2024-03-18T21:40:26.802 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-18T21:40:26.802 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143 --owner scheduled_pdonnell@teuthology --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} -- /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143/orig.config.yaml 2024-03-18T21:40:26.856 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T21:40:26.952 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143 branch: wip-batrick-testing-20240318.181317 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '7609143' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240318.181317 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - PG_AVAILABILITY - PG_DEGRADED - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.dir_split install: ceph: flavor: default sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240318.181317 sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3005 sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 sleep_before_teardown: 0 subset: 433/512 suite: fs suite_branch: wip-batrick-testing-20240318.181317 suite_path: /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN23q79valGzZSCezmtu3H5G8OKoj+SkFhT8hOd86nc1Zovg3u2otRf4t8Mn3Ftf38CLYEaa5077x/XGvfhx+Sw= smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMmiz9uEAhVVpzeDVbCeY00XNBRR3ChEsH59yC2w7169XrawDvSiIBvRQCrtYE05HBaPvA1KDu7DHgxdfJydLm0= smithi113.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFwzKdcwBBvp3d0PPUob2oMiEIIKsMpovmEzwap4MfAZGl6sa4edg2xdBAVqXtWC4dtzB5GC+YIsL6fdlM8lrew= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-18_21:05:49 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-18T21:40:26.952 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa; will attempt to use it 2024-03-18T21:40:26.953 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks 2024-03-18T21:40:26.953 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-18T21:40:26.955 INFO:teuthology.task.internal:Checking packages... 2024-03-18T21:40:26.978 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b6ae24918b03b1b8a19c9263a50655bba80bb810' 2024-03-18T21:40:26.978 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T21:40:26.978 INFO:teuthology.packaging:ref: None 2024-03-18T21:40:26.978 INFO:teuthology.packaging:tag: None 2024-03-18T21:40:26.978 INFO:teuthology.packaging:branch: wip-batrick-testing-20240318.181317 2024-03-18T21:40:26.978 INFO:teuthology.packaging:sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-18T21:40:26.979 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240318.181317 2024-03-18T21:40:27.131 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2206.gb6ae2491 2024-03-18T21:40:27.133 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-18T21:40:27.142 INFO:teuthology.task.internal:no buildpackages task found 2024-03-18T21:40:27.142 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-18T21:40:27.180 INFO:teuthology.task.internal:Saving configuration 2024-03-18T21:40:27.208 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-18T21:40:27.219 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T21:40:27.246 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 21:34:47.270026', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN23q79valGzZSCezmtu3H5G8OKoj+SkFhT8hOd86nc1Zovg3u2otRf4t8Mn3Ftf38CLYEaa5077x/XGvfhx+Sw='} 2024-03-18T21:40:27.269 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 21:34:47.271578', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMmiz9uEAhVVpzeDVbCeY00XNBRR3ChEsH59yC2w7169XrawDvSiIBvRQCrtYE05HBaPvA1KDu7DHgxdfJydLm0='} 2024-03-18T21:40:27.293 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 21:34:47.268353', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFwzKdcwBBvp3d0PPUob2oMiEIIKsMpovmEzwap4MfAZGl6sa4edg2xdBAVqXtWC4dtzB5GC+YIsL6fdlM8lrew='} 2024-03-18T21:40:27.293 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-18T21:40:27.304 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-18T21:40:27.304 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-18T21:40:27.304 INFO:teuthology.task.internal:roles: ubuntu@smithi113.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-18T21:40:27.304 INFO:teuthology.run_tasks:Running task console_log... 2024-03-18T21:40:27.429 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4a3d1eb160>, signals=[15]) 2024-03-18T21:40:27.429 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-18T21:40:27.442 INFO:teuthology.task.internal:Opening connections... 2024-03-18T21:40:27.442 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2024-03-18T21:40:27.444 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T21:40:27.553 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-03-18T21:40:27.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T21:40:27.644 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi113.front.sepia.ceph.com 2024-03-18T21:40:27.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T21:40:27.752 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-18T21:40:27.759 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-03-18T21:40:27.778 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-03-18T21:40:27.778 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-03-18T21:40:27.836 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2024-03-18T21:40:27.836 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="9" 2024-03-18T21:40:27.837 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2024-03-18T21:40:27.837 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2024-03-18T21:40:27.837 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="9" 2024-03-18T21:40:27.837 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el9" 2024-03-18T21:40:27.837 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T21:40:27.837 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2024-03-18T21:40:27.837 INFO:teuthology.orchestra.run.smithi040.stdout:LOGO="fedora-logo-icon" 2024-03-18T21:40:27.837 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T21:40:27.837 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2024-03-18T21:40:27.837 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T21:40:27.837 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T21:40:27.837 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T21:40:27.838 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-03-18T21:40:27.862 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-03-18T21:40:27.879 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-03-18T21:40:27.879 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-03-18T21:40:27.936 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-03-18T21:40:27.937 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-03-18T21:40:27.937 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-03-18T21:40:27.937 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-03-18T21:40:27.937 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-03-18T21:40:27.937 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-03-18T21:40:27.937 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T21:40:27.937 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-03-18T21:40:27.937 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-03-18T21:40:27.937 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T21:40:27.937 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-03-18T21:40:27.937 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T21:40:27.937 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T21:40:27.937 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T21:40:27.938 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-03-18T21:40:27.962 DEBUG:teuthology.orchestra.run.smithi113:> uname -m 2024-03-18T21:40:27.979 INFO:teuthology.orchestra.run.smithi113.stdout:x86_64 2024-03-18T21:40:27.980 DEBUG:teuthology.orchestra.run.smithi113:> cat /etc/os-release 2024-03-18T21:40:28.038 INFO:teuthology.orchestra.run.smithi113.stdout:NAME="CentOS Stream" 2024-03-18T21:40:28.039 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION="9" 2024-03-18T21:40:28.039 INFO:teuthology.orchestra.run.smithi113.stdout:ID="centos" 2024-03-18T21:40:28.039 INFO:teuthology.orchestra.run.smithi113.stdout:ID_LIKE="rhel fedora" 2024-03-18T21:40:28.039 INFO:teuthology.orchestra.run.smithi113.stdout:VERSION_ID="9" 2024-03-18T21:40:28.039 INFO:teuthology.orchestra.run.smithi113.stdout:PLATFORM_ID="platform:el9" 2024-03-18T21:40:28.039 INFO:teuthology.orchestra.run.smithi113.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T21:40:28.039 INFO:teuthology.orchestra.run.smithi113.stdout:ANSI_COLOR="0;31" 2024-03-18T21:40:28.039 INFO:teuthology.orchestra.run.smithi113.stdout:LOGO="fedora-logo-icon" 2024-03-18T21:40:28.039 INFO:teuthology.orchestra.run.smithi113.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T21:40:28.039 INFO:teuthology.orchestra.run.smithi113.stdout:HOME_URL="https://centos.org/" 2024-03-18T21:40:28.039 INFO:teuthology.orchestra.run.smithi113.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T21:40:28.039 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T21:40:28.039 INFO:teuthology.orchestra.run.smithi113.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T21:40:28.040 INFO:teuthology.lock.ops:Updating smithi113.front.sepia.ceph.com on lock server 2024-03-18T21:40:28.062 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-18T21:40:28.074 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-18T21:40:28.096 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-18T21:40:28.097 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2024-03-18T21:40:28.099 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-03-18T21:40:28.101 DEBUG:teuthology.orchestra.run.smithi113:> test '!' -e /home/ubuntu/cephtest 2024-03-18T21:40:28.118 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-18T21:40:28.126 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-18T21:40:28.126 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2024-03-18T21:40:28.157 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-03-18T21:40:28.159 DEBUG:teuthology.orchestra.run.smithi113:> test -z $(ls -A /var/lib/ceph) 2024-03-18T21:40:28.262 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-18T21:40:28.342 INFO:teuthology.run_tasks:Running task kernel... 2024-03-18T21:40:28.361 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-18T21:40:28.361 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-18T21:40:28.361 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T21:40:28.362 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T21:40:28.362 DEBUG:teuthology.orchestra.run.smithi113:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T21:40:28.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T21:40:28.379 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-03-18T21:40:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T21:40:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T21:40:28.381 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-03-18T21:40:28.381 DEBUG:teuthology.orchestra.run.smithi113:> uname -r 2024-03-18T21:40:28.437 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-361.el9.x86_64 2024-03-18T21:40:28.437 INFO:teuthology.task.kernel:Running kernel on smithi103: 5.14.0-361.el9.x86_64 2024-03-18T21:40:28.437 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum install -y kernel 2024-03-18T21:40:28.438 INFO:teuthology.orchestra.run.smithi113.stdout:5.14.0-361.el9.x86_64 2024-03-18T21:40:28.438 INFO:teuthology.task.kernel:Running kernel on smithi113: 5.14.0-361.el9.x86_64 2024-03-18T21:40:28.438 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum install -y kernel 2024-03-18T21:40:28.439 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-361.el9.x86_64 2024-03-18T21:40:28.440 INFO:teuthology.task.kernel:Running kernel on smithi040: 5.14.0-361.el9.x86_64 2024-03-18T21:40:28.440 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum install -y kernel 2024-03-18T21:40:30.556 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-18T21:40:30.608 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-18T21:40:30.728 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-18T21:40:31.940 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-18T21:40:32.025 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-03-18T21:40:32.354 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-18T21:40:33.549 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 6.8 MB/s | 8.0 MB 00:01 2024-03-18T21:40:33.564 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - BaseOS 6.2 MB/s | 8.0 MB 00:01 2024-03-18T21:40:37.736 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.0 MB 00:05 2024-03-18T21:40:42.187 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 19 MB 00:03 2024-03-18T21:40:50.279 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-18T21:40:51.516 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:16 2024-03-18T21:40:56.256 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.1 MB 00:01 2024-03-18T21:40:56.795 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.1 MB 00:03 2024-03-18T21:40:57.742 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-18T21:40:58.254 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-03-18T21:40:59.104 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-18T21:40:59.314 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-18T21:41:04.172 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 22 kB/s | 1.7 kB 00:00 2024-03-18T21:41:04.227 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-03-18T21:41:06.278 INFO:teuthology.orchestra.run.smithi113.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-18T21:41:06.278 INFO:teuthology.orchestra.run.smithi113.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-18T21:41:06.366 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-18T21:41:06.371 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-18T21:41:06.371 INFO:teuthology.orchestra.run.smithi113.stdout: Package Architecture Version Repository Size 2024-03-18T21:41:06.371 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-18T21:41:06.371 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-18T21:41:06.371 INFO:teuthology.orchestra.run.smithi113.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-18T21:41:06.371 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-03-18T21:41:06.372 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-18T21:41:06.372 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-18T21:41:06.372 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-18T21:41:06.372 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T21:41:06.372 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-18T21:41:06.372 INFO:teuthology.orchestra.run.smithi113.stdout:================================================================================ 2024-03-18T21:41:06.372 INFO:teuthology.orchestra.run.smithi113.stdout:Install 4 Packages 2024-03-18T21:41:06.372 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T21:41:06.373 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 76 M 2024-03-18T21:41:06.373 INFO:teuthology.orchestra.run.smithi113.stdout:Installed size: 123 M 2024-03-18T21:41:06.373 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-18T21:41:06.392 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-18T21:41:06.392 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-18T21:41:06.449 INFO:teuthology.orchestra.run.smithi113.stdout:[MIRROR] kernel-core-5.14.0-430.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-430.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-18T21:41:06.449 INFO:teuthology.orchestra.run.smithi113.stdout:[MIRROR] kernel-modules-5.14.0-430.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-430.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-18T21:41:06.450 INFO:teuthology.orchestra.run.smithi113.stdout:[MIRROR] kernel-5.14.0-430.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-430.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-18T21:41:06.486 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-03-18T21:41:06.493 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-03-18T21:41:06.493 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-03-18T21:41:06.493 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-03-18T21:41:06.493 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-03-18T21:41:06.493 INFO:teuthology.orchestra.run.smithi040.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-18T21:41:06.493 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-03-18T21:41:06.493 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-18T21:41:06.493 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-18T21:41:06.494 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-18T21:41:06.494 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T21:41:06.494 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-03-18T21:41:06.494 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-03-18T21:41:06.494 INFO:teuthology.orchestra.run.smithi040.stdout:Install 4 Packages 2024-03-18T21:41:06.494 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T21:41:06.494 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 76 M 2024-03-18T21:41:06.495 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 123 M 2024-03-18T21:41:06.495 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-03-18T21:41:06.582 INFO:teuthology.orchestra.run.smithi113.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 266 kB/s | 44 kB 00:00 2024-03-18T21:41:06.595 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] kernel-5.14.0-430.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-430.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-18T21:41:06.597 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] kernel-core-5.14.0-430.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-430.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-18T21:41:06.597 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] kernel-modules-5.14.0-430.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-430.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-18T21:41:06.937 INFO:teuthology.orchestra.run.smithi040.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 118 kB/s | 44 kB 00:00 2024-03-18T21:41:08.865 INFO:teuthology.orchestra.run.smithi113.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 6.1 MB/s | 15 MB 00:02 2024-03-18T21:41:10.689 INFO:teuthology.orchestra.run.smithi113.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 7.8 MB/s | 33 MB 00:04 2024-03-18T21:41:10.940 INFO:teuthology.orchestra.run.smithi113.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 6.3 MB/s | 27 MB 00:04 2024-03-18T21:41:10.940 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-18T21:41:10.940 INFO:teuthology.orchestra.run.smithi113.stdout:Total 17 MB/s | 76 MB 00:04 2024-03-18T21:41:11.607 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-18T21:41:11.983 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-18T21:41:11.983 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-18T21:41:12.974 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-18T21:41:12.975 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-18T21:41:15.425 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-18T21:41:16.147 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-18T21:41:16.232 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-18T21:41:18.971 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-18T21:41:19.117 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-18T21:41:19.419 INFO:teuthology.orchestra.run.smithi040.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 1.2 MB/s | 15 MB 00:12 2024-03-18T21:41:23.863 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-18T21:41:23.967 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-18T21:41:28.736 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-18T21:41:37.301 INFO:teuthology.orchestra.run.smithi040.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 926 kB/s | 27 MB 00:30 2024-03-18T21:41:43.650 INFO:teuthology.orchestra.run.smithi040.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 923 kB/s | 33 MB 00:37 2024-03-18T21:41:43.651 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-03-18T21:41:43.651 INFO:teuthology.orchestra.run.smithi040.stdout:Total 2.0 MB/s | 76 MB 00:37 2024-03-18T21:41:44.343 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-03-18T21:41:44.776 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-03-18T21:41:44.776 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-03-18T21:41:45.806 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-03-18T21:41:45.806 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-03-18T21:41:48.414 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-03-18T21:41:49.056 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-18T21:41:49.159 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-18T21:41:52.081 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-18T21:41:52.203 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-18T21:41:57.311 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-18T21:41:57.344 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-18T21:41:57.889 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-18T21:41:57.889 INFO:teuthology.orchestra.run.smithi113.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-18T21:41:57.889 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T21:41:57.933 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-18T21:41:58.364 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-18T21:41:58.364 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-18T21:41:58.364 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-18T21:41:58.365 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-18T21:41:58.800 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-18T21:41:58.800 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T21:41:58.800 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-18T21:41:58.800 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-18T21:41:58.800 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-18T21:41:58.800 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-18T21:41:58.800 INFO:teuthology.orchestra.run.smithi113.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-18T21:41:58.800 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T21:41:58.800 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-18T21:41:59.119 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T21:41:59.213 INFO:teuthology.orchestra.run.smithi113.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-18T21:41:59.214 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-18T21:41:59.214 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-18T21:41:59.214 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-18T21:41:59.214 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-18T21:41:59.214 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-18T21:41:59.214 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi113.front.sepia.ceph.com, path=None, version=distro) 2024-03-18T21:41:59.214 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T21:41:59.295 INFO:teuthology.orchestra.run.smithi113.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-18T21:41:59.295 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-18T21:41:59.296 DEBUG:teuthology.orchestra.run.smithi113:> sudo rpm -qi grub2-tools 2024-03-18T21:41:59.328 INFO:teuthology.orchestra.run.smithi113.stdout:Name : grub2-tools 2024-03-18T21:41:59.329 INFO:teuthology.orchestra.run.smithi113.stdout:Epoch : 1 2024-03-18T21:41:59.329 INFO:teuthology.orchestra.run.smithi113.stdout:Version : 2.06 2024-03-18T21:41:59.329 INFO:teuthology.orchestra.run.smithi113.stdout:Release : 68.el9 2024-03-18T21:41:59.329 INFO:teuthology.orchestra.run.smithi113.stdout:Architecture: x86_64 2024-03-18T21:41:59.329 INFO:teuthology.orchestra.run.smithi113.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-18T21:41:59.329 INFO:teuthology.orchestra.run.smithi113.stdout:Group : Unspecified 2024-03-18T21:41:59.329 INFO:teuthology.orchestra.run.smithi113.stdout:Size : 8264254 2024-03-18T21:41:59.329 INFO:teuthology.orchestra.run.smithi113.stdout:License : GPLv3+ 2024-03-18T21:41:59.329 INFO:teuthology.orchestra.run.smithi113.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-18T21:41:59.329 INFO:teuthology.orchestra.run.smithi113.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-18T21:41:59.329 INFO:teuthology.orchestra.run.smithi113.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-18T21:41:59.329 INFO:teuthology.orchestra.run.smithi113.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-18T21:41:59.329 INFO:teuthology.orchestra.run.smithi113.stdout:Packager : builder@centos.org 2024-03-18T21:41:59.329 INFO:teuthology.orchestra.run.smithi113.stdout:Vendor : CentOS 2024-03-18T21:41:59.329 INFO:teuthology.orchestra.run.smithi113.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T21:41:59.330 INFO:teuthology.orchestra.run.smithi113.stdout:Summary : Support tools for GRUB. 2024-03-18T21:41:59.330 INFO:teuthology.orchestra.run.smithi113.stdout:Description : 2024-03-18T21:41:59.330 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T21:41:59.330 INFO:teuthology.orchestra.run.smithi113.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T21:41:59.330 INFO:teuthology.orchestra.run.smithi113.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T21:41:59.330 INFO:teuthology.orchestra.run.smithi113.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T21:41:59.330 INFO:teuthology.orchestra.run.smithi113.stdout:hardware devices. 2024-03-18T21:41:59.330 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T21:41:59.330 INFO:teuthology.orchestra.run.smithi113.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T21:41:59.331 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T21:41:59.331 INFO:teuthology.task.kernel:Updating grub on smithi113 to boot 5.14.0-430.el9.x86_64 2024-03-18T21:41:59.331 DEBUG:teuthology.orchestra.run.smithi113:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T21:41:59.922 INFO:teuthology.orchestra.run.smithi113.stderr:Generating grub configuration file ... 2024-03-18T21:42:01.247 INFO:teuthology.orchestra.run.smithi113.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-18T21:42:01.303 INFO:teuthology.orchestra.run.smithi113.stderr:done 2024-03-18T21:42:01.305 DEBUG:teuthology.orchestra.run.smithi113:> mktemp 2024-03-18T21:42:01.320 INFO:teuthology.orchestra.run.smithi113.stdout:/tmp/tmp.KhCg6nhOoC 2024-03-18T21:42:01.320 DEBUG:teuthology.orchestra.run.smithi113:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KhCg6nhOoC 2024-03-18T21:42:01.389 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 0666 /tmp/tmp.KhCg6nhOoC 2024-03-18T21:42:01.520 DEBUG:teuthology.orchestra.remote:smithi113:/tmp/tmp.KhCg6nhOoC is 6KB 2024-03-18T21:42:01.569 DEBUG:teuthology.orchestra.run.smithi113:> rm -fr /tmp/tmp.KhCg6nhOoC 2024-03-18T21:42:01.583 DEBUG:teuthology.orchestra.run.smithi113:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T21:42:01.649 INFO:teuthology.orchestra.run.smithi113.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-18T21:42:01.649 INFO:teuthology.orchestra.run.smithi113.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-18T21:42:01.649 INFO:teuthology.orchestra.run.smithi113.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-18T21:42:01.649 INFO:teuthology.orchestra.run.smithi113.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-18T21:42:01.650 DEBUG:teuthology.orchestra.run.smithi113:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-18T21:42:01.796 DEBUG:teuthology.orchestra.run.smithi113:> sudo shutdown -r now 2024-03-18T21:42:02.481 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-18T21:42:31.799 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-18T21:42:31.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-18T21:42:31.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T21:42:33.478 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-18T21:42:33.478 INFO:teuthology.orchestra.run.smithi040.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-18T21:42:33.479 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T21:42:33.524 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-18T21:42:33.923 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-18T21:42:33.923 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-18T21:42:33.923 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-18T21:42:33.923 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-18T21:42:34.350 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-18T21:42:34.350 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T21:42:34.350 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-03-18T21:42:34.350 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-18T21:42:34.351 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-18T21:42:34.351 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-18T21:42:34.351 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-18T21:42:34.351 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T21:42:34.351 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-03-18T21:42:34.676 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T21:42:34.774 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-18T21:42:34.774 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-18T21:42:34.774 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-18T21:42:34.774 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-18T21:42:34.775 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-18T21:42:34.775 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-18T21:42:34.775 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi040.front.sepia.ceph.com, path=None, version=distro) 2024-03-18T21:42:34.775 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T21:42:34.858 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-18T21:42:34.858 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-18T21:42:34.858 DEBUG:teuthology.orchestra.run.smithi040:> sudo rpm -qi grub2-tools 2024-03-18T21:42:34.896 INFO:teuthology.orchestra.run.smithi040.stdout:Name : grub2-tools 2024-03-18T21:42:34.896 INFO:teuthology.orchestra.run.smithi040.stdout:Epoch : 1 2024-03-18T21:42:34.896 INFO:teuthology.orchestra.run.smithi040.stdout:Version : 2.06 2024-03-18T21:42:34.896 INFO:teuthology.orchestra.run.smithi040.stdout:Release : 68.el9 2024-03-18T21:42:34.896 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: x86_64 2024-03-18T21:42:34.896 INFO:teuthology.orchestra.run.smithi040.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-18T21:42:34.896 INFO:teuthology.orchestra.run.smithi040.stdout:Group : Unspecified 2024-03-18T21:42:34.897 INFO:teuthology.orchestra.run.smithi040.stdout:Size : 8264254 2024-03-18T21:42:34.897 INFO:teuthology.orchestra.run.smithi040.stdout:License : GPLv3+ 2024-03-18T21:42:34.897 INFO:teuthology.orchestra.run.smithi040.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-18T21:42:34.897 INFO:teuthology.orchestra.run.smithi040.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-18T21:42:34.897 INFO:teuthology.orchestra.run.smithi040.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-18T21:42:34.897 INFO:teuthology.orchestra.run.smithi040.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-18T21:42:34.897 INFO:teuthology.orchestra.run.smithi040.stdout:Packager : builder@centos.org 2024-03-18T21:42:34.897 INFO:teuthology.orchestra.run.smithi040.stdout:Vendor : CentOS 2024-03-18T21:42:34.897 INFO:teuthology.orchestra.run.smithi040.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T21:42:34.897 INFO:teuthology.orchestra.run.smithi040.stdout:Summary : Support tools for GRUB. 2024-03-18T21:42:34.897 INFO:teuthology.orchestra.run.smithi040.stdout:Description : 2024-03-18T21:42:34.897 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T21:42:34.897 INFO:teuthology.orchestra.run.smithi040.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T21:42:34.897 INFO:teuthology.orchestra.run.smithi040.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T21:42:34.898 INFO:teuthology.orchestra.run.smithi040.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T21:42:34.898 INFO:teuthology.orchestra.run.smithi040.stdout:hardware devices. 2024-03-18T21:42:34.898 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T21:42:34.898 INFO:teuthology.orchestra.run.smithi040.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T21:42:34.899 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T21:42:34.899 INFO:teuthology.task.kernel:Updating grub on smithi040 to boot 5.14.0-430.el9.x86_64 2024-03-18T21:42:34.899 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T21:42:35.523 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2024-03-18T21:42:36.937 INFO:teuthology.orchestra.run.smithi040.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-18T21:42:36.972 INFO:teuthology.orchestra.run.smithi040.stderr:done 2024-03-18T21:42:36.974 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-03-18T21:42:36.989 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.nGSFe0oyVB 2024-03-18T21:42:36.989 DEBUG:teuthology.orchestra.run.smithi040:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.nGSFe0oyVB 2024-03-18T21:42:37.065 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.nGSFe0oyVB 2024-03-18T21:42:37.205 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.nGSFe0oyVB is 6KB 2024-03-18T21:42:37.253 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.nGSFe0oyVB 2024-03-18T21:42:37.268 DEBUG:teuthology.orchestra.run.smithi040:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T21:42:37.341 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-18T21:42:37.341 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-18T21:42:37.342 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-18T21:42:37.342 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-18T21:42:37.343 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-18T21:42:37.494 DEBUG:teuthology.orchestra.run.smithi040:> sudo shutdown -r now 2024-03-18T21:42:50.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-03-18T21:42:59.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-18T21:42:59.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T21:43:02.210 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-03-18T21:43:07.497 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-18T21:43:07.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-18T21:43:07.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T21:43:14.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-18T21:43:14.212 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T21:43:17.282 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-03-18T21:43:25.982 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2024-03-18T21:43:32.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-18T21:43:32.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T21:43:34.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-18T21:43:34.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T21:43:50.814 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.113 2024-03-18T21:44:08.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-18T21:44:08.817 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T21:44:09.151 DEBUG:teuthology.orchestra.run.smithi113:> true 2024-03-18T21:44:09.516 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-18T21:44:09.516 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-430.el9.x86_64"... 2024-03-18T21:44:09.517 DEBUG:teuthology.orchestra.run.smithi113:> uname -r 2024-03-18T21:44:09.532 INFO:teuthology.orchestra.run.smithi113.stdout:5.14.0-430.el9.x86_64 2024-03-18T21:44:09.532 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-18T21:44:09.532 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T21:44:09.532 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T21:44:10.533 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-18T21:44:10.533 DEBUG:teuthology.orchestra.run.smithi113:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-18T21:44:10.605 INFO:teuthology.orchestra.run.smithi113.stdout:ttyS1 2024-03-18T21:44:10.634 DEBUG:teuthology.parallel:result is None 2024-03-18T21:44:34.986 DEBUG:teuthology.orchestra.remote:timed out 2024-03-18T21:44:46.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-18T21:44:46.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T21:44:47.260 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-03-18T21:44:47.662 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-18T21:44:47.662 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-18T21:44:47.662 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-03-18T21:44:47.677 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-430.el9.x86_64 2024-03-18T21:44:47.677 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-18T21:44:47.677 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T21:44:47.677 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T21:44:48.678 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-18T21:44:48.678 DEBUG:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-18T21:44:48.768 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2024-03-18T21:44:48.804 DEBUG:teuthology.parallel:result is None 2024-03-18T21:53:50.263 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 8.0 kB/s | 6.1 MB 13:04 2024-03-18T21:53:52.189 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-03-18T21:53:53.242 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-18T21:53:58.265 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 7.3 kB/s | 1.7 kB 00:00 2024-03-18T21:54:00.372 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-18T21:54:00.372 INFO:teuthology.orchestra.run.smithi103.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-18T21:54:00.458 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-18T21:54:00.465 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-18T21:54:00.465 INFO:teuthology.orchestra.run.smithi103.stdout: Package Architecture Version Repository Size 2024-03-18T21:54:00.465 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-18T21:54:00.466 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-18T21:54:00.466 INFO:teuthology.orchestra.run.smithi103.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-18T21:54:00.466 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-18T21:54:00.466 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-18T21:54:00.466 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-18T21:54:00.466 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-18T21:54:00.466 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T21:54:00.466 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-18T21:54:00.466 INFO:teuthology.orchestra.run.smithi103.stdout:================================================================================ 2024-03-18T21:54:00.466 INFO:teuthology.orchestra.run.smithi103.stdout:Install 4 Packages 2024-03-18T21:54:00.466 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T21:54:00.467 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 76 M 2024-03-18T21:54:00.467 INFO:teuthology.orchestra.run.smithi103.stdout:Installed size: 123 M 2024-03-18T21:54:00.467 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-18T21:54:00.623 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-430.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-430.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-18T21:54:00.624 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-430.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-430.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-18T21:54:00.625 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-430.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-430.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-18T21:56:09.429 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-18T21:56:09.430 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-18T21:56:09.430 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-18T21:56:09.595 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-430.el9.x86_64.rpm: Status code: 404 for http://mirror.cpsc.ucalgary.ca/mirror/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-430.el9.x86_64.rpm (IP: 136.159.208.97) 2024-03-18T21:56:09.599 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-430.el9.x86_64.rpm: Status code: 404 for http://mirror.cpsc.ucalgary.ca/mirror/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-430.el9.x86_64.rpm (IP: 136.159.208.97) 2024-03-18T21:56:09.599 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-430.el9.x86_64.rpm: Status code: 404 for http://mirror.cpsc.ucalgary.ca/mirror/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-430.el9.x86_64.rpm (IP: 136.159.208.97) 2024-03-18T21:58:20.425 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-18T21:58:20.425 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-18T21:58:20.425 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-core-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-18T21:58:23.222 INFO:teuthology.orchestra.run.smithi103.stdout:(1/4): kernel-core-5.14.0-430.el9.x86_64.rpm 58 kB/s | 15 MB 04:22 2024-03-18T21:58:23.391 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-430.el9.x86_64.rpm: Status code: 404 for http://mirror.cpsc.ucalgary.ca/mirror/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-430.el9.x86_64.rpm (IP: 136.159.208.97) 2024-03-18T21:58:23.426 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-modules-core-5.14.0-430.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-430.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-18T21:58:25.797 INFO:teuthology.orchestra.run.smithi103.stdout:(2/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 129 kB/s | 33 MB 04:25 2024-03-18T21:58:26.322 INFO:teuthology.orchestra.run.smithi103.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 8.9 MB/s | 27 MB 00:03 2024-03-18T22:00:31.382 INFO:teuthology.orchestra.run.smithi103.stdout:[MIRROR] kernel-5.14.0-430.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-430.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-18T22:00:31.583 INFO:teuthology.orchestra.run.smithi103.stdout:(4/4): kernel-5.14.0-430.el9.x86_64.rpm 116 B/s | 44 kB 06:30 2024-03-18T22:00:31.583 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-18T22:00:31.583 INFO:teuthology.orchestra.run.smithi103.stdout:Total 199 kB/s | 76 MB 06:31 2024-03-18T22:00:32.245 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-18T22:00:32.667 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-18T22:00:32.667 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-18T22:00:33.654 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-18T22:00:33.654 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-18T22:00:36.129 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-18T22:00:36.960 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-18T22:00:37.050 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-18T22:00:39.816 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-18T22:00:39.954 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-18T22:00:44.664 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-18T22:00:44.762 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-18T22:00:49.478 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-18T22:01:18.757 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-18T22:01:18.757 INFO:teuthology.orchestra.run.smithi103.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-18T22:01:18.757 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:01:18.800 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-18T22:01:19.169 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-18T22:01:19.169 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-18T22:01:19.169 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-18T22:01:19.169 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-18T22:01:19.614 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-18T22:01:19.614 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:01:19.614 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-18T22:01:19.614 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-18T22:01:19.615 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-18T22:01:19.615 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-18T22:01:19.615 INFO:teuthology.orchestra.run.smithi103.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-18T22:01:19.615 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:01:19.615 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-18T22:01:19.967 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T22:01:20.074 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-18T22:01:20.075 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-18T22:01:20.075 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-18T22:01:20.075 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-18T22:01:20.075 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-18T22:01:20.075 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-18T22:01:20.075 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=None, version=distro) 2024-03-18T22:01:20.075 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T22:01:20.154 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-18T22:01:20.154 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-18T22:01:20.155 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -qi grub2-tools 2024-03-18T22:01:20.198 INFO:teuthology.orchestra.run.smithi103.stdout:Name : grub2-tools 2024-03-18T22:01:20.198 INFO:teuthology.orchestra.run.smithi103.stdout:Epoch : 1 2024-03-18T22:01:20.199 INFO:teuthology.orchestra.run.smithi103.stdout:Version : 2.06 2024-03-18T22:01:20.199 INFO:teuthology.orchestra.run.smithi103.stdout:Release : 68.el9 2024-03-18T22:01:20.199 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: x86_64 2024-03-18T22:01:20.199 INFO:teuthology.orchestra.run.smithi103.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-18T22:01:20.199 INFO:teuthology.orchestra.run.smithi103.stdout:Group : Unspecified 2024-03-18T22:01:20.199 INFO:teuthology.orchestra.run.smithi103.stdout:Size : 8264254 2024-03-18T22:01:20.199 INFO:teuthology.orchestra.run.smithi103.stdout:License : GPLv3+ 2024-03-18T22:01:20.199 INFO:teuthology.orchestra.run.smithi103.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-18T22:01:20.199 INFO:teuthology.orchestra.run.smithi103.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-18T22:01:20.199 INFO:teuthology.orchestra.run.smithi103.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-18T22:01:20.199 INFO:teuthology.orchestra.run.smithi103.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-18T22:01:20.199 INFO:teuthology.orchestra.run.smithi103.stdout:Packager : builder@centos.org 2024-03-18T22:01:20.199 INFO:teuthology.orchestra.run.smithi103.stdout:Vendor : CentOS 2024-03-18T22:01:20.199 INFO:teuthology.orchestra.run.smithi103.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T22:01:20.199 INFO:teuthology.orchestra.run.smithi103.stdout:Summary : Support tools for GRUB. 2024-03-18T22:01:20.200 INFO:teuthology.orchestra.run.smithi103.stdout:Description : 2024-03-18T22:01:20.200 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:01:20.200 INFO:teuthology.orchestra.run.smithi103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T22:01:20.200 INFO:teuthology.orchestra.run.smithi103.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T22:01:20.200 INFO:teuthology.orchestra.run.smithi103.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T22:01:20.200 INFO:teuthology.orchestra.run.smithi103.stdout:hardware devices. 2024-03-18T22:01:20.200 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:01:20.200 INFO:teuthology.orchestra.run.smithi103.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T22:01:20.201 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T22:01:20.201 INFO:teuthology.task.kernel:Updating grub on smithi103 to boot 5.14.0-430.el9.x86_64 2024-03-18T22:01:20.201 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T22:01:20.760 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2024-03-18T22:01:22.133 INFO:teuthology.orchestra.run.smithi103.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-18T22:01:22.167 INFO:teuthology.orchestra.run.smithi103.stderr:done 2024-03-18T22:01:22.169 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-03-18T22:01:22.184 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.j1mte5pKJO 2024-03-18T22:01:22.184 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.j1mte5pKJO 2024-03-18T22:01:22.253 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.j1mte5pKJO 2024-03-18T22:01:22.391 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.j1mte5pKJO is 6KB 2024-03-18T22:01:22.440 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.j1mte5pKJO 2024-03-18T22:01:22.454 DEBUG:teuthology.orchestra.run.smithi103:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T22:01:22.519 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-18T22:01:22.519 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-18T22:01:22.519 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-18T22:01:22.519 INFO:teuthology.orchestra.run.smithi103.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-18T22:01:22.521 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-18T22:01:22.680 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2024-03-18T22:01:52.684 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-18T22:01:52.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-18T22:01:52.685 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T22:02:27.230 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-03-18T22:02:36.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-18T22:02:36.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T22:02:39.298 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-03-18T22:02:51.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-18T22:02:51.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T22:03:23.279 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-03-18T22:03:23.817 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-18T22:03:23.817 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-18T22:03:23.817 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-03-18T22:03:23.842 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-430.el9.x86_64 2024-03-18T22:03:23.842 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-18T22:03:23.842 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T22:03:23.842 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T22:03:24.843 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-18T22:03:24.844 DEBUG:teuthology.orchestra.run.smithi103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-18T22:03:24.901 INFO:teuthology.orchestra.run.smithi103.stdout:ttyS1 2024-03-18T22:03:24.930 DEBUG:teuthology.parallel:result is None 2024-03-18T22:03:24.931 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-18T22:03:24.939 INFO:teuthology.task.internal:Creating test directory... 2024-03-18T22:03:24.939 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T22:03:24.942 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T22:03:24.944 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T22:03:24.963 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-18T22:03:24.970 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-18T22:03:24.998 INFO:teuthology.task.internal:Creating archive directory... 2024-03-18T22:03:24.999 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T22:03:25.001 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T22:03:25.003 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T22:03:25.102 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-18T22:03:25.109 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-18T22:03:25.109 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T22:03:25.111 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T22:03:25.114 DEBUG:teuthology.orchestra.run.smithi113:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T22:03:25.148 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T22:03:25.152 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T22:03:25.156 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T22:03:25.161 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T22:03:25.166 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T22:03:25.170 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T22:03:25.172 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-18T22:03:25.179 INFO:teuthology.task.internal:Configuring sudo... 2024-03-18T22:03:25.179 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T22:03:25.211 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T22:03:25.213 DEBUG:teuthology.orchestra.run.smithi113:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T22:03:25.243 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-18T22:03:25.252 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-18T22:03:25.252 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T22:03:25.281 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T22:03:25.283 DEBUG:teuthology.orchestra.run.smithi113:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T22:03:25.302 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T22:03:25.378 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T22:03:25.444 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T22:03:25.481 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T22:03:25.530 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:03:25.530 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T22:03:25.598 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T22:03:25.639 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T22:03:25.708 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T22:03:25.746 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T22:03:25.791 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-18T22:03:25.791 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T22:03:25.858 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T22:03:25.901 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T22:03:26.034 DEBUG:teuthology.orchestra.run.smithi113:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T22:03:26.075 DEBUG:teuthology.orchestra.run.smithi113:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T22:03:26.123 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-18T22:03:26.123 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T22:03:26.193 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2024-03-18T22:03:26.195 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-03-18T22:03:26.197 DEBUG:teuthology.orchestra.run.smithi113:> sudo service rsyslog restart 2024-03-18T22:03:26.240 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T22:03:26.244 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T22:03:26.293 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T22:03:26.758 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-18T22:03:26.769 INFO:teuthology.task.internal:Starting timer... 2024-03-18T22:03:26.769 INFO:teuthology.run_tasks:Running task pcp... 2024-03-18T22:03:26.811 INFO:teuthology.run_tasks:Running task selinux... 2024-03-18T22:03:26.826 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-18T22:03:26.826 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2024-03-18T22:03:26.923 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2024-03-18T22:03:26.925 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-03-18T22:03:26.975 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-03-18T22:03:26.977 DEBUG:teuthology.orchestra.run.smithi113:> sudo service auditd rotate 2024-03-18T22:03:27.063 INFO:teuthology.orchestra.run.smithi113.stdout:Rotating logs: 2024-03-18T22:03:27.065 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-18T22:03:27.065 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2024-03-18T22:03:27.088 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2024-03-18T22:03:27.088 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-03-18T22:03:27.106 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-03-18T22:03:27.106 DEBUG:teuthology.orchestra.run.smithi113:> /usr/sbin/getenforce 2024-03-18T22:03:27.131 INFO:teuthology.orchestra.run.smithi113.stdout:Permissive 2024-03-18T22:03:27.131 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi040.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi103.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi113.front.sepia.ceph.com': 'permissive'} 2024-03-18T22:03:27.132 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T22:03:27.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T22:03:27.160 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T22:03:27.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T22:03:27.188 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T22:03:27.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T22:03:27.217 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-18T22:03:27.217 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2024-03-18T22:03:27.255 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-03-18T22:03:27.291 DEBUG:teuthology.orchestra.run.smithi113:> sudo /usr/sbin/setenforce permissive 2024-03-18T22:03:27.324 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-18T22:03:27.333 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-18T22:03:27.342 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-18T22:03:27.459 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-18T22:03:27.473 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-18T22:03:27.475 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi040.front.sepia.ceph.com,smithi103.front.sepia.ceph.com,smithi113.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-18T22:16:27.162 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi103.front.sepia.ceph.com'), Remote(name='ubuntu@smithi113.front.sepia.ceph.com')] 2024-03-18T22:16:27.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-18T22:16:27.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T22:16:27.257 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-03-18T22:16:27.332 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-18T22:16:27.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-18T22:16:27.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T22:16:27.419 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-03-18T22:16:27.497 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-03-18T22:16:27.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-18T22:16:27.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi113.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T22:16:27.584 DEBUG:teuthology.orchestra.run.smithi113:> true 2024-03-18T22:16:27.670 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi113.front.sepia.ceph.com' 2024-03-18T22:16:27.670 INFO:teuthology.run_tasks:Running task clock... 2024-03-18T22:16:27.680 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-18T22:16:27.680 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T22:16:27.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T22:16:27.683 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T22:16:27.683 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T22:16:27.686 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T22:16:27.686 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T22:16:27.714 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T22:16:27.716 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T22:16:27.731 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T22:16:27.733 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T22:16:27.743 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T22:16:27.760 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T22:16:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2024-03-18T22:16:27.763 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-03-18T22:16:27.775 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2024-03-18T22:16:27.777 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-03-18T22:16:27.789 INFO:teuthology.orchestra.run.smithi113.stderr:sudo: ntpd: command not found 2024-03-18T22:16:27.792 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T22:16:27.793 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T22:16:27.803 INFO:teuthology.orchestra.run.smithi113.stdout:506 Cannot talk to daemon 2024-03-18T22:16:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T22:16:27.810 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T22:16:27.820 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T22:16:27.836 INFO:teuthology.orchestra.run.smithi113.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T22:16:27.862 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-03-18T22:16:27.863 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-03-18T22:16:27.865 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T22:16:27.865 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-03-18T22:16:27.865 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T22:16:27.865 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T22:16:27.865 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T22:16:27.866 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T22:16:27.866 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T22:16:27.866 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-03-18T22:16:27.866 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T22:16:27.867 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T22:16:27.867 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T22:16:27.867 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T22:16:27.889 INFO:teuthology.orchestra.run.smithi113.stderr:bash: line 1: ntpq: command not found 2024-03-18T22:16:27.892 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T22:16:27.892 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2024-03-18T22:16:27.892 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T22:16:27.892 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T22:16:27.892 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T22:16:27.892 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T22:16:27.892 INFO:teuthology.run_tasks:Running task install... 2024-03-18T22:16:27.902 DEBUG:teuthology.task.install:project ceph 2024-03-18T22:16:27.903 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}} 2024-03-18T22:16:27.903 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'} 2024-03-18T22:16:27.903 INFO:teuthology.task.install:Using flavor: default 2024-03-18T22:16:27.908 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-18T22:16:27.909 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-18T22:16:27.909 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-18T22:16:27.909 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-18T22:16:27.911 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-18T22:16:27.912 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-18T22:16:27.912 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-18T22:16:27.913 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-18T22:16:28.055 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-18T22:16:28.055 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-18T22:16:28.132 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-18T22:16:28.132 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-18T22:16:28.136 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-18T22:16:28.136 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-18T22:16:28.232 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T22:16:28.232 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-18T22:16:28.232 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T22:16:28.246 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T22:16:28.246 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-18T22:16:28.246 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T22:16:28.260 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T22:16:28.260 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T22:16:28.273 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T22:16:28.273 DEBUG:teuthology.orchestra.run.smithi113:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T22:16:28.300 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T22:16:28.300 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:16:28.300 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T22:16:28.328 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T22:16:28.329 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T22:16:28.329 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T22:16:28.340 DEBUG:teuthology.orchestra.run.smithi113:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T22:16:28.398 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T22:16:28.407 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T22:16:28.419 DEBUG:teuthology.orchestra.run.smithi113:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T22:16:28.473 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-03-18T22:16:28.475 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-03-18T22:16:28.478 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T22:16:28.486 INFO:teuthology.orchestra.run.smithi113.stdout:check_obsoletes = 1 2024-03-18T22:16:28.488 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum clean all 2024-03-18T22:16:28.544 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2024-03-18T22:16:28.545 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2024-03-18T22:16:28.733 INFO:teuthology.orchestra.run.smithi103.stdout:74 files removed 2024-03-18T22:16:28.737 INFO:teuthology.orchestra.run.smithi113.stdout:74 files removed 2024-03-18T22:16:28.760 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T22:16:28.764 DEBUG:teuthology.orchestra.run.smithi113:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T22:16:28.798 INFO:teuthology.orchestra.run.smithi040.stdout:74 files removed 2024-03-18T22:16:28.825 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T22:16:29.460 INFO:teuthology.orchestra.run.smithi113.stdout:ceph packages for x86_64 210 kB/s | 83 kB 00:00 2024-03-18T22:16:29.473 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 200 kB/s | 83 kB 00:00 2024-03-18T22:16:29.594 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 179 kB/s | 83 kB 00:00 2024-03-18T22:16:29.827 INFO:teuthology.orchestra.run.smithi113.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-18T22:16:29.882 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 28 kB/s | 10 kB 00:00 2024-03-18T22:16:30.036 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-18T22:16:30.153 INFO:teuthology.orchestra.run.smithi113.stdout:ceph source packages 6.6 kB/s | 1.9 kB 00:00 2024-03-18T22:16:30.274 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-03-18T22:16:30.478 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-03-18T22:16:30.572 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-18T22:16:30.693 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-18T22:16:30.939 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-18T22:16:31.947 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-18T22:16:32.169 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-18T22:16:32.398 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-18T22:16:33.538 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - BaseOS 6.3 MB/s | 8.0 MB 00:01 2024-03-18T22:16:50.053 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-18T22:16:54.770 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.1 MB 00:01 2024-03-18T22:16:56.204 INFO:teuthology.orchestra.run.smithi113.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-18T22:16:57.292 INFO:teuthology.orchestra.run.smithi113.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-18T22:17:02.164 INFO:teuthology.orchestra.run.smithi113.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-18T22:17:04.603 INFO:teuthology.orchestra.run.smithi113.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T22:17:04.604 INFO:teuthology.orchestra.run.smithi113.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T22:17:04.618 INFO:teuthology.orchestra.run.smithi113.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T22:17:04.619 INFO:teuthology.orchestra.run.smithi113.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T22:17:04.620 INFO:teuthology.orchestra.run.smithi113.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T22:17:04.621 INFO:teuthology.orchestra.run.smithi113.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T22:17:04.623 INFO:teuthology.orchestra.run.smithi113.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T22:17:04.626 INFO:teuthology.orchestra.run.smithi113.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T22:17:04.627 INFO:teuthology.orchestra.run.smithi113.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T22:17:04.706 INFO:teuthology.orchestra.run.smithi113.stdout:Dependencies resolved. 2024-03-18T22:17:04.728 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2024-03-18T22:17:04.728 INFO:teuthology.orchestra.run.smithi113.stdout: Package Arch Version Repository Size 2024-03-18T22:17:04.728 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2024-03-18T22:17:04.728 INFO:teuthology.orchestra.run.smithi113.stdout:Installing: 2024-03-18T22:17:04.728 INFO:teuthology.orchestra.run.smithi113.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T22:17:04.729 INFO:teuthology.orchestra.run.smithi113.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-18T22:17:04.729 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-18T22:17:04.729 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-18T22:17:04.729 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-18T22:17:04.729 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-18T22:17:04.729 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-18T22:17:04.729 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-18T22:17:04.729 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-18T22:17:04.729 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-18T22:17:04.729 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-18T22:17:04.729 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-18T22:17:04.730 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-18T22:17:04.730 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-18T22:17:04.730 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-18T22:17:04.730 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-18T22:17:04.730 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-18T22:17:04.730 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T22:17:04.730 INFO:teuthology.orchestra.run.smithi113.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T22:17:04.730 INFO:teuthology.orchestra.run.smithi113.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T22:17:04.730 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-18T22:17:04.730 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-18T22:17:04.730 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-18T22:17:04.731 INFO:teuthology.orchestra.run.smithi113.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T22:17:04.731 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T22:17:04.731 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T22:17:04.731 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T22:17:04.731 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T22:17:04.731 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T22:17:04.731 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-18T22:17:04.731 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-18T22:17:04.731 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-18T22:17:04.731 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-18T22:17:04.731 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-18T22:17:04.731 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-18T22:17:04.731 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-18T22:17:04.732 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T22:17:04.732 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrading: 2024-03-18T22:17:04.732 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-18T22:17:04.732 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-18T22:17:04.732 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-18T22:17:04.732 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-18T22:17:04.732 INFO:teuthology.orchestra.run.smithi113.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T22:17:04.732 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T22:17:04.732 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T22:17:04.732 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T22:17:04.732 INFO:teuthology.orchestra.run.smithi113.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T22:17:04.732 INFO:teuthology.orchestra.run.smithi113.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T22:17:04.732 INFO:teuthology.orchestra.run.smithi113.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T22:17:04.733 INFO:teuthology.orchestra.run.smithi113.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T22:17:04.733 INFO:teuthology.orchestra.run.smithi113.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T22:17:04.733 INFO:teuthology.orchestra.run.smithi113.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-18T22:17:04.733 INFO:teuthology.orchestra.run.smithi113.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-18T22:17:04.733 INFO:teuthology.orchestra.run.smithi113.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T22:17:04.733 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T22:17:04.733 INFO:teuthology.orchestra.run.smithi113.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T22:17:04.733 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T22:17:04.733 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T22:17:04.733 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T22:17:04.733 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T22:17:04.733 INFO:teuthology.orchestra.run.smithi113.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T22:17:04.733 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T22:17:04.733 INFO:teuthology.orchestra.run.smithi113.stdout:Installing dependencies: 2024-03-18T22:17:04.734 INFO:teuthology.orchestra.run.smithi113.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T22:17:04.734 INFO:teuthology.orchestra.run.smithi113.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T22:17:04.734 INFO:teuthology.orchestra.run.smithi113.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T22:17:04.734 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-18T22:17:04.734 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-18T22:17:04.734 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-18T22:17:04.734 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-18T22:17:04.734 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-18T22:17:04.734 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-18T22:17:04.734 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-18T22:17:04.734 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-18T22:17:04.734 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T22:17:04.734 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T22:17:04.735 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T22:17:04.735 INFO:teuthology.orchestra.run.smithi113.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T22:17:04.735 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T22:17:04.735 INFO:teuthology.orchestra.run.smithi113.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T22:17:04.735 INFO:teuthology.orchestra.run.smithi113.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T22:17:04.735 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T22:17:04.735 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T22:17:04.735 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-18T22:17:04.735 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T22:17:04.735 INFO:teuthology.orchestra.run.smithi113.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T22:17:04.735 INFO:teuthology.orchestra.run.smithi113.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T22:17:04.736 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T22:17:04.736 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T22:17:04.736 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-18T22:17:04.736 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T22:17:04.736 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-18T22:17:04.736 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T22:17:04.736 INFO:teuthology.orchestra.run.smithi113.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T22:17:04.736 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T22:17:04.736 INFO:teuthology.orchestra.run.smithi113.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T22:17:04.736 INFO:teuthology.orchestra.run.smithi113.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T22:17:04.736 INFO:teuthology.orchestra.run.smithi113.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T22:17:04.736 INFO:teuthology.orchestra.run.smithi113.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T22:17:04.736 INFO:teuthology.orchestra.run.smithi113.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T22:17:04.736 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T22:17:04.737 INFO:teuthology.orchestra.run.smithi113.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T22:17:04.737 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T22:17:04.737 INFO:teuthology.orchestra.run.smithi113.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T22:17:04.737 INFO:teuthology.orchestra.run.smithi113.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T22:17:04.737 INFO:teuthology.orchestra.run.smithi113.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T22:17:04.737 INFO:teuthology.orchestra.run.smithi113.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T22:17:04.737 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T22:17:04.737 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T22:17:04.737 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T22:17:04.737 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-18T22:17:04.737 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-18T22:17:04.737 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T22:17:04.737 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T22:17:04.738 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T22:17:04.738 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T22:17:04.738 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T22:17:04.738 INFO:teuthology.orchestra.run.smithi113.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T22:17:04.738 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T22:17:04.738 INFO:teuthology.orchestra.run.smithi113.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T22:17:04.738 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T22:17:04.738 INFO:teuthology.orchestra.run.smithi113.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T22:17:04.738 INFO:teuthology.orchestra.run.smithi113.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T22:17:04.738 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T22:17:04.738 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T22:17:04.738 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T22:17:04.738 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T22:17:04.739 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T22:17:04.739 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T22:17:04.739 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T22:17:04.739 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T22:17:04.739 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T22:17:04.739 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T22:17:04.739 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T22:17:04.739 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T22:17:04.739 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T22:17:04.739 INFO:teuthology.orchestra.run.smithi113.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T22:17:04.739 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T22:17:04.739 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T22:17:04.739 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T22:17:04.739 INFO:teuthology.orchestra.run.smithi113.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T22:17:04.739 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T22:17:04.740 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T22:17:04.740 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T22:17:04.740 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T22:17:04.740 INFO:teuthology.orchestra.run.smithi113.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T22:17:04.740 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T22:17:04.740 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T22:17:04.740 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T22:17:04.740 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T22:17:04.740 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T22:17:04.740 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T22:17:04.740 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T22:17:04.740 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T22:17:04.740 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T22:17:04.740 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T22:17:04.740 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T22:17:04.741 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T22:17:04.741 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T22:17:04.741 INFO:teuthology.orchestra.run.smithi113.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T22:17:04.741 INFO:teuthology.orchestra.run.smithi113.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T22:17:04.741 INFO:teuthology.orchestra.run.smithi113.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T22:17:04.741 INFO:teuthology.orchestra.run.smithi113.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T22:17:04.741 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T22:17:04.741 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T22:17:04.741 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T22:17:04.741 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T22:17:04.741 INFO:teuthology.orchestra.run.smithi113.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T22:17:04.741 INFO:teuthology.orchestra.run.smithi113.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T22:17:04.741 INFO:teuthology.orchestra.run.smithi113.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T22:17:04.741 INFO:teuthology.orchestra.run.smithi113.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T22:17:04.742 INFO:teuthology.orchestra.run.smithi113.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T22:17:04.742 INFO:teuthology.orchestra.run.smithi113.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T22:17:04.742 INFO:teuthology.orchestra.run.smithi113.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T22:17:04.742 INFO:teuthology.orchestra.run.smithi113.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T22:17:04.742 INFO:teuthology.orchestra.run.smithi113.stdout:Installing weak dependencies: 2024-03-18T22:17:04.742 INFO:teuthology.orchestra.run.smithi113.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T22:17:04.742 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:17:04.742 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction Summary 2024-03-18T22:17:04.742 INFO:teuthology.orchestra.run.smithi113.stdout:============================================================================================ 2024-03-18T22:17:04.742 INFO:teuthology.orchestra.run.smithi113.stdout:Install 150 Packages 2024-03-18T22:17:04.742 INFO:teuthology.orchestra.run.smithi113.stdout:Upgrade 24 Packages 2024-03-18T22:17:04.742 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:17:04.743 INFO:teuthology.orchestra.run.smithi113.stdout:Total download size: 224 M 2024-03-18T22:17:04.743 INFO:teuthology.orchestra.run.smithi113.stdout:Downloading Packages: 2024-03-18T22:17:05.515 INFO:teuthology.orchestra.run.smithi113.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-03-18T22:17:05.807 INFO:teuthology.orchestra.run.smithi113.stdout:(2/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 10 MB/s | 5.4 MB 00:00 2024-03-18T22:17:05.857 INFO:teuthology.orchestra.run.smithi113.stdout:(3/174): ceph-immutable-object-cache-19.0.0-220 2.8 MB/s | 145 kB 00:00 2024-03-18T22:17:06.016 INFO:teuthology.orchestra.run.smithi113.stdout:(4/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 15 MB/s | 2.4 MB 00:00 2024-03-18T22:17:06.133 INFO:teuthology.orchestra.run.smithi113.stdout:(5/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 13 MB/s | 1.5 MB 00:00 2024-03-18T22:17:06.158 INFO:teuthology.orchestra.run.smithi113.stdout:(6/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 1.4 MB/s | 891 kB 00:00 2024-03-18T22:17:06.503 INFO:teuthology.orchestra.run.smithi113.stdout:(7/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 13 MB/s | 4.7 MB 00:00 2024-03-18T22:17:07.117 INFO:teuthology.orchestra.run.smithi113.stdout:(8/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el9 16 MB/s | 9.8 MB 00:00 2024-03-18T22:17:07.176 INFO:teuthology.orchestra.run.smithi113.stdout:(9/174): ceph-selinux-19.0.0-2206.gb6ae2491.el9 433 kB/s | 25 kB 00:00 2024-03-18T22:17:08.118 INFO:teuthology.orchestra.run.smithi113.stdout:(10/174): ceph-common-19.0.0-2206.gb6ae2491.el9 7.3 MB/s | 21 MB 00:02 2024-03-18T22:17:08.184 INFO:teuthology.orchestra.run.smithi113.stdout:(11/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 3.2 MB/s | 221 kB 00:00 2024-03-18T22:17:08.268 INFO:teuthology.orchestra.run.smithi113.stdout:(12/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 325 kB/s | 27 kB 00:00 2024-03-18T22:17:08.368 INFO:teuthology.orchestra.run.smithi113.stdout:(13/174): libcephfs-devel-19.0.0-2206.gb6ae2491 317 kB/s | 32 kB 00:00 2024-03-18T22:17:08.535 INFO:teuthology.orchestra.run.smithi113.stdout:(14/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 4.4 MB/s | 746 kB 00:00 2024-03-18T22:17:08.627 INFO:teuthology.orchestra.run.smithi113.stdout:(15/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 1.7 MB/s | 163 kB 00:00 2024-03-18T22:17:08.727 INFO:teuthology.orchestra.run.smithi113.stdout:(16/174): librados-devel-19.0.0-2206.gb6ae2491. 1.2 MB/s | 127 kB 00:00 2024-03-18T22:17:08.845 INFO:teuthology.orchestra.run.smithi113.stdout:(17/174): libradosstriper1-19.0.0-2206.gb6ae249 4.2 MB/s | 501 kB 00:00 2024-03-18T22:17:09.420 INFO:teuthology.orchestra.run.smithi113.stdout:(18/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 9.0 MB/s | 5.1 MB 00:00 2024-03-18T22:17:09.512 INFO:teuthology.orchestra.run.smithi113.stdout:(19/174): python3-ceph-argparse-19.0.0-2206.gb6 492 kB/s | 45 kB 00:00 2024-03-18T22:17:09.612 INFO:teuthology.orchestra.run.smithi113.stdout:(20/174): python3-ceph-common-19.0.0-2206.gb6ae 1.3 MB/s | 131 kB 00:00 2024-03-18T22:17:10.088 INFO:teuthology.orchestra.run.smithi113.stdout:(21/174): ceph-test-19.0.0-2206.gb6ae2491.el9.x 16 MB/s | 48 MB 00:02 2024-03-18T22:17:10.113 INFO:teuthology.orchestra.run.smithi113.stdout:(22/174): python3-cephfs-19.0.0-2206.gb6ae2491. 323 kB/s | 162 kB 00:00 2024-03-18T22:17:10.155 INFO:teuthology.orchestra.run.smithi113.stdout:(23/174): python3-rados-19.0.0-2206.gb6ae2491.e 4.7 MB/s | 321 kB 00:00 2024-03-18T22:17:10.188 INFO:teuthology.orchestra.run.smithi113.stdout:(24/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 3.9 MB/s | 298 kB 00:00 2024-03-18T22:17:10.214 INFO:teuthology.orchestra.run.smithi113.stdout:(25/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 1.7 MB/s | 99 kB 00:00 2024-03-18T22:17:10.264 INFO:teuthology.orchestra.run.smithi113.stdout:(26/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-18T22:17:10.373 INFO:teuthology.orchestra.run.smithi113.stdout:(27/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 1.5 MB/s | 170 kB 00:00 2024-03-18T22:17:10.423 INFO:teuthology.orchestra.run.smithi113.stdout:(28/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 15 MB/s | 3.1 MB 00:00 2024-03-18T22:17:10.440 INFO:teuthology.orchestra.run.smithi113.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2206.g 508 kB/s | 34 kB 00:00 2024-03-18T22:17:10.465 INFO:teuthology.orchestra.run.smithi113.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 3.2 MB/s | 139 kB 00:00 2024-03-18T22:17:10.674 INFO:teuthology.orchestra.run.smithi113.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 7.7 MB/s | 1.8 MB 00:00 2024-03-18T22:17:10.774 INFO:teuthology.orchestra.run.smithi113.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2206.gb6 2.4 MB/s | 248 kB 00:00 2024-03-18T22:17:10.899 INFO:teuthology.orchestra.run.smithi113.stdout:(33/174): ceph-mgr-diskprediction-local-19.0.0- 17 MB/s | 7.4 MB 00:00 2024-03-18T22:17:10.925 INFO:teuthology.orchestra.run.smithi113.stdout:(34/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 328 kB/s | 49 kB 00:00 2024-03-18T22:17:10.950 INFO:teuthology.orchestra.run.smithi113.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2206.gb 329 kB/s | 16 kB 00:00 2024-03-18T22:17:10.992 INFO:teuthology.orchestra.run.smithi113.stdout:(36/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 3.9 MB/s | 264 kB 00:00 2024-03-18T22:17:11.034 INFO:teuthology.orchestra.run.smithi113.stdout:(37/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 8.7 MB/s | 749 kB 00:00 2024-03-18T22:17:11.059 INFO:teuthology.orchestra.run.smithi113.stdout:(38/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 273 kB/s | 18 kB 00:00 2024-03-18T22:17:11.343 INFO:teuthology.orchestra.run.smithi113.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 662 kB 00:00 2024-03-18T22:17:11.418 INFO:teuthology.orchestra.run.smithi113.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 2.7 MB/s | 920 kB 00:00 2024-03-18T22:17:11.443 INFO:teuthology.orchestra.run.smithi113.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.0 MB/s | 106 kB 00:00 2024-03-18T22:17:11.485 INFO:teuthology.orchestra.run.smithi113.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 4.5 MB/s | 307 kB 00:00 2024-03-18T22:17:11.519 INFO:teuthology.orchestra.run.smithi113.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 417 kB/s | 32 kB 00:00 2024-03-18T22:17:11.637 INFO:teuthology.orchestra.run.smithi113.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 147 kB/s | 17 kB 00:00 2024-03-18T22:17:11.686 INFO:teuthology.orchestra.run.smithi113.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 15 MB/s | 3.0 MB 00:00 2024-03-18T22:17:11.712 INFO:teuthology.orchestra.run.smithi113.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 136 kB/s | 10 kB 00:00 2024-03-18T22:17:11.737 INFO:teuthology.orchestra.run.smithi113.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-18T22:17:11.763 INFO:teuthology.orchestra.run.smithi113.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-18T22:17:11.805 INFO:teuthology.orchestra.run.smithi113.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-18T22:17:11.830 INFO:teuthology.orchestra.run.smithi113.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.6 MB/s | 578 kB 00:00 2024-03-18T22:17:11.855 INFO:teuthology.orchestra.run.smithi113.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 939 kB/s | 46 kB 00:00 2024-03-18T22:17:11.889 INFO:teuthology.orchestra.run.smithi113.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-18T22:17:11.914 INFO:teuthology.orchestra.run.smithi113.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-18T22:17:11.939 INFO:teuthology.orchestra.run.smithi113.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-18T22:17:11.964 INFO:teuthology.orchestra.run.smithi113.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 679 kB/s | 34 kB 00:00 2024-03-18T22:17:12.190 INFO:teuthology.orchestra.run.smithi113.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 19 MB/s | 4.7 MB 00:00 2024-03-18T22:17:12.223 INFO:teuthology.orchestra.run.smithi113.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 5.9 MB/s | 1.5 MB 00:00 2024-03-18T22:17:12.283 INFO:teuthology.orchestra.run.smithi113.stdout:(58/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-18T22:17:12.341 INFO:teuthology.orchestra.run.smithi113.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 6.8 MB/s | 811 kB 00:00 2024-03-18T22:17:12.541 INFO:teuthology.orchestra.run.smithi113.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 22 MB/s | 5.7 MB 00:00 2024-03-18T22:17:12.575 INFO:teuthology.orchestra.run.smithi113.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.4 MB/s | 1.0 MB 00:00 2024-03-18T22:17:12.600 INFO:teuthology.orchestra.run.smithi113.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-18T22:17:12.700 INFO:teuthology.orchestra.run.smithi113.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 244 kB/s | 24 kB 00:00 2024-03-18T22:17:12.784 INFO:teuthology.orchestra.run.smithi113.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 28 MB/s | 5.8 MB 00:00 2024-03-18T22:17:12.801 INFO:teuthology.orchestra.run.smithi113.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 182 kB/s | 18 kB 00:00 2024-03-18T22:17:12.826 INFO:teuthology.orchestra.run.smithi113.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-18T22:17:12.843 INFO:teuthology.orchestra.run.smithi113.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 756 kB/s | 32 kB 00:00 2024-03-18T22:17:12.943 INFO:teuthology.orchestra.run.smithi113.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 217 kB 00:00 2024-03-18T22:17:13.010 INFO:teuthology.orchestra.run.smithi113.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2024-03-18T22:17:13.036 INFO:teuthology.orchestra.run.smithi113.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 174 kB 00:00 2024-03-18T22:17:13.053 INFO:teuthology.orchestra.run.smithi113.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.6 MB/s | 69 kB 00:00 2024-03-18T22:17:13.070 INFO:teuthology.orchestra.run.smithi113.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-18T22:17:13.104 INFO:teuthology.orchestra.run.smithi113.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-18T22:17:13.129 INFO:teuthology.orchestra.run.smithi113.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-18T22:17:13.146 INFO:teuthology.orchestra.run.smithi113.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.9 MB/s | 211 kB 00:00 2024-03-18T22:17:13.171 INFO:teuthology.orchestra.run.smithi113.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-18T22:17:13.196 INFO:teuthology.orchestra.run.smithi113.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 856 kB/s | 43 kB 00:00 2024-03-18T22:17:13.413 INFO:teuthology.orchestra.run.smithi113.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 179 kB/s | 39 kB 00:00 2024-03-18T22:17:13.680 INFO:teuthology.orchestra.run.smithi113.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-03-18T22:17:13.706 INFO:teuthology.orchestra.run.smithi113.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 141 kB/s | 41 kB 00:00 2024-03-18T22:17:13.739 INFO:teuthology.orchestra.run.smithi113.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-18T22:17:13.764 INFO:teuthology.orchestra.run.smithi113.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 974 kB/s | 56 kB 00:00 2024-03-18T22:17:13.790 INFO:teuthology.orchestra.run.smithi113.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-18T22:17:13.815 INFO:teuthology.orchestra.run.smithi113.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 877 kB/s | 44 kB 00:00 2024-03-18T22:17:13.839 INFO:teuthology.orchestra.run.smithi113.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-18T22:17:13.873 INFO:teuthology.orchestra.run.smithi113.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.1 MB/s | 241 kB 00:00 2024-03-18T22:17:14.040 INFO:teuthology.orchestra.run.smithi113.stdout:(87/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x8 2.1 MB/s | 17 MB 00:07 2024-03-18T22:17:14.074 INFO:teuthology.orchestra.run.smithi113.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 4.9 MB/s | 1.1 MB 00:00 2024-03-18T22:17:14.099 INFO:teuthology.orchestra.run.smithi113.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 459 kB/s | 103 kB 00:00 2024-03-18T22:17:14.125 INFO:teuthology.orchestra.run.smithi113.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-18T22:17:14.150 INFO:teuthology.orchestra.run.smithi113.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-18T22:17:14.275 INFO:teuthology.orchestra.run.smithi113.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-18T22:17:14.300 INFO:teuthology.orchestra.run.smithi113.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-18T22:17:14.326 INFO:teuthology.orchestra.run.smithi113.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-18T22:17:14.352 INFO:teuthology.orchestra.run.smithi113.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-03-18T22:17:14.410 INFO:teuthology.orchestra.run.smithi113.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 197 kB/s | 21 kB 00:00 2024-03-18T22:17:14.435 INFO:teuthology.orchestra.run.smithi113.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 234 kB/s | 25 kB 00:00 2024-03-18T22:17:14.585 INFO:teuthology.orchestra.run.smithi113.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-18T22:17:14.610 INFO:teuthology.orchestra.run.smithi113.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.4 MB/s | 248 kB 00:00 2024-03-18T22:17:14.661 INFO:teuthology.orchestra.run.smithi113.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 699 kB/s | 35 kB 00:00 2024-03-18T22:17:14.686 INFO:teuthology.orchestra.run.smithi113.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.1 MB/s | 216 kB 00:00 2024-03-18T22:17:14.711 INFO:teuthology.orchestra.run.smithi113.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-18T22:17:14.737 INFO:teuthology.orchestra.run.smithi113.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-18T22:17:14.754 INFO:teuthology.orchestra.run.smithi113.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 532 kB/s | 22 kB 00:00 2024-03-18T22:17:14.854 INFO:teuthology.orchestra.run.smithi113.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 7.2 MB/s | 863 kB 00:00 2024-03-18T22:17:15.096 INFO:teuthology.orchestra.run.smithi113.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-03-18T22:17:15.154 INFO:teuthology.orchestra.run.smithi113.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-03-18T22:17:15.171 INFO:teuthology.orchestra.run.smithi113.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-03-18T22:17:15.230 INFO:teuthology.orchestra.run.smithi113.stdout:(109/174): openssl-devel-3.0.7-27.el9.x86_64.rp 4.7 MB/s | 4.1 MB 00:00 2024-03-18T22:17:15.247 INFO:teuthology.orchestra.run.smithi113.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-03-18T22:17:15.314 INFO:teuthology.orchestra.run.smithi113.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-18T22:17:15.381 INFO:teuthology.orchestra.run.smithi113.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-03-18T22:17:15.473 INFO:teuthology.orchestra.run.smithi113.stdout:(113/174): liboath-2.6.7-2.el9.x86_64.rpm 200 kB/s | 49 kB 00:00 2024-03-18T22:17:15.540 INFO:teuthology.orchestra.run.smithi113.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-18T22:17:15.565 INFO:teuthology.orchestra.run.smithi113.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-03-18T22:17:15.599 INFO:teuthology.orchestra.run.smithi113.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 743 kB/s | 43 kB 00:00 2024-03-18T22:17:15.624 INFO:teuthology.orchestra.run.smithi113.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 547 kB/s | 32 kB 00:00 2024-03-18T22:17:15.658 INFO:teuthology.orchestra.run.smithi113.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 256 kB/s | 15 kB 00:00 2024-03-18T22:17:15.683 INFO:teuthology.orchestra.run.smithi113.stdout:(119/174): python3-asyncssh-2.13.2-4.el9.noarch 2.6 MB/s | 548 kB 00:00 2024-03-18T22:17:15.708 INFO:teuthology.orchestra.run.smithi113.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-03-18T22:17:15.733 INFO:teuthology.orchestra.run.smithi113.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-03-18T22:17:15.767 INFO:teuthology.orchestra.run.smithi113.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-03-18T22:17:15.809 INFO:teuthology.orchestra.run.smithi113.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-03-18T22:17:15.834 INFO:teuthology.orchestra.run.smithi113.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-18T22:17:15.876 INFO:teuthology.orchestra.run.smithi113.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-18T22:17:15.901 INFO:teuthology.orchestra.run.smithi113.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-18T22:17:15.943 INFO:teuthology.orchestra.run.smithi113.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 265 kB/s | 18 kB 00:00 2024-03-18T22:17:15.969 INFO:teuthology.orchestra.run.smithi113.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 346 kB/s | 23 kB 00:00 2024-03-18T22:17:16.019 INFO:teuthology.orchestra.run.smithi113.stdout:(129/174): python3-grpcio-1.46.7-10.el9.x86_64. 8.1 MB/s | 2.0 MB 00:00 2024-03-18T22:17:16.044 INFO:teuthology.orchestra.run.smithi113.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 193 kB/s | 19 kB 00:00 2024-03-18T22:17:16.069 INFO:teuthology.orchestra.run.smithi113.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-03-18T22:17:16.128 INFO:teuthology.orchestra.run.smithi113.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 797 kB/s | 46 kB 00:00 2024-03-18T22:17:16.153 INFO:teuthology.orchestra.run.smithi113.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 945 kB/s | 79 kB 00:00 2024-03-18T22:17:16.187 INFO:teuthology.orchestra.run.smithi113.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-03-18T22:17:16.212 INFO:teuthology.orchestra.run.smithi113.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 688 kB/s | 58 kB 00:00 2024-03-18T22:17:16.237 INFO:teuthology.orchestra.run.smithi113.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-18T22:17:16.282 INFO:teuthology.orchestra.run.smithi113.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 173 kB/s | 16 kB 00:00 2024-03-18T22:17:16.313 INFO:teuthology.orchestra.run.smithi113.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 376 kB/s | 28 kB 00:00 2024-03-18T22:17:16.338 INFO:teuthology.orchestra.run.smithi113.stdout:(139/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 714 kB/s | 90 kB 00:00 2024-03-18T22:17:16.355 INFO:teuthology.orchestra.run.smithi113.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 423 kB/s | 31 kB 00:00 2024-03-18T22:17:16.378 INFO:teuthology.orchestra.run.smithi113.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.9 MB/s | 188 kB 00:00 2024-03-18T22:17:16.403 INFO:teuthology.orchestra.run.smithi113.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 922 kB/s | 59 kB 00:00 2024-03-18T22:17:16.428 INFO:teuthology.orchestra.run.smithi113.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 492 kB/s | 36 kB 00:00 2024-03-18T22:17:16.454 INFO:teuthology.orchestra.run.smithi113.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 675 kB/s | 51 kB 00:00 2024-03-18T22:17:16.479 INFO:teuthology.orchestra.run.smithi113.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-18T22:17:16.504 INFO:teuthology.orchestra.run.smithi113.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-18T22:17:16.529 INFO:teuthology.orchestra.run.smithi113.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-03-18T22:17:16.546 INFO:teuthology.orchestra.run.smithi113.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 297 kB/s | 20 kB 00:00 2024-03-18T22:17:16.571 INFO:teuthology.orchestra.run.smithi113.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-18T22:17:16.672 INFO:teuthology.orchestra.run.smithi113.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-18T22:17:16.714 INFO:teuthology.orchestra.run.smithi113.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.8 MB/s | 321 kB 00:00 2024-03-18T22:17:16.732 INFO:teuthology.orchestra.run.smithi113.stdout:[MIRROR] NetworkManager-1.47.2-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.47.2-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-18T22:17:17.122 INFO:teuthology.orchestra.run.smithi113.stdout:(152/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 5.7 MB/s | 3.1 MB 00:00 2024-03-18T22:17:17.431 INFO:teuthology.orchestra.run.smithi113.stdout:(153/174): librados2-19.0.0-2206.gb6ae2491.el9. 3.8 MB/s | 3.4 MB 00:00 2024-03-18T22:17:17.714 INFO:teuthology.orchestra.run.smithi113.stdout:(154/174): NetworkManager-1.47.2-1.el9.x86_64.r 2.3 MB/s | 2.3 MB 00:01 2024-03-18T22:17:17.773 INFO:teuthology.orchestra.run.smithi113.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 123 kB/s | 42 kB 00:00 2024-03-18T22:17:17.823 INFO:teuthology.orchestra.run.smithi113.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.2 MB/s | 248 kB 00:00 2024-03-18T22:17:17.948 INFO:teuthology.orchestra.run.smithi113.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 291 kB/s | 36 kB 00:00 2024-03-18T22:17:17.966 INFO:teuthology.orchestra.run.smithi113.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 371 kB/s | 71 kB 00:00 2024-03-18T22:17:18.066 INFO:teuthology.orchestra.run.smithi113.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.6 MB/s | 192 kB 00:00 2024-03-18T22:17:18.108 INFO:teuthology.orchestra.run.smithi113.stdout:(160/174): NetworkManager-libnm-1.47.2-1.el9.x8 1.9 MB/s | 1.8 MB 00:00 2024-03-18T22:17:18.175 INFO:teuthology.orchestra.run.smithi113.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 212 kB/s | 23 kB 00:00 2024-03-18T22:17:18.217 INFO:teuthology.orchestra.run.smithi113.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 997 kB/s | 108 kB 00:00 2024-03-18T22:17:18.267 INFO:teuthology.orchestra.run.smithi113.stdout:(163/174): elfutils-libs-0.190-2.el9.x86_64.rpm 850 kB/s | 256 kB 00:00 2024-03-18T22:17:18.292 INFO:teuthology.orchestra.run.smithi113.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 155 kB 00:00 2024-03-18T22:17:18.334 INFO:teuthology.orchestra.run.smithi113.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-18T22:17:18.376 INFO:teuthology.orchestra.run.smithi113.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 585 kB/s | 64 kB 00:00 2024-03-18T22:17:18.402 INFO:teuthology.orchestra.run.smithi113.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 263 kB/s | 29 kB 00:00 2024-03-18T22:17:18.593 INFO:teuthology.orchestra.run.smithi113.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 4.7 MB/s | 1.2 MB 00:00 2024-03-18T22:17:18.744 INFO:teuthology.orchestra.run.smithi113.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 6.7 MB/s | 2.3 MB 00:00 2024-03-18T22:17:18.769 INFO:teuthology.orchestra.run.smithi113.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 2.6 MB/s | 465 kB 00:00 2024-03-18T22:17:18.869 INFO:teuthology.orchestra.run.smithi113.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 248 kB/s | 31 kB 00:00 2024-03-18T22:17:18.911 INFO:teuthology.orchestra.run.smithi113.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.6 MB/s | 1.1 MB 00:00 2024-03-18T22:17:18.995 INFO:teuthology.orchestra.run.smithi113.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 596 kB/s | 74 kB 00:00 2024-03-18T22:17:19.062 INFO:teuthology.orchestra.run.smithi113.stdout:(174/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 3.1 MB/s | 2.2 MB 00:00 2024-03-18T22:17:19.068 INFO:teuthology.orchestra.run.smithi113.stdout:-------------------------------------------------------------------------------- 2024-03-18T22:17:19.068 INFO:teuthology.orchestra.run.smithi113.stdout:Total 16 MB/s | 224 MB 00:14 2024-03-18T22:17:19.533 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction check 2024-03-18T22:17:19.807 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction check succeeded. 2024-03-18T22:17:19.807 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction test 2024-03-18T22:17:21.819 INFO:teuthology.orchestra.run.smithi113.stdout:Transaction test succeeded. 2024-03-18T22:17:21.820 INFO:teuthology.orchestra.run.smithi113.stdout:Running transaction 2024-03-18T22:17:28.773 INFO:teuthology.orchestra.run.smithi113.stdout: Preparing : 1/1 2024-03-18T22:17:28.852 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T22:17:28.892 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T22:17:28.963 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T22:17:29.008 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T22:17:29.242 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T22:17:29.304 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T22:17:29.343 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T22:17:29.376 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T22:17:29.426 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T22:17:29.807 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T22:17:29.843 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-18T22:17:29.963 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-18T22:17:29.995 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-18T22:17:30.093 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-18T22:17:30.125 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-18T22:17:30.175 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T22:17:30.352 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T22:17:30.405 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T22:17:30.433 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-18T22:17:30.535 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-18T22:17:30.559 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-18T22:17:30.624 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-18T22:17:30.672 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T22:17:30.694 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T22:17:30.710 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T22:17:30.909 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T22:17:30.943 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T22:17:30.990 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T22:17:31.167 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T22:17:31.284 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-18T22:17:31.345 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-18T22:17:31.428 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-18T22:17:31.470 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-18T22:17:31.690 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-18T22:17:31.791 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T22:17:31.888 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T22:17:31.934 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T22:17:31.961 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T22:17:31.996 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T22:17:32.071 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T22:17:32.125 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T22:17:32.188 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T22:17:32.252 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T22:17:32.296 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T22:17:32.338 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T22:17:32.372 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T22:17:32.399 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T22:17:32.469 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T22:17:32.504 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-18T22:17:32.547 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-18T22:17:32.578 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-18T22:17:32.645 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T22:17:32.691 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T22:17:32.765 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T22:17:32.898 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T22:17:32.952 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T22:17:33.472 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T22:17:33.557 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T22:17:33.634 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T22:17:33.675 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T22:17:33.698 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T22:17:33.742 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T22:17:33.801 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T22:17:33.835 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-18T22:17:33.866 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T22:17:34.146 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T22:17:34.372 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T22:17:34.451 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T22:17:34.495 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T22:17:34.817 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T22:17:34.899 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T22:17:34.962 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T22:17:35.035 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T22:17:35.074 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T22:17:35.111 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T22:17:35.157 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T22:17:35.193 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T22:17:35.237 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T22:17:35.278 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T22:17:35.412 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T22:17:35.459 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T22:17:35.490 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T22:17:35.540 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T22:17:35.633 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T22:17:35.673 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T22:17:35.713 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T22:17:35.739 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T22:17:35.763 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T22:17:35.848 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T22:17:35.977 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T22:17:36.083 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T22:17:36.159 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T22:17:36.208 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T22:17:36.263 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T22:17:36.458 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T22:17:36.505 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T22:17:36.642 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T22:17:36.678 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T22:17:36.718 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T22:17:36.749 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T22:17:36.777 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T22:17:36.811 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T22:17:36.850 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T22:17:36.882 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T22:17:36.912 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T22:17:36.988 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T22:17:37.022 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T22:17:37.056 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T22:17:37.089 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T22:17:37.132 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T22:17:37.221 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T22:17:37.303 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T22:17:37.346 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T22:17:37.378 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T22:17:37.425 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T22:17:37.522 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T22:17:37.580 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T22:17:38.304 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T22:17:38.360 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T22:17:38.394 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T22:17:38.442 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T22:17:38.532 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T22:17:38.578 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T22:17:38.764 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T22:17:38.793 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T22:17:38.870 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T22:17:39.522 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T22:17:39.632 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T22:17:40.853 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T22:17:40.927 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T22:17:40.958 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T22:17:40.989 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T22:17:41.017 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T22:17:41.332 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T22:17:41.541 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-18T22:17:41.788 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-18T22:17:41.818 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-18T22:17:41.855 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T22:17:42.316 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T22:17:42.341 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-18T22:17:42.395 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-18T22:17:42.418 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-18T22:17:44.629 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-18T22:17:44.665 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-18T22:17:45.209 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-18T22:17:45.251 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-18T22:17:45.342 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-18T22:17:45.342 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T22:17:45.342 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T22:17:45.342 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:17:45.386 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-18T22:17:57.327 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-18T22:17:57.328 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /sys 2024-03-18T22:17:57.328 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /proc 2024-03-18T22:17:57.328 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /mnt 2024-03-18T22:17:57.328 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /var/tmp 2024-03-18T22:17:57.328 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /home 2024-03-18T22:17:57.328 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /root 2024-03-18T22:17:57.328 INFO:teuthology.orchestra.run.smithi113.stdout:skipping the directory /tmp 2024-03-18T22:17:57.328 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:17:57.501 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-18T22:17:57.516 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-18T22:17:57.516 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:17:57.516 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T22:17:57.516 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:17:58.989 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-18T22:17:59.014 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-18T22:17:59.014 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:17:59.014 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T22:17:59.015 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T22:17:59.015 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T22:17:59.015 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:17:59.239 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-18T22:17:59.261 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-18T22:17:59.262 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:17:59.262 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T22:17:59.262 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T22:17:59.262 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T22:17:59.262 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:17:59.698 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-18T22:17:59.721 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-18T22:17:59.721 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:17:59.721 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T22:17:59.721 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T22:17:59.721 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T22:17:59.721 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:17:59.808 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T22:18:00.200 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-18T22:18:00.232 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-18T22:18:00.272 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-18T22:18:00.332 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-18T22:18:00.355 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-18T22:18:00.586 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-18T22:18:00.623 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-18T22:18:01.580 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-18T22:18:01.606 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-18T22:18:01.707 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-18T22:18:01.868 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-18T22:18:01.908 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-18T22:18:01.931 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-18T22:18:01.931 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:18:01.931 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T22:18:01.931 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T22:18:01.931 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T22:18:01.931 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:18:01.976 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-18T22:18:01.986 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-18T22:18:02.033 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-18T22:18:02.903 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T22:18:02.928 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-18T22:18:02.950 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-18T22:18:02.950 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:18:02.950 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T22:18:02.950 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T22:18:02.951 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T22:18:02.951 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:18:03.003 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-18T22:18:03.025 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-18T22:18:03.025 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:18:03.025 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T22:18:03.025 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:18:03.071 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-18T22:18:03.093 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-18T22:18:03.093 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:18:03.093 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T22:18:03.093 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T22:18:03.093 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T22:18:03.093 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:18:03.388 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-18T22:18:03.410 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-18T22:18:03.411 INFO:teuthology.orchestra.run.smithi113.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:18:03.411 INFO:teuthology.orchestra.run.smithi113.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T22:18:03.411 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T22:18:03.411 INFO:teuthology.orchestra.run.smithi113.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T22:18:03.411 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:18:07.876 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-18T22:18:07.916 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-18T22:18:07.947 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-18T22:18:07.974 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T22:18:08.163 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-18T22:18:08.274 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T22:18:08.323 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-18T22:18:08.390 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T22:18:08.449 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T22:18:08.516 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T22:18:08.554 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-18T22:18:08.582 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T22:18:08.620 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-18T22:18:08.676 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-18T22:18:08.765 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T22:18:08.802 INFO:teuthology.orchestra.run.smithi113.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T22:18:09.411 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T22:18:09.450 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T22:18:09.469 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T22:18:10.285 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-18T22:18:10.370 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T22:18:10.411 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T22:18:10.509 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T22:18:10.549 INFO:teuthology.orchestra.run.smithi113.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T22:18:10.574 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T22:18:10.599 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T22:18:10.625 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T22:18:10.626 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T22:18:10.713 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T22:18:10.743 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T22:18:10.743 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T22:18:10.828 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T22:18:10.863 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T22:18:10.896 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T22:18:10.921 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T22:18:10.946 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T22:18:10.981 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T22:18:11.027 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T22:18:11.069 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T22:18:11.093 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T22:18:11.118 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T22:18:11.160 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T22:18:11.185 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T22:18:11.192 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T22:18:11.192 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T22:18:11.380 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T22:18:11.433 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T22:18:11.463 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T22:18:11.488 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T22:18:11.513 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T22:18:11.545 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T22:18:11.572 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T22:18:11.638 INFO:teuthology.orchestra.run.smithi113.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T22:18:16.598 INFO:teuthology.orchestra.run.smithi113.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T22:18:16.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-18T22:18:16.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-18T22:18:16.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-18T22:18:16.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-18T22:18:16.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-18T22:18:16.598 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-18T22:18:16.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-18T22:18:16.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-18T22:18:16.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-18T22:18:16.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-18T22:18:16.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-18T22:18:16.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-18T22:18:16.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-18T22:18:16.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-18T22:18:16.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-18T22:18:16.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-18T22:18:16.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-18T22:18:16.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-18T22:18:16.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-18T22:18:16.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-18T22:18:16.599 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-18T22:18:16.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-18T22:18:16.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-18T22:18:16.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-18T22:18:16.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-18T22:18:16.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-18T22:18:16.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-18T22:18:16.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-18T22:18:16.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-18T22:18:16.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-18T22:18:16.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-18T22:18:16.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-18T22:18:16.600 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-18T22:18:16.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-18T22:18:16.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-18T22:18:16.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-18T22:18:16.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-18T22:18:16.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-18T22:18:16.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-18T22:18:16.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T22:18:16.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T22:18:16.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T22:18:16.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T22:18:16.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T22:18:16.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T22:18:16.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T22:18:16.601 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T22:18:16.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T22:18:16.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T22:18:16.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T22:18:16.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T22:18:16.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T22:18:16.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T22:18:16.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T22:18:16.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T22:18:16.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T22:18:16.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T22:18:16.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T22:18:16.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T22:18:16.602 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T22:18:16.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T22:18:16.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T22:18:16.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T22:18:16.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T22:18:16.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T22:18:16.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T22:18:16.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T22:18:16.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T22:18:16.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T22:18:16.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T22:18:16.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T22:18:16.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T22:18:16.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T22:18:16.603 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T22:18:16.604 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T22:18:16.604 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T22:18:16.604 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T22:18:16.604 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T22:18:16.604 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T22:18:16.604 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T22:18:16.604 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T22:18:16.604 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T22:18:16.604 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T22:18:16.604 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T22:18:16.604 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T22:18:16.604 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T22:18:16.604 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T22:18:16.605 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T22:18:16.605 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T22:18:16.605 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T22:18:16.605 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T22:18:16.605 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T22:18:16.605 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T22:18:16.605 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T22:18:16.605 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T22:18:16.605 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T22:18:16.605 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T22:18:16.605 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T22:18:16.605 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T22:18:16.605 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T22:18:16.605 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T22:18:16.605 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T22:18:16.606 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T22:18:16.606 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T22:18:16.606 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T22:18:16.606 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T22:18:16.606 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T22:18:16.606 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T22:18:16.606 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T22:18:16.606 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T22:18:16.606 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T22:18:16.606 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T22:18:16.606 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T22:18:16.606 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T22:18:16.606 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T22:18:16.606 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T22:18:16.607 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T22:18:16.607 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T22:18:16.607 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T22:18:16.607 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T22:18:16.607 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T22:18:16.607 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T22:18:16.607 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T22:18:16.607 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T22:18:16.607 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T22:18:16.607 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T22:18:16.607 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T22:18:16.607 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T22:18:16.607 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T22:18:16.607 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T22:18:16.608 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T22:18:16.608 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T22:18:16.608 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T22:18:16.608 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T22:18:16.608 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T22:18:16.608 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T22:18:16.608 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T22:18:16.608 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T22:18:16.608 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T22:18:16.608 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T22:18:16.608 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T22:18:16.608 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T22:18:16.608 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T22:18:16.608 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T22:18:16.608 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T22:18:16.609 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T22:18:16.609 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T22:18:16.609 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T22:18:16.609 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T22:18:16.609 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T22:18:16.609 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-18T22:18:16.609 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T22:18:16.609 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-18T22:18:16.609 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T22:18:16.609 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T22:18:16.609 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T22:18:16.609 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-18T22:18:16.609 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T22:18:16.609 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-18T22:18:16.609 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T22:18:16.610 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-18T22:18:16.610 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T22:18:16.610 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-18T22:18:16.610 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T22:18:16.610 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T22:18:16.610 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T22:18:16.610 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T22:18:16.610 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T22:18:16.610 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T22:18:16.610 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T22:18:16.610 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T22:18:16.610 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T22:18:16.610 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T22:18:16.610 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T22:18:16.611 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T22:18:16.612 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T22:18:16.612 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T22:18:16.612 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T22:18:16.612 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T22:18:16.612 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T22:18:16.612 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T22:18:16.612 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T22:18:19.704 INFO:teuthology.orchestra.run.smithi113.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T22:18:19.704 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:18:19.704 INFO:teuthology.orchestra.run.smithi113.stdout:Upgraded: 2024-03-18T22:18:19.705 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-18T22:18:19.705 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-18T22:18:19.705 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-18T22:18:19.705 INFO:teuthology.orchestra.run.smithi113.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-18T22:18:19.705 INFO:teuthology.orchestra.run.smithi113.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T22:18:19.705 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T22:18:19.705 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T22:18:19.705 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T22:18:19.705 INFO:teuthology.orchestra.run.smithi113.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T22:18:19.705 INFO:teuthology.orchestra.run.smithi113.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T22:18:19.705 INFO:teuthology.orchestra.run.smithi113.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T22:18:19.705 INFO:teuthology.orchestra.run.smithi113.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T22:18:19.705 INFO:teuthology.orchestra.run.smithi113.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T22:18:19.706 INFO:teuthology.orchestra.run.smithi113.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.706 INFO:teuthology.orchestra.run.smithi113.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.706 INFO:teuthology.orchestra.run.smithi113.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T22:18:19.706 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T22:18:19.706 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T22:18:19.706 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T22:18:19.706 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T22:18:19.706 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T22:18:19.706 INFO:teuthology.orchestra.run.smithi113.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T22:18:19.706 INFO:teuthology.orchestra.run.smithi113.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T22:18:19.706 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T22:18:19.706 INFO:teuthology.orchestra.run.smithi113.stdout:Installed: 2024-03-18T22:18:19.706 INFO:teuthology.orchestra.run.smithi113.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T22:18:19.706 INFO:teuthology.orchestra.run.smithi113.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T22:18:19.707 INFO:teuthology.orchestra.run.smithi113.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T22:18:19.707 INFO:teuthology.orchestra.run.smithi113.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T22:18:19.707 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.707 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.707 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.707 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.707 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:18:19.707 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.707 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.707 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.707 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:18:19.707 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:18:19.707 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:18:19.708 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:18:19.708 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:18:19.708 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.708 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.708 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:18:19.708 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.708 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.708 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.708 INFO:teuthology.orchestra.run.smithi113.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:18:19.708 INFO:teuthology.orchestra.run.smithi113.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:18:19.708 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.708 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.708 INFO:teuthology.orchestra.run.smithi113.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:18:19.708 INFO:teuthology.orchestra.run.smithi113.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T22:18:19.709 INFO:teuthology.orchestra.run.smithi113.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T22:18:19.709 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T22:18:19.709 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T22:18:19.709 INFO:teuthology.orchestra.run.smithi113.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T22:18:19.709 INFO:teuthology.orchestra.run.smithi113.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T22:18:19.709 INFO:teuthology.orchestra.run.smithi113.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T22:18:19.709 INFO:teuthology.orchestra.run.smithi113.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T22:18:19.709 INFO:teuthology.orchestra.run.smithi113.stdout: jq-1.6-16.el9.x86_64 2024-03-18T22:18:19.709 INFO:teuthology.orchestra.run.smithi113.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T22:18:19.709 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T22:18:19.709 INFO:teuthology.orchestra.run.smithi113.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T22:18:19.709 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.709 INFO:teuthology.orchestra.run.smithi113.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.709 INFO:teuthology.orchestra.run.smithi113.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.710 INFO:teuthology.orchestra.run.smithi113.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T22:18:19.710 INFO:teuthology.orchestra.run.smithi113.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T22:18:19.710 INFO:teuthology.orchestra.run.smithi113.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T22:18:19.710 INFO:teuthology.orchestra.run.smithi113.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T22:18:19.710 INFO:teuthology.orchestra.run.smithi113.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T22:18:19.710 INFO:teuthology.orchestra.run.smithi113.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.710 INFO:teuthology.orchestra.run.smithi113.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.710 INFO:teuthology.orchestra.run.smithi113.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T22:18:19.710 INFO:teuthology.orchestra.run.smithi113.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.710 INFO:teuthology.orchestra.run.smithi113.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T22:18:19.710 INFO:teuthology.orchestra.run.smithi113.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T22:18:19.710 INFO:teuthology.orchestra.run.smithi113.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T22:18:19.710 INFO:teuthology.orchestra.run.smithi113.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T22:18:19.710 INFO:teuthology.orchestra.run.smithi113.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T22:18:19.711 INFO:teuthology.orchestra.run.smithi113.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T22:18:19.711 INFO:teuthology.orchestra.run.smithi113.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T22:18:19.711 INFO:teuthology.orchestra.run.smithi113.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T22:18:19.711 INFO:teuthology.orchestra.run.smithi113.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T22:18:19.711 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T22:18:19.711 INFO:teuthology.orchestra.run.smithi113.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T22:18:19.711 INFO:teuthology.orchestra.run.smithi113.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T22:18:19.711 INFO:teuthology.orchestra.run.smithi113.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T22:18:19.711 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T22:18:19.711 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T22:18:19.711 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T22:18:19.711 INFO:teuthology.orchestra.run.smithi113.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T22:18:19.711 INFO:teuthology.orchestra.run.smithi113.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T22:18:19.711 INFO:teuthology.orchestra.run.smithi113.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T22:18:19.712 INFO:teuthology.orchestra.run.smithi113.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T22:18:19.712 INFO:teuthology.orchestra.run.smithi113.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T22:18:19.712 INFO:teuthology.orchestra.run.smithi113.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T22:18:19.712 INFO:teuthology.orchestra.run.smithi113.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T22:18:19.712 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T22:18:19.712 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.712 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.712 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.712 INFO:teuthology.orchestra.run.smithi113.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T22:18:19.712 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T22:18:19.712 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T22:18:19.712 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T22:18:19.712 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T22:18:19.713 INFO:teuthology.orchestra.run.smithi113.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T22:18:19.713 INFO:teuthology.orchestra.run.smithi113.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T22:18:19.713 INFO:teuthology.orchestra.run.smithi113.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T22:18:19.713 INFO:teuthology.orchestra.run.smithi113.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T22:18:19.713 INFO:teuthology.orchestra.run.smithi113.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T22:18:19.713 INFO:teuthology.orchestra.run.smithi113.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T22:18:19.713 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T22:18:19.713 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T22:18:19.713 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T22:18:19.713 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T22:18:19.713 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T22:18:19.713 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T22:18:19.713 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T22:18:19.714 INFO:teuthology.orchestra.run.smithi113.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T22:18:19.714 INFO:teuthology.orchestra.run.smithi113.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T22:18:19.714 INFO:teuthology.orchestra.run.smithi113.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T22:18:19.714 INFO:teuthology.orchestra.run.smithi113.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T22:18:19.714 INFO:teuthology.orchestra.run.smithi113.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T22:18:19.714 INFO:teuthology.orchestra.run.smithi113.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T22:18:19.714 INFO:teuthology.orchestra.run.smithi113.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T22:18:19.714 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T22:18:19.714 INFO:teuthology.orchestra.run.smithi113.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T22:18:19.714 INFO:teuthology.orchestra.run.smithi113.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T22:18:19.714 INFO:teuthology.orchestra.run.smithi113.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T22:18:19.714 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T22:18:19.714 INFO:teuthology.orchestra.run.smithi113.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T22:18:19.714 INFO:teuthology.orchestra.run.smithi113.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T22:18:19.715 INFO:teuthology.orchestra.run.smithi113.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T22:18:19.715 INFO:teuthology.orchestra.run.smithi113.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T22:18:19.715 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T22:18:19.715 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T22:18:19.715 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T22:18:19.715 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T22:18:19.715 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T22:18:19.715 INFO:teuthology.orchestra.run.smithi113.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T22:18:19.715 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.715 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.715 INFO:teuthology.orchestra.run.smithi113.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T22:18:19.715 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T22:18:19.716 INFO:teuthology.orchestra.run.smithi113.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T22:18:19.716 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.716 INFO:teuthology.orchestra.run.smithi113.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T22:18:19.716 INFO:teuthology.orchestra.run.smithi113.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T22:18:19.716 INFO:teuthology.orchestra.run.smithi113.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T22:18:19.716 INFO:teuthology.orchestra.run.smithi113.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T22:18:19.716 INFO:teuthology.orchestra.run.smithi113.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T22:18:19.716 INFO:teuthology.orchestra.run.smithi113.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T22:18:19.716 INFO:teuthology.orchestra.run.smithi113.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T22:18:19.716 INFO:teuthology.orchestra.run.smithi113.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T22:18:19.716 INFO:teuthology.orchestra.run.smithi113.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T22:18:19.716 INFO:teuthology.orchestra.run.smithi113.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T22:18:19.716 INFO:teuthology.orchestra.run.smithi113.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T22:18:19.716 INFO:teuthology.orchestra.run.smithi113.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T22:18:19.716 INFO:teuthology.orchestra.run.smithi113.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T22:18:19.717 INFO:teuthology.orchestra.run.smithi113.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T22:18:19.717 INFO:teuthology.orchestra.run.smithi113.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T22:18:19.717 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.717 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.717 INFO:teuthology.orchestra.run.smithi113.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:18:19.717 INFO:teuthology.orchestra.run.smithi113.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T22:18:19.717 INFO:teuthology.orchestra.run.smithi113.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T22:18:19.717 INFO:teuthology.orchestra.run.smithi113.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T22:18:19.717 INFO:teuthology.orchestra.run.smithi113.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T22:18:19.717 INFO:teuthology.orchestra.run.smithi113.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T22:18:19.717 INFO:teuthology.orchestra.run.smithi113.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T22:18:19.718 INFO:teuthology.orchestra.run.smithi113.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T22:18:19.718 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:18:19.718 INFO:teuthology.orchestra.run.smithi113.stdout:Complete! 2024-03-18T22:18:20.171 DEBUG:teuthology.parallel:result is None 2024-03-18T22:27:26.609 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 13 kB/s | 8.0 MB 10:53 2024-03-18T22:27:30.248 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 13 kB/s | 8.0 MB 10:57 2024-03-18T22:27:42.415 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-18T22:27:47.583 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.1 MB 00:01 2024-03-18T22:27:47.922 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:16 2024-03-18T22:27:49.217 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-03-18T22:27:50.235 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-18T22:27:52.956 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.1 MB 00:01 2024-03-18T22:27:54.849 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-03-18T22:27:55.285 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-18T22:27:56.375 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-18T22:27:57.748 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T22:27:57.749 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T22:27:57.761 INFO:teuthology.orchestra.run.smithi040.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T22:27:57.762 INFO:teuthology.orchestra.run.smithi040.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T22:27:57.763 INFO:teuthology.orchestra.run.smithi040.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T22:27:57.764 INFO:teuthology.orchestra.run.smithi040.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T22:27:57.766 INFO:teuthology.orchestra.run.smithi040.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T22:27:57.768 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T22:27:57.769 INFO:teuthology.orchestra.run.smithi040.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T22:27:57.853 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-03-18T22:27:57.873 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================ 2024-03-18T22:27:57.873 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-03-18T22:27:57.873 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================ 2024-03-18T22:27:57.873 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-03-18T22:27:57.873 INFO:teuthology.orchestra.run.smithi040.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T22:27:57.873 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-18T22:27:57.873 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-18T22:27:57.873 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-18T22:27:57.873 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-18T22:27:57.873 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-18T22:27:57.874 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-18T22:27:57.874 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-18T22:27:57.874 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-18T22:27:57.874 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-18T22:27:57.874 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-18T22:27:57.874 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-18T22:27:57.874 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-18T22:27:57.874 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-18T22:27:57.874 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-18T22:27:57.874 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-18T22:27:57.874 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-18T22:27:57.874 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T22:27:57.874 INFO:teuthology.orchestra.run.smithi040.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T22:27:57.875 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T22:27:57.875 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-18T22:27:57.875 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-18T22:27:57.875 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-18T22:27:57.875 INFO:teuthology.orchestra.run.smithi040.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T22:27:57.875 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T22:27:57.875 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T22:27:57.875 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T22:27:57.875 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T22:27:57.875 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T22:27:57.875 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-18T22:27:57.876 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-18T22:27:57.876 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-18T22:27:57.876 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-18T22:27:57.876 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-18T22:27:57.876 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-18T22:27:57.876 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-18T22:27:57.876 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T22:27:57.876 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2024-03-18T22:27:57.876 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-18T22:27:57.876 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-18T22:27:57.876 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-18T22:27:57.876 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-18T22:27:57.876 INFO:teuthology.orchestra.run.smithi040.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T22:27:57.877 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T22:27:57.877 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T22:27:57.877 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T22:27:57.877 INFO:teuthology.orchestra.run.smithi040.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T22:27:57.877 INFO:teuthology.orchestra.run.smithi040.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T22:27:57.877 INFO:teuthology.orchestra.run.smithi040.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T22:27:57.877 INFO:teuthology.orchestra.run.smithi040.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T22:27:57.877 INFO:teuthology.orchestra.run.smithi040.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T22:27:57.877 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-18T22:27:57.877 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-18T22:27:57.877 INFO:teuthology.orchestra.run.smithi040.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T22:27:57.878 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T22:27:57.878 INFO:teuthology.orchestra.run.smithi040.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T22:27:57.878 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T22:27:57.878 INFO:teuthology.orchestra.run.smithi040.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T22:27:57.878 INFO:teuthology.orchestra.run.smithi040.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T22:27:57.878 INFO:teuthology.orchestra.run.smithi040.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T22:27:57.878 INFO:teuthology.orchestra.run.smithi040.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T22:27:57.878 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T22:27:57.878 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-03-18T22:27:57.878 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T22:27:57.878 INFO:teuthology.orchestra.run.smithi040.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T22:27:57.878 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T22:27:57.878 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-18T22:27:57.879 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-18T22:27:57.879 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-18T22:27:57.879 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-18T22:27:57.879 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-18T22:27:57.879 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-18T22:27:57.879 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-18T22:27:57.879 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-18T22:27:57.879 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T22:27:57.879 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T22:27:57.879 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T22:27:57.879 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T22:27:57.879 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T22:27:57.879 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T22:27:57.880 INFO:teuthology.orchestra.run.smithi040.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T22:27:57.880 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T22:27:57.880 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T22:27:57.880 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-18T22:27:57.880 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T22:27:57.880 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T22:27:57.880 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T22:27:57.880 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T22:27:57.880 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T22:27:57.880 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-18T22:27:57.880 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T22:27:57.880 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-18T22:27:57.881 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T22:27:57.881 INFO:teuthology.orchestra.run.smithi040.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T22:27:57.881 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T22:27:57.881 INFO:teuthology.orchestra.run.smithi040.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T22:27:57.881 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T22:27:57.881 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T22:27:57.881 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T22:27:57.881 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T22:27:57.881 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T22:27:57.881 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T22:27:57.881 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T22:27:57.881 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T22:27:57.881 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T22:27:57.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T22:27:57.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T22:27:57.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T22:27:57.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T22:27:57.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T22:27:57.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-18T22:27:57.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-18T22:27:57.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T22:27:57.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T22:27:57.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T22:27:57.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T22:27:57.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T22:27:57.882 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T22:27:57.883 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T22:27:57.883 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T22:27:57.883 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T22:27:57.883 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T22:27:57.883 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T22:27:57.883 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T22:27:57.885 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T22:27:57.885 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T22:27:57.885 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T22:27:57.885 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T22:27:57.885 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T22:27:57.885 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T22:27:57.885 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T22:27:57.885 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T22:27:57.885 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T22:27:57.885 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T22:27:57.885 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T22:27:57.885 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T22:27:57.885 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T22:27:57.886 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T22:27:57.886 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T22:27:57.886 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T22:27:57.886 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T22:27:57.886 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T22:27:57.886 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T22:27:57.886 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T22:27:57.886 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T22:27:57.886 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T22:27:57.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T22:27:57.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T22:27:57.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T22:27:57.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T22:27:57.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T22:27:57.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T22:27:57.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T22:27:57.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T22:27:57.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T22:27:57.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T22:27:57.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T22:27:57.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T22:27:57.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T22:27:57.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T22:27:57.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T22:27:57.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T22:27:57.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T22:27:57.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T22:27:57.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T22:27:57.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T22:27:57.888 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T22:27:57.888 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T22:27:57.888 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T22:27:57.888 INFO:teuthology.orchestra.run.smithi040.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T22:27:57.888 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T22:27:57.888 INFO:teuthology.orchestra.run.smithi040.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T22:27:57.888 INFO:teuthology.orchestra.run.smithi040.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T22:27:57.889 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T22:27:57.889 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T22:27:57.889 INFO:teuthology.orchestra.run.smithi040.stdout:Installing weak dependencies: 2024-03-18T22:27:57.889 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T22:27:57.889 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:27:57.889 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-03-18T22:27:57.889 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================================ 2024-03-18T22:27:57.889 INFO:teuthology.orchestra.run.smithi040.stdout:Install 150 Packages 2024-03-18T22:27:57.889 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 24 Packages 2024-03-18T22:27:57.890 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:27:57.890 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 224 M 2024-03-18T22:27:57.890 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-03-18T22:27:58.553 INFO:teuthology.orchestra.run.smithi040.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 30 kB/s | 6.6 kB 00:00 2024-03-18T22:27:58.812 INFO:teuthology.orchestra.run.smithi040.stdout:(2/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 3.4 MB/s | 891 kB 00:00 2024-03-18T22:27:58.912 INFO:teuthology.orchestra.run.smithi040.stdout:(3/174): ceph-immutable-object-cache-19.0.0-220 1.4 MB/s | 145 kB 00:00 2024-03-18T22:27:59.304 INFO:teuthology.orchestra.run.smithi040.stdout:(4/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 6.0 MB/s | 2.4 MB 00:00 2024-03-18T22:27:59.387 INFO:teuthology.orchestra.run.smithi040.stdout:(5/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 5.1 MB/s | 5.4 MB 00:01 2024-03-18T22:27:59.613 INFO:teuthology.orchestra.run.smithi040.stdout:(6/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 4.8 MB/s | 1.5 MB 00:00 2024-03-18T22:27:59.871 INFO:teuthology.orchestra.run.smithi040.stdout:(7/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 9.7 MB/s | 4.7 MB 00:00 2024-03-18T22:28:00.422 INFO:teuthology.orchestra.run.smithi040.stdout:(8/174): ceph-common-19.0.0-2206.gb6ae2491.el9. 9.9 MB/s | 21 MB 00:02 2024-03-18T22:28:00.497 INFO:teuthology.orchestra.run.smithi040.stdout:(9/174): ceph-selinux-19.0.0-2206.gb6ae2491.el9 336 kB/s | 25 kB 00:00 2024-03-18T22:28:00.747 INFO:teuthology.orchestra.run.smithi040.stdout:(10/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el 11 MB/s | 9.8 MB 00:00 2024-03-18T22:28:00.839 INFO:teuthology.orchestra.run.smithi040.stdout:(11/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 2.4 MB/s | 221 kB 00:00 2024-03-18T22:28:00.922 INFO:teuthology.orchestra.run.smithi040.stdout:(12/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 325 kB/s | 27 kB 00:00 2024-03-18T22:28:00.998 INFO:teuthology.orchestra.run.smithi040.stdout:(13/174): libcephfs-devel-19.0.0-2206.gb6ae2491 423 kB/s | 32 kB 00:00 2024-03-18T22:28:01.115 INFO:teuthology.orchestra.run.smithi040.stdout:(14/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 6.2 MB/s | 746 kB 00:00 2024-03-18T22:28:01.192 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-18T22:28:01.198 INFO:teuthology.orchestra.run.smithi040.stdout:(15/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 1.9 MB/s | 163 kB 00:00 2024-03-18T22:28:01.282 INFO:teuthology.orchestra.run.smithi040.stdout:(16/174): librados-devel-19.0.0-2206.gb6ae2491. 1.5 MB/s | 127 kB 00:00 2024-03-18T22:28:01.399 INFO:teuthology.orchestra.run.smithi040.stdout:(17/174): libradosstriper1-19.0.0-2206.gb6ae249 4.2 MB/s | 501 kB 00:00 2024-03-18T22:28:02.016 INFO:teuthology.orchestra.run.smithi040.stdout:(18/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 8.3 MB/s | 5.1 MB 00:00 2024-03-18T22:28:02.192 INFO:teuthology.orchestra.run.smithi040.stdout:(19/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x8 6.5 MB/s | 17 MB 00:02 2024-03-18T22:28:02.216 INFO:teuthology.orchestra.run.smithi040.stdout:(20/174): python3-ceph-argparse-19.0.0-2206.gb6 225 kB/s | 45 kB 00:00 2024-03-18T22:28:02.250 INFO:teuthology.orchestra.run.smithi040.stdout:(21/174): python3-ceph-common-19.0.0-2206.gb6ae 2.2 MB/s | 131 kB 00:00 2024-03-18T22:28:02.275 INFO:teuthology.orchestra.run.smithi040.stdout:(22/174): python3-cephfs-19.0.0-2206.gb6ae2491. 2.7 MB/s | 162 kB 00:00 2024-03-18T22:28:02.325 INFO:teuthology.orchestra.run.smithi040.stdout:(23/174): python3-rados-19.0.0-2206.gb6ae2491.e 4.2 MB/s | 321 kB 00:00 2024-03-18T22:28:02.359 INFO:teuthology.orchestra.run.smithi040.stdout:(24/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 3.5 MB/s | 298 kB 00:00 2024-03-18T22:28:02.409 INFO:teuthology.orchestra.run.smithi040.stdout:(25/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 1.2 MB/s | 99 kB 00:00 2024-03-18T22:28:02.435 INFO:teuthology.orchestra.run.smithi040.stdout:(26/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-18T22:28:02.527 INFO:teuthology.orchestra.run.smithi040.stdout:(27/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-18T22:28:02.619 INFO:teuthology.orchestra.run.smithi040.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2206.g 370 kB/s | 34 kB 00:00 2024-03-18T22:28:02.711 INFO:teuthology.orchestra.run.smithi040.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 1.5 MB/s | 139 kB 00:00 2024-03-18T22:28:02.870 INFO:teuthology.orchestra.run.smithi040.stdout:(30/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 6.7 MB/s | 3.1 MB 00:00 2024-03-18T22:28:02.911 INFO:teuthology.orchestra.run.smithi040.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 9.0 MB/s | 1.8 MB 00:00 2024-03-18T22:28:02.987 INFO:teuthology.orchestra.run.smithi040.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2206.gb6 3.2 MB/s | 248 kB 00:00 2024-03-18T22:28:03.070 INFO:teuthology.orchestra.run.smithi040.stdout:(33/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 593 kB/s | 49 kB 00:00 2024-03-18T22:28:03.154 INFO:teuthology.orchestra.run.smithi040.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2206.gb 198 kB/s | 16 kB 00:00 2024-03-18T22:28:03.263 INFO:teuthology.orchestra.run.smithi040.stdout:(35/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 2.4 MB/s | 264 kB 00:00 2024-03-18T22:28:03.371 INFO:teuthology.orchestra.run.smithi040.stdout:(36/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 6.8 MB/s | 749 kB 00:00 2024-03-18T22:28:03.471 INFO:teuthology.orchestra.run.smithi040.stdout:(37/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 183 kB/s | 18 kB 00:00 2024-03-18T22:28:03.554 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T22:28:03.554 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T22:28:03.566 INFO:teuthology.orchestra.run.smithi103.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T22:28:03.567 INFO:teuthology.orchestra.run.smithi103.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T22:28:03.568 INFO:teuthology.orchestra.run.smithi103.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T22:28:03.569 INFO:teuthology.orchestra.run.smithi103.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T22:28:03.571 INFO:teuthology.orchestra.run.smithi103.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T22:28:03.573 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T22:28:03.574 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T22:28:03.653 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-03-18T22:28:03.686 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================================ 2024-03-18T22:28:03.687 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-03-18T22:28:03.687 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================================ 2024-03-18T22:28:03.687 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-03-18T22:28:03.687 INFO:teuthology.orchestra.run.smithi103.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T22:28:03.687 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-18T22:28:03.687 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-18T22:28:03.687 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-18T22:28:03.687 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-18T22:28:03.687 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-18T22:28:03.687 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-18T22:28:03.687 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-18T22:28:03.687 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-18T22:28:03.687 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-18T22:28:03.688 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-18T22:28:03.688 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-18T22:28:03.688 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-18T22:28:03.688 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-18T22:28:03.688 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-18T22:28:03.688 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-18T22:28:03.688 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-18T22:28:03.688 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T22:28:03.688 INFO:teuthology.orchestra.run.smithi103.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T22:28:03.688 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T22:28:03.688 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-18T22:28:03.688 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-18T22:28:03.688 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-18T22:28:03.688 INFO:teuthology.orchestra.run.smithi103.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T22:28:03.688 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T22:28:03.689 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T22:28:03.689 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T22:28:03.689 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T22:28:03.689 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T22:28:03.689 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-18T22:28:03.689 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-18T22:28:03.689 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-18T22:28:03.689 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-18T22:28:03.689 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-18T22:28:03.689 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-18T22:28:03.689 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-18T22:28:03.689 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T22:28:03.689 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-03-18T22:28:03.690 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-18T22:28:03.690 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-18T22:28:03.690 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-18T22:28:03.690 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-18T22:28:03.690 INFO:teuthology.orchestra.run.smithi103.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T22:28:03.690 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T22:28:03.690 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T22:28:03.690 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T22:28:03.690 INFO:teuthology.orchestra.run.smithi103.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T22:28:03.690 INFO:teuthology.orchestra.run.smithi103.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T22:28:03.690 INFO:teuthology.orchestra.run.smithi103.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T22:28:03.690 INFO:teuthology.orchestra.run.smithi103.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T22:28:03.690 INFO:teuthology.orchestra.run.smithi103.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T22:28:03.691 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-18T22:28:03.691 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-18T22:28:03.691 INFO:teuthology.orchestra.run.smithi103.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T22:28:03.691 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T22:28:03.691 INFO:teuthology.orchestra.run.smithi103.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T22:28:03.691 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T22:28:03.691 INFO:teuthology.orchestra.run.smithi103.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T22:28:03.691 INFO:teuthology.orchestra.run.smithi103.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T22:28:03.691 INFO:teuthology.orchestra.run.smithi103.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T22:28:03.691 INFO:teuthology.orchestra.run.smithi103.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T22:28:03.691 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T22:28:03.691 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-03-18T22:28:03.691 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T22:28:03.691 INFO:teuthology.orchestra.run.smithi103.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T22:28:03.692 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T22:28:03.692 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-18T22:28:03.692 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-18T22:28:03.692 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-18T22:28:03.692 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-18T22:28:03.692 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-18T22:28:03.692 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-18T22:28:03.692 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-18T22:28:03.692 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-18T22:28:03.692 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T22:28:03.692 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T22:28:03.692 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T22:28:03.692 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T22:28:03.693 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T22:28:03.693 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T22:28:03.693 INFO:teuthology.orchestra.run.smithi103.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T22:28:03.693 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T22:28:03.693 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T22:28:03.693 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-18T22:28:03.693 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T22:28:03.693 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T22:28:03.693 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T22:28:03.693 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T22:28:03.693 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T22:28:03.693 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-18T22:28:03.694 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T22:28:03.694 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-18T22:28:03.694 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T22:28:03.694 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T22:28:03.694 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T22:28:03.694 INFO:teuthology.orchestra.run.smithi103.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T22:28:03.694 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T22:28:03.694 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T22:28:03.694 INFO:teuthology.orchestra.run.smithi103.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T22:28:03.694 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T22:28:03.694 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T22:28:03.694 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T22:28:03.695 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T22:28:03.695 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T22:28:03.695 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T22:28:03.695 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T22:28:03.695 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T22:28:03.695 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T22:28:03.695 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T22:28:03.695 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T22:28:03.695 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-18T22:28:03.695 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-18T22:28:03.695 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T22:28:03.695 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T22:28:03.695 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T22:28:03.695 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T22:28:03.696 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T22:28:03.696 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T22:28:03.696 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T22:28:03.696 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T22:28:03.696 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T22:28:03.696 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T22:28:03.696 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T22:28:03.696 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T22:28:03.696 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T22:28:03.696 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T22:28:03.696 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T22:28:03.696 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T22:28:03.696 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T22:28:03.696 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T22:28:03.696 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T22:28:03.697 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T22:28:03.697 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T22:28:03.697 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T22:28:03.697 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T22:28:03.697 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T22:28:03.697 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T22:28:03.697 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T22:28:03.697 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T22:28:03.697 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T22:28:03.697 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T22:28:03.697 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T22:28:03.697 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T22:28:03.697 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T22:28:03.697 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T22:28:03.698 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T22:28:03.698 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T22:28:03.698 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T22:28:03.698 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T22:28:03.698 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T22:28:03.698 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T22:28:03.698 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T22:28:03.698 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T22:28:03.698 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T22:28:03.698 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T22:28:03.698 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T22:28:03.698 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T22:28:03.698 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T22:28:03.699 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T22:28:03.700 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T22:28:03.700 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-03-18T22:28:03.700 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T22:28:03.700 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:28:03.700 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-03-18T22:28:03.700 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================================ 2024-03-18T22:28:03.700 INFO:teuthology.orchestra.run.smithi103.stdout:Install 150 Packages 2024-03-18T22:28:03.700 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 24 Packages 2024-03-18T22:28:03.700 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:28:03.701 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 224 M 2024-03-18T22:28:03.701 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-03-18T22:28:03.905 INFO:teuthology.orchestra.run.smithi040.stdout:(38/174): ceph-test-19.0.0-2206.gb6ae2491.el9.x 14 MB/s | 48 MB 00:03 2024-03-18T22:28:04.055 INFO:teuthology.orchestra.run.smithi040.stdout:(39/174): ceph-mgr-diskprediction-local-19.0.0- 6.2 MB/s | 7.4 MB 00:01 2024-03-18T22:28:04.162 INFO:teuthology.orchestra.run.smithi040.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 3.9 MB/s | 920 kB 00:00 2024-03-18T22:28:04.187 INFO:teuthology.orchestra.run.smithi040.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 925 kB/s | 662 kB 00:00 2024-03-18T22:28:04.212 INFO:teuthology.orchestra.run.smithi040.stdout:(42/174): boost-program-options-1.75.0-8.el9.x8 683 kB/s | 106 kB 00:00 2024-03-18T22:28:04.238 INFO:teuthology.orchestra.run.smithi040.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 4.0 MB/s | 307 kB 00:00 2024-03-18T22:28:04.263 INFO:teuthology.orchestra.run.smithi040.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 631 kB/s | 32 kB 00:00 2024-03-18T22:28:04.364 INFO:teuthology.orchestra.run.smithi040.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 137 kB/s | 17 kB 00:00 2024-03-18T22:28:04.398 INFO:teuthology.orchestra.run.smithi040.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 75 kB/s | 10 kB 00:00 2024-03-18T22:28:04.401 INFO:teuthology.orchestra.run.smithi103.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-03-18T22:28:04.456 INFO:teuthology.orchestra.run.smithi040.stdout:(47/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-03-18T22:28:04.481 INFO:teuthology.orchestra.run.smithi040.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-18T22:28:04.498 INFO:teuthology.orchestra.run.smithi040.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 442 kB/s | 44 kB 00:00 2024-03-18T22:28:04.540 INFO:teuthology.orchestra.run.smithi040.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-03-18T22:28:04.557 INFO:teuthology.orchestra.run.smithi040.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 811 kB/s | 46 kB 00:00 2024-03-18T22:28:04.582 INFO:teuthology.orchestra.run.smithi040.stdout:(52/174): libtool-2.4.6-45.el9.x86_64.rpm 6.8 MB/s | 578 kB 00:00 2024-03-18T22:28:04.608 INFO:teuthology.orchestra.run.smithi040.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-18T22:28:04.633 INFO:teuthology.orchestra.run.smithi040.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-03-18T22:28:04.635 INFO:teuthology.orchestra.run.smithi103.stdout:(2/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 3.7 MB/s | 891 kB 00:00 2024-03-18T22:28:04.658 INFO:teuthology.orchestra.run.smithi040.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-18T22:28:04.683 INFO:teuthology.orchestra.run.smithi040.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 687 kB/s | 34 kB 00:00 2024-03-18T22:28:04.702 INFO:teuthology.orchestra.run.smithi103.stdout:(3/174): ceph-immutable-object-cache-19.0.0-220 2.1 MB/s | 145 kB 00:00 2024-03-18T22:28:04.892 INFO:teuthology.orchestra.run.smithi040.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-03-18T22:28:04.934 INFO:teuthology.orchestra.run.smithi040.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 3.2 MB/s | 811 kB 00:00 2024-03-18T22:28:04.976 INFO:teuthology.orchestra.run.smithi040.stdout:(59/174): postgresql-13.11-1.el9.x86_64.rpm 4.8 MB/s | 1.5 MB 00:00 2024-03-18T22:28:05.027 INFO:teuthology.orchestra.run.smithi103.stdout:(4/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 7.3 MB/s | 2.4 MB 00:00 2024-03-18T22:28:05.084 INFO:teuthology.orchestra.run.smithi040.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 700 kB/s | 134 kB 00:00 2024-03-18T22:28:05.202 INFO:teuthology.orchestra.run.smithi040.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 21 MB/s | 5.7 MB 00:00 2024-03-18T22:28:05.226 INFO:teuthology.orchestra.run.smithi040.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 561 kB/s | 78 kB 00:00 2024-03-18T22:28:05.268 INFO:teuthology.orchestra.run.smithi040.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.6 MB/s | 1.0 MB 00:00 2024-03-18T22:28:05.278 INFO:teuthology.orchestra.run.smithi103.stdout:(5/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 6.0 MB/s | 1.5 MB 00:00 2024-03-18T22:28:05.394 INFO:teuthology.orchestra.run.smithi040.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 5.8 MB 00:00 2024-03-18T22:28:05.419 INFO:teuthology.orchestra.run.smithi040.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 129 kB/s | 24 kB 00:00 2024-03-18T22:28:05.444 INFO:teuthology.orchestra.run.smithi040.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 105 kB/s | 18 kB 00:00 2024-03-18T22:28:05.469 INFO:teuthology.orchestra.run.smithi040.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-18T22:28:05.494 INFO:teuthology.orchestra.run.smithi040.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 635 kB/s | 32 kB 00:00 2024-03-18T22:28:05.595 INFO:teuthology.orchestra.run.smithi040.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-18T22:28:05.620 INFO:teuthology.orchestra.run.smithi040.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-18T22:28:05.645 INFO:teuthology.orchestra.run.smithi040.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-18T22:28:05.670 INFO:teuthology.orchestra.run.smithi040.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 921 kB/s | 69 kB 00:00 2024-03-18T22:28:05.696 INFO:teuthology.orchestra.run.smithi040.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 824 kB/s | 41 kB 00:00 2024-03-18T22:28:05.721 INFO:teuthology.orchestra.run.smithi040.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-18T22:28:05.728 INFO:teuthology.orchestra.run.smithi103.stdout:(6/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 3.6 MB/s | 5.4 MB 00:01 2024-03-18T22:28:05.746 INFO:teuthology.orchestra.run.smithi040.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-18T22:28:05.772 INFO:teuthology.orchestra.run.smithi040.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 948 kB/s | 47 kB 00:00 2024-03-18T22:28:05.797 INFO:teuthology.orchestra.run.smithi040.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-18T22:28:05.823 INFO:teuthology.orchestra.run.smithi040.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 566 kB/s | 43 kB 00:00 2024-03-18T22:28:05.937 INFO:teuthology.orchestra.run.smithi103.stdout:(7/174): ceph-common-19.0.0-2206.gb6ae2491.el9. 12 MB/s | 21 MB 00:01 2024-03-18T22:28:06.003 INFO:teuthology.orchestra.run.smithi103.stdout:(8/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 6.5 MB/s | 4.7 MB 00:00 2024-03-18T22:28:06.087 INFO:teuthology.orchestra.run.smithi103.stdout:(9/174): ceph-selinux-19.0.0-2206.gb6ae2491.el9 302 kB/s | 25 kB 00:00 2024-03-18T22:28:06.171 INFO:teuthology.orchestra.run.smithi040.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-18T22:28:06.198 INFO:teuthology.orchestra.run.smithi040.stdout:(80/174): python3-toml-0.10.2-6.el9.noarch.rpm 104 kB/s | 39 kB 00:00 2024-03-18T22:28:06.223 INFO:teuthology.orchestra.run.smithi040.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 103 kB/s | 41 kB 00:00 2024-03-18T22:28:06.248 INFO:teuthology.orchestra.run.smithi040.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.8 MB/s | 300 kB 00:00 2024-03-18T22:28:06.273 INFO:teuthology.orchestra.run.smithi040.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-18T22:28:06.298 INFO:teuthology.orchestra.run.smithi040.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 839 kB/s | 63 kB 00:00 2024-03-18T22:28:06.324 INFO:teuthology.orchestra.run.smithi040.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 585 kB/s | 44 kB 00:00 2024-03-18T22:28:06.349 INFO:teuthology.orchestra.run.smithi040.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-18T22:28:06.374 INFO:teuthology.orchestra.run.smithi040.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-18T22:28:06.408 INFO:teuthology.orchestra.run.smithi040.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-18T22:28:06.433 INFO:teuthology.orchestra.run.smithi040.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-18T22:28:06.458 INFO:teuthology.orchestra.run.smithi040.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-18T22:28:06.475 INFO:teuthology.orchestra.run.smithi040.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.7 MB/s | 187 kB 00:00 2024-03-18T22:28:06.671 INFO:teuthology.orchestra.run.smithi103.stdout:(10/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el 13 MB/s | 9.8 MB 00:00 2024-03-18T22:28:06.738 INFO:teuthology.orchestra.run.smithi103.stdout:(11/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 3.3 MB/s | 221 kB 00:00 2024-03-18T22:28:06.796 INFO:teuthology.orchestra.run.smithi103.stdout:(12/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 466 kB/s | 27 kB 00:00 2024-03-18T22:28:06.863 INFO:teuthology.orchestra.run.smithi103.stdout:(13/174): libcephfs-devel-19.0.0-2206.gb6ae2491 476 kB/s | 32 kB 00:00 2024-03-18T22:28:06.946 INFO:teuthology.orchestra.run.smithi103.stdout:(14/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 8.8 MB/s | 746 kB 00:00 2024-03-18T22:28:06.975 INFO:teuthology.orchestra.run.smithi040.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 362 kB/s | 187 kB 00:00 2024-03-18T22:28:07.001 INFO:teuthology.orchestra.run.smithi040.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 372 kB/s | 195 kB 00:00 2024-03-18T22:28:07.013 INFO:teuthology.orchestra.run.smithi103.stdout:(15/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 2.4 MB/s | 163 kB 00:00 2024-03-18T22:28:07.072 INFO:teuthology.orchestra.run.smithi103.stdout:(16/174): librados-devel-19.0.0-2206.gb6ae2491. 2.1 MB/s | 127 kB 00:00 2024-03-18T22:28:07.084 INFO:teuthology.orchestra.run.smithi040.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-03-18T22:28:07.101 INFO:teuthology.orchestra.run.smithi040.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 214 kB/s | 21 kB 00:00 2024-03-18T22:28:07.147 INFO:teuthology.orchestra.run.smithi103.stdout:(17/174): libradosstriper1-19.0.0-2206.gb6ae249 6.5 MB/s | 501 kB 00:00 2024-03-18T22:28:07.185 INFO:teuthology.orchestra.run.smithi040.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 254 kB/s | 25 kB 00:00 2024-03-18T22:28:07.210 INFO:teuthology.orchestra.run.smithi040.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-03-18T22:28:07.319 INFO:teuthology.orchestra.run.smithi040.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-03-18T22:28:07.369 INFO:teuthology.orchestra.run.smithi040.stdout:(99/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-18T22:28:07.430 INFO:teuthology.orchestra.run.smithi040.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.9 MB/s | 216 kB 00:00 2024-03-18T22:28:07.471 INFO:teuthology.orchestra.run.smithi040.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 341 kB/s | 35 kB 00:00 2024-03-18T22:28:07.538 INFO:teuthology.orchestra.run.smithi040.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 410 kB/s | 44 kB 00:00 2024-03-18T22:28:07.572 INFO:teuthology.orchestra.run.smithi040.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 934 kB/s | 94 kB 00:00 2024-03-18T22:28:07.573 INFO:teuthology.orchestra.run.smithi103.stdout:(18/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 12 MB/s | 5.1 MB 00:00 2024-03-18T22:28:07.631 INFO:teuthology.orchestra.run.smithi103.stdout:(19/174): python3-ceph-argparse-19.0.0-2206.gb6 776 kB/s | 45 kB 00:00 2024-03-18T22:28:07.639 INFO:teuthology.orchestra.run.smithi040.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-03-18T22:28:07.698 INFO:teuthology.orchestra.run.smithi103.stdout:(20/174): python3-ceph-common-19.0.0-2206.gb6ae 1.9 MB/s | 131 kB 00:00 2024-03-18T22:28:07.765 INFO:teuthology.orchestra.run.smithi103.stdout:(21/174): python3-cephfs-19.0.0-2206.gb6ae2491. 2.4 MB/s | 162 kB 00:00 2024-03-18T22:28:07.840 INFO:teuthology.orchestra.run.smithi103.stdout:(22/174): python3-rados-19.0.0-2206.gb6ae2491.e 4.2 MB/s | 321 kB 00:00 2024-03-18T22:28:07.847 INFO:teuthology.orchestra.run.smithi040.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.1 MB/s | 863 kB 00:00 2024-03-18T22:28:07.915 INFO:teuthology.orchestra.run.smithi103.stdout:(23/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 3.9 MB/s | 298 kB 00:00 2024-03-18T22:28:07.982 INFO:teuthology.orchestra.run.smithi103.stdout:(24/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 1.5 MB/s | 99 kB 00:00 2024-03-18T22:28:07.998 INFO:teuthology.orchestra.run.smithi040.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 4.6 MB/s | 4.1 MB 00:00 2024-03-18T22:28:08.049 INFO:teuthology.orchestra.run.smithi103.stdout:(25/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-18T22:28:08.408 INFO:teuthology.orchestra.run.smithi103.stdout:(26/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x8 6.3 MB/s | 17 MB 00:02 2024-03-18T22:28:08.467 INFO:teuthology.orchestra.run.smithi103.stdout:(27/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 7.4 MB/s | 3.1 MB 00:00 2024-03-18T22:28:08.484 INFO:teuthology.orchestra.run.smithi103.stdout:(28/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 2.2 MB/s | 170 kB 00:00 2024-03-18T22:28:08.526 INFO:teuthology.orchestra.run.smithi103.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2206.g 579 kB/s | 34 kB 00:00 2024-03-18T22:28:08.559 INFO:teuthology.orchestra.run.smithi103.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 1.8 MB/s | 139 kB 00:00 2024-03-18T22:28:08.710 INFO:teuthology.orchestra.run.smithi103.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 9.8 MB/s | 1.8 MB 00:00 2024-03-18T22:28:08.777 INFO:teuthology.orchestra.run.smithi103.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2206.gb6 3.7 MB/s | 248 kB 00:00 2024-03-18T22:28:08.843 INFO:teuthology.orchestra.run.smithi103.stdout:(33/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 743 kB/s | 49 kB 00:00 2024-03-18T22:28:08.910 INFO:teuthology.orchestra.run.smithi103.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2206.gb 246 kB/s | 16 kB 00:00 2024-03-18T22:28:08.994 INFO:teuthology.orchestra.run.smithi103.stdout:(35/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 3.1 MB/s | 264 kB 00:00 2024-03-18T22:28:09.086 INFO:teuthology.orchestra.run.smithi103.stdout:(36/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 8.0 MB/s | 749 kB 00:00 2024-03-18T22:28:09.153 INFO:teuthology.orchestra.run.smithi103.stdout:(37/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 274 kB/s | 18 kB 00:00 2024-03-18T22:28:09.303 INFO:teuthology.orchestra.run.smithi103.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 662 kB 00:00 2024-03-18T22:28:09.370 INFO:teuthology.orchestra.run.smithi103.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 920 kB 00:00 2024-03-18T22:28:09.403 INFO:teuthology.orchestra.run.smithi103.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-18T22:28:09.446 INFO:teuthology.orchestra.run.smithi103.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-18T22:28:09.496 INFO:teuthology.orchestra.run.smithi103.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 636 kB/s | 32 kB 00:00 2024-03-18T22:28:09.580 INFO:teuthology.orchestra.run.smithi103.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 36 MB/s | 3.0 MB 00:00 2024-03-18T22:28:09.672 INFO:teuthology.orchestra.run.smithi103.stdout:(44/174): ceph-mgr-diskprediction-local-19.0.0- 6.6 MB/s | 7.4 MB 00:01 2024-03-18T22:28:09.697 INFO:teuthology.orchestra.run.smithi103.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 146 kB/s | 17 kB 00:00 2024-03-18T22:28:09.722 INFO:teuthology.orchestra.run.smithi103.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-18T22:28:09.748 INFO:teuthology.orchestra.run.smithi103.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-18T22:28:09.773 INFO:teuthology.orchestra.run.smithi103.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-18T22:28:09.806 INFO:teuthology.orchestra.run.smithi103.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-18T22:28:09.831 INFO:teuthology.orchestra.run.smithi103.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-03-18T22:28:09.857 INFO:teuthology.orchestra.run.smithi103.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 940 kB/s | 46 kB 00:00 2024-03-18T22:28:09.882 INFO:teuthology.orchestra.run.smithi103.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-18T22:28:09.907 INFO:teuthology.orchestra.run.smithi103.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-18T22:28:09.933 INFO:teuthology.orchestra.run.smithi103.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-18T22:28:09.958 INFO:teuthology.orchestra.run.smithi103.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 684 kB/s | 34 kB 00:00 2024-03-18T22:28:10.375 INFO:teuthology.orchestra.run.smithi103.stdout:(56/174): ceph-test-19.0.0-2206.gb6ae2491.el9.x 11 MB/s | 48 MB 00:04 2024-03-18T22:28:10.451 INFO:teuthology.orchestra.run.smithi103.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 9.2 MB/s | 4.7 MB 00:00 2024-03-18T22:28:10.493 INFO:teuthology.orchestra.run.smithi103.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 2.9 MB/s | 1.5 MB 00:00 2024-03-18T22:28:10.526 INFO:teuthology.orchestra.run.smithi103.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 5.4 MB/s | 811 kB 00:00 2024-03-18T22:28:10.551 INFO:teuthology.orchestra.run.smithi103.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.3 MB/s | 134 kB 00:00 2024-03-18T22:28:10.602 INFO:teuthology.orchestra.run.smithi103.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-03-18T22:28:10.702 INFO:teuthology.orchestra.run.smithi103.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 32 MB/s | 5.7 MB 00:00 2024-03-18T22:28:10.728 INFO:teuthology.orchestra.run.smithi103.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 445 kB/s | 78 kB 00:00 2024-03-18T22:28:10.811 INFO:teuthology.orchestra.run.smithi103.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 28 MB/s | 5.8 MB 00:00 2024-03-18T22:28:10.836 INFO:teuthology.orchestra.run.smithi103.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 186 kB/s | 24 kB 00:00 2024-03-18T22:28:10.861 INFO:teuthology.orchestra.run.smithi103.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 138 kB/s | 18 kB 00:00 2024-03-18T22:28:10.895 INFO:teuthology.orchestra.run.smithi103.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-18T22:28:10.920 INFO:teuthology.orchestra.run.smithi103.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 542 kB/s | 32 kB 00:00 2024-03-18T22:28:10.995 INFO:teuthology.orchestra.run.smithi103.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-18T22:28:11.021 INFO:teuthology.orchestra.run.smithi103.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-18T22:28:11.046 INFO:teuthology.orchestra.run.smithi103.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 174 kB 00:00 2024-03-18T22:28:11.071 INFO:teuthology.orchestra.run.smithi103.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 914 kB/s | 69 kB 00:00 2024-03-18T22:28:11.096 INFO:teuthology.orchestra.run.smithi103.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-03-18T22:28:11.121 INFO:teuthology.orchestra.run.smithi103.stdout:(74/174): python3-prettytable-0.7.2-27.el9.noar 547 kB/s | 41 kB 00:00 2024-03-18T22:28:11.147 INFO:teuthology.orchestra.run.smithi103.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-18T22:28:11.172 INFO:teuthology.orchestra.run.smithi103.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-18T22:28:11.205 INFO:teuthology.orchestra.run.smithi103.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 742 kB/s | 43 kB 00:00 2024-03-18T22:28:11.231 INFO:teuthology.orchestra.run.smithi103.stdout:(78/174): python3-pytz-2021.1-5.el9.noarch.rpm 570 kB/s | 47 kB 00:00 2024-03-18T22:28:11.381 INFO:teuthology.orchestra.run.smithi103.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 223 kB/s | 39 kB 00:00 2024-03-18T22:28:11.406 INFO:teuthology.orchestra.run.smithi103.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 238 kB/s | 41 kB 00:00 2024-03-18T22:28:11.573 INFO:teuthology.orchestra.run.smithi103.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-18T22:28:11.598 INFO:teuthology.orchestra.run.smithi103.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-18T22:28:11.623 INFO:teuthology.orchestra.run.smithi103.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 261 kB/s | 56 kB 00:00 2024-03-18T22:28:11.649 INFO:teuthology.orchestra.run.smithi103.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-18T22:28:11.674 INFO:teuthology.orchestra.run.smithi103.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-03-18T22:28:11.699 INFO:teuthology.orchestra.run.smithi103.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-18T22:28:11.725 INFO:teuthology.orchestra.run.smithi103.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-18T22:28:11.767 INFO:teuthology.orchestra.run.smithi103.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-18T22:28:11.792 INFO:teuthology.orchestra.run.smithi103.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-18T22:28:11.817 INFO:teuthology.orchestra.run.smithi103.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-18T22:28:11.842 INFO:teuthology.orchestra.run.smithi103.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-18T22:28:12.018 INFO:teuthology.orchestra.run.smithi103.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-18T22:28:12.043 INFO:teuthology.orchestra.run.smithi103.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 747 kB/s | 187 kB 00:00 2024-03-18T22:28:12.101 INFO:teuthology.orchestra.run.smithi103.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-18T22:28:12.127 INFO:teuthology.orchestra.run.smithi103.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 257 kB/s | 21 kB 00:00 2024-03-18T22:28:12.185 INFO:teuthology.orchestra.run.smithi103.stdout:(96/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-03-18T22:28:12.211 INFO:teuthology.orchestra.run.smithi103.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 234 kB/s | 25 kB 00:00 2024-03-18T22:28:12.277 INFO:teuthology.orchestra.run.smithi103.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-03-18T22:28:12.328 INFO:teuthology.orchestra.run.smithi103.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 216 kB 00:00 2024-03-18T22:28:12.353 INFO:teuthology.orchestra.run.smithi103.stdout:(100/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.7 MB/s | 248 kB 00:00 2024-03-18T22:28:12.378 INFO:teuthology.orchestra.run.smithi103.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 696 kB/s | 35 kB 00:00 2024-03-18T22:28:12.403 INFO:teuthology.orchestra.run.smithi103.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-03-18T22:28:12.429 INFO:teuthology.orchestra.run.smithi103.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-18T22:28:12.446 INFO:teuthology.orchestra.run.smithi103.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 534 kB/s | 22 kB 00:00 2024-03-18T22:28:12.513 INFO:teuthology.orchestra.run.smithi103.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 11 MB/s | 4.1 MB 00:00 2024-03-18T22:28:12.755 INFO:teuthology.orchestra.run.smithi103.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.6 MB/s | 863 kB 00:00 2024-03-18T22:28:12.943 INFO:teuthology.orchestra.run.smithi103.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 719 kB/s | 309 kB 00:00 2024-03-18T22:28:12.968 INFO:teuthology.orchestra.run.smithi103.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-03-18T22:28:12.994 INFO:teuthology.orchestra.run.smithi103.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 82 kB/s | 19 kB 00:00 2024-03-18T22:28:13.044 INFO:teuthology.orchestra.run.smithi103.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-03-18T22:28:13.128 INFO:teuthology.orchestra.run.smithi103.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 362 kB/s | 49 kB 00:00 2024-03-18T22:28:13.153 INFO:teuthology.orchestra.run.smithi103.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 619 kB/s | 67 kB 00:00 2024-03-18T22:28:13.270 INFO:teuthology.orchestra.run.smithi103.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-18T22:28:13.295 INFO:teuthology.orchestra.run.smithi103.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 904 kB/s | 151 kB 00:00 2024-03-18T22:28:13.328 INFO:teuthology.orchestra.run.smithi103.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-18T22:28:13.371 INFO:teuthology.orchestra.run.smithi103.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-03-18T22:28:13.388 INFO:teuthology.orchestra.run.smithi103.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 471 kB/s | 43 kB 00:00 2024-03-18T22:28:13.413 INFO:teuthology.orchestra.run.smithi103.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 383 kB/s | 32 kB 00:00 2024-03-18T22:28:13.430 INFO:teuthology.orchestra.run.smithi103.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 255 kB/s | 15 kB 00:00 2024-03-18T22:28:13.497 INFO:teuthology.orchestra.run.smithi103.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-03-18T22:28:13.522 INFO:teuthology.orchestra.run.smithi103.stdout:(121/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-18T22:28:13.556 INFO:teuthology.orchestra.run.smithi103.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-03-18T22:28:13.573 INFO:teuthology.orchestra.run.smithi103.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 468 kB/s | 35 kB 00:00 2024-03-18T22:28:13.609 INFO:teuthology.orchestra.run.smithi103.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-18T22:28:13.640 INFO:teuthology.orchestra.run.smithi103.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-18T22:28:13.682 INFO:teuthology.orchestra.run.smithi103.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-18T22:28:13.707 INFO:teuthology.orchestra.run.smithi103.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 264 kB/s | 18 kB 00:00 2024-03-18T22:28:13.765 INFO:teuthology.orchestra.run.smithi103.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 9.7 MB/s | 2.0 MB 00:00 2024-03-18T22:28:13.791 INFO:teuthology.orchestra.run.smithi103.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-03-18T22:28:13.816 INFO:teuthology.orchestra.run.smithi103.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-03-18T22:28:13.841 INFO:teuthology.orchestra.run.smithi103.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-03-18T22:28:13.908 INFO:teuthology.orchestra.run.smithi103.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 505 kB/s | 46 kB 00:00 2024-03-18T22:28:13.933 INFO:teuthology.orchestra.run.smithi103.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 857 kB/s | 79 kB 00:00 2024-03-18T22:28:13.967 INFO:teuthology.orchestra.run.smithi103.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-03-18T22:28:13.992 INFO:teuthology.orchestra.run.smithi103.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 689 kB/s | 58 kB 00:00 2024-03-18T22:28:14.017 INFO:teuthology.orchestra.run.smithi103.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-18T22:28:14.043 INFO:teuthology.orchestra.run.smithi103.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-03-18T22:28:14.099 INFO:teuthology.orchestra.run.smithi103.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 848 kB/s | 90 kB 00:00 2024-03-18T22:28:14.127 INFO:teuthology.orchestra.run.smithi103.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 259 kB/s | 28 kB 00:00 2024-03-18T22:28:14.152 INFO:teuthology.orchestra.run.smithi103.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-03-18T22:28:14.177 INFO:teuthology.orchestra.run.smithi103.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-18T22:28:14.219 INFO:teuthology.orchestra.run.smithi103.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 885 kB/s | 59 kB 00:00 2024-03-18T22:28:14.244 INFO:teuthology.orchestra.run.smithi103.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 389 kB/s | 36 kB 00:00 2024-03-18T22:28:14.269 INFO:teuthology.orchestra.run.smithi103.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 554 kB/s | 51 kB 00:00 2024-03-18T22:28:14.303 INFO:teuthology.orchestra.run.smithi103.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-18T22:28:14.320 INFO:teuthology.orchestra.run.smithi103.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-18T22:28:14.354 INFO:teuthology.orchestra.run.smithi103.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-18T22:28:14.396 INFO:teuthology.orchestra.run.smithi103.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 216 kB/s | 20 kB 00:00 2024-03-18T22:28:14.421 INFO:teuthology.orchestra.run.smithi103.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-18T22:28:14.463 INFO:teuthology.orchestra.run.smithi103.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-18T22:28:14.505 INFO:teuthology.orchestra.run.smithi103.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.5 MB/s | 321 kB 00:00 2024-03-18T22:28:14.713 INFO:teuthology.orchestra.run.smithi103.stdout:(152/174): librados2-19.0.0-2206.gb6ae2491.el9. 11 MB/s | 3.4 MB 00:00 2024-03-18T22:28:14.914 INFO:teuthology.orchestra.run.smithi103.stdout:(153/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 6.4 MB/s | 3.1 MB 00:00 2024-03-18T22:28:14.972 INFO:teuthology.orchestra.run.smithi103.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 727 kB/s | 42 kB 00:00 2024-03-18T22:28:15.098 INFO:teuthology.orchestra.run.smithi103.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.9 MB/s | 248 kB 00:00 2024-03-18T22:28:15.140 INFO:teuthology.orchestra.run.smithi103.stdout:(156/174): NetworkManager-1.47.2-1.el9.x86_64.r 3.6 MB/s | 2.3 MB 00:00 2024-03-18T22:28:15.165 INFO:teuthology.orchestra.run.smithi103.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-03-18T22:28:15.190 INFO:teuthology.orchestra.run.smithi103.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 726 kB/s | 36 kB 00:00 2024-03-18T22:28:15.249 INFO:teuthology.orchestra.run.smithi103.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.3 MB/s | 192 kB 00:00 2024-03-18T22:28:15.291 INFO:teuthology.orchestra.run.smithi103.stdout:(160/174): NetworkManager-libnm-1.47.2-1.el9.x8 3.2 MB/s | 1.8 MB 00:00 2024-03-18T22:28:15.316 INFO:teuthology.orchestra.run.smithi103.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 344 kB/s | 23 kB 00:00 2024-03-18T22:28:15.341 INFO:teuthology.orchestra.run.smithi103.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.7 MB/s | 256 kB 00:00 2024-03-18T22:28:15.362 INFO:teuthology.orchestra.run.smithi103.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 108 kB 00:00 2024-03-18T22:28:15.396 INFO:teuthology.orchestra.run.smithi103.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.4 MB/s | 136 kB 00:00 2024-03-18T22:28:15.421 INFO:teuthology.orchestra.run.smithi103.stdout:(165/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-18T22:28:15.450 INFO:teuthology.orchestra.run.smithi103.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 760 kB/s | 64 kB 00:00 2024-03-18T22:28:15.472 INFO:teuthology.orchestra.run.smithi103.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 380 kB/s | 29 kB 00:00 2024-03-18T22:28:15.739 INFO:teuthology.orchestra.run.smithi103.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 3.8 MB/s | 1.2 MB 00:00 2024-03-18T22:28:15.856 INFO:teuthology.orchestra.run.smithi103.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.9 MB/s | 465 kB 00:00 2024-03-18T22:28:15.947 INFO:teuthology.orchestra.run.smithi103.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.8 MB/s | 2.2 MB 00:00 2024-03-18T22:28:15.947 INFO:teuthology.orchestra.run.smithi103.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 464 kB/s | 31 kB 00:00 2024-03-18T22:28:15.973 INFO:teuthology.orchestra.run.smithi103.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.6 MB/s | 2.3 MB 00:00 2024-03-18T22:28:16.015 INFO:teuthology.orchestra.run.smithi103.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 809 kB/s | 74 kB 00:00 2024-03-18T22:28:16.157 INFO:teuthology.orchestra.run.smithi103.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 4.1 MB/s | 1.1 MB 00:00 2024-03-18T22:28:16.164 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-03-18T22:28:16.165 INFO:teuthology.orchestra.run.smithi103.stdout:Total 18 MB/s | 224 MB 00:12 2024-03-18T22:28:16.629 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-03-18T22:28:16.906 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-03-18T22:28:16.906 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-03-18T22:28:18.928 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-03-18T22:28:18.929 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-03-18T22:28:25.896 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-03-18T22:28:25.996 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T22:28:26.022 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T22:28:26.060 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T22:28:26.069 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T22:28:26.105 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T22:28:26.167 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T22:28:26.204 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T22:28:26.239 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T22:28:26.290 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T22:28:26.617 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T22:28:26.673 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-18T22:28:26.778 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-18T22:28:26.808 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-18T22:28:26.898 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-18T22:28:26.935 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-18T22:28:26.982 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T22:28:27.162 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T22:28:27.252 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T22:28:27.280 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-18T22:28:27.366 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-18T22:28:27.397 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-18T22:28:27.464 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-18T22:28:27.502 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T22:28:27.524 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T22:28:27.540 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T22:28:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T22:28:27.765 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T22:28:27.813 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T22:28:27.992 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T22:28:28.157 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-18T22:28:28.217 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-18T22:28:28.312 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-18T22:28:28.360 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-18T22:28:28.591 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-18T22:28:28.711 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T22:28:28.803 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T22:28:28.841 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T22:28:28.867 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T22:28:28.903 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T22:28:28.977 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T22:28:29.031 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T22:28:29.069 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T22:28:29.125 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T22:28:29.169 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T22:28:29.213 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T22:28:29.245 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T22:28:29.273 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T22:28:29.342 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T22:28:29.378 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-18T22:28:29.421 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-18T22:28:29.450 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-18T22:28:29.512 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T22:28:29.565 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T22:28:29.639 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T22:28:29.773 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T22:28:29.826 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T22:28:30.335 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T22:28:30.421 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T22:28:30.509 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T22:28:30.550 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T22:28:30.559 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T22:28:30.593 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T22:28:30.648 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T22:28:30.686 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-18T22:28:30.717 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T22:28:31.013 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T22:28:31.220 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T22:28:31.310 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T22:28:31.354 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T22:28:31.774 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T22:28:31.833 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T22:28:31.891 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T22:28:31.971 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T22:28:32.026 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T22:28:32.062 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T22:28:32.109 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T22:28:32.145 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T22:28:32.188 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T22:28:32.230 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T22:28:32.363 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T22:28:32.411 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T22:28:32.441 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T22:28:32.483 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T22:28:32.604 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T22:28:32.650 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T22:28:32.691 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T22:28:32.716 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T22:28:32.741 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T22:28:32.817 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T22:28:32.942 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T22:28:33.048 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T22:28:33.106 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T22:28:33.155 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T22:28:33.211 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T22:28:33.395 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T22:28:33.444 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T22:28:33.579 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T22:28:33.618 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T22:28:33.650 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T22:28:33.680 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T22:28:33.708 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T22:28:33.743 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T22:28:33.774 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T22:28:33.805 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T22:28:33.835 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T22:28:33.913 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T22:28:33.946 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T22:28:33.979 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T22:28:34.012 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T22:28:34.047 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T22:28:34.136 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T22:28:34.219 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T22:28:34.261 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T22:28:34.293 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T22:28:34.347 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T22:28:34.447 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T22:28:34.503 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T22:28:35.226 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T22:28:35.276 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T22:28:35.309 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T22:28:35.357 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T22:28:35.438 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T22:28:35.477 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T22:28:35.647 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T22:28:35.896 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T22:28:36.134 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T22:28:36.769 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T22:28:36.872 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T22:28:38.096 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T22:28:38.186 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T22:28:38.217 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T22:28:38.248 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T22:28:38.276 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T22:28:38.588 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T22:28:38.614 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-18T22:28:38.705 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-18T22:28:38.736 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-18T22:28:38.774 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T22:28:39.226 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T22:28:39.251 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-18T22:28:39.311 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-18T22:28:39.337 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-18T22:28:41.575 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-18T22:28:41.624 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-18T22:28:42.169 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-18T22:28:42.211 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-18T22:28:42.288 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-18T22:28:42.289 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T22:28:42.289 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T22:28:42.289 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:28:42.330 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-18T22:28:54.598 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-18T22:28:54.598 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-03-18T22:28:54.598 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-03-18T22:28:54.598 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-03-18T22:28:54.598 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-03-18T22:28:54.598 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-03-18T22:28:54.598 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-03-18T22:28:54.599 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-03-18T22:28:54.599 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:28:54.782 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-18T22:28:54.797 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-18T22:28:54.798 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:28:54.798 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T22:28:54.798 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:28:56.253 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-18T22:28:56.278 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-18T22:28:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:28:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T22:28:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T22:28:56.278 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T22:28:56.278 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:28:56.503 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-18T22:28:56.526 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-18T22:28:56.526 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:28:56.526 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T22:28:56.526 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T22:28:56.526 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T22:28:56.526 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:28:56.971 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-18T22:28:56.993 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-18T22:28:56.993 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:28:56.993 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T22:28:56.993 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T22:28:56.993 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T22:28:56.993 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:28:57.097 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T22:28:57.499 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-18T22:28:57.530 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-18T22:28:57.561 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-18T22:28:57.621 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-18T22:28:57.644 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-18T22:28:57.876 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-18T22:28:57.912 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-18T22:28:58.874 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-18T22:28:58.896 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-18T22:28:58.996 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-18T22:28:59.159 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-18T22:28:59.190 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-18T22:28:59.213 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-18T22:28:59.213 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:28:59.213 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T22:28:59.213 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T22:28:59.213 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T22:28:59.213 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:28:59.258 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-18T22:28:59.268 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-18T22:28:59.323 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-18T22:29:00.191 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T22:29:00.226 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-18T22:29:00.249 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-18T22:29:00.249 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:29:00.249 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T22:29:00.249 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T22:29:00.249 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T22:29:00.249 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:29:00.294 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-18T22:29:00.316 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-18T22:29:00.316 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:29:00.316 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T22:29:00.316 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:29:00.369 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-18T22:29:00.392 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-18T22:29:00.392 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:29:00.392 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T22:29:00.392 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T22:29:00.392 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T22:29:00.392 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:29:00.698 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-18T22:29:00.718 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-18T22:29:00.718 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:29:00.718 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T22:29:00.718 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T22:29:00.718 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T22:29:00.718 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:29:05.200 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-18T22:29:05.241 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-18T22:29:05.275 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-18T22:29:05.307 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T22:29:05.490 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-18T22:29:05.621 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T22:29:05.665 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-18T22:29:05.719 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T22:29:05.765 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T22:29:05.808 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T22:29:05.849 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-18T22:29:05.890 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T22:29:05.920 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-18T22:29:05.978 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-18T22:29:06.037 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T22:29:06.068 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T22:29:06.617 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T22:29:06.650 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T22:29:06.670 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T22:29:07.488 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-18T22:29:07.580 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T22:29:07.621 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T22:29:07.707 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T22:29:07.743 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T22:29:07.767 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T22:29:07.793 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T22:29:07.819 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T22:29:07.819 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T22:29:07.890 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T22:29:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T22:29:07.929 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T22:29:08.013 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T22:29:08.040 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T22:29:08.073 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T22:29:08.099 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T22:29:08.123 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T22:29:08.157 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T22:29:08.205 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T22:29:08.237 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T22:29:08.262 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T22:29:08.287 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T22:29:08.313 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T22:29:08.338 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T22:29:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T22:29:08.345 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T22:29:08.525 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T22:29:08.525 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T22:29:08.607 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T22:29:08.632 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T22:29:08.658 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T22:29:08.691 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T22:29:08.717 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T22:29:08.783 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T22:29:13.774 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T22:29:13.774 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-18T22:29:13.774 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-18T22:29:13.774 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-18T22:29:13.774 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-18T22:29:13.774 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-18T22:29:13.774 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-18T22:29:13.774 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-18T22:29:13.775 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-18T22:29:13.775 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-18T22:29:13.775 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-18T22:29:13.775 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-18T22:29:13.775 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-18T22:29:13.775 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-18T22:29:13.775 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-18T22:29:13.775 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-18T22:29:13.775 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-18T22:29:13.775 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-18T22:29:13.775 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-18T22:29:13.775 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-18T22:29:13.775 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-18T22:29:13.776 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-18T22:29:13.776 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-18T22:29:13.776 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-18T22:29:13.776 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-18T22:29:13.776 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-18T22:29:13.776 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-18T22:29:13.776 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-18T22:29:13.776 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-18T22:29:13.776 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-18T22:29:13.776 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-18T22:29:13.776 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-18T22:29:13.776 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-18T22:29:13.776 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-18T22:29:13.777 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-18T22:29:13.777 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-18T22:29:13.777 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-18T22:29:13.777 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-18T22:29:13.777 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-18T22:29:13.777 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-18T22:29:13.777 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T22:29:13.777 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T22:29:13.777 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T22:29:13.777 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T22:29:13.777 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T22:29:13.777 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T22:29:13.777 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T22:29:13.777 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T22:29:13.778 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T22:29:13.778 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T22:29:13.778 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T22:29:13.778 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T22:29:13.778 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T22:29:13.778 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T22:29:13.778 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T22:29:13.778 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T22:29:13.778 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T22:29:13.778 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T22:29:13.778 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T22:29:13.778 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T22:29:13.779 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T22:29:13.779 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T22:29:13.779 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T22:29:13.779 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T22:29:13.779 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T22:29:13.779 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T22:29:13.779 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T22:29:13.779 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T22:29:13.779 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T22:29:13.779 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T22:29:13.779 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T22:29:13.779 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T22:29:13.779 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T22:29:13.779 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T22:29:13.780 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T22:29:13.780 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T22:29:13.780 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T22:29:13.780 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T22:29:13.780 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T22:29:13.780 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T22:29:13.780 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T22:29:13.780 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T22:29:13.780 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T22:29:13.780 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T22:29:13.780 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T22:29:13.780 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T22:29:13.780 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T22:29:13.780 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T22:29:13.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T22:29:13.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T22:29:13.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T22:29:13.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T22:29:13.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T22:29:13.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T22:29:13.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T22:29:13.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T22:29:13.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T22:29:13.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T22:29:13.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T22:29:13.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T22:29:13.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T22:29:13.781 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T22:29:13.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T22:29:13.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T22:29:13.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T22:29:13.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T22:29:13.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T22:29:13.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T22:29:13.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T22:29:13.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T22:29:13.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T22:29:13.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T22:29:13.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T22:29:13.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T22:29:13.782 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T22:29:13.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T22:29:13.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T22:29:13.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T22:29:13.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T22:29:13.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T22:29:13.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T22:29:13.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T22:29:13.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T22:29:13.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T22:29:13.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T22:29:13.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T22:29:13.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T22:29:13.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T22:29:13.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T22:29:13.783 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T22:29:13.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T22:29:13.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T22:29:13.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T22:29:13.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T22:29:13.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T22:29:13.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T22:29:13.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T22:29:13.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T22:29:13.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T22:29:13.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T22:29:13.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T22:29:13.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T22:29:13.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T22:29:13.784 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T22:29:13.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T22:29:13.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T22:29:13.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T22:29:13.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T22:29:13.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T22:29:13.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T22:29:13.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T22:29:13.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-18T22:29:13.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T22:29:13.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-18T22:29:13.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T22:29:13.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T22:29:13.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T22:29:13.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-18T22:29:13.785 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T22:29:13.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-18T22:29:13.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T22:29:13.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-18T22:29:13.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T22:29:13.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-18T22:29:13.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T22:29:13.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T22:29:13.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T22:29:13.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T22:29:13.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T22:29:13.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T22:29:13.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T22:29:13.786 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T22:29:13.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T22:29:13.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T22:29:13.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T22:29:13.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T22:29:13.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T22:29:13.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T22:29:13.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T22:29:13.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T22:29:13.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T22:29:13.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T22:29:13.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T22:29:13.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T22:29:13.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T22:29:13.787 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T22:29:13.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T22:29:13.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T22:29:13.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T22:29:13.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T22:29:13.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T22:29:13.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T22:29:13.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T22:29:13.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T22:29:13.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T22:29:13.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T22:29:13.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T22:29:13.788 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T22:29:16.824 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T22:29:16.824 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:29:16.824 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-03-18T22:29:16.824 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-18T22:29:16.824 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-18T22:29:16.824 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-18T22:29:16.825 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-18T22:29:16.825 INFO:teuthology.orchestra.run.smithi103.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T22:29:16.825 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T22:29:16.825 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T22:29:16.825 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T22:29:16.825 INFO:teuthology.orchestra.run.smithi103.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T22:29:16.825 INFO:teuthology.orchestra.run.smithi103.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T22:29:16.825 INFO:teuthology.orchestra.run.smithi103.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T22:29:16.825 INFO:teuthology.orchestra.run.smithi103.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T22:29:16.825 INFO:teuthology.orchestra.run.smithi103.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T22:29:16.825 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.825 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.825 INFO:teuthology.orchestra.run.smithi103.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T22:29:16.826 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T22:29:16.826 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T22:29:16.826 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T22:29:16.826 INFO:teuthology.orchestra.run.smithi103.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T22:29:16.826 INFO:teuthology.orchestra.run.smithi103.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T22:29:16.826 INFO:teuthology.orchestra.run.smithi103.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T22:29:16.826 INFO:teuthology.orchestra.run.smithi103.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T22:29:16.826 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T22:29:16.826 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-03-18T22:29:16.826 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T22:29:16.826 INFO:teuthology.orchestra.run.smithi103.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T22:29:16.826 INFO:teuthology.orchestra.run.smithi103.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T22:29:16.826 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T22:29:16.827 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.827 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.827 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.827 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.827 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:29:16.827 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.827 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.827 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.827 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:29:16.827 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:29:16.827 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:29:16.827 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:29:16.827 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:29:16.828 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.828 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.828 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:29:16.828 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.828 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.828 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.828 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:29:16.828 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:29:16.828 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.828 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.828 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:29:16.828 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T22:29:16.828 INFO:teuthology.orchestra.run.smithi103.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T22:29:16.828 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T22:29:16.829 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T22:29:16.829 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T22:29:16.829 INFO:teuthology.orchestra.run.smithi103.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T22:29:16.829 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T22:29:16.829 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T22:29:16.829 INFO:teuthology.orchestra.run.smithi103.stdout: jq-1.6-16.el9.x86_64 2024-03-18T22:29:16.829 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T22:29:16.829 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T22:29:16.829 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T22:29:16.829 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.829 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.829 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.829 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T22:29:16.830 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T22:29:16.830 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T22:29:16.830 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T22:29:16.830 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T22:29:16.830 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.830 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.830 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T22:29:16.830 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.830 INFO:teuthology.orchestra.run.smithi103.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T22:29:16.830 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T22:29:16.830 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T22:29:16.830 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T22:29:16.830 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T22:29:16.831 INFO:teuthology.orchestra.run.smithi103.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T22:29:16.831 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T22:29:16.831 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T22:29:16.831 INFO:teuthology.orchestra.run.smithi103.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T22:29:16.831 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T22:29:16.831 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T22:29:16.831 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T22:29:16.831 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T22:29:16.831 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T22:29:16.831 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T22:29:16.831 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T22:29:16.831 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T22:29:16.831 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T22:29:16.831 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T22:29:16.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T22:29:16.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T22:29:16.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T22:29:16.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T22:29:16.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T22:29:16.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T22:29:16.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T22:29:16.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T22:29:16.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T22:29:16.832 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T22:29:16.833 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T22:29:16.833 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T22:29:16.833 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T22:29:16.833 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T22:29:16.833 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T22:29:16.833 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T22:29:16.833 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T22:29:16.833 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T22:29:16.833 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T22:29:16.833 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T22:29:16.833 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T22:29:16.833 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T22:29:16.833 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T22:29:16.833 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T22:29:16.834 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T22:29:16.834 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T22:29:16.834 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T22:29:16.834 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T22:29:16.834 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T22:29:16.834 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T22:29:16.834 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T22:29:16.834 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T22:29:16.834 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T22:29:16.834 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T22:29:16.834 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T22:29:16.834 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T22:29:16.834 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T22:29:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T22:29:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T22:29:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T22:29:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T22:29:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T22:29:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T22:29:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T22:29:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T22:29:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T22:29:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T22:29:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T22:29:16.835 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.836 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T22:29:16.836 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T22:29:16.836 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T22:29:16.836 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T22:29:16.836 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T22:29:16.836 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T22:29:16.836 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T22:29:16.836 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T22:29:16.836 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T22:29:16.836 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T22:29:16.836 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T22:29:16.836 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T22:29:16.836 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T22:29:16.837 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T22:29:16.837 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T22:29:16.837 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.837 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.837 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:29:16.837 INFO:teuthology.orchestra.run.smithi103.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T22:29:16.837 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T22:29:16.837 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T22:29:16.837 INFO:teuthology.orchestra.run.smithi103.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T22:29:16.837 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T22:29:16.837 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T22:29:16.838 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T22:29:16.838 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-03-18T22:29:16.838 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-03-18T22:29:17.327 DEBUG:teuthology.parallel:result is None 2024-03-18T22:30:18.091 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] abseil-cpp-20211102.0-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/a/abseil-cpp-20211102.0-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-18T22:30:18.091 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] gperftools-libs-2.9.1-2.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/g/gperftools-libs-2.9.1-2.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-18T22:30:18.091 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] grpc-data-1.46.7-10.el9.noarch.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/fedora-epel/9/Everything/x86_64/Packages/g/grpc-data-1.46.7-10.el9.noarch.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-18T22:30:18.274 INFO:teuthology.orchestra.run.smithi040.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 153 B/s | 19 kB 02:10 2024-03-18T22:30:18.457 INFO:teuthology.orchestra.run.smithi040.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 2.4 kB/s | 309 kB 02:10 2024-03-18T22:30:18.499 INFO:teuthology.orchestra.run.smithi040.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 4.2 kB/s | 548 kB 02:10 2024-03-18T22:30:18.533 INFO:teuthology.orchestra.run.smithi040.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-03-18T22:30:18.575 INFO:teuthology.orchestra.run.smithi040.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 645 kB/s | 49 kB 00:00 2024-03-18T22:30:18.608 INFO:teuthology.orchestra.run.smithi040.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 896 kB/s | 67 kB 00:00 2024-03-18T22:30:18.650 INFO:teuthology.orchestra.run.smithi040.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-18T22:30:18.784 INFO:teuthology.orchestra.run.smithi040.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 8.7 MB/s | 4.4 MB 00:00 2024-03-18T22:30:18.809 INFO:teuthology.orchestra.run.smithi040.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-03-18T22:30:18.834 INFO:teuthology.orchestra.run.smithi040.stdout:(116/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 838 kB 00:00 2024-03-18T22:30:18.860 INFO:teuthology.orchestra.run.smithi040.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 577 kB/s | 43 kB 00:00 2024-03-18T22:30:18.885 INFO:teuthology.orchestra.run.smithi040.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 426 kB/s | 32 kB 00:00 2024-03-18T22:30:18.910 INFO:teuthology.orchestra.run.smithi040.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-03-18T22:30:18.935 INFO:teuthology.orchestra.run.smithi040.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-18T22:30:18.977 INFO:teuthology.orchestra.run.smithi040.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-03-18T22:30:19.003 INFO:teuthology.orchestra.run.smithi040.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-18T22:30:19.028 INFO:teuthology.orchestra.run.smithi040.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 382 kB/s | 35 kB 00:00 2024-03-18T22:30:19.062 INFO:teuthology.orchestra.run.smithi040.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-18T22:30:19.103 INFO:teuthology.orchestra.run.smithi040.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-03-18T22:30:19.129 INFO:teuthology.orchestra.run.smithi040.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-18T22:30:19.179 INFO:teuthology.orchestra.run.smithi040.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-18T22:30:19.221 INFO:teuthology.orchestra.run.smithi040.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 151 kB/s | 18 kB 00:00 2024-03-18T22:30:19.246 INFO:teuthology.orchestra.run.smithi040.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 198 kB/s | 23 kB 00:00 2024-03-18T22:30:19.305 INFO:teuthology.orchestra.run.smithi040.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 155 kB/s | 19 kB 00:00 2024-03-18T22:30:19.330 INFO:teuthology.orchestra.run.smithi040.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-03-18T22:30:19.364 INFO:teuthology.orchestra.run.smithi040.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-03-18T22:30:19.389 INFO:teuthology.orchestra.run.smithi040.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 553 kB/s | 46 kB 00:00 2024-03-18T22:30:19.414 INFO:teuthology.orchestra.run.smithi040.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 942 kB/s | 79 kB 00:00 2024-03-18T22:30:19.440 INFO:teuthology.orchestra.run.smithi040.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 766 kB/s | 58 kB 00:00 2024-03-18T22:30:19.465 INFO:teuthology.orchestra.run.smithi040.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-18T22:30:19.490 INFO:teuthology.orchestra.run.smithi040.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-03-18T22:30:19.516 INFO:teuthology.orchestra.run.smithi040.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-18T22:30:19.541 INFO:teuthology.orchestra.run.smithi040.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 373 kB/s | 28 kB 00:00 2024-03-18T22:30:19.566 INFO:teuthology.orchestra.run.smithi040.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 406 kB/s | 31 kB 00:00 2024-03-18T22:30:19.591 INFO:teuthology.orchestra.run.smithi040.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-18T22:30:19.617 INFO:teuthology.orchestra.run.smithi040.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 782 kB/s | 59 kB 00:00 2024-03-18T22:30:19.642 INFO:teuthology.orchestra.run.smithi040.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-03-18T22:30:19.659 INFO:teuthology.orchestra.run.smithi040.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 757 kB/s | 51 kB 00:00 2024-03-18T22:30:19.693 INFO:teuthology.orchestra.run.smithi040.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-18T22:30:19.726 INFO:teuthology.orchestra.run.smithi040.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-03-18T22:30:19.760 INFO:teuthology.orchestra.run.smithi040.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-03-18T22:30:19.785 INFO:teuthology.orchestra.run.smithi040.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-03-18T22:30:19.810 INFO:teuthology.orchestra.run.smithi040.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-18T22:30:19.886 INFO:teuthology.orchestra.run.smithi040.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-18T22:30:20.044 INFO:teuthology.orchestra.run.smithi040.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.4 MB/s | 321 kB 00:00 2024-03-18T22:30:20.219 INFO:teuthology.orchestra.run.smithi040.stdout:(152/174): librados2-19.0.0-2206.gb6ae2491.el9. 7.8 MB/s | 3.4 MB 00:00 2024-03-18T22:30:20.461 INFO:teuthology.orchestra.run.smithi040.stdout:(153/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 5.0 MB/s | 3.1 MB 00:00 2024-03-18T22:30:20.837 INFO:teuthology.orchestra.run.smithi040.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 112 kB/s | 42 kB 00:00 2024-03-18T22:30:20.928 INFO:teuthology.orchestra.run.smithi040.stdout:(155/174): NetworkManager-1.47.2-1.el9.x86_64.r 2.6 MB/s | 2.3 MB 00:00 2024-03-18T22:30:21.037 INFO:teuthology.orchestra.run.smithi040.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 660 kB/s | 71 kB 00:00 2024-03-18T22:30:21.146 INFO:teuthology.orchestra.run.smithi040.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 336 kB/s | 36 kB 00:00 2024-03-18T22:30:21.179 INFO:teuthology.orchestra.run.smithi040.stdout:(158/174): NetworkManager-tui-1.47.2-1.el9.x86_ 725 kB/s | 248 kB 00:00 2024-03-18T22:30:21.221 INFO:teuthology.orchestra.run.smithi040.stdout:(159/174): NetworkManager-libnm-1.47.2-1.el9.x8 1.8 MB/s | 1.8 MB 00:01 2024-03-18T22:30:21.247 INFO:teuthology.orchestra.run.smithi040.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.9 MB/s | 192 kB 00:00 2024-03-18T22:30:21.289 INFO:teuthology.orchestra.run.smithi040.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-18T22:30:21.322 INFO:teuthology.orchestra.run.smithi040.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 228 kB/s | 23 kB 00:00 2024-03-18T22:30:21.350 INFO:teuthology.orchestra.run.smithi040.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-03-18T22:30:21.398 INFO:teuthology.orchestra.run.smithi040.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-18T22:30:21.431 INFO:teuthology.orchestra.run.smithi040.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-18T22:30:21.457 INFO:teuthology.orchestra.run.smithi040.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 586 kB/s | 64 kB 00:00 2024-03-18T22:30:21.507 INFO:teuthology.orchestra.run.smithi040.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 263 kB/s | 29 kB 00:00 2024-03-18T22:30:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 4.5 MB/s | 1.2 MB 00:00 2024-03-18T22:30:21.824 INFO:teuthology.orchestra.run.smithi040.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.6 MB/s | 465 kB 00:00 2024-03-18T22:30:21.933 INFO:teuthology.orchestra.run.smithi040.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 287 kB/s | 31 kB 00:00 2024-03-18T22:30:21.974 INFO:teuthology.orchestra.run.smithi040.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.9 MB/s | 2.3 MB 00:00 2024-03-18T22:30:22.050 INFO:teuthology.orchestra.run.smithi040.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.2 MB/s | 1.1 MB 00:00 2024-03-18T22:30:22.083 INFO:teuthology.orchestra.run.smithi040.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 685 kB/s | 74 kB 00:00 2024-03-18T22:30:22.175 INFO:teuthology.orchestra.run.smithi040.stdout:(174/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 3.0 MB/s | 2.2 MB 00:00 2024-03-18T22:30:22.180 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-03-18T22:30:22.180 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.6 MB/s | 224 MB 02:24 2024-03-18T22:30:22.629 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-03-18T22:30:22.924 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-03-18T22:30:22.925 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-03-18T22:30:25.045 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-03-18T22:30:25.045 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-03-18T22:30:32.222 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-03-18T22:30:32.322 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T22:30:32.356 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T22:30:32.402 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T22:30:32.411 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T22:30:32.453 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T22:30:32.510 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T22:30:32.550 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T22:30:32.599 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T22:30:32.677 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T22:30:33.033 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T22:30:33.084 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-18T22:30:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-18T22:30:33.219 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-18T22:30:33.295 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-18T22:30:33.332 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-18T22:30:33.370 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T22:30:33.558 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T22:30:33.614 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T22:30:33.641 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-18T22:30:33.729 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-18T22:30:33.758 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-18T22:30:33.807 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-18T22:30:33.854 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T22:30:33.877 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T22:30:33.892 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T22:30:34.074 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T22:30:34.109 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T22:30:34.148 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T22:30:34.326 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T22:30:34.458 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-18T22:30:34.510 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-18T22:30:34.604 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-18T22:30:34.653 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-18T22:30:34.865 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-18T22:30:34.970 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T22:30:35.064 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T22:30:35.100 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T22:30:35.127 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T22:30:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T22:30:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T22:30:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T22:30:35.337 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T22:30:35.402 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T22:30:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T22:30:35.480 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T22:30:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T22:30:35.565 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T22:30:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T22:30:35.670 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-18T22:30:35.714 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-18T22:30:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-18T22:30:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T22:30:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T22:30:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T22:30:36.128 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T22:30:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T22:30:36.725 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T22:30:36.798 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T22:30:36.896 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T22:30:36.941 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T22:30:36.949 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T22:30:37.000 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T22:30:37.048 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T22:30:37.085 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-18T22:30:37.115 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T22:30:37.420 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T22:30:37.584 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T22:30:37.676 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T22:30:37.720 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T22:30:37.873 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T22:30:37.916 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T22:30:37.973 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T22:30:38.046 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T22:30:38.083 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T22:30:38.120 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T22:30:38.167 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T22:30:38.203 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T22:30:38.246 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T22:30:38.288 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T22:30:38.430 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T22:30:38.477 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T22:30:38.507 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T22:30:38.548 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T22:30:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T22:30:38.690 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T22:30:38.730 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T22:30:38.755 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T22:30:38.780 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T22:30:38.859 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T22:30:38.993 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T22:30:39.105 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T22:30:39.148 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T22:30:39.205 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T22:30:39.285 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T22:30:39.483 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T22:30:39.527 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T22:30:39.668 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T22:30:39.717 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T22:30:39.748 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T22:30:39.779 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T22:30:39.807 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T22:30:39.842 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T22:30:39.881 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T22:30:39.921 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T22:30:39.951 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T22:30:40.030 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T22:30:40.061 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T22:30:40.094 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T22:30:40.136 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T22:30:40.170 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T22:30:40.264 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T22:30:40.353 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T22:30:40.401 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T22:30:40.433 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T22:30:40.481 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T22:30:40.583 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T22:30:40.644 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T22:30:41.404 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T22:30:41.464 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T22:30:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T22:30:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T22:30:41.636 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T22:30:41.674 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T22:30:41.858 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T22:30:41.897 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T22:30:41.977 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T22:30:42.647 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T22:30:42.756 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T22:30:44.058 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T22:30:44.122 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T22:30:44.154 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T22:30:44.184 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T22:30:44.212 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T22:30:44.541 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T22:30:44.583 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-18T22:30:44.674 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-18T22:30:44.697 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-18T22:30:44.742 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T22:30:45.228 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T22:30:45.253 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-18T22:30:45.308 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-18T22:30:45.330 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-18T22:30:47.613 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-18T22:30:47.652 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-18T22:30:48.220 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-18T22:30:48.263 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-18T22:30:48.356 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-18T22:30:48.356 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T22:30:48.356 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T22:30:48.356 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:30:48.398 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-18T22:31:00.374 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-18T22:31:00.374 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2024-03-18T22:31:00.375 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2024-03-18T22:31:00.375 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2024-03-18T22:31:00.375 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2024-03-18T22:31:00.375 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2024-03-18T22:31:00.375 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /root 2024-03-18T22:31:00.375 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2024-03-18T22:31:00.375 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:31:00.553 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-18T22:31:00.568 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-18T22:31:00.568 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:31:00.568 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T22:31:00.568 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:31:02.098 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-18T22:31:02.123 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-18T22:31:02.123 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:31:02.123 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T22:31:02.123 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T22:31:02.124 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T22:31:02.124 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:31:02.357 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-18T22:31:02.379 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-18T22:31:02.379 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:31:02.379 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T22:31:02.379 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T22:31:02.379 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T22:31:02.379 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:31:02.849 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-18T22:31:02.871 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-18T22:31:02.871 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:31:02.871 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T22:31:02.872 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T22:31:02.872 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T22:31:02.872 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:31:02.985 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T22:31:03.428 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-18T22:31:03.450 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-18T22:31:03.482 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-18T22:31:03.543 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-18T22:31:03.572 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-18T22:31:03.817 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-18T22:31:03.849 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-18T22:31:04.862 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-18T22:31:04.891 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-18T22:31:04.995 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-18T22:31:05.169 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-18T22:31:05.201 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-18T22:31:05.223 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-18T22:31:05.223 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:31:05.223 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T22:31:05.223 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T22:31:05.223 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T22:31:05.223 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:31:05.268 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-18T22:31:05.279 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-18T22:31:05.330 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-18T22:31:06.254 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T22:31:06.287 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-18T22:31:06.308 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-18T22:31:06.309 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:31:06.309 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T22:31:06.309 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T22:31:06.309 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T22:31:06.309 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:31:06.354 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-18T22:31:06.375 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-18T22:31:06.376 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:31:06.376 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T22:31:06.376 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:31:06.430 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-18T22:31:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-18T22:31:06.452 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:31:06.452 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T22:31:06.452 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T22:31:06.452 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T22:31:06.452 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:31:06.764 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-18T22:31:06.786 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-18T22:31:06.786 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T22:31:06.786 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T22:31:06.786 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T22:31:06.786 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T22:31:06.787 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:31:11.494 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-18T22:31:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-18T22:31:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-18T22:31:11.593 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T22:31:11.794 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-18T22:31:11.917 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T22:31:11.961 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-18T22:31:12.010 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T22:31:12.068 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T22:31:12.103 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T22:31:12.136 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-18T22:31:12.168 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T22:31:12.198 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-18T22:31:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-18T22:31:12.328 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T22:31:12.362 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T22:31:12.967 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T22:31:13.002 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T22:31:13.021 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T22:31:13.887 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-18T22:31:13.956 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T22:31:13.998 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T22:31:14.081 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T22:31:14.118 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T22:31:14.143 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T22:31:14.168 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T22:31:14.195 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T22:31:14.195 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T22:31:14.282 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T22:31:14.312 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T22:31:14.312 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T22:31:14.380 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T22:31:14.406 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T22:31:14.440 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T22:31:14.465 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T22:31:14.481 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T22:31:14.516 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T22:31:14.554 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T22:31:14.586 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T22:31:14.611 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T22:31:14.637 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T22:31:14.670 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T22:31:14.695 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T22:31:14.702 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T22:31:14.702 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T22:31:14.882 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T22:31:14.882 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T22:31:14.956 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T22:31:14.981 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T22:31:15.023 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T22:31:15.039 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T22:31:15.066 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T22:31:15.134 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T22:31:19.849 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T22:31:19.849 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-18T22:31:19.849 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-18T22:31:19.849 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-18T22:31:19.849 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-18T22:31:19.849 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-18T22:31:19.849 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-18T22:31:19.849 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-18T22:31:19.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-18T22:31:19.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-18T22:31:19.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-18T22:31:19.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-18T22:31:19.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-18T22:31:19.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-18T22:31:19.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-18T22:31:19.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-18T22:31:19.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-18T22:31:19.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-18T22:31:19.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-18T22:31:19.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-18T22:31:19.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-18T22:31:19.850 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-18T22:31:19.851 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-18T22:31:19.851 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-18T22:31:19.851 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-18T22:31:19.851 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-18T22:31:19.851 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-18T22:31:19.851 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-18T22:31:19.851 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-18T22:31:19.851 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-18T22:31:19.851 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-18T22:31:19.851 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-18T22:31:19.866 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-18T22:31:19.866 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-18T22:31:19.866 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-18T22:31:19.867 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-18T22:31:19.867 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-18T22:31:19.867 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-18T22:31:19.867 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-18T22:31:19.867 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-18T22:31:19.867 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T22:31:19.867 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T22:31:19.867 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T22:31:19.867 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T22:31:19.867 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T22:31:19.867 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T22:31:19.867 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T22:31:19.868 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T22:31:19.868 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T22:31:19.868 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T22:31:19.868 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T22:31:19.868 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T22:31:19.868 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T22:31:19.868 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T22:31:19.868 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T22:31:19.868 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T22:31:19.868 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T22:31:19.868 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T22:31:19.868 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T22:31:19.868 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T22:31:19.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T22:31:19.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T22:31:19.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T22:31:19.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T22:31:19.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T22:31:19.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T22:31:19.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T22:31:19.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T22:31:19.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T22:31:19.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T22:31:19.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T22:31:19.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T22:31:19.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T22:31:19.869 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T22:31:19.870 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T22:31:19.870 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T22:31:19.870 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T22:31:19.870 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T22:31:19.870 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T22:31:19.870 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T22:31:19.870 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T22:31:19.870 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T22:31:19.870 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T22:31:19.870 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T22:31:19.870 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T22:31:19.870 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T22:31:19.870 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T22:31:19.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T22:31:19.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T22:31:19.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T22:31:19.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T22:31:19.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T22:31:19.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T22:31:19.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T22:31:19.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T22:31:19.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T22:31:19.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T22:31:19.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T22:31:19.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T22:31:19.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T22:31:19.871 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T22:31:19.872 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T22:31:19.872 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T22:31:19.872 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T22:31:19.872 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T22:31:19.872 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T22:31:19.872 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T22:31:19.872 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T22:31:19.872 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T22:31:19.872 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T22:31:19.872 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T22:31:19.872 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T22:31:19.872 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T22:31:19.872 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T22:31:19.872 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T22:31:19.872 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T22:31:19.873 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T22:31:19.873 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T22:31:19.873 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T22:31:19.873 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T22:31:19.873 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T22:31:19.873 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T22:31:19.873 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T22:31:19.873 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T22:31:19.873 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T22:31:19.873 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T22:31:19.873 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T22:31:19.873 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T22:31:19.873 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T22:31:19.873 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T22:31:19.873 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T22:31:19.874 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T22:31:19.874 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T22:31:19.874 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T22:31:19.874 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T22:31:19.874 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T22:31:19.874 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T22:31:19.874 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T22:31:19.874 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T22:31:19.874 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T22:31:19.874 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T22:31:19.874 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T22:31:19.874 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T22:31:19.874 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T22:31:19.875 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T22:31:19.875 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T22:31:19.875 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T22:31:19.875 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T22:31:19.875 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T22:31:19.875 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T22:31:19.875 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T22:31:19.875 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-18T22:31:19.875 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T22:31:19.875 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-18T22:31:19.875 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T22:31:19.875 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T22:31:19.875 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T22:31:19.875 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-18T22:31:19.876 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T22:31:19.876 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-18T22:31:19.876 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T22:31:19.876 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-18T22:31:19.876 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T22:31:19.876 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-18T22:31:19.876 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T22:31:19.876 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T22:31:19.876 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T22:31:19.876 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T22:31:19.876 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T22:31:19.876 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T22:31:19.876 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T22:31:19.876 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T22:31:19.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T22:31:19.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T22:31:19.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T22:31:19.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T22:31:19.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T22:31:19.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T22:31:19.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T22:31:19.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T22:31:19.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T22:31:19.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T22:31:19.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T22:31:19.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T22:31:19.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T22:31:19.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T22:31:19.877 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T22:31:19.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T22:31:19.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T22:31:19.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T22:31:19.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T22:31:19.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T22:31:19.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T22:31:19.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T22:31:19.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T22:31:19.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T22:31:19.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T22:31:19.878 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T22:31:22.669 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T22:31:22.669 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:31:22.669 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2024-03-18T22:31:22.669 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-18T22:31:22.669 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-18T22:31:22.670 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-18T22:31:22.670 INFO:teuthology.orchestra.run.smithi040.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-18T22:31:22.670 INFO:teuthology.orchestra.run.smithi040.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T22:31:22.670 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T22:31:22.670 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T22:31:22.670 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T22:31:22.670 INFO:teuthology.orchestra.run.smithi040.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T22:31:22.670 INFO:teuthology.orchestra.run.smithi040.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T22:31:22.670 INFO:teuthology.orchestra.run.smithi040.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T22:31:22.670 INFO:teuthology.orchestra.run.smithi040.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T22:31:22.670 INFO:teuthology.orchestra.run.smithi040.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T22:31:22.670 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.670 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.671 INFO:teuthology.orchestra.run.smithi040.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T22:31:22.671 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T22:31:22.671 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T22:31:22.671 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T22:31:22.671 INFO:teuthology.orchestra.run.smithi040.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T22:31:22.671 INFO:teuthology.orchestra.run.smithi040.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T22:31:22.671 INFO:teuthology.orchestra.run.smithi040.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T22:31:22.671 INFO:teuthology.orchestra.run.smithi040.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T22:31:22.671 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T22:31:22.671 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-03-18T22:31:22.671 INFO:teuthology.orchestra.run.smithi040.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T22:31:22.671 INFO:teuthology.orchestra.run.smithi040.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T22:31:22.671 INFO:teuthology.orchestra.run.smithi040.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T22:31:22.672 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T22:31:22.672 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.672 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.672 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.672 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.672 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:31:22.672 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.672 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.672 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.672 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:31:22.672 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:31:22.673 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:31:22.673 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:31:22.673 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:31:22.673 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.673 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.673 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:31:22.673 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.673 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.673 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.673 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:31:22.673 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:31:22.674 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.674 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.674 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-18T22:31:22.674 INFO:teuthology.orchestra.run.smithi040.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T22:31:22.674 INFO:teuthology.orchestra.run.smithi040.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T22:31:22.674 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T22:31:22.674 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T22:31:22.674 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T22:31:22.674 INFO:teuthology.orchestra.run.smithi040.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T22:31:22.674 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T22:31:22.674 INFO:teuthology.orchestra.run.smithi040.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T22:31:22.674 INFO:teuthology.orchestra.run.smithi040.stdout: jq-1.6-16.el9.x86_64 2024-03-18T22:31:22.675 INFO:teuthology.orchestra.run.smithi040.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T22:31:22.675 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T22:31:22.675 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T22:31:22.675 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.675 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.675 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.675 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T22:31:22.675 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T22:31:22.675 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T22:31:22.675 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T22:31:22.675 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T22:31:22.675 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.675 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.675 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T22:31:22.675 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.676 INFO:teuthology.orchestra.run.smithi040.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T22:31:22.676 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T22:31:22.676 INFO:teuthology.orchestra.run.smithi040.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T22:31:22.676 INFO:teuthology.orchestra.run.smithi040.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T22:31:22.676 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T22:31:22.676 INFO:teuthology.orchestra.run.smithi040.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T22:31:22.676 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T22:31:22.676 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T22:31:22.676 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T22:31:22.676 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T22:31:22.676 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T22:31:22.677 INFO:teuthology.orchestra.run.smithi040.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T22:31:22.677 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T22:31:22.677 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T22:31:22.677 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T22:31:22.677 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T22:31:22.677 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T22:31:22.677 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T22:31:22.677 INFO:teuthology.orchestra.run.smithi040.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T22:31:22.677 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T22:31:22.677 INFO:teuthology.orchestra.run.smithi040.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T22:31:22.677 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T22:31:22.677 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T22:31:22.677 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T22:31:22.677 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.678 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.678 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.678 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T22:31:22.678 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T22:31:22.678 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T22:31:22.678 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T22:31:22.678 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T22:31:22.678 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T22:31:22.678 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T22:31:22.678 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T22:31:22.679 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T22:31:22.679 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T22:31:22.679 INFO:teuthology.orchestra.run.smithi040.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T22:31:22.679 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T22:31:22.679 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T22:31:22.679 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T22:31:22.679 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T22:31:22.679 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T22:31:22.679 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T22:31:22.679 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T22:31:22.679 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T22:31:22.679 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T22:31:22.679 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T22:31:22.679 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T22:31:22.680 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T22:31:22.681 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T22:31:22.681 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T22:31:22.681 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.681 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.681 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T22:31:22.681 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T22:31:22.681 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T22:31:22.681 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.681 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T22:31:22.681 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T22:31:22.681 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T22:31:22.681 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T22:31:22.682 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T22:31:22.682 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T22:31:22.682 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T22:31:22.682 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T22:31:22.682 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T22:31:22.682 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T22:31:22.682 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T22:31:22.682 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T22:31:22.682 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T22:31:22.682 INFO:teuthology.orchestra.run.smithi040.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T22:31:22.682 INFO:teuthology.orchestra.run.smithi040.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T22:31:22.682 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.682 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.683 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-18T22:31:22.683 INFO:teuthology.orchestra.run.smithi040.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T22:31:22.683 INFO:teuthology.orchestra.run.smithi040.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T22:31:22.683 INFO:teuthology.orchestra.run.smithi040.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T22:31:22.683 INFO:teuthology.orchestra.run.smithi040.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T22:31:22.683 INFO:teuthology.orchestra.run.smithi040.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T22:31:22.683 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T22:31:22.683 INFO:teuthology.orchestra.run.smithi040.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T22:31:22.683 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:31:22.683 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-03-18T22:31:23.331 DEBUG:teuthology.parallel:result is None 2024-03-18T22:31:23.332 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-18T22:31:23.487 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T22:31:23.509 INFO:teuthology.orchestra.run.smithi040.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-18T22:31:23.509 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-18T22:31:23.509 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-18T22:31:23.511 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-18T22:31:23.655 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T22:31:23.680 INFO:teuthology.orchestra.run.smithi103.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-18T22:31:23.680 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-18T22:31:23.680 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-18T22:31:23.682 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-18T22:31:23.905 DEBUG:teuthology.orchestra.run.smithi113:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T22:31:23.931 INFO:teuthology.orchestra.run.smithi113.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-18T22:31:23.931 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-18T22:31:23.931 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-18T22:31:23.933 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-18T22:31:23.934 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:31:23.934 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T22:31:23.968 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-18T22:31:23.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T22:31:24.003 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-18T22:31:24.004 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T22:31:24.038 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-18T22:31:24.039 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:31:24.039 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T22:31:24.069 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T22:31:24.139 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-18T22:31:24.139 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T22:31:24.170 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T22:31:24.239 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-18T22:31:24.240 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T22:31:24.272 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T22:31:24.342 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-18T22:31:24.342 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:31:24.342 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T22:31:24.373 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T22:31:24.442 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-18T22:31:24.443 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T22:31:24.473 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T22:31:24.543 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-18T22:31:24.543 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T22:31:24.574 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T22:31:24.644 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-18T22:31:24.645 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:31:24.645 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T22:31:24.675 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T22:31:24.744 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-18T22:31:24.744 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T22:31:24.776 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T22:31:24.846 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-18T22:31:24.846 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T22:31:24.877 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T22:31:24.947 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-18T22:31:25.042 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'subvols': {'create': 2}} 2024-03-18T22:31:25.042 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-18T22:31:25.043 INFO:tasks.cephadm:Cluster fsid is 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:31:25.043 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-18T22:31:25.043 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.40', 'mon.b': '172.21.15.103', 'mon.c': '172.21.15.113'} 2024-03-18T22:31:25.043 INFO:tasks.cephadm:First mon is mon.a on smithi040 2024-03-18T22:31:25.043 INFO:tasks.cephadm:First mgr is x 2024-03-18T22:31:25.043 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-18T22:31:25.044 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname $(hostname -s) 2024-03-18T22:31:25.075 DEBUG:teuthology.orchestra.run.smithi103:> sudo hostname $(hostname -s) 2024-03-18T22:31:25.107 DEBUG:teuthology.orchestra.run.smithi113:> sudo hostname $(hostname -s) 2024-03-18T22:31:25.138 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-18T22:31:25.138 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-18T22:31:25.285 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77736/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2206-gb6ae2491', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2206.gb6ae2491'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-18 20:42:27.872142', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240318.181317', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-18T22:31:25.502 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-batrick-testing-20240318.181317, sha1 b6ae24918b03b1b8a19c9263a50655bba80bb810 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-18T22:31:25.504 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm 2024-03-18T22:31:25.504 DEBUG:teuthology.orchestra.run.smithi040:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T22:31:26.135 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r--. 1 ubuntu ubuntu 767404 Mar 18 22:31 /home/ubuntu/cephtest/cephadm 2024-03-18T22:31:26.136 DEBUG:teuthology.orchestra.run.smithi103:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T22:31:26.604 INFO:teuthology.orchestra.run.smithi103.stdout:-rw-r--r--. 1 ubuntu ubuntu 767404 Mar 18 22:31 /home/ubuntu/cephtest/cephadm 2024-03-18T22:31:26.605 DEBUG:teuthology.orchestra.run.smithi113:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T22:31:27.141 INFO:teuthology.orchestra.run.smithi113.stdout:-rw-r--r--. 1 ubuntu ubuntu 767404 Mar 18 22:31 /home/ubuntu/cephtest/cephadm 2024-03-18T22:31:27.141 DEBUG:teuthology.orchestra.run.smithi040:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T22:31:27.160 DEBUG:teuthology.orchestra.run.smithi103:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T22:31:27.179 DEBUG:teuthology.orchestra.run.smithi113:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T22:31:27.249 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 on all hosts... 2024-03-18T22:31:27.249 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 pull 2024-03-18T22:31:27.252 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 pull 2024-03-18T22:31:27.254 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 pull 2024-03-18T22:31:27.995 INFO:teuthology.orchestra.run.smithi103.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-18T22:31:28.108 INFO:teuthology.orchestra.run.smithi113.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-18T22:31:28.115 INFO:teuthology.orchestra.run.smithi040.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-18T22:32:32.921 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:32:32.922 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-18T22:32:32.922 INFO:teuthology.orchestra.run.smithi040.stdout: "image_id": "2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c", 2024-03-18T22:32:32.922 INFO:teuthology.orchestra.run.smithi040.stdout: "repo_digests": [ 2024-03-18T22:32:32.922 INFO:teuthology.orchestra.run.smithi040.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639" 2024-03-18T22:32:32.922 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-18T22:32:32.922 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:32:36.696 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-18T22:32:36.696 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-18T22:32:36.696 INFO:teuthology.orchestra.run.smithi103.stdout: "image_id": "2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c", 2024-03-18T22:32:36.696 INFO:teuthology.orchestra.run.smithi103.stdout: "repo_digests": [ 2024-03-18T22:32:36.697 INFO:teuthology.orchestra.run.smithi103.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639" 2024-03-18T22:32:36.697 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-03-18T22:32:36.697 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-18T22:32:38.991 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-18T22:32:38.991 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-18T22:32:38.991 INFO:teuthology.orchestra.run.smithi113.stdout: "image_id": "2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c", 2024-03-18T22:32:38.991 INFO:teuthology.orchestra.run.smithi113.stdout: "repo_digests": [ 2024-03-18T22:32:38.991 INFO:teuthology.orchestra.run.smithi113.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639" 2024-03-18T22:32:38.991 INFO:teuthology.orchestra.run.smithi113.stdout: ] 2024-03-18T22:32:38.991 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-18T22:32:39.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph 2024-03-18T22:32:39.050 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph 2024-03-18T22:32:39.084 DEBUG:teuthology.orchestra.run.smithi113:> sudo mkdir -p /etc/ceph 2024-03-18T22:32:39.113 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /etc/ceph 2024-03-18T22:32:39.144 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /etc/ceph 2024-03-18T22:32:39.173 DEBUG:teuthology.orchestra.run.smithi113:> sudo chmod 777 /etc/ceph 2024-03-18T22:32:39.200 INFO:tasks.cephadm:Writing seed config... 2024-03-18T22:32:39.202 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-18T22:32:39.202 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-18T22:32:39.202 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-18T22:32:39.202 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-18T22:32:39.202 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-18T22:32:39.202 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-18T22:32:39.202 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-18T22:32:39.202 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-18T22:32:39.203 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-18T22:32:39.203 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-18T22:32:39.203 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-18T22:32:39.203 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-18T22:32:39.203 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-18T22:32:39.203 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-18T22:32:39.203 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-18T22:32:39.203 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-18T22:32:39.203 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-18T22:32:39.203 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-18T22:32:39.203 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-18T22:32:39.203 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-18T22:32:39.203 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-18T22:32:39.204 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-18T22:32:39.204 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-18T22:32:39.204 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-18T22:32:39.204 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-18T22:32:39.204 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-18T22:32:39.204 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-18T22:32:39.204 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-18T22:32:39.204 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-18T22:32:39.204 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-18T22:32:39.204 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-18T22:32:39.204 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-18T22:32:39.204 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-18T22:32:39.204 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-18T22:32:39.205 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-03-18T22:32:39.205 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-18T22:32:39.205 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-18T22:32:39.205 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-18T22:32:39.205 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-18T22:32:39.205 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-03-18T22:32:39.205 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-18T22:32:39.205 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-18T22:32:39.205 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-18T22:32:39.205 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-18T22:32:39.206 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:32:39.206 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-18T22:32:39.222 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 410ef414-e577-11ee-95c9-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-18T22:32:39.223 DEBUG:teuthology.orchestra.run.smithi040:mon.a> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.a.service 2024-03-18T22:32:39.265 DEBUG:teuthology.orchestra.run.smithi040:mgr.x> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mgr.x.service 2024-03-18T22:32:39.307 INFO:tasks.cephadm:Bootstrapping... 2024-03-18T22:32:39.307 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 -v bootstrap --fsid 410ef414-e577-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.40 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-18T22:32:39.505 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-03-18T22:32:39.505 INFO:teuthology.orchestra.run.smithi040.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810', '-v', 'bootstrap', '--fsid', '410ef414-e577-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.40', '--skip-admin-label'] 2024-03-18T22:32:39.531 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T22:32:39.531 INFO:teuthology.orchestra.run.smithi040.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-18T22:32:39.531 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying podman|docker is present... 2024-03-18T22:32:39.556 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T22:32:39.556 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying lvm2 is present... 2024-03-18T22:32:39.556 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying time synchronization is in place... 2024-03-18T22:32:39.563 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T22:32:39.563 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T22:32:39.569 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T22:32:39.569 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-03-18T22:32:39.575 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout enabled 2024-03-18T22:32:39.581 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout active 2024-03-18T22:32:39.581 INFO:teuthology.orchestra.run.smithi040.stdout:Unit chronyd.service is enabled and running 2024-03-18T22:32:39.581 INFO:teuthology.orchestra.run.smithi040.stdout:Repeating the final host check... 2024-03-18T22:32:39.604 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T22:32:39.604 INFO:teuthology.orchestra.run.smithi040.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-18T22:32:39.604 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl is present 2024-03-18T22:32:39.604 INFO:teuthology.orchestra.run.smithi040.stdout:lvcreate is present 2024-03-18T22:32:39.610 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T22:32:39.610 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T22:32:39.616 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T22:32:39.616 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-03-18T22:32:39.622 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout enabled 2024-03-18T22:32:39.628 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout active 2024-03-18T22:32:39.629 INFO:teuthology.orchestra.run.smithi040.stdout:Unit chronyd.service is enabled and running 2024-03-18T22:32:39.629 INFO:teuthology.orchestra.run.smithi040.stdout:Host looks OK 2024-03-18T22:32:39.629 INFO:teuthology.orchestra.run.smithi040.stdout:Cluster fsid: 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:32:39.629 INFO:teuthology.orchestra.run.smithi040.stdout:Acquiring lock 140263743585728 on /run/cephadm/410ef414-e577-11ee-95c9-87774f69a715.lock 2024-03-18T22:32:39.629 INFO:teuthology.orchestra.run.smithi040.stdout:Lock 140263743585728 acquired on /run/cephadm/410ef414-e577-11ee-95c9-87774f69a715.lock 2024-03-18T22:32:39.629 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying IP 172.21.15.40 port 3300 ... 2024-03-18T22:32:39.629 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying IP 172.21.15.40 port 6789 ... 2024-03-18T22:32:39.630 INFO:teuthology.orchestra.run.smithi040.stdout:Base mon IP(s) is [172.21.15.40:3300, 172.21.15.40:6789], mon addrv is [v2:172.21.15.40:3300,v1:172.21.15.40:6789] 2024-03-18T22:32:39.633 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.40 metric 100 2024-03-18T22:32:39.633 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.40 metric 100 2024-03-18T22:32:39.635 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-18T22:32:39.635 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-18T22:32:39.635 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-18T22:32:39.637 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-18T22:32:39.637 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-18T22:32:39.638 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T22:32:39.638 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-18T22:32:39.638 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-03-18T22:32:39.638 INFO:teuthology.orchestra.run.smithi040.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T22:32:39.638 INFO:teuthology.orchestra.run.smithi040.stdout:Mon IP `172.21.15.40` is in CIDR network `172.21.0.0/20` 2024-03-18T22:32:39.639 INFO:teuthology.orchestra.run.smithi040.stdout:Mon IP `172.21.15.40` is in CIDR network `172.21.0.0/20` 2024-03-18T22:32:39.639 INFO:teuthology.orchestra.run.smithi040.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-18T22:32:39.639 INFO:teuthology.orchestra.run.smithi040.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-18T22:32:39.640 INFO:teuthology.orchestra.run.smithi040.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-18T22:32:40.015 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stdout 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c 2024-03-18T22:32:40.015 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-18T22:32:40.015 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Getting image source signatures 2024-03-18T22:32:40.015 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-18T22:32:40.015 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Copying blob sha256:93b5a8d362aeee7e8b1f0bbf18238e1bb256c50573ad5b72aff9e382b1bb3ccb 2024-03-18T22:32:40.015 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Copying config sha256:2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c 2024-03-18T22:32:40.016 INFO:teuthology.orchestra.run.smithi040.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-18T22:32:40.961 INFO:teuthology.orchestra.run.smithi040.stdout:ceph: stdout ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev) 2024-03-18T22:32:40.962 INFO:teuthology.orchestra.run.smithi040.stdout:Ceph version: ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev) 2024-03-18T22:32:40.962 INFO:teuthology.orchestra.run.smithi040.stdout:Extracting ceph user uid/gid from container image... 2024-03-18T22:32:41.859 INFO:teuthology.orchestra.run.smithi040.stdout:stat: stdout 167 167 2024-03-18T22:32:41.859 INFO:teuthology.orchestra.run.smithi040.stdout:Creating initial keys... 2024-03-18T22:32:42.733 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph-authtool: stdout AQAKwfhlrhkhCxAAOyCIGQEZ5O0VK8c/9krL5A== 2024-03-18T22:32:43.731 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph-authtool: stdout AQALwfhlF4gBAhAAKShEbMJjudkV7+fKIXrmeQ== 2024-03-18T22:32:44.561 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph-authtool: stdout AQAMwfhlYZpZAhAA8KRckMxtdK6sEUJsnOoJKw== 2024-03-18T22:32:44.562 INFO:teuthology.orchestra.run.smithi040.stdout:Creating initial monmap... 2024-03-18T22:32:45.626 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T22:32:45.627 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-18T22:32:45.627 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:32:45.627 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T22:32:45.627 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool for a [v2:172.21.15.40:3300,v1:172.21.15.40:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T22:32:45.627 INFO:teuthology.orchestra.run.smithi040.stdout:setting min_mon_release = quincy 2024-03-18T22:32:45.627 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: set fsid to 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:32:45.627 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T22:32:45.627 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:32:45.627 INFO:teuthology.orchestra.run.smithi040.stdout:Creating mon... 2024-03-18T22:32:46.573 INFO:teuthology.orchestra.run.smithi040.stdout:create mon.a on 2024-03-18T22:32:46.782 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-18T22:32:46.926 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-18T22:32:47.097 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-410ef414-e577-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-410ef414-e577-11ee-95c9-87774f69a715.target. 2024-03-18T22:32:47.098 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-410ef414-e577-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-410ef414-e577-11ee-95c9-87774f69a715.target. 2024-03-18T22:32:47.351 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.a 2024-03-18T22:32:47.351 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to reset failed state of unit ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-18T22:32:47.520 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-410ef414-e577-11ee-95c9-87774f69a715.target.wants/ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-410ef414-e577-11ee-95c9-87774f69a715@.service. 2024-03-18T22:32:48.049 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T22:32:48.049 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-03-18T22:32:48.057 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T22:32:48.058 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-03-18T22:32:48.058 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-03-18T22:32:48.058 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T22:32:48.058 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mon to start... 2024-03-18T22:32:48.058 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mon... 2024-03-18T22:32:49.056 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout cluster: 2024-03-18T22:32:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout id: 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:32:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-18T22:32:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-03-18T22:32:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout services: 2024-03-18T22:32:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.508922s) 2024-03-18T22:32:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-18T22:32:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-18T22:32:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-03-18T22:32:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout data: 2024-03-18T22:32:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-18T22:32:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-18T22:32:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-18T22:32:49.057 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout pgs: 2024-03-18T22:32:49.058 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-03-18T22:32:49.058 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.460+0000 7f902b30e700 1 Processor -- start 2024-03-18T22:32:49.058 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.460+0000 7f902b30e700 1 -- start start 2024-03-18T22:32:49.058 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.460+0000 7f902b30e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024106a60 0x7f9024106e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:32:49.058 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.460+0000 7f902b30e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9024107450 con 0x7f9024106a60 2024-03-18T22:32:49.058 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.461+0000 7f90290aa700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024106a60 0x7f9024106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:32:49.058 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.461+0000 7f90290aa700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024106a60 0x7f9024106e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40860/0 (socket says 172.21.15.40:40860) 2024-03-18T22:32:49.058 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.461+0000 7f90290aa700 1 -- 172.21.15.40:0/1931412034 learned_addr learned my addr 172.21.15.40:0/1931412034 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:32:49.058 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.461+0000 7f90290aa700 1 -- 172.21.15.40:0/1931412034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9024107c60 con 0x7f9024106a60 2024-03-18T22:32:49.058 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.461+0000 7f90290aa700 1 --2- 172.21.15.40:0/1931412034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024106a60 0x7f9024106e80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f901c009e80 tx=0x7f901c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde9e61db66100bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:32:49.058 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.461+0000 7f9023fff700 1 -- 172.21.15.40:0/1931412034 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f901c017070 con 0x7f9024106a60 2024-03-18T22:32:49.058 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.462+0000 7f9023fff700 1 -- 172.21.15.40:0/1931412034 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f901c003a20 con 0x7f9024106a60 2024-03-18T22:32:49.059 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.462+0000 7f9023fff700 1 -- 172.21.15.40:0/1931412034 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f901c003d20 con 0x7f9024106a60 2024-03-18T22:32:49.059 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.462+0000 7f902b30e700 1 -- 172.21.15.40:0/1931412034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024106a60 msgr2=0x7f9024106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:32:49.059 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.462+0000 7f902b30e700 1 --2- 172.21.15.40:0/1931412034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024106a60 0x7f9024106e80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f901c009e80 tx=0x7f901c00b330 comp rx=0 tx=0).stop 2024-03-18T22:32:49.059 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.462+0000 7f902b30e700 1 -- 172.21.15.40:0/1931412034 shutdown_connections 2024-03-18T22:32:49.059 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.462+0000 7f902b30e700 1 --2- 172.21.15.40:0/1931412034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024106a60 0x7f9024106e80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:32:49.059 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.462+0000 7f902b30e700 1 -- 172.21.15.40:0/1931412034 >> 172.21.15.40:0/1931412034 conn(0x7f9024102020 msgr2=0x7f9024104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:32:49.059 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.462+0000 7f902b30e700 1 -- 172.21.15.40:0/1931412034 shutdown_connections 2024-03-18T22:32:49.059 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.462+0000 7f902b30e700 1 -- 172.21.15.40:0/1931412034 wait complete. 2024-03-18T22:32:49.059 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.463+0000 7f902b30e700 1 Processor -- start 2024-03-18T22:32:49.059 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.463+0000 7f902b30e700 1 -- start start 2024-03-18T22:32:49.059 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.463+0000 7f902b30e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024106a60 0x7f902419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:32:49.059 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.463+0000 7f902b30e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9024108190 con 0x7f9024106a60 2024-03-18T22:32:49.059 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.463+0000 7f90290aa700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024106a60 0x7f902419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:32:49.059 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.463+0000 7f90290aa700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024106a60 0x7f902419b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40862/0 (socket says 172.21.15.40:40862) 2024-03-18T22:32:49.060 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.463+0000 7f90290aa700 1 -- 172.21.15.40:0/1472918767 learned_addr learned my addr 172.21.15.40:0/1472918767 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:32:49.060 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.463+0000 7f90290aa700 1 -- 172.21.15.40:0/1472918767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f902419b7f0 con 0x7f9024106a60 2024-03-18T22:32:49.060 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.463+0000 7f90290aa700 1 --2- 172.21.15.40:0/1472918767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024106a60 0x7f902419b2b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f901c004000 tx=0x7f901c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:32:49.060 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.464+0000 7f90227fc700 1 -- 172.21.15.40:0/1472918767 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f901c022020 con 0x7f9024106a60 2024-03-18T22:32:49.060 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.464+0000 7f90227fc700 1 -- 172.21.15.40:0/1472918767 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f901c018040 con 0x7f9024106a60 2024-03-18T22:32:49.060 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.464+0000 7f902b30e700 1 -- 172.21.15.40:0/1472918767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f902419ba80 con 0x7f9024106a60 2024-03-18T22:32:49.060 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.464+0000 7f90227fc700 1 -- 172.21.15.40:0/1472918767 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f901c017070 con 0x7f9024106a60 2024-03-18T22:32:49.060 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.464+0000 7f902b30e700 1 -- 172.21.15.40:0/1472918767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f902419e770 con 0x7f9024106a60 2024-03-18T22:32:49.060 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.464+0000 7f90227fc700 1 -- 172.21.15.40:0/1472918767 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f901c025020 con 0x7f9024106a60 2024-03-18T22:32:49.060 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.464+0000 7f90227fc700 1 -- 172.21.15.40:0/1472918767 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f901c00f050 con 0x7f9024106a60 2024-03-18T22:32:49.060 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.465+0000 7f902b30e700 1 -- 172.21.15.40:0/1472918767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9024063940 con 0x7f9024106a60 2024-03-18T22:32:49.060 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.467+0000 7f90227fc700 1 -- 172.21.15.40:0/1472918767 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f901c02c030 con 0x7f9024106a60 2024-03-18T22:32:49.060 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.540+0000 7f902b30e700 1 -- 172.21.15.40:0/1472918767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f902419ec90 con 0x7f9024106a60 2024-03-18T22:32:49.060 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.540+0000 7f90227fc700 1 -- 172.21.15.40:0/1472918767 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f901c01e480 con 0x7f9024106a60 2024-03-18T22:32:49.061 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.541+0000 7f902b30e700 1 -- 172.21.15.40:0/1472918767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024106a60 msgr2=0x7f902419b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:32:49.061 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.541+0000 7f902b30e700 1 --2- 172.21.15.40:0/1472918767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024106a60 0x7f902419b2b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f901c004000 tx=0x7f901c004ad0 comp rx=0 tx=0).stop 2024-03-18T22:32:49.061 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.541+0000 7f902b30e700 1 -- 172.21.15.40:0/1472918767 shutdown_connections 2024-03-18T22:32:49.061 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.541+0000 7f902b30e700 1 --2- 172.21.15.40:0/1472918767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9024106a60 0x7f902419b2b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:32:49.061 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.541+0000 7f902b30e700 1 -- 172.21.15.40:0/1472918767 >> 172.21.15.40:0/1472918767 conn(0x7f9024102020 msgr2=0x7f902418e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:32:49.061 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.541+0000 7f902b30e700 1 -- 172.21.15.40:0/1472918767 shutdown_connections 2024-03-18T22:32:49.061 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:48.541+0000 7f902b30e700 1 -- 172.21.15.40:0/1472918767 wait complete. 2024-03-18T22:32:49.061 INFO:teuthology.orchestra.run.smithi040.stdout:mon is available 2024-03-18T22:32:49.061 INFO:teuthology.orchestra.run.smithi040.stdout:Assimilating anything we can from ceph.conf... 2024-03-18T22:32:50.230 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:50 smithi040 ceph-mon[62267]: from='client.? 172.21.15.40:0/1117578278' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-18T22:32:50.241 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-03-18T22:32:50.241 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T22:32:50.241 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout fsid = 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:32:50.241 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.40:3300,v1:172.21.15.40:6789] 2024-03-18T22:32:50.241 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T22:32:50.242 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T22:32:50.242 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T22:32:50.242 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T22:32:50.242 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-03-18T22:32:50.242 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T22:32:50.242 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T22:32:50.242 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-03-18T22:32:50.242 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T22:32:50.242 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-18T22:32:50.242 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T22:32:50.242 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T22:32:50.242 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T22:32:50.242 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T22:32:50.242 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T22:32:50.243 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.467+0000 7f17e9b9d700 1 Processor -- start 2024-03-18T22:32:50.243 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.467+0000 7f17e9b9d700 1 -- start start 2024-03-18T22:32:50.243 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.467+0000 7f17e9b9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4108480 0x7f17e410a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:32:50.243 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.467+0000 7f17e9b9d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f17e407bdd0 con 0x7f17e4108480 2024-03-18T22:32:50.243 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.467+0000 7f17e8b9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4108480 0x7f17e410a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:32:50.243 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.467+0000 7f17e8b9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4108480 0x7f17e410a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42202/0 (socket says 172.21.15.40:42202) 2024-03-18T22:32:50.243 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.467+0000 7f17e8b9b700 1 -- 172.21.15.40:0/2830495153 learned_addr learned my addr 172.21.15.40:0/2830495153 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:32:50.243 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.467+0000 7f17e8b9b700 1 -- 172.21.15.40:0/2830495153 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e410adb0 con 0x7f17e4108480 2024-03-18T22:32:50.243 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.468+0000 7f17e8b9b700 1 --2- 172.21.15.40:0/2830495153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4108480 0x7f17e410a870 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f17d80098a0 tx=0x7f17d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4a017b98285578 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:32:50.243 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.468+0000 7f17e37fe700 1 -- 172.21.15.40:0/2830495153 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f17d8004160 con 0x7f17e4108480 2024-03-18T22:32:50.243 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.468+0000 7f17e37fe700 1 -- 172.21.15.40:0/2830495153 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17d8004300 con 0x7f17e4108480 2024-03-18T22:32:50.243 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.468+0000 7f17e37fe700 1 -- 172.21.15.40:0/2830495153 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f17d80036a0 con 0x7f17e4108480 2024-03-18T22:32:50.243 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.468+0000 7f17e9b9d700 1 -- 172.21.15.40:0/2830495153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4108480 msgr2=0x7f17e410a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:32:50.244 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.468+0000 7f17e9b9d700 1 --2- 172.21.15.40:0/2830495153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4108480 0x7f17e410a870 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f17d80098a0 tx=0x7f17d8009bb0 comp rx=0 tx=0).stop 2024-03-18T22:32:50.244 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.468+0000 7f17e9b9d700 1 -- 172.21.15.40:0/2830495153 shutdown_connections 2024-03-18T22:32:50.244 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.468+0000 7f17e9b9d700 1 --2- 172.21.15.40:0/2830495153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4108480 0x7f17e410a870 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:32:50.244 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.468+0000 7f17e9b9d700 1 -- 172.21.15.40:0/2830495153 >> 172.21.15.40:0/2830495153 conn(0x7f17e4101e70 msgr2=0x7f17e4104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:32:50.244 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.468+0000 7f17e9b9d700 1 -- 172.21.15.40:0/2830495153 shutdown_connections 2024-03-18T22:32:50.244 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.468+0000 7f17e9b9d700 1 -- 172.21.15.40:0/2830495153 wait complete. 2024-03-18T22:32:50.244 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.469+0000 7f17e9b9d700 1 Processor -- start 2024-03-18T22:32:50.244 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.469+0000 7f17e9b9d700 1 -- start start 2024-03-18T22:32:50.244 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.469+0000 7f17e9b9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4108480 0x7f17e419afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:32:50.244 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.469+0000 7f17e9b9d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f17e410b1c0 con 0x7f17e4108480 2024-03-18T22:32:50.244 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.469+0000 7f17e8b9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4108480 0x7f17e419afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:32:50.244 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.469+0000 7f17e8b9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4108480 0x7f17e419afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42210/0 (socket says 172.21.15.40:42210) 2024-03-18T22:32:50.244 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.469+0000 7f17e8b9b700 1 -- 172.21.15.40:0/1117578278 learned_addr learned my addr 172.21.15.40:0/1117578278 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.469+0000 7f17e8b9b700 1 -- 172.21.15.40:0/1117578278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e419b520 con 0x7f17e4108480 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.470+0000 7f17e8b9b700 1 --2- 172.21.15.40:0/1117578278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4108480 0x7f17e419afe0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f17d8000c00 tx=0x7f17d8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.470+0000 7f17e1ffb700 1 -- 172.21.15.40:0/1117578278 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f17d8003db0 con 0x7f17e4108480 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.470+0000 7f17e1ffb700 1 -- 172.21.15.40:0/1117578278 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17d8011630 con 0x7f17e4108480 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.470+0000 7f17e1ffb700 1 -- 172.21.15.40:0/1117578278 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f17d8011950 con 0x7f17e4108480 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.470+0000 7f17e9b9d700 1 -- 172.21.15.40:0/1117578278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17e419b7b0 con 0x7f17e4108480 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.470+0000 7f17e9b9d700 1 -- 172.21.15.40:0/1117578278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17e419e4a0 con 0x7f17e4108480 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.470+0000 7f17e1ffb700 1 -- 172.21.15.40:0/1117578278 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f17d8011af0 con 0x7f17e4108480 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.470+0000 7f17e1ffb700 1 -- 172.21.15.40:0/1117578278 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f17d801c450 con 0x7f17e4108480 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.471+0000 7f17e9b9d700 1 -- 172.21.15.40:0/1117578278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17e4063940 con 0x7f17e4108480 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.473+0000 7f17e1ffb700 1 -- 172.21.15.40:0/1117578278 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f17d8032020 con 0x7f17e4108480 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.543+0000 7f17e9b9d700 1 -- 172.21.15.40:0/1117578278 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f17e402ddc0 con 0x7f17e4108480 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.546+0000 7f17e1ffb700 1 -- 172.21.15.40:0/1117578278 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+454 (secure 0 0 0) 0x7f17d8004260 con 0x7f17e4108480 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.547+0000 7f17e1ffb700 1 -- 172.21.15.40:0/1117578278 <== mon.0 v2:172.21.15.40:3300/0 8 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f17d801b450 con 0x7f17e4108480 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.548+0000 7f17e9b9d700 1 -- 172.21.15.40:0/1117578278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4108480 msgr2=0x7f17e419afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:32:50.245 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.548+0000 7f17e9b9d700 1 --2- 172.21.15.40:0/1117578278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4108480 0x7f17e419afe0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f17d8000c00 tx=0x7f17d8004750 comp rx=0 tx=0).stop 2024-03-18T22:32:50.246 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.548+0000 7f17e9b9d700 1 -- 172.21.15.40:0/1117578278 shutdown_connections 2024-03-18T22:32:50.246 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.548+0000 7f17e9b9d700 1 --2- 172.21.15.40:0/1117578278 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4108480 0x7f17e419afe0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:32:50.246 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.548+0000 7f17e9b9d700 1 -- 172.21.15.40:0/1117578278 >> 172.21.15.40:0/1117578278 conn(0x7f17e4101e70 msgr2=0x7f17e4102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:32:50.246 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.548+0000 7f17e9b9d700 1 -- 172.21.15.40:0/1117578278 shutdown_connections 2024-03-18T22:32:50.246 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:49.548+0000 7f17e9b9d700 1 -- 172.21.15.40:0/1117578278 wait complete. 2024-03-18T22:32:50.246 INFO:teuthology.orchestra.run.smithi040.stdout:Generating new minimal ceph.conf... 2024-03-18T22:32:51.210 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.619+0000 7fe6224cd700 1 Processor -- start 2024-03-18T22:32:51.210 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.619+0000 7fe6224cd700 1 -- start start 2024-03-18T22:32:51.210 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.620+0000 7fe6224cd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe61c108610 0x7fe61c10aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:32:51.210 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.620+0000 7fe6224cd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe61c075980 con 0x7fe61c108610 2024-03-18T22:32:51.210 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.620+0000 7fe61bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe61c108610 0x7fe61c10aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:32:51.210 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.620+0000 7fe61bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe61c108610 0x7fe61c10aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42216/0 (socket says 172.21.15.40:42216) 2024-03-18T22:32:51.211 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.620+0000 7fe61bfff700 1 -- 172.21.15.40:0/219092342 learned_addr learned my addr 172.21.15.40:0/219092342 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:32:51.211 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.620+0000 7fe61bfff700 1 -- 172.21.15.40:0/219092342 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe61c10af40 con 0x7fe61c108610 2024-03-18T22:32:51.211 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.620+0000 7fe61bfff700 1 --2- 172.21.15.40:0/219092342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe61c108610 0x7fe61c10aa00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe60c0098a0 tx=0x7fe60c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ee2dbe58421d20 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:32:51.211 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.620+0000 7fe61affd700 1 -- 172.21.15.40:0/219092342 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe60c004160 con 0x7fe61c108610 2024-03-18T22:32:51.211 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.620+0000 7fe61affd700 1 -- 172.21.15.40:0/219092342 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fe60c004300 con 0x7fe61c108610 2024-03-18T22:32:51.211 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.620+0000 7fe61affd700 1 -- 172.21.15.40:0/219092342 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe60c011a80 con 0x7fe61c108610 2024-03-18T22:32:51.211 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.621+0000 7fe6224cd700 1 -- 172.21.15.40:0/219092342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe61c108610 msgr2=0x7fe61c10aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:32:51.211 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.621+0000 7fe6224cd700 1 --2- 172.21.15.40:0/219092342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe61c108610 0x7fe61c10aa00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe60c0098a0 tx=0x7fe60c009bb0 comp rx=0 tx=0).stop 2024-03-18T22:32:51.211 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.621+0000 7fe6224cd700 1 -- 172.21.15.40:0/219092342 shutdown_connections 2024-03-18T22:32:51.211 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.621+0000 7fe6224cd700 1 --2- 172.21.15.40:0/219092342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe61c108610 0x7fe61c10aa00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:32:51.211 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.621+0000 7fe6224cd700 1 -- 172.21.15.40:0/219092342 >> 172.21.15.40:0/219092342 conn(0x7fe61c102020 msgr2=0x7fe61c104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:32:51.211 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.621+0000 7fe6224cd700 1 -- 172.21.15.40:0/219092342 shutdown_connections 2024-03-18T22:32:51.211 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.621+0000 7fe6224cd700 1 -- 172.21.15.40:0/219092342 wait complete. 2024-03-18T22:32:51.211 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.622+0000 7fe6224cd700 1 Processor -- start 2024-03-18T22:32:51.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.622+0000 7fe6224cd700 1 -- start start 2024-03-18T22:32:51.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.622+0000 7fe6224cd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe61c108610 0x7fe61c19ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:32:51.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.622+0000 7fe6224cd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe61c10bf00 con 0x7fe61c108610 2024-03-18T22:32:51.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.622+0000 7fe61bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe61c108610 0x7fe61c19ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:32:51.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.622+0000 7fe61bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe61c108610 0x7fe61c19ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42218/0 (socket says 172.21.15.40:42218) 2024-03-18T22:32:51.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.622+0000 7fe61bfff700 1 -- 172.21.15.40:0/1549455405 learned_addr learned my addr 172.21.15.40:0/1549455405 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:32:51.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.622+0000 7fe61bfff700 1 -- 172.21.15.40:0/1549455405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe61c19bf90 con 0x7fe61c108610 2024-03-18T22:32:51.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.622+0000 7fe61bfff700 1 --2- 172.21.15.40:0/1549455405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe61c108610 0x7fe61c19ba50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe60c011a50 tx=0x7fe60c004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:32:51.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.623+0000 7fe6197fa700 1 -- 172.21.15.40:0/1549455405 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe60c0039f0 con 0x7fe61c108610 2024-03-18T22:32:51.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.623+0000 7fe6197fa700 1 -- 172.21.15.40:0/1549455405 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fe60c003b90 con 0x7fe61c108610 2024-03-18T22:32:51.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.623+0000 7fe6224cd700 1 -- 172.21.15.40:0/1549455405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe61c19c220 con 0x7fe61c108610 2024-03-18T22:32:51.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.623+0000 7fe6197fa700 1 -- 172.21.15.40:0/1549455405 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe60c01a600 con 0x7fe61c108610 2024-03-18T22:32:51.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.623+0000 7fe6224cd700 1 -- 172.21.15.40:0/1549455405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe61c19ef10 con 0x7fe61c108610 2024-03-18T22:32:51.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.623+0000 7fe6197fa700 1 -- 172.21.15.40:0/1549455405 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe60c01acd0 con 0x7fe61c108610 2024-03-18T22:32:51.212 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.623+0000 7fe6197fa700 1 -- 172.21.15.40:0/1549455405 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe60c019700 con 0x7fe61c108610 2024-03-18T22:32:51.213 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.624+0000 7fe6224cd700 1 -- 172.21.15.40:0/1549455405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe61c063940 con 0x7fe61c108610 2024-03-18T22:32:51.213 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.626+0000 7fe6197fa700 1 -- 172.21.15.40:0/1549455405 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe60c024a40 con 0x7fe61c108610 2024-03-18T22:32:51.213 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.697+0000 7fe6224cd700 1 -- 172.21.15.40:0/1549455405 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fe61c19f4c0 con 0x7fe61c108610 2024-03-18T22:32:51.213 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.698+0000 7fe6197fa700 1 -- 172.21.15.40:0/1549455405 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fe60c024440 con 0x7fe61c108610 2024-03-18T22:32:51.213 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.698+0000 7fe6224cd700 1 -- 172.21.15.40:0/1549455405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe61c108610 msgr2=0x7fe61c19ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:32:51.213 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.698+0000 7fe6224cd700 1 --2- 172.21.15.40:0/1549455405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe61c108610 0x7fe61c19ba50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe60c011a50 tx=0x7fe60c004290 comp rx=0 tx=0).stop 2024-03-18T22:32:51.213 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.698+0000 7fe6224cd700 1 -- 172.21.15.40:0/1549455405 shutdown_connections 2024-03-18T22:32:51.213 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.698+0000 7fe6224cd700 1 --2- 172.21.15.40:0/1549455405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe61c108610 0x7fe61c19ba50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:32:51.213 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.698+0000 7fe6224cd700 1 -- 172.21.15.40:0/1549455405 >> 172.21.15.40:0/1549455405 conn(0x7fe61c102020 msgr2=0x7fe61c102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:32:51.213 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.698+0000 7fe6224cd700 1 -- 172.21.15.40:0/1549455405 shutdown_connections 2024-03-18T22:32:51.213 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:50.698+0000 7fe6224cd700 1 -- 172.21.15.40:0/1549455405 wait complete. 2024-03-18T22:32:51.213 INFO:teuthology.orchestra.run.smithi040.stdout:Restarting the monitor... 2024-03-18T22:32:51.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:51 smithi040 ceph-mon[62267]: from='client.? 172.21.15.40:0/1549455405' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:32:51.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:51 smithi040 systemd[1]: Stopping Ceph mon.a for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:32:51.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:51 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a[62263]: 2024-03-18T22:32:51.489+0000 7fb1f83dd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T22:32:51.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:51 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a[62263]: 2024-03-18T22:32:51.489+0000 7fb1f83dd700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-18T22:32:51.748 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:51 smithi040 podman[62513]: 2024-03-18 22:32:51.509790842 +0000 UTC m=+0.137789587 container died 85a5a7119095505ae27de786b3bb9a75ec734f5c41c17e6e45ec07246bf79afe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-18T22:32:52.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:51 smithi040 podman[62513]: 2024-03-18 22:32:51.846954193 +0000 UTC m=+0.474952939 container cleanup 85a5a7119095505ae27de786b3bb9a75ec734f5c41c17e6e45ec07246bf79afe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0) 2024-03-18T22:32:52.023 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:51 smithi040 bash[62513]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a 2024-03-18T22:32:52.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 podman[62527]: 2024-03-18 22:32:52.022927062 +0000 UTC m=+0.508129823 container remove 85a5a7119095505ae27de786b3bb9a75ec734f5c41c17e6e45ec07246bf79afe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-18T22:32:52.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 systemd[1]: ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-18T22:32:52.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 systemd[1]: Stopped Ceph mon.a for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:32:52.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 systemd[1]: Starting Ceph mon.a for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:32:52.760 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 podman[62624]: 2024-03-18 22:32:52.383045452 +0000 UTC m=+0.018571777 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-18T22:32:52.760 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 podman[62624]: 2024-03-18 22:32:52.498869256 +0000 UTC m=+0.134395562 container create 6bd216c61bbfb5251c558b872e05d9af454c3acbc349d40bdd04261e1280e3ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T22:32:52.760 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 podman[62624]: 2024-03-18 22:32:52.676241574 +0000 UTC m=+0.311767893 container init 6bd216c61bbfb5251c558b872e05d9af454c3acbc349d40bdd04261e1280e3ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-03-18T22:32:52.760 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 podman[62624]: 2024-03-18 22:32:52.67905809 +0000 UTC m=+0.314584398 container start 6bd216c61bbfb5251c558b872e05d9af454c3acbc349d40bdd04261e1280e3ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-18T22:32:52.760 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T22:32:52.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev), process ceph-mon, pid 2 2024-03-18T22:32:52.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: pidfile_write: ignore empty --pid-file 2024-03-18T22:32:52.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: load: jerasure load: lrc 2024-03-18T22:32:52.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: RocksDB version: 7.9.2 2024-03-18T22:32:52.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Git sha 0 2024-03-18T22:32:52.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Compile date 2024-03-18 18:57:12 2024-03-18T22:32:52.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: DB SUMMARY 2024-03-18T22:32:52.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: DB Session ID: EKIV0W15PS97BFL24DFY 2024-03-18T22:32:52.761 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: CURRENT file: CURRENT 2024-03-18T22:32:52.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T22:32:52.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-18T22:32:52.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-18T22:32:52.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 103668 ; 2024-03-18T22:32:52.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.error_if_exists: 0 2024-03-18T22:32:52.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.create_if_missing: 0 2024-03-18T22:32:52.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.paranoid_checks: 1 2024-03-18T22:32:52.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T22:32:52.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T22:32:52.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T22:32:52.762 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.env: 0x55fa9621cc60 2024-03-18T22:32:52.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T22:32:52.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.info_log: 0x55fa9870fae0 2024-03-18T22:32:52.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T22:32:52.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.statistics: (nil) 2024-03-18T22:32:52.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.use_fsync: 0 2024-03-18T22:32:52.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_log_file_size: 0 2024-03-18T22:32:52.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T22:32:52.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T22:32:52.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T22:32:52.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T22:32:52.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.allow_fallocate: 1 2024-03-18T22:32:52.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T22:32:52.763 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T22:32:52.764 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.use_direct_reads: 0 2024-03-18T22:32:52.764 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T22:32:52.764 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T22:32:52.764 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.db_log_dir: 2024-03-18T22:32:52.764 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.wal_dir: 2024-03-18T22:32:52.764 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T22:32:52.764 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T22:32:52.764 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T22:32:52.764 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T22:32:52.764 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T22:32:52.765 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T22:32:52.765 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T22:32:52.765 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T22:32:52.765 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.write_buffer_manager: 0x55fa986e21e0 2024-03-18T22:32:52.765 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T22:32:52.765 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T22:32:52.765 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T22:32:52.765 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T22:32:52.765 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T22:32:52.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T22:32:52.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T22:32:52.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T22:32:52.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.unordered_write: 0 2024-03-18T22:32:52.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T22:32:52.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T22:32:52.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T22:32:52.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T22:32:52.767 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.row_cache: None 2024-03-18T22:32:52.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.wal_filter: None 2024-03-18T22:32:52.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T22:32:52.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T22:32:52.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.two_write_queues: 0 2024-03-18T22:32:52.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T22:32:52.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.wal_compression: 0 2024-03-18T22:32:52.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.atomic_flush: 0 2024-03-18T22:32:52.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T22:32:52.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T22:32:52.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T22:32:52.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.log_readahead_size: 0 2024-03-18T22:32:52.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T22:32:52.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T22:32:52.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T22:32:52.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T22:32:52.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T22:32:52.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T22:32:52.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T22:32:52.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_background_jobs: 2 2024-03-18T22:32:52.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_background_compactions: -1 2024-03-18T22:32:52.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_subcompactions: 1 2024-03-18T22:32:52.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T22:32:52.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T22:32:52.769 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T22:32:52.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T22:32:52.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T22:32:52.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T22:32:52.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T22:32:52.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T22:32:52.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_open_files: -1 2024-03-18T22:32:52.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T22:32:52.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T22:32:52.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T22:32:52.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T22:32:52.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_background_flushes: -1 2024-03-18T22:32:52.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Compression algorithms supported: 2024-03-18T22:32:52.770 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T22:32:52.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: kZSTD supported: 0 2024-03-18T22:32:52.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: kXpressCompression supported: 0 2024-03-18T22:32:52.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T22:32:52.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: kZlibCompression supported: 1 2024-03-18T22:32:52.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: kSnappyCompression supported: 1 2024-03-18T22:32:52.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: kLZ4Compression supported: 1 2024-03-18T22:32:52.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: kBZip2Compression supported: 0 2024-03-18T22:32:52.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T22:32:52.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T22:32:52.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-18T22:32:52.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T22:32:52.771 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T22:32:52.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.merge_operator: 2024-03-18T22:32:52.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compaction_filter: None 2024-03-18T22:32:52.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T22:32:52.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T22:32:52.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T22:32:52.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T22:32:52.772 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fa9870fbe0) 2024-03-18T22:32:52.772 INFO:journalctl@ceph.mon.a.smithi040.stdout: cache_index_and_filter_blocks: 1 2024-03-18T22:32:52.772 INFO:journalctl@ceph.mon.a.smithi040.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T22:32:52.772 INFO:journalctl@ceph.mon.a.smithi040.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T22:32:52.772 INFO:journalctl@ceph.mon.a.smithi040.stdout: pin_top_level_index_and_filter: 1 2024-03-18T22:32:52.772 INFO:journalctl@ceph.mon.a.smithi040.stdout: index_type: 0 2024-03-18T22:32:52.772 INFO:journalctl@ceph.mon.a.smithi040.stdout: data_block_index_type: 0 2024-03-18T22:32:52.773 INFO:journalctl@ceph.mon.a.smithi040.stdout: index_shortening: 1 2024-03-18T22:32:52.773 INFO:journalctl@ceph.mon.a.smithi040.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T22:32:52.773 INFO:journalctl@ceph.mon.a.smithi040.stdout: checksum: 4 2024-03-18T22:32:52.773 INFO:journalctl@ceph.mon.a.smithi040.stdout: no_block_cache: 0 2024-03-18T22:32:52.773 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache: 0x55fa98715090 2024-03-18T22:32:52.773 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache_name: BinnedLRUCache 2024-03-18T22:32:52.773 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache_options: 2024-03-18T22:32:52.773 INFO:journalctl@ceph.mon.a.smithi040.stdout: capacity : 536870912 2024-03-18T22:32:52.773 INFO:journalctl@ceph.mon.a.smithi040.stdout: num_shard_bits : 4 2024-03-18T22:32:52.773 INFO:journalctl@ceph.mon.a.smithi040.stdout: strict_capacity_limit : 0 2024-03-18T22:32:52.773 INFO:journalctl@ceph.mon.a.smithi040.stdout: high_pri_pool_ratio: 0.000 2024-03-18T22:32:52.773 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_cache_compressed: (nil) 2024-03-18T22:32:52.774 INFO:journalctl@ceph.mon.a.smithi040.stdout: persistent_cache: (nil) 2024-03-18T22:32:52.774 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_size: 4096 2024-03-18T22:32:52.774 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_size_deviation: 10 2024-03-18T22:32:52.774 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_restart_interval: 16 2024-03-18T22:32:52.774 INFO:journalctl@ceph.mon.a.smithi040.stdout: index_block_restart_interval: 1 2024-03-18T22:32:52.774 INFO:journalctl@ceph.mon.a.smithi040.stdout: metadata_block_size: 4096 2024-03-18T22:32:52.774 INFO:journalctl@ceph.mon.a.smithi040.stdout: partition_filters: 0 2024-03-18T22:32:52.774 INFO:journalctl@ceph.mon.a.smithi040.stdout: use_delta_encoding: 1 2024-03-18T22:32:52.774 INFO:journalctl@ceph.mon.a.smithi040.stdout: filter_policy: bloomfilter 2024-03-18T22:32:52.774 INFO:journalctl@ceph.mon.a.smithi040.stdout: whole_key_filtering: 1 2024-03-18T22:32:52.774 INFO:journalctl@ceph.mon.a.smithi040.stdout: verify_compression: 0 2024-03-18T22:32:52.774 INFO:journalctl@ceph.mon.a.smithi040.stdout: read_amp_bytes_per_bit: 0 2024-03-18T22:32:52.775 INFO:journalctl@ceph.mon.a.smithi040.stdout: format_version: 5 2024-03-18T22:32:52.775 INFO:journalctl@ceph.mon.a.smithi040.stdout: enable_index_compression: 1 2024-03-18T22:32:52.775 INFO:journalctl@ceph.mon.a.smithi040.stdout: block_align: 0 2024-03-18T22:32:52.775 INFO:journalctl@ceph.mon.a.smithi040.stdout: max_auto_readahead_size: 262144 2024-03-18T22:32:52.775 INFO:journalctl@ceph.mon.a.smithi040.stdout: prepopulate_block_cache: 0 2024-03-18T22:32:52.775 INFO:journalctl@ceph.mon.a.smithi040.stdout: initial_auto_readahead_size: 8192 2024-03-18T22:32:52.775 INFO:journalctl@ceph.mon.a.smithi040.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T22:32:52.775 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T22:32:52.775 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T22:32:52.775 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compression: NoCompression 2024-03-18T22:32:52.775 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T22:32:52.775 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T22:32:52.775 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T22:32:52.776 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.num_levels: 7 2024-03-18T22:32:52.776 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T22:32:52.776 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T22:32:52.776 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T22:32:52.776 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T22:32:52.776 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T22:32:52.776 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T22:32:52.776 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T22:32:52.776 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T22:32:52.776 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T22:32:52.776 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T22:32:52.777 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T22:32:52.777 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T22:32:52.777 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T22:32:52.777 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T22:32:52.777 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T22:32:52.777 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T22:32:52.777 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T22:32:52.777 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T22:32:52.777 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T22:32:52.777 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T22:32:52.777 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T22:32:52.777 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T22:32:52.778 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T22:32:52.778 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T22:32:52.778 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T22:32:52.778 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T22:32:52.778 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T22:32:52.778 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T22:32:52.778 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T22:32:52.778 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T22:32:52.778 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T22:32:52.778 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T22:32:52.778 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T22:32:52.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T22:32:52.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T22:32:52.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T22:32:52.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T22:32:52.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T22:32:52.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T22:32:52.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T22:32:52.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T22:32:52.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T22:32:52.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T22:32:52.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T22:32:52.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T22:32:52.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T22:32:52.779 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T22:32:52.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T22:32:52.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T22:32:52.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T22:32:52.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T22:32:52.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T22:32:52.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T22:32:52.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.table_properties_collectors: 2024-03-18T22:32:52.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.inplace_update_support: 0 2024-03-18T22:32:52.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T22:32:52.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T22:32:52.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T22:32:52.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T22:32:52.780 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.bloom_locality: 0 2024-03-18T22:32:52.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.max_successive_merges: 0 2024-03-18T22:32:52.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T22:32:52.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T22:32:52.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T22:32:52.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T22:32:52.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.ttl: 2592000 2024-03-18T22:32:52.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T22:32:52.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T22:32:52.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T22:32:52.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.enable_blob_files: false 2024-03-18T22:32:52.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.min_blob_size: 0 2024-03-18T22:32:52.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T22:32:52.781 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T22:32:52.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T22:32:52.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T22:32:52.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T22:32:52.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T22:32:52.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T22:32:52.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T22:32:52.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-18T22:32:52.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-18T22:32:52.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: de519530-8f6f-4d12-8772-26296a8741d2 2024-03-18T22:32:52.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710801172707221, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-18T22:32:52.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-18T22:32:52.782 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710801172708241, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 98904, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 320, "table_properties": {"data_size": 96873, "index_size": 313, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13818, "raw_average_key_size": 49, "raw_value_size": 89325, "raw_average_value_size": 320, "num_data_blocks": 12, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710801172, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "de519530-8f6f-4d12-8772-26296a8741d2", "db_session_id": "EKIV0W15PS97BFL24DFY", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-18T22:32:52.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710801172708326, "job": 1, "event": "recovery_finished"} 2024-03-18T22:32:52.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-18T22:32:52.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T22:32:52.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fa98808000 2024-03-18T22:32:52.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: DB pointer 0x55fa987f2000 2024-03-18T22:32:52.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T22:32:52.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T22:32:52.783 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** DB Stats ** 2024-03-18T22:32:52.783 INFO:journalctl@ceph.mon.a.smithi040.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T22:32:52.783 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T22:32:52.783 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T22:32:52.783 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T22:32:52.784 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T22:32:52.784 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T22:32:52.784 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T22:32:52.784 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-03-18T22:32:52.784 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** Compaction Stats [default] ** 2024-03-18T22:32:52.784 INFO:journalctl@ceph.mon.a.smithi040.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T22:32:52.784 INFO:journalctl@ceph.mon.a.smithi040.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T22:32:52.784 INFO:journalctl@ceph.mon.a.smithi040.stdout: L0 2/0 98.42 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T22:32:52.784 INFO:journalctl@ceph.mon.a.smithi040.stdout: Sum 2/0 98.42 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T22:32:52.784 INFO:journalctl@ceph.mon.a.smithi040.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T22:32:52.784 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-03-18T22:32:52.784 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** Compaction Stats [default] ** 2024-03-18T22:32:52.784 INFO:journalctl@ceph.mon.a.smithi040.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T22:32:52.784 INFO:journalctl@ceph.mon.a.smithi040.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T22:32:52.785 INFO:journalctl@ceph.mon.a.smithi040.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 139.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T22:32:52.785 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-03-18T22:32:52.785 INFO:journalctl@ceph.mon.a.smithi040.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T22:32:52.785 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-03-18T22:32:52.785 INFO:journalctl@ceph.mon.a.smithi040.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T22:32:52.785 INFO:journalctl@ceph.mon.a.smithi040.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T22:32:52.785 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T22:32:52.785 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T22:32:52.785 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T22:32:52.785 INFO:journalctl@ceph.mon.a.smithi040.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T22:32:52.785 INFO:journalctl@ceph.mon.a.smithi040.stdout: Cumulative compaction: 0.00 GB write, 31.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T22:32:52.786 INFO:journalctl@ceph.mon.a.smithi040.stdout: Interval compaction: 0.00 GB write, 31.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T22:32:52.786 INFO:journalctl@ceph.mon.a.smithi040.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T22:32:52.786 INFO:journalctl@ceph.mon.a.smithi040.stdout: Block cache BinnedLRUCache@0x55fa98715090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-18T22:32:52.786 INFO:journalctl@ceph.mon.a.smithi040.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-18T22:32:52.786 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-03-18T22:32:52.786 INFO:journalctl@ceph.mon.a.smithi040.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T22:32:52.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: starting mon.a rank 0 at public addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] at bind addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:32:52.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: mon.a@-1(???) e1 preinit fsid 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:32:52.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: mon.a@-1(???).mds e1 new map 2024-03-18T22:32:52.786 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: mon.a@-1(???).mds e1 print_map 2024-03-18T22:32:52.786 INFO:journalctl@ceph.mon.a.smithi040.stdout: e1 2024-03-18T22:32:52.787 INFO:journalctl@ceph.mon.a.smithi040.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T22:32:52.787 INFO:journalctl@ceph.mon.a.smithi040.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T22:32:52.787 INFO:journalctl@ceph.mon.a.smithi040.stdout: legacy client fscid: -1 2024-03-18T22:32:52.787 INFO:journalctl@ceph.mon.a.smithi040.stdout: 2024-03-18T22:32:52.787 INFO:journalctl@ceph.mon.a.smithi040.stdout: No filesystems configured 2024-03-18T22:32:52.787 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T22:32:52.787 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T22:32:52.787 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T22:32:52.787 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T22:32:52.787 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-18T22:32:52.787 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 bash[62624]: 6bd216c61bbfb5251c558b872e05d9af454c3acbc349d40bdd04261e1280e3ac 2024-03-18T22:32:52.787 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 systemd[1]: Started Ceph mon.a for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:32:52.788 INFO:teuthology.orchestra.run.smithi040.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-18T22:32:53.029 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T22:32:53.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: monmap epoch 1 2024-03-18T22:32:53.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: fsid 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:32:53.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: last_changed 2024-03-18T22:32:44.918096+0000 2024-03-18T22:32:53.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: created 2024-03-18T22:32:44.918096+0000 2024-03-18T22:32:53.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: min_mon_release 19 (squid) 2024-03-18T22:32:53.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: election_strategy: 1 2024-03-18T22:32:53.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-03-18T22:32:53.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: fsmap 2024-03-18T22:32:53.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T22:32:53.030 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:52 smithi040 ceph-mon[62643]: mgrmap e1: no daemons active 2024-03-18T22:32:53.743 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.132+0000 7f6d26446700 1 Processor -- start 2024-03-18T22:32:53.748 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.132+0000 7f6d26446700 1 -- start start 2024-03-18T22:32:53.748 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.133+0000 7f6d26446700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d20107b50 0x7f6d20107f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:32:53.748 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.133+0000 7f6d26446700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6d20108540 con 0x7f6d20107b50 2024-03-18T22:32:53.748 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.133+0000 7f6d1ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d20107b50 0x7f6d20107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:32:53.748 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.133+0000 7f6d1ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d20107b50 0x7f6d20107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42220/0 (socket says 172.21.15.40:42220) 2024-03-18T22:32:53.748 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.133+0000 7f6d1ffff700 1 -- 172.21.15.40:0/2603549498 learned_addr learned my addr 172.21.15.40:0/2603549498 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:32:53.749 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.133+0000 7f6d1ffff700 1 -- 172.21.15.40:0/2603549498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d20108d50 con 0x7f6d20107b50 2024-03-18T22:32:53.749 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.133+0000 7f6d1ffff700 1 --2- 172.21.15.40:0/2603549498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d20107b50 0x7f6d20107f70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6d100098a0 tx=0x7f6d10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f70f370e33a9a29 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:32:53.749 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.134+0000 7f6d1effd700 1 -- 172.21.15.40:0/2603549498 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d10004160 con 0x7f6d20107b50 2024-03-18T22:32:53.749 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.134+0000 7f6d1effd700 1 -- 172.21.15.40:0/2603549498 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6d10004300 con 0x7f6d20107b50 2024-03-18T22:32:53.749 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.134+0000 7f6d1effd700 1 -- 172.21.15.40:0/2603549498 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d10011a80 con 0x7f6d20107b50 2024-03-18T22:32:53.749 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.134+0000 7f6d26446700 1 -- 172.21.15.40:0/2603549498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d20107b50 msgr2=0x7f6d20107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:32:53.749 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.134+0000 7f6d26446700 1 --2- 172.21.15.40:0/2603549498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d20107b50 0x7f6d20107f70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6d100098a0 tx=0x7f6d10009bb0 comp rx=0 tx=0).stop 2024-03-18T22:32:53.749 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.134+0000 7f6d26446700 1 -- 172.21.15.40:0/2603549498 shutdown_connections 2024-03-18T22:32:53.749 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.134+0000 7f6d26446700 1 --2- 172.21.15.40:0/2603549498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d20107b50 0x7f6d20107f70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:32:53.749 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.134+0000 7f6d26446700 1 -- 172.21.15.40:0/2603549498 >> 172.21.15.40:0/2603549498 conn(0x7f6d201030f0 msgr2=0x7f6d20105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:32:53.749 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.134+0000 7f6d26446700 1 -- 172.21.15.40:0/2603549498 shutdown_connections 2024-03-18T22:32:53.749 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.134+0000 7f6d26446700 1 -- 172.21.15.40:0/2603549498 wait complete. 2024-03-18T22:32:53.749 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.135+0000 7f6d26446700 1 Processor -- start 2024-03-18T22:32:53.750 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.135+0000 7f6d26446700 1 -- start start 2024-03-18T22:32:53.750 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.135+0000 7f6d26446700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d20107b50 0x7f6d2019bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:32:53.750 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.135+0000 7f6d26446700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2010c100 con 0x7f6d20107b50 2024-03-18T22:32:53.750 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.135+0000 7f6d1ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d20107b50 0x7f6d2019bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:32:53.750 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.135+0000 7f6d1ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d20107b50 0x7f6d2019bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42236/0 (socket says 172.21.15.40:42236) 2024-03-18T22:32:53.750 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.135+0000 7f6d1ffff700 1 -- 172.21.15.40:0/2142694248 learned_addr learned my addr 172.21.15.40:0/2142694248 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:32:53.750 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.136+0000 7f6d1ffff700 1 -- 172.21.15.40:0/2142694248 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2019bff0 con 0x7f6d20107b50 2024-03-18T22:32:53.750 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.136+0000 7f6d1ffff700 1 --2- 172.21.15.40:0/2142694248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d20107b50 0x7f6d2019bab0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6d10000c00 tx=0x7f6d10004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:32:53.750 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.136+0000 7f6d1d7fa700 1 -- 172.21.15.40:0/2142694248 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d100039c0 con 0x7f6d20107b50 2024-03-18T22:32:53.750 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.136+0000 7f6d1d7fa700 1 -- 172.21.15.40:0/2142694248 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6d10003b60 con 0x7f6d20107b50 2024-03-18T22:32:53.750 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.136+0000 7f6d26446700 1 -- 172.21.15.40:0/2142694248 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d2019c280 con 0x7f6d20107b50 2024-03-18T22:32:53.750 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.136+0000 7f6d1d7fa700 1 -- 172.21.15.40:0/2142694248 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d1001a6f0 con 0x7f6d20107b50 2024-03-18T22:32:53.750 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.136+0000 7f6d26446700 1 -- 172.21.15.40:0/2142694248 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d2019ef70 con 0x7f6d20107b50 2024-03-18T22:32:53.750 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.136+0000 7f6d1d7fa700 1 -- 172.21.15.40:0/2142694248 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6d1002d3f0 con 0x7f6d20107b50 2024-03-18T22:32:53.751 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.136+0000 7f6d1d7fa700 1 -- 172.21.15.40:0/2142694248 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6d10019700 con 0x7f6d20107b50 2024-03-18T22:32:53.751 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.137+0000 7f6d26446700 1 -- 172.21.15.40:0/2142694248 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d20063940 con 0x7f6d20107b50 2024-03-18T22:32:53.751 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.139+0000 7f6d1d7fa700 1 -- 172.21.15.40:0/2142694248 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6d10024a40 con 0x7f6d20107b50 2024-03-18T22:32:53.751 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.209+0000 7f6d26446700 1 -- 172.21.15.40:0/2142694248 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f6d2019f530 con 0x7f6d20107b50 2024-03-18T22:32:53.751 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.211+0000 7f6d1d7fa700 1 -- 172.21.15.40:0/2142694248 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f6d10032030 con 0x7f6d20107b50 2024-03-18T22:32:53.751 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.212+0000 7f6d26446700 1 -- 172.21.15.40:0/2142694248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d20107b50 msgr2=0x7f6d2019bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:32:53.751 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.212+0000 7f6d26446700 1 --2- 172.21.15.40:0/2142694248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d20107b50 0x7f6d2019bab0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6d10000c00 tx=0x7f6d10004290 comp rx=0 tx=0).stop 2024-03-18T22:32:53.751 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.212+0000 7f6d26446700 1 -- 172.21.15.40:0/2142694248 shutdown_connections 2024-03-18T22:32:53.751 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.212+0000 7f6d26446700 1 --2- 172.21.15.40:0/2142694248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d20107b50 0x7f6d2019bab0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:32:53.751 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.212+0000 7f6d26446700 1 -- 172.21.15.40:0/2142694248 >> 172.21.15.40:0/2142694248 conn(0x7f6d201030f0 msgr2=0x7f6d20103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:32:53.751 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.212+0000 7f6d26446700 1 -- 172.21.15.40:0/2142694248 shutdown_connections 2024-03-18T22:32:53.751 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:53.212+0000 7f6d26446700 1 -- 172.21.15.40:0/2142694248 wait complete. 2024-03-18T22:32:53.751 INFO:teuthology.orchestra.run.smithi040.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-18T22:32:53.774 INFO:teuthology.orchestra.run.smithi040.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-18T22:32:53.774 INFO:teuthology.orchestra.run.smithi040.stdout:Creating mgr... 2024-03-18T22:32:53.775 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-18T22:32:53.775 INFO:teuthology.orchestra.run.smithi040.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-18T22:32:54.035 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-410ef414-e577-11ee-95c9-87774f69a715@mgr.x 2024-03-18T22:32:54.035 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Failed to reset failed state of unit ceph-410ef414-e577-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-410ef414-e577-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-18T22:32:54.175 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-410ef414-e577-11ee-95c9-87774f69a715.target.wants/ceph-410ef414-e577-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-410ef414-e577-11ee-95c9-87774f69a715@.service. 2024-03-18T22:32:54.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:54 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2142694248' entity='client.admin' 2024-03-18T22:32:54.671 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:54 smithi040 podman[62860]: 2024-03-18 22:32:54.598179769 +0000 UTC m=+0.303302474 container start ea399b57c2c6d8d33433f00ec9eabf65ebeb05ad9a15a5e4c55dee1e6f40a6fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T22:32:54.671 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:54 smithi040 bash[62860]: ea399b57c2c6d8d33433f00ec9eabf65ebeb05ad9a15a5e4c55dee1e6f40a6fe 2024-03-18T22:32:54.686 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T22:32:54.686 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-03-18T22:32:54.694 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T22:32:54.694 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-03-18T22:32:54.694 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-03-18T22:32:54.694 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T22:32:54.702 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T22:32:54.702 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-03-18T22:32:54.709 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T22:32:54.709 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-03-18T22:32:54.709 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-03-18T22:32:54.709 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-18T22:32:54.710 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr to start... 2024-03-18T22:32:54.710 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr... 2024-03-18T22:32:55.143 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:54 smithi040 systemd[1]: Started Ceph mgr.x for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:32:55.644 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:55 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:55.320+0000 7f272952f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T22:32:55.644 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:55 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:55.416+0000 7f272952f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T22:32:56.392 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:56 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:56.080+0000 7f272952f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T22:32:56.392 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:56 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:56.175+0000 7f272952f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T22:32:56.392 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:56 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:56.319+0000 7f272952f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T22:32:56.643 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:56 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:56.418+0000 7f272952f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T22:32:56.643 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:56 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:56.504+0000 7f272952f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T22:32:57.070 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:56 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:56.773+0000 7f272952f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T22:32:57.070 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:56 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:56.885+0000 7f272952f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T22:32:57.071 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:56 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:56.973+0000 7f272952f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T22:32:57.071 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:57 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:57.069+0000 7f272952f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T22:32:57.377 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:57 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:57.248+0000 7f272952f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T22:32:57.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:32:57 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2007153233' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T22:32:58.393 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:58 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:58.275+0000 7f272952f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T22:32:58.752 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:58 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:58.409+0000 7f272952f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T22:32:59.042 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:58 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:58.751+0000 7f272952f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T22:32:59.042 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:58 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:58.842+0000 7f272952f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T22:32:59.042 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:59 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:59.042+0000 7f272952f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T22:32:59.393 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:59 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:59.230+0000 7f272952f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T22:32:59.782 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:59 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:59.521+0000 7f272952f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T22:33:00.143 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:32:59 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:32:59.781+0000 7f272952f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T22:33:00.143 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:00 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:00.024+0000 7f272952f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T22:33:01.145 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:00 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:00.853+0000 7f272952f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T22:33:01.758 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:01.436+0000 7f272952f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T22:33:01.758 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:01.536+0000 7f272952f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T22:33:01.758 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:01.757+0000 7f272952f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T22:33:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-mon[62643]: Activating manager daemon x 2024-03-18T22:33:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-mon[62643]: mgrmap e2: x(active, starting, since 0.00301389s) 2024-03-18T22:33:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-mon[62643]: from='mgr.14100 172.21.15.40:0/896806289' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T22:33:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-mon[62643]: from='mgr.14100 172.21.15.40:0/896806289' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T22:33:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-mon[62643]: from='mgr.14100 172.21.15.40:0/896806289' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T22:33:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-mon[62643]: from='mgr.14100 172.21.15.40:0/896806289' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:33:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-mon[62643]: from='mgr.14100 172.21.15.40:0/896806289' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T22:33:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-mon[62643]: Manager daemon x is now available 2024-03-18T22:33:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-mon[62643]: from='mgr.14100 172.21.15.40:0/896806289' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T22:33:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-mon[62643]: from='mgr.14100 172.21.15.40:0/896806289' entity='mgr.x' 2024-03-18T22:33:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-mon[62643]: from='mgr.14100 172.21.15.40:0/896806289' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T22:33:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-mon[62643]: from='mgr.14100 172.21.15.40:0/896806289' entity='mgr.x' 2024-03-18T22:33:02.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:01 smithi040 ceph-mon[62643]: from='mgr.14100 172.21.15.40:0/896806289' entity='mgr.x' 2024-03-18T22:33:03.500 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-03-18T22:33:03.500 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-03-18T22:33:03.500 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "410ef414-e577-11ee-95c9-87774f69a715", 2024-03-18T22:33:03.500 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T22:33:03.500 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T22:33:03.500 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T22:33:03.500 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T22:33:03.500 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-03-18T22:33:03.500 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T22:33:03.500 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T22:33:03.500 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2024-03-18T22:33:03.501 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-03-18T22:33:03.501 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T22:33:03.501 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T22:33:03.501 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-03-18T22:33:03.501 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-03-18T22:33:03.501 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T22:33:03.501 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T22:33:03.501 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T22:33:03.501 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T22:33:03.501 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-03-18T22:33:03.501 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T22:33:03.501 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T22:33:03.501 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T22:33:03.501 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T22:33:03.501 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T22:33:03.502 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T22:33:03.502 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T22:33:03.502 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T22:33:03.502 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-03-18T22:33:03.509 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T22:33:03.509 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T22:33:03.509 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T22:33:03.510 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T22:33:03.510 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T22:33:03.510 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T22:33:03.510 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T22:33:03.510 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T22:33:03.510 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T22:33:03.510 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-03-18T22:33:03.510 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T22:33:03.510 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T22:33:03.510 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T22:33:03.510 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T22:33:03.510 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-03-18T22:33:03.510 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T22:33:03.510 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T22:33:03.511 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T22:33:03.511 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T22:33:03.511 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T22:33:03.511 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T22:33:03.511 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T22:33:03.511 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-03-18T22:33:03.511 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T22:33:03.511 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-03-18T22:33:03.511 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T22:33:03.511 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T22:33:03.511 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T22:32:48.033280+0000", 2024-03-18T22:33:03.511 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T22:33:03.511 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-03-18T22:33:03.511 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T22:33:03.511 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-03-18T22:33:03.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.477+0000 7fac277c0700 1 Processor -- start 2024-03-18T22:33:03.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.477+0000 7fac277c0700 1 -- start start 2024-03-18T22:33:03.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.477+0000 7fac277c0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac201085d0 0x7fac2010a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:03.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.477+0000 7fac277c0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fac2007be80 con 0x7fac201085d0 2024-03-18T22:33:03.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.477+0000 7fac2555c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac201085d0 0x7fac2010a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:03.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.477+0000 7fac2555c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac201085d0 0x7fac2010a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42260/0 (socket says 172.21.15.40:42260) 2024-03-18T22:33:03.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.477+0000 7fac2555c700 1 -- 172.21.15.40:0/501925298 learned_addr learned my addr 172.21.15.40:0/501925298 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:03.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.478+0000 7fac2555c700 1 -- 172.21.15.40:0/501925298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2010af00 con 0x7fac201085d0 2024-03-18T22:33:03.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.478+0000 7fac2555c700 1 --2- 172.21.15.40:0/501925298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac201085d0 0x7fac2010a9c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0098a0 tx=0x7fac1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4dfcee0cf9a6ed server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:03.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.478+0000 7fac17fff700 1 -- 172.21.15.40:0/501925298 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac1c004160 con 0x7fac201085d0 2024-03-18T22:33:03.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.478+0000 7fac17fff700 1 -- 172.21.15.40:0/501925298 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fac1c004300 con 0x7fac201085d0 2024-03-18T22:33:03.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.478+0000 7fac17fff700 1 -- 172.21.15.40:0/501925298 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac1c011a80 con 0x7fac201085d0 2024-03-18T22:33:03.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.479+0000 7fac277c0700 1 -- 172.21.15.40:0/501925298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac201085d0 msgr2=0x7fac2010a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:03.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.479+0000 7fac277c0700 1 --2- 172.21.15.40:0/501925298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac201085d0 0x7fac2010a9c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0098a0 tx=0x7fac1c009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:03.512 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.479+0000 7fac277c0700 1 -- 172.21.15.40:0/501925298 shutdown_connections 2024-03-18T22:33:03.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.479+0000 7fac277c0700 1 --2- 172.21.15.40:0/501925298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac201085d0 0x7fac2010a9c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:03.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.479+0000 7fac277c0700 1 -- 172.21.15.40:0/501925298 >> 172.21.15.40:0/501925298 conn(0x7fac20102000 msgr2=0x7fac20104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:03.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.479+0000 7fac277c0700 1 -- 172.21.15.40:0/501925298 shutdown_connections 2024-03-18T22:33:03.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.479+0000 7fac277c0700 1 -- 172.21.15.40:0/501925298 wait complete. 2024-03-18T22:33:03.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.479+0000 7fac277c0700 1 Processor -- start 2024-03-18T22:33:03.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.479+0000 7fac277c0700 1 -- start start 2024-03-18T22:33:03.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.479+0000 7fac277c0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac201085d0 0x7fac20107e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:03.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.479+0000 7fac277c0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fac2010bed0 con 0x7fac201085d0 2024-03-18T22:33:03.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.480+0000 7fac2555c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac201085d0 0x7fac20107e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:03.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.480+0000 7fac2555c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac201085d0 0x7fac20107e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42262/0 (socket says 172.21.15.40:42262) 2024-03-18T22:33:03.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.480+0000 7fac2555c700 1 -- 172.21.15.40:0/2007153233 learned_addr learned my addr 172.21.15.40:0/2007153233 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:03.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.480+0000 7fac2555c700 1 -- 172.21.15.40:0/2007153233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac20108390 con 0x7fac201085d0 2024-03-18T22:33:03.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.480+0000 7fac2555c700 1 --2- 172.21.15.40:0/2007153233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac201085d0 0x7fac20107e50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fac1c006af0 tx=0x7fac1c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:03.513 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.480+0000 7fac167fc700 1 -- 172.21.15.40:0/2007153233 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac1c003930 con 0x7fac201085d0 2024-03-18T22:33:03.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.480+0000 7fac167fc700 1 -- 172.21.15.40:0/2007153233 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fac1c003ad0 con 0x7fac201085d0 2024-03-18T22:33:03.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.480+0000 7fac277c0700 1 -- 172.21.15.40:0/2007153233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac20106580 con 0x7fac201085d0 2024-03-18T22:33:03.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.480+0000 7fac167fc700 1 -- 172.21.15.40:0/2007153233 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac1c0195e0 con 0x7fac201085d0 2024-03-18T22:33:03.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.480+0000 7fac277c0700 1 -- 172.21.15.40:0/2007153233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac20106ac0 con 0x7fac201085d0 2024-03-18T22:33:03.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.480+0000 7fac167fc700 1 -- 172.21.15.40:0/2007153233 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fac1c019c70 con 0x7fac201085d0 2024-03-18T22:33:03.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.480+0000 7fac167fc700 1 -- 172.21.15.40:0/2007153233 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fac1c024730 con 0x7fac201085d0 2024-03-18T22:33:03.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.481+0000 7fac277c0700 1 -- 172.21.15.40:0/2007153233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac20063940 con 0x7fac201085d0 2024-03-18T22:33:03.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.483+0000 7fac167fc700 1 -- 172.21.15.40:0/2007153233 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fac1c043dd0 con 0x7fac201085d0 2024-03-18T22:33:03.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.557+0000 7fac277c0700 1 -- 172.21.15.40:0/2007153233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fac20106d70 con 0x7fac201085d0 2024-03-18T22:33:03.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.557+0000 7fac167fc700 1 -- 172.21.15.40:0/2007153233 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fac1c032020 con 0x7fac201085d0 2024-03-18T22:33:03.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.558+0000 7fac277c0700 1 -- 172.21.15.40:0/2007153233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac201085d0 msgr2=0x7fac20107e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:03.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.558+0000 7fac277c0700 1 --2- 172.21.15.40:0/2007153233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac201085d0 0x7fac20107e50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fac1c006af0 tx=0x7fac1c011b70 comp rx=0 tx=0).stop 2024-03-18T22:33:03.514 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.558+0000 7fac277c0700 1 -- 172.21.15.40:0/2007153233 shutdown_connections 2024-03-18T22:33:03.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.558+0000 7fac277c0700 1 --2- 172.21.15.40:0/2007153233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac201085d0 0x7fac20107e50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:03.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.558+0000 7fac277c0700 1 -- 172.21.15.40:0/2007153233 >> 172.21.15.40:0/2007153233 conn(0x7fac20102000 msgr2=0x7fac20102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:03.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.558+0000 7fac277c0700 1 -- 172.21.15.40:0/2007153233 shutdown_connections 2024-03-18T22:33:03.515 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:32:57.558+0000 7fac277c0700 1 -- 172.21.15.40:0/2007153233 wait complete. 2024-03-18T22:33:03.515 INFO:teuthology.orchestra.run.smithi040.stdout:mgr not available, waiting (1/15)... 2024-03-18T22:33:04.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:03 smithi040 ceph-mon[62643]: mgrmap e3: x(active, since 1.00636s) 2024-03-18T22:33:05.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:05 smithi040 ceph-mon[62643]: mgrmap e4: x(active, since 3s) 2024-03-18T22:33:06.818 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-03-18T22:33:06.818 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-03-18T22:33:06.818 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsid": "410ef414-e577-11ee-95c9-87774f69a715", 2024-03-18T22:33:06.818 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T22:33:06.818 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T22:33:06.818 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T22:33:06.818 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T22:33:06.818 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-03-18T22:33:06.818 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T22:33:06.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T22:33:06.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 0 2024-03-18T22:33:06.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-03-18T22:33:06.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T22:33:06.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T22:33:06.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-03-18T22:33:06.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-18T22:33:06.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T22:33:06.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T22:33:06.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T22:33:06.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T22:33:06.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-03-18T22:33:06.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T22:33:06.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T22:33:06.819 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T22:33:06.820 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T22:33:06.820 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T22:33:06.828 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T22:33:06.828 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T22:33:06.828 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T22:33:06.828 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-03-18T22:33:06.828 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T22:33:06.829 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T22:33:06.829 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T22:33:06.829 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T22:33:06.829 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T22:33:06.829 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T22:33:06.829 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T22:33:06.829 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T22:33:06.829 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T22:33:06.829 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-03-18T22:33:06.829 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T22:33:06.829 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T22:33:06.829 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T22:33:06.829 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T22:33:06.829 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-03-18T22:33:06.830 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T22:33:06.830 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T22:33:06.830 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T22:33:06.830 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T22:33:06.830 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T22:33:06.830 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T22:33:06.830 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T22:33:06.830 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ], 2024-03-18T22:33:06.830 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T22:33:06.830 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-03-18T22:33:06.830 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T22:33:06.830 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T22:33:06.830 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T22:32:48.033280+0000", 2024-03-18T22:33:06.830 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T22:33:06.830 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout }, 2024-03-18T22:33:06.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T22:33:06.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-03-18T22:33:06.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.914+0000 7f3f29464700 1 Processor -- start 2024-03-18T22:33:06.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.914+0000 7f3f29464700 1 -- start start 2024-03-18T22:33:06.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.914+0000 7f3f29464700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f241085c0 0x7f3f2410a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:06.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.914+0000 7f3f29464700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2407be80 con 0x7f3f241085c0 2024-03-18T22:33:06.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.915+0000 7f3f22ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f241085c0 0x7f3f2410a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:06.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.915+0000 7f3f22ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f241085c0 0x7f3f2410a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50286/0 (socket says 172.21.15.40:50286) 2024-03-18T22:33:06.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.915+0000 7f3f22ffd700 1 -- 172.21.15.40:0/3235307698 learned_addr learned my addr 172.21.15.40:0/3235307698 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:06.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.915+0000 7f3f22ffd700 1 -- 172.21.15.40:0/3235307698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f2410aef0 con 0x7f3f241085c0 2024-03-18T22:33:06.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.915+0000 7f3f22ffd700 1 --2- 172.21.15.40:0/3235307698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f241085c0 0x7f3f2410a9b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f14009e80 tx=0x7f3f1400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744c6243b45ef332 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:06.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.915+0000 7f3f21ffb700 1 -- 172.21.15.40:0/3235307698 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f14017070 con 0x7f3f241085c0 2024-03-18T22:33:06.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.915+0000 7f3f21ffb700 1 -- 172.21.15.40:0/3235307698 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f3f1400b8c0 con 0x7f3f241085c0 2024-03-18T22:33:06.831 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.915+0000 7f3f21ffb700 1 -- 172.21.15.40:0/3235307698 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f14004520 con 0x7f3f241085c0 2024-03-18T22:33:06.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.916+0000 7f3f29464700 1 -- 172.21.15.40:0/3235307698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f241085c0 msgr2=0x7f3f2410a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:06.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.916+0000 7f3f29464700 1 --2- 172.21.15.40:0/3235307698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f241085c0 0x7f3f2410a9b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f14009e80 tx=0x7f3f1400b330 comp rx=0 tx=0).stop 2024-03-18T22:33:06.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.916+0000 7f3f29464700 1 -- 172.21.15.40:0/3235307698 shutdown_connections 2024-03-18T22:33:06.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.916+0000 7f3f29464700 1 --2- 172.21.15.40:0/3235307698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f241085c0 0x7f3f2410a9b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:06.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.916+0000 7f3f29464700 1 -- 172.21.15.40:0/3235307698 >> 172.21.15.40:0/3235307698 conn(0x7f3f24101ff0 msgr2=0x7f3f24104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:06.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.916+0000 7f3f29464700 1 -- 172.21.15.40:0/3235307698 shutdown_connections 2024-03-18T22:33:06.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.916+0000 7f3f29464700 1 -- 172.21.15.40:0/3235307698 wait complete. 2024-03-18T22:33:06.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.917+0000 7f3f29464700 1 Processor -- start 2024-03-18T22:33:06.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.917+0000 7f3f29464700 1 -- start start 2024-03-18T22:33:06.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.917+0000 7f3f29464700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f241085c0 0x7f3f2419ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:06.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.917+0000 7f3f29464700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2410bec0 con 0x7f3f241085c0 2024-03-18T22:33:06.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.917+0000 7f3f22ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f241085c0 0x7f3f2419ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:06.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.917+0000 7f3f22ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f241085c0 0x7f3f2419ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50288/0 (socket says 172.21.15.40:50288) 2024-03-18T22:33:06.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.917+0000 7f3f22ffd700 1 -- 172.21.15.40:0/2653651384 learned_addr learned my addr 172.21.15.40:0/2653651384 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:06.832 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.917+0000 7f3f22ffd700 1 -- 172.21.15.40:0/2653651384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f2419bfa0 con 0x7f3f241085c0 2024-03-18T22:33:06.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.918+0000 7f3f22ffd700 1 --2- 172.21.15.40:0/2653651384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f241085c0 0x7f3f2419ba60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3f140042b0 tx=0x7f3f14004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:06.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.918+0000 7f3f1bfff700 1 -- 172.21.15.40:0/2653651384 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f14020020 con 0x7f3f241085c0 2024-03-18T22:33:06.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.918+0000 7f3f1bfff700 1 -- 172.21.15.40:0/2653651384 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f3f1400bba0 con 0x7f3f241085c0 2024-03-18T22:33:06.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.918+0000 7f3f29464700 1 -- 172.21.15.40:0/2653651384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f2419c230 con 0x7f3f241085c0 2024-03-18T22:33:06.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.918+0000 7f3f1bfff700 1 -- 172.21.15.40:0/2653651384 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f14017040 con 0x7f3f241085c0 2024-03-18T22:33:06.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.918+0000 7f3f29464700 1 -- 172.21.15.40:0/2653651384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f2419ef20 con 0x7f3f241085c0 2024-03-18T22:33:06.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.919+0000 7f3f1bfff700 1 -- 172.21.15.40:0/2653651384 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f3f14023020 con 0x7f3f241085c0 2024-03-18T22:33:06.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.919+0000 7f3f29464700 1 -- 172.21.15.40:0/2653651384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f24063940 con 0x7f3f241085c0 2024-03-18T22:33:06.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.919+0000 7f3f1bfff700 1 --2- 172.21.15.40:0/2653651384 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7f3f10038ed0 0x7f3f1003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:06.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.919+0000 7f3f1bfff700 1 -- 172.21.15.40:0/2653651384 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3f1400f050 con 0x7f3f241085c0 2024-03-18T22:33:06.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.919+0000 7f3f227fc700 1 --2- 172.21.15.40:0/2653651384 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7f3f10038ed0 0x7f3f1003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:06.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.920+0000 7f3f227fc700 1 --2- 172.21.15.40:0/2653651384 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7f3f10038ed0 0x7f3f1003b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c006fd0 tx=0x7f3f0c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:06.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:05.924+0000 7f3f1bfff700 1 -- 172.21.15.40:0/2653651384 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3f1402a020 con 0x7f3f241085c0 2024-03-18T22:33:06.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:06.212+0000 7f3f29464700 1 -- 172.21.15.40:0/2653651384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f3f2419f0f0 con 0x7f3f241085c0 2024-03-18T22:33:06.833 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:06.212+0000 7f3f1bfff700 1 -- 172.21.15.40:0/2653651384 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f3f2419f0f0 con 0x7f3f241085c0 2024-03-18T22:33:06.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:06.214+0000 7f3f29464700 1 -- 172.21.15.40:0/2653651384 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7f3f10038ed0 msgr2=0x7f3f1003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:06.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:06.214+0000 7f3f29464700 1 --2- 172.21.15.40:0/2653651384 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7f3f10038ed0 0x7f3f1003b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c006fd0 tx=0x7f3f0c006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:06.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:06.214+0000 7f3f29464700 1 -- 172.21.15.40:0/2653651384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f241085c0 msgr2=0x7f3f2419ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:06.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:06.214+0000 7f3f29464700 1 --2- 172.21.15.40:0/2653651384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f241085c0 0x7f3f2419ba60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3f140042b0 tx=0x7f3f14004670 comp rx=0 tx=0).stop 2024-03-18T22:33:06.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:06.214+0000 7f3f29464700 1 -- 172.21.15.40:0/2653651384 shutdown_connections 2024-03-18T22:33:06.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:06.214+0000 7f3f29464700 1 --2- 172.21.15.40:0/2653651384 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7f3f10038ed0 0x7f3f1003b390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:06.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:06.214+0000 7f3f29464700 1 --2- 172.21.15.40:0/2653651384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f241085c0 0x7f3f2419ba60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:06.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:06.214+0000 7f3f29464700 1 -- 172.21.15.40:0/2653651384 >> 172.21.15.40:0/2653651384 conn(0x7f3f24101ff0 msgr2=0x7f3f24102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:06.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:06.215+0000 7f3f29464700 1 -- 172.21.15.40:0/2653651384 shutdown_connections 2024-03-18T22:33:06.834 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:06.215+0000 7f3f29464700 1 -- 172.21.15.40:0/2653651384 wait complete. 2024-03-18T22:33:06.834 INFO:teuthology.orchestra.run.smithi040.stdout:mgr is available 2024-03-18T22:33:06.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:06 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2653651384' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T22:33:07.985 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:07 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2183396204' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T22:33:07.994 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-03-18T22:33:07.995 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T22:33:07.995 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout fsid = 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:33:07.995 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T22:33:07.995 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T22:33:07.995 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T22:33:07.995 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T22:33:07.995 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-03-18T22:33:07.995 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T22:33:07.995 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T22:33:07.995 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 2024-03-18T22:33:07.995 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T22:33:07.995 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-18T22:33:07.995 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T22:33:07.995 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T22:33:07.996 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T22:33:07.996 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T22:33:07.996 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T22:33:07.996 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.207+0000 7ff2d30f5700 1 Processor -- start 2024-03-18T22:33:07.996 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.207+0000 7ff2d30f5700 1 -- start start 2024-03-18T22:33:07.996 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.207+0000 7ff2d30f5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2cc107b50 0x7ff2cc107f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:07.996 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.207+0000 7ff2d30f5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc108540 con 0x7ff2cc107b50 2024-03-18T22:33:07.996 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.207+0000 7ff2d0e91700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2cc107b50 0x7ff2cc107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:07.996 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.207+0000 7ff2d0e91700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2cc107b50 0x7ff2cc107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50302/0 (socket says 172.21.15.40:50302) 2024-03-18T22:33:07.996 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.207+0000 7ff2d0e91700 1 -- 172.21.15.40:0/4034959913 learned_addr learned my addr 172.21.15.40:0/4034959913 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:07.996 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.208+0000 7ff2d0e91700 1 -- 172.21.15.40:0/4034959913 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2cc108d50 con 0x7ff2cc107b50 2024-03-18T22:33:07.996 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.208+0000 7ff2d0e91700 1 --2- 172.21.15.40:0/4034959913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2cc107b50 0x7ff2cc107f70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff2c00098a0 tx=0x7ff2c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cfadc7c3f58d776 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:07.996 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.208+0000 7ff2cb7fe700 1 -- 172.21.15.40:0/4034959913 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2c0004160 con 0x7ff2cc107b50 2024-03-18T22:33:07.996 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.208+0000 7ff2cb7fe700 1 -- 172.21.15.40:0/4034959913 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7ff2c0004300 con 0x7ff2cc107b50 2024-03-18T22:33:07.996 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.208+0000 7ff2cb7fe700 1 -- 172.21.15.40:0/4034959913 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2c0011a80 con 0x7ff2cc107b50 2024-03-18T22:33:07.997 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.209+0000 7ff2d30f5700 1 -- 172.21.15.40:0/4034959913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2cc107b50 msgr2=0x7ff2cc107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:07.997 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.209+0000 7ff2d30f5700 1 --2- 172.21.15.40:0/4034959913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2cc107b50 0x7ff2cc107f70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff2c00098a0 tx=0x7ff2c0009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:07.997 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.209+0000 7ff2d30f5700 1 -- 172.21.15.40:0/4034959913 shutdown_connections 2024-03-18T22:33:07.997 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.209+0000 7ff2d30f5700 1 --2- 172.21.15.40:0/4034959913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2cc107b50 0x7ff2cc107f70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:07.997 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.209+0000 7ff2d30f5700 1 -- 172.21.15.40:0/4034959913 >> 172.21.15.40:0/4034959913 conn(0x7ff2cc1030f0 msgr2=0x7ff2cc105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:07.997 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.209+0000 7ff2d30f5700 1 -- 172.21.15.40:0/4034959913 shutdown_connections 2024-03-18T22:33:07.997 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.209+0000 7ff2d30f5700 1 -- 172.21.15.40:0/4034959913 wait complete. 2024-03-18T22:33:07.997 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.209+0000 7ff2d30f5700 1 Processor -- start 2024-03-18T22:33:07.997 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.209+0000 7ff2d30f5700 1 -- start start 2024-03-18T22:33:07.997 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.210+0000 7ff2d30f5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2cc107b50 0x7ff2cc07db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:07.997 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.210+0000 7ff2d30f5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc10c100 con 0x7ff2cc107b50 2024-03-18T22:33:07.997 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.210+0000 7ff2d0e91700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2cc107b50 0x7ff2cc07db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:07.997 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.210+0000 7ff2d0e91700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2cc107b50 0x7ff2cc07db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50312/0 (socket says 172.21.15.40:50312) 2024-03-18T22:33:07.998 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.210+0000 7ff2d0e91700 1 -- 172.21.15.40:0/2183396204 learned_addr learned my addr 172.21.15.40:0/2183396204 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:07.998 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.210+0000 7ff2d0e91700 1 -- 172.21.15.40:0/2183396204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2cc07e080 con 0x7ff2cc107b50 2024-03-18T22:33:07.998 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.210+0000 7ff2d0e91700 1 --2- 172.21.15.40:0/2183396204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2cc107b50 0x7ff2cc07db40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0000c00 tx=0x7ff2c0004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:07.998 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.210+0000 7ff2c9ffb700 1 -- 172.21.15.40:0/2183396204 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2c00039c0 con 0x7ff2cc107b50 2024-03-18T22:33:07.998 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.210+0000 7ff2c9ffb700 1 -- 172.21.15.40:0/2183396204 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7ff2c0003b60 con 0x7ff2cc107b50 2024-03-18T22:33:07.998 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.210+0000 7ff2d30f5700 1 -- 172.21.15.40:0/2183396204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2cc07c270 con 0x7ff2cc107b50 2024-03-18T22:33:07.998 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.210+0000 7ff2c9ffb700 1 -- 172.21.15.40:0/2183396204 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2c001a6f0 con 0x7ff2cc107b50 2024-03-18T22:33:07.998 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.211+0000 7ff2d30f5700 1 -- 172.21.15.40:0/2183396204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2cc07c7b0 con 0x7ff2cc107b50 2024-03-18T22:33:07.998 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.212+0000 7ff2c9ffb700 1 -- 172.21.15.40:0/2183396204 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7ff2c001a890 con 0x7ff2cc107b50 2024-03-18T22:33:07.998 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.212+0000 7ff2d30f5700 1 -- 172.21.15.40:0/2183396204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2cc063940 con 0x7ff2cc107b50 2024-03-18T22:33:07.998 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.212+0000 7ff2c9ffb700 1 --2- 172.21.15.40:0/2183396204 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7ff2bc038f20 0x7ff2bc03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:07.998 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.212+0000 7ff2c9ffb700 1 -- 172.21.15.40:0/2183396204 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff2c004d070 con 0x7ff2cc107b50 2024-03-18T22:33:07.998 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.212+0000 7ff2cbfff700 1 --2- 172.21.15.40:0/2183396204 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7ff2bc038f20 0x7ff2bc03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:07.998 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.212+0000 7ff2cbfff700 1 --2- 172.21.15.40:0/2183396204 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7ff2bc038f20 0x7ff2bc03b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8006fd0 tx=0x7ff2b8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:07.999 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.217+0000 7ff2c9ffb700 1 -- 172.21.15.40:0/2183396204 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff2c000b5a0 con 0x7ff2cc107b50 2024-03-18T22:33:07.999 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.422+0000 7ff2d30f5700 1 -- 172.21.15.40:0/2183396204 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7ff2cc07cfa0 con 0x7ff2cc107b50 2024-03-18T22:33:07.999 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.423+0000 7ff2c9ffb700 1 -- 172.21.15.40:0/2183396204 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+398 (secure 0 0 0) 0x7ff2c000b800 con 0x7ff2cc107b50 2024-03-18T22:33:07.999 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.425+0000 7ff2d30f5700 1 -- 172.21.15.40:0/2183396204 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7ff2bc038f20 msgr2=0x7ff2bc03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:07.999 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.425+0000 7ff2d30f5700 1 --2- 172.21.15.40:0/2183396204 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7ff2bc038f20 0x7ff2bc03b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8006fd0 tx=0x7ff2b8006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:07.999 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.425+0000 7ff2d30f5700 1 -- 172.21.15.40:0/2183396204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2cc107b50 msgr2=0x7ff2cc07db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:07.999 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.425+0000 7ff2d30f5700 1 --2- 172.21.15.40:0/2183396204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2cc107b50 0x7ff2cc07db40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0000c00 tx=0x7ff2c0004290 comp rx=0 tx=0).stop 2024-03-18T22:33:07.999 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.425+0000 7ff2d30f5700 1 -- 172.21.15.40:0/2183396204 shutdown_connections 2024-03-18T22:33:07.999 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.425+0000 7ff2d30f5700 1 --2- 172.21.15.40:0/2183396204 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7ff2bc038f20 0x7ff2bc03b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:07.999 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.425+0000 7ff2d30f5700 1 --2- 172.21.15.40:0/2183396204 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2cc107b50 0x7ff2cc07db40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:07.999 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.425+0000 7ff2d30f5700 1 -- 172.21.15.40:0/2183396204 >> 172.21.15.40:0/2183396204 conn(0x7ff2cc1030f0 msgr2=0x7ff2cc103c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:07.999 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.425+0000 7ff2d30f5700 1 -- 172.21.15.40:0/2183396204 shutdown_connections 2024-03-18T22:33:07.999 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:07.425+0000 7ff2d30f5700 1 -- 172.21.15.40:0/2183396204 wait complete. 2024-03-18T22:33:07.999 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling cephadm module... 2024-03-18T22:33:09.075 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:08 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1488683504' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-18T22:33:09.075 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:08 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: ignoring --setuser ceph since I am not root 2024-03-18T22:33:09.076 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:08 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: ignoring --setgroup ceph since I am not root 2024-03-18T22:33:09.359 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.409+0000 7fa2de79e700 1 Processor -- start 2024-03-18T22:33:09.359 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.409+0000 7fa2de79e700 1 -- start start 2024-03-18T22:33:09.359 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.409+0000 7fa2de79e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2d8106a80 0x7fa2d8106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:09.359 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.409+0000 7fa2de79e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d8107470 con 0x7fa2d8106a80 2024-03-18T22:33:09.359 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.409+0000 7fa2d7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2d8106a80 0x7fa2d8106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:09.359 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.410+0000 7fa2d7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2d8106a80 0x7fa2d8106ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50322/0 (socket says 172.21.15.40:50322) 2024-03-18T22:33:09.359 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.410+0000 7fa2d7fff700 1 -- 172.21.15.40:0/328807129 learned_addr learned my addr 172.21.15.40:0/328807129 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:09.359 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.410+0000 7fa2d7fff700 1 -- 172.21.15.40:0/328807129 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d8107c80 con 0x7fa2d8106a80 2024-03-18T22:33:09.359 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.410+0000 7fa2d7fff700 1 --2- 172.21.15.40:0/328807129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2d8106a80 0x7fa2d8106ea0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80098a0 tx=0x7fa2c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60668a5ed67405f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:09.359 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.410+0000 7fa2d6ffd700 1 -- 172.21.15.40:0/328807129 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2c8004160 con 0x7fa2d8106a80 2024-03-18T22:33:09.359 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.410+0000 7fa2d6ffd700 1 -- 172.21.15.40:0/328807129 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fa2c8004300 con 0x7fa2d8106a80 2024-03-18T22:33:09.359 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.410+0000 7fa2d6ffd700 1 -- 172.21.15.40:0/328807129 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2c8011a80 con 0x7fa2d8106a80 2024-03-18T22:33:09.360 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.411+0000 7fa2de79e700 1 -- 172.21.15.40:0/328807129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2d8106a80 msgr2=0x7fa2d8106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:09.360 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.411+0000 7fa2de79e700 1 --2- 172.21.15.40:0/328807129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2d8106a80 0x7fa2d8106ea0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80098a0 tx=0x7fa2c8009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:09.360 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.411+0000 7fa2de79e700 1 -- 172.21.15.40:0/328807129 shutdown_connections 2024-03-18T22:33:09.360 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.411+0000 7fa2de79e700 1 --2- 172.21.15.40:0/328807129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2d8106a80 0x7fa2d8106ea0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:09.360 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.411+0000 7fa2de79e700 1 -- 172.21.15.40:0/328807129 >> 172.21.15.40:0/328807129 conn(0x7fa2d8102000 msgr2=0x7fa2d8104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:09.360 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.411+0000 7fa2de79e700 1 -- 172.21.15.40:0/328807129 shutdown_connections 2024-03-18T22:33:09.360 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.411+0000 7fa2de79e700 1 -- 172.21.15.40:0/328807129 wait complete. 2024-03-18T22:33:09.360 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.411+0000 7fa2de79e700 1 Processor -- start 2024-03-18T22:33:09.360 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.411+0000 7fa2de79e700 1 -- start start 2024-03-18T22:33:09.360 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.412+0000 7fa2de79e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2d8106a80 0x7fa2d8191bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:09.360 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.412+0000 7fa2de79e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d810b030 con 0x7fa2d8106a80 2024-03-18T22:33:09.360 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.412+0000 7fa2d7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2d8106a80 0x7fa2d8191bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:09.360 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.412+0000 7fa2d7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2d8106a80 0x7fa2d8191bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50332/0 (socket says 172.21.15.40:50332) 2024-03-18T22:33:09.360 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.412+0000 7fa2d7fff700 1 -- 172.21.15.40:0/1488683504 learned_addr learned my addr 172.21.15.40:0/1488683504 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:09.360 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.412+0000 7fa2d7fff700 1 -- 172.21.15.40:0/1488683504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d8192130 con 0x7fa2d8106a80 2024-03-18T22:33:09.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.412+0000 7fa2d7fff700 1 --2- 172.21.15.40:0/1488683504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2d8106a80 0x7fa2d8191bf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8006af0 tx=0x7fa2c8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:09.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.413+0000 7fa2d57fa700 1 -- 172.21.15.40:0/1488683504 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2c8003930 con 0x7fa2d8106a80 2024-03-18T22:33:09.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.413+0000 7fa2d57fa700 1 -- 172.21.15.40:0/1488683504 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fa2c8003ad0 con 0x7fa2d8106a80 2024-03-18T22:33:09.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.413+0000 7fa2d57fa700 1 -- 172.21.15.40:0/1488683504 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2c80195e0 con 0x7fa2d8106a80 2024-03-18T22:33:09.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.413+0000 7fa2de79e700 1 -- 172.21.15.40:0/1488683504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2d8190320 con 0x7fa2d8106a80 2024-03-18T22:33:09.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.413+0000 7fa2de79e700 1 -- 172.21.15.40:0/1488683504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2d8190860 con 0x7fa2d8106a80 2024-03-18T22:33:09.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.414+0000 7fa2d57fa700 1 -- 172.21.15.40:0/1488683504 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fa2c8019890 con 0x7fa2d8106a80 2024-03-18T22:33:09.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.414+0000 7fa2de79e700 1 -- 172.21.15.40:0/1488683504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2d8063940 con 0x7fa2d8106a80 2024-03-18T22:33:09.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.414+0000 7fa2d57fa700 1 --2- 172.21.15.40:0/1488683504 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7fa2c0038ed0 0x7fa2c003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:09.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.414+0000 7fa2d57fa700 1 -- 172.21.15.40:0/1488683504 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa2c804d200 con 0x7fa2d8106a80 2024-03-18T22:33:09.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.414+0000 7fa2d77fe700 1 --2- 172.21.15.40:0/1488683504 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7fa2c0038ed0 0x7fa2c003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:09.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.414+0000 7fa2d77fe700 1 --2- 172.21.15.40:0/1488683504 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7fa2c0038ed0 0x7fa2c003b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc006fd0 tx=0x7fa2cc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:09.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.419+0000 7fa2d57fa700 1 -- 172.21.15.40:0/1488683504 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa2c801a430 con 0x7fa2d8106a80 2024-03-18T22:33:09.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.683+0000 7fa2de79e700 1 -- 172.21.15.40:0/1488683504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fa2d8195970 con 0x7fa2d8106a80 2024-03-18T22:33:09.361 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.779+0000 7fa2d57fa700 1 -- 172.21.15.40:0/1488683504 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fa2c8019b80 con 0x7fa2d8106a80 2024-03-18T22:33:09.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.781+0000 7fa2d57fa700 1 -- 172.21.15.40:0/1488683504 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fa2c8023390 con 0x7fa2d8106a80 2024-03-18T22:33:09.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.782+0000 7fa2de79e700 1 -- 172.21.15.40:0/1488683504 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7fa2c0038ed0 msgr2=0x7fa2c003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:09.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.782+0000 7fa2de79e700 1 --2- 172.21.15.40:0/1488683504 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7fa2c0038ed0 0x7fa2c003b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc006fd0 tx=0x7fa2cc006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:09.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.782+0000 7fa2de79e700 1 -- 172.21.15.40:0/1488683504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2d8106a80 msgr2=0x7fa2d8191bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:09.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.782+0000 7fa2de79e700 1 --2- 172.21.15.40:0/1488683504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2d8106a80 0x7fa2d8191bf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8006af0 tx=0x7fa2c8011b70 comp rx=0 tx=0).stop 2024-03-18T22:33:09.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.783+0000 7fa2de79e700 1 -- 172.21.15.40:0/1488683504 shutdown_connections 2024-03-18T22:33:09.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.783+0000 7fa2de79e700 1 --2- 172.21.15.40:0/1488683504 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7fa2c0038ed0 0x7fa2c003b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:09.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.783+0000 7fa2de79e700 1 --2- 172.21.15.40:0/1488683504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa2d8106a80 0x7fa2d8191bf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:09.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.783+0000 7fa2de79e700 1 -- 172.21.15.40:0/1488683504 >> 172.21.15.40:0/1488683504 conn(0x7fa2d8102000 msgr2=0x7fa2d8102ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:09.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.783+0000 7fa2de79e700 1 -- 172.21.15.40:0/1488683504 shutdown_connections 2024-03-18T22:33:09.362 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:08.783+0000 7fa2de79e700 1 -- 172.21.15.40:0/1488683504 wait complete. 2024-03-18T22:33:09.603 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:09 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:09.352+0000 7fe8ac669200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T22:33:09.603 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:09 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:09.452+0000 7fe8ac669200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T22:33:09.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:09 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1488683504' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-18T22:33:09.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:09 smithi040 ceph-mon[62643]: mgrmap e5: x(active, since 7s) 2024-03-18T22:33:10.386 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:10 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:10.128+0000 7fe8ac669200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T22:33:10.386 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:10 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:10.223+0000 7fe8ac669200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T22:33:10.387 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:10 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:10.368+0000 7fe8ac669200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T22:33:10.643 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:10 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:10.467+0000 7fe8ac669200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T22:33:10.643 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:10 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:10.553+0000 7fe8ac669200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T22:33:10.982 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-03-18T22:33:10.982 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-18T22:33:10.982 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T22:33:10.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T22:33:10.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T22:33:10.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-03-18T22:33:10.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.778+0000 7fe755ac3700 1 Processor -- start 2024-03-18T22:33:10.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.778+0000 7fe755ac3700 1 -- start start 2024-03-18T22:33:10.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.778+0000 7fe755ac3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe750107b40 0x7fe750107f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:10.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.778+0000 7fe755ac3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe750108530 con 0x7fe750107b40 2024-03-18T22:33:10.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.778+0000 7fe74f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe750107b40 0x7fe750107f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:10.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.778+0000 7fe74f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe750107b40 0x7fe750107f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39274/0 (socket says 172.21.15.40:39274) 2024-03-18T22:33:10.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.778+0000 7fe74f7fe700 1 -- 172.21.15.40:0/695591067 learned_addr learned my addr 172.21.15.40:0/695591067 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:10.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.778+0000 7fe74f7fe700 1 -- 172.21.15.40:0/695591067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe750108d40 con 0x7fe750107b40 2024-03-18T22:33:10.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.778+0000 7fe74f7fe700 1 --2- 172.21.15.40:0/695591067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe750107b40 0x7fe750107f60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe7400098a0 tx=0x7fe740009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed8eb7cab35cce6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:10.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.779+0000 7fe74e7fc700 1 -- 172.21.15.40:0/695591067 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe740004160 con 0x7fe750107b40 2024-03-18T22:33:10.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.779+0000 7fe74e7fc700 1 -- 172.21.15.40:0/695591067 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fe740004300 con 0x7fe750107b40 2024-03-18T22:33:10.983 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.779+0000 7fe74e7fc700 1 -- 172.21.15.40:0/695591067 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe740011a80 con 0x7fe750107b40 2024-03-18T22:33:10.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.779+0000 7fe755ac3700 1 -- 172.21.15.40:0/695591067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe750107b40 msgr2=0x7fe750107f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:10.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.779+0000 7fe755ac3700 1 --2- 172.21.15.40:0/695591067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe750107b40 0x7fe750107f60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe7400098a0 tx=0x7fe740009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:10.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.779+0000 7fe755ac3700 1 -- 172.21.15.40:0/695591067 shutdown_connections 2024-03-18T22:33:10.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.779+0000 7fe755ac3700 1 --2- 172.21.15.40:0/695591067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe750107b40 0x7fe750107f60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:10.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.779+0000 7fe755ac3700 1 -- 172.21.15.40:0/695591067 >> 172.21.15.40:0/695591067 conn(0x7fe750103140 msgr2=0x7fe750105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:10.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.780+0000 7fe755ac3700 1 -- 172.21.15.40:0/695591067 shutdown_connections 2024-03-18T22:33:10.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.780+0000 7fe755ac3700 1 -- 172.21.15.40:0/695591067 wait complete. 2024-03-18T22:33:10.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.780+0000 7fe755ac3700 1 Processor -- start 2024-03-18T22:33:10.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.780+0000 7fe755ac3700 1 -- start start 2024-03-18T22:33:10.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.780+0000 7fe755ac3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe750107b40 0x7fe75007db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:10.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.780+0000 7fe755ac3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe75010c0f0 con 0x7fe750107b40 2024-03-18T22:33:10.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.781+0000 7fe74f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe750107b40 0x7fe75007db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:10.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.781+0000 7fe74f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe750107b40 0x7fe75007db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39290/0 (socket says 172.21.15.40:39290) 2024-03-18T22:33:10.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.781+0000 7fe74f7fe700 1 -- 172.21.15.40:0/1541995125 learned_addr learned my addr 172.21.15.40:0/1541995125 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:10.984 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.781+0000 7fe74f7fe700 1 -- 172.21.15.40:0/1541995125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75007e080 con 0x7fe750107b40 2024-03-18T22:33:10.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.781+0000 7fe74f7fe700 1 --2- 172.21.15.40:0/1541995125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe750107b40 0x7fe75007db40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe740006af0 tx=0x7fe740011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:10.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.783+0000 7fe74cff9700 1 -- 172.21.15.40:0/1541995125 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe740003930 con 0x7fe750107b40 2024-03-18T22:33:10.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.783+0000 7fe74cff9700 1 -- 172.21.15.40:0/1541995125 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fe740003ad0 con 0x7fe750107b40 2024-03-18T22:33:10.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.783+0000 7fe74cff9700 1 -- 172.21.15.40:0/1541995125 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7400195e0 con 0x7fe750107b40 2024-03-18T22:33:10.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.783+0000 7fe755ac3700 1 -- 172.21.15.40:0/1541995125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe75007c270 con 0x7fe750107b40 2024-03-18T22:33:10.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.783+0000 7fe755ac3700 1 -- 172.21.15.40:0/1541995125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe75007c7b0 con 0x7fe750107b40 2024-03-18T22:33:10.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.784+0000 7fe74cff9700 1 -- 172.21.15.40:0/1541995125 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fe740019890 con 0x7fe750107b40 2024-03-18T22:33:10.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.784+0000 7fe74cff9700 1 --2- 172.21.15.40:0/1541995125 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7fe738038f20 0x7fe73803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:10.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.784+0000 7fe74cff9700 1 -- 172.21.15.40:0/1541995125 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe74004e1c0 con 0x7fe750107b40 2024-03-18T22:33:10.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.784+0000 7fe755ac3700 1 -- 172.21.15.40:0/1541995125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe750063940 con 0x7fe750107b40 2024-03-18T22:33:10.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.784+0000 7fe74effd700 1 -- 172.21.15.40:0/1541995125 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7fe738038f20 msgr2=0x7fe73803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1153335782 2024-03-18T22:33:10.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.784+0000 7fe74effd700 1 --2- 172.21.15.40:0/1541995125 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7fe738038f20 0x7fe73803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T22:33:10.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.789+0000 7fe74cff9700 1 -- 172.21.15.40:0/1541995125 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe74002d960 con 0x7fe750107b40 2024-03-18T22:33:10.985 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.985+0000 7fe74effd700 1 -- 172.21.15.40:0/1541995125 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7fe738038f20 msgr2=0x7fe73803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1153335782 2024-03-18T22:33:10.986 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:09.985+0000 7fe74effd700 1 --2- 172.21.15.40:0/1541995125 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7fe738038f20 0x7fe73803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T22:33:10.986 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:10.054+0000 7fe755ac3700 1 -- 172.21.15.40:0/1541995125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fe75007d0c0 con 0x7fe750107b40 2024-03-18T22:33:10.986 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:10.055+0000 7fe74cff9700 1 -- 172.21.15.40:0/1541995125 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fe74002d440 con 0x7fe750107b40 2024-03-18T22:33:10.986 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:10.057+0000 7fe755ac3700 1 -- 172.21.15.40:0/1541995125 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7fe738038f20 msgr2=0x7fe73803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T22:33:10.986 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:10.057+0000 7fe755ac3700 1 --2- 172.21.15.40:0/1541995125 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7fe738038f20 0x7fe73803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:10.986 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:10.057+0000 7fe755ac3700 1 -- 172.21.15.40:0/1541995125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe750107b40 msgr2=0x7fe75007db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:10.986 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:10.057+0000 7fe755ac3700 1 --2- 172.21.15.40:0/1541995125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe750107b40 0x7fe75007db40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe740006af0 tx=0x7fe740011b70 comp rx=0 tx=0).stop 2024-03-18T22:33:10.986 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:10.057+0000 7fe755ac3700 1 -- 172.21.15.40:0/1541995125 shutdown_connections 2024-03-18T22:33:10.986 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:10.057+0000 7fe755ac3700 1 --2- 172.21.15.40:0/1541995125 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7fe738038f20 0x7fe73803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:10.986 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:10.057+0000 7fe755ac3700 1 --2- 172.21.15.40:0/1541995125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe750107b40 0x7fe75007db40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:10.986 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:10.057+0000 7fe755ac3700 1 -- 172.21.15.40:0/1541995125 >> 172.21.15.40:0/1541995125 conn(0x7fe750103140 msgr2=0x7fe750103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:10.986 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:10.057+0000 7fe755ac3700 1 -- 172.21.15.40:0/1541995125 shutdown_connections 2024-03-18T22:33:10.986 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:10.057+0000 7fe755ac3700 1 -- 172.21.15.40:0/1541995125 wait complete. 2024-03-18T22:33:10.986 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for the mgr to restart... 2024-03-18T22:33:10.986 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr epoch 5... 2024-03-18T22:33:11.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:10 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1541995125' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T22:33:11.038 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:10 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:10.824+0000 7fe8ac669200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T22:33:11.038 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:10 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:10.937+0000 7fe8ac669200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T22:33:11.038 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:11 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:11.037+0000 7fe8ac669200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T22:33:11.323 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:11 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:11.141+0000 7fe8ac669200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T22:33:11.584 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:11 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:11.323+0000 7fe8ac669200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T22:33:12.643 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:12 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:12.315+0000 7fe8ac669200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T22:33:12.644 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:12 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:12.449+0000 7fe8ac669200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T22:33:13.089 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:12 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:12.792+0000 7fe8ac669200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T22:33:13.089 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:12 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:12.887+0000 7fe8ac669200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T22:33:13.089 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:13 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:13.088+0000 7fe8ac669200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T22:33:13.393 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:13 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:13.279+0000 7fe8ac669200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T22:33:13.839 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:13 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:13.577+0000 7fe8ac669200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T22:33:13.839 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:13 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:13.838+0000 7fe8ac669200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T22:33:14.143 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:14 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:14.080+0000 7fe8ac669200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T22:33:15.393 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:14 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:14.908+0000 7fe8ac669200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T22:33:15.814 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:15.490+0000 7fe8ac669200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T22:33:15.814 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:15.591+0000 7fe8ac669200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T22:33:15.814 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:15.813+0000 7fe8ac669200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T22:33:16.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-mon[62643]: Active manager daemon x restarted 2024-03-18T22:33:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-mon[62643]: Activating manager daemon x 2024-03-18T22:33:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-mon[62643]: osdmap e2: 0 total, 0 up, 0 in 2024-03-18T22:33:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-mon[62643]: mgrmap e6: x(active, starting, since 0.00337392s) 2024-03-18T22:33:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:33:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T22:33:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T22:33:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T22:33:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T22:33:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-mon[62643]: Manager daemon x is now available 2024-03-18T22:33:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' 2024-03-18T22:33:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' 2024-03-18T22:33:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:33:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:15 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:33:17.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:16 smithi040 ceph-mon[62643]: Found migration_current of "None". Setting to last migration. 2024-03-18T22:33:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:16 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:33:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:16 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T22:33:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:16 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T22:33:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:16 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' 2024-03-18T22:33:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:16 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' 2024-03-18T22:33:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:16 smithi040 ceph-mon[62643]: mgrmap e7: x(active, since 1.00835s) 2024-03-18T22:33:17.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-03-18T22:33:17.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-18T22:33:17.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T22:33:17.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-03-18T22:33:17.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.378+0000 7feec0987700 1 Processor -- start 2024-03-18T22:33:17.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.378+0000 7feec0987700 1 -- start start 2024-03-18T22:33:17.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.378+0000 7feec0987700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc106a60 0x7feebc106e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:17.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.378+0000 7feec0987700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feebc107450 con 0x7feebc106a60 2024-03-18T22:33:17.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.378+0000 7feeba59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc106a60 0x7feebc106e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:17.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.378+0000 7feeba59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc106a60 0x7feebc106e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39302/0 (socket says 172.21.15.40:39302) 2024-03-18T22:33:17.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.378+0000 7feeba59c700 1 -- 172.21.15.40:0/1693567754 learned_addr learned my addr 172.21.15.40:0/1693567754 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:17.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.379+0000 7feeba59c700 1 -- 172.21.15.40:0/1693567754 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feebc107c60 con 0x7feebc106a60 2024-03-18T22:33:17.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.379+0000 7feeba59c700 1 --2- 172.21.15.40:0/1693567754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc106a60 0x7feebc106e80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feeac0098a0 tx=0x7feeac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1398327c8bd62d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:17.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.379+0000 7feeb959a700 1 -- 172.21.15.40:0/1693567754 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feeac004160 con 0x7feebc106a60 2024-03-18T22:33:17.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.379+0000 7feeb959a700 1 -- 172.21.15.40:0/1693567754 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7feeac004300 con 0x7feebc106a60 2024-03-18T22:33:17.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.379+0000 7feeb959a700 1 -- 172.21.15.40:0/1693567754 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feeac011a80 con 0x7feebc106a60 2024-03-18T22:33:17.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.380+0000 7feec0987700 1 -- 172.21.15.40:0/1693567754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc106a60 msgr2=0x7feebc106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:17.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.380+0000 7feec0987700 1 --2- 172.21.15.40:0/1693567754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc106a60 0x7feebc106e80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feeac0098a0 tx=0x7feeac009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:17.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.380+0000 7feec0987700 1 -- 172.21.15.40:0/1693567754 shutdown_connections 2024-03-18T22:33:17.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.380+0000 7feec0987700 1 --2- 172.21.15.40:0/1693567754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc106a60 0x7feebc106e80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:17.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.380+0000 7feec0987700 1 -- 172.21.15.40:0/1693567754 >> 172.21.15.40:0/1693567754 conn(0x7feebc102020 msgr2=0x7feebc104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:17.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.380+0000 7feec0987700 1 -- 172.21.15.40:0/1693567754 shutdown_connections 2024-03-18T22:33:17.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.380+0000 7feec0987700 1 -- 172.21.15.40:0/1693567754 wait complete. 2024-03-18T22:33:17.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.380+0000 7feec0987700 1 Processor -- start 2024-03-18T22:33:17.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.381+0000 7feec0987700 1 -- start start 2024-03-18T22:33:17.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.381+0000 7feec0987700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc106a60 0x7feebc075580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:17.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.381+0000 7feec0987700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feebc10b010 con 0x7feebc106a60 2024-03-18T22:33:17.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.381+0000 7feeba59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc106a60 0x7feebc075580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:17.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.381+0000 7feeba59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc106a60 0x7feebc075580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39304/0 (socket says 172.21.15.40:39304) 2024-03-18T22:33:17.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.381+0000 7feeba59c700 1 -- 172.21.15.40:0/2477063998 learned_addr learned my addr 172.21.15.40:0/2477063998 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.381+0000 7feeba59c700 1 -- 172.21.15.40:0/2477063998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feebc19e7e0 con 0x7feebc106a60 2024-03-18T22:33:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.381+0000 7feeba59c700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc106a60 0x7feebc075580 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7feeac000c00 tx=0x7feeac004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.381+0000 7feeab7fe700 1 -- 172.21.15.40:0/2477063998 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feeac0039c0 con 0x7feebc106a60 2024-03-18T22:33:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.381+0000 7feeab7fe700 1 -- 172.21.15.40:0/2477063998 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7feeac003b60 con 0x7feebc106a60 2024-03-18T22:33:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.381+0000 7feec0987700 1 -- 172.21.15.40:0/2477063998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feebc19e9b0 con 0x7feebc106a60 2024-03-18T22:33:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.382+0000 7feeab7fe700 1 -- 172.21.15.40:0/2477063998 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feeac01a6f0 con 0x7feebc106a60 2024-03-18T22:33:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.382+0000 7feec0987700 1 -- 172.21.15.40:0/2477063998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feebc19edd0 con 0x7feebc106a60 2024-03-18T22:33:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.382+0000 7feeab7fe700 1 -- 172.21.15.40:0/2477063998 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7feeac01a9a0 con 0x7feebc106a60 2024-03-18T22:33:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.383+0000 7feeab7fe700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7feea4038f20 0x7feea403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.383+0000 7feeab7fe700 1 -- 172.21.15.40:0/2477063998 --> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7feea403baf0 con 0x7feea4038f20 2024-03-18T22:33:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.383+0000 7feeab7fe700 1 -- 172.21.15.40:0/2477063998 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7feeac04d080 con 0x7feebc106a60 2024-03-18T22:33:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.383+0000 7feeb9d9b700 1 -- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7feea4038f20 msgr2=0x7feea403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1153335782 2024-03-18T22:33:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.383+0000 7feeb9d9b700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7feea4038f20 0x7feea403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T22:33:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.584+0000 7feeb9d9b700 1 -- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7feea4038f20 msgr2=0x7feea403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1153335782 2024-03-18T22:33:17.402 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.584+0000 7feeb9d9b700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7feea4038f20 0x7feea403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.985+0000 7feeb9d9b700 1 -- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7feea4038f20 msgr2=0x7feea403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1153335782 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:11.985+0000 7feeb9d9b700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7feea4038f20 0x7feea403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:12.786+0000 7feeb9d9b700 1 -- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7feea4038f20 msgr2=0x7feea403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1153335782 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:12.786+0000 7feeb9d9b700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7feea4038f20 0x7feea403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:14.387+0000 7feeb9d9b700 1 -- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7feea4038f20 msgr2=0x7feea403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/1153335782 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:14.387+0000 7feeb9d9b700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7feea4038f20 0x7feea403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:15.828+0000 7feeab7fe700 1 -- 172.21.15.40:0/2477063998 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7feeac02d910 con 0x7feebc106a60 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:15.828+0000 7feeab7fe700 1 -- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7feea4038f20 msgr2=0x7feea403b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:15.828+0000 7feeab7fe700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7feea4038f20 0x7feea403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.832+0000 7feeab7fe700 1 -- 172.21.15.40:0/2477063998 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7feeac04f0b0 con 0x7feebc106a60 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.832+0000 7feeab7fe700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7feea403c810 0x7feea403ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.832+0000 7feeab7fe700 1 -- 172.21.15.40:0/2477063998 --> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7feeac00f500 con 0x7feea403c810 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.833+0000 7feeb9d9b700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7feea403c810 0x7feea403ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.833+0000 7feeb9d9b700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7feea403c810 0x7feea403ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7feeb0003a10 tx=0x7feeb0009290 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.834+0000 7feeab7fe700 1 -- 172.21.15.40:0/2477063998 <== mgr.14116 v2:172.21.15.40:6800/3876208039 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7feeac00f500 con 0x7feea403c810 2024-03-18T22:33:17.403 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.846+0000 7feec0987700 1 -- 172.21.15.40:0/2477063998 --> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7feebc063940 con 0x7feea403c810 2024-03-18T22:33:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.846+0000 7feeab7fe700 1 -- 172.21.15.40:0/2477063998 <== mgr.14116 v2:172.21.15.40:6800/3876208039 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7feebc063940 con 0x7feea403c810 2024-03-18T22:33:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.846+0000 7feec0987700 1 -- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7feea403c810 msgr2=0x7feea403ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.846+0000 7feec0987700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7feea403c810 0x7feea403ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7feeb0003a10 tx=0x7feeb0009290 comp rx=0 tx=0).stop 2024-03-18T22:33:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.846+0000 7feec0987700 1 -- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc106a60 msgr2=0x7feebc075580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.846+0000 7feec0987700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc106a60 0x7feebc075580 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7feeac000c00 tx=0x7feeac004290 comp rx=0 tx=0).stop 2024-03-18T22:33:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.847+0000 7feec0987700 1 -- 172.21.15.40:0/2477063998 shutdown_connections 2024-03-18T22:33:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.847+0000 7feec0987700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7feea403c810 0x7feea403ec00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.847+0000 7feec0987700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feebc106a60 0x7feebc075580 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.847+0000 7feec0987700 1 --2- 172.21.15.40:0/2477063998 >> [v2:172.21.15.40:6800/1153335782,v1:172.21.15.40:6801/1153335782] conn(0x7feea4038f20 0x7feea403b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.847+0000 7feec0987700 1 -- 172.21.15.40:0/2477063998 >> 172.21.15.40:0/2477063998 conn(0x7feebc102020 msgr2=0x7feebc102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.847+0000 7feec0987700 1 -- 172.21.15.40:0/2477063998 shutdown_connections 2024-03-18T22:33:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:16.847+0000 7feec0987700 1 -- 172.21.15.40:0/2477063998 wait complete. 2024-03-18T22:33:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:mgr epoch 5 is available 2024-03-18T22:33:17.404 INFO:teuthology.orchestra.run.smithi040.stdout:Setting orchestrator backend to cephadm... 2024-03-18T22:33:18.034 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:17 smithi040 ceph-mon[62643]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T22:33:18.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:17 smithi040 ceph-mon[62643]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T22:33:18.035 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:17 smithi040 ceph-mon[62643]: [18/Mar/2024:22:33:17] ENGINE Bus STARTING 2024-03-18T22:33:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.792+0000 7f134cc05700 1 Processor -- start 2024-03-18T22:33:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.792+0000 7f134cc05700 1 -- start start 2024-03-18T22:33:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.792+0000 7f134cc05700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348107b70 0x7f1348107f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.792+0000 7f134cc05700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1348108560 con 0x7f1348107b70 2024-03-18T22:33:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.792+0000 7f134659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348107b70 0x7f1348107f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.793+0000 7f134659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348107b70 0x7f1348107f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39390/0 (socket says 172.21.15.40:39390) 2024-03-18T22:33:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.793+0000 7f134659c700 1 -- 172.21.15.40:0/2481908932 learned_addr learned my addr 172.21.15.40:0/2481908932 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.793+0000 7f134659c700 1 -- 172.21.15.40:0/2481908932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1348108d70 con 0x7f1348107b70 2024-03-18T22:33:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.793+0000 7f134659c700 1 --2- 172.21.15.40:0/2481908932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348107b70 0x7f1348107f90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f13300098a0 tx=0x7f1330009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66f814f9b6736e7c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.793+0000 7f134559a700 1 -- 172.21.15.40:0/2481908932 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1330004160 con 0x7f1348107b70 2024-03-18T22:33:18.563 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.793+0000 7f134559a700 1 -- 172.21.15.40:0/2481908932 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1330004300 con 0x7f1348107b70 2024-03-18T22:33:18.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.793+0000 7f134559a700 1 -- 172.21.15.40:0/2481908932 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1330011a80 con 0x7f1348107b70 2024-03-18T22:33:18.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.794+0000 7f134cc05700 1 -- 172.21.15.40:0/2481908932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348107b70 msgr2=0x7f1348107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:18.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.794+0000 7f134cc05700 1 --2- 172.21.15.40:0/2481908932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348107b70 0x7f1348107f90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f13300098a0 tx=0x7f1330009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:18.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.794+0000 7f134cc05700 1 -- 172.21.15.40:0/2481908932 shutdown_connections 2024-03-18T22:33:18.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.794+0000 7f134cc05700 1 --2- 172.21.15.40:0/2481908932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348107b70 0x7f1348107f90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:18.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.794+0000 7f134cc05700 1 -- 172.21.15.40:0/2481908932 >> 172.21.15.40:0/2481908932 conn(0x7f13481030f0 msgr2=0x7f1348105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:18.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.794+0000 7f134cc05700 1 -- 172.21.15.40:0/2481908932 shutdown_connections 2024-03-18T22:33:18.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.794+0000 7f134cc05700 1 -- 172.21.15.40:0/2481908932 wait complete. 2024-03-18T22:33:18.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.795+0000 7f134cc05700 1 Processor -- start 2024-03-18T22:33:18.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.795+0000 7f134cc05700 1 -- start start 2024-03-18T22:33:18.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.795+0000 7f134cc05700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348107b70 0x7f134819bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:18.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.795+0000 7f134cc05700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f134810c120 con 0x7f1348107b70 2024-03-18T22:33:18.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.795+0000 7f134659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348107b70 0x7f134819bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:18.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.795+0000 7f134659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348107b70 0x7f134819bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39402/0 (socket says 172.21.15.40:39402) 2024-03-18T22:33:18.564 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.795+0000 7f134659c700 1 -- 172.21.15.40:0/4199994514 learned_addr learned my addr 172.21.15.40:0/4199994514 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.795+0000 7f134659c700 1 -- 172.21.15.40:0/4199994514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134819c060 con 0x7f1348107b70 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.796+0000 7f134659c700 1 --2- 172.21.15.40:0/4199994514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348107b70 0x7f134819bb20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1330006af0 tx=0x7f1330011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.796+0000 7f133f7fe700 1 -- 172.21.15.40:0/4199994514 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1330003930 con 0x7f1348107b70 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.796+0000 7f133f7fe700 1 -- 172.21.15.40:0/4199994514 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1330003ad0 con 0x7f1348107b70 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.796+0000 7f133f7fe700 1 -- 172.21.15.40:0/4199994514 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13300195e0 con 0x7f1348107b70 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.796+0000 7f134cc05700 1 -- 172.21.15.40:0/4199994514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134819c2f0 con 0x7f1348107b70 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.796+0000 7f134cc05700 1 -- 172.21.15.40:0/4199994514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f134819efe0 con 0x7f1348107b70 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.797+0000 7f133f7fe700 1 -- 172.21.15.40:0/4199994514 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7f133001a430 con 0x7f1348107b70 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.797+0000 7f134cc05700 1 -- 172.21.15.40:0/4199994514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1348063940 con 0x7f1348107b70 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.797+0000 7f133f7fe700 1 --2- 172.21.15.40:0/4199994514 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f1334038da0 0x7f133403b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.797+0000 7f133f7fe700 1 -- 172.21.15.40:0/4199994514 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f133004d130 con 0x7f1348107b70 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.797+0000 7f1345d9b700 1 --2- 172.21.15.40:0/4199994514 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f1334038da0 0x7f133403b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.798+0000 7f1345d9b700 1 --2- 172.21.15.40:0/4199994514 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f1334038da0 0x7f133403b260 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1338006fd0 tx=0x7f1338006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.802+0000 7f133f7fe700 1 -- 172.21.15.40:0/4199994514 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1330023390 con 0x7f1348107b70 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:17.882+0000 7f133f7fe700 1 -- 172.21.15.40:0/4199994514 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f133002dc90 con 0x7f1348107b70 2024-03-18T22:33:18.565 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.023+0000 7f134cc05700 1 -- 172.21.15.40:0/4199994514 --> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f1348104360 con 0x7f1334038da0 2024-03-18T22:33:18.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.031+0000 7f133f7fe700 1 -- 172.21.15.40:0/4199994514 <== mgr.14116 v2:172.21.15.40:6800/3876208039 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1348104360 con 0x7f1334038da0 2024-03-18T22:33:18.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.034+0000 7f134cc05700 1 -- 172.21.15.40:0/4199994514 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f1334038da0 msgr2=0x7f133403b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:18.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.034+0000 7f134cc05700 1 --2- 172.21.15.40:0/4199994514 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f1334038da0 0x7f133403b260 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1338006fd0 tx=0x7f1338006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:18.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.034+0000 7f134cc05700 1 -- 172.21.15.40:0/4199994514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348107b70 msgr2=0x7f134819bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:18.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.034+0000 7f134cc05700 1 --2- 172.21.15.40:0/4199994514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348107b70 0x7f134819bb20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1330006af0 tx=0x7f1330011b70 comp rx=0 tx=0).stop 2024-03-18T22:33:18.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.034+0000 7f134cc05700 1 -- 172.21.15.40:0/4199994514 shutdown_connections 2024-03-18T22:33:18.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.034+0000 7f134cc05700 1 --2- 172.21.15.40:0/4199994514 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f1334038da0 0x7f133403b260 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:18.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.034+0000 7f134cc05700 1 --2- 172.21.15.40:0/4199994514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1348107b70 0x7f134819bb20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:18.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.034+0000 7f134cc05700 1 -- 172.21.15.40:0/4199994514 >> 172.21.15.40:0/4199994514 conn(0x7f13481030f0 msgr2=0x7f1348103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:18.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.034+0000 7f134cc05700 1 -- 172.21.15.40:0/4199994514 shutdown_connections 2024-03-18T22:33:18.566 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.034+0000 7f134cc05700 1 -- 172.21.15.40:0/4199994514 wait complete. 2024-03-18T22:33:19.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:18 smithi040 ceph-mon[62643]: mgrmap e8: x(active, since 2s) 2024-03-18T22:33:19.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:18 smithi040 ceph-mon[62643]: [18/Mar/2024:22:33:17] ENGINE Serving on https://172.21.15.40:7150 2024-03-18T22:33:19.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:18 smithi040 ceph-mon[62643]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:33:19.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:18 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' 2024-03-18T22:33:19.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:18 smithi040 ceph-mon[62643]: [18/Mar/2024:22:33:18] ENGINE Serving on http://172.21.15.40:8765 2024-03-18T22:33:19.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:18 smithi040 ceph-mon[62643]: [18/Mar/2024:22:33:18] ENGINE Bus STARTED 2024-03-18T22:33:19.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:18 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:33:19.711 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-18T22:33:19.711 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.992+0000 7f5ec8e10700 1 Processor -- start 2024-03-18T22:33:19.711 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.992+0000 7f5ec8e10700 1 -- start start 2024-03-18T22:33:19.711 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.993+0000 7f5ec8e10700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ec41085d0 0x7f5ec410a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:19.711 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.993+0000 7f5ec8e10700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec40759c0 con 0x7f5ec41085d0 2024-03-18T22:33:19.711 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.993+0000 7f5ec259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ec41085d0 0x7f5ec410a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:19.711 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.993+0000 7f5ec259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ec41085d0 0x7f5ec410a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39032/0 (socket says 172.21.15.40:39032) 2024-03-18T22:33:19.711 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.993+0000 7f5ec259c700 1 -- 172.21.15.40:0/2187520568 learned_addr learned my addr 172.21.15.40:0/2187520568 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:19.711 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.993+0000 7f5ec259c700 1 -- 172.21.15.40:0/2187520568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ec410af00 con 0x7f5ec41085d0 2024-03-18T22:33:19.711 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.993+0000 7f5ec259c700 1 --2- 172.21.15.40:0/2187520568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ec41085d0 0x7f5ec410a9c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5eb40098a0 tx=0x7f5eb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee34cc775fe80785 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:19.711 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.993+0000 7f5ec159a700 1 -- 172.21.15.40:0/2187520568 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5eb4004160 con 0x7f5ec41085d0 2024-03-18T22:33:19.711 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.993+0000 7f5ec159a700 1 -- 172.21.15.40:0/2187520568 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f5eb4004300 con 0x7f5ec41085d0 2024-03-18T22:33:19.711 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.993+0000 7f5ec159a700 1 -- 172.21.15.40:0/2187520568 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5eb4011a80 con 0x7f5ec41085d0 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.994+0000 7f5ec8e10700 1 -- 172.21.15.40:0/2187520568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ec41085d0 msgr2=0x7f5ec410a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.994+0000 7f5ec8e10700 1 --2- 172.21.15.40:0/2187520568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ec41085d0 0x7f5ec410a9c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5eb40098a0 tx=0x7f5eb4009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.994+0000 7f5ec8e10700 1 -- 172.21.15.40:0/2187520568 shutdown_connections 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.994+0000 7f5ec8e10700 1 --2- 172.21.15.40:0/2187520568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ec41085d0 0x7f5ec410a9c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.994+0000 7f5ec8e10700 1 -- 172.21.15.40:0/2187520568 >> 172.21.15.40:0/2187520568 conn(0x7f5ec4102020 msgr2=0x7f5ec4104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.994+0000 7f5ec8e10700 1 -- 172.21.15.40:0/2187520568 shutdown_connections 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.994+0000 7f5ec8e10700 1 -- 172.21.15.40:0/2187520568 wait complete. 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.995+0000 7f5ec8e10700 1 Processor -- start 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.995+0000 7f5ec8e10700 1 -- start start 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.995+0000 7f5ec8e10700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ec41085d0 0x7f5ec419ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.995+0000 7f5ec8e10700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec410bed0 con 0x7f5ec41085d0 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.995+0000 7f5ec259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ec41085d0 0x7f5ec419ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.995+0000 7f5ec259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ec41085d0 0x7f5ec419ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39048/0 (socket says 172.21.15.40:39048) 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.995+0000 7f5ec259c700 1 -- 172.21.15.40:0/2320767215 learned_addr learned my addr 172.21.15.40:0/2320767215 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.996+0000 7f5ec259c700 1 -- 172.21.15.40:0/2320767215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ec419bf90 con 0x7f5ec41085d0 2024-03-18T22:33:19.712 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.996+0000 7f5ec259c700 1 --2- 172.21.15.40:0/2320767215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ec41085d0 0x7f5ec419ba50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4006af0 tx=0x7f5eb4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:19.713 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.996+0000 7f5eb37fe700 1 -- 172.21.15.40:0/2320767215 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5eb4003930 con 0x7f5ec41085d0 2024-03-18T22:33:19.713 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.996+0000 7f5eb37fe700 1 -- 172.21.15.40:0/2320767215 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f5eb4003ad0 con 0x7f5ec41085d0 2024-03-18T22:33:19.713 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.996+0000 7f5ec8e10700 1 -- 172.21.15.40:0/2320767215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ec419c220 con 0x7f5ec41085d0 2024-03-18T22:33:19.713 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.996+0000 7f5eb37fe700 1 -- 172.21.15.40:0/2320767215 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5eb40195e0 con 0x7f5ec41085d0 2024-03-18T22:33:19.713 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.996+0000 7f5ec8e10700 1 -- 172.21.15.40:0/2320767215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ec419ef10 con 0x7f5ec41085d0 2024-03-18T22:33:19.713 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.997+0000 7f5eb37fe700 1 -- 172.21.15.40:0/2320767215 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f5eb40198e0 con 0x7f5ec41085d0 2024-03-18T22:33:19.713 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.997+0000 7f5ec8e10700 1 -- 172.21.15.40:0/2320767215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ec4063940 con 0x7f5ec41085d0 2024-03-18T22:33:19.713 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.997+0000 7f5eb37fe700 1 --2- 172.21.15.40:0/2320767215 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f5eac038ed0 0x7f5eac03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:19.713 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.997+0000 7f5eb37fe700 1 -- 172.21.15.40:0/2320767215 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5eb404d5c0 con 0x7f5ec41085d0 2024-03-18T22:33:19.713 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.997+0000 7f5ec1d9b700 1 --2- 172.21.15.40:0/2320767215 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f5eac038ed0 0x7f5eac03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:19.713 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:18.998+0000 7f5ec1d9b700 1 --2- 172.21.15.40:0/2320767215 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f5eac038ed0 0x7f5eac03b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8006fd0 tx=0x7f5eb8006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:19.713 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:19.002+0000 7f5eb37fe700 1 -- 172.21.15.40:0/2320767215 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5eb4019cd0 con 0x7f5ec41085d0 2024-03-18T22:33:19.713 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:19.208+0000 7f5ec8e10700 1 -- 172.21.15.40:0/2320767215 --> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f5ec41031c0 con 0x7f5eac038ed0 2024-03-18T22:33:19.713 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:19.208+0000 7f5eb37fe700 1 -- 172.21.15.40:0/2320767215 <== mgr.14116 v2:172.21.15.40:6800/3876208039 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f5ec41031c0 con 0x7f5eac038ed0 2024-03-18T22:33:19.714 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:19.210+0000 7f5ec8e10700 1 -- 172.21.15.40:0/2320767215 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f5eac038ed0 msgr2=0x7f5eac03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:19.714 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:19.210+0000 7f5ec8e10700 1 --2- 172.21.15.40:0/2320767215 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f5eac038ed0 0x7f5eac03b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8006fd0 tx=0x7f5eb8006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:19.714 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:19.210+0000 7f5ec8e10700 1 -- 172.21.15.40:0/2320767215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ec41085d0 msgr2=0x7f5ec419ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:19.714 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:19.210+0000 7f5ec8e10700 1 --2- 172.21.15.40:0/2320767215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ec41085d0 0x7f5ec419ba50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4006af0 tx=0x7f5eb4011b70 comp rx=0 tx=0).stop 2024-03-18T22:33:19.714 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:19.210+0000 7f5ec8e10700 1 -- 172.21.15.40:0/2320767215 shutdown_connections 2024-03-18T22:33:19.714 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:19.210+0000 7f5ec8e10700 1 --2- 172.21.15.40:0/2320767215 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f5eac038ed0 0x7f5eac03b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:19.714 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:19.210+0000 7f5ec8e10700 1 --2- 172.21.15.40:0/2320767215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ec41085d0 0x7f5ec419ba50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:19.714 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:19.210+0000 7f5ec8e10700 1 -- 172.21.15.40:0/2320767215 >> 172.21.15.40:0/2320767215 conn(0x7f5ec4102020 msgr2=0x7f5ec4102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:19.714 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:19.210+0000 7f5ec8e10700 1 -- 172.21.15.40:0/2320767215 shutdown_connections 2024-03-18T22:33:19.714 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:19.210+0000 7f5ec8e10700 1 -- 172.21.15.40:0/2320767215 wait complete. 2024-03-18T22:33:19.714 INFO:teuthology.orchestra.run.smithi040.stdout:Generating ssh key... 2024-03-18T22:33:20.083 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:19 smithi040 ceph-mon[62643]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:33:21.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:20 smithi040 ceph-mon[62643]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:33:21.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:20 smithi040 ceph-mon[62643]: Generating ssh key... 2024-03-18T22:33:21.144 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: Generating public/private rsa key pair. 2024-03-18T22:33:21.144 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: Your identification has been saved in /tmp/tmpde8d6_r0/key. 2024-03-18T22:33:21.144 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: Your public key has been saved in /tmp/tmpde8d6_r0/key.pub. 2024-03-18T22:33:21.144 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: The key fingerprint is: 2024-03-18T22:33:21.144 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: SHA256:xm7sCo/jZkFfRpqktb07PX9WJePi/cZpVkg2CF7eN+o ceph-410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:33:21.144 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: The key's randomart image is: 2024-03-18T22:33:21.144 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: +---[RSA 3072]----+ 2024-03-18T22:33:21.144 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: | | 2024-03-18T22:33:21.144 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: | o . . . | 2024-03-18T22:33:21.145 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: | + * . + o | 2024-03-18T22:33:21.145 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: | o +.+ . o B.o| 2024-03-18T22:33:21.145 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: | . . oS. +.*o| 2024-03-18T22:33:21.145 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: | . .+. ..o o| 2024-03-18T22:33:21.145 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: | .. +o ..o oo| 2024-03-18T22:33:21.145 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: | ++ oo o .E++o| 2024-03-18T22:33:21.145 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: | +o.o... o.ooo.| 2024-03-18T22:33:21.145 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:21 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: +----[SHA256]-----+ 2024-03-18T22:33:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.186+0000 7fd3bb446700 1 Processor -- start 2024-03-18T22:33:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.187+0000 7fd3bb446700 1 -- start start 2024-03-18T22:33:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.187+0000 7fd3bb446700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b4107b50 0x7fd3b4107f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.187+0000 7fd3bb446700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b4108540 con 0x7fd3b4107b50 2024-03-18T22:33:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.187+0000 7fd3b91e2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b4107b50 0x7fd3b4107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.187+0000 7fd3b91e2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b4107b50 0x7fd3b4107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39050/0 (socket says 172.21.15.40:39050) 2024-03-18T22:33:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.187+0000 7fd3b91e2700 1 -- 172.21.15.40:0/3673366109 learned_addr learned my addr 172.21.15.40:0/3673366109 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.187+0000 7fd3b91e2700 1 -- 172.21.15.40:0/3673366109 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b4108d50 con 0x7fd3b4107b50 2024-03-18T22:33:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.187+0000 7fd3b91e2700 1 --2- 172.21.15.40:0/3673366109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b4107b50 0x7fd3b4107f70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80098a0 tx=0x7fd3a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d52b5080c8d76c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.187+0000 7fd3a7fff700 1 -- 172.21.15.40:0/3673366109 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3a8004160 con 0x7fd3b4107b50 2024-03-18T22:33:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.188+0000 7fd3a7fff700 1 -- 172.21.15.40:0/3673366109 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fd3a8004300 con 0x7fd3b4107b50 2024-03-18T22:33:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.188+0000 7fd3a7fff700 1 -- 172.21.15.40:0/3673366109 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3a8011a80 con 0x7fd3b4107b50 2024-03-18T22:33:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.188+0000 7fd3bb446700 1 -- 172.21.15.40:0/3673366109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b4107b50 msgr2=0x7fd3b4107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.188+0000 7fd3bb446700 1 --2- 172.21.15.40:0/3673366109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b4107b50 0x7fd3b4107f70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80098a0 tx=0x7fd3a8009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.188+0000 7fd3bb446700 1 -- 172.21.15.40:0/3673366109 shutdown_connections 2024-03-18T22:33:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.188+0000 7fd3bb446700 1 --2- 172.21.15.40:0/3673366109 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b4107b50 0x7fd3b4107f70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.188+0000 7fd3bb446700 1 -- 172.21.15.40:0/3673366109 >> 172.21.15.40:0/3673366109 conn(0x7fd3b41030f0 msgr2=0x7fd3b4105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.188+0000 7fd3bb446700 1 -- 172.21.15.40:0/3673366109 shutdown_connections 2024-03-18T22:33:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.188+0000 7fd3bb446700 1 -- 172.21.15.40:0/3673366109 wait complete. 2024-03-18T22:33:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.189+0000 7fd3bb446700 1 Processor -- start 2024-03-18T22:33:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.189+0000 7fd3bb446700 1 -- start start 2024-03-18T22:33:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.189+0000 7fd3bb446700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b4107b50 0x7fd3b419bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.189+0000 7fd3bb446700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b410c100 con 0x7fd3b4107b50 2024-03-18T22:33:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.189+0000 7fd3b91e2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b4107b50 0x7fd3b419bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.189+0000 7fd3b91e2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b4107b50 0x7fd3b419bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39056/0 (socket says 172.21.15.40:39056) 2024-03-18T22:33:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.189+0000 7fd3b91e2700 1 -- 172.21.15.40:0/566928552 learned_addr learned my addr 172.21.15.40:0/566928552 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.189+0000 7fd3b91e2700 1 -- 172.21.15.40:0/566928552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b419c000 con 0x7fd3b4107b50 2024-03-18T22:33:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.190+0000 7fd3b91e2700 1 --2- 172.21.15.40:0/566928552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b4107b50 0x7fd3b419bac0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8000c00 tx=0x7fd3a8004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.190+0000 7fd3a67fc700 1 -- 172.21.15.40:0/566928552 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3a80039c0 con 0x7fd3b4107b50 2024-03-18T22:33:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.190+0000 7fd3a67fc700 1 -- 172.21.15.40:0/566928552 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fd3a8003b60 con 0x7fd3b4107b50 2024-03-18T22:33:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.190+0000 7fd3bb446700 1 -- 172.21.15.40:0/566928552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b419c290 con 0x7fd3b4107b50 2024-03-18T22:33:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.190+0000 7fd3a67fc700 1 -- 172.21.15.40:0/566928552 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3a801a6f0 con 0x7fd3b4107b50 2024-03-18T22:33:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.190+0000 7fd3bb446700 1 -- 172.21.15.40:0/566928552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b419ef80 con 0x7fd3b4107b50 2024-03-18T22:33:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.191+0000 7fd3a67fc700 1 -- 172.21.15.40:0/566928552 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fd3a801a890 con 0x7fd3b4107b50 2024-03-18T22:33:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.191+0000 7fd3a67fc700 1 --2- 172.21.15.40:0/566928552 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fd3a0038f70 0x7fd3a003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.191+0000 7fd3a67fc700 1 -- 172.21.15.40:0/566928552 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd3a804d140 con 0x7fd3b4107b50 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.191+0000 7fd3b89e1700 1 --2- 172.21.15.40:0/566928552 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fd3a0038f70 0x7fd3a003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.191+0000 7fd3bb446700 1 -- 172.21.15.40:0/566928552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3b4063940 con 0x7fd3b4107b50 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.191+0000 7fd3b89e1700 1 --2- 172.21.15.40:0/566928552 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fd3a0038f70 0x7fd3a003b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0006fd0 tx=0x7fd3b0006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.196+0000 7fd3a67fc700 1 -- 172.21.15.40:0/566928552 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd3a800bc10 con 0x7fd3b4107b50 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:20.402+0000 7fd3bb446700 1 -- 172.21.15.40:0/566928552 --> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fd3b4104310 con 0x7fd3a0038f70 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.014+0000 7fd3a67fc700 1 -- 172.21.15.40:0/566928552 <== mgr.14116 v2:172.21.15.40:6800/3876208039 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd3b4104310 con 0x7fd3a0038f70 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.016+0000 7fd3bb446700 1 -- 172.21.15.40:0/566928552 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fd3a0038f70 msgr2=0x7fd3a003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.016+0000 7fd3bb446700 1 --2- 172.21.15.40:0/566928552 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fd3a0038f70 0x7fd3a003b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0006fd0 tx=0x7fd3b0006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.016+0000 7fd3bb446700 1 -- 172.21.15.40:0/566928552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b4107b50 msgr2=0x7fd3b419bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.016+0000 7fd3bb446700 1 --2- 172.21.15.40:0/566928552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b4107b50 0x7fd3b419bac0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8000c00 tx=0x7fd3a8004290 comp rx=0 tx=0).stop 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.016+0000 7fd3bb446700 1 -- 172.21.15.40:0/566928552 shutdown_connections 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.016+0000 7fd3bb446700 1 --2- 172.21.15.40:0/566928552 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fd3a0038f70 0x7fd3a003b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.016+0000 7fd3bb446700 1 --2- 172.21.15.40:0/566928552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b4107b50 0x7fd3b419bac0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.016+0000 7fd3bb446700 1 -- 172.21.15.40:0/566928552 >> 172.21.15.40:0/566928552 conn(0x7fd3b41030f0 msgr2=0x7fd3b4103c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.016+0000 7fd3bb446700 1 -- 172.21.15.40:0/566928552 shutdown_connections 2024-03-18T22:33:21.580 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.016+0000 7fd3bb446700 1 -- 172.21.15.40:0/566928552 wait complete. 2024-03-18T22:33:22.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:22 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' 2024-03-18T22:33:22.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:22 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' 2024-03-18T22:33:22.665 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS3KiRwA6QtO9fhKplNG+AOTYKulH7gCG8zmwiM8QRyHA7v0jA5ejd3bIeYrBZ+GHGVCpdIS5uIn+ZuqtTWwIKJoaZfkdiynrQieKgfmrz0gOoM4XWWeaxxO/F2FURm3VylSfKIg2wJcMODCu0JHFC7Bz6wIrgfHnYspR6jNmMw2gj7C1nN2cK8GrcJ6BgpqyJ3DOjN2N2wPTrSGytjUCvV6OM+nlPwTIwwhCE3he5Z6Awn1BqRmXeSfCILz4XrpK/6tcrAe+srpV6PSDQragYAYz8ZSGNvnOCqZ19efcLMsg+qT2NrDxsOn3DA/3UxSuNZTREiZhOe6aV6QNULQE0qgXVWDlWVgd1FakmbBN28ngGDdWgKnof3D3ZhYvkEkBW8TXOiHrkseyh3jgLwRxdOE29GTzEJoraFbxi1jp+UTPE82HBMvMfJTroSzyBgoRNccrH4QsoQYctBkN0Ktr6cMeCfQx/n7EShleaGSUpTYdadQX17lvtbrzYXHuUQPc= ceph-410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:33:22.665 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.977+0000 7fcf5191c700 1 Processor -- start 2024-03-18T22:33:22.666 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.977+0000 7fcf5191c700 1 -- start start 2024-03-18T22:33:22.666 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.977+0000 7fcf5191c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf4c108610 0x7fcf4c10aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:22.666 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.977+0000 7fcf5191c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c07be40 con 0x7fcf4c108610 2024-03-18T22:33:22.666 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.978+0000 7fcf4affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf4c108610 0x7fcf4c10aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:22.666 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.978+0000 7fcf4affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf4c108610 0x7fcf4c10aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39064/0 (socket says 172.21.15.40:39064) 2024-03-18T22:33:22.666 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.978+0000 7fcf4affd700 1 -- 172.21.15.40:0/2673504968 learned_addr learned my addr 172.21.15.40:0/2673504968 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:22.666 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.978+0000 7fcf4affd700 1 -- 172.21.15.40:0/2673504968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4c10af40 con 0x7fcf4c108610 2024-03-18T22:33:22.666 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.978+0000 7fcf4affd700 1 --2- 172.21.15.40:0/2673504968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf4c108610 0x7fcf4c10aa00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c0098a0 tx=0x7fcf3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d6ea2337f2ed69 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:22.666 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.978+0000 7fcf49ffb700 1 -- 172.21.15.40:0/2673504968 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf3c004160 con 0x7fcf4c108610 2024-03-18T22:33:22.666 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.978+0000 7fcf49ffb700 1 -- 172.21.15.40:0/2673504968 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fcf3c004300 con 0x7fcf4c108610 2024-03-18T22:33:22.666 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.978+0000 7fcf49ffb700 1 -- 172.21.15.40:0/2673504968 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf3c011a80 con 0x7fcf4c108610 2024-03-18T22:33:22.666 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.979+0000 7fcf5191c700 1 -- 172.21.15.40:0/2673504968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf4c108610 msgr2=0x7fcf4c10aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:22.666 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.979+0000 7fcf5191c700 1 --2- 172.21.15.40:0/2673504968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf4c108610 0x7fcf4c10aa00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c0098a0 tx=0x7fcf3c009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:22.666 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.979+0000 7fcf5191c700 1 -- 172.21.15.40:0/2673504968 shutdown_connections 2024-03-18T22:33:22.666 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.979+0000 7fcf5191c700 1 --2- 172.21.15.40:0/2673504968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf4c108610 0x7fcf4c10aa00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.979+0000 7fcf5191c700 1 -- 172.21.15.40:0/2673504968 >> 172.21.15.40:0/2673504968 conn(0x7fcf4c102000 msgr2=0x7fcf4c104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.979+0000 7fcf5191c700 1 -- 172.21.15.40:0/2673504968 shutdown_connections 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.979+0000 7fcf5191c700 1 -- 172.21.15.40:0/2673504968 wait complete. 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.980+0000 7fcf5191c700 1 Processor -- start 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.980+0000 7fcf5191c700 1 -- start start 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.980+0000 7fcf5191c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf4c108610 0x7fcf4c19bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.980+0000 7fcf5191c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c10bf00 con 0x7fcf4c108610 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.980+0000 7fcf4affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf4c108610 0x7fcf4c19bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.980+0000 7fcf4affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf4c108610 0x7fcf4c19bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39080/0 (socket says 172.21.15.40:39080) 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.980+0000 7fcf4affd700 1 -- 172.21.15.40:0/3582857708 learned_addr learned my addr 172.21.15.40:0/3582857708 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.980+0000 7fcf4affd700 1 -- 172.21.15.40:0/3582857708 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4c19bff0 con 0x7fcf4c108610 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.980+0000 7fcf4affd700 1 --2- 172.21.15.40:0/3582857708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf4c108610 0x7fcf4c19bab0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c006af0 tx=0x7fcf3c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.981+0000 7fcf5091a700 1 -- 172.21.15.40:0/3582857708 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf3c003930 con 0x7fcf4c108610 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.981+0000 7fcf5091a700 1 -- 172.21.15.40:0/3582857708 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fcf3c003ad0 con 0x7fcf4c108610 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.981+0000 7fcf5191c700 1 -- 172.21.15.40:0/3582857708 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf4c19c280 con 0x7fcf4c108610 2024-03-18T22:33:22.667 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.981+0000 7fcf5091a700 1 -- 172.21.15.40:0/3582857708 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf3c0195e0 con 0x7fcf4c108610 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.981+0000 7fcf5191c700 1 -- 172.21.15.40:0/3582857708 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf4c19ef70 con 0x7fcf4c108610 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.982+0000 7fcf5091a700 1 -- 172.21.15.40:0/3582857708 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fcf3c0198e0 con 0x7fcf4c108610 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.982+0000 7fcf5191c700 1 -- 172.21.15.40:0/3582857708 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf4c063940 con 0x7fcf4c108610 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.982+0000 7fcf5091a700 1 --2- 172.21.15.40:0/3582857708 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fcf34038ed0 0x7fcf3403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.982+0000 7fcf5091a700 1 -- 172.21.15.40:0/3582857708 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcf3c04d530 con 0x7fcf4c108610 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.982+0000 7fcf4a7fc700 1 --2- 172.21.15.40:0/3582857708 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fcf34038ed0 0x7fcf3403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.982+0000 7fcf4a7fc700 1 --2- 172.21.15.40:0/3582857708 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fcf34038ed0 0x7fcf3403b390 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcf40006fd0 tx=0x7fcf40006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:21.987+0000 7fcf5091a700 1 -- 172.21.15.40:0/3582857708 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcf3c019cd0 con 0x7fcf4c108610 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:22.193+0000 7fcf5191c700 1 -- 172.21.15.40:0/3582857708 --> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fcf4c103200 con 0x7fcf34038ed0 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:22.194+0000 7fcf5091a700 1 -- 172.21.15.40:0/3582857708 <== mgr.14116 v2:172.21.15.40:6800/3876208039 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fcf4c103200 con 0x7fcf34038ed0 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:22.196+0000 7fcf5191c700 1 -- 172.21.15.40:0/3582857708 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fcf34038ed0 msgr2=0x7fcf3403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:22.196+0000 7fcf5191c700 1 --2- 172.21.15.40:0/3582857708 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fcf34038ed0 0x7fcf3403b390 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcf40006fd0 tx=0x7fcf40006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:22.196+0000 7fcf5191c700 1 -- 172.21.15.40:0/3582857708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf4c108610 msgr2=0x7fcf4c19bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:22.196+0000 7fcf5191c700 1 --2- 172.21.15.40:0/3582857708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf4c108610 0x7fcf4c19bab0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c006af0 tx=0x7fcf3c011b70 comp rx=0 tx=0).stop 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:22.196+0000 7fcf5191c700 1 -- 172.21.15.40:0/3582857708 shutdown_connections 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:22.196+0000 7fcf5191c700 1 --2- 172.21.15.40:0/3582857708 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fcf34038ed0 0x7fcf3403b390 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:22.668 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:22.196+0000 7fcf5191c700 1 --2- 172.21.15.40:0/3582857708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf4c108610 0x7fcf4c19bab0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:22.669 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:22.196+0000 7fcf5191c700 1 -- 172.21.15.40:0/3582857708 >> 172.21.15.40:0/3582857708 conn(0x7fcf4c102000 msgr2=0x7fcf4c102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:22.669 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:22.196+0000 7fcf5191c700 1 -- 172.21.15.40:0/3582857708 shutdown_connections 2024-03-18T22:33:22.669 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:22.196+0000 7fcf5191c700 1 -- 172.21.15.40:0/3582857708 wait complete. 2024-03-18T22:33:22.669 INFO:teuthology.orchestra.run.smithi040.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-18T22:33:22.669 INFO:teuthology.orchestra.run.smithi040.stdout:Adding key to root@localhost authorized_keys... 2024-03-18T22:33:22.669 INFO:teuthology.orchestra.run.smithi040.stdout:Adding host smithi040... 2024-03-18T22:33:23.094 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:23 smithi040 ceph-mon[62643]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:33:24.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:24 smithi040 ceph-mon[62643]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi040", "addr": "172.21.15.40", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:33:25.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:25 smithi040 ceph-mon[62643]: Deploying cephadm binary to smithi040 2024-03-18T22:33:28.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Added host 'smithi040' with addr '172.21.15.40' 2024-03-18T22:33:28.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.131+0000 7f84328a3700 1 Processor -- start 2024-03-18T22:33:28.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.131+0000 7f84328a3700 1 -- start start 2024-03-18T22:33:28.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.132+0000 7f84328a3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f842c107b80 0x7f842c107fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:28.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.132+0000 7f84328a3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f842c108570 con 0x7f842c107b80 2024-03-18T22:33:28.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.132+0000 7f842bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f842c107b80 0x7f842c107fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:28.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.132+0000 7f842bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f842c107b80 0x7f842c107fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39088/0 (socket says 172.21.15.40:39088) 2024-03-18T22:33:28.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.132+0000 7f842bfff700 1 -- 172.21.15.40:0/2868069689 learned_addr learned my addr 172.21.15.40:0/2868069689 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:28.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.132+0000 7f842bfff700 1 -- 172.21.15.40:0/2868069689 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f842c108d80 con 0x7f842c107b80 2024-03-18T22:33:28.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.132+0000 7f842bfff700 1 --2- 172.21.15.40:0/2868069689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f842c107b80 0x7f842c107fa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f841c0098a0 tx=0x7f841c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7552d68b4dbe2e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:28.374 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.132+0000 7f842affd700 1 -- 172.21.15.40:0/2868069689 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f841c004160 con 0x7f842c107b80 2024-03-18T22:33:28.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.132+0000 7f842affd700 1 -- 172.21.15.40:0/2868069689 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f841c004300 con 0x7f842c107b80 2024-03-18T22:33:28.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.132+0000 7f842affd700 1 -- 172.21.15.40:0/2868069689 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f841c011a80 con 0x7f842c107b80 2024-03-18T22:33:28.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.133+0000 7f84328a3700 1 -- 172.21.15.40:0/2868069689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f842c107b80 msgr2=0x7f842c107fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:28.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.133+0000 7f84328a3700 1 --2- 172.21.15.40:0/2868069689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f842c107b80 0x7f842c107fa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f841c0098a0 tx=0x7f841c009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:28.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.133+0000 7f84328a3700 1 -- 172.21.15.40:0/2868069689 shutdown_connections 2024-03-18T22:33:28.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.133+0000 7f84328a3700 1 --2- 172.21.15.40:0/2868069689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f842c107b80 0x7f842c107fa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:28.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.133+0000 7f84328a3700 1 -- 172.21.15.40:0/2868069689 >> 172.21.15.40:0/2868069689 conn(0x7f842c103120 msgr2=0x7f842c105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:28.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.133+0000 7f84328a3700 1 -- 172.21.15.40:0/2868069689 shutdown_connections 2024-03-18T22:33:28.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.133+0000 7f84328a3700 1 -- 172.21.15.40:0/2868069689 wait complete. 2024-03-18T22:33:28.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.134+0000 7f84328a3700 1 Processor -- start 2024-03-18T22:33:28.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.134+0000 7f84328a3700 1 -- start start 2024-03-18T22:33:28.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.134+0000 7f84328a3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f842c107b80 0x7f842c19baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:28.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.134+0000 7f84328a3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f842c10c130 con 0x7f842c107b80 2024-03-18T22:33:28.375 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.134+0000 7f842bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f842c107b80 0x7f842c19baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:28.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.134+0000 7f842bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f842c107b80 0x7f842c19baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39092/0 (socket says 172.21.15.40:39092) 2024-03-18T22:33:28.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.134+0000 7f842bfff700 1 -- 172.21.15.40:0/1934929614 learned_addr learned my addr 172.21.15.40:0/1934929614 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:28.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.134+0000 7f842bfff700 1 -- 172.21.15.40:0/1934929614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f842c19bfe0 con 0x7f842c107b80 2024-03-18T22:33:28.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.135+0000 7f842bfff700 1 --2- 172.21.15.40:0/1934929614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f842c107b80 0x7f842c19baa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f841c000c00 tx=0x7f841c004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:28.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.135+0000 7f84297fa700 1 -- 172.21.15.40:0/1934929614 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f841c0039c0 con 0x7f842c107b80 2024-03-18T22:33:28.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.135+0000 7f84297fa700 1 -- 172.21.15.40:0/1934929614 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f841c003b60 con 0x7f842c107b80 2024-03-18T22:33:28.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.135+0000 7f84328a3700 1 -- 172.21.15.40:0/1934929614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f842c19c270 con 0x7f842c107b80 2024-03-18T22:33:28.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.135+0000 7f84297fa700 1 -- 172.21.15.40:0/1934929614 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f841c01a6f0 con 0x7f842c107b80 2024-03-18T22:33:28.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.135+0000 7f84328a3700 1 -- 172.21.15.40:0/1934929614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f842c19ef60 con 0x7f842c107b80 2024-03-18T22:33:28.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.136+0000 7f84297fa700 1 -- 172.21.15.40:0/1934929614 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f841c01a890 con 0x7f842c107b80 2024-03-18T22:33:28.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.136+0000 7f84328a3700 1 -- 172.21.15.40:0/1934929614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f842c063940 con 0x7f842c107b80 2024-03-18T22:33:28.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.136+0000 7f84297fa700 1 --2- 172.21.15.40:0/1934929614 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f8414038f70 0x7f841403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:28.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.136+0000 7f84297fa700 1 -- 172.21.15.40:0/1934929614 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f841c04d2f0 con 0x7f842c107b80 2024-03-18T22:33:28.376 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.136+0000 7f842b7fe700 1 --2- 172.21.15.40:0/1934929614 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f8414038f70 0x7f841403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:28.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.137+0000 7f842b7fe700 1 --2- 172.21.15.40:0/1934929614 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f8414038f70 0x7f841403b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8420006fd0 tx=0x7f8420006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:28.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.141+0000 7f84297fa700 1 -- 172.21.15.40:0/1934929614 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f841c01ac30 con 0x7f842c107b80 2024-03-18T22:33:28.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:23.351+0000 7f84328a3700 1 -- 172.21.15.40:0/1934929614 --> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi040", "addr": "172.21.15.40", "target": ["mon-mgr", ""]}) -- 0x7f842c1042f0 con 0x7f8414038f70 2024-03-18T22:33:28.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:27.662+0000 7f84297fa700 1 -- 172.21.15.40:0/1934929614 <== mgr.14116 v2:172.21.15.40:6800/3876208039 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f842c1042f0 con 0x7f8414038f70 2024-03-18T22:33:28.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:27.664+0000 7f84328a3700 1 -- 172.21.15.40:0/1934929614 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f8414038f70 msgr2=0x7f841403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:28.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:27.664+0000 7f84328a3700 1 --2- 172.21.15.40:0/1934929614 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f8414038f70 0x7f841403b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8420006fd0 tx=0x7f8420006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:28.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:27.664+0000 7f84328a3700 1 -- 172.21.15.40:0/1934929614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f842c107b80 msgr2=0x7f842c19baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:28.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:27.664+0000 7f84328a3700 1 --2- 172.21.15.40:0/1934929614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f842c107b80 0x7f842c19baa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f841c000c00 tx=0x7f841c004290 comp rx=0 tx=0).stop 2024-03-18T22:33:28.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:27.664+0000 7f84328a3700 1 -- 172.21.15.40:0/1934929614 shutdown_connections 2024-03-18T22:33:28.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:27.665+0000 7f84328a3700 1 --2- 172.21.15.40:0/1934929614 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f8414038f70 0x7f841403b430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:28.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:27.665+0000 7f84328a3700 1 --2- 172.21.15.40:0/1934929614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f842c107b80 0x7f842c19baa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:28.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:27.665+0000 7f84328a3700 1 -- 172.21.15.40:0/1934929614 >> 172.21.15.40:0/1934929614 conn(0x7f842c103120 msgr2=0x7f842c103be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:28.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:27.665+0000 7f84328a3700 1 -- 172.21.15.40:0/1934929614 shutdown_connections 2024-03-18T22:33:28.377 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:27.665+0000 7f84328a3700 1 -- 172.21.15.40:0/1934929614 wait complete. 2024-03-18T22:33:28.377 INFO:teuthology.orchestra.run.smithi040.stdout:Deploying unmanaged mon service... 2024-03-18T22:33:29.058 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:28 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' 2024-03-18T22:33:29.058 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:28 smithi040 ceph-mon[62643]: Added host smithi040 2024-03-18T22:33:29.059 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:28 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:33:29.808 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-18T22:33:29.808 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.813+0000 7fd4c571b700 1 Processor -- start 2024-03-18T22:33:29.808 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.813+0000 7fd4c571b700 1 -- start start 2024-03-18T22:33:29.808 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.813+0000 7fd4c571b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c0107b50 0x7fd4c0107f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:29.808 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.813+0000 7fd4c571b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c0108540 con 0x7fd4c0107b50 2024-03-18T22:33:29.808 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.814+0000 7fd4beffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c0107b50 0x7fd4c0107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:29.808 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.814+0000 7fd4beffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c0107b50 0x7fd4c0107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33112/0 (socket says 172.21.15.40:33112) 2024-03-18T22:33:29.808 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.814+0000 7fd4beffd700 1 -- 172.21.15.40:0/596784006 learned_addr learned my addr 172.21.15.40:0/596784006 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:29.808 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.814+0000 7fd4beffd700 1 -- 172.21.15.40:0/596784006 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c0108d50 con 0x7fd4c0107b50 2024-03-18T22:33:29.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.814+0000 7fd4beffd700 1 --2- 172.21.15.40:0/596784006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c0107b50 0x7fd4c0107f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00098a0 tx=0x7fd4b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fec847762282580 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:29.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.814+0000 7fd4bdffb700 1 -- 172.21.15.40:0/596784006 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4b0004160 con 0x7fd4c0107b50 2024-03-18T22:33:29.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.815+0000 7fd4bdffb700 1 -- 172.21.15.40:0/596784006 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fd4b0004300 con 0x7fd4c0107b50 2024-03-18T22:33:29.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.815+0000 7fd4bdffb700 1 -- 172.21.15.40:0/596784006 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4b0011a80 con 0x7fd4c0107b50 2024-03-18T22:33:29.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.815+0000 7fd4c571b700 1 -- 172.21.15.40:0/596784006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c0107b50 msgr2=0x7fd4c0107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:29.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.815+0000 7fd4c571b700 1 --2- 172.21.15.40:0/596784006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c0107b50 0x7fd4c0107f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00098a0 tx=0x7fd4b0009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:29.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.815+0000 7fd4c571b700 1 -- 172.21.15.40:0/596784006 shutdown_connections 2024-03-18T22:33:29.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.815+0000 7fd4c571b700 1 --2- 172.21.15.40:0/596784006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c0107b50 0x7fd4c0107f70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:29.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.815+0000 7fd4c571b700 1 -- 172.21.15.40:0/596784006 >> 172.21.15.40:0/596784006 conn(0x7fd4c01030f0 msgr2=0x7fd4c0105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:29.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.815+0000 7fd4c571b700 1 -- 172.21.15.40:0/596784006 shutdown_connections 2024-03-18T22:33:29.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.815+0000 7fd4c571b700 1 -- 172.21.15.40:0/596784006 wait complete. 2024-03-18T22:33:29.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.816+0000 7fd4c571b700 1 Processor -- start 2024-03-18T22:33:29.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.816+0000 7fd4c571b700 1 -- start start 2024-03-18T22:33:29.809 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.816+0000 7fd4c571b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c0107b50 0x7fd4c007db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:29.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.816+0000 7fd4c571b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c010c100 con 0x7fd4c0107b50 2024-03-18T22:33:29.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.816+0000 7fd4beffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c0107b50 0x7fd4c007db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:29.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.816+0000 7fd4beffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c0107b50 0x7fd4c007db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33126/0 (socket says 172.21.15.40:33126) 2024-03-18T22:33:29.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.816+0000 7fd4beffd700 1 -- 172.21.15.40:0/2816646985 learned_addr learned my addr 172.21.15.40:0/2816646985 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:29.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.816+0000 7fd4beffd700 1 -- 172.21.15.40:0/2816646985 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c007e080 con 0x7fd4c0107b50 2024-03-18T22:33:29.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.817+0000 7fd4beffd700 1 --2- 172.21.15.40:0/2816646985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c0107b50 0x7fd4c007db40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0000c00 tx=0x7fd4b0004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:29.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.817+0000 7fd4a7fff700 1 -- 172.21.15.40:0/2816646985 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4b00039c0 con 0x7fd4c0107b50 2024-03-18T22:33:29.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.817+0000 7fd4a7fff700 1 -- 172.21.15.40:0/2816646985 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fd4b0003b60 con 0x7fd4c0107b50 2024-03-18T22:33:29.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.817+0000 7fd4c571b700 1 -- 172.21.15.40:0/2816646985 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c007c270 con 0x7fd4c0107b50 2024-03-18T22:33:29.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.817+0000 7fd4a7fff700 1 -- 172.21.15.40:0/2816646985 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4b001a6f0 con 0x7fd4c0107b50 2024-03-18T22:33:29.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.817+0000 7fd4c571b700 1 -- 172.21.15.40:0/2816646985 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c007c7b0 con 0x7fd4c0107b50 2024-03-18T22:33:29.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.818+0000 7fd4a7fff700 1 -- 172.21.15.40:0/2816646985 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fd4b001a9a0 con 0x7fd4c0107b50 2024-03-18T22:33:29.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.818+0000 7fd4c571b700 1 -- 172.21.15.40:0/2816646985 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c0063940 con 0x7fd4c0107b50 2024-03-18T22:33:29.810 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.818+0000 7fd4a7fff700 1 --2- 172.21.15.40:0/2816646985 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fd4a8038f20 0x7fd4a803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:29.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.818+0000 7fd4a7fff700 1 -- 172.21.15.40:0/2816646985 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd4b004d3d0 con 0x7fd4c0107b50 2024-03-18T22:33:29.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.818+0000 7fd4be7fc700 1 --2- 172.21.15.40:0/2816646985 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fd4a8038f20 0x7fd4a803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:29.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.818+0000 7fd4be7fc700 1 --2- 172.21.15.40:0/2816646985 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fd4a8038f20 0x7fd4a803b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4006fd0 tx=0x7fd4b4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:29.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:28.823+0000 7fd4a7fff700 1 -- 172.21.15.40:0/2816646985 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd4b000bc40 con 0x7fd4c0107b50 2024-03-18T22:33:29.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:29.057+0000 7fd4c571b700 1 -- 172.21.15.40:0/2816646985 --> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fd4c0104340 con 0x7fd4a8038f20 2024-03-18T22:33:29.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:29.064+0000 7fd4a7fff700 1 -- 172.21.15.40:0/2816646985 <== mgr.14116 v2:172.21.15.40:6800/3876208039 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd4c0104340 con 0x7fd4a8038f20 2024-03-18T22:33:29.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:29.066+0000 7fd4c571b700 1 -- 172.21.15.40:0/2816646985 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fd4a8038f20 msgr2=0x7fd4a803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:29.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:29.066+0000 7fd4c571b700 1 --2- 172.21.15.40:0/2816646985 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fd4a8038f20 0x7fd4a803b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4006fd0 tx=0x7fd4b4006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:29.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:29.066+0000 7fd4c571b700 1 -- 172.21.15.40:0/2816646985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c0107b50 msgr2=0x7fd4c007db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:29.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:29.066+0000 7fd4c571b700 1 --2- 172.21.15.40:0/2816646985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c0107b50 0x7fd4c007db40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0000c00 tx=0x7fd4b0004290 comp rx=0 tx=0).stop 2024-03-18T22:33:29.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:29.067+0000 7fd4c571b700 1 -- 172.21.15.40:0/2816646985 shutdown_connections 2024-03-18T22:33:29.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:29.067+0000 7fd4c571b700 1 --2- 172.21.15.40:0/2816646985 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fd4a8038f20 0x7fd4a803b3e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:29.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:29.067+0000 7fd4c571b700 1 --2- 172.21.15.40:0/2816646985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4c0107b50 0x7fd4c007db40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:29.811 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:29.067+0000 7fd4c571b700 1 -- 172.21.15.40:0/2816646985 >> 172.21.15.40:0/2816646985 conn(0x7fd4c01030f0 msgr2=0x7fd4c0103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:29.812 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:29.067+0000 7fd4c571b700 1 -- 172.21.15.40:0/2816646985 shutdown_connections 2024-03-18T22:33:29.812 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:29.067+0000 7fd4c571b700 1 -- 172.21.15.40:0/2816646985 wait complete. 2024-03-18T22:33:29.812 INFO:teuthology.orchestra.run.smithi040.stdout:Deploying unmanaged mgr service... 2024-03-18T22:33:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:30 smithi040 ceph-mon[62643]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:33:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:30 smithi040 ceph-mon[62643]: Saving service mon spec with placement count:5 2024-03-18T22:33:30.362 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:30 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' 2024-03-18T22:33:31.017 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-18T22:33:31.017 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.197+0000 7f33b11a0700 1 Processor -- start 2024-03-18T22:33:31.017 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.197+0000 7f33b11a0700 1 -- start start 2024-03-18T22:33:31.017 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.197+0000 7f33b11a0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33ac106a80 0x7f33ac106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:31.017 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.197+0000 7f33b11a0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac107470 con 0x7f33ac106a80 2024-03-18T22:33:31.017 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.197+0000 7f33aad9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33ac106a80 0x7f33ac106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:31.017 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.197+0000 7f33aad9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33ac106a80 0x7f33ac106ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33136/0 (socket says 172.21.15.40:33136) 2024-03-18T22:33:31.017 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.197+0000 7f33aad9d700 1 -- 172.21.15.40:0/2792207413 learned_addr learned my addr 172.21.15.40:0/2792207413 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:31.017 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.197+0000 7f33aad9d700 1 -- 172.21.15.40:0/2792207413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33ac107c80 con 0x7f33ac106a80 2024-03-18T22:33:31.017 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.198+0000 7f33aad9d700 1 --2- 172.21.15.40:0/2792207413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33ac106a80 0x7f33ac106ea0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f339c0098a0 tx=0x7f339c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4d99bf501bcc7d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:31.018 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.198+0000 7f33a9d9b700 1 -- 172.21.15.40:0/2792207413 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f339c004160 con 0x7f33ac106a80 2024-03-18T22:33:31.018 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.198+0000 7f33a9d9b700 1 -- 172.21.15.40:0/2792207413 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f339c004300 con 0x7f33ac106a80 2024-03-18T22:33:31.018 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.198+0000 7f33a9d9b700 1 -- 172.21.15.40:0/2792207413 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f339c011a80 con 0x7f33ac106a80 2024-03-18T22:33:31.018 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.198+0000 7f33b11a0700 1 -- 172.21.15.40:0/2792207413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33ac106a80 msgr2=0x7f33ac106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:31.018 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.198+0000 7f33b11a0700 1 --2- 172.21.15.40:0/2792207413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33ac106a80 0x7f33ac106ea0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f339c0098a0 tx=0x7f339c009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:31.018 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.198+0000 7f33b11a0700 1 -- 172.21.15.40:0/2792207413 shutdown_connections 2024-03-18T22:33:31.018 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.198+0000 7f33b11a0700 1 --2- 172.21.15.40:0/2792207413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33ac106a80 0x7f33ac106ea0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:31.018 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.198+0000 7f33b11a0700 1 -- 172.21.15.40:0/2792207413 >> 172.21.15.40:0/2792207413 conn(0x7f33ac102000 msgr2=0x7f33ac104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:31.018 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.198+0000 7f33b11a0700 1 -- 172.21.15.40:0/2792207413 shutdown_connections 2024-03-18T22:33:31.018 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.198+0000 7f33b11a0700 1 -- 172.21.15.40:0/2792207413 wait complete. 2024-03-18T22:33:31.018 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.199+0000 7f33b11a0700 1 Processor -- start 2024-03-18T22:33:31.018 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.199+0000 7f33b11a0700 1 -- start start 2024-03-18T22:33:31.018 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.199+0000 7f33b11a0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33ac106a80 0x7f33ac1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:31.018 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.199+0000 7f33b11a0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac07afa0 con 0x7f33ac106a80 2024-03-18T22:33:31.018 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.200+0000 7f33aad9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33ac106a80 0x7f33ac1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:31.019 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.200+0000 7f33aad9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33ac106a80 0x7f33ac1998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33152/0 (socket says 172.21.15.40:33152) 2024-03-18T22:33:31.019 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.200+0000 7f33aad9d700 1 -- 172.21.15.40:0/2913599952 learned_addr learned my addr 172.21.15.40:0/2913599952 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:31.019 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.200+0000 7f33aad9d700 1 -- 172.21.15.40:0/2913599952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33ac199de0 con 0x7f33ac106a80 2024-03-18T22:33:31.019 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.200+0000 7f33aad9d700 1 --2- 172.21.15.40:0/2913599952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33ac106a80 0x7f33ac1998a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f339c006af0 tx=0x7f339c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:31.019 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.200+0000 7f3393fff700 1 -- 172.21.15.40:0/2913599952 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f339c003930 con 0x7f33ac106a80 2024-03-18T22:33:31.019 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.200+0000 7f3393fff700 1 -- 172.21.15.40:0/2913599952 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f339c003ad0 con 0x7f33ac106a80 2024-03-18T22:33:31.019 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.200+0000 7f33b11a0700 1 -- 172.21.15.40:0/2913599952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33ac19a070 con 0x7f33ac106a80 2024-03-18T22:33:31.019 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.200+0000 7f3393fff700 1 -- 172.21.15.40:0/2913599952 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f339c0195e0 con 0x7f33ac106a80 2024-03-18T22:33:31.019 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.200+0000 7f33b11a0700 1 -- 172.21.15.40:0/2913599952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33ac19cd60 con 0x7f33ac106a80 2024-03-18T22:33:31.019 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.201+0000 7f3393fff700 1 -- 172.21.15.40:0/2913599952 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f339c019890 con 0x7f33ac106a80 2024-03-18T22:33:31.019 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.201+0000 7f3393fff700 1 --2- 172.21.15.40:0/2913599952 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f339403b140 0x7f339403d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:31.019 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.201+0000 7f3393fff700 1 -- 172.21.15.40:0/2913599952 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f339c01a430 con 0x7f33ac106a80 2024-03-18T22:33:31.019 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.201+0000 7f33aa59c700 1 --2- 172.21.15.40:0/2913599952 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f339403b140 0x7f339403d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:31.019 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.201+0000 7f33b11a0700 1 -- 172.21.15.40:0/2913599952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33ac063940 con 0x7f33ac106a80 2024-03-18T22:33:31.019 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.202+0000 7f33aa59c700 1 --2- 172.21.15.40:0/2913599952 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f339403b140 0x7f339403d600 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f33a0006fd0 tx=0x7f33a0006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:31.020 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.206+0000 7f3393fff700 1 -- 172.21.15.40:0/2913599952 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f339c02d960 con 0x7f33ac106a80 2024-03-18T22:33:31.020 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.431+0000 7f33b11a0700 1 -- 172.21.15.40:0/2913599952 --> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f33ac062750 con 0x7f339403b140 2024-03-18T22:33:31.020 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.438+0000 7f3393fff700 1 -- 172.21.15.40:0/2913599952 <== mgr.14116 v2:172.21.15.40:6800/3876208039 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f33ac062750 con 0x7f339403b140 2024-03-18T22:33:31.020 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.441+0000 7f33b11a0700 1 -- 172.21.15.40:0/2913599952 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f339403b140 msgr2=0x7f339403d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:31.020 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.441+0000 7f33b11a0700 1 --2- 172.21.15.40:0/2913599952 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f339403b140 0x7f339403d600 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f33a0006fd0 tx=0x7f33a0006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:31.020 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.441+0000 7f33b11a0700 1 -- 172.21.15.40:0/2913599952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33ac106a80 msgr2=0x7f33ac1998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:31.020 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.441+0000 7f33b11a0700 1 --2- 172.21.15.40:0/2913599952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33ac106a80 0x7f33ac1998a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f339c006af0 tx=0x7f339c011b70 comp rx=0 tx=0).stop 2024-03-18T22:33:31.020 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.441+0000 7f33b11a0700 1 -- 172.21.15.40:0/2913599952 shutdown_connections 2024-03-18T22:33:31.020 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.441+0000 7f33b11a0700 1 --2- 172.21.15.40:0/2913599952 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f339403b140 0x7f339403d600 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:31.020 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.441+0000 7f33b11a0700 1 --2- 172.21.15.40:0/2913599952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33ac106a80 0x7f33ac1998a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:31.020 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.441+0000 7f33b11a0700 1 -- 172.21.15.40:0/2913599952 >> 172.21.15.40:0/2913599952 conn(0x7f33ac102000 msgr2=0x7f33ac18fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:31.020 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.441+0000 7f33b11a0700 1 -- 172.21.15.40:0/2913599952 shutdown_connections 2024-03-18T22:33:31.020 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:30.441+0000 7f33b11a0700 1 -- 172.21.15.40:0/2913599952 wait complete. 2024-03-18T22:33:31.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:31 smithi040 ceph-mon[62643]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:33:31.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:31 smithi040 ceph-mon[62643]: Saving service mgr spec with placement count:2 2024-03-18T22:33:31.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:31 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' 2024-03-18T22:33:33.028 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:32 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' 2024-03-18T22:33:33.028 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:32 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3835861628' entity='client.admin' 2024-03-18T22:33:33.028 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:32 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' 2024-03-18T22:33:33.035 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.237+0000 7fa313c2a700 1 Processor -- start 2024-03-18T22:33:33.035 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.237+0000 7fa313c2a700 1 -- start start 2024-03-18T22:33:33.035 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.237+0000 7fa313c2a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30c107b50 0x7fa30c107f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:33.035 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.237+0000 7fa313c2a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c108540 con 0x7fa30c107b50 2024-03-18T22:33:33.035 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.238+0000 7fa3119c6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30c107b50 0x7fa30c107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:33.035 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.238+0000 7fa3119c6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30c107b50 0x7fa30c107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33164/0 (socket says 172.21.15.40:33164) 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.238+0000 7fa3119c6700 1 -- 172.21.15.40:0/3119930878 learned_addr learned my addr 172.21.15.40:0/3119930878 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.238+0000 7fa3119c6700 1 -- 172.21.15.40:0/3119930878 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa30c108d50 con 0x7fa30c107b50 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.238+0000 7fa3119c6700 1 --2- 172.21.15.40:0/3119930878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30c107b50 0x7fa30c107f70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa3080098a0 tx=0x7fa308009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=372fa125533b813f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.238+0000 7fa3109c4700 1 -- 172.21.15.40:0/3119930878 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa308004160 con 0x7fa30c107b50 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.238+0000 7fa3109c4700 1 -- 172.21.15.40:0/3119930878 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa308004300 con 0x7fa30c107b50 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.238+0000 7fa3109c4700 1 -- 172.21.15.40:0/3119930878 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa308011a50 con 0x7fa30c107b50 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.239+0000 7fa313c2a700 1 -- 172.21.15.40:0/3119930878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30c107b50 msgr2=0x7fa30c107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.239+0000 7fa313c2a700 1 --2- 172.21.15.40:0/3119930878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30c107b50 0x7fa30c107f70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa3080098a0 tx=0x7fa308009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.239+0000 7fa313c2a700 1 -- 172.21.15.40:0/3119930878 shutdown_connections 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.239+0000 7fa313c2a700 1 --2- 172.21.15.40:0/3119930878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30c107b50 0x7fa30c107f70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.239+0000 7fa313c2a700 1 -- 172.21.15.40:0/3119930878 >> 172.21.15.40:0/3119930878 conn(0x7fa30c1030f0 msgr2=0x7fa30c105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.239+0000 7fa313c2a700 1 -- 172.21.15.40:0/3119930878 shutdown_connections 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.239+0000 7fa313c2a700 1 -- 172.21.15.40:0/3119930878 wait complete. 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.240+0000 7fa313c2a700 1 Processor -- start 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.240+0000 7fa313c2a700 1 -- start start 2024-03-18T22:33:33.036 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.240+0000 7fa313c2a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30c107b50 0x7fa30c07db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:33.037 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.240+0000 7fa313c2a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c10c100 con 0x7fa30c107b50 2024-03-18T22:33:33.037 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.240+0000 7fa3119c6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30c107b50 0x7fa30c07db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:33.037 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.240+0000 7fa3119c6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30c107b50 0x7fa30c07db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33170/0 (socket says 172.21.15.40:33170) 2024-03-18T22:33:33.037 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.240+0000 7fa3119c6700 1 -- 172.21.15.40:0/3835861628 learned_addr learned my addr 172.21.15.40:0/3835861628 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:33.037 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.240+0000 7fa3119c6700 1 -- 172.21.15.40:0/3835861628 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa30c07e080 con 0x7fa30c107b50 2024-03-18T22:33:33.037 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.240+0000 7fa3119c6700 1 --2- 172.21.15.40:0/3835861628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30c107b50 0x7fa30c07db40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa308011a20 tx=0x7fa308011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:33.037 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.241+0000 7fa302ffd700 1 -- 172.21.15.40:0/3835861628 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa308003af0 con 0x7fa30c107b50 2024-03-18T22:33:33.037 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.241+0000 7fa302ffd700 1 -- 172.21.15.40:0/3835861628 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa30801a8e0 con 0x7fa30c107b50 2024-03-18T22:33:33.037 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.241+0000 7fa313c2a700 1 -- 172.21.15.40:0/3835861628 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa30c07c270 con 0x7fa30c107b50 2024-03-18T22:33:33.037 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.241+0000 7fa302ffd700 1 -- 172.21.15.40:0/3835861628 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa308024860 con 0x7fa30c107b50 2024-03-18T22:33:33.037 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.241+0000 7fa313c2a700 1 -- 172.21.15.40:0/3835861628 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa30c07c7b0 con 0x7fa30c107b50 2024-03-18T22:33:33.037 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.242+0000 7fa302ffd700 1 -- 172.21.15.40:0/3835861628 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fa308022020 con 0x7fa30c107b50 2024-03-18T22:33:33.037 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.242+0000 7fa313c2a700 1 -- 172.21.15.40:0/3835861628 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa30c063940 con 0x7fa30c107b50 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.242+0000 7fa302ffd700 1 --2- 172.21.15.40:0/3835861628 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fa2f8038f70 0x7fa2f803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.242+0000 7fa302ffd700 1 -- 172.21.15.40:0/3835861628 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa30804d0f0 con 0x7fa30c107b50 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.242+0000 7fa3111c5700 1 --2- 172.21.15.40:0/3835861628 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fa2f8038f70 0x7fa2f803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.242+0000 7fa3111c5700 1 --2- 172.21.15.40:0/3835861628 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fa2f8038f70 0x7fa2f803b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc006fd0 tx=0x7fa2fc006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.247+0000 7fa302ffd700 1 -- 172.21.15.40:0/3835861628 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa30800b3e0 con 0x7fa30c107b50 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.453+0000 7fa313c2a700 1 -- 172.21.15.40:0/3835861628 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fa30c07d050 con 0x7fa30c107b50 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.455+0000 7fa302ffd700 1 -- 172.21.15.40:0/3835861628 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fa308029de0 con 0x7fa30c107b50 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.458+0000 7fa313c2a700 1 -- 172.21.15.40:0/3835861628 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fa2f8038f70 msgr2=0x7fa2f803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.458+0000 7fa313c2a700 1 --2- 172.21.15.40:0/3835861628 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fa2f8038f70 0x7fa2f803b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc006fd0 tx=0x7fa2fc006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.458+0000 7fa313c2a700 1 -- 172.21.15.40:0/3835861628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30c107b50 msgr2=0x7fa30c07db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.458+0000 7fa313c2a700 1 --2- 172.21.15.40:0/3835861628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30c107b50 0x7fa30c07db40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa308011a20 tx=0x7fa308011b00 comp rx=0 tx=0).stop 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.458+0000 7fa313c2a700 1 -- 172.21.15.40:0/3835861628 shutdown_connections 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.458+0000 7fa313c2a700 1 --2- 172.21.15.40:0/3835861628 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fa2f8038f70 0x7fa2f803b430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.459+0000 7fa313c2a700 1 --2- 172.21.15.40:0/3835861628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30c107b50 0x7fa30c07db40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.459+0000 7fa313c2a700 1 -- 172.21.15.40:0/3835861628 >> 172.21.15.40:0/3835861628 conn(0x7fa30c1030f0 msgr2=0x7fa30c103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:33.038 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.459+0000 7fa313c2a700 1 -- 172.21.15.40:0/3835861628 shutdown_connections 2024-03-18T22:33:33.039 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:32.459+0000 7fa313c2a700 1 -- 172.21.15.40:0/3835861628 wait complete. 2024-03-18T22:33:34.344 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.444+0000 7fb803884700 1 Processor -- start 2024-03-18T22:33:34.344 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.444+0000 7fb803884700 1 -- start start 2024-03-18T22:33:34.345 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.444+0000 7fb803884700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7fc105e10 0x7fb7fc106230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:34.345 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.444+0000 7fb803884700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc106800 con 0x7fb7fc105e10 2024-03-18T22:33:34.345 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.444+0000 7fb801620700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7fc105e10 0x7fb7fc106230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:34.345 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.444+0000 7fb801620700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7fc105e10 0x7fb7fc106230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33174/0 (socket says 172.21.15.40:33174) 2024-03-18T22:33:34.345 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.444+0000 7fb801620700 1 -- 172.21.15.40:0/4079464861 learned_addr learned my addr 172.21.15.40:0/4079464861 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:34.345 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.444+0000 7fb801620700 1 -- 172.21.15.40:0/4079464861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7fc106980 con 0x7fb7fc105e10 2024-03-18T22:33:34.345 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.445+0000 7fb801620700 1 --2- 172.21.15.40:0/4079464861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7fc105e10 0x7fb7fc106230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80098a0 tx=0x7fb7f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358a0f319faf7306 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:34.345 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.445+0000 7fb7f3fff700 1 -- 172.21.15.40:0/4079464861 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7f8004160 con 0x7fb7fc105e10 2024-03-18T22:33:34.345 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.445+0000 7fb7f3fff700 1 -- 172.21.15.40:0/4079464861 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb7f8004300 con 0x7fb7fc105e10 2024-03-18T22:33:34.345 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.445+0000 7fb7f3fff700 1 -- 172.21.15.40:0/4079464861 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7f8011a50 con 0x7fb7fc105e10 2024-03-18T22:33:34.345 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.446+0000 7fb803884700 1 -- 172.21.15.40:0/4079464861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7fc105e10 msgr2=0x7fb7fc106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:34.345 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.446+0000 7fb803884700 1 --2- 172.21.15.40:0/4079464861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7fc105e10 0x7fb7fc106230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80098a0 tx=0x7fb7f8009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:34.345 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.446+0000 7fb803884700 1 -- 172.21.15.40:0/4079464861 shutdown_connections 2024-03-18T22:33:34.345 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.446+0000 7fb803884700 1 --2- 172.21.15.40:0/4079464861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7fc105e10 0x7fb7fc106230 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:34.346 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.446+0000 7fb803884700 1 -- 172.21.15.40:0/4079464861 >> 172.21.15.40:0/4079464861 conn(0x7fb7fc101770 msgr2=0x7fb7fc103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:34.346 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.446+0000 7fb803884700 1 -- 172.21.15.40:0/4079464861 shutdown_connections 2024-03-18T22:33:34.346 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.446+0000 7fb803884700 1 -- 172.21.15.40:0/4079464861 wait complete. 2024-03-18T22:33:34.346 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.447+0000 7fb803884700 1 Processor -- start 2024-03-18T22:33:34.346 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.447+0000 7fb803884700 1 -- start start 2024-03-18T22:33:34.346 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.447+0000 7fb803884700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7fc105e10 0x7fb7fc108710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:34.346 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.447+0000 7fb803884700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc109d20 con 0x7fb7fc105e10 2024-03-18T22:33:34.346 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.447+0000 7fb801620700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7fc105e10 0x7fb7fc108710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:34.346 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.447+0000 7fb801620700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7fc105e10 0x7fb7fc108710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33178/0 (socket says 172.21.15.40:33178) 2024-03-18T22:33:34.346 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.447+0000 7fb801620700 1 -- 172.21.15.40:0/1550015185 learned_addr learned my addr 172.21.15.40:0/1550015185 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:34.346 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.448+0000 7fb801620700 1 -- 172.21.15.40:0/1550015185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7fc108c50 con 0x7fb7fc105e10 2024-03-18T22:33:34.346 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.448+0000 7fb801620700 1 --2- 172.21.15.40:0/1550015185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7fc105e10 0x7fb7fc108710 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8011710 tx=0x7fb7f8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:34.346 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.448+0000 7fb7f27fc700 1 -- 172.21.15.40:0/1550015185 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7f80039c0 con 0x7fb7fc105e10 2024-03-18T22:33:34.347 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.448+0000 7fb7f27fc700 1 -- 172.21.15.40:0/1550015185 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb7f8003b60 con 0x7fb7fc105e10 2024-03-18T22:33:34.347 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.448+0000 7fb803884700 1 -- 172.21.15.40:0/1550015185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7fc106e40 con 0x7fb7fc105e10 2024-03-18T22:33:34.347 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.448+0000 7fb7f27fc700 1 -- 172.21.15.40:0/1550015185 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7f801a690 con 0x7fb7fc105e10 2024-03-18T22:33:34.347 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.448+0000 7fb803884700 1 -- 172.21.15.40:0/1550015185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7fc107380 con 0x7fb7fc105e10 2024-03-18T22:33:34.347 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.449+0000 7fb7f27fc700 1 -- 172.21.15.40:0/1550015185 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fb7f801a990 con 0x7fb7fc105e10 2024-03-18T22:33:34.347 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.449+0000 7fb7f27fc700 1 --2- 172.21.15.40:0/1550015185 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb7e8038ed0 0x7fb7e803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:34.347 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.449+0000 7fb803884700 1 -- 172.21.15.40:0/1550015185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7fc063940 con 0x7fb7fc105e10 2024-03-18T22:33:34.347 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.449+0000 7fb7f27fc700 1 -- 172.21.15.40:0/1550015185 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb7f804d440 con 0x7fb7fc105e10 2024-03-18T22:33:34.347 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.449+0000 7fb800e1f700 1 --2- 172.21.15.40:0/1550015185 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb7e8038ed0 0x7fb7e803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:34.347 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.450+0000 7fb800e1f700 1 --2- 172.21.15.40:0/1550015185 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb7e8038ed0 0x7fb7e803b390 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec006fd0 tx=0x7fb7ec006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:34.347 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.455+0000 7fb7f27fc700 1 -- 172.21.15.40:0/1550015185 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb7f8019c70 con 0x7fb7fc105e10 2024-03-18T22:33:34.347 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.667+0000 7fb803884700 1 -- 172.21.15.40:0/1550015185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fb7fc107660 con 0x7fb7fc105e10 2024-03-18T22:33:34.347 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.670+0000 7fb7f27fc700 1 -- 172.21.15.40:0/1550015185 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fb7f800f2e0 con 0x7fb7fc105e10 2024-03-18T22:33:34.348 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.672+0000 7fb803884700 1 -- 172.21.15.40:0/1550015185 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb7e8038ed0 msgr2=0x7fb7e803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:34.348 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.672+0000 7fb803884700 1 --2- 172.21.15.40:0/1550015185 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb7e8038ed0 0x7fb7e803b390 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec006fd0 tx=0x7fb7ec006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:34.348 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.672+0000 7fb803884700 1 -- 172.21.15.40:0/1550015185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7fc105e10 msgr2=0x7fb7fc108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:34.348 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.672+0000 7fb803884700 1 --2- 172.21.15.40:0/1550015185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7fc105e10 0x7fb7fc108710 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8011710 tx=0x7fb7f8011ad0 comp rx=0 tx=0).stop 2024-03-18T22:33:34.348 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.672+0000 7fb803884700 1 -- 172.21.15.40:0/1550015185 shutdown_connections 2024-03-18T22:33:34.348 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.672+0000 7fb803884700 1 --2- 172.21.15.40:0/1550015185 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb7e8038ed0 0x7fb7e803b390 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:34.348 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.672+0000 7fb803884700 1 --2- 172.21.15.40:0/1550015185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7fc105e10 0x7fb7fc108710 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:34.348 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.672+0000 7fb803884700 1 -- 172.21.15.40:0/1550015185 >> 172.21.15.40:0/1550015185 conn(0x7fb7fc101770 msgr2=0x7fb7fc102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:34.348 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.673+0000 7fb803884700 1 -- 172.21.15.40:0/1550015185 shutdown_connections 2024-03-18T22:33:34.348 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:33.673+0000 7fb803884700 1 -- 172.21.15.40:0/1550015185 wait complete. 2024-03-18T22:33:34.348 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling the dashboard module... 2024-03-18T22:33:34.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:34 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1550015185' entity='client.admin' 2024-03-18T22:33:36.012 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:35 smithi040 ceph-mon[62643]: from='mgr.14116 172.21.15.40:0/207674194' entity='mgr.x' 2024-03-18T22:33:36.012 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:35 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3426801337' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-18T22:33:36.012 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:35 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: ignoring --setuser ceph since I am not root 2024-03-18T22:33:36.012 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:35 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: ignoring --setgroup ceph since I am not root 2024-03-18T22:33:36.296 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.128+0000 7f02f9dd0700 1 Processor -- start 2024-03-18T22:33:36.296 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.128+0000 7f02f9dd0700 1 -- start start 2024-03-18T22:33:36.296 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.128+0000 7f02f9dd0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02f4108640 0x7f02f410aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:36.296 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.128+0000 7f02f9dd0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f02f407be20 con 0x7f02f4108640 2024-03-18T22:33:36.296 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.128+0000 7f02f37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02f4108640 0x7f02f410aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:36.296 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.128+0000 7f02f37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02f4108640 0x7f02f410aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33188/0 (socket says 172.21.15.40:33188) 2024-03-18T22:33:36.296 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.128+0000 7f02f37fe700 1 -- 172.21.15.40:0/479460772 learned_addr learned my addr 172.21.15.40:0/479460772 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:36.296 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.129+0000 7f02f37fe700 1 -- 172.21.15.40:0/479460772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f410af70 con 0x7f02f4108640 2024-03-18T22:33:36.296 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.129+0000 7f02f37fe700 1 --2- 172.21.15.40:0/479460772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02f4108640 0x7f02f410aa30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f02e40098a0 tx=0x7f02e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad7683a5c9b63c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:36.296 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.129+0000 7f02f27fc700 1 -- 172.21.15.40:0/479460772 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02e4004160 con 0x7f02f4108640 2024-03-18T22:33:36.296 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.129+0000 7f02f27fc700 1 -- 172.21.15.40:0/479460772 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02e4004300 con 0x7f02f4108640 2024-03-18T22:33:36.296 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.129+0000 7f02f27fc700 1 -- 172.21.15.40:0/479460772 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02e4011a50 con 0x7f02f4108640 2024-03-18T22:33:36.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.130+0000 7f02f9dd0700 1 -- 172.21.15.40:0/479460772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02f4108640 msgr2=0x7f02f410aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:36.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.130+0000 7f02f9dd0700 1 --2- 172.21.15.40:0/479460772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02f4108640 0x7f02f410aa30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f02e40098a0 tx=0x7f02e4009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:36.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.130+0000 7f02f9dd0700 1 -- 172.21.15.40:0/479460772 shutdown_connections 2024-03-18T22:33:36.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.130+0000 7f02f9dd0700 1 --2- 172.21.15.40:0/479460772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02f4108640 0x7f02f410aa30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:36.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.130+0000 7f02f9dd0700 1 -- 172.21.15.40:0/479460772 >> 172.21.15.40:0/479460772 conn(0x7f02f4102010 msgr2=0x7f02f4104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:36.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.130+0000 7f02f9dd0700 1 -- 172.21.15.40:0/479460772 shutdown_connections 2024-03-18T22:33:36.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.130+0000 7f02f9dd0700 1 -- 172.21.15.40:0/479460772 wait complete. 2024-03-18T22:33:36.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.131+0000 7f02f9dd0700 1 Processor -- start 2024-03-18T22:33:36.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.131+0000 7f02f9dd0700 1 -- start start 2024-03-18T22:33:36.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.131+0000 7f02f9dd0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02f4108640 0x7f02f419ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:36.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.131+0000 7f02f9dd0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f02f410bf20 con 0x7f02f4108640 2024-03-18T22:33:36.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.131+0000 7f02f37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02f4108640 0x7f02f419ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:36.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.131+0000 7f02f37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02f4108640 0x7f02f419ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33192/0 (socket says 172.21.15.40:33192) 2024-03-18T22:33:36.297 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.131+0000 7f02f37fe700 1 -- 172.21.15.40:0/3426801337 learned_addr learned my addr 172.21.15.40:0/3426801337 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:36.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.131+0000 7f02f37fe700 1 -- 172.21.15.40:0/3426801337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f419bf50 con 0x7f02f4108640 2024-03-18T22:33:36.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.132+0000 7f02f37fe700 1 --2- 172.21.15.40:0/3426801337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02f4108640 0x7f02f419ba10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f02e4011a20 tx=0x7f02e4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:36.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.132+0000 7f02f0ff9700 1 -- 172.21.15.40:0/3426801337 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02e40039c0 con 0x7f02f4108640 2024-03-18T22:33:36.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.132+0000 7f02f9dd0700 1 -- 172.21.15.40:0/3426801337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f419c1e0 con 0x7f02f4108640 2024-03-18T22:33:36.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.132+0000 7f02f0ff9700 1 -- 172.21.15.40:0/3426801337 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02e4003b60 con 0x7f02f4108640 2024-03-18T22:33:36.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.132+0000 7f02f0ff9700 1 -- 172.21.15.40:0/3426801337 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02e401a6b0 con 0x7f02f4108640 2024-03-18T22:33:36.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.132+0000 7f02f9dd0700 1 -- 172.21.15.40:0/3426801337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f419eed0 con 0x7f02f4108640 2024-03-18T22:33:36.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.133+0000 7f02f0ff9700 1 -- 172.21.15.40:0/3426801337 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f02e401a9e0 con 0x7f02f4108640 2024-03-18T22:33:36.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.133+0000 7f02f9dd0700 1 -- 172.21.15.40:0/3426801337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02f4063940 con 0x7f02f4108640 2024-03-18T22:33:36.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.133+0000 7f02f0ff9700 1 --2- 172.21.15.40:0/3426801337 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f02dc038f20 0x7f02dc03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:36.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.133+0000 7f02f0ff9700 1 -- 172.21.15.40:0/3426801337 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f02e404d470 con 0x7f02f4108640 2024-03-18T22:33:36.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.133+0000 7f02f2ffd700 1 --2- 172.21.15.40:0/3426801337 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f02dc038f20 0x7f02dc03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:36.298 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.134+0000 7f02f2ffd700 1 --2- 172.21.15.40:0/3426801337 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f02dc038f20 0x7f02dc03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f02e8006fd0 tx=0x7f02e8006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:36.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.138+0000 7f02f0ff9700 1 -- 172.21.15.40:0/3426801337 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f02e4024a70 con 0x7f02f4108640 2024-03-18T22:33:36.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.405+0000 7f02f9dd0700 1 -- 172.21.15.40:0/3426801337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f02f419f0a0 con 0x7f02f4108640 2024-03-18T22:33:36.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.695+0000 7f02f0ff9700 1 -- 172.21.15.40:0/3426801337 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f02e402d910 con 0x7f02f4108640 2024-03-18T22:33:36.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.697+0000 7f02f0ff9700 1 -- 172.21.15.40:0/3426801337 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f02e402d3f0 con 0x7f02f4108640 2024-03-18T22:33:36.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.699+0000 7f02f9dd0700 1 -- 172.21.15.40:0/3426801337 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f02dc038f20 msgr2=0x7f02dc03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:36.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.699+0000 7f02f9dd0700 1 --2- 172.21.15.40:0/3426801337 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f02dc038f20 0x7f02dc03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f02e8006fd0 tx=0x7f02e8006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:36.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.699+0000 7f02f9dd0700 1 -- 172.21.15.40:0/3426801337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02f4108640 msgr2=0x7f02f419ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:36.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.699+0000 7f02f9dd0700 1 --2- 172.21.15.40:0/3426801337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02f4108640 0x7f02f419ba10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f02e4011a20 tx=0x7f02e4011b00 comp rx=0 tx=0).stop 2024-03-18T22:33:36.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.699+0000 7f02f9dd0700 1 -- 172.21.15.40:0/3426801337 shutdown_connections 2024-03-18T22:33:36.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.699+0000 7f02f9dd0700 1 --2- 172.21.15.40:0/3426801337 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f02dc038f20 0x7f02dc03b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:36.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.699+0000 7f02f9dd0700 1 --2- 172.21.15.40:0/3426801337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f02f4108640 0x7f02f419ba10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:36.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.699+0000 7f02f9dd0700 1 -- 172.21.15.40:0/3426801337 >> 172.21.15.40:0/3426801337 conn(0x7f02f4102010 msgr2=0x7f02f4102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:36.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.699+0000 7f02f9dd0700 1 -- 172.21.15.40:0/3426801337 shutdown_connections 2024-03-18T22:33:36.299 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:35.699+0000 7f02f9dd0700 1 -- 172.21.15.40:0/3426801337 wait complete. 2024-03-18T22:33:36.533 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:36 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:36.279+0000 7fa7f567b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T22:33:36.533 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:36 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:36.379+0000 7fa7f567b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T22:33:36.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:36 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3426801337' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-18T22:33:36.783 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:36 smithi040 ceph-mon[62643]: mgrmap e9: x(active, since 19s) 2024-03-18T22:33:37.393 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:37 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:37.051+0000 7fa7f567b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T22:33:37.394 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:37 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:37.151+0000 7fa7f567b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T22:33:37.394 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:37 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:37.296+0000 7fa7f567b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T22:33:37.585 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-03-18T22:33:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-18T22:33:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T22:33:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T22:33:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T22:33:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-03-18T22:33:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.781+0000 7f650c5ab700 1 Processor -- start 2024-03-18T22:33:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.781+0000 7f650c5ab700 1 -- start start 2024-03-18T22:33:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.781+0000 7f650c5ab700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6504107b70 0x7f6504107f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.781+0000 7f650c5ab700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6504108560 con 0x7f6504107b70 2024-03-18T22:33:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.782+0000 7f650a347700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6504107b70 0x7f6504107f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.782+0000 7f650a347700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6504107b70 0x7f6504107f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33220/0 (socket says 172.21.15.40:33220) 2024-03-18T22:33:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.782+0000 7f650a347700 1 -- 172.21.15.40:0/3885973010 learned_addr learned my addr 172.21.15.40:0/3885973010 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.782+0000 7f650a347700 1 -- 172.21.15.40:0/3885973010 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6504108d70 con 0x7f6504107b70 2024-03-18T22:33:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.782+0000 7f650a347700 1 --2- 172.21.15.40:0/3885973010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6504107b70 0x7f6504107f90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f65000098a0 tx=0x7f6500009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b991469efb3db41 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:37.586 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.782+0000 7f6509345700 1 -- 172.21.15.40:0/3885973010 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6500004160 con 0x7f6504107b70 2024-03-18T22:33:37.587 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.782+0000 7f6509345700 1 -- 172.21.15.40:0/3885973010 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6500004300 con 0x7f6504107b70 2024-03-18T22:33:37.587 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.782+0000 7f6509345700 1 -- 172.21.15.40:0/3885973010 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6500011a50 con 0x7f6504107b70 2024-03-18T22:33:37.587 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.783+0000 7f650c5ab700 1 -- 172.21.15.40:0/3885973010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6504107b70 msgr2=0x7f6504107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:37.587 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.783+0000 7f650c5ab700 1 --2- 172.21.15.40:0/3885973010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6504107b70 0x7f6504107f90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f65000098a0 tx=0x7f6500009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:37.587 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.783+0000 7f650c5ab700 1 -- 172.21.15.40:0/3885973010 shutdown_connections 2024-03-18T22:33:37.587 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.783+0000 7f650c5ab700 1 --2- 172.21.15.40:0/3885973010 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6504107b70 0x7f6504107f90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:37.587 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.783+0000 7f650c5ab700 1 -- 172.21.15.40:0/3885973010 >> 172.21.15.40:0/3885973010 conn(0x7f65041030f0 msgr2=0x7f6504105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:37.587 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.783+0000 7f650c5ab700 1 -- 172.21.15.40:0/3885973010 shutdown_connections 2024-03-18T22:33:37.587 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.783+0000 7f650c5ab700 1 -- 172.21.15.40:0/3885973010 wait complete. 2024-03-18T22:33:37.587 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.784+0000 7f650c5ab700 1 Processor -- start 2024-03-18T22:33:37.587 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.784+0000 7f650c5ab700 1 -- start start 2024-03-18T22:33:37.587 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.784+0000 7f650c5ab700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6504107b70 0x7f650407db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:37.587 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.784+0000 7f650c5ab700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f650410c120 con 0x7f6504107b70 2024-03-18T22:33:37.587 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.784+0000 7f650a347700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6504107b70 0x7f650407db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:37.587 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.784+0000 7f650a347700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6504107b70 0x7f650407db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33234/0 (socket says 172.21.15.40:33234) 2024-03-18T22:33:37.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.784+0000 7f650a347700 1 -- 172.21.15.40:0/515034852 learned_addr learned my addr 172.21.15.40:0/515034852 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:37.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.784+0000 7f650a347700 1 -- 172.21.15.40:0/515034852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f650407e080 con 0x7f6504107b70 2024-03-18T22:33:37.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.785+0000 7f650a347700 1 --2- 172.21.15.40:0/515034852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6504107b70 0x7f650407db40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6500011a20 tx=0x7f6500011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:37.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.785+0000 7f64f77fe700 1 -- 172.21.15.40:0/515034852 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6500003af0 con 0x7f6504107b70 2024-03-18T22:33:37.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.785+0000 7f64f77fe700 1 -- 172.21.15.40:0/515034852 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f650001a8e0 con 0x7f6504107b70 2024-03-18T22:33:37.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.785+0000 7f650c5ab700 1 -- 172.21.15.40:0/515034852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f650407c270 con 0x7f6504107b70 2024-03-18T22:33:37.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.785+0000 7f64f77fe700 1 -- 172.21.15.40:0/515034852 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6500024860 con 0x7f6504107b70 2024-03-18T22:33:37.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.785+0000 7f650c5ab700 1 -- 172.21.15.40:0/515034852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f650407c7b0 con 0x7f6504107b70 2024-03-18T22:33:37.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.786+0000 7f650c5ab700 1 -- 172.21.15.40:0/515034852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6504063940 con 0x7f6504107b70 2024-03-18T22:33:37.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.786+0000 7f64f77fe700 1 -- 172.21.15.40:0/515034852 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f6500022020 con 0x7f6504107b70 2024-03-18T22:33:37.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.786+0000 7f64f77fe700 1 --2- 172.21.15.40:0/515034852 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f64f0038f70 0x7f64f003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:37.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.786+0000 7f64f77fe700 1 -- 172.21.15.40:0/515034852 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f650004d170 con 0x7f6504107b70 2024-03-18T22:33:37.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.786+0000 7f6509b46700 1 -- 172.21.15.40:0/515034852 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f64f0038f70 msgr2=0x7f64f003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/3876208039 2024-03-18T22:33:37.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.786+0000 7f6509b46700 1 --2- 172.21.15.40:0/515034852 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f64f0038f70 0x7f64f003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T22:33:37.588 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.792+0000 7f64f77fe700 1 -- 172.21.15.40:0/515034852 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f650000b3e0 con 0x7f6504107b70 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.987+0000 7f6509b46700 1 -- 172.21.15.40:0/515034852 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f64f0038f70 msgr2=0x7f64f003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/3876208039 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:36.987+0000 7f6509b46700 1 --2- 172.21.15.40:0/515034852 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f64f0038f70 0x7f64f003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:37.056+0000 7f650c5ab700 1 -- 172.21.15.40:0/515034852 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f650407d050 con 0x7f6504107b70 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:37.057+0000 7f64f77fe700 1 -- 172.21.15.40:0/515034852 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f65000172b0 con 0x7f6504107b70 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:37.059+0000 7f650c5ab700 1 -- 172.21.15.40:0/515034852 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f64f0038f70 msgr2=0x7f64f003b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:37.059+0000 7f650c5ab700 1 --2- 172.21.15.40:0/515034852 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f64f0038f70 0x7f64f003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:37.059+0000 7f650c5ab700 1 -- 172.21.15.40:0/515034852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6504107b70 msgr2=0x7f650407db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:37.059+0000 7f650c5ab700 1 --2- 172.21.15.40:0/515034852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6504107b70 0x7f650407db40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6500011a20 tx=0x7f6500011b00 comp rx=0 tx=0).stop 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:37.059+0000 7f650c5ab700 1 -- 172.21.15.40:0/515034852 shutdown_connections 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:37.059+0000 7f650c5ab700 1 --2- 172.21.15.40:0/515034852 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7f64f0038f70 0x7f64f003b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:37.059+0000 7f650c5ab700 1 --2- 172.21.15.40:0/515034852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6504107b70 0x7f650407db40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:37.059+0000 7f650c5ab700 1 -- 172.21.15.40:0/515034852 >> 172.21.15.40:0/515034852 conn(0x7f65041030f0 msgr2=0x7f6504103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:37.059+0000 7f650c5ab700 1 -- 172.21.15.40:0/515034852 shutdown_connections 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:37.059+0000 7f650c5ab700 1 -- 172.21.15.40:0/515034852 wait complete. 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for the mgr to restart... 2024-03-18T22:33:37.589 INFO:teuthology.orchestra.run.smithi040.stdout:Waiting for mgr epoch 9... 2024-03-18T22:33:37.687 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:37 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:37.394+0000 7fa7f567b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T22:33:37.687 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:37 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:37.480+0000 7fa7f567b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T22:33:37.960 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:37 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/515034852' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T22:33:37.960 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:37 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:37.754+0000 7fa7f567b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T22:33:37.960 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:37 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:37.867+0000 7fa7f567b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T22:33:37.960 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:37 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:37.958+0000 7fa7f567b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T22:33:38.238 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:38 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:38.060+0000 7fa7f567b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T22:33:38.639 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:38 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:38.238+0000 7fa7f567b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T22:33:39.643 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:39 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:39.224+0000 7fa7f567b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T22:33:39.644 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:39 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:39.358+0000 7fa7f567b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T22:33:39.993 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:39 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:39.698+0000 7fa7f567b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T22:33:39.993 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:39 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:39.789+0000 7fa7f567b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T22:33:39.994 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:39 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:39.992+0000 7fa7f567b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T22:33:40.393 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:40 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:40.182+0000 7fa7f567b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T22:33:40.737 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:40 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:40.475+0000 7fa7f567b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T22:33:41.042 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:40 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:40.735+0000 7fa7f567b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T22:33:41.042 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:40 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:40.978+0000 7fa7f567b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T22:33:42.143 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:41 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:41.805+0000 7fa7f567b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T22:33:42.643 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:42 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:42.385+0000 7fa7f567b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T22:33:42.643 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:42 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:42.486+0000 7fa7f567b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T22:33:43.144 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:33:42 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:33:42.707+0000 7fa7f567b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T22:33:43.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:42 smithi040 ceph-mon[62643]: Active manager daemon x restarted 2024-03-18T22:33:43.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:42 smithi040 ceph-mon[62643]: Activating manager daemon x 2024-03-18T22:33:43.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:42 smithi040 ceph-mon[62643]: osdmap e3: 0 total, 0 up, 0 in 2024-03-18T22:33:43.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:42 smithi040 ceph-mon[62643]: mgrmap e10: x(active, starting, since 0.00336697s) 2024-03-18T22:33:43.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:33:43.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T22:33:43.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T22:33:43.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T22:33:43.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T22:33:43.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:42 smithi040 ceph-mon[62643]: Manager daemon x is now available 2024-03-18T22:33:43.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:33:43.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:33:43.956 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:43 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T22:33:43.957 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:43 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T22:33:43.957 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:43 smithi040 ceph-mon[62643]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T22:33:43.957 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:43 smithi040 ceph-mon[62643]: mgrmap e11: x(active, since 1.0066s) 2024-03-18T22:33:43.957 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:43 smithi040 ceph-mon[62643]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T22:33:44.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout { 2024-03-18T22:33:44.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-18T22:33:44.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T22:33:44.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout } 2024-03-18T22:33:44.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.031+0000 7fb3533d8700 1 Processor -- start 2024-03-18T22:33:44.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.031+0000 7fb3533d8700 1 -- start start 2024-03-18T22:33:44.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.031+0000 7fb3533d8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb34c106a70 0x7fb34c106e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:44.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.031+0000 7fb3533d8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb34c107460 con 0x7fb34c106a70 2024-03-18T22:33:44.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.031+0000 7fb351174700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb34c106a70 0x7fb34c106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:44.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.031+0000 7fb351174700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb34c106a70 0x7fb34c106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33236/0 (socket says 172.21.15.40:33236) 2024-03-18T22:33:44.675 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.031+0000 7fb351174700 1 -- 172.21.15.40:0/2997504162 learned_addr learned my addr 172.21.15.40:0/2997504162 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:44.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.031+0000 7fb351174700 1 -- 172.21.15.40:0/2997504162 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb34c107c70 con 0x7fb34c106a70 2024-03-18T22:33:44.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.032+0000 7fb351174700 1 --2- 172.21.15.40:0/2997504162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb34c106a70 0x7fb34c106e90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb33c009e80 tx=0x7fb33c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150fee36bd815828 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:44.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.032+0000 7fb34bfff700 1 -- 172.21.15.40:0/2997504162 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb33c017070 con 0x7fb34c106a70 2024-03-18T22:33:44.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.032+0000 7fb34bfff700 1 -- 172.21.15.40:0/2997504162 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb33c00b8c0 con 0x7fb34c106a70 2024-03-18T22:33:44.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.032+0000 7fb34bfff700 1 -- 172.21.15.40:0/2997504162 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb33c004490 con 0x7fb34c106a70 2024-03-18T22:33:44.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.032+0000 7fb3533d8700 1 -- 172.21.15.40:0/2997504162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb34c106a70 msgr2=0x7fb34c106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:44.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.032+0000 7fb3533d8700 1 --2- 172.21.15.40:0/2997504162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb34c106a70 0x7fb34c106e90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb33c009e80 tx=0x7fb33c00b330 comp rx=0 tx=0).stop 2024-03-18T22:33:44.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.032+0000 7fb3533d8700 1 -- 172.21.15.40:0/2997504162 shutdown_connections 2024-03-18T22:33:44.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.032+0000 7fb3533d8700 1 --2- 172.21.15.40:0/2997504162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb34c106a70 0x7fb34c106e90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:44.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.032+0000 7fb3533d8700 1 -- 172.21.15.40:0/2997504162 >> 172.21.15.40:0/2997504162 conn(0x7fb34c102010 msgr2=0x7fb34c104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:44.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.033+0000 7fb3533d8700 1 -- 172.21.15.40:0/2997504162 shutdown_connections 2024-03-18T22:33:44.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.033+0000 7fb3533d8700 1 -- 172.21.15.40:0/2997504162 wait complete. 2024-03-18T22:33:44.676 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.033+0000 7fb3533d8700 1 Processor -- start 2024-03-18T22:33:44.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.033+0000 7fb3533d8700 1 -- start start 2024-03-18T22:33:44.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.033+0000 7fb3533d8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb34c106a70 0x7fb34c191be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:44.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.033+0000 7fb3533d8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb34c10b020 con 0x7fb34c106a70 2024-03-18T22:33:44.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.034+0000 7fb351174700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb34c106a70 0x7fb34c191be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:44.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.034+0000 7fb351174700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb34c106a70 0x7fb34c191be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33248/0 (socket says 172.21.15.40:33248) 2024-03-18T22:33:44.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.034+0000 7fb351174700 1 -- 172.21.15.40:0/1283663578 learned_addr learned my addr 172.21.15.40:0/1283663578 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:44.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.034+0000 7fb351174700 1 -- 172.21.15.40:0/1283663578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb34c192120 con 0x7fb34c106a70 2024-03-18T22:33:44.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.034+0000 7fb351174700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb34c106a70 0x7fb34c191be0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb33c0072a0 tx=0x7fb33c003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:44.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.034+0000 7fb34a7fc700 1 -- 172.21.15.40:0/1283663578 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb33c020020 con 0x7fb34c106a70 2024-03-18T22:33:44.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.034+0000 7fb34a7fc700 1 -- 172.21.15.40:0/1283663578 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb33c00bba0 con 0x7fb34c106a70 2024-03-18T22:33:44.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.034+0000 7fb3533d8700 1 -- 172.21.15.40:0/1283663578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb34c190310 con 0x7fb34c106a70 2024-03-18T22:33:44.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.034+0000 7fb34a7fc700 1 -- 172.21.15.40:0/1283663578 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb33c017040 con 0x7fb34c106a70 2024-03-18T22:33:44.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.034+0000 7fb3533d8700 1 -- 172.21.15.40:0/1283663578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb34c190850 con 0x7fb34c106a70 2024-03-18T22:33:44.677 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.035+0000 7fb34a7fc700 1 -- 172.21.15.40:0/1283663578 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7fb33c023020 con 0x7fb34c106a70 2024-03-18T22:33:44.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.035+0000 7fb34a7fc700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb334038f20 0x7fb33403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:44.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.035+0000 7fb34a7fc700 1 -- 172.21.15.40:0/1283663578 --> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb33403baf0 con 0x7fb334038f20 2024-03-18T22:33:44.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.036+0000 7fb350973700 1 -- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb334038f20 msgr2=0x7fb33403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/3876208039 2024-03-18T22:33:44.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.036+0000 7fb34a7fc700 1 -- 172.21.15.40:0/1283663578 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb33c00f050 con 0x7fb34c106a70 2024-03-18T22:33:44.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.036+0000 7fb350973700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb334038f20 0x7fb33403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T22:33:44.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.237+0000 7fb350973700 1 -- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb334038f20 msgr2=0x7fb33403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/3876208039 2024-03-18T22:33:44.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.237+0000 7fb350973700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb334038f20 0x7fb33403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T22:33:44.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.638+0000 7fb350973700 1 -- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb334038f20 msgr2=0x7fb33403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/3876208039 2024-03-18T22:33:44.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:38.638+0000 7fb350973700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb334038f20 0x7fb33403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T22:33:44.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:39.439+0000 7fb350973700 1 -- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb334038f20 msgr2=0x7fb33403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/3876208039 2024-03-18T22:33:44.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:39.439+0000 7fb350973700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb334038f20 0x7fb33403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T22:33:44.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:41.040+0000 7fb350973700 1 -- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb334038f20 msgr2=0x7fb33403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.40:6800/3876208039 2024-03-18T22:33:44.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:41.040+0000 7fb350973700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb334038f20 0x7fb33403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T22:33:44.678 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:42.722+0000 7fb34a7fc700 1 -- 172.21.15.40:0/1283663578 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7fb33c01c6c0 con 0x7fb34c106a70 2024-03-18T22:33:44.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:42.722+0000 7fb34a7fc700 1 -- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb334038f20 msgr2=0x7fb33403b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T22:33:44.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:42.722+0000 7fb34a7fc700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb334038f20 0x7fb33403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:44.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.724+0000 7fb34a7fc700 1 -- 172.21.15.40:0/1283663578 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7fb33c01ca40 con 0x7fb34c106a70 2024-03-18T22:33:44.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.724+0000 7fb34a7fc700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb33403c9a0 0x7fb33403ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:44.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.724+0000 7fb34a7fc700 1 -- 172.21.15.40:0/1283663578 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb33c00fec0 con 0x7fb33403c9a0 2024-03-18T22:33:44.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.725+0000 7fb350973700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb33403c9a0 0x7fb33403ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:44.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.726+0000 7fb350973700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb33403c9a0 0x7fb33403ed90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb340003a10 tx=0x7fb3400092b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:44.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.726+0000 7fb34a7fc700 1 -- 172.21.15.40:0/1283663578 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fb33c00fec0 con 0x7fb33403c9a0 2024-03-18T22:33:44.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.738+0000 7fb3533d8700 1 -- 172.21.15.40:0/1283663578 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fb34c063940 con 0x7fb33403c9a0 2024-03-18T22:33:44.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.738+0000 7fb34a7fc700 1 -- 172.21.15.40:0/1283663578 <== mgr.14148 v2:172.21.15.40:6800/3109254132 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fb34c063940 con 0x7fb33403c9a0 2024-03-18T22:33:44.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.738+0000 7fb3533d8700 1 -- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb33403c9a0 msgr2=0x7fb33403ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:44.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.738+0000 7fb3533d8700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb33403c9a0 0x7fb33403ed90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb340003a10 tx=0x7fb3400092b0 comp rx=0 tx=0).stop 2024-03-18T22:33:44.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.738+0000 7fb3533d8700 1 -- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb34c106a70 msgr2=0x7fb34c191be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:44.679 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.738+0000 7fb3533d8700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb34c106a70 0x7fb34c191be0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb33c0072a0 tx=0x7fb33c003d00 comp rx=0 tx=0).stop 2024-03-18T22:33:44.680 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.739+0000 7fb3533d8700 1 -- 172.21.15.40:0/1283663578 shutdown_connections 2024-03-18T22:33:44.680 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.739+0000 7fb3533d8700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb33403c9a0 0x7fb33403ed90 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:44.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.739+0000 7fb3533d8700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb34c106a70 0x7fb34c191be0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:44.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.739+0000 7fb3533d8700 1 --2- 172.21.15.40:0/1283663578 >> [v2:172.21.15.40:6800/3876208039,v1:172.21.15.40:6801/3876208039] conn(0x7fb334038f20 0x7fb33403b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:44.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.739+0000 7fb3533d8700 1 -- 172.21.15.40:0/1283663578 >> 172.21.15.40:0/1283663578 conn(0x7fb34c102010 msgr2=0x7fb34c102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:44.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.739+0000 7fb3533d8700 1 -- 172.21.15.40:0/1283663578 shutdown_connections 2024-03-18T22:33:44.681 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:43.739+0000 7fb3533d8700 1 -- 172.21.15.40:0/1283663578 wait complete. 2024-03-18T22:33:44.681 INFO:teuthology.orchestra.run.smithi040.stdout:mgr epoch 9 is available 2024-03-18T22:33:44.681 INFO:teuthology.orchestra.run.smithi040.stdout:Generating a dashboard self-signed certificate... 2024-03-18T22:33:44.958 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:44 smithi040 ceph-mon[62643]: [18/Mar/2024:22:33:44] ENGINE Bus STARTING 2024-03-18T22:33:44.958 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:44 smithi040 ceph-mon[62643]: [18/Mar/2024:22:33:44] ENGINE Serving on https://172.21.15.40:7150 2024-03-18T22:33:44.958 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:44 smithi040 ceph-mon[62643]: [18/Mar/2024:22:33:44] ENGINE Serving on http://172.21.15.40:8765 2024-03-18T22:33:44.958 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:44 smithi040 ceph-mon[62643]: [18/Mar/2024:22:33:44] ENGINE Bus STARTED 2024-03-18T22:33:45.894 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-18T22:33:45.894 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.055+0000 7f4cd334e700 1 Processor -- start 2024-03-18T22:33:45.894 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.055+0000 7f4cd334e700 1 -- start start 2024-03-18T22:33:45.894 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.055+0000 7f4cd334e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ccc107b50 0x7f4ccc107f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:45.894 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.055+0000 7f4cd334e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc108540 con 0x7f4ccc107b50 2024-03-18T22:33:45.895 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.056+0000 7f4cd10ea700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ccc107b50 0x7f4ccc107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:45.895 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.056+0000 7f4cd10ea700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ccc107b50 0x7f4ccc107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50480/0 (socket says 172.21.15.40:50480) 2024-03-18T22:33:45.895 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.056+0000 7f4cd10ea700 1 -- 172.21.15.40:0/2201827376 learned_addr learned my addr 172.21.15.40:0/2201827376 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:45.895 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.056+0000 7f4cd10ea700 1 -- 172.21.15.40:0/2201827376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ccc108d50 con 0x7f4ccc107b50 2024-03-18T22:33:45.895 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.056+0000 7f4cd10ea700 1 --2- 172.21.15.40:0/2201827376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ccc107b50 0x7f4ccc107f70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc009e80 tx=0x7f4cbc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4adc93d6836ad2e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:45.895 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.056+0000 7f4ccbfff700 1 -- 172.21.15.40:0/2201827376 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cbc017070 con 0x7f4ccc107b50 2024-03-18T22:33:45.895 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.056+0000 7f4ccbfff700 1 -- 172.21.15.40:0/2201827376 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cbc00b8c0 con 0x7f4ccc107b50 2024-03-18T22:33:45.895 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.056+0000 7f4ccbfff700 1 -- 172.21.15.40:0/2201827376 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cbc004490 con 0x7f4ccc107b50 2024-03-18T22:33:45.895 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.057+0000 7f4cd334e700 1 -- 172.21.15.40:0/2201827376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ccc107b50 msgr2=0x7f4ccc107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:45.895 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.057+0000 7f4cd334e700 1 --2- 172.21.15.40:0/2201827376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ccc107b50 0x7f4ccc107f70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc009e80 tx=0x7f4cbc00b330 comp rx=0 tx=0).stop 2024-03-18T22:33:45.895 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.057+0000 7f4cd334e700 1 -- 172.21.15.40:0/2201827376 shutdown_connections 2024-03-18T22:33:45.895 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.057+0000 7f4cd334e700 1 --2- 172.21.15.40:0/2201827376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ccc107b50 0x7f4ccc107f70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:45.895 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.057+0000 7f4cd334e700 1 -- 172.21.15.40:0/2201827376 >> 172.21.15.40:0/2201827376 conn(0x7f4ccc1030f0 msgr2=0x7f4ccc105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:45.895 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.057+0000 7f4cd334e700 1 -- 172.21.15.40:0/2201827376 shutdown_connections 2024-03-18T22:33:45.895 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.057+0000 7f4cd334e700 1 -- 172.21.15.40:0/2201827376 wait complete. 2024-03-18T22:33:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.058+0000 7f4cd334e700 1 Processor -- start 2024-03-18T22:33:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.058+0000 7f4cd334e700 1 -- start start 2024-03-18T22:33:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.058+0000 7f4cd334e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ccc107b50 0x7f4ccc19ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.058+0000 7f4cd334e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc10c100 con 0x7f4ccc107b50 2024-03-18T22:33:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.058+0000 7f4cd10ea700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ccc107b50 0x7f4ccc19ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.058+0000 7f4cd10ea700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ccc107b50 0x7f4ccc19ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50496/0 (socket says 172.21.15.40:50496) 2024-03-18T22:33:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.058+0000 7f4cd10ea700 1 -- 172.21.15.40:0/653846745 learned_addr learned my addr 172.21.15.40:0/653846745 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.058+0000 7f4cd10ea700 1 -- 172.21.15.40:0/653846745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ccc19bfc0 con 0x7f4ccc107b50 2024-03-18T22:33:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.058+0000 7f4cd10ea700 1 --2- 172.21.15.40:0/653846745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ccc107b50 0x7f4ccc19ba80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0072a0 tx=0x7f4cbc003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.059+0000 7f4cca7fc700 1 -- 172.21.15.40:0/653846745 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cbc020020 con 0x7f4ccc107b50 2024-03-18T22:33:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.059+0000 7f4cd334e700 1 -- 172.21.15.40:0/653846745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ccc19c250 con 0x7f4ccc107b50 2024-03-18T22:33:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.059+0000 7f4cca7fc700 1 -- 172.21.15.40:0/653846745 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cbc00bba0 con 0x7f4ccc107b50 2024-03-18T22:33:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.059+0000 7f4cca7fc700 1 -- 172.21.15.40:0/653846745 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4cbc017040 con 0x7f4ccc107b50 2024-03-18T22:33:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.059+0000 7f4cd334e700 1 -- 172.21.15.40:0/653846745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ccc19ef40 con 0x7f4ccc107b50 2024-03-18T22:33:45.897 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.059+0000 7f4cca7fc700 1 -- 172.21.15.40:0/653846745 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f4cbc023020 con 0x7f4ccc107b50 2024-03-18T22:33:45.897 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.060+0000 7f4cd334e700 1 -- 172.21.15.40:0/653846745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ccc063940 con 0x7f4ccc107b50 2024-03-18T22:33:45.897 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.060+0000 7f4cca7fc700 1 --2- 172.21.15.40:0/653846745 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4cb4038f20 0x7f4cb403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:45.897 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.060+0000 7f4cca7fc700 1 -- 172.21.15.40:0/653846745 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4cbc00f050 con 0x7f4ccc107b50 2024-03-18T22:33:45.897 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.060+0000 7f4cd08e9700 1 --2- 172.21.15.40:0/653846745 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4cb4038f20 0x7f4cb403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:45.897 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.060+0000 7f4cd08e9700 1 --2- 172.21.15.40:0/653846745 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4cb4038f20 0x7f4cb403b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0006fd0 tx=0x7f4cc0006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:45.897 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.065+0000 7f4cca7fc700 1 -- 172.21.15.40:0/653846745 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4cbc02a020 con 0x7f4ccc107b50 2024-03-18T22:33:45.897 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.283+0000 7f4cd334e700 1 -- 172.21.15.40:0/653846745 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f4ccc1042d0 con 0x7f4cb4038f20 2024-03-18T22:33:45.897 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.373+0000 7f4cca7fc700 1 -- 172.21.15.40:0/653846745 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f4ccc1042d0 con 0x7f4cb4038f20 2024-03-18T22:33:45.897 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.375+0000 7f4cd334e700 1 -- 172.21.15.40:0/653846745 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4cb4038f20 msgr2=0x7f4cb403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:45.897 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.375+0000 7f4cd334e700 1 --2- 172.21.15.40:0/653846745 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4cb4038f20 0x7f4cb403b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0006fd0 tx=0x7f4cc0006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:45.897 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.375+0000 7f4cd334e700 1 -- 172.21.15.40:0/653846745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ccc107b50 msgr2=0x7f4ccc19ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:45.897 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.375+0000 7f4cd334e700 1 --2- 172.21.15.40:0/653846745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ccc107b50 0x7f4ccc19ba80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0072a0 tx=0x7f4cbc003d00 comp rx=0 tx=0).stop 2024-03-18T22:33:45.897 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.375+0000 7f4cd334e700 1 -- 172.21.15.40:0/653846745 shutdown_connections 2024-03-18T22:33:45.898 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.375+0000 7f4cd334e700 1 --2- 172.21.15.40:0/653846745 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4cb4038f20 0x7f4cb403b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:45.898 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.375+0000 7f4cd334e700 1 --2- 172.21.15.40:0/653846745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ccc107b50 0x7f4ccc19ba80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:45.898 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.375+0000 7f4cd334e700 1 -- 172.21.15.40:0/653846745 >> 172.21.15.40:0/653846745 conn(0x7f4ccc1030f0 msgr2=0x7f4ccc103bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:45.898 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.375+0000 7f4cd334e700 1 -- 172.21.15.40:0/653846745 shutdown_connections 2024-03-18T22:33:45.898 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:45.375+0000 7f4cd334e700 1 -- 172.21.15.40:0/653846745 wait complete. 2024-03-18T22:33:45.898 INFO:teuthology.orchestra.run.smithi040.stdout:Creating initial admin user... 2024-03-18T22:33:46.105 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:45 smithi040 ceph-mon[62643]: mgrmap e12: x(active, since 2s) 2024-03-18T22:33:46.106 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:45 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:33:46.106 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:45 smithi040 ceph-mon[62643]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:33:46.106 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:45 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:33:46.106 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:45 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:33:46.810 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:46 smithi040 ceph-mon[62643]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:33:47.337 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$ayx.KCq7LE/7EyGxBCCFkObNFqvHl6kn348xevj87Bw/nMj8uC012", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710801226, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-18T22:33:47.337 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.293+0000 7f51e6e84700 1 Processor -- start 2024-03-18T22:33:47.337 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.293+0000 7f51e6e84700 1 -- start start 2024-03-18T22:33:47.337 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.293+0000 7f51e6e84700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e0108600 0x7f51e010a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:47.337 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.293+0000 7f51e6e84700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f51e007be40 con 0x7f51e0108600 2024-03-18T22:33:47.337 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.293+0000 7f51e4c20700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e0108600 0x7f51e010a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:47.337 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.293+0000 7f51e4c20700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e0108600 0x7f51e010a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50508/0 (socket says 172.21.15.40:50508) 2024-03-18T22:33:47.337 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.293+0000 7f51e4c20700 1 -- 172.21.15.40:0/709764464 learned_addr learned my addr 172.21.15.40:0/709764464 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:47.337 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.293+0000 7f51e4c20700 1 -- 172.21.15.40:0/709764464 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e010af30 con 0x7f51e0108600 2024-03-18T22:33:47.337 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.293+0000 7f51e4c20700 1 --2- 172.21.15.40:0/709764464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e0108600 0x7f51e010a9f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f51d00098a0 tx=0x7f51d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0b96e57f913afe7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:47.337 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.294+0000 7f51df7fe700 1 -- 172.21.15.40:0/709764464 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51d0004160 con 0x7f51e0108600 2024-03-18T22:33:47.338 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.294+0000 7f51df7fe700 1 -- 172.21.15.40:0/709764464 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51d0004300 con 0x7f51e0108600 2024-03-18T22:33:47.338 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.294+0000 7f51df7fe700 1 -- 172.21.15.40:0/709764464 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51d0011a50 con 0x7f51e0108600 2024-03-18T22:33:47.338 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.294+0000 7f51e6e84700 1 -- 172.21.15.40:0/709764464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e0108600 msgr2=0x7f51e010a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:47.338 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.294+0000 7f51e6e84700 1 --2- 172.21.15.40:0/709764464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e0108600 0x7f51e010a9f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f51d00098a0 tx=0x7f51d0009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:47.338 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.294+0000 7f51e6e84700 1 -- 172.21.15.40:0/709764464 shutdown_connections 2024-03-18T22:33:47.338 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.294+0000 7f51e6e84700 1 --2- 172.21.15.40:0/709764464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e0108600 0x7f51e010a9f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:47.338 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.294+0000 7f51e6e84700 1 -- 172.21.15.40:0/709764464 >> 172.21.15.40:0/709764464 conn(0x7f51e0101ff0 msgr2=0x7f51e0104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:47.338 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.294+0000 7f51e6e84700 1 -- 172.21.15.40:0/709764464 shutdown_connections 2024-03-18T22:33:47.338 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.294+0000 7f51e6e84700 1 -- 172.21.15.40:0/709764464 wait complete. 2024-03-18T22:33:47.339 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.295+0000 7f51e6e84700 1 Processor -- start 2024-03-18T22:33:47.339 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.295+0000 7f51e6e84700 1 -- start start 2024-03-18T22:33:47.339 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.295+0000 7f51e6e84700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e0108600 0x7f51e019ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:47.339 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.295+0000 7f51e6e84700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f51e010bf30 con 0x7f51e0108600 2024-03-18T22:33:47.339 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.295+0000 7f51e4c20700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e0108600 0x7f51e019ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:47.339 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.295+0000 7f51e4c20700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e0108600 0x7f51e019ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50522/0 (socket says 172.21.15.40:50522) 2024-03-18T22:33:47.339 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.295+0000 7f51e4c20700 1 -- 172.21.15.40:0/1557523615 learned_addr learned my addr 172.21.15.40:0/1557523615 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:47.339 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.295+0000 7f51e4c20700 1 -- 172.21.15.40:0/1557523615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e019bfa0 con 0x7f51e0108600 2024-03-18T22:33:47.339 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.296+0000 7f51e4c20700 1 --2- 172.21.15.40:0/1557523615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e0108600 0x7f51e019ba60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f51d0011a20 tx=0x7f51d0011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:47.339 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.296+0000 7f51ddffb700 1 -- 172.21.15.40:0/1557523615 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51d00039c0 con 0x7f51e0108600 2024-03-18T22:33:47.339 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.296+0000 7f51e6e84700 1 -- 172.21.15.40:0/1557523615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51e019c230 con 0x7f51e0108600 2024-03-18T22:33:47.339 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.296+0000 7f51ddffb700 1 -- 172.21.15.40:0/1557523615 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51d0003b60 con 0x7f51e0108600 2024-03-18T22:33:47.339 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.296+0000 7f51ddffb700 1 -- 172.21.15.40:0/1557523615 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51d001a6b0 con 0x7f51e0108600 2024-03-18T22:33:47.340 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.296+0000 7f51e6e84700 1 -- 172.21.15.40:0/1557523615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51e019ef20 con 0x7f51e0108600 2024-03-18T22:33:47.340 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.297+0000 7f51ddffb700 1 -- 172.21.15.40:0/1557523615 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f51d0024430 con 0x7f51e0108600 2024-03-18T22:33:47.340 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.297+0000 7f51e6e84700 1 -- 172.21.15.40:0/1557523615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51e0063940 con 0x7f51e0108600 2024-03-18T22:33:47.340 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.297+0000 7f51ddffb700 1 --2- 172.21.15.40:0/1557523615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51c8038f20 0x7f51c803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:47.340 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.297+0000 7f51ddffb700 1 -- 172.21.15.40:0/1557523615 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f51d004d5b0 con 0x7f51e0108600 2024-03-18T22:33:47.340 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.297+0000 7f51dffff700 1 --2- 172.21.15.40:0/1557523615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51c8038f20 0x7f51c803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:47.340 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.297+0000 7f51dffff700 1 --2- 172.21.15.40:0/1557523615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51c8038f20 0x7f51c803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f51d4006fd0 tx=0x7f51d4006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:47.340 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.302+0000 7f51ddffb700 1 -- 172.21.15.40:0/1557523615 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f51d0024a70 con 0x7f51e0108600 2024-03-18T22:33:47.340 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.544+0000 7f51e6e84700 1 -- 172.21.15.40:0/1557523615 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f51e019c710 con 0x7f51c8038f20 2024-03-18T22:33:47.340 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.791+0000 7f51ddffb700 1 -- 172.21.15.40:0/1557523615 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f51e019c710 con 0x7f51c8038f20 2024-03-18T22:33:47.340 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.793+0000 7f51e6e84700 1 -- 172.21.15.40:0/1557523615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51c8038f20 msgr2=0x7f51c803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:47.340 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.793+0000 7f51e6e84700 1 --2- 172.21.15.40:0/1557523615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51c8038f20 0x7f51c803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f51d4006fd0 tx=0x7f51d4006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:47.340 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.793+0000 7f51e6e84700 1 -- 172.21.15.40:0/1557523615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e0108600 msgr2=0x7f51e019ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:47.340 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.793+0000 7f51e6e84700 1 --2- 172.21.15.40:0/1557523615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e0108600 0x7f51e019ba60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f51d0011a20 tx=0x7f51d0011b00 comp rx=0 tx=0).stop 2024-03-18T22:33:47.341 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.793+0000 7f51e6e84700 1 -- 172.21.15.40:0/1557523615 shutdown_connections 2024-03-18T22:33:47.341 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.793+0000 7f51e6e84700 1 --2- 172.21.15.40:0/1557523615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51c8038f20 0x7f51c803b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:47.341 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.793+0000 7f51e6e84700 1 --2- 172.21.15.40:0/1557523615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e0108600 0x7f51e019ba60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:47.341 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.793+0000 7f51e6e84700 1 -- 172.21.15.40:0/1557523615 >> 172.21.15.40:0/1557523615 conn(0x7f51e0101ff0 msgr2=0x7f51e0102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:47.341 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.793+0000 7f51e6e84700 1 -- 172.21.15.40:0/1557523615 shutdown_connections 2024-03-18T22:33:47.341 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:46.793+0000 7f51e6e84700 1 -- 172.21.15.40:0/1557523615 wait complete. 2024-03-18T22:33:47.341 INFO:teuthology.orchestra.run.smithi040.stdout:Fetching dashboard port number... 2024-03-18T22:33:47.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:47 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:33:47.973 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:47 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:33:48.620 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stdout 8443 2024-03-18T22:33:48.620 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.749+0000 7f6547fc2700 1 Processor -- start 2024-03-18T22:33:48.620 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.749+0000 7f6547fc2700 1 -- start start 2024-03-18T22:33:48.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.749+0000 7f6547fc2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6540108620 0x7f654010aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:48.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.749+0000 7f6547fc2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f654007be20 con 0x7f6540108620 2024-03-18T22:33:48.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.749+0000 7f6545d5e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6540108620 0x7f654010aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:48.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.749+0000 7f6545d5e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6540108620 0x7f654010aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50536/0 (socket says 172.21.15.40:50536) 2024-03-18T22:33:48.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.750+0000 7f6545d5e700 1 -- 172.21.15.40:0/788271955 learned_addr learned my addr 172.21.15.40:0/788271955 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:48.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.750+0000 7f6545d5e700 1 -- 172.21.15.40:0/788271955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f654010af50 con 0x7f6540108620 2024-03-18T22:33:48.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.750+0000 7f6545d5e700 1 --2- 172.21.15.40:0/788271955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6540108620 0x7f654010aa10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f653c0098a0 tx=0x7f653c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10fe28faf8709ca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:48.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.750+0000 7f6544d5c700 1 -- 172.21.15.40:0/788271955 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f653c004160 con 0x7f6540108620 2024-03-18T22:33:48.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.750+0000 7f6544d5c700 1 -- 172.21.15.40:0/788271955 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f653c004300 con 0x7f6540108620 2024-03-18T22:33:48.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.750+0000 7f6544d5c700 1 -- 172.21.15.40:0/788271955 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f653c011a50 con 0x7f6540108620 2024-03-18T22:33:48.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.751+0000 7f6547fc2700 1 -- 172.21.15.40:0/788271955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6540108620 msgr2=0x7f654010aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:48.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.751+0000 7f6547fc2700 1 --2- 172.21.15.40:0/788271955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6540108620 0x7f654010aa10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f653c0098a0 tx=0x7f653c009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:48.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.751+0000 7f6547fc2700 1 -- 172.21.15.40:0/788271955 shutdown_connections 2024-03-18T22:33:48.621 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.751+0000 7f6547fc2700 1 --2- 172.21.15.40:0/788271955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6540108620 0x7f654010aa10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.751+0000 7f6547fc2700 1 -- 172.21.15.40:0/788271955 >> 172.21.15.40:0/788271955 conn(0x7f6540101ff0 msgr2=0x7f6540104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.751+0000 7f6547fc2700 1 -- 172.21.15.40:0/788271955 shutdown_connections 2024-03-18T22:33:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.751+0000 7f6547fc2700 1 -- 172.21.15.40:0/788271955 wait complete. 2024-03-18T22:33:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.751+0000 7f6547fc2700 1 Processor -- start 2024-03-18T22:33:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.752+0000 7f6547fc2700 1 -- start start 2024-03-18T22:33:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.752+0000 7f6547fc2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6540108620 0x7f6540107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.752+0000 7f6547fc2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f654010bf00 con 0x7f6540108620 2024-03-18T22:33:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.752+0000 7f6545d5e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6540108620 0x7f6540107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.752+0000 7f6545d5e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6540108620 0x7f6540107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50548/0 (socket says 172.21.15.40:50548) 2024-03-18T22:33:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.752+0000 7f6545d5e700 1 -- 172.21.15.40:0/773941517 learned_addr learned my addr 172.21.15.40:0/773941517 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.752+0000 7f6545d5e700 1 -- 172.21.15.40:0/773941517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65401083e0 con 0x7f6540108620 2024-03-18T22:33:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.752+0000 7f6545d5e700 1 --2- 172.21.15.40:0/773941517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6540108620 0x7f6540107ea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f653c011a00 tx=0x7f653c011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.752+0000 7f6536ffd700 1 -- 172.21.15.40:0/773941517 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f653c003af0 con 0x7f6540108620 2024-03-18T22:33:48.622 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.752+0000 7f6536ffd700 1 -- 172.21.15.40:0/773941517 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f653c01a8e0 con 0x7f6540108620 2024-03-18T22:33:48.623 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.752+0000 7f6547fc2700 1 -- 172.21.15.40:0/773941517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65401065d0 con 0x7f6540108620 2024-03-18T22:33:48.623 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.752+0000 7f6536ffd700 1 -- 172.21.15.40:0/773941517 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f653c024840 con 0x7f6540108620 2024-03-18T22:33:48.623 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.753+0000 7f6547fc2700 1 -- 172.21.15.40:0/773941517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6540106b10 con 0x7f6540108620 2024-03-18T22:33:48.623 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.753+0000 7f6536ffd700 1 -- 172.21.15.40:0/773941517 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f653c022020 con 0x7f6540108620 2024-03-18T22:33:48.623 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.753+0000 7f6547fc2700 1 -- 172.21.15.40:0/773941517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6540063940 con 0x7f6540108620 2024-03-18T22:33:48.623 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.754+0000 7f6536ffd700 1 --2- 172.21.15.40:0/773941517 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f652c038fd0 0x7f652c03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:48.623 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.754+0000 7f6536ffd700 1 -- 172.21.15.40:0/773941517 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f653c04d330 con 0x7f6540108620 2024-03-18T22:33:48.623 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.754+0000 7f654555d700 1 --2- 172.21.15.40:0/773941517 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f652c038fd0 0x7f652c03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:48.623 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.754+0000 7f654555d700 1 --2- 172.21.15.40:0/773941517 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f652c038fd0 0x7f652c03b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6530006fd0 tx=0x7f6530006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:48.623 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.759+0000 7f6536ffd700 1 -- 172.21.15.40:0/773941517 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f653c016260 con 0x7f6540108620 2024-03-18T22:33:48.623 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.971+0000 7f6547fc2700 1 -- 172.21.15.40:0/773941517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f6540106e50 con 0x7f6540108620 2024-03-18T22:33:48.623 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.972+0000 7f6536ffd700 1 -- 172.21.15.40:0/773941517 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f6540106e50 con 0x7f6540108620 2024-03-18T22:33:48.624 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.974+0000 7f6547fc2700 1 -- 172.21.15.40:0/773941517 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f652c038fd0 msgr2=0x7f652c03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:48.624 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.974+0000 7f6547fc2700 1 --2- 172.21.15.40:0/773941517 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f652c038fd0 0x7f652c03b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6530006fd0 tx=0x7f6530006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:48.624 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.974+0000 7f6547fc2700 1 -- 172.21.15.40:0/773941517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6540108620 msgr2=0x7f6540107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:48.624 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.974+0000 7f6547fc2700 1 --2- 172.21.15.40:0/773941517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6540108620 0x7f6540107ea0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f653c011a00 tx=0x7f653c011ae0 comp rx=0 tx=0).stop 2024-03-18T22:33:48.624 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.974+0000 7f6547fc2700 1 -- 172.21.15.40:0/773941517 shutdown_connections 2024-03-18T22:33:48.624 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.974+0000 7f6547fc2700 1 --2- 172.21.15.40:0/773941517 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f652c038fd0 0x7f652c03b490 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:48.624 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.974+0000 7f6547fc2700 1 --2- 172.21.15.40:0/773941517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6540108620 0x7f6540107ea0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:48.624 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.974+0000 7f6547fc2700 1 -- 172.21.15.40:0/773941517 >> 172.21.15.40:0/773941517 conn(0x7f6540101ff0 msgr2=0x7f6540102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:48.624 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.974+0000 7f6547fc2700 1 -- 172.21.15.40:0/773941517 shutdown_connections 2024-03-18T22:33:48.624 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:47.974+0000 7f6547fc2700 1 -- 172.21.15.40:0/773941517 wait complete. 2024-03-18T22:33:48.626 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T22:33:48.627 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout disabled 2024-03-18T22:33:48.633 INFO:teuthology.orchestra.run.smithi040.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T22:33:48.634 INFO:teuthology.orchestra.run.smithi040.stdout:systemctl: stdout inactive 2024-03-18T22:33:48.634 INFO:teuthology.orchestra.run.smithi040.stdout:firewalld.service is not enabled 2024-03-18T22:33:48.634 INFO:teuthology.orchestra.run.smithi040.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-18T22:33:48.635 INFO:teuthology.orchestra.run.smithi040.stdout:Ceph Dashboard is now available at: 2024-03-18T22:33:48.636 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:33:48.636 INFO:teuthology.orchestra.run.smithi040.stdout: URL: https://smithi040.front.sepia.ceph.com:8443/ 2024-03-18T22:33:48.636 INFO:teuthology.orchestra.run.smithi040.stdout: User: admin 2024-03-18T22:33:48.636 INFO:teuthology.orchestra.run.smithi040.stdout: Password: yw358j5hw5 2024-03-18T22:33:48.636 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:33:48.636 INFO:teuthology.orchestra.run.smithi040.stdout:Saving cluster configuration to /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/config directory 2024-03-18T22:33:48.636 INFO:teuthology.orchestra.run.smithi040.stdout:Enabling autotune for osd_memory_target 2024-03-18T22:33:48.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:48 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/773941517' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-18T22:33:49.886 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:49 smithi040 ceph-mon[62643]: mgrmap e13: x(active, since 6s) 2024-03-18T22:33:50.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.659+0000 7f1497e1c700 1 Processor -- start 2024-03-18T22:33:50.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.659+0000 7f1497e1c700 1 -- start start 2024-03-18T22:33:50.397 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.659+0000 7f1497e1c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1490106450 0x7f1490106870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:50.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.659+0000 7f1497e1c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f149007bde0 con 0x7f1490106450 2024-03-18T22:33:50.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.659+0000 7f1495bb8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1490106450 0x7f1490106870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:50.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.659+0000 7f1495bb8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1490106450 0x7f1490106870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55992/0 (socket says 172.21.15.40:55992) 2024-03-18T22:33:50.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.659+0000 7f1495bb8700 1 -- 172.21.15.40:0/1063216016 learned_addr learned my addr 172.21.15.40:0/1063216016 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:50.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.660+0000 7f1495bb8700 1 -- 172.21.15.40:0/1063216016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1490106db0 con 0x7f1490106450 2024-03-18T22:33:50.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.660+0000 7f1495bb8700 1 --2- 172.21.15.40:0/1063216016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1490106450 0x7f1490106870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f148c0098a0 tx=0x7f148c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd8f9edbea029c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:50.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.660+0000 7f1494bb6700 1 -- 172.21.15.40:0/1063216016 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f148c004160 con 0x7f1490106450 2024-03-18T22:33:50.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.660+0000 7f1494bb6700 1 -- 172.21.15.40:0/1063216016 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f148c004300 con 0x7f1490106450 2024-03-18T22:33:50.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.660+0000 7f1494bb6700 1 -- 172.21.15.40:0/1063216016 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f148c011a50 con 0x7f1490106450 2024-03-18T22:33:50.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.660+0000 7f1497e1c700 1 -- 172.21.15.40:0/1063216016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1490106450 msgr2=0x7f1490106870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:50.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.660+0000 7f1497e1c700 1 --2- 172.21.15.40:0/1063216016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1490106450 0x7f1490106870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f148c0098a0 tx=0x7f148c009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:50.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.660+0000 7f1497e1c700 1 -- 172.21.15.40:0/1063216016 shutdown_connections 2024-03-18T22:33:50.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.660+0000 7f1497e1c700 1 --2- 172.21.15.40:0/1063216016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1490106450 0x7f1490106870 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:50.398 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.660+0000 7f1497e1c700 1 -- 172.21.15.40:0/1063216016 >> 172.21.15.40:0/1063216016 conn(0x7f1490101ff0 msgr2=0x7f1490104430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:50.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.661+0000 7f1497e1c700 1 -- 172.21.15.40:0/1063216016 shutdown_connections 2024-03-18T22:33:50.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.661+0000 7f1497e1c700 1 -- 172.21.15.40:0/1063216016 wait complete. 2024-03-18T22:33:50.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.661+0000 7f1497e1c700 1 Processor -- start 2024-03-18T22:33:50.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.661+0000 7f1497e1c700 1 -- start start 2024-03-18T22:33:50.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.661+0000 7f1497e1c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1490106450 0x7f149019dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:50.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.661+0000 7f1497e1c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1490107e20 con 0x7f1490106450 2024-03-18T22:33:50.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.661+0000 7f1495bb8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1490106450 0x7f149019dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:50.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.661+0000 7f1495bb8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1490106450 0x7f149019dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55996/0 (socket says 172.21.15.40:55996) 2024-03-18T22:33:50.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.661+0000 7f1495bb8700 1 -- 172.21.15.40:0/1190936445 learned_addr learned my addr 172.21.15.40:0/1190936445 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:50.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.662+0000 7f1495bb8700 1 -- 172.21.15.40:0/1190936445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149019e170 con 0x7f1490106450 2024-03-18T22:33:50.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.662+0000 7f1495bb8700 1 --2- 172.21.15.40:0/1190936445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1490106450 0x7f149019dc30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f148c011a00 tx=0x7f148c011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:50.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.662+0000 7f1486ffd700 1 -- 172.21.15.40:0/1190936445 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f148c003af0 con 0x7f1490106450 2024-03-18T22:33:50.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.662+0000 7f1486ffd700 1 -- 172.21.15.40:0/1190936445 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f148c01a8e0 con 0x7f1490106450 2024-03-18T22:33:50.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.662+0000 7f1497e1c700 1 -- 172.21.15.40:0/1190936445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f149019e400 con 0x7f1490106450 2024-03-18T22:33:50.399 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.662+0000 7f1486ffd700 1 -- 172.21.15.40:0/1190936445 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f148c024840 con 0x7f1490106450 2024-03-18T22:33:50.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.662+0000 7f1497e1c700 1 -- 172.21.15.40:0/1190936445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14901a10f0 con 0x7f1490106450 2024-03-18T22:33:50.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.663+0000 7f1486ffd700 1 -- 172.21.15.40:0/1190936445 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f148c022020 con 0x7f1490106450 2024-03-18T22:33:50.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.663+0000 7f1486ffd700 1 --2- 172.21.15.40:0/1190936445 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f147c039010 0x7f147c03b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:50.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.663+0000 7f1486ffd700 1 -- 172.21.15.40:0/1190936445 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f148c04d430 con 0x7f1490106450 2024-03-18T22:33:50.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.663+0000 7f1497e1c700 1 -- 172.21.15.40:0/1190936445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1490063940 con 0x7f1490106450 2024-03-18T22:33:50.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.663+0000 7f14953b7700 1 --2- 172.21.15.40:0/1190936445 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f147c039010 0x7f147c03b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:50.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.664+0000 7f14953b7700 1 --2- 172.21.15.40:0/1190936445 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f147c039010 0x7f147c03b4d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1480006fd0 tx=0x7f1480006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:50.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.668+0000 7f1486ffd700 1 -- 172.21.15.40:0/1190936445 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f148c019c70 con 0x7f1490106450 2024-03-18T22:33:50.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.871+0000 7f1497e1c700 1 -- 172.21.15.40:0/1190936445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f149002dda0 con 0x7f1490106450 2024-03-18T22:33:50.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.872+0000 7f1486ffd700 1 -- 172.21.15.40:0/1190936445 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f148c032030 con 0x7f1490106450 2024-03-18T22:33:50.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.874+0000 7f1497e1c700 1 -- 172.21.15.40:0/1190936445 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f147c039010 msgr2=0x7f147c03b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:50.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.874+0000 7f1497e1c700 1 --2- 172.21.15.40:0/1190936445 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f147c039010 0x7f147c03b4d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1480006fd0 tx=0x7f1480006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:50.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.874+0000 7f1497e1c700 1 -- 172.21.15.40:0/1190936445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1490106450 msgr2=0x7f149019dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:50.400 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.874+0000 7f1497e1c700 1 --2- 172.21.15.40:0/1190936445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1490106450 0x7f149019dc30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f148c011a00 tx=0x7f148c011ae0 comp rx=0 tx=0).stop 2024-03-18T22:33:50.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.874+0000 7f1497e1c700 1 -- 172.21.15.40:0/1190936445 shutdown_connections 2024-03-18T22:33:50.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.874+0000 7f1497e1c700 1 --2- 172.21.15.40:0/1190936445 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f147c039010 0x7f147c03b4d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:50.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.874+0000 7f1497e1c700 1 --2- 172.21.15.40:0/1190936445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1490106450 0x7f149019dc30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:50.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.874+0000 7f1497e1c700 1 -- 172.21.15.40:0/1190936445 >> 172.21.15.40:0/1190936445 conn(0x7f1490101ff0 msgr2=0x7f1490194870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:50.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.874+0000 7f1497e1c700 1 -- 172.21.15.40:0/1190936445 shutdown_connections 2024-03-18T22:33:50.401 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:49.874+0000 7f1497e1c700 1 -- 172.21.15.40:0/1190936445 wait complete. 2024-03-18T22:33:51.653 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.751+0000 7f15acd71700 1 Processor -- start 2024-03-18T22:33:51.653 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.751+0000 7f15acd71700 1 -- start start 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.751+0000 7f15acd71700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15a8107b50 0x7f15a8107f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.751+0000 7f15acd71700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f15a8108540 con 0x7f15a8107b50 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.751+0000 7f15a659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15a8107b50 0x7f15a8107f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.751+0000 7f15a659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15a8107b50 0x7f15a8107f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56000/0 (socket says 172.21.15.40:56000) 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.751+0000 7f15a659c700 1 -- 172.21.15.40:0/636217737 learned_addr learned my addr 172.21.15.40:0/636217737 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.752+0000 7f15a659c700 1 -- 172.21.15.40:0/636217737 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a8108d50 con 0x7f15a8107b50 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.752+0000 7f15a659c700 1 --2- 172.21.15.40:0/636217737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15a8107b50 0x7f15a8107f70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f15980098a0 tx=0x7f1598009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f7f3a6735cb0d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.752+0000 7f15a559a700 1 -- 172.21.15.40:0/636217737 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1598004160 con 0x7f15a8107b50 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.752+0000 7f15a559a700 1 -- 172.21.15.40:0/636217737 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1598004300 con 0x7f15a8107b50 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.752+0000 7f15a559a700 1 -- 172.21.15.40:0/636217737 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1598011a50 con 0x7f15a8107b50 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.753+0000 7f15acd71700 1 -- 172.21.15.40:0/636217737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15a8107b50 msgr2=0x7f15a8107f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.753+0000 7f15acd71700 1 --2- 172.21.15.40:0/636217737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15a8107b50 0x7f15a8107f70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f15980098a0 tx=0x7f1598009bb0 comp rx=0 tx=0).stop 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.753+0000 7f15acd71700 1 -- 172.21.15.40:0/636217737 shutdown_connections 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.753+0000 7f15acd71700 1 --2- 172.21.15.40:0/636217737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15a8107b50 0x7f15a8107f70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.753+0000 7f15acd71700 1 -- 172.21.15.40:0/636217737 >> 172.21.15.40:0/636217737 conn(0x7f15a81030f0 msgr2=0x7f15a8105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:51.654 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.753+0000 7f15acd71700 1 -- 172.21.15.40:0/636217737 shutdown_connections 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.753+0000 7f15acd71700 1 -- 172.21.15.40:0/636217737 wait complete. 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.754+0000 7f15acd71700 1 Processor -- start 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.754+0000 7f15acd71700 1 -- start start 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.754+0000 7f15acd71700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15a8107b50 0x7f15a807db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.754+0000 7f15acd71700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f15a810c100 con 0x7f15a8107b50 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.754+0000 7f15a659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15a8107b50 0x7f15a807db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.754+0000 7f15a659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15a8107b50 0x7f15a807db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56008/0 (socket says 172.21.15.40:56008) 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.754+0000 7f15a659c700 1 -- 172.21.15.40:0/1852940510 learned_addr learned my addr 172.21.15.40:0/1852940510 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.754+0000 7f15a659c700 1 -- 172.21.15.40:0/1852940510 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a807e080 con 0x7f15a8107b50 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.754+0000 7f15a659c700 1 --2- 172.21.15.40:0/1852940510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15a8107b50 0x7f15a807db40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1598011710 tx=0x7f1598011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.754+0000 7f15977fe700 1 -- 172.21.15.40:0/1852940510 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15980039c0 con 0x7f15a8107b50 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.755+0000 7f15977fe700 1 -- 172.21.15.40:0/1852940510 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1598003b60 con 0x7f15a8107b50 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.755+0000 7f15977fe700 1 -- 172.21.15.40:0/1852940510 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f159801a690 con 0x7f15a8107b50 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.755+0000 7f15acd71700 1 -- 172.21.15.40:0/1852940510 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15a807c270 con 0x7f15a8107b50 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.755+0000 7f15acd71700 1 -- 172.21.15.40:0/1852940510 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15a807c7b0 con 0x7f15a8107b50 2024-03-18T22:33:51.655 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.755+0000 7f15977fe700 1 -- 172.21.15.40:0/1852940510 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f159801a940 con 0x7f15a8107b50 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.756+0000 7f15977fe700 1 --2- 172.21.15.40:0/1852940510 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1590039020 0x7f159003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.756+0000 7f15977fe700 1 -- 172.21.15.40:0/1852940510 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f159804d640 con 0x7f15a8107b50 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.756+0000 7f15acd71700 1 -- 172.21.15.40:0/1852940510 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15a8063940 con 0x7f15a8107b50 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.756+0000 7f15a5d9b700 1 --2- 172.21.15.40:0/1852940510 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1590039020 0x7f159003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.756+0000 7f15a5d9b700 1 --2- 172.21.15.40:0/1852940510 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1590039020 0x7f159003b4e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f159c006fd0 tx=0x7f159c006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:50.761+0000 7f15977fe700 1 -- 172.21.15.40:0/1852940510 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1598024a70 con 0x7f15a8107b50 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:51.080+0000 7f15acd71700 1 -- 172.21.15.40:0/1852940510 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f15a807ca30 con 0x7f15a8107b50 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:51.083+0000 7f15977fe700 1 -- 172.21.15.40:0/1852940510 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f1598032030 con 0x7f15a8107b50 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:51.085+0000 7f15acd71700 1 -- 172.21.15.40:0/1852940510 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1590039020 msgr2=0x7f159003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:51.085+0000 7f15acd71700 1 --2- 172.21.15.40:0/1852940510 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1590039020 0x7f159003b4e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f159c006fd0 tx=0x7f159c006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:51.085+0000 7f15acd71700 1 -- 172.21.15.40:0/1852940510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15a8107b50 msgr2=0x7f15a807db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:51.085+0000 7f15acd71700 1 --2- 172.21.15.40:0/1852940510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15a8107b50 0x7f15a807db40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1598011710 tx=0x7f1598011ad0 comp rx=0 tx=0).stop 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:51.085+0000 7f15acd71700 1 -- 172.21.15.40:0/1852940510 shutdown_connections 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:51.085+0000 7f15acd71700 1 --2- 172.21.15.40:0/1852940510 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1590039020 0x7f159003b4e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:51.085+0000 7f15acd71700 1 --2- 172.21.15.40:0/1852940510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15a8107b50 0x7f15a807db40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:51.656 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:51.085+0000 7f15acd71700 1 -- 172.21.15.40:0/1852940510 >> 172.21.15.40:0/1852940510 conn(0x7f15a81030f0 msgr2=0x7f15a8103b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:51.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:51.085+0000 7f15acd71700 1 -- 172.21.15.40:0/1852940510 shutdown_connections 2024-03-18T22:33:51.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr 2024-03-18T22:33:51.085+0000 7f15acd71700 1 -- 172.21.15.40:0/1852940510 wait complete. 2024-03-18T22:33:51.657 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-18T22:33:51.657 INFO:teuthology.orchestra.run.smithi040.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-18T22:33:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:33:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-18T22:33:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:33:51.657 INFO:teuthology.orchestra.run.smithi040.stdout:Or, if you are only running a single cluster on this host: 2024-03-18T22:33:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:33:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-18T22:33:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:33:51.657 INFO:teuthology.orchestra.run.smithi040.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-18T22:33:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:33:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: ceph telemetry on 2024-03-18T22:33:51.657 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:33:51.658 INFO:teuthology.orchestra.run.smithi040.stdout:For more information see: 2024-03-18T22:33:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:33:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-18T22:33:51.658 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:33:51.658 INFO:teuthology.orchestra.run.smithi040.stdout:Bootstrap complete. 2024-03-18T22:33:51.693 INFO:tasks.cephadm:Fetching config... 2024-03-18T22:33:51.693 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:33:51.693 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-18T22:33:51.709 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-18T22:33:51.709 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:33:51.709 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-18T22:33:51.764 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-18T22:33:51.764 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:33:51.764 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-18T22:33:51.831 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-18T22:33:51.831 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:33:51.831 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-18T22:33:51.887 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-18T22:33:51.887 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS3KiRwA6QtO9fhKplNG+AOTYKulH7gCG8zmwiM8QRyHA7v0jA5ejd3bIeYrBZ+GHGVCpdIS5uIn+ZuqtTWwIKJoaZfkdiynrQieKgfmrz0gOoM4XWWeaxxO/F2FURm3VylSfKIg2wJcMODCu0JHFC7Bz6wIrgfHnYspR6jNmMw2gj7C1nN2cK8GrcJ6BgpqyJ3DOjN2N2wPTrSGytjUCvV6OM+nlPwTIwwhCE3he5Z6Awn1BqRmXeSfCILz4XrpK/6tcrAe+srpV6PSDQragYAYz8ZSGNvnOCqZ19efcLMsg+qT2NrDxsOn3DA/3UxSuNZTREiZhOe6aV6QNULQE0qgXVWDlWVgd1FakmbBN28ngGDdWgKnof3D3ZhYvkEkBW8TXOiHrkseyh3jgLwRxdOE29GTzEJoraFbxi1jp+UTPE82HBMvMfJTroSzyBgoRNccrH4QsoQYctBkN0Ktr6cMeCfQx/n7EShleaGSUpTYdadQX17lvtbrzYXHuUQPc= ceph-410ef414-e577-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T22:33:51.965 INFO:teuthology.orchestra.run.smithi040.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS3KiRwA6QtO9fhKplNG+AOTYKulH7gCG8zmwiM8QRyHA7v0jA5ejd3bIeYrBZ+GHGVCpdIS5uIn+ZuqtTWwIKJoaZfkdiynrQieKgfmrz0gOoM4XWWeaxxO/F2FURm3VylSfKIg2wJcMODCu0JHFC7Bz6wIrgfHnYspR6jNmMw2gj7C1nN2cK8GrcJ6BgpqyJ3DOjN2N2wPTrSGytjUCvV6OM+nlPwTIwwhCE3he5Z6Awn1BqRmXeSfCILz4XrpK/6tcrAe+srpV6PSDQragYAYz8ZSGNvnOCqZ19efcLMsg+qT2NrDxsOn3DA/3UxSuNZTREiZhOe6aV6QNULQE0qgXVWDlWVgd1FakmbBN28ngGDdWgKnof3D3ZhYvkEkBW8TXOiHrkseyh3jgLwRxdOE29GTzEJoraFbxi1jp+UTPE82HBMvMfJTroSzyBgoRNccrH4QsoQYctBkN0Ktr6cMeCfQx/n7EShleaGSUpTYdadQX17lvtbrzYXHuUQPc= ceph-410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:33:51.979 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS3KiRwA6QtO9fhKplNG+AOTYKulH7gCG8zmwiM8QRyHA7v0jA5ejd3bIeYrBZ+GHGVCpdIS5uIn+ZuqtTWwIKJoaZfkdiynrQieKgfmrz0gOoM4XWWeaxxO/F2FURm3VylSfKIg2wJcMODCu0JHFC7Bz6wIrgfHnYspR6jNmMw2gj7C1nN2cK8GrcJ6BgpqyJ3DOjN2N2wPTrSGytjUCvV6OM+nlPwTIwwhCE3he5Z6Awn1BqRmXeSfCILz4XrpK/6tcrAe+srpV6PSDQragYAYz8ZSGNvnOCqZ19efcLMsg+qT2NrDxsOn3DA/3UxSuNZTREiZhOe6aV6QNULQE0qgXVWDlWVgd1FakmbBN28ngGDdWgKnof3D3ZhYvkEkBW8TXOiHrkseyh3jgLwRxdOE29GTzEJoraFbxi1jp+UTPE82HBMvMfJTroSzyBgoRNccrH4QsoQYctBkN0Ktr6cMeCfQx/n7EShleaGSUpTYdadQX17lvtbrzYXHuUQPc= ceph-410ef414-e577-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T22:33:52.025 INFO:teuthology.orchestra.run.smithi103.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS3KiRwA6QtO9fhKplNG+AOTYKulH7gCG8zmwiM8QRyHA7v0jA5ejd3bIeYrBZ+GHGVCpdIS5uIn+ZuqtTWwIKJoaZfkdiynrQieKgfmrz0gOoM4XWWeaxxO/F2FURm3VylSfKIg2wJcMODCu0JHFC7Bz6wIrgfHnYspR6jNmMw2gj7C1nN2cK8GrcJ6BgpqyJ3DOjN2N2wPTrSGytjUCvV6OM+nlPwTIwwhCE3he5Z6Awn1BqRmXeSfCILz4XrpK/6tcrAe+srpV6PSDQragYAYz8ZSGNvnOCqZ19efcLMsg+qT2NrDxsOn3DA/3UxSuNZTREiZhOe6aV6QNULQE0qgXVWDlWVgd1FakmbBN28ngGDdWgKnof3D3ZhYvkEkBW8TXOiHrkseyh3jgLwRxdOE29GTzEJoraFbxi1jp+UTPE82HBMvMfJTroSzyBgoRNccrH4QsoQYctBkN0Ktr6cMeCfQx/n7EShleaGSUpTYdadQX17lvtbrzYXHuUQPc= ceph-410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:33:52.040 DEBUG:teuthology.orchestra.run.smithi113:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS3KiRwA6QtO9fhKplNG+AOTYKulH7gCG8zmwiM8QRyHA7v0jA5ejd3bIeYrBZ+GHGVCpdIS5uIn+ZuqtTWwIKJoaZfkdiynrQieKgfmrz0gOoM4XWWeaxxO/F2FURm3VylSfKIg2wJcMODCu0JHFC7Bz6wIrgfHnYspR6jNmMw2gj7C1nN2cK8GrcJ6BgpqyJ3DOjN2N2wPTrSGytjUCvV6OM+nlPwTIwwhCE3he5Z6Awn1BqRmXeSfCILz4XrpK/6tcrAe+srpV6PSDQragYAYz8ZSGNvnOCqZ19efcLMsg+qT2NrDxsOn3DA/3UxSuNZTREiZhOe6aV6QNULQE0qgXVWDlWVgd1FakmbBN28ngGDdWgKnof3D3ZhYvkEkBW8TXOiHrkseyh3jgLwRxdOE29GTzEJoraFbxi1jp+UTPE82HBMvMfJTroSzyBgoRNccrH4QsoQYctBkN0Ktr6cMeCfQx/n7EShleaGSUpTYdadQX17lvtbrzYXHuUQPc= ceph-410ef414-e577-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T22:33:52.086 INFO:teuthology.orchestra.run.smithi113.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS3KiRwA6QtO9fhKplNG+AOTYKulH7gCG8zmwiM8QRyHA7v0jA5ejd3bIeYrBZ+GHGVCpdIS5uIn+ZuqtTWwIKJoaZfkdiynrQieKgfmrz0gOoM4XWWeaxxO/F2FURm3VylSfKIg2wJcMODCu0JHFC7Bz6wIrgfHnYspR6jNmMw2gj7C1nN2cK8GrcJ6BgpqyJ3DOjN2N2wPTrSGytjUCvV6OM+nlPwTIwwhCE3he5Z6Awn1BqRmXeSfCILz4XrpK/6tcrAe+srpV6PSDQragYAYz8ZSGNvnOCqZ19efcLMsg+qT2NrDxsOn3DA/3UxSuNZTREiZhOe6aV6QNULQE0qgXVWDlWVgd1FakmbBN28ngGDdWgKnof3D3ZhYvkEkBW8TXOiHrkseyh3jgLwRxdOE29GTzEJoraFbxi1jp+UTPE82HBMvMfJTroSzyBgoRNccrH4QsoQYctBkN0Ktr6cMeCfQx/n7EShleaGSUpTYdadQX17lvtbrzYXHuUQPc= ceph-410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:33:52.101 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-18T22:33:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:33:52.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:52 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1852940510' entity='client.admin' 2024-03-18T22:33:54.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.476+0000 7f4257483700 1 -- 172.21.15.40:0/2069411923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250100010 msgr2=0x7f4250100430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:54.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.476+0000 7f4257483700 1 --2- 172.21.15.40:0/2069411923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250100010 0x7f4250100430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4244009910 tx=0x7f4244009c20 comp rx=0 tx=0).stop 2024-03-18T22:33:54.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.476+0000 7f4257483700 1 -- 172.21.15.40:0/2069411923 shutdown_connections 2024-03-18T22:33:54.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.476+0000 7f4257483700 1 --2- 172.21.15.40:0/2069411923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250100010 0x7f4250100430 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:54.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.476+0000 7f4257483700 1 -- 172.21.15.40:0/2069411923 >> 172.21.15.40:0/2069411923 conn(0x7f42500fb970 msgr2=0x7f42500fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:54.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.476+0000 7f4257483700 1 -- 172.21.15.40:0/2069411923 shutdown_connections 2024-03-18T22:33:54.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.476+0000 7f4257483700 1 -- 172.21.15.40:0/2069411923 wait complete. 2024-03-18T22:33:54.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.477+0000 7f4257483700 1 Processor -- start 2024-03-18T22:33:54.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.477+0000 7f4257483700 1 -- start start 2024-03-18T22:33:54.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.477+0000 7f4257483700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250100010 0x7f425019d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:54.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.477+0000 7f4257483700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4250107c90 con 0x7f4250100010 2024-03-18T22:33:54.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.477+0000 7f425521f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250100010 0x7f425019d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:54.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.477+0000 7f425521f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250100010 0x7f425019d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56024/0 (socket says 172.21.15.40:56024) 2024-03-18T22:33:54.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.477+0000 7f425521f700 1 -- 172.21.15.40:0/690303576 learned_addr learned my addr 172.21.15.40:0/690303576 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:54.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.477+0000 7f425521f700 1 -- 172.21.15.40:0/690303576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f425019df00 con 0x7f4250100010 2024-03-18T22:33:54.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.477+0000 7f425521f700 1 --2- 172.21.15.40:0/690303576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250100010 0x7f425019d9c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4244005800 tx=0x7f4244009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:54.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.478+0000 7f42427fc700 1 -- 172.21.15.40:0/690303576 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4244027070 con 0x7f4250100010 2024-03-18T22:33:54.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.478+0000 7f4257483700 1 -- 172.21.15.40:0/690303576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f425019e190 con 0x7f4250100010 2024-03-18T22:33:54.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.478+0000 7f42427fc700 1 -- 172.21.15.40:0/690303576 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f424400be10 con 0x7f4250100010 2024-03-18T22:33:54.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.478+0000 7f42427fc700 1 -- 172.21.15.40:0/690303576 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f424400f460 con 0x7f4250100010 2024-03-18T22:33:54.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.478+0000 7f4257483700 1 -- 172.21.15.40:0/690303576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42501a0e80 con 0x7f4250100010 2024-03-18T22:33:54.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.479+0000 7f42427fc700 1 -- 172.21.15.40:0/690303576 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f424402a070 con 0x7f4250100010 2024-03-18T22:33:54.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.479+0000 7f4257483700 1 -- 172.21.15.40:0/690303576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4250108410 con 0x7f4250100010 2024-03-18T22:33:54.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.479+0000 7f42427fc700 1 --2- 172.21.15.40:0/690303576 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f423c038fc0 0x7f423c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:54.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.479+0000 7f42427fc700 1 -- 172.21.15.40:0/690303576 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f424401e030 con 0x7f4250100010 2024-03-18T22:33:54.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.479+0000 7f4254a1e700 1 --2- 172.21.15.40:0/690303576 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f423c038fc0 0x7f423c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:54.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.479+0000 7f4254a1e700 1 --2- 172.21.15.40:0/690303576 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f423c038fc0 0x7f423c03b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f424c006fd0 tx=0x7f424c006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:54.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.484+0000 7f42427fc700 1 -- 172.21.15.40:0/690303576 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4244013070 con 0x7f4250100010 2024-03-18T22:33:54.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.687+0000 7f4257483700 1 -- 172.21.15.40:0/690303576 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f425002dcc0 con 0x7f4250100010 2024-03-18T22:33:54.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.690+0000 7f42427fc700 1 -- 172.21.15.40:0/690303576 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f4244031020 con 0x7f4250100010 2024-03-18T22:33:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.692+0000 7f4257483700 1 -- 172.21.15.40:0/690303576 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f423c038fc0 msgr2=0x7f423c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.692+0000 7f4257483700 1 --2- 172.21.15.40:0/690303576 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f423c038fc0 0x7f423c03b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f424c006fd0 tx=0x7f424c006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.692+0000 7f4257483700 1 -- 172.21.15.40:0/690303576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250100010 msgr2=0x7f425019d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.692+0000 7f4257483700 1 --2- 172.21.15.40:0/690303576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250100010 0x7f425019d9c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4244005800 tx=0x7f4244009f60 comp rx=0 tx=0).stop 2024-03-18T22:33:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.692+0000 7f4257483700 1 -- 172.21.15.40:0/690303576 shutdown_connections 2024-03-18T22:33:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.692+0000 7f4257483700 1 --2- 172.21.15.40:0/690303576 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f423c038fc0 0x7f423c03b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:54.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.692+0000 7f4257483700 1 --2- 172.21.15.40:0/690303576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4250100010 0x7f425019d9c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:54.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.692+0000 7f4257483700 1 -- 172.21.15.40:0/690303576 >> 172.21.15.40:0/690303576 conn(0x7f42500fb970 msgr2=0x7f42500fc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:54.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.692+0000 7f4257483700 1 -- 172.21.15.40:0/690303576 shutdown_connections 2024-03-18T22:33:54.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:54.692+0000 7f4257483700 1 -- 172.21.15.40:0/690303576 wait complete. 2024-03-18T22:33:54.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:54 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:33:54.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:54 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:33:54.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:54 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi040", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:33:54.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:54 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:33:54.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:54 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:33:54.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:54 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:33:55.202 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-18T22:33:55.203 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-18T22:33:55.441 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:33:55.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:55 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/690303576' entity='client.admin' 2024-03-18T22:33:55.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:33:55.829 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:33:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.925+0000 7fbfb4745700 1 -- 172.21.15.40:0/2160616244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac100030 msgr2=0x7fbfac100450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.925+0000 7fbfb4745700 1 --2- 172.21.15.40:0/2160616244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac100030 0x7fbfac100450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8009910 tx=0x7fbfa8009c20 comp rx=0 tx=0).stop 2024-03-18T22:33:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.926+0000 7fbfb4745700 1 -- 172.21.15.40:0/2160616244 shutdown_connections 2024-03-18T22:33:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.926+0000 7fbfb4745700 1 --2- 172.21.15.40:0/2160616244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac100030 0x7fbfac100450 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.926+0000 7fbfb4745700 1 -- 172.21.15.40:0/2160616244 >> 172.21.15.40:0/2160616244 conn(0x7fbfac0fb990 msgr2=0x7fbfac0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.926+0000 7fbfb4745700 1 -- 172.21.15.40:0/2160616244 shutdown_connections 2024-03-18T22:33:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.926+0000 7fbfb4745700 1 -- 172.21.15.40:0/2160616244 wait complete. 2024-03-18T22:33:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.926+0000 7fbfb4745700 1 Processor -- start 2024-03-18T22:33:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.926+0000 7fbfb4745700 1 -- start start 2024-03-18T22:33:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.926+0000 7fbfb4745700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac100030 0x7fbfac102890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.926+0000 7fbfb4745700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbfac107cb0 con 0x7fbfac100030 2024-03-18T22:33:56.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.926+0000 7fbfb24e1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac100030 0x7fbfac102890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:56.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.926+0000 7fbfb24e1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac100030 0x7fbfac102890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56038/0 (socket says 172.21.15.40:56038) 2024-03-18T22:33:56.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.926+0000 7fbfb24e1700 1 -- 172.21.15.40:0/1832582444 learned_addr learned my addr 172.21.15.40:0/1832582444 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:56.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.927+0000 7fbfb24e1700 1 -- 172.21.15.40:0/1832582444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfac102dd0 con 0x7fbfac100030 2024-03-18T22:33:56.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.927+0000 7fbfb24e1700 1 --2- 172.21.15.40:0/1832582444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac100030 0x7fbfac102890 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8005800 tx=0x7fbfa8009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:56.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.927+0000 7fbfa37fe700 1 -- 172.21.15.40:0/1832582444 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbfa8027070 con 0x7fbfac100030 2024-03-18T22:33:56.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.927+0000 7fbfa37fe700 1 -- 172.21.15.40:0/1832582444 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfa800be10 con 0x7fbfac100030 2024-03-18T22:33:56.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.927+0000 7fbfb4745700 1 -- 172.21.15.40:0/1832582444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfac100fc0 con 0x7fbfac100030 2024-03-18T22:33:56.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.927+0000 7fbfa37fe700 1 -- 172.21.15.40:0/1832582444 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbfa800f460 con 0x7fbfac100030 2024-03-18T22:33:56.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.927+0000 7fbfb4745700 1 -- 172.21.15.40:0/1832582444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfac101500 con 0x7fbfac100030 2024-03-18T22:33:56.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.928+0000 7fbfa37fe700 1 -- 172.21.15.40:0/1832582444 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fbfa802a070 con 0x7fbfac100030 2024-03-18T22:33:56.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.928+0000 7fbfb4745700 1 -- 172.21.15.40:0/1832582444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfac108430 con 0x7fbfac100030 2024-03-18T22:33:56.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.928+0000 7fbfa37fe700 1 --2- 172.21.15.40:0/1832582444 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbf98038fc0 0x7fbf9803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:56.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.928+0000 7fbfa37fe700 1 -- 172.21.15.40:0/1832582444 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbfa801e030 con 0x7fbfac100030 2024-03-18T22:33:56.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.928+0000 7fbfb1ce0700 1 --2- 172.21.15.40:0/1832582444 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbf98038fc0 0x7fbf9803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:56.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.929+0000 7fbfb1ce0700 1 --2- 172.21.15.40:0/1832582444 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbf98038fc0 0x7fbf9803b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c006fd0 tx=0x7fbf9c006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:56.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:56.933+0000 7fbfa37fe700 1 -- 172.21.15.40:0/1832582444 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbfa8013070 con 0x7fbfac100030 2024-03-18T22:33:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:57.145+0000 7fbfb4745700 1 -- 172.21.15.40:0/1832582444 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fbfac1907c0 con 0x7fbf98038fc0 2024-03-18T22:33:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:57.148+0000 7fbfa37fe700 1 -- 172.21.15.40:0/1832582444 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbfac1907c0 con 0x7fbf98038fc0 2024-03-18T22:33:57.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:57.150+0000 7fbfb4745700 1 -- 172.21.15.40:0/1832582444 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbf98038fc0 msgr2=0x7fbf9803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:57.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:57.150+0000 7fbfb4745700 1 --2- 172.21.15.40:0/1832582444 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbf98038fc0 0x7fbf9803b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c006fd0 tx=0x7fbf9c006e40 comp rx=0 tx=0).stop 2024-03-18T22:33:57.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:57.150+0000 7fbfb4745700 1 -- 172.21.15.40:0/1832582444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac100030 msgr2=0x7fbfac102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:57.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:57.150+0000 7fbfb4745700 1 --2- 172.21.15.40:0/1832582444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac100030 0x7fbfac102890 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8005800 tx=0x7fbfa8009f60 comp rx=0 tx=0).stop 2024-03-18T22:33:57.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:57.150+0000 7fbfb4745700 1 -- 172.21.15.40:0/1832582444 shutdown_connections 2024-03-18T22:33:57.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:57.150+0000 7fbfb4745700 1 --2- 172.21.15.40:0/1832582444 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbf98038fc0 0x7fbf9803b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:57.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:57.150+0000 7fbfb4745700 1 --2- 172.21.15.40:0/1832582444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfac100030 0x7fbfac102890 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:57.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:57.150+0000 7fbfb4745700 1 -- 172.21.15.40:0/1832582444 >> 172.21.15.40:0/1832582444 conn(0x7fbfac0fb990 msgr2=0x7fbfac0fc520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:57.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:57.151+0000 7fbfb4745700 1 -- 172.21.15.40:0/1832582444 shutdown_connections 2024-03-18T22:33:57.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:57.151+0000 7fbfb4745700 1 -- 172.21.15.40:0/1832582444 wait complete. 2024-03-18T22:33:57.921 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi103 2024-03-18T22:33:57.921 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-18T22:33:57.921 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/etc/ceph/ceph.conf 2024-03-18T22:33:57.940 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-18T22:33:57.940 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T22:33:57.998 INFO:tasks.cephadm:Adding host smithi103 to orchestrator... 2024-03-18T22:33:57.998 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch host add smithi103 2024-03-18T22:33:58.241 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:33:58.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:58 smithi040 ceph-mon[62643]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:33:58.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:58 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:33:58.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:58 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:33:58.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:58 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:33:58.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:58 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:33:58.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:58 smithi040 ceph-mon[62643]: Updating smithi040:/etc/ceph/ceph.conf 2024-03-18T22:33:58.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:58 smithi040 ceph-mon[62643]: Updating smithi040:/var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T22:33:58.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:58 smithi040 ceph-mon[62643]: Updating smithi040:/etc/ceph/ceph.client.admin.keyring 2024-03-18T22:33:58.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:58 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:33:58.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:58 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:33:58.461 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:58 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:33:59.412 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:33:59 smithi040 ceph-mon[62643]: Updating smithi040:/var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T22:33:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.697+0000 7f538edb0700 1 -- 172.21.15.40:0/4178740882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5388100e50 msgr2=0x7f5388101230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:33:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.697+0000 7f538edb0700 1 --2- 172.21.15.40:0/4178740882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5388100e50 0x7f5388101230 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5378009910 tx=0x7f5378009c20 comp rx=0 tx=0).stop 2024-03-18T22:33:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.697+0000 7f538edb0700 1 -- 172.21.15.40:0/4178740882 shutdown_connections 2024-03-18T22:33:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.697+0000 7f538edb0700 1 --2- 172.21.15.40:0/4178740882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5388100e50 0x7f5388101230 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:33:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.697+0000 7f538edb0700 1 -- 172.21.15.40:0/4178740882 >> 172.21.15.40:0/4178740882 conn(0x7f53880fc700 msgr2=0x7f53880feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:33:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.697+0000 7f538edb0700 1 -- 172.21.15.40:0/4178740882 shutdown_connections 2024-03-18T22:33:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.697+0000 7f538edb0700 1 -- 172.21.15.40:0/4178740882 wait complete. 2024-03-18T22:33:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.697+0000 7f538edb0700 1 Processor -- start 2024-03-18T22:33:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.698+0000 7f538edb0700 1 -- start start 2024-03-18T22:33:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.698+0000 7f538edb0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5388100e50 0x7f53881976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:59.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.698+0000 7f538edb0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f538810b0d0 con 0x7f5388100e50 2024-03-18T22:33:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.698+0000 7f538cb4c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5388100e50 0x7f53881976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.698+0000 7f538cb4c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5388100e50 0x7f53881976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55934/0 (socket says 172.21.15.40:55934) 2024-03-18T22:33:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.698+0000 7f538cb4c700 1 -- 172.21.15.40:0/480886161 learned_addr learned my addr 172.21.15.40:0/480886161 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:33:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.698+0000 7f538cb4c700 1 -- 172.21.15.40:0/480886161 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5388197c30 con 0x7f5388100e50 2024-03-18T22:33:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.698+0000 7f538cb4c700 1 --2- 172.21.15.40:0/480886161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5388100e50 0x7f53881976f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f53780056d0 tx=0x7f53780057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.698+0000 7f5385ffb700 1 -- 172.21.15.40:0/480886161 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5378026070 con 0x7f5388100e50 2024-03-18T22:33:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.698+0000 7f538edb0700 1 -- 172.21.15.40:0/480886161 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5388197f20 con 0x7f5388100e50 2024-03-18T22:33:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.698+0000 7f5385ffb700 1 -- 172.21.15.40:0/480886161 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f537800bba0 con 0x7f5388100e50 2024-03-18T22:33:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.699+0000 7f5385ffb700 1 -- 172.21.15.40:0/480886161 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53780037f0 con 0x7f5388100e50 2024-03-18T22:33:59.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.699+0000 7f538edb0700 1 -- 172.21.15.40:0/480886161 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f538819bfc0 con 0x7f5388100e50 2024-03-18T22:33:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.699+0000 7f5385ffb700 1 -- 172.21.15.40:0/480886161 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f5378029070 con 0x7f5388100e50 2024-03-18T22:33:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.699+0000 7f538edb0700 1 -- 172.21.15.40:0/480886161 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5388102330 con 0x7f5388100e50 2024-03-18T22:33:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.700+0000 7f5385ffb700 1 --2- 172.21.15.40:0/480886161 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5370038fc0 0x7f537003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:33:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.700+0000 7f5385ffb700 1 -- 172.21.15.40:0/480886161 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f537801e070 con 0x7f5388100e50 2024-03-18T22:33:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.700+0000 7f5387fff700 1 --2- 172.21.15.40:0/480886161 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5370038fc0 0x7f537003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:33:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.700+0000 7f5387fff700 1 --2- 172.21.15.40:0/480886161 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5370038fc0 0x7f537003b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f537c006fd0 tx=0x7f537c006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:33:59.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.705+0000 7f5385ffb700 1 -- 172.21.15.40:0/480886161 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5378013070 con 0x7f5388100e50 2024-03-18T22:33:59.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:33:59.913+0000 7f538edb0700 1 -- 172.21.15.40:0/480886161 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi103", "target": ["mon-mgr", ""]}) -- 0x7f5388198a50 con 0x7f5370038fc0 2024-03-18T22:34:01.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:00 smithi040 ceph-mon[62643]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi103", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:34:01.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:00 smithi040 ceph-mon[62643]: Deploying cephadm binary to smithi103 2024-03-18T22:34:04.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:03 smithi040 ceph-mon[62643]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:04.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:04.302+0000 7f5385ffb700 1 -- 172.21.15.40:0/480886161 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f5388198a50 con 0x7f5370038fc0 2024-03-18T22:34:04.303 INFO:teuthology.orchestra.run.smithi040.stdout:Added host 'smithi103' with addr '172.21.15.103' 2024-03-18T22:34:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:04.304+0000 7f538edb0700 1 -- 172.21.15.40:0/480886161 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5370038fc0 msgr2=0x7f537003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:04.304+0000 7f538edb0700 1 --2- 172.21.15.40:0/480886161 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5370038fc0 0x7f537003b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f537c006fd0 tx=0x7f537c006e40 comp rx=0 tx=0).stop 2024-03-18T22:34:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:04.304+0000 7f538edb0700 1 -- 172.21.15.40:0/480886161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5388100e50 msgr2=0x7f53881976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:04.304+0000 7f538edb0700 1 --2- 172.21.15.40:0/480886161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5388100e50 0x7f53881976f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f53780056d0 tx=0x7f53780057b0 comp rx=0 tx=0).stop 2024-03-18T22:34:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:04.304+0000 7f538edb0700 1 -- 172.21.15.40:0/480886161 shutdown_connections 2024-03-18T22:34:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:04.304+0000 7f538edb0700 1 --2- 172.21.15.40:0/480886161 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5370038fc0 0x7f537003b480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:04.304+0000 7f538edb0700 1 --2- 172.21.15.40:0/480886161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5388100e50 0x7f53881976f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:04.304+0000 7f538edb0700 1 -- 172.21.15.40:0/480886161 >> 172.21.15.40:0/480886161 conn(0x7f53880fc700 msgr2=0x7f538806a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:04.304+0000 7f538edb0700 1 -- 172.21.15.40:0/480886161 shutdown_connections 2024-03-18T22:34:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:04.304+0000 7f538edb0700 1 -- 172.21.15.40:0/480886161 wait complete. 2024-03-18T22:34:04.909 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T22:34:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:34:05.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:05 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:05.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:05 smithi040 ceph-mon[62643]: Added host smithi103 2024-03-18T22:34:05.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:05 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:34:05.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:05 smithi040 ceph-mon[62643]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:05.566 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:05 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:06.581 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:06 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:06.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.684+0000 7f782f3d3700 1 -- 172.21.15.40:0/178668714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7828102ee0 msgr2=0x7f78281032c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:06.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.684+0000 7f782f3d3700 1 --2- 172.21.15.40:0/178668714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7828102ee0 0x7f78281032c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7824009910 tx=0x7f7824009c20 comp rx=0 tx=0).stop 2024-03-18T22:34:06.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.684+0000 7f782f3d3700 1 -- 172.21.15.40:0/178668714 shutdown_connections 2024-03-18T22:34:06.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.684+0000 7f782f3d3700 1 --2- 172.21.15.40:0/178668714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7828102ee0 0x7f78281032c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:06.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.684+0000 7f782f3d3700 1 -- 172.21.15.40:0/178668714 >> 172.21.15.40:0/178668714 conn(0x7f78280fc6d0 msgr2=0x7f78280feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:06.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.685+0000 7f782f3d3700 1 -- 172.21.15.40:0/178668714 shutdown_connections 2024-03-18T22:34:06.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.685+0000 7f782f3d3700 1 -- 172.21.15.40:0/178668714 wait complete. 2024-03-18T22:34:06.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.685+0000 7f782f3d3700 1 Processor -- start 2024-03-18T22:34:06.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.685+0000 7f782f3d3700 1 -- start start 2024-03-18T22:34:06.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.686+0000 7f782f3d3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7828102ee0 0x7f782819d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:06.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.686+0000 7f782f3d3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f782810c220 con 0x7f7828102ee0 2024-03-18T22:34:06.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.686+0000 7f782d16f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7828102ee0 0x7f782819d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:06.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.686+0000 7f782d16f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7828102ee0 0x7f782819d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55950/0 (socket says 172.21.15.40:55950) 2024-03-18T22:34:06.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.686+0000 7f782d16f700 1 -- 172.21.15.40:0/3023000590 learned_addr learned my addr 172.21.15.40:0/3023000590 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:34:06.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.686+0000 7f782d16f700 1 -- 172.21.15.40:0/3023000590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f782819d920 con 0x7f7828102ee0 2024-03-18T22:34:06.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.686+0000 7f782d16f700 1 --2- 172.21.15.40:0/3023000590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7828102ee0 0x7f782819d3e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f78240056d0 tx=0x7f78240057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:06.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.686+0000 7f781e7fc700 1 -- 172.21.15.40:0/3023000590 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7824026070 con 0x7f7828102ee0 2024-03-18T22:34:06.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.686+0000 7f781e7fc700 1 -- 172.21.15.40:0/3023000590 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7824004370 con 0x7f7828102ee0 2024-03-18T22:34:06.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.686+0000 7f782f3d3700 1 -- 172.21.15.40:0/3023000590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f782819dbb0 con 0x7f7828102ee0 2024-03-18T22:34:06.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.686+0000 7f781e7fc700 1 -- 172.21.15.40:0/3023000590 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78240039b0 con 0x7f7828102ee0 2024-03-18T22:34:06.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.686+0000 7f782f3d3700 1 -- 172.21.15.40:0/3023000590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78281975a0 con 0x7f7828102ee0 2024-03-18T22:34:06.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.687+0000 7f781e7fc700 1 -- 172.21.15.40:0/3023000590 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f7824029070 con 0x7f7828102ee0 2024-03-18T22:34:06.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.687+0000 7f782f3d3700 1 -- 172.21.15.40:0/3023000590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f782806a990 con 0x7f7828102ee0 2024-03-18T22:34:06.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.688+0000 7f781e7fc700 1 --2- 172.21.15.40:0/3023000590 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7814039020 0x7f781403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:06.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.688+0000 7f781e7fc700 1 -- 172.21.15.40:0/3023000590 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f782401e070 con 0x7f7828102ee0 2024-03-18T22:34:06.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.688+0000 7f782c96e700 1 --2- 172.21.15.40:0/3023000590 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7814039020 0x7f781403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:06.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.688+0000 7f782c96e700 1 --2- 172.21.15.40:0/3023000590 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7814039020 0x7f781403b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7818006fd0 tx=0x7f7818006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:06.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.693+0000 7f781e7fc700 1 -- 172.21.15.40:0/3023000590 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7824013070 con 0x7f7828102ee0 2024-03-18T22:34:06.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.905+0000 7f782f3d3700 1 -- 172.21.15.40:0/3023000590 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7828198ad0 con 0x7f7814039020 2024-03-18T22:34:06.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.906+0000 7f781e7fc700 1 -- 172.21.15.40:0/3023000590 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f7828198ad0 con 0x7f7814039020 2024-03-18T22:34:06.907 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:34:06.907 INFO:teuthology.orchestra.run.smithi040.stdout:[{"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}, {"addr": "172.21.15.103", "hostname": "smithi103", "labels": [], "status": ""}] 2024-03-18T22:34:06.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.908+0000 7f782f3d3700 1 -- 172.21.15.40:0/3023000590 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7814039020 msgr2=0x7f781403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:06.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.908+0000 7f782f3d3700 1 --2- 172.21.15.40:0/3023000590 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7814039020 0x7f781403b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7818006fd0 tx=0x7f7818006e40 comp rx=0 tx=0).stop 2024-03-18T22:34:06.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.908+0000 7f782f3d3700 1 -- 172.21.15.40:0/3023000590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7828102ee0 msgr2=0x7f782819d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:06.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.908+0000 7f782f3d3700 1 --2- 172.21.15.40:0/3023000590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7828102ee0 0x7f782819d3e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f78240056d0 tx=0x7f78240057b0 comp rx=0 tx=0).stop 2024-03-18T22:34:06.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.908+0000 7f782f3d3700 1 -- 172.21.15.40:0/3023000590 shutdown_connections 2024-03-18T22:34:06.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.908+0000 7f782f3d3700 1 --2- 172.21.15.40:0/3023000590 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7814039020 0x7f781403b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:06.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.909+0000 7f782f3d3700 1 --2- 172.21.15.40:0/3023000590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7828102ee0 0x7f782819d3e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:06.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.909+0000 7f782f3d3700 1 -- 172.21.15.40:0/3023000590 >> 172.21.15.40:0/3023000590 conn(0x7f78280fc6d0 msgr2=0x7f7828103c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:06.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.909+0000 7f782f3d3700 1 -- 172.21.15.40:0/3023000590 shutdown_connections 2024-03-18T22:34:06.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:06.909+0000 7f782f3d3700 1 -- 172.21.15.40:0/3023000590 wait complete. 2024-03-18T22:34:07.444 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:07 smithi040 ceph-mon[62643]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:07.444 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:07.445 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi113 2024-03-18T22:34:07.445 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-18T22:34:07.445 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/etc/ceph/ceph.conf 2024-03-18T22:34:07.465 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-18T22:34:07.465 DEBUG:teuthology.orchestra.run.smithi113:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T22:34:07.523 INFO:tasks.cephadm:Adding host smithi113 to orchestrator... 2024-03-18T22:34:07.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch host add smithi113 2024-03-18T22:34:07.761 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:34:08.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:08 smithi040 ceph-mon[62643]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:34:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.215+0000 7fbc143cb700 1 -- 172.21.15.40:0/987283442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc0c100e40 msgr2=0x7fbc0c101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.215+0000 7fbc143cb700 1 --2- 172.21.15.40:0/987283442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc0c100e40 0x7fbc0c101220 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009910 tx=0x7fbc08009c20 comp rx=0 tx=0).stop 2024-03-18T22:34:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.215+0000 7fbc143cb700 1 -- 172.21.15.40:0/987283442 shutdown_connections 2024-03-18T22:34:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.215+0000 7fbc143cb700 1 --2- 172.21.15.40:0/987283442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc0c100e40 0x7fbc0c101220 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.215+0000 7fbc143cb700 1 -- 172.21.15.40:0/987283442 >> 172.21.15.40:0/987283442 conn(0x7fbc0c077850 msgr2=0x7fbc0c0feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:09.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.215+0000 7fbc143cb700 1 -- 172.21.15.40:0/987283442 shutdown_connections 2024-03-18T22:34:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.215+0000 7fbc143cb700 1 -- 172.21.15.40:0/987283442 wait complete. 2024-03-18T22:34:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.215+0000 7fbc143cb700 1 Processor -- start 2024-03-18T22:34:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.215+0000 7fbc143cb700 1 -- start start 2024-03-18T22:34:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.216+0000 7fbc143cb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc0c100e40 0x7fbc0c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.216+0000 7fbc143cb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c10b0c0 con 0x7fbc0c100e40 2024-03-18T22:34:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.216+0000 7fbc12167700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc0c100e40 0x7fbc0c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.216+0000 7fbc12167700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc0c100e40 0x7fbc0c198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39312/0 (socket says 172.21.15.40:39312) 2024-03-18T22:34:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.216+0000 7fbc12167700 1 -- 172.21.15.40:0/2197767914 learned_addr learned my addr 172.21.15.40:0/2197767914 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:34:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.216+0000 7fbc12167700 1 -- 172.21.15.40:0/2197767914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0c198aa0 con 0x7fbc0c100e40 2024-03-18T22:34:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.216+0000 7fbc12167700 1 --2- 172.21.15.40:0/2197767914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc0c100e40 0x7fbc0c198560 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbc08006010 tx=0x7fbc08005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.216+0000 7fbc037fe700 1 -- 172.21.15.40:0/2197767914 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc08026070 con 0x7fbc0c100e40 2024-03-18T22:34:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.216+0000 7fbc037fe700 1 -- 172.21.15.40:0/2197767914 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc0800bba0 con 0x7fbc0c100e40 2024-03-18T22:34:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.216+0000 7fbc143cb700 1 -- 172.21.15.40:0/2197767914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc0c198d90 con 0x7fbc0c100e40 2024-03-18T22:34:09.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.216+0000 7fbc037fe700 1 -- 172.21.15.40:0/2197767914 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc080038a0 con 0x7fbc0c100e40 2024-03-18T22:34:09.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.216+0000 7fbc143cb700 1 -- 172.21.15.40:0/2197767914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc0c199250 con 0x7fbc0c100e40 2024-03-18T22:34:09.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.217+0000 7fbc037fe700 1 -- 172.21.15.40:0/2197767914 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fbc08029070 con 0x7fbc0c100e40 2024-03-18T22:34:09.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.217+0000 7fbc143cb700 1 -- 172.21.15.40:0/2197767914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc0c102320 con 0x7fbc0c100e40 2024-03-18T22:34:09.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.218+0000 7fbc037fe700 1 --2- 172.21.15.40:0/2197767914 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbbf8038fc0 0x7fbbf803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:09.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.218+0000 7fbc037fe700 1 -- 172.21.15.40:0/2197767914 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbc0801e070 con 0x7fbc0c100e40 2024-03-18T22:34:09.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.218+0000 7fbc11966700 1 --2- 172.21.15.40:0/2197767914 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbbf8038fc0 0x7fbbf803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:09.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.218+0000 7fbc11966700 1 --2- 172.21.15.40:0/2197767914 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbbf8038fc0 0x7fbbf803b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc006fd0 tx=0x7fbbfc006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:09.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.223+0000 7fbc037fe700 1 -- 172.21.15.40:0/2197767914 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbc08013070 con 0x7fbc0c100e40 2024-03-18T22:34:09.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:09.432+0000 7fbc143cb700 1 -- 172.21.15.40:0/2197767914 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi113", "target": ["mon-mgr", ""]}) -- 0x7fbc0c069d50 con 0x7fbbf8038fc0 2024-03-18T22:34:09.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:09 smithi040 ceph-mon[62643]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:10.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:10 smithi040 ceph-mon[62643]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi113", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:34:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:10 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:10 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:10 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:10 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:10 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:34:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:10 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:10 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:34:11.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:11 smithi040 ceph-mon[62643]: Updating smithi103:/etc/ceph/ceph.conf 2024-03-18T22:34:11.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:11 smithi040 ceph-mon[62643]: Deploying cephadm binary to smithi113 2024-03-18T22:34:11.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:11 smithi040 ceph-mon[62643]: Updating smithi103:/var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T22:34:11.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:11 smithi040 ceph-mon[62643]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-03-18T22:34:11.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:11 smithi040 ceph-mon[62643]: Updating smithi103:/var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T22:34:11.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:11 smithi040 ceph-mon[62643]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:13.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:12 smithi040 ceph-mon[62643]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:13.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:13.853+0000 7fbc037fe700 1 -- 172.21.15.40:0/2197767914 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fbc0c069d50 con 0x7fbbf8038fc0 2024-03-18T22:34:13.854 INFO:teuthology.orchestra.run.smithi040.stdout:Added host 'smithi113' with addr '172.21.15.113' 2024-03-18T22:34:13.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:13.855+0000 7fbc143cb700 1 -- 172.21.15.40:0/2197767914 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbbf8038fc0 msgr2=0x7fbbf803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:13.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:13.855+0000 7fbc143cb700 1 --2- 172.21.15.40:0/2197767914 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbbf8038fc0 0x7fbbf803b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc006fd0 tx=0x7fbbfc006e40 comp rx=0 tx=0).stop 2024-03-18T22:34:13.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:13.856+0000 7fbc143cb700 1 -- 172.21.15.40:0/2197767914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc0c100e40 msgr2=0x7fbc0c198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:13.856+0000 7fbc143cb700 1 --2- 172.21.15.40:0/2197767914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc0c100e40 0x7fbc0c198560 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbc08006010 tx=0x7fbc08005780 comp rx=0 tx=0).stop 2024-03-18T22:34:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:13.856+0000 7fbc143cb700 1 -- 172.21.15.40:0/2197767914 shutdown_connections 2024-03-18T22:34:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:13.856+0000 7fbc143cb700 1 --2- 172.21.15.40:0/2197767914 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbbf8038fc0 0x7fbbf803b480 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:13.856+0000 7fbc143cb700 1 --2- 172.21.15.40:0/2197767914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc0c100e40 0x7fbc0c198560 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:13.856+0000 7fbc143cb700 1 -- 172.21.15.40:0/2197767914 >> 172.21.15.40:0/2197767914 conn(0x7fbc0c077850 msgr2=0x7fbc0c102810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:13.856+0000 7fbc143cb700 1 -- 172.21.15.40:0/2197767914 shutdown_connections 2024-03-18T22:34:13.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:13.856+0000 7fbc143cb700 1 -- 172.21.15.40:0/2197767914 wait complete. 2024-03-18T22:34:14.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:14 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:14.421 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:14 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:14.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:14 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:14.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:14 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:14.422 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:14 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:34:14.453 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T22:34:14.693 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:34:15.753 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:15 smithi040 ceph-mon[62643]: Added host smithi113 2024-03-18T22:34:15.754 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:15 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:15.754 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:15 smithi040 ceph-mon[62643]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:15.754 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:15 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:16.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.103+0000 7f8eaadf7700 1 -- 172.21.15.40:0/813083993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ea4102ee0 msgr2=0x7f8ea41032c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:16.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.103+0000 7f8eaadf7700 1 --2- 172.21.15.40:0/813083993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ea4102ee0 0x7f8ea41032c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8e94009910 tx=0x7f8e94009c20 comp rx=0 tx=0).stop 2024-03-18T22:34:16.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.103+0000 7f8eaadf7700 1 -- 172.21.15.40:0/813083993 shutdown_connections 2024-03-18T22:34:16.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.103+0000 7f8eaadf7700 1 --2- 172.21.15.40:0/813083993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ea4102ee0 0x7f8ea41032c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:16.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.103+0000 7f8eaadf7700 1 -- 172.21.15.40:0/813083993 >> 172.21.15.40:0/813083993 conn(0x7f8ea40fc750 msgr2=0x7f8ea40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:16.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.103+0000 7f8eaadf7700 1 -- 172.21.15.40:0/813083993 shutdown_connections 2024-03-18T22:34:16.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.103+0000 7f8eaadf7700 1 -- 172.21.15.40:0/813083993 wait complete. 2024-03-18T22:34:16.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.103+0000 7f8eaadf7700 1 Processor -- start 2024-03-18T22:34:16.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.103+0000 7f8eaadf7700 1 -- start start 2024-03-18T22:34:16.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.103+0000 7f8eaadf7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ea4102ee0 0x7f8ea419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:16.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.103+0000 7f8eaadf7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea410c260 con 0x7f8ea4102ee0 2024-03-18T22:34:16.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.104+0000 7f8ea8b93700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ea4102ee0 0x7f8ea419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:16.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.104+0000 7f8ea8b93700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ea4102ee0 0x7f8ea419d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39328/0 (socket says 172.21.15.40:39328) 2024-03-18T22:34:16.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.104+0000 7f8ea8b93700 1 -- 172.21.15.40:0/3654890991 learned_addr learned my addr 172.21.15.40:0/3654890991 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:34:16.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.104+0000 7f8ea8b93700 1 -- 172.21.15.40:0/3654890991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea419db30 con 0x7f8ea4102ee0 2024-03-18T22:34:16.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.104+0000 7f8ea8b93700 1 --2- 172.21.15.40:0/3654890991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ea4102ee0 0x7f8ea419d5f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8e940056d0 tx=0x7f8e940057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:16.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.104+0000 7f8ea1ffb700 1 -- 172.21.15.40:0/3654890991 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e94026070 con 0x7f8ea4102ee0 2024-03-18T22:34:16.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.104+0000 7f8ea1ffb700 1 -- 172.21.15.40:0/3654890991 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e94004370 con 0x7f8ea4102ee0 2024-03-18T22:34:16.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.104+0000 7f8eaadf7700 1 -- 172.21.15.40:0/3654890991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ea419ddc0 con 0x7f8ea4102ee0 2024-03-18T22:34:16.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.104+0000 7f8ea1ffb700 1 -- 172.21.15.40:0/3654890991 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e940039b0 con 0x7f8ea4102ee0 2024-03-18T22:34:16.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.104+0000 7f8eaadf7700 1 -- 172.21.15.40:0/3654890991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ea41976c0 con 0x7f8ea4102ee0 2024-03-18T22:34:16.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.105+0000 7f8ea1ffb700 1 -- 172.21.15.40:0/3654890991 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f8e94029070 con 0x7f8ea4102ee0 2024-03-18T22:34:16.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.105+0000 7f8eaadf7700 1 -- 172.21.15.40:0/3654890991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ea406aa30 con 0x7f8ea4102ee0 2024-03-18T22:34:16.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.105+0000 7f8ea1ffb700 1 --2- 172.21.15.40:0/3654890991 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e8c039020 0x7f8e8c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:16.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.106+0000 7f8ea1ffb700 1 -- 172.21.15.40:0/3654890991 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8e9401e070 con 0x7f8ea4102ee0 2024-03-18T22:34:16.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.106+0000 7f8ea3fff700 1 --2- 172.21.15.40:0/3654890991 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e8c039020 0x7f8e8c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:16.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.106+0000 7f8ea3fff700 1 --2- 172.21.15.40:0/3654890991 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e8c039020 0x7f8e8c03b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8e98006fd0 tx=0x7f8e98006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:16.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.110+0000 7f8ea1ffb700 1 -- 172.21.15.40:0/3654890991 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8e94013070 con 0x7f8ea4102ee0 2024-03-18T22:34:16.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.321+0000 7f8eaadf7700 1 -- 172.21.15.40:0/3654890991 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8ea4063940 con 0x7f8e8c039020 2024-03-18T22:34:16.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.322+0000 7f8ea1ffb700 1 -- 172.21.15.40:0/3654890991 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f8ea4063940 con 0x7f8e8c039020 2024-03-18T22:34:16.322 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:34:16.323 INFO:teuthology.orchestra.run.smithi040.stdout:[{"addr": "172.21.15.40", "hostname": "smithi040", "labels": [], "status": ""}, {"addr": "172.21.15.103", "hostname": "smithi103", "labels": [], "status": ""}, {"addr": "172.21.15.113", "hostname": "smithi113", "labels": [], "status": ""}] 2024-03-18T22:34:16.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.324+0000 7f8eaadf7700 1 -- 172.21.15.40:0/3654890991 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e8c039020 msgr2=0x7f8e8c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:16.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.324+0000 7f8eaadf7700 1 --2- 172.21.15.40:0/3654890991 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e8c039020 0x7f8e8c03b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8e98006fd0 tx=0x7f8e98006e40 comp rx=0 tx=0).stop 2024-03-18T22:34:16.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.324+0000 7f8eaadf7700 1 -- 172.21.15.40:0/3654890991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ea4102ee0 msgr2=0x7f8ea419d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:16.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.324+0000 7f8eaadf7700 1 --2- 172.21.15.40:0/3654890991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ea4102ee0 0x7f8ea419d5f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8e940056d0 tx=0x7f8e940057b0 comp rx=0 tx=0).stop 2024-03-18T22:34:16.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.324+0000 7f8eaadf7700 1 -- 172.21.15.40:0/3654890991 shutdown_connections 2024-03-18T22:34:16.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.324+0000 7f8eaadf7700 1 --2- 172.21.15.40:0/3654890991 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e8c039020 0x7f8e8c03b4e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:16.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.324+0000 7f8eaadf7700 1 --2- 172.21.15.40:0/3654890991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ea4102ee0 0x7f8ea419d5f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:16.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.324+0000 7f8eaadf7700 1 -- 172.21.15.40:0/3654890991 >> 172.21.15.40:0/3654890991 conn(0x7f8ea40fc750 msgr2=0x7f8ea4103c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:16.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.325+0000 7f8eaadf7700 1 -- 172.21.15.40:0/3654890991 shutdown_connections 2024-03-18T22:34:16.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:16.325+0000 7f8eaadf7700 1 -- 172.21.15.40:0/3654890991 wait complete. 2024-03-18T22:34:16.914 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-18T22:34:16.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-18T22:34:17.155 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:34:17.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:17 smithi040 ceph-mon[62643]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:34:17.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:17.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:17 smithi040 ceph-mon[62643]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.615+0000 7f29577a4700 1 -- 172.21.15.40:0/4194250309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2950101f50 msgr2=0x7f2950102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.615+0000 7f29577a4700 1 --2- 172.21.15.40:0/4194250309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2950101f50 0x7f2950102330 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f294c009910 tx=0x7f294c009c20 comp rx=0 tx=0).stop 2024-03-18T22:34:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.615+0000 7f29577a4700 1 -- 172.21.15.40:0/4194250309 shutdown_connections 2024-03-18T22:34:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.615+0000 7f29577a4700 1 --2- 172.21.15.40:0/4194250309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2950101f50 0x7f2950102330 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.615+0000 7f29577a4700 1 -- 172.21.15.40:0/4194250309 >> 172.21.15.40:0/4194250309 conn(0x7f29500fd820 msgr2=0x7f29500ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.616+0000 7f29577a4700 1 -- 172.21.15.40:0/4194250309 shutdown_connections 2024-03-18T22:34:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.616+0000 7f29577a4700 1 -- 172.21.15.40:0/4194250309 wait complete. 2024-03-18T22:34:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.616+0000 7f29577a4700 1 Processor -- start 2024-03-18T22:34:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.616+0000 7f29577a4700 1 -- start start 2024-03-18T22:34:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.616+0000 7f29577a4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2950101f50 0x7f295019d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:18.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.616+0000 7f29577a4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f295010c1d0 con 0x7f2950101f50 2024-03-18T22:34:18.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.616+0000 7f2955540700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2950101f50 0x7f295019d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:18.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.617+0000 7f2955540700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2950101f50 0x7f295019d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39344/0 (socket says 172.21.15.40:39344) 2024-03-18T22:34:18.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.617+0000 7f2955540700 1 -- 172.21.15.40:0/1689863752 learned_addr learned my addr 172.21.15.40:0/1689863752 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:34:18.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.617+0000 7f2955540700 1 -- 172.21.15.40:0/1689863752 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f295019d910 con 0x7f2950101f50 2024-03-18T22:34:18.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.617+0000 7f2955540700 1 --2- 172.21.15.40:0/1689863752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2950101f50 0x7f295019d3d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f294c0056d0 tx=0x7f294c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:18.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.617+0000 7f29467fc700 1 -- 172.21.15.40:0/1689863752 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f294c026070 con 0x7f2950101f50 2024-03-18T22:34:18.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.617+0000 7f29467fc700 1 -- 172.21.15.40:0/1689863752 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f294c00bba0 con 0x7f2950101f50 2024-03-18T22:34:18.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.617+0000 7f29467fc700 1 -- 172.21.15.40:0/1689863752 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f294c0037f0 con 0x7f2950101f50 2024-03-18T22:34:18.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.617+0000 7f29577a4700 1 -- 172.21.15.40:0/1689863752 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f295019dc00 con 0x7f2950101f50 2024-03-18T22:34:18.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.617+0000 7f29577a4700 1 -- 172.21.15.40:0/1689863752 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2950197590 con 0x7f2950101f50 2024-03-18T22:34:18.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.618+0000 7f29467fc700 1 -- 172.21.15.40:0/1689863752 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f294c029070 con 0x7f2950101f50 2024-03-18T22:34:18.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.618+0000 7f29577a4700 1 -- 172.21.15.40:0/1689863752 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2950103430 con 0x7f2950101f50 2024-03-18T22:34:18.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.618+0000 7f29467fc700 1 --2- 172.21.15.40:0/1689863752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f293c039020 0x7f293c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:18.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.618+0000 7f29467fc700 1 -- 172.21.15.40:0/1689863752 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f294c01e070 con 0x7f2950101f50 2024-03-18T22:34:18.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.618+0000 7f2954d3f700 1 --2- 172.21.15.40:0/1689863752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f293c039020 0x7f293c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:18.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.619+0000 7f2954d3f700 1 --2- 172.21.15.40:0/1689863752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f293c039020 0x7f293c03b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2940006fd0 tx=0x7f2940006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:18.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.623+0000 7f29467fc700 1 -- 172.21.15.40:0/1689863752 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f294c013070 con 0x7f2950101f50 2024-03-18T22:34:18.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:18.834+0000 7f29577a4700 1 -- 172.21.15.40:0/1689863752 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f2950063940 con 0x7f2950101f50 2024-03-18T22:34:18.835 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:18 smithi040 ceph-mon[62643]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:19.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:19.816+0000 7f29467fc700 1 -- 172.21.15.40:0/1689863752 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f294c00fe20 con 0x7f2950101f50 2024-03-18T22:34:19.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:19.819+0000 7f29577a4700 1 -- 172.21.15.40:0/1689863752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f293c039020 msgr2=0x7f293c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:19.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:19.819+0000 7f29577a4700 1 --2- 172.21.15.40:0/1689863752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f293c039020 0x7f293c03b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2940006fd0 tx=0x7f2940006e40 comp rx=0 tx=0).stop 2024-03-18T22:34:19.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:19.819+0000 7f29577a4700 1 -- 172.21.15.40:0/1689863752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2950101f50 msgr2=0x7f295019d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:19.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:19.819+0000 7f29577a4700 1 --2- 172.21.15.40:0/1689863752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2950101f50 0x7f295019d3d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f294c0056d0 tx=0x7f294c0057b0 comp rx=0 tx=0).stop 2024-03-18T22:34:19.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:19.819+0000 7f29577a4700 1 -- 172.21.15.40:0/1689863752 shutdown_connections 2024-03-18T22:34:19.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:19.819+0000 7f29577a4700 1 --2- 172.21.15.40:0/1689863752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f293c039020 0x7f293c03b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:19.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:19.819+0000 7f29577a4700 1 --2- 172.21.15.40:0/1689863752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2950101f50 0x7f295019d3d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:19.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:19.819+0000 7f29577a4700 1 -- 172.21.15.40:0/1689863752 >> 172.21.15.40:0/1689863752 conn(0x7f29500fd820 msgr2=0x7f2950103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:19.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:19.819+0000 7f29577a4700 1 -- 172.21.15.40:0/1689863752 shutdown_connections 2024-03-18T22:34:19.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:19.819+0000 7f29577a4700 1 -- 172.21.15.40:0/1689863752 wait complete. 2024-03-18T22:34:19.821 INFO:teuthology.orchestra.run.smithi040.stderr:adjusted tunables profile to default 2024-03-18T22:34:20.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:19 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1689863752' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T22:34:20.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:19 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:20.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:19 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:20.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:19 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:20.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:19 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:20.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:19 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi113", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:34:20.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:19 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:20.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:19 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:34:20.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:19 smithi040 ceph-mon[62643]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-18T22:34:20.492 INFO:tasks.cephadm:Adding mon.a on smithi040 2024-03-18T22:34:20.493 INFO:tasks.cephadm:Adding mon.b on smithi103 2024-03-18T22:34:20.493 INFO:tasks.cephadm:Adding mon.c on smithi113 2024-03-18T22:34:20.493 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi040:172.21.15.40=a;smithi103:172.21.15.103=b;smithi113:172.21.15.113=c' 2024-03-18T22:34:20.720 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T22:34:21.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:20 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1689863752' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T22:34:21.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:20 smithi040 ceph-mon[62643]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T22:34:21.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:20 smithi040 ceph-mon[62643]: Updating smithi113:/var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T22:34:21.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:20 smithi040 ceph-mon[62643]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-03-18T22:34:21.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:20 smithi040 ceph-mon[62643]: Updating smithi113:/var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T22:34:21.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:21.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:21.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:21.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:20 smithi040 ceph-mon[62643]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:22.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.091+0000 7f47a8fc8700 1 -- 172.21.15.113:0/3736292514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a4101f50 msgr2=0x7f47a4102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:22.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.091+0000 7f47a8fc8700 1 --2- 172.21.15.113:0/3736292514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a4101f50 0x7f47a4102330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4794009910 tx=0x7f4794009c20 comp rx=0 tx=0).stop 2024-03-18T22:34:22.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.091+0000 7f47a8fc8700 1 -- 172.21.15.113:0/3736292514 shutdown_connections 2024-03-18T22:34:22.093 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.091+0000 7f47a8fc8700 1 --2- 172.21.15.113:0/3736292514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a4101f50 0x7f47a4102330 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.091+0000 7f47a8fc8700 1 -- 172.21.15.113:0/3736292514 >> 172.21.15.113:0/3736292514 conn(0x7f47a40fd820 msgr2=0x7f47a40ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.091+0000 7f47a8fc8700 1 -- 172.21.15.113:0/3736292514 shutdown_connections 2024-03-18T22:34:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.091+0000 7f47a8fc8700 1 -- 172.21.15.113:0/3736292514 wait complete. 2024-03-18T22:34:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.092+0000 7f47a8fc8700 1 Processor -- start 2024-03-18T22:34:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.092+0000 7f47a8fc8700 1 -- start start 2024-03-18T22:34:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.092+0000 7f47a8fc8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a4101f50 0x7f47a4074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.092+0000 7f47a8fc8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f47a410c1d0 con 0x7f47a4101f50 2024-03-18T22:34:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.092+0000 7f47a259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a4101f50 0x7f47a4074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.092+0000 7f47a259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a4101f50 0x7f47a4074610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.113:46218/0 (socket says 172.21.15.113:46218) 2024-03-18T22:34:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.092+0000 7f47a259c700 1 -- 172.21.15.113:0/4267284051 learned_addr learned my addr 172.21.15.113:0/4267284051 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-18T22:34:22.094 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.093+0000 7f47a259c700 1 -- 172.21.15.113:0/4267284051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a4076500 con 0x7f47a4101f50 2024-03-18T22:34:22.095 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.093+0000 7f47a259c700 1 --2- 172.21.15.113:0/4267284051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a4101f50 0x7f47a4074610 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f47940056d0 tx=0x7f47940057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:22.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.093+0000 7f47937fe700 1 -- 172.21.15.113:0/4267284051 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4794026070 con 0x7f47a4101f50 2024-03-18T22:34:22.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.093+0000 7f47937fe700 1 -- 172.21.15.113:0/4267284051 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f479400bc00 con 0x7f47a4101f50 2024-03-18T22:34:22.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.093+0000 7f47a8fc8700 1 -- 172.21.15.113:0/4267284051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47a4074bb0 con 0x7f47a4101f50 2024-03-18T22:34:22.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.093+0000 7f47937fe700 1 -- 172.21.15.113:0/4267284051 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47940039b0 con 0x7f47a4101f50 2024-03-18T22:34:22.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.093+0000 7f47a8fc8700 1 -- 172.21.15.113:0/4267284051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47a4075070 con 0x7f47a4101f50 2024-03-18T22:34:22.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.094+0000 7f47a8fc8700 1 -- 172.21.15.113:0/4267284051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47a4103430 con 0x7f47a4101f50 2024-03-18T22:34:22.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.094+0000 7f47937fe700 1 -- 172.21.15.113:0/4267284051 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f4794029070 con 0x7f47a4101f50 2024-03-18T22:34:22.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.094+0000 7f47937fe700 1 --2- 172.21.15.113:0/4267284051 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f478c038fc0 0x7f478c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:22.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.094+0000 7f47937fe700 1 -- 172.21.15.113:0/4267284051 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f479401e070 con 0x7f47a4101f50 2024-03-18T22:34:22.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.094+0000 7f47a1d9b700 1 --2- 172.21.15.113:0/4267284051 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f478c038fc0 0x7f478c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:22.096 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.095+0000 7f47a1d9b700 1 --2- 172.21.15.113:0/4267284051 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f478c038fc0 0x7f478c03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4798006fd0 tx=0x7f4798006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:22.100 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.099+0000 7f47937fe700 1 -- 172.21.15.113:0/4267284051 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4794013070 con 0x7f47a4101f50 2024-03-18T22:34:22.305 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.304+0000 7f47a8fc8700 1 -- 172.21.15.113:0/4267284051 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi040:172.21.15.40=a;smithi103:172.21.15.103=b;smithi113:172.21.15.113=c", "target": ["mon-mgr", ""]}) -- 0x7f47a4075300 con 0x7f478c038fc0 2024-03-18T22:34:22.312 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.311+0000 7f47937fe700 1 -- 172.21.15.113:0/4267284051 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f47a4075300 con 0x7f478c038fc0 2024-03-18T22:34:22.312 INFO:teuthology.orchestra.run.smithi113.stdout:Scheduled mon update... 2024-03-18T22:34:22.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.313+0000 7f47a8fc8700 1 -- 172.21.15.113:0/4267284051 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f478c038fc0 msgr2=0x7f478c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:22.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.313+0000 7f47a8fc8700 1 --2- 172.21.15.113:0/4267284051 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f478c038fc0 0x7f478c03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4798006fd0 tx=0x7f4798006e40 comp rx=0 tx=0).stop 2024-03-18T22:34:22.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.313+0000 7f47a8fc8700 1 -- 172.21.15.113:0/4267284051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a4101f50 msgr2=0x7f47a4074610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:22.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.313+0000 7f47a8fc8700 1 --2- 172.21.15.113:0/4267284051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a4101f50 0x7f47a4074610 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f47940056d0 tx=0x7f47940057b0 comp rx=0 tx=0).stop 2024-03-18T22:34:22.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.313+0000 7f47a8fc8700 1 -- 172.21.15.113:0/4267284051 shutdown_connections 2024-03-18T22:34:22.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.313+0000 7f47a8fc8700 1 --2- 172.21.15.113:0/4267284051 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f478c038fc0 0x7f478c03b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:22.314 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.313+0000 7f47a8fc8700 1 --2- 172.21.15.113:0/4267284051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a4101f50 0x7f47a4074610 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:22.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.313+0000 7f47a8fc8700 1 -- 172.21.15.113:0/4267284051 >> 172.21.15.113:0/4267284051 conn(0x7f47a40fd820 msgr2=0x7f47a4103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:22.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.313+0000 7f47a8fc8700 1 -- 172.21.15.113:0/4267284051 shutdown_connections 2024-03-18T22:34:22.315 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:22.313+0000 7f47a8fc8700 1 -- 172.21.15.113:0/4267284051 wait complete. 2024-03-18T22:34:22.890 DEBUG:teuthology.orchestra.run.smithi103:mon.b> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.b.service 2024-03-18T22:34:22.892 DEBUG:teuthology.orchestra.run.smithi113:mon.c> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.c.service 2024-03-18T22:34:22.895 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T22:34:22.895 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T22:34:23.578 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T22:34:23.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:23 smithi040 ceph-mon[62643]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi040:172.21.15.40=a;smithi103:172.21.15.103=b;smithi113:172.21.15.113=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:34:23.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:23 smithi040 ceph-mon[62643]: Saving service mon spec with placement smithi040:172.21.15.40=a;smithi103:172.21.15.103=b;smithi113:172.21.15.113=c;count:3 2024-03-18T22:34:23.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:23.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:34:23.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:23.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:34:23.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:23.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T22:34:23.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:23.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:23 smithi040 ceph-mon[62643]: Deploying daemon mon.c on smithi113 2024-03-18T22:34:23.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:23 smithi040 ceph-mon[62643]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:25.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:24 smithi040 ceph-mon[62643]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:26.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.679+0000 7f7fc1e56700 1 -- 172.21.15.113:0/1038311779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fbc102f50 msgr2=0x7f7fbc103330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:26.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.679+0000 7f7fc1e56700 1 --2- 172.21.15.113:0/1038311779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fbc102f50 0x7f7fbc103330 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009910 tx=0x7f7fac009c20 comp rx=0 tx=0).stop 2024-03-18T22:34:26.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.679+0000 7f7fc1e56700 1 -- 172.21.15.113:0/1038311779 shutdown_connections 2024-03-18T22:34:26.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.679+0000 7f7fc1e56700 1 --2- 172.21.15.113:0/1038311779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fbc102f50 0x7f7fbc103330 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:26.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.679+0000 7f7fc1e56700 1 -- 172.21.15.113:0/1038311779 >> 172.21.15.113:0/1038311779 conn(0x7f7fbc0fc700 msgr2=0x7f7fbc0feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:26.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.679+0000 7f7fc1e56700 1 -- 172.21.15.113:0/1038311779 shutdown_connections 2024-03-18T22:34:26.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.679+0000 7f7fc1e56700 1 -- 172.21.15.113:0/1038311779 wait complete. 2024-03-18T22:34:26.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.679+0000 7f7fc1e56700 1 Processor -- start 2024-03-18T22:34:26.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.679+0000 7f7fc1e56700 1 -- start start 2024-03-18T22:34:26.681 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.680+0000 7f7fc1e56700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fbc102f50 0x7f7fbc100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:26.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.680+0000 7f7fc1e56700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc10c250 con 0x7f7fbc102f50 2024-03-18T22:34:26.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.680+0000 7f7fbb7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fbc102f50 0x7f7fbc100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:26.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.680+0000 7f7fbb7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fbc102f50 0x7f7fbc100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.113:39990/0 (socket says 172.21.15.113:39990) 2024-03-18T22:34:26.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.680+0000 7f7fbb7fe700 1 -- 172.21.15.113:0/10330065 learned_addr learned my addr 172.21.15.113:0/10330065 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-18T22:34:26.682 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.680+0000 7f7fbb7fe700 1 -- 172.21.15.113:0/10330065 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fbc102aa0 con 0x7f7fbc102f50 2024-03-18T22:34:26.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.680+0000 7f7fbb7fe700 1 --2- 172.21.15.113:0/10330065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fbc102f50 0x7f7fbc100bb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0056d0 tx=0x7f7fac0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:26.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.681+0000 7f7fb8ff9700 1 -- 172.21.15.113:0/10330065 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fac026070 con 0x7f7fbc102f50 2024-03-18T22:34:26.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.681+0000 7f7fb8ff9700 1 -- 172.21.15.113:0/10330065 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7fac004370 con 0x7f7fbc102f50 2024-03-18T22:34:26.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.681+0000 7f7fc1e56700 1 -- 172.21.15.113:0/10330065 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fbc1010f0 con 0x7f7fbc102f50 2024-03-18T22:34:26.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.681+0000 7f7fb8ff9700 1 -- 172.21.15.113:0/10330065 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fac0039b0 con 0x7f7fbc102f50 2024-03-18T22:34:26.683 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.681+0000 7f7fc1e56700 1 -- 172.21.15.113:0/10330065 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fbc1015b0 con 0x7f7fbc102f50 2024-03-18T22:34:26.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.682+0000 7f7fc1e56700 1 -- 172.21.15.113:0/10330065 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fbc06a9b0 con 0x7f7fbc102f50 2024-03-18T22:34:26.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.682+0000 7f7fb8ff9700 1 -- 172.21.15.113:0/10330065 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f7fac029070 con 0x7f7fbc102f50 2024-03-18T22:34:26.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.682+0000 7f7fb8ff9700 1 --2- 172.21.15.113:0/10330065 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7fa4038fc0 0x7f7fa403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:26.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.682+0000 7f7fb8ff9700 1 -- 172.21.15.113:0/10330065 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7fac01e070 con 0x7f7fbc102f50 2024-03-18T22:34:26.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.682+0000 7f7fbaffd700 1 --2- 172.21.15.113:0/10330065 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7fa4038fc0 0x7f7fa403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:26.684 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.683+0000 7f7fbaffd700 1 --2- 172.21.15.113:0/10330065 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7fa4038fc0 0x7f7fa403b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0006fd0 tx=0x7f7fb0006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:26.688 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.688+0000 7f7fb8ff9700 1 -- 172.21.15.113:0/10330065 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7fac013070 con 0x7f7fbc102f50 2024-03-18T22:34:26.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:26 smithi113 systemd[1]: Starting Ceph mon.c for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:34:26.993 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.992+0000 7f7fc1e56700 1 -- 172.21.15.113:0/10330065 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7fbc102180 con 0x7f7fbc102f50 2024-03-18T22:34:26.994 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.993+0000 7f7fb8ff9700 1 -- 172.21.15.113:0/10330065 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f7fac030020 con 0x7f7fbc102f50 2024-03-18T22:34:26.994 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:34:26.994 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":1,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","modified":"2024-03-18T22:32:44.918096Z","created":"2024-03-18T22:32:44.918096Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T22:34:26.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.995+0000 7f7fc1e56700 1 -- 172.21.15.113:0/10330065 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7fa4038fc0 msgr2=0x7f7fa403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:26.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.995+0000 7f7fc1e56700 1 --2- 172.21.15.113:0/10330065 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7fa4038fc0 0x7f7fa403b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0006fd0 tx=0x7f7fb0006e40 comp rx=0 tx=0).stop 2024-03-18T22:34:26.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.995+0000 7f7fc1e56700 1 -- 172.21.15.113:0/10330065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fbc102f50 msgr2=0x7f7fbc100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:26.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.995+0000 7f7fc1e56700 1 --2- 172.21.15.113:0/10330065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fbc102f50 0x7f7fbc100bb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0056d0 tx=0x7f7fac0057b0 comp rx=0 tx=0).stop 2024-03-18T22:34:26.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.995+0000 7f7fc1e56700 1 -- 172.21.15.113:0/10330065 shutdown_connections 2024-03-18T22:34:26.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.995+0000 7f7fc1e56700 1 --2- 172.21.15.113:0/10330065 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7fa4038fc0 0x7f7fa403b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:26.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.995+0000 7f7fc1e56700 1 --2- 172.21.15.113:0/10330065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7fbc102f50 0x7f7fbc100bb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:26.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.995+0000 7f7fc1e56700 1 -- 172.21.15.113:0/10330065 >> 172.21.15.113:0/10330065 conn(0x7f7fbc0fc700 msgr2=0x7f7fbc103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:26.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.995+0000 7f7fc1e56700 1 -- 172.21.15.113:0/10330065 shutdown_connections 2024-03-18T22:34:26.996 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:26.995+0000 7f7fc1e56700 1 -- 172.21.15.113:0/10330065 wait complete. 2024-03-18T22:34:26.997 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 1 2024-03-18T22:34:27.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:26 smithi040 ceph-mon[62643]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:27.273 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:26 smithi113 podman[66739]: 2024-03-18 22:34:26.91050418 +0000 UTC m=+0.107981026 container create 3c9a2ff1f2041c946902d53fa437fcd818a051ccfd170fd7eadca1acdf7d4ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-18T22:34:27.273 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:26 smithi113 podman[66739]: 2024-03-18 22:34:26.821074123 +0000 UTC m=+0.018550962 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T22:34:27.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 podman[66739]: 2024-03-18 22:34:27.633142984 +0000 UTC m=+0.830619830 container init 3c9a2ff1f2041c946902d53fa437fcd818a051ccfd170fd7eadca1acdf7d4ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T22:34:27.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 podman[66739]: 2024-03-18 22:34:27.636500799 +0000 UTC m=+0.833977630 container start 3c9a2ff1f2041c946902d53fa437fcd818a051ccfd170fd7eadca1acdf7d4ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-18T22:34:27.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T22:34:27.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev), process ceph-mon, pid 2 2024-03-18T22:34:27.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: pidfile_write: ignore empty --pid-file 2024-03-18T22:34:27.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: load: jerasure load: lrc 2024-03-18T22:34:27.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: RocksDB version: 7.9.2 2024-03-18T22:34:27.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Git sha 0 2024-03-18T22:34:27.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Compile date 2024-03-18 18:57:12 2024-03-18T22:34:27.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: DB SUMMARY 2024-03-18T22:34:27.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: DB Session ID: L7CK5HBD7TJ83S8FGVVH 2024-03-18T22:34:27.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: CURRENT file: CURRENT 2024-03-18T22:34:27.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T22:34:27.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-18T22:34:27.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-18T22:34:27.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-18T22:34:27.832 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.error_if_exists: 0 2024-03-18T22:34:27.832 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.create_if_missing: 0 2024-03-18T22:34:27.832 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.paranoid_checks: 1 2024-03-18T22:34:27.832 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T22:34:27.832 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T22:34:27.832 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T22:34:27.832 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.env: 0x55632353ac60 2024-03-18T22:34:27.832 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T22:34:27.832 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.info_log: 0x5563244e7ae0 2024-03-18T22:34:27.832 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T22:34:27.832 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.statistics: (nil) 2024-03-18T22:34:27.832 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.use_fsync: 0 2024-03-18T22:34:27.832 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_log_file_size: 0 2024-03-18T22:34:27.832 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T22:34:27.833 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T22:34:27.833 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T22:34:27.833 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T22:34:27.833 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.allow_fallocate: 1 2024-03-18T22:34:27.833 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T22:34:27.833 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T22:34:27.833 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.use_direct_reads: 0 2024-03-18T22:34:27.833 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T22:34:27.833 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T22:34:27.833 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.db_log_dir: 2024-03-18T22:34:27.833 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.wal_dir: 2024-03-18T22:34:27.833 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T22:34:27.833 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T22:34:27.833 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T22:34:27.833 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T22:34:27.834 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T22:34:27.834 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T22:34:27.834 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T22:34:27.834 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T22:34:27.834 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.write_buffer_manager: 0x5563244ba1e0 2024-03-18T22:34:27.834 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T22:34:27.834 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T22:34:27.834 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T22:34:27.834 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T22:34:27.834 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T22:34:27.834 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T22:34:27.834 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T22:34:27.834 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T22:34:27.834 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.unordered_write: 0 2024-03-18T22:34:27.835 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T22:34:27.835 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T22:34:27.835 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T22:34:27.835 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T22:34:27.835 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.row_cache: None 2024-03-18T22:34:27.835 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.wal_filter: None 2024-03-18T22:34:27.836 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T22:34:27.837 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T22:34:27.837 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.two_write_queues: 0 2024-03-18T22:34:27.837 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T22:34:27.837 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.wal_compression: 0 2024-03-18T22:34:27.837 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.atomic_flush: 0 2024-03-18T22:34:27.837 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T22:34:27.837 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T22:34:27.837 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T22:34:27.837 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.log_readahead_size: 0 2024-03-18T22:34:27.837 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T22:34:27.837 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T22:34:27.837 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T22:34:27.838 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T22:34:27.838 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T22:34:27.838 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T22:34:27.838 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T22:34:27.838 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_background_jobs: 2 2024-03-18T22:34:27.838 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_background_compactions: -1 2024-03-18T22:34:27.838 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_subcompactions: 1 2024-03-18T22:34:27.838 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T22:34:27.838 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T22:34:27.838 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T22:34:27.838 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T22:34:27.838 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T22:34:27.838 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T22:34:27.838 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_open_files: -1 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_background_flushes: -1 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Compression algorithms supported: 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: kZSTD supported: 0 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: kXpressCompression supported: 0 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: kZlibCompression supported: 1 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: kSnappyCompression supported: 1 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: kLZ4Compression supported: 1 2024-03-18T22:34:27.839 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: kBZip2Compression supported: 0 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.merge_operator: 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compaction_filter: None 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5563244e7be0) 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout: cache_index_and_filter_blocks: 1 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T22:34:27.840 INFO:journalctl@ceph.mon.c.smithi113.stdout: pin_top_level_index_and_filter: 1 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: index_type: 0 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: data_block_index_type: 0 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: index_shortening: 1 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: checksum: 4 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: no_block_cache: 0 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: block_cache: 0x5563244ed090 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: block_cache_name: BinnedLRUCache 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: block_cache_options: 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: capacity : 536870912 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: num_shard_bits : 4 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: strict_capacity_limit : 0 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: high_pri_pool_ratio: 0.000 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: block_cache_compressed: (nil) 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: persistent_cache: (nil) 2024-03-18T22:34:27.841 INFO:journalctl@ceph.mon.c.smithi113.stdout: block_size: 4096 2024-03-18T22:34:27.842 INFO:journalctl@ceph.mon.c.smithi113.stdout: block_size_deviation: 10 2024-03-18T22:34:27.842 INFO:journalctl@ceph.mon.c.smithi113.stdout: block_restart_interval: 16 2024-03-18T22:34:27.842 INFO:journalctl@ceph.mon.c.smithi113.stdout: index_block_restart_interval: 1 2024-03-18T22:34:27.842 INFO:journalctl@ceph.mon.c.smithi113.stdout: metadata_block_size: 4096 2024-03-18T22:34:27.842 INFO:journalctl@ceph.mon.c.smithi113.stdout: partition_filters: 0 2024-03-18T22:34:27.842 INFO:journalctl@ceph.mon.c.smithi113.stdout: use_delta_encoding: 1 2024-03-18T22:34:27.842 INFO:journalctl@ceph.mon.c.smithi113.stdout: filter_policy: bloomfilter 2024-03-18T22:34:27.842 INFO:journalctl@ceph.mon.c.smithi113.stdout: whole_key_filtering: 1 2024-03-18T22:34:27.842 INFO:journalctl@ceph.mon.c.smithi113.stdout: verify_compression: 0 2024-03-18T22:34:27.842 INFO:journalctl@ceph.mon.c.smithi113.stdout: read_amp_bytes_per_bit: 0 2024-03-18T22:34:27.842 INFO:journalctl@ceph.mon.c.smithi113.stdout: format_version: 5 2024-03-18T22:34:27.842 INFO:journalctl@ceph.mon.c.smithi113.stdout: enable_index_compression: 1 2024-03-18T22:34:27.842 INFO:journalctl@ceph.mon.c.smithi113.stdout: block_align: 0 2024-03-18T22:34:27.842 INFO:journalctl@ceph.mon.c.smithi113.stdout: max_auto_readahead_size: 262144 2024-03-18T22:34:27.842 INFO:journalctl@ceph.mon.c.smithi113.stdout: prepopulate_block_cache: 0 2024-03-18T22:34:27.843 INFO:journalctl@ceph.mon.c.smithi113.stdout: initial_auto_readahead_size: 8192 2024-03-18T22:34:27.843 INFO:journalctl@ceph.mon.c.smithi113.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T22:34:27.843 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T22:34:27.843 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T22:34:27.843 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compression: NoCompression 2024-03-18T22:34:27.843 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T22:34:27.843 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T22:34:27.843 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T22:34:27.843 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.num_levels: 7 2024-03-18T22:34:27.843 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T22:34:27.843 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T22:34:27.843 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T22:34:27.843 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T22:34:27.843 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T22:34:27.843 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T22:34:27.844 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T22:34:27.844 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T22:34:27.844 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T22:34:27.844 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T22:34:27.844 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T22:34:27.844 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T22:34:27.844 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T22:34:27.844 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T22:34:27.844 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T22:34:27.844 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T22:34:27.844 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T22:34:27.844 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T22:34:27.844 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T22:34:27.844 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T22:34:27.844 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T22:34:27.845 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T22:34:27.846 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T22:34:27.846 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T22:34:27.846 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T22:34:27.846 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T22:34:27.846 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T22:34:27.846 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T22:34:27.846 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T22:34:27.846 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T22:34:27.846 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T22:34:27.846 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T22:34:27.846 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T22:34:27.846 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T22:34:27.846 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T22:34:27.846 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T22:34:27.846 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T22:34:27.847 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T22:34:27.847 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.table_properties_collectors: 2024-03-18T22:34:27.847 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.inplace_update_support: 0 2024-03-18T22:34:27.847 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T22:34:27.847 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T22:34:27.847 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T22:34:27.847 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T22:34:27.847 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.bloom_locality: 0 2024-03-18T22:34:27.847 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.max_successive_merges: 0 2024-03-18T22:34:27.847 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T22:34:27.847 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T22:34:27.847 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T22:34:27.847 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T22:34:27.847 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.ttl: 2592000 2024-03-18T22:34:27.847 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T22:34:27.848 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T22:34:27.848 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T22:34:27.848 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.enable_blob_files: false 2024-03-18T22:34:27.848 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.min_blob_size: 0 2024-03-18T22:34:27.848 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T22:34:27.848 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T22:34:27.848 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T22:34:27.848 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T22:34:27.848 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T22:34:27.848 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T22:34:27.848 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T22:34:27.848 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T22:34:27.848 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b6909de7-94e7-44f0-886f-8ecec505ff93 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710801267678979, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710801267679402, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710801267, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b6909de7-94e7-44f0-886f-8ecec505ff93", "db_session_id": "L7CK5HBD7TJ83S8FGVVH", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710801267679479, "job": 1, "event": "recovery_finished"} 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5563245e0000 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: DB pointer 0x5563245ca000 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout: ** DB Stats ** 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T22:34:27.849 INFO:journalctl@ceph.mon.c.smithi113.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T22:34:27.850 INFO:journalctl@ceph.mon.c.smithi113.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T22:34:27.850 INFO:journalctl@ceph.mon.c.smithi113.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T22:34:27.850 INFO:journalctl@ceph.mon.c.smithi113.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T22:34:27.850 INFO:journalctl@ceph.mon.c.smithi113.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T22:34:27.850 INFO:journalctl@ceph.mon.c.smithi113.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T22:34:27.850 INFO:journalctl@ceph.mon.c.smithi113.stdout: 2024-03-18T22:34:27.850 INFO:journalctl@ceph.mon.c.smithi113.stdout: ** Compaction Stats [default] ** 2024-03-18T22:34:27.850 INFO:journalctl@ceph.mon.c.smithi113.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T22:34:27.850 INFO:journalctl@ceph.mon.c.smithi113.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T22:34:27.850 INFO:journalctl@ceph.mon.c.smithi113.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T22:34:27.850 INFO:journalctl@ceph.mon.c.smithi113.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T22:34:27.850 INFO:journalctl@ceph.mon.c.smithi113.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T22:34:27.850 INFO:journalctl@ceph.mon.c.smithi113.stdout: 2024-03-18T22:34:27.850 INFO:journalctl@ceph.mon.c.smithi113.stdout: ** Compaction Stats [default] ** 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T22:34:27.851 INFO:journalctl@ceph.mon.c.smithi113.stdout: Block cache BinnedLRUCache@0x5563244ed090#2 capacity: 512.00 MB usage: 0.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout: Block cache entry stats(count,size,portion): DataBlock(1,0.64 KB,0.00012219%) FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout: 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: using public_addr v2:172.21.15.113:0/0 -> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: starting mon.c rank -1 at public addrs [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] at bind addrs [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: mon.c@-1(???) e0 preinit fsid 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: mon.c@-1(synchronizing).mds e1 new map 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout: e1 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout: legacy client fscid: -1 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout: 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout: No filesystems configured 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-18T22:34:27.852 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-18T22:34:27.853 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-18T22:34:27.853 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-18T22:34:27.853 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-18T22:34:27.853 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-18T22:34:27.853 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T22:34:27.853 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T22:34:27.853 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T22:34:27.853 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T22:34:27.853 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:27.853 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.853 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:34:27.853 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:27.853 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi113", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:34:27.853 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.853 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.854 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.854 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.854 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:34:27.854 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:27.854 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:34:27.854 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: Updating smithi103:/etc/ceph/ceph.conf 2024-03-18T22:34:27.854 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: Deploying cephadm binary to smithi113 2024-03-18T22:34:27.854 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: Updating smithi103:/var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T22:34:27.854 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-03-18T22:34:27.854 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: Updating smithi103:/var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T22:34:27.854 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:27.854 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:27.854 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.854 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.854 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: Added host smithi113 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1689863752' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi113", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: Updating smithi113:/etc/ceph/ceph.conf 2024-03-18T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1689863752' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: Updating smithi113:/var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: Updating smithi113:/etc/ceph/ceph.client.admin.keyring 2024-03-18T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: Updating smithi113:/var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi040:172.21.15.40=a;smithi103:172.21.15.103=b;smithi113:172.21.15.113=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: Saving service mon spec with placement smithi040:172.21.15.40=a;smithi103:172.21.15.103=b;smithi113:172.21.15.113=c;count:3 2024-03-18T22:34:27.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:34:27.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:27.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:34:27.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:27.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T22:34:27.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:27.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: Deploying daemon mon.c on smithi113 2024-03-18T22:34:27.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:27.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:27.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:27.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 ceph-mon[66775]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-18T22:34:27.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 bash[66739]: 3c9a2ff1f2041c946902d53fa437fcd818a051ccfd170fd7eadca1acdf7d4ddb 2024-03-18T22:34:27.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:27 smithi113 systemd[1]: Started Ceph mon.c for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:34:28.576 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T22:34:28.577 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T22:34:28.816 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:34:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:34:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: mon.a calling monitor election 2024-03-18T22:34:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: mon.c calling monitor election 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: monmap epoch 2 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: fsid 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: last_changed 2024-03-18T22:34:27.706954+0000 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: created 2024-03-18T22:32:44.918096+0000 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: min_mon_release 19 (squid) 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: election_strategy: 1 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: 1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.c 2024-03-18T22:34:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: fsmap 2024-03-18T22:34:33.082 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T22:34:33.082 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: mgrmap e13: x(active, since 50s) 2024-03-18T22:34:33.082 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: overall HEALTH_OK 2024-03-18T22:34:33.082 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:33.082 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:34:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: mon.a calling monitor election 2024-03-18T22:34:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: mon.c calling monitor election 2024-03-18T22:34:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: monmap epoch 2 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: fsid 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: last_changed 2024-03-18T22:34:27.706954+0000 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: created 2024-03-18T22:32:44.918096+0000 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: min_mon_release 19 (squid) 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: election_strategy: 1 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: 1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.c 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: fsmap 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: mgrmap e13: x(active, since 50s) 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: overall HEALTH_OK 2024-03-18T22:34:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:33.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:33.150 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.149+0000 7f998822d700 1 -- 172.21.15.113:0/1002652330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f996c0035c0 msgr2=0x7f996c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:33.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.149+0000 7f998822d700 1 --2- 172.21.15.113:0/1002652330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f996c0035c0 0x7f996c005a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f998006a570 tx=0x7f997000feb0 comp rx=0 tx=0).stop 2024-03-18T22:34:33.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.149+0000 7f998822d700 1 -- 172.21.15.113:0/1002652330 shutdown_connections 2024-03-18T22:34:33.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.149+0000 7f998822d700 1 --2- 172.21.15.113:0/1002652330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f996c0035c0 0x7f996c005a50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:33.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.149+0000 7f998822d700 1 -- 172.21.15.113:0/1002652330 >> 172.21.15.113:0/1002652330 conn(0x7f99800fc7b0 msgr2=0x7f99800febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:33.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.149+0000 7f998822d700 1 -- 172.21.15.113:0/1002652330 shutdown_connections 2024-03-18T22:34:33.151 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.149+0000 7f998822d700 1 -- 172.21.15.113:0/1002652330 wait complete. 2024-03-18T22:34:33.152 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.150+0000 7f998822d700 1 Processor -- start 2024-03-18T22:34:33.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.150+0000 7f998822d700 1 -- start start 2024-03-18T22:34:33.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.150+0000 7f998822d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f996c0035c0 0x7f998019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:33.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.151+0000 7f998822d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f998019fc30 0x7f99801998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:33.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.151+0000 7f998822d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9980109d20 con 0x7f996c0035c0 2024-03-18T22:34:33.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.151+0000 7f998822d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9980109ba0 con 0x7f998019fc30 2024-03-18T22:34:33.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.151+0000 7f99857c8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f998019fc30 0x7f99801998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:33.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.151+0000 7f9985fc9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f996c0035c0 0x7f998019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:33.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.151+0000 7f99857c8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f998019fc30 0x7f99801998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41188/0 (socket says 172.21.15.113:41188) 2024-03-18T22:34:33.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.151+0000 7f99857c8700 1 -- 172.21.15.113:0/2225612533 learned_addr learned my addr 172.21.15.113:0/2225612533 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-18T22:34:33.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.151+0000 7f99857c8700 1 -- 172.21.15.113:0/2225612533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f998019fc30 msgr2=0x7f99801998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-18T22:34:33.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.151+0000 7f99857c8700 1 -- 172.21.15.113:0/2225612533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f998019fc30 msgr2=0x7f99801998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T22:34:33.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.151+0000 7f99857c8700 1 --2- 172.21.15.113:0/2225612533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f998019fc30 0x7f99801998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T22:34:33.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.151+0000 7f99857c8700 1 --2- 172.21.15.113:0/2225612533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f998019fc30 0x7f99801998b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T22:34:33.153 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.151+0000 7f9985fc9700 1 -- 172.21.15.113:0/2225612533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f998019fc30 msgr2=0x7f99801998b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T22:34:33.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.151+0000 7f9985fc9700 1 --2- 172.21.15.113:0/2225612533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f998019fc30 0x7f99801998b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:33.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.151+0000 7f9985fc9700 1 -- 172.21.15.113:0/2225612533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99801a0270 con 0x7f996c0035c0 2024-03-18T22:34:33.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.151+0000 7f9985fc9700 1 --2- 172.21.15.113:0/2225612533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f996c0035c0 0x7f998019f6f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9970010010 tx=0x7f9970005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:33.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.152+0000 7f9976ffd700 1 -- 172.21.15.113:0/2225612533 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9970004aa0 con 0x7f996c0035c0 2024-03-18T22:34:33.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.152+0000 7f9976ffd700 1 -- 172.21.15.113:0/2225612533 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9970004030 con 0x7f996c0035c0 2024-03-18T22:34:33.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.152+0000 7f998822d700 1 -- 172.21.15.113:0/2225612533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9980199df0 con 0x7f996c0035c0 2024-03-18T22:34:33.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.152+0000 7f9976ffd700 1 -- 172.21.15.113:0/2225612533 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9970011b30 con 0x7f996c0035c0 2024-03-18T22:34:33.154 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.152+0000 7f998822d700 1 -- 172.21.15.113:0/2225612533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f998019a380 con 0x7f996c0035c0 2024-03-18T22:34:33.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.153+0000 7f9976ffd700 1 -- 172.21.15.113:0/2225612533 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f9970023070 con 0x7f996c0035c0 2024-03-18T22:34:33.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.153+0000 7f9976ffd700 1 --2- 172.21.15.113:0/2225612533 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9964038db0 0x7f996403b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:33.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.153+0000 7f998822d700 1 -- 172.21.15.113:0/2225612533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f998006ad70 con 0x7f996c0035c0 2024-03-18T22:34:33.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.153+0000 7f9976ffd700 1 -- 172.21.15.113:0/2225612533 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9970020050 con 0x7f996c0035c0 2024-03-18T22:34:33.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.153+0000 7f99857c8700 1 --2- 172.21.15.113:0/2225612533 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9964038db0 0x7f996403b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:33.155 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.154+0000 7f99857c8700 1 --2- 172.21.15.113:0/2225612533 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9964038db0 0x7f996403b270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f997c009180 tx=0x7f997c008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:33.159 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.159+0000 7f9976ffd700 1 -- 172.21.15.113:0/2225612533 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9970028020 con 0x7f996c0035c0 2024-03-18T22:34:33.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.437+0000 7f998822d700 1 -- 172.21.15.113:0/2225612533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f998019ade0 con 0x7f996c0035c0 2024-03-18T22:34:33.438 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.437+0000 7f9976ffd700 1 -- 172.21.15.113:0/2225612533 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f997001a390 con 0x7f996c0035c0 2024-03-18T22:34:33.438 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:34:33.439 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":2,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","modified":"2024-03-18T22:34:27.706954Z","created":"2024-03-18T22:32:44.918096Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T22:34:33.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.440+0000 7f998822d700 1 -- 172.21.15.113:0/2225612533 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9964038db0 msgr2=0x7f996403b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:33.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.440+0000 7f998822d700 1 --2- 172.21.15.113:0/2225612533 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9964038db0 0x7f996403b270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f997c009180 tx=0x7f997c008040 comp rx=0 tx=0).stop 2024-03-18T22:34:33.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.440+0000 7f998822d700 1 -- 172.21.15.113:0/2225612533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f996c0035c0 msgr2=0x7f998019f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:33.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.440+0000 7f998822d700 1 --2- 172.21.15.113:0/2225612533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f996c0035c0 0x7f998019f6f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9970010010 tx=0x7f9970005020 comp rx=0 tx=0).stop 2024-03-18T22:34:33.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.440+0000 7f998822d700 1 -- 172.21.15.113:0/2225612533 shutdown_connections 2024-03-18T22:34:33.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.440+0000 7f998822d700 1 --2- 172.21.15.113:0/2225612533 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9964038db0 0x7f996403b270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:33.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.440+0000 7f998822d700 1 --2- 172.21.15.113:0/2225612533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f996c0035c0 0x7f998019f6f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:33.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.440+0000 7f998822d700 1 --2- 172.21.15.113:0/2225612533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f998019fc30 0x7f99801998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:33.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.440+0000 7f998822d700 1 -- 172.21.15.113:0/2225612533 >> 172.21.15.113:0/2225612533 conn(0x7f99800fc7b0 msgr2=0x7f9980101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:33.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.440+0000 7f998822d700 1 -- 172.21.15.113:0/2225612533 shutdown_connections 2024-03-18T22:34:33.441 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:33.440+0000 7f998822d700 1 -- 172.21.15.113:0/2225612533 wait complete. 2024-03-18T22:34:33.442 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 2 2024-03-18T22:34:33.979 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:33 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:33.979 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:33 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T22:34:33.979 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:33 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:33.979 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:33 smithi113 ceph-mon[66775]: Deploying daemon mon.b on smithi103 2024-03-18T22:34:33.979 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:33 smithi113 ceph-mon[66775]: from='client.? 172.21.15.113:0/2225612533' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T22:34:33.979 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:33 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:34.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:33 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:34.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:33 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T22:34:34.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:33 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:34.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:33 smithi040 ceph-mon[62643]: Deploying daemon mon.b on smithi103 2024-03-18T22:34:34.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:33 smithi040 ceph-mon[62643]: from='client.? 172.21.15.113:0/2225612533' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T22:34:34.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:33 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:35.005 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T22:34:35.006 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T22:34:35.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:34 smithi113 ceph-mon[66775]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:35.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:34 smithi040 ceph-mon[62643]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:35.248 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:34:36.756 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.753+0000 7f32cd72f700 1 -- 172.21.15.113:0/1240932102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32c8102060 msgr2=0x7f32c8102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:36.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.753+0000 7f32cd72f700 1 --2- 172.21.15.113:0/1240932102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32c8102060 0x7f32c8102440 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f32b8009910 tx=0x7f32b8009c20 comp rx=0 tx=0).stop 2024-03-18T22:34:36.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.754+0000 7f32cd72f700 1 -- 172.21.15.113:0/1240932102 shutdown_connections 2024-03-18T22:34:36.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.754+0000 7f32cd72f700 1 --2- 172.21.15.113:0/1240932102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32c8102060 0x7f32c8102440 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:36.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.754+0000 7f32cd72f700 1 -- 172.21.15.113:0/1240932102 >> 172.21.15.113:0/1240932102 conn(0x7f32c80fd8d0 msgr2=0x7f32c80ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:36.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.754+0000 7f32cd72f700 1 -- 172.21.15.113:0/1240932102 shutdown_connections 2024-03-18T22:34:36.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.754+0000 7f32cd72f700 1 -- 172.21.15.113:0/1240932102 wait complete. 2024-03-18T22:34:36.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.754+0000 7f32cd72f700 1 Processor -- start 2024-03-18T22:34:36.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.754+0000 7f32cd72f700 1 -- start start 2024-03-18T22:34:36.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32cd72f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c8102060 0x7f32c819d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:36.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32cd72f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32c819dba0 0x7f32c81977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:36.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32cd72f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32c810c2e0 con 0x7f32c819dba0 2024-03-18T22:34:36.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32cd72f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f32c810c160 con 0x7f32c8102060 2024-03-18T22:34:36.757 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32c6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c8102060 0x7f32c819d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:36.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32c6ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c8102060 0x7f32c819d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:41194/0 (socket says 172.21.15.113:41194) 2024-03-18T22:34:36.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32c6ffd700 1 -- 172.21.15.113:0/4194941680 learned_addr learned my addr 172.21.15.113:0/4194941680 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-18T22:34:36.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32c6ffd700 1 -- 172.21.15.113:0/4194941680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c8102060 msgr2=0x7f32c819d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-18T22:34:36.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32c6ffd700 1 -- 172.21.15.113:0/4194941680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c8102060 msgr2=0x7f32c819d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T22:34:36.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32c6ffd700 1 --2- 172.21.15.113:0/4194941680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c8102060 0x7f32c819d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T22:34:36.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32c67fc700 1 --2- 172.21.15.113:0/4194941680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32c819dba0 0x7f32c81977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:36.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32c6ffd700 1 --2- 172.21.15.113:0/4194941680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c8102060 0x7f32c819d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T22:34:36.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32c67fc700 1 -- 172.21.15.113:0/4194941680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c8102060 msgr2=0x7f32c819d660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T22:34:36.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32c67fc700 1 --2- 172.21.15.113:0/4194941680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c8102060 0x7f32c819d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:36.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32c67fc700 1 -- 172.21.15.113:0/4194941680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32c819e1e0 con 0x7f32c819dba0 2024-03-18T22:34:36.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.755+0000 7f32c67fc700 1 --2- 172.21.15.113:0/4194941680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32c819dba0 0x7f32c81977e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f32bc00b4d0 tx=0x7f32bc00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:36.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.756+0000 7f32affff700 1 -- 172.21.15.113:0/4194941680 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f32bc011630 con 0x7f32c819dba0 2024-03-18T22:34:36.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.756+0000 7f32cd72f700 1 -- 172.21.15.113:0/4194941680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32c8197d80 con 0x7f32c819dba0 2024-03-18T22:34:36.758 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.756+0000 7f32affff700 1 -- 172.21.15.113:0/4194941680 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32bc011c80 con 0x7f32c819dba0 2024-03-18T22:34:36.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.756+0000 7f32affff700 1 -- 172.21.15.113:0/4194941680 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f32bc010da0 con 0x7f32c819dba0 2024-03-18T22:34:36.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.756+0000 7f32cd72f700 1 -- 172.21.15.113:0/4194941680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32c81982e0 con 0x7f32c819dba0 2024-03-18T22:34:36.759 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.757+0000 7f32affff700 1 -- 172.21.15.113:0/4194941680 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f32bc018020 con 0x7f32c819dba0 2024-03-18T22:34:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.757+0000 7f32cd72f700 1 -- 172.21.15.113:0/4194941680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32c8103540 con 0x7f32c819dba0 2024-03-18T22:34:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.757+0000 7f32affff700 1 --2- 172.21.15.113:0/4194941680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f32b0038d60 0x7f32b003b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.757+0000 7f32affff700 1 -- 172.21.15.113:0/4194941680 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f32bc04bab0 con 0x7f32c819dba0 2024-03-18T22:34:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.757+0000 7f32c6ffd700 1 --2- 172.21.15.113:0/4194941680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f32b0038d60 0x7f32b003b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:36.760 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.757+0000 7f32c6ffd700 1 --2- 172.21.15.113:0/4194941680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f32b0038d60 0x7f32b003b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f32b8009750 tx=0x7f32b801a040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:36.763 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.762+0000 7f32affff700 1 -- 172.21.15.113:0/4194941680 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f32bc01f020 con 0x7f32c819dba0 2024-03-18T22:34:36.900 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:36.899+0000 7f32affff700 1 -- 172.21.15.113:0/4194941680 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32bc01a400 con 0x7f32c819dba0 2024-03-18T22:34:37.041 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:37.038+0000 7f32cd72f700 1 -- 172.21.15.113:0/4194941680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f32c8198650 con 0x7f32c819dba0 2024-03-18T22:34:37.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:36 smithi113 ceph-mon[66775]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:37.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:36 smithi103 systemd[1]: Started Ceph mon.b for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:34:37.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:36 smithi040 ceph-mon[62643]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:41.932 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:41.931+0000 7f32affff700 1 -- 172.21.15.113:0/4194941680 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f32bc03e520 con 0x7f32c819dba0 2024-03-18T22:34:41.933 INFO:teuthology.orchestra.run.smithi113.stdout: 2024-03-18T22:34:41.933 INFO:teuthology.orchestra.run.smithi113.stdout:{"epoch":3,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","modified":"2024-03-18T22:34:36.898839Z","created":"2024-03-18T22:32:44.918096Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:3300","nonce":0},{"type":"v1","addr":"172.21.15.40:6789","nonce":0}]},"addr":"172.21.15.40:6789/0","public_addr":"172.21.15.40:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:3300","nonce":0},{"type":"v1","addr":"172.21.15.113:6789","nonce":0}]},"addr":"172.21.15.113:6789/0","public_addr":"172.21.15.113:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T22:34:41.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:41.934+0000 7f32cd72f700 1 -- 172.21.15.113:0/4194941680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f32b0038d60 msgr2=0x7f32b003b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:41.935 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:41.934+0000 7f32cd72f700 1 --2- 172.21.15.113:0/4194941680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f32b0038d60 0x7f32b003b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f32b8009750 tx=0x7f32b801a040 comp rx=0 tx=0).stop 2024-03-18T22:34:41.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:41.934+0000 7f32cd72f700 1 -- 172.21.15.113:0/4194941680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32c819dba0 msgr2=0x7f32c81977e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:41.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:41.934+0000 7f32cd72f700 1 --2- 172.21.15.113:0/4194941680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32c819dba0 0x7f32c81977e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f32bc00b4d0 tx=0x7f32bc00b890 comp rx=0 tx=0).stop 2024-03-18T22:34:41.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:41.934+0000 7f32cd72f700 1 -- 172.21.15.113:0/4194941680 shutdown_connections 2024-03-18T22:34:41.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:41.934+0000 7f32cd72f700 1 --2- 172.21.15.113:0/4194941680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f32b0038d60 0x7f32b003b220 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:41.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:41.934+0000 7f32cd72f700 1 --2- 172.21.15.113:0/4194941680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f32c8102060 0x7f32c819d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:41.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:41.934+0000 7f32cd72f700 1 --2- 172.21.15.113:0/4194941680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32c819dba0 0x7f32c81977e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:41.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:41.934+0000 7f32cd72f700 1 -- 172.21.15.113:0/4194941680 >> 172.21.15.113:0/4194941680 conn(0x7f32c80fd8d0 msgr2=0x7f32c8103a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:41.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:41.935+0000 7f32cd72f700 1 -- 172.21.15.113:0/4194941680 shutdown_connections 2024-03-18T22:34:41.936 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:41.935+0000 7f32cd72f700 1 -- 172.21.15.113:0/4194941680 wait complete. 2024-03-18T22:34:41.937 INFO:teuthology.orchestra.run.smithi113.stderr:dumped monmap epoch 3 2024-03-18T22:34:42.290 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: mon.c calling monitor election 2024-03-18T22:34:42.290 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:34:42.291 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: mon.a calling monitor election 2024-03-18T22:34:42.291 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:42.291 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:42.291 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: from='client.? 172.21.15.113:0/4194941680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T22:34:42.291 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:42.291 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:42.291 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:42.291 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:42.291 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:42.291 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:42.291 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:42.291 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T22:34:42.291 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: monmap epoch 3 2024-03-18T22:34:42.292 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: fsid 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:34:42.292 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: last_changed 2024-03-18T22:34:36.898839+0000 2024-03-18T22:34:42.292 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: created 2024-03-18T22:32:44.918096+0000 2024-03-18T22:34:42.292 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: min_mon_release 19 (squid) 2024-03-18T22:34:42.292 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: election_strategy: 1 2024-03-18T22:34:42.292 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-03-18T22:34:42.292 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: 1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.c 2024-03-18T22:34:42.292 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: 2: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-03-18T22:34:42.292 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: fsmap 2024-03-18T22:34:42.292 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T22:34:42.292 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: mgrmap e13: x(active, since 59s) 2024-03-18T22:34:42.292 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T22:34:42.292 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T22:34:42.292 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T22:34:42.292 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: mon.b (rank 2) addr [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] is down (out of quorum) 2024-03-18T22:34:42.293 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:42.293 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:42.293 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:42.293 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:42.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: mon.c calling monitor election 2024-03-18T22:34:42.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:34:42.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: mon.a calling monitor election 2024-03-18T22:34:42.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:42.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:42.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: from='client.? 172.21.15.113:0/4194941680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T22:34:42.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:42.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:42.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:42.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:42.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:42.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:42.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:42.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T22:34:42.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: monmap epoch 3 2024-03-18T22:34:42.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: fsid 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:34:42.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: last_changed 2024-03-18T22:34:36.898839+0000 2024-03-18T22:34:42.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: created 2024-03-18T22:32:44.918096+0000 2024-03-18T22:34:42.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: min_mon_release 19 (squid) 2024-03-18T22:34:42.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: election_strategy: 1 2024-03-18T22:34:42.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-03-18T22:34:42.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: 1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.c 2024-03-18T22:34:42.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: 2: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-03-18T22:34:42.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: fsmap 2024-03-18T22:34:42.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T22:34:42.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: mgrmap e13: x(active, since 59s) 2024-03-18T22:34:42.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T22:34:42.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T22:34:42.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T22:34:42.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: mon.b (rank 2) addr [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] is down (out of quorum) 2024-03-18T22:34:42.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:42.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:42.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:42.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:42.570 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-18T22:34:42.570 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-18T22:34:42.825 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:34:43.200 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:34:43.201 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:42 smithi040 ceph-mon[62643]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:43.201 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:43.201 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:43.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:42 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:34:43.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:42 smithi113 ceph-mon[66775]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:43.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:42 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:43.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:42 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: mon.c calling monitor election 2024-03-18T22:34:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:34:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: mon.a calling monitor election 2024-03-18T22:34:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:34:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='client.? 172.21.15.113:0/4194941680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T22:34:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T22:34:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: monmap epoch 3 2024-03-18T22:34:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: fsid 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:34:44.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: last_changed 2024-03-18T22:34:36.898839+0000 2024-03-18T22:34:44.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: created 2024-03-18T22:32:44.918096+0000 2024-03-18T22:34:44.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: min_mon_release 19 (squid) 2024-03-18T22:34:44.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: election_strategy: 1 2024-03-18T22:34:44.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-03-18T22:34:44.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: 1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.c 2024-03-18T22:34:44.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: 2: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-03-18T22:34:44.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: fsmap 2024-03-18T22:34:44.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T22:34:44.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: mgrmap e13: x(active, since 59s) 2024-03-18T22:34:44.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T22:34:44.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T22:34:44.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T22:34:44.362 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: mon.b (rank 2) addr [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] is down (out of quorum) 2024-03-18T22:34:44.362 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:44.362 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:44.362 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:44.362 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:44.362 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:34:44.362 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:44.362 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:44.362 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:44.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.387+0000 7fd139ab6700 1 -- 172.21.15.40:0/3938509758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd134100e40 msgr2=0x7fd134101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:44.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.387+0000 7fd139ab6700 1 --2- 172.21.15.40:0/3938509758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd134100e40 0x7fd134101220 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd128009910 tx=0x7fd128009c20 comp rx=0 tx=0).stop 2024-03-18T22:34:44.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.387+0000 7fd139ab6700 1 -- 172.21.15.40:0/3938509758 shutdown_connections 2024-03-18T22:34:44.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.387+0000 7fd139ab6700 1 --2- 172.21.15.40:0/3938509758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd134100e40 0x7fd134101220 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:44.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.387+0000 7fd139ab6700 1 -- 172.21.15.40:0/3938509758 >> 172.21.15.40:0/3938509758 conn(0x7fd134077850 msgr2=0x7fd1340feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:44.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.387+0000 7fd139ab6700 1 -- 172.21.15.40:0/3938509758 shutdown_connections 2024-03-18T22:34:44.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.387+0000 7fd139ab6700 1 -- 172.21.15.40:0/3938509758 wait complete. 2024-03-18T22:34:44.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.388+0000 7fd139ab6700 1 Processor -- start 2024-03-18T22:34:44.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.388+0000 7fd139ab6700 1 -- start start 2024-03-18T22:34:44.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.388+0000 7fd139ab6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd134100e40 0x7fd1341028e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:44.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.388+0000 7fd139ab6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd134102e20 0x7fd1341032a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:44.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.388+0000 7fd139ab6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1341a3b80 0x7fd1341a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:44.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.388+0000 7fd139ab6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd13410b0c0 con 0x7fd1341a3b80 2024-03-18T22:34:44.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.388+0000 7fd139ab6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd13410af40 con 0x7fd134100e40 2024-03-18T22:34:44.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.388+0000 7fd139ab6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd13410b240 con 0x7fd134102e20 2024-03-18T22:34:44.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.388+0000 7fd133fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1341a3b80 0x7fd1341a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:44.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.388+0000 7fd133fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1341a3b80 0x7fd1341a5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51674/0 (socket says 172.21.15.40:51674) 2024-03-18T22:34:44.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.388+0000 7fd133fff700 1 -- 172.21.15.40:0/1486551397 learned_addr learned my addr 172.21.15.40:0/1486551397 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:34:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.388+0000 7fd132ffd700 1 --2- 172.21.15.40:0/1486551397 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd134102e20 0x7fd1341032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.388+0000 7fd1337fe700 1 --2- 172.21.15.40:0/1486551397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd134100e40 0x7fd1341028e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.389+0000 7fd133fff700 1 -- 172.21.15.40:0/1486551397 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd134102e20 msgr2=0x7fd1341032a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.389+0000 7fd133fff700 1 --2- 172.21.15.40:0/1486551397 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd134102e20 0x7fd1341032a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.389+0000 7fd133fff700 1 -- 172.21.15.40:0/1486551397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd134100e40 msgr2=0x7fd1341028e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.389+0000 7fd133fff700 1 --2- 172.21.15.40:0/1486551397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd134100e40 0x7fd1341028e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.389+0000 7fd133fff700 1 -- 172.21.15.40:0/1486551397 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1341a66a0 con 0x7fd1341a3b80 2024-03-18T22:34:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.389+0000 7fd1337fe700 1 --2- 172.21.15.40:0/1486551397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd134100e40 0x7fd1341028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:34:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.389+0000 7fd133fff700 1 --2- 172.21.15.40:0/1486551397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1341a3b80 0x7fd1341a5f70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd12c00ba50 tx=0x7fd12c00be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.389+0000 7fd130ff9700 1 -- 172.21.15.40:0/1486551397 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c019070 con 0x7fd1341a3b80 2024-03-18T22:34:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.389+0000 7fd130ff9700 1 -- 172.21.15.40:0/1486551397 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd12c014440 con 0x7fd1341a3b80 2024-03-18T22:34:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.389+0000 7fd139ab6700 1 -- 172.21.15.40:0/1486551397 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1341a6990 con 0x7fd1341a3b80 2024-03-18T22:34:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.389+0000 7fd130ff9700 1 -- 172.21.15.40:0/1486551397 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c007590 con 0x7fd1341a3b80 2024-03-18T22:34:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.389+0000 7fd139ab6700 1 -- 172.21.15.40:0/1486551397 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1341a6ed0 con 0x7fd1341a3b80 2024-03-18T22:34:44.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.390+0000 7fd139ab6700 1 -- 172.21.15.40:0/1486551397 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd134063940 con 0x7fd1341a3b80 2024-03-18T22:34:44.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.390+0000 7fd130ff9700 1 -- 172.21.15.40:0/1486551397 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fd12c01c070 con 0x7fd1341a3b80 2024-03-18T22:34:44.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.390+0000 7fd130ff9700 1 --2- 172.21.15.40:0/1486551397 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd11c039450 0x7fd11c03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:44.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.390+0000 7fd130ff9700 1 -- 172.21.15.40:0/1486551397 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd12c04b6d0 con 0x7fd1341a3b80 2024-03-18T22:34:44.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.390+0000 7fd1337fe700 1 --2- 172.21.15.40:0/1486551397 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd11c039450 0x7fd11c03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:44.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.391+0000 7fd1337fe700 1 --2- 172.21.15.40:0/1486551397 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd11c039450 0x7fd11c03b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd128006010 tx=0x7fd12801a040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:44.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.400+0000 7fd130ff9700 1 -- 172.21.15.40:0/1486551397 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd12c014980 con 0x7fd1341a3b80 2024-03-18T22:34:44.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.612+0000 7fd139ab6700 1 -- 172.21.15.40:0/1486551397 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fd1341a7160 con 0x7fd1341a3b80 2024-03-18T22:34:44.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.613+0000 7fd130ff9700 1 -- 172.21.15.40:0/1486551397 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7fd12c01d3b0 con 0x7fd1341a3b80 2024-03-18T22:34:44.614 INFO:teuthology.orchestra.run.smithi040.stdout:# minimal ceph.conf for 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:34:44.614 INFO:teuthology.orchestra.run.smithi040.stdout:[global] 2024-03-18T22:34:44.614 INFO:teuthology.orchestra.run.smithi040.stdout: fsid = 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:34:44.614 INFO:teuthology.orchestra.run.smithi040.stdout: mon_host = [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] 2024-03-18T22:34:44.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.615+0000 7fd139ab6700 1 -- 172.21.15.40:0/1486551397 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd11c039450 msgr2=0x7fd11c03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:44.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.615+0000 7fd139ab6700 1 --2- 172.21.15.40:0/1486551397 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd11c039450 0x7fd11c03b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd128006010 tx=0x7fd12801a040 comp rx=0 tx=0).stop 2024-03-18T22:34:44.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.615+0000 7fd139ab6700 1 -- 172.21.15.40:0/1486551397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1341a3b80 msgr2=0x7fd1341a5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:44.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.615+0000 7fd139ab6700 1 --2- 172.21.15.40:0/1486551397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1341a3b80 0x7fd1341a5f70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd12c00ba50 tx=0x7fd12c00be10 comp rx=0 tx=0).stop 2024-03-18T22:34:44.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.615+0000 7fd139ab6700 1 -- 172.21.15.40:0/1486551397 shutdown_connections 2024-03-18T22:34:44.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.615+0000 7fd139ab6700 1 --2- 172.21.15.40:0/1486551397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1341a3b80 0x7fd1341a5f70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:44.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.615+0000 7fd139ab6700 1 --2- 172.21.15.40:0/1486551397 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd11c039450 0x7fd11c03b910 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:44.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.615+0000 7fd139ab6700 1 --2- 172.21.15.40:0/1486551397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd134100e40 0x7fd1341028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:44.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.615+0000 7fd139ab6700 1 --2- 172.21.15.40:0/1486551397 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd134102e20 0x7fd1341032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:44.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.615+0000 7fd139ab6700 1 -- 172.21.15.40:0/1486551397 >> 172.21.15.40:0/1486551397 conn(0x7fd134077850 msgr2=0x7fd134069700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:44.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.615+0000 7fd139ab6700 1 -- 172.21.15.40:0/1486551397 shutdown_connections 2024-03-18T22:34:44.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:44.615+0000 7fd139ab6700 1 -- 172.21.15.40:0/1486551397 wait complete. 2024-03-18T22:34:45.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: mon.b calling monitor election 2024-03-18T22:34:45.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: mon.b calling monitor election 2024-03-18T22:34:45.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: mon.c calling monitor election 2024-03-18T22:34:45.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: mon.a calling monitor election 2024-03-18T22:34:45.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T22:34:45.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: monmap epoch 3 2024-03-18T22:34:45.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: fsid 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:34:45.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: last_changed 2024-03-18T22:34:36.898839+0000 2024-03-18T22:34:45.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: created 2024-03-18T22:32:44.918096+0000 2024-03-18T22:34:45.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: min_mon_release 19 (squid) 2024-03-18T22:34:45.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: election_strategy: 1 2024-03-18T22:34:45.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-03-18T22:34:45.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: 1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.c 2024-03-18T22:34:45.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: 2: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-03-18T22:34:45.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: fsmap 2024-03-18T22:34:45.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T22:34:45.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: mgrmap e13: x(active, since 61s) 2024-03-18T22:34:45.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T22:34:45.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: Cluster is now healthy 2024-03-18T22:34:45.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: overall HEALTH_OK 2024-03-18T22:34:45.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T22:34:45.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T22:34:45.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T22:34:45.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:45.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: Reconfiguring daemon mon.a on smithi040 2024-03-18T22:34:45.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1486551397' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:45.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:45.147 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: mon.b calling monitor election 2024-03-18T22:34:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: mon.b calling monitor election 2024-03-18T22:34:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: mon.c calling monitor election 2024-03-18T22:34:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: mon.a calling monitor election 2024-03-18T22:34:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T22:34:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: monmap epoch 3 2024-03-18T22:34:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: fsid 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:34:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: last_changed 2024-03-18T22:34:36.898839+0000 2024-03-18T22:34:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: created 2024-03-18T22:32:44.918096+0000 2024-03-18T22:34:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: min_mon_release 19 (squid) 2024-03-18T22:34:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: election_strategy: 1 2024-03-18T22:34:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-03-18T22:34:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: 1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.c 2024-03-18T22:34:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: 2: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-03-18T22:34:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: fsmap 2024-03-18T22:34:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T22:34:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: mgrmap e13: x(active, since 61s) 2024-03-18T22:34:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T22:34:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: Cluster is now healthy 2024-03-18T22:34:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: overall HEALTH_OK 2024-03-18T22:34:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T22:34:45.333 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T22:34:45.333 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T22:34:45.333 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:45.333 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: Reconfiguring daemon mon.a on smithi040 2024-03-18T22:34:45.333 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1486551397' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:45.333 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:45.333 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: mon.b calling monitor election 2024-03-18T22:34:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: mon.b calling monitor election 2024-03-18T22:34:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: mon.c calling monitor election 2024-03-18T22:34:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: mon.a calling monitor election 2024-03-18T22:34:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T22:34:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: monmap epoch 3 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: fsid 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: last_changed 2024-03-18T22:34:36.898839+0000 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: created 2024-03-18T22:32:44.918096+0000 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: min_mon_release 19 (squid) 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: election_strategy: 1 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: 0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: 1: [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] mon.c 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: 2: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: fsmap 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: mgrmap e13: x(active, since 61s) 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: Cluster is now healthy 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: overall HEALTH_OK 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:45.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T22:34:45.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T22:34:45.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T22:34:45.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:45.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: Reconfiguring daemon mon.a on smithi040 2024-03-18T22:34:45.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1486551397' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:45.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:45.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:34:46.179 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-18T22:34:46.179 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:34:46.179 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T22:34:46.208 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:34:46.208 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T22:34:46.275 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-18T22:34:46.275 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T22:34:46.303 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-18T22:34:46.304 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T22:34:46.371 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-18T22:34:46.371 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T22:34:46.406 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-18T22:34:46.406 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T22:34:46.477 INFO:tasks.cephadm:Adding mgr.x on smithi040 2024-03-18T22:34:46.477 INFO:tasks.cephadm:Adding mgr.y on smithi103 2024-03-18T22:34:46.477 INFO:tasks.cephadm:Adding mgr.z on smithi113 2024-03-18T22:34:46.477 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi040=x;smithi103=y;smithi113=z' 2024-03-18T22:34:46.748 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:34:46.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:46 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:46.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:46 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:46.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:46 smithi103 ceph-mon[66377]: Reconfiguring mon.b (monmap changed)... 2024-03-18T22:34:46.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:46 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T22:34:46.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:46 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T22:34:46.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:46 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:46.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:46 smithi103 ceph-mon[66377]: Reconfiguring daemon mon.b on smithi103 2024-03-18T22:34:46.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:46 smithi103 ceph-mon[66377]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:46 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:46 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:46 smithi113 ceph-mon[66775]: Reconfiguring mon.b (monmap changed)... 2024-03-18T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:46 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:46 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:46 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:47.082 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:46 smithi113 ceph-mon[66775]: Reconfiguring daemon mon.b on smithi103 2024-03-18T22:34:47.083 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:46 smithi113 ceph-mon[66775]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:47.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:46 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:47.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:46 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:47.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:46 smithi040 ceph-mon[62643]: Reconfiguring mon.b (monmap changed)... 2024-03-18T22:34:47.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:46 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T22:34:47.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:46 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T22:34:47.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:46 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:47.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:46 smithi040 ceph-mon[62643]: Reconfiguring daemon mon.b on smithi103 2024-03-18T22:34:47.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:46 smithi040 ceph-mon[62643]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:48.140 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.139+0000 7ff46d6ca700 1 -- 172.21.15.113:0/3731449355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff468101e60 msgr2=0x7ff468102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:48.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.139+0000 7ff46d6ca700 1 --2- 172.21.15.113:0/3731449355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff468101e60 0x7ff468102280 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff45c009910 tx=0x7ff45c009c20 comp rx=0 tx=0).stop 2024-03-18T22:34:48.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.140+0000 7ff46d6ca700 1 -- 172.21.15.113:0/3731449355 shutdown_connections 2024-03-18T22:34:48.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.140+0000 7ff46d6ca700 1 --2- 172.21.15.113:0/3731449355 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff468101e60 0x7ff468102280 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:48.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.140+0000 7ff46d6ca700 1 -- 172.21.15.113:0/3731449355 >> 172.21.15.113:0/3731449355 conn(0x7ff4680fd400 msgr2=0x7ff4680ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:48.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.140+0000 7ff46d6ca700 1 -- 172.21.15.113:0/3731449355 shutdown_connections 2024-03-18T22:34:48.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.140+0000 7ff46d6ca700 1 -- 172.21.15.113:0/3731449355 wait complete. 2024-03-18T22:34:48.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.140+0000 7ff46d6ca700 1 Processor -- start 2024-03-18T22:34:48.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.140+0000 7ff46d6ca700 1 -- start start 2024-03-18T22:34:48.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff46d6ca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff468197760 0x7ff468197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:48.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff46d6ca700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4681980c0 0x7ff46819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:48.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff46d6ca700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff46819d290 0x7ff46819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:48.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff46d6ca700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff468107d20 con 0x7ff46819d290 2024-03-18T22:34:48.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff46d6ca700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff468107ba0 con 0x7ff468197760 2024-03-18T22:34:48.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff46d6ca700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff468107ea0 con 0x7ff4681980c0 2024-03-18T22:34:48.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff4667fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4681980c0 0x7ff46819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:48.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff4667fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4681980c0 0x7ff46819cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:56184/0 (socket says 172.21.15.113:56184) 2024-03-18T22:34:48.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff4667fc700 1 -- 172.21.15.113:0/1777183125 learned_addr learned my addr 172.21.15.113:0/1777183125 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-18T22:34:48.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff466ffd700 1 --2- 172.21.15.113:0/1777183125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff468197760 0x7ff468197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:48.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff4677fe700 1 --2- 172.21.15.113:0/1777183125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff46819d290 0x7ff46819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff4667fc700 1 -- 172.21.15.113:0/1777183125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff468197760 msgr2=0x7ff468197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff4667fc700 1 --2- 172.21.15.113:0/1777183125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff468197760 0x7ff468197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff4667fc700 1 -- 172.21.15.113:0/1777183125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff46819d290 msgr2=0x7ff46819f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff4667fc700 1 --2- 172.21.15.113:0/1777183125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff46819d290 0x7ff46819f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff4667fc700 1 -- 172.21.15.113:0/1777183125 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff46819ffb0 con 0x7ff4681980c0 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff4667fc700 1 --2- 172.21.15.113:0/1777183125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4681980c0 0x7ff46819cd50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff45800b780 tx=0x7ff45800ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.141+0000 7ff44ffff700 1 -- 172.21.15.113:0/1777183125 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff458016070 con 0x7ff4681980c0 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.142+0000 7ff46d6ca700 1 -- 172.21.15.113:0/1777183125 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4681a02a0 con 0x7ff4681980c0 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.142+0000 7ff44ffff700 1 -- 172.21.15.113:0/1777183125 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4580036a0 con 0x7ff4681980c0 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.142+0000 7ff44ffff700 1 -- 172.21.15.113:0/1777183125 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4580119c0 con 0x7ff4681980c0 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.142+0000 7ff46d6ca700 1 -- 172.21.15.113:0/1777183125 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4681a0860 con 0x7ff4681980c0 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.142+0000 7ff44ffff700 1 -- 172.21.15.113:0/1777183125 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7ff458019070 con 0x7ff4681980c0 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.143+0000 7ff44ffff700 1 --2- 172.21.15.113:0/1777183125 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff450039450 0x7ff45003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.143+0000 7ff44ffff700 1 -- 172.21.15.113:0/1777183125 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff45804abc0 con 0x7ff4681980c0 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.143+0000 7ff46d6ca700 1 -- 172.21.15.113:0/1777183125 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff468063940 con 0x7ff4681980c0 2024-03-18T22:34:48.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.143+0000 7ff466ffd700 1 --2- 172.21.15.113:0/1777183125 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff450039450 0x7ff45003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:48.146 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.143+0000 7ff466ffd700 1 --2- 172.21.15.113:0/1777183125 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff450039450 0x7ff45003b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff45c005e00 tx=0x7ff45c01a040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:48.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.148+0000 7ff44ffff700 1 -- 172.21.15.113:0/1777183125 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff458003bc0 con 0x7ff4681980c0 2024-03-18T22:34:48.362 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.362+0000 7ff46d6ca700 1 -- 172.21.15.113:0/1777183125 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi103=y;smithi113=z", "target": ["mon-mgr", ""]}) -- 0x7ff4680ff410 con 0x7ff450039450 2024-03-18T22:34:48.368 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.368+0000 7ff44ffff700 1 -- 172.21.15.113:0/1777183125 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ff4680ff410 con 0x7ff450039450 2024-03-18T22:34:48.369 INFO:teuthology.orchestra.run.smithi113.stdout:Scheduled mgr update... 2024-03-18T22:34:48.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.370+0000 7ff46d6ca700 1 -- 172.21.15.113:0/1777183125 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff450039450 msgr2=0x7ff45003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:48.371 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.370+0000 7ff46d6ca700 1 --2- 172.21.15.113:0/1777183125 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff450039450 0x7ff45003b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff45c005e00 tx=0x7ff45c01a040 comp rx=0 tx=0).stop 2024-03-18T22:34:48.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.370+0000 7ff46d6ca700 1 -- 172.21.15.113:0/1777183125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4681980c0 msgr2=0x7ff46819cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:48.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.370+0000 7ff46d6ca700 1 --2- 172.21.15.113:0/1777183125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4681980c0 0x7ff46819cd50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff45800b780 tx=0x7ff45800ba90 comp rx=0 tx=0).stop 2024-03-18T22:34:48.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.370+0000 7ff46d6ca700 1 -- 172.21.15.113:0/1777183125 shutdown_connections 2024-03-18T22:34:48.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.370+0000 7ff46d6ca700 1 --2- 172.21.15.113:0/1777183125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff46819d290 0x7ff46819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:48.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.370+0000 7ff46d6ca700 1 --2- 172.21.15.113:0/1777183125 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff450039450 0x7ff45003b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:48.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.370+0000 7ff46d6ca700 1 --2- 172.21.15.113:0/1777183125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff468197760 0x7ff468197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:48.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.370+0000 7ff46d6ca700 1 --2- 172.21.15.113:0/1777183125 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff4681980c0 0x7ff46819cd50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:48.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.371+0000 7ff46d6ca700 1 -- 172.21.15.113:0/1777183125 >> 172.21.15.113:0/1777183125 conn(0x7ff4680fd400 msgr2=0x7ff4680fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:48.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.371+0000 7ff46d6ca700 1 -- 172.21.15.113:0/1777183125 shutdown_connections 2024-03-18T22:34:48.372 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:34:48.371+0000 7ff46d6ca700 1 -- 172.21.15.113:0/1777183125 wait complete. 2024-03-18T22:34:48.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:48.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:48.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:48 smithi113 ceph-mon[66775]: Reconfiguring mon.c (monmap changed)... 2024-03-18T22:34:48.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T22:34:48.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T22:34:48.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:48.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:48 smithi113 ceph-mon[66775]: Reconfiguring daemon mon.c on smithi113 2024-03-18T22:34:48.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:48.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:48.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:48 smithi103 ceph-mon[66377]: Reconfiguring mon.c (monmap changed)... 2024-03-18T22:34:48.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T22:34:48.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T22:34:48.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:48.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:48 smithi103 ceph-mon[66377]: Reconfiguring daemon mon.c on smithi113 2024-03-18T22:34:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:48 smithi040 ceph-mon[62643]: Reconfiguring mon.c (monmap changed)... 2024-03-18T22:34:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T22:34:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T22:34:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:48 smithi040 ceph-mon[62643]: Reconfiguring daemon mon.c on smithi113 2024-03-18T22:34:49.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:49 smithi113 ceph-mon[66775]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi103=y;smithi113=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:34:49.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:49 smithi113 ceph-mon[66775]: Saving service mgr spec with placement smithi040=x;smithi103=y;smithi113=z;count:3 2024-03-18T22:34:49.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:49 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:49.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:49 smithi113 ceph-mon[66775]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:49.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:49 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:49.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:49 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:49.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:49 smithi040 ceph-mon[62643]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi103=y;smithi113=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:34:49.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:49 smithi040 ceph-mon[62643]: Saving service mgr spec with placement smithi040=x;smithi103=y;smithi113=z;count:3 2024-03-18T22:34:49.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:49 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:49.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:49 smithi040 ceph-mon[62643]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:49.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:49 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:49.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:49 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:49.748 DEBUG:teuthology.orchestra.run.smithi103:mgr.y> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mgr.y.service 2024-03-18T22:34:49.751 DEBUG:teuthology.orchestra.run.smithi113:mgr.z> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mgr.z.service 2024-03-18T22:34:49.753 INFO:tasks.cephadm:Deploying OSDs... 2024-03-18T22:34:49.754 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:34:49.754 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T22:34:49.769 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T22:34:49.769 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2024-03-18T22:34:49.778 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:49 smithi103 ceph-mon[66377]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi040=x;smithi103=y;smithi113=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:34:49.778 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:49 smithi103 ceph-mon[66377]: Saving service mgr spec with placement smithi040=x;smithi103=y;smithi113=z;count:3 2024-03-18T22:34:49.778 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:49 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:49.778 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:49 smithi103 ceph-mon[66377]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:49.778 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:49 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:49.778 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:49 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:49.824 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T22:34:49.824 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T22:34:49.824 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-18T22:34:49.824 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T22:34:49.824 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T22:34:49.824 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-03-18 22:33:52.629722913 +0000 2024-03-18T22:34:49.824 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-03-18 22:16:03.953300360 +0000 2024-03-18T22:34:49.824 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-03-18 22:16:03.953300360 +0000 2024-03-18T22:34:49.824 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-03-18 22:16:03.953300360 +0000 2024-03-18T22:34:49.825 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T22:34:49.890 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-03-18T22:34:49.890 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-03-18T22:34:49.890 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000254108 s, 2.0 MB/s 2024-03-18T22:34:49.891 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T22:34:49.946 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2024-03-18T22:34:50.000 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T22:34:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T22:34:50.001 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-18T22:34:50.001 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T22:34:50.001 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T22:34:50.001 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-03-18 22:33:52.657722543 +0000 2024-03-18T22:34:50.001 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-03-18 22:16:04.324295623 +0000 2024-03-18T22:34:50.001 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-03-18 22:16:04.324295623 +0000 2024-03-18T22:34:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-03-18 22:16:04.324295623 +0000 2024-03-18T22:34:50.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T22:34:50.066 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-03-18T22:34:50.067 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-03-18T22:34:50.067 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000194444 s, 2.6 MB/s 2024-03-18T22:34:50.068 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T22:34:50.123 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2024-03-18T22:34:50.178 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T22:34:50.178 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T22:34:50.178 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-18T22:34:50.178 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T22:34:50.178 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T22:34:50.178 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-03-18 22:33:52.683722200 +0000 2024-03-18T22:34:50.178 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-03-18 22:16:04.678291102 +0000 2024-03-18T22:34:50.178 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-03-18 22:16:04.678291102 +0000 2024-03-18T22:34:50.178 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-03-18 22:16:04.678291102 +0000 2024-03-18T22:34:50.178 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T22:34:50.243 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-03-18T22:34:50.243 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-03-18T22:34:50.243 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000196678 s, 2.6 MB/s 2024-03-18T22:34:50.244 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T22:34:50.299 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2024-03-18T22:34:50.354 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T22:34:50.354 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T22:34:50.354 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-18T22:34:50.354 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T22:34:50.354 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T22:34:50.354 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-03-18 22:33:52.709721856 +0000 2024-03-18T22:34:50.354 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-03-18 22:16:05.027286646 +0000 2024-03-18T22:34:50.354 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-03-18 22:16:05.027286646 +0000 2024-03-18T22:34:50.354 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-03-18 22:16:05.027286646 +0000 2024-03-18T22:34:50.355 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T22:34:50.420 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-03-18T22:34:50.420 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-03-18T22:34:50.420 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000216876 s, 2.4 MB/s 2024-03-18T22:34:50.421 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T22:34:50.476 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-03-18T22:34:50.476 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T22:34:50.495 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T22:34:50.495 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-03-18T22:34:50.553 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T22:34:50.553 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T22:34:50.553 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-18T22:34:50.554 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T22:34:50.554 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T22:34:50.554 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-03-18 22:34:36.271089385 +0000 2024-03-18T22:34:50.554 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-03-18 22:15:57.770969305 +0000 2024-03-18T22:34:50.554 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-03-18 22:15:57.770969305 +0000 2024-03-18T22:34:50.554 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-03-18 22:15:57.770969305 +0000 2024-03-18T22:34:50.554 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T22:34:50.624 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-03-18T22:34:50.624 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-03-18T22:34:50.624 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000235033 s, 2.2 MB/s 2024-03-18T22:34:50.625 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T22:34:50.631 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:50 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:34:50.632 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:50 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:50.632 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:50 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:34:50.632 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:50 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:50.632 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:50 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T22:34:50.632 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:50 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T22:34:50.632 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:50 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T22:34:50.632 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:50 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:50.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:50 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:34:50.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:50 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:50.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:50 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:34:50.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:50 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:50.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:50 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T22:34:50.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:50 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T22:34:50.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:50 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T22:34:50.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:50 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:50.683 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-03-18T22:34:50.740 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T22:34:50.740 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T22:34:50.740 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-18T22:34:50.740 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T22:34:50.740 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T22:34:50.740 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-03-18 22:34:36.271089385 +0000 2024-03-18T22:34:50.740 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-03-18 22:15:58.126962271 +0000 2024-03-18T22:34:50.740 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-03-18 22:15:58.126962271 +0000 2024-03-18T22:34:50.740 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-03-18 22:15:58.126962271 +0000 2024-03-18T22:34:50.741 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T22:34:50.805 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:50 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:34:50.806 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:50 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:50.806 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:50 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:34:50.806 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:50 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:50.806 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:50 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T22:34:50.806 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:50 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T22:34:50.806 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:50 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T22:34:50.806 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:50 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:50.808 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-03-18T22:34:50.808 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-03-18T22:34:50.808 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000269525 s, 1.9 MB/s 2024-03-18T22:34:50.809 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T22:34:50.866 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-03-18T22:34:50.922 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T22:34:50.922 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T22:34:50.922 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-18T22:34:50.922 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T22:34:50.922 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T22:34:50.923 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-03-18 22:34:36.272089365 +0000 2024-03-18T22:34:50.923 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-03-18 22:15:58.457955730 +0000 2024-03-18T22:34:50.923 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-03-18 22:15:58.457955730 +0000 2024-03-18T22:34:50.923 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-03-18 22:15:58.457955730 +0000 2024-03-18T22:34:50.923 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T22:34:50.989 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-03-18T22:34:50.989 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-03-18T22:34:50.989 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000212301 s, 2.4 MB/s 2024-03-18T22:34:50.990 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T22:34:51.047 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-03-18T22:34:51.103 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T22:34:51.103 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T22:34:51.103 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-18T22:34:51.103 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T22:34:51.103 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T22:34:51.103 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-03-18 22:34:36.272089365 +0000 2024-03-18T22:34:51.103 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-03-18 22:15:58.786949229 +0000 2024-03-18T22:34:51.103 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-03-18 22:15:58.786949229 +0000 2024-03-18T22:34:51.104 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-03-18 22:15:58.786949229 +0000 2024-03-18T22:34:51.104 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T22:34:51.169 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-03-18T22:34:51.169 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-03-18T22:34:51.169 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000191334 s, 2.7 MB/s 2024-03-18T22:34:51.170 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T22:34:51.226 DEBUG:teuthology.orchestra.run.smithi113:> set -ex 2024-03-18T22:34:51.226 DEBUG:teuthology.orchestra.run.smithi113:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T22:34:51.243 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T22:34:51.243 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_1 2024-03-18T22:34:51.299 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T22:34:51.299 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T22:34:51.300 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-18T22:34:51.300 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T22:34:51.300 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T22:34:51.300 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-03-18 22:34:26.613924494 +0000 2024-03-18T22:34:51.300 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-03-18 22:16:02.031241572 +0000 2024-03-18T22:34:51.300 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-03-18 22:16:02.031241572 +0000 2024-03-18T22:34:51.300 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-03-18 22:16:02.031241572 +0000 2024-03-18T22:34:51.300 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T22:34:51.367 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-03-18T22:34:51.367 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-03-18T22:34:51.368 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000200261 s, 2.6 MB/s 2024-03-18T22:34:51.369 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T22:34:51.424 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_2 2024-03-18T22:34:51.484 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T22:34:51.484 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T22:34:51.484 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-18T22:34:51.484 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T22:34:51.484 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T22:34:51.484 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-03-18 22:34:26.613924494 +0000 2024-03-18T22:34:51.484 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-03-18 22:16:02.367234669 +0000 2024-03-18T22:34:51.484 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-03-18 22:16:02.367234669 +0000 2024-03-18T22:34:51.484 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-03-18 22:16:02.367234669 +0000 2024-03-18T22:34:51.484 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T22:34:51.589 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-03-18T22:34:51.589 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-03-18T22:34:51.589 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000187278 s, 2.7 MB/s 2024-03-18T22:34:51.590 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T22:34:51.606 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_3 2024-03-18T22:34:51.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:51 smithi103 ceph-mon[66377]: Deploying daemon mgr.z on smithi113 2024-03-18T22:34:51.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:51 smithi103 ceph-mon[66377]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:51.614 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:51 smithi113 ceph-mon[66775]: Deploying daemon mgr.z on smithi113 2024-03-18T22:34:51.615 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:51 smithi113 ceph-mon[66775]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:51.628 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T22:34:51.628 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T22:34:51.628 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-18T22:34:51.628 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T22:34:51.628 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T22:34:51.629 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-03-18 22:34:51.543393599 +0000 2024-03-18T22:34:51.629 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-03-18 22:16:02.725227314 +0000 2024-03-18T22:34:51.629 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-03-18 22:16:02.725227314 +0000 2024-03-18T22:34:51.629 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-03-18 22:16:02.725227314 +0000 2024-03-18T22:34:51.629 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T22:34:51.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:51 smithi040 ceph-mon[62643]: Deploying daemon mgr.z on smithi113 2024-03-18T22:34:51.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:51 smithi040 ceph-mon[62643]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:51.764 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-03-18T22:34:51.764 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-03-18T22:34:51.764 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000224527 s, 2.3 MB/s 2024-03-18T22:34:51.765 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T22:34:51.780 DEBUG:teuthology.orchestra.run.smithi113:> stat /dev/vg_nvme/lv_4 2024-03-18T22:34:51.836 INFO:teuthology.orchestra.run.smithi113.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T22:34:51.836 INFO:teuthology.orchestra.run.smithi113.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T22:34:51.836 INFO:teuthology.orchestra.run.smithi113.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-18T22:34:51.836 INFO:teuthology.orchestra.run.smithi113.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T22:34:51.836 INFO:teuthology.orchestra.run.smithi113.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T22:34:51.836 INFO:teuthology.orchestra.run.smithi113.stdout:Access: 2024-03-18 22:34:51.722389787 +0000 2024-03-18T22:34:51.836 INFO:teuthology.orchestra.run.smithi113.stdout:Modify: 2024-03-18 22:16:03.072220185 +0000 2024-03-18T22:34:51.836 INFO:teuthology.orchestra.run.smithi113.stdout:Change: 2024-03-18 22:16:03.072220185 +0000 2024-03-18T22:34:51.836 INFO:teuthology.orchestra.run.smithi113.stdout: Birth: 2024-03-18 22:16:03.072220185 +0000 2024-03-18T22:34:51.837 DEBUG:teuthology.orchestra.run.smithi113:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T22:34:51.902 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:51 smithi113 systemd[1]: Starting Ceph mgr.z for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:34:51.905 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records in 2024-03-18T22:34:51.905 INFO:teuthology.orchestra.run.smithi113.stderr:1+0 records out 2024-03-18T22:34:51.905 INFO:teuthology.orchestra.run.smithi113.stderr:512 bytes copied, 0.000222557 s, 2.3 MB/s 2024-03-18T22:34:51.907 DEBUG:teuthology.orchestra.run.smithi113:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T22:34:51.962 INFO:tasks.cephadm:Deploying osd.0 on smithi040 with /dev/vg_nvme/lv_4... 2024-03-18T22:34:51.963 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T22:34:52.211 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:34:52.218 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:52 smithi113 podman[68580]: 2024-03-18 22:34:51.909631039 +0000 UTC m=+0.018836881 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T22:34:52.218 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:52 smithi113 podman[68580]: 2024-03-18 22:34:52.006955504 +0000 UTC m=+0.116161344 container create 94352a482c0bd225209cd86341d3ef289c9236b1a5409f7e84b674822f2a1add (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-18T22:34:52.218 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:52 smithi113 podman[68580]: 2024-03-18 22:34:52.217734354 +0000 UTC m=+0.326940198 container init 94352a482c0bd225209cd86341d3ef289c9236b1a5409f7e84b674822f2a1add (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T22:34:52.580 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:52 smithi113 podman[68580]: 2024-03-18 22:34:52.220699253 +0000 UTC m=+0.329905100 container start 94352a482c0bd225209cd86341d3ef289c9236b1a5409f7e84b674822f2a1add (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T22:34:52.580 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:52 smithi113 ceph-mgr[68628]: -- 172.21.15.113:0/3693393737 <== mon.1 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560a7570c5a0 con 0x560a756e1000 2024-03-18T22:34:52.580 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:52 smithi113 bash[68580]: 94352a482c0bd225209cd86341d3ef289c9236b1a5409f7e84b674822f2a1add 2024-03-18T22:34:52.580 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:52 smithi113 systemd[1]: Started Ceph mgr.z for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:34:53.080 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:52 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:52.752+0000 7fe942c10200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T22:34:53.080 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:52 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:52.842+0000 7fe942c10200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T22:34:53.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:53.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:53.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:53.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T22:34:53.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T22:34:53.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T22:34:53.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:53.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:53 smithi040 ceph-mon[62643]: Deploying daemon mgr.y on smithi103 2024-03-18T22:34:53.536 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:53 smithi040 ceph-mon[62643]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:53.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:53.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:53.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:53.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T22:34:53.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T22:34:53.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T22:34:53.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:53.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:53 smithi103 ceph-mon[66377]: Deploying daemon mgr.y on smithi103 2024-03-18T22:34:53.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:53 smithi103 ceph-mon[66377]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:53.629 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:53 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:53.473+0000 7fe942c10200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T22:34:53.629 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:53 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:53.566+0000 7fe942c10200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T22:34:53.629 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:53.629 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:53.629 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:53.629 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T22:34:53.629 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T22:34:53.629 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T22:34:53.630 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:53.630 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:53 smithi113 ceph-mon[66775]: Deploying daemon mgr.y on smithi103 2024-03-18T22:34:53.630 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:53 smithi113 ceph-mon[66775]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:53.894 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:53 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:53.708+0000 7fe942c10200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T22:34:53.894 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:53 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:53.808+0000 7fe942c10200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T22:34:53.894 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:53 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:53.893+0000 7fe942c10200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T22:34:54.100 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T22:34:54.100 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T22:34:54.100 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-03-18T22:34:54.100 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-03-18T22:34:54.101 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011248 s, 932 MB/s 2024-03-18T22:34:54.101 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-03-18T22:34:54.447 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:54 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:54.157+0000 7fe942c10200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T22:34:54.447 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:54 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:54.266+0000 7fe942c10200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T22:34:54.447 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:54 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:54.352+0000 7fe942c10200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T22:34:54.447 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:54 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:54.445+0000 7fe942c10200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T22:34:54.508 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:54 smithi103 systemd[1]: Starting Ceph mgr.y for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:34:54.703 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi040:vg_nvme/lv_4 2024-03-18T22:34:54.802 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:54 smithi103 podman[67740]: 2024-03-18 22:34:54.507776372 +0000 UTC m=+0.098266812 container create 1cad52968eff6701ce3e140fd0a99616d9451d84a5cd6e79d705268bb1ebea9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-18T22:34:54.802 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:54 smithi103 podman[67740]: 2024-03-18 22:34:54.427970688 +0000 UTC m=+0.018461092 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T22:34:54.802 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:54 smithi103 podman[67740]: 2024-03-18 22:34:54.726494191 +0000 UTC m=+0.316984598 container init 1cad52968eff6701ce3e140fd0a99616d9451d84a5cd6e79d705268bb1ebea9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-18T22:34:54.802 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:54 smithi103 podman[67740]: 2024-03-18 22:34:54.7289374 +0000 UTC m=+0.319427797 container start 1cad52968eff6701ce3e140fd0a99616d9451d84a5cd6e79d705268bb1ebea9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-18T22:34:54.802 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:54 smithi103 ceph-mgr[67759]: -- 172.21.15.103:0/1936117427 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f19c13a5a0 con 0x55f19c10f000 2024-03-18T22:34:54.802 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:54 smithi103 bash[67740]: 1cad52968eff6701ce3e140fd0a99616d9451d84a5cd6e79d705268bb1ebea9a 2024-03-18T22:34:54.820 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:54 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:54.620+0000 7fe942c10200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T22:34:54.950 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:34:55.059 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:54 smithi040 ceph-mon[62643]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:55.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:54 smithi113 ceph-mon[66775]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:55.109 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:54 smithi103 systemd[1]: Started Ceph mgr.y for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:34:55.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:54 smithi103 ceph-mon[66377]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:55.609 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:55 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:55.318+0000 7fa2cb7eb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T22:34:55.609 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:55 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:55.419+0000 7fa2cb7eb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T22:34:55.875 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:55 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:55.608+0000 7fe942c10200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T22:34:55.875 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:55 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:55.745+0000 7fe942c10200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T22:34:56.129 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:56 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:56.118+0000 7fa2cb7eb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T22:34:56.130 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:56.130 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:56.130 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:56.130 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:56.130 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:34:56.136 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:56 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:56.077+0000 7fe942c10200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T22:34:56.136 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:56.136 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:56.136 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:56.137 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:56.137 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:34:56.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:56.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:56.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:56.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:56.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:34:56.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.426+0000 7f99b6ac0700 1 -- 172.21.15.40:0/4087414470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99b0101240 msgr2=0x7f99b0101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:56.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.426+0000 7f99b6ac0700 1 --2- 172.21.15.40:0/4087414470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99b0101240 0x7f99b0101660 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f99a40099c0 tx=0x7f99a4009cd0 comp rx=0 tx=0).stop 2024-03-18T22:34:56.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.426+0000 7f99b6ac0700 1 -- 172.21.15.40:0/4087414470 shutdown_connections 2024-03-18T22:34:56.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.426+0000 7f99b6ac0700 1 --2- 172.21.15.40:0/4087414470 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99b0102de0 0x7f99b0109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:56.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.426+0000 7f99b6ac0700 1 --2- 172.21.15.40:0/4087414470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99b0101240 0x7f99b0101660 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:56.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.426+0000 7f99b6ac0700 1 --2- 172.21.15.40:0/4087414470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99b0102440 0x7f99b01028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:56.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.426+0000 7f99b6ac0700 1 -- 172.21.15.40:0/4087414470 >> 172.21.15.40:0/4087414470 conn(0x7f99b00fc7e0 msgr2=0x7f99b00fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:34:56.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.426+0000 7f99b6ac0700 1 -- 172.21.15.40:0/4087414470 shutdown_connections 2024-03-18T22:34:56.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.426+0000 7f99b6ac0700 1 -- 172.21.15.40:0/4087414470 wait complete. 2024-03-18T22:34:56.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.427+0000 7f99b6ac0700 1 Processor -- start 2024-03-18T22:34:56.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.427+0000 7f99b6ac0700 1 -- start start 2024-03-18T22:34:56.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b6ac0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99b0101240 0x7f99b01985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:56.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b6ac0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99b0102440 0x7f99b0198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:56.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b6ac0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99b0102de0 0x7f99b019fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b6ac0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f99b010cd30 con 0x7f99b0102de0 2024-03-18T22:34:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b6ac0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f99b010cbb0 con 0x7f99b0101240 2024-03-18T22:34:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b6ac0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f99b010ceb0 con 0x7f99b0102440 2024-03-18T22:34:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b505d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99b0102de0 0x7f99b019fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b505d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99b0102de0 0x7f99b019fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40940/0 (socket says 172.21.15.40:40940) 2024-03-18T22:34:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b505d700 1 -- 172.21.15.40:0/986351739 learned_addr learned my addr 172.21.15.40:0/986351739 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:34:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b485c700 1 --2- 172.21.15.40:0/986351739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99b0101240 0x7f99b01985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99affff700 1 --2- 172.21.15.40:0/986351739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99b0102440 0x7f99b0198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b505d700 1 -- 172.21.15.40:0/986351739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99b0102440 msgr2=0x7f99b0198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b505d700 1 --2- 172.21.15.40:0/986351739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99b0102440 0x7f99b0198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b505d700 1 -- 172.21.15.40:0/986351739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99b0101240 msgr2=0x7f99b01985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:34:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b505d700 1 --2- 172.21.15.40:0/986351739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99b0101240 0x7f99b01985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:34:56.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b505d700 1 -- 172.21.15.40:0/986351739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b01a01b0 con 0x7f99b0102de0 2024-03-18T22:34:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99affff700 1 --2- 172.21.15.40:0/986351739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99b0102440 0x7f99b0198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:34:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b505d700 1 --2- 172.21.15.40:0/986351739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99b0102de0 0x7f99b019fb10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f99a800ef60 tx=0x7f99a800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99adffb700 1 -- 172.21.15.40:0/986351739 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a80092e0 con 0x7f99b0102de0 2024-03-18T22:34:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99adffb700 1 -- 172.21.15.40:0/986351739 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99a8004bb0 con 0x7f99b0102de0 2024-03-18T22:34:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99adffb700 1 -- 172.21.15.40:0/986351739 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a8004280 con 0x7f99b0102de0 2024-03-18T22:34:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.428+0000 7f99b6ac0700 1 -- 172.21.15.40:0/986351739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b01a0440 con 0x7f99b0102de0 2024-03-18T22:34:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.429+0000 7f99b6ac0700 1 -- 172.21.15.40:0/986351739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b01a09d0 con 0x7f99b0102de0 2024-03-18T22:34:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.429+0000 7f99adffb700 1 -- 172.21.15.40:0/986351739 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f99a801b030 con 0x7f99b0102de0 2024-03-18T22:34:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.430+0000 7f99adffb700 1 --2- 172.21.15.40:0/986351739 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f99a0039200 0x7f99a003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:34:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.430+0000 7f99adffb700 1 -- 172.21.15.40:0/986351739 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f99a804caf0 con 0x7f99b0102de0 2024-03-18T22:34:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.430+0000 7f99b485c700 1 --2- 172.21.15.40:0/986351739 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f99a0039200 0x7f99a003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:34:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.430+0000 7f99b6ac0700 1 -- 172.21.15.40:0/986351739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99b0068fd0 con 0x7f99b0102de0 2024-03-18T22:34:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.430+0000 7f99b485c700 1 --2- 172.21.15.40:0/986351739 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f99a0039200 0x7f99a003b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f99a4000c00 tx=0x7f99a4005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:34:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.435+0000 7f99adffb700 1 -- 172.21.15.40:0/986351739 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f99a8022020 con 0x7f99b0102de0 2024-03-18T22:34:56.482 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:56 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:56.214+0000 7fa2cb7eb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T22:34:56.482 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:56 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:56.371+0000 7fa2cb7eb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T22:34:56.482 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:56 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:56.480+0000 7fa2cb7eb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T22:34:56.560 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:56 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:56.171+0000 7fe942c10200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T22:34:56.560 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:56 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:56.364+0000 7fe942c10200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T22:34:56.560 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:56 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:56.559+0000 7fe942c10200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T22:34:56.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:56.654+0000 7f99b6ac0700 1 -- 172.21.15.40:0/986351739 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f99b006a740 con 0x7f99a0039200 2024-03-18T22:34:56.831 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:56 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:56.564+0000 7fa2cb7eb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T22:34:56.831 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:56 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:56.829+0000 7fa2cb7eb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T22:34:57.095 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:56 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:56.841+0000 7fe942c10200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T22:34:57.109 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:56 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:56.939+0000 7fa2cb7eb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T22:34:57.109 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:57.025+0000 7fa2cb7eb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T22:34:57.411 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:57.094+0000 7fe942c10200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T22:34:57.411 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:57.330+0000 7fe942c10200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T22:34:57.411 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:57.120+0000 7fa2cb7eb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T22:34:57.411 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:57.294+0000 7fa2cb7eb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T22:34:57.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:57.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:57.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:57.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:57.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:57.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:34:57.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:57.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:34:57.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:34:57.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:34:57.553 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:57.554 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T22:34:57.554 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T22:34:57.554 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T22:34:57.554 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:57.554 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: Reconfiguring daemon mgr.x on smithi040 2024-03-18T22:34:57.554 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:57 smithi040 ceph-mon[62643]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:57.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:57.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:57.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:57.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:57.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:57.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:34:57.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:57.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:34:57.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:34:57.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:34:57.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:57.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T22:34:57.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T22:34:57.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T22:34:57.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:57.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: Reconfiguring daemon mgr.x on smithi040 2024-03-18T22:34:57.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:57 smithi113 ceph-mon[66775]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:57.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:57.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:57.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:57.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:57.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:57.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:34:57.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:57.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:34:57.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:34:57.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:34:57.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:57.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T22:34:57.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T22:34:57.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T22:34:57.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:34:57.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: Reconfiguring daemon mgr.x on smithi040 2024-03-18T22:34:57.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:57 smithi103 ceph-mon[66377]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:58.579 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:58 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:58.125+0000 7fe942c10200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T22:34:58.609 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:58 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:58.262+0000 7fa2cb7eb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T22:34:58.609 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:58 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:58.392+0000 7fa2cb7eb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T22:34:58.861 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:58 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:58.861 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:58 smithi040 ceph-mon[62643]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:59.002 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:58 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:59.002 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:58 smithi103 ceph-mon[66377]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:59.002 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:58 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:58.721+0000 7fa2cb7eb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T22:34:59.002 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:58 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:58.809+0000 7fa2cb7eb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T22:34:59.002 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:59.000+0000 7fa2cb7eb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T22:34:59.012 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:58 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:34:59.012 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:58 smithi113 ceph-mon[66775]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:34:59.013 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:58 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:58.701+0000 7fe942c10200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T22:34:59.013 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:58 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:58.798+0000 7fe942c10200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T22:34:59.013 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:34:59 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z[68624]: 2024-03-18T22:34:59.011+0000 7fe942c10200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T22:34:59.359 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:59.186+0000 7fa2cb7eb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T22:34:59.725 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:59.468+0000 7fa2cb7eb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T22:34:59.726 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:59.724+0000 7fa2cb7eb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T22:34:59.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:34:59.982+0000 7f99adffb700 1 -- 172.21.15.40:0/986351739 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 14) ==== 91698+0+0 (secure 0 0 0) 0x7f99a8012050 con 0x7f99b0102de0 2024-03-18T22:35:00.109 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:34:59.957+0000 7fa2cb7eb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T22:35:00.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:00.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:00.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:35:00.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:00.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:35:00.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:00.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-mon[66377]: Standby manager daemon z started 2024-03-18T22:35:00.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-mon[66377]: from='mgr.? 172.21.15.113:0/654817116' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T22:35:00.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-mon[66377]: from='mgr.? 172.21.15.113:0/654817116' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T22:35:00.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-mon[66377]: from='mgr.? 172.21.15.113:0/654817116' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T22:35:00.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:34:59 smithi103 ceph-mon[66377]: from='mgr.? 172.21.15.113:0/654817116' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T22:35:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:35:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:35:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:59 smithi113 ceph-mon[66775]: Standby manager daemon z started 2024-03-18T22:35:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:59 smithi113 ceph-mon[66775]: from='mgr.? 172.21.15.113:0/654817116' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T22:35:00.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:59 smithi113 ceph-mon[66775]: from='mgr.? 172.21.15.113:0/654817116' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T22:35:00.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:59 smithi113 ceph-mon[66775]: from='mgr.? 172.21.15.113:0/654817116' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T22:35:00.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:34:59 smithi113 ceph-mon[66775]: from='mgr.? 172.21.15.113:0/654817116' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T22:35:00.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:35:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:35:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:59 smithi040 ceph-mon[62643]: Standby manager daemon z started 2024-03-18T22:35:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:59 smithi040 ceph-mon[62643]: from='mgr.? 172.21.15.113:0/654817116' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T22:35:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:59 smithi040 ceph-mon[62643]: from='mgr.? 172.21.15.113:0/654817116' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T22:35:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:59 smithi040 ceph-mon[62643]: from='mgr.? 172.21.15.113:0/654817116' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T22:35:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:34:59 smithi040 ceph-mon[62643]: from='mgr.? 172.21.15.113:0/654817116' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T22:35:01.109 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:35:00 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:35:00.755+0000 7fa2cb7eb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T22:35:01.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:00 smithi103 ceph-mon[66377]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-18T22:35:01.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:00 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T22:35:01.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:00 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2903962570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f712e31-408e-427e-8826-2afdfc70bf80"}]: dispatch 2024-03-18T22:35:01.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:00 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2903962570' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f712e31-408e-427e-8826-2afdfc70bf80"}]': finished 2024-03-18T22:35:01.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:00 smithi103 ceph-mon[66377]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T22:35:01.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:00 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:01.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:00 smithi103 ceph-mon[66377]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:01.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:00 smithi040 ceph-mon[62643]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-18T22:35:01.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:00 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T22:35:01.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:00 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2903962570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f712e31-408e-427e-8826-2afdfc70bf80"}]: dispatch 2024-03-18T22:35:01.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:00 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2903962570' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f712e31-408e-427e-8826-2afdfc70bf80"}]': finished 2024-03-18T22:35:01.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:00 smithi040 ceph-mon[62643]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T22:35:01.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:00 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:01.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:00 smithi040 ceph-mon[62643]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:00 smithi113 ceph-mon[66775]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-18T22:35:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:00 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T22:35:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:00 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2903962570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f712e31-408e-427e-8826-2afdfc70bf80"}]: dispatch 2024-03-18T22:35:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:00 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2903962570' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f712e31-408e-427e-8826-2afdfc70bf80"}]': finished 2024-03-18T22:35:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:00 smithi113 ceph-mon[66775]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T22:35:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:00 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:00 smithi113 ceph-mon[66775]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:01.609 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:35:01 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:35:01.319+0000 7fa2cb7eb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T22:35:01.609 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:35:01 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:35:01.416+0000 7fa2cb7eb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T22:35:01.984 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:35:01 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y[67755]: 2024-03-18T22:35:01.630+0000 7fa2cb7eb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T22:35:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:01 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1375972696' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:35:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:01 smithi113 ceph-mon[66775]: Standby manager daemon y started 2024-03-18T22:35:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:01 smithi113 ceph-mon[66775]: from='mgr.? 172.21.15.103:0/2207414773' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T22:35:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:01 smithi113 ceph-mon[66775]: from='mgr.? 172.21.15.103:0/2207414773' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T22:35:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:01 smithi113 ceph-mon[66775]: from='mgr.? 172.21.15.103:0/2207414773' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T22:35:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:01 smithi113 ceph-mon[66775]: from='mgr.? 172.21.15.103:0/2207414773' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T22:35:02.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:01 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1375972696' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:35:02.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:01 smithi103 ceph-mon[66377]: Standby manager daemon y started 2024-03-18T22:35:02.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:01 smithi103 ceph-mon[66377]: from='mgr.? 172.21.15.103:0/2207414773' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T22:35:02.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:01 smithi103 ceph-mon[66377]: from='mgr.? 172.21.15.103:0/2207414773' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T22:35:02.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:01 smithi103 ceph-mon[66377]: from='mgr.? 172.21.15.103:0/2207414773' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T22:35:02.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:01 smithi103 ceph-mon[66377]: from='mgr.? 172.21.15.103:0/2207414773' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T22:35:02.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:01 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1375972696' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:35:02.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:01 smithi040 ceph-mon[62643]: Standby manager daemon y started 2024-03-18T22:35:02.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:01 smithi040 ceph-mon[62643]: from='mgr.? 172.21.15.103:0/2207414773' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T22:35:02.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:01 smithi040 ceph-mon[62643]: from='mgr.? 172.21.15.103:0/2207414773' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T22:35:02.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:01 smithi040 ceph-mon[62643]: from='mgr.? 172.21.15.103:0/2207414773' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T22:35:02.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:01 smithi040 ceph-mon[62643]: from='mgr.? 172.21.15.103:0/2207414773' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T22:35:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:35:02.411+0000 7f99adffb700 1 -- 172.21.15.40:0/986351739 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f99a8007500 con 0x7f99b0102de0 2024-03-18T22:35:03.795 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:03 smithi040 ceph-mon[62643]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-18T22:35:03.795 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:03 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T22:35:03.795 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:03 smithi040 ceph-mon[62643]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:03.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:03 smithi113 ceph-mon[66775]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-18T22:35:03.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:03 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T22:35:03.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:03 smithi113 ceph-mon[66775]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:03.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:03 smithi103 ceph-mon[66377]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-18T22:35:03.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:03 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T22:35:03.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:03 smithi103 ceph-mon[66377]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:04.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:04 smithi040 ceph-mon[62643]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:05.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:04 smithi113 ceph-mon[66775]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:05.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:04 smithi103 ceph-mon[66377]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:07.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:06 smithi113 ceph-mon[66775]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:07.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:06 smithi103 ceph-mon[66377]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:07.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:06 smithi040 ceph-mon[62643]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:08.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T22:35:08.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:08.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:07 smithi113 ceph-mon[66775]: Deploying daemon osd.0 on smithi040 2024-03-18T22:35:08.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T22:35:08.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:08.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:07 smithi103 ceph-mon[66377]: Deploying daemon osd.0 on smithi040 2024-03-18T22:35:08.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T22:35:08.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:08.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:07 smithi040 ceph-mon[62643]: Deploying daemon osd.0 on smithi040 2024-03-18T22:35:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:08 smithi113 ceph-mon[66775]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:09.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:08 smithi103 ceph-mon[66377]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:09.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:08 smithi040 ceph-mon[62643]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:11.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:10 smithi113 ceph-mon[66775]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:11.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:10 smithi103 ceph-mon[66377]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:11.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:10 smithi040 ceph-mon[62643]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:12.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:12 smithi040 ceph-mon[62643]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:13.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:12 smithi113 ceph-mon[66775]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:13.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:12 smithi103 ceph-mon[66377]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:14.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:13 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:14.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:13 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:14.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:13 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:35:14.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:13 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:14.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:13 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:14.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:13 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:35:14.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:13 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:14.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:13 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:14.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:13 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:35:15.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:14 smithi103 ceph-mon[66377]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:15.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:14 smithi040 ceph-mon[62643]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:15.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:14 smithi113 ceph-mon[66775]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:16.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:35:16.513+0000 7f99adffb700 1 -- 172.21.15.40:0/986351739 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f99b006a740 con 0x7f99a0039200 2024-03-18T22:35:16.514 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 0 on host 'smithi040' 2024-03-18T22:35:16.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:35:16.515+0000 7f99b6ac0700 1 -- 172.21.15.40:0/986351739 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f99a0039200 msgr2=0x7f99a003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:35:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:35:16.515+0000 7f99b6ac0700 1 --2- 172.21.15.40:0/986351739 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f99a0039200 0x7f99a003b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f99a4000c00 tx=0x7f99a4005fb0 comp rx=0 tx=0).stop 2024-03-18T22:35:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:35:16.515+0000 7f99b6ac0700 1 -- 172.21.15.40:0/986351739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99b0102de0 msgr2=0x7f99b019fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:35:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:35:16.515+0000 7f99b6ac0700 1 --2- 172.21.15.40:0/986351739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99b0102de0 0x7f99b019fb10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f99a800ef60 tx=0x7f99a800ef90 comp rx=0 tx=0).stop 2024-03-18T22:35:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:35:16.515+0000 7f99b6ac0700 1 -- 172.21.15.40:0/986351739 shutdown_connections 2024-03-18T22:35:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:35:16.515+0000 7f99b6ac0700 1 --2- 172.21.15.40:0/986351739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f99b0102de0 0x7f99b019fb10 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:35:16.515+0000 7f99b6ac0700 1 --2- 172.21.15.40:0/986351739 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f99a0039200 0x7f99a003b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:35:16.515+0000 7f99b6ac0700 1 --2- 172.21.15.40:0/986351739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99b0101240 0x7f99b01985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:35:16.515+0000 7f99b6ac0700 1 --2- 172.21.15.40:0/986351739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f99b0102440 0x7f99b0198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:35:16.515+0000 7f99b6ac0700 1 -- 172.21.15.40:0/986351739 >> 172.21.15.40:0/986351739 conn(0x7f99b00fc7e0 msgr2=0x7f99b006a030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:35:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:35:16.516+0000 7f99b6ac0700 1 -- 172.21.15.40:0/986351739 shutdown_connections 2024-03-18T22:35:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:35:16.516+0000 7f99b6ac0700 1 -- 172.21.15.40:0/986351739 wait complete. 2024-03-18T22:35:16.750 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:16.751 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:16.751 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:16.751 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:35:16.751 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:16.751 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:16.751 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:16.751 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:35:16.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:16.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:16.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:16.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:35:16.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:16.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:16.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:16.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:35:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:35:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:35:17.222 DEBUG:teuthology.orchestra.run.smithi040:osd.0> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.0.service 2024-03-18T22:35:17.224 INFO:tasks.cephadm:Deploying osd.1 on smithi103 with /dev/vg_nvme/lv_4... 2024-03-18T22:35:17.224 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T22:35:17.467 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T22:35:17.774 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:17 smithi040 ceph-mon[62643]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:17.774 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:17 smithi040 ceph-mon[62643]: from='osd.0 [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T22:35:17.775 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:17 smithi040 ceph-mon[62643]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T22:35:17.830 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:17 smithi103 ceph-mon[66377]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:17.830 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:17 smithi103 ceph-mon[66377]: from='osd.0 [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T22:35:17.830 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:17 smithi103 ceph-mon[66377]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T22:35:18.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:17 smithi113 ceph-mon[66775]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:18.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:17 smithi113 ceph-mon[66775]: from='osd.0 [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T22:35:18.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:17 smithi113 ceph-mon[66775]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T22:35:18.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:18 smithi040 ceph-mon[62643]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T22:35:18.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:18 smithi040 ceph-mon[62643]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T22:35:18.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:18 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:18.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:18 smithi040 ceph-mon[62643]: from='osd.0 [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-18T22:35:18.700 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:18 smithi040 ceph-mon[62643]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-18T22:35:18.771 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:18 smithi103 ceph-mon[66377]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T22:35:18.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:18 smithi103 ceph-mon[66377]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T22:35:18.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:18 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:18.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:18 smithi103 ceph-mon[66377]: from='osd.0 [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-18T22:35:18.772 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:18 smithi103 ceph-mon[66377]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-18T22:35:19.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:18 smithi113 ceph-mon[66775]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T22:35:19.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:18 smithi113 ceph-mon[66775]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T22:35:19.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:18 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:19.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:18 smithi113 ceph-mon[66775]: from='osd.0 [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-18T22:35:19.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:18 smithi113 ceph-mon[66775]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-18T22:35:19.321 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T22:35:19.321 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T22:35:19.321 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-03-18T22:35:19.321 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-03-18T22:35:19.321 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130254 s, 805 MB/s 2024-03-18T22:35:19.321 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-03-18T22:35:19.599 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:35:19 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-0[74673]: 2024-03-18T22:35:19.331+0000 7f0332321700 -1 osd.0 0 waiting for initial osdmap 2024-03-18T22:35:19.599 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:35:19 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-0[74673]: 2024-03-18T22:35:19.335+0000 7f032990b700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T22:35:19.691 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:19 smithi103 ceph-mon[66377]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-18T22:35:19.691 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:19 smithi103 ceph-mon[66377]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T22:35:19.691 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:19 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:19.691 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:19 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:19.691 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:19 smithi103 ceph-mon[66377]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:19.692 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:19 smithi103 ceph-mon[66377]: from='osd.0 ' entity='osd.0' 2024-03-18T22:35:19.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:19 smithi040 ceph-mon[62643]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-18T22:35:19.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:19 smithi040 ceph-mon[62643]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T22:35:19.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:19 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:19.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:19 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:19.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:19 smithi040 ceph-mon[62643]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:19.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:19 smithi040 ceph-mon[62643]: from='osd.0 ' entity='osd.0' 2024-03-18T22:35:19.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi103:vg_nvme/lv_4 2024-03-18T22:35:20.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:19 smithi113 ceph-mon[66775]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-18T22:35:20.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:19 smithi113 ceph-mon[66775]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T22:35:20.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:19 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:20.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:19 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:20.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:19 smithi113 ceph-mon[66775]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T22:35:20.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:19 smithi113 ceph-mon[66775]: from='osd.0 ' entity='osd.0' 2024-03-18T22:35:20.210 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T22:35:20.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:20 smithi103 ceph-mon[66377]: purged_snaps scrub starts 2024-03-18T22:35:20.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:20 smithi103 ceph-mon[66377]: purged_snaps scrub ok 2024-03-18T22:35:20.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:20.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:20.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:20.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:35:20.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:20.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:20.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:35:20.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:20.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:20 smithi103 ceph-mon[66377]: osd.0 [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248] boot 2024-03-18T22:35:20.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:20 smithi103 ceph-mon[66377]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T22:35:20.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:20.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:20 smithi040 ceph-mon[62643]: purged_snaps scrub starts 2024-03-18T22:35:20.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:20 smithi040 ceph-mon[62643]: purged_snaps scrub ok 2024-03-18T22:35:20.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:20.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:20.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:20.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:35:20.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:20.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:20.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:35:20.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:20.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:20 smithi040 ceph-mon[62643]: osd.0 [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248] boot 2024-03-18T22:35:20.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:20 smithi040 ceph-mon[62643]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T22:35:20.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:21.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:20 smithi113 ceph-mon[66775]: purged_snaps scrub starts 2024-03-18T22:35:21.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:20 smithi113 ceph-mon[66775]: purged_snaps scrub ok 2024-03-18T22:35:21.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:21.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:21.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:21.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:35:21.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:21.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:21.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:35:21.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:21.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:20 smithi113 ceph-mon[66775]: osd.0 [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248] boot 2024-03-18T22:35:21.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:20 smithi113 ceph-mon[66775]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T22:35:21.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T22:35:21.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.713+0000 7f50326c8700 1 -- 172.21.15.103:0/3012223399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f502c102360 msgr2=0x7f502c102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:35:21.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.713+0000 7f50326c8700 1 --2- 172.21.15.103:0/3012223399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f502c102360 0x7f502c102780 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f501c0099c0 tx=0x7f501c009cd0 comp rx=0 tx=0).stop 2024-03-18T22:35:21.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.713+0000 7f50326c8700 1 -- 172.21.15.103:0/3012223399 shutdown_connections 2024-03-18T22:35:21.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.713+0000 7f50326c8700 1 --2- 172.21.15.103:0/3012223399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f502c103f00 0x7f502c10a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:21.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.713+0000 7f50326c8700 1 --2- 172.21.15.103:0/3012223399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f502c102360 0x7f502c102780 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:21.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.713+0000 7f50326c8700 1 --2- 172.21.15.103:0/3012223399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f502c103560 0x7f502c1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:21.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.713+0000 7f50326c8700 1 -- 172.21.15.103:0/3012223399 >> 172.21.15.103:0/3012223399 conn(0x7f502c0fd920 msgr2=0x7f502c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:35:21.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.713+0000 7f50326c8700 1 -- 172.21.15.103:0/3012223399 shutdown_connections 2024-03-18T22:35:21.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.713+0000 7f50326c8700 1 -- 172.21.15.103:0/3012223399 wait complete. 2024-03-18T22:35:21.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.713+0000 7f50326c8700 1 Processor -- start 2024-03-18T22:35:21.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.713+0000 7f50326c8700 1 -- start start 2024-03-18T22:35:21.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f50326c8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f502c102360 0x7f502c0760b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:35:21.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f50326c8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f502c103560 0x7f502c074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:35:21.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f50326c8700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f502c103f00 0x7f502c074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:35:21.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f50326c8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f502c10de00 con 0x7f502c102360 2024-03-18T22:35:21.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f50326c8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f502c10dc80 con 0x7f502c103560 2024-03-18T22:35:21.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f50326c8700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f502c10df80 con 0x7f502c103f00 2024-03-18T22:35:21.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f502b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f502c103560 0x7f502c074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:35:21.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f502b7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f502c103560 0x7f502c074700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33644/0 (socket says 172.21.15.103:33644) 2024-03-18T22:35:21.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f502b7fe700 1 -- 172.21.15.103:0/1413038159 learned_addr learned my addr 172.21.15.103:0/1413038159 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-18T22:35:21.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f502bfff700 1 --2- 172.21.15.103:0/1413038159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f502c102360 0x7f502c0760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:35:21.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f5030c65700 1 --2- 172.21.15.103:0/1413038159 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f502c103f00 0x7f502c074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:35:21.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f502b7fe700 1 -- 172.21.15.103:0/1413038159 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f502c103f00 msgr2=0x7f502c074c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:35:21.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f502b7fe700 1 --2- 172.21.15.103:0/1413038159 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f502c103f00 0x7f502c074c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:21.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f502b7fe700 1 -- 172.21.15.103:0/1413038159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f502c102360 msgr2=0x7f502c0760b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:35:21.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f502b7fe700 1 --2- 172.21.15.103:0/1413038159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f502c102360 0x7f502c0760b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:21.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f502b7fe700 1 -- 172.21.15.103:0/1413038159 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f502c0754f0 con 0x7f502c103560 2024-03-18T22:35:21.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f502b7fe700 1 --2- 172.21.15.103:0/1413038159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f502c103560 0x7f502c074700 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f502000d980 tx=0x7f502000dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:35:21.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f50297fa700 1 -- 172.21.15.103:0/1413038159 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5020016070 con 0x7f502c103560 2024-03-18T22:35:21.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f50297fa700 1 -- 172.21.15.103:0/1413038159 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f50200044d0 con 0x7f502c103560 2024-03-18T22:35:21.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f50326c8700 1 -- 172.21.15.103:0/1413038159 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f502c1a2630 con 0x7f502c103560 2024-03-18T22:35:21.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.714+0000 7f50297fa700 1 -- 172.21.15.103:0/1413038159 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5020004cf0 con 0x7f502c103560 2024-03-18T22:35:21.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.715+0000 7f50326c8700 1 -- 172.21.15.103:0/1413038159 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f502c1a2b70 con 0x7f502c103560 2024-03-18T22:35:21.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.715+0000 7f50326c8700 1 -- 172.21.15.103:0/1413038159 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f502c068fd0 con 0x7f502c103560 2024-03-18T22:35:21.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.717+0000 7f50297fa700 1 -- 172.21.15.103:0/1413038159 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f5020004020 con 0x7f502c103560 2024-03-18T22:35:21.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.717+0000 7f50297fa700 1 --2- 172.21.15.103:0/1413038159 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f50140a2880 0x7f50140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:35:21.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.717+0000 7f50297fa700 1 -- 172.21.15.103:0/1413038159 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f50200cd370 con 0x7f502c103560 2024-03-18T22:35:21.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.717+0000 7f502bfff700 1 --2- 172.21.15.103:0/1413038159 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f50140a2880 0x7f50140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:35:21.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.718+0000 7f502bfff700 1 --2- 172.21.15.103:0/1413038159 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f50140a2880 0x7f50140a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f501c00b580 tx=0x7f501c005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:35:21.721 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.721+0000 7f50297fa700 1 -- 172.21.15.103:0/1413038159 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f502001f020 con 0x7f502c103560 2024-03-18T22:35:21.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:21 smithi103 ceph-mon[66377]: Detected new or changed devices on smithi040 2024-03-18T22:35:21.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:21 smithi103 ceph-mon[66377]: Adjusting osd_memory_target on smithi040 to 17093M 2024-03-18T22:35:21.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:21 smithi103 ceph-mon[66377]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:21.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:21 smithi040 ceph-mon[62643]: Detected new or changed devices on smithi040 2024-03-18T22:35:21.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:21 smithi040 ceph-mon[62643]: Adjusting osd_memory_target on smithi040 to 17093M 2024-03-18T22:35:21.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:21 smithi040 ceph-mon[62643]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:21.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:21.929+0000 7f50326c8700 1 -- 172.21.15.103:0/1413038159 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f502c062750 con 0x7f50140a2880 2024-03-18T22:35:22.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:21 smithi113 ceph-mon[66775]: Detected new or changed devices on smithi040 2024-03-18T22:35:22.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:21 smithi113 ceph-mon[66775]: Adjusting osd_memory_target on smithi040 to 17093M 2024-03-18T22:35:22.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:21 smithi113 ceph-mon[66775]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:22.846 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:22 smithi103 ceph-mon[66377]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T22:35:22.846 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:22 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:35:22.846 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:22 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:35:22.846 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:22 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:22.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:22 smithi040 ceph-mon[62643]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T22:35:22.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:22 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:35:22.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:22 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:35:22.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:22 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:22.981 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:22 smithi113 ceph-mon[66775]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T22:35:22.981 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:22 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:35:22.981 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:22 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:35:22.982 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:22 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:23.727 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:23 smithi103 ceph-mon[66377]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:35:23.727 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:23 smithi103 ceph-mon[66377]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:23.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:23 smithi040 ceph-mon[62643]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:35:23.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:23 smithi040 ceph-mon[62643]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:24.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:23 smithi113 ceph-mon[66775]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:35:24.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:23 smithi113 ceph-mon[66775]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:24.871 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:24 smithi103 ceph-mon[66377]: from='client.? 172.21.15.103:0/879393176' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9"}]: dispatch 2024-03-18T22:35:24.871 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:24 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9"}]: dispatch 2024-03-18T22:35:24.871 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:24 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9"}]': finished 2024-03-18T22:35:24.871 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:24 smithi103 ceph-mon[66377]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T22:35:24.871 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:24.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:24 smithi040 ceph-mon[62643]: from='client.? 172.21.15.103:0/879393176' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9"}]: dispatch 2024-03-18T22:35:24.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:24 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9"}]: dispatch 2024-03-18T22:35:24.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:24 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9"}]': finished 2024-03-18T22:35:24.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:24 smithi040 ceph-mon[62643]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T22:35:24.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:25.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:24 smithi113 ceph-mon[66775]: from='client.? 172.21.15.103:0/879393176' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9"}]: dispatch 2024-03-18T22:35:25.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:24 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9"}]: dispatch 2024-03-18T22:35:25.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:24 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9"}]': finished 2024-03-18T22:35:25.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:24 smithi113 ceph-mon[66775]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T22:35:25.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:25.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:25 smithi040 ceph-mon[62643]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:25.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:25 smithi040 ceph-mon[62643]: from='client.? 172.21.15.103:0/947569186' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:35:26.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:25 smithi113 ceph-mon[66775]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:26.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:25 smithi113 ceph-mon[66775]: from='client.? 172.21.15.103:0/947569186' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:35:26.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:25 smithi103 ceph-mon[66377]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:26.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:25 smithi103 ceph-mon[66377]: from='client.? 172.21.15.103:0/947569186' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:35:27.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:26 smithi113 ceph-mon[66775]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:27.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:26 smithi103 ceph-mon[66377]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:27.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:26 smithi040 ceph-mon[62643]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:29.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:28 smithi113 ceph-mon[66775]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:29.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:28 smithi103 ceph-mon[66377]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:29.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:28 smithi040 ceph-mon[62643]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:30.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:30 smithi103 ceph-mon[66377]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:31.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:30 smithi113 ceph-mon[66775]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:31.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:30 smithi040 ceph-mon[62643]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:31 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T22:35:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:31 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:31 smithi113 ceph-mon[66775]: Deploying daemon osd.1 on smithi103 2024-03-18T22:35:32.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:31 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T22:35:32.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:31 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:32.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:31 smithi103 ceph-mon[66377]: Deploying daemon osd.1 on smithi103 2024-03-18T22:35:32.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:31 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T22:35:32.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:31 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:32.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:31 smithi040 ceph-mon[62643]: Deploying daemon osd.1 on smithi103 2024-03-18T22:35:33.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:32 smithi103 ceph-mon[66377]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:33.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:32 smithi040 ceph-mon[62643]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:33.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:32 smithi113 ceph-mon[66775]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:35.076 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:34 smithi103 ceph-mon[66377]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:35.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:34 smithi113 ceph-mon[66775]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:35.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:34 smithi040 ceph-mon[62643]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:38.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:37 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:38.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:37 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:38.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:37 smithi113 ceph-mon[66775]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:38.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:37 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:35:38.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:37 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:38.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:37 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:38.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:37 smithi103 ceph-mon[66377]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:38.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:37 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:35:38.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:37 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:38.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:37 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:38.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:37 smithi040 ceph-mon[62643]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:38.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:37 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:35:39.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:38 smithi113 ceph-mon[66775]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:39.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:38 smithi103 ceph-mon[66377]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:39.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:38 smithi040 ceph-mon[62643]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:40.244 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 1 on host 'smithi103' 2024-03-18T22:35:40.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:40.242+0000 7f50297fa700 1 -- 172.21.15.103:0/1413038159 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f502c062750 con 0x7f50140a2880 2024-03-18T22:35:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:40.244+0000 7f50326c8700 1 -- 172.21.15.103:0/1413038159 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f50140a2880 msgr2=0x7f50140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:35:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:40.244+0000 7f50326c8700 1 --2- 172.21.15.103:0/1413038159 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f50140a2880 0x7f50140a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f501c00b580 tx=0x7f501c005fb0 comp rx=0 tx=0).stop 2024-03-18T22:35:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:40.244+0000 7f50326c8700 1 -- 172.21.15.103:0/1413038159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f502c103560 msgr2=0x7f502c074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:35:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:40.244+0000 7f50326c8700 1 --2- 172.21.15.103:0/1413038159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f502c103560 0x7f502c074700 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f502000d980 tx=0x7f502000dd40 comp rx=0 tx=0).stop 2024-03-18T22:35:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:40.244+0000 7f50326c8700 1 -- 172.21.15.103:0/1413038159 shutdown_connections 2024-03-18T22:35:40.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:40.244+0000 7f50326c8700 1 --2- 172.21.15.103:0/1413038159 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f502c103f00 0x7f502c074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:40.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:40.244+0000 7f50326c8700 1 --2- 172.21.15.103:0/1413038159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f502c102360 0x7f502c0760b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:40.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:40.244+0000 7f50326c8700 1 --2- 172.21.15.103:0/1413038159 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f50140a2880 0x7f50140a4d40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:40.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:40.245+0000 7f50326c8700 1 --2- 172.21.15.103:0/1413038159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f502c103560 0x7f502c074700 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:40.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:40.245+0000 7f50326c8700 1 -- 172.21.15.103:0/1413038159 >> 172.21.15.103:0/1413038159 conn(0x7f502c0fd920 msgr2=0x7f502c106980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:35:40.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:40.245+0000 7f50326c8700 1 -- 172.21.15.103:0/1413038159 shutdown_connections 2024-03-18T22:35:40.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:35:40.245+0000 7f50326c8700 1 -- 172.21.15.103:0/1413038159 wait complete. 2024-03-18T22:35:40.596 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:40.596 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:40.596 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:40.596 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:35:40.597 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:40.597 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:40.597 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:35:40.597 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:40.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:40.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:40.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:40.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:35:40.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:40.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:40.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:35:40.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:40.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:40.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:40.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:40.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:35:40.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:40.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:40.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:35:40.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:40.932 DEBUG:teuthology.orchestra.run.smithi103:osd.1> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.1.service 2024-03-18T22:35:40.934 INFO:tasks.cephadm:Deploying osd.2 on smithi113 with /dev/vg_nvme/lv_4... 2024-03-18T22:35:40.934 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T22:35:41.178 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:35:41.550 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:41 smithi103 ceph-mon[66377]: from='osd.1 [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T22:35:41.550 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:41 smithi103 ceph-mon[66377]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T22:35:41.550 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:41 smithi103 ceph-mon[66377]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:41.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:41 smithi113 ceph-mon[66775]: from='osd.1 [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T22:35:41.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:41 smithi113 ceph-mon[66775]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T22:35:41.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:41 smithi113 ceph-mon[66775]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:41.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:41 smithi040 ceph-mon[62643]: from='osd.1 [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T22:35:41.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:41 smithi040 ceph-mon[62643]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T22:35:41.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:41 smithi040 ceph-mon[62643]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:42.524 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:42 smithi103 ceph-mon[66377]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T22:35:42.524 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:42 smithi103 ceph-mon[66377]: from='osd.1 [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:35:42.524 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:42 smithi103 ceph-mon[66377]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T22:35:42.524 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:42 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:42.524 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:42 smithi103 ceph-mon[66377]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:35:42.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:42 smithi113 ceph-mon[66775]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T22:35:42.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:42 smithi113 ceph-mon[66775]: from='osd.1 [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:35:42.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:42 smithi113 ceph-mon[66775]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T22:35:42.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:42 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:42.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:42 smithi113 ceph-mon[66775]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:35:42.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:42 smithi040 ceph-mon[62643]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T22:35:42.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:42 smithi040 ceph-mon[62643]: from='osd.1 [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:35:42.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:42 smithi040 ceph-mon[62643]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T22:35:42.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:42.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:42 smithi040 ceph-mon[62643]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:35:42.940 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T22:35:42.941 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T22:35:42.941 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-03-18T22:35:42.941 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-03-18T22:35:42.941 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00968586 s, 1.1 GB/s 2024-03-18T22:35:42.941 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-03-18T22:35:43.359 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:35:43 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-1[70958]: 2024-03-18T22:35:43.055+0000 7f50a8a95700 -1 osd.1 0 waiting for initial osdmap 2024-03-18T22:35:43.360 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:35:43 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-1[70958]: 2024-03-18T22:35:43.059+0000 7f50a0880700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T22:35:43.513 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi113:vg_nvme/lv_4 2024-03-18T22:35:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:43 smithi113 ceph-mon[66775]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-18T22:35:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:43 smithi113 ceph-mon[66775]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T22:35:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:43 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:43 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:43 smithi113 ceph-mon[66775]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:43 smithi113 ceph-mon[66775]: from='osd.1 ' entity='osd.1' 2024-03-18T22:35:43.702 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:43 smithi103 ceph-mon[66377]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-18T22:35:43.702 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:43 smithi103 ceph-mon[66377]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T22:35:43.702 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:43.703 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:43.703 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:43 smithi103 ceph-mon[66377]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:43.703 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:43 smithi103 ceph-mon[66377]: from='osd.1 ' entity='osd.1' 2024-03-18T22:35:43.753 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:35:43.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:43 smithi040 ceph-mon[62643]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-18T22:35:43.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:43 smithi040 ceph-mon[62643]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T22:35:43.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:43 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:43.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:43 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:43.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:43 smithi040 ceph-mon[62643]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-18T22:35:43.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:43 smithi040 ceph-mon[62643]: from='osd.1 ' entity='osd.1' 2024-03-18T22:35:44.727 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:44 smithi113 ceph-mon[66775]: purged_snaps scrub starts 2024-03-18T22:35:44.727 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:44 smithi113 ceph-mon[66775]: purged_snaps scrub ok 2024-03-18T22:35:44.727 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:44.727 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:44 smithi113 ceph-mon[66775]: osd.1 [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755] boot 2024-03-18T22:35:44.727 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:44 smithi113 ceph-mon[66775]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T22:35:44.727 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:44.727 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:44.727 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:44.727 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:35:44.727 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:44.728 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:44.728 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:35:44.728 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:44.773 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:44 smithi040 ceph-mon[62643]: purged_snaps scrub starts 2024-03-18T22:35:44.773 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:44 smithi040 ceph-mon[62643]: purged_snaps scrub ok 2024-03-18T22:35:44.774 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:44.774 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:44 smithi040 ceph-mon[62643]: osd.1 [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755] boot 2024-03-18T22:35:44.774 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:44 smithi040 ceph-mon[62643]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T22:35:44.774 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:44.774 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:44.774 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:44.774 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:35:44.774 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:44.774 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:44.774 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:35:44.774 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:44.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:44 smithi103 ceph-mon[66377]: purged_snaps scrub starts 2024-03-18T22:35:44.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:44 smithi103 ceph-mon[66377]: purged_snaps scrub ok 2024-03-18T22:35:44.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:44.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:44 smithi103 ceph-mon[66377]: osd.1 [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755] boot 2024-03-18T22:35:44.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:44 smithi103 ceph-mon[66377]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T22:35:44.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T22:35:44.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:44.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:44.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:35:44.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:44.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:44.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:35:44.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:35:45.234 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.231+0000 7fdaaaa6f700 1 -- 172.21.15.113:0/797727122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdaa4101be0 msgr2=0x7fdaa4102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:35:45.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.231+0000 7fdaaaa6f700 1 --2- 172.21.15.113:0/797727122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdaa4101be0 0x7fdaa4102040 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fda90009990 tx=0x7fda90009ca0 comp rx=0 tx=0).stop 2024-03-18T22:35:45.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.231+0000 7fdaaaa6f700 1 -- 172.21.15.113:0/797727122 shutdown_connections 2024-03-18T22:35:45.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.231+0000 7fdaaaa6f700 1 --2- 172.21.15.113:0/797727122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4102580 0x7fdaa4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:45.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.231+0000 7fdaaaa6f700 1 --2- 172.21.15.113:0/797727122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaa41009e0 0x7fdaa4100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:45.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.231+0000 7fdaaaa6f700 1 --2- 172.21.15.113:0/797727122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdaa4101be0 0x7fdaa4102040 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:45.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.231+0000 7fdaaaa6f700 1 -- 172.21.15.113:0/797727122 >> 172.21.15.113:0/797727122 conn(0x7fdaa40fbf60 msgr2=0x7fdaa40fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:35:45.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.231+0000 7fdaaaa6f700 1 -- 172.21.15.113:0/797727122 shutdown_connections 2024-03-18T22:35:45.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.231+0000 7fdaaaa6f700 1 -- 172.21.15.113:0/797727122 wait complete. 2024-03-18T22:35:45.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.232+0000 7fdaaaa6f700 1 Processor -- start 2024-03-18T22:35:45.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.232+0000 7fdaaaa6f700 1 -- start start 2024-03-18T22:35:45.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.232+0000 7fdaaaa6f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaa41009e0 0x7fdaa4198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:35:45.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.232+0000 7fdaaaa6f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4101be0 0x7fdaa41989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:35:45.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.232+0000 7fdaaaa6f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdaa4102580 0x7fdaa419fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:35:45.235 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.232+0000 7fdaaaa6f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa410deb0 con 0x7fdaa41009e0 2024-03-18T22:35:45.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.232+0000 7fdaaaa6f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa410dd30 con 0x7fdaa4101be0 2024-03-18T22:35:45.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.232+0000 7fdaaaa6f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa410e030 con 0x7fdaa4102580 2024-03-18T22:35:45.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.232+0000 7fdaa900c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdaa4102580 0x7fdaa419fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:35:45.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.232+0000 7fdaa900c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdaa4102580 0x7fdaa419fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:57536/0 (socket says 172.21.15.113:57536) 2024-03-18T22:35:45.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.232+0000 7fdaa900c700 1 -- 172.21.15.113:0/471710656 learned_addr learned my addr 172.21.15.113:0/471710656 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-18T22:35:45.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.232+0000 7fdaa880b700 1 --2- 172.21.15.113:0/471710656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaa41009e0 0x7fdaa4198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:35:45.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.232+0000 7fdaa3fff700 1 --2- 172.21.15.113:0/471710656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4101be0 0x7fdaa41989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:35:45.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.233+0000 7fdaa900c700 1 -- 172.21.15.113:0/471710656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4101be0 msgr2=0x7fdaa41989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:35:45.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.233+0000 7fdaa900c700 1 --2- 172.21.15.113:0/471710656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4101be0 0x7fdaa41989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:45.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.233+0000 7fdaa900c700 1 -- 172.21.15.113:0/471710656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaa41009e0 msgr2=0x7fdaa4198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:35:45.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.233+0000 7fdaa900c700 1 --2- 172.21.15.113:0/471710656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaa41009e0 0x7fdaa4198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:35:45.236 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.233+0000 7fdaa900c700 1 -- 172.21.15.113:0/471710656 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa41a00c0 con 0x7fdaa4102580 2024-03-18T22:35:45.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.233+0000 7fdaa900c700 1 --2- 172.21.15.113:0/471710656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdaa4102580 0x7fdaa419fa20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00add0 tx=0x7fda9c00c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:35:45.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.233+0000 7fdaa1ffb700 1 -- 172.21.15.113:0/471710656 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c017020 con 0x7fdaa4102580 2024-03-18T22:35:45.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.233+0000 7fdaa1ffb700 1 -- 172.21.15.113:0/471710656 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fda9c00ae00 con 0x7fdaa4102580 2024-03-18T22:35:45.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.233+0000 7fdaaaa6f700 1 -- 172.21.15.113:0/471710656 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa41a03b0 con 0x7fdaa4102580 2024-03-18T22:35:45.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.233+0000 7fdaa1ffb700 1 -- 172.21.15.113:0/471710656 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c005280 con 0x7fdaa4102580 2024-03-18T22:35:45.237 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.233+0000 7fdaaaa6f700 1 -- 172.21.15.113:0/471710656 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa41a0940 con 0x7fdaa4102580 2024-03-18T22:35:45.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.234+0000 7fdaaaa6f700 1 -- 172.21.15.113:0/471710656 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaa4068fd0 con 0x7fdaa4102580 2024-03-18T22:35:45.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.235+0000 7fdaa1ffb700 1 -- 172.21.15.113:0/471710656 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7fda9c00c890 con 0x7fdaa4102580 2024-03-18T22:35:45.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.236+0000 7fdaa1ffb700 1 --2- 172.21.15.113:0/471710656 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fda940a2880 0x7fda940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:35:45.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.236+0000 7fdaa1ffb700 1 -- 172.21.15.113:0/471710656 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7fda9c0cc860 con 0x7fdaa4102580 2024-03-18T22:35:45.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.236+0000 7fdaa880b700 1 --2- 172.21.15.113:0/471710656 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fda940a2880 0x7fda940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:35:45.238 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.236+0000 7fdaa880b700 1 --2- 172.21.15.113:0/471710656 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fda940a2880 0x7fda940a4d40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4101a40 tx=0x7fda9800a3b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:35:45.241 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.240+0000 7fdaa1ffb700 1 -- 172.21.15.113:0/471710656 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fda9c021020 con 0x7fdaa4102580 2024-03-18T22:35:45.445 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:35:45.445+0000 7fdaaaa6f700 1 -- 172.21.15.113:0/471710656 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fdaa4062750 con 0x7fda940a2880 2024-03-18T22:35:45.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:45 smithi040 ceph-mon[62643]: Detected new or changed devices on smithi103 2024-03-18T22:35:45.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:45 smithi040 ceph-mon[62643]: Adjusting osd_memory_target on smithi103 to 17088M 2024-03-18T22:35:45.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:45 smithi040 ceph-mon[62643]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:45.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:45 smithi113 ceph-mon[66775]: Detected new or changed devices on smithi103 2024-03-18T22:35:45.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:45 smithi113 ceph-mon[66775]: Adjusting osd_memory_target on smithi103 to 17088M 2024-03-18T22:35:45.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:45 smithi113 ceph-mon[66775]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:45.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:45 smithi103 ceph-mon[66377]: Detected new or changed devices on smithi103 2024-03-18T22:35:45.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:45 smithi103 ceph-mon[66377]: Adjusting osd_memory_target on smithi103 to 17088M 2024-03-18T22:35:45.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:45 smithi103 ceph-mon[66377]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:46.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:46 smithi113 ceph-mon[66775]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T22:35:46.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:46 smithi113 ceph-mon[66775]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:35:46.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:46 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:35:46.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:46 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:35:46.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:46 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:46.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:46 smithi103 ceph-mon[66377]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T22:35:46.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:46 smithi103 ceph-mon[66377]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:35:46.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:46 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:35:46.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:46 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:35:46.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:46 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:46.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:46 smithi040 ceph-mon[62643]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T22:35:46.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:46 smithi040 ceph-mon[62643]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:35:46.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:46 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:35:46.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:46 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:35:46.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:46 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:47.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:47 smithi113 ceph-mon[66775]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:47.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:47 smithi103 ceph-mon[66377]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:47.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:47 smithi040 ceph-mon[62643]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:48 smithi113 ceph-mon[66775]: from='client.? 172.21.15.113:0/3942071213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69ba7362-43de-46f4-a5fa-b8492210fa27"}]: dispatch 2024-03-18T22:35:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:48 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69ba7362-43de-46f4-a5fa-b8492210fa27"}]: dispatch 2024-03-18T22:35:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:48 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "69ba7362-43de-46f4-a5fa-b8492210fa27"}]': finished 2024-03-18T22:35:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:48 smithi113 ceph-mon[66775]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T22:35:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T22:35:48.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:48 smithi103 ceph-mon[66377]: from='client.? 172.21.15.113:0/3942071213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69ba7362-43de-46f4-a5fa-b8492210fa27"}]: dispatch 2024-03-18T22:35:48.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:48 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69ba7362-43de-46f4-a5fa-b8492210fa27"}]: dispatch 2024-03-18T22:35:48.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:48 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "69ba7362-43de-46f4-a5fa-b8492210fa27"}]': finished 2024-03-18T22:35:48.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:48 smithi103 ceph-mon[66377]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T22:35:48.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T22:35:48.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:48 smithi040 ceph-mon[62643]: from='client.? 172.21.15.113:0/3942071213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69ba7362-43de-46f4-a5fa-b8492210fa27"}]: dispatch 2024-03-18T22:35:48.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:48 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69ba7362-43de-46f4-a5fa-b8492210fa27"}]: dispatch 2024-03-18T22:35:48.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:48 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "69ba7362-43de-46f4-a5fa-b8492210fa27"}]': finished 2024-03-18T22:35:48.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:48 smithi040 ceph-mon[62643]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T22:35:48.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T22:35:49.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:49 smithi113 ceph-mon[66775]: from='client.? 172.21.15.113:0/3020264194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:35:49.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:49 smithi113 ceph-mon[66775]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:49.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:49 smithi103 ceph-mon[66377]: from='client.? 172.21.15.113:0/3020264194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:35:49.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:49 smithi103 ceph-mon[66377]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:49.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:49 smithi040 ceph-mon[62643]: from='client.? 172.21.15.113:0/3020264194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:35:49.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:49 smithi040 ceph-mon[62643]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:51.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:50 smithi113 ceph-mon[66775]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:51.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:50 smithi103 ceph-mon[66377]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:51.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:50 smithi040 ceph-mon[62643]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:53.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:52 smithi113 ceph-mon[66775]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:53.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:52 smithi103 ceph-mon[66377]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:53.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:52 smithi040 ceph-mon[62643]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:55.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:54 smithi113 ceph-mon[66775]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:55.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:54 smithi103 ceph-mon[66377]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:55.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:54 smithi040 ceph-mon[62643]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:55.946 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T22:35:55.946 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:55.946 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:55 smithi113 ceph-mon[66775]: Deploying daemon osd.2 on smithi113 2024-03-18T22:35:56.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T22:35:56.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:56.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:55 smithi103 ceph-mon[66377]: Deploying daemon osd.2 on smithi113 2024-03-18T22:35:56.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T22:35:56.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:35:56.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:55 smithi040 ceph-mon[62643]: Deploying daemon osd.2 on smithi113 2024-03-18T22:35:56.943 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:56 smithi113 ceph-mon[66775]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:57.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:56 smithi103 ceph-mon[66377]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:57.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:56 smithi040 ceph-mon[62643]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:59.022 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:35:58 smithi113 ceph-mon[66775]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:59.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:35:58 smithi103 ceph-mon[66377]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:35:59.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:35:58 smithi040 ceph-mon[62643]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:36:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:01 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:01 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:01 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:01 smithi113 ceph-mon[66775]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:36:01.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:01 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:01.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:01 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:01.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:01 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:01.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:01 smithi103 ceph-mon[66377]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:36:01.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:01 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:01.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:01 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:01.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:01 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:01.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:01 smithi040 ceph-mon[62643]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:36:03.566 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:03.565+0000 7fdaa1ffb700 1 -- 172.21.15.113:0/471710656 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdaa4062750 con 0x7fda940a2880 2024-03-18T22:36:03.566 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 2 on host 'smithi113' 2024-03-18T22:36:03.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:03.567+0000 7fdaaaa6f700 1 -- 172.21.15.113:0/471710656 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fda940a2880 msgr2=0x7fda940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:36:03.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:03.567+0000 7fdaaaa6f700 1 --2- 172.21.15.113:0/471710656 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fda940a2880 0x7fda940a4d40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4101a40 tx=0x7fda9800a3b0 comp rx=0 tx=0).stop 2024-03-18T22:36:03.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:03.567+0000 7fdaaaa6f700 1 -- 172.21.15.113:0/471710656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdaa4102580 msgr2=0x7fdaa419fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:36:03.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:03.567+0000 7fdaaaa6f700 1 --2- 172.21.15.113:0/471710656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdaa4102580 0x7fdaa419fa20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00add0 tx=0x7fda9c00c330 comp rx=0 tx=0).stop 2024-03-18T22:36:03.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:03.567+0000 7fdaaaa6f700 1 -- 172.21.15.113:0/471710656 shutdown_connections 2024-03-18T22:36:03.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:03.567+0000 7fdaaaa6f700 1 --2- 172.21.15.113:0/471710656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdaa4102580 0x7fdaa419fa20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:03.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:03.567+0000 7fdaaaa6f700 1 --2- 172.21.15.113:0/471710656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdaa41009e0 0x7fdaa4198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:03.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:03.567+0000 7fdaaaa6f700 1 --2- 172.21.15.113:0/471710656 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fda940a2880 0x7fda940a4d40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:03.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:03.567+0000 7fdaaaa6f700 1 --2- 172.21.15.113:0/471710656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdaa4101be0 0x7fdaa41989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:03.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:03.567+0000 7fdaaaa6f700 1 -- 172.21.15.113:0/471710656 >> 172.21.15.113:0/471710656 conn(0x7fdaa40fbf60 msgr2=0x7fdaa4104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:36:03.568 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:03.567+0000 7fdaaaa6f700 1 -- 172.21.15.113:0/471710656 shutdown_connections 2024-03-18T22:36:03.569 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:03.567+0000 7fdaaaa6f700 1 -- 172.21.15.113:0/471710656 wait complete. 2024-03-18T22:36:04.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:03 smithi113 ceph-mon[66775]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:36:04.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:03 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:04.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:03 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:04.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:03 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:04.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:03 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:04.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:03 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:04.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:03 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:04.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:03 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:04.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:03 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:04.097 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:03 smithi040 ceph-mon[62643]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:36:04.097 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:03 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:04.097 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:03 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:04.097 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:03 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:04.097 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:03 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:04.097 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:03 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:04.097 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:03 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:04.097 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:03 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:04.097 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:03 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:04.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:03 smithi103 ceph-mon[66377]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:36:04.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:03 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:04.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:03 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:04.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:03 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:04.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:03 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:04.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:03 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:04.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:03 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:04.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:03 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:04.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:03 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:04.177 DEBUG:teuthology.orchestra.run.smithi113:osd.2> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.2.service 2024-03-18T22:36:04.180 INFO:tasks.cephadm:Deploying osd.3 on smithi040 with /dev/vg_nvme/lv_3... 2024-03-18T22:36:04.180 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T22:36:04.431 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:36:04.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:04 smithi040 ceph-mon[62643]: from='osd.2 [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T22:36:04.876 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:04 smithi040 ceph-mon[62643]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T22:36:05.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:04 smithi113 ceph-mon[66775]: from='osd.2 [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T22:36:05.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:04 smithi113 ceph-mon[66775]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T22:36:05.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:04 smithi103 ceph-mon[66377]: from='osd.2 [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T22:36:05.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:04 smithi103 ceph-mon[66377]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T22:36:05.954 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:05 smithi040 ceph-mon[62643]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:36:05.954 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:05 smithi040 ceph-mon[62643]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T22:36:05.954 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:05 smithi040 ceph-mon[62643]: from='osd.2 [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:36:05.954 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:05 smithi040 ceph-mon[62643]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T22:36:05.954 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:05 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T22:36:05.954 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:05 smithi040 ceph-mon[62643]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:36:06.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:05 smithi113 ceph-mon[66775]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:36:06.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:05 smithi113 ceph-mon[66775]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T22:36:06.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:05 smithi113 ceph-mon[66775]: from='osd.2 [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:36:06.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:05 smithi113 ceph-mon[66775]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T22:36:06.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:05 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T22:36:06.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:05 smithi113 ceph-mon[66775]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:36:06.080 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 22:36:05 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-2[71925]: 2024-03-18T22:36:05.806+0000 7f15b43b7700 -1 osd.2 0 waiting for initial osdmap 2024-03-18T22:36:06.081 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 22:36:05 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-2[71925]: 2024-03-18T22:36:05.809+0000 7f15b01aa700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T22:36:06.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:05 smithi103 ceph-mon[66377]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:36:06.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:05 smithi103 ceph-mon[66377]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T22:36:06.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:05 smithi103 ceph-mon[66377]: from='osd.2 [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:36:06.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:05 smithi103 ceph-mon[66377]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T22:36:06.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:05 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T22:36:06.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:05 smithi103 ceph-mon[66377]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:36:06.538 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T22:36:06.538 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T22:36:06.538 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-03-18T22:36:06.538 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-03-18T22:36:06.538 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111142 s, 943 MB/s 2024-03-18T22:36:06.538 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-03-18T22:36:07.041 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:06 smithi113 ceph-mon[66775]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-18T22:36:07.041 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:06 smithi113 ceph-mon[66775]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T22:36:07.041 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:06 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T22:36:07.041 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:06 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T22:36:07.041 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:06 smithi113 ceph-mon[66775]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:36:07.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:06 smithi103 ceph-mon[66377]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-18T22:36:07.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:06 smithi103 ceph-mon[66377]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T22:36:07.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:06 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T22:36:07.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:06 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T22:36:07.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:06 smithi103 ceph-mon[66377]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:36:07.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:06 smithi040 ceph-mon[62643]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-18T22:36:07.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:06 smithi040 ceph-mon[62643]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T22:36:07.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:06 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T22:36:07.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:06 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T22:36:07.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:06 smithi040 ceph-mon[62643]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T22:36:07.252 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi040:vg_nvme/lv_3 2024-03-18T22:36:07.495 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:36:07.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:07 smithi040 ceph-mon[62643]: purged_snaps scrub starts 2024-03-18T22:36:07.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:07 smithi040 ceph-mon[62643]: purged_snaps scrub ok 2024-03-18T22:36:07.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:07 smithi040 ceph-mon[62643]: osd.2 [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822] boot 2024-03-18T22:36:07.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:07 smithi040 ceph-mon[62643]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T22:36:07.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T22:36:07.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T22:36:07.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:07 smithi040 ceph-mon[62643]: Detected new or changed devices on smithi113 2024-03-18T22:36:07.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:07.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:07.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:36:07.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:07 smithi040 ceph-mon[62643]: Adjusting osd_memory_target on smithi113 to 17088M 2024-03-18T22:36:07.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:07.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:07.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:07.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:08.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:07 smithi113 ceph-mon[66775]: purged_snaps scrub starts 2024-03-18T22:36:08.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:07 smithi113 ceph-mon[66775]: purged_snaps scrub ok 2024-03-18T22:36:08.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:07 smithi113 ceph-mon[66775]: osd.2 [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822] boot 2024-03-18T22:36:08.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:07 smithi113 ceph-mon[66775]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T22:36:08.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T22:36:08.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T22:36:08.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:07 smithi113 ceph-mon[66775]: Detected new or changed devices on smithi113 2024-03-18T22:36:08.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:08.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:08.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:36:08.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:07 smithi113 ceph-mon[66775]: Adjusting osd_memory_target on smithi113 to 17088M 2024-03-18T22:36:08.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:08.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:08.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:08.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:08.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:07 smithi103 ceph-mon[66377]: purged_snaps scrub starts 2024-03-18T22:36:08.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:07 smithi103 ceph-mon[66377]: purged_snaps scrub ok 2024-03-18T22:36:08.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:07 smithi103 ceph-mon[66377]: osd.2 [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822] boot 2024-03-18T22:36:08.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:07 smithi103 ceph-mon[66377]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T22:36:08.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T22:36:08.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T22:36:08.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:07 smithi103 ceph-mon[66377]: Detected new or changed devices on smithi113 2024-03-18T22:36:08.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:08.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:08.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:36:08.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:07 smithi103 ceph-mon[66377]: Adjusting osd_memory_target on smithi113 to 17088M 2024-03-18T22:36:08.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:08.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:08.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:08.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:08.993 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:08 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T22:36:08.994 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:08 smithi040 ceph-mon[62643]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T22:36:08.994 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:08 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T22:36:08.994 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:08 smithi040 ceph-mon[62643]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:08 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T22:36:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:08 smithi113 ceph-mon[66775]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T22:36:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:08 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T22:36:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:08 smithi113 ceph-mon[66775]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:09.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:08 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T22:36:09.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:08 smithi103 ceph-mon[66377]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T22:36:09.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:08 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T22:36:09.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:08 smithi103 ceph-mon[66377]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.198+0000 7f7efa4d4700 1 -- 172.21.15.40:0/4102247760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ef41056f0 msgr2=0x7f7ef4107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:36:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.198+0000 7f7efa4d4700 1 --2- 172.21.15.40:0/4102247760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ef41056f0 0x7f7ef4107ae0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009990 tx=0x7f7ee8009ca0 comp rx=0 tx=0).stop 2024-03-18T22:36:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.199+0000 7f7efa4d4700 1 -- 172.21.15.40:0/4102247760 shutdown_connections 2024-03-18T22:36:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.199+0000 7f7efa4d4700 1 --2- 172.21.15.40:0/4102247760 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ef4108020 0x7f7ef410a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.199+0000 7f7efa4d4700 1 --2- 172.21.15.40:0/4102247760 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef4102dc0 0x7f7ef41051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.199+0000 7f7efa4d4700 1 --2- 172.21.15.40:0/4102247760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ef41056f0 0x7f7ef4107ae0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.199+0000 7f7efa4d4700 1 -- 172.21.15.40:0/4102247760 >> 172.21.15.40:0/4102247760 conn(0x7f7ef40fc7d0 msgr2=0x7f7ef40fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:36:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.199+0000 7f7efa4d4700 1 -- 172.21.15.40:0/4102247760 shutdown_connections 2024-03-18T22:36:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.199+0000 7f7efa4d4700 1 -- 172.21.15.40:0/4102247760 wait complete. 2024-03-18T22:36:09.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.199+0000 7f7efa4d4700 1 Processor -- start 2024-03-18T22:36:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.199+0000 7f7efa4d4700 1 -- start start 2024-03-18T22:36:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7efa4d4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef4102dc0 0x7f7ef4198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:36:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7efa4d4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ef41056f0 0x7f7ef41989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:36:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7efa4d4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ef4108020 0x7f7ef419fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:36:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7efa4d4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef410ddf0 con 0x7f7ef41056f0 2024-03-18T22:36:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7efa4d4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef410dc70 con 0x7f7ef4102dc0 2024-03-18T22:36:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7efa4d4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef410df70 con 0x7f7ef4108020 2024-03-18T22:36:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7ef37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ef41056f0 0x7f7ef41989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:36:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7ef37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ef41056f0 0x7f7ef41989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60734/0 (socket says 172.21.15.40:60734) 2024-03-18T22:36:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7ef37fe700 1 -- 172.21.15.40:0/3760638111 learned_addr learned my addr 172.21.15.40:0/3760638111 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:36:09.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7ef3fff700 1 --2- 172.21.15.40:0/3760638111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef4102dc0 0x7f7ef4198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:36:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7ef8a71700 1 --2- 172.21.15.40:0/3760638111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ef4108020 0x7f7ef419fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:36:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7ef37fe700 1 -- 172.21.15.40:0/3760638111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ef4108020 msgr2=0x7f7ef419fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:36:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7ef37fe700 1 --2- 172.21.15.40:0/3760638111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ef4108020 0x7f7ef419fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7ef37fe700 1 -- 172.21.15.40:0/3760638111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef4102dc0 msgr2=0x7f7ef4198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:36:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7ef37fe700 1 --2- 172.21.15.40:0/3760638111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef4102dc0 0x7f7ef4198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7ef37fe700 1 -- 172.21.15.40:0/3760638111 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef41a00e0 con 0x7f7ef41056f0 2024-03-18T22:36:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7ef8a71700 1 --2- 172.21.15.40:0/3760638111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ef4108020 0x7f7ef419fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:36:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7ef37fe700 1 --2- 172.21.15.40:0/3760638111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ef41056f0 0x7f7ef41989c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8000c00 tx=0x7f7ee80057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:36:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7ef17fa700 1 -- 172.21.15.40:0/3760638111 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8027070 con 0x7f7ef41056f0 2024-03-18T22:36:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7efa4d4700 1 -- 172.21.15.40:0/3760638111 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef41a0370 con 0x7f7ef41056f0 2024-03-18T22:36:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.200+0000 7f7efa4d4700 1 -- 172.21.15.40:0/3760638111 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef41a0820 con 0x7f7ef41056f0 2024-03-18T22:36:09.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.201+0000 7f7ef17fa700 1 -- 172.21.15.40:0/3760638111 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ee8022470 con 0x7f7ef41056f0 2024-03-18T22:36:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.201+0000 7f7ef17fa700 1 -- 172.21.15.40:0/3760638111 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee800f6c0 con 0x7f7ef41056f0 2024-03-18T22:36:09.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.201+0000 7f7efa4d4700 1 -- 172.21.15.40:0/3760638111 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ee0005360 con 0x7f7ef41056f0 2024-03-18T22:36:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.203+0000 7f7ef17fa700 1 -- 172.21.15.40:0/3760638111 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f7ee802a070 con 0x7f7ef41056f0 2024-03-18T22:36:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.203+0000 7f7ef17fa700 1 --2- 172.21.15.40:0/3760638111 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7edc0a2880 0x7f7edc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:36:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.203+0000 7f7ef3fff700 1 --2- 172.21.15.40:0/3760638111 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7edc0a2880 0x7f7edc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:36:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.203+0000 7f7ef17fa700 1 -- 172.21.15.40:0/3760638111 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7f7ee80ce440 con 0x7f7ef41056f0 2024-03-18T22:36:09.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.204+0000 7f7ef3fff700 1 --2- 172.21.15.40:0/3760638111 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7edc0a2880 0x7f7edc0a4d40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4005950 tx=0x7f7ee400a300 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:36:09.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.210+0000 7f7ef17fa700 1 -- 172.21.15.40:0/3760638111 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7ee8013070 con 0x7f7ef41056f0 2024-03-18T22:36:09.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.426+0000 7f7efa4d4700 1 -- 172.21.15.40:0/3760638111 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f7ee0000bf0 con 0x7f7edc0a2880 2024-03-18T22:36:09.622 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:36:09 smithi040 sudo[78953]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T22:36:09.622 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:36:09 smithi040 sudo[78953]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T22:36:09.622 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:36:09 smithi040 sudo[78953]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T22:36:09.622 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:36:09 smithi040 sudo[78953]: pam_unix(sudo:session): session closed for user root 2024-03-18T22:36:09.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:09.829+0000 7f7ef17fa700 1 -- 172.21.15.40:0/3760638111 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7ee80050c0 con 0x7f7ef41056f0 2024-03-18T22:36:09.893 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:36:09 smithi040 sudo[79023]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T22:36:09.893 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:36:09 smithi040 sudo[79023]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T22:36:09.894 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:36:09 smithi040 sudo[79023]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T22:36:09.894 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:36:09 smithi040 sudo[79023]: pam_unix(sudo:session): session closed for user root 2024-03-18T22:36:09.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:09 smithi040 ceph-mon[62643]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T22:36:09.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:09 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T22:36:09.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:09 smithi040 ceph-mon[62643]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T22:36:09.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:09 smithi040 ceph-mon[62643]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:36:09.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:09 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:36:09.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:09 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:36:09.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:09 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:09.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:09 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:09 smithi113 ceph-mon[66775]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T22:36:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:09 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T22:36:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:09 smithi113 ceph-mon[66775]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T22:36:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:09 smithi113 ceph-mon[66775]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:36:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:09 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:36:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:09 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:36:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:09 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:09 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:10.109 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:36:09 smithi103 sudo[74647]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T22:36:10.109 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:36:09 smithi103 sudo[74647]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T22:36:10.109 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:36:09 smithi103 sudo[74647]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T22:36:10.109 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:36:09 smithi103 sudo[74647]: pam_unix(sudo:session): session closed for user root 2024-03-18T22:36:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:09 smithi103 ceph-mon[66377]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T22:36:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:09 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T22:36:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:09 smithi103 ceph-mon[66377]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T22:36:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:09 smithi103 ceph-mon[66377]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:36:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:09 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:36:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:09 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:36:10.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:09 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:10.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:09 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:10.609 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:36:10 smithi103 sudo[74650]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T22:36:10.609 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:36:10 smithi103 sudo[74650]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T22:36:10.609 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:36:10 smithi103 sudo[74650]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T22:36:10.609 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:36:10 smithi103 sudo[74650]: pam_unix(sudo:session): session closed for user root 2024-03-18T22:36:10.671 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 22:36:10 smithi113 sudo[75971]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T22:36:10.672 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 22:36:10 smithi113 sudo[75971]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T22:36:10.672 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 22:36:10 smithi113 sudo[75971]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T22:36:10.672 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 22:36:10 smithi113 sudo[75971]: pam_unix(sudo:session): session closed for user root 2024-03-18T22:36:11.080 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 22:36:10 smithi113 sudo[75974]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T22:36:11.080 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 22:36:10 smithi113 sudo[75974]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T22:36:11.080 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 22:36:10 smithi113 sudo[75974]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T22:36:11.080 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 22:36:10 smithi113 sudo[75974]: pam_unix(sudo:session): session closed for user root 2024-03-18T22:36:11.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:10 smithi040 sudo[79104]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T22:36:11.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:10 smithi040 sudo[79104]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T22:36:11.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:10 smithi040 sudo[79104]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T22:36:11.216 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:10 smithi040 sudo[79104]: pam_unix(sudo:session): session closed for user root 2024-03-18T22:36:11.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:11 smithi040 sudo[79128]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T22:36:11.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:11 smithi040 sudo[79128]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T22:36:11.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:11 smithi040 sudo[79128]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T22:36:11.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:11 smithi040 sudo[79128]: pam_unix(sudo:session): session closed for user root 2024-03-18T22:36:11.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:11 smithi040 ceph-mon[62643]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T22:36:11.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:11 smithi040 ceph-mon[62643]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T22:36:11.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:11 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:11.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:11 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:11.467 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:11 smithi040 ceph-mon[62643]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T22:36:11.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:11 smithi113 ceph-mon[66775]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T22:36:11.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:11 smithi113 ceph-mon[66775]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T22:36:11.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:11 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:11.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:11 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:11.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:11 smithi113 ceph-mon[66775]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T22:36:11.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:11 smithi113 sudo[75977]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T22:36:11.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:11 smithi113 sudo[75977]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T22:36:11.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:11 smithi113 sudo[75977]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T22:36:11.477 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:11 smithi113 sudo[75977]: pam_unix(sudo:session): session closed for user root 2024-03-18T22:36:11.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:11 smithi103 ceph-mon[66377]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T22:36:11.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:11 smithi103 ceph-mon[66377]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T22:36:11.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:11 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:11.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:11 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:11.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:11 smithi103 ceph-mon[66377]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T22:36:11.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:11 smithi113 sudo[75980]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T22:36:11.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:11 smithi113 sudo[75980]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T22:36:11.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:11 smithi113 sudo[75980]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T22:36:11.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:11 smithi113 sudo[75980]: pam_unix(sudo:session): session closed for user root 2024-03-18T22:36:12.209 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:11 smithi103 sudo[74653]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T22:36:12.210 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:11 smithi103 sudo[74653]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T22:36:12.210 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:11 smithi103 sudo[74653]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T22:36:12.210 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:11 smithi103 sudo[74653]: pam_unix(sudo:session): session closed for user root 2024-03-18T22:36:12.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:12.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T22:36:12.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:36:12.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:36:12.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:36:12.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T22:36:12.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:36:12.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:36:12.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:36:12.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T22:36:12.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: Cluster is now healthy 2024-03-18T22:36:12.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T22:36:12.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T22:36:12.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:36:12.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:36:12.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:36:12.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1482298340' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "627e5c6b-419e-4c32-9207-0447a88e18d4"}]: dispatch 2024-03-18T22:36:12.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T22:36:12.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1482298340' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "627e5c6b-419e-4c32-9207-0447a88e18d4"}]': finished 2024-03-18T22:36:12.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T22:36:12.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T22:36:12.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:12.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T22:36:12.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:36:12.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:36:12.607 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:36:12.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T22:36:12.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:36:12.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:36:12.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:36:12.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T22:36:12.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: Cluster is now healthy 2024-03-18T22:36:12.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T22:36:12.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T22:36:12.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:36:12.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:36:12.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:36:12.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1482298340' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "627e5c6b-419e-4c32-9207-0447a88e18d4"}]: dispatch 2024-03-18T22:36:12.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T22:36:12.608 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1482298340' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "627e5c6b-419e-4c32-9207-0447a88e18d4"}]': finished 2024-03-18T22:36:12.609 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T22:36:12.609 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T22:36:12.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 sudo[74656]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T22:36:12.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 sudo[74656]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T22:36:12.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 sudo[74656]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T22:36:12.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 sudo[74656]: pam_unix(sudo:session): session closed for user root 2024-03-18T22:36:12.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:12.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T22:36:12.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:36:12.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:36:12.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:36:12.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T22:36:12.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:36:12.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:36:12.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:36:12.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T22:36:12.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: Cluster is now healthy 2024-03-18T22:36:12.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T22:36:12.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T22:36:12.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T22:36:12.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T22:36:12.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T22:36:12.611 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1482298340' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "627e5c6b-419e-4c32-9207-0447a88e18d4"}]: dispatch 2024-03-18T22:36:12.611 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T22:36:12.611 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1482298340' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "627e5c6b-419e-4c32-9207-0447a88e18d4"}]': finished 2024-03-18T22:36:12.611 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T22:36:12.611 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T22:36:13.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:13 smithi113 ceph-mon[66775]: pgmap v87: 1 pgs: 1 active+clean; 449 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:13.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:13 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2656963999' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:36:13.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:13 smithi103 ceph-mon[66377]: pgmap v87: 1 pgs: 1 active+clean; 449 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:13.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:13 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2656963999' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:36:13.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:13 smithi040 ceph-mon[62643]: pgmap v87: 1 pgs: 1 active+clean; 449 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:13.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:13 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2656963999' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:36:15.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:14 smithi103 ceph-mon[66377]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 7.3 KiB/s rd, 371 KiB/s wr, 24 op/s 2024-03-18T22:36:15.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:14 smithi040 ceph-mon[62643]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 7.3 KiB/s rd, 371 KiB/s wr, 24 op/s 2024-03-18T22:36:15.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:14 smithi113 ceph-mon[66775]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 7.3 KiB/s rd, 371 KiB/s wr, 24 op/s 2024-03-18T22:36:16.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:16 smithi040 ceph-mon[62643]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.2 KiB/s rd, 264 KiB/s wr, 17 op/s 2024-03-18T22:36:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:16 smithi103 ceph-mon[66377]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.2 KiB/s rd, 264 KiB/s wr, 17 op/s 2024-03-18T22:36:17.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:16 smithi113 ceph-mon[66775]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 5.2 KiB/s rd, 264 KiB/s wr, 17 op/s 2024-03-18T22:36:19.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:18 smithi103 ceph-mon[66377]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 205 KiB/s wr, 13 op/s 2024-03-18T22:36:19.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:18 smithi040 ceph-mon[62643]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 205 KiB/s wr, 13 op/s 2024-03-18T22:36:19.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:18 smithi113 ceph-mon[66775]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 205 KiB/s wr, 13 op/s 2024-03-18T22:36:20.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:19 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T22:36:20.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:19 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:20.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:19 smithi103 ceph-mon[66377]: Deploying daemon osd.3 on smithi040 2024-03-18T22:36:20.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:19 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T22:36:20.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:19 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:20.115 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:19 smithi040 ceph-mon[62643]: Deploying daemon osd.3 on smithi040 2024-03-18T22:36:20.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:19 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T22:36:20.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:19 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:20.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:19 smithi113 ceph-mon[66775]: Deploying daemon osd.3 on smithi040 2024-03-18T22:36:21.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:20 smithi103 ceph-mon[66377]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-18T22:36:21.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:20 smithi040 ceph-mon[62643]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-18T22:36:21.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:20 smithi113 ceph-mon[66775]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-18T22:36:22.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:22 smithi040 ceph-mon[62643]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 173 KiB/s wr, 11 op/s 2024-03-18T22:36:23.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:22 smithi103 ceph-mon[66377]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 173 KiB/s wr, 11 op/s 2024-03-18T22:36:23.132 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:22 smithi113 ceph-mon[66775]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 173 KiB/s wr, 11 op/s 2024-03-18T22:36:25.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:24 smithi103 ceph-mon[66377]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-18T22:36:25.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:24 smithi040 ceph-mon[62643]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-18T22:36:25.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:24 smithi113 ceph-mon[66775]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-18T22:36:25.979 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:25 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:25.979 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:25 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:25.979 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:25 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:26.133 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:25 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:26.134 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:25 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:26.134 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:25 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:25 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:25 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:25 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:27.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:26 smithi040 ceph-mon[62643]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:27.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:26 smithi113 ceph-mon[66775]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:27.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:26 smithi103 ceph-mon[66377]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:28.438 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 3 on host 'smithi040' 2024-03-18T22:36:28.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:28.436+0000 7f7ef17fa700 1 -- 172.21.15.40:0/3760638111 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7ee0000bf0 con 0x7f7edc0a2880 2024-03-18T22:36:28.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:28.439+0000 7f7efa4d4700 1 -- 172.21.15.40:0/3760638111 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7edc0a2880 msgr2=0x7f7edc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:36:28.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:28.439+0000 7f7efa4d4700 1 --2- 172.21.15.40:0/3760638111 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7edc0a2880 0x7f7edc0a4d40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4005950 tx=0x7f7ee400a300 comp rx=0 tx=0).stop 2024-03-18T22:36:28.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:28.439+0000 7f7efa4d4700 1 -- 172.21.15.40:0/3760638111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ef41056f0 msgr2=0x7f7ef41989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:36:28.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:28.439+0000 7f7efa4d4700 1 --2- 172.21.15.40:0/3760638111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ef41056f0 0x7f7ef41989c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8000c00 tx=0x7f7ee80057c0 comp rx=0 tx=0).stop 2024-03-18T22:36:28.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:28.439+0000 7f7efa4d4700 1 -- 172.21.15.40:0/3760638111 shutdown_connections 2024-03-18T22:36:28.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:28.439+0000 7f7efa4d4700 1 --2- 172.21.15.40:0/3760638111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ef4108020 0x7f7ef419fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:28.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:28.439+0000 7f7efa4d4700 1 --2- 172.21.15.40:0/3760638111 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7edc0a2880 0x7f7edc0a4d40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:28.439+0000 7f7efa4d4700 1 --2- 172.21.15.40:0/3760638111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ef4102dc0 0x7f7ef4198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:28.439+0000 7f7efa4d4700 1 --2- 172.21.15.40:0/3760638111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ef41056f0 0x7f7ef41989c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:28.439+0000 7f7efa4d4700 1 -- 172.21.15.40:0/3760638111 >> 172.21.15.40:0/3760638111 conn(0x7f7ef40fc7d0 msgr2=0x7f7ef4103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:36:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:28.439+0000 7f7efa4d4700 1 -- 172.21.15.40:0/3760638111 shutdown_connections 2024-03-18T22:36:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:36:28.439+0000 7f7efa4d4700 1 -- 172.21.15.40:0/3760638111 wait complete. 2024-03-18T22:36:29.121 DEBUG:teuthology.orchestra.run.smithi040:osd.3> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.3.service 2024-03-18T22:36:29.123 INFO:tasks.cephadm:Deploying osd.4 on smithi103 with /dev/vg_nvme/lv_3... 2024-03-18T22:36:29.123 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T22:36:29.370 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T22:36:29.729 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:29.729 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:29.729 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:29.729 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:29.729 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:29 smithi103 ceph-mon[66377]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:29.730 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:29 smithi103 ceph-mon[66377]: from='osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T22:36:29.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:29.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:29.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:29.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:29.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:29 smithi113 ceph-mon[66775]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:29.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:29 smithi113 ceph-mon[66775]: from='osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T22:36:29.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:29 smithi040 ceph-mon[62643]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:29 smithi040 ceph-mon[62643]: from='osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T22:36:30.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:30 smithi040 ceph-mon[62643]: from='osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T22:36:30.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:30 smithi040 ceph-mon[62643]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T22:36:30.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:30 smithi040 ceph-mon[62643]: from='osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-18T22:36:30.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:30 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T22:36:31.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:30 smithi113 ceph-mon[66775]: from='osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T22:36:31.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:30 smithi113 ceph-mon[66775]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T22:36:31.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:30 smithi113 ceph-mon[66775]: from='osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-18T22:36:31.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:30 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T22:36:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:30 smithi103 ceph-mon[66377]: from='osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T22:36:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:30 smithi103 ceph-mon[66377]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T22:36:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:30 smithi103 ceph-mon[66377]: from='osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-18T22:36:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:30 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T22:36:31.313 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T22:36:31.313 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T22:36:31.314 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-03-18T22:36:31.314 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-03-18T22:36:31.314 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111535 s, 940 MB/s 2024-03-18T22:36:31.314 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-03-18T22:36:31.747 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 22:36:31 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-3[81228]: 2024-03-18T22:36:31.399+0000 7f27deb09700 -1 osd.3 0 waiting for initial osdmap 2024-03-18T22:36:31.747 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 22:36:31 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-3[81228]: 2024-03-18T22:36:31.405+0000 7f27d68f4700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T22:36:32.049 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:31 smithi103 ceph-mon[66377]: from='osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-18T22:36:32.050 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:31 smithi103 ceph-mon[66377]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T22:36:32.050 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:31 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T22:36:32.050 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:31 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T22:36:32.050 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:31 smithi103 ceph-mon[66377]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:32.050 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:31 smithi103 ceph-mon[66377]: from='osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891]' entity='osd.3' 2024-03-18T22:36:32.050 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:31 smithi103 ceph-mon[66377]: osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891] boot 2024-03-18T22:36:32.050 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:31 smithi103 ceph-mon[66377]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T22:36:32.050 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:31 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T22:36:32.070 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:31 smithi040 ceph-mon[62643]: from='osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-18T22:36:32.070 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:31 smithi040 ceph-mon[62643]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T22:36:32.071 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:31 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T22:36:32.071 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:31 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T22:36:32.071 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:31 smithi040 ceph-mon[62643]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:32.071 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:31 smithi040 ceph-mon[62643]: from='osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891]' entity='osd.3' 2024-03-18T22:36:32.071 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:31 smithi040 ceph-mon[62643]: osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891] boot 2024-03-18T22:36:32.071 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:31 smithi040 ceph-mon[62643]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T22:36:32.071 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:31 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T22:36:32.076 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi103:vg_nvme/lv_3 2024-03-18T22:36:32.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:31 smithi113 ceph-mon[66775]: from='osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-18T22:36:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:31 smithi113 ceph-mon[66775]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T22:36:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:31 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T22:36:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:31 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T22:36:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:31 smithi113 ceph-mon[66775]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T22:36:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:31 smithi113 ceph-mon[66775]: from='osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891]' entity='osd.3' 2024-03-18T22:36:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:31 smithi113 ceph-mon[66775]: osd.3 [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891] boot 2024-03-18T22:36:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:31 smithi113 ceph-mon[66775]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T22:36:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:31 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T22:36:32.319 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T22:36:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:32 smithi113 ceph-mon[66775]: purged_snaps scrub starts 2024-03-18T22:36:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:32 smithi113 ceph-mon[66775]: purged_snaps scrub ok 2024-03-18T22:36:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:36:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:36:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:32 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:33.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:32 smithi113 ceph-mon[66775]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T22:36:33.081 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:32 smithi103 ceph-mon[66377]: purged_snaps scrub starts 2024-03-18T22:36:33.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:32 smithi103 ceph-mon[66377]: purged_snaps scrub ok 2024-03-18T22:36:33.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:32 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:33.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:32 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:33.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:32 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:36:33.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:32 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:36:33.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:32 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:33.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:32 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:33.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:32 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:33.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:32 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:33.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:32 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:33.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:32 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:33.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:32 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:33.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:32 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:33.083 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:32 smithi103 ceph-mon[66377]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T22:36:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:32 smithi040 ceph-mon[62643]: purged_snaps scrub starts 2024-03-18T22:36:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:32 smithi040 ceph-mon[62643]: purged_snaps scrub ok 2024-03-18T22:36:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:36:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:36:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:33.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:32 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:33.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:32 smithi040 ceph-mon[62643]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T22:36:33.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.868+0000 7f36a4005700 1 -- 172.21.15.103:0/3475590504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c102e80 msgr2=0x7f369c109730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:36:33.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.868+0000 7f36a4005700 1 --2- 172.21.15.103:0/3475590504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c102e80 0x7f369c109730 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f369400b0d0 tx=0x7f369400b3e0 comp rx=0 tx=0).stop 2024-03-18T22:36:33.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.868+0000 7f36a4005700 1 -- 172.21.15.103:0/3475590504 shutdown_connections 2024-03-18T22:36:33.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.868+0000 7f36a4005700 1 --2- 172.21.15.103:0/3475590504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c102e80 0x7f369c109730 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:33.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.868+0000 7f36a4005700 1 --2- 172.21.15.103:0/3475590504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f369c1012e0 0x7f369c101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:33.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.868+0000 7f36a4005700 1 --2- 172.21.15.103:0/3475590504 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f369c1024e0 0x7f369c102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:33.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.868+0000 7f36a4005700 1 -- 172.21.15.103:0/3475590504 >> 172.21.15.103:0/3475590504 conn(0x7f369c0fc8a0 msgr2=0x7f369c0fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:36:33.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.869+0000 7f36a4005700 1 -- 172.21.15.103:0/3475590504 shutdown_connections 2024-03-18T22:36:33.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.869+0000 7f36a4005700 1 -- 172.21.15.103:0/3475590504 wait complete. 2024-03-18T22:36:33.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.869+0000 7f36a4005700 1 Processor -- start 2024-03-18T22:36:33.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.869+0000 7f36a4005700 1 -- start start 2024-03-18T22:36:33.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.869+0000 7f36a4005700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c1012e0 0x7f369c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:36:33.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.869+0000 7f36a4005700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f369c1024e0 0x7f369c198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:36:33.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.869+0000 7f36a4005700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f369c102e80 0x7f369c19f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:36:33.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.869+0000 7f36a4005700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f369c10cd00 con 0x7f369c102e80 2024-03-18T22:36:33.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.869+0000 7f36a4005700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f369c10cb80 con 0x7f369c1012e0 2024-03-18T22:36:33.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.869+0000 7f36a4005700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f369c10ce80 con 0x7f369c1024e0 2024-03-18T22:36:33.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.869+0000 7f36a1da1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c1012e0 0x7f369c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:36:33.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.870+0000 7f36a1da1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c1012e0 0x7f369c198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:52060/0 (socket says 172.21.15.103:52060) 2024-03-18T22:36:33.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.870+0000 7f36a1da1700 1 -- 172.21.15.103:0/199046410 learned_addr learned my addr 172.21.15.103:0/199046410 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-18T22:36:33.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.870+0000 7f36a25a2700 1 --2- 172.21.15.103:0/199046410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f369c102e80 0x7f369c19f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:36:33.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.870+0000 7f36a15a0700 1 --2- 172.21.15.103:0/199046410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f369c1024e0 0x7f369c198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:36:33.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.870+0000 7f36a1da1700 1 -- 172.21.15.103:0/199046410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f369c1024e0 msgr2=0x7f369c198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:36:33.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.870+0000 7f36a1da1700 1 --2- 172.21.15.103:0/199046410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f369c1024e0 0x7f369c198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:33.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.870+0000 7f36a1da1700 1 -- 172.21.15.103:0/199046410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f369c102e80 msgr2=0x7f369c19f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:36:33.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.870+0000 7f36a1da1700 1 --2- 172.21.15.103:0/199046410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f369c102e80 0x7f369c19f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:33.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.870+0000 7f36a1da1700 1 -- 172.21.15.103:0/199046410 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f369c1a0090 con 0x7f369c1012e0 2024-03-18T22:36:33.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.870+0000 7f36a1da1700 1 --2- 172.21.15.103:0/199046410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c1012e0 0x7f369c198430 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f369800d980 tx=0x7f369800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:36:33.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.870+0000 7f3692ffd700 1 -- 172.21.15.103:0/199046410 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3698017030 con 0x7f369c1012e0 2024-03-18T22:36:33.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.870+0000 7f36a4005700 1 -- 172.21.15.103:0/199046410 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f369c1a0380 con 0x7f369c1012e0 2024-03-18T22:36:33.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.870+0000 7f3692ffd700 1 -- 172.21.15.103:0/199046410 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36980044d0 con 0x7f369c1012e0 2024-03-18T22:36:33.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.870+0000 7f3692ffd700 1 -- 172.21.15.103:0/199046410 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3698003950 con 0x7f369c1012e0 2024-03-18T22:36:33.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.870+0000 7f36a4005700 1 -- 172.21.15.103:0/199046410 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f369c1a0910 con 0x7f369c1012e0 2024-03-18T22:36:33.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.871+0000 7f36a4005700 1 -- 172.21.15.103:0/199046410 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f369c068fd0 con 0x7f369c1012e0 2024-03-18T22:36:33.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.873+0000 7f3692ffd700 1 -- 172.21.15.103:0/199046410 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f369801a070 con 0x7f369c1012e0 2024-03-18T22:36:33.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.873+0000 7f3692ffd700 1 --2- 172.21.15.103:0/199046410 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f36880a2890 0x7f36880a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:36:33.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.874+0000 7f3692ffd700 1 -- 172.21.15.103:0/199046410 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f36980cd410 con 0x7f369c1012e0 2024-03-18T22:36:33.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.874+0000 7f36a15a0700 1 --2- 172.21.15.103:0/199046410 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f36880a2890 0x7f36880a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:36:33.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.874+0000 7f36a15a0700 1 --2- 172.21.15.103:0/199046410 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f36880a2890 0x7f36880a4d50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f368c004630 tx=0x7f368c00b3c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:36:33.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:33.877+0000 7f3692ffd700 1 -- 172.21.15.103:0/199046410 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f369809b6b0 con 0x7f369c1012e0 2024-03-18T22:36:34.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:33 smithi040 ceph-mon[62643]: Detected new or changed devices on smithi040 2024-03-18T22:36:34.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:33 smithi040 ceph-mon[62643]: Adjusting osd_memory_target on smithi040 to 8546M 2024-03-18T22:36:34.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:33 smithi040 ceph-mon[62643]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:34.046 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:33 smithi040 ceph-mon[62643]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T22:36:34.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:33 smithi113 ceph-mon[66775]: Detected new or changed devices on smithi040 2024-03-18T22:36:34.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:33 smithi113 ceph-mon[66775]: Adjusting osd_memory_target on smithi040 to 8546M 2024-03-18T22:36:34.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:33 smithi113 ceph-mon[66775]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:34.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:33 smithi113 ceph-mon[66775]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T22:36:34.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:34.080+0000 7f36a4005700 1 -- 172.21.15.103:0/199046410 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f369c06b120 con 0x7f36880a2890 2024-03-18T22:36:34.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:33 smithi103 ceph-mon[66377]: Detected new or changed devices on smithi040 2024-03-18T22:36:34.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:33 smithi103 ceph-mon[66377]: Adjusting osd_memory_target on smithi040 to 8546M 2024-03-18T22:36:34.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:33 smithi103 ceph-mon[66377]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:34.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:33 smithi103 ceph-mon[66377]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T22:36:34.937 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:34 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:36:34.937 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:34 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:36:34.937 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:34 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:34.937 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:34 smithi103 ceph-mon[66377]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T22:36:35.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:34 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:36:35.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:34 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:36:35.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:34 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:35.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:34 smithi113 ceph-mon[66775]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T22:36:35.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:34 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:36:35.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:34 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:36:35.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:34 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:35.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:34 smithi040 ceph-mon[62643]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T22:36:35.962 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:35 smithi103 ceph-mon[66377]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:36:35.963 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:35 smithi103 ceph-mon[66377]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 368 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T22:36:35.963 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:35 smithi103 ceph-mon[66377]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T22:36:36.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:35 smithi113 ceph-mon[66775]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:36:36.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:35 smithi113 ceph-mon[66775]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 368 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T22:36:36.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:35 smithi113 ceph-mon[66775]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T22:36:36.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:35 smithi040 ceph-mon[62643]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:36:36.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:35 smithi040 ceph-mon[62643]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 368 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T22:36:36.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:35 smithi040 ceph-mon[62643]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T22:36:38.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:37 smithi113 ceph-mon[66775]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 142 KiB/s, 0 objects/s recovering 2024-03-18T22:36:38.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:37 smithi113 ceph-mon[66775]: from='client.? 172.21.15.103:0/907674302' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34f3434e-5092-4639-9c2c-f64a196ea4b0"}]: dispatch 2024-03-18T22:36:38.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:37 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34f3434e-5092-4639-9c2c-f64a196ea4b0"}]: dispatch 2024-03-18T22:36:38.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:37 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34f3434e-5092-4639-9c2c-f64a196ea4b0"}]': finished 2024-03-18T22:36:38.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:37 smithi113 ceph-mon[66775]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T22:36:38.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:37 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:38.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:37 smithi113 ceph-mon[66775]: from='client.? 172.21.15.103:0/4166455192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:36:38.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:37 smithi103 ceph-mon[66377]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 142 KiB/s, 0 objects/s recovering 2024-03-18T22:36:38.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:37 smithi103 ceph-mon[66377]: from='client.? 172.21.15.103:0/907674302' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34f3434e-5092-4639-9c2c-f64a196ea4b0"}]: dispatch 2024-03-18T22:36:38.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:37 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34f3434e-5092-4639-9c2c-f64a196ea4b0"}]: dispatch 2024-03-18T22:36:38.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:37 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34f3434e-5092-4639-9c2c-f64a196ea4b0"}]': finished 2024-03-18T22:36:38.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:37 smithi103 ceph-mon[66377]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T22:36:38.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:37 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:38.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:37 smithi103 ceph-mon[66377]: from='client.? 172.21.15.103:0/4166455192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:36:38.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:37 smithi040 ceph-mon[62643]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 142 KiB/s, 0 objects/s recovering 2024-03-18T22:36:38.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:37 smithi040 ceph-mon[62643]: from='client.? 172.21.15.103:0/907674302' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34f3434e-5092-4639-9c2c-f64a196ea4b0"}]: dispatch 2024-03-18T22:36:38.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:37 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34f3434e-5092-4639-9c2c-f64a196ea4b0"}]: dispatch 2024-03-18T22:36:38.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:37 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34f3434e-5092-4639-9c2c-f64a196ea4b0"}]': finished 2024-03-18T22:36:38.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:37 smithi040 ceph-mon[62643]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T22:36:38.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:37 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:38.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:37 smithi040 ceph-mon[62643]: from='client.? 172.21.15.103:0/4166455192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:36:40.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:39 smithi113 ceph-mon[66775]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-18T22:36:40.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:39 smithi103 ceph-mon[66377]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-18T22:36:40.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:39 smithi040 ceph-mon[62643]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-18T22:36:42.056 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:41 smithi103 ceph-mon[66377]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:42.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:41 smithi113 ceph-mon[66775]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:42.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:41 smithi040 ceph-mon[62643]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:44.026 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:43 smithi103 ceph-mon[66377]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:44.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:43 smithi113 ceph-mon[66775]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:44.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:43 smithi040 ceph-mon[62643]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:44.979 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T22:36:44.979 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:45.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T22:36:45.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:45.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T22:36:45.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:45.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:45 smithi103 ceph-mon[66377]: Deploying daemon osd.4 on smithi103 2024-03-18T22:36:45.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:45 smithi103 ceph-mon[66377]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:46.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:45 smithi113 ceph-mon[66775]: Deploying daemon osd.4 on smithi103 2024-03-18T22:36:46.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:45 smithi113 ceph-mon[66775]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:46.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:45 smithi040 ceph-mon[62643]: Deploying daemon osd.4 on smithi103 2024-03-18T22:36:46.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:45 smithi040 ceph-mon[62643]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:47.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:47 smithi103 ceph-mon[66377]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:48.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:47 smithi113 ceph-mon[66775]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:48.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:47 smithi040 ceph-mon[62643]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:50.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:49 smithi103 ceph-mon[66377]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:50.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:49 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:50.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:49 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:50.029 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:49 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:50.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:49 smithi113 ceph-mon[66775]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:50.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:49 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:50.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:49 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:50.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:49 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:50.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:49 smithi040 ceph-mon[62643]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:50.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:49 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:50.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:49 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:50.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:49 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:51.948 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:51 smithi103 ceph-mon[66377]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:52.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:51 smithi113 ceph-mon[66775]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:52.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:51 smithi040 ceph-mon[62643]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:53.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:53.144+0000 7f3692ffd700 1 -- 172.21.15.103:0/199046410 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f369c06b120 con 0x7f36880a2890 2024-03-18T22:36:53.144 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 4 on host 'smithi103' 2024-03-18T22:36:53.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:53.146+0000 7f36a4005700 1 -- 172.21.15.103:0/199046410 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f36880a2890 msgr2=0x7f36880a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:36:53.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:53.146+0000 7f36a4005700 1 --2- 172.21.15.103:0/199046410 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f36880a2890 0x7f36880a4d50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f368c004630 tx=0x7f368c00b3c0 comp rx=0 tx=0).stop 2024-03-18T22:36:53.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:53.146+0000 7f36a4005700 1 -- 172.21.15.103:0/199046410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c1012e0 msgr2=0x7f369c198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:36:53.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:53.146+0000 7f36a4005700 1 --2- 172.21.15.103:0/199046410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c1012e0 0x7f369c198430 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f369800d980 tx=0x7f369800dc90 comp rx=0 tx=0).stop 2024-03-18T22:36:53.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:53.146+0000 7f36a4005700 1 -- 172.21.15.103:0/199046410 shutdown_connections 2024-03-18T22:36:53.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:53.146+0000 7f36a4005700 1 --2- 172.21.15.103:0/199046410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f369c102e80 0x7f369c19f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:53.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:53.146+0000 7f36a4005700 1 --2- 172.21.15.103:0/199046410 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f36880a2890 0x7f36880a4d50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:53.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:53.146+0000 7f36a4005700 1 --2- 172.21.15.103:0/199046410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f369c1012e0 0x7f369c198430 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:53.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:53.146+0000 7f36a4005700 1 --2- 172.21.15.103:0/199046410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f369c1024e0 0x7f369c198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:53.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:53.146+0000 7f36a4005700 1 -- 172.21.15.103:0/199046410 >> 172.21.15.103:0/199046410 conn(0x7f369c0fc8a0 msgr2=0x7f369c06aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:36:53.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:53.146+0000 7f36a4005700 1 -- 172.21.15.103:0/199046410 shutdown_connections 2024-03-18T22:36:53.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:36:53.146+0000 7f36a4005700 1 -- 172.21.15.103:0/199046410 wait complete. 2024-03-18T22:36:53.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:53.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:53.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:53.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:53.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:53.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:53 smithi103 ceph-mon[66377]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:53.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:53.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:53.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:53 smithi113 ceph-mon[66775]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:53.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:53.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:53.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:53.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:53.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:53.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:53 smithi040 ceph-mon[62643]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:53.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:53.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:53.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:36:53.773 DEBUG:teuthology.orchestra.run.smithi103:osd.4> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.4.service 2024-03-18T22:36:53.775 INFO:tasks.cephadm:Deploying osd.5 on smithi113 with /dev/vg_nvme/lv_3... 2024-03-18T22:36:53.775 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T22:36:54.031 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:36:54.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:54 smithi113 ceph-mon[66775]: from='osd.4 [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T22:36:54.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:54 smithi113 ceph-mon[66775]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T22:36:54.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:54 smithi103 ceph-mon[66377]: from='osd.4 [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T22:36:54.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:54 smithi103 ceph-mon[66377]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T22:36:54.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:54 smithi040 ceph-mon[62643]: from='osd.4 [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T22:36:54.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:54 smithi040 ceph-mon[62643]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T22:36:55.300 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:55 smithi113 ceph-mon[66775]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T22:36:55.300 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:55 smithi113 ceph-mon[66775]: from='osd.4 [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:36:55.300 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:55 smithi113 ceph-mon[66775]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T22:36:55.301 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:55.301 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:55 smithi113 ceph-mon[66775]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:36:55.301 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:55 smithi113 ceph-mon[66775]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:55.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:55 smithi103 ceph-mon[66377]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T22:36:55.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:55 smithi103 ceph-mon[66377]: from='osd.4 [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:36:55.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:55 smithi103 ceph-mon[66377]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T22:36:55.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:55.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:55 smithi103 ceph-mon[66377]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:36:55.489 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:55 smithi103 ceph-mon[66377]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:55.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:55 smithi040 ceph-mon[62643]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T22:36:55.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:55 smithi040 ceph-mon[62643]: from='osd.4 [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:36:55.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:55 smithi040 ceph-mon[62643]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T22:36:55.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:55.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:55 smithi040 ceph-mon[62643]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:36:55.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:55 smithi040 ceph-mon[62643]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:55.861 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T22:36:55.861 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T22:36:55.861 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-03-18T22:36:55.861 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-03-18T22:36:55.861 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00810543 s, 1.3 GB/s 2024-03-18T22:36:55.861 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-03-18T22:36:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:56 smithi103 ceph-mon[66377]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-18T22:36:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:56 smithi103 ceph-mon[66377]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T22:36:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:56 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:56 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:56 smithi103 ceph-mon[66377]: from='osd.4 ' entity='osd.4' 2024-03-18T22:36:56.360 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 22:36:56 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-4[77442]: 2024-03-18T22:36:56.011+0000 7f9694129700 -1 osd.4 0 waiting for initial osdmap 2024-03-18T22:36:56.360 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 22:36:56 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-4[77442]: 2024-03-18T22:36:56.019+0000 7f968b713700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T22:36:56.466 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:56 smithi113 ceph-mon[66775]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-18T22:36:56.466 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:56 smithi113 ceph-mon[66775]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T22:36:56.466 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:56 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:56.466 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:56 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:56.466 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:56 smithi113 ceph-mon[66775]: from='osd.4 ' entity='osd.4' 2024-03-18T22:36:56.467 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi113:vg_nvme/lv_3 2024-03-18T22:36:56.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:56 smithi040 ceph-mon[62643]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-18T22:36:56.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:56 smithi040 ceph-mon[62643]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T22:36:56.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:56 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:56.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:56 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:56.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:56 smithi040 ceph-mon[62643]: from='osd.4 ' entity='osd.4' 2024-03-18T22:36:56.701 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:36:57.472 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: purged_snaps scrub starts 2024-03-18T22:36:57.472 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: purged_snaps scrub ok 2024-03-18T22:36:57.472 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:57.472 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:57.472 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: Detected new or changed devices on smithi103 2024-03-18T22:36:57.472 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:57.472 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:57.472 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:36:57.473 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:36:57.473 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: Adjusting osd_memory_target on smithi103 to 8544M 2024-03-18T22:36:57.473 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:57.473 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:57.473 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:57.473 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:57.473 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: osd.4 [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460] boot 2024-03-18T22:36:57.473 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T22:36:57.473 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:57.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: purged_snaps scrub starts 2024-03-18T22:36:57.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: purged_snaps scrub ok 2024-03-18T22:36:57.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:57.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: Detected new or changed devices on smithi103 2024-03-18T22:36:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:36:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:36:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: Adjusting osd_memory_target on smithi103 to 8544M 2024-03-18T22:36:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: osd.4 [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460] boot 2024-03-18T22:36:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T22:36:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:57.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: purged_snaps scrub starts 2024-03-18T22:36:57.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: purged_snaps scrub ok 2024-03-18T22:36:57.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:57.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-03-18T22:36:57.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: Detected new or changed devices on smithi103 2024-03-18T22:36:57.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:57.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:57.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:36:57.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:36:57.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: Adjusting osd_memory_target on smithi103 to 8544M 2024-03-18T22:36:57.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:57.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:57.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:36:57.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:36:57.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: osd.4 [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460] boot 2024-03-18T22:36:57.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T22:36:57.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T22:36:58.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.193+0000 7f483fe7e700 1 -- 172.21.15.113:0/2953871598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4838102440 msgr2=0x7f48381028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:36:58.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.193+0000 7f483fe7e700 1 --2- 172.21.15.113:0/2953871598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4838102440 0x7f48381028a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4828009990 tx=0x7f4828009ca0 comp rx=0 tx=0).stop 2024-03-18T22:36:58.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.193+0000 7f483fe7e700 1 -- 172.21.15.113:0/2953871598 shutdown_connections 2024-03-18T22:36:58.196 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.193+0000 7f483fe7e700 1 --2- 172.21.15.113:0/2953871598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4838102de0 0x7f4838109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:58.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.193+0000 7f483fe7e700 1 --2- 172.21.15.113:0/2953871598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4838101240 0x7f4838101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:58.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.193+0000 7f483fe7e700 1 --2- 172.21.15.113:0/2953871598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4838102440 0x7f48381028a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:58.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.193+0000 7f483fe7e700 1 -- 172.21.15.113:0/2953871598 >> 172.21.15.113:0/2953871598 conn(0x7f48380fc7e0 msgr2=0x7f48380fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:36:58.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.193+0000 7f483fe7e700 1 -- 172.21.15.113:0/2953871598 shutdown_connections 2024-03-18T22:36:58.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.193+0000 7f483fe7e700 1 -- 172.21.15.113:0/2953871598 wait complete. 2024-03-18T22:36:58.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483fe7e700 1 Processor -- start 2024-03-18T22:36:58.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483fe7e700 1 -- start start 2024-03-18T22:36:58.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483fe7e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4838101240 0x7f483806ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:36:58.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483fe7e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4838102440 0x7f483819d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:36:58.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483fe7e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4838102de0 0x7f483819f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:36:58.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483fe7e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f483810ccc0 con 0x7f4838102de0 2024-03-18T22:36:58.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483fe7e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f483810cb40 con 0x7f4838102440 2024-03-18T22:36:58.197 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483fe7e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f483810ce40 con 0x7f4838101240 2024-03-18T22:36:58.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483dc1a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4838101240 0x7f483806ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:36:58.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483dc1a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4838101240 0x7f483806ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:42418/0 (socket says 172.21.15.113:42418) 2024-03-18T22:36:58.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483dc1a700 1 -- 172.21.15.113:0/2663546336 learned_addr learned my addr 172.21.15.113:0/2663546336 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-18T22:36:58.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483e41b700 1 --2- 172.21.15.113:0/2663546336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4838102de0 0x7f483819f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:36:58.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483d419700 1 --2- 172.21.15.113:0/2663546336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4838102440 0x7f483819d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:36:58.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483dc1a700 1 -- 172.21.15.113:0/2663546336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4838102440 msgr2=0x7f483819d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:36:58.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483dc1a700 1 --2- 172.21.15.113:0/2663546336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4838102440 0x7f483819d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:58.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483dc1a700 1 -- 172.21.15.113:0/2663546336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4838102de0 msgr2=0x7f483819f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:36:58.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483dc1a700 1 --2- 172.21.15.113:0/2663546336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4838102de0 0x7f483819f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:36:58.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483dc1a700 1 -- 172.21.15.113:0/2663546336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48381a0140 con 0x7f4838101240 2024-03-18T22:36:58.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483d419700 1 --2- 172.21.15.113:0/2663546336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4838102440 0x7f483819d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:36:58.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.194+0000 7f483dc1a700 1 --2- 172.21.15.113:0/2663546336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4838101240 0x7f483806ab20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f483400d980 tx=0x7f483400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:36:58.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.195+0000 7f482effd700 1 -- 172.21.15.113:0/2663546336 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4834017030 con 0x7f4838101240 2024-03-18T22:36:58.198 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.195+0000 7f483fe7e700 1 -- 172.21.15.113:0/2663546336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48381a0430 con 0x7f4838101240 2024-03-18T22:36:58.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.195+0000 7f482effd700 1 -- 172.21.15.113:0/2663546336 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48340044d0 con 0x7f4838101240 2024-03-18T22:36:58.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.195+0000 7f482effd700 1 -- 172.21.15.113:0/2663546336 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48340039e0 con 0x7f4838101240 2024-03-18T22:36:58.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.195+0000 7f483fe7e700 1 -- 172.21.15.113:0/2663546336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48381a0970 con 0x7f4838101240 2024-03-18T22:36:58.199 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.195+0000 7f483fe7e700 1 -- 172.21.15.113:0/2663546336 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4838068fd0 con 0x7f4838101240 2024-03-18T22:36:58.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.197+0000 7f482effd700 1 -- 172.21.15.113:0/2663546336 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f483401a070 con 0x7f4838101240 2024-03-18T22:36:58.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.197+0000 7f482effd700 1 --2- 172.21.15.113:0/2663546336 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f48240a2890 0x7f48240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:36:58.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.197+0000 7f482effd700 1 -- 172.21.15.113:0/2663546336 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f48340cd6b0 con 0x7f4838101240 2024-03-18T22:36:58.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.197+0000 7f483d419700 1 --2- 172.21.15.113:0/2663546336 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f48240a2890 0x7f48240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:36:58.200 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.198+0000 7f483d419700 1 --2- 172.21.15.113:0/2663546336 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f48240a2890 0x7f48240a4d50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f482800b5c0 tx=0x7f4828005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:36:58.202 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.202+0000 7f482effd700 1 -- 172.21.15.113:0/2663546336 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f483409b7c0 con 0x7f4838101240 2024-03-18T22:36:58.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:36:58.405+0000 7f483fe7e700 1 -- 172.21.15.113:0/2663546336 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f4838062750 con 0x7f48240a2890 2024-03-18T22:36:59.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:59 smithi113 ceph-mon[66775]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T22:36:59.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:59 smithi113 ceph-mon[66775]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:36:59.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:36:59.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:36:59.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:59.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:36:59 smithi113 ceph-mon[66775]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:36:59.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:59 smithi103 ceph-mon[66377]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T22:36:59.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:59 smithi103 ceph-mon[66377]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:36:59.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:36:59.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:36:59.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:59.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:36:59 smithi103 ceph-mon[66377]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:36:59.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:59 smithi040 ceph-mon[62643]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T22:36:59.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:59 smithi040 ceph-mon[62643]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:36:59.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:36:59.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:36:59.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:36:59.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:36:59 smithi040 ceph-mon[62643]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:01.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:00 smithi103 ceph-mon[66377]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:01.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:00 smithi040 ceph-mon[62643]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:00 smithi113 ceph-mon[66775]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:02.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:01 smithi103 ceph-mon[66377]: from='client.? 172.21.15.113:0/103295413' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9617fbc-5b2c-4e57-a926-b1b3a0953e58"}]: dispatch 2024-03-18T22:37:02.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:01 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9617fbc-5b2c-4e57-a926-b1b3a0953e58"}]: dispatch 2024-03-18T22:37:02.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:01 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9617fbc-5b2c-4e57-a926-b1b3a0953e58"}]': finished 2024-03-18T22:37:02.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:01 smithi103 ceph-mon[66377]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T22:37:02.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:01 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T22:37:02.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:01 smithi103 ceph-mon[66377]: from='client.? 172.21.15.113:0/3955968865' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:37:02.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:01 smithi040 ceph-mon[62643]: from='client.? 172.21.15.113:0/103295413' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9617fbc-5b2c-4e57-a926-b1b3a0953e58"}]: dispatch 2024-03-18T22:37:02.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:01 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9617fbc-5b2c-4e57-a926-b1b3a0953e58"}]: dispatch 2024-03-18T22:37:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:01 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9617fbc-5b2c-4e57-a926-b1b3a0953e58"}]': finished 2024-03-18T22:37:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:01 smithi040 ceph-mon[62643]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T22:37:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:01 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T22:37:02.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:01 smithi040 ceph-mon[62643]: from='client.? 172.21.15.113:0/3955968865' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:37:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:01 smithi113 ceph-mon[66775]: from='client.? 172.21.15.113:0/103295413' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9617fbc-5b2c-4e57-a926-b1b3a0953e58"}]: dispatch 2024-03-18T22:37:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:01 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9617fbc-5b2c-4e57-a926-b1b3a0953e58"}]: dispatch 2024-03-18T22:37:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:01 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9617fbc-5b2c-4e57-a926-b1b3a0953e58"}]': finished 2024-03-18T22:37:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:01 smithi113 ceph-mon[66775]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T22:37:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:01 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T22:37:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:01 smithi113 ceph-mon[66775]: from='client.? 172.21.15.113:0/3955968865' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:37:03.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:02 smithi103 ceph-mon[66377]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:03.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:02 smithi040 ceph-mon[62643]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:03.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:02 smithi113 ceph-mon[66775]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:05.023 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:04 smithi113 ceph-mon[66775]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:05.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:04 smithi103 ceph-mon[66377]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:05.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:04 smithi040 ceph-mon[62643]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:07.106 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:06 smithi113 ceph-mon[66775]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:07.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:06 smithi103 ceph-mon[66377]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:07.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:06 smithi040 ceph-mon[62643]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:08.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:08 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T22:37:08.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:08 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:08.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:08 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T22:37:08.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:08 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:08.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:08 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T22:37:08.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:08 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:09.582 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:09 smithi113 ceph-mon[66775]: Deploying daemon osd.5 on smithi113 2024-03-18T22:37:09.582 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:09 smithi113 ceph-mon[66775]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:09.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:09 smithi040 ceph-mon[62643]: Deploying daemon osd.5 on smithi113 2024-03-18T22:37:09.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:09 smithi040 ceph-mon[62643]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:09.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:09 smithi103 ceph-mon[66377]: Deploying daemon osd.5 on smithi113 2024-03-18T22:37:09.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:09 smithi103 ceph-mon[66377]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:11.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:10 smithi103 ceph-mon[66377]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:11.135 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:10 smithi113 ceph-mon[66775]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:11.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:10 smithi040 ceph-mon[62643]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:13.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:12 smithi103 ceph-mon[66377]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:13.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:12 smithi040 ceph-mon[62643]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:13.160 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:12 smithi113 ceph-mon[66775]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:15.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:14 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:15.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:14 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:15.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:14 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:37:15.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:14 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:15.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:14 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:15.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:14 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:37:15.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:14 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:15.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:14 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:15.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:14 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:37:16.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:15 smithi113 ceph-mon[66775]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:16.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:15 smithi103 ceph-mon[66377]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:16.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:15 smithi040 ceph-mon[62643]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:17.625 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:37:17.623+0000 7f482effd700 1 -- 172.21.15.113:0/2663546336 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4838062750 con 0x7f48240a2890 2024-03-18T22:37:17.625 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 5 on host 'smithi113' 2024-03-18T22:37:17.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:37:17.626+0000 7f483fe7e700 1 -- 172.21.15.113:0/2663546336 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f48240a2890 msgr2=0x7f48240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:37:17.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:37:17.626+0000 7f483fe7e700 1 --2- 172.21.15.113:0/2663546336 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f48240a2890 0x7f48240a4d50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f482800b5c0 tx=0x7f4828005fb0 comp rx=0 tx=0).stop 2024-03-18T22:37:17.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:37:17.626+0000 7f483fe7e700 1 -- 172.21.15.113:0/2663546336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4838101240 msgr2=0x7f483806ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:37:17.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:37:17.626+0000 7f483fe7e700 1 --2- 172.21.15.113:0/2663546336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4838101240 0x7f483806ab20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f483400d980 tx=0x7f483400dc90 comp rx=0 tx=0).stop 2024-03-18T22:37:17.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:37:17.626+0000 7f483fe7e700 1 -- 172.21.15.113:0/2663546336 shutdown_connections 2024-03-18T22:37:17.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:37:17.626+0000 7f483fe7e700 1 --2- 172.21.15.113:0/2663546336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4838102de0 0x7f483819f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:17.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:37:17.626+0000 7f483fe7e700 1 --2- 172.21.15.113:0/2663546336 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4838101240 0x7f483806ab20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:17.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:37:17.626+0000 7f483fe7e700 1 --2- 172.21.15.113:0/2663546336 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f48240a2890 0x7f48240a4d50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:17.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:37:17.626+0000 7f483fe7e700 1 --2- 172.21.15.113:0/2663546336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4838102440 0x7f483819d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:17.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:37:17.626+0000 7f483fe7e700 1 -- 172.21.15.113:0/2663546336 >> 172.21.15.113:0/2663546336 conn(0x7f48380fc7e0 msgr2=0x7f4838105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:37:17.627 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:37:17.626+0000 7f483fe7e700 1 -- 172.21.15.113:0/2663546336 shutdown_connections 2024-03-18T22:37:17.628 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:37:17.626+0000 7f483fe7e700 1 -- 172.21.15.113:0/2663546336 wait complete. 2024-03-18T22:37:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:17 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:17 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:17 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:17 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:37:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:17 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:17 smithi113 ceph-mon[66775]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:17 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:17 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:17 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:37:17.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:17 smithi113 ceph-mon[66775]: from='osd.5 [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T22:37:17.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:17 smithi113 ceph-mon[66775]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T22:37:18.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:17 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:18.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:17 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:18.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:17 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:18.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:17 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:37:18.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:17 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:18.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:17 smithi103 ceph-mon[66377]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:18.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:17 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:18.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:17 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:18.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:17 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:37:18.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:17 smithi103 ceph-mon[66377]: from='osd.5 [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T22:37:18.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:17 smithi103 ceph-mon[66377]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T22:37:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:37:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:17 smithi040 ceph-mon[62643]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:37:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:17 smithi040 ceph-mon[62643]: from='osd.5 [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T22:37:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:17 smithi040 ceph-mon[62643]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T22:37:18.233 DEBUG:teuthology.orchestra.run.smithi113:osd.5> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.5.service 2024-03-18T22:37:18.235 INFO:tasks.cephadm:Deploying osd.6 on smithi040 with /dev/vg_nvme/lv_2... 2024-03-18T22:37:18.235 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T22:37:18.489 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:37:19.605 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:19 smithi113 ceph-mon[66775]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T22:37:19.605 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:19 smithi113 ceph-mon[66775]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T22:37:19.606 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:19 smithi113 ceph-mon[66775]: from='osd.5 [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:37:19.606 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:19 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T22:37:19.606 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:19 smithi113 ceph-mon[66775]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:37:19.606 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:19 smithi113 ceph-mon[66775]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:19.606 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 22:37:19 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-5[78843]: 2024-03-18T22:37:19.598+0000 7f452d9af700 -1 osd.5 0 waiting for initial osdmap 2024-03-18T22:37:19.606 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 22:37:19 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-5[78843]: 2024-03-18T22:37:19.604+0000 7f45297a2700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T22:37:19.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:19 smithi103 ceph-mon[66377]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T22:37:19.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:19 smithi103 ceph-mon[66377]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T22:37:19.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:19 smithi103 ceph-mon[66377]: from='osd.5 [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:37:19.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:19 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T22:37:19.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:19 smithi103 ceph-mon[66377]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:37:19.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:19 smithi103 ceph-mon[66377]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:19.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:19 smithi040 ceph-mon[62643]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T22:37:19.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:19 smithi040 ceph-mon[62643]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T22:37:19.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:19 smithi040 ceph-mon[62643]: from='osd.5 [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:37:19.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:19 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T22:37:19.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:19 smithi040 ceph-mon[62643]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:37:19.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:19 smithi040 ceph-mon[62643]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T22:37:20.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:20 smithi040 ceph-mon[62643]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-18T22:37:20.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:20 smithi040 ceph-mon[62643]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T22:37:20.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T22:37:20.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T22:37:20.807 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:20 smithi113 ceph-mon[66775]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-18T22:37:20.807 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:20 smithi113 ceph-mon[66775]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T22:37:20.807 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T22:37:20.807 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T22:37:20.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:20 smithi103 ceph-mon[66377]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-18T22:37:20.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:20 smithi103 ceph-mon[66377]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T22:37:20.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T22:37:20.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T22:37:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T22:37:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T22:37:20.873 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-03-18T22:37:20.873 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-03-18T22:37:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126381 s, 830 MB/s 2024-03-18T22:37:20.874 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 2024-03-18T22:37:20.874 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-03-18T22:37:21.486 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi040:vg_nvme/lv_2 2024-03-18T22:37:21.732 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:37:21.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:21 smithi040 ceph-mon[62643]: purged_snaps scrub starts 2024-03-18T22:37:21.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:21 smithi040 ceph-mon[62643]: purged_snaps scrub ok 2024-03-18T22:37:21.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:21 smithi040 ceph-mon[62643]: osd.5 [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324] boot 2024-03-18T22:37:21.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:21 smithi040 ceph-mon[62643]: osdmap e38: 6 total, 6 up, 6 in 2024-03-18T22:37:21.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:21 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T22:37:21.853 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:21 smithi040 ceph-mon[62643]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:21.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:21 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:21.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:21 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:21.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:21 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:37:21.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:21 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:37:21.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:21 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:21.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:21 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:21.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:21 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:37:21.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:21 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:21.854 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:21 smithi040 ceph-mon[62643]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T22:37:22.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:21 smithi113 ceph-mon[66775]: purged_snaps scrub starts 2024-03-18T22:37:22.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:21 smithi113 ceph-mon[66775]: purged_snaps scrub ok 2024-03-18T22:37:22.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:21 smithi113 ceph-mon[66775]: osd.5 [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324] boot 2024-03-18T22:37:22.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:21 smithi113 ceph-mon[66775]: osdmap e38: 6 total, 6 up, 6 in 2024-03-18T22:37:22.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:21 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T22:37:22.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:21 smithi113 ceph-mon[66775]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:22.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:21 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:22.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:21 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:22.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:21 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:37:22.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:21 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:37:22.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:21 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:22.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:21 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:22.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:21 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:37:22.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:21 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:22.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:21 smithi113 ceph-mon[66775]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T22:37:22.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:21 smithi103 ceph-mon[66377]: purged_snaps scrub starts 2024-03-18T22:37:22.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:21 smithi103 ceph-mon[66377]: purged_snaps scrub ok 2024-03-18T22:37:22.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:21 smithi103 ceph-mon[66377]: osd.5 [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324] boot 2024-03-18T22:37:22.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:21 smithi103 ceph-mon[66377]: osdmap e38: 6 total, 6 up, 6 in 2024-03-18T22:37:22.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:21 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T22:37:22.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:21 smithi103 ceph-mon[66377]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:22.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:21 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:22.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:21 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:22.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:21 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:37:22.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:21 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:37:22.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:21 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:22.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:21 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:22.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:21 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:37:22.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:21 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:22.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:21 smithi103 ceph-mon[66377]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T22:37:22.842 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:22 smithi040 ceph-mon[62643]: Detected new or changed devices on smithi113 2024-03-18T22:37:22.842 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:22 smithi040 ceph-mon[62643]: Adjusting osd_memory_target on smithi113 to 8544M 2024-03-18T22:37:23.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:22 smithi113 ceph-mon[66775]: Detected new or changed devices on smithi113 2024-03-18T22:37:23.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:22 smithi113 ceph-mon[66775]: Adjusting osd_memory_target on smithi113 to 8544M 2024-03-18T22:37:23.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:22 smithi103 ceph-mon[66377]: Detected new or changed devices on smithi113 2024-03-18T22:37:23.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:22 smithi103 ceph-mon[66377]: Adjusting osd_memory_target on smithi113 to 8544M 2024-03-18T22:37:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.385+0000 7f7d815ab700 1 -- 172.21.15.40:0/1839942527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d7c103540 msgr2=0x7f7d7c1039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:37:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.385+0000 7f7d815ab700 1 --2- 172.21.15.40:0/1839942527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d7c103540 0x7f7d7c1039a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7d640099c0 tx=0x7f7d64009cd0 comp rx=0 tx=0).stop 2024-03-18T22:37:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.385+0000 7f7d815ab700 1 -- 172.21.15.40:0/1839942527 shutdown_connections 2024-03-18T22:37:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.385+0000 7f7d815ab700 1 --2- 172.21.15.40:0/1839942527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d7c103ee0 0x7f7d7c10a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.385+0000 7f7d815ab700 1 --2- 172.21.15.40:0/1839942527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d7c102340 0x7f7d7c102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.385+0000 7f7d815ab700 1 --2- 172.21.15.40:0/1839942527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d7c103540 0x7f7d7c1039a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.385+0000 7f7d815ab700 1 -- 172.21.15.40:0/1839942527 >> 172.21.15.40:0/1839942527 conn(0x7f7d7c0fd920 msgr2=0x7f7d7c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:37:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.385+0000 7f7d815ab700 1 -- 172.21.15.40:0/1839942527 shutdown_connections 2024-03-18T22:37:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.385+0000 7f7d815ab700 1 -- 172.21.15.40:0/1839942527 wait complete. 2024-03-18T22:37:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.386+0000 7f7d815ab700 1 Processor -- start 2024-03-18T22:37:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.386+0000 7f7d815ab700 1 -- start start 2024-03-18T22:37:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.386+0000 7f7d815ab700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d7c102340 0x7f7d7c1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:37:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.386+0000 7f7d815ab700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d7c103540 0x7f7d7c198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:37:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.386+0000 7f7d815ab700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d7c103ee0 0x7f7d7c19fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:37:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.386+0000 7f7d815ab700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7c10ded0 con 0x7f7d7c103540 2024-03-18T22:37:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.386+0000 7f7d815ab700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7c10dd50 con 0x7f7d7c102340 2024-03-18T22:37:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.386+0000 7f7d815ab700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7c10e050 con 0x7f7d7c103ee0 2024-03-18T22:37:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.386+0000 7f7d7a7fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d7c103540 0x7f7d7c198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:37:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.386+0000 7f7d7a7fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d7c103540 0x7f7d7c198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48548/0 (socket says 172.21.15.40:48548) 2024-03-18T22:37:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.386+0000 7f7d7a7fc700 1 -- 172.21.15.40:0/3007651124 learned_addr learned my addr 172.21.15.40:0/3007651124 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:37:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.386+0000 7f7d7b7fe700 1 --2- 172.21.15.40:0/3007651124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d7c103ee0 0x7f7d7c19fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:37:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.386+0000 7f7d7affd700 1 --2- 172.21.15.40:0/3007651124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d7c102340 0x7f7d7c1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:37:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.387+0000 7f7d7a7fc700 1 -- 172.21.15.40:0/3007651124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d7c103ee0 msgr2=0x7f7d7c19fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:37:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.387+0000 7f7d7a7fc700 1 --2- 172.21.15.40:0/3007651124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d7c103ee0 0x7f7d7c19fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.387+0000 7f7d7a7fc700 1 -- 172.21.15.40:0/3007651124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d7c102340 msgr2=0x7f7d7c1985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:37:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.387+0000 7f7d7a7fc700 1 --2- 172.21.15.40:0/3007651124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d7c102340 0x7f7d7c1985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.387+0000 7f7d7a7fc700 1 -- 172.21.15.40:0/3007651124 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d7c1a0230 con 0x7f7d7c103540 2024-03-18T22:37:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.387+0000 7f7d7a7fc700 1 --2- 172.21.15.40:0/3007651124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d7c103540 0x7f7d7c198b10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7d64000c00 tx=0x7f7d640047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:37:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.387+0000 7f7d73fff700 1 -- 172.21.15.40:0/3007651124 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d64027070 con 0x7f7d7c103540 2024-03-18T22:37:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.387+0000 7f7d815ab700 1 -- 172.21.15.40:0/3007651124 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d7c1a04c0 con 0x7f7d7c103540 2024-03-18T22:37:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.387+0000 7f7d73fff700 1 -- 172.21.15.40:0/3007651124 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d640040d0 con 0x7f7d7c103540 2024-03-18T22:37:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.387+0000 7f7d73fff700 1 -- 172.21.15.40:0/3007651124 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6400f460 con 0x7f7d7c103540 2024-03-18T22:37:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.387+0000 7f7d815ab700 1 -- 172.21.15.40:0/3007651124 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d7c1a09f0 con 0x7f7d7c103540 2024-03-18T22:37:23.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.388+0000 7f7d815ab700 1 -- 172.21.15.40:0/3007651124 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d7c068fd0 con 0x7f7d7c103540 2024-03-18T22:37:23.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.389+0000 7f7d73fff700 1 -- 172.21.15.40:0/3007651124 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7d6402a070 con 0x7f7d7c103540 2024-03-18T22:37:23.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.390+0000 7f7d73fff700 1 --2- 172.21.15.40:0/3007651124 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7d680a2910 0x7f7d680a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:37:23.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.390+0000 7f7d7affd700 1 --2- 172.21.15.40:0/3007651124 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7d680a2910 0x7f7d680a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:37:23.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.390+0000 7f7d73fff700 1 -- 172.21.15.40:0/3007651124 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4533+0+0 (secure 0 0 0) 0x7f7d640cf1a0 con 0x7f7d7c103540 2024-03-18T22:37:23.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.390+0000 7f7d7affd700 1 --2- 172.21.15.40:0/3007651124 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7d680a2910 0x7f7d680a4dd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c1033a0 tx=0x7f7d6c009190 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:37:23.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.394+0000 7f7d73fff700 1 -- 172.21.15.40:0/3007651124 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7d64031070 con 0x7f7d7c103540 2024-03-18T22:37:23.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:23.613+0000 7f7d815ab700 1 -- 172.21.15.40:0/3007651124 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7d7c062750 con 0x7f7d680a2910 2024-03-18T22:37:23.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:23 smithi040 ceph-mon[62643]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T22:37:23.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:23 smithi040 ceph-mon[62643]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:23.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:23 smithi040 ceph-mon[62643]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T22:37:23.981 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:23 smithi113 ceph-mon[66775]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T22:37:23.981 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:23 smithi113 ceph-mon[66775]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:23.981 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:23 smithi113 ceph-mon[66775]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T22:37:24.010 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:23 smithi103 ceph-mon[66377]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T22:37:24.010 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:23 smithi103 ceph-mon[66377]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:24.011 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:23 smithi103 ceph-mon[66377]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T22:37:24.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:24 smithi040 ceph-mon[62643]: from='client.14340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:37:24.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:37:24.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:37:24.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:24.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:24 smithi040 ceph-mon[62643]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T22:37:25.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:24 smithi113 ceph-mon[66775]: from='client.14340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:37:25.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:37:25.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:37:25.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:25.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:24 smithi113 ceph-mon[66775]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T22:37:25.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:24 smithi103 ceph-mon[66377]: from='client.14340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:37:25.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:37:25.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:37:25.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:25.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:24 smithi103 ceph-mon[66377]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T22:37:25.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:25 smithi040 ceph-mon[62643]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T22:37:26.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:25 smithi113 ceph-mon[66775]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T22:37:26.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:25 smithi103 ceph-mon[66377]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T22:37:26.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:26 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3101547899' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "42df7e48-779a-4aee-bf6b-8c9fc8627d99"}]: dispatch 2024-03-18T22:37:26.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:26 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3101547899' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "42df7e48-779a-4aee-bf6b-8c9fc8627d99"}]': finished 2024-03-18T22:37:26.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:26 smithi040 ceph-mon[62643]: osdmap e43: 7 total, 6 up, 7 in 2024-03-18T22:37:26.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:27.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:26 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3101547899' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "42df7e48-779a-4aee-bf6b-8c9fc8627d99"}]: dispatch 2024-03-18T22:37:27.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:26 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3101547899' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "42df7e48-779a-4aee-bf6b-8c9fc8627d99"}]': finished 2024-03-18T22:37:27.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:26 smithi113 ceph-mon[66775]: osdmap e43: 7 total, 6 up, 7 in 2024-03-18T22:37:27.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:27.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:26 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3101547899' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "42df7e48-779a-4aee-bf6b-8c9fc8627d99"}]: dispatch 2024-03-18T22:37:27.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:26 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3101547899' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "42df7e48-779a-4aee-bf6b-8c9fc8627d99"}]': finished 2024-03-18T22:37:27.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:26 smithi103 ceph-mon[66377]: osdmap e43: 7 total, 6 up, 7 in 2024-03-18T22:37:27.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:27.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:27 smithi040 ceph-mon[62643]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-03-18T22:37:27.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:27 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1181929082' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:37:28.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:27 smithi113 ceph-mon[66775]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-03-18T22:37:28.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:27 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1181929082' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:37:28.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:27 smithi103 ceph-mon[66377]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-03-18T22:37:28.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:27 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1181929082' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:37:29.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:28 smithi103 ceph-mon[66377]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T22:37:29.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:28 smithi040 ceph-mon[62643]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T22:37:29.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:28 smithi113 ceph-mon[66775]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T22:37:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:30 smithi103 ceph-mon[66377]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T22:37:31.118 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:30 smithi040 ceph-mon[62643]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T22:37:31.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:30 smithi113 ceph-mon[66775]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T22:37:32.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:32 smithi040 ceph-mon[62643]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-18T22:37:33.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:32 smithi103 ceph-mon[66377]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-18T22:37:33.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:32 smithi113 ceph-mon[66775]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-18T22:37:34.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:33 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T22:37:34.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:33 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:34.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:33 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T22:37:34.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:33 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:34.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:33 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T22:37:34.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:33 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:35.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:34 smithi113 ceph-mon[66775]: Deploying daemon osd.6 on smithi040 2024-03-18T22:37:35.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:34 smithi113 ceph-mon[66775]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:35.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:34 smithi103 ceph-mon[66377]: Deploying daemon osd.6 on smithi040 2024-03-18T22:37:35.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:34 smithi103 ceph-mon[66377]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:35.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:34 smithi040 ceph-mon[62643]: Deploying daemon osd.6 on smithi040 2024-03-18T22:37:35.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:34 smithi040 ceph-mon[62643]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:37.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:36 smithi103 ceph-mon[66377]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:37.135 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:36 smithi040 ceph-mon[62643]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:37.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:36 smithi113 ceph-mon[66775]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:39.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:38 smithi103 ceph-mon[66377]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:39.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:38 smithi040 ceph-mon[62643]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:39.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:38 smithi113 ceph-mon[66775]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:40.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:39 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:37:40.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:39 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:40.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:39 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:40.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:39 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:37:40.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:39 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:40.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:39 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:40.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:39 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:37:40.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:39 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:40.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:39 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:41.008 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:40 smithi040 ceph-mon[62643]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:41.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:40 smithi103 ceph-mon[66377]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:41.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:40 smithi113 ceph-mon[66775]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:43.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:43.233+0000 7f7d73fff700 1 -- 172.21.15.40:0/3007651124 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7d7c062750 con 0x7f7d680a2910 2024-03-18T22:37:43.235 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 6 on host 'smithi040' 2024-03-18T22:37:43.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:43.235+0000 7f7d815ab700 1 -- 172.21.15.40:0/3007651124 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7d680a2910 msgr2=0x7f7d680a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:37:43.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:43.235+0000 7f7d815ab700 1 --2- 172.21.15.40:0/3007651124 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7d680a2910 0x7f7d680a4dd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c1033a0 tx=0x7f7d6c009190 comp rx=0 tx=0).stop 2024-03-18T22:37:43.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:43.235+0000 7f7d815ab700 1 -- 172.21.15.40:0/3007651124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d7c103540 msgr2=0x7f7d7c198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:37:43.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:43.235+0000 7f7d815ab700 1 --2- 172.21.15.40:0/3007651124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d7c103540 0x7f7d7c198b10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7d64000c00 tx=0x7f7d640047b0 comp rx=0 tx=0).stop 2024-03-18T22:37:43.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:43.235+0000 7f7d815ab700 1 -- 172.21.15.40:0/3007651124 shutdown_connections 2024-03-18T22:37:43.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:43.235+0000 7f7d815ab700 1 --2- 172.21.15.40:0/3007651124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d7c103ee0 0x7f7d7c19fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:43.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:43.235+0000 7f7d815ab700 1 --2- 172.21.15.40:0/3007651124 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7d680a2910 0x7f7d680a4dd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:43.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:43.235+0000 7f7d815ab700 1 --2- 172.21.15.40:0/3007651124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d7c102340 0x7f7d7c1985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:43.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:43.235+0000 7f7d815ab700 1 --2- 172.21.15.40:0/3007651124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d7c103540 0x7f7d7c198b10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:43.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:43.235+0000 7f7d815ab700 1 -- 172.21.15.40:0/3007651124 >> 172.21.15.40:0/3007651124 conn(0x7f7d7c0fd920 msgr2=0x7f7d7c106a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:37:43.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:43.235+0000 7f7d815ab700 1 -- 172.21.15.40:0/3007651124 shutdown_connections 2024-03-18T22:37:43.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:37:43.235+0000 7f7d815ab700 1 -- 172.21.15.40:0/3007651124 wait complete. 2024-03-18T22:37:43.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:43 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:43.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:43 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:43.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:43 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:43.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:43 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:37:43.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:43 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:43.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:43 smithi040 ceph-mon[62643]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:43.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:43 smithi040 ceph-mon[62643]: from='osd.6 [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T22:37:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:43 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:43 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:43 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:43 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:37:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:43 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:43 smithi113 ceph-mon[66775]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:43 smithi113 ceph-mon[66775]: from='osd.6 [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T22:37:43.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:43.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:43.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:43.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:37:43.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:43.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:43 smithi103 ceph-mon[66377]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:43.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:43 smithi103 ceph-mon[66377]: from='osd.6 [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T22:37:43.859 DEBUG:teuthology.orchestra.run.smithi040:osd.6> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.6.service 2024-03-18T22:37:43.861 INFO:tasks.cephadm:Deploying osd.7 on smithi103 with /dev/vg_nvme/lv_2... 2024-03-18T22:37:43.861 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T22:37:44.112 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T22:37:44.435 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:44 smithi103 ceph-mon[66377]: from='osd.6 [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T22:37:44.435 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:44 smithi103 ceph-mon[66377]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T22:37:44.435 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:44 smithi103 ceph-mon[66377]: from='osd.6 [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-18T22:37:44.435 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:44.436 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:44.436 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:44.436 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:37:44.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:44 smithi113 ceph-mon[66775]: from='osd.6 [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T22:37:44.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:44 smithi113 ceph-mon[66775]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T22:37:44.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:44 smithi113 ceph-mon[66775]: from='osd.6 [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-18T22:37:44.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:44.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:44.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:44.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:37:44.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:44 smithi040 ceph-mon[62643]: from='osd.6 [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T22:37:44.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:44 smithi040 ceph-mon[62643]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T22:37:44.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:44 smithi040 ceph-mon[62643]: from='osd.6 [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-18T22:37:44.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:44.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:44.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:44.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:37:44.644 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 22:37:44 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-6[87962]: 2024-03-18T22:37:44.179+0000 7f740af83700 -1 osd.6 0 waiting for initial osdmap 2024-03-18T22:37:44.644 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 22:37:44 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-6[87962]: 2024-03-18T22:37:44.187+0000 7f7406d76700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T22:37:45.331 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:45 smithi040 ceph-mon[62643]: from='osd.6 [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-18T22:37:45.331 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:45 smithi040 ceph-mon[62643]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T22:37:45.332 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:45 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:45.332 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:45 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:45.332 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:45 smithi040 ceph-mon[62643]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:45.332 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:45 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:45.449 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:45 smithi103 ceph-mon[66377]: from='osd.6 [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-18T22:37:45.449 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:45 smithi103 ceph-mon[66377]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T22:37:45.449 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:45 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:45.449 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:45 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:45.449 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:45 smithi103 ceph-mon[66377]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:45.450 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:45 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:45.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:45 smithi113 ceph-mon[66775]: from='osd.6 [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-18T22:37:45.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:45 smithi113 ceph-mon[66775]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T22:37:45.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:45 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:45.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:45 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:45.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:45 smithi113 ceph-mon[66775]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-18T22:37:45.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:45 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:46.133 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T22:37:46.133 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T22:37:46.133 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-03-18T22:37:46.133 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-03-18T22:37:46.133 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111439 s, 941 MB/s 2024-03-18T22:37:46.133 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-03-18T22:37:46.430 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:46 smithi103 ceph-mon[66377]: purged_snaps scrub starts 2024-03-18T22:37:46.431 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:46 smithi103 ceph-mon[66377]: purged_snaps scrub ok 2024-03-18T22:37:46.431 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:46 smithi103 ceph-mon[66377]: osd.6 [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763] boot 2024-03-18T22:37:46.431 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:46 smithi103 ceph-mon[66377]: osdmap e46: 7 total, 7 up, 7 in 2024-03-18T22:37:46.431 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:46 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:46.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:46 smithi113 ceph-mon[66775]: purged_snaps scrub starts 2024-03-18T22:37:46.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:46 smithi113 ceph-mon[66775]: purged_snaps scrub ok 2024-03-18T22:37:46.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:46 smithi113 ceph-mon[66775]: osd.6 [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763] boot 2024-03-18T22:37:46.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:46 smithi113 ceph-mon[66775]: osdmap e46: 7 total, 7 up, 7 in 2024-03-18T22:37:46.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:46 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:46.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:46 smithi040 ceph-mon[62643]: purged_snaps scrub starts 2024-03-18T22:37:46.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:46 smithi040 ceph-mon[62643]: purged_snaps scrub ok 2024-03-18T22:37:46.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:46 smithi040 ceph-mon[62643]: osd.6 [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763] boot 2024-03-18T22:37:46.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:46 smithi040 ceph-mon[62643]: osdmap e46: 7 total, 7 up, 7 in 2024-03-18T22:37:46.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:46 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T22:37:46.859 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi103:vg_nvme/lv_2 2024-03-18T22:37:47.100 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T22:37:47.450 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:47 smithi103 ceph-mon[66377]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T22:37:47.450 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:47 smithi103 ceph-mon[66377]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:47.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:47 smithi113 ceph-mon[66775]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T22:37:47.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:47 smithi113 ceph-mon[66775]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:47.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:47 smithi040 ceph-mon[62643]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T22:37:47.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:47 smithi040 ceph-mon[62643]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:48.486 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:48 smithi103 ceph-mon[66377]: Detected new or changed devices on smithi040 2024-03-18T22:37:48.486 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:48.486 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:48.486 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:37:48.486 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:37:48.486 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:37:48.486 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:48 smithi103 ceph-mon[66377]: Adjusting osd_memory_target on smithi040 to 5697M 2024-03-18T22:37:48.486 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:48.486 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:48.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:37:48.487 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:48 smithi113 ceph-mon[66775]: Detected new or changed devices on smithi040 2024-03-18T22:37:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:37:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:37:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:37:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:48 smithi113 ceph-mon[66775]: Adjusting osd_memory_target on smithi040 to 5697M 2024-03-18T22:37:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:37:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:48.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:48 smithi040 ceph-mon[62643]: Detected new or changed devices on smithi040 2024-03-18T22:37:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:37:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:37:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:37:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:48 smithi040 ceph-mon[62643]: Adjusting osd_memory_target on smithi040 to 5697M 2024-03-18T22:37:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:37:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:37:48.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.704+0000 7ff9d536a700 1 -- 172.21.15.103:0/101072890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d0102430 msgr2=0x7ff9d0102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:37:48.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.704+0000 7ff9d536a700 1 --2- 172.21.15.103:0/101072890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d0102430 0x7ff9d0102890 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff9c40099c0 tx=0x7ff9c4009cd0 comp rx=0 tx=0).stop 2024-03-18T22:37:48.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.704+0000 7ff9d536a700 1 -- 172.21.15.103:0/101072890 shutdown_connections 2024-03-18T22:37:48.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.704+0000 7ff9d536a700 1 --2- 172.21.15.103:0/101072890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9d0102dd0 0x7ff9d0109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:48.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.704+0000 7ff9d536a700 1 --2- 172.21.15.103:0/101072890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9d0101230 0x7ff9d0101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:48.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.704+0000 7ff9d536a700 1 --2- 172.21.15.103:0/101072890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d0102430 0x7ff9d0102890 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:48.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.704+0000 7ff9d536a700 1 -- 172.21.15.103:0/101072890 >> 172.21.15.103:0/101072890 conn(0x7ff9d00fc7d0 msgr2=0x7ff9d00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:37:48.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.704+0000 7ff9d536a700 1 -- 172.21.15.103:0/101072890 shutdown_connections 2024-03-18T22:37:48.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.704+0000 7ff9d536a700 1 -- 172.21.15.103:0/101072890 wait complete. 2024-03-18T22:37:48.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.705+0000 7ff9d536a700 1 Processor -- start 2024-03-18T22:37:48.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.705+0000 7ff9d536a700 1 -- start start 2024-03-18T22:37:48.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.705+0000 7ff9d536a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9d0101230 0x7ff9d01051c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:37:48.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.705+0000 7ff9d536a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d0102430 0x7ff9d0103810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:37:48.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.705+0000 7ff9d536a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9d0102dd0 0x7ff9d0103d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:37:48.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.705+0000 7ff9d536a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d010ccd0 con 0x7ff9d0101230 2024-03-18T22:37:48.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.705+0000 7ff9d536a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d010cb50 con 0x7ff9d0102430 2024-03-18T22:37:48.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.705+0000 7ff9d536a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d010ce50 con 0x7ff9d0102dd0 2024-03-18T22:37:48.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9ce7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d0102430 0x7ff9d0103810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:37:48.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9ce7fc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d0102430 0x7ff9d0103810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50128/0 (socket says 172.21.15.103:50128) 2024-03-18T22:37:48.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9ce7fc700 1 -- 172.21.15.103:0/4131377890 learned_addr learned my addr 172.21.15.103:0/4131377890 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-18T22:37:48.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9cf7fe700 1 --2- 172.21.15.103:0/4131377890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9d0102dd0 0x7ff9d0103d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:37:48.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9ceffd700 1 --2- 172.21.15.103:0/4131377890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9d0101230 0x7ff9d01051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:37:48.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9ce7fc700 1 -- 172.21.15.103:0/4131377890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9d0102dd0 msgr2=0x7ff9d0103d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:37:48.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9ce7fc700 1 --2- 172.21.15.103:0/4131377890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9d0102dd0 0x7ff9d0103d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:48.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9ce7fc700 1 -- 172.21.15.103:0/4131377890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9d0101230 msgr2=0x7ff9d01051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:37:48.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9ce7fc700 1 --2- 172.21.15.103:0/4131377890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9d0101230 0x7ff9d01051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:37:48.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9ce7fc700 1 -- 172.21.15.103:0/4131377890 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d0104600 con 0x7ff9d0102430 2024-03-18T22:37:48.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9ce7fc700 1 --2- 172.21.15.103:0/4131377890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d0102430 0x7ff9d0103810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4006010 tx=0x7ff9c40047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:37:48.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9b7fff700 1 -- 172.21.15.103:0/4131377890 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4027070 con 0x7ff9d0102430 2024-03-18T22:37:48.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9d536a700 1 -- 172.21.15.103:0/4131377890 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9d01a25b0 con 0x7ff9d0102430 2024-03-18T22:37:48.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9b7fff700 1 -- 172.21.15.103:0/4131377890 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9c4004050 con 0x7ff9d0102430 2024-03-18T22:37:48.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9b7fff700 1 -- 172.21.15.103:0/4131377890 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4022e10 con 0x7ff9d0102430 2024-03-18T22:37:48.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.706+0000 7ff9d536a700 1 -- 172.21.15.103:0/4131377890 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9d01a2a00 con 0x7ff9d0102430 2024-03-18T22:37:48.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.707+0000 7ff9d536a700 1 -- 172.21.15.103:0/4131377890 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9d0068fd0 con 0x7ff9d0102430 2024-03-18T22:37:48.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.708+0000 7ff9b7fff700 1 -- 172.21.15.103:0/4131377890 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff9c402a070 con 0x7ff9d0102430 2024-03-18T22:37:48.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.709+0000 7ff9b7fff700 1 --2- 172.21.15.103:0/4131377890 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff9b80a2890 0x7ff9b80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:37:48.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.709+0000 7ff9b7fff700 1 -- 172.21.15.103:0/4131377890 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7ff9c40cf000 con 0x7ff9d0102430 2024-03-18T22:37:48.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.709+0000 7ff9ceffd700 1 --2- 172.21.15.103:0/4131377890 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff9b80a2890 0x7ff9b80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:37:48.711 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.709+0000 7ff9ceffd700 1 --2- 172.21.15.103:0/4131377890 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff9b80a2890 0x7ff9b80a4d50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0005950 tx=0x7ff9c00058e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:37:48.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.712+0000 7ff9b7fff700 1 -- 172.21.15.103:0/4131377890 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff9c4013070 con 0x7ff9d0102430 2024-03-18T22:37:48.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:37:48.915+0000 7ff9d536a700 1 -- 172.21.15.103:0/4131377890 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff9d00fea00 con 0x7ff9b80a2890 2024-03-18T22:37:49.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:49 smithi113 ceph-mon[66775]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:49.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:49 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:37:49.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:49 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:37:49.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:49 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:49.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:49 smithi103 ceph-mon[66377]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:49.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:49 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:37:49.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:49 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:37:49.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:49 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:49.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:49 smithi040 ceph-mon[62643]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:49.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:49 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:37:49.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:49 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:37:49.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:49 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:50.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:50 smithi113 ceph-mon[66775]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:37:50.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:50 smithi103 ceph-mon[66377]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:37:50.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:50 smithi040 ceph-mon[62643]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:37:51.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:51 smithi113 ceph-mon[66775]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:51.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:51 smithi103 ceph-mon[66377]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:51.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:51 smithi040 ceph-mon[62643]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:52.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:52 smithi113 ceph-mon[66775]: from='client.? 172.21.15.103:0/235275459' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f3ce8a7-42f4-4582-9343-e34951f6b3a4"}]: dispatch 2024-03-18T22:37:52.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:52 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f3ce8a7-42f4-4582-9343-e34951f6b3a4"}]: dispatch 2024-03-18T22:37:52.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:52 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f3ce8a7-42f4-4582-9343-e34951f6b3a4"}]': finished 2024-03-18T22:37:52.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:52 smithi113 ceph-mon[66775]: osdmap e48: 8 total, 7 up, 8 in 2024-03-18T22:37:52.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:52 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:37:52.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:52 smithi113 ceph-mon[66775]: from='client.? 172.21.15.103:0/1116954456' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:37:52.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:52 smithi103 ceph-mon[66377]: from='client.? 172.21.15.103:0/235275459' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f3ce8a7-42f4-4582-9343-e34951f6b3a4"}]: dispatch 2024-03-18T22:37:52.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:52 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f3ce8a7-42f4-4582-9343-e34951f6b3a4"}]: dispatch 2024-03-18T22:37:52.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:52 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f3ce8a7-42f4-4582-9343-e34951f6b3a4"}]': finished 2024-03-18T22:37:52.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:52 smithi103 ceph-mon[66377]: osdmap e48: 8 total, 7 up, 8 in 2024-03-18T22:37:52.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:52 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:37:52.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:52 smithi103 ceph-mon[66377]: from='client.? 172.21.15.103:0/1116954456' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:37:52.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:52 smithi040 ceph-mon[62643]: from='client.? 172.21.15.103:0/235275459' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f3ce8a7-42f4-4582-9343-e34951f6b3a4"}]: dispatch 2024-03-18T22:37:52.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:52 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f3ce8a7-42f4-4582-9343-e34951f6b3a4"}]: dispatch 2024-03-18T22:37:52.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:52 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f3ce8a7-42f4-4582-9343-e34951f6b3a4"}]': finished 2024-03-18T22:37:52.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:52 smithi040 ceph-mon[62643]: osdmap e48: 8 total, 7 up, 8 in 2024-03-18T22:37:52.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:52 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:37:52.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:52 smithi040 ceph-mon[62643]: from='client.? 172.21.15.103:0/1116954456' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:37:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:53 smithi113 ceph-mon[66775]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:53.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:53 smithi103 ceph-mon[66377]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:53.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:53 smithi040 ceph-mon[62643]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:55.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:54 smithi113 ceph-mon[66775]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:55.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:54 smithi103 ceph-mon[66377]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:55.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:54 smithi040 ceph-mon[62643]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:56.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:56 smithi103 ceph-mon[66377]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:57.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:56 smithi040 ceph-mon[62643]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:57.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:56 smithi113 ceph-mon[66775]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:59.102 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:58 smithi103 ceph-mon[66377]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:59.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:58 smithi040 ceph-mon[62643]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:59.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:58 smithi113 ceph-mon[66775]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:37:59.988 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T22:37:59.989 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:37:59.989 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:37:59 smithi103 ceph-mon[66377]: Deploying daemon osd.7 on smithi103 2024-03-18T22:38:00.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T22:38:00.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:00.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:37:59 smithi040 ceph-mon[62643]: Deploying daemon osd.7 on smithi103 2024-03-18T22:38:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T22:38:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:37:59 smithi113 ceph-mon[66775]: Deploying daemon osd.7 on smithi103 2024-03-18T22:38:01.027 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:00 smithi103 ceph-mon[66377]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:01.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:00 smithi040 ceph-mon[62643]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:00 smithi113 ceph-mon[66775]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:03.065 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:02 smithi103 ceph-mon[66377]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:03.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:02 smithi040 ceph-mon[62643]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:03.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:02 smithi113 ceph-mon[66775]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:05.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:05 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:05.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:05 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:05.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:05 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:38:05.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:05 smithi103 ceph-mon[66377]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:05.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:05 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:05.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:05 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:05.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:05 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:38:05.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:05 smithi040 ceph-mon[62643]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:06.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:05 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:06.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:05 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:06.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:05 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:38:06.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:05 smithi113 ceph-mon[66775]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:07.105 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:06 smithi040 ceph-mon[62643]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:07.108 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:06 smithi103 ceph-mon[66377]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:07.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:06 smithi113 ceph-mon[66775]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:08.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:38:08.531+0000 7ff9b7fff700 1 -- 172.21.15.103:0/4131377890 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff9d00fea00 con 0x7ff9b80a2890 2024-03-18T22:38:08.533 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 7 on host 'smithi103' 2024-03-18T22:38:08.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:38:08.534+0000 7ff9d536a700 1 -- 172.21.15.103:0/4131377890 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff9b80a2890 msgr2=0x7ff9b80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:38:08.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:38:08.534+0000 7ff9d536a700 1 --2- 172.21.15.103:0/4131377890 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff9b80a2890 0x7ff9b80a4d50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0005950 tx=0x7ff9c00058e0 comp rx=0 tx=0).stop 2024-03-18T22:38:08.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:38:08.534+0000 7ff9d536a700 1 -- 172.21.15.103:0/4131377890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d0102430 msgr2=0x7ff9d0103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:38:08.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:38:08.534+0000 7ff9d536a700 1 --2- 172.21.15.103:0/4131377890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d0102430 0x7ff9d0103810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4006010 tx=0x7ff9c40047b0 comp rx=0 tx=0).stop 2024-03-18T22:38:08.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:38:08.534+0000 7ff9d536a700 1 -- 172.21.15.103:0/4131377890 shutdown_connections 2024-03-18T22:38:08.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:38:08.534+0000 7ff9d536a700 1 --2- 172.21.15.103:0/4131377890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff9d0102dd0 0x7ff9d0103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:08.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:38:08.534+0000 7ff9d536a700 1 --2- 172.21.15.103:0/4131377890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9d0101230 0x7ff9d01051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:08.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:38:08.534+0000 7ff9d536a700 1 --2- 172.21.15.103:0/4131377890 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff9b80a2890 0x7ff9b80a4d50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:08.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:38:08.534+0000 7ff9d536a700 1 --2- 172.21.15.103:0/4131377890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff9d0102430 0x7ff9d0103810 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:08.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:38:08.534+0000 7ff9d536a700 1 -- 172.21.15.103:0/4131377890 >> 172.21.15.103:0/4131377890 conn(0x7ff9d00fc7d0 msgr2=0x7ff9d00fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:38:08.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:38:08.534+0000 7ff9d536a700 1 -- 172.21.15.103:0/4131377890 shutdown_connections 2024-03-18T22:38:08.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:38:08.534+0000 7ff9d536a700 1 -- 172.21.15.103:0/4131377890 wait complete. 2024-03-18T22:38:08.847 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:08 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:08.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:08 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:08.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:08 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:08.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:08 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:38:08.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:08 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:08.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:08 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:08.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:08 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:08.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:08 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:38:08.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:08 smithi103 ceph-mon[66377]: from='osd.7 [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T22:38:08.848 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:08 smithi103 ceph-mon[66377]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T22:38:08.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:08 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:08.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:08 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:08.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:08 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:08.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:08 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:38:08.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:08 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:08.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:08 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:08.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:08 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:08.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:08 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:38:08.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:08 smithi040 ceph-mon[62643]: from='osd.7 [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T22:38:08.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:08 smithi040 ceph-mon[62643]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T22:38:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:08 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:08 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:08 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:08 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:38:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:08 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:08 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:08 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:08 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:38:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:08 smithi113 ceph-mon[66775]: from='osd.7 [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T22:38:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:08 smithi113 ceph-mon[66775]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T22:38:09.185 DEBUG:teuthology.orchestra.run.smithi103:osd.7> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.7.service 2024-03-18T22:38:09.187 INFO:tasks.cephadm:Deploying osd.8 on smithi113 with /dev/vg_nvme/lv_2... 2024-03-18T22:38:09.187 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T22:38:09.435 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:38:09.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:09 smithi113 ceph-mon[66775]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:09.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:09 smithi113 ceph-mon[66775]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T22:38:09.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:09 smithi113 ceph-mon[66775]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T22:38:09.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:09 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:38:09.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:09 smithi113 ceph-mon[66775]: from='osd.7 [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:38:09.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:09 smithi113 ceph-mon[66775]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:38:09.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:09 smithi103 ceph-mon[66377]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:09.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:09 smithi103 ceph-mon[66377]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T22:38:09.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:09 smithi103 ceph-mon[66377]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T22:38:09.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:09 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:38:09.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:09 smithi103 ceph-mon[66377]: from='osd.7 [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:38:09.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:09 smithi103 ceph-mon[66377]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:38:09.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:09 smithi040 ceph-mon[62643]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:09.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:09 smithi040 ceph-mon[62643]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T22:38:09.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:09 smithi040 ceph-mon[62643]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T22:38:09.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:09 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:38:09.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:09 smithi040 ceph-mon[62643]: from='osd.7 [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:38:09.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:09 smithi040 ceph-mon[62643]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:38:10.499 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 22:38:10 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-7[84132]: 2024-03-18T22:38:10.489+0000 7f4a81644700 -1 osd.7 0 waiting for initial osdmap 2024-03-18T22:38:10.500 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 22:38:10 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-7[84132]: 2024-03-18T22:38:10.498+0000 7f4a7d437700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T22:38:11.572 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T22:38:11.572 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T22:38:11.572 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-03-18T22:38:11.572 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-03-18T22:38:11.572 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00853362 s, 1.2 GB/s 2024-03-18T22:38:11.572 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-03-18T22:38:11.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:11 smithi103 ceph-mon[66377]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-18T22:38:11.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:11 smithi103 ceph-mon[66377]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T22:38:11.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:11 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:38:11.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:11 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:38:11.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:11 smithi103 ceph-mon[66377]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:11.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:11 smithi113 ceph-mon[66775]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-18T22:38:11.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:11 smithi113 ceph-mon[66775]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T22:38:11.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:11 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:38:11.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:11 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:38:11.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:11 smithi113 ceph-mon[66775]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:11.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:11 smithi040 ceph-mon[62643]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-18T22:38:11.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:11 smithi040 ceph-mon[62643]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T22:38:11.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:11 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:38:11.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:11 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:38:11.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:11 smithi040 ceph-mon[62643]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-18T22:38:12.227 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi113:vg_nvme/lv_2 2024-03-18T22:38:12.479 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:38:12.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:12 smithi113 ceph-mon[66775]: purged_snaps scrub starts 2024-03-18T22:38:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:12 smithi113 ceph-mon[66775]: purged_snaps scrub ok 2024-03-18T22:38:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:38:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:12 smithi113 ceph-mon[66775]: osd.7 [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261] boot 2024-03-18T22:38:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:12 smithi113 ceph-mon[66775]: osdmap e51: 8 total, 8 up, 8 in 2024-03-18T22:38:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:38:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:12.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:38:12.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:12.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:12 smithi103 ceph-mon[66377]: purged_snaps scrub starts 2024-03-18T22:38:12.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:12 smithi103 ceph-mon[66377]: purged_snaps scrub ok 2024-03-18T22:38:12.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:38:12.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:12 smithi103 ceph-mon[66377]: osd.7 [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261] boot 2024-03-18T22:38:12.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:12 smithi103 ceph-mon[66377]: osdmap e51: 8 total, 8 up, 8 in 2024-03-18T22:38:12.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:38:12.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:12.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:12.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:12.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:12.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:12.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:12.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:12.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:38:12.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:12.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:12 smithi040 ceph-mon[62643]: purged_snaps scrub starts 2024-03-18T22:38:12.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:12 smithi040 ceph-mon[62643]: purged_snaps scrub ok 2024-03-18T22:38:12.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:38:12.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:12 smithi040 ceph-mon[62643]: osd.7 [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261] boot 2024-03-18T22:38:12.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:12 smithi040 ceph-mon[62643]: osdmap e51: 8 total, 8 up, 8 in 2024-03-18T22:38:12.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T22:38:12.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:12.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:12.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:12.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:12.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:12.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:12.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:12.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:38:12.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:13.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:13 smithi113 ceph-mon[66775]: Detected new or changed devices on smithi103 2024-03-18T22:38:13.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:13 smithi113 ceph-mon[66775]: Adjusting osd_memory_target on smithi103 to 5696M 2024-03-18T22:38:13.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:13 smithi113 ceph-mon[66775]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T22:38:13.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:13 smithi113 ceph-mon[66775]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:13.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:13 smithi103 ceph-mon[66377]: Detected new or changed devices on smithi103 2024-03-18T22:38:13.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:13 smithi103 ceph-mon[66377]: Adjusting osd_memory_target on smithi103 to 5696M 2024-03-18T22:38:13.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:13 smithi103 ceph-mon[66377]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T22:38:13.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:13 smithi103 ceph-mon[66377]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:13.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:13 smithi040 ceph-mon[62643]: Detected new or changed devices on smithi103 2024-03-18T22:38:13.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:13 smithi040 ceph-mon[62643]: Adjusting osd_memory_target on smithi103 to 5696M 2024-03-18T22:38:13.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:13 smithi040 ceph-mon[62643]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T22:38:13.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:13 smithi040 ceph-mon[62643]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:14.141 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.139+0000 7f317568a700 1 -- 172.21.15.113:0/84468751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3170102340 msgr2=0x7f3170102760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:38:14.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.139+0000 7f317568a700 1 --2- 172.21.15.113:0/84468751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3170102340 0x7f3170102760 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3160009990 tx=0x7f3160009ca0 comp rx=0 tx=0).stop 2024-03-18T22:38:14.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.139+0000 7f317568a700 1 -- 172.21.15.113:0/84468751 shutdown_connections 2024-03-18T22:38:14.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.139+0000 7f317568a700 1 --2- 172.21.15.113:0/84468751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3170103ee0 0x7f317010a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:14.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.139+0000 7f317568a700 1 --2- 172.21.15.113:0/84468751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3170102340 0x7f3170102760 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:14.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.139+0000 7f317568a700 1 --2- 172.21.15.113:0/84468751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3170103540 0x7f31701039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:14.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.139+0000 7f317568a700 1 -- 172.21.15.113:0/84468751 >> 172.21.15.113:0/84468751 conn(0x7f31700fd920 msgr2=0x7f31700ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:38:14.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.139+0000 7f317568a700 1 -- 172.21.15.113:0/84468751 shutdown_connections 2024-03-18T22:38:14.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.139+0000 7f317568a700 1 -- 172.21.15.113:0/84468751 wait complete. 2024-03-18T22:38:14.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.139+0000 7f317568a700 1 Processor -- start 2024-03-18T22:38:14.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.139+0000 7f317568a700 1 -- start start 2024-03-18T22:38:14.142 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f317568a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3170102340 0x7f31701984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:38:14.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f317568a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3170103540 0x7f3170198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:38:14.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f317568a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3170103ee0 0x7f317019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:38:14.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f317568a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f317010dde0 con 0x7f3170103540 2024-03-18T22:38:14.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f317568a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f317010dc60 con 0x7f3170102340 2024-03-18T22:38:14.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f317568a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f317010df60 con 0x7f3170103ee0 2024-03-18T22:38:14.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f316f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3170103ee0 0x7f317019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:38:14.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f316f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3170103ee0 0x7f317019fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:53952/0 (socket says 172.21.15.113:53952) 2024-03-18T22:38:14.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f316f7fe700 1 -- 172.21.15.113:0/2178233762 learned_addr learned my addr 172.21.15.113:0/2178233762 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-18T22:38:14.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f3167fff700 1 --2- 172.21.15.113:0/2178233762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3170103540 0x7f3170198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:38:14.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f316effd700 1 --2- 172.21.15.113:0/2178233762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3170102340 0x7f31701984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:38:14.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f316f7fe700 1 -- 172.21.15.113:0/2178233762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3170102340 msgr2=0x7f31701984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:38:14.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f316f7fe700 1 --2- 172.21.15.113:0/2178233762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3170102340 0x7f31701984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:14.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f316f7fe700 1 -- 172.21.15.113:0/2178233762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3170103540 msgr2=0x7f3170198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:38:14.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f316f7fe700 1 --2- 172.21.15.113:0/2178233762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3170103540 0x7f3170198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:14.143 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f316f7fe700 1 -- 172.21.15.113:0/2178233762 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31701a0100 con 0x7f3170103ee0 2024-03-18T22:38:14.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f316effd700 1 --2- 172.21.15.113:0/2178233762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3170102340 0x7f31701984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:38:14.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f316f7fe700 1 --2- 172.21.15.113:0/2178233762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3170103ee0 0x7f317019fa60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f316800add0 tx=0x7f316800c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:38:14.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f316cff9700 1 -- 172.21.15.113:0/2178233762 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168017020 con 0x7f3170103ee0 2024-03-18T22:38:14.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f316cff9700 1 -- 172.21.15.113:0/2178233762 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f316800ae00 con 0x7f3170103ee0 2024-03-18T22:38:14.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f317568a700 1 -- 172.21.15.113:0/2178233762 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31701a03f0 con 0x7f3170103ee0 2024-03-18T22:38:14.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f316cff9700 1 -- 172.21.15.113:0/2178233762 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168005350 con 0x7f3170103ee0 2024-03-18T22:38:14.144 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.140+0000 7f317568a700 1 -- 172.21.15.113:0/2178233762 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31701a08e0 con 0x7f3170103ee0 2024-03-18T22:38:14.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.141+0000 7f317568a700 1 -- 172.21.15.113:0/2178233762 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3170068fd0 con 0x7f3170103ee0 2024-03-18T22:38:14.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.142+0000 7f316cff9700 1 -- 172.21.15.113:0/2178233762 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f316800c940 con 0x7f3170103ee0 2024-03-18T22:38:14.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.143+0000 7f316cff9700 1 --2- 172.21.15.113:0/2178233762 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f315c0a2890 0x7f315c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:38:14.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.143+0000 7f316cff9700 1 -- 172.21.15.113:0/2178233762 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f31680cd700 con 0x7f3170103ee0 2024-03-18T22:38:14.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.143+0000 7f316effd700 1 --2- 172.21.15.113:0/2178233762 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f315c0a2890 0x7f315c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:38:14.145 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.144+0000 7f316effd700 1 --2- 172.21.15.113:0/2178233762 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f315c0a2890 0x7f315c0a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3160003920 tx=0x7f3160005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:38:14.148 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.147+0000 7f316cff9700 1 -- 172.21.15.113:0/2178233762 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3168021020 con 0x7f3170103ee0 2024-03-18T22:38:14.355 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:14.354+0000 7f317568a700 1 -- 172.21.15.113:0/2178233762 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3170062750 con 0x7f315c0a2890 2024-03-18T22:38:14.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:14 smithi113 ceph-mon[66775]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T22:38:14.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:14 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:38:14.857 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:14 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:38:14.858 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:14 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:14.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:14 smithi103 ceph-mon[66377]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T22:38:14.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:14 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:38:14.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:14 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:38:14.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:14 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:14.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:14 smithi040 ceph-mon[62643]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T22:38:14.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:14 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:38:14.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:14 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:38:14.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:14 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:15.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:15 smithi113 ceph-mon[66775]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:38:15.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:15 smithi113 ceph-mon[66775]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T22:38:15.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:15 smithi113 ceph-mon[66775]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:15.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:15 smithi103 ceph-mon[66377]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:38:15.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:15 smithi103 ceph-mon[66377]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T22:38:15.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:15 smithi103 ceph-mon[66377]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:15.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:15 smithi040 ceph-mon[62643]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:38:15.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:15 smithi040 ceph-mon[62643]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T22:38:15.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:15 smithi040 ceph-mon[62643]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:16.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:16 smithi113 ceph-mon[66775]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T22:38:16.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:16 smithi103 ceph-mon[66377]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T22:38:16.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:16 smithi040 ceph-mon[62643]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T22:38:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:17 smithi113 ceph-mon[66775]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:17 smithi113 ceph-mon[66775]: from='client.? 172.21.15.113:0/2629964199' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66b85f48-4e20-4563-b495-b50e8fe79c74"}]: dispatch 2024-03-18T22:38:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:17 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66b85f48-4e20-4563-b495-b50e8fe79c74"}]: dispatch 2024-03-18T22:38:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:17 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66b85f48-4e20-4563-b495-b50e8fe79c74"}]': finished 2024-03-18T22:38:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:17 smithi113 ceph-mon[66775]: osdmap e56: 9 total, 8 up, 9 in 2024-03-18T22:38:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:17 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T22:38:17.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:17 smithi103 ceph-mon[66377]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:17.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:17 smithi103 ceph-mon[66377]: from='client.? 172.21.15.113:0/2629964199' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66b85f48-4e20-4563-b495-b50e8fe79c74"}]: dispatch 2024-03-18T22:38:17.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:17 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66b85f48-4e20-4563-b495-b50e8fe79c74"}]: dispatch 2024-03-18T22:38:17.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:17 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66b85f48-4e20-4563-b495-b50e8fe79c74"}]': finished 2024-03-18T22:38:17.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:17 smithi103 ceph-mon[66377]: osdmap e56: 9 total, 8 up, 9 in 2024-03-18T22:38:17.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:17 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T22:38:17.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:17 smithi040 ceph-mon[62643]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:17.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:17 smithi040 ceph-mon[62643]: from='client.? 172.21.15.113:0/2629964199' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66b85f48-4e20-4563-b495-b50e8fe79c74"}]: dispatch 2024-03-18T22:38:17.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:17 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66b85f48-4e20-4563-b495-b50e8fe79c74"}]: dispatch 2024-03-18T22:38:17.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:17 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66b85f48-4e20-4563-b495-b50e8fe79c74"}]': finished 2024-03-18T22:38:17.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:17 smithi040 ceph-mon[62643]: osdmap e56: 9 total, 8 up, 9 in 2024-03-18T22:38:17.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T22:38:18.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:18 smithi113 ceph-mon[66775]: from='client.? 172.21.15.113:0/1179554958' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:38:18.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:18 smithi103 ceph-mon[66377]: from='client.? 172.21.15.113:0/1179554958' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:38:18.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:18 smithi040 ceph-mon[62643]: from='client.? 172.21.15.113:0/1179554958' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:38:19.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:19 smithi113 ceph-mon[66775]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:19.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:19 smithi103 ceph-mon[66377]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:19.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:19 smithi040 ceph-mon[62643]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:21.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:20 smithi113 ceph-mon[66775]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:21.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:20 smithi103 ceph-mon[66377]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:21.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:20 smithi040 ceph-mon[62643]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:22.959 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:22 smithi113 ceph-mon[66775]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:23.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:22 smithi103 ceph-mon[66377]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:23.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:22 smithi040 ceph-mon[62643]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:24.490 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T22:38:24.490 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:24.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T22:38:24.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:24.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T22:38:24.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:25.457 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:25 smithi113 ceph-mon[66775]: Deploying daemon osd.8 on smithi113 2024-03-18T22:38:25.457 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:25 smithi113 ceph-mon[66775]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:25.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:25 smithi103 ceph-mon[66377]: Deploying daemon osd.8 on smithi113 2024-03-18T22:38:25.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:25 smithi103 ceph-mon[66377]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:25.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:25 smithi040 ceph-mon[62643]: Deploying daemon osd.8 on smithi113 2024-03-18T22:38:25.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:25 smithi040 ceph-mon[62643]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:27.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:26 smithi103 ceph-mon[66377]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:27.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:26 smithi040 ceph-mon[62643]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:27.201 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:26 smithi113 ceph-mon[66775]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:29.017 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:28 smithi113 ceph-mon[66775]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:29.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:28 smithi103 ceph-mon[66377]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:29.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:28 smithi040 ceph-mon[62643]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:31.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:30 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:31.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:30 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:31.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:30 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:38:31.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:30 smithi113 ceph-mon[66775]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:31.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:30 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:31.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:30 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:31.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:30 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:38:31.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:30 smithi103 ceph-mon[66377]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:31.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:30 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:31.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:30 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:31.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:30 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:38:31.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:30 smithi040 ceph-mon[62643]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:32 smithi113 ceph-mon[66775]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:33.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:32 smithi103 ceph-mon[66377]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:33.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:32 smithi040 ceph-mon[62643]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:33.477 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:33.476+0000 7f316cff9700 1 -- 172.21.15.113:0/2178233762 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3170062750 con 0x7f315c0a2890 2024-03-18T22:38:33.478 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 8 on host 'smithi113' 2024-03-18T22:38:33.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:33.478+0000 7f317568a700 1 -- 172.21.15.113:0/2178233762 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f315c0a2890 msgr2=0x7f315c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:38:33.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:33.478+0000 7f317568a700 1 --2- 172.21.15.113:0/2178233762 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f315c0a2890 0x7f315c0a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3160003920 tx=0x7f3160005fb0 comp rx=0 tx=0).stop 2024-03-18T22:38:33.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:33.478+0000 7f317568a700 1 -- 172.21.15.113:0/2178233762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3170103ee0 msgr2=0x7f317019fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:38:33.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:33.478+0000 7f317568a700 1 --2- 172.21.15.113:0/2178233762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3170103ee0 0x7f317019fa60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f316800add0 tx=0x7f316800c330 comp rx=0 tx=0).stop 2024-03-18T22:38:33.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:33.478+0000 7f317568a700 1 -- 172.21.15.113:0/2178233762 shutdown_connections 2024-03-18T22:38:33.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:33.478+0000 7f317568a700 1 --2- 172.21.15.113:0/2178233762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3170103ee0 0x7f317019fa60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:33.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:33.478+0000 7f317568a700 1 --2- 172.21.15.113:0/2178233762 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f315c0a2890 0x7f315c0a4d50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:33.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:33.478+0000 7f317568a700 1 --2- 172.21.15.113:0/2178233762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3170102340 0x7f31701984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:33.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:33.478+0000 7f317568a700 1 --2- 172.21.15.113:0/2178233762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3170103540 0x7f3170198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:33.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:33.479+0000 7f317568a700 1 -- 172.21.15.113:0/2178233762 >> 172.21.15.113:0/2178233762 conn(0x7f31700fd920 msgr2=0x7f3170106950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:38:33.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:33.479+0000 7f317568a700 1 -- 172.21.15.113:0/2178233762 shutdown_connections 2024-03-18T22:38:33.480 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:38:33.479+0000 7f317568a700 1 -- 172.21.15.113:0/2178233762 wait complete. 2024-03-18T22:38:34.104 DEBUG:teuthology.orchestra.run.smithi113:osd.8> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.8.service 2024-03-18T22:38:34.107 INFO:tasks.cephadm:Deploying osd.9 on smithi040 with /dev/vg_nvme/lv_1... 2024-03-18T22:38:34.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T22:38:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:38:34.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:34 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:34.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:34 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:34.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:34 smithi040 ceph-mon[62643]: from='osd.8 [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T22:38:34.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:34 smithi040 ceph-mon[62643]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T22:38:34.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:34 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:34.729 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:34 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:34.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:34 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:34.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:34 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:34.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:34 smithi113 ceph-mon[66775]: from='osd.8 [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T22:38:34.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:34 smithi113 ceph-mon[66775]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T22:38:34.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:34 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:34.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:34 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:34.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:34 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:34.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:34 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:34.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:34 smithi103 ceph-mon[66377]: from='osd.8 [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T22:38:34.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:34 smithi103 ceph-mon[66377]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T22:38:34.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:34 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:34.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:34 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:35.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:35 smithi040 ceph-mon[62643]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T22:38:35.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:35 smithi040 ceph-mon[62643]: from='osd.8 [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:38:35.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:35 smithi040 ceph-mon[62643]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T22:38:35.726 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:35 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T22:38:35.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:35 smithi040 ceph-mon[62643]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:38:35.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:35 smithi040 ceph-mon[62643]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:35.778 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:35 smithi113 ceph-mon[66775]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T22:38:35.778 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:35 smithi113 ceph-mon[66775]: from='osd.8 [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:38:35.778 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:35 smithi113 ceph-mon[66775]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T22:38:35.779 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:35 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T22:38:35.779 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:35 smithi113 ceph-mon[66775]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:38:35.779 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:35 smithi113 ceph-mon[66775]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:35.779 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 22:38:35 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-8[85673]: 2024-03-18T22:38:35.440+0000 7f6ac2940700 -1 osd.8 0 waiting for initial osdmap 2024-03-18T22:38:35.779 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 22:38:35 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-8[85673]: 2024-03-18T22:38:35.452+0000 7f6abe733700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T22:38:35.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:35 smithi103 ceph-mon[66377]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T22:38:35.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:35 smithi103 ceph-mon[66377]: from='osd.8 [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:38:35.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:35 smithi103 ceph-mon[66377]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T22:38:35.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:35 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T22:38:35.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:35 smithi103 ceph-mon[66377]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:38:35.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:35 smithi103 ceph-mon[66377]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-03-18T22:38:36.433 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T22:38:36.433 INFO:teuthology.orchestra.run.smithi040.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T22:38:36.433 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 10+0 records in 2024-03-18T22:38:36.433 INFO:teuthology.orchestra.run.smithi040.stderr:10+0 records out 2024-03-18T22:38:36.433 INFO:teuthology.orchestra.run.smithi040.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.01152 s, 910 MB/s 2024-03-18T22:38:36.433 INFO:teuthology.orchestra.run.smithi040.stderr: stderr: 2024-03-18T22:38:36.433 INFO:teuthology.orchestra.run.smithi040.stderr:--> Zapping successful for: 2024-03-18T22:38:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:36 smithi103 ceph-mon[66377]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-18T22:38:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:36 smithi103 ceph-mon[66377]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T22:38:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:36 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T22:38:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:36 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T22:38:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:36 smithi103 ceph-mon[66377]: osd.8 [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708] boot 2024-03-18T22:38:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:36 smithi103 ceph-mon[66377]: osdmap e59: 9 total, 9 up, 9 in 2024-03-18T22:38:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:36 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T22:38:36.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:36 smithi040 ceph-mon[62643]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-18T22:38:36.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:36 smithi040 ceph-mon[62643]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T22:38:36.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:36 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T22:38:36.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:36 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T22:38:36.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:36 smithi040 ceph-mon[62643]: osd.8 [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708] boot 2024-03-18T22:38:36.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:36 smithi040 ceph-mon[62643]: osdmap e59: 9 total, 9 up, 9 in 2024-03-18T22:38:36.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:36 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T22:38:36.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:36 smithi113 ceph-mon[66775]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-18T22:38:36.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:36 smithi113 ceph-mon[66775]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T22:38:36.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:36 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T22:38:36.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:36 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T22:38:36.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:36 smithi113 ceph-mon[66775]: osd.8 [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708] boot 2024-03-18T22:38:36.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:36 smithi113 ceph-mon[66775]: osdmap e59: 9 total, 9 up, 9 in 2024-03-18T22:38:36.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:36 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T22:38:37.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi040:vg_nvme/lv_1 2024-03-18T22:38:37.365 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:38:37.570 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:37 smithi113 ceph-mon[66775]: purged_snaps scrub starts 2024-03-18T22:38:37.570 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:37 smithi113 ceph-mon[66775]: purged_snaps scrub ok 2024-03-18T22:38:37.570 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:37 smithi113 ceph-mon[66775]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:37.570 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:37 smithi113 ceph-mon[66775]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T22:38:37.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:37 smithi040 ceph-mon[62643]: purged_snaps scrub starts 2024-03-18T22:38:37.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:37 smithi040 ceph-mon[62643]: purged_snaps scrub ok 2024-03-18T22:38:37.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:37 smithi040 ceph-mon[62643]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:37.721 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:37 smithi040 ceph-mon[62643]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T22:38:37.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:37 smithi103 ceph-mon[66377]: purged_snaps scrub starts 2024-03-18T22:38:37.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:37 smithi103 ceph-mon[66377]: purged_snaps scrub ok 2024-03-18T22:38:37.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:37 smithi103 ceph-mon[66377]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:37.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:37 smithi103 ceph-mon[66377]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T22:38:38.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: Detected new or changed devices on smithi113 2024-03-18T22:38:38.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:38.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:38.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:38.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:38.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:38.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: Adjusting osd_memory_target on smithi113 to 5696M 2024-03-18T22:38:38.864 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:38.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:38.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:38:38.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:38.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:38:38.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:38.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:38:38.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:38.865 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:38 smithi040 ceph-mon[62643]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T22:38:39.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: Detected new or changed devices on smithi113 2024-03-18T22:38:39.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:39.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:39.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:39.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:39.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:39.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: Adjusting osd_memory_target on smithi113 to 5696M 2024-03-18T22:38:39.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:39.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:39.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:38:39.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:39.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:38:39.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:39.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:38:39.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:39.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:38 smithi113 ceph-mon[66775]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T22:38:39.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.102+0000 7f7da1c84700 1 -- 172.21.15.40:0/4036163258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d9c101510 msgr2=0x7f7d9c101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:38:39.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.102+0000 7f7da1c84700 1 --2- 172.21.15.40:0/4036163258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d9c101510 0x7f7d9c101970 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7d900099c0 tx=0x7f7d90009cd0 comp rx=0 tx=0).stop 2024-03-18T22:38:39.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.102+0000 7f7da1c84700 1 -- 172.21.15.40:0/4036163258 shutdown_connections 2024-03-18T22:38:39.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.102+0000 7f7da1c84700 1 --2- 172.21.15.40:0/4036163258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d9c101eb0 0x7f7d9c109770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:39.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.102+0000 7f7da1c84700 1 --2- 172.21.15.40:0/4036163258 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d9c1027d0 0x7f7d9c102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:39.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.102+0000 7f7da1c84700 1 --2- 172.21.15.40:0/4036163258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d9c101510 0x7f7d9c101970 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:39.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.102+0000 7f7da1c84700 1 -- 172.21.15.40:0/4036163258 >> 172.21.15.40:0/4036163258 conn(0x7f7d9c0fbf80 msgr2=0x7f7d9c0fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:38:39.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.102+0000 7f7da1c84700 1 -- 172.21.15.40:0/4036163258 shutdown_connections 2024-03-18T22:38:39.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.102+0000 7f7da1c84700 1 -- 172.21.15.40:0/4036163258 wait complete. 2024-03-18T22:38:39.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.103+0000 7f7da1c84700 1 Processor -- start 2024-03-18T22:38:39.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.103+0000 7f7da1c84700 1 -- start start 2024-03-18T22:38:39.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.103+0000 7f7da1c84700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d9c101510 0x7f7d9c1052b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:38:39.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.103+0000 7f7da1c84700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d9c101eb0 0x7f7d9c103900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:38:39.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.103+0000 7f7da1c84700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d9c1027d0 0x7f7d9c103e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:38:39.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.103+0000 7f7da1c84700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c10ccb0 con 0x7f7d9c101eb0 2024-03-18T22:38:39.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.103+0000 7f7da1c84700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c10cb30 con 0x7f7d9c1027d0 2024-03-18T22:38:39.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.103+0000 7f7da1c84700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c10ce30 con 0x7f7d9c101510 2024-03-18T22:38:39.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.103+0000 7f7d9affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d9c101eb0 0x7f7d9c103900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:38:39.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.103+0000 7f7d9affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d9c101eb0 0x7f7d9c103900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45484/0 (socket says 172.21.15.40:45484) 2024-03-18T22:38:39.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.103+0000 7f7d9affd700 1 -- 172.21.15.40:0/2083969107 learned_addr learned my addr 172.21.15.40:0/2083969107 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:38:39.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.103+0000 7f7d9bfff700 1 --2- 172.21.15.40:0/2083969107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d9c1027d0 0x7f7d9c103e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:38:39.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.104+0000 7f7d9b7fe700 1 --2- 172.21.15.40:0/2083969107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d9c101510 0x7f7d9c1052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:38:39.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.104+0000 7f7d9affd700 1 -- 172.21.15.40:0/2083969107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d9c101510 msgr2=0x7f7d9c1052b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:38:39.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.104+0000 7f7d9affd700 1 --2- 172.21.15.40:0/2083969107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d9c101510 0x7f7d9c1052b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:39.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.104+0000 7f7d9affd700 1 -- 172.21.15.40:0/2083969107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d9c1027d0 msgr2=0x7f7d9c103e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:38:39.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.104+0000 7f7d9affd700 1 --2- 172.21.15.40:0/2083969107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d9c1027d0 0x7f7d9c103e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:38:39.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.104+0000 7f7d9affd700 1 -- 172.21.15.40:0/2083969107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d9c1046f0 con 0x7f7d9c101eb0 2024-03-18T22:38:39.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.104+0000 7f7d9b7fe700 1 --2- 172.21.15.40:0/2083969107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d9c101510 0x7f7d9c1052b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:38:39.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.104+0000 7f7d9affd700 1 --2- 172.21.15.40:0/2083969107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d9c101eb0 0x7f7d9c103900 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7d90006010 tx=0x7f7d900047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:38:39.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.104+0000 7f7d98ff9700 1 -- 172.21.15.40:0/2083969107 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d90027070 con 0x7f7d9c101eb0 2024-03-18T22:38:39.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.104+0000 7f7d98ff9700 1 -- 172.21.15.40:0/2083969107 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d900040d0 con 0x7f7d9c101eb0 2024-03-18T22:38:39.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.104+0000 7f7d98ff9700 1 -- 172.21.15.40:0/2083969107 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9000f460 con 0x7f7d9c101eb0 2024-03-18T22:38:39.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.104+0000 7f7da1c84700 1 -- 172.21.15.40:0/2083969107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d9c1a25f0 con 0x7f7d9c101eb0 2024-03-18T22:38:39.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.104+0000 7f7da1c84700 1 -- 172.21.15.40:0/2083969107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d9c1a2a40 con 0x7f7d9c101eb0 2024-03-18T22:38:39.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.105+0000 7f7da1c84700 1 -- 172.21.15.40:0/2083969107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d9c068fd0 con 0x7f7d9c101eb0 2024-03-18T22:38:39.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.106+0000 7f7d98ff9700 1 -- 172.21.15.40:0/2083969107 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7d9002a070 con 0x7f7d9c101eb0 2024-03-18T22:38:39.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.107+0000 7f7d98ff9700 1 --2- 172.21.15.40:0/2083969107 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7d840a2910 0x7f7d840a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:38:39.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.107+0000 7f7d9b7fe700 1 --2- 172.21.15.40:0/2083969107 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7d840a2910 0x7f7d840a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:38:39.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.107+0000 7f7d98ff9700 1 -- 172.21.15.40:0/2083969107 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5858+0+0 (secure 0 0 0) 0x7f7d900cf8c0 con 0x7f7d9c101eb0 2024-03-18T22:38:39.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.108+0000 7f7d9b7fe700 1 --2- 172.21.15.40:0/2083969107 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7d840a2910 0x7f7d840a4dd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c005fd0 tx=0x7f7d8c005f00 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:38:39.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: Detected new or changed devices on smithi113 2024-03-18T22:38:39.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:39.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:39.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:39.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:39.111 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:38:39.111 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: Adjusting osd_memory_target on smithi113 to 5696M 2024-03-18T22:38:39.111 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:39.111 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:39.111 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:38:39.111 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:39.111 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:38:39.111 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:39.111 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:38:39.111 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:39.111 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:38 smithi103 ceph-mon[66377]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T22:38:39.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.111+0000 7f7d98ff9700 1 -- 172.21.15.40:0/2083969107 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7d90031070 con 0x7f7d9c101eb0 2024-03-18T22:38:39.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:38:39.326+0000 7f7da1c84700 1 -- 172.21.15.40:0/2083969107 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f7d9c0fe1a0 con 0x7f7d840a2910 2024-03-18T22:38:39.948 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:39 smithi040 ceph-mon[62643]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:39.948 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:39 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:38:39.948 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:39 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:38:39.948 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:39 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:39.949 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:39 smithi040 ceph-mon[62643]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T22:38:40.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:39 smithi113 ceph-mon[66775]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:40.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:39 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:38:40.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:39 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:38:40.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:39 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:40.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:39 smithi113 ceph-mon[66775]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T22:38:40.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:39 smithi103 ceph-mon[66377]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:40.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:39 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:38:40.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:39 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:38:40.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:39 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:40.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:39 smithi103 ceph-mon[66377]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T22:38:40.929 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:40 smithi040 ceph-mon[62643]: from='client.14400 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:38:40.929 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:40 smithi040 ceph-mon[62643]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T22:38:41.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:40 smithi113 ceph-mon[66775]: from='client.14400 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:38:41.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:40 smithi113 ceph-mon[66775]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T22:38:41.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:40 smithi103 ceph-mon[66377]: from='client.14400 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi040:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:38:41.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:40 smithi103 ceph-mon[66377]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T22:38:42.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:41 smithi113 ceph-mon[66775]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:42.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:41 smithi103 ceph-mon[66377]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:42.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:41 smithi040 ceph-mon[62643]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:42.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:42 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/467152635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e46e9424-c131-4559-9a43-55c7cf9c8eeb"}]: dispatch 2024-03-18T22:38:42.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:42 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/467152635' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e46e9424-c131-4559-9a43-55c7cf9c8eeb"}]': finished 2024-03-18T22:38:42.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:42 smithi040 ceph-mon[62643]: osdmap e64: 10 total, 9 up, 10 in 2024-03-18T22:38:42.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T22:38:43.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:42 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/467152635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e46e9424-c131-4559-9a43-55c7cf9c8eeb"}]: dispatch 2024-03-18T22:38:43.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:42 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/467152635' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e46e9424-c131-4559-9a43-55c7cf9c8eeb"}]': finished 2024-03-18T22:38:43.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:42 smithi113 ceph-mon[66775]: osdmap e64: 10 total, 9 up, 10 in 2024-03-18T22:38:43.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:42 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T22:38:43.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:42 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/467152635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e46e9424-c131-4559-9a43-55c7cf9c8eeb"}]: dispatch 2024-03-18T22:38:43.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:42 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/467152635' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e46e9424-c131-4559-9a43-55c7cf9c8eeb"}]': finished 2024-03-18T22:38:43.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:42 smithi103 ceph-mon[66377]: osdmap e64: 10 total, 9 up, 10 in 2024-03-18T22:38:43.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:42 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T22:38:44.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:43 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2858342755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:38:44.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:43 smithi113 ceph-mon[66775]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-03-18T22:38:44.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:43 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2858342755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:38:44.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:43 smithi103 ceph-mon[66377]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-03-18T22:38:44.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:43 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2858342755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:38:44.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:43 smithi040 ceph-mon[62643]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-03-18T22:38:45.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:44 smithi103 ceph-mon[66377]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T22:38:45.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:44 smithi040 ceph-mon[62643]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T22:38:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:44 smithi113 ceph-mon[66775]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T22:38:47.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:46 smithi103 ceph-mon[66377]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-18T22:38:47.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:46 smithi040 ceph-mon[62643]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-18T22:38:47.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:46 smithi113 ceph-mon[66775]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-18T22:38:49.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:48 smithi103 ceph-mon[66377]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-18T22:38:49.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:48 smithi040 ceph-mon[62643]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-18T22:38:49.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:48 smithi113 ceph-mon[66775]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-18T22:38:50.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:49 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T22:38:50.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:49 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:50.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:49 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T22:38:50.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:49 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:50.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:49 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T22:38:50.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:49 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:38:51.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:50 smithi040 ceph-mon[62643]: Deploying daemon osd.9 on smithi040 2024-03-18T22:38:51.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:50 smithi040 ceph-mon[62643]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T22:38:51.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:50 smithi113 ceph-mon[66775]: Deploying daemon osd.9 on smithi040 2024-03-18T22:38:51.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:50 smithi113 ceph-mon[66775]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T22:38:51.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:50 smithi103 ceph-mon[66377]: Deploying daemon osd.9 on smithi040 2024-03-18T22:38:51.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:50 smithi103 ceph-mon[66377]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T22:38:52.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:52 smithi040 ceph-mon[62643]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-18T22:38:53.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:52 smithi103 ceph-mon[66377]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-18T22:38:53.132 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:52 smithi113 ceph-mon[66775]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-18T22:38:55.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:54 smithi103 ceph-mon[66377]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:55.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:54 smithi040 ceph-mon[62643]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:55.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:54 smithi113 ceph-mon[66775]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:57.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:56 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:57.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:56 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:57.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:56 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:38:57.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:56 smithi113 ceph-mon[66775]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:57.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:56 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:57.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:56 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:57.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:56 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:38:57.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:56 smithi103 ceph-mon[66377]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:57.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:56 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:57.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:56 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:38:57.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:56 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:38:57.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:56 smithi040 ceph-mon[62643]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:59.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:58 smithi103 ceph-mon[66377]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:59.110 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:58 smithi040 ceph-mon[62643]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:38:59.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:58 smithi113 ceph-mon[66775]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:39:00.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:00.038+0000 7f7d98ff9700 1 -- 172.21.15.40:0/2083969107 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7d9c0fe1a0 con 0x7f7d840a2910 2024-03-18T22:39:00.041 INFO:teuthology.orchestra.run.smithi040.stdout:Created osd(s) 9 on host 'smithi040' 2024-03-18T22:39:00.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:00.040+0000 7f7da1c84700 1 -- 172.21.15.40:0/2083969107 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7d840a2910 msgr2=0x7f7d840a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:00.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:00.040+0000 7f7da1c84700 1 --2- 172.21.15.40:0/2083969107 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7d840a2910 0x7f7d840a4dd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c005fd0 tx=0x7f7d8c005f00 comp rx=0 tx=0).stop 2024-03-18T22:39:00.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:00.040+0000 7f7da1c84700 1 -- 172.21.15.40:0/2083969107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d9c101eb0 msgr2=0x7f7d9c103900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:00.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:00.040+0000 7f7da1c84700 1 --2- 172.21.15.40:0/2083969107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d9c101eb0 0x7f7d9c103900 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7d90006010 tx=0x7f7d900047b0 comp rx=0 tx=0).stop 2024-03-18T22:39:00.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:00.041+0000 7f7da1c84700 1 -- 172.21.15.40:0/2083969107 shutdown_connections 2024-03-18T22:39:00.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:00.041+0000 7f7da1c84700 1 --2- 172.21.15.40:0/2083969107 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7d840a2910 0x7f7d840a4dd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:00.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:00.041+0000 7f7da1c84700 1 --2- 172.21.15.40:0/2083969107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d9c1027d0 0x7f7d9c103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:00.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:00.041+0000 7f7da1c84700 1 --2- 172.21.15.40:0/2083969107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7d9c101510 0x7f7d9c1052b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:00.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:00.041+0000 7f7da1c84700 1 --2- 172.21.15.40:0/2083969107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d9c101eb0 0x7f7d9c103900 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:00.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:00.041+0000 7f7da1c84700 1 -- 172.21.15.40:0/2083969107 >> 172.21.15.40:0/2083969107 conn(0x7f7d9c0fbf80 msgr2=0x7f7d9c0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:39:00.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:00.041+0000 7f7da1c84700 1 -- 172.21.15.40:0/2083969107 shutdown_connections 2024-03-18T22:39:00.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:00.041+0000 7f7da1c84700 1 -- 172.21.15.40:0/2083969107 wait complete. 2024-03-18T22:39:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:59 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:38:59 smithi113 ceph-mon[66775]: from='osd.9 [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T22:39:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:59 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:38:59 smithi103 ceph-mon[66377]: from='osd.9 [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T22:39:00.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:00.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:00.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:00.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:59 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:38:59 smithi040 ceph-mon[62643]: from='osd.9 [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T22:39:00.721 DEBUG:teuthology.orchestra.run.smithi040:osd.9> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.9.service 2024-03-18T22:39:00.724 INFO:tasks.cephadm:Deploying osd.10 on smithi103 with /dev/vg_nvme/lv_1... 2024-03-18T22:39:00.724 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T22:39:00.972 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T22:39:01.260 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:01 smithi103 ceph-mon[66377]: from='osd.9 [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T22:39:01.260 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:01 smithi103 ceph-mon[66377]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T22:39:01.260 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:01 smithi103 ceph-mon[66377]: from='osd.9 [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-18T22:39:01.260 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:01 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T22:39:01.260 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:01 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:01.260 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:01 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:01.260 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:01 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:39:01.260 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:01 smithi103 ceph-mon[66377]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:39:01.272 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:01 smithi040 ceph-mon[62643]: from='osd.9 [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T22:39:01.273 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:01 smithi040 ceph-mon[62643]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T22:39:01.273 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:01 smithi040 ceph-mon[62643]: from='osd.9 [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-18T22:39:01.273 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:01 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T22:39:01.273 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:01 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:01.273 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:01 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:01.273 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:01 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:39:01.273 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:01 smithi040 ceph-mon[62643]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:39:01.273 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 22:39:01 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T22:39:01.005+0000 7f6d0f8b3700 -1 osd.9 0 waiting for initial osdmap 2024-03-18T22:39:01.273 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 22:39:01 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T22:39:01.017+0000 7f6d0b6a6700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T22:39:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:01 smithi113 ceph-mon[66775]: from='osd.9 [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T22:39:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:01 smithi113 ceph-mon[66775]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T22:39:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:01 smithi113 ceph-mon[66775]: from='osd.9 [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]: dispatch 2024-03-18T22:39:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:01 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T22:39:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:01 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:01 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:01 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:39:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:01 smithi113 ceph-mon[66775]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:39:02.164 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:02 smithi103 ceph-mon[66377]: from='osd.9 [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-18T22:39:02.164 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:02 smithi103 ceph-mon[66377]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T22:39:02.165 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:02 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T22:39:02.165 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:02 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T22:39:02.243 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:02 smithi040 ceph-mon[62643]: from='osd.9 [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-18T22:39:02.243 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:02 smithi040 ceph-mon[62643]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T22:39:02.244 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:02 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T22:39:02.244 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:02 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T22:39:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:02 smithi113 ceph-mon[66775]: from='osd.9 [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi040", "root=default"]}]': finished 2024-03-18T22:39:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:02 smithi113 ceph-mon[66775]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T22:39:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:02 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T22:39:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:02 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T22:39:03.054 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:03 smithi040 ceph-mon[62643]: purged_snaps scrub starts 2024-03-18T22:39:03.054 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:03 smithi040 ceph-mon[62643]: purged_snaps scrub ok 2024-03-18T22:39:03.054 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:03 smithi040 ceph-mon[62643]: osd.9 [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401] boot 2024-03-18T22:39:03.054 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:03 smithi040 ceph-mon[62643]: osdmap e67: 10 total, 10 up, 10 in 2024-03-18T22:39:03.054 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:03 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T22:39:03.054 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:03 smithi040 ceph-mon[62643]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:39:03.087 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T22:39:03.087 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T22:39:03.087 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-03-18T22:39:03.088 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-03-18T22:39:03.088 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130565 s, 803 MB/s 2024-03-18T22:39:03.088 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-03-18T22:39:03.111 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:03 smithi103 ceph-mon[66377]: purged_snaps scrub starts 2024-03-18T22:39:03.111 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:03 smithi103 ceph-mon[66377]: purged_snaps scrub ok 2024-03-18T22:39:03.111 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:03 smithi103 ceph-mon[66377]: osd.9 [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401] boot 2024-03-18T22:39:03.112 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:03 smithi103 ceph-mon[66377]: osdmap e67: 10 total, 10 up, 10 in 2024-03-18T22:39:03.112 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:03 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T22:39:03.112 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:03 smithi103 ceph-mon[66377]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:39:03.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:03 smithi113 ceph-mon[66775]: purged_snaps scrub starts 2024-03-18T22:39:03.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:03 smithi113 ceph-mon[66775]: purged_snaps scrub ok 2024-03-18T22:39:03.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:03 smithi113 ceph-mon[66775]: osd.9 [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401] boot 2024-03-18T22:39:03.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:03 smithi113 ceph-mon[66775]: osdmap e67: 10 total, 10 up, 10 in 2024-03-18T22:39:03.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:03 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T22:39:03.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:03 smithi113 ceph-mon[66775]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-18T22:39:03.764 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi103:vg_nvme/lv_1 2024-03-18T22:39:04.008 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T22:39:04.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:04 smithi103 ceph-mon[66377]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T22:39:04.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:04 smithi103 ceph-mon[66377]: Detected new or changed devices on smithi040 2024-03-18T22:39:04.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:04.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:04.021 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:04.276 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:04.276 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:04.276 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:04.276 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:04 smithi103 ceph-mon[66377]: Adjusting osd_memory_target on smithi040 to 4273M 2024-03-18T22:39:04.276 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:04.276 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:04.276 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:04.276 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:04.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:04 smithi113 ceph-mon[66775]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T22:39:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:04 smithi113 ceph-mon[66775]: Detected new or changed devices on smithi040 2024-03-18T22:39:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:04 smithi113 ceph-mon[66775]: Adjusting osd_memory_target on smithi040 to 4273M 2024-03-18T22:39:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:04.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:04 smithi040 ceph-mon[62643]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T22:39:04.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:04 smithi040 ceph-mon[62643]: Detected new or changed devices on smithi040 2024-03-18T22:39:04.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:04.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:04.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:04.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:04.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:04.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:04.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:04 smithi040 ceph-mon[62643]: Adjusting osd_memory_target on smithi040 to 4273M 2024-03-18T22:39:04.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:04.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:04.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:04.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:05.099 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:05 smithi103 ceph-mon[66377]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:05.295 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:05 smithi040 ceph-mon[62643]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:05.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:05 smithi113 ceph-mon[66775]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:05.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.644+0000 7fd755ef9700 1 -- 172.21.15.103:0/1890823432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd750102de0 msgr2=0x7fd750109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:05.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.644+0000 7fd755ef9700 1 --2- 172.21.15.103:0/1890823432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd750102de0 0x7fd750109690 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd74800b0d0 tx=0x7fd74800b3e0 comp rx=0 tx=0).stop 2024-03-18T22:39:05.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.644+0000 7fd755ef9700 1 -- 172.21.15.103:0/1890823432 shutdown_connections 2024-03-18T22:39:05.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.644+0000 7fd755ef9700 1 --2- 172.21.15.103:0/1890823432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd750102de0 0x7fd750109690 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:05.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.644+0000 7fd755ef9700 1 --2- 172.21.15.103:0/1890823432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd750101240 0x7fd750101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:05.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.644+0000 7fd755ef9700 1 --2- 172.21.15.103:0/1890823432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd750102440 0x7fd7501028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:05.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.644+0000 7fd755ef9700 1 -- 172.21.15.103:0/1890823432 >> 172.21.15.103:0/1890823432 conn(0x7fd7500fc7e0 msgr2=0x7fd7500fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:39:05.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.644+0000 7fd755ef9700 1 -- 172.21.15.103:0/1890823432 shutdown_connections 2024-03-18T22:39:05.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.644+0000 7fd755ef9700 1 -- 172.21.15.103:0/1890823432 wait complete. 2024-03-18T22:39:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.645+0000 7fd755ef9700 1 Processor -- start 2024-03-18T22:39:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.645+0000 7fd755ef9700 1 -- start start 2024-03-18T22:39:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.645+0000 7fd755ef9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd750101240 0x7fd7501984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.645+0000 7fd755ef9700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd750102440 0x7fd7501989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.645+0000 7fd755ef9700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd750102de0 0x7fd75019fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.645+0000 7fd755ef9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd75010ccd0 con 0x7fd750101240 2024-03-18T22:39:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.645+0000 7fd755ef9700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd75010cb50 con 0x7fd750102de0 2024-03-18T22:39:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.645+0000 7fd755ef9700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd75010ce50 con 0x7fd750102440 2024-03-18T22:39:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.645+0000 7fd74ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd750102de0 0x7fd75019fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.645+0000 7fd74ffff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd750102de0 0x7fd75019fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58956/0 (socket says 172.21.15.103:58956) 2024-03-18T22:39:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.645+0000 7fd74ffff700 1 -- 172.21.15.103:0/2183231887 learned_addr learned my addr 172.21.15.103:0/2183231887 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-03-18T22:39:05.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.645+0000 7fd74f7fe700 1 --2- 172.21.15.103:0/2183231887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd750101240 0x7fd7501984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:05.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.646+0000 7fd746ffd700 1 --2- 172.21.15.103:0/2183231887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd750102440 0x7fd7501989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:05.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.646+0000 7fd74f7fe700 1 -- 172.21.15.103:0/2183231887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd750102440 msgr2=0x7fd7501989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:05.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.646+0000 7fd74f7fe700 1 --2- 172.21.15.103:0/2183231887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd750102440 0x7fd7501989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:05.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.646+0000 7fd74f7fe700 1 -- 172.21.15.103:0/2183231887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd750102de0 msgr2=0x7fd75019fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:05.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.646+0000 7fd74f7fe700 1 --2- 172.21.15.103:0/2183231887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd750102de0 0x7fd75019fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:05.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.646+0000 7fd74f7fe700 1 -- 172.21.15.103:0/2183231887 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7501a00a0 con 0x7fd750101240 2024-03-18T22:39:05.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.646+0000 7fd74ffff700 1 --2- 172.21.15.103:0/2183231887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd750102de0 0x7fd75019fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:39:05.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.646+0000 7fd74f7fe700 1 --2- 172.21.15.103:0/2183231887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd750101240 0x7fd7501984b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd74000e9f0 tx=0x7fd74000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:39:05.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.646+0000 7fd74d7fa700 1 -- 172.21.15.103:0/2183231887 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd740004220 con 0x7fd750101240 2024-03-18T22:39:05.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.646+0000 7fd755ef9700 1 -- 172.21.15.103:0/2183231887 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7501a0390 con 0x7fd750101240 2024-03-18T22:39:05.650 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.646+0000 7fd74d7fa700 1 -- 172.21.15.103:0/2183231887 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd7400043c0 con 0x7fd750101240 2024-03-18T22:39:05.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.646+0000 7fd74d7fa700 1 -- 172.21.15.103:0/2183231887 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd740003c40 con 0x7fd750101240 2024-03-18T22:39:05.651 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.646+0000 7fd755ef9700 1 -- 172.21.15.103:0/2183231887 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7501a0920 con 0x7fd750101240 2024-03-18T22:39:05.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.647+0000 7fd755ef9700 1 -- 172.21.15.103:0/2183231887 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd750068fd0 con 0x7fd750101240 2024-03-18T22:39:05.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.648+0000 7fd74d7fa700 1 -- 172.21.15.103:0/2183231887 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd74000c7c0 con 0x7fd750101240 2024-03-18T22:39:05.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.649+0000 7fd74d7fa700 1 --2- 172.21.15.103:0/2183231887 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd73c0a2890 0x7fd73c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:05.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.649+0000 7fd74d7fa700 1 -- 172.21.15.103:0/2183231887 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7fd7400ce5f0 con 0x7fd750101240 2024-03-18T22:39:05.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.649+0000 7fd746ffd700 1 --2- 172.21.15.103:0/2183231887 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd73c0a2890 0x7fd73c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:05.652 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.649+0000 7fd746ffd700 1 --2- 172.21.15.103:0/2183231887 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd73c0a2890 0x7fd73c0a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd738004630 tx=0x7fd73800b3c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:39:05.654 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.652+0000 7fd74d7fa700 1 -- 172.21.15.103:0/2183231887 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd74009be80 con 0x7fd750101240 2024-03-18T22:39:05.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:05.855+0000 7fd755ef9700 1 -- 172.21.15.103:0/2183231887 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd7500fe9c0 con 0x7fd73c0a2890 2024-03-18T22:39:06.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:06 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:39:06.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:06 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:39:06.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:06 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:06.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:06 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:39:06.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:06 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:39:06.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:06 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:06.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:06 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:39:06.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:06 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:39:06.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:06 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:07.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:07 smithi113 ceph-mon[66775]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:39:07.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:07 smithi113 ceph-mon[66775]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:07.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:07 smithi103 ceph-mon[66377]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:39:07.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:07 smithi103 ceph-mon[66377]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:07.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:07 smithi040 ceph-mon[62643]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:39:07.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:07 smithi040 ceph-mon[62643]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 288 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:08 smithi113 ceph-mon[66775]: from='client.? 172.21.15.103:0/3253424390' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df88c341-1ebe-45e5-aa2e-f8a0696c3770"}]: dispatch 2024-03-18T22:39:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:08 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df88c341-1ebe-45e5-aa2e-f8a0696c3770"}]: dispatch 2024-03-18T22:39:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:08 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df88c341-1ebe-45e5-aa2e-f8a0696c3770"}]': finished 2024-03-18T22:39:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:08 smithi113 ceph-mon[66775]: osdmap e69: 11 total, 10 up, 11 in 2024-03-18T22:39:09.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:08 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:09.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:08 smithi113 ceph-mon[66775]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:09.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:08 smithi103 ceph-mon[66377]: from='client.? 172.21.15.103:0/3253424390' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df88c341-1ebe-45e5-aa2e-f8a0696c3770"}]: dispatch 2024-03-18T22:39:09.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:08 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df88c341-1ebe-45e5-aa2e-f8a0696c3770"}]: dispatch 2024-03-18T22:39:09.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:08 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df88c341-1ebe-45e5-aa2e-f8a0696c3770"}]': finished 2024-03-18T22:39:09.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:08 smithi103 ceph-mon[66377]: osdmap e69: 11 total, 10 up, 11 in 2024-03-18T22:39:09.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:08 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:09.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:08 smithi103 ceph-mon[66377]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:09.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:08 smithi040 ceph-mon[62643]: from='client.? 172.21.15.103:0/3253424390' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df88c341-1ebe-45e5-aa2e-f8a0696c3770"}]: dispatch 2024-03-18T22:39:09.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:08 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df88c341-1ebe-45e5-aa2e-f8a0696c3770"}]: dispatch 2024-03-18T22:39:09.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:08 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df88c341-1ebe-45e5-aa2e-f8a0696c3770"}]': finished 2024-03-18T22:39:09.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:08 smithi040 ceph-mon[62643]: osdmap e69: 11 total, 10 up, 11 in 2024-03-18T22:39:09.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:08 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:09.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:08 smithi040 ceph-mon[62643]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:09 smithi113 ceph-mon[66775]: from='client.? 172.21.15.103:0/2402755489' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:39:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:09 smithi103 ceph-mon[66377]: from='client.? 172.21.15.103:0/2402755489' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:39:10.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:09 smithi040 ceph-mon[62643]: from='client.? 172.21.15.103:0/2402755489' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:39:11.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:10 smithi113 ceph-mon[66775]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:11.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:10 smithi103 ceph-mon[66377]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:11.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:10 smithi040 ceph-mon[62643]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:13.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:12 smithi103 ceph-mon[66377]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:13.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:12 smithi040 ceph-mon[62643]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:13.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:12 smithi113 ceph-mon[66775]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:15.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:14 smithi103 ceph-mon[66377]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:15.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:14 smithi040 ceph-mon[62643]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:15.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:14 smithi113 ceph-mon[66775]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:16.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T22:39:16.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:16.610 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T22:39:16.610 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:16.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T22:39:16.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:17.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:17 smithi103 ceph-mon[66377]: Deploying daemon osd.10 on smithi103 2024-03-18T22:39:17.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:17 smithi103 ceph-mon[66377]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:17.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:17 smithi040 ceph-mon[62643]: Deploying daemon osd.10 on smithi103 2024-03-18T22:39:17.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:17 smithi040 ceph-mon[62643]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:17 smithi113 ceph-mon[66775]: Deploying daemon osd.10 on smithi103 2024-03-18T22:39:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:17 smithi113 ceph-mon[66775]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:19.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:18 smithi040 ceph-mon[62643]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:19.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:18 smithi103 ceph-mon[66377]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:19.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:18 smithi113 ceph-mon[66775]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:21.069 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:20 smithi103 ceph-mon[66377]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:21.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:20 smithi040 ceph-mon[62643]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:21.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:20 smithi113 ceph-mon[66775]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:23.305 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:22 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:23.306 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:22 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:23.306 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:22 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:39:23.306 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:22 smithi103 ceph-mon[66377]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:22 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:22 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:22 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:39:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:22 smithi113 ceph-mon[66775]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:22 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:22 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:22 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:39:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:22 smithi040 ceph-mon[62643]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:25.090 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:24 smithi103 ceph-mon[66377]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:25.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:24 smithi040 ceph-mon[62643]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:25.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:24 smithi113 ceph-mon[66775]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:25.823 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 10 on host 'smithi103' 2024-03-18T22:39:25.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:25.822+0000 7fd74d7fa700 1 -- 172.21.15.103:0/2183231887 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fd7500fe9c0 con 0x7fd73c0a2890 2024-03-18T22:39:25.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:25.825+0000 7fd755ef9700 1 -- 172.21.15.103:0/2183231887 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd73c0a2890 msgr2=0x7fd73c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:25.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:25.825+0000 7fd755ef9700 1 --2- 172.21.15.103:0/2183231887 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd73c0a2890 0x7fd73c0a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd738004630 tx=0x7fd73800b3c0 comp rx=0 tx=0).stop 2024-03-18T22:39:25.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:25.825+0000 7fd755ef9700 1 -- 172.21.15.103:0/2183231887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd750101240 msgr2=0x7fd7501984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:25.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:25.825+0000 7fd755ef9700 1 --2- 172.21.15.103:0/2183231887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd750101240 0x7fd7501984b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd74000e9f0 tx=0x7fd74000ed00 comp rx=0 tx=0).stop 2024-03-18T22:39:25.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:25.825+0000 7fd755ef9700 1 -- 172.21.15.103:0/2183231887 shutdown_connections 2024-03-18T22:39:25.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:25.825+0000 7fd755ef9700 1 --2- 172.21.15.103:0/2183231887 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd73c0a2890 0x7fd73c0a4d50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:25.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:25.825+0000 7fd755ef9700 1 --2- 172.21.15.103:0/2183231887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd750102de0 0x7fd75019fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:25.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:25.825+0000 7fd755ef9700 1 --2- 172.21.15.103:0/2183231887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd750101240 0x7fd7501984b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:25.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:25.825+0000 7fd755ef9700 1 --2- 172.21.15.103:0/2183231887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd750102440 0x7fd7501989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:25.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:25.825+0000 7fd755ef9700 1 -- 172.21.15.103:0/2183231887 >> 172.21.15.103:0/2183231887 conn(0x7fd7500fc7e0 msgr2=0x7fd7500fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:39:25.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:25.825+0000 7fd755ef9700 1 -- 172.21.15.103:0/2183231887 shutdown_connections 2024-03-18T22:39:25.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-03-18T22:39:25.825+0000 7fd755ef9700 1 -- 172.21.15.103:0/2183231887 wait complete. 2024-03-18T22:39:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:39:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:26.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:26.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:26 smithi113 ceph-mon[66775]: from='osd.10 [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T22:39:26.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:26 smithi113 ceph-mon[66775]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T22:39:26.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:26.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:26.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:26.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:26.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:26.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:39:26.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:26.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:26.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:26 smithi103 ceph-mon[66377]: from='osd.10 [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T22:39:26.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:26 smithi103 ceph-mon[66377]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T22:39:26.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:26.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:26.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:26.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:26.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:26.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:39:26.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:26.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:26.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:26 smithi040 ceph-mon[62643]: from='osd.10 [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T22:39:26.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:26 smithi040 ceph-mon[62643]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T22:39:26.527 DEBUG:teuthology.orchestra.run.smithi103:osd.10> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.10.service 2024-03-18T22:39:26.529 INFO:tasks.cephadm:Deploying osd.11 on smithi113 with /dev/vg_nvme/lv_1... 2024-03-18T22:39:26.529 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T22:39:26.776 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:39:27.969 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:27 smithi103 ceph-mon[66377]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T22:39:27.969 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:27 smithi103 ceph-mon[66377]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:27.969 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:27 smithi103 ceph-mon[66377]: from='osd.10 [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:39:27.969 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:27 smithi103 ceph-mon[66377]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T22:39:27.969 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:27 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:27.969 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:27 smithi103 ceph-mon[66377]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:39:28.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:27 smithi113 ceph-mon[66775]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T22:39:28.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:27 smithi113 ceph-mon[66775]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:28.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:27 smithi113 ceph-mon[66775]: from='osd.10 [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:39:28.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:27 smithi113 ceph-mon[66775]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T22:39:28.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:28.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:27 smithi113 ceph-mon[66775]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:39:28.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:27 smithi040 ceph-mon[62643]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T22:39:28.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:27 smithi040 ceph-mon[62643]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:28.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:27 smithi040 ceph-mon[62643]: from='osd.10 [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:39:28.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:27 smithi040 ceph-mon[62643]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T22:39:28.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:27 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:28.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:27 smithi040 ceph-mon[62643]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-03-18T22:39:28.838 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T22:39:28.838 INFO:teuthology.orchestra.run.smithi113.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T22:39:28.838 INFO:teuthology.orchestra.run.smithi113.stderr: stderr: 10+0 records in 2024-03-18T22:39:28.838 INFO:teuthology.orchestra.run.smithi113.stderr:10+0 records out 2024-03-18T22:39:28.838 INFO:teuthology.orchestra.run.smithi113.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00847871 s, 1.2 GB/s 2024-03-18T22:39:28.838 INFO:teuthology.orchestra.run.smithi113.stderr:--> Zapping successful for: 2024-03-18T22:39:28.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:28 smithi103 ceph-mon[66377]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-18T22:39:28.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:28 smithi103 ceph-mon[66377]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T22:39:28.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:28 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:28.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:28 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:28.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:28 smithi103 ceph-mon[66377]: from='osd.10 ' entity='osd.10' 2024-03-18T22:39:28.860 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 22:39:28 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T22:39:28.577+0000 7f5409499700 -1 osd.10 0 waiting for initial osdmap 2024-03-18T22:39:28.860 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 22:39:28 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T22:39:28.589+0000 7f5401284700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T22:39:29.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:28 smithi113 ceph-mon[66775]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-18T22:39:29.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:28 smithi113 ceph-mon[66775]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T22:39:29.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:28 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:29.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:28 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:29.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:28 smithi113 ceph-mon[66775]: from='osd.10 ' entity='osd.10' 2024-03-18T22:39:29.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:28 smithi040 ceph-mon[62643]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-03-18T22:39:29.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:28 smithi040 ceph-mon[62643]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T22:39:29.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:28 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:29.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:28 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:29.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:28 smithi040 ceph-mon[62643]: from='osd.10 ' entity='osd.10' 2024-03-18T22:39:29.528 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi113:vg_nvme/lv_1 2024-03-18T22:39:29.767 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:39:30.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:29 smithi113 ceph-mon[66775]: purged_snaps scrub starts 2024-03-18T22:39:30.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:29 smithi113 ceph-mon[66775]: purged_snaps scrub ok 2024-03-18T22:39:30.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:29 smithi113 ceph-mon[66775]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:30.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:30.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:29 smithi113 ceph-mon[66775]: osd.10 [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819] boot 2024-03-18T22:39:30.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:29 smithi113 ceph-mon[66775]: osdmap e72: 11 total, 11 up, 11 in 2024-03-18T22:39:30.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:30.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:29 smithi103 ceph-mon[66377]: purged_snaps scrub starts 2024-03-18T22:39:30.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:29 smithi103 ceph-mon[66377]: purged_snaps scrub ok 2024-03-18T22:39:30.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:29 smithi103 ceph-mon[66377]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:30.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:30.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:29 smithi103 ceph-mon[66377]: osd.10 [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819] boot 2024-03-18T22:39:30.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:29 smithi103 ceph-mon[66377]: osdmap e72: 11 total, 11 up, 11 in 2024-03-18T22:39:30.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:30.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:29 smithi040 ceph-mon[62643]: purged_snaps scrub starts 2024-03-18T22:39:30.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:29 smithi040 ceph-mon[62643]: purged_snaps scrub ok 2024-03-18T22:39:30.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:29 smithi040 ceph-mon[62643]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 289 MiB used, 894 GiB / 894 GiB avail 2024-03-18T22:39:30.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:30.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:29 smithi040 ceph-mon[62643]: osd.10 [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819] boot 2024-03-18T22:39:30.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:29 smithi040 ceph-mon[62643]: osdmap e72: 11 total, 11 up, 11 in 2024-03-18T22:39:30.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T22:39:31.064 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:30 smithi113 ceph-mon[66775]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T22:39:31.065 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:30 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:31.065 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:30 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:31.065 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:30 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:31.065 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:30 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:31.065 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:30 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:31.065 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:30 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:31.065 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:30 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:31.065 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:30 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:31.065 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:30 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:31.065 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:30 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:30 smithi103 ceph-mon[66377]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T22:39:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:30 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:30 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:30 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:30 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:30 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:30 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:30 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:31.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:30 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:31.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:30 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:31.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:30 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:31.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:30 smithi040 ceph-mon[62643]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T22:39:31.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:30 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:31.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:30 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:31.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:30 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:31.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:30 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:31.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:30 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:31.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:30 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:31.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:30 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:31.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:30 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:31.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:30 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:31.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:30 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:31.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.269+0000 7f5a95bd2700 1 -- 172.21.15.113:0/1944887501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a90103540 msgr2=0x7f5a901039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:31.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.269+0000 7f5a95bd2700 1 --2- 172.21.15.113:0/1944887501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a90103540 0x7f5a901039a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5a84009990 tx=0x7f5a84009ca0 comp rx=0 tx=0).stop 2024-03-18T22:39:31.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.269+0000 7f5a95bd2700 1 -- 172.21.15.113:0/1944887501 shutdown_connections 2024-03-18T22:39:31.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.269+0000 7f5a95bd2700 1 --2- 172.21.15.113:0/1944887501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a90103ee0 0x7f5a9010a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:31.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.269+0000 7f5a95bd2700 1 --2- 172.21.15.113:0/1944887501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a90102340 0x7f5a90102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:31.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.269+0000 7f5a95bd2700 1 --2- 172.21.15.113:0/1944887501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a90103540 0x7f5a901039a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:31.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.269+0000 7f5a95bd2700 1 -- 172.21.15.113:0/1944887501 >> 172.21.15.113:0/1944887501 conn(0x7f5a900fd920 msgr2=0x7f5a900ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:39:31.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.269+0000 7f5a95bd2700 1 -- 172.21.15.113:0/1944887501 shutdown_connections 2024-03-18T22:39:31.272 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.269+0000 7f5a95bd2700 1 -- 172.21.15.113:0/1944887501 wait complete. 2024-03-18T22:39:31.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.269+0000 7f5a95bd2700 1 Processor -- start 2024-03-18T22:39:31.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a95bd2700 1 -- start start 2024-03-18T22:39:31.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a95bd2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a90102340 0x7f5a90198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:31.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a95bd2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a90103540 0x7f5a90198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:31.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a95bd2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a90103ee0 0x7f5a9019faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:31.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a95bd2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9010de50 con 0x7f5a90103540 2024-03-18T22:39:31.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a95bd2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9010dcd0 con 0x7f5a90103ee0 2024-03-18T22:39:31.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a95bd2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9010dfd0 con 0x7f5a90102340 2024-03-18T22:39:31.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a8f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a90102340 0x7f5a90198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:31.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a8f7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a90102340 0x7f5a90198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:34710/0 (socket says 172.21.15.113:34710) 2024-03-18T22:39:31.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a8f7fe700 1 -- 172.21.15.113:0/1153114449 learned_addr learned my addr 172.21.15.113:0/1153114449 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-18T22:39:31.273 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a8effd700 1 --2- 172.21.15.113:0/1153114449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a90103540 0x7f5a90198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:31.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a8ffff700 1 --2- 172.21.15.113:0/1153114449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a90103ee0 0x7f5a9019faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:31.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a8f7fe700 1 -- 172.21.15.113:0/1153114449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a90103ee0 msgr2=0x7f5a9019faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:31.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a8f7fe700 1 --2- 172.21.15.113:0/1153114449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a90103ee0 0x7f5a9019faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:31.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a8f7fe700 1 -- 172.21.15.113:0/1153114449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a90103540 msgr2=0x7f5a90198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:31.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a8f7fe700 1 --2- 172.21.15.113:0/1153114449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a90103540 0x7f5a90198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:31.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a8f7fe700 1 -- 172.21.15.113:0/1153114449 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a901a0190 con 0x7f5a90102340 2024-03-18T22:39:31.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a8ffff700 1 --2- 172.21.15.113:0/1153114449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a90103ee0 0x7f5a9019faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:39:31.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.270+0000 7f5a8f7fe700 1 --2- 172.21.15.113:0/1153114449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a90102340 0x7f5a90198580 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5a8000b9f0 tx=0x7f5a8000bd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:39:31.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.271+0000 7f5a8cff9700 1 -- 172.21.15.113:0/1153114449 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a80019070 con 0x7f5a90102340 2024-03-18T22:39:31.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.271+0000 7f5a8cff9700 1 -- 172.21.15.113:0/1153114449 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a8000ce00 con 0x7f5a90102340 2024-03-18T22:39:31.274 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.271+0000 7f5a95bd2700 1 -- 172.21.15.113:0/1153114449 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a901a0480 con 0x7f5a90102340 2024-03-18T22:39:31.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.271+0000 7f5a8cff9700 1 -- 172.21.15.113:0/1153114449 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a80012340 con 0x7f5a90102340 2024-03-18T22:39:31.277 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.271+0000 7f5a95bd2700 1 -- 172.21.15.113:0/1153114449 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a901a0a40 con 0x7f5a90102340 2024-03-18T22:39:31.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.272+0000 7f5a95bd2700 1 -- 172.21.15.113:0/1153114449 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a90068fd0 con 0x7f5a90102340 2024-03-18T22:39:31.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.273+0000 7f5a8cff9700 1 -- 172.21.15.113:0/1153114449 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5a8001c070 con 0x7f5a90102340 2024-03-18T22:39:31.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.273+0000 7f5a8cff9700 1 --2- 172.21.15.113:0/1153114449 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5a780a2890 0x7f5a780a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:31.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.273+0000 7f5a8cff9700 1 -- 172.21.15.113:0/1153114449 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7f5a800cd7c0 con 0x7f5a90102340 2024-03-18T22:39:31.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.273+0000 7f5a8effd700 1 --2- 172.21.15.113:0/1153114449 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5a780a2890 0x7f5a780a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:31.278 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.274+0000 7f5a8effd700 1 --2- 172.21.15.113:0/1153114449 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5a780a2890 0x7f5a780a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5a8400b5c0 tx=0x7f5a84005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:39:31.279 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.278+0000 7f5a8cff9700 1 -- 172.21.15.113:0/1153114449 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5a8009ae90 con 0x7f5a90102340 2024-03-18T22:39:31.487 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:31.486+0000 7f5a95bd2700 1 -- 172.21.15.113:0/1153114449 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f5a90062750 con 0x7f5a780a2890 2024-03-18T22:39:31.813 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:31 smithi113 ceph-mon[66775]: Detected new or changed devices on smithi103 2024-03-18T22:39:31.814 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:31 smithi113 ceph-mon[66775]: Adjusting osd_memory_target on smithi103 to 4272M 2024-03-18T22:39:31.814 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:31 smithi113 ceph-mon[66775]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:31 smithi113 ceph-mon[66775]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T22:39:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:31 smithi113 ceph-mon[66775]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:39:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:31 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:39:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:31 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:39:32.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:31 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:32.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:31 smithi103 ceph-mon[66377]: Detected new or changed devices on smithi103 2024-03-18T22:39:32.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:31 smithi103 ceph-mon[66377]: Adjusting osd_memory_target on smithi103 to 4272M 2024-03-18T22:39:32.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:31 smithi103 ceph-mon[66377]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:32.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:31 smithi103 ceph-mon[66377]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T22:39:32.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:31 smithi103 ceph-mon[66377]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:39:32.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:31 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:39:32.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:31 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:39:32.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:31 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:32.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:31 smithi040 ceph-mon[62643]: Detected new or changed devices on smithi103 2024-03-18T22:39:32.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:31 smithi040 ceph-mon[62643]: Adjusting osd_memory_target on smithi103 to 4272M 2024-03-18T22:39:32.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:31 smithi040 ceph-mon[62643]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:32.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:31 smithi040 ceph-mon[62643]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T22:39:32.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:31 smithi040 ceph-mon[62643]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi113:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:39:32.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:31 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T22:39:32.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:31 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T22:39:32.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:31 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:32 smithi113 ceph-mon[66775]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T22:39:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:32 smithi113 ceph-mon[66775]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T22:39:33.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:32 smithi103 ceph-mon[66377]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T22:39:33.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:32 smithi103 ceph-mon[66377]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T22:39:33.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:32 smithi040 ceph-mon[62643]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T22:39:33.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:32 smithi040 ceph-mon[62643]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T22:39:34.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:33 smithi113 ceph-mon[66775]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T22:39:34.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:33 smithi103 ceph-mon[66377]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T22:39:34.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:33 smithi040 ceph-mon[62643]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T22:39:35.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:34 smithi113 ceph-mon[66775]: from='client.? 172.21.15.113:0/787384322' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97b51a30-f636-4149-a06a-f536f1671467"}]: dispatch 2024-03-18T22:39:35.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:34 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97b51a30-f636-4149-a06a-f536f1671467"}]: dispatch 2024-03-18T22:39:35.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:34 smithi113 ceph-mon[66775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97b51a30-f636-4149-a06a-f536f1671467"}]': finished 2024-03-18T22:39:35.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:34 smithi113 ceph-mon[66775]: osdmap e77: 12 total, 11 up, 12 in 2024-03-18T22:39:35.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:34 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:35.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:34 smithi113 ceph-mon[66775]: from='client.? 172.21.15.113:0/2523693325' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:39:35.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:34 smithi113 ceph-mon[66775]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T22:39:35.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:34 smithi103 ceph-mon[66377]: from='client.? 172.21.15.113:0/787384322' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97b51a30-f636-4149-a06a-f536f1671467"}]: dispatch 2024-03-18T22:39:35.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:34 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97b51a30-f636-4149-a06a-f536f1671467"}]: dispatch 2024-03-18T22:39:35.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:34 smithi103 ceph-mon[66377]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97b51a30-f636-4149-a06a-f536f1671467"}]': finished 2024-03-18T22:39:35.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:34 smithi103 ceph-mon[66377]: osdmap e77: 12 total, 11 up, 12 in 2024-03-18T22:39:35.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:34 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:35.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:34 smithi103 ceph-mon[66377]: from='client.? 172.21.15.113:0/2523693325' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:39:35.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:34 smithi103 ceph-mon[66377]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T22:39:35.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:34 smithi040 ceph-mon[62643]: from='client.? 172.21.15.113:0/787384322' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97b51a30-f636-4149-a06a-f536f1671467"}]: dispatch 2024-03-18T22:39:35.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:34 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97b51a30-f636-4149-a06a-f536f1671467"}]: dispatch 2024-03-18T22:39:35.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:34 smithi040 ceph-mon[62643]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97b51a30-f636-4149-a06a-f536f1671467"}]': finished 2024-03-18T22:39:35.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:34 smithi040 ceph-mon[62643]: osdmap e77: 12 total, 11 up, 12 in 2024-03-18T22:39:35.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:34 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:35.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:34 smithi040 ceph-mon[62643]: from='client.? 172.21.15.113:0/2523693325' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T22:39:35.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:34 smithi040 ceph-mon[62643]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T22:39:37.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:36 smithi040 ceph-mon[62643]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T22:39:37.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:36 smithi113 ceph-mon[66775]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T22:39:37.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:36 smithi103 ceph-mon[66377]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T22:39:39.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:38 smithi103 ceph-mon[66377]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-18T22:39:39.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:38 smithi040 ceph-mon[62643]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-18T22:39:39.160 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:38 smithi113 ceph-mon[66775]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-18T22:39:41.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:40 smithi040 ceph-mon[62643]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:41.173 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:40 smithi113 ceph-mon[66775]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:41.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:40 smithi103 ceph-mon[66377]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:42.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T22:39:42.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:42.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:41 smithi040 ceph-mon[62643]: Deploying daemon osd.11 on smithi113 2024-03-18T22:39:42.231 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T22:39:42.231 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:42.231 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:41 smithi113 ceph-mon[66775]: Deploying daemon osd.11 on smithi113 2024-03-18T22:39:42.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:41 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T22:39:42.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:41 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:42.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:41 smithi103 ceph-mon[66377]: Deploying daemon osd.11 on smithi113 2024-03-18T22:39:43.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:42 smithi040 ceph-mon[62643]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:43.190 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:42 smithi113 ceph-mon[66775]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:43.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:42 smithi103 ceph-mon[66377]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:45.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:44 smithi040 ceph-mon[62643]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:45.318 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:44 smithi113 ceph-mon[66775]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:44 smithi103 ceph-mon[66377]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:46.938 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:46 smithi113 ceph-mon[66775]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:47.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:46 smithi040 ceph-mon[62643]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:47.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:46 smithi103 ceph-mon[66377]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:48.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:39:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:48.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:48.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:39:48.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:48.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:39:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:49.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:49 smithi113 ceph-mon[66775]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:49.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:49 smithi103 ceph-mon[66377]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:49.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:49 smithi040 ceph-mon[62643]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:51.405 INFO:teuthology.orchestra.run.smithi113.stdout:Created osd(s) 11 on host 'smithi113' 2024-03-18T22:39:51.406 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:51.404+0000 7f5a8cff9700 1 -- 172.21.15.113:0/1153114449 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f5a90062750 con 0x7f5a780a2890 2024-03-18T22:39:51.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:51.406+0000 7f5a95bd2700 1 -- 172.21.15.113:0/1153114449 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5a780a2890 msgr2=0x7f5a780a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:51.407 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:51.406+0000 7f5a95bd2700 1 --2- 172.21.15.113:0/1153114449 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5a780a2890 0x7f5a780a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5a8400b5c0 tx=0x7f5a84005fd0 comp rx=0 tx=0).stop 2024-03-18T22:39:51.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:51.406+0000 7f5a95bd2700 1 -- 172.21.15.113:0/1153114449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a90102340 msgr2=0x7f5a90198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:51.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:51.406+0000 7f5a95bd2700 1 --2- 172.21.15.113:0/1153114449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a90102340 0x7f5a90198580 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5a8000b9f0 tx=0x7f5a8000bd00 comp rx=0 tx=0).stop 2024-03-18T22:39:51.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:51.406+0000 7f5a95bd2700 1 -- 172.21.15.113:0/1153114449 shutdown_connections 2024-03-18T22:39:51.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:51.406+0000 7f5a95bd2700 1 --2- 172.21.15.113:0/1153114449 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5a780a2890 0x7f5a780a4d50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:51.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:51.406+0000 7f5a95bd2700 1 --2- 172.21.15.113:0/1153114449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a90103ee0 0x7f5a9019faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:51.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:51.406+0000 7f5a95bd2700 1 --2- 172.21.15.113:0/1153114449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a90102340 0x7f5a90198580 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:51.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:51.406+0000 7f5a95bd2700 1 --2- 172.21.15.113:0/1153114449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a90103540 0x7f5a90198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:51.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:51.406+0000 7f5a95bd2700 1 -- 172.21.15.113:0/1153114449 >> 172.21.15.113:0/1153114449 conn(0x7f5a900fd920 msgr2=0x7f5a90106980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:39:51.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:51.407+0000 7f5a95bd2700 1 -- 172.21.15.113:0/1153114449 shutdown_connections 2024-03-18T22:39:51.408 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:39:51.407+0000 7f5a95bd2700 1 -- 172.21.15.113:0/1153114449 wait complete. 2024-03-18T22:39:51.762 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:51 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:51.762 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:51 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:51.762 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:51 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:51.762 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:51 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:51.762 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:51 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:51.762 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:51 smithi113 ceph-mon[66775]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:51.762 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:51 smithi113 ceph-mon[66775]: from='osd.11 [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T22:39:51.762 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:51 smithi113 ceph-mon[66775]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T22:39:51.762 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:51 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:51.762 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:51 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:51.762 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:51 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:39:51.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:51 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:51.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:51 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:51.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:51 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:51.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:51 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:51.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:51 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:51.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:51 smithi103 ceph-mon[66377]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:51.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:51 smithi103 ceph-mon[66377]: from='osd.11 [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T22:39:51.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:51 smithi103 ceph-mon[66377]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T22:39:51.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:51 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:51.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:51 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:51.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:51 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:39:51.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:51 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:51.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:51 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:51.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:51 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:51.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:51 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:51.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:51 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:51.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:51 smithi040 ceph-mon[62643]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:51.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:51 smithi040 ceph-mon[62643]: from='osd.11 [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T22:39:51.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:51 smithi040 ceph-mon[62643]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T22:39:51.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:51 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:51.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:51 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:51.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:51 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:39:52.106 DEBUG:teuthology.orchestra.run.smithi113:osd.11> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.11.service 2024-03-18T22:39:52.108 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-18T22:39:52.108 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-18T22:39:52.362 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:39:52.633 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:52 smithi113 ceph-mon[66775]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T22:39:52.633 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:52 smithi113 ceph-mon[66775]: from='osd.11 [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:39:52.633 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:52 smithi113 ceph-mon[66775]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T22:39:52.633 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:52 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:52.633 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:52 smithi113 ceph-mon[66775]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:39:52.730 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:52 smithi040 ceph-mon[62643]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T22:39:52.731 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:52 smithi040 ceph-mon[62643]: from='osd.11 [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:39:52.731 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:52 smithi040 ceph-mon[62643]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T22:39:52.731 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:52 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:52.731 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:52 smithi040 ceph-mon[62643]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:39:52.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:52 smithi103 ceph-mon[66377]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T22:39:52.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:52 smithi103 ceph-mon[66377]: from='osd.11 [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:39:52.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:52 smithi103 ceph-mon[66377]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T22:39:52.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:52 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:52.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:52 smithi103 ceph-mon[66377]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]: dispatch 2024-03-18T22:39:53.304 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 22:39:52 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T22:39:52.927+0000 7f2d2da4e700 -1 osd.11 0 waiting for initial osdmap 2024-03-18T22:39:53.304 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 22:39:52 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T22:39:52.944+0000 7f2d25839700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T22:39:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:53 smithi113 ceph-mon[66775]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-18T22:39:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:53 smithi113 ceph-mon[66775]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T22:39:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:53 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:53 smithi113 ceph-mon[66775]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:53.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:53 smithi113 ceph-mon[66775]: from='osd.11 ' entity='osd.11' 2024-03-18T22:39:53.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:53 smithi040 ceph-mon[62643]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-18T22:39:53.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:53 smithi040 ceph-mon[62643]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T22:39:53.723 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:53.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:53 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:53.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:53 smithi040 ceph-mon[62643]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:53.724 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:53 smithi040 ceph-mon[62643]: from='osd.11 ' entity='osd.11' 2024-03-18T22:39:53.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:53 smithi103 ceph-mon[66377]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi113", "root=default"]}]': finished 2024-03-18T22:39:53.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:53 smithi103 ceph-mon[66377]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T22:39:53.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:53.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:53 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:53.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:53 smithi103 ceph-mon[66377]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-03-18T22:39:53.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:53 smithi103 ceph-mon[66377]: from='osd.11 ' entity='osd.11' 2024-03-18T22:39:54.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.056+0000 7f3d39778700 1 -- 172.21.15.40:0/2973683864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d34102de0 msgr2=0x7f3d34109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:54.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.056+0000 7f3d39778700 1 --2- 172.21.15.40:0/2973683864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d34102de0 0x7f3d34109690 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00b0d0 tx=0x7f3d2c00b3e0 comp rx=0 tx=0).stop 2024-03-18T22:39:54.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.056+0000 7f3d39778700 1 -- 172.21.15.40:0/2973683864 shutdown_connections 2024-03-18T22:39:54.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.056+0000 7f3d39778700 1 --2- 172.21.15.40:0/2973683864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d34102de0 0x7f3d34109690 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:54.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.056+0000 7f3d39778700 1 --2- 172.21.15.40:0/2973683864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d34101240 0x7f3d34101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:54.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.056+0000 7f3d39778700 1 --2- 172.21.15.40:0/2973683864 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d34102440 0x7f3d341028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:54.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.056+0000 7f3d39778700 1 -- 172.21.15.40:0/2973683864 >> 172.21.15.40:0/2973683864 conn(0x7f3d340fc7e0 msgr2=0x7f3d340fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:39:54.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.056+0000 7f3d39778700 1 -- 172.21.15.40:0/2973683864 shutdown_connections 2024-03-18T22:39:54.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.056+0000 7f3d39778700 1 -- 172.21.15.40:0/2973683864 wait complete. 2024-03-18T22:39:54.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.057+0000 7f3d39778700 1 Processor -- start 2024-03-18T22:39:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.057+0000 7f3d39778700 1 -- start start 2024-03-18T22:39:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.057+0000 7f3d39778700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d34101240 0x7f3d3406aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.057+0000 7f3d39778700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d34102440 0x7f3d3419d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.057+0000 7f3d39778700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d34102de0 0x7f3d3419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.057+0000 7f3d39778700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3410ccd0 con 0x7f3d34101240 2024-03-18T22:39:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.057+0000 7f3d39778700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3410cb50 con 0x7f3d34102de0 2024-03-18T22:39:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.057+0000 7f3d39778700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3410ce50 con 0x7f3d34102440 2024-03-18T22:39:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.057+0000 7f3d32ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d34101240 0x7f3d3406aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.057+0000 7f3d32ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d34101240 0x7f3d3406aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38594/0 (socket says 172.21.15.40:38594) 2024-03-18T22:39:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.057+0000 7f3d32ffd700 1 -- 172.21.15.40:0/207505983 learned_addr learned my addr 172.21.15.40:0/207505983 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:39:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.057+0000 7f3d327fc700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d34102440 0x7f3d3419d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:54.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.057+0000 7f3d337fe700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d34102de0 0x7f3d3419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.057+0000 7f3d32ffd700 1 -- 172.21.15.40:0/207505983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d34102440 msgr2=0x7f3d3419d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.057+0000 7f3d32ffd700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d34102440 0x7f3d3419d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.058+0000 7f3d32ffd700 1 -- 172.21.15.40:0/207505983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d34102de0 msgr2=0x7f3d3419f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.058+0000 7f3d32ffd700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d34102de0 0x7f3d3419f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.058+0000 7f3d32ffd700 1 -- 172.21.15.40:0/207505983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d341a00a0 con 0x7f3d34101240 2024-03-18T22:39:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.058+0000 7f3d337fe700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d34102de0 0x7f3d3419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:39:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.058+0000 7f3d327fc700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d34102440 0x7f3d3419d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:39:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.058+0000 7f3d32ffd700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d34101240 0x7f3d3406aaf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400d980 tx=0x7f3d2400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:39:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.058+0000 7f3d1bfff700 1 -- 172.21.15.40:0/207505983 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24017030 con 0x7f3d34101240 2024-03-18T22:39:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.058+0000 7f3d1bfff700 1 -- 172.21.15.40:0/207505983 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d240044d0 con 0x7f3d34101240 2024-03-18T22:39:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.058+0000 7f3d39778700 1 -- 172.21.15.40:0/207505983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d341a0390 con 0x7f3d34101240 2024-03-18T22:39:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.058+0000 7f3d1bfff700 1 -- 172.21.15.40:0/207505983 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d240039e0 con 0x7f3d34101240 2024-03-18T22:39:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.058+0000 7f3d39778700 1 -- 172.21.15.40:0/207505983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d341a08d0 con 0x7f3d34101240 2024-03-18T22:39:54.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.059+0000 7f3d39778700 1 -- 172.21.15.40:0/207505983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d34068fd0 con 0x7f3d34101240 2024-03-18T22:39:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.060+0000 7f3d1bfff700 1 -- 172.21.15.40:0/207505983 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3d2401a070 con 0x7f3d34101240 2024-03-18T22:39:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.061+0000 7f3d1bfff700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3d1c0a2890 0x7f3d1c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:54.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.061+0000 7f3d327fc700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3d1c0a2890 0x7f3d1c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:54.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.061+0000 7f3d1bfff700 1 -- 172.21.15.40:0/207505983 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7127+0+0 (secure 0 0 0) 0x7f3d240ce2b0 con 0x7f3d34101240 2024-03-18T22:39:54.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.061+0000 7f3d327fc700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3d1c0a2890 0x7f3d1c0a4d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3d28011490 tx=0x7f3d28009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:39:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.069+0000 7f3d1bfff700 1 -- 172.21.15.40:0/207505983 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3d2409b7f0 con 0x7f3d34101240 2024-03-18T22:39:54.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.275+0000 7f3d39778700 1 -- 172.21.15.40:0/207505983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f3d3402dbb0 con 0x7f3d34101240 2024-03-18T22:39:54.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.275+0000 7f3d1bfff700 1 -- 172.21.15.40:0/207505983 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f3d2409b0f0 con 0x7f3d34101240 2024-03-18T22:39:54.276 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:39:54.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.279+0000 7f3d39778700 1 -- 172.21.15.40:0/207505983 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3d1c0a2890 msgr2=0x7f3d1c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:54.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.279+0000 7f3d39778700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3d1c0a2890 0x7f3d1c0a4d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3d28011490 tx=0x7f3d28009260 comp rx=0 tx=0).stop 2024-03-18T22:39:54.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.279+0000 7f3d39778700 1 -- 172.21.15.40:0/207505983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d34101240 msgr2=0x7f3d3406aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:54.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.279+0000 7f3d39778700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d34101240 0x7f3d3406aaf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400d980 tx=0x7f3d2400dc90 comp rx=0 tx=0).stop 2024-03-18T22:39:54.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.279+0000 7f3d39778700 1 -- 172.21.15.40:0/207505983 shutdown_connections 2024-03-18T22:39:54.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.279+0000 7f3d39778700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3d1c0a2890 0x7f3d1c0a4d50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:54.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.279+0000 7f3d39778700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d34102de0 0x7f3d3419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:54.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.279+0000 7f3d39778700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d34101240 0x7f3d3406aaf0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:54.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.279+0000 7f3d39778700 1 --2- 172.21.15.40:0/207505983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d34102440 0x7f3d3419d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:54.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.279+0000 7f3d39778700 1 -- 172.21.15.40:0/207505983 >> 172.21.15.40:0/207505983 conn(0x7f3d340fc7e0 msgr2=0x7f3d34105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:39:54.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.279+0000 7f3d39778700 1 -- 172.21.15.40:0/207505983 shutdown_connections 2024-03-18T22:39:54.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:54.279+0000 7f3d39778700 1 -- 172.21.15.40:0/207505983 wait complete. 2024-03-18T22:39:54.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:54 smithi040 ceph-mon[62643]: purged_snaps scrub starts 2024-03-18T22:39:54.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:54 smithi040 ceph-mon[62643]: purged_snaps scrub ok 2024-03-18T22:39:54.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:54 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:54.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:54 smithi040 ceph-mon[62643]: osd.11 [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143] boot 2024-03-18T22:39:54.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:54 smithi040 ceph-mon[62643]: osdmap e80: 12 total, 12 up, 12 in 2024-03-18T22:39:54.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:54 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:54.614 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:54 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/207505983' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T22:39:54.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:54 smithi113 ceph-mon[66775]: purged_snaps scrub starts 2024-03-18T22:39:54.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:54 smithi113 ceph-mon[66775]: purged_snaps scrub ok 2024-03-18T22:39:54.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:54 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:54.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:54 smithi113 ceph-mon[66775]: osd.11 [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143] boot 2024-03-18T22:39:54.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:54 smithi113 ceph-mon[66775]: osdmap e80: 12 total, 12 up, 12 in 2024-03-18T22:39:54.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:54 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:54.744 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:54 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/207505983' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T22:39:54.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:54 smithi103 ceph-mon[66377]: purged_snaps scrub starts 2024-03-18T22:39:54.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:54 smithi103 ceph-mon[66377]: purged_snaps scrub ok 2024-03-18T22:39:54.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:54 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:54.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:54 smithi103 ceph-mon[66377]: osd.11 [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143] boot 2024-03-18T22:39:54.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:54 smithi103 ceph-mon[66377]: osdmap e80: 12 total, 12 up, 12 in 2024-03-18T22:39:54.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:54 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T22:39:54.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:54 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/207505983' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T22:39:55.159 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1710801593,"num_in_osds":12,"osd_in_since":1710801574,"num_remapped_pgs":0} 2024-03-18T22:39:55.160 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T22:39:55.411 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:39:55.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:55 smithi113 ceph-mon[66775]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:39:55.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:55 smithi113 ceph-mon[66775]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T22:39:55.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:55.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:55.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:55.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:55.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:55.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:55.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:55.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:55.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:55.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:55 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:55.744 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:55 smithi040 ceph-mon[62643]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:39:55.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:55 smithi040 ceph-mon[62643]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T22:39:55.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:55.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:55.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:55.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:55.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:55.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:55.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:55.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:55.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:55.745 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:55 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:55.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:55 smithi103 ceph-mon[66377]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:39:55.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:55 smithi103 ceph-mon[66377]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T22:39:55.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:55.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:55.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:55.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:55.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:55.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:39:55.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:55.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:39:55.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:39:55.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:55 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:39:56.751 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:56 smithi040 ceph-mon[62643]: Detected new or changed devices on smithi113 2024-03-18T22:39:56.751 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:56 smithi040 ceph-mon[62643]: Adjusting osd_memory_target on smithi113 to 4272M 2024-03-18T22:39:56.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:56 smithi103 ceph-mon[66377]: Detected new or changed devices on smithi113 2024-03-18T22:39:56.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:56 smithi103 ceph-mon[66377]: Adjusting osd_memory_target on smithi113 to 4272M 2024-03-18T22:39:56.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:56 smithi113 ceph-mon[66775]: Detected new or changed devices on smithi113 2024-03-18T22:39:56.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:56 smithi113 ceph-mon[66775]: Adjusting osd_memory_target on smithi113 to 4272M 2024-03-18T22:39:57.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.090+0000 7fb7f4de0700 1 -- 172.21.15.40:0/3387712405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f0102fa0 msgr2=0x7fb7f0106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.090+0000 7fb7f4de0700 1 --2- 172.21.15.40:0/3387712405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f0102fa0 0x7fb7f0106f40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0009990 tx=0x7fb7e0009ca0 comp rx=0 tx=0).stop 2024-03-18T22:39:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.090+0000 7fb7f4de0700 1 -- 172.21.15.40:0/3387712405 shutdown_connections 2024-03-18T22:39:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.090+0000 7fb7f4de0700 1 --2- 172.21.15.40:0/3387712405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7f0107620 0x7fb7f010f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.090+0000 7fb7f4de0700 1 --2- 172.21.15.40:0/3387712405 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f01025f0 0x7fb7f01029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.090+0000 7fb7f4de0700 1 --2- 172.21.15.40:0/3387712405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f0102fa0 0x7fb7f0106f40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.090+0000 7fb7f4de0700 1 -- 172.21.15.40:0/3387712405 >> 172.21.15.40:0/3387712405 conn(0x7fb7f00793a0 msgr2=0x7fb7f00797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:39:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.090+0000 7fb7f4de0700 1 -- 172.21.15.40:0/3387712405 shutdown_connections 2024-03-18T22:39:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.091+0000 7fb7f4de0700 1 -- 172.21.15.40:0/3387712405 wait complete. 2024-03-18T22:39:57.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.091+0000 7fb7f4de0700 1 Processor -- start 2024-03-18T22:39:57.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.091+0000 7fb7f4de0700 1 -- start start 2024-03-18T22:39:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7f4de0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f01025f0 0x7fb7f019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7f4de0700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0102fa0 0x7fb7f019eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7f4de0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7f0107620 0x7fb7f01987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7f4de0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f0112490 con 0x7fb7f01025f0 2024-03-18T22:39:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7f4de0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f0112310 con 0x7fb7f0107620 2024-03-18T22:39:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7f4de0700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f0112610 con 0x7fb7f0102fa0 2024-03-18T22:39:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7ee59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f01025f0 0x7fb7f019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7ee59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f01025f0 0x7fb7f019e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38610/0 (socket says 172.21.15.40:38610) 2024-03-18T22:39:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7ee59c700 1 -- 172.21.15.40:0/4235862854 learned_addr learned my addr 172.21.15.40:0/4235862854 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:39:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7eed9d700 1 --2- 172.21.15.40:0/4235862854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7f0107620 0x7fb7f01987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7edd9b700 1 --2- 172.21.15.40:0/4235862854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0102fa0 0x7fb7f019eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:57.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7ee59c700 1 -- 172.21.15.40:0/4235862854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0102fa0 msgr2=0x7fb7f019eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7ee59c700 1 --2- 172.21.15.40:0/4235862854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0102fa0 0x7fb7f019eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7ee59c700 1 -- 172.21.15.40:0/4235862854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7f0107620 msgr2=0x7fb7f01987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7ee59c700 1 --2- 172.21.15.40:0/4235862854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7f0107620 0x7fb7f01987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7ee59c700 1 -- 172.21.15.40:0/4235862854 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f0198fb0 con 0x7fb7f01025f0 2024-03-18T22:39:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.092+0000 7fb7ee59c700 1 --2- 172.21.15.40:0/4235862854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f01025f0 0x7fb7f019e650 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800c970 tx=0x7fb7d800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:39:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.093+0000 7fb7e77fe700 1 -- 172.21.15.40:0/4235862854 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d80078d0 con 0x7fb7f01025f0 2024-03-18T22:39:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.093+0000 7fb7f4de0700 1 -- 172.21.15.40:0/4235862854 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7f01992a0 con 0x7fb7f01025f0 2024-03-18T22:39:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.093+0000 7fb7e77fe700 1 -- 172.21.15.40:0/4235862854 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb7d8018480 con 0x7fb7f01025f0 2024-03-18T22:39:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.093+0000 7fb7e77fe700 1 -- 172.21.15.40:0/4235862854 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d800f6b0 con 0x7fb7f01025f0 2024-03-18T22:39:57.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.093+0000 7fb7f4de0700 1 -- 172.21.15.40:0/4235862854 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7f01a54b0 con 0x7fb7f01025f0 2024-03-18T22:39:57.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.094+0000 7fb7f4de0700 1 -- 172.21.15.40:0/4235862854 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7f0103a20 con 0x7fb7f01025f0 2024-03-18T22:39:57.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.095+0000 7fb7e77fe700 1 -- 172.21.15.40:0/4235862854 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb7d8018630 con 0x7fb7f01025f0 2024-03-18T22:39:57.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.095+0000 7fb7e77fe700 1 --2- 172.21.15.40:0/4235862854 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb7dc0a2890 0x7fb7dc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:39:57.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.096+0000 7fb7edd9b700 1 --2- 172.21.15.40:0/4235862854 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb7dc0a2890 0x7fb7dc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:39:57.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.096+0000 7fb7e77fe700 1 -- 172.21.15.40:0/4235862854 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb7d80ce980 con 0x7fb7f01025f0 2024-03-18T22:39:57.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.096+0000 7fb7edd9b700 1 --2- 172.21.15.40:0/4235862854 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb7dc0a2890 0x7fb7dc0a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000b5c0 tx=0x7fb7e0005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:39:57.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.099+0000 7fb7e77fe700 1 -- 172.21.15.40:0/4235862854 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb7d809bfe0 con 0x7fb7f01025f0 2024-03-18T22:39:57.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.315+0000 7fb7f4de0700 1 -- 172.21.15.40:0/4235862854 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7f002dbb0 con 0x7fb7f01025f0 2024-03-18T22:39:57.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.316+0000 7fb7e77fe700 1 -- 172.21.15.40:0/4235862854 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18457 (secure 0 0 0) 0x7fb7d809b8e0 con 0x7fb7f01025f0 2024-03-18T22:39:57.318 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:39:57.319 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":81,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:39:54.932203+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:39:57.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.319+0000 7fb7f4de0700 1 -- 172.21.15.40:0/4235862854 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb7dc0a2890 msgr2=0x7fb7dc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:57.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.319+0000 7fb7f4de0700 1 --2- 172.21.15.40:0/4235862854 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb7dc0a2890 0x7fb7dc0a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000b5c0 tx=0x7fb7e0005960 comp rx=0 tx=0).stop 2024-03-18T22:39:57.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.319+0000 7fb7f4de0700 1 -- 172.21.15.40:0/4235862854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f01025f0 msgr2=0x7fb7f019e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:39:57.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.319+0000 7fb7f4de0700 1 --2- 172.21.15.40:0/4235862854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f01025f0 0x7fb7f019e650 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800c970 tx=0x7fb7d800cc80 comp rx=0 tx=0).stop 2024-03-18T22:39:57.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.319+0000 7fb7f4de0700 1 -- 172.21.15.40:0/4235862854 shutdown_connections 2024-03-18T22:39:57.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.319+0000 7fb7f4de0700 1 --2- 172.21.15.40:0/4235862854 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb7dc0a2890 0x7fb7dc0a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:57.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.319+0000 7fb7f4de0700 1 --2- 172.21.15.40:0/4235862854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7f0107620 0x7fb7f01987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:57.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.319+0000 7fb7f4de0700 1 --2- 172.21.15.40:0/4235862854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f01025f0 0x7fb7f019e650 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:57.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.319+0000 7fb7f4de0700 1 --2- 172.21.15.40:0/4235862854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7f0102fa0 0x7fb7f019eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:39:57.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.319+0000 7fb7f4de0700 1 -- 172.21.15.40:0/4235862854 >> 172.21.15.40:0/4235862854 conn(0x7fb7f00793a0 msgr2=0x7fb7f01006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:39:57.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.319+0000 7fb7f4de0700 1 -- 172.21.15.40:0/4235862854 shutdown_connections 2024-03-18T22:39:57.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:39:57.319+0000 7fb7f4de0700 1 -- 172.21.15.40:0/4235862854 wait complete. 2024-03-18T22:39:57.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:57 smithi113 ceph-mon[66775]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:39:57.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:57 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/4235862854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:39:57.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:57 smithi103 ceph-mon[66377]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:39:57.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:57 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/4235862854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:39:57.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:57 smithi040 ceph-mon[62643]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:39:57.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:57 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/4235862854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:39:58.131 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T22:36:07.060691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T22:39:58.131 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-18T22:39:58.377 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:39:59.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:39:58 smithi040 ceph-mon[62643]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:39:59.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:39:58 smithi113 ceph-mon[66775]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:39:59.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:39:58 smithi103 ceph-mon[66377]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:00.078 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:00 smithi040 ceph-mon[62643]: overall HEALTH_OK 2024-03-18T22:40:00.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.174+0000 7f6efbb20700 1 -- 172.21.15.40:0/3671259748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ef4107650 msgr2=0x7f6ef410f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:00.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.174+0000 7f6efbb20700 1 --2- 172.21.15.40:0/3671259748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ef4107650 0x7f6ef410f200 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4009990 tx=0x7f6ee4009ca0 comp rx=0 tx=0).stop 2024-03-18T22:40:00.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.174+0000 7f6efbb20700 1 -- 172.21.15.40:0/3671259748 shutdown_connections 2024-03-18T22:40:00.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.174+0000 7f6efbb20700 1 --2- 172.21.15.40:0/3671259748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ef4107650 0x7f6ef410f200 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.174+0000 7f6efbb20700 1 --2- 172.21.15.40:0/3671259748 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ef406ac70 0x7f6ef406b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.174+0000 7f6efbb20700 1 --2- 172.21.15.40:0/3671259748 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ef406b620 0x7f6ef4106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.174+0000 7f6efbb20700 1 -- 172.21.15.40:0/3671259748 >> 172.21.15.40:0/3671259748 conn(0x7f6ef4077260 msgr2=0x7f6ef4077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.174+0000 7f6efbb20700 1 -- 172.21.15.40:0/3671259748 shutdown_connections 2024-03-18T22:40:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.174+0000 7f6efbb20700 1 -- 172.21.15.40:0/3671259748 wait complete. 2024-03-18T22:40:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6efbb20700 1 Processor -- start 2024-03-18T22:40:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6efbb20700 1 -- start start 2024-03-18T22:40:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6efbb20700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ef406ac70 0x7f6ef41a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6efbb20700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ef406b620 0x7f6ef41a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6efbb20700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ef4107650 0x7f6ef419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6efbb20700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef4112520 con 0x7f6ef406ac70 2024-03-18T22:40:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6efbb20700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef41123a0 con 0x7f6ef4107650 2024-03-18T22:40:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6efbb20700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef41126a0 con 0x7f6ef406b620 2024-03-18T22:40:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6ef98bc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ef406ac70 0x7f6ef41a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6ef98bc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ef406ac70 0x7f6ef41a0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50340/0 (socket says 172.21.15.40:50340) 2024-03-18T22:40:00.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6ef98bc700 1 -- 172.21.15.40:0/356814795 learned_addr learned my addr 172.21.15.40:0/356814795 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6ef90bb700 1 --2- 172.21.15.40:0/356814795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ef406b620 0x7f6ef41a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6efa0bd700 1 --2- 172.21.15.40:0/356814795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ef4107650 0x7f6ef419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6ef98bc700 1 -- 172.21.15.40:0/356814795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ef406b620 msgr2=0x7f6ef41a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6ef98bc700 1 --2- 172.21.15.40:0/356814795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ef406b620 0x7f6ef41a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6ef98bc700 1 -- 172.21.15.40:0/356814795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ef4107650 msgr2=0x7f6ef419a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6ef98bc700 1 --2- 172.21.15.40:0/356814795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ef4107650 0x7f6ef419a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6ef98bc700 1 -- 172.21.15.40:0/356814795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef419b060 con 0x7f6ef406ac70 2024-03-18T22:40:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6efa0bd700 1 --2- 172.21.15.40:0/356814795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ef4107650 0x7f6ef419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:40:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.175+0000 7f6ef98bc700 1 --2- 172.21.15.40:0/356814795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ef406ac70 0x7f6ef41a0740 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6eec007e20 tx=0x7f6eec00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.176+0000 7f6ef2ffd700 1 -- 172.21.15.40:0/356814795 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec016020 con 0x7f6ef406ac70 2024-03-18T22:40:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.176+0000 7f6ef2ffd700 1 -- 172.21.15.40:0/356814795 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6eec00ae00 con 0x7f6ef406ac70 2024-03-18T22:40:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.176+0000 7f6efbb20700 1 -- 172.21.15.40:0/356814795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef419b350 con 0x7f6ef406ac70 2024-03-18T22:40:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.176+0000 7f6ef2ffd700 1 -- 172.21.15.40:0/356814795 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec011700 con 0x7f6ef406ac70 2024-03-18T22:40:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.176+0000 7f6efbb20700 1 -- 172.21.15.40:0/356814795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef41a7610 con 0x7f6ef406ac70 2024-03-18T22:40:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.176+0000 7f6efbb20700 1 -- 172.21.15.40:0/356814795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ef4068fd0 con 0x7f6ef406ac70 2024-03-18T22:40:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.178+0000 7f6ef2ffd700 1 -- 172.21.15.40:0/356814795 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f6eec019020 con 0x7f6ef406ac70 2024-03-18T22:40:00.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.178+0000 7f6ef2ffd700 1 --2- 172.21.15.40:0/356814795 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6edc0a2890 0x7f6edc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:00.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.178+0000 7f6ef90bb700 1 --2- 172.21.15.40:0/356814795 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6edc0a2890 0x7f6edc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:00.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.178+0000 7f6ef2ffd700 1 -- 172.21.15.40:0/356814795 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6eec0ce7a0 con 0x7f6ef406ac70 2024-03-18T22:40:00.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.179+0000 7f6ef90bb700 1 --2- 172.21.15.40:0/356814795 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6edc0a2890 0x7f6edc0a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8006fd0 tx=0x7f6ee8008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:00.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.182+0000 7f6ef2ffd700 1 -- 172.21.15.40:0/356814795 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6eec09bce0 con 0x7f6ef406ac70 2024-03-18T22:40:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:00 smithi113 ceph-mon[66775]: overall HEALTH_OK 2024-03-18T22:40:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:00 smithi103 ceph-mon[66377]: overall HEALTH_OK 2024-03-18T22:40:00.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.383+0000 7f6efbb20700 1 -- 172.21.15.40:0/356814795 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6ef419be60 con 0x7f6ef406ac70 2024-03-18T22:40:00.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.383+0000 7f6ef2ffd700 1 -- 172.21.15.40:0/356814795 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f6ef419be60 con 0x7f6ef406ac70 2024-03-18T22:40:00.385 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-18T22:40:00.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.386+0000 7f6efbb20700 1 -- 172.21.15.40:0/356814795 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6edc0a2890 msgr2=0x7f6edc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:00.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.386+0000 7f6efbb20700 1 --2- 172.21.15.40:0/356814795 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6edc0a2890 0x7f6edc0a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8006fd0 tx=0x7f6ee8008040 comp rx=0 tx=0).stop 2024-03-18T22:40:00.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.386+0000 7f6efbb20700 1 -- 172.21.15.40:0/356814795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ef406ac70 msgr2=0x7f6ef41a0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:00.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.386+0000 7f6efbb20700 1 --2- 172.21.15.40:0/356814795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ef406ac70 0x7f6ef41a0740 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6eec007e20 tx=0x7f6eec00a340 comp rx=0 tx=0).stop 2024-03-18T22:40:00.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.386+0000 7f6efbb20700 1 -- 172.21.15.40:0/356814795 shutdown_connections 2024-03-18T22:40:00.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.386+0000 7f6efbb20700 1 --2- 172.21.15.40:0/356814795 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6edc0a2890 0x7f6edc0a4d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:00.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.387+0000 7f6efbb20700 1 --2- 172.21.15.40:0/356814795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ef4107650 0x7f6ef419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:00.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.387+0000 7f6efbb20700 1 --2- 172.21.15.40:0/356814795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ef406ac70 0x7f6ef41a0740 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:00.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.387+0000 7f6efbb20700 1 --2- 172.21.15.40:0/356814795 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ef406b620 0x7f6ef41a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:00.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.387+0000 7f6efbb20700 1 -- 172.21.15.40:0/356814795 >> 172.21.15.40:0/356814795 conn(0x7f6ef4077260 msgr2=0x7f6ef410d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:00.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.387+0000 7f6efbb20700 1 -- 172.21.15.40:0/356814795 shutdown_connections 2024-03-18T22:40:00.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:00.387+0000 7f6efbb20700 1 -- 172.21.15.40:0/356814795 wait complete. 2024-03-18T22:40:01.007 INFO:tasks.cephadm:Adding mds.a on smithi040 2024-03-18T22:40:01.007 INFO:tasks.cephadm:Adding mds.d on smithi040 2024-03-18T22:40:01.007 INFO:tasks.cephadm:Adding mds.g on smithi040 2024-03-18T22:40:01.007 INFO:tasks.cephadm:Adding mds.j on smithi040 2024-03-18T22:40:01.007 INFO:tasks.cephadm:Adding mds.b on smithi103 2024-03-18T22:40:01.007 INFO:tasks.cephadm:Adding mds.e on smithi103 2024-03-18T22:40:01.008 INFO:tasks.cephadm:Adding mds.h on smithi103 2024-03-18T22:40:01.008 INFO:tasks.cephadm:Adding mds.k on smithi103 2024-03-18T22:40:01.008 INFO:tasks.cephadm:Adding mds.c on smithi113 2024-03-18T22:40:01.008 INFO:tasks.cephadm:Adding mds.f on smithi113 2024-03-18T22:40:01.008 INFO:tasks.cephadm:Adding mds.i on smithi113 2024-03-18T22:40:01.008 INFO:tasks.cephadm:Adding mds.l on smithi113 2024-03-18T22:40:01.008 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l' 2024-03-18T22:40:01.265 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:40:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:01 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/356814795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:40:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:01 smithi113 ceph-mon[66775]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:01.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:01 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/356814795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:40:01.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:01 smithi103 ceph-mon[66377]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:01.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:01 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/356814795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:40:01.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:01 smithi040 ceph-mon[62643]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:02.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.953+0000 7fdc40b6e700 1 -- 172.21.15.113:0/2288620051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c102370 msgr2=0x7fdc3c102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:02.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.953+0000 7fdc40b6e700 1 --2- 172.21.15.113:0/2288620051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c102370 0x7fdc3c102790 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009990 tx=0x7fdc2c009ca0 comp rx=0 tx=0).stop 2024-03-18T22:40:02.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.953+0000 7fdc40b6e700 1 -- 172.21.15.113:0/2288620051 shutdown_connections 2024-03-18T22:40:02.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.953+0000 7fdc40b6e700 1 --2- 172.21.15.113:0/2288620051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc3c103f10 0x7fdc3c10a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:02.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.953+0000 7fdc40b6e700 1 --2- 172.21.15.113:0/2288620051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c102370 0x7fdc3c102790 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:02.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.953+0000 7fdc40b6e700 1 --2- 172.21.15.113:0/2288620051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc3c103570 0x7fdc3c1039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:02.954 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.953+0000 7fdc40b6e700 1 -- 172.21.15.113:0/2288620051 >> 172.21.15.113:0/2288620051 conn(0x7fdc3c0fd910 msgr2=0x7fdc3c0ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:02.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.953+0000 7fdc40b6e700 1 -- 172.21.15.113:0/2288620051 shutdown_connections 2024-03-18T22:40:02.955 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.953+0000 7fdc40b6e700 1 -- 172.21.15.113:0/2288620051 wait complete. 2024-03-18T22:40:02.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.953+0000 7fdc40b6e700 1 Processor -- start 2024-03-18T22:40:02.956 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc40b6e700 1 -- start start 2024-03-18T22:40:02.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc40b6e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c102370 0x7fdc3c0760b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:02.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc40b6e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc3c103570 0x7fdc3c074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:02.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc40b6e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc3c103f10 0x7fdc3c074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:02.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc40b6e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c10ddf0 con 0x7fdc3c103f10 2024-03-18T22:40:02.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc40b6e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c10dc70 con 0x7fdc3c103570 2024-03-18T22:40:02.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc40b6e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c10df70 con 0x7fdc3c102370 2024-03-18T22:40:02.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc3a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c102370 0x7fdc3c0760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:02.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc3a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c102370 0x7fdc3c0760b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.113:59226/0 (socket says 172.21.15.113:59226) 2024-03-18T22:40:02.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc3a59c700 1 -- 172.21.15.113:0/1088039203 learned_addr learned my addr 172.21.15.113:0/1088039203 (peer_addr_for_me v2:172.21.15.113:0/0) 2024-03-18T22:40:02.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc3ad9d700 1 --2- 172.21.15.113:0/1088039203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc3c103f10 0x7fdc3c074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:02.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc3a59c700 1 -- 172.21.15.113:0/1088039203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc3c103570 msgr2=0x7fdc3c074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:02.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc3a59c700 1 --2- 172.21.15.113:0/1088039203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc3c103570 0x7fdc3c074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:02.957 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc3a59c700 1 -- 172.21.15.113:0/1088039203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc3c103f10 msgr2=0x7fdc3c074c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:02.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc3a59c700 1 --2- 172.21.15.113:0/1088039203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc3c103f10 0x7fdc3c074c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:02.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc3a59c700 1 -- 172.21.15.113:0/1088039203 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3c075180 con 0x7fdc3c102370 2024-03-18T22:40:02.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc3ad9d700 1 --2- 172.21.15.113:0/1088039203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc3c103f10 0x7fdc3c074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:40:02.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc3a59c700 1 --2- 172.21.15.113:0/1088039203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c102370 0x7fdc3c0760b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c003920 tx=0x7fdc2c0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:02.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc2b7fe700 1 -- 172.21.15.113:0/1088039203 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c003c90 con 0x7fdc3c102370 2024-03-18T22:40:02.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc2b7fe700 1 -- 172.21.15.113:0/1088039203 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc2c004830 con 0x7fdc3c102370 2024-03-18T22:40:02.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc40b6e700 1 -- 172.21.15.113:0/1088039203 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc3c075410 con 0x7fdc3c102370 2024-03-18T22:40:02.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.954+0000 7fdc2b7fe700 1 -- 172.21.15.113:0/1088039203 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c005480 con 0x7fdc3c102370 2024-03-18T22:40:02.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.955+0000 7fdc40b6e700 1 -- 172.21.15.113:0/1088039203 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc3c1a6e60 con 0x7fdc3c102370 2024-03-18T22:40:02.958 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.956+0000 7fdc40b6e700 1 -- 172.21.15.113:0/1088039203 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc3c068fd0 con 0x7fdc3c102370 2024-03-18T22:40:02.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.957+0000 7fdc2b7fe700 1 -- 172.21.15.113:0/1088039203 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fdc2c003e30 con 0x7fdc3c102370 2024-03-18T22:40:02.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.957+0000 7fdc2b7fe700 1 --2- 172.21.15.113:0/1088039203 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc240a2890 0x7fdc240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:02.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.957+0000 7fdc2b7fe700 1 -- 172.21.15.113:0/1088039203 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fdc2c0cfbf0 con 0x7fdc3c102370 2024-03-18T22:40:02.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.957+0000 7fdc39d9b700 1 --2- 172.21.15.113:0/1088039203 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc240a2890 0x7fdc240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:02.959 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.958+0000 7fdc39d9b700 1 --2- 172.21.15.113:0/1088039203 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc240a2890 0x7fdc240a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdc30005fd0 tx=0x7fdc30005d00 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:02.961 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:02.961+0000 7fdc2b7fe700 1 -- 172.21.15.113:0/1088039203 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdc2c013070 con 0x7fdc3c102370 2024-03-18T22:40:03.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:02 smithi113 ceph-mon[66775]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:03.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:02 smithi040 ceph-mon[62643]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:03.169 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:03.168+0000 7fdc40b6e700 1 -- 172.21.15.113:0/1088039203 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l", "target": ["mon-mgr", ""]}) -- 0x7fdc3c062910 con 0x7fdc240a2890 2024-03-18T22:40:03.177 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:03.176+0000 7fdc2b7fe700 1 -- 172.21.15.113:0/1088039203 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fdc3c062910 con 0x7fdc240a2890 2024-03-18T22:40:03.177 INFO:teuthology.orchestra.run.smithi113.stdout:Scheduled mds.all update... 2024-03-18T22:40:03.179 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:03.178+0000 7fdc40b6e700 1 -- 172.21.15.113:0/1088039203 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc240a2890 msgr2=0x7fdc240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:03.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:03.178+0000 7fdc40b6e700 1 --2- 172.21.15.113:0/1088039203 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc240a2890 0x7fdc240a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdc30005fd0 tx=0x7fdc30005d00 comp rx=0 tx=0).stop 2024-03-18T22:40:03.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:03.178+0000 7fdc40b6e700 1 -- 172.21.15.113:0/1088039203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c102370 msgr2=0x7fdc3c0760b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:03.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:03.178+0000 7fdc40b6e700 1 --2- 172.21.15.113:0/1088039203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c102370 0x7fdc3c0760b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c003920 tx=0x7fdc2c0047b0 comp rx=0 tx=0).stop 2024-03-18T22:40:03.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:03.178+0000 7fdc40b6e700 1 -- 172.21.15.113:0/1088039203 shutdown_connections 2024-03-18T22:40:03.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:03.178+0000 7fdc40b6e700 1 --2- 172.21.15.113:0/1088039203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc3c103f10 0x7fdc3c074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:03.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:03.179+0000 7fdc40b6e700 1 --2- 172.21.15.113:0/1088039203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc3c102370 0x7fdc3c0760b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:03.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:03.179+0000 7fdc40b6e700 1 --2- 172.21.15.113:0/1088039203 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc240a2890 0x7fdc240a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:03.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:03.179+0000 7fdc40b6e700 1 --2- 172.21.15.113:0/1088039203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc3c103570 0x7fdc3c074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:03.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:03.179+0000 7fdc40b6e700 1 -- 172.21.15.113:0/1088039203 >> 172.21.15.113:0/1088039203 conn(0x7fdc3c0fd910 msgr2=0x7fdc3c106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:03.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:03.179+0000 7fdc40b6e700 1 -- 172.21.15.113:0/1088039203 shutdown_connections 2024-03-18T22:40:03.180 INFO:teuthology.orchestra.run.smithi113.stderr:2024-03-18T22:40:03.179+0000 7fdc40b6e700 1 -- 172.21.15.113:0/1088039203 wait complete. 2024-03-18T22:40:03.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:02 smithi103 ceph-mon[66377]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:03.842 DEBUG:teuthology.orchestra.run.smithi040:mds.a> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.a.service 2024-03-18T22:40:03.844 DEBUG:teuthology.orchestra.run.smithi040:mds.d> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.d.service 2024-03-18T22:40:03.887 DEBUG:teuthology.orchestra.run.smithi040:mds.g> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.g.service 2024-03-18T22:40:03.931 DEBUG:teuthology.orchestra.run.smithi040:mds.j> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.j.service 2024-03-18T22:40:03.974 DEBUG:teuthology.orchestra.run.smithi103:mds.b> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.b.service 2024-03-18T22:40:03.977 DEBUG:teuthology.orchestra.run.smithi103:mds.e> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.e.service 2024-03-18T22:40:04.020 DEBUG:teuthology.orchestra.run.smithi103:mds.h> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.h.service 2024-03-18T22:40:04.063 DEBUG:teuthology.orchestra.run.smithi103:mds.k> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.k.service 2024-03-18T22:40:04.106 DEBUG:teuthology.orchestra.run.smithi113:mds.c> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.c.service 2024-03-18T22:40:04.108 DEBUG:teuthology.orchestra.run.smithi113:mds.f> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.f.service 2024-03-18T22:40:04.151 DEBUG:teuthology.orchestra.run.smithi113:mds.i> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.i.service 2024-03-18T22:40:04.194 DEBUG:teuthology.orchestra.run.smithi113:mds.l> sudo journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.l.service 2024-03-18T22:40:04.237 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-18T22:40:04.238 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:40:04.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.348+0000 7fc226061640 1 -- 172.21.15.40:0/908258622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc220101840 msgr2=0x7fc220101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:04.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.348+0000 7fc226061640 1 --2- 172.21.15.40:0/908258622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc220101840 0x7fc220101c60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc210009900 tx=0x7fc21002bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:04.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.349+0000 7fc226061640 1 -- 172.21.15.40:0/908258622 shutdown_connections 2024-03-18T22:40:04.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.349+0000 7fc226061640 1 --2- 172.21.15.40:0/908258622 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2201033f0 0x7fc220109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.349+0000 7fc226061640 1 --2- 172.21.15.40:0/908258622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc220102a30 0x7fc220102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.349+0000 7fc226061640 1 --2- 172.21.15.40:0/908258622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc220101840 0x7fc220101c60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.349+0000 7fc226061640 1 -- 172.21.15.40:0/908258622 >> 172.21.15.40:0/908258622 conn(0x7fc2200fcdd0 msgr2=0x7fc2200ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:04.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.349+0000 7fc226061640 1 -- 172.21.15.40:0/908258622 shutdown_connections 2024-03-18T22:40:04.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.349+0000 7fc226061640 1 -- 172.21.15.40:0/908258622 wait complete. 2024-03-18T22:40:04.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.349+0000 7fc226061640 1 Processor -- start 2024-03-18T22:40:04.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.349+0000 7fc226061640 1 -- start start 2024-03-18T22:40:04.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.349+0000 7fc226061640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc220101840 0x7fc220197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.349+0000 7fc226061640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc220102a30 0x7fc220197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.349+0000 7fc226061640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2201033f0 0x7fc22019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.349+0000 7fc226061640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc22010d360 con 0x7fc220101840 2024-03-18T22:40:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.349+0000 7fc226061640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc22010d1e0 con 0x7fc220102a30 2024-03-18T22:40:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.349+0000 7fc226061640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc22010d4e0 con 0x7fc2201033f0 2024-03-18T22:40:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc21f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc220101840 0x7fc220197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc21f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc220101840 0x7fc220197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50354/0 (socket says 172.21.15.40:50354) 2024-03-18T22:40:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc21f7fe640 1 -- 172.21.15.40:0/1910954878 learned_addr learned my addr 172.21.15.40:0/1910954878 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc21effd640 1 --2- 172.21.15.40:0/1910954878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc220102a30 0x7fc220197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc21ffff640 1 --2- 172.21.15.40:0/1910954878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2201033f0 0x7fc22019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc21f7fe640 1 -- 172.21.15.40:0/1910954878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2201033f0 msgr2=0x7fc22019f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc21f7fe640 1 --2- 172.21.15.40:0/1910954878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2201033f0 0x7fc22019f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc21f7fe640 1 -- 172.21.15.40:0/1910954878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc220102a30 msgr2=0x7fc220197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc21f7fe640 1 --2- 172.21.15.40:0/1910954878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc220102a30 0x7fc220197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc21f7fe640 1 -- 172.21.15.40:0/1910954878 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22019f6b0 con 0x7fc220101840 2024-03-18T22:40:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc21f7fe640 1 --2- 172.21.15.40:0/1910954878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc220101840 0x7fc220197a90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc210000c00 tx=0x7fc210005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc21cff9640 1 -- 172.21.15.40:0/1910954878 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210043020 con 0x7fc220101840 2024-03-18T22:40:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc226061640 1 -- 172.21.15.40:0/1910954878 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc22019f940 con 0x7fc220101840 2024-03-18T22:40:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc21cff9640 1 -- 172.21.15.40:0/1910954878 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc210004350 con 0x7fc220101840 2024-03-18T22:40:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc21cff9640 1 -- 172.21.15.40:0/1910954878 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc210003b80 con 0x7fc220101840 2024-03-18T22:40:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.350+0000 7fc226061640 1 -- 172.21.15.40:0/1910954878 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc22019fe90 con 0x7fc220101840 2024-03-18T22:40:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.351+0000 7fc226061640 1 -- 172.21.15.40:0/1910954878 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc220108c30 con 0x7fc220101840 2024-03-18T22:40:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.352+0000 7fc21cff9640 1 -- 172.21.15.40:0/1910954878 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc210046020 con 0x7fc220101840 2024-03-18T22:40:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.353+0000 7fc21cff9640 1 --2- 172.21.15.40:0/1910954878 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2080a2800 0x7fc2080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.353+0000 7fc21effd640 1 --2- 172.21.15.40:0/1910954878 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2080a2800 0x7fc2080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.353+0000 7fc21cff9640 1 -- 172.21.15.40:0/1910954878 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc2100f0aa0 con 0x7fc220101840 2024-03-18T22:40:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.353+0000 7fc21effd640 1 --2- 172.21.15.40:0/1910954878 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2080a2800 0x7fc2080a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc220198fa0 tx=0x7fc214005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:04.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.357+0000 7fc21cff9640 1 -- 172.21.15.40:0/1910954878 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc21004d020 con 0x7fc220101840 2024-03-18T22:40:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:04 smithi103 ceph-mon[66377]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:40:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:04 smithi103 ceph-mon[66377]: Saving service mds.all spec with placement smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l;count:12 2024-03-18T22:40:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:40:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:40:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:04 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:04.447 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:04 smithi113 ceph-mon[66775]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:40:04.447 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:04 smithi113 ceph-mon[66775]: Saving service mds.all spec with placement smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l;count:12 2024-03-18T22:40:04.447 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:04.447 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:40:04.448 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:04.448 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:40:04.448 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:04.448 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:04.448 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:04.448 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:04 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:04.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.541+0000 7fc226061640 1 -- 172.21.15.40:0/1910954878 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc220062420 con 0x7fc220101840 2024-03-18T22:40:04.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.542+0000 7fc21cff9640 1 -- 172.21.15.40:0/1910954878 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18457 (secure 0 0 0) 0x7fc2100bdfe0 con 0x7fc220101840 2024-03-18T22:40:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:04.544 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":81,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:39:54.932203+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:40:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.544+0000 7fc226061640 1 -- 172.21.15.40:0/1910954878 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2080a2800 msgr2=0x7fc2080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.544+0000 7fc226061640 1 --2- 172.21.15.40:0/1910954878 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2080a2800 0x7fc2080a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc220198fa0 tx=0x7fc214005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.544+0000 7fc226061640 1 -- 172.21.15.40:0/1910954878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc220101840 msgr2=0x7fc220197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.544+0000 7fc226061640 1 --2- 172.21.15.40:0/1910954878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc220101840 0x7fc220197a90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc210000c00 tx=0x7fc210005dc0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.544+0000 7fc226061640 1 -- 172.21.15.40:0/1910954878 shutdown_connections 2024-03-18T22:40:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.544+0000 7fc226061640 1 --2- 172.21.15.40:0/1910954878 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2080a2800 0x7fc2080a4cc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.544+0000 7fc226061640 1 --2- 172.21.15.40:0/1910954878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2201033f0 0x7fc22019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.544+0000 7fc226061640 1 --2- 172.21.15.40:0/1910954878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc220102a30 0x7fc220197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.544+0000 7fc226061640 1 --2- 172.21.15.40:0/1910954878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc220101840 0x7fc220197a90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.544+0000 7fc226061640 1 -- 172.21.15.40:0/1910954878 >> 172.21.15.40:0/1910954878 conn(0x7fc2200fcdd0 msgr2=0x7fc220105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:04.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.544+0000 7fc226061640 1 -- 172.21.15.40:0/1910954878 shutdown_connections 2024-03-18T22:40:04.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.544+0000 7fc226061640 1 -- 172.21.15.40:0/1910954878 wait complete. 2024-03-18T22:40:04.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:04 smithi040 ceph-mon[62643]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:40:04.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:04 smithi040 ceph-mon[62643]: Saving service mds.all spec with placement smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l;count:12 2024-03-18T22:40:04.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:04.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:40:04.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:04.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:40:04.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:04.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:04.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:04.556 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:04 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:04.558 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T22:36:07.060691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T22:40:04.558 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T22:40:04.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.662+0000 7fc8e0b90640 1 -- 172.21.15.40:0/3976308737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8dc101910 msgr2=0x7fc8dc101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:04.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.662+0000 7fc8e0b90640 1 --2- 172.21.15.40:0/3976308737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8dc101910 0x7fc8dc101d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009900 tx=0x7fc8c402bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:04.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.662+0000 7fc8e0b90640 1 -- 172.21.15.40:0/3976308737 shutdown_connections 2024-03-18T22:40:04.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.662+0000 7fc8e0b90640 1 --2- 172.21.15.40:0/3976308737 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8dc1022d0 0x7fc8dc108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.662+0000 7fc8e0b90640 1 --2- 172.21.15.40:0/3976308737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8dc101910 0x7fc8dc101d90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.662+0000 7fc8e0b90640 1 --2- 172.21.15.40:0/3976308737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8dc100720 0x7fc8dc100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.662+0000 7fc8e0b90640 1 -- 172.21.15.40:0/3976308737 >> 172.21.15.40:0/3976308737 conn(0x7fc8dc0fbc90 msgr2=0x7fc8dc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:04.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.662+0000 7fc8e0b90640 1 -- 172.21.15.40:0/3976308737 shutdown_connections 2024-03-18T22:40:04.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.662+0000 7fc8e0b90640 1 -- 172.21.15.40:0/3976308737 wait complete. 2024-03-18T22:40:04.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.662+0000 7fc8e0b90640 1 Processor -- start 2024-03-18T22:40:04.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8e0b90640 1 -- start start 2024-03-18T22:40:04.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8e0b90640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8dc100720 0x7fc8dc196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:04.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8e0b90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8dc101910 0x7fc8dc196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:04.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8e0b90640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8dc1022d0 0x7fc8dc19ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8e0b90640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc10c230 con 0x7fc8dc101910 2024-03-18T22:40:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8e0b90640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc10c0b0 con 0x7fc8dc1022d0 2024-03-18T22:40:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8e0b90640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc10c3b0 con 0x7fc8dc100720 2024-03-18T22:40:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8d9d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8dc101910 0x7fc8dc196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8d9d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8dc101910 0x7fc8dc196e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50378/0 (socket says 172.21.15.40:50378) 2024-03-18T22:40:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8d9d9b640 1 -- 172.21.15.40:0/2792902933 learned_addr learned my addr 172.21.15.40:0/2792902933 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8dad9d640 1 --2- 172.21.15.40:0/2792902933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8dc1022d0 0x7fc8dc19ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8da59c640 1 --2- 172.21.15.40:0/2792902933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8dc100720 0x7fc8dc196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8d9d9b640 1 -- 172.21.15.40:0/2792902933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8dc100720 msgr2=0x7fc8dc196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8d9d9b640 1 --2- 172.21.15.40:0/2792902933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8dc100720 0x7fc8dc196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8d9d9b640 1 -- 172.21.15.40:0/2792902933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8dc1022d0 msgr2=0x7fc8dc19ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8d9d9b640 1 --2- 172.21.15.40:0/2792902933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8dc1022d0 0x7fc8dc19ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8d9d9b640 1 -- 172.21.15.40:0/2792902933 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8dc19e540 con 0x7fc8dc101910 2024-03-18T22:40:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8d9d9b640 1 --2- 172.21.15.40:0/2792902933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8dc101910 0x7fc8dc196e60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4006010 tx=0x7fc8c4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:04.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.663+0000 7fc8d37fe640 1 -- 172.21.15.40:0/2792902933 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4043020 con 0x7fc8dc101910 2024-03-18T22:40:04.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.664+0000 7fc8e0b90640 1 -- 172.21.15.40:0/2792902933 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8dc19e7d0 con 0x7fc8dc101910 2024-03-18T22:40:04.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.664+0000 7fc8d37fe640 1 -- 172.21.15.40:0/2792902933 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc8c4004440 con 0x7fc8dc101910 2024-03-18T22:40:04.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.664+0000 7fc8d37fe640 1 -- 172.21.15.40:0/2792902933 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4003bf0 con 0x7fc8dc101910 2024-03-18T22:40:04.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.664+0000 7fc8e0b90640 1 -- 172.21.15.40:0/2792902933 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8dc19ed70 con 0x7fc8dc101910 2024-03-18T22:40:04.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.664+0000 7fc8e0b90640 1 -- 172.21.15.40:0/2792902933 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8dc107b30 con 0x7fc8dc101910 2024-03-18T22:40:04.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.666+0000 7fc8d37fe640 1 -- 172.21.15.40:0/2792902933 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc8c4046020 con 0x7fc8dc101910 2024-03-18T22:40:04.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.666+0000 7fc8d37fe640 1 --2- 172.21.15.40:0/2792902933 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8c80a2800 0x7fc8c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:04.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.666+0000 7fc8da59c640 1 --2- 172.21.15.40:0/2792902933 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8c80a2800 0x7fc8c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:04.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.666+0000 7fc8d37fe640 1 -- 172.21.15.40:0/2792902933 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc8c40f0f00 con 0x7fc8dc101910 2024-03-18T22:40:04.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.667+0000 7fc8da59c640 1 --2- 172.21.15.40:0/2792902933 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8c80a2800 0x7fc8c80a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc101770 tx=0x7fc8d4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:04.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.670+0000 7fc8d37fe640 1 -- 172.21.15.40:0/2792902933 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc8c404d020 con 0x7fc8dc101910 2024-03-18T22:40:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.853+0000 7fc8e0b90640 1 -- 172.21.15.40:0/2792902933 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc8dc02d1f0 con 0x7fc8dc101910 2024-03-18T22:40:04.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.854+0000 7fc8d37fe640 1 -- 172.21.15.40:0/2792902933 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fc8c40be440 con 0x7fc8dc101910 2024-03-18T22:40:04.855 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-18T22:40:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.856+0000 7fc8e0b90640 1 -- 172.21.15.40:0/2792902933 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8c80a2800 msgr2=0x7fc8c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.856+0000 7fc8e0b90640 1 --2- 172.21.15.40:0/2792902933 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8c80a2800 0x7fc8c80a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc101770 tx=0x7fc8d4005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:04.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.856+0000 7fc8e0b90640 1 -- 172.21.15.40:0/2792902933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8dc101910 msgr2=0x7fc8dc196e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:04.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.856+0000 7fc8e0b90640 1 --2- 172.21.15.40:0/2792902933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8dc101910 0x7fc8dc196e60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4006010 tx=0x7fc8c4004040 comp rx=0 tx=0).stop 2024-03-18T22:40:04.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.856+0000 7fc8e0b90640 1 -- 172.21.15.40:0/2792902933 shutdown_connections 2024-03-18T22:40:04.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.856+0000 7fc8e0b90640 1 --2- 172.21.15.40:0/2792902933 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8c80a2800 0x7fc8c80a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.856+0000 7fc8e0b90640 1 --2- 172.21.15.40:0/2792902933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8dc1022d0 0x7fc8dc19ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.856+0000 7fc8e0b90640 1 --2- 172.21.15.40:0/2792902933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8dc101910 0x7fc8dc196e60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.856+0000 7fc8e0b90640 1 --2- 172.21.15.40:0/2792902933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8dc100720 0x7fc8dc196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.856+0000 7fc8e0b90640 1 -- 172.21.15.40:0/2792902933 >> 172.21.15.40:0/2792902933 conn(0x7fc8dc0fbc90 msgr2=0x7fc8dc104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:04.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.856+0000 7fc8e0b90640 1 -- 172.21.15.40:0/2792902933 shutdown_connections 2024-03-18T22:40:04.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.856+0000 7fc8e0b90640 1 -- 172.21.15.40:0/2792902933 wait complete. 2024-03-18T22:40:04.868 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:04.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.969+0000 7f4ce433b640 1 -- 172.21.15.40:0/3539053507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cdc101950 msgr2=0x7f4cdc101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:04.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.969+0000 7f4ce433b640 1 --2- 172.21.15.40:0/3539053507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cdc101950 0x7f4cdc101dd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009900 tx=0x7f4ccc02bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.969+0000 7f4ce433b640 1 -- 172.21.15.40:0/3539053507 shutdown_connections 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.969+0000 7f4ce433b640 1 --2- 172.21.15.40:0/3539053507 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cdc102310 0x7f4cdc108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.969+0000 7f4ce433b640 1 --2- 172.21.15.40:0/3539053507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cdc101950 0x7f4cdc101dd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.969+0000 7f4ce433b640 1 --2- 172.21.15.40:0/3539053507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cdc100760 0x7f4cdc100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.969+0000 7f4ce433b640 1 -- 172.21.15.40:0/3539053507 >> 172.21.15.40:0/3539053507 conn(0x7f4cdc0fbcd0 msgr2=0x7f4cdc0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.969+0000 7f4ce433b640 1 -- 172.21.15.40:0/3539053507 shutdown_connections 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.969+0000 7f4ce433b640 1 -- 172.21.15.40:0/3539053507 wait complete. 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.969+0000 7f4ce433b640 1 Processor -- start 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.969+0000 7f4ce433b640 1 -- start start 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce433b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cdc100760 0x7f4cdc06a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce433b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cdc101950 0x7f4cdc19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce433b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cdc102310 0x7f4cdc19eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce433b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc10c290 con 0x7f4cdc101950 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce433b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc10c110 con 0x7f4cdc100760 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce433b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc10c410 con 0x7f4cdc102310 2024-03-18T22:40:04.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce18d6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cdc101950 0x7f4cdc19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce18d6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cdc101950 0x7f4cdc19c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50402/0 (socket says 172.21.15.40:50402) 2024-03-18T22:40:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce18d6640 1 -- 172.21.15.40:0/810363063 learned_addr learned my addr 172.21.15.40:0/810363063 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce20d7640 1 --2- 172.21.15.40:0/810363063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cdc100760 0x7f4cdc06a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce28d8640 1 --2- 172.21.15.40:0/810363063 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cdc102310 0x7f4cdc19eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce18d6640 1 -- 172.21.15.40:0/810363063 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cdc102310 msgr2=0x7f4cdc19eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce18d6640 1 --2- 172.21.15.40:0/810363063 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cdc102310 0x7f4cdc19eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce18d6640 1 -- 172.21.15.40:0/810363063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cdc100760 msgr2=0x7f4cdc06a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce18d6640 1 --2- 172.21.15.40:0/810363063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cdc100760 0x7f4cdc06a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce18d6640 1 -- 172.21.15.40:0/810363063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cdc19f6b0 con 0x7f4cdc101950 2024-03-18T22:40:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce18d6640 1 --2- 172.21.15.40:0/810363063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cdc101950 0x7f4cdc19c780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc006010 tx=0x7f4ccc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4cd37fe640 1 -- 172.21.15.40:0/810363063 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc043020 con 0x7f4cdc101950 2024-03-18T22:40:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4ce433b640 1 -- 172.21.15.40:0/810363063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cdc19f8e0 con 0x7f4cdc101950 2024-03-18T22:40:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4cd37fe640 1 -- 172.21.15.40:0/810363063 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ccc004440 con 0x7f4cdc101950 2024-03-18T22:40:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.970+0000 7f4cd37fe640 1 -- 172.21.15.40:0/810363063 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ccc003bf0 con 0x7f4cdc101950 2024-03-18T22:40:04.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.971+0000 7f4ce433b640 1 -- 172.21.15.40:0/810363063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cdc19fe30 con 0x7f4cdc101950 2024-03-18T22:40:04.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.971+0000 7f4ce433b640 1 -- 172.21.15.40:0/810363063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cdc068a60 con 0x7f4cdc101950 2024-03-18T22:40:04.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.972+0000 7f4cd37fe640 1 -- 172.21.15.40:0/810363063 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4ccc046020 con 0x7f4cdc101950 2024-03-18T22:40:04.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.973+0000 7f4cd37fe640 1 --2- 172.21.15.40:0/810363063 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4cc80a2800 0x7f4cc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:04.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.973+0000 7f4ce20d7640 1 --2- 172.21.15.40:0/810363063 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4cc80a2800 0x7f4cc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:04.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.973+0000 7f4cd37fe640 1 -- 172.21.15.40:0/810363063 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4ccc0f08d0 con 0x7f4cdc101950 2024-03-18T22:40:04.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.973+0000 7f4ce20d7640 1 --2- 172.21.15.40:0/810363063 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4cc80a2800 0x7f4cc80a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc1017b0 tx=0x7f4cd8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:04.977+0000 7f4cd37fe640 1 -- 172.21.15.40:0/810363063 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4ccc04d020 con 0x7f4cdc101950 2024-03-18T22:40:05.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:05.205+0000 7f4ce433b640 1 -- 172.21.15.40:0/810363063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cdc107a80 con 0x7f4cdc101950 2024-03-18T22:40:05.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:05.205+0000 7f4cd37fe640 1 -- 172.21.15.40:0/810363063 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f4ccc0bde10 con 0x7f4cdc101950 2024-03-18T22:40:05.207 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:05.207 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-18T22:40:05.207 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 1 2024-03-18T22:40:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:05.207+0000 7f4ce433b640 1 -- 172.21.15.40:0/810363063 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4cc80a2800 msgr2=0x7f4cc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:05.207+0000 7f4ce433b640 1 --2- 172.21.15.40:0/810363063 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4cc80a2800 0x7f4cc80a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc1017b0 tx=0x7f4cd8005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:05.207+0000 7f4ce433b640 1 -- 172.21.15.40:0/810363063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cdc101950 msgr2=0x7f4cdc19c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:05.207+0000 7f4ce433b640 1 --2- 172.21.15.40:0/810363063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cdc101950 0x7f4cdc19c780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc006010 tx=0x7f4ccc004040 comp rx=0 tx=0).stop 2024-03-18T22:40:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:05.207+0000 7f4ce433b640 1 -- 172.21.15.40:0/810363063 shutdown_connections 2024-03-18T22:40:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:05.207+0000 7f4ce433b640 1 --2- 172.21.15.40:0/810363063 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4cc80a2800 0x7f4cc80a4cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:05.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:05.207+0000 7f4ce433b640 1 --2- 172.21.15.40:0/810363063 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4cdc102310 0x7f4cdc19eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:05.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:05.207+0000 7f4ce433b640 1 --2- 172.21.15.40:0/810363063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cdc101950 0x7f4cdc19c780 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:05.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:05.207+0000 7f4ce433b640 1 --2- 172.21.15.40:0/810363063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cdc100760 0x7f4cdc06a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:05.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:05.207+0000 7f4ce433b640 1 -- 172.21.15.40:0/810363063 >> 172.21.15.40:0/810363063 conn(0x7f4cdc0fbcd0 msgr2=0x7f4cdc104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:05.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:05.208+0000 7f4ce433b640 1 -- 172.21.15.40:0/810363063 shutdown_connections 2024-03-18T22:40:05.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:05.208+0000 7f4ce433b640 1 -- 172.21.15.40:0/810363063 wait complete. 2024-03-18T22:40:05.446 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:05 smithi113 ceph-mon[66775]: Deploying daemon mds.l on smithi113 2024-03-18T22:40:05.447 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:05 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1910954878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:05.447 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:05 smithi113 ceph-mon[66775]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:05.447 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:05 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2792902933' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:40:05.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:05 smithi103 ceph-mon[66377]: Deploying daemon mds.l on smithi113 2024-03-18T22:40:05.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:05 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1910954878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:05.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:05 smithi103 ceph-mon[66377]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:05.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:05 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2792902933' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:40:05.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:05 smithi040 ceph-mon[62643]: Deploying daemon mds.l on smithi113 2024-03-18T22:40:05.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:05 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1910954878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:05.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:05 smithi040 ceph-mon[62643]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:05.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:05 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2792902933' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:40:06.080 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 22:40:05 smithi113 systemd[1]: Starting Ceph mds.l for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:40:06.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:06 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/810363063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:06.418 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:06 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/810363063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:06.419 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 22:40:06 smithi113 podman[97474]: 2024-03-18 22:40:06.056375507 +0000 UTC m=+0.018378538 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T22:40:06.419 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 22:40:06 smithi113 podman[97474]: 2024-03-18 22:40:06.165714306 +0000 UTC m=+0.127717380 container create 382717cf9b9cc87d190d47aebb02266367fdc1344d08baa55c8cee14f5a1434d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-18T22:40:06.419 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 22:40:06 smithi113 podman[97474]: 2024-03-18 22:40:06.418514647 +0000 UTC m=+0.380517678 container init 382717cf9b9cc87d190d47aebb02266367fdc1344d08baa55c8cee14f5a1434d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-18T22:40:06.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:06 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/810363063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:06.830 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 22:40:06 smithi113 podman[97474]: 2024-03-18 22:40:06.421328757 +0000 UTC m=+0.383331798 container start 382717cf9b9cc87d190d47aebb02266367fdc1344d08baa55c8cee14f5a1434d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-l, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-18T22:40:06.830 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 22:40:06 smithi113 ceph-mds[97493]: -- 172.21.15.113:0/2758356830 <== mon.1 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5651feacba40 con 0x5651fea9ec00 2024-03-18T22:40:06.830 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 22:40:06 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-l[97489]: starting mds.l at 2024-03-18T22:40:06.830 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 22:40:06 smithi113 bash[97474]: 382717cf9b9cc87d190d47aebb02266367fdc1344d08baa55c8cee14f5a1434d 2024-03-18T22:40:06.830 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 22:40:06 smithi113 systemd[1]: Started Ceph mds.l for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:40:07.221 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:07.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.336+0000 7f1a3613b640 1 -- 172.21.15.40:0/3395093753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a30102b00 msgr2=0x7f1a30102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:07.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.336+0000 7f1a3613b640 1 --2- 172.21.15.40:0/3395093753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a30102b00 0x7f1a30102f80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009930 tx=0x7f1a2402bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:07.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.336+0000 7f1a3613b640 1 -- 172.21.15.40:0/3395093753 shutdown_connections 2024-03-18T22:40:07.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.336+0000 7f1a3613b640 1 --2- 172.21.15.40:0/3395093753 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a301034c0 0x7f1a30109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:07.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.336+0000 7f1a3613b640 1 --2- 172.21.15.40:0/3395093753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a30102b00 0x7f1a30102f80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:07.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.336+0000 7f1a3613b640 1 --2- 172.21.15.40:0/3395093753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a300ffb30 0x7f1a300fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:07.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.336+0000 7f1a3613b640 1 -- 172.21.15.40:0/3395093753 >> 172.21.15.40:0/3395093753 conn(0x7f1a300fb460 msgr2=0x7f1a300fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:07.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.336+0000 7f1a3613b640 1 -- 172.21.15.40:0/3395093753 shutdown_connections 2024-03-18T22:40:07.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.336+0000 7f1a3613b640 1 -- 172.21.15.40:0/3395093753 wait complete. 2024-03-18T22:40:07.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.337+0000 7f1a3613b640 1 Processor -- start 2024-03-18T22:40:07.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.337+0000 7f1a3613b640 1 -- start start 2024-03-18T22:40:07.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.337+0000 7f1a3613b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a300ffb30 0x7f1a30102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:07.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.337+0000 7f1a3613b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a30102b00 0x7f1a30100ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:07.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.337+0000 7f1a3613b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a301034c0 0x7f1a30100ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:07.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.337+0000 7f1a3613b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3010d3c0 con 0x7f1a300ffb30 2024-03-18T22:40:07.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.337+0000 7f1a3613b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3010d240 con 0x7f1a30102b00 2024-03-18T22:40:07.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.337+0000 7f1a3613b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3010d540 con 0x7f1a301034c0 2024-03-18T22:40:07.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.337+0000 7f1a2f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a300ffb30 0x7f1a30102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:07.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.337+0000 7f1a2f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a300ffb30 0x7f1a30102470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50422/0 (socket says 172.21.15.40:50422) 2024-03-18T22:40:07.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.337+0000 7f1a2f7fe640 1 -- 172.21.15.40:0/1180289337 learned_addr learned my addr 172.21.15.40:0/1180289337 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:07.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.338+0000 7f1a2effd640 1 --2- 172.21.15.40:0/1180289337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a30102b00 0x7f1a30100ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:07.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.338+0000 7f1a2ffff640 1 --2- 172.21.15.40:0/1180289337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a301034c0 0x7f1a30100ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:07.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.338+0000 7f1a2f7fe640 1 -- 172.21.15.40:0/1180289337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a301034c0 msgr2=0x7f1a30100ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:07.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.338+0000 7f1a2f7fe640 1 --2- 172.21.15.40:0/1180289337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a301034c0 0x7f1a30100ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:07.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.338+0000 7f1a2f7fe640 1 -- 172.21.15.40:0/1180289337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a30102b00 msgr2=0x7f1a30100ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:07.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.338+0000 7f1a2f7fe640 1 --2- 172.21.15.40:0/1180289337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a30102b00 0x7f1a30100ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:07.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.338+0000 7f1a2f7fe640 1 -- 172.21.15.40:0/1180289337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a30101880 con 0x7f1a300ffb30 2024-03-18T22:40:07.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.338+0000 7f1a2effd640 1 --2- 172.21.15.40:0/1180289337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a30102b00 0x7f1a30100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:40:07.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.338+0000 7f1a2f7fe640 1 --2- 172.21.15.40:0/1180289337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a300ffb30 0x7f1a30102470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1a2000e9c0 tx=0x7f1a2000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:07.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.338+0000 7f1a2cff9640 1 -- 172.21.15.40:0/1180289337 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2000cde0 con 0x7f1a300ffb30 2024-03-18T22:40:07.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.338+0000 7f1a2cff9640 1 -- 172.21.15.40:0/1180289337 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a200108e0 con 0x7f1a300ffb30 2024-03-18T22:40:07.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.338+0000 7f1a3613b640 1 -- 172.21.15.40:0/1180289337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a301a1c10 con 0x7f1a300ffb30 2024-03-18T22:40:07.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.338+0000 7f1a2cff9640 1 -- 172.21.15.40:0/1180289337 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a20018950 con 0x7f1a300ffb30 2024-03-18T22:40:07.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.338+0000 7f1a3613b640 1 -- 172.21.15.40:0/1180289337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a301a2130 con 0x7f1a300ffb30 2024-03-18T22:40:07.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.339+0000 7f1a3613b640 1 -- 172.21.15.40:0/1180289337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a30108d20 con 0x7f1a300ffb30 2024-03-18T22:40:07.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.340+0000 7f1a2cff9640 1 -- 172.21.15.40:0/1180289337 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1a20010a90 con 0x7f1a300ffb30 2024-03-18T22:40:07.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.341+0000 7f1a2cff9640 1 --2- 172.21.15.40:0/1180289337 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a180a2800 0x7f1a180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:07.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.341+0000 7f1a2effd640 1 --2- 172.21.15.40:0/1180289337 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a180a2800 0x7f1a180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:07.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.341+0000 7f1a2cff9640 1 -- 172.21.15.40:0/1180289337 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1a200cebe0 con 0x7f1a300ffb30 2024-03-18T22:40:07.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.341+0000 7f1a2effd640 1 --2- 172.21.15.40:0/1180289337 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a180a2800 0x7f1a180a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009900 tx=0x7f1a24005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:07.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.345+0000 7f1a2cff9640 1 -- 172.21.15.40:0/1180289337 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1a20053740 con 0x7f1a300ffb30 2024-03-18T22:40:07.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:07 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] up:boot 2024-03-18T22:40:07.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:07 smithi103 ceph-mon[66377]: fsmap 1 up:standby 2024-03-18T22:40:07.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T22:40:07.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:07.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:07.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:07.532 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:07.532 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:07.532 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:07 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:07.532 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:07 smithi103 ceph-mon[66377]: Deploying daemon mds.k on smithi103 2024-03-18T22:40:07.532 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:07 smithi103 ceph-mon[66377]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:07.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.571+0000 7f1a3613b640 1 -- 172.21.15.40:0/1180289337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a3002d1f0 con 0x7f1a300ffb30 2024-03-18T22:40:07.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.572+0000 7f1a2cff9640 1 -- 172.21.15.40:0/1180289337 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f1a2009c120 con 0x7f1a300ffb30 2024-03-18T22:40:07.573 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:07.573 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T22:40:07.574 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 2 2024-03-18T22:40:07.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.573+0000 7f1a3613b640 1 -- 172.21.15.40:0/1180289337 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a180a2800 msgr2=0x7f1a180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:07.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.573+0000 7f1a3613b640 1 --2- 172.21.15.40:0/1180289337 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a180a2800 0x7f1a180a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009900 tx=0x7f1a24005fd0 comp rx=0 tx=0).stop 2024-03-18T22:40:07.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.574+0000 7f1a3613b640 1 -- 172.21.15.40:0/1180289337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a300ffb30 msgr2=0x7f1a30102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:07.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.574+0000 7f1a3613b640 1 --2- 172.21.15.40:0/1180289337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a300ffb30 0x7f1a30102470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1a2000e9c0 tx=0x7f1a2000ee80 comp rx=0 tx=0).stop 2024-03-18T22:40:07.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.574+0000 7f1a3613b640 1 -- 172.21.15.40:0/1180289337 shutdown_connections 2024-03-18T22:40:07.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.574+0000 7f1a3613b640 1 --2- 172.21.15.40:0/1180289337 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a180a2800 0x7f1a180a4cc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:07.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.574+0000 7f1a3613b640 1 --2- 172.21.15.40:0/1180289337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a301034c0 0x7f1a30100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:07.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.574+0000 7f1a3613b640 1 --2- 172.21.15.40:0/1180289337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a30102b00 0x7f1a30100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:07.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.574+0000 7f1a3613b640 1 --2- 172.21.15.40:0/1180289337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a300ffb30 0x7f1a30102470 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:07.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.574+0000 7f1a3613b640 1 -- 172.21.15.40:0/1180289337 >> 172.21.15.40:0/1180289337 conn(0x7f1a300fb460 msgr2=0x7f1a30105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:07.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.574+0000 7f1a3613b640 1 -- 172.21.15.40:0/1180289337 shutdown_connections 2024-03-18T22:40:07.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:07.574+0000 7f1a3613b640 1 -- 172.21.15.40:0/1180289337 wait complete. 2024-03-18T22:40:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:07 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] up:boot 2024-03-18T22:40:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:07 smithi040 ceph-mon[62643]: fsmap 1 up:standby 2024-03-18T22:40:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T22:40:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:07 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:07 smithi040 ceph-mon[62643]: Deploying daemon mds.k on smithi103 2024-03-18T22:40:07.585 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:07 smithi040 ceph-mon[62643]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:07.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:07 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] up:boot 2024-03-18T22:40:07.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:07 smithi113 ceph-mon[66775]: fsmap 1 up:standby 2024-03-18T22:40:07.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T22:40:07.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:07.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:07.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:07.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:07.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:07.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:07 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:07.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:07 smithi113 ceph-mon[66775]: Deploying daemon mds.k on smithi103 2024-03-18T22:40:07.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:07 smithi113 ceph-mon[66775]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:08.723 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:08 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1180289337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:08.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:08 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1180289337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:08.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:08 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1180289337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:09.109 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 22:40:08 smithi103 systemd[1]: Starting Ceph mds.k for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:40:09.370 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 22:40:09 smithi103 podman[94979]: 2024-03-18 22:40:09.021744722 +0000 UTC m=+0.017816453 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T22:40:09.371 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 22:40:09 smithi103 podman[94979]: 2024-03-18 22:40:09.131262591 +0000 UTC m=+0.127334314 container create 3acf8ab35b6bf453877ccb8a1d855dd8ba418707a3a7734438e8bd6cf81ada55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-k, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-18T22:40:09.371 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 22:40:09 smithi103 podman[94979]: 2024-03-18 22:40:09.367478912 +0000 UTC m=+0.363550640 container init 3acf8ab35b6bf453877ccb8a1d855dd8ba418707a3a7734438e8bd6cf81ada55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-k, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-18T22:40:09.371 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 22:40:09 smithi103 podman[94979]: 2024-03-18 22:40:09.370603157 +0000 UTC m=+0.366674883 container start 3acf8ab35b6bf453877ccb8a1d855dd8ba418707a3a7734438e8bd6cf81ada55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-k, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-18T22:40:09.587 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:09.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.693+0000 7fe956977640 1 -- 172.21.15.40:0/2829365708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe950100740 msgr2=0x7fe950100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.693+0000 7fe956977640 1 --2- 172.21.15.40:0/2829365708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe950100740 0x7fe950100b60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe938009900 tx=0x7fe93802bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.693+0000 7fe956977640 1 -- 172.21.15.40:0/2829365708 shutdown_connections 2024-03-18T22:40:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.693+0000 7fe956977640 1 --2- 172.21.15.40:0/2829365708 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe9501022f0 0x7fe950108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.693+0000 7fe956977640 1 --2- 172.21.15.40:0/2829365708 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe950101930 0x7fe950101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.693+0000 7fe956977640 1 --2- 172.21.15.40:0/2829365708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe950100740 0x7fe950100b60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.693+0000 7fe956977640 1 -- 172.21.15.40:0/2829365708 >> 172.21.15.40:0/2829365708 conn(0x7fe9500fbcb0 msgr2=0x7fe9500fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.693+0000 7fe956977640 1 -- 172.21.15.40:0/2829365708 shutdown_connections 2024-03-18T22:40:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.693+0000 7fe956977640 1 -- 172.21.15.40:0/2829365708 wait complete. 2024-03-18T22:40:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.693+0000 7fe956977640 1 Processor -- start 2024-03-18T22:40:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.693+0000 7fe956977640 1 -- start start 2024-03-18T22:40:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe956977640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe950100740 0x7fe950197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe956977640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe950101930 0x7fe950198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe956977640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9501022f0 0x7fe95019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe956977640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe95010c240 con 0x7fe9501022f0 2024-03-18T22:40:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe956977640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe95010c0c0 con 0x7fe950101930 2024-03-18T22:40:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe956977640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe95010c3c0 con 0x7fe950100740 2024-03-18T22:40:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe954f14640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9501022f0 0x7fe95019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe954f14640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9501022f0 0x7fe95019f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52620/0 (socket says 172.21.15.40:52620) 2024-03-18T22:40:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe954f14640 1 -- 172.21.15.40:0/4009966950 learned_addr learned my addr 172.21.15.40:0/4009966950 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe94f7fe640 1 --2- 172.21.15.40:0/4009966950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe950101930 0x7fe950198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe94ffff640 1 --2- 172.21.15.40:0/4009966950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe950100740 0x7fe950197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe954f14640 1 -- 172.21.15.40:0/4009966950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe950100740 msgr2=0x7fe950197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe954f14640 1 --2- 172.21.15.40:0/4009966950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe950100740 0x7fe950197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe954f14640 1 -- 172.21.15.40:0/4009966950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe950101930 msgr2=0x7fe950198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe954f14640 1 --2- 172.21.15.40:0/4009966950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe950101930 0x7fe950198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:09.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe954f14640 1 -- 172.21.15.40:0/4009966950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe95019f710 con 0x7fe9501022f0 2024-03-18T22:40:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe954f14640 1 --2- 172.21.15.40:0/4009966950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9501022f0 0x7fe95019f0a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe94800eaa0 tx=0x7fe94800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.694+0000 7fe94d7fa640 1 -- 172.21.15.40:0/4009966950 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94800cdb0 con 0x7fe9501022f0 2024-03-18T22:40:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.695+0000 7fe956977640 1 -- 172.21.15.40:0/4009966950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe95019fa00 con 0x7fe9501022f0 2024-03-18T22:40:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.695+0000 7fe94d7fa640 1 -- 172.21.15.40:0/4009966950 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe948005020 con 0x7fe9501022f0 2024-03-18T22:40:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.695+0000 7fe94d7fa640 1 -- 172.21.15.40:0/4009966950 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9480116d0 con 0x7fe9501022f0 2024-03-18T22:40:09.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.695+0000 7fe956977640 1 -- 172.21.15.40:0/4009966950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe95019ff00 con 0x7fe9501022f0 2024-03-18T22:40:09.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.696+0000 7fe956977640 1 -- 172.21.15.40:0/4009966950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9501a01f0 con 0x7fe9501022f0 2024-03-18T22:40:09.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.697+0000 7fe94d7fa640 1 -- 172.21.15.40:0/4009966950 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe9480051d0 con 0x7fe9501022f0 2024-03-18T22:40:09.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.698+0000 7fe94d7fa640 1 --2- 172.21.15.40:0/4009966950 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe93c0a2800 0x7fe93c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:09.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.698+0000 7fe94d7fa640 1 -- 172.21.15.40:0/4009966950 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe9480ceae0 con 0x7fe9501022f0 2024-03-18T22:40:09.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.698+0000 7fe94ffff640 1 --2- 172.21.15.40:0/4009966950 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe93c0a2800 0x7fe93c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:09.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.699+0000 7fe94ffff640 1 --2- 172.21.15.40:0/4009966950 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe93c0a2800 0x7fe93c0a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe938005ab0 tx=0x7fe938005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:09.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.701+0000 7fe94d7fa640 1 -- 172.21.15.40:0/4009966950 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe948016030 con 0x7fe9501022f0 2024-03-18T22:40:09.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:09 smithi113 ceph-mon[66775]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:09.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:09 smithi103 ceph-mon[66377]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:09.859 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 22:40:09 smithi103 ceph-mds[94998]: -- 172.21.15.103:0/2427172975 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a2f497fa40 con 0x55a2f4952800 2024-03-18T22:40:09.859 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 22:40:09 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-k[94994]: starting mds.k at 2024-03-18T22:40:09.859 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 22:40:09 smithi103 bash[94979]: 3acf8ab35b6bf453877ccb8a1d855dd8ba418707a3a7734438e8bd6cf81ada55 2024-03-18T22:40:09.859 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 22:40:09 smithi103 systemd[1]: Started Ceph mds.k for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:40:09.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:09 smithi040 ceph-mon[62643]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.924+0000 7fe956977640 1 -- 172.21.15.40:0/4009966950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9501a03c0 con 0x7fe9501022f0 2024-03-18T22:40:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.924+0000 7fe94d7fa640 1 -- 172.21.15.40:0/4009966950 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7fe94809b9d0 con 0x7fe9501022f0 2024-03-18T22:40:09.928 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:09.928 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-18T22:40:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 3 2024-03-18T22:40:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.926+0000 7fe956977640 1 -- 172.21.15.40:0/4009966950 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe93c0a2800 msgr2=0x7fe93c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.926+0000 7fe956977640 1 --2- 172.21.15.40:0/4009966950 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe93c0a2800 0x7fe93c0a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe938005ab0 tx=0x7fe938005fd0 comp rx=0 tx=0).stop 2024-03-18T22:40:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.926+0000 7fe956977640 1 -- 172.21.15.40:0/4009966950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9501022f0 msgr2=0x7fe95019f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.926+0000 7fe956977640 1 --2- 172.21.15.40:0/4009966950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9501022f0 0x7fe95019f0a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe94800eaa0 tx=0x7fe94800ef60 comp rx=0 tx=0).stop 2024-03-18T22:40:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.927+0000 7fe956977640 1 -- 172.21.15.40:0/4009966950 shutdown_connections 2024-03-18T22:40:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.927+0000 7fe956977640 1 --2- 172.21.15.40:0/4009966950 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe93c0a2800 0x7fe93c0a4cc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.927+0000 7fe956977640 1 --2- 172.21.15.40:0/4009966950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9501022f0 0x7fe95019f0a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.927+0000 7fe956977640 1 --2- 172.21.15.40:0/4009966950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe950101930 0x7fe950198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.927+0000 7fe956977640 1 --2- 172.21.15.40:0/4009966950 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe950100740 0x7fe950197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.927+0000 7fe956977640 1 -- 172.21.15.40:0/4009966950 >> 172.21.15.40:0/4009966950 conn(0x7fe9500fbcb0 msgr2=0x7fe950068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.927+0000 7fe956977640 1 -- 172.21.15.40:0/4009966950 shutdown_connections 2024-03-18T22:40:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:09.927+0000 7fe956977640 1 -- 172.21.15.40:0/4009966950 wait complete. 2024-03-18T22:40:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:10 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] up:boot 2024-03-18T22:40:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:10 smithi040 ceph-mon[62643]: fsmap 2 up:standby 2024-03-18T22:40:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:10 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T22:40:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:10 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:10 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:10 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:10 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:10 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:10 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:10 smithi040 ceph-mon[62643]: Deploying daemon mds.j on smithi040 2024-03-18T22:40:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:10 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/4009966950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:10.788 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:10 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] up:boot 2024-03-18T22:40:10.788 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:10 smithi103 ceph-mon[66377]: fsmap 2 up:standby 2024-03-18T22:40:10.788 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:10 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T22:40:10.788 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:10 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:10.788 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:10 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:10.788 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:10 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:10.789 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:10 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:10.789 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:10 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:10.789 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:10 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:10.789 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:10 smithi103 ceph-mon[66377]: Deploying daemon mds.j on smithi040 2024-03-18T22:40:10.789 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:10 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/4009966950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:10.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:10 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] up:boot 2024-03-18T22:40:10.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:10 smithi113 ceph-mon[66775]: fsmap 2 up:standby 2024-03-18T22:40:10.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:10 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T22:40:10.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:10 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:10.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:10 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:10.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:10 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:10.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:10 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:10.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:10 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:10.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:10 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:10.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:10 smithi113 ceph-mon[66775]: Deploying daemon mds.j on smithi040 2024-03-18T22:40:10.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:10 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/4009966950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:11.731 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:11 smithi103 ceph-mon[66377]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:11.766 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:11 smithi040 ceph-mon[62643]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:11.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:11 smithi113 ceph-mon[66775]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:11.940 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.047+0000 7f873d880640 1 -- 172.21.15.40:0/2146580441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87381033e0 msgr2=0x7f8738109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.047+0000 7f873d880640 1 --2- 172.21.15.40:0/2146580441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87381033e0 0x7f8738109c70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8730009ca0 tx=0x7f873002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:40:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.047+0000 7f873d880640 1 -- 172.21.15.40:0/2146580441 shutdown_connections 2024-03-18T22:40:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.047+0000 7f873d880640 1 --2- 172.21.15.40:0/2146580441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87381033e0 0x7f8738109c70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.047+0000 7f873d880640 1 --2- 172.21.15.40:0/2146580441 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8738102a20 0x7f8738102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.047+0000 7f873d880640 1 --2- 172.21.15.40:0/2146580441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8738101830 0x7f8738101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.047+0000 7f873d880640 1 -- 172.21.15.40:0/2146580441 >> 172.21.15.40:0/2146580441 conn(0x7f87380fce00 msgr2=0x7f87380ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.047+0000 7f873d880640 1 -- 172.21.15.40:0/2146580441 shutdown_connections 2024-03-18T22:40:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.047+0000 7f873d880640 1 -- 172.21.15.40:0/2146580441 wait complete. 2024-03-18T22:40:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f873d880640 1 Processor -- start 2024-03-18T22:40:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f873d880640 1 -- start start 2024-03-18T22:40:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f873d880640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8738101830 0x7f8738197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f873d880640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8738102a20 0x7f8738197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f873d880640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87381033e0 0x7f873819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f873d880640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f873810d360 con 0x7f87381033e0 2024-03-18T22:40:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f873d880640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f873810d1e0 con 0x7f8738102a20 2024-03-18T22:40:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f873d880640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f873810d4e0 con 0x7f8738101830 2024-03-18T22:40:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f87377fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87381033e0 0x7f873819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f87377fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87381033e0 0x7f873819f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52648/0 (socket says 172.21.15.40:52648) 2024-03-18T22:40:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f87377fe640 1 -- 172.21.15.40:0/2541135211 learned_addr learned my addr 172.21.15.40:0/2541135211 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f8736ffd640 1 --2- 172.21.15.40:0/2541135211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8738101830 0x7f8738197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f87367fc640 1 --2- 172.21.15.40:0/2541135211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8738102a20 0x7f8738197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f87377fe640 1 -- 172.21.15.40:0/2541135211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8738101830 msgr2=0x7f8738197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f87377fe640 1 --2- 172.21.15.40:0/2541135211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8738101830 0x7f8738197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f87377fe640 1 -- 172.21.15.40:0/2541135211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8738102a20 msgr2=0x7f8738197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f87377fe640 1 --2- 172.21.15.40:0/2541135211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8738102a20 0x7f8738197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.048+0000 7f87377fe640 1 -- 172.21.15.40:0/2541135211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f873819f680 con 0x7f87381033e0 2024-03-18T22:40:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.049+0000 7f87377fe640 1 --2- 172.21.15.40:0/2541135211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87381033e0 0x7f873819f010 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f873002c700 tx=0x7f87300328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.049+0000 7f873c87e640 1 -- 172.21.15.40:0/2541135211 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8730041020 con 0x7f87381033e0 2024-03-18T22:40:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.049+0000 7f873c87e640 1 -- 172.21.15.40:0/2541135211 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8730003e20 con 0x7f87381033e0 2024-03-18T22:40:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.049+0000 7f873d880640 1 -- 172.21.15.40:0/2541135211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f873819f910 con 0x7f87381033e0 2024-03-18T22:40:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.049+0000 7f873c87e640 1 -- 172.21.15.40:0/2541135211 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8730039040 con 0x7f87381033e0 2024-03-18T22:40:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.049+0000 7f873d880640 1 -- 172.21.15.40:0/2541135211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f873819fe00 con 0x7f87381033e0 2024-03-18T22:40:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.050+0000 7f873d880640 1 -- 172.21.15.40:0/2541135211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8738108b70 con 0x7f87381033e0 2024-03-18T22:40:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.051+0000 7f873c87e640 1 -- 172.21.15.40:0/2541135211 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8730044020 con 0x7f87381033e0 2024-03-18T22:40:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.051+0000 7f873c87e640 1 --2- 172.21.15.40:0/2541135211 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f87240a2800 0x7f87240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.051+0000 7f8736ffd640 1 --2- 172.21.15.40:0/2541135211 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f87240a2800 0x7f87240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.052+0000 7f873c87e640 1 -- 172.21.15.40:0/2541135211 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f87300f0980 con 0x7f87381033e0 2024-03-18T22:40:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.052+0000 7f8736ffd640 1 --2- 172.21.15.40:0/2541135211 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f87240a2800 0x7f87240a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8738102880 tx=0x7f8720005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:12.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.056+0000 7f873c87e640 1 -- 172.21.15.40:0/2541135211 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f873004b020 con 0x7f87381033e0 2024-03-18T22:40:12.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.283+0000 7f873d880640 1 -- 172.21.15.40:0/2541135211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8738062420 con 0x7f87381033e0 2024-03-18T22:40:12.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.284+0000 7f873c87e640 1 -- 172.21.15.40:0/2541135211 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f87300bde20 con 0x7f87381033e0 2024-03-18T22:40:12.285 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:12.286 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-18T22:40:12.286 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 3 2024-03-18T22:40:12.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.286+0000 7f873d880640 1 -- 172.21.15.40:0/2541135211 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f87240a2800 msgr2=0x7f87240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:12.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.286+0000 7f873d880640 1 --2- 172.21.15.40:0/2541135211 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f87240a2800 0x7f87240a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8738102880 tx=0x7f8720005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:12.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.286+0000 7f873d880640 1 -- 172.21.15.40:0/2541135211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87381033e0 msgr2=0x7f873819f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:12.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.286+0000 7f873d880640 1 --2- 172.21.15.40:0/2541135211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87381033e0 0x7f873819f010 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f873002c700 tx=0x7f87300328a0 comp rx=0 tx=0).stop 2024-03-18T22:40:12.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.286+0000 7f873d880640 1 -- 172.21.15.40:0/2541135211 shutdown_connections 2024-03-18T22:40:12.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.286+0000 7f873d880640 1 --2- 172.21.15.40:0/2541135211 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f87240a2800 0x7f87240a4cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:12.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.286+0000 7f873d880640 1 --2- 172.21.15.40:0/2541135211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87381033e0 0x7f873819f010 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:12.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.286+0000 7f873d880640 1 --2- 172.21.15.40:0/2541135211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8738102a20 0x7f8738197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:12.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.286+0000 7f873d880640 1 --2- 172.21.15.40:0/2541135211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8738101830 0x7f8738197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:12.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.286+0000 7f873d880640 1 -- 172.21.15.40:0/2541135211 >> 172.21.15.40:0/2541135211 conn(0x7f87380fce00 msgr2=0x7f8738105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:12.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.286+0000 7f873d880640 1 -- 172.21.15.40:0/2541135211 shutdown_connections 2024-03-18T22:40:12.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:12.286+0000 7f873d880640 1 -- 172.21.15.40:0/2541135211 wait complete. 2024-03-18T22:40:12.481 INFO:journalctl@ceph.mds.j.smithi040.stdout:Mar 18 22:40:12 smithi040 systemd[1]: Started Ceph mds.j for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:40:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:12 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2541135211' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:12.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:12.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:12.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:12.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:12 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2541135211' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:12.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:12.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:12.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:12.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:12.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:12.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:12.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:12 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2541135211' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:12.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:12.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:12.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:12.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:12.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:12.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:13.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:13 smithi040 ceph-mon[62643]: Deploying daemon mds.d on smithi040 2024-03-18T22:40:13.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:13 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.40:6834/823346277,v1:172.21.15.40:6835/823346277] up:boot 2024-03-18T22:40:13.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:13 smithi040 ceph-mon[62643]: fsmap 3 up:standby 2024-03-18T22:40:13.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:13 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T22:40:13.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:13 smithi040 ceph-mon[62643]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:13.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:13 smithi113 ceph-mon[66775]: Deploying daemon mds.d on smithi040 2024-03-18T22:40:13.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:13 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.40:6834/823346277,v1:172.21.15.40:6835/823346277] up:boot 2024-03-18T22:40:13.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:13 smithi113 ceph-mon[66775]: fsmap 3 up:standby 2024-03-18T22:40:13.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:13 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T22:40:13.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:13 smithi113 ceph-mon[66775]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:13.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:13 smithi103 ceph-mon[66377]: Deploying daemon mds.d on smithi040 2024-03-18T22:40:13.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:13 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.40:6834/823346277,v1:172.21.15.40:6835/823346277] up:boot 2024-03-18T22:40:13.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:13 smithi103 ceph-mon[66377]: fsmap 3 up:standby 2024-03-18T22:40:13.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:13 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T22:40:13.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:13 smithi103 ceph-mon[66377]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:14.300 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.473+0000 7f5c12359640 1 -- 172.21.15.40:0/354282191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c0c101930 msgr2=0x7f5c0c101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.473+0000 7f5c12359640 1 --2- 172.21.15.40:0/354282191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c0c101930 0x7f5c0c101db0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5c00009900 tx=0x7f5c0002bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.473+0000 7f5c12359640 1 -- 172.21.15.40:0/354282191 shutdown_connections 2024-03-18T22:40:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.473+0000 7f5c12359640 1 --2- 172.21.15.40:0/354282191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c0c1022f0 0x7f5c0c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.473+0000 7f5c12359640 1 --2- 172.21.15.40:0/354282191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c0c101930 0x7f5c0c101db0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.473+0000 7f5c12359640 1 --2- 172.21.15.40:0/354282191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c100740 0x7f5c0c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.473+0000 7f5c12359640 1 -- 172.21.15.40:0/354282191 >> 172.21.15.40:0/354282191 conn(0x7f5c0c0fbcb0 msgr2=0x7f5c0c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.473+0000 7f5c12359640 1 -- 172.21.15.40:0/354282191 shutdown_connections 2024-03-18T22:40:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.473+0000 7f5c12359640 1 -- 172.21.15.40:0/354282191 wait complete. 2024-03-18T22:40:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.474+0000 7f5c12359640 1 Processor -- start 2024-03-18T22:40:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.474+0000 7f5c12359640 1 -- start start 2024-03-18T22:40:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.474+0000 7f5c12359640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c0c100740 0x7f5c0c0694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.474+0000 7f5c12359640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c101930 0x7f5c0c19c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.474+0000 7f5c12359640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c0c1022f0 0x7f5c0c19ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.474+0000 7f5c12359640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c10c270 con 0x7f5c0c1022f0 2024-03-18T22:40:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.474+0000 7f5c12359640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c10c0f0 con 0x7f5c0c101930 2024-03-18T22:40:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.474+0000 7f5c12359640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c10c3f0 con 0x7f5c0c100740 2024-03-18T22:40:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c108f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c0c1022f0 0x7f5c0c19ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c108f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c0c1022f0 0x7f5c0c19ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52690/0 (socket says 172.21.15.40:52690) 2024-03-18T22:40:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c108f6640 1 -- 172.21.15.40:0/3660727668 learned_addr learned my addr 172.21.15.40:0/3660727668 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c0bfff640 1 --2- 172.21.15.40:0/3660727668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c0c100740 0x7f5c0c0694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c0b7fe640 1 --2- 172.21.15.40:0/3660727668 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c101930 0x7f5c0c19c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c108f6640 1 -- 172.21.15.40:0/3660727668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c0c100740 msgr2=0x7f5c0c0694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c108f6640 1 --2- 172.21.15.40:0/3660727668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c0c100740 0x7f5c0c0694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c108f6640 1 -- 172.21.15.40:0/3660727668 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c101930 msgr2=0x7f5c0c19c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c108f6640 1 --2- 172.21.15.40:0/3660727668 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c101930 0x7f5c0c19c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c108f6640 1 -- 172.21.15.40:0/3660727668 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0c19f690 con 0x7f5c0c1022f0 2024-03-18T22:40:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c108f6640 1 --2- 172.21.15.40:0/3660727668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c0c1022f0 0x7f5c0c19ed80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400eb10 tx=0x7f5c0400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c097fa640 1 -- 172.21.15.40:0/3660727668 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c04004b20 con 0x7f5c0c1022f0 2024-03-18T22:40:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c097fa640 1 -- 172.21.15.40:0/3660727668 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5c04004cc0 con 0x7f5c0c1022f0 2024-03-18T22:40:14.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c12359640 1 -- 172.21.15.40:0/3660727668 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c0c19f920 con 0x7f5c0c1022f0 2024-03-18T22:40:14.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c097fa640 1 -- 172.21.15.40:0/3660727668 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c04010690 con 0x7f5c0c1022f0 2024-03-18T22:40:14.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.475+0000 7f5c12359640 1 -- 172.21.15.40:0/3660727668 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c0c19fe40 con 0x7f5c0c1022f0 2024-03-18T22:40:14.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.476+0000 7f5c12359640 1 -- 172.21.15.40:0/3660727668 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c0c107a60 con 0x7f5c0c1022f0 2024-03-18T22:40:14.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.477+0000 7f5c097fa640 1 -- 172.21.15.40:0/3660727668 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5c04005020 con 0x7f5c0c1022f0 2024-03-18T22:40:14.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.478+0000 7f5c097fa640 1 --2- 172.21.15.40:0/3660727668 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5bf40a2800 0x7f5bf40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:14.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.478+0000 7f5c0bfff640 1 --2- 172.21.15.40:0/3660727668 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5bf40a2800 0x7f5bf40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:14.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.478+0000 7f5c097fa640 1 -- 172.21.15.40:0/3660727668 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5c040ced20 con 0x7f5c0c1022f0 2024-03-18T22:40:14.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.478+0000 7f5c0bfff640 1 --2- 172.21.15.40:0/3660727668 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5bf40a2800 0x7f5bf40a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c101790 tx=0x7f5bfc005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:14.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.482+0000 7f5c097fa640 1 -- 172.21.15.40:0/3660727668 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5c0409c1c0 con 0x7f5c0c1022f0 2024-03-18T22:40:14.644 INFO:journalctl@ceph.mds.d.smithi040.stdout:Mar 18 22:40:14 smithi040 systemd[1]: Starting Ceph mds.d for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:40:14.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.723+0000 7f5c12359640 1 -- 172.21.15.40:0/3660727668 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c0c02d240 con 0x7f5c0c1022f0 2024-03-18T22:40:14.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.723+0000 7f5c097fa640 1 -- 172.21.15.40:0/3660727668 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7f5c0409bac0 con 0x7f5c0c1022f0 2024-03-18T22:40:14.725 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:14.725 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-18T22:40:14.725 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 4 2024-03-18T22:40:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.725+0000 7f5c12359640 1 -- 172.21.15.40:0/3660727668 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5bf40a2800 msgr2=0x7f5bf40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.725+0000 7f5c12359640 1 --2- 172.21.15.40:0/3660727668 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5bf40a2800 0x7f5bf40a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c101790 tx=0x7f5bfc005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.725+0000 7f5c12359640 1 -- 172.21.15.40:0/3660727668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c0c1022f0 msgr2=0x7f5c0c19ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.725+0000 7f5c12359640 1 --2- 172.21.15.40:0/3660727668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c0c1022f0 0x7f5c0c19ed80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400eb10 tx=0x7f5c0400c2d0 comp rx=0 tx=0).stop 2024-03-18T22:40:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.725+0000 7f5c12359640 1 -- 172.21.15.40:0/3660727668 shutdown_connections 2024-03-18T22:40:14.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.726+0000 7f5c12359640 1 --2- 172.21.15.40:0/3660727668 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5bf40a2800 0x7f5bf40a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:14.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.726+0000 7f5c12359640 1 --2- 172.21.15.40:0/3660727668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c0c1022f0 0x7f5c0c19ed80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:14.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.726+0000 7f5c12359640 1 --2- 172.21.15.40:0/3660727668 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c101930 0x7f5c0c19c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:14.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.726+0000 7f5c12359640 1 --2- 172.21.15.40:0/3660727668 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5c0c100740 0x7f5c0c0694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:14.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.726+0000 7f5c12359640 1 -- 172.21.15.40:0/3660727668 >> 172.21.15.40:0/3660727668 conn(0x7f5c0c0fbcb0 msgr2=0x7f5c0c104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:14.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.726+0000 7f5c12359640 1 -- 172.21.15.40:0/3660727668 shutdown_connections 2024-03-18T22:40:14.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:14.726+0000 7f5c12359640 1 -- 172.21.15.40:0/3660727668 wait complete. 2024-03-18T22:40:15.043 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:14 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3660727668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:15.043 INFO:journalctl@ceph.mds.d.smithi040.stdout:Mar 18 22:40:14 smithi040 podman[100386]: 2024-03-18 22:40:14.720106447 +0000 UTC m=+0.019553659 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T22:40:15.043 INFO:journalctl@ceph.mds.d.smithi040.stdout:Mar 18 22:40:14 smithi040 podman[100386]: 2024-03-18 22:40:14.823723333 +0000 UTC m=+0.123170544 container create 9ac863149aff646b3173d74a875d4bbf4fd9ac7875ef92dc704e29981fdf6c4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-18T22:40:15.044 INFO:journalctl@ceph.mds.d.smithi040.stdout:Mar 18 22:40:15 smithi040 podman[100386]: 2024-03-18 22:40:15.043008848 +0000 UTC m=+0.342456070 container init 9ac863149aff646b3173d74a875d4bbf4fd9ac7875ef92dc704e29981fdf6c4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T22:40:15.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:14 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3660727668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:15.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:14 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3660727668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:15.393 INFO:journalctl@ceph.mds.d.smithi040.stdout:Mar 18 22:40:15 smithi040 podman[100386]: 2024-03-18 22:40:15.045730589 +0000 UTC m=+0.345177799 container start 9ac863149aff646b3173d74a875d4bbf4fd9ac7875ef92dc704e29981fdf6c4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-18T22:40:15.394 INFO:journalctl@ceph.mds.d.smithi040.stdout:Mar 18 22:40:15 smithi040 ceph-mds[100407]: -- 172.21.15.40:0/4039203218 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55632ff35a40 con 0x55632ff08800 2024-03-18T22:40:15.394 INFO:journalctl@ceph.mds.d.smithi040.stdout:Mar 18 22:40:15 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-d[100403]: starting mds.d at 2024-03-18T22:40:15.394 INFO:journalctl@ceph.mds.d.smithi040.stdout:Mar 18 22:40:15 smithi040 bash[100386]: 9ac863149aff646b3173d74a875d4bbf4fd9ac7875ef92dc704e29981fdf6c4f 2024-03-18T22:40:15.394 INFO:journalctl@ceph.mds.d.smithi040.stdout:Mar 18 22:40:15 smithi040 systemd[1]: Started Ceph mds.d for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:40:16.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:15 smithi113 ceph-mon[66775]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:16.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:15 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:16.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:15 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:16.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:15 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:16.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:15 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:16.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:15 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:16.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:15 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:16.105 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:15 smithi103 ceph-mon[66377]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:16.105 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:15 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:16.105 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:15 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:16.105 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:15 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:16.105 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:15 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:16.105 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:15 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:16.105 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:15 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:16.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:15 smithi040 ceph-mon[62643]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:15 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:15 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:15 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:15 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:15 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:15 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:16.739 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:16.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.856+0000 7faf3fdc6640 1 -- 172.21.15.40:0/168095672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf380ffb30 msgr2=0x7faf380fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:16.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.856+0000 7faf3fdc6640 1 --2- 172.21.15.40:0/168095672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf380ffb30 0x7faf380fff50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7faf34009930 tx=0x7faf3402bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:16.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.856+0000 7faf3fdc6640 1 -- 172.21.15.40:0/168095672 shutdown_connections 2024-03-18T22:40:16.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.856+0000 7faf3fdc6640 1 --2- 172.21.15.40:0/168095672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf381034c0 0x7faf38109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:16.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.856+0000 7faf3fdc6640 1 --2- 172.21.15.40:0/168095672 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf38102b00 0x7faf38102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:16.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.856+0000 7faf3fdc6640 1 --2- 172.21.15.40:0/168095672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf380ffb30 0x7faf380fff50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:16.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.856+0000 7faf3fdc6640 1 -- 172.21.15.40:0/168095672 >> 172.21.15.40:0/168095672 conn(0x7faf380fb460 msgr2=0x7faf380fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:16.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.856+0000 7faf3fdc6640 1 -- 172.21.15.40:0/168095672 shutdown_connections 2024-03-18T22:40:16.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.856+0000 7faf3fdc6640 1 -- 172.21.15.40:0/168095672 wait complete. 2024-03-18T22:40:16.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3fdc6640 1 Processor -- start 2024-03-18T22:40:16.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3fdc6640 1 -- start start 2024-03-18T22:40:16.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3fdc6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf380ffb30 0x7faf38197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:16.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3fdc6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf38102b00 0x7faf38198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:16.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3fdc6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf381034c0 0x7faf3819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:16.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3fdc6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faf3810d390 con 0x7faf38102b00 2024-03-18T22:40:16.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3fdc6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faf3810d210 con 0x7faf380ffb30 2024-03-18T22:40:16.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3fdc6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7faf3810d510 con 0x7faf381034c0 2024-03-18T22:40:16.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3d361640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf38102b00 0x7faf38198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:16.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3d361640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf38102b00 0x7faf38198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52722/0 (socket says 172.21.15.40:52722) 2024-03-18T22:40:16.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3d361640 1 -- 172.21.15.40:0/4289549185 learned_addr learned my addr 172.21.15.40:0/4289549185 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:16.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3db62640 1 --2- 172.21.15.40:0/4289549185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf380ffb30 0x7faf38197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:16.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3e363640 1 --2- 172.21.15.40:0/4289549185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf381034c0 0x7faf3819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:16.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3d361640 1 -- 172.21.15.40:0/4289549185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf381034c0 msgr2=0x7faf3819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:16.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3d361640 1 --2- 172.21.15.40:0/4289549185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf381034c0 0x7faf3819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:16.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3d361640 1 -- 172.21.15.40:0/4289549185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf380ffb30 msgr2=0x7faf38197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:16.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3d361640 1 --2- 172.21.15.40:0/4289549185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf380ffb30 0x7faf38197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:16.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3d361640 1 -- 172.21.15.40:0/4289549185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3819f6e0 con 0x7faf38102b00 2024-03-18T22:40:16.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.857+0000 7faf3d361640 1 --2- 172.21.15.40:0/4289549185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf38102b00 0x7faf38198000 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7faf2800d950 tx=0x7faf2800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:16.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.858+0000 7faf2effd640 1 -- 172.21.15.40:0/4289549185 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf28017030 con 0x7faf38102b00 2024-03-18T22:40:16.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.858+0000 7faf2effd640 1 -- 172.21.15.40:0/4289549185 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf28004020 con 0x7faf38102b00 2024-03-18T22:40:16.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.858+0000 7faf3fdc6640 1 -- 172.21.15.40:0/4289549185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf3819f9d0 con 0x7faf38102b00 2024-03-18T22:40:16.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.858+0000 7faf2effd640 1 -- 172.21.15.40:0/4289549185 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf280047a0 con 0x7faf38102b00 2024-03-18T22:40:16.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.858+0000 7faf3fdc6640 1 -- 172.21.15.40:0/4289549185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf3819ff20 con 0x7faf38102b00 2024-03-18T22:40:16.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.859+0000 7faf3fdc6640 1 -- 172.21.15.40:0/4289549185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf38108d00 con 0x7faf38102b00 2024-03-18T22:40:16.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.861+0000 7faf2effd640 1 -- 172.21.15.40:0/4289549185 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7faf280041d0 con 0x7faf38102b00 2024-03-18T22:40:16.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.861+0000 7faf2effd640 1 --2- 172.21.15.40:0/4289549185 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faf240a2800 0x7faf240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:16.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.861+0000 7faf3db62640 1 --2- 172.21.15.40:0/4289549185 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faf240a2800 0x7faf240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:16.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.861+0000 7faf2effd640 1 -- 172.21.15.40:0/4289549185 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7faf280ce810 con 0x7faf38102b00 2024-03-18T22:40:16.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.862+0000 7faf3db62640 1 --2- 172.21.15.40:0/4289549185 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faf240a2800 0x7faf240a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faf34000c00 tx=0x7faf34005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:16.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:16.865+0000 7faf2effd640 1 -- 172.21.15.40:0/4289549185 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faf2809bcb0 con 0x7faf38102b00 2024-03-18T22:40:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:16 smithi113 ceph-mon[66775]: Deploying daemon mds.e on smithi103 2024-03-18T22:40:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:16 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.40:6836/3562391432,v1:172.21.15.40:6837/3562391432] up:boot 2024-03-18T22:40:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:16 smithi113 ceph-mon[66775]: fsmap 4 up:standby 2024-03-18T22:40:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T22:40:17.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:17.087+0000 7faf3fdc6640 1 -- 172.21.15.40:0/4289549185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf3802d1f0 con 0x7faf38102b00 2024-03-18T22:40:17.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:17.087+0000 7faf2effd640 1 -- 172.21.15.40:0/4289549185 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7faf2809b5b0 con 0x7faf38102b00 2024-03-18T22:40:17.089 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:17.089 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-18T22:40:17.089 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 5 2024-03-18T22:40:17.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:17.089+0000 7faf3fdc6640 1 -- 172.21.15.40:0/4289549185 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faf240a2800 msgr2=0x7faf240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:17.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:17.089+0000 7faf3fdc6640 1 --2- 172.21.15.40:0/4289549185 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faf240a2800 0x7faf240a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faf34000c00 tx=0x7faf34005fd0 comp rx=0 tx=0).stop 2024-03-18T22:40:17.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:17.089+0000 7faf3fdc6640 1 -- 172.21.15.40:0/4289549185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf38102b00 msgr2=0x7faf38198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:17.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:17.089+0000 7faf3fdc6640 1 --2- 172.21.15.40:0/4289549185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf38102b00 0x7faf38198000 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7faf2800d950 tx=0x7faf2800de10 comp rx=0 tx=0).stop 2024-03-18T22:40:17.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:17.089+0000 7faf3fdc6640 1 -- 172.21.15.40:0/4289549185 shutdown_connections 2024-03-18T22:40:17.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:17.089+0000 7faf3fdc6640 1 --2- 172.21.15.40:0/4289549185 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faf240a2800 0x7faf240a4cc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:17.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:17.089+0000 7faf3fdc6640 1 --2- 172.21.15.40:0/4289549185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faf381034c0 0x7faf3819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:17.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:17.089+0000 7faf3fdc6640 1 --2- 172.21.15.40:0/4289549185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf38102b00 0x7faf38198000 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:17.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:17.089+0000 7faf3fdc6640 1 --2- 172.21.15.40:0/4289549185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf380ffb30 0x7faf38197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:17.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:17.089+0000 7faf3fdc6640 1 -- 172.21.15.40:0/4289549185 >> 172.21.15.40:0/4289549185 conn(0x7faf380fb460 msgr2=0x7faf38105ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:17.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:17.089+0000 7faf3fdc6640 1 -- 172.21.15.40:0/4289549185 shutdown_connections 2024-03-18T22:40:17.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:17.089+0000 7faf3fdc6640 1 -- 172.21.15.40:0/4289549185 wait complete. 2024-03-18T22:40:17.100 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:16 smithi040 ceph-mon[62643]: Deploying daemon mds.e on smithi103 2024-03-18T22:40:17.100 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:16 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.40:6836/3562391432,v1:172.21.15.40:6837/3562391432] up:boot 2024-03-18T22:40:17.101 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:16 smithi040 ceph-mon[62643]: fsmap 4 up:standby 2024-03-18T22:40:17.101 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T22:40:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:16 smithi103 ceph-mon[66377]: Deploying daemon mds.e on smithi103 2024-03-18T22:40:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:16 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.40:6836/3562391432,v1:172.21.15.40:6837/3562391432] up:boot 2024-03-18T22:40:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:16 smithi103 ceph-mon[66377]: fsmap 4 up:standby 2024-03-18T22:40:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T22:40:17.578 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 22:40:17 smithi103 systemd[1]: Starting Ceph mds.e for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:40:17.832 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:17 smithi103 ceph-mon[66377]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:17.832 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:17 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/4289549185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:17.832 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 22:40:17 smithi103 podman[95300]: 2024-03-18 22:40:17.481158418 +0000 UTC m=+0.018366947 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T22:40:17.833 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 22:40:17 smithi103 podman[95300]: 2024-03-18 22:40:17.596208351 +0000 UTC m=+0.133416859 container create 2ede6c30917ebeebc014eacdafb23849a8dd1c76b65c462a1cc9a8bd8a8db610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-e, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T22:40:17.833 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 22:40:17 smithi103 podman[95300]: 2024-03-18 22:40:17.83236657 +0000 UTC m=+0.369575099 container init 2ede6c30917ebeebc014eacdafb23849a8dd1c76b65c462a1cc9a8bd8a8db610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-e, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-18T22:40:18.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:17 smithi113 ceph-mon[66775]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:18.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:17 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/4289549185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:18.109 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 22:40:17 smithi103 podman[95300]: 2024-03-18 22:40:17.835117643 +0000 UTC m=+0.372326152 container start 2ede6c30917ebeebc014eacdafb23849a8dd1c76b65c462a1cc9a8bd8a8db610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-18T22:40:18.109 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 22:40:17 smithi103 ceph-mds[95319]: -- 172.21.15.103:0/803321353 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c6eaf4fa40 con 0x55c6eaf22c00 2024-03-18T22:40:18.109 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 22:40:17 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-e[95315]: starting mds.e at 2024-03-18T22:40:18.109 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 22:40:17 smithi103 bash[95300]: 2ede6c30917ebeebc014eacdafb23849a8dd1c76b65c462a1cc9a8bd8a8db610 2024-03-18T22:40:18.110 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 22:40:17 smithi103 systemd[1]: Started Ceph mds.e for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:40:18.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:17 smithi040 ceph-mon[62643]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:18.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:17 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/4289549185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:19.103 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:19.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.217+0000 7fd5e24dc640 1 -- 172.21.15.40:0/2504087644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5dc1013d0 msgr2=0x7fd5dc108c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:19.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.217+0000 7fd5e24dc640 1 --2- 172.21.15.40:0/2504087644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5dc1013d0 0x7fd5dc108c70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4009ca0 tx=0x7fd5d402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:40:19.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.217+0000 7fd5e24dc640 1 -- 172.21.15.40:0/2504087644 shutdown_connections 2024-03-18T22:40:19.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.217+0000 7fd5e24dc640 1 --2- 172.21.15.40:0/2504087644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5dc1013d0 0x7fd5dc108c70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:19.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.217+0000 7fd5e24dc640 1 --2- 172.21.15.40:0/2504087644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5dc100a10 0x7fd5dc100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:19.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.217+0000 7fd5e24dc640 1 --2- 172.21.15.40:0/2504087644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5dc101ce0 0x7fd5dc102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:19.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.217+0000 7fd5e24dc640 1 -- 172.21.15.40:0/2504087644 >> 172.21.15.40:0/2504087644 conn(0x7fd5dc0fb440 msgr2=0x7fd5dc0fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:19.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.217+0000 7fd5e24dc640 1 -- 172.21.15.40:0/2504087644 shutdown_connections 2024-03-18T22:40:19.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.217+0000 7fd5e24dc640 1 -- 172.21.15.40:0/2504087644 wait complete. 2024-03-18T22:40:19.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.218+0000 7fd5e24dc640 1 Processor -- start 2024-03-18T22:40:19.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.218+0000 7fd5e24dc640 1 -- start start 2024-03-18T22:40:19.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.218+0000 7fd5e24dc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5dc100a10 0x7fd5dc1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:19.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.218+0000 7fd5e24dc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5dc1013d0 0x7fd5dc197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.218+0000 7fd5e24dc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5dc101ce0 0x7fd5dc19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.218+0000 7fd5e24dc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc10c260 con 0x7fd5dc100a10 2024-03-18T22:40:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.218+0000 7fd5e24dc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc10c0e0 con 0x7fd5dc101ce0 2024-03-18T22:40:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.218+0000 7fd5e24dc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc10c3e0 con 0x7fd5dc1013d0 2024-03-18T22:40:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5dbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5dc100a10 0x7fd5dc1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5dbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5dc100a10 0x7fd5dc1979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42900/0 (socket says 172.21.15.40:42900) 2024-03-18T22:40:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5dbfff640 1 -- 172.21.15.40:0/1010921819 learned_addr learned my addr 172.21.15.40:0/1010921819 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5e0a79640 1 --2- 172.21.15.40:0/1010921819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5dc101ce0 0x7fd5dc19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5db7fe640 1 --2- 172.21.15.40:0/1010921819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5dc1013d0 0x7fd5dc197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5dbfff640 1 -- 172.21.15.40:0/1010921819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5dc1013d0 msgr2=0x7fd5dc197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5dbfff640 1 --2- 172.21.15.40:0/1010921819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5dc1013d0 0x7fd5dc197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5dbfff640 1 -- 172.21.15.40:0/1010921819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5dc101ce0 msgr2=0x7fd5dc19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:19.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5dbfff640 1 --2- 172.21.15.40:0/1010921819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5dc101ce0 0x7fd5dc19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5dbfff640 1 -- 172.21.15.40:0/1010921819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5dc19f610 con 0x7fd5dc100a10 2024-03-18T22:40:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5e0a79640 1 --2- 172.21.15.40:0/1010921819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5dc101ce0 0x7fd5dc19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:40:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5dbfff640 1 --2- 172.21.15.40:0/1010921819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5dc100a10 0x7fd5dc1979f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400e9c0 tx=0x7fd5c400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5d97fa640 1 -- 172.21.15.40:0/1010921819 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c400cd80 con 0x7fd5dc100a10 2024-03-18T22:40:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5d97fa640 1 -- 172.21.15.40:0/1010921819 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5c40035f0 con 0x7fd5dc100a10 2024-03-18T22:40:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5e24dc640 1 -- 172.21.15.40:0/1010921819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5dc19f900 con 0x7fd5dc100a10 2024-03-18T22:40:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5d97fa640 1 -- 172.21.15.40:0/1010921819 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4010690 con 0x7fd5dc100a10 2024-03-18T22:40:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.219+0000 7fd5e24dc640 1 -- 172.21.15.40:0/1010921819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5dc19fe50 con 0x7fd5dc100a10 2024-03-18T22:40:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.220+0000 7fd5e24dc640 1 -- 172.21.15.40:0/1010921819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5dc068a60 con 0x7fd5dc100a10 2024-03-18T22:40:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.221+0000 7fd5d97fa640 1 -- 172.21.15.40:0/1010921819 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd5c40037a0 con 0x7fd5dc100a10 2024-03-18T22:40:19.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.222+0000 7fd5d97fa640 1 --2- 172.21.15.40:0/1010921819 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5c80a2800 0x7fd5c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:19.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.222+0000 7fd5db7fe640 1 --2- 172.21.15.40:0/1010921819 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5c80a2800 0x7fd5c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:19.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.222+0000 7fd5d97fa640 1 -- 172.21.15.40:0/1010921819 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd5c40ceb40 con 0x7fd5dc100a10 2024-03-18T22:40:19.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.222+0000 7fd5db7fe640 1 --2- 172.21.15.40:0/1010921819 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5c80a2800 0x7fd5c80a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc005fd0 tx=0x7fd5cc005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:19.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.227+0000 7fd5d97fa640 1 -- 172.21.15.40:0/1010921819 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd5c409bfe0 con 0x7fd5dc100a10 2024-03-18T22:40:19.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:18 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:19.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:18 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:19.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:18 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:19.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:18 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:19.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:18 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:19.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:18 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:19.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:18 smithi113 ceph-mon[66775]: Deploying daemon mds.f on smithi113 2024-03-18T22:40:19.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:18 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.103:6834/2536694691,v1:172.21.15.103:6835/2536694691] up:boot 2024-03-18T22:40:19.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:18 smithi113 ceph-mon[66775]: fsmap 5 up:standby 2024-03-18T22:40:19.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:18 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T22:40:19.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:18 smithi113 ceph-mon[66775]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:19.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:18 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:19.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:18 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:19.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:18 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:19.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:18 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:19.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:18 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:19.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:18 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:19.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:18 smithi103 ceph-mon[66377]: Deploying daemon mds.f on smithi113 2024-03-18T22:40:19.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:18 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.103:6834/2536694691,v1:172.21.15.103:6835/2536694691] up:boot 2024-03-18T22:40:19.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:18 smithi103 ceph-mon[66377]: fsmap 5 up:standby 2024-03-18T22:40:19.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:18 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T22:40:19.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:18 smithi103 ceph-mon[66377]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:19.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:18 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:19.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:18 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:19.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:18 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:19.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:18 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:19.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:18 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:19.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:18 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:19.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:18 smithi040 ceph-mon[62643]: Deploying daemon mds.f on smithi113 2024-03-18T22:40:19.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:18 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.103:6834/2536694691,v1:172.21.15.103:6835/2536694691] up:boot 2024-03-18T22:40:19.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:18 smithi040 ceph-mon[62643]: fsmap 5 up:standby 2024-03-18T22:40:19.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:18 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T22:40:19.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:18 smithi040 ceph-mon[62643]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.448+0000 7fd5e24dc640 1 -- 172.21.15.40:0/1010921819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5dc1a00e0 con 0x7fd5dc100a10 2024-03-18T22:40:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.449+0000 7fd5d97fa640 1 -- 172.21.15.40:0/1010921819 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4699 (secure 0 0 0) 0x7fd5c409b8e0 con 0x7fd5dc100a10 2024-03-18T22:40:19.450 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:19.450 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-18T22:40:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 6 2024-03-18T22:40:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.450+0000 7fd5e24dc640 1 -- 172.21.15.40:0/1010921819 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5c80a2800 msgr2=0x7fd5c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.450+0000 7fd5e24dc640 1 --2- 172.21.15.40:0/1010921819 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5c80a2800 0x7fd5c80a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc005fd0 tx=0x7fd5cc005850 comp rx=0 tx=0).stop 2024-03-18T22:40:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.450+0000 7fd5e24dc640 1 -- 172.21.15.40:0/1010921819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5dc100a10 msgr2=0x7fd5dc1979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.450+0000 7fd5e24dc640 1 --2- 172.21.15.40:0/1010921819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5dc100a10 0x7fd5dc1979f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400e9c0 tx=0x7fd5c400ee80 comp rx=0 tx=0).stop 2024-03-18T22:40:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.451+0000 7fd5e24dc640 1 -- 172.21.15.40:0/1010921819 shutdown_connections 2024-03-18T22:40:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.451+0000 7fd5e24dc640 1 --2- 172.21.15.40:0/1010921819 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5c80a2800 0x7fd5c80a4cc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.451+0000 7fd5e24dc640 1 --2- 172.21.15.40:0/1010921819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5dc101ce0 0x7fd5dc19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.451+0000 7fd5e24dc640 1 --2- 172.21.15.40:0/1010921819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5dc1013d0 0x7fd5dc197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.451+0000 7fd5e24dc640 1 --2- 172.21.15.40:0/1010921819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5dc100a10 0x7fd5dc1979f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.451+0000 7fd5e24dc640 1 -- 172.21.15.40:0/1010921819 >> 172.21.15.40:0/1010921819 conn(0x7fd5dc0fb440 msgr2=0x7fd5dc0fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.451+0000 7fd5e24dc640 1 -- 172.21.15.40:0/1010921819 shutdown_connections 2024-03-18T22:40:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:19.451+0000 7fd5e24dc640 1 -- 172.21.15.40:0/1010921819 wait complete. 2024-03-18T22:40:19.984 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:19 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1010921819' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:20.330 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 22:40:20 smithi113 systemd[1]: Starting Ceph mds.f for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:40:20.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:19 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1010921819' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:20.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:19 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1010921819' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:20.672 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 22:40:20 smithi113 podman[97789]: 2024-03-18 22:40:20.285097943 +0000 UTC m=+0.018137997 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T22:40:20.672 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 22:40:20 smithi113 podman[97789]: 2024-03-18 22:40:20.411636804 +0000 UTC m=+0.144676840 container create a6d4038b9ceab36e5f3c0c59a48034e5fbc496c5a036e17f7da718cefce343e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-f, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-18T22:40:20.673 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 22:40:20 smithi113 podman[97789]: 2024-03-18 22:40:20.672592966 +0000 UTC m=+0.405633008 container init a6d4038b9ceab36e5f3c0c59a48034e5fbc496c5a036e17f7da718cefce343e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-f, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T22:40:20.992 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 22:40:20 smithi113 podman[97789]: 2024-03-18 22:40:20.675279894 +0000 UTC m=+0.408319928 container start a6d4038b9ceab36e5f3c0c59a48034e5fbc496c5a036e17f7da718cefce343e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-f, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307) 2024-03-18T22:40:20.992 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 22:40:20 smithi113 ceph-mds[97808]: -- 172.21.15.113:0/2320905129 <== mon.1 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ebd99b5a40 con 0x55ebd9988800 2024-03-18T22:40:20.992 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 22:40:20 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-f[97804]: starting mds.f at 2024-03-18T22:40:20.992 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 22:40:20 smithi113 bash[97789]: a6d4038b9ceab36e5f3c0c59a48034e5fbc496c5a036e17f7da718cefce343e0 2024-03-18T22:40:20.992 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 22:40:20 smithi113 systemd[1]: Started Ceph mds.f for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:40:20.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:20 smithi113 ceph-mon[66775]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:20.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:20.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:20.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:20.992 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:20.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:20.993 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:21.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:20 smithi103 ceph-mon[66377]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:21.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:21.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:21.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:21.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:21.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:21.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:21.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:20 smithi040 ceph-mon[62643]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:21.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:21.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:21.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:21.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:21.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:21.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:21.464 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:21.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.577+0000 7f37c26f7640 1 -- 172.21.15.40:0/1363978707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc100750 msgr2=0x7f37bc100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:21.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.577+0000 7f37c26f7640 1 --2- 172.21.15.40:0/1363978707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc100750 0x7f37bc100b70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009900 tx=0x7f37ac02bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:21.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.577+0000 7f37c26f7640 1 -- 172.21.15.40:0/1363978707 shutdown_connections 2024-03-18T22:40:21.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.577+0000 7f37c26f7640 1 --2- 172.21.15.40:0/1363978707 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37bc102300 0x7f37bc108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:21.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.577+0000 7f37c26f7640 1 --2- 172.21.15.40:0/1363978707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37bc101940 0x7f37bc101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:21.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.577+0000 7f37c26f7640 1 --2- 172.21.15.40:0/1363978707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc100750 0x7f37bc100b70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:21.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.577+0000 7f37c26f7640 1 -- 172.21.15.40:0/1363978707 >> 172.21.15.40:0/1363978707 conn(0x7f37bc0fbd00 msgr2=0x7f37bc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:21.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.577+0000 7f37c26f7640 1 -- 172.21.15.40:0/1363978707 shutdown_connections 2024-03-18T22:40:21.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.577+0000 7f37c26f7640 1 -- 172.21.15.40:0/1363978707 wait complete. 2024-03-18T22:40:21.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.578+0000 7f37c26f7640 1 Processor -- start 2024-03-18T22:40:21.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.578+0000 7f37c26f7640 1 -- start start 2024-03-18T22:40:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.578+0000 7f37c26f7640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37bc100750 0x7f37bc197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.578+0000 7f37c26f7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37bc101940 0x7f37bc198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.578+0000 7f37c26f7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc102300 0x7f37bc19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.578+0000 7f37c26f7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc10c280 con 0x7f37bc102300 2024-03-18T22:40:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.578+0000 7f37c26f7640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc10c100 con 0x7f37bc101940 2024-03-18T22:40:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.578+0000 7f37c26f7640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc10c400 con 0x7f37bc100750 2024-03-18T22:40:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.578+0000 7f37c0c94640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc102300 0x7f37bc19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.578+0000 7f37c0c94640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc102300 0x7f37bc19f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42912/0 (socket says 172.21.15.40:42912) 2024-03-18T22:40:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.578+0000 7f37c0c94640 1 -- 172.21.15.40:0/3332632898 learned_addr learned my addr 172.21.15.40:0/3332632898 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.578+0000 7f37bbfff640 1 --2- 172.21.15.40:0/3332632898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37bc100750 0x7f37bc197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.578+0000 7f37bb7fe640 1 --2- 172.21.15.40:0/3332632898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37bc101940 0x7f37bc198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.579+0000 7f37c0c94640 1 -- 172.21.15.40:0/3332632898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37bc100750 msgr2=0x7f37bc197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.579+0000 7f37c0c94640 1 --2- 172.21.15.40:0/3332632898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37bc100750 0x7f37bc197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.579+0000 7f37c0c94640 1 -- 172.21.15.40:0/3332632898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37bc101940 msgr2=0x7f37bc198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.579+0000 7f37c0c94640 1 --2- 172.21.15.40:0/3332632898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37bc101940 0x7f37bc198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.579+0000 7f37c0c94640 1 -- 172.21.15.40:0/3332632898 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37bc19f6c0 con 0x7f37bc102300 2024-03-18T22:40:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.579+0000 7f37c0c94640 1 --2- 172.21.15.40:0/3332632898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc102300 0x7f37bc19f050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f37b400eb10 tx=0x7f37b400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.579+0000 7f37b97fa640 1 -- 172.21.15.40:0/3332632898 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b4004b20 con 0x7f37bc102300 2024-03-18T22:40:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.579+0000 7f37b97fa640 1 -- 172.21.15.40:0/3332632898 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37b4004cc0 con 0x7f37bc102300 2024-03-18T22:40:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.579+0000 7f37c26f7640 1 -- 172.21.15.40:0/3332632898 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37bc19f9b0 con 0x7f37bc102300 2024-03-18T22:40:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.579+0000 7f37b97fa640 1 -- 172.21.15.40:0/3332632898 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b4010690 con 0x7f37bc102300 2024-03-18T22:40:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.579+0000 7f37c26f7640 1 -- 172.21.15.40:0/3332632898 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37bc19ff50 con 0x7f37bc102300 2024-03-18T22:40:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.580+0000 7f37c26f7640 1 -- 172.21.15.40:0/3332632898 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37bc1a0240 con 0x7f37bc102300 2024-03-18T22:40:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.581+0000 7f37b97fa640 1 -- 172.21.15.40:0/3332632898 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f37b4005020 con 0x7f37bc102300 2024-03-18T22:40:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.581+0000 7f37b97fa640 1 --2- 172.21.15.40:0/3332632898 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f37a40a2880 0x7f37a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.582+0000 7f37bbfff640 1 --2- 172.21.15.40:0/3332632898 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f37a40a2880 0x7f37a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.582+0000 7f37b97fa640 1 -- 172.21.15.40:0/3332632898 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f37b40ced50 con 0x7f37bc102300 2024-03-18T22:40:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.582+0000 7f37bbfff640 1 --2- 172.21.15.40:0/3332632898 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f37a40a2880 0x7f37a40a4d40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f37ac005ab0 tx=0x7f37ac005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:21.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.586+0000 7f37b97fa640 1 -- 172.21.15.40:0/3332632898 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f37b409c1f0 con 0x7f37bc102300 2024-03-18T22:40:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.811+0000 7f37c26f7640 1 -- 172.21.15.40:0/3332632898 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37bc1a0410 con 0x7f37bc102300 2024-03-18T22:40:21.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.812+0000 7f37b97fa640 1 -- 172.21.15.40:0/3332632898 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7f37b409ce20 con 0x7f37bc102300 2024-03-18T22:40:21.814 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:21.814 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-18T22:40:21.814 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 7 2024-03-18T22:40:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.814+0000 7f37c26f7640 1 -- 172.21.15.40:0/3332632898 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f37a40a2880 msgr2=0x7f37a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.814+0000 7f37c26f7640 1 --2- 172.21.15.40:0/3332632898 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f37a40a2880 0x7f37a40a4d40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f37ac005ab0 tx=0x7f37ac005fd0 comp rx=0 tx=0).stop 2024-03-18T22:40:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.814+0000 7f37c26f7640 1 -- 172.21.15.40:0/3332632898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc102300 msgr2=0x7f37bc19f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.814+0000 7f37c26f7640 1 --2- 172.21.15.40:0/3332632898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc102300 0x7f37bc19f050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f37b400eb10 tx=0x7f37b400c2d0 comp rx=0 tx=0).stop 2024-03-18T22:40:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.814+0000 7f37c26f7640 1 -- 172.21.15.40:0/3332632898 shutdown_connections 2024-03-18T22:40:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.814+0000 7f37c26f7640 1 --2- 172.21.15.40:0/3332632898 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f37a40a2880 0x7f37a40a4d40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.814+0000 7f37c26f7640 1 --2- 172.21.15.40:0/3332632898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37bc102300 0x7f37bc19f050 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.814+0000 7f37c26f7640 1 --2- 172.21.15.40:0/3332632898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37bc101940 0x7f37bc198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.814+0000 7f37c26f7640 1 --2- 172.21.15.40:0/3332632898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f37bc100750 0x7f37bc197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.814+0000 7f37c26f7640 1 -- 172.21.15.40:0/3332632898 >> 172.21.15.40:0/3332632898 conn(0x7f37bc0fbd00 msgr2=0x7f37bc069e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.814+0000 7f37c26f7640 1 -- 172.21.15.40:0/3332632898 shutdown_connections 2024-03-18T22:40:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:21.814+0000 7f37c26f7640 1 -- 172.21.15.40:0/3332632898 wait complete. 2024-03-18T22:40:22.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:21 smithi103 ceph-mon[66377]: Deploying daemon mds.c on smithi113 2024-03-18T22:40:22.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:21 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.113:6834/4040610039,v1:172.21.15.113:6835/4040610039] up:boot 2024-03-18T22:40:22.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:21 smithi103 ceph-mon[66377]: fsmap 6 up:standby 2024-03-18T22:40:22.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:21 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T22:40:22.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:21 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3332632898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:22.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:21 smithi040 ceph-mon[62643]: Deploying daemon mds.c on smithi113 2024-03-18T22:40:22.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:21 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.113:6834/4040610039,v1:172.21.15.113:6835/4040610039] up:boot 2024-03-18T22:40:22.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:21 smithi040 ceph-mon[62643]: fsmap 6 up:standby 2024-03-18T22:40:22.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:21 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T22:40:22.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:21 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3332632898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:22.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:21 smithi113 ceph-mon[66775]: Deploying daemon mds.c on smithi113 2024-03-18T22:40:22.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:21 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.113:6834/4040610039,v1:172.21.15.113:6835/4040610039] up:boot 2024-03-18T22:40:22.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:21 smithi113 ceph-mon[66775]: fsmap 6 up:standby 2024-03-18T22:40:22.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:21 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T22:40:22.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:21 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3332632898' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:23.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:23 smithi113 ceph-mon[66775]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:23 smithi103 ceph-mon[66377]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:23.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:23 smithi040 ceph-mon[62643]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:23.828 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:23.829 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 18 22:40:23 smithi113 systemd[1]: Started Ceph mds.c for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:40:23.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 -- 172.21.15.40:0/2491589493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870101930 msgr2=0x7f0870101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:23.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 --2- 172.21.15.40:0/2491589493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870101930 0x7f0870101db0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0864009900 tx=0x7f086402bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:23.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 -- 172.21.15.40:0/2491589493 shutdown_connections 2024-03-18T22:40:23.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 --2- 172.21.15.40:0/2491589493 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08701022f0 0x7f0870108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:23.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 --2- 172.21.15.40:0/2491589493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870101930 0x7f0870101db0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:23.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 --2- 172.21.15.40:0/2491589493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0870100740 0x7f0870100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:23.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 -- 172.21.15.40:0/2491589493 >> 172.21.15.40:0/2491589493 conn(0x7f08700fbcb0 msgr2=0x7f08700fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:23.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 -- 172.21.15.40:0/2491589493 shutdown_connections 2024-03-18T22:40:23.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 -- 172.21.15.40:0/2491589493 wait complete. 2024-03-18T22:40:23.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 Processor -- start 2024-03-18T22:40:23.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 -- start start 2024-03-18T22:40:23.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870100740 0x7f08701046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:23.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0870101930 0x7f0870102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08701022f0 0x7f0870103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f087010c270 con 0x7f0870100740 2024-03-18T22:40:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.942+0000 7f0877de1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f087010c0f0 con 0x7f08701022f0 2024-03-18T22:40:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f0877de1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f087010c3f0 con 0x7f0870101930 2024-03-18T22:40:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f0875b7d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870100740 0x7f08701046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f0875b7d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870100740 0x7f08701046d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42924/0 (socket says 172.21.15.40:42924) 2024-03-18T22:40:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f0875b7d640 1 -- 172.21.15.40:0/2940352912 learned_addr learned my addr 172.21.15.40:0/2940352912 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f087637e640 1 --2- 172.21.15.40:0/2940352912 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08701022f0 0x7f0870103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f087537c640 1 --2- 172.21.15.40:0/2940352912 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0870101930 0x7f0870102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f0875b7d640 1 -- 172.21.15.40:0/2940352912 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0870101930 msgr2=0x7f0870102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f0875b7d640 1 --2- 172.21.15.40:0/2940352912 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0870101930 0x7f0870102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f0875b7d640 1 -- 172.21.15.40:0/2940352912 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08701022f0 msgr2=0x7f0870103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f0875b7d640 1 --2- 172.21.15.40:0/2940352912 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08701022f0 0x7f0870103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f0875b7d640 1 -- 172.21.15.40:0/2940352912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0870103ae0 con 0x7f0870100740 2024-03-18T22:40:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f0875b7d640 1 --2- 172.21.15.40:0/2940352912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870100740 0x7f08701046d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f086800c970 tx=0x7f086800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:23.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f086effd640 1 -- 172.21.15.40:0/2940352912 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0868004220 con 0x7f0870100740 2024-03-18T22:40:23.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f086effd640 1 -- 172.21.15.40:0/2940352912 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08680043c0 con 0x7f0870100740 2024-03-18T22:40:23.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f0877de1640 1 -- 172.21.15.40:0/2940352912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08701a1be0 con 0x7f0870100740 2024-03-18T22:40:23.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f086effd640 1 -- 172.21.15.40:0/2940352912 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0868004a00 con 0x7f0870100740 2024-03-18T22:40:23.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.943+0000 7f0877de1640 1 -- 172.21.15.40:0/2940352912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08701a20b0 con 0x7f0870100740 2024-03-18T22:40:23.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.944+0000 7f0877de1640 1 -- 172.21.15.40:0/2940352912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0870068610 con 0x7f0870100740 2024-03-18T22:40:23.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.945+0000 7f086effd640 1 -- 172.21.15.40:0/2940352912 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f08680079b0 con 0x7f0870100740 2024-03-18T22:40:23.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.946+0000 7f086effd640 1 --2- 172.21.15.40:0/2940352912 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f08580a2800 0x7f08580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:23.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.946+0000 7f087537c640 1 --2- 172.21.15.40:0/2940352912 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f08580a2800 0x7f08580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:23.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.946+0000 7f086effd640 1 -- 172.21.15.40:0/2940352912 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f08680ce2d0 con 0x7f0870100740 2024-03-18T22:40:23.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.946+0000 7f087537c640 1 --2- 172.21.15.40:0/2940352912 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f08580a2800 0x7f08580a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0864009230 tx=0x7f0864005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:23.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:23.950+0000 7f086effd640 1 -- 172.21.15.40:0/2940352912 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0868014030 con 0x7f0870100740 2024-03-18T22:40:24.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:24.175+0000 7f0877de1640 1 -- 172.21.15.40:0/2940352912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08701a2280 con 0x7f0870100740 2024-03-18T22:40:24.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:24.175+0000 7f086effd640 1 -- 172.21.15.40:0/2940352912 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6355 (secure 0 0 0) 0x7f086809b1c0 con 0x7f0870100740 2024-03-18T22:40:24.177 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:24.177 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-18T22:40:24.178 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 8 2024-03-18T22:40:24.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:24.177+0000 7f0877de1640 1 -- 172.21.15.40:0/2940352912 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f08580a2800 msgr2=0x7f08580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:24.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:24.177+0000 7f0877de1640 1 --2- 172.21.15.40:0/2940352912 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f08580a2800 0x7f08580a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0864009230 tx=0x7f0864005fd0 comp rx=0 tx=0).stop 2024-03-18T22:40:24.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:24.177+0000 7f0877de1640 1 -- 172.21.15.40:0/2940352912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870100740 msgr2=0x7f08701046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:24.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:24.177+0000 7f0877de1640 1 --2- 172.21.15.40:0/2940352912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870100740 0x7f08701046d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f086800c970 tx=0x7f086800ce30 comp rx=0 tx=0).stop 2024-03-18T22:40:24.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:24.177+0000 7f0877de1640 1 -- 172.21.15.40:0/2940352912 shutdown_connections 2024-03-18T22:40:24.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:24.178+0000 7f0877de1640 1 --2- 172.21.15.40:0/2940352912 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f08580a2800 0x7f08580a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:24.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:24.178+0000 7f0877de1640 1 --2- 172.21.15.40:0/2940352912 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08701022f0 0x7f0870103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:24.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:24.178+0000 7f0877de1640 1 --2- 172.21.15.40:0/2940352912 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0870101930 0x7f0870102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:24.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:24.178+0000 7f0877de1640 1 --2- 172.21.15.40:0/2940352912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0870100740 0x7f08701046d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:24.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:24.178+0000 7f0877de1640 1 -- 172.21.15.40:0/2940352912 >> 172.21.15.40:0/2940352912 conn(0x7f08700fbcb0 msgr2=0x7f08700689d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:24.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:24.178+0000 7f0877de1640 1 -- 172.21.15.40:0/2940352912 shutdown_connections 2024-03-18T22:40:24.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:24.178+0000 7f0877de1640 1 -- 172.21.15.40:0/2940352912 wait complete. 2024-03-18T22:40:24.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:24.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:24.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:24.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:24.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:24.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:24.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:24 smithi103 ceph-mon[66377]: Deploying daemon mds.h on smithi103 2024-03-18T22:40:24.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:24 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.113:6836/4212429112,v1:172.21.15.113:6837/4212429112] up:boot 2024-03-18T22:40:24.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:24 smithi103 ceph-mon[66377]: fsmap 7 up:standby 2024-03-18T22:40:24.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T22:40:24.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:24 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2940352912' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:24.923 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:24.923 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:24.923 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:24.923 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:24.923 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:24.923 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:24.924 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:24 smithi113 ceph-mon[66775]: Deploying daemon mds.h on smithi103 2024-03-18T22:40:24.924 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:24 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.113:6836/4212429112,v1:172.21.15.113:6837/4212429112] up:boot 2024-03-18T22:40:24.924 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:24 smithi113 ceph-mon[66775]: fsmap 7 up:standby 2024-03-18T22:40:24.924 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T22:40:24.924 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:24 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2940352912' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:25.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:25.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:25.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:25.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:25.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:25.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:25.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:24 smithi040 ceph-mon[62643]: Deploying daemon mds.h on smithi103 2024-03-18T22:40:25.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:24 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.113:6836/4212429112,v1:172.21.15.113:6837/4212429112] up:boot 2024-03-18T22:40:25.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:24 smithi040 ceph-mon[62643]: fsmap 7 up:standby 2024-03-18T22:40:25.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T22:40:25.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:24 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2940352912' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:26.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:25 smithi113 ceph-mon[66775]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:26.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:25 smithi103 ceph-mon[66377]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:26.109 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 22:40:25 smithi103 systemd[1]: Starting Ceph mds.h for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:40:26.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:25 smithi040 ceph-mon[62643]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:26.191 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:26.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.305+0000 7fe661897640 1 -- 172.21.15.40:0/2427241307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe65c104bf0 msgr2=0x7fe65c106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:26.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.305+0000 7fe661897640 1 --2- 172.21.15.40:0/2427241307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe65c104bf0 0x7fe65c106fe0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe650009860 tx=0x7fe65002bb70 comp rx=0 tx=0).stop 2024-03-18T22:40:26.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.305+0000 7fe661897640 1 -- 172.21.15.40:0/2427241307 shutdown_connections 2024-03-18T22:40:26.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.305+0000 7fe661897640 1 --2- 172.21.15.40:0/2427241307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe65c107520 0x7fe65c109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:26.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.306+0000 7fe661897640 1 --2- 172.21.15.40:0/2427241307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe65c104bf0 0x7fe65c106fe0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:26.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.306+0000 7fe661897640 1 --2- 172.21.15.40:0/2427241307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe65c1022c0 0x7fe65c1046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:26.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.306+0000 7fe661897640 1 -- 172.21.15.40:0/2427241307 >> 172.21.15.40:0/2427241307 conn(0x7fe65c0fbcb0 msgr2=0x7fe65c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:26.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.306+0000 7fe661897640 1 -- 172.21.15.40:0/2427241307 shutdown_connections 2024-03-18T22:40:26.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.306+0000 7fe661897640 1 -- 172.21.15.40:0/2427241307 wait complete. 2024-03-18T22:40:26.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.306+0000 7fe661897640 1 Processor -- start 2024-03-18T22:40:26.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.306+0000 7fe661897640 1 -- start start 2024-03-18T22:40:26.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe661897640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe65c1022c0 0x7fe65c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:26.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe661897640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe65c104bf0 0x7fe65c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:26.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe661897640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe65c107520 0x7fe65c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:26.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe661897640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c10d360 con 0x7fe65c107520 2024-03-18T22:40:26.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe661897640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c10d1e0 con 0x7fe65c1022c0 2024-03-18T22:40:26.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe661897640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c10d4e0 con 0x7fe65c104bf0 2024-03-18T22:40:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe65b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe65c107520 0x7fe65c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe65b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe65c107520 0x7fe65c19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42940/0 (socket says 172.21.15.40:42940) 2024-03-18T22:40:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe65b7fe640 1 -- 172.21.15.40:0/1093434295 learned_addr learned my addr 172.21.15.40:0/1093434295 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe65a7fc640 1 --2- 172.21.15.40:0/1093434295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe65c104bf0 0x7fe65c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe65b7fe640 1 -- 172.21.15.40:0/1093434295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe65c104bf0 msgr2=0x7fe65c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe65b7fe640 1 --2- 172.21.15.40:0/1093434295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe65c104bf0 0x7fe65c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe65b7fe640 1 -- 172.21.15.40:0/1093434295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe65c1022c0 msgr2=0x7fe65c197a90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T22:40:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe65b7fe640 1 --2- 172.21.15.40:0/1093434295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe65c1022c0 0x7fe65c197a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe65b7fe640 1 -- 172.21.15.40:0/1093434295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe65c19f6b0 con 0x7fe65c107520 2024-03-18T22:40:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe65a7fc640 1 --2- 172.21.15.40:0/1093434295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe65c104bf0 0x7fe65c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:40:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe65b7fe640 1 --2- 172.21.15.40:0/1093434295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe65c107520 0x7fe65c19f040 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe64c009fd0 tx=0x7fe64c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe660895640 1 -- 172.21.15.40:0/1093434295 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c017070 con 0x7fe65c107520 2024-03-18T22:40:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe660895640 1 -- 172.21.15.40:0/1093434295 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe64c003e20 con 0x7fe65c107520 2024-03-18T22:40:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe661897640 1 -- 172.21.15.40:0/1093434295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe65c19f9a0 con 0x7fe65c107520 2024-03-18T22:40:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe660895640 1 -- 172.21.15.40:0/1093434295 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c010460 con 0x7fe65c107520 2024-03-18T22:40:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.307+0000 7fe661897640 1 -- 172.21.15.40:0/1093434295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe65c19fea0 con 0x7fe65c107520 2024-03-18T22:40:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.308+0000 7fe661897640 1 -- 172.21.15.40:0/1093434295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe65c0fd7a0 con 0x7fe65c107520 2024-03-18T22:40:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.309+0000 7fe660895640 1 -- 172.21.15.40:0/1093434295 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe64c003970 con 0x7fe65c107520 2024-03-18T22:40:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.310+0000 7fe660895640 1 --2- 172.21.15.40:0/1093434295 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe6440a2800 0x7fe6440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.310+0000 7fe65affd640 1 --2- 172.21.15.40:0/1093434295 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe6440a2800 0x7fe6440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.310+0000 7fe660895640 1 -- 172.21.15.40:0/1093434295 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe64c0ce470 con 0x7fe65c107520 2024-03-18T22:40:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.310+0000 7fe65affd640 1 --2- 172.21.15.40:0/1093434295 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe6440a2800 0x7fe6440a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe6540096c0 tx=0x7fe654009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.314+0000 7fe660895640 1 -- 172.21.15.40:0/1093434295 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe64c09b9b0 con 0x7fe65c107520 2024-03-18T22:40:26.406 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 22:40:26 smithi103 podman[95604]: 2024-03-18 22:40:26.023806847 +0000 UTC m=+0.018211372 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T22:40:26.406 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 22:40:26 smithi103 podman[95604]: 2024-03-18 22:40:26.12817871 +0000 UTC m=+0.122583230 container create ae455a62f9386640c6a1a3ce012ffbf8d20686a4482703c2531b435991845573 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-h, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T22:40:26.406 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 22:40:26 smithi103 podman[95604]: 2024-03-18 22:40:26.406093672 +0000 UTC m=+0.400498190 container init ae455a62f9386640c6a1a3ce012ffbf8d20686a4482703c2531b435991845573 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-h, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-18T22:40:26.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.537+0000 7fe661897640 1 -- 172.21.15.40:0/1093434295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe65c062420 con 0x7fe65c107520 2024-03-18T22:40:26.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.537+0000 7fe660895640 1 -- 172.21.15.40:0/1093434295 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6355 (secure 0 0 0) 0x7fe64c095e20 con 0x7fe65c107520 2024-03-18T22:40:26.538 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:26.538 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-18T22:40:26.538 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 8 2024-03-18T22:40:26.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.539+0000 7fe661897640 1 -- 172.21.15.40:0/1093434295 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe6440a2800 msgr2=0x7fe6440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:26.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.539+0000 7fe661897640 1 --2- 172.21.15.40:0/1093434295 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe6440a2800 0x7fe6440a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe6540096c0 tx=0x7fe654009260 comp rx=0 tx=0).stop 2024-03-18T22:40:26.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.539+0000 7fe661897640 1 -- 172.21.15.40:0/1093434295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe65c107520 msgr2=0x7fe65c19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:26.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.539+0000 7fe661897640 1 --2- 172.21.15.40:0/1093434295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe65c107520 0x7fe65c19f040 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe64c009fd0 tx=0x7fe64c00b260 comp rx=0 tx=0).stop 2024-03-18T22:40:26.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.539+0000 7fe661897640 1 -- 172.21.15.40:0/1093434295 shutdown_connections 2024-03-18T22:40:26.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.539+0000 7fe661897640 1 --2- 172.21.15.40:0/1093434295 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe6440a2800 0x7fe6440a4cc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:26.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.539+0000 7fe661897640 1 --2- 172.21.15.40:0/1093434295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe65c107520 0x7fe65c19f040 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:26.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.539+0000 7fe661897640 1 --2- 172.21.15.40:0/1093434295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe65c104bf0 0x7fe65c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:26.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.539+0000 7fe661897640 1 --2- 172.21.15.40:0/1093434295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe65c1022c0 0x7fe65c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:26.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.539+0000 7fe661897640 1 -- 172.21.15.40:0/1093434295 >> 172.21.15.40:0/1093434295 conn(0x7fe65c0fbcb0 msgr2=0x7fe65c102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:26.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.539+0000 7fe661897640 1 -- 172.21.15.40:0/1093434295 shutdown_connections 2024-03-18T22:40:26.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:26.539+0000 7fe661897640 1 -- 172.21.15.40:0/1093434295 wait complete. 2024-03-18T22:40:26.665 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 22:40:26 smithi103 podman[95604]: 2024-03-18 22:40:26.408782323 +0000 UTC m=+0.403186844 container start ae455a62f9386640c6a1a3ce012ffbf8d20686a4482703c2531b435991845573 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-h, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T22:40:26.665 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 22:40:26 smithi103 ceph-mds[95623]: -- 172.21.15.103:0/3872970608 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56291c91ba40 con 0x56291c8ee800 2024-03-18T22:40:26.665 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 22:40:26 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-h[95619]: starting mds.h at 2024-03-18T22:40:26.665 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 22:40:26 smithi103 bash[95604]: ae455a62f9386640c6a1a3ce012ffbf8d20686a4482703c2531b435991845573 2024-03-18T22:40:26.665 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 22:40:26 smithi103 systemd[1]: Started Ceph mds.h for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:40:26.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:26 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1093434295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:26.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:26.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:26.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:26.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:26.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:26.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:27.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:26 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1093434295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:27.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:27.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:27.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:27.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:27.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:27.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:27.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:26 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1093434295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:27.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:27.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:27.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:27.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:27.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:27.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:27.827 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:27 smithi103 ceph-mon[66377]: Deploying daemon mds.i on smithi113 2024-03-18T22:40:27.827 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:27 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] up:boot 2024-03-18T22:40:27.828 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:27 smithi103 ceph-mon[66377]: fsmap 8 up:standby 2024-03-18T22:40:27.828 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:27 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T22:40:27.828 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:27 smithi103 ceph-mon[66377]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:27.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:27 smithi113 ceph-mon[66775]: Deploying daemon mds.i on smithi113 2024-03-18T22:40:27.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:27 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] up:boot 2024-03-18T22:40:27.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:27 smithi113 ceph-mon[66775]: fsmap 8 up:standby 2024-03-18T22:40:27.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:27 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T22:40:27.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:27 smithi113 ceph-mon[66775]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:28.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:27 smithi040 ceph-mon[62643]: Deploying daemon mds.i on smithi113 2024-03-18T22:40:28.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:27 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] up:boot 2024-03-18T22:40:28.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:27 smithi040 ceph-mon[62643]: fsmap 8 up:standby 2024-03-18T22:40:28.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:27 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T22:40:28.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:27 smithi040 ceph-mon[62643]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:28.552 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:28.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.666+0000 7fb274db3640 1 -- 172.21.15.40:0/316901314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270102a50 msgr2=0x7fb270102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:28.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.666+0000 7fb274db3640 1 --2- 172.21.15.40:0/316901314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270102a50 0x7fb270102ed0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb258009900 tx=0x7fb25802bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:28.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.667+0000 7fb274db3640 1 -- 172.21.15.40:0/316901314 shutdown_connections 2024-03-18T22:40:28.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.667+0000 7fb274db3640 1 --2- 172.21.15.40:0/316901314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb270103410 0x7fb270109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:28.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.667+0000 7fb274db3640 1 --2- 172.21.15.40:0/316901314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270102a50 0x7fb270102ed0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:28.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.667+0000 7fb274db3640 1 --2- 172.21.15.40:0/316901314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb270101860 0x7fb270101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:28.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.667+0000 7fb274db3640 1 -- 172.21.15.40:0/316901314 >> 172.21.15.40:0/316901314 conn(0x7fb2700fcdd0 msgr2=0x7fb2700ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:28.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.667+0000 7fb274db3640 1 -- 172.21.15.40:0/316901314 shutdown_connections 2024-03-18T22:40:28.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.667+0000 7fb274db3640 1 -- 172.21.15.40:0/316901314 wait complete. 2024-03-18T22:40:28.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.667+0000 7fb274db3640 1 Processor -- start 2024-03-18T22:40:28.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.667+0000 7fb274db3640 1 -- start start 2024-03-18T22:40:28.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb274db3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270101860 0x7fb270197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:28.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb274db3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb270102a50 0x7fb270197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:28.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb274db3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb270103410 0x7fb27019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:28.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb274db3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb27010d3c0 con 0x7fb270101860 2024-03-18T22:40:28.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb274db3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb27010d240 con 0x7fb270102a50 2024-03-18T22:40:28.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb274db3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb27010d540 con 0x7fb270103410 2024-03-18T22:40:28.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb26e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270101860 0x7fb270197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:28.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb26e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270101860 0x7fb270197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38064/0 (socket says 172.21.15.40:38064) 2024-03-18T22:40:28.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb26e59c640 1 -- 172.21.15.40:0/3583187064 learned_addr learned my addr 172.21.15.40:0/3583187064 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:28.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb26ed9d640 1 --2- 172.21.15.40:0/3583187064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb270103410 0x7fb27019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb26dd9b640 1 --2- 172.21.15.40:0/3583187064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb270102a50 0x7fb270197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb26e59c640 1 -- 172.21.15.40:0/3583187064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb270103410 msgr2=0x7fb27019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb26e59c640 1 --2- 172.21.15.40:0/3583187064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb270103410 0x7fb27019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb26e59c640 1 -- 172.21.15.40:0/3583187064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb270102a50 msgr2=0x7fb270197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb26e59c640 1 --2- 172.21.15.40:0/3583187064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb270102a50 0x7fb270197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb26e59c640 1 -- 172.21.15.40:0/3583187064 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27019f6d0 con 0x7fb270101860 2024-03-18T22:40:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb26e59c640 1 --2- 172.21.15.40:0/3583187064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270101860 0x7fb270197ab0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb26800d950 tx=0x7fb26800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb2677fe640 1 -- 172.21.15.40:0/3583187064 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb268017020 con 0x7fb270101860 2024-03-18T22:40:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.668+0000 7fb274db3640 1 -- 172.21.15.40:0/3583187064 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb27019f9c0 con 0x7fb270101860 2024-03-18T22:40:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.669+0000 7fb2677fe640 1 -- 172.21.15.40:0/3583187064 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb268009c30 con 0x7fb270101860 2024-03-18T22:40:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.669+0000 7fb2677fe640 1 -- 172.21.15.40:0/3583187064 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26800f6b0 con 0x7fb270101860 2024-03-18T22:40:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.669+0000 7fb274db3640 1 -- 172.21.15.40:0/3583187064 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb27019ff30 con 0x7fb270101860 2024-03-18T22:40:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.670+0000 7fb274db3640 1 -- 172.21.15.40:0/3583187064 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb270108c50 con 0x7fb270101860 2024-03-18T22:40:28.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.671+0000 7fb2677fe640 1 -- 172.21.15.40:0/3583187064 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb26801a020 con 0x7fb270101860 2024-03-18T22:40:28.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.671+0000 7fb2677fe640 1 --2- 172.21.15.40:0/3583187064 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb25c0a2800 0x7fb25c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:28.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.671+0000 7fb26dd9b640 1 --2- 172.21.15.40:0/3583187064 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb25c0a2800 0x7fb25c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:28.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.671+0000 7fb2677fe640 1 -- 172.21.15.40:0/3583187064 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb2680ce1f0 con 0x7fb270101860 2024-03-18T22:40:28.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.672+0000 7fb26dd9b640 1 --2- 172.21.15.40:0/3583187064 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb25c0a2800 0x7fb25c0a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb258000c00 tx=0x7fb25802c7e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:28.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.676+0000 7fb2677fe640 1 -- 172.21.15.40:0/3583187064 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb268009de0 con 0x7fb270101860 2024-03-18T22:40:28.830 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 22:40:28 smithi113 systemd[1]: Starting Ceph mds.i for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:40:28.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.899+0000 7fb274db3640 1 -- 172.21.15.40:0/3583187064 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb27002d410 con 0x7fb270101860 2024-03-18T22:40:28.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.899+0000 7fb2677fe640 1 -- 172.21.15.40:0/3583187064 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7183 (secure 0 0 0) 0x7fb26809b730 con 0x7fb270101860 2024-03-18T22:40:28.900 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:28.900 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-18T22:40:28.901 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 9 2024-03-18T22:40:28.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.902+0000 7fb274db3640 1 -- 172.21.15.40:0/3583187064 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb25c0a2800 msgr2=0x7fb25c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:28.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.902+0000 7fb274db3640 1 --2- 172.21.15.40:0/3583187064 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb25c0a2800 0x7fb25c0a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb258000c00 tx=0x7fb25802c7e0 comp rx=0 tx=0).stop 2024-03-18T22:40:28.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.902+0000 7fb274db3640 1 -- 172.21.15.40:0/3583187064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270101860 msgr2=0x7fb270197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:28.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.902+0000 7fb274db3640 1 --2- 172.21.15.40:0/3583187064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270101860 0x7fb270197ab0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb26800d950 tx=0x7fb26800de10 comp rx=0 tx=0).stop 2024-03-18T22:40:28.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.902+0000 7fb274db3640 1 -- 172.21.15.40:0/3583187064 shutdown_connections 2024-03-18T22:40:28.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.902+0000 7fb274db3640 1 --2- 172.21.15.40:0/3583187064 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb25c0a2800 0x7fb25c0a4cc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:28.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.902+0000 7fb274db3640 1 --2- 172.21.15.40:0/3583187064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb270103410 0x7fb27019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:28.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.902+0000 7fb274db3640 1 --2- 172.21.15.40:0/3583187064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb270102a50 0x7fb270197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:28.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.902+0000 7fb274db3640 1 --2- 172.21.15.40:0/3583187064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb270101860 0x7fb270197ab0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:28.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.902+0000 7fb274db3640 1 -- 172.21.15.40:0/3583187064 >> 172.21.15.40:0/3583187064 conn(0x7fb2700fcdd0 msgr2=0x7fb270105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:28.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.902+0000 7fb274db3640 1 -- 172.21.15.40:0/3583187064 shutdown_connections 2024-03-18T22:40:28.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:28.902+0000 7fb274db3640 1 -- 172.21.15.40:0/3583187064 wait complete. 2024-03-18T22:40:29.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:28 smithi103 ceph-mon[66377]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:29.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:28 smithi040 ceph-mon[62643]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:29.177 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:28 smithi113 ceph-mon[66775]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:29.178 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 22:40:28 smithi113 podman[98422]: 2024-03-18 22:40:28.811955009 +0000 UTC m=+0.017487937 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T22:40:29.178 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 22:40:28 smithi113 podman[98422]: 2024-03-18 22:40:28.930949229 +0000 UTC m=+0.136482151 container create 124c2cdc599d40bbf23c3b1922404257e247b2687696f9df684e7504eea43be5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-i, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T22:40:29.178 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 22:40:29 smithi113 podman[98422]: 2024-03-18 22:40:29.177606115 +0000 UTC m=+0.383139035 container init 124c2cdc599d40bbf23c3b1922404257e247b2687696f9df684e7504eea43be5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-i, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-18T22:40:29.580 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 22:40:29 smithi113 podman[98422]: 2024-03-18 22:40:29.180414617 +0000 UTC m=+0.385947539 container start 124c2cdc599d40bbf23c3b1922404257e247b2687696f9df684e7504eea43be5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-i, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T22:40:29.580 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 22:40:29 smithi113 ceph-mds[98441]: -- 172.21.15.113:0/3707031092 <== mon.1 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5612f4b0da40 con 0x5612f4ae0c00 2024-03-18T22:40:29.580 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 22:40:29 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-i[98437]: starting mds.i at 2024-03-18T22:40:29.580 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 22:40:29 smithi113 bash[98422]: 124c2cdc599d40bbf23c3b1922404257e247b2687696f9df684e7504eea43be5 2024-03-18T22:40:29.580 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 22:40:29 smithi113 systemd[1]: Started Ceph mds.i for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:40:30.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:29 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3583187064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:30.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:30.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:30.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:30.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:30.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:30.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:30.121 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:29 smithi040 ceph-mon[62643]: Deploying daemon mds.g on smithi040 2024-03-18T22:40:30.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:29 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3583187064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:30.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:30.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:30.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:30.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:30.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:30.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:30.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:29 smithi113 ceph-mon[66775]: Deploying daemon mds.g on smithi040 2024-03-18T22:40:30.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:29 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3583187064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:30.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:30.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:30.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:30.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:30.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:30.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:30.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:29 smithi103 ceph-mon[66377]: Deploying daemon mds.g on smithi040 2024-03-18T22:40:30.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:31.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.032+0000 7fd642e85640 1 -- 172.21.15.40:0/1401207471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63c0ffe80 msgr2=0x7fd63c1002a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:31.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.032+0000 7fd642e85640 1 --2- 172.21.15.40:0/1401207471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63c0ffe80 0x7fd63c1002a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd630009900 tx=0x7fd63002bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:31.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.032+0000 7fd642e85640 1 -- 172.21.15.40:0/1401207471 shutdown_connections 2024-03-18T22:40:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd642e85640 1 --2- 172.21.15.40:0/1401207471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd63c101a20 0x7fd63c108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd642e85640 1 --2- 172.21.15.40:0/1401207471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd63c101060 0x7fd63c1014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd642e85640 1 --2- 172.21.15.40:0/1401207471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63c0ffe80 0x7fd63c1002a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd642e85640 1 -- 172.21.15.40:0/1401207471 >> 172.21.15.40:0/1401207471 conn(0x7fd63c0fb440 msgr2=0x7fd63c0fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd642e85640 1 -- 172.21.15.40:0/1401207471 shutdown_connections 2024-03-18T22:40:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd642e85640 1 -- 172.21.15.40:0/1401207471 wait complete. 2024-03-18T22:40:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd642e85640 1 Processor -- start 2024-03-18T22:40:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd642e85640 1 -- start start 2024-03-18T22:40:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd642e85640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63c0ffe80 0x7fd63c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd642e85640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd63c101060 0x7fd63c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd642e85640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd63c101a20 0x7fd63c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd642e85640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c10b1a0 con 0x7fd63c0ffe80 2024-03-18T22:40:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd642e85640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c10b020 con 0x7fd63c101060 2024-03-18T22:40:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd642e85640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c10b320 con 0x7fd63c101a20 2024-03-18T22:40:31.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd640c21640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63c0ffe80 0x7fd63c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd640c21640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63c0ffe80 0x7fd63c197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38082/0 (socket says 172.21.15.40:38082) 2024-03-18T22:40:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.033+0000 7fd640c21640 1 -- 172.21.15.40:0/2690717224 learned_addr learned my addr 172.21.15.40:0/2690717224 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.034+0000 7fd63bfff640 1 --2- 172.21.15.40:0/2690717224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd63c101060 0x7fd63c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.034+0000 7fd641422640 1 --2- 172.21.15.40:0/2690717224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd63c101a20 0x7fd63c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.034+0000 7fd640c21640 1 -- 172.21.15.40:0/2690717224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd63c101a20 msgr2=0x7fd63c19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.034+0000 7fd640c21640 1 --2- 172.21.15.40:0/2690717224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd63c101a20 0x7fd63c19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.034+0000 7fd640c21640 1 -- 172.21.15.40:0/2690717224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd63c101060 msgr2=0x7fd63c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.034+0000 7fd640c21640 1 --2- 172.21.15.40:0/2690717224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd63c101060 0x7fd63c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.034+0000 7fd640c21640 1 -- 172.21.15.40:0/2690717224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63c19f6d0 con 0x7fd63c0ffe80 2024-03-18T22:40:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.034+0000 7fd640c21640 1 --2- 172.21.15.40:0/2690717224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63c0ffe80 0x7fd63c197ab0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd630000c00 tx=0x7fd630004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.034+0000 7fd639ffb640 1 -- 172.21.15.40:0/2690717224 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd630043020 con 0x7fd63c0ffe80 2024-03-18T22:40:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.034+0000 7fd639ffb640 1 -- 172.21.15.40:0/2690717224 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd630004aa0 con 0x7fd63c0ffe80 2024-03-18T22:40:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.034+0000 7fd642e85640 1 -- 172.21.15.40:0/2690717224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd63c19f960 con 0x7fd63c0ffe80 2024-03-18T22:40:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.034+0000 7fd639ffb640 1 -- 172.21.15.40:0/2690717224 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd630003d00 con 0x7fd63c0ffe80 2024-03-18T22:40:31.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.034+0000 7fd642e85640 1 -- 172.21.15.40:0/2690717224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd63c19ff00 con 0x7fd63c0ffe80 2024-03-18T22:40:31.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.035+0000 7fd642e85640 1 -- 172.21.15.40:0/2690717224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd63c1a0130 con 0x7fd63c0ffe80 2024-03-18T22:40:31.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.036+0000 7fd639ffb640 1 -- 172.21.15.40:0/2690717224 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd630046020 con 0x7fd63c0ffe80 2024-03-18T22:40:31.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.037+0000 7fd639ffb640 1 --2- 172.21.15.40:0/2690717224 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd62c0a2880 0x7fd62c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:31.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.037+0000 7fd639ffb640 1 -- 172.21.15.40:0/2690717224 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd6300f0f90 con 0x7fd63c0ffe80 2024-03-18T22:40:31.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.037+0000 7fd63bfff640 1 --2- 172.21.15.40:0/2690717224 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd62c0a2880 0x7fd62c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:31.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.038+0000 7fd63bfff640 1 --2- 172.21.15.40:0/2690717224 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd62c0a2880 0x7fd62c0a4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd628005fd0 tx=0x7fd628005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:31.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.041+0000 7fd639ffb640 1 -- 172.21.15.40:0/2690717224 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd63004d020 con 0x7fd63c0ffe80 2024-03-18T22:40:31.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:30 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] up:boot 2024-03-18T22:40:31.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:30 smithi040 ceph-mon[62643]: fsmap 9 up:standby 2024-03-18T22:40:31.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:30 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T22:40:31.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:30 smithi040 ceph-mon[62643]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:31.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.272+0000 7fd642e85640 1 -- 172.21.15.40:0/2690717224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd63c1a0300 con 0x7fd63c0ffe80 2024-03-18T22:40:31.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.273+0000 7fd639ffb640 1 -- 172.21.15.40:0/2690717224 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7fd6300be4b0 con 0x7fd63c0ffe80 2024-03-18T22:40:31.274 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:31.274 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T22:40:31.274 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 10 2024-03-18T22:40:31.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.275+0000 7fd642e85640 1 -- 172.21.15.40:0/2690717224 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd62c0a2880 msgr2=0x7fd62c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:31.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.275+0000 7fd642e85640 1 --2- 172.21.15.40:0/2690717224 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd62c0a2880 0x7fd62c0a4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd628005fd0 tx=0x7fd628005850 comp rx=0 tx=0).stop 2024-03-18T22:40:31.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.275+0000 7fd642e85640 1 -- 172.21.15.40:0/2690717224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63c0ffe80 msgr2=0x7fd63c197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:31.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.275+0000 7fd642e85640 1 --2- 172.21.15.40:0/2690717224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63c0ffe80 0x7fd63c197ab0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd630000c00 tx=0x7fd630004270 comp rx=0 tx=0).stop 2024-03-18T22:40:31.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.275+0000 7fd642e85640 1 -- 172.21.15.40:0/2690717224 shutdown_connections 2024-03-18T22:40:31.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.275+0000 7fd642e85640 1 --2- 172.21.15.40:0/2690717224 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd62c0a2880 0x7fd62c0a4d40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:31.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.275+0000 7fd642e85640 1 --2- 172.21.15.40:0/2690717224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd63c101a20 0x7fd63c19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:31.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.275+0000 7fd642e85640 1 --2- 172.21.15.40:0/2690717224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd63c101060 0x7fd63c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:31.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.275+0000 7fd642e85640 1 --2- 172.21.15.40:0/2690717224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63c0ffe80 0x7fd63c197ab0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:31.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.275+0000 7fd642e85640 1 -- 172.21.15.40:0/2690717224 >> 172.21.15.40:0/2690717224 conn(0x7fd63c0fb440 msgr2=0x7fd63c106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:31.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.275+0000 7fd642e85640 1 -- 172.21.15.40:0/2690717224 shutdown_connections 2024-03-18T22:40:31.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:31.275+0000 7fd642e85640 1 -- 172.21.15.40:0/2690717224 wait complete. 2024-03-18T22:40:31.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:30 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] up:boot 2024-03-18T22:40:31.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:30 smithi113 ceph-mon[66775]: fsmap 9 up:standby 2024-03-18T22:40:31.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:30 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T22:40:31.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:30 smithi113 ceph-mon[66775]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:31.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:30 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] up:boot 2024-03-18T22:40:31.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:30 smithi103 ceph-mon[66377]: fsmap 9 up:standby 2024-03-18T22:40:31.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:30 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T22:40:31.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:30 smithi103 ceph-mon[66377]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:32.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:31 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2690717224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:32.144 INFO:journalctl@ceph.mds.g.smithi040.stdout:Mar 18 22:40:32 smithi040 systemd[1]: Started Ceph mds.g for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:40:32.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:31 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2690717224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:32.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:31 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2690717224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:33.288 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:33.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:33 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:33.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:33 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:33.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:33 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:33.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:33 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:33.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:33 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:33.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:33 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:33.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:33 smithi040 ceph-mon[62643]: Deploying daemon mds.a on smithi040 2024-03-18T22:40:33.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:33 smithi040 ceph-mon[62643]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:33.315 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:33 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.40:6838/4264194754,v1:172.21.15.40:6839/4264194754] up:boot 2024-03-18T22:40:33.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:33 smithi040 ceph-mon[62643]: fsmap 10 up:standby 2024-03-18T22:40:33.316 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:33 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T22:40:33.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.399+0000 7f7803812640 1 -- 172.21.15.40:0/1632782525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77fc105610 msgr2=0x7f77fc10bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.399+0000 7f7803812640 1 --2- 172.21.15.40:0/1632782525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77fc105610 0x7f77fc10bea0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f77f4009900 tx=0x7f77f402bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.399+0000 7f7803812640 1 -- 172.21.15.40:0/1632782525 shutdown_connections 2024-03-18T22:40:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.399+0000 7f7803812640 1 --2- 172.21.15.40:0/1632782525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77fc105610 0x7f77fc10bea0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.399+0000 7f7803812640 1 --2- 172.21.15.40:0/1632782525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77fc104c50 0x7f77fc1050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.399+0000 7f7803812640 1 --2- 172.21.15.40:0/1632782525 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77fc103a60 0x7f77fc103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.399+0000 7f7803812640 1 -- 172.21.15.40:0/1632782525 >> 172.21.15.40:0/1632782525 conn(0x7f77fc0fefd0 msgr2=0x7f77fc101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.399+0000 7f7803812640 1 -- 172.21.15.40:0/1632782525 shutdown_connections 2024-03-18T22:40:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.399+0000 7f7803812640 1 -- 172.21.15.40:0/1632782525 wait complete. 2024-03-18T22:40:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f7803812640 1 Processor -- start 2024-03-18T22:40:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f7803812640 1 -- start start 2024-03-18T22:40:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f7803812640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77fc103a60 0x7f77fc199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f7803812640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77fc104c50 0x7f77fc19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:33.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f7803812640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77fc105610 0x7f77fc1a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f7803812640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f77fc10f590 con 0x7f77fc103a60 2024-03-18T22:40:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f7803812640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f77fc10f410 con 0x7f77fc105610 2024-03-18T22:40:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f7803812640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f77fc10f710 con 0x7f77fc104c50 2024-03-18T22:40:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f78015ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77fc103a60 0x7f77fc199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f78015ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77fc103a60 0x7f77fc199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38134/0 (socket says 172.21.15.40:38134) 2024-03-18T22:40:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f78015ae640 1 -- 172.21.15.40:0/2788658675 learned_addr learned my addr 172.21.15.40:0/2788658675 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f7801daf640 1 --2- 172.21.15.40:0/2788658675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77fc105610 0x7f77fc1a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f7800dad640 1 --2- 172.21.15.40:0/2788658675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77fc104c50 0x7f77fc19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f78015ae640 1 -- 172.21.15.40:0/2788658675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77fc104c50 msgr2=0x7f77fc19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f78015ae640 1 --2- 172.21.15.40:0/2788658675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77fc104c50 0x7f77fc19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f78015ae640 1 -- 172.21.15.40:0/2788658675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77fc105610 msgr2=0x7f77fc1a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f78015ae640 1 --2- 172.21.15.40:0/2788658675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77fc105610 0x7f77fc1a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.400+0000 7f78015ae640 1 -- 172.21.15.40:0/2788658675 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77fc1a18b0 con 0x7f77fc103a60 2024-03-18T22:40:33.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.401+0000 7f78015ae640 1 --2- 172.21.15.40:0/2788658675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77fc103a60 0x7f77fc199c70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f77f800eb70 tx=0x7f77f800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:33.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.401+0000 7f77f27fc640 1 -- 172.21.15.40:0/2788658675 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f8003b70 con 0x7f77fc103a60 2024-03-18T22:40:33.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.401+0000 7f77f27fc640 1 -- 172.21.15.40:0/2788658675 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77f800ee70 con 0x7f77fc103a60 2024-03-18T22:40:33.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.401+0000 7f77f27fc640 1 -- 172.21.15.40:0/2788658675 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f8010650 con 0x7f77fc103a60 2024-03-18T22:40:33.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.401+0000 7f7803812640 1 -- 172.21.15.40:0/2788658675 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77fc1a1ba0 con 0x7f77fc103a60 2024-03-18T22:40:33.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.401+0000 7f7803812640 1 -- 172.21.15.40:0/2788658675 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77fc1a20f0 con 0x7f77fc103a60 2024-03-18T22:40:33.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.402+0000 7f7803812640 1 -- 172.21.15.40:0/2788658675 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77fc10ae00 con 0x7f77fc103a60 2024-03-18T22:40:33.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.403+0000 7f77f27fc640 1 -- 172.21.15.40:0/2788658675 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f77f8004a40 con 0x7f77fc103a60 2024-03-18T22:40:33.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.403+0000 7f77f27fc640 1 --2- 172.21.15.40:0/2788658675 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77e80a2880 0x7f77e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:33.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.404+0000 7f7800dad640 1 --2- 172.21.15.40:0/2788658675 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77e80a2880 0x7f77e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:33.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.404+0000 7f77f27fc640 1 -- 172.21.15.40:0/2788658675 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f77f80cecb0 con 0x7f77fc103a60 2024-03-18T22:40:33.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.404+0000 7f7800dad640 1 --2- 172.21.15.40:0/2788658675 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77e80a2880 0x7f77e80a4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f77fc19b170 tx=0x7f77ec005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:33.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.407+0000 7f77f27fc640 1 -- 172.21.15.40:0/2788658675 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f77f809c1f0 con 0x7f77fc103a60 2024-03-18T22:40:33.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:33 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:33.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:33 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:33.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:33 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:33.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:33 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:33.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:33 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:33.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:33 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:33.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:33 smithi113 ceph-mon[66775]: Deploying daemon mds.a on smithi040 2024-03-18T22:40:33.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:33 smithi113 ceph-mon[66775]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:33.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:33 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.40:6838/4264194754,v1:172.21.15.40:6839/4264194754] up:boot 2024-03-18T22:40:33.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:33 smithi113 ceph-mon[66775]: fsmap 10 up:standby 2024-03-18T22:40:33.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:33 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T22:40:33.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:33 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:33.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:33 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:33.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:33 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:33.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:33 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:33.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:33 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:33.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:33 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:33.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:33 smithi103 ceph-mon[66377]: Deploying daemon mds.a on smithi040 2024-03-18T22:40:33.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:33 smithi103 ceph-mon[66377]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:33.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:33 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.40:6838/4264194754,v1:172.21.15.40:6839/4264194754] up:boot 2024-03-18T22:40:33.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:33 smithi103 ceph-mon[66377]: fsmap 10 up:standby 2024-03-18T22:40:33.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:33 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T22:40:33.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.633+0000 7f7803812640 1 -- 172.21.15.40:0/2788658675 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77fc02d1f0 con 0x7f77fc103a60 2024-03-18T22:40:33.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.634+0000 7f77f27fc640 1 -- 172.21.15.40:0/2788658675 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8839 (secure 0 0 0) 0x7f77f809baf0 con 0x7f77fc103a60 2024-03-18T22:40:33.635 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:33.635 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T22:40:33.635 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 11 2024-03-18T22:40:33.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.636+0000 7f7803812640 1 -- 172.21.15.40:0/2788658675 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77e80a2880 msgr2=0x7f77e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:33.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.636+0000 7f7803812640 1 --2- 172.21.15.40:0/2788658675 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77e80a2880 0x7f77e80a4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f77fc19b170 tx=0x7f77ec005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:33.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.636+0000 7f7803812640 1 -- 172.21.15.40:0/2788658675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77fc103a60 msgr2=0x7f77fc199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:33.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.636+0000 7f7803812640 1 --2- 172.21.15.40:0/2788658675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77fc103a60 0x7f77fc199c70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f77f800eb70 tx=0x7f77f800c6a0 comp rx=0 tx=0).stop 2024-03-18T22:40:33.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.637+0000 7f7803812640 1 -- 172.21.15.40:0/2788658675 shutdown_connections 2024-03-18T22:40:33.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.637+0000 7f7803812640 1 --2- 172.21.15.40:0/2788658675 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77e80a2880 0x7f77e80a4d40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:33.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.637+0000 7f7803812640 1 --2- 172.21.15.40:0/2788658675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77fc105610 0x7f77fc1a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:33.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.637+0000 7f7803812640 1 --2- 172.21.15.40:0/2788658675 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77fc104c50 0x7f77fc19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:33.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.637+0000 7f7803812640 1 --2- 172.21.15.40:0/2788658675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77fc103a60 0x7f77fc199c70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:33.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.637+0000 7f7803812640 1 -- 172.21.15.40:0/2788658675 >> 172.21.15.40:0/2788658675 conn(0x7f77fc0fefd0 msgr2=0x7f77fc107fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:33.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.637+0000 7f7803812640 1 -- 172.21.15.40:0/2788658675 shutdown_connections 2024-03-18T22:40:33.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:33.637+0000 7f7803812640 1 -- 172.21.15.40:0/2788658675 wait complete. 2024-03-18T22:40:34.310 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:34 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2788658675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:34.310 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 22:40:34 smithi040 systemd[1]: Starting Ceph mds.a for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:40:34.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:34 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2788658675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:34.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:34 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2788658675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:34.644 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 22:40:34 smithi040 podman[101349]: 2024-03-18 22:40:34.361200987 +0000 UTC m=+0.019722401 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T22:40:34.644 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 22:40:34 smithi040 podman[101349]: 2024-03-18 22:40:34.479498272 +0000 UTC m=+0.138019715 container create 555cd76a8073ffd1b6aa943eb519d0e052ff16a9d20f306f982c4a1114f86fcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T22:40:35.144 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 22:40:34 smithi040 podman[101349]: 2024-03-18 22:40:34.840908775 +0000 UTC m=+0.499430182 container init 555cd76a8073ffd1b6aa943eb519d0e052ff16a9d20f306f982c4a1114f86fcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T22:40:35.144 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 22:40:34 smithi040 podman[101349]: 2024-03-18 22:40:34.843551361 +0000 UTC m=+0.502072759 container start 555cd76a8073ffd1b6aa943eb519d0e052ff16a9d20f306f982c4a1114f86fcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-a, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS) 2024-03-18T22:40:35.144 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 22:40:34 smithi040 ceph-mds[101369]: -- 172.21.15.40:0/3371710850 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cfeede7a40 con 0x55cfeedba800 2024-03-18T22:40:35.144 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 22:40:34 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-a[101365]: starting mds.a at 2024-03-18T22:40:35.144 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 22:40:34 smithi040 bash[101349]: 555cd76a8073ffd1b6aa943eb519d0e052ff16a9d20f306f982c4a1114f86fcb 2024-03-18T22:40:35.144 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 22:40:34 smithi040 systemd[1]: Started Ceph mds.a for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:40:35.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:35 smithi113 ceph-mon[66775]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:35.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:35 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:35.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:35 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:35.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:35 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:35.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:35 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:35.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:35 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:35.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:35 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:35.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:35 smithi103 ceph-mon[66377]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:35.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:35 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:35.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:35 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:35.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:35 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:35.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:35 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:35.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:35 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:35.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:35 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:35.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:35 smithi040 ceph-mon[62643]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:35.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:35 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:35.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:35 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:35.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:35 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:35.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:35 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T22:40:35.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:35 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T22:40:35.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:35 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:35.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:35.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.763+0000 7f2f948c0640 1 -- 172.21.15.40:0/3643281807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f901022f0 msgr2=0x7f2f90108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:35.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.763+0000 7f2f948c0640 1 --2- 172.21.15.40:0/3643281807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f901022f0 0x7f2f90108b80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2f80009ca0 tx=0x7f2f8002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:40:35.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.763+0000 7f2f948c0640 1 -- 172.21.15.40:0/3643281807 shutdown_connections 2024-03-18T22:40:35.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.763+0000 7f2f948c0640 1 --2- 172.21.15.40:0/3643281807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f901022f0 0x7f2f90108b80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:35.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.763+0000 7f2f948c0640 1 --2- 172.21.15.40:0/3643281807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f90101930 0x7f2f90101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:35.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.763+0000 7f2f948c0640 1 --2- 172.21.15.40:0/3643281807 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f90100740 0x7f2f90100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:35.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.763+0000 7f2f948c0640 1 -- 172.21.15.40:0/3643281807 >> 172.21.15.40:0/3643281807 conn(0x7f2f900fbcb0 msgr2=0x7f2f900fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:35.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.763+0000 7f2f948c0640 1 -- 172.21.15.40:0/3643281807 shutdown_connections 2024-03-18T22:40:35.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.763+0000 7f2f948c0640 1 -- 172.21.15.40:0/3643281807 wait complete. 2024-03-18T22:40:35.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.764+0000 7f2f948c0640 1 Processor -- start 2024-03-18T22:40:35.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.764+0000 7f2f948c0640 1 -- start start 2024-03-18T22:40:35.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.764+0000 7f2f948c0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f90100740 0x7f2f90197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:35.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.764+0000 7f2f948c0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f90101930 0x7f2f90198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:35.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.764+0000 7f2f948c0640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f901022f0 0x7f2f9019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:35.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.764+0000 7f2f948c0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9010c2f0 con 0x7f2f90101930 2024-03-18T22:40:35.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.764+0000 7f2f948c0640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9010c170 con 0x7f2f901022f0 2024-03-18T22:40:35.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.764+0000 7f2f948c0640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9010c470 con 0x7f2f90100740 2024-03-18T22:40:35.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.764+0000 7f2f8dd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f90101930 0x7f2f90198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:35.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.764+0000 7f2f8dd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f90101930 0x7f2f90198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38166/0 (socket says 172.21.15.40:38166) 2024-03-18T22:40:35.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.764+0000 7f2f8dd9b640 1 -- 172.21.15.40:0/2869683707 learned_addr learned my addr 172.21.15.40:0/2869683707 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:35.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.765+0000 7f2f8ed9d640 1 --2- 172.21.15.40:0/2869683707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f901022f0 0x7f2f9019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:35.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.765+0000 7f2f8e59c640 1 --2- 172.21.15.40:0/2869683707 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f90100740 0x7f2f90197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:35.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.765+0000 7f2f8dd9b640 1 -- 172.21.15.40:0/2869683707 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f90100740 msgr2=0x7f2f90197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:35.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.765+0000 7f2f8dd9b640 1 --2- 172.21.15.40:0/2869683707 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f90100740 0x7f2f90197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:35.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.765+0000 7f2f8dd9b640 1 -- 172.21.15.40:0/2869683707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f901022f0 msgr2=0x7f2f9019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:35.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.765+0000 7f2f8dd9b640 1 --2- 172.21.15.40:0/2869683707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f901022f0 0x7f2f9019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:35.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.765+0000 7f2f8dd9b640 1 -- 172.21.15.40:0/2869683707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9019f730 con 0x7f2f90101930 2024-03-18T22:40:35.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.765+0000 7f2f8dd9b640 1 --2- 172.21.15.40:0/2869683707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f90101930 0x7f2f90198050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2f7800b4a0 tx=0x7f2f7800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:35.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.765+0000 7f2f877fe640 1 -- 172.21.15.40:0/2869683707 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f78014070 con 0x7f2f90101930 2024-03-18T22:40:35.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.765+0000 7f2f948c0640 1 -- 172.21.15.40:0/2869683707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f9019fa20 con 0x7f2f90101930 2024-03-18T22:40:35.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.765+0000 7f2f877fe640 1 -- 172.21.15.40:0/2869683707 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f78010ac0 con 0x7f2f90101930 2024-03-18T22:40:35.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.765+0000 7f2f877fe640 1 -- 172.21.15.40:0/2869683707 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f78017c30 con 0x7f2f90101930 2024-03-18T22:40:35.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.765+0000 7f2f948c0640 1 -- 172.21.15.40:0/2869683707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f9019fff0 con 0x7f2f90101930 2024-03-18T22:40:35.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.766+0000 7f2f948c0640 1 -- 172.21.15.40:0/2869683707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f901a0280 con 0x7f2f90101930 2024-03-18T22:40:35.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.767+0000 7f2f877fe640 1 -- 172.21.15.40:0/2869683707 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2f78017070 con 0x7f2f90101930 2024-03-18T22:40:35.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.768+0000 7f2f877fe640 1 --2- 172.21.15.40:0/2869683707 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2f7c0a2800 0x7f2f7c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:35.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.768+0000 7f2f8e59c640 1 --2- 172.21.15.40:0/2869683707 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2f7c0a2800 0x7f2f7c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:35.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.768+0000 7f2f877fe640 1 -- 172.21.15.40:0/2869683707 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2f780cdca0 con 0x7f2f90101930 2024-03-18T22:40:35.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.769+0000 7f2f8e59c640 1 --2- 172.21.15.40:0/2869683707 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2f7c0a2800 0x7f2f7c0a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2f90101790 tx=0x7f2f88005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:35.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.773+0000 7f2f877fe640 1 -- 172.21.15.40:0/2869683707 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2f7809aaf0 con 0x7f2f90101930 2024-03-18T22:40:35.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.995+0000 7f2f948c0640 1 -- 172.21.15.40:0/2869683707 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f901a0450 con 0x7f2f90101930 2024-03-18T22:40:35.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.996+0000 7f2f877fe640 1 -- 172.21.15.40:0/2869683707 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9665 (secure 0 0 0) 0x7f2f780aee20 con 0x7f2f90101930 2024-03-18T22:40:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:35.997 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14604,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T22:40:35.997 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 12 2024-03-18T22:40:35.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.998+0000 7f2f948c0640 1 -- 172.21.15.40:0/2869683707 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2f7c0a2800 msgr2=0x7f2f7c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:35.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.998+0000 7f2f948c0640 1 --2- 172.21.15.40:0/2869683707 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2f7c0a2800 0x7f2f7c0a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2f90101790 tx=0x7f2f88005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:35.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.998+0000 7f2f948c0640 1 -- 172.21.15.40:0/2869683707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f90101930 msgr2=0x7f2f90198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:35.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.998+0000 7f2f948c0640 1 --2- 172.21.15.40:0/2869683707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f90101930 0x7f2f90198050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2f7800b4a0 tx=0x7f2f7800b960 comp rx=0 tx=0).stop 2024-03-18T22:40:35.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.998+0000 7f2f948c0640 1 -- 172.21.15.40:0/2869683707 shutdown_connections 2024-03-18T22:40:35.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.998+0000 7f2f948c0640 1 --2- 172.21.15.40:0/2869683707 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2f7c0a2800 0x7f2f7c0a4cc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:35.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.998+0000 7f2f948c0640 1 --2- 172.21.15.40:0/2869683707 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f901022f0 0x7f2f9019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:35.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.998+0000 7f2f948c0640 1 --2- 172.21.15.40:0/2869683707 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f90101930 0x7f2f90198050 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:35.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.998+0000 7f2f948c0640 1 --2- 172.21.15.40:0/2869683707 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2f90100740 0x7f2f90197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:35.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.998+0000 7f2f948c0640 1 -- 172.21.15.40:0/2869683707 >> 172.21.15.40:0/2869683707 conn(0x7f2f900fbcb0 msgr2=0x7f2f90068a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:35.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.998+0000 7f2f948c0640 1 -- 172.21.15.40:0/2869683707 shutdown_connections 2024-03-18T22:40:35.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:35.998+0000 7f2f948c0640 1 -- 172.21.15.40:0/2869683707 wait complete. 2024-03-18T22:40:36.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:36 smithi113 ceph-mon[66775]: Deploying daemon mds.b on smithi103 2024-03-18T22:40:36.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:36 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.40:6840/1807327423,v1:172.21.15.40:6841/1807327423] up:boot 2024-03-18T22:40:36.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:36 smithi113 ceph-mon[66775]: fsmap 11 up:standby 2024-03-18T22:40:36.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:36 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T22:40:36.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:36 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2869683707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:36.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:36 smithi040 ceph-mon[62643]: Deploying daemon mds.b on smithi103 2024-03-18T22:40:36.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:36 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.40:6840/1807327423,v1:172.21.15.40:6841/1807327423] up:boot 2024-03-18T22:40:36.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:36 smithi040 ceph-mon[62643]: fsmap 11 up:standby 2024-03-18T22:40:36.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:36 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T22:40:36.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:36 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2869683707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:36 smithi103 ceph-mon[66377]: Deploying daemon mds.b on smithi103 2024-03-18T22:40:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:36 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.40:6840/1807327423,v1:172.21.15.40:6841/1807327423] up:boot 2024-03-18T22:40:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:36 smithi103 ceph-mon[66377]: fsmap 11 up:standby 2024-03-18T22:40:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:36 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T22:40:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:36 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2869683707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:37.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:37 smithi103 ceph-mon[66377]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:37.359 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 22:40:37 smithi103 systemd[1]: Starting Ceph mds.b for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:40:37.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:37 smithi113 ceph-mon[66775]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:37.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:37 smithi040 ceph-mon[62643]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:37.688 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 22:40:37 smithi103 podman[95921]: 2024-03-18 22:40:37.335158582 +0000 UTC m=+0.017760030 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T22:40:37.688 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 22:40:37 smithi103 podman[95921]: 2024-03-18 22:40:37.443672749 +0000 UTC m=+0.126274258 container create 670bfc4263edee67d786551cbece1203495faad959ff7fe53e39f9cad9c2ff69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-18T22:40:37.688 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 22:40:37 smithi103 podman[95921]: 2024-03-18 22:40:37.6882477 +0000 UTC m=+0.370849151 container init 670bfc4263edee67d786551cbece1203495faad959ff7fe53e39f9cad9c2ff69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-18T22:40:38.010 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:38.109 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 22:40:37 smithi103 podman[95921]: 2024-03-18 22:40:37.691251549 +0000 UTC m=+0.373853001 container start 670bfc4263edee67d786551cbece1203495faad959ff7fe53e39f9cad9c2ff69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-18T22:40:38.109 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 22:40:37 smithi103 ceph-mds[95940]: -- 172.21.15.103:0/1520207435 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562edd33fa40 con 0x562edd312c00 2024-03-18T22:40:38.109 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 22:40:37 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-b[95936]: starting mds.b at 2024-03-18T22:40:38.109 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 22:40:37 smithi103 bash[95921]: 670bfc4263edee67d786551cbece1203495faad959ff7fe53e39f9cad9c2ff69 2024-03-18T22:40:38.109 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 22:40:37 smithi103 systemd[1]: Started Ceph mds.b for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:40:38.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.124+0000 7f9895a44640 1 -- 172.21.15.40:0/2697904998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98901022f0 msgr2=0x7f9890108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:38.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.124+0000 7f9895a44640 1 --2- 172.21.15.40:0/2697904998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98901022f0 0x7f9890108b80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9888009ca0 tx=0x7f988802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.124+0000 7f9895a44640 1 -- 172.21.15.40:0/2697904998 shutdown_connections 2024-03-18T22:40:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.124+0000 7f9895a44640 1 --2- 172.21.15.40:0/2697904998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98901022f0 0x7f9890108b80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.124+0000 7f9895a44640 1 --2- 172.21.15.40:0/2697904998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9890101930 0x7f9890101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.124+0000 7f9895a44640 1 --2- 172.21.15.40:0/2697904998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9890100740 0x7f9890100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.124+0000 7f9895a44640 1 -- 172.21.15.40:0/2697904998 >> 172.21.15.40:0/2697904998 conn(0x7f98900fbcd0 msgr2=0x7f98900fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.124+0000 7f9895a44640 1 -- 172.21.15.40:0/2697904998 shutdown_connections 2024-03-18T22:40:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.124+0000 7f9895a44640 1 -- 172.21.15.40:0/2697904998 wait complete. 2024-03-18T22:40:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.124+0000 7f9895a44640 1 Processor -- start 2024-03-18T22:40:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.124+0000 7f9895a44640 1 -- start start 2024-03-18T22:40:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f9895a44640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9890100740 0x7f9890197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f9895a44640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9890101930 0x7f9890198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f9895a44640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98901022f0 0x7f989019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f9895a44640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f989010c270 con 0x7f9890101930 2024-03-18T22:40:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f9895a44640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f989010c0f0 con 0x7f9890100740 2024-03-18T22:40:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f9895a44640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f989010c3f0 con 0x7f98901022f0 2024-03-18T22:40:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f9887fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9890101930 0x7f9890198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f9887fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9890101930 0x7f9890198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38176/0 (socket says 172.21.15.40:38176) 2024-03-18T22:40:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f9887fff640 1 -- 172.21.15.40:0/4147436548 learned_addr learned my addr 172.21.15.40:0/4147436548 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f988effd640 1 --2- 172.21.15.40:0/4147436548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9890100740 0x7f9890197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f988f7fe640 1 --2- 172.21.15.40:0/4147436548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98901022f0 0x7f989019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f9887fff640 1 -- 172.21.15.40:0/4147436548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98901022f0 msgr2=0x7f989019f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f9887fff640 1 --2- 172.21.15.40:0/4147436548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98901022f0 0x7f989019f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f9887fff640 1 -- 172.21.15.40:0/4147436548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9890100740 msgr2=0x7f9890197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f9887fff640 1 --2- 172.21.15.40:0/4147436548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9890100740 0x7f9890197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f9887fff640 1 -- 172.21.15.40:0/4147436548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f989019f670 con 0x7f9890101930 2024-03-18T22:40:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f988effd640 1 --2- 172.21.15.40:0/4147436548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9890100740 0x7f9890197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:40:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.125+0000 7f9887fff640 1 --2- 172.21.15.40:0/4147436548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9890101930 0x7f9890198000 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f987800e9f0 tx=0x7f987800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.126+0000 7f9894a42640 1 -- 172.21.15.40:0/4147436548 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987800cd80 con 0x7f9890101930 2024-03-18T22:40:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.126+0000 7f9894a42640 1 -- 172.21.15.40:0/4147436548 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98780054d0 con 0x7f9890101930 2024-03-18T22:40:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.126+0000 7f9894a42640 1 -- 172.21.15.40:0/4147436548 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9878010490 con 0x7f9890101930 2024-03-18T22:40:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.126+0000 7f9895a44640 1 -- 172.21.15.40:0/4147436548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f989019f960 con 0x7f9890101930 2024-03-18T22:40:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.126+0000 7f9895a44640 1 -- 172.21.15.40:0/4147436548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f989019fe60 con 0x7f9890101930 2024-03-18T22:40:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.127+0000 7f9895a44640 1 -- 172.21.15.40:0/4147436548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98901a0090 con 0x7f9890101930 2024-03-18T22:40:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.128+0000 7f9894a42640 1 -- 172.21.15.40:0/4147436548 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9878005020 con 0x7f9890101930 2024-03-18T22:40:38.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.129+0000 7f9894a42640 1 --2- 172.21.15.40:0/4147436548 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f987c0a2800 0x7f987c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:38.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.129+0000 7f988effd640 1 --2- 172.21.15.40:0/4147436548 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f987c0a2800 0x7f987c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:38.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.129+0000 7f9894a42640 1 -- 172.21.15.40:0/4147436548 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f98780ce960 con 0x7f9890101930 2024-03-18T22:40:38.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.130+0000 7f988effd640 1 --2- 172.21.15.40:0/4147436548 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f987c0a2800 0x7f987c0a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9890101790 tx=0x7f9880005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:38.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.134+0000 7f9894a42640 1 -- 172.21.15.40:0/4147436548 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f987809bea0 con 0x7f9890101930 2024-03-18T22:40:38.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.358+0000 7f9895a44640 1 -- 172.21.15.40:0/4147436548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98901a0300 con 0x7f9890101930 2024-03-18T22:40:38.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.358+0000 7f9894a42640 1 -- 172.21.15.40:0/4147436548 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10494 (secure 0 0 0) 0x7f987809b7a0 con 0x7f9890101930 2024-03-18T22:40:38.359 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:38.359 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14604,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24509,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-18T22:40:38.359 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 13 2024-03-18T22:40:38.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.360+0000 7f9895a44640 1 -- 172.21.15.40:0/4147436548 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f987c0a2800 msgr2=0x7f987c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:38.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.360+0000 7f9895a44640 1 --2- 172.21.15.40:0/4147436548 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f987c0a2800 0x7f987c0a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9890101790 tx=0x7f9880005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:38.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.360+0000 7f9895a44640 1 -- 172.21.15.40:0/4147436548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9890101930 msgr2=0x7f9890198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:38.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.360+0000 7f9895a44640 1 --2- 172.21.15.40:0/4147436548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9890101930 0x7f9890198000 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f987800e9f0 tx=0x7f987800edf0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.360+0000 7f9895a44640 1 -- 172.21.15.40:0/4147436548 shutdown_connections 2024-03-18T22:40:38.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.360+0000 7f9895a44640 1 --2- 172.21.15.40:0/4147436548 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f987c0a2800 0x7f987c0a4cc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.360+0000 7f9895a44640 1 --2- 172.21.15.40:0/4147436548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f98901022f0 0x7f989019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.360+0000 7f9895a44640 1 --2- 172.21.15.40:0/4147436548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9890101930 0x7f9890198000 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.360+0000 7f9895a44640 1 --2- 172.21.15.40:0/4147436548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9890100740 0x7f9890197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.360+0000 7f9895a44640 1 -- 172.21.15.40:0/4147436548 >> 172.21.15.40:0/4147436548 conn(0x7f98900fbcd0 msgr2=0x7f98900fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:38.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.361+0000 7f9895a44640 1 -- 172.21.15.40:0/4147436548 shutdown_connections 2024-03-18T22:40:38.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.361+0000 7f9895a44640 1 -- 172.21.15.40:0/4147436548 wait complete. 2024-03-18T22:40:38.373 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:40:38.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.476+0000 7f0cf25bc640 1 -- 172.21.15.40:0/2202667449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cec103420 msgr2=0x7f0cec109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:38.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.476+0000 7f0cf25bc640 1 --2- 172.21.15.40:0/2202667449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cec103420 0x7f0cec109cb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009ca0 tx=0x7f0ce402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.476+0000 7f0cf25bc640 1 -- 172.21.15.40:0/2202667449 shutdown_connections 2024-03-18T22:40:38.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.476+0000 7f0cf25bc640 1 --2- 172.21.15.40:0/2202667449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cec103420 0x7f0cec109cb0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.476+0000 7f0cf25bc640 1 --2- 172.21.15.40:0/2202667449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cec102a60 0x7f0cec102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.476+0000 7f0cf25bc640 1 --2- 172.21.15.40:0/2202667449 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cec101870 0x7f0cec101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.476+0000 7f0cf25bc640 1 -- 172.21.15.40:0/2202667449 >> 172.21.15.40:0/2202667449 conn(0x7f0cec0fcdc0 msgr2=0x7f0cec0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:38.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.476+0000 7f0cf25bc640 1 -- 172.21.15.40:0/2202667449 shutdown_connections 2024-03-18T22:40:38.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.476+0000 7f0cf25bc640 1 -- 172.21.15.40:0/2202667449 wait complete. 2024-03-18T22:40:38.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.477+0000 7f0cf25bc640 1 Processor -- start 2024-03-18T22:40:38.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.477+0000 7f0cf25bc640 1 -- start start 2024-03-18T22:40:38.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.477+0000 7f0cf25bc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cec101870 0x7f0cec197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:38.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.477+0000 7f0cf25bc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cec102a60 0x7f0cec197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:38.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.477+0000 7f0cf25bc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cec103420 0x7f0cec19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:38.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.477+0000 7f0cf25bc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec10d380 con 0x7f0cec103420 2024-03-18T22:40:38.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.477+0000 7f0cf25bc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec10d200 con 0x7f0cec101870 2024-03-18T22:40:38.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.477+0000 7f0cf25bc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec10d500 con 0x7f0cec102a60 2024-03-18T22:40:38.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.477+0000 7f0cf0b59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cec103420 0x7f0cec19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:38.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.477+0000 7f0cebfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cec101870 0x7f0cec197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:38.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.477+0000 7f0cf0b59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cec103420 0x7f0cec19efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38180/0 (socket says 172.21.15.40:38180) 2024-03-18T22:40:38.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.477+0000 7f0cebfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cec101870 0x7f0cec197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56890/0 (socket says 172.21.15.40:56890) 2024-03-18T22:40:38.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.477+0000 7f0cf0b59640 1 -- 172.21.15.40:0/1957823073 learned_addr learned my addr 172.21.15.40:0/1957823073 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:38.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.477+0000 7f0ceb7fe640 1 --2- 172.21.15.40:0/1957823073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cec102a60 0x7f0cec197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:38.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.478+0000 7f0cf0b59640 1 -- 172.21.15.40:0/1957823073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cec102a60 msgr2=0x7f0cec197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:38.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.478+0000 7f0cf0b59640 1 --2- 172.21.15.40:0/1957823073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cec102a60 0x7f0cec197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.478+0000 7f0cf0b59640 1 -- 172.21.15.40:0/1957823073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cec101870 msgr2=0x7f0cec197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:38.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.478+0000 7f0cf0b59640 1 --2- 172.21.15.40:0/1957823073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cec101870 0x7f0cec197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.478+0000 7f0cf0b59640 1 -- 172.21.15.40:0/1957823073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cec19f650 con 0x7f0cec103420 2024-03-18T22:40:38.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.478+0000 7f0cebfff640 1 --2- 172.21.15.40:0/1957823073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cec101870 0x7f0cec197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:40:38.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.478+0000 7f0ceb7fe640 1 --2- 172.21.15.40:0/1957823073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cec102a60 0x7f0cec197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:40:38.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.478+0000 7f0cf0b59640 1 --2- 172.21.15.40:0/1957823073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cec103420 0x7f0cec19efe0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0ce402c700 tx=0x7f0ce40328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:38.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.478+0000 7f0ce97fa640 1 -- 172.21.15.40:0/1957823073 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4041020 con 0x7f0cec103420 2024-03-18T22:40:38.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.478+0000 7f0cf25bc640 1 -- 172.21.15.40:0/1957823073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cec19f8e0 con 0x7f0cec103420 2024-03-18T22:40:38.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.478+0000 7f0ce97fa640 1 -- 172.21.15.40:0/1957823073 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ce4003e20 con 0x7f0cec103420 2024-03-18T22:40:38.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.478+0000 7f0ce97fa640 1 -- 172.21.15.40:0/1957823073 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4039040 con 0x7f0cec103420 2024-03-18T22:40:38.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.478+0000 7f0cf25bc640 1 -- 172.21.15.40:0/1957823073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cec19fdd0 con 0x7f0cec103420 2024-03-18T22:40:38.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.479+0000 7f0cf25bc640 1 -- 172.21.15.40:0/1957823073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cec108b90 con 0x7f0cec103420 2024-03-18T22:40:38.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.480+0000 7f0ce97fa640 1 -- 172.21.15.40:0/1957823073 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0ce4044020 con 0x7f0cec103420 2024-03-18T22:40:38.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.481+0000 7f0ce97fa640 1 --2- 172.21.15.40:0/1957823073 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0cd80a2800 0x7f0cd80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:38.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.481+0000 7f0ce97fa640 1 -- 172.21.15.40:0/1957823073 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0ce40f0b50 con 0x7f0cec103420 2024-03-18T22:40:38.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.481+0000 7f0cebfff640 1 --2- 172.21.15.40:0/1957823073 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0cd80a2800 0x7f0cd80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:38.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.481+0000 7f0cebfff640 1 --2- 172.21.15.40:0/1957823073 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0cd80a2800 0x7f0cd80a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0cec1028c0 tx=0x7f0cd4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:38.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.489+0000 7f0ce97fa640 1 -- 172.21.15.40:0/1957823073 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0ce404b020 con 0x7f0cec103420 2024-03-18T22:40:38.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.707+0000 7f0cf25bc640 1 -- 172.21.15.40:0/1957823073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0cec02d3c0 con 0x7f0cec103420 2024-03-18T22:40:38.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.708+0000 7f0ce97fa640 1 -- 172.21.15.40:0/1957823073 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18457 (secure 0 0 0) 0x7f0ce40be090 con 0x7f0cec103420 2024-03-18T22:40:38.709 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:38.709 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":81,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:39:54.932203+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:40:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.710+0000 7f0cf25bc640 1 -- 172.21.15.40:0/1957823073 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0cd80a2800 msgr2=0x7f0cd80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.710+0000 7f0cf25bc640 1 --2- 172.21.15.40:0/1957823073 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0cd80a2800 0x7f0cd80a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0cec1028c0 tx=0x7f0cd4005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.710+0000 7f0cf25bc640 1 -- 172.21.15.40:0/1957823073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cec103420 msgr2=0x7f0cec19efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.710+0000 7f0cf25bc640 1 --2- 172.21.15.40:0/1957823073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cec103420 0x7f0cec19efe0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0ce402c700 tx=0x7f0ce40328a0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.710+0000 7f0cf25bc640 1 -- 172.21.15.40:0/1957823073 shutdown_connections 2024-03-18T22:40:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.710+0000 7f0cf25bc640 1 --2- 172.21.15.40:0/1957823073 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0cd80a2800 0x7f0cd80a4cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.710+0000 7f0cf25bc640 1 --2- 172.21.15.40:0/1957823073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0cec103420 0x7f0cec19efe0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.710+0000 7f0cf25bc640 1 --2- 172.21.15.40:0/1957823073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0cec102a60 0x7f0cec197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.710+0000 7f0cf25bc640 1 --2- 172.21.15.40:0/1957823073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0cec101870 0x7f0cec197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.710+0000 7f0cf25bc640 1 -- 172.21.15.40:0/1957823073 >> 172.21.15.40:0/1957823073 conn(0x7f0cec0fcdc0 msgr2=0x7f0cec105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.711+0000 7f0cf25bc640 1 -- 172.21.15.40:0/1957823073 shutdown_connections 2024-03-18T22:40:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.711+0000 7f0cf25bc640 1 -- 172.21.15.40:0/1957823073 wait complete. 2024-03-18T22:40:38.723 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T22:36:07.060691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T22:40:38.724 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T22:40:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.829+0000 7fdb33162640 1 -- 172.21.15.40:0/345152253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c101910 msgr2=0x7fdb2c101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.829+0000 7fdb33162640 1 --2- 172.21.15.40:0/345152253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c101910 0x7fdb2c101d90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009900 tx=0x7fdb1c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.829+0000 7fdb33162640 1 -- 172.21.15.40:0/345152253 shutdown_connections 2024-03-18T22:40:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.829+0000 7fdb33162640 1 --2- 172.21.15.40:0/345152253 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c1022d0 0x7fdb2c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.829+0000 7fdb33162640 1 --2- 172.21.15.40:0/345152253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c101910 0x7fdb2c101d90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.829+0000 7fdb33162640 1 --2- 172.21.15.40:0/345152253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb2c100720 0x7fdb2c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.829+0000 7fdb33162640 1 -- 172.21.15.40:0/345152253 >> 172.21.15.40:0/345152253 conn(0x7fdb2c0fbcb0 msgr2=0x7fdb2c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.829+0000 7fdb33162640 1 -- 172.21.15.40:0/345152253 shutdown_connections 2024-03-18T22:40:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.829+0000 7fdb33162640 1 -- 172.21.15.40:0/345152253 wait complete. 2024-03-18T22:40:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb33162640 1 Processor -- start 2024-03-18T22:40:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb33162640 1 -- start start 2024-03-18T22:40:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb33162640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb2c100720 0x7fdb2c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb33162640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c101910 0x7fdb2c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb33162640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c1022d0 0x7fdb2c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb33162640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c10c250 con 0x7fdb2c1022d0 2024-03-18T22:40:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb33162640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c10c0d0 con 0x7fdb2c100720 2024-03-18T22:40:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb33162640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c10c3d0 con 0x7fdb2c101910 2024-03-18T22:40:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb316ff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c1022d0 0x7fdb2c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb316ff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c1022d0 0x7fdb2c19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50984/0 (socket says 172.21.15.40:50984) 2024-03-18T22:40:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb316ff640 1 -- 172.21.15.40:0/3867134208 learned_addr learned my addr 172.21.15.40:0/3867134208 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb30efe640 1 --2- 172.21.15.40:0/3867134208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb2c100720 0x7fdb2c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb23fff640 1 --2- 172.21.15.40:0/3867134208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c101910 0x7fdb2c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb316ff640 1 -- 172.21.15.40:0/3867134208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c101910 msgr2=0x7fdb2c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb316ff640 1 --2- 172.21.15.40:0/3867134208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c101910 0x7fdb2c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb316ff640 1 -- 172.21.15.40:0/3867134208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb2c100720 msgr2=0x7fdb2c197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb316ff640 1 --2- 172.21.15.40:0/3867134208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb2c100720 0x7fdb2c197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.830+0000 7fdb316ff640 1 -- 172.21.15.40:0/3867134208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb2c19f6b0 con 0x7fdb2c1022d0 2024-03-18T22:40:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.831+0000 7fdb316ff640 1 --2- 172.21.15.40:0/3867134208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c1022d0 0x7fdb2c19f040 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdb2400eaa0 tx=0x7fdb2400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.831+0000 7fdb21ffb640 1 -- 172.21.15.40:0/3867134208 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2400cdb0 con 0x7fdb2c1022d0 2024-03-18T22:40:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.831+0000 7fdb21ffb640 1 -- 172.21.15.40:0/3867134208 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb24005020 con 0x7fdb2c1022d0 2024-03-18T22:40:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.831+0000 7fdb33162640 1 -- 172.21.15.40:0/3867134208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb2c19f9a0 con 0x7fdb2c1022d0 2024-03-18T22:40:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.831+0000 7fdb21ffb640 1 -- 172.21.15.40:0/3867134208 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb240116d0 con 0x7fdb2c1022d0 2024-03-18T22:40:38.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.831+0000 7fdb33162640 1 -- 172.21.15.40:0/3867134208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb2c19fef0 con 0x7fdb2c1022d0 2024-03-18T22:40:38.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.832+0000 7fdb33162640 1 -- 172.21.15.40:0/3867134208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb2c1a0120 con 0x7fdb2c1022d0 2024-03-18T22:40:38.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.833+0000 7fdb21ffb640 1 -- 172.21.15.40:0/3867134208 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fdb240051d0 con 0x7fdb2c1022d0 2024-03-18T22:40:38.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.834+0000 7fdb21ffb640 1 --2- 172.21.15.40:0/3867134208 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb140a2800 0x7fdb140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:38.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.834+0000 7fdb30efe640 1 --2- 172.21.15.40:0/3867134208 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb140a2800 0x7fdb140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:38.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.834+0000 7fdb21ffb640 1 -- 172.21.15.40:0/3867134208 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fdb240ce810 con 0x7fdb2c1022d0 2024-03-18T22:40:38.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.834+0000 7fdb30efe640 1 --2- 172.21.15.40:0/3867134208 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb140a2800 0x7fdb140a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c101770 tx=0x7fdb2800b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:38.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:38.838+0000 7fdb21ffb640 1 -- 172.21.15.40:0/3867134208 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdb24016030 con 0x7fdb2c1022d0 2024-03-18T22:40:39.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.022+0000 7fdb33162640 1 -- 172.21.15.40:0/3867134208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdb2c1a02f0 con 0x7fdb2c1022d0 2024-03-18T22:40:39.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.023+0000 7fdb21ffb640 1 -- 172.21.15.40:0/3867134208 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fdb2409bcb0 con 0x7fdb2c1022d0 2024-03-18T22:40:39.024 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-18T22:40:39.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.025+0000 7fdb33162640 1 -- 172.21.15.40:0/3867134208 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb140a2800 msgr2=0x7fdb140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:39.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.025+0000 7fdb33162640 1 --2- 172.21.15.40:0/3867134208 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb140a2800 0x7fdb140a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c101770 tx=0x7fdb2800b390 comp rx=0 tx=0).stop 2024-03-18T22:40:39.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.025+0000 7fdb33162640 1 -- 172.21.15.40:0/3867134208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c1022d0 msgr2=0x7fdb2c19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:39.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.025+0000 7fdb33162640 1 --2- 172.21.15.40:0/3867134208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c1022d0 0x7fdb2c19f040 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdb2400eaa0 tx=0x7fdb2400ef60 comp rx=0 tx=0).stop 2024-03-18T22:40:39.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.025+0000 7fdb33162640 1 -- 172.21.15.40:0/3867134208 shutdown_connections 2024-03-18T22:40:39.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.025+0000 7fdb33162640 1 --2- 172.21.15.40:0/3867134208 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb140a2800 0x7fdb140a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.025+0000 7fdb33162640 1 --2- 172.21.15.40:0/3867134208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c1022d0 0x7fdb2c19f040 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.025+0000 7fdb33162640 1 --2- 172.21.15.40:0/3867134208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c101910 0x7fdb2c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.025+0000 7fdb33162640 1 --2- 172.21.15.40:0/3867134208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb2c100720 0x7fdb2c197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.025+0000 7fdb33162640 1 -- 172.21.15.40:0/3867134208 >> 172.21.15.40:0/3867134208 conn(0x7fdb2c0fbcb0 msgr2=0x7fdb2c068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:39.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.025+0000 7fdb33162640 1 -- 172.21.15.40:0/3867134208 shutdown_connections 2024-03-18T22:40:39.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.025+0000 7fdb33162640 1 -- 172.21.15.40:0/3867134208 wait complete. 2024-03-18T22:40:39.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:39.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:39.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:39.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:39.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:39.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:40:39.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:38 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] up:boot 2024-03-18T22:40:39.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:38 smithi040 ceph-mon[62643]: fsmap 12 up:standby 2024-03-18T22:40:39.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T22:40:39.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:38 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/4147436548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:39.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:38 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1957823073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:39.037 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:38 smithi040 ceph-mon[62643]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:39.039 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T22:40:39.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:39.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:39.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:39.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:39.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:39.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:40:39.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:38 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] up:boot 2024-03-18T22:40:39.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:38 smithi103 ceph-mon[66377]: fsmap 12 up:standby 2024-03-18T22:40:39.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T22:40:39.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:38 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/4147436548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:39.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:38 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1957823073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:39.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:38 smithi103 ceph-mon[66377]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:39.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.143+0000 7fd154f7e640 1 -- 172.21.15.40:0/590732918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd150102e50 msgr2=0x7fd150103270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:39.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.143+0000 7fd154f7e640 1 --2- 172.21.15.40:0/590732918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd150102e50 0x7fd150103270 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd140009ca0 tx=0x7fd14002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.143+0000 7fd154f7e640 1 -- 172.21.15.40:0/590732918 shutdown_connections 2024-03-18T22:40:39.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.143+0000 7fd154f7e640 1 --2- 172.21.15.40:0/590732918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd150100280 0x7fd150109e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.143+0000 7fd154f7e640 1 --2- 172.21.15.40:0/590732918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1500ff8c0 0x7fd1500ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.143+0000 7fd154f7e640 1 --2- 172.21.15.40:0/590732918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd150102e50 0x7fd150103270 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.143+0000 7fd154f7e640 1 -- 172.21.15.40:0/590732918 >> 172.21.15.40:0/590732918 conn(0x7fd1500fb460 msgr2=0x7fd1500fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:39.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.143+0000 7fd154f7e640 1 -- 172.21.15.40:0/590732918 shutdown_connections 2024-03-18T22:40:39.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.143+0000 7fd154f7e640 1 -- 172.21.15.40:0/590732918 wait complete. 2024-03-18T22:40:39.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.144+0000 7fd154f7e640 1 Processor -- start 2024-03-18T22:40:39.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.144+0000 7fd154f7e640 1 -- start start 2024-03-18T22:40:39.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.144+0000 7fd154f7e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1500ff8c0 0x7fd150197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:39.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.144+0000 7fd154f7e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd150100280 0x7fd150198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:39.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.144+0000 7fd154f7e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd150102e50 0x7fd15019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:39.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.144+0000 7fd154f7e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd15010d3a0 con 0x7fd150102e50 2024-03-18T22:40:39.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.144+0000 7fd154f7e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd15010d220 con 0x7fd1500ff8c0 2024-03-18T22:40:39.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.144+0000 7fd154f7e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd15010d520 con 0x7fd150100280 2024-03-18T22:40:39.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.144+0000 7fd14ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd150102e50 0x7fd15019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:39.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.144+0000 7fd14ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd150102e50 0x7fd15019f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51002/0 (socket says 172.21.15.40:51002) 2024-03-18T22:40:39.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.144+0000 7fd14ed9d640 1 -- 172.21.15.40:0/650623574 learned_addr learned my addr 172.21.15.40:0/650623574 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:39.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.144+0000 7fd14dd9b640 1 --2- 172.21.15.40:0/650623574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd150100280 0x7fd150198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:39.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.145+0000 7fd14e59c640 1 --2- 172.21.15.40:0/650623574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1500ff8c0 0x7fd150197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:39.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.145+0000 7fd14ed9d640 1 -- 172.21.15.40:0/650623574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd150100280 msgr2=0x7fd150198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:39.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.145+0000 7fd14ed9d640 1 --2- 172.21.15.40:0/650623574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd150100280 0x7fd150198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.145+0000 7fd14ed9d640 1 -- 172.21.15.40:0/650623574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1500ff8c0 msgr2=0x7fd150197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:39.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.145+0000 7fd14ed9d640 1 --2- 172.21.15.40:0/650623574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1500ff8c0 0x7fd150197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.145+0000 7fd14ed9d640 1 -- 172.21.15.40:0/650623574 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd15019f6f0 con 0x7fd150102e50 2024-03-18T22:40:39.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.145+0000 7fd14ed9d640 1 --2- 172.21.15.40:0/650623574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd150102e50 0x7fd15019f080 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd14800e970 tx=0x7fd14800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:39.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.145+0000 7fd13f7fe640 1 -- 172.21.15.40:0/650623574 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14800ccc0 con 0x7fd150102e50 2024-03-18T22:40:39.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.145+0000 7fd154f7e640 1 -- 172.21.15.40:0/650623574 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd15019f9e0 con 0x7fd150102e50 2024-03-18T22:40:39.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.145+0000 7fd13f7fe640 1 -- 172.21.15.40:0/650623574 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd14800ce60 con 0x7fd150102e50 2024-03-18T22:40:39.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.145+0000 7fd13f7fe640 1 -- 172.21.15.40:0/650623574 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148010430 con 0x7fd150102e50 2024-03-18T22:40:39.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.145+0000 7fd154f7e640 1 -- 172.21.15.40:0/650623574 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd15019ff80 con 0x7fd150102e50 2024-03-18T22:40:39.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.146+0000 7fd154f7e640 1 -- 172.21.15.40:0/650623574 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd150108d90 con 0x7fd150102e50 2024-03-18T22:40:39.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.147+0000 7fd13f7fe640 1 -- 172.21.15.40:0/650623574 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd148004f90 con 0x7fd150102e50 2024-03-18T22:40:39.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.148+0000 7fd13f7fe640 1 --2- 172.21.15.40:0/650623574 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1380a2800 0x7fd1380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:39.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.148+0000 7fd14e59c640 1 --2- 172.21.15.40:0/650623574 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1380a2800 0x7fd1380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:39.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.148+0000 7fd13f7fe640 1 -- 172.21.15.40:0/650623574 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd1480ce960 con 0x7fd150102e50 2024-03-18T22:40:39.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.148+0000 7fd14e59c640 1 --2- 172.21.15.40:0/650623574 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1380a2800 0x7fd1380a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd14002c630 tx=0x7fd140032340 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:39.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.152+0000 7fd13f7fe640 1 -- 172.21.15.40:0/650623574 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd14809bea0 con 0x7fd150102e50 2024-03-18T22:40:39.155 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:39.156 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:39.156 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:39.156 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:39.156 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:39.156 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:40:39.156 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:38 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] up:boot 2024-03-18T22:40:39.156 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:38 smithi113 ceph-mon[66775]: fsmap 12 up:standby 2024-03-18T22:40:39.156 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T22:40:39.156 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:38 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/4147436548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:39.156 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:38 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1957823073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:39.156 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:38 smithi113 ceph-mon[66775]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:39.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.340+0000 7fd154f7e640 1 -- 172.21.15.40:0/650623574 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd15002d3a0 con 0x7fd150102e50 2024-03-18T22:40:39.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.340+0000 7fd13f7fe640 1 -- 172.21.15.40:0/650623574 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fd14809b7a0 con 0x7fd150102e50 2024-03-18T22:40:39.341 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:39.341 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-03-18T22:40:39.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:40:39.342 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-03-18T22:40:39.342 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-03-18T22:40:39.342 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:40:39.342 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-03-18T22:40:39.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.342+0000 7fd154f7e640 1 -- 172.21.15.40:0/650623574 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1380a2800 msgr2=0x7fd1380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:39.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.342+0000 7fd154f7e640 1 --2- 172.21.15.40:0/650623574 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1380a2800 0x7fd1380a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd14002c630 tx=0x7fd140032340 comp rx=0 tx=0).stop 2024-03-18T22:40:39.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.342+0000 7fd154f7e640 1 -- 172.21.15.40:0/650623574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd150102e50 msgr2=0x7fd15019f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:39.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.342+0000 7fd154f7e640 1 --2- 172.21.15.40:0/650623574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd150102e50 0x7fd15019f080 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd14800e970 tx=0x7fd14800ed70 comp rx=0 tx=0).stop 2024-03-18T22:40:39.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.342+0000 7fd154f7e640 1 -- 172.21.15.40:0/650623574 shutdown_connections 2024-03-18T22:40:39.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.342+0000 7fd154f7e640 1 --2- 172.21.15.40:0/650623574 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1380a2800 0x7fd1380a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.342+0000 7fd154f7e640 1 --2- 172.21.15.40:0/650623574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd150102e50 0x7fd15019f080 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.342+0000 7fd154f7e640 1 --2- 172.21.15.40:0/650623574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd150100280 0x7fd150198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.342+0000 7fd154f7e640 1 --2- 172.21.15.40:0/650623574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1500ff8c0 0x7fd150197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.342+0000 7fd154f7e640 1 -- 172.21.15.40:0/650623574 >> 172.21.15.40:0/650623574 conn(0x7fd1500fb460 msgr2=0x7fd150105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:39.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.342+0000 7fd154f7e640 1 -- 172.21.15.40:0/650623574 shutdown_connections 2024-03-18T22:40:39.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.342+0000 7fd154f7e640 1 -- 172.21.15.40:0/650623574 wait complete. 2024-03-18T22:40:39.354 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T22:40:39.354 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T22:40:39.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.471+0000 7f41a4cdc640 1 -- 172.21.15.40:0/3102169478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a0103470 msgr2=0x7f41a0109d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:39.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.471+0000 7f41a4cdc640 1 --2- 172.21.15.40:0/3102169478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a0103470 0x7f41a0109d00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4198009930 tx=0x7f419802bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:39.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.471+0000 7f41a4cdc640 1 -- 172.21.15.40:0/3102169478 shutdown_connections 2024-03-18T22:40:39.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.471+0000 7f41a4cdc640 1 --2- 172.21.15.40:0/3102169478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a0103470 0x7f41a0109d00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.471+0000 7f41a4cdc640 1 --2- 172.21.15.40:0/3102169478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41a0102ab0 0x7f41a0102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.471+0000 7f41a4cdc640 1 --2- 172.21.15.40:0/3102169478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41a00ffae0 0x7f41a00fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.471+0000 7f41a4cdc640 1 -- 172.21.15.40:0/3102169478 >> 172.21.15.40:0/3102169478 conn(0x7f41a00fb410 msgr2=0x7f41a00fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:39.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.471+0000 7f41a4cdc640 1 -- 172.21.15.40:0/3102169478 shutdown_connections 2024-03-18T22:40:39.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.471+0000 7f41a4cdc640 1 -- 172.21.15.40:0/3102169478 wait complete. 2024-03-18T22:40:39.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.472+0000 7f41a4cdc640 1 Processor -- start 2024-03-18T22:40:39.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.472+0000 7f41a4cdc640 1 -- start start 2024-03-18T22:40:39.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.472+0000 7f41a4cdc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00ffae0 0x7f41a0102420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:39.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.472+0000 7f41a4cdc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41a0102ab0 0x7f41a0100a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:39.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.472+0000 7f41a4cdc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41a0103470 0x7f41a0100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:39.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.472+0000 7f41a4cdc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f41a010d360 con 0x7f41a00ffae0 2024-03-18T22:40:39.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.472+0000 7f41a4cdc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f41a010d1e0 con 0x7f41a0103470 2024-03-18T22:40:39.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.472+0000 7f41a4cdc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f41a010d4e0 con 0x7f41a0102ab0 2024-03-18T22:40:39.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.472+0000 7f419e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00ffae0 0x7f41a0102420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:39.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.472+0000 7f419e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00ffae0 0x7f41a0102420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51016/0 (socket says 172.21.15.40:51016) 2024-03-18T22:40:39.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.472+0000 7f419ed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41a0103470 0x7f41a0100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:39.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.472+0000 7f419e59c640 1 -- 172.21.15.40:0/894593156 learned_addr learned my addr 172.21.15.40:0/894593156 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:39.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.472+0000 7f419dd9b640 1 --2- 172.21.15.40:0/894593156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41a0102ab0 0x7f41a0100a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:39.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.473+0000 7f419e59c640 1 -- 172.21.15.40:0/894593156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41a0102ab0 msgr2=0x7f41a0100a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:39.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.473+0000 7f419e59c640 1 --2- 172.21.15.40:0/894593156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41a0102ab0 0x7f41a0100a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.473+0000 7f419e59c640 1 -- 172.21.15.40:0/894593156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41a0103470 msgr2=0x7f41a0100fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:39.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.473+0000 7f419e59c640 1 --2- 172.21.15.40:0/894593156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41a0103470 0x7f41a0100fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:39.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.473+0000 7f419e59c640 1 -- 172.21.15.40:0/894593156 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a0101830 con 0x7f41a00ffae0 2024-03-18T22:40:39.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.473+0000 7f419ed9d640 1 --2- 172.21.15.40:0/894593156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41a0103470 0x7f41a0100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:40:39.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.473+0000 7f419e59c640 1 --2- 172.21.15.40:0/894593156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00ffae0 0x7f41a0102420 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f419000eb70 tx=0x7f419000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:39.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.473+0000 7f418f7fe640 1 -- 172.21.15.40:0/894593156 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190003b70 con 0x7f41a00ffae0 2024-03-18T22:40:39.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.473+0000 7f41a4cdc640 1 -- 172.21.15.40:0/894593156 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41a01a1b50 con 0x7f41a00ffae0 2024-03-18T22:40:39.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.473+0000 7f418f7fe640 1 -- 172.21.15.40:0/894593156 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f419000ee70 con 0x7f41a00ffae0 2024-03-18T22:40:39.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.473+0000 7f418f7fe640 1 -- 172.21.15.40:0/894593156 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190010650 con 0x7f41a00ffae0 2024-03-18T22:40:39.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.473+0000 7f41a4cdc640 1 -- 172.21.15.40:0/894593156 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41a01a2050 con 0x7f41a00ffae0 2024-03-18T22:40:39.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.474+0000 7f41a4cdc640 1 -- 172.21.15.40:0/894593156 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41a0108cf0 con 0x7f41a00ffae0 2024-03-18T22:40:39.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.475+0000 7f418f7fe640 1 -- 172.21.15.40:0/894593156 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4190004020 con 0x7f41a00ffae0 2024-03-18T22:40:39.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.476+0000 7f418f7fe640 1 --2- 172.21.15.40:0/894593156 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f41880a2800 0x7f41880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:39.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.476+0000 7f419dd9b640 1 --2- 172.21.15.40:0/894593156 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f41880a2800 0x7f41880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:39.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.476+0000 7f418f7fe640 1 -- 172.21.15.40:0/894593156 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f41900ceca0 con 0x7f41a00ffae0 2024-03-18T22:40:39.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.476+0000 7f419dd9b640 1 --2- 172.21.15.40:0/894593156 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f41880a2800 0x7f41880a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f41a0102190 tx=0x7f4194005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:39.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.481+0000 7f418f7fe640 1 -- 172.21.15.40:0/894593156 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f419009c1e0 con 0x7f41a00ffae0 2024-03-18T22:40:39.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:39.673+0000 7f41a4cdc640 1 -- 172.21.15.40:0/894593156 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f41a002d3c0 con 0x7f41a00ffae0 2024-03-18T22:40:40.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.191+0000 7f418f7fe640 1 -- 172.21.15.40:0/894593156 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f419009bae0 con 0x7f41a00ffae0 2024-03-18T22:40:40.192 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-03-18T22:40:40.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.194+0000 7f41a4cdc640 1 -- 172.21.15.40:0/894593156 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f41880a2800 msgr2=0x7f41880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:40.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.194+0000 7f41a4cdc640 1 --2- 172.21.15.40:0/894593156 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f41880a2800 0x7f41880a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f41a0102190 tx=0x7f4194005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:40.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.194+0000 7f41a4cdc640 1 -- 172.21.15.40:0/894593156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00ffae0 msgr2=0x7f41a0102420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:40.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.194+0000 7f41a4cdc640 1 --2- 172.21.15.40:0/894593156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00ffae0 0x7f41a0102420 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f419000eb70 tx=0x7f419000c6a0 comp rx=0 tx=0).stop 2024-03-18T22:40:40.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.194+0000 7f41a4cdc640 1 -- 172.21.15.40:0/894593156 shutdown_connections 2024-03-18T22:40:40.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.194+0000 7f41a4cdc640 1 --2- 172.21.15.40:0/894593156 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f41880a2800 0x7f41880a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:40.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.194+0000 7f41a4cdc640 1 --2- 172.21.15.40:0/894593156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41a0103470 0x7f41a0100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:40.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.194+0000 7f41a4cdc640 1 --2- 172.21.15.40:0/894593156 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f41a0102ab0 0x7f41a0100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:40.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.194+0000 7f41a4cdc640 1 --2- 172.21.15.40:0/894593156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f41a00ffae0 0x7f41a0102420 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:40.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.194+0000 7f41a4cdc640 1 -- 172.21.15.40:0/894593156 >> 172.21.15.40:0/894593156 conn(0x7f41a00fb410 msgr2=0x7f41a0105ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:40.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.194+0000 7f41a4cdc640 1 -- 172.21.15.40:0/894593156 shutdown_connections 2024-03-18T22:40:40.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.195+0000 7f41a4cdc640 1 -- 172.21.15.40:0/894593156 wait complete. 2024-03-18T22:40:40.210 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:39 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3867134208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:40:40.210 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:39 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/650623574' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T22:40:40.211 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-18T22:40:40.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.321+0000 7ff361f87640 1 -- 172.21.15.40:0/612322885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff35c101910 msgr2=0x7ff35c101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:40.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.321+0000 7ff361f87640 1 --2- 172.21.15.40:0/612322885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff35c101910 0x7ff35c101d90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff350009860 tx=0x7ff35002bb70 comp rx=0 tx=0).stop 2024-03-18T22:40:40.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.322+0000 7ff361f87640 1 -- 172.21.15.40:0/612322885 shutdown_connections 2024-03-18T22:40:40.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.322+0000 7ff361f87640 1 --2- 172.21.15.40:0/612322885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff35c1022d0 0x7ff35c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.322+0000 7ff361f87640 1 --2- 172.21.15.40:0/612322885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff35c101910 0x7ff35c101d90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.322+0000 7ff361f87640 1 --2- 172.21.15.40:0/612322885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff35c100720 0x7ff35c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.322+0000 7ff361f87640 1 -- 172.21.15.40:0/612322885 >> 172.21.15.40:0/612322885 conn(0x7ff35c0fbcb0 msgr2=0x7ff35c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.322+0000 7ff361f87640 1 -- 172.21.15.40:0/612322885 shutdown_connections 2024-03-18T22:40:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.322+0000 7ff361f87640 1 -- 172.21.15.40:0/612322885 wait complete. 2024-03-18T22:40:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.322+0000 7ff361f87640 1 Processor -- start 2024-03-18T22:40:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.322+0000 7ff361f87640 1 -- start start 2024-03-18T22:40:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.322+0000 7ff361f87640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff35c100720 0x7ff35c199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.322+0000 7ff361f87640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff35c101910 0x7ff35c19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff361f87640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff35c1022d0 0x7ff35c1a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff361f87640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff35c10c250 con 0x7ff35c101910 2024-03-18T22:40:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff361f87640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff35c10c0d0 con 0x7ff35c1022d0 2024-03-18T22:40:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff361f87640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff35c10c3d0 con 0x7ff35c100720 2024-03-18T22:40:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff35affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff35c101910 0x7ff35c19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff35affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff35c101910 0x7ff35c19a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51038/0 (socket says 172.21.15.40:51038) 2024-03-18T22:40:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff35bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff35c1022d0 0x7ff35c1a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff35b7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff35c100720 0x7ff35c199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff35affd640 1 -- 172.21.15.40:0/1065492783 learned_addr learned my addr 172.21.15.40:0/1065492783 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff35bfff640 1 -- 172.21.15.40:0/1065492783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff35c100720 msgr2=0x7ff35c199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff35bfff640 1 --2- 172.21.15.40:0/1065492783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff35c100720 0x7ff35c199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff35bfff640 1 -- 172.21.15.40:0/1065492783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff35c101910 msgr2=0x7ff35c19a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff35bfff640 1 --2- 172.21.15.40:0/1065492783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff35c101910 0x7ff35c19a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff35bfff640 1 -- 172.21.15.40:0/1065492783 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff35c1a1940 con 0x7ff35c1022d0 2024-03-18T22:40:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff35affd640 1 --2- 172.21.15.40:0/1065492783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff35c101910 0x7ff35c19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:40:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff35b7fe640 1 --2- 172.21.15.40:0/1065492783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff35c100720 0x7ff35c199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:40:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff35bfff640 1 --2- 172.21.15.40:0/1065492783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff35c1022d0 0x7ff35c1a12a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff354005fd0 tx=0x7ff35400ed20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff358ff9640 1 -- 172.21.15.40:0/1065492783 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35400ccc0 con 0x7ff35c1022d0 2024-03-18T22:40:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff361f87640 1 -- 172.21.15.40:0/1065492783 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff35c1a1c30 con 0x7ff35c1022d0 2024-03-18T22:40:40.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff358ff9640 1 -- 172.21.15.40:0/1065492783 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff3540035f0 con 0x7ff35c1022d0 2024-03-18T22:40:40.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff358ff9640 1 -- 172.21.15.40:0/1065492783 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35400ce60 con 0x7ff35c1022d0 2024-03-18T22:40:40.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.323+0000 7ff361f87640 1 -- 172.21.15.40:0/1065492783 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff35c1a2180 con 0x7ff35c1022d0 2024-03-18T22:40:40.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.324+0000 7ff361f87640 1 -- 172.21.15.40:0/1065492783 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff35c1a23b0 con 0x7ff35c1022d0 2024-03-18T22:40:40.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.326+0000 7ff358ff9640 1 -- 172.21.15.40:0/1065492783 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff3540037a0 con 0x7ff35c1022d0 2024-03-18T22:40:40.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.326+0000 7ff358ff9640 1 --2- 172.21.15.40:0/1065492783 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff3440a2800 0x7ff3440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:40.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.326+0000 7ff35b7fe640 1 --2- 172.21.15.40:0/1065492783 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff3440a2800 0x7ff3440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:40.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.326+0000 7ff358ff9640 1 -- 172.21.15.40:0/1065492783 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7ff3540cebf0 con 0x7ff35c1022d0 2024-03-18T22:40:40.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.327+0000 7ff35b7fe640 1 --2- 172.21.15.40:0/1065492783 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff3440a2800 0x7ff3440a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff35c19b200 tx=0x7ff34c009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:40.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:39 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3867134208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:40:40.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:39 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/650623574' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T22:40:40.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.334+0000 7ff358ff9640 1 -- 172.21.15.40:0/1065492783 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff354009510 con 0x7ff35c1022d0 2024-03-18T22:40:40.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:39 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3867134208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:40:40.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:39 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/650623574' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T22:40:40.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:40.519+0000 7ff361f87640 1 -- 172.21.15.40:0/1065492783 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7ff35c1a2580 con 0x7ff35c1022d0 2024-03-18T22:40:41.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/894593156' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T22:40:41.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:41.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:40:41.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:40:41.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/894593156' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T22:40:41.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T22:40:41.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1065492783' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T22:40:41.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T22:40:41.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:40 smithi103 ceph-mon[66377]: pgmap v281: 33 pgs: 5 creating+peering, 1 active+clean, 27 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:41.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/894593156' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T22:40:41.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:41.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:40:41.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:40:41.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/894593156' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T22:40:41.126 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T22:40:41.127 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1065492783' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T22:40:41.127 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T22:40:41.127 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:40 smithi040 ceph-mon[62643]: pgmap v281: 33 pgs: 5 creating+peering, 1 active+clean, 27 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:41.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.200+0000 7ff358ff9640 1 -- 172.21.15.40:0/1065492783 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7ff35409bf20 con 0x7ff35c1022d0 2024-03-18T22:40:41.201 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-03-18T22:40:41.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.205+0000 7ff361f87640 1 -- 172.21.15.40:0/1065492783 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff3440a2800 msgr2=0x7ff3440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:41.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.205+0000 7ff361f87640 1 --2- 172.21.15.40:0/1065492783 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff3440a2800 0x7ff3440a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff35c19b200 tx=0x7ff34c009500 comp rx=0 tx=0).stop 2024-03-18T22:40:41.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.205+0000 7ff361f87640 1 -- 172.21.15.40:0/1065492783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff35c1022d0 msgr2=0x7ff35c1a12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:41.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.205+0000 7ff361f87640 1 --2- 172.21.15.40:0/1065492783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff35c1022d0 0x7ff35c1a12a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff354005fd0 tx=0x7ff35400ed20 comp rx=0 tx=0).stop 2024-03-18T22:40:41.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.205+0000 7ff361f87640 1 -- 172.21.15.40:0/1065492783 shutdown_connections 2024-03-18T22:40:41.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.205+0000 7ff361f87640 1 --2- 172.21.15.40:0/1065492783 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff3440a2800 0x7ff3440a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:41.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.205+0000 7ff361f87640 1 --2- 172.21.15.40:0/1065492783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff35c1022d0 0x7ff35c1a12a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:41.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.205+0000 7ff361f87640 1 --2- 172.21.15.40:0/1065492783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff35c101910 0x7ff35c19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:41.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.205+0000 7ff361f87640 1 --2- 172.21.15.40:0/1065492783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff35c100720 0x7ff35c199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:41.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.205+0000 7ff361f87640 1 -- 172.21.15.40:0/1065492783 >> 172.21.15.40:0/1065492783 conn(0x7ff35c0fbcb0 msgr2=0x7ff35c0689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:41.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.206+0000 7ff361f87640 1 -- 172.21.15.40:0/1065492783 shutdown_connections 2024-03-18T22:40:41.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.206+0000 7ff361f87640 1 -- 172.21.15.40:0/1065492783 wait complete. 2024-03-18T22:40:41.225 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T22:40:41.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/894593156' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T22:40:41.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:41.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:40:41.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:41.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:40:41.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/894593156' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T22:40:41.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T22:40:41.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1065492783' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T22:40:41.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T22:40:41.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:40 smithi113 ceph-mon[66775]: pgmap v281: 33 pgs: 5 creating+peering, 1 active+clean, 27 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T22:40:41.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.330+0000 7feba30d3640 1 -- 172.21.15.40:0/3090461240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb9c101930 msgr2=0x7feb9c101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:41.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.330+0000 7feba30d3640 1 --2- 172.21.15.40:0/3090461240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb9c101930 0x7feb9c101db0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feb88009900 tx=0x7feb8802bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:41.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.330+0000 7feba30d3640 1 -- 172.21.15.40:0/3090461240 shutdown_connections 2024-03-18T22:40:41.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.330+0000 7feba30d3640 1 --2- 172.21.15.40:0/3090461240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb9c1022f0 0x7feb9c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:41.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.330+0000 7feba30d3640 1 --2- 172.21.15.40:0/3090461240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb9c101930 0x7feb9c101db0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:41.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.330+0000 7feba30d3640 1 --2- 172.21.15.40:0/3090461240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb9c100740 0x7feb9c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:41.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.330+0000 7feba30d3640 1 -- 172.21.15.40:0/3090461240 >> 172.21.15.40:0/3090461240 conn(0x7feb9c0fbcd0 msgr2=0x7feb9c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:41.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.330+0000 7feba30d3640 1 -- 172.21.15.40:0/3090461240 shutdown_connections 2024-03-18T22:40:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.330+0000 7feba30d3640 1 -- 172.21.15.40:0/3090461240 wait complete. 2024-03-18T22:40:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba30d3640 1 Processor -- start 2024-03-18T22:40:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba30d3640 1 -- start start 2024-03-18T22:40:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba30d3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb9c100740 0x7feb9c199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba30d3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb9c101930 0x7feb9c19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba30d3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb9c1022f0 0x7feb9c1a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba30d3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb9c10c250 con 0x7feb9c1022f0 2024-03-18T22:40:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba30d3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feb9c10c0d0 con 0x7feb9c100740 2024-03-18T22:40:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba30d3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7feb9c10c3d0 con 0x7feb9c101930 2024-03-18T22:40:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba1670640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb9c1022f0 0x7feb9c1a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba1670640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb9c1022f0 0x7feb9c1a1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51054/0 (socket says 172.21.15.40:51054) 2024-03-18T22:40:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba1670640 1 -- 172.21.15.40:0/2011374709 learned_addr learned my addr 172.21.15.40:0/2011374709 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feb9bfff640 1 --2- 172.21.15.40:0/2011374709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb9c101930 0x7feb9c19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba0e6f640 1 --2- 172.21.15.40:0/2011374709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb9c100740 0x7feb9c199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:41.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba1670640 1 -- 172.21.15.40:0/2011374709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb9c101930 msgr2=0x7feb9c19a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba1670640 1 --2- 172.21.15.40:0/2011374709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb9c101930 0x7feb9c19a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba1670640 1 -- 172.21.15.40:0/2011374709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb9c100740 msgr2=0x7feb9c199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba1670640 1 --2- 172.21.15.40:0/2011374709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb9c100740 0x7feb9c199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba1670640 1 -- 172.21.15.40:0/2011374709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9c1a18c0 con 0x7feb9c1022f0 2024-03-18T22:40:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.331+0000 7feba1670640 1 --2- 172.21.15.40:0/2011374709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb9c1022f0 0x7feb9c1a1220 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feb9400eb10 tx=0x7feb9400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.332+0000 7feb99ffb640 1 -- 172.21.15.40:0/2011374709 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb94004b20 con 0x7feb9c1022f0 2024-03-18T22:40:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.332+0000 7feb99ffb640 1 -- 172.21.15.40:0/2011374709 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb94004cc0 con 0x7feb9c1022f0 2024-03-18T22:40:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.332+0000 7feba30d3640 1 -- 172.21.15.40:0/2011374709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb9c1a1bb0 con 0x7feb9c1022f0 2024-03-18T22:40:41.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.332+0000 7feb99ffb640 1 -- 172.21.15.40:0/2011374709 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb94010690 con 0x7feb9c1022f0 2024-03-18T22:40:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.332+0000 7feba30d3640 1 -- 172.21.15.40:0/2011374709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb9c1a2080 con 0x7feb9c1022f0 2024-03-18T22:40:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.333+0000 7feba30d3640 1 -- 172.21.15.40:0/2011374709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb9c1a22b0 con 0x7feb9c1022f0 2024-03-18T22:40:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.334+0000 7feb99ffb640 1 -- 172.21.15.40:0/2011374709 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7feb94005020 con 0x7feb9c1022f0 2024-03-18T22:40:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.334+0000 7feb99ffb640 1 --2- 172.21.15.40:0/2011374709 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb8c0a2800 0x7feb8c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:41.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.334+0000 7feba0e6f640 1 --2- 172.21.15.40:0/2011374709 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb8c0a2800 0x7feb8c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:41.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.334+0000 7feb99ffb640 1 -- 172.21.15.40:0/2011374709 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7feb940cee20 con 0x7feb9c1022f0 2024-03-18T22:40:41.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.335+0000 7feba0e6f640 1 --2- 172.21.15.40:0/2011374709 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb8c0a2800 0x7feb8c0a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feb9c19ab50 tx=0x7feb90005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:41.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.339+0000 7feb99ffb640 1 -- 172.21.15.40:0/2011374709 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feb9409c070 con 0x7feb9c1022f0 2024-03-18T22:40:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:41.571+0000 7feba30d3640 1 -- 172.21.15.40:0/2011374709 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7feb9c1a2480 con 0x7feb9c1022f0 2024-03-18T22:40:42.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.197+0000 7feb99ffb640 1 -- 172.21.15.40:0/2011374709 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7feb9409b970 con 0x7feb9c1022f0 2024-03-18T22:40:42.198 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 2 and data pool 3 2024-03-18T22:40:42.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.199+0000 7feba30d3640 1 -- 172.21.15.40:0/2011374709 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb8c0a2800 msgr2=0x7feb8c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:42.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.199+0000 7feba30d3640 1 --2- 172.21.15.40:0/2011374709 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb8c0a2800 0x7feb8c0a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feb9c19ab50 tx=0x7feb90005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:42.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.199+0000 7feba30d3640 1 -- 172.21.15.40:0/2011374709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb9c1022f0 msgr2=0x7feb9c1a1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:42.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.199+0000 7feba30d3640 1 --2- 172.21.15.40:0/2011374709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb9c1022f0 0x7feb9c1a1220 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feb9400eb10 tx=0x7feb9400c2d0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.200+0000 7feba30d3640 1 -- 172.21.15.40:0/2011374709 shutdown_connections 2024-03-18T22:40:42.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.200+0000 7feba30d3640 1 --2- 172.21.15.40:0/2011374709 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb8c0a2800 0x7feb8c0a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.200+0000 7feba30d3640 1 --2- 172.21.15.40:0/2011374709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb9c1022f0 0x7feb9c1a1220 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.200+0000 7feba30d3640 1 --2- 172.21.15.40:0/2011374709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb9c101930 0x7feb9c19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.200+0000 7feba30d3640 1 --2- 172.21.15.40:0/2011374709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb9c100740 0x7feb9c199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.200+0000 7feba30d3640 1 -- 172.21.15.40:0/2011374709 >> 172.21.15.40:0/2011374709 conn(0x7feb9c0fbcd0 msgr2=0x7feb9c069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:42.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.200+0000 7feba30d3640 1 -- 172.21.15.40:0/2011374709 shutdown_connections 2024-03-18T22:40:42.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.200+0000 7feba30d3640 1 -- 172.21.15.40:0/2011374709 wait complete. 2024-03-18T22:40:42.213 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:41 smithi040 ceph-mon[62643]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T22:40:42.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:41 smithi040 ceph-mon[62643]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-18T22:40:42.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:41 smithi040 ceph-mon[62643]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T22:40:42.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:41 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2011374709' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T22:40:42.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:41 smithi040 ceph-mon[62643]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T22:40:42.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:41 smithi040 ceph-mon[62643]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T22:40:42.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:42.214 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:40:42.215 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.215 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-18T22:40:42.216 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-18T22:40:42.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 -- 172.21.15.40:0/4241820606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54ac1022f0 msgr2=0x7f54ac108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:42.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 --2- 172.21.15.40:0/4241820606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54ac1022f0 0x7f54ac108b80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009ca0 tx=0x7f54a402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 -- 172.21.15.40:0/4241820606 shutdown_connections 2024-03-18T22:40:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 --2- 172.21.15.40:0/4241820606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54ac1022f0 0x7f54ac108b80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 --2- 172.21.15.40:0/4241820606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54ac101930 0x7f54ac101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 --2- 172.21.15.40:0/4241820606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54ac100740 0x7f54ac100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 -- 172.21.15.40:0/4241820606 >> 172.21.15.40:0/4241820606 conn(0x7f54ac0fbcd0 msgr2=0x7f54ac0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 -- 172.21.15.40:0/4241820606 shutdown_connections 2024-03-18T22:40:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 -- 172.21.15.40:0/4241820606 wait complete. 2024-03-18T22:40:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 Processor -- start 2024-03-18T22:40:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 -- start start 2024-03-18T22:40:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54ac100740 0x7f54ac197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54ac101930 0x7f54ac197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54ac1022f0 0x7f54ac19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f54ac10c270 con 0x7f54ac101930 2024-03-18T22:40:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f54ac10c0f0 con 0x7f54ac100740 2024-03-18T22:40:42.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.325+0000 7f54b2db9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f54ac10c3f0 con 0x7f54ac1022f0 2024-03-18T22:40:42.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54abfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54ac101930 0x7f54ac197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:42.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54abfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54ac101930 0x7f54ac197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51072/0 (socket says 172.21.15.40:51072) 2024-03-18T22:40:42.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54abfff640 1 -- 172.21.15.40:0/1337661621 learned_addr learned my addr 172.21.15.40:0/1337661621 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:42.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54b0b55640 1 --2- 172.21.15.40:0/1337661621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54ac100740 0x7f54ac197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:42.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54abfff640 1 -- 172.21.15.40:0/1337661621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54ac1022f0 msgr2=0x7f54ac19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:42.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54b1356640 1 --2- 172.21.15.40:0/1337661621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54ac1022f0 0x7f54ac19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:42.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54abfff640 1 --2- 172.21.15.40:0/1337661621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54ac1022f0 0x7f54ac19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54abfff640 1 -- 172.21.15.40:0/1337661621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54ac100740 msgr2=0x7f54ac197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:42.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54abfff640 1 --2- 172.21.15.40:0/1337661621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54ac100740 0x7f54ac197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54abfff640 1 -- 172.21.15.40:0/1337661621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54ac19f6c0 con 0x7f54ac101930 2024-03-18T22:40:42.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54b1356640 1 --2- 172.21.15.40:0/1337661621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54ac1022f0 0x7f54ac19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:40:42.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54abfff640 1 --2- 172.21.15.40:0/1337661621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54ac101930 0x7f54ac197fe0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f549800d770 tx=0x7f549800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:42.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54a9ffb640 1 -- 172.21.15.40:0/1337661621 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5498017070 con 0x7f54ac101930 2024-03-18T22:40:42.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54a9ffb640 1 -- 172.21.15.40:0/1337661621 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54980054d0 con 0x7f54ac101930 2024-03-18T22:40:42.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54b2db9640 1 -- 172.21.15.40:0/1337661621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54ac19f9b0 con 0x7f54ac101930 2024-03-18T22:40:42.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54a9ffb640 1 -- 172.21.15.40:0/1337661621 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5498004cf0 con 0x7f54ac101930 2024-03-18T22:40:42.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.326+0000 7f54b2db9640 1 -- 172.21.15.40:0/1337661621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54ac19ff00 con 0x7f54ac101930 2024-03-18T22:40:42.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.327+0000 7f54b2db9640 1 -- 172.21.15.40:0/1337661621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54ac1a0130 con 0x7f54ac101930 2024-03-18T22:40:42.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:41 smithi113 ceph-mon[66775]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T22:40:42.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:41 smithi113 ceph-mon[66775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-18T22:40:42.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:41 smithi113 ceph-mon[66775]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T22:40:42.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:41 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2011374709' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T22:40:42.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:41 smithi113 ceph-mon[66775]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T22:40:42.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:41 smithi113 ceph-mon[66775]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T22:40:42.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:42.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:40:42.333 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.328+0000 7f54a9ffb640 1 -- 172.21.15.40:0/1337661621 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5498004e90 con 0x7f54ac101930 2024-03-18T22:40:42.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.329+0000 7f54a9ffb640 1 --2- 172.21.15.40:0/1337661621 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f549c0a2880 0x7f549c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:42.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.329+0000 7f54b0b55640 1 --2- 172.21.15.40:0/1337661621 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f549c0a2880 0x7f549c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:42.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.329+0000 7f54a9ffb640 1 -- 172.21.15.40:0/1337661621 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f54980ce2f0 con 0x7f54ac101930 2024-03-18T22:40:42.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.329+0000 7f54b0b55640 1 --2- 172.21.15.40:0/1337661621 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f549c0a2880 0x7f549c0a4d40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f54ac101790 tx=0x7f54a000b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:42.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.333+0000 7f54a9ffb640 1 -- 172.21.15.40:0/1337661621 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f549809b4e0 con 0x7f54ac101930 2024-03-18T22:40:42.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:41 smithi103 ceph-mon[66377]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T22:40:42.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:41 smithi103 ceph-mon[66377]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-18T22:40:42.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:41 smithi103 ceph-mon[66377]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T22:40:42.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:41 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2011374709' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T22:40:42.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:41 smithi103 ceph-mon[66377]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T22:40:42.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:41 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:41 smithi103 ceph-mon[66377]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T22:40:42.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:41 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:41 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:41 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:41 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:41 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:41 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:40:42.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:41 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:40:42.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:41 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:42.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.528+0000 7f54b2db9640 1 -- 172.21.15.40:0/1337661621 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f54ac1a0300 con 0x7f54ac101930 2024-03-18T22:40:42.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.799+0000 7f54a9ffb640 1 -- 172.21.15.40:0/1337661621 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f549809ade0 con 0x7f54ac101930 2024-03-18T22:40:42.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.802+0000 7f54b2db9640 1 -- 172.21.15.40:0/1337661621 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f549c0a2880 msgr2=0x7f549c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:42.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.802+0000 7f54b2db9640 1 --2- 172.21.15.40:0/1337661621 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f549c0a2880 0x7f549c0a4d40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f54ac101790 tx=0x7f54a000b390 comp rx=0 tx=0).stop 2024-03-18T22:40:42.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.802+0000 7f54b2db9640 1 -- 172.21.15.40:0/1337661621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54ac101930 msgr2=0x7f54ac197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:42.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.802+0000 7f54b2db9640 1 --2- 172.21.15.40:0/1337661621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54ac101930 0x7f54ac197fe0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f549800d770 tx=0x7f549800dc30 comp rx=0 tx=0).stop 2024-03-18T22:40:42.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.802+0000 7f54b2db9640 1 -- 172.21.15.40:0/1337661621 shutdown_connections 2024-03-18T22:40:42.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.802+0000 7f54b2db9640 1 --2- 172.21.15.40:0/1337661621 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f549c0a2880 0x7f549c0a4d40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.802+0000 7f54b2db9640 1 --2- 172.21.15.40:0/1337661621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f54ac1022f0 0x7f54ac19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.802+0000 7f54b2db9640 1 --2- 172.21.15.40:0/1337661621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f54ac101930 0x7f54ac197fe0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.802+0000 7f54b2db9640 1 --2- 172.21.15.40:0/1337661621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54ac100740 0x7f54ac197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.802+0000 7f54b2db9640 1 -- 172.21.15.40:0/1337661621 >> 172.21.15.40:0/1337661621 conn(0x7f54ac0fbcd0 msgr2=0x7f54ac0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:42.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.803+0000 7f54b2db9640 1 -- 172.21.15.40:0/1337661621 shutdown_connections 2024-03-18T22:40:42.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.803+0000 7f54b2db9640 1 -- 172.21.15.40:0/1337661621 wait complete. 2024-03-18T22:40:42.816 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T22:40:42.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.927+0000 7f9a2ddd4640 1 -- 172.21.15.40:0/1312258426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a281033e0 msgr2=0x7f9a28109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:42.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.927+0000 7f9a2ddd4640 1 --2- 172.21.15.40:0/1312258426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a281033e0 0x7f9a28109c70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9a20009900 tx=0x7f9a2002bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:42.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.927+0000 7f9a2ddd4640 1 -- 172.21.15.40:0/1312258426 shutdown_connections 2024-03-18T22:40:42.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.927+0000 7f9a2ddd4640 1 --2- 172.21.15.40:0/1312258426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a281033e0 0x7f9a28109c70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.927+0000 7f9a2ddd4640 1 --2- 172.21.15.40:0/1312258426 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a28102a20 0x7f9a28102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.927+0000 7f9a2ddd4640 1 --2- 172.21.15.40:0/1312258426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a28101830 0x7f9a28101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.927+0000 7f9a2ddd4640 1 -- 172.21.15.40:0/1312258426 >> 172.21.15.40:0/1312258426 conn(0x7f9a280fce00 msgr2=0x7f9a280ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:42.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.927+0000 7f9a2ddd4640 1 -- 172.21.15.40:0/1312258426 shutdown_connections 2024-03-18T22:40:42.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.927+0000 7f9a2ddd4640 1 -- 172.21.15.40:0/1312258426 wait complete. 2024-03-18T22:40:42.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a2ddd4640 1 Processor -- start 2024-03-18T22:40:42.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a2ddd4640 1 -- start start 2024-03-18T22:40:42.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a2ddd4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a28101830 0x7f9a28197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:42.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a2ddd4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a28102a20 0x7f9a28197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:42.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a2ddd4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a281033e0 0x7f9a2819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:42.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a2ddd4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2810d340 con 0x7f9a281033e0 2024-03-18T22:40:42.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a2ddd4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2810d1c0 con 0x7f9a28102a20 2024-03-18T22:40:42.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a2ddd4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2810d4c0 con 0x7f9a28101830 2024-03-18T22:40:42.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a27fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a281033e0 0x7f9a2819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:42.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a27fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a281033e0 0x7f9a2819efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51092/0 (socket says 172.21.15.40:51092) 2024-03-18T22:40:42.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a27fff640 1 -- 172.21.15.40:0/482805757 learned_addr learned my addr 172.21.15.40:0/482805757 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:42.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a26ffd640 1 --2- 172.21.15.40:0/482805757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a28102a20 0x7f9a28197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:42.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a277fe640 1 --2- 172.21.15.40:0/482805757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a28101830 0x7f9a28197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:42.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a27fff640 1 -- 172.21.15.40:0/482805757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a28101830 msgr2=0x7f9a28197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:42.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a27fff640 1 --2- 172.21.15.40:0/482805757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a28101830 0x7f9a28197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a27fff640 1 -- 172.21.15.40:0/482805757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a28102a20 msgr2=0x7f9a28197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:42.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a27fff640 1 --2- 172.21.15.40:0/482805757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a28102a20 0x7f9a28197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:42.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.928+0000 7f9a27fff640 1 -- 172.21.15.40:0/482805757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2819f630 con 0x7f9a281033e0 2024-03-18T22:40:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.929+0000 7f9a27fff640 1 --2- 172.21.15.40:0/482805757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a281033e0 0x7f9a2819efc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9a20009230 tx=0x7f9a200040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.929+0000 7f9a24ff9640 1 -- 172.21.15.40:0/482805757 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a20043020 con 0x7f9a281033e0 2024-03-18T22:40:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.929+0000 7f9a24ff9640 1 -- 172.21.15.40:0/482805757 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a200044c0 con 0x7f9a281033e0 2024-03-18T22:40:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.929+0000 7f9a2ddd4640 1 -- 172.21.15.40:0/482805757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a2819f8c0 con 0x7f9a281033e0 2024-03-18T22:40:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.929+0000 7f9a24ff9640 1 -- 172.21.15.40:0/482805757 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a20003bc0 con 0x7f9a281033e0 2024-03-18T22:40:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.929+0000 7f9a2ddd4640 1 -- 172.21.15.40:0/482805757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a2819fe60 con 0x7f9a281033e0 2024-03-18T22:40:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.930+0000 7f9a2ddd4640 1 -- 172.21.15.40:0/482805757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a28108ba0 con 0x7f9a281033e0 2024-03-18T22:40:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.931+0000 7f9a24ff9640 1 -- 172.21.15.40:0/482805757 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9a20046020 con 0x7f9a281033e0 2024-03-18T22:40:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.931+0000 7f9a24ff9640 1 --2- 172.21.15.40:0/482805757 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9a140a2880 0x7f9a140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.931+0000 7f9a277fe640 1 --2- 172.21.15.40:0/482805757 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9a140a2880 0x7f9a140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.932+0000 7f9a24ff9640 1 -- 172.21.15.40:0/482805757 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7f9a200f11b0 con 0x7f9a281033e0 2024-03-18T22:40:42.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.932+0000 7f9a277fe640 1 --2- 172.21.15.40:0/482805757 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9a140a2880 0x7f9a140a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9a28102880 tx=0x7f9a18009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:42.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:42.935+0000 7f9a24ff9640 1 -- 172.21.15.40:0/482805757 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9a2004d020 con 0x7f9a281033e0 2024-03-18T22:40:43.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:43.119+0000 7f9a2ddd4640 1 -- 172.21.15.40:0/482805757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f9a28062420 con 0x7f9a281033e0 2024-03-18T22:40:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:43 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2011374709' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T22:40:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:43 smithi113 ceph-mon[66775]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T22:40:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:43 smithi113 ceph-mon[66775]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T22:40:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:43 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1337661621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T22:40:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:43 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1337661621' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-18T22:40:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:43 smithi113 ceph-mon[66775]: pgmap v285: 97 pgs: 20 creating+peering, 32 active+clean, 45 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-18T22:40:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:43 smithi113 ceph-mon[66775]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T22:40:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:43 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:43.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:43 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] up:active 2024-03-18T22:40:43.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:43 smithi113 ceph-mon[66775]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T22:40:43.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:43 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/482805757' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T22:40:43.609 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 22:40:43 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-b[95936]: 2024-03-18T22:40:43.193+0000 7f747196f700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T22:40:43.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:43 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2011374709' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T22:40:43.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:43 smithi103 ceph-mon[66377]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T22:40:43.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:43 smithi103 ceph-mon[66377]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T22:40:43.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:43 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1337661621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T22:40:43.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:43 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1337661621' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-18T22:40:43.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:43 smithi103 ceph-mon[66377]: pgmap v285: 97 pgs: 20 creating+peering, 32 active+clean, 45 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-18T22:40:43.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:43 smithi103 ceph-mon[66377]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T22:40:43.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:43.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:43 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] up:active 2024-03-18T22:40:43.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:43 smithi103 ceph-mon[66377]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T22:40:43.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:43 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/482805757' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T22:40:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:43 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2011374709' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T22:40:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:43 smithi040 ceph-mon[62643]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T22:40:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:43 smithi040 ceph-mon[62643]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T22:40:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:43 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1337661621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T22:40:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:43 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1337661621' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-18T22:40:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:43 smithi040 ceph-mon[62643]: pgmap v285: 97 pgs: 20 creating+peering, 32 active+clean, 45 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-18T22:40:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:43 smithi040 ceph-mon[62643]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T22:40:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:43 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:40:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:43 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] up:active 2024-03-18T22:40:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:43 smithi040 ceph-mon[62643]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T22:40:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:43 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/482805757' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T22:40:44.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:44 smithi113 ceph-mon[66775]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T22:40:44.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:44 smithi113 ceph-mon[66775]: Cluster is now healthy 2024-03-18T22:40:44.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:44 smithi113 ceph-mon[66775]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T22:40:44.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:44 smithi103 ceph-mon[66377]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T22:40:44.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:44 smithi103 ceph-mon[66377]: Cluster is now healthy 2024-03-18T22:40:44.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:44 smithi103 ceph-mon[66377]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T22:40:44.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:44 smithi040 ceph-mon[62643]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T22:40:44.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:44 smithi040 ceph-mon[62643]: Cluster is now healthy 2024-03-18T22:40:44.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:44 smithi040 ceph-mon[62643]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T22:40:44.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.818+0000 7f9a24ff9640 1 -- 172.21.15.40:0/482805757 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7f9a200be1c0 con 0x7f9a281033e0 2024-03-18T22:40:44.819 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data_ec' created 2024-03-18T22:40:44.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.821+0000 7f9a2ddd4640 1 -- 172.21.15.40:0/482805757 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9a140a2880 msgr2=0x7f9a140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:44.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.822+0000 7f9a2ddd4640 1 --2- 172.21.15.40:0/482805757 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9a140a2880 0x7f9a140a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9a28102880 tx=0x7f9a18009500 comp rx=0 tx=0).stop 2024-03-18T22:40:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.822+0000 7f9a2ddd4640 1 -- 172.21.15.40:0/482805757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a281033e0 msgr2=0x7f9a2819efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.822+0000 7f9a2ddd4640 1 --2- 172.21.15.40:0/482805757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a281033e0 0x7f9a2819efc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9a20009230 tx=0x7f9a200040c0 comp rx=0 tx=0).stop 2024-03-18T22:40:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.822+0000 7f9a2ddd4640 1 -- 172.21.15.40:0/482805757 shutdown_connections 2024-03-18T22:40:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.822+0000 7f9a2ddd4640 1 --2- 172.21.15.40:0/482805757 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9a140a2880 0x7f9a140a4d40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.822+0000 7f9a2ddd4640 1 --2- 172.21.15.40:0/482805757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a281033e0 0x7f9a2819efc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.822+0000 7f9a2ddd4640 1 --2- 172.21.15.40:0/482805757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a28102a20 0x7f9a28197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.822+0000 7f9a2ddd4640 1 --2- 172.21.15.40:0/482805757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9a28101830 0x7f9a28197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.822+0000 7f9a2ddd4640 1 -- 172.21.15.40:0/482805757 >> 172.21.15.40:0/482805757 conn(0x7f9a280fce00 msgr2=0x7f9a28105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.822+0000 7f9a2ddd4640 1 -- 172.21.15.40:0/482805757 shutdown_connections 2024-03-18T22:40:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.822+0000 7f9a2ddd4640 1 -- 172.21.15.40:0/482805757 wait complete. 2024-03-18T22:40:44.840 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-18T22:40:44.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.951+0000 7f17e407e640 1 -- 172.21.15.40:0/2437920527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17dc101920 msgr2=0x7f17dc101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:44.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.951+0000 7f17e407e640 1 --2- 172.21.15.40:0/2437920527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17dc101920 0x7f17dc101da0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009900 tx=0x7f17cc02bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:44.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.951+0000 7f17e407e640 1 -- 172.21.15.40:0/2437920527 shutdown_connections 2024-03-18T22:40:44.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.951+0000 7f17e407e640 1 --2- 172.21.15.40:0/2437920527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17dc1022e0 0x7f17dc108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:44.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.951+0000 7f17e407e640 1 --2- 172.21.15.40:0/2437920527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17dc101920 0x7f17dc101da0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:44.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.951+0000 7f17e407e640 1 --2- 172.21.15.40:0/2437920527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17dc100730 0x7f17dc100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:44.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.951+0000 7f17e407e640 1 -- 172.21.15.40:0/2437920527 >> 172.21.15.40:0/2437920527 conn(0x7f17dc0fbcc0 msgr2=0x7f17dc0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:44.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.951+0000 7f17e407e640 1 -- 172.21.15.40:0/2437920527 shutdown_connections 2024-03-18T22:40:44.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.951+0000 7f17e407e640 1 -- 172.21.15.40:0/2437920527 wait complete. 2024-03-18T22:40:44.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e407e640 1 Processor -- start 2024-03-18T22:40:44.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e407e640 1 -- start start 2024-03-18T22:40:44.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e407e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17dc100730 0x7f17dc197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e407e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17dc101920 0x7f17dc197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e407e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17dc1022e0 0x7f17dc19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e407e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f17dc10c260 con 0x7f17dc101920 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e407e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f17dc10c0e0 con 0x7f17dc1022e0 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e407e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f17dc10c3e0 con 0x7f17dc100730 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e1619640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17dc101920 0x7f17dc197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e1619640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17dc101920 0x7f17dc197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51108/0 (socket says 172.21.15.40:51108) 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e1619640 1 -- 172.21.15.40:0/1623687199 learned_addr learned my addr 172.21.15.40:0/1623687199 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e261b640 1 --2- 172.21.15.40:0/1623687199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17dc1022e0 0x7f17dc19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e1e1a640 1 --2- 172.21.15.40:0/1623687199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17dc100730 0x7f17dc197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e1619640 1 -- 172.21.15.40:0/1623687199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17dc100730 msgr2=0x7f17dc197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e1619640 1 --2- 172.21.15.40:0/1623687199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17dc100730 0x7f17dc197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e1619640 1 -- 172.21.15.40:0/1623687199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17dc1022e0 msgr2=0x7f17dc19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e1619640 1 --2- 172.21.15.40:0/1623687199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17dc1022e0 0x7f17dc19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e1619640 1 -- 172.21.15.40:0/1623687199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17dc19f6d0 con 0x7f17dc101920 2024-03-18T22:40:44.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17e1619640 1 --2- 172.21.15.40:0/1623687199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17dc101920 0x7f17dc197ff0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f17cc006010 tx=0x7f17cc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.952+0000 7f17d2ffd640 1 -- 172.21.15.40:0/1623687199 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc043020 con 0x7f17dc101920 2024-03-18T22:40:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.953+0000 7f17d2ffd640 1 -- 172.21.15.40:0/1623687199 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17cc004440 con 0x7f17dc101920 2024-03-18T22:40:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.953+0000 7f17e407e640 1 -- 172.21.15.40:0/1623687199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17dc19f960 con 0x7f17dc101920 2024-03-18T22:40:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.953+0000 7f17d2ffd640 1 -- 172.21.15.40:0/1623687199 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc003bf0 con 0x7f17dc101920 2024-03-18T22:40:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.953+0000 7f17e407e640 1 -- 172.21.15.40:0/1623687199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17dc19feb0 con 0x7f17dc101920 2024-03-18T22:40:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.954+0000 7f17e407e640 1 -- 172.21.15.40:0/1623687199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17dc1a0080 con 0x7f17dc101920 2024-03-18T22:40:44.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.955+0000 7f17d2ffd640 1 -- 172.21.15.40:0/1623687199 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f17cc046020 con 0x7f17dc101920 2024-03-18T22:40:44.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.956+0000 7f17d2ffd640 1 --2- 172.21.15.40:0/1623687199 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17c80a2800 0x7f17c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:44.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.956+0000 7f17e1e1a640 1 --2- 172.21.15.40:0/1623687199 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17c80a2800 0x7f17c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:44.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.956+0000 7f17d2ffd640 1 -- 172.21.15.40:0/1623687199 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7f17cc0f15d0 con 0x7f17dc101920 2024-03-18T22:40:44.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.957+0000 7f17e1e1a640 1 --2- 172.21.15.40:0/1623687199 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17c80a2800 0x7f17c80a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f17dc101780 tx=0x7f17d4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:44.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:44.960+0000 7f17d2ffd640 1 -- 172.21.15.40:0/1623687199 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f17cc04d020 con 0x7f17dc101920 2024-03-18T22:40:45.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.141+0000 7f17e407e640 1 -- 172.21.15.40:0/1623687199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f17dc1a0250 con 0x7f17dc101920 2024-03-18T22:40:45.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.820+0000 7f17d2ffd640 1 -- 172.21.15.40:0/1623687199 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7f17cc0be450 con 0x7f17dc101920 2024-03-18T22:40:45.821 INFO:teuthology.orchestra.run.smithi040.stderr:set pool 4 allow_ec_overwrites to true 2024-03-18T22:40:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.824+0000 7f17e407e640 1 -- 172.21.15.40:0/1623687199 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17c80a2800 msgr2=0x7f17c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.824+0000 7f17e407e640 1 --2- 172.21.15.40:0/1623687199 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17c80a2800 0x7f17c80a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f17dc101780 tx=0x7f17d4005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.824+0000 7f17e407e640 1 -- 172.21.15.40:0/1623687199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17dc101920 msgr2=0x7f17dc197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.824+0000 7f17e407e640 1 --2- 172.21.15.40:0/1623687199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17dc101920 0x7f17dc197ff0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f17cc006010 tx=0x7f17cc004040 comp rx=0 tx=0).stop 2024-03-18T22:40:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.825+0000 7f17e407e640 1 -- 172.21.15.40:0/1623687199 shutdown_connections 2024-03-18T22:40:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.825+0000 7f17e407e640 1 --2- 172.21.15.40:0/1623687199 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17c80a2800 0x7f17c80a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.825+0000 7f17e407e640 1 --2- 172.21.15.40:0/1623687199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17dc1022e0 0x7f17dc19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.825+0000 7f17e407e640 1 --2- 172.21.15.40:0/1623687199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17dc101920 0x7f17dc197ff0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:45.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.825+0000 7f17e407e640 1 --2- 172.21.15.40:0/1623687199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17dc100730 0x7f17dc197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:45.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.825+0000 7f17e407e640 1 -- 172.21.15.40:0/1623687199 >> 172.21.15.40:0/1623687199 conn(0x7f17dc0fbcc0 msgr2=0x7f17dc069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:45.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.825+0000 7f17e407e640 1 -- 172.21.15.40:0/1623687199 shutdown_connections 2024-03-18T22:40:45.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.825+0000 7f17e407e640 1 -- 172.21.15.40:0/1623687199 wait complete. 2024-03-18T22:40:45.850 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-18T22:40:45.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.960+0000 7fc09c044640 1 -- 172.21.15.40:0/1572795569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc094102a20 msgr2=0x7fc094102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:45.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.960+0000 7fc09c044640 1 --2- 172.21.15.40:0/1572795569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc094102a20 0x7fc094102ea0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc084009900 tx=0x7fc08402bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:45.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.960+0000 7fc09c044640 1 -- 172.21.15.40:0/1572795569 shutdown_connections 2024-03-18T22:40:45.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.960+0000 7fc09c044640 1 --2- 172.21.15.40:0/1572795569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0941033e0 0x7fc094109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:45.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.960+0000 7fc09c044640 1 --2- 172.21.15.40:0/1572795569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc094102a20 0x7fc094102ea0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:45.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.960+0000 7fc09c044640 1 --2- 172.21.15.40:0/1572795569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc094101830 0x7fc094101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:45.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.960+0000 7fc09c044640 1 -- 172.21.15.40:0/1572795569 >> 172.21.15.40:0/1572795569 conn(0x7fc0940fcdc0 msgr2=0x7fc0940ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:45.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.960+0000 7fc09c044640 1 -- 172.21.15.40:0/1572795569 shutdown_connections 2024-03-18T22:40:45.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.960+0000 7fc09c044640 1 -- 172.21.15.40:0/1572795569 wait complete. 2024-03-18T22:40:45.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc09c044640 1 Processor -- start 2024-03-18T22:40:45.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc09c044640 1 -- start start 2024-03-18T22:40:45.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc09c044640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc094101830 0x7fc094197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:45.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc09c044640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc094102a20 0x7fc094197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:45.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc09c044640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0941033e0 0x7fc09419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:45.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc09c044640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc09410d360 con 0x7fc094102a20 2024-03-18T22:40:45.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc09c044640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc09410d1e0 con 0x7fc0941033e0 2024-03-18T22:40:45.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc09c044640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc09410d4e0 con 0x7fc094101830 2024-03-18T22:40:45.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc0995df640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc094102a20 0x7fc094197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:45.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc0995df640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc094102a20 0x7fc094197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51124/0 (socket says 172.21.15.40:51124) 2024-03-18T22:40:45.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc0995df640 1 -- 172.21.15.40:0/322025845 learned_addr learned my addr 172.21.15.40:0/322025845 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:45.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc09a5e1640 1 --2- 172.21.15.40:0/322025845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0941033e0 0x7fc09419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:45.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc099de0640 1 --2- 172.21.15.40:0/322025845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc094101830 0x7fc094197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:45.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc0995df640 1 -- 172.21.15.40:0/322025845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc094101830 msgr2=0x7fc094197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:45.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc0995df640 1 --2- 172.21.15.40:0/322025845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc094101830 0x7fc094197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:45.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc0995df640 1 -- 172.21.15.40:0/322025845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0941033e0 msgr2=0x7fc09419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:45.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc0995df640 1 --2- 172.21.15.40:0/322025845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0941033e0 0x7fc09419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:45.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.961+0000 7fc0995df640 1 -- 172.21.15.40:0/322025845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc09419f6c0 con 0x7fc094102a20 2024-03-18T22:40:45.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.962+0000 7fc0995df640 1 --2- 172.21.15.40:0/322025845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc094102a20 0x7fc094197fe0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc084006010 tx=0x7fc084004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:45.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.962+0000 7fc08affd640 1 -- 172.21.15.40:0/322025845 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc084043020 con 0x7fc094102a20 2024-03-18T22:40:45.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.962+0000 7fc09c044640 1 -- 172.21.15.40:0/322025845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc09419f950 con 0x7fc094102a20 2024-03-18T22:40:45.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.962+0000 7fc08affd640 1 -- 172.21.15.40:0/322025845 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc084004440 con 0x7fc094102a20 2024-03-18T22:40:45.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.962+0000 7fc08affd640 1 -- 172.21.15.40:0/322025845 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc084003a80 con 0x7fc094102a20 2024-03-18T22:40:45.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.962+0000 7fc09c044640 1 -- 172.21.15.40:0/322025845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc09419fea0 con 0x7fc094102a20 2024-03-18T22:40:45.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.963+0000 7fc09c044640 1 -- 172.21.15.40:0/322025845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc094108c70 con 0x7fc094102a20 2024-03-18T22:40:45.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.964+0000 7fc08affd640 1 -- 172.21.15.40:0/322025845 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc084046020 con 0x7fc094102a20 2024-03-18T22:40:45.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.964+0000 7fc08affd640 1 --2- 172.21.15.40:0/322025845 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc0800a2800 0x7fc0800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:45.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.965+0000 7fc099de0640 1 --2- 172.21.15.40:0/322025845 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc0800a2800 0x7fc0800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:45.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.965+0000 7fc08affd640 1 -- 172.21.15.40:0/322025845 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7fc0840f0ef0 con 0x7fc094102a20 2024-03-18T22:40:45.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.965+0000 7fc099de0640 1 --2- 172.21.15.40:0/322025845 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc0800a2800 0x7fc0800a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc090005fd0 tx=0x7fc09000b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:45.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:45.969+0000 7fc08affd640 1 -- 172.21.15.40:0/322025845 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc08404d020 con 0x7fc094102a20 2024-03-18T22:40:46.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:45 smithi103 ceph-mon[66377]: pgmap v287: 97 pgs: 20 creating+peering, 51 active+clean, 26 unknown; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 1 op/s 2024-03-18T22:40:46.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:45 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/482805757' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T22:40:46.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:45 smithi103 ceph-mon[66377]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T22:40:46.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:45 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1623687199' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T22:40:46.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:45 smithi040 ceph-mon[62643]: pgmap v287: 97 pgs: 20 creating+peering, 51 active+clean, 26 unknown; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 1 op/s 2024-03-18T22:40:46.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:45 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/482805757' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T22:40:46.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:45 smithi040 ceph-mon[62643]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T22:40:46.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:45 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1623687199' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T22:40:46.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.195+0000 7fc09c044640 1 -- 172.21.15.40:0/322025845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fc094062420 con 0x7fc094102a20 2024-03-18T22:40:46.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:45 smithi113 ceph-mon[66775]: pgmap v287: 97 pgs: 20 creating+peering, 51 active+clean, 26 unknown; 577 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 1 op/s 2024-03-18T22:40:46.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:45 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/482805757' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T22:40:46.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:45 smithi113 ceph-mon[66775]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T22:40:46.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:45 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1623687199' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T22:40:46.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.838+0000 7fc08affd640 1 -- 172.21.15.40:0/322025845 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) ==== 137+0+0 (secure 0 0 0) 0x7fc0840bdd70 con 0x7fc094102a20 2024-03-18T22:40:46.839 INFO:teuthology.orchestra.run.smithi040.stderr:added data pool 4 to fsmap 2024-03-18T22:40:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.842+0000 7fc09c044640 1 -- 172.21.15.40:0/322025845 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc0800a2800 msgr2=0x7fc0800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.842+0000 7fc09c044640 1 --2- 172.21.15.40:0/322025845 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc0800a2800 0x7fc0800a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc090005fd0 tx=0x7fc09000b390 comp rx=0 tx=0).stop 2024-03-18T22:40:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.842+0000 7fc09c044640 1 -- 172.21.15.40:0/322025845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc094102a20 msgr2=0x7fc094197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.842+0000 7fc09c044640 1 --2- 172.21.15.40:0/322025845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc094102a20 0x7fc094197fe0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc084006010 tx=0x7fc084004040 comp rx=0 tx=0).stop 2024-03-18T22:40:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.842+0000 7fc09c044640 1 -- 172.21.15.40:0/322025845 shutdown_connections 2024-03-18T22:40:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.842+0000 7fc09c044640 1 --2- 172.21.15.40:0/322025845 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc0800a2800 0x7fc0800a4cc0 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc090005fd0 tx=0x7fc09000b390 comp rx=0 tx=0).stop 2024-03-18T22:40:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.842+0000 7fc09c044640 1 --2- 172.21.15.40:0/322025845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0941033e0 0x7fc09419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.842+0000 7fc09c044640 1 --2- 172.21.15.40:0/322025845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc094102a20 0x7fc094197fe0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.842+0000 7fc09c044640 1 --2- 172.21.15.40:0/322025845 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc094101830 0x7fc094197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.842+0000 7fc09c044640 1 -- 172.21.15.40:0/322025845 >> 172.21.15.40:0/322025845 conn(0x7fc0940fcdc0 msgr2=0x7fc094105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.842+0000 7fc09c044640 1 -- 172.21.15.40:0/322025845 shutdown_connections 2024-03-18T22:40:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.842+0000 7fc09c044640 1 -- 172.21.15.40:0/322025845 wait complete. 2024-03-18T22:40:46.858 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:46.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.965+0000 7fdec1910640 1 -- 172.21.15.40:0/1790129295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdebc1075c0 msgr2=0x7fdebc1099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:46.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.965+0000 7fdec1910640 1 --2- 172.21.15.40:0/1790129295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdebc1075c0 0x7fdebc1099b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4009ca0 tx=0x7fdeb402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:40:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.965+0000 7fdec1910640 1 -- 172.21.15.40:0/1790129295 shutdown_connections 2024-03-18T22:40:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.965+0000 7fdec1910640 1 --2- 172.21.15.40:0/1790129295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdebc1075c0 0x7fdebc1099b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.965+0000 7fdec1910640 1 --2- 172.21.15.40:0/1790129295 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc104c90 0x7fdebc107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.965+0000 7fdec1910640 1 --2- 172.21.15.40:0/1790129295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdebc102360 0x7fdebc104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.965+0000 7fdec1910640 1 -- 172.21.15.40:0/1790129295 >> 172.21.15.40:0/1790129295 conn(0x7fdebc0fbcb0 msgr2=0x7fdebc0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.966+0000 7fdec1910640 1 -- 172.21.15.40:0/1790129295 shutdown_connections 2024-03-18T22:40:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.966+0000 7fdec1910640 1 -- 172.21.15.40:0/1790129295 wait complete. 2024-03-18T22:40:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.966+0000 7fdec1910640 1 Processor -- start 2024-03-18T22:40:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.966+0000 7fdec1910640 1 -- start start 2024-03-18T22:40:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.966+0000 7fdec1910640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdebc102360 0x7fdebc197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.966+0000 7fdec1910640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc104c90 0x7fdebc197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.966+0000 7fdec1910640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdebc1075c0 0x7fdebc19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.966+0000 7fdec1910640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc10d390 con 0x7fdebc1075c0 2024-03-18T22:40:46.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.966+0000 7fdec1910640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc10d210 con 0x7fdebc102360 2024-03-18T22:40:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.966+0000 7fdec1910640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc10d510 con 0x7fdebc104c90 2024-03-18T22:40:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.966+0000 7fdebb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdebc1075c0 0x7fdebc19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.966+0000 7fdebb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdebc1075c0 0x7fdebc19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51132/0 (socket says 172.21.15.40:51132) 2024-03-18T22:40:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.966+0000 7fdebb7fe640 1 -- 172.21.15.40:0/923429626 learned_addr learned my addr 172.21.15.40:0/923429626 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.966+0000 7fdebaffd640 1 --2- 172.21.15.40:0/923429626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdebc102360 0x7fdebc197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.966+0000 7fdeb3fff640 1 --2- 172.21.15.40:0/923429626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc104c90 0x7fdebc197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.967+0000 7fdebb7fe640 1 -- 172.21.15.40:0/923429626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc104c90 msgr2=0x7fdebc197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.967+0000 7fdebb7fe640 1 --2- 172.21.15.40:0/923429626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc104c90 0x7fdebc197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.967+0000 7fdebb7fe640 1 -- 172.21.15.40:0/923429626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdebc102360 msgr2=0x7fdebc197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.967+0000 7fdebb7fe640 1 --2- 172.21.15.40:0/923429626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdebc102360 0x7fdebc197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.967+0000 7fdebb7fe640 1 -- 172.21.15.40:0/923429626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdebc19f6b0 con 0x7fdebc1075c0 2024-03-18T22:40:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.967+0000 7fdeb3fff640 1 --2- 172.21.15.40:0/923429626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc104c90 0x7fdebc197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:40:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.967+0000 7fdebb7fe640 1 --2- 172.21.15.40:0/923429626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdebc1075c0 0x7fdebc19f040 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdeb402c700 tx=0x7fdeb40328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.967+0000 7fdec090e640 1 -- 172.21.15.40:0/923429626 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4041020 con 0x7fdebc1075c0 2024-03-18T22:40:46.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.967+0000 7fdec1910640 1 -- 172.21.15.40:0/923429626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdebc19f940 con 0x7fdebc1075c0 2024-03-18T22:40:46.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.967+0000 7fdec090e640 1 -- 172.21.15.40:0/923429626 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdeb4003e20 con 0x7fdebc1075c0 2024-03-18T22:40:46.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.967+0000 7fdec090e640 1 -- 172.21.15.40:0/923429626 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4039040 con 0x7fdebc1075c0 2024-03-18T22:40:46.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.967+0000 7fdec1910640 1 -- 172.21.15.40:0/923429626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdebc19fe80 con 0x7fdebc1075c0 2024-03-18T22:40:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.968+0000 7fdec1910640 1 -- 172.21.15.40:0/923429626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdebc0fd7a0 con 0x7fdebc1075c0 2024-03-18T22:40:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.969+0000 7fdec090e640 1 -- 172.21.15.40:0/923429626 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fdeb4044020 con 0x7fdebc1075c0 2024-03-18T22:40:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.970+0000 7fdec090e640 1 --2- 172.21.15.40:0/923429626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdea80a2800 0x7fdea80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.970+0000 7fdebaffd640 1 --2- 172.21.15.40:0/923429626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdea80a2800 0x7fdea80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.970+0000 7fdec090e640 1 -- 172.21.15.40:0/923429626 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdeb40f0fb0 con 0x7fdebc1075c0 2024-03-18T22:40:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.970+0000 7fdebaffd640 1 --2- 172.21.15.40:0/923429626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdea80a2800 0x7fdea80a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009bd0 tx=0x7fdeac005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:46.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:46.974+0000 7fdec090e640 1 -- 172.21.15.40:0/923429626 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdeb404b020 con 0x7fdebc1075c0 2024-03-18T22:40:47.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:46 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1623687199' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-18T22:40:47.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:46 smithi103 ceph-mon[66377]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T22:40:47.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:46 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/322025845' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T22:40:47.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:46 smithi103 ceph-mon[66377]: pgmap v290: 129 pgs: 6 creating+peering, 103 active+clean, 20 unknown; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 9 op/s 2024-03-18T22:40:47.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:46 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1623687199' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-18T22:40:47.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:46 smithi040 ceph-mon[62643]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T22:40:47.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:46 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/322025845' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T22:40:47.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:46 smithi040 ceph-mon[62643]: pgmap v290: 129 pgs: 6 creating+peering, 103 active+clean, 20 unknown; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 9 op/s 2024-03-18T22:40:47.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.197+0000 7fdec1910640 1 -- 172.21.15.40:0/923429626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdebc062420 con 0x7fdebc1075c0 2024-03-18T22:40:47.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.198+0000 7fdec090e640 1 -- 172.21.15.40:0/923429626 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11812 (secure 0 0 0) 0x7fdeb40bde10 con 0x7fdebc1075c0 2024-03-18T22:40:47.199 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:47.199 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14604,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:40:46.196914+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24509,"qdb_cluster":[24509]},"id":1}]} 2024-03-18T22:40:47.199 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 16 2024-03-18T22:40:47.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.201+0000 7fdec1910640 1 -- 172.21.15.40:0/923429626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdea80a2800 msgr2=0x7fdea80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:47.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.201+0000 7fdec1910640 1 --2- 172.21.15.40:0/923429626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdea80a2800 0x7fdea80a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009bd0 tx=0x7fdeac005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:47.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.201+0000 7fdec1910640 1 -- 172.21.15.40:0/923429626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdebc1075c0 msgr2=0x7fdebc19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:47.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.201+0000 7fdec1910640 1 --2- 172.21.15.40:0/923429626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdebc1075c0 0x7fdebc19f040 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdeb402c700 tx=0x7fdeb40328a0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.201+0000 7fdec1910640 1 -- 172.21.15.40:0/923429626 shutdown_connections 2024-03-18T22:40:47.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.201+0000 7fdec1910640 1 --2- 172.21.15.40:0/923429626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdea80a2800 0x7fdea80a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.201+0000 7fdec1910640 1 --2- 172.21.15.40:0/923429626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdebc1075c0 0x7fdebc19f040 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.201+0000 7fdec1910640 1 --2- 172.21.15.40:0/923429626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdebc104c90 0x7fdebc197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.201+0000 7fdec1910640 1 --2- 172.21.15.40:0/923429626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdebc102360 0x7fdebc197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.201+0000 7fdec1910640 1 -- 172.21.15.40:0/923429626 >> 172.21.15.40:0/923429626 conn(0x7fdebc0fbcb0 msgr2=0x7fdebc102e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:47.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.202+0000 7fdec1910640 1 -- 172.21.15.40:0/923429626 shutdown_connections 2024-03-18T22:40:47.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.202+0000 7fdec1910640 1 -- 172.21.15.40:0/923429626 wait complete. 2024-03-18T22:40:47.214 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:40:47.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.315+0000 7fab1d6b2640 1 -- 172.21.15.40:0/4147623413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab18104c10 msgr2=0x7fab18107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:47.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.315+0000 7fab1d6b2640 1 --2- 172.21.15.40:0/4147623413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab18104c10 0x7fab18107000 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fab00009900 tx=0x7fab0002bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:47.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.315+0000 7fab1d6b2640 1 -- 172.21.15.40:0/4147623413 shutdown_connections 2024-03-18T22:40:47.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.315+0000 7fab1d6b2640 1 --2- 172.21.15.40:0/4147623413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab18107540 0x7fab18109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.315+0000 7fab1d6b2640 1 --2- 172.21.15.40:0/4147623413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab18104c10 0x7fab18107000 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.315+0000 7fab1d6b2640 1 --2- 172.21.15.40:0/4147623413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab181022e0 0x7fab181046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.315+0000 7fab1d6b2640 1 -- 172.21.15.40:0/4147623413 >> 172.21.15.40:0/4147623413 conn(0x7fab180fbcb0 msgr2=0x7fab180fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:47.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.315+0000 7fab1d6b2640 1 -- 172.21.15.40:0/4147623413 shutdown_connections 2024-03-18T22:40:47.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.315+0000 7fab1d6b2640 1 -- 172.21.15.40:0/4147623413 wait complete. 2024-03-18T22:40:47.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.316+0000 7fab1d6b2640 1 Processor -- start 2024-03-18T22:40:47.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.316+0000 7fab1d6b2640 1 -- start start 2024-03-18T22:40:47.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.316+0000 7fab1d6b2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab181022e0 0x7fab18197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:47.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.316+0000 7fab1d6b2640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab18104c10 0x7fab18197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:47.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.316+0000 7fab1d6b2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab18107540 0x7fab1819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:47.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.316+0000 7fab1d6b2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fab1810d380 con 0x7fab18107540 2024-03-18T22:40:47.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.316+0000 7fab1d6b2640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fab1810d200 con 0x7fab18104c10 2024-03-18T22:40:47.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.316+0000 7fab1d6b2640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fab1810d500 con 0x7fab181022e0 2024-03-18T22:40:47.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.316+0000 7fab177fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab18107540 0x7fab1819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:47.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.316+0000 7fab177fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab18107540 0x7fab1819f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51148/0 (socket says 172.21.15.40:51148) 2024-03-18T22:40:47.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.316+0000 7fab177fe640 1 -- 172.21.15.40:0/73639603 learned_addr learned my addr 172.21.15.40:0/73639603 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:47.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.316+0000 7fab167fc640 1 --2- 172.21.15.40:0/73639603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab18104c10 0x7fab18197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:47.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.317+0000 7fab16ffd640 1 --2- 172.21.15.40:0/73639603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab181022e0 0x7fab18197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:47.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.317+0000 7fab177fe640 1 -- 172.21.15.40:0/73639603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab181022e0 msgr2=0x7fab18197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:47.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.317+0000 7fab177fe640 1 --2- 172.21.15.40:0/73639603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab181022e0 0x7fab18197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.317+0000 7fab177fe640 1 -- 172.21.15.40:0/73639603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab18104c10 msgr2=0x7fab18197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:47.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.317+0000 7fab177fe640 1 --2- 172.21.15.40:0/73639603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab18104c10 0x7fab18197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.317+0000 7fab177fe640 1 -- 172.21.15.40:0/73639603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1819f6b0 con 0x7fab18107540 2024-03-18T22:40:47.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.317+0000 7fab16ffd640 1 --2- 172.21.15.40:0/73639603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab181022e0 0x7fab18197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:40:47.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.317+0000 7fab177fe640 1 --2- 172.21.15.40:0/73639603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab18107540 0x7fab1819f040 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fab0800eb10 tx=0x7fab0800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:47.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.317+0000 7fab0ffff640 1 -- 172.21.15.40:0/73639603 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab08004b20 con 0x7fab18107540 2024-03-18T22:40:47.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.317+0000 7fab1d6b2640 1 -- 172.21.15.40:0/73639603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab1819f9a0 con 0x7fab18107540 2024-03-18T22:40:47.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.317+0000 7fab0ffff640 1 -- 172.21.15.40:0/73639603 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab08004cc0 con 0x7fab18107540 2024-03-18T22:40:47.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.317+0000 7fab0ffff640 1 -- 172.21.15.40:0/73639603 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab08010690 con 0x7fab18107540 2024-03-18T22:40:47.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.317+0000 7fab1d6b2640 1 -- 172.21.15.40:0/73639603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab1819ff10 con 0x7fab18107540 2024-03-18T22:40:47.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.318+0000 7fab1d6b2640 1 -- 172.21.15.40:0/73639603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab180fd7a0 con 0x7fab18107540 2024-03-18T22:40:47.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.319+0000 7fab0ffff640 1 -- 172.21.15.40:0/73639603 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fab08005000 con 0x7fab18107540 2024-03-18T22:40:47.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.320+0000 7fab0ffff640 1 --2- 172.21.15.40:0/73639603 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab040a2800 0x7fab040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:47.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.320+0000 7fab16ffd640 1 --2- 172.21.15.40:0/73639603 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab040a2800 0x7fab040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:47.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.320+0000 7fab0ffff640 1 -- 172.21.15.40:0/73639603 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fab080cf790 con 0x7fab18107540 2024-03-18T22:40:47.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.320+0000 7fab16ffd640 1 --2- 172.21.15.40:0/73639603 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab040a2800 0x7fab040a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fab10009b50 tx=0x7fab10005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:47.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.323+0000 7fab0ffff640 1 -- 172.21.15.40:0/73639603 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fab0809c5f0 con 0x7fab18107540 2024-03-18T22:40:47.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:46 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1623687199' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-18T22:40:47.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:46 smithi113 ceph-mon[66775]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T22:40:47.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:46 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/322025845' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T22:40:47.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:46 smithi113 ceph-mon[66775]: pgmap v290: 129 pgs: 6 creating+peering, 103 active+clean, 20 unknown; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 9 op/s 2024-03-18T22:40:47.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.509+0000 7fab1d6b2640 1 -- 172.21.15.40:0/73639603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fab1802d1f0 con 0x7fab18107540 2024-03-18T22:40:47.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.510+0000 7fab0ffff640 1 -- 172.21.15.40:0/73639603 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24129 (secure 0 0 0) 0x7fab0809bef0 con 0x7fab18107540 2024-03-18T22:40:47.511 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:47.511 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":89,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:40:46.196741+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:40:47.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.513+0000 7fab1d6b2640 1 -- 172.21.15.40:0/73639603 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab040a2800 msgr2=0x7fab040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:47.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.513+0000 7fab1d6b2640 1 --2- 172.21.15.40:0/73639603 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab040a2800 0x7fab040a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fab10009b50 tx=0x7fab10005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:47.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.513+0000 7fab1d6b2640 1 -- 172.21.15.40:0/73639603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab18107540 msgr2=0x7fab1819f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:47.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.513+0000 7fab1d6b2640 1 --2- 172.21.15.40:0/73639603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab18107540 0x7fab1819f040 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fab0800eb10 tx=0x7fab0800c2d0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.513+0000 7fab1d6b2640 1 -- 172.21.15.40:0/73639603 shutdown_connections 2024-03-18T22:40:47.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.513+0000 7fab1d6b2640 1 --2- 172.21.15.40:0/73639603 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab040a2800 0x7fab040a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.513+0000 7fab1d6b2640 1 --2- 172.21.15.40:0/73639603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab18107540 0x7fab1819f040 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.513+0000 7fab1d6b2640 1 --2- 172.21.15.40:0/73639603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab18104c10 0x7fab18197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.513+0000 7fab1d6b2640 1 --2- 172.21.15.40:0/73639603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab181022e0 0x7fab18197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.513+0000 7fab1d6b2640 1 -- 172.21.15.40:0/73639603 >> 172.21.15.40:0/73639603 conn(0x7fab180fbcb0 msgr2=0x7fab18102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:47.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.513+0000 7fab1d6b2640 1 -- 172.21.15.40:0/73639603 shutdown_connections 2024-03-18T22:40:47.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.513+0000 7fab1d6b2640 1 -- 172.21.15.40:0/73639603 wait complete. 2024-03-18T22:40:47.525 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:40:47.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.627+0000 7f2d87de6640 1 -- 172.21.15.40:0/1829475350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d801022f0 msgr2=0x7f2d80108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:47.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.627+0000 7f2d87de6640 1 --2- 172.21.15.40:0/1829475350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d801022f0 0x7f2d80108b80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009be0 tx=0x7f2d7802bef0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d87de6640 1 -- 172.21.15.40:0/1829475350 shutdown_connections 2024-03-18T22:40:47.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d87de6640 1 --2- 172.21.15.40:0/1829475350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d801022f0 0x7f2d80108b80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d87de6640 1 --2- 172.21.15.40:0/1829475350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d80101930 0x7f2d80101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d87de6640 1 --2- 172.21.15.40:0/1829475350 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d80100740 0x7f2d80100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d87de6640 1 -- 172.21.15.40:0/1829475350 >> 172.21.15.40:0/1829475350 conn(0x7f2d800fbcb0 msgr2=0x7f2d800fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:47.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d87de6640 1 -- 172.21.15.40:0/1829475350 shutdown_connections 2024-03-18T22:40:47.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d87de6640 1 -- 172.21.15.40:0/1829475350 wait complete. 2024-03-18T22:40:47.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d87de6640 1 Processor -- start 2024-03-18T22:40:47.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d87de6640 1 -- start start 2024-03-18T22:40:47.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d87de6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d80100740 0x7f2d80196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d87de6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d80101930 0x7f2d80196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d87de6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d801022f0 0x7f2d8019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d87de6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8010c270 con 0x7f2d80100740 2024-03-18T22:40:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d87de6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8010c0f0 con 0x7f2d80101930 2024-03-18T22:40:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d87de6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8010c3f0 con 0x7f2d801022f0 2024-03-18T22:40:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d85b82640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d80100740 0x7f2d80196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d85b82640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d80100740 0x7f2d80196940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51160/0 (socket says 172.21.15.40:51160) 2024-03-18T22:40:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d85b82640 1 -- 172.21.15.40:0/726399764 learned_addr learned my addr 172.21.15.40:0/726399764 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.628+0000 7f2d85381640 1 --2- 172.21.15.40:0/726399764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d80101930 0x7f2d80196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.629+0000 7f2d86383640 1 --2- 172.21.15.40:0/726399764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d801022f0 0x7f2d8019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.629+0000 7f2d85b82640 1 -- 172.21.15.40:0/726399764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d801022f0 msgr2=0x7f2d8019def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.629+0000 7f2d85b82640 1 --2- 172.21.15.40:0/726399764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d801022f0 0x7f2d8019def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.629+0000 7f2d85b82640 1 -- 172.21.15.40:0/726399764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d80101930 msgr2=0x7f2d80196e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.629+0000 7f2d85b82640 1 --2- 172.21.15.40:0/726399764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d80101930 0x7f2d80196e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.629+0000 7f2d85b82640 1 -- 172.21.15.40:0/726399764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8019e560 con 0x7f2d80100740 2024-03-18T22:40:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.629+0000 7f2d85b82640 1 --2- 172.21.15.40:0/726399764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d80100740 0x7f2d80196940 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00d950 tx=0x7f2d7c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.629+0000 7f2d76ffd640 1 -- 172.21.15.40:0/726399764 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c017070 con 0x7f2d80100740 2024-03-18T22:40:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.629+0000 7f2d87de6640 1 -- 172.21.15.40:0/726399764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8019e850 con 0x7f2d80100740 2024-03-18T22:40:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.629+0000 7f2d76ffd640 1 -- 172.21.15.40:0/726399764 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d7c009c30 con 0x7f2d80100740 2024-03-18T22:40:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.629+0000 7f2d76ffd640 1 -- 172.21.15.40:0/726399764 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c00f620 con 0x7f2d80100740 2024-03-18T22:40:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.629+0000 7f2d87de6640 1 -- 172.21.15.40:0/726399764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d8019ed50 con 0x7f2d80100740 2024-03-18T22:40:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.630+0000 7f2d87de6640 1 -- 172.21.15.40:0/726399764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d80107ab0 con 0x7f2d80100740 2024-03-18T22:40:47.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.631+0000 7f2d76ffd640 1 -- 172.21.15.40:0/726399764 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2d7c01a070 con 0x7f2d80100740 2024-03-18T22:40:47.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.632+0000 7f2d76ffd640 1 --2- 172.21.15.40:0/726399764 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d6c0a2800 0x7f2d6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:47.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.632+0000 7f2d85381640 1 --2- 172.21.15.40:0/726399764 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d6c0a2800 0x7f2d6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:47.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.632+0000 7f2d76ffd640 1 -- 172.21.15.40:0/726399764 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2d7c0ce620 con 0x7f2d80100740 2024-03-18T22:40:47.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.632+0000 7f2d85381640 1 --2- 172.21.15.40:0/726399764 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d6c0a2800 0x7f2d6c0a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2d80197e50 tx=0x7f2d70009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:47.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.636+0000 7f2d76ffd640 1 -- 172.21.15.40:0/726399764 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2d7c009de0 con 0x7f2d80100740 2024-03-18T22:40:47.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.821+0000 7f2d87de6640 1 -- 172.21.15.40:0/726399764 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d80062420 con 0x7f2d80100740 2024-03-18T22:40:47.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.823+0000 7f2d76ffd640 1 -- 172.21.15.40:0/726399764 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24129 (secure 0 0 0) 0x7f2d7c09ae30 con 0x7f2d80100740 2024-03-18T22:40:47.824 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:47.824 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":89,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:40:46.196741+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:40:47.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.825+0000 7f2d87de6640 1 -- 172.21.15.40:0/726399764 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d6c0a2800 msgr2=0x7f2d6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:47.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.825+0000 7f2d87de6640 1 --2- 172.21.15.40:0/726399764 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d6c0a2800 0x7f2d6c0a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2d80197e50 tx=0x7f2d70009260 comp rx=0 tx=0).stop 2024-03-18T22:40:47.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.825+0000 7f2d87de6640 1 -- 172.21.15.40:0/726399764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d80100740 msgr2=0x7f2d80196940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:47.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.825+0000 7f2d87de6640 1 --2- 172.21.15.40:0/726399764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d80100740 0x7f2d80196940 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00d950 tx=0x7f2d7c00de10 comp rx=0 tx=0).stop 2024-03-18T22:40:47.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.825+0000 7f2d87de6640 1 -- 172.21.15.40:0/726399764 shutdown_connections 2024-03-18T22:40:47.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.825+0000 7f2d87de6640 1 --2- 172.21.15.40:0/726399764 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d6c0a2800 0x7f2d6c0a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.825+0000 7f2d87de6640 1 --2- 172.21.15.40:0/726399764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d801022f0 0x7f2d8019def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.825+0000 7f2d87de6640 1 --2- 172.21.15.40:0/726399764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d80101930 0x7f2d80196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.825+0000 7f2d87de6640 1 --2- 172.21.15.40:0/726399764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d80100740 0x7f2d80196940 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.825+0000 7f2d87de6640 1 -- 172.21.15.40:0/726399764 >> 172.21.15.40:0/726399764 conn(0x7f2d800fbcb0 msgr2=0x7f2d80104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:47.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.825+0000 7f2d87de6640 1 -- 172.21.15.40:0/726399764 shutdown_connections 2024-03-18T22:40:47.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.825+0000 7f2d87de6640 1 -- 172.21.15.40:0/726399764 wait complete. 2024-03-18T22:40:47.853 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:40:47.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.954+0000 7fed15659640 1 -- 172.21.15.40:0/3084546309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed10101930 msgr2=0x7fed10101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:47.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.954+0000 7fed15659640 1 --2- 172.21.15.40:0/3084546309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed10101930 0x7fed10101db0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fed04009900 tx=0x7fed0402bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:47.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.954+0000 7fed15659640 1 -- 172.21.15.40:0/3084546309 shutdown_connections 2024-03-18T22:40:47.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.954+0000 7fed15659640 1 --2- 172.21.15.40:0/3084546309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed101022f0 0x7fed10108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.954+0000 7fed15659640 1 --2- 172.21.15.40:0/3084546309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed10101930 0x7fed10101db0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.954+0000 7fed15659640 1 --2- 172.21.15.40:0/3084546309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed10100740 0x7fed10100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.954+0000 7fed15659640 1 -- 172.21.15.40:0/3084546309 >> 172.21.15.40:0/3084546309 conn(0x7fed100fbcb0 msgr2=0x7fed100fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:47.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.954+0000 7fed15659640 1 -- 172.21.15.40:0/3084546309 shutdown_connections 2024-03-18T22:40:47.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.954+0000 7fed15659640 1 -- 172.21.15.40:0/3084546309 wait complete. 2024-03-18T22:40:47.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed15659640 1 Processor -- start 2024-03-18T22:40:47.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed15659640 1 -- start start 2024-03-18T22:40:47.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed15659640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed10100740 0x7fed10197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:47.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed15659640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed10101930 0x7fed10197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:47.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed15659640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed101022f0 0x7fed1019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:47.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed15659640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fed1010c250 con 0x7fed10101930 2024-03-18T22:40:47.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed15659640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fed1010c0d0 con 0x7fed101022f0 2024-03-18T22:40:47.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed15659640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fed1010c3d0 con 0x7fed10100740 2024-03-18T22:40:47.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed0e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed10101930 0x7fed10197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:47.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed0e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed10101930 0x7fed10197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51184/0 (socket says 172.21.15.40:51184) 2024-03-18T22:40:47.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed0e7fc640 1 -- 172.21.15.40:0/651472851 learned_addr learned my addr 172.21.15.40:0/651472851 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:47.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed0f7fe640 1 --2- 172.21.15.40:0/651472851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed101022f0 0x7fed1019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:47.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed0effd640 1 --2- 172.21.15.40:0/651472851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed10100740 0x7fed10197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:47.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed0e7fc640 1 -- 172.21.15.40:0/651472851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed10100740 msgr2=0x7fed10197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:47.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed0e7fc640 1 --2- 172.21.15.40:0/651472851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed10100740 0x7fed10197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed0e7fc640 1 -- 172.21.15.40:0/651472851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed101022f0 msgr2=0x7fed1019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:47.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed0e7fc640 1 --2- 172.21.15.40:0/651472851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed101022f0 0x7fed1019f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:47.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.955+0000 7fed0e7fc640 1 -- 172.21.15.40:0/651472851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1019f690 con 0x7fed10101930 2024-03-18T22:40:47.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.956+0000 7fed0e7fc640 1 --2- 172.21.15.40:0/651472851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed10101930 0x7fed10197fb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fed04006010 tx=0x7fed04004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:47.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.956+0000 7fecf7fff640 1 -- 172.21.15.40:0/651472851 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed04043020 con 0x7fed10101930 2024-03-18T22:40:47.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.956+0000 7fed15659640 1 -- 172.21.15.40:0/651472851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1019f920 con 0x7fed10101930 2024-03-18T22:40:47.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.956+0000 7fecf7fff640 1 -- 172.21.15.40:0/651472851 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed04004440 con 0x7fed10101930 2024-03-18T22:40:47.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.956+0000 7fecf7fff640 1 -- 172.21.15.40:0/651472851 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed04003bf0 con 0x7fed10101930 2024-03-18T22:40:47.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.956+0000 7fed15659640 1 -- 172.21.15.40:0/651472851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed1019fec0 con 0x7fed10101930 2024-03-18T22:40:47.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.957+0000 7fed15659640 1 -- 172.21.15.40:0/651472851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed101a0090 con 0x7fed10101930 2024-03-18T22:40:47.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.958+0000 7fecf7fff640 1 -- 172.21.15.40:0/651472851 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fed04046020 con 0x7fed10101930 2024-03-18T22:40:47.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.958+0000 7fecf7fff640 1 --2- 172.21.15.40:0/651472851 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fecf80a2800 0x7fecf80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:47.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.958+0000 7fed0effd640 1 --2- 172.21.15.40:0/651472851 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fecf80a2800 0x7fecf80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:47.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.958+0000 7fecf7fff640 1 -- 172.21.15.40:0/651472851 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fed040f10e0 con 0x7fed10101930 2024-03-18T22:40:47.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.959+0000 7fed0effd640 1 --2- 172.21.15.40:0/651472851 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fecf80a2800 0x7fecf80a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fed10101790 tx=0x7fed08005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:47.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:47.963+0000 7fecf7fff640 1 -- 172.21.15.40:0/651472851 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fed0404d020 con 0x7fed10101930 2024-03-18T22:40:48.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:47 smithi103 ceph-mon[66377]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T22:40:48.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:47 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/322025845' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-18T22:40:48.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:47 smithi103 ceph-mon[66377]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T22:40:48.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:47 smithi103 ceph-mon[66377]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T22:40:48.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:47 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/923429626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:48.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:47 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/73639603' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:48.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:47 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/726399764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:48.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:47 smithi040 ceph-mon[62643]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T22:40:48.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:47 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/322025845' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-18T22:40:48.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:47 smithi040 ceph-mon[62643]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T22:40:48.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:47 smithi040 ceph-mon[62643]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T22:40:48.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:47 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/923429626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:48.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:47 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/73639603' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:48.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:47 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/726399764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:48.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.188+0000 7fed15659640 1 -- 172.21.15.40:0/651472851 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed101a0260 con 0x7fed10101930 2024-03-18T22:40:48.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.189+0000 7fecf7fff640 1 -- 172.21.15.40:0/651472851 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11812 (secure 0 0 0) 0x7fed040bdee0 con 0x7fed10101930 2024-03-18T22:40:48.190 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:48.190 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14592,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14604,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24388,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:40:46.196914+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24509,"qdb_cluster":[24509]},"id":1}]} 2024-03-18T22:40:48.190 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 16 2024-03-18T22:40:48.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.191+0000 7fed15659640 1 -- 172.21.15.40:0/651472851 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fecf80a2800 msgr2=0x7fecf80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.191+0000 7fed15659640 1 --2- 172.21.15.40:0/651472851 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fecf80a2800 0x7fecf80a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fed10101790 tx=0x7fed08005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:48.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.191+0000 7fed15659640 1 -- 172.21.15.40:0/651472851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed10101930 msgr2=0x7fed10197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.191+0000 7fed15659640 1 --2- 172.21.15.40:0/651472851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed10101930 0x7fed10197fb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fed04006010 tx=0x7fed04004040 comp rx=0 tx=0).stop 2024-03-18T22:40:48.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.192+0000 7fed15659640 1 -- 172.21.15.40:0/651472851 shutdown_connections 2024-03-18T22:40:48.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.192+0000 7fed15659640 1 --2- 172.21.15.40:0/651472851 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fecf80a2800 0x7fecf80a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.192+0000 7fed15659640 1 --2- 172.21.15.40:0/651472851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed101022f0 0x7fed1019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.192+0000 7fed15659640 1 --2- 172.21.15.40:0/651472851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fed10101930 0x7fed10197fb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.192+0000 7fed15659640 1 --2- 172.21.15.40:0/651472851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fed10100740 0x7fed10197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.192+0000 7fed15659640 1 -- 172.21.15.40:0/651472851 >> 172.21.15.40:0/651472851 conn(0x7fed100fbcb0 msgr2=0x7fed100fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:48.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.192+0000 7fed15659640 1 -- 172.21.15.40:0/651472851 shutdown_connections 2024-03-18T22:40:48.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.192+0000 7fed15659640 1 -- 172.21.15.40:0/651472851 wait complete. 2024-03-18T22:40:48.204 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:40:48.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.306+0000 7f18e6953640 1 -- 172.21.15.40:0/540336461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e0102ae0 msgr2=0x7f18e0102f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.306+0000 7f18e6953640 1 --2- 172.21.15.40:0/540336461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e0102ae0 0x7f18e0102f60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f18d0009930 tx=0x7f18d002bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:48.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.306+0000 7f18e6953640 1 -- 172.21.15.40:0/540336461 shutdown_connections 2024-03-18T22:40:48.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.306+0000 7f18e6953640 1 --2- 172.21.15.40:0/540336461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18e01034a0 0x7f18e0109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.306+0000 7f18e6953640 1 --2- 172.21.15.40:0/540336461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e0102ae0 0x7f18e0102f60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.306+0000 7f18e6953640 1 --2- 172.21.15.40:0/540336461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18e00ffb10 0x7f18e00fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.306+0000 7f18e6953640 1 -- 172.21.15.40:0/540336461 >> 172.21.15.40:0/540336461 conn(0x7f18e00fb440 msgr2=0x7f18e00fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:48.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.306+0000 7f18e6953640 1 -- 172.21.15.40:0/540336461 shutdown_connections 2024-03-18T22:40:48.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.306+0000 7f18e6953640 1 -- 172.21.15.40:0/540336461 wait complete. 2024-03-18T22:40:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18e6953640 1 Processor -- start 2024-03-18T22:40:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18e6953640 1 -- start start 2024-03-18T22:40:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18e6953640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e00ffb10 0x7f18e0197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18e6953640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18e0102ae0 0x7f18e0198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18e6953640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18e01034a0 0x7f18e019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18e6953640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f18e010d3a0 con 0x7f18e00ffb10 2024-03-18T22:40:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18e6953640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18e010d220 con 0x7f18e01034a0 2024-03-18T22:40:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18e6953640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f18e010d520 con 0x7f18e0102ae0 2024-03-18T22:40:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18dffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e00ffb10 0x7f18e0197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18dffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e00ffb10 0x7f18e0197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51202/0 (socket says 172.21.15.40:51202) 2024-03-18T22:40:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18dffff640 1 -- 172.21.15.40:0/2521139522 learned_addr learned my addr 172.21.15.40:0/2521139522 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18e4ef0640 1 --2- 172.21.15.40:0/2521139522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18e01034a0 0x7f18e019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18df7fe640 1 --2- 172.21.15.40:0/2521139522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18e0102ae0 0x7f18e0198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18dffff640 1 -- 172.21.15.40:0/2521139522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18e0102ae0 msgr2=0x7f18e0198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18dffff640 1 --2- 172.21.15.40:0/2521139522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18e0102ae0 0x7f18e0198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18dffff640 1 -- 172.21.15.40:0/2521139522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18e01034a0 msgr2=0x7f18e019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18dffff640 1 --2- 172.21.15.40:0/2521139522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18e01034a0 0x7f18e019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.307+0000 7f18dffff640 1 -- 172.21.15.40:0/2521139522 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e019f700 con 0x7f18e00ffb10 2024-03-18T22:40:48.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.308+0000 7f18dffff640 1 --2- 172.21.15.40:0/2521139522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e00ffb10 0x7f18e0197ae0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f18d400e9c0 tx=0x7f18d400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:48.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.308+0000 7f18dd7fa640 1 -- 172.21.15.40:0/2521139522 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d400cde0 con 0x7f18e00ffb10 2024-03-18T22:40:48.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.308+0000 7f18dd7fa640 1 -- 172.21.15.40:0/2521139522 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18d40108e0 con 0x7f18e00ffb10 2024-03-18T22:40:48.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.308+0000 7f18dd7fa640 1 -- 172.21.15.40:0/2521139522 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18d4018990 con 0x7f18e00ffb10 2024-03-18T22:40:48.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.308+0000 7f18e6953640 1 -- 172.21.15.40:0/2521139522 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18e019f9f0 con 0x7f18e00ffb10 2024-03-18T22:40:48.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.308+0000 7f18e6953640 1 -- 172.21.15.40:0/2521139522 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18e019ff60 con 0x7f18e00ffb10 2024-03-18T22:40:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.309+0000 7f18e6953640 1 -- 172.21.15.40:0/2521139522 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18e0108da0 con 0x7f18e00ffb10 2024-03-18T22:40:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.310+0000 7f18dd7fa640 1 -- 172.21.15.40:0/2521139522 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f18d401f020 con 0x7f18e00ffb10 2024-03-18T22:40:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.310+0000 7f18dd7fa640 1 --2- 172.21.15.40:0/2521139522 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18c80a2880 0x7f18c80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.311+0000 7f18df7fe640 1 --2- 172.21.15.40:0/2521139522 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18c80a2880 0x7f18c80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.311+0000 7f18dd7fa640 1 -- 172.21.15.40:0/2521139522 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f18d40ceef0 con 0x7f18e00ffb10 2024-03-18T22:40:48.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.311+0000 7f18df7fe640 1 --2- 172.21.15.40:0/2521139522 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18c80a2880 0x7f18c80a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f18d0006010 tx=0x7f18d002c800 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:48.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.315+0000 7f18dd7fa640 1 -- 172.21.15.40:0/2521139522 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f18d409bcf0 con 0x7f18e00ffb10 2024-03-18T22:40:48.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:47 smithi113 ceph-mon[66775]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T22:40:48.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:47 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/322025845' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-18T22:40:48.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:47 smithi113 ceph-mon[66775]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T22:40:48.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:47 smithi113 ceph-mon[66775]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T22:40:48.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:47 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/923429626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:48.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:47 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/73639603' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:48.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:47 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/726399764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:48.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.498+0000 7f18e6953640 1 -- 172.21.15.40:0/2521139522 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f18e0062420 con 0x7f18e00ffb10 2024-03-18T22:40:48.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.499+0000 7f18dd7fa640 1 -- 172.21.15.40:0/2521139522 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24129 (secure 0 0 0) 0x7f18d409b5f0 con 0x7f18e00ffb10 2024-03-18T22:40:48.500 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:48.501 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":89,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:40:46.196741+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:40:48.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.501+0000 7f18e6953640 1 -- 172.21.15.40:0/2521139522 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18c80a2880 msgr2=0x7f18c80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.501+0000 7f18e6953640 1 --2- 172.21.15.40:0/2521139522 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18c80a2880 0x7f18c80a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f18d0006010 tx=0x7f18d002c800 comp rx=0 tx=0).stop 2024-03-18T22:40:48.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.501+0000 7f18e6953640 1 -- 172.21.15.40:0/2521139522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e00ffb10 msgr2=0x7f18e0197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.501+0000 7f18e6953640 1 --2- 172.21.15.40:0/2521139522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e00ffb10 0x7f18e0197ae0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f18d400e9c0 tx=0x7f18d400ee80 comp rx=0 tx=0).stop 2024-03-18T22:40:48.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.501+0000 7f18e6953640 1 -- 172.21.15.40:0/2521139522 shutdown_connections 2024-03-18T22:40:48.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.501+0000 7f18e6953640 1 --2- 172.21.15.40:0/2521139522 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18c80a2880 0x7f18c80a4d40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.501+0000 7f18e6953640 1 --2- 172.21.15.40:0/2521139522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18e01034a0 0x7f18e019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.501+0000 7f18e6953640 1 --2- 172.21.15.40:0/2521139522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18e0102ae0 0x7f18e0198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.502+0000 7f18e6953640 1 --2- 172.21.15.40:0/2521139522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18e00ffb10 0x7f18e0197ae0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.502+0000 7f18e6953640 1 -- 172.21.15.40:0/2521139522 >> 172.21.15.40:0/2521139522 conn(0x7f18e00fb440 msgr2=0x7f18e0105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:48.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.502+0000 7f18e6953640 1 -- 172.21.15.40:0/2521139522 shutdown_connections 2024-03-18T22:40:48.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.502+0000 7f18e6953640 1 -- 172.21.15.40:0/2521139522 wait complete. 2024-03-18T22:40:48.514 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T22:40:41.572773+0000', 'modified': '2024-03-18T22:40:46.196914+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24509}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.103:6839/2609728202', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6838', 'nonce': 2609728202}, {'type': 'v1', 'addr': '172.21.15.103:6839', 'nonce': 2609728202}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24509, 'qdb_cluster': [24509]} 2024-03-18T22:40:48.514 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T22:40:48.514 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T22:40:48.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.620+0000 7f9e1d7b3640 1 -- 172.21.15.40:0/81374737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e18100270 msgr2=0x7f9e18109e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.620+0000 7f9e1d7b3640 1 --2- 172.21.15.40:0/81374737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e18100270 0x7f9e18109e20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009ca0 tx=0x7f9e0802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.620+0000 7f9e1d7b3640 1 -- 172.21.15.40:0/81374737 shutdown_connections 2024-03-18T22:40:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.620+0000 7f9e1d7b3640 1 --2- 172.21.15.40:0/81374737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e18100270 0x7f9e18109e20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.620+0000 7f9e1d7b3640 1 --2- 172.21.15.40:0/81374737 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e180ff8b0 0x7f9e180ffd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.620+0000 7f9e1d7b3640 1 --2- 172.21.15.40:0/81374737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e18102e40 0x7f9e18103260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.620+0000 7f9e1d7b3640 1 -- 172.21.15.40:0/81374737 >> 172.21.15.40:0/81374737 conn(0x7f9e180fb450 msgr2=0x7f9e180fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.620+0000 7f9e1d7b3640 1 -- 172.21.15.40:0/81374737 shutdown_connections 2024-03-18T22:40:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.620+0000 7f9e1d7b3640 1 -- 172.21.15.40:0/81374737 wait complete. 2024-03-18T22:40:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.620+0000 7f9e1d7b3640 1 Processor -- start 2024-03-18T22:40:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.620+0000 7f9e1d7b3640 1 -- start start 2024-03-18T22:40:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e1d7b3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e180ff8b0 0x7f9e18197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e1d7b3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e18100270 0x7f9e18197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e1d7b3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e18102e40 0x7f9e1819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e1d7b3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1810d370 con 0x7f9e18102e40 2024-03-18T22:40:48.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e1d7b3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1810d1f0 con 0x7f9e180ff8b0 2024-03-18T22:40:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e1d7b3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1810d4f0 con 0x7f9e18100270 2024-03-18T22:40:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e177fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e18102e40 0x7f9e1819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e177fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e18102e40 0x7f9e1819efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37530/0 (socket says 172.21.15.40:37530) 2024-03-18T22:40:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e177fe640 1 -- 172.21.15.40:0/1318159328 learned_addr learned my addr 172.21.15.40:0/1318159328 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e167fc640 1 --2- 172.21.15.40:0/1318159328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e18100270 0x7f9e18197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e16ffd640 1 --2- 172.21.15.40:0/1318159328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e180ff8b0 0x7f9e18197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e177fe640 1 -- 172.21.15.40:0/1318159328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e18100270 msgr2=0x7f9e18197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e177fe640 1 --2- 172.21.15.40:0/1318159328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e18100270 0x7f9e18197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e177fe640 1 -- 172.21.15.40:0/1318159328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e180ff8b0 msgr2=0x7f9e18197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e177fe640 1 --2- 172.21.15.40:0/1318159328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e180ff8b0 0x7f9e18197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e177fe640 1 -- 172.21.15.40:0/1318159328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1819f630 con 0x7f9e18102e40 2024-03-18T22:40:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e177fe640 1 --2- 172.21.15.40:0/1318159328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e18102e40 0x7f9e1819efc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9e0802c700 tx=0x7f9e080328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9dfffff640 1 -- 172.21.15.40:0/1318159328 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08041020 con 0x7f9e18102e40 2024-03-18T22:40:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9dfffff640 1 -- 172.21.15.40:0/1318159328 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e08003e20 con 0x7f9e18102e40 2024-03-18T22:40:48.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e1d7b3640 1 -- 172.21.15.40:0/1318159328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e1819f8c0 con 0x7f9e18102e40 2024-03-18T22:40:48.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9dfffff640 1 -- 172.21.15.40:0/1318159328 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08039040 con 0x7f9e18102e40 2024-03-18T22:40:48.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.621+0000 7f9e1d7b3640 1 -- 172.21.15.40:0/1318159328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e1819fe00 con 0x7f9e18102e40 2024-03-18T22:40:48.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.624+0000 7f9dfffff640 1 -- 172.21.15.40:0/1318159328 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9e08044020 con 0x7f9e18102e40 2024-03-18T22:40:48.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.624+0000 7f9e20399740 1 Processor -- start 2024-03-18T22:40:48.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.624+0000 7f9e20399740 1 -- start start 2024-03-18T22:40:48.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.624+0000 7f9e20399740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246eeff0 0x5604246ef410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.624+0000 7f9e20399740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x5604246f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.624+0000 7f9e20399740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246f4b50 0x5604246f6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.624+0000 7f9e20399740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560424653de0 con 0x5604246ef9e0 2024-03-18T22:40:48.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.624+0000 7f9e20399740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5604245fcc80 con 0x5604246eeff0 2024-03-18T22:40:48.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.624+0000 7f9e20399740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56042464fe80 con 0x5604246f4b50 2024-03-18T22:40:48.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.624+0000 7f9dfffff640 1 --2- 172.21.15.40:0/1318159328 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9e000a2880 0x7f9e000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.624+0000 7f9e16ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x5604246f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.624+0000 7f9e16ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x5604246f44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37542/0 (socket says 172.21.15.40:37542) 2024-03-18T22:40:48.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.624+0000 7f9e16ffd640 1 -- 172.21.15.40:0/2949886287 learned_addr learned my addr 172.21.15.40:0/2949886287 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:48.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.624+0000 7f9e177fe640 1 --2- 172.21.15.40:0/1318159328 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9e000a2880 0x7f9e000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.624+0000 7f9e167fc640 1 --2- 172.21.15.40:0/2949886287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246eeff0 0x5604246ef410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.625+0000 7f9e167fc640 1 --2- 172.21.15.40:0/2949886287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246f4b50 0x5604246f6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.625+0000 7f9dfffff640 1 -- 172.21.15.40:0/1318159328 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9e080f0e30 con 0x7f9e18102e40 2024-03-18T22:40:48.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.625+0000 7f9e16ffd640 1 -- 172.21.15.40:0/2949886287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246f4b50 msgr2=0x5604246f6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:40:48.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.625+0000 7f9e16ffd640 1 --2- 172.21.15.40:0/2949886287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246f4b50 0x5604246f6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.625+0000 7f9e16ffd640 1 -- 172.21.15.40:0/2949886287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246eeff0 msgr2=0x5604246ef410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:40:48.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.625+0000 7f9e16ffd640 1 --2- 172.21.15.40:0/2949886287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246eeff0 0x5604246ef410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.625+0000 7f9e16ffd640 1 -- 172.21.15.40:0/2949886287 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56042460efb0 con 0x5604246ef9e0 2024-03-18T22:40:48.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.625+0000 7f9e167fc640 1 --2- 172.21.15.40:0/2949886287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246f4b50 0x5604246f6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:40:48.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.625+0000 7f9e177fe640 1 --2- 172.21.15.40:0/1318159328 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9e000a2880 0x7f9e000a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9e08032380 tx=0x7f9e080902c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:48.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.625+0000 7f9e16ffd640 1 --2- 172.21.15.40:0/2949886287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x5604246f44e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9e10009c20 tx=0x7f9e10011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=752aa740b1ec5c38 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:48.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.625+0000 7f9dfd7fa640 1 -- 172.21.15.40:0/2949886287 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e10004030 con 0x5604246ef9e0 2024-03-18T22:40:48.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.625+0000 7f9dfd7fa640 1 -- 172.21.15.40:0/2949886287 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e100035f0 con 0x5604246ef9e0 2024-03-18T22:40:48.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.625+0000 7f9dfd7fa640 1 -- 172.21.15.40:0/2949886287 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e10019620 con 0x5604246ef9e0 2024-03-18T22:40:48.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 -- 172.21.15.40:0/2949886287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 msgr2=0x5604246f44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 --2- 172.21.15.40:0/2949886287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x5604246f44e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9e10009c20 tx=0x7f9e10011810 comp rx=0 tx=0).stop 2024-03-18T22:40:48.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 -- 172.21.15.40:0/2949886287 shutdown_connections 2024-03-18T22:40:48.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 --2- 172.21.15.40:0/2949886287 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246f4b50 0x5604246f6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 --2- 172.21.15.40:0/2949886287 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x5604246f44e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 --2- 172.21.15.40:0/2949886287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246eeff0 0x5604246ef410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 -- 172.21.15.40:0/2949886287 >> 172.21.15.40:0/2949886287 conn(0x5604246ea560 msgr2=0x5604246ea960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:48.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 -- 172.21.15.40:0/2949886287 shutdown_connections 2024-03-18T22:40:48.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 -- 172.21.15.40:0/2949886287 wait complete. 2024-03-18T22:40:48.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 -- start start 2024-03-18T22:40:48.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 Processor -- start 2024-03-18T22:40:48.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246eeff0 0x5604246f9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x5604246f9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 0x560424700a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5604246f7b00 con 0x5604246ef9e0 2024-03-18T22:40:48.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56042460ffc0 con 0x5604246f4b50 2024-03-18T22:40:48.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5604246f7c80 con 0x5604246eeff0 2024-03-18T22:40:48.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e16ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x5604246f9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e20399740 10 client.0 _ll_register_callbacks cb 0x56042464ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:40:48.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e16ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x5604246f9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37552/0 (socket says 172.21.15.40:37552) 2024-03-18T22:40:48.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.626+0000 7f9e16ffd640 1 -- 172.21.15.40:0/558520533 learned_addr learned my addr 172.21.15.40:0/558520533 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:48.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9e167fc640 1 --2- 172.21.15.40:0/558520533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 0x560424700a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9e167fc640 1 --2- 172.21.15.40:0/558520533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246eeff0 0x5604246f9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9e16ffd640 1 -- 172.21.15.40:0/558520533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246eeff0 msgr2=0x5604246f9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:40:48.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9e16ffd640 1 --2- 172.21.15.40:0/558520533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246eeff0 0x5604246f9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9e16ffd640 1 -- 172.21.15.40:0/558520533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 msgr2=0x560424700a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:40:48.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9e16ffd640 1 --2- 172.21.15.40:0/558520533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 0x560424700a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9e16ffd640 1 -- 172.21.15.40:0/558520533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604243682d0 con 0x5604246ef9e0 2024-03-18T22:40:48.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9e167fc640 1 --2- 172.21.15.40:0/558520533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 0x560424700a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:40:48.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9e16ffd640 1 --2- 172.21.15.40:0/558520533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x5604246f9a50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9e10011d10 tx=0x7f9e100042b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2fec2903b374621 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:48.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9dfaff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:40:48.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9dfaff5640 1 -- 172.21.15.40:0/558520533 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e10028020 con 0x5604246ef9e0 2024-03-18T22:40:48.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9dfaff5640 1 -- 172.21.15.40:0/558520533 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9e10005040 con 0x5604246ef9e0 2024-03-18T22:40:48.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9e20399740 1 -- 172.21.15.40:0/558520533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e100035f0 con 0x5604246ef9e0 2024-03-18T22:40:48.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9dfaff5640 1 -- 172.21.15.40:0/558520533 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f9e10033050 con 0x5604246ef9e0 2024-03-18T22:40:48.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9e20399740 10 client.14721 fetch_fsmap learned FSMap version 16 2024-03-18T22:40:48.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9e20399740 1 -- 172.21.15.40:0/558520533 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x5604244326b0 con 0x5604246ef9e0 2024-03-18T22:40:48.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.627+0000 7f9dfaff5640 1 -- 172.21.15.40:0/558520533 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e100035f0 con 0x5604246ef9e0 2024-03-18T22:40:48.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.628+0000 7f9dfaff5640 1 -- 172.21.15.40:0/558520533 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7f9e10018070 con 0x5604246ef9e0 2024-03-18T22:40:48.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.628+0000 7f9e20399740 10 client.14721 fetch_fsmap finished waiting for FSMap version 16 2024-03-18T22:40:48.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.628+0000 7f9e20399740 10 client.14721 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T22:40:48.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.628+0000 7f9e20399740 20 client.14721 populate_metadata read hostname 'smithi040' 2024-03-18T22:40:48.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.628+0000 7f9e20399740 1 --2- 172.21.15.40:0/558520533 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560424706020 0x560424707490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.628+0000 7f9e20399740 4 client.14721 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:40:48.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.628+0000 7f9e20399740 1 -- 172.21.15.40:0/558520533 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560424661320 con 0x560424706020 2024-03-18T22:40:48.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.628+0000 7f9e167fc640 1 --2- 172.21.15.40:0/558520533 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560424706020 0x560424707490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.628+0000 7f9dfaff5640 10 client.14721 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:40:48.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.628+0000 7f9e167fc640 1 --2- 172.21.15.40:0/558520533 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560424706020 0x560424707490 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12f9b564440131ab server_cookie=c7f9df7fab9c36f8 in_seq=0 out_seq=0 2024-03-18T22:40:48.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.629+0000 7f9dfaff5640 1 -- 172.21.15.40:0/558520533 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560424661320 con 0x560424706020 2024-03-18T22:40:48.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.629+0000 7f9dfaff5640 10 client.14721 handle_command_reply: tid=0 2024-03-18T22:40:48.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.629+0000 7f9e20399740 1 client.14721 shutdown 2024-03-18T22:40:48.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.629+0000 7f9e20399740 1 -- 172.21.15.40:0/558520533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 msgr2=0x5604246f9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.629+0000 7f9e20399740 1 --2- 172.21.15.40:0/558520533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x5604246f9a50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9e10011d10 tx=0x7f9e100042b0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.629+0000 7f9e20399740 1 -- 172.21.15.40:0/558520533 shutdown_connections 2024-03-18T22:40:48.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.629+0000 7f9e20399740 1 --2- 172.21.15.40:0/558520533 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560424706020 0x560424707490 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.629+0000 7f9e20399740 1 --2- 172.21.15.40:0/558520533 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 0x560424700a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.629+0000 7f9dfaff5640 0 client.14721 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:40:48.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.629+0000 7f9e20399740 1 --2- 172.21.15.40:0/558520533 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x5604246f9a50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.629+0000 7f9e20399740 1 --2- 172.21.15.40:0/558520533 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246eeff0 0x5604246f9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.629+0000 7f9e20399740 1 -- 172.21.15.40:0/558520533 >> 172.21.15.40:0/558520533 conn(0x5604246ea560 msgr2=0x5604246e97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:48.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.629+0000 7f9e20399740 1 -- 172.21.15.40:0/558520533 shutdown_connections 2024-03-18T22:40:48.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.629+0000 7f9e20399740 1 -- 172.21.15.40:0/558520533 wait complete. 2024-03-18T22:40:48.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.629+0000 7f9e20399740 20 client.14721 trim_cache size 0 max 16384 2024-03-18T22:40:48.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.647+0000 7f9e20399740 1 Processor -- start 2024-03-18T22:40:48.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.647+0000 7f9e20399740 1 -- start start 2024-03-18T22:40:48.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.647+0000 7f9e20399740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246eeff0 0x5604246ea940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.647+0000 7f9e20399740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x560424704fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.647+0000 7f9e20399740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 0x5604246e8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.647+0000 7f9e20399740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5604247057d0 con 0x5604246ef9e0 2024-03-18T22:40:48.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.647+0000 7f9e20399740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560424705ea0 con 0x5604246f4b50 2024-03-18T22:40:48.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.647+0000 7f9e20399740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560424706570 con 0x5604246eeff0 2024-03-18T22:40:48.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.648+0000 7f9e16ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x560424704fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.648+0000 7f9e167fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246eeff0 0x5604246ea940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.648+0000 7f9e16ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x560424704fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37562/0 (socket says 172.21.15.40:37562) 2024-03-18T22:40:48.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.648+0000 7f9e16ffd640 1 -- 172.21.15.40:0/162887961 learned_addr learned my addr 172.21.15.40:0/162887961 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:48.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.648+0000 7f9e167fc640 1 --2- 172.21.15.40:0/162887961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 0x5604246e8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.648+0000 7f9e16ffd640 1 -- 172.21.15.40:0/162887961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246eeff0 msgr2=0x5604246ea940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:40:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.648+0000 7f9e16ffd640 1 --2- 172.21.15.40:0/162887961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246eeff0 0x5604246ea940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.648+0000 7f9e16ffd640 1 -- 172.21.15.40:0/162887961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 msgr2=0x5604246e8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:40:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.648+0000 7f9e16ffd640 1 --2- 172.21.15.40:0/162887961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 0x5604246e8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.648+0000 7f9e16ffd640 1 -- 172.21.15.40:0/162887961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604246ea100 con 0x5604246ef9e0 2024-03-18T22:40:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.648+0000 7f9e16ffd640 1 --2- 172.21.15.40:0/162887961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x560424704fe0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9e10004de0 tx=0x7f9e10009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eeb12301a443004 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.648+0000 7f9dfcff9640 1 -- 172.21.15.40:0/162887961 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e10028040 con 0x5604246ef9e0 2024-03-18T22:40:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.648+0000 7f9dfcff9640 1 -- 172.21.15.40:0/162887961 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e100035f0 con 0x5604246ef9e0 2024-03-18T22:40:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.648+0000 7f9dfcff9640 1 -- 172.21.15.40:0/162887961 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1001f040 con 0x5604246ef9e0 2024-03-18T22:40:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 -- 172.21.15.40:0/162887961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 msgr2=0x560424704fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 --2- 172.21.15.40:0/162887961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x560424704fe0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9e10004de0 tx=0x7f9e10009540 comp rx=0 tx=0).stop 2024-03-18T22:40:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 -- 172.21.15.40:0/162887961 shutdown_connections 2024-03-18T22:40:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 --2- 172.21.15.40:0/162887961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 0x5604246e8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 --2- 172.21.15.40:0/162887961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x560424704fe0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 --2- 172.21.15.40:0/162887961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246eeff0 0x5604246ea940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 -- 172.21.15.40:0/162887961 >> 172.21.15.40:0/162887961 conn(0x5604246ea560 msgr2=0x5604246f80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 -- 172.21.15.40:0/162887961 shutdown_connections 2024-03-18T22:40:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 -- 172.21.15.40:0/162887961 wait complete. 2024-03-18T22:40:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 -- start start 2024-03-18T22:40:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 Processor -- start 2024-03-18T22:40:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246eeff0 0x560424769d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x56042476c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 0x56042476e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560424707310 con 0x5604246ef9e0 2024-03-18T22:40:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560424706c40 con 0x5604246f4b50 2024-03-18T22:40:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5604247079e0 con 0x5604246eeff0 2024-03-18T22:40:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e20399740 10 client.0 _ll_register_callbacks cb 0x7f9e0c00bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:40:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e16ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x56042476c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e16ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x56042476c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37564/0 (socket says 172.21.15.40:37564) 2024-03-18T22:40:48.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e16ffd640 1 -- 172.21.15.40:0/1068355266 learned_addr learned my addr 172.21.15.40:0/1068355266 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:48.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e167fc640 1 --2- 172.21.15.40:0/1068355266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246eeff0 0x560424769d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.649+0000 7f9e167fc640 1 --2- 172.21.15.40:0/1068355266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 0x56042476e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9e16ffd640 1 -- 172.21.15.40:0/1068355266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246eeff0 msgr2=0x560424769d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:40:48.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9e16ffd640 1 --2- 172.21.15.40:0/1068355266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246eeff0 0x560424769d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9e16ffd640 1 -- 172.21.15.40:0/1068355266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 msgr2=0x56042476e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:40:48.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9e16ffd640 1 --2- 172.21.15.40:0/1068355266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 0x56042476e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9e16ffd640 1 -- 172.21.15.40:0/1068355266 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56042476ede0 con 0x5604246ef9e0 2024-03-18T22:40:48.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9e16ffd640 1 --2- 172.21.15.40:0/1068355266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x56042476c2d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9e10011850 tx=0x7f9e100098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a6268193bcb1ec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:48.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9dfaff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:40:48.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9dfaff5640 1 -- 172.21.15.40:0/1068355266 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e10003e60 con 0x5604246ef9e0 2024-03-18T22:40:48.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9dfaff5640 1 -- 172.21.15.40:0/1068355266 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9e100044c0 con 0x5604246ef9e0 2024-03-18T22:40:48.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9e20399740 1 -- 172.21.15.40:0/1068355266 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e100035f0 con 0x5604246ef9e0 2024-03-18T22:40:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9dfaff5640 1 -- 172.21.15.40:0/1068355266 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f9e10045050 con 0x5604246ef9e0 2024-03-18T22:40:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9e20399740 10 client.14727 fetch_fsmap learned FSMap version 16 2024-03-18T22:40:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9e20399740 1 -- 172.21.15.40:0/1068355266 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x5604247700c0 con 0x5604246ef9e0 2024-03-18T22:40:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9dfaff5640 1 -- 172.21.15.40:0/1068355266 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e100035f0 con 0x5604246ef9e0 2024-03-18T22:40:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9dfaff5640 1 -- 172.21.15.40:0/1068355266 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7f9e10018070 con 0x5604246ef9e0 2024-03-18T22:40:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9e20399740 10 client.14727 fetch_fsmap finished waiting for FSMap version 16 2024-03-18T22:40:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9e20399740 10 client.14727 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T22:40:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9e20399740 20 client.14727 populate_metadata read hostname 'smithi040' 2024-03-18T22:40:48.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9e20399740 1 --2- 172.21.15.40:0/1068355266 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560424774930 0x560424773640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9e20399740 4 client.14727 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T22:40:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.650+0000 7f9e20399740 1 -- 172.21.15.40:0/1068355266 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "status"}) -- 0x56042465b670 con 0x560424774930 2024-03-18T22:40:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9e167fc640 1 --2- 172.21.15.40:0/1068355266 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560424774930 0x560424773640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9dfaff5640 10 client.14727 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:40:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9e167fc640 1 --2- 172.21.15.40:0/1068355266 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560424774930 0x560424773640 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f23e7346fb0e7224 server_cookie=b3cd5a380e8981cf in_seq=0 out_seq=0 2024-03-18T22:40:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9dfaff5640 1 -- 172.21.15.40:0/1068355266 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56042465b670 con 0x560424774930 2024-03-18T22:40:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9dfaff5640 10 client.14727 handle_command_reply: tid=0 2024-03-18T22:40:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9e20399740 1 client.14727 shutdown 2024-03-18T22:40:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9e20399740 1 -- 172.21.15.40:0/1068355266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 msgr2=0x56042476c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9e20399740 1 --2- 172.21.15.40:0/1068355266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x56042476c2d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9e10011850 tx=0x7f9e100098e0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9e20399740 1 -- 172.21.15.40:0/1068355266 shutdown_connections 2024-03-18T22:40:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9e20399740 1 --2- 172.21.15.40:0/1068355266 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560424774930 0x560424773640 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9e20399740 1 --2- 172.21.15.40:0/1068355266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604246f4b50 0x56042476e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9e20399740 1 --2- 172.21.15.40:0/1068355266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604246ef9e0 0x56042476c2d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9dfaff5640 0 client.14727 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:40:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9e20399740 1 --2- 172.21.15.40:0/1068355266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604246eeff0 0x560424769d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9e20399740 1 -- 172.21.15.40:0/1068355266 >> 172.21.15.40:0/1068355266 conn(0x5604246ea560 msgr2=0x560424704fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9e20399740 1 -- 172.21.15.40:0/1068355266 shutdown_connections 2024-03-18T22:40:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9e20399740 1 -- 172.21.15.40:0/1068355266 wait complete. 2024-03-18T22:40:48.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.651+0000 7f9e20399740 20 client.14727 trim_cache size 0 max 16384 2024-03-18T22:40:48.661 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:40:48.661 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "410ef414-e577-11ee-95c9-87774f69a715", 2024-03-18T22:40:48.661 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-18T22:40:48.661 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24509, 2024-03-18T22:40:48.661 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-18T22:40:48.661 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-18T22:40:48.661 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-18T22:40:48.661 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 6.4527322030000001, 2024-03-18T22:40:48.661 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 16, 2024-03-18T22:40:48.661 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 89, 2024-03-18T22:40:48.661 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 88, 2024-03-18T22:40:48.662 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 10.935592227000001 2024-03-18T22:40:48.662 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:40:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.652+0000 7f9e1d7b3640 1 -- 172.21.15.40:0/1318159328 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9e000a2880 msgr2=0x7f9e000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.652+0000 7f9e1d7b3640 1 --2- 172.21.15.40:0/1318159328 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9e000a2880 0x7f9e000a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9e08032380 tx=0x7f9e080902c0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.652+0000 7f9e1d7b3640 1 -- 172.21.15.40:0/1318159328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e18102e40 msgr2=0x7f9e1819efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.652+0000 7f9e1d7b3640 1 --2- 172.21.15.40:0/1318159328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e18102e40 0x7f9e1819efc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9e0802c700 tx=0x7f9e080328a0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.652+0000 7f9e1d7b3640 1 -- 172.21.15.40:0/1318159328 shutdown_connections 2024-03-18T22:40:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.652+0000 7f9e1d7b3640 1 --2- 172.21.15.40:0/1318159328 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9e000a2880 0x7f9e000a4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.652+0000 7f9e1d7b3640 1 --2- 172.21.15.40:0/1318159328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e18102e40 0x7f9e1819efc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.652+0000 7f9e1d7b3640 1 --2- 172.21.15.40:0/1318159328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e18100270 0x7f9e18197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.652+0000 7f9e1d7b3640 1 --2- 172.21.15.40:0/1318159328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e180ff8b0 0x7f9e18197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.652+0000 7f9e1d7b3640 1 -- 172.21.15.40:0/1318159328 >> 172.21.15.40:0/1318159328 conn(0x7f9e180fb450 msgr2=0x7f9e18105ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.652+0000 7f9e1d7b3640 1 -- 172.21.15.40:0/1318159328 shutdown_connections 2024-03-18T22:40:48.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:48.652+0000 7f9e1d7b3640 1 -- 172.21.15.40:0/1318159328 wait complete. 2024-03-18T22:40:48.665 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T22:40:48.665 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T22:40:48.666 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-18T22:40:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:40:49.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:48 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/651472851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:49.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:48 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2521139522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:49.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:48 smithi103 ceph-mon[66377]: pgmap v292: 129 pgs: 6 creating+peering, 119 active+clean, 4 unknown; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-03-18T22:40:49.124 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:48 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/651472851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:48 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2521139522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:49.125 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:48 smithi040 ceph-mon[62643]: pgmap v292: 129 pgs: 6 creating+peering, 119 active+clean, 4 unknown; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-03-18T22:40:49.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:48 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/651472851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:40:49.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:48 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2521139522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:49.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:48 smithi113 ceph-mon[66775]: pgmap v292: 129 pgs: 6 creating+peering, 119 active+clean, 4 unknown; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-03-18T22:40:50.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:49 smithi103 ceph-mon[66377]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T22:40:50.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:49 smithi103 ceph-mon[66377]: Cluster is now healthy 2024-03-18T22:40:50.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:49 smithi040 ceph-mon[62643]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T22:40:50.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:49 smithi040 ceph-mon[62643]: Cluster is now healthy 2024-03-18T22:40:50.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:49 smithi113 ceph-mon[66775]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T22:40:50.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:49 smithi113 ceph-mon[66775]: Cluster is now healthy 2024-03-18T22:40:50.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.576+0000 7fac2cd11700 1 -- 172.21.15.40:0/2756412970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac28100900 msgr2=0x7fac28100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:50.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.576+0000 7fac2cd11700 1 --2- 172.21.15.40:0/2756412970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac28100900 0x7fac28100d80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fac10009990 tx=0x7fac10009ca0 comp rx=0 tx=0).stop 2024-03-18T22:40:50.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.576+0000 7fac2cd11700 1 -- 172.21.15.40:0/2756412970 shutdown_connections 2024-03-18T22:40:50.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.576+0000 7fac2cd11700 1 --2- 172.21.15.40:0/2756412970 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac281012c0 0x7fac2810f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:50.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.576+0000 7fac2cd11700 1 --2- 172.21.15.40:0/2756412970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac28107380 0x7fac28107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:50.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.576+0000 7fac2cd11700 1 --2- 172.21.15.40:0/2756412970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac28100900 0x7fac28100d80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:50.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.576+0000 7fac2cd11700 1 -- 172.21.15.40:0/2756412970 >> 172.21.15.40:0/2756412970 conn(0x7fac28077810 msgr2=0x7fac280fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:50.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.576+0000 7fac2cd11700 1 -- 172.21.15.40:0/2756412970 shutdown_connections 2024-03-18T22:40:50.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.576+0000 7fac2cd11700 1 -- 172.21.15.40:0/2756412970 wait complete. 2024-03-18T22:40:50.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.577+0000 7fac2cd11700 1 Processor -- start 2024-03-18T22:40:50.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.577+0000 7fac2cd11700 1 -- start start 2024-03-18T22:40:50.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.577+0000 7fac2cd11700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac28100900 0x7fac2819e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:50.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.577+0000 7fac2cd11700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac281012c0 0x7fac2819ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:50.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.577+0000 7fac2cd11700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac28107380 0x7fac281988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:50.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.577+0000 7fac2cd11700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fac281125f0 con 0x7fac281012c0 2024-03-18T22:40:50.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.577+0000 7fac2cd11700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fac28112470 con 0x7fac28107380 2024-03-18T22:40:50.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.577+0000 7fac2cd11700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fac28112770 con 0x7fac28100900 2024-03-18T22:40:50.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.577+0000 7fac25d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac281012c0 0x7fac2819ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:50.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.578+0000 7fac25d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac281012c0 0x7fac2819ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37580/0 (socket says 172.21.15.40:37580) 2024-03-18T22:40:50.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.578+0000 7fac25d9b700 1 -- 172.21.15.40:0/2121198274 learned_addr learned my addr 172.21.15.40:0/2121198274 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:50.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.578+0000 7fac26d9d700 1 --2- 172.21.15.40:0/2121198274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac28107380 0x7fac281988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:50.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.578+0000 7fac2659c700 1 --2- 172.21.15.40:0/2121198274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac28100900 0x7fac2819e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:50.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.578+0000 7fac26d9d700 1 -- 172.21.15.40:0/2121198274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac28100900 msgr2=0x7fac2819e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:50.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.578+0000 7fac26d9d700 1 --2- 172.21.15.40:0/2121198274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac28100900 0x7fac2819e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:50.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.578+0000 7fac26d9d700 1 -- 172.21.15.40:0/2121198274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac281012c0 msgr2=0x7fac2819ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:50.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.578+0000 7fac26d9d700 1 --2- 172.21.15.40:0/2121198274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac281012c0 0x7fac2819ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:50.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.578+0000 7fac26d9d700 1 -- 172.21.15.40:0/2121198274 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac28199030 con 0x7fac28107380 2024-03-18T22:40:50.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.578+0000 7fac26d9d700 1 --2- 172.21.15.40:0/2121198274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac28107380 0x7fac281988d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fac1800c940 tx=0x7fac1800cc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:50.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.578+0000 7fac1f7fe700 1 -- 172.21.15.40:0/2121198274 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac18004230 con 0x7fac28107380 2024-03-18T22:40:50.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.578+0000 7fac1f7fe700 1 -- 172.21.15.40:0/2121198274 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac180043d0 con 0x7fac28107380 2024-03-18T22:40:50.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.578+0000 7fac2cd11700 1 -- 172.21.15.40:0/2121198274 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac28199320 con 0x7fac28107380 2024-03-18T22:40:50.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.578+0000 7fac1f7fe700 1 -- 172.21.15.40:0/2121198274 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac18003b10 con 0x7fac28107380 2024-03-18T22:40:50.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.578+0000 7fac2cd11700 1 -- 172.21.15.40:0/2121198274 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac281a55e0 con 0x7fac28107380 2024-03-18T22:40:50.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.579+0000 7fac2cd11700 1 -- 172.21.15.40:0/2121198274 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac28068fd0 con 0x7fac28107380 2024-03-18T22:40:50.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.580+0000 7fac1f7fe700 1 -- 172.21.15.40:0/2121198274 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fac18003d30 con 0x7fac28107380 2024-03-18T22:40:50.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.581+0000 7fac1f7fe700 1 --2- 172.21.15.40:0/2121198274 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fac140a2890 0x7fac140a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:50.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.581+0000 7fac2659c700 1 --2- 172.21.15.40:0/2121198274 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fac140a2890 0x7fac140a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:50.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.581+0000 7fac1f7fe700 1 -- 172.21.15.40:0/2121198274 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fac180ceb80 con 0x7fac28107380 2024-03-18T22:40:50.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.581+0000 7fac2659c700 1 --2- 172.21.15.40:0/2121198274 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fac140a2890 0x7fac140a4d50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fac28199a10 tx=0x7fac2000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:50.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.585+0000 7fac1f7fe700 1 -- 172.21.15.40:0/2121198274 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fac18014070 con 0x7fac28107380 2024-03-18T22:40:50.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.843+0000 7fac2cd11700 1 -- 172.21.15.40:0/2121198274 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fac28199e00 con 0x7fac28107380 2024-03-18T22:40:50.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.843+0000 7fac1f7fe700 1 -- 172.21.15.40:0/2121198274 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+88 (secure 0 0 0) 0x7fac1809b980 con 0x7fac28107380 2024-03-18T22:40:50.844 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T22:40:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.846+0000 7fac2cd11700 1 -- 172.21.15.40:0/2121198274 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fac140a2890 msgr2=0x7fac140a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.846+0000 7fac2cd11700 1 --2- 172.21.15.40:0/2121198274 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fac140a2890 0x7fac140a4d50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fac28199a10 tx=0x7fac2000b040 comp rx=0 tx=0).stop 2024-03-18T22:40:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.846+0000 7fac2cd11700 1 -- 172.21.15.40:0/2121198274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac28107380 msgr2=0x7fac281988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.846+0000 7fac2cd11700 1 --2- 172.21.15.40:0/2121198274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac28107380 0x7fac281988d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fac1800c940 tx=0x7fac1800cc50 comp rx=0 tx=0).stop 2024-03-18T22:40:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.846+0000 7fac2cd11700 1 -- 172.21.15.40:0/2121198274 shutdown_connections 2024-03-18T22:40:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.846+0000 7fac2cd11700 1 --2- 172.21.15.40:0/2121198274 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fac140a2890 0x7fac140a4d50 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.846+0000 7fac2cd11700 1 --2- 172.21.15.40:0/2121198274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac28107380 0x7fac281988d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.846+0000 7fac2cd11700 1 --2- 172.21.15.40:0/2121198274 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac28100900 0x7fac2819e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.846+0000 7fac2cd11700 1 --2- 172.21.15.40:0/2121198274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac281012c0 0x7fac2819ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.846+0000 7fac2cd11700 1 -- 172.21.15.40:0/2121198274 >> 172.21.15.40:0/2121198274 conn(0x7fac28077810 msgr2=0x7fac28105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.846+0000 7fac2cd11700 1 -- 172.21.15.40:0/2121198274 shutdown_connections 2024-03-18T22:40:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:50.847+0000 7fac2cd11700 1 -- 172.21.15.40:0/2121198274 wait complete. 2024-03-18T22:40:51.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:50 smithi040 ceph-mon[62643]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-18T22:40:51.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:50 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2121198274' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T22:40:51.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:50 smithi113 ceph-mon[66775]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-18T22:40:51.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:50 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2121198274' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T22:40:51.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:50 smithi103 ceph-mon[66377]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-03-18T22:40:51.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:50 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2121198274' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T22:40:51.623 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-18T22:40:51.623 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T22:40:51.623 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-03-18T22:40:51.623 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-18T22:40:51.624 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T22:40:51.624 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-18T22:40:51.624 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T22:40:51.624 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T22:40:51.624 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T22:40:51.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:40:51.624 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-03-18T22:40:51.640 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T22:40:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T22:40:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T22:40:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-18T22:40:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-03-18T22:40:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-18T22:40:51.640 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T22:40:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-03-18T22:40:51.640 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-03-18T22:40:51.641 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T22:40:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-03-18T22:40:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-03-18T22:40:51.641 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T22:40:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-03-18T22:40:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-03-18T22:40:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T22:40:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 39803sec preferred_lft 39803sec 2024-03-18T22:40:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-03-18T22:40:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-18T22:40:51.641 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T22:40:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-03-18T22:40:51.641 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-03-18T22:40:51.642 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T22:40:51.642 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T22:40:51.642 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-18T22:40:51.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add name ceph-brx type bridge 2024-03-18T22:40:51.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr flush dev ceph-brx 2024-03-18T22:40:51.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set ceph-brx up 2024-03-18T22:40:51.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T22:40:51.642 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-18T22:40:51.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T22:40:51.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T22:40:51.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:40:51.853 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T22:40:51.880 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-03-18T22:40:51.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:40:51.881 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-03-18T22:40:51.950 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-03-18T22:40:51.950 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T22:40:51.951 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T22:40:51.951 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T22:40:51.951 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T22:40:51.951 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T22:40:51.951 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-18T22:40:51.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T22:40:51.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T22:40:51.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T22:40:51.951 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-18T22:40:52.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T22:40:52.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T22:40:52.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:40:52.258 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-18T22:40:52.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:40:52.273 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list-id 2024-03-18T22:40:52.328 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T22:40:52.328 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-18T22:40:52.329 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T22:40:52.329 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-18T22:40:52.329 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-18T22:40:52.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T22:40:52.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T22:40:52.451 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-18T22:40:52.451 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T22:40:52.451 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-18T22:40:52.451 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-18T22:40:52.451 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T22:40:52.451 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-18T22:40:52.451 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-18T22:40:52.451 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-18T22:40:52.451 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-18T22:40:52.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T22:40:52.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T22:40:52.637 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T22:40:52.637 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-18T22:40:52.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-03-18T22:40:52.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T22:40:52.637 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-18T22:40:52.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T22:40:52.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T22:40:52.718 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-18T22:40:52.718 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T22:40:52.719 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-18T22:40:52.733 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T22:40:52.733 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T22:40:52.733 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-18T22:40:52.787 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-18T22:40:52.855 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T22:40:52.911 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-18T22:40:52.911 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-18T22:40:52.911 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-18T22:40:52.911 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-18T22:40:52.911 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-18T22:40:52.911 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-18T22:40:52.912 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-18T22:40:52.912 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-18T22:40:52.912 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-18T22:40:52.912 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-18T22:40:52.912 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-18T22:40:52.912 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/selinux 2024-03-18T22:40:52.912 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T22:40:52.912 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-18T22:40:52.912 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-18T22:40:52.912 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-18T22:40:52.912 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-18T22:40:52.912 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T22:40:52.912 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-18T22:40:52.912 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-18T22:40:52.913 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T22:40:52.913 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T22:40:52.913 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T22:40:52.913 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-18T22:40:52.913 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-18T22:40:52.913 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay 2024-03-18T22:40:52.913 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/ffadd4dbcadfe7df76393e76b841afbc43fe1c6b0adb1d7adc056ea3cdd53b80/merged 2024-03-18T22:40:52.913 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/e5d1e4b8fdfb35be92f152a64ac0762ea523644962df8b9d571d94edbe59ddb1/merged 2024-03-18T22:40:52.913 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/0 2024-03-18T22:40:52.913 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T22:40:52.913 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/3c1ba48e38cf5203cf83e1c1d44d8ef91a0126a3906b39c3738ac667c369bc37/merged 2024-03-18T22:40:52.913 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/69a5ef89d1322a3b59be2d8ccdaea68c725b78aa7476835d0b8d6c8cf353c092/merged 2024-03-18T22:40:52.913 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/46ba45855c1c5875dfc6f00e3b2812f6978c6dc7e326ce36de4564840e3fbc8a/merged 2024-03-18T22:40:52.914 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/0ef730bb9f35d48fce64218d197e565540ec315beae77cd2ebb785b711b84283/merged 2024-03-18T22:40:52.914 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/107f9392d7587e7089941e3003d627a1a7436dcd3ddb79b4b8ddceec20f12a75/merged 2024-03-18T22:40:52.914 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/466af6a30704e702d0289e0940b69c651437cdc7e0388f8c1a73ae9945fdf6e4/merged 2024-03-18T22:40:52.914 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/c34a43cdd3aea4b12cfb8caf97ed276dc3fee813385dcdfaa3e559dfd432412b/merged 2024-03-18T22:40:52.914 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/457d84b9f6629e8752ef7eab7df5be8812b168869f1c8529c825e17d9df9ad41/merged 2024-03-18T22:40:52.914 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-18T22:40:52.914 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T22:40:52.914 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T22:40:52.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:40:52.914 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-18T22:40:52.966 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T22:40:52.966 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-18T22:40:53.008 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-18T22:40:53.034 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T22:40:53.089 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-18T22:40:53.090 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-18T22:40:53.090 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-18T22:40:53.090 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-18T22:40:53.090 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-18T22:40:53.090 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-18T22:40:53.090 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-18T22:40:53.090 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-18T22:40:53.090 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-18T22:40:53.090 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-18T22:40:53.090 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-18T22:40:53.090 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/selinux 2024-03-18T22:40:53.090 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/ffadd4dbcadfe7df76393e76b841afbc43fe1c6b0adb1d7adc056ea3cdd53b80/merged 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/e5d1e4b8fdfb35be92f152a64ac0762ea523644962df8b9d571d94edbe59ddb1/merged 2024-03-18T22:40:53.091 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/0 2024-03-18T22:40:53.092 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T22:40:53.092 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/3c1ba48e38cf5203cf83e1c1d44d8ef91a0126a3906b39c3738ac667c369bc37/merged 2024-03-18T22:40:53.092 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/69a5ef89d1322a3b59be2d8ccdaea68c725b78aa7476835d0b8d6c8cf353c092/merged 2024-03-18T22:40:53.092 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/46ba45855c1c5875dfc6f00e3b2812f6978c6dc7e326ce36de4564840e3fbc8a/merged 2024-03-18T22:40:53.092 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/0ef730bb9f35d48fce64218d197e565540ec315beae77cd2ebb785b711b84283/merged 2024-03-18T22:40:53.092 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/107f9392d7587e7089941e3003d627a1a7436dcd3ddb79b4b8ddceec20f12a75/merged 2024-03-18T22:40:53.092 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/466af6a30704e702d0289e0940b69c651437cdc7e0388f8c1a73ae9945fdf6e4/merged 2024-03-18T22:40:53.092 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/c34a43cdd3aea4b12cfb8caf97ed276dc3fee813385dcdfaa3e559dfd432412b/merged 2024-03-18T22:40:53.092 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/457d84b9f6629e8752ef7eab7df5be8812b168869f1c8529c825e17d9df9ad41/merged 2024-03-18T22:40:53.092 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-18T22:40:53.092 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T22:40:53.092 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T22:40:53.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:40:53.093 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-18T22:40:53.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:2024-03-18T22:40:53.101+0000 7f257175e480 -1 init, newargv = 0x55b44e08f9f0 newargc=15 2024-03-18T22:40:53.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:ceph-fuse[103476]: starting ceph client 2024-03-18T22:40:53.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:ceph-fuse[103476]: starting fuse 2024-03-18T22:40:53.115 INFO:teuthology.orchestra.run.smithi040.stdout:119 2024-03-18T22:40:53.116 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-18T22:40:53.116 DEBUG:teuthology.orchestra.run.smithi040:> sudo stdin-killer -- python3 -c ' 2024-03-18T22:40:53.116 DEBUG:teuthology.orchestra.run.smithi040:> import glob 2024-03-18T22:40:53.116 DEBUG:teuthology.orchestra.run.smithi040:> import re 2024-03-18T22:40:53.116 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-03-18T22:40:53.116 DEBUG:teuthology.orchestra.run.smithi040:> import subprocess 2024-03-18T22:40:53.116 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-18T22:40:53.116 DEBUG:teuthology.orchestra.run.smithi040:> def _find_admin_socket(client_name): 2024-03-18T22:40:53.116 DEBUG:teuthology.orchestra.run.smithi040:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-18T22:40:53.116 DEBUG:teuthology.orchestra.run.smithi040:> files = glob.glob(asok_path) 2024-03-18T22:40:53.116 DEBUG:teuthology.orchestra.run.smithi040:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> # Given a non-glob path, it better be there 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> if "*" not in asok_path: 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> assert(len(files) == 1) 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> return files[0] 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> for f in files: 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> contents = proc_f.read() 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> if mountpoint in contents: 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> return f 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-18T22:40:53.117 DEBUG:teuthology.orchestra.run.smithi040:> print(_find_admin_socket("client.admin")) 2024-03-18T22:40:53.118 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-03-18T22:40:53.246 INFO:teuthology.orchestra.run.smithi040.stdout:/var/run/ceph/ceph-client.admin.103476.asok 2024-03-18T22:40:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T22:40:53.257 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.103476.asok 2024-03-18T22:40:53.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:40:53.258 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.103476.asok status 2024-03-18T22:40:53.394 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:40:53.394 INFO:teuthology.orchestra.run.smithi040.stdout: "metadata": { 2024-03-18T22:40:53.394 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_sha1": "b6ae24918b03b1b8a19c9263a50655bba80bb810", 2024-03-18T22:40:53.395 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-18T22:40:53.395 INFO:teuthology.orchestra.run.smithi040.stdout: "entity_id": "admin", 2024-03-18T22:40:53.395 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": "smithi040", 2024-03-18T22:40:53.395 INFO:teuthology.orchestra.run.smithi040.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-18T22:40:53.395 INFO:teuthology.orchestra.run.smithi040.stdout: "pid": "103476", 2024-03-18T22:40:53.395 INFO:teuthology.orchestra.run.smithi040.stdout: "root": "/" 2024-03-18T22:40:53.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:40:53.395 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_count": 0, 2024-03-18T22:40:53.395 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_pinned_count": 0, 2024-03-18T22:40:53.395 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 14739, 2024-03-18T22:40:53.395 INFO:teuthology.orchestra.run.smithi040.stdout: "inst": { 2024-03-18T22:40:53.395 INFO:teuthology.orchestra.run.smithi040.stdout: "name": { 2024-03-18T22:40:53.395 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "client", 2024-03-18T22:40:53.395 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 14739 2024-03-18T22:40:53.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:40:53.396 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-18T22:40:53.396 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-18T22:40:53.396 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-18T22:40:53.396 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 35282407 2024-03-18T22:40:53.396 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:40:53.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:40:53.396 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-18T22:40:53.396 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-18T22:40:53.396 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-18T22:40:53.396 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 35282407 2024-03-18T22:40:53.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:40:53.396 INFO:teuthology.orchestra.run.smithi040.stdout: "inst_str": "client.14739 192.168.0.1:0/35282407", 2024-03-18T22:40:53.396 INFO:teuthology.orchestra.run.smithi040.stdout: "addr_str": "192.168.0.1:0/35282407", 2024-03-18T22:40:53.397 INFO:teuthology.orchestra.run.smithi040.stdout: "inode_count": 1, 2024-03-18T22:40:53.397 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_epoch": 16, 2024-03-18T22:40:53.397 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch": 89, 2024-03-18T22:40:53.397 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch_barrier": 0, 2024-03-18T22:40:53.397 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklisted": false, 2024-03-18T22:40:53.397 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs" 2024-03-18T22:40:53.397 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:40:53.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:40:53.403 DEBUG:teuthology.orchestra.run.smithi040:> stat --file-system '--printf=%T 2024-03-18T22:40:53.403 DEBUG:teuthology.orchestra.run.smithi040:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-18T22:40:53.418 INFO:teuthology.orchestra.run.smithi040.stdout:fuseblk 2024-03-18T22:40:53.418 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-18T22:40:53.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:40:53.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-18T22:40:53.485 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-18T22:40:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T22:40:53.594 INFO:teuthology.orchestra.run.smithi040.stdout:# file: . 2024-03-18T22:40:53.595 INFO:teuthology.orchestra.run.smithi040.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-18T22:40:53.595 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:53.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T22:40:53.601 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:40:53.601 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T22:40:53.616 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:40:53.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.758+0000 7fd3034dc640 1 -- 172.21.15.40:0/2685709498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2fc1075d0 msgr2=0x7fd2fc1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.758+0000 7fd3034dc640 1 --2- 172.21.15.40:0/2685709498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2fc1075d0 0x7fd2fc1099c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009ca0 tx=0x7fd2f402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:40:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.758+0000 7fd3034dc640 1 -- 172.21.15.40:0/2685709498 shutdown_connections 2024-03-18T22:40:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.758+0000 7fd3034dc640 1 --2- 172.21.15.40:0/2685709498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2fc1075d0 0x7fd2fc1099c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.758+0000 7fd3034dc640 1 --2- 172.21.15.40:0/2685709498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2fc104ca0 0x7fd2fc107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.758+0000 7fd3034dc640 1 --2- 172.21.15.40:0/2685709498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2fc102370 0x7fd2fc104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.758+0000 7fd3034dc640 1 -- 172.21.15.40:0/2685709498 >> 172.21.15.40:0/2685709498 conn(0x7fd2fc0fbd00 msgr2=0x7fd2fc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.758+0000 7fd3034dc640 1 -- 172.21.15.40:0/2685709498 shutdown_connections 2024-03-18T22:40:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.758+0000 7fd3034dc640 1 -- 172.21.15.40:0/2685709498 wait complete. 2024-03-18T22:40:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.758+0000 7fd3034dc640 1 Processor -- start 2024-03-18T22:40:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.758+0000 7fd3034dc640 1 -- start start 2024-03-18T22:40:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd3034dc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2fc102370 0x7fd2fc197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd3034dc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2fc104ca0 0x7fd2fc197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd3034dc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2fc1075d0 0x7fd2fc19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd3034dc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc10d380 con 0x7fd2fc104ca0 2024-03-18T22:40:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd3034dc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc10d200 con 0x7fd2fc1075d0 2024-03-18T22:40:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd3034dc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc10d500 con 0x7fd2fc102370 2024-03-18T22:40:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd300a77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2fc104ca0 0x7fd2fc197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd300a77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2fc104ca0 0x7fd2fc197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37612/0 (socket says 172.21.15.40:37612) 2024-03-18T22:40:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd300a77640 1 -- 172.21.15.40:0/1169262739 learned_addr learned my addr 172.21.15.40:0/1169262739 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd301a79640 1 --2- 172.21.15.40:0/1169262739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2fc1075d0 0x7fd2fc19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd301278640 1 --2- 172.21.15.40:0/1169262739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2fc102370 0x7fd2fc197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd300a77640 1 -- 172.21.15.40:0/1169262739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2fc102370 msgr2=0x7fd2fc197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd300a77640 1 --2- 172.21.15.40:0/1169262739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2fc102370 0x7fd2fc197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd300a77640 1 -- 172.21.15.40:0/1169262739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2fc1075d0 msgr2=0x7fd2fc19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd300a77640 1 --2- 172.21.15.40:0/1169262739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2fc1075d0 0x7fd2fc19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd300a77640 1 -- 172.21.15.40:0/1169262739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2fc19f670 con 0x7fd2fc104ca0 2024-03-18T22:40:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd300a77640 1 --2- 172.21.15.40:0/1169262739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2fc104ca0 0x7fd2fc197f90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00c970 tx=0x7fd2ec00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd2f27fc640 1 -- 172.21.15.40:0/1169262739 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec007bb0 con 0x7fd2fc104ca0 2024-03-18T22:40:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd2f27fc640 1 -- 172.21.15.40:0/1169262739 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2ec007d50 con 0x7fd2fc104ca0 2024-03-18T22:40:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd3034dc640 1 -- 172.21.15.40:0/1169262739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2fc19f960 con 0x7fd2fc104ca0 2024-03-18T22:40:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd2f27fc640 1 -- 172.21.15.40:0/1169262739 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec00f450 con 0x7fd2fc104ca0 2024-03-18T22:40:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.759+0000 7fd3034dc640 1 -- 172.21.15.40:0/1169262739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2fc19ff00 con 0x7fd2fc104ca0 2024-03-18T22:40:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.760+0000 7fd3034dc640 1 -- 172.21.15.40:0/1169262739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2fc0fd7f0 con 0x7fd2fc104ca0 2024-03-18T22:40:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.761+0000 7fd2f27fc640 1 -- 172.21.15.40:0/1169262739 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd2ec004f20 con 0x7fd2fc104ca0 2024-03-18T22:40:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.762+0000 7fd2f27fc640 1 --2- 172.21.15.40:0/1169262739 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd2e80a2880 0x7fd2e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.762+0000 7fd301278640 1 --2- 172.21.15.40:0/1169262739 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd2e80a2880 0x7fd2e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.762+0000 7fd2f27fc640 1 -- 172.21.15.40:0/1169262739 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd2ec0cec50 con 0x7fd2fc104ca0 2024-03-18T22:40:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.763+0000 7fd301278640 1 --2- 172.21.15.40:0/1169262739 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd2e80a2880 0x7fd2e80a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009b60 tx=0x7fd2f8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:53.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.766+0000 7fd2f27fc640 1 -- 172.21.15.40:0/1169262739 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd2ec014030 con 0x7fd2fc104ca0 2024-03-18T22:40:53.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.952+0000 7fd3034dc640 1 -- 172.21.15.40:0/1169262739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2fc02d3c0 con 0x7fd2fc104ca0 2024-03-18T22:40:53.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.953+0000 7fd2f27fc640 1 -- 172.21.15.40:0/1169262739 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24129 (secure 0 0 0) 0x7fd2ec09bb30 con 0x7fd2fc104ca0 2024-03-18T22:40:53.954 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:53.954 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":89,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:40:46.196741+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:40:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.955+0000 7fd3034dc640 1 -- 172.21.15.40:0/1169262739 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd2e80a2880 msgr2=0x7fd2e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.955+0000 7fd3034dc640 1 --2- 172.21.15.40:0/1169262739 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd2e80a2880 0x7fd2e80a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009b60 tx=0x7fd2f8005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.955+0000 7fd3034dc640 1 -- 172.21.15.40:0/1169262739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2fc104ca0 msgr2=0x7fd2fc197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.955+0000 7fd3034dc640 1 --2- 172.21.15.40:0/1169262739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2fc104ca0 0x7fd2fc197f90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00c970 tx=0x7fd2ec00ce30 comp rx=0 tx=0).stop 2024-03-18T22:40:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.955+0000 7fd3034dc640 1 -- 172.21.15.40:0/1169262739 shutdown_connections 2024-03-18T22:40:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.955+0000 7fd3034dc640 1 --2- 172.21.15.40:0/1169262739 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd2e80a2880 0x7fd2e80a4d40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.955+0000 7fd3034dc640 1 --2- 172.21.15.40:0/1169262739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2fc1075d0 0x7fd2fc19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.955+0000 7fd3034dc640 1 --2- 172.21.15.40:0/1169262739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2fc104ca0 0x7fd2fc197f90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.955+0000 7fd3034dc640 1 --2- 172.21.15.40:0/1169262739 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2fc102370 0x7fd2fc197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.955+0000 7fd3034dc640 1 -- 172.21.15.40:0/1169262739 >> 172.21.15.40:0/1169262739 conn(0x7fd2fc0fbd00 msgr2=0x7fd2fc102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.955+0000 7fd3034dc640 1 -- 172.21.15.40:0/1169262739 shutdown_connections 2024-03-18T22:40:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:53.955+0000 7fd3034dc640 1 -- 172.21.15.40:0/1169262739 wait complete. 2024-03-18T22:40:53.965 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:53 smithi040 ceph-mon[62643]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-18T22:40:53.967 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T22:36:07.060691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T22:40:39.925147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T22:40:40.773757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T22:40:44.068183+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T22:40:53.967 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T22:40:54.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.072+0000 7fa94a46c640 1 -- 172.21.15.40:0/709602304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944100730 msgr2=0x7fa944100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:54.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.072+0000 7fa94a46c640 1 --2- 172.21.15.40:0/709602304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944100730 0x7fa944100b50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa934009900 tx=0x7fa93402bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:54.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.072+0000 7fa94a46c640 1 -- 172.21.15.40:0/709602304 shutdown_connections 2024-03-18T22:40:54.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.072+0000 7fa94a46c640 1 --2- 172.21.15.40:0/709602304 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9441022e0 0x7fa944108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.072+0000 7fa94a46c640 1 --2- 172.21.15.40:0/709602304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa944101920 0x7fa944101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.072+0000 7fa94a46c640 1 --2- 172.21.15.40:0/709602304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944100730 0x7fa944100b50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.072+0000 7fa94a46c640 1 -- 172.21.15.40:0/709602304 >> 172.21.15.40:0/709602304 conn(0x7fa9440fbcc0 msgr2=0x7fa9440fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:54.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.072+0000 7fa94a46c640 1 -- 172.21.15.40:0/709602304 shutdown_connections 2024-03-18T22:40:54.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.072+0000 7fa94a46c640 1 -- 172.21.15.40:0/709602304 wait complete. 2024-03-18T22:40:54.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.072+0000 7fa94a46c640 1 Processor -- start 2024-03-18T22:40:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.072+0000 7fa94a46c640 1 -- start start 2024-03-18T22:40:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa94a46c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa944100730 0x7fa944197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa94a46c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944101920 0x7fa944198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa94a46c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9441022e0 0x7fa94419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa94a46c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa94410c260 con 0x7fa944101920 2024-03-18T22:40:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa94a46c640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa94410c0e0 con 0x7fa944100730 2024-03-18T22:40:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa94a46c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa94410c3e0 con 0x7fa9441022e0 2024-03-18T22:40:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa9437fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944101920 0x7fa944198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa9437fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944101920 0x7fa944198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37626/0 (socket says 172.21.15.40:37626) 2024-03-18T22:40:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa9437fe640 1 -- 172.21.15.40:0/1662855466 learned_addr learned my addr 172.21.15.40:0/1662855466 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa943fff640 1 --2- 172.21.15.40:0/1662855466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa944100730 0x7fa944197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa948a09640 1 --2- 172.21.15.40:0/1662855466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9441022e0 0x7fa94419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa9437fe640 1 -- 172.21.15.40:0/1662855466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9441022e0 msgr2=0x7fa94419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:54.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa9437fe640 1 --2- 172.21.15.40:0/1662855466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9441022e0 0x7fa94419f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa9437fe640 1 -- 172.21.15.40:0/1662855466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa944100730 msgr2=0x7fa944197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:54.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa9437fe640 1 --2- 172.21.15.40:0/1662855466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa944100730 0x7fa944197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa9437fe640 1 -- 172.21.15.40:0/1662855466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94419f700 con 0x7fa944101920 2024-03-18T22:40:54.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa9437fe640 1 --2- 172.21.15.40:0/1662855466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944101920 0x7fa944198020 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa93800ea60 tx=0x7fa93800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:54.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa9417fa640 1 -- 172.21.15.40:0/1662855466 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93800cde0 con 0x7fa944101920 2024-03-18T22:40:54.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa9417fa640 1 -- 172.21.15.40:0/1662855466 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9380035f0 con 0x7fa944101920 2024-03-18T22:40:54.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa9417fa640 1 -- 172.21.15.40:0/1662855466 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa938010690 con 0x7fa944101920 2024-03-18T22:40:54.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.073+0000 7fa94a46c640 1 -- 172.21.15.40:0/1662855466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94419f9f0 con 0x7fa944101920 2024-03-18T22:40:54.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.074+0000 7fa94a46c640 1 -- 172.21.15.40:0/1662855466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94419ff40 con 0x7fa944101920 2024-03-18T22:40:54.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.074+0000 7fa94a46c640 1 -- 172.21.15.40:0/1662855466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9441a01d0 con 0x7fa944101920 2024-03-18T22:40:54.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.076+0000 7fa9417fa640 1 -- 172.21.15.40:0/1662855466 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa9380037a0 con 0x7fa944101920 2024-03-18T22:40:54.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.076+0000 7fa9417fa640 1 --2- 172.21.15.40:0/1662855466 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa92c0a2800 0x7fa92c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:54.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.076+0000 7fa943fff640 1 --2- 172.21.15.40:0/1662855466 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa92c0a2800 0x7fa92c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:54.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.076+0000 7fa9417fa640 1 -- 172.21.15.40:0/1662855466 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa9380cf1b0 con 0x7fa944101920 2024-03-18T22:40:54.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.077+0000 7fa943fff640 1 --2- 172.21.15.40:0/1662855466 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa92c0a2800 0x7fa92c0a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa934005ab0 tx=0x7fa934005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:54.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:53 smithi113 ceph-mon[66775]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-18T22:40:54.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.080+0000 7fa9417fa640 1 -- 172.21.15.40:0/1662855466 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa938053740 con 0x7fa944101920 2024-03-18T22:40:54.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:53 smithi103 ceph-mon[66377]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-18T22:40:54.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.262+0000 7fa94a46c640 1 -- 172.21.15.40:0/1662855466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa9441a03a0 con 0x7fa944101920 2024-03-18T22:40:54.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.262+0000 7fa9417fa640 1 -- 172.21.15.40:0/1662855466 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fa93809c010 con 0x7fa944101920 2024-03-18T22:40:54.263 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-18T22:40:54.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.264+0000 7fa94a46c640 1 -- 172.21.15.40:0/1662855466 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa92c0a2800 msgr2=0x7fa92c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:54.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.264+0000 7fa94a46c640 1 --2- 172.21.15.40:0/1662855466 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa92c0a2800 0x7fa92c0a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa934005ab0 tx=0x7fa934005fd0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.265+0000 7fa94a46c640 1 -- 172.21.15.40:0/1662855466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944101920 msgr2=0x7fa944198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:54.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.265+0000 7fa94a46c640 1 --2- 172.21.15.40:0/1662855466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944101920 0x7fa944198020 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa93800ea60 tx=0x7fa93800ef20 comp rx=0 tx=0).stop 2024-03-18T22:40:54.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.265+0000 7fa94a46c640 1 -- 172.21.15.40:0/1662855466 shutdown_connections 2024-03-18T22:40:54.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.265+0000 7fa94a46c640 1 --2- 172.21.15.40:0/1662855466 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa92c0a2800 0x7fa92c0a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.265+0000 7fa94a46c640 1 --2- 172.21.15.40:0/1662855466 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9441022e0 0x7fa94419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.265+0000 7fa94a46c640 1 --2- 172.21.15.40:0/1662855466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa944101920 0x7fa944198020 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.265+0000 7fa94a46c640 1 --2- 172.21.15.40:0/1662855466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa944100730 0x7fa944197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.265+0000 7fa94a46c640 1 -- 172.21.15.40:0/1662855466 >> 172.21.15.40:0/1662855466 conn(0x7fa9440fbcc0 msgr2=0x7fa944069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:54.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.265+0000 7fa94a46c640 1 -- 172.21.15.40:0/1662855466 shutdown_connections 2024-03-18T22:40:54.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.265+0000 7fa94a46c640 1 -- 172.21.15.40:0/1662855466 wait complete. 2024-03-18T22:40:54.276 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T22:40:54.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.380+0000 7ffa230c6640 1 -- 172.21.15.40:0/2956716821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa1c1033e0 msgr2=0x7ffa1c109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:54.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.380+0000 7ffa230c6640 1 --2- 172.21.15.40:0/2956716821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa1c1033e0 0x7ffa1c109c70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009900 tx=0x7ffa0402bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:54.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa230c6640 1 -- 172.21.15.40:0/2956716821 shutdown_connections 2024-03-18T22:40:54.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa230c6640 1 --2- 172.21.15.40:0/2956716821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa1c1033e0 0x7ffa1c109c70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa230c6640 1 --2- 172.21.15.40:0/2956716821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa1c102a20 0x7ffa1c102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa230c6640 1 --2- 172.21.15.40:0/2956716821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa1c101830 0x7ffa1c101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa230c6640 1 -- 172.21.15.40:0/2956716821 >> 172.21.15.40:0/2956716821 conn(0x7ffa1c0fcdc0 msgr2=0x7ffa1c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:54.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa230c6640 1 -- 172.21.15.40:0/2956716821 shutdown_connections 2024-03-18T22:40:54.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa230c6640 1 -- 172.21.15.40:0/2956716821 wait complete. 2024-03-18T22:40:54.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa230c6640 1 Processor -- start 2024-03-18T22:40:54.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa230c6640 1 -- start start 2024-03-18T22:40:54.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa230c6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa1c101830 0x7ffa1c197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:54.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa230c6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa1c102a20 0x7ffa1c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa230c6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa1c1033e0 0x7ffa1c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa230c6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c10d340 con 0x7ffa1c101830 2024-03-18T22:40:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa230c6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c10d1c0 con 0x7ffa1c1033e0 2024-03-18T22:40:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa230c6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c10d4c0 con 0x7ffa1c102a20 2024-03-18T22:40:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa20e62640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa1c101830 0x7ffa1c197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa20e62640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa1c101830 0x7ffa1c197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37654/0 (socket says 172.21.15.40:37654) 2024-03-18T22:40:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa20e62640 1 -- 172.21.15.40:0/4041439961 learned_addr learned my addr 172.21.15.40:0/4041439961 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa1bfff640 1 --2- 172.21.15.40:0/4041439961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa1c102a20 0x7ffa1c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.381+0000 7ffa21663640 1 --2- 172.21.15.40:0/4041439961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa1c1033e0 0x7ffa1c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.382+0000 7ffa20e62640 1 -- 172.21.15.40:0/4041439961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa1c102a20 msgr2=0x7ffa1c197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.382+0000 7ffa20e62640 1 --2- 172.21.15.40:0/4041439961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa1c102a20 0x7ffa1c197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.382+0000 7ffa20e62640 1 -- 172.21.15.40:0/4041439961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa1c1033e0 msgr2=0x7ffa1c19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.382+0000 7ffa20e62640 1 --2- 172.21.15.40:0/4041439961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa1c1033e0 0x7ffa1c19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.382+0000 7ffa20e62640 1 -- 172.21.15.40:0/4041439961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1c19f630 con 0x7ffa1c101830 2024-03-18T22:40:54.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.382+0000 7ffa20e62640 1 --2- 172.21.15.40:0/4041439961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa1c101830 0x7ffa1c197a10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400e9c0 tx=0x7ffa1400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:54.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.382+0000 7ffa19ffb640 1 -- 172.21.15.40:0/4041439961 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1400cd80 con 0x7ffa1c101830 2024-03-18T22:40:54.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.382+0000 7ffa19ffb640 1 -- 172.21.15.40:0/4041439961 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffa140035f0 con 0x7ffa1c101830 2024-03-18T22:40:54.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.382+0000 7ffa230c6640 1 -- 172.21.15.40:0/4041439961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa1c19f920 con 0x7ffa1c101830 2024-03-18T22:40:54.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.382+0000 7ffa19ffb640 1 -- 172.21.15.40:0/4041439961 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa14010690 con 0x7ffa1c101830 2024-03-18T22:40:54.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.382+0000 7ffa230c6640 1 -- 172.21.15.40:0/4041439961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa1c19fe70 con 0x7ffa1c101830 2024-03-18T22:40:54.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.383+0000 7ffa230c6640 1 -- 172.21.15.40:0/4041439961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa1c068a60 con 0x7ffa1c101830 2024-03-18T22:40:54.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.384+0000 7ffa19ffb640 1 -- 172.21.15.40:0/4041439961 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ffa140037a0 con 0x7ffa1c101830 2024-03-18T22:40:54.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.385+0000 7ffa19ffb640 1 --2- 172.21.15.40:0/4041439961 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ffa080a2800 0x7ffa080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:54.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.385+0000 7ffa1bfff640 1 --2- 172.21.15.40:0/4041439961 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ffa080a2800 0x7ffa080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:54.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.385+0000 7ffa19ffb640 1 -- 172.21.15.40:0/4041439961 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ffa140cf030 con 0x7ffa1c101830 2024-03-18T22:40:54.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.385+0000 7ffa1bfff640 1 --2- 172.21.15.40:0/4041439961 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ffa080a2800 0x7ffa080a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c198f20 tx=0x7ffa0c009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:54.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.389+0000 7ffa19ffb640 1 -- 172.21.15.40:0/4041439961 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffa1409be90 con 0x7ffa1c101830 2024-03-18T22:40:54.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.571+0000 7ffa230c6640 1 -- 172.21.15.40:0/4041439961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffa1c108c20 con 0x7ffa1c101830 2024-03-18T22:40:54.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.571+0000 7ffa19ffb640 1 -- 172.21.15.40:0/4041439961 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7ffa1c108c20 con 0x7ffa1c101830 2024-03-18T22:40:54.572 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:40:54.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.573+0000 7ffa230c6640 1 -- 172.21.15.40:0/4041439961 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ffa080a2800 msgr2=0x7ffa080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:54.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.573+0000 7ffa230c6640 1 --2- 172.21.15.40:0/4041439961 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ffa080a2800 0x7ffa080a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c198f20 tx=0x7ffa0c009500 comp rx=0 tx=0).stop 2024-03-18T22:40:54.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.573+0000 7ffa230c6640 1 -- 172.21.15.40:0/4041439961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa1c101830 msgr2=0x7ffa1c197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:54.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.573+0000 7ffa230c6640 1 --2- 172.21.15.40:0/4041439961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa1c101830 0x7ffa1c197a10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400e9c0 tx=0x7ffa1400ee80 comp rx=0 tx=0).stop 2024-03-18T22:40:54.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.573+0000 7ffa230c6640 1 -- 172.21.15.40:0/4041439961 shutdown_connections 2024-03-18T22:40:54.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.573+0000 7ffa230c6640 1 --2- 172.21.15.40:0/4041439961 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ffa080a2800 0x7ffa080a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.573+0000 7ffa230c6640 1 --2- 172.21.15.40:0/4041439961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa1c1033e0 0x7ffa1c19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.573+0000 7ffa230c6640 1 --2- 172.21.15.40:0/4041439961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa1c102a20 0x7ffa1c197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.573+0000 7ffa230c6640 1 --2- 172.21.15.40:0/4041439961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa1c101830 0x7ffa1c197a10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.573+0000 7ffa230c6640 1 -- 172.21.15.40:0/4041439961 >> 172.21.15.40:0/4041439961 conn(0x7ffa1c0fcdc0 msgr2=0x7ffa1c105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:54.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.573+0000 7ffa230c6640 1 -- 172.21.15.40:0/4041439961 shutdown_connections 2024-03-18T22:40:54.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.573+0000 7ffa230c6640 1 -- 172.21.15.40:0/4041439961 wait complete. 2024-03-18T22:40:54.585 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T22:40:54.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.689+0000 7f4245b97640 1 -- 172.21.15.40:0/2830454556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4240101910 msgr2=0x7f4240101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:54.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.689+0000 7f4245b97640 1 --2- 172.21.15.40:0/2830454556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4240101910 0x7f4240101d90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4234009900 tx=0x7f423402bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:54.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.689+0000 7f4245b97640 1 -- 172.21.15.40:0/2830454556 shutdown_connections 2024-03-18T22:40:54.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.689+0000 7f4245b97640 1 --2- 172.21.15.40:0/2830454556 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42401022d0 0x7f4240108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.689+0000 7f4245b97640 1 --2- 172.21.15.40:0/2830454556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4240101910 0x7f4240101d90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.689+0000 7f4245b97640 1 --2- 172.21.15.40:0/2830454556 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4240100720 0x7f4240100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.689+0000 7f4245b97640 1 -- 172.21.15.40:0/2830454556 >> 172.21.15.40:0/2830454556 conn(0x7f42400fbcb0 msgr2=0x7f42400fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.689+0000 7f4245b97640 1 -- 172.21.15.40:0/2830454556 shutdown_connections 2024-03-18T22:40:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.689+0000 7f4245b97640 1 -- 172.21.15.40:0/2830454556 wait complete. 2024-03-18T22:40:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.690+0000 7f4245b97640 1 Processor -- start 2024-03-18T22:40:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.690+0000 7f4245b97640 1 -- start start 2024-03-18T22:40:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.690+0000 7f4245b97640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4240100720 0x7f4240197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.690+0000 7f4245b97640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4240101910 0x7f4240197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.690+0000 7f4245b97640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42401022d0 0x7f424019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.690+0000 7f4245b97640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f424010c230 con 0x7f4240100720 2024-03-18T22:40:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.690+0000 7f4245b97640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f424010c0b0 con 0x7f42401022d0 2024-03-18T22:40:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.690+0000 7f4245b97640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f424010c3b0 con 0x7f4240101910 2024-03-18T22:40:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.690+0000 7f423f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4240100720 0x7f4240197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.690+0000 7f423f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4240100720 0x7f4240197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37672/0 (socket says 172.21.15.40:37672) 2024-03-18T22:40:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.690+0000 7f423f7fe640 1 -- 172.21.15.40:0/2504621363 learned_addr learned my addr 172.21.15.40:0/2504621363 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.690+0000 7f423ffff640 1 --2- 172.21.15.40:0/2504621363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42401022d0 0x7f424019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.690+0000 7f423effd640 1 --2- 172.21.15.40:0/2504621363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4240101910 0x7f4240197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.691+0000 7f423f7fe640 1 -- 172.21.15.40:0/2504621363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4240101910 msgr2=0x7f4240197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.691+0000 7f423f7fe640 1 --2- 172.21.15.40:0/2504621363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4240101910 0x7f4240197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.691+0000 7f423f7fe640 1 -- 172.21.15.40:0/2504621363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42401022d0 msgr2=0x7f424019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.691+0000 7f423f7fe640 1 --2- 172.21.15.40:0/2504621363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42401022d0 0x7f424019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.691+0000 7f423f7fe640 1 -- 172.21.15.40:0/2504621363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424019f6a0 con 0x7f4240100720 2024-03-18T22:40:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.691+0000 7f423f7fe640 1 --2- 172.21.15.40:0/2504621363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4240100720 0x7f4240197a80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f423000d950 tx=0x7f423000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.691+0000 7f423cff9640 1 -- 172.21.15.40:0/2504621363 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4230017020 con 0x7f4240100720 2024-03-18T22:40:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.691+0000 7f4245b97640 1 -- 172.21.15.40:0/2504621363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f424019f990 con 0x7f4240100720 2024-03-18T22:40:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.691+0000 7f423cff9640 1 -- 172.21.15.40:0/2504621363 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4230009c30 con 0x7f4240100720 2024-03-18T22:40:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.691+0000 7f423cff9640 1 -- 172.21.15.40:0/2504621363 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423000f6b0 con 0x7f4240100720 2024-03-18T22:40:54.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.691+0000 7f4245b97640 1 -- 172.21.15.40:0/2504621363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f424019feb0 con 0x7f4240100720 2024-03-18T22:40:54.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.692+0000 7f4245b97640 1 -- 172.21.15.40:0/2504621363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4240068610 con 0x7f4240100720 2024-03-18T22:40:54.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.693+0000 7f423cff9640 1 -- 172.21.15.40:0/2504621363 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f423001a020 con 0x7f4240100720 2024-03-18T22:40:54.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.694+0000 7f423cff9640 1 --2- 172.21.15.40:0/2504621363 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f42280a2800 0x7f42280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:54.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.694+0000 7f423effd640 1 --2- 172.21.15.40:0/2504621363 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f42280a2800 0x7f42280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:54.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.694+0000 7f423cff9640 1 -- 172.21.15.40:0/2504621363 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f42300ce930 con 0x7f4240100720 2024-03-18T22:40:54.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.694+0000 7f423effd640 1 --2- 172.21.15.40:0/2504621363 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f42280a2800 0x7f42280a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4234009230 tx=0x7f4234005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:54.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.698+0000 7f423cff9640 1 -- 172.21.15.40:0/2504621363 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4230009de0 con 0x7f4240100720 2024-03-18T22:40:54.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.882+0000 7f4245b97640 1 -- 172.21.15.40:0/2504621363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f42401a0840 con 0x7f4240100720 2024-03-18T22:40:54.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.883+0000 7f423cff9640 1 -- 172.21.15.40:0/2504621363 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f423009b730 con 0x7f4240100720 2024-03-18T22:40:54.883 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-18T22:40:54.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.884+0000 7f4245b97640 1 -- 172.21.15.40:0/2504621363 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f42280a2800 msgr2=0x7f42280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:54.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.884+0000 7f4245b97640 1 --2- 172.21.15.40:0/2504621363 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f42280a2800 0x7f42280a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4234009230 tx=0x7f4234005fd0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.884+0000 7f4245b97640 1 -- 172.21.15.40:0/2504621363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4240100720 msgr2=0x7f4240197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:54.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.884+0000 7f4245b97640 1 --2- 172.21.15.40:0/2504621363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4240100720 0x7f4240197a80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f423000d950 tx=0x7f423000de10 comp rx=0 tx=0).stop 2024-03-18T22:40:54.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.885+0000 7f4245b97640 1 -- 172.21.15.40:0/2504621363 shutdown_connections 2024-03-18T22:40:54.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.885+0000 7f4245b97640 1 --2- 172.21.15.40:0/2504621363 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f42280a2800 0x7f42280a4cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.885+0000 7f4245b97640 1 --2- 172.21.15.40:0/2504621363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f42401022d0 0x7f424019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.885+0000 7f4245b97640 1 --2- 172.21.15.40:0/2504621363 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4240101910 0x7f4240197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.885+0000 7f4245b97640 1 --2- 172.21.15.40:0/2504621363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4240100720 0x7f4240197a80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:54.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.885+0000 7f4245b97640 1 -- 172.21.15.40:0/2504621363 >> 172.21.15.40:0/2504621363 conn(0x7f42400fbcb0 msgr2=0x7f42400689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:54.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.885+0000 7f4245b97640 1 -- 172.21.15.40:0/2504621363 shutdown_connections 2024-03-18T22:40:54.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:54.885+0000 7f4245b97640 1 -- 172.21.15.40:0/2504621363 wait complete. 2024-03-18T22:40:54.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:54 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1169262739' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:54.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:54 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1662855466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:40:54.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:54 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/4041439961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:40:54.896 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T22:40:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.001+0000 7f659ed0a640 1 -- 172.21.15.40:0/1960403128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6598100250 msgr2=0x7f6598109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.001+0000 7f659ed0a640 1 --2- 172.21.15.40:0/1960403128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6598100250 0x7f6598109e00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6588009900 tx=0x7f658802bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.001+0000 7f659ed0a640 1 -- 172.21.15.40:0/1960403128 shutdown_connections 2024-03-18T22:40:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.001+0000 7f659ed0a640 1 --2- 172.21.15.40:0/1960403128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6598100250 0x7f6598109e00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.001+0000 7f659ed0a640 1 --2- 172.21.15.40:0/1960403128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65980ff890 0x7f65980ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.001+0000 7f659ed0a640 1 --2- 172.21.15.40:0/1960403128 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6598102e20 0x7f6598103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.001+0000 7f659ed0a640 1 -- 172.21.15.40:0/1960403128 >> 172.21.15.40:0/1960403128 conn(0x7f65980fb450 msgr2=0x7f65980fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.001+0000 7f659ed0a640 1 -- 172.21.15.40:0/1960403128 shutdown_connections 2024-03-18T22:40:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.001+0000 7f659ed0a640 1 -- 172.21.15.40:0/1960403128 wait complete. 2024-03-18T22:40:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.001+0000 7f659ed0a640 1 Processor -- start 2024-03-18T22:40:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.001+0000 7f659ed0a640 1 -- start start 2024-03-18T22:40:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659ed0a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65980ff890 0x7f65981979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659ed0a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6598100250 0x7f6598197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659ed0a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6598102e20 0x7f659819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659ed0a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f659810d3c0 con 0x7f65980ff890 2024-03-18T22:40:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659ed0a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f659810d240 con 0x7f6598100250 2024-03-18T22:40:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659ed0a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f659810d540 con 0x7f6598102e20 2024-03-18T22:40:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659caa6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65980ff890 0x7f65981979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659caa6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65980ff890 0x7f65981979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37694/0 (socket says 172.21.15.40:37694) 2024-03-18T22:40:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659caa6640 1 -- 172.21.15.40:0/3406159150 learned_addr learned my addr 172.21.15.40:0/3406159150 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f6597fff640 1 --2- 172.21.15.40:0/3406159150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6598100250 0x7f6598197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659d2a7640 1 --2- 172.21.15.40:0/3406159150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6598102e20 0x7f659819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659caa6640 1 -- 172.21.15.40:0/3406159150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6598102e20 msgr2=0x7f659819ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659caa6640 1 --2- 172.21.15.40:0/3406159150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6598102e20 0x7f659819ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659caa6640 1 -- 172.21.15.40:0/3406159150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6598100250 msgr2=0x7f6598197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:55.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659caa6640 1 --2- 172.21.15.40:0/3406159150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6598100250 0x7f6598197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659caa6640 1 -- 172.21.15.40:0/3406159150 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f659819f5f0 con 0x7f65980ff890 2024-03-18T22:40:55.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659caa6640 1 --2- 172.21.15.40:0/3406159150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65980ff890 0x7f65981979d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f659000eb70 tx=0x7f659000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:55.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f6595ffb640 1 -- 172.21.15.40:0/3406159150 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6590003b70 con 0x7f65980ff890 2024-03-18T22:40:55.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f6595ffb640 1 -- 172.21.15.40:0/3406159150 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f659000ee70 con 0x7f65980ff890 2024-03-18T22:40:55.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f659ed0a640 1 -- 172.21.15.40:0/3406159150 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f659819f8e0 con 0x7f65980ff890 2024-03-18T22:40:55.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.002+0000 7f6595ffb640 1 -- 172.21.15.40:0/3406159150 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6590010650 con 0x7f65980ff890 2024-03-18T22:40:55.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.003+0000 7f659ed0a640 1 -- 172.21.15.40:0/3406159150 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f659819fe30 con 0x7f65980ff890 2024-03-18T22:40:55.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.003+0000 7f659ed0a640 1 -- 172.21.15.40:0/3406159150 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6598108ce0 con 0x7f65980ff890 2024-03-18T22:40:55.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.004+0000 7f6595ffb640 1 -- 172.21.15.40:0/3406159150 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f6590004020 con 0x7f65980ff890 2024-03-18T22:40:55.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.005+0000 7f6595ffb640 1 --2- 172.21.15.40:0/3406159150 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f65800a2880 0x7f65800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:55.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.005+0000 7f6597fff640 1 --2- 172.21.15.40:0/3406159150 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f65800a2880 0x7f65800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:55.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.005+0000 7f6595ffb640 1 -- 172.21.15.40:0/3406159150 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f65900cf2f0 con 0x7f65980ff890 2024-03-18T22:40:55.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.005+0000 7f6597fff640 1 --2- 172.21.15.40:0/3406159150 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f65800a2880 0x7f65800a4d40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6598198ee0 tx=0x7f658c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:55.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.009+0000 7f6595ffb640 1 -- 172.21.15.40:0/3406159150 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f659009c0f0 con 0x7f65980ff890 2024-03-18T22:40:55.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:54 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1169262739' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:55.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:54 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1662855466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:40:55.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:54 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/4041439961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:40:55.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:54 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1169262739' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:55.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:54 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1662855466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:40:55.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:54 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/4041439961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:40:55.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.195+0000 7f659ed0a640 1 -- 172.21.15.40:0/3406159150 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6598062420 con 0x7f65980ff890 2024-03-18T22:40:55.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.195+0000 7f6595ffb640 1 -- 172.21.15.40:0/3406159150 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f659009b9f0 con 0x7f65980ff890 2024-03-18T22:40:55.196 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:40:55.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.197+0000 7f659ed0a640 1 -- 172.21.15.40:0/3406159150 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f65800a2880 msgr2=0x7f65800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:55.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.198+0000 7f659ed0a640 1 --2- 172.21.15.40:0/3406159150 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f65800a2880 0x7f65800a4d40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6598198ee0 tx=0x7f658c005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:55.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.198+0000 7f659ed0a640 1 -- 172.21.15.40:0/3406159150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65980ff890 msgr2=0x7f65981979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:55.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.198+0000 7f659ed0a640 1 --2- 172.21.15.40:0/3406159150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65980ff890 0x7f65981979d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f659000eb70 tx=0x7f659000c6a0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.198+0000 7f659ed0a640 1 -- 172.21.15.40:0/3406159150 shutdown_connections 2024-03-18T22:40:55.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.198+0000 7f659ed0a640 1 --2- 172.21.15.40:0/3406159150 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f65800a2880 0x7f65800a4d40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.198+0000 7f659ed0a640 1 --2- 172.21.15.40:0/3406159150 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6598102e20 0x7f659819ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.198+0000 7f659ed0a640 1 --2- 172.21.15.40:0/3406159150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6598100250 0x7f6598197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.198+0000 7f659ed0a640 1 --2- 172.21.15.40:0/3406159150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65980ff890 0x7f65981979d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.198+0000 7f659ed0a640 1 -- 172.21.15.40:0/3406159150 >> 172.21.15.40:0/3406159150 conn(0x7f65980fb450 msgr2=0x7f6598105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:55.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.198+0000 7f659ed0a640 1 -- 172.21.15.40:0/3406159150 shutdown_connections 2024-03-18T22:40:55.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.198+0000 7f659ed0a640 1 -- 172.21.15.40:0/3406159150 wait complete. 2024-03-18T22:40:55.210 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T22:40:55.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.313+0000 7fabf4a75640 1 -- 172.21.15.40:0/2682376254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabf01022e0 msgr2=0x7fabf0108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:55.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.313+0000 7fabf4a75640 1 --2- 172.21.15.40:0/2682376254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabf01022e0 0x7fabf0108b70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fabe8009ca0 tx=0x7fabe802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.313+0000 7fabf4a75640 1 -- 172.21.15.40:0/2682376254 shutdown_connections 2024-03-18T22:40:55.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.313+0000 7fabf4a75640 1 --2- 172.21.15.40:0/2682376254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabf01022e0 0x7fabf0108b70 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.313+0000 7fabf4a75640 1 --2- 172.21.15.40:0/2682376254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf0101920 0x7fabf0101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.313+0000 7fabf4a75640 1 --2- 172.21.15.40:0/2682376254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabf0100730 0x7fabf0100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.313+0000 7fabf4a75640 1 -- 172.21.15.40:0/2682376254 >> 172.21.15.40:0/2682376254 conn(0x7fabf00fbcc0 msgr2=0x7fabf00fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:55.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.313+0000 7fabf4a75640 1 -- 172.21.15.40:0/2682376254 shutdown_connections 2024-03-18T22:40:55.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.313+0000 7fabf4a75640 1 -- 172.21.15.40:0/2682376254 wait complete. 2024-03-18T22:40:55.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabf4a75640 1 Processor -- start 2024-03-18T22:40:55.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabf4a75640 1 -- start start 2024-03-18T22:40:55.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabf4a75640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabf0100730 0x7fabf01046c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:55.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabf4a75640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf0101920 0x7fabf0102d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:55.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabf4a75640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabf01022e0 0x7fabf0103240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:55.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabf4a75640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fabf010c260 con 0x7fabf0100730 2024-03-18T22:40:55.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabf4a75640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fabf010c0e0 con 0x7fabf01022e0 2024-03-18T22:40:55.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabf4a75640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fabf010c3e0 con 0x7fabf0101920 2024-03-18T22:40:55.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabee59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabf0100730 0x7fabf01046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:55.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabee59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabf0100730 0x7fabf01046c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37716/0 (socket says 172.21.15.40:37716) 2024-03-18T22:40:55.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabee59c640 1 -- 172.21.15.40:0/730826785 learned_addr learned my addr 172.21.15.40:0/730826785 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:55.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabeed9d640 1 --2- 172.21.15.40:0/730826785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabf01022e0 0x7fabf0103240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:55.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabedd9b640 1 --2- 172.21.15.40:0/730826785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf0101920 0x7fabf0102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:55.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabee59c640 1 -- 172.21.15.40:0/730826785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf0101920 msgr2=0x7fabf0102d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:55.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabee59c640 1 --2- 172.21.15.40:0/730826785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf0101920 0x7fabf0102d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabee59c640 1 -- 172.21.15.40:0/730826785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabf01022e0 msgr2=0x7fabf0103240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:55.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabee59c640 1 --2- 172.21.15.40:0/730826785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabf01022e0 0x7fabf0103240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.314+0000 7fabee59c640 1 -- 172.21.15.40:0/730826785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf0103ad0 con 0x7fabf0100730 2024-03-18T22:40:55.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.315+0000 7fabee59c640 1 --2- 172.21.15.40:0/730826785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabf0100730 0x7fabf01046c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fabe000e9c0 tx=0x7fabe000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:55.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.315+0000 7fabdf7fe640 1 -- 172.21.15.40:0/730826785 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe000cde0 con 0x7fabf0100730 2024-03-18T22:40:55.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.315+0000 7fabdf7fe640 1 -- 172.21.15.40:0/730826785 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabe00108e0 con 0x7fabf0100730 2024-03-18T22:40:55.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.315+0000 7fabdf7fe640 1 -- 172.21.15.40:0/730826785 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe0018950 con 0x7fabf0100730 2024-03-18T22:40:55.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.315+0000 7fabf4a75640 1 -- 172.21.15.40:0/730826785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabf01a1b70 con 0x7fabf0100730 2024-03-18T22:40:55.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.315+0000 7fabf4a75640 1 -- 172.21.15.40:0/730826785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabf01a2070 con 0x7fabf0100730 2024-03-18T22:40:55.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.316+0000 7fabf4a75640 1 -- 172.21.15.40:0/730826785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabf01a2240 con 0x7fabf0100730 2024-03-18T22:40:55.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.317+0000 7fabdf7fe640 1 -- 172.21.15.40:0/730826785 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fabe0010a90 con 0x7fabf0100730 2024-03-18T22:40:55.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.317+0000 7fabdf7fe640 1 --2- 172.21.15.40:0/730826785 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fabd80a2880 0x7fabd80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:55.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.317+0000 7fabdf7fe640 1 -- 172.21.15.40:0/730826785 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fabe00cf280 con 0x7fabf0100730 2024-03-18T22:40:55.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.318+0000 7fabedd9b640 1 --2- 172.21.15.40:0/730826785 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fabd80a2880 0x7fabd80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:55.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.318+0000 7fabedd9b640 1 --2- 172.21.15.40:0/730826785 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fabd80a2880 0x7fabd80a4d40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fabf0104430 tx=0x7fabe4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:55.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.322+0000 7fabdf7fe640 1 -- 172.21.15.40:0/730826785 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fabe0053740 con 0x7fabf0100730 2024-03-18T22:40:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.516+0000 7fabf4a75640 1 -- 172.21.15.40:0/730826785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fabf01a2530 con 0x7fabf0100730 2024-03-18T22:40:55.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.516+0000 7fabdf7fe640 1 -- 172.21.15.40:0/730826785 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+591 (secure 0 0 0) 0x7fabe009c080 con 0x7fabf0100730 2024-03-18T22:40:55.517 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/347668672 2024-03-19T22:33:42.721571+0000 2024-03-18T22:40:55.517 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/4237962144 2024-03-19T22:33:42.721571+0000 2024-03-18T22:40:55.517 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/1454961757 2024-03-19T22:33:15.826478+0000 2024-03-18T22:40:55.517 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3047621005 2024-03-19T22:33:15.826478+0000 2024-03-18T22:40:55.517 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/2121391701 2024-03-19T22:33:15.826478+0000 2024-03-18T22:40:55.517 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/3876208039 2024-03-19T22:33:42.721571+0000 2024-03-18T22:40:55.517 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/3876208039 2024-03-19T22:33:42.721571+0000 2024-03-18T22:40:55.517 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/2591529486 2024-03-19T22:33:42.721571+0000 2024-03-18T22:40:55.517 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/1153335782 2024-03-19T22:33:15.826478+0000 2024-03-18T22:40:55.517 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/1153335782 2024-03-19T22:33:15.826478+0000 2024-03-18T22:40:55.517 INFO:teuthology.orchestra.run.smithi040.stderr:listed 10 entries 2024-03-18T22:40:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.518+0000 7fabf4a75640 1 -- 172.21.15.40:0/730826785 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fabd80a2880 msgr2=0x7fabd80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.518+0000 7fabf4a75640 1 --2- 172.21.15.40:0/730826785 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fabd80a2880 0x7fabd80a4d40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fabf0104430 tx=0x7fabe4005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.518+0000 7fabf4a75640 1 -- 172.21.15.40:0/730826785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabf0100730 msgr2=0x7fabf01046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.518+0000 7fabf4a75640 1 --2- 172.21.15.40:0/730826785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabf0100730 0x7fabf01046c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fabe000e9c0 tx=0x7fabe000ee80 comp rx=0 tx=0).stop 2024-03-18T22:40:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.518+0000 7fabf4a75640 1 -- 172.21.15.40:0/730826785 shutdown_connections 2024-03-18T22:40:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.518+0000 7fabf4a75640 1 --2- 172.21.15.40:0/730826785 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fabd80a2880 0x7fabd80a4d40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.518+0000 7fabf4a75640 1 --2- 172.21.15.40:0/730826785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabf01022e0 0x7fabf0103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.518+0000 7fabf4a75640 1 --2- 172.21.15.40:0/730826785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fabf0101920 0x7fabf0102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.518+0000 7fabf4a75640 1 --2- 172.21.15.40:0/730826785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabf0100730 0x7fabf01046c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.518+0000 7fabf4a75640 1 -- 172.21.15.40:0/730826785 >> 172.21.15.40:0/730826785 conn(0x7fabf00fbcc0 msgr2=0x7fabf00fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.518+0000 7fabf4a75640 1 -- 172.21.15.40:0/730826785 shutdown_connections 2024-03-18T22:40:55.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.518+0000 7fabf4a75640 1 -- 172.21.15.40:0/730826785 wait complete. 2024-03-18T22:40:55.530 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:40:55.530 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T22:40:55.545 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:40:55.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.687+0000 7f2c8d859640 1 -- 172.21.15.40:0/124814827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c88101930 msgr2=0x7f2c88101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:55.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.687+0000 7f2c8d859640 1 --2- 172.21.15.40:0/124814827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c88101930 0x7f2c88101db0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2c70009900 tx=0x7f2c7002bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:55.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.687+0000 7f2c8d859640 1 -- 172.21.15.40:0/124814827 shutdown_connections 2024-03-18T22:40:55.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.687+0000 7f2c8d859640 1 --2- 172.21.15.40:0/124814827 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c881022f0 0x7f2c88108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.687+0000 7f2c8d859640 1 --2- 172.21.15.40:0/124814827 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c88101930 0x7f2c88101db0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.687+0000 7f2c8d859640 1 --2- 172.21.15.40:0/124814827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c88100740 0x7f2c88100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.687+0000 7f2c8d859640 1 -- 172.21.15.40:0/124814827 >> 172.21.15.40:0/124814827 conn(0x7f2c880fbcb0 msgr2=0x7f2c880fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:55.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.687+0000 7f2c8d859640 1 -- 172.21.15.40:0/124814827 shutdown_connections 2024-03-18T22:40:55.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.687+0000 7f2c8d859640 1 -- 172.21.15.40:0/124814827 wait complete. 2024-03-18T22:40:55.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.688+0000 7f2c8d859640 1 Processor -- start 2024-03-18T22:40:55.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.688+0000 7f2c8d859640 1 -- start start 2024-03-18T22:40:55.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.688+0000 7f2c8d859640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c88100740 0x7f2c881046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:55.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.688+0000 7f2c8d859640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c88101930 0x7f2c88102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:55.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.688+0000 7f2c8d859640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c881022f0 0x7f2c88103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:55.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.688+0000 7f2c8d859640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8810c270 con 0x7f2c88100740 2024-03-18T22:40:55.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.688+0000 7f2c8d859640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8810c0f0 con 0x7f2c88101930 2024-03-18T22:40:55.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.688+0000 7f2c8d859640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8810c3f0 con 0x7f2c881022f0 2024-03-18T22:40:55.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.688+0000 7f2c86ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c88100740 0x7f2c881046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:55.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.688+0000 7f2c86ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c88100740 0x7f2c881046d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37722/0 (socket says 172.21.15.40:37722) 2024-03-18T22:40:55.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.688+0000 7f2c86ffd640 1 -- 172.21.15.40:0/30062867 learned_addr learned my addr 172.21.15.40:0/30062867 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:55.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.688+0000 7f2c867fc640 1 --2- 172.21.15.40:0/30062867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c88101930 0x7f2c88102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:55.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.688+0000 7f2c877fe640 1 --2- 172.21.15.40:0/30062867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c881022f0 0x7f2c88103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:55.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.689+0000 7f2c867fc640 1 -- 172.21.15.40:0/30062867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c881022f0 msgr2=0x7f2c88103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:55.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.689+0000 7f2c867fc640 1 --2- 172.21.15.40:0/30062867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c881022f0 0x7f2c88103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.689+0000 7f2c867fc640 1 -- 172.21.15.40:0/30062867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c88100740 msgr2=0x7f2c881046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:55.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.689+0000 7f2c867fc640 1 --2- 172.21.15.40:0/30062867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c88100740 0x7f2c881046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.689+0000 7f2c867fc640 1 -- 172.21.15.40:0/30062867 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c88103ae0 con 0x7f2c88101930 2024-03-18T22:40:55.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.689+0000 7f2c86ffd640 1 --2- 172.21.15.40:0/30062867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c88100740 0x7f2c881046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:40:55.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.689+0000 7f2c867fc640 1 --2- 172.21.15.40:0/30062867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c88101930 0x7f2c88102d10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2c70000c00 tx=0x7f2c70004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:55.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.689+0000 7f2c8c857640 1 -- 172.21.15.40:0/30062867 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c70043020 con 0x7f2c88101930 2024-03-18T22:40:55.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.689+0000 7f2c8d859640 1 -- 172.21.15.40:0/30062867 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c881a1c10 con 0x7f2c88101930 2024-03-18T22:40:55.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.689+0000 7f2c8c857640 1 -- 172.21.15.40:0/30062867 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c70004440 con 0x7f2c88101930 2024-03-18T22:40:55.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.689+0000 7f2c8c857640 1 -- 172.21.15.40:0/30062867 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c70003c60 con 0x7f2c88101930 2024-03-18T22:40:55.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.689+0000 7f2c8d859640 1 -- 172.21.15.40:0/30062867 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c881a2160 con 0x7f2c88101930 2024-03-18T22:40:55.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.690+0000 7f2c8d859640 1 -- 172.21.15.40:0/30062867 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c881a2330 con 0x7f2c88101930 2024-03-18T22:40:55.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.691+0000 7f2c8c857640 1 -- 172.21.15.40:0/30062867 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2c70046020 con 0x7f2c88101930 2024-03-18T22:40:55.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.692+0000 7f2c8c857640 1 --2- 172.21.15.40:0/30062867 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2c740a2800 0x7f2c740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:55.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.692+0000 7f2c86ffd640 1 --2- 172.21.15.40:0/30062867 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2c740a2800 0x7f2c740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:55.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.692+0000 7f2c8c857640 1 -- 172.21.15.40:0/30062867 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2c700f13c0 con 0x7f2c88101930 2024-03-18T22:40:55.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.692+0000 7f2c86ffd640 1 --2- 172.21.15.40:0/30062867 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2c740a2800 0x7f2c740a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800f7b0 tx=0x7f2c78009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:55.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.696+0000 7f2c8c857640 1 -- 172.21.15.40:0/30062867 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2c7004d020 con 0x7f2c88101930 2024-03-18T22:40:55.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.882+0000 7f2c8d859640 1 -- 172.21.15.40:0/30062867 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c881a2500 con 0x7f2c88101930 2024-03-18T22:40:55.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.884+0000 7f2c8c857640 1 -- 172.21.15.40:0/30062867 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24129 (secure 0 0 0) 0x7f2c700be1c0 con 0x7f2c88101930 2024-03-18T22:40:55.885 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:40:55.885 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":89,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:40:46.196741+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:40:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.886+0000 7f2c8d859640 1 -- 172.21.15.40:0/30062867 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2c740a2800 msgr2=0x7f2c740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.886+0000 7f2c8d859640 1 --2- 172.21.15.40:0/30062867 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2c740a2800 0x7f2c740a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800f7b0 tx=0x7f2c78009500 comp rx=0 tx=0).stop 2024-03-18T22:40:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.886+0000 7f2c8d859640 1 -- 172.21.15.40:0/30062867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c88101930 msgr2=0x7f2c88102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.886+0000 7f2c8d859640 1 --2- 172.21.15.40:0/30062867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c88101930 0x7f2c88102d10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2c70000c00 tx=0x7f2c70004040 comp rx=0 tx=0).stop 2024-03-18T22:40:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.886+0000 7f2c8d859640 1 -- 172.21.15.40:0/30062867 shutdown_connections 2024-03-18T22:40:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.886+0000 7f2c8d859640 1 --2- 172.21.15.40:0/30062867 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2c740a2800 0x7f2c740a4cc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.886+0000 7f2c8d859640 1 --2- 172.21.15.40:0/30062867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2c881022f0 0x7f2c88103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.886+0000 7f2c8d859640 1 --2- 172.21.15.40:0/30062867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2c88101930 0x7f2c88102d10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.886+0000 7f2c8d859640 1 --2- 172.21.15.40:0/30062867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c88100740 0x7f2c881046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:55.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.886+0000 7f2c8d859640 1 -- 172.21.15.40:0/30062867 >> 172.21.15.40:0/30062867 conn(0x7f2c880fbcb0 msgr2=0x7f2c880689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.886+0000 7f2c8d859640 1 -- 172.21.15.40:0/30062867 shutdown_connections 2024-03-18T22:40:55.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:55.886+0000 7f2c8d859640 1 -- 172.21.15.40:0/30062867 wait complete. 2024-03-18T22:40:55.898 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T22:36:07.060691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T22:40:39.925147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T22:40:40.773757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T22:40:44.068183+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T22:40:55.898 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T22:40:56.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.003+0000 7fc0acab2640 1 -- 172.21.15.40:0/4203964994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0a8101910 msgr2=0x7fc0a8101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.003+0000 7fc0acab2640 1 --2- 172.21.15.40:0/4203964994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0a8101910 0x7fc0a8101d90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc090009900 tx=0x7fc09002bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:56.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.003+0000 7fc0acab2640 1 -- 172.21.15.40:0/4203964994 shutdown_connections 2024-03-18T22:40:56.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.003+0000 7fc0acab2640 1 --2- 172.21.15.40:0/4203964994 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0a81022d0 0x7fc0a8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.004+0000 7fc0acab2640 1 --2- 172.21.15.40:0/4203964994 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0a8101910 0x7fc0a8101d90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.004+0000 7fc0acab2640 1 --2- 172.21.15.40:0/4203964994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0a8100720 0x7fc0a8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.004+0000 7fc0acab2640 1 -- 172.21.15.40:0/4203964994 >> 172.21.15.40:0/4203964994 conn(0x7fc0a80fbcb0 msgr2=0x7fc0a80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:56.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.004+0000 7fc0acab2640 1 -- 172.21.15.40:0/4203964994 shutdown_connections 2024-03-18T22:40:56.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.004+0000 7fc0acab2640 1 -- 172.21.15.40:0/4203964994 wait complete. 2024-03-18T22:40:56.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.004+0000 7fc0acab2640 1 Processor -- start 2024-03-18T22:40:56.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.004+0000 7fc0acab2640 1 -- start start 2024-03-18T22:40:56.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.004+0000 7fc0acab2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0a8100720 0x7fc0a8197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.004+0000 7fc0acab2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0a8101910 0x7fc0a8197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.004+0000 7fc0acab2640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0a81022d0 0x7fc0a819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.004+0000 7fc0acab2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a810d370 con 0x7fc0a8101910 2024-03-18T22:40:56.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.004+0000 7fc0acab2640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a810d1f0 con 0x7fc0a81022d0 2024-03-18T22:40:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.004+0000 7fc0acab2640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a810d4f0 con 0x7fc0a8100720 2024-03-18T22:40:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc0a5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0a8101910 0x7fc0a8197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc0a5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0a8101910 0x7fc0a8197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37730/0 (socket says 172.21.15.40:37730) 2024-03-18T22:40:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc0a5d9b640 1 -- 172.21.15.40:0/3122293266 learned_addr learned my addr 172.21.15.40:0/3122293266 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc0a6d9d640 1 --2- 172.21.15.40:0/3122293266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0a81022d0 0x7fc0a819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc0a659c640 1 --2- 172.21.15.40:0/3122293266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0a8100720 0x7fc0a8197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc0a6d9d640 1 -- 172.21.15.40:0/3122293266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0a8100720 msgr2=0x7fc0a8197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc0a6d9d640 1 --2- 172.21.15.40:0/3122293266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0a8100720 0x7fc0a8197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc0a6d9d640 1 -- 172.21.15.40:0/3122293266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0a8101910 msgr2=0x7fc0a8197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc0a6d9d640 1 --2- 172.21.15.40:0/3122293266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0a8101910 0x7fc0a8197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc0a6d9d640 1 -- 172.21.15.40:0/3122293266 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0a819f690 con 0x7fc0a81022d0 2024-03-18T22:40:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc0a5d9b640 1 --2- 172.21.15.40:0/3122293266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0a8101910 0x7fc0a8197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:40:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc0a6d9d640 1 --2- 172.21.15.40:0/3122293266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0a81022d0 0x7fc0a819f020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc098009fd0 tx=0x7fc09800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc09f7fe640 1 -- 172.21.15.40:0/3122293266 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09800ce20 con 0x7fc0a81022d0 2024-03-18T22:40:56.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc09f7fe640 1 -- 172.21.15.40:0/3122293266 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc098005020 con 0x7fc0a81022d0 2024-03-18T22:40:56.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc0acab2640 1 -- 172.21.15.40:0/3122293266 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0a819f980 con 0x7fc0a81022d0 2024-03-18T22:40:56.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc09f7fe640 1 -- 172.21.15.40:0/3122293266 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc098010690 con 0x7fc0a81022d0 2024-03-18T22:40:56.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.005+0000 7fc0acab2640 1 -- 172.21.15.40:0/3122293266 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0a819fed0 con 0x7fc0a81022d0 2024-03-18T22:40:56.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.006+0000 7fc0acab2640 1 -- 172.21.15.40:0/3122293266 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0a8107b60 con 0x7fc0a81022d0 2024-03-18T22:40:56.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.007+0000 7fc09f7fe640 1 -- 172.21.15.40:0/3122293266 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc0980051d0 con 0x7fc0a81022d0 2024-03-18T22:40:56.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.008+0000 7fc09f7fe640 1 --2- 172.21.15.40:0/3122293266 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc0940a2800 0x7fc0940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.008+0000 7fc0a659c640 1 --2- 172.21.15.40:0/3122293266 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc0940a2800 0x7fc0940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.008+0000 7fc09f7fe640 1 -- 172.21.15.40:0/3122293266 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc0980cf3d0 con 0x7fc0a81022d0 2024-03-18T22:40:56.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.008+0000 7fc0a659c640 1 --2- 172.21.15.40:0/3122293266 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc0940a2800 0x7fc0940a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8101770 tx=0x7fc0a0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:56.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.012+0000 7fc09f7fe640 1 -- 172.21.15.40:0/3122293266 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc09809c1d0 con 0x7fc0a81022d0 2024-03-18T22:40:56.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:55 smithi113 ceph-mon[66775]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-03-18T22:40:56.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:55 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2504621363' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:40:56.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:55 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3406159150' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:40:56.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:55 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/730826785' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T22:40:56.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:55 smithi103 ceph-mon[66377]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-03-18T22:40:56.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:55 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2504621363' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:40:56.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:55 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3406159150' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:40:56.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:55 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/730826785' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T22:40:56.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:55 smithi040 ceph-mon[62643]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-03-18T22:40:56.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:55 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2504621363' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:40:56.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:55 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3406159150' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:40:56.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:55 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/730826785' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T22:40:56.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.195+0000 7fc0acab2640 1 -- 172.21.15.40:0/3122293266 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc0a802d1f0 con 0x7fc0a81022d0 2024-03-18T22:40:56.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.196+0000 7fc09f7fe640 1 -- 172.21.15.40:0/3122293266 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fc09809bad0 con 0x7fc0a81022d0 2024-03-18T22:40:56.197 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-18T22:40:56.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.198+0000 7fc0acab2640 1 -- 172.21.15.40:0/3122293266 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc0940a2800 msgr2=0x7fc0940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.198+0000 7fc0acab2640 1 --2- 172.21.15.40:0/3122293266 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc0940a2800 0x7fc0940a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8101770 tx=0x7fc0a0005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:56.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.198+0000 7fc0acab2640 1 -- 172.21.15.40:0/3122293266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0a81022d0 msgr2=0x7fc0a819f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.198+0000 7fc0acab2640 1 --2- 172.21.15.40:0/3122293266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0a81022d0 0x7fc0a819f020 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc098009fd0 tx=0x7fc09800ef10 comp rx=0 tx=0).stop 2024-03-18T22:40:56.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.198+0000 7fc0acab2640 1 -- 172.21.15.40:0/3122293266 shutdown_connections 2024-03-18T22:40:56.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.198+0000 7fc0acab2640 1 --2- 172.21.15.40:0/3122293266 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc0940a2800 0x7fc0940a4cc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.198+0000 7fc0acab2640 1 --2- 172.21.15.40:0/3122293266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0a81022d0 0x7fc0a819f020 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.198+0000 7fc0acab2640 1 --2- 172.21.15.40:0/3122293266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0a8101910 0x7fc0a8197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.198+0000 7fc0acab2640 1 --2- 172.21.15.40:0/3122293266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc0a8100720 0x7fc0a8197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.198+0000 7fc0acab2640 1 -- 172.21.15.40:0/3122293266 >> 172.21.15.40:0/3122293266 conn(0x7fc0a80fbcb0 msgr2=0x7fc0a8104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:56.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.198+0000 7fc0acab2640 1 -- 172.21.15.40:0/3122293266 shutdown_connections 2024-03-18T22:40:56.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.198+0000 7fc0acab2640 1 -- 172.21.15.40:0/3122293266 wait complete. 2024-03-18T22:40:56.209 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T22:40:56.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.314+0000 7f6ce40d1640 1 -- 172.21.15.40:0/4136283480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cdc100730 msgr2=0x7f6cdc100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.314+0000 7f6ce40d1640 1 --2- 172.21.15.40:0/4136283480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cdc100730 0x7f6cdc100b50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4009900 tx=0x7f6cd402bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:56.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.314+0000 7f6ce40d1640 1 -- 172.21.15.40:0/4136283480 shutdown_connections 2024-03-18T22:40:56.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.314+0000 7f6ce40d1640 1 --2- 172.21.15.40:0/4136283480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cdc1022e0 0x7f6cdc108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.314+0000 7f6ce40d1640 1 --2- 172.21.15.40:0/4136283480 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cdc101920 0x7f6cdc101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.314+0000 7f6ce40d1640 1 --2- 172.21.15.40:0/4136283480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cdc100730 0x7f6cdc100b50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.314+0000 7f6ce40d1640 1 -- 172.21.15.40:0/4136283480 >> 172.21.15.40:0/4136283480 conn(0x7f6cdc0fbcc0 msgr2=0x7f6cdc0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:56.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.314+0000 7f6ce40d1640 1 -- 172.21.15.40:0/4136283480 shutdown_connections 2024-03-18T22:40:56.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.314+0000 7f6ce40d1640 1 -- 172.21.15.40:0/4136283480 wait complete. 2024-03-18T22:40:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.314+0000 7f6ce40d1640 1 Processor -- start 2024-03-18T22:40:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.314+0000 7f6ce40d1640 1 -- start start 2024-03-18T22:40:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce40d1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cdc100730 0x7f6cdc196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce40d1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cdc101920 0x7f6cdc196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce40d1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cdc1022e0 0x7f6cdc19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce40d1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc10c2d0 con 0x7f6cdc1022e0 2024-03-18T22:40:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce40d1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc10c150 con 0x7f6cdc100730 2024-03-18T22:40:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce40d1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc10c450 con 0x7f6cdc101920 2024-03-18T22:40:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce266e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cdc1022e0 0x7f6cdc19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce266e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cdc1022e0 0x7f6cdc19dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37746/0 (socket says 172.21.15.40:37746) 2024-03-18T22:40:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce266e640 1 -- 172.21.15.40:0/3108408071 learned_addr learned my addr 172.21.15.40:0/3108408071 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce1e6d640 1 --2- 172.21.15.40:0/3108408071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cdc100730 0x7f6cdc196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce166c640 1 --2- 172.21.15.40:0/3108408071 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cdc101920 0x7f6cdc196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce266e640 1 -- 172.21.15.40:0/3108408071 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cdc101920 msgr2=0x7f6cdc196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce266e640 1 --2- 172.21.15.40:0/3108408071 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cdc101920 0x7f6cdc196f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce266e640 1 -- 172.21.15.40:0/3108408071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cdc100730 msgr2=0x7f6cdc196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce266e640 1 --2- 172.21.15.40:0/3108408071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cdc100730 0x7f6cdc196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce266e640 1 -- 172.21.15.40:0/3108408071 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cdc19e650 con 0x7f6cdc1022e0 2024-03-18T22:40:56.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce266e640 1 --2- 172.21.15.40:0/3108408071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cdc1022e0 0x7f6cdc19dfe0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00b970 tx=0x7f6ccc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:56.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6cdaffd640 1 -- 172.21.15.40:0/3108408071 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc019070 con 0x7f6cdc1022e0 2024-03-18T22:40:56.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6cdaffd640 1 -- 172.21.15.40:0/3108408071 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ccc014440 con 0x7f6cdc1022e0 2024-03-18T22:40:56.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6cdaffd640 1 -- 172.21.15.40:0/3108408071 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc0075d0 con 0x7f6cdc1022e0 2024-03-18T22:40:56.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce40d1640 1 -- 172.21.15.40:0/3108408071 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cdc19e940 con 0x7f6cdc1022e0 2024-03-18T22:40:56.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.315+0000 7f6ce40d1640 1 -- 172.21.15.40:0/3108408071 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cdc19ef10 con 0x7f6cdc1022e0 2024-03-18T22:40:56.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.316+0000 7f6ce40d1640 1 -- 172.21.15.40:0/3108408071 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cdc107b70 con 0x7f6cdc1022e0 2024-03-18T22:40:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.318+0000 7f6cdaffd640 1 -- 172.21.15.40:0/3108408071 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f6ccc01c070 con 0x7f6cdc1022e0 2024-03-18T22:40:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.318+0000 7f6cdaffd640 1 --2- 172.21.15.40:0/3108408071 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6cc40a2880 0x7f6cc40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.318+0000 7f6ce1e6d640 1 --2- 172.21.15.40:0/3108408071 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6cc40a2880 0x7f6cc40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.318+0000 7f6cdaffd640 1 -- 172.21.15.40:0/3108408071 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6ccc0ce6a0 con 0x7f6cdc1022e0 2024-03-18T22:40:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.319+0000 7f6ce1e6d640 1 --2- 172.21.15.40:0/3108408071 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6cc40a2880 0x7f6cc40a4d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4005ab0 tx=0x7f6cd4005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:56.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.322+0000 7f6cdaffd640 1 -- 172.21.15.40:0/3108408071 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6ccc09aeb0 con 0x7f6cdc1022e0 2024-03-18T22:40:56.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.503+0000 7f6ce40d1640 1 -- 172.21.15.40:0/3108408071 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6cdc02d400 con 0x7f6cdc1022e0 2024-03-18T22:40:56.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.503+0000 7f6cdaffd640 1 -- 172.21.15.40:0/3108408071 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f6ccc09ef50 con 0x7f6cdc1022e0 2024-03-18T22:40:56.504 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:40:56.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.505+0000 7f6ce40d1640 1 -- 172.21.15.40:0/3108408071 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6cc40a2880 msgr2=0x7f6cc40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.505+0000 7f6ce40d1640 1 --2- 172.21.15.40:0/3108408071 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6cc40a2880 0x7f6cc40a4d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4005ab0 tx=0x7f6cd4005fd0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.505+0000 7f6ce40d1640 1 -- 172.21.15.40:0/3108408071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cdc1022e0 msgr2=0x7f6cdc19dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.505+0000 7f6ce40d1640 1 --2- 172.21.15.40:0/3108408071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cdc1022e0 0x7f6cdc19dfe0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc00b970 tx=0x7f6ccc00be30 comp rx=0 tx=0).stop 2024-03-18T22:40:56.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.505+0000 7f6ce40d1640 1 -- 172.21.15.40:0/3108408071 shutdown_connections 2024-03-18T22:40:56.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.505+0000 7f6ce40d1640 1 --2- 172.21.15.40:0/3108408071 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6cc40a2880 0x7f6cc40a4d40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.505+0000 7f6ce40d1640 1 --2- 172.21.15.40:0/3108408071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cdc1022e0 0x7f6cdc19dfe0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.505+0000 7f6ce40d1640 1 --2- 172.21.15.40:0/3108408071 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cdc101920 0x7f6cdc196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.505+0000 7f6ce40d1640 1 --2- 172.21.15.40:0/3108408071 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cdc100730 0x7f6cdc196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.505+0000 7f6ce40d1640 1 -- 172.21.15.40:0/3108408071 >> 172.21.15.40:0/3108408071 conn(0x7f6cdc0fbcc0 msgr2=0x7f6cdc104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:56.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.506+0000 7f6ce40d1640 1 -- 172.21.15.40:0/3108408071 shutdown_connections 2024-03-18T22:40:56.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.506+0000 7f6ce40d1640 1 -- 172.21.15.40:0/3108408071 wait complete. 2024-03-18T22:40:56.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T22:40:56.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.621+0000 7f723deea640 1 -- 172.21.15.40:0/3478080963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72381022e0 msgr2=0x7f7238108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.621+0000 7f723deea640 1 --2- 172.21.15.40:0/3478080963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72381022e0 0x7f7238108b70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7230009ca0 tx=0x7f723002c100 comp rx=0 tx=0).stop 2024-03-18T22:40:56.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.622+0000 7f723deea640 1 -- 172.21.15.40:0/3478080963 shutdown_connections 2024-03-18T22:40:56.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.622+0000 7f723deea640 1 --2- 172.21.15.40:0/3478080963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72381022e0 0x7f7238108b70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.622+0000 7f723deea640 1 --2- 172.21.15.40:0/3478080963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7238101920 0x7f7238101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.622+0000 7f723deea640 1 --2- 172.21.15.40:0/3478080963 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7238100730 0x7f7238100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.622+0000 7f723deea640 1 -- 172.21.15.40:0/3478080963 >> 172.21.15.40:0/3478080963 conn(0x7f72380fbcc0 msgr2=0x7f72380fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:56.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.622+0000 7f723deea640 1 -- 172.21.15.40:0/3478080963 shutdown_connections 2024-03-18T22:40:56.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.622+0000 7f723deea640 1 -- 172.21.15.40:0/3478080963 wait complete. 2024-03-18T22:40:56.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.622+0000 7f723deea640 1 Processor -- start 2024-03-18T22:40:56.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.622+0000 7f723deea640 1 -- start start 2024-03-18T22:40:56.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.622+0000 7f723deea640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7238100730 0x7f7238199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.622+0000 7f723deea640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7238101920 0x7f723819a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.622+0000 7f723deea640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72381022e0 0x7f72381a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.622+0000 7f723deea640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f723810c240 con 0x7f72381022e0 2024-03-18T22:40:56.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.622+0000 7f723deea640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f723810c0c0 con 0x7f7238100730 2024-03-18T22:40:56.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f723deea640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f723810c3c0 con 0x7f7238101920 2024-03-18T22:40:56.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f7237fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72381022e0 0x7f72381a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f72377fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7238100730 0x7f7238199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f7237fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72381022e0 0x7f72381a1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37764/0 (socket says 172.21.15.40:37764) 2024-03-18T22:40:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f7237fff640 1 -- 172.21.15.40:0/2960927230 learned_addr learned my addr 172.21.15.40:0/2960927230 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f7236ffd640 1 --2- 172.21.15.40:0/2960927230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7238101920 0x7f723819a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f7237fff640 1 -- 172.21.15.40:0/2960927230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7238101920 msgr2=0x7f723819a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f7237fff640 1 --2- 172.21.15.40:0/2960927230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7238101920 0x7f723819a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f7237fff640 1 -- 172.21.15.40:0/2960927230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7238100730 msgr2=0x7f7238199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f7237fff640 1 --2- 172.21.15.40:0/2960927230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7238100730 0x7f7238199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f7237fff640 1 -- 172.21.15.40:0/2960927230 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72381a17b0 con 0x7f72381022e0 2024-03-18T22:40:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f7236ffd640 1 --2- 172.21.15.40:0/2960927230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7238101920 0x7f723819a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:40:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f72377fe640 1 --2- 172.21.15.40:0/2960927230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7238100730 0x7f7238199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:40:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f7237fff640 1 --2- 172.21.15.40:0/2960927230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72381022e0 0x7f72381a1140 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f723002c710 tx=0x7f7230035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f7234ff9640 1 -- 172.21.15.40:0/2960927230 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230039040 con 0x7f72381022e0 2024-03-18T22:40:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f7234ff9640 1 -- 172.21.15.40:0/2960927230 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7230044070 con 0x7f72381022e0 2024-03-18T22:40:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f723deea640 1 -- 172.21.15.40:0/2960927230 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72381a1a40 con 0x7f72381022e0 2024-03-18T22:40:56.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f7234ff9640 1 -- 172.21.15.40:0/2960927230 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72300044a0 con 0x7f72381022e0 2024-03-18T22:40:56.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.623+0000 7f723deea640 1 -- 172.21.15.40:0/2960927230 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72381a1f30 con 0x7f72381022e0 2024-03-18T22:40:56.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.624+0000 7f723deea640 1 -- 172.21.15.40:0/2960927230 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72381a2160 con 0x7f72381022e0 2024-03-18T22:40:56.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.625+0000 7f7234ff9640 1 -- 172.21.15.40:0/2960927230 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f72300327c0 con 0x7f72381022e0 2024-03-18T22:40:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.626+0000 7f7234ff9640 1 --2- 172.21.15.40:0/2960927230 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72200a2800 0x7f72200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.626+0000 7f72377fe640 1 --2- 172.21.15.40:0/2960927230 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72200a2800 0x7f72200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.626+0000 7f7234ff9640 1 -- 172.21.15.40:0/2960927230 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f72300f1300 con 0x7f72381022e0 2024-03-18T22:40:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.626+0000 7f72377fe640 1 --2- 172.21.15.40:0/2960927230 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72200a2800 0x7f72200a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7238101780 tx=0x7f7228005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:56.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.630+0000 7f7234ff9640 1 -- 172.21.15.40:0/2960927230 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f72300be160 con 0x7f72381022e0 2024-03-18T22:40:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.815+0000 7f723deea640 1 -- 172.21.15.40:0/2960927230 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f72381a24f0 con 0x7f72381022e0 2024-03-18T22:40:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.816+0000 7f7234ff9640 1 -- 172.21.15.40:0/2960927230 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f72300bda60 con 0x7f72381022e0 2024-03-18T22:40:56.816 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-18T22:40:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.818+0000 7f723deea640 1 -- 172.21.15.40:0/2960927230 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72200a2800 msgr2=0x7f72200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.818+0000 7f723deea640 1 --2- 172.21.15.40:0/2960927230 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72200a2800 0x7f72200a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7238101780 tx=0x7f7228005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.818+0000 7f723deea640 1 -- 172.21.15.40:0/2960927230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72381022e0 msgr2=0x7f72381a1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.818+0000 7f723deea640 1 --2- 172.21.15.40:0/2960927230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72381022e0 0x7f72381a1140 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f723002c710 tx=0x7f7230035d60 comp rx=0 tx=0).stop 2024-03-18T22:40:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.818+0000 7f723deea640 1 -- 172.21.15.40:0/2960927230 shutdown_connections 2024-03-18T22:40:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.818+0000 7f723deea640 1 --2- 172.21.15.40:0/2960927230 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72200a2800 0x7f72200a4cc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.818+0000 7f723deea640 1 --2- 172.21.15.40:0/2960927230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72381022e0 0x7f72381a1140 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.818+0000 7f723deea640 1 --2- 172.21.15.40:0/2960927230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7238101920 0x7f723819a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.818+0000 7f723deea640 1 --2- 172.21.15.40:0/2960927230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7238100730 0x7f7238199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.818+0000 7f723deea640 1 -- 172.21.15.40:0/2960927230 >> 172.21.15.40:0/2960927230 conn(0x7f72380fbcc0 msgr2=0x7f72380fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:56.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.818+0000 7f723deea640 1 -- 172.21.15.40:0/2960927230 shutdown_connections 2024-03-18T22:40:56.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.818+0000 7f723deea640 1 -- 172.21.15.40:0/2960927230 wait complete. 2024-03-18T22:40:56.832 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T22:40:56.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.936+0000 7fb0c74bd640 1 -- 172.21.15.40:0/3496622614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c0100720 msgr2=0x7fb0c0100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.936+0000 7fb0c74bd640 1 --2- 172.21.15.40:0/3496622614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c0100720 0x7fb0c0100b40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009900 tx=0x7fb0b802bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:56.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.936+0000 7fb0c74bd640 1 -- 172.21.15.40:0/3496622614 shutdown_connections 2024-03-18T22:40:56.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.936+0000 7fb0c74bd640 1 --2- 172.21.15.40:0/3496622614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0c01022d0 0x7fb0c0108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.936+0000 7fb0c74bd640 1 --2- 172.21.15.40:0/3496622614 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0c0101910 0x7fb0c0101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.936+0000 7fb0c74bd640 1 --2- 172.21.15.40:0/3496622614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c0100720 0x7fb0c0100b40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.936+0000 7fb0c74bd640 1 -- 172.21.15.40:0/3496622614 >> 172.21.15.40:0/3496622614 conn(0x7fb0c00fbcb0 msgr2=0x7fb0c00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.937+0000 7fb0c74bd640 1 -- 172.21.15.40:0/3496622614 shutdown_connections 2024-03-18T22:40:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.937+0000 7fb0c74bd640 1 -- 172.21.15.40:0/3496622614 wait complete. 2024-03-18T22:40:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.937+0000 7fb0c74bd640 1 Processor -- start 2024-03-18T22:40:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.937+0000 7fb0c74bd640 1 -- start start 2024-03-18T22:40:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.937+0000 7fb0c74bd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0c0100720 0x7fb0c0197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.937+0000 7fb0c74bd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c0101910 0x7fb0c0198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.937+0000 7fb0c74bd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0c01022d0 0x7fb0c019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.937+0000 7fb0c74bd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c010c240 con 0x7fb0c0101910 2024-03-18T22:40:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.937+0000 7fb0c74bd640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c010c0c0 con 0x7fb0c0100720 2024-03-18T22:40:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.937+0000 7fb0c74bd640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c010c3c0 con 0x7fb0c01022d0 2024-03-18T22:40:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.937+0000 7fb0c4a58640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c0101910 0x7fb0c0198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.937+0000 7fb0c4a58640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c0101910 0x7fb0c0198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37780/0 (socket says 172.21.15.40:37780) 2024-03-18T22:40:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.937+0000 7fb0c4a58640 1 -- 172.21.15.40:0/574311843 learned_addr learned my addr 172.21.15.40:0/574311843 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.937+0000 7fb0c5259640 1 --2- 172.21.15.40:0/574311843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0c0100720 0x7fb0c0197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.937+0000 7fb0c5a5a640 1 --2- 172.21.15.40:0/574311843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0c01022d0 0x7fb0c019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.938+0000 7fb0c4a58640 1 -- 172.21.15.40:0/574311843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0c01022d0 msgr2=0x7fb0c019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.938+0000 7fb0c4a58640 1 --2- 172.21.15.40:0/574311843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0c01022d0 0x7fb0c019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.938+0000 7fb0c4a58640 1 -- 172.21.15.40:0/574311843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0c0100720 msgr2=0x7fb0c0197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.938+0000 7fb0c4a58640 1 --2- 172.21.15.40:0/574311843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0c0100720 0x7fb0c0197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.938+0000 7fb0c4a58640 1 -- 172.21.15.40:0/574311843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c019f710 con 0x7fb0c0101910 2024-03-18T22:40:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.938+0000 7fb0c4a58640 1 --2- 172.21.15.40:0/574311843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c0101910 0x7fb0c0198030 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb0b400d770 tx=0x7fb0b400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.938+0000 7fb0be7fc640 1 -- 172.21.15.40:0/574311843 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4017070 con 0x7fb0c0101910 2024-03-18T22:40:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.938+0000 7fb0be7fc640 1 -- 172.21.15.40:0/574311843 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0b40054d0 con 0x7fb0c0101910 2024-03-18T22:40:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.938+0000 7fb0c74bd640 1 -- 172.21.15.40:0/574311843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c019fa00 con 0x7fb0c0101910 2024-03-18T22:40:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.938+0000 7fb0be7fc640 1 -- 172.21.15.40:0/574311843 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4004cf0 con 0x7fb0c0101910 2024-03-18T22:40:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.938+0000 7fb0c74bd640 1 -- 172.21.15.40:0/574311843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c019ff00 con 0x7fb0c0101910 2024-03-18T22:40:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.939+0000 7fb0c74bd640 1 -- 172.21.15.40:0/574311843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0c01a0130 con 0x7fb0c0101910 2024-03-18T22:40:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.940+0000 7fb0be7fc640 1 -- 172.21.15.40:0/574311843 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb0b4004e90 con 0x7fb0c0101910 2024-03-18T22:40:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.941+0000 7fb0be7fc640 1 --2- 172.21.15.40:0/574311843 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0a80a2800 0x7fb0a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.941+0000 7fb0c5259640 1 --2- 172.21.15.40:0/574311843 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0a80a2800 0x7fb0a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.941+0000 7fb0be7fc640 1 -- 172.21.15.40:0/574311843 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb0b40ce6b0 con 0x7fb0c0101910 2024-03-18T22:40:56.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.941+0000 7fb0c5259640 1 --2- 172.21.15.40:0/574311843 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0a80a2800 0x7fb0a80a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8005ab0 tx=0x7fb0b8005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:56.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:56.945+0000 7fb0be7fc640 1 -- 172.21.15.40:0/574311843 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb0b409ae60 con 0x7fb0c0101910 2024-03-18T22:40:57.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:56 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/30062867' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:57.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:56 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3122293266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:40:57.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:56 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3108408071' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:40:57.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:56 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2960927230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:40:57.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:56 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/30062867' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:57.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:56 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3122293266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:40:57.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:56 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3108408071' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:40:57.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:56 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2960927230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:40:57.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.127+0000 7fb0c74bd640 1 -- 172.21.15.40:0/574311843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb0c01a0300 con 0x7fb0c0101910 2024-03-18T22:40:57.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.127+0000 7fb0be7fc640 1 -- 172.21.15.40:0/574311843 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7fb0b409ef00 con 0x7fb0c0101910 2024-03-18T22:40:57.128 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:40:57.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.129+0000 7fb0c74bd640 1 -- 172.21.15.40:0/574311843 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0a80a2800 msgr2=0x7fb0a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:57.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.129+0000 7fb0c74bd640 1 --2- 172.21.15.40:0/574311843 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0a80a2800 0x7fb0a80a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8005ab0 tx=0x7fb0b8005fd0 comp rx=0 tx=0).stop 2024-03-18T22:40:57.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.129+0000 7fb0c74bd640 1 -- 172.21.15.40:0/574311843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c0101910 msgr2=0x7fb0c0198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.129+0000 7fb0c74bd640 1 --2- 172.21.15.40:0/574311843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c0101910 0x7fb0c0198030 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb0b400d770 tx=0x7fb0b400dc30 comp rx=0 tx=0).stop 2024-03-18T22:40:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.129+0000 7fb0c74bd640 1 -- 172.21.15.40:0/574311843 shutdown_connections 2024-03-18T22:40:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.129+0000 7fb0c74bd640 1 --2- 172.21.15.40:0/574311843 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0a80a2800 0x7fb0a80a4cc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.129+0000 7fb0c74bd640 1 --2- 172.21.15.40:0/574311843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0c01022d0 0x7fb0c019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.129+0000 7fb0c74bd640 1 --2- 172.21.15.40:0/574311843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c0101910 0x7fb0c0198030 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.129+0000 7fb0c74bd640 1 --2- 172.21.15.40:0/574311843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0c0100720 0x7fb0c0197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.129+0000 7fb0c74bd640 1 -- 172.21.15.40:0/574311843 >> 172.21.15.40:0/574311843 conn(0x7fb0c00fbcb0 msgr2=0x7fb0c00689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.130+0000 7fb0c74bd640 1 -- 172.21.15.40:0/574311843 shutdown_connections 2024-03-18T22:40:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.130+0000 7fb0c74bd640 1 -- 172.21.15.40:0/574311843 wait complete. 2024-03-18T22:40:57.140 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:56 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/30062867' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:40:57.140 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:56 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3122293266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:40:57.140 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:56 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3108408071' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:40:57.140 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:56 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2960927230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:40:57.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T22:40:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.244+0000 7fec9d80c640 1 -- 172.21.15.40:0/1345975580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec98101930 msgr2=0x7fec98101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.244+0000 7fec9d80c640 1 --2- 172.21.15.40:0/1345975580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec98101930 0x7fec98101db0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fec8c009900 tx=0x7fec8c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:40:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.244+0000 7fec9d80c640 1 -- 172.21.15.40:0/1345975580 shutdown_connections 2024-03-18T22:40:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.244+0000 7fec9d80c640 1 --2- 172.21.15.40:0/1345975580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec981022f0 0x7fec98108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.244+0000 7fec9d80c640 1 --2- 172.21.15.40:0/1345975580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec98101930 0x7fec98101db0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.244+0000 7fec9d80c640 1 --2- 172.21.15.40:0/1345975580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec98100740 0x7fec98100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.244+0000 7fec9d80c640 1 -- 172.21.15.40:0/1345975580 >> 172.21.15.40:0/1345975580 conn(0x7fec980fbcd0 msgr2=0x7fec980fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.244+0000 7fec9d80c640 1 -- 172.21.15.40:0/1345975580 shutdown_connections 2024-03-18T22:40:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.244+0000 7fec9d80c640 1 -- 172.21.15.40:0/1345975580 wait complete. 2024-03-18T22:40:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.244+0000 7fec9d80c640 1 Processor -- start 2024-03-18T22:40:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec9d80c640 1 -- start start 2024-03-18T22:40:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec9d80c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec98100740 0x7fec98197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec9d80c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec98101930 0x7fec98198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec9d80c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec981022f0 0x7fec9819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec9d80c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fec9810c270 con 0x7fec98101930 2024-03-18T22:40:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec9d80c640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fec9810c0f0 con 0x7fec981022f0 2024-03-18T22:40:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec9d80c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fec9810c3f0 con 0x7fec98100740 2024-03-18T22:40:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec967fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec98101930 0x7fec98198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec967fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec98101930 0x7fec98198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37796/0 (socket says 172.21.15.40:37796) 2024-03-18T22:40:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec967fc640 1 -- 172.21.15.40:0/2751648282 learned_addr learned my addr 172.21.15.40:0/2751648282 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:40:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec977fe640 1 --2- 172.21.15.40:0/2751648282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec981022f0 0x7fec9819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec96ffd640 1 --2- 172.21.15.40:0/2751648282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec98100740 0x7fec98197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec967fc640 1 -- 172.21.15.40:0/2751648282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec98100740 msgr2=0x7fec98197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec967fc640 1 --2- 172.21.15.40:0/2751648282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec98100740 0x7fec98197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec967fc640 1 -- 172.21.15.40:0/2751648282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec981022f0 msgr2=0x7fec9819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec967fc640 1 --2- 172.21.15.40:0/2751648282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec981022f0 0x7fec9819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec967fc640 1 -- 172.21.15.40:0/2751648282 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9819f6e0 con 0x7fec98101930 2024-03-18T22:40:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec96ffd640 1 --2- 172.21.15.40:0/2751648282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec98100740 0x7fec98197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:40:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.245+0000 7fec967fc640 1 --2- 172.21.15.40:0/2751648282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec98101930 0x7fec98198000 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fec8c006010 tx=0x7fec8c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.246+0000 7fec9c80a640 1 -- 172.21.15.40:0/2751648282 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8c043020 con 0x7fec98101930 2024-03-18T22:40:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.246+0000 7fec9d80c640 1 -- 172.21.15.40:0/2751648282 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec9819f970 con 0x7fec98101930 2024-03-18T22:40:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.246+0000 7fec9c80a640 1 -- 172.21.15.40:0/2751648282 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec8c0043d0 con 0x7fec98101930 2024-03-18T22:40:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.246+0000 7fec9c80a640 1 -- 172.21.15.40:0/2751648282 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8c003b80 con 0x7fec98101930 2024-03-18T22:40:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.246+0000 7fec9d80c640 1 -- 172.21.15.40:0/2751648282 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec9819fec0 con 0x7fec98101930 2024-03-18T22:40:57.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.247+0000 7fec9d80c640 1 -- 172.21.15.40:0/2751648282 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec981a0090 con 0x7fec98101930 2024-03-18T22:40:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.248+0000 7fec9c80a640 1 -- 172.21.15.40:0/2751648282 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fec8c046020 con 0x7fec98101930 2024-03-18T22:40:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.248+0000 7fec9c80a640 1 --2- 172.21.15.40:0/2751648282 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec800a2800 0x7fec800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:40:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.248+0000 7fec96ffd640 1 --2- 172.21.15.40:0/2751648282 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec800a2800 0x7fec800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:40:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.248+0000 7fec9c80a640 1 -- 172.21.15.40:0/2751648282 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fec8c0f0f40 con 0x7fec98101930 2024-03-18T22:40:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.249+0000 7fec96ffd640 1 --2- 172.21.15.40:0/2751648282 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec800a2800 0x7fec800a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fec98101790 tx=0x7fec90005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:40:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.253+0000 7fec9c80a640 1 -- 172.21.15.40:0/2751648282 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fec8c04d020 con 0x7fec98101930 2024-03-18T22:40:57.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.448+0000 7fec9d80c640 1 -- 172.21.15.40:0/2751648282 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fec981a0260 con 0x7fec98101930 2024-03-18T22:40:57.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.448+0000 7fec9c80a640 1 -- 172.21.15.40:0/2751648282 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+591 (secure 0 0 0) 0x7fec8c0bdda0 con 0x7fec98101930 2024-03-18T22:40:57.449 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/347668672 2024-03-19T22:33:42.721571+0000 2024-03-18T22:40:57.449 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/4237962144 2024-03-19T22:33:42.721571+0000 2024-03-18T22:40:57.449 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/1454961757 2024-03-19T22:33:15.826478+0000 2024-03-18T22:40:57.449 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3047621005 2024-03-19T22:33:15.826478+0000 2024-03-18T22:40:57.450 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/2121391701 2024-03-19T22:33:15.826478+0000 2024-03-18T22:40:57.450 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/3876208039 2024-03-19T22:33:42.721571+0000 2024-03-18T22:40:57.450 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/3876208039 2024-03-19T22:33:42.721571+0000 2024-03-18T22:40:57.450 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/2591529486 2024-03-19T22:33:42.721571+0000 2024-03-18T22:40:57.450 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/1153335782 2024-03-19T22:33:15.826478+0000 2024-03-18T22:40:57.450 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/1153335782 2024-03-19T22:33:15.826478+0000 2024-03-18T22:40:57.450 INFO:teuthology.orchestra.run.smithi040.stderr:listed 10 entries 2024-03-18T22:40:57.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.451+0000 7fec9d80c640 1 -- 172.21.15.40:0/2751648282 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec800a2800 msgr2=0x7fec800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:57.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.451+0000 7fec9d80c640 1 --2- 172.21.15.40:0/2751648282 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec800a2800 0x7fec800a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fec98101790 tx=0x7fec90005f90 comp rx=0 tx=0).stop 2024-03-18T22:40:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.451+0000 7fec9d80c640 1 -- 172.21.15.40:0/2751648282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec98101930 msgr2=0x7fec98198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:40:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.451+0000 7fec9d80c640 1 --2- 172.21.15.40:0/2751648282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec98101930 0x7fec98198000 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fec8c006010 tx=0x7fec8c005e40 comp rx=0 tx=0).stop 2024-03-18T22:40:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.451+0000 7fec9d80c640 1 -- 172.21.15.40:0/2751648282 shutdown_connections 2024-03-18T22:40:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.451+0000 7fec9d80c640 1 --2- 172.21.15.40:0/2751648282 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec800a2800 0x7fec800a4cc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.451+0000 7fec9d80c640 1 --2- 172.21.15.40:0/2751648282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec981022f0 0x7fec9819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.451+0000 7fec9d80c640 1 --2- 172.21.15.40:0/2751648282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec98101930 0x7fec98198000 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.451+0000 7fec9d80c640 1 --2- 172.21.15.40:0/2751648282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec98100740 0x7fec98197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:40:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.451+0000 7fec9d80c640 1 -- 172.21.15.40:0/2751648282 >> 172.21.15.40:0/2751648282 conn(0x7fec980fbcd0 msgr2=0x7fec980fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:40:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.451+0000 7fec9d80c640 1 -- 172.21.15.40:0/2751648282 shutdown_connections 2024-03-18T22:40:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:40:57.451+0000 7fec9d80c640 1 -- 172.21.15.40:0/2751648282 wait complete. 2024-03-18T22:40:57.462 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi040.front.sepia.ceph.com... 2024-03-18T22:40:57.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:40:57.462 DEBUG:teuthology.orchestra.run.smithi040:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-18T22:40:57.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi040.stderr:ceph-fuse[103476]: fuse finished with error 0 and tester_r 0 2024-03-18T22:40:57.489 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T22:40:58.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:57 smithi113 ceph-mon[66775]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 0 op/s 2024-03-18T22:40:58.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:57 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/574311843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:40:58.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:57 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2751648282' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T22:40:58.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:57 smithi103 ceph-mon[66377]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 0 op/s 2024-03-18T22:40:58.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:57 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/574311843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:40:58.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:57 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2751648282' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T22:40:58.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:57 smithi040 ceph-mon[62643]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 0 op/s 2024-03-18T22:40:58.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:57 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/574311843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:40:58.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:57 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2751648282' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T22:40:59.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:40:58 smithi103 ceph-mon[66377]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 256 B/s wr, 0 op/s 2024-03-18T22:40:59.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:40:58 smithi040 ceph-mon[62643]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 256 B/s wr, 0 op/s 2024-03-18T22:40:59.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:40:58 smithi113 ceph-mon[66775]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 256 B/s wr, 0 op/s 2024-03-18T22:41:01.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:00 smithi040 ceph-mon[62643]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T22:41:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:00 smithi113 ceph-mon[66775]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T22:41:01.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:00 smithi103 ceph-mon[66377]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T22:41:03.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:02 smithi040 ceph-mon[62643]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T22:41:03.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:02 smithi113 ceph-mon[66775]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T22:41:03.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:02 smithi103 ceph-mon[66377]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T22:41:03.591 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-18T22:41:03.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:41:03.591 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-18T22:41:03.613 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:41:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.770+0000 7ff18266a640 1 -- 172.21.15.40:0/1138854549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff17c1033f0 msgr2=0x7ff17c109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.770+0000 7ff18266a640 1 --2- 172.21.15.40:0/1138854549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff17c1033f0 0x7ff17c109c80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff174009ca0 tx=0x7ff17402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:41:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.770+0000 7ff18266a640 1 -- 172.21.15.40:0/1138854549 shutdown_connections 2024-03-18T22:41:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.770+0000 7ff18266a640 1 --2- 172.21.15.40:0/1138854549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff17c1033f0 0x7ff17c109c80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.770+0000 7ff18266a640 1 --2- 172.21.15.40:0/1138854549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff17c102a30 0x7ff17c102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.770+0000 7ff18266a640 1 --2- 172.21.15.40:0/1138854549 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff17c101840 0x7ff17c101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.770+0000 7ff18266a640 1 -- 172.21.15.40:0/1138854549 >> 172.21.15.40:0/1138854549 conn(0x7ff17c0fcdd0 msgr2=0x7ff17c0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.770+0000 7ff18266a640 1 -- 172.21.15.40:0/1138854549 shutdown_connections 2024-03-18T22:41:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.770+0000 7ff18266a640 1 -- 172.21.15.40:0/1138854549 wait complete. 2024-03-18T22:41:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.770+0000 7ff18266a640 1 Processor -- start 2024-03-18T22:41:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff18266a640 1 -- start start 2024-03-18T22:41:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff18266a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff17c101840 0x7ff17c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff18266a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff17c102a30 0x7ff17c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff18266a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff17c1033f0 0x7ff17c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff18266a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c10d370 con 0x7ff17c101840 2024-03-18T22:41:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff18266a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c10d1f0 con 0x7ff17c1033f0 2024-03-18T22:41:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff18266a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff17c10d4f0 con 0x7ff17c102a30 2024-03-18T22:41:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff17bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff17c101840 0x7ff17c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff17bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff17c101840 0x7ff17c197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50166/0 (socket says 172.21.15.40:50166) 2024-03-18T22:41:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff17bfff640 1 -- 172.21.15.40:0/1830760676 learned_addr learned my addr 172.21.15.40:0/1830760676 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff17b7fe640 1 --2- 172.21.15.40:0/1830760676 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff17c102a30 0x7ff17c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff180c07640 1 --2- 172.21.15.40:0/1830760676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff17c1033f0 0x7ff17c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:03.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff17bfff640 1 -- 172.21.15.40:0/1830760676 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff17c102a30 msgr2=0x7ff17c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:03.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff17bfff640 1 --2- 172.21.15.40:0/1830760676 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff17c102a30 0x7ff17c197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:03.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff17bfff640 1 -- 172.21.15.40:0/1830760676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff17c1033f0 msgr2=0x7ff17c19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:03.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff17bfff640 1 --2- 172.21.15.40:0/1830760676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff17c1033f0 0x7ff17c19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:03.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff17bfff640 1 -- 172.21.15.40:0/1830760676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff17c19f620 con 0x7ff17c101840 2024-03-18T22:41:03.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.771+0000 7ff180c07640 1 --2- 172.21.15.40:0/1830760676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff17c1033f0 0x7ff17c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:03.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.772+0000 7ff17bfff640 1 --2- 172.21.15.40:0/1830760676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff17c101840 0x7ff17c197a00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff16c00d950 tx=0x7ff16c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:03.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.772+0000 7ff1797fa640 1 -- 172.21.15.40:0/1830760676 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c017070 con 0x7ff17c101840 2024-03-18T22:41:03.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.772+0000 7ff1797fa640 1 -- 172.21.15.40:0/1830760676 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff16c009c30 con 0x7ff17c101840 2024-03-18T22:41:03.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.772+0000 7ff18266a640 1 -- 172.21.15.40:0/1830760676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff17c19f910 con 0x7ff17c101840 2024-03-18T22:41:03.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.772+0000 7ff1797fa640 1 -- 172.21.15.40:0/1830760676 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c00f620 con 0x7ff17c101840 2024-03-18T22:41:03.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.772+0000 7ff18266a640 1 -- 172.21.15.40:0/1830760676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff17c19fe10 con 0x7ff17c101840 2024-03-18T22:41:03.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.773+0000 7ff18266a640 1 -- 172.21.15.40:0/1830760676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff17c108c00 con 0x7ff17c101840 2024-03-18T22:41:03.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.774+0000 7ff1797fa640 1 -- 172.21.15.40:0/1830760676 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff16c01a070 con 0x7ff17c101840 2024-03-18T22:41:03.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.774+0000 7ff1797fa640 1 --2- 172.21.15.40:0/1830760676 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1680a2800 0x7ff1680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:03.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.775+0000 7ff17b7fe640 1 --2- 172.21.15.40:0/1830760676 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1680a2800 0x7ff1680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:03.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.775+0000 7ff1797fa640 1 -- 172.21.15.40:0/1830760676 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff16c0ce640 con 0x7ff17c101840 2024-03-18T22:41:03.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.775+0000 7ff17b7fe640 1 --2- 172.21.15.40:0/1830760676 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1680a2800 0x7ff1680a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff17c198f10 tx=0x7ff164005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:03.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.778+0000 7ff1797fa640 1 -- 172.21.15.40:0/1830760676 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff16c009de0 con 0x7ff17c101840 2024-03-18T22:41:03.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.963+0000 7ff18266a640 1 -- 172.21.15.40:0/1830760676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff17c02d390 con 0x7ff17c101840 2024-03-18T22:41:03.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.964+0000 7ff1797fa640 1 -- 172.21.15.40:0/1830760676 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24129 (secure 0 0 0) 0x7ff16c09ae50 con 0x7ff17c101840 2024-03-18T22:41:03.965 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:41:03.965 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":89,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:40:46.196741+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:41:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.966+0000 7ff18266a640 1 -- 172.21.15.40:0/1830760676 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1680a2800 msgr2=0x7ff1680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.966+0000 7ff18266a640 1 --2- 172.21.15.40:0/1830760676 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1680a2800 0x7ff1680a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff17c198f10 tx=0x7ff164005f90 comp rx=0 tx=0).stop 2024-03-18T22:41:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.966+0000 7ff18266a640 1 -- 172.21.15.40:0/1830760676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff17c101840 msgr2=0x7ff17c197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.966+0000 7ff18266a640 1 --2- 172.21.15.40:0/1830760676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff17c101840 0x7ff17c197a00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff16c00d950 tx=0x7ff16c00de10 comp rx=0 tx=0).stop 2024-03-18T22:41:03.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.967+0000 7ff18266a640 1 -- 172.21.15.40:0/1830760676 shutdown_connections 2024-03-18T22:41:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.967+0000 7ff18266a640 1 --2- 172.21.15.40:0/1830760676 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1680a2800 0x7ff1680a4cc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.967+0000 7ff18266a640 1 --2- 172.21.15.40:0/1830760676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff17c1033f0 0x7ff17c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.967+0000 7ff18266a640 1 --2- 172.21.15.40:0/1830760676 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff17c102a30 0x7ff17c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.967+0000 7ff18266a640 1 --2- 172.21.15.40:0/1830760676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff17c101840 0x7ff17c197a00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.967+0000 7ff18266a640 1 -- 172.21.15.40:0/1830760676 >> 172.21.15.40:0/1830760676 conn(0x7ff17c0fcdd0 msgr2=0x7ff17c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.967+0000 7ff18266a640 1 -- 172.21.15.40:0/1830760676 shutdown_connections 2024-03-18T22:41:03.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:03.967+0000 7ff18266a640 1 -- 172.21.15.40:0/1830760676 wait complete. 2024-03-18T22:41:03.979 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:41:04.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.081+0000 7f1085ae9640 1 -- 172.21.15.40:0/2964767218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1080100a90 msgr2=0x7f1080100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:04.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.081+0000 7f1085ae9640 1 --2- 172.21.15.40:0/2964767218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1080100a90 0x7f1080100ef0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1074009900 tx=0x7f107402bcd0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.081+0000 7f1085ae9640 1 -- 172.21.15.40:0/2964767218 shutdown_connections 2024-03-18T22:41:04.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.081+0000 7f1085ae9640 1 --2- 172.21.15.40:0/2964767218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1080101430 0x7f10801018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.081+0000 7f1085ae9640 1 --2- 172.21.15.40:0/2964767218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1080100a90 0x7f1080100ef0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.081+0000 7f1085ae9640 1 --2- 172.21.15.40:0/2964767218 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1080100130 0x7f1080100550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.081+0000 7f1085ae9640 1 -- 172.21.15.40:0/2964767218 >> 172.21.15.40:0/2964767218 conn(0x7f10800fbcb0 msgr2=0x7f10800fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:04.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.081+0000 7f1085ae9640 1 -- 172.21.15.40:0/2964767218 shutdown_connections 2024-03-18T22:41:04.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.081+0000 7f1085ae9640 1 -- 172.21.15.40:0/2964767218 wait complete. 2024-03-18T22:41:04.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f1085ae9640 1 Processor -- start 2024-03-18T22:41:04.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f1085ae9640 1 -- start start 2024-03-18T22:41:04.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f1085ae9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1080100130 0x7f1080199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:04.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f1085ae9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1080100a90 0x7f108019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:04.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f1085ae9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1080101430 0x7f10801a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:04.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f1085ae9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f108010f550 con 0x7f1080101430 2024-03-18T22:41:04.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f1085ae9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f108010f3d0 con 0x7f1080100a90 2024-03-18T22:41:04.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f1085ae9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f108010f6d0 con 0x7f1080100130 2024-03-18T22:41:04.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f107ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1080101430 0x7f10801a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:04.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f107ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1080101430 0x7f10801a1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50192/0 (socket says 172.21.15.40:50192) 2024-03-18T22:41:04.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f107ffff640 1 -- 172.21.15.40:0/3031576481 learned_addr learned my addr 172.21.15.40:0/3031576481 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:04.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f107effd640 1 --2- 172.21.15.40:0/3031576481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1080100a90 0x7f108019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:04.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f107f7fe640 1 --2- 172.21.15.40:0/3031576481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1080100130 0x7f1080199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:04.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f107ffff640 1 -- 172.21.15.40:0/3031576481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1080100130 msgr2=0x7f1080199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:04.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f107ffff640 1 --2- 172.21.15.40:0/3031576481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1080100130 0x7f1080199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f107ffff640 1 -- 172.21.15.40:0/3031576481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1080100a90 msgr2=0x7f108019a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:04.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f107ffff640 1 --2- 172.21.15.40:0/3031576481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1080100a90 0x7f108019a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.082+0000 7f107ffff640 1 -- 172.21.15.40:0/3031576481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10801a18c0 con 0x7f1080101430 2024-03-18T22:41:04.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.083+0000 7f107ffff640 1 --2- 172.21.15.40:0/3031576481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1080101430 0x7f10801a1220 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f107800eb10 tx=0x7f107800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:04.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.083+0000 7f107cff9640 1 -- 172.21.15.40:0/3031576481 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1078004b20 con 0x7f1080101430 2024-03-18T22:41:04.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.083+0000 7f1085ae9640 1 -- 172.21.15.40:0/3031576481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10801a1bb0 con 0x7f1080101430 2024-03-18T22:41:04.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.083+0000 7f107cff9640 1 -- 172.21.15.40:0/3031576481 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1078004cc0 con 0x7f1080101430 2024-03-18T22:41:04.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.083+0000 7f107cff9640 1 -- 172.21.15.40:0/3031576481 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1078010690 con 0x7f1080101430 2024-03-18T22:41:04.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.083+0000 7f1085ae9640 1 -- 172.21.15.40:0/3031576481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10801a2080 con 0x7f1080101430 2024-03-18T22:41:04.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.084+0000 7f1085ae9640 1 -- 172.21.15.40:0/3031576481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f108010b0d0 con 0x7f1080101430 2024-03-18T22:41:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.085+0000 7f107cff9640 1 -- 172.21.15.40:0/3031576481 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1078005000 con 0x7f1080101430 2024-03-18T22:41:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.085+0000 7f107cff9640 1 --2- 172.21.15.40:0/3031576481 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f10680a2800 0x7f10680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.086+0000 7f107cff9640 1 -- 172.21.15.40:0/3031576481 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f10780cf500 con 0x7f1080101430 2024-03-18T22:41:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.086+0000 7f107f7fe640 1 --2- 172.21.15.40:0/3031576481 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f10680a2800 0x7f10680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.086+0000 7f107f7fe640 1 --2- 172.21.15.40:0/3031576481 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f10680a2800 0x7f10680a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1070005fd0 tx=0x7f1070005e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.089+0000 7f107cff9640 1 -- 172.21.15.40:0/3031576481 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1078053740 con 0x7f1080101430 2024-03-18T22:41:04.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.273+0000 7f1085ae9640 1 -- 172.21.15.40:0/3031576481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f108002d3c0 con 0x7f1080101430 2024-03-18T22:41:04.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.275+0000 7f107cff9640 1 -- 172.21.15.40:0/3031576481 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24129 (secure 0 0 0) 0x7f107809c300 con 0x7f1080101430 2024-03-18T22:41:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:41:04.276 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":89,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:40:46.196741+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:41:04.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.277+0000 7f1085ae9640 1 -- 172.21.15.40:0/3031576481 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f10680a2800 msgr2=0x7f10680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.277+0000 7f1085ae9640 1 --2- 172.21.15.40:0/3031576481 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f10680a2800 0x7f10680a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1070005fd0 tx=0x7f1070005e40 comp rx=0 tx=0).stop 2024-03-18T22:41:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.277+0000 7f1085ae9640 1 -- 172.21.15.40:0/3031576481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1080101430 msgr2=0x7f10801a1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.277+0000 7f1085ae9640 1 --2- 172.21.15.40:0/3031576481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1080101430 0x7f10801a1220 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f107800eb10 tx=0x7f107800c2d0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.277+0000 7f1085ae9640 1 -- 172.21.15.40:0/3031576481 shutdown_connections 2024-03-18T22:41:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.277+0000 7f1085ae9640 1 --2- 172.21.15.40:0/3031576481 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f10680a2800 0x7f10680a4cc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.277+0000 7f1085ae9640 1 --2- 172.21.15.40:0/3031576481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1080101430 0x7f10801a1220 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.277+0000 7f1085ae9640 1 --2- 172.21.15.40:0/3031576481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1080100a90 0x7f108019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.277+0000 7f1085ae9640 1 --2- 172.21.15.40:0/3031576481 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1080100130 0x7f1080199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.277+0000 7f1085ae9640 1 -- 172.21.15.40:0/3031576481 >> 172.21.15.40:0/3031576481 conn(0x7f10800fbcb0 msgr2=0x7f10800fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.277+0000 7f1085ae9640 1 -- 172.21.15.40:0/3031576481 shutdown_connections 2024-03-18T22:41:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.277+0000 7f1085ae9640 1 -- 172.21.15.40:0/3031576481 wait complete. 2024-03-18T22:41:04.287 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:04 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1830760676' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:41:04.289 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T22:41:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:04 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1830760676' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:41:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:04 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1830760676' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:41:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.393+0000 7f2b1ec6e640 1 -- 172.21.15.40:0/734785463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b18100740 msgr2=0x7f2b18100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.393+0000 7f2b1ec6e640 1 --2- 172.21.15.40:0/734785463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b18100740 0x7f2b18100b60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009900 tx=0x7f2b1402bc30 comp rx=0 tx=0).stop 2024-03-18T22:41:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.393+0000 7f2b1ec6e640 1 -- 172.21.15.40:0/734785463 shutdown_connections 2024-03-18T22:41:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.393+0000 7f2b1ec6e640 1 --2- 172.21.15.40:0/734785463 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b181022f0 0x7f2b18108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.393+0000 7f2b1ec6e640 1 --2- 172.21.15.40:0/734785463 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b18101930 0x7f2b18101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.393+0000 7f2b1ec6e640 1 --2- 172.21.15.40:0/734785463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b18100740 0x7f2b18100b60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.393+0000 7f2b1ec6e640 1 -- 172.21.15.40:0/734785463 >> 172.21.15.40:0/734785463 conn(0x7f2b180fbcd0 msgr2=0x7f2b180fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.393+0000 7f2b1ec6e640 1 -- 172.21.15.40:0/734785463 shutdown_connections 2024-03-18T22:41:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.393+0000 7f2b1ec6e640 1 -- 172.21.15.40:0/734785463 wait complete. 2024-03-18T22:41:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1ec6e640 1 Processor -- start 2024-03-18T22:41:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1ec6e640 1 -- start start 2024-03-18T22:41:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1ec6e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b18100740 0x7f2b1806a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1ec6e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b18101930 0x7f2b1819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1ec6e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b181022f0 0x7f2b1819ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1ec6e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1810c260 con 0x7f2b181022f0 2024-03-18T22:41:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1ec6e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1810c0e0 con 0x7f2b18100740 2024-03-18T22:41:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1ec6e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1810c3e0 con 0x7f2b18101930 2024-03-18T22:41:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1d20b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b181022f0 0x7f2b1819ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1d20b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b181022f0 0x7f2b1819ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50208/0 (socket says 172.21.15.40:50208) 2024-03-18T22:41:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1d20b640 1 -- 172.21.15.40:0/2679041041 learned_addr learned my addr 172.21.15.40:0/2679041041 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1ca0a640 1 --2- 172.21.15.40:0/2679041041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b18100740 0x7f2b1806a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b0ffff640 1 --2- 172.21.15.40:0/2679041041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b18101930 0x7f2b1819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1d20b640 1 -- 172.21.15.40:0/2679041041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b18101930 msgr2=0x7f2b1819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1d20b640 1 --2- 172.21.15.40:0/2679041041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b18101930 0x7f2b1819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1d20b640 1 -- 172.21.15.40:0/2679041041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b18100740 msgr2=0x7f2b1806a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1d20b640 1 --2- 172.21.15.40:0/2679041041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b18100740 0x7f2b1806a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:04.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1d20b640 1 -- 172.21.15.40:0/2679041041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1819f6d0 con 0x7f2b181022f0 2024-03-18T22:41:04.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b0ffff640 1 --2- 172.21.15.40:0/2679041041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b18101930 0x7f2b1819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:41:04.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.394+0000 7f2b1d20b640 1 --2- 172.21.15.40:0/2679041041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b181022f0 0x7f2b1819ee10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000efd0 tx=0x7f2b1000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:04.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.395+0000 7f2b0dffb640 1 -- 172.21.15.40:0/2679041041 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b10003b70 con 0x7f2b181022f0 2024-03-18T22:41:04.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.395+0000 7f2b1ec6e640 1 -- 172.21.15.40:0/2679041041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b1819f9c0 con 0x7f2b181022f0 2024-03-18T22:41:04.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.395+0000 7f2b0dffb640 1 -- 172.21.15.40:0/2679041041 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b10003d10 con 0x7f2b181022f0 2024-03-18T22:41:04.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.395+0000 7f2b0dffb640 1 -- 172.21.15.40:0/2679041041 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b10010980 con 0x7f2b181022f0 2024-03-18T22:41:04.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.395+0000 7f2b1ec6e640 1 -- 172.21.15.40:0/2679041041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b1819ff10 con 0x7f2b181022f0 2024-03-18T22:41:04.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.396+0000 7f2b1ec6e640 1 -- 172.21.15.40:0/2679041041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b18107b80 con 0x7f2b181022f0 2024-03-18T22:41:04.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.397+0000 7f2b0dffb640 1 -- 172.21.15.40:0/2679041041 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2b10004020 con 0x7f2b181022f0 2024-03-18T22:41:04.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.397+0000 7f2b0dffb640 1 --2- 172.21.15.40:0/2679041041 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2b000a2800 0x7f2b000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:04.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.398+0000 7f2b1ca0a640 1 --2- 172.21.15.40:0/2679041041 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2b000a2800 0x7f2b000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:04.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.398+0000 7f2b0dffb640 1 -- 172.21.15.40:0/2679041041 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2b100cf210 con 0x7f2b181022f0 2024-03-18T22:41:04.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.398+0000 7f2b1ca0a640 1 --2- 172.21.15.40:0/2679041041 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2b000a2800 0x7f2b000a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2b14005ab0 tx=0x7f2b14005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:04.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.402+0000 7f2b0dffb640 1 -- 172.21.15.40:0/2679041041 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2b1009c010 con 0x7f2b181022f0 2024-03-18T22:41:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:04.622+0000 7f2b1ec6e640 1 -- 172.21.15.40:0/2679041041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f2b18062420 con 0x7f2b181022f0 2024-03-18T22:41:05.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.021+0000 7f2b0dffb640 1 -- 172.21.15.40:0/2679041041 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f2b1009b910 con 0x7f2b181022f0 2024-03-18T22:41:05.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.023+0000 7f2b1ec6e640 1 -- 172.21.15.40:0/2679041041 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2b000a2800 msgr2=0x7f2b000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:05.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.023+0000 7f2b1ec6e640 1 --2- 172.21.15.40:0/2679041041 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2b000a2800 0x7f2b000a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2b14005ab0 tx=0x7f2b14005fd0 comp rx=0 tx=0).stop 2024-03-18T22:41:05.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.023+0000 7f2b1ec6e640 1 -- 172.21.15.40:0/2679041041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b181022f0 msgr2=0x7f2b1819ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:05.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.023+0000 7f2b1ec6e640 1 --2- 172.21.15.40:0/2679041041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b181022f0 0x7f2b1819ee10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000efd0 tx=0x7f2b1000c6a0 comp rx=0 tx=0).stop 2024-03-18T22:41:05.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.023+0000 7f2b1ec6e640 1 -- 172.21.15.40:0/2679041041 shutdown_connections 2024-03-18T22:41:05.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.024+0000 7f2b1ec6e640 1 --2- 172.21.15.40:0/2679041041 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2b000a2800 0x7f2b000a4cc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:05.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.024+0000 7f2b1ec6e640 1 --2- 172.21.15.40:0/2679041041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b181022f0 0x7f2b1819ee10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:05.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.024+0000 7f2b1ec6e640 1 --2- 172.21.15.40:0/2679041041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b18101930 0x7f2b1819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:05.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.024+0000 7f2b1ec6e640 1 --2- 172.21.15.40:0/2679041041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b18100740 0x7f2b1806a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:05.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.024+0000 7f2b1ec6e640 1 -- 172.21.15.40:0/2679041041 >> 172.21.15.40:0/2679041041 conn(0x7f2b180fbcd0 msgr2=0x7f2b18104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:05.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.024+0000 7f2b1ec6e640 1 -- 172.21.15.40:0/2679041041 shutdown_connections 2024-03-18T22:41:05.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.024+0000 7f2b1ec6e640 1 -- 172.21.15.40:0/2679041041 wait complete. 2024-03-18T22:41:05.036 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-18T22:41:05.036 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-18T22:41:05.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.143+0000 7f31d1cac640 1 -- 172.21.15.40:0/3998857503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31cc1033e0 msgr2=0x7f31cc109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:05.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.143+0000 7f31d1cac640 1 --2- 172.21.15.40:0/3998857503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31cc1033e0 0x7f31cc109c70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f31c4009ca0 tx=0x7f31c402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:41:05.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.144+0000 7f31d1cac640 1 -- 172.21.15.40:0/3998857503 shutdown_connections 2024-03-18T22:41:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.144+0000 7f31d1cac640 1 --2- 172.21.15.40:0/3998857503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31cc1033e0 0x7f31cc109c70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.144+0000 7f31d1cac640 1 --2- 172.21.15.40:0/3998857503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31cc102a20 0x7f31cc102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.144+0000 7f31d1cac640 1 --2- 172.21.15.40:0/3998857503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31cc101830 0x7f31cc101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.144+0000 7f31d1cac640 1 -- 172.21.15.40:0/3998857503 >> 172.21.15.40:0/3998857503 conn(0x7f31cc0fcdc0 msgr2=0x7f31cc0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.144+0000 7f31d1cac640 1 -- 172.21.15.40:0/3998857503 shutdown_connections 2024-03-18T22:41:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.144+0000 7f31d1cac640 1 -- 172.21.15.40:0/3998857503 wait complete. 2024-03-18T22:41:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.144+0000 7f31d1cac640 1 Processor -- start 2024-03-18T22:41:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.144+0000 7f31d1cac640 1 -- start start 2024-03-18T22:41:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.144+0000 7f31d1cac640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31cc101830 0x7f31cc0755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.144+0000 7f31d1cac640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31cc102a20 0x7f31cc073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.144+0000 7f31d1cac640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31cc1033e0 0x7f31cc074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.144+0000 7f31d1cac640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f31cc10d360 con 0x7f31cc101830 2024-03-18T22:41:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.144+0000 7f31d1cac640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f31cc10d1e0 con 0x7f31cc102a20 2024-03-18T22:41:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.144+0000 7f31d1cac640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f31cc10d4e0 con 0x7f31cc1033e0 2024-03-18T22:41:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31cb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31cc101830 0x7f31cc0755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31cb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31cc101830 0x7f31cc0755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50220/0 (socket says 172.21.15.40:50220) 2024-03-18T22:41:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31cb7fe640 1 -- 172.21.15.40:0/1474734210 learned_addr learned my addr 172.21.15.40:0/1474734210 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31caffd640 1 --2- 172.21.15.40:0/1474734210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31cc102a20 0x7f31cc073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31cbfff640 1 --2- 172.21.15.40:0/1474734210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31cc1033e0 0x7f31cc074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31cb7fe640 1 -- 172.21.15.40:0/1474734210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31cc1033e0 msgr2=0x7f31cc074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31cb7fe640 1 --2- 172.21.15.40:0/1474734210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31cc1033e0 0x7f31cc074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31cb7fe640 1 -- 172.21.15.40:0/1474734210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31cc102a20 msgr2=0x7f31cc073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31cb7fe640 1 --2- 172.21.15.40:0/1474734210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31cc102a20 0x7f31cc073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31cb7fe640 1 -- 172.21.15.40:0/1474734210 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31cc0749e0 con 0x7f31cc101830 2024-03-18T22:41:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31cb7fe640 1 --2- 172.21.15.40:0/1474734210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31cc101830 0x7f31cc0755d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00e9c0 tx=0x7f31bc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31c8ff9640 1 -- 172.21.15.40:0/1474734210 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc00cd80 con 0x7f31cc101830 2024-03-18T22:41:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31c8ff9640 1 -- 172.21.15.40:0/1474734210 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f31bc0035f0 con 0x7f31cc101830 2024-03-18T22:41:05.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31d1cac640 1 -- 172.21.15.40:0/1474734210 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31cc1a1b30 con 0x7f31cc101830 2024-03-18T22:41:05.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31c8ff9640 1 -- 172.21.15.40:0/1474734210 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc010690 con 0x7f31cc101830 2024-03-18T22:41:05.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.145+0000 7f31d1cac640 1 -- 172.21.15.40:0/1474734210 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31cc1a2080 con 0x7f31cc101830 2024-03-18T22:41:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.146+0000 7f31d1cac640 1 -- 172.21.15.40:0/1474734210 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31cc108b50 con 0x7f31cc101830 2024-03-18T22:41:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.147+0000 7f31c8ff9640 1 -- 172.21.15.40:0/1474734210 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f31bc0037a0 con 0x7f31cc101830 2024-03-18T22:41:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.148+0000 7f31c8ff9640 1 --2- 172.21.15.40:0/1474734210 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f31b40a2800 0x7f31b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.148+0000 7f31c8ff9640 1 -- 172.21.15.40:0/1474734210 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f31bc0cf140 con 0x7f31cc101830 2024-03-18T22:41:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.148+0000 7f31caffd640 1 --2- 172.21.15.40:0/1474734210 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f31b40a2800 0x7f31b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:05.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.148+0000 7f31caffd640 1 --2- 172.21.15.40:0/1474734210 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f31b40a2800 0x7f31b40a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f31cc075340 tx=0x7f31c000b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.152+0000 7f31c8ff9640 1 -- 172.21.15.40:0/1474734210 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f31bc09bfa0 con 0x7f31cc101830 2024-03-18T22:41:05.304 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:05 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3031576481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:41:05.304 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:05 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2679041041' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T22:41:05.304 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:05 smithi113 ceph-mon[66775]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T22:41:05.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:05 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3031576481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:41:05.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:05 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2679041041' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T22:41:05.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:05 smithi103 ceph-mon[66377]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T22:41:05.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:05.378+0000 7f31d1cac640 1 -- 172.21.15.40:0/1474734210 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f31cc02d450 con 0x7f31cc101830 2024-03-18T22:41:05.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:05 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3031576481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:41:05.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:05 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2679041041' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T22:41:05.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:05 smithi040 ceph-mon[62643]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T22:41:06.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.022+0000 7f31c8ff9640 1 -- 172.21.15.40:0/1474734210 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f31bc09b8a0 con 0x7f31cc101830 2024-03-18T22:41:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.025+0000 7f31d1cac640 1 -- 172.21.15.40:0/1474734210 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f31b40a2800 msgr2=0x7f31b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.025+0000 7f31d1cac640 1 --2- 172.21.15.40:0/1474734210 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f31b40a2800 0x7f31b40a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f31cc075340 tx=0x7f31c000b390 comp rx=0 tx=0).stop 2024-03-18T22:41:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.025+0000 7f31d1cac640 1 -- 172.21.15.40:0/1474734210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31cc101830 msgr2=0x7f31cc0755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.025+0000 7f31d1cac640 1 --2- 172.21.15.40:0/1474734210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31cc101830 0x7f31cc0755d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00e9c0 tx=0x7f31bc00ee80 comp rx=0 tx=0).stop 2024-03-18T22:41:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.025+0000 7f31d1cac640 1 -- 172.21.15.40:0/1474734210 shutdown_connections 2024-03-18T22:41:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.025+0000 7f31d1cac640 1 --2- 172.21.15.40:0/1474734210 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f31b40a2800 0x7f31b40a4cc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.025+0000 7f31d1cac640 1 --2- 172.21.15.40:0/1474734210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31cc1033e0 0x7f31cc074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.025+0000 7f31d1cac640 1 --2- 172.21.15.40:0/1474734210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31cc102a20 0x7f31cc073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.025+0000 7f31d1cac640 1 --2- 172.21.15.40:0/1474734210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31cc101830 0x7f31cc0755d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.025+0000 7f31d1cac640 1 -- 172.21.15.40:0/1474734210 >> 172.21.15.40:0/1474734210 conn(0x7f31cc0fcdc0 msgr2=0x7f31cc105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.025+0000 7f31d1cac640 1 -- 172.21.15.40:0/1474734210 shutdown_connections 2024-03-18T22:41:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.025+0000 7f31d1cac640 1 -- 172.21.15.40:0/1474734210 wait complete. 2024-03-18T22:41:06.042 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-18T22:41:06.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.150+0000 7f5b851c4640 1 -- 172.21.15.40:0/1759738492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b80101910 msgr2=0x7f5b80101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:06.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.150+0000 7f5b851c4640 1 --2- 172.21.15.40:0/1759738492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b80101910 0x7f5b80101d90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009900 tx=0x7f5b7402bc30 comp rx=0 tx=0).stop 2024-03-18T22:41:06.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.150+0000 7f5b851c4640 1 -- 172.21.15.40:0/1759738492 shutdown_connections 2024-03-18T22:41:06.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.150+0000 7f5b851c4640 1 --2- 172.21.15.40:0/1759738492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b801022d0 0x7f5b80108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:06.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.150+0000 7f5b851c4640 1 --2- 172.21.15.40:0/1759738492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b80101910 0x7f5b80101d90 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:06.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.150+0000 7f5b851c4640 1 --2- 172.21.15.40:0/1759738492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b80100720 0x7f5b80100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:06.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.150+0000 7f5b851c4640 1 -- 172.21.15.40:0/1759738492 >> 172.21.15.40:0/1759738492 conn(0x7f5b800fbcb0 msgr2=0x7f5b800fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:06.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.150+0000 7f5b851c4640 1 -- 172.21.15.40:0/1759738492 shutdown_connections 2024-03-18T22:41:06.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.150+0000 7f5b851c4640 1 -- 172.21.15.40:0/1759738492 wait complete. 2024-03-18T22:41:06.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.151+0000 7f5b851c4640 1 Processor -- start 2024-03-18T22:41:06.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.151+0000 7f5b851c4640 1 -- start start 2024-03-18T22:41:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.151+0000 7f5b851c4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b80100720 0x7f5b80197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.151+0000 7f5b851c4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b80101910 0x7f5b80198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.151+0000 7f5b851c4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b801022d0 0x7f5b8019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.151+0000 7f5b851c4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8010c250 con 0x7f5b801022d0 2024-03-18T22:41:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.151+0000 7f5b851c4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8010c0d0 con 0x7f5b80101910 2024-03-18T22:41:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.151+0000 7f5b851c4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8010c3d0 con 0x7f5b80100720 2024-03-18T22:41:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.151+0000 7f5b7f59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b801022d0 0x7f5b8019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.151+0000 7f5b7f59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b801022d0 0x7f5b8019f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50242/0 (socket says 172.21.15.40:50242) 2024-03-18T22:41:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.151+0000 7f5b7f59e640 1 -- 172.21.15.40:0/4208737107 learned_addr learned my addr 172.21.15.40:0/4208737107 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.151+0000 7f5b7e59c640 1 --2- 172.21.15.40:0/4208737107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b80101910 0x7f5b80198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.151+0000 7f5b7ed9d640 1 --2- 172.21.15.40:0/4208737107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b80100720 0x7f5b80197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.152+0000 7f5b7f59e640 1 -- 172.21.15.40:0/4208737107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b80100720 msgr2=0x7f5b80197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.152+0000 7f5b7f59e640 1 --2- 172.21.15.40:0/4208737107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b80100720 0x7f5b80197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.152+0000 7f5b7f59e640 1 -- 172.21.15.40:0/4208737107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b80101910 msgr2=0x7f5b80198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.152+0000 7f5b7f59e640 1 --2- 172.21.15.40:0/4208737107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b80101910 0x7f5b80198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.152+0000 7f5b7f59e640 1 -- 172.21.15.40:0/4208737107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8019f6b0 con 0x7f5b801022d0 2024-03-18T22:41:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.152+0000 7f5b7f59e640 1 --2- 172.21.15.40:0/4208737107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b801022d0 0x7f5b8019f040 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800eb10 tx=0x7f5b7800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.152+0000 7f5b67fff640 1 -- 172.21.15.40:0/4208737107 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b78004b20 con 0x7f5b801022d0 2024-03-18T22:41:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.152+0000 7f5b851c4640 1 -- 172.21.15.40:0/4208737107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b8019f9a0 con 0x7f5b801022d0 2024-03-18T22:41:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.152+0000 7f5b67fff640 1 -- 172.21.15.40:0/4208737107 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b78004cc0 con 0x7f5b801022d0 2024-03-18T22:41:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.152+0000 7f5b67fff640 1 -- 172.21.15.40:0/4208737107 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b78010690 con 0x7f5b801022d0 2024-03-18T22:41:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.152+0000 7f5b851c4640 1 -- 172.21.15.40:0/4208737107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b8019ff10 con 0x7f5b801022d0 2024-03-18T22:41:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.153+0000 7f5b851c4640 1 -- 172.21.15.40:0/4208737107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b801a0200 con 0x7f5b801022d0 2024-03-18T22:41:06.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.154+0000 7f5b67fff640 1 -- 172.21.15.40:0/4208737107 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5b78005000 con 0x7f5b801022d0 2024-03-18T22:41:06.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.155+0000 7f5b67fff640 1 --2- 172.21.15.40:0/4208737107 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b680a2800 0x7f5b680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:06.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.155+0000 7f5b7ed9d640 1 --2- 172.21.15.40:0/4208737107 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b680a2800 0x7f5b680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:06.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.155+0000 7f5b67fff640 1 -- 172.21.15.40:0/4208737107 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5b780cf740 con 0x7f5b801022d0 2024-03-18T22:41:06.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.155+0000 7f5b7ed9d640 1 --2- 172.21.15.40:0/4208737107 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b680a2800 0x7f5b680a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5b700095f0 tx=0x7f5b70005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:06.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.162+0000 7f5b67fff640 1 -- 172.21.15.40:0/4208737107 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5b7809c540 con 0x7f5b801022d0 2024-03-18T22:41:06.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:06 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2679041041' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T22:41:06.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:06 smithi113 ceph-mon[66775]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T22:41:06.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:06 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1474734210' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T22:41:06.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:06 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2679041041' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T22:41:06.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:06 smithi103 ceph-mon[66377]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T22:41:06.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:06 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1474734210' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T22:41:06.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:06.386+0000 7f5b851c4640 1 -- 172.21.15.40:0/4208737107 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f5b801a03d0 con 0x7f5b801022d0 2024-03-18T22:41:06.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:06 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2679041041' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T22:41:06.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:06 smithi040 ceph-mon[62643]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T22:41:06.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:06 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1474734210' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T22:41:07.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.033+0000 7f5b67fff640 1 -- 172.21.15.40:0/4208737107 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f5b7809be40 con 0x7f5b801022d0 2024-03-18T22:41:07.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.036+0000 7f5b851c4640 1 -- 172.21.15.40:0/4208737107 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b680a2800 msgr2=0x7f5b680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:07.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.036+0000 7f5b851c4640 1 --2- 172.21.15.40:0/4208737107 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b680a2800 0x7f5b680a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5b700095f0 tx=0x7f5b70005f90 comp rx=0 tx=0).stop 2024-03-18T22:41:07.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.036+0000 7f5b851c4640 1 -- 172.21.15.40:0/4208737107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b801022d0 msgr2=0x7f5b8019f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:07.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.036+0000 7f5b851c4640 1 --2- 172.21.15.40:0/4208737107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b801022d0 0x7f5b8019f040 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800eb10 tx=0x7f5b7800c2d0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.036+0000 7f5b851c4640 1 -- 172.21.15.40:0/4208737107 shutdown_connections 2024-03-18T22:41:07.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.036+0000 7f5b851c4640 1 --2- 172.21.15.40:0/4208737107 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b680a2800 0x7f5b680a4cc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.036+0000 7f5b851c4640 1 --2- 172.21.15.40:0/4208737107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b801022d0 0x7f5b8019f040 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.036+0000 7f5b851c4640 1 --2- 172.21.15.40:0/4208737107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b80101910 0x7f5b80198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.036+0000 7f5b851c4640 1 --2- 172.21.15.40:0/4208737107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b80100720 0x7f5b80197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.036+0000 7f5b851c4640 1 -- 172.21.15.40:0/4208737107 >> 172.21.15.40:0/4208737107 conn(0x7f5b800fbcb0 msgr2=0x7f5b80068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:07.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.036+0000 7f5b851c4640 1 -- 172.21.15.40:0/4208737107 shutdown_connections 2024-03-18T22:41:07.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.036+0000 7f5b851c4640 1 -- 172.21.15.40:0/4208737107 wait complete. 2024-03-18T22:41:07.052 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-18T22:41:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.160+0000 7f1f0051d640 1 -- 172.21.15.40:0/2775634884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef8100740 msgr2=0x7f1ef8100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.160+0000 7f1f0051d640 1 --2- 172.21.15.40:0/2775634884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef8100740 0x7f1ef8100b60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0009ca0 tx=0x7f1ef002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.160+0000 7f1f0051d640 1 -- 172.21.15.40:0/2775634884 shutdown_connections 2024-03-18T22:41:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.160+0000 7f1f0051d640 1 --2- 172.21.15.40:0/2775634884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef81022f0 0x7f1ef8108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.160+0000 7f1f0051d640 1 --2- 172.21.15.40:0/2775634884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ef8101930 0x7f1ef8101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.160+0000 7f1f0051d640 1 --2- 172.21.15.40:0/2775634884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef8100740 0x7f1ef8100b60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.160+0000 7f1f0051d640 1 -- 172.21.15.40:0/2775634884 >> 172.21.15.40:0/2775634884 conn(0x7f1ef80fbcd0 msgr2=0x7f1ef80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.160+0000 7f1f0051d640 1 -- 172.21.15.40:0/2775634884 shutdown_connections 2024-03-18T22:41:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.160+0000 7f1f0051d640 1 -- 172.21.15.40:0/2775634884 wait complete. 2024-03-18T22:41:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.161+0000 7f1f0051d640 1 Processor -- start 2024-03-18T22:41:07.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.161+0000 7f1f0051d640 1 -- start start 2024-03-18T22:41:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.161+0000 7f1f0051d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ef8100740 0x7f1ef810af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.161+0000 7f1f0051d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef8101930 0x7f1ef8109580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.161+0000 7f1f0051d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef81022f0 0x7f1ef8109ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.161+0000 7f1f0051d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef810d390 con 0x7f1ef81022f0 2024-03-18T22:41:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.161+0000 7f1f0051d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef810d210 con 0x7f1ef8101930 2024-03-18T22:41:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.161+0000 7f1f0051d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef810d510 con 0x7f1ef8100740 2024-03-18T22:41:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.161+0000 7f1efeaba640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef81022f0 0x7f1ef8109ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.161+0000 7f1efeaba640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef81022f0 0x7f1ef8109ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50266/0 (socket says 172.21.15.40:50266) 2024-03-18T22:41:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.161+0000 7f1efe2b9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ef8100740 0x7f1ef810af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.161+0000 7f1efeaba640 1 -- 172.21.15.40:0/568096307 learned_addr learned my addr 172.21.15.40:0/568096307 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.161+0000 7f1efdab8640 1 --2- 172.21.15.40:0/568096307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef8101930 0x7f1ef8109580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.162+0000 7f1efe2b9640 1 -- 172.21.15.40:0/568096307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef8101930 msgr2=0x7f1ef8109580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:07.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.162+0000 7f1efe2b9640 1 --2- 172.21.15.40:0/568096307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef8101930 0x7f1ef8109580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.162+0000 7f1efe2b9640 1 -- 172.21.15.40:0/568096307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef81022f0 msgr2=0x7f1ef8109ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.162+0000 7f1efe2b9640 1 --2- 172.21.15.40:0/568096307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef81022f0 0x7f1ef8109ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.162+0000 7f1efe2b9640 1 -- 172.21.15.40:0/568096307 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ef810a350 con 0x7f1ef8100740 2024-03-18T22:41:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.162+0000 7f1efeaba640 1 --2- 172.21.15.40:0/568096307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef81022f0 0x7f1ef8109ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.162+0000 7f1efe2b9640 1 --2- 172.21.15.40:0/568096307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ef8100740 0x7f1ef810af40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1ef002c630 tx=0x7f1ef00328a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.162+0000 7f1ef77fe640 1 -- 172.21.15.40:0/568096307 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0041020 con 0x7f1ef8100740 2024-03-18T22:41:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.162+0000 7f1ef77fe640 1 -- 172.21.15.40:0/568096307 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ef0003e20 con 0x7f1ef8100740 2024-03-18T22:41:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.162+0000 7f1ef77fe640 1 -- 172.21.15.40:0/568096307 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0039040 con 0x7f1ef8100740 2024-03-18T22:41:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.162+0000 7f1f0051d640 1 -- 172.21.15.40:0/568096307 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ef81a1b60 con 0x7f1ef8100740 2024-03-18T22:41:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.162+0000 7f1f0051d640 1 -- 172.21.15.40:0/568096307 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ef81a2080 con 0x7f1ef8100740 2024-03-18T22:41:07.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.163+0000 7f1f0051d640 1 -- 172.21.15.40:0/568096307 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ef8107b30 con 0x7f1ef8100740 2024-03-18T22:41:07.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.164+0000 7f1ef77fe640 1 -- 172.21.15.40:0/568096307 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1ef0044020 con 0x7f1ef8100740 2024-03-18T22:41:07.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.165+0000 7f1ef77fe640 1 --2- 172.21.15.40:0/568096307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1ee00a2800 0x7f1ee00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:07.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.165+0000 7f1efdab8640 1 --2- 172.21.15.40:0/568096307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1ee00a2800 0x7f1ee00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:07.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.165+0000 7f1ef77fe640 1 -- 172.21.15.40:0/568096307 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1ef00f10f0 con 0x7f1ef8100740 2024-03-18T22:41:07.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.165+0000 7f1efdab8640 1 --2- 172.21.15.40:0/568096307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1ee00a2800 0x7f1ee00a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1ef810acb0 tx=0x7f1eec005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:07.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.172+0000 7f1ef77fe640 1 -- 172.21.15.40:0/568096307 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1ef004b020 con 0x7f1ef8100740 2024-03-18T22:41:07.321 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:07 smithi103 ceph-mon[66377]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T22:41:07.321 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:07 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1474734210' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-18T22:41:07.321 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:07 smithi103 ceph-mon[66377]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T22:41:07.321 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:07 smithi103 ceph-mon[66377]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T22:41:07.321 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:07 smithi103 ceph-mon[66377]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T22:41:07.321 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:07 smithi103 ceph-mon[66377]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T22:41:07.321 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:07 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/4208737107' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T22:41:07.321 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:07 smithi103 ceph-mon[66377]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 1 op/s 2024-03-18T22:41:07.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:07 smithi113 ceph-mon[66775]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T22:41:07.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:07 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1474734210' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-18T22:41:07.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:07 smithi113 ceph-mon[66775]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T22:41:07.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:07 smithi113 ceph-mon[66775]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T22:41:07.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:07 smithi113 ceph-mon[66775]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T22:41:07.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:07 smithi113 ceph-mon[66775]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T22:41:07.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:07 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/4208737107' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T22:41:07.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:07 smithi113 ceph-mon[66775]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 1 op/s 2024-03-18T22:41:07.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:07 smithi040 ceph-mon[62643]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T22:41:07.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:07 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1474734210' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-18T22:41:07.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:07 smithi040 ceph-mon[62643]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T22:41:07.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:07 smithi040 ceph-mon[62643]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T22:41:07.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:07 smithi040 ceph-mon[62643]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T22:41:07.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:07 smithi040 ceph-mon[62643]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T22:41:07.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:07 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/4208737107' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T22:41:07.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:07 smithi040 ceph-mon[62643]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 1 op/s 2024-03-18T22:41:07.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.395+0000 7f1f0051d640 1 -- 172.21.15.40:0/568096307 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f1ef802d1f0 con 0x7f1ef8100740 2024-03-18T22:41:07.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.819+0000 7f1ef77fe640 1 -- 172.21.15.40:0/568096307 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7f1ef00bdef0 con 0x7f1ef8100740 2024-03-18T22:41:07.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.822+0000 7f1f0051d640 1 -- 172.21.15.40:0/568096307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1ee00a2800 msgr2=0x7f1ee00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:07.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.822+0000 7f1f0051d640 1 --2- 172.21.15.40:0/568096307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1ee00a2800 0x7f1ee00a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1ef810acb0 tx=0x7f1eec005f90 comp rx=0 tx=0).stop 2024-03-18T22:41:07.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.822+0000 7f1f0051d640 1 -- 172.21.15.40:0/568096307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ef8100740 msgr2=0x7f1ef810af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:07.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.822+0000 7f1f0051d640 1 --2- 172.21.15.40:0/568096307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ef8100740 0x7f1ef810af40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1ef002c630 tx=0x7f1ef00328a0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.822+0000 7f1f0051d640 1 -- 172.21.15.40:0/568096307 shutdown_connections 2024-03-18T22:41:07.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.822+0000 7f1f0051d640 1 --2- 172.21.15.40:0/568096307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1ee00a2800 0x7f1ee00a4cc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.822+0000 7f1f0051d640 1 --2- 172.21.15.40:0/568096307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ef81022f0 0x7f1ef8109ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.822+0000 7f1f0051d640 1 --2- 172.21.15.40:0/568096307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1ef8101930 0x7f1ef8109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.822+0000 7f1f0051d640 1 --2- 172.21.15.40:0/568096307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1ef8100740 0x7f1ef810af40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.822+0000 7f1f0051d640 1 -- 172.21.15.40:0/568096307 >> 172.21.15.40:0/568096307 conn(0x7f1ef80fbcd0 msgr2=0x7f1ef8104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:07.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.822+0000 7f1f0051d640 1 -- 172.21.15.40:0/568096307 shutdown_connections 2024-03-18T22:41:07.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.822+0000 7f1f0051d640 1 -- 172.21.15.40:0/568096307 wait complete. 2024-03-18T22:41:07.837 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-18T22:41:07.837 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-18T22:41:07.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.942+0000 7f6f1ecf1640 1 -- 172.21.15.40:0/3676831175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f18102300 msgr2=0x7f6f181046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:07.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.942+0000 7f6f1ecf1640 1 --2- 172.21.15.40:0/3676831175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f18102300 0x7f6f181046f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6f10009ca0 tx=0x7f6f1002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.942+0000 7f6f1ecf1640 1 -- 172.21.15.40:0/3676831175 shutdown_connections 2024-03-18T22:41:07.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.942+0000 7f6f1ecf1640 1 --2- 172.21.15.40:0/3676831175 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f18107560 0x7f6f18109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.942+0000 7f6f1ecf1640 1 --2- 172.21.15.40:0/3676831175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f18104c30 0x7f6f18107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.942+0000 7f6f1ecf1640 1 --2- 172.21.15.40:0/3676831175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f18102300 0x7f6f181046f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.942+0000 7f6f1ecf1640 1 -- 172.21.15.40:0/3676831175 >> 172.21.15.40:0/3676831175 conn(0x7f6f180fbcb0 msgr2=0x7f6f180fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:07.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.942+0000 7f6f1ecf1640 1 -- 172.21.15.40:0/3676831175 shutdown_connections 2024-03-18T22:41:07.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.942+0000 7f6f1ecf1640 1 -- 172.21.15.40:0/3676831175 wait complete. 2024-03-18T22:41:07.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f1ecf1640 1 Processor -- start 2024-03-18T22:41:07.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f1ecf1640 1 -- start start 2024-03-18T22:41:07.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f1ecf1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f18102300 0x7f6f18197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:07.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f1ecf1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f18104c30 0x7f6f18198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:07.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f1ecf1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f18107560 0x7f6f1819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:07.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f1ecf1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1810d370 con 0x7f6f18104c30 2024-03-18T22:41:07.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f1ecf1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1810d1f0 con 0x7f6f18102300 2024-03-18T22:41:07.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f1ecf1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1810d4f0 con 0x7f6f18107560 2024-03-18T22:41:07.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f17fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f18104c30 0x7f6f18198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:07.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f17fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f18104c30 0x7f6f18198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50278/0 (socket says 172.21.15.40:50278) 2024-03-18T22:41:07.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f17fff640 1 -- 172.21.15.40:0/807027872 learned_addr learned my addr 172.21.15.40:0/807027872 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:07.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f1d28e640 1 --2- 172.21.15.40:0/807027872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f18107560 0x7f6f1819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:07.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f1ca8d640 1 --2- 172.21.15.40:0/807027872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f18102300 0x7f6f18197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:07.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f17fff640 1 -- 172.21.15.40:0/807027872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f18107560 msgr2=0x7f6f1819f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:07.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f17fff640 1 --2- 172.21.15.40:0/807027872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f18107560 0x7f6f1819f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f17fff640 1 -- 172.21.15.40:0/807027872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f18102300 msgr2=0x7f6f18197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:07.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f17fff640 1 --2- 172.21.15.40:0/807027872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f18102300 0x7f6f18197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:07.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.943+0000 7f6f17fff640 1 -- 172.21.15.40:0/807027872 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1819f700 con 0x7f6f18104c30 2024-03-18T22:41:07.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.944+0000 7f6f17fff640 1 --2- 172.21.15.40:0/807027872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f18104c30 0x7f6f18198020 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e9c0 tx=0x7f6f0c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:07.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.944+0000 7f6f15ffb640 1 -- 172.21.15.40:0/807027872 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c00cd80 con 0x7f6f18104c30 2024-03-18T22:41:07.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.944+0000 7f6f15ffb640 1 -- 172.21.15.40:0/807027872 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f0c0054d0 con 0x7f6f18104c30 2024-03-18T22:41:07.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.944+0000 7f6f1ecf1640 1 -- 172.21.15.40:0/807027872 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f1819f9f0 con 0x7f6f18104c30 2024-03-18T22:41:07.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.944+0000 7f6f15ffb640 1 -- 172.21.15.40:0/807027872 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c010430 con 0x7f6f18104c30 2024-03-18T22:41:07.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.944+0000 7f6f1ecf1640 1 -- 172.21.15.40:0/807027872 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f1819ff40 con 0x7f6f18104c30 2024-03-18T22:41:07.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.945+0000 7f6f1ecf1640 1 -- 172.21.15.40:0/807027872 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f180fd7a0 con 0x7f6f18104c30 2024-03-18T22:41:07.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.946+0000 7f6f15ffb640 1 -- 172.21.15.40:0/807027872 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f6f0c004fd0 con 0x7f6f18104c30 2024-03-18T22:41:07.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.946+0000 7f6f15ffb640 1 --2- 172.21.15.40:0/807027872 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6f000a2800 0x7f6f000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:07.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.946+0000 7f6f15ffb640 1 -- 172.21.15.40:0/807027872 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6f0c0cf0d0 con 0x7f6f18104c30 2024-03-18T22:41:07.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.946+0000 7f6f1ca8d640 1 --2- 172.21.15.40:0/807027872 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6f000a2800 0x7f6f000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:07.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.947+0000 7f6f1ca8d640 1 --2- 172.21.15.40:0/807027872 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6f000a2800 0x7f6f000a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6f1002c700 tx=0x7f6f10000f40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:07.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:07.951+0000 7f6f15ffb640 1 -- 172.21.15.40:0/807027872 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6f0c09bed0 con 0x7f6f18104c30 2024-03-18T22:41:08.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/4208737107' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T22:41:08.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] up:active 2024-03-18T22:41:08.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-mon[62643]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T22:41:08.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-mon[62643]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T22:41:08.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-mon[62643]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-18T22:41:08.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-mon[62643]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-18T22:41:08.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/568096307' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T22:41:08.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-mon[62643]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T22:41:08.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-mon[62643]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T22:41:08.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] up:active 2024-03-18T22:41:08.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-mon[62643]: daemon mds.k assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-18T22:41:08.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-mon[62643]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T22:41:08.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-mon[62643]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=k=up:creating} 3 up:standby-replay 5 up:standby 2024-03-18T22:41:08.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-mon[62643]: daemon mds.k is now active in filesystem cephfs as rank 3 2024-03-18T22:41:08.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:08.163+0000 7f6f1ecf1640 1 -- 172.21.15.40:0/807027872 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f6f18061c60 con 0x7f6f000a2800 2024-03-18T22:41:08.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:08 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/4208737107' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T22:41:08.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:08 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] up:active 2024-03-18T22:41:08.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:08 smithi113 ceph-mon[66775]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T22:41:08.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:08 smithi113 ceph-mon[66775]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T22:41:08.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:08 smithi113 ceph-mon[66775]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-18T22:41:08.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:08 smithi113 ceph-mon[66775]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-18T22:41:08.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:08 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/568096307' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T22:41:08.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:08 smithi113 ceph-mon[66775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T22:41:08.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:08 smithi113 ceph-mon[66775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T22:41:08.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:08 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] up:active 2024-03-18T22:41:08.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:08 smithi113 ceph-mon[66775]: daemon mds.k assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-18T22:41:08.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:08 smithi113 ceph-mon[66775]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T22:41:08.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:08 smithi113 ceph-mon[66775]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=k=up:creating} 3 up:standby-replay 5 up:standby 2024-03-18T22:41:08.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:08 smithi113 ceph-mon[66775]: daemon mds.k is now active in filesystem cephfs as rank 3 2024-03-18T22:41:08.359 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 22:41:08 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-b[95936]: 2024-03-18T22:41:08.195+0000 7f747196f700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-18T22:41:08.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:08 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/4208737107' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T22:41:08.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:08 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] up:active 2024-03-18T22:41:08.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:08 smithi103 ceph-mon[66377]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T22:41:08.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:08 smithi103 ceph-mon[66377]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T22:41:08.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:08 smithi103 ceph-mon[66377]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-18T22:41:08.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:08 smithi103 ceph-mon[66377]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-18T22:41:08.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:08 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/568096307' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T22:41:08.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:08 smithi103 ceph-mon[66377]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T22:41:08.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:08 smithi103 ceph-mon[66377]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T22:41:08.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:08 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] up:active 2024-03-18T22:41:08.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:08 smithi103 ceph-mon[66377]: daemon mds.k assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-18T22:41:08.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:08 smithi103 ceph-mon[66377]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-18T22:41:08.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:08 smithi103 ceph-mon[66377]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=k=up:creating} 3 up:standby-replay 5 up:standby 2024-03-18T22:41:08.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:08 smithi103 ceph-mon[66377]: daemon mds.k is now active in filesystem cephfs as rank 3 2024-03-18T22:41:08.644 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:41:08.168+0000 7fa757b4f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T22:41:08.644 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:41:08.168+0000 7fa757b4f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T22:41:08.644 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:41:08.168+0000 7fa757b4f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T22:41:08.644 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:41:08.168+0000 7fa757b4f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T22:41:08.644 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:41:08 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:41:08.168+0000 7fa757b4f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T22:41:09.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.856+0000 7f6f15ffb640 1 -- 172.21.15.40:0/807027872 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6f18061c60 con 0x7f6f000a2800 2024-03-18T22:41:09.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.858+0000 7f6f1ecf1640 1 -- 172.21.15.40:0/807027872 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6f000a2800 msgr2=0x7f6f000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:09.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.858+0000 7f6f1ecf1640 1 --2- 172.21.15.40:0/807027872 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6f000a2800 0x7f6f000a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6f1002c700 tx=0x7f6f10000f40 comp rx=0 tx=0).stop 2024-03-18T22:41:09.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.858+0000 7f6f1ecf1640 1 -- 172.21.15.40:0/807027872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f18104c30 msgr2=0x7f6f18198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:09.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.858+0000 7f6f1ecf1640 1 --2- 172.21.15.40:0/807027872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f18104c30 0x7f6f18198020 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c00e9c0 tx=0x7f6f0c00ee80 comp rx=0 tx=0).stop 2024-03-18T22:41:09.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.858+0000 7f6f1ecf1640 1 -- 172.21.15.40:0/807027872 shutdown_connections 2024-03-18T22:41:09.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.858+0000 7f6f1ecf1640 1 --2- 172.21.15.40:0/807027872 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6f000a2800 0x7f6f000a4cc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:09.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.858+0000 7f6f1ecf1640 1 --2- 172.21.15.40:0/807027872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6f18107560 0x7f6f1819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:09.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.858+0000 7f6f1ecf1640 1 --2- 172.21.15.40:0/807027872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6f18104c30 0x7f6f18198020 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:09.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.858+0000 7f6f1ecf1640 1 --2- 172.21.15.40:0/807027872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f18102300 0x7f6f18197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:09.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.858+0000 7f6f1ecf1640 1 -- 172.21.15.40:0/807027872 >> 172.21.15.40:0/807027872 conn(0x7f6f180fbcb0 msgr2=0x7f6f18102e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:09.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.859+0000 7f6f1ecf1640 1 -- 172.21.15.40:0/807027872 shutdown_connections 2024-03-18T22:41:09.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.859+0000 7f6f1ecf1640 1 -- 172.21.15.40:0/807027872 wait complete. 2024-03-18T22:41:09.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-18T22:41:09.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.984+0000 7fecc7510640 1 -- 172.21.15.40:0/1121337621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0102a20 msgr2=0x7fecc0102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:09.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.984+0000 7fecc7510640 1 --2- 172.21.15.40:0/1121337621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0102a20 0x7fecc0102ea0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fecb0009900 tx=0x7fecb002bc30 comp rx=0 tx=0).stop 2024-03-18T22:41:09.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.984+0000 7fecc7510640 1 -- 172.21.15.40:0/1121337621 shutdown_connections 2024-03-18T22:41:09.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.984+0000 7fecc7510640 1 --2- 172.21.15.40:0/1121337621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecc01033e0 0x7fecc0109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:09.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.984+0000 7fecc7510640 1 --2- 172.21.15.40:0/1121337621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0102a20 0x7fecc0102ea0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:09.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.984+0000 7fecc7510640 1 --2- 172.21.15.40:0/1121337621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc0101830 0x7fecc0101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:09.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.984+0000 7fecc7510640 1 -- 172.21.15.40:0/1121337621 >> 172.21.15.40:0/1121337621 conn(0x7fecc00fcdc0 msgr2=0x7fecc00ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:09.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.984+0000 7fecc7510640 1 -- 172.21.15.40:0/1121337621 shutdown_connections 2024-03-18T22:41:09.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.984+0000 7fecc7510640 1 -- 172.21.15.40:0/1121337621 wait complete. 2024-03-18T22:41:09.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc7510640 1 Processor -- start 2024-03-18T22:41:09.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc7510640 1 -- start start 2024-03-18T22:41:09.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc7510640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecc0101830 0x7fecc0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc7510640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0102a20 0x7fecc0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc7510640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc01033e0 0x7fecc019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc7510640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fecc010d340 con 0x7fecc0102a20 2024-03-18T22:41:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc7510640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fecc010d1c0 con 0x7fecc01033e0 2024-03-18T22:41:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc7510640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fecc010d4c0 con 0x7fecc0101830 2024-03-18T22:41:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc4aab640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0102a20 0x7fecc0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc4aab640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0102a20 0x7fecc0198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40686/0 (socket says 172.21.15.40:40686) 2024-03-18T22:41:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc4aab640 1 -- 172.21.15.40:0/1823350703 learned_addr learned my addr 172.21.15.40:0/1823350703 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc52ac640 1 --2- 172.21.15.40:0/1823350703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecc0101830 0x7fecc0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc5aad640 1 --2- 172.21.15.40:0/1823350703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc01033e0 0x7fecc019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc4aab640 1 -- 172.21.15.40:0/1823350703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecc0101830 msgr2=0x7fecc0197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc4aab640 1 --2- 172.21.15.40:0/1823350703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecc0101830 0x7fecc0197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc4aab640 1 -- 172.21.15.40:0/1823350703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc01033e0 msgr2=0x7fecc019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc4aab640 1 --2- 172.21.15.40:0/1823350703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc01033e0 0x7fecc019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:09.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc4aab640 1 -- 172.21.15.40:0/1823350703 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc019f6e0 con 0x7fecc0102a20 2024-03-18T22:41:09.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc52ac640 1 --2- 172.21.15.40:0/1823350703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecc0101830 0x7fecc0197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:09.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.985+0000 7fecc4aab640 1 --2- 172.21.15.40:0/1823350703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0102a20 0x7fecc0198000 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fecb0006010 tx=0x7fecb0005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:09.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.986+0000 7fecb67fc640 1 -- 172.21.15.40:0/1823350703 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb0043020 con 0x7fecc0102a20 2024-03-18T22:41:09.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.986+0000 7fecc7510640 1 -- 172.21.15.40:0/1823350703 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecc019f970 con 0x7fecc0102a20 2024-03-18T22:41:09.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.986+0000 7fecb67fc640 1 -- 172.21.15.40:0/1823350703 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fecb00043d0 con 0x7fecc0102a20 2024-03-18T22:41:09.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.986+0000 7fecb67fc640 1 -- 172.21.15.40:0/1823350703 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb0003a80 con 0x7fecc0102a20 2024-03-18T22:41:09.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.986+0000 7fecc7510640 1 -- 172.21.15.40:0/1823350703 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc019ff10 con 0x7fecc0102a20 2024-03-18T22:41:09.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.987+0000 7fecc7510640 1 -- 172.21.15.40:0/1823350703 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecc0108c10 con 0x7fecc0102a20 2024-03-18T22:41:09.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.988+0000 7fecb67fc640 1 -- 172.21.15.40:0/1823350703 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fecb0046020 con 0x7fecc0102a20 2024-03-18T22:41:09.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.988+0000 7fecb67fc640 1 --2- 172.21.15.40:0/1823350703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fecac0a2800 0x7fecac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:09.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.988+0000 7fecc52ac640 1 --2- 172.21.15.40:0/1823350703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fecac0a2800 0x7fecac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:09.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.988+0000 7fecb67fc640 1 -- 172.21.15.40:0/1823350703 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fecb00f0e90 con 0x7fecc0102a20 2024-03-18T22:41:09.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.989+0000 7fecc52ac640 1 --2- 172.21.15.40:0/1823350703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fecac0a2800 0x7fecac0a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fecc0102880 tx=0x7fecbc005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:09.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:09.992+0000 7fecb67fc640 1 -- 172.21.15.40:0/1823350703 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fecb004d020 con 0x7fecc0102a20 2024-03-18T22:41:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:09 smithi113 ceph-mon[66775]: from='client.14853 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:41:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:09 smithi113 ceph-mon[66775]: pgmap v302: 129 pgs: 129 active+clean; 582 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 1023 B/s wr, 4 op/s 2024-03-18T22:41:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:09 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] up:active 2024-03-18T22:41:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:09 smithi113 ceph-mon[66775]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-18T22:41:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:09 smithi113 ceph-mon[66775]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T22:41:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:09 smithi113 ceph-mon[66775]: Cluster is now healthy 2024-03-18T22:41:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:09 smithi113 ceph-mon[66775]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=k=up:active} 3 up:standby-replay 5 up:standby 2024-03-18T22:41:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:09 smithi113 ceph-mon[66775]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-03-18T22:41:10.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:09 smithi113 ceph-mon[66775]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-03-18T22:41:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:09 smithi103 ceph-mon[66377]: from='client.14853 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:41:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:09 smithi103 ceph-mon[66377]: pgmap v302: 129 pgs: 129 active+clean; 582 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 1023 B/s wr, 4 op/s 2024-03-18T22:41:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:09 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] up:active 2024-03-18T22:41:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:09 smithi103 ceph-mon[66377]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-18T22:41:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:09 smithi103 ceph-mon[66377]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T22:41:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:09 smithi103 ceph-mon[66377]: Cluster is now healthy 2024-03-18T22:41:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:09 smithi103 ceph-mon[66377]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=k=up:active} 3 up:standby-replay 5 up:standby 2024-03-18T22:41:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:09 smithi103 ceph-mon[66377]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-03-18T22:41:10.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:09 smithi103 ceph-mon[66377]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-03-18T22:41:10.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:09 smithi040 ceph-mon[62643]: from='client.14853 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:41:10.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:09 smithi040 ceph-mon[62643]: pgmap v302: 129 pgs: 129 active+clean; 582 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 1023 B/s wr, 4 op/s 2024-03-18T22:41:10.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:09 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] up:active 2024-03-18T22:41:10.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:09 smithi040 ceph-mon[62643]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-18T22:41:10.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:09 smithi040 ceph-mon[62643]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T22:41:10.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:09 smithi040 ceph-mon[62643]: Cluster is now healthy 2024-03-18T22:41:10.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:09 smithi040 ceph-mon[62643]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=k=up:active} 3 up:standby-replay 5 up:standby 2024-03-18T22:41:10.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:09 smithi040 ceph-mon[62643]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-03-18T22:41:10.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:09 smithi040 ceph-mon[62643]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-03-18T22:41:10.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.204+0000 7fecc7510640 1 -- 172.21.15.40:0/1823350703 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fecc0061c60 con 0x7fecac0a2800 2024-03-18T22:41:10.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.227+0000 7fecb67fc640 1 -- 172.21.15.40:0/1823350703 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fecc0061c60 con 0x7fecac0a2800 2024-03-18T22:41:10.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.229+0000 7fecc7510640 1 -- 172.21.15.40:0/1823350703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fecac0a2800 msgr2=0x7fecac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:10.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.229+0000 7fecc7510640 1 --2- 172.21.15.40:0/1823350703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fecac0a2800 0x7fecac0a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fecc0102880 tx=0x7fecbc005f90 comp rx=0 tx=0).stop 2024-03-18T22:41:10.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.229+0000 7fecc7510640 1 -- 172.21.15.40:0/1823350703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0102a20 msgr2=0x7fecc0198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:10.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.229+0000 7fecc7510640 1 --2- 172.21.15.40:0/1823350703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0102a20 0x7fecc0198000 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fecb0006010 tx=0x7fecb0005e40 comp rx=0 tx=0).stop 2024-03-18T22:41:10.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.229+0000 7fecc7510640 1 -- 172.21.15.40:0/1823350703 shutdown_connections 2024-03-18T22:41:10.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.229+0000 7fecc7510640 1 --2- 172.21.15.40:0/1823350703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fecac0a2800 0x7fecac0a4cc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.229+0000 7fecc7510640 1 --2- 172.21.15.40:0/1823350703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc01033e0 0x7fecc019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.229+0000 7fecc7510640 1 --2- 172.21.15.40:0/1823350703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0102a20 0x7fecc0198000 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.229+0000 7fecc7510640 1 --2- 172.21.15.40:0/1823350703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecc0101830 0x7fecc0197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.229+0000 7fecc7510640 1 -- 172.21.15.40:0/1823350703 >> 172.21.15.40:0/1823350703 conn(0x7fecc00fcdc0 msgr2=0x7fecc0105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.229+0000 7fecc7510640 1 -- 172.21.15.40:0/1823350703 shutdown_connections 2024-03-18T22:41:10.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.229+0000 7fecc7510640 1 -- 172.21.15.40:0/1823350703 wait complete. 2024-03-18T22:41:10.240 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-18T22:41:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.347+0000 7f275889a640 1 -- 172.21.15.40:0/4236750595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27541033f0 msgr2=0x7f2754109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.347+0000 7f275889a640 1 --2- 172.21.15.40:0/4236750595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27541033f0 0x7f2754109c80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2744009ca0 tx=0x7f274402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.347+0000 7f275889a640 1 -- 172.21.15.40:0/4236750595 shutdown_connections 2024-03-18T22:41:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.347+0000 7f275889a640 1 --2- 172.21.15.40:0/4236750595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27541033f0 0x7f2754109c80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.347+0000 7f275889a640 1 --2- 172.21.15.40:0/4236750595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2754102a30 0x7f2754102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.347+0000 7f275889a640 1 --2- 172.21.15.40:0/4236750595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2754101840 0x7f2754101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.347+0000 7f275889a640 1 -- 172.21.15.40:0/4236750595 >> 172.21.15.40:0/4236750595 conn(0x7f27540fcdb0 msgr2=0x7f27540ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.347+0000 7f275889a640 1 -- 172.21.15.40:0/4236750595 shutdown_connections 2024-03-18T22:41:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.347+0000 7f275889a640 1 -- 172.21.15.40:0/4236750595 wait complete. 2024-03-18T22:41:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f275889a640 1 Processor -- start 2024-03-18T22:41:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f275889a640 1 -- start start 2024-03-18T22:41:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f275889a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2754101840 0x7f2754197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f275889a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2754102a30 0x7f2754197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f275889a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27541033f0 0x7f275419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f275889a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f275410d370 con 0x7f2754102a30 2024-03-18T22:41:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f275889a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f275410d1f0 con 0x7f2754101840 2024-03-18T22:41:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f275889a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f275410d4f0 con 0x7f27541033f0 2024-03-18T22:41:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f2751d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2754102a30 0x7f2754197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f2751d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2754102a30 0x7f2754197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40708/0 (socket says 172.21.15.40:40708) 2024-03-18T22:41:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f2751d9b640 1 -- 172.21.15.40:0/2803085879 learned_addr learned my addr 172.21.15.40:0/2803085879 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f2752d9d640 1 --2- 172.21.15.40:0/2803085879 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27541033f0 0x7f275419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f275259c640 1 --2- 172.21.15.40:0/2803085879 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2754101840 0x7f2754197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f2751d9b640 1 -- 172.21.15.40:0/2803085879 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27541033f0 msgr2=0x7f275419efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f2751d9b640 1 --2- 172.21.15.40:0/2803085879 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27541033f0 0x7f275419efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f2751d9b640 1 -- 172.21.15.40:0/2803085879 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2754101840 msgr2=0x7f2754197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f2751d9b640 1 --2- 172.21.15.40:0/2803085879 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2754101840 0x7f2754197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f2751d9b640 1 -- 172.21.15.40:0/2803085879 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f275419f650 con 0x7f2754102a30 2024-03-18T22:41:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.348+0000 7f2751d9b640 1 --2- 172.21.15.40:0/2803085879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2754102a30 0x7f2754197f70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f273c00e9f0 tx=0x7f273c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.349+0000 7f274b7fe640 1 -- 172.21.15.40:0/2803085879 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273c00cd80 con 0x7f2754102a30 2024-03-18T22:41:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.349+0000 7f274b7fe640 1 -- 172.21.15.40:0/2803085879 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f273c0054d0 con 0x7f2754102a30 2024-03-18T22:41:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.349+0000 7f275889a640 1 -- 172.21.15.40:0/2803085879 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f275419f940 con 0x7f2754102a30 2024-03-18T22:41:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.349+0000 7f274b7fe640 1 -- 172.21.15.40:0/2803085879 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273c010490 con 0x7f2754102a30 2024-03-18T22:41:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.349+0000 7f275889a640 1 -- 172.21.15.40:0/2803085879 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f275419fe40 con 0x7f2754102a30 2024-03-18T22:41:10.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.350+0000 7f275889a640 1 -- 172.21.15.40:0/2803085879 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2754108c30 con 0x7f2754102a30 2024-03-18T22:41:10.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.351+0000 7f274b7fe640 1 -- 172.21.15.40:0/2803085879 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f273c005020 con 0x7f2754102a30 2024-03-18T22:41:10.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.351+0000 7f274b7fe640 1 --2- 172.21.15.40:0/2803085879 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f27400a2800 0x7f27400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:10.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.351+0000 7f275259c640 1 --2- 172.21.15.40:0/2803085879 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f27400a2800 0x7f27400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:10.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.351+0000 7f274b7fe640 1 -- 172.21.15.40:0/2803085879 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f273c0cf040 con 0x7f2754102a30 2024-03-18T22:41:10.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.352+0000 7f275259c640 1 --2- 172.21.15.40:0/2803085879 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f27400a2800 0x7f27400a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2754102890 tx=0x7f274c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:10.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.356+0000 7f274b7fe640 1 -- 172.21.15.40:0/2803085879 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f273c09bea0 con 0x7f2754102a30 2024-03-18T22:41:10.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.565+0000 7f275889a640 1 -- 172.21.15.40:0/2803085879 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f2754061c60 con 0x7f27400a2800 2024-03-18T22:41:10.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.569+0000 7f274b7fe640 1 -- 172.21.15.40:0/2803085879 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f2754061c60 con 0x7f27400a2800 2024-03-18T22:41:10.570 INFO:teuthology.orchestra.run.smithi040.stdout:/volumes/qa/sv_0/bdd30de8-89b1-40ee-b382-961fdfc3fe80 2024-03-18T22:41:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.571+0000 7f275889a640 1 -- 172.21.15.40:0/2803085879 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f27400a2800 msgr2=0x7f27400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.571+0000 7f275889a640 1 --2- 172.21.15.40:0/2803085879 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f27400a2800 0x7f27400a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2754102890 tx=0x7f274c005f90 comp rx=0 tx=0).stop 2024-03-18T22:41:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.571+0000 7f275889a640 1 -- 172.21.15.40:0/2803085879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2754102a30 msgr2=0x7f2754197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.571+0000 7f275889a640 1 --2- 172.21.15.40:0/2803085879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2754102a30 0x7f2754197f70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f273c00e9f0 tx=0x7f273c00edf0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.571+0000 7f275889a640 1 -- 172.21.15.40:0/2803085879 shutdown_connections 2024-03-18T22:41:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.571+0000 7f275889a640 1 --2- 172.21.15.40:0/2803085879 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f27400a2800 0x7f27400a4cc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.571+0000 7f275889a640 1 --2- 172.21.15.40:0/2803085879 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f27541033f0 0x7f275419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.571+0000 7f275889a640 1 --2- 172.21.15.40:0/2803085879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2754102a30 0x7f2754197f70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.571+0000 7f275889a640 1 --2- 172.21.15.40:0/2803085879 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2754101840 0x7f2754197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.571+0000 7f275889a640 1 -- 172.21.15.40:0/2803085879 >> 172.21.15.40:0/2803085879 conn(0x7f27540fcdb0 msgr2=0x7f2754105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:10.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.571+0000 7f275889a640 1 -- 172.21.15.40:0/2803085879 shutdown_connections 2024-03-18T22:41:10.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.571+0000 7f275889a640 1 -- 172.21.15.40:0/2803085879 wait complete. 2024-03-18T22:41:10.582 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-18T22:41:10.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.689+0000 7f00dd910640 1 -- 172.21.15.40:0/40295107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00d8109700 msgr2=0x7f00d810bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:10.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.689+0000 7f00dd910640 1 --2- 172.21.15.40:0/40295107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00d8109700 0x7f00d810bb90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f00c8009ca0 tx=0x7f00c802c180 comp rx=0 tx=0).stop 2024-03-18T22:41:10.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.689+0000 7f00dd910640 1 -- 172.21.15.40:0/40295107 shutdown_connections 2024-03-18T22:41:10.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.689+0000 7f00dd910640 1 --2- 172.21.15.40:0/40295107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00d8109700 0x7f00d810bb90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.689+0000 7f00dd910640 1 --2- 172.21.15.40:0/40295107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00d8074200 0x7f00d8074660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.689+0000 7f00dd910640 1 --2- 172.21.15.40:0/40295107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00d80757e0 0x7f00d8073c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.689+0000 7f00dd910640 1 -- 172.21.15.40:0/40295107 >> 172.21.15.40:0/40295107 conn(0x7f00d80fcdc0 msgr2=0x7f00d80ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:10.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.689+0000 7f00dd910640 1 -- 172.21.15.40:0/40295107 shutdown_connections 2024-03-18T22:41:10.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.689+0000 7f00dd910640 1 -- 172.21.15.40:0/40295107 wait complete. 2024-03-18T22:41:10.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.690+0000 7f00dd910640 1 Processor -- start 2024-03-18T22:41:10.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.690+0000 7f00dd910640 1 -- start start 2024-03-18T22:41:10.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.690+0000 7f00dd910640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00d8074200 0x7f00d8199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:10.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.690+0000 7f00dd910640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00d80757e0 0x7f00d819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:10.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.690+0000 7f00dd910640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00d8109700 0x7f00d81a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:10.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.690+0000 7f00dd910640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f00d810f590 con 0x7f00d8109700 2024-03-18T22:41:10.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.690+0000 7f00dd910640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f00d810f410 con 0x7f00d8074200 2024-03-18T22:41:10.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.690+0000 7f00dd910640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f00d810f710 con 0x7f00d80757e0 2024-03-18T22:41:10.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.690+0000 7f00d77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00d8109700 0x7f00d81a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:10.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.690+0000 7f00d67fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00d80757e0 0x7f00d819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:10.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.690+0000 7f00d77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00d8109700 0x7f00d81a11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40724/0 (socket says 172.21.15.40:40724) 2024-03-18T22:41:10.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.690+0000 7f00d77fe640 1 -- 172.21.15.40:0/3713308105 learned_addr learned my addr 172.21.15.40:0/3713308105 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:10.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.690+0000 7f00d6ffd640 1 --2- 172.21.15.40:0/3713308105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00d8074200 0x7f00d8199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:10.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.691+0000 7f00d67fc640 1 -- 172.21.15.40:0/3713308105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00d8074200 msgr2=0x7f00d8199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:10.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.691+0000 7f00d67fc640 1 --2- 172.21.15.40:0/3713308105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00d8074200 0x7f00d8199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.691+0000 7f00d67fc640 1 -- 172.21.15.40:0/3713308105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00d8109700 msgr2=0x7f00d81a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:10.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.691+0000 7f00d67fc640 1 --2- 172.21.15.40:0/3713308105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00d8109700 0x7f00d81a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.691+0000 7f00d67fc640 1 -- 172.21.15.40:0/3713308105 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d81a1890 con 0x7f00d80757e0 2024-03-18T22:41:10.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.691+0000 7f00d77fe640 1 --2- 172.21.15.40:0/3713308105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00d8109700 0x7f00d81a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:10.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.691+0000 7f00d67fc640 1 --2- 172.21.15.40:0/3713308105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00d80757e0 0x7f00d819a1c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00c000e9f0 tx=0x7f00c000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:10.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.691+0000 7f00dc90e640 1 -- 172.21.15.40:0/3713308105 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c000ccc0 con 0x7f00d80757e0 2024-03-18T22:41:10.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.691+0000 7f00dd910640 1 -- 172.21.15.40:0/3713308105 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d81a1b80 con 0x7f00d80757e0 2024-03-18T22:41:10.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.691+0000 7f00dc90e640 1 -- 172.21.15.40:0/3713308105 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00c0005020 con 0x7f00d80757e0 2024-03-18T22:41:10.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.691+0000 7f00dd910640 1 -- 172.21.15.40:0/3713308105 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d81a2080 con 0x7f00d80757e0 2024-03-18T22:41:10.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.691+0000 7f00dc90e640 1 -- 172.21.15.40:0/3713308105 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c001c070 con 0x7f00d80757e0 2024-03-18T22:41:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.692+0000 7f00dd910640 1 -- 172.21.15.40:0/3713308105 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00d81084c0 con 0x7f00d80757e0 2024-03-18T22:41:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.693+0000 7f00dc90e640 1 -- 172.21.15.40:0/3713308105 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f00c00049a0 con 0x7f00d80757e0 2024-03-18T22:41:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.694+0000 7f00dc90e640 1 --2- 172.21.15.40:0/3713308105 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00c40a2800 0x7f00c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.694+0000 7f00d6ffd640 1 --2- 172.21.15.40:0/3713308105 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00c40a2800 0x7f00c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.694+0000 7f00dc90e640 1 -- 172.21.15.40:0/3713308105 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f00c00ced60 con 0x7f00d80757e0 2024-03-18T22:41:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.694+0000 7f00d6ffd640 1 --2- 172.21.15.40:0/3713308105 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00c40a2800 0x7f00c40a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f00d819b150 tx=0x7f00d0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:10.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.697+0000 7f00dc90e640 1 -- 172.21.15.40:0/3713308105 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f00c009bbc0 con 0x7f00d80757e0 2024-03-18T22:41:10.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.911+0000 7f00dd910640 1 -- 172.21.15.40:0/3713308105 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f00d8061c60 con 0x7f00c40a2800 2024-03-18T22:41:10.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.935+0000 7f00dc90e640 1 -- 172.21.15.40:0/3713308105 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f00d8061c60 con 0x7f00c40a2800 2024-03-18T22:41:10.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.937+0000 7f00dd910640 1 -- 172.21.15.40:0/3713308105 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00c40a2800 msgr2=0x7f00c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:10.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.937+0000 7f00dd910640 1 --2- 172.21.15.40:0/3713308105 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00c40a2800 0x7f00c40a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f00d819b150 tx=0x7f00d0005f90 comp rx=0 tx=0).stop 2024-03-18T22:41:10.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.937+0000 7f00dd910640 1 -- 172.21.15.40:0/3713308105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00d80757e0 msgr2=0x7f00d819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:10.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.937+0000 7f00dd910640 1 --2- 172.21.15.40:0/3713308105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00d80757e0 0x7f00d819a1c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00c000e9f0 tx=0x7f00c000eeb0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.937+0000 7f00dd910640 1 -- 172.21.15.40:0/3713308105 shutdown_connections 2024-03-18T22:41:10.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.937+0000 7f00dd910640 1 --2- 172.21.15.40:0/3713308105 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00c40a2800 0x7f00c40a4cc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.937+0000 7f00dd910640 1 --2- 172.21.15.40:0/3713308105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00d8109700 0x7f00d81a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.937+0000 7f00dd910640 1 --2- 172.21.15.40:0/3713308105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00d80757e0 0x7f00d819a1c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.937+0000 7f00dd910640 1 --2- 172.21.15.40:0/3713308105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00d8074200 0x7f00d8199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:10.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.937+0000 7f00dd910640 1 -- 172.21.15.40:0/3713308105 >> 172.21.15.40:0/3713308105 conn(0x7f00d80fcdc0 msgr2=0x7f00d8105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:10.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.937+0000 7f00dd910640 1 -- 172.21.15.40:0/3713308105 shutdown_connections 2024-03-18T22:41:10.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:10.937+0000 7f00dd910640 1 -- 172.21.15.40:0/3713308105 wait complete. 2024-03-18T22:41:10.948 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-18T22:41:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.055+0000 7f6162910640 1 -- 172.21.15.40:0/3966621619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f615c101920 msgr2=0x7f615c101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.055+0000 7f6162910640 1 --2- 172.21.15.40:0/3966621619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f615c101920 0x7f615c101da0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6144009900 tx=0x7f614402bc30 comp rx=0 tx=0).stop 2024-03-18T22:41:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.055+0000 7f6162910640 1 -- 172.21.15.40:0/3966621619 shutdown_connections 2024-03-18T22:41:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.055+0000 7f6162910640 1 --2- 172.21.15.40:0/3966621619 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f615c1022e0 0x7f615c108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.055+0000 7f6162910640 1 --2- 172.21.15.40:0/3966621619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f615c101920 0x7f615c101da0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.055+0000 7f6162910640 1 --2- 172.21.15.40:0/3966621619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f615c100730 0x7f615c100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.055+0000 7f6162910640 1 -- 172.21.15.40:0/3966621619 >> 172.21.15.40:0/3966621619 conn(0x7f615c0fbcc0 msgr2=0x7f615c0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.055+0000 7f6162910640 1 -- 172.21.15.40:0/3966621619 shutdown_connections 2024-03-18T22:41:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.055+0000 7f6162910640 1 -- 172.21.15.40:0/3966621619 wait complete. 2024-03-18T22:41:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.056+0000 7f6162910640 1 Processor -- start 2024-03-18T22:41:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.056+0000 7f6162910640 1 -- start start 2024-03-18T22:41:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.056+0000 7f6162910640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f615c100730 0x7f615c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.056+0000 7f6162910640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f615c101920 0x7f615c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.056+0000 7f6162910640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f615c1022e0 0x7f615c1a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.056+0000 7f6162910640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f615c10b160 con 0x7f615c100730 2024-03-18T22:41:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.056+0000 7f6162910640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f615c10afe0 con 0x7f615c1022e0 2024-03-18T22:41:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.056+0000 7f6162910640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f615c10b2e0 con 0x7f615c101920 2024-03-18T22:41:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.056+0000 7f615bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f615c100730 0x7f615c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.056+0000 7f615bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f615c100730 0x7f615c199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40736/0 (socket says 172.21.15.40:40736) 2024-03-18T22:41:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.056+0000 7f615bfff640 1 -- 172.21.15.40:0/2610726650 learned_addr learned my addr 172.21.15.40:0/2610726650 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.056+0000 7f6160ead640 1 --2- 172.21.15.40:0/2610726650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f615c1022e0 0x7f615c1a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.056+0000 7f615b7fe640 1 --2- 172.21.15.40:0/2610726650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f615c101920 0x7f615c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.057+0000 7f615bfff640 1 -- 172.21.15.40:0/2610726650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f615c101920 msgr2=0x7f615c19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.057+0000 7f615bfff640 1 --2- 172.21.15.40:0/2610726650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f615c101920 0x7f615c19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.057+0000 7f615bfff640 1 -- 172.21.15.40:0/2610726650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f615c1022e0 msgr2=0x7f615c1a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.057+0000 7f615bfff640 1 --2- 172.21.15.40:0/2610726650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f615c1022e0 0x7f615c1a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.057+0000 7f615bfff640 1 -- 172.21.15.40:0/2610726650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615c1a18b0 con 0x7f615c100730 2024-03-18T22:41:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.057+0000 7f615bfff640 1 --2- 172.21.15.40:0/2610726650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f615c100730 0x7f615c199c70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f614c00bfd0 tx=0x7f614c00ae10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.057+0000 7f61597fa640 1 -- 172.21.15.40:0/2610726650 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c019070 con 0x7f615c100730 2024-03-18T22:41:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.057+0000 7f61597fa640 1 -- 172.21.15.40:0/2610726650 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f614c014410 con 0x7f615c100730 2024-03-18T22:41:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.057+0000 7f6162910640 1 -- 172.21.15.40:0/2610726650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f615c1a1ba0 con 0x7f615c100730 2024-03-18T22:41:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.057+0000 7f61597fa640 1 -- 172.21.15.40:0/2610726650 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c013610 con 0x7f615c100730 2024-03-18T22:41:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.057+0000 7f6162910640 1 -- 172.21.15.40:0/2610726650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f615c1a20c0 con 0x7f615c100730 2024-03-18T22:41:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.058+0000 7f6162910640 1 -- 172.21.15.40:0/2610726650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f615c107b20 con 0x7f615c100730 2024-03-18T22:41:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.059+0000 7f61597fa640 1 -- 172.21.15.40:0/2610726650 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f614c01c070 con 0x7f615c100730 2024-03-18T22:41:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.059+0000 7f61597fa640 1 --2- 172.21.15.40:0/2610726650 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f61480a2800 0x7f61480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.060+0000 7f615b7fe640 1 --2- 172.21.15.40:0/2610726650 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f61480a2800 0x7f61480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.060+0000 7f61597fa640 1 -- 172.21.15.40:0/2610726650 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f614c0cdf80 con 0x7f615c100730 2024-03-18T22:41:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.060+0000 7f615b7fe640 1 --2- 172.21.15.40:0/2610726650 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f61480a2800 0x7f61480a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6144009230 tx=0x7f6144005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.064+0000 7f61597fa640 1 -- 172.21.15.40:0/2610726650 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f614c09a730 con 0x7f615c100730 2024-03-18T22:41:11.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:10 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] up:active 2024-03-18T22:41:11.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:10 smithi103 ceph-mon[66377]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-18T22:41:11.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:10 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:41:11.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:10 smithi103 ceph-mon[66377]: from='client.14865 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:41:11.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:10 smithi103 ceph-mon[66377]: from='client.14871 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:41:11.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:10 smithi103 ceph-mon[66377]: pgmap v303: 129 pgs: 129 active+clean; 585 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-18T22:41:11.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:10 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] up:active 2024-03-18T22:41:11.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:10 smithi040 ceph-mon[62643]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-18T22:41:11.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:10 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:41:11.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:10 smithi040 ceph-mon[62643]: from='client.14865 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:41:11.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:10 smithi040 ceph-mon[62643]: from='client.14871 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:41:11.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:10 smithi040 ceph-mon[62643]: pgmap v303: 129 pgs: 129 active+clean; 585 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-18T22:41:11.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.277+0000 7f6162910640 1 -- 172.21.15.40:0/2610726650 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f615c061c60 con 0x7f61480a2800 2024-03-18T22:41:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.281+0000 7f61597fa640 1 -- 172.21.15.40:0/2610726650 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f615c061c60 con 0x7f61480a2800 2024-03-18T22:41:11.282 INFO:teuthology.orchestra.run.smithi040.stdout:/volumes/qa/sv_1/6096fb71-95e4-4a1c-a901-2f5ab4abcede 2024-03-18T22:41:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.283+0000 7f6162910640 1 -- 172.21.15.40:0/2610726650 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f61480a2800 msgr2=0x7f61480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.283+0000 7f6162910640 1 --2- 172.21.15.40:0/2610726650 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f61480a2800 0x7f61480a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6144009230 tx=0x7f6144005fd0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.283+0000 7f6162910640 1 -- 172.21.15.40:0/2610726650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f615c100730 msgr2=0x7f615c199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.283+0000 7f6162910640 1 --2- 172.21.15.40:0/2610726650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f615c100730 0x7f615c199c70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f614c00bfd0 tx=0x7f614c00ae10 comp rx=0 tx=0).stop 2024-03-18T22:41:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.283+0000 7f6162910640 1 -- 172.21.15.40:0/2610726650 shutdown_connections 2024-03-18T22:41:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.283+0000 7f6162910640 1 --2- 172.21.15.40:0/2610726650 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f61480a2800 0x7f61480a4cc0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.283+0000 7f6162910640 1 --2- 172.21.15.40:0/2610726650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f615c1022e0 0x7f615c1a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.283+0000 7f6162910640 1 --2- 172.21.15.40:0/2610726650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f615c101920 0x7f615c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.283+0000 7f6162910640 1 --2- 172.21.15.40:0/2610726650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f615c100730 0x7f615c199c70 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.283+0000 7f6162910640 1 -- 172.21.15.40:0/2610726650 >> 172.21.15.40:0/2610726650 conn(0x7f615c0fbcc0 msgr2=0x7f615c104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.283+0000 7f6162910640 1 -- 172.21.15.40:0/2610726650 shutdown_connections 2024-03-18T22:41:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.283+0000 7f6162910640 1 -- 172.21.15.40:0/2610726650 wait complete. 2024-03-18T22:41:11.294 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:41:11.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:10 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] up:active 2024-03-18T22:41:11.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:10 smithi113 ceph-mon[66775]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-18T22:41:11.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:10 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:41:11.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:10 smithi113 ceph-mon[66775]: from='client.14865 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:41:11.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:10 smithi113 ceph-mon[66775]: from='client.14871 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:41:11.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:10 smithi113 ceph-mon[66775]: pgmap v303: 129 pgs: 129 active+clean; 585 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-03-18T22:41:11.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.395+0000 7fb535174640 1 -- 172.21.15.40:0/3173412613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb530101930 msgr2=0x7fb530101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.395+0000 7fb535174640 1 --2- 172.21.15.40:0/3173412613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb530101930 0x7fb530101db0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb518009900 tx=0x7fb51802bc30 comp rx=0 tx=0).stop 2024-03-18T22:41:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.395+0000 7fb535174640 1 -- 172.21.15.40:0/3173412613 shutdown_connections 2024-03-18T22:41:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.395+0000 7fb535174640 1 --2- 172.21.15.40:0/3173412613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb5301022f0 0x7fb530108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.396+0000 7fb535174640 1 --2- 172.21.15.40:0/3173412613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb530101930 0x7fb530101db0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.396+0000 7fb535174640 1 --2- 172.21.15.40:0/3173412613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb530100740 0x7fb530100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.396+0000 7fb535174640 1 -- 172.21.15.40:0/3173412613 >> 172.21.15.40:0/3173412613 conn(0x7fb5300fbcb0 msgr2=0x7fb5300fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.396+0000 7fb535174640 1 -- 172.21.15.40:0/3173412613 shutdown_connections 2024-03-18T22:41:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.396+0000 7fb535174640 1 -- 172.21.15.40:0/3173412613 wait complete. 2024-03-18T22:41:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.396+0000 7fb535174640 1 Processor -- start 2024-03-18T22:41:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.396+0000 7fb535174640 1 -- start start 2024-03-18T22:41:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb535174640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb530100740 0x7fb530197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb535174640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb530101930 0x7fb530198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb535174640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5301022f0 0x7fb53019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb535174640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb53010c250 con 0x7fb5301022f0 2024-03-18T22:41:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb535174640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb53010c0d0 con 0x7fb530100740 2024-03-18T22:41:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb535174640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb53010c3d0 con 0x7fb530101930 2024-03-18T22:41:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb52f59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5301022f0 0x7fb53019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb52f59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5301022f0 0x7fb53019f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40752/0 (socket says 172.21.15.40:40752) 2024-03-18T22:41:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb52f59e640 1 -- 172.21.15.40:0/3515068510 learned_addr learned my addr 172.21.15.40:0/3515068510 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb52ed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb530100740 0x7fb530197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb52659c640 1 --2- 172.21.15.40:0/3515068510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb530101930 0x7fb530198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb52f59e640 1 -- 172.21.15.40:0/3515068510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb530101930 msgr2=0x7fb530198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb52f59e640 1 --2- 172.21.15.40:0/3515068510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb530101930 0x7fb530198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb52f59e640 1 -- 172.21.15.40:0/3515068510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb530100740 msgr2=0x7fb530197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb52f59e640 1 --2- 172.21.15.40:0/3515068510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb530100740 0x7fb530197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb52f59e640 1 -- 172.21.15.40:0/3515068510 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53019f6f0 con 0x7fb5301022f0 2024-03-18T22:41:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb52659c640 1 --2- 172.21.15.40:0/3515068510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb530101930 0x7fb530198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb52f59e640 1 --2- 172.21.15.40:0/3515068510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5301022f0 0x7fb53019f080 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb52800eb10 tx=0x7fb52800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb52cd99640 1 -- 172.21.15.40:0/3515068510 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528004b20 con 0x7fb5301022f0 2024-03-18T22:41:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb535174640 1 -- 172.21.15.40:0/3515068510 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb53019f9e0 con 0x7fb5301022f0 2024-03-18T22:41:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb52cd99640 1 -- 172.21.15.40:0/3515068510 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb528004cc0 con 0x7fb5301022f0 2024-03-18T22:41:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb52cd99640 1 -- 172.21.15.40:0/3515068510 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528010690 con 0x7fb5301022f0 2024-03-18T22:41:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.397+0000 7fb535174640 1 -- 172.21.15.40:0/3515068510 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb53019ff00 con 0x7fb5301022f0 2024-03-18T22:41:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.398+0000 7fb535174640 1 -- 172.21.15.40:0/3515068510 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5301a01c0 con 0x7fb5301022f0 2024-03-18T22:41:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.399+0000 7fb52cd99640 1 -- 172.21.15.40:0/3515068510 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb528005020 con 0x7fb5301022f0 2024-03-18T22:41:11.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.400+0000 7fb52cd99640 1 --2- 172.21.15.40:0/3515068510 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb51c0a2800 0x7fb51c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:11.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.400+0000 7fb52ed9d640 1 --2- 172.21.15.40:0/3515068510 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb51c0a2800 0x7fb51c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:11.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.400+0000 7fb52cd99640 1 -- 172.21.15.40:0/3515068510 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb5280cf290 con 0x7fb5301022f0 2024-03-18T22:41:11.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.400+0000 7fb52ed9d640 1 --2- 172.21.15.40:0/3515068510 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb51c0a2800 0x7fb51c0a4cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb530101790 tx=0x7fb520005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:11.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.405+0000 7fb52cd99640 1 -- 172.21.15.40:0/3515068510 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb52809c090 con 0x7fb5301022f0 2024-03-18T22:41:11.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.628+0000 7fb535174640 1 -- 172.21.15.40:0/3515068510 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5301a0390 con 0x7fb5301022f0 2024-03-18T22:41:11.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.629+0000 7fb52cd99640 1 -- 172.21.15.40:0/3515068510 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7fb52809b990 con 0x7fb5301022f0 2024-03-18T22:41:11.630 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:41:11.630 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:41:09.839105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24481,24509,24473,24401,24388]},"id":1}]} 2024-03-18T22:41:11.630 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 27 2024-03-18T22:41:11.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.631+0000 7fb535174640 1 -- 172.21.15.40:0/3515068510 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb51c0a2800 msgr2=0x7fb51c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:11.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.631+0000 7fb535174640 1 --2- 172.21.15.40:0/3515068510 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb51c0a2800 0x7fb51c0a4cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb530101790 tx=0x7fb520005f90 comp rx=0 tx=0).stop 2024-03-18T22:41:11.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.631+0000 7fb535174640 1 -- 172.21.15.40:0/3515068510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5301022f0 msgr2=0x7fb53019f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:11.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.631+0000 7fb535174640 1 --2- 172.21.15.40:0/3515068510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5301022f0 0x7fb53019f080 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb52800eb10 tx=0x7fb52800c2d0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.631+0000 7fb535174640 1 -- 172.21.15.40:0/3515068510 shutdown_connections 2024-03-18T22:41:11.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.631+0000 7fb535174640 1 --2- 172.21.15.40:0/3515068510 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb51c0a2800 0x7fb51c0a4cc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.631+0000 7fb535174640 1 --2- 172.21.15.40:0/3515068510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5301022f0 0x7fb53019f080 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.631+0000 7fb535174640 1 --2- 172.21.15.40:0/3515068510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb530101930 0x7fb530198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.631+0000 7fb535174640 1 --2- 172.21.15.40:0/3515068510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb530100740 0x7fb530197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.631+0000 7fb535174640 1 -- 172.21.15.40:0/3515068510 >> 172.21.15.40:0/3515068510 conn(0x7fb5300fbcb0 msgr2=0x7fb530068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:11.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.631+0000 7fb535174640 1 -- 172.21.15.40:0/3515068510 shutdown_connections 2024-03-18T22:41:11.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.631+0000 7fb535174640 1 -- 172.21.15.40:0/3515068510 wait complete. 2024-03-18T22:41:11.643 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:41:11.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.745+0000 7fcddd7ff640 1 -- 172.21.15.40:0/2771732401 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdd8100590 msgr2=0x7fcdd81009b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:11.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.745+0000 7fcddd7ff640 1 --2- 172.21.15.40:0/2771732401 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdd8100590 0x7fcdd81009b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009900 tx=0x7fcdc802bc30 comp rx=0 tx=0).stop 2024-03-18T22:41:11.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.745+0000 7fcddd7ff640 1 -- 172.21.15.40:0/2771732401 shutdown_connections 2024-03-18T22:41:11.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.745+0000 7fcddd7ff640 1 --2- 172.21.15.40:0/2771732401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd8102140 0x7fcdd81089d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.745+0000 7fcddd7ff640 1 --2- 172.21.15.40:0/2771732401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdd8101780 0x7fcdd8101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.745+0000 7fcddd7ff640 1 --2- 172.21.15.40:0/2771732401 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdd8100590 0x7fcdd81009b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.745+0000 7fcddd7ff640 1 -- 172.21.15.40:0/2771732401 >> 172.21.15.40:0/2771732401 conn(0x7fcdd80fbb00 msgr2=0x7fcdd80fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:11.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.745+0000 7fcddd7ff640 1 -- 172.21.15.40:0/2771732401 shutdown_connections 2024-03-18T22:41:11.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.745+0000 7fcddd7ff640 1 -- 172.21.15.40:0/2771732401 wait complete. 2024-03-18T22:41:11.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcddd7ff640 1 Processor -- start 2024-03-18T22:41:11.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcddd7ff640 1 -- start start 2024-03-18T22:41:11.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcddd7ff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdd8100590 0x7fcdd8104520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:11.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcddd7ff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdd8101780 0x7fcdd8102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:11.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcddd7ff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd8102140 0x7fcdd81030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:11.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcddd7ff640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd810c100 con 0x7fcdd8102140 2024-03-18T22:41:11.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcddd7ff640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd810bf80 con 0x7fcdd8101780 2024-03-18T22:41:11.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcddd7ff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd810c280 con 0x7fcdd8100590 2024-03-18T22:41:11.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcdd6ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdd8100590 0x7fcdd8104520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:11.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd8102140 0x7fcdd81030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:11.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcdd6ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdd8100590 0x7fcdd8104520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:49772/0 (socket says 172.21.15.40:49772) 2024-03-18T22:41:11.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcdd6ffd640 1 -- 172.21.15.40:0/1303051821 learned_addr learned my addr 172.21.15.40:0/1303051821 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:11.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcdcffff640 1 --2- 172.21.15.40:0/1303051821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdd8101780 0x7fcdd8102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:11.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcdd6ffd640 1 -- 172.21.15.40:0/1303051821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdd8101780 msgr2=0x7fcdd8102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:11.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcdd6ffd640 1 --2- 172.21.15.40:0/1303051821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdd8101780 0x7fcdd8102b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcdd6ffd640 1 -- 172.21.15.40:0/1303051821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd8102140 msgr2=0x7fcdd81030a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:11.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcdd6ffd640 1 --2- 172.21.15.40:0/1303051821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd8102140 0x7fcdd81030a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcdd6ffd640 1 -- 172.21.15.40:0/1303051821 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd8103930 con 0x7fcdd8100590 2024-03-18T22:41:11.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.746+0000 7fcdd77fe640 1 --2- 172.21.15.40:0/1303051821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd8102140 0x7fcdd81030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:11.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.747+0000 7fcdd6ffd640 1 --2- 172.21.15.40:0/1303051821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdd8100590 0x7fcdd8104520 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8000c00 tx=0x7fcdc8005e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:11.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.747+0000 7fcdd4ff9640 1 -- 172.21.15.40:0/1303051821 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8043020 con 0x7fcdd8100590 2024-03-18T22:41:11.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.747+0000 7fcdd4ff9640 1 -- 172.21.15.40:0/1303051821 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdc80043d0 con 0x7fcdd8100590 2024-03-18T22:41:11.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.747+0000 7fcddd7ff640 1 -- 172.21.15.40:0/1303051821 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd81a1a20 con 0x7fcdd8100590 2024-03-18T22:41:11.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.747+0000 7fcdd4ff9640 1 -- 172.21.15.40:0/1303051821 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8003bb0 con 0x7fcdd8100590 2024-03-18T22:41:11.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.747+0000 7fcddd7ff640 1 -- 172.21.15.40:0/1303051821 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd81a1f70 con 0x7fcdd8100590 2024-03-18T22:41:11.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.748+0000 7fcddd7ff640 1 -- 172.21.15.40:0/1303051821 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdd81a2140 con 0x7fcdd8100590 2024-03-18T22:41:11.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.749+0000 7fcdd4ff9640 1 -- 172.21.15.40:0/1303051821 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fcdc8046020 con 0x7fcdd8100590 2024-03-18T22:41:11.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.749+0000 7fcdd4ff9640 1 --2- 172.21.15.40:0/1303051821 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcdc40a2800 0x7fcdc40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:11.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.750+0000 7fcdcffff640 1 --2- 172.21.15.40:0/1303051821 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcdc40a2800 0x7fcdc40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:11.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.750+0000 7fcdd4ff9640 1 -- 172.21.15.40:0/1303051821 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcdc80f0e20 con 0x7fcdd8100590 2024-03-18T22:41:11.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.750+0000 7fcdcffff640 1 --2- 172.21.15.40:0/1303051821 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcdc40a2800 0x7fcdc40a4cc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8104290 tx=0x7fcdc0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:11.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.754+0000 7fcdd4ff9640 1 -- 172.21.15.40:0/1303051821 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcdc804d020 con 0x7fcdd8100590 2024-03-18T22:41:11.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.858+0000 7fcdd4ff9640 1 -- 172.21.15.40:0/1303051821 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fcdc80f6020 con 0x7fcdd8100590 2024-03-18T22:41:11.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:11 smithi040 ceph-mon[62643]: from='client.24751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:41:11.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:11 smithi040 ceph-mon[62643]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:41:11.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:11 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3515068510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:41:11.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.946+0000 7fcddd7ff640 1 -- 172.21.15.40:0/1303051821 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcdd81a2310 con 0x7fcdd8100590 2024-03-18T22:41:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.948+0000 7fcdd4ff9640 1 -- 172.21.15.40:0/1303051821 <== mon.1 v2:172.21.15.113:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24129 (secure 0 0 0) 0x7fcdc80bdc20 con 0x7fcdd8100590 2024-03-18T22:41:11.949 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:41:11.949 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":89,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:40:46.196741+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:41:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.950+0000 7fcddd7ff640 1 -- 172.21.15.40:0/1303051821 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcdc40a2800 msgr2=0x7fcdc40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.950+0000 7fcddd7ff640 1 --2- 172.21.15.40:0/1303051821 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcdc40a2800 0x7fcdc40a4cc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8104290 tx=0x7fcdc0005f90 comp rx=0 tx=0).stop 2024-03-18T22:41:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.950+0000 7fcddd7ff640 1 -- 172.21.15.40:0/1303051821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdd8100590 msgr2=0x7fcdd8104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.950+0000 7fcddd7ff640 1 --2- 172.21.15.40:0/1303051821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdd8100590 0x7fcdd8104520 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8000c00 tx=0x7fcdc8005e60 comp rx=0 tx=0).stop 2024-03-18T22:41:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.950+0000 7fcddd7ff640 1 -- 172.21.15.40:0/1303051821 shutdown_connections 2024-03-18T22:41:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.950+0000 7fcddd7ff640 1 --2- 172.21.15.40:0/1303051821 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcdc40a2800 0x7fcdc40a4cc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.950+0000 7fcddd7ff640 1 --2- 172.21.15.40:0/1303051821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcdd8102140 0x7fcdd81030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.950+0000 7fcddd7ff640 1 --2- 172.21.15.40:0/1303051821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdd8101780 0x7fcdd8102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.950+0000 7fcddd7ff640 1 --2- 172.21.15.40:0/1303051821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcdd8100590 0x7fcdd8104520 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.950+0000 7fcddd7ff640 1 -- 172.21.15.40:0/1303051821 >> 172.21.15.40:0/1303051821 conn(0x7fcdd80fbb00 msgr2=0x7fcdd80fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.950+0000 7fcddd7ff640 1 -- 172.21.15.40:0/1303051821 shutdown_connections 2024-03-18T22:41:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:11.950+0000 7fcddd7ff640 1 -- 172.21.15.40:0/1303051821 wait complete. 2024-03-18T22:41:11.963 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T22:41:11.963 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T22:41:12.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.065+0000 7fe2f626c640 1 -- 172.21.15.40:0/2206535116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2f0102320 msgr2=0x7fe2f0104710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:12.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.065+0000 7fe2f626c640 1 --2- 172.21.15.40:0/2206535116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2f0102320 0x7fe2f0104710 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009900 tx=0x7fe2e002bc30 comp rx=0 tx=0).stop 2024-03-18T22:41:12.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.065+0000 7fe2f626c640 1 -- 172.21.15.40:0/2206535116 shutdown_connections 2024-03-18T22:41:12.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.065+0000 7fe2f626c640 1 --2- 172.21.15.40:0/2206535116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f0107580 0x7fe2f0109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.065+0000 7fe2f626c640 1 --2- 172.21.15.40:0/2206535116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2f0104c50 0x7fe2f0107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.065+0000 7fe2f626c640 1 --2- 172.21.15.40:0/2206535116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2f0102320 0x7fe2f0104710 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.065+0000 7fe2f626c640 1 -- 172.21.15.40:0/2206535116 >> 172.21.15.40:0/2206535116 conn(0x7fe2f00fbcb0 msgr2=0x7fe2f00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:12.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.066+0000 7fe2f626c640 1 -- 172.21.15.40:0/2206535116 shutdown_connections 2024-03-18T22:41:12.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.066+0000 7fe2f626c640 1 -- 172.21.15.40:0/2206535116 wait complete. 2024-03-18T22:41:12.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.066+0000 7fe2f626c640 1 Processor -- start 2024-03-18T22:41:12.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.066+0000 7fe2f626c640 1 -- start start 2024-03-18T22:41:12.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.066+0000 7fe2f626c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f0102320 0x7fe2f0199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:12.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.066+0000 7fe2f626c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2f0104c50 0x7fe2f019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:12.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.066+0000 7fe2f626c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2f0107580 0x7fe2f01a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:12.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.066+0000 7fe2f626c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f010d390 con 0x7fe2f0107580 2024-03-18T22:41:12.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.066+0000 7fe2f626c640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f010d210 con 0x7fe2f0102320 2024-03-18T22:41:12.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.066+0000 7fe2f626c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f010d510 con 0x7fe2f0104c50 2024-03-18T22:41:12.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2f4809640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2f0107580 0x7fe2f01a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:12.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2effff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f0102320 0x7fe2f0199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:12.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2effff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f0102320 0x7fe2f0199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:41082/0 (socket says 172.21.15.40:41082) 2024-03-18T22:41:12.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2effff640 1 -- 172.21.15.40:0/1988044124 learned_addr learned my addr 172.21.15.40:0/1988044124 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:12.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2ef7fe640 1 --2- 172.21.15.40:0/1988044124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2f0104c50 0x7fe2f019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:12.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2effff640 1 -- 172.21.15.40:0/1988044124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2f0104c50 msgr2=0x7fe2f019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:12.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2effff640 1 --2- 172.21.15.40:0/1988044124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2f0104c50 0x7fe2f019a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2effff640 1 -- 172.21.15.40:0/1988044124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2f0107580 msgr2=0x7fe2f01a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:12.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2effff640 1 --2- 172.21.15.40:0/1988044124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2f0107580 0x7fe2f01a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2effff640 1 -- 172.21.15.40:0/1988044124 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f01a18e0 con 0x7fe2f0102320 2024-03-18T22:41:12.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2effff640 1 --2- 172.21.15.40:0/1988044124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f0102320 0x7fe2f0199ca0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0006010 tx=0x7fe2e0005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:12.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2ed7fa640 1 -- 172.21.15.40:0/1988044124 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0043020 con 0x7fe2f0102320 2024-03-18T22:41:12.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2f626c640 1 -- 172.21.15.40:0/1988044124 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f01a1b70 con 0x7fe2f0102320 2024-03-18T22:41:12.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2ed7fa640 1 -- 172.21.15.40:0/1988044124 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2e00043d0 con 0x7fe2f0102320 2024-03-18T22:41:12.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2ed7fa640 1 -- 172.21.15.40:0/1988044124 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0003b80 con 0x7fe2f0102320 2024-03-18T22:41:12.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.067+0000 7fe2f626c640 1 -- 172.21.15.40:0/1988044124 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f01a20c0 con 0x7fe2f0102320 2024-03-18T22:41:12.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.068+0000 7fe2f626c640 1 -- 172.21.15.40:0/1988044124 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f0109070 con 0x7fe2f0102320 2024-03-18T22:41:12.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.069+0000 7fe2ed7fa640 1 -- 172.21.15.40:0/1988044124 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe2e0046020 con 0x7fe2f0102320 2024-03-18T22:41:12.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.070+0000 7fe2ed7fa640 1 --2- 172.21.15.40:0/1988044124 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe2d80a2890 0x7fe2d80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:12.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.070+0000 7fe2ef7fe640 1 --2- 172.21.15.40:0/1988044124 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe2d80a2890 0x7fe2d80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:12.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.070+0000 7fe2ed7fa640 1 -- 172.21.15.40:0/1988044124 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe2e00f1580 con 0x7fe2f0102320 2024-03-18T22:41:12.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.070+0000 7fe2ef7fe640 1 --2- 172.21.15.40:0/1988044124 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe2d80a2890 0x7fe2d80a4d50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe2f019b1a0 tx=0x7fe2e4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:12.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.078+0000 7fe2ed7fa640 1 -- 172.21.15.40:0/1988044124 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe2e004d020 con 0x7fe2f0102320 2024-03-18T22:41:12.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:11 smithi103 ceph-mon[66377]: from='client.24751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:41:12.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:11 smithi103 ceph-mon[66377]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:41:12.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:11 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3515068510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:41:12.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.257+0000 7fe2f626c640 1 -- 172.21.15.40:0/1988044124 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe2f0061c60 con 0x7fe2d80a2890 2024-03-18T22:41:12.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.267+0000 7fe2ed7fa640 1 -- 172.21.15.40:0/1988044124 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340307 (secure 0 0 0) 0x7fe2f0061c60 con 0x7fe2d80a2890 2024-03-18T22:41:12.272 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:41:12.274 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-18T22:41:12.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.271+0000 7fe2f626c640 1 -- 172.21.15.40:0/1988044124 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe2d80a2890 msgr2=0x7fe2d80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:12.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.271+0000 7fe2f626c640 1 --2- 172.21.15.40:0/1988044124 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe2d80a2890 0x7fe2d80a4d50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe2f019b1a0 tx=0x7fe2e4005f90 comp rx=0 tx=0).stop 2024-03-18T22:41:12.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.271+0000 7fe2f626c640 1 -- 172.21.15.40:0/1988044124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f0102320 msgr2=0x7fe2f0199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:12.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.271+0000 7fe2f626c640 1 --2- 172.21.15.40:0/1988044124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f0102320 0x7fe2f0199ca0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0006010 tx=0x7fe2e0005e40 comp rx=0 tx=0).stop 2024-03-18T22:41:12.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.271+0000 7fe2f626c640 1 -- 172.21.15.40:0/1988044124 shutdown_connections 2024-03-18T22:41:12.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.271+0000 7fe2f626c640 1 --2- 172.21.15.40:0/1988044124 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe2d80a2890 0x7fe2d80a4d50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.271+0000 7fe2f626c640 1 --2- 172.21.15.40:0/1988044124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe2f0107580 0x7fe2f01a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.271+0000 7fe2f626c640 1 --2- 172.21.15.40:0/1988044124 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe2f0104c50 0x7fe2f019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.271+0000 7fe2f626c640 1 --2- 172.21.15.40:0/1988044124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe2f0102320 0x7fe2f0199ca0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.271+0000 7fe2f626c640 1 -- 172.21.15.40:0/1988044124 >> 172.21.15.40:0/1988044124 conn(0x7fe2f00fbcb0 msgr2=0x7fe2f0101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:12.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.271+0000 7fe2f626c640 1 -- 172.21.15.40:0/1988044124 shutdown_connections 2024-03-18T22:41:12.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.271+0000 7fe2f626c640 1 -- 172.21.15.40:0/1988044124 wait complete. 2024-03-18T22:41:12.282 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-03-18T22:41:10.809019+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598581,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":313,"num_write_kb":2982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":253,"ondisk_log_size":253,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":379316,"kb_used_data":13520,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562508,"statfs":{"total":1151940427776,"available":1151552008192,"internally_reserved":0,"allocated":13844480,"data_stored":7526600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3713,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":85,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003927"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203760+0000","last_change":"2024-03-18T22:40:41.806379+0000","last_active":"2024-03-18T22:40:47.203760+0000","last_peered":"2024-03-18T22:40:47.203760+0000","last_clean":"2024-03-18T22:40:47.203760+0000","last_became_active":"2024-03-18T22:40:41.806058+0000","last_became_peered":"2024-03-18T22:40:41.806058+0000","last_unstale":"2024-03-18T22:40:47.203760+0000","last_undegraded":"2024-03-18T22:40:47.203760+0000","last_fullsized":"2024-03-18T22:40:47.203760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:45.834396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843169+0000","last_change":"2024-03-18T22:40:41.813722+0000","last_active":"2024-03-18T22:40:46.843169+0000","last_peered":"2024-03-18T22:40:46.843169+0000","last_clean":"2024-03-18T22:40:46.843169+0000","last_became_active":"2024-03-18T22:40:41.813284+0000","last_became_peered":"2024-03-18T22:40:41.813284+0000","last_unstale":"2024-03-18T22:40:46.843169+0000","last_undegraded":"2024-03-18T22:40:46.843169+0000","last_fullsized":"2024-03-18T22:40:46.843169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:46.374117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.842444+0000","last_change":"2024-03-18T22:40:41.808340+0000","last_active":"2024-03-18T22:40:46.842444+0000","last_peered":"2024-03-18T22:40:46.842444+0000","last_clean":"2024-03-18T22:40:46.842444+0000","last_became_active":"2024-03-18T22:40:41.808002+0000","last_became_peered":"2024-03-18T22:40:41.808002+0000","last_unstale":"2024-03-18T22:40:46.842444+0000","last_undegraded":"2024-03-18T22:40:46.842444+0000","last_fullsized":"2024-03-18T22:40:46.842444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:40:32.122124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202258+0000","last_change":"2024-03-18T22:40:41.809959+0000","last_active":"2024-03-18T22:40:47.202258+0000","last_peered":"2024-03-18T22:40:47.202258+0000","last_clean":"2024-03-18T22:40:47.202258+0000","last_became_active":"2024-03-18T22:40:41.809676+0000","last_became_peered":"2024-03-18T22:40:41.809676+0000","last_unstale":"2024-03-18T22:40:47.202258+0000","last_undegraded":"2024-03-18T22:40:47.202258+0000","last_fullsized":"2024-03-18T22:40:47.202258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:29:45.263011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202250+0000","last_change":"2024-03-18T22:40:41.804583+0000","last_active":"2024-03-18T22:40:47.202250+0000","last_peered":"2024-03-18T22:40:47.202250+0000","last_clean":"2024-03-18T22:40:47.202250+0000","last_became_active":"2024-03-18T22:40:41.804064+0000","last_became_peered":"2024-03-18T22:40:41.804064+0000","last_unstale":"2024-03-18T22:40:47.202250+0000","last_undegraded":"2024-03-18T22:40:47.202250+0000","last_fullsized":"2024-03-18T22:40:47.202250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:46:23.455106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202510+0000","last_change":"2024-03-18T22:40:41.811230+0000","last_active":"2024-03-18T22:40:47.202510+0000","last_peered":"2024-03-18T22:40:47.202510+0000","last_clean":"2024-03-18T22:40:47.202510+0000","last_became_active":"2024-03-18T22:40:41.810904+0000","last_became_peered":"2024-03-18T22:40:41.810904+0000","last_unstale":"2024-03-18T22:40:47.202510+0000","last_undegraded":"2024-03-18T22:40:47.202510+0000","last_fullsized":"2024-03-18T22:40:47.202510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:51.302617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843440+0000","last_change":"2024-03-18T22:40:41.814971+0000","last_active":"2024-03-18T22:40:46.843440+0000","last_peered":"2024-03-18T22:40:46.843440+0000","last_clean":"2024-03-18T22:40:46.843440+0000","last_became_active":"2024-03-18T22:40:41.814305+0000","last_became_peered":"2024-03-18T22:40:41.814305+0000","last_unstale":"2024-03-18T22:40:46.843440+0000","last_undegraded":"2024-03-18T22:40:46.843440+0000","last_fullsized":"2024-03-18T22:40:46.843440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:07.390157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201249+0000","last_change":"2024-03-18T22:40:41.815674+0000","last_active":"2024-03-18T22:40:47.201249+0000","last_peered":"2024-03-18T22:40:47.201249+0000","last_clean":"2024-03-18T22:40:47.201249+0000","last_became_active":"2024-03-18T22:40:41.813560+0000","last_became_peered":"2024-03-18T22:40:41.813560+0000","last_unstale":"2024-03-18T22:40:47.201249+0000","last_undegraded":"2024-03-18T22:40:47.201249+0000","last_fullsized":"2024-03-18T22:40:47.201249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:42:32.409338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203022+0000","last_change":"2024-03-18T22:40:41.817508+0000","last_active":"2024-03-18T22:40:47.203022+0000","last_peered":"2024-03-18T22:40:47.203022+0000","last_clean":"2024-03-18T22:40:47.203022+0000","last_became_active":"2024-03-18T22:40:41.817184+0000","last_became_peered":"2024-03-18T22:40:41.817184+0000","last_unstale":"2024-03-18T22:40:47.203022+0000","last_undegraded":"2024-03-18T22:40:47.203022+0000","last_fullsized":"2024-03-18T22:40:47.203022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:10.255238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202325+0000","last_change":"2024-03-18T22:40:41.806235+0000","last_active":"2024-03-18T22:40:47.202325+0000","last_peered":"2024-03-18T22:40:47.202325+0000","last_clean":"2024-03-18T22:40:47.202325+0000","last_became_active":"2024-03-18T22:40:41.805913+0000","last_became_peered":"2024-03-18T22:40:41.805913+0000","last_unstale":"2024-03-18T22:40:47.202325+0000","last_undegraded":"2024-03-18T22:40:47.202325+0000","last_fullsized":"2024-03-18T22:40:47.202325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:39:06.299989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843867+0000","last_change":"2024-03-18T22:40:41.814735+0000","last_active":"2024-03-18T22:40:46.843867+0000","last_peered":"2024-03-18T22:40:46.843867+0000","last_clean":"2024-03-18T22:40:46.843867+0000","last_became_active":"2024-03-18T22:40:41.814291+0000","last_became_peered":"2024-03-18T22:40:41.814291+0000","last_unstale":"2024-03-18T22:40:46.843867+0000","last_undegraded":"2024-03-18T22:40:46.843867+0000","last_fullsized":"2024-03-18T22:40:46.843867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:49.610674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202165+0000","last_change":"2024-03-18T22:40:41.815561+0000","last_active":"2024-03-18T22:40:47.202165+0000","last_peered":"2024-03-18T22:40:47.202165+0000","last_clean":"2024-03-18T22:40:47.202165+0000","last_became_active":"2024-03-18T22:40:41.815252+0000","last_became_peered":"2024-03-18T22:40:41.815252+0000","last_unstale":"2024-03-18T22:40:47.202165+0000","last_undegraded":"2024-03-18T22:40:47.202165+0000","last_fullsized":"2024-03-18T22:40:47.202165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:34.769168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844354+0000","last_change":"2024-03-18T22:40:41.803325+0000","last_active":"2024-03-18T22:40:46.844354+0000","last_peered":"2024-03-18T22:40:46.844354+0000","last_clean":"2024-03-18T22:40:46.844354+0000","last_became_active":"2024-03-18T22:40:41.803008+0000","last_became_peered":"2024-03-18T22:40:41.803008+0000","last_unstale":"2024-03-18T22:40:46.844354+0000","last_undegraded":"2024-03-18T22:40:46.844354+0000","last_fullsized":"2024-03-18T22:40:46.844354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:14:01.335548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.842231+0000","last_change":"2024-03-18T22:40:41.812541+0000","last_active":"2024-03-18T22:40:46.842231+0000","last_peered":"2024-03-18T22:40:46.842231+0000","last_clean":"2024-03-18T22:40:46.842231+0000","last_became_active":"2024-03-18T22:40:41.812153+0000","last_became_peered":"2024-03-18T22:40:41.812153+0000","last_unstale":"2024-03-18T22:40:46.842231+0000","last_undegraded":"2024-03-18T22:40:46.842231+0000","last_fullsized":"2024-03-18T22:40:46.842231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:04.927057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.842356+0000","last_change":"2024-03-18T22:40:41.815790+0000","last_active":"2024-03-18T22:40:46.842356+0000","last_peered":"2024-03-18T22:40:46.842356+0000","last_clean":"2024-03-18T22:40:46.842356+0000","last_became_active":"2024-03-18T22:40:41.814949+0000","last_became_peered":"2024-03-18T22:40:41.814949+0000","last_unstale":"2024-03-18T22:40:46.842356+0000","last_undegraded":"2024-03-18T22:40:46.842356+0000","last_fullsized":"2024-03-18T22:40:46.842356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:10:15.874285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201282+0000","last_change":"2024-03-18T22:40:41.809980+0000","last_active":"2024-03-18T22:40:47.201282+0000","last_peered":"2024-03-18T22:40:47.201282+0000","last_clean":"2024-03-18T22:40:47.201282+0000","last_became_active":"2024-03-18T22:40:41.809640+0000","last_became_peered":"2024-03-18T22:40:41.809640+0000","last_unstale":"2024-03-18T22:40:47.201282+0000","last_undegraded":"2024-03-18T22:40:47.201282+0000","last_fullsized":"2024-03-18T22:40:47.201282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:25.389027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203649+0000","last_change":"2024-03-18T22:40:41.807088+0000","last_active":"2024-03-18T22:40:47.203649+0000","last_peered":"2024-03-18T22:40:47.203649+0000","last_clean":"2024-03-18T22:40:47.203649+0000","last_became_active":"2024-03-18T22:40:41.806770+0000","last_became_peered":"2024-03-18T22:40:41.806770+0000","last_unstale":"2024-03-18T22:40:47.203649+0000","last_undegraded":"2024-03-18T22:40:47.203649+0000","last_fullsized":"2024-03-18T22:40:47.203649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:53:14.001247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202008+0000","last_change":"2024-03-18T22:40:41.812685+0000","last_active":"2024-03-18T22:40:47.202008+0000","last_peered":"2024-03-18T22:40:47.202008+0000","last_clean":"2024-03-18T22:40:47.202008+0000","last_became_active":"2024-03-18T22:40:41.812346+0000","last_became_peered":"2024-03-18T22:40:41.812346+0000","last_unstale":"2024-03-18T22:40:47.202008+0000","last_undegraded":"2024-03-18T22:40:47.202008+0000","last_fullsized":"2024-03-18T22:40:47.202008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:25.033460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.204135+0000","last_change":"2024-03-18T22:40:41.812627+0000","last_active":"2024-03-18T22:40:47.204135+0000","last_peered":"2024-03-18T22:40:47.204135+0000","last_clean":"2024-03-18T22:40:47.204135+0000","last_became_active":"2024-03-18T22:40:41.812327+0000","last_became_peered":"2024-03-18T22:40:41.812327+0000","last_unstale":"2024-03-18T22:40:47.204135+0000","last_undegraded":"2024-03-18T22:40:47.204135+0000","last_fullsized":"2024-03-18T22:40:47.204135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:58:22.088589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202449+0000","last_change":"2024-03-18T22:40:45.847822+0000","last_active":"2024-03-18T22:40:47.202449+0000","last_peered":"2024-03-18T22:40:47.202449+0000","last_clean":"2024-03-18T22:40:47.202449+0000","last_became_active":"2024-03-18T22:40:45.847407+0000","last_became_peered":"2024-03-18T22:40:45.847407+0000","last_unstale":"2024-03-18T22:40:47.202449+0000","last_undegraded":"2024-03-18T22:40:47.202449+0000","last_fullsized":"2024-03-18T22:40:47.202449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:44.825997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'11","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.832346+0000","last_change":"2024-03-18T22:40:41.218797+0000","last_active":"2024-03-18T22:41:07.832346+0000","last_peered":"2024-03-18T22:41:07.832346+0000","last_clean":"2024-03-18T22:41:07.832346+0000","last_became_active":"2024-03-18T22:40:41.218472+0000","last_became_peered":"2024-03-18T22:40:41.218472+0000","last_unstale":"2024-03-18T22:41:07.832346+0000","last_undegraded":"2024-03-18T22:41:07.832346+0000","last_fullsized":"2024-03-18T22:41:07.832346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:10.230257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201431+0000","last_change":"2024-03-18T22:40:41.810692+0000","last_active":"2024-03-18T22:40:47.201431+0000","last_peered":"2024-03-18T22:40:47.201431+0000","last_clean":"2024-03-18T22:40:47.201431+0000","last_became_active":"2024-03-18T22:40:41.810390+0000","last_became_peered":"2024-03-18T22:40:41.810390+0000","last_unstale":"2024-03-18T22:40:47.201431+0000","last_undegraded":"2024-03-18T22:40:47.201431+0000","last_fullsized":"2024-03-18T22:40:47.201431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:57:05.649622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201468+0000","last_change":"2024-03-18T22:40:45.851126+0000","last_active":"2024-03-18T22:40:47.201468+0000","last_peered":"2024-03-18T22:40:47.201468+0000","last_clean":"2024-03-18T22:40:47.201468+0000","last_became_active":"2024-03-18T22:40:45.850549+0000","last_became_peered":"2024-03-18T22:40:45.850549+0000","last_unstale":"2024-03-18T22:40:47.201468+0000","last_undegraded":"2024-03-18T22:40:47.201468+0000","last_fullsized":"2024-03-18T22:40:47.201468+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:39.359263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201615+0000","last_change":"2024-03-18T22:40:41.220944+0000","last_active":"2024-03-18T22:40:47.201615+0000","last_peered":"2024-03-18T22:40:47.201615+0000","last_clean":"2024-03-18T22:40:47.201615+0000","last_became_active":"2024-03-18T22:40:41.220612+0000","last_became_peered":"2024-03-18T22:40:41.220612+0000","last_unstale":"2024-03-18T22:40:47.201615+0000","last_undegraded":"2024-03-18T22:40:47.201615+0000","last_fullsized":"2024-03-18T22:40:47.201615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:16.083134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.842940+0000","last_change":"2024-03-18T22:40:41.818031+0000","last_active":"2024-03-18T22:40:46.842940+0000","last_peered":"2024-03-18T22:40:46.842940+0000","last_clean":"2024-03-18T22:40:46.842940+0000","last_became_active":"2024-03-18T22:40:41.817693+0000","last_became_peered":"2024-03-18T22:40:41.817693+0000","last_unstale":"2024-03-18T22:40:46.842940+0000","last_undegraded":"2024-03-18T22:40:46.842940+0000","last_fullsized":"2024-03-18T22:40:46.842940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:05.901203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201799+0000","last_change":"2024-03-18T22:40:45.850749+0000","last_active":"2024-03-18T22:40:47.201799+0000","last_peered":"2024-03-18T22:40:47.201799+0000","last_clean":"2024-03-18T22:40:47.201799+0000","last_became_active":"2024-03-18T22:40:45.850318+0000","last_became_peered":"2024-03-18T22:40:45.850318+0000","last_unstale":"2024-03-18T22:40:47.201799+0000","last_undegraded":"2024-03-18T22:40:47.201799+0000","last_fullsized":"2024-03-18T22:40:47.201799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:56.047413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202763+0000","last_change":"2024-03-18T22:40:41.220775+0000","last_active":"2024-03-18T22:40:47.202763+0000","last_peered":"2024-03-18T22:40:47.202763+0000","last_clean":"2024-03-18T22:40:47.202763+0000","last_became_active":"2024-03-18T22:40:41.220313+0000","last_became_peered":"2024-03-18T22:40:41.220313+0000","last_unstale":"2024-03-18T22:40:47.202763+0000","last_undegraded":"2024-03-18T22:40:47.202763+0000","last_fullsized":"2024-03-18T22:40:47.202763+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:24.180884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202737+0000","last_change":"2024-03-18T22:40:41.818448+0000","last_active":"2024-03-18T22:40:47.202737+0000","last_peered":"2024-03-18T22:40:47.202737+0000","last_clean":"2024-03-18T22:40:47.202737+0000","last_became_active":"2024-03-18T22:40:41.818088+0000","last_became_peered":"2024-03-18T22:40:41.818088+0000","last_unstale":"2024-03-18T22:40:47.202737+0000","last_undegraded":"2024-03-18T22:40:47.202737+0000","last_fullsized":"2024-03-18T22:40:47.202737+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:34:54.298226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201508+0000","last_change":"2024-03-18T22:40:45.846250+0000","last_active":"2024-03-18T22:40:47.201508+0000","last_peered":"2024-03-18T22:40:47.201508+0000","last_clean":"2024-03-18T22:40:47.201508+0000","last_became_active":"2024-03-18T22:40:45.845859+0000","last_became_peered":"2024-03-18T22:40:45.845859+0000","last_unstale":"2024-03-18T22:40:47.201508+0000","last_undegraded":"2024-03-18T22:40:47.201508+0000","last_fullsized":"2024-03-18T22:40:47.201508+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:57.855301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202381+0000","last_change":"2024-03-18T22:40:41.216569+0000","last_active":"2024-03-18T22:40:47.202381+0000","last_peered":"2024-03-18T22:40:47.202381+0000","last_clean":"2024-03-18T22:40:47.202381+0000","last_became_active":"2024-03-18T22:40:41.216255+0000","last_became_peered":"2024-03-18T22:40:41.216255+0000","last_unstale":"2024-03-18T22:40:47.202381+0000","last_undegraded":"2024-03-18T22:40:47.202381+0000","last_fullsized":"2024-03-18T22:40:47.202381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:23.016346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203900+0000","last_change":"2024-03-18T22:40:41.818645+0000","last_active":"2024-03-18T22:40:47.203900+0000","last_peered":"2024-03-18T22:40:47.203900+0000","last_clean":"2024-03-18T22:40:47.203900+0000","last_became_active":"2024-03-18T22:40:41.818322+0000","last_became_peered":"2024-03-18T22:40:41.818322+0000","last_unstale":"2024-03-18T22:40:47.203900+0000","last_undegraded":"2024-03-18T22:40:47.203900+0000","last_fullsized":"2024-03-18T22:40:47.203900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:46.288533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203848+0000","last_change":"2024-03-18T22:40:45.839244+0000","last_active":"2024-03-18T22:40:47.203848+0000","last_peered":"2024-03-18T22:40:47.203848+0000","last_clean":"2024-03-18T22:40:47.203848+0000","last_became_active":"2024-03-18T22:40:45.838849+0000","last_became_peered":"2024-03-18T22:40:45.838849+0000","last_unstale":"2024-03-18T22:40:47.203848+0000","last_undegraded":"2024-03-18T22:40:47.203848+0000","last_fullsized":"2024-03-18T22:40:47.203848+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:28:55.327999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:08.827091+0000","last_change":"2024-03-18T22:40:41.210196+0000","last_active":"2024-03-18T22:41:08.827091+0000","last_peered":"2024-03-18T22:41:08.827091+0000","last_clean":"2024-03-18T22:41:08.827091+0000","last_became_active":"2024-03-18T22:40:41.209888+0000","last_became_peered":"2024-03-18T22:40:41.209888+0000","last_unstale":"2024-03-18T22:41:08.827091+0000","last_undegraded":"2024-03-18T22:41:08.827091+0000","last_fullsized":"2024-03-18T22:41:08.827091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:06:31.134688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202849+0000","last_change":"2024-03-18T22:40:41.806646+0000","last_active":"2024-03-18T22:40:47.202849+0000","last_peered":"2024-03-18T22:40:47.202849+0000","last_clean":"2024-03-18T22:40:47.202849+0000","last_became_active":"2024-03-18T22:40:41.806106+0000","last_became_peered":"2024-03-18T22:40:41.806106+0000","last_unstale":"2024-03-18T22:40:47.202849+0000","last_undegraded":"2024-03-18T22:40:47.202849+0000","last_fullsized":"2024-03-18T22:40:47.202849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:11:06.369619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202154+0000","last_change":"2024-03-18T22:40:45.852982+0000","last_active":"2024-03-18T22:40:47.202154+0000","last_peered":"2024-03-18T22:40:47.202154+0000","last_clean":"2024-03-18T22:40:47.202154+0000","last_became_active":"2024-03-18T22:40:45.852432+0000","last_became_peered":"2024-03-18T22:40:45.852432+0000","last_unstale":"2024-03-18T22:40:47.202154+0000","last_undegraded":"2024-03-18T22:40:47.202154+0000","last_fullsized":"2024-03-18T22:40:47.202154+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:06:03.475875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.832107+0000","last_change":"2024-03-18T22:40:41.217449+0000","last_active":"2024-03-18T22:41:07.832107+0000","last_peered":"2024-03-18T22:41:07.832107+0000","last_clean":"2024-03-18T22:41:07.832107+0000","last_became_active":"2024-03-18T22:40:41.217127+0000","last_became_peered":"2024-03-18T22:40:41.217127+0000","last_unstale":"2024-03-18T22:41:07.832107+0000","last_undegraded":"2024-03-18T22:41:07.832107+0000","last_fullsized":"2024-03-18T22:41:07.832107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:08:00.178720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203995+0000","last_change":"2024-03-18T22:40:41.814154+0000","last_active":"2024-03-18T22:40:47.203995+0000","last_peered":"2024-03-18T22:40:47.203995+0000","last_clean":"2024-03-18T22:40:47.203995+0000","last_became_active":"2024-03-18T22:40:41.813824+0000","last_became_peered":"2024-03-18T22:40:41.813824+0000","last_unstale":"2024-03-18T22:40:47.203995+0000","last_undegraded":"2024-03-18T22:40:47.203995+0000","last_fullsized":"2024-03-18T22:40:47.203995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:12.412689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203586+0000","last_change":"2024-03-18T22:40:45.852711+0000","last_active":"2024-03-18T22:40:47.203586+0000","last_peered":"2024-03-18T22:40:47.203586+0000","last_clean":"2024-03-18T22:40:47.203586+0000","last_became_active":"2024-03-18T22:40:45.852247+0000","last_became_peered":"2024-03-18T22:40:45.852247+0000","last_unstale":"2024-03-18T22:40:47.203586+0000","last_undegraded":"2024-03-18T22:40:47.203586+0000","last_fullsized":"2024-03-18T22:40:47.203586+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:03.061689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843844+0000","last_change":"2024-03-18T22:40:41.212399+0000","last_active":"2024-03-18T22:40:46.843844+0000","last_peered":"2024-03-18T22:40:46.843844+0000","last_clean":"2024-03-18T22:40:46.843844+0000","last_became_active":"2024-03-18T22:40:41.207416+0000","last_became_peered":"2024-03-18T22:40:41.207416+0000","last_unstale":"2024-03-18T22:40:46.843844+0000","last_undegraded":"2024-03-18T22:40:46.843844+0000","last_fullsized":"2024-03-18T22:40:46.843844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:11.406342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201357+0000","last_change":"2024-03-18T22:40:41.807313+0000","last_active":"2024-03-18T22:40:47.201357+0000","last_peered":"2024-03-18T22:40:47.201357+0000","last_clean":"2024-03-18T22:40:47.201357+0000","last_became_active":"2024-03-18T22:40:41.806029+0000","last_became_peered":"2024-03-18T22:40:41.806029+0000","last_unstale":"2024-03-18T22:40:47.201357+0000","last_undegraded":"2024-03-18T22:40:47.201357+0000","last_fullsized":"2024-03-18T22:40:47.201357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:23:43.973443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.842932+0000","last_change":"2024-03-18T22:40:45.835479+0000","last_active":"2024-03-18T22:40:46.842932+0000","last_peered":"2024-03-18T22:40:46.842932+0000","last_clean":"2024-03-18T22:40:46.842932+0000","last_became_active":"2024-03-18T22:40:45.834994+0000","last_became_peered":"2024-03-18T22:40:45.834994+0000","last_unstale":"2024-03-18T22:40:46.842932+0000","last_undegraded":"2024-03-18T22:40:46.842932+0000","last_fullsized":"2024-03-18T22:40:46.842932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:31.317843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:08.840903+0000","last_change":"2024-03-18T22:40:41.212110+0000","last_active":"2024-03-18T22:41:08.840903+0000","last_peered":"2024-03-18T22:41:08.840903+0000","last_clean":"2024-03-18T22:41:08.840903+0000","last_became_active":"2024-03-18T22:40:41.211803+0000","last_became_peered":"2024-03-18T22:40:41.211803+0000","last_unstale":"2024-03-18T22:41:08.840903+0000","last_undegraded":"2024-03-18T22:41:08.840903+0000","last_fullsized":"2024-03-18T22:41:08.840903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:40.482983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201396+0000","last_change":"2024-03-18T22:40:41.812585+0000","last_active":"2024-03-18T22:40:47.201396+0000","last_peered":"2024-03-18T22:40:47.201396+0000","last_clean":"2024-03-18T22:40:47.201396+0000","last_became_active":"2024-03-18T22:40:41.812260+0000","last_became_peered":"2024-03-18T22:40:41.812260+0000","last_unstale":"2024-03-18T22:40:47.201396+0000","last_undegraded":"2024-03-18T22:40:47.201396+0000","last_fullsized":"2024-03-18T22:40:47.201396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:12:59.999503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202071+0000","last_change":"2024-03-18T22:40:45.842252+0000","last_active":"2024-03-18T22:40:47.202071+0000","last_peered":"2024-03-18T22:40:47.202071+0000","last_clean":"2024-03-18T22:40:47.202071+0000","last_became_active":"2024-03-18T22:40:45.841849+0000","last_became_peered":"2024-03-18T22:40:45.841849+0000","last_unstale":"2024-03-18T22:40:47.202071+0000","last_undegraded":"2024-03-18T22:40:47.202071+0000","last_fullsized":"2024-03-18T22:40:47.202071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:21.253947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.833971+0000","last_change":"2024-03-18T22:40:41.214253+0000","last_active":"2024-03-18T22:41:07.833971+0000","last_peered":"2024-03-18T22:41:07.833971+0000","last_clean":"2024-03-18T22:41:07.833971+0000","last_became_active":"2024-03-18T22:40:41.213911+0000","last_became_peered":"2024-03-18T22:40:41.213911+0000","last_unstale":"2024-03-18T22:41:07.833971+0000","last_undegraded":"2024-03-18T22:41:07.833971+0000","last_fullsized":"2024-03-18T22:41:07.833971+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:19.185714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202726+0000","last_change":"2024-03-18T22:40:41.813281+0000","last_active":"2024-03-18T22:40:47.202726+0000","last_peered":"2024-03-18T22:40:47.202726+0000","last_clean":"2024-03-18T22:40:47.202726+0000","last_became_active":"2024-03-18T22:40:41.812909+0000","last_became_peered":"2024-03-18T22:40:41.812909+0000","last_unstale":"2024-03-18T22:40:47.202726+0000","last_undegraded":"2024-03-18T22:40:47.202726+0000","last_fullsized":"2024-03-18T22:40:47.202726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:17:26.619441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201925+0000","last_change":"2024-03-18T22:40:45.846956+0000","last_active":"2024-03-18T22:40:47.201925+0000","last_peered":"2024-03-18T22:40:47.201925+0000","last_clean":"2024-03-18T22:40:47.201925+0000","last_became_active":"2024-03-18T22:40:45.843061+0000","last_became_peered":"2024-03-18T22:40:45.843061+0000","last_unstale":"2024-03-18T22:40:47.201925+0000","last_undegraded":"2024-03-18T22:40:47.201925+0000","last_fullsized":"2024-03-18T22:40:47.201925+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:52.043424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.040678+0000","last_change":"2024-03-18T22:40:41.214933+0000","last_active":"2024-03-18T22:41:06.040678+0000","last_peered":"2024-03-18T22:41:06.040678+0000","last_clean":"2024-03-18T22:41:06.040678+0000","last_became_active":"2024-03-18T22:40:41.214606+0000","last_became_peered":"2024-03-18T22:40:41.214606+0000","last_unstale":"2024-03-18T22:41:06.040678+0000","last_undegraded":"2024-03-18T22:41:06.040678+0000","last_fullsized":"2024-03-18T22:41:06.040678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:08:06.230288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843060+0000","last_change":"2024-03-18T22:40:41.817590+0000","last_active":"2024-03-18T22:40:46.843060+0000","last_peered":"2024-03-18T22:40:46.843060+0000","last_clean":"2024-03-18T22:40:46.843060+0000","last_became_active":"2024-03-18T22:40:41.815809+0000","last_became_peered":"2024-03-18T22:40:41.815809+0000","last_unstale":"2024-03-18T22:40:46.843060+0000","last_undegraded":"2024-03-18T22:40:46.843060+0000","last_fullsized":"2024-03-18T22:40:46.843060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:44:54.043647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202049+0000","last_change":"2024-03-18T22:40:45.844767+0000","last_active":"2024-03-18T22:40:47.202049+0000","last_peered":"2024-03-18T22:40:47.202049+0000","last_clean":"2024-03-18T22:40:47.202049+0000","last_became_active":"2024-03-18T22:40:45.844251+0000","last_became_peered":"2024-03-18T22:40:45.844251+0000","last_unstale":"2024-03-18T22:40:47.202049+0000","last_undegraded":"2024-03-18T22:40:47.202049+0000","last_fullsized":"2024-03-18T22:40:47.202049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:43.756959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.824859+0000","last_change":"2024-03-18T22:40:41.214310+0000","last_active":"2024-03-18T22:41:07.824859+0000","last_peered":"2024-03-18T22:41:07.824859+0000","last_clean":"2024-03-18T22:41:07.824859+0000","last_became_active":"2024-03-18T22:40:41.213963+0000","last_became_peered":"2024-03-18T22:40:41.213963+0000","last_unstale":"2024-03-18T22:41:07.824859+0000","last_undegraded":"2024-03-18T22:41:07.824859+0000","last_fullsized":"2024-03-18T22:41:07.824859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:03.401220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202884+0000","last_change":"2024-03-18T22:40:41.810876+0000","last_active":"2024-03-18T22:40:47.202884+0000","last_peered":"2024-03-18T22:40:47.202884+0000","last_clean":"2024-03-18T22:40:47.202884+0000","last_became_active":"2024-03-18T22:40:41.810104+0000","last_became_peered":"2024-03-18T22:40:41.810104+0000","last_unstale":"2024-03-18T22:40:47.202884+0000","last_undegraded":"2024-03-18T22:40:47.202884+0000","last_fullsized":"2024-03-18T22:40:47.202884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:41.294286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201637+0000","last_change":"2024-03-18T22:40:45.841830+0000","last_active":"2024-03-18T22:40:47.201637+0000","last_peered":"2024-03-18T22:40:47.201637+0000","last_clean":"2024-03-18T22:40:47.201637+0000","last_became_active":"2024-03-18T22:40:45.841426+0000","last_became_peered":"2024-03-18T22:40:45.841426+0000","last_unstale":"2024-03-18T22:40:47.201637+0000","last_undegraded":"2024-03-18T22:40:47.201637+0000","last_fullsized":"2024-03-18T22:40:47.201637+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:20:13.227650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'15","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:08.839243+0000","last_change":"2024-03-18T22:40:41.209837+0000","last_active":"2024-03-18T22:41:08.839243+0000","last_peered":"2024-03-18T22:41:08.839243+0000","last_clean":"2024-03-18T22:41:08.839243+0000","last_became_active":"2024-03-18T22:40:41.209465+0000","last_became_peered":"2024-03-18T22:40:41.209465+0000","last_unstale":"2024-03-18T22:41:08.839243+0000","last_undegraded":"2024-03-18T22:41:08.839243+0000","last_fullsized":"2024-03-18T22:41:08.839243+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:45:15.765475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4234,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":7,"num_write":17,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202532+0000","last_change":"2024-03-18T22:40:41.810058+0000","last_active":"2024-03-18T22:40:47.202532+0000","last_peered":"2024-03-18T22:40:47.202532+0000","last_clean":"2024-03-18T22:40:47.202532+0000","last_became_active":"2024-03-18T22:40:41.809677+0000","last_became_peered":"2024-03-18T22:40:41.809677+0000","last_unstale":"2024-03-18T22:40:47.202532+0000","last_undegraded":"2024-03-18T22:40:47.202532+0000","last_fullsized":"2024-03-18T22:40:47.202532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:03.935036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.204326+0000","last_change":"2024-03-18T22:40:45.845638+0000","last_active":"2024-03-18T22:40:47.204326+0000","last_peered":"2024-03-18T22:40:47.204326+0000","last_clean":"2024-03-18T22:40:47.204326+0000","last_became_active":"2024-03-18T22:40:45.844994+0000","last_became_peered":"2024-03-18T22:40:45.844994+0000","last_unstale":"2024-03-18T22:40:47.204326+0000","last_undegraded":"2024-03-18T22:40:47.204326+0000","last_fullsized":"2024-03-18T22:40:47.204326+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:21.551271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.038829+0000","last_change":"2024-03-18T22:40:41.218397+0000","last_active":"2024-03-18T22:41:06.038829+0000","last_peered":"2024-03-18T22:41:06.038829+0000","last_clean":"2024-03-18T22:41:06.038829+0000","last_became_active":"2024-03-18T22:40:41.217587+0000","last_became_peered":"2024-03-18T22:40:41.217587+0000","last_unstale":"2024-03-18T22:41:06.038829+0000","last_undegraded":"2024-03-18T22:41:06.038829+0000","last_fullsized":"2024-03-18T22:41:06.038829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:36.524954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843987+0000","last_change":"2024-03-18T22:40:41.812026+0000","last_active":"2024-03-18T22:40:46.843987+0000","last_peered":"2024-03-18T22:40:46.843987+0000","last_clean":"2024-03-18T22:40:46.843987+0000","last_became_active":"2024-03-18T22:40:41.811668+0000","last_became_peered":"2024-03-18T22:40:41.811668+0000","last_unstale":"2024-03-18T22:40:46.843987+0000","last_undegraded":"2024-03-18T22:40:46.843987+0000","last_fullsized":"2024-03-18T22:40:46.843987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:17:54.775327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202967+0000","last_change":"2024-03-18T22:40:45.848466+0000","last_active":"2024-03-18T22:40:47.202967+0000","last_peered":"2024-03-18T22:40:47.202967+0000","last_clean":"2024-03-18T22:40:47.202967+0000","last_became_active":"2024-03-18T22:40:45.848078+0000","last_became_peered":"2024-03-18T22:40:45.848078+0000","last_unstale":"2024-03-18T22:40:47.202967+0000","last_undegraded":"2024-03-18T22:40:47.202967+0000","last_fullsized":"2024-03-18T22:40:47.202967+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:28.515368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:08.841241+0000","last_change":"2024-03-18T22:40:41.216434+0000","last_active":"2024-03-18T22:41:08.841241+0000","last_peered":"2024-03-18T22:41:08.841241+0000","last_clean":"2024-03-18T22:41:08.841241+0000","last_became_active":"2024-03-18T22:40:41.216103+0000","last_became_peered":"2024-03-18T22:40:41.216103+0000","last_unstale":"2024-03-18T22:41:08.841241+0000","last_undegraded":"2024-03-18T22:41:08.841241+0000","last_fullsized":"2024-03-18T22:41:08.841241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:39.516463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843591+0000","last_change":"2024-03-18T22:40:41.808405+0000","last_active":"2024-03-18T22:40:46.843591+0000","last_peered":"2024-03-18T22:40:46.843591+0000","last_clean":"2024-03-18T22:40:46.843591+0000","last_became_active":"2024-03-18T22:40:41.808111+0000","last_became_peered":"2024-03-18T22:40:41.808111+0000","last_unstale":"2024-03-18T22:40:46.843591+0000","last_undegraded":"2024-03-18T22:40:46.843591+0000","last_fullsized":"2024-03-18T22:40:46.843591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:37:20.808163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202091+0000","last_change":"2024-03-18T22:40:45.848409+0000","last_active":"2024-03-18T22:40:47.202091+0000","last_peered":"2024-03-18T22:40:47.202091+0000","last_clean":"2024-03-18T22:40:47.202091+0000","last_became_active":"2024-03-18T22:40:45.847687+0000","last_became_peered":"2024-03-18T22:40:45.847687+0000","last_unstale":"2024-03-18T22:40:47.202091+0000","last_undegraded":"2024-03-18T22:40:47.202091+0000","last_fullsized":"2024-03-18T22:40:47.202091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:32.894885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.046173+0000","last_change":"2024-03-18T22:40:41.208408+0000","last_active":"2024-03-18T22:41:07.046173+0000","last_peered":"2024-03-18T22:41:07.046173+0000","last_clean":"2024-03-18T22:41:07.046173+0000","last_became_active":"2024-03-18T22:40:41.207976+0000","last_became_peered":"2024-03-18T22:40:41.207976+0000","last_unstale":"2024-03-18T22:41:07.046173+0000","last_undegraded":"2024-03-18T22:41:07.046173+0000","last_fullsized":"2024-03-18T22:41:07.046173+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:43:24.416974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843529+0000","last_change":"2024-03-18T22:40:41.813547+0000","last_active":"2024-03-18T22:40:46.843529+0000","last_peered":"2024-03-18T22:40:46.843529+0000","last_clean":"2024-03-18T22:40:46.843529+0000","last_became_active":"2024-03-18T22:40:41.813250+0000","last_became_peered":"2024-03-18T22:40:41.813250+0000","last_unstale":"2024-03-18T22:40:46.843529+0000","last_undegraded":"2024-03-18T22:40:46.843529+0000","last_fullsized":"2024-03-18T22:40:46.843529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:54.428485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844337+0000","last_change":"2024-03-18T22:40:45.845797+0000","last_active":"2024-03-18T22:40:46.844337+0000","last_peered":"2024-03-18T22:40:46.844337+0000","last_clean":"2024-03-18T22:40:46.844337+0000","last_became_active":"2024-03-18T22:40:45.845324+0000","last_became_peered":"2024-03-18T22:40:45.845324+0000","last_unstale":"2024-03-18T22:40:46.844337+0000","last_undegraded":"2024-03-18T22:40:46.844337+0000","last_fullsized":"2024-03-18T22:40:46.844337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:21.613203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'10","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.041222+0000","last_change":"2024-03-18T22:40:41.222127+0000","last_active":"2024-03-18T22:41:06.041222+0000","last_peered":"2024-03-18T22:41:06.041222+0000","last_clean":"2024-03-18T22:41:06.041222+0000","last_became_active":"2024-03-18T22:40:41.221830+0000","last_became_peered":"2024-03-18T22:40:41.221830+0000","last_unstale":"2024-03-18T22:41:06.041222+0000","last_undegraded":"2024-03-18T22:41:06.041222+0000","last_fullsized":"2024-03-18T22:41:06.041222+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:24:02.235914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203971+0000","last_change":"2024-03-18T22:40:41.814606+0000","last_active":"2024-03-18T22:40:47.203971+0000","last_peered":"2024-03-18T22:40:47.203971+0000","last_clean":"2024-03-18T22:40:47.203971+0000","last_became_active":"2024-03-18T22:40:41.814309+0000","last_became_peered":"2024-03-18T22:40:41.814309+0000","last_unstale":"2024-03-18T22:40:47.203971+0000","last_undegraded":"2024-03-18T22:40:47.203971+0000","last_fullsized":"2024-03-18T22:40:47.203971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:33.086229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.204356+0000","last_change":"2024-03-18T22:40:45.845711+0000","last_active":"2024-03-18T22:40:47.204356+0000","last_peered":"2024-03-18T22:40:47.204356+0000","last_clean":"2024-03-18T22:40:47.204356+0000","last_became_active":"2024-03-18T22:40:45.845094+0000","last_became_peered":"2024-03-18T22:40:45.845094+0000","last_unstale":"2024-03-18T22:40:47.204356+0000","last_undegraded":"2024-03-18T22:40:47.204356+0000","last_fullsized":"2024-03-18T22:40:47.204356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:08.372875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202536+0000","last_change":"2024-03-18T22:40:41.219076+0000","last_active":"2024-03-18T22:40:47.202536+0000","last_peered":"2024-03-18T22:40:47.202536+0000","last_clean":"2024-03-18T22:40:47.202536+0000","last_became_active":"2024-03-18T22:40:41.217580+0000","last_became_peered":"2024-03-18T22:40:41.217580+0000","last_unstale":"2024-03-18T22:40:47.202536+0000","last_undegraded":"2024-03-18T22:40:47.202536+0000","last_fullsized":"2024-03-18T22:40:47.202536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:38.112325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203173+0000","last_change":"2024-03-18T22:40:41.810267+0000","last_active":"2024-03-18T22:40:47.203173+0000","last_peered":"2024-03-18T22:40:47.203173+0000","last_clean":"2024-03-18T22:40:47.203173+0000","last_became_active":"2024-03-18T22:40:41.809960+0000","last_became_peered":"2024-03-18T22:40:41.809960+0000","last_unstale":"2024-03-18T22:40:47.203173+0000","last_undegraded":"2024-03-18T22:40:47.203173+0000","last_fullsized":"2024-03-18T22:40:47.203173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:00.973817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202789+0000","last_change":"2024-03-18T22:40:45.848461+0000","last_active":"2024-03-18T22:40:47.202789+0000","last_peered":"2024-03-18T22:40:47.202789+0000","last_clean":"2024-03-18T22:40:47.202789+0000","last_became_active":"2024-03-18T22:40:45.848043+0000","last_became_peered":"2024-03-18T22:40:45.848043+0000","last_unstale":"2024-03-18T22:40:47.202789+0000","last_undegraded":"2024-03-18T22:40:47.202789+0000","last_fullsized":"2024-03-18T22:40:47.202789+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:38.865565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202161+0000","last_change":"2024-03-18T22:40:41.213069+0000","last_active":"2024-03-18T22:40:47.202161+0000","last_peered":"2024-03-18T22:40:47.202161+0000","last_clean":"2024-03-18T22:40:47.202161+0000","last_became_active":"2024-03-18T22:40:41.212725+0000","last_became_peered":"2024-03-18T22:40:41.212725+0000","last_unstale":"2024-03-18T22:40:47.202161+0000","last_undegraded":"2024-03-18T22:40:47.202161+0000","last_fullsized":"2024-03-18T22:40:47.202161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:50.219433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201736+0000","last_change":"2024-03-18T22:40:41.808343+0000","last_active":"2024-03-18T22:40:47.201736+0000","last_peered":"2024-03-18T22:40:47.201736+0000","last_clean":"2024-03-18T22:40:47.201736+0000","last_became_active":"2024-03-18T22:40:41.808046+0000","last_became_peered":"2024-03-18T22:40:41.808046+0000","last_unstale":"2024-03-18T22:40:47.201736+0000","last_undegraded":"2024-03-18T22:40:47.201736+0000","last_fullsized":"2024-03-18T22:40:47.201736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:09.357992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844268+0000","last_change":"2024-03-18T22:40:45.840058+0000","last_active":"2024-03-18T22:40:46.844268+0000","last_peered":"2024-03-18T22:40:46.844268+0000","last_clean":"2024-03-18T22:40:46.844268+0000","last_became_active":"2024-03-18T22:40:45.839661+0000","last_became_peered":"2024-03-18T22:40:45.839661+0000","last_unstale":"2024-03-18T22:40:46.844268+0000","last_undegraded":"2024-03-18T22:40:46.844268+0000","last_fullsized":"2024-03-18T22:40:46.844268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:35:14.323972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843724+0000","last_change":"2024-03-18T22:40:41.215988+0000","last_active":"2024-03-18T22:40:46.843724+0000","last_peered":"2024-03-18T22:40:46.843724+0000","last_clean":"2024-03-18T22:40:46.843724+0000","last_became_active":"2024-03-18T22:40:41.215680+0000","last_became_peered":"2024-03-18T22:40:41.215680+0000","last_unstale":"2024-03-18T22:40:46.843724+0000","last_undegraded":"2024-03-18T22:40:46.843724+0000","last_fullsized":"2024-03-18T22:40:46.843724+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:43.412362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201944+0000","last_change":"2024-03-18T22:40:41.812521+0000","last_active":"2024-03-18T22:40:47.201944+0000","last_peered":"2024-03-18T22:40:47.201944+0000","last_clean":"2024-03-18T22:40:47.201944+0000","last_became_active":"2024-03-18T22:40:41.812223+0000","last_became_peered":"2024-03-18T22:40:41.812223+0000","last_unstale":"2024-03-18T22:40:47.201944+0000","last_undegraded":"2024-03-18T22:40:47.201944+0000","last_fullsized":"2024-03-18T22:40:47.201944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:35:37.945144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844055+0000","last_change":"2024-03-18T22:40:45.838372+0000","last_active":"2024-03-18T22:40:46.844055+0000","last_peered":"2024-03-18T22:40:46.844055+0000","last_clean":"2024-03-18T22:40:46.844055+0000","last_became_active":"2024-03-18T22:40:45.837962+0000","last_became_peered":"2024-03-18T22:40:45.837962+0000","last_unstale":"2024-03-18T22:40:46.844055+0000","last_undegraded":"2024-03-18T22:40:46.844055+0000","last_fullsized":"2024-03-18T22:40:46.844055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:18:39.323023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203367+0000","last_change":"2024-03-18T22:40:41.812146+0000","last_active":"2024-03-18T22:40:47.203367+0000","last_peered":"2024-03-18T22:40:47.203367+0000","last_clean":"2024-03-18T22:40:47.203367+0000","last_became_active":"2024-03-18T22:40:41.811840+0000","last_became_peered":"2024-03-18T22:40:41.811840+0000","last_unstale":"2024-03-18T22:40:47.203367+0000","last_undegraded":"2024-03-18T22:40:47.203367+0000","last_fullsized":"2024-03-18T22:40:47.203367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:30.829187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843766+0000","last_change":"2024-03-18T22:40:41.212133+0000","last_active":"2024-03-18T22:40:46.843766+0000","last_peered":"2024-03-18T22:40:46.843766+0000","last_clean":"2024-03-18T22:40:46.843766+0000","last_became_active":"2024-03-18T22:40:41.211581+0000","last_became_peered":"2024-03-18T22:40:41.211581+0000","last_unstale":"2024-03-18T22:40:46.843766+0000","last_undegraded":"2024-03-18T22:40:46.843766+0000","last_fullsized":"2024-03-18T22:40:46.843766+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:44.287725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843870+0000","last_change":"2024-03-18T22:40:45.843660+0000","last_active":"2024-03-18T22:40:46.843870+0000","last_peered":"2024-03-18T22:40:46.843870+0000","last_clean":"2024-03-18T22:40:46.843870+0000","last_became_active":"2024-03-18T22:40:45.843233+0000","last_became_peered":"2024-03-18T22:40:45.843233+0000","last_unstale":"2024-03-18T22:40:46.843870+0000","last_undegraded":"2024-03-18T22:40:46.843870+0000","last_fullsized":"2024-03-18T22:40:46.843870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:08:19.758496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201521+0000","last_change":"2024-03-18T22:40:41.811798+0000","last_active":"2024-03-18T22:40:47.201521+0000","last_peered":"2024-03-18T22:40:47.201521+0000","last_clean":"2024-03-18T22:40:47.201521+0000","last_became_active":"2024-03-18T22:40:41.811422+0000","last_became_peered":"2024-03-18T22:40:41.811422+0000","last_unstale":"2024-03-18T22:40:47.201521+0000","last_undegraded":"2024-03-18T22:40:47.201521+0000","last_fullsized":"2024-03-18T22:40:47.201521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:53.944556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.039621+0000","last_change":"2024-03-18T22:40:41.210052+0000","last_active":"2024-03-18T22:41:06.039621+0000","last_peered":"2024-03-18T22:41:06.039621+0000","last_clean":"2024-03-18T22:41:06.039621+0000","last_became_active":"2024-03-18T22:40:41.209647+0000","last_became_peered":"2024-03-18T22:40:41.209647+0000","last_unstale":"2024-03-18T22:41:06.039621+0000","last_undegraded":"2024-03-18T22:41:06.039621+0000","last_fullsized":"2024-03-18T22:41:06.039621+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:57:35.187348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843338+0000","last_change":"2024-03-18T22:39:32.831758+0000","last_active":"2024-03-18T22:40:46.843338+0000","last_peered":"2024-03-18T22:40:46.843338+0000","last_clean":"2024-03-18T22:40:46.843338+0000","last_became_active":"2024-03-18T22:39:32.831287+0000","last_became_peered":"2024-03-18T22:39:32.831287+0000","last_unstale":"2024-03-18T22:40:46.843338+0000","last_undegraded":"2024-03-18T22:40:46.843338+0000","last_fullsized":"2024-03-18T22:40:46.843338+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:36:07.806854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:36:07.806854+0000","last_clean_scrub_stamp":"2024-03-18T22:36:07.806854+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:42.449107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.204239+0000","last_change":"2024-03-18T22:40:45.842008+0000","last_active":"2024-03-18T22:40:47.204239+0000","last_peered":"2024-03-18T22:40:47.204239+0000","last_clean":"2024-03-18T22:40:47.204239+0000","last_became_active":"2024-03-18T22:40:45.841231+0000","last_became_peered":"2024-03-18T22:40:45.841231+0000","last_unstale":"2024-03-18T22:40:47.204239+0000","last_undegraded":"2024-03-18T22:40:47.204239+0000","last_fullsized":"2024-03-18T22:40:47.204239+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:06.812049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843850+0000","last_change":"2024-03-18T22:40:41.807216+0000","last_active":"2024-03-18T22:40:46.843850+0000","last_peered":"2024-03-18T22:40:46.843850+0000","last_clean":"2024-03-18T22:40:46.843850+0000","last_became_active":"2024-03-18T22:40:41.806797+0000","last_became_peered":"2024-03-18T22:40:41.806797+0000","last_unstale":"2024-03-18T22:40:46.843850+0000","last_undegraded":"2024-03-18T22:40:46.843850+0000","last_fullsized":"2024-03-18T22:40:46.843850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:26.422329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.047645+0000","last_change":"2024-03-18T22:40:41.216555+0000","last_active":"2024-03-18T22:41:07.047645+0000","last_peered":"2024-03-18T22:41:07.047645+0000","last_clean":"2024-03-18T22:41:07.047645+0000","last_became_active":"2024-03-18T22:40:41.216240+0000","last_became_peered":"2024-03-18T22:40:41.216240+0000","last_unstale":"2024-03-18T22:41:07.047645+0000","last_undegraded":"2024-03-18T22:41:07.047645+0000","last_fullsized":"2024-03-18T22:41:07.047645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:04.655182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202780+0000","last_change":"2024-03-18T22:40:45.839889+0000","last_active":"2024-03-18T22:40:47.202780+0000","last_peered":"2024-03-18T22:40:47.202780+0000","last_clean":"2024-03-18T22:40:47.202780+0000","last_became_active":"2024-03-18T22:40:45.839414+0000","last_became_peered":"2024-03-18T22:40:45.839414+0000","last_unstale":"2024-03-18T22:40:47.202780+0000","last_undegraded":"2024-03-18T22:40:47.202780+0000","last_fullsized":"2024-03-18T22:40:47.202780+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:52.497378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202718+0000","last_change":"2024-03-18T22:40:41.221026+0000","last_active":"2024-03-18T22:40:47.202718+0000","last_peered":"2024-03-18T22:40:47.202718+0000","last_clean":"2024-03-18T22:40:47.202718+0000","last_became_active":"2024-03-18T22:40:41.220681+0000","last_became_peered":"2024-03-18T22:40:41.220681+0000","last_unstale":"2024-03-18T22:40:47.202718+0000","last_undegraded":"2024-03-18T22:40:47.202718+0000","last_fullsized":"2024-03-18T22:40:47.202718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:11.490445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844508+0000","last_change":"2024-03-18T22:40:41.810500+0000","last_active":"2024-03-18T22:40:46.844508+0000","last_peered":"2024-03-18T22:40:46.844508+0000","last_clean":"2024-03-18T22:40:46.844508+0000","last_became_active":"2024-03-18T22:40:41.809723+0000","last_became_peered":"2024-03-18T22:40:41.809723+0000","last_unstale":"2024-03-18T22:40:46.844508+0000","last_undegraded":"2024-03-18T22:40:46.844508+0000","last_fullsized":"2024-03-18T22:40:46.844508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:18:37.053809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202465+0000","last_change":"2024-03-18T22:40:45.844323+0000","last_active":"2024-03-18T22:40:47.202465+0000","last_peered":"2024-03-18T22:40:47.202465+0000","last_clean":"2024-03-18T22:40:47.202465+0000","last_became_active":"2024-03-18T22:40:45.843146+0000","last_became_peered":"2024-03-18T22:40:45.843146+0000","last_unstale":"2024-03-18T22:40:47.202465+0000","last_undegraded":"2024-03-18T22:40:47.202465+0000","last_fullsized":"2024-03-18T22:40:47.202465+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:14:59.881467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.833378+0000","last_change":"2024-03-18T22:40:41.212126+0000","last_active":"2024-03-18T22:41:07.833378+0000","last_peered":"2024-03-18T22:41:07.833378+0000","last_clean":"2024-03-18T22:41:07.833378+0000","last_became_active":"2024-03-18T22:40:41.211772+0000","last_became_peered":"2024-03-18T22:40:41.211772+0000","last_unstale":"2024-03-18T22:41:07.833378+0000","last_undegraded":"2024-03-18T22:41:07.833378+0000","last_fullsized":"2024-03-18T22:41:07.833378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:03.215751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202031+0000","last_change":"2024-03-18T22:40:41.820476+0000","last_active":"2024-03-18T22:40:47.202031+0000","last_peered":"2024-03-18T22:40:47.202031+0000","last_clean":"2024-03-18T22:40:47.202031+0000","last_became_active":"2024-03-18T22:40:41.820084+0000","last_became_peered":"2024-03-18T22:40:41.820084+0000","last_unstale":"2024-03-18T22:40:47.202031+0000","last_undegraded":"2024-03-18T22:40:47.202031+0000","last_fullsized":"2024-03-18T22:40:47.202031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:51:50.580208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201852+0000","last_change":"2024-03-18T22:40:45.838588+0000","last_active":"2024-03-18T22:40:47.201852+0000","last_peered":"2024-03-18T22:40:47.201852+0000","last_clean":"2024-03-18T22:40:47.201852+0000","last_became_active":"2024-03-18T22:40:45.838068+0000","last_became_peered":"2024-03-18T22:40:45.838068+0000","last_unstale":"2024-03-18T22:40:47.201852+0000","last_undegraded":"2024-03-18T22:40:47.201852+0000","last_fullsized":"2024-03-18T22:40:47.201852+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:11:02.162668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.041019+0000","last_change":"2024-03-18T22:40:41.210108+0000","last_active":"2024-03-18T22:41:06.041019+0000","last_peered":"2024-03-18T22:41:06.041019+0000","last_clean":"2024-03-18T22:41:06.041019+0000","last_became_active":"2024-03-18T22:40:41.209781+0000","last_became_peered":"2024-03-18T22:40:41.209781+0000","last_unstale":"2024-03-18T22:41:06.041019+0000","last_undegraded":"2024-03-18T22:41:06.041019+0000","last_fullsized":"2024-03-18T22:41:06.041019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:10.670822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843171+0000","last_change":"2024-03-18T22:40:41.814637+0000","last_active":"2024-03-18T22:40:46.843171+0000","last_peered":"2024-03-18T22:40:46.843171+0000","last_clean":"2024-03-18T22:40:46.843171+0000","last_became_active":"2024-03-18T22:40:41.814279+0000","last_became_peered":"2024-03-18T22:40:41.814279+0000","last_unstale":"2024-03-18T22:40:46.843171+0000","last_undegraded":"2024-03-18T22:40:46.843171+0000","last_fullsized":"2024-03-18T22:40:46.843171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:12.496230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201586+0000","last_change":"2024-03-18T22:40:45.851560+0000","last_active":"2024-03-18T22:40:47.201586+0000","last_peered":"2024-03-18T22:40:47.201586+0000","last_clean":"2024-03-18T22:40:47.201586+0000","last_became_active":"2024-03-18T22:40:45.850884+0000","last_became_peered":"2024-03-18T22:40:45.850884+0000","last_unstale":"2024-03-18T22:40:47.201586+0000","last_undegraded":"2024-03-18T22:40:47.201586+0000","last_fullsized":"2024-03-18T22:40:47.201586+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:04.114468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.039626+0000","last_change":"2024-03-18T22:40:41.210156+0000","last_active":"2024-03-18T22:41:06.039626+0000","last_peered":"2024-03-18T22:41:06.039626+0000","last_clean":"2024-03-18T22:41:06.039626+0000","last_became_active":"2024-03-18T22:40:41.209855+0000","last_became_peered":"2024-03-18T22:40:41.209855+0000","last_unstale":"2024-03-18T22:41:06.039626+0000","last_undegraded":"2024-03-18T22:41:06.039626+0000","last_fullsized":"2024-03-18T22:41:06.039626+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:02.752573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202142+0000","last_change":"2024-03-18T22:40:41.803681+0000","last_active":"2024-03-18T22:40:47.202142+0000","last_peered":"2024-03-18T22:40:47.202142+0000","last_clean":"2024-03-18T22:40:47.202142+0000","last_became_active":"2024-03-18T22:40:41.803244+0000","last_became_peered":"2024-03-18T22:40:41.803244+0000","last_unstale":"2024-03-18T22:40:47.202142+0000","last_undegraded":"2024-03-18T22:40:47.202142+0000","last_fullsized":"2024-03-18T22:40:47.202142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:55.944575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202264+0000","last_change":"2024-03-18T22:40:45.851040+0000","last_active":"2024-03-18T22:40:47.202264+0000","last_peered":"2024-03-18T22:40:47.202264+0000","last_clean":"2024-03-18T22:40:47.202264+0000","last_became_active":"2024-03-18T22:40:45.848300+0000","last_became_peered":"2024-03-18T22:40:45.848300+0000","last_unstale":"2024-03-18T22:40:47.202264+0000","last_undegraded":"2024-03-18T22:40:47.202264+0000","last_fullsized":"2024-03-18T22:40:47.202264+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:43:10.821075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.040152+0000","last_change":"2024-03-18T22:40:41.210830+0000","last_active":"2024-03-18T22:41:06.040152+0000","last_peered":"2024-03-18T22:41:06.040152+0000","last_clean":"2024-03-18T22:41:06.040152+0000","last_became_active":"2024-03-18T22:40:41.210476+0000","last_became_peered":"2024-03-18T22:40:41.210476+0000","last_unstale":"2024-03-18T22:41:06.040152+0000","last_undegraded":"2024-03-18T22:41:06.040152+0000","last_fullsized":"2024-03-18T22:41:06.040152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:13.243788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202078+0000","last_change":"2024-03-18T22:40:41.807678+0000","last_active":"2024-03-18T22:40:47.202078+0000","last_peered":"2024-03-18T22:40:47.202078+0000","last_clean":"2024-03-18T22:40:47.202078+0000","last_became_active":"2024-03-18T22:40:41.807383+0000","last_became_peered":"2024-03-18T22:40:41.807383+0000","last_unstale":"2024-03-18T22:40:47.202078+0000","last_undegraded":"2024-03-18T22:40:47.202078+0000","last_fullsized":"2024-03-18T22:40:47.202078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:27:55.930293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202000+0000","last_change":"2024-03-18T22:40:45.850982+0000","last_active":"2024-03-18T22:40:47.202000+0000","last_peered":"2024-03-18T22:40:47.202000+0000","last_clean":"2024-03-18T22:40:47.202000+0000","last_became_active":"2024-03-18T22:40:45.850526+0000","last_became_peered":"2024-03-18T22:40:45.850526+0000","last_unstale":"2024-03-18T22:40:47.202000+0000","last_undegraded":"2024-03-18T22:40:47.202000+0000","last_fullsized":"2024-03-18T22:40:47.202000+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:05:24.450986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201971+0000","last_change":"2024-03-18T22:40:41.219540+0000","last_active":"2024-03-18T22:40:47.201971+0000","last_peered":"2024-03-18T22:40:47.201971+0000","last_clean":"2024-03-18T22:40:47.201971+0000","last_became_active":"2024-03-18T22:40:41.219206+0000","last_became_peered":"2024-03-18T22:40:41.219206+0000","last_unstale":"2024-03-18T22:40:47.201971+0000","last_undegraded":"2024-03-18T22:40:47.201971+0000","last_fullsized":"2024-03-18T22:40:47.201971+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:33.586368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201618+0000","last_change":"2024-03-18T22:40:41.815561+0000","last_active":"2024-03-18T22:40:47.201618+0000","last_peered":"2024-03-18T22:40:47.201618+0000","last_clean":"2024-03-18T22:40:47.201618+0000","last_became_active":"2024-03-18T22:40:41.815082+0000","last_became_peered":"2024-03-18T22:40:41.815082+0000","last_unstale":"2024-03-18T22:40:47.201618+0000","last_undegraded":"2024-03-18T22:40:47.201618+0000","last_fullsized":"2024-03-18T22:40:47.201618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:01.676907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202496+0000","last_change":"2024-03-18T22:40:45.845386+0000","last_active":"2024-03-18T22:40:47.202496+0000","last_peered":"2024-03-18T22:40:47.202496+0000","last_clean":"2024-03-18T22:40:47.202496+0000","last_became_active":"2024-03-18T22:40:45.845007+0000","last_became_peered":"2024-03-18T22:40:45.845007+0000","last_unstale":"2024-03-18T22:40:47.202496+0000","last_undegraded":"2024-03-18T22:40:47.202496+0000","last_fullsized":"2024-03-18T22:40:47.202496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:41.429195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.041468+0000","last_change":"2024-03-18T22:40:41.215559+0000","last_active":"2024-03-18T22:41:06.041468+0000","last_peered":"2024-03-18T22:41:06.041468+0000","last_clean":"2024-03-18T22:41:06.041468+0000","last_became_active":"2024-03-18T22:40:41.215253+0000","last_became_peered":"2024-03-18T22:40:41.215253+0000","last_unstale":"2024-03-18T22:41:06.041468+0000","last_undegraded":"2024-03-18T22:41:06.041468+0000","last_fullsized":"2024-03-18T22:41:06.041468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:19.706011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843521+0000","last_change":"2024-03-18T22:40:41.807533+0000","last_active":"2024-03-18T22:40:46.843521+0000","last_peered":"2024-03-18T22:40:46.843521+0000","last_clean":"2024-03-18T22:40:46.843521+0000","last_became_active":"2024-03-18T22:40:41.807219+0000","last_became_peered":"2024-03-18T22:40:41.807219+0000","last_unstale":"2024-03-18T22:40:46.843521+0000","last_undegraded":"2024-03-18T22:40:46.843521+0000","last_fullsized":"2024-03-18T22:40:46.843521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:15:24.214367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202061+0000","last_change":"2024-03-18T22:40:45.839670+0000","last_active":"2024-03-18T22:40:47.202061+0000","last_peered":"2024-03-18T22:40:47.202061+0000","last_clean":"2024-03-18T22:40:47.202061+0000","last_became_active":"2024-03-18T22:40:45.839263+0000","last_became_peered":"2024-03-18T22:40:45.839263+0000","last_unstale":"2024-03-18T22:40:47.202061+0000","last_undegraded":"2024-03-18T22:40:47.202061+0000","last_fullsized":"2024-03-18T22:40:47.202061+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:48.991959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202445+0000","last_change":"2024-03-18T22:40:41.209617+0000","last_active":"2024-03-18T22:40:47.202445+0000","last_peered":"2024-03-18T22:40:47.202445+0000","last_clean":"2024-03-18T22:40:47.202445+0000","last_became_active":"2024-03-18T22:40:41.209311+0000","last_became_peered":"2024-03-18T22:40:41.209311+0000","last_unstale":"2024-03-18T22:40:47.202445+0000","last_undegraded":"2024-03-18T22:40:47.202445+0000","last_fullsized":"2024-03-18T22:40:47.202445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:44:06.019870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843930+0000","last_change":"2024-03-18T22:40:41.808210+0000","last_active":"2024-03-18T22:40:46.843930+0000","last_peered":"2024-03-18T22:40:46.843930+0000","last_clean":"2024-03-18T22:40:46.843930+0000","last_became_active":"2024-03-18T22:40:41.807898+0000","last_became_peered":"2024-03-18T22:40:41.807898+0000","last_unstale":"2024-03-18T22:40:46.843930+0000","last_undegraded":"2024-03-18T22:40:46.843930+0000","last_fullsized":"2024-03-18T22:40:46.843930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:20.401165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.842467+0000","last_change":"2024-03-18T22:40:45.841905+0000","last_active":"2024-03-18T22:40:46.842467+0000","last_peered":"2024-03-18T22:40:46.842467+0000","last_clean":"2024-03-18T22:40:46.842467+0000","last_became_active":"2024-03-18T22:40:45.841464+0000","last_became_peered":"2024-03-18T22:40:45.841464+0000","last_unstale":"2024-03-18T22:40:46.842467+0000","last_undegraded":"2024-03-18T22:40:46.842467+0000","last_fullsized":"2024-03-18T22:40:46.842467+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:30.435788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.832580+0000","last_change":"2024-03-18T22:40:41.216919+0000","last_active":"2024-03-18T22:41:07.832580+0000","last_peered":"2024-03-18T22:41:07.832580+0000","last_clean":"2024-03-18T22:41:07.832580+0000","last_became_active":"2024-03-18T22:40:41.216608+0000","last_became_peered":"2024-03-18T22:40:41.216608+0000","last_unstale":"2024-03-18T22:41:07.832580+0000","last_undegraded":"2024-03-18T22:41:07.832580+0000","last_fullsized":"2024-03-18T22:41:07.832580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:10:42.230268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843684+0000","last_change":"2024-03-18T22:40:41.813879+0000","last_active":"2024-03-18T22:40:46.843684+0000","last_peered":"2024-03-18T22:40:46.843684+0000","last_clean":"2024-03-18T22:40:46.843684+0000","last_became_active":"2024-03-18T22:40:41.813584+0000","last_became_peered":"2024-03-18T22:40:41.813584+0000","last_unstale":"2024-03-18T22:40:46.843684+0000","last_undegraded":"2024-03-18T22:40:46.843684+0000","last_fullsized":"2024-03-18T22:40:46.843684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:20:44.808087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203076+0000","last_change":"2024-03-18T22:40:45.838033+0000","last_active":"2024-03-18T22:40:47.203076+0000","last_peered":"2024-03-18T22:40:47.203076+0000","last_clean":"2024-03-18T22:40:47.203076+0000","last_became_active":"2024-03-18T22:40:45.837530+0000","last_became_peered":"2024-03-18T22:40:45.837530+0000","last_unstale":"2024-03-18T22:40:47.203076+0000","last_undegraded":"2024-03-18T22:40:47.203076+0000","last_fullsized":"2024-03-18T22:40:47.203076+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:43:43.444631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844373+0000","last_change":"2024-03-18T22:40:41.215686+0000","last_active":"2024-03-18T22:40:46.844373+0000","last_peered":"2024-03-18T22:40:46.844373+0000","last_clean":"2024-03-18T22:40:46.844373+0000","last_became_active":"2024-03-18T22:40:41.215362+0000","last_became_peered":"2024-03-18T22:40:41.215362+0000","last_unstale":"2024-03-18T22:40:46.844373+0000","last_undegraded":"2024-03-18T22:40:46.844373+0000","last_fullsized":"2024-03-18T22:40:46.844373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:46:30.546304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844074+0000","last_change":"2024-03-18T22:40:41.808959+0000","last_active":"2024-03-18T22:40:46.844074+0000","last_peered":"2024-03-18T22:40:46.844074+0000","last_clean":"2024-03-18T22:40:46.844074+0000","last_became_active":"2024-03-18T22:40:41.808671+0000","last_became_peered":"2024-03-18T22:40:41.808671+0000","last_unstale":"2024-03-18T22:40:46.844074+0000","last_undegraded":"2024-03-18T22:40:46.844074+0000","last_fullsized":"2024-03-18T22:40:46.844074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:37.115318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202207+0000","last_change":"2024-03-18T22:40:41.814240+0000","last_active":"2024-03-18T22:40:47.202207+0000","last_peered":"2024-03-18T22:40:47.202207+0000","last_clean":"2024-03-18T22:40:47.202207+0000","last_became_active":"2024-03-18T22:40:41.813925+0000","last_became_peered":"2024-03-18T22:40:41.813925+0000","last_unstale":"2024-03-18T22:40:47.202207+0000","last_undegraded":"2024-03-18T22:40:47.202207+0000","last_fullsized":"2024-03-18T22:40:47.202207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:38.999803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202253+0000","last_change":"2024-03-18T22:40:41.813381+0000","last_active":"2024-03-18T22:40:47.202253+0000","last_peered":"2024-03-18T22:40:47.202253+0000","last_clean":"2024-03-18T22:40:47.202253+0000","last_became_active":"2024-03-18T22:40:41.812927+0000","last_became_peered":"2024-03-18T22:40:41.812927+0000","last_unstale":"2024-03-18T22:40:47.202253+0000","last_undegraded":"2024-03-18T22:40:47.202253+0000","last_fullsized":"2024-03-18T22:40:47.202253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:23:09.178485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201795+0000","last_change":"2024-03-18T22:40:41.812513+0000","last_active":"2024-03-18T22:40:47.201795+0000","last_peered":"2024-03-18T22:40:47.201795+0000","last_clean":"2024-03-18T22:40:47.201795+0000","last_became_active":"2024-03-18T22:40:41.811882+0000","last_became_peered":"2024-03-18T22:40:41.811882+0000","last_unstale":"2024-03-18T22:40:47.201795+0000","last_undegraded":"2024-03-18T22:40:47.201795+0000","last_fullsized":"2024-03-18T22:40:47.201795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:45.596911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.842621+0000","last_change":"2024-03-18T22:40:41.814960+0000","last_active":"2024-03-18T22:40:46.842621+0000","last_peered":"2024-03-18T22:40:46.842621+0000","last_clean":"2024-03-18T22:40:46.842621+0000","last_became_active":"2024-03-18T22:40:41.814620+0000","last_became_peered":"2024-03-18T22:40:41.814620+0000","last_unstale":"2024-03-18T22:40:46.842621+0000","last_undegraded":"2024-03-18T22:40:46.842621+0000","last_fullsized":"2024-03-18T22:40:46.842621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:42.205885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844492+0000","last_change":"2024-03-18T22:40:41.810023+0000","last_active":"2024-03-18T22:40:46.844492+0000","last_peered":"2024-03-18T22:40:46.844492+0000","last_clean":"2024-03-18T22:40:46.844492+0000","last_became_active":"2024-03-18T22:40:41.806027+0000","last_became_peered":"2024-03-18T22:40:41.806027+0000","last_unstale":"2024-03-18T22:40:46.844492+0000","last_undegraded":"2024-03-18T22:40:46.844492+0000","last_fullsized":"2024-03-18T22:40:46.844492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:46.472567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203520+0000","last_change":"2024-03-18T22:40:41.808718+0000","last_active":"2024-03-18T22:40:47.203520+0000","last_peered":"2024-03-18T22:40:47.203520+0000","last_clean":"2024-03-18T22:40:47.203520+0000","last_became_active":"2024-03-18T22:40:41.808379+0000","last_became_peered":"2024-03-18T22:40:41.808379+0000","last_unstale":"2024-03-18T22:40:47.203520+0000","last_undegraded":"2024-03-18T22:40:47.203520+0000","last_fullsized":"2024-03-18T22:40:47.203520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:26.637531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202815+0000","last_change":"2024-03-18T22:40:41.818046+0000","last_active":"2024-03-18T22:40:47.202815+0000","last_peered":"2024-03-18T22:40:47.202815+0000","last_clean":"2024-03-18T22:40:47.202815+0000","last_became_active":"2024-03-18T22:40:41.817698+0000","last_became_peered":"2024-03-18T22:40:41.817698+0000","last_unstale":"2024-03-18T22:40:47.202815+0000","last_undegraded":"2024-03-18T22:40:47.202815+0000","last_fullsized":"2024-03-18T22:40:47.202815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:22.967673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202222+0000","last_change":"2024-03-18T22:40:41.819337+0000","last_active":"2024-03-18T22:40:47.202222+0000","last_peered":"2024-03-18T22:40:47.202222+0000","last_clean":"2024-03-18T22:40:47.202222+0000","last_became_active":"2024-03-18T22:40:41.818987+0000","last_became_peered":"2024-03-18T22:40:41.818987+0000","last_unstale":"2024-03-18T22:40:47.202222+0000","last_undegraded":"2024-03-18T22:40:47.202222+0000","last_fullsized":"2024-03-18T22:40:47.202222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:07:04.529454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203272+0000","last_change":"2024-03-18T22:40:41.813262+0000","last_active":"2024-03-18T22:40:47.203272+0000","last_peered":"2024-03-18T22:40:47.203272+0000","last_clean":"2024-03-18T22:40:47.203272+0000","last_became_active":"2024-03-18T22:40:41.812956+0000","last_became_peered":"2024-03-18T22:40:41.812956+0000","last_unstale":"2024-03-18T22:40:47.203272+0000","last_undegraded":"2024-03-18T22:40:47.203272+0000","last_fullsized":"2024-03-18T22:40:47.203272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:19.940097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202324+0000","last_change":"2024-03-18T22:40:41.811897+0000","last_active":"2024-03-18T22:40:47.202324+0000","last_peered":"2024-03-18T22:40:47.202324+0000","last_clean":"2024-03-18T22:40:47.202324+0000","last_became_active":"2024-03-18T22:40:41.811575+0000","last_became_peered":"2024-03-18T22:40:41.811575+0000","last_unstale":"2024-03-18T22:40:47.202324+0000","last_undegraded":"2024-03-18T22:40:47.202324+0000","last_fullsized":"2024-03-18T22:40:47.202324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:45:43.321057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202268+0000","last_change":"2024-03-18T22:40:41.811757+0000","last_active":"2024-03-18T22:40:47.202268+0000","last_peered":"2024-03-18T22:40:47.202268+0000","last_clean":"2024-03-18T22:40:47.202268+0000","last_became_active":"2024-03-18T22:40:41.811405+0000","last_became_peered":"2024-03-18T22:40:41.811405+0000","last_unstale":"2024-03-18T22:40:47.202268+0000","last_undegraded":"2024-03-18T22:40:47.202268+0000","last_fullsized":"2024-03-18T22:40:47.202268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:39.224604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844157+0000","last_change":"2024-03-18T22:40:41.809987+0000","last_active":"2024-03-18T22:40:46.844157+0000","last_peered":"2024-03-18T22:40:46.844157+0000","last_clean":"2024-03-18T22:40:46.844157+0000","last_became_active":"2024-03-18T22:40:41.809544+0000","last_became_peered":"2024-03-18T22:40:41.809544+0000","last_unstale":"2024-03-18T22:40:46.844157+0000","last_undegraded":"2024-03-18T22:40:46.844157+0000","last_fullsized":"2024-03-18T22:40:46.844157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:23.800949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203338+0000","last_change":"2024-03-18T22:40:41.810963+0000","last_active":"2024-03-18T22:40:47.203338+0000","last_peered":"2024-03-18T22:40:47.203338+0000","last_clean":"2024-03-18T22:40:47.203338+0000","last_became_active":"2024-03-18T22:40:41.810641+0000","last_became_peered":"2024-03-18T22:40:41.810641+0000","last_unstale":"2024-03-18T22:40:47.203338+0000","last_undegraded":"2024-03-18T22:40:47.203338+0000","last_fullsized":"2024-03-18T22:40:47.203338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:37:48.872961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8213,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":132,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":27591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":478407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36932,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708220,"statfs":{"total":95995035648,"available":95957217280,"internally_reserved":0,"allocated":1634304,"data_stored":1073944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808859,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":478123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070495,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1068787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":481640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":479393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":481714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":477409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":477237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28604,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716548,"statfs":{"total":95995035648,"available":95965745152,"internally_reserved":0,"allocated":1626112,"data_stored":1074890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":6361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T22:41:12.288 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T22:41:12.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:11 smithi113 ceph-mon[66775]: from='client.24751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:41:12.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:11 smithi113 ceph-mon[66775]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:41:12.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:11 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3515068510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:41:12.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.424+0000 7f4275f22640 1 -- 172.21.15.40:0/2959457408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42701009b0 msgr2=0x7f4270100e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:12.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.424+0000 7f4275f22640 1 --2- 172.21.15.40:0/2959457408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42701009b0 0x7f4270100e30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4264009900 tx=0x7f426402bc30 comp rx=0 tx=0).stop 2024-03-18T22:41:12.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.424+0000 7f4275f22640 1 -- 172.21.15.40:0/2959457408 shutdown_connections 2024-03-18T22:41:12.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.424+0000 7f4275f22640 1 --2- 172.21.15.40:0/2959457408 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4270101370 0x7f4270108c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.424+0000 7f4275f22640 1 --2- 172.21.15.40:0/2959457408 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42701009b0 0x7f4270100e30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.424+0000 7f4275f22640 1 --2- 172.21.15.40:0/2959457408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4270101d60 0x7f4270102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.424+0000 7f4275f22640 1 -- 172.21.15.40:0/2959457408 >> 172.21.15.40:0/2959457408 conn(0x7f42700fb440 msgr2=0x7f42700fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:12.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.424+0000 7f4275f22640 1 -- 172.21.15.40:0/2959457408 shutdown_connections 2024-03-18T22:41:12.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.424+0000 7f4275f22640 1 -- 172.21.15.40:0/2959457408 wait complete. 2024-03-18T22:41:12.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f4275f22640 1 Processor -- start 2024-03-18T22:41:12.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f4275f22640 1 -- start start 2024-03-18T22:41:12.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f4275f22640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42701009b0 0x7f4270197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:12.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f4275f22640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4270101370 0x7f4270197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:12.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f4275f22640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4270101d60 0x7f427019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:12.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f4275f22640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f427010d400 con 0x7f42701009b0 2024-03-18T22:41:12.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f4275f22640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f427010d280 con 0x7f4270101370 2024-03-18T22:41:12.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f4275f22640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f427010d580 con 0x7f4270101d60 2024-03-18T22:41:12.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f426f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42701009b0 0x7f4270197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:12.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f426f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42701009b0 0x7f4270197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40800/0 (socket says 172.21.15.40:40800) 2024-03-18T22:41:12.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f426f7fe640 1 -- 172.21.15.40:0/2110244918 learned_addr learned my addr 172.21.15.40:0/2110244918 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:12.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f426effd640 1 --2- 172.21.15.40:0/2110244918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4270101370 0x7f4270197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:12.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f426ffff640 1 --2- 172.21.15.40:0/2110244918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4270101d60 0x7f427019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:12.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f426f7fe640 1 -- 172.21.15.40:0/2110244918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4270101d60 msgr2=0x7f427019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:12.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f426f7fe640 1 --2- 172.21.15.40:0/2110244918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4270101d60 0x7f427019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f426f7fe640 1 -- 172.21.15.40:0/2110244918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4270101370 msgr2=0x7f4270197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:12.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f426f7fe640 1 --2- 172.21.15.40:0/2110244918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4270101370 0x7f4270197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f426f7fe640 1 -- 172.21.15.40:0/2110244918 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427019f6c0 con 0x7f42701009b0 2024-03-18T22:41:12.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.425+0000 7f426effd640 1 --2- 172.21.15.40:0/2110244918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4270101370 0x7f4270197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:12.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.426+0000 7f426f7fe640 1 --2- 172.21.15.40:0/2110244918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42701009b0 0x7f4270197aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f426000e9c0 tx=0x7f426000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:12.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.426+0000 7f426cff9640 1 -- 172.21.15.40:0/2110244918 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426000ccc0 con 0x7f42701009b0 2024-03-18T22:41:12.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.426+0000 7f4275f22640 1 -- 172.21.15.40:0/2110244918 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f427019f9b0 con 0x7f42701009b0 2024-03-18T22:41:12.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.426+0000 7f426cff9640 1 -- 172.21.15.40:0/2110244918 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f42600035f0 con 0x7f42701009b0 2024-03-18T22:41:12.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.426+0000 7f426cff9640 1 -- 172.21.15.40:0/2110244918 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426000ce60 con 0x7f42701009b0 2024-03-18T22:41:12.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.426+0000 7f4275f22640 1 -- 172.21.15.40:0/2110244918 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427019fed0 con 0x7f42701009b0 2024-03-18T22:41:12.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.427+0000 7f4275f22640 1 -- 172.21.15.40:0/2110244918 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4270107cb0 con 0x7f42701009b0 2024-03-18T22:41:12.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.428+0000 7f426cff9640 1 -- 172.21.15.40:0/2110244918 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f42600037a0 con 0x7f42701009b0 2024-03-18T22:41:12.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.428+0000 7f426cff9640 1 --2- 172.21.15.40:0/2110244918 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f42580a2890 0x7f42580a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:12.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.428+0000 7f426effd640 1 --2- 172.21.15.40:0/2110244918 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f42580a2890 0x7f42580a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:12.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.428+0000 7f426cff9640 1 -- 172.21.15.40:0/2110244918 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f42600cf1e0 con 0x7f42701009b0 2024-03-18T22:41:12.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.429+0000 7f426effd640 1 --2- 172.21.15.40:0/2110244918 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f42580a2890 0x7f42580a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4264005ab0 tx=0x7f4264005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:12.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.433+0000 7f426cff9640 1 -- 172.21.15.40:0/2110244918 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f426009bee0 con 0x7f42701009b0 2024-03-18T22:41:12.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.613+0000 7f4275f22640 1 -- 172.21.15.40:0/2110244918 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4270061c60 con 0x7f42580a2890 2024-03-18T22:41:12.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.623+0000 7f426cff9640 1 -- 172.21.15.40:0/2110244918 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340307 (secure 0 0 0) 0x7f4270061c60 con 0x7f42580a2890 2024-03-18T22:41:12.627 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:41:12.629 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-18T22:41:12.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.626+0000 7f4275f22640 1 -- 172.21.15.40:0/2110244918 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f42580a2890 msgr2=0x7f42580a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:12.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.626+0000 7f4275f22640 1 --2- 172.21.15.40:0/2110244918 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f42580a2890 0x7f42580a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4264005ab0 tx=0x7f4264005fd0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.626+0000 7f4275f22640 1 -- 172.21.15.40:0/2110244918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42701009b0 msgr2=0x7f4270197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:12.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.626+0000 7f4275f22640 1 --2- 172.21.15.40:0/2110244918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42701009b0 0x7f4270197aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f426000e9c0 tx=0x7f426000edc0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.626+0000 7f4275f22640 1 -- 172.21.15.40:0/2110244918 shutdown_connections 2024-03-18T22:41:12.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.626+0000 7f4275f22640 1 --2- 172.21.15.40:0/2110244918 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f42580a2890 0x7f42580a4d50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.626+0000 7f4275f22640 1 --2- 172.21.15.40:0/2110244918 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4270101d60 0x7f427019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.626+0000 7f4275f22640 1 --2- 172.21.15.40:0/2110244918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4270101370 0x7f4270197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.626+0000 7f4275f22640 1 --2- 172.21.15.40:0/2110244918 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42701009b0 0x7f4270197aa0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:12.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.626+0000 7f4275f22640 1 -- 172.21.15.40:0/2110244918 >> 172.21.15.40:0/2110244918 conn(0x7f42700fb440 msgr2=0x7f4270104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:12.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.626+0000 7f4275f22640 1 -- 172.21.15.40:0/2110244918 shutdown_connections 2024-03-18T22:41:12.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:12.626+0000 7f4275f22640 1 -- 172.21.15.40:0/2110244918 wait complete. 2024-03-18T22:41:12.637 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-03-18T22:41:10.809019+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598581,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":313,"num_write_kb":2982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":253,"ondisk_log_size":253,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":379316,"kb_used_data":13520,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562508,"statfs":{"total":1151940427776,"available":1151552008192,"internally_reserved":0,"allocated":13844480,"data_stored":7526600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3713,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":85,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003927"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203760+0000","last_change":"2024-03-18T22:40:41.806379+0000","last_active":"2024-03-18T22:40:47.203760+0000","last_peered":"2024-03-18T22:40:47.203760+0000","last_clean":"2024-03-18T22:40:47.203760+0000","last_became_active":"2024-03-18T22:40:41.806058+0000","last_became_peered":"2024-03-18T22:40:41.806058+0000","last_unstale":"2024-03-18T22:40:47.203760+0000","last_undegraded":"2024-03-18T22:40:47.203760+0000","last_fullsized":"2024-03-18T22:40:47.203760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:45.834396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843169+0000","last_change":"2024-03-18T22:40:41.813722+0000","last_active":"2024-03-18T22:40:46.843169+0000","last_peered":"2024-03-18T22:40:46.843169+0000","last_clean":"2024-03-18T22:40:46.843169+0000","last_became_active":"2024-03-18T22:40:41.813284+0000","last_became_peered":"2024-03-18T22:40:41.813284+0000","last_unstale":"2024-03-18T22:40:46.843169+0000","last_undegraded":"2024-03-18T22:40:46.843169+0000","last_fullsized":"2024-03-18T22:40:46.843169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:46.374117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.842444+0000","last_change":"2024-03-18T22:40:41.808340+0000","last_active":"2024-03-18T22:40:46.842444+0000","last_peered":"2024-03-18T22:40:46.842444+0000","last_clean":"2024-03-18T22:40:46.842444+0000","last_became_active":"2024-03-18T22:40:41.808002+0000","last_became_peered":"2024-03-18T22:40:41.808002+0000","last_unstale":"2024-03-18T22:40:46.842444+0000","last_undegraded":"2024-03-18T22:40:46.842444+0000","last_fullsized":"2024-03-18T22:40:46.842444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:40:32.122124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202258+0000","last_change":"2024-03-18T22:40:41.809959+0000","last_active":"2024-03-18T22:40:47.202258+0000","last_peered":"2024-03-18T22:40:47.202258+0000","last_clean":"2024-03-18T22:40:47.202258+0000","last_became_active":"2024-03-18T22:40:41.809676+0000","last_became_peered":"2024-03-18T22:40:41.809676+0000","last_unstale":"2024-03-18T22:40:47.202258+0000","last_undegraded":"2024-03-18T22:40:47.202258+0000","last_fullsized":"2024-03-18T22:40:47.202258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:29:45.263011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202250+0000","last_change":"2024-03-18T22:40:41.804583+0000","last_active":"2024-03-18T22:40:47.202250+0000","last_peered":"2024-03-18T22:40:47.202250+0000","last_clean":"2024-03-18T22:40:47.202250+0000","last_became_active":"2024-03-18T22:40:41.804064+0000","last_became_peered":"2024-03-18T22:40:41.804064+0000","last_unstale":"2024-03-18T22:40:47.202250+0000","last_undegraded":"2024-03-18T22:40:47.202250+0000","last_fullsized":"2024-03-18T22:40:47.202250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:46:23.455106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202510+0000","last_change":"2024-03-18T22:40:41.811230+0000","last_active":"2024-03-18T22:40:47.202510+0000","last_peered":"2024-03-18T22:40:47.202510+0000","last_clean":"2024-03-18T22:40:47.202510+0000","last_became_active":"2024-03-18T22:40:41.810904+0000","last_became_peered":"2024-03-18T22:40:41.810904+0000","last_unstale":"2024-03-18T22:40:47.202510+0000","last_undegraded":"2024-03-18T22:40:47.202510+0000","last_fullsized":"2024-03-18T22:40:47.202510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:51.302617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843440+0000","last_change":"2024-03-18T22:40:41.814971+0000","last_active":"2024-03-18T22:40:46.843440+0000","last_peered":"2024-03-18T22:40:46.843440+0000","last_clean":"2024-03-18T22:40:46.843440+0000","last_became_active":"2024-03-18T22:40:41.814305+0000","last_became_peered":"2024-03-18T22:40:41.814305+0000","last_unstale":"2024-03-18T22:40:46.843440+0000","last_undegraded":"2024-03-18T22:40:46.843440+0000","last_fullsized":"2024-03-18T22:40:46.843440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:07.390157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201249+0000","last_change":"2024-03-18T22:40:41.815674+0000","last_active":"2024-03-18T22:40:47.201249+0000","last_peered":"2024-03-18T22:40:47.201249+0000","last_clean":"2024-03-18T22:40:47.201249+0000","last_became_active":"2024-03-18T22:40:41.813560+0000","last_became_peered":"2024-03-18T22:40:41.813560+0000","last_unstale":"2024-03-18T22:40:47.201249+0000","last_undegraded":"2024-03-18T22:40:47.201249+0000","last_fullsized":"2024-03-18T22:40:47.201249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:42:32.409338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203022+0000","last_change":"2024-03-18T22:40:41.817508+0000","last_active":"2024-03-18T22:40:47.203022+0000","last_peered":"2024-03-18T22:40:47.203022+0000","last_clean":"2024-03-18T22:40:47.203022+0000","last_became_active":"2024-03-18T22:40:41.817184+0000","last_became_peered":"2024-03-18T22:40:41.817184+0000","last_unstale":"2024-03-18T22:40:47.203022+0000","last_undegraded":"2024-03-18T22:40:47.203022+0000","last_fullsized":"2024-03-18T22:40:47.203022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:10.255238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202325+0000","last_change":"2024-03-18T22:40:41.806235+0000","last_active":"2024-03-18T22:40:47.202325+0000","last_peered":"2024-03-18T22:40:47.202325+0000","last_clean":"2024-03-18T22:40:47.202325+0000","last_became_active":"2024-03-18T22:40:41.805913+0000","last_became_peered":"2024-03-18T22:40:41.805913+0000","last_unstale":"2024-03-18T22:40:47.202325+0000","last_undegraded":"2024-03-18T22:40:47.202325+0000","last_fullsized":"2024-03-18T22:40:47.202325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:39:06.299989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843867+0000","last_change":"2024-03-18T22:40:41.814735+0000","last_active":"2024-03-18T22:40:46.843867+0000","last_peered":"2024-03-18T22:40:46.843867+0000","last_clean":"2024-03-18T22:40:46.843867+0000","last_became_active":"2024-03-18T22:40:41.814291+0000","last_became_peered":"2024-03-18T22:40:41.814291+0000","last_unstale":"2024-03-18T22:40:46.843867+0000","last_undegraded":"2024-03-18T22:40:46.843867+0000","last_fullsized":"2024-03-18T22:40:46.843867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:49.610674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202165+0000","last_change":"2024-03-18T22:40:41.815561+0000","last_active":"2024-03-18T22:40:47.202165+0000","last_peered":"2024-03-18T22:40:47.202165+0000","last_clean":"2024-03-18T22:40:47.202165+0000","last_became_active":"2024-03-18T22:40:41.815252+0000","last_became_peered":"2024-03-18T22:40:41.815252+0000","last_unstale":"2024-03-18T22:40:47.202165+0000","last_undegraded":"2024-03-18T22:40:47.202165+0000","last_fullsized":"2024-03-18T22:40:47.202165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:34.769168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844354+0000","last_change":"2024-03-18T22:40:41.803325+0000","last_active":"2024-03-18T22:40:46.844354+0000","last_peered":"2024-03-18T22:40:46.844354+0000","last_clean":"2024-03-18T22:40:46.844354+0000","last_became_active":"2024-03-18T22:40:41.803008+0000","last_became_peered":"2024-03-18T22:40:41.803008+0000","last_unstale":"2024-03-18T22:40:46.844354+0000","last_undegraded":"2024-03-18T22:40:46.844354+0000","last_fullsized":"2024-03-18T22:40:46.844354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:14:01.335548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.842231+0000","last_change":"2024-03-18T22:40:41.812541+0000","last_active":"2024-03-18T22:40:46.842231+0000","last_peered":"2024-03-18T22:40:46.842231+0000","last_clean":"2024-03-18T22:40:46.842231+0000","last_became_active":"2024-03-18T22:40:41.812153+0000","last_became_peered":"2024-03-18T22:40:41.812153+0000","last_unstale":"2024-03-18T22:40:46.842231+0000","last_undegraded":"2024-03-18T22:40:46.842231+0000","last_fullsized":"2024-03-18T22:40:46.842231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:04.927057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.842356+0000","last_change":"2024-03-18T22:40:41.815790+0000","last_active":"2024-03-18T22:40:46.842356+0000","last_peered":"2024-03-18T22:40:46.842356+0000","last_clean":"2024-03-18T22:40:46.842356+0000","last_became_active":"2024-03-18T22:40:41.814949+0000","last_became_peered":"2024-03-18T22:40:41.814949+0000","last_unstale":"2024-03-18T22:40:46.842356+0000","last_undegraded":"2024-03-18T22:40:46.842356+0000","last_fullsized":"2024-03-18T22:40:46.842356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:10:15.874285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201282+0000","last_change":"2024-03-18T22:40:41.809980+0000","last_active":"2024-03-18T22:40:47.201282+0000","last_peered":"2024-03-18T22:40:47.201282+0000","last_clean":"2024-03-18T22:40:47.201282+0000","last_became_active":"2024-03-18T22:40:41.809640+0000","last_became_peered":"2024-03-18T22:40:41.809640+0000","last_unstale":"2024-03-18T22:40:47.201282+0000","last_undegraded":"2024-03-18T22:40:47.201282+0000","last_fullsized":"2024-03-18T22:40:47.201282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:25.389027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203649+0000","last_change":"2024-03-18T22:40:41.807088+0000","last_active":"2024-03-18T22:40:47.203649+0000","last_peered":"2024-03-18T22:40:47.203649+0000","last_clean":"2024-03-18T22:40:47.203649+0000","last_became_active":"2024-03-18T22:40:41.806770+0000","last_became_peered":"2024-03-18T22:40:41.806770+0000","last_unstale":"2024-03-18T22:40:47.203649+0000","last_undegraded":"2024-03-18T22:40:47.203649+0000","last_fullsized":"2024-03-18T22:40:47.203649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:53:14.001247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202008+0000","last_change":"2024-03-18T22:40:41.812685+0000","last_active":"2024-03-18T22:40:47.202008+0000","last_peered":"2024-03-18T22:40:47.202008+0000","last_clean":"2024-03-18T22:40:47.202008+0000","last_became_active":"2024-03-18T22:40:41.812346+0000","last_became_peered":"2024-03-18T22:40:41.812346+0000","last_unstale":"2024-03-18T22:40:47.202008+0000","last_undegraded":"2024-03-18T22:40:47.202008+0000","last_fullsized":"2024-03-18T22:40:47.202008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:25.033460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.204135+0000","last_change":"2024-03-18T22:40:41.812627+0000","last_active":"2024-03-18T22:40:47.204135+0000","last_peered":"2024-03-18T22:40:47.204135+0000","last_clean":"2024-03-18T22:40:47.204135+0000","last_became_active":"2024-03-18T22:40:41.812327+0000","last_became_peered":"2024-03-18T22:40:41.812327+0000","last_unstale":"2024-03-18T22:40:47.204135+0000","last_undegraded":"2024-03-18T22:40:47.204135+0000","last_fullsized":"2024-03-18T22:40:47.204135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:58:22.088589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202449+0000","last_change":"2024-03-18T22:40:45.847822+0000","last_active":"2024-03-18T22:40:47.202449+0000","last_peered":"2024-03-18T22:40:47.202449+0000","last_clean":"2024-03-18T22:40:47.202449+0000","last_became_active":"2024-03-18T22:40:45.847407+0000","last_became_peered":"2024-03-18T22:40:45.847407+0000","last_unstale":"2024-03-18T22:40:47.202449+0000","last_undegraded":"2024-03-18T22:40:47.202449+0000","last_fullsized":"2024-03-18T22:40:47.202449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:44.825997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'11","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.832346+0000","last_change":"2024-03-18T22:40:41.218797+0000","last_active":"2024-03-18T22:41:07.832346+0000","last_peered":"2024-03-18T22:41:07.832346+0000","last_clean":"2024-03-18T22:41:07.832346+0000","last_became_active":"2024-03-18T22:40:41.218472+0000","last_became_peered":"2024-03-18T22:40:41.218472+0000","last_unstale":"2024-03-18T22:41:07.832346+0000","last_undegraded":"2024-03-18T22:41:07.832346+0000","last_fullsized":"2024-03-18T22:41:07.832346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:10.230257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201431+0000","last_change":"2024-03-18T22:40:41.810692+0000","last_active":"2024-03-18T22:40:47.201431+0000","last_peered":"2024-03-18T22:40:47.201431+0000","last_clean":"2024-03-18T22:40:47.201431+0000","last_became_active":"2024-03-18T22:40:41.810390+0000","last_became_peered":"2024-03-18T22:40:41.810390+0000","last_unstale":"2024-03-18T22:40:47.201431+0000","last_undegraded":"2024-03-18T22:40:47.201431+0000","last_fullsized":"2024-03-18T22:40:47.201431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:57:05.649622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201468+0000","last_change":"2024-03-18T22:40:45.851126+0000","last_active":"2024-03-18T22:40:47.201468+0000","last_peered":"2024-03-18T22:40:47.201468+0000","last_clean":"2024-03-18T22:40:47.201468+0000","last_became_active":"2024-03-18T22:40:45.850549+0000","last_became_peered":"2024-03-18T22:40:45.850549+0000","last_unstale":"2024-03-18T22:40:47.201468+0000","last_undegraded":"2024-03-18T22:40:47.201468+0000","last_fullsized":"2024-03-18T22:40:47.201468+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:39.359263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201615+0000","last_change":"2024-03-18T22:40:41.220944+0000","last_active":"2024-03-18T22:40:47.201615+0000","last_peered":"2024-03-18T22:40:47.201615+0000","last_clean":"2024-03-18T22:40:47.201615+0000","last_became_active":"2024-03-18T22:40:41.220612+0000","last_became_peered":"2024-03-18T22:40:41.220612+0000","last_unstale":"2024-03-18T22:40:47.201615+0000","last_undegraded":"2024-03-18T22:40:47.201615+0000","last_fullsized":"2024-03-18T22:40:47.201615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:16.083134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.842940+0000","last_change":"2024-03-18T22:40:41.818031+0000","last_active":"2024-03-18T22:40:46.842940+0000","last_peered":"2024-03-18T22:40:46.842940+0000","last_clean":"2024-03-18T22:40:46.842940+0000","last_became_active":"2024-03-18T22:40:41.817693+0000","last_became_peered":"2024-03-18T22:40:41.817693+0000","last_unstale":"2024-03-18T22:40:46.842940+0000","last_undegraded":"2024-03-18T22:40:46.842940+0000","last_fullsized":"2024-03-18T22:40:46.842940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:05.901203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201799+0000","last_change":"2024-03-18T22:40:45.850749+0000","last_active":"2024-03-18T22:40:47.201799+0000","last_peered":"2024-03-18T22:40:47.201799+0000","last_clean":"2024-03-18T22:40:47.201799+0000","last_became_active":"2024-03-18T22:40:45.850318+0000","last_became_peered":"2024-03-18T22:40:45.850318+0000","last_unstale":"2024-03-18T22:40:47.201799+0000","last_undegraded":"2024-03-18T22:40:47.201799+0000","last_fullsized":"2024-03-18T22:40:47.201799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:56.047413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202763+0000","last_change":"2024-03-18T22:40:41.220775+0000","last_active":"2024-03-18T22:40:47.202763+0000","last_peered":"2024-03-18T22:40:47.202763+0000","last_clean":"2024-03-18T22:40:47.202763+0000","last_became_active":"2024-03-18T22:40:41.220313+0000","last_became_peered":"2024-03-18T22:40:41.220313+0000","last_unstale":"2024-03-18T22:40:47.202763+0000","last_undegraded":"2024-03-18T22:40:47.202763+0000","last_fullsized":"2024-03-18T22:40:47.202763+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:24.180884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202737+0000","last_change":"2024-03-18T22:40:41.818448+0000","last_active":"2024-03-18T22:40:47.202737+0000","last_peered":"2024-03-18T22:40:47.202737+0000","last_clean":"2024-03-18T22:40:47.202737+0000","last_became_active":"2024-03-18T22:40:41.818088+0000","last_became_peered":"2024-03-18T22:40:41.818088+0000","last_unstale":"2024-03-18T22:40:47.202737+0000","last_undegraded":"2024-03-18T22:40:47.202737+0000","last_fullsized":"2024-03-18T22:40:47.202737+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:34:54.298226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201508+0000","last_change":"2024-03-18T22:40:45.846250+0000","last_active":"2024-03-18T22:40:47.201508+0000","last_peered":"2024-03-18T22:40:47.201508+0000","last_clean":"2024-03-18T22:40:47.201508+0000","last_became_active":"2024-03-18T22:40:45.845859+0000","last_became_peered":"2024-03-18T22:40:45.845859+0000","last_unstale":"2024-03-18T22:40:47.201508+0000","last_undegraded":"2024-03-18T22:40:47.201508+0000","last_fullsized":"2024-03-18T22:40:47.201508+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:57.855301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202381+0000","last_change":"2024-03-18T22:40:41.216569+0000","last_active":"2024-03-18T22:40:47.202381+0000","last_peered":"2024-03-18T22:40:47.202381+0000","last_clean":"2024-03-18T22:40:47.202381+0000","last_became_active":"2024-03-18T22:40:41.216255+0000","last_became_peered":"2024-03-18T22:40:41.216255+0000","last_unstale":"2024-03-18T22:40:47.202381+0000","last_undegraded":"2024-03-18T22:40:47.202381+0000","last_fullsized":"2024-03-18T22:40:47.202381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:23.016346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203900+0000","last_change":"2024-03-18T22:40:41.818645+0000","last_active":"2024-03-18T22:40:47.203900+0000","last_peered":"2024-03-18T22:40:47.203900+0000","last_clean":"2024-03-18T22:40:47.203900+0000","last_became_active":"2024-03-18T22:40:41.818322+0000","last_became_peered":"2024-03-18T22:40:41.818322+0000","last_unstale":"2024-03-18T22:40:47.203900+0000","last_undegraded":"2024-03-18T22:40:47.203900+0000","last_fullsized":"2024-03-18T22:40:47.203900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:46.288533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203848+0000","last_change":"2024-03-18T22:40:45.839244+0000","last_active":"2024-03-18T22:40:47.203848+0000","last_peered":"2024-03-18T22:40:47.203848+0000","last_clean":"2024-03-18T22:40:47.203848+0000","last_became_active":"2024-03-18T22:40:45.838849+0000","last_became_peered":"2024-03-18T22:40:45.838849+0000","last_unstale":"2024-03-18T22:40:47.203848+0000","last_undegraded":"2024-03-18T22:40:47.203848+0000","last_fullsized":"2024-03-18T22:40:47.203848+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:28:55.327999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:08.827091+0000","last_change":"2024-03-18T22:40:41.210196+0000","last_active":"2024-03-18T22:41:08.827091+0000","last_peered":"2024-03-18T22:41:08.827091+0000","last_clean":"2024-03-18T22:41:08.827091+0000","last_became_active":"2024-03-18T22:40:41.209888+0000","last_became_peered":"2024-03-18T22:40:41.209888+0000","last_unstale":"2024-03-18T22:41:08.827091+0000","last_undegraded":"2024-03-18T22:41:08.827091+0000","last_fullsized":"2024-03-18T22:41:08.827091+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:06:31.134688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202849+0000","last_change":"2024-03-18T22:40:41.806646+0000","last_active":"2024-03-18T22:40:47.202849+0000","last_peered":"2024-03-18T22:40:47.202849+0000","last_clean":"2024-03-18T22:40:47.202849+0000","last_became_active":"2024-03-18T22:40:41.806106+0000","last_became_peered":"2024-03-18T22:40:41.806106+0000","last_unstale":"2024-03-18T22:40:47.202849+0000","last_undegraded":"2024-03-18T22:40:47.202849+0000","last_fullsized":"2024-03-18T22:40:47.202849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:11:06.369619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202154+0000","last_change":"2024-03-18T22:40:45.852982+0000","last_active":"2024-03-18T22:40:47.202154+0000","last_peered":"2024-03-18T22:40:47.202154+0000","last_clean":"2024-03-18T22:40:47.202154+0000","last_became_active":"2024-03-18T22:40:45.852432+0000","last_became_peered":"2024-03-18T22:40:45.852432+0000","last_unstale":"2024-03-18T22:40:47.202154+0000","last_undegraded":"2024-03-18T22:40:47.202154+0000","last_fullsized":"2024-03-18T22:40:47.202154+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:06:03.475875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.832107+0000","last_change":"2024-03-18T22:40:41.217449+0000","last_active":"2024-03-18T22:41:07.832107+0000","last_peered":"2024-03-18T22:41:07.832107+0000","last_clean":"2024-03-18T22:41:07.832107+0000","last_became_active":"2024-03-18T22:40:41.217127+0000","last_became_peered":"2024-03-18T22:40:41.217127+0000","last_unstale":"2024-03-18T22:41:07.832107+0000","last_undegraded":"2024-03-18T22:41:07.832107+0000","last_fullsized":"2024-03-18T22:41:07.832107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:08:00.178720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203995+0000","last_change":"2024-03-18T22:40:41.814154+0000","last_active":"2024-03-18T22:40:47.203995+0000","last_peered":"2024-03-18T22:40:47.203995+0000","last_clean":"2024-03-18T22:40:47.203995+0000","last_became_active":"2024-03-18T22:40:41.813824+0000","last_became_peered":"2024-03-18T22:40:41.813824+0000","last_unstale":"2024-03-18T22:40:47.203995+0000","last_undegraded":"2024-03-18T22:40:47.203995+0000","last_fullsized":"2024-03-18T22:40:47.203995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:12.412689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203586+0000","last_change":"2024-03-18T22:40:45.852711+0000","last_active":"2024-03-18T22:40:47.203586+0000","last_peered":"2024-03-18T22:40:47.203586+0000","last_clean":"2024-03-18T22:40:47.203586+0000","last_became_active":"2024-03-18T22:40:45.852247+0000","last_became_peered":"2024-03-18T22:40:45.852247+0000","last_unstale":"2024-03-18T22:40:47.203586+0000","last_undegraded":"2024-03-18T22:40:47.203586+0000","last_fullsized":"2024-03-18T22:40:47.203586+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:03.061689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843844+0000","last_change":"2024-03-18T22:40:41.212399+0000","last_active":"2024-03-18T22:40:46.843844+0000","last_peered":"2024-03-18T22:40:46.843844+0000","last_clean":"2024-03-18T22:40:46.843844+0000","last_became_active":"2024-03-18T22:40:41.207416+0000","last_became_peered":"2024-03-18T22:40:41.207416+0000","last_unstale":"2024-03-18T22:40:46.843844+0000","last_undegraded":"2024-03-18T22:40:46.843844+0000","last_fullsized":"2024-03-18T22:40:46.843844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:11.406342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201357+0000","last_change":"2024-03-18T22:40:41.807313+0000","last_active":"2024-03-18T22:40:47.201357+0000","last_peered":"2024-03-18T22:40:47.201357+0000","last_clean":"2024-03-18T22:40:47.201357+0000","last_became_active":"2024-03-18T22:40:41.806029+0000","last_became_peered":"2024-03-18T22:40:41.806029+0000","last_unstale":"2024-03-18T22:40:47.201357+0000","last_undegraded":"2024-03-18T22:40:47.201357+0000","last_fullsized":"2024-03-18T22:40:47.201357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:23:43.973443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.842932+0000","last_change":"2024-03-18T22:40:45.835479+0000","last_active":"2024-03-18T22:40:46.842932+0000","last_peered":"2024-03-18T22:40:46.842932+0000","last_clean":"2024-03-18T22:40:46.842932+0000","last_became_active":"2024-03-18T22:40:45.834994+0000","last_became_peered":"2024-03-18T22:40:45.834994+0000","last_unstale":"2024-03-18T22:40:46.842932+0000","last_undegraded":"2024-03-18T22:40:46.842932+0000","last_fullsized":"2024-03-18T22:40:46.842932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:31.317843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:08.840903+0000","last_change":"2024-03-18T22:40:41.212110+0000","last_active":"2024-03-18T22:41:08.840903+0000","last_peered":"2024-03-18T22:41:08.840903+0000","last_clean":"2024-03-18T22:41:08.840903+0000","last_became_active":"2024-03-18T22:40:41.211803+0000","last_became_peered":"2024-03-18T22:40:41.211803+0000","last_unstale":"2024-03-18T22:41:08.840903+0000","last_undegraded":"2024-03-18T22:41:08.840903+0000","last_fullsized":"2024-03-18T22:41:08.840903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:40.482983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201396+0000","last_change":"2024-03-18T22:40:41.812585+0000","last_active":"2024-03-18T22:40:47.201396+0000","last_peered":"2024-03-18T22:40:47.201396+0000","last_clean":"2024-03-18T22:40:47.201396+0000","last_became_active":"2024-03-18T22:40:41.812260+0000","last_became_peered":"2024-03-18T22:40:41.812260+0000","last_unstale":"2024-03-18T22:40:47.201396+0000","last_undegraded":"2024-03-18T22:40:47.201396+0000","last_fullsized":"2024-03-18T22:40:47.201396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:12:59.999503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202071+0000","last_change":"2024-03-18T22:40:45.842252+0000","last_active":"2024-03-18T22:40:47.202071+0000","last_peered":"2024-03-18T22:40:47.202071+0000","last_clean":"2024-03-18T22:40:47.202071+0000","last_became_active":"2024-03-18T22:40:45.841849+0000","last_became_peered":"2024-03-18T22:40:45.841849+0000","last_unstale":"2024-03-18T22:40:47.202071+0000","last_undegraded":"2024-03-18T22:40:47.202071+0000","last_fullsized":"2024-03-18T22:40:47.202071+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:21.253947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.833971+0000","last_change":"2024-03-18T22:40:41.214253+0000","last_active":"2024-03-18T22:41:07.833971+0000","last_peered":"2024-03-18T22:41:07.833971+0000","last_clean":"2024-03-18T22:41:07.833971+0000","last_became_active":"2024-03-18T22:40:41.213911+0000","last_became_peered":"2024-03-18T22:40:41.213911+0000","last_unstale":"2024-03-18T22:41:07.833971+0000","last_undegraded":"2024-03-18T22:41:07.833971+0000","last_fullsized":"2024-03-18T22:41:07.833971+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:19.185714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202726+0000","last_change":"2024-03-18T22:40:41.813281+0000","last_active":"2024-03-18T22:40:47.202726+0000","last_peered":"2024-03-18T22:40:47.202726+0000","last_clean":"2024-03-18T22:40:47.202726+0000","last_became_active":"2024-03-18T22:40:41.812909+0000","last_became_peered":"2024-03-18T22:40:41.812909+0000","last_unstale":"2024-03-18T22:40:47.202726+0000","last_undegraded":"2024-03-18T22:40:47.202726+0000","last_fullsized":"2024-03-18T22:40:47.202726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:17:26.619441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201925+0000","last_change":"2024-03-18T22:40:45.846956+0000","last_active":"2024-03-18T22:40:47.201925+0000","last_peered":"2024-03-18T22:40:47.201925+0000","last_clean":"2024-03-18T22:40:47.201925+0000","last_became_active":"2024-03-18T22:40:45.843061+0000","last_became_peered":"2024-03-18T22:40:45.843061+0000","last_unstale":"2024-03-18T22:40:47.201925+0000","last_undegraded":"2024-03-18T22:40:47.201925+0000","last_fullsized":"2024-03-18T22:40:47.201925+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:52.043424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.040678+0000","last_change":"2024-03-18T22:40:41.214933+0000","last_active":"2024-03-18T22:41:06.040678+0000","last_peered":"2024-03-18T22:41:06.040678+0000","last_clean":"2024-03-18T22:41:06.040678+0000","last_became_active":"2024-03-18T22:40:41.214606+0000","last_became_peered":"2024-03-18T22:40:41.214606+0000","last_unstale":"2024-03-18T22:41:06.040678+0000","last_undegraded":"2024-03-18T22:41:06.040678+0000","last_fullsized":"2024-03-18T22:41:06.040678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:08:06.230288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843060+0000","last_change":"2024-03-18T22:40:41.817590+0000","last_active":"2024-03-18T22:40:46.843060+0000","last_peered":"2024-03-18T22:40:46.843060+0000","last_clean":"2024-03-18T22:40:46.843060+0000","last_became_active":"2024-03-18T22:40:41.815809+0000","last_became_peered":"2024-03-18T22:40:41.815809+0000","last_unstale":"2024-03-18T22:40:46.843060+0000","last_undegraded":"2024-03-18T22:40:46.843060+0000","last_fullsized":"2024-03-18T22:40:46.843060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:44:54.043647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202049+0000","last_change":"2024-03-18T22:40:45.844767+0000","last_active":"2024-03-18T22:40:47.202049+0000","last_peered":"2024-03-18T22:40:47.202049+0000","last_clean":"2024-03-18T22:40:47.202049+0000","last_became_active":"2024-03-18T22:40:45.844251+0000","last_became_peered":"2024-03-18T22:40:45.844251+0000","last_unstale":"2024-03-18T22:40:47.202049+0000","last_undegraded":"2024-03-18T22:40:47.202049+0000","last_fullsized":"2024-03-18T22:40:47.202049+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:43.756959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.824859+0000","last_change":"2024-03-18T22:40:41.214310+0000","last_active":"2024-03-18T22:41:07.824859+0000","last_peered":"2024-03-18T22:41:07.824859+0000","last_clean":"2024-03-18T22:41:07.824859+0000","last_became_active":"2024-03-18T22:40:41.213963+0000","last_became_peered":"2024-03-18T22:40:41.213963+0000","last_unstale":"2024-03-18T22:41:07.824859+0000","last_undegraded":"2024-03-18T22:41:07.824859+0000","last_fullsized":"2024-03-18T22:41:07.824859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:03.401220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202884+0000","last_change":"2024-03-18T22:40:41.810876+0000","last_active":"2024-03-18T22:40:47.202884+0000","last_peered":"2024-03-18T22:40:47.202884+0000","last_clean":"2024-03-18T22:40:47.202884+0000","last_became_active":"2024-03-18T22:40:41.810104+0000","last_became_peered":"2024-03-18T22:40:41.810104+0000","last_unstale":"2024-03-18T22:40:47.202884+0000","last_undegraded":"2024-03-18T22:40:47.202884+0000","last_fullsized":"2024-03-18T22:40:47.202884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:41.294286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201637+0000","last_change":"2024-03-18T22:40:45.841830+0000","last_active":"2024-03-18T22:40:47.201637+0000","last_peered":"2024-03-18T22:40:47.201637+0000","last_clean":"2024-03-18T22:40:47.201637+0000","last_became_active":"2024-03-18T22:40:45.841426+0000","last_became_peered":"2024-03-18T22:40:45.841426+0000","last_unstale":"2024-03-18T22:40:47.201637+0000","last_undegraded":"2024-03-18T22:40:47.201637+0000","last_fullsized":"2024-03-18T22:40:47.201637+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:20:13.227650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'15","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:08.839243+0000","last_change":"2024-03-18T22:40:41.209837+0000","last_active":"2024-03-18T22:41:08.839243+0000","last_peered":"2024-03-18T22:41:08.839243+0000","last_clean":"2024-03-18T22:41:08.839243+0000","last_became_active":"2024-03-18T22:40:41.209465+0000","last_became_peered":"2024-03-18T22:40:41.209465+0000","last_unstale":"2024-03-18T22:41:08.839243+0000","last_undegraded":"2024-03-18T22:41:08.839243+0000","last_fullsized":"2024-03-18T22:41:08.839243+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:45:15.765475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4234,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":7,"num_write":17,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202532+0000","last_change":"2024-03-18T22:40:41.810058+0000","last_active":"2024-03-18T22:40:47.202532+0000","last_peered":"2024-03-18T22:40:47.202532+0000","last_clean":"2024-03-18T22:40:47.202532+0000","last_became_active":"2024-03-18T22:40:41.809677+0000","last_became_peered":"2024-03-18T22:40:41.809677+0000","last_unstale":"2024-03-18T22:40:47.202532+0000","last_undegraded":"2024-03-18T22:40:47.202532+0000","last_fullsized":"2024-03-18T22:40:47.202532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:03.935036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.204326+0000","last_change":"2024-03-18T22:40:45.845638+0000","last_active":"2024-03-18T22:40:47.204326+0000","last_peered":"2024-03-18T22:40:47.204326+0000","last_clean":"2024-03-18T22:40:47.204326+0000","last_became_active":"2024-03-18T22:40:45.844994+0000","last_became_peered":"2024-03-18T22:40:45.844994+0000","last_unstale":"2024-03-18T22:40:47.204326+0000","last_undegraded":"2024-03-18T22:40:47.204326+0000","last_fullsized":"2024-03-18T22:40:47.204326+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:21.551271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.038829+0000","last_change":"2024-03-18T22:40:41.218397+0000","last_active":"2024-03-18T22:41:06.038829+0000","last_peered":"2024-03-18T22:41:06.038829+0000","last_clean":"2024-03-18T22:41:06.038829+0000","last_became_active":"2024-03-18T22:40:41.217587+0000","last_became_peered":"2024-03-18T22:40:41.217587+0000","last_unstale":"2024-03-18T22:41:06.038829+0000","last_undegraded":"2024-03-18T22:41:06.038829+0000","last_fullsized":"2024-03-18T22:41:06.038829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:36.524954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843987+0000","last_change":"2024-03-18T22:40:41.812026+0000","last_active":"2024-03-18T22:40:46.843987+0000","last_peered":"2024-03-18T22:40:46.843987+0000","last_clean":"2024-03-18T22:40:46.843987+0000","last_became_active":"2024-03-18T22:40:41.811668+0000","last_became_peered":"2024-03-18T22:40:41.811668+0000","last_unstale":"2024-03-18T22:40:46.843987+0000","last_undegraded":"2024-03-18T22:40:46.843987+0000","last_fullsized":"2024-03-18T22:40:46.843987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:17:54.775327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202967+0000","last_change":"2024-03-18T22:40:45.848466+0000","last_active":"2024-03-18T22:40:47.202967+0000","last_peered":"2024-03-18T22:40:47.202967+0000","last_clean":"2024-03-18T22:40:47.202967+0000","last_became_active":"2024-03-18T22:40:45.848078+0000","last_became_peered":"2024-03-18T22:40:45.848078+0000","last_unstale":"2024-03-18T22:40:47.202967+0000","last_undegraded":"2024-03-18T22:40:47.202967+0000","last_fullsized":"2024-03-18T22:40:47.202967+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:28.515368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:08.841241+0000","last_change":"2024-03-18T22:40:41.216434+0000","last_active":"2024-03-18T22:41:08.841241+0000","last_peered":"2024-03-18T22:41:08.841241+0000","last_clean":"2024-03-18T22:41:08.841241+0000","last_became_active":"2024-03-18T22:40:41.216103+0000","last_became_peered":"2024-03-18T22:40:41.216103+0000","last_unstale":"2024-03-18T22:41:08.841241+0000","last_undegraded":"2024-03-18T22:41:08.841241+0000","last_fullsized":"2024-03-18T22:41:08.841241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:39.516463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843591+0000","last_change":"2024-03-18T22:40:41.808405+0000","last_active":"2024-03-18T22:40:46.843591+0000","last_peered":"2024-03-18T22:40:46.843591+0000","last_clean":"2024-03-18T22:40:46.843591+0000","last_became_active":"2024-03-18T22:40:41.808111+0000","last_became_peered":"2024-03-18T22:40:41.808111+0000","last_unstale":"2024-03-18T22:40:46.843591+0000","last_undegraded":"2024-03-18T22:40:46.843591+0000","last_fullsized":"2024-03-18T22:40:46.843591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:37:20.808163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202091+0000","last_change":"2024-03-18T22:40:45.848409+0000","last_active":"2024-03-18T22:40:47.202091+0000","last_peered":"2024-03-18T22:40:47.202091+0000","last_clean":"2024-03-18T22:40:47.202091+0000","last_became_active":"2024-03-18T22:40:45.847687+0000","last_became_peered":"2024-03-18T22:40:45.847687+0000","last_unstale":"2024-03-18T22:40:47.202091+0000","last_undegraded":"2024-03-18T22:40:47.202091+0000","last_fullsized":"2024-03-18T22:40:47.202091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:32.894885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.046173+0000","last_change":"2024-03-18T22:40:41.208408+0000","last_active":"2024-03-18T22:41:07.046173+0000","last_peered":"2024-03-18T22:41:07.046173+0000","last_clean":"2024-03-18T22:41:07.046173+0000","last_became_active":"2024-03-18T22:40:41.207976+0000","last_became_peered":"2024-03-18T22:40:41.207976+0000","last_unstale":"2024-03-18T22:41:07.046173+0000","last_undegraded":"2024-03-18T22:41:07.046173+0000","last_fullsized":"2024-03-18T22:41:07.046173+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:43:24.416974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843529+0000","last_change":"2024-03-18T22:40:41.813547+0000","last_active":"2024-03-18T22:40:46.843529+0000","last_peered":"2024-03-18T22:40:46.843529+0000","last_clean":"2024-03-18T22:40:46.843529+0000","last_became_active":"2024-03-18T22:40:41.813250+0000","last_became_peered":"2024-03-18T22:40:41.813250+0000","last_unstale":"2024-03-18T22:40:46.843529+0000","last_undegraded":"2024-03-18T22:40:46.843529+0000","last_fullsized":"2024-03-18T22:40:46.843529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:54.428485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844337+0000","last_change":"2024-03-18T22:40:45.845797+0000","last_active":"2024-03-18T22:40:46.844337+0000","last_peered":"2024-03-18T22:40:46.844337+0000","last_clean":"2024-03-18T22:40:46.844337+0000","last_became_active":"2024-03-18T22:40:45.845324+0000","last_became_peered":"2024-03-18T22:40:45.845324+0000","last_unstale":"2024-03-18T22:40:46.844337+0000","last_undegraded":"2024-03-18T22:40:46.844337+0000","last_fullsized":"2024-03-18T22:40:46.844337+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:21.613203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'10","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.041222+0000","last_change":"2024-03-18T22:40:41.222127+0000","last_active":"2024-03-18T22:41:06.041222+0000","last_peered":"2024-03-18T22:41:06.041222+0000","last_clean":"2024-03-18T22:41:06.041222+0000","last_became_active":"2024-03-18T22:40:41.221830+0000","last_became_peered":"2024-03-18T22:40:41.221830+0000","last_unstale":"2024-03-18T22:41:06.041222+0000","last_undegraded":"2024-03-18T22:41:06.041222+0000","last_fullsized":"2024-03-18T22:41:06.041222+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:24:02.235914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203971+0000","last_change":"2024-03-18T22:40:41.814606+0000","last_active":"2024-03-18T22:40:47.203971+0000","last_peered":"2024-03-18T22:40:47.203971+0000","last_clean":"2024-03-18T22:40:47.203971+0000","last_became_active":"2024-03-18T22:40:41.814309+0000","last_became_peered":"2024-03-18T22:40:41.814309+0000","last_unstale":"2024-03-18T22:40:47.203971+0000","last_undegraded":"2024-03-18T22:40:47.203971+0000","last_fullsized":"2024-03-18T22:40:47.203971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:33.086229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.204356+0000","last_change":"2024-03-18T22:40:45.845711+0000","last_active":"2024-03-18T22:40:47.204356+0000","last_peered":"2024-03-18T22:40:47.204356+0000","last_clean":"2024-03-18T22:40:47.204356+0000","last_became_active":"2024-03-18T22:40:45.845094+0000","last_became_peered":"2024-03-18T22:40:45.845094+0000","last_unstale":"2024-03-18T22:40:47.204356+0000","last_undegraded":"2024-03-18T22:40:47.204356+0000","last_fullsized":"2024-03-18T22:40:47.204356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:08.372875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202536+0000","last_change":"2024-03-18T22:40:41.219076+0000","last_active":"2024-03-18T22:40:47.202536+0000","last_peered":"2024-03-18T22:40:47.202536+0000","last_clean":"2024-03-18T22:40:47.202536+0000","last_became_active":"2024-03-18T22:40:41.217580+0000","last_became_peered":"2024-03-18T22:40:41.217580+0000","last_unstale":"2024-03-18T22:40:47.202536+0000","last_undegraded":"2024-03-18T22:40:47.202536+0000","last_fullsized":"2024-03-18T22:40:47.202536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:38.112325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203173+0000","last_change":"2024-03-18T22:40:41.810267+0000","last_active":"2024-03-18T22:40:47.203173+0000","last_peered":"2024-03-18T22:40:47.203173+0000","last_clean":"2024-03-18T22:40:47.203173+0000","last_became_active":"2024-03-18T22:40:41.809960+0000","last_became_peered":"2024-03-18T22:40:41.809960+0000","last_unstale":"2024-03-18T22:40:47.203173+0000","last_undegraded":"2024-03-18T22:40:47.203173+0000","last_fullsized":"2024-03-18T22:40:47.203173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:00.973817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202789+0000","last_change":"2024-03-18T22:40:45.848461+0000","last_active":"2024-03-18T22:40:47.202789+0000","last_peered":"2024-03-18T22:40:47.202789+0000","last_clean":"2024-03-18T22:40:47.202789+0000","last_became_active":"2024-03-18T22:40:45.848043+0000","last_became_peered":"2024-03-18T22:40:45.848043+0000","last_unstale":"2024-03-18T22:40:47.202789+0000","last_undegraded":"2024-03-18T22:40:47.202789+0000","last_fullsized":"2024-03-18T22:40:47.202789+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:38.865565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202161+0000","last_change":"2024-03-18T22:40:41.213069+0000","last_active":"2024-03-18T22:40:47.202161+0000","last_peered":"2024-03-18T22:40:47.202161+0000","last_clean":"2024-03-18T22:40:47.202161+0000","last_became_active":"2024-03-18T22:40:41.212725+0000","last_became_peered":"2024-03-18T22:40:41.212725+0000","last_unstale":"2024-03-18T22:40:47.202161+0000","last_undegraded":"2024-03-18T22:40:47.202161+0000","last_fullsized":"2024-03-18T22:40:47.202161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:50.219433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201736+0000","last_change":"2024-03-18T22:40:41.808343+0000","last_active":"2024-03-18T22:40:47.201736+0000","last_peered":"2024-03-18T22:40:47.201736+0000","last_clean":"2024-03-18T22:40:47.201736+0000","last_became_active":"2024-03-18T22:40:41.808046+0000","last_became_peered":"2024-03-18T22:40:41.808046+0000","last_unstale":"2024-03-18T22:40:47.201736+0000","last_undegraded":"2024-03-18T22:40:47.201736+0000","last_fullsized":"2024-03-18T22:40:47.201736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:09.357992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844268+0000","last_change":"2024-03-18T22:40:45.840058+0000","last_active":"2024-03-18T22:40:46.844268+0000","last_peered":"2024-03-18T22:40:46.844268+0000","last_clean":"2024-03-18T22:40:46.844268+0000","last_became_active":"2024-03-18T22:40:45.839661+0000","last_became_peered":"2024-03-18T22:40:45.839661+0000","last_unstale":"2024-03-18T22:40:46.844268+0000","last_undegraded":"2024-03-18T22:40:46.844268+0000","last_fullsized":"2024-03-18T22:40:46.844268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:35:14.323972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843724+0000","last_change":"2024-03-18T22:40:41.215988+0000","last_active":"2024-03-18T22:40:46.843724+0000","last_peered":"2024-03-18T22:40:46.843724+0000","last_clean":"2024-03-18T22:40:46.843724+0000","last_became_active":"2024-03-18T22:40:41.215680+0000","last_became_peered":"2024-03-18T22:40:41.215680+0000","last_unstale":"2024-03-18T22:40:46.843724+0000","last_undegraded":"2024-03-18T22:40:46.843724+0000","last_fullsized":"2024-03-18T22:40:46.843724+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:43.412362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201944+0000","last_change":"2024-03-18T22:40:41.812521+0000","last_active":"2024-03-18T22:40:47.201944+0000","last_peered":"2024-03-18T22:40:47.201944+0000","last_clean":"2024-03-18T22:40:47.201944+0000","last_became_active":"2024-03-18T22:40:41.812223+0000","last_became_peered":"2024-03-18T22:40:41.812223+0000","last_unstale":"2024-03-18T22:40:47.201944+0000","last_undegraded":"2024-03-18T22:40:47.201944+0000","last_fullsized":"2024-03-18T22:40:47.201944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:35:37.945144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844055+0000","last_change":"2024-03-18T22:40:45.838372+0000","last_active":"2024-03-18T22:40:46.844055+0000","last_peered":"2024-03-18T22:40:46.844055+0000","last_clean":"2024-03-18T22:40:46.844055+0000","last_became_active":"2024-03-18T22:40:45.837962+0000","last_became_peered":"2024-03-18T22:40:45.837962+0000","last_unstale":"2024-03-18T22:40:46.844055+0000","last_undegraded":"2024-03-18T22:40:46.844055+0000","last_fullsized":"2024-03-18T22:40:46.844055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:18:39.323023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203367+0000","last_change":"2024-03-18T22:40:41.812146+0000","last_active":"2024-03-18T22:40:47.203367+0000","last_peered":"2024-03-18T22:40:47.203367+0000","last_clean":"2024-03-18T22:40:47.203367+0000","last_became_active":"2024-03-18T22:40:41.811840+0000","last_became_peered":"2024-03-18T22:40:41.811840+0000","last_unstale":"2024-03-18T22:40:47.203367+0000","last_undegraded":"2024-03-18T22:40:47.203367+0000","last_fullsized":"2024-03-18T22:40:47.203367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:30.829187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843766+0000","last_change":"2024-03-18T22:40:41.212133+0000","last_active":"2024-03-18T22:40:46.843766+0000","last_peered":"2024-03-18T22:40:46.843766+0000","last_clean":"2024-03-18T22:40:46.843766+0000","last_became_active":"2024-03-18T22:40:41.211581+0000","last_became_peered":"2024-03-18T22:40:41.211581+0000","last_unstale":"2024-03-18T22:40:46.843766+0000","last_undegraded":"2024-03-18T22:40:46.843766+0000","last_fullsized":"2024-03-18T22:40:46.843766+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:44.287725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843870+0000","last_change":"2024-03-18T22:40:45.843660+0000","last_active":"2024-03-18T22:40:46.843870+0000","last_peered":"2024-03-18T22:40:46.843870+0000","last_clean":"2024-03-18T22:40:46.843870+0000","last_became_active":"2024-03-18T22:40:45.843233+0000","last_became_peered":"2024-03-18T22:40:45.843233+0000","last_unstale":"2024-03-18T22:40:46.843870+0000","last_undegraded":"2024-03-18T22:40:46.843870+0000","last_fullsized":"2024-03-18T22:40:46.843870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:08:19.758496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201521+0000","last_change":"2024-03-18T22:40:41.811798+0000","last_active":"2024-03-18T22:40:47.201521+0000","last_peered":"2024-03-18T22:40:47.201521+0000","last_clean":"2024-03-18T22:40:47.201521+0000","last_became_active":"2024-03-18T22:40:41.811422+0000","last_became_peered":"2024-03-18T22:40:41.811422+0000","last_unstale":"2024-03-18T22:40:47.201521+0000","last_undegraded":"2024-03-18T22:40:47.201521+0000","last_fullsized":"2024-03-18T22:40:47.201521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:53.944556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.039621+0000","last_change":"2024-03-18T22:40:41.210052+0000","last_active":"2024-03-18T22:41:06.039621+0000","last_peered":"2024-03-18T22:41:06.039621+0000","last_clean":"2024-03-18T22:41:06.039621+0000","last_became_active":"2024-03-18T22:40:41.209647+0000","last_became_peered":"2024-03-18T22:40:41.209647+0000","last_unstale":"2024-03-18T22:41:06.039621+0000","last_undegraded":"2024-03-18T22:41:06.039621+0000","last_fullsized":"2024-03-18T22:41:06.039621+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:57:35.187348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843338+0000","last_change":"2024-03-18T22:39:32.831758+0000","last_active":"2024-03-18T22:40:46.843338+0000","last_peered":"2024-03-18T22:40:46.843338+0000","last_clean":"2024-03-18T22:40:46.843338+0000","last_became_active":"2024-03-18T22:39:32.831287+0000","last_became_peered":"2024-03-18T22:39:32.831287+0000","last_unstale":"2024-03-18T22:40:46.843338+0000","last_undegraded":"2024-03-18T22:40:46.843338+0000","last_fullsized":"2024-03-18T22:40:46.843338+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:36:07.806854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:36:07.806854+0000","last_clean_scrub_stamp":"2024-03-18T22:36:07.806854+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:42.449107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.204239+0000","last_change":"2024-03-18T22:40:45.842008+0000","last_active":"2024-03-18T22:40:47.204239+0000","last_peered":"2024-03-18T22:40:47.204239+0000","last_clean":"2024-03-18T22:40:47.204239+0000","last_became_active":"2024-03-18T22:40:45.841231+0000","last_became_peered":"2024-03-18T22:40:45.841231+0000","last_unstale":"2024-03-18T22:40:47.204239+0000","last_undegraded":"2024-03-18T22:40:47.204239+0000","last_fullsized":"2024-03-18T22:40:47.204239+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:06.812049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843850+0000","last_change":"2024-03-18T22:40:41.807216+0000","last_active":"2024-03-18T22:40:46.843850+0000","last_peered":"2024-03-18T22:40:46.843850+0000","last_clean":"2024-03-18T22:40:46.843850+0000","last_became_active":"2024-03-18T22:40:41.806797+0000","last_became_peered":"2024-03-18T22:40:41.806797+0000","last_unstale":"2024-03-18T22:40:46.843850+0000","last_undegraded":"2024-03-18T22:40:46.843850+0000","last_fullsized":"2024-03-18T22:40:46.843850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:26.422329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.047645+0000","last_change":"2024-03-18T22:40:41.216555+0000","last_active":"2024-03-18T22:41:07.047645+0000","last_peered":"2024-03-18T22:41:07.047645+0000","last_clean":"2024-03-18T22:41:07.047645+0000","last_became_active":"2024-03-18T22:40:41.216240+0000","last_became_peered":"2024-03-18T22:40:41.216240+0000","last_unstale":"2024-03-18T22:41:07.047645+0000","last_undegraded":"2024-03-18T22:41:07.047645+0000","last_fullsized":"2024-03-18T22:41:07.047645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:04.655182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202780+0000","last_change":"2024-03-18T22:40:45.839889+0000","last_active":"2024-03-18T22:40:47.202780+0000","last_peered":"2024-03-18T22:40:47.202780+0000","last_clean":"2024-03-18T22:40:47.202780+0000","last_became_active":"2024-03-18T22:40:45.839414+0000","last_became_peered":"2024-03-18T22:40:45.839414+0000","last_unstale":"2024-03-18T22:40:47.202780+0000","last_undegraded":"2024-03-18T22:40:47.202780+0000","last_fullsized":"2024-03-18T22:40:47.202780+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:52.497378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202718+0000","last_change":"2024-03-18T22:40:41.221026+0000","last_active":"2024-03-18T22:40:47.202718+0000","last_peered":"2024-03-18T22:40:47.202718+0000","last_clean":"2024-03-18T22:40:47.202718+0000","last_became_active":"2024-03-18T22:40:41.220681+0000","last_became_peered":"2024-03-18T22:40:41.220681+0000","last_unstale":"2024-03-18T22:40:47.202718+0000","last_undegraded":"2024-03-18T22:40:47.202718+0000","last_fullsized":"2024-03-18T22:40:47.202718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:11.490445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844508+0000","last_change":"2024-03-18T22:40:41.810500+0000","last_active":"2024-03-18T22:40:46.844508+0000","last_peered":"2024-03-18T22:40:46.844508+0000","last_clean":"2024-03-18T22:40:46.844508+0000","last_became_active":"2024-03-18T22:40:41.809723+0000","last_became_peered":"2024-03-18T22:40:41.809723+0000","last_unstale":"2024-03-18T22:40:46.844508+0000","last_undegraded":"2024-03-18T22:40:46.844508+0000","last_fullsized":"2024-03-18T22:40:46.844508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:18:37.053809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202465+0000","last_change":"2024-03-18T22:40:45.844323+0000","last_active":"2024-03-18T22:40:47.202465+0000","last_peered":"2024-03-18T22:40:47.202465+0000","last_clean":"2024-03-18T22:40:47.202465+0000","last_became_active":"2024-03-18T22:40:45.843146+0000","last_became_peered":"2024-03-18T22:40:45.843146+0000","last_unstale":"2024-03-18T22:40:47.202465+0000","last_undegraded":"2024-03-18T22:40:47.202465+0000","last_fullsized":"2024-03-18T22:40:47.202465+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:14:59.881467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.833378+0000","last_change":"2024-03-18T22:40:41.212126+0000","last_active":"2024-03-18T22:41:07.833378+0000","last_peered":"2024-03-18T22:41:07.833378+0000","last_clean":"2024-03-18T22:41:07.833378+0000","last_became_active":"2024-03-18T22:40:41.211772+0000","last_became_peered":"2024-03-18T22:40:41.211772+0000","last_unstale":"2024-03-18T22:41:07.833378+0000","last_undegraded":"2024-03-18T22:41:07.833378+0000","last_fullsized":"2024-03-18T22:41:07.833378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:03.215751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202031+0000","last_change":"2024-03-18T22:40:41.820476+0000","last_active":"2024-03-18T22:40:47.202031+0000","last_peered":"2024-03-18T22:40:47.202031+0000","last_clean":"2024-03-18T22:40:47.202031+0000","last_became_active":"2024-03-18T22:40:41.820084+0000","last_became_peered":"2024-03-18T22:40:41.820084+0000","last_unstale":"2024-03-18T22:40:47.202031+0000","last_undegraded":"2024-03-18T22:40:47.202031+0000","last_fullsized":"2024-03-18T22:40:47.202031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:51:50.580208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201852+0000","last_change":"2024-03-18T22:40:45.838588+0000","last_active":"2024-03-18T22:40:47.201852+0000","last_peered":"2024-03-18T22:40:47.201852+0000","last_clean":"2024-03-18T22:40:47.201852+0000","last_became_active":"2024-03-18T22:40:45.838068+0000","last_became_peered":"2024-03-18T22:40:45.838068+0000","last_unstale":"2024-03-18T22:40:47.201852+0000","last_undegraded":"2024-03-18T22:40:47.201852+0000","last_fullsized":"2024-03-18T22:40:47.201852+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:11:02.162668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.041019+0000","last_change":"2024-03-18T22:40:41.210108+0000","last_active":"2024-03-18T22:41:06.041019+0000","last_peered":"2024-03-18T22:41:06.041019+0000","last_clean":"2024-03-18T22:41:06.041019+0000","last_became_active":"2024-03-18T22:40:41.209781+0000","last_became_peered":"2024-03-18T22:40:41.209781+0000","last_unstale":"2024-03-18T22:41:06.041019+0000","last_undegraded":"2024-03-18T22:41:06.041019+0000","last_fullsized":"2024-03-18T22:41:06.041019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:10.670822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843171+0000","last_change":"2024-03-18T22:40:41.814637+0000","last_active":"2024-03-18T22:40:46.843171+0000","last_peered":"2024-03-18T22:40:46.843171+0000","last_clean":"2024-03-18T22:40:46.843171+0000","last_became_active":"2024-03-18T22:40:41.814279+0000","last_became_peered":"2024-03-18T22:40:41.814279+0000","last_unstale":"2024-03-18T22:40:46.843171+0000","last_undegraded":"2024-03-18T22:40:46.843171+0000","last_fullsized":"2024-03-18T22:40:46.843171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:12.496230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201586+0000","last_change":"2024-03-18T22:40:45.851560+0000","last_active":"2024-03-18T22:40:47.201586+0000","last_peered":"2024-03-18T22:40:47.201586+0000","last_clean":"2024-03-18T22:40:47.201586+0000","last_became_active":"2024-03-18T22:40:45.850884+0000","last_became_peered":"2024-03-18T22:40:45.850884+0000","last_unstale":"2024-03-18T22:40:47.201586+0000","last_undegraded":"2024-03-18T22:40:47.201586+0000","last_fullsized":"2024-03-18T22:40:47.201586+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:04.114468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.039626+0000","last_change":"2024-03-18T22:40:41.210156+0000","last_active":"2024-03-18T22:41:06.039626+0000","last_peered":"2024-03-18T22:41:06.039626+0000","last_clean":"2024-03-18T22:41:06.039626+0000","last_became_active":"2024-03-18T22:40:41.209855+0000","last_became_peered":"2024-03-18T22:40:41.209855+0000","last_unstale":"2024-03-18T22:41:06.039626+0000","last_undegraded":"2024-03-18T22:41:06.039626+0000","last_fullsized":"2024-03-18T22:41:06.039626+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:02.752573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202142+0000","last_change":"2024-03-18T22:40:41.803681+0000","last_active":"2024-03-18T22:40:47.202142+0000","last_peered":"2024-03-18T22:40:47.202142+0000","last_clean":"2024-03-18T22:40:47.202142+0000","last_became_active":"2024-03-18T22:40:41.803244+0000","last_became_peered":"2024-03-18T22:40:41.803244+0000","last_unstale":"2024-03-18T22:40:47.202142+0000","last_undegraded":"2024-03-18T22:40:47.202142+0000","last_fullsized":"2024-03-18T22:40:47.202142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:55.944575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202264+0000","last_change":"2024-03-18T22:40:45.851040+0000","last_active":"2024-03-18T22:40:47.202264+0000","last_peered":"2024-03-18T22:40:47.202264+0000","last_clean":"2024-03-18T22:40:47.202264+0000","last_became_active":"2024-03-18T22:40:45.848300+0000","last_became_peered":"2024-03-18T22:40:45.848300+0000","last_unstale":"2024-03-18T22:40:47.202264+0000","last_undegraded":"2024-03-18T22:40:47.202264+0000","last_fullsized":"2024-03-18T22:40:47.202264+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:43:10.821075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.040152+0000","last_change":"2024-03-18T22:40:41.210830+0000","last_active":"2024-03-18T22:41:06.040152+0000","last_peered":"2024-03-18T22:41:06.040152+0000","last_clean":"2024-03-18T22:41:06.040152+0000","last_became_active":"2024-03-18T22:40:41.210476+0000","last_became_peered":"2024-03-18T22:40:41.210476+0000","last_unstale":"2024-03-18T22:41:06.040152+0000","last_undegraded":"2024-03-18T22:41:06.040152+0000","last_fullsized":"2024-03-18T22:41:06.040152+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:13.243788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202078+0000","last_change":"2024-03-18T22:40:41.807678+0000","last_active":"2024-03-18T22:40:47.202078+0000","last_peered":"2024-03-18T22:40:47.202078+0000","last_clean":"2024-03-18T22:40:47.202078+0000","last_became_active":"2024-03-18T22:40:41.807383+0000","last_became_peered":"2024-03-18T22:40:41.807383+0000","last_unstale":"2024-03-18T22:40:47.202078+0000","last_undegraded":"2024-03-18T22:40:47.202078+0000","last_fullsized":"2024-03-18T22:40:47.202078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:27:55.930293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202000+0000","last_change":"2024-03-18T22:40:45.850982+0000","last_active":"2024-03-18T22:40:47.202000+0000","last_peered":"2024-03-18T22:40:47.202000+0000","last_clean":"2024-03-18T22:40:47.202000+0000","last_became_active":"2024-03-18T22:40:45.850526+0000","last_became_peered":"2024-03-18T22:40:45.850526+0000","last_unstale":"2024-03-18T22:40:47.202000+0000","last_undegraded":"2024-03-18T22:40:47.202000+0000","last_fullsized":"2024-03-18T22:40:47.202000+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:05:24.450986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201971+0000","last_change":"2024-03-18T22:40:41.219540+0000","last_active":"2024-03-18T22:40:47.201971+0000","last_peered":"2024-03-18T22:40:47.201971+0000","last_clean":"2024-03-18T22:40:47.201971+0000","last_became_active":"2024-03-18T22:40:41.219206+0000","last_became_peered":"2024-03-18T22:40:41.219206+0000","last_unstale":"2024-03-18T22:40:47.201971+0000","last_undegraded":"2024-03-18T22:40:47.201971+0000","last_fullsized":"2024-03-18T22:40:47.201971+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:33.586368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201618+0000","last_change":"2024-03-18T22:40:41.815561+0000","last_active":"2024-03-18T22:40:47.201618+0000","last_peered":"2024-03-18T22:40:47.201618+0000","last_clean":"2024-03-18T22:40:47.201618+0000","last_became_active":"2024-03-18T22:40:41.815082+0000","last_became_peered":"2024-03-18T22:40:41.815082+0000","last_unstale":"2024-03-18T22:40:47.201618+0000","last_undegraded":"2024-03-18T22:40:47.201618+0000","last_fullsized":"2024-03-18T22:40:47.201618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:01.676907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202496+0000","last_change":"2024-03-18T22:40:45.845386+0000","last_active":"2024-03-18T22:40:47.202496+0000","last_peered":"2024-03-18T22:40:47.202496+0000","last_clean":"2024-03-18T22:40:47.202496+0000","last_became_active":"2024-03-18T22:40:45.845007+0000","last_became_peered":"2024-03-18T22:40:45.845007+0000","last_unstale":"2024-03-18T22:40:47.202496+0000","last_undegraded":"2024-03-18T22:40:47.202496+0000","last_fullsized":"2024-03-18T22:40:47.202496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:41.429195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:06.041468+0000","last_change":"2024-03-18T22:40:41.215559+0000","last_active":"2024-03-18T22:41:06.041468+0000","last_peered":"2024-03-18T22:41:06.041468+0000","last_clean":"2024-03-18T22:41:06.041468+0000","last_became_active":"2024-03-18T22:40:41.215253+0000","last_became_peered":"2024-03-18T22:40:41.215253+0000","last_unstale":"2024-03-18T22:41:06.041468+0000","last_undegraded":"2024-03-18T22:41:06.041468+0000","last_fullsized":"2024-03-18T22:41:06.041468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:19.706011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843521+0000","last_change":"2024-03-18T22:40:41.807533+0000","last_active":"2024-03-18T22:40:46.843521+0000","last_peered":"2024-03-18T22:40:46.843521+0000","last_clean":"2024-03-18T22:40:46.843521+0000","last_became_active":"2024-03-18T22:40:41.807219+0000","last_became_peered":"2024-03-18T22:40:41.807219+0000","last_unstale":"2024-03-18T22:40:46.843521+0000","last_undegraded":"2024-03-18T22:40:46.843521+0000","last_fullsized":"2024-03-18T22:40:46.843521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:15:24.214367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202061+0000","last_change":"2024-03-18T22:40:45.839670+0000","last_active":"2024-03-18T22:40:47.202061+0000","last_peered":"2024-03-18T22:40:47.202061+0000","last_clean":"2024-03-18T22:40:47.202061+0000","last_became_active":"2024-03-18T22:40:45.839263+0000","last_became_peered":"2024-03-18T22:40:45.839263+0000","last_unstale":"2024-03-18T22:40:47.202061+0000","last_undegraded":"2024-03-18T22:40:47.202061+0000","last_fullsized":"2024-03-18T22:40:47.202061+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:48.991959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202445+0000","last_change":"2024-03-18T22:40:41.209617+0000","last_active":"2024-03-18T22:40:47.202445+0000","last_peered":"2024-03-18T22:40:47.202445+0000","last_clean":"2024-03-18T22:40:47.202445+0000","last_became_active":"2024-03-18T22:40:41.209311+0000","last_became_peered":"2024-03-18T22:40:41.209311+0000","last_unstale":"2024-03-18T22:40:47.202445+0000","last_undegraded":"2024-03-18T22:40:47.202445+0000","last_fullsized":"2024-03-18T22:40:47.202445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:44:06.019870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843930+0000","last_change":"2024-03-18T22:40:41.808210+0000","last_active":"2024-03-18T22:40:46.843930+0000","last_peered":"2024-03-18T22:40:46.843930+0000","last_clean":"2024-03-18T22:40:46.843930+0000","last_became_active":"2024-03-18T22:40:41.807898+0000","last_became_peered":"2024-03-18T22:40:41.807898+0000","last_unstale":"2024-03-18T22:40:46.843930+0000","last_undegraded":"2024-03-18T22:40:46.843930+0000","last_fullsized":"2024-03-18T22:40:46.843930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:20.401165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.842467+0000","last_change":"2024-03-18T22:40:45.841905+0000","last_active":"2024-03-18T22:40:46.842467+0000","last_peered":"2024-03-18T22:40:46.842467+0000","last_clean":"2024-03-18T22:40:46.842467+0000","last_became_active":"2024-03-18T22:40:45.841464+0000","last_became_peered":"2024-03-18T22:40:45.841464+0000","last_unstale":"2024-03-18T22:40:46.842467+0000","last_undegraded":"2024-03-18T22:40:46.842467+0000","last_fullsized":"2024-03-18T22:40:46.842467+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:30.435788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:41:07.832580+0000","last_change":"2024-03-18T22:40:41.216919+0000","last_active":"2024-03-18T22:41:07.832580+0000","last_peered":"2024-03-18T22:41:07.832580+0000","last_clean":"2024-03-18T22:41:07.832580+0000","last_became_active":"2024-03-18T22:40:41.216608+0000","last_became_peered":"2024-03-18T22:40:41.216608+0000","last_unstale":"2024-03-18T22:41:07.832580+0000","last_undegraded":"2024-03-18T22:41:07.832580+0000","last_fullsized":"2024-03-18T22:41:07.832580+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:10:42.230268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.843684+0000","last_change":"2024-03-18T22:40:41.813879+0000","last_active":"2024-03-18T22:40:46.843684+0000","last_peered":"2024-03-18T22:40:46.843684+0000","last_clean":"2024-03-18T22:40:46.843684+0000","last_became_active":"2024-03-18T22:40:41.813584+0000","last_became_peered":"2024-03-18T22:40:41.813584+0000","last_unstale":"2024-03-18T22:40:46.843684+0000","last_undegraded":"2024-03-18T22:40:46.843684+0000","last_fullsized":"2024-03-18T22:40:46.843684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:20:44.808087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203076+0000","last_change":"2024-03-18T22:40:45.838033+0000","last_active":"2024-03-18T22:40:47.203076+0000","last_peered":"2024-03-18T22:40:47.203076+0000","last_clean":"2024-03-18T22:40:47.203076+0000","last_became_active":"2024-03-18T22:40:45.837530+0000","last_became_peered":"2024-03-18T22:40:45.837530+0000","last_unstale":"2024-03-18T22:40:47.203076+0000","last_undegraded":"2024-03-18T22:40:47.203076+0000","last_fullsized":"2024-03-18T22:40:47.203076+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:43:43.444631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844373+0000","last_change":"2024-03-18T22:40:41.215686+0000","last_active":"2024-03-18T22:40:46.844373+0000","last_peered":"2024-03-18T22:40:46.844373+0000","last_clean":"2024-03-18T22:40:46.844373+0000","last_became_active":"2024-03-18T22:40:41.215362+0000","last_became_peered":"2024-03-18T22:40:41.215362+0000","last_unstale":"2024-03-18T22:40:46.844373+0000","last_undegraded":"2024-03-18T22:40:46.844373+0000","last_fullsized":"2024-03-18T22:40:46.844373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:46:30.546304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844074+0000","last_change":"2024-03-18T22:40:41.808959+0000","last_active":"2024-03-18T22:40:46.844074+0000","last_peered":"2024-03-18T22:40:46.844074+0000","last_clean":"2024-03-18T22:40:46.844074+0000","last_became_active":"2024-03-18T22:40:41.808671+0000","last_became_peered":"2024-03-18T22:40:41.808671+0000","last_unstale":"2024-03-18T22:40:46.844074+0000","last_undegraded":"2024-03-18T22:40:46.844074+0000","last_fullsized":"2024-03-18T22:40:46.844074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:37.115318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202207+0000","last_change":"2024-03-18T22:40:41.814240+0000","last_active":"2024-03-18T22:40:47.202207+0000","last_peered":"2024-03-18T22:40:47.202207+0000","last_clean":"2024-03-18T22:40:47.202207+0000","last_became_active":"2024-03-18T22:40:41.813925+0000","last_became_peered":"2024-03-18T22:40:41.813925+0000","last_unstale":"2024-03-18T22:40:47.202207+0000","last_undegraded":"2024-03-18T22:40:47.202207+0000","last_fullsized":"2024-03-18T22:40:47.202207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:38.999803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202253+0000","last_change":"2024-03-18T22:40:41.813381+0000","last_active":"2024-03-18T22:40:47.202253+0000","last_peered":"2024-03-18T22:40:47.202253+0000","last_clean":"2024-03-18T22:40:47.202253+0000","last_became_active":"2024-03-18T22:40:41.812927+0000","last_became_peered":"2024-03-18T22:40:41.812927+0000","last_unstale":"2024-03-18T22:40:47.202253+0000","last_undegraded":"2024-03-18T22:40:47.202253+0000","last_fullsized":"2024-03-18T22:40:47.202253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:23:09.178485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.201795+0000","last_change":"2024-03-18T22:40:41.812513+0000","last_active":"2024-03-18T22:40:47.201795+0000","last_peered":"2024-03-18T22:40:47.201795+0000","last_clean":"2024-03-18T22:40:47.201795+0000","last_became_active":"2024-03-18T22:40:41.811882+0000","last_became_peered":"2024-03-18T22:40:41.811882+0000","last_unstale":"2024-03-18T22:40:47.201795+0000","last_undegraded":"2024-03-18T22:40:47.201795+0000","last_fullsized":"2024-03-18T22:40:47.201795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:45.596911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.842621+0000","last_change":"2024-03-18T22:40:41.814960+0000","last_active":"2024-03-18T22:40:46.842621+0000","last_peered":"2024-03-18T22:40:46.842621+0000","last_clean":"2024-03-18T22:40:46.842621+0000","last_became_active":"2024-03-18T22:40:41.814620+0000","last_became_peered":"2024-03-18T22:40:41.814620+0000","last_unstale":"2024-03-18T22:40:46.842621+0000","last_undegraded":"2024-03-18T22:40:46.842621+0000","last_fullsized":"2024-03-18T22:40:46.842621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:42.205885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844492+0000","last_change":"2024-03-18T22:40:41.810023+0000","last_active":"2024-03-18T22:40:46.844492+0000","last_peered":"2024-03-18T22:40:46.844492+0000","last_clean":"2024-03-18T22:40:46.844492+0000","last_became_active":"2024-03-18T22:40:41.806027+0000","last_became_peered":"2024-03-18T22:40:41.806027+0000","last_unstale":"2024-03-18T22:40:46.844492+0000","last_undegraded":"2024-03-18T22:40:46.844492+0000","last_fullsized":"2024-03-18T22:40:46.844492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:46.472567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203520+0000","last_change":"2024-03-18T22:40:41.808718+0000","last_active":"2024-03-18T22:40:47.203520+0000","last_peered":"2024-03-18T22:40:47.203520+0000","last_clean":"2024-03-18T22:40:47.203520+0000","last_became_active":"2024-03-18T22:40:41.808379+0000","last_became_peered":"2024-03-18T22:40:41.808379+0000","last_unstale":"2024-03-18T22:40:47.203520+0000","last_undegraded":"2024-03-18T22:40:47.203520+0000","last_fullsized":"2024-03-18T22:40:47.203520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:26.637531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202815+0000","last_change":"2024-03-18T22:40:41.818046+0000","last_active":"2024-03-18T22:40:47.202815+0000","last_peered":"2024-03-18T22:40:47.202815+0000","last_clean":"2024-03-18T22:40:47.202815+0000","last_became_active":"2024-03-18T22:40:41.817698+0000","last_became_peered":"2024-03-18T22:40:41.817698+0000","last_unstale":"2024-03-18T22:40:47.202815+0000","last_undegraded":"2024-03-18T22:40:47.202815+0000","last_fullsized":"2024-03-18T22:40:47.202815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:22.967673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202222+0000","last_change":"2024-03-18T22:40:41.819337+0000","last_active":"2024-03-18T22:40:47.202222+0000","last_peered":"2024-03-18T22:40:47.202222+0000","last_clean":"2024-03-18T22:40:47.202222+0000","last_became_active":"2024-03-18T22:40:41.818987+0000","last_became_peered":"2024-03-18T22:40:41.818987+0000","last_unstale":"2024-03-18T22:40:47.202222+0000","last_undegraded":"2024-03-18T22:40:47.202222+0000","last_fullsized":"2024-03-18T22:40:47.202222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:07:04.529454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203272+0000","last_change":"2024-03-18T22:40:41.813262+0000","last_active":"2024-03-18T22:40:47.203272+0000","last_peered":"2024-03-18T22:40:47.203272+0000","last_clean":"2024-03-18T22:40:47.203272+0000","last_became_active":"2024-03-18T22:40:41.812956+0000","last_became_peered":"2024-03-18T22:40:41.812956+0000","last_unstale":"2024-03-18T22:40:47.203272+0000","last_undegraded":"2024-03-18T22:40:47.203272+0000","last_fullsized":"2024-03-18T22:40:47.203272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:19.940097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202324+0000","last_change":"2024-03-18T22:40:41.811897+0000","last_active":"2024-03-18T22:40:47.202324+0000","last_peered":"2024-03-18T22:40:47.202324+0000","last_clean":"2024-03-18T22:40:47.202324+0000","last_became_active":"2024-03-18T22:40:41.811575+0000","last_became_peered":"2024-03-18T22:40:41.811575+0000","last_unstale":"2024-03-18T22:40:47.202324+0000","last_undegraded":"2024-03-18T22:40:47.202324+0000","last_fullsized":"2024-03-18T22:40:47.202324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:45:43.321057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.202268+0000","last_change":"2024-03-18T22:40:41.811757+0000","last_active":"2024-03-18T22:40:47.202268+0000","last_peered":"2024-03-18T22:40:47.202268+0000","last_clean":"2024-03-18T22:40:47.202268+0000","last_became_active":"2024-03-18T22:40:41.811405+0000","last_became_peered":"2024-03-18T22:40:41.811405+0000","last_unstale":"2024-03-18T22:40:47.202268+0000","last_undegraded":"2024-03-18T22:40:47.202268+0000","last_fullsized":"2024-03-18T22:40:47.202268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:39.224604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:46.844157+0000","last_change":"2024-03-18T22:40:41.809987+0000","last_active":"2024-03-18T22:40:46.844157+0000","last_peered":"2024-03-18T22:40:46.844157+0000","last_clean":"2024-03-18T22:40:46.844157+0000","last_became_active":"2024-03-18T22:40:41.809544+0000","last_became_peered":"2024-03-18T22:40:41.809544+0000","last_unstale":"2024-03-18T22:40:46.844157+0000","last_undegraded":"2024-03-18T22:40:46.844157+0000","last_fullsized":"2024-03-18T22:40:46.844157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:23.800949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T22:40:47.203338+0000","last_change":"2024-03-18T22:40:41.810963+0000","last_active":"2024-03-18T22:40:47.203338+0000","last_peered":"2024-03-18T22:40:47.203338+0000","last_clean":"2024-03-18T22:40:47.203338+0000","last_became_active":"2024-03-18T22:40:41.810641+0000","last_became_peered":"2024-03-18T22:40:41.810641+0000","last_unstale":"2024-03-18T22:40:47.203338+0000","last_undegraded":"2024-03-18T22:40:47.203338+0000","last_fullsized":"2024-03-18T22:40:47.203338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:37:48.872961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8213,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":132,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":27591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":478407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36932,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708220,"statfs":{"total":95995035648,"available":95957217280,"internally_reserved":0,"allocated":1634304,"data_stored":1073944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808859,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":478123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070495,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1068787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":481640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":479393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":481714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":477409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":477237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757296,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28604,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716548,"statfs":{"total":95995035648,"available":95965745152,"internally_reserved":0,"allocated":1626112,"data_stored":1074890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":6361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T22:41:12.642 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T22:41:12.643 INFO:tasks.cephadm:Setting up client nodes... 2024-03-18T22:41:12.643 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-18T22:41:12.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:12 smithi040 ceph-mon[62643]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-18T22:41:12.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:12 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1303051821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:41:12.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:12 smithi040 ceph-mon[62643]: from='client.24791 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:41:12.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:12 smithi040 ceph-mon[62643]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:41:12.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:12 smithi040 ceph-mon[62643]: pgmap v304: 129 pgs: 129 active+clean; 588 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 4.4 KiB/s wr, 18 op/s 2024-03-18T22:41:12.924 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:13.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:12 smithi103 ceph-mon[66377]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-18T22:41:13.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:12 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1303051821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:41:13.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:12 smithi103 ceph-mon[66377]: from='client.24791 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:41:13.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:12 smithi103 ceph-mon[66377]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:41:13.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:12 smithi103 ceph-mon[66377]: pgmap v304: 129 pgs: 129 active+clean; 588 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 4.4 KiB/s wr, 18 op/s 2024-03-18T22:41:13.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:12 smithi113 ceph-mon[66775]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-18T22:41:13.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:12 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1303051821' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:41:13.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:12 smithi113 ceph-mon[66775]: from='client.24791 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:41:13.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:12 smithi113 ceph-mon[66775]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:41:13.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:12 smithi113 ceph-mon[66775]: pgmap v304: 129 pgs: 129 active+clean; 588 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 4.4 KiB/s wr, 18 op/s 2024-03-18T22:41:14.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.696+0000 7f7ed8ea7700 1 -- 172.21.15.40:0/322210218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed4102360 msgr2=0x7f7ed4102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:14.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.696+0000 7f7ed8ea7700 1 --2- 172.21.15.40:0/322210218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed4102360 0x7f7ed4102780 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40099c0 tx=0x7f7ec4009cd0 comp rx=0 tx=0).stop 2024-03-18T22:41:14.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.697+0000 7f7ed8ea7700 1 -- 172.21.15.40:0/322210218 shutdown_connections 2024-03-18T22:41:14.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.697+0000 7f7ed8ea7700 1 --2- 172.21.15.40:0/322210218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ed4103f00 0x7f7ed410a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:14.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.697+0000 7f7ed8ea7700 1 --2- 172.21.15.40:0/322210218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed4102360 0x7f7ed4102780 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:14.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.697+0000 7f7ed8ea7700 1 --2- 172.21.15.40:0/322210218 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed4103560 0x7f7ed41039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.697+0000 7f7ed8ea7700 1 -- 172.21.15.40:0/322210218 >> 172.21.15.40:0/322210218 conn(0x7f7ed40fd920 msgr2=0x7f7ed40ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.697+0000 7f7ed8ea7700 1 -- 172.21.15.40:0/322210218 shutdown_connections 2024-03-18T22:41:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.697+0000 7f7ed8ea7700 1 -- 172.21.15.40:0/322210218 wait complete. 2024-03-18T22:41:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.697+0000 7f7ed8ea7700 1 Processor -- start 2024-03-18T22:41:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.697+0000 7f7ed8ea7700 1 -- start start 2024-03-18T22:41:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed8ea7700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ed4102360 0x7f7ed4198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed8ea7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed4103560 0x7f7ed4198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed8ea7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed4103f00 0x7f7ed419fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed8ea7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed410de40 con 0x7f7ed4103560 2024-03-18T22:41:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed8ea7700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed410dcc0 con 0x7f7ed4102360 2024-03-18T22:41:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed8ea7700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed410dfc0 con 0x7f7ed4103f00 2024-03-18T22:41:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed1d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed4103560 0x7f7ed4198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ed4102360 0x7f7ed4198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed2d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed4103f00 0x7f7ed419fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:14.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed259c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ed4102360 0x7f7ed4198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:41110/0 (socket says 172.21.15.40:41110) 2024-03-18T22:41:14.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed1d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed4103560 0x7f7ed4198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40832/0 (socket says 172.21.15.40:40832) 2024-03-18T22:41:14.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed2d9d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed4103f00 0x7f7ed419fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:49828/0 (socket says 172.21.15.40:49828) 2024-03-18T22:41:14.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed259c700 1 -- 172.21.15.40:0/3787200604 learned_addr learned my addr 172.21.15.40:0/3787200604 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:14.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed259c700 1 -- 172.21.15.40:0/3787200604 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed4103f00 msgr2=0x7f7ed419fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:14.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed259c700 1 --2- 172.21.15.40:0/3787200604 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed4103f00 0x7f7ed419fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:14.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed259c700 1 -- 172.21.15.40:0/3787200604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed4103560 msgr2=0x7f7ed4198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:14.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed259c700 1 --2- 172.21.15.40:0/3787200604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed4103560 0x7f7ed4198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:14.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed259c700 1 -- 172.21.15.40:0/3787200604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed41a00b0 con 0x7f7ed4102360 2024-03-18T22:41:14.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed1d9b700 1 --2- 172.21.15.40:0/3787200604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed4103560 0x7f7ed4198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:14.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed2d9d700 1 --2- 172.21.15.40:0/3787200604 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed4103f00 0x7f7ed419fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:14.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ed259c700 1 --2- 172.21.15.40:0/3787200604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ed4102360 0x7f7ed4198450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4009990 tx=0x7f7ec400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:14.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.698+0000 7f7ec37fe700 1 -- 172.21.15.40:0/3787200604 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4009e60 con 0x7f7ed4102360 2024-03-18T22:41:14.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.699+0000 7f7ec37fe700 1 -- 172.21.15.40:0/3787200604 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ec40038c0 con 0x7f7ed4102360 2024-03-18T22:41:14.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.699+0000 7f7ed8ea7700 1 -- 172.21.15.40:0/3787200604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed41a0340 con 0x7f7ed4102360 2024-03-18T22:41:14.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.699+0000 7f7ec37fe700 1 -- 172.21.15.40:0/3787200604 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4005600 con 0x7f7ed4102360 2024-03-18T22:41:14.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.699+0000 7f7ed8ea7700 1 -- 172.21.15.40:0/3787200604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed41a0790 con 0x7f7ed4102360 2024-03-18T22:41:14.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.699+0000 7f7ed8ea7700 1 -- 172.21.15.40:0/3787200604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ed4050820 con 0x7f7ed4102360 2024-03-18T22:41:14.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.701+0000 7f7ec37fe700 1 -- 172.21.15.40:0/3787200604 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7ec4003a70 con 0x7f7ed4102360 2024-03-18T22:41:14.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.702+0000 7f7ec37fe700 1 --2- 172.21.15.40:0/3787200604 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ebc0a29d0 0x7f7ebc0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:14.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.702+0000 7f7ed1d9b700 1 --2- 172.21.15.40:0/3787200604 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ebc0a29d0 0x7f7ebc0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:14.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.702+0000 7f7ec37fe700 1 -- 172.21.15.40:0/3787200604 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7ec40cfed0 con 0x7f7ed4102360 2024-03-18T22:41:14.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.702+0000 7f7ed1d9b700 1 --2- 172.21.15.40:0/3787200604 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ebc0a29d0 0x7f7ebc0a4e90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8011490 tx=0x7f7ec8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:14.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.710+0000 7f7ec37fe700 1 -- 172.21.15.40:0/3787200604 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7ec4013070 con 0x7f7ed4102360 2024-03-18T22:41:14.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:14.997+0000 7f7ed8ea7700 1 -- 172.21.15.40:0/3787200604 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f7ed402dbb0 con 0x7f7ed4102360 2024-03-18T22:41:14.998 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:14 smithi040 ceph-mon[62643]: pgmap v305: 129 pgs: 129 active+clean; 629 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.5 KiB/s wr, 22 op/s 2024-03-18T22:41:15.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:15.002+0000 7f7ec37fe700 1 -- 172.21.15.40:0/3787200604 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f7ec409cc60 con 0x7f7ed4102360 2024-03-18T22:41:15.002 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-03-18T22:41:15.003 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAKw/hlGg+POxAAjZoyd1enoA3u+nkutiVSZg== 2024-03-18T22:41:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:15.004+0000 7f7ed8ea7700 1 -- 172.21.15.40:0/3787200604 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ebc0a29d0 msgr2=0x7f7ebc0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:15.004+0000 7f7ed8ea7700 1 --2- 172.21.15.40:0/3787200604 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ebc0a29d0 0x7f7ebc0a4e90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8011490 tx=0x7f7ec8009260 comp rx=0 tx=0).stop 2024-03-18T22:41:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:15.004+0000 7f7ed8ea7700 1 -- 172.21.15.40:0/3787200604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ed4102360 msgr2=0x7f7ed4198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:15.004+0000 7f7ed8ea7700 1 --2- 172.21.15.40:0/3787200604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ed4102360 0x7f7ed4198450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4009990 tx=0x7f7ec400f690 comp rx=0 tx=0).stop 2024-03-18T22:41:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:15.004+0000 7f7ed8ea7700 1 -- 172.21.15.40:0/3787200604 shutdown_connections 2024-03-18T22:41:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:15.004+0000 7f7ed8ea7700 1 --2- 172.21.15.40:0/3787200604 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ed4103f00 0x7f7ed419fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:15.004+0000 7f7ed8ea7700 1 --2- 172.21.15.40:0/3787200604 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ebc0a29d0 0x7f7ebc0a4e90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:15.004+0000 7f7ed8ea7700 1 --2- 172.21.15.40:0/3787200604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ed4102360 0x7f7ed4198450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:15.004+0000 7f7ed8ea7700 1 --2- 172.21.15.40:0/3787200604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ed4103560 0x7f7ed4198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:15.004+0000 7f7ed8ea7700 1 -- 172.21.15.40:0/3787200604 >> 172.21.15.40:0/3787200604 conn(0x7f7ed40fd920 msgr2=0x7f7ed41068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:15.004+0000 7f7ed8ea7700 1 -- 172.21.15.40:0/3787200604 shutdown_connections 2024-03-18T22:41:15.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:15.004+0000 7f7ed8ea7700 1 -- 172.21.15.40:0/3787200604 wait complete. 2024-03-18T22:41:15.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:14 smithi113 ceph-mon[66775]: pgmap v305: 129 pgs: 129 active+clean; 629 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.5 KiB/s wr, 22 op/s 2024-03-18T22:41:15.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:14 smithi103 ceph-mon[66377]: pgmap v305: 129 pgs: 129 active+clean; 629 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.5 KiB/s wr, 22 op/s 2024-03-18T22:41:15.729 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:41:15.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-18T22:41:15.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-18T22:41:15.772 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-18T22:41:15.772 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-18T22:41:15.772 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-18T22:41:16.060 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:16.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:15 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3787200604' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T22:41:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:15 smithi040 ceph-mon[62643]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T22:41:16.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:15 smithi040 ceph-mon[62643]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T22:41:16.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:15 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3787200604' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T22:41:16.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:15 smithi113 ceph-mon[66775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T22:41:16.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:15 smithi113 ceph-mon[66775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T22:41:16.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:15 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3787200604' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T22:41:16.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:15 smithi103 ceph-mon[66377]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T22:41:16.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:15 smithi103 ceph-mon[66377]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T22:41:17.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:16 smithi040 ceph-mon[62643]: pgmap v306: 129 pgs: 129 active+clean; 629 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.5 KiB/s wr, 23 op/s 2024-03-18T22:41:17.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:16 smithi113 ceph-mon[66775]: pgmap v306: 129 pgs: 129 active+clean; 629 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.5 KiB/s wr, 23 op/s 2024-03-18T22:41:17.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:16 smithi103 ceph-mon[66377]: pgmap v306: 129 pgs: 129 active+clean; 629 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.5 KiB/s wr, 23 op/s 2024-03-18T22:41:17.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.833+0000 7fdd798f5700 1 -- 172.21.15.40:0/1367921644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd7406aab0 msgr2=0x7fdd7406ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:17.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.833+0000 7fdd798f5700 1 --2- 172.21.15.40:0/1367921644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd7406aab0 0x7fdd7406ae90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdd640099c0 tx=0x7fdd64009cd0 comp rx=0 tx=0).stop 2024-03-18T22:41:17.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.834+0000 7fdd798f5700 1 -- 172.21.15.40:0/1367921644 shutdown_connections 2024-03-18T22:41:17.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.834+0000 7fdd798f5700 1 --2- 172.21.15.40:0/1367921644 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd74107690 0x7fdd7410f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:17.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.834+0000 7fdd798f5700 1 --2- 172.21.15.40:0/1367921644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd7406aab0 0x7fdd7406ae90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:17.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.834+0000 7fdd798f5700 1 --2- 172.21.15.40:0/1367921644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7406b460 0x7fdd74106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:17.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.834+0000 7fdd798f5700 1 -- 172.21.15.40:0/1367921644 >> 172.21.15.40:0/1367921644 conn(0x7fdd74077260 msgr2=0x7fdd74077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:17.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.834+0000 7fdd798f5700 1 -- 172.21.15.40:0/1367921644 shutdown_connections 2024-03-18T22:41:17.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.834+0000 7fdd798f5700 1 -- 172.21.15.40:0/1367921644 wait complete. 2024-03-18T22:41:17.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.834+0000 7fdd798f5700 1 Processor -- start 2024-03-18T22:41:17.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.834+0000 7fdd798f5700 1 -- start start 2024-03-18T22:41:17.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd798f5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7406aab0 0x7fdd7419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:17.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd798f5700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd7406b460 0x7fdd7419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:17.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd798f5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd74107690 0x7fdd74198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:17.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd798f5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdd741124d0 con 0x7fdd74107690 2024-03-18T22:41:17.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd798f5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdd74112350 con 0x7fdd7406aab0 2024-03-18T22:41:17.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd798f5700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdd74112650 con 0x7fdd7406b460 2024-03-18T22:41:17.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd727fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd7406b460 0x7fdd7419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:17.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd72ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7406aab0 0x7fdd7419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:17.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd737fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd74107690 0x7fdd74198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:17.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd727fc700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd7406b460 0x7fdd7419ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:49852/0 (socket says 172.21.15.40:49852) 2024-03-18T22:41:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd72ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7406aab0 0x7fdd7419e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:41122/0 (socket says 172.21.15.40:41122) 2024-03-18T22:41:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd727fc700 1 -- 172.21.15.40:0/612813615 learned_addr learned my addr 172.21.15.40:0/612813615 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd727fc700 1 -- 172.21.15.40:0/612813615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7406aab0 msgr2=0x7fdd7419e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd727fc700 1 --2- 172.21.15.40:0/612813615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7406aab0 0x7fdd7419e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd727fc700 1 -- 172.21.15.40:0/612813615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd74107690 msgr2=0x7fdd74198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd727fc700 1 --2- 172.21.15.40:0/612813615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd74107690 0x7fdd74198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd727fc700 1 -- 172.21.15.40:0/612813615 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd74199070 con 0x7fdd7406b460 2024-03-18T22:41:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.835+0000 7fdd727fc700 1 --2- 172.21.15.40:0/612813615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd7406b460 0x7fdd7419ec00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdd6800e940 tx=0x7fdd6800ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.836+0000 7fdd788f3700 1 -- 172.21.15.40:0/612813615 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6800c9c0 con 0x7fdd7406b460 2024-03-18T22:41:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.836+0000 7fdd798f5700 1 -- 172.21.15.40:0/612813615 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd74199360 con 0x7fdd7406b460 2024-03-18T22:41:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.836+0000 7fdd788f3700 1 -- 172.21.15.40:0/612813615 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd6800cb60 con 0x7fdd7406b460 2024-03-18T22:41:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.836+0000 7fdd788f3700 1 -- 172.21.15.40:0/612813615 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd68018680 con 0x7fdd7406b460 2024-03-18T22:41:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.836+0000 7fdd798f5700 1 -- 172.21.15.40:0/612813615 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd741a5520 con 0x7fdd7406b460 2024-03-18T22:41:17.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.836+0000 7fdd798f5700 1 -- 172.21.15.40:0/612813615 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd7410a860 con 0x7fdd7406b460 2024-03-18T22:41:17.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.838+0000 7fdd788f3700 1 -- 172.21.15.40:0/612813615 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fdd6800cd10 con 0x7fdd7406b460 2024-03-18T22:41:17.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.839+0000 7fdd788f3700 1 --2- 172.21.15.40:0/612813615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdd5c0a2920 0x7fdd5c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:17.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.839+0000 7fdd72ffd700 1 --2- 172.21.15.40:0/612813615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdd5c0a2920 0x7fdd5c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:17.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.839+0000 7fdd788f3700 1 -- 172.21.15.40:0/612813615 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdd680cf2c0 con 0x7fdd7406b460 2024-03-18T22:41:17.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.839+0000 7fdd72ffd700 1 --2- 172.21.15.40:0/612813615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdd5c0a2920 0x7fdd5c0a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdd64000c00 tx=0x7fdd64005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:17.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:17.847+0000 7fdd788f3700 1 -- 172.21.15.40:0/612813615 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdd6809bf60 con 0x7fdd7406b460 2024-03-18T22:41:18.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:18.112+0000 7fdd798f5700 1 -- 172.21.15.40:0/612813615 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fdd74199d70 con 0x7fdd7406b460 2024-03-18T22:41:18.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:18.117+0000 7fdd788f3700 1 -- 172.21.15.40:0/612813615 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262658 (secure 0 0 0) 0x7fdd6809b860 con 0x7fdd7406b460 2024-03-18T22:41:18.122 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:41:18.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:18.120+0000 7fdd798f5700 1 -- 172.21.15.40:0/612813615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdd5c0a2920 msgr2=0x7fdd5c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:18.120+0000 7fdd798f5700 1 --2- 172.21.15.40:0/612813615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdd5c0a2920 0x7fdd5c0a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdd64000c00 tx=0x7fdd64005fb0 comp rx=0 tx=0).stop 2024-03-18T22:41:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:18.120+0000 7fdd798f5700 1 -- 172.21.15.40:0/612813615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd7406b460 msgr2=0x7fdd7419ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:18.120+0000 7fdd798f5700 1 --2- 172.21.15.40:0/612813615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd7406b460 0x7fdd7419ec00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdd6800e940 tx=0x7fdd6800ec50 comp rx=0 tx=0).stop 2024-03-18T22:41:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:18.120+0000 7fdd798f5700 1 -- 172.21.15.40:0/612813615 shutdown_connections 2024-03-18T22:41:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:18.120+0000 7fdd798f5700 1 --2- 172.21.15.40:0/612813615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd74107690 0x7fdd74198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:18.120+0000 7fdd798f5700 1 --2- 172.21.15.40:0/612813615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdd5c0a2920 0x7fdd5c0a4de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:18.120+0000 7fdd798f5700 1 --2- 172.21.15.40:0/612813615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd7406aab0 0x7fdd7419e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:18.120+0000 7fdd798f5700 1 --2- 172.21.15.40:0/612813615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd7406b460 0x7fdd7419ec00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:18.120+0000 7fdd798f5700 1 -- 172.21.15.40:0/612813615 >> 172.21.15.40:0/612813615 conn(0x7fdd74077260 msgr2=0x7fdd740ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:18.120+0000 7fdd798f5700 1 -- 172.21.15.40:0/612813615 shutdown_connections 2024-03-18T22:41:18.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:18.120+0000 7fdd798f5700 1 -- 172.21.15.40:0/612813615 wait complete. 2024-03-18T22:41:18.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:18 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/612813615' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T22:41:18.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:18 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/612813615' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T22:41:18.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:18 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/612813615' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T22:41:18.876 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":17,"flags":0,"active_gid":14148,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":3109254132},{"type":"v1","addr":"172.21.15.40:6801","nonce":3109254132}]},"active_addr":"172.21.15.40:6801/3109254132","active_change":"2024-03-18T22:33:42.721723+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.40:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":1880935272}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":841930532}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":89988959}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":2728419226}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.40:0","nonce":2463894675}]}]} 2024-03-18T22:41:18.881 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-18T22:41:18.881 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-18T22:41:18.881 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T22:41:19.132 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:19.478 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:19 smithi040 ceph-mon[62643]: pgmap v307: 129 pgs: 129 active+clean; 629 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.4 KiB/s wr, 24 op/s 2024-03-18T22:41:19.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:19 smithi113 ceph-mon[66775]: pgmap v307: 129 pgs: 129 active+clean; 629 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.4 KiB/s wr, 24 op/s 2024-03-18T22:41:19.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:19 smithi103 ceph-mon[66377]: pgmap v307: 129 pgs: 129 active+clean; 629 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.4 KiB/s wr, 24 op/s 2024-03-18T22:41:20.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.881+0000 7fc3208f2700 1 -- 172.21.15.40:0/4168107993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc31c101eb0 msgr2=0x7fc31c10e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:20.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.881+0000 7fc3208f2700 1 --2- 172.21.15.40:0/4168107993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc31c101eb0 0x7fc31c10e0e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00b0a0 tx=0x7fc30c00b3b0 comp rx=0 tx=0).stop 2024-03-18T22:41:20.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.881+0000 7fc3208f2700 1 -- 172.21.15.40:0/4168107993 shutdown_connections 2024-03-18T22:41:20.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.881+0000 7fc3208f2700 1 --2- 172.21.15.40:0/4168107993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc31c101eb0 0x7fc31c10e0e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:20.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.881+0000 7fc3208f2700 1 --2- 172.21.15.40:0/4168107993 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc31c107400 0x7fc31c1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:20.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.881+0000 7fc3208f2700 1 --2- 172.21.15.40:0/4168107993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc31c1014f0 0x7fc31c101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:20.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.881+0000 7fc3208f2700 1 -- 172.21.15.40:0/4168107993 >> 172.21.15.40:0/4168107993 conn(0x7fc31c078070 msgr2=0x7fc31c078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:20.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.882+0000 7fc3208f2700 1 -- 172.21.15.40:0/4168107993 shutdown_connections 2024-03-18T22:41:20.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.882+0000 7fc3208f2700 1 -- 172.21.15.40:0/4168107993 wait complete. 2024-03-18T22:41:20.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.882+0000 7fc3208f2700 1 Processor -- start 2024-03-18T22:41:20.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.882+0000 7fc3208f2700 1 -- start start 2024-03-18T22:41:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.882+0000 7fc3208f2700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc31c1014f0 0x7fc31c10e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.882+0000 7fc3208f2700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc31c101eb0 0x7fc31c10edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.882+0000 7fc3208f2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc31c107400 0x7fc31c10f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.882+0000 7fc3208f2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c112540 con 0x7fc31c107400 2024-03-18T22:41:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.882+0000 7fc3208f2700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c1123c0 con 0x7fc31c101eb0 2024-03-18T22:41:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc3208f2700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c1126c0 con 0x7fc31c1014f0 2024-03-18T22:41:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc319d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc31c101eb0 0x7fc31c10edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc31ad9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc31c107400 0x7fc31c10f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc319d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc31c101eb0 0x7fc31c10edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:44280/0 (socket says 172.21.15.40:44280) 2024-03-18T22:41:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc31a59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc31c1014f0 0x7fc31c10e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc319d9b700 1 -- 172.21.15.40:0/3015181428 learned_addr learned my addr 172.21.15.40:0/3015181428 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc31a59c700 1 -- 172.21.15.40:0/3015181428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc31c101eb0 msgr2=0x7fc31c10edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc31a59c700 1 --2- 172.21.15.40:0/3015181428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc31c101eb0 0x7fc31c10edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc31a59c700 1 -- 172.21.15.40:0/3015181428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc31c107400 msgr2=0x7fc31c10f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc31a59c700 1 --2- 172.21.15.40:0/3015181428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc31c107400 0x7fc31c10f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc31a59c700 1 -- 172.21.15.40:0/3015181428 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31c1a6ef0 con 0x7fc31c1014f0 2024-03-18T22:41:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc319d9b700 1 --2- 172.21.15.40:0/3015181428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc31c101eb0 0x7fc31c10edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc31a59c700 1 --2- 172.21.15.40:0/3015181428 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc31c1014f0 0x7fc31c10e870 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc31400e9f0 tx=0x7fc31400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc3137fe700 1 -- 172.21.15.40:0/3015181428 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31400c9c0 con 0x7fc31c1014f0 2024-03-18T22:41:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc3208f2700 1 -- 172.21.15.40:0/3015181428 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31c1a71e0 con 0x7fc31c1014f0 2024-03-18T22:41:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc3137fe700 1 -- 172.21.15.40:0/3015181428 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc31400cb60 con 0x7fc31c1014f0 2024-03-18T22:41:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc3137fe700 1 -- 172.21.15.40:0/3015181428 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc314018680 con 0x7fc31c1014f0 2024-03-18T22:41:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.883+0000 7fc3208f2700 1 -- 172.21.15.40:0/3015181428 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc31c1a7720 con 0x7fc31c1014f0 2024-03-18T22:41:20.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.884+0000 7fc3208f2700 1 -- 172.21.15.40:0/3015181428 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc31c102990 con 0x7fc31c1014f0 2024-03-18T22:41:20.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.885+0000 7fc3137fe700 1 -- 172.21.15.40:0/3015181428 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc314010430 con 0x7fc31c1014f0 2024-03-18T22:41:20.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.886+0000 7fc3137fe700 1 --2- 172.21.15.40:0/3015181428 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc3080a2920 0x7fc3080a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:20.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.886+0000 7fc319d9b700 1 --2- 172.21.15.40:0/3015181428 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc3080a2920 0x7fc3080a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:20.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.886+0000 7fc3137fe700 1 -- 172.21.15.40:0/3015181428 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc3140ceef0 con 0x7fc31c1014f0 2024-03-18T22:41:20.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.886+0000 7fc319d9b700 1 --2- 172.21.15.40:0/3015181428 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc3080a2920 0x7fc3080a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc31c078670 tx=0x7fc304009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:20.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:20.890+0000 7fc3137fe700 1 -- 172.21.15.40:0/3015181428 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc31409bb90 con 0x7fc31c1014f0 2024-03-18T22:41:21.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:21.096+0000 7fc3208f2700 1 -- 172.21.15.40:0/3015181428 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc31c02dbb0 con 0x7fc31c1014f0 2024-03-18T22:41:21.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:21.097+0000 7fc3137fe700 1 -- 172.21.15.40:0/3015181428 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24129 (secure 0 0 0) 0x7fc31409b490 con 0x7fc31c1014f0 2024-03-18T22:41:21.099 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:41:21.099 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":89,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:40:46.196741+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:41:21.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:21.099+0000 7fc3208f2700 1 -- 172.21.15.40:0/3015181428 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc3080a2920 msgr2=0x7fc3080a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:21.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:21.099+0000 7fc3208f2700 1 --2- 172.21.15.40:0/3015181428 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc3080a2920 0x7fc3080a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc31c078670 tx=0x7fc304009260 comp rx=0 tx=0).stop 2024-03-18T22:41:21.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:21.099+0000 7fc3208f2700 1 -- 172.21.15.40:0/3015181428 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc31c1014f0 msgr2=0x7fc31c10e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:21.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:21.099+0000 7fc3208f2700 1 --2- 172.21.15.40:0/3015181428 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc31c1014f0 0x7fc31c10e870 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc31400e9f0 tx=0x7fc31400ed00 comp rx=0 tx=0).stop 2024-03-18T22:41:21.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:21.099+0000 7fc3208f2700 1 -- 172.21.15.40:0/3015181428 shutdown_connections 2024-03-18T22:41:21.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:21.099+0000 7fc3208f2700 1 --2- 172.21.15.40:0/3015181428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc31c107400 0x7fc31c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:21.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:21.099+0000 7fc3208f2700 1 --2- 172.21.15.40:0/3015181428 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc31c1014f0 0x7fc31c10e870 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:21.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:21.099+0000 7fc3208f2700 1 --2- 172.21.15.40:0/3015181428 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc3080a2920 0x7fc3080a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:21.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:21.099+0000 7fc3208f2700 1 --2- 172.21.15.40:0/3015181428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc31c101eb0 0x7fc31c10edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:21.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:21.099+0000 7fc3208f2700 1 -- 172.21.15.40:0/3015181428 >> 172.21.15.40:0/3015181428 conn(0x7fc31c078070 msgr2=0x7fc31c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:21.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:21.099+0000 7fc3208f2700 1 -- 172.21.15.40:0/3015181428 shutdown_connections 2024-03-18T22:41:21.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:21.099+0000 7fc3208f2700 1 -- 172.21.15.40:0/3015181428 wait complete. 2024-03-18T22:41:21.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:20 smithi040 ceph-mon[62643]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.0 KiB/s wr, 23 op/s 2024-03-18T22:41:21.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:20 smithi113 ceph-mon[66775]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.0 KiB/s wr, 23 op/s 2024-03-18T22:41:21.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:20 smithi103 ceph-mon[66377]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.0 KiB/s wr, 23 op/s 2024-03-18T22:41:21.886 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-18T22:41:21.887 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T22:41:22.140 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:22.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:21 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3015181428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:41:22.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:21 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3015181428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:41:22.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:21 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3015181428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:41:23.146 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:22 smithi040 ceph-mon[62643]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 20 op/s 2024-03-18T22:41:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:22 smithi113 ceph-mon[66775]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 20 op/s 2024-03-18T22:41:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:22 smithi103 ceph-mon[66377]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 20 op/s 2024-03-18T22:41:24.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:23.999+0000 7f5d27dc6700 1 -- 172.21.15.40:0/3086853540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d20108340 msgr2=0x7f5d20108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:24.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:23.999+0000 7f5d27dc6700 1 --2- 172.21.15.40:0/3086853540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d20108340 0x7f5d20108720 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5d140099c0 tx=0x7f5d14009cd0 comp rx=0 tx=0).stop 2024-03-18T22:41:24.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:23.999+0000 7f5d27dc6700 1 -- 172.21.15.40:0/3086853540 shutdown_connections 2024-03-18T22:41:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:23.999+0000 7f5d27dc6700 1 --2- 172.21.15.40:0/3086853540 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d20102d20 0x7f5d2010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:23.999+0000 7f5d27dc6700 1 --2- 172.21.15.40:0/3086853540 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d20108340 0x7f5d20108720 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:23.999+0000 7f5d27dc6700 1 --2- 172.21.15.40:0/3086853540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d20102360 0x7f5d201027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:23.999+0000 7f5d27dc6700 1 -- 172.21.15.40:0/3086853540 >> 172.21.15.40:0/3086853540 conn(0x7f5d20079440 msgr2=0x7f5d20079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.000+0000 7f5d27dc6700 1 -- 172.21.15.40:0/3086853540 shutdown_connections 2024-03-18T22:41:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.000+0000 7f5d27dc6700 1 -- 172.21.15.40:0/3086853540 wait complete. 2024-03-18T22:41:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.000+0000 7f5d27dc6700 1 Processor -- start 2024-03-18T22:41:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.000+0000 7f5d27dc6700 1 -- start start 2024-03-18T22:41:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.000+0000 7f5d27dc6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d20102360 0x7f5d20074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.000+0000 7f5d27dc6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d20102d20 0x7f5d20074b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.000+0000 7f5d27dc6700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d20108340 0x7f5d20075090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.000+0000 7f5d27dc6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5d201124a0 con 0x7f5d20102d20 2024-03-18T22:41:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.000+0000 7f5d27dc6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5d20112320 con 0x7f5d20102360 2024-03-18T22:41:24.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.000+0000 7f5d27dc6700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5d20112620 con 0x7f5d20108340 2024-03-18T22:41:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d25361700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d20102d20 0x7f5d20074b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d25361700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d20102d20 0x7f5d20074b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54178/0 (socket says 172.21.15.40:54178) 2024-03-18T22:41:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d25b62700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d20102360 0x7f5d20074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d25361700 1 -- 172.21.15.40:0/272616518 learned_addr learned my addr 172.21.15.40:0/272616518 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d26363700 1 --2- 172.21.15.40:0/272616518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d20108340 0x7f5d20075090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d26363700 1 -- 172.21.15.40:0/272616518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d20102360 msgr2=0x7f5d20074610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d26363700 1 --2- 172.21.15.40:0/272616518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d20102360 0x7f5d20074610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d26363700 1 -- 172.21.15.40:0/272616518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d20102d20 msgr2=0x7f5d20074b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d26363700 1 --2- 172.21.15.40:0/272616518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d20102d20 0x7f5d20074b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d26363700 1 -- 172.21.15.40:0/272616518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d201a6e70 con 0x7f5d20108340 2024-03-18T22:41:24.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d25361700 1 --2- 172.21.15.40:0/272616518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d20102d20 0x7f5d20074b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:24.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d26363700 1 --2- 172.21.15.40:0/272616518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d20108340 0x7f5d20075090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5d1800efc0 tx=0x7f5d1800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:24.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d12ffd700 1 -- 172.21.15.40:0/272616518 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d180092e0 con 0x7f5d20108340 2024-03-18T22:41:24.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d27dc6700 1 -- 172.21.15.40:0/272616518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d201a7100 con 0x7f5d20108340 2024-03-18T22:41:24.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d12ffd700 1 -- 172.21.15.40:0/272616518 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5d18004bb0 con 0x7f5d20108340 2024-03-18T22:41:24.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d12ffd700 1 -- 172.21.15.40:0/272616518 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d18004020 con 0x7f5d20108340 2024-03-18T22:41:24.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.001+0000 7f5d27dc6700 1 -- 172.21.15.40:0/272616518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d201a7640 con 0x7f5d20108340 2024-03-18T22:41:24.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.002+0000 7f5d27dc6700 1 -- 172.21.15.40:0/272616518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d20068fd0 con 0x7f5d20108340 2024-03-18T22:41:24.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.003+0000 7f5d12ffd700 1 -- 172.21.15.40:0/272616518 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5d1801b030 con 0x7f5d20108340 2024-03-18T22:41:24.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.004+0000 7f5d12ffd700 1 --2- 172.21.15.40:0/272616518 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5d0c0a2920 0x7f5d0c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:24.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.004+0000 7f5d25b62700 1 --2- 172.21.15.40:0/272616518 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5d0c0a2920 0x7f5d0c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:24.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.004+0000 7f5d12ffd700 1 -- 172.21.15.40:0/272616518 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5d180cea20 con 0x7f5d20108340 2024-03-18T22:41:24.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.004+0000 7f5d25b62700 1 --2- 172.21.15.40:0/272616518 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5d0c0a2920 0x7f5d0c0a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5d14003760 tx=0x7f5d14005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:24.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.007+0000 7f5d12ffd700 1 -- 172.21.15.40:0/272616518 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5d18022020 con 0x7f5d20108340 2024-03-18T22:41:24.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.215+0000 7f5d27dc6700 1 -- 172.21.15.40:0/272616518 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d2002dcd0 con 0x7f5d20108340 2024-03-18T22:41:24.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.216+0000 7f5d12ffd700 1 -- 172.21.15.40:0/272616518 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24129 (secure 0 0 0) 0x7f5d18010040 con 0x7f5d20108340 2024-03-18T22:41:24.218 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:41:24.218 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":89,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:40:46.196741+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:41:24.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.219+0000 7f5d27dc6700 1 -- 172.21.15.40:0/272616518 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5d0c0a2920 msgr2=0x7f5d0c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:24.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.219+0000 7f5d27dc6700 1 --2- 172.21.15.40:0/272616518 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5d0c0a2920 0x7f5d0c0a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5d14003760 tx=0x7f5d14005fb0 comp rx=0 tx=0).stop 2024-03-18T22:41:24.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.219+0000 7f5d27dc6700 1 -- 172.21.15.40:0/272616518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d20108340 msgr2=0x7f5d20075090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:24.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.219+0000 7f5d27dc6700 1 --2- 172.21.15.40:0/272616518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d20108340 0x7f5d20075090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5d1800efc0 tx=0x7f5d1800c3a0 comp rx=0 tx=0).stop 2024-03-18T22:41:24.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.219+0000 7f5d27dc6700 1 -- 172.21.15.40:0/272616518 shutdown_connections 2024-03-18T22:41:24.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.219+0000 7f5d27dc6700 1 --2- 172.21.15.40:0/272616518 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d20108340 0x7f5d20075090 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:24.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.219+0000 7f5d27dc6700 1 --2- 172.21.15.40:0/272616518 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5d0c0a2920 0x7f5d0c0a4de0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:24.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.219+0000 7f5d27dc6700 1 --2- 172.21.15.40:0/272616518 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d20102360 0x7f5d20074610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:24.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.219+0000 7f5d27dc6700 1 --2- 172.21.15.40:0/272616518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d20102d20 0x7f5d20074b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:24.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.219+0000 7f5d27dc6700 1 -- 172.21.15.40:0/272616518 >> 172.21.15.40:0/272616518 conn(0x7f5d20079440 msgr2=0x7f5d20101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:24.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.219+0000 7f5d27dc6700 1 -- 172.21.15.40:0/272616518 shutdown_connections 2024-03-18T22:41:24.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:24.219+0000 7f5d27dc6700 1 -- 172.21.15.40:0/272616518 wait complete. 2024-03-18T22:41:24.528 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:24 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/272616518' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:41:24.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:24 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/272616518' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:41:24.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:24 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/272616518' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:41:24.869 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-18T22:41:24.869 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-18T22:41:24.870 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-18T22:41:24.870 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-18T22:41:24.870 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-18T22:41:24.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-18T22:41:24.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-18T22:41:24.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-18T22:41:24.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-18T22:41:24.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-18T22:41:24.873 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-18T22:41:24.873 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-18T22:41:25.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:25 smithi040 ceph-mon[62643]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.4 KiB/s wr, 12 op/s 2024-03-18T22:41:25.563 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:25.569 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:25.576 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:25.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:25 smithi113 ceph-mon[66775]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.4 KiB/s wr, 12 op/s 2024-03-18T22:41:25.583 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:25.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:25 smithi103 ceph-mon[66377]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.4 KiB/s wr, 12 op/s 2024-03-18T22:41:25.878 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:25.879 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:25.891 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:25.892 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:27.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:26 smithi040 ceph-mon[62643]: pgmap v311: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-18T22:41:27.314 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:27.315 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:27.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:26 smithi113 ceph-mon[66775]: pgmap v311: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-18T22:41:27.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:26 smithi103 ceph-mon[66377]: pgmap v311: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-18T22:41:29.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:28 smithi040 ceph-mon[62643]: pgmap v312: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-18T22:41:29.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:28 smithi113 ceph-mon[66775]: pgmap v312: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-18T22:41:29.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:28 smithi103 ceph-mon[66377]: pgmap v312: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-18T22:41:31.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:30 smithi040 ceph-mon[62643]: pgmap v313: 129 pgs: 129 active+clean; 644 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-18T22:41:31.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:30 smithi113 ceph-mon[66775]: pgmap v313: 129 pgs: 129 active+clean; 644 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-18T22:41:31.354 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:31.356 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:31.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:30 smithi103 ceph-mon[66377]: pgmap v313: 129 pgs: 129 active+clean; 644 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-18T22:41:32.938 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:32 smithi040 ceph-mon[62643]: pgmap v314: 129 pgs: 129 active+clean; 644 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T22:41:33.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:32 smithi113 ceph-mon[66775]: pgmap v314: 129 pgs: 129 active+clean; 644 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T22:41:33.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:32 smithi103 ceph-mon[66377]: pgmap v314: 129 pgs: 129 active+clean; 644 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T22:41:35.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:34 smithi040 ceph-mon[62643]: pgmap v315: 129 pgs: 129 active+clean; 644 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T22:41:35.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:34 smithi113 ceph-mon[66775]: pgmap v315: 129 pgs: 129 active+clean; 644 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T22:41:35.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:34 smithi103 ceph-mon[66377]: pgmap v315: 129 pgs: 129 active+clean; 644 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T22:41:35.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.406+0000 7fcc84587700 1 -- 172.21.15.40:0/763354794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c102360 msgr2=0x7fcc7c1027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:35.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.406+0000 7fcc84587700 1 --2- 172.21.15.40:0/763354794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c102360 0x7fcc7c1027e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcc780099c0 tx=0x7fcc78009cd0 comp rx=0 tx=0).stop 2024-03-18T22:41:35.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.406+0000 7fcc84587700 1 -- 172.21.15.40:0/763354794 shutdown_connections 2024-03-18T22:41:35.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.406+0000 7fcc84587700 1 --2- 172.21.15.40:0/763354794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc7c102d20 0x7fcc7c10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:35.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.406+0000 7fcc84587700 1 --2- 172.21.15.40:0/763354794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc7c108340 0x7fcc7c108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:35.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.406+0000 7fcc84587700 1 --2- 172.21.15.40:0/763354794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c102360 0x7fcc7c1027e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:35.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.406+0000 7fcc84587700 1 -- 172.21.15.40:0/763354794 >> 172.21.15.40:0/763354794 conn(0x7fcc7c079440 msgr2=0x7fcc7c079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:35.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.406+0000 7fcc84587700 1 -- 172.21.15.40:0/763354794 shutdown_connections 2024-03-18T22:41:35.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.406+0000 7fcc84587700 1 -- 172.21.15.40:0/763354794 wait complete. 2024-03-18T22:41:35.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.407+0000 7fcc84587700 1 Processor -- start 2024-03-18T22:41:35.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.407+0000 7fcc84587700 1 -- start start 2024-03-18T22:41:35.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.407+0000 7fcc84587700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc7c102360 0x7fcc7c1a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:35.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.407+0000 7fcc84587700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c102d20 0x7fcc7c1a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:35.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.407+0000 7fcc84587700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc7c108340 0x7fcc7c10e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:35.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.407+0000 7fcc84587700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c1124a0 con 0x7fcc7c102d20 2024-03-18T22:41:35.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.407+0000 7fcc84587700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c112320 con 0x7fcc7c102360 2024-03-18T22:41:35.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.407+0000 7fcc84587700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c112620 con 0x7fcc7c108340 2024-03-18T22:41:35.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.407+0000 7fcc82323700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc7c102360 0x7fcc7c1a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:35.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.407+0000 7fcc81b22700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c102d20 0x7fcc7c1a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:35.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.407+0000 7fcc82323700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc7c102360 0x7fcc7c1a4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:55796/0 (socket says 172.21.15.40:55796) 2024-03-18T22:41:35.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.407+0000 7fcc82323700 1 -- 172.21.15.40:0/3298237002 learned_addr learned my addr 172.21.15.40:0/3298237002 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:35.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.407+0000 7fcc82b24700 1 --2- 172.21.15.40:0/3298237002 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc7c108340 0x7fcc7c10e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:35.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.408+0000 7fcc82323700 1 -- 172.21.15.40:0/3298237002 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc7c108340 msgr2=0x7fcc7c10e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:35.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.408+0000 7fcc82323700 1 --2- 172.21.15.40:0/3298237002 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc7c108340 0x7fcc7c10e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:35.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.408+0000 7fcc82323700 1 -- 172.21.15.40:0/3298237002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c102d20 msgr2=0x7fcc7c1a5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:35.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.408+0000 7fcc82323700 1 --2- 172.21.15.40:0/3298237002 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c102d20 0x7fcc7c1a5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:35.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.408+0000 7fcc82323700 1 -- 172.21.15.40:0/3298237002 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7c10eab0 con 0x7fcc7c102360 2024-03-18T22:41:35.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.408+0000 7fcc82b24700 1 --2- 172.21.15.40:0/3298237002 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcc7c108340 0x7fcc7c10e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:35.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.408+0000 7fcc82323700 1 --2- 172.21.15.40:0/3298237002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc7c102360 0x7fcc7c1a4bf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c00e9a0 tx=0x7fcc6c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:35.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.408+0000 7fcc737fe700 1 -- 172.21.15.40:0/3298237002 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c00ca10 con 0x7fcc7c102360 2024-03-18T22:41:35.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.408+0000 7fcc737fe700 1 -- 172.21.15.40:0/3298237002 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc6c00cbb0 con 0x7fcc7c102360 2024-03-18T22:41:35.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.408+0000 7fcc737fe700 1 -- 172.21.15.40:0/3298237002 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c018680 con 0x7fcc7c102360 2024-03-18T22:41:35.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.408+0000 7fcc84587700 1 -- 172.21.15.40:0/3298237002 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc7c10eda0 con 0x7fcc7c102360 2024-03-18T22:41:35.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.408+0000 7fcc84587700 1 -- 172.21.15.40:0/3298237002 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc7c1aba50 con 0x7fcc7c102360 2024-03-18T22:41:35.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.409+0000 7fcc84587700 1 -- 172.21.15.40:0/3298237002 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcc7c0752a0 con 0x7fcc7c102360 2024-03-18T22:41:35.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.411+0000 7fcc737fe700 1 -- 172.21.15.40:0/3298237002 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fcc6c010430 con 0x7fcc7c102360 2024-03-18T22:41:35.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.411+0000 7fcc737fe700 1 --2- 172.21.15.40:0/3298237002 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcc680a2920 0x7fcc680a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:35.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.411+0000 7fcc81b22700 1 --2- 172.21.15.40:0/3298237002 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcc680a2920 0x7fcc680a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:35.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.411+0000 7fcc737fe700 1 -- 172.21.15.40:0/3298237002 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcc6c0cf4b0 con 0x7fcc7c102360 2024-03-18T22:41:35.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.411+0000 7fcc737fe700 1 --2- 172.21.15.40:0/3298237002 >> [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891] conn(0x7fcc680ad5e0 0x7fcc680afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:35.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.411+0000 7fcc737fe700 1 -- 172.21.15.40:0/3298237002 --> [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc6c01bec0 con 0x7fcc680ad5e0 2024-03-18T22:41:35.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.411+0000 7fcc737fe700 1 -- 172.21.15.40:0/3298237002 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fcc6c0cf860 con 0x7fcc7c102360 2024-03-18T22:41:35.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.412+0000 7fcc82b24700 1 --2- 172.21.15.40:0/3298237002 >> [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891] conn(0x7fcc680ad5e0 0x7fcc680afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:35.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.412+0000 7fcc81b22700 1 --2- 172.21.15.40:0/3298237002 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcc680a2920 0x7fcc680a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c1a0500 tx=0x7fcc78005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:35.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.412+0000 7fcc82b24700 1 --2- 172.21.15.40:0/3298237002 >> [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891] conn(0x7fcc680ad5e0 0x7fcc680afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:35.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.413+0000 7fcc737fe700 1 -- 172.21.15.40:0/3298237002 <== osd.3 v2:172.21.15.40:6810/3224857891 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc6c01bec0 con 0x7fcc680ad5e0 2024-03-18T22:41:35.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.430+0000 7fcc84587700 1 -- 172.21.15.40:0/3298237002 --> [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcc7c1039f0 con 0x7fcc680ad5e0 2024-03-18T22:41:35.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.430+0000 7fcc737fe700 1 -- 172.21.15.40:0/3298237002 <== osd.3 v2:172.21.15.40:6810/3224857891 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fcc7c1039f0 con 0x7fcc680ad5e0 2024-03-18T22:41:35.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.431+0000 7fcc84587700 1 -- 172.21.15.40:0/3298237002 >> [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891] conn(0x7fcc680ad5e0 msgr2=0x7fcc680afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:35.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.431+0000 7fcc84587700 1 --2- 172.21.15.40:0/3298237002 >> [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891] conn(0x7fcc680ad5e0 0x7fcc680afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:35.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.431+0000 7fcc84587700 1 -- 172.21.15.40:0/3298237002 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcc680a2920 msgr2=0x7fcc680a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:35.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.431+0000 7fcc84587700 1 --2- 172.21.15.40:0/3298237002 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcc680a2920 0x7fcc680a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c1a0500 tx=0x7fcc78005fb0 comp rx=0 tx=0).stop 2024-03-18T22:41:35.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.431+0000 7fcc84587700 1 -- 172.21.15.40:0/3298237002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc7c102360 msgr2=0x7fcc7c1a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:35.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.431+0000 7fcc84587700 1 --2- 172.21.15.40:0/3298237002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc7c102360 0x7fcc7c1a4bf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c00e9a0 tx=0x7fcc6c00ecb0 comp rx=0 tx=0).stop 2024-03-18T22:41:35.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.431+0000 7fcc82b24700 1 -- 172.21.15.40:0/3298237002 reap_dead start 2024-03-18T22:41:35.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.431+0000 7fcc84587700 1 -- 172.21.15.40:0/3298237002 shutdown_connections 2024-03-18T22:41:35.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.431+0000 7fcc84587700 1 -- 172.21.15.40:0/3298237002 >> 172.21.15.40:0/3298237002 conn(0x7fcc7c079440 msgr2=0x7fcc7c074880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:35.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.431+0000 7fcc84587700 1 -- 172.21.15.40:0/3298237002 shutdown_connections 2024-03-18T22:41:35.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:35.431+0000 7fcc84587700 1 -- 172.21.15.40:0/3298237002 wait complete. 2024-03-18T22:41:37.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:36 smithi040 ceph-mon[62643]: pgmap v316: 129 pgs: 129 active+clean; 644 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T22:41:37.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:36 smithi113 ceph-mon[66775]: pgmap v316: 129 pgs: 129 active+clean; 644 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T22:41:37.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.355+0000 7f5bb0e53700 1 -- 172.21.15.40:0/3944275498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bac1012c0 msgr2=0x7f5bac10f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.355+0000 7f5bb0e53700 1 --2- 172.21.15.40:0/3944275498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bac1012c0 0x7f5bac10f220 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5ba400b0a0 tx=0x7f5ba400b3b0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.355+0000 7f5bb0e53700 1 -- 172.21.15.40:0/3944275498 shutdown_connections 2024-03-18T22:41:37.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.355+0000 7f5bb0e53700 1 --2- 172.21.15.40:0/3944275498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bac1012c0 0x7f5bac10f220 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.355+0000 7f5bb0e53700 1 --2- 172.21.15.40:0/3944275498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac107380 0x7f5bac107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.355+0000 7f5bb0e53700 1 --2- 172.21.15.40:0/3944275498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bac100900 0x7f5bac100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.355+0000 7f5bb0e53700 1 -- 172.21.15.40:0/3944275498 >> 172.21.15.40:0/3944275498 conn(0x7f5bac077810 msgr2=0x7f5bac0fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:37.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.355+0000 7f5bb0e53700 1 -- 172.21.15.40:0/3944275498 shutdown_connections 2024-03-18T22:41:37.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.355+0000 7f5bb0e53700 1 -- 172.21.15.40:0/3944275498 wait complete. 2024-03-18T22:41:37.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.356+0000 7f5bb0e53700 1 Processor -- start 2024-03-18T22:41:37.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.356+0000 7f5bb0e53700 1 -- start start 2024-03-18T22:41:37.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.356+0000 7f5bb0e53700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac100900 0x7f5bac101ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:37.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.356+0000 7f5bb0e53700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bac1012c0 0x7f5bac102020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.356+0000 7f5bb0e53700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bac107380 0x7f5bac102560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.356+0000 7f5bb0e53700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac112610 con 0x7f5bac1012c0 2024-03-18T22:41:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.356+0000 7f5bb0e53700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac112490 con 0x7f5bac100900 2024-03-18T22:41:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.356+0000 7f5bb0e53700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac112790 con 0x7f5bac107380 2024-03-18T22:41:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.356+0000 7f5baa59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac100900 0x7f5bac101ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.356+0000 7f5baa59c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac100900 0x7f5bac101ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:55828/0 (socket says 172.21.15.40:55828) 2024-03-18T22:41:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.356+0000 7f5ba9d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bac1012c0 0x7f5bac102020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.356+0000 7f5baa59c700 1 -- 172.21.15.40:0/864533925 learned_addr learned my addr 172.21.15.40:0/864533925 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.356+0000 7f5baad9d700 1 --2- 172.21.15.40:0/864533925 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bac107380 0x7f5bac102560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.357+0000 7f5baa59c700 1 -- 172.21.15.40:0/864533925 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bac107380 msgr2=0x7f5bac102560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.357+0000 7f5baa59c700 1 --2- 172.21.15.40:0/864533925 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5bac107380 0x7f5bac102560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.357+0000 7f5baa59c700 1 -- 172.21.15.40:0/864533925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bac1012c0 msgr2=0x7f5bac102020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.357+0000 7f5baa59c700 1 --2- 172.21.15.40:0/864533925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bac1012c0 0x7f5bac102020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.357+0000 7f5baa59c700 1 -- 172.21.15.40:0/864533925 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bac1a6ff0 con 0x7f5bac100900 2024-03-18T22:41:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.357+0000 7f5baa59c700 1 --2- 172.21.15.40:0/864533925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac100900 0x7f5bac101ae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5b9400e9c0 tx=0x7f5b9400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.357+0000 7f5ba37fe700 1 -- 172.21.15.40:0/864533925 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9400cbd0 con 0x7f5bac100900 2024-03-18T22:41:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.357+0000 7f5bb0e53700 1 -- 172.21.15.40:0/864533925 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bac1a72e0 con 0x7f5bac100900 2024-03-18T22:41:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.357+0000 7f5ba37fe700 1 -- 172.21.15.40:0/864533925 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b9400ee60 con 0x7f5bac100900 2024-03-18T22:41:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.357+0000 7f5bb0e53700 1 -- 172.21.15.40:0/864533925 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bac1a7820 con 0x7f5bac100900 2024-03-18T22:41:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.357+0000 7f5ba37fe700 1 -- 172.21.15.40:0/864533925 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b940055b0 con 0x7f5bac100900 2024-03-18T22:41:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.358+0000 7f5bb0e53700 1 -- 172.21.15.40:0/864533925 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5b8c001030 con 0x7f5bac100900 2024-03-18T22:41:37.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:36 smithi103 ceph-mon[66377]: pgmap v316: 129 pgs: 129 active+clean; 644 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T22:41:37.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.359+0000 7f5ba37fe700 1 -- 172.21.15.40:0/864533925 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5b94004750 con 0x7f5bac100900 2024-03-18T22:41:37.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.360+0000 7f5ba37fe700 1 --2- 172.21.15.40:0/864533925 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b980a29a0 0x7f5b980a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.360+0000 7f5ba9d9b700 1 --2- 172.21.15.40:0/864533925 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b980a29a0 0x7f5b980a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.360+0000 7f5ba37fe700 1 -- 172.21.15.40:0/864533925 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5b940cf2a0 con 0x7f5bac100900 2024-03-18T22:41:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.360+0000 7f5ba37fe700 1 --2- 172.21.15.40:0/864533925 >> [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248] conn(0x7f5b980ad660 0x7f5b980afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.360+0000 7f5ba37fe700 1 -- 172.21.15.40:0/864533925 --> [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b980b0170 con 0x7f5b980ad660 2024-03-18T22:41:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.360+0000 7f5ba37fe700 1 -- 172.21.15.40:0/864533925 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f5b940cf6f0 con 0x7f5bac100900 2024-03-18T22:41:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.360+0000 7f5baad9d700 1 --2- 172.21.15.40:0/864533925 >> [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248] conn(0x7f5b980ad660 0x7f5b980afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.360+0000 7f5ba9d9b700 1 --2- 172.21.15.40:0/864533925 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b980a29a0 0x7f5b980a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5bac0ffe70 tx=0x7f5b9c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.361+0000 7f5baad9d700 1 --2- 172.21.15.40:0/864533925 >> [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248] conn(0x7f5b980ad660 0x7f5b980afaa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.361+0000 7f5ba37fe700 1 -- 172.21.15.40:0/864533925 <== osd.0 v2:172.21.15.40:6802/1816003248 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5b980b0170 con 0x7f5b980ad660 2024-03-18T22:41:37.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.379+0000 7f5bb0e53700 1 -- 172.21.15.40:0/864533925 --> [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5b8c002e60 con 0x7f5b980ad660 2024-03-18T22:41:37.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.379+0000 7f5ba37fe700 1 -- 172.21.15.40:0/864533925 <== osd.0 v2:172.21.15.40:6802/1816003248 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5b8c002e60 con 0x7f5b980ad660 2024-03-18T22:41:37.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.379+0000 7f5bb0e53700 1 -- 172.21.15.40:0/864533925 >> [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248] conn(0x7f5b980ad660 msgr2=0x7f5b980afaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.379+0000 7f5bb0e53700 1 --2- 172.21.15.40:0/864533925 >> [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248] conn(0x7f5b980ad660 0x7f5b980afaa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.379+0000 7f5bb0e53700 1 -- 172.21.15.40:0/864533925 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b980a29a0 msgr2=0x7f5b980a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.379+0000 7f5bb0e53700 1 --2- 172.21.15.40:0/864533925 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b980a29a0 0x7f5b980a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5bac0ffe70 tx=0x7f5b9c009260 comp rx=0 tx=0).stop 2024-03-18T22:41:37.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.380+0000 7f5bb0e53700 1 -- 172.21.15.40:0/864533925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac100900 msgr2=0x7f5bac101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.380+0000 7f5bb0e53700 1 --2- 172.21.15.40:0/864533925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bac100900 0x7f5bac101ae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5b9400e9c0 tx=0x7f5b9400ecd0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.380+0000 7f5baad9d700 1 -- 172.21.15.40:0/864533925 reap_dead start 2024-03-18T22:41:37.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.380+0000 7f5bb0e53700 1 -- 172.21.15.40:0/864533925 shutdown_connections 2024-03-18T22:41:37.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.380+0000 7f5bb0e53700 1 -- 172.21.15.40:0/864533925 >> 172.21.15.40:0/864533925 conn(0x7f5bac077810 msgr2=0x7f5bac105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:37.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.380+0000 7f5bb0e53700 1 -- 172.21.15.40:0/864533925 shutdown_connections 2024-03-18T22:41:37.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.380+0000 7f5bb0e53700 1 -- 172.21.15.40:0/864533925 wait complete. 2024-03-18T22:41:37.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.582+0000 7fb05d42e700 1 -- 172.21.15.40:0/2523587632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb05806b5c0 msgr2=0x7fb058106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.582+0000 7fb05d42e700 1 --2- 172.21.15.40:0/2523587632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb05806b5c0 0x7fb058106f50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb0400099c0 tx=0x7fb040009cd0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.582+0000 7fb05d42e700 1 -- 172.21.15.40:0/2523587632 shutdown_connections 2024-03-18T22:41:37.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.582+0000 7fb05d42e700 1 --2- 172.21.15.40:0/2523587632 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb058107680 0x7fb05810f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.582+0000 7fb05d42e700 1 --2- 172.21.15.40:0/2523587632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb05806ac10 0x7fb05806aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.582+0000 7fb05d42e700 1 --2- 172.21.15.40:0/2523587632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb05806b5c0 0x7fb058106f50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.582+0000 7fb05d42e700 1 -- 172.21.15.40:0/2523587632 >> 172.21.15.40:0/2523587632 conn(0x7fb058078180 msgr2=0x7fb058078590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:37.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.582+0000 7fb05d42e700 1 -- 172.21.15.40:0/2523587632 shutdown_connections 2024-03-18T22:41:37.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.582+0000 7fb05d42e700 1 -- 172.21.15.40:0/2523587632 wait complete. 2024-03-18T22:41:37.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.583+0000 7fb05d42e700 1 Processor -- start 2024-03-18T22:41:37.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.583+0000 7fb05d42e700 1 -- start start 2024-03-18T22:41:37.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.583+0000 7fb05d42e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb05806ac10 0x7fb05819e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:37.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.583+0000 7fb05d42e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb05806b5c0 0x7fb05819ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:37.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.583+0000 7fb05d42e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb058107680 0x7fb058198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:37.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.583+0000 7fb05d42e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb058112480 con 0x7fb05806ac10 2024-03-18T22:41:37.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.583+0000 7fb05d42e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb058112300 con 0x7fb058107680 2024-03-18T22:41:37.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.583+0000 7fb05d42e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb058112600 con 0x7fb05806b5c0 2024-03-18T22:41:37.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.583+0000 7fb056ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb05806ac10 0x7fb05819e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:37.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.583+0000 7fb056ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb05806ac10 0x7fb05819e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47926/0 (socket says 172.21.15.40:47926) 2024-03-18T22:41:37.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.583+0000 7fb056ffd700 1 -- 172.21.15.40:0/4019132704 learned_addr learned my addr 172.21.15.40:0/4019132704 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:37.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.583+0000 7fb0577fe700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb058107680 0x7fb058198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:37.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.583+0000 7fb0567fc700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb05806b5c0 0x7fb05819ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:37.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.584+0000 7fb056ffd700 1 -- 172.21.15.40:0/4019132704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb05806b5c0 msgr2=0x7fb05819ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.584+0000 7fb056ffd700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb05806b5c0 0x7fb05819ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.584+0000 7fb056ffd700 1 -- 172.21.15.40:0/4019132704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb058107680 msgr2=0x7fb058198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.584+0000 7fb056ffd700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb058107680 0x7fb058198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.584+0000 7fb056ffd700 1 -- 172.21.15.40:0/4019132704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb058199060 con 0x7fb05806ac10 2024-03-18T22:41:37.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.584+0000 7fb0577fe700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb058107680 0x7fb058198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:37.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.584+0000 7fb0567fc700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb05806b5c0 0x7fb05819ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:37.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.584+0000 7fb056ffd700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb05806ac10 0x7fb05819e6f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb04800c370 tx=0x7fb04800c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:37.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.584+0000 7fb04ffff700 1 -- 172.21.15.40:0/4019132704 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0480092e0 con 0x7fb05806ac10 2024-03-18T22:41:37.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.584+0000 7fb05d42e700 1 -- 172.21.15.40:0/4019132704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0581992f0 con 0x7fb05806ac10 2024-03-18T22:41:37.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.584+0000 7fb04ffff700 1 -- 172.21.15.40:0/4019132704 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb04800cbe0 con 0x7fb05806ac10 2024-03-18T22:41:37.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.584+0000 7fb04ffff700 1 -- 172.21.15.40:0/4019132704 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb048004710 con 0x7fb05806ac10 2024-03-18T22:41:37.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.584+0000 7fb05d42e700 1 -- 172.21.15.40:0/4019132704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0581a55c0 con 0x7fb05806ac10 2024-03-18T22:41:37.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.585+0000 7fb05d42e700 1 -- 172.21.15.40:0/4019132704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb058062a20 con 0x7fb05806ac10 2024-03-18T22:41:37.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.586+0000 7fb04ffff700 1 -- 172.21.15.40:0/4019132704 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fb04801a020 con 0x7fb05806ac10 2024-03-18T22:41:37.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.587+0000 7fb04ffff700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0440a2920 0x7fb0440a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:37.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.587+0000 7fb0567fc700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0440a2920 0x7fb0440a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:37.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.587+0000 7fb04ffff700 1 -- 172.21.15.40:0/4019132704 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb0480cf2b0 con 0x7fb05806ac10 2024-03-18T22:41:37.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.587+0000 7fb04ffff700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763] conn(0x7fb0440ad5e0 0x7fb0440afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:37.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.587+0000 7fb04ffff700 1 -- 172.21.15.40:0/4019132704 --> [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb048008ec0 con 0x7fb0440ad5e0 2024-03-18T22:41:37.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.587+0000 7fb04ffff700 1 -- 172.21.15.40:0/4019132704 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fb0480cf660 con 0x7fb05806ac10 2024-03-18T22:41:37.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.587+0000 7fb0577fe700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763] conn(0x7fb0440ad5e0 0x7fb0440afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:37.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.587+0000 7fb0567fc700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0440a2920 0x7fb0440a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb040009990 tx=0x7fb040005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:37.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.587+0000 7fb0577fe700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763] conn(0x7fb0440ad5e0 0x7fb0440afa20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:37.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.588+0000 7fb04ffff700 1 -- 172.21.15.40:0/4019132704 <== osd.6 v2:172.21.15.40:6818/1125393763 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb048008ec0 con 0x7fb0440ad5e0 2024-03-18T22:41:37.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.605+0000 7fb05d42e700 1 -- 172.21.15.40:0/4019132704 --> [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb058062ea0 con 0x7fb0440ad5e0 2024-03-18T22:41:37.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.605+0000 7fb04ffff700 1 -- 172.21.15.40:0/4019132704 <== osd.6 v2:172.21.15.40:6818/1125393763 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb058062ea0 con 0x7fb0440ad5e0 2024-03-18T22:41:37.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.605+0000 7fb05d42e700 1 -- 172.21.15.40:0/4019132704 >> [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763] conn(0x7fb0440ad5e0 msgr2=0x7fb0440afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.605+0000 7fb05d42e700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763] conn(0x7fb0440ad5e0 0x7fb0440afa20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.605+0000 7fb05d42e700 1 -- 172.21.15.40:0/4019132704 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0440a2920 msgr2=0x7fb0440a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.605+0000 7fb05d42e700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0440a2920 0x7fb0440a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb040009990 tx=0x7fb040005fb0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.605+0000 7fb05d42e700 1 -- 172.21.15.40:0/4019132704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb05806ac10 msgr2=0x7fb05819e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.605+0000 7fb05d42e700 1 --2- 172.21.15.40:0/4019132704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb05806ac10 0x7fb05819e6f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb04800c370 tx=0x7fb04800c680 comp rx=0 tx=0).stop 2024-03-18T22:41:37.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.605+0000 7fb0577fe700 1 -- 172.21.15.40:0/4019132704 reap_dead start 2024-03-18T22:41:37.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.605+0000 7fb05d42e700 1 -- 172.21.15.40:0/4019132704 shutdown_connections 2024-03-18T22:41:37.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.605+0000 7fb05d42e700 1 -- 172.21.15.40:0/4019132704 >> 172.21.15.40:0/4019132704 conn(0x7fb058078180 msgr2=0x7fb0580ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:37.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.606+0000 7fb05d42e700 1 -- 172.21.15.40:0/4019132704 shutdown_connections 2024-03-18T22:41:37.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.606+0000 7fb05d42e700 1 -- 172.21.15.40:0/4019132704 wait complete. 2024-03-18T22:41:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.166+0000 7f1a7369f700 1 -- 172.21.15.40:0/4160990246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a6c1014a0 msgr2=0x7f1a6c101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.166+0000 7f1a7369f700 1 --2- 172.21.15.40:0/4160990246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a6c1014a0 0x7f1a6c101880 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1a680099c0 tx=0x7f1a68009cd0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.166+0000 7f1a7369f700 1 -- 172.21.15.40:0/4160990246 shutdown_connections 2024-03-18T22:41:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.166+0000 7f1a7369f700 1 --2- 172.21.15.40:0/4160990246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a6c1064f0 0x7f1a6c10e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.166+0000 7f1a7369f700 1 --2- 172.21.15.40:0/4160990246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a6c1014a0 0x7f1a6c101880 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.166+0000 7f1a7369f700 1 --2- 172.21.15.40:0/4160990246 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a6c101e50 0x7f1a6c105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.166+0000 7f1a7369f700 1 -- 172.21.15.40:0/4160990246 >> 172.21.15.40:0/4160990246 conn(0x7f1a6c077260 msgr2=0x7f1a6c077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.166+0000 7f1a7369f700 1 -- 172.21.15.40:0/4160990246 shutdown_connections 2024-03-18T22:41:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.166+0000 7f1a7369f700 1 -- 172.21.15.40:0/4160990246 wait complete. 2024-03-18T22:41:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.166+0000 7f1a7369f700 1 Processor -- start 2024-03-18T22:41:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.166+0000 7f1a7369f700 1 -- start start 2024-03-18T22:41:37.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a7369f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a6c1014a0 0x7f1a6c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a7369f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a6c101e50 0x7f1a6c19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a7369f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a6c1064f0 0x7f1a6c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a7369f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6c111380 con 0x7f1a6c1064f0 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a7369f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6c111200 con 0x7f1a6c101e50 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a7369f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6c111500 con 0x7f1a6c1014a0 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a7143b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a6c1014a0 0x7f1a6c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a71c3c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a6c1064f0 0x7f1a6c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a7143b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a6c1014a0 0x7f1a6c19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:50902/0 (socket says 172.21.15.40:50902) 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a7143b700 1 -- 172.21.15.40:0/2507320360 learned_addr learned my addr 172.21.15.40:0/2507320360 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a70c3a700 1 --2- 172.21.15.40:0/2507320360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a6c101e50 0x7f1a6c19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a7143b700 1 -- 172.21.15.40:0/2507320360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a6c101e50 msgr2=0x7f1a6c19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a7143b700 1 --2- 172.21.15.40:0/2507320360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a6c101e50 0x7f1a6c19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a7143b700 1 -- 172.21.15.40:0/2507320360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a6c1064f0 msgr2=0x7f1a6c1987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a7143b700 1 --2- 172.21.15.40:0/2507320360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a6c1064f0 0x7f1a6c1987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a7143b700 1 -- 172.21.15.40:0/2507320360 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a6c198fd0 con 0x7f1a6c1014a0 2024-03-18T22:41:37.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a7143b700 1 --2- 172.21.15.40:0/2507320360 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a6c1014a0 0x7f1a6c19e670 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1a68006010 tx=0x7f1a68005280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:37.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a5e7fc700 1 -- 172.21.15.40:0/2507320360 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a68026070 con 0x7f1a6c1014a0 2024-03-18T22:41:37.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.167+0000 7f1a5e7fc700 1 -- 172.21.15.40:0/2507320360 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a68005680 con 0x7f1a6c1014a0 2024-03-18T22:41:37.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.168+0000 7f1a7369f700 1 -- 172.21.15.40:0/2507320360 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a6c199260 con 0x7f1a6c1014a0 2024-03-18T22:41:37.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.168+0000 7f1a5e7fc700 1 -- 172.21.15.40:0/2507320360 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6800fb20 con 0x7f1a6c1014a0 2024-03-18T22:41:37.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.168+0000 7f1a7369f700 1 -- 172.21.15.40:0/2507320360 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a6c1a54d0 con 0x7f1a6c1014a0 2024-03-18T22:41:37.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.168+0000 7f1a7369f700 1 -- 172.21.15.40:0/2507320360 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1a6c062aa0 con 0x7f1a6c1014a0 2024-03-18T22:41:37.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.169+0000 7f1a5e7fc700 1 -- 172.21.15.40:0/2507320360 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1a68029070 con 0x7f1a6c1014a0 2024-03-18T22:41:37.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.170+0000 7f1a5e7fc700 1 --2- 172.21.15.40:0/2507320360 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a580a2920 0x7f1a580a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:37.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.170+0000 7f1a70c3a700 1 --2- 172.21.15.40:0/2507320360 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a580a2920 0x7f1a580a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:37.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.170+0000 7f1a5e7fc700 1 -- 172.21.15.40:0/2507320360 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1a680cf5b0 con 0x7f1a6c1014a0 2024-03-18T22:41:37.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.170+0000 7f1a5e7fc700 1 --2- 172.21.15.40:0/2507320360 >> [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708] conn(0x7f1a580ad5e0 0x7f1a580afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:37.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.170+0000 7f1a5e7fc700 1 -- 172.21.15.40:0/2507320360 --> [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1a68019e50 con 0x7f1a580ad5e0 2024-03-18T22:41:37.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.170+0000 7f1a5e7fc700 1 -- 172.21.15.40:0/2507320360 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f1a680d6050 con 0x7f1a6c1014a0 2024-03-18T22:41:37.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.170+0000 7f1a71c3c700 1 --2- 172.21.15.40:0/2507320360 >> [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708] conn(0x7f1a580ad5e0 0x7f1a580afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:37.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.170+0000 7f1a70c3a700 1 --2- 172.21.15.40:0/2507320360 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a580a2920 0x7f1a580a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c100fb0 tx=0x7f1a60009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:37.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.171+0000 7f1a71c3c700 1 --2- 172.21.15.40:0/2507320360 >> [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708] conn(0x7f1a580ad5e0 0x7f1a580afa20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:37.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.172+0000 7f1a5e7fc700 1 -- 172.21.15.40:0/2507320360 <== osd.8 v2:172.21.15.113:6816/2089621708 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1a68019e50 con 0x7f1a580ad5e0 2024-03-18T22:41:37.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.204+0000 7f1a7369f700 1 -- 172.21.15.40:0/2507320360 --> [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1a6c062f20 con 0x7f1a580ad5e0 2024-03-18T22:41:37.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.204+0000 7f1a5e7fc700 1 -- 172.21.15.40:0/2507320360 <== osd.8 v2:172.21.15.113:6816/2089621708 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1a6c062f20 con 0x7f1a580ad5e0 2024-03-18T22:41:37.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.205+0000 7f1a7369f700 1 -- 172.21.15.40:0/2507320360 >> [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708] conn(0x7f1a580ad5e0 msgr2=0x7f1a580afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.205+0000 7f1a7369f700 1 --2- 172.21.15.40:0/2507320360 >> [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708] conn(0x7f1a580ad5e0 0x7f1a580afa20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:37.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.205+0000 7f1a7369f700 1 -- 172.21.15.40:0/2507320360 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a580a2920 msgr2=0x7f1a580a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.205+0000 7f1a7369f700 1 --2- 172.21.15.40:0/2507320360 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a580a2920 0x7f1a580a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c100fb0 tx=0x7f1a60009260 comp rx=0 tx=0).stop 2024-03-18T22:41:37.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.205+0000 7f1a7369f700 1 -- 172.21.15.40:0/2507320360 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a6c1014a0 msgr2=0x7f1a6c19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:37.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.205+0000 7f1a7369f700 1 --2- 172.21.15.40:0/2507320360 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a6c1014a0 0x7f1a6c19e670 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1a68006010 tx=0x7f1a68005280 comp rx=0 tx=0).stop 2024-03-18T22:41:37.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.205+0000 7f1a71c3c700 1 -- 172.21.15.40:0/2507320360 reap_dead start 2024-03-18T22:41:37.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.205+0000 7f1a7369f700 1 -- 172.21.15.40:0/2507320360 shutdown_connections 2024-03-18T22:41:37.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.205+0000 7f1a7369f700 1 -- 172.21.15.40:0/2507320360 >> 172.21.15.40:0/2507320360 conn(0x7f1a6c077260 msgr2=0x7f1a6c1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:37.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.205+0000 7f1a7369f700 1 -- 172.21.15.40:0/2507320360 shutdown_connections 2024-03-18T22:41:37.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:37.205+0000 7f1a7369f700 1 -- 172.21.15.40:0/2507320360 wait complete. 2024-03-18T22:41:38.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.093+0000 7efccd9b1700 1 -- 172.21.15.40:0/3658028357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcc8101210 msgr2=0x7efcc8101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.093+0000 7efccd9b1700 1 --2- 172.21.15.40:0/3658028357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcc8101210 0x7efcc8101670 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7efcbc009990 tx=0x7efcbc009ca0 comp rx=0 tx=0).stop 2024-03-18T22:41:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.093+0000 7efccd9b1700 1 -- 172.21.15.40:0/3658028357 shutdown_connections 2024-03-18T22:41:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.094+0000 7efccd9b1700 1 --2- 172.21.15.40:0/3658028357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc8101bb0 0x7efcc810e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.094+0000 7efccd9b1700 1 --2- 172.21.15.40:0/3658028357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcc8107250 0x7efcc8107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.094+0000 7efccd9b1700 1 --2- 172.21.15.40:0/3658028357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcc8101210 0x7efcc8101670 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.094+0000 7efccd9b1700 1 -- 172.21.15.40:0/3658028357 >> 172.21.15.40:0/3658028357 conn(0x7efcc8078140 msgr2=0x7efcc8078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.094+0000 7efccd9b1700 1 -- 172.21.15.40:0/3658028357 shutdown_connections 2024-03-18T22:41:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.094+0000 7efccd9b1700 1 -- 172.21.15.40:0/3658028357 wait complete. 2024-03-18T22:41:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.094+0000 7efccd9b1700 1 Processor -- start 2024-03-18T22:41:38.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.094+0000 7efccd9b1700 1 -- start start 2024-03-18T22:41:38.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efccd9b1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc8101210 0x7efcc819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:38.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efccd9b1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcc8101bb0 0x7efcc819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:38.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efccd9b1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcc8107250 0x7efcc8198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:38.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efccd9b1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efcc8111360 con 0x7efcc8101bb0 2024-03-18T22:41:38.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efccd9b1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efcc81111e0 con 0x7efcc8107250 2024-03-18T22:41:38.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efccd9b1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7efcc81114e0 con 0x7efcc8101210 2024-03-18T22:41:38.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efcc67fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcc8101bb0 0x7efcc819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efcc67fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcc8101bb0 0x7efcc819ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47944/0 (socket says 172.21.15.40:47944) 2024-03-18T22:41:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efcc67fc700 1 -- 172.21.15.40:0/48827476 learned_addr learned my addr 172.21.15.40:0/48827476 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efcc77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcc8107250 0x7efcc8198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efcc6ffd700 1 --2- 172.21.15.40:0/48827476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc8101210 0x7efcc819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efcc6ffd700 1 -- 172.21.15.40:0/48827476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcc8107250 msgr2=0x7efcc8198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efcc6ffd700 1 --2- 172.21.15.40:0/48827476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcc8107250 0x7efcc8198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efcc6ffd700 1 -- 172.21.15.40:0/48827476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcc8101bb0 msgr2=0x7efcc819ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efcc6ffd700 1 --2- 172.21.15.40:0/48827476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcc8101bb0 0x7efcc819ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efcc6ffd700 1 -- 172.21.15.40:0/48827476 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc8198fd0 con 0x7efcc8101210 2024-03-18T22:41:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efcc67fc700 1 --2- 172.21.15.40:0/48827476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcc8101bb0 0x7efcc819ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efcc77fe700 1 --2- 172.21.15.40:0/48827476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efcc8107250 0x7efcc8198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efcc6ffd700 1 --2- 172.21.15.40:0/48827476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc8101210 0x7efcc819e6b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efcb800e9f0 tx=0x7efcb800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efccc9af700 1 -- 172.21.15.40:0/48827476 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb800c9c0 con 0x7efcc8101210 2024-03-18T22:41:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efccd9b1700 1 -- 172.21.15.40:0/48827476 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcc81992c0 con 0x7efcc8101210 2024-03-18T22:41:38.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.095+0000 7efccc9af700 1 -- 172.21.15.40:0/48827476 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcb800cb60 con 0x7efcc8101210 2024-03-18T22:41:38.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.096+0000 7efccc9af700 1 -- 172.21.15.40:0/48827476 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb8018680 con 0x7efcc8101210 2024-03-18T22:41:38.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.096+0000 7efccd9b1700 1 -- 172.21.15.40:0/48827476 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcc81a5580 con 0x7efcc8101210 2024-03-18T22:41:38.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.096+0000 7efccd9b1700 1 -- 172.21.15.40:0/48827476 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efcc81062e0 con 0x7efcc8101210 2024-03-18T22:41:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.098+0000 7efccc9af700 1 -- 172.21.15.40:0/48827476 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7efcb8010430 con 0x7efcc8101210 2024-03-18T22:41:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.098+0000 7efccc9af700 1 --2- 172.21.15.40:0/48827476 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efcb00a2920 0x7efcb00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.098+0000 7efcc67fc700 1 --2- 172.21.15.40:0/48827476 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efcb00a2920 0x7efcb00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.098+0000 7efccc9af700 1 -- 172.21.15.40:0/48827476 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7efcb80cf410 con 0x7efcc8101210 2024-03-18T22:41:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.098+0000 7efccc9af700 1 --2- 172.21.15.40:0/48827476 >> [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324] conn(0x7efcb00ad560 0x7efcb00af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.099+0000 7efccc9af700 1 -- 172.21.15.40:0/48827476 --> [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efcb00b0070 con 0x7efcb00ad560 2024-03-18T22:41:38.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.099+0000 7efccc9af700 1 -- 172.21.15.40:0/48827476 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7efcb80cf860 con 0x7efcc8101210 2024-03-18T22:41:38.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.099+0000 7efcc77fe700 1 --2- 172.21.15.40:0/48827476 >> [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324] conn(0x7efcb00ad560 0x7efcb00af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:38.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.099+0000 7efcc67fc700 1 --2- 172.21.15.40:0/48827476 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efcb00a2920 0x7efcb00a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efcc80786f0 tx=0x7efcbc00b630 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:38.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.099+0000 7efcc77fe700 1 --2- 172.21.15.40:0/48827476 >> [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324] conn(0x7efcb00ad560 0x7efcb00af9a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:38.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.100+0000 7efccc9af700 1 -- 172.21.15.40:0/48827476 <== osd.5 v2:172.21.15.113:6808/3666840324 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efcb00b0070 con 0x7efcb00ad560 2024-03-18T22:41:38.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.116+0000 7efccd9b1700 1 -- 172.21.15.40:0/48827476 --> [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efcc8102900 con 0x7efcb00ad560 2024-03-18T22:41:38.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.116+0000 7efccc9af700 1 -- 172.21.15.40:0/48827476 <== osd.5 v2:172.21.15.113:6808/3666840324 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7efcc8102900 con 0x7efcb00ad560 2024-03-18T22:41:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.117+0000 7efccd9b1700 1 -- 172.21.15.40:0/48827476 >> [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324] conn(0x7efcb00ad560 msgr2=0x7efcb00af9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.117+0000 7efccd9b1700 1 --2- 172.21.15.40:0/48827476 >> [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324] conn(0x7efcb00ad560 0x7efcb00af9a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.117+0000 7efccd9b1700 1 -- 172.21.15.40:0/48827476 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efcb00a2920 msgr2=0x7efcb00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.117+0000 7efccd9b1700 1 --2- 172.21.15.40:0/48827476 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efcb00a2920 0x7efcb00a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efcc80786f0 tx=0x7efcbc00b630 comp rx=0 tx=0).stop 2024-03-18T22:41:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.117+0000 7efccd9b1700 1 -- 172.21.15.40:0/48827476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc8101210 msgr2=0x7efcc819e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.117+0000 7efccd9b1700 1 --2- 172.21.15.40:0/48827476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efcc8101210 0x7efcc819e6b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efcb800e9f0 tx=0x7efcb800ed00 comp rx=0 tx=0).stop 2024-03-18T22:41:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.117+0000 7efcc77fe700 1 -- 172.21.15.40:0/48827476 reap_dead start 2024-03-18T22:41:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.117+0000 7efccd9b1700 1 -- 172.21.15.40:0/48827476 shutdown_connections 2024-03-18T22:41:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.117+0000 7efccd9b1700 1 -- 172.21.15.40:0/48827476 >> 172.21.15.40:0/48827476 conn(0x7efcc8078140 msgr2=0x7efcc806a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.117+0000 7efccd9b1700 1 -- 172.21.15.40:0/48827476 shutdown_connections 2024-03-18T22:41:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.117+0000 7efccd9b1700 1 -- 172.21.15.40:0/48827476 wait complete. 2024-03-18T22:41:39.119 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:38 smithi040 ceph-mon[62643]: pgmap v317: 129 pgs: 129 active+clean; 644 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T22:41:39.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:38 smithi113 ceph-mon[66775]: pgmap v317: 129 pgs: 129 active+clean; 644 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T22:41:39.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:38 smithi103 ceph-mon[66377]: pgmap v317: 129 pgs: 129 active+clean; 644 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T22:41:39.830 INFO:teuthology.orchestra.run.smithi040.stdout:107374182463 2024-03-18T22:41:39.830 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-18T22:41:39.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.778+0000 7f8bd4c49700 1 -- 172.21.15.40:0/451282247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bd0102d50 msgr2=0x7f8bd010f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:39.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.778+0000 7f8bd4c49700 1 --2- 172.21.15.40:0/451282247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bd0102d50 0x7f8bd010f1b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800b0a0 tx=0x7f8bc800b3b0 comp rx=0 tx=0).stop 2024-03-18T22:41:39.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.778+0000 7f8bd4c49700 1 -- 172.21.15.40:0/451282247 shutdown_connections 2024-03-18T22:41:39.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.778+0000 7f8bd4c49700 1 --2- 172.21.15.40:0/451282247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bd0102d50 0x7f8bd010f1b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:39.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.778+0000 7f8bd4c49700 1 --2- 172.21.15.40:0/451282247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd0108470 0x7f8bd0108850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:39.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.778+0000 7f8bd4c49700 1 --2- 172.21.15.40:0/451282247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bd0102390 0x7f8bd0102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:39.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.778+0000 7f8bd4c49700 1 -- 172.21.15.40:0/451282247 >> 172.21.15.40:0/451282247 conn(0x7f8bd00793d0 msgr2=0x7f8bd00797e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:39.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.778+0000 7f8bd4c49700 1 -- 172.21.15.40:0/451282247 shutdown_connections 2024-03-18T22:41:39.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.778+0000 7f8bd4c49700 1 -- 172.21.15.40:0/451282247 wait complete. 2024-03-18T22:41:39.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.779+0000 7f8bd4c49700 1 Processor -- start 2024-03-18T22:41:39.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.779+0000 7f8bd4c49700 1 -- start start 2024-03-18T22:41:39.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bd4c49700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bd0102390 0x7f8bd019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:39.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bd4c49700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd0102d50 0x7f8bd019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bd4c49700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bd0108470 0x7f8bd0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bd4c49700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd0112480 con 0x7f8bd0108470 2024-03-18T22:41:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bd4c49700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd0112300 con 0x7f8bd0102d50 2024-03-18T22:41:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bd4c49700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd0112600 con 0x7f8bd0102390 2024-03-18T22:41:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bce59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bd0102390 0x7f8bd019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bc7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd0102d50 0x7f8bd019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bced9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bd0108470 0x7f8bd0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bce59c700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bd0102390 0x7f8bd019e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:36326/0 (socket says 172.21.15.40:36326) 2024-03-18T22:41:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bce59c700 1 -- 172.21.15.40:0/4010011163 learned_addr learned my addr 172.21.15.40:0/4010011163 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bc7fff700 1 -- 172.21.15.40:0/4010011163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bd0102390 msgr2=0x7f8bd019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:39.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bc7fff700 1 --2- 172.21.15.40:0/4010011163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bd0102390 0x7f8bd019e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:39.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bc7fff700 1 -- 172.21.15.40:0/4010011163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bd0108470 msgr2=0x7f8bd0198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:39.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bc7fff700 1 --2- 172.21.15.40:0/4010011163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bd0108470 0x7f8bd0198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:39.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bc7fff700 1 -- 172.21.15.40:0/4010011163 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd0199020 con 0x7f8bd0102d50 2024-03-18T22:41:39.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bce59c700 1 --2- 172.21.15.40:0/4010011163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8bd0102390 0x7f8bd019e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:39.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.780+0000 7f8bced9d700 1 --2- 172.21.15.40:0/4010011163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bd0108470 0x7f8bd0198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:39.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.781+0000 7f8bc7fff700 1 --2- 172.21.15.40:0/4010011163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd0102d50 0x7f8bd019ebb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8bb800e9f0 tx=0x7f8bb800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:39.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.781+0000 7f8bc77fe700 1 -- 172.21.15.40:0/4010011163 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb800c9c0 con 0x7f8bd0102d50 2024-03-18T22:41:39.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.781+0000 7f8bc77fe700 1 -- 172.21.15.40:0/4010011163 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8bb800cb60 con 0x7f8bd0102d50 2024-03-18T22:41:39.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.781+0000 7f8bd4c49700 1 -- 172.21.15.40:0/4010011163 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd0199310 con 0x7f8bd0102d50 2024-03-18T22:41:39.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.781+0000 7f8bc77fe700 1 -- 172.21.15.40:0/4010011163 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8018680 con 0x7f8bd0102d50 2024-03-18T22:41:39.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.781+0000 7f8bd4c49700 1 -- 172.21.15.40:0/4010011163 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd01a55f0 con 0x7f8bd0102d50 2024-03-18T22:41:39.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.782+0000 7f8bd4c49700 1 -- 172.21.15.40:0/4010011163 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8bd0062a60 con 0x7f8bd0102d50 2024-03-18T22:41:39.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.783+0000 7f8bc77fe700 1 -- 172.21.15.40:0/4010011163 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8bb8010430 con 0x7f8bd0102d50 2024-03-18T22:41:39.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.784+0000 7f8bc77fe700 1 --2- 172.21.15.40:0/4010011163 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8bbc0a2920 0x7f8bbc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:39.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.784+0000 7f8bce59c700 1 --2- 172.21.15.40:0/4010011163 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8bbc0a2920 0x7f8bbc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:39.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.784+0000 7f8bc77fe700 1 -- 172.21.15.40:0/4010011163 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8bb80cf4d0 con 0x7f8bd0102d50 2024-03-18T22:41:39.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.784+0000 7f8bc77fe700 1 --2- 172.21.15.40:0/4010011163 >> [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822] conn(0x7f8bbc0ad560 0x7f8bbc0af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:39.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.784+0000 7f8bc77fe700 1 -- 172.21.15.40:0/4010011163 --> [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8bbc0b0070 con 0x7f8bbc0ad560 2024-03-18T22:41:39.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.784+0000 7f8bc77fe700 1 -- 172.21.15.40:0/4010011163 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f8bb80cf920 con 0x7f8bd0102d50 2024-03-18T22:41:39.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.784+0000 7f8bced9d700 1 --2- 172.21.15.40:0/4010011163 >> [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822] conn(0x7f8bbc0ad560 0x7f8bbc0af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.785+0000 7f8bce59c700 1 --2- 172.21.15.40:0/4010011163 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8bbc0a2920 0x7f8bbc0a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0011550 tx=0x7f8bc0009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.785+0000 7f8bced9d700 1 --2- 172.21.15.40:0/4010011163 >> [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822] conn(0x7f8bbc0ad560 0x7f8bbc0af9a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.786+0000 7f8bc77fe700 1 -- 172.21.15.40:0/4010011163 <== osd.2 v2:172.21.15.113:6800/954444822 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8bbc0b0070 con 0x7f8bbc0ad560 2024-03-18T22:41:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.803+0000 7f8bd4c49700 1 -- 172.21.15.40:0/4010011163 --> [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8bd0062ee0 con 0x7f8bbc0ad560 2024-03-18T22:41:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.803+0000 7f8bc77fe700 1 -- 172.21.15.40:0/4010011163 <== osd.2 v2:172.21.15.113:6800/954444822 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8bd0062ee0 con 0x7f8bbc0ad560 2024-03-18T22:41:39.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.803+0000 7f8bd4c49700 1 -- 172.21.15.40:0/4010011163 >> [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822] conn(0x7f8bbc0ad560 msgr2=0x7f8bbc0af9a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:39.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.803+0000 7f8bd4c49700 1 --2- 172.21.15.40:0/4010011163 >> [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822] conn(0x7f8bbc0ad560 0x7f8bbc0af9a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:39.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.804+0000 7f8bd4c49700 1 -- 172.21.15.40:0/4010011163 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8bbc0a2920 msgr2=0x7f8bbc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:39.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.804+0000 7f8bd4c49700 1 --2- 172.21.15.40:0/4010011163 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8bbc0a2920 0x7f8bbc0a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0011550 tx=0x7f8bc0009260 comp rx=0 tx=0).stop 2024-03-18T22:41:39.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.804+0000 7f8bd4c49700 1 -- 172.21.15.40:0/4010011163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd0102d50 msgr2=0x7f8bd019ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:39.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.804+0000 7f8bd4c49700 1 --2- 172.21.15.40:0/4010011163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bd0102d50 0x7f8bd019ebb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8bb800e9f0 tx=0x7f8bb800ed00 comp rx=0 tx=0).stop 2024-03-18T22:41:39.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.804+0000 7f8bced9d700 1 -- 172.21.15.40:0/4010011163 reap_dead start 2024-03-18T22:41:39.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.804+0000 7f8bd4c49700 1 -- 172.21.15.40:0/4010011163 shutdown_connections 2024-03-18T22:41:39.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.804+0000 7f8bd4c49700 1 -- 172.21.15.40:0/4010011163 >> 172.21.15.40:0/4010011163 conn(0x7f8bd00793d0 msgr2=0x7f8bd0101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:39.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.804+0000 7f8bd4c49700 1 -- 172.21.15.40:0/4010011163 shutdown_connections 2024-03-18T22:41:39.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:38.804+0000 7f8bd4c49700 1 -- 172.21.15.40:0/4010011163 wait complete. 2024-03-18T22:41:40.474 INFO:teuthology.orchestra.run.smithi040.stdout:77309411396 2024-03-18T22:41:40.474 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-18T22:41:41.103 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:40 smithi040 ceph-mon[62643]: pgmap v318: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-18T22:41:41.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:40 smithi113 ceph-mon[66775]: pgmap v318: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-18T22:41:41.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:40 smithi103 ceph-mon[66377]: pgmap v318: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-18T22:41:41.981 INFO:teuthology.orchestra.run.smithi040.stdout:163208757302 2024-03-18T22:41:41.982 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-18T22:41:42.270 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:41 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:41:42.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:41 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:41:42.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:41 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:41:43.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:42 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-03-18T22:41:43.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:42 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2024-03-18T22:41:43.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:42 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-18T22:41:43.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:42 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-03-18T22:41:43.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:42 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-18T22:41:43.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:42 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-03-18T22:41:43.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:42 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-18T22:41:43.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:42 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-18T22:41:43.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:42 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-03-18T22:41:43.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:42 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]: dispatch 2024-03-18T22:41:43.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:42 smithi103 ceph-mon[66377]: pgmap v319: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T22:41:43.197 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-03-18T22:41:43.197 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2024-03-18T22:41:43.197 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-18T22:41:43.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-03-18T22:41:43.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-18T22:41:43.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-03-18T22:41:43.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-18T22:41:43.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-18T22:41:43.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-03-18T22:41:43.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:42 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]: dispatch 2024-03-18T22:41:43.198 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:42 smithi040 ceph-mon[62643]: pgmap v319: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T22:41:43.261 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:42 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-03-18T22:41:43.262 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:42 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2024-03-18T22:41:43.262 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:42 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-18T22:41:43.262 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:42 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-03-18T22:41:43.262 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:42 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-18T22:41:43.262 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:42 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-03-18T22:41:43.262 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:42 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-18T22:41:43.262 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:42 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-18T22:41:43.262 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:42 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-03-18T22:41:43.262 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:42 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]: dispatch 2024-03-18T22:41:43.262 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:42 smithi113 ceph-mon[66775]: pgmap v319: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T22:41:43.520 INFO:teuthology.orchestra.run.smithi040.stdout:197568495666 2024-03-18T22:41:43.520 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-18T22:41:43.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.862+0000 7fbe1754d700 1 -- 172.21.15.40:0/1879178301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe101023c0 msgr2=0x7fbe10102840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:43.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.862+0000 7fbe1754d700 1 --2- 172.21.15.40:0/1879178301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe101023c0 0x7fbe10102840 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009990 tx=0x7fbe0c009ca0 comp rx=0 tx=0).stop 2024-03-18T22:41:43.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.862+0000 7fbe1754d700 1 -- 172.21.15.40:0/1879178301 shutdown_connections 2024-03-18T22:41:43.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.862+0000 7fbe1754d700 1 --2- 172.21.15.40:0/1879178301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe10102d80 0x7fbe1010f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:43.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.862+0000 7fbe1754d700 1 --2- 172.21.15.40:0/1879178301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe10108430 0x7fbe10108810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:43.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.862+0000 7fbe1754d700 1 --2- 172.21.15.40:0/1879178301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe101023c0 0x7fbe10102840 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:43.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.862+0000 7fbe1754d700 1 -- 172.21.15.40:0/1879178301 >> 172.21.15.40:0/1879178301 conn(0x7fbe100793e0 msgr2=0x7fbe100797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.862+0000 7fbe1754d700 1 -- 172.21.15.40:0/1879178301 shutdown_connections 2024-03-18T22:41:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.862+0000 7fbe1754d700 1 -- 172.21.15.40:0/1879178301 wait complete. 2024-03-18T22:41:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.863+0000 7fbe1754d700 1 Processor -- start 2024-03-18T22:41:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.863+0000 7fbe1754d700 1 -- start start 2024-03-18T22:41:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.863+0000 7fbe1754d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe101023c0 0x7fbe1019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.863+0000 7fbe1754d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe10102d80 0x7fbe1019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.863+0000 7fbe1754d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe10108430 0x7fbe10198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.863+0000 7fbe1754d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbe10112500 con 0x7fbe101023c0 2024-03-18T22:41:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.863+0000 7fbe1754d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbe10112380 con 0x7fbe10108430 2024-03-18T22:41:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.863+0000 7fbe1754d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbe10112680 con 0x7fbe10102d80 2024-03-18T22:41:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.863+0000 7fbe152e9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe101023c0 0x7fbe1019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:43.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.863+0000 7fbe152e9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe101023c0 0x7fbe1019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33772/0 (socket says 172.21.15.40:33772) 2024-03-18T22:41:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.863+0000 7fbe152e9700 1 -- 172.21.15.40:0/22646551 learned_addr learned my addr 172.21.15.40:0/22646551 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.863+0000 7fbe14ae8700 1 --2- 172.21.15.40:0/22646551 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe10102d80 0x7fbe1019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.863+0000 7fbe15aea700 1 --2- 172.21.15.40:0/22646551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe10108430 0x7fbe10198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.864+0000 7fbe152e9700 1 -- 172.21.15.40:0/22646551 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe10102d80 msgr2=0x7fbe1019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.864+0000 7fbe152e9700 1 --2- 172.21.15.40:0/22646551 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe10102d80 0x7fbe1019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.864+0000 7fbe152e9700 1 -- 172.21.15.40:0/22646551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe10108430 msgr2=0x7fbe10198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.864+0000 7fbe152e9700 1 --2- 172.21.15.40:0/22646551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe10108430 0x7fbe10198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.864+0000 7fbe152e9700 1 -- 172.21.15.40:0/22646551 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe10198fe0 con 0x7fbe101023c0 2024-03-18T22:41:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.864+0000 7fbe14ae8700 1 --2- 172.21.15.40:0/22646551 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe10102d80 0x7fbe1019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.864+0000 7fbe152e9700 1 --2- 172.21.15.40:0/22646551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe101023c0 0x7fbe1019e680 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbe0000e9a0 tx=0x7fbe0000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.864+0000 7fbe067fc700 1 -- 172.21.15.40:0/22646551 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0000ca10 con 0x7fbe101023c0 2024-03-18T22:41:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.864+0000 7fbe1754d700 1 -- 172.21.15.40:0/22646551 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe101992d0 con 0x7fbe101023c0 2024-03-18T22:41:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.864+0000 7fbe067fc700 1 -- 172.21.15.40:0/22646551 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe0000cbb0 con 0x7fbe101023c0 2024-03-18T22:41:43.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.864+0000 7fbe067fc700 1 -- 172.21.15.40:0/22646551 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe00018680 con 0x7fbe101023c0 2024-03-18T22:41:43.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.864+0000 7fbe1754d700 1 -- 172.21.15.40:0/22646551 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe101a54e0 con 0x7fbe101023c0 2024-03-18T22:41:43.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.865+0000 7fbe1754d700 1 -- 172.21.15.40:0/22646551 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbe10062a20 con 0x7fbe101023c0 2024-03-18T22:41:43.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.866+0000 7fbe067fc700 1 -- 172.21.15.40:0/22646551 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fbe00010430 con 0x7fbe101023c0 2024-03-18T22:41:43.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.867+0000 7fbe067fc700 1 --2- 172.21.15.40:0/22646551 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbdfc0a2920 0x7fbdfc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:43.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.867+0000 7fbe14ae8700 1 --2- 172.21.15.40:0/22646551 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbdfc0a2920 0x7fbdfc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:43.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.867+0000 7fbe067fc700 1 -- 172.21.15.40:0/22646551 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9092+0+0 (secure 0 0 0) 0x7fbe000cf5d0 con 0x7fbe101023c0 2024-03-18T22:41:43.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.867+0000 7fbe067fc700 1 --2- 172.21.15.40:0/22646551 >> [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460] conn(0x7fbdfc0ada80 0x7fbdfc0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:43.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.867+0000 7fbe067fc700 1 -- 172.21.15.40:0/22646551 --> [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbe0001bec0 con 0x7fbdfc0ada80 2024-03-18T22:41:43.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.867+0000 7fbe067fc700 1 -- 172.21.15.40:0/22646551 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fbe000cf980 con 0x7fbe101023c0 2024-03-18T22:41:43.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.867+0000 7fbe14ae8700 1 --2- 172.21.15.40:0/22646551 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbdfc0a2920 0x7fbdfc0a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00b2e0 tx=0x7fbe0c01a040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:43.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.867+0000 7fbe15aea700 1 --2- 172.21.15.40:0/22646551 >> [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460] conn(0x7fbdfc0ada80 0x7fbdfc0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:43.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.868+0000 7fbe15aea700 1 --2- 172.21.15.40:0/22646551 >> [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460] conn(0x7fbdfc0ada80 0x7fbdfc0aff20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:43.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.869+0000 7fbe067fc700 1 -- 172.21.15.40:0/22646551 <== osd.4 v2:172.21.15.103:6808/3777182460 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbe0001bec0 con 0x7fbdfc0ada80 2024-03-18T22:41:43.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.885+0000 7fbe1754d700 1 -- 172.21.15.40:0/22646551 --> [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbe10062e20 con 0x7fbdfc0ada80 2024-03-18T22:41:43.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.885+0000 7fbe067fc700 1 -- 172.21.15.40:0/22646551 <== osd.4 v2:172.21.15.103:6808/3777182460 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbe10062e20 con 0x7fbdfc0ada80 2024-03-18T22:41:43.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.886+0000 7fbe1754d700 1 -- 172.21.15.40:0/22646551 >> [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460] conn(0x7fbdfc0ada80 msgr2=0x7fbdfc0aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:43.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.886+0000 7fbe1754d700 1 --2- 172.21.15.40:0/22646551 >> [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460] conn(0x7fbdfc0ada80 0x7fbdfc0aff20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:43.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.886+0000 7fbe1754d700 1 -- 172.21.15.40:0/22646551 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbdfc0a2920 msgr2=0x7fbdfc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:43.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.886+0000 7fbe1754d700 1 --2- 172.21.15.40:0/22646551 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbdfc0a2920 0x7fbdfc0a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00b2e0 tx=0x7fbe0c01a040 comp rx=0 tx=0).stop 2024-03-18T22:41:43.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.886+0000 7fbe1754d700 1 -- 172.21.15.40:0/22646551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe101023c0 msgr2=0x7fbe1019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:43.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.886+0000 7fbe1754d700 1 --2- 172.21.15.40:0/22646551 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe101023c0 0x7fbe1019e680 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbe0000e9a0 tx=0x7fbe0000ecb0 comp rx=0 tx=0).stop 2024-03-18T22:41:43.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.886+0000 7fbe15aea700 1 -- 172.21.15.40:0/22646551 reap_dead start 2024-03-18T22:41:43.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.886+0000 7fbe1754d700 1 -- 172.21.15.40:0/22646551 shutdown_connections 2024-03-18T22:41:43.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.886+0000 7fbe1754d700 1 -- 172.21.15.40:0/22646551 >> 172.21.15.40:0/22646551 conn(0x7fbe100793e0 msgr2=0x7fbe101010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:43.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.886+0000 7fbe1754d700 1 -- 172.21.15.40:0/22646551 shutdown_connections 2024-03-18T22:41:43.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:43.886+0000 7fbe1754d700 1 -- 172.21.15.40:0/22646551 wait complete. 2024-03-18T22:41:44.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2024-03-18T22:41:44.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2024-03-18T22:41:44.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-18T22:41:44.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-03-18T22:41:44.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-18T22:41:44.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-03-18T22:41:44.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-18T22:41:44.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-18T22:41:44.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-03-18T22:41:44.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]': finished 2024-03-18T22:41:44.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:44 smithi040 ceph-mon[62643]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T22:41:44.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:41:44.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:41:44.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:41:44.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:41:44.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2024-03-18T22:41:44.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2024-03-18T22:41:44.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-18T22:41:44.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-03-18T22:41:44.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-18T22:41:44.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-03-18T22:41:44.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-18T22:41:44.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-18T22:41:44.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-03-18T22:41:44.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]': finished 2024-03-18T22:41:44.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:44 smithi113 ceph-mon[66775]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T22:41:44.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:41:44.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:41:44.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:41:44.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:41:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2024-03-18T22:41:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2024-03-18T22:41:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-18T22:41:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-03-18T22:41:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-18T22:41:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-03-18T22:41:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-18T22:41:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-18T22:41:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-03-18T22:41:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]': finished 2024-03-18T22:41:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:44 smithi103 ceph-mon[66377]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T22:41:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:41:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:41:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:41:44.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:41:44.698 INFO:teuthology.orchestra.run.smithi040.stdout:253403070502 2024-03-18T22:41:44.699 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-18T22:41:45.116 INFO:teuthology.orchestra.run.smithi040.stdout:141733920829 2024-03-18T22:41:45.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-18T22:41:45.154 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:45 smithi040 ceph-mon[62643]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T22:41:45.154 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:45 smithi040 ceph-mon[62643]: pgmap v322: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-18T22:41:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:45 smithi113 ceph-mon[66775]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T22:41:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:45 smithi113 ceph-mon[66775]: pgmap v322: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-18T22:41:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:45 smithi103 ceph-mon[66377]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T22:41:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:45 smithi103 ceph-mon[66377]: pgmap v322: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-18T22:41:45.483 INFO:teuthology.orchestra.run.smithi040.stdout:34359738447 2024-03-18T22:41:45.483 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-18T22:41:46.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.000+0000 7f0773582700 1 -- 172.21.15.40:0/2487393239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f076c106590 msgr2=0x7f076c10e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.000+0000 7f0773582700 1 --2- 172.21.15.40:0/2487393239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f076c106590 0x7f076c10e160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f075c0098f0 tx=0x7f075c009c00 comp rx=0 tx=0).stop 2024-03-18T22:41:46.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.000+0000 7f0773582700 1 -- 172.21.15.40:0/2487393239 shutdown_connections 2024-03-18T22:41:46.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.000+0000 7f0773582700 1 --2- 172.21.15.40:0/2487393239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f076c106590 0x7f076c10e160 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.000+0000 7f0773582700 1 --2- 172.21.15.40:0/2487393239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f076c101c70 0x7f076c102050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.000+0000 7f0773582700 1 --2- 172.21.15.40:0/2487393239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f076c102620 0x7f076c105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.000+0000 7f0773582700 1 -- 172.21.15.40:0/2487393239 >> 172.21.15.40:0/2487393239 conn(0x7f076c077810 msgr2=0x7f076c0fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:46.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.000+0000 7f0773582700 1 -- 172.21.15.40:0/2487393239 shutdown_connections 2024-03-18T22:41:46.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.000+0000 7f0773582700 1 -- 172.21.15.40:0/2487393239 wait complete. 2024-03-18T22:41:46.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0773582700 1 Processor -- start 2024-03-18T22:41:46.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0773582700 1 -- start start 2024-03-18T22:41:46.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0773582700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f076c101c70 0x7f076c0fccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:46.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0773582700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f076c102620 0x7f076c0fd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:46.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0773582700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f076c106590 0x7f076c0fd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:46.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0773582700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f076c111430 con 0x7f076c106590 2024-03-18T22:41:46.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0773582700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f076c1112b0 con 0x7f076c101c70 2024-03-18T22:41:46.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0773582700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f076c1115b0 con 0x7f076c102620 2024-03-18T22:41:46.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0771b1f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f076c106590 0x7f076c0fd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:46.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0770b1d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f076c102620 0x7f076c0fd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:46.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f077131e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f076c101c70 0x7f076c0fccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:46.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0771b1f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f076c106590 0x7f076c0fd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33830/0 (socket says 172.21.15.40:33830) 2024-03-18T22:41:46.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0771b1f700 1 -- 172.21.15.40:0/3153974340 learned_addr learned my addr 172.21.15.40:0/3153974340 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:46.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0771b1f700 1 -- 172.21.15.40:0/3153974340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f076c102620 msgr2=0x7f076c0fd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0771b1f700 1 --2- 172.21.15.40:0/3153974340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f076c102620 0x7f076c0fd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0771b1f700 1 -- 172.21.15.40:0/3153974340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f076c101c70 msgr2=0x7f076c0fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0771b1f700 1 --2- 172.21.15.40:0/3153974340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f076c101c70 0x7f076c0fccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f0771b1f700 1 -- 172.21.15.40:0/3153974340 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076c1a6fc0 con 0x7f076c106590 2024-03-18T22:41:46.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.001+0000 7f077131e700 1 --2- 172.21.15.40:0/3153974340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f076c101c70 0x7f076c0fccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:46.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.002+0000 7f0771b1f700 1 --2- 172.21.15.40:0/3153974340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f076c106590 0x7f076c0fd770 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f075c009810 tx=0x7f075c00fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:46.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.002+0000 7f076a7fc700 1 -- 172.21.15.40:0/3153974340 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075c0050c0 con 0x7f076c106590 2024-03-18T22:41:46.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.002+0000 7f0773582700 1 -- 172.21.15.40:0/3153974340 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f076c1a7250 con 0x7f076c106590 2024-03-18T22:41:46.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.002+0000 7f076a7fc700 1 -- 172.21.15.40:0/3153974340 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f075c00b560 con 0x7f076c106590 2024-03-18T22:41:46.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.002+0000 7f076a7fc700 1 -- 172.21.15.40:0/3153974340 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075c003a20 con 0x7f076c106590 2024-03-18T22:41:46.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.002+0000 7f0773582700 1 -- 172.21.15.40:0/3153974340 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f076c1a7700 con 0x7f076c106590 2024-03-18T22:41:46.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.003+0000 7f0773582700 1 -- 172.21.15.40:0/3153974340 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f076c105710 con 0x7f076c106590 2024-03-18T22:41:46.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.004+0000 7f076a7fc700 1 -- 172.21.15.40:0/3153974340 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f075c00b710 con 0x7f076c106590 2024-03-18T22:41:46.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.004+0000 7f076a7fc700 1 --2- 172.21.15.40:0/3153974340 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f07540a2920 0x7f07540a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:46.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.005+0000 7f077131e700 1 --2- 172.21.15.40:0/3153974340 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f07540a2920 0x7f07540a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:46.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.005+0000 7f076a7fc700 1 -- 172.21.15.40:0/3153974340 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f075c0d0470 con 0x7f076c106590 2024-03-18T22:41:46.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.005+0000 7f076a7fc700 1 --2- 172.21.15.40:0/3153974340 >> [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401] conn(0x7f07540ada70 0x7f07540aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:46.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.005+0000 7f076a7fc700 1 -- 172.21.15.40:0/3153974340 --> [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07540b05e0 con 0x7f07540ada70 2024-03-18T22:41:46.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.005+0000 7f076a7fc700 1 -- 172.21.15.40:0/3153974340 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f075c0d08c0 con 0x7f076c106590 2024-03-18T22:41:46.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.005+0000 7f0770b1d700 1 --2- 172.21.15.40:0/3153974340 >> [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401] conn(0x7f07540ada70 0x7f07540aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:46.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.006+0000 7f077131e700 1 --2- 172.21.15.40:0/3153974340 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f07540a2920 0x7f07540a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0764009480 tx=0x7f076400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:46.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.006+0000 7f0770b1d700 1 --2- 172.21.15.40:0/3153974340 >> [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401] conn(0x7f07540ada70 0x7f07540aff10 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:46.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.007+0000 7f076a7fc700 1 -- 172.21.15.40:0/3153974340 <== osd.9 v2:172.21.15.40:6826/4067616401 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f07540b05e0 con 0x7f07540ada70 2024-03-18T22:41:46.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.024+0000 7f0773582700 1 -- 172.21.15.40:0/3153974340 --> [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f076c102a20 con 0x7f07540ada70 2024-03-18T22:41:46.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.024+0000 7f076a7fc700 1 -- 172.21.15.40:0/3153974340 <== osd.9 v2:172.21.15.40:6826/4067616401 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f076c102a20 con 0x7f07540ada70 2024-03-18T22:41:46.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.025+0000 7f0773582700 1 -- 172.21.15.40:0/3153974340 >> [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401] conn(0x7f07540ada70 msgr2=0x7f07540aff10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.025+0000 7f0773582700 1 --2- 172.21.15.40:0/3153974340 >> [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401] conn(0x7f07540ada70 0x7f07540aff10 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.025+0000 7f0773582700 1 -- 172.21.15.40:0/3153974340 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f07540a2920 msgr2=0x7f07540a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.025+0000 7f0773582700 1 --2- 172.21.15.40:0/3153974340 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f07540a2920 0x7f07540a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0764009480 tx=0x7f076400b040 comp rx=0 tx=0).stop 2024-03-18T22:41:46.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.025+0000 7f0773582700 1 -- 172.21.15.40:0/3153974340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f076c106590 msgr2=0x7f076c0fd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.025+0000 7f0773582700 1 --2- 172.21.15.40:0/3153974340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f076c106590 0x7f076c0fd770 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f075c009810 tx=0x7f075c00fa00 comp rx=0 tx=0).stop 2024-03-18T22:41:46.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.025+0000 7f0771b1f700 1 -- 172.21.15.40:0/3153974340 reap_dead start 2024-03-18T22:41:46.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.025+0000 7f0773582700 1 -- 172.21.15.40:0/3153974340 shutdown_connections 2024-03-18T22:41:46.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.025+0000 7f0773582700 1 -- 172.21.15.40:0/3153974340 >> 172.21.15.40:0/3153974340 conn(0x7f076c077810 msgr2=0x7f076c100c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:46.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.025+0000 7f0773582700 1 -- 172.21.15.40:0/3153974340 shutdown_connections 2024-03-18T22:41:46.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.025+0000 7f0773582700 1 -- 172.21.15.40:0/3153974340 wait complete. 2024-03-18T22:41:46.036 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:46 smithi040 ceph-mon[62643]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-18T22:41:46.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.880+0000 7f524d8ca700 1 -- 172.21.15.40:0/202673805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5248107250 msgr2=0x7f5248107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.880+0000 7f524d8ca700 1 --2- 172.21.15.40:0/202673805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5248107250 0x7f5248107630 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5238009990 tx=0x7f5238009ca0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.880+0000 7f524d8ca700 1 -- 172.21.15.40:0/202673805 shutdown_connections 2024-03-18T22:41:46.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.880+0000 7f524d8ca700 1 --2- 172.21.15.40:0/202673805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5248101bb0 0x7f524810e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.880+0000 7f524d8ca700 1 --2- 172.21.15.40:0/202673805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5248107250 0x7f5248107630 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.880+0000 7f524d8ca700 1 --2- 172.21.15.40:0/202673805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5248101210 0x7f5248101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.880+0000 7f524d8ca700 1 -- 172.21.15.40:0/202673805 >> 172.21.15.40:0/202673805 conn(0x7f5248078140 msgr2=0x7f5248078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:46.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.880+0000 7f524d8ca700 1 -- 172.21.15.40:0/202673805 shutdown_connections 2024-03-18T22:41:46.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.880+0000 7f524d8ca700 1 -- 172.21.15.40:0/202673805 wait complete. 2024-03-18T22:41:46.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.881+0000 7f524d8ca700 1 Processor -- start 2024-03-18T22:41:46.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.881+0000 7f524d8ca700 1 -- start start 2024-03-18T22:41:46.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f524d8ca700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5248101210 0x7f5248199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:46.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f524d8ca700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5248101bb0 0x7f5248199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:46.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f524d8ca700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5248107250 0x7f52481a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:46.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f524d8ca700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5248111360 con 0x7f5248101bb0 2024-03-18T22:41:46.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f524d8ca700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f52481111e0 con 0x7f5248101210 2024-03-18T22:41:46.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f524d8ca700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f52481114e0 con 0x7f5248107250 2024-03-18T22:41:46.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f52467fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5248101bb0 0x7f5248199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:46.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f5246ffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5248101210 0x7f5248199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:46.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f52467fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5248101bb0 0x7f5248199d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33798/0 (socket says 172.21.15.40:33798) 2024-03-18T22:41:46.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f52467fc700 1 -- 172.21.15.40:0/368218535 learned_addr learned my addr 172.21.15.40:0/368218535 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:46.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f52477fe700 1 --2- 172.21.15.40:0/368218535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5248107250 0x7f52481a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:46.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f52467fc700 1 -- 172.21.15.40:0/368218535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5248107250 msgr2=0x7f52481a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f52467fc700 1 --2- 172.21.15.40:0/368218535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5248107250 0x7f52481a48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f52467fc700 1 -- 172.21.15.40:0/368218535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5248101210 msgr2=0x7f5248199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f52467fc700 1 --2- 172.21.15.40:0/368218535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5248101210 0x7f5248199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f52467fc700 1 -- 172.21.15.40:0/368218535 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52481a4e00 con 0x7f5248101bb0 2024-03-18T22:41:46.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f5246ffd700 1 --2- 172.21.15.40:0/368218535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5248101210 0x7f5248199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:46.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f52467fc700 1 --2- 172.21.15.40:0/368218535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5248101bb0 0x7f5248199d80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f523c00e9c0 tx=0x7f523c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:46.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f524c8c8700 1 -- 172.21.15.40:0/368218535 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c00cbd0 con 0x7f5248101bb0 2024-03-18T22:41:46.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f524c8c8700 1 -- 172.21.15.40:0/368218535 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f523c00ee50 con 0x7f5248101bb0 2024-03-18T22:41:46.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f524d8ca700 1 -- 172.21.15.40:0/368218535 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52481a4fd0 con 0x7f5248101bb0 2024-03-18T22:41:46.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f524c8c8700 1 -- 172.21.15.40:0/368218535 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c005550 con 0x7f5248101bb0 2024-03-18T22:41:46.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.882+0000 7f524d8ca700 1 -- 172.21.15.40:0/368218535 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52481a5490 con 0x7f5248101bb0 2024-03-18T22:41:46.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.883+0000 7f524d8ca700 1 -- 172.21.15.40:0/368218535 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f52480629a0 con 0x7f5248101bb0 2024-03-18T22:41:46.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.884+0000 7f524c8c8700 1 -- 172.21.15.40:0/368218535 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f523c004750 con 0x7f5248101bb0 2024-03-18T22:41:46.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.885+0000 7f524c8c8700 1 --2- 172.21.15.40:0/368218535 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f52300a2920 0x7f52300a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:46.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.885+0000 7f5246ffd700 1 --2- 172.21.15.40:0/368218535 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f52300a2920 0x7f52300a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:46.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.885+0000 7f524c8c8700 1 -- 172.21.15.40:0/368218535 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f523c0cf520 con 0x7f5248101bb0 2024-03-18T22:41:46.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.885+0000 7f524c8c8700 1 --2- 172.21.15.40:0/368218535 >> [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819] conn(0x7f52300ada80 0x7f52300aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:46.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.885+0000 7f524c8c8700 1 -- 172.21.15.40:0/368218535 --> [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f52300b05f0 con 0x7f52300ada80 2024-03-18T22:41:46.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.885+0000 7f524c8c8700 1 -- 172.21.15.40:0/368218535 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f523c0cf970 con 0x7f5248101bb0 2024-03-18T22:41:46.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.885+0000 7f52477fe700 1 --2- 172.21.15.40:0/368218535 >> [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819] conn(0x7f52300ada80 0x7f52300aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:46.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.886+0000 7f5246ffd700 1 --2- 172.21.15.40:0/368218535 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f52300a2920 0x7f52300a4de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f523800b2e0 tx=0x7f523801a040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:46.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.886+0000 7f52477fe700 1 --2- 172.21.15.40:0/368218535 >> [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819] conn(0x7f52300ada80 0x7f52300aff20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:46.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.887+0000 7f524c8c8700 1 -- 172.21.15.40:0/368218535 <== osd.10 v2:172.21.15.103:6824/1414934819 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f52300b05f0 con 0x7f52300ada80 2024-03-18T22:41:46.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.904+0000 7f524d8ca700 1 -- 172.21.15.40:0/368218535 --> [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5248062dd0 con 0x7f52300ada80 2024-03-18T22:41:46.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.904+0000 7f524c8c8700 1 -- 172.21.15.40:0/368218535 <== osd.10 v2:172.21.15.103:6824/1414934819 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5248062dd0 con 0x7f52300ada80 2024-03-18T22:41:46.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.905+0000 7f524d8ca700 1 -- 172.21.15.40:0/368218535 >> [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819] conn(0x7f52300ada80 msgr2=0x7f52300aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.905+0000 7f524d8ca700 1 --2- 172.21.15.40:0/368218535 >> [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819] conn(0x7f52300ada80 0x7f52300aff20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.905+0000 7f524d8ca700 1 -- 172.21.15.40:0/368218535 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f52300a2920 msgr2=0x7f52300a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.905+0000 7f524d8ca700 1 --2- 172.21.15.40:0/368218535 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f52300a2920 0x7f52300a4de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f523800b2e0 tx=0x7f523801a040 comp rx=0 tx=0).stop 2024-03-18T22:41:46.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.905+0000 7f524d8ca700 1 -- 172.21.15.40:0/368218535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5248101bb0 msgr2=0x7f5248199d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.905+0000 7f524d8ca700 1 --2- 172.21.15.40:0/368218535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5248101bb0 0x7f5248199d80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f523c00e9c0 tx=0x7f523c00ecd0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.905+0000 7f52477fe700 1 -- 172.21.15.40:0/368218535 reap_dead start 2024-03-18T22:41:46.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.905+0000 7f524d8ca700 1 -- 172.21.15.40:0/368218535 shutdown_connections 2024-03-18T22:41:46.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.905+0000 7f524d8ca700 1 -- 172.21.15.40:0/368218535 >> 172.21.15.40:0/368218535 conn(0x7f5248078140 msgr2=0x7f52481000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:46.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.905+0000 7f524d8ca700 1 -- 172.21.15.40:0/368218535 shutdown_connections 2024-03-18T22:41:46.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:45.905+0000 7f524d8ca700 1 -- 172.21.15.40:0/368218535 wait complete. 2024-03-18T22:41:46.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:46 smithi113 ceph-mon[66775]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-18T22:41:46.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:46 smithi103 ceph-mon[66377]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-18T22:41:46.732 INFO:teuthology.orchestra.run.smithi040.stdout:309237645343 2024-03-18T22:41:46.732 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-18T22:41:46.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.939+0000 7fbd72baa700 1 -- 172.21.15.40:0/1296449725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd6c101bb0 msgr2=0x7fbd6c10e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.939+0000 7fbd72baa700 1 --2- 172.21.15.40:0/1296449725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd6c101bb0 0x7fbd6c10e090 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbd64009990 tx=0x7fbd64009ca0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.939+0000 7fbd72baa700 1 -- 172.21.15.40:0/1296449725 shutdown_connections 2024-03-18T22:41:46.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.939+0000 7fbd72baa700 1 --2- 172.21.15.40:0/1296449725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd6c101bb0 0x7fbd6c10e090 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.939+0000 7fbd72baa700 1 --2- 172.21.15.40:0/1296449725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd6c107250 0x7fbd6c107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.939+0000 7fbd72baa700 1 --2- 172.21.15.40:0/1296449725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd6c101210 0x7fbd6c101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.939+0000 7fbd72baa700 1 -- 172.21.15.40:0/1296449725 >> 172.21.15.40:0/1296449725 conn(0x7fbd6c078140 msgr2=0x7fbd6c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:46.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.939+0000 7fbd72baa700 1 -- 172.21.15.40:0/1296449725 shutdown_connections 2024-03-18T22:41:46.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.939+0000 7fbd72baa700 1 -- 172.21.15.40:0/1296449725 wait complete. 2024-03-18T22:41:46.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.940+0000 7fbd72baa700 1 Processor -- start 2024-03-18T22:41:46.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.940+0000 7fbd72baa700 1 -- start start 2024-03-18T22:41:46.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.940+0000 7fbd72baa700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd6c101210 0x7fbd6c1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:46.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.940+0000 7fbd72baa700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd6c101bb0 0x7fbd6c199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.940+0000 7fbd72baa700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd6c107250 0x7fbd6c1a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.940+0000 7fbd72baa700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6c111350 con 0x7fbd6c107250 2024-03-18T22:41:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.940+0000 7fbd72baa700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6c1111d0 con 0x7fbd6c101bb0 2024-03-18T22:41:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.940+0000 7fbd72baa700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6c1114d0 con 0x7fbd6c101210 2024-03-18T22:41:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.940+0000 7fbd71147700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd6c107250 0x7fbd6c1a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.940+0000 7fbd71147700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd6c107250 0x7fbd6c1a4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33844/0 (socket says 172.21.15.40:33844) 2024-03-18T22:41:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.940+0000 7fbd71147700 1 -- 172.21.15.40:0/3462246161 learned_addr learned my addr 172.21.15.40:0/3462246161 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.940+0000 7fbd6bfff700 1 --2- 172.21.15.40:0/3462246161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd6c101bb0 0x7fbd6c199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.940+0000 7fbd70946700 1 --2- 172.21.15.40:0/3462246161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd6c101210 0x7fbd6c1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.941+0000 7fbd71147700 1 -- 172.21.15.40:0/3462246161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd6c101210 msgr2=0x7fbd6c1997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.941+0000 7fbd71147700 1 --2- 172.21.15.40:0/3462246161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbd6c101210 0x7fbd6c1997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.941+0000 7fbd71147700 1 -- 172.21.15.40:0/3462246161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd6c101bb0 msgr2=0x7fbd6c199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.941+0000 7fbd71147700 1 --2- 172.21.15.40:0/3462246161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd6c101bb0 0x7fbd6c199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.941+0000 7fbd71147700 1 -- 172.21.15.40:0/3462246161 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd6c1a4d70 con 0x7fbd6c107250 2024-03-18T22:41:46.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.941+0000 7fbd71147700 1 --2- 172.21.15.40:0/3462246161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd6c107250 0x7fbd6c1a4830 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbd64004790 tx=0x7fbd640047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:46.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.941+0000 7fbd69ffb700 1 -- 172.21.15.40:0/3462246161 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd64027070 con 0x7fbd6c107250 2024-03-18T22:41:46.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.941+0000 7fbd72baa700 1 -- 172.21.15.40:0/3462246161 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd6c1a4fa0 con 0x7fbd6c107250 2024-03-18T22:41:46.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.941+0000 7fbd69ffb700 1 -- 172.21.15.40:0/3462246161 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd64022470 con 0x7fbd6c107250 2024-03-18T22:41:46.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.941+0000 7fbd69ffb700 1 -- 172.21.15.40:0/3462246161 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6400b610 con 0x7fbd6c107250 2024-03-18T22:41:46.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.941+0000 7fbd72baa700 1 -- 172.21.15.40:0/3462246161 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd6c1a53f0 con 0x7fbd6c107250 2024-03-18T22:41:46.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.942+0000 7fbd72baa700 1 -- 172.21.15.40:0/3462246161 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbd6c0629a0 con 0x7fbd6c107250 2024-03-18T22:41:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.943+0000 7fbd69ffb700 1 -- 172.21.15.40:0/3462246161 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fbd6402a070 con 0x7fbd6c107250 2024-03-18T22:41:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.944+0000 7fbd69ffb700 1 --2- 172.21.15.40:0/3462246161 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbd580a2920 0x7fbd580a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.944+0000 7fbd70946700 1 --2- 172.21.15.40:0/3462246161 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbd580a2920 0x7fbd580a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.944+0000 7fbd69ffb700 1 -- 172.21.15.40:0/3462246161 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbd640d04c0 con 0x7fbd6c107250 2024-03-18T22:41:46.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.944+0000 7fbd69ffb700 1 --2- 172.21.15.40:0/3462246161 >> [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261] conn(0x7fbd580ada80 0x7fbd580aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.944+0000 7fbd69ffb700 1 -- 172.21.15.40:0/3462246161 --> [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbd640095d0 con 0x7fbd580ada80 2024-03-18T22:41:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.944+0000 7fbd69ffb700 1 -- 172.21.15.40:0/3462246161 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fbd640d0870 con 0x7fbd6c107250 2024-03-18T22:41:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.944+0000 7fbd6bfff700 1 --2- 172.21.15.40:0/3462246161 >> [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261] conn(0x7fbd580ada80 0x7fbd580aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.944+0000 7fbd70946700 1 --2- 172.21.15.40:0/3462246161 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbd580a2920 0x7fbd580a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00f540 tx=0x7fbd5c00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.945+0000 7fbd6bfff700 1 --2- 172.21.15.40:0/3462246161 >> [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261] conn(0x7fbd580ada80 0x7fbd580aff20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:46.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.946+0000 7fbd69ffb700 1 -- 172.21.15.40:0/3462246161 <== osd.7 v2:172.21.15.103:6816/542253261 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbd640095d0 con 0x7fbd580ada80 2024-03-18T22:41:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.969+0000 7fbd72baa700 1 -- 172.21.15.40:0/3462246161 --> [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbd6c062da0 con 0x7fbd580ada80 2024-03-18T22:41:46.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.970+0000 7fbd69ffb700 1 -- 172.21.15.40:0/3462246161 <== osd.7 v2:172.21.15.103:6816/542253261 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbd6c062da0 con 0x7fbd580ada80 2024-03-18T22:41:46.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.970+0000 7fbd72baa700 1 -- 172.21.15.40:0/3462246161 >> [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261] conn(0x7fbd580ada80 msgr2=0x7fbd580aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.970+0000 7fbd72baa700 1 --2- 172.21.15.40:0/3462246161 >> [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261] conn(0x7fbd580ada80 0x7fbd580aff20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.970+0000 7fbd72baa700 1 -- 172.21.15.40:0/3462246161 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbd580a2920 msgr2=0x7fbd580a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.970+0000 7fbd72baa700 1 --2- 172.21.15.40:0/3462246161 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbd580a2920 0x7fbd580a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00f540 tx=0x7fbd5c00b040 comp rx=0 tx=0).stop 2024-03-18T22:41:46.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.970+0000 7fbd72baa700 1 -- 172.21.15.40:0/3462246161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd6c107250 msgr2=0x7fbd6c1a4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.970+0000 7fbd72baa700 1 --2- 172.21.15.40:0/3462246161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd6c107250 0x7fbd6c1a4830 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbd64004790 tx=0x7fbd640047c0 comp rx=0 tx=0).stop 2024-03-18T22:41:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.970+0000 7fbd71147700 1 -- 172.21.15.40:0/3462246161 reap_dead start 2024-03-18T22:41:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.970+0000 7fbd72baa700 1 -- 172.21.15.40:0/3462246161 shutdown_connections 2024-03-18T22:41:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.970+0000 7fbd72baa700 1 -- 172.21.15.40:0/3462246161 >> 172.21.15.40:0/3462246161 conn(0x7fbd6c078140 msgr2=0x7fbd6c100040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.970+0000 7fbd72baa700 1 -- 172.21.15.40:0/3462246161 shutdown_connections 2024-03-18T22:41:46.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:46.970+0000 7fbd72baa700 1 -- 172.21.15.40:0/3462246161 wait complete. 2024-03-18T22:41:47.298 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:47 smithi040 ceph-mon[62643]: pgmap v323: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-18T22:41:47.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:47 smithi113 ceph-mon[66775]: pgmap v323: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-18T22:41:47.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:47 smithi103 ceph-mon[66377]: pgmap v323: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-18T22:41:47.608 INFO:teuthology.orchestra.run.smithi040.stdout:287762808867 2024-03-18T22:41:47.608 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-18T22:41:48.282 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:48 smithi040 ceph-mon[62643]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-18T22:41:48.283 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:48 smithi040 ceph-mon[62643]: Cluster is now healthy 2024-03-18T22:41:48.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:48 smithi113 ceph-mon[66775]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-18T22:41:48.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:48 smithi113 ceph-mon[66775]: Cluster is now healthy 2024-03-18T22:41:48.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:48 smithi103 ceph-mon[66377]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-18T22:41:48.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:48 smithi103 ceph-mon[66377]: Cluster is now healthy 2024-03-18T22:41:48.581 INFO:teuthology.orchestra.run.smithi040.stdout:219043332142 2024-03-18T22:41:48.582 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-18T22:41:49.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:49 smithi040 ceph-mon[62643]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:41:49.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:49 smithi113 ceph-mon[66775]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:41:49.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:49 smithi103 ceph-mon[66377]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:41:49.775 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:49.775 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:49.776 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:49.776 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.831+0000 7f6d5186a700 1 -- 172.21.15.40:0/738052169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d4c102fa0 msgr2=0x7f6d4c106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.831+0000 7f6d5186a700 1 --2- 172.21.15.40:0/738052169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d4c102fa0 0x7f6d4c106f70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c0099c0 tx=0x7f6d3c009cd0 comp rx=0 tx=0).stop 2024-03-18T22:41:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.831+0000 7f6d5186a700 1 -- 172.21.15.40:0/738052169 shutdown_connections 2024-03-18T22:41:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.831+0000 7f6d5186a700 1 --2- 172.21.15.40:0/738052169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d4c1076a0 0x7f6d4c10f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:49.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.831+0000 7f6d5186a700 1 --2- 172.21.15.40:0/738052169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d4c1025f0 0x7f6d4c1029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:49.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.831+0000 7f6d5186a700 1 --2- 172.21.15.40:0/738052169 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d4c102fa0 0x7f6d4c106f70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:49.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.831+0000 7f6d5186a700 1 -- 172.21.15.40:0/738052169 >> 172.21.15.40:0/738052169 conn(0x7f6d4c079430 msgr2=0x7f6d4c079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:49.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.831+0000 7f6d5186a700 1 -- 172.21.15.40:0/738052169 shutdown_connections 2024-03-18T22:41:49.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.831+0000 7f6d5186a700 1 -- 172.21.15.40:0/738052169 wait complete. 2024-03-18T22:41:49.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.831+0000 7f6d5186a700 1 Processor -- start 2024-03-18T22:41:49.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.831+0000 7f6d5186a700 1 -- start start 2024-03-18T22:41:49.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d5186a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d4c1025f0 0x7f6d4c19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:49.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d5186a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d4c102fa0 0x7f6d4c19ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:49.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d5186a700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d4c1076a0 0x7f6d4c198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:49.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d5186a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c1124a0 con 0x7f6d4c1025f0 2024-03-18T22:41:49.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d5186a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c112320 con 0x7f6d4c102fa0 2024-03-18T22:41:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d5186a700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c112620 con 0x7f6d4c1076a0 2024-03-18T22:41:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d4affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d4c1025f0 0x7f6d4c19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d4affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d4c1025f0 0x7f6d4c19e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32980/0 (socket says 172.21.15.40:32980) 2024-03-18T22:41:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d4affd700 1 -- 172.21.15.40:0/750082335 learned_addr learned my addr 172.21.15.40:0/750082335 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d4a7fc700 1 --2- 172.21.15.40:0/750082335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d4c102fa0 0x7f6d4c19ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d4b7fe700 1 --2- 172.21.15.40:0/750082335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d4c1076a0 0x7f6d4c198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d4affd700 1 -- 172.21.15.40:0/750082335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d4c1076a0 msgr2=0x7f6d4c198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d4affd700 1 --2- 172.21.15.40:0/750082335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d4c1076a0 0x7f6d4c198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d4affd700 1 -- 172.21.15.40:0/750082335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d4c102fa0 msgr2=0x7f6d4c19ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d4affd700 1 --2- 172.21.15.40:0/750082335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d4c102fa0 0x7f6d4c19ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d4affd700 1 -- 172.21.15.40:0/750082335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4c198fe0 con 0x7f6d4c1025f0 2024-03-18T22:41:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d4a7fc700 1 --2- 172.21.15.40:0/750082335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d4c102fa0 0x7f6d4c19ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.832+0000 7f6d4affd700 1 --2- 172.21.15.40:0/750082335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d4c1025f0 0x7f6d4c19e680 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000eb30 tx=0x7f6d4000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:49.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.833+0000 7f6d50868700 1 -- 172.21.15.40:0/750082335 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4000cbd0 con 0x7f6d4c1025f0 2024-03-18T22:41:49.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.833+0000 7f6d5186a700 1 -- 172.21.15.40:0/750082335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d4c1992d0 con 0x7f6d4c1025f0 2024-03-18T22:41:49.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.833+0000 7f6d50868700 1 -- 172.21.15.40:0/750082335 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d400044d0 con 0x7f6d4c1025f0 2024-03-18T22:41:49.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.833+0000 7f6d50868700 1 -- 172.21.15.40:0/750082335 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d40003e30 con 0x7f6d4c1025f0 2024-03-18T22:41:49.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.833+0000 7f6d5186a700 1 -- 172.21.15.40:0/750082335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d4c1a54e0 con 0x7f6d4c1025f0 2024-03-18T22:41:49.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.834+0000 7f6d5186a700 1 -- 172.21.15.40:0/750082335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6d4c062aa0 con 0x7f6d4c1025f0 2024-03-18T22:41:49.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.835+0000 7f6d50868700 1 -- 172.21.15.40:0/750082335 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6d40004000 con 0x7f6d4c1025f0 2024-03-18T22:41:49.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.835+0000 7f6d50868700 1 --2- 172.21.15.40:0/750082335 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6d340a2920 0x7f6d340a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:49.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.835+0000 7f6d4a7fc700 1 --2- 172.21.15.40:0/750082335 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6d340a2920 0x7f6d340a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:49.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.835+0000 7f6d50868700 1 -- 172.21.15.40:0/750082335 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6d400cf900 con 0x7f6d4c1025f0 2024-03-18T22:41:49.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.836+0000 7f6d50868700 1 --2- 172.21.15.40:0/750082335 >> [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143] conn(0x7f6d340ada80 0x7f6d340aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:49.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.836+0000 7f6d50868700 1 -- 172.21.15.40:0/750082335 --> [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d40021e50 con 0x7f6d340ada80 2024-03-18T22:41:49.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.836+0000 7f6d50868700 1 -- 172.21.15.40:0/750082335 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f6d400cfd50 con 0x7f6d4c1025f0 2024-03-18T22:41:49.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.836+0000 7f6d4a7fc700 1 --2- 172.21.15.40:0/750082335 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6d340a2920 0x7f6d340a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c009990 tx=0x7f6d3c005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:49.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.836+0000 7f6d4b7fe700 1 --2- 172.21.15.40:0/750082335 >> [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143] conn(0x7f6d340ada80 0x7f6d340aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:49.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.836+0000 7f6d4b7fe700 1 --2- 172.21.15.40:0/750082335 >> [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143] conn(0x7f6d340ada80 0x7f6d340aff20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:49.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.837+0000 7f6d50868700 1 -- 172.21.15.40:0/750082335 <== osd.11 v2:172.21.15.113:6824/287160143 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6d40021e50 con 0x7f6d340ada80 2024-03-18T22:41:49.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.853+0000 7f6d5186a700 1 -- 172.21.15.40:0/750082335 --> [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6d4c062ea0 con 0x7f6d340ada80 2024-03-18T22:41:49.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.854+0000 7f6d50868700 1 -- 172.21.15.40:0/750082335 <== osd.11 v2:172.21.15.113:6824/287160143 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6d4c062ea0 con 0x7f6d340ada80 2024-03-18T22:41:49.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.854+0000 7f6d5186a700 1 -- 172.21.15.40:0/750082335 >> [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143] conn(0x7f6d340ada80 msgr2=0x7f6d340aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:49.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.854+0000 7f6d5186a700 1 --2- 172.21.15.40:0/750082335 >> [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143] conn(0x7f6d340ada80 0x7f6d340aff20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:49.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.854+0000 7f6d5186a700 1 -- 172.21.15.40:0/750082335 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6d340a2920 msgr2=0x7f6d340a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:49.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.854+0000 7f6d5186a700 1 --2- 172.21.15.40:0/750082335 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6d340a2920 0x7f6d340a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c009990 tx=0x7f6d3c005fb0 comp rx=0 tx=0).stop 2024-03-18T22:41:49.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.854+0000 7f6d5186a700 1 -- 172.21.15.40:0/750082335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d4c1025f0 msgr2=0x7f6d4c19e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:49.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.854+0000 7f6d5186a700 1 --2- 172.21.15.40:0/750082335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d4c1025f0 0x7f6d4c19e680 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000eb30 tx=0x7f6d4000ee40 comp rx=0 tx=0).stop 2024-03-18T22:41:49.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.854+0000 7f6d4b7fe700 1 -- 172.21.15.40:0/750082335 reap_dead start 2024-03-18T22:41:49.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.854+0000 7f6d5186a700 1 -- 172.21.15.40:0/750082335 shutdown_connections 2024-03-18T22:41:49.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.854+0000 7f6d5186a700 1 -- 172.21.15.40:0/750082335 >> 172.21.15.40:0/750082335 conn(0x7f6d4c079430 msgr2=0x7f6d4c100760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:49.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.854+0000 7f6d5186a700 1 -- 172.21.15.40:0/750082335 shutdown_connections 2024-03-18T22:41:49.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:49.854+0000 7f6d5186a700 1 -- 172.21.15.40:0/750082335 wait complete. 2024-03-18T22:41:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.105+0000 7f5543bab700 1 -- 172.21.15.40:0/1154983038 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f553c101210 msgr2=0x7f553c101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.105+0000 7f5543bab700 1 --2- 172.21.15.40:0/1154983038 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f553c101210 0x7f553c101670 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f55380098f0 tx=0x7f5538009c00 comp rx=0 tx=0).stop 2024-03-18T22:41:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.105+0000 7f5543bab700 1 -- 172.21.15.40:0/1154983038 shutdown_connections 2024-03-18T22:41:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.105+0000 7f5543bab700 1 --2- 172.21.15.40:0/1154983038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f553c101bb0 0x7f553c10e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.105+0000 7f5543bab700 1 --2- 172.21.15.40:0/1154983038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f553c107250 0x7f553c107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.105+0000 7f5543bab700 1 --2- 172.21.15.40:0/1154983038 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f553c101210 0x7f553c101670 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.105+0000 7f5543bab700 1 -- 172.21.15.40:0/1154983038 >> 172.21.15.40:0/1154983038 conn(0x7f553c078140 msgr2=0x7f553c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.105+0000 7f5543bab700 1 -- 172.21.15.40:0/1154983038 shutdown_connections 2024-03-18T22:41:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.105+0000 7f5543bab700 1 -- 172.21.15.40:0/1154983038 wait complete. 2024-03-18T22:41:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5543bab700 1 Processor -- start 2024-03-18T22:41:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5543bab700 1 -- start start 2024-03-18T22:41:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5543bab700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f553c101210 0x7f553c19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5543bab700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f553c101bb0 0x7f553c19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5543bab700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f553c107250 0x7f553c198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5543bab700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f553c111360 con 0x7f553c107250 2024-03-18T22:41:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5543bab700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f553c1111e0 con 0x7f553c101210 2024-03-18T22:41:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5543bab700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f553c1114e0 con 0x7f553c101bb0 2024-03-18T22:41:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5542148700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f553c107250 0x7f553c198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5542148700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f553c107250 0x7f553c198860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33000/0 (socket says 172.21.15.40:33000) 2024-03-18T22:41:50.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5542148700 1 -- 172.21.15.40:0/1749925916 learned_addr learned my addr 172.21.15.40:0/1749925916 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:50.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5541146700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f553c101bb0 0x7f553c19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:50.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5541947700 1 --2- 172.21.15.40:0/1749925916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f553c101210 0x7f553c19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:50.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5542148700 1 -- 172.21.15.40:0/1749925916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f553c101bb0 msgr2=0x7f553c19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:50.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5542148700 1 --2- 172.21.15.40:0/1749925916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f553c101bb0 0x7f553c19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:50.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5542148700 1 -- 172.21.15.40:0/1749925916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f553c101210 msgr2=0x7f553c19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:50.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.106+0000 7f5542148700 1 --2- 172.21.15.40:0/1749925916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f553c101210 0x7f553c19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:50.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.107+0000 7f5542148700 1 -- 172.21.15.40:0/1749925916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f553c198fc0 con 0x7f553c107250 2024-03-18T22:41:50.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.107+0000 7f5541146700 1 --2- 172.21.15.40:0/1749925916 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f553c101bb0 0x7f553c19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:50.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.107+0000 7f5541947700 1 --2- 172.21.15.40:0/1749925916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f553c101210 0x7f553c19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:50.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.107+0000 7f5542148700 1 --2- 172.21.15.40:0/1749925916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f553c107250 0x7f553c198860 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f553400e3c0 tx=0x7f553400e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:50.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.107+0000 7f552effd700 1 -- 172.21.15.40:0/1749925916 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55340090d0 con 0x7f553c107250 2024-03-18T22:41:50.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.107+0000 7f552effd700 1 -- 172.21.15.40:0/1749925916 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5534009270 con 0x7f553c107250 2024-03-18T22:41:50.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.107+0000 7f5543bab700 1 -- 172.21.15.40:0/1749925916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f553c1992b0 con 0x7f553c107250 2024-03-18T22:41:50.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.107+0000 7f552effd700 1 -- 172.21.15.40:0/1749925916 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5534007e30 con 0x7f553c107250 2024-03-18T22:41:50.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.107+0000 7f5543bab700 1 -- 172.21.15.40:0/1749925916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f553c1a5570 con 0x7f553c107250 2024-03-18T22:41:50.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.108+0000 7f5543bab700 1 -- 172.21.15.40:0/1749925916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f553c106360 con 0x7f553c107250 2024-03-18T22:41:50.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.109+0000 7f552effd700 1 -- 172.21.15.40:0/1749925916 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5534012050 con 0x7f553c107250 2024-03-18T22:41:50.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.110+0000 7f552effd700 1 --2- 172.21.15.40:0/1749925916 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f55280a29a0 0x7f55280a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:50.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.110+0000 7f5541947700 1 --2- 172.21.15.40:0/1749925916 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f55280a29a0 0x7f55280a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:50.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.110+0000 7f552effd700 1 -- 172.21.15.40:0/1749925916 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f55340cf160 con 0x7f553c107250 2024-03-18T22:41:50.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.110+0000 7f552effd700 1 --2- 172.21.15.40:0/1749925916 >> [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755] conn(0x7f55280adb00 0x7f55280affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.110+0000 7f552effd700 1 -- 172.21.15.40:0/1749925916 --> [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f55280b0670 con 0x7f55280adb00 2024-03-18T22:41:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.110+0000 7f552effd700 1 -- 172.21.15.40:0/1749925916 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f55340cf510 con 0x7f553c107250 2024-03-18T22:41:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.110+0000 7f5541947700 1 --2- 172.21.15.40:0/1749925916 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f55280a29a0 0x7f55280a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f553c1999a0 tx=0x7f5530009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.110+0000 7f5541146700 1 --2- 172.21.15.40:0/1749925916 >> [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755] conn(0x7f55280adb00 0x7f55280affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.111+0000 7f5541146700 1 --2- 172.21.15.40:0/1749925916 >> [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755] conn(0x7f55280adb00 0x7f55280affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.112+0000 7f552effd700 1 -- 172.21.15.40:0/1749925916 <== osd.1 v2:172.21.15.103:6800/2851705755 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f55280b0670 con 0x7f55280adb00 2024-03-18T22:41:50.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.128+0000 7f5543bab700 1 -- 172.21.15.40:0/1749925916 --> [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f553c102900 con 0x7f55280adb00 2024-03-18T22:41:50.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.128+0000 7f552effd700 1 -- 172.21.15.40:0/1749925916 <== osd.1 v2:172.21.15.103:6800/2851705755 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f553c102900 con 0x7f55280adb00 2024-03-18T22:41:50.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.129+0000 7f5543bab700 1 -- 172.21.15.40:0/1749925916 >> [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755] conn(0x7f55280adb00 msgr2=0x7f55280affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:50.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.129+0000 7f5543bab700 1 --2- 172.21.15.40:0/1749925916 >> [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755] conn(0x7f55280adb00 0x7f55280affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:50.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.129+0000 7f5543bab700 1 -- 172.21.15.40:0/1749925916 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f55280a29a0 msgr2=0x7f55280a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:50.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.129+0000 7f5543bab700 1 --2- 172.21.15.40:0/1749925916 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f55280a29a0 0x7f55280a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f553c1999a0 tx=0x7f5530009260 comp rx=0 tx=0).stop 2024-03-18T22:41:50.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.129+0000 7f5543bab700 1 -- 172.21.15.40:0/1749925916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f553c107250 msgr2=0x7f553c198860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:50.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.129+0000 7f5543bab700 1 --2- 172.21.15.40:0/1749925916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f553c107250 0x7f553c198860 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f553400e3c0 tx=0x7f553400e6d0 comp rx=0 tx=0).stop 2024-03-18T22:41:50.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.129+0000 7f5542148700 1 -- 172.21.15.40:0/1749925916 reap_dead start 2024-03-18T22:41:50.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.129+0000 7f5543bab700 1 -- 172.21.15.40:0/1749925916 shutdown_connections 2024-03-18T22:41:50.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.129+0000 7f5543bab700 1 -- 172.21.15.40:0/1749925916 >> 172.21.15.40:0/1749925916 conn(0x7f553c078140 msgr2=0x7f553c06a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:50.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.129+0000 7f5543bab700 1 -- 172.21.15.40:0/1749925916 shutdown_connections 2024-03-18T22:41:50.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:50.129+0000 7f5543bab700 1 -- 172.21.15.40:0/1749925916 wait complete. 2024-03-18T22:41:50.308 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:50.309 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:50.309 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:50.942 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:50 smithi040 ceph-mon[62643]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-03-18T22:41:50.992 INFO:teuthology.orchestra.run.smithi040.stdout:343597383706 2024-03-18T22:41:50.992 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-18T22:41:51.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:50 smithi113 ceph-mon[66775]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-03-18T22:41:51.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:50 smithi103 ceph-mon[66377]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-03-18T22:41:53.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:52 smithi040 ceph-mon[62643]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-18T22:41:53.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:52 smithi113 ceph-mon[66775]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-18T22:41:53.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:52 smithi103 ceph-mon[66377]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-03-18T22:41:54.890 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:54.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:54 smithi040 ceph-mon[62643]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-18T22:41:55.132 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:54 smithi113 ceph-mon[66775]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-18T22:41:55.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:54 smithi103 ceph-mon[66377]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-18T22:41:55.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.517+0000 7fc41a7b4700 1 -- 172.21.15.40:0/1589175772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc414101460 msgr2=0x7fc414101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:55.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.517+0000 7fc41a7b4700 1 --2- 172.21.15.40:0/1589175772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc414101460 0x7fc414101840 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc404009990 tx=0x7fc404009ca0 comp rx=0 tx=0).stop 2024-03-18T22:41:55.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.517+0000 7fc41a7b4700 1 -- 172.21.15.40:0/1589175772 shutdown_connections 2024-03-18T22:41:55.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.517+0000 7fc41a7b4700 1 --2- 172.21.15.40:0/1589175772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc414106520 0x7fc41410e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:55.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.517+0000 7fc41a7b4700 1 --2- 172.21.15.40:0/1589175772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc414101460 0x7fc414101840 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:55.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.517+0000 7fc41a7b4700 1 --2- 172.21.15.40:0/1589175772 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc414101e10 0x7fc414105e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:55.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.517+0000 7fc41a7b4700 1 -- 172.21.15.40:0/1589175772 >> 172.21.15.40:0/1589175772 conn(0x7fc414078180 msgr2=0x7fc414078590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:55.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.517+0000 7fc41a7b4700 1 -- 172.21.15.40:0/1589175772 shutdown_connections 2024-03-18T22:41:55.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.517+0000 7fc41a7b4700 1 -- 172.21.15.40:0/1589175772 wait complete. 2024-03-18T22:41:55.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc41a7b4700 1 Processor -- start 2024-03-18T22:41:55.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc41a7b4700 1 -- start start 2024-03-18T22:41:55.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc41a7b4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc414101460 0x7fc41419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:55.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc41a7b4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc414101e10 0x7fc41419ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:55.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc41a7b4700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc414106520 0x7fc414198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:55.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc41a7b4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc414111390 con 0x7fc414101e10 2024-03-18T22:41:55.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc41a7b4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc414111210 con 0x7fc414101460 2024-03-18T22:41:55.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc41a7b4700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc414111510 con 0x7fc414106520 2024-03-18T22:41:55.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc4137fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc414101e10 0x7fc41419ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:55.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc413fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc414101460 0x7fc41419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:55.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc418d51700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc414106520 0x7fc414198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:55.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc4137fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc414101e10 0x7fc41419ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33020/0 (socket says 172.21.15.40:33020) 2024-03-18T22:41:55.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc413fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc414101460 0x7fc41419e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:37666/0 (socket says 172.21.15.40:37666) 2024-03-18T22:41:55.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc4137fe700 1 -- 172.21.15.40:0/3587473316 learned_addr learned my addr 172.21.15.40:0/3587473316 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:41:55.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc413fff700 1 -- 172.21.15.40:0/3587473316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc414106520 msgr2=0x7fc414198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:55.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc413fff700 1 --2- 172.21.15.40:0/3587473316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc414106520 0x7fc414198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:55.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc413fff700 1 -- 172.21.15.40:0/3587473316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc414101e10 msgr2=0x7fc41419ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:55.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc413fff700 1 --2- 172.21.15.40:0/3587473316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc414101e10 0x7fc41419ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:55.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc413fff700 1 -- 172.21.15.40:0/3587473316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc414199010 con 0x7fc414101460 2024-03-18T22:41:55.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.518+0000 7fc4137fe700 1 --2- 172.21.15.40:0/3587473316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc414101e10 0x7fc41419ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:41:55.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.519+0000 7fc413fff700 1 --2- 172.21.15.40:0/3587473316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc414101460 0x7fc41419e6b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc4040059d0 tx=0x7fc404004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:55.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.519+0000 7fc4117fa700 1 -- 172.21.15.40:0/3587473316 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc404027070 con 0x7fc414101460 2024-03-18T22:41:55.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.519+0000 7fc4117fa700 1 -- 172.21.15.40:0/3587473316 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc404022470 con 0x7fc414101460 2024-03-18T22:41:55.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.519+0000 7fc41a7b4700 1 -- 172.21.15.40:0/3587473316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4141992a0 con 0x7fc414101460 2024-03-18T22:41:55.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.519+0000 7fc4117fa700 1 -- 172.21.15.40:0/3587473316 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40400b610 con 0x7fc414101460 2024-03-18T22:41:55.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.519+0000 7fc41a7b4700 1 -- 172.21.15.40:0/3587473316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4141a5510 con 0x7fc414101460 2024-03-18T22:41:55.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.520+0000 7fc41a7b4700 1 -- 172.21.15.40:0/3587473316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc41406a260 con 0x7fc414101460 2024-03-18T22:41:55.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.521+0000 7fc4117fa700 1 -- 172.21.15.40:0/3587473316 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc40402a070 con 0x7fc414101460 2024-03-18T22:41:55.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.522+0000 7fc4117fa700 1 --2- 172.21.15.40:0/3587473316 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc3fc0a2920 0x7fc3fc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:41:55.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.522+0000 7fc4137fe700 1 --2- 172.21.15.40:0/3587473316 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc3fc0a2920 0x7fc3fc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:41:55.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.522+0000 7fc4117fa700 1 -- 172.21.15.40:0/3587473316 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc4040cfba0 con 0x7fc414101460 2024-03-18T22:41:55.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.522+0000 7fc4137fe700 1 --2- 172.21.15.40:0/3587473316 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc3fc0a2920 0x7fc3fc0a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc414106080 tx=0x7fc408009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:41:55.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.527+0000 7fc4117fa700 1 -- 172.21.15.40:0/3587473316 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc404013070 con 0x7fc414101460 2024-03-18T22:41:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.739+0000 7fc41a7b4700 1 -- 172.21.15.40:0/3587473316 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fc414199d60 con 0x7fc414101460 2024-03-18T22:41:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.740+0000 7fc4117fa700 1 -- 172.21.15.40:0/3587473316 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc40409c7b0 con 0x7fc414101460 2024-03-18T22:41:55.741 INFO:teuthology.orchestra.run.smithi040.stdout:34359738450 2024-03-18T22:41:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.742+0000 7fc41a7b4700 1 -- 172.21.15.40:0/3587473316 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc3fc0a2920 msgr2=0x7fc3fc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.742+0000 7fc41a7b4700 1 --2- 172.21.15.40:0/3587473316 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc3fc0a2920 0x7fc3fc0a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc414106080 tx=0x7fc408009260 comp rx=0 tx=0).stop 2024-03-18T22:41:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.742+0000 7fc41a7b4700 1 -- 172.21.15.40:0/3587473316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc414101460 msgr2=0x7fc41419e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:41:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.742+0000 7fc41a7b4700 1 --2- 172.21.15.40:0/3587473316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc414101460 0x7fc41419e6b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc4040059d0 tx=0x7fc404004790 comp rx=0 tx=0).stop 2024-03-18T22:41:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.742+0000 7fc41a7b4700 1 -- 172.21.15.40:0/3587473316 shutdown_connections 2024-03-18T22:41:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.742+0000 7fc41a7b4700 1 --2- 172.21.15.40:0/3587473316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc414106520 0x7fc414198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.742+0000 7fc41a7b4700 1 --2- 172.21.15.40:0/3587473316 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc3fc0a2920 0x7fc3fc0a4de0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.742+0000 7fc41a7b4700 1 --2- 172.21.15.40:0/3587473316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc414101460 0x7fc41419e6b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.742+0000 7fc41a7b4700 1 --2- 172.21.15.40:0/3587473316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc414101e10 0x7fc41419ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:41:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.742+0000 7fc41a7b4700 1 -- 172.21.15.40:0/3587473316 >> 172.21.15.40:0/3587473316 conn(0x7fc414078180 msgr2=0x7fc41406ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:41:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.742+0000 7fc41a7b4700 1 -- 172.21.15.40:0/3587473316 shutdown_connections 2024-03-18T22:41:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:41:55.742+0000 7fc41a7b4700 1 -- 172.21.15.40:0/3587473316 wait complete. 2024-03-18T22:41:56.143 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:55 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3587473316' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T22:41:56.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:55 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3587473316' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T22:41:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:55 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3587473316' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T22:41:56.672 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:41:56.696 INFO:teuthology.orchestra.run.smithi040.stdout:55834574923 2024-03-18T22:41:56.697 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-18T22:41:57.093 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:56 smithi040 ceph-mon[62643]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T22:41:57.260 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:56 smithi103 ceph-mon[66377]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T22:41:57.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:56 smithi113 ceph-mon[66775]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T22:41:58.257 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738447 got 34359738450 for osd.0 2024-03-18T22:41:58.257 DEBUG:teuthology.parallel:result is None 2024-03-18T22:41:59.133 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:41:58 smithi040 ceph-mon[62643]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-03-18T22:41:59.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:41:58 smithi113 ceph-mon[66775]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-03-18T22:41:59.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:41:58 smithi103 ceph-mon[66377]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-03-18T22:42:00.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:00 smithi040 ceph-mon[62643]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T22:42:01.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:00 smithi113 ceph-mon[66775]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T22:42:01.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:00 smithi103 ceph-mon[66377]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T22:42:01.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.543+0000 7f0d9183f700 1 -- 172.21.15.40:0/1291444473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d8c1072a0 msgr2=0x7f0d8c107680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:01.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.543+0000 7f0d9183f700 1 --2- 172.21.15.40:0/1291444473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d8c1072a0 0x7f0d8c107680 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0098f0 tx=0x7f0d7c009c00 comp rx=0 tx=0).stop 2024-03-18T22:42:01.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.543+0000 7f0d9183f700 1 -- 172.21.15.40:0/1291444473 shutdown_connections 2024-03-18T22:42:01.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.543+0000 7f0d9183f700 1 --2- 172.21.15.40:0/1291444473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d8c101bb0 0x7f0d8c10e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:01.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.543+0000 7f0d9183f700 1 --2- 172.21.15.40:0/1291444473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d8c1072a0 0x7f0d8c107680 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:01.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.543+0000 7f0d9183f700 1 --2- 172.21.15.40:0/1291444473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d8c1011f0 0x7f0d8c101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:01.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.543+0000 7f0d9183f700 1 -- 172.21.15.40:0/1291444473 >> 172.21.15.40:0/1291444473 conn(0x7f0d8c078220 msgr2=0x7f0d8c078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:01.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.543+0000 7f0d9183f700 1 -- 172.21.15.40:0/1291444473 shutdown_connections 2024-03-18T22:42:01.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.543+0000 7f0d9183f700 1 -- 172.21.15.40:0/1291444473 wait complete. 2024-03-18T22:42:01.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.544+0000 7f0d9183f700 1 Processor -- start 2024-03-18T22:42:01.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.544+0000 7f0d9183f700 1 -- start start 2024-03-18T22:42:01.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.544+0000 7f0d9183f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d8c1011f0 0x7f0d8c19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:01.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.544+0000 7f0d9183f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d8c101bb0 0x7f0d8c19eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:01.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.544+0000 7f0d9183f700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d8c1072a0 0x7f0d8c198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:01.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.544+0000 7f0d9183f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8c06b470 con 0x7f0d8c1011f0 2024-03-18T22:42:01.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.544+0000 7f0d9183f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8c06b2f0 con 0x7f0d8c101bb0 2024-03-18T22:42:01.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.544+0000 7f0d9183f700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8c06b5f0 con 0x7f0d8c1072a0 2024-03-18T22:42:01.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.544+0000 7f0d8affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d8c1011f0 0x7f0d8c19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:01.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.544+0000 7f0d8affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d8c1011f0 0x7f0d8c19e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46670/0 (socket says 172.21.15.40:46670) 2024-03-18T22:42:01.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.544+0000 7f0d8affd700 1 -- 172.21.15.40:0/1207114883 learned_addr learned my addr 172.21.15.40:0/1207114883 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:01.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.544+0000 7f0d8a7fc700 1 --2- 172.21.15.40:0/1207114883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d8c101bb0 0x7f0d8c19eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:01.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.544+0000 7f0d8b7fe700 1 --2- 172.21.15.40:0/1207114883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d8c1072a0 0x7f0d8c198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:01.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.545+0000 7f0d8affd700 1 -- 172.21.15.40:0/1207114883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d8c1072a0 msgr2=0x7f0d8c198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:01.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.545+0000 7f0d8affd700 1 --2- 172.21.15.40:0/1207114883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d8c1072a0 0x7f0d8c198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:01.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.545+0000 7f0d8affd700 1 -- 172.21.15.40:0/1207114883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d8c101bb0 msgr2=0x7f0d8c19eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:01.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.545+0000 7f0d8affd700 1 --2- 172.21.15.40:0/1207114883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d8c101bb0 0x7f0d8c19eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:01.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.545+0000 7f0d8affd700 1 -- 172.21.15.40:0/1207114883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d8c198f60 con 0x7f0d8c1011f0 2024-03-18T22:42:01.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.545+0000 7f0d8a7fc700 1 --2- 172.21.15.40:0/1207114883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d8c101bb0 0x7f0d8c19eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:01.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.545+0000 7f0d8affd700 1 --2- 172.21.15.40:0/1207114883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d8c1011f0 0x7f0d8c19e600 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c00f990 tx=0x7f0d7c00f9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:01.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.545+0000 7f0d9083d700 1 -- 172.21.15.40:0/1207114883 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c027070 con 0x7f0d8c1011f0 2024-03-18T22:42:01.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.545+0000 7f0d9183f700 1 -- 172.21.15.40:0/1207114883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d8c1991f0 con 0x7f0d8c1011f0 2024-03-18T22:42:01.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.545+0000 7f0d9083d700 1 -- 172.21.15.40:0/1207114883 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d7c005570 con 0x7f0d8c1011f0 2024-03-18T22:42:01.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.545+0000 7f0d9083d700 1 -- 172.21.15.40:0/1207114883 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c004930 con 0x7f0d8c1011f0 2024-03-18T22:42:01.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.545+0000 7f0d9183f700 1 -- 172.21.15.40:0/1207114883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d8c1a5460 con 0x7f0d8c1011f0 2024-03-18T22:42:01.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.546+0000 7f0d9183f700 1 -- 172.21.15.40:0/1207114883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d8c1028e0 con 0x7f0d8c1011f0 2024-03-18T22:42:01.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.547+0000 7f0d9083d700 1 -- 172.21.15.40:0/1207114883 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f0d7c02a070 con 0x7f0d8c1011f0 2024-03-18T22:42:01.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.548+0000 7f0d9083d700 1 --2- 172.21.15.40:0/1207114883 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0d740a2920 0x7f0d740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:01.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.548+0000 7f0d8a7fc700 1 --2- 172.21.15.40:0/1207114883 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0d740a2920 0x7f0d740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:01.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.548+0000 7f0d9083d700 1 -- 172.21.15.40:0/1207114883 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0d7c0d0070 con 0x7f0d8c1011f0 2024-03-18T22:42:01.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.548+0000 7f0d8a7fc700 1 --2- 172.21.15.40:0/1207114883 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0d740a2920 0x7f0d740a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009750 tx=0x7f0d80009310 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:01.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.552+0000 7f0d9083d700 1 -- 172.21.15.40:0/1207114883 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0d7c031070 con 0x7f0d8c1011f0 2024-03-18T22:42:01.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.760+0000 7f0d9183f700 1 -- 172.21.15.40:0/1207114883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f0d8c02dbb0 con 0x7f0d8c1011f0 2024-03-18T22:42:01.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.760+0000 7f0d9083d700 1 -- 172.21.15.40:0/1207114883 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0d7c013030 con 0x7f0d8c1011f0 2024-03-18T22:42:01.761 INFO:teuthology.orchestra.run.smithi040.stdout:141733920832 2024-03-18T22:42:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.762+0000 7f0d9183f700 1 -- 172.21.15.40:0/1207114883 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0d740a2920 msgr2=0x7f0d740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.762+0000 7f0d9183f700 1 --2- 172.21.15.40:0/1207114883 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0d740a2920 0x7f0d740a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009750 tx=0x7f0d80009310 comp rx=0 tx=0).stop 2024-03-18T22:42:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.762+0000 7f0d9183f700 1 -- 172.21.15.40:0/1207114883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d8c1011f0 msgr2=0x7f0d8c19e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.762+0000 7f0d9183f700 1 --2- 172.21.15.40:0/1207114883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d8c1011f0 0x7f0d8c19e600 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c00f990 tx=0x7f0d7c00f9c0 comp rx=0 tx=0).stop 2024-03-18T22:42:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.762+0000 7f0d9183f700 1 -- 172.21.15.40:0/1207114883 shutdown_connections 2024-03-18T22:42:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.762+0000 7f0d9183f700 1 --2- 172.21.15.40:0/1207114883 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0d8c1072a0 0x7f0d8c198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.762+0000 7f0d9183f700 1 --2- 172.21.15.40:0/1207114883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d8c1011f0 0x7f0d8c19e600 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.762+0000 7f0d9183f700 1 --2- 172.21.15.40:0/1207114883 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0d740a2920 0x7f0d740a4de0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.762+0000 7f0d9183f700 1 --2- 172.21.15.40:0/1207114883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0d8c101bb0 0x7f0d8c19eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.762+0000 7f0d9183f700 1 -- 172.21.15.40:0/1207114883 >> 172.21.15.40:0/1207114883 conn(0x7f0d8c078220 msgr2=0x7f0d8c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.762+0000 7f0d9183f700 1 -- 172.21.15.40:0/1207114883 shutdown_connections 2024-03-18T22:42:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:01.762+0000 7f0d9183f700 1 -- 172.21.15.40:0/1207114883 wait complete. 2024-03-18T22:42:02.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:01 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1207114883' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T22:42:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:01 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1207114883' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T22:42:02.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:01 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1207114883' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T22:42:02.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.383+0000 7f63d6d52700 1 -- 172.21.15.40:0/2113630016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d0101e50 msgr2=0x7f63d0105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:02.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.383+0000 7f63d6d52700 1 --2- 172.21.15.40:0/2113630016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d0101e50 0x7f63d0105e10 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0099c0 tx=0x7f63bc009cd0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.383+0000 7f63d6d52700 1 -- 172.21.15.40:0/2113630016 shutdown_connections 2024-03-18T22:42:02.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.383+0000 7f63d6d52700 1 --2- 172.21.15.40:0/2113630016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63d01064f0 0x7f63d010e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.383+0000 7f63d6d52700 1 --2- 172.21.15.40:0/2113630016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63d01014a0 0x7f63d0101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.383+0000 7f63d6d52700 1 --2- 172.21.15.40:0/2113630016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d0101e50 0x7f63d0105e10 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.383+0000 7f63d6d52700 1 -- 172.21.15.40:0/2113630016 >> 172.21.15.40:0/2113630016 conn(0x7f63d0077260 msgr2=0x7f63d0077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:02.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.383+0000 7f63d6d52700 1 -- 172.21.15.40:0/2113630016 shutdown_connections 2024-03-18T22:42:02.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.383+0000 7f63d6d52700 1 -- 172.21.15.40:0/2113630016 wait complete. 2024-03-18T22:42:02.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63d6d52700 1 Processor -- start 2024-03-18T22:42:02.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63d6d52700 1 -- start start 2024-03-18T22:42:02.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63d6d52700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63d01014a0 0x7f63d019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:02.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63d6d52700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63d0101e50 0x7f63d019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:02.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63d6d52700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d01064f0 0x7f63d0198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:02.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63d6d52700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63d0111360 con 0x7f63d01064f0 2024-03-18T22:42:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63d6d52700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63d01111e0 con 0x7f63d01014a0 2024-03-18T22:42:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63d6d52700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f63d01114e0 con 0x7f63d0101e50 2024-03-18T22:42:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63cffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63d0101e50 0x7f63d019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63d4aee700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63d01014a0 0x7f63d019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63d52ef700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d01064f0 0x7f63d0198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63cffff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63d0101e50 0x7f63d019ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:59132/0 (socket says 172.21.15.40:59132) 2024-03-18T22:42:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63d4aee700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63d01014a0 0x7f63d019e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:57898/0 (socket says 172.21.15.40:57898) 2024-03-18T22:42:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63cffff700 1 -- 172.21.15.40:0/3617908693 learned_addr learned my addr 172.21.15.40:0/3617908693 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63cffff700 1 -- 172.21.15.40:0/3617908693 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63d01014a0 msgr2=0x7f63d019e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63cffff700 1 --2- 172.21.15.40:0/3617908693 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63d01014a0 0x7f63d019e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63cffff700 1 -- 172.21.15.40:0/3617908693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d01064f0 msgr2=0x7f63d0198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:02.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63cffff700 1 --2- 172.21.15.40:0/3617908693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d01064f0 0x7f63d0198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.384+0000 7f63cffff700 1 -- 172.21.15.40:0/3617908693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63d0199040 con 0x7f63d0101e50 2024-03-18T22:42:02.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.385+0000 7f63cffff700 1 --2- 172.21.15.40:0/3617908693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63d0101e50 0x7f63d019ebd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f63bc005f50 tx=0x7f63bc004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:02.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.385+0000 7f63cdffb700 1 -- 172.21.15.40:0/3617908693 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc026070 con 0x7f63d0101e50 2024-03-18T22:42:02.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.385+0000 7f63d6d52700 1 -- 172.21.15.40:0/3617908693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63d01992d0 con 0x7f63d0101e50 2024-03-18T22:42:02.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.385+0000 7f63cdffb700 1 -- 172.21.15.40:0/3617908693 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63bc003c70 con 0x7f63d0101e50 2024-03-18T22:42:02.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.385+0000 7f63cdffb700 1 -- 172.21.15.40:0/3617908693 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc00f9c0 con 0x7f63d0101e50 2024-03-18T22:42:02.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.385+0000 7f63d6d52700 1 -- 172.21.15.40:0/3617908693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63d01a54f0 con 0x7f63d0101e50 2024-03-18T22:42:02.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.386+0000 7f63d6d52700 1 -- 172.21.15.40:0/3617908693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63d01028f0 con 0x7f63d0101e50 2024-03-18T22:42:02.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.387+0000 7f63cdffb700 1 -- 172.21.15.40:0/3617908693 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f63bc029070 con 0x7f63d0101e50 2024-03-18T22:42:02.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.388+0000 7f63cdffb700 1 --2- 172.21.15.40:0/3617908693 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63c00a2920 0x7f63c00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:02.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.388+0000 7f63d4aee700 1 --2- 172.21.15.40:0/3617908693 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63c00a2920 0x7f63c00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:02.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.388+0000 7f63cdffb700 1 -- 172.21.15.40:0/3617908693 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f63bc0cf730 con 0x7f63d0101e50 2024-03-18T22:42:02.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.388+0000 7f63d4aee700 1 --2- 172.21.15.40:0/3617908693 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63c00a2920 0x7f63c00a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009c60 tx=0x7f63c4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:02.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.393+0000 7f63cdffb700 1 -- 172.21.15.40:0/3617908693 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f63bc05d740 con 0x7f63d0101e50 2024-03-18T22:42:02.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.606+0000 7f63d6d52700 1 -- 172.21.15.40:0/3617908693 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f63d0199d40 con 0x7f63d0101e50 2024-03-18T22:42:02.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.606+0000 7f63cdffb700 1 -- 172.21.15.40:0/3617908693 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f63bc013070 con 0x7f63d0101e50 2024-03-18T22:42:02.607 INFO:teuthology.orchestra.run.smithi040.stdout:163208757307 2024-03-18T22:42:02.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.601+0000 7fea9edb1700 1 -- 172.21.15.40:0/164575325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea98102620 msgr2=0x7fea98105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:02.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.601+0000 7fea9edb1700 1 --2- 172.21.15.40:0/164575325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea98102620 0x7fea98105eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fea84009920 tx=0x7fea84009c30 comp rx=0 tx=0).stop 2024-03-18T22:42:02.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.601+0000 7fea9edb1700 1 -- 172.21.15.40:0/164575325 shutdown_connections 2024-03-18T22:42:02.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.601+0000 7fea9edb1700 1 --2- 172.21.15.40:0/164575325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea98106590 0x7fea9810e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.601+0000 7fea9edb1700 1 --2- 172.21.15.40:0/164575325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea98101c70 0x7fea98102050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.601+0000 7fea9edb1700 1 --2- 172.21.15.40:0/164575325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea98102620 0x7fea98105eb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.601+0000 7fea9edb1700 1 -- 172.21.15.40:0/164575325 >> 172.21.15.40:0/164575325 conn(0x7fea98077810 msgr2=0x7fea980fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:02.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.601+0000 7fea9edb1700 1 -- 172.21.15.40:0/164575325 shutdown_connections 2024-03-18T22:42:02.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.602+0000 7fea9edb1700 1 -- 172.21.15.40:0/164575325 wait complete. 2024-03-18T22:42:02.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.602+0000 7fea9edb1700 1 Processor -- start 2024-03-18T22:42:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.602+0000 7fea9edb1700 1 -- start start 2024-03-18T22:42:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea9edb1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea98101c70 0x7fea980fccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea9edb1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea98102620 0x7fea980fd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea9edb1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea98106590 0x7fea980fd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea9edb1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fea98111400 con 0x7fea98106590 2024-03-18T22:42:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea9edb1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea98111280 con 0x7fea98101c70 2024-03-18T22:42:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea9edb1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fea98111580 con 0x7fea98102620 2024-03-18T22:42:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea9d34e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea98106590 0x7fea980fd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea97fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea98102620 0x7fea980fd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea97fff700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea98102620 0x7fea980fd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:59150/0 (socket says 172.21.15.40:59150) 2024-03-18T22:42:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea97fff700 1 -- 172.21.15.40:0/1918782653 learned_addr learned my addr 172.21.15.40:0/1918782653 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea9cb4d700 1 --2- 172.21.15.40:0/1918782653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea98101c70 0x7fea980fccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea9d34e700 1 --2- 172.21.15.40:0/1918782653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea98106590 0x7fea980fd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46708/0 (socket says 172.21.15.40:46708) 2024-03-18T22:42:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea9cb4d700 1 -- 172.21.15.40:0/1918782653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea98102620 msgr2=0x7fea980fd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea9cb4d700 1 --2- 172.21.15.40:0/1918782653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea98102620 0x7fea980fd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea9cb4d700 1 -- 172.21.15.40:0/1918782653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea98106590 msgr2=0x7fea980fd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:02.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea9cb4d700 1 --2- 172.21.15.40:0/1918782653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea98106590 0x7fea980fd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea9cb4d700 1 -- 172.21.15.40:0/1918782653 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea981a70b0 con 0x7fea98101c70 2024-03-18T22:42:02.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.603+0000 7fea97fff700 1 --2- 172.21.15.40:0/1918782653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea98102620 0x7fea980fd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:02.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.604+0000 7fea9cb4d700 1 --2- 172.21.15.40:0/1918782653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea98101c70 0x7fea980fccf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00e9f0 tx=0x7fea8c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:02.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.604+0000 7fea95ffb700 1 -- 172.21.15.40:0/1918782653 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c00c9c0 con 0x7fea98101c70 2024-03-18T22:42:02.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.604+0000 7fea95ffb700 1 -- 172.21.15.40:0/1918782653 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea8c00cb60 con 0x7fea98101c70 2024-03-18T22:42:02.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.604+0000 7fea9edb1700 1 -- 172.21.15.40:0/1918782653 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea981a73a0 con 0x7fea98101c70 2024-03-18T22:42:02.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.604+0000 7fea95ffb700 1 -- 172.21.15.40:0/1918782653 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c018680 con 0x7fea98101c70 2024-03-18T22:42:02.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.604+0000 7fea9edb1700 1 -- 172.21.15.40:0/1918782653 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea981a7930 con 0x7fea98101c70 2024-03-18T22:42:02.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.605+0000 7fea9edb1700 1 -- 172.21.15.40:0/1918782653 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea98068fd0 con 0x7fea98101c70 2024-03-18T22:42:02.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.606+0000 7fea95ffb700 1 -- 172.21.15.40:0/1918782653 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fea8c047a90 con 0x7fea98101c70 2024-03-18T22:42:02.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.606+0000 7fea95ffb700 1 --2- 172.21.15.40:0/1918782653 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fea880a29a0 0x7fea880a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:02.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.607+0000 7fea97fff700 1 --2- 172.21.15.40:0/1918782653 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fea880a29a0 0x7fea880a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:02.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.607+0000 7fea95ffb700 1 -- 172.21.15.40:0/1918782653 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fea8c0cf0e0 con 0x7fea98101c70 2024-03-18T22:42:02.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.607+0000 7fea97fff700 1 --2- 172.21.15.40:0/1918782653 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fea880a29a0 0x7fea880a4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fea98104030 tx=0x7fea84019040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:02.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.608+0000 7f63d6d52700 1 -- 172.21.15.40:0/3617908693 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63c00a2920 msgr2=0x7f63c00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:02.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.608+0000 7f63d6d52700 1 --2- 172.21.15.40:0/3617908693 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63c00a2920 0x7f63c00a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009c60 tx=0x7f63c4009260 comp rx=0 tx=0).stop 2024-03-18T22:42:02.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.608+0000 7f63d6d52700 1 -- 172.21.15.40:0/3617908693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63d0101e50 msgr2=0x7f63d019ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:02.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.608+0000 7f63d6d52700 1 --2- 172.21.15.40:0/3617908693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63d0101e50 0x7f63d019ebd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f63bc005f50 tx=0x7f63bc004270 comp rx=0 tx=0).stop 2024-03-18T22:42:02.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.609+0000 7f63d6d52700 1 -- 172.21.15.40:0/3617908693 shutdown_connections 2024-03-18T22:42:02.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.609+0000 7f63d6d52700 1 --2- 172.21.15.40:0/3617908693 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63d01064f0 0x7f63d0198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.609+0000 7f63d6d52700 1 --2- 172.21.15.40:0/3617908693 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63c00a2920 0x7f63c00a4de0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.609+0000 7f63d6d52700 1 --2- 172.21.15.40:0/3617908693 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63d01014a0 0x7f63d019e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.609+0000 7f63d6d52700 1 --2- 172.21.15.40:0/3617908693 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63d0101e50 0x7f63d019ebd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.609+0000 7f63d6d52700 1 -- 172.21.15.40:0/3617908693 >> 172.21.15.40:0/3617908693 conn(0x7f63d0077260 msgr2=0x7f63d006ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:02.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.609+0000 7f63d6d52700 1 -- 172.21.15.40:0/3617908693 shutdown_connections 2024-03-18T22:42:02.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.609+0000 7f63d6d52700 1 -- 172.21.15.40:0/3617908693 wait complete. 2024-03-18T22:42:02.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.610+0000 7fea95ffb700 1 -- 172.21.15.40:0/1918782653 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fea8c09bc60 con 0x7fea98101c70 2024-03-18T22:42:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.818+0000 7fea9edb1700 1 -- 172.21.15.40:0/1918782653 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fea980fe220 con 0x7fea98101c70 2024-03-18T22:42:02.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.819+0000 7fea95ffb700 1 -- 172.21.15.40:0/1918782653 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fea8c09b560 con 0x7fea98101c70 2024-03-18T22:42:02.819 INFO:teuthology.orchestra.run.smithi040.stdout:77309411400 2024-03-18T22:42:02.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.821+0000 7fea9edb1700 1 -- 172.21.15.40:0/1918782653 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fea880a29a0 msgr2=0x7fea880a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.821+0000 7fea9edb1700 1 --2- 172.21.15.40:0/1918782653 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fea880a29a0 0x7fea880a4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fea98104030 tx=0x7fea84019040 comp rx=0 tx=0).stop 2024-03-18T22:42:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.821+0000 7fea9edb1700 1 -- 172.21.15.40:0/1918782653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea98101c70 msgr2=0x7fea980fccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.821+0000 7fea9edb1700 1 --2- 172.21.15.40:0/1918782653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea98101c70 0x7fea980fccf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00e9f0 tx=0x7fea8c00ed00 comp rx=0 tx=0).stop 2024-03-18T22:42:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.821+0000 7fea9edb1700 1 -- 172.21.15.40:0/1918782653 shutdown_connections 2024-03-18T22:42:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.821+0000 7fea9edb1700 1 --2- 172.21.15.40:0/1918782653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea98106590 0x7fea980fd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.821+0000 7fea9edb1700 1 --2- 172.21.15.40:0/1918782653 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fea880a29a0 0x7fea880a4e60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.821+0000 7fea9edb1700 1 --2- 172.21.15.40:0/1918782653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea98101c70 0x7fea980fccf0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.821+0000 7fea9edb1700 1 --2- 172.21.15.40:0/1918782653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea98102620 0x7fea980fd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.821+0000 7fea9edb1700 1 -- 172.21.15.40:0/1918782653 >> 172.21.15.40:0/1918782653 conn(0x7fea98077810 msgr2=0x7fea98100c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.821+0000 7fea9edb1700 1 -- 172.21.15.40:0/1918782653 shutdown_connections 2024-03-18T22:42:02.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:02.821+0000 7fea9edb1700 1 -- 172.21.15.40:0/1918782653 wait complete. 2024-03-18T22:42:03.225 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:02 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3617908693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T22:42:03.225 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:02 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1918782653' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T22:42:03.226 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:02 smithi040 ceph-mon[62643]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:03.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:02 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3617908693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T22:42:03.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:02 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1918782653' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T22:42:03.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:02 smithi113 ceph-mon[66775]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:03.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:02 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3617908693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T22:42:03.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:02 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1918782653' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T22:42:03.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:02 smithi103 ceph-mon[66377]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:04.249 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:03 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/556650867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T22:42:04.277 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757302 got 163208757307 for osd.5 2024-03-18T22:42:04.277 DEBUG:teuthology.parallel:result is None 2024-03-18T22:42:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:03 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/556650867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T22:42:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:03 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/556650867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T22:42:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.037+0000 7f67b762b700 1 -- 172.21.15.40:0/2740289368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b0101210 msgr2=0x7f67b0101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.037+0000 7f67b762b700 1 --2- 172.21.15.40:0/2740289368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b0101210 0x7f67b0101670 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f67a0009990 tx=0x7f67a0009ca0 comp rx=0 tx=0).stop 2024-03-18T22:42:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.037+0000 7f67b762b700 1 -- 172.21.15.40:0/2740289368 shutdown_connections 2024-03-18T22:42:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.037+0000 7f67b762b700 1 --2- 172.21.15.40:0/2740289368 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67b0101bb0 0x7f67b010e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.037+0000 7f67b762b700 1 --2- 172.21.15.40:0/2740289368 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67b0107250 0x7f67b0107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.037+0000 7f67b762b700 1 --2- 172.21.15.40:0/2740289368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b0101210 0x7f67b0101670 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.037+0000 7f67b762b700 1 -- 172.21.15.40:0/2740289368 >> 172.21.15.40:0/2740289368 conn(0x7f67b0078140 msgr2=0x7f67b0078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.037+0000 7f67b762b700 1 -- 172.21.15.40:0/2740289368 shutdown_connections 2024-03-18T22:42:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.037+0000 7f67b762b700 1 -- 172.21.15.40:0/2740289368 wait complete. 2024-03-18T22:42:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b762b700 1 Processor -- start 2024-03-18T22:42:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b762b700 1 -- start start 2024-03-18T22:42:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b762b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67b0101210 0x7f67b0198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b762b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67b0101bb0 0x7f67b0198fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b762b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b0107250 0x7f67b019d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b762b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f67b0111360 con 0x7f67b0107250 2024-03-18T22:42:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b762b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f67b01111e0 con 0x7f67b0101bb0 2024-03-18T22:42:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b762b700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f67b01114e0 con 0x7f67b0101210 2024-03-18T22:42:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b53c7700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67b0101210 0x7f67b0198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b5bc8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b0107250 0x7f67b019d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b5bc8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b0107250 0x7f67b019d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46734/0 (socket says 172.21.15.40:46734) 2024-03-18T22:42:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b4bc6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67b0101bb0 0x7f67b0198fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b5bc8700 1 -- 172.21.15.40:0/556650867 learned_addr learned my addr 172.21.15.40:0/556650867 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b53c7700 1 --2- 172.21.15.40:0/556650867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67b0101210 0x7f67b0198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:59162/0 (socket says 172.21.15.40:59162) 2024-03-18T22:42:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b5bc8700 1 -- 172.21.15.40:0/556650867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67b0101210 msgr2=0x7f67b0198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b5bc8700 1 --2- 172.21.15.40:0/556650867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67b0101210 0x7f67b0198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b5bc8700 1 -- 172.21.15.40:0/556650867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67b0101bb0 msgr2=0x7f67b0198fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b5bc8700 1 --2- 172.21.15.40:0/556650867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67b0101bb0 0x7f67b0198fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b5bc8700 1 -- 172.21.15.40:0/556650867 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b019da30 con 0x7f67b0107250 2024-03-18T22:42:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.038+0000 7f67b4bc6700 1 --2- 172.21.15.40:0/556650867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67b0101bb0 0x7f67b0198fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.039+0000 7f67b5bc8700 1 --2- 172.21.15.40:0/556650867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b0107250 0x7f67b019d330 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f67a800e940 tx=0x7f67a800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.039+0000 7f67a67fc700 1 -- 172.21.15.40:0/556650867 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a8004a30 con 0x7f67b0107250 2024-03-18T22:42:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.039+0000 7f67b762b700 1 -- 172.21.15.40:0/556650867 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67b019dd20 con 0x7f67b0107250 2024-03-18T22:42:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.039+0000 7f67a67fc700 1 -- 172.21.15.40:0/556650867 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67a80054d0 con 0x7f67b0107250 2024-03-18T22:42:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.039+0000 7f67a67fc700 1 -- 172.21.15.40:0/556650867 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a800ccf0 con 0x7f67b0107250 2024-03-18T22:42:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.039+0000 7f67b762b700 1 -- 172.21.15.40:0/556650867 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67b01a55e0 con 0x7f67b0107250 2024-03-18T22:42:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.040+0000 7f67b762b700 1 -- 172.21.15.40:0/556650867 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67b0068fd0 con 0x7f67b0107250 2024-03-18T22:42:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.041+0000 7f67a67fc700 1 -- 172.21.15.40:0/556650867 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f67a8005020 con 0x7f67b0107250 2024-03-18T22:42:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.041+0000 7f67a67fc700 1 --2- 172.21.15.40:0/556650867 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f679c0a2920 0x7f679c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.042+0000 7f67b53c7700 1 --2- 172.21.15.40:0/556650867 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f679c0a2920 0x7f679c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.042+0000 7f67a67fc700 1 -- 172.21.15.40:0/556650867 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f67a80cf0d0 con 0x7f67b0107250 2024-03-18T22:42:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.042+0000 7f67b53c7700 1 --2- 172.21.15.40:0/556650867 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f679c0a2920 0x7f679c0a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f67ac00f540 tx=0x7f67ac00b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.046+0000 7f67a67fc700 1 -- 172.21.15.40:0/556650867 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f67a809bcb0 con 0x7f67b0107250 2024-03-18T22:42:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.263+0000 7f67b762b700 1 -- 172.21.15.40:0/556650867 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f67b01a57b0 con 0x7f67b0107250 2024-03-18T22:42:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.264+0000 7f67a67fc700 1 -- 172.21.15.40:0/556650867 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f67a809b5b0 con 0x7f67b0107250 2024-03-18T22:42:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.266+0000 7f67b762b700 1 -- 172.21.15.40:0/556650867 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f679c0a2920 msgr2=0x7f679c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.266+0000 7f67b762b700 1 --2- 172.21.15.40:0/556650867 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f679c0a2920 0x7f679c0a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f67ac00f540 tx=0x7f67ac00b040 comp rx=0 tx=0).stop 2024-03-18T22:42:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.266+0000 7f67b762b700 1 -- 172.21.15.40:0/556650867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b0107250 msgr2=0x7f67b019d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.266+0000 7f67b762b700 1 --2- 172.21.15.40:0/556650867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b0107250 0x7f67b019d330 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f67a800e940 tx=0x7f67a800ec50 comp rx=0 tx=0).stop 2024-03-18T22:42:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.266+0000 7f67b762b700 1 -- 172.21.15.40:0/556650867 shutdown_connections 2024-03-18T22:42:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.266+0000 7f67b762b700 1 --2- 172.21.15.40:0/556650867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67b0107250 0x7f67b019d330 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.266+0000 7f67b762b700 1 --2- 172.21.15.40:0/556650867 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67b0101210 0x7f67b0198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.266+0000 7f67b762b700 1 --2- 172.21.15.40:0/556650867 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f679c0a2920 0x7f679c0a4de0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.266+0000 7f67b762b700 1 --2- 172.21.15.40:0/556650867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67b0101bb0 0x7f67b0198fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.266+0000 7f67b762b700 1 -- 172.21.15.40:0/556650867 >> 172.21.15.40:0/556650867 conn(0x7f67b0078140 msgr2=0x7f67b006a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.266+0000 7f67b762b700 1 -- 172.21.15.40:0/556650867 shutdown_connections 2024-03-18T22:42:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:03.266+0000 7f67b762b700 1 -- 172.21.15.40:0/556650867 wait complete. 2024-03-18T22:42:04.369 INFO:teuthology.orchestra.run.smithi040.stdout:197568495671 2024-03-18T22:42:04.936 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:04 smithi040 ceph-mon[62643]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T22:42:04.974 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495666 got 197568495671 for osd.6 2024-03-18T22:42:04.974 DEBUG:teuthology.parallel:result is None 2024-03-18T22:42:05.321 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:42:05.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:04 smithi113 ceph-mon[66775]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T22:42:05.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:04 smithi103 ceph-mon[66377]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T22:42:06.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.150+0000 7f78087ad700 1 -- 172.21.15.40:0/1383002769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78001025c0 msgr2=0x7f78001029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:06.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.150+0000 7f78087ad700 1 --2- 172.21.15.40:0/1383002769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78001025c0 0x7f78001029a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f77f00099c0 tx=0x7f77f0009cd0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.150+0000 7f78087ad700 1 -- 172.21.15.40:0/1383002769 shutdown_connections 2024-03-18T22:42:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.150+0000 7f78087ad700 1 --2- 172.21.15.40:0/1383002769 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7800107690 0x7f780010f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.150+0000 7f78087ad700 1 --2- 172.21.15.40:0/1383002769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78001025c0 0x7f78001029a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.150+0000 7f78087ad700 1 --2- 172.21.15.40:0/1383002769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7800102f70 0x7f7800106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.150+0000 7f78087ad700 1 -- 172.21.15.40:0/1383002769 >> 172.21.15.40:0/1383002769 conn(0x7f7800079470 msgr2=0x7f7800079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.150+0000 7f78087ad700 1 -- 172.21.15.40:0/1383002769 shutdown_connections 2024-03-18T22:42:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.150+0000 7f78087ad700 1 -- 172.21.15.40:0/1383002769 wait complete. 2024-03-18T22:42:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.151+0000 7f78087ad700 1 Processor -- start 2024-03-18T22:42:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.151+0000 7f78087ad700 1 -- start start 2024-03-18T22:42:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.151+0000 7f78087ad700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78001025c0 0x7f7800074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.151+0000 7f78087ad700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7800102f70 0x7f7800074b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.151+0000 7f78087ad700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7800107690 0x7f7800075090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.151+0000 7f78087ad700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f78001125a0 con 0x7f7800107690 2024-03-18T22:42:06.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.151+0000 7f78087ad700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7800112420 con 0x7f78001025c0 2024-03-18T22:42:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.151+0000 7f78087ad700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7800112720 con 0x7f7800102f70 2024-03-18T22:42:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.151+0000 7f7806d4a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7800107690 0x7f7800075090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.151+0000 7f7805d48700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7800102f70 0x7f7800074b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.151+0000 7f7806549700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78001025c0 0x7f7800074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.151+0000 7f7806d4a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7800107690 0x7f7800075090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46778/0 (socket says 172.21.15.40:46778) 2024-03-18T22:42:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.151+0000 7f7806d4a700 1 -- 172.21.15.40:0/2386418677 learned_addr learned my addr 172.21.15.40:0/2386418677 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.152+0000 7f7806549700 1 -- 172.21.15.40:0/2386418677 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7800102f70 msgr2=0x7f7800074b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.152+0000 7f7806549700 1 --2- 172.21.15.40:0/2386418677 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7800102f70 0x7f7800074b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.152+0000 7f7806549700 1 -- 172.21.15.40:0/2386418677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7800107690 msgr2=0x7f7800075090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.152+0000 7f7806549700 1 --2- 172.21.15.40:0/2386418677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7800107690 0x7f7800075090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.152+0000 7f7806549700 1 -- 172.21.15.40:0/2386418677 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78001a7000 con 0x7f78001025c0 2024-03-18T22:42:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.152+0000 7f7806549700 1 --2- 172.21.15.40:0/2386418677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78001025c0 0x7f7800074610 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f77f00098e0 tx=0x7f77f0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.152+0000 7f77f77fe700 1 -- 172.21.15.40:0/2386418677 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f0026070 con 0x7f78001025c0 2024-03-18T22:42:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.152+0000 7f78087ad700 1 -- 172.21.15.40:0/2386418677 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78001a7290 con 0x7f78001025c0 2024-03-18T22:42:06.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.152+0000 7f77f77fe700 1 -- 172.21.15.40:0/2386418677 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77f0003c70 con 0x7f78001025c0 2024-03-18T22:42:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.152+0000 7f77f77fe700 1 -- 172.21.15.40:0/2386418677 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f000f9e0 con 0x7f78001025c0 2024-03-18T22:42:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.152+0000 7f78087ad700 1 -- 172.21.15.40:0/2386418677 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78001a7740 con 0x7f78001025c0 2024-03-18T22:42:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.153+0000 7f78087ad700 1 -- 172.21.15.40:0/2386418677 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7800068fd0 con 0x7f78001025c0 2024-03-18T22:42:06.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.154+0000 7f77f77fe700 1 -- 172.21.15.40:0/2386418677 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f77f0029070 con 0x7f78001025c0 2024-03-18T22:42:06.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.155+0000 7f77f77fe700 1 --2- 172.21.15.40:0/2386418677 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77ec0a2920 0x7f77ec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:06.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.155+0000 7f7805d48700 1 --2- 172.21.15.40:0/2386418677 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77ec0a2920 0x7f77ec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:06.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.155+0000 7f77f77fe700 1 -- 172.21.15.40:0/2386418677 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f77f00cf9c0 con 0x7f78001025c0 2024-03-18T22:42:06.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.156+0000 7f7805d48700 1 --2- 172.21.15.40:0/2386418677 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77ec0a2920 0x7f77ec0a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7800101fa0 tx=0x7f77fc009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:06.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.162+0000 7f77f77fe700 1 -- 172.21.15.40:0/2386418677 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f77f0013070 con 0x7f78001025c0 2024-03-18T22:42:06.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.773+0000 7f2db1bbe700 1 -- 172.21.15.40:0/2868047645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac107250 msgr2=0x7f2dac107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:06.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.773+0000 7f2db1bbe700 1 --2- 172.21.15.40:0/2868047645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac107250 0x7f2dac107630 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c009990 tx=0x7f2d9c009ca0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.773+0000 7f2db1bbe700 1 -- 172.21.15.40:0/2868047645 shutdown_connections 2024-03-18T22:42:06.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.773+0000 7f2db1bbe700 1 --2- 172.21.15.40:0/2868047645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dac101bb0 0x7f2dac10e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.773+0000 7f2db1bbe700 1 --2- 172.21.15.40:0/2868047645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac107250 0x7f2dac107630 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.773+0000 7f2db1bbe700 1 --2- 172.21.15.40:0/2868047645 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dac101210 0x7f2dac101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.773+0000 7f2db1bbe700 1 -- 172.21.15.40:0/2868047645 >> 172.21.15.40:0/2868047645 conn(0x7f2dac078140 msgr2=0x7f2dac078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.773+0000 7f2db1bbe700 1 -- 172.21.15.40:0/2868047645 shutdown_connections 2024-03-18T22:42:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.773+0000 7f2db1bbe700 1 -- 172.21.15.40:0/2868047645 wait complete. 2024-03-18T22:42:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.774+0000 7f2db1bbe700 1 Processor -- start 2024-03-18T22:42:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.774+0000 7f2db1bbe700 1 -- start start 2024-03-18T22:42:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.774+0000 7f2db1bbe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dac101210 0x7f2dac19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.774+0000 7f2db1bbe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dac101bb0 0x7f2dac19eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.774+0000 7f2db1bbe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac107250 0x7f2dac198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.774+0000 7f2db1bbe700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac111340 con 0x7f2dac107250 2024-03-18T22:42:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.774+0000 7f2db1bbe700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac1111c0 con 0x7f2dac101bb0 2024-03-18T22:42:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.774+0000 7f2db1bbe700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac1114c0 con 0x7f2dac101210 2024-03-18T22:42:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.774+0000 7f2dabfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac107250 0x7f2dac198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:06.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.774+0000 7f2dab7fe700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dac101210 0x7f2dac19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:06.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.774+0000 7f2dabfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac107250 0x7f2dac198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46760/0 (socket says 172.21.15.40:46760) 2024-03-18T22:42:06.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.774+0000 7f2dabfff700 1 -- 172.21.15.40:0/3873346275 learned_addr learned my addr 172.21.15.40:0/3873346275 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:06.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.774+0000 7f2daaffd700 1 --2- 172.21.15.40:0/3873346275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dac101bb0 0x7f2dac19eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:06.299 INFO:teuthology.orchestra.run.smithi040.stdout:219043332146 2024-03-18T22:42:06.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.775+0000 7f2dabfff700 1 -- 172.21.15.40:0/3873346275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dac101210 msgr2=0x7f2dac19e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.775+0000 7f2dabfff700 1 --2- 172.21.15.40:0/3873346275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dac101210 0x7f2dac19e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.775+0000 7f2dabfff700 1 -- 172.21.15.40:0/3873346275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dac101bb0 msgr2=0x7f2dac19eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.775+0000 7f2dabfff700 1 --2- 172.21.15.40:0/3873346275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dac101bb0 0x7f2dac19eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.775+0000 7f2dabfff700 1 -- 172.21.15.40:0/3873346275 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dac198f60 con 0x7f2dac107250 2024-03-18T22:42:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.775+0000 7f2dab7fe700 1 --2- 172.21.15.40:0/3873346275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dac101210 0x7f2dac19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.775+0000 7f2dabfff700 1 --2- 172.21.15.40:0/3873346275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac107250 0x7f2dac198780 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2da400ef30 tx=0x7f2da400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.775+0000 7f2da8ff9700 1 -- 172.21.15.40:0/3873346275 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da40092e0 con 0x7f2dac107250 2024-03-18T22:42:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.775+0000 7f2da8ff9700 1 -- 172.21.15.40:0/3873346275 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2da4004bb0 con 0x7f2dac107250 2024-03-18T22:42:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.775+0000 7f2da8ff9700 1 -- 172.21.15.40:0/3873346275 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da4004020 con 0x7f2dac107250 2024-03-18T22:42:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.775+0000 7f2db1bbe700 1 -- 172.21.15.40:0/3873346275 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dac1991f0 con 0x7f2dac107250 2024-03-18T22:42:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.775+0000 7f2db1bbe700 1 -- 172.21.15.40:0/3873346275 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dac1a5460 con 0x7f2dac107250 2024-03-18T22:42:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.776+0000 7f2db1bbe700 1 -- 172.21.15.40:0/3873346275 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dac102900 con 0x7f2dac107250 2024-03-18T22:42:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.778+0000 7f2da8ff9700 1 -- 172.21.15.40:0/3873346275 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2da401b030 con 0x7f2dac107250 2024-03-18T22:42:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.778+0000 7f2da8ff9700 1 --2- 172.21.15.40:0/3873346275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d940a2920 0x7f2d940a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.779+0000 7f2da8ff9700 1 -- 172.21.15.40:0/3873346275 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2da40cf3a0 con 0x7f2dac107250 2024-03-18T22:42:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.779+0000 7f2dab7fe700 1 --2- 172.21.15.40:0/3873346275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d940a2920 0x7f2d940a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.779+0000 7f2dab7fe700 1 --2- 172.21.15.40:0/3873346275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d940a2920 0x7f2d940a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c0098e0 tx=0x7f2d9c005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.782+0000 7f2da8ff9700 1 -- 172.21.15.40:0/3873346275 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2da4022020 con 0x7f2dac107250 2024-03-18T22:42:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.992+0000 7f2db1bbe700 1 -- 172.21.15.40:0/3873346275 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f2dac1a5690 con 0x7f2dac107250 2024-03-18T22:42:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.992+0000 7f2da8ff9700 1 -- 172.21.15.40:0/3873346275 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2da4010040 con 0x7f2dac107250 2024-03-18T22:42:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.994+0000 7f2db1bbe700 1 -- 172.21.15.40:0/3873346275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d940a2920 msgr2=0x7f2d940a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.994+0000 7f2db1bbe700 1 --2- 172.21.15.40:0/3873346275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d940a2920 0x7f2d940a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c0098e0 tx=0x7f2d9c005960 comp rx=0 tx=0).stop 2024-03-18T22:42:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.994+0000 7f2db1bbe700 1 -- 172.21.15.40:0/3873346275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac107250 msgr2=0x7f2dac198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.994+0000 7f2db1bbe700 1 --2- 172.21.15.40:0/3873346275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac107250 0x7f2dac198780 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2da400ef30 tx=0x7f2da400c3a0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.994+0000 7f2db1bbe700 1 -- 172.21.15.40:0/3873346275 shutdown_connections 2024-03-18T22:42:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.994+0000 7f2db1bbe700 1 --2- 172.21.15.40:0/3873346275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2dac107250 0x7f2dac198780 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.994+0000 7f2db1bbe700 1 --2- 172.21.15.40:0/3873346275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2dac101210 0x7f2dac19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.994+0000 7f2db1bbe700 1 --2- 172.21.15.40:0/3873346275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d940a2920 0x7f2d940a4de0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.994+0000 7f2db1bbe700 1 --2- 172.21.15.40:0/3873346275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2dac101bb0 0x7f2dac19eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.994+0000 7f2db1bbe700 1 -- 172.21.15.40:0/3873346275 >> 172.21.15.40:0/3873346275 conn(0x7f2dac078140 msgr2=0x7f2dac06a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:06.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.995+0000 7f2db1bbe700 1 -- 172.21.15.40:0/3873346275 shutdown_connections 2024-03-18T22:42:06.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:05.995+0000 7f2db1bbe700 1 -- 172.21.15.40:0/3873346275 wait complete. 2024-03-18T22:42:06.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:06 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3873346275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T22:42:06.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:06 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3873346275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T22:42:06.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:06 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3873346275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T22:42:06.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.371+0000 7f78087ad700 1 -- 172.21.15.40:0/2386418677 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f780002dbb0 con 0x7f78001025c0 2024-03-18T22:42:06.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.372+0000 7f77f77fe700 1 -- 172.21.15.40:0/2386418677 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f77f009c630 con 0x7f78001025c0 2024-03-18T22:42:06.373 INFO:teuthology.orchestra.run.smithi040.stdout:253403070507 2024-03-18T22:42:06.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.374+0000 7f78087ad700 1 -- 172.21.15.40:0/2386418677 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77ec0a2920 msgr2=0x7f77ec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:06.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.374+0000 7f78087ad700 1 --2- 172.21.15.40:0/2386418677 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77ec0a2920 0x7f77ec0a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7800101fa0 tx=0x7f77fc009260 comp rx=0 tx=0).stop 2024-03-18T22:42:06.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.374+0000 7f78087ad700 1 -- 172.21.15.40:0/2386418677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78001025c0 msgr2=0x7f7800074610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:06.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.374+0000 7f78087ad700 1 --2- 172.21.15.40:0/2386418677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78001025c0 0x7f7800074610 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f77f00098e0 tx=0x7f77f0004270 comp rx=0 tx=0).stop 2024-03-18T22:42:06.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.374+0000 7f78087ad700 1 -- 172.21.15.40:0/2386418677 shutdown_connections 2024-03-18T22:42:06.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.374+0000 7f78087ad700 1 --2- 172.21.15.40:0/2386418677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7800107690 0x7f7800075090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.374+0000 7f78087ad700 1 --2- 172.21.15.40:0/2386418677 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77ec0a2920 0x7f77ec0a4de0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.374+0000 7f78087ad700 1 --2- 172.21.15.40:0/2386418677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f78001025c0 0x7f7800074610 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.374+0000 7f78087ad700 1 --2- 172.21.15.40:0/2386418677 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7800102f70 0x7f7800074b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:06.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.374+0000 7f78087ad700 1 -- 172.21.15.40:0/2386418677 >> 172.21.15.40:0/2386418677 conn(0x7f7800079470 msgr2=0x7f78001006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:06.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.374+0000 7f78087ad700 1 -- 172.21.15.40:0/2386418677 shutdown_connections 2024-03-18T22:42:06.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:06.374+0000 7f78087ad700 1 -- 172.21.15.40:0/2386418677 wait complete. 2024-03-18T22:42:06.924 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332142 got 219043332146 for osd.7 2024-03-18T22:42:06.924 DEBUG:teuthology.parallel:result is None 2024-03-18T22:42:07.157 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:07 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2386418677' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T22:42:07.157 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:07 smithi040 ceph-mon[62643]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:07.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:07 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2386418677' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T22:42:07.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:07 smithi113 ceph-mon[66775]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:07.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:07 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2386418677' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T22:42:07.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:07 smithi103 ceph-mon[66377]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:07.478 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920829 got 141733920832 for osd.4 2024-03-18T22:42:07.478 DEBUG:teuthology.parallel:result is None 2024-03-18T22:42:08.238 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411396 got 77309411400 for osd.2 2024-03-18T22:42:08.238 DEBUG:teuthology.parallel:result is None 2024-03-18T22:42:09.018 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:08 smithi040 ceph-mon[62643]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:09.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:08 smithi113 ceph-mon[66775]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:09.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:08 smithi103 ceph-mon[66377]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:09.481 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070502 got 253403070507 for osd.8 2024-03-18T22:42:09.481 DEBUG:teuthology.parallel:result is None 2024-03-18T22:42:09.911 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:42:11.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:10 smithi040 ceph-mon[62643]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:42:11.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:10 smithi113 ceph-mon[66775]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:42:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:42:11.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:10 smithi103 ceph-mon[66377]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:42:11.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.839+0000 7f18f6b8d700 1 -- 172.21.15.40:0/3657294888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18f0069fc0 msgr2=0x7f18f006a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:11.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.839+0000 7f18f6b8d700 1 --2- 172.21.15.40:0/3657294888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18f0069fc0 0x7f18f006a420 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f18dc0099c0 tx=0x7f18dc009cd0 comp rx=0 tx=0).stop 2024-03-18T22:42:11.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.839+0000 7f18f6b8d700 1 -- 172.21.15.40:0/3657294888 shutdown_connections 2024-03-18T22:42:11.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.839+0000 7f18f6b8d700 1 --2- 172.21.15.40:0/3657294888 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18f006a960 0x7f18f010f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:11.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.839+0000 7f18f6b8d700 1 --2- 172.21.15.40:0/3657294888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18f0108360 0x7f18f0108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:11.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.839+0000 7f18f6b8d700 1 --2- 172.21.15.40:0/3657294888 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18f0069fc0 0x7f18f006a420 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:11.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.839+0000 7f18f6b8d700 1 -- 172.21.15.40:0/3657294888 >> 172.21.15.40:0/3657294888 conn(0x7f18f0078140 msgr2=0x7f18f0078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:11.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.839+0000 7f18f6b8d700 1 -- 172.21.15.40:0/3657294888 shutdown_connections 2024-03-18T22:42:11.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.839+0000 7f18f6b8d700 1 -- 172.21.15.40:0/3657294888 wait complete. 2024-03-18T22:42:11.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.840+0000 7f18f6b8d700 1 Processor -- start 2024-03-18T22:42:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.840+0000 7f18f6b8d700 1 -- start start 2024-03-18T22:42:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.840+0000 7f18f6b8d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18f0069fc0 0x7f18f019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.840+0000 7f18f6b8d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18f006a960 0x7f18f019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.840+0000 7f18f6b8d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18f0108360 0x7f18f0198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.840+0000 7f18f6b8d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f18f01124e0 con 0x7f18f006a960 2024-03-18T22:42:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.840+0000 7f18f6b8d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18f0112360 con 0x7f18f0108360 2024-03-18T22:42:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.840+0000 7f18f6b8d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f18f0112660 con 0x7f18f0069fc0 2024-03-18T22:42:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.840+0000 7f18f4929700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18f0069fc0 0x7f18f019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.840+0000 7f18f4929700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18f0069fc0 0x7f18f019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55544/0 (socket says 172.21.15.40:55544) 2024-03-18T22:42:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.840+0000 7f18effff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18f006a960 0x7f18f019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.840+0000 7f18f4929700 1 -- 172.21.15.40:0/401722629 learned_addr learned my addr 172.21.15.40:0/401722629 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.840+0000 7f18f512a700 1 --2- 172.21.15.40:0/401722629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18f0108360 0x7f18f0198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.841+0000 7f18f4929700 1 -- 172.21.15.40:0/401722629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18f0108360 msgr2=0x7f18f0198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.841+0000 7f18f4929700 1 --2- 172.21.15.40:0/401722629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18f0108360 0x7f18f0198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:11.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.841+0000 7f18f4929700 1 -- 172.21.15.40:0/401722629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18f006a960 msgr2=0x7f18f019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:11.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.841+0000 7f18f4929700 1 --2- 172.21.15.40:0/401722629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18f006a960 0x7f18f019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:11.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.841+0000 7f18f4929700 1 -- 172.21.15.40:0/401722629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f0198fe0 con 0x7f18f0069fc0 2024-03-18T22:42:11.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.841+0000 7f18f512a700 1 --2- 172.21.15.40:0/401722629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18f0108360 0x7f18f0198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:11.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.841+0000 7f18f4929700 1 --2- 172.21.15.40:0/401722629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18f0069fc0 0x7f18f019e680 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f18e400d980 tx=0x7f18e400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:11.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.841+0000 7f18edffb700 1 -- 172.21.15.40:0/401722629 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e4016070 con 0x7f18f0069fc0 2024-03-18T22:42:11.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.841+0000 7f18f6b8d700 1 -- 172.21.15.40:0/401722629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18f01992d0 con 0x7f18f0069fc0 2024-03-18T22:42:11.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.841+0000 7f18edffb700 1 -- 172.21.15.40:0/401722629 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18e4004830 con 0x7f18f0069fc0 2024-03-18T22:42:11.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.841+0000 7f18edffb700 1 -- 172.21.15.40:0/401722629 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e4005530 con 0x7f18f0069fc0 2024-03-18T22:42:11.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.841+0000 7f18f6b8d700 1 -- 172.21.15.40:0/401722629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18f01a54e0 con 0x7f18f0069fc0 2024-03-18T22:42:11.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.842+0000 7f18f6b8d700 1 -- 172.21.15.40:0/401722629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18f0103a10 con 0x7f18f0069fc0 2024-03-18T22:42:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.843+0000 7f18edffb700 1 -- 172.21.15.40:0/401722629 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f18e40049e0 con 0x7f18f0069fc0 2024-03-18T22:42:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.844+0000 7f18edffb700 1 --2- 172.21.15.40:0/401722629 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18e00a2920 0x7f18e00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.844+0000 7f18edffb700 1 -- 172.21.15.40:0/401722629 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f18e40cf260 con 0x7f18f0069fc0 2024-03-18T22:42:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.844+0000 7f18effff700 1 --2- 172.21.15.40:0/401722629 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18e00a2920 0x7f18e00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.845+0000 7f18effff700 1 --2- 172.21.15.40:0/401722629 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18e00a2920 0x7f18e00a4de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f18f00787b0 tx=0x7f18dc005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:11.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:11.852+0000 7f18edffb700 1 -- 172.21.15.40:0/401722629 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f18e401f020 con 0x7f18f0069fc0 2024-03-18T22:42:12.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.045+0000 7fc4bbcc1700 1 -- 172.21.15.40:0/3825235360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4b41014a0 msgr2=0x7fc4b4101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:12.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.045+0000 7fc4bbcc1700 1 --2- 172.21.15.40:0/3825235360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4b41014a0 0x7fc4b4101880 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80099c0 tx=0x7fc4a8009cd0 comp rx=0 tx=0).stop 2024-03-18T22:42:12.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.045+0000 7fc4bbcc1700 1 -- 172.21.15.40:0/3825235360 shutdown_connections 2024-03-18T22:42:12.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.045+0000 7fc4bbcc1700 1 --2- 172.21.15.40:0/3825235360 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4b41064f0 0x7fc4b410e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:12.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.045+0000 7fc4bbcc1700 1 --2- 172.21.15.40:0/3825235360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4b41014a0 0x7fc4b4101880 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:12.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.045+0000 7fc4bbcc1700 1 --2- 172.21.15.40:0/3825235360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4b4101e50 0x7fc4b4105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:12.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.045+0000 7fc4bbcc1700 1 -- 172.21.15.40:0/3825235360 >> 172.21.15.40:0/3825235360 conn(0x7fc4b4077260 msgr2=0x7fc4b4077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:12.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.046+0000 7fc4bbcc1700 1 -- 172.21.15.40:0/3825235360 shutdown_connections 2024-03-18T22:42:12.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.046+0000 7fc4bbcc1700 1 -- 172.21.15.40:0/3825235360 wait complete. 2024-03-18T22:42:12.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.046+0000 7fc4bbcc1700 1 Processor -- start 2024-03-18T22:42:12.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.046+0000 7fc4bbcc1700 1 -- start start 2024-03-18T22:42:12.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.046+0000 7fc4bbcc1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4b41014a0 0x7fc4b419e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:12.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.046+0000 7fc4bbcc1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4b4101e50 0x7fc4b419ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:12.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.046+0000 7fc4bbcc1700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4b41064f0 0x7fc4b4198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:12.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.046+0000 7fc4bbcc1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b4111380 con 0x7fc4b41014a0 2024-03-18T22:42:12.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.046+0000 7fc4bbcc1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b4111200 con 0x7fc4b4101e50 2024-03-18T22:42:12.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.046+0000 7fc4bbcc1700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b4111500 con 0x7fc4b41064f0 2024-03-18T22:42:12.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.046+0000 7fc4b9a5d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4b41014a0 0x7fc4b419e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:12.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.047+0000 7fc4b9a5d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4b41014a0 0x7fc4b419e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39974/0 (socket says 172.21.15.40:39974) 2024-03-18T22:42:12.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.047+0000 7fc4b9a5d700 1 -- 172.21.15.40:0/1234693066 learned_addr learned my addr 172.21.15.40:0/1234693066 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:12.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.047+0000 7fc4b925c700 1 --2- 172.21.15.40:0/1234693066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4b4101e50 0x7fc4b419ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:12.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.047+0000 7fc4ba25e700 1 --2- 172.21.15.40:0/1234693066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4b41064f0 0x7fc4b4198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:12.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.047+0000 7fc4b9a5d700 1 -- 172.21.15.40:0/1234693066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4b41064f0 msgr2=0x7fc4b4198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:12.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.047+0000 7fc4b9a5d700 1 --2- 172.21.15.40:0/1234693066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4b41064f0 0x7fc4b4198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:12.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.047+0000 7fc4b9a5d700 1 -- 172.21.15.40:0/1234693066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4b4101e50 msgr2=0x7fc4b419ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:12.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.047+0000 7fc4b9a5d700 1 --2- 172.21.15.40:0/1234693066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4b4101e50 0x7fc4b419ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:12.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.047+0000 7fc4b9a5d700 1 -- 172.21.15.40:0/1234693066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b4199070 con 0x7fc4b41014a0 2024-03-18T22:42:12.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.047+0000 7fc4b9a5d700 1 --2- 172.21.15.40:0/1234693066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4b41014a0 0x7fc4b419e750 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8006010 tx=0x7fc4a8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:12.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.047+0000 7fc4a6ffd700 1 -- 172.21.15.40:0/1234693066 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8026070 con 0x7fc4b41014a0 2024-03-18T22:42:12.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.047+0000 7fc4bbcc1700 1 -- 172.21.15.40:0/1234693066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b4199300 con 0x7fc4b41014a0 2024-03-18T22:42:12.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.047+0000 7fc4a6ffd700 1 -- 172.21.15.40:0/1234693066 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4a8004630 con 0x7fc4b41014a0 2024-03-18T22:42:12.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.047+0000 7fc4a6ffd700 1 -- 172.21.15.40:0/1234693066 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a800fb20 con 0x7fc4b41014a0 2024-03-18T22:42:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.047+0000 7fc4bbcc1700 1 -- 172.21.15.40:0/1234693066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4b41a5620 con 0x7fc4b41014a0 2024-03-18T22:42:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.048+0000 7fc4bbcc1700 1 -- 172.21.15.40:0/1234693066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4b4068fd0 con 0x7fc4b41014a0 2024-03-18T22:42:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.050+0000 7fc4a6ffd700 1 -- 172.21.15.40:0/1234693066 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc4a8029070 con 0x7fc4b41014a0 2024-03-18T22:42:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.050+0000 7fc4a6ffd700 1 --2- 172.21.15.40:0/1234693066 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc4a00a2920 0x7fc4a00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.050+0000 7fc4b925c700 1 --2- 172.21.15.40:0/1234693066 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc4a00a2920 0x7fc4a00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.050+0000 7fc4a6ffd700 1 -- 172.21.15.40:0/1234693066 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc4a80cfb60 con 0x7fc4b41014a0 2024-03-18T22:42:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.051+0000 7fc4b925c700 1 --2- 172.21.15.40:0/1234693066 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc4a00a2920 0x7fc4a00a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4107cb0 tx=0x7fc4b0009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.054+0000 7fc4a6ffd700 1 -- 172.21.15.40:0/1234693066 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc4a8030080 con 0x7fc4b41014a0 2024-03-18T22:42:12.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.067+0000 7f18f6b8d700 1 -- 172.21.15.40:0/401722629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f18f002dbb0 con 0x7f18f0069fc0 2024-03-18T22:42:12.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.068+0000 7f18edffb700 1 -- 172.21.15.40:0/401722629 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f18e409be40 con 0x7f18f0069fc0 2024-03-18T22:42:12.068 INFO:teuthology.orchestra.run.smithi040.stdout:287762808872 2024-03-18T22:42:12.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.070+0000 7f18f6b8d700 1 -- 172.21.15.40:0/401722629 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18e00a2920 msgr2=0x7f18e00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:12.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.070+0000 7f18f6b8d700 1 --2- 172.21.15.40:0/401722629 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18e00a2920 0x7f18e00a4de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f18f00787b0 tx=0x7f18dc005fb0 comp rx=0 tx=0).stop 2024-03-18T22:42:12.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.070+0000 7f18f6b8d700 1 -- 172.21.15.40:0/401722629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18f0069fc0 msgr2=0x7f18f019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:12.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.070+0000 7f18f6b8d700 1 --2- 172.21.15.40:0/401722629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18f0069fc0 0x7f18f019e680 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f18e400d980 tx=0x7f18e400dc90 comp rx=0 tx=0).stop 2024-03-18T22:42:12.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.070+0000 7f18f6b8d700 1 -- 172.21.15.40:0/401722629 shutdown_connections 2024-03-18T22:42:12.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.070+0000 7f18f6b8d700 1 --2- 172.21.15.40:0/401722629 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18e00a2920 0x7f18e00a4de0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:12.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.070+0000 7f18f6b8d700 1 --2- 172.21.15.40:0/401722629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18f0108360 0x7f18f0198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:12.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.070+0000 7f18f6b8d700 1 --2- 172.21.15.40:0/401722629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f18f0069fc0 0x7f18f019e680 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:12.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.070+0000 7f18f6b8d700 1 --2- 172.21.15.40:0/401722629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f18f006a960 0x7f18f019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:12.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.070+0000 7f18f6b8d700 1 -- 172.21.15.40:0/401722629 >> 172.21.15.40:0/401722629 conn(0x7f18f0078140 msgr2=0x7f18f00ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:12.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.070+0000 7f18f6b8d700 1 -- 172.21.15.40:0/401722629 shutdown_connections 2024-03-18T22:42:12.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.070+0000 7f18f6b8d700 1 -- 172.21.15.40:0/401722629 wait complete. 2024-03-18T22:42:12.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.268+0000 7fc4bbcc1700 1 -- 172.21.15.40:0/1234693066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fc4b4199e40 con 0x7fc4b41014a0 2024-03-18T22:42:12.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.268+0000 7fc4a6ffd700 1 -- 172.21.15.40:0/1234693066 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fc4a8013070 con 0x7fc4b41014a0 2024-03-18T22:42:12.269 INFO:teuthology.orchestra.run.smithi040.stdout:309237645348 2024-03-18T22:42:12.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.270+0000 7fc4bbcc1700 1 -- 172.21.15.40:0/1234693066 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc4a00a2920 msgr2=0x7fc4a00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:12.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.270+0000 7fc4bbcc1700 1 --2- 172.21.15.40:0/1234693066 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc4a00a2920 0x7fc4a00a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4107cb0 tx=0x7fc4b0009260 comp rx=0 tx=0).stop 2024-03-18T22:42:12.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.270+0000 7fc4bbcc1700 1 -- 172.21.15.40:0/1234693066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4b41014a0 msgr2=0x7fc4b419e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:12.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.270+0000 7fc4bbcc1700 1 --2- 172.21.15.40:0/1234693066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4b41014a0 0x7fc4b419e750 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8006010 tx=0x7fc4a8004270 comp rx=0 tx=0).stop 2024-03-18T22:42:12.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.271+0000 7fc4bbcc1700 1 -- 172.21.15.40:0/1234693066 shutdown_connections 2024-03-18T22:42:12.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.271+0000 7fc4bbcc1700 1 --2- 172.21.15.40:0/1234693066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc4b41064f0 0x7fc4b4198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:12.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.271+0000 7fc4bbcc1700 1 --2- 172.21.15.40:0/1234693066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4b41014a0 0x7fc4b419e750 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:12.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.271+0000 7fc4bbcc1700 1 --2- 172.21.15.40:0/1234693066 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc4a00a2920 0x7fc4a00a4de0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:12.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.271+0000 7fc4bbcc1700 1 --2- 172.21.15.40:0/1234693066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4b4101e50 0x7fc4b419ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:12.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.271+0000 7fc4bbcc1700 1 -- 172.21.15.40:0/1234693066 >> 172.21.15.40:0/1234693066 conn(0x7fc4b4077260 msgr2=0x7fc4b406ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:12.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.271+0000 7fc4bbcc1700 1 -- 172.21.15.40:0/1234693066 shutdown_connections 2024-03-18T22:42:12.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:12.271+0000 7fc4bbcc1700 1 -- 172.21.15.40:0/1234693066 wait complete. 2024-03-18T22:42:12.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:42:12.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:12 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:42:12.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:12 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/401722629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T22:42:12.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:12 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1234693066' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T22:42:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:42:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:12 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:42:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:12 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/401722629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T22:42:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:12 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1234693066' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T22:42:12.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:42:12.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:12 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:42:12.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:12 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/401722629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T22:42:12.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:12 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1234693066' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T22:42:13.054 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645343 got 309237645348 for osd.10 2024-03-18T22:42:13.054 DEBUG:teuthology.parallel:result is None 2024-03-18T22:42:13.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.231+0000 7f4d1cdeb700 1 -- 172.21.15.40:0/486833335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d18102fa0 msgr2=0x7f4d18106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:13.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.231+0000 7f4d1cdeb700 1 --2- 172.21.15.40:0/486833335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d18102fa0 0x7f4d18106f70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4d080099c0 tx=0x7f4d08009cd0 comp rx=0 tx=0).stop 2024-03-18T22:42:13.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.232+0000 7f4d1cdeb700 1 -- 172.21.15.40:0/486833335 shutdown_connections 2024-03-18T22:42:13.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.232+0000 7f4d1cdeb700 1 --2- 172.21.15.40:0/486833335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d181076a0 0x7f4d1810f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:13.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.232+0000 7f4d1cdeb700 1 --2- 172.21.15.40:0/486833335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d181025f0 0x7f4d181029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:13.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.232+0000 7f4d1cdeb700 1 --2- 172.21.15.40:0/486833335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d18102fa0 0x7f4d18106f70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:13.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.232+0000 7f4d1cdeb700 1 -- 172.21.15.40:0/486833335 >> 172.21.15.40:0/486833335 conn(0x7f4d18079430 msgr2=0x7f4d18079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:13.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.232+0000 7f4d1cdeb700 1 -- 172.21.15.40:0/486833335 shutdown_connections 2024-03-18T22:42:13.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.232+0000 7f4d1cdeb700 1 -- 172.21.15.40:0/486833335 wait complete. 2024-03-18T22:42:13.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.232+0000 7f4d1cdeb700 1 Processor -- start 2024-03-18T22:42:13.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d1cdeb700 1 -- start start 2024-03-18T22:42:13.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d1cdeb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d181025f0 0x7f4d1819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:13.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d1cdeb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d18102fa0 0x7f4d1819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:13.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d1cdeb700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d181076a0 0x7f4d181987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:13.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d1cdeb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4d181124c0 con 0x7f4d181025f0 2024-03-18T22:42:13.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d1cdeb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4d18112340 con 0x7f4d18102fa0 2024-03-18T22:42:13.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d1cdeb700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4d18112640 con 0x7f4d181076a0 2024-03-18T22:42:13.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d1659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d181025f0 0x7f4d1819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:13.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d1659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d181025f0 0x7f4d1819e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39994/0 (socket says 172.21.15.40:39994) 2024-03-18T22:42:13.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d1659c700 1 -- 172.21.15.40:0/2989091787 learned_addr learned my addr 172.21.15.40:0/2989091787 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:13.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d15d9b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d18102fa0 0x7f4d1819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:13.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d16d9d700 1 --2- 172.21.15.40:0/2989091787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d181076a0 0x7f4d181987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:13.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d15d9b700 1 -- 172.21.15.40:0/2989091787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d181076a0 msgr2=0x7f4d181987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:13.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d15d9b700 1 --2- 172.21.15.40:0/2989091787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d181076a0 0x7f4d181987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:13.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d15d9b700 1 -- 172.21.15.40:0/2989091787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d181025f0 msgr2=0x7f4d1819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:13.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d15d9b700 1 --2- 172.21.15.40:0/2989091787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d181025f0 0x7f4d1819e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:13.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d15d9b700 1 -- 172.21.15.40:0/2989091787 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d18198fa0 con 0x7f4d18102fa0 2024-03-18T22:42:13.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d1659c700 1 --2- 172.21.15.40:0/2989091787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d181025f0 0x7f4d1819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:13.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.233+0000 7f4d15d9b700 1 --2- 172.21.15.40:0/2989091787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d18102fa0 0x7f4d1819eb80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4d080098e0 tx=0x7f4d08004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:13.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.234+0000 7f4d077fe700 1 -- 172.21.15.40:0/2989091787 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d08026070 con 0x7f4d18102fa0 2024-03-18T22:42:13.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.234+0000 7f4d077fe700 1 -- 172.21.15.40:0/2989091787 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d08003c70 con 0x7f4d18102fa0 2024-03-18T22:42:13.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.234+0000 7f4d1cdeb700 1 -- 172.21.15.40:0/2989091787 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d18199230 con 0x7f4d18102fa0 2024-03-18T22:42:13.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.234+0000 7f4d077fe700 1 -- 172.21.15.40:0/2989091787 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0800f9c0 con 0x7f4d18102fa0 2024-03-18T22:42:13.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.234+0000 7f4d1cdeb700 1 -- 172.21.15.40:0/2989091787 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d181a54a0 con 0x7f4d18102fa0 2024-03-18T22:42:13.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.235+0000 7f4d1cdeb700 1 -- 172.21.15.40:0/2989091787 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d18103a50 con 0x7f4d18102fa0 2024-03-18T22:42:13.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.236+0000 7f4d077fe700 1 -- 172.21.15.40:0/2989091787 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4d08029070 con 0x7f4d18102fa0 2024-03-18T22:42:13.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.237+0000 7f4d077fe700 1 --2- 172.21.15.40:0/2989091787 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4d000a2920 0x7f4d000a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:13.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.237+0000 7f4d1659c700 1 --2- 172.21.15.40:0/2989091787 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4d000a2920 0x7f4d000a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:13.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.237+0000 7f4d077fe700 1 -- 172.21.15.40:0/2989091787 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4d080cfa80 con 0x7f4d18102fa0 2024-03-18T22:42:13.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.237+0000 7f4d1659c700 1 --2- 172.21.15.40:0/2989091787 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4d000a2920 0x7f4d000a4de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c011550 tx=0x7f4d0c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:13.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.242+0000 7f4d077fe700 1 -- 172.21.15.40:0/2989091787 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4d08030020 con 0x7f4d18102fa0 2024-03-18T22:42:13.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.447+0000 7f4d1cdeb700 1 -- 172.21.15.40:0/2989091787 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f4d18199cf0 con 0x7f4d18102fa0 2024-03-18T22:42:13.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.447+0000 7f4d077fe700 1 -- 172.21.15.40:0/2989091787 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4d08013070 con 0x7f4d18102fa0 2024-03-18T22:42:13.448 INFO:teuthology.orchestra.run.smithi040.stdout:107374182471 2024-03-18T22:42:13.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.449+0000 7f4d1cdeb700 1 -- 172.21.15.40:0/2989091787 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4d000a2920 msgr2=0x7f4d000a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:13.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.449+0000 7f4d1cdeb700 1 --2- 172.21.15.40:0/2989091787 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4d000a2920 0x7f4d000a4de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c011550 tx=0x7f4d0c009260 comp rx=0 tx=0).stop 2024-03-18T22:42:13.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.449+0000 7f4d1cdeb700 1 -- 172.21.15.40:0/2989091787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d18102fa0 msgr2=0x7f4d1819eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:13.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.449+0000 7f4d1cdeb700 1 --2- 172.21.15.40:0/2989091787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d18102fa0 0x7f4d1819eb80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4d080098e0 tx=0x7f4d08004270 comp rx=0 tx=0).stop 2024-03-18T22:42:13.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.450+0000 7f4d1cdeb700 1 -- 172.21.15.40:0/2989091787 shutdown_connections 2024-03-18T22:42:13.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.450+0000 7f4d1cdeb700 1 --2- 172.21.15.40:0/2989091787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4d181076a0 0x7f4d181987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:13.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.450+0000 7f4d1cdeb700 1 --2- 172.21.15.40:0/2989091787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d181025f0 0x7f4d1819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:13.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.450+0000 7f4d1cdeb700 1 --2- 172.21.15.40:0/2989091787 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4d000a2920 0x7f4d000a4de0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:13.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.450+0000 7f4d1cdeb700 1 --2- 172.21.15.40:0/2989091787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d18102fa0 0x7f4d1819eb80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:13.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.450+0000 7f4d1cdeb700 1 -- 172.21.15.40:0/2989091787 >> 172.21.15.40:0/2989091787 conn(0x7f4d18079430 msgr2=0x7f4d18100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:13.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.450+0000 7f4d1cdeb700 1 -- 172.21.15.40:0/2989091787 shutdown_connections 2024-03-18T22:42:13.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:13.450+0000 7f4d1cdeb700 1 -- 172.21.15.40:0/2989091787 wait complete. 2024-03-18T22:42:13.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:13 smithi040 ceph-mon[62643]: pgmap v336: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:13.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:13 smithi113 ceph-mon[66775]: pgmap v336: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:13.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:13 smithi103 ceph-mon[66377]: pgmap v336: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:14.342 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808867 got 287762808872 for osd.9 2024-03-18T22:42:14.342 DEBUG:teuthology.parallel:result is None 2024-03-18T22:42:14.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:14 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2989091787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T22:42:14.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:14 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:42:14.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:14 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:42:14.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:14 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:42:14.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:14 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2989091787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T22:42:14.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:14 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:42:14.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:14 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:42:14.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:14 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:42:14.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:14 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2989091787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T22:42:14.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:14 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:42:14.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:14 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:42:14.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:14 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:42:15.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:15 smithi040 ceph-mon[62643]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-18T22:42:15.558 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:15 smithi040 ceph-mon[62643]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T22:42:15.593 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182463 got 107374182471 for osd.3 2024-03-18T22:42:15.593 DEBUG:teuthology.parallel:result is None 2024-03-18T22:42:15.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:15 smithi113 ceph-mon[66775]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-18T22:42:15.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:15 smithi113 ceph-mon[66775]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T22:42:15.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:15 smithi103 ceph-mon[66377]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-18T22:42:15.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:15 smithi103 ceph-mon[66377]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T22:42:16.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.359+0000 7fb94d016700 1 -- 172.21.15.40:0/3080162340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb94806b460 msgr2=0x7fb948106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:16.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.359+0000 7fb94d016700 1 --2- 172.21.15.40:0/3080162340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb94806b460 0x7fb948106f40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009990 tx=0x7fb93c009ca0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.359+0000 7fb94d016700 1 -- 172.21.15.40:0/3080162340 shutdown_connections 2024-03-18T22:42:16.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.359+0000 7fb94d016700 1 --2- 172.21.15.40:0/3080162340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb948107620 0x7fb94810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.359+0000 7fb94d016700 1 --2- 172.21.15.40:0/3080162340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb94806aab0 0x7fb94806ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.359+0000 7fb94d016700 1 --2- 172.21.15.40:0/3080162340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb94806b460 0x7fb948106f40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.359+0000 7fb94d016700 1 -- 172.21.15.40:0/3080162340 >> 172.21.15.40:0/3080162340 conn(0x7fb948077260 msgr2=0x7fb948077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:16.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.360+0000 7fb94d016700 1 -- 172.21.15.40:0/3080162340 shutdown_connections 2024-03-18T22:42:16.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.360+0000 7fb94d016700 1 -- 172.21.15.40:0/3080162340 wait complete. 2024-03-18T22:42:16.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.360+0000 7fb94d016700 1 Processor -- start 2024-03-18T22:42:16.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.360+0000 7fb94d016700 1 -- start start 2024-03-18T22:42:16.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94d016700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb94806aab0 0x7fb9481a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:16.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94d016700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb94806b460 0x7fb9481a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:16.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94d016700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948107620 0x7fb94819a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:16.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94d016700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb948112450 con 0x7fb948107620 2024-03-18T22:42:16.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94d016700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb9481122d0 con 0x7fb94806aab0 2024-03-18T22:42:16.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94d016700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb9481125d0 con 0x7fb94806b460 2024-03-18T22:42:16.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94759e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948107620 0x7fb94819a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:16.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94759e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948107620 0x7fb94819a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40028/0 (socket says 172.21.15.40:40028) 2024-03-18T22:42:16.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94759e700 1 -- 172.21.15.40:0/2805228244 learned_addr learned my addr 172.21.15.40:0/2805228244 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:16.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94659c700 1 --2- 172.21.15.40:0/2805228244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb94806b460 0x7fb9481a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:16.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb946d9d700 1 --2- 172.21.15.40:0/2805228244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb94806aab0 0x7fb9481a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:16.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94759e700 1 -- 172.21.15.40:0/2805228244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb94806b460 msgr2=0x7fb9481a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:16.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94759e700 1 --2- 172.21.15.40:0/2805228244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb94806b460 0x7fb9481a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94759e700 1 -- 172.21.15.40:0/2805228244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb94806aab0 msgr2=0x7fb9481a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:16.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94759e700 1 --2- 172.21.15.40:0/2805228244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb94806aab0 0x7fb9481a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94759e700 1 -- 172.21.15.40:0/2805228244 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb94819b0f0 con 0x7fb948107620 2024-03-18T22:42:16.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94659c700 1 --2- 172.21.15.40:0/2805228244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb94806b460 0x7fb9481a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:16.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.361+0000 7fb94759e700 1 --2- 172.21.15.40:0/2805228244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948107620 0x7fb94819a990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb94000ef30 tx=0x7fb94000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:16.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.362+0000 7fb92ffff700 1 -- 172.21.15.40:0/2805228244 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9400092e0 con 0x7fb948107620 2024-03-18T22:42:16.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.362+0000 7fb94d016700 1 -- 172.21.15.40:0/2805228244 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb94819b380 con 0x7fb948107620 2024-03-18T22:42:16.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.362+0000 7fb92ffff700 1 -- 172.21.15.40:0/2805228244 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb940004bb0 con 0x7fb948107620 2024-03-18T22:42:16.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.362+0000 7fb92ffff700 1 -- 172.21.15.40:0/2805228244 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb940004020 con 0x7fb948107620 2024-03-18T22:42:16.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.362+0000 7fb94d016700 1 -- 172.21.15.40:0/2805228244 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9481a7650 con 0x7fb948107620 2024-03-18T22:42:16.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.363+0000 7fb94d016700 1 -- 172.21.15.40:0/2805228244 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb948068fd0 con 0x7fb948107620 2024-03-18T22:42:16.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.364+0000 7fb92ffff700 1 -- 172.21.15.40:0/2805228244 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb94001b030 con 0x7fb948107620 2024-03-18T22:42:16.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.364+0000 7fb92ffff700 1 --2- 172.21.15.40:0/2805228244 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9300a2890 0x7fb9300a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:16.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.365+0000 7fb946d9d700 1 --2- 172.21.15.40:0/2805228244 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9300a2890 0x7fb9300a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:16.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.365+0000 7fb92ffff700 1 -- 172.21.15.40:0/2805228244 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb9400ce8a0 con 0x7fb948107620 2024-03-18T22:42:16.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.365+0000 7fb946d9d700 1 --2- 172.21.15.40:0/2805228244 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9300a2890 0x7fb9300a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb9481072c0 tx=0x7fb938009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:16.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.369+0000 7fb92ffff700 1 -- 172.21.15.40:0/2805228244 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb940022020 con 0x7fb948107620 2024-03-18T22:42:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.252+0000 7f9633cee700 1 -- 172.21.15.40:0/2556276401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c102360 msgr2=0x7f962c1027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.252+0000 7f9633cee700 1 --2- 172.21.15.40:0/2556276401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c102360 0x7f962c1027e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f96200099c0 tx=0x7f9620009cd0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.252+0000 7f9633cee700 1 -- 172.21.15.40:0/2556276401 shutdown_connections 2024-03-18T22:42:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.252+0000 7f9633cee700 1 --2- 172.21.15.40:0/2556276401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f962c102d20 0x7f962c10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.252+0000 7f9633cee700 1 --2- 172.21.15.40:0/2556276401 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f962c108340 0x7f962c108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.252+0000 7f9633cee700 1 --2- 172.21.15.40:0/2556276401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c102360 0x7f962c1027e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.252+0000 7f9633cee700 1 -- 172.21.15.40:0/2556276401 >> 172.21.15.40:0/2556276401 conn(0x7f962c079440 msgr2=0x7f962c079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.252+0000 7f9633cee700 1 -- 172.21.15.40:0/2556276401 shutdown_connections 2024-03-18T22:42:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.252+0000 7f9633cee700 1 -- 172.21.15.40:0/2556276401 wait complete. 2024-03-18T22:42:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.253+0000 7f9633cee700 1 Processor -- start 2024-03-18T22:42:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.253+0000 7f9633cee700 1 -- start start 2024-03-18T22:42:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.253+0000 7f9633cee700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f962c102360 0x7f962c19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.253+0000 7f9633cee700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c102d20 0x7f962c19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.253+0000 7f9633cee700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f962c108340 0x7f962c198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.253+0000 7f9633cee700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f962c112490 con 0x7f962c102d20 2024-03-18T22:42:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.253+0000 7f9633cee700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f962c112310 con 0x7f962c108340 2024-03-18T22:42:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.253+0000 7f9633cee700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f962c112610 con 0x7f962c102360 2024-03-18T22:42:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.253+0000 7f9631289700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c102d20 0x7f962c19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.253+0000 7f9631289700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c102d20 0x7f962c19eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40002/0 (socket says 172.21.15.40:40002) 2024-03-18T22:42:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.253+0000 7f9631289700 1 -- 172.21.15.40:0/1089804593 learned_addr learned my addr 172.21.15.40:0/1089804593 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.253+0000 7f9631a8a700 1 --2- 172.21.15.40:0/1089804593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f962c102360 0x7f962c19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.253+0000 7f963228b700 1 --2- 172.21.15.40:0/1089804593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f962c108340 0x7f962c198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:16.495 INFO:teuthology.orchestra.run.smithi040.stdout:343597383711 2024-03-18T22:42:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.254+0000 7f9631289700 1 -- 172.21.15.40:0/1089804593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f962c102360 msgr2=0x7f962c19e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.254+0000 7f9631289700 1 --2- 172.21.15.40:0/1089804593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f962c102360 0x7f962c19e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.254+0000 7f9631289700 1 -- 172.21.15.40:0/1089804593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f962c108340 msgr2=0x7f962c198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.254+0000 7f9631289700 1 --2- 172.21.15.40:0/1089804593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f962c108340 0x7f962c198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.254+0000 7f9631289700 1 -- 172.21.15.40:0/1089804593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f962c198f70 con 0x7f962c102d20 2024-03-18T22:42:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.254+0000 7f9631289700 1 --2- 172.21.15.40:0/1089804593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c102d20 0x7f962c19eb50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9620006010 tx=0x7f9620004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.254+0000 7f961effd700 1 -- 172.21.15.40:0/1089804593 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9620026070 con 0x7f962c102d20 2024-03-18T22:42:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.254+0000 7f961effd700 1 -- 172.21.15.40:0/1089804593 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9620004630 con 0x7f962c102d20 2024-03-18T22:42:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.254+0000 7f9633cee700 1 -- 172.21.15.40:0/1089804593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f962c199200 con 0x7f962c102d20 2024-03-18T22:42:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.254+0000 7f961effd700 1 -- 172.21.15.40:0/1089804593 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962000fb20 con 0x7f962c102d20 2024-03-18T22:42:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.254+0000 7f9633cee700 1 -- 172.21.15.40:0/1089804593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f962c1a5470 con 0x7f962c102d20 2024-03-18T22:42:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.255+0000 7f9633cee700 1 -- 172.21.15.40:0/1089804593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f962c1039f0 con 0x7f962c102d20 2024-03-18T22:42:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.256+0000 7f961effd700 1 -- 172.21.15.40:0/1089804593 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9620029070 con 0x7f962c102d20 2024-03-18T22:42:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.257+0000 7f961effd700 1 --2- 172.21.15.40:0/1089804593 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f96180a2890 0x7f96180a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.257+0000 7f961effd700 1 -- 172.21.15.40:0/1089804593 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f96200cf9d0 con 0x7f962c102d20 2024-03-18T22:42:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.257+0000 7f9631a8a700 1 --2- 172.21.15.40:0/1089804593 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f96180a2890 0x7f96180a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.258+0000 7f9631a8a700 1 --2- 172.21.15.40:0/1089804593 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f96180a2890 0x7f96180a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9628009c60 tx=0x7f9628009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.262+0000 7f961effd700 1 -- 172.21.15.40:0/1089804593 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9620030080 con 0x7f962c102d20 2024-03-18T22:42:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.470+0000 7f9633cee700 1 -- 172.21.15.40:0/1089804593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f962c199cc0 con 0x7f962c102d20 2024-03-18T22:42:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.470+0000 7f961effd700 1 -- 172.21.15.40:0/1089804593 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f9620013070 con 0x7f962c102d20 2024-03-18T22:42:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.472+0000 7f9633cee700 1 -- 172.21.15.40:0/1089804593 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f96180a2890 msgr2=0x7f96180a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.472+0000 7f9633cee700 1 --2- 172.21.15.40:0/1089804593 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f96180a2890 0x7f96180a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9628009c60 tx=0x7f9628009260 comp rx=0 tx=0).stop 2024-03-18T22:42:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.472+0000 7f9633cee700 1 -- 172.21.15.40:0/1089804593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c102d20 msgr2=0x7f962c19eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.472+0000 7f9633cee700 1 --2- 172.21.15.40:0/1089804593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c102d20 0x7f962c19eb50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9620006010 tx=0x7f9620004270 comp rx=0 tx=0).stop 2024-03-18T22:42:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.472+0000 7f9633cee700 1 -- 172.21.15.40:0/1089804593 shutdown_connections 2024-03-18T22:42:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.472+0000 7f9633cee700 1 --2- 172.21.15.40:0/1089804593 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f96180a2890 0x7f96180a4d50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.472+0000 7f9633cee700 1 --2- 172.21.15.40:0/1089804593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f962c108340 0x7f962c198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.472+0000 7f9633cee700 1 --2- 172.21.15.40:0/1089804593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f962c102360 0x7f962c19e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.472+0000 7f9633cee700 1 --2- 172.21.15.40:0/1089804593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f962c102d20 0x7f962c19eb50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.472+0000 7f9633cee700 1 -- 172.21.15.40:0/1089804593 >> 172.21.15.40:0/1089804593 conn(0x7f962c079440 msgr2=0x7f962c1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.472+0000 7f9633cee700 1 -- 172.21.15.40:0/1089804593 shutdown_connections 2024-03-18T22:42:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.472+0000 7f9633cee700 1 -- 172.21.15.40:0/1089804593 wait complete. 2024-03-18T22:42:16.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.577+0000 7fb94d016700 1 -- 172.21.15.40:0/2805228244 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fb94802dbb0 con 0x7fb948107620 2024-03-18T22:42:16.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.577+0000 7fb92ffff700 1 -- 172.21.15.40:0/2805228244 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb940010040 con 0x7fb948107620 2024-03-18T22:42:16.578 INFO:teuthology.orchestra.run.smithi040.stdout:55834574928 2024-03-18T22:42:16.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.579+0000 7fb94d016700 1 -- 172.21.15.40:0/2805228244 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9300a2890 msgr2=0x7fb9300a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:16.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.579+0000 7fb94d016700 1 --2- 172.21.15.40:0/2805228244 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9300a2890 0x7fb9300a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb9481072c0 tx=0x7fb938009260 comp rx=0 tx=0).stop 2024-03-18T22:42:16.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.579+0000 7fb94d016700 1 -- 172.21.15.40:0/2805228244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948107620 msgr2=0x7fb94819a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:16.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.579+0000 7fb94d016700 1 --2- 172.21.15.40:0/2805228244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948107620 0x7fb94819a990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb94000ef30 tx=0x7fb94000c3a0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.579+0000 7fb94d016700 1 -- 172.21.15.40:0/2805228244 shutdown_connections 2024-03-18T22:42:16.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.579+0000 7fb94d016700 1 --2- 172.21.15.40:0/2805228244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb948107620 0x7fb94819a990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.579+0000 7fb94d016700 1 --2- 172.21.15.40:0/2805228244 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9300a2890 0x7fb9300a4d50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.579+0000 7fb94d016700 1 --2- 172.21.15.40:0/2805228244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb94806aab0 0x7fb9481a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.579+0000 7fb94d016700 1 --2- 172.21.15.40:0/2805228244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb94806b460 0x7fb9481a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:16.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.579+0000 7fb94d016700 1 -- 172.21.15.40:0/2805228244 >> 172.21.15.40:0/2805228244 conn(0x7fb948077260 msgr2=0x7fb94810d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:16.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.579+0000 7fb94d016700 1 -- 172.21.15.40:0/2805228244 shutdown_connections 2024-03-18T22:42:16.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:16.579+0000 7fb94d016700 1 -- 172.21.15.40:0/2805228244 wait complete. 2024-03-18T22:42:16.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:16 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1089804593' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T22:42:16.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:16 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1089804593' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T22:42:16.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:16 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1089804593' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T22:42:17.281 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574923 got 55834574928 for osd.1 2024-03-18T22:42:17.281 DEBUG:teuthology.parallel:result is None 2024-03-18T22:42:17.766 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383706 got 343597383711 for osd.11 2024-03-18T22:42:17.767 DEBUG:teuthology.parallel:result is None 2024-03-18T22:42:17.767 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-18T22:42:17.767 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T22:42:17.794 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:17 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2805228244' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T22:42:17.794 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:17 smithi040 ceph-mon[62643]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-18T22:42:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:17 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2805228244' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T22:42:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:17 smithi113 ceph-mon[66775]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-18T22:42:17.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:17 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2805228244' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T22:42:17.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:17 smithi103 ceph-mon[66377]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-18T22:42:18.025 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:42:19.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.650+0000 7f8e6b970700 1 -- 172.21.15.40:0/1961334069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e641064f0 msgr2=0x7f8e6410e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:19.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.650+0000 7f8e6b970700 1 --2- 172.21.15.40:0/1961334069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e641064f0 0x7f8e6410e0c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0099c0 tx=0x7f8e5c009cd0 comp rx=0 tx=0).stop 2024-03-18T22:42:19.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.650+0000 7f8e6b970700 1 -- 172.21.15.40:0/1961334069 shutdown_connections 2024-03-18T22:42:19.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.650+0000 7f8e6b970700 1 --2- 172.21.15.40:0/1961334069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e641064f0 0x7f8e6410e0c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:19.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.650+0000 7f8e6b970700 1 --2- 172.21.15.40:0/1961334069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e641014a0 0x7f8e64101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:19.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.650+0000 7f8e6b970700 1 --2- 172.21.15.40:0/1961334069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e64101e50 0x7f8e64105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:19.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.650+0000 7f8e6b970700 1 -- 172.21.15.40:0/1961334069 >> 172.21.15.40:0/1961334069 conn(0x7f8e64077260 msgr2=0x7f8e64077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:19.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.650+0000 7f8e6b970700 1 -- 172.21.15.40:0/1961334069 shutdown_connections 2024-03-18T22:42:19.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.650+0000 7f8e6b970700 1 -- 172.21.15.40:0/1961334069 wait complete. 2024-03-18T22:42:19.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6b970700 1 Processor -- start 2024-03-18T22:42:19.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6b970700 1 -- start start 2024-03-18T22:42:19.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6b970700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e641014a0 0x7f8e6419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:19.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6b970700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e64101e50 0x7f8e6419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:19.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6b970700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e641064f0 0x7f8e641987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:19.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6b970700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8e641113b0 con 0x7f8e641014a0 2024-03-18T22:42:19.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6b970700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8e64111230 con 0x7f8e641064f0 2024-03-18T22:42:19.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6b970700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8e64111530 con 0x7f8e64101e50 2024-03-18T22:42:19.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6970c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e641014a0 0x7f8e6419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:19.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6970c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e641014a0 0x7f8e6419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60980/0 (socket says 172.21.15.40:60980) 2024-03-18T22:42:19.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6970c700 1 -- 172.21.15.40:0/267914829 learned_addr learned my addr 172.21.15.40:0/267914829 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:19.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e68f0b700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e64101e50 0x7f8e6419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:19.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e69f0d700 1 --2- 172.21.15.40:0/267914829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e641064f0 0x7f8e641987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:19.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6970c700 1 -- 172.21.15.40:0/267914829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e64101e50 msgr2=0x7f8e6419eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:19.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6970c700 1 --2- 172.21.15.40:0/267914829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e64101e50 0x7f8e6419eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:19.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6970c700 1 -- 172.21.15.40:0/267914829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e641064f0 msgr2=0x7f8e641987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:19.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6970c700 1 --2- 172.21.15.40:0/267914829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e641064f0 0x7f8e641987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:19.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e6970c700 1 -- 172.21.15.40:0/267914829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e64198fa0 con 0x7f8e641014a0 2024-03-18T22:42:19.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e68f0b700 1 --2- 172.21.15.40:0/267914829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e64101e50 0x7f8e6419eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:19.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.651+0000 7f8e69f0d700 1 --2- 172.21.15.40:0/267914829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e641064f0 0x7f8e641987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:19.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.652+0000 7f8e6970c700 1 --2- 172.21.15.40:0/267914829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e641014a0 0x7f8e6419e640 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400ef10 tx=0x7f8e5400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:19.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.652+0000 7f8e627fc700 1 -- 172.21.15.40:0/267914829 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5400c9c0 con 0x7f8e641014a0 2024-03-18T22:42:19.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.652+0000 7f8e6b970700 1 -- 172.21.15.40:0/267914829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e64199230 con 0x7f8e641014a0 2024-03-18T22:42:19.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.652+0000 7f8e627fc700 1 -- 172.21.15.40:0/267914829 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e54019070 con 0x7f8e641014a0 2024-03-18T22:42:19.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.652+0000 7f8e627fc700 1 -- 172.21.15.40:0/267914829 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e54014480 con 0x7f8e641014a0 2024-03-18T22:42:19.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.652+0000 7f8e6b970700 1 -- 172.21.15.40:0/267914829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e641a54a0 con 0x7f8e641014a0 2024-03-18T22:42:19.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.653+0000 7f8e6b970700 1 -- 172.21.15.40:0/267914829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e641028f0 con 0x7f8e641014a0 2024-03-18T22:42:19.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.654+0000 7f8e627fc700 1 -- 172.21.15.40:0/267914829 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8e54012050 con 0x7f8e641014a0 2024-03-18T22:42:19.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.654+0000 7f8e627fc700 1 --2- 172.21.15.40:0/267914829 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e4c0a2890 0x7f8e4c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:19.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.654+0000 7f8e68f0b700 1 --2- 172.21.15.40:0/267914829 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e4c0a2890 0x7f8e4c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:19.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.654+0000 7f8e627fc700 1 -- 172.21.15.40:0/267914829 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8e540ced50 con 0x7f8e641014a0 2024-03-18T22:42:19.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.655+0000 7f8e68f0b700 1 --2- 172.21.15.40:0/267914829 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e4c0a2890 0x7f8e4c0a4d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8e6406b930 tx=0x7f8e58009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:19.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.659+0000 7f8e627fc700 1 -- 172.21.15.40:0/267914829 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8e540d2030 con 0x7f8e641014a0 2024-03-18T22:42:19.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.858+0000 7f8e6b970700 1 -- 172.21.15.40:0/267914829 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8e6406a850 con 0x7f8e4c0a2890 2024-03-18T22:42:19.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.868+0000 7f8e627fc700 1 -- 172.21.15.40:0/267914829 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340649 (secure 0 0 0) 0x7f8e6406a850 con 0x7f8e4c0a2890 2024-03-18T22:42:19.874 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.871+0000 7f8e6b970700 1 -- 172.21.15.40:0/267914829 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e4c0a2890 msgr2=0x7f8e4c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.871+0000 7f8e6b970700 1 --2- 172.21.15.40:0/267914829 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e4c0a2890 0x7f8e4c0a4d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8e6406b930 tx=0x7f8e58009260 comp rx=0 tx=0).stop 2024-03-18T22:42:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.871+0000 7f8e6b970700 1 -- 172.21.15.40:0/267914829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e641014a0 msgr2=0x7f8e6419e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.871+0000 7f8e6b970700 1 --2- 172.21.15.40:0/267914829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e641014a0 0x7f8e6419e640 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400ef10 tx=0x7f8e5400c3a0 comp rx=0 tx=0).stop 2024-03-18T22:42:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.872+0000 7f8e6b970700 1 -- 172.21.15.40:0/267914829 shutdown_connections 2024-03-18T22:42:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.872+0000 7f8e6b970700 1 --2- 172.21.15.40:0/267914829 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e4c0a2890 0x7f8e4c0a4d50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.872+0000 7f8e6b970700 1 --2- 172.21.15.40:0/267914829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e641064f0 0x7f8e641987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.872+0000 7f8e6b970700 1 --2- 172.21.15.40:0/267914829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e641014a0 0x7f8e6419e640 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.872+0000 7f8e6b970700 1 --2- 172.21.15.40:0/267914829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e64101e50 0x7f8e6419eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.872+0000 7f8e6b970700 1 -- 172.21.15.40:0/267914829 >> 172.21.15.40:0/267914829 conn(0x7f8e64077260 msgr2=0x7f8e64105190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.872+0000 7f8e6b970700 1 -- 172.21.15.40:0/267914829 shutdown_connections 2024-03-18T22:42:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:19.872+0000 7f8e6b970700 1 -- 172.21.15.40:0/267914829 wait complete. 2024-03-18T22:42:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-18T22:42:20.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:19 smithi040 ceph-mon[62643]: pgmap v339: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T22:42:20.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:19 smithi113 ceph-mon[66775]: pgmap v339: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T22:42:20.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:19 smithi103 ceph-mon[66377]: pgmap v339: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T22:42:20.630 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-03-18T22:42:18.831918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662226,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":848,"num_read_kb":549,"num_write":437,"num_write_kb":3091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":861,"ondisk_log_size":861,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384192,"kb_used_data":14240,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557632,"statfs":{"total":1151940427776,"available":1151547015168,"internally_reserved":0,"allocated":14581760,"data_stored":7972388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":124,"num_read_kb":59,"num_write":11,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003854"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.861943+0000","last_change":"2024-03-18T22:40:41.806379+0000","last_active":"2024-03-18T22:42:12.861943+0000","last_peered":"2024-03-18T22:42:12.861943+0000","last_clean":"2024-03-18T22:42:12.861943+0000","last_became_active":"2024-03-18T22:40:41.806058+0000","last_became_peered":"2024-03-18T22:40:41.806058+0000","last_unstale":"2024-03-18T22:42:12.861943+0000","last_undegraded":"2024-03-18T22:42:12.861943+0000","last_fullsized":"2024-03-18T22:42:12.861943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:45.834396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.851643+0000","last_change":"2024-03-18T22:40:41.813722+0000","last_active":"2024-03-18T22:42:12.851643+0000","last_peered":"2024-03-18T22:42:12.851643+0000","last_clean":"2024-03-18T22:42:12.851643+0000","last_became_active":"2024-03-18T22:40:41.813284+0000","last_became_peered":"2024-03-18T22:40:41.813284+0000","last_unstale":"2024-03-18T22:42:12.851643+0000","last_undegraded":"2024-03-18T22:42:12.851643+0000","last_fullsized":"2024-03-18T22:42:12.851643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:46.374117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.857256+0000","last_change":"2024-03-18T22:40:41.808340+0000","last_active":"2024-03-18T22:42:12.857256+0000","last_peered":"2024-03-18T22:42:12.857256+0000","last_clean":"2024-03-18T22:42:12.857256+0000","last_became_active":"2024-03-18T22:40:41.808002+0000","last_became_peered":"2024-03-18T22:40:41.808002+0000","last_unstale":"2024-03-18T22:42:12.857256+0000","last_undegraded":"2024-03-18T22:42:12.857256+0000","last_fullsized":"2024-03-18T22:42:12.857256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:40:32.122124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.832643+0000","last_change":"2024-03-18T22:40:41.809959+0000","last_active":"2024-03-18T22:42:12.832643+0000","last_peered":"2024-03-18T22:42:12.832643+0000","last_clean":"2024-03-18T22:42:12.832643+0000","last_became_active":"2024-03-18T22:40:41.809676+0000","last_became_peered":"2024-03-18T22:40:41.809676+0000","last_unstale":"2024-03-18T22:42:12.832643+0000","last_undegraded":"2024-03-18T22:42:12.832643+0000","last_fullsized":"2024-03-18T22:42:12.832643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:29:45.263011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.825175+0000","last_change":"2024-03-18T22:40:41.804583+0000","last_active":"2024-03-18T22:42:12.825175+0000","last_peered":"2024-03-18T22:42:12.825175+0000","last_clean":"2024-03-18T22:42:12.825175+0000","last_became_active":"2024-03-18T22:40:41.804064+0000","last_became_peered":"2024-03-18T22:40:41.804064+0000","last_unstale":"2024-03-18T22:42:12.825175+0000","last_undegraded":"2024-03-18T22:42:12.825175+0000","last_fullsized":"2024-03-18T22:42:12.825175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:46:23.455106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.855509+0000","last_change":"2024-03-18T22:41:44.301195+0000","last_active":"2024-03-18T22:42:12.855509+0000","last_peered":"2024-03-18T22:42:12.855509+0000","last_clean":"2024-03-18T22:42:12.855509+0000","last_became_active":"2024-03-18T22:41:44.300801+0000","last_became_peered":"2024-03-18T22:41:44.300801+0000","last_unstale":"2024-03-18T22:42:12.855509+0000","last_undegraded":"2024-03-18T22:42:12.855509+0000","last_fullsized":"2024-03-18T22:42:12.855509+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:12:55.579977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.832006+0000","last_change":"2024-03-18T22:40:41.814971+0000","last_active":"2024-03-18T22:42:12.832006+0000","last_peered":"2024-03-18T22:42:12.832006+0000","last_clean":"2024-03-18T22:42:12.832006+0000","last_became_active":"2024-03-18T22:40:41.814305+0000","last_became_peered":"2024-03-18T22:40:41.814305+0000","last_unstale":"2024-03-18T22:42:12.832006+0000","last_undegraded":"2024-03-18T22:42:12.832006+0000","last_fullsized":"2024-03-18T22:42:12.832006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:07.390157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.851956+0000","last_change":"2024-03-18T22:40:41.815674+0000","last_active":"2024-03-18T22:42:12.851956+0000","last_peered":"2024-03-18T22:42:12.851956+0000","last_clean":"2024-03-18T22:42:12.851956+0000","last_became_active":"2024-03-18T22:40:41.813560+0000","last_became_peered":"2024-03-18T22:40:41.813560+0000","last_unstale":"2024-03-18T22:42:12.851956+0000","last_undegraded":"2024-03-18T22:42:12.851956+0000","last_fullsized":"2024-03-18T22:42:12.851956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:42:32.409338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.862612+0000","last_change":"2024-03-18T22:40:41.817508+0000","last_active":"2024-03-18T22:42:12.862612+0000","last_peered":"2024-03-18T22:42:12.862612+0000","last_clean":"2024-03-18T22:42:12.862612+0000","last_became_active":"2024-03-18T22:40:41.817184+0000","last_became_peered":"2024-03-18T22:40:41.817184+0000","last_unstale":"2024-03-18T22:42:12.862612+0000","last_undegraded":"2024-03-18T22:42:12.862612+0000","last_fullsized":"2024-03-18T22:42:12.862612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:10.255238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.848287+0000","last_change":"2024-03-18T22:40:41.806235+0000","last_active":"2024-03-18T22:42:12.848287+0000","last_peered":"2024-03-18T22:42:12.848287+0000","last_clean":"2024-03-18T22:42:12.848287+0000","last_became_active":"2024-03-18T22:40:41.805913+0000","last_became_peered":"2024-03-18T22:40:41.805913+0000","last_unstale":"2024-03-18T22:42:12.848287+0000","last_undegraded":"2024-03-18T22:42:12.848287+0000","last_fullsized":"2024-03-18T22:42:12.848287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:39:06.299989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.856751+0000","last_change":"2024-03-18T22:40:41.814735+0000","last_active":"2024-03-18T22:42:12.856751+0000","last_peered":"2024-03-18T22:42:12.856751+0000","last_clean":"2024-03-18T22:42:12.856751+0000","last_became_active":"2024-03-18T22:40:41.814291+0000","last_became_peered":"2024-03-18T22:40:41.814291+0000","last_unstale":"2024-03-18T22:42:12.856751+0000","last_undegraded":"2024-03-18T22:42:12.856751+0000","last_fullsized":"2024-03-18T22:42:12.856751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:49.610674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.851736+0000","last_change":"2024-03-18T22:40:41.815561+0000","last_active":"2024-03-18T22:42:12.851736+0000","last_peered":"2024-03-18T22:42:12.851736+0000","last_clean":"2024-03-18T22:42:12.851736+0000","last_became_active":"2024-03-18T22:40:41.815252+0000","last_became_peered":"2024-03-18T22:40:41.815252+0000","last_unstale":"2024-03-18T22:42:12.851736+0000","last_undegraded":"2024-03-18T22:42:12.851736+0000","last_fullsized":"2024-03-18T22:42:12.851736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:34.769168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.848344+0000","last_change":"2024-03-18T22:40:41.803325+0000","last_active":"2024-03-18T22:42:12.848344+0000","last_peered":"2024-03-18T22:42:12.848344+0000","last_clean":"2024-03-18T22:42:12.848344+0000","last_became_active":"2024-03-18T22:40:41.803008+0000","last_became_peered":"2024-03-18T22:40:41.803008+0000","last_unstale":"2024-03-18T22:42:12.848344+0000","last_undegraded":"2024-03-18T22:42:12.848344+0000","last_fullsized":"2024-03-18T22:42:12.848344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:14:01.335548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.851770+0000","last_change":"2024-03-18T22:40:41.812541+0000","last_active":"2024-03-18T22:42:12.851770+0000","last_peered":"2024-03-18T22:42:12.851770+0000","last_clean":"2024-03-18T22:42:12.851770+0000","last_became_active":"2024-03-18T22:40:41.812153+0000","last_became_peered":"2024-03-18T22:40:41.812153+0000","last_unstale":"2024-03-18T22:42:12.851770+0000","last_undegraded":"2024-03-18T22:42:12.851770+0000","last_fullsized":"2024-03-18T22:42:12.851770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:04.927057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.845234+0000","last_change":"2024-03-18T22:40:41.815790+0000","last_active":"2024-03-18T22:42:12.845234+0000","last_peered":"2024-03-18T22:42:12.845234+0000","last_clean":"2024-03-18T22:42:12.845234+0000","last_became_active":"2024-03-18T22:40:41.814949+0000","last_became_peered":"2024-03-18T22:40:41.814949+0000","last_unstale":"2024-03-18T22:42:12.845234+0000","last_undegraded":"2024-03-18T22:42:12.845234+0000","last_fullsized":"2024-03-18T22:42:12.845234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:10:15.874285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.858727+0000","last_change":"2024-03-18T22:40:41.809980+0000","last_active":"2024-03-18T22:42:12.858727+0000","last_peered":"2024-03-18T22:42:12.858727+0000","last_clean":"2024-03-18T22:42:12.858727+0000","last_became_active":"2024-03-18T22:40:41.809640+0000","last_became_peered":"2024-03-18T22:40:41.809640+0000","last_unstale":"2024-03-18T22:42:12.858727+0000","last_undegraded":"2024-03-18T22:42:12.858727+0000","last_fullsized":"2024-03-18T22:42:12.858727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:25.389027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.862116+0000","last_change":"2024-03-18T22:40:41.807088+0000","last_active":"2024-03-18T22:42:12.862116+0000","last_peered":"2024-03-18T22:42:12.862116+0000","last_clean":"2024-03-18T22:42:12.862116+0000","last_became_active":"2024-03-18T22:40:41.806770+0000","last_became_peered":"2024-03-18T22:40:41.806770+0000","last_unstale":"2024-03-18T22:42:12.862116+0000","last_undegraded":"2024-03-18T22:42:12.862116+0000","last_fullsized":"2024-03-18T22:42:12.862116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:53:14.001247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.851434+0000","last_change":"2024-03-18T22:40:41.812685+0000","last_active":"2024-03-18T22:42:12.851434+0000","last_peered":"2024-03-18T22:42:12.851434+0000","last_clean":"2024-03-18T22:42:12.851434+0000","last_became_active":"2024-03-18T22:40:41.812346+0000","last_became_peered":"2024-03-18T22:40:41.812346+0000","last_unstale":"2024-03-18T22:42:12.851434+0000","last_undegraded":"2024-03-18T22:42:12.851434+0000","last_fullsized":"2024-03-18T22:42:12.851434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:25.033460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.827948+0000","last_change":"2024-03-18T22:40:41.812627+0000","last_active":"2024-03-18T22:42:12.827948+0000","last_peered":"2024-03-18T22:42:12.827948+0000","last_clean":"2024-03-18T22:42:12.827948+0000","last_became_active":"2024-03-18T22:40:41.812327+0000","last_became_peered":"2024-03-18T22:40:41.812327+0000","last_unstale":"2024-03-18T22:42:12.827948+0000","last_undegraded":"2024-03-18T22:42:12.827948+0000","last_fullsized":"2024-03-18T22:42:12.827948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:58:22.088589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001429+0000","last_change":"2024-03-18T22:40:45.847822+0000","last_active":"2024-03-18T22:41:44.001429+0000","last_peered":"2024-03-18T22:41:44.001429+0000","last_clean":"2024-03-18T22:41:44.001429+0000","last_became_active":"2024-03-18T22:40:45.847407+0000","last_became_peered":"2024-03-18T22:40:45.847407+0000","last_unstale":"2024-03-18T22:41:44.001429+0000","last_undegraded":"2024-03-18T22:41:44.001429+0000","last_fullsized":"2024-03-18T22:41:44.001429+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:44.825997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'14","reported_seq":111,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:16.932250+0000","last_change":"2024-03-18T22:40:41.218797+0000","last_active":"2024-03-18T22:42:16.932250+0000","last_peered":"2024-03-18T22:42:16.932250+0000","last_clean":"2024-03-18T22:42:16.932250+0000","last_became_active":"2024-03-18T22:40:41.218472+0000","last_became_peered":"2024-03-18T22:40:41.218472+0000","last_unstale":"2024-03-18T22:42:16.932250+0000","last_undegraded":"2024-03-18T22:42:16.932250+0000","last_fullsized":"2024-03-18T22:42:16.932250+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:10.230257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.831788+0000","last_change":"2024-03-18T22:41:44.005444+0000","last_active":"2024-03-18T22:42:12.831788+0000","last_peered":"2024-03-18T22:42:12.831788+0000","last_clean":"2024-03-18T22:42:12.831788+0000","last_became_active":"2024-03-18T22:41:44.005157+0000","last_became_peered":"2024-03-18T22:41:44.005157+0000","last_unstale":"2024-03-18T22:42:12.831788+0000","last_undegraded":"2024-03-18T22:42:12.831788+0000","last_fullsized":"2024-03-18T22:42:12.831788+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:53:16.249875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997343+0000","last_change":"2024-03-18T22:40:45.851126+0000","last_active":"2024-03-18T22:41:43.997343+0000","last_peered":"2024-03-18T22:41:43.997343+0000","last_clean":"2024-03-18T22:41:43.997343+0000","last_became_active":"2024-03-18T22:40:45.850549+0000","last_became_peered":"2024-03-18T22:40:45.850549+0000","last_unstale":"2024-03-18T22:41:43.997343+0000","last_undegraded":"2024-03-18T22:41:43.997343+0000","last_fullsized":"2024-03-18T22:41:43.997343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:39.359263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999819+0000","last_change":"2024-03-18T22:40:41.220944+0000","last_active":"2024-03-18T22:41:43.999819+0000","last_peered":"2024-03-18T22:41:43.999819+0000","last_clean":"2024-03-18T22:41:43.999819+0000","last_became_active":"2024-03-18T22:40:41.220612+0000","last_became_peered":"2024-03-18T22:40:41.220612+0000","last_unstale":"2024-03-18T22:41:43.999819+0000","last_undegraded":"2024-03-18T22:41:43.999819+0000","last_fullsized":"2024-03-18T22:41:43.999819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:16.083134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.857160+0000","last_change":"2024-03-18T22:40:41.818031+0000","last_active":"2024-03-18T22:42:12.857160+0000","last_peered":"2024-03-18T22:42:12.857160+0000","last_clean":"2024-03-18T22:42:12.857160+0000","last_became_active":"2024-03-18T22:40:41.817693+0000","last_became_peered":"2024-03-18T22:40:41.817693+0000","last_unstale":"2024-03-18T22:42:12.857160+0000","last_undegraded":"2024-03-18T22:42:12.857160+0000","last_fullsized":"2024-03-18T22:42:12.857160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:05.901203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.998307+0000","last_change":"2024-03-18T22:40:45.850749+0000","last_active":"2024-03-18T22:41:43.998307+0000","last_peered":"2024-03-18T22:41:43.998307+0000","last_clean":"2024-03-18T22:41:43.998307+0000","last_became_active":"2024-03-18T22:40:45.850318+0000","last_became_peered":"2024-03-18T22:40:45.850318+0000","last_unstale":"2024-03-18T22:41:43.998307+0000","last_undegraded":"2024-03-18T22:41:43.998307+0000","last_fullsized":"2024-03-18T22:41:43.998307+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:56.047413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997214+0000","last_change":"2024-03-18T22:40:41.220775+0000","last_active":"2024-03-18T22:41:43.997214+0000","last_peered":"2024-03-18T22:41:43.997214+0000","last_clean":"2024-03-18T22:41:43.997214+0000","last_became_active":"2024-03-18T22:40:41.220313+0000","last_became_peered":"2024-03-18T22:40:41.220313+0000","last_unstale":"2024-03-18T22:41:43.997214+0000","last_undegraded":"2024-03-18T22:41:43.997214+0000","last_fullsized":"2024-03-18T22:41:43.997214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:24.180884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.858766+0000","last_change":"2024-03-18T22:40:41.818448+0000","last_active":"2024-03-18T22:42:12.858766+0000","last_peered":"2024-03-18T22:42:12.858766+0000","last_clean":"2024-03-18T22:42:12.858766+0000","last_became_active":"2024-03-18T22:40:41.818088+0000","last_became_peered":"2024-03-18T22:40:41.818088+0000","last_unstale":"2024-03-18T22:42:12.858766+0000","last_undegraded":"2024-03-18T22:42:12.858766+0000","last_fullsized":"2024-03-18T22:42:12.858766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:34:54.298226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997546+0000","last_change":"2024-03-18T22:40:45.846250+0000","last_active":"2024-03-18T22:41:43.997546+0000","last_peered":"2024-03-18T22:41:43.997546+0000","last_clean":"2024-03-18T22:41:43.997546+0000","last_became_active":"2024-03-18T22:40:45.845859+0000","last_became_peered":"2024-03-18T22:40:45.845859+0000","last_unstale":"2024-03-18T22:41:43.997546+0000","last_undegraded":"2024-03-18T22:41:43.997546+0000","last_fullsized":"2024-03-18T22:41:43.997546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:57.855301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997708+0000","last_change":"2024-03-18T22:40:41.216569+0000","last_active":"2024-03-18T22:41:43.997708+0000","last_peered":"2024-03-18T22:41:43.997708+0000","last_clean":"2024-03-18T22:41:43.997708+0000","last_became_active":"2024-03-18T22:40:41.216255+0000","last_became_peered":"2024-03-18T22:40:41.216255+0000","last_unstale":"2024-03-18T22:41:43.997708+0000","last_undegraded":"2024-03-18T22:41:43.997708+0000","last_fullsized":"2024-03-18T22:41:43.997708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:23.016346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.859296+0000","last_change":"2024-03-18T22:41:44.301578+0000","last_active":"2024-03-18T22:42:12.859296+0000","last_peered":"2024-03-18T22:42:12.859296+0000","last_clean":"2024-03-18T22:42:12.859296+0000","last_became_active":"2024-03-18T22:41:44.301178+0000","last_became_peered":"2024-03-18T22:41:44.301178+0000","last_unstale":"2024-03-18T22:42:12.859296+0000","last_undegraded":"2024-03-18T22:42:12.859296+0000","last_fullsized":"2024-03-18T22:42:12.859296+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:25:50.635812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.302390+0000","last_change":"2024-03-18T22:41:44.302390+0000","last_active":"2024-03-18T22:41:44.302390+0000","last_peered":"2024-03-18T22:41:44.302390+0000","last_clean":"2024-03-18T22:41:44.302390+0000","last_became_active":"2024-03-18T22:41:44.301920+0000","last_became_peered":"2024-03-18T22:41:44.301920+0000","last_unstale":"2024-03-18T22:41:44.302390+0000","last_undegraded":"2024-03-18T22:41:44.302390+0000","last_fullsized":"2024-03-18T22:41:44.302390+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:56:12.561520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,7,11],"acting":[2,8,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:13.891549+0000","last_change":"2024-03-18T22:40:41.210196+0000","last_active":"2024-03-18T22:42:13.891549+0000","last_peered":"2024-03-18T22:42:13.891549+0000","last_clean":"2024-03-18T22:42:13.891549+0000","last_became_active":"2024-03-18T22:40:41.209888+0000","last_became_peered":"2024-03-18T22:40:41.209888+0000","last_unstale":"2024-03-18T22:42:13.891549+0000","last_undegraded":"2024-03-18T22:42:13.891549+0000","last_fullsized":"2024-03-18T22:42:13.891549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:06:31.134688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.835531+0000","last_change":"2024-03-18T22:41:44.004814+0000","last_active":"2024-03-18T22:42:12.835531+0000","last_peered":"2024-03-18T22:42:12.835531+0000","last_clean":"2024-03-18T22:42:12.835531+0000","last_became_active":"2024-03-18T22:41:44.004219+0000","last_became_peered":"2024-03-18T22:41:44.004219+0000","last_unstale":"2024-03-18T22:42:12.835531+0000","last_undegraded":"2024-03-18T22:42:12.835531+0000","last_fullsized":"2024-03-18T22:42:12.835531+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:29:03.746508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.005790+0000","last_change":"2024-03-18T22:41:44.005790+0000","last_active":"2024-03-18T22:41:44.005790+0000","last_peered":"2024-03-18T22:41:44.005790+0000","last_clean":"2024-03-18T22:41:44.005790+0000","last_became_active":"2024-03-18T22:41:44.005386+0000","last_became_peered":"2024-03-18T22:41:44.005386+0000","last_unstale":"2024-03-18T22:41:44.005790+0000","last_undegraded":"2024-03-18T22:41:44.005790+0000","last_fullsized":"2024-03-18T22:41:44.005790+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:49:08.212639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.000320+0000","last_change":"2024-03-18T22:40:41.217449+0000","last_active":"2024-03-18T22:41:44.000320+0000","last_peered":"2024-03-18T22:41:44.000320+0000","last_clean":"2024-03-18T22:41:44.000320+0000","last_became_active":"2024-03-18T22:40:41.217127+0000","last_became_peered":"2024-03-18T22:40:41.217127+0000","last_unstale":"2024-03-18T22:41:44.000320+0000","last_undegraded":"2024-03-18T22:41:44.000320+0000","last_fullsized":"2024-03-18T22:41:44.000320+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:08:00.178720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.838151+0000","last_change":"2024-03-18T22:40:41.814154+0000","last_active":"2024-03-18T22:42:12.838151+0000","last_peered":"2024-03-18T22:42:12.838151+0000","last_clean":"2024-03-18T22:42:12.838151+0000","last_became_active":"2024-03-18T22:40:41.813824+0000","last_became_peered":"2024-03-18T22:40:41.813824+0000","last_unstale":"2024-03-18T22:42:12.838151+0000","last_undegraded":"2024-03-18T22:42:12.838151+0000","last_fullsized":"2024-03-18T22:42:12.838151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:12.412689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.265037+0000","last_change":"2024-03-18T22:40:45.852711+0000","last_active":"2024-03-18T22:41:44.265037+0000","last_peered":"2024-03-18T22:41:44.265037+0000","last_clean":"2024-03-18T22:41:44.265037+0000","last_became_active":"2024-03-18T22:40:45.852247+0000","last_became_peered":"2024-03-18T22:40:45.852247+0000","last_unstale":"2024-03-18T22:41:44.265037+0000","last_undegraded":"2024-03-18T22:41:44.265037+0000","last_fullsized":"2024-03-18T22:41:44.265037+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:03.061689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001812+0000","last_change":"2024-03-18T22:40:41.212399+0000","last_active":"2024-03-18T22:41:44.001812+0000","last_peered":"2024-03-18T22:41:44.001812+0000","last_clean":"2024-03-18T22:41:44.001812+0000","last_became_active":"2024-03-18T22:40:41.207416+0000","last_became_peered":"2024-03-18T22:40:41.207416+0000","last_unstale":"2024-03-18T22:41:44.001812+0000","last_undegraded":"2024-03-18T22:41:44.001812+0000","last_fullsized":"2024-03-18T22:41:44.001812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:11.406342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.831198+0000","last_change":"2024-03-18T22:40:41.807313+0000","last_active":"2024-03-18T22:42:12.831198+0000","last_peered":"2024-03-18T22:42:12.831198+0000","last_clean":"2024-03-18T22:42:12.831198+0000","last_became_active":"2024-03-18T22:40:41.806029+0000","last_became_peered":"2024-03-18T22:40:41.806029+0000","last_unstale":"2024-03-18T22:42:12.831198+0000","last_undegraded":"2024-03-18T22:42:12.831198+0000","last_fullsized":"2024-03-18T22:42:12.831198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:23:43.973443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.333855+0000","last_change":"2024-03-18T22:40:45.835479+0000","last_active":"2024-03-18T22:41:44.333855+0000","last_peered":"2024-03-18T22:41:44.333855+0000","last_clean":"2024-03-18T22:41:44.333855+0000","last_became_active":"2024-03-18T22:40:45.834994+0000","last_became_peered":"2024-03-18T22:40:45.834994+0000","last_unstale":"2024-03-18T22:41:44.333855+0000","last_undegraded":"2024-03-18T22:41:44.333855+0000","last_fullsized":"2024-03-18T22:41:44.333855+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:31.317843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999237+0000","last_change":"2024-03-18T22:40:41.212110+0000","last_active":"2024-03-18T22:41:43.999237+0000","last_peered":"2024-03-18T22:41:43.999237+0000","last_clean":"2024-03-18T22:41:43.999237+0000","last_became_active":"2024-03-18T22:40:41.211803+0000","last_became_peered":"2024-03-18T22:40:41.211803+0000","last_unstale":"2024-03-18T22:41:43.999237+0000","last_undegraded":"2024-03-18T22:41:43.999237+0000","last_fullsized":"2024-03-18T22:41:43.999237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:40.482983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.842272+0000","last_change":"2024-03-18T22:40:41.812585+0000","last_active":"2024-03-18T22:42:12.842272+0000","last_peered":"2024-03-18T22:42:12.842272+0000","last_clean":"2024-03-18T22:42:12.842272+0000","last_became_active":"2024-03-18T22:40:41.812260+0000","last_became_peered":"2024-03-18T22:40:41.812260+0000","last_unstale":"2024-03-18T22:42:12.842272+0000","last_undegraded":"2024-03-18T22:42:12.842272+0000","last_fullsized":"2024-03-18T22:42:12.842272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:12:59.999503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999606+0000","last_change":"2024-03-18T22:40:45.842252+0000","last_active":"2024-03-18T22:41:43.999606+0000","last_peered":"2024-03-18T22:41:43.999606+0000","last_clean":"2024-03-18T22:41:43.999606+0000","last_became_active":"2024-03-18T22:40:45.841849+0000","last_became_peered":"2024-03-18T22:40:45.841849+0000","last_unstale":"2024-03-18T22:41:43.999606+0000","last_undegraded":"2024-03-18T22:41:43.999606+0000","last_fullsized":"2024-03-18T22:41:43.999606+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:21.253947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":164,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:13.890917+0000","last_change":"2024-03-18T22:40:41.214253+0000","last_active":"2024-03-18T22:42:13.890917+0000","last_peered":"2024-03-18T22:42:13.890917+0000","last_clean":"2024-03-18T22:42:13.890917+0000","last_became_active":"2024-03-18T22:40:41.213911+0000","last_became_peered":"2024-03-18T22:40:41.213911+0000","last_unstale":"2024-03-18T22:42:13.890917+0000","last_undegraded":"2024-03-18T22:42:13.890917+0000","last_fullsized":"2024-03-18T22:42:13.890917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:19.185714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.858816+0000","last_change":"2024-03-18T22:40:41.813281+0000","last_active":"2024-03-18T22:42:12.858816+0000","last_peered":"2024-03-18T22:42:12.858816+0000","last_clean":"2024-03-18T22:42:12.858816+0000","last_became_active":"2024-03-18T22:40:41.812909+0000","last_became_peered":"2024-03-18T22:40:41.812909+0000","last_unstale":"2024-03-18T22:42:12.858816+0000","last_undegraded":"2024-03-18T22:42:12.858816+0000","last_fullsized":"2024-03-18T22:42:12.858816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:17:26.619441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001859+0000","last_change":"2024-03-18T22:40:45.846956+0000","last_active":"2024-03-18T22:41:44.001859+0000","last_peered":"2024-03-18T22:41:44.001859+0000","last_clean":"2024-03-18T22:41:44.001859+0000","last_became_active":"2024-03-18T22:40:45.843061+0000","last_became_peered":"2024-03-18T22:40:45.843061+0000","last_unstale":"2024-03-18T22:41:44.001859+0000","last_undegraded":"2024-03-18T22:41:44.001859+0000","last_fullsized":"2024-03-18T22:41:44.001859+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:52.043424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.300418+0000","last_change":"2024-03-18T22:40:41.214933+0000","last_active":"2024-03-18T22:41:44.300418+0000","last_peered":"2024-03-18T22:41:44.300418+0000","last_clean":"2024-03-18T22:41:44.300418+0000","last_became_active":"2024-03-18T22:40:41.214606+0000","last_became_peered":"2024-03-18T22:40:41.214606+0000","last_unstale":"2024-03-18T22:41:44.300418+0000","last_undegraded":"2024-03-18T22:41:44.300418+0000","last_fullsized":"2024-03-18T22:41:44.300418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:08:06.230288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.845541+0000","last_change":"2024-03-18T22:40:41.817590+0000","last_active":"2024-03-18T22:42:12.845541+0000","last_peered":"2024-03-18T22:42:12.845541+0000","last_clean":"2024-03-18T22:42:12.845541+0000","last_became_active":"2024-03-18T22:40:41.815809+0000","last_became_peered":"2024-03-18T22:40:41.815809+0000","last_unstale":"2024-03-18T22:42:12.845541+0000","last_undegraded":"2024-03-18T22:42:12.845541+0000","last_fullsized":"2024-03-18T22:42:12.845541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:44:54.043647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.998518+0000","last_change":"2024-03-18T22:40:45.844767+0000","last_active":"2024-03-18T22:41:43.998518+0000","last_peered":"2024-03-18T22:41:43.998518+0000","last_clean":"2024-03-18T22:41:43.998518+0000","last_became_active":"2024-03-18T22:40:45.844251+0000","last_became_peered":"2024-03-18T22:40:45.844251+0000","last_unstale":"2024-03-18T22:41:43.998518+0000","last_undegraded":"2024-03-18T22:41:43.998518+0000","last_fullsized":"2024-03-18T22:41:43.998518+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:43.756959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.000056+0000","last_change":"2024-03-18T22:40:41.214310+0000","last_active":"2024-03-18T22:41:44.000056+0000","last_peered":"2024-03-18T22:41:44.000056+0000","last_clean":"2024-03-18T22:41:44.000056+0000","last_became_active":"2024-03-18T22:40:41.213963+0000","last_became_peered":"2024-03-18T22:40:41.213963+0000","last_unstale":"2024-03-18T22:41:44.000056+0000","last_undegraded":"2024-03-18T22:41:44.000056+0000","last_fullsized":"2024-03-18T22:41:44.000056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:03.401220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.860732+0000","last_change":"2024-03-18T22:40:41.810876+0000","last_active":"2024-03-18T22:42:12.860732+0000","last_peered":"2024-03-18T22:42:12.860732+0000","last_clean":"2024-03-18T22:42:12.860732+0000","last_became_active":"2024-03-18T22:40:41.810104+0000","last_became_peered":"2024-03-18T22:40:41.810104+0000","last_unstale":"2024-03-18T22:42:12.860732+0000","last_undegraded":"2024-03-18T22:42:12.860732+0000","last_fullsized":"2024-03-18T22:42:12.860732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:41.294286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.266735+0000","last_change":"2024-03-18T22:41:44.266318+0000","last_active":"2024-03-18T22:41:44.266735+0000","last_peered":"2024-03-18T22:41:44.266735+0000","last_clean":"2024-03-18T22:41:44.266735+0000","last_became_active":"2024-03-18T22:41:44.265903+0000","last_became_peered":"2024-03-18T22:41:44.265903+0000","last_unstale":"2024-03-18T22:41:44.266735+0000","last_undegraded":"2024-03-18T22:41:44.266735+0000","last_fullsized":"2024-03-18T22:41:44.266735+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:44.307096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":209,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:13.890900+0000","last_change":"2024-03-18T22:40:41.209837+0000","last_active":"2024-03-18T22:42:13.890900+0000","last_peered":"2024-03-18T22:42:13.890900+0000","last_clean":"2024-03-18T22:42:13.890900+0000","last_became_active":"2024-03-18T22:40:41.209465+0000","last_became_peered":"2024-03-18T22:40:41.209465+0000","last_unstale":"2024-03-18T22:42:13.890900+0000","last_undegraded":"2024-03-18T22:42:13.890900+0000","last_fullsized":"2024-03-18T22:42:13.890900+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:45:15.765475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63606,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":144,"num_read_kb":135,"num_write":40,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.854214+0000","last_change":"2024-03-18T22:40:41.810058+0000","last_active":"2024-03-18T22:42:12.854214+0000","last_peered":"2024-03-18T22:42:12.854214+0000","last_clean":"2024-03-18T22:42:12.854214+0000","last_became_active":"2024-03-18T22:40:41.809677+0000","last_became_peered":"2024-03-18T22:40:41.809677+0000","last_unstale":"2024-03-18T22:42:12.854214+0000","last_undegraded":"2024-03-18T22:42:12.854214+0000","last_fullsized":"2024-03-18T22:42:12.854214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:03.935036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.000269+0000","last_change":"2024-03-18T22:40:45.845638+0000","last_active":"2024-03-18T22:41:44.000269+0000","last_peered":"2024-03-18T22:41:44.000269+0000","last_clean":"2024-03-18T22:41:44.000269+0000","last_became_active":"2024-03-18T22:40:45.844994+0000","last_became_peered":"2024-03-18T22:40:45.844994+0000","last_unstale":"2024-03-18T22:41:44.000269+0000","last_undegraded":"2024-03-18T22:41:44.000269+0000","last_fullsized":"2024-03-18T22:41:44.000269+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:21.551271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001706+0000","last_change":"2024-03-18T22:40:41.218397+0000","last_active":"2024-03-18T22:41:44.001706+0000","last_peered":"2024-03-18T22:41:44.001706+0000","last_clean":"2024-03-18T22:41:44.001706+0000","last_became_active":"2024-03-18T22:40:41.217587+0000","last_became_peered":"2024-03-18T22:40:41.217587+0000","last_unstale":"2024-03-18T22:41:44.001706+0000","last_undegraded":"2024-03-18T22:41:44.001706+0000","last_fullsized":"2024-03-18T22:41:44.001706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:36.524954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.839536+0000","last_change":"2024-03-18T22:40:41.812026+0000","last_active":"2024-03-18T22:42:12.839536+0000","last_peered":"2024-03-18T22:42:12.839536+0000","last_clean":"2024-03-18T22:42:12.839536+0000","last_became_active":"2024-03-18T22:40:41.811668+0000","last_became_peered":"2024-03-18T22:40:41.811668+0000","last_unstale":"2024-03-18T22:42:12.839536+0000","last_undegraded":"2024-03-18T22:42:12.839536+0000","last_fullsized":"2024-03-18T22:42:12.839536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:17:54.775327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.265213+0000","last_change":"2024-03-18T22:40:45.848466+0000","last_active":"2024-03-18T22:41:44.265213+0000","last_peered":"2024-03-18T22:41:44.265213+0000","last_clean":"2024-03-18T22:41:44.265213+0000","last_became_active":"2024-03-18T22:40:45.848078+0000","last_became_peered":"2024-03-18T22:40:45.848078+0000","last_unstale":"2024-03-18T22:41:44.265213+0000","last_undegraded":"2024-03-18T22:41:44.265213+0000","last_fullsized":"2024-03-18T22:41:44.265213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:28.515368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999779+0000","last_change":"2024-03-18T22:40:41.216434+0000","last_active":"2024-03-18T22:41:43.999779+0000","last_peered":"2024-03-18T22:41:43.999779+0000","last_clean":"2024-03-18T22:41:43.999779+0000","last_became_active":"2024-03-18T22:40:41.216103+0000","last_became_peered":"2024-03-18T22:40:41.216103+0000","last_unstale":"2024-03-18T22:41:43.999779+0000","last_undegraded":"2024-03-18T22:41:43.999779+0000","last_fullsized":"2024-03-18T22:41:43.999779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:39.516463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.854916+0000","last_change":"2024-03-18T22:40:41.808405+0000","last_active":"2024-03-18T22:42:12.854916+0000","last_peered":"2024-03-18T22:42:12.854916+0000","last_clean":"2024-03-18T22:42:12.854916+0000","last_became_active":"2024-03-18T22:40:41.808111+0000","last_became_peered":"2024-03-18T22:40:41.808111+0000","last_unstale":"2024-03-18T22:42:12.854916+0000","last_undegraded":"2024-03-18T22:42:12.854916+0000","last_fullsized":"2024-03-18T22:42:12.854916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:37:20.808163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.002088+0000","last_change":"2024-03-18T22:40:45.848409+0000","last_active":"2024-03-18T22:41:44.002088+0000","last_peered":"2024-03-18T22:41:44.002088+0000","last_clean":"2024-03-18T22:41:44.002088+0000","last_became_active":"2024-03-18T22:40:45.847687+0000","last_became_peered":"2024-03-18T22:40:45.847687+0000","last_unstale":"2024-03-18T22:41:44.002088+0000","last_undegraded":"2024-03-18T22:41:44.002088+0000","last_fullsized":"2024-03-18T22:41:44.002088+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:32.894885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:15.899663+0000","last_change":"2024-03-18T22:40:41.208408+0000","last_active":"2024-03-18T22:42:15.899663+0000","last_peered":"2024-03-18T22:42:15.899663+0000","last_clean":"2024-03-18T22:42:15.899663+0000","last_became_active":"2024-03-18T22:40:41.207976+0000","last_became_peered":"2024-03-18T22:40:41.207976+0000","last_unstale":"2024-03-18T22:42:15.899663+0000","last_undegraded":"2024-03-18T22:42:15.899663+0000","last_fullsized":"2024-03-18T22:42:15.899663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:43:24.416974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.848042+0000","last_change":"2024-03-18T22:40:41.813547+0000","last_active":"2024-03-18T22:42:12.848042+0000","last_peered":"2024-03-18T22:42:12.848042+0000","last_clean":"2024-03-18T22:42:12.848042+0000","last_became_active":"2024-03-18T22:40:41.813250+0000","last_became_peered":"2024-03-18T22:40:41.813250+0000","last_unstale":"2024-03-18T22:42:12.848042+0000","last_undegraded":"2024-03-18T22:42:12.848042+0000","last_fullsized":"2024-03-18T22:42:12.848042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:54.428485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001295+0000","last_change":"2024-03-18T22:40:45.845797+0000","last_active":"2024-03-18T22:41:44.001295+0000","last_peered":"2024-03-18T22:41:44.001295+0000","last_clean":"2024-03-18T22:41:44.001295+0000","last_became_active":"2024-03-18T22:40:45.845324+0000","last_became_peered":"2024-03-18T22:40:45.845324+0000","last_unstale":"2024-03-18T22:41:44.001295+0000","last_undegraded":"2024-03-18T22:41:44.001295+0000","last_fullsized":"2024-03-18T22:41:44.001295+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:21.613203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997100+0000","last_change":"2024-03-18T22:40:41.222127+0000","last_active":"2024-03-18T22:41:43.997100+0000","last_peered":"2024-03-18T22:41:43.997100+0000","last_clean":"2024-03-18T22:41:43.997100+0000","last_became_active":"2024-03-18T22:40:41.221830+0000","last_became_peered":"2024-03-18T22:40:41.221830+0000","last_unstale":"2024-03-18T22:41:43.997100+0000","last_undegraded":"2024-03-18T22:41:43.997100+0000","last_fullsized":"2024-03-18T22:41:43.997100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:24:02.235914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.856654+0000","last_change":"2024-03-18T22:40:41.814606+0000","last_active":"2024-03-18T22:42:12.856654+0000","last_peered":"2024-03-18T22:42:12.856654+0000","last_clean":"2024-03-18T22:42:12.856654+0000","last_became_active":"2024-03-18T22:40:41.814309+0000","last_became_peered":"2024-03-18T22:40:41.814309+0000","last_unstale":"2024-03-18T22:42:12.856654+0000","last_undegraded":"2024-03-18T22:42:12.856654+0000","last_fullsized":"2024-03-18T22:42:12.856654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:33.086229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999592+0000","last_change":"2024-03-18T22:40:45.845711+0000","last_active":"2024-03-18T22:41:43.999592+0000","last_peered":"2024-03-18T22:41:43.999592+0000","last_clean":"2024-03-18T22:41:43.999592+0000","last_became_active":"2024-03-18T22:40:45.845094+0000","last_became_peered":"2024-03-18T22:40:45.845094+0000","last_unstale":"2024-03-18T22:41:43.999592+0000","last_undegraded":"2024-03-18T22:41:43.999592+0000","last_fullsized":"2024-03-18T22:41:43.999592+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:08.372875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997818+0000","last_change":"2024-03-18T22:40:41.219076+0000","last_active":"2024-03-18T22:41:43.997818+0000","last_peered":"2024-03-18T22:41:43.997818+0000","last_clean":"2024-03-18T22:41:43.997818+0000","last_became_active":"2024-03-18T22:40:41.217580+0000","last_became_peered":"2024-03-18T22:40:41.217580+0000","last_unstale":"2024-03-18T22:41:43.997818+0000","last_undegraded":"2024-03-18T22:41:43.997818+0000","last_fullsized":"2024-03-18T22:41:43.997818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:38.112325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.845514+0000","last_change":"2024-03-18T22:40:41.810267+0000","last_active":"2024-03-18T22:42:12.845514+0000","last_peered":"2024-03-18T22:42:12.845514+0000","last_clean":"2024-03-18T22:42:12.845514+0000","last_became_active":"2024-03-18T22:40:41.809960+0000","last_became_peered":"2024-03-18T22:40:41.809960+0000","last_unstale":"2024-03-18T22:42:12.845514+0000","last_undegraded":"2024-03-18T22:42:12.845514+0000","last_fullsized":"2024-03-18T22:42:12.845514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:00.973817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997199+0000","last_change":"2024-03-18T22:40:45.848461+0000","last_active":"2024-03-18T22:41:43.997199+0000","last_peered":"2024-03-18T22:41:43.997199+0000","last_clean":"2024-03-18T22:41:43.997199+0000","last_became_active":"2024-03-18T22:40:45.848043+0000","last_became_peered":"2024-03-18T22:40:45.848043+0000","last_unstale":"2024-03-18T22:41:43.997199+0000","last_undegraded":"2024-03-18T22:41:43.997199+0000","last_fullsized":"2024-03-18T22:41:43.997199+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:38.865565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.002567+0000","last_change":"2024-03-18T22:40:41.213069+0000","last_active":"2024-03-18T22:41:44.002567+0000","last_peered":"2024-03-18T22:41:44.002567+0000","last_clean":"2024-03-18T22:41:44.002567+0000","last_became_active":"2024-03-18T22:40:41.212725+0000","last_became_peered":"2024-03-18T22:40:41.212725+0000","last_unstale":"2024-03-18T22:41:44.002567+0000","last_undegraded":"2024-03-18T22:41:44.002567+0000","last_fullsized":"2024-03-18T22:41:44.002567+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:50.219433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.848639+0000","last_change":"2024-03-18T22:41:44.004772+0000","last_active":"2024-03-18T22:42:12.848639+0000","last_peered":"2024-03-18T22:42:12.848639+0000","last_clean":"2024-03-18T22:42:12.848639+0000","last_became_active":"2024-03-18T22:41:44.004342+0000","last_became_peered":"2024-03-18T22:41:44.004342+0000","last_unstale":"2024-03-18T22:42:12.848639+0000","last_undegraded":"2024-03-18T22:42:12.848639+0000","last_fullsized":"2024-03-18T22:42:12.848639+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:18:38.674911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.000521+0000","last_change":"2024-03-18T22:40:45.840058+0000","last_active":"2024-03-18T22:41:44.000521+0000","last_peered":"2024-03-18T22:41:44.000521+0000","last_clean":"2024-03-18T22:41:44.000521+0000","last_became_active":"2024-03-18T22:40:45.839661+0000","last_became_peered":"2024-03-18T22:40:45.839661+0000","last_unstale":"2024-03-18T22:41:44.000521+0000","last_undegraded":"2024-03-18T22:41:44.000521+0000","last_fullsized":"2024-03-18T22:41:44.000521+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:35:14.323972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'7","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:15.931529+0000","last_change":"2024-03-18T22:40:41.215988+0000","last_active":"2024-03-18T22:42:15.931529+0000","last_peered":"2024-03-18T22:42:15.931529+0000","last_clean":"2024-03-18T22:42:15.931529+0000","last_became_active":"2024-03-18T22:40:41.215680+0000","last_became_peered":"2024-03-18T22:40:41.215680+0000","last_unstale":"2024-03-18T22:42:15.931529+0000","last_undegraded":"2024-03-18T22:42:15.931529+0000","last_fullsized":"2024-03-18T22:42:15.931529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:43.412362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.850700+0000","last_change":"2024-03-18T22:40:41.812521+0000","last_active":"2024-03-18T22:42:12.850700+0000","last_peered":"2024-03-18T22:42:12.850700+0000","last_clean":"2024-03-18T22:42:12.850700+0000","last_became_active":"2024-03-18T22:40:41.812223+0000","last_became_peered":"2024-03-18T22:40:41.812223+0000","last_unstale":"2024-03-18T22:42:12.850700+0000","last_undegraded":"2024-03-18T22:42:12.850700+0000","last_fullsized":"2024-03-18T22:42:12.850700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:35:37.945144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999563+0000","last_change":"2024-03-18T22:40:45.838372+0000","last_active":"2024-03-18T22:41:43.999563+0000","last_peered":"2024-03-18T22:41:43.999563+0000","last_clean":"2024-03-18T22:41:43.999563+0000","last_became_active":"2024-03-18T22:40:45.837962+0000","last_became_peered":"2024-03-18T22:40:45.837962+0000","last_unstale":"2024-03-18T22:41:43.999563+0000","last_undegraded":"2024-03-18T22:41:43.999563+0000","last_fullsized":"2024-03-18T22:41:43.999563+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:18:39.323023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.838523+0000","last_change":"2024-03-18T22:40:41.812146+0000","last_active":"2024-03-18T22:42:12.838523+0000","last_peered":"2024-03-18T22:42:12.838523+0000","last_clean":"2024-03-18T22:42:12.838523+0000","last_became_active":"2024-03-18T22:40:41.811840+0000","last_became_peered":"2024-03-18T22:40:41.811840+0000","last_unstale":"2024-03-18T22:42:12.838523+0000","last_undegraded":"2024-03-18T22:42:12.838523+0000","last_fullsized":"2024-03-18T22:42:12.838523+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:30.829187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:14.897912+0000","last_change":"2024-03-18T22:40:41.212133+0000","last_active":"2024-03-18T22:42:14.897912+0000","last_peered":"2024-03-18T22:42:14.897912+0000","last_clean":"2024-03-18T22:42:14.897912+0000","last_became_active":"2024-03-18T22:40:41.211581+0000","last_became_peered":"2024-03-18T22:40:41.211581+0000","last_unstale":"2024-03-18T22:42:14.897912+0000","last_undegraded":"2024-03-18T22:42:14.897912+0000","last_fullsized":"2024-03-18T22:42:14.897912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:44.287725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.300965+0000","last_change":"2024-03-18T22:40:45.843660+0000","last_active":"2024-03-18T22:41:44.300965+0000","last_peered":"2024-03-18T22:41:44.300965+0000","last_clean":"2024-03-18T22:41:44.300965+0000","last_became_active":"2024-03-18T22:40:45.843233+0000","last_became_peered":"2024-03-18T22:40:45.843233+0000","last_unstale":"2024-03-18T22:41:44.300965+0000","last_undegraded":"2024-03-18T22:41:44.300965+0000","last_fullsized":"2024-03-18T22:41:44.300965+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:08:19.758496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.838172+0000","last_change":"2024-03-18T22:40:41.811798+0000","last_active":"2024-03-18T22:42:12.838172+0000","last_peered":"2024-03-18T22:42:12.838172+0000","last_clean":"2024-03-18T22:42:12.838172+0000","last_became_active":"2024-03-18T22:40:41.811422+0000","last_became_peered":"2024-03-18T22:40:41.811422+0000","last_unstale":"2024-03-18T22:42:12.838172+0000","last_undegraded":"2024-03-18T22:42:12.838172+0000","last_fullsized":"2024-03-18T22:42:12.838172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:53.944556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.998409+0000","last_change":"2024-03-18T22:40:41.210052+0000","last_active":"2024-03-18T22:41:43.998409+0000","last_peered":"2024-03-18T22:41:43.998409+0000","last_clean":"2024-03-18T22:41:43.998409+0000","last_became_active":"2024-03-18T22:40:41.209647+0000","last_became_peered":"2024-03-18T22:40:41.209647+0000","last_unstale":"2024-03-18T22:41:43.998409+0000","last_undegraded":"2024-03-18T22:41:43.998409+0000","last_fullsized":"2024-03-18T22:41:43.998409+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:57:35.187348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.299453+0000","last_change":"2024-03-18T22:39:32.831758+0000","last_active":"2024-03-18T22:41:44.299453+0000","last_peered":"2024-03-18T22:41:44.299453+0000","last_clean":"2024-03-18T22:41:44.299453+0000","last_became_active":"2024-03-18T22:39:32.831287+0000","last_became_peered":"2024-03-18T22:39:32.831287+0000","last_unstale":"2024-03-18T22:41:44.299453+0000","last_undegraded":"2024-03-18T22:41:44.299453+0000","last_fullsized":"2024-03-18T22:41:44.299453+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:36:07.806854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:36:07.806854+0000","last_clean_scrub_stamp":"2024-03-18T22:36:07.806854+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:42.449107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.000291+0000","last_change":"2024-03-18T22:40:45.842008+0000","last_active":"2024-03-18T22:41:44.000291+0000","last_peered":"2024-03-18T22:41:44.000291+0000","last_clean":"2024-03-18T22:41:44.000291+0000","last_became_active":"2024-03-18T22:40:45.841231+0000","last_became_peered":"2024-03-18T22:40:45.841231+0000","last_unstale":"2024-03-18T22:41:44.000291+0000","last_undegraded":"2024-03-18T22:41:44.000291+0000","last_fullsized":"2024-03-18T22:41:44.000291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:06.812049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.836440+0000","last_change":"2024-03-18T22:40:41.807216+0000","last_active":"2024-03-18T22:42:12.836440+0000","last_peered":"2024-03-18T22:42:12.836440+0000","last_clean":"2024-03-18T22:42:12.836440+0000","last_became_active":"2024-03-18T22:40:41.806797+0000","last_became_peered":"2024-03-18T22:40:41.806797+0000","last_unstale":"2024-03-18T22:42:12.836440+0000","last_undegraded":"2024-03-18T22:42:12.836440+0000","last_fullsized":"2024-03-18T22:42:12.836440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:26.422329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.299426+0000","last_change":"2024-03-18T22:40:41.216555+0000","last_active":"2024-03-18T22:41:44.299426+0000","last_peered":"2024-03-18T22:41:44.299426+0000","last_clean":"2024-03-18T22:41:44.299426+0000","last_became_active":"2024-03-18T22:40:41.216240+0000","last_became_peered":"2024-03-18T22:40:41.216240+0000","last_unstale":"2024-03-18T22:41:44.299426+0000","last_undegraded":"2024-03-18T22:41:44.299426+0000","last_fullsized":"2024-03-18T22:41:44.299426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:04.655182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.264909+0000","last_change":"2024-03-18T22:40:45.839889+0000","last_active":"2024-03-18T22:41:44.264909+0000","last_peered":"2024-03-18T22:41:44.264909+0000","last_clean":"2024-03-18T22:41:44.264909+0000","last_became_active":"2024-03-18T22:40:45.839414+0000","last_became_peered":"2024-03-18T22:40:45.839414+0000","last_unstale":"2024-03-18T22:41:44.264909+0000","last_undegraded":"2024-03-18T22:41:44.264909+0000","last_fullsized":"2024-03-18T22:41:44.264909+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:52.497378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997719+0000","last_change":"2024-03-18T22:40:41.221026+0000","last_active":"2024-03-18T22:41:43.997719+0000","last_peered":"2024-03-18T22:41:43.997719+0000","last_clean":"2024-03-18T22:41:43.997719+0000","last_became_active":"2024-03-18T22:40:41.220681+0000","last_became_peered":"2024-03-18T22:40:41.220681+0000","last_unstale":"2024-03-18T22:41:43.997719+0000","last_undegraded":"2024-03-18T22:41:43.997719+0000","last_fullsized":"2024-03-18T22:41:43.997719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:11.490445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.836527+0000","last_change":"2024-03-18T22:40:41.810500+0000","last_active":"2024-03-18T22:42:12.836527+0000","last_peered":"2024-03-18T22:42:12.836527+0000","last_clean":"2024-03-18T22:42:12.836527+0000","last_became_active":"2024-03-18T22:40:41.809723+0000","last_became_peered":"2024-03-18T22:40:41.809723+0000","last_unstale":"2024-03-18T22:42:12.836527+0000","last_undegraded":"2024-03-18T22:42:12.836527+0000","last_fullsized":"2024-03-18T22:42:12.836527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:18:37.053809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999142+0000","last_change":"2024-03-18T22:40:45.844323+0000","last_active":"2024-03-18T22:41:43.999142+0000","last_peered":"2024-03-18T22:41:43.999142+0000","last_clean":"2024-03-18T22:41:43.999142+0000","last_became_active":"2024-03-18T22:40:45.843146+0000","last_became_peered":"2024-03-18T22:40:45.843146+0000","last_unstale":"2024-03-18T22:41:43.999142+0000","last_undegraded":"2024-03-18T22:41:43.999142+0000","last_fullsized":"2024-03-18T22:41:43.999142+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:14:59.881467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'13","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.002718+0000","last_change":"2024-03-18T22:40:41.212126+0000","last_active":"2024-03-18T22:41:44.002718+0000","last_peered":"2024-03-18T22:41:44.002718+0000","last_clean":"2024-03-18T22:41:44.002718+0000","last_became_active":"2024-03-18T22:40:41.211772+0000","last_became_peered":"2024-03-18T22:40:41.211772+0000","last_unstale":"2024-03-18T22:41:44.002718+0000","last_undegraded":"2024-03-18T22:41:44.002718+0000","last_fullsized":"2024-03-18T22:41:44.002718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:03.215751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.844532+0000","last_change":"2024-03-18T22:40:41.820476+0000","last_active":"2024-03-18T22:42:12.844532+0000","last_peered":"2024-03-18T22:42:12.844532+0000","last_clean":"2024-03-18T22:42:12.844532+0000","last_became_active":"2024-03-18T22:40:41.820084+0000","last_became_peered":"2024-03-18T22:40:41.820084+0000","last_unstale":"2024-03-18T22:42:12.844532+0000","last_undegraded":"2024-03-18T22:42:12.844532+0000","last_fullsized":"2024-03-18T22:42:12.844532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:51:50.580208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001221+0000","last_change":"2024-03-18T22:40:45.838588+0000","last_active":"2024-03-18T22:41:44.001221+0000","last_peered":"2024-03-18T22:41:44.001221+0000","last_clean":"2024-03-18T22:41:44.001221+0000","last_became_active":"2024-03-18T22:40:45.838068+0000","last_became_peered":"2024-03-18T22:40:45.838068+0000","last_unstale":"2024-03-18T22:41:44.001221+0000","last_undegraded":"2024-03-18T22:41:44.001221+0000","last_fullsized":"2024-03-18T22:41:44.001221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:11:02.162668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.998852+0000","last_change":"2024-03-18T22:40:41.210108+0000","last_active":"2024-03-18T22:41:43.998852+0000","last_peered":"2024-03-18T22:41:43.998852+0000","last_clean":"2024-03-18T22:41:43.998852+0000","last_became_active":"2024-03-18T22:40:41.209781+0000","last_became_peered":"2024-03-18T22:40:41.209781+0000","last_unstale":"2024-03-18T22:41:43.998852+0000","last_undegraded":"2024-03-18T22:41:43.998852+0000","last_fullsized":"2024-03-18T22:41:43.998852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:10.670822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.845491+0000","last_change":"2024-03-18T22:40:41.814637+0000","last_active":"2024-03-18T22:42:12.845491+0000","last_peered":"2024-03-18T22:42:12.845491+0000","last_clean":"2024-03-18T22:42:12.845491+0000","last_became_active":"2024-03-18T22:40:41.814279+0000","last_became_peered":"2024-03-18T22:40:41.814279+0000","last_unstale":"2024-03-18T22:42:12.845491+0000","last_undegraded":"2024-03-18T22:42:12.845491+0000","last_fullsized":"2024-03-18T22:42:12.845491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:12.496230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997696+0000","last_change":"2024-03-18T22:40:45.851560+0000","last_active":"2024-03-18T22:41:43.997696+0000","last_peered":"2024-03-18T22:41:43.997696+0000","last_clean":"2024-03-18T22:41:43.997696+0000","last_became_active":"2024-03-18T22:40:45.850884+0000","last_became_peered":"2024-03-18T22:40:45.850884+0000","last_unstale":"2024-03-18T22:41:43.997696+0000","last_undegraded":"2024-03-18T22:41:43.997696+0000","last_fullsized":"2024-03-18T22:41:43.997696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:04.114468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.298816+0000","last_change":"2024-03-18T22:40:41.210156+0000","last_active":"2024-03-18T22:41:44.298816+0000","last_peered":"2024-03-18T22:41:44.298816+0000","last_clean":"2024-03-18T22:41:44.298816+0000","last_became_active":"2024-03-18T22:40:41.209855+0000","last_became_peered":"2024-03-18T22:40:41.209855+0000","last_unstale":"2024-03-18T22:41:44.298816+0000","last_undegraded":"2024-03-18T22:41:44.298816+0000","last_fullsized":"2024-03-18T22:41:44.298816+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:02.752573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.803809+0000","last_change":"2024-03-18T22:40:41.803681+0000","last_active":"2024-03-18T22:42:12.803809+0000","last_peered":"2024-03-18T22:42:12.803809+0000","last_clean":"2024-03-18T22:42:12.803809+0000","last_became_active":"2024-03-18T22:40:41.803244+0000","last_became_peered":"2024-03-18T22:40:41.803244+0000","last_unstale":"2024-03-18T22:42:12.803809+0000","last_undegraded":"2024-03-18T22:42:12.803809+0000","last_fullsized":"2024-03-18T22:42:12.803809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:55.944575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.006624+0000","last_change":"2024-03-18T22:41:44.005707+0000","last_active":"2024-03-18T22:41:44.006624+0000","last_peered":"2024-03-18T22:41:44.006624+0000","last_clean":"2024-03-18T22:41:44.006624+0000","last_became_active":"2024-03-18T22:41:44.004338+0000","last_became_peered":"2024-03-18T22:41:44.004338+0000","last_unstale":"2024-03-18T22:41:44.006624+0000","last_undegraded":"2024-03-18T22:41:44.006624+0000","last_fullsized":"2024-03-18T22:41:44.006624+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:46.054087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999722+0000","last_change":"2024-03-18T22:40:41.210830+0000","last_active":"2024-03-18T22:41:43.999722+0000","last_peered":"2024-03-18T22:41:43.999722+0000","last_clean":"2024-03-18T22:41:43.999722+0000","last_became_active":"2024-03-18T22:40:41.210476+0000","last_became_peered":"2024-03-18T22:40:41.210476+0000","last_unstale":"2024-03-18T22:41:43.999722+0000","last_undegraded":"2024-03-18T22:41:43.999722+0000","last_fullsized":"2024-03-18T22:41:43.999722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:13.243788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.842580+0000","last_change":"2024-03-18T22:40:41.807678+0000","last_active":"2024-03-18T22:42:12.842580+0000","last_peered":"2024-03-18T22:42:12.842580+0000","last_clean":"2024-03-18T22:42:12.842580+0000","last_became_active":"2024-03-18T22:40:41.807383+0000","last_became_peered":"2024-03-18T22:40:41.807383+0000","last_unstale":"2024-03-18T22:42:12.842580+0000","last_undegraded":"2024-03-18T22:42:12.842580+0000","last_fullsized":"2024-03-18T22:42:12.842580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:27:55.930293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997772+0000","last_change":"2024-03-18T22:40:45.850982+0000","last_active":"2024-03-18T22:41:43.997772+0000","last_peered":"2024-03-18T22:41:43.997772+0000","last_clean":"2024-03-18T22:41:43.997772+0000","last_became_active":"2024-03-18T22:40:45.850526+0000","last_became_peered":"2024-03-18T22:40:45.850526+0000","last_unstale":"2024-03-18T22:41:43.997772+0000","last_undegraded":"2024-03-18T22:41:43.997772+0000","last_fullsized":"2024-03-18T22:41:43.997772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:05:24.450986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001540+0000","last_change":"2024-03-18T22:40:41.219540+0000","last_active":"2024-03-18T22:41:44.001540+0000","last_peered":"2024-03-18T22:41:44.001540+0000","last_clean":"2024-03-18T22:41:44.001540+0000","last_became_active":"2024-03-18T22:40:41.219206+0000","last_became_peered":"2024-03-18T22:40:41.219206+0000","last_unstale":"2024-03-18T22:41:44.001540+0000","last_undegraded":"2024-03-18T22:41:44.001540+0000","last_fullsized":"2024-03-18T22:41:44.001540+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:33.586368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.852188+0000","last_change":"2024-03-18T22:40:41.815561+0000","last_active":"2024-03-18T22:42:12.852188+0000","last_peered":"2024-03-18T22:42:12.852188+0000","last_clean":"2024-03-18T22:42:12.852188+0000","last_became_active":"2024-03-18T22:40:41.815082+0000","last_became_peered":"2024-03-18T22:40:41.815082+0000","last_unstale":"2024-03-18T22:42:12.852188+0000","last_undegraded":"2024-03-18T22:42:12.852188+0000","last_fullsized":"2024-03-18T22:42:12.852188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:01.676907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997674+0000","last_change":"2024-03-18T22:40:45.845386+0000","last_active":"2024-03-18T22:41:43.997674+0000","last_peered":"2024-03-18T22:41:43.997674+0000","last_clean":"2024-03-18T22:41:43.997674+0000","last_became_active":"2024-03-18T22:40:45.845007+0000","last_became_peered":"2024-03-18T22:40:45.845007+0000","last_unstale":"2024-03-18T22:41:43.997674+0000","last_undegraded":"2024-03-18T22:41:43.997674+0000","last_fullsized":"2024-03-18T22:41:43.997674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:41.429195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997236+0000","last_change":"2024-03-18T22:40:41.215559+0000","last_active":"2024-03-18T22:41:43.997236+0000","last_peered":"2024-03-18T22:41:43.997236+0000","last_clean":"2024-03-18T22:41:43.997236+0000","last_became_active":"2024-03-18T22:40:41.215253+0000","last_became_peered":"2024-03-18T22:40:41.215253+0000","last_unstale":"2024-03-18T22:41:43.997236+0000","last_undegraded":"2024-03-18T22:41:43.997236+0000","last_fullsized":"2024-03-18T22:41:43.997236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:19.706011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.857216+0000","last_change":"2024-03-18T22:40:41.807533+0000","last_active":"2024-03-18T22:42:12.857216+0000","last_peered":"2024-03-18T22:42:12.857216+0000","last_clean":"2024-03-18T22:42:12.857216+0000","last_became_active":"2024-03-18T22:40:41.807219+0000","last_became_peered":"2024-03-18T22:40:41.807219+0000","last_unstale":"2024-03-18T22:42:12.857216+0000","last_undegraded":"2024-03-18T22:42:12.857216+0000","last_fullsized":"2024-03-18T22:42:12.857216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:15:24.214367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.299142+0000","last_change":"2024-03-18T22:40:45.839670+0000","last_active":"2024-03-18T22:41:44.299142+0000","last_peered":"2024-03-18T22:41:44.299142+0000","last_clean":"2024-03-18T22:41:44.299142+0000","last_became_active":"2024-03-18T22:40:45.839263+0000","last_became_peered":"2024-03-18T22:40:45.839263+0000","last_unstale":"2024-03-18T22:41:44.299142+0000","last_undegraded":"2024-03-18T22:41:44.299142+0000","last_fullsized":"2024-03-18T22:41:44.299142+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:48.991959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:14.933774+0000","last_change":"2024-03-18T22:40:41.209617+0000","last_active":"2024-03-18T22:42:14.933774+0000","last_peered":"2024-03-18T22:42:14.933774+0000","last_clean":"2024-03-18T22:42:14.933774+0000","last_became_active":"2024-03-18T22:40:41.209311+0000","last_became_peered":"2024-03-18T22:40:41.209311+0000","last_unstale":"2024-03-18T22:42:14.933774+0000","last_undegraded":"2024-03-18T22:42:14.933774+0000","last_fullsized":"2024-03-18T22:42:14.933774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:44:06.019870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":69,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.858901+0000","last_change":"2024-03-18T22:40:41.808210+0000","last_active":"2024-03-18T22:42:12.858901+0000","last_peered":"2024-03-18T22:42:12.858901+0000","last_clean":"2024-03-18T22:42:12.858901+0000","last_became_active":"2024-03-18T22:40:41.807898+0000","last_became_peered":"2024-03-18T22:40:41.807898+0000","last_unstale":"2024-03-18T22:42:12.858901+0000","last_undegraded":"2024-03-18T22:42:12.858901+0000","last_fullsized":"2024-03-18T22:42:12.858901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:20.401165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.333674+0000","last_change":"2024-03-18T22:40:45.841905+0000","last_active":"2024-03-18T22:41:44.333674+0000","last_peered":"2024-03-18T22:41:44.333674+0000","last_clean":"2024-03-18T22:41:44.333674+0000","last_became_active":"2024-03-18T22:40:45.841464+0000","last_became_peered":"2024-03-18T22:40:45.841464+0000","last_unstale":"2024-03-18T22:41:44.333674+0000","last_undegraded":"2024-03-18T22:41:44.333674+0000","last_fullsized":"2024-03-18T22:41:44.333674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:30.435788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.000167+0000","last_change":"2024-03-18T22:40:41.216919+0000","last_active":"2024-03-18T22:41:44.000167+0000","last_peered":"2024-03-18T22:41:44.000167+0000","last_clean":"2024-03-18T22:41:44.000167+0000","last_became_active":"2024-03-18T22:40:41.216608+0000","last_became_peered":"2024-03-18T22:40:41.216608+0000","last_unstale":"2024-03-18T22:41:44.000167+0000","last_undegraded":"2024-03-18T22:41:44.000167+0000","last_fullsized":"2024-03-18T22:41:44.000167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:10:42.230268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.838803+0000","last_change":"2024-03-18T22:40:41.813879+0000","last_active":"2024-03-18T22:42:12.838803+0000","last_peered":"2024-03-18T22:42:12.838803+0000","last_clean":"2024-03-18T22:42:12.838803+0000","last_became_active":"2024-03-18T22:40:41.813584+0000","last_became_peered":"2024-03-18T22:40:41.813584+0000","last_unstale":"2024-03-18T22:42:12.838803+0000","last_undegraded":"2024-03-18T22:42:12.838803+0000","last_fullsized":"2024-03-18T22:42:12.838803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:20:44.808087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999146+0000","last_change":"2024-03-18T22:40:45.838033+0000","last_active":"2024-03-18T22:41:43.999146+0000","last_peered":"2024-03-18T22:41:43.999146+0000","last_clean":"2024-03-18T22:41:43.999146+0000","last_became_active":"2024-03-18T22:40:45.837530+0000","last_became_peered":"2024-03-18T22:40:45.837530+0000","last_unstale":"2024-03-18T22:41:43.999146+0000","last_undegraded":"2024-03-18T22:41:43.999146+0000","last_fullsized":"2024-03-18T22:41:43.999146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:43:43.444631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001391+0000","last_change":"2024-03-18T22:40:41.215686+0000","last_active":"2024-03-18T22:41:44.001391+0000","last_peered":"2024-03-18T22:41:44.001391+0000","last_clean":"2024-03-18T22:41:44.001391+0000","last_became_active":"2024-03-18T22:40:41.215362+0000","last_became_peered":"2024-03-18T22:40:41.215362+0000","last_unstale":"2024-03-18T22:41:44.001391+0000","last_undegraded":"2024-03-18T22:41:44.001391+0000","last_fullsized":"2024-03-18T22:41:44.001391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:46:30.546304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.853485+0000","last_change":"2024-03-18T22:40:41.808959+0000","last_active":"2024-03-18T22:42:12.853485+0000","last_peered":"2024-03-18T22:42:12.853485+0000","last_clean":"2024-03-18T22:42:12.853485+0000","last_became_active":"2024-03-18T22:40:41.808671+0000","last_became_peered":"2024-03-18T22:40:41.808671+0000","last_unstale":"2024-03-18T22:42:12.853485+0000","last_undegraded":"2024-03-18T22:42:12.853485+0000","last_fullsized":"2024-03-18T22:42:12.853485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:37.115318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.845325+0000","last_change":"2024-03-18T22:40:41.814240+0000","last_active":"2024-03-18T22:42:12.845325+0000","last_peered":"2024-03-18T22:42:12.845325+0000","last_clean":"2024-03-18T22:42:12.845325+0000","last_became_active":"2024-03-18T22:40:41.813925+0000","last_became_peered":"2024-03-18T22:40:41.813925+0000","last_unstale":"2024-03-18T22:42:12.845325+0000","last_undegraded":"2024-03-18T22:42:12.845325+0000","last_fullsized":"2024-03-18T22:42:12.845325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:38.999803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.839477+0000","last_change":"2024-03-18T22:40:41.813381+0000","last_active":"2024-03-18T22:42:12.839477+0000","last_peered":"2024-03-18T22:42:12.839477+0000","last_clean":"2024-03-18T22:42:12.839477+0000","last_became_active":"2024-03-18T22:40:41.812927+0000","last_became_peered":"2024-03-18T22:40:41.812927+0000","last_unstale":"2024-03-18T22:42:12.839477+0000","last_undegraded":"2024-03-18T22:42:12.839477+0000","last_fullsized":"2024-03-18T22:42:12.839477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:23:09.178485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.838241+0000","last_change":"2024-03-18T22:40:41.812513+0000","last_active":"2024-03-18T22:42:12.838241+0000","last_peered":"2024-03-18T22:42:12.838241+0000","last_clean":"2024-03-18T22:42:12.838241+0000","last_became_active":"2024-03-18T22:40:41.811882+0000","last_became_peered":"2024-03-18T22:40:41.811882+0000","last_unstale":"2024-03-18T22:42:12.838241+0000","last_undegraded":"2024-03-18T22:42:12.838241+0000","last_fullsized":"2024-03-18T22:42:12.838241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:45.596911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.845097+0000","last_change":"2024-03-18T22:40:41.814960+0000","last_active":"2024-03-18T22:42:12.845097+0000","last_peered":"2024-03-18T22:42:12.845097+0000","last_clean":"2024-03-18T22:42:12.845097+0000","last_became_active":"2024-03-18T22:40:41.814620+0000","last_became_peered":"2024-03-18T22:40:41.814620+0000","last_unstale":"2024-03-18T22:42:12.845097+0000","last_undegraded":"2024-03-18T22:42:12.845097+0000","last_fullsized":"2024-03-18T22:42:12.845097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:42.205885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.839450+0000","last_change":"2024-03-18T22:41:44.334913+0000","last_active":"2024-03-18T22:42:12.839450+0000","last_peered":"2024-03-18T22:42:12.839450+0000","last_clean":"2024-03-18T22:42:12.839450+0000","last_became_active":"2024-03-18T22:41:44.334554+0000","last_became_peered":"2024-03-18T22:41:44.334554+0000","last_unstale":"2024-03-18T22:42:12.839450+0000","last_undegraded":"2024-03-18T22:42:12.839450+0000","last_fullsized":"2024-03-18T22:42:12.839450+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:00:17.369015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.842272+0000","last_change":"2024-03-18T22:40:41.808718+0000","last_active":"2024-03-18T22:42:12.842272+0000","last_peered":"2024-03-18T22:42:12.842272+0000","last_clean":"2024-03-18T22:42:12.842272+0000","last_became_active":"2024-03-18T22:40:41.808379+0000","last_became_peered":"2024-03-18T22:40:41.808379+0000","last_unstale":"2024-03-18T22:42:12.842272+0000","last_undegraded":"2024-03-18T22:42:12.842272+0000","last_fullsized":"2024-03-18T22:42:12.842272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:26.637531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.847565+0000","last_change":"2024-03-18T22:40:41.818046+0000","last_active":"2024-03-18T22:42:12.847565+0000","last_peered":"2024-03-18T22:42:12.847565+0000","last_clean":"2024-03-18T22:42:12.847565+0000","last_became_active":"2024-03-18T22:40:41.817698+0000","last_became_peered":"2024-03-18T22:40:41.817698+0000","last_unstale":"2024-03-18T22:42:12.847565+0000","last_undegraded":"2024-03-18T22:42:12.847565+0000","last_fullsized":"2024-03-18T22:42:12.847565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:22.967673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.821054+0000","last_change":"2024-03-18T22:40:41.819337+0000","last_active":"2024-03-18T22:42:12.821054+0000","last_peered":"2024-03-18T22:42:12.821054+0000","last_clean":"2024-03-18T22:42:12.821054+0000","last_became_active":"2024-03-18T22:40:41.818987+0000","last_became_peered":"2024-03-18T22:40:41.818987+0000","last_unstale":"2024-03-18T22:42:12.821054+0000","last_undegraded":"2024-03-18T22:42:12.821054+0000","last_fullsized":"2024-03-18T22:42:12.821054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:07:04.529454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.835632+0000","last_change":"2024-03-18T22:40:41.813262+0000","last_active":"2024-03-18T22:42:12.835632+0000","last_peered":"2024-03-18T22:42:12.835632+0000","last_clean":"2024-03-18T22:42:12.835632+0000","last_became_active":"2024-03-18T22:40:41.812956+0000","last_became_peered":"2024-03-18T22:40:41.812956+0000","last_unstale":"2024-03-18T22:42:12.835632+0000","last_undegraded":"2024-03-18T22:42:12.835632+0000","last_fullsized":"2024-03-18T22:42:12.835632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:19.940097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.829350+0000","last_change":"2024-03-18T22:40:41.811897+0000","last_active":"2024-03-18T22:42:12.829350+0000","last_peered":"2024-03-18T22:42:12.829350+0000","last_clean":"2024-03-18T22:42:12.829350+0000","last_became_active":"2024-03-18T22:40:41.811575+0000","last_became_peered":"2024-03-18T22:40:41.811575+0000","last_unstale":"2024-03-18T22:42:12.829350+0000","last_undegraded":"2024-03-18T22:42:12.829350+0000","last_fullsized":"2024-03-18T22:42:12.829350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:45:43.321057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.859376+0000","last_change":"2024-03-18T22:40:41.811757+0000","last_active":"2024-03-18T22:42:12.859376+0000","last_peered":"2024-03-18T22:42:12.859376+0000","last_clean":"2024-03-18T22:42:12.859376+0000","last_became_active":"2024-03-18T22:40:41.811405+0000","last_became_peered":"2024-03-18T22:40:41.811405+0000","last_unstale":"2024-03-18T22:42:12.859376+0000","last_undegraded":"2024-03-18T22:42:12.859376+0000","last_fullsized":"2024-03-18T22:42:12.859376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:39.224604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.839451+0000","last_change":"2024-03-18T22:40:41.809987+0000","last_active":"2024-03-18T22:42:12.839451+0000","last_peered":"2024-03-18T22:42:12.839451+0000","last_clean":"2024-03-18T22:42:12.839451+0000","last_became_active":"2024-03-18T22:40:41.809544+0000","last_became_peered":"2024-03-18T22:40:41.809544+0000","last_unstale":"2024-03-18T22:42:12.839451+0000","last_undegraded":"2024-03-18T22:42:12.839451+0000","last_fullsized":"2024-03-18T22:42:12.839451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:23.800949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.847977+0000","last_change":"2024-03-18T22:40:41.810963+0000","last_active":"2024-03-18T22:42:12.847977+0000","last_peered":"2024-03-18T22:42:12.847977+0000","last_clean":"2024-03-18T22:42:12.847977+0000","last_became_active":"2024-03-18T22:40:41.810641+0000","last_became_peered":"2024-03-18T22:40:41.810641+0000","last_unstale":"2024-03-18T22:42:12.847977+0000","last_undegraded":"2024-03-18T22:42:12.847977+0000","last_fullsized":"2024-03-18T22:42:12.847977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:37:48.872961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71624,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":722,"num_read_kb":440,"num_write":254,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383711,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":501425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645349,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37024,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708128,"statfs":{"total":95995035648,"available":95957123072,"internally_reserved":0,"allocated":1728512,"data_stored":1152379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808874,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":505451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070510,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":501733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332149,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1099435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":497391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1101824,"data_stored":562027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":498315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1122304,"data_stored":562247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":496567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":501465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1658880,"data_stored":1093953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":65890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":65874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T22:42:20.640 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T22:42:20.898 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:42:21.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:20 smithi040 ceph-mon[62643]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:42:21.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:20 smithi113 ceph-mon[66775]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:42:21.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:20 smithi103 ceph-mon[66377]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:42:21.988 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:21 smithi040 ceph-mon[62643]: pgmap v340: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T22:42:22.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:21 smithi113 ceph-mon[66775]: pgmap v340: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T22:42:22.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:21 smithi103 ceph-mon[66377]: pgmap v340: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T22:42:22.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.528+0000 7f654a78d700 1 -- 172.21.15.40:0/1255896498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6544101210 msgr2=0x7f6544101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:22.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.528+0000 7f654a78d700 1 --2- 172.21.15.40:0/1255896498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6544101210 0x7f6544101670 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6538009990 tx=0x7f6538009ca0 comp rx=0 tx=0).stop 2024-03-18T22:42:22.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.528+0000 7f654a78d700 1 -- 172.21.15.40:0/1255896498 shutdown_connections 2024-03-18T22:42:22.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.528+0000 7f654a78d700 1 --2- 172.21.15.40:0/1255896498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6544101bb0 0x7f654410e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:22.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.528+0000 7f654a78d700 1 --2- 172.21.15.40:0/1255896498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6544107250 0x7f6544107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:22.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.528+0000 7f654a78d700 1 --2- 172.21.15.40:0/1255896498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6544101210 0x7f6544101670 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:22.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.528+0000 7f654a78d700 1 -- 172.21.15.40:0/1255896498 >> 172.21.15.40:0/1255896498 conn(0x7f6544078140 msgr2=0x7f6544078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:22.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.528+0000 7f654a78d700 1 -- 172.21.15.40:0/1255896498 shutdown_connections 2024-03-18T22:42:22.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.528+0000 7f654a78d700 1 -- 172.21.15.40:0/1255896498 wait complete. 2024-03-18T22:42:22.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.528+0000 7f654a78d700 1 Processor -- start 2024-03-18T22:42:22.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f654a78d700 1 -- start start 2024-03-18T22:42:22.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f654a78d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6544101210 0x7f65441032a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:22.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f654a78d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6544101bb0 0x7f65441037e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:22.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f654a78d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6544107250 0x7f6544103d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:22.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f654a78d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6544111380 con 0x7f6544101bb0 2024-03-18T22:42:22.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f654a78d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6544111200 con 0x7f6544101210 2024-03-18T22:42:22.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f654a78d700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6544111500 con 0x7f6544107250 2024-03-18T22:42:22.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f65437fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6544101bb0 0x7f65441037e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:22.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f65437fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6544101bb0 0x7f65441037e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32772/0 (socket says 172.21.15.40:32772) 2024-03-18T22:42:22.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f65437fe700 1 -- 172.21.15.40:0/335649066 learned_addr learned my addr 172.21.15.40:0/335649066 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:22.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f6543fff700 1 --2- 172.21.15.40:0/335649066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6544101210 0x7f65441032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:22.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f6548d2a700 1 --2- 172.21.15.40:0/335649066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6544107250 0x7f6544103d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:22.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f65437fe700 1 -- 172.21.15.40:0/335649066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6544107250 msgr2=0x7f6544103d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:22.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f65437fe700 1 --2- 172.21.15.40:0/335649066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6544107250 0x7f6544103d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:22.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f65437fe700 1 -- 172.21.15.40:0/335649066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6544101210 msgr2=0x7f65441032a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:22.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f65437fe700 1 --2- 172.21.15.40:0/335649066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6544101210 0x7f65441032a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:22.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f65437fe700 1 -- 172.21.15.40:0/335649066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65441a6fd0 con 0x7f6544101bb0 2024-03-18T22:42:22.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f6543fff700 1 --2- 172.21.15.40:0/335649066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6544101210 0x7f65441032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:22.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.529+0000 7f65437fe700 1 --2- 172.21.15.40:0/335649066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6544101bb0 0x7f65441037e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f65380057d0 tx=0x7f6538005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:22.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.530+0000 7f65417fa700 1 -- 172.21.15.40:0/335649066 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538026070 con 0x7f6544101bb0 2024-03-18T22:42:22.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.530+0000 7f654a78d700 1 -- 172.21.15.40:0/335649066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65441a7260 con 0x7f6544101bb0 2024-03-18T22:42:22.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.530+0000 7f65417fa700 1 -- 172.21.15.40:0/335649066 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6538021410 con 0x7f6544101bb0 2024-03-18T22:42:22.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.530+0000 7f65417fa700 1 -- 172.21.15.40:0/335649066 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653800b610 con 0x7f6544101bb0 2024-03-18T22:42:22.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.530+0000 7f654a78d700 1 -- 172.21.15.40:0/335649066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65441a7740 con 0x7f6544101bb0 2024-03-18T22:42:22.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.531+0000 7f654a78d700 1 -- 172.21.15.40:0/335649066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6544068fd0 con 0x7f6544101bb0 2024-03-18T22:42:22.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.532+0000 7f65417fa700 1 -- 172.21.15.40:0/335649066 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6538029070 con 0x7f6544101bb0 2024-03-18T22:42:22.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.532+0000 7f65417fa700 1 --2- 172.21.15.40:0/335649066 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f652c0a2890 0x7f652c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:22.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.532+0000 7f6543fff700 1 --2- 172.21.15.40:0/335649066 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f652c0a2890 0x7f652c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:22.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.533+0000 7f65417fa700 1 -- 172.21.15.40:0/335649066 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f65380cfc70 con 0x7f6544101bb0 2024-03-18T22:42:22.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.533+0000 7f6543fff700 1 --2- 172.21.15.40:0/335649066 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f652c0a2890 0x7f652c0a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6534009fd0 tx=0x7f6534009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:22.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.536+0000 7f65417fa700 1 -- 172.21.15.40:0/335649066 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6538030080 con 0x7f6544101bb0 2024-03-18T22:42:22.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.736+0000 7f654a78d700 1 -- 172.21.15.40:0/335649066 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6544069fc0 con 0x7f652c0a2890 2024-03-18T22:42:22.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.746+0000 7f65417fa700 1 -- 172.21.15.40:0/335649066 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340585 (secure 0 0 0) 0x7f6544069fc0 con 0x7f652c0a2890 2024-03-18T22:42:22.751 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:22.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.749+0000 7f654a78d700 1 -- 172.21.15.40:0/335649066 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f652c0a2890 msgr2=0x7f652c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:22.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.749+0000 7f654a78d700 1 --2- 172.21.15.40:0/335649066 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f652c0a2890 0x7f652c0a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6534009fd0 tx=0x7f6534009260 comp rx=0 tx=0).stop 2024-03-18T22:42:22.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.749+0000 7f654a78d700 1 -- 172.21.15.40:0/335649066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6544101bb0 msgr2=0x7f65441037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:22.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.749+0000 7f654a78d700 1 --2- 172.21.15.40:0/335649066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6544101bb0 0x7f65441037e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f65380057d0 tx=0x7f6538005800 comp rx=0 tx=0).stop 2024-03-18T22:42:22.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.750+0000 7f654a78d700 1 -- 172.21.15.40:0/335649066 shutdown_connections 2024-03-18T22:42:22.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.750+0000 7f654a78d700 1 --2- 172.21.15.40:0/335649066 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6544107250 0x7f6544103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:22.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.750+0000 7f654a78d700 1 --2- 172.21.15.40:0/335649066 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f652c0a2890 0x7f652c0a4d50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:22.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.750+0000 7f654a78d700 1 --2- 172.21.15.40:0/335649066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6544101210 0x7f65441032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:22.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.750+0000 7f654a78d700 1 --2- 172.21.15.40:0/335649066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6544101bb0 0x7f65441037e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:22.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.750+0000 7f654a78d700 1 -- 172.21.15.40:0/335649066 >> 172.21.15.40:0/335649066 conn(0x7f6544078140 msgr2=0x7f6544105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:22.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.750+0000 7f654a78d700 1 -- 172.21.15.40:0/335649066 shutdown_connections 2024-03-18T22:42:22.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:22.750+0000 7f654a78d700 1 -- 172.21.15.40:0/335649066 wait complete. 2024-03-18T22:42:22.754 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-18T22:42:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:22 smithi113 ceph-mon[66775]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:42:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:22 smithi103 ceph-mon[66377]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:42:23.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:22 smithi040 ceph-mon[62643]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:42:23.520 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-03-18T22:42:20.832452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662226,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":873,"num_read_kb":564,"num_write":437,"num_write_kb":3091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":861,"ondisk_log_size":861,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384192,"kb_used_data":14240,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557632,"statfs":{"total":1151940427776,"available":1151547015168,"internally_reserved":0,"allocated":14581760,"data_stored":7972388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":129,"num_read_kb":69,"num_write":11,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003660"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.861943+0000","last_change":"2024-03-18T22:40:41.806379+0000","last_active":"2024-03-18T22:42:12.861943+0000","last_peered":"2024-03-18T22:42:12.861943+0000","last_clean":"2024-03-18T22:42:12.861943+0000","last_became_active":"2024-03-18T22:40:41.806058+0000","last_became_peered":"2024-03-18T22:40:41.806058+0000","last_unstale":"2024-03-18T22:42:12.861943+0000","last_undegraded":"2024-03-18T22:42:12.861943+0000","last_fullsized":"2024-03-18T22:42:12.861943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:45.834396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.851643+0000","last_change":"2024-03-18T22:40:41.813722+0000","last_active":"2024-03-18T22:42:12.851643+0000","last_peered":"2024-03-18T22:42:12.851643+0000","last_clean":"2024-03-18T22:42:12.851643+0000","last_became_active":"2024-03-18T22:40:41.813284+0000","last_became_peered":"2024-03-18T22:40:41.813284+0000","last_unstale":"2024-03-18T22:42:12.851643+0000","last_undegraded":"2024-03-18T22:42:12.851643+0000","last_fullsized":"2024-03-18T22:42:12.851643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:46.374117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.857256+0000","last_change":"2024-03-18T22:40:41.808340+0000","last_active":"2024-03-18T22:42:12.857256+0000","last_peered":"2024-03-18T22:42:12.857256+0000","last_clean":"2024-03-18T22:42:12.857256+0000","last_became_active":"2024-03-18T22:40:41.808002+0000","last_became_peered":"2024-03-18T22:40:41.808002+0000","last_unstale":"2024-03-18T22:42:12.857256+0000","last_undegraded":"2024-03-18T22:42:12.857256+0000","last_fullsized":"2024-03-18T22:42:12.857256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:40:32.122124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.832643+0000","last_change":"2024-03-18T22:40:41.809959+0000","last_active":"2024-03-18T22:42:12.832643+0000","last_peered":"2024-03-18T22:42:12.832643+0000","last_clean":"2024-03-18T22:42:12.832643+0000","last_became_active":"2024-03-18T22:40:41.809676+0000","last_became_peered":"2024-03-18T22:40:41.809676+0000","last_unstale":"2024-03-18T22:42:12.832643+0000","last_undegraded":"2024-03-18T22:42:12.832643+0000","last_fullsized":"2024-03-18T22:42:12.832643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:29:45.263011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.825175+0000","last_change":"2024-03-18T22:40:41.804583+0000","last_active":"2024-03-18T22:42:12.825175+0000","last_peered":"2024-03-18T22:42:12.825175+0000","last_clean":"2024-03-18T22:42:12.825175+0000","last_became_active":"2024-03-18T22:40:41.804064+0000","last_became_peered":"2024-03-18T22:40:41.804064+0000","last_unstale":"2024-03-18T22:42:12.825175+0000","last_undegraded":"2024-03-18T22:42:12.825175+0000","last_fullsized":"2024-03-18T22:42:12.825175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:46:23.455106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.855509+0000","last_change":"2024-03-18T22:41:44.301195+0000","last_active":"2024-03-18T22:42:12.855509+0000","last_peered":"2024-03-18T22:42:12.855509+0000","last_clean":"2024-03-18T22:42:12.855509+0000","last_became_active":"2024-03-18T22:41:44.300801+0000","last_became_peered":"2024-03-18T22:41:44.300801+0000","last_unstale":"2024-03-18T22:42:12.855509+0000","last_undegraded":"2024-03-18T22:42:12.855509+0000","last_fullsized":"2024-03-18T22:42:12.855509+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:12:55.579977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.832006+0000","last_change":"2024-03-18T22:40:41.814971+0000","last_active":"2024-03-18T22:42:12.832006+0000","last_peered":"2024-03-18T22:42:12.832006+0000","last_clean":"2024-03-18T22:42:12.832006+0000","last_became_active":"2024-03-18T22:40:41.814305+0000","last_became_peered":"2024-03-18T22:40:41.814305+0000","last_unstale":"2024-03-18T22:42:12.832006+0000","last_undegraded":"2024-03-18T22:42:12.832006+0000","last_fullsized":"2024-03-18T22:42:12.832006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:07.390157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.851956+0000","last_change":"2024-03-18T22:40:41.815674+0000","last_active":"2024-03-18T22:42:12.851956+0000","last_peered":"2024-03-18T22:42:12.851956+0000","last_clean":"2024-03-18T22:42:12.851956+0000","last_became_active":"2024-03-18T22:40:41.813560+0000","last_became_peered":"2024-03-18T22:40:41.813560+0000","last_unstale":"2024-03-18T22:42:12.851956+0000","last_undegraded":"2024-03-18T22:42:12.851956+0000","last_fullsized":"2024-03-18T22:42:12.851956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:42:32.409338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.862612+0000","last_change":"2024-03-18T22:40:41.817508+0000","last_active":"2024-03-18T22:42:12.862612+0000","last_peered":"2024-03-18T22:42:12.862612+0000","last_clean":"2024-03-18T22:42:12.862612+0000","last_became_active":"2024-03-18T22:40:41.817184+0000","last_became_peered":"2024-03-18T22:40:41.817184+0000","last_unstale":"2024-03-18T22:42:12.862612+0000","last_undegraded":"2024-03-18T22:42:12.862612+0000","last_fullsized":"2024-03-18T22:42:12.862612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:10.255238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.848287+0000","last_change":"2024-03-18T22:40:41.806235+0000","last_active":"2024-03-18T22:42:12.848287+0000","last_peered":"2024-03-18T22:42:12.848287+0000","last_clean":"2024-03-18T22:42:12.848287+0000","last_became_active":"2024-03-18T22:40:41.805913+0000","last_became_peered":"2024-03-18T22:40:41.805913+0000","last_unstale":"2024-03-18T22:42:12.848287+0000","last_undegraded":"2024-03-18T22:42:12.848287+0000","last_fullsized":"2024-03-18T22:42:12.848287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:39:06.299989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.856751+0000","last_change":"2024-03-18T22:40:41.814735+0000","last_active":"2024-03-18T22:42:12.856751+0000","last_peered":"2024-03-18T22:42:12.856751+0000","last_clean":"2024-03-18T22:42:12.856751+0000","last_became_active":"2024-03-18T22:40:41.814291+0000","last_became_peered":"2024-03-18T22:40:41.814291+0000","last_unstale":"2024-03-18T22:42:12.856751+0000","last_undegraded":"2024-03-18T22:42:12.856751+0000","last_fullsized":"2024-03-18T22:42:12.856751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:49.610674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.851736+0000","last_change":"2024-03-18T22:40:41.815561+0000","last_active":"2024-03-18T22:42:12.851736+0000","last_peered":"2024-03-18T22:42:12.851736+0000","last_clean":"2024-03-18T22:42:12.851736+0000","last_became_active":"2024-03-18T22:40:41.815252+0000","last_became_peered":"2024-03-18T22:40:41.815252+0000","last_unstale":"2024-03-18T22:42:12.851736+0000","last_undegraded":"2024-03-18T22:42:12.851736+0000","last_fullsized":"2024-03-18T22:42:12.851736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:34.769168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.848344+0000","last_change":"2024-03-18T22:40:41.803325+0000","last_active":"2024-03-18T22:42:12.848344+0000","last_peered":"2024-03-18T22:42:12.848344+0000","last_clean":"2024-03-18T22:42:12.848344+0000","last_became_active":"2024-03-18T22:40:41.803008+0000","last_became_peered":"2024-03-18T22:40:41.803008+0000","last_unstale":"2024-03-18T22:42:12.848344+0000","last_undegraded":"2024-03-18T22:42:12.848344+0000","last_fullsized":"2024-03-18T22:42:12.848344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:14:01.335548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.851770+0000","last_change":"2024-03-18T22:40:41.812541+0000","last_active":"2024-03-18T22:42:12.851770+0000","last_peered":"2024-03-18T22:42:12.851770+0000","last_clean":"2024-03-18T22:42:12.851770+0000","last_became_active":"2024-03-18T22:40:41.812153+0000","last_became_peered":"2024-03-18T22:40:41.812153+0000","last_unstale":"2024-03-18T22:42:12.851770+0000","last_undegraded":"2024-03-18T22:42:12.851770+0000","last_fullsized":"2024-03-18T22:42:12.851770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:04.927057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.845234+0000","last_change":"2024-03-18T22:40:41.815790+0000","last_active":"2024-03-18T22:42:12.845234+0000","last_peered":"2024-03-18T22:42:12.845234+0000","last_clean":"2024-03-18T22:42:12.845234+0000","last_became_active":"2024-03-18T22:40:41.814949+0000","last_became_peered":"2024-03-18T22:40:41.814949+0000","last_unstale":"2024-03-18T22:42:12.845234+0000","last_undegraded":"2024-03-18T22:42:12.845234+0000","last_fullsized":"2024-03-18T22:42:12.845234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:10:15.874285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.858727+0000","last_change":"2024-03-18T22:40:41.809980+0000","last_active":"2024-03-18T22:42:12.858727+0000","last_peered":"2024-03-18T22:42:12.858727+0000","last_clean":"2024-03-18T22:42:12.858727+0000","last_became_active":"2024-03-18T22:40:41.809640+0000","last_became_peered":"2024-03-18T22:40:41.809640+0000","last_unstale":"2024-03-18T22:42:12.858727+0000","last_undegraded":"2024-03-18T22:42:12.858727+0000","last_fullsized":"2024-03-18T22:42:12.858727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:25.389027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.862116+0000","last_change":"2024-03-18T22:40:41.807088+0000","last_active":"2024-03-18T22:42:12.862116+0000","last_peered":"2024-03-18T22:42:12.862116+0000","last_clean":"2024-03-18T22:42:12.862116+0000","last_became_active":"2024-03-18T22:40:41.806770+0000","last_became_peered":"2024-03-18T22:40:41.806770+0000","last_unstale":"2024-03-18T22:42:12.862116+0000","last_undegraded":"2024-03-18T22:42:12.862116+0000","last_fullsized":"2024-03-18T22:42:12.862116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:53:14.001247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.851434+0000","last_change":"2024-03-18T22:40:41.812685+0000","last_active":"2024-03-18T22:42:12.851434+0000","last_peered":"2024-03-18T22:42:12.851434+0000","last_clean":"2024-03-18T22:42:12.851434+0000","last_became_active":"2024-03-18T22:40:41.812346+0000","last_became_peered":"2024-03-18T22:40:41.812346+0000","last_unstale":"2024-03-18T22:42:12.851434+0000","last_undegraded":"2024-03-18T22:42:12.851434+0000","last_fullsized":"2024-03-18T22:42:12.851434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:25.033460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.827948+0000","last_change":"2024-03-18T22:40:41.812627+0000","last_active":"2024-03-18T22:42:12.827948+0000","last_peered":"2024-03-18T22:42:12.827948+0000","last_clean":"2024-03-18T22:42:12.827948+0000","last_became_active":"2024-03-18T22:40:41.812327+0000","last_became_peered":"2024-03-18T22:40:41.812327+0000","last_unstale":"2024-03-18T22:42:12.827948+0000","last_undegraded":"2024-03-18T22:42:12.827948+0000","last_fullsized":"2024-03-18T22:42:12.827948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:58:22.088589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001429+0000","last_change":"2024-03-18T22:40:45.847822+0000","last_active":"2024-03-18T22:41:44.001429+0000","last_peered":"2024-03-18T22:41:44.001429+0000","last_clean":"2024-03-18T22:41:44.001429+0000","last_became_active":"2024-03-18T22:40:45.847407+0000","last_became_peered":"2024-03-18T22:40:45.847407+0000","last_unstale":"2024-03-18T22:41:44.001429+0000","last_undegraded":"2024-03-18T22:41:44.001429+0000","last_fullsized":"2024-03-18T22:41:44.001429+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:44.825997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'14","reported_seq":111,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:16.932250+0000","last_change":"2024-03-18T22:40:41.218797+0000","last_active":"2024-03-18T22:42:16.932250+0000","last_peered":"2024-03-18T22:42:16.932250+0000","last_clean":"2024-03-18T22:42:16.932250+0000","last_became_active":"2024-03-18T22:40:41.218472+0000","last_became_peered":"2024-03-18T22:40:41.218472+0000","last_unstale":"2024-03-18T22:42:16.932250+0000","last_undegraded":"2024-03-18T22:42:16.932250+0000","last_fullsized":"2024-03-18T22:42:16.932250+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:10.230257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.831788+0000","last_change":"2024-03-18T22:41:44.005444+0000","last_active":"2024-03-18T22:42:12.831788+0000","last_peered":"2024-03-18T22:42:12.831788+0000","last_clean":"2024-03-18T22:42:12.831788+0000","last_became_active":"2024-03-18T22:41:44.005157+0000","last_became_peered":"2024-03-18T22:41:44.005157+0000","last_unstale":"2024-03-18T22:42:12.831788+0000","last_undegraded":"2024-03-18T22:42:12.831788+0000","last_fullsized":"2024-03-18T22:42:12.831788+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:53:16.249875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997343+0000","last_change":"2024-03-18T22:40:45.851126+0000","last_active":"2024-03-18T22:41:43.997343+0000","last_peered":"2024-03-18T22:41:43.997343+0000","last_clean":"2024-03-18T22:41:43.997343+0000","last_became_active":"2024-03-18T22:40:45.850549+0000","last_became_peered":"2024-03-18T22:40:45.850549+0000","last_unstale":"2024-03-18T22:41:43.997343+0000","last_undegraded":"2024-03-18T22:41:43.997343+0000","last_fullsized":"2024-03-18T22:41:43.997343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:39.359263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999819+0000","last_change":"2024-03-18T22:40:41.220944+0000","last_active":"2024-03-18T22:41:43.999819+0000","last_peered":"2024-03-18T22:41:43.999819+0000","last_clean":"2024-03-18T22:41:43.999819+0000","last_became_active":"2024-03-18T22:40:41.220612+0000","last_became_peered":"2024-03-18T22:40:41.220612+0000","last_unstale":"2024-03-18T22:41:43.999819+0000","last_undegraded":"2024-03-18T22:41:43.999819+0000","last_fullsized":"2024-03-18T22:41:43.999819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:16.083134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.857160+0000","last_change":"2024-03-18T22:40:41.818031+0000","last_active":"2024-03-18T22:42:12.857160+0000","last_peered":"2024-03-18T22:42:12.857160+0000","last_clean":"2024-03-18T22:42:12.857160+0000","last_became_active":"2024-03-18T22:40:41.817693+0000","last_became_peered":"2024-03-18T22:40:41.817693+0000","last_unstale":"2024-03-18T22:42:12.857160+0000","last_undegraded":"2024-03-18T22:42:12.857160+0000","last_fullsized":"2024-03-18T22:42:12.857160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:05.901203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.998307+0000","last_change":"2024-03-18T22:40:45.850749+0000","last_active":"2024-03-18T22:41:43.998307+0000","last_peered":"2024-03-18T22:41:43.998307+0000","last_clean":"2024-03-18T22:41:43.998307+0000","last_became_active":"2024-03-18T22:40:45.850318+0000","last_became_peered":"2024-03-18T22:40:45.850318+0000","last_unstale":"2024-03-18T22:41:43.998307+0000","last_undegraded":"2024-03-18T22:41:43.998307+0000","last_fullsized":"2024-03-18T22:41:43.998307+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:56.047413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997214+0000","last_change":"2024-03-18T22:40:41.220775+0000","last_active":"2024-03-18T22:41:43.997214+0000","last_peered":"2024-03-18T22:41:43.997214+0000","last_clean":"2024-03-18T22:41:43.997214+0000","last_became_active":"2024-03-18T22:40:41.220313+0000","last_became_peered":"2024-03-18T22:40:41.220313+0000","last_unstale":"2024-03-18T22:41:43.997214+0000","last_undegraded":"2024-03-18T22:41:43.997214+0000","last_fullsized":"2024-03-18T22:41:43.997214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:24.180884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.858766+0000","last_change":"2024-03-18T22:40:41.818448+0000","last_active":"2024-03-18T22:42:12.858766+0000","last_peered":"2024-03-18T22:42:12.858766+0000","last_clean":"2024-03-18T22:42:12.858766+0000","last_became_active":"2024-03-18T22:40:41.818088+0000","last_became_peered":"2024-03-18T22:40:41.818088+0000","last_unstale":"2024-03-18T22:42:12.858766+0000","last_undegraded":"2024-03-18T22:42:12.858766+0000","last_fullsized":"2024-03-18T22:42:12.858766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:34:54.298226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997546+0000","last_change":"2024-03-18T22:40:45.846250+0000","last_active":"2024-03-18T22:41:43.997546+0000","last_peered":"2024-03-18T22:41:43.997546+0000","last_clean":"2024-03-18T22:41:43.997546+0000","last_became_active":"2024-03-18T22:40:45.845859+0000","last_became_peered":"2024-03-18T22:40:45.845859+0000","last_unstale":"2024-03-18T22:41:43.997546+0000","last_undegraded":"2024-03-18T22:41:43.997546+0000","last_fullsized":"2024-03-18T22:41:43.997546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:57.855301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997708+0000","last_change":"2024-03-18T22:40:41.216569+0000","last_active":"2024-03-18T22:41:43.997708+0000","last_peered":"2024-03-18T22:41:43.997708+0000","last_clean":"2024-03-18T22:41:43.997708+0000","last_became_active":"2024-03-18T22:40:41.216255+0000","last_became_peered":"2024-03-18T22:40:41.216255+0000","last_unstale":"2024-03-18T22:41:43.997708+0000","last_undegraded":"2024-03-18T22:41:43.997708+0000","last_fullsized":"2024-03-18T22:41:43.997708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:23.016346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.859296+0000","last_change":"2024-03-18T22:41:44.301578+0000","last_active":"2024-03-18T22:42:12.859296+0000","last_peered":"2024-03-18T22:42:12.859296+0000","last_clean":"2024-03-18T22:42:12.859296+0000","last_became_active":"2024-03-18T22:41:44.301178+0000","last_became_peered":"2024-03-18T22:41:44.301178+0000","last_unstale":"2024-03-18T22:42:12.859296+0000","last_undegraded":"2024-03-18T22:42:12.859296+0000","last_fullsized":"2024-03-18T22:42:12.859296+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:25:50.635812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.302390+0000","last_change":"2024-03-18T22:41:44.302390+0000","last_active":"2024-03-18T22:41:44.302390+0000","last_peered":"2024-03-18T22:41:44.302390+0000","last_clean":"2024-03-18T22:41:44.302390+0000","last_became_active":"2024-03-18T22:41:44.301920+0000","last_became_peered":"2024-03-18T22:41:44.301920+0000","last_unstale":"2024-03-18T22:41:44.302390+0000","last_undegraded":"2024-03-18T22:41:44.302390+0000","last_fullsized":"2024-03-18T22:41:44.302390+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:56:12.561520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,7,11],"acting":[2,8,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":111,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:18.896241+0000","last_change":"2024-03-18T22:40:41.210196+0000","last_active":"2024-03-18T22:42:18.896241+0000","last_peered":"2024-03-18T22:42:18.896241+0000","last_clean":"2024-03-18T22:42:18.896241+0000","last_became_active":"2024-03-18T22:40:41.209888+0000","last_became_peered":"2024-03-18T22:40:41.209888+0000","last_unstale":"2024-03-18T22:42:18.896241+0000","last_undegraded":"2024-03-18T22:42:18.896241+0000","last_fullsized":"2024-03-18T22:42:18.896241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:06:31.134688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.835531+0000","last_change":"2024-03-18T22:41:44.004814+0000","last_active":"2024-03-18T22:42:12.835531+0000","last_peered":"2024-03-18T22:42:12.835531+0000","last_clean":"2024-03-18T22:42:12.835531+0000","last_became_active":"2024-03-18T22:41:44.004219+0000","last_became_peered":"2024-03-18T22:41:44.004219+0000","last_unstale":"2024-03-18T22:42:12.835531+0000","last_undegraded":"2024-03-18T22:42:12.835531+0000","last_fullsized":"2024-03-18T22:42:12.835531+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:29:03.746508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.005790+0000","last_change":"2024-03-18T22:41:44.005790+0000","last_active":"2024-03-18T22:41:44.005790+0000","last_peered":"2024-03-18T22:41:44.005790+0000","last_clean":"2024-03-18T22:41:44.005790+0000","last_became_active":"2024-03-18T22:41:44.005386+0000","last_became_peered":"2024-03-18T22:41:44.005386+0000","last_unstale":"2024-03-18T22:41:44.005790+0000","last_undegraded":"2024-03-18T22:41:44.005790+0000","last_fullsized":"2024-03-18T22:41:44.005790+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:49:08.212639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.000320+0000","last_change":"2024-03-18T22:40:41.217449+0000","last_active":"2024-03-18T22:41:44.000320+0000","last_peered":"2024-03-18T22:41:44.000320+0000","last_clean":"2024-03-18T22:41:44.000320+0000","last_became_active":"2024-03-18T22:40:41.217127+0000","last_became_peered":"2024-03-18T22:40:41.217127+0000","last_unstale":"2024-03-18T22:41:44.000320+0000","last_undegraded":"2024-03-18T22:41:44.000320+0000","last_fullsized":"2024-03-18T22:41:44.000320+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:08:00.178720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.838151+0000","last_change":"2024-03-18T22:40:41.814154+0000","last_active":"2024-03-18T22:42:12.838151+0000","last_peered":"2024-03-18T22:42:12.838151+0000","last_clean":"2024-03-18T22:42:12.838151+0000","last_became_active":"2024-03-18T22:40:41.813824+0000","last_became_peered":"2024-03-18T22:40:41.813824+0000","last_unstale":"2024-03-18T22:42:12.838151+0000","last_undegraded":"2024-03-18T22:42:12.838151+0000","last_fullsized":"2024-03-18T22:42:12.838151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:12.412689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.265037+0000","last_change":"2024-03-18T22:40:45.852711+0000","last_active":"2024-03-18T22:41:44.265037+0000","last_peered":"2024-03-18T22:41:44.265037+0000","last_clean":"2024-03-18T22:41:44.265037+0000","last_became_active":"2024-03-18T22:40:45.852247+0000","last_became_peered":"2024-03-18T22:40:45.852247+0000","last_unstale":"2024-03-18T22:41:44.265037+0000","last_undegraded":"2024-03-18T22:41:44.265037+0000","last_fullsized":"2024-03-18T22:41:44.265037+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:03.061689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001812+0000","last_change":"2024-03-18T22:40:41.212399+0000","last_active":"2024-03-18T22:41:44.001812+0000","last_peered":"2024-03-18T22:41:44.001812+0000","last_clean":"2024-03-18T22:41:44.001812+0000","last_became_active":"2024-03-18T22:40:41.207416+0000","last_became_peered":"2024-03-18T22:40:41.207416+0000","last_unstale":"2024-03-18T22:41:44.001812+0000","last_undegraded":"2024-03-18T22:41:44.001812+0000","last_fullsized":"2024-03-18T22:41:44.001812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:11.406342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.831198+0000","last_change":"2024-03-18T22:40:41.807313+0000","last_active":"2024-03-18T22:42:12.831198+0000","last_peered":"2024-03-18T22:42:12.831198+0000","last_clean":"2024-03-18T22:42:12.831198+0000","last_became_active":"2024-03-18T22:40:41.806029+0000","last_became_peered":"2024-03-18T22:40:41.806029+0000","last_unstale":"2024-03-18T22:42:12.831198+0000","last_undegraded":"2024-03-18T22:42:12.831198+0000","last_fullsized":"2024-03-18T22:42:12.831198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:23:43.973443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.333855+0000","last_change":"2024-03-18T22:40:45.835479+0000","last_active":"2024-03-18T22:41:44.333855+0000","last_peered":"2024-03-18T22:41:44.333855+0000","last_clean":"2024-03-18T22:41:44.333855+0000","last_became_active":"2024-03-18T22:40:45.834994+0000","last_became_peered":"2024-03-18T22:40:45.834994+0000","last_unstale":"2024-03-18T22:41:44.333855+0000","last_undegraded":"2024-03-18T22:41:44.333855+0000","last_fullsized":"2024-03-18T22:41:44.333855+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:31.317843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999237+0000","last_change":"2024-03-18T22:40:41.212110+0000","last_active":"2024-03-18T22:41:43.999237+0000","last_peered":"2024-03-18T22:41:43.999237+0000","last_clean":"2024-03-18T22:41:43.999237+0000","last_became_active":"2024-03-18T22:40:41.211803+0000","last_became_peered":"2024-03-18T22:40:41.211803+0000","last_unstale":"2024-03-18T22:41:43.999237+0000","last_undegraded":"2024-03-18T22:41:43.999237+0000","last_fullsized":"2024-03-18T22:41:43.999237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:40.482983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.842272+0000","last_change":"2024-03-18T22:40:41.812585+0000","last_active":"2024-03-18T22:42:12.842272+0000","last_peered":"2024-03-18T22:42:12.842272+0000","last_clean":"2024-03-18T22:42:12.842272+0000","last_became_active":"2024-03-18T22:40:41.812260+0000","last_became_peered":"2024-03-18T22:40:41.812260+0000","last_unstale":"2024-03-18T22:42:12.842272+0000","last_undegraded":"2024-03-18T22:42:12.842272+0000","last_fullsized":"2024-03-18T22:42:12.842272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:12:59.999503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999606+0000","last_change":"2024-03-18T22:40:45.842252+0000","last_active":"2024-03-18T22:41:43.999606+0000","last_peered":"2024-03-18T22:41:43.999606+0000","last_clean":"2024-03-18T22:41:43.999606+0000","last_became_active":"2024-03-18T22:40:45.841849+0000","last_became_peered":"2024-03-18T22:40:45.841849+0000","last_unstale":"2024-03-18T22:41:43.999606+0000","last_undegraded":"2024-03-18T22:41:43.999606+0000","last_fullsized":"2024-03-18T22:41:43.999606+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:21.253947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":174,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:18.895904+0000","last_change":"2024-03-18T22:40:41.214253+0000","last_active":"2024-03-18T22:42:18.895904+0000","last_peered":"2024-03-18T22:42:18.895904+0000","last_clean":"2024-03-18T22:42:18.895904+0000","last_became_active":"2024-03-18T22:40:41.213911+0000","last_became_peered":"2024-03-18T22:40:41.213911+0000","last_unstale":"2024-03-18T22:42:18.895904+0000","last_undegraded":"2024-03-18T22:42:18.895904+0000","last_fullsized":"2024-03-18T22:42:18.895904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:19.185714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":142,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.858816+0000","last_change":"2024-03-18T22:40:41.813281+0000","last_active":"2024-03-18T22:42:12.858816+0000","last_peered":"2024-03-18T22:42:12.858816+0000","last_clean":"2024-03-18T22:42:12.858816+0000","last_became_active":"2024-03-18T22:40:41.812909+0000","last_became_peered":"2024-03-18T22:40:41.812909+0000","last_unstale":"2024-03-18T22:42:12.858816+0000","last_undegraded":"2024-03-18T22:42:12.858816+0000","last_fullsized":"2024-03-18T22:42:12.858816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:17:26.619441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001859+0000","last_change":"2024-03-18T22:40:45.846956+0000","last_active":"2024-03-18T22:41:44.001859+0000","last_peered":"2024-03-18T22:41:44.001859+0000","last_clean":"2024-03-18T22:41:44.001859+0000","last_became_active":"2024-03-18T22:40:45.843061+0000","last_became_peered":"2024-03-18T22:40:45.843061+0000","last_unstale":"2024-03-18T22:41:44.001859+0000","last_undegraded":"2024-03-18T22:41:44.001859+0000","last_fullsized":"2024-03-18T22:41:44.001859+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:52.043424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.300418+0000","last_change":"2024-03-18T22:40:41.214933+0000","last_active":"2024-03-18T22:41:44.300418+0000","last_peered":"2024-03-18T22:41:44.300418+0000","last_clean":"2024-03-18T22:41:44.300418+0000","last_became_active":"2024-03-18T22:40:41.214606+0000","last_became_peered":"2024-03-18T22:40:41.214606+0000","last_unstale":"2024-03-18T22:41:44.300418+0000","last_undegraded":"2024-03-18T22:41:44.300418+0000","last_fullsized":"2024-03-18T22:41:44.300418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:08:06.230288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.845541+0000","last_change":"2024-03-18T22:40:41.817590+0000","last_active":"2024-03-18T22:42:12.845541+0000","last_peered":"2024-03-18T22:42:12.845541+0000","last_clean":"2024-03-18T22:42:12.845541+0000","last_became_active":"2024-03-18T22:40:41.815809+0000","last_became_peered":"2024-03-18T22:40:41.815809+0000","last_unstale":"2024-03-18T22:42:12.845541+0000","last_undegraded":"2024-03-18T22:42:12.845541+0000","last_fullsized":"2024-03-18T22:42:12.845541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:44:54.043647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.998518+0000","last_change":"2024-03-18T22:40:45.844767+0000","last_active":"2024-03-18T22:41:43.998518+0000","last_peered":"2024-03-18T22:41:43.998518+0000","last_clean":"2024-03-18T22:41:43.998518+0000","last_became_active":"2024-03-18T22:40:45.844251+0000","last_became_peered":"2024-03-18T22:40:45.844251+0000","last_unstale":"2024-03-18T22:41:43.998518+0000","last_undegraded":"2024-03-18T22:41:43.998518+0000","last_fullsized":"2024-03-18T22:41:43.998518+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:43.756959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.000056+0000","last_change":"2024-03-18T22:40:41.214310+0000","last_active":"2024-03-18T22:41:44.000056+0000","last_peered":"2024-03-18T22:41:44.000056+0000","last_clean":"2024-03-18T22:41:44.000056+0000","last_became_active":"2024-03-18T22:40:41.213963+0000","last_became_peered":"2024-03-18T22:40:41.213963+0000","last_unstale":"2024-03-18T22:41:44.000056+0000","last_undegraded":"2024-03-18T22:41:44.000056+0000","last_fullsized":"2024-03-18T22:41:44.000056+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:03.401220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.860732+0000","last_change":"2024-03-18T22:40:41.810876+0000","last_active":"2024-03-18T22:42:12.860732+0000","last_peered":"2024-03-18T22:42:12.860732+0000","last_clean":"2024-03-18T22:42:12.860732+0000","last_became_active":"2024-03-18T22:40:41.810104+0000","last_became_peered":"2024-03-18T22:40:41.810104+0000","last_unstale":"2024-03-18T22:42:12.860732+0000","last_undegraded":"2024-03-18T22:42:12.860732+0000","last_fullsized":"2024-03-18T22:42:12.860732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:41.294286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.266735+0000","last_change":"2024-03-18T22:41:44.266318+0000","last_active":"2024-03-18T22:41:44.266735+0000","last_peered":"2024-03-18T22:41:44.266735+0000","last_clean":"2024-03-18T22:41:44.266735+0000","last_became_active":"2024-03-18T22:41:44.265903+0000","last_became_peered":"2024-03-18T22:41:44.265903+0000","last_unstale":"2024-03-18T22:41:44.266735+0000","last_undegraded":"2024-03-18T22:41:44.266735+0000","last_fullsized":"2024-03-18T22:41:44.266735+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:44.307096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":219,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:18.895690+0000","last_change":"2024-03-18T22:40:41.209837+0000","last_active":"2024-03-18T22:42:18.895690+0000","last_peered":"2024-03-18T22:42:18.895690+0000","last_clean":"2024-03-18T22:42:18.895690+0000","last_became_active":"2024-03-18T22:40:41.209465+0000","last_became_peered":"2024-03-18T22:40:41.209465+0000","last_unstale":"2024-03-18T22:42:18.895690+0000","last_undegraded":"2024-03-18T22:42:18.895690+0000","last_fullsized":"2024-03-18T22:42:18.895690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:45:15.765475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63606,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":154,"num_read_kb":140,"num_write":40,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.854214+0000","last_change":"2024-03-18T22:40:41.810058+0000","last_active":"2024-03-18T22:42:12.854214+0000","last_peered":"2024-03-18T22:42:12.854214+0000","last_clean":"2024-03-18T22:42:12.854214+0000","last_became_active":"2024-03-18T22:40:41.809677+0000","last_became_peered":"2024-03-18T22:40:41.809677+0000","last_unstale":"2024-03-18T22:42:12.854214+0000","last_undegraded":"2024-03-18T22:42:12.854214+0000","last_fullsized":"2024-03-18T22:42:12.854214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:03.935036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.000269+0000","last_change":"2024-03-18T22:40:45.845638+0000","last_active":"2024-03-18T22:41:44.000269+0000","last_peered":"2024-03-18T22:41:44.000269+0000","last_clean":"2024-03-18T22:41:44.000269+0000","last_became_active":"2024-03-18T22:40:45.844994+0000","last_became_peered":"2024-03-18T22:40:45.844994+0000","last_unstale":"2024-03-18T22:41:44.000269+0000","last_undegraded":"2024-03-18T22:41:44.000269+0000","last_fullsized":"2024-03-18T22:41:44.000269+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:21.551271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001706+0000","last_change":"2024-03-18T22:40:41.218397+0000","last_active":"2024-03-18T22:41:44.001706+0000","last_peered":"2024-03-18T22:41:44.001706+0000","last_clean":"2024-03-18T22:41:44.001706+0000","last_became_active":"2024-03-18T22:40:41.217587+0000","last_became_peered":"2024-03-18T22:40:41.217587+0000","last_unstale":"2024-03-18T22:41:44.001706+0000","last_undegraded":"2024-03-18T22:41:44.001706+0000","last_fullsized":"2024-03-18T22:41:44.001706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:36.524954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.839536+0000","last_change":"2024-03-18T22:40:41.812026+0000","last_active":"2024-03-18T22:42:12.839536+0000","last_peered":"2024-03-18T22:42:12.839536+0000","last_clean":"2024-03-18T22:42:12.839536+0000","last_became_active":"2024-03-18T22:40:41.811668+0000","last_became_peered":"2024-03-18T22:40:41.811668+0000","last_unstale":"2024-03-18T22:42:12.839536+0000","last_undegraded":"2024-03-18T22:42:12.839536+0000","last_fullsized":"2024-03-18T22:42:12.839536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:17:54.775327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.265213+0000","last_change":"2024-03-18T22:40:45.848466+0000","last_active":"2024-03-18T22:41:44.265213+0000","last_peered":"2024-03-18T22:41:44.265213+0000","last_clean":"2024-03-18T22:41:44.265213+0000","last_became_active":"2024-03-18T22:40:45.848078+0000","last_became_peered":"2024-03-18T22:40:45.848078+0000","last_unstale":"2024-03-18T22:41:44.265213+0000","last_undegraded":"2024-03-18T22:41:44.265213+0000","last_fullsized":"2024-03-18T22:41:44.265213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:28.515368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999779+0000","last_change":"2024-03-18T22:40:41.216434+0000","last_active":"2024-03-18T22:41:43.999779+0000","last_peered":"2024-03-18T22:41:43.999779+0000","last_clean":"2024-03-18T22:41:43.999779+0000","last_became_active":"2024-03-18T22:40:41.216103+0000","last_became_peered":"2024-03-18T22:40:41.216103+0000","last_unstale":"2024-03-18T22:41:43.999779+0000","last_undegraded":"2024-03-18T22:41:43.999779+0000","last_fullsized":"2024-03-18T22:41:43.999779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:39.516463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.854916+0000","last_change":"2024-03-18T22:40:41.808405+0000","last_active":"2024-03-18T22:42:12.854916+0000","last_peered":"2024-03-18T22:42:12.854916+0000","last_clean":"2024-03-18T22:42:12.854916+0000","last_became_active":"2024-03-18T22:40:41.808111+0000","last_became_peered":"2024-03-18T22:40:41.808111+0000","last_unstale":"2024-03-18T22:42:12.854916+0000","last_undegraded":"2024-03-18T22:42:12.854916+0000","last_fullsized":"2024-03-18T22:42:12.854916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:37:20.808163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.002088+0000","last_change":"2024-03-18T22:40:45.848409+0000","last_active":"2024-03-18T22:41:44.002088+0000","last_peered":"2024-03-18T22:41:44.002088+0000","last_clean":"2024-03-18T22:41:44.002088+0000","last_became_active":"2024-03-18T22:40:45.847687+0000","last_became_peered":"2024-03-18T22:40:45.847687+0000","last_unstale":"2024-03-18T22:41:44.002088+0000","last_undegraded":"2024-03-18T22:41:44.002088+0000","last_fullsized":"2024-03-18T22:41:44.002088+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:32.894885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:15.899663+0000","last_change":"2024-03-18T22:40:41.208408+0000","last_active":"2024-03-18T22:42:15.899663+0000","last_peered":"2024-03-18T22:42:15.899663+0000","last_clean":"2024-03-18T22:42:15.899663+0000","last_became_active":"2024-03-18T22:40:41.207976+0000","last_became_peered":"2024-03-18T22:40:41.207976+0000","last_unstale":"2024-03-18T22:42:15.899663+0000","last_undegraded":"2024-03-18T22:42:15.899663+0000","last_fullsized":"2024-03-18T22:42:15.899663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:43:24.416974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.848042+0000","last_change":"2024-03-18T22:40:41.813547+0000","last_active":"2024-03-18T22:42:12.848042+0000","last_peered":"2024-03-18T22:42:12.848042+0000","last_clean":"2024-03-18T22:42:12.848042+0000","last_became_active":"2024-03-18T22:40:41.813250+0000","last_became_peered":"2024-03-18T22:40:41.813250+0000","last_unstale":"2024-03-18T22:42:12.848042+0000","last_undegraded":"2024-03-18T22:42:12.848042+0000","last_fullsized":"2024-03-18T22:42:12.848042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:54.428485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001295+0000","last_change":"2024-03-18T22:40:45.845797+0000","last_active":"2024-03-18T22:41:44.001295+0000","last_peered":"2024-03-18T22:41:44.001295+0000","last_clean":"2024-03-18T22:41:44.001295+0000","last_became_active":"2024-03-18T22:40:45.845324+0000","last_became_peered":"2024-03-18T22:40:45.845324+0000","last_unstale":"2024-03-18T22:41:44.001295+0000","last_undegraded":"2024-03-18T22:41:44.001295+0000","last_fullsized":"2024-03-18T22:41:44.001295+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:21.613203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997100+0000","last_change":"2024-03-18T22:40:41.222127+0000","last_active":"2024-03-18T22:41:43.997100+0000","last_peered":"2024-03-18T22:41:43.997100+0000","last_clean":"2024-03-18T22:41:43.997100+0000","last_became_active":"2024-03-18T22:40:41.221830+0000","last_became_peered":"2024-03-18T22:40:41.221830+0000","last_unstale":"2024-03-18T22:41:43.997100+0000","last_undegraded":"2024-03-18T22:41:43.997100+0000","last_fullsized":"2024-03-18T22:41:43.997100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:24:02.235914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.856654+0000","last_change":"2024-03-18T22:40:41.814606+0000","last_active":"2024-03-18T22:42:12.856654+0000","last_peered":"2024-03-18T22:42:12.856654+0000","last_clean":"2024-03-18T22:42:12.856654+0000","last_became_active":"2024-03-18T22:40:41.814309+0000","last_became_peered":"2024-03-18T22:40:41.814309+0000","last_unstale":"2024-03-18T22:42:12.856654+0000","last_undegraded":"2024-03-18T22:42:12.856654+0000","last_fullsized":"2024-03-18T22:42:12.856654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:33.086229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999592+0000","last_change":"2024-03-18T22:40:45.845711+0000","last_active":"2024-03-18T22:41:43.999592+0000","last_peered":"2024-03-18T22:41:43.999592+0000","last_clean":"2024-03-18T22:41:43.999592+0000","last_became_active":"2024-03-18T22:40:45.845094+0000","last_became_peered":"2024-03-18T22:40:45.845094+0000","last_unstale":"2024-03-18T22:41:43.999592+0000","last_undegraded":"2024-03-18T22:41:43.999592+0000","last_fullsized":"2024-03-18T22:41:43.999592+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:08.372875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997818+0000","last_change":"2024-03-18T22:40:41.219076+0000","last_active":"2024-03-18T22:41:43.997818+0000","last_peered":"2024-03-18T22:41:43.997818+0000","last_clean":"2024-03-18T22:41:43.997818+0000","last_became_active":"2024-03-18T22:40:41.217580+0000","last_became_peered":"2024-03-18T22:40:41.217580+0000","last_unstale":"2024-03-18T22:41:43.997818+0000","last_undegraded":"2024-03-18T22:41:43.997818+0000","last_fullsized":"2024-03-18T22:41:43.997818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:38.112325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.845514+0000","last_change":"2024-03-18T22:40:41.810267+0000","last_active":"2024-03-18T22:42:12.845514+0000","last_peered":"2024-03-18T22:42:12.845514+0000","last_clean":"2024-03-18T22:42:12.845514+0000","last_became_active":"2024-03-18T22:40:41.809960+0000","last_became_peered":"2024-03-18T22:40:41.809960+0000","last_unstale":"2024-03-18T22:42:12.845514+0000","last_undegraded":"2024-03-18T22:42:12.845514+0000","last_fullsized":"2024-03-18T22:42:12.845514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:00.973817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997199+0000","last_change":"2024-03-18T22:40:45.848461+0000","last_active":"2024-03-18T22:41:43.997199+0000","last_peered":"2024-03-18T22:41:43.997199+0000","last_clean":"2024-03-18T22:41:43.997199+0000","last_became_active":"2024-03-18T22:40:45.848043+0000","last_became_peered":"2024-03-18T22:40:45.848043+0000","last_unstale":"2024-03-18T22:41:43.997199+0000","last_undegraded":"2024-03-18T22:41:43.997199+0000","last_fullsized":"2024-03-18T22:41:43.997199+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:38.865565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.002567+0000","last_change":"2024-03-18T22:40:41.213069+0000","last_active":"2024-03-18T22:41:44.002567+0000","last_peered":"2024-03-18T22:41:44.002567+0000","last_clean":"2024-03-18T22:41:44.002567+0000","last_became_active":"2024-03-18T22:40:41.212725+0000","last_became_peered":"2024-03-18T22:40:41.212725+0000","last_unstale":"2024-03-18T22:41:44.002567+0000","last_undegraded":"2024-03-18T22:41:44.002567+0000","last_fullsized":"2024-03-18T22:41:44.002567+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:50.219433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.848639+0000","last_change":"2024-03-18T22:41:44.004772+0000","last_active":"2024-03-18T22:42:12.848639+0000","last_peered":"2024-03-18T22:42:12.848639+0000","last_clean":"2024-03-18T22:42:12.848639+0000","last_became_active":"2024-03-18T22:41:44.004342+0000","last_became_peered":"2024-03-18T22:41:44.004342+0000","last_unstale":"2024-03-18T22:42:12.848639+0000","last_undegraded":"2024-03-18T22:42:12.848639+0000","last_fullsized":"2024-03-18T22:42:12.848639+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:18:38.674911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.000521+0000","last_change":"2024-03-18T22:40:45.840058+0000","last_active":"2024-03-18T22:41:44.000521+0000","last_peered":"2024-03-18T22:41:44.000521+0000","last_clean":"2024-03-18T22:41:44.000521+0000","last_became_active":"2024-03-18T22:40:45.839661+0000","last_became_peered":"2024-03-18T22:40:45.839661+0000","last_unstale":"2024-03-18T22:41:44.000521+0000","last_undegraded":"2024-03-18T22:41:44.000521+0000","last_fullsized":"2024-03-18T22:41:44.000521+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:35:14.323972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'7","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:15.931529+0000","last_change":"2024-03-18T22:40:41.215988+0000","last_active":"2024-03-18T22:42:15.931529+0000","last_peered":"2024-03-18T22:42:15.931529+0000","last_clean":"2024-03-18T22:42:15.931529+0000","last_became_active":"2024-03-18T22:40:41.215680+0000","last_became_peered":"2024-03-18T22:40:41.215680+0000","last_unstale":"2024-03-18T22:42:15.931529+0000","last_undegraded":"2024-03-18T22:42:15.931529+0000","last_fullsized":"2024-03-18T22:42:15.931529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:43.412362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.850700+0000","last_change":"2024-03-18T22:40:41.812521+0000","last_active":"2024-03-18T22:42:12.850700+0000","last_peered":"2024-03-18T22:42:12.850700+0000","last_clean":"2024-03-18T22:42:12.850700+0000","last_became_active":"2024-03-18T22:40:41.812223+0000","last_became_peered":"2024-03-18T22:40:41.812223+0000","last_unstale":"2024-03-18T22:42:12.850700+0000","last_undegraded":"2024-03-18T22:42:12.850700+0000","last_fullsized":"2024-03-18T22:42:12.850700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:35:37.945144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999563+0000","last_change":"2024-03-18T22:40:45.838372+0000","last_active":"2024-03-18T22:41:43.999563+0000","last_peered":"2024-03-18T22:41:43.999563+0000","last_clean":"2024-03-18T22:41:43.999563+0000","last_became_active":"2024-03-18T22:40:45.837962+0000","last_became_peered":"2024-03-18T22:40:45.837962+0000","last_unstale":"2024-03-18T22:41:43.999563+0000","last_undegraded":"2024-03-18T22:41:43.999563+0000","last_fullsized":"2024-03-18T22:41:43.999563+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:18:39.323023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.838523+0000","last_change":"2024-03-18T22:40:41.812146+0000","last_active":"2024-03-18T22:42:12.838523+0000","last_peered":"2024-03-18T22:42:12.838523+0000","last_clean":"2024-03-18T22:42:12.838523+0000","last_became_active":"2024-03-18T22:40:41.811840+0000","last_became_peered":"2024-03-18T22:40:41.811840+0000","last_unstale":"2024-03-18T22:42:12.838523+0000","last_undegraded":"2024-03-18T22:42:12.838523+0000","last_fullsized":"2024-03-18T22:42:12.838523+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:30.829187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:14.897912+0000","last_change":"2024-03-18T22:40:41.212133+0000","last_active":"2024-03-18T22:42:14.897912+0000","last_peered":"2024-03-18T22:42:14.897912+0000","last_clean":"2024-03-18T22:42:14.897912+0000","last_became_active":"2024-03-18T22:40:41.211581+0000","last_became_peered":"2024-03-18T22:40:41.211581+0000","last_unstale":"2024-03-18T22:42:14.897912+0000","last_undegraded":"2024-03-18T22:42:14.897912+0000","last_fullsized":"2024-03-18T22:42:14.897912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:44.287725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.300965+0000","last_change":"2024-03-18T22:40:45.843660+0000","last_active":"2024-03-18T22:41:44.300965+0000","last_peered":"2024-03-18T22:41:44.300965+0000","last_clean":"2024-03-18T22:41:44.300965+0000","last_became_active":"2024-03-18T22:40:45.843233+0000","last_became_peered":"2024-03-18T22:40:45.843233+0000","last_unstale":"2024-03-18T22:41:44.300965+0000","last_undegraded":"2024-03-18T22:41:44.300965+0000","last_fullsized":"2024-03-18T22:41:44.300965+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:08:19.758496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.838172+0000","last_change":"2024-03-18T22:40:41.811798+0000","last_active":"2024-03-18T22:42:12.838172+0000","last_peered":"2024-03-18T22:42:12.838172+0000","last_clean":"2024-03-18T22:42:12.838172+0000","last_became_active":"2024-03-18T22:40:41.811422+0000","last_became_peered":"2024-03-18T22:40:41.811422+0000","last_unstale":"2024-03-18T22:42:12.838172+0000","last_undegraded":"2024-03-18T22:42:12.838172+0000","last_fullsized":"2024-03-18T22:42:12.838172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:53.944556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.998409+0000","last_change":"2024-03-18T22:40:41.210052+0000","last_active":"2024-03-18T22:41:43.998409+0000","last_peered":"2024-03-18T22:41:43.998409+0000","last_clean":"2024-03-18T22:41:43.998409+0000","last_became_active":"2024-03-18T22:40:41.209647+0000","last_became_peered":"2024-03-18T22:40:41.209647+0000","last_unstale":"2024-03-18T22:41:43.998409+0000","last_undegraded":"2024-03-18T22:41:43.998409+0000","last_fullsized":"2024-03-18T22:41:43.998409+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:57:35.187348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.299453+0000","last_change":"2024-03-18T22:39:32.831758+0000","last_active":"2024-03-18T22:41:44.299453+0000","last_peered":"2024-03-18T22:41:44.299453+0000","last_clean":"2024-03-18T22:41:44.299453+0000","last_became_active":"2024-03-18T22:39:32.831287+0000","last_became_peered":"2024-03-18T22:39:32.831287+0000","last_unstale":"2024-03-18T22:41:44.299453+0000","last_undegraded":"2024-03-18T22:41:44.299453+0000","last_fullsized":"2024-03-18T22:41:44.299453+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:36:07.806854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:36:07.806854+0000","last_clean_scrub_stamp":"2024-03-18T22:36:07.806854+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:42.449107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.000291+0000","last_change":"2024-03-18T22:40:45.842008+0000","last_active":"2024-03-18T22:41:44.000291+0000","last_peered":"2024-03-18T22:41:44.000291+0000","last_clean":"2024-03-18T22:41:44.000291+0000","last_became_active":"2024-03-18T22:40:45.841231+0000","last_became_peered":"2024-03-18T22:40:45.841231+0000","last_unstale":"2024-03-18T22:41:44.000291+0000","last_undegraded":"2024-03-18T22:41:44.000291+0000","last_fullsized":"2024-03-18T22:41:44.000291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:06.812049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.836440+0000","last_change":"2024-03-18T22:40:41.807216+0000","last_active":"2024-03-18T22:42:12.836440+0000","last_peered":"2024-03-18T22:42:12.836440+0000","last_clean":"2024-03-18T22:42:12.836440+0000","last_became_active":"2024-03-18T22:40:41.806797+0000","last_became_peered":"2024-03-18T22:40:41.806797+0000","last_unstale":"2024-03-18T22:42:12.836440+0000","last_undegraded":"2024-03-18T22:42:12.836440+0000","last_fullsized":"2024-03-18T22:42:12.836440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:26.422329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.299426+0000","last_change":"2024-03-18T22:40:41.216555+0000","last_active":"2024-03-18T22:41:44.299426+0000","last_peered":"2024-03-18T22:41:44.299426+0000","last_clean":"2024-03-18T22:41:44.299426+0000","last_became_active":"2024-03-18T22:40:41.216240+0000","last_became_peered":"2024-03-18T22:40:41.216240+0000","last_unstale":"2024-03-18T22:41:44.299426+0000","last_undegraded":"2024-03-18T22:41:44.299426+0000","last_fullsized":"2024-03-18T22:41:44.299426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:04.655182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.264909+0000","last_change":"2024-03-18T22:40:45.839889+0000","last_active":"2024-03-18T22:41:44.264909+0000","last_peered":"2024-03-18T22:41:44.264909+0000","last_clean":"2024-03-18T22:41:44.264909+0000","last_became_active":"2024-03-18T22:40:45.839414+0000","last_became_peered":"2024-03-18T22:40:45.839414+0000","last_unstale":"2024-03-18T22:41:44.264909+0000","last_undegraded":"2024-03-18T22:41:44.264909+0000","last_fullsized":"2024-03-18T22:41:44.264909+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:52.497378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997719+0000","last_change":"2024-03-18T22:40:41.221026+0000","last_active":"2024-03-18T22:41:43.997719+0000","last_peered":"2024-03-18T22:41:43.997719+0000","last_clean":"2024-03-18T22:41:43.997719+0000","last_became_active":"2024-03-18T22:40:41.220681+0000","last_became_peered":"2024-03-18T22:40:41.220681+0000","last_unstale":"2024-03-18T22:41:43.997719+0000","last_undegraded":"2024-03-18T22:41:43.997719+0000","last_fullsized":"2024-03-18T22:41:43.997719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:11.490445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.836527+0000","last_change":"2024-03-18T22:40:41.810500+0000","last_active":"2024-03-18T22:42:12.836527+0000","last_peered":"2024-03-18T22:42:12.836527+0000","last_clean":"2024-03-18T22:42:12.836527+0000","last_became_active":"2024-03-18T22:40:41.809723+0000","last_became_peered":"2024-03-18T22:40:41.809723+0000","last_unstale":"2024-03-18T22:42:12.836527+0000","last_undegraded":"2024-03-18T22:42:12.836527+0000","last_fullsized":"2024-03-18T22:42:12.836527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:18:37.053809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999142+0000","last_change":"2024-03-18T22:40:45.844323+0000","last_active":"2024-03-18T22:41:43.999142+0000","last_peered":"2024-03-18T22:41:43.999142+0000","last_clean":"2024-03-18T22:41:43.999142+0000","last_became_active":"2024-03-18T22:40:45.843146+0000","last_became_peered":"2024-03-18T22:40:45.843146+0000","last_unstale":"2024-03-18T22:41:43.999142+0000","last_undegraded":"2024-03-18T22:41:43.999142+0000","last_fullsized":"2024-03-18T22:41:43.999142+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:14:59.881467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'13","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.002718+0000","last_change":"2024-03-18T22:40:41.212126+0000","last_active":"2024-03-18T22:41:44.002718+0000","last_peered":"2024-03-18T22:41:44.002718+0000","last_clean":"2024-03-18T22:41:44.002718+0000","last_became_active":"2024-03-18T22:40:41.211772+0000","last_became_peered":"2024-03-18T22:40:41.211772+0000","last_unstale":"2024-03-18T22:41:44.002718+0000","last_undegraded":"2024-03-18T22:41:44.002718+0000","last_fullsized":"2024-03-18T22:41:44.002718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:03.215751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.844532+0000","last_change":"2024-03-18T22:40:41.820476+0000","last_active":"2024-03-18T22:42:12.844532+0000","last_peered":"2024-03-18T22:42:12.844532+0000","last_clean":"2024-03-18T22:42:12.844532+0000","last_became_active":"2024-03-18T22:40:41.820084+0000","last_became_peered":"2024-03-18T22:40:41.820084+0000","last_unstale":"2024-03-18T22:42:12.844532+0000","last_undegraded":"2024-03-18T22:42:12.844532+0000","last_fullsized":"2024-03-18T22:42:12.844532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:51:50.580208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001221+0000","last_change":"2024-03-18T22:40:45.838588+0000","last_active":"2024-03-18T22:41:44.001221+0000","last_peered":"2024-03-18T22:41:44.001221+0000","last_clean":"2024-03-18T22:41:44.001221+0000","last_became_active":"2024-03-18T22:40:45.838068+0000","last_became_peered":"2024-03-18T22:40:45.838068+0000","last_unstale":"2024-03-18T22:41:44.001221+0000","last_undegraded":"2024-03-18T22:41:44.001221+0000","last_fullsized":"2024-03-18T22:41:44.001221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:11:02.162668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.998852+0000","last_change":"2024-03-18T22:40:41.210108+0000","last_active":"2024-03-18T22:41:43.998852+0000","last_peered":"2024-03-18T22:41:43.998852+0000","last_clean":"2024-03-18T22:41:43.998852+0000","last_became_active":"2024-03-18T22:40:41.209781+0000","last_became_peered":"2024-03-18T22:40:41.209781+0000","last_unstale":"2024-03-18T22:41:43.998852+0000","last_undegraded":"2024-03-18T22:41:43.998852+0000","last_fullsized":"2024-03-18T22:41:43.998852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:10.670822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.845491+0000","last_change":"2024-03-18T22:40:41.814637+0000","last_active":"2024-03-18T22:42:12.845491+0000","last_peered":"2024-03-18T22:42:12.845491+0000","last_clean":"2024-03-18T22:42:12.845491+0000","last_became_active":"2024-03-18T22:40:41.814279+0000","last_became_peered":"2024-03-18T22:40:41.814279+0000","last_unstale":"2024-03-18T22:42:12.845491+0000","last_undegraded":"2024-03-18T22:42:12.845491+0000","last_fullsized":"2024-03-18T22:42:12.845491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:12.496230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997696+0000","last_change":"2024-03-18T22:40:45.851560+0000","last_active":"2024-03-18T22:41:43.997696+0000","last_peered":"2024-03-18T22:41:43.997696+0000","last_clean":"2024-03-18T22:41:43.997696+0000","last_became_active":"2024-03-18T22:40:45.850884+0000","last_became_peered":"2024-03-18T22:40:45.850884+0000","last_unstale":"2024-03-18T22:41:43.997696+0000","last_undegraded":"2024-03-18T22:41:43.997696+0000","last_fullsized":"2024-03-18T22:41:43.997696+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:04.114468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.298816+0000","last_change":"2024-03-18T22:40:41.210156+0000","last_active":"2024-03-18T22:41:44.298816+0000","last_peered":"2024-03-18T22:41:44.298816+0000","last_clean":"2024-03-18T22:41:44.298816+0000","last_became_active":"2024-03-18T22:40:41.209855+0000","last_became_peered":"2024-03-18T22:40:41.209855+0000","last_unstale":"2024-03-18T22:41:44.298816+0000","last_undegraded":"2024-03-18T22:41:44.298816+0000","last_fullsized":"2024-03-18T22:41:44.298816+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:02.752573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.803809+0000","last_change":"2024-03-18T22:40:41.803681+0000","last_active":"2024-03-18T22:42:12.803809+0000","last_peered":"2024-03-18T22:42:12.803809+0000","last_clean":"2024-03-18T22:42:12.803809+0000","last_became_active":"2024-03-18T22:40:41.803244+0000","last_became_peered":"2024-03-18T22:40:41.803244+0000","last_unstale":"2024-03-18T22:42:12.803809+0000","last_undegraded":"2024-03-18T22:42:12.803809+0000","last_fullsized":"2024-03-18T22:42:12.803809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:55.944575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.006624+0000","last_change":"2024-03-18T22:41:44.005707+0000","last_active":"2024-03-18T22:41:44.006624+0000","last_peered":"2024-03-18T22:41:44.006624+0000","last_clean":"2024-03-18T22:41:44.006624+0000","last_became_active":"2024-03-18T22:41:44.004338+0000","last_became_peered":"2024-03-18T22:41:44.004338+0000","last_unstale":"2024-03-18T22:41:44.006624+0000","last_undegraded":"2024-03-18T22:41:44.006624+0000","last_fullsized":"2024-03-18T22:41:44.006624+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:46.054087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999722+0000","last_change":"2024-03-18T22:40:41.210830+0000","last_active":"2024-03-18T22:41:43.999722+0000","last_peered":"2024-03-18T22:41:43.999722+0000","last_clean":"2024-03-18T22:41:43.999722+0000","last_became_active":"2024-03-18T22:40:41.210476+0000","last_became_peered":"2024-03-18T22:40:41.210476+0000","last_unstale":"2024-03-18T22:41:43.999722+0000","last_undegraded":"2024-03-18T22:41:43.999722+0000","last_fullsized":"2024-03-18T22:41:43.999722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:13.243788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.842580+0000","last_change":"2024-03-18T22:40:41.807678+0000","last_active":"2024-03-18T22:42:12.842580+0000","last_peered":"2024-03-18T22:42:12.842580+0000","last_clean":"2024-03-18T22:42:12.842580+0000","last_became_active":"2024-03-18T22:40:41.807383+0000","last_became_peered":"2024-03-18T22:40:41.807383+0000","last_unstale":"2024-03-18T22:42:12.842580+0000","last_undegraded":"2024-03-18T22:42:12.842580+0000","last_fullsized":"2024-03-18T22:42:12.842580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:27:55.930293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997772+0000","last_change":"2024-03-18T22:40:45.850982+0000","last_active":"2024-03-18T22:41:43.997772+0000","last_peered":"2024-03-18T22:41:43.997772+0000","last_clean":"2024-03-18T22:41:43.997772+0000","last_became_active":"2024-03-18T22:40:45.850526+0000","last_became_peered":"2024-03-18T22:40:45.850526+0000","last_unstale":"2024-03-18T22:41:43.997772+0000","last_undegraded":"2024-03-18T22:41:43.997772+0000","last_fullsized":"2024-03-18T22:41:43.997772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:05:24.450986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001540+0000","last_change":"2024-03-18T22:40:41.219540+0000","last_active":"2024-03-18T22:41:44.001540+0000","last_peered":"2024-03-18T22:41:44.001540+0000","last_clean":"2024-03-18T22:41:44.001540+0000","last_became_active":"2024-03-18T22:40:41.219206+0000","last_became_peered":"2024-03-18T22:40:41.219206+0000","last_unstale":"2024-03-18T22:41:44.001540+0000","last_undegraded":"2024-03-18T22:41:44.001540+0000","last_fullsized":"2024-03-18T22:41:44.001540+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:33.586368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.852188+0000","last_change":"2024-03-18T22:40:41.815561+0000","last_active":"2024-03-18T22:42:12.852188+0000","last_peered":"2024-03-18T22:42:12.852188+0000","last_clean":"2024-03-18T22:42:12.852188+0000","last_became_active":"2024-03-18T22:40:41.815082+0000","last_became_peered":"2024-03-18T22:40:41.815082+0000","last_unstale":"2024-03-18T22:42:12.852188+0000","last_undegraded":"2024-03-18T22:42:12.852188+0000","last_fullsized":"2024-03-18T22:42:12.852188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:01.676907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997674+0000","last_change":"2024-03-18T22:40:45.845386+0000","last_active":"2024-03-18T22:41:43.997674+0000","last_peered":"2024-03-18T22:41:43.997674+0000","last_clean":"2024-03-18T22:41:43.997674+0000","last_became_active":"2024-03-18T22:40:45.845007+0000","last_became_peered":"2024-03-18T22:40:45.845007+0000","last_unstale":"2024-03-18T22:41:43.997674+0000","last_undegraded":"2024-03-18T22:41:43.997674+0000","last_fullsized":"2024-03-18T22:41:43.997674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:41.429195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.997236+0000","last_change":"2024-03-18T22:40:41.215559+0000","last_active":"2024-03-18T22:41:43.997236+0000","last_peered":"2024-03-18T22:41:43.997236+0000","last_clean":"2024-03-18T22:41:43.997236+0000","last_became_active":"2024-03-18T22:40:41.215253+0000","last_became_peered":"2024-03-18T22:40:41.215253+0000","last_unstale":"2024-03-18T22:41:43.997236+0000","last_undegraded":"2024-03-18T22:41:43.997236+0000","last_fullsized":"2024-03-18T22:41:43.997236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:00:19.706011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.857216+0000","last_change":"2024-03-18T22:40:41.807533+0000","last_active":"2024-03-18T22:42:12.857216+0000","last_peered":"2024-03-18T22:42:12.857216+0000","last_clean":"2024-03-18T22:42:12.857216+0000","last_became_active":"2024-03-18T22:40:41.807219+0000","last_became_peered":"2024-03-18T22:40:41.807219+0000","last_unstale":"2024-03-18T22:42:12.857216+0000","last_undegraded":"2024-03-18T22:42:12.857216+0000","last_fullsized":"2024-03-18T22:42:12.857216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:15:24.214367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.299142+0000","last_change":"2024-03-18T22:40:45.839670+0000","last_active":"2024-03-18T22:41:44.299142+0000","last_peered":"2024-03-18T22:41:44.299142+0000","last_clean":"2024-03-18T22:41:44.299142+0000","last_became_active":"2024-03-18T22:40:45.839263+0000","last_became_peered":"2024-03-18T22:40:45.839263+0000","last_unstale":"2024-03-18T22:41:44.299142+0000","last_undegraded":"2024-03-18T22:41:44.299142+0000","last_fullsized":"2024-03-18T22:41:44.299142+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:48.991959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":101,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:14.933774+0000","last_change":"2024-03-18T22:40:41.209617+0000","last_active":"2024-03-18T22:42:14.933774+0000","last_peered":"2024-03-18T22:42:14.933774+0000","last_clean":"2024-03-18T22:42:14.933774+0000","last_became_active":"2024-03-18T22:40:41.209311+0000","last_became_peered":"2024-03-18T22:40:41.209311+0000","last_unstale":"2024-03-18T22:42:14.933774+0000","last_undegraded":"2024-03-18T22:42:14.933774+0000","last_fullsized":"2024-03-18T22:42:14.933774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:44:06.019870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":69,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.858901+0000","last_change":"2024-03-18T22:40:41.808210+0000","last_active":"2024-03-18T22:42:12.858901+0000","last_peered":"2024-03-18T22:42:12.858901+0000","last_clean":"2024-03-18T22:42:12.858901+0000","last_became_active":"2024-03-18T22:40:41.807898+0000","last_became_peered":"2024-03-18T22:40:41.807898+0000","last_unstale":"2024-03-18T22:42:12.858901+0000","last_undegraded":"2024-03-18T22:42:12.858901+0000","last_fullsized":"2024-03-18T22:42:12.858901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:20.401165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.333674+0000","last_change":"2024-03-18T22:40:45.841905+0000","last_active":"2024-03-18T22:41:44.333674+0000","last_peered":"2024-03-18T22:41:44.333674+0000","last_clean":"2024-03-18T22:41:44.333674+0000","last_became_active":"2024-03-18T22:40:45.841464+0000","last_became_peered":"2024-03-18T22:40:45.841464+0000","last_unstale":"2024-03-18T22:41:44.333674+0000","last_undegraded":"2024-03-18T22:41:44.333674+0000","last_fullsized":"2024-03-18T22:41:44.333674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:30.435788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.000167+0000","last_change":"2024-03-18T22:40:41.216919+0000","last_active":"2024-03-18T22:41:44.000167+0000","last_peered":"2024-03-18T22:41:44.000167+0000","last_clean":"2024-03-18T22:41:44.000167+0000","last_became_active":"2024-03-18T22:40:41.216608+0000","last_became_peered":"2024-03-18T22:40:41.216608+0000","last_unstale":"2024-03-18T22:41:44.000167+0000","last_undegraded":"2024-03-18T22:41:44.000167+0000","last_fullsized":"2024-03-18T22:41:44.000167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:10:42.230268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.838803+0000","last_change":"2024-03-18T22:40:41.813879+0000","last_active":"2024-03-18T22:42:12.838803+0000","last_peered":"2024-03-18T22:42:12.838803+0000","last_clean":"2024-03-18T22:42:12.838803+0000","last_became_active":"2024-03-18T22:40:41.813584+0000","last_became_peered":"2024-03-18T22:40:41.813584+0000","last_unstale":"2024-03-18T22:42:12.838803+0000","last_undegraded":"2024-03-18T22:42:12.838803+0000","last_fullsized":"2024-03-18T22:42:12.838803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:20:44.808087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:43.999146+0000","last_change":"2024-03-18T22:40:45.838033+0000","last_active":"2024-03-18T22:41:43.999146+0000","last_peered":"2024-03-18T22:41:43.999146+0000","last_clean":"2024-03-18T22:41:43.999146+0000","last_became_active":"2024-03-18T22:40:45.837530+0000","last_became_peered":"2024-03-18T22:40:45.837530+0000","last_unstale":"2024-03-18T22:41:43.999146+0000","last_undegraded":"2024-03-18T22:41:43.999146+0000","last_fullsized":"2024-03-18T22:41:43.999146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:43:43.444631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:41:44.001391+0000","last_change":"2024-03-18T22:40:41.215686+0000","last_active":"2024-03-18T22:41:44.001391+0000","last_peered":"2024-03-18T22:41:44.001391+0000","last_clean":"2024-03-18T22:41:44.001391+0000","last_became_active":"2024-03-18T22:40:41.215362+0000","last_became_peered":"2024-03-18T22:40:41.215362+0000","last_unstale":"2024-03-18T22:41:44.001391+0000","last_undegraded":"2024-03-18T22:41:44.001391+0000","last_fullsized":"2024-03-18T22:41:44.001391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:46:30.546304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.853485+0000","last_change":"2024-03-18T22:40:41.808959+0000","last_active":"2024-03-18T22:42:12.853485+0000","last_peered":"2024-03-18T22:42:12.853485+0000","last_clean":"2024-03-18T22:42:12.853485+0000","last_became_active":"2024-03-18T22:40:41.808671+0000","last_became_peered":"2024-03-18T22:40:41.808671+0000","last_unstale":"2024-03-18T22:42:12.853485+0000","last_undegraded":"2024-03-18T22:42:12.853485+0000","last_fullsized":"2024-03-18T22:42:12.853485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:37.115318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.845325+0000","last_change":"2024-03-18T22:40:41.814240+0000","last_active":"2024-03-18T22:42:12.845325+0000","last_peered":"2024-03-18T22:42:12.845325+0000","last_clean":"2024-03-18T22:42:12.845325+0000","last_became_active":"2024-03-18T22:40:41.813925+0000","last_became_peered":"2024-03-18T22:40:41.813925+0000","last_unstale":"2024-03-18T22:42:12.845325+0000","last_undegraded":"2024-03-18T22:42:12.845325+0000","last_fullsized":"2024-03-18T22:42:12.845325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:38.999803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.839477+0000","last_change":"2024-03-18T22:40:41.813381+0000","last_active":"2024-03-18T22:42:12.839477+0000","last_peered":"2024-03-18T22:42:12.839477+0000","last_clean":"2024-03-18T22:42:12.839477+0000","last_became_active":"2024-03-18T22:40:41.812927+0000","last_became_peered":"2024-03-18T22:40:41.812927+0000","last_unstale":"2024-03-18T22:42:12.839477+0000","last_undegraded":"2024-03-18T22:42:12.839477+0000","last_fullsized":"2024-03-18T22:42:12.839477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:23:09.178485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.838241+0000","last_change":"2024-03-18T22:40:41.812513+0000","last_active":"2024-03-18T22:42:12.838241+0000","last_peered":"2024-03-18T22:42:12.838241+0000","last_clean":"2024-03-18T22:42:12.838241+0000","last_became_active":"2024-03-18T22:40:41.811882+0000","last_became_peered":"2024-03-18T22:40:41.811882+0000","last_unstale":"2024-03-18T22:42:12.838241+0000","last_undegraded":"2024-03-18T22:42:12.838241+0000","last_fullsized":"2024-03-18T22:42:12.838241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:45.596911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.845097+0000","last_change":"2024-03-18T22:40:41.814960+0000","last_active":"2024-03-18T22:42:12.845097+0000","last_peered":"2024-03-18T22:42:12.845097+0000","last_clean":"2024-03-18T22:42:12.845097+0000","last_became_active":"2024-03-18T22:40:41.814620+0000","last_became_peered":"2024-03-18T22:40:41.814620+0000","last_unstale":"2024-03-18T22:42:12.845097+0000","last_undegraded":"2024-03-18T22:42:12.845097+0000","last_fullsized":"2024-03-18T22:42:12.845097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:42.205885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.839450+0000","last_change":"2024-03-18T22:41:44.334913+0000","last_active":"2024-03-18T22:42:12.839450+0000","last_peered":"2024-03-18T22:42:12.839450+0000","last_clean":"2024-03-18T22:42:12.839450+0000","last_became_active":"2024-03-18T22:41:44.334554+0000","last_became_peered":"2024-03-18T22:41:44.334554+0000","last_unstale":"2024-03-18T22:42:12.839450+0000","last_undegraded":"2024-03-18T22:42:12.839450+0000","last_fullsized":"2024-03-18T22:42:12.839450+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:00:17.369015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.842272+0000","last_change":"2024-03-18T22:40:41.808718+0000","last_active":"2024-03-18T22:42:12.842272+0000","last_peered":"2024-03-18T22:42:12.842272+0000","last_clean":"2024-03-18T22:42:12.842272+0000","last_became_active":"2024-03-18T22:40:41.808379+0000","last_became_peered":"2024-03-18T22:40:41.808379+0000","last_unstale":"2024-03-18T22:42:12.842272+0000","last_undegraded":"2024-03-18T22:42:12.842272+0000","last_fullsized":"2024-03-18T22:42:12.842272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:26.637531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.847565+0000","last_change":"2024-03-18T22:40:41.818046+0000","last_active":"2024-03-18T22:42:12.847565+0000","last_peered":"2024-03-18T22:42:12.847565+0000","last_clean":"2024-03-18T22:42:12.847565+0000","last_became_active":"2024-03-18T22:40:41.817698+0000","last_became_peered":"2024-03-18T22:40:41.817698+0000","last_unstale":"2024-03-18T22:42:12.847565+0000","last_undegraded":"2024-03-18T22:42:12.847565+0000","last_fullsized":"2024-03-18T22:42:12.847565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:22.967673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.821054+0000","last_change":"2024-03-18T22:40:41.819337+0000","last_active":"2024-03-18T22:42:12.821054+0000","last_peered":"2024-03-18T22:42:12.821054+0000","last_clean":"2024-03-18T22:42:12.821054+0000","last_became_active":"2024-03-18T22:40:41.818987+0000","last_became_peered":"2024-03-18T22:40:41.818987+0000","last_unstale":"2024-03-18T22:42:12.821054+0000","last_undegraded":"2024-03-18T22:42:12.821054+0000","last_fullsized":"2024-03-18T22:42:12.821054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:07:04.529454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.835632+0000","last_change":"2024-03-18T22:40:41.813262+0000","last_active":"2024-03-18T22:42:12.835632+0000","last_peered":"2024-03-18T22:42:12.835632+0000","last_clean":"2024-03-18T22:42:12.835632+0000","last_became_active":"2024-03-18T22:40:41.812956+0000","last_became_peered":"2024-03-18T22:40:41.812956+0000","last_unstale":"2024-03-18T22:42:12.835632+0000","last_undegraded":"2024-03-18T22:42:12.835632+0000","last_fullsized":"2024-03-18T22:42:12.835632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:19.940097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.829350+0000","last_change":"2024-03-18T22:40:41.811897+0000","last_active":"2024-03-18T22:42:12.829350+0000","last_peered":"2024-03-18T22:42:12.829350+0000","last_clean":"2024-03-18T22:42:12.829350+0000","last_became_active":"2024-03-18T22:40:41.811575+0000","last_became_peered":"2024-03-18T22:40:41.811575+0000","last_unstale":"2024-03-18T22:42:12.829350+0000","last_undegraded":"2024-03-18T22:42:12.829350+0000","last_fullsized":"2024-03-18T22:42:12.829350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:45:43.321057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.859376+0000","last_change":"2024-03-18T22:40:41.811757+0000","last_active":"2024-03-18T22:42:12.859376+0000","last_peered":"2024-03-18T22:42:12.859376+0000","last_clean":"2024-03-18T22:42:12.859376+0000","last_became_active":"2024-03-18T22:40:41.811405+0000","last_became_peered":"2024-03-18T22:40:41.811405+0000","last_unstale":"2024-03-18T22:42:12.859376+0000","last_undegraded":"2024-03-18T22:42:12.859376+0000","last_fullsized":"2024-03-18T22:42:12.859376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:39.224604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.839451+0000","last_change":"2024-03-18T22:40:41.809987+0000","last_active":"2024-03-18T22:42:12.839451+0000","last_peered":"2024-03-18T22:42:12.839451+0000","last_clean":"2024-03-18T22:42:12.839451+0000","last_became_active":"2024-03-18T22:40:41.809544+0000","last_became_peered":"2024-03-18T22:40:41.809544+0000","last_unstale":"2024-03-18T22:42:12.839451+0000","last_undegraded":"2024-03-18T22:42:12.839451+0000","last_fullsized":"2024-03-18T22:42:12.839451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:23.800949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T22:42:12.847977+0000","last_change":"2024-03-18T22:40:41.810963+0000","last_active":"2024-03-18T22:42:12.847977+0000","last_peered":"2024-03-18T22:42:12.847977+0000","last_clean":"2024-03-18T22:42:12.847977+0000","last_became_active":"2024-03-18T22:40:41.810641+0000","last_became_peered":"2024-03-18T22:40:41.810641+0000","last_unstale":"2024-03-18T22:42:12.847977+0000","last_undegraded":"2024-03-18T22:42:12.847977+0000","last_fullsized":"2024-03-18T22:42:12.847977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:37:48.872961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71624,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":747,"num_read_kb":455,"num_write":254,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383712,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":501425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37024,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708128,"statfs":{"total":95995035648,"available":95957123072,"internally_reserved":0,"allocated":1728512,"data_stored":1152379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808874,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":505451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070510,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":501733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332149,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1099435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":497391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1101824,"data_stored":562027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":498315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1122304,"data_stored":562247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":496567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":501465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757311,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1658880,"data_stored":1093953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":65890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":65874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T22:42:23.525 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-18T22:42:23.525 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-18T22:42:23.526 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-18T22:42:23.526 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-18T22:42:23.776 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:42:24.302 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:23 smithi040 ceph-mon[62643]: pgmap v341: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T22:42:24.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:23 smithi113 ceph-mon[66775]: pgmap v341: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T22:42:24.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:23 smithi103 ceph-mon[66377]: pgmap v341: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T22:42:25.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.626+0000 7f0935446700 1 -- 172.21.15.40:0/748646492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0930101bb0 msgr2=0x7f093010e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:25.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.626+0000 7f0935446700 1 --2- 172.21.15.40:0/748646492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0930101bb0 0x7f093010e050 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0928009f90 tx=0x7f092800b330 comp rx=0 tx=0).stop 2024-03-18T22:42:25.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.626+0000 7f0935446700 1 -- 172.21.15.40:0/748646492 shutdown_connections 2024-03-18T22:42:25.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.626+0000 7f0935446700 1 --2- 172.21.15.40:0/748646492 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0930101bb0 0x7f093010e050 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:25.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.626+0000 7f0935446700 1 --2- 172.21.15.40:0/748646492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09301072a0 0x7f0930107680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:25.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.626+0000 7f0935446700 1 --2- 172.21.15.40:0/748646492 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09301011f0 0x7f0930101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:25.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.626+0000 7f0935446700 1 -- 172.21.15.40:0/748646492 >> 172.21.15.40:0/748646492 conn(0x7f0930078220 msgr2=0x7f0930078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:25.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.626+0000 7f0935446700 1 -- 172.21.15.40:0/748646492 shutdown_connections 2024-03-18T22:42:25.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.626+0000 7f0935446700 1 -- 172.21.15.40:0/748646492 wait complete. 2024-03-18T22:42:25.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.626+0000 7f0935446700 1 Processor -- start 2024-03-18T22:42:25.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f0935446700 1 -- start start 2024-03-18T22:42:25.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f0935446700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09301011f0 0x7f093019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:25.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f0935446700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0930101bb0 0x7f093019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:25.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f0935446700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09301072a0 0x7f09301986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:25.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f0935446700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0930112480 con 0x7f09301072a0 2024-03-18T22:42:25.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f0935446700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0930112300 con 0x7f0930101bb0 2024-03-18T22:42:25.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f0935446700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0930112600 con 0x7f09301011f0 2024-03-18T22:42:25.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f092effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09301011f0 0x7f093019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:25.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f092f7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09301072a0 0x7f09301986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:25.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f092effd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09301011f0 0x7f093019e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:50692/0 (socket says 172.21.15.40:50692) 2024-03-18T22:42:25.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f092effd700 1 -- 172.21.15.40:0/90701214 learned_addr learned my addr 172.21.15.40:0/90701214 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:25.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f092e7fc700 1 --2- 172.21.15.40:0/90701214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0930101bb0 0x7f093019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:25.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f092effd700 1 -- 172.21.15.40:0/90701214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0930101bb0 msgr2=0x7f093019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:25.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f092effd700 1 --2- 172.21.15.40:0/90701214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0930101bb0 0x7f093019eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:25.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f092effd700 1 -- 172.21.15.40:0/90701214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09301072a0 msgr2=0x7f09301986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:25.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f092effd700 1 --2- 172.21.15.40:0/90701214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09301072a0 0x7f09301986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:25.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f092effd700 1 -- 172.21.15.40:0/90701214 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0930198fc0 con 0x7f09301011f0 2024-03-18T22:42:25.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.627+0000 7f092f7fe700 1 --2- 172.21.15.40:0/90701214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09301072a0 0x7f09301986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:25.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.628+0000 7f092effd700 1 --2- 172.21.15.40:0/90701214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09301011f0 0x7f093019e610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f092000ec10 tx=0x7f092000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:25.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.628+0000 7f0917fff700 1 -- 172.21.15.40:0/90701214 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092000cc90 con 0x7f09301011f0 2024-03-18T22:42:25.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.628+0000 7f0917fff700 1 -- 172.21.15.40:0/90701214 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f092000ce30 con 0x7f09301011f0 2024-03-18T22:42:25.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.628+0000 7f0935446700 1 -- 172.21.15.40:0/90701214 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09301992b0 con 0x7f09301011f0 2024-03-18T22:42:25.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.628+0000 7f0917fff700 1 -- 172.21.15.40:0/90701214 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920010430 con 0x7f09301011f0 2024-03-18T22:42:25.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.628+0000 7f0935446700 1 -- 172.21.15.40:0/90701214 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09301a5590 con 0x7f09301011f0 2024-03-18T22:42:25.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.629+0000 7f0935446700 1 -- 172.21.15.40:0/90701214 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09301028e0 con 0x7f09301011f0 2024-03-18T22:42:25.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.630+0000 7f0917fff700 1 -- 172.21.15.40:0/90701214 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0920004000 con 0x7f09301011f0 2024-03-18T22:42:25.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.630+0000 7f0917fff700 1 --2- 172.21.15.40:0/90701214 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09180a2890 0x7f09180a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:25.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.630+0000 7f092e7fc700 1 --2- 172.21.15.40:0/90701214 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09180a2890 0x7f09180a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:25.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.630+0000 7f0917fff700 1 -- 172.21.15.40:0/90701214 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f09200cf050 con 0x7f09301011f0 2024-03-18T22:42:25.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.631+0000 7f092e7fc700 1 --2- 172.21.15.40:0/90701214 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09180a2890 0x7f09180a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0924009680 tx=0x7f0924009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:25.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.635+0000 7f0917fff700 1 -- 172.21.15.40:0/90701214 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f092009bd90 con 0x7f09301011f0 2024-03-18T22:42:25.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.885+0000 7f0935446700 1 -- 172.21.15.40:0/90701214 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0930199c10 con 0x7f09301011f0 2024-03-18T22:42:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.885+0000 7f0917fff700 1 -- 172.21.15.40:0/90701214 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f092009b690 con 0x7f09301011f0 2024-03-18T22:42:25.886 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:25.886 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-18T22:42:25.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.887+0000 7f0935446700 1 -- 172.21.15.40:0/90701214 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09180a2890 msgr2=0x7f09180a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:25.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.887+0000 7f0935446700 1 --2- 172.21.15.40:0/90701214 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09180a2890 0x7f09180a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0924009680 tx=0x7f0924009260 comp rx=0 tx=0).stop 2024-03-18T22:42:25.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.887+0000 7f0935446700 1 -- 172.21.15.40:0/90701214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09301011f0 msgr2=0x7f093019e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:25.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.887+0000 7f0935446700 1 --2- 172.21.15.40:0/90701214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09301011f0 0x7f093019e610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f092000ec10 tx=0x7f092000ef20 comp rx=0 tx=0).stop 2024-03-18T22:42:25.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.887+0000 7f0935446700 1 -- 172.21.15.40:0/90701214 shutdown_connections 2024-03-18T22:42:25.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.887+0000 7f0935446700 1 --2- 172.21.15.40:0/90701214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09301072a0 0x7f09301986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:25.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.887+0000 7f0935446700 1 --2- 172.21.15.40:0/90701214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09301011f0 0x7f093019e610 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:25.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.887+0000 7f0935446700 1 --2- 172.21.15.40:0/90701214 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09180a2890 0x7f09180a4d50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:25.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.887+0000 7f0935446700 1 --2- 172.21.15.40:0/90701214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0930101bb0 0x7f093019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:25.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.887+0000 7f0935446700 1 -- 172.21.15.40:0/90701214 >> 172.21.15.40:0/90701214 conn(0x7f0930078220 msgr2=0x7f09300ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:25.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.888+0000 7f0935446700 1 -- 172.21.15.40:0/90701214 shutdown_connections 2024-03-18T22:42:25.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:25.888+0000 7f0935446700 1 -- 172.21.15.40:0/90701214 wait complete. 2024-03-18T22:42:26.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:25 smithi040 ceph-mon[62643]: pgmap v342: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T22:42:26.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:25 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/90701214' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T22:42:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:25 smithi113 ceph-mon[66775]: pgmap v342: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T22:42:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:25 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/90701214' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T22:42:26.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:25 smithi103 ceph-mon[66377]: pgmap v342: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T22:42:26.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:25 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/90701214' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T22:42:26.484 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-18T22:42:26.484 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:42:26.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.588+0000 7f1660737640 1 -- 172.21.15.40:0/4188041227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1658101840 msgr2=0x7f1658101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:26.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.588+0000 7f1660737640 1 --2- 172.21.15.40:0/4188041227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1658101840 0x7f1658101c60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1650009900 tx=0x7f165002bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:26.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.588+0000 7f1660737640 1 -- 172.21.15.40:0/4188041227 shutdown_connections 2024-03-18T22:42:26.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.588+0000 7f1660737640 1 --2- 172.21.15.40:0/4188041227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16581033f0 0x7f1658109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.588+0000 7f1660737640 1 --2- 172.21.15.40:0/4188041227 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1658102a30 0x7f1658102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.588+0000 7f1660737640 1 --2- 172.21.15.40:0/4188041227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1658101840 0x7f1658101c60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.588+0000 7f1660737640 1 -- 172.21.15.40:0/4188041227 >> 172.21.15.40:0/4188041227 conn(0x7f16580fcdd0 msgr2=0x7f16580ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:26.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.588+0000 7f1660737640 1 -- 172.21.15.40:0/4188041227 shutdown_connections 2024-03-18T22:42:26.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.588+0000 7f1660737640 1 -- 172.21.15.40:0/4188041227 wait complete. 2024-03-18T22:42:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f1660737640 1 Processor -- start 2024-03-18T22:42:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f1660737640 1 -- start start 2024-03-18T22:42:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f1660737640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1658101840 0x7f1658197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f1660737640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1658102a30 0x7f1658197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f1660737640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16581033f0 0x7f165819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f1660737640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f165810d360 con 0x7f16581033f0 2024-03-18T22:42:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f1660737640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f165810d1e0 con 0x7f1658101840 2024-03-18T22:42:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f1660737640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f165810d4e0 con 0x7f1658102a30 2024-03-18T22:42:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f165ecd4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16581033f0 0x7f165819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f165ecd4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16581033f0 0x7f165819f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32814/0 (socket says 172.21.15.40:32814) 2024-03-18T22:42:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f165ecd4640 1 -- 172.21.15.40:0/1472213703 learned_addr learned my addr 172.21.15.40:0/1472213703 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f165e4d3640 1 --2- 172.21.15.40:0/1472213703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1658101840 0x7f1658197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f165dcd2640 1 --2- 172.21.15.40:0/1472213703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1658102a30 0x7f1658197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f165ecd4640 1 -- 172.21.15.40:0/1472213703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1658102a30 msgr2=0x7f1658197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:26.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f165ecd4640 1 --2- 172.21.15.40:0/1472213703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1658102a30 0x7f1658197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f165ecd4640 1 -- 172.21.15.40:0/1472213703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1658101840 msgr2=0x7f1658197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f165ecd4640 1 --2- 172.21.15.40:0/1472213703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1658101840 0x7f1658197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.589+0000 7f165ecd4640 1 -- 172.21.15.40:0/1472213703 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165819f6b0 con 0x7f16581033f0 2024-03-18T22:42:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.590+0000 7f165ecd4640 1 --2- 172.21.15.40:0/1472213703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16581033f0 0x7f165819f040 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f165400eb10 tx=0x7f165400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.590+0000 7f164f7fe640 1 -- 172.21.15.40:0/1472213703 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1654004b20 con 0x7f16581033f0 2024-03-18T22:42:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.590+0000 7f164f7fe640 1 -- 172.21.15.40:0/1472213703 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1654004cc0 con 0x7f16581033f0 2024-03-18T22:42:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.590+0000 7f1660737640 1 -- 172.21.15.40:0/1472213703 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f165819f9a0 con 0x7f16581033f0 2024-03-18T22:42:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.590+0000 7f164f7fe640 1 -- 172.21.15.40:0/1472213703 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1654010690 con 0x7f16581033f0 2024-03-18T22:42:26.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.590+0000 7f1660737640 1 -- 172.21.15.40:0/1472213703 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f165819fef0 con 0x7f16581033f0 2024-03-18T22:42:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.591+0000 7f1660737640 1 -- 172.21.15.40:0/1472213703 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1658108be0 con 0x7f16581033f0 2024-03-18T22:42:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.592+0000 7f164f7fe640 1 -- 172.21.15.40:0/1472213703 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1654005020 con 0x7f16581033f0 2024-03-18T22:42:26.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.592+0000 7f164f7fe640 1 --2- 172.21.15.40:0/1472213703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f16440a2880 0x7f16440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.593+0000 7f165e4d3640 1 --2- 172.21.15.40:0/1472213703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f16440a2880 0x7f16440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.593+0000 7f164f7fe640 1 -- 172.21.15.40:0/1472213703 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f16540cf2d0 con 0x7f16581033f0 2024-03-18T22:42:26.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.593+0000 7f165e4d3640 1 --2- 172.21.15.40:0/1472213703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f16440a2880 0x7f16440a4d40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1650005ab0 tx=0x7f1650005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:26.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.597+0000 7f164f7fe640 1 -- 172.21.15.40:0/1472213703 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f165409c010 con 0x7f16581033f0 2024-03-18T22:42:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.780+0000 7f1660737640 1 -- 172.21.15.40:0/1472213703 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1658062420 con 0x7f16581033f0 2024-03-18T22:42:26.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.782+0000 7f164f7fe640 1 -- 172.21.15.40:0/1472213703 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24600 (secure 0 0 0) 0x7f165409b910 con 0x7f16581033f0 2024-03-18T22:42:26.783 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:26.783 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":91,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:41:43.993830+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:42:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.784+0000 7f1660737640 1 -- 172.21.15.40:0/1472213703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f16440a2880 msgr2=0x7f16440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.784+0000 7f1660737640 1 --2- 172.21.15.40:0/1472213703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f16440a2880 0x7f16440a4d40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1650005ab0 tx=0x7f1650005fd0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.784+0000 7f1660737640 1 -- 172.21.15.40:0/1472213703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16581033f0 msgr2=0x7f165819f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.784+0000 7f1660737640 1 --2- 172.21.15.40:0/1472213703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16581033f0 0x7f165819f040 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f165400eb10 tx=0x7f165400c2d0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.784+0000 7f1660737640 1 -- 172.21.15.40:0/1472213703 shutdown_connections 2024-03-18T22:42:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.784+0000 7f1660737640 1 --2- 172.21.15.40:0/1472213703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f16440a2880 0x7f16440a4d40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.784+0000 7f1660737640 1 --2- 172.21.15.40:0/1472213703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f16581033f0 0x7f165819f040 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.784+0000 7f1660737640 1 --2- 172.21.15.40:0/1472213703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1658102a30 0x7f1658197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.784+0000 7f1660737640 1 --2- 172.21.15.40:0/1472213703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1658101840 0x7f1658197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.784+0000 7f1660737640 1 -- 172.21.15.40:0/1472213703 >> 172.21.15.40:0/1472213703 conn(0x7f16580fcdd0 msgr2=0x7f1658105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.784+0000 7f1660737640 1 -- 172.21.15.40:0/1472213703 shutdown_connections 2024-03-18T22:42:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.784+0000 7f1660737640 1 -- 172.21.15.40:0/1472213703 wait complete. 2024-03-18T22:42:26.796 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T22:36:07.060691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T22:40:39.925147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T22:40:40.773757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T22:40:44.068183+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T22:42:26.797 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T22:42:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.904+0000 7f72e7976640 1 -- 172.21.15.40:0/2949840102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72e0101910 msgr2=0x7f72e0101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.904+0000 7f72e7976640 1 --2- 172.21.15.40:0/2949840102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72e0101910 0x7f72e0101d90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009900 tx=0x7f72d002bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.904+0000 7f72e7976640 1 -- 172.21.15.40:0/2949840102 shutdown_connections 2024-03-18T22:42:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.904+0000 7f72e7976640 1 --2- 172.21.15.40:0/2949840102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72e01022d0 0x7f72e0108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.904+0000 7f72e7976640 1 --2- 172.21.15.40:0/2949840102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72e0101910 0x7f72e0101d90 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.904+0000 7f72e7976640 1 --2- 172.21.15.40:0/2949840102 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72e0100720 0x7f72e0100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.904+0000 7f72e7976640 1 -- 172.21.15.40:0/2949840102 >> 172.21.15.40:0/2949840102 conn(0x7f72e00fbcb0 msgr2=0x7f72e00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.904+0000 7f72e7976640 1 -- 172.21.15.40:0/2949840102 shutdown_connections 2024-03-18T22:42:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.904+0000 7f72e7976640 1 -- 172.21.15.40:0/2949840102 wait complete. 2024-03-18T22:42:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.905+0000 7f72e7976640 1 Processor -- start 2024-03-18T22:42:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.905+0000 7f72e7976640 1 -- start start 2024-03-18T22:42:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.905+0000 7f72e7976640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72e0100720 0x7f72e0197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.905+0000 7f72e7976640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72e0101910 0x7f72e0198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.905+0000 7f72e7976640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72e01022d0 0x7f72e019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.905+0000 7f72e7976640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f72e0069280 con 0x7f72e01022d0 2024-03-18T22:42:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.905+0000 7f72e7976640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72e0069100 con 0x7f72e0100720 2024-03-18T22:42:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.905+0000 7f72e7976640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f72e0069400 con 0x7f72e0101910 2024-03-18T22:42:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.905+0000 7f72e5f13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72e01022d0 0x7f72e019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.905+0000 7f72e4f11640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72e0101910 0x7f72e0198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.905+0000 7f72e5f13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72e01022d0 0x7f72e019f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32828/0 (socket says 172.21.15.40:32828) 2024-03-18T22:42:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.905+0000 7f72e5f13640 1 -- 172.21.15.40:0/2495177899 learned_addr learned my addr 172.21.15.40:0/2495177899 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.905+0000 7f72e5712640 1 --2- 172.21.15.40:0/2495177899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72e0100720 0x7f72e0197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.906+0000 7f72e5f13640 1 -- 172.21.15.40:0/2495177899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72e0101910 msgr2=0x7f72e0198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.906+0000 7f72e5f13640 1 --2- 172.21.15.40:0/2495177899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72e0101910 0x7f72e0198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.906+0000 7f72e5f13640 1 -- 172.21.15.40:0/2495177899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72e0100720 msgr2=0x7f72e0197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.906+0000 7f72e5f13640 1 --2- 172.21.15.40:0/2495177899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72e0100720 0x7f72e0197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.906+0000 7f72e5f13640 1 -- 172.21.15.40:0/2495177899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e019f6f0 con 0x7f72e01022d0 2024-03-18T22:42:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.906+0000 7f72e5f13640 1 --2- 172.21.15.40:0/2495177899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72e01022d0 0x7f72e019f080 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f72d800eb10 tx=0x7f72d800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.907+0000 7f72d67fc640 1 -- 172.21.15.40:0/2495177899 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d8004b20 con 0x7f72e01022d0 2024-03-18T22:42:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.907+0000 7f72d67fc640 1 -- 172.21.15.40:0/2495177899 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72d8004cc0 con 0x7f72e01022d0 2024-03-18T22:42:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.907+0000 7f72d67fc640 1 -- 172.21.15.40:0/2495177899 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d8010690 con 0x7f72e01022d0 2024-03-18T22:42:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.907+0000 7f72e7976640 1 -- 172.21.15.40:0/2495177899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e019f9e0 con 0x7f72e01022d0 2024-03-18T22:42:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.907+0000 7f72e7976640 1 -- 172.21.15.40:0/2495177899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e019feb0 con 0x7f72e01022d0 2024-03-18T22:42:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.908+0000 7f72e7976640 1 -- 172.21.15.40:0/2495177899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72e0107b60 con 0x7f72e01022d0 2024-03-18T22:42:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.909+0000 7f72d67fc640 1 -- 172.21.15.40:0/2495177899 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f72d8005020 con 0x7f72e01022d0 2024-03-18T22:42:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.909+0000 7f72d67fc640 1 --2- 172.21.15.40:0/2495177899 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72cc0a2800 0x7f72cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.910+0000 7f72e5712640 1 --2- 172.21.15.40:0/2495177899 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72cc0a2800 0x7f72cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.910+0000 7f72d67fc640 1 -- 172.21.15.40:0/2495177899 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f72d80cf330 con 0x7f72e01022d0 2024-03-18T22:42:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.910+0000 7f72e5712640 1 --2- 172.21.15.40:0/2495177899 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72cc0a2800 0x7f72cc0a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f72e0101770 tx=0x7f72dc005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:26.914+0000 7f72d67fc640 1 -- 172.21.15.40:0/2495177899 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f72d809c070 con 0x7f72e01022d0 2024-03-18T22:42:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.096+0000 7f72e7976640 1 -- 172.21.15.40:0/2495177899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f72e0062420 con 0x7f72e01022d0 2024-03-18T22:42:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.097+0000 7f72d67fc640 1 -- 172.21.15.40:0/2495177899 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f72d809b970 con 0x7f72e01022d0 2024-03-18T22:42:27.097 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-18T22:42:27.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.098+0000 7f72e7976640 1 -- 172.21.15.40:0/2495177899 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72cc0a2800 msgr2=0x7f72cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:27.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.098+0000 7f72e7976640 1 --2- 172.21.15.40:0/2495177899 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72cc0a2800 0x7f72cc0a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f72e0101770 tx=0x7f72dc005f90 comp rx=0 tx=0).stop 2024-03-18T22:42:27.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.098+0000 7f72e7976640 1 -- 172.21.15.40:0/2495177899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72e01022d0 msgr2=0x7f72e019f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:27.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.098+0000 7f72e7976640 1 --2- 172.21.15.40:0/2495177899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72e01022d0 0x7f72e019f080 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f72d800eb10 tx=0x7f72d800c2d0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.098+0000 7f72e7976640 1 -- 172.21.15.40:0/2495177899 shutdown_connections 2024-03-18T22:42:27.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.098+0000 7f72e7976640 1 --2- 172.21.15.40:0/2495177899 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72cc0a2800 0x7f72cc0a4cc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.098+0000 7f72e7976640 1 --2- 172.21.15.40:0/2495177899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72e01022d0 0x7f72e019f080 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.098+0000 7f72e7976640 1 --2- 172.21.15.40:0/2495177899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72e0101910 0x7f72e0198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.098+0000 7f72e7976640 1 --2- 172.21.15.40:0/2495177899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72e0100720 0x7f72e0197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.098+0000 7f72e7976640 1 -- 172.21.15.40:0/2495177899 >> 172.21.15.40:0/2495177899 conn(0x7f72e00fbcb0 msgr2=0x7f72e0104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:27.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.099+0000 7f72e7976640 1 -- 172.21.15.40:0/2495177899 shutdown_connections 2024-03-18T22:42:27.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.099+0000 7f72e7976640 1 -- 172.21.15.40:0/2495177899 wait complete. 2024-03-18T22:42:27.109 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:26 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1472213703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:27.110 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T22:42:27.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.215+0000 7fd18189a640 1 -- 172.21.15.40:0/2311223192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd17c1022f0 msgr2=0x7fd17c108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:27.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.215+0000 7fd18189a640 1 --2- 172.21.15.40:0/2311223192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd17c1022f0 0x7fd17c108b80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009900 tx=0x7fd16c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:27.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.215+0000 7fd18189a640 1 -- 172.21.15.40:0/2311223192 shutdown_connections 2024-03-18T22:42:27.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.215+0000 7fd18189a640 1 --2- 172.21.15.40:0/2311223192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd17c1022f0 0x7fd17c108b80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.215+0000 7fd18189a640 1 --2- 172.21.15.40:0/2311223192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd17c101930 0x7fd17c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.215+0000 7fd18189a640 1 --2- 172.21.15.40:0/2311223192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd17c100740 0x7fd17c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.215+0000 7fd18189a640 1 -- 172.21.15.40:0/2311223192 >> 172.21.15.40:0/2311223192 conn(0x7fd17c0fbcd0 msgr2=0x7fd17c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:27.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.215+0000 7fd18189a640 1 -- 172.21.15.40:0/2311223192 shutdown_connections 2024-03-18T22:42:27.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.215+0000 7fd18189a640 1 -- 172.21.15.40:0/2311223192 wait complete. 2024-03-18T22:42:27.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.216+0000 7fd18189a640 1 Processor -- start 2024-03-18T22:42:27.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.216+0000 7fd18189a640 1 -- start start 2024-03-18T22:42:27.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.216+0000 7fd18189a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd17c100740 0x7fd17c1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:27.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.216+0000 7fd18189a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd17c101930 0x7fd17c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.216+0000 7fd18189a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd17c1022f0 0x7fd17c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.216+0000 7fd18189a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd17c10d390 con 0x7fd17c1022f0 2024-03-18T22:42:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.216+0000 7fd18189a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd17c10d210 con 0x7fd17c100740 2024-03-18T22:42:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.216+0000 7fd18189a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd17c10d510 con 0x7fd17c101930 2024-03-18T22:42:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.216+0000 7fd17b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd17c1022f0 0x7fd17c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.216+0000 7fd17b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd17c1022f0 0x7fd17c19efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32854/0 (socket says 172.21.15.40:32854) 2024-03-18T22:42:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.216+0000 7fd17b7fe640 1 -- 172.21.15.40:0/3405982604 learned_addr learned my addr 172.21.15.40:0/3405982604 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.216+0000 7fd17affd640 1 --2- 172.21.15.40:0/3405982604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd17c100740 0x7fd17c1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.216+0000 7fd17a7fc640 1 --2- 172.21.15.40:0/3405982604 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd17c101930 0x7fd17c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.217+0000 7fd17b7fe640 1 -- 172.21.15.40:0/3405982604 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd17c101930 msgr2=0x7fd17c197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.217+0000 7fd17b7fe640 1 --2- 172.21.15.40:0/3405982604 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd17c101930 0x7fd17c197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.217+0000 7fd17b7fe640 1 -- 172.21.15.40:0/3405982604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd17c100740 msgr2=0x7fd17c1979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:27.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.217+0000 7fd17b7fe640 1 --2- 172.21.15.40:0/3405982604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd17c100740 0x7fd17c1979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.217+0000 7fd17b7fe640 1 -- 172.21.15.40:0/3405982604 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17c19f610 con 0x7fd17c1022f0 2024-03-18T22:42:27.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.217+0000 7fd17b7fe640 1 --2- 172.21.15.40:0/3405982604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd17c1022f0 0x7fd17c19efa0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009230 tx=0x7fd16c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:27.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.217+0000 7fd180898640 1 -- 172.21.15.40:0/3405982604 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c043020 con 0x7fd17c1022f0 2024-03-18T22:42:27.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.217+0000 7fd18189a640 1 -- 172.21.15.40:0/3405982604 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd17c19f8a0 con 0x7fd17c1022f0 2024-03-18T22:42:27.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.217+0000 7fd180898640 1 -- 172.21.15.40:0/3405982604 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd16c0043d0 con 0x7fd17c1022f0 2024-03-18T22:42:27.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.217+0000 7fd180898640 1 -- 172.21.15.40:0/3405982604 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c003b40 con 0x7fd17c1022f0 2024-03-18T22:42:27.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.217+0000 7fd18189a640 1 -- 172.21.15.40:0/3405982604 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd17c19fe40 con 0x7fd17c1022f0 2024-03-18T22:42:27.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.218+0000 7fd18189a640 1 -- 172.21.15.40:0/3405982604 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd17c107a60 con 0x7fd17c1022f0 2024-03-18T22:42:27.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.220+0000 7fd180898640 1 -- 172.21.15.40:0/3405982604 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd16c046020 con 0x7fd17c1022f0 2024-03-18T22:42:27.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.220+0000 7fd180898640 1 --2- 172.21.15.40:0/3405982604 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1680a2800 0x7fd1680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:27.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.220+0000 7fd17affd640 1 --2- 172.21.15.40:0/3405982604 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1680a2800 0x7fd1680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:27.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.220+0000 7fd180898640 1 -- 172.21.15.40:0/3405982604 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd16c0f1380 con 0x7fd17c1022f0 2024-03-18T22:42:27.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.221+0000 7fd17affd640 1 --2- 172.21.15.40:0/3405982604 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1680a2800 0x7fd1680a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd17c101790 tx=0x7fd174009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:27.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.224+0000 7fd180898640 1 -- 172.21.15.40:0/3405982604 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd16c04d020 con 0x7fd17c1022f0 2024-03-18T22:42:27.259 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:26 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1472213703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:27.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:26 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1472213703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:27.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.409+0000 7fd18189a640 1 -- 172.21.15.40:0/3405982604 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd17c02d1f0 con 0x7fd17c1022f0 2024-03-18T22:42:27.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.409+0000 7fd180898640 1 -- 172.21.15.40:0/3405982604 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fd16c0be0c0 con 0x7fd17c1022f0 2024-03-18T22:42:27.410 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:42:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.411+0000 7fd18189a640 1 -- 172.21.15.40:0/3405982604 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1680a2800 msgr2=0x7fd1680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.411+0000 7fd18189a640 1 --2- 172.21.15.40:0/3405982604 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1680a2800 0x7fd1680a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd17c101790 tx=0x7fd174009500 comp rx=0 tx=0).stop 2024-03-18T22:42:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.411+0000 7fd18189a640 1 -- 172.21.15.40:0/3405982604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd17c1022f0 msgr2=0x7fd17c19efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.411+0000 7fd18189a640 1 --2- 172.21.15.40:0/3405982604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd17c1022f0 0x7fd17c19efa0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009230 tx=0x7fd16c005e40 comp rx=0 tx=0).stop 2024-03-18T22:42:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.411+0000 7fd18189a640 1 -- 172.21.15.40:0/3405982604 shutdown_connections 2024-03-18T22:42:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.411+0000 7fd18189a640 1 --2- 172.21.15.40:0/3405982604 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1680a2800 0x7fd1680a4cc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.411+0000 7fd18189a640 1 --2- 172.21.15.40:0/3405982604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd17c1022f0 0x7fd17c19efa0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.411+0000 7fd18189a640 1 --2- 172.21.15.40:0/3405982604 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd17c101930 0x7fd17c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.411+0000 7fd18189a640 1 --2- 172.21.15.40:0/3405982604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd17c100740 0x7fd17c1979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.411+0000 7fd18189a640 1 -- 172.21.15.40:0/3405982604 >> 172.21.15.40:0/3405982604 conn(0x7fd17c0fbcd0 msgr2=0x7fd17c104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.411+0000 7fd18189a640 1 -- 172.21.15.40:0/3405982604 shutdown_connections 2024-03-18T22:42:27.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.411+0000 7fd18189a640 1 -- 172.21.15.40:0/3405982604 wait complete. 2024-03-18T22:42:27.423 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T22:42:27.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.527+0000 7f3d280bf640 1 -- 172.21.15.40:0/2037919239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d20101930 msgr2=0x7f3d20101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.527+0000 7f3d280bf640 1 --2- 172.21.15.40:0/2037919239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d20101930 0x7f3d20101db0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009860 tx=0x7f3d1002bb70 comp rx=0 tx=0).stop 2024-03-18T22:42:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.527+0000 7f3d280bf640 1 -- 172.21.15.40:0/2037919239 shutdown_connections 2024-03-18T22:42:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.527+0000 7f3d280bf640 1 --2- 172.21.15.40:0/2037919239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d201022f0 0x7f3d20108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.527+0000 7f3d280bf640 1 --2- 172.21.15.40:0/2037919239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d20101930 0x7f3d20101db0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.527+0000 7f3d280bf640 1 --2- 172.21.15.40:0/2037919239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d20100740 0x7f3d20100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d280bf640 1 -- 172.21.15.40:0/2037919239 >> 172.21.15.40:0/2037919239 conn(0x7f3d200fbcd0 msgr2=0x7f3d200fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d280bf640 1 -- 172.21.15.40:0/2037919239 shutdown_connections 2024-03-18T22:42:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d280bf640 1 -- 172.21.15.40:0/2037919239 wait complete. 2024-03-18T22:42:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d280bf640 1 Processor -- start 2024-03-18T22:42:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d280bf640 1 -- start start 2024-03-18T22:42:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d280bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d20100740 0x7f3d20197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d280bf640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d20101930 0x7f3d20197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d280bf640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d201022f0 0x7f3d2019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:27.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d280bf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2010d390 con 0x7f3d20100740 2024-03-18T22:42:27.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d280bf640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2010d210 con 0x7f3d201022f0 2024-03-18T22:42:27.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d280bf640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2010d510 con 0x7f3d20101930 2024-03-18T22:42:27.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d25e5b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d20100740 0x7f3d20197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:27.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d25e5b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d20100740 0x7f3d20197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32872/0 (socket says 172.21.15.40:32872) 2024-03-18T22:42:27.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d25e5b640 1 -- 172.21.15.40:0/648045209 learned_addr learned my addr 172.21.15.40:0/648045209 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:27.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d2565a640 1 --2- 172.21.15.40:0/648045209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d20101930 0x7f3d20197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:27.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.528+0000 7f3d2665c640 1 --2- 172.21.15.40:0/648045209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d201022f0 0x7f3d2019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:27.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.529+0000 7f3d25e5b640 1 -- 172.21.15.40:0/648045209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d20101930 msgr2=0x7f3d20197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:27.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.529+0000 7f3d25e5b640 1 --2- 172.21.15.40:0/648045209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d20101930 0x7f3d20197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.529+0000 7f3d25e5b640 1 -- 172.21.15.40:0/648045209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d201022f0 msgr2=0x7f3d2019f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:27.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.529+0000 7f3d25e5b640 1 --2- 172.21.15.40:0/648045209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d201022f0 0x7f3d2019f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.529+0000 7f3d25e5b640 1 -- 172.21.15.40:0/648045209 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2019f6b0 con 0x7f3d20100740 2024-03-18T22:42:27.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.529+0000 7f3d2665c640 1 --2- 172.21.15.40:0/648045209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d201022f0 0x7f3d2019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:42:27.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.529+0000 7f3d25e5b640 1 --2- 172.21.15.40:0/648045209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d20100740 0x7f3d20197a90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00e9a0 tx=0x7f3d1c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:27.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.529+0000 7f3d16ffd640 1 -- 172.21.15.40:0/648045209 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c00cd80 con 0x7f3d20100740 2024-03-18T22:42:27.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.529+0000 7f3d280bf640 1 -- 172.21.15.40:0/648045209 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d2019f9a0 con 0x7f3d20100740 2024-03-18T22:42:27.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.529+0000 7f3d16ffd640 1 -- 172.21.15.40:0/648045209 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d1c0108e0 con 0x7f3d20100740 2024-03-18T22:42:27.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.529+0000 7f3d16ffd640 1 -- 172.21.15.40:0/648045209 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0188a0 con 0x7f3d20100740 2024-03-18T22:42:27.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.529+0000 7f3d280bf640 1 -- 172.21.15.40:0/648045209 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d2019ff10 con 0x7f3d20100740 2024-03-18T22:42:27.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.530+0000 7f3d280bf640 1 -- 172.21.15.40:0/648045209 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d20107b80 con 0x7f3d20100740 2024-03-18T22:42:27.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.531+0000 7f3d16ffd640 1 -- 172.21.15.40:0/648045209 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3d1c010a90 con 0x7f3d20100740 2024-03-18T22:42:27.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.532+0000 7f3d16ffd640 1 --2- 172.21.15.40:0/648045209 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3d0c0a2800 0x7f3d0c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:27.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.532+0000 7f3d16ffd640 1 -- 172.21.15.40:0/648045209 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3d1c0cf1f0 con 0x7f3d20100740 2024-03-18T22:42:27.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.532+0000 7f3d2565a640 1 --2- 172.21.15.40:0/648045209 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3d0c0a2800 0x7f3d0c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:27.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.532+0000 7f3d2565a640 1 --2- 172.21.15.40:0/648045209 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3d0c0a2800 0x7f3d0c0a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3d10000c00 tx=0x7f3d100349a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:27.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.536+0000 7f3d16ffd640 1 -- 172.21.15.40:0/648045209 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3d1c053740 con 0x7f3d20100740 2024-03-18T22:42:27.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.718+0000 7f3d280bf640 1 -- 172.21.15.40:0/648045209 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3d20062420 con 0x7f3d20100740 2024-03-18T22:42:27.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.718+0000 7f3d16ffd640 1 -- 172.21.15.40:0/648045209 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f3d1c09bf30 con 0x7f3d20100740 2024-03-18T22:42:27.719 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-18T22:42:27.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.720+0000 7f3d280bf640 1 -- 172.21.15.40:0/648045209 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3d0c0a2800 msgr2=0x7f3d0c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:27.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.720+0000 7f3d280bf640 1 --2- 172.21.15.40:0/648045209 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3d0c0a2800 0x7f3d0c0a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3d10000c00 tx=0x7f3d100349a0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.720+0000 7f3d280bf640 1 -- 172.21.15.40:0/648045209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d20100740 msgr2=0x7f3d20197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:27.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.720+0000 7f3d280bf640 1 --2- 172.21.15.40:0/648045209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d20100740 0x7f3d20197a90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00e9a0 tx=0x7f3d1c00ee60 comp rx=0 tx=0).stop 2024-03-18T22:42:27.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.720+0000 7f3d280bf640 1 -- 172.21.15.40:0/648045209 shutdown_connections 2024-03-18T22:42:27.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.720+0000 7f3d280bf640 1 --2- 172.21.15.40:0/648045209 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3d0c0a2800 0x7f3d0c0a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.720+0000 7f3d280bf640 1 --2- 172.21.15.40:0/648045209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d201022f0 0x7f3d2019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.720+0000 7f3d280bf640 1 --2- 172.21.15.40:0/648045209 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3d20101930 0x7f3d20197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.720+0000 7f3d280bf640 1 --2- 172.21.15.40:0/648045209 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d20100740 0x7f3d20197a90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.720+0000 7f3d280bf640 1 -- 172.21.15.40:0/648045209 >> 172.21.15.40:0/648045209 conn(0x7f3d200fbcd0 msgr2=0x7f3d20104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:27.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.720+0000 7f3d280bf640 1 -- 172.21.15.40:0/648045209 shutdown_connections 2024-03-18T22:42:27.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.720+0000 7f3d280bf640 1 -- 172.21.15.40:0/648045209 wait complete. 2024-03-18T22:42:27.732 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T22:42:27.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.837+0000 7f6a496e8640 1 -- 172.21.15.40:0/1529575459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a441033e0 msgr2=0x7f6a44109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:27.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.837+0000 7f6a496e8640 1 --2- 172.21.15.40:0/1529575459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a441033e0 0x7f6a44109c70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009ca0 tx=0x7f6a3c02c0f0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.837+0000 7f6a496e8640 1 -- 172.21.15.40:0/1529575459 shutdown_connections 2024-03-18T22:42:27.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.837+0000 7f6a496e8640 1 --2- 172.21.15.40:0/1529575459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a441033e0 0x7f6a44109c70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.837+0000 7f6a496e8640 1 --2- 172.21.15.40:0/1529575459 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a44102a20 0x7f6a44102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.837+0000 7f6a496e8640 1 --2- 172.21.15.40:0/1529575459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a44101830 0x7f6a44101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.837+0000 7f6a496e8640 1 -- 172.21.15.40:0/1529575459 >> 172.21.15.40:0/1529575459 conn(0x7f6a440fcd80 msgr2=0x7f6a440ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:27.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.837+0000 7f6a496e8640 1 -- 172.21.15.40:0/1529575459 shutdown_connections 2024-03-18T22:42:27.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.837+0000 7f6a496e8640 1 -- 172.21.15.40:0/1529575459 wait complete. 2024-03-18T22:42:27.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a496e8640 1 Processor -- start 2024-03-18T22:42:27.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a496e8640 1 -- start start 2024-03-18T22:42:27.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a496e8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a44101830 0x7f6a44197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:27.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a496e8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a44102a20 0x7f6a44197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:27.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a496e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a441033e0 0x7f6a4419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:27.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a496e8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4410d340 con 0x7f6a441033e0 2024-03-18T22:42:27.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a496e8640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4410d1c0 con 0x7f6a44101830 2024-03-18T22:42:27.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a496e8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4410d4c0 con 0x7f6a44102a20 2024-03-18T22:42:27.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a437fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a441033e0 0x7f6a4419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:27.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a437fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a441033e0 0x7f6a4419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32892/0 (socket says 172.21.15.40:32892) 2024-03-18T22:42:27.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a437fe640 1 -- 172.21.15.40:0/2957532378 learned_addr learned my addr 172.21.15.40:0/2957532378 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:27.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a42ffd640 1 --2- 172.21.15.40:0/2957532378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a44101830 0x7f6a44197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:27.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a3a7fc640 1 --2- 172.21.15.40:0/2957532378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a44102a20 0x7f6a44197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:27.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a437fe640 1 -- 172.21.15.40:0/2957532378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a44102a20 msgr2=0x7f6a44197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:27.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a437fe640 1 --2- 172.21.15.40:0/2957532378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a44102a20 0x7f6a44197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a437fe640 1 -- 172.21.15.40:0/2957532378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a44101830 msgr2=0x7f6a44197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:27.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a437fe640 1 --2- 172.21.15.40:0/2957532378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a44101830 0x7f6a44197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:27.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a437fe640 1 -- 172.21.15.40:0/2957532378 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4419f660 con 0x7f6a441033e0 2024-03-18T22:42:27.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.838+0000 7f6a437fe640 1 --2- 172.21.15.40:0/2957532378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a441033e0 0x7f6a4419eff0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c032380 tx=0x7f6a3c032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:27.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.839+0000 7f6a40ff9640 1 -- 172.21.15.40:0/2957532378 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c041020 con 0x7f6a441033e0 2024-03-18T22:42:27.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.839+0000 7f6a40ff9640 1 -- 172.21.15.40:0/2957532378 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a3c003e20 con 0x7f6a441033e0 2024-03-18T22:42:27.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.839+0000 7f6a496e8640 1 -- 172.21.15.40:0/2957532378 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a4419f8f0 con 0x7f6a441033e0 2024-03-18T22:42:27.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.839+0000 7f6a40ff9640 1 -- 172.21.15.40:0/2957532378 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c039040 con 0x7f6a441033e0 2024-03-18T22:42:27.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.839+0000 7f6a496e8640 1 -- 172.21.15.40:0/2957532378 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a4419fde0 con 0x7f6a441033e0 2024-03-18T22:42:27.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.840+0000 7f6a496e8640 1 -- 172.21.15.40:0/2957532378 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a44108c20 con 0x7f6a441033e0 2024-03-18T22:42:27.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.841+0000 7f6a40ff9640 1 -- 172.21.15.40:0/2957532378 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6a3c044020 con 0x7f6a441033e0 2024-03-18T22:42:27.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.841+0000 7f6a40ff9640 1 --2- 172.21.15.40:0/2957532378 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a300a2800 0x7f6a300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:27.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.841+0000 7f6a42ffd640 1 --2- 172.21.15.40:0/2957532378 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a300a2800 0x7f6a300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:27.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.842+0000 7f6a40ff9640 1 -- 172.21.15.40:0/2957532378 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6a3c0f0eb0 con 0x7f6a441033e0 2024-03-18T22:42:27.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.842+0000 7f6a42ffd640 1 --2- 172.21.15.40:0/2957532378 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a300a2800 0x7f6a300a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6a44102880 tx=0x7f6a34005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:27.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:27.846+0000 7f6a40ff9640 1 -- 172.21.15.40:0/2957532378 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6a3c04b020 con 0x7f6a441033e0 2024-03-18T22:42:28.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.030+0000 7f6a496e8640 1 -- 172.21.15.40:0/2957532378 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6a44062420 con 0x7f6a441033e0 2024-03-18T22:42:28.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.030+0000 7f6a40ff9640 1 -- 172.21.15.40:0/2957532378 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f6a3c0bdb90 con 0x7f6a441033e0 2024-03-18T22:42:28.031 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:42:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.032+0000 7f6a496e8640 1 -- 172.21.15.40:0/2957532378 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a300a2800 msgr2=0x7f6a300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.032+0000 7f6a496e8640 1 --2- 172.21.15.40:0/2957532378 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a300a2800 0x7f6a300a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6a44102880 tx=0x7f6a34005f90 comp rx=0 tx=0).stop 2024-03-18T22:42:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.032+0000 7f6a496e8640 1 -- 172.21.15.40:0/2957532378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a441033e0 msgr2=0x7f6a4419eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.032+0000 7f6a496e8640 1 --2- 172.21.15.40:0/2957532378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a441033e0 0x7f6a4419eff0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c032380 tx=0x7f6a3c032820 comp rx=0 tx=0).stop 2024-03-18T22:42:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.032+0000 7f6a496e8640 1 -- 172.21.15.40:0/2957532378 shutdown_connections 2024-03-18T22:42:28.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.032+0000 7f6a496e8640 1 --2- 172.21.15.40:0/2957532378 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a300a2800 0x7f6a300a4cc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.032+0000 7f6a496e8640 1 --2- 172.21.15.40:0/2957532378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a441033e0 0x7f6a4419eff0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.032+0000 7f6a496e8640 1 --2- 172.21.15.40:0/2957532378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a44102a20 0x7f6a44197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.032+0000 7f6a496e8640 1 --2- 172.21.15.40:0/2957532378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a44101830 0x7f6a44197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.032+0000 7f6a496e8640 1 -- 172.21.15.40:0/2957532378 >> 172.21.15.40:0/2957532378 conn(0x7f6a440fcd80 msgr2=0x7f6a44105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.033+0000 7f6a496e8640 1 -- 172.21.15.40:0/2957532378 shutdown_connections 2024-03-18T22:42:28.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.033+0000 7f6a496e8640 1 -- 172.21.15.40:0/2957532378 wait complete. 2024-03-18T22:42:28.043 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:27 smithi040 ceph-mon[62643]: pgmap v343: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-18T22:42:28.043 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:27 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2495177899' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:42:28.043 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:27 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3405982604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:42:28.044 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:27 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/648045209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:42:28.045 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:42:28.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 -- 172.21.15.40:0/2589089910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f220c102a30 msgr2=0x7f220c102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:28.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 --2- 172.21.15.40:0/2589089910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f220c102a30 0x7f220c102eb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2200009900 tx=0x7f220002bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:28.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 -- 172.21.15.40:0/2589089910 shutdown_connections 2024-03-18T22:42:28.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 --2- 172.21.15.40:0/2589089910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f220c1033f0 0x7f220c109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 --2- 172.21.15.40:0/2589089910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f220c102a30 0x7f220c102eb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 --2- 172.21.15.40:0/2589089910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f220c101840 0x7f220c101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 -- 172.21.15.40:0/2589089910 >> 172.21.15.40:0/2589089910 conn(0x7f220c0fcdd0 msgr2=0x7f220c0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 -- 172.21.15.40:0/2589089910 shutdown_connections 2024-03-18T22:42:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 -- 172.21.15.40:0/2589089910 wait complete. 2024-03-18T22:42:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 Processor -- start 2024-03-18T22:42:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 -- start start 2024-03-18T22:42:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f220c101840 0x7f220c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f220c102a30 0x7f220c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f220c1033f0 0x7f220c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f220c10d350 con 0x7f220c1033f0 2024-03-18T22:42:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f220c10d1d0 con 0x7f220c101840 2024-03-18T22:42:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.147+0000 7f2211668640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f220c10d4d0 con 0x7f220c102a30 2024-03-18T22:42:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f220b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f220c1033f0 0x7f220c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:28.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f220affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f220c101840 0x7f220c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:28.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f220b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f220c1033f0 0x7f220c19f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32918/0 (socket says 172.21.15.40:32918) 2024-03-18T22:42:28.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f220affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f220c101840 0x7f220c197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:57480/0 (socket says 172.21.15.40:57480) 2024-03-18T22:42:28.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f220b7fe640 1 -- 172.21.15.40:0/1320784680 learned_addr learned my addr 172.21.15.40:0/1320784680 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:28.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f220a7fc640 1 --2- 172.21.15.40:0/1320784680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f220c102a30 0x7f220c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:28.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f220b7fe640 1 -- 172.21.15.40:0/1320784680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f220c102a30 msgr2=0x7f220c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:28.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f220b7fe640 1 --2- 172.21.15.40:0/1320784680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f220c102a30 0x7f220c197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f220b7fe640 1 -- 172.21.15.40:0/1320784680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f220c101840 msgr2=0x7f220c197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:28.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f220b7fe640 1 --2- 172.21.15.40:0/1320784680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f220c101840 0x7f220c197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f220b7fe640 1 -- 172.21.15.40:0/1320784680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220c19f670 con 0x7f220c1033f0 2024-03-18T22:42:28.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f220b7fe640 1 --2- 172.21.15.40:0/1320784680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f220c1033f0 0x7f220c19f000 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f220400eb10 tx=0x7f220400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:28.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f21f3fff640 1 -- 172.21.15.40:0/1320784680 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204004b20 con 0x7f220c1033f0 2024-03-18T22:42:28.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f21f3fff640 1 -- 172.21.15.40:0/1320784680 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2204004cc0 con 0x7f220c1033f0 2024-03-18T22:42:28.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f2211668640 1 -- 172.21.15.40:0/1320784680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f220c19f960 con 0x7f220c1033f0 2024-03-18T22:42:28.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f21f3fff640 1 -- 172.21.15.40:0/1320784680 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204010690 con 0x7f220c1033f0 2024-03-18T22:42:28.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.148+0000 7f2211668640 1 -- 172.21.15.40:0/1320784680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f220c19fe30 con 0x7f220c1033f0 2024-03-18T22:42:28.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.149+0000 7f2211668640 1 -- 172.21.15.40:0/1320784680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f220c108c30 con 0x7f220c1033f0 2024-03-18T22:42:28.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.150+0000 7f21f3fff640 1 -- 172.21.15.40:0/1320784680 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2204005020 con 0x7f220c1033f0 2024-03-18T22:42:28.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.151+0000 7f21f3fff640 1 --2- 172.21.15.40:0/1320784680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21f40a2800 0x7f21f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:28.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.151+0000 7f220affd640 1 --2- 172.21.15.40:0/1320784680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21f40a2800 0x7f21f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:28.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.151+0000 7f21f3fff640 1 -- 172.21.15.40:0/1320784680 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f22040cf3b0 con 0x7f220c1033f0 2024-03-18T22:42:28.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.151+0000 7f220affd640 1 --2- 172.21.15.40:0/1320784680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21f40a2800 0x7f21f40a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f220c102890 tx=0x7f21fc005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:28.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.155+0000 7f21f3fff640 1 -- 172.21.15.40:0/1320784680 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f220409c090 con 0x7f220c1033f0 2024-03-18T22:42:28.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:27 smithi113 ceph-mon[66775]: pgmap v343: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-18T22:42:28.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:27 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2495177899' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:42:28.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:27 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3405982604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:42:28.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:27 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/648045209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:42:28.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:27 smithi103 ceph-mon[66377]: pgmap v343: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-18T22:42:28.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:27 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2495177899' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:42:28.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:27 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3405982604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:42:28.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:27 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/648045209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:42:28.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.377+0000 7f2211668640 1 -- 172.21.15.40:0/1320784680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f220c02d1f0 con 0x7f220c1033f0 2024-03-18T22:42:28.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.378+0000 7f21f3fff640 1 -- 172.21.15.40:0/1320784680 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7f220409b990 con 0x7f220c1033f0 2024-03-18T22:42:28.379 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:28.379 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:41:09.839105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24481,24509,24473,24401,24388]},"id":1}]} 2024-03-18T22:42:28.379 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 27 2024-03-18T22:42:28.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.380+0000 7f2211668640 1 -- 172.21.15.40:0/1320784680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21f40a2800 msgr2=0x7f21f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:28.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.380+0000 7f2211668640 1 --2- 172.21.15.40:0/1320784680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21f40a2800 0x7f21f40a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f220c102890 tx=0x7f21fc005f90 comp rx=0 tx=0).stop 2024-03-18T22:42:28.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.380+0000 7f2211668640 1 -- 172.21.15.40:0/1320784680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f220c1033f0 msgr2=0x7f220c19f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:28.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.380+0000 7f2211668640 1 --2- 172.21.15.40:0/1320784680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f220c1033f0 0x7f220c19f000 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f220400eb10 tx=0x7f220400c2d0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.380+0000 7f2211668640 1 -- 172.21.15.40:0/1320784680 shutdown_connections 2024-03-18T22:42:28.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.380+0000 7f2211668640 1 --2- 172.21.15.40:0/1320784680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21f40a2800 0x7f21f40a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.380+0000 7f2211668640 1 --2- 172.21.15.40:0/1320784680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f220c1033f0 0x7f220c19f000 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.380+0000 7f2211668640 1 --2- 172.21.15.40:0/1320784680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f220c102a30 0x7f220c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.380+0000 7f2211668640 1 --2- 172.21.15.40:0/1320784680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f220c101840 0x7f220c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.380+0000 7f2211668640 1 -- 172.21.15.40:0/1320784680 >> 172.21.15.40:0/1320784680 conn(0x7f220c0fcdd0 msgr2=0x7f220c105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:28.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.380+0000 7f2211668640 1 -- 172.21.15.40:0/1320784680 shutdown_connections 2024-03-18T22:42:28.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.380+0000 7f2211668640 1 -- 172.21.15.40:0/1320784680 wait complete. 2024-03-18T22:42:28.392 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:42:28.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.494+0000 7fd60a6d3640 1 -- 172.21.15.40:0/2835994967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd604101840 msgr2=0x7fd604101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:28.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.494+0000 7fd60a6d3640 1 --2- 172.21.15.40:0/2835994967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd604101840 0x7fd604101c60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4009860 tx=0x7fd5f402bb70 comp rx=0 tx=0).stop 2024-03-18T22:42:28.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.494+0000 7fd60a6d3640 1 -- 172.21.15.40:0/2835994967 shutdown_connections 2024-03-18T22:42:28.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.494+0000 7fd60a6d3640 1 --2- 172.21.15.40:0/2835994967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6041033f0 0x7fd604109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.494+0000 7fd60a6d3640 1 --2- 172.21.15.40:0/2835994967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd604102a30 0x7fd604102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.494+0000 7fd60a6d3640 1 --2- 172.21.15.40:0/2835994967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd604101840 0x7fd604101c60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.494+0000 7fd60a6d3640 1 -- 172.21.15.40:0/2835994967 >> 172.21.15.40:0/2835994967 conn(0x7fd6040fcdd0 msgr2=0x7fd6040ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:28.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.494+0000 7fd60a6d3640 1 -- 172.21.15.40:0/2835994967 shutdown_connections 2024-03-18T22:42:28.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.494+0000 7fd60a6d3640 1 -- 172.21.15.40:0/2835994967 wait complete. 2024-03-18T22:42:28.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd60a6d3640 1 Processor -- start 2024-03-18T22:42:28.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd60a6d3640 1 -- start start 2024-03-18T22:42:28.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd60a6d3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd604101840 0x7fd604197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:28.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd60a6d3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd604102a30 0x7fd604197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:28.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd60a6d3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6041033f0 0x7fd60419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:28.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd60a6d3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd60410d340 con 0x7fd604101840 2024-03-18T22:42:28.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd60a6d3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd60410d1c0 con 0x7fd604102a30 2024-03-18T22:42:28.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd60a6d3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd60410d4c0 con 0x7fd6041033f0 2024-03-18T22:42:28.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd603fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd604101840 0x7fd604197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd608c70640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6041033f0 0x7fd60419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd603fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd604101840 0x7fd604197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32936/0 (socket says 172.21.15.40:32936) 2024-03-18T22:42:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd608c70640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6041033f0 0x7fd60419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:50812/0 (socket says 172.21.15.40:50812) 2024-03-18T22:42:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd603fff640 1 -- 172.21.15.40:0/3208964147 learned_addr learned my addr 172.21.15.40:0/3208964147 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd6037fe640 1 --2- 172.21.15.40:0/3208964147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd604102a30 0x7fd604197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd608c70640 1 -- 172.21.15.40:0/3208964147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd604102a30 msgr2=0x7fd604197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd608c70640 1 --2- 172.21.15.40:0/3208964147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd604102a30 0x7fd604197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd608c70640 1 -- 172.21.15.40:0/3208964147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd604101840 msgr2=0x7fd604197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd608c70640 1 --2- 172.21.15.40:0/3208964147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd604101840 0x7fd604197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd608c70640 1 -- 172.21.15.40:0/3208964147 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60419f6d0 con 0x7fd6041033f0 2024-03-18T22:42:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.495+0000 7fd603fff640 1 --2- 172.21.15.40:0/3208964147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd604101840 0x7fd604197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.496+0000 7fd608c70640 1 --2- 172.21.15.40:0/3208964147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6041033f0 0x7fd60419f060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00ec50 tx=0x7fd5fc00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.496+0000 7fd6017fa640 1 -- 172.21.15.40:0/3208964147 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc003b70 con 0x7fd6041033f0 2024-03-18T22:42:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.496+0000 7fd6017fa640 1 -- 172.21.15.40:0/3208964147 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5fc00ee90 con 0x7fd6041033f0 2024-03-18T22:42:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.496+0000 7fd60a6d3640 1 -- 172.21.15.40:0/3208964147 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd60419f9c0 con 0x7fd6041033f0 2024-03-18T22:42:28.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.496+0000 7fd6017fa640 1 -- 172.21.15.40:0/3208964147 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc010670 con 0x7fd6041033f0 2024-03-18T22:42:28.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.496+0000 7fd60a6d3640 1 -- 172.21.15.40:0/3208964147 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd60419fec0 con 0x7fd6041033f0 2024-03-18T22:42:28.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.497+0000 7fd60a6d3640 1 -- 172.21.15.40:0/3208964147 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd604108bd0 con 0x7fd6041033f0 2024-03-18T22:42:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.498+0000 7fd6017fa640 1 -- 172.21.15.40:0/3208964147 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd5fc004b40 con 0x7fd6041033f0 2024-03-18T22:42:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.498+0000 7fd6017fa640 1 --2- 172.21.15.40:0/3208964147 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5ec0a2800 0x7fd5ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.498+0000 7fd603fff640 1 --2- 172.21.15.40:0/3208964147 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5ec0a2800 0x7fd5ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.498+0000 7fd6017fa640 1 -- 172.21.15.40:0/3208964147 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd5fc0cf290 con 0x7fd6041033f0 2024-03-18T22:42:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.499+0000 7fd603fff640 1 --2- 172.21.15.40:0/3208964147 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5ec0a2800 0x7fd5ec0a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd5f40095f0 tx=0x7fd5f40058b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.502+0000 7fd6017fa640 1 -- 172.21.15.40:0/3208964147 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd5fc09bfd0 con 0x7fd6041033f0 2024-03-18T22:42:28.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.686+0000 7fd60a6d3640 1 -- 172.21.15.40:0/3208964147 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd60402d440 con 0x7fd6041033f0 2024-03-18T22:42:28.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.688+0000 7fd6017fa640 1 -- 172.21.15.40:0/3208964147 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24600 (secure 0 0 0) 0x7fd5fc09b8d0 con 0x7fd6041033f0 2024-03-18T22:42:28.689 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:28.689 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":91,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:41:43.993830+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:42:28.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.690+0000 7fd60a6d3640 1 -- 172.21.15.40:0/3208964147 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5ec0a2800 msgr2=0x7fd5ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:28.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.690+0000 7fd60a6d3640 1 --2- 172.21.15.40:0/3208964147 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5ec0a2800 0x7fd5ec0a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd5f40095f0 tx=0x7fd5f40058b0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.690+0000 7fd60a6d3640 1 -- 172.21.15.40:0/3208964147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6041033f0 msgr2=0x7fd60419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.690+0000 7fd60a6d3640 1 --2- 172.21.15.40:0/3208964147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6041033f0 0x7fd60419f060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00ec50 tx=0x7fd5fc00c6a0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.690+0000 7fd60a6d3640 1 -- 172.21.15.40:0/3208964147 shutdown_connections 2024-03-18T22:42:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.690+0000 7fd60a6d3640 1 --2- 172.21.15.40:0/3208964147 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5ec0a2800 0x7fd5ec0a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.690+0000 7fd60a6d3640 1 --2- 172.21.15.40:0/3208964147 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd6041033f0 0x7fd60419f060 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.690+0000 7fd60a6d3640 1 --2- 172.21.15.40:0/3208964147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd604102a30 0x7fd604197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.690+0000 7fd60a6d3640 1 --2- 172.21.15.40:0/3208964147 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd604101840 0x7fd604197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.690+0000 7fd60a6d3640 1 -- 172.21.15.40:0/3208964147 >> 172.21.15.40:0/3208964147 conn(0x7fd6040fcdd0 msgr2=0x7fd604105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.690+0000 7fd60a6d3640 1 -- 172.21.15.40:0/3208964147 shutdown_connections 2024-03-18T22:42:28.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.690+0000 7fd60a6d3640 1 -- 172.21.15.40:0/3208964147 wait complete. 2024-03-18T22:42:28.702 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T22:36:07.060691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T22:40:39.925147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T22:40:40.773757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T22:40:44.068183+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T22:42:28.703 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T22:42:28.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.808+0000 7fd1d3f9d640 1 -- 172.21.15.40:0/4124306497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1cc1022e0 msgr2=0x7fd1cc108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:28.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.808+0000 7fd1d3f9d640 1 --2- 172.21.15.40:0/4124306497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1cc1022e0 0x7fd1cc108b70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009be0 tx=0x7fd1c402bef0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.808+0000 7fd1d3f9d640 1 -- 172.21.15.40:0/4124306497 shutdown_connections 2024-03-18T22:42:28.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.808+0000 7fd1d3f9d640 1 --2- 172.21.15.40:0/4124306497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1cc1022e0 0x7fd1cc108b70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.808+0000 7fd1d3f9d640 1 --2- 172.21.15.40:0/4124306497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1cc101920 0x7fd1cc101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.808+0000 7fd1d3f9d640 1 --2- 172.21.15.40:0/4124306497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1cc100730 0x7fd1cc100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.808+0000 7fd1d3f9d640 1 -- 172.21.15.40:0/4124306497 >> 172.21.15.40:0/4124306497 conn(0x7fd1cc0fbcc0 msgr2=0x7fd1cc0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:28.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.808+0000 7fd1d3f9d640 1 -- 172.21.15.40:0/4124306497 shutdown_connections 2024-03-18T22:42:28.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.808+0000 7fd1d3f9d640 1 -- 172.21.15.40:0/4124306497 wait complete. 2024-03-18T22:42:28.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d3f9d640 1 Processor -- start 2024-03-18T22:42:28.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d3f9d640 1 -- start start 2024-03-18T22:42:28.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d3f9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1cc100730 0x7fd1cc1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:28.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d3f9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1cc101920 0x7fd1cc197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:28.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d3f9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1cc1022e0 0x7fd1cc19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:28.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d3f9d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc10c240 con 0x7fd1cc101920 2024-03-18T22:42:28.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d3f9d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc10c0c0 con 0x7fd1cc100730 2024-03-18T22:42:28.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d3f9d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc10c3c0 con 0x7fd1cc1022e0 2024-03-18T22:42:28.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d1538640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1cc101920 0x7fd1cc197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:28.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d1538640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1cc101920 0x7fd1cc197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50760/0 (socket says 172.21.15.40:50760) 2024-03-18T22:42:28.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d1538640 1 -- 172.21.15.40:0/106191514 learned_addr learned my addr 172.21.15.40:0/106191514 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:28.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d1d39640 1 --2- 172.21.15.40:0/106191514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1cc100730 0x7fd1cc1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:28.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d253a640 1 --2- 172.21.15.40:0/106191514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1cc1022e0 0x7fd1cc19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:28.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d1538640 1 -- 172.21.15.40:0/106191514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1cc1022e0 msgr2=0x7fd1cc19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:28.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d1538640 1 --2- 172.21.15.40:0/106191514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1cc1022e0 0x7fd1cc19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d1538640 1 -- 172.21.15.40:0/106191514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1cc100730 msgr2=0x7fd1cc1979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:28.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d1538640 1 --2- 172.21.15.40:0/106191514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1cc100730 0x7fd1cc1979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:28.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.809+0000 7fd1d1538640 1 -- 172.21.15.40:0/106191514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1cc19f610 con 0x7fd1cc101920 2024-03-18T22:42:28.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.810+0000 7fd1d1538640 1 --2- 172.21.15.40:0/106191514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1cc101920 0x7fd1cc197f30 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00e9c0 tx=0x7fd1bc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:28.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.810+0000 7fd1c2ffd640 1 -- 172.21.15.40:0/106191514 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc00ccc0 con 0x7fd1cc101920 2024-03-18T22:42:28.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.810+0000 7fd1c2ffd640 1 -- 172.21.15.40:0/106191514 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1bc0054d0 con 0x7fd1cc101920 2024-03-18T22:42:28.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.810+0000 7fd1d3f9d640 1 -- 172.21.15.40:0/106191514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1cc19f900 con 0x7fd1cc101920 2024-03-18T22:42:28.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.810+0000 7fd1c2ffd640 1 -- 172.21.15.40:0/106191514 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc010430 con 0x7fd1cc101920 2024-03-18T22:42:28.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.810+0000 7fd1d3f9d640 1 -- 172.21.15.40:0/106191514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1cc19fe50 con 0x7fd1cc101920 2024-03-18T22:42:28.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.811+0000 7fd1d3f9d640 1 -- 172.21.15.40:0/106191514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1cc1a00e0 con 0x7fd1cc101920 2024-03-18T22:42:28.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.812+0000 7fd1c2ffd640 1 -- 172.21.15.40:0/106191514 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd1bc0046c0 con 0x7fd1cc101920 2024-03-18T22:42:28.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.812+0000 7fd1c2ffd640 1 --2- 172.21.15.40:0/106191514 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1b80a2800 0x7fd1b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:28.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.812+0000 7fd1d1d39640 1 --2- 172.21.15.40:0/106191514 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1b80a2800 0x7fd1b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:28.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.812+0000 7fd1c2ffd640 1 -- 172.21.15.40:0/106191514 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd1bc0cf010 con 0x7fd1cc101920 2024-03-18T22:42:28.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.813+0000 7fd1d1d39640 1 --2- 172.21.15.40:0/106191514 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1b80a2800 0x7fd1b80a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc101780 tx=0x7fd1c8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:28.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:28.817+0000 7fd1c2ffd640 1 -- 172.21.15.40:0/106191514 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd1bc09bd50 con 0x7fd1cc101920 2024-03-18T22:42:29.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.001+0000 7fd1d3f9d640 1 -- 172.21.15.40:0/106191514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd1cc1a02b0 con 0x7fd1cc101920 2024-03-18T22:42:29.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.002+0000 7fd1c2ffd640 1 -- 172.21.15.40:0/106191514 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fd1bc09b650 con 0x7fd1cc101920 2024-03-18T22:42:29.003 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-18T22:42:29.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.004+0000 7fd1d3f9d640 1 -- 172.21.15.40:0/106191514 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1b80a2800 msgr2=0x7fd1b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.004+0000 7fd1d3f9d640 1 --2- 172.21.15.40:0/106191514 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1b80a2800 0x7fd1b80a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc101780 tx=0x7fd1c8005f90 comp rx=0 tx=0).stop 2024-03-18T22:42:29.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.004+0000 7fd1d3f9d640 1 -- 172.21.15.40:0/106191514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1cc101920 msgr2=0x7fd1cc197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.004+0000 7fd1d3f9d640 1 --2- 172.21.15.40:0/106191514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1cc101920 0x7fd1cc197f30 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00e9c0 tx=0x7fd1bc00ee80 comp rx=0 tx=0).stop 2024-03-18T22:42:29.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.004+0000 7fd1d3f9d640 1 -- 172.21.15.40:0/106191514 shutdown_connections 2024-03-18T22:42:29.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.004+0000 7fd1d3f9d640 1 --2- 172.21.15.40:0/106191514 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1b80a2800 0x7fd1b80a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.004+0000 7fd1d3f9d640 1 --2- 172.21.15.40:0/106191514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1cc1022e0 0x7fd1cc19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.004+0000 7fd1d3f9d640 1 --2- 172.21.15.40:0/106191514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1cc101920 0x7fd1cc197f30 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.004+0000 7fd1d3f9d640 1 --2- 172.21.15.40:0/106191514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1cc100730 0x7fd1cc1979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.004+0000 7fd1d3f9d640 1 -- 172.21.15.40:0/106191514 >> 172.21.15.40:0/106191514 conn(0x7fd1cc0fbcc0 msgr2=0x7fd1cc069e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:29.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.004+0000 7fd1d3f9d640 1 -- 172.21.15.40:0/106191514 shutdown_connections 2024-03-18T22:42:29.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.004+0000 7fd1d3f9d640 1 -- 172.21.15.40:0/106191514 wait complete. 2024-03-18T22:42:29.014 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:28 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2957532378' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:42:29.014 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:28 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1320784680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:42:29.014 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:28 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3208964147' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:29.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T22:42:29.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.120+0000 7fdfa8261640 1 -- 172.21.15.40:0/2739455871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa0100250 msgr2=0x7fdfa0109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.120+0000 7fdfa8261640 1 --2- 172.21.15.40:0/2739455871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa0100250 0x7fdfa0109e00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdf98009ca0 tx=0x7fdf9802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.120+0000 7fdfa8261640 1 -- 172.21.15.40:0/2739455871 shutdown_connections 2024-03-18T22:42:29.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.120+0000 7fdfa8261640 1 --2- 172.21.15.40:0/2739455871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa0100250 0x7fdfa0109e00 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.120+0000 7fdfa8261640 1 --2- 172.21.15.40:0/2739455871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfa00ff890 0x7fdfa00ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.120+0000 7fdfa8261640 1 --2- 172.21.15.40:0/2739455871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdfa0102e20 0x7fdfa0103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.120+0000 7fdfa8261640 1 -- 172.21.15.40:0/2739455871 >> 172.21.15.40:0/2739455871 conn(0x7fdfa00fb410 msgr2=0x7fdfa00fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:29.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.120+0000 7fdfa8261640 1 -- 172.21.15.40:0/2739455871 shutdown_connections 2024-03-18T22:42:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.120+0000 7fdfa8261640 1 -- 172.21.15.40:0/2739455871 wait complete. 2024-03-18T22:42:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa8261640 1 Processor -- start 2024-03-18T22:42:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa8261640 1 -- start start 2024-03-18T22:42:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa8261640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfa00ff890 0x7fdfa01979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa8261640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdfa0100250 0x7fdfa0197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa8261640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa0102e20 0x7fdfa019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa8261640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa010d330 con 0x7fdfa0102e20 2024-03-18T22:42:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa8261640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa010d1b0 con 0x7fdfa00ff890 2024-03-18T22:42:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa8261640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa010d4b0 con 0x7fdfa0100250 2024-03-18T22:42:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa67fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa0102e20 0x7fdfa019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa67fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa0102e20 0x7fdfa019ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50788/0 (socket says 172.21.15.40:50788) 2024-03-18T22:42:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa57fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdfa0100250 0x7fdfa0197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa67fe640 1 -- 172.21.15.40:0/2053735745 learned_addr learned my addr 172.21.15.40:0/2053735745 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa5ffd640 1 --2- 172.21.15.40:0/2053735745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfa00ff890 0x7fdfa01979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:29.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa67fe640 1 -- 172.21.15.40:0/2053735745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdfa0100250 msgr2=0x7fdfa0197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa67fe640 1 --2- 172.21.15.40:0/2053735745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdfa0100250 0x7fdfa0197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa67fe640 1 -- 172.21.15.40:0/2053735745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfa00ff890 msgr2=0x7fdfa01979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa67fe640 1 --2- 172.21.15.40:0/2053735745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfa00ff890 0x7fdfa01979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.121+0000 7fdfa67fe640 1 -- 172.21.15.40:0/2053735745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa019f600 con 0x7fdfa0102e20 2024-03-18T22:42:29.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.122+0000 7fdfa67fe640 1 --2- 172.21.15.40:0/2053735745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa0102e20 0x7fdfa019ef90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdf9802c700 tx=0x7fdf980328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:29.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.122+0000 7fdf96ffd640 1 -- 172.21.15.40:0/2053735745 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf98041020 con 0x7fdfa0102e20 2024-03-18T22:42:29.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.122+0000 7fdf96ffd640 1 -- 172.21.15.40:0/2053735745 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf98003e20 con 0x7fdfa0102e20 2024-03-18T22:42:29.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.122+0000 7fdfa8261640 1 -- 172.21.15.40:0/2053735745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfa019f890 con 0x7fdfa0102e20 2024-03-18T22:42:29.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.122+0000 7fdf96ffd640 1 -- 172.21.15.40:0/2053735745 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf98039040 con 0x7fdfa0102e20 2024-03-18T22:42:29.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.122+0000 7fdfa8261640 1 -- 172.21.15.40:0/2053735745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfa019fd80 con 0x7fdfa0102e20 2024-03-18T22:42:29.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.123+0000 7fdfa8261640 1 -- 172.21.15.40:0/2053735745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfa0108ce0 con 0x7fdfa0102e20 2024-03-18T22:42:29.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.124+0000 7fdf96ffd640 1 -- 172.21.15.40:0/2053735745 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdf98044020 con 0x7fdfa0102e20 2024-03-18T22:42:29.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.124+0000 7fdf96ffd640 1 --2- 172.21.15.40:0/2053735745 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf8c0a2800 0x7fdf8c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:29.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.125+0000 7fdfa5ffd640 1 --2- 172.21.15.40:0/2053735745 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf8c0a2800 0x7fdf8c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:29.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.125+0000 7fdf96ffd640 1 -- 172.21.15.40:0/2053735745 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdf980f1350 con 0x7fdfa0102e20 2024-03-18T22:42:29.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.125+0000 7fdfa5ffd640 1 --2- 172.21.15.40:0/2053735745 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf8c0a2800 0x7fdf8c0a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0068680 tx=0x7fdf9c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:29.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.128+0000 7fdf96ffd640 1 -- 172.21.15.40:0/2053735745 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdf9804b020 con 0x7fdfa0102e20 2024-03-18T22:42:29.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.310+0000 7fdfa8261640 1 -- 172.21.15.40:0/2053735745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdfa002d1f0 con 0x7fdfa0102e20 2024-03-18T22:42:29.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.310+0000 7fdf96ffd640 1 -- 172.21.15.40:0/2053735745 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fdf980be090 con 0x7fdfa0102e20 2024-03-18T22:42:29.311 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:42:29.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.312+0000 7fdfa8261640 1 -- 172.21.15.40:0/2053735745 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf8c0a2800 msgr2=0x7fdf8c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.312+0000 7fdfa8261640 1 --2- 172.21.15.40:0/2053735745 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf8c0a2800 0x7fdf8c0a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0068680 tx=0x7fdf9c005f90 comp rx=0 tx=0).stop 2024-03-18T22:42:29.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.312+0000 7fdfa8261640 1 -- 172.21.15.40:0/2053735745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa0102e20 msgr2=0x7fdfa019ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.312+0000 7fdfa8261640 1 --2- 172.21.15.40:0/2053735745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa0102e20 0x7fdfa019ef90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdf9802c700 tx=0x7fdf980328a0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.312+0000 7fdfa8261640 1 -- 172.21.15.40:0/2053735745 shutdown_connections 2024-03-18T22:42:29.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.312+0000 7fdfa8261640 1 --2- 172.21.15.40:0/2053735745 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf8c0a2800 0x7fdf8c0a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.312+0000 7fdfa8261640 1 --2- 172.21.15.40:0/2053735745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfa0102e20 0x7fdfa019ef90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.312+0000 7fdfa8261640 1 --2- 172.21.15.40:0/2053735745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdfa0100250 0x7fdfa0197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.312+0000 7fdfa8261640 1 --2- 172.21.15.40:0/2053735745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfa00ff890 0x7fdfa01979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.312+0000 7fdfa8261640 1 -- 172.21.15.40:0/2053735745 >> 172.21.15.40:0/2053735745 conn(0x7fdfa00fb410 msgr2=0x7fdfa0105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:29.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.312+0000 7fdfa8261640 1 -- 172.21.15.40:0/2053735745 shutdown_connections 2024-03-18T22:42:29.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.312+0000 7fdfa8261640 1 -- 172.21.15.40:0/2053735745 wait complete. 2024-03-18T22:42:29.324 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T22:42:29.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:28 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2957532378' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:42:29.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:28 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1320784680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:42:29.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:28 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3208964147' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:29.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:28 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2957532378' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:42:29.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:28 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1320784680' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:42:29.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:28 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3208964147' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:29.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.428+0000 7fb7b8227640 1 -- 172.21.15.40:0/809971027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7b01075c0 msgr2=0x7fb7b01099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.428+0000 7fb7b8227640 1 --2- 172.21.15.40:0/809971027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7b01075c0 0x7fb7b01099b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb7a0009ca0 tx=0x7fb7a002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.428+0000 7fb7b8227640 1 -- 172.21.15.40:0/809971027 shutdown_connections 2024-03-18T22:42:29.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.428+0000 7fb7b8227640 1 --2- 172.21.15.40:0/809971027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7b01075c0 0x7fb7b01099b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.428+0000 7fb7b8227640 1 --2- 172.21.15.40:0/809971027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7b0104c90 0x7fb7b0107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.428+0000 7fb7b8227640 1 --2- 172.21.15.40:0/809971027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7b0102360 0x7fb7b0104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.428+0000 7fb7b8227640 1 -- 172.21.15.40:0/809971027 >> 172.21.15.40:0/809971027 conn(0x7fb7b00fbcb0 msgr2=0x7fb7b00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.429+0000 7fb7b8227640 1 -- 172.21.15.40:0/809971027 shutdown_connections 2024-03-18T22:42:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.429+0000 7fb7b8227640 1 -- 172.21.15.40:0/809971027 wait complete. 2024-03-18T22:42:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.429+0000 7fb7b8227640 1 Processor -- start 2024-03-18T22:42:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.429+0000 7fb7b8227640 1 -- start start 2024-03-18T22:42:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.429+0000 7fb7b8227640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7b0102360 0x7fb7b0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.429+0000 7fb7b8227640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7b0104c90 0x7fb7b0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.429+0000 7fb7b8227640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7b01075c0 0x7fb7b019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.429+0000 7fb7b8227640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b010d380 con 0x7fb7b0102360 2024-03-18T22:42:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.429+0000 7fb7b8227640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b010d200 con 0x7fb7b0104c90 2024-03-18T22:42:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.429+0000 7fb7b8227640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b010d500 con 0x7fb7b01075c0 2024-03-18T22:42:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.429+0000 7fb7b5fc3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7b0102360 0x7fb7b0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.429+0000 7fb7b5fc3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7b0102360 0x7fb7b0197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50806/0 (socket says 172.21.15.40:50806) 2024-03-18T22:42:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.429+0000 7fb7b5fc3640 1 -- 172.21.15.40:0/3201677197 learned_addr learned my addr 172.21.15.40:0/3201677197 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.429+0000 7fb7b57c2640 1 --2- 172.21.15.40:0/3201677197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7b0104c90 0x7fb7b0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.429+0000 7fb7b67c4640 1 --2- 172.21.15.40:0/3201677197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7b01075c0 0x7fb7b019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.430+0000 7fb7b5fc3640 1 -- 172.21.15.40:0/3201677197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7b01075c0 msgr2=0x7fb7b019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.430+0000 7fb7b5fc3640 1 --2- 172.21.15.40:0/3201677197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7b01075c0 0x7fb7b019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.430+0000 7fb7b5fc3640 1 -- 172.21.15.40:0/3201677197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7b0104c90 msgr2=0x7fb7b0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.430+0000 7fb7b5fc3640 1 --2- 172.21.15.40:0/3201677197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7b0104c90 0x7fb7b0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.430+0000 7fb7b5fc3640 1 -- 172.21.15.40:0/3201677197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b019f6d0 con 0x7fb7b0102360 2024-03-18T22:42:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.430+0000 7fb7b5fc3640 1 --2- 172.21.15.40:0/3201677197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7b0102360 0x7fb7b0197ab0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800e9f0 tx=0x7fb7a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.430+0000 7fb7aeffd640 1 -- 172.21.15.40:0/3201677197 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a800cd80 con 0x7fb7b0102360 2024-03-18T22:42:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.430+0000 7fb7aeffd640 1 -- 172.21.15.40:0/3201677197 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb7a80054d0 con 0x7fb7b0102360 2024-03-18T22:42:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.430+0000 7fb7b8227640 1 -- 172.21.15.40:0/3201677197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b019f9c0 con 0x7fb7b0102360 2024-03-18T22:42:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.430+0000 7fb7aeffd640 1 -- 172.21.15.40:0/3201677197 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8010490 con 0x7fb7b0102360 2024-03-18T22:42:29.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.430+0000 7fb7b8227640 1 -- 172.21.15.40:0/3201677197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b019fec0 con 0x7fb7b0102360 2024-03-18T22:42:29.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.431+0000 7fb7b8227640 1 -- 172.21.15.40:0/3201677197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7b00fd7a0 con 0x7fb7b0102360 2024-03-18T22:42:29.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.432+0000 7fb7aeffd640 1 -- 172.21.15.40:0/3201677197 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb7a8005020 con 0x7fb7b0102360 2024-03-18T22:42:29.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.433+0000 7fb7aeffd640 1 --2- 172.21.15.40:0/3201677197 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb7980a2800 0x7fb7980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:29.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.433+0000 7fb7b57c2640 1 --2- 172.21.15.40:0/3201677197 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb7980a2800 0x7fb7980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:29.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.433+0000 7fb7aeffd640 1 -- 172.21.15.40:0/3201677197 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb7a80cf140 con 0x7fb7b0102360 2024-03-18T22:42:29.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.434+0000 7fb7b57c2640 1 --2- 172.21.15.40:0/3201677197 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb7980a2800 0x7fb7980a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb7a40095f0 tx=0x7fb7a4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.438+0000 7fb7aeffd640 1 -- 172.21.15.40:0/3201677197 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb7a809be80 con 0x7fb7b0102360 2024-03-18T22:42:29.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.620+0000 7fb7b8227640 1 -- 172.21.15.40:0/3201677197 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb7b0062420 con 0x7fb7b0102360 2024-03-18T22:42:29.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.621+0000 7fb7aeffd640 1 -- 172.21.15.40:0/3201677197 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fb7a809b780 con 0x7fb7b0102360 2024-03-18T22:42:29.622 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-18T22:42:29.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.623+0000 7fb7b8227640 1 -- 172.21.15.40:0/3201677197 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb7980a2800 msgr2=0x7fb7980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.623+0000 7fb7b8227640 1 --2- 172.21.15.40:0/3201677197 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb7980a2800 0x7fb7980a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb7a40095f0 tx=0x7fb7a4005f90 comp rx=0 tx=0).stop 2024-03-18T22:42:29.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.623+0000 7fb7b8227640 1 -- 172.21.15.40:0/3201677197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7b0102360 msgr2=0x7fb7b0197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.623+0000 7fb7b8227640 1 --2- 172.21.15.40:0/3201677197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7b0102360 0x7fb7b0197ab0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800e9f0 tx=0x7fb7a800edf0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.623+0000 7fb7b8227640 1 -- 172.21.15.40:0/3201677197 shutdown_connections 2024-03-18T22:42:29.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.623+0000 7fb7b8227640 1 --2- 172.21.15.40:0/3201677197 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb7980a2800 0x7fb7980a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.623+0000 7fb7b8227640 1 --2- 172.21.15.40:0/3201677197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb7b01075c0 0x7fb7b019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.623+0000 7fb7b8227640 1 --2- 172.21.15.40:0/3201677197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb7b0104c90 0x7fb7b0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.623+0000 7fb7b8227640 1 --2- 172.21.15.40:0/3201677197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7b0102360 0x7fb7b0197ab0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.623+0000 7fb7b8227640 1 -- 172.21.15.40:0/3201677197 >> 172.21.15.40:0/3201677197 conn(0x7fb7b00fbcb0 msgr2=0x7fb7b0102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:29.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.623+0000 7fb7b8227640 1 -- 172.21.15.40:0/3201677197 shutdown_connections 2024-03-18T22:42:29.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.623+0000 7fb7b8227640 1 -- 172.21.15.40:0/3201677197 wait complete. 2024-03-18T22:42:29.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T22:42:29.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.740+0000 7f7aeb019640 1 -- 172.21.15.40:0/464693571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ae4101830 msgr2=0x7f7ae4101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.740+0000 7f7aeb019640 1 --2- 172.21.15.40:0/464693571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ae4101830 0x7f7ae4101c50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009be0 tx=0x7f7adc02bef0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.740+0000 7f7aeb019640 1 -- 172.21.15.40:0/464693571 shutdown_connections 2024-03-18T22:42:29.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.740+0000 7f7aeb019640 1 --2- 172.21.15.40:0/464693571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ae41033e0 0x7f7ae4109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.740+0000 7f7aeb019640 1 --2- 172.21.15.40:0/464693571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ae4102a20 0x7f7ae4102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.740+0000 7f7aeb019640 1 --2- 172.21.15.40:0/464693571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ae4101830 0x7f7ae4101c50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.740+0000 7f7aeb019640 1 -- 172.21.15.40:0/464693571 >> 172.21.15.40:0/464693571 conn(0x7f7ae40fcdc0 msgr2=0x7f7ae40ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:29.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.740+0000 7f7aeb019640 1 -- 172.21.15.40:0/464693571 shutdown_connections 2024-03-18T22:42:29.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.740+0000 7f7aeb019640 1 -- 172.21.15.40:0/464693571 wait complete. 2024-03-18T22:42:29.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.740+0000 7f7aeb019640 1 Processor -- start 2024-03-18T22:42:29.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.740+0000 7f7aeb019640 1 -- start start 2024-03-18T22:42:29.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7aeb019640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ae4101830 0x7f7ae4197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:29.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7aeb019640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ae4102a20 0x7f7ae4198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:29.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7aeb019640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ae41033e0 0x7f7ae419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:29.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7aeb019640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae410d330 con 0x7f7ae4101830 2024-03-18T22:42:29.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7aeb019640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae410d1b0 con 0x7f7ae41033e0 2024-03-18T22:42:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7aeb019640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae410d4b0 con 0x7f7ae4102a20 2024-03-18T22:42:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7ae8db5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ae4101830 0x7f7ae4197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7ae8db5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ae4101830 0x7f7ae4197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50814/0 (socket says 172.21.15.40:50814) 2024-03-18T22:42:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7ae8db5640 1 -- 172.21.15.40:0/4136493536 learned_addr learned my addr 172.21.15.40:0/4136493536 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7ae3fff640 1 --2- 172.21.15.40:0/4136493536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ae4102a20 0x7f7ae4198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7ae95b6640 1 --2- 172.21.15.40:0/4136493536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ae41033e0 0x7f7ae419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7ae8db5640 1 -- 172.21.15.40:0/4136493536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ae4102a20 msgr2=0x7f7ae4198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7ae8db5640 1 --2- 172.21.15.40:0/4136493536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ae4102a20 0x7f7ae4198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7ae8db5640 1 -- 172.21.15.40:0/4136493536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ae41033e0 msgr2=0x7f7ae419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7ae8db5640 1 --2- 172.21.15.40:0/4136493536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ae41033e0 0x7f7ae419f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7ae8db5640 1 -- 172.21.15.40:0/4136493536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae419f700 con 0x7f7ae4101830 2024-03-18T22:42:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7ae3fff640 1 --2- 172.21.15.40:0/4136493536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ae4102a20 0x7f7ae4198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7ae8db5640 1 --2- 172.21.15.40:0/4136493536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ae4101830 0x7f7ae4197ae0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7adc000f80 tx=0x7f7adc033a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7ae1ffb640 1 -- 172.21.15.40:0/4136493536 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc043020 con 0x7f7ae4101830 2024-03-18T22:42:29.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7aeb019640 1 -- 172.21.15.40:0/4136493536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ae419f990 con 0x7f7ae4101830 2024-03-18T22:42:29.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7ae1ffb640 1 -- 172.21.15.40:0/4136493536 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7adc033e60 con 0x7f7ae4101830 2024-03-18T22:42:29.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.741+0000 7f7ae1ffb640 1 -- 172.21.15.40:0/4136493536 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc034430 con 0x7f7ae4101830 2024-03-18T22:42:29.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.742+0000 7f7aeb019640 1 -- 172.21.15.40:0/4136493536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ae419fee0 con 0x7f7ae4101830 2024-03-18T22:42:29.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.743+0000 7f7aeb019640 1 -- 172.21.15.40:0/4136493536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ae4068a60 con 0x7f7ae4101830 2024-03-18T22:42:29.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.744+0000 7f7ae1ffb640 1 -- 172.21.15.40:0/4136493536 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7adc046020 con 0x7f7ae4101830 2024-03-18T22:42:29.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.744+0000 7f7ae1ffb640 1 --2- 172.21.15.40:0/4136493536 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ad00a2800 0x7f7ad00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:29.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.744+0000 7f7ae3fff640 1 --2- 172.21.15.40:0/4136493536 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ad00a2800 0x7f7ad00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:29.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.745+0000 7f7ae1ffb640 1 -- 172.21.15.40:0/4136493536 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7adc0f1430 con 0x7f7ae4101830 2024-03-18T22:42:29.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.745+0000 7f7ae3fff640 1 --2- 172.21.15.40:0/4136493536 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ad00a2800 0x7f7ad00a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4198ff0 tx=0x7f7acc009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:29.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.749+0000 7f7ae1ffb640 1 -- 172.21.15.40:0/4136493536 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7adc04d020 con 0x7f7ae4101830 2024-03-18T22:42:29.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.934+0000 7f7aeb019640 1 -- 172.21.15.40:0/4136493536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7ae4062420 con 0x7f7ae4101830 2024-03-18T22:42:29.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.934+0000 7f7ae1ffb640 1 -- 172.21.15.40:0/4136493536 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f7adc0be170 con 0x7f7ae4101830 2024-03-18T22:42:29.935 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:42:29.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.936+0000 7f7aeb019640 1 -- 172.21.15.40:0/4136493536 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ad00a2800 msgr2=0x7f7ad00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.936+0000 7f7aeb019640 1 --2- 172.21.15.40:0/4136493536 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ad00a2800 0x7f7ad00a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4198ff0 tx=0x7f7acc009260 comp rx=0 tx=0).stop 2024-03-18T22:42:29.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.936+0000 7f7aeb019640 1 -- 172.21.15.40:0/4136493536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ae4101830 msgr2=0x7f7ae4197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:29.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.936+0000 7f7aeb019640 1 --2- 172.21.15.40:0/4136493536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ae4101830 0x7f7ae4197ae0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7adc000f80 tx=0x7f7adc033a60 comp rx=0 tx=0).stop 2024-03-18T22:42:29.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.936+0000 7f7aeb019640 1 -- 172.21.15.40:0/4136493536 shutdown_connections 2024-03-18T22:42:29.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.936+0000 7f7aeb019640 1 --2- 172.21.15.40:0/4136493536 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ad00a2800 0x7f7ad00a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.936+0000 7f7aeb019640 1 --2- 172.21.15.40:0/4136493536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ae41033e0 0x7f7ae419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.936+0000 7f7aeb019640 1 --2- 172.21.15.40:0/4136493536 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ae4102a20 0x7f7ae4198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.936+0000 7f7aeb019640 1 --2- 172.21.15.40:0/4136493536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ae4101830 0x7f7ae4197ae0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.936+0000 7f7aeb019640 1 -- 172.21.15.40:0/4136493536 >> 172.21.15.40:0/4136493536 conn(0x7f7ae40fcdc0 msgr2=0x7f7ae4105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.936+0000 7f7aeb019640 1 -- 172.21.15.40:0/4136493536 shutdown_connections 2024-03-18T22:42:29.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:29.936+0000 7f7aeb019640 1 -- 172.21.15.40:0/4136493536 wait complete. 2024-03-18T22:42:29.948 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:42:30.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.050+0000 7fb386377640 1 -- 172.21.15.40:0/809044931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb380100a10 msgr2=0x7fb380100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.050+0000 7fb386377640 1 --2- 172.21.15.40:0/809044931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb380100a10 0x7fb380100e90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb370009900 tx=0x7fb37002bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.050+0000 7fb386377640 1 -- 172.21.15.40:0/809044931 shutdown_connections 2024-03-18T22:42:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.050+0000 7fb386377640 1 --2- 172.21.15.40:0/809044931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb3801013d0 0x7fb380108c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.050+0000 7fb386377640 1 --2- 172.21.15.40:0/809044931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb380100a10 0x7fb380100e90 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.050+0000 7fb386377640 1 --2- 172.21.15.40:0/809044931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb380101ce0 0x7fb380102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.050+0000 7fb386377640 1 -- 172.21.15.40:0/809044931 >> 172.21.15.40:0/809044931 conn(0x7fb3800fb440 msgr2=0x7fb3800fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.050+0000 7fb386377640 1 -- 172.21.15.40:0/809044931 shutdown_connections 2024-03-18T22:42:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.050+0000 7fb386377640 1 -- 172.21.15.40:0/809044931 wait complete. 2024-03-18T22:42:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.051+0000 7fb386377640 1 Processor -- start 2024-03-18T22:42:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.051+0000 7fb386377640 1 -- start start 2024-03-18T22:42:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.051+0000 7fb386377640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb380100a10 0x7fb380104960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.051+0000 7fb386377640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3801013d0 0x7fb380102fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.051+0000 7fb386377640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb380101ce0 0x7fb3801034e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:30.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.051+0000 7fb386377640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb38010c270 con 0x7fb3801013d0 2024-03-18T22:42:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.051+0000 7fb386377640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb38010c0f0 con 0x7fb380101ce0 2024-03-18T22:42:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.051+0000 7fb386377640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb38010c3f0 con 0x7fb380100a10 2024-03-18T22:42:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.051+0000 7fb37f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3801013d0 0x7fb380102fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.051+0000 7fb37f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3801013d0 0x7fb380102fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50826/0 (socket says 172.21.15.40:50826) 2024-03-18T22:42:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.051+0000 7fb37f7fe640 1 -- 172.21.15.40:0/1644230716 learned_addr learned my addr 172.21.15.40:0/1644230716 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.051+0000 7fb384914640 1 --2- 172.21.15.40:0/1644230716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb380101ce0 0x7fb3801034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.051+0000 7fb37ffff640 1 --2- 172.21.15.40:0/1644230716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb380100a10 0x7fb380104960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.052+0000 7fb37f7fe640 1 -- 172.21.15.40:0/1644230716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb380100a10 msgr2=0x7fb380104960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.052+0000 7fb37f7fe640 1 --2- 172.21.15.40:0/1644230716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb380100a10 0x7fb380104960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.052+0000 7fb37f7fe640 1 -- 172.21.15.40:0/1644230716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb380101ce0 msgr2=0x7fb3801034e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.052+0000 7fb37f7fe640 1 --2- 172.21.15.40:0/1644230716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb380101ce0 0x7fb3801034e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.052+0000 7fb37f7fe640 1 -- 172.21.15.40:0/1644230716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb380103d70 con 0x7fb3801013d0 2024-03-18T22:42:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.052+0000 7fb37f7fe640 1 --2- 172.21.15.40:0/1644230716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3801013d0 0x7fb380102fa0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb370005ab0 tx=0x7fb3700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:30.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.052+0000 7fb37d7fa640 1 -- 172.21.15.40:0/1644230716 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb370043020 con 0x7fb3801013d0 2024-03-18T22:42:30.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.052+0000 7fb37d7fa640 1 -- 172.21.15.40:0/1644230716 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb370004a50 con 0x7fb3801013d0 2024-03-18T22:42:30.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.052+0000 7fb386377640 1 -- 172.21.15.40:0/1644230716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3801a1b90 con 0x7fb3801013d0 2024-03-18T22:42:30.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.052+0000 7fb37d7fa640 1 -- 172.21.15.40:0/1644230716 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb370003ba0 con 0x7fb3801013d0 2024-03-18T22:42:30.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.052+0000 7fb386377640 1 -- 172.21.15.40:0/1644230716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3801a20e0 con 0x7fb3801013d0 2024-03-18T22:42:30.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.053+0000 7fb386377640 1 -- 172.21.15.40:0/1644230716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3801a22b0 con 0x7fb3801013d0 2024-03-18T22:42:30.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.054+0000 7fb37d7fa640 1 -- 172.21.15.40:0/1644230716 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb370046020 con 0x7fb3801013d0 2024-03-18T22:42:30.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.055+0000 7fb37d7fa640 1 --2- 172.21.15.40:0/1644230716 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb36c0a2800 0x7fb36c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:30.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.055+0000 7fb37ffff640 1 --2- 172.21.15.40:0/1644230716 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb36c0a2800 0x7fb36c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:30.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.055+0000 7fb37d7fa640 1 -- 172.21.15.40:0/1644230716 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb3700f0fa0 con 0x7fb3801013d0 2024-03-18T22:42:30.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.055+0000 7fb37ffff640 1 --2- 172.21.15.40:0/1644230716 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb36c0a2800 0x7fb36c0a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb380069c50 tx=0x7fb368005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:30.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.060+0000 7fb37d7fa640 1 -- 172.21.15.40:0/1644230716 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb37004d020 con 0x7fb3801013d0 2024-03-18T22:42:30.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.281+0000 7fb386377640 1 -- 172.21.15.40:0/1644230716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3801a2640 con 0x7fb3801013d0 2024-03-18T22:42:30.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.282+0000 7fb37d7fa640 1 -- 172.21.15.40:0/1644230716 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7fb3801a2640 con 0x7fb3801013d0 2024-03-18T22:42:30.282 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:30.283 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:41:09.839105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24481,24509,24473,24401,24388]},"id":1}]} 2024-03-18T22:42:30.283 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 27 2024-03-18T22:42:30.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.284+0000 7fb386377640 1 -- 172.21.15.40:0/1644230716 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb36c0a2800 msgr2=0x7fb36c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:30.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.284+0000 7fb386377640 1 --2- 172.21.15.40:0/1644230716 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb36c0a2800 0x7fb36c0a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb380069c50 tx=0x7fb368005f90 comp rx=0 tx=0).stop 2024-03-18T22:42:30.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.284+0000 7fb386377640 1 -- 172.21.15.40:0/1644230716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3801013d0 msgr2=0x7fb380102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:30.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.284+0000 7fb386377640 1 --2- 172.21.15.40:0/1644230716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3801013d0 0x7fb380102fa0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb370005ab0 tx=0x7fb3700042f0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.284+0000 7fb386377640 1 -- 172.21.15.40:0/1644230716 shutdown_connections 2024-03-18T22:42:30.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.284+0000 7fb386377640 1 --2- 172.21.15.40:0/1644230716 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb36c0a2800 0x7fb36c0a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.284+0000 7fb386377640 1 --2- 172.21.15.40:0/1644230716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb380101ce0 0x7fb3801034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.284+0000 7fb386377640 1 --2- 172.21.15.40:0/1644230716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3801013d0 0x7fb380102fa0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.284+0000 7fb386377640 1 --2- 172.21.15.40:0/1644230716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb380100a10 0x7fb380104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.284+0000 7fb386377640 1 -- 172.21.15.40:0/1644230716 >> 172.21.15.40:0/1644230716 conn(0x7fb3800fb440 msgr2=0x7fb3800ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:30.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.284+0000 7fb386377640 1 -- 172.21.15.40:0/1644230716 shutdown_connections 2024-03-18T22:42:30.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.284+0000 7fb386377640 1 -- 172.21.15.40:0/1644230716 wait complete. 2024-03-18T22:42:30.294 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:29 smithi040 ceph-mon[62643]: pgmap v344: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T22:42:30.295 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:29 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/106191514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:42:30.295 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:29 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2053735745' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:42:30.295 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:29 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3201677197' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:42:30.296 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:42:30.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:29 smithi113 ceph-mon[66775]: pgmap v344: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T22:42:30.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:29 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/106191514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:42:30.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:29 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2053735745' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:42:30.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:29 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3201677197' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:42:30.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:29 smithi103 ceph-mon[66377]: pgmap v344: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T22:42:30.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:29 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/106191514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:42:30.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:29 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2053735745' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:42:30.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:29 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3201677197' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:42:30.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.398+0000 7f30f460b640 1 -- 172.21.15.40:0/602212885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30ec101410 msgr2=0x7f30ec101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:30.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.398+0000 7f30f460b640 1 --2- 172.21.15.40:0/602212885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30ec101410 0x7f30ec101890 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009be0 tx=0x7f30dc02bef0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.398+0000 7f30f460b640 1 -- 172.21.15.40:0/602212885 shutdown_connections 2024-03-18T22:42:30.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.398+0000 7f30f460b640 1 --2- 172.21.15.40:0/602212885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30ec101410 0x7f30ec101890 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.398+0000 7f30f460b640 1 --2- 172.21.15.40:0/602212885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30ec100a70 0x7f30ec100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.398+0000 7f30f460b640 1 --2- 172.21.15.40:0/602212885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30ec100110 0x7f30ec100530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.398+0000 7f30f460b640 1 -- 172.21.15.40:0/602212885 >> 172.21.15.40:0/602212885 conn(0x7f30ec0fbcb0 msgr2=0x7f30ec0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:30.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.398+0000 7f30f460b640 1 -- 172.21.15.40:0/602212885 shutdown_connections 2024-03-18T22:42:30.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.399+0000 7f30f460b640 1 -- 172.21.15.40:0/602212885 wait complete. 2024-03-18T22:42:30.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.399+0000 7f30f460b640 1 Processor -- start 2024-03-18T22:42:30.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.399+0000 7f30f460b640 1 -- start start 2024-03-18T22:42:30.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.399+0000 7f30f460b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30ec100110 0x7f30ec199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:30.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.399+0000 7f30f460b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30ec100a70 0x7f30ec19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:30.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.399+0000 7f30f460b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30ec101410 0x7f30ec1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:30.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.399+0000 7f30f460b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec10f560 con 0x7f30ec100a70 2024-03-18T22:42:30.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.399+0000 7f30f460b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec10f3e0 con 0x7f30ec100110 2024-03-18T22:42:30.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.399+0000 7f30f460b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec10f6e0 con 0x7f30ec101410 2024-03-18T22:42:30.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.399+0000 7f30f1ba6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30ec100a70 0x7f30ec19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:30.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.399+0000 7f30f1ba6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30ec100a70 0x7f30ec19a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50852/0 (socket says 172.21.15.40:50852) 2024-03-18T22:42:30.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.399+0000 7f30f23a7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30ec100110 0x7f30ec199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:30.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.399+0000 7f30f1ba6640 1 -- 172.21.15.40:0/1479115768 learned_addr learned my addr 172.21.15.40:0/1479115768 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:30.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.399+0000 7f30f2ba8640 1 --2- 172.21.15.40:0/1479115768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30ec101410 0x7f30ec1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:30.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.400+0000 7f30f1ba6640 1 -- 172.21.15.40:0/1479115768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30ec101410 msgr2=0x7f30ec1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:30.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.400+0000 7f30f1ba6640 1 --2- 172.21.15.40:0/1479115768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30ec101410 0x7f30ec1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.400+0000 7f30f1ba6640 1 -- 172.21.15.40:0/1479115768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30ec100110 msgr2=0x7f30ec199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:30.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.400+0000 7f30f1ba6640 1 --2- 172.21.15.40:0/1479115768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30ec100110 0x7f30ec199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.400+0000 7f30f1ba6640 1 -- 172.21.15.40:0/1479115768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ec1a1840 con 0x7f30ec100a70 2024-03-18T22:42:30.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.400+0000 7f30f1ba6640 1 --2- 172.21.15.40:0/1479115768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30ec100a70 0x7f30ec19a170 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f30e000e9c0 tx=0x7f30e000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:30.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.400+0000 7f30eb7fe640 1 -- 172.21.15.40:0/1479115768 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e000cde0 con 0x7f30ec100a70 2024-03-18T22:42:30.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.400+0000 7f30f460b640 1 -- 172.21.15.40:0/1479115768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30ec1a1b30 con 0x7f30ec100a70 2024-03-18T22:42:30.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.400+0000 7f30eb7fe640 1 -- 172.21.15.40:0/1479115768 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30e00035f0 con 0x7f30ec100a70 2024-03-18T22:42:30.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.400+0000 7f30eb7fe640 1 -- 172.21.15.40:0/1479115768 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e0010690 con 0x7f30ec100a70 2024-03-18T22:42:30.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.400+0000 7f30f460b640 1 -- 172.21.15.40:0/1479115768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30ec1a2080 con 0x7f30ec100a70 2024-03-18T22:42:30.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.401+0000 7f30f460b640 1 -- 172.21.15.40:0/1479115768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30ec10b040 con 0x7f30ec100a70 2024-03-18T22:42:30.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.402+0000 7f30eb7fe640 1 -- 172.21.15.40:0/1479115768 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f30e00037a0 con 0x7f30ec100a70 2024-03-18T22:42:30.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.402+0000 7f30eb7fe640 1 --2- 172.21.15.40:0/1479115768 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f30d40a2800 0x7f30d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:30.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.403+0000 7f30f23a7640 1 --2- 172.21.15.40:0/1479115768 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f30d40a2800 0x7f30d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:30.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.403+0000 7f30eb7fe640 1 -- 172.21.15.40:0/1479115768 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f30e00cf110 con 0x7f30ec100a70 2024-03-18T22:42:30.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.403+0000 7f30f23a7640 1 --2- 172.21.15.40:0/1479115768 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f30d40a2800 0x7f30d40a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f30ec19b100 tx=0x7f30e4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:30.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.407+0000 7f30eb7fe640 1 -- 172.21.15.40:0/1479115768 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f30e0009510 con 0x7f30ec100a70 2024-03-18T22:42:30.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.595+0000 7f30f460b640 1 -- 172.21.15.40:0/1479115768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f30ec02d3c0 con 0x7f30ec100a70 2024-03-18T22:42:30.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.596+0000 7f30eb7fe640 1 -- 172.21.15.40:0/1479115768 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24600 (secure 0 0 0) 0x7f30e009be50 con 0x7f30ec100a70 2024-03-18T22:42:30.597 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:30.597 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":91,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:41:43.993830+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:42:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.598+0000 7f30f460b640 1 -- 172.21.15.40:0/1479115768 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f30d40a2800 msgr2=0x7f30d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.598+0000 7f30f460b640 1 --2- 172.21.15.40:0/1479115768 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f30d40a2800 0x7f30d40a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f30ec19b100 tx=0x7f30e4005f90 comp rx=0 tx=0).stop 2024-03-18T22:42:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.598+0000 7f30f460b640 1 -- 172.21.15.40:0/1479115768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30ec100a70 msgr2=0x7f30ec19a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.598+0000 7f30f460b640 1 --2- 172.21.15.40:0/1479115768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30ec100a70 0x7f30ec19a170 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f30e000e9c0 tx=0x7f30e000ee80 comp rx=0 tx=0).stop 2024-03-18T22:42:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.598+0000 7f30f460b640 1 -- 172.21.15.40:0/1479115768 shutdown_connections 2024-03-18T22:42:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.598+0000 7f30f460b640 1 --2- 172.21.15.40:0/1479115768 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f30d40a2800 0x7f30d40a4cc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.598+0000 7f30f460b640 1 --2- 172.21.15.40:0/1479115768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30ec101410 0x7f30ec1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.598+0000 7f30f460b640 1 --2- 172.21.15.40:0/1479115768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30ec100a70 0x7f30ec19a170 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.598+0000 7f30f460b640 1 --2- 172.21.15.40:0/1479115768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30ec100110 0x7f30ec199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.598+0000 7f30f460b640 1 -- 172.21.15.40:0/1479115768 >> 172.21.15.40:0/1479115768 conn(0x7f30ec0fbcb0 msgr2=0x7f30ec0fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.598+0000 7f30f460b640 1 -- 172.21.15.40:0/1479115768 shutdown_connections 2024-03-18T22:42:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.599+0000 7f30f460b640 1 -- 172.21.15.40:0/1479115768 wait complete. 2024-03-18T22:42:30.611 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:42:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.712+0000 7ff4245df640 1 -- 172.21.15.40:0/2409973790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff41c100740 msgr2=0x7ff41c100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.712+0000 7ff4245df640 1 --2- 172.21.15.40:0/2409973790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff41c100740 0x7ff41c100b60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff414009900 tx=0x7ff41402bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.712+0000 7ff4245df640 1 -- 172.21.15.40:0/2409973790 shutdown_connections 2024-03-18T22:42:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.712+0000 7ff4245df640 1 --2- 172.21.15.40:0/2409973790 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff41c1022f0 0x7ff41c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff4245df640 1 --2- 172.21.15.40:0/2409973790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff41c101930 0x7ff41c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff4245df640 1 --2- 172.21.15.40:0/2409973790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff41c100740 0x7ff41c100b60 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff4245df640 1 -- 172.21.15.40:0/2409973790 >> 172.21.15.40:0/2409973790 conn(0x7ff41c0fbcb0 msgr2=0x7ff41c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff4245df640 1 -- 172.21.15.40:0/2409973790 shutdown_connections 2024-03-18T22:42:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff4245df640 1 -- 172.21.15.40:0/2409973790 wait complete. 2024-03-18T22:42:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff4245df640 1 Processor -- start 2024-03-18T22:42:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff4245df640 1 -- start start 2024-03-18T22:42:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff4245df640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff41c100740 0x7ff41c10adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff4245df640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff41c101930 0x7ff41c109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff4245df640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff41c1022f0 0x7ff41c109940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff4245df640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff41c10d380 con 0x7ff41c100740 2024-03-18T22:42:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff4245df640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff41c10d200 con 0x7ff41c101930 2024-03-18T22:42:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff4245df640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff41c10d500 con 0x7ff41c1022f0 2024-03-18T22:42:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff42237b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff41c100740 0x7ff41c10adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff42237b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff41c100740 0x7ff41c10adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50874/0 (socket says 172.21.15.40:50874) 2024-03-18T22:42:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff42237b640 1 -- 172.21.15.40:0/1065282537 learned_addr learned my addr 172.21.15.40:0/1065282537 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff422b7c640 1 --2- 172.21.15.40:0/1065282537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff41c1022f0 0x7ff41c109940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.713+0000 7ff421b7a640 1 --2- 172.21.15.40:0/1065282537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff41c101930 0x7ff41c109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.714+0000 7ff42237b640 1 -- 172.21.15.40:0/1065282537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff41c1022f0 msgr2=0x7ff41c109940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.714+0000 7ff42237b640 1 --2- 172.21.15.40:0/1065282537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff41c1022f0 0x7ff41c109940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.714+0000 7ff42237b640 1 -- 172.21.15.40:0/1065282537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff41c101930 msgr2=0x7ff41c109400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.714+0000 7ff42237b640 1 --2- 172.21.15.40:0/1065282537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff41c101930 0x7ff41c109400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.714+0000 7ff42237b640 1 -- 172.21.15.40:0/1065282537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff41c10a1d0 con 0x7ff41c100740 2024-03-18T22:42:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.714+0000 7ff421b7a640 1 --2- 172.21.15.40:0/1065282537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff41c101930 0x7ff41c109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:42:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.714+0000 7ff42237b640 1 --2- 172.21.15.40:0/1065282537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff41c100740 0x7ff41c10adc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff414000c00 tx=0x7ff414005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.714+0000 7ff41b7fe640 1 -- 172.21.15.40:0/1065282537 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff414043020 con 0x7ff41c100740 2024-03-18T22:42:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.714+0000 7ff4245df640 1 -- 172.21.15.40:0/1065282537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff41c1a1b80 con 0x7ff41c100740 2024-03-18T22:42:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.714+0000 7ff41b7fe640 1 -- 172.21.15.40:0/1065282537 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff414004350 con 0x7ff41c100740 2024-03-18T22:42:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.714+0000 7ff41b7fe640 1 -- 172.21.15.40:0/1065282537 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff414003b80 con 0x7ff41c100740 2024-03-18T22:42:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.714+0000 7ff4245df640 1 -- 172.21.15.40:0/1065282537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff41c1a2120 con 0x7ff41c100740 2024-03-18T22:42:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.715+0000 7ff4245df640 1 -- 172.21.15.40:0/1065282537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff41c107b30 con 0x7ff41c100740 2024-03-18T22:42:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.716+0000 7ff41b7fe640 1 -- 172.21.15.40:0/1065282537 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff414046020 con 0x7ff41c100740 2024-03-18T22:42:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.717+0000 7ff41b7fe640 1 --2- 172.21.15.40:0/1065282537 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff4040a2800 0x7ff4040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.717+0000 7ff421b7a640 1 --2- 172.21.15.40:0/1065282537 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff4040a2800 0x7ff4040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.717+0000 7ff41b7fe640 1 -- 172.21.15.40:0/1065282537 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff4140f1350 con 0x7ff41c100740 2024-03-18T22:42:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.717+0000 7ff421b7a640 1 --2- 172.21.15.40:0/1065282537 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff4040a2800 0x7ff4040a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff41c10ab30 tx=0x7ff410005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:30.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.720+0000 7ff41b7fe640 1 -- 172.21.15.40:0/1065282537 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff41404d020 con 0x7ff41c100740 2024-03-18T22:42:30.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.944+0000 7ff4245df640 1 -- 172.21.15.40:0/1065282537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff41c02d1f0 con 0x7ff41c100740 2024-03-18T22:42:30.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.944+0000 7ff41b7fe640 1 -- 172.21.15.40:0/1065282537 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7ff4140be090 con 0x7ff41c100740 2024-03-18T22:42:30.945 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:30.945 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:41:09.839105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24481,24509,24473,24401,24388]},"id":1}]} 2024-03-18T22:42:30.946 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 27 2024-03-18T22:42:30.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.947+0000 7ff4245df640 1 -- 172.21.15.40:0/1065282537 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff4040a2800 msgr2=0x7ff4040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:30.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.947+0000 7ff4245df640 1 --2- 172.21.15.40:0/1065282537 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff4040a2800 0x7ff4040a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff41c10ab30 tx=0x7ff410005f90 comp rx=0 tx=0).stop 2024-03-18T22:42:30.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.947+0000 7ff4245df640 1 -- 172.21.15.40:0/1065282537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff41c100740 msgr2=0x7ff41c10adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:30.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.947+0000 7ff4245df640 1 --2- 172.21.15.40:0/1065282537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff41c100740 0x7ff41c10adc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff414000c00 tx=0x7ff414005dc0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.947+0000 7ff4245df640 1 -- 172.21.15.40:0/1065282537 shutdown_connections 2024-03-18T22:42:30.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.947+0000 7ff4245df640 1 --2- 172.21.15.40:0/1065282537 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff4040a2800 0x7ff4040a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.947+0000 7ff4245df640 1 --2- 172.21.15.40:0/1065282537 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff41c1022f0 0x7ff41c109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.947+0000 7ff4245df640 1 --2- 172.21.15.40:0/1065282537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff41c101930 0x7ff41c109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.947+0000 7ff4245df640 1 --2- 172.21.15.40:0/1065282537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff41c100740 0x7ff41c10adc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:30.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.947+0000 7ff4245df640 1 -- 172.21.15.40:0/1065282537 >> 172.21.15.40:0/1065282537 conn(0x7ff41c0fbcb0 msgr2=0x7ff41c104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:30.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.947+0000 7ff4245df640 1 -- 172.21.15.40:0/1065282537 shutdown_connections 2024-03-18T22:42:30.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:30.947+0000 7ff4245df640 1 -- 172.21.15.40:0/1065282537 wait complete. 2024-03-18T22:42:30.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T22:40:41.572773+0000', 'modified': '2024-03-18T22:41:09.839105+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24509, 'mds_1': 24481, 'mds_2': 24473, 'mds_3': 24401, 'mds_4': 24388}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14592': {'gid': 14592, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6839/4264194754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 4264194754}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 4264194754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14604': {'gid': 14604, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6841/1807327423', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6840', 'nonce': 1807327423}, {'type': 'v1', 'addr': '172.21.15.40:6841', 'nonce': 1807327423}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24388': {'gid': 24388, 'name': 'l', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.113:6833/2086888629', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6832', 'nonce': 2086888629}, {'type': 'v1', 'addr': '172.21.15.113:6833', 'nonce': 2086888629}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24401': {'gid': 24401, 'name': 'k', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.103:6833/3761932784', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6832', 'nonce': 3761932784}, {'type': 'v1', 'addr': '172.21.15.103:6833', 'nonce': 3761932784}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.103:6835/2536694691', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6834', 'nonce': 2536694691}, {'type': 'v1', 'addr': '172.21.15.103:6835', 'nonce': 2536694691}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.113:6835/4040610039', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6834', 'nonce': 4040610039}, {'type': 'v1', 'addr': '172.21.15.113:6835', 'nonce': 4040610039}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.113:6837/4212429112', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6836', 'nonce': 4212429112}, {'type': 'v1', 'addr': '172.21.15.113:6837', 'nonce': 4212429112}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.103:6837/1713853423', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6836', 'nonce': 1713853423}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 1713853423}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.113:6839/2986827418', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6838', 'nonce': 2986827418}, {'type': 'v1', 'addr': '172.21.15.113:6839', 'nonce': 2986827418}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.103:6839/2609728202', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6838', 'nonce': 2609728202}, {'type': 'v1', 'addr': '172.21.15.103:6839', 'nonce': 2609728202}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24509, 'qdb_cluster': [24481, 24509, 24473, 24401, 24388]} 2024-03-18T22:42:30.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-18T22:42:30.959 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-03-18T22:42:31.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.062+0000 7fec6d885640 1 -- 172.21.15.40:0/985506836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec680ffb20 msgr2=0x7fec680fff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.062+0000 7fec6d885640 1 --2- 172.21.15.40:0/985506836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec680ffb20 0x7fec680fff40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fec58009930 tx=0x7fec5802bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:31.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.062+0000 7fec6d885640 1 -- 172.21.15.40:0/985506836 shutdown_connections 2024-03-18T22:42:31.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.062+0000 7fec6d885640 1 --2- 172.21.15.40:0/985506836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec68103550 0x7fec68109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.062+0000 7fec6d885640 1 --2- 172.21.15.40:0/985506836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec68102b90 0x7fec68103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.062+0000 7fec6d885640 1 --2- 172.21.15.40:0/985506836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec680ffb20 0x7fec680fff40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.062+0000 7fec6d885640 1 -- 172.21.15.40:0/985506836 >> 172.21.15.40:0/985506836 conn(0x7fec680fb450 msgr2=0x7fec680fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.062+0000 7fec6d885640 1 -- 172.21.15.40:0/985506836 shutdown_connections 2024-03-18T22:42:31.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.062+0000 7fec6d885640 1 -- 172.21.15.40:0/985506836 wait complete. 2024-03-18T22:42:31.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec6d885640 1 Processor -- start 2024-03-18T22:42:31.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec6d885640 1 -- start start 2024-03-18T22:42:31.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec6d885640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec680ffb20 0x7fec68197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec6d885640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec68102b90 0x7fec68198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec6d885640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec68103550 0x7fec6819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec6d885640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fec6810d360 con 0x7fec68102b90 2024-03-18T22:42:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec6d885640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fec6810d1e0 con 0x7fec68103550 2024-03-18T22:42:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec6d885640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fec6810d4e0 con 0x7fec680ffb20 2024-03-18T22:42:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec667fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec68102b90 0x7fec68198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec667fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec68102b90 0x7fec68198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50888/0 (socket says 172.21.15.40:50888) 2024-03-18T22:42:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec667fc640 1 -- 172.21.15.40:0/3928451761 learned_addr learned my addr 172.21.15.40:0/3928451761 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec677fe640 1 --2- 172.21.15.40:0/3928451761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec68103550 0x7fec6819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec66ffd640 1 --2- 172.21.15.40:0/3928451761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec680ffb20 0x7fec68197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec667fc640 1 -- 172.21.15.40:0/3928451761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec680ffb20 msgr2=0x7fec68197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec667fc640 1 --2- 172.21.15.40:0/3928451761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec680ffb20 0x7fec68197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec667fc640 1 -- 172.21.15.40:0/3928451761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec68103550 msgr2=0x7fec6819f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec667fc640 1 --2- 172.21.15.40:0/3928451761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec68103550 0x7fec6819f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec667fc640 1 -- 172.21.15.40:0/3928451761 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec6819f700 con 0x7fec68102b90 2024-03-18T22:42:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.063+0000 7fec667fc640 1 --2- 172.21.15.40:0/3928451761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec68102b90 0x7fec68198020 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00e9f0 tx=0x7fec5c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.064+0000 7fec6c883640 1 -- 172.21.15.40:0/3928451761 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c00cd80 con 0x7fec68102b90 2024-03-18T22:42:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.064+0000 7fec6c883640 1 -- 172.21.15.40:0/3928451761 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec5c0108e0 con 0x7fec68102b90 2024-03-18T22:42:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.064+0000 7fec6d885640 1 -- 172.21.15.40:0/3928451761 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec6819f9f0 con 0x7fec68102b90 2024-03-18T22:42:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.064+0000 7fec6c883640 1 -- 172.21.15.40:0/3928451761 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c018920 con 0x7fec68102b90 2024-03-18T22:42:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.064+0000 7fec6d885640 1 -- 172.21.15.40:0/3928451761 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec6819ff40 con 0x7fec68102b90 2024-03-18T22:42:31.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.066+0000 7fec6c883640 1 -- 172.21.15.40:0/3928451761 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fec5c010a90 con 0x7fec68102b90 2024-03-18T22:42:31.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.066+0000 7fec70476740 1 Processor -- start 2024-03-18T22:42:31.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.066+0000 7fec70476740 1 -- start start 2024-03-18T22:42:31.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.066+0000 7fec70476740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076e6400 0x5641076e6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.066+0000 7fec70476740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6df0 0x5641076eb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.066+0000 7fec70476740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076ebf60 0x5641076ee3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.066+0000 7fec70476740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564107646de0 con 0x5641076e6400 2024-03-18T22:42:31.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.066+0000 7fec70476740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5641075efc80 con 0x5641076e6df0 2024-03-18T22:42:31.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.066+0000 7fec70476740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x564107642e80 con 0x5641076ebf60 2024-03-18T22:42:31.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.066+0000 7fec6c883640 1 --2- 172.21.15.40:0/3928451761 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec500a2800 0x7fec500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.066+0000 7fec66ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6df0 0x5641076eb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.066+0000 7fec677fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076e6400 0x5641076e6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.066+0000 7fec667fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076ebf60 0x5641076ee3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.066+0000 7fec677fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076e6400 0x5641076e6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50890/0 (socket says 172.21.15.40:50890) 2024-03-18T22:42:31.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.067+0000 7fec66ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6df0 0x5641076eb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49296/0 (socket says 172.21.15.40:49296) 2024-03-18T22:42:31.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.067+0000 7fec677fe640 1 -- 172.21.15.40:0/1709123326 learned_addr learned my addr 172.21.15.40:0/1709123326 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.067+0000 7fec677fe640 1 --2- 172.21.15.40:0/3928451761 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec500a2800 0x7fec500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.067+0000 7fec6c883640 1 -- 172.21.15.40:0/3928451761 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fec5c0cf0b0 con 0x7fec68102b90 2024-03-18T22:42:31.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.067+0000 7fec66ffd640 1 -- 172.21.15.40:0/1709123326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076ebf60 msgr2=0x5641076ee3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.067+0000 7fec66ffd640 1 --2- 172.21.15.40:0/1709123326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076ebf60 0x5641076ee3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.067+0000 7fec66ffd640 1 -- 172.21.15.40:0/1709123326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076e6400 msgr2=0x5641076e6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.067+0000 7fec66ffd640 1 --2- 172.21.15.40:0/1709123326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076e6400 0x5641076e6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.067+0000 7fec66ffd640 1 -- 172.21.15.40:0/1709123326 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564107601fb0 con 0x5641076e6df0 2024-03-18T22:42:31.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.067+0000 7fec677fe640 1 --2- 172.21.15.40:0/3928451761 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec500a2800 0x7fec500a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5641076e7650 tx=0x7fec60009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.067+0000 7fec66ffd640 1 --2- 172.21.15.40:0/1709123326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6df0 0x5641076eb8f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fec5802c170 tx=0x7fec58009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db9ffc2dda4de111 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.067+0000 7fec4dffb640 1 -- 172.21.15.40:0/1709123326 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec58045020 con 0x5641076e6df0 2024-03-18T22:42:31.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.067+0000 7fec4dffb640 1 -- 172.21.15.40:0/1709123326 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec58004040 con 0x5641076e6df0 2024-03-18T22:42:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.067+0000 7fec4dffb640 1 -- 172.21.15.40:0/1709123326 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec580055d0 con 0x5641076e6df0 2024-03-18T22:42:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.068+0000 7fec70476740 1 -- 172.21.15.40:0/1709123326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6df0 msgr2=0x5641076eb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.068+0000 7fec70476740 1 --2- 172.21.15.40:0/1709123326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6df0 0x5641076eb8f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fec5802c170 tx=0x7fec58009560 comp rx=0 tx=0).stop 2024-03-18T22:42:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.068+0000 7fec70476740 1 -- 172.21.15.40:0/1709123326 shutdown_connections 2024-03-18T22:42:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.068+0000 7fec70476740 1 --2- 172.21.15.40:0/1709123326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076ebf60 0x5641076ee3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.068+0000 7fec70476740 1 --2- 172.21.15.40:0/1709123326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6df0 0x5641076eb8f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.068+0000 7fec70476740 1 --2- 172.21.15.40:0/1709123326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076e6400 0x5641076e6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.068+0000 7fec70476740 1 -- 172.21.15.40:0/1709123326 >> 172.21.15.40:0/1709123326 conn(0x5641076dd560 msgr2=0x5641076dd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.068+0000 7fec70476740 1 -- 172.21.15.40:0/1709123326 shutdown_connections 2024-03-18T22:42:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.068+0000 7fec70476740 1 -- 172.21.15.40:0/1709123326 wait complete. 2024-03-18T22:42:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.068+0000 7fec70476740 1 -- start start 2024-03-18T22:42:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.068+0000 7fec70476740 1 Processor -- start 2024-03-18T22:42:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec70476740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6400 0x5641076f08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec70476740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076e6df0 0x5641076f0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec70476740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076ebf60 0x5641076f7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec70476740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5641076eeeb0 con 0x5641076ebf60 2024-03-18T22:42:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec70476740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564107602fc0 con 0x5641076e6400 2024-03-18T22:42:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec70476740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5641076ef030 con 0x5641076e6df0 2024-03-18T22:42:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec70476740 10 client.0 _ll_register_callbacks cb 0x56410763ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec66ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6400 0x5641076f08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec677fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076ebf60 0x5641076f7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec667fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076e6df0 0x5641076f0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec66ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6400 0x5641076f08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49302/0 (socket says 172.21.15.40:49302) 2024-03-18T22:42:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec66ffd640 1 -- 172.21.15.40:0/2462005927 learned_addr learned my addr 172.21.15.40:0/2462005927 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec66ffd640 1 -- 172.21.15.40:0/2462005927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076e6df0 msgr2=0x5641076f0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec66ffd640 1 --2- 172.21.15.40:0/2462005927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076e6df0 0x5641076f0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec66ffd640 1 -- 172.21.15.40:0/2462005927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076ebf60 msgr2=0x5641076f7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec66ffd640 1 --2- 172.21.15.40:0/2462005927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076ebf60 0x5641076f7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec66ffd640 1 -- 172.21.15.40:0/2462005927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56410735b2d0 con 0x5641076e6400 2024-03-18T22:42:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec66ffd640 1 --2- 172.21.15.40:0/2462005927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6400 0x5641076f08f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fec58009230 tx=0x7fec58004480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff0937df70455cd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:42:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.069+0000 7fec46ffd640 1 -- 172.21.15.40:0/2462005927 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec58045070 con 0x5641076e6400 2024-03-18T22:42:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.070+0000 7fec70476740 1 -- 172.21.15.40:0/2462005927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec58004040 con 0x5641076e6400 2024-03-18T22:42:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.070+0000 7fec46ffd640 1 -- 172.21.15.40:0/2462005927 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fec580035f0 con 0x5641076e6400 2024-03-18T22:42:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.070+0000 7fec46ffd640 1 -- 172.21.15.40:0/2462005927 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fec5805a050 con 0x5641076e6400 2024-03-18T22:42:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.070+0000 7fec70476740 10 client.25079 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.070+0000 7fec70476740 1 -- 172.21.15.40:0/2462005927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5641074256b0 con 0x5641076e6400 2024-03-18T22:42:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.070+0000 7fec46ffd640 1 -- 172.21.15.40:0/2462005927 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec58004040 con 0x5641076e6400 2024-03-18T22:42:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.070+0000 7fec46ffd640 1 -- 172.21.15.40:0/2462005927 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fec580049b0 con 0x5641076e6400 2024-03-18T22:42:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.070+0000 7fec70476740 10 client.25079 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.070+0000 7fec70476740 10 client.25079 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-18T22:42:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.070+0000 7fec70476740 20 client.25079 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.070+0000 7fec70476740 1 --2- 172.21.15.40:0/2462005927 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x5641076fd2c0 0x5641076fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.070+0000 7fec70476740 4 client.25079 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:42:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.070+0000 7fec70476740 1 -- 172.21.15.40:0/2462005927 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564107654320 con 0x5641076fd2c0 2024-03-18T22:42:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.070+0000 7fec667fc640 1 --2- 172.21.15.40:0/2462005927 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x5641076fd2c0 0x5641076fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.071+0000 7fec667fc640 1 --2- 172.21.15.40:0/2462005927 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x5641076fd2c0 0x5641076fe730 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=749966d7b3fa0249 server_cookie=380937c7ea363497 in_seq=0 out_seq=0 2024-03-18T22:42:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.071+0000 7fec46ffd640 10 client.25079 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:42:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.072+0000 7fec46ffd640 1 -- 172.21.15.40:0/2462005927 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564107654320 con 0x5641076fd2c0 2024-03-18T22:42:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.072+0000 7fec46ffd640 10 client.25079 handle_command_reply: tid=0 2024-03-18T22:42:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.072+0000 7fec70476740 1 client.25079 shutdown 2024-03-18T22:42:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.072+0000 7fec70476740 1 -- 172.21.15.40:0/2462005927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6400 msgr2=0x5641076f08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.072+0000 7fec70476740 1 --2- 172.21.15.40:0/2462005927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6400 0x5641076f08f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fec58009230 tx=0x7fec58004480 comp rx=0 tx=0).stop 2024-03-18T22:42:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.072+0000 7fec70476740 1 -- 172.21.15.40:0/2462005927 shutdown_connections 2024-03-18T22:42:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.072+0000 7fec70476740 1 --2- 172.21.15.40:0/2462005927 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x5641076fd2c0 0x5641076fe730 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.072+0000 7fec70476740 1 --2- 172.21.15.40:0/2462005927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076ebf60 0x5641076f7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.072+0000 7fec70476740 1 --2- 172.21.15.40:0/2462005927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076e6df0 0x5641076f0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.072+0000 7fec46ffd640 0 client.25079 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:42:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.072+0000 7fec70476740 1 --2- 172.21.15.40:0/2462005927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6400 0x5641076f08f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.072+0000 7fec70476740 1 -- 172.21.15.40:0/2462005927 >> 172.21.15.40:0/2462005927 conn(0x5641076dd560 msgr2=0x5641076dc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.072+0000 7fec70476740 1 -- 172.21.15.40:0/2462005927 shutdown_connections 2024-03-18T22:42:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.072+0000 7fec70476740 1 -- 172.21.15.40:0/2462005927 wait complete. 2024-03-18T22:42:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.072+0000 7fec70476740 20 client.25079 trim_cache size 0 max 16384 2024-03-18T22:42:31.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.090+0000 7fec70476740 1 Processor -- start 2024-03-18T22:42:31.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.090+0000 7fec70476740 1 -- start start 2024-03-18T22:42:31.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.090+0000 7fec70476740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6400 0x5641076e7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.090+0000 7fec70476740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076e6df0 0x5641076f7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.090+0000 7fec70476740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076ebf60 0x5641076dd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.090+0000 7fec70476740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5641076fca70 con 0x5641076ebf60 2024-03-18T22:42:31.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.090+0000 7fec70476740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5641076fd140 con 0x5641076e6400 2024-03-18T22:42:31.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.090+0000 7fec70476740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5641076fd810 con 0x5641076e6df0 2024-03-18T22:42:31.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.091+0000 7fec677fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076ebf60 0x5641076dd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.091+0000 7fec66ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6400 0x5641076e7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.091+0000 7fec677fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076ebf60 0x5641076dd940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50896/0 (socket says 172.21.15.40:50896) 2024-03-18T22:42:31.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.091+0000 7fec677fe640 1 -- 172.21.15.40:0/297301813 learned_addr learned my addr 172.21.15.40:0/297301813 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.091+0000 7fec667fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076e6df0 0x5641076f7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.091+0000 7fec677fe640 1 -- 172.21.15.40:0/297301813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076e6df0 msgr2=0x5641076f7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.091+0000 7fec677fe640 1 --2- 172.21.15.40:0/297301813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076e6df0 0x5641076f7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.091+0000 7fec677fe640 1 -- 172.21.15.40:0/297301813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6400 msgr2=0x5641076e7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.091+0000 7fec677fe640 1 --2- 172.21.15.40:0/297301813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6400 0x5641076e7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.091+0000 7fec677fe640 1 -- 172.21.15.40:0/297301813 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641076d8670 con 0x5641076ebf60 2024-03-18T22:42:31.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.091+0000 7fec677fe640 1 --2- 172.21.15.40:0/297301813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076ebf60 0x5641076dd940 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fec60011f90 tx=0x7fec60009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c898fcdc45a109d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.091+0000 7fec4d7fa640 1 -- 172.21.15.40:0/297301813 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6000fd80 con 0x5641076ebf60 2024-03-18T22:42:31.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.091+0000 7fec4d7fa640 1 -- 172.21.15.40:0/297301813 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec60004040 con 0x5641076ebf60 2024-03-18T22:42:31.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.091+0000 7fec4d7fa640 1 -- 172.21.15.40:0/297301813 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec60005280 con 0x5641076ebf60 2024-03-18T22:42:31.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 -- 172.21.15.40:0/297301813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076ebf60 msgr2=0x5641076dd940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 --2- 172.21.15.40:0/297301813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076ebf60 0x5641076dd940 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fec60011f90 tx=0x7fec60009b30 comp rx=0 tx=0).stop 2024-03-18T22:42:31.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 -- 172.21.15.40:0/297301813 shutdown_connections 2024-03-18T22:42:31.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 --2- 172.21.15.40:0/297301813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076ebf60 0x5641076dd940 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 --2- 172.21.15.40:0/297301813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076e6df0 0x5641076f7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 --2- 172.21.15.40:0/297301813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076e6400 0x5641076e7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 -- 172.21.15.40:0/297301813 >> 172.21.15.40:0/297301813 conn(0x5641076dd560 msgr2=0x5641076e1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 -- 172.21.15.40:0/297301813 shutdown_connections 2024-03-18T22:42:31.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 -- 172.21.15.40:0/297301813 wait complete. 2024-03-18T22:42:31.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 -- start start 2024-03-18T22:42:31.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 Processor -- start 2024-03-18T22:42:31.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076e6400 0x5641076f4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076e6df0 0x5641076f45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076ebf60 0x5641076f7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5641076fe5b0 con 0x5641076e6400 2024-03-18T22:42:31.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5641076fdee0 con 0x5641076ebf60 2024-03-18T22:42:31.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5641076fec80 con 0x5641076e6df0 2024-03-18T22:42:31.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec70476740 10 client.0 _ll_register_callbacks cb 0x7fec60011460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:31.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.092+0000 7fec66ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076e6400 0x5641076f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec66ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076e6400 0x5641076f4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50904/0 (socket says 172.21.15.40:50904) 2024-03-18T22:42:31.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec66ffd640 1 -- 172.21.15.40:0/3104031273 learned_addr learned my addr 172.21.15.40:0/3104031273 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec677fe640 1 --2- 172.21.15.40:0/3104031273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076ebf60 0x5641076f7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec667fc640 1 --2- 172.21.15.40:0/3104031273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076e6df0 0x5641076f45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec66ffd640 1 -- 172.21.15.40:0/3104031273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076e6df0 msgr2=0x5641076f45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec66ffd640 1 --2- 172.21.15.40:0/3104031273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076e6df0 0x5641076f45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec66ffd640 1 -- 172.21.15.40:0/3104031273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076ebf60 msgr2=0x5641076f7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec66ffd640 1 --2- 172.21.15.40:0/3104031273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076ebf60 0x5641076f7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec66ffd640 1 -- 172.21.15.40:0/3104031273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641076f7b80 con 0x5641076e6400 2024-03-18T22:42:31.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec66ffd640 1 --2- 172.21.15.40:0/3104031273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076e6400 0x5641076f4060 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fec58004f90 tx=0x7fec5802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b25b4fcc484360b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:31.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec46ffd640 1 -- 172.21.15.40:0/3104031273 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec58063060 con 0x5641076e6400 2024-03-18T22:42:31.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec46ffd640 1 -- 172.21.15.40:0/3104031273 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fec580048c0 con 0x5641076e6400 2024-03-18T22:42:31.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec70476740 1 -- 172.21.15.40:0/3104031273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec60004040 con 0x5641076e6400 2024-03-18T22:42:31.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec46ffd640 1 -- 172.21.15.40:0/3104031273 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fec5806a050 con 0x5641076e6400 2024-03-18T22:42:31.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec70476740 10 client.15195 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:31.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec70476740 1 -- 172.21.15.40:0/3104031273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564107763b50 con 0x5641076e6400 2024-03-18T22:42:31.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.093+0000 7fec46ffd640 1 -- 172.21.15.40:0/3104031273 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec60004040 con 0x5641076e6400 2024-03-18T22:42:31.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.094+0000 7fec46ffd640 1 -- 172.21.15.40:0/3104031273 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fec58041930 con 0x5641076e6400 2024-03-18T22:42:31.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.094+0000 7fec70476740 10 client.15195 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:31.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.094+0000 7fec70476740 10 client.15195 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-18T22:42:31.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.094+0000 7fec70476740 20 client.15195 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:31.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.094+0000 7fec70476740 1 --2- 172.21.15.40:0/3104031273 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x564107768760 0x564107767470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.094+0000 7fec70476740 4 client.15195 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T22:42:31.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.094+0000 7fec70476740 1 -- 172.21.15.40:0/3104031273 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "status"}) -- 0x56410764e670 con 0x564107768760 2024-03-18T22:42:31.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.094+0000 7fec667fc640 1 --2- 172.21.15.40:0/3104031273 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x564107768760 0x564107767470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.094+0000 7fec667fc640 1 --2- 172.21.15.40:0/3104031273 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x564107768760 0x564107767470 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a6251c75044cd06f server_cookie=5c51a80e789b64fc in_seq=0 out_seq=0 2024-03-18T22:42:31.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.094+0000 7fec46ffd640 10 client.15195 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:42:31.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.094+0000 7fec46ffd640 1 -- 172.21.15.40:0/3104031273 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56410764e670 con 0x564107768760 2024-03-18T22:42:31.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.094+0000 7fec46ffd640 10 client.15195 handle_command_reply: tid=0 2024-03-18T22:42:31.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.094+0000 7fec70476740 1 client.15195 shutdown 2024-03-18T22:42:31.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec70476740 1 -- 172.21.15.40:0/3104031273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076e6400 msgr2=0x5641076f4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec70476740 1 --2- 172.21.15.40:0/3104031273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076e6400 0x5641076f4060 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fec58004f90 tx=0x7fec5802bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:31.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec70476740 1 -- 172.21.15.40:0/3104031273 shutdown_connections 2024-03-18T22:42:31.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec70476740 1 --2- 172.21.15.40:0/3104031273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5641076ebf60 0x5641076f7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec70476740 1 --2- 172.21.15.40:0/3104031273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5641076e6df0 0x5641076f45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec70476740 1 --2- 172.21.15.40:0/3104031273 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x564107768760 0x564107767470 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec70476740 1 --2- 172.21.15.40:0/3104031273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5641076e6400 0x5641076f4060 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec46ffd640 0 client.15195 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:42:31.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec70476740 1 -- 172.21.15.40:0/3104031273 >> 172.21.15.40:0/3104031273 conn(0x5641076dd560 msgr2=0x5641076e07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec70476740 1 -- 172.21.15.40:0/3104031273 shutdown_connections 2024-03-18T22:42:31.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec70476740 1 -- 172.21.15.40:0/3104031273 wait complete. 2024-03-18T22:42:31.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec70476740 20 client.15195 trim_cache size 0 max 16384 2024-03-18T22:42:31.104 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:42:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "410ef414-e577-11ee-95c9-87774f69a715", 2024-03-18T22:42:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 4, 2024-03-18T22:42:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24388, 2024-03-18T22:42:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-18T22:42:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-18T22:42:31.104 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-18T22:42:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 82.264113051999999, 2024-03-18T22:42:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 27, 2024-03-18T22:42:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 91, 2024-03-18T22:42:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T22:42:31.105 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 144.648712265 2024-03-18T22:42:31.105 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:42:31.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec6d885640 1 -- 172.21.15.40:0/3928451761 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec500a2800 msgr2=0x7fec500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec6d885640 1 --2- 172.21.15.40:0/3928451761 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec500a2800 0x7fec500a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5641076e7650 tx=0x7fec60009500 comp rx=0 tx=0).stop 2024-03-18T22:42:31.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec6d885640 1 -- 172.21.15.40:0/3928451761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec68102b90 msgr2=0x7fec68198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec6d885640 1 --2- 172.21.15.40:0/3928451761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec68102b90 0x7fec68198020 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00e9f0 tx=0x7fec5c00eeb0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec6d885640 1 -- 172.21.15.40:0/3928451761 shutdown_connections 2024-03-18T22:42:31.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec6d885640 1 --2- 172.21.15.40:0/3928451761 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec500a2800 0x7fec500a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec6d885640 1 --2- 172.21.15.40:0/3928451761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec68103550 0x7fec6819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec6d885640 1 --2- 172.21.15.40:0/3928451761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec68102b90 0x7fec68198020 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec6d885640 1 --2- 172.21.15.40:0/3928451761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec680ffb20 0x7fec68197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec6d885640 1 -- 172.21.15.40:0/3928451761 >> 172.21.15.40:0/3928451761 conn(0x7fec680fb450 msgr2=0x7fec68105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec6d885640 1 -- 172.21.15.40:0/3928451761 shutdown_connections 2024-03-18T22:42:31.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.095+0000 7fec6d885640 1 -- 172.21.15.40:0/3928451761 wait complete. 2024-03-18T22:42:31.108 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-03-18T22:42:31.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 -- 172.21.15.40:0/1393899176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd258101920 msgr2=0x7fd258101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 --2- 172.21.15.40:0/1393899176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd258101920 0x7fd258101da0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd248009900 tx=0x7fd24802bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:31.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 -- 172.21.15.40:0/1393899176 shutdown_connections 2024-03-18T22:42:31.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 --2- 172.21.15.40:0/1393899176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2581022e0 0x7fd258108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 --2- 172.21.15.40:0/1393899176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd258101920 0x7fd258101da0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 --2- 172.21.15.40:0/1393899176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd258100730 0x7fd258100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 -- 172.21.15.40:0/1393899176 >> 172.21.15.40:0/1393899176 conn(0x7fd2580fbcc0 msgr2=0x7fd2580fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 -- 172.21.15.40:0/1393899176 shutdown_connections 2024-03-18T22:42:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 -- 172.21.15.40:0/1393899176 wait complete. 2024-03-18T22:42:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 Processor -- start 2024-03-18T22:42:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 -- start start 2024-03-18T22:42:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd258100730 0x7fd25810b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd258101920 0x7fd2581096f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2581022e0 0x7fd258109c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd25810d380 con 0x7fd258100730 2024-03-18T22:42:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd25810d200 con 0x7fd2581022e0 2024-03-18T22:42:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.211+0000 7fd25f40b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd25810d500 con 0x7fd258101920 2024-03-18T22:42:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd25d1a7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd258100730 0x7fd25810b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd25d1a7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd258100730 0x7fd25810b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50932/0 (socket says 172.21.15.40:50932) 2024-03-18T22:42:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd25d1a7640 1 -- 172.21.15.40:0/2648418580 learned_addr learned my addr 172.21.15.40:0/2648418580 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd25d9a8640 1 --2- 172.21.15.40:0/2648418580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2581022e0 0x7fd258109c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd25c9a6640 1 --2- 172.21.15.40:0/2648418580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd258101920 0x7fd2581096f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd25d1a7640 1 -- 172.21.15.40:0/2648418580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd258101920 msgr2=0x7fd2581096f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd25d1a7640 1 --2- 172.21.15.40:0/2648418580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd258101920 0x7fd2581096f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd25d1a7640 1 -- 172.21.15.40:0/2648418580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2581022e0 msgr2=0x7fd258109c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd25d1a7640 1 --2- 172.21.15.40:0/2648418580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2581022e0 0x7fd258109c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd25d1a7640 1 -- 172.21.15.40:0/2648418580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25810a4c0 con 0x7fd258100730 2024-03-18T22:42:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd25d1a7640 1 --2- 172.21.15.40:0/2648418580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd258100730 0x7fd25810b0b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd25400e9c0 tx=0x7fd25400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd24e7fc640 1 -- 172.21.15.40:0/2648418580 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25400cd80 con 0x7fd258100730 2024-03-18T22:42:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd24e7fc640 1 -- 172.21.15.40:0/2648418580 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2540035f0 con 0x7fd258100730 2024-03-18T22:42:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd24e7fc640 1 -- 172.21.15.40:0/2648418580 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254010690 con 0x7fd258100730 2024-03-18T22:42:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd25f40b640 1 -- 172.21.15.40:0/2648418580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2581a1b70 con 0x7fd258100730 2024-03-18T22:42:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.212+0000 7fd25f40b640 1 -- 172.21.15.40:0/2648418580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2581a2090 con 0x7fd258100730 2024-03-18T22:42:31.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.214+0000 7fd24e7fc640 1 -- 172.21.15.40:0/2648418580 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd2540037a0 con 0x7fd258100730 2024-03-18T22:42:31.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd261ffc740 1 Processor -- start 2024-03-18T22:42:31.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd261ffc740 1 -- start start 2024-03-18T22:42:31.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd261ffc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05cff0 0x55749e05d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd261ffc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e05d9e0 0x55749e0624e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd261ffc740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e062b50 0x55749e064fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd261ffc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55749dfc1de0 con 0x55749e05cff0 2024-03-18T22:42:31.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd261ffc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55749df6ac80 con 0x55749e05d9e0 2024-03-18T22:42:31.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd261ffc740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55749dfbde80 con 0x55749e062b50 2024-03-18T22:42:31.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd24e7fc640 1 --2- 172.21.15.40:0/2648418580 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd2440a2800 0x7fd2440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd25d9a8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05cff0 0x55749e05d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd25c9a6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e062b50 0x55749e064fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd25d1a7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e05d9e0 0x55749e0624e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd25d9a8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05cff0 0x55749e05d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50948/0 (socket says 172.21.15.40:50948) 2024-03-18T22:42:31.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd25d9a8640 1 -- 172.21.15.40:0/2865910314 learned_addr learned my addr 172.21.15.40:0/2865910314 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd25d9a8640 1 --2- 172.21.15.40:0/2648418580 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd2440a2800 0x7fd2440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd24e7fc640 1 -- 172.21.15.40:0/2648418580 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd2540cf1f0 con 0x7fd258100730 2024-03-18T22:42:31.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd25c9a6640 1 -- 172.21.15.40:0/2865910314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e05d9e0 msgr2=0x55749e0624e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd25c9a6640 1 --2- 172.21.15.40:0/2865910314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e05d9e0 0x55749e0624e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd25c9a6640 1 -- 172.21.15.40:0/2865910314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05cff0 msgr2=0x55749e05d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd25c9a6640 1 --2- 172.21.15.40:0/2865910314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05cff0 0x55749e05d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.215+0000 7fd25c9a6640 1 -- 172.21.15.40:0/2865910314 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55749df7cfb0 con 0x55749e062b50 2024-03-18T22:42:31.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.216+0000 7fd25d9a8640 1 --2- 172.21.15.40:0/2648418580 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd2440a2800 0x7fd2440a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x55749e05e240 tx=0x7fd250009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.216+0000 7fd25c9a6640 1 --2- 172.21.15.40:0/2865910314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e062b50 0x55749e064fe0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd248000c00 tx=0x7fd248009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86d8fa5d02d84bbf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.216+0000 7fd2437fe640 1 -- 172.21.15.40:0/2865910314 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd248045020 con 0x55749e062b50 2024-03-18T22:42:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.216+0000 7fd2437fe640 1 -- 172.21.15.40:0/2865910314 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd248004040 con 0x55749e062b50 2024-03-18T22:42:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.216+0000 7fd2437fe640 1 -- 172.21.15.40:0/2865910314 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd248005530 con 0x55749e062b50 2024-03-18T22:42:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.216+0000 7fd261ffc740 1 -- 172.21.15.40:0/2865910314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e062b50 msgr2=0x55749e064fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.216+0000 7fd261ffc740 1 --2- 172.21.15.40:0/2865910314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e062b50 0x55749e064fe0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd248000c00 tx=0x7fd248009560 comp rx=0 tx=0).stop 2024-03-18T22:42:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 1 -- 172.21.15.40:0/2865910314 shutdown_connections 2024-03-18T22:42:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 1 --2- 172.21.15.40:0/2865910314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e062b50 0x55749e064fe0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 1 --2- 172.21.15.40:0/2865910314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e05d9e0 0x55749e0624e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 1 --2- 172.21.15.40:0/2865910314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05cff0 0x55749e05d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 1 -- 172.21.15.40:0/2865910314 >> 172.21.15.40:0/2865910314 conn(0x55749e058560 msgr2=0x55749e058960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 1 -- 172.21.15.40:0/2865910314 shutdown_connections 2024-03-18T22:42:31.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 1 -- 172.21.15.40:0/2865910314 wait complete. 2024-03-18T22:42:31.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 1 -- start start 2024-03-18T22:42:31.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 1 Processor -- start 2024-03-18T22:42:31.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05cff0 0x55749e067450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e05d9e0 0x55749e067990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e062b50 0x55749e06e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55749e065aa0 con 0x55749e05cff0 2024-03-18T22:42:31.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55749df7dfc0 con 0x55749e062b50 2024-03-18T22:42:31.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55749e065c20 con 0x55749e05d9e0 2024-03-18T22:42:31.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd261ffc740 10 client.0 _ll_register_callbacks cb 0x55749dfb9a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:31.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd25d1a7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05cff0 0x55749e067450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd25d1a7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05cff0 0x55749e067450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50958/0 (socket says 172.21.15.40:50958) 2024-03-18T22:42:31.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd25d1a7640 1 -- 172.21.15.40:0/433965903 learned_addr learned my addr 172.21.15.40:0/433965903 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.217+0000 7fd25d9a8640 1 --2- 172.21.15.40:0/433965903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e062b50 0x55749e06e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd25c9a6640 1 --2- 172.21.15.40:0/433965903 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e05d9e0 0x55749e067990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd25d1a7640 1 -- 172.21.15.40:0/433965903 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e05d9e0 msgr2=0x55749e067990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd25d1a7640 1 --2- 172.21.15.40:0/433965903 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e05d9e0 0x55749e067990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd25d1a7640 1 -- 172.21.15.40:0/433965903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e062b50 msgr2=0x55749e06e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd25d1a7640 1 --2- 172.21.15.40:0/433965903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e062b50 0x55749e06e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd25d1a7640 1 -- 172.21.15.40:0/433965903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55749dcd62d0 con 0x55749e05cff0 2024-03-18T22:42:31.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd25d1a7640 1 --2- 172.21.15.40:0/433965903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05cff0 0x55749e067450 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd254018000 tx=0x7fd254066a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5147c19b370a8fc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd240ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:31.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd240ff9640 1 -- 172.21.15.40:0/433965903 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25409bed0 con 0x55749e05cff0 2024-03-18T22:42:31.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd240ff9640 1 -- 172.21.15.40:0/433965903 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd2540860d0 con 0x55749e05cff0 2024-03-18T22:42:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd261ffc740 1 -- 172.21.15.40:0/433965903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd248004040 con 0x55749e05cff0 2024-03-18T22:42:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd240ff9640 1 -- 172.21.15.40:0/433965903 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd2540a3f70 con 0x55749e05cff0 2024-03-18T22:42:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd261ffc740 10 client.15207 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd261ffc740 1 -- 172.21.15.40:0/433965903 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55749dda06b0 con 0x55749e05cff0 2024-03-18T22:42:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd240ff9640 1 -- 172.21.15.40:0/433965903 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd248004040 con 0x55749e05cff0 2024-03-18T22:42:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.218+0000 7fd240ff9640 1 -- 172.21.15.40:0/433965903 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fd25406a200 con 0x55749e05cff0 2024-03-18T22:42:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.219+0000 7fd261ffc740 10 client.15207 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.219+0000 7fd261ffc740 10 client.15207 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-18T22:42:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.219+0000 7fd261ffc740 20 client.15207 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.219+0000 7fd261ffc740 1 --2- 172.21.15.40:0/433965903 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55749e0742c0 0x55749e075730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.219+0000 7fd261ffc740 4 client.15207 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:42:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.219+0000 7fd261ffc740 1 -- 172.21.15.40:0/433965903 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55749dfcf320 con 0x55749e0742c0 2024-03-18T22:42:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.219+0000 7fd25c9a6640 1 --2- 172.21.15.40:0/433965903 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55749e0742c0 0x55749e075730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.219+0000 7fd25c9a6640 1 --2- 172.21.15.40:0/433965903 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55749e0742c0 0x55749e075730 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e88377c9fc50bf07 server_cookie=d20d73ee9a39f664 in_seq=0 out_seq=0 2024-03-18T22:42:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.219+0000 7fd240ff9640 10 client.15207 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:42:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.220+0000 7fd240ff9640 1 -- 172.21.15.40:0/433965903 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55749dfcf320 con 0x55749e0742c0 2024-03-18T22:42:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.220+0000 7fd240ff9640 10 client.15207 handle_command_reply: tid=0 2024-03-18T22:42:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.220+0000 7fd261ffc740 1 client.15207 shutdown 2024-03-18T22:42:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.220+0000 7fd261ffc740 1 -- 172.21.15.40:0/433965903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05cff0 msgr2=0x55749e067450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.220+0000 7fd261ffc740 1 --2- 172.21.15.40:0/433965903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05cff0 0x55749e067450 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd254018000 tx=0x7fd254066a30 comp rx=0 tx=0).stop 2024-03-18T22:42:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.220+0000 7fd261ffc740 1 -- 172.21.15.40:0/433965903 shutdown_connections 2024-03-18T22:42:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.220+0000 7fd261ffc740 1 --2- 172.21.15.40:0/433965903 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55749e0742c0 0x55749e075730 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.220+0000 7fd261ffc740 1 --2- 172.21.15.40:0/433965903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e062b50 0x55749e06e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.220+0000 7fd261ffc740 1 --2- 172.21.15.40:0/433965903 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e05d9e0 0x55749e067990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.220+0000 7fd261ffc740 1 --2- 172.21.15.40:0/433965903 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05cff0 0x55749e067450 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.220+0000 7fd240ff9640 0 client.15207 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:42:31.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.220+0000 7fd261ffc740 1 -- 172.21.15.40:0/433965903 >> 172.21.15.40:0/433965903 conn(0x55749e058560 msgr2=0x55749e0577e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.220+0000 7fd261ffc740 1 -- 172.21.15.40:0/433965903 shutdown_connections 2024-03-18T22:42:31.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.220+0000 7fd261ffc740 1 -- 172.21.15.40:0/433965903 wait complete. 2024-03-18T22:42:31.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.220+0000 7fd261ffc740 20 client.15207 trim_cache size 0 max 16384 2024-03-18T22:42:31.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.238+0000 7fd261ffc740 1 Processor -- start 2024-03-18T22:42:31.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.238+0000 7fd261ffc740 1 -- start start 2024-03-18T22:42:31.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.238+0000 7fd261ffc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e058560 0x55749e05eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.238+0000 7fd261ffc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e05cff0 0x55749e05f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.238+0000 7fd261ffc740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e05d9e0 0x55749e06e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.238+0000 7fd261ffc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55749e073a70 con 0x55749e058560 2024-03-18T22:42:31.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.238+0000 7fd261ffc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55749e074140 con 0x55749e05cff0 2024-03-18T22:42:31.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.238+0000 7fd261ffc740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55749e074810 con 0x55749e05d9e0 2024-03-18T22:42:31.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.238+0000 7fd25d1a7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e058560 0x55749e05eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.238+0000 7fd25d1a7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e058560 0x55749e05eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50964/0 (socket says 172.21.15.40:50964) 2024-03-18T22:42:31.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.238+0000 7fd25d1a7640 1 -- 172.21.15.40:0/814967961 learned_addr learned my addr 172.21.15.40:0/814967961 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.239+0000 7fd25c9a6640 1 --2- 172.21.15.40:0/814967961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e05cff0 0x55749e05f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.239+0000 7fd25d9a8640 1 --2- 172.21.15.40:0/814967961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e05d9e0 0x55749e06e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.239+0000 7fd25d1a7640 1 -- 172.21.15.40:0/814967961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e05d9e0 msgr2=0x55749e06e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.239+0000 7fd25d1a7640 1 --2- 172.21.15.40:0/814967961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e05d9e0 0x55749e06e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.239+0000 7fd25d1a7640 1 -- 172.21.15.40:0/814967961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e05cff0 msgr2=0x55749e05f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.239+0000 7fd25d1a7640 1 --2- 172.21.15.40:0/814967961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e05cff0 0x55749e05f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.239+0000 7fd25d1a7640 1 -- 172.21.15.40:0/814967961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55749e05ddc0 con 0x55749e058560 2024-03-18T22:42:31.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.239+0000 7fd25c9a6640 1 --2- 172.21.15.40:0/814967961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e05cff0 0x55749e05f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:31.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.239+0000 7fd25d1a7640 1 --2- 172.21.15.40:0/814967961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e058560 0x55749e05eee0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd2540058c0 tx=0x7fd25409bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8961dda8caf315f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.239+0000 7fd242ffd640 1 -- 172.21.15.40:0/814967961 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254064070 con 0x55749e058560 2024-03-18T22:42:31.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.239+0000 7fd242ffd640 1 -- 172.21.15.40:0/814967961 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd25409bed0 con 0x55749e058560 2024-03-18T22:42:31.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.239+0000 7fd242ffd640 1 -- 172.21.15.40:0/814967961 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254066310 con 0x55749e058560 2024-03-18T22:42:31.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 -- 172.21.15.40:0/814967961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e058560 msgr2=0x55749e05eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 --2- 172.21.15.40:0/814967961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e058560 0x55749e05eee0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd2540058c0 tx=0x7fd25409bd60 comp rx=0 tx=0).stop 2024-03-18T22:42:31.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 -- 172.21.15.40:0/814967961 shutdown_connections 2024-03-18T22:42:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 --2- 172.21.15.40:0/814967961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e05d9e0 0x55749e06e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 --2- 172.21.15.40:0/814967961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e05cff0 0x55749e05f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 --2- 172.21.15.40:0/814967961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e058560 0x55749e05eee0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 -- 172.21.15.40:0/814967961 >> 172.21.15.40:0/814967961 conn(0x55749e058cb0 msgr2=0x55749e073100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 -- 172.21.15.40:0/814967961 shutdown_connections 2024-03-18T22:42:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 -- 172.21.15.40:0/814967961 wait complete. 2024-03-18T22:42:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 -- start start 2024-03-18T22:42:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 Processor -- start 2024-03-18T22:42:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e058560 0x55749e06ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e05cff0 0x55749e06b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05d9e0 0x55749e06df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55749e0755b0 con 0x55749e05d9e0 2024-03-18T22:42:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55749e074ee0 con 0x55749e058560 2024-03-18T22:42:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55749e075c80 con 0x55749e05cff0 2024-03-18T22:42:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd261ffc740 10 client.0 _ll_register_callbacks cb 0x7fd254097ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd25d9a8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05d9e0 0x55749e06df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd25d9a8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05d9e0 0x55749e06df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50966/0 (socket says 172.21.15.40:50966) 2024-03-18T22:42:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd25d9a8640 1 -- 172.21.15.40:0/2039221212 learned_addr learned my addr 172.21.15.40:0/2039221212 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd25d1a7640 1 --2- 172.21.15.40:0/2039221212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e058560 0x55749e06ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.240+0000 7fd25c9a6640 1 --2- 172.21.15.40:0/2039221212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e05cff0 0x55749e06b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.241+0000 7fd25d9a8640 1 -- 172.21.15.40:0/2039221212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e05cff0 msgr2=0x55749e06b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.241+0000 7fd25d9a8640 1 --2- 172.21.15.40:0/2039221212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e05cff0 0x55749e06b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.241+0000 7fd25d9a8640 1 -- 172.21.15.40:0/2039221212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e058560 msgr2=0x55749e06ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.241+0000 7fd25d9a8640 1 --2- 172.21.15.40:0/2039221212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e058560 0x55749e06ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.241+0000 7fd25d9a8640 1 -- 172.21.15.40:0/2039221212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55749e06e5a0 con 0x55749e05d9e0 2024-03-18T22:42:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.241+0000 7fd25d9a8640 1 --2- 172.21.15.40:0/2039221212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05d9e0 0x55749e06df30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd2500099e0 tx=0x7fd250004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75532c3179a9db66 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.241+0000 7fd240ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.241+0000 7fd240ff9640 1 -- 172.21.15.40:0/2039221212 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25002f060 con 0x55749e05d9e0 2024-03-18T22:42:31.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.241+0000 7fd240ff9640 1 -- 172.21.15.40:0/2039221212 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd250004710 con 0x55749e05d9e0 2024-03-18T22:42:31.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.241+0000 7fd261ffc740 1 -- 172.21.15.40:0/2039221212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd25409bed0 con 0x55749e05d9e0 2024-03-18T22:42:31.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.241+0000 7fd240ff9640 1 -- 172.21.15.40:0/2039221212 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd250036050 con 0x55749e05d9e0 2024-03-18T22:42:31.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.241+0000 7fd261ffc740 10 client.15213 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:31.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.241+0000 7fd261ffc740 1 -- 172.21.15.40:0/2039221212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55749e0dab00 con 0x55749e05d9e0 2024-03-18T22:42:31.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.241+0000 7fd240ff9640 1 -- 172.21.15.40:0/2039221212 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd25409bed0 con 0x55749e05d9e0 2024-03-18T22:42:31.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.241+0000 7fd240ff9640 1 -- 172.21.15.40:0/2039221212 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fd25001d070 con 0x55749e05d9e0 2024-03-18T22:42:31.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd261ffc740 10 client.15213 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:31.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd261ffc740 10 client.15213 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-18T22:42:31.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd261ffc740 20 client.15213 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:31.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd261ffc740 1 --2- 172.21.15.40:0/2039221212 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55749e062b50 0x55749e0de420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd261ffc740 4 client.15213 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T22:42:31.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd261ffc740 1 -- 172.21.15.40:0/2039221212 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "status"}) -- 0x55749dfc9670 con 0x55749e062b50 2024-03-18T22:42:31.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd25d1a7640 1 --2- 172.21.15.40:0/2039221212 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55749e062b50 0x55749e0de420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd25d1a7640 1 --2- 172.21.15.40:0/2039221212 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55749e062b50 0x55749e0de420 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b0e562188383a739 server_cookie=2916e54017eeb1f2 in_seq=0 out_seq=0 2024-03-18T22:42:31.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd240ff9640 10 client.15213 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:42:31.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd240ff9640 1 -- 172.21.15.40:0/2039221212 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55749dfc9670 con 0x55749e062b50 2024-03-18T22:42:31.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd240ff9640 10 client.15213 handle_command_reply: tid=0 2024-03-18T22:42:31.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd261ffc740 1 client.15213 shutdown 2024-03-18T22:42:31.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd261ffc740 1 -- 172.21.15.40:0/2039221212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05d9e0 msgr2=0x55749e06df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd261ffc740 1 --2- 172.21.15.40:0/2039221212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05d9e0 0x55749e06df30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd2500099e0 tx=0x7fd250004ab0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd261ffc740 1 -- 172.21.15.40:0/2039221212 shutdown_connections 2024-03-18T22:42:31.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd261ffc740 1 --2- 172.21.15.40:0/2039221212 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55749e062b50 0x55749e0de420 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd261ffc740 1 --2- 172.21.15.40:0/2039221212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55749e05d9e0 0x55749e06df30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd261ffc740 1 --2- 172.21.15.40:0/2039221212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55749e05cff0 0x55749e06b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd240ff9640 0 client.15213 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:42:31.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd261ffc740 1 --2- 172.21.15.40:0/2039221212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55749e058560 0x55749e06ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.242+0000 7fd261ffc740 1 -- 172.21.15.40:0/2039221212 >> 172.21.15.40:0/2039221212 conn(0x55749e058cb0 msgr2=0x55749e05eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.243+0000 7fd261ffc740 1 -- 172.21.15.40:0/2039221212 shutdown_connections 2024-03-18T22:42:31.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.243+0000 7fd261ffc740 1 -- 172.21.15.40:0/2039221212 wait complete. 2024-03-18T22:42:31.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.243+0000 7fd261ffc740 20 client.15213 trim_cache size 0 max 16384 2024-03-18T22:42:31.251 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:42:31.251 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "410ef414-e577-11ee-95c9-87774f69a715", 2024-03-18T22:42:31.251 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 3, 2024-03-18T22:42:31.251 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24401, 2024-03-18T22:42:31.251 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-18T22:42:31.251 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-18T22:42:31.251 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-18T22:42:31.251 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 83.418784872000003, 2024-03-18T22:42:31.252 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 27, 2024-03-18T22:42:31.252 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 91, 2024-03-18T22:42:31.252 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T22:42:31.252 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 141.84683515 2024-03-18T22:42:31.252 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:42:31.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.243+0000 7fd25f40b640 1 -- 172.21.15.40:0/2648418580 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd2440a2800 msgr2=0x7fd2440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.243+0000 7fd25f40b640 1 --2- 172.21.15.40:0/2648418580 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd2440a2800 0x7fd2440a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x55749e05e240 tx=0x7fd250009500 comp rx=0 tx=0).stop 2024-03-18T22:42:31.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.243+0000 7fd25f40b640 1 -- 172.21.15.40:0/2648418580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd258100730 msgr2=0x7fd25810b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.243+0000 7fd25f40b640 1 --2- 172.21.15.40:0/2648418580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd258100730 0x7fd25810b0b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd25400e9c0 tx=0x7fd25400ee80 comp rx=0 tx=0).stop 2024-03-18T22:42:31.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.243+0000 7fd25f40b640 1 -- 172.21.15.40:0/2648418580 shutdown_connections 2024-03-18T22:42:31.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.243+0000 7fd25f40b640 1 --2- 172.21.15.40:0/2648418580 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd2440a2800 0x7fd2440a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.243+0000 7fd25f40b640 1 --2- 172.21.15.40:0/2648418580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2581022e0 0x7fd258109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.243+0000 7fd25f40b640 1 --2- 172.21.15.40:0/2648418580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd258101920 0x7fd2581096f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.243+0000 7fd25f40b640 1 --2- 172.21.15.40:0/2648418580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd258100730 0x7fd25810b0b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.243+0000 7fd25f40b640 1 -- 172.21.15.40:0/2648418580 >> 172.21.15.40:0/2648418580 conn(0x7fd2580fbcc0 msgr2=0x7fd258104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.243+0000 7fd25f40b640 1 -- 172.21.15.40:0/2648418580 shutdown_connections 2024-03-18T22:42:31.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.243+0000 7fd25f40b640 1 -- 172.21.15.40:0/2648418580 wait complete. 2024-03-18T22:42:31.254 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:30 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/4136493536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:42:31.254 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:30 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1644230716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:42:31.254 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:30 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1479115768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:31.255 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-18T22:42:31.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:30 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/4136493536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:42:31.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:30 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1644230716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:42:31.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:30 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1479115768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:31.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:30 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/4136493536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:42:31.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:30 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1644230716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:42:31.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:30 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1479115768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:31.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.359+0000 7ff33637e640 1 -- 172.21.15.40:0/3543641587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff330101050 msgr2=0x7ff3301014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.359+0000 7ff33637e640 1 --2- 172.21.15.40:0/3543641587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff330101050 0x7ff3301014d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff324009900 tx=0x7ff32402bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:31.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.359+0000 7ff33637e640 1 -- 172.21.15.40:0/3543641587 shutdown_connections 2024-03-18T22:42:31.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.359+0000 7ff33637e640 1 --2- 172.21.15.40:0/3543641587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff330101a10 0x7ff330108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.359+0000 7ff33637e640 1 --2- 172.21.15.40:0/3543641587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff330101050 0x7ff3301014d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.359+0000 7ff33637e640 1 --2- 172.21.15.40:0/3543641587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3300ffe70 0x7ff330100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.359+0000 7ff33637e640 1 -- 172.21.15.40:0/3543641587 >> 172.21.15.40:0/3543641587 conn(0x7ff3300fb450 msgr2=0x7ff3300fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.359+0000 7ff33637e640 1 -- 172.21.15.40:0/3543641587 shutdown_connections 2024-03-18T22:42:31.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.359+0000 7ff33637e640 1 -- 172.21.15.40:0/3543641587 wait complete. 2024-03-18T22:42:31.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.359+0000 7ff33637e640 1 Processor -- start 2024-03-18T22:42:31.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.359+0000 7ff33637e640 1 -- start start 2024-03-18T22:42:31.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff33637e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3300ffe70 0x7ff330197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff33637e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff330101050 0x7ff330198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff33637e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff330101a10 0x7ff33019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff33637e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff33010c290 con 0x7ff330101050 2024-03-18T22:42:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff33637e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff33010c110 con 0x7ff330101a10 2024-03-18T22:42:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff33637e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff33010c410 con 0x7ff3300ffe70 2024-03-18T22:42:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff32f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff330101050 0x7ff330198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff32f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff330101050 0x7ff330198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50980/0 (socket says 172.21.15.40:50980) 2024-03-18T22:42:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff32f7fe640 1 -- 172.21.15.40:0/142662880 learned_addr learned my addr 172.21.15.40:0/142662880 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff32ffff640 1 --2- 172.21.15.40:0/142662880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3300ffe70 0x7ff330197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff33491b640 1 --2- 172.21.15.40:0/142662880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff330101a10 0x7ff33019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff32f7fe640 1 -- 172.21.15.40:0/142662880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3300ffe70 msgr2=0x7ff330197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff32f7fe640 1 --2- 172.21.15.40:0/142662880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3300ffe70 0x7ff330197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff32f7fe640 1 -- 172.21.15.40:0/142662880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff330101a10 msgr2=0x7ff33019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff32f7fe640 1 --2- 172.21.15.40:0/142662880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff330101a10 0x7ff33019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff32f7fe640 1 -- 172.21.15.40:0/142662880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33019f700 con 0x7ff330101050 2024-03-18T22:42:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff32f7fe640 1 --2- 172.21.15.40:0/142662880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff330101050 0x7ff330198020 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff324006010 tx=0x7ff324005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.360+0000 7ff32d7fa640 1 -- 172.21.15.40:0/142662880 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324043020 con 0x7ff330101050 2024-03-18T22:42:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.361+0000 7ff33637e640 1 -- 172.21.15.40:0/142662880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff33019f990 con 0x7ff330101050 2024-03-18T22:42:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.361+0000 7ff32d7fa640 1 -- 172.21.15.40:0/142662880 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff3240043d0 con 0x7ff330101050 2024-03-18T22:42:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.361+0000 7ff32d7fa640 1 -- 172.21.15.40:0/142662880 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324003b80 con 0x7ff330101050 2024-03-18T22:42:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.361+0000 7ff33637e640 1 -- 172.21.15.40:0/142662880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff33019ff30 con 0x7ff330101050 2024-03-18T22:42:31.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff32d7fa640 1 -- 172.21.15.40:0/142662880 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff324046020 con 0x7ff330101050 2024-03-18T22:42:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff338f6e740 1 Processor -- start 2024-03-18T22:42:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff338f6e740 1 -- start start 2024-03-18T22:42:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff32d7fa640 1 --2- 172.21.15.40:0/142662880 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff3180a2800 0x7ff3180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff338f6e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d20410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff338f6e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 0x559c07d254e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff338f6e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d25b50 0x559c07d27fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff338f6e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559c07c84de0 con 0x559c07d1fff0 2024-03-18T22:42:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff338f6e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559c07c2dc80 con 0x559c07d209e0 2024-03-18T22:42:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff338f6e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559c07c80e80 con 0x559c07d25b50 2024-03-18T22:42:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff33491b640 1 --2- 172.21.15.40:0/142662880 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff3180a2800 0x7ff3180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff32ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d20410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff32ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d20410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50996/0 (socket says 172.21.15.40:50996) 2024-03-18T22:42:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff32ffff640 1 -- 172.21.15.40:0/1289257257 learned_addr learned my addr 172.21.15.40:0/1289257257 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff32d7fa640 1 -- 172.21.15.40:0/142662880 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff3240f0f70 con 0x7ff330101050 2024-03-18T22:42:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.363+0000 7ff32f7fe640 1 --2- 172.21.15.40:0/1289257257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 0x559c07d254e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.364+0000 7ff33491b640 1 --2- 172.21.15.40:0/1289257257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d25b50 0x559c07d27fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.364+0000 7ff32ffff640 1 -- 172.21.15.40:0/1289257257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d25b50 msgr2=0x559c07d27fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.364+0000 7ff32ffff640 1 --2- 172.21.15.40:0/1289257257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d25b50 0x559c07d27fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.364+0000 7ff32ffff640 1 -- 172.21.15.40:0/1289257257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 msgr2=0x559c07d254e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.364+0000 7ff32ffff640 1 --2- 172.21.15.40:0/1289257257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 0x559c07d254e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.364+0000 7ff32ffff640 1 -- 172.21.15.40:0/1289257257 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c07c3ffb0 con 0x559c07d1fff0 2024-03-18T22:42:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.364+0000 7ff33491b640 1 --2- 172.21.15.40:0/1289257257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d25b50 0x559c07d27fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:42:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.364+0000 7ff32ffff640 1 --2- 172.21.15.40:0/1289257257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d20410 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff320009ca0 tx=0x7ff320011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e198808c4398af server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.364+0000 7ff33491b640 1 --2- 172.21.15.40:0/142662880 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff3180a2800 0x7ff3180a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff328005fd0 tx=0x7ff328009b10 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.364+0000 7ff3167fc640 1 -- 172.21.15.40:0/1289257257 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff320004220 con 0x559c07d1fff0 2024-03-18T22:42:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.364+0000 7ff3167fc640 1 -- 172.21.15.40:0/1289257257 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff3200043c0 con 0x559c07d1fff0 2024-03-18T22:42:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.364+0000 7ff3167fc640 1 -- 172.21.15.40:0/1289257257 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff320019890 con 0x559c07d1fff0 2024-03-18T22:42:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.365+0000 7ff338f6e740 1 -- 172.21.15.40:0/1289257257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 msgr2=0x559c07d20410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.365+0000 7ff338f6e740 1 --2- 172.21.15.40:0/1289257257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d20410 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff320009ca0 tx=0x7ff320011810 comp rx=0 tx=0).stop 2024-03-18T22:42:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.365+0000 7ff338f6e740 1 -- 172.21.15.40:0/1289257257 shutdown_connections 2024-03-18T22:42:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.365+0000 7ff338f6e740 1 --2- 172.21.15.40:0/1289257257 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d25b50 0x559c07d27fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.365+0000 7ff338f6e740 1 --2- 172.21.15.40:0/1289257257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 0x559c07d254e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.365+0000 7ff338f6e740 1 --2- 172.21.15.40:0/1289257257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d20410 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.365+0000 7ff338f6e740 1 -- 172.21.15.40:0/1289257257 >> 172.21.15.40:0/1289257257 conn(0x559c07d1b560 msgr2=0x559c07d1b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.365+0000 7ff338f6e740 1 -- 172.21.15.40:0/1289257257 shutdown_connections 2024-03-18T22:42:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.365+0000 7ff338f6e740 1 -- 172.21.15.40:0/1289257257 wait complete. 2024-03-18T22:42:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.365+0000 7ff338f6e740 1 -- start start 2024-03-18T22:42:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.365+0000 7ff338f6e740 1 Processor -- start 2024-03-18T22:42:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff338f6e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d2e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff338f6e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 0x559c07d2edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff338f6e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d25b50 0x559c07d35df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff338f6e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559c07d2cf10 con 0x559c07d1fff0 2024-03-18T22:42:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff338f6e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559c07c40fc0 con 0x559c07d209e0 2024-03-18T22:42:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff338f6e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559c07d2d090 con 0x559c07d25b50 2024-03-18T22:42:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff32ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d2e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff338f6e740 10 client.0 _ll_register_callbacks cb 0x559c07c7ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff32ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d2e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50998/0 (socket says 172.21.15.40:50998) 2024-03-18T22:42:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff32ffff640 1 -- 172.21.15.40:0/2673403629 learned_addr learned my addr 172.21.15.40:0/2673403629 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff32f7fe640 1 --2- 172.21.15.40:0/2673403629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 0x559c07d2edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff33491b640 1 --2- 172.21.15.40:0/2673403629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d25b50 0x559c07d35df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff32ffff640 1 -- 172.21.15.40:0/2673403629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d25b50 msgr2=0x559c07d35df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff32ffff640 1 --2- 172.21.15.40:0/2673403629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d25b50 0x559c07d35df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff32ffff640 1 -- 172.21.15.40:0/2673403629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 msgr2=0x559c07d2edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff32ffff640 1 --2- 172.21.15.40:0/2673403629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 0x559c07d2edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff32ffff640 1 -- 172.21.15.40:0/2673403629 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c079992d0 con 0x559c07d1fff0 2024-03-18T22:42:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff32f7fe640 1 --2- 172.21.15.40:0/2673403629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 0x559c07d2edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff32ffff640 1 --2- 172.21.15.40:0/2673403629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d2e870 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff320009870 tx=0x7ff320005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=427b24603b36ee8a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff30f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff30f7fe640 1 -- 172.21.15.40:0/2673403629 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff320028070 con 0x559c07d1fff0 2024-03-18T22:42:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff30f7fe640 1 -- 172.21.15.40:0/2673403629 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff3200035f0 con 0x559c07d1fff0 2024-03-18T22:42:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.366+0000 7ff338f6e740 1 -- 172.21.15.40:0/2673403629 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3200043c0 con 0x559c07d1fff0 2024-03-18T22:42:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.367+0000 7ff30f7fe640 1 -- 172.21.15.40:0/2673403629 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff320034050 con 0x559c07d1fff0 2024-03-18T22:42:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.367+0000 7ff338f6e740 10 client.15225 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.367+0000 7ff338f6e740 1 -- 172.21.15.40:0/2673403629 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559c07a636b0 con 0x559c07d1fff0 2024-03-18T22:42:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.367+0000 7ff30f7fe640 1 -- 172.21.15.40:0/2673403629 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff3200043c0 con 0x559c07d1fff0 2024-03-18T22:42:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.367+0000 7ff30f7fe640 1 -- 172.21.15.40:0/2673403629 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ff320004930 con 0x559c07d1fff0 2024-03-18T22:42:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.367+0000 7ff338f6e740 10 client.15225 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.367+0000 7ff338f6e740 10 client.15225 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T22:42:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.367+0000 7ff338f6e740 20 client.15225 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:31.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.367+0000 7ff338f6e740 1 --2- 172.21.15.40:0/2673403629 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x559c07d3b4d0 0x559c07d3c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.367+0000 7ff338f6e740 4 client.15225 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:42:31.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.367+0000 7ff338f6e740 1 -- 172.21.15.40:0/2673403629 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c07c92320 con 0x559c07d3b4d0 2024-03-18T22:42:31.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.367+0000 7ff32f7fe640 1 --2- 172.21.15.40:0/2673403629 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x559c07d3b4d0 0x559c07d3c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.367+0000 7ff32f7fe640 1 --2- 172.21.15.40:0/2673403629 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x559c07d3b4d0 0x559c07d3c940 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a2a69e9ac7559c30 server_cookie=71c7ee6932cc624e in_seq=0 out_seq=0 2024-03-18T22:42:31.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.367+0000 7ff30f7fe640 10 client.15225 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:42:31.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.369+0000 7ff30f7fe640 1 -- 172.21.15.40:0/2673403629 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559c07c92320 con 0x559c07d3b4d0 2024-03-18T22:42:31.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.369+0000 7ff30f7fe640 10 client.15225 handle_command_reply: tid=0 2024-03-18T22:42:31.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.369+0000 7ff338f6e740 1 client.15225 shutdown 2024-03-18T22:42:31.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.369+0000 7ff338f6e740 1 -- 172.21.15.40:0/2673403629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 msgr2=0x559c07d2e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.369+0000 7ff338f6e740 1 --2- 172.21.15.40:0/2673403629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d2e870 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff320009870 tx=0x7ff320005270 comp rx=0 tx=0).stop 2024-03-18T22:42:31.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.369+0000 7ff338f6e740 1 -- 172.21.15.40:0/2673403629 shutdown_connections 2024-03-18T22:42:31.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.369+0000 7ff338f6e740 1 --2- 172.21.15.40:0/2673403629 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x559c07d3b4d0 0x559c07d3c940 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.369+0000 7ff338f6e740 1 --2- 172.21.15.40:0/2673403629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d25b50 0x559c07d35df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.369+0000 7ff338f6e740 1 --2- 172.21.15.40:0/2673403629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 0x559c07d2edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.369+0000 7ff338f6e740 1 --2- 172.21.15.40:0/2673403629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d2e870 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.369+0000 7ff30f7fe640 0 client.15225 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:42:31.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.369+0000 7ff338f6e740 1 -- 172.21.15.40:0/2673403629 >> 172.21.15.40:0/2673403629 conn(0x559c07d1b560 msgr2=0x559c07d1a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.369+0000 7ff338f6e740 1 -- 172.21.15.40:0/2673403629 shutdown_connections 2024-03-18T22:42:31.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.369+0000 7ff338f6e740 1 -- 172.21.15.40:0/2673403629 wait complete. 2024-03-18T22:42:31.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.369+0000 7ff338f6e740 20 client.15225 trim_cache size 0 max 16384 2024-03-18T22:42:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff338f6e740 1 Processor -- start 2024-03-18T22:42:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff338f6e740 1 -- start start 2024-03-18T22:42:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff338f6e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d2d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff338f6e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d209e0 0x559c07d35d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff338f6e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d25b50 0x559c07d1b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff338f6e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559c07d3ac80 con 0x559c07d1fff0 2024-03-18T22:42:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff338f6e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559c07d3b350 con 0x559c07d25b50 2024-03-18T22:42:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff338f6e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559c07d3ba20 con 0x559c07d209e0 2024-03-18T22:42:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff32ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d2d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff32ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d2d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51002/0 (socket says 172.21.15.40:51002) 2024-03-18T22:42:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff32ffff640 1 -- 172.21.15.40:0/244461478 learned_addr learned my addr 172.21.15.40:0/244461478 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff33491b640 1 --2- 172.21.15.40:0/244461478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d25b50 0x559c07d1b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff32f7fe640 1 --2- 172.21.15.40:0/244461478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d209e0 0x559c07d35d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff32ffff640 1 -- 172.21.15.40:0/244461478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d209e0 msgr2=0x559c07d35d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff32ffff640 1 --2- 172.21.15.40:0/244461478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d209e0 0x559c07d35d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff32ffff640 1 -- 172.21.15.40:0/244461478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d25b50 msgr2=0x559c07d1b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff32ffff640 1 --2- 172.21.15.40:0/244461478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d25b50 0x559c07d1b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.387+0000 7ff32ffff640 1 -- 172.21.15.40:0/244461478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c07d20dc0 con 0x559c07d1fff0 2024-03-18T22:42:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.388+0000 7ff32ffff640 1 --2- 172.21.15.40:0/244461478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d2d460 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff320009380 tx=0x7ff320009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee0a5fe59e15ef25 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.388+0000 7ff315ffb640 1 -- 172.21.15.40:0/244461478 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32003d070 con 0x559c07d1fff0 2024-03-18T22:42:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.388+0000 7ff315ffb640 1 -- 172.21.15.40:0/244461478 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff320031030 con 0x559c07d1fff0 2024-03-18T22:42:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.388+0000 7ff315ffb640 1 -- 172.21.15.40:0/244461478 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff320028040 con 0x559c07d1fff0 2024-03-18T22:42:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.388+0000 7ff338f6e740 1 -- 172.21.15.40:0/244461478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 msgr2=0x559c07d2d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.388+0000 7ff338f6e740 1 --2- 172.21.15.40:0/244461478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d2d460 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff320009380 tx=0x7ff320009f80 comp rx=0 tx=0).stop 2024-03-18T22:42:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.388+0000 7ff338f6e740 1 -- 172.21.15.40:0/244461478 shutdown_connections 2024-03-18T22:42:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.388+0000 7ff338f6e740 1 --2- 172.21.15.40:0/244461478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d25b50 0x559c07d1b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.388+0000 7ff338f6e740 1 --2- 172.21.15.40:0/244461478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d209e0 0x559c07d35d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.388+0000 7ff338f6e740 1 --2- 172.21.15.40:0/244461478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d1fff0 0x559c07d2d460 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.388+0000 7ff338f6e740 1 -- 172.21.15.40:0/244461478 >> 172.21.15.40:0/244461478 conn(0x559c07d1b560 msgr2=0x559c07d213f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.388+0000 7ff338f6e740 1 -- 172.21.15.40:0/244461478 shutdown_connections 2024-03-18T22:42:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.388+0000 7ff338f6e740 1 -- 172.21.15.40:0/244461478 wait complete. 2024-03-18T22:42:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff338f6e740 1 -- start start 2024-03-18T22:42:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff338f6e740 1 Processor -- start 2024-03-18T22:42:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff338f6e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d1fff0 0x559c07d33d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff338f6e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 0x559c07d9d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff338f6e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d25b50 0x559c07d9f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff338f6e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559c07d3c7c0 con 0x559c07d25b50 2024-03-18T22:42:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff338f6e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559c07d3c0f0 con 0x559c07d209e0 2024-03-18T22:42:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff338f6e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559c07d3ce90 con 0x559c07d1fff0 2024-03-18T22:42:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff338f6e740 10 client.0 _ll_register_callbacks cb 0x7ff324005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff33491b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d25b50 0x559c07d9f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff33491b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d25b50 0x559c07d9f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51006/0 (socket says 172.21.15.40:51006) 2024-03-18T22:42:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff33491b640 1 -- 172.21.15.40:0/103113854 learned_addr learned my addr 172.21.15.40:0/103113854 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff32f7fe640 1 --2- 172.21.15.40:0/103113854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 0x559c07d9d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff32ffff640 1 --2- 172.21.15.40:0/103113854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d1fff0 0x559c07d33d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff33491b640 1 -- 172.21.15.40:0/103113854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d1fff0 msgr2=0x559c07d33d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff33491b640 1 --2- 172.21.15.40:0/103113854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d1fff0 0x559c07d33d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff33491b640 1 -- 172.21.15.40:0/103113854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 msgr2=0x559c07d9d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff33491b640 1 --2- 172.21.15.40:0/103113854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 0x559c07d9d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.389+0000 7ff33491b640 1 -- 172.21.15.40:0/103113854 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c07d9ff80 con 0x559c07d25b50 2024-03-18T22:42:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff33491b640 1 --2- 172.21.15.40:0/103113854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d25b50 0x559c07d9f770 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff328013430 tx=0x7ff328009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c7bf1eae8c9fea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff30f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff30f7fe640 1 -- 172.21.15.40:0/103113854 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3280136c0 con 0x559c07d25b50 2024-03-18T22:42:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff338f6e740 1 -- 172.21.15.40:0/103113854 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff320031030 con 0x559c07d25b50 2024-03-18T22:42:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff30f7fe640 1 -- 172.21.15.40:0/103113854 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff3280045b0 con 0x559c07d25b50 2024-03-18T22:42:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff30f7fe640 1 -- 172.21.15.40:0/103113854 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff3280047b0 con 0x559c07d25b50 2024-03-18T22:42:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff338f6e740 10 client.15231 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff338f6e740 1 -- 172.21.15.40:0/103113854 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559c07da12f0 con 0x559c07d25b50 2024-03-18T22:42:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff30f7fe640 1 -- 172.21.15.40:0/103113854 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff320031030 con 0x559c07d25b50 2024-03-18T22:42:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff30f7fe640 1 -- 172.21.15.40:0/103113854 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ff328029070 con 0x559c07d25b50 2024-03-18T22:42:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff338f6e740 10 client.15231 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff338f6e740 10 client.15231 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T22:42:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff338f6e740 20 client.15231 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff338f6e740 1 --2- 172.21.15.40:0/103113854 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x559c07da5fd0 0x559c07da4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff338f6e740 4 client.15231 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T22:42:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.390+0000 7ff338f6e740 1 -- 172.21.15.40:0/103113854 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "status"}) -- 0x559c07c8c670 con 0x559c07da5fd0 2024-03-18T22:42:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff32ffff640 1 --2- 172.21.15.40:0/103113854 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x559c07da5fd0 0x559c07da4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff32ffff640 1 --2- 172.21.15.40:0/103113854 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x559c07da5fd0 0x559c07da4ce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=756a11ea37d5edf8 server_cookie=e027d3622cb1f864 in_seq=0 out_seq=0 2024-03-18T22:42:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff30f7fe640 10 client.15231 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:42:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff30f7fe640 1 -- 172.21.15.40:0/103113854 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559c07c8c670 con 0x559c07da5fd0 2024-03-18T22:42:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff30f7fe640 10 client.15231 handle_command_reply: tid=0 2024-03-18T22:42:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff338f6e740 1 client.15231 shutdown 2024-03-18T22:42:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff338f6e740 1 -- 172.21.15.40:0/103113854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d25b50 msgr2=0x559c07d9f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff338f6e740 1 --2- 172.21.15.40:0/103113854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d25b50 0x559c07d9f770 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff328013430 tx=0x7ff328009500 comp rx=0 tx=0).stop 2024-03-18T22:42:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff338f6e740 1 -- 172.21.15.40:0/103113854 shutdown_connections 2024-03-18T22:42:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff338f6e740 1 --2- 172.21.15.40:0/103113854 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x559c07da5fd0 0x559c07da4ce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff338f6e740 1 --2- 172.21.15.40:0/103113854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c07d25b50 0x559c07d9f770 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff338f6e740 1 --2- 172.21.15.40:0/103113854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c07d209e0 0x559c07d9d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff30f7fe640 0 client.15231 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:42:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff338f6e740 1 --2- 172.21.15.40:0/103113854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c07d1fff0 0x559c07d33d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff338f6e740 1 -- 172.21.15.40:0/103113854 >> 172.21.15.40:0/103113854 conn(0x559c07d1b560 msgr2=0x559c07d25040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff338f6e740 1 -- 172.21.15.40:0/103113854 shutdown_connections 2024-03-18T22:42:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff338f6e740 1 -- 172.21.15.40:0/103113854 wait complete. 2024-03-18T22:42:31.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.391+0000 7ff338f6e740 20 client.15231 trim_cache size 0 max 16384 2024-03-18T22:42:31.399 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:42:31.399 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "410ef414-e577-11ee-95c9-87774f69a715", 2024-03-18T22:42:31.399 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-03-18T22:42:31.399 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24473, 2024-03-18T22:42:31.399 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-18T22:42:31.400 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-18T22:42:31.400 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-18T22:42:31.400 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 84.352496943000006, 2024-03-18T22:42:31.400 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 27, 2024-03-18T22:42:31.400 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 91, 2024-03-18T22:42:31.400 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T22:42:31.400 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 124.957825904 2024-03-18T22:42:31.400 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:42:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.392+0000 7ff33637e640 1 -- 172.21.15.40:0/142662880 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff3180a2800 msgr2=0x7ff3180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.392+0000 7ff33637e640 1 --2- 172.21.15.40:0/142662880 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff3180a2800 0x7ff3180a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff328005fd0 tx=0x7ff328009b10 comp rx=0 tx=0).stop 2024-03-18T22:42:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.392+0000 7ff33637e640 1 -- 172.21.15.40:0/142662880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff330101050 msgr2=0x7ff330198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.392+0000 7ff33637e640 1 --2- 172.21.15.40:0/142662880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff330101050 0x7ff330198020 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff324006010 tx=0x7ff324005e40 comp rx=0 tx=0).stop 2024-03-18T22:42:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.392+0000 7ff33637e640 1 -- 172.21.15.40:0/142662880 shutdown_connections 2024-03-18T22:42:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.392+0000 7ff33637e640 1 --2- 172.21.15.40:0/142662880 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff3180a2800 0x7ff3180a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.392+0000 7ff33637e640 1 --2- 172.21.15.40:0/142662880 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff330101a10 0x7ff33019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.392+0000 7ff33637e640 1 --2- 172.21.15.40:0/142662880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff330101050 0x7ff330198020 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.392+0000 7ff33637e640 1 --2- 172.21.15.40:0/142662880 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff3300ffe70 0x7ff330197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.392+0000 7ff33637e640 1 -- 172.21.15.40:0/142662880 >> 172.21.15.40:0/142662880 conn(0x7ff3300fb450 msgr2=0x7ff3300fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.392+0000 7ff33637e640 1 -- 172.21.15.40:0/142662880 shutdown_connections 2024-03-18T22:42:31.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.392+0000 7ff33637e640 1 -- 172.21.15.40:0/142662880 wait complete. 2024-03-18T22:42:31.403 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T22:42:31.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.506+0000 7fe772bd4640 1 -- 172.21.15.40:0/3936635716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe76c100730 msgr2=0x7fe76c100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.506+0000 7fe772bd4640 1 --2- 172.21.15.40:0/3936635716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe76c100730 0x7fe76c100b50 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe760009900 tx=0x7fe76002bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:31.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.506+0000 7fe772bd4640 1 -- 172.21.15.40:0/3936635716 shutdown_connections 2024-03-18T22:42:31.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.506+0000 7fe772bd4640 1 --2- 172.21.15.40:0/3936635716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe76c1022e0 0x7fe76c108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.506+0000 7fe772bd4640 1 --2- 172.21.15.40:0/3936635716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe76c101920 0x7fe76c101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.506+0000 7fe772bd4640 1 --2- 172.21.15.40:0/3936635716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe76c100730 0x7fe76c100b50 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.506+0000 7fe772bd4640 1 -- 172.21.15.40:0/3936635716 >> 172.21.15.40:0/3936635716 conn(0x7fe76c0fbcc0 msgr2=0x7fe76c0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.506+0000 7fe772bd4640 1 -- 172.21.15.40:0/3936635716 shutdown_connections 2024-03-18T22:42:31.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.506+0000 7fe772bd4640 1 -- 172.21.15.40:0/3936635716 wait complete. 2024-03-18T22:42:31.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe772bd4640 1 Processor -- start 2024-03-18T22:42:31.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe772bd4640 1 -- start start 2024-03-18T22:42:31.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe772bd4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe76c100730 0x7fe76c1046c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe772bd4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe76c101920 0x7fe76c102d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe772bd4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe76c1022e0 0x7fe76c103240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe772bd4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe76c10c230 con 0x7fe76c101920 2024-03-18T22:42:31.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe772bd4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe76c10c0b0 con 0x7fe76c100730 2024-03-18T22:42:31.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe772bd4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe76c10c3b0 con 0x7fe76c1022e0 2024-03-18T22:42:31.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe76bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe76c101920 0x7fe76c102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe76bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe76c101920 0x7fe76c102d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51034/0 (socket says 172.21.15.40:51034) 2024-03-18T22:42:31.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe76bfff640 1 -- 172.21.15.40:0/3286152573 learned_addr learned my addr 172.21.15.40:0/3286152573 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe770970640 1 --2- 172.21.15.40:0/3286152573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe76c100730 0x7fe76c1046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe771171640 1 --2- 172.21.15.40:0/3286152573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe76c1022e0 0x7fe76c103240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe76bfff640 1 -- 172.21.15.40:0/3286152573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe76c1022e0 msgr2=0x7fe76c103240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe76bfff640 1 --2- 172.21.15.40:0/3286152573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe76c1022e0 0x7fe76c103240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe76bfff640 1 -- 172.21.15.40:0/3286152573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe76c100730 msgr2=0x7fe76c1046c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe76bfff640 1 --2- 172.21.15.40:0/3286152573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe76c100730 0x7fe76c1046c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe76bfff640 1 -- 172.21.15.40:0/3286152573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe76c103ad0 con 0x7fe76c101920 2024-03-18T22:42:31.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.507+0000 7fe76bfff640 1 --2- 172.21.15.40:0/3286152573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe76c101920 0x7fe76c102d00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe75800e9f0 tx=0x7fe75800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.508+0000 7fe769ffb640 1 -- 172.21.15.40:0/3286152573 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75800ccc0 con 0x7fe76c101920 2024-03-18T22:42:31.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.508+0000 7fe769ffb640 1 -- 172.21.15.40:0/3286152573 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7580108e0 con 0x7fe76c101920 2024-03-18T22:42:31.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.508+0000 7fe772bd4640 1 -- 172.21.15.40:0/3286152573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe76c1a1be0 con 0x7fe76c101920 2024-03-18T22:42:31.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.508+0000 7fe769ffb640 1 -- 172.21.15.40:0/3286152573 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7580187c0 con 0x7fe76c101920 2024-03-18T22:42:31.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.508+0000 7fe772bd4640 1 -- 172.21.15.40:0/3286152573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe76c1a2130 con 0x7fe76c101920 2024-03-18T22:42:31.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.510+0000 7fe769ffb640 1 -- 172.21.15.40:0/3286152573 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe758010a90 con 0x7fe76c101920 2024-03-18T22:42:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.510+0000 7fe7757c4740 1 Processor -- start 2024-03-18T22:42:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.510+0000 7fe7757c4740 1 -- start start 2024-03-18T22:42:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.510+0000 7fe7757c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc4ff0 0x5631c3bc5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.510+0000 7fe769ffb640 1 --2- 172.21.15.40:0/3286152573 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe75c0a2800 0x7fe75c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.510+0000 7fe7757c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc59e0 0x5631c3bca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.510+0000 7fe7757c4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bcab50 0x5631c3bccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.510+0000 7fe7757c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5631c3b29de0 con 0x5631c3bc4ff0 2024-03-18T22:42:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.510+0000 7fe7757c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5631c3ad2c80 con 0x5631c3bc59e0 2024-03-18T22:42:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.510+0000 7fe7757c4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5631c3b25e80 con 0x5631c3bcab50 2024-03-18T22:42:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe771171640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc4ff0 0x5631c3bc5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe770970640 1 --2- 172.21.15.40:0/3286152573 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe75c0a2800 0x7fe75c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe771171640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc4ff0 0x5631c3bc5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51046/0 (socket says 172.21.15.40:51046) 2024-03-18T22:42:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe771171640 1 -- 172.21.15.40:0/2061556750 learned_addr learned my addr 172.21.15.40:0/2061556750 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe76bfff640 1 --2- 172.21.15.40:0/2061556750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc59e0 0x5631c3bca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe769ffb640 1 -- 172.21.15.40:0/3286152573 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe7580cf1b0 con 0x7fe76c101920 2024-03-18T22:42:31.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe771171640 1 --2- 172.21.15.40:0/2061556750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bcab50 0x5631c3bccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe771171640 1 -- 172.21.15.40:0/2061556750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bcab50 msgr2=0x5631c3bccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe771171640 1 --2- 172.21.15.40:0/2061556750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bcab50 0x5631c3bccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe771171640 1 -- 172.21.15.40:0/2061556750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc59e0 msgr2=0x5631c3bca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe771171640 1 --2- 172.21.15.40:0/2061556750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc59e0 0x5631c3bca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe771171640 1 -- 172.21.15.40:0/2061556750 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631c3ae4fb0 con 0x5631c3bc4ff0 2024-03-18T22:42:31.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe76bfff640 1 --2- 172.21.15.40:0/2061556750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc59e0 0x5631c3bca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:31.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe770970640 1 --2- 172.21.15.40:0/3286152573 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe75c0a2800 0x7fe75c0a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe760005ab0 tx=0x7fe760005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe771171640 1 --2- 172.21.15.40:0/2061556750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc4ff0 0x5631c3bc5410 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe764013970 tx=0x7fe764009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afac610d7d80b7ca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe756ffd640 1 -- 172.21.15.40:0/2061556750 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe764003970 con 0x5631c3bc4ff0 2024-03-18T22:42:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe756ffd640 1 -- 172.21.15.40:0/2061556750 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe76400f8c0 con 0x5631c3bc4ff0 2024-03-18T22:42:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.511+0000 7fe756ffd640 1 -- 172.21.15.40:0/2061556750 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe764004430 con 0x5631c3bc4ff0 2024-03-18T22:42:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.512+0000 7fe7757c4740 1 -- 172.21.15.40:0/2061556750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc4ff0 msgr2=0x5631c3bc5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.512+0000 7fe7757c4740 1 --2- 172.21.15.40:0/2061556750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc4ff0 0x5631c3bc5410 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe764013970 tx=0x7fe764009bf0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.512+0000 7fe7757c4740 1 -- 172.21.15.40:0/2061556750 shutdown_connections 2024-03-18T22:42:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.512+0000 7fe7757c4740 1 --2- 172.21.15.40:0/2061556750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bcab50 0x5631c3bccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.512+0000 7fe7757c4740 1 --2- 172.21.15.40:0/2061556750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc59e0 0x5631c3bca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.512+0000 7fe7757c4740 1 --2- 172.21.15.40:0/2061556750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc4ff0 0x5631c3bc5410 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.512+0000 7fe7757c4740 1 -- 172.21.15.40:0/2061556750 >> 172.21.15.40:0/2061556750 conn(0x5631c3bc0560 msgr2=0x5631c3bc0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.512+0000 7fe7757c4740 1 -- 172.21.15.40:0/2061556750 shutdown_connections 2024-03-18T22:42:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.512+0000 7fe7757c4740 1 -- 172.21.15.40:0/2061556750 wait complete. 2024-03-18T22:42:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.512+0000 7fe7757c4740 1 -- start start 2024-03-18T22:42:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.512+0000 7fe7757c4740 1 Processor -- start 2024-03-18T22:42:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe7757c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc4ff0 0x5631c3bcf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe7757c4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bc59e0 0x5631c3bcf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe7757c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bcab50 0x5631c3bd6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe7757c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5631c3bcdb00 con 0x5631c3bcab50 2024-03-18T22:42:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe7757c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5631c3ae5fc0 con 0x5631c3bc4ff0 2024-03-18T22:42:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe7757c4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5631c3bcdc80 con 0x5631c3bc59e0 2024-03-18T22:42:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe7757c4740 10 client.0 _ll_register_callbacks cb 0x5631c3b21a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe771171640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bcab50 0x5631c3bd6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe771171640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bcab50 0x5631c3bd6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51048/0 (socket says 172.21.15.40:51048) 2024-03-18T22:42:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe771171640 1 -- 172.21.15.40:0/2089535390 learned_addr learned my addr 172.21.15.40:0/2089535390 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe770970640 1 --2- 172.21.15.40:0/2089535390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc4ff0 0x5631c3bcf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe76bfff640 1 --2- 172.21.15.40:0/2089535390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bc59e0 0x5631c3bcf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe771171640 1 -- 172.21.15.40:0/2089535390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bc59e0 msgr2=0x5631c3bcf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe771171640 1 --2- 172.21.15.40:0/2089535390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bc59e0 0x5631c3bcf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe771171640 1 -- 172.21.15.40:0/2089535390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc4ff0 msgr2=0x5631c3bcf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe771171640 1 --2- 172.21.15.40:0/2089535390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc4ff0 0x5631c3bcf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe771171640 1 -- 172.21.15.40:0/2089535390 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631c383e2d0 con 0x5631c3bcab50 2024-03-18T22:42:31.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe771171640 1 --2- 172.21.15.40:0/2089535390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bcab50 0x5631c3bd6a30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe76400ff50 tx=0x7fe764013580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2034da3fdaf63930 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe74bfff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:31.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe74bfff640 1 -- 172.21.15.40:0/2089535390 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe764003c80 con 0x5631c3bcab50 2024-03-18T22:42:31.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe74bfff640 1 -- 172.21.15.40:0/2089535390 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe764005050 con 0x5631c3bcab50 2024-03-18T22:42:31.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.513+0000 7fe7757c4740 1 -- 172.21.15.40:0/2089535390 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe76400f8c0 con 0x5631c3bcab50 2024-03-18T22:42:31.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.514+0000 7fe74bfff640 1 -- 172.21.15.40:0/2089535390 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe764035050 con 0x5631c3bcab50 2024-03-18T22:42:31.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.514+0000 7fe7757c4740 10 client.15243 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:31.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.514+0000 7fe7757c4740 1 -- 172.21.15.40:0/2089535390 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5631c39086b0 con 0x5631c3bcab50 2024-03-18T22:42:31.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.514+0000 7fe74bfff640 1 -- 172.21.15.40:0/2089535390 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe76400f8c0 con 0x5631c3bcab50 2024-03-18T22:42:31.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.514+0000 7fe74bfff640 1 -- 172.21.15.40:0/2089535390 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe76401e070 con 0x5631c3bcab50 2024-03-18T22:42:31.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.514+0000 7fe7757c4740 10 client.15243 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:31.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.514+0000 7fe7757c4740 10 client.15243 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T22:42:31.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.514+0000 7fe7757c4740 20 client.15243 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:31.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.514+0000 7fe7757c4740 1 --2- 172.21.15.40:0/2089535390 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5631c3bdc070 0x5631c3bdd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.514+0000 7fe7757c4740 4 client.15243 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:42:31.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.514+0000 7fe7757c4740 1 -- 172.21.15.40:0/2089535390 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631c3b37320 con 0x5631c3bdc070 2024-03-18T22:42:31.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.514+0000 7fe770970640 1 --2- 172.21.15.40:0/2089535390 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5631c3bdc070 0x5631c3bdd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.515+0000 7fe74bfff640 10 client.15243 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:42:31.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.515+0000 7fe770970640 1 --2- 172.21.15.40:0/2089535390 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5631c3bdc070 0x5631c3bdd4e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=71e77f02e2eafcf3 server_cookie=989c76dc2e0f9953 in_seq=0 out_seq=0 2024-03-18T22:42:31.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.515+0000 7fe74bfff640 1 -- 172.21.15.40:0/2089535390 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5631c3b37320 con 0x5631c3bdc070 2024-03-18T22:42:31.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.515+0000 7fe74bfff640 10 client.15243 handle_command_reply: tid=0 2024-03-18T22:42:31.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.515+0000 7fe7757c4740 1 client.15243 shutdown 2024-03-18T22:42:31.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.515+0000 7fe7757c4740 1 -- 172.21.15.40:0/2089535390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bcab50 msgr2=0x5631c3bd6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.515+0000 7fe7757c4740 1 --2- 172.21.15.40:0/2089535390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bcab50 0x5631c3bd6a30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe76400ff50 tx=0x7fe764013580 comp rx=0 tx=0).stop 2024-03-18T22:42:31.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.515+0000 7fe7757c4740 1 -- 172.21.15.40:0/2089535390 shutdown_connections 2024-03-18T22:42:31.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.515+0000 7fe7757c4740 1 --2- 172.21.15.40:0/2089535390 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5631c3bdc070 0x5631c3bdd4e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.515+0000 7fe7757c4740 1 --2- 172.21.15.40:0/2089535390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bcab50 0x5631c3bd6a30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.515+0000 7fe7757c4740 1 --2- 172.21.15.40:0/2089535390 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bc59e0 0x5631c3bcf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.515+0000 7fe74bfff640 0 client.15243 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:42:31.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.515+0000 7fe7757c4740 1 --2- 172.21.15.40:0/2089535390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc4ff0 0x5631c3bcf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.515+0000 7fe7757c4740 1 -- 172.21.15.40:0/2089535390 >> 172.21.15.40:0/2089535390 conn(0x5631c3bc0560 msgr2=0x5631c3bbf7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.516+0000 7fe7757c4740 1 -- 172.21.15.40:0/2089535390 shutdown_connections 2024-03-18T22:42:31.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.516+0000 7fe7757c4740 1 -- 172.21.15.40:0/2089535390 wait complete. 2024-03-18T22:42:31.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.516+0000 7fe7757c4740 20 client.15243 trim_cache size 0 max 16384 2024-03-18T22:42:31.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe7757c4740 1 Processor -- start 2024-03-18T22:42:31.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe7757c4740 1 -- start start 2024-03-18T22:42:31.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe7757c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc4ff0 0x5631c3bcd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe7757c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc59e0 0x5631c3bd6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe7757c4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bcab50 0x5631c3bcead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe7757c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5631c3bdb820 con 0x5631c3bc59e0 2024-03-18T22:42:31.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe7757c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5631c3bdbef0 con 0x5631c3bc4ff0 2024-03-18T22:42:31.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe7757c4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5631c3bdc5c0 con 0x5631c3bcab50 2024-03-18T22:42:31.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe76bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc59e0 0x5631c3bd6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe76bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc59e0 0x5631c3bd6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51060/0 (socket says 172.21.15.40:51060) 2024-03-18T22:42:31.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe76bfff640 1 -- 172.21.15.40:0/459026881 learned_addr learned my addr 172.21.15.40:0/459026881 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe770970640 1 --2- 172.21.15.40:0/459026881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc4ff0 0x5631c3bcd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe771171640 1 --2- 172.21.15.40:0/459026881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bcab50 0x5631c3bcead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe76bfff640 1 -- 172.21.15.40:0/459026881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bcab50 msgr2=0x5631c3bcead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe76bfff640 1 --2- 172.21.15.40:0/459026881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bcab50 0x5631c3bcead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe76bfff640 1 -- 172.21.15.40:0/459026881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc4ff0 msgr2=0x5631c3bcd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe76bfff640 1 --2- 172.21.15.40:0/459026881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc4ff0 0x5631c3bcd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe76bfff640 1 -- 172.21.15.40:0/459026881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631c3bdca70 con 0x5631c3bc59e0 2024-03-18T22:42:31.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe76bfff640 1 --2- 172.21.15.40:0/459026881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc59e0 0x5631c3bd6950 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe7580cccc0 tx=0x7fe758009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31f629db68bce8a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe7567fc640 1 -- 172.21.15.40:0/459026881 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe758026040 con 0x5631c3bc59e0 2024-03-18T22:42:31.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe7567fc640 1 -- 172.21.15.40:0/459026881 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe75809b7f0 con 0x5631c3bc59e0 2024-03-18T22:42:31.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.534+0000 7fe7567fc640 1 -- 172.21.15.40:0/459026881 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe758097e60 con 0x5631c3bc59e0 2024-03-18T22:42:31.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.535+0000 7fe7757c4740 1 -- 172.21.15.40:0/459026881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc59e0 msgr2=0x5631c3bd6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.535+0000 7fe7757c4740 1 --2- 172.21.15.40:0/459026881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc59e0 0x5631c3bd6950 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe7580cccc0 tx=0x7fe758009700 comp rx=0 tx=0).stop 2024-03-18T22:42:31.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.535+0000 7fe7757c4740 1 -- 172.21.15.40:0/459026881 shutdown_connections 2024-03-18T22:42:31.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.535+0000 7fe7757c4740 1 --2- 172.21.15.40:0/459026881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bcab50 0x5631c3bcead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.535+0000 7fe7757c4740 1 --2- 172.21.15.40:0/459026881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc59e0 0x5631c3bd6950 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.535+0000 7fe7757c4740 1 --2- 172.21.15.40:0/459026881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc4ff0 0x5631c3bcd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.535+0000 7fe7757c4740 1 -- 172.21.15.40:0/459026881 >> 172.21.15.40:0/459026881 conn(0x5631c3bc0560 msgr2=0x5631c3bc0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.535+0000 7fe7757c4740 1 -- 172.21.15.40:0/459026881 shutdown_connections 2024-03-18T22:42:31.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.535+0000 7fe7757c4740 1 -- 172.21.15.40:0/459026881 wait complete. 2024-03-18T22:42:31.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.535+0000 7fe7757c4740 1 -- start start 2024-03-18T22:42:31.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.535+0000 7fe7757c4740 1 Processor -- start 2024-03-18T22:42:31.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe7757c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc4ff0 0x5631c3bd4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe7757c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc59e0 0x5631c3c3ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe7757c4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bcab50 0x5631c3c40310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe7757c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5631c3bdd360 con 0x5631c3bc4ff0 2024-03-18T22:42:31.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe7757c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5631c3bdcc90 con 0x5631c3bc59e0 2024-03-18T22:42:31.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe7757c4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5631c3bdda30 con 0x5631c3bcab50 2024-03-18T22:42:31.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe7757c4740 10 client.0 _ll_register_callbacks cb 0x7fe758097370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:31.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe770970640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc4ff0 0x5631c3bd4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe770970640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc4ff0 0x5631c3bd4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51076/0 (socket says 172.21.15.40:51076) 2024-03-18T22:42:31.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe770970640 1 -- 172.21.15.40:0/3727412429 learned_addr learned my addr 172.21.15.40:0/3727412429 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe76bfff640 1 --2- 172.21.15.40:0/3727412429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc59e0 0x5631c3c3ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe771171640 1 --2- 172.21.15.40:0/3727412429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bcab50 0x5631c3c40310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe770970640 1 -- 172.21.15.40:0/3727412429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bcab50 msgr2=0x5631c3c40310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe770970640 1 --2- 172.21.15.40:0/3727412429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bcab50 0x5631c3c40310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe770970640 1 -- 172.21.15.40:0/3727412429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc59e0 msgr2=0x5631c3c3ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe770970640 1 --2- 172.21.15.40:0/3727412429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc59e0 0x5631c3c3ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe770970640 1 -- 172.21.15.40:0/3727412429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631c3c40a80 con 0x5631c3bc4ff0 2024-03-18T22:42:31.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe770970640 1 --2- 172.21.15.40:0/3727412429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc4ff0 0x5631c3bd4940 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe76002cc30 tx=0x7fe76002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c880b1cc5032271f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe74bfff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:31.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe74bfff640 1 -- 172.21.15.40:0/3727412429 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7600042c0 con 0x5631c3bc4ff0 2024-03-18T22:42:31.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe74bfff640 1 -- 172.21.15.40:0/3727412429 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe760004460 con 0x5631c3bc4ff0 2024-03-18T22:42:31.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.536+0000 7fe7757c4740 1 -- 172.21.15.40:0/3727412429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe75809b7f0 con 0x5631c3bc4ff0 2024-03-18T22:42:31.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.537+0000 7fe74bfff640 1 -- 172.21.15.40:0/3727412429 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe760064050 con 0x5631c3bc4ff0 2024-03-18T22:42:31.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.537+0000 7fe7757c4740 10 client.15249 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.537+0000 7fe7757c4740 1 -- 172.21.15.40:0/3727412429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5631c3c41e50 con 0x5631c3bc4ff0 2024-03-18T22:42:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.537+0000 7fe74bfff640 1 -- 172.21.15.40:0/3727412429 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe75809b7f0 con 0x5631c3bc4ff0 2024-03-18T22:42:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.537+0000 7fe74bfff640 1 -- 172.21.15.40:0/3727412429 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe760003a50 con 0x5631c3bc4ff0 2024-03-18T22:42:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.537+0000 7fe7757c4740 10 client.15249 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.537+0000 7fe7757c4740 10 client.15249 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T22:42:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.537+0000 7fe7757c4740 20 client.15249 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.537+0000 7fe7757c4740 1 --2- 172.21.15.40:0/3727412429 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5631c3c46a00 0x5631c3c45710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.537+0000 7fe7757c4740 4 client.15249 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T22:42:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.537+0000 7fe7757c4740 1 -- 172.21.15.40:0/3727412429 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "status"}) -- 0x5631c3b31670 con 0x5631c3c46a00 2024-03-18T22:42:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.537+0000 7fe76bfff640 1 --2- 172.21.15.40:0/3727412429 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5631c3c46a00 0x5631c3c45710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe74bfff640 10 client.15249 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:42:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe76bfff640 1 --2- 172.21.15.40:0/3727412429 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5631c3c46a00 0x5631c3c45710 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=88032f03f1fcfa37 server_cookie=69ef9accbb7b1985 in_seq=0 out_seq=0 2024-03-18T22:42:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe74bfff640 1 -- 172.21.15.40:0/3727412429 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x5631c3b31670 con 0x5631c3c46a00 2024-03-18T22:42:31.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe74bfff640 10 client.15249 handle_command_reply: tid=0 2024-03-18T22:42:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe7757c4740 1 client.15249 shutdown 2024-03-18T22:42:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe7757c4740 1 -- 172.21.15.40:0/3727412429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc4ff0 msgr2=0x5631c3bd4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe7757c4740 1 --2- 172.21.15.40:0/3727412429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc4ff0 0x5631c3bd4940 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe76002cc30 tx=0x7fe76002bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe7757c4740 1 -- 172.21.15.40:0/3727412429 shutdown_connections 2024-03-18T22:42:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe7757c4740 1 --2- 172.21.15.40:0/3727412429 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5631c3c46a00 0x5631c3c45710 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe7757c4740 1 --2- 172.21.15.40:0/3727412429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5631c3bcab50 0x5631c3c40310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe74bfff640 0 client.15249 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:42:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe7757c4740 1 --2- 172.21.15.40:0/3727412429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5631c3bc59e0 0x5631c3c3ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe7757c4740 1 --2- 172.21.15.40:0/3727412429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5631c3bc4ff0 0x5631c3bd4940 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe7757c4740 1 -- 172.21.15.40:0/3727412429 >> 172.21.15.40:0/3727412429 conn(0x5631c3bc0560 msgr2=0x5631c3bcead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe7757c4740 1 -- 172.21.15.40:0/3727412429 shutdown_connections 2024-03-18T22:42:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe7757c4740 1 -- 172.21.15.40:0/3727412429 wait complete. 2024-03-18T22:42:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.538+0000 7fe7757c4740 20 client.15249 trim_cache size 0 max 16384 2024-03-18T22:42:31.547 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:42:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "410ef414-e577-11ee-95c9-87774f69a715", 2024-03-18T22:42:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-03-18T22:42:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24481, 2024-03-18T22:42:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-18T22:42:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-18T22:42:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-18T22:42:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 85.508905532, 2024-03-18T22:42:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 27, 2024-03-18T22:42:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 91, 2024-03-18T22:42:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T22:42:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 122.332835266 2024-03-18T22:42:31.548 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:42:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.539+0000 7fe772bd4640 1 -- 172.21.15.40:0/3286152573 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe75c0a2800 msgr2=0x7fe75c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.539+0000 7fe772bd4640 1 --2- 172.21.15.40:0/3286152573 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe75c0a2800 0x7fe75c0a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe760005ab0 tx=0x7fe760005fd0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.539+0000 7fe772bd4640 1 -- 172.21.15.40:0/3286152573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe76c101920 msgr2=0x7fe76c102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.539+0000 7fe772bd4640 1 --2- 172.21.15.40:0/3286152573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe76c101920 0x7fe76c102d00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe75800e9f0 tx=0x7fe75800edf0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.539+0000 7fe772bd4640 1 -- 172.21.15.40:0/3286152573 shutdown_connections 2024-03-18T22:42:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.539+0000 7fe772bd4640 1 --2- 172.21.15.40:0/3286152573 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe75c0a2800 0x7fe75c0a4cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.539+0000 7fe772bd4640 1 --2- 172.21.15.40:0/3286152573 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe76c1022e0 0x7fe76c103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.539+0000 7fe772bd4640 1 --2- 172.21.15.40:0/3286152573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe76c101920 0x7fe76c102d00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.539+0000 7fe772bd4640 1 --2- 172.21.15.40:0/3286152573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe76c100730 0x7fe76c1046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.539+0000 7fe772bd4640 1 -- 172.21.15.40:0/3286152573 >> 172.21.15.40:0/3286152573 conn(0x7fe76c0fbcc0 msgr2=0x7fe76c0fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.539+0000 7fe772bd4640 1 -- 172.21.15.40:0/3286152573 shutdown_connections 2024-03-18T22:42:31.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.539+0000 7fe772bd4640 1 -- 172.21.15.40:0/3286152573 wait complete. 2024-03-18T22:42:31.551 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T22:42:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.654+0000 7f788bc4b640 1 -- 172.21.15.40:0/1880032698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78841033e0 msgr2=0x7f7884109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.654+0000 7f788bc4b640 1 --2- 172.21.15.40:0/1880032698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78841033e0 0x7f7884109c70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f787c009ca0 tx=0x7f787c02c0f0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.654+0000 7f788bc4b640 1 -- 172.21.15.40:0/1880032698 shutdown_connections 2024-03-18T22:42:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.654+0000 7f788bc4b640 1 --2- 172.21.15.40:0/1880032698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78841033e0 0x7f7884109c70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.654+0000 7f788bc4b640 1 --2- 172.21.15.40:0/1880032698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884102a20 0x7f7884102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.654+0000 7f788bc4b640 1 --2- 172.21.15.40:0/1880032698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7884101830 0x7f7884101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.654+0000 7f788bc4b640 1 -- 172.21.15.40:0/1880032698 >> 172.21.15.40:0/1880032698 conn(0x7f78840fcdc0 msgr2=0x7f78840ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.654+0000 7f788bc4b640 1 -- 172.21.15.40:0/1880032698 shutdown_connections 2024-03-18T22:42:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.654+0000 7f788bc4b640 1 -- 172.21.15.40:0/1880032698 wait complete. 2024-03-18T22:42:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f788bc4b640 1 Processor -- start 2024-03-18T22:42:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f788bc4b640 1 -- start start 2024-03-18T22:42:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f788bc4b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7884101830 0x7f7884195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f788bc4b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884102a20 0x7f7884195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f788bc4b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78841033e0 0x7f788419cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f788bc4b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f788410b180 con 0x7f7884102a20 2024-03-18T22:42:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f788bc4b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f788410b000 con 0x7f7884101830 2024-03-18T22:42:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f788bc4b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f788410b300 con 0x7f78841033e0 2024-03-18T22:42:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f78891e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884102a20 0x7f7884195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f78891e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884102a20 0x7f7884195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51086/0 (socket says 172.21.15.40:51086) 2024-03-18T22:42:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f78891e6640 1 -- 172.21.15.40:0/2047484749 learned_addr learned my addr 172.21.15.40:0/2047484749 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f78899e7640 1 --2- 172.21.15.40:0/2047484749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7884101830 0x7f7884195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f788a1e8640 1 --2- 172.21.15.40:0/2047484749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78841033e0 0x7f788419cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f78891e6640 1 -- 172.21.15.40:0/2047484749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78841033e0 msgr2=0x7f788419cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f78891e6640 1 --2- 172.21.15.40:0/2047484749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78841033e0 0x7f788419cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f78891e6640 1 -- 172.21.15.40:0/2047484749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7884101830 msgr2=0x7f7884195810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f78891e6640 1 --2- 172.21.15.40:0/2047484749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7884101830 0x7f7884195810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f78891e6640 1 -- 172.21.15.40:0/2047484749 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788419d430 con 0x7f7884102a20 2024-03-18T22:42:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f788a1e8640 1 --2- 172.21.15.40:0/2047484749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78841033e0 0x7f788419cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:42:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.655+0000 7f78891e6640 1 --2- 172.21.15.40:0/2047484749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884102a20 0x7f7884195d50 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7874009be0 tx=0x7f78740071c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.656+0000 7f787affd640 1 -- 172.21.15.40:0/2047484749 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874007e00 con 0x7f7884102a20 2024-03-18T22:42:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.656+0000 7f787affd640 1 -- 172.21.15.40:0/2047484749 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f787401e070 con 0x7f7884102a20 2024-03-18T22:42:31.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.656+0000 7f787affd640 1 -- 172.21.15.40:0/2047484749 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874018730 con 0x7f7884102a20 2024-03-18T22:42:31.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.656+0000 7f788bc4b640 1 -- 172.21.15.40:0/2047484749 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f788419d720 con 0x7f7884102a20 2024-03-18T22:42:31.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.656+0000 7f788bc4b640 1 -- 172.21.15.40:0/2047484749 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f788419dcc0 con 0x7f7884102a20 2024-03-18T22:42:31.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.658+0000 7f787affd640 1 -- 172.21.15.40:0/2047484749 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7874019b30 con 0x7f7884102a20 2024-03-18T22:42:31.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.659+0000 7f788e832740 1 Processor -- start 2024-03-18T22:42:31.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.659+0000 7f788e832740 1 -- start start 2024-03-18T22:42:31.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.659+0000 7f788e832740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab2aff0 0x561f0ab2b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.659+0000 7f788e832740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2b9e0 0x561f0ab304e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.659+0000 7f788e832740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab30b50 0x561f0ab32fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.659+0000 7f788e832740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561f0aa8fde0 con 0x561f0ab30b50 2024-03-18T22:42:31.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.659+0000 7f788e832740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561f0aa38c80 con 0x561f0ab2aff0 2024-03-18T22:42:31.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.659+0000 7f788e832740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x561f0aa8be80 con 0x561f0ab2b9e0 2024-03-18T22:42:31.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.659+0000 7f787affd640 1 --2- 172.21.15.40:0/2047484749 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f78700a2880 0x7f78700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.659+0000 7f78891e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab30b50 0x561f0ab32fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.659+0000 7f78899e7640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2b9e0 0x561f0ab304e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.659+0000 7f788a1e8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab2aff0 0x561f0ab2b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.659+0000 7f78899e7640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2b9e0 0x561f0ab304e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:43002/0 (socket says 172.21.15.40:43002) 2024-03-18T22:42:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.659+0000 7f78899e7640 1 -- 172.21.15.40:0/55215180 learned_addr learned my addr 172.21.15.40:0/55215180 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.659+0000 7f788a1e8640 1 --2- 172.21.15.40:0/2047484749 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f78700a2880 0x7f78700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.660+0000 7f787affd640 1 -- 172.21.15.40:0/2047484749 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f78740d2950 con 0x7f7884102a20 2024-03-18T22:42:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.660+0000 7f78899e7640 1 -- 172.21.15.40:0/55215180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab2aff0 msgr2=0x561f0ab2b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.660+0000 7f78899e7640 1 --2- 172.21.15.40:0/55215180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab2aff0 0x561f0ab2b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.660+0000 7f78899e7640 1 -- 172.21.15.40:0/55215180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab30b50 msgr2=0x561f0ab32fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.660+0000 7f78899e7640 1 --2- 172.21.15.40:0/55215180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab30b50 0x561f0ab32fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.660+0000 7f78899e7640 1 -- 172.21.15.40:0/55215180 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f0aa4afb0 con 0x561f0ab2b9e0 2024-03-18T22:42:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.660+0000 7f788a1e8640 1 --2- 172.21.15.40:0/2047484749 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f78700a2880 0x7f78700a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f787c02c630 tx=0x7f787c004a40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.660+0000 7f78899e7640 1 --2- 172.21.15.40:0/55215180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2b9e0 0x561f0ab304e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7880009ca0 tx=0x7f7880011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10ea1ec4316b83a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.660+0000 7f786ffff640 1 -- 172.21.15.40:0/55215180 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880004220 con 0x561f0ab2b9e0 2024-03-18T22:42:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.660+0000 7f786ffff640 1 -- 172.21.15.40:0/55215180 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7880009e50 con 0x561f0ab2b9e0 2024-03-18T22:42:31.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.660+0000 7f786ffff640 1 -- 172.21.15.40:0/55215180 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880018650 con 0x561f0ab2b9e0 2024-03-18T22:42:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.661+0000 7f788e832740 1 -- 172.21.15.40:0/55215180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2b9e0 msgr2=0x561f0ab304e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.661+0000 7f788e832740 1 --2- 172.21.15.40:0/55215180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2b9e0 0x561f0ab304e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7880009ca0 tx=0x7f7880011810 comp rx=0 tx=0).stop 2024-03-18T22:42:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.661+0000 7f788e832740 1 -- 172.21.15.40:0/55215180 shutdown_connections 2024-03-18T22:42:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.661+0000 7f788e832740 1 --2- 172.21.15.40:0/55215180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab30b50 0x561f0ab32fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.661+0000 7f788e832740 1 --2- 172.21.15.40:0/55215180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2b9e0 0x561f0ab304e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.661+0000 7f788e832740 1 --2- 172.21.15.40:0/55215180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab2aff0 0x561f0ab2b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.661+0000 7f788e832740 1 -- 172.21.15.40:0/55215180 >> 172.21.15.40:0/55215180 conn(0x561f0ab26560 msgr2=0x561f0ab26960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.661+0000 7f788e832740 1 -- 172.21.15.40:0/55215180 shutdown_connections 2024-03-18T22:42:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.661+0000 7f788e832740 1 -- 172.21.15.40:0/55215180 wait complete. 2024-03-18T22:42:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.661+0000 7f788e832740 1 -- start start 2024-03-18T22:42:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.661+0000 7f788e832740 1 Processor -- start 2024-03-18T22:42:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f788e832740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2aff0 0x561f0ab35560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f788e832740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2b9e0 0x561f0ab35aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f788e832740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab30b50 0x561f0ab3cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f788e832740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561f0ab33b00 con 0x561f0ab2b9e0 2024-03-18T22:42:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f788e832740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561f0aa4bfc0 con 0x561f0ab30b50 2024-03-18T22:42:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f788e832740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x561f0ab33c80 con 0x561f0ab2aff0 2024-03-18T22:42:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f78891e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2b9e0 0x561f0ab35aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f788e832740 10 client.0 _ll_register_callbacks cb 0x561f0aa87a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f78891e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2b9e0 0x561f0ab35aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51092/0 (socket says 172.21.15.40:51092) 2024-03-18T22:42:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f78891e6640 1 -- 172.21.15.40:0/1207526214 learned_addr learned my addr 172.21.15.40:0/1207526214 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f788a1e8640 1 --2- 172.21.15.40:0/1207526214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab30b50 0x561f0ab3cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f78899e7640 1 --2- 172.21.15.40:0/1207526214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2aff0 0x561f0ab35560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f78891e6640 1 -- 172.21.15.40:0/1207526214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2aff0 msgr2=0x561f0ab35560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f78891e6640 1 --2- 172.21.15.40:0/1207526214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2aff0 0x561f0ab35560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f78891e6640 1 -- 172.21.15.40:0/1207526214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab30b50 msgr2=0x561f0ab3cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f78891e6640 1 --2- 172.21.15.40:0/1207526214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab30b50 0x561f0ab3cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f78891e6640 1 -- 172.21.15.40:0/1207526214 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f0a7a42d0 con 0x561f0ab2b9e0 2024-03-18T22:42:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f78891e6640 1 --2- 172.21.15.40:0/1207526214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2b9e0 0x561f0ab35aa0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f787409ec30 tx=0x7f787406ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acc4eb313548cd92 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f786d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f786d7fa640 1 -- 172.21.15.40:0/1207526214 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787409e580 con 0x561f0ab2b9e0 2024-03-18T22:42:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f786d7fa640 1 -- 172.21.15.40:0/1207526214 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f787409ac80 con 0x561f0ab2b9e0 2024-03-18T22:42:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.662+0000 7f788e832740 1 -- 172.21.15.40:0/1207526214 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7880009e50 con 0x561f0ab2b9e0 2024-03-18T22:42:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.663+0000 7f786d7fa640 1 -- 172.21.15.40:0/1207526214 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f787409ef30 con 0x561f0ab2b9e0 2024-03-18T22:42:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.663+0000 7f788e832740 10 client.15261 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.663+0000 7f788e832740 1 -- 172.21.15.40:0/1207526214 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561f0a86e6b0 con 0x561f0ab2b9e0 2024-03-18T22:42:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.663+0000 7f786d7fa640 1 -- 172.21.15.40:0/1207526214 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7880009e50 con 0x561f0ab2b9e0 2024-03-18T22:42:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.663+0000 7f786d7fa640 1 -- 172.21.15.40:0/1207526214 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f78740a2050 con 0x561f0ab2b9e0 2024-03-18T22:42:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.663+0000 7f788e832740 10 client.15261 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.663+0000 7f788e832740 10 client.15261 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T22:42:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.663+0000 7f788e832740 20 client.15261 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.663+0000 7f788e832740 1 --2- 172.21.15.40:0/1207526214 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561f0ab42470 0x561f0ab438e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.663+0000 7f788e832740 4 client.15261 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:42:31.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.663+0000 7f788e832740 1 -- 172.21.15.40:0/1207526214 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f0aa9d320 con 0x561f0ab42470 2024-03-18T22:42:31.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.663+0000 7f78899e7640 1 --2- 172.21.15.40:0/1207526214 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561f0ab42470 0x561f0ab438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.664+0000 7f78899e7640 1 --2- 172.21.15.40:0/1207526214 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561f0ab42470 0x561f0ab438e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57a1804d79581f63 server_cookie=e10da7600a9c3abc in_seq=0 out_seq=0 2024-03-18T22:42:31.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.664+0000 7f786d7fa640 10 client.15261 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:42:31.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.665+0000 7f786d7fa640 1 -- 172.21.15.40:0/1207526214 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561f0aa9d320 con 0x561f0ab42470 2024-03-18T22:42:31.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.665+0000 7f786d7fa640 10 client.15261 handle_command_reply: tid=0 2024-03-18T22:42:31.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.665+0000 7f788e832740 1 client.15261 shutdown 2024-03-18T22:42:31.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.665+0000 7f788e832740 1 -- 172.21.15.40:0/1207526214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2b9e0 msgr2=0x561f0ab35aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.665+0000 7f788e832740 1 --2- 172.21.15.40:0/1207526214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2b9e0 0x561f0ab35aa0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f787409ec30 tx=0x7f787406ea50 comp rx=0 tx=0).stop 2024-03-18T22:42:31.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.665+0000 7f788e832740 1 -- 172.21.15.40:0/1207526214 shutdown_connections 2024-03-18T22:42:31.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.665+0000 7f788e832740 1 --2- 172.21.15.40:0/1207526214 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561f0ab42470 0x561f0ab438e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.665+0000 7f788e832740 1 --2- 172.21.15.40:0/1207526214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab30b50 0x561f0ab3cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.665+0000 7f788e832740 1 --2- 172.21.15.40:0/1207526214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2b9e0 0x561f0ab35aa0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.665+0000 7f788e832740 1 --2- 172.21.15.40:0/1207526214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2aff0 0x561f0ab35560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.665+0000 7f788e832740 1 -- 172.21.15.40:0/1207526214 >> 172.21.15.40:0/1207526214 conn(0x561f0ab26560 msgr2=0x561f0ab257e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.665+0000 7f786d7fa640 0 client.15261 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:42:31.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.665+0000 7f788e832740 1 -- 172.21.15.40:0/1207526214 shutdown_connections 2024-03-18T22:42:31.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.665+0000 7f788e832740 1 -- 172.21.15.40:0/1207526214 wait complete. 2024-03-18T22:42:31.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.665+0000 7f788e832740 20 client.15261 trim_cache size 0 max 16384 2024-03-18T22:42:31.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f788e832740 1 Processor -- start 2024-03-18T22:42:31.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f788e832740 1 -- start start 2024-03-18T22:42:31.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f788e832740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2aff0 0x561f0ab2cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f788e832740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2b9e0 0x561f0ab2d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f788e832740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab30b50 0x561f0ab3caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f788e832740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561f0ab41c20 con 0x561f0ab2aff0 2024-03-18T22:42:31.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f788e832740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561f0ab422f0 con 0x561f0ab30b50 2024-03-18T22:42:31.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f788e832740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x561f0ab429c0 con 0x561f0ab2b9e0 2024-03-18T22:42:31.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f78899e7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2aff0 0x561f0ab2cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f78899e7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2aff0 0x561f0ab2cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51096/0 (socket says 172.21.15.40:51096) 2024-03-18T22:42:31.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f78899e7640 1 -- 172.21.15.40:0/150369774 learned_addr learned my addr 172.21.15.40:0/150369774 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f78891e6640 1 --2- 172.21.15.40:0/150369774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2b9e0 0x561f0ab2d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f788a1e8640 1 --2- 172.21.15.40:0/150369774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab30b50 0x561f0ab3caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f78899e7640 1 -- 172.21.15.40:0/150369774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2b9e0 msgr2=0x561f0ab2d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f78899e7640 1 --2- 172.21.15.40:0/150369774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2b9e0 0x561f0ab2d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f78899e7640 1 -- 172.21.15.40:0/150369774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab30b50 msgr2=0x561f0ab3caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f78899e7640 1 --2- 172.21.15.40:0/150369774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab30b50 0x561f0ab3caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.683+0000 7f78899e7640 1 -- 172.21.15.40:0/150369774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f0ab272d0 con 0x561f0ab2aff0 2024-03-18T22:42:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.684+0000 7f78899e7640 1 --2- 172.21.15.40:0/150369774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2aff0 0x561f0ab2cee0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7880009ca0 tx=0x7f78800096b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447cf6853e469e4b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.684+0000 7f786f7fe640 1 -- 172.21.15.40:0/150369774 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788002e070 con 0x561f0ab2aff0 2024-03-18T22:42:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.684+0000 7f786f7fe640 1 -- 172.21.15.40:0/150369774 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7880003d20 con 0x561f0ab2aff0 2024-03-18T22:42:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.684+0000 7f786f7fe640 1 -- 172.21.15.40:0/150369774 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788001e040 con 0x561f0ab2aff0 2024-03-18T22:42:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.684+0000 7f788e832740 1 -- 172.21.15.40:0/150369774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2aff0 msgr2=0x561f0ab2cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.684+0000 7f788e832740 1 --2- 172.21.15.40:0/150369774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2aff0 0x561f0ab2cee0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7880009ca0 tx=0x7f78800096b0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.684+0000 7f788e832740 1 -- 172.21.15.40:0/150369774 shutdown_connections 2024-03-18T22:42:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.684+0000 7f788e832740 1 --2- 172.21.15.40:0/150369774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab30b50 0x561f0ab3caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.684+0000 7f788e832740 1 --2- 172.21.15.40:0/150369774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab2b9e0 0x561f0ab2d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.684+0000 7f788e832740 1 --2- 172.21.15.40:0/150369774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2aff0 0x561f0ab2cee0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.684+0000 7f788e832740 1 -- 172.21.15.40:0/150369774 >> 172.21.15.40:0/150369774 conn(0x561f0ab26560 msgr2=0x561f0ab34050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.684+0000 7f788e832740 1 -- 172.21.15.40:0/150369774 shutdown_connections 2024-03-18T22:42:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.684+0000 7f788e832740 1 -- 172.21.15.40:0/150369774 wait complete. 2024-03-18T22:42:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f788e832740 1 -- start start 2024-03-18T22:42:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f788e832740 1 Processor -- start 2024-03-18T22:42:31.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f788e832740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab2aff0 0x561f0ab37e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f788e832740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2b9e0 0x561f0ab383a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f788e832740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab30b50 0x561f0ab3b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f788e832740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561f0ab43760 con 0x561f0ab2b9e0 2024-03-18T22:42:31.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f788e832740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561f0ab43090 con 0x561f0ab2aff0 2024-03-18T22:42:31.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f788e832740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x561f0ab43e30 con 0x561f0ab30b50 2024-03-18T22:42:31.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f788e832740 10 client.0 _ll_register_callbacks cb 0x7f787409ed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:31.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f78891e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2b9e0 0x561f0ab383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f78899e7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab2aff0 0x561f0ab37e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f78891e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2b9e0 0x561f0ab383a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51100/0 (socket says 172.21.15.40:51100) 2024-03-18T22:42:31.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f78891e6640 1 -- 172.21.15.40:0/2191407609 learned_addr learned my addr 172.21.15.40:0/2191407609 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:31.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f788a1e8640 1 --2- 172.21.15.40:0/2191407609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab30b50 0x561f0ab3b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f78891e6640 1 -- 172.21.15.40:0/2191407609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab30b50 msgr2=0x561f0ab3b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f78891e6640 1 --2- 172.21.15.40:0/2191407609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab30b50 0x561f0ab3b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f78891e6640 1 -- 172.21.15.40:0/2191407609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab2aff0 msgr2=0x561f0ab37e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:31.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f78891e6640 1 --2- 172.21.15.40:0/2191407609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab2aff0 0x561f0ab37e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f78891e6640 1 -- 172.21.15.40:0/2191407609 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f0ab3b9b0 con 0x561f0ab2b9e0 2024-03-18T22:42:31.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f78899e7640 1 --2- 172.21.15.40:0/2191407609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab2aff0 0x561f0ab37e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:31.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f78891e6640 1 --2- 172.21.15.40:0/2191407609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2b9e0 0x561f0ab383a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f787409e310 tx=0x7f787409af30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7137e6ecd555c3e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:31.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.685+0000 7f786d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:31.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.686+0000 7f786d7fa640 1 -- 172.21.15.40:0/2191407609 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787409b1d0 con 0x561f0ab2b9e0 2024-03-18T22:42:31.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.686+0000 7f788e832740 1 -- 172.21.15.40:0/2191407609 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7880003d20 con 0x561f0ab2b9e0 2024-03-18T22:42:31.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.686+0000 7f786d7fa640 1 -- 172.21.15.40:0/2191407609 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f787406eac0 con 0x561f0ab2b9e0 2024-03-18T22:42:31.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.686+0000 7f786d7fa640 1 -- 172.21.15.40:0/2191407609 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f78740a76d0 con 0x561f0ab2b9e0 2024-03-18T22:42:31.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.686+0000 7f788e832740 10 client.15267 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:31.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.686+0000 7f788e832740 1 -- 172.21.15.40:0/2191407609 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561f0ab3cd20 con 0x561f0ab2b9e0 2024-03-18T22:42:31.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.686+0000 7f786d7fa640 1 -- 172.21.15.40:0/2191407609 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7880003d20 con 0x561f0ab2b9e0 2024-03-18T22:42:31.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.686+0000 7f786d7fa640 1 -- 172.21.15.40:0/2191407609 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f78740a2070 con 0x561f0ab2b9e0 2024-03-18T22:42:31.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.686+0000 7f788e832740 10 client.15267 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:31.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.686+0000 7f788e832740 10 client.15267 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T22:42:31.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.686+0000 7f788e832740 20 client.15267 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:31.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.686+0000 7f788e832740 1 --2- 172.21.15.40:0/2191407609 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561f0abaca80 0x561f0abab790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:31.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.686+0000 7f788e832740 4 client.15267 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T22:42:31.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.686+0000 7f788e832740 1 -- 172.21.15.40:0/2191407609 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "status"}) -- 0x561f0aa97670 con 0x561f0abaca80 2024-03-18T22:42:31.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.686+0000 7f78899e7640 1 --2- 172.21.15.40:0/2191407609 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561f0abaca80 0x561f0abab790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:31.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f78899e7640 1 --2- 172.21.15.40:0/2191407609 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561f0abaca80 0x561f0abab790 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b890509933304a server_cookie=94a6c5ff4f895e61 in_seq=0 out_seq=0 2024-03-18T22:42:31.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f786d7fa640 10 client.15267 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:42:31.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f786d7fa640 1 -- 172.21.15.40:0/2191407609 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x561f0aa97670 con 0x561f0abaca80 2024-03-18T22:42:31.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f786d7fa640 10 client.15267 handle_command_reply: tid=0 2024-03-18T22:42:31.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f788e832740 1 client.15267 shutdown 2024-03-18T22:42:31.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f788e832740 1 -- 172.21.15.40:0/2191407609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2b9e0 msgr2=0x561f0ab383a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f788e832740 1 --2- 172.21.15.40:0/2191407609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2b9e0 0x561f0ab383a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f787409e310 tx=0x7f787409af30 comp rx=0 tx=0).stop 2024-03-18T22:42:31.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f788e832740 1 -- 172.21.15.40:0/2191407609 shutdown_connections 2024-03-18T22:42:31.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f788e832740 1 --2- 172.21.15.40:0/2191407609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f0ab30b50 0x561f0ab3b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f788e832740 1 --2- 172.21.15.40:0/2191407609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f0ab2b9e0 0x561f0ab383a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f788e832740 1 --2- 172.21.15.40:0/2191407609 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561f0abaca80 0x561f0abab790 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f788e832740 1 --2- 172.21.15.40:0/2191407609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f0ab2aff0 0x561f0ab37e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f786d7fa640 0 client.15267 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:42:31.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f788e832740 1 -- 172.21.15.40:0/2191407609 >> 172.21.15.40:0/2191407609 conn(0x561f0ab26560 msgr2=0x561f0ab34050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f788e832740 1 -- 172.21.15.40:0/2191407609 shutdown_connections 2024-03-18T22:42:31.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f788e832740 1 -- 172.21.15.40:0/2191407609 wait complete. 2024-03-18T22:42:31.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.687+0000 7f788e832740 20 client.15267 trim_cache size 0 max 16384 2024-03-18T22:42:31.698 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:42:31.699 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "410ef414-e577-11ee-95c9-87774f69a715", 2024-03-18T22:42:31.699 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-18T22:42:31.699 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24509, 2024-03-18T22:42:31.699 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-18T22:42:31.699 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-18T22:42:31.699 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-18T22:42:31.699 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 109.488573871, 2024-03-18T22:42:31.699 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 27, 2024-03-18T22:42:31.699 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 91, 2024-03-18T22:42:31.699 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T22:42:31.699 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 113.971437982 2024-03-18T22:42:31.699 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:42:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.688+0000 7f788bc4b640 1 -- 172.21.15.40:0/2047484749 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f78700a2880 msgr2=0x7f78700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.688+0000 7f788bc4b640 1 --2- 172.21.15.40:0/2047484749 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f78700a2880 0x7f78700a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f787c02c630 tx=0x7f787c004a40 comp rx=0 tx=0).stop 2024-03-18T22:42:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.688+0000 7f788bc4b640 1 -- 172.21.15.40:0/2047484749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884102a20 msgr2=0x7f7884195d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.688+0000 7f788bc4b640 1 --2- 172.21.15.40:0/2047484749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884102a20 0x7f7884195d50 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7874009be0 tx=0x7f78740071c0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.688+0000 7f788bc4b640 1 -- 172.21.15.40:0/2047484749 shutdown_connections 2024-03-18T22:42:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.688+0000 7f788bc4b640 1 --2- 172.21.15.40:0/2047484749 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f78700a2880 0x7f78700a4d40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.688+0000 7f788bc4b640 1 --2- 172.21.15.40:0/2047484749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f78841033e0 0x7f788419cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.688+0000 7f788bc4b640 1 --2- 172.21.15.40:0/2047484749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7884102a20 0x7f7884195d50 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.688+0000 7f788bc4b640 1 --2- 172.21.15.40:0/2047484749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7884101830 0x7f7884195810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.688+0000 7f788bc4b640 1 -- 172.21.15.40:0/2047484749 >> 172.21.15.40:0/2047484749 conn(0x7f78840fcdc0 msgr2=0x7f7884105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.688+0000 7f788bc4b640 1 -- 172.21.15.40:0/2047484749 shutdown_connections 2024-03-18T22:42:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:31.688+0000 7f788bc4b640 1 -- 172.21.15.40:0/2047484749 wait complete. 2024-03-18T22:42:31.702 INFO:tasks.cephadm:Setup complete, yielding 2024-03-18T22:42:31.702 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T22:42:31.712 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2024-03-18T22:42:31.712 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-18T22:42:31.958 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:42:32.192 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:31 smithi040 ceph-mon[62643]: pgmap v345: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:42:32.192 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:31 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1065282537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:42:32.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:31 smithi113 ceph-mon[66775]: pgmap v345: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:42:32.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:31 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1065282537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:42:32.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:31 smithi103 ceph-mon[66377]: pgmap v345: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:42:32.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:31 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1065282537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:42:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.715+0000 7fde8c738700 1 -- 172.21.15.40:0/3553782817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde84102e50 msgr2=0x7fde84109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.715+0000 7fde8c738700 1 --2- 172.21.15.40:0/3553782817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde84102e50 0x7fde84109700 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0099c0 tx=0x7fde7c009cd0 comp rx=0 tx=0).stop 2024-03-18T22:42:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.715+0000 7fde8c738700 1 -- 172.21.15.40:0/3553782817 shutdown_connections 2024-03-18T22:42:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.715+0000 7fde8c738700 1 --2- 172.21.15.40:0/3553782817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde84102e50 0x7fde84109700 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.715+0000 7fde8c738700 1 --2- 172.21.15.40:0/3553782817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde841012b0 0x7fde841016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.715+0000 7fde8c738700 1 --2- 172.21.15.40:0/3553782817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde841024b0 0x7fde84102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.715+0000 7fde8c738700 1 -- 172.21.15.40:0/3553782817 >> 172.21.15.40:0/3553782817 conn(0x7fde840fc870 msgr2=0x7fde840fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.715+0000 7fde8c738700 1 -- 172.21.15.40:0/3553782817 shutdown_connections 2024-03-18T22:42:33.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.715+0000 7fde8c738700 1 -- 172.21.15.40:0/3553782817 wait complete. 2024-03-18T22:42:33.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.716+0000 7fde8c738700 1 Processor -- start 2024-03-18T22:42:33.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.716+0000 7fde8c738700 1 -- start start 2024-03-18T22:42:33.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.716+0000 7fde8c738700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde841012b0 0x7fde8406b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:33.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.716+0000 7fde8c738700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde841024b0 0x7fde8419d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:33.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.716+0000 7fde8c738700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde84102e50 0x7fde8419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:33.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.716+0000 7fde8c738700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde8410ccb0 con 0x7fde84102e50 2024-03-18T22:42:33.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.716+0000 7fde8c738700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde8410cb30 con 0x7fde841012b0 2024-03-18T22:42:33.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.716+0000 7fde8c738700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fde8410ce30 con 0x7fde841024b0 2024-03-18T22:42:33.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.716+0000 7fde8acd5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde84102e50 0x7fde8419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:33.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.716+0000 7fde8a4d4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde841012b0 0x7fde8406b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:33.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.716+0000 7fde8acd5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde84102e50 0x7fde8419f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51114/0 (socket says 172.21.15.40:51114) 2024-03-18T22:42:33.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.716+0000 7fde8acd5700 1 -- 172.21.15.40:0/2922237022 learned_addr learned my addr 172.21.15.40:0/2922237022 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:33.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.716+0000 7fde89cd3700 1 --2- 172.21.15.40:0/2922237022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde841024b0 0x7fde8419d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.717+0000 7fde8acd5700 1 -- 172.21.15.40:0/2922237022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde841024b0 msgr2=0x7fde8419d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.717+0000 7fde8acd5700 1 --2- 172.21.15.40:0/2922237022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde841024b0 0x7fde8419d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.717+0000 7fde8acd5700 1 -- 172.21.15.40:0/2922237022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde841012b0 msgr2=0x7fde8406b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.717+0000 7fde8acd5700 1 --2- 172.21.15.40:0/2922237022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde841012b0 0x7fde8406b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.717+0000 7fde8acd5700 1 -- 172.21.15.40:0/2922237022 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde841a0040 con 0x7fde84102e50 2024-03-18T22:42:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.717+0000 7fde89cd3700 1 --2- 172.21.15.40:0/2922237022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde841024b0 0x7fde8419d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.717+0000 7fde8a4d4700 1 --2- 172.21.15.40:0/2922237022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde841012b0 0x7fde8406b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.717+0000 7fde8acd5700 1 --2- 172.21.15.40:0/2922237022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde84102e50 0x7fde8419f770 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fde7c003820 tx=0x7fde7c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.717+0000 7fde837fe700 1 -- 172.21.15.40:0/2922237022 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7c009e60 con 0x7fde84102e50 2024-03-18T22:42:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.717+0000 7fde837fe700 1 -- 172.21.15.40:0/2922237022 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde7c0038c0 con 0x7fde84102e50 2024-03-18T22:42:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.717+0000 7fde837fe700 1 -- 172.21.15.40:0/2922237022 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7c0055f0 con 0x7fde84102e50 2024-03-18T22:42:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.717+0000 7fde8c738700 1 -- 172.21.15.40:0/2922237022 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde841a02d0 con 0x7fde84102e50 2024-03-18T22:42:33.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.717+0000 7fde8c738700 1 -- 172.21.15.40:0/2922237022 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde841a0670 con 0x7fde84102e50 2024-03-18T22:42:33.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.718+0000 7fde8c738700 1 -- 172.21.15.40:0/2922237022 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde84068fd0 con 0x7fde84102e50 2024-03-18T22:42:33.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.719+0000 7fde837fe700 1 -- 172.21.15.40:0/2922237022 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fde7c003a70 con 0x7fde84102e50 2024-03-18T22:42:33.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.720+0000 7fde837fe700 1 --2- 172.21.15.40:0/2922237022 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fde6c0a2890 0x7fde6c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:33.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.720+0000 7fde8a4d4700 1 --2- 172.21.15.40:0/2922237022 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fde6c0a2890 0x7fde6c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:33.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.720+0000 7fde837fe700 1 -- 172.21.15.40:0/2922237022 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fde7c0cfd10 con 0x7fde84102e50 2024-03-18T22:42:33.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.721+0000 7fde8a4d4700 1 --2- 172.21.15.40:0/2922237022 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fde6c0a2890 0x7fde6c0a4d50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fde84102310 tx=0x7fde7400b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:33.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.725+0000 7fde837fe700 1 -- 172.21.15.40:0/2922237022 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fde7c0d5020 con 0x7fde84102e50 2024-03-18T22:42:33.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.948+0000 7fde8c738700 1 -- 172.21.15.40:0/2922237022 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fde84062750 con 0x7fde6c0a2890 2024-03-18T22:42:33.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.950+0000 7fde837fe700 1 -- 172.21.15.40:0/2922237022 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fde84062750 con 0x7fde6c0a2890 2024-03-18T22:42:33.951 INFO:teuthology.orchestra.run.smithi040.stdout:Backend: cephadm 2024-03-18T22:42:33.951 INFO:teuthology.orchestra.run.smithi040.stdout:Available: Yes 2024-03-18T22:42:33.951 INFO:teuthology.orchestra.run.smithi040.stdout:Paused: No 2024-03-18T22:42:33.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.952+0000 7fde8c738700 1 -- 172.21.15.40:0/2922237022 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fde6c0a2890 msgr2=0x7fde6c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:33.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.952+0000 7fde8c738700 1 --2- 172.21.15.40:0/2922237022 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fde6c0a2890 0x7fde6c0a4d50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fde84102310 tx=0x7fde7400b040 comp rx=0 tx=0).stop 2024-03-18T22:42:33.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.952+0000 7fde8c738700 1 -- 172.21.15.40:0/2922237022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde84102e50 msgr2=0x7fde8419f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:33.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.952+0000 7fde8c738700 1 --2- 172.21.15.40:0/2922237022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde84102e50 0x7fde8419f770 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fde7c003820 tx=0x7fde7c00f690 comp rx=0 tx=0).stop 2024-03-18T22:42:33.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.952+0000 7fde8c738700 1 -- 172.21.15.40:0/2922237022 shutdown_connections 2024-03-18T22:42:33.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.952+0000 7fde8c738700 1 --2- 172.21.15.40:0/2922237022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde84102e50 0x7fde8419f770 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:33.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.952+0000 7fde8c738700 1 --2- 172.21.15.40:0/2922237022 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fde6c0a2890 0x7fde6c0a4d50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:33.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.952+0000 7fde8c738700 1 --2- 172.21.15.40:0/2922237022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde841012b0 0x7fde8406b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:33.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.952+0000 7fde8c738700 1 --2- 172.21.15.40:0/2922237022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde841024b0 0x7fde8419d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:33.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.952+0000 7fde8c738700 1 -- 172.21.15.40:0/2922237022 >> 172.21.15.40:0/2922237022 conn(0x7fde840fc870 msgr2=0x7fde84105880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:33.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.952+0000 7fde8c738700 1 -- 172.21.15.40:0/2922237022 shutdown_connections 2024-03-18T22:42:33.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:33.952+0000 7fde8c738700 1 -- 172.21.15.40:0/2922237022 wait complete. 2024-03-18T22:42:34.318 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:33 smithi040 ceph-mon[62643]: pgmap v346: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:34.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:33 smithi113 ceph-mon[66775]: pgmap v346: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:34.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:33 smithi103 ceph-mon[66377]: pgmap v346: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:34.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-18T22:42:34.983 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:42:35.104 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:34 smithi040 ceph-mon[62643]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:42:35.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:34 smithi113 ceph-mon[66775]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:42:35.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:34 smithi103 ceph-mon[66377]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:42:36.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:35 smithi040 ceph-mon[62643]: pgmap v347: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T22:42:36.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:35 smithi113 ceph-mon[66775]: pgmap v347: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T22:42:36.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:35 smithi103 ceph-mon[66377]: pgmap v347: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T22:42:36.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.761+0000 7f2b41c4e700 1 -- 172.21.15.40:0/315576284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b3c103560 msgr2=0x7f2b3c1039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:36.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.761+0000 7f2b41c4e700 1 --2- 172.21.15.40:0/315576284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b3c103560 0x7f2b3c1039c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2b300099c0 tx=0x7f2b30009cd0 comp rx=0 tx=0).stop 2024-03-18T22:42:36.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.761+0000 7f2b41c4e700 1 -- 172.21.15.40:0/315576284 shutdown_connections 2024-03-18T22:42:36.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.761+0000 7f2b41c4e700 1 --2- 172.21.15.40:0/315576284 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b3c103f00 0x7f2b3c10a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:36.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.761+0000 7f2b41c4e700 1 --2- 172.21.15.40:0/315576284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3c102360 0x7f2b3c102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:36.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.761+0000 7f2b41c4e700 1 --2- 172.21.15.40:0/315576284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b3c103560 0x7f2b3c1039c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:36.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.761+0000 7f2b41c4e700 1 -- 172.21.15.40:0/315576284 >> 172.21.15.40:0/315576284 conn(0x7f2b3c0fd8c0 msgr2=0x7f2b3c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:36.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.761+0000 7f2b41c4e700 1 -- 172.21.15.40:0/315576284 shutdown_connections 2024-03-18T22:42:36.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.761+0000 7f2b41c4e700 1 -- 172.21.15.40:0/315576284 wait complete. 2024-03-18T22:42:36.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.762+0000 7f2b41c4e700 1 Processor -- start 2024-03-18T22:42:36.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.762+0000 7f2b41c4e700 1 -- start start 2024-03-18T22:42:36.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.762+0000 7f2b41c4e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b3c102360 0x7f2b3c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:36.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.762+0000 7f2b41c4e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b3c103560 0x7f2b3c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:36.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.762+0000 7f2b41c4e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3c103f00 0x7f2b3c19fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:36.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.762+0000 7f2b41c4e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3c10dde0 con 0x7f2b3c103560 2024-03-18T22:42:36.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.762+0000 7f2b41c4e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3c10dc60 con 0x7f2b3c103f00 2024-03-18T22:42:36.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.762+0000 7f2b41c4e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3c10df60 con 0x7f2b3c102360 2024-03-18T22:42:36.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.762+0000 7f2b3affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b3c103560 0x7f2b3c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:36.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.763+0000 7f2b3affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b3c103560 0x7f2b3c1989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51128/0 (socket says 172.21.15.40:51128) 2024-03-18T22:42:36.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.763+0000 7f2b3affd700 1 -- 172.21.15.40:0/2036852626 learned_addr learned my addr 172.21.15.40:0/2036852626 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:36.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.763+0000 7f2b3bfff700 1 --2- 172.21.15.40:0/2036852626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3c103f00 0x7f2b3c19fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.763+0000 7f2b3b7fe700 1 --2- 172.21.15.40:0/2036852626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b3c102360 0x7f2b3c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.763+0000 7f2b3affd700 1 -- 172.21.15.40:0/2036852626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b3c102360 msgr2=0x7f2b3c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.763+0000 7f2b3affd700 1 --2- 172.21.15.40:0/2036852626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b3c102360 0x7f2b3c198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.763+0000 7f2b3affd700 1 -- 172.21.15.40:0/2036852626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3c103f00 msgr2=0x7f2b3c19fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.763+0000 7f2b3affd700 1 --2- 172.21.15.40:0/2036852626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3c103f00 0x7f2b3c19fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.763+0000 7f2b3affd700 1 -- 172.21.15.40:0/2036852626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b3c1a00f0 con 0x7f2b3c103560 2024-03-18T22:42:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.763+0000 7f2b3affd700 1 --2- 172.21.15.40:0/2036852626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b3c103560 0x7f2b3c1989d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2b30005910 tx=0x7f2b3000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.763+0000 7f2b38ff9700 1 -- 172.21.15.40:0/2036852626 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b30009e60 con 0x7f2b3c103560 2024-03-18T22:42:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.763+0000 7f2b38ff9700 1 -- 172.21.15.40:0/2036852626 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b300038c0 con 0x7f2b3c103560 2024-03-18T22:42:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.763+0000 7f2b38ff9700 1 -- 172.21.15.40:0/2036852626 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b30005640 con 0x7f2b3c103560 2024-03-18T22:42:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.763+0000 7f2b41c4e700 1 -- 172.21.15.40:0/2036852626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b3c1a0380 con 0x7f2b3c103560 2024-03-18T22:42:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.763+0000 7f2b41c4e700 1 -- 172.21.15.40:0/2036852626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b3c1a07d0 con 0x7f2b3c103560 2024-03-18T22:42:36.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.764+0000 7f2b41c4e700 1 -- 172.21.15.40:0/2036852626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b3c068fd0 con 0x7f2b3c103560 2024-03-18T22:42:36.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.765+0000 7f2b38ff9700 1 -- 172.21.15.40:0/2036852626 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2b30003a70 con 0x7f2b3c103560 2024-03-18T22:42:36.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.766+0000 7f2b38ff9700 1 --2- 172.21.15.40:0/2036852626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2b240a2910 0x7f2b240a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:36.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.766+0000 7f2b3b7fe700 1 --2- 172.21.15.40:0/2036852626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2b240a2910 0x7f2b240a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:36.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.766+0000 7f2b38ff9700 1 -- 172.21.15.40:0/2036852626 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2b300cfe80 con 0x7f2b3c103560 2024-03-18T22:42:36.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.766+0000 7f2b3b7fe700 1 --2- 172.21.15.40:0/2036852626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2b240a2910 0x7f2b240a4dd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c1033c0 tx=0x7f2b2c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:36.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.769+0000 7f2b38ff9700 1 -- 172.21.15.40:0/2036852626 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2b30013070 con 0x7f2b3c103560 2024-03-18T22:42:36.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.983+0000 7f2b41c4e700 1 -- 172.21.15.40:0/2036852626 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f2b3c062810 con 0x7f2b240a2910 2024-03-18T22:42:36.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.997+0000 7f2b38ff9700 1 -- 172.21.15.40:0/2036852626 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f2b3c062810 con 0x7f2b240a2910 2024-03-18T22:42:36.999 INFO:teuthology.orchestra.run.smithi040.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-18T22:42:36.999 INFO:teuthology.orchestra.run.smithi040.stdout:mds.a smithi040 running (2m) 25s ago 2m 18.6M - 19.0.0-2206-gb6ae2491 2312c8e42d00 555cd76a8073 2024-03-18T22:42:36.999 INFO:teuthology.orchestra.run.smithi040.stdout:mds.b smithi103 running (119s) 53s ago 119s 26.5M - 19.0.0-2206-gb6ae2491 2312c8e42d00 670bfc4263ed 2024-03-18T22:42:36.999 INFO:teuthology.orchestra.run.smithi040.stdout:mds.c smithi113 running (2m) 53s ago 2m 16.2M - 19.0.0-2206-gb6ae2491 2312c8e42d00 6830d5373e13 2024-03-18T22:42:36.999 INFO:teuthology.orchestra.run.smithi040.stdout:mds.d smithi040 running (2m) 25s ago 2m 14.1M - 19.0.0-2206-gb6ae2491 2312c8e42d00 9ac863149aff 2024-03-18T22:42:36.999 INFO:teuthology.orchestra.run.smithi040.stdout:mds.e smithi103 running (2m) 53s ago 2m 16.2M - 19.0.0-2206-gb6ae2491 2312c8e42d00 2ede6c30917e 2024-03-18T22:42:36.999 INFO:teuthology.orchestra.run.smithi040.stdout:mds.f smithi113 running (2m) 53s ago 2m 18.3M - 19.0.0-2206-gb6ae2491 2312c8e42d00 a6d4038b9cea 2024-03-18T22:42:36.999 INFO:teuthology.orchestra.run.smithi040.stdout:mds.g smithi040 running (2m) 25s ago 2m 18.8M - 19.0.0-2206-gb6ae2491 2312c8e42d00 0741cacc890b 2024-03-18T22:42:36.999 INFO:teuthology.orchestra.run.smithi040.stdout:mds.h smithi103 running (2m) 53s ago 2m 16.9M - 19.0.0-2206-gb6ae2491 2312c8e42d00 ae455a62f938 2024-03-18T22:42:36.999 INFO:teuthology.orchestra.run.smithi040.stdout:mds.i smithi113 running (2m) 53s ago 2m 16.7M - 19.0.0-2206-gb6ae2491 2312c8e42d00 124c2cdc599d 2024-03-18T22:42:36.999 INFO:teuthology.orchestra.run.smithi040.stdout:mds.j smithi040 running (2m) 25s ago 2m 14.0M - 19.0.0-2206-gb6ae2491 2312c8e42d00 312e652f8959 2024-03-18T22:42:36.999 INFO:teuthology.orchestra.run.smithi040.stdout:mds.k smithi103 running (2m) 53s ago 2m 17.4M - 19.0.0-2206-gb6ae2491 2312c8e42d00 3acf8ab35b6b 2024-03-18T22:42:36.999 INFO:teuthology.orchestra.run.smithi040.stdout:mds.l smithi113 running (2m) 53s ago 2m 18.7M - 19.0.0-2206-gb6ae2491 2312c8e42d00 382717cf9b9c 2024-03-18T22:42:37.000 INFO:teuthology.orchestra.run.smithi040.stdout:mgr.x smithi040 *:9283,8765 running (9m) 25s ago 9m 513M - 19.0.0-2206-gb6ae2491 2312c8e42d00 ea399b57c2c6 2024-03-18T22:42:37.000 INFO:teuthology.orchestra.run.smithi040.stdout:mgr.y smithi103 *:8443,8765 running (7m) 53s ago 7m 453M - 19.0.0-2206-gb6ae2491 2312c8e42d00 1cad52968eff 2024-03-18T22:42:37.000 INFO:teuthology.orchestra.run.smithi040.stdout:mgr.z smithi113 *:8443,8765 running (7m) 53s ago 7m 451M - 19.0.0-2206-gb6ae2491 2312c8e42d00 94352a482c0b 2024-03-18T22:42:37.000 INFO:teuthology.orchestra.run.smithi040.stdout:mon.a smithi040 running (9m) 25s ago 9m 49.5M 2048M 19.0.0-2206-gb6ae2491 2312c8e42d00 6bd216c61bbf 2024-03-18T22:42:37.000 INFO:teuthology.orchestra.run.smithi040.stdout:mon.b smithi103 running (8m) 53s ago 8m 42.4M 2048M 19.0.0-2206-gb6ae2491 2312c8e42d00 c4d19a7c8bce 2024-03-18T22:42:37.000 INFO:teuthology.orchestra.run.smithi040.stdout:mon.c smithi113 running (8m) 53s ago 8m 40.6M 2048M 19.0.0-2206-gb6ae2491 2312c8e42d00 3c9a2ff1f204 2024-03-18T22:42:37.000 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0 smithi040 running (7m) 25s ago 7m 53.3M 4273M 19.0.0-2206-gb6ae2491 2312c8e42d00 c8b687eb0042 2024-03-18T22:42:37.000 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1 smithi103 running (7m) 53s ago 7m 54.0M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 f3670ae93f49 2024-03-18T22:42:37.000 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2 smithi113 running (6m) 53s ago 6m 46.2M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 0c44a058bdf8 2024-03-18T22:42:37.000 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3 smithi040 running (6m) 25s ago 6m 52.5M 4273M 19.0.0-2206-gb6ae2491 2312c8e42d00 ee8635ed7f43 2024-03-18T22:42:37.000 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4 smithi103 running (5m) 53s ago 5m 50.6M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 b151016593fd 2024-03-18T22:42:37.000 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5 smithi113 running (5m) 53s ago 5m 44.3M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 dd9c38d46eea 2024-03-18T22:42:37.000 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6 smithi040 running (4m) 25s ago 4m 42.3M 4273M 19.0.0-2206-gb6ae2491 2312c8e42d00 b6980037c010 2024-03-18T22:42:37.000 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 smithi103 running (4m) 53s ago 4m 44.8M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 68bd9931a8c1 2024-03-18T22:42:37.000 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8 smithi113 running (4m) 53s ago 4m 43.0M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 4846d388d3c6 2024-03-18T22:42:37.001 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9 smithi040 running (3m) 25s ago 3m 41.8M 4273M 19.0.0-2206-gb6ae2491 2312c8e42d00 5aa5fddb7c22 2024-03-18T22:42:37.001 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10 smithi103 running (3m) 53s ago 3m 52.9M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 e6156fe9b109 2024-03-18T22:42:37.001 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11 smithi113 running (2m) 53s ago 2m 48.3M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 e2838393f202 2024-03-18T22:42:37.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.999+0000 7f2b41c4e700 1 -- 172.21.15.40:0/2036852626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2b240a2910 msgr2=0x7f2b240a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:37.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.999+0000 7f2b41c4e700 1 --2- 172.21.15.40:0/2036852626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2b240a2910 0x7f2b240a4dd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c1033c0 tx=0x7f2b2c009260 comp rx=0 tx=0).stop 2024-03-18T22:42:37.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.999+0000 7f2b41c4e700 1 -- 172.21.15.40:0/2036852626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b3c103560 msgr2=0x7f2b3c1989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.999+0000 7f2b41c4e700 1 --2- 172.21.15.40:0/2036852626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b3c103560 0x7f2b3c1989d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2b30005910 tx=0x7f2b3000f690 comp rx=0 tx=0).stop 2024-03-18T22:42:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.999+0000 7f2b41c4e700 1 -- 172.21.15.40:0/2036852626 shutdown_connections 2024-03-18T22:42:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.999+0000 7f2b41c4e700 1 --2- 172.21.15.40:0/2036852626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2b240a2910 0x7f2b240a4dd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.999+0000 7f2b41c4e700 1 --2- 172.21.15.40:0/2036852626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b3c103f00 0x7f2b3c19fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.999+0000 7f2b41c4e700 1 --2- 172.21.15.40:0/2036852626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b3c102360 0x7f2b3c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.999+0000 7f2b41c4e700 1 --2- 172.21.15.40:0/2036852626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b3c103560 0x7f2b3c1989d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.999+0000 7f2b41c4e700 1 -- 172.21.15.40:0/2036852626 >> 172.21.15.40:0/2036852626 conn(0x7f2b3c0fd8c0 msgr2=0x7f2b3c1068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.999+0000 7f2b41c4e700 1 -- 172.21.15.40:0/2036852626 shutdown_connections 2024-03-18T22:42:37.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:36.999+0000 7f2b41c4e700 1 -- 172.21.15.40:0/2036852626 wait complete. 2024-03-18T22:42:37.618 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-18T22:42:37.870 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:42:38.234 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:37 smithi040 ceph-mon[62643]: pgmap v348: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:38.234 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:37 smithi040 ceph-mon[62643]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:42:38.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:37 smithi113 ceph-mon[66775]: pgmap v348: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:38.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:37 smithi113 ceph-mon[66775]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:42:38.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:37 smithi103 ceph-mon[66377]: pgmap v348: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:38.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:37 smithi103 ceph-mon[66377]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:42:39.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.607+0000 7fccc6f81700 1 -- 172.21.15.40:0/1581802443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103ee0 msgr2=0x7fccc010a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:39.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.607+0000 7fccc6f81700 1 --2- 172.21.15.40:0/1581802443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103ee0 0x7fccc010a790 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fccb800b0d0 tx=0x7fccb800b3e0 comp rx=0 tx=0).stop 2024-03-18T22:42:39.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.607+0000 7fccc6f81700 1 -- 172.21.15.40:0/1581802443 shutdown_connections 2024-03-18T22:42:39.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.607+0000 7fccc6f81700 1 --2- 172.21.15.40:0/1581802443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103ee0 0x7fccc010a790 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:39.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.607+0000 7fccc6f81700 1 --2- 172.21.15.40:0/1581802443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccc0102340 0x7fccc0102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:39.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.607+0000 7fccc6f81700 1 --2- 172.21.15.40:0/1581802443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc0103540 0x7fccc01039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:39.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.607+0000 7fccc6f81700 1 -- 172.21.15.40:0/1581802443 >> 172.21.15.40:0/1581802443 conn(0x7fccc00fd920 msgr2=0x7fccc00ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:39.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.607+0000 7fccc6f81700 1 -- 172.21.15.40:0/1581802443 shutdown_connections 2024-03-18T22:42:39.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.607+0000 7fccc6f81700 1 -- 172.21.15.40:0/1581802443 wait complete. 2024-03-18T22:42:39.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.607+0000 7fccc6f81700 1 Processor -- start 2024-03-18T22:42:39.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccc6f81700 1 -- start start 2024-03-18T22:42:39.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccc6f81700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc0102340 0x7fccc00760b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:39.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccc6f81700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103540 0x7fccc0074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:39.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccc6f81700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccc0103ee0 0x7fccc0074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:39.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccc6f81700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fccc010ddc0 con 0x7fccc0103540 2024-03-18T22:42:39.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccc6f81700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fccc010dc40 con 0x7fccc0103ee0 2024-03-18T22:42:39.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccc6f81700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fccc010df40 con 0x7fccc0102340 2024-03-18T22:42:39.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccbffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103540 0x7fccc0074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:39.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccc4d1d700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc0102340 0x7fccc00760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccbffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103540 0x7fccc0074700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38370/0 (socket says 172.21.15.40:38370) 2024-03-18T22:42:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccbffff700 1 -- 172.21.15.40:0/2979902869 learned_addr learned my addr 172.21.15.40:0/2979902869 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccc551e700 1 --2- 172.21.15.40:0/2979902869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccc0103ee0 0x7fccc0074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccbffff700 1 -- 172.21.15.40:0/2979902869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc0102340 msgr2=0x7fccc00760b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccbffff700 1 --2- 172.21.15.40:0/2979902869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc0102340 0x7fccc00760b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccbffff700 1 -- 172.21.15.40:0/2979902869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccc0103ee0 msgr2=0x7fccc0074c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccbffff700 1 --2- 172.21.15.40:0/2979902869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccc0103ee0 0x7fccc0074c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccbffff700 1 -- 172.21.15.40:0/2979902869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc00754f0 con 0x7fccc0103540 2024-03-18T22:42:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccc4d1d700 1 --2- 172.21.15.40:0/2979902869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc0102340 0x7fccc00760b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.608+0000 7fccbffff700 1 --2- 172.21.15.40:0/2979902869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103540 0x7fccc0074700 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fccac00e9a0 tx=0x7fccac00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.609+0000 7fccbdffb700 1 -- 172.21.15.40:0/2979902869 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccac00ca10 con 0x7fccc0103540 2024-03-18T22:42:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.609+0000 7fccbdffb700 1 -- 172.21.15.40:0/2979902869 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fccac00cbb0 con 0x7fccc0103540 2024-03-18T22:42:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.609+0000 7fccc6f81700 1 -- 172.21.15.40:0/2979902869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccc01a2590 con 0x7fccc0103540 2024-03-18T22:42:39.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.609+0000 7fccbdffb700 1 -- 172.21.15.40:0/2979902869 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccac018680 con 0x7fccc0103540 2024-03-18T22:42:39.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.609+0000 7fccc6f81700 1 -- 172.21.15.40:0/2979902869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccc01a2aa0 con 0x7fccc0103540 2024-03-18T22:42:39.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.610+0000 7fccc6f81700 1 -- 172.21.15.40:0/2979902869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccc0068fd0 con 0x7fccc0103540 2024-03-18T22:42:39.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.611+0000 7fccbdffb700 1 -- 172.21.15.40:0/2979902869 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fccac010430 con 0x7fccc0103540 2024-03-18T22:42:39.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.611+0000 7fccbdffb700 1 --2- 172.21.15.40:0/2979902869 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fccb00a2890 0x7fccb00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:39.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.611+0000 7fccbdffb700 1 -- 172.21.15.40:0/2979902869 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fccac0cf180 con 0x7fccc0103540 2024-03-18T22:42:39.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.611+0000 7fccc4d1d700 1 --2- 172.21.15.40:0/2979902869 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fccb00a2890 0x7fccb00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:39.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.612+0000 7fccc4d1d700 1 --2- 172.21.15.40:0/2979902869 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fccb00a2890 0x7fccb00a4d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fccc01033a0 tx=0x7fccb4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:39.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.615+0000 7fccbdffb700 1 -- 172.21.15.40:0/2979902869 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fccac09bec0 con 0x7fccc0103540 2024-03-18T22:42:39.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.830+0000 7fccc6f81700 1 -- 172.21.15.40:0/2979902869 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fccc0062750 con 0x7fccb00a2890 2024-03-18T22:42:39.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.836+0000 7fccbdffb700 1 -- 172.21.15.40:0/2979902869 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fccc0062750 con 0x7fccb00a2890 2024-03-18T22:42:39.837 INFO:teuthology.orchestra.run.smithi040.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-18T22:42:39.838 INFO:teuthology.orchestra.run.smithi040.stdout:mds.all 12/12 56s ago 2m smithi040=a;smithi040=d;smithi040=g;smithi040=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi113=c;smithi113=f;smithi113=i;smithi113=l;count:12 2024-03-18T22:42:39.838 INFO:teuthology.orchestra.run.smithi040.stdout:mgr 3/3 56s ago 7m smithi040=x;smithi103=y;smithi113=z;count:3 2024-03-18T22:42:39.838 INFO:teuthology.orchestra.run.smithi040.stdout:mon 3/3 56s ago 8m smithi040:172.21.15.40=a;smithi103:172.21.15.103=b;smithi113:172.21.15.113=c;count:3 2024-03-18T22:42:39.838 INFO:teuthology.orchestra.run.smithi040.stdout:osd 12 56s ago - 2024-03-18T22:42:39.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.839+0000 7fccc6f81700 1 -- 172.21.15.40:0/2979902869 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fccb00a2890 msgr2=0x7fccb00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:39.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.839+0000 7fccc6f81700 1 --2- 172.21.15.40:0/2979902869 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fccb00a2890 0x7fccb00a4d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fccc01033a0 tx=0x7fccb4009260 comp rx=0 tx=0).stop 2024-03-18T22:42:39.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.839+0000 7fccc6f81700 1 -- 172.21.15.40:0/2979902869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103540 msgr2=0x7fccc0074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:39.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.839+0000 7fccc6f81700 1 --2- 172.21.15.40:0/2979902869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103540 0x7fccc0074700 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fccac00e9a0 tx=0x7fccac00ecb0 comp rx=0 tx=0).stop 2024-03-18T22:42:39.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.839+0000 7fccc6f81700 1 -- 172.21.15.40:0/2979902869 shutdown_connections 2024-03-18T22:42:39.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.839+0000 7fccc6f81700 1 --2- 172.21.15.40:0/2979902869 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fccb00a2890 0x7fccb00a4d50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:39.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.839+0000 7fccc6f81700 1 --2- 172.21.15.40:0/2979902869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccc0103ee0 0x7fccc0074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:39.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.839+0000 7fccc6f81700 1 --2- 172.21.15.40:0/2979902869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccc0102340 0x7fccc00760b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:39.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.839+0000 7fccc6f81700 1 --2- 172.21.15.40:0/2979902869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc0103540 0x7fccc0074700 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:39.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.839+0000 7fccc6f81700 1 -- 172.21.15.40:0/2979902869 >> 172.21.15.40:0/2979902869 conn(0x7fccc00fd920 msgr2=0x7fccc0106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:39.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.839+0000 7fccc6f81700 1 -- 172.21.15.40:0/2979902869 shutdown_connections 2024-03-18T22:42:39.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:39.839+0000 7fccc6f81700 1 -- 172.21.15.40:0/2979902869 wait complete. 2024-03-18T22:42:40.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:39 smithi040 ceph-mon[62643]: pgmap v349: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:40.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:39 smithi040 ceph-mon[62643]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:42:40.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:39 smithi113 ceph-mon[66775]: pgmap v349: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:40.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:39 smithi113 ceph-mon[66775]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:42:40.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:39 smithi103 ceph-mon[66377]: pgmap v349: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:40.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:39 smithi103 ceph-mon[66377]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:42:40.641 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-18T22:42:40.892 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:42:42.196 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:41 smithi040 ceph-mon[62643]: pgmap v350: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:42:42.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:41 smithi113 ceph-mon[66775]: pgmap v350: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:42:42.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:41 smithi103 ceph-mon[66377]: pgmap v350: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:42:42.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.583+0000 7f51fcf0e700 1 -- 172.21.15.40:0/1247037881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51f8103530 msgr2=0x7f51f8103990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:42.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.583+0000 7f51fcf0e700 1 --2- 172.21.15.40:0/1247037881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51f8103530 0x7f51f8103990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f51ec0099c0 tx=0x7f51ec009cd0 comp rx=0 tx=0).stop 2024-03-18T22:42:42.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.583+0000 7f51fcf0e700 1 -- 172.21.15.40:0/1247037881 shutdown_connections 2024-03-18T22:42:42.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.583+0000 7f51fcf0e700 1 --2- 172.21.15.40:0/1247037881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51f8103ed0 0x7f51f810a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:42.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.583+0000 7f51fcf0e700 1 --2- 172.21.15.40:0/1247037881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51f8102330 0x7f51f8102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:42.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.583+0000 7f51fcf0e700 1 --2- 172.21.15.40:0/1247037881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51f8103530 0x7f51f8103990 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:42.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.583+0000 7f51fcf0e700 1 -- 172.21.15.40:0/1247037881 >> 172.21.15.40:0/1247037881 conn(0x7f51f80fd8f0 msgr2=0x7f51f80ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:42.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.583+0000 7f51fcf0e700 1 -- 172.21.15.40:0/1247037881 shutdown_connections 2024-03-18T22:42:42.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.583+0000 7f51fcf0e700 1 -- 172.21.15.40:0/1247037881 wait complete. 2024-03-18T22:42:42.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51fcf0e700 1 Processor -- start 2024-03-18T22:42:42.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51fcf0e700 1 -- start start 2024-03-18T22:42:42.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51fcf0e700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51f8102330 0x7f51f80760b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:42.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51fcf0e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51f8103530 0x7f51f8074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:42.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51fcf0e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51f8103ed0 0x7f51f8074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:42.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51fcf0e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f51f810ddf0 con 0x7f51f8103530 2024-03-18T22:42:42.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51fcf0e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f51f810dc70 con 0x7f51f8103ed0 2024-03-18T22:42:42.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51fcf0e700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f51f810df70 con 0x7f51f8102330 2024-03-18T22:42:42.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51f5d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51f8103530 0x7f51f8074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:42.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51f5d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51f8103530 0x7f51f8074700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38390/0 (socket says 172.21.15.40:38390) 2024-03-18T22:42:42.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51f5d9b700 1 -- 172.21.15.40:0/1754895436 learned_addr learned my addr 172.21.15.40:0/1754895436 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:42.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51f659c700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51f8102330 0x7f51f80760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:42.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51f6d9d700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51f8103ed0 0x7f51f8074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:42.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51f5d9b700 1 -- 172.21.15.40:0/1754895436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51f8102330 msgr2=0x7f51f80760b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:42.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51f5d9b700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51f8102330 0x7f51f80760b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:42.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51f5d9b700 1 -- 172.21.15.40:0/1754895436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51f8103ed0 msgr2=0x7f51f8074c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:42.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51f5d9b700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51f8103ed0 0x7f51f8074c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:42.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51f5d9b700 1 -- 172.21.15.40:0/1754895436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f80754f0 con 0x7f51f8103530 2024-03-18T22:42:42.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51f659c700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51f8102330 0x7f51f80760b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:42.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.584+0000 7f51f6d9d700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51f8103ed0 0x7f51f8074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:42.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.585+0000 7f51f5d9b700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51f8103530 0x7f51f8074700 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f51ec009990 tx=0x7f51ec00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:42.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.585+0000 7f51e77fe700 1 -- 172.21.15.40:0/1754895436 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec009e60 con 0x7f51f8103530 2024-03-18T22:42:42.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.585+0000 7f51fcf0e700 1 -- 172.21.15.40:0/1754895436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f81a25c0 con 0x7f51f8103530 2024-03-18T22:42:42.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.585+0000 7f51e77fe700 1 -- 172.21.15.40:0/1754895436 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51ec003d70 con 0x7f51f8103530 2024-03-18T22:42:42.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.585+0000 7f51e77fe700 1 -- 172.21.15.40:0/1754895436 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0055b0 con 0x7f51f8103530 2024-03-18T22:42:42.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.585+0000 7f51fcf0e700 1 -- 172.21.15.40:0/1754895436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f81a2a10 con 0x7f51f8103530 2024-03-18T22:42:42.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.586+0000 7f51fcf0e700 1 -- 172.21.15.40:0/1754895436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51f8068fd0 con 0x7f51f8103530 2024-03-18T22:42:42.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.587+0000 7f51e77fe700 1 -- 172.21.15.40:0/1754895436 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f51ec00fc40 con 0x7f51f8103530 2024-03-18T22:42:42.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.587+0000 7f51e77fe700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51e00a2890 0x7f51e00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:42.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.588+0000 7f51f659c700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51e00a2890 0x7f51e00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:42.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.588+0000 7f51e77fe700 1 -- 172.21.15.40:0/1754895436 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f51ec0cfc00 con 0x7f51f8103530 2024-03-18T22:42:42.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.588+0000 7f51f659c700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51e00a2890 0x7f51e00a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f51f8103390 tx=0x7f51e8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:42.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.592+0000 7f51e77fe700 1 -- 172.21.15.40:0/1754895436 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f51ec013070 con 0x7f51f8103530 2024-03-18T22:42:42.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.803+0000 7f51fcf0e700 1 -- 172.21.15.40:0/1754895436 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f51f8062750 con 0x7f51e00a2890 2024-03-18T22:42:42.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.805+0000 7f51e77fe700 1 -- 172.21.15.40:0/1754895436 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f51f8062750 con 0x7f51e00a2890 2024-03-18T22:42:42.806 INFO:teuthology.orchestra.run.smithi040.stdout:HOST ADDR LABELS STATUS 2024-03-18T22:42:42.806 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040 172.21.15.40 2024-03-18T22:42:42.807 INFO:teuthology.orchestra.run.smithi040.stdout:smithi103 172.21.15.103 2024-03-18T22:42:42.807 INFO:teuthology.orchestra.run.smithi040.stdout:smithi113 172.21.15.113 2024-03-18T22:42:42.807 INFO:teuthology.orchestra.run.smithi040.stdout:3 hosts in cluster 2024-03-18T22:42:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.807+0000 7f51fcf0e700 1 -- 172.21.15.40:0/1754895436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51e00a2890 msgr2=0x7f51e00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.807+0000 7f51fcf0e700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51e00a2890 0x7f51e00a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f51f8103390 tx=0x7f51e8009260 comp rx=0 tx=0).stop 2024-03-18T22:42:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.807+0000 7f51fcf0e700 1 -- 172.21.15.40:0/1754895436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51f8103530 msgr2=0x7f51f8074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.807+0000 7f51fcf0e700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51f8103530 0x7f51f8074700 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f51ec009990 tx=0x7f51ec00f690 comp rx=0 tx=0).stop 2024-03-18T22:42:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.807+0000 7f51fcf0e700 1 -- 172.21.15.40:0/1754895436 shutdown_connections 2024-03-18T22:42:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.807+0000 7f51fcf0e700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51e00a2890 0x7f51e00a4d50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.807+0000 7f51fcf0e700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51f8103ed0 0x7f51f8074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.807+0000 7f51fcf0e700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51f8102330 0x7f51f80760b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.807+0000 7f51fcf0e700 1 --2- 172.21.15.40:0/1754895436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51f8103530 0x7f51f8074700 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:42.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.807+0000 7f51fcf0e700 1 -- 172.21.15.40:0/1754895436 >> 172.21.15.40:0/1754895436 conn(0x7f51f80fd8f0 msgr2=0x7f51f81068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:42.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.807+0000 7f51fcf0e700 1 -- 172.21.15.40:0/1754895436 shutdown_connections 2024-03-18T22:42:42.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:42.807+0000 7f51fcf0e700 1 -- 172.21.15.40:0/1754895436 wait complete. 2024-03-18T22:42:43.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:42 smithi040 ceph-mon[62643]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:42:43.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:42 smithi113 ceph-mon[66775]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:42:43.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:42 smithi103 ceph-mon[66377]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:42:43.563 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-18T22:42:43.816 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:42:44.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:43 smithi040 ceph-mon[62643]: pgmap v351: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:44.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:43 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-03-18T22:42:44.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:43 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11, 2, 6]}]: dispatch 2024-03-18T22:42:44.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:43 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-03-18T22:42:44.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:43 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-03-18T22:42:44.276 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:43 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]: dispatch 2024-03-18T22:42:44.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:43 smithi113 ceph-mon[66775]: pgmap v351: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:44.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:43 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-03-18T22:42:44.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:43 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11, 2, 6]}]: dispatch 2024-03-18T22:42:44.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:43 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-03-18T22:42:44.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:43 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-03-18T22:42:44.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:43 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]: dispatch 2024-03-18T22:42:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:43 smithi103 ceph-mon[66377]: pgmap v351: 129 pgs: 129 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:42:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-03-18T22:42:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11, 2, 6]}]: dispatch 2024-03-18T22:42:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-03-18T22:42:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]: dispatch 2024-03-18T22:42:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:43 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]: dispatch 2024-03-18T22:42:45.285 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-03-18T22:42:45.285 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11, 2, 6]}]': finished 2024-03-18T22:42:45.285 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-03-18T22:42:45.285 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-03-18T22:42:45.285 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]': finished 2024-03-18T22:42:45.285 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:44 smithi040 ceph-mon[62643]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T22:42:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-03-18T22:42:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11, 2, 6]}]': finished 2024-03-18T22:42:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-03-18T22:42:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-03-18T22:42:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]': finished 2024-03-18T22:42:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:44 smithi113 ceph-mon[66775]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T22:42:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-03-18T22:42:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11, 2, 6]}]': finished 2024-03-18T22:42:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-03-18T22:42:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 1]}]': finished 2024-03-18T22:42:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]': finished 2024-03-18T22:42:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:44 smithi103 ceph-mon[66377]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T22:42:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.530+0000 7f7f6de19700 1 -- 172.21.15.40:0/1485432362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f68103ee0 msgr2=0x7f7f6810a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:45.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.530+0000 7f7f6de19700 1 --2- 172.21.15.40:0/1485432362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f68103ee0 0x7f7f6810a790 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7f6000b0d0 tx=0x7f7f6000b3e0 comp rx=0 tx=0).stop 2024-03-18T22:42:45.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.531+0000 7f7f6de19700 1 -- 172.21.15.40:0/1485432362 shutdown_connections 2024-03-18T22:42:45.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.531+0000 7f7f6de19700 1 --2- 172.21.15.40:0/1485432362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f68103ee0 0x7f7f6810a790 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:45.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.531+0000 7f7f6de19700 1 --2- 172.21.15.40:0/1485432362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68102340 0x7f7f68102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:45.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.531+0000 7f7f6de19700 1 --2- 172.21.15.40:0/1485432362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f68103540 0x7f7f681039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:45.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.531+0000 7f7f6de19700 1 -- 172.21.15.40:0/1485432362 >> 172.21.15.40:0/1485432362 conn(0x7f7f680fd920 msgr2=0x7f7f680ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:45.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.531+0000 7f7f6de19700 1 -- 172.21.15.40:0/1485432362 shutdown_connections 2024-03-18T22:42:45.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.531+0000 7f7f6de19700 1 -- 172.21.15.40:0/1485432362 wait complete. 2024-03-18T22:42:45.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.531+0000 7f7f6de19700 1 Processor -- start 2024-03-18T22:42:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.531+0000 7f7f6de19700 1 -- start start 2024-03-18T22:42:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f6de19700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68102340 0x7f7f68198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f6de19700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f68103540 0x7f7f68198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f6de19700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f68103ee0 0x7f7f6819f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f6de19700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6810ddc0 con 0x7f7f68103ee0 2024-03-18T22:42:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f6de19700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6810dc40 con 0x7f7f68102340 2024-03-18T22:42:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f6de19700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6810df40 con 0x7f7f68103540 2024-03-18T22:42:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f67fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f68103ee0 0x7f7f6819f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f66ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f68103540 0x7f7f68198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f66ffd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f68103540 0x7f7f68198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:37982/0 (socket says 172.21.15.40:37982) 2024-03-18T22:42:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f67fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f68103ee0 0x7f7f6819f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38412/0 (socket says 172.21.15.40:38412) 2024-03-18T22:42:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f66ffd700 1 -- 172.21.15.40:0/2574192898 learned_addr learned my addr 172.21.15.40:0/2574192898 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f677fe700 1 --2- 172.21.15.40:0/2574192898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68102340 0x7f7f68198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f66ffd700 1 -- 172.21.15.40:0/2574192898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68102340 msgr2=0x7f7f68198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f66ffd700 1 --2- 172.21.15.40:0/2574192898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68102340 0x7f7f68198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f66ffd700 1 -- 172.21.15.40:0/2574192898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f68103ee0 msgr2=0x7f7f6819f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f66ffd700 1 --2- 172.21.15.40:0/2574192898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f68103ee0 0x7f7f6819f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f66ffd700 1 -- 172.21.15.40:0/2574192898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f681a0060 con 0x7f7f68103540 2024-03-18T22:42:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f66ffd700 1 --2- 172.21.15.40:0/2574192898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f68103540 0x7f7f68198940 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00e9d0 tx=0x7f7f5c00ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f64ff9700 1 -- 172.21.15.40:0/2574192898 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c00cbd0 con 0x7f7f68103540 2024-03-18T22:42:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f6de19700 1 -- 172.21.15.40:0/2574192898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f681a0350 con 0x7f7f68103540 2024-03-18T22:42:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f64ff9700 1 -- 172.21.15.40:0/2574192898 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7f5c00ee70 con 0x7f7f68103540 2024-03-18T22:42:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.532+0000 7f7f64ff9700 1 -- 172.21.15.40:0/2574192898 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c018680 con 0x7f7f68103540 2024-03-18T22:42:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.533+0000 7f7f6de19700 1 -- 172.21.15.40:0/2574192898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f681a08e0 con 0x7f7f68103540 2024-03-18T22:42:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.533+0000 7f7f6de19700 1 -- 172.21.15.40:0/2574192898 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f68068fd0 con 0x7f7f68103540 2024-03-18T22:42:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.535+0000 7f7f64ff9700 1 -- 172.21.15.40:0/2574192898 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7f5c047a60 con 0x7f7f68103540 2024-03-18T22:42:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.535+0000 7f7f64ff9700 1 --2- 172.21.15.40:0/2574192898 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7f500a2890 0x7f7f500a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.535+0000 7f7f677fe700 1 --2- 172.21.15.40:0/2574192898 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7f500a2890 0x7f7f500a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.535+0000 7f7f64ff9700 1 -- 172.21.15.40:0/2574192898 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7f5c0cf140 con 0x7f7f68103540 2024-03-18T22:42:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.536+0000 7f7f677fe700 1 --2- 172.21.15.40:0/2574192898 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7f500a2890 0x7f7f500a4d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7f681033a0 tx=0x7f7f58009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:45.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.538+0000 7f7f64ff9700 1 -- 172.21.15.40:0/2574192898 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7f5c00cd70 con 0x7f7f68103540 2024-03-18T22:42:45.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.750+0000 7f7f6de19700 1 -- 172.21.15.40:0/2574192898 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f7f68062750 con 0x7f7f500a2890 2024-03-18T22:42:45.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.752+0000 7f7f64ff9700 1 -- 172.21.15.40:0/2574192898 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f7f68062750 con 0x7f7f500a2890 2024-03-18T22:42:45.754 INFO:teuthology.orchestra.run.smithi040.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-18T22:42:45.754 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009J400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-18T22:42:45.754 INFO:teuthology.orchestra.run.smithi040.stdout:smithi103 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204012Z400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-18T22:42:45.754 INFO:teuthology.orchestra.run.smithi040.stdout:smithi113 /dev/nvme0n1 ssd INTEL_SSDPEDMX020T4_CVPD5171000M2P0SGN 1863G No 2m ago Has a FileSystem, LVM detected 2024-03-18T22:42:45.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.754+0000 7f7f6de19700 1 -- 172.21.15.40:0/2574192898 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7f500a2890 msgr2=0x7f7f500a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:45.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.754+0000 7f7f6de19700 1 --2- 172.21.15.40:0/2574192898 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7f500a2890 0x7f7f500a4d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7f681033a0 tx=0x7f7f58009260 comp rx=0 tx=0).stop 2024-03-18T22:42:45.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.754+0000 7f7f6de19700 1 -- 172.21.15.40:0/2574192898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f68103540 msgr2=0x7f7f68198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:45.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.754+0000 7f7f6de19700 1 --2- 172.21.15.40:0/2574192898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f68103540 0x7f7f68198940 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00e9d0 tx=0x7f7f5c00ece0 comp rx=0 tx=0).stop 2024-03-18T22:42:45.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.754+0000 7f7f6de19700 1 -- 172.21.15.40:0/2574192898 shutdown_connections 2024-03-18T22:42:45.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.754+0000 7f7f6de19700 1 --2- 172.21.15.40:0/2574192898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f68103ee0 0x7f7f6819f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:45.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.754+0000 7f7f6de19700 1 --2- 172.21.15.40:0/2574192898 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7f500a2890 0x7f7f500a4d50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:45.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.754+0000 7f7f6de19700 1 --2- 172.21.15.40:0/2574192898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f68102340 0x7f7f68198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:45.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.754+0000 7f7f6de19700 1 --2- 172.21.15.40:0/2574192898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7f68103540 0x7f7f68198940 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:45.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.754+0000 7f7f6de19700 1 -- 172.21.15.40:0/2574192898 >> 172.21.15.40:0/2574192898 conn(0x7f7f680fd920 msgr2=0x7f7f68106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:45.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.754+0000 7f7f6de19700 1 -- 172.21.15.40:0/2574192898 shutdown_connections 2024-03-18T22:42:45.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:45.754+0000 7f7f6de19700 1 -- 172.21.15.40:0/2574192898 wait complete. 2024-03-18T22:42:46.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:45 smithi040 ceph-mon[62643]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4 op/s 2024-03-18T22:42:46.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:45 smithi040 ceph-mon[62643]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-18T22:42:46.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:45 smithi040 ceph-mon[62643]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T22:42:46.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:45 smithi040 ceph-mon[62643]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:42:46.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:45 smithi113 ceph-mon[66775]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4 op/s 2024-03-18T22:42:46.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:45 smithi113 ceph-mon[66775]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-18T22:42:46.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:45 smithi113 ceph-mon[66775]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T22:42:46.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:45 smithi113 ceph-mon[66775]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:42:46.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:45 smithi103 ceph-mon[66377]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4 op/s 2024-03-18T22:42:46.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:45 smithi103 ceph-mon[66377]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-18T22:42:46.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:45 smithi103 ceph-mon[66377]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T22:42:46.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:45 smithi103 ceph-mon[66377]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T22:42:46.501 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T22:42:46.512 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2024-03-18T22:42:46.512 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-18T22:42:46.758 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:42:48.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:47 smithi040 ceph-mon[62643]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3 op/s; 13 B/s, 1 objects/s recovering 2024-03-18T22:42:48.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:47 smithi113 ceph-mon[66775]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3 op/s; 13 B/s, 1 objects/s recovering 2024-03-18T22:42:48.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:47 smithi103 ceph-mon[66377]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3 op/s; 13 B/s, 1 objects/s recovering 2024-03-18T22:42:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.416+0000 7f0201345700 1 -- 172.21.15.40:0/2545684174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc102360 msgr2=0x7f01fc102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.416+0000 7f0201345700 1 --2- 172.21.15.40:0/2545684174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc102360 0x7f01fc102780 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f01ec00b0d0 tx=0x7f01ec00b3e0 comp rx=0 tx=0).stop 2024-03-18T22:42:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.416+0000 7f0201345700 1 -- 172.21.15.40:0/2545684174 shutdown_connections 2024-03-18T22:42:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.416+0000 7f0201345700 1 --2- 172.21.15.40:0/2545684174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01fc103f00 0x7f01fc10a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.416+0000 7f0201345700 1 --2- 172.21.15.40:0/2545684174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc102360 0x7f01fc102780 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:48.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.416+0000 7f0201345700 1 --2- 172.21.15.40:0/2545684174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc103560 0x7f01fc1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.416+0000 7f0201345700 1 -- 172.21.15.40:0/2545684174 >> 172.21.15.40:0/2545684174 conn(0x7f01fc0fd8c0 msgr2=0x7f01fc0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.416+0000 7f0201345700 1 -- 172.21.15.40:0/2545684174 shutdown_connections 2024-03-18T22:42:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.416+0000 7f0201345700 1 -- 172.21.15.40:0/2545684174 wait complete. 2024-03-18T22:42:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f0201345700 1 Processor -- start 2024-03-18T22:42:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f0201345700 1 -- start start 2024-03-18T22:42:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f0201345700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc102360 0x7f01fc1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f0201345700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01fc103560 0x7f01fc1989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f0201345700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc103f00 0x7f01fc19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f0201345700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc10ddd0 con 0x7f01fc102360 2024-03-18T22:42:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f0201345700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc10dc50 con 0x7f01fc103f00 2024-03-18T22:42:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f0201345700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc10df50 con 0x7f01fc103560 2024-03-18T22:42:48.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f01faffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc102360 0x7f01fc1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f01faffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc102360 0x7f01fc1984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38422/0 (socket says 172.21.15.40:38422) 2024-03-18T22:42:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f01faffd700 1 -- 172.21.15.40:0/3033066991 learned_addr learned my addr 172.21.15.40:0/3033066991 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f01fa7fc700 1 --2- 172.21.15.40:0/3033066991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01fc103560 0x7f01fc1989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f01fb7fe700 1 --2- 172.21.15.40:0/3033066991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc103f00 0x7f01fc19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f01faffd700 1 -- 172.21.15.40:0/3033066991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01fc103560 msgr2=0x7f01fc1989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f01faffd700 1 --2- 172.21.15.40:0/3033066991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01fc103560 0x7f01fc1989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f01faffd700 1 -- 172.21.15.40:0/3033066991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc103f00 msgr2=0x7f01fc19fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f01faffd700 1 --2- 172.21.15.40:0/3033066991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc103f00 0x7f01fc19fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f01faffd700 1 -- 172.21.15.40:0/3033066991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01fc1a0100 con 0x7f01fc102360 2024-03-18T22:42:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.417+0000 7f01fa7fc700 1 --2- 172.21.15.40:0/3033066991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01fc103560 0x7f01fc1989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:42:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.418+0000 7f01faffd700 1 --2- 172.21.15.40:0/3033066991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc102360 0x7f01fc1984a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f01ec00b770 tx=0x7f01ec004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.418+0000 7f01f3fff700 1 -- 172.21.15.40:0/3033066991 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec01f070 con 0x7f01fc102360 2024-03-18T22:42:48.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.418+0000 7f01f3fff700 1 -- 172.21.15.40:0/3033066991 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01ec007b80 con 0x7f01fc102360 2024-03-18T22:42:48.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.418+0000 7f0201345700 1 -- 172.21.15.40:0/3033066991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01fc1a0390 con 0x7f01fc102360 2024-03-18T22:42:48.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.418+0000 7f01f3fff700 1 -- 172.21.15.40:0/3033066991 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec01b690 con 0x7f01fc102360 2024-03-18T22:42:48.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.418+0000 7f0201345700 1 -- 172.21.15.40:0/3033066991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01fc1a07e0 con 0x7f01fc102360 2024-03-18T22:42:48.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.419+0000 7f0201345700 1 -- 172.21.15.40:0/3033066991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01fc068fd0 con 0x7f01fc102360 2024-03-18T22:42:48.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.420+0000 7f01f3fff700 1 -- 172.21.15.40:0/3033066991 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f01ec022070 con 0x7f01fc102360 2024-03-18T22:42:48.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.421+0000 7f01f3fff700 1 --2- 172.21.15.40:0/3033066991 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f01e80a2890 0x7f01e80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:48.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.421+0000 7f01fa7fc700 1 --2- 172.21.15.40:0/3033066991 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f01e80a2890 0x7f01e80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:48.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.421+0000 7f01f3fff700 1 -- 172.21.15.40:0/3033066991 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f01ec00f020 con 0x7f01fc102360 2024-03-18T22:42:48.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.421+0000 7f01fa7fc700 1 --2- 172.21.15.40:0/3033066991 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f01e80a2890 0x7f01e80a4d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009cb0 tx=0x7f01e4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:48.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.424+0000 7f01f3fff700 1 -- 172.21.15.40:0/3033066991 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f01ec029020 con 0x7f01fc102360 2024-03-18T22:42:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.674+0000 7f0201345700 1 -- 172.21.15.40:0/3033066991 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f01fc02ddc0 con 0x7f01fc102360 2024-03-18T22:42:48.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.675+0000 7f01f3fff700 1 -- 172.21.15.40:0/3033066991 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) ==== 76+0+3102 (secure 0 0 0) 0x7f01ec09d1d0 con 0x7f01fc102360 2024-03-18T22:42:48.677 INFO:teuthology.orchestra.run.smithi040.stdout:e27 2024-03-18T22:42:48.677 INFO:teuthology.orchestra.run.smithi040.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T22:42:48.677 INFO:teuthology.orchestra.run.smithi040.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T22:42:48.677 INFO:teuthology.orchestra.run.smithi040.stdout:legacy client fscid: 1 2024-03-18T22:42:48.677 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:48.677 INFO:teuthology.orchestra.run.smithi040.stdout:Filesystem 'cephfs' (1) 2024-03-18T22:42:48.677 INFO:teuthology.orchestra.run.smithi040.stdout:fs_name cephfs 2024-03-18T22:42:48.677 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 27 2024-03-18T22:42:48.677 INFO:teuthology.orchestra.run.smithi040.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-18T22:42:48.677 INFO:teuthology.orchestra.run.smithi040.stdout:created 2024-03-18T22:40:41.572773+0000 2024-03-18T22:42:48.677 INFO:teuthology.orchestra.run.smithi040.stdout:modified 2024-03-18T22:41:09.839105+0000 2024-03-18T22:42:48.678 INFO:teuthology.orchestra.run.smithi040.stdout:tableserver 0 2024-03-18T22:42:48.678 INFO:teuthology.orchestra.run.smithi040.stdout:root 0 2024-03-18T22:42:48.678 INFO:teuthology.orchestra.run.smithi040.stdout:session_timeout 300 2024-03-18T22:42:48.678 INFO:teuthology.orchestra.run.smithi040.stdout:session_autoclose 300 2024-03-18T22:42:48.678 INFO:teuthology.orchestra.run.smithi040.stdout:max_file_size 1099511627776 2024-03-18T22:42:48.678 INFO:teuthology.orchestra.run.smithi040.stdout:max_xattr_size 65536 2024-03-18T22:42:48.678 INFO:teuthology.orchestra.run.smithi040.stdout:required_client_features {} 2024-03-18T22:42:48.678 INFO:teuthology.orchestra.run.smithi040.stdout:last_failure 0 2024-03-18T22:42:48.678 INFO:teuthology.orchestra.run.smithi040.stdout:last_failure_osd_epoch 0 2024-03-18T22:42:48.678 INFO:teuthology.orchestra.run.smithi040.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T22:42:48.678 INFO:teuthology.orchestra.run.smithi040.stdout:max_mds 5 2024-03-18T22:42:48.678 INFO:teuthology.orchestra.run.smithi040.stdout:in 0,1,2,3,4 2024-03-18T22:42:48.678 INFO:teuthology.orchestra.run.smithi040.stdout:up {0=24509,1=24481,2=24473,3=24401,4=24388} 2024-03-18T22:42:48.679 INFO:teuthology.orchestra.run.smithi040.stdout:failed 2024-03-18T22:42:48.679 INFO:teuthology.orchestra.run.smithi040.stdout:damaged 2024-03-18T22:42:48.679 INFO:teuthology.orchestra.run.smithi040.stdout:stopped 2024-03-18T22:42:48.679 INFO:teuthology.orchestra.run.smithi040.stdout:data_pools [3,4] 2024-03-18T22:42:48.679 INFO:teuthology.orchestra.run.smithi040.stdout:metadata_pool 2 2024-03-18T22:42:48.679 INFO:teuthology.orchestra.run.smithi040.stdout:inline_data disabled 2024-03-18T22:42:48.679 INFO:teuthology.orchestra.run.smithi040.stdout:balancer 2024-03-18T22:42:48.679 INFO:teuthology.orchestra.run.smithi040.stdout:bal_rank_mask -1 2024-03-18T22:42:48.679 INFO:teuthology.orchestra.run.smithi040.stdout:standby_count_wanted 0 2024-03-18T22:42:48.679 INFO:teuthology.orchestra.run.smithi040.stdout:qdb_cluster leader: 24509 members: 24481,24509,24473,24401,24388 2024-03-18T22:42:48.679 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.b{0:24509} state up:active seq 3 addr [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T22:42:48.679 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.c{0:24457} state up:standby-replay seq 1 addr [v2:172.21.15.113:6836/4212429112,v1:172.21.15.113:6837/4212429112] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T22:42:48.679 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.i{1:24481} state up:active seq 11 addr [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T22:42:48.679 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.f{1:24445} state up:standby-replay seq 1 addr [v2:172.21.15.113:6834/4040610039,v1:172.21.15.113:6835/4040610039] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T22:42:48.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.h{2:24473} state up:active seq 12 addr [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T22:42:48.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.e{2:24437} state up:standby-replay seq 1 addr [v2:172.21.15.103:6834/2536694691,v1:172.21.15.103:6835/2536694691] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T22:42:48.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.k{3:24401} state up:active seq 16 addr [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T22:42:48.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.a{3:14604} state up:standby-replay seq 1 addr [v2:172.21.15.40:6840/1807327423,v1:172.21.15.40:6841/1807327423] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T22:42:48.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.l{4:24388} state up:active seq 17 addr [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T22:42:48.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.g{4:14592} state up:standby-replay seq 1 addr [v2:172.21.15.40:6838/4264194754,v1:172.21.15.40:6839/4264194754] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T22:42:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:48.680 INFO:teuthology.orchestra.run.smithi040.stdout:Standby daemons: 2024-03-18T22:42:48.680 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:48.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.j{-1:14517} state up:standby seq 1 addr [v2:172.21.15.40:6834/823346277,v1:172.21.15.40:6835/823346277] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T22:42:48.680 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.d{-1:14529} state up:standby seq 1 addr [v2:172.21.15.40:6836/3562391432,v1:172.21.15.40:6837/3562391432] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T22:42:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.678+0000 7f0201345700 1 -- 172.21.15.40:0/3033066991 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f01e80a2890 msgr2=0x7f01e80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.678+0000 7f0201345700 1 --2- 172.21.15.40:0/3033066991 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f01e80a2890 0x7f01e80a4d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009cb0 tx=0x7f01e4009260 comp rx=0 tx=0).stop 2024-03-18T22:42:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.678+0000 7f0201345700 1 -- 172.21.15.40:0/3033066991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc102360 msgr2=0x7f01fc1984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.678+0000 7f0201345700 1 --2- 172.21.15.40:0/3033066991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc102360 0x7f01fc1984a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f01ec00b770 tx=0x7f01ec004d90 comp rx=0 tx=0).stop 2024-03-18T22:42:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.678+0000 7f0201345700 1 -- 172.21.15.40:0/3033066991 shutdown_connections 2024-03-18T22:42:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.678+0000 7f0201345700 1 --2- 172.21.15.40:0/3033066991 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f01e80a2890 0x7f01e80a4d50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.678+0000 7f0201345700 1 --2- 172.21.15.40:0/3033066991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc103f00 0x7f01fc19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.678+0000 7f0201345700 1 --2- 172.21.15.40:0/3033066991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01fc102360 0x7f01fc1984a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.678+0000 7f0201345700 1 --2- 172.21.15.40:0/3033066991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01fc103560 0x7f01fc1989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.678+0000 7f0201345700 1 -- 172.21.15.40:0/3033066991 >> 172.21.15.40:0/3033066991 conn(0x7f01fc0fd8c0 msgr2=0x7f01fc1068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:48.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.678+0000 7f0201345700 1 -- 172.21.15.40:0/3033066991 shutdown_connections 2024-03-18T22:42:48.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:48.678+0000 7f0201345700 1 -- 172.21.15.40:0/3033066991 wait complete. 2024-03-18T22:42:48.682 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 27 2024-03-18T22:42:49.310 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:48 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3033066991' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T22:42:49.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:48 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3033066991' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T22:42:49.339 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-18T22:42:49.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:48 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3033066991' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T22:42:49.600 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:42:50.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:49 smithi113 ceph-mon[66775]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2 op/s; 13 B/s, 1 objects/s recovering 2024-03-18T22:42:50.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:49 smithi103 ceph-mon[66377]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2 op/s; 13 B/s, 1 objects/s recovering 2024-03-18T22:42:50.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:49 smithi040 ceph-mon[62643]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2 op/s; 13 B/s, 1 objects/s recovering 2024-03-18T22:42:51.277 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:51 smithi040 ceph-mon[62643]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-18T22:42:51.278 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:51 smithi040 ceph-mon[62643]: Cluster is now healthy 2024-03-18T22:42:51.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.275+0000 7f1124089700 1 -- 172.21.15.40:0/805581450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f111c102430 msgr2=0x7f111c102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.275+0000 7f1124089700 1 --2- 172.21.15.40:0/805581450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f111c102430 0x7f111c102890 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f110c0099c0 tx=0x7f110c009cd0 comp rx=0 tx=0).stop 2024-03-18T22:42:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.275+0000 7f1124089700 1 -- 172.21.15.40:0/805581450 shutdown_connections 2024-03-18T22:42:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.275+0000 7f1124089700 1 --2- 172.21.15.40:0/805581450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f111c102dd0 0x7f111c109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.275+0000 7f1124089700 1 --2- 172.21.15.40:0/805581450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f111c101230 0x7f111c101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.275+0000 7f1124089700 1 --2- 172.21.15.40:0/805581450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f111c102430 0x7f111c102890 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.275+0000 7f1124089700 1 -- 172.21.15.40:0/805581450 >> 172.21.15.40:0/805581450 conn(0x7f111c0fc7d0 msgr2=0x7f111c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.275+0000 7f1124089700 1 -- 172.21.15.40:0/805581450 shutdown_connections 2024-03-18T22:42:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.275+0000 7f1124089700 1 -- 172.21.15.40:0/805581450 wait complete. 2024-03-18T22:42:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.276+0000 7f1124089700 1 Processor -- start 2024-03-18T22:42:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.276+0000 7f1124089700 1 -- start start 2024-03-18T22:42:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.276+0000 7f1124089700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f111c101230 0x7f111c06b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.276+0000 7f1124089700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f111c102430 0x7f111c19d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.276+0000 7f1124089700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f111c102dd0 0x7f111c19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:51.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.276+0000 7f1124089700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f111c10ccb0 con 0x7f111c101230 2024-03-18T22:42:51.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.276+0000 7f1124089700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f111c10cb30 con 0x7f111c102430 2024-03-18T22:42:51.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.276+0000 7f1124089700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f111c10ce30 con 0x7f111c102dd0 2024-03-18T22:42:51.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.276+0000 7f1121e25700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f111c101230 0x7f111c06b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:51.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.276+0000 7f1121e25700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f111c101230 0x7f111c06b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39226/0 (socket says 172.21.15.40:39226) 2024-03-18T22:42:51.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.276+0000 7f1121e25700 1 -- 172.21.15.40:0/1224884747 learned_addr learned my addr 172.21.15.40:0/1224884747 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:51.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.276+0000 7f1121624700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f111c102430 0x7f111c19d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:51.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.276+0000 7f1122626700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f111c102dd0 0x7f111c19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:51.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.277+0000 7f1121e25700 1 -- 172.21.15.40:0/1224884747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f111c102dd0 msgr2=0x7f111c19f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:51.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.277+0000 7f1121e25700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f111c102dd0 0x7f111c19f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:51.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.277+0000 7f1121e25700 1 -- 172.21.15.40:0/1224884747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f111c102430 msgr2=0x7f111c19d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:51.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.277+0000 7f1121e25700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f111c102430 0x7f111c19d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:51.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.277+0000 7f1121e25700 1 -- 172.21.15.40:0/1224884747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111c197010 con 0x7f111c101230 2024-03-18T22:42:51.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.277+0000 7f1122626700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f111c102dd0 0x7f111c19f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.277+0000 7f1121624700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f111c102430 0x7f111c19d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.277+0000 7f1121e25700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f111c101230 0x7f111c06b3b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f111800e970 tx=0x7f111800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.277+0000 7f1112ffd700 1 -- 172.21.15.40:0/1224884747 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1118004220 con 0x7f111c101230 2024-03-18T22:42:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.277+0000 7f1124089700 1 -- 172.21.15.40:0/1224884747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f111c197300 con 0x7f111c101230 2024-03-18T22:42:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.277+0000 7f1112ffd700 1 -- 172.21.15.40:0/1224884747 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f11180043c0 con 0x7f111c101230 2024-03-18T22:42:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.277+0000 7f1112ffd700 1 -- 172.21.15.40:0/1224884747 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1118003d50 con 0x7f111c101230 2024-03-18T22:42:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.277+0000 7f1124089700 1 -- 172.21.15.40:0/1224884747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f111c197860 con 0x7f111c101230 2024-03-18T22:42:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.278+0000 7f1124089700 1 -- 172.21.15.40:0/1224884747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f111c068fd0 con 0x7f111c101230 2024-03-18T22:42:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.279+0000 7f1112ffd700 1 -- 172.21.15.40:0/1224884747 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f111800c7e0 con 0x7f111c101230 2024-03-18T22:42:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.279+0000 7f1112ffd700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f11080a2890 0x7f11080a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.280+0000 7f1121624700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f11080a2890 0x7f11080a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.280+0000 7f1112ffd700 1 -- 172.21.15.40:0/1224884747 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f11180cf0e0 con 0x7f111c101230 2024-03-18T22:42:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.280+0000 7f1121624700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f11080a2890 0x7f11080a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f110c000c00 tx=0x7f110c005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:51.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.284+0000 7f1112ffd700 1 -- 172.21.15.40:0/1224884747 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f111809bd30 con 0x7f111c101230 2024-03-18T22:42:51.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:51 smithi113 ceph-mon[66775]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-18T22:42:51.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:51 smithi113 ceph-mon[66775]: Cluster is now healthy 2024-03-18T22:42:51.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:51 smithi103 ceph-mon[66377]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-18T22:42:51.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:51 smithi103 ceph-mon[66377]: Cluster is now healthy 2024-03-18T22:42:51.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.491+0000 7f1124089700 1 -- 172.21.15.40:0/1224884747 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f111c108240 con 0x7f111c101230 2024-03-18T22:42:51.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.492+0000 7f1112ffd700 1 -- 172.21.15.40:0/1224884747 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5736 (secure 0 0 0) 0x7f111809b630 con 0x7f111c101230 2024-03-18T22:42:51.494 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 93 2024-03-18T22:42:51.494 INFO:teuthology.orchestra.run.smithi040.stdout:fsid 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T22:42:51.494 INFO:teuthology.orchestra.run.smithi040.stdout:created 2024-03-18T22:32:48.032793+0000 2024-03-18T22:42:51.494 INFO:teuthology.orchestra.run.smithi040.stdout:modified 2024-03-18T22:42:44.941873+0000 2024-03-18T22:42:51.494 INFO:teuthology.orchestra.run.smithi040.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-18T22:42:51.494 INFO:teuthology.orchestra.run.smithi040.stdout:crush_version 27 2024-03-18T22:42:51.494 INFO:teuthology.orchestra.run.smithi040.stdout:full_ratio 0.95 2024-03-18T22:42:51.494 INFO:teuthology.orchestra.run.smithi040.stdout:backfillfull_ratio 0.9 2024-03-18T22:42:51.494 INFO:teuthology.orchestra.run.smithi040.stdout:nearfull_ratio 0.85 2024-03-18T22:42:51.494 INFO:teuthology.orchestra.run.smithi040.stdout:require_min_compat_client luminous 2024-03-18T22:42:51.494 INFO:teuthology.orchestra.run.smithi040.stdout:min_compat_client luminous 2024-03-18T22:42:51.494 INFO:teuthology.orchestra.run.smithi040.stdout:require_osd_release squid 2024-03-18T22:42:51.495 INFO:teuthology.orchestra.run.smithi040.stdout:stretch_mode_enabled false 2024-03-18T22:42:51.495 INFO:teuthology.orchestra.run.smithi040.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-18T22:42:51.495 INFO:teuthology.orchestra.run.smithi040.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-18T22:42:51.495 INFO:teuthology.orchestra.run.smithi040.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-18T22:42:51.495 INFO:teuthology.orchestra.run.smithi040.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-18T22:42:51.495 INFO:teuthology.orchestra.run.smithi040.stdout:max_osd 12 2024-03-18T22:42:51.495 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6802/1816003248,v1:172.21.15.40:6803/1816003248] [v2:172.21.15.40:6804/1816003248,v1:172.21.15.40:6805/1816003248] exists,up 0f712e31-408e-427e-8826-2afdfc70bf80 2024-03-18T22:42:51.495 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6800/2851705755,v1:172.21.15.103:6801/2851705755] [v2:172.21.15.103:6802/2851705755,v1:172.21.15.103:6803/2851705755] exists,up 5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9 2024-03-18T22:42:51.495 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6800/954444822,v1:172.21.15.113:6801/954444822] [v2:172.21.15.113:6802/954444822,v1:172.21.15.113:6803/954444822] exists,up 69ba7362-43de-46f4-a5fa-b8492210fa27 2024-03-18T22:42:51.495 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6810/3224857891,v1:172.21.15.40:6811/3224857891] [v2:172.21.15.40:6812/3224857891,v1:172.21.15.40:6813/3224857891] exists,up 627e5c6b-419e-4c32-9207-0447a88e18d4 2024-03-18T22:42:51.495 INFO:teuthology.orchestra.run.smithi040.stdout:osd.4 up in weight 1 up_from 33 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6808/3777182460,v1:172.21.15.103:6809/3777182460] [v2:172.21.15.103:6810/3777182460,v1:172.21.15.103:6811/3777182460] exists,up 34f3434e-5092-4639-9c2c-f64a196ea4b0 2024-03-18T22:42:51.495 INFO:teuthology.orchestra.run.smithi040.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6808/3666840324,v1:172.21.15.113:6809/3666840324] [v2:172.21.15.113:6810/3666840324,v1:172.21.15.113:6811/3666840324] exists,up f9617fbc-5b2c-4e57-a926-b1b3a0953e58 2024-03-18T22:42:51.495 INFO:teuthology.orchestra.run.smithi040.stdout:osd.6 up in weight 1 up_from 46 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6818/1125393763,v1:172.21.15.40:6819/1125393763] [v2:172.21.15.40:6820/1125393763,v1:172.21.15.40:6821/1125393763] exists,up 42df7e48-779a-4aee-bf6b-8c9fc8627d99 2024-03-18T22:42:51.495 INFO:teuthology.orchestra.run.smithi040.stdout:osd.7 up in weight 1 up_from 51 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6816/542253261,v1:172.21.15.103:6817/542253261] [v2:172.21.15.103:6818/542253261,v1:172.21.15.103:6819/542253261] exists,up 1f3ce8a7-42f4-4582-9343-e34951f6b3a4 2024-03-18T22:42:51.496 INFO:teuthology.orchestra.run.smithi040.stdout:osd.8 up in weight 1 up_from 59 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6816/2089621708,v1:172.21.15.113:6817/2089621708] [v2:172.21.15.113:6818/2089621708,v1:172.21.15.113:6819/2089621708] exists,up 66b85f48-4e20-4563-b495-b50e8fe79c74 2024-03-18T22:42:51.496 INFO:teuthology.orchestra.run.smithi040.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.40:6826/4067616401,v1:172.21.15.40:6827/4067616401] [v2:172.21.15.40:6828/4067616401,v1:172.21.15.40:6829/4067616401] exists,up e46e9424-c131-4559-9a43-55c7cf9c8eeb 2024-03-18T22:42:51.496 INFO:teuthology.orchestra.run.smithi040.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6824/1414934819,v1:172.21.15.103:6825/1414934819] [v2:172.21.15.103:6826/1414934819,v1:172.21.15.103:6827/1414934819] exists,up df88c341-1ebe-45e5-aa2e-f8a0696c3770 2024-03-18T22:42:51.496 INFO:teuthology.orchestra.run.smithi040.stdout:osd.11 up in weight 1 up_from 80 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.113:6824/287160143,v1:172.21.15.113:6825/287160143] [v2:172.21.15.113:6826/287160143,v1:172.21.15.113:6827/287160143] exists,up 97b51a30-f636-4149-a06a-f536f1671467 2024-03-18T22:42:51.496 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 2.0 [8,4] 2024-03-18T22:42:51.496 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 2.b [8,11,2,6] 2024-03-18T22:42:51.496 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 2.e [2,4] 2024-03-18T22:42:51.496 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 2.10 [2,1] 2024-03-18T22:42:51.496 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 2.1b [5,11] 2024-03-18T22:42:51.496 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.5 [3,9] 2024-03-18T22:42:51.496 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.1a [5,6] 2024-03-18T22:42:51.496 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.1d [4,8] 2024-03-18T22:42:51.496 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.1e [3,9] 2024-03-18T22:42:51.496 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.24 [4,9] 2024-03-18T22:42:51.497 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 3.3a [4,7] 2024-03-18T22:42:51.497 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 4.f [9,8] 2024-03-18T22:42:51.497 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 4.12 [9,10] 2024-03-18T22:42:51.497 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 4.1c [9,8] 2024-03-18T22:42:51.497 INFO:teuthology.orchestra.run.smithi040.stdout:pg_upmap_items 4.1d [5,8] 2024-03-18T22:42:51.497 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/347668672 expires 2024-03-19T22:33:42.721571+0000 2024-03-18T22:42:51.497 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/4237962144 expires 2024-03-19T22:33:42.721571+0000 2024-03-18T22:42:51.497 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/1454961757 expires 2024-03-19T22:33:15.826478+0000 2024-03-18T22:42:51.497 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/3047621005 expires 2024-03-19T22:33:15.826478+0000 2024-03-18T22:42:51.497 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/2121391701 expires 2024-03-19T22:33:15.826478+0000 2024-03-18T22:42:51.497 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6801/3876208039 expires 2024-03-19T22:33:42.721571+0000 2024-03-18T22:42:51.497 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6800/3876208039 expires 2024-03-19T22:33:42.721571+0000 2024-03-18T22:42:51.497 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:0/2591529486 expires 2024-03-19T22:33:42.721571+0000 2024-03-18T22:42:51.497 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6800/1153335782 expires 2024-03-19T22:33:15.826478+0000 2024-03-18T22:42:51.497 INFO:teuthology.orchestra.run.smithi040.stdout:blocklist 172.21.15.40:6801/1153335782 expires 2024-03-19T22:33:15.826478+0000 2024-03-18T22:42:51.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.495+0000 7f1124089700 1 -- 172.21.15.40:0/1224884747 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f11080a2890 msgr2=0x7f11080a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:51.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.495+0000 7f1124089700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f11080a2890 0x7f11080a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f110c000c00 tx=0x7f110c005fb0 comp rx=0 tx=0).stop 2024-03-18T22:42:51.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.495+0000 7f1124089700 1 -- 172.21.15.40:0/1224884747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f111c101230 msgr2=0x7f111c06b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:51.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.495+0000 7f1124089700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f111c101230 0x7f111c06b3b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f111800e970 tx=0x7f111800ec80 comp rx=0 tx=0).stop 2024-03-18T22:42:51.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.495+0000 7f1124089700 1 -- 172.21.15.40:0/1224884747 shutdown_connections 2024-03-18T22:42:51.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.495+0000 7f1124089700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f111c102dd0 0x7f111c19f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:51.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.495+0000 7f1124089700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f111c101230 0x7f111c06b3b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:51.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.495+0000 7f1124089700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f11080a2890 0x7f11080a4d50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:51.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.495+0000 7f1124089700 1 --2- 172.21.15.40:0/1224884747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f111c102430 0x7f111c19d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:51.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.495+0000 7f1124089700 1 -- 172.21.15.40:0/1224884747 >> 172.21.15.40:0/1224884747 conn(0x7f111c0fc7d0 msgr2=0x7f111c1057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:51.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.495+0000 7f1124089700 1 -- 172.21.15.40:0/1224884747 shutdown_connections 2024-03-18T22:42:51.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:51.495+0000 7f1124089700 1 -- 172.21.15.40:0/1224884747 wait complete. 2024-03-18T22:42:52.139 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:52 smithi040 ceph-mon[62643]: pgmap v357: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3 op/s; 158 B/s, 1 objects/s recovering 2024-03-18T22:42:52.139 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:52 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1224884747' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T22:42:52.140 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-18T22:42:52.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:42:52.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.257+0000 7f3cdfd1b640 1 -- 172.21.15.40:0/84722410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd8104c70 msgr2=0x7f3cd8107060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:52.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.257+0000 7f3cdfd1b640 1 --2- 172.21.15.40:0/84722410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd8104c70 0x7f3cd8107060 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8009900 tx=0x7f3cc802bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:52.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.257+0000 7f3cdfd1b640 1 -- 172.21.15.40:0/84722410 shutdown_connections 2024-03-18T22:42:52.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.257+0000 7f3cdfd1b640 1 --2- 172.21.15.40:0/84722410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cd81075a0 0x7f3cd81099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.257+0000 7f3cdfd1b640 1 --2- 172.21.15.40:0/84722410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd8104c70 0x7f3cd8107060 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.257+0000 7f3cdfd1b640 1 --2- 172.21.15.40:0/84722410 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd8102340 0x7f3cd8104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.257+0000 7f3cdfd1b640 1 -- 172.21.15.40:0/84722410 >> 172.21.15.40:0/84722410 conn(0x7f3cd80fbd50 msgr2=0x7f3cd80fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:52.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.257+0000 7f3cdfd1b640 1 -- 172.21.15.40:0/84722410 shutdown_connections 2024-03-18T22:42:52.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.257+0000 7f3cdfd1b640 1 -- 172.21.15.40:0/84722410 wait complete. 2024-03-18T22:42:52.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cdfd1b640 1 Processor -- start 2024-03-18T22:42:52.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cdfd1b640 1 -- start start 2024-03-18T22:42:52.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cdfd1b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd8102340 0x7f3cd8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:52.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cdfd1b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd8104c70 0x7f3cd8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:52.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cdfd1b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cd81075a0 0x7f3cd819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:52.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cdfd1b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd810d390 con 0x7f3cd8102340 2024-03-18T22:42:52.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cdfd1b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd810d210 con 0x7f3cd81075a0 2024-03-18T22:42:52.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cdfd1b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd810d510 con 0x7f3cd8104c70 2024-03-18T22:42:52.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cddab7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd8102340 0x7f3cd8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:52.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cddab7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd8102340 0x7f3cd8197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39238/0 (socket says 172.21.15.40:39238) 2024-03-18T22:42:52.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cddab7640 1 -- 172.21.15.40:0/846063596 learned_addr learned my addr 172.21.15.40:0/846063596 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:52.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cdd2b6640 1 --2- 172.21.15.40:0/846063596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd8104c70 0x7f3cd8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:52.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cde2b8640 1 --2- 172.21.15.40:0/846063596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cd81075a0 0x7f3cd819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:52.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cddab7640 1 -- 172.21.15.40:0/846063596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd8104c70 msgr2=0x7f3cd8198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:52.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cddab7640 1 --2- 172.21.15.40:0/846063596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd8104c70 0x7f3cd8198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cddab7640 1 -- 172.21.15.40:0/846063596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cd81075a0 msgr2=0x7f3cd819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:52.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cddab7640 1 --2- 172.21.15.40:0/846063596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cd81075a0 0x7f3cd819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.258+0000 7f3cddab7640 1 -- 172.21.15.40:0/846063596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd819f6e0 con 0x7f3cd8102340 2024-03-18T22:42:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.259+0000 7f3cddab7640 1 --2- 172.21.15.40:0/846063596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd8102340 0x7f3cd8197ac0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400e9f0 tx=0x7f3cd400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.259+0000 7f3cceffd640 1 -- 172.21.15.40:0/846063596 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd400cd80 con 0x7f3cd8102340 2024-03-18T22:42:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.259+0000 7f3cdfd1b640 1 -- 172.21.15.40:0/846063596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd819f9d0 con 0x7f3cd8102340 2024-03-18T22:42:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.259+0000 7f3cceffd640 1 -- 172.21.15.40:0/846063596 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3cd40054d0 con 0x7f3cd8102340 2024-03-18T22:42:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.259+0000 7f3cceffd640 1 -- 172.21.15.40:0/846063596 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4010490 con 0x7f3cd8102340 2024-03-18T22:42:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.259+0000 7f3cdfd1b640 1 -- 172.21.15.40:0/846063596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cd819fea0 con 0x7f3cd8102340 2024-03-18T22:42:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.260+0000 7f3cdfd1b640 1 -- 172.21.15.40:0/846063596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cd80fd840 con 0x7f3cd8102340 2024-03-18T22:42:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.261+0000 7f3cceffd640 1 -- 172.21.15.40:0/846063596 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3cd4005020 con 0x7f3cd8102340 2024-03-18T22:42:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.261+0000 7f3cceffd640 1 --2- 172.21.15.40:0/846063596 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3cc40a2800 0x7f3cc40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.261+0000 7f3cdd2b6640 1 --2- 172.21.15.40:0/846063596 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3cc40a2800 0x7f3cc40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.261+0000 7f3cceffd640 1 -- 172.21.15.40:0/846063596 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f3cd40cf1b0 con 0x7f3cd8102340 2024-03-18T22:42:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.262+0000 7f3cdd2b6640 1 --2- 172.21.15.40:0/846063596 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3cc40a2800 0x7f3cc40a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8005ab0 tx=0x7f3cc8005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:52.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.266+0000 7f3cceffd640 1 -- 172.21.15.40:0/846063596 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3cd409be60 con 0x7f3cd8102340 2024-03-18T22:42:52.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:52 smithi113 ceph-mon[66775]: pgmap v357: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3 op/s; 158 B/s, 1 objects/s recovering 2024-03-18T22:42:52.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:52 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1224884747' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T22:42:52.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:52 smithi103 ceph-mon[66377]: pgmap v357: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3 op/s; 158 B/s, 1 objects/s recovering 2024-03-18T22:42:52.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:52 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1224884747' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T22:42:52.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.462+0000 7f3cdfd1b640 1 -- 172.21.15.40:0/846063596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3cd8062450 con 0x7f3cd8102340 2024-03-18T22:42:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.463+0000 7f3cceffd640 1 -- 172.21.15.40:0/846063596 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24908 (secure 0 0 0) 0x7f3cd409b760 con 0x7f3cd8102340 2024-03-18T22:42:52.465 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:52.466 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":93,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:42:44.941873+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:42:52.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.466+0000 7f3cdfd1b640 1 -- 172.21.15.40:0/846063596 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3cc40a2800 msgr2=0x7f3cc40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.466+0000 7f3cdfd1b640 1 --2- 172.21.15.40:0/846063596 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3cc40a2800 0x7f3cc40a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8005ab0 tx=0x7f3cc8005fd0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.466+0000 7f3cdfd1b640 1 -- 172.21.15.40:0/846063596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd8102340 msgr2=0x7f3cd8197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.466+0000 7f3cdfd1b640 1 --2- 172.21.15.40:0/846063596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd8102340 0x7f3cd8197ac0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400e9f0 tx=0x7f3cd400edf0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.466+0000 7f3cdfd1b640 1 -- 172.21.15.40:0/846063596 shutdown_connections 2024-03-18T22:42:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.466+0000 7f3cdfd1b640 1 --2- 172.21.15.40:0/846063596 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3cc40a2800 0x7f3cc40a4cc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.466+0000 7f3cdfd1b640 1 --2- 172.21.15.40:0/846063596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3cd81075a0 0x7f3cd819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.466+0000 7f3cdfd1b640 1 --2- 172.21.15.40:0/846063596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3cd8104c70 0x7f3cd8198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.466+0000 7f3cdfd1b640 1 --2- 172.21.15.40:0/846063596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cd8102340 0x7f3cd8197ac0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.466+0000 7f3cdfd1b640 1 -- 172.21.15.40:0/846063596 >> 172.21.15.40:0/846063596 conn(0x7f3cd80fbd50 msgr2=0x7f3cd8102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.466+0000 7f3cdfd1b640 1 -- 172.21.15.40:0/846063596 shutdown_connections 2024-03-18T22:42:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.466+0000 7f3cdfd1b640 1 -- 172.21.15.40:0/846063596 wait complete. 2024-03-18T22:42:52.479 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T22:36:07.060691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T22:40:39.925147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T22:40:40.773757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T22:40:44.068183+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T22:42:52.479 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T22:42:52.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 -- 172.21.15.40:0/1138132097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe518101910 msgr2=0x7fe518101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:52.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 --2- 172.21.15.40:0/1138132097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe518101910 0x7fe518101d90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe508009860 tx=0x7fe50802bb70 comp rx=0 tx=0).stop 2024-03-18T22:42:52.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 -- 172.21.15.40:0/1138132097 shutdown_connections 2024-03-18T22:42:52.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 --2- 172.21.15.40:0/1138132097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5181022d0 0x7fe518108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 --2- 172.21.15.40:0/1138132097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe518101910 0x7fe518101d90 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 --2- 172.21.15.40:0/1138132097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe518100720 0x7fe518100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 -- 172.21.15.40:0/1138132097 >> 172.21.15.40:0/1138132097 conn(0x7fe5180fbcb0 msgr2=0x7fe5180fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:52.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 -- 172.21.15.40:0/1138132097 shutdown_connections 2024-03-18T22:42:52.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 -- 172.21.15.40:0/1138132097 wait complete. 2024-03-18T22:42:52.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 Processor -- start 2024-03-18T22:42:52.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 -- start start 2024-03-18T22:42:52.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe518100720 0x7fe518197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:52.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe518101910 0x7fe518197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:52.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5181022d0 0x7fe51819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:52.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe51810d370 con 0x7fe518101910 2024-03-18T22:42:52.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe51810d1f0 con 0x7fe5181022d0 2024-03-18T22:42:52.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.584+0000 7fe51ffc6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe51810d4f0 con 0x7fe518100720 2024-03-18T22:42:52.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe51d561640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe518101910 0x7fe518197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:52.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe51dd62640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe518100720 0x7fe518197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:52.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe51d561640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe518101910 0x7fe518197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39260/0 (socket says 172.21.15.40:39260) 2024-03-18T22:42:52.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe51dd62640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe518100720 0x7fe518197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:40416/0 (socket says 172.21.15.40:40416) 2024-03-18T22:42:52.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe51d561640 1 -- 172.21.15.40:0/2787735692 learned_addr learned my addr 172.21.15.40:0/2787735692 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:52.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe51e563640 1 --2- 172.21.15.40:0/2787735692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5181022d0 0x7fe51819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:52.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe51d561640 1 -- 172.21.15.40:0/2787735692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe518100720 msgr2=0x7fe518197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:52.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe51d561640 1 --2- 172.21.15.40:0/2787735692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe518100720 0x7fe518197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe51d561640 1 -- 172.21.15.40:0/2787735692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5181022d0 msgr2=0x7fe51819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:52.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe51d561640 1 --2- 172.21.15.40:0/2787735692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5181022d0 0x7fe51819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe51d561640 1 -- 172.21.15.40:0/2787735692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51819f690 con 0x7fe518101910 2024-03-18T22:42:52.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe51d561640 1 --2- 172.21.15.40:0/2787735692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe518101910 0x7fe518197fb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe5080057c0 tx=0x7fe5080042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:52.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe50effd640 1 -- 172.21.15.40:0/2787735692 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508041030 con 0x7fe518101910 2024-03-18T22:42:52.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe51ffc6640 1 -- 172.21.15.40:0/2787735692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51819f920 con 0x7fe518101910 2024-03-18T22:42:52.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe50effd640 1 -- 172.21.15.40:0/2787735692 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe50802c9c0 con 0x7fe518101910 2024-03-18T22:42:52.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe50effd640 1 -- 172.21.15.40:0/2787735692 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508039040 con 0x7fe518101910 2024-03-18T22:42:52.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.585+0000 7fe51ffc6640 1 -- 172.21.15.40:0/2787735692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe51819fe10 con 0x7fe518101910 2024-03-18T22:42:52.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.586+0000 7fe51ffc6640 1 -- 172.21.15.40:0/2787735692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe518107b60 con 0x7fe518101910 2024-03-18T22:42:52.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.587+0000 7fe50effd640 1 -- 172.21.15.40:0/2787735692 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe508044070 con 0x7fe518101910 2024-03-18T22:42:52.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.588+0000 7fe50effd640 1 --2- 172.21.15.40:0/2787735692 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe5040a2800 0x7fe5040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:52.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.588+0000 7fe50effd640 1 -- 172.21.15.40:0/2787735692 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe5080f03c0 con 0x7fe518101910 2024-03-18T22:42:52.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.588+0000 7fe51dd62640 1 --2- 172.21.15.40:0/2787735692 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe5040a2800 0x7fe5040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:52.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.588+0000 7fe51dd62640 1 --2- 172.21.15.40:0/2787735692 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe5040a2800 0x7fe5040a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe518101770 tx=0x7fe514009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:52.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.592+0000 7fe50effd640 1 -- 172.21.15.40:0/2787735692 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe5080bd070 con 0x7fe518101910 2024-03-18T22:42:52.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.775+0000 7fe51ffc6640 1 -- 172.21.15.40:0/2787735692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe518062420 con 0x7fe518101910 2024-03-18T22:42:52.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.776+0000 7fe50effd640 1 -- 172.21.15.40:0/2787735692 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fe50802cb90 con 0x7fe518101910 2024-03-18T22:42:52.778 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-18T22:42:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.778+0000 7fe51ffc6640 1 -- 172.21.15.40:0/2787735692 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe5040a2800 msgr2=0x7fe5040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.778+0000 7fe51ffc6640 1 --2- 172.21.15.40:0/2787735692 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe5040a2800 0x7fe5040a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe518101770 tx=0x7fe514009260 comp rx=0 tx=0).stop 2024-03-18T22:42:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.778+0000 7fe51ffc6640 1 -- 172.21.15.40:0/2787735692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe518101910 msgr2=0x7fe518197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.778+0000 7fe51ffc6640 1 --2- 172.21.15.40:0/2787735692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe518101910 0x7fe518197fb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe5080057c0 tx=0x7fe5080042f0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.778+0000 7fe51ffc6640 1 -- 172.21.15.40:0/2787735692 shutdown_connections 2024-03-18T22:42:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.778+0000 7fe51ffc6640 1 --2- 172.21.15.40:0/2787735692 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe5040a2800 0x7fe5040a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.778+0000 7fe51ffc6640 1 --2- 172.21.15.40:0/2787735692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5181022d0 0x7fe51819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.778+0000 7fe51ffc6640 1 --2- 172.21.15.40:0/2787735692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe518101910 0x7fe518197fb0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.778+0000 7fe51ffc6640 1 --2- 172.21.15.40:0/2787735692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe518100720 0x7fe518197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.778+0000 7fe51ffc6640 1 -- 172.21.15.40:0/2787735692 >> 172.21.15.40:0/2787735692 conn(0x7fe5180fbcb0 msgr2=0x7fe518104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.779+0000 7fe51ffc6640 1 -- 172.21.15.40:0/2787735692 shutdown_connections 2024-03-18T22:42:52.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.779+0000 7fe51ffc6640 1 -- 172.21.15.40:0/2787735692 wait complete. 2024-03-18T22:42:52.791 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T22:42:52.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.896+0000 7fc865190640 1 -- 172.21.15.40:0/1755065826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc860102a20 msgr2=0x7fc860102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:52.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.896+0000 7fc865190640 1 --2- 172.21.15.40:0/1755065826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc860102a20 0x7fc860102ea0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc854009860 tx=0x7fc85402bb70 comp rx=0 tx=0).stop 2024-03-18T22:42:52.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.896+0000 7fc865190640 1 -- 172.21.15.40:0/1755065826 shutdown_connections 2024-03-18T22:42:52.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.896+0000 7fc865190640 1 --2- 172.21.15.40:0/1755065826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8601033e0 0x7fc860109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.896+0000 7fc865190640 1 --2- 172.21.15.40:0/1755065826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc860102a20 0x7fc860102ea0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.896+0000 7fc865190640 1 --2- 172.21.15.40:0/1755065826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc860101830 0x7fc860101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.896+0000 7fc865190640 1 -- 172.21.15.40:0/1755065826 >> 172.21.15.40:0/1755065826 conn(0x7fc8600fcdc0 msgr2=0x7fc8600ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:52.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.896+0000 7fc865190640 1 -- 172.21.15.40:0/1755065826 shutdown_connections 2024-03-18T22:42:52.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.896+0000 7fc865190640 1 -- 172.21.15.40:0/1755065826 wait complete. 2024-03-18T22:42:52.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc865190640 1 Processor -- start 2024-03-18T22:42:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc865190640 1 -- start start 2024-03-18T22:42:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc865190640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc860101830 0x7fc8600755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc865190640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc860102a20 0x7fc860073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc865190640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8601033e0 0x7fc860074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc865190640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc86010d360 con 0x7fc860101830 2024-03-18T22:42:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc865190640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc86010d1e0 con 0x7fc860102a20 2024-03-18T22:42:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc865190640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc86010d4e0 con 0x7fc8601033e0 2024-03-18T22:42:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc85ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc860101830 0x7fc8600755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc85ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc860101830 0x7fc8600755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39286/0 (socket says 172.21.15.40:39286) 2024-03-18T22:42:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc85ed9d640 1 -- 172.21.15.40:0/2946710294 learned_addr learned my addr 172.21.15.40:0/2946710294 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc85f59e640 1 --2- 172.21.15.40:0/2946710294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8601033e0 0x7fc860074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc85e59c640 1 --2- 172.21.15.40:0/2946710294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc860102a20 0x7fc860073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:52.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc85ed9d640 1 -- 172.21.15.40:0/2946710294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8601033e0 msgr2=0x7fc860074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:52.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc85ed9d640 1 --2- 172.21.15.40:0/2946710294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8601033e0 0x7fc860074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc85ed9d640 1 -- 172.21.15.40:0/2946710294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc860102a20 msgr2=0x7fc860073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:52.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc85ed9d640 1 --2- 172.21.15.40:0/2946710294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc860102a20 0x7fc860073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:52.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.897+0000 7fc85ed9d640 1 -- 172.21.15.40:0/2946710294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8600749e0 con 0x7fc860101830 2024-03-18T22:42:52.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.898+0000 7fc85ed9d640 1 --2- 172.21.15.40:0/2946710294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc860101830 0x7fc8600755d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc85000e9c0 tx=0x7fc85000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:52.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.898+0000 7fc847fff640 1 -- 172.21.15.40:0/2946710294 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85000cd80 con 0x7fc860101830 2024-03-18T22:42:52.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.898+0000 7fc865190640 1 -- 172.21.15.40:0/2946710294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8601a1bf0 con 0x7fc860101830 2024-03-18T22:42:52.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.898+0000 7fc847fff640 1 -- 172.21.15.40:0/2946710294 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc8500035f0 con 0x7fc860101830 2024-03-18T22:42:52.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.898+0000 7fc847fff640 1 -- 172.21.15.40:0/2946710294 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc850010690 con 0x7fc860101830 2024-03-18T22:42:52.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.898+0000 7fc865190640 1 -- 172.21.15.40:0/2946710294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8601a20c0 con 0x7fc860101830 2024-03-18T22:42:52.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.899+0000 7fc865190640 1 -- 172.21.15.40:0/2946710294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc860068a60 con 0x7fc860101830 2024-03-18T22:42:52.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.900+0000 7fc847fff640 1 -- 172.21.15.40:0/2946710294 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc8500037a0 con 0x7fc860101830 2024-03-18T22:42:52.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.900+0000 7fc847fff640 1 --2- 172.21.15.40:0/2946710294 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8480a2800 0x7fc8480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:52.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.901+0000 7fc85e59c640 1 --2- 172.21.15.40:0/2946710294 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8480a2800 0x7fc8480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:52.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.901+0000 7fc847fff640 1 -- 172.21.15.40:0/2946710294 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc8500cf400 con 0x7fc860101830 2024-03-18T22:42:52.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.901+0000 7fc85e59c640 1 --2- 172.21.15.40:0/2946710294 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8480a2800 0x7fc8480a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc85402c180 tx=0x7fc854034980 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:52.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:52.905+0000 7fc847fff640 1 -- 172.21.15.40:0/2946710294 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc85009c050 con 0x7fc860101830 2024-03-18T22:42:53.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.092+0000 7fc865190640 1 -- 172.21.15.40:0/2946710294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc860062420 con 0x7fc860101830 2024-03-18T22:42:53.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.092+0000 7fc847fff640 1 -- 172.21.15.40:0/2946710294 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fc85009b950 con 0x7fc860101830 2024-03-18T22:42:53.094 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:42:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.094+0000 7fc865190640 1 -- 172.21.15.40:0/2946710294 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8480a2800 msgr2=0x7fc8480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.095+0000 7fc865190640 1 --2- 172.21.15.40:0/2946710294 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8480a2800 0x7fc8480a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc85402c180 tx=0x7fc854034980 comp rx=0 tx=0).stop 2024-03-18T22:42:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.095+0000 7fc865190640 1 -- 172.21.15.40:0/2946710294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc860101830 msgr2=0x7fc8600755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.095+0000 7fc865190640 1 --2- 172.21.15.40:0/2946710294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc860101830 0x7fc8600755d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc85000e9c0 tx=0x7fc85000ee80 comp rx=0 tx=0).stop 2024-03-18T22:42:53.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.095+0000 7fc865190640 1 -- 172.21.15.40:0/2946710294 shutdown_connections 2024-03-18T22:42:53.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.095+0000 7fc865190640 1 --2- 172.21.15.40:0/2946710294 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8480a2800 0x7fc8480a4cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.095+0000 7fc865190640 1 --2- 172.21.15.40:0/2946710294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc8601033e0 0x7fc860074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.095+0000 7fc865190640 1 --2- 172.21.15.40:0/2946710294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc860102a20 0x7fc860073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.095+0000 7fc865190640 1 --2- 172.21.15.40:0/2946710294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc860101830 0x7fc8600755d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.095+0000 7fc865190640 1 -- 172.21.15.40:0/2946710294 >> 172.21.15.40:0/2946710294 conn(0x7fc8600fcdc0 msgr2=0x7fc860105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:53.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.095+0000 7fc865190640 1 -- 172.21.15.40:0/2946710294 shutdown_connections 2024-03-18T22:42:53.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.095+0000 7fc865190640 1 -- 172.21.15.40:0/2946710294 wait complete. 2024-03-18T22:42:53.106 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:53 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/846063596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:53.106 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:53 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2787735692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:42:53.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T22:42:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.211+0000 7f6aee2f8640 1 -- 172.21.15.40:0/2643006878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae81022d0 msgr2=0x7f6ae8108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.211+0000 7f6aee2f8640 1 --2- 172.21.15.40:0/2643006878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae81022d0 0x7f6ae8108b60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0009ca0 tx=0x7f6ae002c100 comp rx=0 tx=0).stop 2024-03-18T22:42:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.211+0000 7f6aee2f8640 1 -- 172.21.15.40:0/2643006878 shutdown_connections 2024-03-18T22:42:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.211+0000 7f6aee2f8640 1 --2- 172.21.15.40:0/2643006878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae81022d0 0x7f6ae8108b60 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.211+0000 7f6aee2f8640 1 --2- 172.21.15.40:0/2643006878 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ae8101910 0x7f6ae8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.211+0000 7f6aee2f8640 1 --2- 172.21.15.40:0/2643006878 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ae8100720 0x7f6ae8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.211+0000 7f6aee2f8640 1 -- 172.21.15.40:0/2643006878 >> 172.21.15.40:0/2643006878 conn(0x7f6ae80fbcb0 msgr2=0x7f6ae80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.211+0000 7f6aee2f8640 1 -- 172.21.15.40:0/2643006878 shutdown_connections 2024-03-18T22:42:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.211+0000 7f6aee2f8640 1 -- 172.21.15.40:0/2643006878 wait complete. 2024-03-18T22:42:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.212+0000 7f6aee2f8640 1 Processor -- start 2024-03-18T22:42:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.212+0000 7f6aee2f8640 1 -- start start 2024-03-18T22:42:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.212+0000 7f6aee2f8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ae8100720 0x7f6ae80694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.212+0000 7f6aee2f8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae8101910 0x7f6ae819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.212+0000 7f6aee2f8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ae81022d0 0x7f6ae819ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.212+0000 7f6aee2f8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae810c250 con 0x7f6ae8101910 2024-03-18T22:42:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.212+0000 7f6aee2f8640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae810c0d0 con 0x7f6ae81022d0 2024-03-18T22:42:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.212+0000 7f6aee2f8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae810c3d0 con 0x7f6ae8100720 2024-03-18T22:42:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.212+0000 7f6ae77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae8101910 0x7f6ae819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.212+0000 7f6ae77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae8101910 0x7f6ae819c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39294/0 (socket says 172.21.15.40:39294) 2024-03-18T22:42:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.212+0000 7f6ae77fe640 1 -- 172.21.15.40:0/2370109495 learned_addr learned my addr 172.21.15.40:0/2370109495 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.212+0000 7f6ae7fff640 1 --2- 172.21.15.40:0/2370109495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ae8100720 0x7f6ae80694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.212+0000 7f6aec895640 1 --2- 172.21.15.40:0/2370109495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ae81022d0 0x7f6ae819ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:53.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.213+0000 7f6ae77fe640 1 -- 172.21.15.40:0/2370109495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ae8100720 msgr2=0x7f6ae80694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:53.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.213+0000 7f6ae77fe640 1 --2- 172.21.15.40:0/2370109495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ae8100720 0x7f6ae80694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.213+0000 7f6ae77fe640 1 -- 172.21.15.40:0/2370109495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ae81022d0 msgr2=0x7f6ae819ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:53.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.213+0000 7f6ae77fe640 1 --2- 172.21.15.40:0/2370109495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ae81022d0 0x7f6ae819ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.213+0000 7f6ae77fe640 1 -- 172.21.15.40:0/2370109495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae819f6f0 con 0x7f6ae8101910 2024-03-18T22:42:53.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.213+0000 7f6ae77fe640 1 --2- 172.21.15.40:0/2370109495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae8101910 0x7f6ae819c7c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6ad800e9f0 tx=0x7f6ad800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:53.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.213+0000 7f6ae57fa640 1 -- 172.21.15.40:0/2370109495 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad800cd80 con 0x7f6ae8101910 2024-03-18T22:42:53.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.213+0000 7f6aee2f8640 1 -- 172.21.15.40:0/2370109495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae819f980 con 0x7f6ae8101910 2024-03-18T22:42:53.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.213+0000 7f6ae57fa640 1 -- 172.21.15.40:0/2370109495 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ad80108e0 con 0x7f6ae8101910 2024-03-18T22:42:53.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.213+0000 7f6ae57fa640 1 -- 172.21.15.40:0/2370109495 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8018920 con 0x7f6ae8101910 2024-03-18T22:42:53.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.213+0000 7f6aee2f8640 1 -- 172.21.15.40:0/2370109495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae819ff20 con 0x7f6ae8101910 2024-03-18T22:42:53.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.214+0000 7f6aee2f8640 1 -- 172.21.15.40:0/2370109495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae8107b60 con 0x7f6ae8101910 2024-03-18T22:42:53.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.215+0000 7f6ae57fa640 1 -- 172.21.15.40:0/2370109495 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6ad8010a90 con 0x7f6ae8101910 2024-03-18T22:42:53.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.216+0000 7f6ae57fa640 1 --2- 172.21.15.40:0/2370109495 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ad40a2800 0x7f6ad40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.216+0000 7f6ae7fff640 1 --2- 172.21.15.40:0/2370109495 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ad40a2800 0x7f6ad40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.216+0000 7f6ae57fa640 1 -- 172.21.15.40:0/2370109495 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6ad80cf5e0 con 0x7f6ae8101910 2024-03-18T22:42:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.216+0000 7f6ae7fff640 1 --2- 172.21.15.40:0/2370109495 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ad40a2800 0x7f6ad40a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8101770 tx=0x7f6ad0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.220+0000 7f6ae57fa640 1 -- 172.21.15.40:0/2370109495 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6ad809bc40 con 0x7f6ae8101910 2024-03-18T22:42:53.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:53 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/846063596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:53.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:53 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2787735692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:42:53.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:53 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/846063596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:53.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:53 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2787735692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:42:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.404+0000 7f6aee2f8640 1 -- 172.21.15.40:0/2370109495 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6ae802d1f0 con 0x7f6ae8101910 2024-03-18T22:42:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.404+0000 7f6ae57fa640 1 -- 172.21.15.40:0/2370109495 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f6ad809fce0 con 0x7f6ae8101910 2024-03-18T22:42:53.406 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-18T22:42:53.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.406+0000 7f6aee2f8640 1 -- 172.21.15.40:0/2370109495 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ad40a2800 msgr2=0x7f6ad40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:53.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.406+0000 7f6aee2f8640 1 --2- 172.21.15.40:0/2370109495 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ad40a2800 0x7f6ad40a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8101770 tx=0x7f6ad0005f90 comp rx=0 tx=0).stop 2024-03-18T22:42:53.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.406+0000 7f6aee2f8640 1 -- 172.21.15.40:0/2370109495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae8101910 msgr2=0x7f6ae819c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:53.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.406+0000 7f6aee2f8640 1 --2- 172.21.15.40:0/2370109495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae8101910 0x7f6ae819c7c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6ad800e9f0 tx=0x7f6ad800eeb0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.406+0000 7f6aee2f8640 1 -- 172.21.15.40:0/2370109495 shutdown_connections 2024-03-18T22:42:53.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.406+0000 7f6aee2f8640 1 --2- 172.21.15.40:0/2370109495 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ad40a2800 0x7f6ad40a4cc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.406+0000 7f6aee2f8640 1 --2- 172.21.15.40:0/2370109495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ae81022d0 0x7f6ae819ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.406+0000 7f6aee2f8640 1 --2- 172.21.15.40:0/2370109495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ae8101910 0x7f6ae819c7c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.406+0000 7f6aee2f8640 1 --2- 172.21.15.40:0/2370109495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ae8100720 0x7f6ae80694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.406+0000 7f6aee2f8640 1 -- 172.21.15.40:0/2370109495 >> 172.21.15.40:0/2370109495 conn(0x7f6ae80fbcb0 msgr2=0x7f6ae8104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:53.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.406+0000 7f6aee2f8640 1 -- 172.21.15.40:0/2370109495 shutdown_connections 2024-03-18T22:42:53.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.406+0000 7f6aee2f8640 1 -- 172.21.15.40:0/2370109495 wait complete. 2024-03-18T22:42:53.419 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T22:42:53.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.523+0000 7f90baab2640 1 -- 172.21.15.40:0/185177192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b4101830 msgr2=0x7f90b4101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:53.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.523+0000 7f90baab2640 1 --2- 172.21.15.40:0/185177192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b4101830 0x7f90b4101c50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f90a8009900 tx=0x7f90a802bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:53.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.523+0000 7f90baab2640 1 -- 172.21.15.40:0/185177192 shutdown_connections 2024-03-18T22:42:53.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.523+0000 7f90baab2640 1 --2- 172.21.15.40:0/185177192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b41033e0 0x7f90b4109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.523+0000 7f90baab2640 1 --2- 172.21.15.40:0/185177192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b4102a20 0x7f90b4102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.523+0000 7f90baab2640 1 --2- 172.21.15.40:0/185177192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b4101830 0x7f90b4101c50 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.523+0000 7f90baab2640 1 -- 172.21.15.40:0/185177192 >> 172.21.15.40:0/185177192 conn(0x7f90b40fcdc0 msgr2=0x7f90b40ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:53.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.523+0000 7f90baab2640 1 -- 172.21.15.40:0/185177192 shutdown_connections 2024-03-18T22:42:53.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.523+0000 7f90baab2640 1 -- 172.21.15.40:0/185177192 wait complete. 2024-03-18T22:42:53.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.523+0000 7f90baab2640 1 Processor -- start 2024-03-18T22:42:53.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.523+0000 7f90baab2640 1 -- start start 2024-03-18T22:42:53.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90baab2640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b4101830 0x7f90b40755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:53.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90baab2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b4102a20 0x7f90b4073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:53.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90baab2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b41033e0 0x7f90b4074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:53.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90baab2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f90b410d330 con 0x7f90b41033e0 2024-03-18T22:42:53.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90baab2640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f90b410d1b0 con 0x7f90b4101830 2024-03-18T22:42:53.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90baab2640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f90b410d4b0 con 0x7f90b4102a20 2024-03-18T22:42:53.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90b904f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b41033e0 0x7f90b4074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:53.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90b904f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b41033e0 0x7f90b4074150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39306/0 (socket says 172.21.15.40:39306) 2024-03-18T22:42:53.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90b904f640 1 -- 172.21.15.40:0/51289717 learned_addr learned my addr 172.21.15.40:0/51289717 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:53.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90b3fff640 1 --2- 172.21.15.40:0/51289717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b4102a20 0x7f90b4073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:53.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90b884e640 1 --2- 172.21.15.40:0/51289717 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b4101830 0x7f90b40755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:53.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90b904f640 1 -- 172.21.15.40:0/51289717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b4102a20 msgr2=0x7f90b4073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:53.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90b904f640 1 --2- 172.21.15.40:0/51289717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b4102a20 0x7f90b4073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90b904f640 1 -- 172.21.15.40:0/51289717 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b4101830 msgr2=0x7f90b40755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:53.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90b904f640 1 --2- 172.21.15.40:0/51289717 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b4101830 0x7f90b40755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90b904f640 1 -- 172.21.15.40:0/51289717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b40749e0 con 0x7f90b41033e0 2024-03-18T22:42:53.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90b904f640 1 --2- 172.21.15.40:0/51289717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b41033e0 0x7f90b4074150 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00eb10 tx=0x7f90ac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:53.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90b1ffb640 1 -- 172.21.15.40:0/51289717 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ac004b20 con 0x7f90b41033e0 2024-03-18T22:42:53.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90b1ffb640 1 -- 172.21.15.40:0/51289717 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90ac004cc0 con 0x7f90b41033e0 2024-03-18T22:42:53.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90baab2640 1 -- 172.21.15.40:0/51289717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b41a1bd0 con 0x7f90b41033e0 2024-03-18T22:42:53.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.524+0000 7f90b1ffb640 1 -- 172.21.15.40:0/51289717 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ac010690 con 0x7f90b41033e0 2024-03-18T22:42:53.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.525+0000 7f90baab2640 1 -- 172.21.15.40:0/51289717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b41a2120 con 0x7f90b41033e0 2024-03-18T22:42:53.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.526+0000 7f90baab2640 1 -- 172.21.15.40:0/51289717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90b4108ce0 con 0x7f90b41033e0 2024-03-18T22:42:53.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.526+0000 7f90b1ffb640 1 -- 172.21.15.40:0/51289717 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f90ac005020 con 0x7f90b41033e0 2024-03-18T22:42:53.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.527+0000 7f90b1ffb640 1 --2- 172.21.15.40:0/51289717 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90a40a2880 0x7f90a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:53.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.527+0000 7f90b884e640 1 --2- 172.21.15.40:0/51289717 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90a40a2880 0x7f90a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:53.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.527+0000 7f90b1ffb640 1 -- 172.21.15.40:0/51289717 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f90ac0cf3d0 con 0x7f90b41033e0 2024-03-18T22:42:53.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.527+0000 7f90b884e640 1 --2- 172.21.15.40:0/51289717 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90a40a2880 0x7f90a40a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f90a8005ab0 tx=0x7f90a8005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:53.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.531+0000 7f90b1ffb640 1 -- 172.21.15.40:0/51289717 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f90ac09c080 con 0x7f90b41033e0 2024-03-18T22:42:53.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.713+0000 7f90baab2640 1 -- 172.21.15.40:0/51289717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f90b4062420 con 0x7f90b41033e0 2024-03-18T22:42:53.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.714+0000 7f90b1ffb640 1 -- 172.21.15.40:0/51289717 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f90ac09b980 con 0x7f90b41033e0 2024-03-18T22:42:53.715 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:42:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.716+0000 7f90baab2640 1 -- 172.21.15.40:0/51289717 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90a40a2880 msgr2=0x7f90a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.716+0000 7f90baab2640 1 --2- 172.21.15.40:0/51289717 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90a40a2880 0x7f90a40a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f90a8005ab0 tx=0x7f90a8005fd0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.716+0000 7f90baab2640 1 -- 172.21.15.40:0/51289717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b41033e0 msgr2=0x7f90b4074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.716+0000 7f90baab2640 1 --2- 172.21.15.40:0/51289717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b41033e0 0x7f90b4074150 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00eb10 tx=0x7f90ac00c2d0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.716+0000 7f90baab2640 1 -- 172.21.15.40:0/51289717 shutdown_connections 2024-03-18T22:42:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.716+0000 7f90baab2640 1 --2- 172.21.15.40:0/51289717 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90a40a2880 0x7f90a40a4d40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.716+0000 7f90baab2640 1 --2- 172.21.15.40:0/51289717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b41033e0 0x7f90b4074150 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.716+0000 7f90baab2640 1 --2- 172.21.15.40:0/51289717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90b4102a20 0x7f90b4073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.716+0000 7f90baab2640 1 --2- 172.21.15.40:0/51289717 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f90b4101830 0x7f90b40755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.716+0000 7f90baab2640 1 -- 172.21.15.40:0/51289717 >> 172.21.15.40:0/51289717 conn(0x7f90b40fcdc0 msgr2=0x7f90b4105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.716+0000 7f90baab2640 1 -- 172.21.15.40:0/51289717 shutdown_connections 2024-03-18T22:42:53.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.716+0000 7f90baab2640 1 -- 172.21.15.40:0/51289717 wait complete. 2024-03-18T22:42:53.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:42:53.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.829+0000 7fd3d40ac640 1 -- 172.21.15.40:0/948016462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc102360 msgr2=0x7fd3cc104750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:53.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.829+0000 7fd3d40ac640 1 --2- 172.21.15.40:0/948016462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc102360 0x7fd3cc104750 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009900 tx=0x7fd3c802bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:53.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.829+0000 7fd3d40ac640 1 -- 172.21.15.40:0/948016462 shutdown_connections 2024-03-18T22:42:53.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.829+0000 7fd3d40ac640 1 --2- 172.21.15.40:0/948016462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3cc1075c0 0x7fd3cc1099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.829+0000 7fd3d40ac640 1 --2- 172.21.15.40:0/948016462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3cc104c90 0x7fd3cc107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.829+0000 7fd3d40ac640 1 --2- 172.21.15.40:0/948016462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc102360 0x7fd3cc104750 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.829+0000 7fd3d40ac640 1 -- 172.21.15.40:0/948016462 >> 172.21.15.40:0/948016462 conn(0x7fd3cc0fbcd0 msgr2=0x7fd3cc0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.829+0000 7fd3d40ac640 1 -- 172.21.15.40:0/948016462 shutdown_connections 2024-03-18T22:42:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.829+0000 7fd3d40ac640 1 -- 172.21.15.40:0/948016462 wait complete. 2024-03-18T22:42:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d40ac640 1 Processor -- start 2024-03-18T22:42:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d40ac640 1 -- start start 2024-03-18T22:42:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d40ac640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3cc102360 0x7fd3cc197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d40ac640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3cc104c90 0x7fd3cc197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d40ac640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc1075c0 0x7fd3cc19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d40ac640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc10d380 con 0x7fd3cc1075c0 2024-03-18T22:42:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d40ac640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc10d200 con 0x7fd3cc104c90 2024-03-18T22:42:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d40ac640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc10d500 con 0x7fd3cc102360 2024-03-18T22:42:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d2649640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc1075c0 0x7fd3cc19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d1e48640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3cc102360 0x7fd3cc197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d2649640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc1075c0 0x7fd3cc19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39330/0 (socket says 172.21.15.40:39330) 2024-03-18T22:42:53.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d1e48640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3cc102360 0x7fd3cc197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:40522/0 (socket says 172.21.15.40:40522) 2024-03-18T22:42:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d2649640 1 -- 172.21.15.40:0/1512891141 learned_addr learned my addr 172.21.15.40:0/1512891141 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d1647640 1 --2- 172.21.15.40:0/1512891141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3cc104c90 0x7fd3cc197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d2649640 1 -- 172.21.15.40:0/1512891141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3cc102360 msgr2=0x7fd3cc197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d2649640 1 --2- 172.21.15.40:0/1512891141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3cc102360 0x7fd3cc197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d2649640 1 -- 172.21.15.40:0/1512891141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3cc104c90 msgr2=0x7fd3cc197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d2649640 1 --2- 172.21.15.40:0/1512891141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3cc104c90 0x7fd3cc197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d2649640 1 -- 172.21.15.40:0/1512891141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3cc19f6b0 con 0x7fd3cc1075c0 2024-03-18T22:42:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d1e48640 1 --2- 172.21.15.40:0/1512891141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3cc102360 0x7fd3cc197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.830+0000 7fd3d2649640 1 --2- 172.21.15.40:0/1512891141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc1075c0 0x7fd3cc19f040 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400eb10 tx=0x7fd3c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.831+0000 7fd3c2ffd640 1 -- 172.21.15.40:0/1512891141 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4004b20 con 0x7fd3cc1075c0 2024-03-18T22:42:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.831+0000 7fd3c2ffd640 1 -- 172.21.15.40:0/1512891141 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3c4004cc0 con 0x7fd3cc1075c0 2024-03-18T22:42:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.831+0000 7fd3d40ac640 1 -- 172.21.15.40:0/1512891141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3cc19f9a0 con 0x7fd3cc1075c0 2024-03-18T22:42:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.831+0000 7fd3c2ffd640 1 -- 172.21.15.40:0/1512891141 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4010690 con 0x7fd3cc1075c0 2024-03-18T22:42:53.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.831+0000 7fd3d40ac640 1 -- 172.21.15.40:0/1512891141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3cc19fef0 con 0x7fd3cc1075c0 2024-03-18T22:42:53.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.832+0000 7fd3d40ac640 1 -- 172.21.15.40:0/1512891141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3cc0fd7c0 con 0x7fd3cc1075c0 2024-03-18T22:42:53.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.833+0000 7fd3c2ffd640 1 -- 172.21.15.40:0/1512891141 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd3c4005020 con 0x7fd3cc1075c0 2024-03-18T22:42:53.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.833+0000 7fd3c2ffd640 1 --2- 172.21.15.40:0/1512891141 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd3b80a2800 0x7fd3b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:53.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.833+0000 7fd3c2ffd640 1 -- 172.21.15.40:0/1512891141 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd3c40cf510 con 0x7fd3cc1075c0 2024-03-18T22:42:53.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.833+0000 7fd3d1e48640 1 --2- 172.21.15.40:0/1512891141 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd3b80a2800 0x7fd3b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:53.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.834+0000 7fd3d1e48640 1 --2- 172.21.15.40:0/1512891141 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd3b80a2800 0x7fd3b80a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009230 tx=0x7fd3c8005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:53.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:53.838+0000 7fd3c2ffd640 1 -- 172.21.15.40:0/1512891141 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd3c409c1c0 con 0x7fd3cc1075c0 2024-03-18T22:42:54.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.061+0000 7fd3d40ac640 1 -- 172.21.15.40:0/1512891141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3cc062420 con 0x7fd3cc1075c0 2024-03-18T22:42:54.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.062+0000 7fd3c2ffd640 1 -- 172.21.15.40:0/1512891141 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7fd3c409bac0 con 0x7fd3cc1075c0 2024-03-18T22:42:54.064 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:54.064 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:41:09.839105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24481,24509,24473,24401,24388]},"id":1}]} 2024-03-18T22:42:54.064 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 27 2024-03-18T22:42:54.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.063+0000 7fd3d40ac640 1 -- 172.21.15.40:0/1512891141 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd3b80a2800 msgr2=0x7fd3b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:54.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.063+0000 7fd3d40ac640 1 --2- 172.21.15.40:0/1512891141 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd3b80a2800 0x7fd3b80a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009230 tx=0x7fd3c8005fd0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.063+0000 7fd3d40ac640 1 -- 172.21.15.40:0/1512891141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc1075c0 msgr2=0x7fd3cc19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:54.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.063+0000 7fd3d40ac640 1 --2- 172.21.15.40:0/1512891141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc1075c0 0x7fd3cc19f040 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400eb10 tx=0x7fd3c400c2d0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.064+0000 7fd3d40ac640 1 -- 172.21.15.40:0/1512891141 shutdown_connections 2024-03-18T22:42:54.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.064+0000 7fd3d40ac640 1 --2- 172.21.15.40:0/1512891141 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd3b80a2800 0x7fd3b80a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.064+0000 7fd3d40ac640 1 --2- 172.21.15.40:0/1512891141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3cc1075c0 0x7fd3cc19f040 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.064+0000 7fd3d40ac640 1 --2- 172.21.15.40:0/1512891141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd3cc104c90 0x7fd3cc197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.064+0000 7fd3d40ac640 1 --2- 172.21.15.40:0/1512891141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd3cc102360 0x7fd3cc197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.064+0000 7fd3d40ac640 1 -- 172.21.15.40:0/1512891141 >> 172.21.15.40:0/1512891141 conn(0x7fd3cc0fbcd0 msgr2=0x7fd3cc102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:54.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.064+0000 7fd3d40ac640 1 -- 172.21.15.40:0/1512891141 shutdown_connections 2024-03-18T22:42:54.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.064+0000 7fd3d40ac640 1 -- 172.21.15.40:0/1512891141 wait complete. 2024-03-18T22:42:54.077 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:42:54.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.178+0000 7f5f12576640 1 -- 172.21.15.40:0/3075405982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f0c102a40 msgr2=0x7f5f0c102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:54.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.178+0000 7f5f12576640 1 --2- 172.21.15.40:0/3075405982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f0c102a40 0x7f5f0c102ec0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5f00009900 tx=0x7f5f0002bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:54.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.178+0000 7f5f12576640 1 -- 172.21.15.40:0/3075405982 shutdown_connections 2024-03-18T22:42:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.178+0000 7f5f12576640 1 --2- 172.21.15.40:0/3075405982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f0c103400 0x7f5f0c109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.178+0000 7f5f12576640 1 --2- 172.21.15.40:0/3075405982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f0c102a40 0x7f5f0c102ec0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.178+0000 7f5f12576640 1 --2- 172.21.15.40:0/3075405982 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f0c101850 0x7f5f0c101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.178+0000 7f5f12576640 1 -- 172.21.15.40:0/3075405982 >> 172.21.15.40:0/3075405982 conn(0x7f5f0c0fce00 msgr2=0x7f5f0c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.178+0000 7f5f12576640 1 -- 172.21.15.40:0/3075405982 shutdown_connections 2024-03-18T22:42:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.178+0000 7f5f12576640 1 -- 172.21.15.40:0/3075405982 wait complete. 2024-03-18T22:42:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.178+0000 7f5f12576640 1 Processor -- start 2024-03-18T22:42:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.178+0000 7f5f12576640 1 -- start start 2024-03-18T22:42:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f12576640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f0c101850 0x7f5f0c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f12576640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f0c102a40 0x7f5f0c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f12576640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f0c103400 0x7f5f0c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f12576640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c10d380 con 0x7f5f0c103400 2024-03-18T22:42:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f12576640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c10d200 con 0x7f5f0c101850 2024-03-18T22:42:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f12576640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c10d500 con 0x7f5f0c102a40 2024-03-18T22:42:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f10b13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f0c103400 0x7f5f0c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f0bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f0c101850 0x7f5f0c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f0bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f0c101850 0x7f5f0c197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:44420/0 (socket says 172.21.15.40:44420) 2024-03-18T22:42:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f0bfff640 1 -- 172.21.15.40:0/194475718 learned_addr learned my addr 172.21.15.40:0/194475718 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f0b7fe640 1 --2- 172.21.15.40:0/194475718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f0c102a40 0x7f5f0c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f0bfff640 1 -- 172.21.15.40:0/194475718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f0c102a40 msgr2=0x7f5f0c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f0bfff640 1 --2- 172.21.15.40:0/194475718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f0c102a40 0x7f5f0c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f0bfff640 1 -- 172.21.15.40:0/194475718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f0c103400 msgr2=0x7f5f0c19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f0bfff640 1 --2- 172.21.15.40:0/194475718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f0c103400 0x7f5f0c19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f0bfff640 1 -- 172.21.15.40:0/194475718 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f0c19f6d0 con 0x7f5f0c101850 2024-03-18T22:42:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f0bfff640 1 --2- 172.21.15.40:0/194475718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f0c101850 0x7f5f0c197ab0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5efc00e9c0 tx=0x7f5efc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f097fa640 1 -- 172.21.15.40:0/194475718 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5efc00cd80 con 0x7f5f0c101850 2024-03-18T22:42:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f12576640 1 -- 172.21.15.40:0/194475718 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f0c19f9c0 con 0x7f5f0c101850 2024-03-18T22:42:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f097fa640 1 -- 172.21.15.40:0/194475718 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5efc0035f0 con 0x7f5f0c101850 2024-03-18T22:42:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f097fa640 1 -- 172.21.15.40:0/194475718 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5efc010690 con 0x7f5f0c101850 2024-03-18T22:42:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.179+0000 7f5f12576640 1 -- 172.21.15.40:0/194475718 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f0c19fee0 con 0x7f5f0c101850 2024-03-18T22:42:54.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.180+0000 7f5f12576640 1 -- 172.21.15.40:0/194475718 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f0c108c60 con 0x7f5f0c101850 2024-03-18T22:42:54.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.181+0000 7f5f097fa640 1 -- 172.21.15.40:0/194475718 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5efc0037a0 con 0x7f5f0c101850 2024-03-18T22:42:54.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.182+0000 7f5f097fa640 1 --2- 172.21.15.40:0/194475718 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5ef40a2800 0x7f5ef40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:54.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.182+0000 7f5f0b7fe640 1 --2- 172.21.15.40:0/194475718 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5ef40a2800 0x7f5ef40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:54.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.182+0000 7f5f097fa640 1 -- 172.21.15.40:0/194475718 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5efc0cf490 con 0x7f5f0c101850 2024-03-18T22:42:54.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.182+0000 7f5f0b7fe640 1 --2- 172.21.15.40:0/194475718 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5ef40a2800 0x7f5ef40a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5f00009230 tx=0x7f5f00005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:54.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.190+0000 7f5f097fa640 1 -- 172.21.15.40:0/194475718 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5efc09c140 con 0x7f5f0c101850 2024-03-18T22:42:54.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:54 smithi113 ceph-mon[66775]: pgmap v358: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4 op/s; 142 B/s, 1 objects/s recovering 2024-03-18T22:42:54.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:54 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2946710294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:42:54.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:54 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2370109495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:42:54.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:54 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/51289717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:42:54.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:54 smithi103 ceph-mon[66377]: pgmap v358: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4 op/s; 142 B/s, 1 objects/s recovering 2024-03-18T22:42:54.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:54 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2946710294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:42:54.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:54 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2370109495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:42:54.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:54 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/51289717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:42:54.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.375+0000 7f5f12576640 1 -- 172.21.15.40:0/194475718 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f0c02d4a0 con 0x7f5f0c101850 2024-03-18T22:42:54.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.377+0000 7f5f097fa640 1 -- 172.21.15.40:0/194475718 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24908 (secure 0 0 0) 0x7f5efc09ba40 con 0x7f5f0c101850 2024-03-18T22:42:54.379 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:54.379 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":93,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:42:44.941873+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:42:54.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.379+0000 7f5f12576640 1 -- 172.21.15.40:0/194475718 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5ef40a2800 msgr2=0x7f5ef40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:54.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.379+0000 7f5f12576640 1 --2- 172.21.15.40:0/194475718 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5ef40a2800 0x7f5ef40a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5f00009230 tx=0x7f5f00005fd0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.379+0000 7f5f12576640 1 -- 172.21.15.40:0/194475718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f0c101850 msgr2=0x7f5f0c197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:54.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.379+0000 7f5f12576640 1 --2- 172.21.15.40:0/194475718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f0c101850 0x7f5f0c197ab0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5efc00e9c0 tx=0x7f5efc00ee80 comp rx=0 tx=0).stop 2024-03-18T22:42:54.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.379+0000 7f5f12576640 1 -- 172.21.15.40:0/194475718 shutdown_connections 2024-03-18T22:42:54.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.379+0000 7f5f12576640 1 --2- 172.21.15.40:0/194475718 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5ef40a2800 0x7f5ef40a4cc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.379+0000 7f5f12576640 1 --2- 172.21.15.40:0/194475718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f0c103400 0x7f5f0c19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.379+0000 7f5f12576640 1 --2- 172.21.15.40:0/194475718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5f0c102a40 0x7f5f0c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.379+0000 7f5f12576640 1 --2- 172.21.15.40:0/194475718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f0c101850 0x7f5f0c197ab0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.379+0000 7f5f12576640 1 -- 172.21.15.40:0/194475718 >> 172.21.15.40:0/194475718 conn(0x7f5f0c0fce00 msgr2=0x7f5f0c105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:54.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.380+0000 7f5f12576640 1 -- 172.21.15.40:0/194475718 shutdown_connections 2024-03-18T22:42:54.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.380+0000 7f5f12576640 1 -- 172.21.15.40:0/194475718 wait complete. 2024-03-18T22:42:54.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:54 smithi040 ceph-mon[62643]: pgmap v358: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4 op/s; 142 B/s, 1 objects/s recovering 2024-03-18T22:42:54.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:54 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2946710294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:42:54.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:54 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2370109495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:42:54.391 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:54 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/51289717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:42:54.393 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T22:36:07.060691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T22:40:39.925147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T22:40:40.773757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T22:40:44.068183+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T22:42:54.393 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T22:42:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.498+0000 7fa0d65e6640 1 -- 172.21.15.40:0/897611564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0d0101920 msgr2=0x7fa0d0101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.498+0000 7fa0d65e6640 1 --2- 172.21.15.40:0/897611564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0d0101920 0x7fa0d0101da0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009860 tx=0x7fa0c002bb70 comp rx=0 tx=0).stop 2024-03-18T22:42:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.498+0000 7fa0d65e6640 1 -- 172.21.15.40:0/897611564 shutdown_connections 2024-03-18T22:42:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.498+0000 7fa0d65e6640 1 --2- 172.21.15.40:0/897611564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0d01022e0 0x7fa0d0108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.498+0000 7fa0d65e6640 1 --2- 172.21.15.40:0/897611564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0d0101920 0x7fa0d0101da0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.498+0000 7fa0d65e6640 1 --2- 172.21.15.40:0/897611564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0d0100730 0x7fa0d0100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.498+0000 7fa0d65e6640 1 -- 172.21.15.40:0/897611564 >> 172.21.15.40:0/897611564 conn(0x7fa0d00fbcc0 msgr2=0x7fa0d00fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.498+0000 7fa0d65e6640 1 -- 172.21.15.40:0/897611564 shutdown_connections 2024-03-18T22:42:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.498+0000 7fa0d65e6640 1 -- 172.21.15.40:0/897611564 wait complete. 2024-03-18T22:42:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0d65e6640 1 Processor -- start 2024-03-18T22:42:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0d65e6640 1 -- start start 2024-03-18T22:42:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0d65e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0d0100730 0x7fa0d006a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0d65e6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0d0101920 0x7fa0d019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0d65e6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0d01022e0 0x7fa0d019ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0d65e6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d010c240 con 0x7fa0d0100730 2024-03-18T22:42:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0d65e6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d010c0c0 con 0x7fa0d01022e0 2024-03-18T22:42:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0d65e6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d010c3c0 con 0x7fa0d0101920 2024-03-18T22:42:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0cffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0d0100730 0x7fa0d006a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0cf7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0d0101920 0x7fa0d019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0cffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0d0100730 0x7fa0d006a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39368/0 (socket says 172.21.15.40:39368) 2024-03-18T22:42:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0cffff640 1 -- 172.21.15.40:0/1477743327 learned_addr learned my addr 172.21.15.40:0/1477743327 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0d4b83640 1 --2- 172.21.15.40:0/1477743327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0d01022e0 0x7fa0d019ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0cf7fe640 1 -- 172.21.15.40:0/1477743327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0d01022e0 msgr2=0x7fa0d019ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:54.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0cf7fe640 1 --2- 172.21.15.40:0/1477743327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0d01022e0 0x7fa0d019ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0cf7fe640 1 -- 172.21.15.40:0/1477743327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0d0100730 msgr2=0x7fa0d006a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:54.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0cf7fe640 1 --2- 172.21.15.40:0/1477743327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0d0100730 0x7fa0d006a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.499+0000 7fa0cf7fe640 1 -- 172.21.15.40:0/1477743327 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d019f720 con 0x7fa0d0101920 2024-03-18T22:42:54.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.500+0000 7fa0cf7fe640 1 --2- 172.21.15.40:0/1477743327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0d0101920 0x7fa0d019c7f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0037040 tx=0x7fa0c00040e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:54.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.500+0000 7fa0cd7fa640 1 -- 172.21.15.40:0/1477743327 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0041020 con 0x7fa0d0101920 2024-03-18T22:42:54.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.500+0000 7fa0cd7fa640 1 -- 172.21.15.40:0/1477743327 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0c00044d0 con 0x7fa0d0101920 2024-03-18T22:42:54.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.500+0000 7fa0d65e6640 1 -- 172.21.15.40:0/1477743327 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d019f950 con 0x7fa0d0101920 2024-03-18T22:42:54.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.500+0000 7fa0cd7fa640 1 -- 172.21.15.40:0/1477743327 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0039040 con 0x7fa0d0101920 2024-03-18T22:42:54.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.500+0000 7fa0d65e6640 1 -- 172.21.15.40:0/1477743327 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d019fe40 con 0x7fa0d0101920 2024-03-18T22:42:54.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.501+0000 7fa0d65e6640 1 -- 172.21.15.40:0/1477743327 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d0107b40 con 0x7fa0d0101920 2024-03-18T22:42:54.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.502+0000 7fa0cd7fa640 1 -- 172.21.15.40:0/1477743327 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa0c0044020 con 0x7fa0d0101920 2024-03-18T22:42:54.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.502+0000 7fa0cd7fa640 1 --2- 172.21.15.40:0/1477743327 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa0bc0a2800 0x7fa0bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:54.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.503+0000 7fa0cffff640 1 --2- 172.21.15.40:0/1477743327 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa0bc0a2800 0x7fa0bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:54.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.503+0000 7fa0cd7fa640 1 -- 172.21.15.40:0/1477743327 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa0c00f0b20 con 0x7fa0d0101920 2024-03-18T22:42:54.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.503+0000 7fa0cffff640 1 --2- 172.21.15.40:0/1477743327 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa0bc0a2800 0x7fa0bc0a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009c60 tx=0x7fa0b8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:54.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.506+0000 7fa0cd7fa640 1 -- 172.21.15.40:0/1477743327 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa0c004b020 con 0x7fa0d0101920 2024-03-18T22:42:54.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.688+0000 7fa0d65e6640 1 -- 172.21.15.40:0/1477743327 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa0d002d400 con 0x7fa0d0101920 2024-03-18T22:42:54.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.689+0000 7fa0cd7fa640 1 -- 172.21.15.40:0/1477743327 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fa0c00bd770 con 0x7fa0d0101920 2024-03-18T22:42:54.690 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-18T22:42:54.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.690+0000 7fa0d65e6640 1 -- 172.21.15.40:0/1477743327 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa0bc0a2800 msgr2=0x7fa0bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:54.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.690+0000 7fa0d65e6640 1 --2- 172.21.15.40:0/1477743327 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa0bc0a2800 0x7fa0bc0a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009c60 tx=0x7fa0b8009260 comp rx=0 tx=0).stop 2024-03-18T22:42:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.690+0000 7fa0d65e6640 1 -- 172.21.15.40:0/1477743327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0d0101920 msgr2=0x7fa0d019c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.690+0000 7fa0d65e6640 1 --2- 172.21.15.40:0/1477743327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0d0101920 0x7fa0d019c7f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0037040 tx=0x7fa0c00040e0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.691+0000 7fa0d65e6640 1 -- 172.21.15.40:0/1477743327 shutdown_connections 2024-03-18T22:42:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.691+0000 7fa0d65e6640 1 --2- 172.21.15.40:0/1477743327 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa0bc0a2800 0x7fa0bc0a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.691+0000 7fa0d65e6640 1 --2- 172.21.15.40:0/1477743327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa0d01022e0 0x7fa0d019ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.691+0000 7fa0d65e6640 1 --2- 172.21.15.40:0/1477743327 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa0d0101920 0x7fa0d019c7f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.691+0000 7fa0d65e6640 1 --2- 172.21.15.40:0/1477743327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0d0100730 0x7fa0d006a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.691+0000 7fa0d65e6640 1 -- 172.21.15.40:0/1477743327 >> 172.21.15.40:0/1477743327 conn(0x7fa0d00fbcc0 msgr2=0x7fa0d0104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.691+0000 7fa0d65e6640 1 -- 172.21.15.40:0/1477743327 shutdown_connections 2024-03-18T22:42:54.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.691+0000 7fa0d65e6640 1 -- 172.21.15.40:0/1477743327 wait complete. 2024-03-18T22:42:54.703 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T22:42:54.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.807+0000 7f102c1b2640 1 -- 172.21.15.40:0/1692559871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101920 msgr2=0x7f1024101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:54.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.807+0000 7f102c1b2640 1 --2- 172.21.15.40:0/1692559871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101920 0x7f1024101da0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1014009900 tx=0x7f101402bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:54.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.807+0000 7f102c1b2640 1 -- 172.21.15.40:0/1692559871 shutdown_connections 2024-03-18T22:42:54.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.807+0000 7f102c1b2640 1 --2- 172.21.15.40:0/1692559871 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10241022e0 0x7f1024108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.807+0000 7f102c1b2640 1 --2- 172.21.15.40:0/1692559871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101920 0x7f1024101da0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.807+0000 7f102c1b2640 1 --2- 172.21.15.40:0/1692559871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1024100730 0x7f1024100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.807+0000 7f102c1b2640 1 -- 172.21.15.40:0/1692559871 >> 172.21.15.40:0/1692559871 conn(0x7f10240fbcc0 msgr2=0x7f10240fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:54.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.807+0000 7f102c1b2640 1 -- 172.21.15.40:0/1692559871 shutdown_connections 2024-03-18T22:42:54.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.807+0000 7f102c1b2640 1 -- 172.21.15.40:0/1692559871 wait complete. 2024-03-18T22:42:54.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.807+0000 7f102c1b2640 1 Processor -- start 2024-03-18T22:42:54.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.807+0000 7f102c1b2640 1 -- start start 2024-03-18T22:42:54.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102c1b2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1024100730 0x7f1024196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:54.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102c1b2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101920 0x7f1024196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:54.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102c1b2640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10241022e0 0x7f102419df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:54.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102c1b2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f102410c260 con 0x7f1024101920 2024-03-18T22:42:54.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102c1b2640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f102410c0e0 con 0x7f10241022e0 2024-03-18T22:42:54.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102c1b2640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f102410c3e0 con 0x7f1024100730 2024-03-18T22:42:54.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102974d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101920 0x7f1024196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:54.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f1029f4e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1024100730 0x7f1024196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:54.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102974d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101920 0x7f1024196eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39390/0 (socket says 172.21.15.40:39390) 2024-03-18T22:42:54.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102974d640 1 -- 172.21.15.40:0/847231503 learned_addr learned my addr 172.21.15.40:0/847231503 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:54.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102a74f640 1 --2- 172.21.15.40:0/847231503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10241022e0 0x7f102419df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:54.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102974d640 1 -- 172.21.15.40:0/847231503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1024100730 msgr2=0x7f1024196970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:54.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102974d640 1 --2- 172.21.15.40:0/847231503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1024100730 0x7f1024196970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102974d640 1 -- 172.21.15.40:0/847231503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10241022e0 msgr2=0x7f102419df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:54.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102974d640 1 --2- 172.21.15.40:0/847231503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10241022e0 0x7f102419df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:54.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102974d640 1 -- 172.21.15.40:0/847231503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f102419e590 con 0x7f1024101920 2024-03-18T22:42:54.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f1029f4e640 1 --2- 172.21.15.40:0/847231503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1024100730 0x7f1024196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:54.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102974d640 1 --2- 172.21.15.40:0/847231503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101920 0x7f1024196eb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1014006010 tx=0x7f1014005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:54.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f101affd640 1 -- 172.21.15.40:0/847231503 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1014004e00 con 0x7f1024101920 2024-03-18T22:42:54.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102c1b2640 1 -- 172.21.15.40:0/847231503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1024069df0 con 0x7f1024101920 2024-03-18T22:42:54.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f101affd640 1 -- 172.21.15.40:0/847231503 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1014005450 con 0x7f1024101920 2024-03-18T22:42:54.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f101affd640 1 -- 172.21.15.40:0/847231503 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101403c450 con 0x7f1024101920 2024-03-18T22:42:54.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.808+0000 7f102c1b2640 1 -- 172.21.15.40:0/847231503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f102406a340 con 0x7f1024101920 2024-03-18T22:42:54.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.809+0000 7f102c1b2640 1 -- 172.21.15.40:0/847231503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1024068a60 con 0x7f1024101920 2024-03-18T22:42:54.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.811+0000 7f101affd640 1 -- 172.21.15.40:0/847231503 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f10140036f0 con 0x7f1024101920 2024-03-18T22:42:54.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.811+0000 7f101affd640 1 --2- 172.21.15.40:0/847231503 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f10100a2800 0x7f10100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:54.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.811+0000 7f1029f4e640 1 --2- 172.21.15.40:0/847231503 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f10100a2800 0x7f10100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:54.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.811+0000 7f101affd640 1 -- 172.21.15.40:0/847231503 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f10140f30a0 con 0x7f1024101920 2024-03-18T22:42:54.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.812+0000 7f1029f4e640 1 --2- 172.21.15.40:0/847231503 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f10100a2800 0x7f10100a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1024101780 tx=0x7f1020005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:54.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.816+0000 7f101affd640 1 -- 172.21.15.40:0/847231503 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f10140bfd50 con 0x7f1024101920 2024-03-18T22:42:55.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.999+0000 7f102c1b2640 1 -- 172.21.15.40:0/847231503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f102419e7c0 con 0x7f1024101920 2024-03-18T22:42:55.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:54.999+0000 7f101affd640 1 -- 172.21.15.40:0/847231503 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f10140bf650 con 0x7f1024101920 2024-03-18T22:42:55.001 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:42:55.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.001+0000 7f102c1b2640 1 -- 172.21.15.40:0/847231503 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f10100a2800 msgr2=0x7f10100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.001+0000 7f102c1b2640 1 --2- 172.21.15.40:0/847231503 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f10100a2800 0x7f10100a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1024101780 tx=0x7f1020005f90 comp rx=0 tx=0).stop 2024-03-18T22:42:55.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.001+0000 7f102c1b2640 1 -- 172.21.15.40:0/847231503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101920 msgr2=0x7f1024196eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.001+0000 7f102c1b2640 1 --2- 172.21.15.40:0/847231503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101920 0x7f1024196eb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1014006010 tx=0x7f1014005e40 comp rx=0 tx=0).stop 2024-03-18T22:42:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.001+0000 7f102c1b2640 1 -- 172.21.15.40:0/847231503 shutdown_connections 2024-03-18T22:42:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.001+0000 7f102c1b2640 1 --2- 172.21.15.40:0/847231503 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f10100a2800 0x7f10100a4cc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.001+0000 7f102c1b2640 1 --2- 172.21.15.40:0/847231503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10241022e0 0x7f102419df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.001+0000 7f102c1b2640 1 --2- 172.21.15.40:0/847231503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1024101920 0x7f1024196eb0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.001+0000 7f102c1b2640 1 --2- 172.21.15.40:0/847231503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1024100730 0x7f1024196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.001+0000 7f102c1b2640 1 -- 172.21.15.40:0/847231503 >> 172.21.15.40:0/847231503 conn(0x7f10240fbcc0 msgr2=0x7f1024104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.002+0000 7f102c1b2640 1 -- 172.21.15.40:0/847231503 shutdown_connections 2024-03-18T22:42:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.002+0000 7f102c1b2640 1 -- 172.21.15.40:0/847231503 wait complete. 2024-03-18T22:42:55.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T22:42:55.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.119+0000 7f6ea9f5b640 1 -- 172.21.15.40:0/1083344307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea40ffb00 msgr2=0x7f6ea40fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.119+0000 7f6ea9f5b640 1 --2- 172.21.15.40:0/1083344307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea40ffb00 0x7f6ea40fff20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6e94009930 tx=0x7f6e9402bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:55.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.119+0000 7f6ea9f5b640 1 -- 172.21.15.40:0/1083344307 shutdown_connections 2024-03-18T22:42:55.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.119+0000 7f6ea9f5b640 1 --2- 172.21.15.40:0/1083344307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ea4103490 0x7f6ea4109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.119+0000 7f6ea9f5b640 1 --2- 172.21.15.40:0/1083344307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ea4102ad0 0x7f6ea4102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.119+0000 7f6ea9f5b640 1 --2- 172.21.15.40:0/1083344307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea40ffb00 0x7f6ea40fff20 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.119+0000 7f6ea9f5b640 1 -- 172.21.15.40:0/1083344307 >> 172.21.15.40:0/1083344307 conn(0x7f6ea40fb430 msgr2=0x7f6ea40fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:55.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.119+0000 7f6ea9f5b640 1 -- 172.21.15.40:0/1083344307 shutdown_connections 2024-03-18T22:42:55.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.119+0000 7f6ea9f5b640 1 -- 172.21.15.40:0/1083344307 wait complete. 2024-03-18T22:42:55.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.120+0000 7f6ea9f5b640 1 Processor -- start 2024-03-18T22:42:55.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.120+0000 7f6ea9f5b640 1 -- start start 2024-03-18T22:42:55.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.120+0000 7f6ea9f5b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea40ffb00 0x7f6ea4102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:55.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.120+0000 7f6ea9f5b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ea4102ad0 0x7f6ea4100a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:55.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.120+0000 7f6ea9f5b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ea4103490 0x7f6ea4100fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:55.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.120+0000 7f6ea9f5b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea410d360 con 0x7f6ea40ffb00 2024-03-18T22:42:55.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.120+0000 7f6ea9f5b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea410d1e0 con 0x7f6ea4103490 2024-03-18T22:42:55.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.120+0000 7f6ea9f5b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea410d4e0 con 0x7f6ea4102ad0 2024-03-18T22:42:55.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.120+0000 7f6ea37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea40ffb00 0x7f6ea4102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:55.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.120+0000 7f6ea37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea40ffb00 0x7f6ea4102440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39408/0 (socket says 172.21.15.40:39408) 2024-03-18T22:42:55.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.120+0000 7f6ea37fe640 1 -- 172.21.15.40:0/1203483089 learned_addr learned my addr 172.21.15.40:0/1203483089 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:55.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.120+0000 7f6ea3fff640 1 --2- 172.21.15.40:0/1203483089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ea4103490 0x7f6ea4100fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:55.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.120+0000 7f6ea2ffd640 1 --2- 172.21.15.40:0/1203483089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ea4102ad0 0x7f6ea4100a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:55.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.121+0000 7f6ea37fe640 1 -- 172.21.15.40:0/1203483089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ea4102ad0 msgr2=0x7f6ea4100a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.121+0000 7f6ea37fe640 1 --2- 172.21.15.40:0/1203483089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ea4102ad0 0x7f6ea4100a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.121+0000 7f6ea37fe640 1 -- 172.21.15.40:0/1203483089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ea4103490 msgr2=0x7f6ea4100fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.121+0000 7f6ea37fe640 1 --2- 172.21.15.40:0/1203483089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ea4103490 0x7f6ea4100fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.121+0000 7f6ea37fe640 1 -- 172.21.15.40:0/1203483089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea4101850 con 0x7f6ea40ffb00 2024-03-18T22:42:55.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.121+0000 7f6ea3fff640 1 --2- 172.21.15.40:0/1203483089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ea4103490 0x7f6ea4100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:55.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.121+0000 7f6ea37fe640 1 --2- 172.21.15.40:0/1203483089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea40ffb00 0x7f6ea4102440 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6e94000c00 tx=0x7f6e94005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:55.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.121+0000 7f6ea0ff9640 1 -- 172.21.15.40:0/1203483089 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e94043020 con 0x7f6ea40ffb00 2024-03-18T22:42:55.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.121+0000 7f6ea9f5b640 1 -- 172.21.15.40:0/1203483089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea41a1c30 con 0x7f6ea40ffb00 2024-03-18T22:42:55.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.121+0000 7f6ea0ff9640 1 -- 172.21.15.40:0/1203483089 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e940043d0 con 0x7f6ea40ffb00 2024-03-18T22:42:55.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.121+0000 7f6ea0ff9640 1 -- 172.21.15.40:0/1203483089 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e94003ae0 con 0x7f6ea40ffb00 2024-03-18T22:42:55.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.121+0000 7f6ea9f5b640 1 -- 172.21.15.40:0/1203483089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea41a21d0 con 0x7f6ea40ffb00 2024-03-18T22:42:55.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.122+0000 7f6ea9f5b640 1 -- 172.21.15.40:0/1203483089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ea4108d20 con 0x7f6ea40ffb00 2024-03-18T22:42:55.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.123+0000 7f6ea0ff9640 1 -- 172.21.15.40:0/1203483089 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6e94046020 con 0x7f6ea40ffb00 2024-03-18T22:42:55.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.124+0000 7f6ea0ff9640 1 --2- 172.21.15.40:0/1203483089 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6e8c0a2800 0x7f6e8c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:55.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.124+0000 7f6ea2ffd640 1 --2- 172.21.15.40:0/1203483089 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6e8c0a2800 0x7f6e8c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:55.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.124+0000 7f6ea0ff9640 1 -- 172.21.15.40:0/1203483089 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6e940f0f30 con 0x7f6ea40ffb00 2024-03-18T22:42:55.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.124+0000 7f6ea2ffd640 1 --2- 172.21.15.40:0/1203483089 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6e8c0a2800 0x7f6e8c0a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6ea41021b0 tx=0x7f6e98005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:55.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.128+0000 7f6ea0ff9640 1 -- 172.21.15.40:0/1203483089 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6e9404d020 con 0x7f6ea40ffb00 2024-03-18T22:42:55.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.311+0000 7f6ea9f5b640 1 -- 172.21.15.40:0/1203483089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6ea4062420 con 0x7f6ea40ffb00 2024-03-18T22:42:55.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.311+0000 7f6ea0ff9640 1 -- 172.21.15.40:0/1203483089 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f6e940bdbe0 con 0x7f6ea40ffb00 2024-03-18T22:42:55.313 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-18T22:42:55.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.313+0000 7f6ea9f5b640 1 -- 172.21.15.40:0/1203483089 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6e8c0a2800 msgr2=0x7f6e8c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.313+0000 7f6ea9f5b640 1 --2- 172.21.15.40:0/1203483089 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6e8c0a2800 0x7f6e8c0a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6ea41021b0 tx=0x7f6e98005f90 comp rx=0 tx=0).stop 2024-03-18T22:42:55.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.313+0000 7f6ea9f5b640 1 -- 172.21.15.40:0/1203483089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea40ffb00 msgr2=0x7f6ea4102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.313+0000 7f6ea9f5b640 1 --2- 172.21.15.40:0/1203483089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea40ffb00 0x7f6ea4102440 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6e94000c00 tx=0x7f6e94005e70 comp rx=0 tx=0).stop 2024-03-18T22:42:55.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.313+0000 7f6ea9f5b640 1 -- 172.21.15.40:0/1203483089 shutdown_connections 2024-03-18T22:42:55.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.313+0000 7f6ea9f5b640 1 --2- 172.21.15.40:0/1203483089 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6e8c0a2800 0x7f6e8c0a4cc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.313+0000 7f6ea9f5b640 1 --2- 172.21.15.40:0/1203483089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ea4103490 0x7f6ea4100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.313+0000 7f6ea9f5b640 1 --2- 172.21.15.40:0/1203483089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ea4102ad0 0x7f6ea4100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.313+0000 7f6ea9f5b640 1 --2- 172.21.15.40:0/1203483089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea40ffb00 0x7f6ea4102440 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.313+0000 7f6ea9f5b640 1 -- 172.21.15.40:0/1203483089 >> 172.21.15.40:0/1203483089 conn(0x7f6ea40fb430 msgr2=0x7f6ea4105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:55.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.313+0000 7f6ea9f5b640 1 -- 172.21.15.40:0/1203483089 shutdown_connections 2024-03-18T22:42:55.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.313+0000 7f6ea9f5b640 1 -- 172.21.15.40:0/1203483089 wait complete. 2024-03-18T22:42:55.325 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:55 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1512891141' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:42:55.325 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:55 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/194475718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:55.325 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:55 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1477743327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:42:55.325 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:55 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/847231503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:42:55.326 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T22:42:55.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:55 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1512891141' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:42:55.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:55 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/194475718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:55.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:55 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1477743327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:42:55.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:55 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/847231503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:42:55.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:55 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1512891141' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:42:55.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:55 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/194475718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:55.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:55 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1477743327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:42:55.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:55 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/847231503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:42:55.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.430+0000 7faa756c6640 1 -- 172.21.15.40:0/3190695027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa700ffe70 msgr2=0x7faa70100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.430+0000 7faa756c6640 1 --2- 172.21.15.40:0/3190695027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa700ffe70 0x7faa70100290 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7faa60009900 tx=0x7faa6002bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:55.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.430+0000 7faa756c6640 1 -- 172.21.15.40:0/3190695027 shutdown_connections 2024-03-18T22:42:55.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.430+0000 7faa756c6640 1 --2- 172.21.15.40:0/3190695027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa70101a10 0x7faa70108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.430+0000 7faa756c6640 1 --2- 172.21.15.40:0/3190695027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa70101050 0x7faa701014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.430+0000 7faa756c6640 1 --2- 172.21.15.40:0/3190695027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa700ffe70 0x7faa70100290 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.430+0000 7faa756c6640 1 -- 172.21.15.40:0/3190695027 >> 172.21.15.40:0/3190695027 conn(0x7faa700fb430 msgr2=0x7faa700fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:55.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.430+0000 7faa756c6640 1 -- 172.21.15.40:0/3190695027 shutdown_connections 2024-03-18T22:42:55.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.430+0000 7faa756c6640 1 -- 172.21.15.40:0/3190695027 wait complete. 2024-03-18T22:42:55.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa756c6640 1 Processor -- start 2024-03-18T22:42:55.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa756c6640 1 -- start start 2024-03-18T22:42:55.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa756c6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa700ffe70 0x7faa70103e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:55.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa756c6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa70101050 0x7faa701024c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:55.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa756c6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa70101a10 0x7faa70102a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:55.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa756c6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa7010c2b0 con 0x7faa70101050 2024-03-18T22:42:55.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa756c6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faa7010c130 con 0x7faa700ffe70 2024-03-18T22:42:55.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa756c6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7faa7010c430 con 0x7faa70101a10 2024-03-18T22:42:55.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa6e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa70101050 0x7faa701024c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:55.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa6effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa700ffe70 0x7faa70103e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:55.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa6e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa70101050 0x7faa701024c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39434/0 (socket says 172.21.15.40:39434) 2024-03-18T22:42:55.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa6e7fc640 1 -- 172.21.15.40:0/1316811177 learned_addr learned my addr 172.21.15.40:0/1316811177 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:55.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa6f7fe640 1 --2- 172.21.15.40:0/1316811177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa70101a10 0x7faa70102a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:55.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa6e7fc640 1 -- 172.21.15.40:0/1316811177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa70101a10 msgr2=0x7faa70102a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa6e7fc640 1 --2- 172.21.15.40:0/1316811177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa70101a10 0x7faa70102a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa6e7fc640 1 -- 172.21.15.40:0/1316811177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa700ffe70 msgr2=0x7faa70103e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa6e7fc640 1 --2- 172.21.15.40:0/1316811177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa700ffe70 0x7faa70103e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa6e7fc640 1 -- 172.21.15.40:0/1316811177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa701032c0 con 0x7faa70101050 2024-03-18T22:42:55.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.431+0000 7faa6f7fe640 1 --2- 172.21.15.40:0/1316811177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa70101a10 0x7faa70102a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:55.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.432+0000 7faa6e7fc640 1 --2- 172.21.15.40:0/1316811177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa70101050 0x7faa701024c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7faa6400e9f0 tx=0x7faa6400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:55.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.432+0000 7faa57fff640 1 -- 172.21.15.40:0/1316811177 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6400ccc0 con 0x7faa70101050 2024-03-18T22:42:55.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.432+0000 7faa756c6640 1 -- 172.21.15.40:0/1316811177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa701a1c50 con 0x7faa70101050 2024-03-18T22:42:55.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.432+0000 7faa57fff640 1 -- 172.21.15.40:0/1316811177 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa640108e0 con 0x7faa70101050 2024-03-18T22:42:55.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.432+0000 7faa57fff640 1 -- 172.21.15.40:0/1316811177 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa640187c0 con 0x7faa70101050 2024-03-18T22:42:55.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.432+0000 7faa756c6640 1 -- 172.21.15.40:0/1316811177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa701a2170 con 0x7faa70101050 2024-03-18T22:42:55.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.433+0000 7faa756c6640 1 -- 172.21.15.40:0/1316811177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa701a2340 con 0x7faa70101050 2024-03-18T22:42:55.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.434+0000 7faa57fff640 1 -- 172.21.15.40:0/1316811177 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7faa64010a90 con 0x7faa70101050 2024-03-18T22:42:55.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.435+0000 7faa57fff640 1 --2- 172.21.15.40:0/1316811177 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa580a2800 0x7faa580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:55.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.435+0000 7faa6effd640 1 --2- 172.21.15.40:0/1316811177 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa580a2800 0x7faa580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:55.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.435+0000 7faa57fff640 1 -- 172.21.15.40:0/1316811177 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7faa640cf690 con 0x7faa70101050 2024-03-18T22:42:55.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.435+0000 7faa6effd640 1 --2- 172.21.15.40:0/1316811177 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa580a2800 0x7faa580a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7faa6002c240 tx=0x7faa6002c7e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:55.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.439+0000 7faa57fff640 1 -- 172.21.15.40:0/1316811177 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faa6409c340 con 0x7faa70101050 2024-03-18T22:42:55.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.621+0000 7faa756c6640 1 -- 172.21.15.40:0/1316811177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7faa701a2630 con 0x7faa70101050 2024-03-18T22:42:55.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.621+0000 7faa57fff640 1 -- 172.21.15.40:0/1316811177 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7faa6409bc40 con 0x7faa70101050 2024-03-18T22:42:55.623 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:42:55.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.623+0000 7faa756c6640 1 -- 172.21.15.40:0/1316811177 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa580a2800 msgr2=0x7faa580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.623+0000 7faa756c6640 1 --2- 172.21.15.40:0/1316811177 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa580a2800 0x7faa580a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7faa6002c240 tx=0x7faa6002c7e0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.623+0000 7faa756c6640 1 -- 172.21.15.40:0/1316811177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa70101050 msgr2=0x7faa701024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.623+0000 7faa756c6640 1 --2- 172.21.15.40:0/1316811177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa70101050 0x7faa701024c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7faa6400e9f0 tx=0x7faa6400edf0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.623+0000 7faa756c6640 1 -- 172.21.15.40:0/1316811177 shutdown_connections 2024-03-18T22:42:55.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.623+0000 7faa756c6640 1 --2- 172.21.15.40:0/1316811177 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa580a2800 0x7faa580a4cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.623+0000 7faa756c6640 1 --2- 172.21.15.40:0/1316811177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa70101a10 0x7faa70102a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.623+0000 7faa756c6640 1 --2- 172.21.15.40:0/1316811177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa70101050 0x7faa701024c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.623+0000 7faa756c6640 1 --2- 172.21.15.40:0/1316811177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa700ffe70 0x7faa70103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.623+0000 7faa756c6640 1 -- 172.21.15.40:0/1316811177 >> 172.21.15.40:0/1316811177 conn(0x7faa700fb430 msgr2=0x7faa700fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:55.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.623+0000 7faa756c6640 1 -- 172.21.15.40:0/1316811177 shutdown_connections 2024-03-18T22:42:55.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.623+0000 7faa756c6640 1 -- 172.21.15.40:0/1316811177 wait complete. 2024-03-18T22:42:55.636 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:42:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.737+0000 7fbea0ff3640 1 -- 172.21.15.40:0/2781764251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9c102a60 msgr2=0x7fbe9c102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.737+0000 7fbea0ff3640 1 --2- 172.21.15.40:0/2781764251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9c102a60 0x7fbe9c102ee0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fbe84009900 tx=0x7fbe8402bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.737+0000 7fbea0ff3640 1 -- 172.21.15.40:0/2781764251 shutdown_connections 2024-03-18T22:42:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.737+0000 7fbea0ff3640 1 --2- 172.21.15.40:0/2781764251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe9c103420 0x7fbe9c109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.737+0000 7fbea0ff3640 1 --2- 172.21.15.40:0/2781764251 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9c102a60 0x7fbe9c102ee0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.737+0000 7fbea0ff3640 1 --2- 172.21.15.40:0/2781764251 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe9c101870 0x7fbe9c101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.737+0000 7fbea0ff3640 1 -- 172.21.15.40:0/2781764251 >> 172.21.15.40:0/2781764251 conn(0x7fbe9c0fcdc0 msgr2=0x7fbe9c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.737+0000 7fbea0ff3640 1 -- 172.21.15.40:0/2781764251 shutdown_connections 2024-03-18T22:42:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.737+0000 7fbea0ff3640 1 -- 172.21.15.40:0/2781764251 wait complete. 2024-03-18T22:42:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbea0ff3640 1 Processor -- start 2024-03-18T22:42:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbea0ff3640 1 -- start start 2024-03-18T22:42:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbea0ff3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe9c101870 0x7fbe9c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbea0ff3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe9c102a60 0x7fbe9c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbea0ff3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9c103420 0x7fbe9c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbea0ff3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9c10d3c0 con 0x7fbe9c103420 2024-03-18T22:42:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbea0ff3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9c10d240 con 0x7fbe9c101870 2024-03-18T22:42:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbea0ff3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9c10d540 con 0x7fbe9c102a60 2024-03-18T22:42:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbe9ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9c103420 0x7fbe9c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbe9ad9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9c103420 0x7fbe9c19f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39452/0 (socket says 172.21.15.40:39452) 2024-03-18T22:42:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbe9ad9d640 1 -- 172.21.15.40:0/1221930593 learned_addr learned my addr 172.21.15.40:0/1221930593 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbe9a59c640 1 --2- 172.21.15.40:0/1221930593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe9c101870 0x7fbe9c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbe93fff640 1 --2- 172.21.15.40:0/1221930593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe9c102a60 0x7fbe9c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbe9ad9d640 1 -- 172.21.15.40:0/1221930593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe9c102a60 msgr2=0x7fbe9c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbe9ad9d640 1 --2- 172.21.15.40:0/1221930593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe9c102a60 0x7fbe9c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbe9ad9d640 1 -- 172.21.15.40:0/1221930593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe9c101870 msgr2=0x7fbe9c197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbe9ad9d640 1 --2- 172.21.15.40:0/1221930593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe9c101870 0x7fbe9c197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbe9ad9d640 1 -- 172.21.15.40:0/1221930593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9c19f6a0 con 0x7fbe9c103420 2024-03-18T22:42:55.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.738+0000 7fbe9ad9d640 1 --2- 172.21.15.40:0/1221930593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9c103420 0x7fbe9c19f030 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400eb10 tx=0x7fbe9400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:55.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.739+0000 7fbe937fe640 1 -- 172.21.15.40:0/1221930593 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe94004b20 con 0x7fbe9c103420 2024-03-18T22:42:55.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.739+0000 7fbe937fe640 1 -- 172.21.15.40:0/1221930593 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe94004cc0 con 0x7fbe9c103420 2024-03-18T22:42:55.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.739+0000 7fbe937fe640 1 -- 172.21.15.40:0/1221930593 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe94010690 con 0x7fbe9c103420 2024-03-18T22:42:55.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.739+0000 7fbea0ff3640 1 -- 172.21.15.40:0/1221930593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe9c19f990 con 0x7fbe9c103420 2024-03-18T22:42:55.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.739+0000 7fbea0ff3640 1 -- 172.21.15.40:0/1221930593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe9c19feb0 con 0x7fbe9c103420 2024-03-18T22:42:55.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.740+0000 7fbea0ff3640 1 -- 172.21.15.40:0/1221930593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe9c108b90 con 0x7fbe9c103420 2024-03-18T22:42:55.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.741+0000 7fbe937fe640 1 -- 172.21.15.40:0/1221930593 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbe94005020 con 0x7fbe9c103420 2024-03-18T22:42:55.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.741+0000 7fbe937fe640 1 --2- 172.21.15.40:0/1221930593 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbe880a2800 0x7fbe880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:55.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.742+0000 7fbe9a59c640 1 --2- 172.21.15.40:0/1221930593 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbe880a2800 0x7fbe880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:55.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.742+0000 7fbe937fe640 1 -- 172.21.15.40:0/1221930593 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fbe940cf500 con 0x7fbe9c103420 2024-03-18T22:42:55.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.742+0000 7fbe9a59c640 1 --2- 172.21.15.40:0/1221930593 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbe880a2800 0x7fbe880a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c1028c0 tx=0x7fbe8c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:55.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.746+0000 7fbe937fe640 1 -- 172.21.15.40:0/1221930593 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbe9409c1b0 con 0x7fbe9c103420 2024-03-18T22:42:55.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.969+0000 7fbea0ff3640 1 -- 172.21.15.40:0/1221930593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe9c02d1f0 con 0x7fbe9c103420 2024-03-18T22:42:55.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.969+0000 7fbe937fe640 1 -- 172.21.15.40:0/1221930593 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11947 (secure 0 0 0) 0x7fbe9409bab0 con 0x7fbe9c103420 2024-03-18T22:42:55.971 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:55.971 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:41:09.839105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24481,24509,24473,24401,24388]},"id":1}]} 2024-03-18T22:42:55.972 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 27 2024-03-18T22:42:55.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.971+0000 7fbea0ff3640 1 -- 172.21.15.40:0/1221930593 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbe880a2800 msgr2=0x7fbe880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.971+0000 7fbea0ff3640 1 --2- 172.21.15.40:0/1221930593 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbe880a2800 0x7fbe880a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c1028c0 tx=0x7fbe8c005f90 comp rx=0 tx=0).stop 2024-03-18T22:42:55.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.971+0000 7fbea0ff3640 1 -- 172.21.15.40:0/1221930593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9c103420 msgr2=0x7fbe9c19f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:55.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.971+0000 7fbea0ff3640 1 --2- 172.21.15.40:0/1221930593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9c103420 0x7fbe9c19f030 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbe9400eb10 tx=0x7fbe9400c2d0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.972+0000 7fbea0ff3640 1 -- 172.21.15.40:0/1221930593 shutdown_connections 2024-03-18T22:42:55.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.972+0000 7fbea0ff3640 1 --2- 172.21.15.40:0/1221930593 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbe880a2800 0x7fbe880a4cc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.972+0000 7fbea0ff3640 1 --2- 172.21.15.40:0/1221930593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9c103420 0x7fbe9c19f030 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.972+0000 7fbea0ff3640 1 --2- 172.21.15.40:0/1221930593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe9c102a60 0x7fbe9c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.972+0000 7fbea0ff3640 1 --2- 172.21.15.40:0/1221930593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe9c101870 0x7fbe9c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:55.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.972+0000 7fbea0ff3640 1 -- 172.21.15.40:0/1221930593 >> 172.21.15.40:0/1221930593 conn(0x7fbe9c0fcdc0 msgr2=0x7fbe9c105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:55.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.972+0000 7fbea0ff3640 1 -- 172.21.15.40:0/1221930593 shutdown_connections 2024-03-18T22:42:55.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:55.972+0000 7fbea0ff3640 1 -- 172.21.15.40:0/1221930593 wait complete. 2024-03-18T22:42:55.984 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:42:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.087+0000 7f848b59e640 1 -- 172.21.15.40:0/2873625671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f848c101b90 msgr2=0x7f848c101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.087+0000 7f848b59e640 1 --2- 172.21.15.40:0/2873625671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f848c101b90 0x7f848c101fb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f847c009860 tx=0x7f847c02bb70 comp rx=0 tx=0).stop 2024-03-18T22:42:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.087+0000 7f848b59e640 1 -- 172.21.15.40:0/2873625671 shutdown_connections 2024-03-18T22:42:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.087+0000 7f848b59e640 1 --2- 172.21.15.40:0/2873625671 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f848c1011a0 0x7f848c108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.087+0000 7f848b59e640 1 --2- 172.21.15.40:0/2873625671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f848c1007e0 0x7f848c100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.087+0000 7f848b59e640 1 --2- 172.21.15.40:0/2873625671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f848c101b90 0x7f848c101fb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.087+0000 7f848b59e640 1 -- 172.21.15.40:0/2873625671 >> 172.21.15.40:0/2873625671 conn(0x7f848c0fb270 msgr2=0x7f848c0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.087+0000 7f848b59e640 1 -- 172.21.15.40:0/2873625671 shutdown_connections 2024-03-18T22:42:56.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.087+0000 7f848b59e640 1 -- 172.21.15.40:0/2873625671 wait complete. 2024-03-18T22:42:56.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848b59e640 1 Processor -- start 2024-03-18T22:42:56.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848b59e640 1 -- start start 2024-03-18T22:42:56.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848b59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f848c1007e0 0x7f848c196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848b59e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f848c1011a0 0x7f848c196cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848b59e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f848c101b90 0x7f848c19dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848b59e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f848c10c0c0 con 0x7f848c1007e0 2024-03-18T22:42:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848b59e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f848c10bf40 con 0x7f848c101b90 2024-03-18T22:42:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848b59e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f848c10c240 con 0x7f848c1011a0 2024-03-18T22:42:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f848c1007e0 0x7f848c196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848a59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f848c1007e0 0x7f848c196790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39482/0 (socket says 172.21.15.40:39482) 2024-03-18T22:42:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848a59c640 1 -- 172.21.15.40:0/2059503345 learned_addr learned my addr 172.21.15.40:0/2059503345 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f8489d9b640 1 --2- 172.21.15.40:0/2059503345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f848c1011a0 0x7f848c196cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848ad9d640 1 --2- 172.21.15.40:0/2059503345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f848c101b90 0x7f848c19dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848a59c640 1 -- 172.21.15.40:0/2059503345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f848c1011a0 msgr2=0x7f848c196cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848a59c640 1 --2- 172.21.15.40:0/2059503345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f848c1011a0 0x7f848c196cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848a59c640 1 -- 172.21.15.40:0/2059503345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f848c101b90 msgr2=0x7f848c19dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848a59c640 1 --2- 172.21.15.40:0/2059503345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f848c101b90 0x7f848c19dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.088+0000 7f848a59c640 1 -- 172.21.15.40:0/2059503345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f848c19e3b0 con 0x7f848c1007e0 2024-03-18T22:42:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.089+0000 7f848a59c640 1 --2- 172.21.15.40:0/2059503345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f848c1007e0 0x7f848c196790 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f847c005940 tx=0x7f847c035d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.089+0000 7f847b7fe640 1 -- 172.21.15.40:0/2059503345 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f847c004240 con 0x7f848c1007e0 2024-03-18T22:42:56.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.089+0000 7f848b59e640 1 -- 172.21.15.40:0/2059503345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f848c0ff6d0 con 0x7f848c1007e0 2024-03-18T22:42:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.089+0000 7f847b7fe640 1 -- 172.21.15.40:0/2059503345 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f847c02c9c0 con 0x7f848c1007e0 2024-03-18T22:42:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.089+0000 7f847b7fe640 1 -- 172.21.15.40:0/2059503345 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f847c003d50 con 0x7f848c1007e0 2024-03-18T22:42:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.089+0000 7f848b59e640 1 -- 172.21.15.40:0/2059503345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f848c0ffbc0 con 0x7f848c1007e0 2024-03-18T22:42:56.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.090+0000 7f848b59e640 1 -- 172.21.15.40:0/2059503345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f848c068980 con 0x7f848c1007e0 2024-03-18T22:42:56.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.091+0000 7f847b7fe640 1 -- 172.21.15.40:0/2059503345 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f847c02c510 con 0x7f848c1007e0 2024-03-18T22:42:56.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.091+0000 7f847b7fe640 1 --2- 172.21.15.40:0/2059503345 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f84740a2800 0x7f84740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.092+0000 7f8489d9b640 1 --2- 172.21.15.40:0/2059503345 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f84740a2800 0x7f84740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.092+0000 7f847b7fe640 1 -- 172.21.15.40:0/2059503345 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f847c0f2610 con 0x7f848c1007e0 2024-03-18T22:42:56.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.092+0000 7f8489d9b640 1 --2- 172.21.15.40:0/2059503345 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f84740a2800 0x7f84740a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f848c197ca0 tx=0x7f8480009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.096+0000 7f847b7fe640 1 -- 172.21.15.40:0/2059503345 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f847c0bf260 con 0x7f848c1007e0 2024-03-18T22:42:56.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.280+0000 7f848b59e640 1 -- 172.21.15.40:0/2059503345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f848c19e640 con 0x7f848c1007e0 2024-03-18T22:42:56.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.282+0000 7f847b7fe640 1 -- 172.21.15.40:0/2059503345 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24908 (secure 0 0 0) 0x7f847c0beb60 con 0x7f848c1007e0 2024-03-18T22:42:56.284 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:42:56.284 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":93,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:42:44.941873+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:42:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.284+0000 7f848b59e640 1 -- 172.21.15.40:0/2059503345 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f84740a2800 msgr2=0x7f84740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.284+0000 7f848b59e640 1 --2- 172.21.15.40:0/2059503345 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f84740a2800 0x7f84740a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f848c197ca0 tx=0x7f8480009260 comp rx=0 tx=0).stop 2024-03-18T22:42:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.284+0000 7f848b59e640 1 -- 172.21.15.40:0/2059503345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f848c1007e0 msgr2=0x7f848c196790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.284+0000 7f848b59e640 1 --2- 172.21.15.40:0/2059503345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f848c1007e0 0x7f848c196790 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f847c005940 tx=0x7f847c035d40 comp rx=0 tx=0).stop 2024-03-18T22:42:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.284+0000 7f848b59e640 1 -- 172.21.15.40:0/2059503345 shutdown_connections 2024-03-18T22:42:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.284+0000 7f848b59e640 1 --2- 172.21.15.40:0/2059503345 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f84740a2800 0x7f84740a4cc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.284+0000 7f848b59e640 1 --2- 172.21.15.40:0/2059503345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f848c101b90 0x7f848c19dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.284+0000 7f848b59e640 1 --2- 172.21.15.40:0/2059503345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f848c1011a0 0x7f848c196cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.284+0000 7f848b59e640 1 --2- 172.21.15.40:0/2059503345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f848c1007e0 0x7f848c196790 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.284+0000 7f848b59e640 1 -- 172.21.15.40:0/2059503345 >> 172.21.15.40:0/2059503345 conn(0x7f848c0fb270 msgr2=0x7f848c104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.284+0000 7f848b59e640 1 -- 172.21.15.40:0/2059503345 shutdown_connections 2024-03-18T22:42:56.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.284+0000 7f848b59e640 1 -- 172.21.15.40:0/2059503345 wait complete. 2024-03-18T22:42:56.295 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:56 smithi040 ceph-mon[62643]: pgmap v359: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 126 B/s, 1 objects/s recovering 2024-03-18T22:42:56.295 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:56 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1203483089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:42:56.295 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:56 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1316811177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:42:56.296 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:56 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1221930593' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:42:56.297 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T22:40:41.572773+0000', 'modified': '2024-03-18T22:41:09.839105+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24509, 'mds_1': 24481, 'mds_2': 24473, 'mds_3': 24401, 'mds_4': 24388}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14592': {'gid': 14592, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6839/4264194754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6838', 'nonce': 4264194754}, {'type': 'v1', 'addr': '172.21.15.40:6839', 'nonce': 4264194754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14604': {'gid': 14604, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.40:6841/1807327423', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6840', 'nonce': 1807327423}, {'type': 'v1', 'addr': '172.21.15.40:6841', 'nonce': 1807327423}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24388': {'gid': 24388, 'name': 'l', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.113:6833/2086888629', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6832', 'nonce': 2086888629}, {'type': 'v1', 'addr': '172.21.15.113:6833', 'nonce': 2086888629}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24401': {'gid': 24401, 'name': 'k', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.103:6833/3761932784', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6832', 'nonce': 3761932784}, {'type': 'v1', 'addr': '172.21.15.103:6833', 'nonce': 3761932784}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.103:6835/2536694691', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6834', 'nonce': 2536694691}, {'type': 'v1', 'addr': '172.21.15.103:6835', 'nonce': 2536694691}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.113:6835/4040610039', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6834', 'nonce': 4040610039}, {'type': 'v1', 'addr': '172.21.15.113:6835', 'nonce': 4040610039}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.113:6837/4212429112', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6836', 'nonce': 4212429112}, {'type': 'v1', 'addr': '172.21.15.113:6837', 'nonce': 4212429112}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.103:6837/1713853423', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6836', 'nonce': 1713853423}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 1713853423}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.113:6839/2986827418', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.113:6838', 'nonce': 2986827418}, {'type': 'v1', 'addr': '172.21.15.113:6839', 'nonce': 2986827418}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.103:6839/2609728202', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6838', 'nonce': 2609728202}, {'type': 'v1', 'addr': '172.21.15.103:6839', 'nonce': 2609728202}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24509, 'qdb_cluster': [24481, 24509, 24473, 24401, 24388]} 2024-03-18T22:42:56.298 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-18T22:42:56.298 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-03-18T22:42:56.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:56 smithi113 ceph-mon[66775]: pgmap v359: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 126 B/s, 1 objects/s recovering 2024-03-18T22:42:56.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:56 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1203483089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:42:56.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:56 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1316811177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:42:56.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:56 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1221930593' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:42:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:56 smithi103 ceph-mon[66377]: pgmap v359: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 126 B/s, 1 objects/s recovering 2024-03-18T22:42:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:56 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1203483089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:42:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:56 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1316811177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:42:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:56 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1221930593' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:42:56.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.400+0000 7f7bf0636640 1 -- 172.21.15.40:0/1860791366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be8100720 msgr2=0x7f7be8100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.400+0000 7f7bf0636640 1 --2- 172.21.15.40:0/1860791366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be8100720 0x7f7be8100b40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009900 tx=0x7f7be402bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:56.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.400+0000 7f7bf0636640 1 -- 172.21.15.40:0/1860791366 shutdown_connections 2024-03-18T22:42:56.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.400+0000 7f7bf0636640 1 --2- 172.21.15.40:0/1860791366 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be81022d0 0x7f7be8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.400+0000 7f7bf0636640 1 --2- 172.21.15.40:0/1860791366 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be8101910 0x7f7be8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.400+0000 7f7bf0636640 1 --2- 172.21.15.40:0/1860791366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be8100720 0x7f7be8100b40 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.400+0000 7f7bf0636640 1 -- 172.21.15.40:0/1860791366 >> 172.21.15.40:0/1860791366 conn(0x7f7be80fbcd0 msgr2=0x7f7be80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.400+0000 7f7bf0636640 1 -- 172.21.15.40:0/1860791366 shutdown_connections 2024-03-18T22:42:56.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.400+0000 7f7bf0636640 1 -- 172.21.15.40:0/1860791366 wait complete. 2024-03-18T22:42:56.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.400+0000 7f7bf0636640 1 Processor -- start 2024-03-18T22:42:56.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bf0636640 1 -- start start 2024-03-18T22:42:56.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bf0636640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be8100720 0x7f7be8197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bf0636640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be8101910 0x7f7be8197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bf0636640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be81022d0 0x7f7be819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bf0636640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7be810c220 con 0x7f7be8100720 2024-03-18T22:42:56.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bf0636640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7be810c0a0 con 0x7f7be81022d0 2024-03-18T22:42:56.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bf0636640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7be810c3a0 con 0x7f7be8101910 2024-03-18T22:42:56.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bee3d2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be8100720 0x7f7be8197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bee3d2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be8100720 0x7f7be8197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39496/0 (socket says 172.21.15.40:39496) 2024-03-18T22:42:56.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bee3d2640 1 -- 172.21.15.40:0/2722221674 learned_addr learned my addr 172.21.15.40:0/2722221674 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7beebd3640 1 --2- 172.21.15.40:0/2722221674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be81022d0 0x7f7be819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bedbd1640 1 --2- 172.21.15.40:0/2722221674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be8101910 0x7f7be8197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bee3d2640 1 -- 172.21.15.40:0/2722221674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be8101910 msgr2=0x7f7be8197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/2722221674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be8101910 0x7f7be8197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bee3d2640 1 -- 172.21.15.40:0/2722221674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be81022d0 msgr2=0x7f7be819f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/2722221674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be81022d0 0x7f7be819f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bee3d2640 1 -- 172.21.15.40:0/2722221674 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be819f6c0 con 0x7f7be8100720 2024-03-18T22:42:56.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7beebd3640 1 --2- 172.21.15.40:0/2722221674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be81022d0 0x7f7be819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:56.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/2722221674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be8100720 0x7f7be8197aa0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7be4000c00 tx=0x7f7be4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.401+0000 7f7bdf7fe640 1 -- 172.21.15.40:0/2722221674 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be4043020 con 0x7f7be8100720 2024-03-18T22:42:56.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.402+0000 7f7bdf7fe640 1 -- 172.21.15.40:0/2722221674 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7be4004350 con 0x7f7be8100720 2024-03-18T22:42:56.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.402+0000 7f7bf0636640 1 -- 172.21.15.40:0/2722221674 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be819f950 con 0x7f7be8100720 2024-03-18T22:42:56.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.402+0000 7f7bdf7fe640 1 -- 172.21.15.40:0/2722221674 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be4003b80 con 0x7f7be8100720 2024-03-18T22:42:56.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.402+0000 7f7bf0636640 1 -- 172.21.15.40:0/2722221674 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be819fea0 con 0x7f7be8100720 2024-03-18T22:42:56.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.403+0000 7f7bdf7fe640 1 -- 172.21.15.40:0/2722221674 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7be4046020 con 0x7f7be8100720 2024-03-18T22:42:56.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.404+0000 7f7bf3225740 1 Processor -- start 2024-03-18T22:42:56.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.404+0000 7f7bf3225740 1 -- start start 2024-03-18T22:42:56.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.404+0000 7f7bf3225740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f268ff0 0x55790f269410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.404+0000 7f7bf3225740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 0x55790f26e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.404+0000 7f7bf3225740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f26eb50 0x55790f270fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.404+0000 7f7bf3225740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55790f1cdde0 con 0x55790f26eb50 2024-03-18T22:42:56.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.404+0000 7f7bf3225740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55790f176c80 con 0x55790f268ff0 2024-03-18T22:42:56.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.404+0000 7f7bf3225740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55790f1c9e80 con 0x55790f2699e0 2024-03-18T22:42:56.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.404+0000 7f7bdf7fe640 1 --2- 172.21.15.40:0/2722221674 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7bd40a2800 0x7f7bd40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7bee3d2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 0x55790f26e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7bedbd1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f26eb50 0x55790f270fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7beebd3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f268ff0 0x55790f269410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7bedbd1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f26eb50 0x55790f270fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39500/0 (socket says 172.21.15.40:39500) 2024-03-18T22:42:56.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7bedbd1640 1 -- 172.21.15.40:0/2456844654 learned_addr learned my addr 172.21.15.40:0/2456844654 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7beebd3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f268ff0 0x55790f269410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:44548/0 (socket says 172.21.15.40:44548) 2024-03-18T22:42:56.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7beebd3640 1 --2- 172.21.15.40:0/2722221674 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7bd40a2800 0x7f7bd40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7bdf7fe640 1 -- 172.21.15.40:0/2722221674 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7be40f0fd0 con 0x7f7be8100720 2024-03-18T22:42:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7bee3d2640 1 -- 172.21.15.40:0/2456844654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f268ff0 msgr2=0x55790f269410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/2456844654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f268ff0 0x55790f269410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7bee3d2640 1 -- 172.21.15.40:0/2456844654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f26eb50 msgr2=0x55790f270fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/2456844654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f26eb50 0x55790f270fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7bee3d2640 1 -- 172.21.15.40:0/2456844654 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55790f188fb0 con 0x55790f2699e0 2024-03-18T22:42:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7beebd3640 1 --2- 172.21.15.40:0/2722221674 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7bd40a2800 0x7f7bd40a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7be0005fd0 tx=0x7f7be0011810 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/2456844654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 0x55790f26e4e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7be40ee650 tx=0x7f7be40e7f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81eb910999521a0e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7bdcbfe640 1 -- 172.21.15.40:0/2456844654 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be40bd2f0 con 0x55790f2699e0 2024-03-18T22:42:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7bdcbfe640 1 -- 172.21.15.40:0/2456844654 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7be40bdc20 con 0x55790f2699e0 2024-03-18T22:42:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.405+0000 7f7bdcbfe640 1 -- 172.21.15.40:0/2456844654 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be40bce80 con 0x55790f2699e0 2024-03-18T22:42:56.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.406+0000 7f7bf3225740 1 -- 172.21.15.40:0/2456844654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 msgr2=0x55790f26e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.406+0000 7f7bf3225740 1 --2- 172.21.15.40:0/2456844654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 0x55790f26e4e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7be40ee650 tx=0x7f7be40e7f70 comp rx=0 tx=0).stop 2024-03-18T22:42:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.406+0000 7f7bf3225740 1 -- 172.21.15.40:0/2456844654 shutdown_connections 2024-03-18T22:42:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.406+0000 7f7bf3225740 1 --2- 172.21.15.40:0/2456844654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f26eb50 0x55790f270fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.406+0000 7f7bf3225740 1 --2- 172.21.15.40:0/2456844654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 0x55790f26e4e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.406+0000 7f7bf3225740 1 --2- 172.21.15.40:0/2456844654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f268ff0 0x55790f269410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.406+0000 7f7bf3225740 1 -- 172.21.15.40:0/2456844654 >> 172.21.15.40:0/2456844654 conn(0x55790f264560 msgr2=0x55790f264960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.406+0000 7f7bf3225740 1 -- 172.21.15.40:0/2456844654 shutdown_connections 2024-03-18T22:42:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.406+0000 7f7bf3225740 1 -- 172.21.15.40:0/2456844654 wait complete. 2024-03-18T22:42:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.406+0000 7f7bf3225740 1 -- start start 2024-03-18T22:42:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bf3225740 1 Processor -- start 2024-03-18T22:42:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bf3225740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f268ff0 0x55790f273250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bf3225740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f2699e0 0x55790f273790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bf3225740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f26eb50 0x55790f27a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bf3225740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55790f2718c0 con 0x55790f2699e0 2024-03-18T22:42:56.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bf3225740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55790f189fc0 con 0x55790f26eb50 2024-03-18T22:42:56.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bf3225740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55790f271a40 con 0x55790f268ff0 2024-03-18T22:42:56.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bf3225740 10 client.0 _ll_register_callbacks cb 0x55790f1c5a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:56.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bedbd1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f2699e0 0x55790f273790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bedbd1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f2699e0 0x55790f273790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39516/0 (socket says 172.21.15.40:39516) 2024-03-18T22:42:56.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bedbd1640 1 -- 172.21.15.40:0/2318375404 learned_addr learned my addr 172.21.15.40:0/2318375404 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/2318375404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f268ff0 0x55790f273250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7beebd3640 1 --2- 172.21.15.40:0/2318375404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f26eb50 0x55790f27a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bedbd1640 1 -- 172.21.15.40:0/2318375404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f268ff0 msgr2=0x55790f273250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bedbd1640 1 --2- 172.21.15.40:0/2318375404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f268ff0 0x55790f273250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bedbd1640 1 -- 172.21.15.40:0/2318375404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f26eb50 msgr2=0x55790f27a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bedbd1640 1 --2- 172.21.15.40:0/2318375404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f26eb50 0x55790f27a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bedbd1640 1 -- 172.21.15.40:0/2318375404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55790eee22d0 con 0x55790f2699e0 2024-03-18T22:42:56.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/2318375404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f268ff0 0x55790f273250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:56.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bedbd1640 1 --2- 172.21.15.40:0/2318375404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f2699e0 0x55790f273790 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8012760 tx=0x7f7bd8012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbaacc3525eaf45 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.407+0000 7f7bd1ffb640 1 -- 172.21.15.40:0/2318375404 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8004020 con 0x55790f2699e0 2024-03-18T22:42:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.408+0000 7f7bf3225740 1 -- 172.21.15.40:0/2318375404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7be40bdc20 con 0x55790f2699e0 2024-03-18T22:42:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.408+0000 7f7bd1ffb640 1 -- 172.21.15.40:0/2318375404 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7bd80041c0 con 0x55790f2699e0 2024-03-18T22:42:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.408+0000 7f7bd1ffb640 1 -- 172.21.15.40:0/2318375404 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7bd8025050 con 0x55790f2699e0 2024-03-18T22:42:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.408+0000 7f7bf3225740 10 client.15399 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.408+0000 7f7bf3225740 1 -- 172.21.15.40:0/2318375404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55790efac6b0 con 0x55790f2699e0 2024-03-18T22:42:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.408+0000 7f7bd1ffb640 1 -- 172.21.15.40:0/2318375404 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7be40bdc20 con 0x55790f2699e0 2024-03-18T22:42:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.408+0000 7f7bd1ffb640 1 -- 172.21.15.40:0/2318375404 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f7bd80038e0 con 0x55790f2699e0 2024-03-18T22:42:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.408+0000 7f7bf3225740 10 client.15399 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.408+0000 7f7bf3225740 10 client.15399 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-18T22:42:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.408+0000 7f7bf3225740 20 client.15399 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:56.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.408+0000 7f7bf3225740 1 --2- 172.21.15.40:0/2318375404 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55790f280110 0x55790f281580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.408+0000 7f7bf3225740 4 client.15399 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:42:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.408+0000 7f7bf3225740 1 -- 172.21.15.40:0/2318375404 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55790f1db320 con 0x55790f280110 2024-03-18T22:42:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.408+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/2318375404 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55790f280110 0x55790f281580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.409+0000 7f7bd1ffb640 10 client.15399 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:42:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.409+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/2318375404 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55790f280110 0x55790f281580 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=884ee0f9c3ef0bfb server_cookie=7899c5d50cb0f285 in_seq=0 out_seq=0 2024-03-18T22:42:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.409+0000 7f7bd1ffb640 1 -- 172.21.15.40:0/2318375404 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55790f1db320 con 0x55790f280110 2024-03-18T22:42:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.409+0000 7f7bd1ffb640 10 client.15399 handle_command_reply: tid=0 2024-03-18T22:42:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.409+0000 7f7bf3225740 1 client.15399 shutdown 2024-03-18T22:42:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.410+0000 7f7bf3225740 1 -- 172.21.15.40:0/2318375404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f2699e0 msgr2=0x55790f273790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.410+0000 7f7bf3225740 1 --2- 172.21.15.40:0/2318375404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f2699e0 0x55790f273790 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8012760 tx=0x7f7bd8012b60 comp rx=0 tx=0).stop 2024-03-18T22:42:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.410+0000 7f7bf3225740 1 -- 172.21.15.40:0/2318375404 shutdown_connections 2024-03-18T22:42:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.410+0000 7f7bf3225740 1 --2- 172.21.15.40:0/2318375404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f26eb50 0x55790f27a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.410+0000 7f7bf3225740 1 --2- 172.21.15.40:0/2318375404 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55790f280110 0x55790f281580 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.410+0000 7f7bf3225740 1 --2- 172.21.15.40:0/2318375404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f2699e0 0x55790f273790 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.410+0000 7f7bd1ffb640 0 client.15399 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:42:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.410+0000 7f7bf3225740 1 --2- 172.21.15.40:0/2318375404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f268ff0 0x55790f273250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.410+0000 7f7bf3225740 1 -- 172.21.15.40:0/2318375404 >> 172.21.15.40:0/2318375404 conn(0x55790f264560 msgr2=0x55790f2637e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.410+0000 7f7bf3225740 1 -- 172.21.15.40:0/2318375404 shutdown_connections 2024-03-18T22:42:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.410+0000 7f7bf3225740 1 -- 172.21.15.40:0/2318375404 wait complete. 2024-03-18T22:42:56.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.410+0000 7f7bf3225740 20 client.15399 trim_cache size 0 max 16384 2024-03-18T22:42:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bf3225740 1 Processor -- start 2024-03-18T22:42:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bf3225740 1 -- start start 2024-03-18T22:42:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bf3225740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f268ff0 0x55790f280eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bf3225740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 0x55790f27a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bf3225740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f26eb50 0x55790f264940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bf3225740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55790f27f8c0 con 0x55790f268ff0 2024-03-18T22:42:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bf3225740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55790f27ff90 con 0x55790f26eb50 2024-03-18T22:42:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bf3225740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55790f280660 con 0x55790f2699e0 2024-03-18T22:42:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bee3d2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f268ff0 0x55790f280eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bee3d2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f268ff0 0x55790f280eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39526/0 (socket says 172.21.15.40:39526) 2024-03-18T22:42:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bee3d2640 1 -- 172.21.15.40:0/3954901149 learned_addr learned my addr 172.21.15.40:0/3954901149 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7beebd3640 1 --2- 172.21.15.40:0/3954901149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f26eb50 0x55790f264940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bedbd1640 1 --2- 172.21.15.40:0/3954901149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 0x55790f27a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bee3d2640 1 -- 172.21.15.40:0/3954901149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 msgr2=0x55790f27a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/3954901149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 0x55790f27a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bee3d2640 1 -- 172.21.15.40:0/3954901149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f26eb50 msgr2=0x55790f264940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/3954901149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f26eb50 0x55790f264940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bee3d2640 1 -- 172.21.15.40:0/3954901149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55790f26e180 con 0x55790f268ff0 2024-03-18T22:42:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/3954901149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f268ff0 0x55790f280eb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7be40b9980 tx=0x7f7be40af040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8fdc585d5727c46 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bd3fff640 1 -- 172.21.15.40:0/3954901149 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be40bd060 con 0x55790f268ff0 2024-03-18T22:42:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bd3fff640 1 -- 172.21.15.40:0/3954901149 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7be40bdc20 con 0x55790f268ff0 2024-03-18T22:42:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.428+0000 7f7bd3fff640 1 -- 172.21.15.40:0/3954901149 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be4003d20 con 0x55790f268ff0 2024-03-18T22:42:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.429+0000 7f7bf3225740 1 -- 172.21.15.40:0/3954901149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f268ff0 msgr2=0x55790f280eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.429+0000 7f7bf3225740 1 --2- 172.21.15.40:0/3954901149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f268ff0 0x55790f280eb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7be40b9980 tx=0x7f7be40af040 comp rx=0 tx=0).stop 2024-03-18T22:42:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.429+0000 7f7bf3225740 1 -- 172.21.15.40:0/3954901149 shutdown_connections 2024-03-18T22:42:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.429+0000 7f7bf3225740 1 --2- 172.21.15.40:0/3954901149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f26eb50 0x55790f264940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.429+0000 7f7bf3225740 1 --2- 172.21.15.40:0/3954901149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 0x55790f27a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.429+0000 7f7bf3225740 1 --2- 172.21.15.40:0/3954901149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f268ff0 0x55790f280eb0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.429+0000 7f7bf3225740 1 -- 172.21.15.40:0/3954901149 >> 172.21.15.40:0/3954901149 conn(0x55790f264560 msgr2=0x55790f26a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.429+0000 7f7bf3225740 1 -- 172.21.15.40:0/3954901149 shutdown_connections 2024-03-18T22:42:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.429+0000 7f7bf3225740 1 -- 172.21.15.40:0/3954901149 wait complete. 2024-03-18T22:42:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.429+0000 7f7bf3225740 1 -- start start 2024-03-18T22:42:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.429+0000 7f7bf3225740 1 Processor -- start 2024-03-18T22:42:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bf3225740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f268ff0 0x55790f275b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bf3225740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 0x55790f276070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bf3225740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f26eb50 0x55790f279090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bf3225740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55790f281400 con 0x55790f26eb50 2024-03-18T22:42:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bf3225740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55790f280d30 con 0x55790f268ff0 2024-03-18T22:42:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bf3225740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55790f281ad0 con 0x55790f2699e0 2024-03-18T22:42:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bf3225740 10 client.0 _ll_register_callbacks cb 0x7f7bd8009f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7beebd3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f26eb50 0x55790f279090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7beebd3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f26eb50 0x55790f279090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39540/0 (socket says 172.21.15.40:39540) 2024-03-18T22:42:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7beebd3640 1 -- 172.21.15.40:0/871921705 learned_addr learned my addr 172.21.15.40:0/871921705 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bedbd1640 1 --2- 172.21.15.40:0/871921705 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 0x55790f276070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/871921705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f268ff0 0x55790f275b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7beebd3640 1 -- 172.21.15.40:0/871921705 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 msgr2=0x55790f276070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7beebd3640 1 --2- 172.21.15.40:0/871921705 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 0x55790f276070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7beebd3640 1 -- 172.21.15.40:0/871921705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f268ff0 msgr2=0x55790f275b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7beebd3640 1 --2- 172.21.15.40:0/871921705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f268ff0 0x55790f275b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7beebd3640 1 -- 172.21.15.40:0/871921705 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55790f279680 con 0x55790f26eb50 2024-03-18T22:42:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7beebd3640 1 --2- 172.21.15.40:0/871921705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f26eb50 0x55790f279090 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7be0003e70 tx=0x7f7be001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3a6d77aee9824d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bd1ffb640 1 -- 172.21.15.40:0/871921705 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be0011220 con 0x55790f26eb50 2024-03-18T22:42:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bd1ffb640 1 -- 172.21.15.40:0/871921705 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7be0004b20 con 0x55790f26eb50 2024-03-18T22:42:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bf3225740 1 -- 172.21.15.40:0/871921705 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7be40bdc20 con 0x55790f26eb50 2024-03-18T22:42:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bd1ffb640 1 -- 172.21.15.40:0/871921705 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7be0036050 con 0x55790f26eb50 2024-03-18T22:42:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bf3225740 10 client.15405 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.430+0000 7f7bf3225740 1 -- 172.21.15.40:0/871921705 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55790f27a9f0 con 0x55790f26eb50 2024-03-18T22:42:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.431+0000 7f7bd1ffb640 1 -- 172.21.15.40:0/871921705 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7be40bdc20 con 0x55790f26eb50 2024-03-18T22:42:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.431+0000 7f7bd1ffb640 1 -- 172.21.15.40:0/871921705 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f7be0005020 con 0x55790f26eb50 2024-03-18T22:42:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.431+0000 7f7bf3225740 10 client.15405 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.431+0000 7f7bf3225740 10 client.15405 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-18T22:42:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.431+0000 7f7bf3225740 20 client.15405 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.431+0000 7f7bf3225740 1 --2- 172.21.15.40:0/871921705 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55790f2ea800 0x55790f2e9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.431+0000 7f7bf3225740 4 client.15405 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T22:42:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.431+0000 7f7bf3225740 1 -- 172.21.15.40:0/871921705 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "status"}) -- 0x55790f1d5670 con 0x55790f2ea800 2024-03-18T22:42:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.431+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/871921705 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55790f2ea800 0x55790f2e9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.431+0000 7f7bd1ffb640 10 client.15405 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:42:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.431+0000 7f7bee3d2640 1 --2- 172.21.15.40:0/871921705 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55790f2ea800 0x55790f2e9510 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=338d7215c844f48a server_cookie=237260c0c69320cd in_seq=0 out_seq=0 2024-03-18T22:42:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bd1ffb640 1 -- 172.21.15.40:0/871921705 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55790f1d5670 con 0x55790f2ea800 2024-03-18T22:42:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bd1ffb640 10 client.15405 handle_command_reply: tid=0 2024-03-18T22:42:56.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf3225740 1 client.15405 shutdown 2024-03-18T22:42:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf3225740 1 -- 172.21.15.40:0/871921705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f26eb50 msgr2=0x55790f279090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf3225740 1 --2- 172.21.15.40:0/871921705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f26eb50 0x55790f279090 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7be0003e70 tx=0x7f7be001d000 comp rx=0 tx=0).stop 2024-03-18T22:42:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf3225740 1 -- 172.21.15.40:0/871921705 shutdown_connections 2024-03-18T22:42:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf3225740 1 --2- 172.21.15.40:0/871921705 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55790f2ea800 0x55790f2e9510 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf3225740 1 --2- 172.21.15.40:0/871921705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55790f26eb50 0x55790f279090 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf3225740 1 --2- 172.21.15.40:0/871921705 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55790f2699e0 0x55790f276070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bd1ffb640 0 client.15405 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:42:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf3225740 1 --2- 172.21.15.40:0/871921705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55790f268ff0 0x55790f275b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf3225740 1 -- 172.21.15.40:0/871921705 >> 172.21.15.40:0/871921705 conn(0x55790f264560 msgr2=0x55790f26aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf3225740 1 -- 172.21.15.40:0/871921705 shutdown_connections 2024-03-18T22:42:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf3225740 1 -- 172.21.15.40:0/871921705 wait complete. 2024-03-18T22:42:56.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf3225740 20 client.15405 trim_cache size 0 max 16384 2024-03-18T22:42:56.445 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:42:56.445 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "410ef414-e577-11ee-95c9-87774f69a715", 2024-03-18T22:42:56.445 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 4, 2024-03-18T22:42:56.445 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24388, 2024-03-18T22:42:56.446 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-18T22:42:56.446 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-18T22:42:56.446 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-18T22:42:56.446 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 107.602289856, 2024-03-18T22:42:56.446 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 27, 2024-03-18T22:42:56.447 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 93, 2024-03-18T22:42:56.447 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T22:42:56.447 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 169.986890082 2024-03-18T22:42:56.447 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:42:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf0636640 1 -- 172.21.15.40:0/2722221674 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7bd40a2800 msgr2=0x7f7bd40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf0636640 1 --2- 172.21.15.40:0/2722221674 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7bd40a2800 0x7f7bd40a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7be0005fd0 tx=0x7f7be0011810 comp rx=0 tx=0).stop 2024-03-18T22:42:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf0636640 1 -- 172.21.15.40:0/2722221674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be8100720 msgr2=0x7f7be8197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf0636640 1 --2- 172.21.15.40:0/2722221674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be8100720 0x7f7be8197aa0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7be4000c00 tx=0x7f7be4005dc0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf0636640 1 -- 172.21.15.40:0/2722221674 shutdown_connections 2024-03-18T22:42:56.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf0636640 1 --2- 172.21.15.40:0/2722221674 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7bd40a2800 0x7f7bd40a4cc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf0636640 1 --2- 172.21.15.40:0/2722221674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be81022d0 0x7f7be819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf0636640 1 --2- 172.21.15.40:0/2722221674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7be8101910 0x7f7be8197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf0636640 1 --2- 172.21.15.40:0/2722221674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7be8100720 0x7f7be8197aa0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf0636640 1 -- 172.21.15.40:0/2722221674 >> 172.21.15.40:0/2722221674 conn(0x7f7be80fbcd0 msgr2=0x7f7be8069f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf0636640 1 -- 172.21.15.40:0/2722221674 shutdown_connections 2024-03-18T22:42:56.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.432+0000 7f7bf0636640 1 -- 172.21.15.40:0/2722221674 wait complete. 2024-03-18T22:42:56.452 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-03-18T22:42:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.556+0000 7f677813b640 1 -- 172.21.15.40:0/174413053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770101a10 msgr2=0x7f6770108330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.556+0000 7f677813b640 1 --2- 172.21.15.40:0/174413053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770101a10 0x7f6770108330 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6760009ca0 tx=0x7f676002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.556+0000 7f677813b640 1 -- 172.21.15.40:0/174413053 shutdown_connections 2024-03-18T22:42:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.556+0000 7f677813b640 1 --2- 172.21.15.40:0/174413053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770101a10 0x7f6770108330 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.556+0000 7f677813b640 1 --2- 172.21.15.40:0/174413053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6770101050 0x7f67701014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.556+0000 7f677813b640 1 --2- 172.21.15.40:0/174413053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700ffe70 0x7f6770100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.556+0000 7f677813b640 1 -- 172.21.15.40:0/174413053 >> 172.21.15.40:0/174413053 conn(0x7f67700fb450 msgr2=0x7f67700fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.556+0000 7f677813b640 1 -- 172.21.15.40:0/174413053 shutdown_connections 2024-03-18T22:42:56.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.556+0000 7f677813b640 1 -- 172.21.15.40:0/174413053 wait complete. 2024-03-18T22:42:56.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.556+0000 7f677813b640 1 Processor -- start 2024-03-18T22:42:56.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.556+0000 7f677813b640 1 -- start start 2024-03-18T22:42:56.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f677813b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700ffe70 0x7f6770109580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f677813b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770101050 0x7f677019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f677813b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6770101a10 0x7f677019ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f677813b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f677010c270 con 0x7f6770101050 2024-03-18T22:42:56.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f677813b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f677010c0f0 con 0x7f6770101a10 2024-03-18T22:42:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f677813b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f677010c3f0 con 0x7f67700ffe70 2024-03-18T22:42:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f67756d6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770101050 0x7f677019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f67756d6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770101050 0x7f677019c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39558/0 (socket says 172.21.15.40:39558) 2024-03-18T22:42:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f67756d6640 1 -- 172.21.15.40:0/1279629166 learned_addr learned my addr 172.21.15.40:0/1279629166 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f6775ed7640 1 --2- 172.21.15.40:0/1279629166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700ffe70 0x7f6770109580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f67766d8640 1 --2- 172.21.15.40:0/1279629166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6770101a10 0x7f677019ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f67756d6640 1 -- 172.21.15.40:0/1279629166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700ffe70 msgr2=0x7f6770109580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f67756d6640 1 --2- 172.21.15.40:0/1279629166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700ffe70 0x7f6770109580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f67756d6640 1 -- 172.21.15.40:0/1279629166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6770101a10 msgr2=0x7f677019ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f67756d6640 1 --2- 172.21.15.40:0/1279629166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6770101a10 0x7f677019ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f67756d6640 1 -- 172.21.15.40:0/1279629166 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f677019f620 con 0x7f6770101050 2024-03-18T22:42:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f67756d6640 1 --2- 172.21.15.40:0/1279629166 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770101050 0x7f677019c720 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f676400e9f0 tx=0x7f676400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f676effd640 1 -- 172.21.15.40:0/1279629166 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676400cd80 con 0x7f6770101050 2024-03-18T22:42:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f676effd640 1 -- 172.21.15.40:0/1279629166 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67640108e0 con 0x7f6770101050 2024-03-18T22:42:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f677813b640 1 -- 172.21.15.40:0/1279629166 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f677019f910 con 0x7f6770101050 2024-03-18T22:42:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f676effd640 1 -- 172.21.15.40:0/1279629166 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764018920 con 0x7f6770101050 2024-03-18T22:42:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.557+0000 7f677813b640 1 -- 172.21.15.40:0/1279629166 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f677019fe60 con 0x7f6770101050 2024-03-18T22:42:56.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.559+0000 7f676effd640 1 -- 172.21.15.40:0/1279629166 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6764010a90 con 0x7f6770101050 2024-03-18T22:42:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.560+0000 7f677ae0a740 1 Processor -- start 2024-03-18T22:42:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.560+0000 7f677ae0a740 1 -- start start 2024-03-18T22:42:56.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.560+0000 7f676effd640 1 --2- 172.21.15.40:0/1279629166 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f67580a2800 0x7f67580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.560+0000 7f677ae0a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a0d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.560+0000 7f677ae0a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a124e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.560+0000 7f676effd640 1 -- 172.21.15.40:0/1279629166 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f67640cf140 con 0x7f6770101050 2024-03-18T22:42:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.560+0000 7f677ae0a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 0x55bcd9a14fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.560+0000 7f677ae0a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bcd9971de0 con 0x55bcd9a0cff0 2024-03-18T22:42:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.560+0000 7f67766d8640 1 --2- 172.21.15.40:0/1279629166 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f67580a2800 0x7f67580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.560+0000 7f677ae0a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bcd991ac80 con 0x55bcd9a0d9e0 2024-03-18T22:42:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.560+0000 7f677ae0a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55bcd996de80 con 0x55bcd9a12b50 2024-03-18T22:42:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.560+0000 7f6775ed7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a0d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.560+0000 7f6775ed7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a0d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39562/0 (socket says 172.21.15.40:39562) 2024-03-18T22:42:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.560+0000 7f6775ed7640 1 -- 172.21.15.40:0/149844094 learned_addr learned my addr 172.21.15.40:0/149844094 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.561+0000 7f67756d6640 1 --2- 172.21.15.40:0/149844094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a124e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.561+0000 7f67766d8640 1 --2- 172.21.15.40:0/149844094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 0x55bcd9a14fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.561+0000 7f6775ed7640 1 -- 172.21.15.40:0/149844094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 msgr2=0x55bcd9a14fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.561+0000 7f6775ed7640 1 --2- 172.21.15.40:0/149844094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 0x55bcd9a14fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.561+0000 7f6775ed7640 1 -- 172.21.15.40:0/149844094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a0d9e0 msgr2=0x55bcd9a124e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.561+0000 7f6775ed7640 1 --2- 172.21.15.40:0/149844094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a124e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.561+0000 7f6775ed7640 1 -- 172.21.15.40:0/149844094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcd992cfb0 con 0x55bcd9a0cff0 2024-03-18T22:42:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.561+0000 7f67766d8640 1 --2- 172.21.15.40:0/1279629166 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f67580a2800 0x7f67580a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6760006010 tx=0x7f6760032ea0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.561+0000 7f6775ed7640 1 --2- 172.21.15.40:0/149844094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a0d410 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6768009c80 tx=0x7f6768011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=304a3e687366388a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.561+0000 7f6757fff640 1 -- 172.21.15.40:0/149844094 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676801e070 con 0x55bcd9a0cff0 2024-03-18T22:42:56.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.561+0000 7f6757fff640 1 -- 172.21.15.40:0/149844094 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6768009e30 con 0x55bcd9a0cff0 2024-03-18T22:42:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.561+0000 7f6757fff640 1 -- 172.21.15.40:0/149844094 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676800ea10 con 0x55bcd9a0cff0 2024-03-18T22:42:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.562+0000 7f677ae0a740 1 -- 172.21.15.40:0/149844094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 msgr2=0x55bcd9a0d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.562+0000 7f677ae0a740 1 --2- 172.21.15.40:0/149844094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a0d410 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6768009c80 tx=0x7f6768011810 comp rx=0 tx=0).stop 2024-03-18T22:42:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.562+0000 7f677ae0a740 1 -- 172.21.15.40:0/149844094 shutdown_connections 2024-03-18T22:42:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.562+0000 7f677ae0a740 1 --2- 172.21.15.40:0/149844094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 0x55bcd9a14fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.562+0000 7f677ae0a740 1 --2- 172.21.15.40:0/149844094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a124e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.562+0000 7f677ae0a740 1 --2- 172.21.15.40:0/149844094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a0d410 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.562+0000 7f677ae0a740 1 -- 172.21.15.40:0/149844094 >> 172.21.15.40:0/149844094 conn(0x55bcd9a08560 msgr2=0x55bcd9a08960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.562+0000 7f677ae0a740 1 -- 172.21.15.40:0/149844094 shutdown_connections 2024-03-18T22:42:56.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.562+0000 7f677ae0a740 1 -- 172.21.15.40:0/149844094 wait complete. 2024-03-18T22:42:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.562+0000 7f677ae0a740 1 -- start start 2024-03-18T22:42:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.562+0000 7f677ae0a740 1 Processor -- start 2024-03-18T22:42:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f677ae0a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a174b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f677ae0a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a179f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f677ae0a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 0x55bcd9a1ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f677ae0a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bcd9a15b00 con 0x55bcd9a0cff0 2024-03-18T22:42:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f677ae0a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bcd992dfc0 con 0x55bcd9a0d9e0 2024-03-18T22:42:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f677ae0a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55bcd9a15c80 con 0x55bcd9a12b50 2024-03-18T22:42:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f677ae0a740 10 client.0 _ll_register_callbacks cb 0x55bcd9969a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:56.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f6775ed7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a174b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f6775ed7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a174b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39574/0 (socket says 172.21.15.40:39574) 2024-03-18T22:42:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f6775ed7640 1 -- 172.21.15.40:0/2941708193 learned_addr learned my addr 172.21.15.40:0/2941708193 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f67756d6640 1 --2- 172.21.15.40:0/2941708193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a179f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f67766d8640 1 --2- 172.21.15.40:0/2941708193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 0x55bcd9a1ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f6775ed7640 1 -- 172.21.15.40:0/2941708193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 msgr2=0x55bcd9a1ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f6775ed7640 1 --2- 172.21.15.40:0/2941708193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 0x55bcd9a1ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f6775ed7640 1 -- 172.21.15.40:0/2941708193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a0d9e0 msgr2=0x55bcd9a179f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f6775ed7640 1 --2- 172.21.15.40:0/2941708193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a179f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f6775ed7640 1 -- 172.21.15.40:0/2941708193 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcd96862d0 con 0x55bcd9a0cff0 2024-03-18T22:42:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f67766d8640 1 --2- 172.21.15.40:0/2941708193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 0x55bcd9a1ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:42:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f6775ed7640 1 --2- 172.21.15.40:0/2941708193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a174b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f67680097b0 tx=0x7f6768005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe326d104866cf26 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f67557fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:56.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f67557fa640 1 -- 172.21.15.40:0/2941708193 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676801e040 con 0x55bcd9a0cff0 2024-03-18T22:42:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f67557fa640 1 -- 172.21.15.40:0/2941708193 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6768004230 con 0x55bcd9a0cff0 2024-03-18T22:42:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f677ae0a740 1 -- 172.21.15.40:0/2941708193 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6768009e30 con 0x55bcd9a0cff0 2024-03-18T22:42:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f67557fa640 1 -- 172.21.15.40:0/2941708193 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6768032050 con 0x55bcd9a0cff0 2024-03-18T22:42:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.563+0000 7f677ae0a740 10 client.15417 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.564+0000 7f677ae0a740 1 -- 172.21.15.40:0/2941708193 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bcd97506b0 con 0x55bcd9a0cff0 2024-03-18T22:42:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.564+0000 7f67557fa640 1 -- 172.21.15.40:0/2941708193 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6768009e30 con 0x55bcd9a0cff0 2024-03-18T22:42:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.564+0000 7f67557fa640 1 -- 172.21.15.40:0/2941708193 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6768021030 con 0x55bcd9a0cff0 2024-03-18T22:42:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.564+0000 7f677ae0a740 10 client.15417 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.564+0000 7f677ae0a740 10 client.15417 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-18T22:42:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.564+0000 7f677ae0a740 20 client.15417 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.564+0000 7f677ae0a740 1 --2- 172.21.15.40:0/2941708193 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bcd9a24020 0x55bcd9a25490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.564+0000 7f677ae0a740 4 client.15417 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:42:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.564+0000 7f677ae0a740 1 -- 172.21.15.40:0/2941708193 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcd997f320 con 0x55bcd9a24020 2024-03-18T22:42:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.564+0000 7f67756d6640 1 --2- 172.21.15.40:0/2941708193 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bcd9a24020 0x55bcd9a25490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.565+0000 7f67557fa640 10 client.15417 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:42:56.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.565+0000 7f67756d6640 1 --2- 172.21.15.40:0/2941708193 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bcd9a24020 0x55bcd9a25490 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c72055b01b35319e server_cookie=56f3223f8ccfc4a4 in_seq=0 out_seq=0 2024-03-18T22:42:56.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.565+0000 7f67557fa640 1 -- 172.21.15.40:0/2941708193 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bcd997f320 con 0x55bcd9a24020 2024-03-18T22:42:56.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.565+0000 7f67557fa640 10 client.15417 handle_command_reply: tid=0 2024-03-18T22:42:56.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.565+0000 7f677ae0a740 1 client.15417 shutdown 2024-03-18T22:42:56.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.565+0000 7f677ae0a740 1 -- 172.21.15.40:0/2941708193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 msgr2=0x55bcd9a174b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.565+0000 7f677ae0a740 1 --2- 172.21.15.40:0/2941708193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a174b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f67680097b0 tx=0x7f6768005720 comp rx=0 tx=0).stop 2024-03-18T22:42:56.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.565+0000 7f677ae0a740 1 -- 172.21.15.40:0/2941708193 shutdown_connections 2024-03-18T22:42:56.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.565+0000 7f677ae0a740 1 --2- 172.21.15.40:0/2941708193 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bcd9a24020 0x55bcd9a25490 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.565+0000 7f677ae0a740 1 --2- 172.21.15.40:0/2941708193 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 0x55bcd9a1ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.565+0000 7f677ae0a740 1 --2- 172.21.15.40:0/2941708193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a179f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.565+0000 7f67557fa640 0 client.15417 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:42:56.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.565+0000 7f677ae0a740 1 --2- 172.21.15.40:0/2941708193 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a174b0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.565+0000 7f677ae0a740 1 -- 172.21.15.40:0/2941708193 >> 172.21.15.40:0/2941708193 conn(0x55bcd9a08560 msgr2=0x55bcd9a077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.565+0000 7f677ae0a740 1 -- 172.21.15.40:0/2941708193 shutdown_connections 2024-03-18T22:42:56.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.566+0000 7f677ae0a740 1 -- 172.21.15.40:0/2941708193 wait complete. 2024-03-18T22:42:56.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.566+0000 7f677ae0a740 20 client.15417 trim_cache size 0 max 16384 2024-03-18T22:42:56.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f677ae0a740 1 Processor -- start 2024-03-18T22:42:56.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f677ae0a740 1 -- start start 2024-03-18T22:42:56.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f677ae0a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a06f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f677ae0a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a1e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f677ae0a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 0x55bcd9a07490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f677ae0a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bcd9a237d0 con 0x55bcd9a0cff0 2024-03-18T22:42:56.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f677ae0a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bcd9a23ea0 con 0x55bcd9a0d9e0 2024-03-18T22:42:56.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f677ae0a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55bcd9a24570 con 0x55bcd9a12b50 2024-03-18T22:42:56.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f6775ed7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a06f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f6775ed7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a06f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39586/0 (socket says 172.21.15.40:39586) 2024-03-18T22:42:56.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f6775ed7640 1 -- 172.21.15.40:0/3669444013 learned_addr learned my addr 172.21.15.40:0/3669444013 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f67756d6640 1 --2- 172.21.15.40:0/3669444013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a1e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f67766d8640 1 --2- 172.21.15.40:0/3669444013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 0x55bcd9a07490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f6775ed7640 1 -- 172.21.15.40:0/3669444013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 msgr2=0x55bcd9a07490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f6775ed7640 1 --2- 172.21.15.40:0/3669444013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 0x55bcd9a07490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f6775ed7640 1 -- 172.21.15.40:0/3669444013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a0d9e0 msgr2=0x55bcd9a1e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f6775ed7640 1 --2- 172.21.15.40:0/3669444013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a1e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f6775ed7640 1 -- 172.21.15.40:0/3669444013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcd9a185e0 con 0x55bcd9a0cff0 2024-03-18T22:42:56.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f6775ed7640 1 --2- 172.21.15.40:0/3669444013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a06f50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6768009c80 tx=0x7f6768009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67699dfef08e31a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.584+0000 7f67577fe640 1 -- 172.21.15.40:0/3669444013 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676801e040 con 0x55bcd9a0cff0 2024-03-18T22:42:56.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.585+0000 7f67577fe640 1 -- 172.21.15.40:0/3669444013 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6768009850 con 0x55bcd9a0cff0 2024-03-18T22:42:56.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.585+0000 7f67577fe640 1 -- 172.21.15.40:0/3669444013 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67680035f0 con 0x55bcd9a0cff0 2024-03-18T22:42:56.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.585+0000 7f677ae0a740 1 -- 172.21.15.40:0/3669444013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 msgr2=0x55bcd9a06f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.585+0000 7f677ae0a740 1 --2- 172.21.15.40:0/3669444013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a06f50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6768009c80 tx=0x7f6768009530 comp rx=0 tx=0).stop 2024-03-18T22:42:56.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.585+0000 7f677ae0a740 1 -- 172.21.15.40:0/3669444013 shutdown_connections 2024-03-18T22:42:56.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.585+0000 7f677ae0a740 1 --2- 172.21.15.40:0/3669444013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a12b50 0x55bcd9a07490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.585+0000 7f677ae0a740 1 --2- 172.21.15.40:0/3669444013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a1e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.585+0000 7f677ae0a740 1 --2- 172.21.15.40:0/3669444013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a06f50 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.585+0000 7f677ae0a740 1 -- 172.21.15.40:0/3669444013 >> 172.21.15.40:0/3669444013 conn(0x55bcd9a08560 msgr2=0x55bcd9a22fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.585+0000 7f677ae0a740 1 -- 172.21.15.40:0/3669444013 shutdown_connections 2024-03-18T22:42:56.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.585+0000 7f677ae0a740 1 -- 172.21.15.40:0/3669444013 wait complete. 2024-03-18T22:42:56.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.585+0000 7f677ae0a740 1 -- start start 2024-03-18T22:42:56.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.585+0000 7f677ae0a740 1 Processor -- start 2024-03-18T22:42:56.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f677ae0a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a1a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f677ae0a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a1add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f677ae0a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a12b50 0x55bcd9a1ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f677ae0a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bcd9a25310 con 0x55bcd9a0cff0 2024-03-18T22:42:56.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f677ae0a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bcd9a24c40 con 0x55bcd9a12b50 2024-03-18T22:42:56.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f677ae0a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55bcd9a259e0 con 0x55bcd9a0d9e0 2024-03-18T22:42:56.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f677ae0a740 10 client.0 _ll_register_callbacks cb 0x7f676801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:56.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f6775ed7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a1a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f6775ed7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a1a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39602/0 (socket says 172.21.15.40:39602) 2024-03-18T22:42:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f6775ed7640 1 -- 172.21.15.40:0/3644450286 learned_addr learned my addr 172.21.15.40:0/3644450286 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f67756d6640 1 --2- 172.21.15.40:0/3644450286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a1add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f67766d8640 1 --2- 172.21.15.40:0/3644450286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a12b50 0x55bcd9a1ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f6775ed7640 1 -- 172.21.15.40:0/3644450286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a0d9e0 msgr2=0x55bcd9a1add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f6775ed7640 1 --2- 172.21.15.40:0/3644450286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a1add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f6775ed7640 1 -- 172.21.15.40:0/3644450286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a12b50 msgr2=0x55bcd9a1ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f6775ed7640 1 --2- 172.21.15.40:0/3644450286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a12b50 0x55bcd9a1ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f6775ed7640 1 -- 172.21.15.40:0/3644450286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcd9a1e3e0 con 0x55bcd9a0cff0 2024-03-18T22:42:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f6775ed7640 1 --2- 172.21.15.40:0/3644450286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a1a890 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f67680048d0 tx=0x7f6768005350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f001934c2ef32d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f67557fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f67557fa640 1 -- 172.21.15.40:0/3644450286 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676801e040 con 0x55bcd9a0cff0 2024-03-18T22:42:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f67557fa640 1 -- 172.21.15.40:0/3644450286 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6768004240 con 0x55bcd9a0cff0 2024-03-18T22:42:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.586+0000 7f677ae0a740 1 -- 172.21.15.40:0/3644450286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6768009850 con 0x55bcd9a0cff0 2024-03-18T22:42:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.587+0000 7f67557fa640 1 -- 172.21.15.40:0/3644450286 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6768046050 con 0x55bcd9a0cff0 2024-03-18T22:42:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.587+0000 7f677ae0a740 10 client.15423 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:56.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.587+0000 7f677ae0a740 1 -- 172.21.15.40:0/3644450286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bcd9a8a490 con 0x55bcd9a0cff0 2024-03-18T22:42:56.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.587+0000 7f67557fa640 1 -- 172.21.15.40:0/3644450286 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6768009850 con 0x55bcd9a0cff0 2024-03-18T22:42:56.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.587+0000 7f67557fa640 1 -- 172.21.15.40:0/3644450286 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f676803c030 con 0x55bcd9a0cff0 2024-03-18T22:42:56.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.587+0000 7f677ae0a740 10 client.15423 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:56.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.587+0000 7f677ae0a740 10 client.15423 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-18T22:42:56.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.587+0000 7f677ae0a740 20 client.15423 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:56.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.587+0000 7f677ae0a740 1 --2- 172.21.15.40:0/3644450286 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bcd9a8ecd0 0x55bcd9a8d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.587+0000 7f677ae0a740 4 client.15423 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T22:42:56.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.587+0000 7f677ae0a740 1 -- 172.21.15.40:0/3644450286 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "status"}) -- 0x55bcd9979670 con 0x55bcd9a8ecd0 2024-03-18T22:42:56.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.587+0000 7f67756d6640 1 --2- 172.21.15.40:0/3644450286 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bcd9a8ecd0 0x55bcd9a8d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f67557fa640 10 client.15423 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:42:56.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f67756d6640 1 --2- 172.21.15.40:0/3644450286 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bcd9a8ecd0 0x55bcd9a8d9e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fd01e33199d2181d server_cookie=97d0a7ac282024a2 in_seq=0 out_seq=0 2024-03-18T22:42:56.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f67557fa640 1 -- 172.21.15.40:0/3644450286 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55bcd9979670 con 0x55bcd9a8ecd0 2024-03-18T22:42:56.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f67557fa640 10 client.15423 handle_command_reply: tid=0 2024-03-18T22:42:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f677ae0a740 1 client.15423 shutdown 2024-03-18T22:42:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f677ae0a740 1 -- 172.21.15.40:0/3644450286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 msgr2=0x55bcd9a1a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f677ae0a740 1 --2- 172.21.15.40:0/3644450286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a1a890 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f67680048d0 tx=0x7f6768005350 comp rx=0 tx=0).stop 2024-03-18T22:42:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f677ae0a740 1 -- 172.21.15.40:0/3644450286 shutdown_connections 2024-03-18T22:42:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f677ae0a740 1 --2- 172.21.15.40:0/3644450286 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bcd9a8ecd0 0x55bcd9a8d9e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f677ae0a740 1 --2- 172.21.15.40:0/3644450286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcd9a12b50 0x55bcd9a1ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f677ae0a740 1 --2- 172.21.15.40:0/3644450286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bcd9a0d9e0 0x55bcd9a1add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f67557fa640 0 client.15423 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:42:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f677ae0a740 1 --2- 172.21.15.40:0/3644450286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bcd9a0cff0 0x55bcd9a1a890 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f677ae0a740 1 -- 172.21.15.40:0/3644450286 >> 172.21.15.40:0/3644450286 conn(0x55bcd9a08560 msgr2=0x55bcd9a0b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f677ae0a740 1 -- 172.21.15.40:0/3644450286 shutdown_connections 2024-03-18T22:42:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f677ae0a740 1 -- 172.21.15.40:0/3644450286 wait complete. 2024-03-18T22:42:56.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.588+0000 7f677ae0a740 20 client.15423 trim_cache size 0 max 16384 2024-03-18T22:42:56.604 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:42:56.604 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "410ef414-e577-11ee-95c9-87774f69a715", 2024-03-18T22:42:56.604 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 3, 2024-03-18T22:42:56.604 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24401, 2024-03-18T22:42:56.604 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-18T22:42:56.604 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-18T22:42:56.604 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-18T22:42:56.604 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 108.76530085100001, 2024-03-18T22:42:56.604 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 27, 2024-03-18T22:42:56.604 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 93, 2024-03-18T22:42:56.604 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T22:42:56.604 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 167.193354091 2024-03-18T22:42:56.605 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:42:56.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.589+0000 7f677813b640 1 -- 172.21.15.40:0/1279629166 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f67580a2800 msgr2=0x7f67580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.589+0000 7f677813b640 1 --2- 172.21.15.40:0/1279629166 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f67580a2800 0x7f67580a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6760006010 tx=0x7f6760032ea0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.589+0000 7f677813b640 1 -- 172.21.15.40:0/1279629166 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770101050 msgr2=0x7f677019c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.589+0000 7f677813b640 1 --2- 172.21.15.40:0/1279629166 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770101050 0x7f677019c720 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f676400e9f0 tx=0x7f676400eeb0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.589+0000 7f677813b640 1 -- 172.21.15.40:0/1279629166 shutdown_connections 2024-03-18T22:42:56.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.589+0000 7f677813b640 1 --2- 172.21.15.40:0/1279629166 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f67580a2800 0x7f67580a4cc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.589+0000 7f677813b640 1 --2- 172.21.15.40:0/1279629166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6770101a10 0x7f677019ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.589+0000 7f677813b640 1 --2- 172.21.15.40:0/1279629166 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6770101050 0x7f677019c720 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.589+0000 7f677813b640 1 --2- 172.21.15.40:0/1279629166 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67700ffe70 0x7f6770109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.589+0000 7f677813b640 1 -- 172.21.15.40:0/1279629166 >> 172.21.15.40:0/1279629166 conn(0x7f67700fb450 msgr2=0x7f67701043e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.589+0000 7f677813b640 1 -- 172.21.15.40:0/1279629166 shutdown_connections 2024-03-18T22:42:56.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.589+0000 7f677813b640 1 -- 172.21.15.40:0/1279629166 wait complete. 2024-03-18T22:42:56.607 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-18T22:42:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.709+0000 7f942cd10640 1 -- 172.21.15.40:0/1503754964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9428101930 msgr2=0x7f9428101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.709+0000 7f942cd10640 1 --2- 172.21.15.40:0/1503754964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9428101930 0x7f9428101db0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9410009900 tx=0x7f941002bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.709+0000 7f942cd10640 1 -- 172.21.15.40:0/1503754964 shutdown_connections 2024-03-18T22:42:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.709+0000 7f942cd10640 1 --2- 172.21.15.40:0/1503754964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94281022f0 0x7f9428108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.709+0000 7f942cd10640 1 --2- 172.21.15.40:0/1503754964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9428101930 0x7f9428101db0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.709+0000 7f942cd10640 1 --2- 172.21.15.40:0/1503754964 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9428100740 0x7f9428100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.709+0000 7f942cd10640 1 -- 172.21.15.40:0/1503754964 >> 172.21.15.40:0/1503754964 conn(0x7f94280fbcd0 msgr2=0x7f94280fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.709+0000 7f942cd10640 1 -- 172.21.15.40:0/1503754964 shutdown_connections 2024-03-18T22:42:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.709+0000 7f942cd10640 1 -- 172.21.15.40:0/1503754964 wait complete. 2024-03-18T22:42:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942cd10640 1 Processor -- start 2024-03-18T22:42:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942cd10640 1 -- start start 2024-03-18T22:42:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942cd10640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9428100740 0x7f942806a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942cd10640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9428101930 0x7f942819ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942cd10640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94281022f0 0x7f94281a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942cd10640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f942810c250 con 0x7f9428100740 2024-03-18T22:42:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942cd10640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f942810c0d0 con 0x7f94281022f0 2024-03-18T22:42:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942cd10640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f942810c3d0 con 0x7f9428101930 2024-03-18T22:42:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9428100740 0x7f942806a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9428100740 0x7f942806a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39616/0 (socket says 172.21.15.40:39616) 2024-03-18T22:42:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942659c640 1 -- 172.21.15.40:0/1115719485 learned_addr learned my addr 172.21.15.40:0/1115719485 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f9426d9d640 1 --2- 172.21.15.40:0/1115719485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94281022f0 0x7f94281a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f941ffff640 1 --2- 172.21.15.40:0/1115719485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9428101930 0x7f942819ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942659c640 1 -- 172.21.15.40:0/1115719485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9428101930 msgr2=0x7f942819ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942659c640 1 --2- 172.21.15.40:0/1115719485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9428101930 0x7f942819ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942659c640 1 -- 172.21.15.40:0/1115719485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94281022f0 msgr2=0x7f94281a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942659c640 1 --2- 172.21.15.40:0/1115719485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94281022f0 0x7f94281a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942659c640 1 -- 172.21.15.40:0/1115719485 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94281a1910 con 0x7f9428100740 2024-03-18T22:42:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.710+0000 7f942659c640 1 --2- 172.21.15.40:0/1115719485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9428100740 0x7f942806a9c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f941800e9c0 tx=0x7f941800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.711+0000 7f941f7fe640 1 -- 172.21.15.40:0/1115719485 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941800cd80 con 0x7f9428100740 2024-03-18T22:42:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.711+0000 7f942cd10640 1 -- 172.21.15.40:0/1115719485 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94281a1ba0 con 0x7f9428100740 2024-03-18T22:42:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.711+0000 7f941f7fe640 1 -- 172.21.15.40:0/1115719485 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94180035f0 con 0x7f9428100740 2024-03-18T22:42:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.711+0000 7f941f7fe640 1 -- 172.21.15.40:0/1115719485 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418010690 con 0x7f9428100740 2024-03-18T22:42:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.711+0000 7f942cd10640 1 -- 172.21.15.40:0/1115719485 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94281a20c0 con 0x7f9428100740 2024-03-18T22:42:56.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.713+0000 7f941f7fe640 1 -- 172.21.15.40:0/1115719485 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f94180037a0 con 0x7f9428100740 2024-03-18T22:42:56.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.713+0000 7f942f9db740 1 Processor -- start 2024-03-18T22:42:56.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.713+0000 7f942f9db740 1 -- start start 2024-03-18T22:42:56.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f942f9db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025dff0 0x55efc025e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f942f9db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025e9e0 0x55efc02634e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f941f7fe640 1 --2- 172.21.15.40:0/1115719485 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f94140a2830 0x7f94140a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f942f9db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc0263b50 0x55efc0265fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f942f9db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55efc01c2de0 con 0x55efc025e9e0 2024-03-18T22:42:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f942f9db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55efc016bc80 con 0x55efc0263b50 2024-03-18T22:42:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f942f9db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55efc01bee80 con 0x55efc025dff0 2024-03-18T22:42:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f942659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025e9e0 0x55efc02634e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f941ffff640 1 --2- 172.21.15.40:0/1115719485 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f94140a2830 0x7f94140a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f9426d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025dff0 0x55efc025e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f942659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025e9e0 0x55efc02634e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39622/0 (socket says 172.21.15.40:39622) 2024-03-18T22:42:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f942659c640 1 -- 172.21.15.40:0/825002330 learned_addr learned my addr 172.21.15.40:0/825002330 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f9426d9d640 1 --2- 172.21.15.40:0/825002330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc0263b50 0x55efc0265fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f941f7fe640 1 -- 172.21.15.40:0/1115719485 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f94180cf170 con 0x7f9428100740 2024-03-18T22:42:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f9426d9d640 1 -- 172.21.15.40:0/825002330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc0263b50 msgr2=0x55efc0265fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f9426d9d640 1 --2- 172.21.15.40:0/825002330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc0263b50 0x55efc0265fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f9426d9d640 1 -- 172.21.15.40:0/825002330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025e9e0 msgr2=0x55efc02634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f9426d9d640 1 --2- 172.21.15.40:0/825002330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025e9e0 0x55efc02634e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f9426d9d640 1 -- 172.21.15.40:0/825002330 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efc017dfb0 con 0x55efc025dff0 2024-03-18T22:42:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f942659c640 1 --2- 172.21.15.40:0/825002330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025e9e0 0x55efc02634e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f941ffff640 1 --2- 172.21.15.40:0/1115719485 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f94140a2830 0x7f94140a4cf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9410009230 tx=0x7f9410005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f9426d9d640 1 --2- 172.21.15.40:0/825002330 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025dff0 0x55efc025e410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9420011a70 tx=0x7f9420004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=697241c672b14b49 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f941cff9640 1 -- 172.21.15.40:0/825002330 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420004b20 con 0x55efc025dff0 2024-03-18T22:42:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f941cff9640 1 -- 172.21.15.40:0/825002330 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9420004cc0 con 0x55efc025dff0 2024-03-18T22:42:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.714+0000 7f941cff9640 1 -- 172.21.15.40:0/825002330 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420005550 con 0x55efc025dff0 2024-03-18T22:42:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.715+0000 7f942f9db740 1 -- 172.21.15.40:0/825002330 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025dff0 msgr2=0x55efc025e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.715+0000 7f942f9db740 1 --2- 172.21.15.40:0/825002330 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025dff0 0x55efc025e410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9420011a70 tx=0x7f9420004ab0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.715+0000 7f942f9db740 1 -- 172.21.15.40:0/825002330 shutdown_connections 2024-03-18T22:42:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.715+0000 7f942f9db740 1 --2- 172.21.15.40:0/825002330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc0263b50 0x55efc0265fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.715+0000 7f942f9db740 1 --2- 172.21.15.40:0/825002330 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025e9e0 0x55efc02634e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.715+0000 7f942f9db740 1 --2- 172.21.15.40:0/825002330 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025dff0 0x55efc025e410 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.715+0000 7f942f9db740 1 -- 172.21.15.40:0/825002330 >> 172.21.15.40:0/825002330 conn(0x55efc0259560 msgr2=0x55efc0259960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.715+0000 7f942f9db740 1 -- 172.21.15.40:0/825002330 shutdown_connections 2024-03-18T22:42:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.715+0000 7f942f9db740 1 -- 172.21.15.40:0/825002330 wait complete. 2024-03-18T22:42:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.715+0000 7f942f9db740 1 -- start start 2024-03-18T22:42:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942f9db740 1 Processor -- start 2024-03-18T22:42:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942f9db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025dff0 0x55efc0268510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942f9db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc025e9e0 0x55efc0268a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942f9db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc0263b50 0x55efc026fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942f9db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55efc0266b00 con 0x55efc025dff0 2024-03-18T22:42:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942f9db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55efc017efc0 con 0x55efc025e9e0 2024-03-18T22:42:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942f9db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55efc0266c80 con 0x55efc0263b50 2024-03-18T22:42:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942f9db740 10 client.0 _ll_register_callbacks cb 0x55efc01baa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025dff0 0x55efc0268510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025dff0 0x55efc0268510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39638/0 (socket says 172.21.15.40:39638) 2024-03-18T22:42:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942659c640 1 -- 172.21.15.40:0/3373385928 learned_addr learned my addr 172.21.15.40:0/3373385928 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f9426d9d640 1 --2- 172.21.15.40:0/3373385928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc0263b50 0x55efc026fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f941ffff640 1 --2- 172.21.15.40:0/3373385928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc025e9e0 0x55efc0268a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942659c640 1 -- 172.21.15.40:0/3373385928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc0263b50 msgr2=0x55efc026fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942659c640 1 --2- 172.21.15.40:0/3373385928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc0263b50 0x55efc026fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942659c640 1 -- 172.21.15.40:0/3373385928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc025e9e0 msgr2=0x55efc0268a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942659c640 1 --2- 172.21.15.40:0/3373385928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc025e9e0 0x55efc0268a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942659c640 1 -- 172.21.15.40:0/3373385928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efbfed72d0 con 0x55efc025dff0 2024-03-18T22:42:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f9426d9d640 1 --2- 172.21.15.40:0/3373385928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc0263b50 0x55efc026fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f942659c640 1 --2- 172.21.15.40:0/3373385928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025dff0 0x55efc0268510 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f94180058c0 tx=0x7f941806b240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc4d82a998489814 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f940dffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.716+0000 7f940dffb640 1 -- 172.21.15.40:0/3373385928 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941809b080 con 0x55efc025dff0 2024-03-18T22:42:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.717+0000 7f940dffb640 1 -- 172.21.15.40:0/3373385928 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f94180c7700 con 0x55efc025dff0 2024-03-18T22:42:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.717+0000 7f942f9db740 1 -- 172.21.15.40:0/3373385928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9420004cc0 con 0x55efc025dff0 2024-03-18T22:42:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.717+0000 7f940dffb640 1 -- 172.21.15.40:0/3373385928 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9418003b30 con 0x55efc025dff0 2024-03-18T22:42:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.717+0000 7f942f9db740 10 client.15435 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.717+0000 7f942f9db740 1 -- 172.21.15.40:0/3373385928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55efbffa16b0 con 0x55efc025dff0 2024-03-18T22:42:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.717+0000 7f940dffb640 1 -- 172.21.15.40:0/3373385928 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9420004cc0 con 0x55efc025dff0 2024-03-18T22:42:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.717+0000 7f940dffb640 1 -- 172.21.15.40:0/3373385928 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f9418066550 con 0x55efc025dff0 2024-03-18T22:42:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.717+0000 7f942f9db740 10 client.15435 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.717+0000 7f942f9db740 10 client.15435 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T22:42:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.717+0000 7f942f9db740 20 client.15435 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.717+0000 7f942f9db740 1 --2- 172.21.15.40:0/3373385928 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55efc0275330 0x55efc02767a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.717+0000 7f942f9db740 4 client.15435 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:42:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.717+0000 7f942f9db740 1 -- 172.21.15.40:0/3373385928 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efc01d0320 con 0x55efc0275330 2024-03-18T22:42:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.717+0000 7f941ffff640 1 --2- 172.21.15.40:0/3373385928 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55efc0275330 0x55efc02767a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.718+0000 7f940dffb640 10 client.15435 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:42:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.718+0000 7f941ffff640 1 --2- 172.21.15.40:0/3373385928 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55efc0275330 0x55efc02767a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=16783a8084b10358 server_cookie=5c27c2e97c3d391f in_seq=0 out_seq=0 2024-03-18T22:42:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.718+0000 7f940dffb640 1 -- 172.21.15.40:0/3373385928 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55efc01d0320 con 0x55efc0275330 2024-03-18T22:42:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.718+0000 7f940dffb640 10 client.15435 handle_command_reply: tid=0 2024-03-18T22:42:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.718+0000 7f942f9db740 1 client.15435 shutdown 2024-03-18T22:42:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.718+0000 7f942f9db740 1 -- 172.21.15.40:0/3373385928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025dff0 msgr2=0x55efc0268510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.718+0000 7f942f9db740 1 --2- 172.21.15.40:0/3373385928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025dff0 0x55efc0268510 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f94180058c0 tx=0x7f941806b240 comp rx=0 tx=0).stop 2024-03-18T22:42:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.718+0000 7f942f9db740 1 -- 172.21.15.40:0/3373385928 shutdown_connections 2024-03-18T22:42:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.718+0000 7f942f9db740 1 --2- 172.21.15.40:0/3373385928 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55efc0275330 0x55efc02767a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.718+0000 7f942f9db740 1 --2- 172.21.15.40:0/3373385928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc0263b50 0x55efc026fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.718+0000 7f942f9db740 1 --2- 172.21.15.40:0/3373385928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc025e9e0 0x55efc0268a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.718+0000 7f940dffb640 0 client.15435 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:42:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.718+0000 7f942f9db740 1 --2- 172.21.15.40:0/3373385928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025dff0 0x55efc0268510 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.718+0000 7f942f9db740 1 -- 172.21.15.40:0/3373385928 >> 172.21.15.40:0/3373385928 conn(0x55efc0259560 msgr2=0x55efc02587e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.719+0000 7f942f9db740 1 -- 172.21.15.40:0/3373385928 shutdown_connections 2024-03-18T22:42:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.719+0000 7f942f9db740 1 -- 172.21.15.40:0/3373385928 wait complete. 2024-03-18T22:42:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.719+0000 7f942f9db740 20 client.15435 trim_cache size 0 max 16384 2024-03-18T22:42:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942f9db740 1 Processor -- start 2024-03-18T22:42:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942f9db740 1 -- start start 2024-03-18T22:42:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942f9db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025dff0 0x55efc025f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942f9db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025e9e0 0x55efc026fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942f9db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc0263b50 0x55efc025ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942f9db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55efc0274ae0 con 0x55efc025dff0 2024-03-18T22:42:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942f9db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55efc02751b0 con 0x55efc0263b50 2024-03-18T22:42:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942f9db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55efc0275880 con 0x55efc025e9e0 2024-03-18T22:42:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025dff0 0x55efc025f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025dff0 0x55efc025f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39640/0 (socket says 172.21.15.40:39640) 2024-03-18T22:42:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942659c640 1 -- 172.21.15.40:0/1986306430 learned_addr learned my addr 172.21.15.40:0/1986306430 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f941ffff640 1 --2- 172.21.15.40:0/1986306430 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025e9e0 0x55efc026fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f9426d9d640 1 --2- 172.21.15.40:0/1986306430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc0263b50 0x55efc025ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942659c640 1 -- 172.21.15.40:0/1986306430 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025e9e0 msgr2=0x55efc026fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942659c640 1 --2- 172.21.15.40:0/1986306430 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025e9e0 0x55efc026fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942659c640 1 -- 172.21.15.40:0/1986306430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc0263b50 msgr2=0x55efc025ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942659c640 1 --2- 172.21.15.40:0/1986306430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc0263b50 0x55efc025ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942659c640 1 -- 172.21.15.40:0/1986306430 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efc0272cc0 con 0x55efc025dff0 2024-03-18T22:42:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f942659c640 1 --2- 172.21.15.40:0/1986306430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025dff0 0x55efc025f270 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f941809b420 tx=0x7f9418099c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d733038c4f935958 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f940ffff640 1 -- 172.21.15.40:0/1986306430 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941809b080 con 0x55efc025dff0 2024-03-18T22:42:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f940ffff640 1 -- 172.21.15.40:0/1986306430 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f941809be20 con 0x55efc025dff0 2024-03-18T22:42:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.737+0000 7f940ffff640 1 -- 172.21.15.40:0/1986306430 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94180afb00 con 0x55efc025dff0 2024-03-18T22:42:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.738+0000 7f942f9db740 1 -- 172.21.15.40:0/1986306430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025dff0 msgr2=0x55efc025f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.738+0000 7f942f9db740 1 --2- 172.21.15.40:0/1986306430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025dff0 0x55efc025f270 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f941809b420 tx=0x7f9418099c30 comp rx=0 tx=0).stop 2024-03-18T22:42:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.738+0000 7f942f9db740 1 -- 172.21.15.40:0/1986306430 shutdown_connections 2024-03-18T22:42:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.738+0000 7f942f9db740 1 --2- 172.21.15.40:0/1986306430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc0263b50 0x55efc025ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.738+0000 7f942f9db740 1 --2- 172.21.15.40:0/1986306430 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025e9e0 0x55efc026fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.738+0000 7f942f9db740 1 --2- 172.21.15.40:0/1986306430 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc025dff0 0x55efc025f270 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.738+0000 7f942f9db740 1 -- 172.21.15.40:0/1986306430 >> 172.21.15.40:0/1986306430 conn(0x55efc0259560 msgr2=0x55efc0259940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.738+0000 7f942f9db740 1 -- 172.21.15.40:0/1986306430 shutdown_connections 2024-03-18T22:42:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.738+0000 7f942f9db740 1 -- 172.21.15.40:0/1986306430 wait complete. 2024-03-18T22:42:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.738+0000 7f942f9db740 1 -- start start 2024-03-18T22:42:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.738+0000 7f942f9db740 1 Processor -- start 2024-03-18T22:42:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f942f9db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc025dff0 0x55efc026bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f942f9db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025e9e0 0x55efc026c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f942f9db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc0263b50 0x55efc026eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f942f9db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55efc0276620 con 0x55efc0263b50 2024-03-18T22:42:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f942f9db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55efc0275f50 con 0x55efc025dff0 2024-03-18T22:42:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f942f9db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55efc0276cf0 con 0x55efc025e9e0 2024-03-18T22:42:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f942f9db740 10 client.0 _ll_register_callbacks cb 0x7f9418096100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f9426d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc0263b50 0x55efc026eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f942659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc025dff0 0x55efc026bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f9426d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc0263b50 0x55efc026eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39646/0 (socket says 172.21.15.40:39646) 2024-03-18T22:42:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f9426d9d640 1 -- 172.21.15.40:0/3139971120 learned_addr learned my addr 172.21.15.40:0/3139971120 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f941ffff640 1 --2- 172.21.15.40:0/3139971120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025e9e0 0x55efc026c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f9426d9d640 1 -- 172.21.15.40:0/3139971120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025e9e0 msgr2=0x55efc026c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f9426d9d640 1 --2- 172.21.15.40:0/3139971120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025e9e0 0x55efc026c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f9426d9d640 1 -- 172.21.15.40:0/3139971120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc025dff0 msgr2=0x55efc026bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f9426d9d640 1 --2- 172.21.15.40:0/3139971120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc025dff0 0x55efc026bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f9426d9d640 1 -- 172.21.15.40:0/3139971120 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efc026f120 con 0x55efc0263b50 2024-03-18T22:42:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f9426d9d640 1 --2- 172.21.15.40:0/3139971120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc0263b50 0x55efc026eb30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9420011a70 tx=0x7f9420009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd964603b3481f69 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f940dffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f940dffb640 1 -- 172.21.15.40:0/3139971120 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420004d30 con 0x55efc0263b50 2024-03-18T22:42:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f940dffb640 1 -- 172.21.15.40:0/3139971120 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9420011040 con 0x55efc0263b50 2024-03-18T22:42:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f942f9db740 1 -- 172.21.15.40:0/3139971120 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f941809be20 con 0x55efc0263b50 2024-03-18T22:42:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f940dffb640 1 -- 172.21.15.40:0/3139971120 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9420037050 con 0x55efc0263b50 2024-03-18T22:42:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f942f9db740 10 client.15441 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.739+0000 7f942f9db740 1 -- 172.21.15.40:0/3139971120 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55efc02db510 con 0x55efc0263b50 2024-03-18T22:42:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.740+0000 7f940dffb640 1 -- 172.21.15.40:0/3139971120 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f941809be20 con 0x55efc0263b50 2024-03-18T22:42:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.740+0000 7f940dffb640 1 -- 172.21.15.40:0/3139971120 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f942001d030 con 0x55efc0263b50 2024-03-18T22:42:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.740+0000 7f942f9db740 10 client.15441 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.740+0000 7f942f9db740 10 client.15441 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T22:42:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.740+0000 7f942f9db740 20 client.15441 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.740+0000 7f942f9db740 1 --2- 172.21.15.40:0/3139971120 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55efc02dfe50 0x55efc02deb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.740+0000 7f942f9db740 4 client.15441 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T22:42:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.740+0000 7f942f9db740 1 -- 172.21.15.40:0/3139971120 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "status"}) -- 0x55efc01ca670 con 0x55efc02dfe50 2024-03-18T22:42:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.740+0000 7f942659c640 1 --2- 172.21.15.40:0/3139971120 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55efc02dfe50 0x55efc02deb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.740+0000 7f940dffb640 10 client.15441 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:42:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.740+0000 7f942659c640 1 --2- 172.21.15.40:0/3139971120 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55efc02dfe50 0x55efc02deb60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3f6c5fa9ca48697f server_cookie=46b3bef8eaafee19 in_seq=0 out_seq=0 2024-03-18T22:42:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f940dffb640 1 -- 172.21.15.40:0/3139971120 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55efc01ca670 con 0x55efc02dfe50 2024-03-18T22:42:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f940dffb640 10 client.15441 handle_command_reply: tid=0 2024-03-18T22:42:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942f9db740 1 client.15441 shutdown 2024-03-18T22:42:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942f9db740 1 -- 172.21.15.40:0/3139971120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc0263b50 msgr2=0x55efc026eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942f9db740 1 --2- 172.21.15.40:0/3139971120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc0263b50 0x55efc026eb30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9420011a70 tx=0x7f9420009e00 comp rx=0 tx=0).stop 2024-03-18T22:42:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942f9db740 1 -- 172.21.15.40:0/3139971120 shutdown_connections 2024-03-18T22:42:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942f9db740 1 --2- 172.21.15.40:0/3139971120 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55efc02dfe50 0x55efc02deb60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942f9db740 1 --2- 172.21.15.40:0/3139971120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55efc0263b50 0x55efc026eb30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f940dffb640 0 client.15441 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:42:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942f9db740 1 --2- 172.21.15.40:0/3139971120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55efc025e9e0 0x55efc026c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942f9db740 1 --2- 172.21.15.40:0/3139971120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efc025dff0 0x55efc026bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942f9db740 1 -- 172.21.15.40:0/3139971120 >> 172.21.15.40:0/3139971120 conn(0x55efc0259560 msgr2=0x55efc025f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942f9db740 1 -- 172.21.15.40:0/3139971120 shutdown_connections 2024-03-18T22:42:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942f9db740 1 -- 172.21.15.40:0/3139971120 wait complete. 2024-03-18T22:42:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942f9db740 20 client.15441 trim_cache size 0 max 16384 2024-03-18T22:42:56.750 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:42:56.750 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "410ef414-e577-11ee-95c9-87774f69a715", 2024-03-18T22:42:56.750 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-03-18T22:42:56.750 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24473, 2024-03-18T22:42:56.750 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-18T22:42:56.750 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-18T22:42:56.750 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-18T22:42:56.750 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 109.702912059, 2024-03-18T22:42:56.750 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 27, 2024-03-18T22:42:56.751 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 93, 2024-03-18T22:42:56.751 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T22:42:56.751 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 150.30824458000001 2024-03-18T22:42:56.751 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:42:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942cd10640 1 -- 172.21.15.40:0/1115719485 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f94140a2830 msgr2=0x7f94140a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942cd10640 1 --2- 172.21.15.40:0/1115719485 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f94140a2830 0x7f94140a4cf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9410009230 tx=0x7f9410005fd0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942cd10640 1 -- 172.21.15.40:0/1115719485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9428100740 msgr2=0x7f942806a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942cd10640 1 --2- 172.21.15.40:0/1115719485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9428100740 0x7f942806a9c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f941800e9c0 tx=0x7f941800ee80 comp rx=0 tx=0).stop 2024-03-18T22:42:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942cd10640 1 -- 172.21.15.40:0/1115719485 shutdown_connections 2024-03-18T22:42:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942cd10640 1 --2- 172.21.15.40:0/1115719485 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f94140a2830 0x7f94140a4cf0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942cd10640 1 --2- 172.21.15.40:0/1115719485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94281022f0 0x7f94281a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942cd10640 1 --2- 172.21.15.40:0/1115719485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9428101930 0x7f942819ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942cd10640 1 --2- 172.21.15.40:0/1115719485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9428100740 0x7f942806a9c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942cd10640 1 -- 172.21.15.40:0/1115719485 >> 172.21.15.40:0/1115719485 conn(0x7f94280fbcd0 msgr2=0x7f9428104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942cd10640 1 -- 172.21.15.40:0/1115719485 shutdown_connections 2024-03-18T22:42:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.741+0000 7f942cd10640 1 -- 172.21.15.40:0/1115719485 wait complete. 2024-03-18T22:42:56.754 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T22:42:56.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.856+0000 7f5e88bf3640 1 -- 172.21.15.40:0/2454482755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e84101cf0 msgr2=0x7f5e84102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.856+0000 7f5e88bf3640 1 --2- 172.21.15.40:0/2454482755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e84101cf0 0x7f5e84102110 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009900 tx=0x7f5e7402bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:56.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.856+0000 7f5e88bf3640 1 -- 172.21.15.40:0/2454482755 shutdown_connections 2024-03-18T22:42:56.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.856+0000 7f5e88bf3640 1 --2- 172.21.15.40:0/2454482755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e841013e0 0x7f5e84108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.856+0000 7f5e88bf3640 1 --2- 172.21.15.40:0/2454482755 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e84100a20 0x7f5e84100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.856+0000 7f5e88bf3640 1 --2- 172.21.15.40:0/2454482755 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e84101cf0 0x7f5e84102110 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.856+0000 7f5e88bf3640 1 -- 172.21.15.40:0/2454482755 >> 172.21.15.40:0/2454482755 conn(0x7f5e840fb450 msgr2=0x7f5e840fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.856+0000 7f5e88bf3640 1 -- 172.21.15.40:0/2454482755 shutdown_connections 2024-03-18T22:42:56.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.856+0000 7f5e88bf3640 1 -- 172.21.15.40:0/2454482755 wait complete. 2024-03-18T22:42:56.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.857+0000 7f5e88bf3640 1 Processor -- start 2024-03-18T22:42:56.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.857+0000 7f5e88bf3640 1 -- start start 2024-03-18T22:42:56.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.857+0000 7f5e88bf3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e84100a20 0x7f5e841047d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.857+0000 7f5e88bf3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e841013e0 0x7f5e84102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.857+0000 7f5e88bf3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e84101cf0 0x7f5e84103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.857+0000 7f5e88bf3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8410c270 con 0x7f5e84101cf0 2024-03-18T22:42:56.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.857+0000 7f5e88bf3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8410c0f0 con 0x7f5e84100a20 2024-03-18T22:42:56.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.857+0000 7f5e88bf3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8410c3f0 con 0x7f5e841013e0 2024-03-18T22:42:56.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.857+0000 7f5e82d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e84101cf0 0x7f5e84103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.857+0000 7f5e8259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e84100a20 0x7f5e841047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.857+0000 7f5e81d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e841013e0 0x7f5e84102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.858+0000 7f5e8259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e84100a20 0x7f5e841047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:44690/0 (socket says 172.21.15.40:44690) 2024-03-18T22:42:56.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.858+0000 7f5e8259c640 1 -- 172.21.15.40:0/864828522 learned_addr learned my addr 172.21.15.40:0/864828522 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.858+0000 7f5e82d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e84101cf0 0x7f5e84103350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39664/0 (socket says 172.21.15.40:39664) 2024-03-18T22:42:56.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.858+0000 7f5e8259c640 1 -- 172.21.15.40:0/864828522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e841013e0 msgr2=0x7f5e84102e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.858+0000 7f5e8259c640 1 --2- 172.21.15.40:0/864828522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e841013e0 0x7f5e84102e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.858+0000 7f5e8259c640 1 -- 172.21.15.40:0/864828522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e84101cf0 msgr2=0x7f5e84103350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.858+0000 7f5e8259c640 1 --2- 172.21.15.40:0/864828522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e84101cf0 0x7f5e84103350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.858+0000 7f5e8259c640 1 -- 172.21.15.40:0/864828522 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e84103be0 con 0x7f5e84100a20 2024-03-18T22:42:56.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.858+0000 7f5e82d9d640 1 --2- 172.21.15.40:0/864828522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e84101cf0 0x7f5e84103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:56.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.858+0000 7f5e8259c640 1 --2- 172.21.15.40:0/864828522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e84100a20 0x7f5e841047d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5e74000c00 tx=0x7f5e74005e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.858+0000 7f5e737fe640 1 -- 172.21.15.40:0/864828522 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74043020 con 0x7f5e84100a20 2024-03-18T22:42:56.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.858+0000 7f5e737fe640 1 -- 172.21.15.40:0/864828522 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e740043d0 con 0x7f5e84100a20 2024-03-18T22:42:56.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.858+0000 7f5e88bf3640 1 -- 172.21.15.40:0/864828522 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e841a1c10 con 0x7f5e84100a20 2024-03-18T22:42:56.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.858+0000 7f5e737fe640 1 -- 172.21.15.40:0/864828522 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74003b10 con 0x7f5e84100a20 2024-03-18T22:42:56.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.858+0000 7f5e88bf3640 1 -- 172.21.15.40:0/864828522 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e841a2160 con 0x7f5e84100a20 2024-03-18T22:42:56.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.860+0000 7f5e737fe640 1 -- 172.21.15.40:0/864828522 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5e74046020 con 0x7f5e84100a20 2024-03-18T22:42:56.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e8b7db740 1 Processor -- start 2024-03-18T22:42:56.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e8b7db740 1 -- start start 2024-03-18T22:42:56.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e8b7db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abcff0 0x557505abd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e8b7db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abd9e0 0x557505ac24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e8b7db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505ac2b50 0x557505ac4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e8b7db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557505a21de0 con 0x557505abcff0 2024-03-18T22:42:56.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e8b7db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5575059cac80 con 0x557505ac2b50 2024-03-18T22:42:56.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e8b7db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557505a1de80 con 0x557505abd9e0 2024-03-18T22:42:56.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e737fe640 1 --2- 172.21.15.40:0/864828522 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5e6c0a2800 0x7f5e6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e82d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abcff0 0x557505abd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e81d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505ac2b50 0x557505ac4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e82d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abcff0 0x557505abd410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39674/0 (socket says 172.21.15.40:39674) 2024-03-18T22:42:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e82d9d640 1 -- 172.21.15.40:0/3682617141 learned_addr learned my addr 172.21.15.40:0/3682617141 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e8259c640 1 --2- 172.21.15.40:0/3682617141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abd9e0 0x557505ac24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e82d9d640 1 --2- 172.21.15.40:0/864828522 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5e6c0a2800 0x7f5e6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e81d9b640 1 -- 172.21.15.40:0/3682617141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abd9e0 msgr2=0x557505ac24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e737fe640 1 -- 172.21.15.40:0/864828522 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5e740f0fe0 con 0x7f5e84100a20 2024-03-18T22:42:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e81d9b640 1 --2- 172.21.15.40:0/3682617141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abd9e0 0x557505ac24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e81d9b640 1 -- 172.21.15.40:0/3682617141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abcff0 msgr2=0x557505abd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e81d9b640 1 --2- 172.21.15.40:0/3682617141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abcff0 0x557505abd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e81d9b640 1 -- 172.21.15.40:0/3682617141 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575059dcfb0 con 0x557505ac2b50 2024-03-18T22:42:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.861+0000 7f5e82d9d640 1 --2- 172.21.15.40:0/3682617141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abcff0 0x557505abd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.862+0000 7f5e81d9b640 1 --2- 172.21.15.40:0/3682617141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505ac2b50 0x557505ac4fe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009bf0 tx=0x7f5e78011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f734a5ad66037c5c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.862+0000 7f5e82d9d640 1 --2- 172.21.15.40:0/864828522 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5e6c0a2800 0x7f5e6c0a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x557505abe240 tx=0x7f5e7c009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.862+0000 7f5e70ff9640 1 -- 172.21.15.40:0/3682617141 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78004050 con 0x557505ac2b50 2024-03-18T22:42:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.862+0000 7f5e70ff9640 1 -- 172.21.15.40:0/3682617141 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e780054d0 con 0x557505ac2b50 2024-03-18T22:42:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.862+0000 7f5e70ff9640 1 -- 172.21.15.40:0/3682617141 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78004cc0 con 0x557505ac2b50 2024-03-18T22:42:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.862+0000 7f5e8b7db740 1 -- 172.21.15.40:0/3682617141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505ac2b50 msgr2=0x557505ac4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.862+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/3682617141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505ac2b50 0x557505ac4fe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009bf0 tx=0x7f5e78011810 comp rx=0 tx=0).stop 2024-03-18T22:42:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.862+0000 7f5e8b7db740 1 -- 172.21.15.40:0/3682617141 shutdown_connections 2024-03-18T22:42:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.862+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/3682617141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505ac2b50 0x557505ac4fe0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.862+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/3682617141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abd9e0 0x557505ac24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.862+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/3682617141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abcff0 0x557505abd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e8b7db740 1 -- 172.21.15.40:0/3682617141 >> 172.21.15.40:0/3682617141 conn(0x557505ab8560 msgr2=0x557505ab8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e8b7db740 1 -- 172.21.15.40:0/3682617141 shutdown_connections 2024-03-18T22:42:56.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e8b7db740 1 -- 172.21.15.40:0/3682617141 wait complete. 2024-03-18T22:42:56.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e8b7db740 1 -- start start 2024-03-18T22:42:56.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e8b7db740 1 Processor -- start 2024-03-18T22:42:56.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e8b7db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abcff0 0x557505ac74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e8b7db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abd9e0 0x557505ac79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e8b7db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505ac2b50 0x557505acea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e8b7db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557505ac5b00 con 0x557505abd9e0 2024-03-18T22:42:56.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e8b7db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5575059ddfc0 con 0x557505ac2b50 2024-03-18T22:42:56.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e8b7db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557505ac5c80 con 0x557505abcff0 2024-03-18T22:42:56.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e8b7db740 10 client.0 _ll_register_callbacks cb 0x557505a19a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:56.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e81d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abd9e0 0x557505ac79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e81d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abd9e0 0x557505ac79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39676/0 (socket says 172.21.15.40:39676) 2024-03-18T22:42:56.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e81d9b640 1 -- 172.21.15.40:0/2221783502 learned_addr learned my addr 172.21.15.40:0/2221783502 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e8259c640 1 --2- 172.21.15.40:0/2221783502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abcff0 0x557505ac74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.863+0000 7f5e82d9d640 1 --2- 172.21.15.40:0/2221783502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505ac2b50 0x557505acea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e8259c640 1 -- 172.21.15.40:0/2221783502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505ac2b50 msgr2=0x557505acea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e8259c640 1 --2- 172.21.15.40:0/2221783502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505ac2b50 0x557505acea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e8259c640 1 -- 172.21.15.40:0/2221783502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abd9e0 msgr2=0x557505ac79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e8259c640 1 --2- 172.21.15.40:0/2221783502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abd9e0 0x557505ac79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e8259c640 1 -- 172.21.15.40:0/2221783502 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575057362d0 con 0x557505abcff0 2024-03-18T22:42:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e81d9b640 1 --2- 172.21.15.40:0/2221783502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abd9e0 0x557505ac79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e8259c640 1 --2- 172.21.15.40:0/2221783502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abcff0 0x557505ac74a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x557505abe450 tx=0x7f5e74009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84e2def001ab953a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:42:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e69ffb640 1 -- 172.21.15.40:0/2221783502 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e740bdc50 con 0x557505abcff0 2024-03-18T22:42:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e69ffb640 1 -- 172.21.15.40:0/2221783502 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5e74003cb0 con 0x557505abcff0 2024-03-18T22:42:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2221783502 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e780054d0 con 0x557505abcff0 2024-03-18T22:42:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e69ffb640 1 -- 172.21.15.40:0/2221783502 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5e740bd550 con 0x557505abcff0 2024-03-18T22:42:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e8b7db740 10 client.25327 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2221783502 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5575058006b0 con 0x557505abcff0 2024-03-18T22:42:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e69ffb640 1 -- 172.21.15.40:0/2221783502 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e780054d0 con 0x557505abcff0 2024-03-18T22:42:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e69ffb640 1 -- 172.21.15.40:0/2221783502 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f5e740d6270 con 0x557505abcff0 2024-03-18T22:42:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e8b7db740 10 client.25327 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e8b7db740 10 client.25327 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T22:42:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e8b7db740 20 client.25327 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/2221783502 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557505ad4300 0x557505ad5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e8b7db740 4 client.25327 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:42:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.864+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2221783502 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557505a2f320 con 0x557505ad4300 2024-03-18T22:42:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.865+0000 7f5e81d9b640 1 --2- 172.21.15.40:0/2221783502 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557505ad4300 0x557505ad5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.865+0000 7f5e69ffb640 10 client.25327 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:42:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.865+0000 7f5e81d9b640 1 --2- 172.21.15.40:0/2221783502 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557505ad4300 0x557505ad5770 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2300696b5410e121 server_cookie=e2220528c67cf14c in_seq=0 out_seq=0 2024-03-18T22:42:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.866+0000 7f5e69ffb640 1 -- 172.21.15.40:0/2221783502 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557505a2f320 con 0x557505ad4300 2024-03-18T22:42:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.866+0000 7f5e69ffb640 10 client.25327 handle_command_reply: tid=0 2024-03-18T22:42:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.866+0000 7f5e8b7db740 1 client.25327 shutdown 2024-03-18T22:42:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.866+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2221783502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abcff0 msgr2=0x557505ac74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.866+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/2221783502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abcff0 0x557505ac74a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x557505abe450 tx=0x7f5e74009620 comp rx=0 tx=0).stop 2024-03-18T22:42:56.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.866+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2221783502 shutdown_connections 2024-03-18T22:42:56.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.866+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/2221783502 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557505ad4300 0x557505ad5770 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.866+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/2221783502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505ac2b50 0x557505acea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.866+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/2221783502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abd9e0 0x557505ac79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.866+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/2221783502 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abcff0 0x557505ac74a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.866+0000 7f5e69ffb640 0 client.25327 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:42:56.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.866+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2221783502 >> 172.21.15.40:0/2221783502 conn(0x557505ab8560 msgr2=0x557505ab77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.866+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2221783502 shutdown_connections 2024-03-18T22:42:56.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.866+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2221783502 wait complete. 2024-03-18T22:42:56.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.866+0000 7f5e8b7db740 20 client.25327 trim_cache size 0 max 16384 2024-03-18T22:42:56.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.884+0000 7f5e8b7db740 1 Processor -- start 2024-03-18T22:42:56.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.884+0000 7f5e8b7db740 1 -- start start 2024-03-18T22:42:56.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.884+0000 7f5e8b7db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505abcff0 0x557505ac4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.884+0000 7f5e8b7db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abd9e0 0x557505ace9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.884+0000 7f5e8b7db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505ac2b50 0x557505ab3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.884+0000 7f5e8b7db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557505ad3ab0 con 0x557505ac2b50 2024-03-18T22:42:56.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.884+0000 7f5e8b7db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557505ad4180 con 0x557505abcff0 2024-03-18T22:42:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.884+0000 7f5e8b7db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557505ad4850 con 0x557505abd9e0 2024-03-18T22:42:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.884+0000 7f5e82d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505ac2b50 0x557505ab3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.884+0000 7f5e82d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505ac2b50 0x557505ab3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39688/0 (socket says 172.21.15.40:39688) 2024-03-18T22:42:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.884+0000 7f5e8259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505abcff0 0x557505ac4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.884+0000 7f5e81d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abd9e0 0x557505ace9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.884+0000 7f5e82d9d640 1 -- 172.21.15.40:0/735915411 learned_addr learned my addr 172.21.15.40:0/735915411 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e82d9d640 1 -- 172.21.15.40:0/735915411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abd9e0 msgr2=0x557505ace9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e82d9d640 1 --2- 172.21.15.40:0/735915411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abd9e0 0x557505ace9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e82d9d640 1 -- 172.21.15.40:0/735915411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505abcff0 msgr2=0x557505ac4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e82d9d640 1 --2- 172.21.15.40:0/735915411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505abcff0 0x557505ac4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e82d9d640 1 -- 172.21.15.40:0/735915411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557505abe270 con 0x557505ac2b50 2024-03-18T22:42:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e81d9b640 1 --2- 172.21.15.40:0/735915411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abd9e0 0x557505ace9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e8259c640 1 --2- 172.21.15.40:0/735915411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505abcff0 0x557505ac4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e82d9d640 1 --2- 172.21.15.40:0/735915411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505ac2b50 0x557505ab3870 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c005c00 tx=0x7f5e7c008000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3108ecad3db1e840 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e6bfff640 1 -- 172.21.15.40:0/735915411 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c0047b0 con 0x557505ac2b50 2024-03-18T22:42:56.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e6bfff640 1 -- 172.21.15.40:0/735915411 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e7c005020 con 0x557505ac2b50 2024-03-18T22:42:56.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e6bfff640 1 -- 172.21.15.40:0/735915411 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c02a6e0 con 0x557505ac2b50 2024-03-18T22:42:56.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e8b7db740 1 -- 172.21.15.40:0/735915411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505ac2b50 msgr2=0x557505ab3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/735915411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505ac2b50 0x557505ab3870 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c005c00 tx=0x7f5e7c008000 comp rx=0 tx=0).stop 2024-03-18T22:42:56.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e8b7db740 1 -- 172.21.15.40:0/735915411 shutdown_connections 2024-03-18T22:42:56.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/735915411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505ac2b50 0x557505ab3870 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/735915411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505abd9e0 0x557505ace9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/735915411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505abcff0 0x557505ac4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.885+0000 7f5e8b7db740 1 -- 172.21.15.40:0/735915411 >> 172.21.15.40:0/735915411 conn(0x557505ab8560 msgr2=0x557505ab8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8b7db740 1 -- 172.21.15.40:0/735915411 shutdown_connections 2024-03-18T22:42:56.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8b7db740 1 -- 172.21.15.40:0/735915411 wait complete. 2024-03-18T22:42:56.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8b7db740 1 -- start start 2024-03-18T22:42:56.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8b7db740 1 Processor -- start 2024-03-18T22:42:56.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8b7db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505abcff0 0x557505acc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8b7db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abd9e0 0x557505b36050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8b7db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505ac2b50 0x557505b385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8b7db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557505ad55f0 con 0x557505abd9e0 2024-03-18T22:42:56.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8b7db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557505ad4f20 con 0x557505abcff0 2024-03-18T22:42:56.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8b7db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557505ad5cc0 con 0x557505ac2b50 2024-03-18T22:42:56.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8b7db740 10 client.0 _ll_register_callbacks cb 0x7f5e74009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:56.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e81d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abd9e0 0x557505b36050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505abcff0 0x557505acc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e82d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505ac2b50 0x557505b385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e81d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abd9e0 0x557505b36050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39702/0 (socket says 172.21.15.40:39702) 2024-03-18T22:42:56.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e81d9b640 1 -- 172.21.15.40:0/2211427564 learned_addr learned my addr 172.21.15.40:0/2211427564 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:56.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8259c640 1 -- 172.21.15.40:0/2211427564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505ac2b50 msgr2=0x557505b385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8259c640 1 --2- 172.21.15.40:0/2211427564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505ac2b50 0x557505b385a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8259c640 1 -- 172.21.15.40:0/2211427564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abd9e0 msgr2=0x557505b36050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:56.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8259c640 1 --2- 172.21.15.40:0/2211427564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abd9e0 0x557505b36050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e8259c640 1 -- 172.21.15.40:0/2211427564 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557505b38c90 con 0x557505abcff0 2024-03-18T22:42:56.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.886+0000 7f5e81d9b640 1 --2- 172.21.15.40:0/2211427564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abd9e0 0x557505b36050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:56.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e8259c640 1 --2- 172.21.15.40:0/2211427564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505abcff0 0x557505acc9d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5e7402cdd0 tx=0x7f5e74005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6b673c32dfe5037 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:56.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:42:56.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e69ffb640 1 -- 172.21.15.40:0/2211427564 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e740c5900 con 0x557505abcff0 2024-03-18T22:42:56.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2211427564 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e7c005020 con 0x557505abcff0 2024-03-18T22:42:56.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e69ffb640 1 -- 172.21.15.40:0/2211427564 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5e740ee4b0 con 0x557505abcff0 2024-03-18T22:42:56.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e69ffb640 1 -- 172.21.15.40:0/2211427564 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5e740d6410 con 0x557505abcff0 2024-03-18T22:42:56.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e8b7db740 10 client.25349 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:56.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2211427564 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557505b39fe0 con 0x557505abcff0 2024-03-18T22:42:56.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e69ffb640 1 -- 172.21.15.40:0/2211427564 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e7c005020 con 0x557505abcff0 2024-03-18T22:42:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e69ffb640 1 -- 172.21.15.40:0/2211427564 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f5e740b9430 con 0x557505abcff0 2024-03-18T22:42:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e8b7db740 10 client.25349 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e8b7db740 10 client.25349 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T22:42:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e8b7db740 20 client.25349 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/2211427564 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557505b3e9d0 0x557505b3d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e8b7db740 4 client.25349 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T22:42:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.887+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2211427564 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "status"}) -- 0x557505a29670 con 0x557505b3e9d0 2024-03-18T22:42:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e81d9b640 1 --2- 172.21.15.40:0/2211427564 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557505b3e9d0 0x557505b3d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e81d9b640 1 --2- 172.21.15.40:0/2211427564 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557505b3e9d0 0x557505b3d6e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=edc131c616547b3f server_cookie=e9114ddb39db5382 in_seq=0 out_seq=0 2024-03-18T22:42:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e69ffb640 10 client.25349 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:42:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e69ffb640 1 -- 172.21.15.40:0/2211427564 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x557505a29670 con 0x557505b3e9d0 2024-03-18T22:42:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e69ffb640 10 client.25349 handle_command_reply: tid=0 2024-03-18T22:42:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e8b7db740 1 client.25349 shutdown 2024-03-18T22:42:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2211427564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505abcff0 msgr2=0x557505acc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/2211427564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505abcff0 0x557505acc9d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5e7402cdd0 tx=0x7f5e74005fd0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2211427564 shutdown_connections 2024-03-18T22:42:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/2211427564 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557505b3e9d0 0x557505b3d6e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/2211427564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557505ac2b50 0x557505b385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e69ffb640 0 client.25349 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:42:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/2211427564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557505abd9e0 0x557505b36050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e8b7db740 1 --2- 172.21.15.40:0/2211427564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557505abcff0 0x557505acc9d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2211427564 >> 172.21.15.40:0/2211427564 conn(0x557505ab8560 msgr2=0x557505ad2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2211427564 shutdown_connections 2024-03-18T22:42:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e8b7db740 1 -- 172.21.15.40:0/2211427564 wait complete. 2024-03-18T22:42:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.888+0000 7f5e8b7db740 20 client.25349 trim_cache size 0 max 16384 2024-03-18T22:42:56.898 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:42:56.898 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "410ef414-e577-11ee-95c9-87774f69a715", 2024-03-18T22:42:56.898 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-03-18T22:42:56.898 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24481, 2024-03-18T22:42:56.898 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-18T22:42:56.898 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-18T22:42:56.898 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-18T22:42:56.898 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 110.86027737000001, 2024-03-18T22:42:56.898 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 27, 2024-03-18T22:42:56.898 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 93, 2024-03-18T22:42:56.898 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T22:42:56.898 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 147.68420626400001 2024-03-18T22:42:56.898 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:42:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.889+0000 7f5e88bf3640 1 -- 172.21.15.40:0/864828522 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5e6c0a2800 msgr2=0x7f5e6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.889+0000 7f5e88bf3640 1 --2- 172.21.15.40:0/864828522 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5e6c0a2800 0x7f5e6c0a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x557505abe240 tx=0x7f5e7c009500 comp rx=0 tx=0).stop 2024-03-18T22:42:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.889+0000 7f5e88bf3640 1 -- 172.21.15.40:0/864828522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e84100a20 msgr2=0x7f5e841047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.889+0000 7f5e88bf3640 1 --2- 172.21.15.40:0/864828522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e84100a20 0x7f5e841047d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5e74000c00 tx=0x7f5e74005e70 comp rx=0 tx=0).stop 2024-03-18T22:42:56.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.889+0000 7f5e88bf3640 1 -- 172.21.15.40:0/864828522 shutdown_connections 2024-03-18T22:42:56.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.889+0000 7f5e88bf3640 1 --2- 172.21.15.40:0/864828522 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5e6c0a2800 0x7f5e6c0a4cc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.889+0000 7f5e88bf3640 1 --2- 172.21.15.40:0/864828522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e84101cf0 0x7f5e84103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.889+0000 7f5e88bf3640 1 --2- 172.21.15.40:0/864828522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e841013e0 0x7f5e84102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.889+0000 7f5e88bf3640 1 --2- 172.21.15.40:0/864828522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e84100a20 0x7f5e841047d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:56.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.889+0000 7f5e88bf3640 1 -- 172.21.15.40:0/864828522 >> 172.21.15.40:0/864828522 conn(0x7f5e840fb450 msgr2=0x7f5e840ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:56.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.889+0000 7f5e88bf3640 1 -- 172.21.15.40:0/864828522 shutdown_connections 2024-03-18T22:42:56.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:56.889+0000 7f5e88bf3640 1 -- 172.21.15.40:0/864828522 wait complete. 2024-03-18T22:42:56.901 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T22:42:57.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.004+0000 7f2416e59640 1 -- 172.21.15.40:0/1554882947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24101033e0 msgr2=0x7f2410109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.004+0000 7f2416e59640 1 --2- 172.21.15.40:0/1554882947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24101033e0 0x7f2410109c70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009900 tx=0x7f23f802bc30 comp rx=0 tx=0).stop 2024-03-18T22:42:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.004+0000 7f2416e59640 1 -- 172.21.15.40:0/1554882947 shutdown_connections 2024-03-18T22:42:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.004+0000 7f2416e59640 1 --2- 172.21.15.40:0/1554882947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f24101033e0 0x7f2410109c70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.004+0000 7f2416e59640 1 --2- 172.21.15.40:0/1554882947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2410102a20 0x7f2410102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.004+0000 7f2416e59640 1 --2- 172.21.15.40:0/1554882947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2410101830 0x7f2410101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.004+0000 7f2416e59640 1 -- 172.21.15.40:0/1554882947 >> 172.21.15.40:0/1554882947 conn(0x7f24100fcdc0 msgr2=0x7f24100ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.004+0000 7f2416e59640 1 -- 172.21.15.40:0/1554882947 shutdown_connections 2024-03-18T22:42:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.004+0000 7f2416e59640 1 -- 172.21.15.40:0/1554882947 wait complete. 2024-03-18T22:42:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.004+0000 7f2416e59640 1 Processor -- start 2024-03-18T22:42:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f2416e59640 1 -- start start 2024-03-18T22:42:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f2416e59640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2410101830 0x7f2410197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f2416e59640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2410102a20 0x7f2410197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f2416e59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24101033e0 0x7f241019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f2416e59640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f241010d360 con 0x7f24101033e0 2024-03-18T22:42:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f2416e59640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f241010d1e0 con 0x7f2410101830 2024-03-18T22:42:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f2416e59640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f241010d4e0 con 0x7f2410102a20 2024-03-18T22:42:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f24153f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24101033e0 0x7f241019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f240ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2410102a20 0x7f2410197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f240ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2410102a20 0x7f2410197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:40856/0 (socket says 172.21.15.40:40856) 2024-03-18T22:42:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f2414bf5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2410101830 0x7f2410197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f240ffff640 1 -- 172.21.15.40:0/3505309927 learned_addr learned my addr 172.21.15.40:0/3505309927 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f240ffff640 1 -- 172.21.15.40:0/3505309927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2410101830 msgr2=0x7f2410197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f240ffff640 1 --2- 172.21.15.40:0/3505309927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2410101830 0x7f2410197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f240ffff640 1 -- 172.21.15.40:0/3505309927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24101033e0 msgr2=0x7f241019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f240ffff640 1 --2- 172.21.15.40:0/3505309927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24101033e0 0x7f241019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f240ffff640 1 -- 172.21.15.40:0/3505309927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f241019f650 con 0x7f2410102a20 2024-03-18T22:42:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f240ffff640 1 --2- 172.21.15.40:0/3505309927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2410102a20 0x7f2410197f70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f240000e9f0 tx=0x7f240000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.005+0000 7f240dffb640 1 -- 172.21.15.40:0/3505309927 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240000cd80 con 0x7f2410102a20 2024-03-18T22:42:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.006+0000 7f2416e59640 1 -- 172.21.15.40:0/3505309927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f241019f940 con 0x7f2410102a20 2024-03-18T22:42:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.006+0000 7f240dffb640 1 -- 172.21.15.40:0/3505309927 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24000108e0 con 0x7f2410102a20 2024-03-18T22:42:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.006+0000 7f240dffb640 1 -- 172.21.15.40:0/3505309927 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2400018920 con 0x7f2410102a20 2024-03-18T22:42:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.006+0000 7f2416e59640 1 -- 172.21.15.40:0/3505309927 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f241019fee0 con 0x7f2410102a20 2024-03-18T22:42:57.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.008+0000 7f240dffb640 1 -- 172.21.15.40:0/3505309927 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2400010a90 con 0x7f2410102a20 2024-03-18T22:42:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.008+0000 7f2419a44740 1 Processor -- start 2024-03-18T22:42:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.008+0000 7f2419a44740 1 -- start start 2024-03-18T22:42:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.008+0000 7f240dffb640 1 --2- 172.21.15.40:0/3505309927 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f23fc0a2800 0x7f23fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.008+0000 7f2419a44740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3caff0 0x55c41f3cb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.008+0000 7f2419a44740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3cb9e0 0x55c41f3d04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.008+0000 7f2419a44740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3d0b50 0x55c41f3d2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.008+0000 7f2419a44740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c41f32fde0 con 0x55c41f3d0b50 2024-03-18T22:42:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.008+0000 7f2419a44740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c41f2d8c80 con 0x55c41f3caff0 2024-03-18T22:42:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.008+0000 7f2419a44740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c41f32be80 con 0x55c41f3cb9e0 2024-03-18T22:42:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.008+0000 7f240dffb640 1 -- 172.21.15.40:0/3505309927 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f24000cf140 con 0x7f2410102a20 2024-03-18T22:42:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.008+0000 7f24153f6640 1 --2- 172.21.15.40:0/3505309927 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f23fc0a2800 0x7f23fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.009+0000 7f2414bf5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3caff0 0x55c41f3cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.009+0000 7f2414bf5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3caff0 0x55c41f3cb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:44742/0 (socket says 172.21.15.40:44742) 2024-03-18T22:42:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.009+0000 7f240ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3cb9e0 0x55c41f3d04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.009+0000 7f2414bf5640 1 -- 172.21.15.40:0/1903623266 learned_addr learned my addr 172.21.15.40:0/1903623266 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.009+0000 7f24153f6640 1 --2- 172.21.15.40:0/1903623266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3d0b50 0x55c41f3d2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.009+0000 7f240ffff640 1 -- 172.21.15.40:0/1903623266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3caff0 msgr2=0x55c41f3cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.009+0000 7f240ffff640 1 --2- 172.21.15.40:0/1903623266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3caff0 0x55c41f3cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.009+0000 7f240ffff640 1 -- 172.21.15.40:0/1903623266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3d0b50 msgr2=0x55c41f3d2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.009+0000 7f240ffff640 1 --2- 172.21.15.40:0/1903623266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3d0b50 0x55c41f3d2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.009+0000 7f240ffff640 1 -- 172.21.15.40:0/1903623266 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c41f2eafb0 con 0x55c41f3cb9e0 2024-03-18T22:42:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.009+0000 7f24153f6640 1 --2- 172.21.15.40:0/3505309927 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f23fc0a2800 0x7f23fc0a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f23f8006010 tx=0x7f23f8005e60 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.009+0000 7f240ffff640 1 --2- 172.21.15.40:0/1903623266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3cb9e0 0x55c41f3d04e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f24000cc850 tx=0x7f2400010ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc4d3887f243c43f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.009+0000 7f2406ffd640 1 -- 172.21.15.40:0/1903623266 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24000a3e90 con 0x55c41f3cb9e0 2024-03-18T22:42:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.009+0000 7f2406ffd640 1 -- 172.21.15.40:0/1903623266 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2400099aa0 con 0x55c41f3cb9e0 2024-03-18T22:42:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.009+0000 7f2406ffd640 1 -- 172.21.15.40:0/1903623266 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24000974f0 con 0x55c41f3cb9e0 2024-03-18T22:42:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.010+0000 7f2419a44740 1 -- 172.21.15.40:0/1903623266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3cb9e0 msgr2=0x55c41f3d04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.010+0000 7f2419a44740 1 --2- 172.21.15.40:0/1903623266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3cb9e0 0x55c41f3d04e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f24000cc850 tx=0x7f2400010ea0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.010+0000 7f2419a44740 1 -- 172.21.15.40:0/1903623266 shutdown_connections 2024-03-18T22:42:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.010+0000 7f2419a44740 1 --2- 172.21.15.40:0/1903623266 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3d0b50 0x55c41f3d2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.010+0000 7f2419a44740 1 --2- 172.21.15.40:0/1903623266 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3cb9e0 0x55c41f3d04e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.010+0000 7f2419a44740 1 --2- 172.21.15.40:0/1903623266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3caff0 0x55c41f3cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.010+0000 7f2419a44740 1 -- 172.21.15.40:0/1903623266 >> 172.21.15.40:0/1903623266 conn(0x55c41f3c6560 msgr2=0x55c41f3c6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.010+0000 7f2419a44740 1 -- 172.21.15.40:0/1903623266 shutdown_connections 2024-03-18T22:42:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.010+0000 7f2419a44740 1 -- 172.21.15.40:0/1903623266 wait complete. 2024-03-18T22:42:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.010+0000 7f2419a44740 1 -- start start 2024-03-18T22:42:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.010+0000 7f2419a44740 1 Processor -- start 2024-03-18T22:42:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f2419a44740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3caff0 0x55c41f3d5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f2419a44740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3cb9e0 0x55c41f3d5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f2419a44740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3d0b50 0x55c41f3dcac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f2419a44740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c41f3d3b00 con 0x55c41f3d0b50 2024-03-18T22:42:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f2419a44740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c41f2ebfc0 con 0x55c41f3cb9e0 2024-03-18T22:42:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f2419a44740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c41f3d3c80 con 0x55c41f3caff0 2024-03-18T22:42:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f2419a44740 10 client.0 _ll_register_callbacks cb 0x55c41f327a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f24153f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3d0b50 0x55c41f3dcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f24153f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3d0b50 0x55c41f3dcac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39736/0 (socket says 172.21.15.40:39736) 2024-03-18T22:42:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f24153f6640 1 -- 172.21.15.40:0/3081244945 learned_addr learned my addr 172.21.15.40:0/3081244945 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f240ffff640 1 --2- 172.21.15.40:0/3081244945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3cb9e0 0x55c41f3d5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f2414bf5640 1 --2- 172.21.15.40:0/3081244945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3caff0 0x55c41f3d5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f24153f6640 1 -- 172.21.15.40:0/3081244945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3caff0 msgr2=0x55c41f3d5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f24153f6640 1 --2- 172.21.15.40:0/3081244945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3caff0 0x55c41f3d5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f24153f6640 1 -- 172.21.15.40:0/3081244945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3cb9e0 msgr2=0x55c41f3d5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f24153f6640 1 --2- 172.21.15.40:0/3081244945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3cb9e0 0x55c41f3d5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f24153f6640 1 -- 172.21.15.40:0/3081244945 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c41f0442d0 con 0x55c41f3d0b50 2024-03-18T22:42:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f24153f6640 1 --2- 172.21.15.40:0/3081244945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3d0b50 0x55c41f3dcac0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f23f8035000 tx=0x7f23f802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e6895f6cf4dc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f23effff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f23effff640 1 -- 172.21.15.40:0/3081244945 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f8055060 con 0x55c41f3d0b50 2024-03-18T22:42:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f23effff640 1 -- 172.21.15.40:0/3081244945 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f23f8004bc0 con 0x55c41f3d0b50 2024-03-18T22:42:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.011+0000 7f2419a44740 1 -- 172.21.15.40:0/3081244945 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2400099aa0 con 0x55c41f3d0b50 2024-03-18T22:42:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.012+0000 7f23effff640 1 -- 172.21.15.40:0/3081244945 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f23f805c050 con 0x55c41f3d0b50 2024-03-18T22:42:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.012+0000 7f2419a44740 10 client.15471 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.012+0000 7f2419a44740 1 -- 172.21.15.40:0/3081244945 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c41f10e6b0 con 0x55c41f3d0b50 2024-03-18T22:42:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.012+0000 7f23effff640 1 -- 172.21.15.40:0/3081244945 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2400099aa0 con 0x55c41f3d0b50 2024-03-18T22:42:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.012+0000 7f23effff640 1 -- 172.21.15.40:0/3081244945 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f23f8043450 con 0x55c41f3d0b50 2024-03-18T22:42:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.012+0000 7f2419a44740 10 client.15471 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.012+0000 7f2419a44740 10 client.15471 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T22:42:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.012+0000 7f2419a44740 20 client.15471 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.012+0000 7f2419a44740 1 --2- 172.21.15.40:0/3081244945 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c41f3e23e0 0x55c41f3e3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.012+0000 7f2419a44740 4 client.15471 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:42:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.012+0000 7f2419a44740 1 -- 172.21.15.40:0/3081244945 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c41f33d320 con 0x55c41f3e23e0 2024-03-18T22:42:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.012+0000 7f2414bf5640 1 --2- 172.21.15.40:0/3081244945 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c41f3e23e0 0x55c41f3e3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.013+0000 7f23effff640 10 client.15471 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:42:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.013+0000 7f2414bf5640 1 --2- 172.21.15.40:0/3081244945 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c41f3e23e0 0x55c41f3e3850 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53e499463a847cf server_cookie=2adf40382265e776 in_seq=0 out_seq=0 2024-03-18T22:42:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.014+0000 7f23effff640 1 -- 172.21.15.40:0/3081244945 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c41f33d320 con 0x55c41f3e23e0 2024-03-18T22:42:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.014+0000 7f23effff640 10 client.15471 handle_command_reply: tid=0 2024-03-18T22:42:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.014+0000 7f2419a44740 1 client.15471 shutdown 2024-03-18T22:42:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.014+0000 7f2419a44740 1 -- 172.21.15.40:0/3081244945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3d0b50 msgr2=0x55c41f3dcac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.014+0000 7f2419a44740 1 --2- 172.21.15.40:0/3081244945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3d0b50 0x55c41f3dcac0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f23f8035000 tx=0x7f23f802c7e0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.014+0000 7f2419a44740 1 -- 172.21.15.40:0/3081244945 shutdown_connections 2024-03-18T22:42:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.014+0000 7f2419a44740 1 --2- 172.21.15.40:0/3081244945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3d0b50 0x55c41f3dcac0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.014+0000 7f2419a44740 1 --2- 172.21.15.40:0/3081244945 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c41f3e23e0 0x55c41f3e3850 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.014+0000 7f2419a44740 1 --2- 172.21.15.40:0/3081244945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3cb9e0 0x55c41f3d5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.014+0000 7f23effff640 0 client.15471 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:42:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.014+0000 7f2419a44740 1 --2- 172.21.15.40:0/3081244945 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3caff0 0x55c41f3d5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.014+0000 7f2419a44740 1 -- 172.21.15.40:0/3081244945 >> 172.21.15.40:0/3081244945 conn(0x55c41f3c6560 msgr2=0x55c41f3c57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.014+0000 7f2419a44740 1 -- 172.21.15.40:0/3081244945 shutdown_connections 2024-03-18T22:42:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.014+0000 7f2419a44740 1 -- 172.21.15.40:0/3081244945 wait complete. 2024-03-18T22:42:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.014+0000 7f2419a44740 20 client.15471 trim_cache size 0 max 16384 2024-03-18T22:42:57.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.032+0000 7f2419a44740 1 Processor -- start 2024-03-18T22:42:57.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.032+0000 7f2419a44740 1 -- start start 2024-03-18T22:42:57.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.032+0000 7f2419a44740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3caff0 0x55c41f3e11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.032+0000 7f2419a44740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3cb9e0 0x55c41f3dc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.032+0000 7f2419a44740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3d0b50 0x55c41f3c6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.032+0000 7f2419a44740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c41f3e1b90 con 0x55c41f3cb9e0 2024-03-18T22:42:57.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.032+0000 7f2419a44740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c41f3e2260 con 0x55c41f3d0b50 2024-03-18T22:42:57.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.032+0000 7f2419a44740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c41f3e2930 con 0x55c41f3caff0 2024-03-18T22:42:57.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.032+0000 7f240ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3cb9e0 0x55c41f3dc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.032+0000 7f240ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3cb9e0 0x55c41f3dc9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39742/0 (socket says 172.21.15.40:39742) 2024-03-18T22:42:57.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.032+0000 7f240ffff640 1 -- 172.21.15.40:0/299155609 learned_addr learned my addr 172.21.15.40:0/299155609 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:57.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.032+0000 7f24153f6640 1 --2- 172.21.15.40:0/299155609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3d0b50 0x55c41f3c6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.033+0000 7f2414bf5640 1 --2- 172.21.15.40:0/299155609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3caff0 0x55c41f3e11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.033+0000 7f240ffff640 1 -- 172.21.15.40:0/299155609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3caff0 msgr2=0x55c41f3e11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.033+0000 7f240ffff640 1 --2- 172.21.15.40:0/299155609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3caff0 0x55c41f3e11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.033+0000 7f240ffff640 1 -- 172.21.15.40:0/299155609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3d0b50 msgr2=0x55c41f3c6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.033+0000 7f240ffff640 1 --2- 172.21.15.40:0/299155609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3d0b50 0x55c41f3c6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.033+0000 7f240ffff640 1 -- 172.21.15.40:0/299155609 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c41f3dfd70 con 0x55c41f3cb9e0 2024-03-18T22:42:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.033+0000 7f240ffff640 1 --2- 172.21.15.40:0/299155609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3cb9e0 0x55c41f3dc9e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f240009b270 tx=0x7f2400097b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=584b8147a8fc97fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.033+0000 7f24067fc640 1 -- 172.21.15.40:0/299155609 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24000a3e90 con 0x55c41f3cb9e0 2024-03-18T22:42:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.033+0000 7f24067fc640 1 -- 172.21.15.40:0/299155609 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24000860d0 con 0x55c41f3cb9e0 2024-03-18T22:42:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.033+0000 7f24067fc640 1 -- 172.21.15.40:0/299155609 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24000974f0 con 0x55c41f3cb9e0 2024-03-18T22:42:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.034+0000 7f2419a44740 1 -- 172.21.15.40:0/299155609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3cb9e0 msgr2=0x55c41f3dc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.034+0000 7f2419a44740 1 --2- 172.21.15.40:0/299155609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3cb9e0 0x55c41f3dc9e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f240009b270 tx=0x7f2400097b70 comp rx=0 tx=0).stop 2024-03-18T22:42:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.034+0000 7f2419a44740 1 -- 172.21.15.40:0/299155609 shutdown_connections 2024-03-18T22:42:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.034+0000 7f2419a44740 1 --2- 172.21.15.40:0/299155609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3d0b50 0x55c41f3c6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.034+0000 7f2419a44740 1 --2- 172.21.15.40:0/299155609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3cb9e0 0x55c41f3dc9e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.034+0000 7f2419a44740 1 --2- 172.21.15.40:0/299155609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3caff0 0x55c41f3e11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.034+0000 7f2419a44740 1 -- 172.21.15.40:0/299155609 >> 172.21.15.40:0/299155609 conn(0x55c41f3c6560 msgr2=0x55c41f3c4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.034+0000 7f2419a44740 1 -- 172.21.15.40:0/299155609 shutdown_connections 2024-03-18T22:42:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.034+0000 7f2419a44740 1 -- 172.21.15.40:0/299155609 wait complete. 2024-03-18T22:42:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.034+0000 7f2419a44740 1 -- start start 2024-03-18T22:42:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.034+0000 7f2419a44740 1 Processor -- start 2024-03-18T22:42:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f2419a44740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3caff0 0x55c41f3d8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f2419a44740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3cb9e0 0x55c41f3d8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f2419a44740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3d0b50 0x55c41f3dbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f2419a44740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c41f3e36d0 con 0x55c41f3cb9e0 2024-03-18T22:42:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f2419a44740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c41f3e3000 con 0x55c41f3d0b50 2024-03-18T22:42:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f2419a44740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c41f3e3da0 con 0x55c41f3caff0 2024-03-18T22:42:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f2419a44740 10 client.0 _ll_register_callbacks cb 0x7f23f8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:42:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f240ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3cb9e0 0x55c41f3d8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f2414bf5640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3caff0 0x55c41f3d8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f240ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3cb9e0 0x55c41f3d8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39758/0 (socket says 172.21.15.40:39758) 2024-03-18T22:42:57.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f240ffff640 1 -- 172.21.15.40:0/1078095191 learned_addr learned my addr 172.21.15.40:0/1078095191 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:57.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f24153f6640 1 --2- 172.21.15.40:0/1078095191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3d0b50 0x55c41f3dbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f240ffff640 1 -- 172.21.15.40:0/1078095191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3caff0 msgr2=0x55c41f3d8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:57.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f240ffff640 1 --2- 172.21.15.40:0/1078095191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3caff0 0x55c41f3d8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f240ffff640 1 -- 172.21.15.40:0/1078095191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3d0b50 msgr2=0x55c41f3dbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:42:57.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f240ffff640 1 --2- 172.21.15.40:0/1078095191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3d0b50 0x55c41f3dbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f240ffff640 1 -- 172.21.15.40:0/1078095191 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c41f3dc150 con 0x55c41f3cb9e0 2024-03-18T22:42:57.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f240ffff640 1 --2- 172.21.15.40:0/1078095191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3cb9e0 0x55c41f3d8b90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f240009b240 tx=0x7f2400097a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1989a3d8d0d6559 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:57.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f23effff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:42:57.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f23effff640 1 -- 172.21.15.40:0/1078095191 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2400010630 con 0x55c41f3cb9e0 2024-03-18T22:42:57.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f23effff640 1 -- 172.21.15.40:0/1078095191 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f24000974f0 con 0x55c41f3cb9e0 2024-03-18T22:42:57.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f2419a44740 1 -- 172.21.15.40:0/1078095191 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24000860d0 con 0x55c41f3cb9e0 2024-03-18T22:42:57.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f23effff640 1 -- 172.21.15.40:0/1078095191 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2400099aa0 con 0x55c41f3cb9e0 2024-03-18T22:42:57.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f2419a44740 10 client.15477 fetch_fsmap learned FSMap version 27 2024-03-18T22:42:57.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.035+0000 7f2419a44740 1 -- 172.21.15.40:0/1078095191 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c41f448630 con 0x55c41f3cb9e0 2024-03-18T22:42:57.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.036+0000 7f23effff640 1 -- 172.21.15.40:0/1078095191 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24000860d0 con 0x55c41f3cb9e0 2024-03-18T22:42:57.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.036+0000 7f23effff640 1 -- 172.21.15.40:0/1078095191 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f2400018ac0 con 0x55c41f3cb9e0 2024-03-18T22:42:57.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.036+0000 7f2419a44740 10 client.15477 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T22:42:57.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.036+0000 7f2419a44740 10 client.15477 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T22:42:57.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.036+0000 7f2419a44740 20 client.15477 populate_metadata read hostname 'smithi040' 2024-03-18T22:42:57.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.036+0000 7f2419a44740 1 --2- 172.21.15.40:0/1078095191 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c41f44ca30 0x55c41f44b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:57.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.036+0000 7f2419a44740 4 client.15477 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T22:42:57.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.036+0000 7f2419a44740 1 -- 172.21.15.40:0/1078095191 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "status"}) -- 0x55c41f337670 con 0x55c41f44ca30 2024-03-18T22:42:57.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.036+0000 7f2414bf5640 1 --2- 172.21.15.40:0/1078095191 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c41f44ca30 0x55c41f44b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:57.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.036+0000 7f23effff640 10 client.15477 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:42:57.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.036+0000 7f2414bf5640 1 --2- 172.21.15.40:0/1078095191 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c41f44ca30 0x55c41f44b740 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b8a93271f5c804b server_cookie=a2079cf8bb893cb9 in_seq=0 out_seq=0 2024-03-18T22:42:57.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.037+0000 7f23effff640 1 -- 172.21.15.40:0/1078095191 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55c41f337670 con 0x55c41f44ca30 2024-03-18T22:42:57.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.037+0000 7f23effff640 10 client.15477 handle_command_reply: tid=0 2024-03-18T22:42:57.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.037+0000 7f2419a44740 1 client.15477 shutdown 2024-03-18T22:42:57.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.037+0000 7f2419a44740 1 -- 172.21.15.40:0/1078095191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3cb9e0 msgr2=0x55c41f3d8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:57.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.037+0000 7f2419a44740 1 --2- 172.21.15.40:0/1078095191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3cb9e0 0x55c41f3d8b90 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f240009b240 tx=0x7f2400097a40 comp rx=0 tx=0).stop 2024-03-18T22:42:57.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.037+0000 7f2419a44740 1 -- 172.21.15.40:0/1078095191 shutdown_connections 2024-03-18T22:42:57.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.037+0000 7f2419a44740 1 --2- 172.21.15.40:0/1078095191 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c41f44ca30 0x55c41f44b740 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.037+0000 7f2419a44740 1 --2- 172.21.15.40:0/1078095191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c41f3d0b50 0x55c41f3dbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.037+0000 7f2419a44740 1 --2- 172.21.15.40:0/1078095191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c41f3cb9e0 0x55c41f3d8b90 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.037+0000 7f23effff640 0 client.15477 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:42:57.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.037+0000 7f2419a44740 1 --2- 172.21.15.40:0/1078095191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c41f3caff0 0x55c41f3d8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.037+0000 7f2419a44740 1 -- 172.21.15.40:0/1078095191 >> 172.21.15.40:0/1078095191 conn(0x55c41f3c6560 msgr2=0x55c41f3ca050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:57.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.037+0000 7f2419a44740 1 -- 172.21.15.40:0/1078095191 shutdown_connections 2024-03-18T22:42:57.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.037+0000 7f2419a44740 1 -- 172.21.15.40:0/1078095191 wait complete. 2024-03-18T22:42:57.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.037+0000 7f2419a44740 20 client.15477 trim_cache size 0 max 16384 2024-03-18T22:42:57.046 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:42:57.046 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "410ef414-e577-11ee-95c9-87774f69a715", 2024-03-18T22:42:57.046 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-18T22:42:57.046 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 24509, 2024-03-18T22:42:57.046 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-18T22:42:57.046 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-18T22:42:57.046 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-18T22:42:57.046 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 134.839271134, 2024-03-18T22:42:57.046 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 27, 2024-03-18T22:42:57.047 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 93, 2024-03-18T22:42:57.047 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T22:42:57.047 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 139.32213574100001 2024-03-18T22:42:57.047 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:42:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.038+0000 7f2416e59640 1 -- 172.21.15.40:0/3505309927 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f23fc0a2800 msgr2=0x7f23fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.038+0000 7f2416e59640 1 --2- 172.21.15.40:0/3505309927 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f23fc0a2800 0x7f23fc0a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f23f8006010 tx=0x7f23f8005e60 comp rx=0 tx=0).stop 2024-03-18T22:42:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.038+0000 7f2416e59640 1 -- 172.21.15.40:0/3505309927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2410102a20 msgr2=0x7f2410197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.038+0000 7f2416e59640 1 --2- 172.21.15.40:0/3505309927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2410102a20 0x7f2410197f70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f240000e9f0 tx=0x7f240000eeb0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.038+0000 7f2416e59640 1 -- 172.21.15.40:0/3505309927 shutdown_connections 2024-03-18T22:42:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.038+0000 7f2416e59640 1 --2- 172.21.15.40:0/3505309927 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f23fc0a2800 0x7f23fc0a4cc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.038+0000 7f2416e59640 1 --2- 172.21.15.40:0/3505309927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24101033e0 0x7f241019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.038+0000 7f2416e59640 1 --2- 172.21.15.40:0/3505309927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2410102a20 0x7f2410197f70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.038+0000 7f2416e59640 1 --2- 172.21.15.40:0/3505309927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2410101830 0x7f2410197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.038+0000 7f2416e59640 1 -- 172.21.15.40:0/3505309927 >> 172.21.15.40:0/3505309927 conn(0x7f24100fcdc0 msgr2=0x7f2410105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.038+0000 7f2416e59640 1 -- 172.21.15.40:0/3505309927 shutdown_connections 2024-03-18T22:42:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:57.038+0000 7f2416e59640 1 -- 172.21.15.40:0/3505309927 wait complete. 2024-03-18T22:42:57.050 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-18T22:42:57.061 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-18T22:42:57.061 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-03-18T22:42:57.061 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-03-18T22:42:57.061 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T22:42:57.061 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T22:42:57.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:42:57.062 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-18T22:42:57.078 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-18T22:42:57.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:42:57.078 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T22:42:57.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:42:57.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link delete ceph-brx 2024-03-18T22:42:57.153 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:57 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2059503345' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:57.215 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-18T22:42:57.215 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-18T22:42:57.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:57 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2059503345' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:57.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:57 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2059503345' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:42:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:42:58.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:42:58 smithi040 ceph-mon[62643]: pgmap v360: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s; 106 B/s, 1 objects/s recovering 2024-03-18T22:42:58.284 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:42:58 smithi103 ceph-mon[66377]: pgmap v360: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s; 106 B/s, 1 objects/s recovering 2024-03-18T22:42:58.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:42:58 smithi113 ceph-mon[66775]: pgmap v360: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s; 106 B/s, 1 objects/s recovering 2024-03-18T22:42:59.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.185+0000 7fd051755700 1 -- 172.21.15.40:0/3993856972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c101bb0 msgr2=0x7fd04c10e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:59.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.185+0000 7fd051755700 1 --2- 172.21.15.40:0/3993856972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c101bb0 0x7fd04c10e090 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd04400b0a0 tx=0x7fd04400b3b0 comp rx=0 tx=0).stop 2024-03-18T22:42:59.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.185+0000 7fd051755700 1 -- 172.21.15.40:0/3993856972 shutdown_connections 2024-03-18T22:42:59.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.185+0000 7fd051755700 1 --2- 172.21.15.40:0/3993856972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c101bb0 0x7fd04c10e090 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:59.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.185+0000 7fd051755700 1 --2- 172.21.15.40:0/3993856972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04c107250 0x7fd04c107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:59.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.185+0000 7fd051755700 1 --2- 172.21.15.40:0/3993856972 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd04c101210 0x7fd04c101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:59.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.185+0000 7fd051755700 1 -- 172.21.15.40:0/3993856972 >> 172.21.15.40:0/3993856972 conn(0x7fd04c078140 msgr2=0x7fd04c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:59.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.185+0000 7fd051755700 1 -- 172.21.15.40:0/3993856972 shutdown_connections 2024-03-18T22:42:59.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.185+0000 7fd051755700 1 -- 172.21.15.40:0/3993856972 wait complete. 2024-03-18T22:42:59.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.186+0000 7fd051755700 1 Processor -- start 2024-03-18T22:42:59.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.186+0000 7fd051755700 1 -- start start 2024-03-18T22:42:59.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd051755700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd04c101210 0x7fd04c19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:59.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd051755700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c101bb0 0x7fd04c19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:59.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd051755700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04c107250 0x7fd04c1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:59.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd051755700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c111380 con 0x7fd04c101bb0 2024-03-18T22:42:59.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd051755700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c111200 con 0x7fd04c107250 2024-03-18T22:42:59.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd051755700 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd04c111500 con 0x7fd04c101210 2024-03-18T22:42:59.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd04a7fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c101bb0 0x7fd04c19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd04a7fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c101bb0 0x7fd04c19eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37144/0 (socket says 172.21.15.40:37144) 2024-03-18T22:42:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd04a7fc700 1 -- 172.21.15.40:0/2292429094 learned_addr learned my addr 172.21.15.40:0/2292429094 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:42:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd04affd700 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd04c101210 0x7fd04c19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd04b7fe700 1 --2- 172.21.15.40:0/2292429094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04c107250 0x7fd04c1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd04a7fc700 1 -- 172.21.15.40:0/2292429094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd04c101210 msgr2=0x7fd04c19e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd04a7fc700 1 --2- 172.21.15.40:0/2292429094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd04c101210 0x7fd04c19e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd04a7fc700 1 -- 172.21.15.40:0/2292429094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04c107250 msgr2=0x7fd04c1987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd04a7fc700 1 --2- 172.21.15.40:0/2292429094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04c107250 0x7fd04c1987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd04a7fc700 1 -- 172.21.15.40:0/2292429094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd04c198f90 con 0x7fd04c101bb0 2024-03-18T22:42:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd04affd700 1 --2- 172.21.15.40:0/2292429094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd04c101210 0x7fd04c19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:42:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd04a7fc700 1 --2- 172.21.15.40:0/2292429094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c101bb0 0x7fd04c19eb70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd04000e9c0 tx=0x7fd04000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd033fff700 1 -- 172.21.15.40:0/2292429094 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04000cbd0 con 0x7fd04c101bb0 2024-03-18T22:42:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd051755700 1 -- 172.21.15.40:0/2292429094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd04c199280 con 0x7fd04c101bb0 2024-03-18T22:42:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd033fff700 1 -- 172.21.15.40:0/2292429094 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd04000ee60 con 0x7fd04c101bb0 2024-03-18T22:42:59.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.187+0000 7fd033fff700 1 -- 172.21.15.40:0/2292429094 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0400055b0 con 0x7fd04c101bb0 2024-03-18T22:42:59.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.188+0000 7fd051755700 1 -- 172.21.15.40:0/2292429094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd04c1a5490 con 0x7fd04c101bb0 2024-03-18T22:42:59.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.188+0000 7fd051755700 1 -- 172.21.15.40:0/2292429094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd04c102900 con 0x7fd04c101bb0 2024-03-18T22:42:59.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.189+0000 7fd033fff700 1 -- 172.21.15.40:0/2292429094 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd040004750 con 0x7fd04c101bb0 2024-03-18T22:42:59.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.190+0000 7fd033fff700 1 --2- 172.21.15.40:0/2292429094 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0340a2890 0x7fd0340a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:42:59.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.190+0000 7fd04affd700 1 --2- 172.21.15.40:0/2292429094 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0340a2890 0x7fd0340a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:42:59.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.190+0000 7fd033fff700 1 -- 172.21.15.40:0/2292429094 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd0400cf0c0 con 0x7fd04c101bb0 2024-03-18T22:42:59.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.191+0000 7fd04affd700 1 --2- 172.21.15.40:0/2292429094 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0340a2890 0x7fd0340a4d50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009fd0 tx=0x7fd03c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:42:59.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.195+0000 7fd033fff700 1 -- 172.21.15.40:0/2292429094 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd04009bd70 con 0x7fd04c101bb0 2024-03-18T22:42:59.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.447+0000 7fd051755700 1 -- 172.21.15.40:0/2292429094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd04c1a5720 con 0x7fd04c101bb0 2024-03-18T22:42:59.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.447+0000 7fd033fff700 1 -- 172.21.15.40:0/2292429094 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+88 (secure 0 0 0) 0x7fd04009b670 con 0x7fd04c101bb0 2024-03-18T22:42:59.449 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T22:42:59.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.450+0000 7fd051755700 1 -- 172.21.15.40:0/2292429094 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0340a2890 msgr2=0x7fd0340a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:59.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.450+0000 7fd051755700 1 --2- 172.21.15.40:0/2292429094 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0340a2890 0x7fd0340a4d50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009fd0 tx=0x7fd03c009260 comp rx=0 tx=0).stop 2024-03-18T22:42:59.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.451+0000 7fd051755700 1 -- 172.21.15.40:0/2292429094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c101bb0 msgr2=0x7fd04c19eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:42:59.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.451+0000 7fd051755700 1 --2- 172.21.15.40:0/2292429094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c101bb0 0x7fd04c19eb70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd04000e9c0 tx=0x7fd04000ecd0 comp rx=0 tx=0).stop 2024-03-18T22:42:59.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.451+0000 7fd051755700 1 -- 172.21.15.40:0/2292429094 shutdown_connections 2024-03-18T22:42:59.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.451+0000 7fd051755700 1 --2- 172.21.15.40:0/2292429094 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0340a2890 0x7fd0340a4d50 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:59.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.451+0000 7fd051755700 1 --2- 172.21.15.40:0/2292429094 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd04c107250 0x7fd04c1987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:59.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.451+0000 7fd051755700 1 --2- 172.21.15.40:0/2292429094 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd04c101210 0x7fd04c19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:59.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.451+0000 7fd051755700 1 --2- 172.21.15.40:0/2292429094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd04c101bb0 0x7fd04c19eb70 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:42:59.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.451+0000 7fd051755700 1 -- 172.21.15.40:0/2292429094 >> 172.21.15.40:0/2292429094 conn(0x7fd04c078140 msgr2=0x7fd04c06a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:42:59.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.451+0000 7fd051755700 1 -- 172.21.15.40:0/2292429094 shutdown_connections 2024-03-18T22:42:59.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:42:59.451+0000 7fd051755700 1 -- 172.21.15.40:0/2292429094 wait complete. 2024-03-18T22:43:00.027 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T22:43:00.028 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T22:43:00.028 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-03-18T22:43:00.028 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T22:43:00.028 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T22:43:00.028 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T22:43:00.028 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T22:43:00.028 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T22:43:00.028 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T22:43:00.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:43:00.028 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-03-18T22:43:00.045 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T22:43:00.045 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T22:43:00.045 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T22:43:00.045 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-18T22:43:00.045 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-03-18T22:43:00.045 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-18T22:43:00.045 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T22:43:00.045 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-03-18T22:43:00.045 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-03-18T22:43:00.045 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T22:43:00.046 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-03-18T22:43:00.046 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-03-18T22:43:00.046 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T22:43:00.046 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-03-18T22:43:00.046 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-03-18T22:43:00.046 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T22:43:00.046 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 39674sec preferred_lft 39674sec 2024-03-18T22:43:00.046 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link noprefixroute 2024-03-18T22:43:00.046 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-18T22:43:00.046 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T22:43:00.046 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-03-18T22:43:00.046 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-03-18T22:43:00.047 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T22:43:00.047 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T22:43:00.047 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-18T22:43:00.047 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add name ceph-brx type bridge 2024-03-18T22:43:00.047 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr flush dev ceph-brx 2024-03-18T22:43:00.047 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set ceph-brx up 2024-03-18T22:43:00.047 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T22:43:00.047 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-18T22:43:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T22:43:00.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T22:43:00.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:43:00.198 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T22:43:00.224 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-03-18T22:43:00.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:43:00.226 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-03-18T22:43:00.282 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-03-18T22:43:00.282 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T22:43:00.282 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T22:43:00.282 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T22:43:00.282 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T22:43:00.283 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T22:43:00.283 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-18T22:43:00.283 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T22:43:00.283 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T22:43:00.283 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T22:43:00.283 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-18T22:43:00.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:00 smithi113 ceph-mon[66775]: pgmap v361: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 96 B/s, 0 objects/s recovering 2024-03-18T22:43:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:00 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2292429094' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T22:43:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:00 smithi103 ceph-mon[66377]: pgmap v361: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 96 B/s, 0 objects/s recovering 2024-03-18T22:43:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:00 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2292429094' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T22:43:00.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T22:43:00.381 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:00 smithi040 ceph-mon[62643]: pgmap v361: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 96 B/s, 0 objects/s recovering 2024-03-18T22:43:00.381 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:00 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2292429094' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T22:43:00.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T22:43:00.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:43:00.418 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-18T22:43:00.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:43:00.433 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list-id 2024-03-18T22:43:00.488 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T22:43:00.488 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-18T22:43:00.488 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T22:43:00.488 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-18T22:43:00.488 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-18T22:43:00.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T22:43:00.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T22:43:00.611 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-18T22:43:00.611 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T22:43:00.611 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-18T22:43:00.611 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-18T22:43:00.611 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T22:43:00.611 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-18T22:43:00.611 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-18T22:43:00.611 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-18T22:43:00.611 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-18T22:43:00.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T22:43:00.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T22:43:00.740 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T22:43:00.740 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-18T22:43:00.740 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-03-18T22:43:00.740 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T22:43:00.740 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-18T22:43:00.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T22:43:00.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T22:43:00.822 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-03-18T22:43:00.822 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T22:43:00.822 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T22:43:00.837 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T22:43:00.837 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T22:43:00.837 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T22:43:00.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-18T22:43:00.960 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T22:43:01.016 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-18T22:43:01.016 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-18T22:43:01.016 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-18T22:43:01.016 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-18T22:43:01.016 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-18T22:43:01.016 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-18T22:43:01.016 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-18T22:43:01.016 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-18T22:43:01.016 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-18T22:43:01.016 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-18T22:43:01.017 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-18T22:43:01.017 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/selinux 2024-03-18T22:43:01.017 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T22:43:01.017 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-18T22:43:01.017 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-18T22:43:01.017 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-18T22:43:01.017 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-18T22:43:01.017 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T22:43:01.017 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-18T22:43:01.017 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-18T22:43:01.017 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T22:43:01.017 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T22:43:01.017 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T22:43:01.017 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-18T22:43:01.017 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-18T22:43:01.018 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay 2024-03-18T22:43:01.018 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/ffadd4dbcadfe7df76393e76b841afbc43fe1c6b0adb1d7adc056ea3cdd53b80/merged 2024-03-18T22:43:01.018 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/e5d1e4b8fdfb35be92f152a64ac0762ea523644962df8b9d571d94edbe59ddb1/merged 2024-03-18T22:43:01.018 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/0 2024-03-18T22:43:01.018 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T22:43:01.018 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/3c1ba48e38cf5203cf83e1c1d44d8ef91a0126a3906b39c3738ac667c369bc37/merged 2024-03-18T22:43:01.018 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/69a5ef89d1322a3b59be2d8ccdaea68c725b78aa7476835d0b8d6c8cf353c092/merged 2024-03-18T22:43:01.018 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/46ba45855c1c5875dfc6f00e3b2812f6978c6dc7e326ce36de4564840e3fbc8a/merged 2024-03-18T22:43:01.018 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/0ef730bb9f35d48fce64218d197e565540ec315beae77cd2ebb785b711b84283/merged 2024-03-18T22:43:01.018 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/107f9392d7587e7089941e3003d627a1a7436dcd3ddb79b4b8ddceec20f12a75/merged 2024-03-18T22:43:01.018 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/466af6a30704e702d0289e0940b69c651437cdc7e0388f8c1a73ae9945fdf6e4/merged 2024-03-18T22:43:01.018 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/c34a43cdd3aea4b12cfb8caf97ed276dc3fee813385dcdfaa3e559dfd432412b/merged 2024-03-18T22:43:01.018 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/457d84b9f6629e8752ef7eab7df5be8812b168869f1c8529c825e17d9df9ad41/merged 2024-03-18T22:43:01.018 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-18T22:43:01.019 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T22:43:01.019 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T22:43:01.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:43:01.019 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-18T22:43:01.071 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T22:43:01.071 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/bdd30de8-89b1-40ee-b382-961fdfc3fe80 --client_fs=cephfs) 2024-03-18T22:43:01.114 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-18T22:43:01.141 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T22:43:01.197 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-18T22:43:01.197 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-18T22:43:01.197 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-18T22:43:01.197 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-18T22:43:01.197 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-18T22:43:01.197 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-18T22:43:01.197 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-18T22:43:01.198 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-18T22:43:01.198 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-18T22:43:01.198 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-18T22:43:01.198 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-18T22:43:01.198 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/selinux 2024-03-18T22:43:01.198 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T22:43:01.198 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-18T22:43:01.198 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-18T22:43:01.198 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-18T22:43:01.198 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-18T22:43:01.198 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T22:43:01.198 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-18T22:43:01.199 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-18T22:43:01.199 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T22:43:01.199 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T22:43:01.199 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T22:43:01.199 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-18T22:43:01.200 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-18T22:43:01.200 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay 2024-03-18T22:43:01.200 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/ffadd4dbcadfe7df76393e76b841afbc43fe1c6b0adb1d7adc056ea3cdd53b80/merged 2024-03-18T22:43:01.200 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/e5d1e4b8fdfb35be92f152a64ac0762ea523644962df8b9d571d94edbe59ddb1/merged 2024-03-18T22:43:01.200 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/0 2024-03-18T22:43:01.200 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T22:43:01.200 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/3c1ba48e38cf5203cf83e1c1d44d8ef91a0126a3906b39c3738ac667c369bc37/merged 2024-03-18T22:43:01.200 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/69a5ef89d1322a3b59be2d8ccdaea68c725b78aa7476835d0b8d6c8cf353c092/merged 2024-03-18T22:43:01.201 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/46ba45855c1c5875dfc6f00e3b2812f6978c6dc7e326ce36de4564840e3fbc8a/merged 2024-03-18T22:43:01.201 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/0ef730bb9f35d48fce64218d197e565540ec315beae77cd2ebb785b711b84283/merged 2024-03-18T22:43:01.201 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/107f9392d7587e7089941e3003d627a1a7436dcd3ddb79b4b8ddceec20f12a75/merged 2024-03-18T22:43:01.201 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/466af6a30704e702d0289e0940b69c651437cdc7e0388f8c1a73ae9945fdf6e4/merged 2024-03-18T22:43:01.201 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/c34a43cdd3aea4b12cfb8caf97ed276dc3fee813385dcdfaa3e559dfd432412b/merged 2024-03-18T22:43:01.201 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/457d84b9f6629e8752ef7eab7df5be8812b168869f1c8529c825e17d9df9ad41/merged 2024-03-18T22:43:01.201 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-18T22:43:01.201 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T22:43:01.201 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T22:43:01.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:43:01.202 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-18T22:43:01.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:2024-03-18T22:43:01.198+0000 7fc778cb5480 -1 init, newargv = 0x56007fd902f0 newargc=15 2024-03-18T22:43:01.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[113620]: starting ceph client 2024-03-18T22:43:01.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[113620]: starting fuse 2024-03-18T22:43:02.319 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-18T22:43:02.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:02 smithi113 ceph-mon[66775]: pgmap v362: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 96 B/s, 0 objects/s recovering 2024-03-18T22:43:02.348 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:02 smithi040 ceph-mon[62643]: pgmap v362: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 96 B/s, 0 objects/s recovering 2024-03-18T22:43:02.353 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T22:43:02.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:02 smithi103 ceph-mon[66377]: pgmap v362: 129 pgs: 129 active+clean; 647 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 96 B/s, 0 objects/s recovering 2024-03-18T22:43:02.411 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-18T22:43:02.411 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-18T22:43:02.411 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-18T22:43:02.411 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-18T22:43:02.411 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-18T22:43:02.411 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-18T22:43:02.411 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-18T22:43:02.411 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-18T22:43:02.411 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-18T22:43:02.412 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-18T22:43:02.412 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-18T22:43:02.412 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/selinux 2024-03-18T22:43:02.412 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T22:43:02.412 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-18T22:43:02.412 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-18T22:43:02.412 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-18T22:43:02.412 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-18T22:43:02.412 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T22:43:02.412 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-18T22:43:02.412 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-18T22:43:02.412 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T22:43:02.412 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T22:43:02.412 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T22:43:02.413 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-18T22:43:02.413 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-18T22:43:02.413 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay 2024-03-18T22:43:02.413 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/ffadd4dbcadfe7df76393e76b841afbc43fe1c6b0adb1d7adc056ea3cdd53b80/merged 2024-03-18T22:43:02.413 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/e5d1e4b8fdfb35be92f152a64ac0762ea523644962df8b9d571d94edbe59ddb1/merged 2024-03-18T22:43:02.413 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/0 2024-03-18T22:43:02.413 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T22:43:02.413 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/3c1ba48e38cf5203cf83e1c1d44d8ef91a0126a3906b39c3738ac667c369bc37/merged 2024-03-18T22:43:02.413 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/69a5ef89d1322a3b59be2d8ccdaea68c725b78aa7476835d0b8d6c8cf353c092/merged 2024-03-18T22:43:02.413 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/46ba45855c1c5875dfc6f00e3b2812f6978c6dc7e326ce36de4564840e3fbc8a/merged 2024-03-18T22:43:02.413 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/0ef730bb9f35d48fce64218d197e565540ec315beae77cd2ebb785b711b84283/merged 2024-03-18T22:43:02.413 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/107f9392d7587e7089941e3003d627a1a7436dcd3ddb79b4b8ddceec20f12a75/merged 2024-03-18T22:43:02.413 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/466af6a30704e702d0289e0940b69c651437cdc7e0388f8c1a73ae9945fdf6e4/merged 2024-03-18T22:43:02.413 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/c34a43cdd3aea4b12cfb8caf97ed276dc3fee813385dcdfaa3e559dfd432412b/merged 2024-03-18T22:43:02.413 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/containers/storage/overlay/457d84b9f6629e8752ef7eab7df5be8812b168869f1c8529c825e17d9df9ad41/merged 2024-03-18T22:43:02.414 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-18T22:43:02.414 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T22:43:02.414 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T22:43:02.414 INFO:teuthology.orchestra.run.smithi040.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-18T22:43:02.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:43:02.414 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-18T22:43:02.468 INFO:teuthology.orchestra.run.smithi040.stdout:119 2024-03-18T22:43:02.468 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-18T22:43:02.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo stdin-killer -- python3 -c ' 2024-03-18T22:43:02.468 DEBUG:teuthology.orchestra.run.smithi040:> import glob 2024-03-18T22:43:02.468 DEBUG:teuthology.orchestra.run.smithi040:> import re 2024-03-18T22:43:02.468 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-03-18T22:43:02.468 DEBUG:teuthology.orchestra.run.smithi040:> import subprocess 2024-03-18T22:43:02.468 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-18T22:43:02.468 DEBUG:teuthology.orchestra.run.smithi040:> def _find_admin_socket(client_name): 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> files = glob.glob(asok_path) 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> # Given a non-glob path, it better be there 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> if "*" not in asok_path: 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> assert(len(files) == 1) 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> return files[0] 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> for f in files: 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> contents = proc_f.read() 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> if mountpoint in contents: 2024-03-18T22:43:02.469 DEBUG:teuthology.orchestra.run.smithi040:> return f 2024-03-18T22:43:02.470 DEBUG:teuthology.orchestra.run.smithi040:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T22:43:02.470 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-18T22:43:02.470 DEBUG:teuthology.orchestra.run.smithi040:> print(_find_admin_socket("client.0")) 2024-03-18T22:43:02.470 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-03-18T22:43:02.598 INFO:teuthology.orchestra.run.smithi040.stdout:/var/run/ceph/ceph-client.0.113620.asok 2024-03-18T22:43:02.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T22:43:02.610 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.113620.asok 2024-03-18T22:43:02.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:43:02.610 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.113620.asok status 2024-03-18T22:43:02.750 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:43:02.750 INFO:teuthology.orchestra.run.smithi040.stdout: "metadata": { 2024-03-18T22:43:02.750 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_sha1": "b6ae24918b03b1b8a19c9263a50655bba80bb810", 2024-03-18T22:43:02.750 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-18T22:43:02.750 INFO:teuthology.orchestra.run.smithi040.stdout: "entity_id": "0", 2024-03-18T22:43:02.750 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": "smithi040", 2024-03-18T22:43:02.750 INFO:teuthology.orchestra.run.smithi040.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-18T22:43:02.750 INFO:teuthology.orchestra.run.smithi040.stdout: "pid": "113620", 2024-03-18T22:43:02.750 INFO:teuthology.orchestra.run.smithi040.stdout: "root": "/volumes/qa/sv_0/bdd30de8-89b1-40ee-b382-961fdfc3fe80" 2024-03-18T22:43:02.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:43:02.750 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_count": 0, 2024-03-18T22:43:02.751 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_pinned_count": 0, 2024-03-18T22:43:02.751 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 15489, 2024-03-18T22:43:02.751 INFO:teuthology.orchestra.run.smithi040.stdout: "inst": { 2024-03-18T22:43:02.751 INFO:teuthology.orchestra.run.smithi040.stdout: "name": { 2024-03-18T22:43:02.751 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "client", 2024-03-18T22:43:02.751 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 15489 2024-03-18T22:43:02.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:43:02.751 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-18T22:43:02.751 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-18T22:43:02.751 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-18T22:43:02.751 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2905687954 2024-03-18T22:43:02.751 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:43:02.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:43:02.751 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-18T22:43:02.752 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-18T22:43:02.752 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-18T22:43:02.752 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2905687954 2024-03-18T22:43:02.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:43:02.752 INFO:teuthology.orchestra.run.smithi040.stdout: "inst_str": "client.15489 192.168.0.1:0/2905687954", 2024-03-18T22:43:02.752 INFO:teuthology.orchestra.run.smithi040.stdout: "addr_str": "192.168.0.1:0/2905687954", 2024-03-18T22:43:02.752 INFO:teuthology.orchestra.run.smithi040.stdout: "inode_count": 5, 2024-03-18T22:43:02.752 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_epoch": 27, 2024-03-18T22:43:02.752 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch": 93, 2024-03-18T22:43:02.752 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch_barrier": 0, 2024-03-18T22:43:02.752 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklisted": false, 2024-03-18T22:43:02.754 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs" 2024-03-18T22:43:02.754 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:43:02.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:43:02.759 DEBUG:teuthology.orchestra.run.smithi040:> stat --file-system '--printf=%T 2024-03-18T22:43:02.760 DEBUG:teuthology.orchestra.run.smithi040:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-18T22:43:02.774 INFO:teuthology.orchestra.run.smithi040.stdout:fuseblk 2024-03-18T22:43:02.775 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-18T22:43:02.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:43:02.775 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T22:43:02.844 INFO:teuthology.run_tasks:Running task exec... 2024-03-18T22:43:02.854 INFO:teuthology.task.exec:Executing custom commands... 2024-03-18T22:43:02.854 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi040.front.sepia.ceph.com 2024-03-18T22:43:02.854 DEBUG:teuthology.orchestra.run.smithi040:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-18T22:43:02.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.987+0000 7f2552885640 1 -- 172.21.15.40:0/3129525208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f254c100740 msgr2=0x7f254c100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:02.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.987+0000 7f2552885640 1 --2- 172.21.15.40:0/3129525208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f254c100740 0x7f254c100b60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f253c009900 tx=0x7f253c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:43:02.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.987+0000 7f2552885640 1 -- 172.21.15.40:0/3129525208 shutdown_connections 2024-03-18T22:43:02.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.987+0000 7f2552885640 1 --2- 172.21.15.40:0/3129525208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f254c1022f0 0x7f254c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:02.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.987+0000 7f2552885640 1 --2- 172.21.15.40:0/3129525208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f254c101930 0x7f254c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:02.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.987+0000 7f2552885640 1 --2- 172.21.15.40:0/3129525208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f254c100740 0x7f254c100b60 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:02.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.987+0000 7f2552885640 1 -- 172.21.15.40:0/3129525208 >> 172.21.15.40:0/3129525208 conn(0x7f254c0fbcb0 msgr2=0x7f254c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:02.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.987+0000 7f2552885640 1 -- 172.21.15.40:0/3129525208 shutdown_connections 2024-03-18T22:43:02.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.987+0000 7f2552885640 1 -- 172.21.15.40:0/3129525208 wait complete. 2024-03-18T22:43:02.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.987+0000 7f2552885640 1 Processor -- start 2024-03-18T22:43:02.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f2552885640 1 -- start start 2024-03-18T22:43:02.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f2552885640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f254c100740 0x7f254c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:02.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f2552885640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f254c101930 0x7f254c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:02.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f2552885640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f254c1022f0 0x7f254c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:02.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f2552885640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f254c10c240 con 0x7f254c100740 2024-03-18T22:43:02.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f2552885640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f254c10c0c0 con 0x7f254c1022f0 2024-03-18T22:43:02.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f2552885640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f254c10c3c0 con 0x7f254c101930 2024-03-18T22:43:02.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f254bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f254c100740 0x7f254c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:02.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f254bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f254c100740 0x7f254c197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37158/0 (socket says 172.21.15.40:37158) 2024-03-18T22:43:02.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f254bfff640 1 -- 172.21.15.40:0/3954596320 learned_addr learned my addr 172.21.15.40:0/3954596320 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:02.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f254b7fe640 1 --2- 172.21.15.40:0/3954596320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f254c101930 0x7f254c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:02.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f2550e22640 1 --2- 172.21.15.40:0/3954596320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f254c1022f0 0x7f254c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:02.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f254b7fe640 1 -- 172.21.15.40:0/3954596320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f254c1022f0 msgr2=0x7f254c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:02.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f254b7fe640 1 --2- 172.21.15.40:0/3954596320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f254c1022f0 0x7f254c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:02.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f254b7fe640 1 -- 172.21.15.40:0/3954596320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f254c100740 msgr2=0x7f254c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:02.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f254b7fe640 1 --2- 172.21.15.40:0/3954596320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f254c100740 0x7f254c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:02.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f254b7fe640 1 -- 172.21.15.40:0/3954596320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254c19f650 con 0x7f254c101930 2024-03-18T22:43:02.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f254bfff640 1 --2- 172.21.15.40:0/3954596320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f254c100740 0x7f254c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:02.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.988+0000 7f254b7fe640 1 --2- 172.21.15.40:0/3954596320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f254c101930 0x7f254c197f70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f254000e9f0 tx=0x7f254000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:02.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.989+0000 7f25497fa640 1 -- 172.21.15.40:0/3954596320 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254000ccc0 con 0x7f254c101930 2024-03-18T22:43:02.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.989+0000 7f25497fa640 1 -- 172.21.15.40:0/3954596320 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25400035f0 con 0x7f254c101930 2024-03-18T22:43:02.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.989+0000 7f2552885640 1 -- 172.21.15.40:0/3954596320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f254c19f940 con 0x7f254c101930 2024-03-18T22:43:02.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.989+0000 7f25497fa640 1 -- 172.21.15.40:0/3954596320 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254000ce60 con 0x7f254c101930 2024-03-18T22:43:02.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.989+0000 7f2552885640 1 -- 172.21.15.40:0/3954596320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f254c19fe90 con 0x7f254c101930 2024-03-18T22:43:02.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.990+0000 7f2552885640 1 -- 172.21.15.40:0/3954596320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f254c1a0120 con 0x7f254c101930 2024-03-18T22:43:02.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.991+0000 7f25497fa640 1 -- 172.21.15.40:0/3954596320 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f25400037a0 con 0x7f254c101930 2024-03-18T22:43:02.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.991+0000 7f25497fa640 1 --2- 172.21.15.40:0/3954596320 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25340a28a0 0x7f25340a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:02.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.991+0000 7f254bfff640 1 --2- 172.21.15.40:0/3954596320 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25340a28a0 0x7f25340a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:02.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.991+0000 7f25497fa640 1 -- 172.21.15.40:0/3954596320 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f25400cf0a0 con 0x7f254c101930 2024-03-18T22:43:02.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.992+0000 7f254bfff640 1 --2- 172.21.15.40:0/3954596320 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25340a28a0 0x7f25340a4d60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f253c02c840 tx=0x7f253c0095e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:02.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:02.996+0000 7f25497fa640 1 -- 172.21.15.40:0/3954596320 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f254009bcf0 con 0x7f254c101930 2024-03-18T22:43:03.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:03.218+0000 7f2552885640 1 -- 172.21.15.40:0/3954596320 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f254c1a0470 con 0x7f254c101930 2024-03-18T22:43:04.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.048+0000 7f25497fa640 1 -- 172.21.15.40:0/3954596320 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v28) ==== 117+0+0 (secure 0 0 0) 0x7f254009b5f0 con 0x7f254c101930 2024-03-18T22:43:04.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.051+0000 7f2552885640 1 -- 172.21.15.40:0/3954596320 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25340a28a0 msgr2=0x7f25340a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:04.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.051+0000 7f2552885640 1 --2- 172.21.15.40:0/3954596320 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25340a28a0 0x7f25340a4d60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f253c02c840 tx=0x7f253c0095e0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.051+0000 7f2552885640 1 -- 172.21.15.40:0/3954596320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f254c101930 msgr2=0x7f254c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:04.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.051+0000 7f2552885640 1 --2- 172.21.15.40:0/3954596320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f254c101930 0x7f254c197f70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f254000e9f0 tx=0x7f254000edf0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.051+0000 7f2552885640 1 -- 172.21.15.40:0/3954596320 shutdown_connections 2024-03-18T22:43:04.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.051+0000 7f2552885640 1 --2- 172.21.15.40:0/3954596320 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25340a28a0 0x7f25340a4d60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.051+0000 7f2552885640 1 --2- 172.21.15.40:0/3954596320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f254c1022f0 0x7f254c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.051+0000 7f2552885640 1 --2- 172.21.15.40:0/3954596320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f254c101930 0x7f254c197f70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.051+0000 7f2552885640 1 --2- 172.21.15.40:0/3954596320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f254c100740 0x7f254c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.051+0000 7f2552885640 1 -- 172.21.15.40:0/3954596320 >> 172.21.15.40:0/3954596320 conn(0x7f254c0fbcb0 msgr2=0x7f254c0fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:04.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.051+0000 7f2552885640 1 -- 172.21.15.40:0/3954596320 shutdown_connections 2024-03-18T22:43:04.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.051+0000 7f2552885640 1 -- 172.21.15.40:0/3954596320 wait complete. 2024-03-18T22:43:04.065 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-18T22:43:04.076 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-18T22:43:04.121 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:43:04.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.232+0000 7f1490234640 1 -- 172.21.15.40:0/4239860034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1488101840 msgr2=0x7f1488101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.232+0000 7f1490234640 1 --2- 172.21.15.40:0/4239860034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1488101840 0x7f1488101c60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1484009900 tx=0x7f148402bc30 comp rx=0 tx=0).stop 2024-03-18T22:43:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.232+0000 7f1490234640 1 -- 172.21.15.40:0/4239860034 shutdown_connections 2024-03-18T22:43:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.232+0000 7f1490234640 1 --2- 172.21.15.40:0/4239860034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14881033f0 0x7f1488109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.232+0000 7f1490234640 1 --2- 172.21.15.40:0/4239860034 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1488102a30 0x7f1488102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.232+0000 7f1490234640 1 --2- 172.21.15.40:0/4239860034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1488101840 0x7f1488101c60 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.232+0000 7f1490234640 1 -- 172.21.15.40:0/4239860034 >> 172.21.15.40:0/4239860034 conn(0x7f14880fcdd0 msgr2=0x7f14880ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.232+0000 7f1490234640 1 -- 172.21.15.40:0/4239860034 shutdown_connections 2024-03-18T22:43:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.232+0000 7f1490234640 1 -- 172.21.15.40:0/4239860034 wait complete. 2024-03-18T22:43:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f1490234640 1 Processor -- start 2024-03-18T22:43:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f1490234640 1 -- start start 2024-03-18T22:43:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f1490234640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1488101840 0x7f1488197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:04.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f1490234640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1488102a30 0x7f1488197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:04.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f1490234640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14881033f0 0x7f148819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:04.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f1490234640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f148810d340 con 0x7f14881033f0 2024-03-18T22:43:04.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f1490234640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f148810d1c0 con 0x7f1488101840 2024-03-18T22:43:04.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f1490234640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f148810d4c0 con 0x7f1488102a30 2024-03-18T22:43:04.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f148e7d1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14881033f0 0x7f148819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:04.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f148dfd0640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1488101840 0x7f1488197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:04.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f148e7d1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14881033f0 0x7f148819f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37174/0 (socket says 172.21.15.40:37174) 2024-03-18T22:43:04.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f148e7d1640 1 -- 172.21.15.40:0/2289878931 learned_addr learned my addr 172.21.15.40:0/2289878931 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:04.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f148d7cf640 1 --2- 172.21.15.40:0/2289878931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1488102a30 0x7f1488197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:04.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f148e7d1640 1 -- 172.21.15.40:0/2289878931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1488102a30 msgr2=0x7f1488197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:04.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f148e7d1640 1 --2- 172.21.15.40:0/2289878931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1488102a30 0x7f1488197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f148e7d1640 1 -- 172.21.15.40:0/2289878931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1488101840 msgr2=0x7f1488197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:04.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f148e7d1640 1 --2- 172.21.15.40:0/2289878931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1488101840 0x7f1488197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f148e7d1640 1 -- 172.21.15.40:0/2289878931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f148819f690 con 0x7f14881033f0 2024-03-18T22:43:04.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f148dfd0640 1 --2- 172.21.15.40:0/2289878931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1488101840 0x7f1488197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:04.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.233+0000 7f148e7d1640 1 --2- 172.21.15.40:0/2289878931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14881033f0 0x7f148819f020 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f148000eb10 tx=0x7f148000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:04.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.234+0000 7f147effd640 1 -- 172.21.15.40:0/2289878931 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480004b20 con 0x7f14881033f0 2024-03-18T22:43:04.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.234+0000 7f147effd640 1 -- 172.21.15.40:0/2289878931 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1480004cc0 con 0x7f14881033f0 2024-03-18T22:43:04.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.234+0000 7f1490234640 1 -- 172.21.15.40:0/2289878931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f148819f980 con 0x7f14881033f0 2024-03-18T22:43:04.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.234+0000 7f147effd640 1 -- 172.21.15.40:0/2289878931 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480010690 con 0x7f14881033f0 2024-03-18T22:43:04.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.234+0000 7f1490234640 1 -- 172.21.15.40:0/2289878931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f148819fed0 con 0x7f14881033f0 2024-03-18T22:43:04.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.235+0000 7f1490234640 1 -- 172.21.15.40:0/2289878931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1488108c30 con 0x7f14881033f0 2024-03-18T22:43:04.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.236+0000 7f147effd640 1 -- 172.21.15.40:0/2289878931 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1480005020 con 0x7f14881033f0 2024-03-18T22:43:04.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.236+0000 7f147effd640 1 --2- 172.21.15.40:0/2289878931 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f14740a2880 0x7f14740a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:04.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.237+0000 7f148dfd0640 1 --2- 172.21.15.40:0/2289878931 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f14740a2880 0x7f14740a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:04.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.237+0000 7f147effd640 1 -- 172.21.15.40:0/2289878931 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f14800cf360 con 0x7f14881033f0 2024-03-18T22:43:04.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.237+0000 7f148dfd0640 1 --2- 172.21.15.40:0/2289878931 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f14740a2880 0x7f14740a4d40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1484005ab0 tx=0x7f1484005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:04.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.241+0000 7f147effd640 1 -- 172.21.15.40:0/2289878931 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f148009c010 con 0x7f14881033f0 2024-03-18T22:43:04.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:04 smithi113 ceph-mon[66775]: pgmap v363: 129 pgs: 129 active+clean; 648 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T22:43:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:04 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3954596320' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T22:43:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:04 smithi113 ceph-mon[66775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T22:43:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:04 smithi103 ceph-mon[66377]: pgmap v363: 129 pgs: 129 active+clean; 648 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T22:43:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:04 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3954596320' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T22:43:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:04 smithi103 ceph-mon[66377]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T22:43:04.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:04 smithi040 ceph-mon[62643]: pgmap v363: 129 pgs: 129 active+clean; 648 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T22:43:04.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:04 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3954596320' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T22:43:04.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:04 smithi040 ceph-mon[62643]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T22:43:04.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.425+0000 7f1490234640 1 -- 172.21.15.40:0/2289878931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1488062420 con 0x7f14881033f0 2024-03-18T22:43:04.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.426+0000 7f147effd640 1 -- 172.21.15.40:0/2289878931 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24908 (secure 0 0 0) 0x7f148009b910 con 0x7f14881033f0 2024-03-18T22:43:04.427 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:43:04.428 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":93,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:42:44.941873+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:43:04.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.428+0000 7f1490234640 1 -- 172.21.15.40:0/2289878931 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f14740a2880 msgr2=0x7f14740a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:04.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.428+0000 7f1490234640 1 --2- 172.21.15.40:0/2289878931 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f14740a2880 0x7f14740a4d40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1484005ab0 tx=0x7f1484005fd0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.428+0000 7f1490234640 1 -- 172.21.15.40:0/2289878931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14881033f0 msgr2=0x7f148819f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:04.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.428+0000 7f1490234640 1 --2- 172.21.15.40:0/2289878931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14881033f0 0x7f148819f020 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f148000eb10 tx=0x7f148000c2d0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.428+0000 7f1490234640 1 -- 172.21.15.40:0/2289878931 shutdown_connections 2024-03-18T22:43:04.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.428+0000 7f1490234640 1 --2- 172.21.15.40:0/2289878931 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f14740a2880 0x7f14740a4d40 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.428+0000 7f1490234640 1 --2- 172.21.15.40:0/2289878931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14881033f0 0x7f148819f020 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.428+0000 7f1490234640 1 --2- 172.21.15.40:0/2289878931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1488102a30 0x7f1488197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.428+0000 7f1490234640 1 --2- 172.21.15.40:0/2289878931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1488101840 0x7f1488197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.428+0000 7f1490234640 1 -- 172.21.15.40:0/2289878931 >> 172.21.15.40:0/2289878931 conn(0x7f14880fcdd0 msgr2=0x7f1488105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:04.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.429+0000 7f1490234640 1 -- 172.21.15.40:0/2289878931 shutdown_connections 2024-03-18T22:43:04.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.429+0000 7f1490234640 1 -- 172.21.15.40:0/2289878931 wait complete. 2024-03-18T22:43:04.440 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T22:36:07.060691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T22:40:39.925147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T22:40:40.773757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T22:40:44.068183+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T22:43:04.441 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T22:43:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.546+0000 7f704be63640 1 -- 172.21.15.40:0/2031512647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70440ffe70 msgr2=0x7f7044100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.546+0000 7f704be63640 1 --2- 172.21.15.40:0/2031512647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70440ffe70 0x7f7044100290 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7040009860 tx=0x7f704002bb70 comp rx=0 tx=0).stop 2024-03-18T22:43:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.547+0000 7f704be63640 1 -- 172.21.15.40:0/2031512647 shutdown_connections 2024-03-18T22:43:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.547+0000 7f704be63640 1 --2- 172.21.15.40:0/2031512647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7044101a10 0x7f7044108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.547+0000 7f704be63640 1 --2- 172.21.15.40:0/2031512647 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7044101050 0x7f70441014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.547+0000 7f704be63640 1 --2- 172.21.15.40:0/2031512647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70440ffe70 0x7f7044100290 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.547+0000 7f704be63640 1 -- 172.21.15.40:0/2031512647 >> 172.21.15.40:0/2031512647 conn(0x7f70440fb430 msgr2=0x7f70440fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.547+0000 7f704be63640 1 -- 172.21.15.40:0/2031512647 shutdown_connections 2024-03-18T22:43:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.547+0000 7f704be63640 1 -- 172.21.15.40:0/2031512647 wait complete. 2024-03-18T22:43:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.547+0000 7f704be63640 1 Processor -- start 2024-03-18T22:43:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.547+0000 7f704be63640 1 -- start start 2024-03-18T22:43:04.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.547+0000 7f704be63640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70440ffe70 0x7f70440fd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f704be63640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7044101050 0x7f704419ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f704be63640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7044101a10 0x7f70441a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f704be63640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f704410b9e0 con 0x7f7044101050 2024-03-18T22:43:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f704be63640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f704410b860 con 0x7f7044101a10 2024-03-18T22:43:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f704be63640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f704410bb60 con 0x7f70440ffe70 2024-03-18T22:43:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f70493fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7044101050 0x7f704419ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f70493fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7044101050 0x7f704419ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37184/0 (socket says 172.21.15.40:37184) 2024-03-18T22:43:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f70493fe640 1 -- 172.21.15.40:0/3972058153 learned_addr learned my addr 172.21.15.40:0/3972058153 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f7049bff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70440ffe70 0x7f70440fd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f704a400640 1 --2- 172.21.15.40:0/3972058153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7044101a10 0x7f70441a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f7049bff640 1 -- 172.21.15.40:0/3972058153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7044101a10 msgr2=0x7f70441a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f7049bff640 1 --2- 172.21.15.40:0/3972058153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7044101a10 0x7f70441a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f7049bff640 1 -- 172.21.15.40:0/3972058153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7044101050 msgr2=0x7f704419ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:04.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f7049bff640 1 --2- 172.21.15.40:0/3972058153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7044101050 0x7f704419ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f7049bff640 1 -- 172.21.15.40:0/3972058153 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70441a1960 con 0x7f70440ffe70 2024-03-18T22:43:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f70493fe640 1 --2- 172.21.15.40:0/3972058153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7044101050 0x7f704419ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f7049bff640 1 --2- 172.21.15.40:0/3972058153 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70440ffe70 0x7f70440fd5d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7040037040 tx=0x7f70400040e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f703affd640 1 -- 172.21.15.40:0/3972058153 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040041020 con 0x7f70440ffe70 2024-03-18T22:43:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f703affd640 1 -- 172.21.15.40:0/3972058153 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70400044d0 con 0x7f70440ffe70 2024-03-18T22:43:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f704be63640 1 -- 172.21.15.40:0/3972058153 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70441a1b90 con 0x7f70440ffe70 2024-03-18T22:43:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f703affd640 1 -- 172.21.15.40:0/3972058153 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040039040 con 0x7f70440ffe70 2024-03-18T22:43:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.548+0000 7f704be63640 1 -- 172.21.15.40:0/3972058153 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70441a2080 con 0x7f70440ffe70 2024-03-18T22:43:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.549+0000 7f704be63640 1 -- 172.21.15.40:0/3972058153 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7044107ab0 con 0x7f70440ffe70 2024-03-18T22:43:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.550+0000 7f703affd640 1 -- 172.21.15.40:0/3972058153 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7040044020 con 0x7f70440ffe70 2024-03-18T22:43:04.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.551+0000 7f703affd640 1 --2- 172.21.15.40:0/3972058153 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70300a2830 0x7f70300a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:04.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.551+0000 7f70493fe640 1 --2- 172.21.15.40:0/3972058153 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70300a2830 0x7f70300a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:04.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.551+0000 7f703affd640 1 -- 172.21.15.40:0/3972058153 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f70400f09f0 con 0x7f70440ffe70 2024-03-18T22:43:04.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.551+0000 7f70493fe640 1 --2- 172.21.15.40:0/3972058153 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70300a2830 0x7f70300a4cf0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f70440fd340 tx=0x7f7034005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:04.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.554+0000 7f703affd640 1 -- 172.21.15.40:0/3972058153 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f704004b020 con 0x7f70440ffe70 2024-03-18T22:43:04.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.738+0000 7f704be63640 1 -- 172.21.15.40:0/3972058153 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f704402d380 con 0x7f70440ffe70 2024-03-18T22:43:04.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.739+0000 7f703affd640 1 -- 172.21.15.40:0/3972058153 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f70400bd6a0 con 0x7f70440ffe70 2024-03-18T22:43:04.739 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-18T22:43:04.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.740+0000 7f704be63640 1 -- 172.21.15.40:0/3972058153 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70300a2830 msgr2=0x7f70300a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.740+0000 7f704be63640 1 --2- 172.21.15.40:0/3972058153 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70300a2830 0x7f70300a4cf0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f70440fd340 tx=0x7f7034005f90 comp rx=0 tx=0).stop 2024-03-18T22:43:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.740+0000 7f704be63640 1 -- 172.21.15.40:0/3972058153 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70440ffe70 msgr2=0x7f70440fd5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.740+0000 7f704be63640 1 --2- 172.21.15.40:0/3972058153 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70440ffe70 0x7f70440fd5d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7040037040 tx=0x7f70400040e0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.740+0000 7f704be63640 1 -- 172.21.15.40:0/3972058153 shutdown_connections 2024-03-18T22:43:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.740+0000 7f704be63640 1 --2- 172.21.15.40:0/3972058153 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70300a2830 0x7f70300a4cf0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.740+0000 7f704be63640 1 --2- 172.21.15.40:0/3972058153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7044101a10 0x7f70441a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.740+0000 7f704be63640 1 --2- 172.21.15.40:0/3972058153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7044101050 0x7f704419ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.740+0000 7f704be63640 1 --2- 172.21.15.40:0/3972058153 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f70440ffe70 0x7f70440fd5d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.740+0000 7f704be63640 1 -- 172.21.15.40:0/3972058153 >> 172.21.15.40:0/3972058153 conn(0x7f70440fb430 msgr2=0x7f7044102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.741+0000 7f704be63640 1 -- 172.21.15.40:0/3972058153 shutdown_connections 2024-03-18T22:43:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.741+0000 7f704be63640 1 -- 172.21.15.40:0/3972058153 wait complete. 2024-03-18T22:43:04.752 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T22:43:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.857+0000 7f4b3dbd4640 1 -- 172.21.15.40:0/2769884464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38100720 msgr2=0x7f4b38100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.857+0000 7f4b3dbd4640 1 --2- 172.21.15.40:0/2769884464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38100720 0x7f4b38100b40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f4b28009ca0 tx=0x7f4b2802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.857+0000 7f4b3dbd4640 1 -- 172.21.15.40:0/2769884464 shutdown_connections 2024-03-18T22:43:04.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.857+0000 7f4b3dbd4640 1 --2- 172.21.15.40:0/2769884464 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b381022d0 0x7f4b38108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.857+0000 7f4b3dbd4640 1 --2- 172.21.15.40:0/2769884464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b38101910 0x7f4b38101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.857+0000 7f4b3dbd4640 1 --2- 172.21.15.40:0/2769884464 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38100720 0x7f4b38100b40 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.857+0000 7f4b3dbd4640 1 -- 172.21.15.40:0/2769884464 >> 172.21.15.40:0/2769884464 conn(0x7f4b380fbcb0 msgr2=0x7f4b380fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.857+0000 7f4b3dbd4640 1 -- 172.21.15.40:0/2769884464 shutdown_connections 2024-03-18T22:43:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.857+0000 7f4b3dbd4640 1 -- 172.21.15.40:0/2769884464 wait complete. 2024-03-18T22:43:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.858+0000 7f4b3dbd4640 1 Processor -- start 2024-03-18T22:43:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.858+0000 7f4b3dbd4640 1 -- start start 2024-03-18T22:43:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.858+0000 7f4b3dbd4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38100720 0x7f4b38197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.858+0000 7f4b3dbd4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b38101910 0x7f4b38198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.858+0000 7f4b3dbd4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b381022d0 0x7f4b3819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.858+0000 7f4b3dbd4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3810d370 con 0x7f4b38100720 2024-03-18T22:43:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.858+0000 7f4b3dbd4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3810d1f0 con 0x7f4b38101910 2024-03-18T22:43:04.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.858+0000 7f4b3dbd4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3810d4f0 con 0x7f4b381022d0 2024-03-18T22:43:04.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.858+0000 7f4b377fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38100720 0x7f4b38197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:04.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.858+0000 7f4b377fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38100720 0x7f4b38197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37200/0 (socket says 172.21.15.40:37200) 2024-03-18T22:43:04.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.858+0000 7f4b377fe640 1 -- 172.21.15.40:0/1281877988 learned_addr learned my addr 172.21.15.40:0/1281877988 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:04.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.858+0000 7f4b37fff640 1 --2- 172.21.15.40:0/1281877988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b381022d0 0x7f4b3819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:04.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.858+0000 7f4b36ffd640 1 --2- 172.21.15.40:0/1281877988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b38101910 0x7f4b38198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:04.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.859+0000 7f4b377fe640 1 -- 172.21.15.40:0/1281877988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b381022d0 msgr2=0x7f4b3819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:04.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.859+0000 7f4b377fe640 1 --2- 172.21.15.40:0/1281877988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b381022d0 0x7f4b3819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.859+0000 7f4b377fe640 1 -- 172.21.15.40:0/1281877988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b38101910 msgr2=0x7f4b38198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:04.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.859+0000 7f4b377fe640 1 --2- 172.21.15.40:0/1281877988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b38101910 0x7f4b38198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:04.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.859+0000 7f4b377fe640 1 -- 172.21.15.40:0/1281877988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b3819f6e0 con 0x7f4b38100720 2024-03-18T22:43:04.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.859+0000 7f4b377fe640 1 --2- 172.21.15.40:0/1281877988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38100720 0x7f4b38197ac0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4b28000f80 tx=0x7f4b280328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:04.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.859+0000 7f4b34ff9640 1 -- 172.21.15.40:0/1281877988 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b28041020 con 0x7f4b38100720 2024-03-18T22:43:04.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.859+0000 7f4b34ff9640 1 -- 172.21.15.40:0/1281877988 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b28034720 con 0x7f4b38100720 2024-03-18T22:43:04.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.859+0000 7f4b3dbd4640 1 -- 172.21.15.40:0/1281877988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b3819f970 con 0x7f4b38100720 2024-03-18T22:43:04.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.859+0000 7f4b34ff9640 1 -- 172.21.15.40:0/1281877988 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b28039040 con 0x7f4b38100720 2024-03-18T22:43:04.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.859+0000 7f4b3dbd4640 1 -- 172.21.15.40:0/1281877988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b3819feb0 con 0x7f4b38100720 2024-03-18T22:43:04.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.860+0000 7f4b3dbd4640 1 -- 172.21.15.40:0/1281877988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b38107bb0 con 0x7f4b38100720 2024-03-18T22:43:04.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.861+0000 7f4b34ff9640 1 -- 172.21.15.40:0/1281877988 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4b28044020 con 0x7f4b38100720 2024-03-18T22:43:04.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.861+0000 7f4b34ff9640 1 --2- 172.21.15.40:0/1281877988 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b240a2880 0x7f4b240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:04.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.862+0000 7f4b36ffd640 1 --2- 172.21.15.40:0/1281877988 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b240a2880 0x7f4b240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:04.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.862+0000 7f4b34ff9640 1 -- 172.21.15.40:0/1281877988 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4b280f0eb0 con 0x7f4b38100720 2024-03-18T22:43:04.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.862+0000 7f4b36ffd640 1 --2- 172.21.15.40:0/1281877988 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b240a2880 0x7f4b240a4d40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4b38198fd0 tx=0x7f4b20005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:04.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:04.866+0000 7f4b34ff9640 1 -- 172.21.15.40:0/1281877988 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4b2804b020 con 0x7f4b38100720 2024-03-18T22:43:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.047+0000 7f4b3dbd4640 1 -- 172.21.15.40:0/1281877988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4b38062420 con 0x7f4b38100720 2024-03-18T22:43:05.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.049+0000 7f4b34ff9640 1 -- 172.21.15.40:0/1281877988 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f4b280bdb60 con 0x7f4b38100720 2024-03-18T22:43:05.049 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:43:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.051+0000 7f4b3dbd4640 1 -- 172.21.15.40:0/1281877988 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b240a2880 msgr2=0x7f4b240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.051+0000 7f4b3dbd4640 1 --2- 172.21.15.40:0/1281877988 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b240a2880 0x7f4b240a4d40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4b38198fd0 tx=0x7f4b20005f90 comp rx=0 tx=0).stop 2024-03-18T22:43:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.051+0000 7f4b3dbd4640 1 -- 172.21.15.40:0/1281877988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38100720 msgr2=0x7f4b38197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.051+0000 7f4b3dbd4640 1 --2- 172.21.15.40:0/1281877988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38100720 0x7f4b38197ac0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4b28000f80 tx=0x7f4b280328a0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.051+0000 7f4b3dbd4640 1 -- 172.21.15.40:0/1281877988 shutdown_connections 2024-03-18T22:43:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.051+0000 7f4b3dbd4640 1 --2- 172.21.15.40:0/1281877988 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b240a2880 0x7f4b240a4d40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.051+0000 7f4b3dbd4640 1 --2- 172.21.15.40:0/1281877988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b381022d0 0x7f4b3819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.051+0000 7f4b3dbd4640 1 --2- 172.21.15.40:0/1281877988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b38101910 0x7f4b38198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.051+0000 7f4b3dbd4640 1 --2- 172.21.15.40:0/1281877988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38100720 0x7f4b38197ac0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.051+0000 7f4b3dbd4640 1 -- 172.21.15.40:0/1281877988 >> 172.21.15.40:0/1281877988 conn(0x7f4b380fbcb0 msgr2=0x7f4b38104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.051+0000 7f4b3dbd4640 1 -- 172.21.15.40:0/1281877988 shutdown_connections 2024-03-18T22:43:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.051+0000 7f4b3dbd4640 1 -- 172.21.15.40:0/1281877988 wait complete. 2024-03-18T22:43:05.063 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T22:43:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.170+0000 7f6d14ff4640 1 -- 172.21.15.40:0/267061763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d101022d0 msgr2=0x7f6d10108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.170+0000 7f6d14ff4640 1 --2- 172.21.15.40:0/267061763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d101022d0 0x7f6d10108b60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6d08009ca0 tx=0x7f6d0802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.170+0000 7f6d14ff4640 1 -- 172.21.15.40:0/267061763 shutdown_connections 2024-03-18T22:43:05.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.170+0000 7f6d14ff4640 1 --2- 172.21.15.40:0/267061763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d101022d0 0x7f6d10108b60 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.170+0000 7f6d14ff4640 1 --2- 172.21.15.40:0/267061763 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d10101910 0x7f6d10101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.170+0000 7f6d14ff4640 1 --2- 172.21.15.40:0/267061763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d10100720 0x7f6d10100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.170+0000 7f6d14ff4640 1 -- 172.21.15.40:0/267061763 >> 172.21.15.40:0/267061763 conn(0x7f6d100fbcb0 msgr2=0x7f6d100fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:05.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.170+0000 7f6d14ff4640 1 -- 172.21.15.40:0/267061763 shutdown_connections 2024-03-18T22:43:05.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.170+0000 7f6d14ff4640 1 -- 172.21.15.40:0/267061763 wait complete. 2024-03-18T22:43:05.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.171+0000 7f6d14ff4640 1 Processor -- start 2024-03-18T22:43:05.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.171+0000 7f6d14ff4640 1 -- start start 2024-03-18T22:43:05.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.171+0000 7f6d14ff4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d10100720 0x7f6d101046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:05.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.171+0000 7f6d14ff4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d10101910 0x7f6d10102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.171+0000 7f6d14ff4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d101022d0 0x7f6d10103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.171+0000 7f6d14ff4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1010c250 con 0x7f6d10101910 2024-03-18T22:43:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.171+0000 7f6d14ff4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1010c0d0 con 0x7f6d101022d0 2024-03-18T22:43:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.171+0000 7f6d14ff4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6d1010c3d0 con 0x7f6d10100720 2024-03-18T22:43:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.171+0000 7f6d0dd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d10101910 0x7f6d10102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.171+0000 7f6d0ed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d101022d0 0x7f6d10103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.171+0000 7f6d0dd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d10101910 0x7f6d10102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37218/0 (socket says 172.21.15.40:37218) 2024-03-18T22:43:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.171+0000 7f6d0e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d10100720 0x7f6d101046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.171+0000 7f6d0dd9b640 1 -- 172.21.15.40:0/2648259975 learned_addr learned my addr 172.21.15.40:0/2648259975 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.172+0000 7f6d0ed9d640 1 -- 172.21.15.40:0/2648259975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d10100720 msgr2=0x7f6d101046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.172+0000 7f6d0ed9d640 1 --2- 172.21.15.40:0/2648259975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d10100720 0x7f6d101046b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.172+0000 7f6d0ed9d640 1 -- 172.21.15.40:0/2648259975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d10101910 msgr2=0x7f6d10102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.172+0000 7f6d0ed9d640 1 --2- 172.21.15.40:0/2648259975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d10101910 0x7f6d10102cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.172+0000 7f6d0ed9d640 1 -- 172.21.15.40:0/2648259975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d10103ac0 con 0x7f6d101022d0 2024-03-18T22:43:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.172+0000 7f6d0ed9d640 1 --2- 172.21.15.40:0/2648259975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d101022d0 0x7f6d10103230 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6d0802c630 tx=0x7f6d080328a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.172+0000 7f6cff7fe640 1 -- 172.21.15.40:0/2648259975 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d08041020 con 0x7f6d101022d0 2024-03-18T22:43:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.172+0000 7f6cff7fe640 1 -- 172.21.15.40:0/2648259975 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6d08003e20 con 0x7f6d101022d0 2024-03-18T22:43:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.172+0000 7f6d14ff4640 1 -- 172.21.15.40:0/2648259975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d101a1b10 con 0x7f6d101022d0 2024-03-18T22:43:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.172+0000 7f6cff7fe640 1 -- 172.21.15.40:0/2648259975 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d08039040 con 0x7f6d101022d0 2024-03-18T22:43:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.172+0000 7f6d14ff4640 1 -- 172.21.15.40:0/2648259975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d101a20b0 con 0x7f6d101022d0 2024-03-18T22:43:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.173+0000 7f6d14ff4640 1 -- 172.21.15.40:0/2648259975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d101a2280 con 0x7f6d101022d0 2024-03-18T22:43:05.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.174+0000 7f6cff7fe640 1 -- 172.21.15.40:0/2648259975 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6d08044020 con 0x7f6d101022d0 2024-03-18T22:43:05.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.174+0000 7f6cff7fe640 1 --2- 172.21.15.40:0/2648259975 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6cf80a2880 0x7f6cf80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:05.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.175+0000 7f6d0e59c640 1 --2- 172.21.15.40:0/2648259975 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6cf80a2880 0x7f6cf80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:05.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.175+0000 7f6cff7fe640 1 -- 172.21.15.40:0/2648259975 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6d080f0f50 con 0x7f6d101022d0 2024-03-18T22:43:05.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.175+0000 7f6d0e59c640 1 --2- 172.21.15.40:0/2648259975 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6cf80a2880 0x7f6cf80a4d40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6d040095f0 tx=0x7f6d04005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:05.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.179+0000 7f6cff7fe640 1 -- 172.21.15.40:0/2648259975 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6d0804b020 con 0x7f6d101022d0 2024-03-18T22:43:05.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:05 smithi113 ceph-mon[66775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-18T22:43:05.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:05 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:43:05.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:05 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2289878931' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:43:05.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:05 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3972058153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:43:05.334 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:05 smithi040 ceph-mon[62643]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-18T22:43:05.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:05 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:43:05.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:05 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2289878931' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:43:05.335 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:05 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3972058153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:43:05.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:05 smithi103 ceph-mon[66377]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-18T22:43:05.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:05 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:43:05.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:05 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2289878931' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:43:05.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:05 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3972058153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:43:05.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.366+0000 7f6d14ff4640 1 -- 172.21.15.40:0/2648259975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6d101a2610 con 0x7f6d101022d0 2024-03-18T22:43:05.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.366+0000 7f6cff7fe640 1 -- 172.21.15.40:0/2648259975 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f6d080bdc00 con 0x7f6d101022d0 2024-03-18T22:43:05.367 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-18T22:43:05.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.368+0000 7f6d14ff4640 1 -- 172.21.15.40:0/2648259975 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6cf80a2880 msgr2=0x7f6cf80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:05.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.368+0000 7f6d14ff4640 1 --2- 172.21.15.40:0/2648259975 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6cf80a2880 0x7f6cf80a4d40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6d040095f0 tx=0x7f6d04005f90 comp rx=0 tx=0).stop 2024-03-18T22:43:05.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.368+0000 7f6d14ff4640 1 -- 172.21.15.40:0/2648259975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d101022d0 msgr2=0x7f6d10103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:05.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.368+0000 7f6d14ff4640 1 --2- 172.21.15.40:0/2648259975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d101022d0 0x7f6d10103230 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6d0802c630 tx=0x7f6d080328a0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.368+0000 7f6d14ff4640 1 -- 172.21.15.40:0/2648259975 shutdown_connections 2024-03-18T22:43:05.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.368+0000 7f6d14ff4640 1 --2- 172.21.15.40:0/2648259975 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6cf80a2880 0x7f6cf80a4d40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.368+0000 7f6d14ff4640 1 --2- 172.21.15.40:0/2648259975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d101022d0 0x7f6d10103230 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.368+0000 7f6d14ff4640 1 --2- 172.21.15.40:0/2648259975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d10101910 0x7f6d10102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.368+0000 7f6d14ff4640 1 --2- 172.21.15.40:0/2648259975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6d10100720 0x7f6d101046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.368+0000 7f6d14ff4640 1 -- 172.21.15.40:0/2648259975 >> 172.21.15.40:0/2648259975 conn(0x7f6d100fbcb0 msgr2=0x7f6d100fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:05.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.368+0000 7f6d14ff4640 1 -- 172.21.15.40:0/2648259975 shutdown_connections 2024-03-18T22:43:05.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.368+0000 7f6d14ff4640 1 -- 172.21.15.40:0/2648259975 wait complete. 2024-03-18T22:43:05.380 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T22:43:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.487+0000 7f344707e640 1 -- 172.21.15.40:0/3369730944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3440101910 msgr2=0x7f3440101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.487+0000 7f344707e640 1 --2- 172.21.15.40:0/3369730944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3440101910 0x7f3440101d90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f342c009860 tx=0x7f342c02bb70 comp rx=0 tx=0).stop 2024-03-18T22:43:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.487+0000 7f344707e640 1 -- 172.21.15.40:0/3369730944 shutdown_connections 2024-03-18T22:43:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.487+0000 7f344707e640 1 --2- 172.21.15.40:0/3369730944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34401022d0 0x7f3440108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.487+0000 7f344707e640 1 --2- 172.21.15.40:0/3369730944 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3440101910 0x7f3440101d90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.487+0000 7f344707e640 1 --2- 172.21.15.40:0/3369730944 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3440100720 0x7f3440100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.487+0000 7f344707e640 1 -- 172.21.15.40:0/3369730944 >> 172.21.15.40:0/3369730944 conn(0x7f34400fbcb0 msgr2=0x7f34400fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.487+0000 7f344707e640 1 -- 172.21.15.40:0/3369730944 shutdown_connections 2024-03-18T22:43:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.487+0000 7f344707e640 1 -- 172.21.15.40:0/3369730944 wait complete. 2024-03-18T22:43:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f344707e640 1 Processor -- start 2024-03-18T22:43:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f344707e640 1 -- start start 2024-03-18T22:43:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f344707e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3440100720 0x7f34400694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f344707e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3440101910 0x7f344019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f344707e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34401022d0 0x7f344019ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f344707e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f344010c230 con 0x7f34401022d0 2024-03-18T22:43:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f344707e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f344010c0b0 con 0x7f3440100720 2024-03-18T22:43:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f344707e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f344010c3b0 con 0x7f3440101910 2024-03-18T22:43:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f344561b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34401022d0 0x7f344019ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:05.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f344561b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34401022d0 0x7f344019ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37230/0 (socket says 172.21.15.40:37230) 2024-03-18T22:43:05.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f344561b640 1 -- 172.21.15.40:0/2925852445 learned_addr learned my addr 172.21.15.40:0/2925852445 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:05.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f343ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3440101910 0x7f344019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:05.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f3444e1a640 1 --2- 172.21.15.40:0/2925852445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3440100720 0x7f34400694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:05.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f343ffff640 1 -- 172.21.15.40:0/2925852445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3440100720 msgr2=0x7f34400694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:05.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f343ffff640 1 --2- 172.21.15.40:0/2925852445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3440100720 0x7f34400694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f343ffff640 1 -- 172.21.15.40:0/2925852445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34401022d0 msgr2=0x7f344019ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:05.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f343ffff640 1 --2- 172.21.15.40:0/2925852445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34401022d0 0x7f344019ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f343ffff640 1 -- 172.21.15.40:0/2925852445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344019f720 con 0x7f3440101910 2024-03-18T22:43:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f344561b640 1 --2- 172.21.15.40:0/2925852445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34401022d0 0x7f344019ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.488+0000 7f343ffff640 1 --2- 172.21.15.40:0/2925852445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3440101910 0x7f344019c7f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f342c02c0b0 tx=0x7f342c0040e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.489+0000 7f343dffb640 1 -- 172.21.15.40:0/2925852445 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342c041020 con 0x7f3440101910 2024-03-18T22:43:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.489+0000 7f343dffb640 1 -- 172.21.15.40:0/2925852445 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f342c0044d0 con 0x7f3440101910 2024-03-18T22:43:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.489+0000 7f343dffb640 1 -- 172.21.15.40:0/2925852445 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342c039040 con 0x7f3440101910 2024-03-18T22:43:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.489+0000 7f344707e640 1 -- 172.21.15.40:0/2925852445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f344019f950 con 0x7f3440101910 2024-03-18T22:43:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.489+0000 7f344707e640 1 -- 172.21.15.40:0/2925852445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f344019fe90 con 0x7f3440101910 2024-03-18T22:43:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.490+0000 7f344707e640 1 -- 172.21.15.40:0/2925852445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3440107ab0 con 0x7f3440101910 2024-03-18T22:43:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.491+0000 7f343dffb640 1 -- 172.21.15.40:0/2925852445 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f342c044020 con 0x7f3440101910 2024-03-18T22:43:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.491+0000 7f343dffb640 1 --2- 172.21.15.40:0/2925852445 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f34300a2880 0x7f34300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.492+0000 7f3444e1a640 1 --2- 172.21.15.40:0/2925852445 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f34300a2880 0x7f34300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:05.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.492+0000 7f343dffb640 1 -- 172.21.15.40:0/2925852445 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f342c0f09b0 con 0x7f3440101910 2024-03-18T22:43:05.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.492+0000 7f3444e1a640 1 --2- 172.21.15.40:0/2925852445 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f34300a2880 0x7f34300a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3434009530 tx=0x7f3434009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:05.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.495+0000 7f343dffb640 1 -- 172.21.15.40:0/2925852445 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f342c04b020 con 0x7f3440101910 2024-03-18T22:43:05.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.680+0000 7f344707e640 1 -- 172.21.15.40:0/2925852445 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f344002d440 con 0x7f3440101910 2024-03-18T22:43:05.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.680+0000 7f343dffb640 1 -- 172.21.15.40:0/2925852445 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f342c0bd680 con 0x7f3440101910 2024-03-18T22:43:05.681 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:43:05.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.683+0000 7f344707e640 1 -- 172.21.15.40:0/2925852445 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f34300a2880 msgr2=0x7f34300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:05.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.683+0000 7f344707e640 1 --2- 172.21.15.40:0/2925852445 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f34300a2880 0x7f34300a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3434009530 tx=0x7f3434009260 comp rx=0 tx=0).stop 2024-03-18T22:43:05.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.683+0000 7f344707e640 1 -- 172.21.15.40:0/2925852445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3440101910 msgr2=0x7f344019c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:05.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.683+0000 7f344707e640 1 --2- 172.21.15.40:0/2925852445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3440101910 0x7f344019c7f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f342c02c0b0 tx=0x7f342c0040e0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.683+0000 7f344707e640 1 -- 172.21.15.40:0/2925852445 shutdown_connections 2024-03-18T22:43:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.683+0000 7f344707e640 1 --2- 172.21.15.40:0/2925852445 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f34300a2880 0x7f34300a4d40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.683+0000 7f344707e640 1 --2- 172.21.15.40:0/2925852445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34401022d0 0x7f344019ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.683+0000 7f344707e640 1 --2- 172.21.15.40:0/2925852445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3440101910 0x7f344019c7f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.683+0000 7f344707e640 1 --2- 172.21.15.40:0/2925852445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3440100720 0x7f34400694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.683+0000 7f344707e640 1 -- 172.21.15.40:0/2925852445 >> 172.21.15.40:0/2925852445 conn(0x7f34400fbcb0 msgr2=0x7f3440104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.683+0000 7f344707e640 1 -- 172.21.15.40:0/2925852445 shutdown_connections 2024-03-18T22:43:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.683+0000 7f344707e640 1 -- 172.21.15.40:0/2925852445 wait complete. 2024-03-18T22:43:05.694 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:43:05.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.797+0000 7fa7e47eb640 1 -- 172.21.15.40:0/1376194749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7dc1075c0 msgr2=0x7fa7dc1099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:05.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.797+0000 7fa7e47eb640 1 --2- 172.21.15.40:0/1376194749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7dc1075c0 0x7fa7dc1099b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009be0 tx=0x7fa7d402bef0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.797+0000 7fa7e47eb640 1 -- 172.21.15.40:0/1376194749 shutdown_connections 2024-03-18T22:43:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.797+0000 7fa7e47eb640 1 --2- 172.21.15.40:0/1376194749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7dc1075c0 0x7fa7dc1099b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.797+0000 7fa7e47eb640 1 --2- 172.21.15.40:0/1376194749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7dc104c90 0x7fa7dc107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.797+0000 7fa7e47eb640 1 --2- 172.21.15.40:0/1376194749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7dc102360 0x7fa7dc104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.797+0000 7fa7e47eb640 1 -- 172.21.15.40:0/1376194749 >> 172.21.15.40:0/1376194749 conn(0x7fa7dc0fbcb0 msgr2=0x7fa7dc0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.797+0000 7fa7e47eb640 1 -- 172.21.15.40:0/1376194749 shutdown_connections 2024-03-18T22:43:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.797+0000 7fa7e47eb640 1 -- 172.21.15.40:0/1376194749 wait complete. 2024-03-18T22:43:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e47eb640 1 Processor -- start 2024-03-18T22:43:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e47eb640 1 -- start start 2024-03-18T22:43:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e47eb640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7dc102360 0x7fa7dc199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e47eb640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7dc104c90 0x7fa7dc19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e47eb640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7dc1075c0 0x7fa7dc1a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e47eb640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc10d390 con 0x7fa7dc104c90 2024-03-18T22:43:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e47eb640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc10d210 con 0x7fa7dc1075c0 2024-03-18T22:43:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e47eb640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc10d510 con 0x7fa7dc102360 2024-03-18T22:43:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e1d86640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7dc104c90 0x7fa7dc19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e2587640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7dc102360 0x7fa7dc199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e1d86640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7dc104c90 0x7fa7dc19a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37254/0 (socket says 172.21.15.40:37254) 2024-03-18T22:43:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e1d86640 1 -- 172.21.15.40:0/3123390234 learned_addr learned my addr 172.21.15.40:0/3123390234 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e2d88640 1 --2- 172.21.15.40:0/3123390234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7dc1075c0 0x7fa7dc1a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e1d86640 1 -- 172.21.15.40:0/3123390234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7dc102360 msgr2=0x7fa7dc199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e1d86640 1 --2- 172.21.15.40:0/3123390234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7dc102360 0x7fa7dc199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e1d86640 1 -- 172.21.15.40:0/3123390234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7dc1075c0 msgr2=0x7fa7dc1a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e1d86640 1 --2- 172.21.15.40:0/3123390234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7dc1075c0 0x7fa7dc1a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e1d86640 1 -- 172.21.15.40:0/3123390234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7dc1a18c0 con 0x7fa7dc104c90 2024-03-18T22:43:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.798+0000 7fa7e2587640 1 --2- 172.21.15.40:0/3123390234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7dc102360 0x7fa7dc199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.799+0000 7fa7e1d86640 1 --2- 172.21.15.40:0/3123390234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7dc104c90 0x7fa7dc19a1c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00e9f0 tx=0x7fa7cc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.799+0000 7fa7d37fe640 1 -- 172.21.15.40:0/3123390234 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc00ccc0 con 0x7fa7dc104c90 2024-03-18T22:43:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.799+0000 7fa7e47eb640 1 -- 172.21.15.40:0/3123390234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7dc1a1bb0 con 0x7fa7dc104c90 2024-03-18T22:43:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.799+0000 7fa7d37fe640 1 -- 172.21.15.40:0/3123390234 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7cc0108e0 con 0x7fa7dc104c90 2024-03-18T22:43:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.799+0000 7fa7d37fe640 1 -- 172.21.15.40:0/3123390234 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc018880 con 0x7fa7dc104c90 2024-03-18T22:43:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.799+0000 7fa7e47eb640 1 -- 172.21.15.40:0/3123390234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7dc1a2150 con 0x7fa7dc104c90 2024-03-18T22:43:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.800+0000 7fa7e47eb640 1 -- 172.21.15.40:0/3123390234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7dc1090b0 con 0x7fa7dc104c90 2024-03-18T22:43:05.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.801+0000 7fa7d37fe640 1 -- 172.21.15.40:0/3123390234 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa7cc010a90 con 0x7fa7dc104c90 2024-03-18T22:43:05.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.802+0000 7fa7d37fe640 1 --2- 172.21.15.40:0/3123390234 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa7c80a2800 0x7fa7c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:05.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.802+0000 7fa7e2587640 1 --2- 172.21.15.40:0/3123390234 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa7c80a2800 0x7fa7c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:05.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.802+0000 7fa7d37fe640 1 -- 172.21.15.40:0/3123390234 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa7cc0cf550 con 0x7fa7dc104c90 2024-03-18T22:43:05.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.802+0000 7fa7e2587640 1 --2- 172.21.15.40:0/3123390234 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa7c80a2800 0x7fa7c80a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc19ab50 tx=0x7fa7d8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:05.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.809+0000 7fa7d37fe640 1 -- 172.21.15.40:0/3123390234 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa7cc09bb50 con 0x7fa7dc104c90 2024-03-18T22:43:05.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:05.997+0000 7fa7e47eb640 1 -- 172.21.15.40:0/3123390234 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa7dc02d370 con 0x7fa7dc104c90 2024-03-18T22:43:06.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.000+0000 7fa7d37fe640 1 -- 172.21.15.40:0/3123390234 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24908 (secure 0 0 0) 0x7fa7cc09fbf0 con 0x7fa7dc104c90 2024-03-18T22:43:06.000 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:43:06.001 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":93,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:42:44.941873+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:43:06.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.002+0000 7fa7e47eb640 1 -- 172.21.15.40:0/3123390234 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa7c80a2800 msgr2=0x7fa7c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.002+0000 7fa7e47eb640 1 --2- 172.21.15.40:0/3123390234 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa7c80a2800 0x7fa7c80a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc19ab50 tx=0x7fa7d8005f90 comp rx=0 tx=0).stop 2024-03-18T22:43:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.002+0000 7fa7e47eb640 1 -- 172.21.15.40:0/3123390234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7dc104c90 msgr2=0x7fa7dc19a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.002+0000 7fa7e47eb640 1 --2- 172.21.15.40:0/3123390234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7dc104c90 0x7fa7dc19a1c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00e9f0 tx=0x7fa7cc00eeb0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.002+0000 7fa7e47eb640 1 -- 172.21.15.40:0/3123390234 shutdown_connections 2024-03-18T22:43:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.002+0000 7fa7e47eb640 1 --2- 172.21.15.40:0/3123390234 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa7c80a2800 0x7fa7c80a4cc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.002+0000 7fa7e47eb640 1 --2- 172.21.15.40:0/3123390234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7dc1075c0 0x7fa7dc1a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.002+0000 7fa7e47eb640 1 --2- 172.21.15.40:0/3123390234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7dc104c90 0x7fa7dc19a1c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.002+0000 7fa7e47eb640 1 --2- 172.21.15.40:0/3123390234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa7dc102360 0x7fa7dc199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.002+0000 7fa7e47eb640 1 -- 172.21.15.40:0/3123390234 >> 172.21.15.40:0/3123390234 conn(0x7fa7dc0fbcb0 msgr2=0x7fa7dc101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.002+0000 7fa7e47eb640 1 -- 172.21.15.40:0/3123390234 shutdown_connections 2024-03-18T22:43:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.002+0000 7fa7e47eb640 1 -- 172.21.15.40:0/3123390234 wait complete. 2024-03-18T22:43:06.014 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T22:36:07.060691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T22:40:39.925147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T22:40:40.773757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T22:40:44.068183+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T22:43:06.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T22:43:06.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.122+0000 7fbfdab62640 1 -- 172.21.15.40:0/462764478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd4101910 msgr2=0x7fbfd4101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.122+0000 7fbfdab62640 1 --2- 172.21.15.40:0/462764478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd4101910 0x7fbfd4101d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009900 tx=0x7fbfc002bc30 comp rx=0 tx=0).stop 2024-03-18T22:43:06.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.122+0000 7fbfdab62640 1 -- 172.21.15.40:0/462764478 shutdown_connections 2024-03-18T22:43:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.122+0000 7fbfdab62640 1 --2- 172.21.15.40:0/462764478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfd41022d0 0x7fbfd4108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.122+0000 7fbfdab62640 1 --2- 172.21.15.40:0/462764478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd4101910 0x7fbfd4101d90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.122+0000 7fbfdab62640 1 --2- 172.21.15.40:0/462764478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd4100720 0x7fbfd4100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.122+0000 7fbfdab62640 1 -- 172.21.15.40:0/462764478 >> 172.21.15.40:0/462764478 conn(0x7fbfd40fbcb0 msgr2=0x7fbfd40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.122+0000 7fbfdab62640 1 -- 172.21.15.40:0/462764478 shutdown_connections 2024-03-18T22:43:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.122+0000 7fbfdab62640 1 -- 172.21.15.40:0/462764478 wait complete. 2024-03-18T22:43:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.123+0000 7fbfdab62640 1 Processor -- start 2024-03-18T22:43:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.123+0000 7fbfdab62640 1 -- start start 2024-03-18T22:43:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.123+0000 7fbfdab62640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd4100720 0x7fbfd41046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.123+0000 7fbfdab62640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfd4101910 0x7fbfd4102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.123+0000 7fbfdab62640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd41022d0 0x7fbfd4103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.123+0000 7fbfdab62640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd410c280 con 0x7fbfd41022d0 2024-03-18T22:43:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.123+0000 7fbfdab62640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd410c100 con 0x7fbfd4100720 2024-03-18T22:43:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.123+0000 7fbfdab62640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd410c400 con 0x7fbfd4101910 2024-03-18T22:43:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.123+0000 7fbfd88fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd4100720 0x7fbfd41046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.123+0000 7fbfd90ff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd41022d0 0x7fbfd4103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.123+0000 7fbfd88fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd4100720 0x7fbfd41046b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:39862/0 (socket says 172.21.15.40:39862) 2024-03-18T22:43:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.123+0000 7fbfd88fe640 1 -- 172.21.15.40:0/3556458307 learned_addr learned my addr 172.21.15.40:0/3556458307 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.123+0000 7fbfd3fff640 1 --2- 172.21.15.40:0/3556458307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfd4101910 0x7fbfd4102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.124+0000 7fbfd88fe640 1 -- 172.21.15.40:0/3556458307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfd4101910 msgr2=0x7fbfd4102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.124+0000 7fbfd88fe640 1 --2- 172.21.15.40:0/3556458307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfd4101910 0x7fbfd4102cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.124+0000 7fbfd88fe640 1 -- 172.21.15.40:0/3556458307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd41022d0 msgr2=0x7fbfd4103230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.124+0000 7fbfd88fe640 1 --2- 172.21.15.40:0/3556458307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd41022d0 0x7fbfd4103230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.124+0000 7fbfd88fe640 1 -- 172.21.15.40:0/3556458307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd4103ac0 con 0x7fbfd4100720 2024-03-18T22:43:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.124+0000 7fbfd90ff640 1 --2- 172.21.15.40:0/3556458307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd41022d0 0x7fbfd4103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.124+0000 7fbfd3fff640 1 --2- 172.21.15.40:0/3556458307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfd4101910 0x7fbfd4102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.124+0000 7fbfd88fe640 1 --2- 172.21.15.40:0/3556458307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd4100720 0x7fbfd41046b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbfc800e9c0 tx=0x7fbfc800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.124+0000 7fbfd1ffb640 1 -- 172.21.15.40:0/3556458307 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc800cde0 con 0x7fbfd4100720 2024-03-18T22:43:06.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.124+0000 7fbfd1ffb640 1 -- 172.21.15.40:0/3556458307 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfc80108e0 con 0x7fbfd4100720 2024-03-18T22:43:06.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.124+0000 7fbfd1ffb640 1 -- 172.21.15.40:0/3556458307 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8018950 con 0x7fbfd4100720 2024-03-18T22:43:06.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.124+0000 7fbfdab62640 1 -- 172.21.15.40:0/3556458307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd41a1bf0 con 0x7fbfd4100720 2024-03-18T22:43:06.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.124+0000 7fbfdab62640 1 -- 172.21.15.40:0/3556458307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd41a2110 con 0x7fbfd4100720 2024-03-18T22:43:06.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.125+0000 7fbfdab62640 1 -- 172.21.15.40:0/3556458307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfd41a22e0 con 0x7fbfd4100720 2024-03-18T22:43:06.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.126+0000 7fbfd1ffb640 1 -- 172.21.15.40:0/3556458307 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbfc8010a90 con 0x7fbfd4100720 2024-03-18T22:43:06.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.127+0000 7fbfd1ffb640 1 --2- 172.21.15.40:0/3556458307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbfc40a2800 0x7fbfc40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:06.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.127+0000 7fbfd3fff640 1 --2- 172.21.15.40:0/3556458307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbfc40a2800 0x7fbfc40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:06.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.127+0000 7fbfd1ffb640 1 -- 172.21.15.40:0/3556458307 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fbfc80cf500 con 0x7fbfd4100720 2024-03-18T22:43:06.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.127+0000 7fbfd3fff640 1 --2- 172.21.15.40:0/3556458307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbfc40a2800 0x7fbfc40a4cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0000c00 tx=0x7fbfc002c7e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:06.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.130+0000 7fbfd1ffb640 1 -- 172.21.15.40:0/3556458307 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbfc809c1b0 con 0x7fbfd4100720 2024-03-18T22:43:06.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:06 smithi040 ceph-mon[62643]: pgmap v364: 129 pgs: 129 active+clean; 649 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 853 B/s wr, 12 op/s 2024-03-18T22:43:06.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:06 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1281877988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:43:06.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:06 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2648259975' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:43:06.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:06 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2925852445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:43:06.312 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:06 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3123390234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:43:06.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.313+0000 7fbfdab62640 1 -- 172.21.15.40:0/3556458307 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbfd41a2670 con 0x7fbfd4100720 2024-03-18T22:43:06.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.314+0000 7fbfd1ffb640 1 -- 172.21.15.40:0/3556458307 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fbfc809bab0 con 0x7fbfd4100720 2024-03-18T22:43:06.314 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-18T22:43:06.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.316+0000 7fbfdab62640 1 -- 172.21.15.40:0/3556458307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbfc40a2800 msgr2=0x7fbfc40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.316+0000 7fbfdab62640 1 --2- 172.21.15.40:0/3556458307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbfc40a2800 0x7fbfc40a4cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0000c00 tx=0x7fbfc002c7e0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.316+0000 7fbfdab62640 1 -- 172.21.15.40:0/3556458307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd4100720 msgr2=0x7fbfd41046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.316+0000 7fbfdab62640 1 --2- 172.21.15.40:0/3556458307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd4100720 0x7fbfd41046b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbfc800e9c0 tx=0x7fbfc800ee80 comp rx=0 tx=0).stop 2024-03-18T22:43:06.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.316+0000 7fbfdab62640 1 -- 172.21.15.40:0/3556458307 shutdown_connections 2024-03-18T22:43:06.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.316+0000 7fbfdab62640 1 --2- 172.21.15.40:0/3556458307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbfc40a2800 0x7fbfc40a4cc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.316+0000 7fbfdab62640 1 --2- 172.21.15.40:0/3556458307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfd41022d0 0x7fbfd4103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.316+0000 7fbfdab62640 1 --2- 172.21.15.40:0/3556458307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbfd4101910 0x7fbfd4102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.316+0000 7fbfdab62640 1 --2- 172.21.15.40:0/3556458307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd4100720 0x7fbfd41046b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.316+0000 7fbfdab62640 1 -- 172.21.15.40:0/3556458307 >> 172.21.15.40:0/3556458307 conn(0x7fbfd40fbcb0 msgr2=0x7fbfd40fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:06.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.316+0000 7fbfdab62640 1 -- 172.21.15.40:0/3556458307 shutdown_connections 2024-03-18T22:43:06.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.316+0000 7fbfdab62640 1 -- 172.21.15.40:0/3556458307 wait complete. 2024-03-18T22:43:06.328 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T22:43:06.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:06 smithi113 ceph-mon[66775]: pgmap v364: 129 pgs: 129 active+clean; 649 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 853 B/s wr, 12 op/s 2024-03-18T22:43:06.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:06 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1281877988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:43:06.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:06 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2648259975' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:43:06.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:06 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2925852445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:43:06.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:06 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3123390234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:43:06.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:06 smithi103 ceph-mon[66377]: pgmap v364: 129 pgs: 129 active+clean; 649 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 853 B/s wr, 12 op/s 2024-03-18T22:43:06.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:06 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1281877988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:43:06.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:06 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2648259975' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:43:06.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:06 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2925852445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:43:06.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:06 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3123390234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:43:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.433+0000 7f3e1f587640 1 -- 172.21.15.40:0/3327729194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e181022e0 msgr2=0x7f3e18108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.433+0000 7f3e1f587640 1 --2- 172.21.15.40:0/3327729194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e181022e0 0x7f3e18108b70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3e10009ca0 tx=0x7f3e1002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.433+0000 7f3e1f587640 1 -- 172.21.15.40:0/3327729194 shutdown_connections 2024-03-18T22:43:06.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.433+0000 7f3e1f587640 1 --2- 172.21.15.40:0/3327729194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e181022e0 0x7f3e18108b70 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.433+0000 7f3e1f587640 1 --2- 172.21.15.40:0/3327729194 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e18101920 0x7f3e18101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.433+0000 7f3e1f587640 1 --2- 172.21.15.40:0/3327729194 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e18100730 0x7f3e18100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.433+0000 7f3e1f587640 1 -- 172.21.15.40:0/3327729194 >> 172.21.15.40:0/3327729194 conn(0x7f3e180fbcc0 msgr2=0x7f3e180fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:06.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.433+0000 7f3e1f587640 1 -- 172.21.15.40:0/3327729194 shutdown_connections 2024-03-18T22:43:06.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.433+0000 7f3e1f587640 1 -- 172.21.15.40:0/3327729194 wait complete. 2024-03-18T22:43:06.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1f587640 1 Processor -- start 2024-03-18T22:43:06.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1f587640 1 -- start start 2024-03-18T22:43:06.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1f587640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e18100730 0x7f3e181979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:06.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1f587640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e18101920 0x7f3e18197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:06.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1f587640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e181022e0 0x7f3e1819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:06.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1f587640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1810d380 con 0x7f3e18101920 2024-03-18T22:43:06.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1f587640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1810d200 con 0x7f3e18100730 2024-03-18T22:43:06.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1f587640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1810d500 con 0x7f3e181022e0 2024-03-18T22:43:06.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1cb22640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e18101920 0x7f3e18197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:06.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1cb22640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e18101920 0x7f3e18197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37298/0 (socket says 172.21.15.40:37298) 2024-03-18T22:43:06.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1cb22640 1 -- 172.21.15.40:0/3742901588 learned_addr learned my addr 172.21.15.40:0/3742901588 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:06.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1db24640 1 --2- 172.21.15.40:0/3742901588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e181022e0 0x7f3e1819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:06.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1d323640 1 --2- 172.21.15.40:0/3742901588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e18100730 0x7f3e181979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:06.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1cb22640 1 -- 172.21.15.40:0/3742901588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e181022e0 msgr2=0x7f3e1819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1cb22640 1 --2- 172.21.15.40:0/3742901588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e181022e0 0x7f3e1819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1cb22640 1 -- 172.21.15.40:0/3742901588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e18100730 msgr2=0x7f3e181979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1cb22640 1 --2- 172.21.15.40:0/3742901588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e18100730 0x7f3e181979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1cb22640 1 -- 172.21.15.40:0/3742901588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1819f600 con 0x7f3e18101920 2024-03-18T22:43:06.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e1cb22640 1 --2- 172.21.15.40:0/3742901588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e18101920 0x7f3e18197f20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800e9f0 tx=0x7f3e0800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:06.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.434+0000 7f3e0e7fc640 1 -- 172.21.15.40:0/3742901588 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0800cd80 con 0x7f3e18101920 2024-03-18T22:43:06.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.435+0000 7f3e0e7fc640 1 -- 172.21.15.40:0/3742901588 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e080108e0 con 0x7f3e18101920 2024-03-18T22:43:06.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.435+0000 7f3e1f587640 1 -- 172.21.15.40:0/3742901588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e1819f8f0 con 0x7f3e18101920 2024-03-18T22:43:06.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.435+0000 7f3e0e7fc640 1 -- 172.21.15.40:0/3742901588 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e08018920 con 0x7f3e18101920 2024-03-18T22:43:06.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.435+0000 7f3e1f587640 1 -- 172.21.15.40:0/3742901588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e1819fe40 con 0x7f3e18101920 2024-03-18T22:43:06.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.436+0000 7f3e1f587640 1 -- 172.21.15.40:0/3742901588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e18068a60 con 0x7f3e18101920 2024-03-18T22:43:06.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.437+0000 7f3e0e7fc640 1 -- 172.21.15.40:0/3742901588 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3e08010a90 con 0x7f3e18101920 2024-03-18T22:43:06.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.437+0000 7f3e0e7fc640 1 --2- 172.21.15.40:0/3742901588 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3e040a2800 0x7f3e040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:06.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.437+0000 7f3e1d323640 1 --2- 172.21.15.40:0/3742901588 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3e040a2800 0x7f3e040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:06.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.437+0000 7f3e0e7fc640 1 -- 172.21.15.40:0/3742901588 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f3e080cf610 con 0x7f3e18101920 2024-03-18T22:43:06.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.438+0000 7f3e1d323640 1 --2- 172.21.15.40:0/3742901588 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3e040a2800 0x7f3e040a4cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3e140095f0 tx=0x7f3e14005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:06.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.441+0000 7f3e0e7fc640 1 -- 172.21.15.40:0/3742901588 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3e0809bc70 con 0x7f3e18101920 2024-03-18T22:43:06.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.623+0000 7f3e1f587640 1 -- 172.21.15.40:0/3742901588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3e1802d220 con 0x7f3e18101920 2024-03-18T22:43:06.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.623+0000 7f3e0e7fc640 1 -- 172.21.15.40:0/3742901588 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f3e0809fd10 con 0x7f3e18101920 2024-03-18T22:43:06.624 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:43:06.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.625+0000 7f3e1f587640 1 -- 172.21.15.40:0/3742901588 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3e040a2800 msgr2=0x7f3e040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.625+0000 7f3e1f587640 1 --2- 172.21.15.40:0/3742901588 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3e040a2800 0x7f3e040a4cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3e140095f0 tx=0x7f3e14005f90 comp rx=0 tx=0).stop 2024-03-18T22:43:06.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.625+0000 7f3e1f587640 1 -- 172.21.15.40:0/3742901588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e18101920 msgr2=0x7f3e18197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.625+0000 7f3e1f587640 1 --2- 172.21.15.40:0/3742901588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e18101920 0x7f3e18197f20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800e9f0 tx=0x7f3e0800eeb0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.625+0000 7f3e1f587640 1 -- 172.21.15.40:0/3742901588 shutdown_connections 2024-03-18T22:43:06.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.625+0000 7f3e1f587640 1 --2- 172.21.15.40:0/3742901588 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3e040a2800 0x7f3e040a4cc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.625+0000 7f3e1f587640 1 --2- 172.21.15.40:0/3742901588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3e181022e0 0x7f3e1819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.625+0000 7f3e1f587640 1 --2- 172.21.15.40:0/3742901588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e18101920 0x7f3e18197f20 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.625+0000 7f3e1f587640 1 --2- 172.21.15.40:0/3742901588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e18100730 0x7f3e181979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.625+0000 7f3e1f587640 1 -- 172.21.15.40:0/3742901588 >> 172.21.15.40:0/3742901588 conn(0x7f3e180fbcc0 msgr2=0x7f3e18104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:06.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.625+0000 7f3e1f587640 1 -- 172.21.15.40:0/3742901588 shutdown_connections 2024-03-18T22:43:06.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.625+0000 7f3e1f587640 1 -- 172.21.15.40:0/3742901588 wait complete. 2024-03-18T22:43:06.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T22:43:06.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.741+0000 7f6af15d4640 1 -- 172.21.15.40:0/3108164611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aec0ffe70 msgr2=0x7f6aec100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.741+0000 7f6af15d4640 1 --2- 172.21.15.40:0/3108164611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aec0ffe70 0x7f6aec100290 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009900 tx=0x7f6adc02bc30 comp rx=0 tx=0).stop 2024-03-18T22:43:06.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.742+0000 7f6af15d4640 1 -- 172.21.15.40:0/3108164611 shutdown_connections 2024-03-18T22:43:06.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.742+0000 7f6af15d4640 1 --2- 172.21.15.40:0/3108164611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec101a10 0x7f6aec108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.742+0000 7f6af15d4640 1 --2- 172.21.15.40:0/3108164611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6aec101050 0x7f6aec1014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.742+0000 7f6af15d4640 1 --2- 172.21.15.40:0/3108164611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aec0ffe70 0x7f6aec100290 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.742+0000 7f6af15d4640 1 -- 172.21.15.40:0/3108164611 >> 172.21.15.40:0/3108164611 conn(0x7f6aec0fb450 msgr2=0x7f6aec0fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:06.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.742+0000 7f6af15d4640 1 -- 172.21.15.40:0/3108164611 shutdown_connections 2024-03-18T22:43:06.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.742+0000 7f6af15d4640 1 -- 172.21.15.40:0/3108164611 wait complete. 2024-03-18T22:43:06.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.742+0000 7f6af15d4640 1 Processor -- start 2024-03-18T22:43:06.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.742+0000 7f6af15d4640 1 -- start start 2024-03-18T22:43:06.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.742+0000 7f6af15d4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec0ffe70 0x7f6aec103e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:06.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.742+0000 7f6af15d4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6aec101050 0x7f6aec1024c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:06.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.742+0000 7f6af15d4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aec101a10 0x7f6aec102a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:06.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.742+0000 7f6af15d4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec10b9e0 con 0x7f6aec101a10 2024-03-18T22:43:06.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.742+0000 7f6af15d4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec10b860 con 0x7f6aec0ffe70 2024-03-18T22:43:06.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.742+0000 7f6af15d4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec10bb60 con 0x7f6aec101050 2024-03-18T22:43:06.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6aeb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aec101a10 0x7f6aec102a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:06.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6aeaffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec0ffe70 0x7f6aec103e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:06.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6aeb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aec101a10 0x7f6aec102a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37310/0 (socket says 172.21.15.40:37310) 2024-03-18T22:43:06.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6aeb7fe640 1 -- 172.21.15.40:0/2807692769 learned_addr learned my addr 172.21.15.40:0/2807692769 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:06.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6aea7fc640 1 --2- 172.21.15.40:0/2807692769 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6aec101050 0x7f6aec1024c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:06.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6aeb7fe640 1 -- 172.21.15.40:0/2807692769 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6aec101050 msgr2=0x7f6aec1024c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6aeb7fe640 1 --2- 172.21.15.40:0/2807692769 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6aec101050 0x7f6aec1024c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6aeb7fe640 1 -- 172.21.15.40:0/2807692769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec0ffe70 msgr2=0x7f6aec103e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6aeb7fe640 1 --2- 172.21.15.40:0/2807692769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec0ffe70 0x7f6aec103e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6aeb7fe640 1 -- 172.21.15.40:0/2807692769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aec103290 con 0x7f6aec101a10 2024-03-18T22:43:06.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6aeb7fe640 1 --2- 172.21.15.40:0/2807692769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aec101a10 0x7f6aec102a00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6ae400eb10 tx=0x7f6ae400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:06.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6ad3fff640 1 -- 172.21.15.40:0/2807692769 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae4004b20 con 0x7f6aec101a10 2024-03-18T22:43:06.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6ad3fff640 1 -- 172.21.15.40:0/2807692769 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ae4004cc0 con 0x7f6aec101a10 2024-03-18T22:43:06.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6af15d4640 1 -- 172.21.15.40:0/2807692769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aec1a1c50 con 0x7f6aec101a10 2024-03-18T22:43:06.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6ad3fff640 1 -- 172.21.15.40:0/2807692769 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae4010690 con 0x7f6aec101a10 2024-03-18T22:43:06.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.743+0000 7f6af15d4640 1 -- 172.21.15.40:0/2807692769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aec1a2150 con 0x7f6aec101a10 2024-03-18T22:43:06.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.744+0000 7f6af15d4640 1 -- 172.21.15.40:0/2807692769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aec107c30 con 0x7f6aec101a10 2024-03-18T22:43:06.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.745+0000 7f6ad3fff640 1 -- 172.21.15.40:0/2807692769 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6ae4005020 con 0x7f6aec101a10 2024-03-18T22:43:06.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.746+0000 7f6ad3fff640 1 --2- 172.21.15.40:0/2807692769 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ad40a2800 0x7f6ad40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:06.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.746+0000 7f6aeaffd640 1 --2- 172.21.15.40:0/2807692769 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ad40a2800 0x7f6ad40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:06.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.746+0000 7f6ad3fff640 1 -- 172.21.15.40:0/2807692769 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6ae40cf3b0 con 0x7f6aec101a10 2024-03-18T22:43:06.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.747+0000 7f6aeaffd640 1 --2- 172.21.15.40:0/2807692769 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ad40a2800 0x7f6ad40a4cc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009230 tx=0x7f6adc02c740 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:06.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.751+0000 7f6ad3fff640 1 -- 172.21.15.40:0/2807692769 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6ae409c100 con 0x7f6aec101a10 2024-03-18T22:43:06.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.935+0000 7f6af15d4640 1 -- 172.21.15.40:0/2807692769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6aec02d440 con 0x7f6aec101a10 2024-03-18T22:43:06.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.936+0000 7f6ad3fff640 1 -- 172.21.15.40:0/2807692769 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f6ae409ba00 con 0x7f6aec101a10 2024-03-18T22:43:06.953 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-18T22:43:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.937+0000 7f6af15d4640 1 -- 172.21.15.40:0/2807692769 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ad40a2800 msgr2=0x7f6ad40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.937+0000 7f6af15d4640 1 --2- 172.21.15.40:0/2807692769 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ad40a2800 0x7f6ad40a4cc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6adc009230 tx=0x7f6adc02c740 comp rx=0 tx=0).stop 2024-03-18T22:43:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.937+0000 7f6af15d4640 1 -- 172.21.15.40:0/2807692769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aec101a10 msgr2=0x7f6aec102a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:06.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.937+0000 7f6af15d4640 1 --2- 172.21.15.40:0/2807692769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aec101a10 0x7f6aec102a00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6ae400eb10 tx=0x7f6ae400c2d0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.937+0000 7f6af15d4640 1 -- 172.21.15.40:0/2807692769 shutdown_connections 2024-03-18T22:43:06.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.937+0000 7f6af15d4640 1 --2- 172.21.15.40:0/2807692769 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ad40a2800 0x7f6ad40a4cc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.937+0000 7f6af15d4640 1 --2- 172.21.15.40:0/2807692769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6aec101a10 0x7f6aec102a00 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.937+0000 7f6af15d4640 1 --2- 172.21.15.40:0/2807692769 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6aec101050 0x7f6aec1024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.937+0000 7f6af15d4640 1 --2- 172.21.15.40:0/2807692769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec0ffe70 0x7f6aec103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:06.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.937+0000 7f6af15d4640 1 -- 172.21.15.40:0/2807692769 >> 172.21.15.40:0/2807692769 conn(0x7f6aec0fb450 msgr2=0x7f6aec0fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:06.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.938+0000 7f6af15d4640 1 -- 172.21.15.40:0/2807692769 shutdown_connections 2024-03-18T22:43:06.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:06.938+0000 7f6af15d4640 1 -- 172.21.15.40:0/2807692769 wait complete. 2024-03-18T22:43:06.955 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T22:43:07.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.061+0000 7fb1581d4640 1 -- 172.21.15.40:0/1025656924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb150104c10 msgr2=0x7fb150107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:07.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.061+0000 7fb1581d4640 1 --2- 172.21.15.40:0/1025656924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb150104c10 0x7fb150107000 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb144009900 tx=0x7fb14402bc30 comp rx=0 tx=0).stop 2024-03-18T22:43:07.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.061+0000 7fb1581d4640 1 -- 172.21.15.40:0/1025656924 shutdown_connections 2024-03-18T22:43:07.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.061+0000 7fb1581d4640 1 --2- 172.21.15.40:0/1025656924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb150107540 0x7fb150109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.061+0000 7fb1581d4640 1 --2- 172.21.15.40:0/1025656924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb150104c10 0x7fb150107000 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.061+0000 7fb1581d4640 1 --2- 172.21.15.40:0/1025656924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1501022e0 0x7fb1501046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.061+0000 7fb1581d4640 1 -- 172.21.15.40:0/1025656924 >> 172.21.15.40:0/1025656924 conn(0x7fb1500fbcb0 msgr2=0x7fb1500fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:07.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.061+0000 7fb1581d4640 1 -- 172.21.15.40:0/1025656924 shutdown_connections 2024-03-18T22:43:07.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.061+0000 7fb1581d4640 1 -- 172.21.15.40:0/1025656924 wait complete. 2024-03-18T22:43:07.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.061+0000 7fb1581d4640 1 Processor -- start 2024-03-18T22:43:07.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.061+0000 7fb1581d4640 1 -- start start 2024-03-18T22:43:07.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb1581d4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1501022e0 0x7fb150197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:07.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb1581d4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb150104c10 0x7fb1501980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:07.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb1581d4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb150107540 0x7fb15019f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:07.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb1581d4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb15010d350 con 0x7fb150107540 2024-03-18T22:43:07.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb1581d4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb15010d1d0 con 0x7fb150104c10 2024-03-18T22:43:07.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb1581d4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb15010d4d0 con 0x7fb1501022e0 2024-03-18T22:43:07.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb156771640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb150107540 0x7fb15019f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:07.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb156771640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb150107540 0x7fb15019f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37328/0 (socket says 172.21.15.40:37328) 2024-03-18T22:43:07.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb156771640 1 -- 172.21.15.40:0/544968523 learned_addr learned my addr 172.21.15.40:0/544968523 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:07.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb15576f640 1 --2- 172.21.15.40:0/544968523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb150104c10 0x7fb1501980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:07.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb155f70640 1 --2- 172.21.15.40:0/544968523 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1501022e0 0x7fb150197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:07.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb156771640 1 -- 172.21.15.40:0/544968523 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1501022e0 msgr2=0x7fb150197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:07.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb156771640 1 --2- 172.21.15.40:0/544968523 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1501022e0 0x7fb150197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb156771640 1 -- 172.21.15.40:0/544968523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb150104c10 msgr2=0x7fb1501980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:07.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb156771640 1 --2- 172.21.15.40:0/544968523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb150104c10 0x7fb1501980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb156771640 1 -- 172.21.15.40:0/544968523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb15019f780 con 0x7fb150107540 2024-03-18T22:43:07.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb156771640 1 --2- 172.21.15.40:0/544968523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb150107540 0x7fb15019f110 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb148009fd0 tx=0x7fb14800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:07.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb142ffd640 1 -- 172.21.15.40:0/544968523 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb148017070 con 0x7fb150107540 2024-03-18T22:43:07.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb142ffd640 1 -- 172.21.15.40:0/544968523 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb148003e20 con 0x7fb150107540 2024-03-18T22:43:07.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb142ffd640 1 -- 172.21.15.40:0/544968523 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb148010460 con 0x7fb150107540 2024-03-18T22:43:07.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb1581d4640 1 -- 172.21.15.40:0/544968523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb15019fa70 con 0x7fb150107540 2024-03-18T22:43:07.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.062+0000 7fb1581d4640 1 -- 172.21.15.40:0/544968523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb15019ff90 con 0x7fb150107540 2024-03-18T22:43:07.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.063+0000 7fb1581d4640 1 -- 172.21.15.40:0/544968523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1500fd7a0 con 0x7fb150107540 2024-03-18T22:43:07.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.064+0000 7fb142ffd640 1 -- 172.21.15.40:0/544968523 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb148003970 con 0x7fb150107540 2024-03-18T22:43:07.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.065+0000 7fb142ffd640 1 --2- 172.21.15.40:0/544968523 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb13c0a2880 0x7fb13c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:07.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.065+0000 7fb155f70640 1 --2- 172.21.15.40:0/544968523 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb13c0a2880 0x7fb13c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:07.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.065+0000 7fb142ffd640 1 -- 172.21.15.40:0/544968523 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb1480ce7d0 con 0x7fb150107540 2024-03-18T22:43:07.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.065+0000 7fb155f70640 1 --2- 172.21.15.40:0/544968523 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb13c0a2880 0x7fb13c0a4d40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb14c009bd0 tx=0x7fb14c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:07.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.069+0000 7fb142ffd640 1 -- 172.21.15.40:0/544968523 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb14809ae30 con 0x7fb150107540 2024-03-18T22:43:07.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.253+0000 7fb1581d4640 1 -- 172.21.15.40:0/544968523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb150062420 con 0x7fb150107540 2024-03-18T22:43:07.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.254+0000 7fb142ffd640 1 -- 172.21.15.40:0/544968523 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fb14809eed0 con 0x7fb150107540 2024-03-18T22:43:07.255 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:43:07.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.256+0000 7fb1581d4640 1 -- 172.21.15.40:0/544968523 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb13c0a2880 msgr2=0x7fb13c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:07.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.256+0000 7fb1581d4640 1 --2- 172.21.15.40:0/544968523 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb13c0a2880 0x7fb13c0a4d40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb14c009bd0 tx=0x7fb14c005f90 comp rx=0 tx=0).stop 2024-03-18T22:43:07.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.256+0000 7fb1581d4640 1 -- 172.21.15.40:0/544968523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb150107540 msgr2=0x7fb15019f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:07.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.256+0000 7fb1581d4640 1 --2- 172.21.15.40:0/544968523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb150107540 0x7fb15019f110 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb148009fd0 tx=0x7fb14800b260 comp rx=0 tx=0).stop 2024-03-18T22:43:07.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.256+0000 7fb1581d4640 1 -- 172.21.15.40:0/544968523 shutdown_connections 2024-03-18T22:43:07.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.256+0000 7fb1581d4640 1 --2- 172.21.15.40:0/544968523 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb13c0a2880 0x7fb13c0a4d40 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.256+0000 7fb1581d4640 1 --2- 172.21.15.40:0/544968523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb150107540 0x7fb15019f110 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.256+0000 7fb1581d4640 1 --2- 172.21.15.40:0/544968523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb150104c10 0x7fb1501980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.256+0000 7fb1581d4640 1 --2- 172.21.15.40:0/544968523 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb1501022e0 0x7fb150197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.256+0000 7fb1581d4640 1 -- 172.21.15.40:0/544968523 >> 172.21.15.40:0/544968523 conn(0x7fb1500fbcb0 msgr2=0x7fb150102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:07.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.256+0000 7fb1581d4640 1 -- 172.21.15.40:0/544968523 shutdown_connections 2024-03-18T22:43:07.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.256+0000 7fb1581d4640 1 -- 172.21.15.40:0/544968523 wait complete. 2024-03-18T22:43:07.266 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:07 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3556458307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:43:07.266 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:07 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3742901588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:43:07.266 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:07 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2807692769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:43:07.267 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-18T22:43:07.267 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:43:07.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:07 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3556458307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:43:07.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:07 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3742901588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:43:07.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:07 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2807692769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:43:07.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:07 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3556458307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:43:07.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:07 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3742901588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:43:07.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:07 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2807692769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:43:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.369+0000 7f97583d4640 1 -- 172.21.15.40:0/3095750742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9750104cd0 msgr2=0x7f97501070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.369+0000 7f97583d4640 1 --2- 172.21.15.40:0/3095750742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9750104cd0 0x7f97501070c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9740009900 tx=0x7f974002bc30 comp rx=0 tx=0).stop 2024-03-18T22:43:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.369+0000 7f97583d4640 1 -- 172.21.15.40:0/3095750742 shutdown_connections 2024-03-18T22:43:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.369+0000 7f97583d4640 1 --2- 172.21.15.40:0/3095750742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9750107600 0x7f97501099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.369+0000 7f97583d4640 1 --2- 172.21.15.40:0/3095750742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9750104cd0 0x7f97501070c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.369+0000 7f97583d4640 1 --2- 172.21.15.40:0/3095750742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97501023a0 0x7f9750104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.369+0000 7f97583d4640 1 -- 172.21.15.40:0/3095750742 >> 172.21.15.40:0/3095750742 conn(0x7f97500fbd50 msgr2=0x7f97500fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.369+0000 7f97583d4640 1 -- 172.21.15.40:0/3095750742 shutdown_connections 2024-03-18T22:43:07.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.369+0000 7f97583d4640 1 -- 172.21.15.40:0/3095750742 wait complete. 2024-03-18T22:43:07.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f97583d4640 1 Processor -- start 2024-03-18T22:43:07.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f97583d4640 1 -- start start 2024-03-18T22:43:07.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f97583d4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97501023a0 0x7f9750197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:07.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f97583d4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9750104cd0 0x7f9750198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:07.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f97583d4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9750107600 0x7f975019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:07.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f97583d4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f975010d350 con 0x7f97501023a0 2024-03-18T22:43:07.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f97583d4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f975010d1d0 con 0x7f9750107600 2024-03-18T22:43:07.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f97583d4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f975010d4d0 con 0x7f9750104cd0 2024-03-18T22:43:07.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f9756170640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97501023a0 0x7f9750197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:07.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f9756170640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97501023a0 0x7f9750197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37338/0 (socket says 172.21.15.40:37338) 2024-03-18T22:43:07.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f9756170640 1 -- 172.21.15.40:0/3816420584 learned_addr learned my addr 172.21.15.40:0/3816420584 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:07.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f975596f640 1 --2- 172.21.15.40:0/3816420584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9750104cd0 0x7f9750198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:07.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f9756971640 1 --2- 172.21.15.40:0/3816420584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9750107600 0x7f975019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:07.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f9756170640 1 -- 172.21.15.40:0/3816420584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9750104cd0 msgr2=0x7f9750198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:07.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f9756170640 1 --2- 172.21.15.40:0/3816420584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9750104cd0 0x7f9750198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f9756170640 1 -- 172.21.15.40:0/3816420584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9750107600 msgr2=0x7f975019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:07.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f9756170640 1 --2- 172.21.15.40:0/3816420584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9750107600 0x7f975019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f9756170640 1 -- 172.21.15.40:0/3816420584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f975019f6e0 con 0x7f97501023a0 2024-03-18T22:43:07.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.370+0000 7f9756170640 1 --2- 172.21.15.40:0/3816420584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97501023a0 0x7f9750197ac0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f974c00ea60 tx=0x7f974c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:07.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.371+0000 7f97477fe640 1 -- 172.21.15.40:0/3816420584 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c00cde0 con 0x7f97501023a0 2024-03-18T22:43:07.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.371+0000 7f97477fe640 1 -- 172.21.15.40:0/3816420584 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f974c0035f0 con 0x7f97501023a0 2024-03-18T22:43:07.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.371+0000 7f97583d4640 1 -- 172.21.15.40:0/3816420584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f975019f9d0 con 0x7f97501023a0 2024-03-18T22:43:07.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.371+0000 7f97477fe640 1 -- 172.21.15.40:0/3816420584 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c010690 con 0x7f97501023a0 2024-03-18T22:43:07.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.371+0000 7f97583d4640 1 -- 172.21.15.40:0/3816420584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f975019ff40 con 0x7f97501023a0 2024-03-18T22:43:07.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.372+0000 7f97583d4640 1 -- 172.21.15.40:0/3816420584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97500fd840 con 0x7f97501023a0 2024-03-18T22:43:07.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.373+0000 7f97477fe640 1 -- 172.21.15.40:0/3816420584 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f974c0037a0 con 0x7f97501023a0 2024-03-18T22:43:07.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.373+0000 7f97477fe640 1 --2- 172.21.15.40:0/3816420584 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f973c0a2880 0x7f973c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:07.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.373+0000 7f975596f640 1 --2- 172.21.15.40:0/3816420584 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f973c0a2880 0x7f973c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:07.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.373+0000 7f97477fe640 1 -- 172.21.15.40:0/3816420584 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f974c0cf2a0 con 0x7f97501023a0 2024-03-18T22:43:07.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.374+0000 7f975596f640 1 --2- 172.21.15.40:0/3816420584 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f973c0a2880 0x7f973c0a4d40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9740005ab0 tx=0x7f9740005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:07.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.377+0000 7f97477fe640 1 -- 172.21.15.40:0/3816420584 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f974c053740 con 0x7f97501023a0 2024-03-18T22:43:07.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.597+0000 7f97583d4640 1 -- 172.21.15.40:0/3816420584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f975002d1f0 con 0x7f97501023a0 2024-03-18T22:43:07.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.598+0000 7f97477fe640 1 -- 172.21.15.40:0/3816420584 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11947 (secure 0 0 0) 0x7f974c09bf50 con 0x7f97501023a0 2024-03-18T22:43:07.599 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:43:07.599 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:43:03.220450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24388,24401,24473,24481,24509]},"id":1}]} 2024-03-18T22:43:07.599 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 28 2024-03-18T22:43:07.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.600+0000 7f97583d4640 1 -- 172.21.15.40:0/3816420584 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f973c0a2880 msgr2=0x7f973c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:07.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.600+0000 7f97583d4640 1 --2- 172.21.15.40:0/3816420584 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f973c0a2880 0x7f973c0a4d40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9740005ab0 tx=0x7f9740005fd0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.600+0000 7f97583d4640 1 -- 172.21.15.40:0/3816420584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97501023a0 msgr2=0x7f9750197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:07.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.600+0000 7f97583d4640 1 --2- 172.21.15.40:0/3816420584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97501023a0 0x7f9750197ac0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f974c00ea60 tx=0x7f974c00ef20 comp rx=0 tx=0).stop 2024-03-18T22:43:07.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.600+0000 7f97583d4640 1 -- 172.21.15.40:0/3816420584 shutdown_connections 2024-03-18T22:43:07.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.600+0000 7f97583d4640 1 --2- 172.21.15.40:0/3816420584 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f973c0a2880 0x7f973c0a4d40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.600+0000 7f97583d4640 1 --2- 172.21.15.40:0/3816420584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9750107600 0x7f975019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.600+0000 7f97583d4640 1 --2- 172.21.15.40:0/3816420584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9750104cd0 0x7f9750198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.600+0000 7f97583d4640 1 --2- 172.21.15.40:0/3816420584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97501023a0 0x7f9750197ac0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.600+0000 7f97583d4640 1 -- 172.21.15.40:0/3816420584 >> 172.21.15.40:0/3816420584 conn(0x7f97500fbd50 msgr2=0x7f9750102ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:07.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.600+0000 7f97583d4640 1 -- 172.21.15.40:0/3816420584 shutdown_connections 2024-03-18T22:43:07.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.600+0000 7f97583d4640 1 -- 172.21.15.40:0/3816420584 wait complete. 2024-03-18T22:43:07.611 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-18T22:43:07.611 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-18T22:43:07.612 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T22:43:07.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.714+0000 7fccc0777640 1 -- 172.21.15.40:0/1149497503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccb8100740 msgr2=0x7fccb8100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:07.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.714+0000 7fccc0777640 1 --2- 172.21.15.40:0/1149497503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccb8100740 0x7fccb8100b60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009900 tx=0x7fccb002bc30 comp rx=0 tx=0).stop 2024-03-18T22:43:07.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.714+0000 7fccc0777640 1 -- 172.21.15.40:0/1149497503 shutdown_connections 2024-03-18T22:43:07.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.714+0000 7fccc0777640 1 --2- 172.21.15.40:0/1149497503 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb81022f0 0x7fccb8108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.714+0000 7fccc0777640 1 --2- 172.21.15.40:0/1149497503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb8101930 0x7fccb8101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.714+0000 7fccc0777640 1 --2- 172.21.15.40:0/1149497503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccb8100740 0x7fccb8100b60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.714+0000 7fccc0777640 1 -- 172.21.15.40:0/1149497503 >> 172.21.15.40:0/1149497503 conn(0x7fccb80fbcb0 msgr2=0x7fccb80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:07.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.714+0000 7fccc0777640 1 -- 172.21.15.40:0/1149497503 shutdown_connections 2024-03-18T22:43:07.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.714+0000 7fccc0777640 1 -- 172.21.15.40:0/1149497503 wait complete. 2024-03-18T22:43:07.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccc0777640 1 Processor -- start 2024-03-18T22:43:07.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccc0777640 1 -- start start 2024-03-18T22:43:07.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccc0777640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb8100740 0x7fccb8197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:07.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccc0777640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb8101930 0x7fccb8198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:07.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccc0777640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccb81022f0 0x7fccb819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:07.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccc0777640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fccb810c240 con 0x7fccb8101930 2024-03-18T22:43:07.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccc0777640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fccb810c0c0 con 0x7fccb81022f0 2024-03-18T22:43:07.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccc0777640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fccb810c3c0 con 0x7fccb8100740 2024-03-18T22:43:07.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccbdd12640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb8101930 0x7fccb8198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:07.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccbdd12640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb8101930 0x7fccb8198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37350/0 (socket says 172.21.15.40:37350) 2024-03-18T22:43:07.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccbdd12640 1 -- 172.21.15.40:0/3052992715 learned_addr learned my addr 172.21.15.40:0/3052992715 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:07.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccbe513640 1 --2- 172.21.15.40:0/3052992715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb8100740 0x7fccb8197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:07.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccbed14640 1 --2- 172.21.15.40:0/3052992715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccb81022f0 0x7fccb819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:07.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccbdd12640 1 -- 172.21.15.40:0/3052992715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb8100740 msgr2=0x7fccb8197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:07.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccbdd12640 1 --2- 172.21.15.40:0/3052992715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb8100740 0x7fccb8197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccbdd12640 1 -- 172.21.15.40:0/3052992715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccb81022f0 msgr2=0x7fccb819f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:07.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccbdd12640 1 --2- 172.21.15.40:0/3052992715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccb81022f0 0x7fccb819f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccbdd12640 1 -- 172.21.15.40:0/3052992715 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb819f710 con 0x7fccb8101930 2024-03-18T22:43:07.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccbe513640 1 --2- 172.21.15.40:0/3052992715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb8100740 0x7fccb8197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:07.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.715+0000 7fccbdd12640 1 --2- 172.21.15.40:0/3052992715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb8101930 0x7fccb8198030 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fccac00e9f0 tx=0x7fccac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:07.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.716+0000 7fccb77fe640 1 -- 172.21.15.40:0/3052992715 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccac00ccc0 con 0x7fccb8101930 2024-03-18T22:43:07.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.716+0000 7fccb77fe640 1 -- 172.21.15.40:0/3052992715 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fccac0108e0 con 0x7fccb8101930 2024-03-18T22:43:07.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.716+0000 7fccc0777640 1 -- 172.21.15.40:0/3052992715 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccb819fa00 con 0x7fccb8101930 2024-03-18T22:43:07.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.716+0000 7fccb77fe640 1 -- 172.21.15.40:0/3052992715 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccac0187c0 con 0x7fccb8101930 2024-03-18T22:43:07.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.716+0000 7fccc0777640 1 -- 172.21.15.40:0/3052992715 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccb819ff50 con 0x7fccb8101930 2024-03-18T22:43:07.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.717+0000 7fccc0777640 1 -- 172.21.15.40:0/3052992715 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccb81a0180 con 0x7fccb8101930 2024-03-18T22:43:07.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.718+0000 7fccb77fe640 1 -- 172.21.15.40:0/3052992715 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fccac010a90 con 0x7fccb8101930 2024-03-18T22:43:07.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.718+0000 7fccb77fe640 1 --2- 172.21.15.40:0/3052992715 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcca00a2800 0x7fcca00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:07.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.718+0000 7fccbe513640 1 --2- 172.21.15.40:0/3052992715 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcca00a2800 0x7fcca00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:07.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.718+0000 7fccb77fe640 1 -- 172.21.15.40:0/3052992715 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fccac0cf270 con 0x7fccb8101930 2024-03-18T22:43:07.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.719+0000 7fccbe513640 1 --2- 172.21.15.40:0/3052992715 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcca00a2800 0x7fcca00a4cc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fccb0005ab0 tx=0x7fccb0005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:07.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.723+0000 7fccb77fe640 1 -- 172.21.15.40:0/3052992715 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fccac09bf20 con 0x7fccb8101930 2024-03-18T22:43:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.904+0000 7fccc0777640 1 -- 172.21.15.40:0/3052992715 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fccb806a610 con 0x7fcca00a2800 2024-03-18T22:43:07.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.913+0000 7fccb77fe640 1 -- 172.21.15.40:0/3052992715 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340540 (secure 0 0 0) 0x7fccb806a610 con 0x7fcca00a2800 2024-03-18T22:43:07.918 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:43:07.920 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-18T22:43:07.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.916+0000 7fccc0777640 1 -- 172.21.15.40:0/3052992715 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcca00a2800 msgr2=0x7fcca00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:07.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.916+0000 7fccc0777640 1 --2- 172.21.15.40:0/3052992715 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcca00a2800 0x7fcca00a4cc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fccb0005ab0 tx=0x7fccb0005fd0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.916+0000 7fccc0777640 1 -- 172.21.15.40:0/3052992715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb8101930 msgr2=0x7fccb8198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:07.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.916+0000 7fccc0777640 1 --2- 172.21.15.40:0/3052992715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb8101930 0x7fccb8198030 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fccac00e9f0 tx=0x7fccac00edf0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.916+0000 7fccc0777640 1 -- 172.21.15.40:0/3052992715 shutdown_connections 2024-03-18T22:43:07.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.916+0000 7fccc0777640 1 --2- 172.21.15.40:0/3052992715 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcca00a2800 0x7fcca00a4cc0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.916+0000 7fccc0777640 1 --2- 172.21.15.40:0/3052992715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccb81022f0 0x7fccb819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.916+0000 7fccc0777640 1 --2- 172.21.15.40:0/3052992715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccb8101930 0x7fccb8198030 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.916+0000 7fccc0777640 1 --2- 172.21.15.40:0/3052992715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fccb8100740 0x7fccb8197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:07.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.916+0000 7fccc0777640 1 -- 172.21.15.40:0/3052992715 >> 172.21.15.40:0/3052992715 conn(0x7fccb80fbcb0 msgr2=0x7fccb8069f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:07.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.916+0000 7fccc0777640 1 -- 172.21.15.40:0/3052992715 shutdown_connections 2024-03-18T22:43:07.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:07.916+0000 7fccc0777640 1 -- 172.21.15.40:0/3052992715 wait complete. 2024-03-18T22:43:07.928 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-03-18T22:43:06.844822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664696,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1277,"num_read_kb":795,"num_write":451,"num_write_kb":3103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1182002,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":873,"ondisk_log_size":873,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384576,"kb_used_data":14624,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557248,"statfs":{"total":1151940427776,"available":1151546621952,"internally_reserved":0,"allocated":14974976,"data_stored":8208698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2470,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":109,"num_read_kb":59,"num_write":14,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003309"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947520+0000","last_change":"2024-03-18T22:40:41.806379+0000","last_active":"2024-03-18T22:42:44.947520+0000","last_peered":"2024-03-18T22:42:44.947520+0000","last_clean":"2024-03-18T22:42:44.947520+0000","last_became_active":"2024-03-18T22:40:41.806058+0000","last_became_peered":"2024-03-18T22:40:41.806058+0000","last_unstale":"2024-03-18T22:42:44.947520+0000","last_undegraded":"2024-03-18T22:42:44.947520+0000","last_fullsized":"2024-03-18T22:42:44.947520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:45.834396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955707+0000","last_change":"2024-03-18T22:40:41.813722+0000","last_active":"2024-03-18T22:42:44.955707+0000","last_peered":"2024-03-18T22:42:44.955707+0000","last_clean":"2024-03-18T22:42:44.955707+0000","last_became_active":"2024-03-18T22:40:41.813284+0000","last_became_peered":"2024-03-18T22:40:41.813284+0000","last_unstale":"2024-03-18T22:42:44.955707+0000","last_undegraded":"2024-03-18T22:42:44.955707+0000","last_fullsized":"2024-03-18T22:42:44.955707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:46.374117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955331+0000","last_change":"2024-03-18T22:40:41.808340+0000","last_active":"2024-03-18T22:42:44.955331+0000","last_peered":"2024-03-18T22:42:44.955331+0000","last_clean":"2024-03-18T22:42:44.955331+0000","last_became_active":"2024-03-18T22:40:41.808002+0000","last_became_peered":"2024-03-18T22:40:41.808002+0000","last_unstale":"2024-03-18T22:42:44.955331+0000","last_undegraded":"2024-03-18T22:42:44.955331+0000","last_fullsized":"2024-03-18T22:42:44.955331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:40:32.122124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945721+0000","last_change":"2024-03-18T22:40:41.809959+0000","last_active":"2024-03-18T22:42:44.945721+0000","last_peered":"2024-03-18T22:42:44.945721+0000","last_clean":"2024-03-18T22:42:44.945721+0000","last_became_active":"2024-03-18T22:40:41.809676+0000","last_became_peered":"2024-03-18T22:40:41.809676+0000","last_unstale":"2024-03-18T22:42:44.945721+0000","last_undegraded":"2024-03-18T22:42:44.945721+0000","last_fullsized":"2024-03-18T22:42:44.945721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:29:45.263011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950820+0000","last_change":"2024-03-18T22:40:41.804583+0000","last_active":"2024-03-18T22:42:44.950820+0000","last_peered":"2024-03-18T22:42:44.950820+0000","last_clean":"2024-03-18T22:42:44.950820+0000","last_became_active":"2024-03-18T22:40:41.804064+0000","last_became_peered":"2024-03-18T22:40:41.804064+0000","last_unstale":"2024-03-18T22:42:44.950820+0000","last_undegraded":"2024-03-18T22:42:44.950820+0000","last_fullsized":"2024-03-18T22:42:44.950820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:46:23.455106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955919+0000","last_change":"2024-03-18T22:41:44.301195+0000","last_active":"2024-03-18T22:42:44.955919+0000","last_peered":"2024-03-18T22:42:44.955919+0000","last_clean":"2024-03-18T22:42:44.955919+0000","last_became_active":"2024-03-18T22:41:44.300801+0000","last_became_peered":"2024-03-18T22:41:44.300801+0000","last_unstale":"2024-03-18T22:42:44.955919+0000","last_undegraded":"2024-03-18T22:42:44.955919+0000","last_fullsized":"2024-03-18T22:42:44.955919+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:12:55.579977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947351+0000","last_change":"2024-03-18T22:40:41.814971+0000","last_active":"2024-03-18T22:42:44.947351+0000","last_peered":"2024-03-18T22:42:44.947351+0000","last_clean":"2024-03-18T22:42:44.947351+0000","last_became_active":"2024-03-18T22:40:41.814305+0000","last_became_peered":"2024-03-18T22:40:41.814305+0000","last_unstale":"2024-03-18T22:42:44.947351+0000","last_undegraded":"2024-03-18T22:42:44.947351+0000","last_fullsized":"2024-03-18T22:42:44.947351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:07.390157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950536+0000","last_change":"2024-03-18T22:40:41.815674+0000","last_active":"2024-03-18T22:42:44.950536+0000","last_peered":"2024-03-18T22:42:44.950536+0000","last_clean":"2024-03-18T22:42:44.950536+0000","last_became_active":"2024-03-18T22:40:41.813560+0000","last_became_peered":"2024-03-18T22:40:41.813560+0000","last_unstale":"2024-03-18T22:42:44.950536+0000","last_undegraded":"2024-03-18T22:42:44.950536+0000","last_fullsized":"2024-03-18T22:42:44.950536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:42:32.409338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.958219+0000","last_change":"2024-03-18T22:40:41.817508+0000","last_active":"2024-03-18T22:42:44.958219+0000","last_peered":"2024-03-18T22:42:44.958219+0000","last_clean":"2024-03-18T22:42:44.958219+0000","last_became_active":"2024-03-18T22:40:41.817184+0000","last_became_peered":"2024-03-18T22:40:41.817184+0000","last_unstale":"2024-03-18T22:42:44.958219+0000","last_undegraded":"2024-03-18T22:42:44.958219+0000","last_fullsized":"2024-03-18T22:42:44.958219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:10.255238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956881+0000","last_change":"2024-03-18T22:40:41.806235+0000","last_active":"2024-03-18T22:42:44.956881+0000","last_peered":"2024-03-18T22:42:44.956881+0000","last_clean":"2024-03-18T22:42:44.956881+0000","last_became_active":"2024-03-18T22:40:41.805913+0000","last_became_peered":"2024-03-18T22:40:41.805913+0000","last_unstale":"2024-03-18T22:42:44.956881+0000","last_undegraded":"2024-03-18T22:42:44.956881+0000","last_fullsized":"2024-03-18T22:42:44.956881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:39:06.299989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947974+0000","last_change":"2024-03-18T22:40:41.814735+0000","last_active":"2024-03-18T22:42:44.947974+0000","last_peered":"2024-03-18T22:42:44.947974+0000","last_clean":"2024-03-18T22:42:44.947974+0000","last_became_active":"2024-03-18T22:40:41.814291+0000","last_became_peered":"2024-03-18T22:40:41.814291+0000","last_unstale":"2024-03-18T22:42:44.947974+0000","last_undegraded":"2024-03-18T22:42:44.947974+0000","last_fullsized":"2024-03-18T22:42:44.947974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:49.610674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945638+0000","last_change":"2024-03-18T22:40:41.815561+0000","last_active":"2024-03-18T22:42:44.945638+0000","last_peered":"2024-03-18T22:42:44.945638+0000","last_clean":"2024-03-18T22:42:44.945638+0000","last_became_active":"2024-03-18T22:40:41.815252+0000","last_became_peered":"2024-03-18T22:40:41.815252+0000","last_unstale":"2024-03-18T22:42:44.945638+0000","last_undegraded":"2024-03-18T22:42:44.945638+0000","last_fullsized":"2024-03-18T22:42:44.945638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:34.769168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947541+0000","last_change":"2024-03-18T22:40:41.803325+0000","last_active":"2024-03-18T22:42:44.947541+0000","last_peered":"2024-03-18T22:42:44.947541+0000","last_clean":"2024-03-18T22:42:44.947541+0000","last_became_active":"2024-03-18T22:40:41.803008+0000","last_became_peered":"2024-03-18T22:40:41.803008+0000","last_unstale":"2024-03-18T22:42:44.947541+0000","last_undegraded":"2024-03-18T22:42:44.947541+0000","last_fullsized":"2024-03-18T22:42:44.947541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:14:01.335548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.954563+0000","last_change":"2024-03-18T22:40:41.812541+0000","last_active":"2024-03-18T22:42:44.954563+0000","last_peered":"2024-03-18T22:42:44.954563+0000","last_clean":"2024-03-18T22:42:44.954563+0000","last_became_active":"2024-03-18T22:40:41.812153+0000","last_became_peered":"2024-03-18T22:40:41.812153+0000","last_unstale":"2024-03-18T22:42:44.954563+0000","last_undegraded":"2024-03-18T22:42:44.954563+0000","last_fullsized":"2024-03-18T22:42:44.954563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:04.927057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.954565+0000","last_change":"2024-03-18T22:40:41.815790+0000","last_active":"2024-03-18T22:42:44.954565+0000","last_peered":"2024-03-18T22:42:44.954565+0000","last_clean":"2024-03-18T22:42:44.954565+0000","last_became_active":"2024-03-18T22:40:41.814949+0000","last_became_peered":"2024-03-18T22:40:41.814949+0000","last_unstale":"2024-03-18T22:42:44.954565+0000","last_undegraded":"2024-03-18T22:42:44.954565+0000","last_fullsized":"2024-03-18T22:42:44.954565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:10:15.874285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957433+0000","last_change":"2024-03-18T22:40:41.809980+0000","last_active":"2024-03-18T22:42:44.957433+0000","last_peered":"2024-03-18T22:42:44.957433+0000","last_clean":"2024-03-18T22:42:44.957433+0000","last_became_active":"2024-03-18T22:40:41.809640+0000","last_became_peered":"2024-03-18T22:40:41.809640+0000","last_unstale":"2024-03-18T22:42:44.957433+0000","last_undegraded":"2024-03-18T22:42:44.957433+0000","last_fullsized":"2024-03-18T22:42:44.957433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:25.389027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947447+0000","last_change":"2024-03-18T22:40:41.807088+0000","last_active":"2024-03-18T22:42:44.947447+0000","last_peered":"2024-03-18T22:42:44.947447+0000","last_clean":"2024-03-18T22:42:44.947447+0000","last_became_active":"2024-03-18T22:40:41.806770+0000","last_became_peered":"2024-03-18T22:40:41.806770+0000","last_unstale":"2024-03-18T22:42:44.947447+0000","last_undegraded":"2024-03-18T22:42:44.947447+0000","last_fullsized":"2024-03-18T22:42:44.947447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:53:14.001247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956723+0000","last_change":"2024-03-18T22:40:41.812685+0000","last_active":"2024-03-18T22:42:44.956723+0000","last_peered":"2024-03-18T22:42:44.956723+0000","last_clean":"2024-03-18T22:42:44.956723+0000","last_became_active":"2024-03-18T22:40:41.812346+0000","last_became_peered":"2024-03-18T22:40:41.812346+0000","last_unstale":"2024-03-18T22:42:44.956723+0000","last_undegraded":"2024-03-18T22:42:44.956723+0000","last_fullsized":"2024-03-18T22:42:44.956723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:25.033460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950634+0000","last_change":"2024-03-18T22:40:41.812627+0000","last_active":"2024-03-18T22:42:44.950634+0000","last_peered":"2024-03-18T22:42:44.950634+0000","last_clean":"2024-03-18T22:42:44.950634+0000","last_became_active":"2024-03-18T22:40:41.812327+0000","last_became_peered":"2024-03-18T22:40:41.812327+0000","last_unstale":"2024-03-18T22:42:44.950634+0000","last_undegraded":"2024-03-18T22:42:44.950634+0000","last_fullsized":"2024-03-18T22:42:44.950634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:58:22.088589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949400+0000","last_change":"2024-03-18T22:40:45.847822+0000","last_active":"2024-03-18T22:42:44.949400+0000","last_peered":"2024-03-18T22:42:44.949400+0000","last_clean":"2024-03-18T22:42:44.949400+0000","last_became_active":"2024-03-18T22:40:45.847407+0000","last_became_peered":"2024-03-18T22:40:45.847407+0000","last_unstale":"2024-03-18T22:42:44.949400+0000","last_undegraded":"2024-03-18T22:42:44.949400+0000","last_fullsized":"2024-03-18T22:42:44.949400+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:44.825997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'15","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:01.983401+0000","last_change":"2024-03-18T22:40:41.218797+0000","last_active":"2024-03-18T22:43:01.983401+0000","last_peered":"2024-03-18T22:43:01.983401+0000","last_clean":"2024-03-18T22:43:01.983401+0000","last_became_active":"2024-03-18T22:40:41.218472+0000","last_became_peered":"2024-03-18T22:40:41.218472+0000","last_unstale":"2024-03-18T22:43:01.983401+0000","last_undegraded":"2024-03-18T22:43:01.983401+0000","last_fullsized":"2024-03-18T22:43:01.983401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:10.230257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956484+0000","last_change":"2024-03-18T22:41:44.005444+0000","last_active":"2024-03-18T22:42:44.956484+0000","last_peered":"2024-03-18T22:42:44.956484+0000","last_clean":"2024-03-18T22:42:44.956484+0000","last_became_active":"2024-03-18T22:41:44.005157+0000","last_became_peered":"2024-03-18T22:41:44.005157+0000","last_unstale":"2024-03-18T22:42:44.956484+0000","last_undegraded":"2024-03-18T22:42:44.956484+0000","last_fullsized":"2024-03-18T22:42:44.956484+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:53:16.249875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945333+0000","last_change":"2024-03-18T22:40:45.851126+0000","last_active":"2024-03-18T22:42:44.945333+0000","last_peered":"2024-03-18T22:42:44.945333+0000","last_clean":"2024-03-18T22:42:44.945333+0000","last_became_active":"2024-03-18T22:40:45.850549+0000","last_became_peered":"2024-03-18T22:40:45.850549+0000","last_unstale":"2024-03-18T22:42:44.945333+0000","last_undegraded":"2024-03-18T22:42:44.945333+0000","last_fullsized":"2024-03-18T22:42:44.945333+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:39.359263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951119+0000","last_change":"2024-03-18T22:40:41.220944+0000","last_active":"2024-03-18T22:42:44.951119+0000","last_peered":"2024-03-18T22:42:44.951119+0000","last_clean":"2024-03-18T22:42:44.951119+0000","last_became_active":"2024-03-18T22:40:41.220612+0000","last_became_peered":"2024-03-18T22:40:41.220612+0000","last_unstale":"2024-03-18T22:42:44.951119+0000","last_undegraded":"2024-03-18T22:42:44.951119+0000","last_fullsized":"2024-03-18T22:42:44.951119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:16.083134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.954672+0000","last_change":"2024-03-18T22:40:41.818031+0000","last_active":"2024-03-18T22:42:44.954672+0000","last_peered":"2024-03-18T22:42:44.954672+0000","last_clean":"2024-03-18T22:42:44.954672+0000","last_became_active":"2024-03-18T22:40:41.817693+0000","last_became_peered":"2024-03-18T22:40:41.817693+0000","last_unstale":"2024-03-18T22:42:44.954672+0000","last_undegraded":"2024-03-18T22:42:44.954672+0000","last_fullsized":"2024-03-18T22:42:44.954672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:05.901203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945296+0000","last_change":"2024-03-18T22:40:45.850749+0000","last_active":"2024-03-18T22:42:44.945296+0000","last_peered":"2024-03-18T22:42:44.945296+0000","last_clean":"2024-03-18T22:42:44.945296+0000","last_became_active":"2024-03-18T22:40:45.850318+0000","last_became_peered":"2024-03-18T22:40:45.850318+0000","last_unstale":"2024-03-18T22:42:44.945296+0000","last_undegraded":"2024-03-18T22:42:44.945296+0000","last_fullsized":"2024-03-18T22:42:44.945296+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:56.047413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950968+0000","last_change":"2024-03-18T22:40:41.220775+0000","last_active":"2024-03-18T22:42:44.950968+0000","last_peered":"2024-03-18T22:42:44.950968+0000","last_clean":"2024-03-18T22:42:44.950968+0000","last_became_active":"2024-03-18T22:40:41.220313+0000","last_became_peered":"2024-03-18T22:40:41.220313+0000","last_unstale":"2024-03-18T22:42:44.950968+0000","last_undegraded":"2024-03-18T22:42:44.950968+0000","last_fullsized":"2024-03-18T22:42:44.950968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:24.180884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957754+0000","last_change":"2024-03-18T22:40:41.818448+0000","last_active":"2024-03-18T22:42:44.957754+0000","last_peered":"2024-03-18T22:42:44.957754+0000","last_clean":"2024-03-18T22:42:44.957754+0000","last_became_active":"2024-03-18T22:40:41.818088+0000","last_became_peered":"2024-03-18T22:40:41.818088+0000","last_unstale":"2024-03-18T22:42:44.957754+0000","last_undegraded":"2024-03-18T22:42:44.957754+0000","last_fullsized":"2024-03-18T22:42:44.957754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:34:54.298226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945216+0000","last_change":"2024-03-18T22:40:45.846250+0000","last_active":"2024-03-18T22:42:44.945216+0000","last_peered":"2024-03-18T22:42:44.945216+0000","last_clean":"2024-03-18T22:42:44.945216+0000","last_became_active":"2024-03-18T22:40:45.845859+0000","last_became_peered":"2024-03-18T22:40:45.845859+0000","last_unstale":"2024-03-18T22:42:44.945216+0000","last_undegraded":"2024-03-18T22:42:44.945216+0000","last_fullsized":"2024-03-18T22:42:44.945216+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:57.855301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951899+0000","last_change":"2024-03-18T22:40:41.216569+0000","last_active":"2024-03-18T22:42:44.951899+0000","last_peered":"2024-03-18T22:42:44.951899+0000","last_clean":"2024-03-18T22:42:44.951899+0000","last_became_active":"2024-03-18T22:40:41.216255+0000","last_became_peered":"2024-03-18T22:40:41.216255+0000","last_unstale":"2024-03-18T22:42:44.951899+0000","last_undegraded":"2024-03-18T22:42:44.951899+0000","last_fullsized":"2024-03-18T22:42:44.951899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:23.016346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945576+0000","last_change":"2024-03-18T22:41:44.301578+0000","last_active":"2024-03-18T22:42:44.945576+0000","last_peered":"2024-03-18T22:42:44.945576+0000","last_clean":"2024-03-18T22:42:44.945576+0000","last_became_active":"2024-03-18T22:41:44.301178+0000","last_became_peered":"2024-03-18T22:41:44.301178+0000","last_unstale":"2024-03-18T22:42:44.945576+0000","last_undegraded":"2024-03-18T22:42:44.945576+0000","last_fullsized":"2024-03-18T22:42:44.945576+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:25:50.635812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949131+0000","last_change":"2024-03-18T22:41:44.302390+0000","last_active":"2024-03-18T22:42:44.949131+0000","last_peered":"2024-03-18T22:42:44.949131+0000","last_clean":"2024-03-18T22:42:44.949131+0000","last_became_active":"2024-03-18T22:41:44.301920+0000","last_became_peered":"2024-03-18T22:41:44.301920+0000","last_unstale":"2024-03-18T22:42:44.949131+0000","last_undegraded":"2024-03-18T22:42:44.949131+0000","last_fullsized":"2024-03-18T22:42:44.949131+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:56:12.561520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,7,11],"acting":[2,8,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:02.973659+0000","last_change":"2024-03-18T22:42:44.962925+0000","last_active":"2024-03-18T22:43:02.973659+0000","last_peered":"2024-03-18T22:43:02.973659+0000","last_clean":"2024-03-18T22:43:02.973659+0000","last_became_active":"2024-03-18T22:42:44.951614+0000","last_became_peered":"2024-03-18T22:42:44.951614+0000","last_unstale":"2024-03-18T22:43:02.973659+0000","last_undegraded":"2024-03-18T22:43:02.973659+0000","last_fullsized":"2024-03-18T22:43:02.973659+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:18:00.200940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950735+0000","last_change":"2024-03-18T22:41:44.004814+0000","last_active":"2024-03-18T22:42:44.950735+0000","last_peered":"2024-03-18T22:42:44.950735+0000","last_clean":"2024-03-18T22:42:44.950735+0000","last_became_active":"2024-03-18T22:41:44.004219+0000","last_became_peered":"2024-03-18T22:41:44.004219+0000","last_unstale":"2024-03-18T22:42:44.950735+0000","last_undegraded":"2024-03-18T22:42:44.950735+0000","last_fullsized":"2024-03-18T22:42:44.950735+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:29:03.746508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949773+0000","last_change":"2024-03-18T22:41:44.005790+0000","last_active":"2024-03-18T22:42:44.949773+0000","last_peered":"2024-03-18T22:42:44.949773+0000","last_clean":"2024-03-18T22:42:44.949773+0000","last_became_active":"2024-03-18T22:41:44.005386+0000","last_became_peered":"2024-03-18T22:41:44.005386+0000","last_unstale":"2024-03-18T22:42:44.949773+0000","last_undegraded":"2024-03-18T22:42:44.949773+0000","last_fullsized":"2024-03-18T22:42:44.949773+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:49:08.212639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955868+0000","last_change":"2024-03-18T22:40:41.217449+0000","last_active":"2024-03-18T22:42:44.955868+0000","last_peered":"2024-03-18T22:42:44.955868+0000","last_clean":"2024-03-18T22:42:44.955868+0000","last_became_active":"2024-03-18T22:40:41.217127+0000","last_became_peered":"2024-03-18T22:40:41.217127+0000","last_unstale":"2024-03-18T22:42:44.955868+0000","last_undegraded":"2024-03-18T22:42:44.955868+0000","last_fullsized":"2024-03-18T22:42:44.955868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:08:00.178720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945729+0000","last_change":"2024-03-18T22:40:41.814154+0000","last_active":"2024-03-18T22:42:44.945729+0000","last_peered":"2024-03-18T22:42:44.945729+0000","last_clean":"2024-03-18T22:42:44.945729+0000","last_became_active":"2024-03-18T22:40:41.813824+0000","last_became_peered":"2024-03-18T22:40:41.813824+0000","last_unstale":"2024-03-18T22:42:44.945729+0000","last_undegraded":"2024-03-18T22:42:44.945729+0000","last_fullsized":"2024-03-18T22:42:44.945729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:12.412689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947414+0000","last_change":"2024-03-18T22:40:45.852711+0000","last_active":"2024-03-18T22:42:44.947414+0000","last_peered":"2024-03-18T22:42:44.947414+0000","last_clean":"2024-03-18T22:42:44.947414+0000","last_became_active":"2024-03-18T22:40:45.852247+0000","last_became_peered":"2024-03-18T22:40:45.852247+0000","last_unstale":"2024-03-18T22:42:44.947414+0000","last_undegraded":"2024-03-18T22:42:44.947414+0000","last_fullsized":"2024-03-18T22:42:44.947414+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:03.061689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947647+0000","last_change":"2024-03-18T22:40:41.212399+0000","last_active":"2024-03-18T22:42:44.947647+0000","last_peered":"2024-03-18T22:42:44.947647+0000","last_clean":"2024-03-18T22:42:44.947647+0000","last_became_active":"2024-03-18T22:40:41.207416+0000","last_became_peered":"2024-03-18T22:40:41.207416+0000","last_unstale":"2024-03-18T22:42:44.947647+0000","last_undegraded":"2024-03-18T22:42:44.947647+0000","last_fullsized":"2024-03-18T22:42:44.947647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:11.406342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950757+0000","last_change":"2024-03-18T22:40:41.807313+0000","last_active":"2024-03-18T22:42:44.950757+0000","last_peered":"2024-03-18T22:42:44.950757+0000","last_clean":"2024-03-18T22:42:44.950757+0000","last_became_active":"2024-03-18T22:40:41.806029+0000","last_became_peered":"2024-03-18T22:40:41.806029+0000","last_unstale":"2024-03-18T22:42:44.950757+0000","last_undegraded":"2024-03-18T22:42:44.950757+0000","last_fullsized":"2024-03-18T22:42:44.950757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:23:43.973443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955513+0000","last_change":"2024-03-18T22:40:45.835479+0000","last_active":"2024-03-18T22:42:44.955513+0000","last_peered":"2024-03-18T22:42:44.955513+0000","last_clean":"2024-03-18T22:42:44.955513+0000","last_became_active":"2024-03-18T22:40:45.834994+0000","last_became_peered":"2024-03-18T22:40:45.834994+0000","last_unstale":"2024-03-18T22:42:44.955513+0000","last_undegraded":"2024-03-18T22:42:44.955513+0000","last_fullsized":"2024-03-18T22:42:44.955513+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:31.317843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945288+0000","last_change":"2024-03-18T22:40:41.212110+0000","last_active":"2024-03-18T22:42:44.945288+0000","last_peered":"2024-03-18T22:42:44.945288+0000","last_clean":"2024-03-18T22:42:44.945288+0000","last_became_active":"2024-03-18T22:40:41.211803+0000","last_became_peered":"2024-03-18T22:40:41.211803+0000","last_unstale":"2024-03-18T22:42:44.945288+0000","last_undegraded":"2024-03-18T22:42:44.945288+0000","last_fullsized":"2024-03-18T22:42:44.945288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:40.482983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956612+0000","last_change":"2024-03-18T22:40:41.812585+0000","last_active":"2024-03-18T22:42:44.956612+0000","last_peered":"2024-03-18T22:42:44.956612+0000","last_clean":"2024-03-18T22:42:44.956612+0000","last_became_active":"2024-03-18T22:40:41.812260+0000","last_became_peered":"2024-03-18T22:40:41.812260+0000","last_unstale":"2024-03-18T22:42:44.956612+0000","last_undegraded":"2024-03-18T22:42:44.956612+0000","last_fullsized":"2024-03-18T22:42:44.956612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:12:59.999503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950731+0000","last_change":"2024-03-18T22:40:45.842252+0000","last_active":"2024-03-18T22:42:44.950731+0000","last_peered":"2024-03-18T22:42:44.950731+0000","last_clean":"2024-03-18T22:42:44.950731+0000","last_became_active":"2024-03-18T22:40:45.841849+0000","last_became_peered":"2024-03-18T22:40:45.841849+0000","last_unstale":"2024-03-18T22:42:44.950731+0000","last_undegraded":"2024-03-18T22:42:44.950731+0000","last_fullsized":"2024-03-18T22:42:44.950731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:21.253947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'7","reported_seq":270,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:03.974303+0000","last_change":"2024-03-18T22:40:41.214253+0000","last_active":"2024-03-18T22:43:03.974303+0000","last_peered":"2024-03-18T22:43:03.974303+0000","last_clean":"2024-03-18T22:43:03.974303+0000","last_became_active":"2024-03-18T22:40:41.213911+0000","last_became_peered":"2024-03-18T22:40:41.213911+0000","last_unstale":"2024-03-18T22:43:03.974303+0000","last_undegraded":"2024-03-18T22:43:03.974303+0000","last_fullsized":"2024-03-18T22:43:03.974303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:19.185714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":232,"num_read_kb":232,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.958079+0000","last_change":"2024-03-18T22:40:41.813281+0000","last_active":"2024-03-18T22:42:44.958079+0000","last_peered":"2024-03-18T22:42:44.958079+0000","last_clean":"2024-03-18T22:42:44.958079+0000","last_became_active":"2024-03-18T22:40:41.812909+0000","last_became_peered":"2024-03-18T22:40:41.812909+0000","last_unstale":"2024-03-18T22:42:44.958079+0000","last_undegraded":"2024-03-18T22:42:44.958079+0000","last_fullsized":"2024-03-18T22:42:44.958079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:17:26.619441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956423+0000","last_change":"2024-03-18T22:40:45.846956+0000","last_active":"2024-03-18T22:42:44.956423+0000","last_peered":"2024-03-18T22:42:44.956423+0000","last_clean":"2024-03-18T22:42:44.956423+0000","last_became_active":"2024-03-18T22:40:45.843061+0000","last_became_peered":"2024-03-18T22:40:45.843061+0000","last_unstale":"2024-03-18T22:42:44.956423+0000","last_undegraded":"2024-03-18T22:42:44.956423+0000","last_fullsized":"2024-03-18T22:42:44.956423+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:52.043424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947970+0000","last_change":"2024-03-18T22:40:41.214933+0000","last_active":"2024-03-18T22:42:44.947970+0000","last_peered":"2024-03-18T22:42:44.947970+0000","last_clean":"2024-03-18T22:42:44.947970+0000","last_became_active":"2024-03-18T22:40:41.214606+0000","last_became_peered":"2024-03-18T22:40:41.214606+0000","last_unstale":"2024-03-18T22:42:44.947970+0000","last_undegraded":"2024-03-18T22:42:44.947970+0000","last_fullsized":"2024-03-18T22:42:44.947970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:08:06.230288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955583+0000","last_change":"2024-03-18T22:40:41.817590+0000","last_active":"2024-03-18T22:42:44.955583+0000","last_peered":"2024-03-18T22:42:44.955583+0000","last_clean":"2024-03-18T22:42:44.955583+0000","last_became_active":"2024-03-18T22:40:41.815809+0000","last_became_peered":"2024-03-18T22:40:41.815809+0000","last_unstale":"2024-03-18T22:42:44.955583+0000","last_undegraded":"2024-03-18T22:42:44.955583+0000","last_fullsized":"2024-03-18T22:42:44.955583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:44:54.043647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945405+0000","last_change":"2024-03-18T22:40:45.844767+0000","last_active":"2024-03-18T22:42:44.945405+0000","last_peered":"2024-03-18T22:42:44.945405+0000","last_clean":"2024-03-18T22:42:44.945405+0000","last_became_active":"2024-03-18T22:40:45.844251+0000","last_became_peered":"2024-03-18T22:40:45.844251+0000","last_unstale":"2024-03-18T22:42:44.945405+0000","last_undegraded":"2024-03-18T22:42:44.945405+0000","last_fullsized":"2024-03-18T22:42:44.945405+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:43.756959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945512+0000","last_change":"2024-03-18T22:40:41.214310+0000","last_active":"2024-03-18T22:42:44.945512+0000","last_peered":"2024-03-18T22:42:44.945512+0000","last_clean":"2024-03-18T22:42:44.945512+0000","last_became_active":"2024-03-18T22:40:41.213963+0000","last_became_peered":"2024-03-18T22:40:41.213963+0000","last_unstale":"2024-03-18T22:42:44.945512+0000","last_undegraded":"2024-03-18T22:42:44.945512+0000","last_fullsized":"2024-03-18T22:42:44.945512+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:03.401220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957904+0000","last_change":"2024-03-18T22:40:41.810876+0000","last_active":"2024-03-18T22:42:44.957904+0000","last_peered":"2024-03-18T22:42:44.957904+0000","last_clean":"2024-03-18T22:42:44.957904+0000","last_became_active":"2024-03-18T22:40:41.810104+0000","last_became_peered":"2024-03-18T22:40:41.810104+0000","last_unstale":"2024-03-18T22:42:44.957904+0000","last_undegraded":"2024-03-18T22:42:44.957904+0000","last_fullsized":"2024-03-18T22:42:44.957904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:41.294286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955701+0000","last_change":"2024-03-18T22:41:44.266318+0000","last_active":"2024-03-18T22:42:44.955701+0000","last_peered":"2024-03-18T22:42:44.955701+0000","last_clean":"2024-03-18T22:42:44.955701+0000","last_became_active":"2024-03-18T22:41:44.265903+0000","last_became_peered":"2024-03-18T22:41:44.265903+0000","last_unstale":"2024-03-18T22:42:44.955701+0000","last_undegraded":"2024-03-18T22:42:44.955701+0000","last_fullsized":"2024-03-18T22:42:44.955701+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:44.307096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'40","reported_seq":317,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:03.972467+0000","last_change":"2024-03-18T22:40:41.209837+0000","last_active":"2024-03-18T22:43:03.972467+0000","last_peered":"2024-03-18T22:43:03.972467+0000","last_clean":"2024-03-18T22:43:03.972467+0000","last_became_active":"2024-03-18T22:40:41.209465+0000","last_became_peered":"2024-03-18T22:40:41.209465+0000","last_unstale":"2024-03-18T22:43:03.972467+0000","last_undegraded":"2024-03-18T22:43:03.972467+0000","last_fullsized":"2024-03-18T22:43:03.972467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:45:15.765475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64100,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":245,"num_read_kb":186,"num_write":42,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951123+0000","last_change":"2024-03-18T22:40:41.810058+0000","last_active":"2024-03-18T22:42:44.951123+0000","last_peered":"2024-03-18T22:42:44.951123+0000","last_clean":"2024-03-18T22:42:44.951123+0000","last_became_active":"2024-03-18T22:40:41.809677+0000","last_became_peered":"2024-03-18T22:40:41.809677+0000","last_unstale":"2024-03-18T22:42:44.951123+0000","last_undegraded":"2024-03-18T22:42:44.951123+0000","last_fullsized":"2024-03-18T22:42:44.951123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:03.935036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945802+0000","last_change":"2024-03-18T22:40:45.845638+0000","last_active":"2024-03-18T22:42:44.945802+0000","last_peered":"2024-03-18T22:42:44.945802+0000","last_clean":"2024-03-18T22:42:44.945802+0000","last_became_active":"2024-03-18T22:40:45.844994+0000","last_became_peered":"2024-03-18T22:40:45.844994+0000","last_unstale":"2024-03-18T22:42:44.945802+0000","last_undegraded":"2024-03-18T22:42:44.945802+0000","last_fullsized":"2024-03-18T22:42:44.945802+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:21.551271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950170+0000","last_change":"2024-03-18T22:40:41.218397+0000","last_active":"2024-03-18T22:42:44.950170+0000","last_peered":"2024-03-18T22:42:44.950170+0000","last_clean":"2024-03-18T22:42:44.950170+0000","last_became_active":"2024-03-18T22:40:41.217587+0000","last_became_peered":"2024-03-18T22:40:41.217587+0000","last_unstale":"2024-03-18T22:42:44.950170+0000","last_undegraded":"2024-03-18T22:42:44.950170+0000","last_fullsized":"2024-03-18T22:42:44.950170+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:36.524954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947698+0000","last_change":"2024-03-18T22:40:41.812026+0000","last_active":"2024-03-18T22:42:44.947698+0000","last_peered":"2024-03-18T22:42:44.947698+0000","last_clean":"2024-03-18T22:42:44.947698+0000","last_became_active":"2024-03-18T22:40:41.811668+0000","last_became_peered":"2024-03-18T22:40:41.811668+0000","last_unstale":"2024-03-18T22:42:44.947698+0000","last_undegraded":"2024-03-18T22:42:44.947698+0000","last_fullsized":"2024-03-18T22:42:44.947698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:17:54.775327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947145+0000","last_change":"2024-03-18T22:40:45.848466+0000","last_active":"2024-03-18T22:42:44.947145+0000","last_peered":"2024-03-18T22:42:44.947145+0000","last_clean":"2024-03-18T22:42:44.947145+0000","last_became_active":"2024-03-18T22:40:45.848078+0000","last_became_peered":"2024-03-18T22:40:45.848078+0000","last_unstale":"2024-03-18T22:42:44.947145+0000","last_undegraded":"2024-03-18T22:42:44.947145+0000","last_fullsized":"2024-03-18T22:42:44.947145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:28.515368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945811+0000","last_change":"2024-03-18T22:40:41.216434+0000","last_active":"2024-03-18T22:42:44.945811+0000","last_peered":"2024-03-18T22:42:44.945811+0000","last_clean":"2024-03-18T22:42:44.945811+0000","last_became_active":"2024-03-18T22:40:41.216103+0000","last_became_peered":"2024-03-18T22:40:41.216103+0000","last_unstale":"2024-03-18T22:42:44.945811+0000","last_undegraded":"2024-03-18T22:42:44.945811+0000","last_fullsized":"2024-03-18T22:42:44.945811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:39.516463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947732+0000","last_change":"2024-03-18T22:40:41.808405+0000","last_active":"2024-03-18T22:42:44.947732+0000","last_peered":"2024-03-18T22:42:44.947732+0000","last_clean":"2024-03-18T22:42:44.947732+0000","last_became_active":"2024-03-18T22:40:41.808111+0000","last_became_peered":"2024-03-18T22:40:41.808111+0000","last_unstale":"2024-03-18T22:42:44.947732+0000","last_undegraded":"2024-03-18T22:42:44.947732+0000","last_fullsized":"2024-03-18T22:42:44.947732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:37:20.808163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956599+0000","last_change":"2024-03-18T22:40:45.848409+0000","last_active":"2024-03-18T22:42:44.956599+0000","last_peered":"2024-03-18T22:42:44.956599+0000","last_clean":"2024-03-18T22:42:44.956599+0000","last_became_active":"2024-03-18T22:40:45.847687+0000","last_became_peered":"2024-03-18T22:40:45.847687+0000","last_unstale":"2024-03-18T22:42:44.956599+0000","last_undegraded":"2024-03-18T22:42:44.956599+0000","last_fullsized":"2024-03-18T22:42:44.956599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:32.894885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'8","reported_seq":160,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:01.950868+0000","last_change":"2024-03-18T22:40:41.208408+0000","last_active":"2024-03-18T22:43:01.950868+0000","last_peered":"2024-03-18T22:43:01.950868+0000","last_clean":"2024-03-18T22:43:01.950868+0000","last_became_active":"2024-03-18T22:40:41.207976+0000","last_became_peered":"2024-03-18T22:40:41.207976+0000","last_unstale":"2024-03-18T22:43:01.950868+0000","last_undegraded":"2024-03-18T22:43:01.950868+0000","last_fullsized":"2024-03-18T22:43:01.950868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:43:24.416974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1614,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947842+0000","last_change":"2024-03-18T22:40:41.813547+0000","last_active":"2024-03-18T22:42:44.947842+0000","last_peered":"2024-03-18T22:42:44.947842+0000","last_clean":"2024-03-18T22:42:44.947842+0000","last_became_active":"2024-03-18T22:40:41.813250+0000","last_became_peered":"2024-03-18T22:40:41.813250+0000","last_unstale":"2024-03-18T22:42:44.947842+0000","last_undegraded":"2024-03-18T22:42:44.947842+0000","last_fullsized":"2024-03-18T22:42:44.947842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:54.428485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948147+0000","last_change":"2024-03-18T22:40:45.845797+0000","last_active":"2024-03-18T22:42:44.948147+0000","last_peered":"2024-03-18T22:42:44.948147+0000","last_clean":"2024-03-18T22:42:44.948147+0000","last_became_active":"2024-03-18T22:40:45.845324+0000","last_became_peered":"2024-03-18T22:40:45.845324+0000","last_unstale":"2024-03-18T22:42:44.948147+0000","last_undegraded":"2024-03-18T22:42:44.948147+0000","last_fullsized":"2024-03-18T22:42:44.948147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:21.613203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.969073+0000","last_change":"2024-03-18T22:42:44.968575+0000","last_active":"2024-03-18T22:42:44.969073+0000","last_peered":"2024-03-18T22:42:44.969073+0000","last_clean":"2024-03-18T22:42:44.969073+0000","last_became_active":"2024-03-18T22:42:44.957518+0000","last_became_peered":"2024-03-18T22:42:44.957518+0000","last_unstale":"2024-03-18T22:42:44.969073+0000","last_undegraded":"2024-03-18T22:42:44.969073+0000","last_fullsized":"2024-03-18T22:42:44.969073+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:42:40.590291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945444+0000","last_change":"2024-03-18T22:40:41.814606+0000","last_active":"2024-03-18T22:42:44.945444+0000","last_peered":"2024-03-18T22:42:44.945444+0000","last_clean":"2024-03-18T22:42:44.945444+0000","last_became_active":"2024-03-18T22:40:41.814309+0000","last_became_peered":"2024-03-18T22:40:41.814309+0000","last_unstale":"2024-03-18T22:42:44.945444+0000","last_undegraded":"2024-03-18T22:42:44.945444+0000","last_fullsized":"2024-03-18T22:42:44.945444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:33.086229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945376+0000","last_change":"2024-03-18T22:40:45.845711+0000","last_active":"2024-03-18T22:42:44.945376+0000","last_peered":"2024-03-18T22:42:44.945376+0000","last_clean":"2024-03-18T22:42:44.945376+0000","last_became_active":"2024-03-18T22:40:45.845094+0000","last_became_peered":"2024-03-18T22:40:45.845094+0000","last_unstale":"2024-03-18T22:42:44.945376+0000","last_undegraded":"2024-03-18T22:42:44.945376+0000","last_fullsized":"2024-03-18T22:42:44.945376+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:08.372875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:56.474783+0000","last_change":"2024-03-18T22:40:41.219076+0000","last_active":"2024-03-18T22:42:56.474783+0000","last_peered":"2024-03-18T22:42:56.474783+0000","last_clean":"2024-03-18T22:42:56.474783+0000","last_became_active":"2024-03-18T22:40:41.217580+0000","last_became_peered":"2024-03-18T22:40:41.217580+0000","last_unstale":"2024-03-18T22:42:56.474783+0000","last_undegraded":"2024-03-18T22:42:56.474783+0000","last_fullsized":"2024-03-18T22:42:56.474783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:38.112325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947271+0000","last_change":"2024-03-18T22:40:41.810267+0000","last_active":"2024-03-18T22:42:44.947271+0000","last_peered":"2024-03-18T22:42:44.947271+0000","last_clean":"2024-03-18T22:42:44.947271+0000","last_became_active":"2024-03-18T22:40:41.809960+0000","last_became_peered":"2024-03-18T22:40:41.809960+0000","last_unstale":"2024-03-18T22:42:44.947271+0000","last_undegraded":"2024-03-18T22:42:44.947271+0000","last_fullsized":"2024-03-18T22:42:44.947271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:00.973817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950675+0000","last_change":"2024-03-18T22:40:45.848461+0000","last_active":"2024-03-18T22:42:44.950675+0000","last_peered":"2024-03-18T22:42:44.950675+0000","last_clean":"2024-03-18T22:42:44.950675+0000","last_became_active":"2024-03-18T22:40:45.848043+0000","last_became_peered":"2024-03-18T22:40:45.848043+0000","last_unstale":"2024-03-18T22:42:44.950675+0000","last_undegraded":"2024-03-18T22:42:44.950675+0000","last_fullsized":"2024-03-18T22:42:44.950675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:38.865565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957138+0000","last_change":"2024-03-18T22:40:41.213069+0000","last_active":"2024-03-18T22:42:44.957138+0000","last_peered":"2024-03-18T22:42:44.957138+0000","last_clean":"2024-03-18T22:42:44.957138+0000","last_became_active":"2024-03-18T22:40:41.212725+0000","last_became_peered":"2024-03-18T22:40:41.212725+0000","last_unstale":"2024-03-18T22:42:44.957138+0000","last_undegraded":"2024-03-18T22:42:44.957138+0000","last_fullsized":"2024-03-18T22:42:44.957138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:50.219433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956533+0000","last_change":"2024-03-18T22:41:44.004772+0000","last_active":"2024-03-18T22:42:44.956533+0000","last_peered":"2024-03-18T22:42:44.956533+0000","last_clean":"2024-03-18T22:42:44.956533+0000","last_became_active":"2024-03-18T22:41:44.004342+0000","last_became_peered":"2024-03-18T22:41:44.004342+0000","last_unstale":"2024-03-18T22:42:44.956533+0000","last_undegraded":"2024-03-18T22:42:44.956533+0000","last_fullsized":"2024-03-18T22:42:44.956533+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:18:38.674911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947926+0000","last_change":"2024-03-18T22:40:45.840058+0000","last_active":"2024-03-18T22:42:44.947926+0000","last_peered":"2024-03-18T22:42:44.947926+0000","last_clean":"2024-03-18T22:42:44.947926+0000","last_became_active":"2024-03-18T22:40:45.839661+0000","last_became_peered":"2024-03-18T22:40:45.839661+0000","last_unstale":"2024-03-18T22:42:44.947926+0000","last_undegraded":"2024-03-18T22:42:44.947926+0000","last_fullsized":"2024-03-18T22:42:44.947926+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:35:14.323972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'8","reported_seq":156,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:01.983783+0000","last_change":"2024-03-18T22:40:41.215988+0000","last_active":"2024-03-18T22:43:01.983783+0000","last_peered":"2024-03-18T22:43:01.983783+0000","last_clean":"2024-03-18T22:43:01.983783+0000","last_became_active":"2024-03-18T22:40:41.215680+0000","last_became_peered":"2024-03-18T22:40:41.215680+0000","last_unstale":"2024-03-18T22:43:01.983783+0000","last_undegraded":"2024-03-18T22:43:01.983783+0000","last_fullsized":"2024-03-18T22:43:01.983783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:43.412362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1614,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956119+0000","last_change":"2024-03-18T22:40:41.812521+0000","last_active":"2024-03-18T22:42:44.956119+0000","last_peered":"2024-03-18T22:42:44.956119+0000","last_clean":"2024-03-18T22:42:44.956119+0000","last_became_active":"2024-03-18T22:40:41.812223+0000","last_became_peered":"2024-03-18T22:40:41.812223+0000","last_unstale":"2024-03-18T22:42:44.956119+0000","last_undegraded":"2024-03-18T22:42:44.956119+0000","last_fullsized":"2024-03-18T22:42:44.956119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:35:37.945144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948016+0000","last_change":"2024-03-18T22:40:45.838372+0000","last_active":"2024-03-18T22:42:44.948016+0000","last_peered":"2024-03-18T22:42:44.948016+0000","last_clean":"2024-03-18T22:42:44.948016+0000","last_became_active":"2024-03-18T22:40:45.837962+0000","last_became_peered":"2024-03-18T22:40:45.837962+0000","last_unstale":"2024-03-18T22:42:44.948016+0000","last_undegraded":"2024-03-18T22:42:44.948016+0000","last_fullsized":"2024-03-18T22:42:44.948016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:18:39.323023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945400+0000","last_change":"2024-03-18T22:40:41.812146+0000","last_active":"2024-03-18T22:42:44.945400+0000","last_peered":"2024-03-18T22:42:44.945400+0000","last_clean":"2024-03-18T22:42:44.945400+0000","last_became_active":"2024-03-18T22:40:41.811840+0000","last_became_peered":"2024-03-18T22:40:41.811840+0000","last_unstale":"2024-03-18T22:42:44.945400+0000","last_undegraded":"2024-03-18T22:42:44.945400+0000","last_fullsized":"2024-03-18T22:42:44.945400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:30.829187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'7","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:04.953826+0000","last_change":"2024-03-18T22:40:41.212133+0000","last_active":"2024-03-18T22:43:04.953826+0000","last_peered":"2024-03-18T22:43:04.953826+0000","last_clean":"2024-03-18T22:43:04.953826+0000","last_became_active":"2024-03-18T22:40:41.211581+0000","last_became_peered":"2024-03-18T22:40:41.211581+0000","last_unstale":"2024-03-18T22:43:04.953826+0000","last_undegraded":"2024-03-18T22:43:04.953826+0000","last_fullsized":"2024-03-18T22:43:04.953826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:44.287725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947800+0000","last_change":"2024-03-18T22:40:45.843660+0000","last_active":"2024-03-18T22:42:44.947800+0000","last_peered":"2024-03-18T22:42:44.947800+0000","last_clean":"2024-03-18T22:42:44.947800+0000","last_became_active":"2024-03-18T22:40:45.843233+0000","last_became_peered":"2024-03-18T22:40:45.843233+0000","last_unstale":"2024-03-18T22:42:44.947800+0000","last_undegraded":"2024-03-18T22:42:44.947800+0000","last_fullsized":"2024-03-18T22:42:44.947800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:08:19.758496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950814+0000","last_change":"2024-03-18T22:40:41.811798+0000","last_active":"2024-03-18T22:42:44.950814+0000","last_peered":"2024-03-18T22:42:44.950814+0000","last_clean":"2024-03-18T22:42:44.950814+0000","last_became_active":"2024-03-18T22:40:41.811422+0000","last_became_peered":"2024-03-18T22:40:41.811422+0000","last_unstale":"2024-03-18T22:42:44.950814+0000","last_undegraded":"2024-03-18T22:42:44.950814+0000","last_fullsized":"2024-03-18T22:42:44.950814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:53.944556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945349+0000","last_change":"2024-03-18T22:40:41.210052+0000","last_active":"2024-03-18T22:42:44.945349+0000","last_peered":"2024-03-18T22:42:44.945349+0000","last_clean":"2024-03-18T22:42:44.945349+0000","last_became_active":"2024-03-18T22:40:41.209647+0000","last_became_peered":"2024-03-18T22:40:41.209647+0000","last_unstale":"2024-03-18T22:42:44.945349+0000","last_undegraded":"2024-03-18T22:42:44.945349+0000","last_fullsized":"2024-03-18T22:42:44.945349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:57:35.187348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947306+0000","last_change":"2024-03-18T22:39:32.831758+0000","last_active":"2024-03-18T22:42:44.947306+0000","last_peered":"2024-03-18T22:42:44.947306+0000","last_clean":"2024-03-18T22:42:44.947306+0000","last_became_active":"2024-03-18T22:39:32.831287+0000","last_became_peered":"2024-03-18T22:39:32.831287+0000","last_unstale":"2024-03-18T22:42:44.947306+0000","last_undegraded":"2024-03-18T22:42:44.947306+0000","last_fullsized":"2024-03-18T22:42:44.947306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:36:07.806854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:36:07.806854+0000","last_clean_scrub_stamp":"2024-03-18T22:36:07.806854+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:42.449107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945891+0000","last_change":"2024-03-18T22:40:45.842008+0000","last_active":"2024-03-18T22:42:44.945891+0000","last_peered":"2024-03-18T22:42:44.945891+0000","last_clean":"2024-03-18T22:42:44.945891+0000","last_became_active":"2024-03-18T22:40:45.841231+0000","last_became_peered":"2024-03-18T22:40:45.841231+0000","last_unstale":"2024-03-18T22:42:44.945891+0000","last_undegraded":"2024-03-18T22:42:44.945891+0000","last_fullsized":"2024-03-18T22:42:44.945891+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:06.812049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947613+0000","last_change":"2024-03-18T22:40:41.807216+0000","last_active":"2024-03-18T22:42:44.947613+0000","last_peered":"2024-03-18T22:42:44.947613+0000","last_clean":"2024-03-18T22:42:44.947613+0000","last_became_active":"2024-03-18T22:40:41.806797+0000","last_became_peered":"2024-03-18T22:40:41.806797+0000","last_unstale":"2024-03-18T22:42:44.947613+0000","last_undegraded":"2024-03-18T22:42:44.947613+0000","last_fullsized":"2024-03-18T22:42:44.947613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:26.422329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955531+0000","last_change":"2024-03-18T22:42:44.955531+0000","last_active":"2024-03-18T22:42:44.955531+0000","last_peered":"2024-03-18T22:42:44.955531+0000","last_clean":"2024-03-18T22:42:44.955531+0000","last_became_active":"2024-03-18T22:42:44.949859+0000","last_became_peered":"2024-03-18T22:42:44.949859+0000","last_unstale":"2024-03-18T22:42:44.955531+0000","last_undegraded":"2024-03-18T22:42:44.955531+0000","last_fullsized":"2024-03-18T22:42:44.955531+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:50.522069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947630+0000","last_change":"2024-03-18T22:40:45.839889+0000","last_active":"2024-03-18T22:42:44.947630+0000","last_peered":"2024-03-18T22:42:44.947630+0000","last_clean":"2024-03-18T22:42:44.947630+0000","last_became_active":"2024-03-18T22:40:45.839414+0000","last_became_peered":"2024-03-18T22:40:45.839414+0000","last_unstale":"2024-03-18T22:42:44.947630+0000","last_undegraded":"2024-03-18T22:42:44.947630+0000","last_fullsized":"2024-03-18T22:42:44.947630+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:52.497378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951108+0000","last_change":"2024-03-18T22:40:41.221026+0000","last_active":"2024-03-18T22:42:44.951108+0000","last_peered":"2024-03-18T22:42:44.951108+0000","last_clean":"2024-03-18T22:42:44.951108+0000","last_became_active":"2024-03-18T22:40:41.220681+0000","last_became_peered":"2024-03-18T22:40:41.220681+0000","last_unstale":"2024-03-18T22:42:44.951108+0000","last_undegraded":"2024-03-18T22:42:44.951108+0000","last_fullsized":"2024-03-18T22:42:44.951108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:11.490445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948129+0000","last_change":"2024-03-18T22:40:41.810500+0000","last_active":"2024-03-18T22:42:44.948129+0000","last_peered":"2024-03-18T22:42:44.948129+0000","last_clean":"2024-03-18T22:42:44.948129+0000","last_became_active":"2024-03-18T22:40:41.809723+0000","last_became_peered":"2024-03-18T22:40:41.809723+0000","last_unstale":"2024-03-18T22:42:44.948129+0000","last_undegraded":"2024-03-18T22:42:44.948129+0000","last_fullsized":"2024-03-18T22:42:44.948129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:18:37.053809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950609+0000","last_change":"2024-03-18T22:40:45.844323+0000","last_active":"2024-03-18T22:42:44.950609+0000","last_peered":"2024-03-18T22:42:44.950609+0000","last_clean":"2024-03-18T22:42:44.950609+0000","last_became_active":"2024-03-18T22:40:45.843146+0000","last_became_peered":"2024-03-18T22:40:45.843146+0000","last_unstale":"2024-03-18T22:42:44.950609+0000","last_undegraded":"2024-03-18T22:42:44.950609+0000","last_fullsized":"2024-03-18T22:42:44.950609+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:14:59.881467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'14","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:53.026843+0000","last_change":"2024-03-18T22:40:41.212126+0000","last_active":"2024-03-18T22:42:53.026843+0000","last_peered":"2024-03-18T22:42:53.026843+0000","last_clean":"2024-03-18T22:42:53.026843+0000","last_became_active":"2024-03-18T22:40:41.211772+0000","last_became_peered":"2024-03-18T22:40:41.211772+0000","last_unstale":"2024-03-18T22:42:53.026843+0000","last_undegraded":"2024-03-18T22:42:53.026843+0000","last_fullsized":"2024-03-18T22:42:53.026843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:03.215751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951248+0000","last_change":"2024-03-18T22:40:41.820476+0000","last_active":"2024-03-18T22:42:44.951248+0000","last_peered":"2024-03-18T22:42:44.951248+0000","last_clean":"2024-03-18T22:42:44.951248+0000","last_became_active":"2024-03-18T22:40:41.820084+0000","last_became_peered":"2024-03-18T22:40:41.820084+0000","last_unstale":"2024-03-18T22:42:44.951248+0000","last_undegraded":"2024-03-18T22:42:44.951248+0000","last_fullsized":"2024-03-18T22:42:44.951248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:51:50.580208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949154+0000","last_change":"2024-03-18T22:40:45.838588+0000","last_active":"2024-03-18T22:42:44.949154+0000","last_peered":"2024-03-18T22:42:44.949154+0000","last_clean":"2024-03-18T22:42:44.949154+0000","last_became_active":"2024-03-18T22:40:45.838068+0000","last_became_peered":"2024-03-18T22:40:45.838068+0000","last_unstale":"2024-03-18T22:42:44.949154+0000","last_undegraded":"2024-03-18T22:42:44.949154+0000","last_fullsized":"2024-03-18T22:42:44.949154+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:11:02.162668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945922+0000","last_change":"2024-03-18T22:40:41.210108+0000","last_active":"2024-03-18T22:42:44.945922+0000","last_peered":"2024-03-18T22:42:44.945922+0000","last_clean":"2024-03-18T22:42:44.945922+0000","last_became_active":"2024-03-18T22:40:41.209781+0000","last_became_peered":"2024-03-18T22:40:41.209781+0000","last_unstale":"2024-03-18T22:42:44.945922+0000","last_undegraded":"2024-03-18T22:42:44.945922+0000","last_fullsized":"2024-03-18T22:42:44.945922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:10.670822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955691+0000","last_change":"2024-03-18T22:40:41.814637+0000","last_active":"2024-03-18T22:42:44.955691+0000","last_peered":"2024-03-18T22:42:44.955691+0000","last_clean":"2024-03-18T22:42:44.955691+0000","last_became_active":"2024-03-18T22:40:41.814279+0000","last_became_peered":"2024-03-18T22:40:41.814279+0000","last_unstale":"2024-03-18T22:42:44.955691+0000","last_undegraded":"2024-03-18T22:42:44.955691+0000","last_fullsized":"2024-03-18T22:42:44.955691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:12.496230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945752+0000","last_change":"2024-03-18T22:40:45.851560+0000","last_active":"2024-03-18T22:42:44.945752+0000","last_peered":"2024-03-18T22:42:44.945752+0000","last_clean":"2024-03-18T22:42:44.945752+0000","last_became_active":"2024-03-18T22:40:45.850884+0000","last_became_peered":"2024-03-18T22:40:45.850884+0000","last_unstale":"2024-03-18T22:42:44.945752+0000","last_undegraded":"2024-03-18T22:42:44.945752+0000","last_fullsized":"2024-03-18T22:42:44.945752+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:04.114468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957432+0000","last_change":"2024-03-18T22:40:41.210156+0000","last_active":"2024-03-18T22:42:44.957432+0000","last_peered":"2024-03-18T22:42:44.957432+0000","last_clean":"2024-03-18T22:42:44.957432+0000","last_became_active":"2024-03-18T22:40:41.209855+0000","last_became_peered":"2024-03-18T22:40:41.209855+0000","last_unstale":"2024-03-18T22:42:44.957432+0000","last_undegraded":"2024-03-18T22:42:44.957432+0000","last_fullsized":"2024-03-18T22:42:44.957432+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:02.752573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949647+0000","last_change":"2024-03-18T22:40:41.803681+0000","last_active":"2024-03-18T22:42:44.949647+0000","last_peered":"2024-03-18T22:42:44.949647+0000","last_clean":"2024-03-18T22:42:44.949647+0000","last_became_active":"2024-03-18T22:40:41.803244+0000","last_became_peered":"2024-03-18T22:40:41.803244+0000","last_unstale":"2024-03-18T22:42:44.949647+0000","last_undegraded":"2024-03-18T22:42:44.949647+0000","last_fullsized":"2024-03-18T22:42:44.949647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:55.944575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951408+0000","last_change":"2024-03-18T22:41:44.005707+0000","last_active":"2024-03-18T22:42:44.951408+0000","last_peered":"2024-03-18T22:42:44.951408+0000","last_clean":"2024-03-18T22:42:44.951408+0000","last_became_active":"2024-03-18T22:41:44.004338+0000","last_became_peered":"2024-03-18T22:41:44.004338+0000","last_unstale":"2024-03-18T22:42:44.951408+0000","last_undegraded":"2024-03-18T22:42:44.951408+0000","last_fullsized":"2024-03-18T22:42:44.951408+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:46.054087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951208+0000","last_change":"2024-03-18T22:40:41.210830+0000","last_active":"2024-03-18T22:42:44.951208+0000","last_peered":"2024-03-18T22:42:44.951208+0000","last_clean":"2024-03-18T22:42:44.951208+0000","last_became_active":"2024-03-18T22:40:41.210476+0000","last_became_peered":"2024-03-18T22:40:41.210476+0000","last_unstale":"2024-03-18T22:42:44.951208+0000","last_undegraded":"2024-03-18T22:42:44.951208+0000","last_fullsized":"2024-03-18T22:42:44.951208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:13.243788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947025+0000","last_change":"2024-03-18T22:40:41.807678+0000","last_active":"2024-03-18T22:42:44.947025+0000","last_peered":"2024-03-18T22:42:44.947025+0000","last_clean":"2024-03-18T22:42:44.947025+0000","last_became_active":"2024-03-18T22:40:41.807383+0000","last_became_peered":"2024-03-18T22:40:41.807383+0000","last_unstale":"2024-03-18T22:42:44.947025+0000","last_undegraded":"2024-03-18T22:42:44.947025+0000","last_fullsized":"2024-03-18T22:42:44.947025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:27:55.930293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945556+0000","last_change":"2024-03-18T22:40:45.850982+0000","last_active":"2024-03-18T22:42:44.945556+0000","last_peered":"2024-03-18T22:42:44.945556+0000","last_clean":"2024-03-18T22:42:44.945556+0000","last_became_active":"2024-03-18T22:40:45.850526+0000","last_became_peered":"2024-03-18T22:40:45.850526+0000","last_unstale":"2024-03-18T22:42:44.945556+0000","last_undegraded":"2024-03-18T22:42:44.945556+0000","last_fullsized":"2024-03-18T22:42:44.945556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:05:24.450986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949309+0000","last_change":"2024-03-18T22:40:41.219540+0000","last_active":"2024-03-18T22:42:44.949309+0000","last_peered":"2024-03-18T22:42:44.949309+0000","last_clean":"2024-03-18T22:42:44.949309+0000","last_became_active":"2024-03-18T22:40:41.219206+0000","last_became_peered":"2024-03-18T22:40:41.219206+0000","last_unstale":"2024-03-18T22:42:44.949309+0000","last_undegraded":"2024-03-18T22:42:44.949309+0000","last_fullsized":"2024-03-18T22:42:44.949309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:33.586368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951047+0000","last_change":"2024-03-18T22:40:41.815561+0000","last_active":"2024-03-18T22:42:44.951047+0000","last_peered":"2024-03-18T22:42:44.951047+0000","last_clean":"2024-03-18T22:42:44.951047+0000","last_became_active":"2024-03-18T22:40:41.815082+0000","last_became_peered":"2024-03-18T22:40:41.815082+0000","last_unstale":"2024-03-18T22:42:44.951047+0000","last_undegraded":"2024-03-18T22:42:44.951047+0000","last_fullsized":"2024-03-18T22:42:44.951047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:01.676907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945700+0000","last_change":"2024-03-18T22:40:45.845386+0000","last_active":"2024-03-18T22:42:44.945700+0000","last_peered":"2024-03-18T22:42:44.945700+0000","last_clean":"2024-03-18T22:42:44.945700+0000","last_became_active":"2024-03-18T22:40:45.845007+0000","last_became_peered":"2024-03-18T22:40:45.845007+0000","last_unstale":"2024-03-18T22:42:44.945700+0000","last_undegraded":"2024-03-18T22:42:44.945700+0000","last_fullsized":"2024-03-18T22:42:44.945700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:41.429195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.960602+0000","last_change":"2024-03-18T22:42:44.960602+0000","last_active":"2024-03-18T22:42:44.960602+0000","last_peered":"2024-03-18T22:42:44.960602+0000","last_clean":"2024-03-18T22:42:44.960602+0000","last_became_active":"2024-03-18T22:42:44.951902+0000","last_became_peered":"2024-03-18T22:42:44.951902+0000","last_unstale":"2024-03-18T22:42:44.960602+0000","last_undegraded":"2024-03-18T22:42:44.960602+0000","last_fullsized":"2024-03-18T22:42:44.960602+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:07.038253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,7],"acting":[11,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947336+0000","last_change":"2024-03-18T22:40:41.807533+0000","last_active":"2024-03-18T22:42:44.947336+0000","last_peered":"2024-03-18T22:42:44.947336+0000","last_clean":"2024-03-18T22:42:44.947336+0000","last_became_active":"2024-03-18T22:40:41.807219+0000","last_became_peered":"2024-03-18T22:40:41.807219+0000","last_unstale":"2024-03-18T22:42:44.947336+0000","last_undegraded":"2024-03-18T22:42:44.947336+0000","last_fullsized":"2024-03-18T22:42:44.947336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:15:24.214367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957534+0000","last_change":"2024-03-18T22:40:45.839670+0000","last_active":"2024-03-18T22:42:44.957534+0000","last_peered":"2024-03-18T22:42:44.957534+0000","last_clean":"2024-03-18T22:42:44.957534+0000","last_became_active":"2024-03-18T22:40:45.839263+0000","last_became_peered":"2024-03-18T22:40:45.839263+0000","last_unstale":"2024-03-18T22:42:44.957534+0000","last_undegraded":"2024-03-18T22:42:44.957534+0000","last_fullsized":"2024-03-18T22:42:44.957534+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:48.991959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'6","reported_seq":152,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:00.986309+0000","last_change":"2024-03-18T22:40:41.209617+0000","last_active":"2024-03-18T22:43:00.986309+0000","last_peered":"2024-03-18T22:43:00.986309+0000","last_clean":"2024-03-18T22:43:00.986309+0000","last_became_active":"2024-03-18T22:40:41.209311+0000","last_became_peered":"2024-03-18T22:40:41.209311+0000","last_unstale":"2024-03-18T22:43:00.986309+0000","last_undegraded":"2024-03-18T22:43:00.986309+0000","last_fullsized":"2024-03-18T22:43:00.986309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:44:06.019870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948110+0000","last_change":"2024-03-18T22:40:41.808210+0000","last_active":"2024-03-18T22:42:44.948110+0000","last_peered":"2024-03-18T22:42:44.948110+0000","last_clean":"2024-03-18T22:42:44.948110+0000","last_became_active":"2024-03-18T22:40:41.807898+0000","last_became_peered":"2024-03-18T22:40:41.807898+0000","last_unstale":"2024-03-18T22:42:44.948110+0000","last_undegraded":"2024-03-18T22:42:44.948110+0000","last_fullsized":"2024-03-18T22:42:44.948110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:20.401165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.954699+0000","last_change":"2024-03-18T22:40:45.841905+0000","last_active":"2024-03-18T22:42:44.954699+0000","last_peered":"2024-03-18T22:42:44.954699+0000","last_clean":"2024-03-18T22:42:44.954699+0000","last_became_active":"2024-03-18T22:40:45.841464+0000","last_became_peered":"2024-03-18T22:40:45.841464+0000","last_unstale":"2024-03-18T22:42:44.954699+0000","last_undegraded":"2024-03-18T22:42:44.954699+0000","last_fullsized":"2024-03-18T22:42:44.954699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:30.435788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945748+0000","last_change":"2024-03-18T22:40:41.216919+0000","last_active":"2024-03-18T22:42:44.945748+0000","last_peered":"2024-03-18T22:42:44.945748+0000","last_clean":"2024-03-18T22:42:44.945748+0000","last_became_active":"2024-03-18T22:40:41.216608+0000","last_became_peered":"2024-03-18T22:40:41.216608+0000","last_unstale":"2024-03-18T22:42:44.945748+0000","last_undegraded":"2024-03-18T22:42:44.945748+0000","last_fullsized":"2024-03-18T22:42:44.945748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:10:42.230268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947883+0000","last_change":"2024-03-18T22:40:41.813879+0000","last_active":"2024-03-18T22:42:44.947883+0000","last_peered":"2024-03-18T22:42:44.947883+0000","last_clean":"2024-03-18T22:42:44.947883+0000","last_became_active":"2024-03-18T22:40:41.813584+0000","last_became_peered":"2024-03-18T22:40:41.813584+0000","last_unstale":"2024-03-18T22:42:44.947883+0000","last_undegraded":"2024-03-18T22:42:44.947883+0000","last_fullsized":"2024-03-18T22:42:44.947883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:20:44.808087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945203+0000","last_change":"2024-03-18T22:40:45.838033+0000","last_active":"2024-03-18T22:42:44.945203+0000","last_peered":"2024-03-18T22:42:44.945203+0000","last_clean":"2024-03-18T22:42:44.945203+0000","last_became_active":"2024-03-18T22:40:45.837530+0000","last_became_peered":"2024-03-18T22:40:45.837530+0000","last_unstale":"2024-03-18T22:42:44.945203+0000","last_undegraded":"2024-03-18T22:42:44.945203+0000","last_fullsized":"2024-03-18T22:42:44.945203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:43:43.444631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.961111+0000","last_change":"2024-03-18T22:42:44.960540+0000","last_active":"2024-03-18T22:42:44.961111+0000","last_peered":"2024-03-18T22:42:44.961111+0000","last_clean":"2024-03-18T22:42:44.961111+0000","last_became_active":"2024-03-18T22:42:44.952010+0000","last_became_peered":"2024-03-18T22:42:44.952010+0000","last_unstale":"2024-03-18T22:42:44.961111+0000","last_undegraded":"2024-03-18T22:42:44.961111+0000","last_fullsized":"2024-03-18T22:42:44.961111+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:46:48.257820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948217+0000","last_change":"2024-03-18T22:40:41.808959+0000","last_active":"2024-03-18T22:42:44.948217+0000","last_peered":"2024-03-18T22:42:44.948217+0000","last_clean":"2024-03-18T22:42:44.948217+0000","last_became_active":"2024-03-18T22:40:41.808671+0000","last_became_peered":"2024-03-18T22:40:41.808671+0000","last_unstale":"2024-03-18T22:42:44.948217+0000","last_undegraded":"2024-03-18T22:42:44.948217+0000","last_fullsized":"2024-03-18T22:42:44.948217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:37.115318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947087+0000","last_change":"2024-03-18T22:40:41.814240+0000","last_active":"2024-03-18T22:42:44.947087+0000","last_peered":"2024-03-18T22:42:44.947087+0000","last_clean":"2024-03-18T22:42:44.947087+0000","last_became_active":"2024-03-18T22:40:41.813925+0000","last_became_peered":"2024-03-18T22:40:41.813925+0000","last_unstale":"2024-03-18T22:42:44.947087+0000","last_undegraded":"2024-03-18T22:42:44.947087+0000","last_fullsized":"2024-03-18T22:42:44.947087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:38.999803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949347+0000","last_change":"2024-03-18T22:40:41.813381+0000","last_active":"2024-03-18T22:42:44.949347+0000","last_peered":"2024-03-18T22:42:44.949347+0000","last_clean":"2024-03-18T22:42:44.949347+0000","last_became_active":"2024-03-18T22:40:41.812927+0000","last_became_peered":"2024-03-18T22:40:41.812927+0000","last_unstale":"2024-03-18T22:42:44.949347+0000","last_undegraded":"2024-03-18T22:42:44.949347+0000","last_fullsized":"2024-03-18T22:42:44.949347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:23:09.178485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949543+0000","last_change":"2024-03-18T22:40:41.812513+0000","last_active":"2024-03-18T22:42:44.949543+0000","last_peered":"2024-03-18T22:42:44.949543+0000","last_clean":"2024-03-18T22:42:44.949543+0000","last_became_active":"2024-03-18T22:40:41.811882+0000","last_became_peered":"2024-03-18T22:40:41.811882+0000","last_unstale":"2024-03-18T22:42:44.949543+0000","last_undegraded":"2024-03-18T22:42:44.949543+0000","last_fullsized":"2024-03-18T22:42:44.949543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:45.596911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955001+0000","last_change":"2024-03-18T22:40:41.814960+0000","last_active":"2024-03-18T22:42:44.955001+0000","last_peered":"2024-03-18T22:42:44.955001+0000","last_clean":"2024-03-18T22:42:44.955001+0000","last_became_active":"2024-03-18T22:40:41.814620+0000","last_became_peered":"2024-03-18T22:40:41.814620+0000","last_unstale":"2024-03-18T22:42:44.955001+0000","last_undegraded":"2024-03-18T22:42:44.955001+0000","last_fullsized":"2024-03-18T22:42:44.955001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:42.205885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948057+0000","last_change":"2024-03-18T22:41:44.334913+0000","last_active":"2024-03-18T22:42:44.948057+0000","last_peered":"2024-03-18T22:42:44.948057+0000","last_clean":"2024-03-18T22:42:44.948057+0000","last_became_active":"2024-03-18T22:41:44.334554+0000","last_became_peered":"2024-03-18T22:41:44.334554+0000","last_unstale":"2024-03-18T22:42:44.948057+0000","last_undegraded":"2024-03-18T22:42:44.948057+0000","last_fullsized":"2024-03-18T22:42:44.948057+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:00:17.369015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947401+0000","last_change":"2024-03-18T22:40:41.808718+0000","last_active":"2024-03-18T22:42:44.947401+0000","last_peered":"2024-03-18T22:42:44.947401+0000","last_clean":"2024-03-18T22:42:44.947401+0000","last_became_active":"2024-03-18T22:40:41.808379+0000","last_became_peered":"2024-03-18T22:40:41.808379+0000","last_unstale":"2024-03-18T22:42:44.947401+0000","last_undegraded":"2024-03-18T22:42:44.947401+0000","last_fullsized":"2024-03-18T22:42:44.947401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:26.637531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950358+0000","last_change":"2024-03-18T22:40:41.818046+0000","last_active":"2024-03-18T22:42:44.950358+0000","last_peered":"2024-03-18T22:42:44.950358+0000","last_clean":"2024-03-18T22:42:44.950358+0000","last_became_active":"2024-03-18T22:40:41.817698+0000","last_became_peered":"2024-03-18T22:40:41.817698+0000","last_unstale":"2024-03-18T22:42:44.950358+0000","last_undegraded":"2024-03-18T22:42:44.950358+0000","last_fullsized":"2024-03-18T22:42:44.950358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:22.967673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951447+0000","last_change":"2024-03-18T22:40:41.819337+0000","last_active":"2024-03-18T22:42:44.951447+0000","last_peered":"2024-03-18T22:42:44.951447+0000","last_clean":"2024-03-18T22:42:44.951447+0000","last_became_active":"2024-03-18T22:40:41.818987+0000","last_became_peered":"2024-03-18T22:40:41.818987+0000","last_unstale":"2024-03-18T22:42:44.951447+0000","last_undegraded":"2024-03-18T22:42:44.951447+0000","last_fullsized":"2024-03-18T22:42:44.951447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:07:04.529454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945317+0000","last_change":"2024-03-18T22:40:41.813262+0000","last_active":"2024-03-18T22:42:44.945317+0000","last_peered":"2024-03-18T22:42:44.945317+0000","last_clean":"2024-03-18T22:42:44.945317+0000","last_became_active":"2024-03-18T22:40:41.812956+0000","last_became_peered":"2024-03-18T22:40:41.812956+0000","last_unstale":"2024-03-18T22:42:44.945317+0000","last_undegraded":"2024-03-18T22:42:44.945317+0000","last_fullsized":"2024-03-18T22:42:44.945317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:19.940097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951429+0000","last_change":"2024-03-18T22:40:41.811897+0000","last_active":"2024-03-18T22:42:44.951429+0000","last_peered":"2024-03-18T22:42:44.951429+0000","last_clean":"2024-03-18T22:42:44.951429+0000","last_became_active":"2024-03-18T22:40:41.811575+0000","last_became_peered":"2024-03-18T22:40:41.811575+0000","last_unstale":"2024-03-18T22:42:44.951429+0000","last_undegraded":"2024-03-18T22:42:44.951429+0000","last_fullsized":"2024-03-18T22:42:44.951429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:45:43.321057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957689+0000","last_change":"2024-03-18T22:40:41.811757+0000","last_active":"2024-03-18T22:42:44.957689+0000","last_peered":"2024-03-18T22:42:44.957689+0000","last_clean":"2024-03-18T22:42:44.957689+0000","last_became_active":"2024-03-18T22:40:41.811405+0000","last_became_peered":"2024-03-18T22:40:41.811405+0000","last_unstale":"2024-03-18T22:42:44.957689+0000","last_undegraded":"2024-03-18T22:42:44.957689+0000","last_fullsized":"2024-03-18T22:42:44.957689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:39.224604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947749+0000","last_change":"2024-03-18T22:40:41.809987+0000","last_active":"2024-03-18T22:42:44.947749+0000","last_peered":"2024-03-18T22:42:44.947749+0000","last_clean":"2024-03-18T22:42:44.947749+0000","last_became_active":"2024-03-18T22:40:41.809544+0000","last_became_peered":"2024-03-18T22:40:41.809544+0000","last_unstale":"2024-03-18T22:42:44.947749+0000","last_undegraded":"2024-03-18T22:42:44.947749+0000","last_fullsized":"2024-03-18T22:42:44.947749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:23.800949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947201+0000","last_change":"2024-03-18T22:40:41.810963+0000","last_active":"2024-03-18T22:42:44.947201+0000","last_peered":"2024-03-18T22:42:44.947201+0000","last_clean":"2024-03-18T22:42:44.947201+0000","last_became_active":"2024-03-18T22:40:41.810641+0000","last_became_peered":"2024-03-18T22:40:41.810641+0000","last_unstale":"2024-03-18T22:42:44.947201+0000","last_undegraded":"2024-03-18T22:42:44.947201+0000","last_fullsized":"2024-03-18T22:42:44.947201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:37:48.872961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74094,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1151,"num_read_kb":686,"num_write":268,"num_write_kb":163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":222282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":234,"ondisk_log_size":234,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383721,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":522642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37056,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708096,"statfs":{"total":95995035648,"available":95957090304,"internally_reserved":0,"allocated":1761280,"data_stored":1172442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808883,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":524526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":520808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28660,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716492,"statfs":{"total":95995035648,"available":95965687808,"internally_reserved":0,"allocated":1683456,"data_stored":1119992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":516960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36452,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708700,"statfs":{"total":95995035648,"available":95957708800,"internally_reserved":0,"allocated":1142784,"data_stored":582202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":517884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1146880,"data_stored":582692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":515642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1073152,"data_stored":521034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28660,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716492,"statfs":{"total":95995035648,"available":95965687808,"internally_reserved":0,"allocated":1683456,"data_stored":1111874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":66990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":67480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T22:43:07.933 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T22:43:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.071+0000 7fab5d0f8640 1 -- 172.21.15.40:0/448840530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab581022f0 msgr2=0x7fab58108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.071+0000 7fab5d0f8640 1 --2- 172.21.15.40:0/448840530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab581022f0 0x7fab58108b80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fab50009ca0 tx=0x7fab5002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.071+0000 7fab5d0f8640 1 -- 172.21.15.40:0/448840530 shutdown_connections 2024-03-18T22:43:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.071+0000 7fab5d0f8640 1 --2- 172.21.15.40:0/448840530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab581022f0 0x7fab58108b80 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.071+0000 7fab5d0f8640 1 --2- 172.21.15.40:0/448840530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab58101930 0x7fab58101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.071+0000 7fab5d0f8640 1 --2- 172.21.15.40:0/448840530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab58100740 0x7fab58100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.071+0000 7fab5d0f8640 1 -- 172.21.15.40:0/448840530 >> 172.21.15.40:0/448840530 conn(0x7fab580fbcb0 msgr2=0x7fab580fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.071+0000 7fab5d0f8640 1 -- 172.21.15.40:0/448840530 shutdown_connections 2024-03-18T22:43:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.071+0000 7fab5d0f8640 1 -- 172.21.15.40:0/448840530 wait complete. 2024-03-18T22:43:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.071+0000 7fab5d0f8640 1 Processor -- start 2024-03-18T22:43:08.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5d0f8640 1 -- start start 2024-03-18T22:43:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5d0f8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab58100740 0x7fab581979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5d0f8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab58101930 0x7fab58197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5d0f8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab581022f0 0x7fab5819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5d0f8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fab5810d390 con 0x7fab58101930 2024-03-18T22:43:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5d0f8640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fab5810d210 con 0x7fab58100740 2024-03-18T22:43:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5d0f8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fab5810d510 con 0x7fab581022f0 2024-03-18T22:43:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab58101930 0x7fab58197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab58101930 0x7fab58197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37368/0 (socket says 172.21.15.40:37368) 2024-03-18T22:43:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5659c640 1 -- 172.21.15.40:0/3139557289 learned_addr learned my addr 172.21.15.40:0/3139557289 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5759e640 1 --2- 172.21.15.40:0/3139557289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab581022f0 0x7fab5819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab56d9d640 1 --2- 172.21.15.40:0/3139557289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab58100740 0x7fab581979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5659c640 1 -- 172.21.15.40:0/3139557289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab581022f0 msgr2=0x7fab5819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5659c640 1 --2- 172.21.15.40:0/3139557289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab581022f0 0x7fab5819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5659c640 1 -- 172.21.15.40:0/3139557289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab58100740 msgr2=0x7fab581979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5659c640 1 --2- 172.21.15.40:0/3139557289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab58100740 0x7fab581979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5659c640 1 -- 172.21.15.40:0/3139557289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5819f610 con 0x7fab58101930 2024-03-18T22:43:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.072+0000 7fab5659c640 1 --2- 172.21.15.40:0/3139557289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab58101930 0x7fab58197f30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00e9f0 tx=0x7fab4c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.073+0000 7fab3ffff640 1 -- 172.21.15.40:0/3139557289 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c00cd80 con 0x7fab58101930 2024-03-18T22:43:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.073+0000 7fab3ffff640 1 -- 172.21.15.40:0/3139557289 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab4c0108e0 con 0x7fab58101930 2024-03-18T22:43:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.073+0000 7fab5d0f8640 1 -- 172.21.15.40:0/3139557289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab5819f900 con 0x7fab58101930 2024-03-18T22:43:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.073+0000 7fab3ffff640 1 -- 172.21.15.40:0/3139557289 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c018920 con 0x7fab58101930 2024-03-18T22:43:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.073+0000 7fab5d0f8640 1 -- 172.21.15.40:0/3139557289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab5819fea0 con 0x7fab58101930 2024-03-18T22:43:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.074+0000 7fab5d0f8640 1 -- 172.21.15.40:0/3139557289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab58107b30 con 0x7fab58101930 2024-03-18T22:43:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.075+0000 7fab3ffff640 1 -- 172.21.15.40:0/3139557289 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fab4c010a90 con 0x7fab58101930 2024-03-18T22:43:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.075+0000 7fab3ffff640 1 --2- 172.21.15.40:0/3139557289 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab400a2880 0x7fab400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.075+0000 7fab56d9d640 1 --2- 172.21.15.40:0/3139557289 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab400a2880 0x7fab400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.075+0000 7fab3ffff640 1 -- 172.21.15.40:0/3139557289 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fab4c0cf0f0 con 0x7fab58101930 2024-03-18T22:43:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.076+0000 7fab56d9d640 1 --2- 172.21.15.40:0/3139557289 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab400a2880 0x7fab400a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fab58101790 tx=0x7fab48005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:08.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.080+0000 7fab3ffff640 1 -- 172.21.15.40:0/3139557289 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fab4c09b750 con 0x7fab58101930 2024-03-18T22:43:08.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.261+0000 7fab5d0f8640 1 -- 172.21.15.40:0/3139557289 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fab58061c60 con 0x7fab400a2880 2024-03-18T22:43:08.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.270+0000 7fab3ffff640 1 -- 172.21.15.40:0/3139557289 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340540 (secure 0 0 0) 0x7fab58061c60 con 0x7fab400a2880 2024-03-18T22:43:08.274 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:43:08.276 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-18T22:43:08.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.273+0000 7fab5d0f8640 1 -- 172.21.15.40:0/3139557289 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab400a2880 msgr2=0x7fab400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:08.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.273+0000 7fab5d0f8640 1 --2- 172.21.15.40:0/3139557289 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab400a2880 0x7fab400a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fab58101790 tx=0x7fab48005f90 comp rx=0 tx=0).stop 2024-03-18T22:43:08.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.273+0000 7fab5d0f8640 1 -- 172.21.15.40:0/3139557289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab58101930 msgr2=0x7fab58197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:08.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.273+0000 7fab5d0f8640 1 --2- 172.21.15.40:0/3139557289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab58101930 0x7fab58197f30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00e9f0 tx=0x7fab4c00eeb0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.274+0000 7fab5d0f8640 1 -- 172.21.15.40:0/3139557289 shutdown_connections 2024-03-18T22:43:08.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.274+0000 7fab5d0f8640 1 --2- 172.21.15.40:0/3139557289 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab400a2880 0x7fab400a4d40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.274+0000 7fab5d0f8640 1 --2- 172.21.15.40:0/3139557289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab581022f0 0x7fab5819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.274+0000 7fab5d0f8640 1 --2- 172.21.15.40:0/3139557289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab58101930 0x7fab58197f30 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.274+0000 7fab5d0f8640 1 --2- 172.21.15.40:0/3139557289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab58100740 0x7fab581979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.274+0000 7fab5d0f8640 1 -- 172.21.15.40:0/3139557289 >> 172.21.15.40:0/3139557289 conn(0x7fab580fbcb0 msgr2=0x7fab58104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:08.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.274+0000 7fab5d0f8640 1 -- 172.21.15.40:0/3139557289 shutdown_connections 2024-03-18T22:43:08.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.274+0000 7fab5d0f8640 1 -- 172.21.15.40:0/3139557289 wait complete. 2024-03-18T22:43:08.284 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:08 smithi040 ceph-mon[62643]: pgmap v365: 129 pgs: 129 active+clean; 649 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1023 B/s wr, 10 op/s 2024-03-18T22:43:08.284 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:08 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/544968523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:43:08.284 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:08 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3816420584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:43:08.285 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-03-18T22:43:06.844822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664696,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1277,"num_read_kb":795,"num_write":451,"num_write_kb":3103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1182002,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":873,"ondisk_log_size":873,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384576,"kb_used_data":14624,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557248,"statfs":{"total":1151940427776,"available":1151546621952,"internally_reserved":0,"allocated":14974976,"data_stored":8208698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2470,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":109,"num_read_kb":59,"num_write":14,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003309"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947520+0000","last_change":"2024-03-18T22:40:41.806379+0000","last_active":"2024-03-18T22:42:44.947520+0000","last_peered":"2024-03-18T22:42:44.947520+0000","last_clean":"2024-03-18T22:42:44.947520+0000","last_became_active":"2024-03-18T22:40:41.806058+0000","last_became_peered":"2024-03-18T22:40:41.806058+0000","last_unstale":"2024-03-18T22:42:44.947520+0000","last_undegraded":"2024-03-18T22:42:44.947520+0000","last_fullsized":"2024-03-18T22:42:44.947520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:45.834396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955707+0000","last_change":"2024-03-18T22:40:41.813722+0000","last_active":"2024-03-18T22:42:44.955707+0000","last_peered":"2024-03-18T22:42:44.955707+0000","last_clean":"2024-03-18T22:42:44.955707+0000","last_became_active":"2024-03-18T22:40:41.813284+0000","last_became_peered":"2024-03-18T22:40:41.813284+0000","last_unstale":"2024-03-18T22:42:44.955707+0000","last_undegraded":"2024-03-18T22:42:44.955707+0000","last_fullsized":"2024-03-18T22:42:44.955707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:46.374117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955331+0000","last_change":"2024-03-18T22:40:41.808340+0000","last_active":"2024-03-18T22:42:44.955331+0000","last_peered":"2024-03-18T22:42:44.955331+0000","last_clean":"2024-03-18T22:42:44.955331+0000","last_became_active":"2024-03-18T22:40:41.808002+0000","last_became_peered":"2024-03-18T22:40:41.808002+0000","last_unstale":"2024-03-18T22:42:44.955331+0000","last_undegraded":"2024-03-18T22:42:44.955331+0000","last_fullsized":"2024-03-18T22:42:44.955331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:40:32.122124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945721+0000","last_change":"2024-03-18T22:40:41.809959+0000","last_active":"2024-03-18T22:42:44.945721+0000","last_peered":"2024-03-18T22:42:44.945721+0000","last_clean":"2024-03-18T22:42:44.945721+0000","last_became_active":"2024-03-18T22:40:41.809676+0000","last_became_peered":"2024-03-18T22:40:41.809676+0000","last_unstale":"2024-03-18T22:42:44.945721+0000","last_undegraded":"2024-03-18T22:42:44.945721+0000","last_fullsized":"2024-03-18T22:42:44.945721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:29:45.263011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950820+0000","last_change":"2024-03-18T22:40:41.804583+0000","last_active":"2024-03-18T22:42:44.950820+0000","last_peered":"2024-03-18T22:42:44.950820+0000","last_clean":"2024-03-18T22:42:44.950820+0000","last_became_active":"2024-03-18T22:40:41.804064+0000","last_became_peered":"2024-03-18T22:40:41.804064+0000","last_unstale":"2024-03-18T22:42:44.950820+0000","last_undegraded":"2024-03-18T22:42:44.950820+0000","last_fullsized":"2024-03-18T22:42:44.950820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:46:23.455106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955919+0000","last_change":"2024-03-18T22:41:44.301195+0000","last_active":"2024-03-18T22:42:44.955919+0000","last_peered":"2024-03-18T22:42:44.955919+0000","last_clean":"2024-03-18T22:42:44.955919+0000","last_became_active":"2024-03-18T22:41:44.300801+0000","last_became_peered":"2024-03-18T22:41:44.300801+0000","last_unstale":"2024-03-18T22:42:44.955919+0000","last_undegraded":"2024-03-18T22:42:44.955919+0000","last_fullsized":"2024-03-18T22:42:44.955919+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:12:55.579977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947351+0000","last_change":"2024-03-18T22:40:41.814971+0000","last_active":"2024-03-18T22:42:44.947351+0000","last_peered":"2024-03-18T22:42:44.947351+0000","last_clean":"2024-03-18T22:42:44.947351+0000","last_became_active":"2024-03-18T22:40:41.814305+0000","last_became_peered":"2024-03-18T22:40:41.814305+0000","last_unstale":"2024-03-18T22:42:44.947351+0000","last_undegraded":"2024-03-18T22:42:44.947351+0000","last_fullsized":"2024-03-18T22:42:44.947351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:07.390157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950536+0000","last_change":"2024-03-18T22:40:41.815674+0000","last_active":"2024-03-18T22:42:44.950536+0000","last_peered":"2024-03-18T22:42:44.950536+0000","last_clean":"2024-03-18T22:42:44.950536+0000","last_became_active":"2024-03-18T22:40:41.813560+0000","last_became_peered":"2024-03-18T22:40:41.813560+0000","last_unstale":"2024-03-18T22:42:44.950536+0000","last_undegraded":"2024-03-18T22:42:44.950536+0000","last_fullsized":"2024-03-18T22:42:44.950536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:42:32.409338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.958219+0000","last_change":"2024-03-18T22:40:41.817508+0000","last_active":"2024-03-18T22:42:44.958219+0000","last_peered":"2024-03-18T22:42:44.958219+0000","last_clean":"2024-03-18T22:42:44.958219+0000","last_became_active":"2024-03-18T22:40:41.817184+0000","last_became_peered":"2024-03-18T22:40:41.817184+0000","last_unstale":"2024-03-18T22:42:44.958219+0000","last_undegraded":"2024-03-18T22:42:44.958219+0000","last_fullsized":"2024-03-18T22:42:44.958219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:10.255238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956881+0000","last_change":"2024-03-18T22:40:41.806235+0000","last_active":"2024-03-18T22:42:44.956881+0000","last_peered":"2024-03-18T22:42:44.956881+0000","last_clean":"2024-03-18T22:42:44.956881+0000","last_became_active":"2024-03-18T22:40:41.805913+0000","last_became_peered":"2024-03-18T22:40:41.805913+0000","last_unstale":"2024-03-18T22:42:44.956881+0000","last_undegraded":"2024-03-18T22:42:44.956881+0000","last_fullsized":"2024-03-18T22:42:44.956881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:39:06.299989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947974+0000","last_change":"2024-03-18T22:40:41.814735+0000","last_active":"2024-03-18T22:42:44.947974+0000","last_peered":"2024-03-18T22:42:44.947974+0000","last_clean":"2024-03-18T22:42:44.947974+0000","last_became_active":"2024-03-18T22:40:41.814291+0000","last_became_peered":"2024-03-18T22:40:41.814291+0000","last_unstale":"2024-03-18T22:42:44.947974+0000","last_undegraded":"2024-03-18T22:42:44.947974+0000","last_fullsized":"2024-03-18T22:42:44.947974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:49.610674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945638+0000","last_change":"2024-03-18T22:40:41.815561+0000","last_active":"2024-03-18T22:42:44.945638+0000","last_peered":"2024-03-18T22:42:44.945638+0000","last_clean":"2024-03-18T22:42:44.945638+0000","last_became_active":"2024-03-18T22:40:41.815252+0000","last_became_peered":"2024-03-18T22:40:41.815252+0000","last_unstale":"2024-03-18T22:42:44.945638+0000","last_undegraded":"2024-03-18T22:42:44.945638+0000","last_fullsized":"2024-03-18T22:42:44.945638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:34.769168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947541+0000","last_change":"2024-03-18T22:40:41.803325+0000","last_active":"2024-03-18T22:42:44.947541+0000","last_peered":"2024-03-18T22:42:44.947541+0000","last_clean":"2024-03-18T22:42:44.947541+0000","last_became_active":"2024-03-18T22:40:41.803008+0000","last_became_peered":"2024-03-18T22:40:41.803008+0000","last_unstale":"2024-03-18T22:42:44.947541+0000","last_undegraded":"2024-03-18T22:42:44.947541+0000","last_fullsized":"2024-03-18T22:42:44.947541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:14:01.335548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.954563+0000","last_change":"2024-03-18T22:40:41.812541+0000","last_active":"2024-03-18T22:42:44.954563+0000","last_peered":"2024-03-18T22:42:44.954563+0000","last_clean":"2024-03-18T22:42:44.954563+0000","last_became_active":"2024-03-18T22:40:41.812153+0000","last_became_peered":"2024-03-18T22:40:41.812153+0000","last_unstale":"2024-03-18T22:42:44.954563+0000","last_undegraded":"2024-03-18T22:42:44.954563+0000","last_fullsized":"2024-03-18T22:42:44.954563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:04.927057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.954565+0000","last_change":"2024-03-18T22:40:41.815790+0000","last_active":"2024-03-18T22:42:44.954565+0000","last_peered":"2024-03-18T22:42:44.954565+0000","last_clean":"2024-03-18T22:42:44.954565+0000","last_became_active":"2024-03-18T22:40:41.814949+0000","last_became_peered":"2024-03-18T22:40:41.814949+0000","last_unstale":"2024-03-18T22:42:44.954565+0000","last_undegraded":"2024-03-18T22:42:44.954565+0000","last_fullsized":"2024-03-18T22:42:44.954565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:10:15.874285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957433+0000","last_change":"2024-03-18T22:40:41.809980+0000","last_active":"2024-03-18T22:42:44.957433+0000","last_peered":"2024-03-18T22:42:44.957433+0000","last_clean":"2024-03-18T22:42:44.957433+0000","last_became_active":"2024-03-18T22:40:41.809640+0000","last_became_peered":"2024-03-18T22:40:41.809640+0000","last_unstale":"2024-03-18T22:42:44.957433+0000","last_undegraded":"2024-03-18T22:42:44.957433+0000","last_fullsized":"2024-03-18T22:42:44.957433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:25.389027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947447+0000","last_change":"2024-03-18T22:40:41.807088+0000","last_active":"2024-03-18T22:42:44.947447+0000","last_peered":"2024-03-18T22:42:44.947447+0000","last_clean":"2024-03-18T22:42:44.947447+0000","last_became_active":"2024-03-18T22:40:41.806770+0000","last_became_peered":"2024-03-18T22:40:41.806770+0000","last_unstale":"2024-03-18T22:42:44.947447+0000","last_undegraded":"2024-03-18T22:42:44.947447+0000","last_fullsized":"2024-03-18T22:42:44.947447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:53:14.001247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956723+0000","last_change":"2024-03-18T22:40:41.812685+0000","last_active":"2024-03-18T22:42:44.956723+0000","last_peered":"2024-03-18T22:42:44.956723+0000","last_clean":"2024-03-18T22:42:44.956723+0000","last_became_active":"2024-03-18T22:40:41.812346+0000","last_became_peered":"2024-03-18T22:40:41.812346+0000","last_unstale":"2024-03-18T22:42:44.956723+0000","last_undegraded":"2024-03-18T22:42:44.956723+0000","last_fullsized":"2024-03-18T22:42:44.956723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:25.033460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950634+0000","last_change":"2024-03-18T22:40:41.812627+0000","last_active":"2024-03-18T22:42:44.950634+0000","last_peered":"2024-03-18T22:42:44.950634+0000","last_clean":"2024-03-18T22:42:44.950634+0000","last_became_active":"2024-03-18T22:40:41.812327+0000","last_became_peered":"2024-03-18T22:40:41.812327+0000","last_unstale":"2024-03-18T22:42:44.950634+0000","last_undegraded":"2024-03-18T22:42:44.950634+0000","last_fullsized":"2024-03-18T22:42:44.950634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:58:22.088589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949400+0000","last_change":"2024-03-18T22:40:45.847822+0000","last_active":"2024-03-18T22:42:44.949400+0000","last_peered":"2024-03-18T22:42:44.949400+0000","last_clean":"2024-03-18T22:42:44.949400+0000","last_became_active":"2024-03-18T22:40:45.847407+0000","last_became_peered":"2024-03-18T22:40:45.847407+0000","last_unstale":"2024-03-18T22:42:44.949400+0000","last_undegraded":"2024-03-18T22:42:44.949400+0000","last_fullsized":"2024-03-18T22:42:44.949400+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:44.825997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'15","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:01.983401+0000","last_change":"2024-03-18T22:40:41.218797+0000","last_active":"2024-03-18T22:43:01.983401+0000","last_peered":"2024-03-18T22:43:01.983401+0000","last_clean":"2024-03-18T22:43:01.983401+0000","last_became_active":"2024-03-18T22:40:41.218472+0000","last_became_peered":"2024-03-18T22:40:41.218472+0000","last_unstale":"2024-03-18T22:43:01.983401+0000","last_undegraded":"2024-03-18T22:43:01.983401+0000","last_fullsized":"2024-03-18T22:43:01.983401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:10.230257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956484+0000","last_change":"2024-03-18T22:41:44.005444+0000","last_active":"2024-03-18T22:42:44.956484+0000","last_peered":"2024-03-18T22:42:44.956484+0000","last_clean":"2024-03-18T22:42:44.956484+0000","last_became_active":"2024-03-18T22:41:44.005157+0000","last_became_peered":"2024-03-18T22:41:44.005157+0000","last_unstale":"2024-03-18T22:42:44.956484+0000","last_undegraded":"2024-03-18T22:42:44.956484+0000","last_fullsized":"2024-03-18T22:42:44.956484+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:53:16.249875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945333+0000","last_change":"2024-03-18T22:40:45.851126+0000","last_active":"2024-03-18T22:42:44.945333+0000","last_peered":"2024-03-18T22:42:44.945333+0000","last_clean":"2024-03-18T22:42:44.945333+0000","last_became_active":"2024-03-18T22:40:45.850549+0000","last_became_peered":"2024-03-18T22:40:45.850549+0000","last_unstale":"2024-03-18T22:42:44.945333+0000","last_undegraded":"2024-03-18T22:42:44.945333+0000","last_fullsized":"2024-03-18T22:42:44.945333+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:39.359263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951119+0000","last_change":"2024-03-18T22:40:41.220944+0000","last_active":"2024-03-18T22:42:44.951119+0000","last_peered":"2024-03-18T22:42:44.951119+0000","last_clean":"2024-03-18T22:42:44.951119+0000","last_became_active":"2024-03-18T22:40:41.220612+0000","last_became_peered":"2024-03-18T22:40:41.220612+0000","last_unstale":"2024-03-18T22:42:44.951119+0000","last_undegraded":"2024-03-18T22:42:44.951119+0000","last_fullsized":"2024-03-18T22:42:44.951119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:16.083134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.954672+0000","last_change":"2024-03-18T22:40:41.818031+0000","last_active":"2024-03-18T22:42:44.954672+0000","last_peered":"2024-03-18T22:42:44.954672+0000","last_clean":"2024-03-18T22:42:44.954672+0000","last_became_active":"2024-03-18T22:40:41.817693+0000","last_became_peered":"2024-03-18T22:40:41.817693+0000","last_unstale":"2024-03-18T22:42:44.954672+0000","last_undegraded":"2024-03-18T22:42:44.954672+0000","last_fullsized":"2024-03-18T22:42:44.954672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:05.901203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945296+0000","last_change":"2024-03-18T22:40:45.850749+0000","last_active":"2024-03-18T22:42:44.945296+0000","last_peered":"2024-03-18T22:42:44.945296+0000","last_clean":"2024-03-18T22:42:44.945296+0000","last_became_active":"2024-03-18T22:40:45.850318+0000","last_became_peered":"2024-03-18T22:40:45.850318+0000","last_unstale":"2024-03-18T22:42:44.945296+0000","last_undegraded":"2024-03-18T22:42:44.945296+0000","last_fullsized":"2024-03-18T22:42:44.945296+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:56.047413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950968+0000","last_change":"2024-03-18T22:40:41.220775+0000","last_active":"2024-03-18T22:42:44.950968+0000","last_peered":"2024-03-18T22:42:44.950968+0000","last_clean":"2024-03-18T22:42:44.950968+0000","last_became_active":"2024-03-18T22:40:41.220313+0000","last_became_peered":"2024-03-18T22:40:41.220313+0000","last_unstale":"2024-03-18T22:42:44.950968+0000","last_undegraded":"2024-03-18T22:42:44.950968+0000","last_fullsized":"2024-03-18T22:42:44.950968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:24.180884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957754+0000","last_change":"2024-03-18T22:40:41.818448+0000","last_active":"2024-03-18T22:42:44.957754+0000","last_peered":"2024-03-18T22:42:44.957754+0000","last_clean":"2024-03-18T22:42:44.957754+0000","last_became_active":"2024-03-18T22:40:41.818088+0000","last_became_peered":"2024-03-18T22:40:41.818088+0000","last_unstale":"2024-03-18T22:42:44.957754+0000","last_undegraded":"2024-03-18T22:42:44.957754+0000","last_fullsized":"2024-03-18T22:42:44.957754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:34:54.298226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945216+0000","last_change":"2024-03-18T22:40:45.846250+0000","last_active":"2024-03-18T22:42:44.945216+0000","last_peered":"2024-03-18T22:42:44.945216+0000","last_clean":"2024-03-18T22:42:44.945216+0000","last_became_active":"2024-03-18T22:40:45.845859+0000","last_became_peered":"2024-03-18T22:40:45.845859+0000","last_unstale":"2024-03-18T22:42:44.945216+0000","last_undegraded":"2024-03-18T22:42:44.945216+0000","last_fullsized":"2024-03-18T22:42:44.945216+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:57.855301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951899+0000","last_change":"2024-03-18T22:40:41.216569+0000","last_active":"2024-03-18T22:42:44.951899+0000","last_peered":"2024-03-18T22:42:44.951899+0000","last_clean":"2024-03-18T22:42:44.951899+0000","last_became_active":"2024-03-18T22:40:41.216255+0000","last_became_peered":"2024-03-18T22:40:41.216255+0000","last_unstale":"2024-03-18T22:42:44.951899+0000","last_undegraded":"2024-03-18T22:42:44.951899+0000","last_fullsized":"2024-03-18T22:42:44.951899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:23.016346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945576+0000","last_change":"2024-03-18T22:41:44.301578+0000","last_active":"2024-03-18T22:42:44.945576+0000","last_peered":"2024-03-18T22:42:44.945576+0000","last_clean":"2024-03-18T22:42:44.945576+0000","last_became_active":"2024-03-18T22:41:44.301178+0000","last_became_peered":"2024-03-18T22:41:44.301178+0000","last_unstale":"2024-03-18T22:42:44.945576+0000","last_undegraded":"2024-03-18T22:42:44.945576+0000","last_fullsized":"2024-03-18T22:42:44.945576+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:25:50.635812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949131+0000","last_change":"2024-03-18T22:41:44.302390+0000","last_active":"2024-03-18T22:42:44.949131+0000","last_peered":"2024-03-18T22:42:44.949131+0000","last_clean":"2024-03-18T22:42:44.949131+0000","last_became_active":"2024-03-18T22:41:44.301920+0000","last_became_peered":"2024-03-18T22:41:44.301920+0000","last_unstale":"2024-03-18T22:42:44.949131+0000","last_undegraded":"2024-03-18T22:42:44.949131+0000","last_fullsized":"2024-03-18T22:42:44.949131+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:56:12.561520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,7,11],"acting":[2,8,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:02.973659+0000","last_change":"2024-03-18T22:42:44.962925+0000","last_active":"2024-03-18T22:43:02.973659+0000","last_peered":"2024-03-18T22:43:02.973659+0000","last_clean":"2024-03-18T22:43:02.973659+0000","last_became_active":"2024-03-18T22:42:44.951614+0000","last_became_peered":"2024-03-18T22:42:44.951614+0000","last_unstale":"2024-03-18T22:43:02.973659+0000","last_undegraded":"2024-03-18T22:43:02.973659+0000","last_fullsized":"2024-03-18T22:43:02.973659+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:18:00.200940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950735+0000","last_change":"2024-03-18T22:41:44.004814+0000","last_active":"2024-03-18T22:42:44.950735+0000","last_peered":"2024-03-18T22:42:44.950735+0000","last_clean":"2024-03-18T22:42:44.950735+0000","last_became_active":"2024-03-18T22:41:44.004219+0000","last_became_peered":"2024-03-18T22:41:44.004219+0000","last_unstale":"2024-03-18T22:42:44.950735+0000","last_undegraded":"2024-03-18T22:42:44.950735+0000","last_fullsized":"2024-03-18T22:42:44.950735+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:29:03.746508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949773+0000","last_change":"2024-03-18T22:41:44.005790+0000","last_active":"2024-03-18T22:42:44.949773+0000","last_peered":"2024-03-18T22:42:44.949773+0000","last_clean":"2024-03-18T22:42:44.949773+0000","last_became_active":"2024-03-18T22:41:44.005386+0000","last_became_peered":"2024-03-18T22:41:44.005386+0000","last_unstale":"2024-03-18T22:42:44.949773+0000","last_undegraded":"2024-03-18T22:42:44.949773+0000","last_fullsized":"2024-03-18T22:42:44.949773+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:49:08.212639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955868+0000","last_change":"2024-03-18T22:40:41.217449+0000","last_active":"2024-03-18T22:42:44.955868+0000","last_peered":"2024-03-18T22:42:44.955868+0000","last_clean":"2024-03-18T22:42:44.955868+0000","last_became_active":"2024-03-18T22:40:41.217127+0000","last_became_peered":"2024-03-18T22:40:41.217127+0000","last_unstale":"2024-03-18T22:42:44.955868+0000","last_undegraded":"2024-03-18T22:42:44.955868+0000","last_fullsized":"2024-03-18T22:42:44.955868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:08:00.178720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945729+0000","last_change":"2024-03-18T22:40:41.814154+0000","last_active":"2024-03-18T22:42:44.945729+0000","last_peered":"2024-03-18T22:42:44.945729+0000","last_clean":"2024-03-18T22:42:44.945729+0000","last_became_active":"2024-03-18T22:40:41.813824+0000","last_became_peered":"2024-03-18T22:40:41.813824+0000","last_unstale":"2024-03-18T22:42:44.945729+0000","last_undegraded":"2024-03-18T22:42:44.945729+0000","last_fullsized":"2024-03-18T22:42:44.945729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:12.412689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947414+0000","last_change":"2024-03-18T22:40:45.852711+0000","last_active":"2024-03-18T22:42:44.947414+0000","last_peered":"2024-03-18T22:42:44.947414+0000","last_clean":"2024-03-18T22:42:44.947414+0000","last_became_active":"2024-03-18T22:40:45.852247+0000","last_became_peered":"2024-03-18T22:40:45.852247+0000","last_unstale":"2024-03-18T22:42:44.947414+0000","last_undegraded":"2024-03-18T22:42:44.947414+0000","last_fullsized":"2024-03-18T22:42:44.947414+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:03.061689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947647+0000","last_change":"2024-03-18T22:40:41.212399+0000","last_active":"2024-03-18T22:42:44.947647+0000","last_peered":"2024-03-18T22:42:44.947647+0000","last_clean":"2024-03-18T22:42:44.947647+0000","last_became_active":"2024-03-18T22:40:41.207416+0000","last_became_peered":"2024-03-18T22:40:41.207416+0000","last_unstale":"2024-03-18T22:42:44.947647+0000","last_undegraded":"2024-03-18T22:42:44.947647+0000","last_fullsized":"2024-03-18T22:42:44.947647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:11.406342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950757+0000","last_change":"2024-03-18T22:40:41.807313+0000","last_active":"2024-03-18T22:42:44.950757+0000","last_peered":"2024-03-18T22:42:44.950757+0000","last_clean":"2024-03-18T22:42:44.950757+0000","last_became_active":"2024-03-18T22:40:41.806029+0000","last_became_peered":"2024-03-18T22:40:41.806029+0000","last_unstale":"2024-03-18T22:42:44.950757+0000","last_undegraded":"2024-03-18T22:42:44.950757+0000","last_fullsized":"2024-03-18T22:42:44.950757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:23:43.973443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955513+0000","last_change":"2024-03-18T22:40:45.835479+0000","last_active":"2024-03-18T22:42:44.955513+0000","last_peered":"2024-03-18T22:42:44.955513+0000","last_clean":"2024-03-18T22:42:44.955513+0000","last_became_active":"2024-03-18T22:40:45.834994+0000","last_became_peered":"2024-03-18T22:40:45.834994+0000","last_unstale":"2024-03-18T22:42:44.955513+0000","last_undegraded":"2024-03-18T22:42:44.955513+0000","last_fullsized":"2024-03-18T22:42:44.955513+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:31.317843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945288+0000","last_change":"2024-03-18T22:40:41.212110+0000","last_active":"2024-03-18T22:42:44.945288+0000","last_peered":"2024-03-18T22:42:44.945288+0000","last_clean":"2024-03-18T22:42:44.945288+0000","last_became_active":"2024-03-18T22:40:41.211803+0000","last_became_peered":"2024-03-18T22:40:41.211803+0000","last_unstale":"2024-03-18T22:42:44.945288+0000","last_undegraded":"2024-03-18T22:42:44.945288+0000","last_fullsized":"2024-03-18T22:42:44.945288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:40.482983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956612+0000","last_change":"2024-03-18T22:40:41.812585+0000","last_active":"2024-03-18T22:42:44.956612+0000","last_peered":"2024-03-18T22:42:44.956612+0000","last_clean":"2024-03-18T22:42:44.956612+0000","last_became_active":"2024-03-18T22:40:41.812260+0000","last_became_peered":"2024-03-18T22:40:41.812260+0000","last_unstale":"2024-03-18T22:42:44.956612+0000","last_undegraded":"2024-03-18T22:42:44.956612+0000","last_fullsized":"2024-03-18T22:42:44.956612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:12:59.999503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950731+0000","last_change":"2024-03-18T22:40:45.842252+0000","last_active":"2024-03-18T22:42:44.950731+0000","last_peered":"2024-03-18T22:42:44.950731+0000","last_clean":"2024-03-18T22:42:44.950731+0000","last_became_active":"2024-03-18T22:40:45.841849+0000","last_became_peered":"2024-03-18T22:40:45.841849+0000","last_unstale":"2024-03-18T22:42:44.950731+0000","last_undegraded":"2024-03-18T22:42:44.950731+0000","last_fullsized":"2024-03-18T22:42:44.950731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:21.253947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'7","reported_seq":270,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:03.974303+0000","last_change":"2024-03-18T22:40:41.214253+0000","last_active":"2024-03-18T22:43:03.974303+0000","last_peered":"2024-03-18T22:43:03.974303+0000","last_clean":"2024-03-18T22:43:03.974303+0000","last_became_active":"2024-03-18T22:40:41.213911+0000","last_became_peered":"2024-03-18T22:40:41.213911+0000","last_unstale":"2024-03-18T22:43:03.974303+0000","last_undegraded":"2024-03-18T22:43:03.974303+0000","last_fullsized":"2024-03-18T22:43:03.974303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:19.185714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":232,"num_read_kb":232,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.958079+0000","last_change":"2024-03-18T22:40:41.813281+0000","last_active":"2024-03-18T22:42:44.958079+0000","last_peered":"2024-03-18T22:42:44.958079+0000","last_clean":"2024-03-18T22:42:44.958079+0000","last_became_active":"2024-03-18T22:40:41.812909+0000","last_became_peered":"2024-03-18T22:40:41.812909+0000","last_unstale":"2024-03-18T22:42:44.958079+0000","last_undegraded":"2024-03-18T22:42:44.958079+0000","last_fullsized":"2024-03-18T22:42:44.958079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:17:26.619441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956423+0000","last_change":"2024-03-18T22:40:45.846956+0000","last_active":"2024-03-18T22:42:44.956423+0000","last_peered":"2024-03-18T22:42:44.956423+0000","last_clean":"2024-03-18T22:42:44.956423+0000","last_became_active":"2024-03-18T22:40:45.843061+0000","last_became_peered":"2024-03-18T22:40:45.843061+0000","last_unstale":"2024-03-18T22:42:44.956423+0000","last_undegraded":"2024-03-18T22:42:44.956423+0000","last_fullsized":"2024-03-18T22:42:44.956423+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:52.043424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947970+0000","last_change":"2024-03-18T22:40:41.214933+0000","last_active":"2024-03-18T22:42:44.947970+0000","last_peered":"2024-03-18T22:42:44.947970+0000","last_clean":"2024-03-18T22:42:44.947970+0000","last_became_active":"2024-03-18T22:40:41.214606+0000","last_became_peered":"2024-03-18T22:40:41.214606+0000","last_unstale":"2024-03-18T22:42:44.947970+0000","last_undegraded":"2024-03-18T22:42:44.947970+0000","last_fullsized":"2024-03-18T22:42:44.947970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:08:06.230288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955583+0000","last_change":"2024-03-18T22:40:41.817590+0000","last_active":"2024-03-18T22:42:44.955583+0000","last_peered":"2024-03-18T22:42:44.955583+0000","last_clean":"2024-03-18T22:42:44.955583+0000","last_became_active":"2024-03-18T22:40:41.815809+0000","last_became_peered":"2024-03-18T22:40:41.815809+0000","last_unstale":"2024-03-18T22:42:44.955583+0000","last_undegraded":"2024-03-18T22:42:44.955583+0000","last_fullsized":"2024-03-18T22:42:44.955583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:44:54.043647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945405+0000","last_change":"2024-03-18T22:40:45.844767+0000","last_active":"2024-03-18T22:42:44.945405+0000","last_peered":"2024-03-18T22:42:44.945405+0000","last_clean":"2024-03-18T22:42:44.945405+0000","last_became_active":"2024-03-18T22:40:45.844251+0000","last_became_peered":"2024-03-18T22:40:45.844251+0000","last_unstale":"2024-03-18T22:42:44.945405+0000","last_undegraded":"2024-03-18T22:42:44.945405+0000","last_fullsized":"2024-03-18T22:42:44.945405+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:43.756959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945512+0000","last_change":"2024-03-18T22:40:41.214310+0000","last_active":"2024-03-18T22:42:44.945512+0000","last_peered":"2024-03-18T22:42:44.945512+0000","last_clean":"2024-03-18T22:42:44.945512+0000","last_became_active":"2024-03-18T22:40:41.213963+0000","last_became_peered":"2024-03-18T22:40:41.213963+0000","last_unstale":"2024-03-18T22:42:44.945512+0000","last_undegraded":"2024-03-18T22:42:44.945512+0000","last_fullsized":"2024-03-18T22:42:44.945512+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:03.401220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957904+0000","last_change":"2024-03-18T22:40:41.810876+0000","last_active":"2024-03-18T22:42:44.957904+0000","last_peered":"2024-03-18T22:42:44.957904+0000","last_clean":"2024-03-18T22:42:44.957904+0000","last_became_active":"2024-03-18T22:40:41.810104+0000","last_became_peered":"2024-03-18T22:40:41.810104+0000","last_unstale":"2024-03-18T22:42:44.957904+0000","last_undegraded":"2024-03-18T22:42:44.957904+0000","last_fullsized":"2024-03-18T22:42:44.957904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:41.294286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955701+0000","last_change":"2024-03-18T22:41:44.266318+0000","last_active":"2024-03-18T22:42:44.955701+0000","last_peered":"2024-03-18T22:42:44.955701+0000","last_clean":"2024-03-18T22:42:44.955701+0000","last_became_active":"2024-03-18T22:41:44.265903+0000","last_became_peered":"2024-03-18T22:41:44.265903+0000","last_unstale":"2024-03-18T22:42:44.955701+0000","last_undegraded":"2024-03-18T22:42:44.955701+0000","last_fullsized":"2024-03-18T22:42:44.955701+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:44.307096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'40","reported_seq":317,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:03.972467+0000","last_change":"2024-03-18T22:40:41.209837+0000","last_active":"2024-03-18T22:43:03.972467+0000","last_peered":"2024-03-18T22:43:03.972467+0000","last_clean":"2024-03-18T22:43:03.972467+0000","last_became_active":"2024-03-18T22:40:41.209465+0000","last_became_peered":"2024-03-18T22:40:41.209465+0000","last_unstale":"2024-03-18T22:43:03.972467+0000","last_undegraded":"2024-03-18T22:43:03.972467+0000","last_fullsized":"2024-03-18T22:43:03.972467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:45:15.765475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64100,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":245,"num_read_kb":186,"num_write":42,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951123+0000","last_change":"2024-03-18T22:40:41.810058+0000","last_active":"2024-03-18T22:42:44.951123+0000","last_peered":"2024-03-18T22:42:44.951123+0000","last_clean":"2024-03-18T22:42:44.951123+0000","last_became_active":"2024-03-18T22:40:41.809677+0000","last_became_peered":"2024-03-18T22:40:41.809677+0000","last_unstale":"2024-03-18T22:42:44.951123+0000","last_undegraded":"2024-03-18T22:42:44.951123+0000","last_fullsized":"2024-03-18T22:42:44.951123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:03.935036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945802+0000","last_change":"2024-03-18T22:40:45.845638+0000","last_active":"2024-03-18T22:42:44.945802+0000","last_peered":"2024-03-18T22:42:44.945802+0000","last_clean":"2024-03-18T22:42:44.945802+0000","last_became_active":"2024-03-18T22:40:45.844994+0000","last_became_peered":"2024-03-18T22:40:45.844994+0000","last_unstale":"2024-03-18T22:42:44.945802+0000","last_undegraded":"2024-03-18T22:42:44.945802+0000","last_fullsized":"2024-03-18T22:42:44.945802+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:21.551271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950170+0000","last_change":"2024-03-18T22:40:41.218397+0000","last_active":"2024-03-18T22:42:44.950170+0000","last_peered":"2024-03-18T22:42:44.950170+0000","last_clean":"2024-03-18T22:42:44.950170+0000","last_became_active":"2024-03-18T22:40:41.217587+0000","last_became_peered":"2024-03-18T22:40:41.217587+0000","last_unstale":"2024-03-18T22:42:44.950170+0000","last_undegraded":"2024-03-18T22:42:44.950170+0000","last_fullsized":"2024-03-18T22:42:44.950170+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:36.524954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947698+0000","last_change":"2024-03-18T22:40:41.812026+0000","last_active":"2024-03-18T22:42:44.947698+0000","last_peered":"2024-03-18T22:42:44.947698+0000","last_clean":"2024-03-18T22:42:44.947698+0000","last_became_active":"2024-03-18T22:40:41.811668+0000","last_became_peered":"2024-03-18T22:40:41.811668+0000","last_unstale":"2024-03-18T22:42:44.947698+0000","last_undegraded":"2024-03-18T22:42:44.947698+0000","last_fullsized":"2024-03-18T22:42:44.947698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:17:54.775327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947145+0000","last_change":"2024-03-18T22:40:45.848466+0000","last_active":"2024-03-18T22:42:44.947145+0000","last_peered":"2024-03-18T22:42:44.947145+0000","last_clean":"2024-03-18T22:42:44.947145+0000","last_became_active":"2024-03-18T22:40:45.848078+0000","last_became_peered":"2024-03-18T22:40:45.848078+0000","last_unstale":"2024-03-18T22:42:44.947145+0000","last_undegraded":"2024-03-18T22:42:44.947145+0000","last_fullsized":"2024-03-18T22:42:44.947145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:28.515368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945811+0000","last_change":"2024-03-18T22:40:41.216434+0000","last_active":"2024-03-18T22:42:44.945811+0000","last_peered":"2024-03-18T22:42:44.945811+0000","last_clean":"2024-03-18T22:42:44.945811+0000","last_became_active":"2024-03-18T22:40:41.216103+0000","last_became_peered":"2024-03-18T22:40:41.216103+0000","last_unstale":"2024-03-18T22:42:44.945811+0000","last_undegraded":"2024-03-18T22:42:44.945811+0000","last_fullsized":"2024-03-18T22:42:44.945811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:39.516463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947732+0000","last_change":"2024-03-18T22:40:41.808405+0000","last_active":"2024-03-18T22:42:44.947732+0000","last_peered":"2024-03-18T22:42:44.947732+0000","last_clean":"2024-03-18T22:42:44.947732+0000","last_became_active":"2024-03-18T22:40:41.808111+0000","last_became_peered":"2024-03-18T22:40:41.808111+0000","last_unstale":"2024-03-18T22:42:44.947732+0000","last_undegraded":"2024-03-18T22:42:44.947732+0000","last_fullsized":"2024-03-18T22:42:44.947732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:37:20.808163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956599+0000","last_change":"2024-03-18T22:40:45.848409+0000","last_active":"2024-03-18T22:42:44.956599+0000","last_peered":"2024-03-18T22:42:44.956599+0000","last_clean":"2024-03-18T22:42:44.956599+0000","last_became_active":"2024-03-18T22:40:45.847687+0000","last_became_peered":"2024-03-18T22:40:45.847687+0000","last_unstale":"2024-03-18T22:42:44.956599+0000","last_undegraded":"2024-03-18T22:42:44.956599+0000","last_fullsized":"2024-03-18T22:42:44.956599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:32.894885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'8","reported_seq":160,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:01.950868+0000","last_change":"2024-03-18T22:40:41.208408+0000","last_active":"2024-03-18T22:43:01.950868+0000","last_peered":"2024-03-18T22:43:01.950868+0000","last_clean":"2024-03-18T22:43:01.950868+0000","last_became_active":"2024-03-18T22:40:41.207976+0000","last_became_peered":"2024-03-18T22:40:41.207976+0000","last_unstale":"2024-03-18T22:43:01.950868+0000","last_undegraded":"2024-03-18T22:43:01.950868+0000","last_fullsized":"2024-03-18T22:43:01.950868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:43:24.416974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1614,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947842+0000","last_change":"2024-03-18T22:40:41.813547+0000","last_active":"2024-03-18T22:42:44.947842+0000","last_peered":"2024-03-18T22:42:44.947842+0000","last_clean":"2024-03-18T22:42:44.947842+0000","last_became_active":"2024-03-18T22:40:41.813250+0000","last_became_peered":"2024-03-18T22:40:41.813250+0000","last_unstale":"2024-03-18T22:42:44.947842+0000","last_undegraded":"2024-03-18T22:42:44.947842+0000","last_fullsized":"2024-03-18T22:42:44.947842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:54.428485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948147+0000","last_change":"2024-03-18T22:40:45.845797+0000","last_active":"2024-03-18T22:42:44.948147+0000","last_peered":"2024-03-18T22:42:44.948147+0000","last_clean":"2024-03-18T22:42:44.948147+0000","last_became_active":"2024-03-18T22:40:45.845324+0000","last_became_peered":"2024-03-18T22:40:45.845324+0000","last_unstale":"2024-03-18T22:42:44.948147+0000","last_undegraded":"2024-03-18T22:42:44.948147+0000","last_fullsized":"2024-03-18T22:42:44.948147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:21.613203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.969073+0000","last_change":"2024-03-18T22:42:44.968575+0000","last_active":"2024-03-18T22:42:44.969073+0000","last_peered":"2024-03-18T22:42:44.969073+0000","last_clean":"2024-03-18T22:42:44.969073+0000","last_became_active":"2024-03-18T22:42:44.957518+0000","last_became_peered":"2024-03-18T22:42:44.957518+0000","last_unstale":"2024-03-18T22:42:44.969073+0000","last_undegraded":"2024-03-18T22:42:44.969073+0000","last_fullsized":"2024-03-18T22:42:44.969073+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:42:40.590291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945444+0000","last_change":"2024-03-18T22:40:41.814606+0000","last_active":"2024-03-18T22:42:44.945444+0000","last_peered":"2024-03-18T22:42:44.945444+0000","last_clean":"2024-03-18T22:42:44.945444+0000","last_became_active":"2024-03-18T22:40:41.814309+0000","last_became_peered":"2024-03-18T22:40:41.814309+0000","last_unstale":"2024-03-18T22:42:44.945444+0000","last_undegraded":"2024-03-18T22:42:44.945444+0000","last_fullsized":"2024-03-18T22:42:44.945444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:33.086229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945376+0000","last_change":"2024-03-18T22:40:45.845711+0000","last_active":"2024-03-18T22:42:44.945376+0000","last_peered":"2024-03-18T22:42:44.945376+0000","last_clean":"2024-03-18T22:42:44.945376+0000","last_became_active":"2024-03-18T22:40:45.845094+0000","last_became_peered":"2024-03-18T22:40:45.845094+0000","last_unstale":"2024-03-18T22:42:44.945376+0000","last_undegraded":"2024-03-18T22:42:44.945376+0000","last_fullsized":"2024-03-18T22:42:44.945376+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:08.372875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:56.474783+0000","last_change":"2024-03-18T22:40:41.219076+0000","last_active":"2024-03-18T22:42:56.474783+0000","last_peered":"2024-03-18T22:42:56.474783+0000","last_clean":"2024-03-18T22:42:56.474783+0000","last_became_active":"2024-03-18T22:40:41.217580+0000","last_became_peered":"2024-03-18T22:40:41.217580+0000","last_unstale":"2024-03-18T22:42:56.474783+0000","last_undegraded":"2024-03-18T22:42:56.474783+0000","last_fullsized":"2024-03-18T22:42:56.474783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:38.112325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947271+0000","last_change":"2024-03-18T22:40:41.810267+0000","last_active":"2024-03-18T22:42:44.947271+0000","last_peered":"2024-03-18T22:42:44.947271+0000","last_clean":"2024-03-18T22:42:44.947271+0000","last_became_active":"2024-03-18T22:40:41.809960+0000","last_became_peered":"2024-03-18T22:40:41.809960+0000","last_unstale":"2024-03-18T22:42:44.947271+0000","last_undegraded":"2024-03-18T22:42:44.947271+0000","last_fullsized":"2024-03-18T22:42:44.947271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:00.973817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950675+0000","last_change":"2024-03-18T22:40:45.848461+0000","last_active":"2024-03-18T22:42:44.950675+0000","last_peered":"2024-03-18T22:42:44.950675+0000","last_clean":"2024-03-18T22:42:44.950675+0000","last_became_active":"2024-03-18T22:40:45.848043+0000","last_became_peered":"2024-03-18T22:40:45.848043+0000","last_unstale":"2024-03-18T22:42:44.950675+0000","last_undegraded":"2024-03-18T22:42:44.950675+0000","last_fullsized":"2024-03-18T22:42:44.950675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:38.865565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957138+0000","last_change":"2024-03-18T22:40:41.213069+0000","last_active":"2024-03-18T22:42:44.957138+0000","last_peered":"2024-03-18T22:42:44.957138+0000","last_clean":"2024-03-18T22:42:44.957138+0000","last_became_active":"2024-03-18T22:40:41.212725+0000","last_became_peered":"2024-03-18T22:40:41.212725+0000","last_unstale":"2024-03-18T22:42:44.957138+0000","last_undegraded":"2024-03-18T22:42:44.957138+0000","last_fullsized":"2024-03-18T22:42:44.957138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:50.219433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956533+0000","last_change":"2024-03-18T22:41:44.004772+0000","last_active":"2024-03-18T22:42:44.956533+0000","last_peered":"2024-03-18T22:42:44.956533+0000","last_clean":"2024-03-18T22:42:44.956533+0000","last_became_active":"2024-03-18T22:41:44.004342+0000","last_became_peered":"2024-03-18T22:41:44.004342+0000","last_unstale":"2024-03-18T22:42:44.956533+0000","last_undegraded":"2024-03-18T22:42:44.956533+0000","last_fullsized":"2024-03-18T22:42:44.956533+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:18:38.674911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947926+0000","last_change":"2024-03-18T22:40:45.840058+0000","last_active":"2024-03-18T22:42:44.947926+0000","last_peered":"2024-03-18T22:42:44.947926+0000","last_clean":"2024-03-18T22:42:44.947926+0000","last_became_active":"2024-03-18T22:40:45.839661+0000","last_became_peered":"2024-03-18T22:40:45.839661+0000","last_unstale":"2024-03-18T22:42:44.947926+0000","last_undegraded":"2024-03-18T22:42:44.947926+0000","last_fullsized":"2024-03-18T22:42:44.947926+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:35:14.323972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'8","reported_seq":156,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:01.983783+0000","last_change":"2024-03-18T22:40:41.215988+0000","last_active":"2024-03-18T22:43:01.983783+0000","last_peered":"2024-03-18T22:43:01.983783+0000","last_clean":"2024-03-18T22:43:01.983783+0000","last_became_active":"2024-03-18T22:40:41.215680+0000","last_became_peered":"2024-03-18T22:40:41.215680+0000","last_unstale":"2024-03-18T22:43:01.983783+0000","last_undegraded":"2024-03-18T22:43:01.983783+0000","last_fullsized":"2024-03-18T22:43:01.983783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:43.412362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1614,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956119+0000","last_change":"2024-03-18T22:40:41.812521+0000","last_active":"2024-03-18T22:42:44.956119+0000","last_peered":"2024-03-18T22:42:44.956119+0000","last_clean":"2024-03-18T22:42:44.956119+0000","last_became_active":"2024-03-18T22:40:41.812223+0000","last_became_peered":"2024-03-18T22:40:41.812223+0000","last_unstale":"2024-03-18T22:42:44.956119+0000","last_undegraded":"2024-03-18T22:42:44.956119+0000","last_fullsized":"2024-03-18T22:42:44.956119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:35:37.945144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948016+0000","last_change":"2024-03-18T22:40:45.838372+0000","last_active":"2024-03-18T22:42:44.948016+0000","last_peered":"2024-03-18T22:42:44.948016+0000","last_clean":"2024-03-18T22:42:44.948016+0000","last_became_active":"2024-03-18T22:40:45.837962+0000","last_became_peered":"2024-03-18T22:40:45.837962+0000","last_unstale":"2024-03-18T22:42:44.948016+0000","last_undegraded":"2024-03-18T22:42:44.948016+0000","last_fullsized":"2024-03-18T22:42:44.948016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:18:39.323023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945400+0000","last_change":"2024-03-18T22:40:41.812146+0000","last_active":"2024-03-18T22:42:44.945400+0000","last_peered":"2024-03-18T22:42:44.945400+0000","last_clean":"2024-03-18T22:42:44.945400+0000","last_became_active":"2024-03-18T22:40:41.811840+0000","last_became_peered":"2024-03-18T22:40:41.811840+0000","last_unstale":"2024-03-18T22:42:44.945400+0000","last_undegraded":"2024-03-18T22:42:44.945400+0000","last_fullsized":"2024-03-18T22:42:44.945400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:30.829187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'7","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:04.953826+0000","last_change":"2024-03-18T22:40:41.212133+0000","last_active":"2024-03-18T22:43:04.953826+0000","last_peered":"2024-03-18T22:43:04.953826+0000","last_clean":"2024-03-18T22:43:04.953826+0000","last_became_active":"2024-03-18T22:40:41.211581+0000","last_became_peered":"2024-03-18T22:40:41.211581+0000","last_unstale":"2024-03-18T22:43:04.953826+0000","last_undegraded":"2024-03-18T22:43:04.953826+0000","last_fullsized":"2024-03-18T22:43:04.953826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:44.287725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947800+0000","last_change":"2024-03-18T22:40:45.843660+0000","last_active":"2024-03-18T22:42:44.947800+0000","last_peered":"2024-03-18T22:42:44.947800+0000","last_clean":"2024-03-18T22:42:44.947800+0000","last_became_active":"2024-03-18T22:40:45.843233+0000","last_became_peered":"2024-03-18T22:40:45.843233+0000","last_unstale":"2024-03-18T22:42:44.947800+0000","last_undegraded":"2024-03-18T22:42:44.947800+0000","last_fullsized":"2024-03-18T22:42:44.947800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:08:19.758496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950814+0000","last_change":"2024-03-18T22:40:41.811798+0000","last_active":"2024-03-18T22:42:44.950814+0000","last_peered":"2024-03-18T22:42:44.950814+0000","last_clean":"2024-03-18T22:42:44.950814+0000","last_became_active":"2024-03-18T22:40:41.811422+0000","last_became_peered":"2024-03-18T22:40:41.811422+0000","last_unstale":"2024-03-18T22:42:44.950814+0000","last_undegraded":"2024-03-18T22:42:44.950814+0000","last_fullsized":"2024-03-18T22:42:44.950814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:53.944556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945349+0000","last_change":"2024-03-18T22:40:41.210052+0000","last_active":"2024-03-18T22:42:44.945349+0000","last_peered":"2024-03-18T22:42:44.945349+0000","last_clean":"2024-03-18T22:42:44.945349+0000","last_became_active":"2024-03-18T22:40:41.209647+0000","last_became_peered":"2024-03-18T22:40:41.209647+0000","last_unstale":"2024-03-18T22:42:44.945349+0000","last_undegraded":"2024-03-18T22:42:44.945349+0000","last_fullsized":"2024-03-18T22:42:44.945349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:57:35.187348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947306+0000","last_change":"2024-03-18T22:39:32.831758+0000","last_active":"2024-03-18T22:42:44.947306+0000","last_peered":"2024-03-18T22:42:44.947306+0000","last_clean":"2024-03-18T22:42:44.947306+0000","last_became_active":"2024-03-18T22:39:32.831287+0000","last_became_peered":"2024-03-18T22:39:32.831287+0000","last_unstale":"2024-03-18T22:42:44.947306+0000","last_undegraded":"2024-03-18T22:42:44.947306+0000","last_fullsized":"2024-03-18T22:42:44.947306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:36:07.806854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:36:07.806854+0000","last_clean_scrub_stamp":"2024-03-18T22:36:07.806854+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:42.449107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945891+0000","last_change":"2024-03-18T22:40:45.842008+0000","last_active":"2024-03-18T22:42:44.945891+0000","last_peered":"2024-03-18T22:42:44.945891+0000","last_clean":"2024-03-18T22:42:44.945891+0000","last_became_active":"2024-03-18T22:40:45.841231+0000","last_became_peered":"2024-03-18T22:40:45.841231+0000","last_unstale":"2024-03-18T22:42:44.945891+0000","last_undegraded":"2024-03-18T22:42:44.945891+0000","last_fullsized":"2024-03-18T22:42:44.945891+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:06.812049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947613+0000","last_change":"2024-03-18T22:40:41.807216+0000","last_active":"2024-03-18T22:42:44.947613+0000","last_peered":"2024-03-18T22:42:44.947613+0000","last_clean":"2024-03-18T22:42:44.947613+0000","last_became_active":"2024-03-18T22:40:41.806797+0000","last_became_peered":"2024-03-18T22:40:41.806797+0000","last_unstale":"2024-03-18T22:42:44.947613+0000","last_undegraded":"2024-03-18T22:42:44.947613+0000","last_fullsized":"2024-03-18T22:42:44.947613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:26.422329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955531+0000","last_change":"2024-03-18T22:42:44.955531+0000","last_active":"2024-03-18T22:42:44.955531+0000","last_peered":"2024-03-18T22:42:44.955531+0000","last_clean":"2024-03-18T22:42:44.955531+0000","last_became_active":"2024-03-18T22:42:44.949859+0000","last_became_peered":"2024-03-18T22:42:44.949859+0000","last_unstale":"2024-03-18T22:42:44.955531+0000","last_undegraded":"2024-03-18T22:42:44.955531+0000","last_fullsized":"2024-03-18T22:42:44.955531+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:50.522069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947630+0000","last_change":"2024-03-18T22:40:45.839889+0000","last_active":"2024-03-18T22:42:44.947630+0000","last_peered":"2024-03-18T22:42:44.947630+0000","last_clean":"2024-03-18T22:42:44.947630+0000","last_became_active":"2024-03-18T22:40:45.839414+0000","last_became_peered":"2024-03-18T22:40:45.839414+0000","last_unstale":"2024-03-18T22:42:44.947630+0000","last_undegraded":"2024-03-18T22:42:44.947630+0000","last_fullsized":"2024-03-18T22:42:44.947630+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:52.497378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951108+0000","last_change":"2024-03-18T22:40:41.221026+0000","last_active":"2024-03-18T22:42:44.951108+0000","last_peered":"2024-03-18T22:42:44.951108+0000","last_clean":"2024-03-18T22:42:44.951108+0000","last_became_active":"2024-03-18T22:40:41.220681+0000","last_became_peered":"2024-03-18T22:40:41.220681+0000","last_unstale":"2024-03-18T22:42:44.951108+0000","last_undegraded":"2024-03-18T22:42:44.951108+0000","last_fullsized":"2024-03-18T22:42:44.951108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:11.490445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948129+0000","last_change":"2024-03-18T22:40:41.810500+0000","last_active":"2024-03-18T22:42:44.948129+0000","last_peered":"2024-03-18T22:42:44.948129+0000","last_clean":"2024-03-18T22:42:44.948129+0000","last_became_active":"2024-03-18T22:40:41.809723+0000","last_became_peered":"2024-03-18T22:40:41.809723+0000","last_unstale":"2024-03-18T22:42:44.948129+0000","last_undegraded":"2024-03-18T22:42:44.948129+0000","last_fullsized":"2024-03-18T22:42:44.948129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:18:37.053809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950609+0000","last_change":"2024-03-18T22:40:45.844323+0000","last_active":"2024-03-18T22:42:44.950609+0000","last_peered":"2024-03-18T22:42:44.950609+0000","last_clean":"2024-03-18T22:42:44.950609+0000","last_became_active":"2024-03-18T22:40:45.843146+0000","last_became_peered":"2024-03-18T22:40:45.843146+0000","last_unstale":"2024-03-18T22:42:44.950609+0000","last_undegraded":"2024-03-18T22:42:44.950609+0000","last_fullsized":"2024-03-18T22:42:44.950609+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:14:59.881467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'14","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:53.026843+0000","last_change":"2024-03-18T22:40:41.212126+0000","last_active":"2024-03-18T22:42:53.026843+0000","last_peered":"2024-03-18T22:42:53.026843+0000","last_clean":"2024-03-18T22:42:53.026843+0000","last_became_active":"2024-03-18T22:40:41.211772+0000","last_became_peered":"2024-03-18T22:40:41.211772+0000","last_unstale":"2024-03-18T22:42:53.026843+0000","last_undegraded":"2024-03-18T22:42:53.026843+0000","last_fullsized":"2024-03-18T22:42:53.026843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:03.215751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951248+0000","last_change":"2024-03-18T22:40:41.820476+0000","last_active":"2024-03-18T22:42:44.951248+0000","last_peered":"2024-03-18T22:42:44.951248+0000","last_clean":"2024-03-18T22:42:44.951248+0000","last_became_active":"2024-03-18T22:40:41.820084+0000","last_became_peered":"2024-03-18T22:40:41.820084+0000","last_unstale":"2024-03-18T22:42:44.951248+0000","last_undegraded":"2024-03-18T22:42:44.951248+0000","last_fullsized":"2024-03-18T22:42:44.951248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:51:50.580208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949154+0000","last_change":"2024-03-18T22:40:45.838588+0000","last_active":"2024-03-18T22:42:44.949154+0000","last_peered":"2024-03-18T22:42:44.949154+0000","last_clean":"2024-03-18T22:42:44.949154+0000","last_became_active":"2024-03-18T22:40:45.838068+0000","last_became_peered":"2024-03-18T22:40:45.838068+0000","last_unstale":"2024-03-18T22:42:44.949154+0000","last_undegraded":"2024-03-18T22:42:44.949154+0000","last_fullsized":"2024-03-18T22:42:44.949154+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:11:02.162668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945922+0000","last_change":"2024-03-18T22:40:41.210108+0000","last_active":"2024-03-18T22:42:44.945922+0000","last_peered":"2024-03-18T22:42:44.945922+0000","last_clean":"2024-03-18T22:42:44.945922+0000","last_became_active":"2024-03-18T22:40:41.209781+0000","last_became_peered":"2024-03-18T22:40:41.209781+0000","last_unstale":"2024-03-18T22:42:44.945922+0000","last_undegraded":"2024-03-18T22:42:44.945922+0000","last_fullsized":"2024-03-18T22:42:44.945922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:10.670822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955691+0000","last_change":"2024-03-18T22:40:41.814637+0000","last_active":"2024-03-18T22:42:44.955691+0000","last_peered":"2024-03-18T22:42:44.955691+0000","last_clean":"2024-03-18T22:42:44.955691+0000","last_became_active":"2024-03-18T22:40:41.814279+0000","last_became_peered":"2024-03-18T22:40:41.814279+0000","last_unstale":"2024-03-18T22:42:44.955691+0000","last_undegraded":"2024-03-18T22:42:44.955691+0000","last_fullsized":"2024-03-18T22:42:44.955691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:12.496230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945752+0000","last_change":"2024-03-18T22:40:45.851560+0000","last_active":"2024-03-18T22:42:44.945752+0000","last_peered":"2024-03-18T22:42:44.945752+0000","last_clean":"2024-03-18T22:42:44.945752+0000","last_became_active":"2024-03-18T22:40:45.850884+0000","last_became_peered":"2024-03-18T22:40:45.850884+0000","last_unstale":"2024-03-18T22:42:44.945752+0000","last_undegraded":"2024-03-18T22:42:44.945752+0000","last_fullsized":"2024-03-18T22:42:44.945752+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:04.114468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957432+0000","last_change":"2024-03-18T22:40:41.210156+0000","last_active":"2024-03-18T22:42:44.957432+0000","last_peered":"2024-03-18T22:42:44.957432+0000","last_clean":"2024-03-18T22:42:44.957432+0000","last_became_active":"2024-03-18T22:40:41.209855+0000","last_became_peered":"2024-03-18T22:40:41.209855+0000","last_unstale":"2024-03-18T22:42:44.957432+0000","last_undegraded":"2024-03-18T22:42:44.957432+0000","last_fullsized":"2024-03-18T22:42:44.957432+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:02.752573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949647+0000","last_change":"2024-03-18T22:40:41.803681+0000","last_active":"2024-03-18T22:42:44.949647+0000","last_peered":"2024-03-18T22:42:44.949647+0000","last_clean":"2024-03-18T22:42:44.949647+0000","last_became_active":"2024-03-18T22:40:41.803244+0000","last_became_peered":"2024-03-18T22:40:41.803244+0000","last_unstale":"2024-03-18T22:42:44.949647+0000","last_undegraded":"2024-03-18T22:42:44.949647+0000","last_fullsized":"2024-03-18T22:42:44.949647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:55.944575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951408+0000","last_change":"2024-03-18T22:41:44.005707+0000","last_active":"2024-03-18T22:42:44.951408+0000","last_peered":"2024-03-18T22:42:44.951408+0000","last_clean":"2024-03-18T22:42:44.951408+0000","last_became_active":"2024-03-18T22:41:44.004338+0000","last_became_peered":"2024-03-18T22:41:44.004338+0000","last_unstale":"2024-03-18T22:42:44.951408+0000","last_undegraded":"2024-03-18T22:42:44.951408+0000","last_fullsized":"2024-03-18T22:42:44.951408+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:46.054087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951208+0000","last_change":"2024-03-18T22:40:41.210830+0000","last_active":"2024-03-18T22:42:44.951208+0000","last_peered":"2024-03-18T22:42:44.951208+0000","last_clean":"2024-03-18T22:42:44.951208+0000","last_became_active":"2024-03-18T22:40:41.210476+0000","last_became_peered":"2024-03-18T22:40:41.210476+0000","last_unstale":"2024-03-18T22:42:44.951208+0000","last_undegraded":"2024-03-18T22:42:44.951208+0000","last_fullsized":"2024-03-18T22:42:44.951208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:13.243788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947025+0000","last_change":"2024-03-18T22:40:41.807678+0000","last_active":"2024-03-18T22:42:44.947025+0000","last_peered":"2024-03-18T22:42:44.947025+0000","last_clean":"2024-03-18T22:42:44.947025+0000","last_became_active":"2024-03-18T22:40:41.807383+0000","last_became_peered":"2024-03-18T22:40:41.807383+0000","last_unstale":"2024-03-18T22:42:44.947025+0000","last_undegraded":"2024-03-18T22:42:44.947025+0000","last_fullsized":"2024-03-18T22:42:44.947025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:27:55.930293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945556+0000","last_change":"2024-03-18T22:40:45.850982+0000","last_active":"2024-03-18T22:42:44.945556+0000","last_peered":"2024-03-18T22:42:44.945556+0000","last_clean":"2024-03-18T22:42:44.945556+0000","last_became_active":"2024-03-18T22:40:45.850526+0000","last_became_peered":"2024-03-18T22:40:45.850526+0000","last_unstale":"2024-03-18T22:42:44.945556+0000","last_undegraded":"2024-03-18T22:42:44.945556+0000","last_fullsized":"2024-03-18T22:42:44.945556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:05:24.450986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949309+0000","last_change":"2024-03-18T22:40:41.219540+0000","last_active":"2024-03-18T22:42:44.949309+0000","last_peered":"2024-03-18T22:42:44.949309+0000","last_clean":"2024-03-18T22:42:44.949309+0000","last_became_active":"2024-03-18T22:40:41.219206+0000","last_became_peered":"2024-03-18T22:40:41.219206+0000","last_unstale":"2024-03-18T22:42:44.949309+0000","last_undegraded":"2024-03-18T22:42:44.949309+0000","last_fullsized":"2024-03-18T22:42:44.949309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:33.586368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951047+0000","last_change":"2024-03-18T22:40:41.815561+0000","last_active":"2024-03-18T22:42:44.951047+0000","last_peered":"2024-03-18T22:42:44.951047+0000","last_clean":"2024-03-18T22:42:44.951047+0000","last_became_active":"2024-03-18T22:40:41.815082+0000","last_became_peered":"2024-03-18T22:40:41.815082+0000","last_unstale":"2024-03-18T22:42:44.951047+0000","last_undegraded":"2024-03-18T22:42:44.951047+0000","last_fullsized":"2024-03-18T22:42:44.951047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:01.676907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945700+0000","last_change":"2024-03-18T22:40:45.845386+0000","last_active":"2024-03-18T22:42:44.945700+0000","last_peered":"2024-03-18T22:42:44.945700+0000","last_clean":"2024-03-18T22:42:44.945700+0000","last_became_active":"2024-03-18T22:40:45.845007+0000","last_became_peered":"2024-03-18T22:40:45.845007+0000","last_unstale":"2024-03-18T22:42:44.945700+0000","last_undegraded":"2024-03-18T22:42:44.945700+0000","last_fullsized":"2024-03-18T22:42:44.945700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:41.429195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.960602+0000","last_change":"2024-03-18T22:42:44.960602+0000","last_active":"2024-03-18T22:42:44.960602+0000","last_peered":"2024-03-18T22:42:44.960602+0000","last_clean":"2024-03-18T22:42:44.960602+0000","last_became_active":"2024-03-18T22:42:44.951902+0000","last_became_peered":"2024-03-18T22:42:44.951902+0000","last_unstale":"2024-03-18T22:42:44.960602+0000","last_undegraded":"2024-03-18T22:42:44.960602+0000","last_fullsized":"2024-03-18T22:42:44.960602+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:07.038253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,7],"acting":[11,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947336+0000","last_change":"2024-03-18T22:40:41.807533+0000","last_active":"2024-03-18T22:42:44.947336+0000","last_peered":"2024-03-18T22:42:44.947336+0000","last_clean":"2024-03-18T22:42:44.947336+0000","last_became_active":"2024-03-18T22:40:41.807219+0000","last_became_peered":"2024-03-18T22:40:41.807219+0000","last_unstale":"2024-03-18T22:42:44.947336+0000","last_undegraded":"2024-03-18T22:42:44.947336+0000","last_fullsized":"2024-03-18T22:42:44.947336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:15:24.214367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957534+0000","last_change":"2024-03-18T22:40:45.839670+0000","last_active":"2024-03-18T22:42:44.957534+0000","last_peered":"2024-03-18T22:42:44.957534+0000","last_clean":"2024-03-18T22:42:44.957534+0000","last_became_active":"2024-03-18T22:40:45.839263+0000","last_became_peered":"2024-03-18T22:40:45.839263+0000","last_unstale":"2024-03-18T22:42:44.957534+0000","last_undegraded":"2024-03-18T22:42:44.957534+0000","last_fullsized":"2024-03-18T22:42:44.957534+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:48.991959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'6","reported_seq":152,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:00.986309+0000","last_change":"2024-03-18T22:40:41.209617+0000","last_active":"2024-03-18T22:43:00.986309+0000","last_peered":"2024-03-18T22:43:00.986309+0000","last_clean":"2024-03-18T22:43:00.986309+0000","last_became_active":"2024-03-18T22:40:41.209311+0000","last_became_peered":"2024-03-18T22:40:41.209311+0000","last_unstale":"2024-03-18T22:43:00.986309+0000","last_undegraded":"2024-03-18T22:43:00.986309+0000","last_fullsized":"2024-03-18T22:43:00.986309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:44:06.019870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948110+0000","last_change":"2024-03-18T22:40:41.808210+0000","last_active":"2024-03-18T22:42:44.948110+0000","last_peered":"2024-03-18T22:42:44.948110+0000","last_clean":"2024-03-18T22:42:44.948110+0000","last_became_active":"2024-03-18T22:40:41.807898+0000","last_became_peered":"2024-03-18T22:40:41.807898+0000","last_unstale":"2024-03-18T22:42:44.948110+0000","last_undegraded":"2024-03-18T22:42:44.948110+0000","last_fullsized":"2024-03-18T22:42:44.948110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:20.401165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.954699+0000","last_change":"2024-03-18T22:40:45.841905+0000","last_active":"2024-03-18T22:42:44.954699+0000","last_peered":"2024-03-18T22:42:44.954699+0000","last_clean":"2024-03-18T22:42:44.954699+0000","last_became_active":"2024-03-18T22:40:45.841464+0000","last_became_peered":"2024-03-18T22:40:45.841464+0000","last_unstale":"2024-03-18T22:42:44.954699+0000","last_undegraded":"2024-03-18T22:42:44.954699+0000","last_fullsized":"2024-03-18T22:42:44.954699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:30.435788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945748+0000","last_change":"2024-03-18T22:40:41.216919+0000","last_active":"2024-03-18T22:42:44.945748+0000","last_peered":"2024-03-18T22:42:44.945748+0000","last_clean":"2024-03-18T22:42:44.945748+0000","last_became_active":"2024-03-18T22:40:41.216608+0000","last_became_peered":"2024-03-18T22:40:41.216608+0000","last_unstale":"2024-03-18T22:42:44.945748+0000","last_undegraded":"2024-03-18T22:42:44.945748+0000","last_fullsized":"2024-03-18T22:42:44.945748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:10:42.230268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947883+0000","last_change":"2024-03-18T22:40:41.813879+0000","last_active":"2024-03-18T22:42:44.947883+0000","last_peered":"2024-03-18T22:42:44.947883+0000","last_clean":"2024-03-18T22:42:44.947883+0000","last_became_active":"2024-03-18T22:40:41.813584+0000","last_became_peered":"2024-03-18T22:40:41.813584+0000","last_unstale":"2024-03-18T22:42:44.947883+0000","last_undegraded":"2024-03-18T22:42:44.947883+0000","last_fullsized":"2024-03-18T22:42:44.947883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:20:44.808087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945203+0000","last_change":"2024-03-18T22:40:45.838033+0000","last_active":"2024-03-18T22:42:44.945203+0000","last_peered":"2024-03-18T22:42:44.945203+0000","last_clean":"2024-03-18T22:42:44.945203+0000","last_became_active":"2024-03-18T22:40:45.837530+0000","last_became_peered":"2024-03-18T22:40:45.837530+0000","last_unstale":"2024-03-18T22:42:44.945203+0000","last_undegraded":"2024-03-18T22:42:44.945203+0000","last_fullsized":"2024-03-18T22:42:44.945203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:43:43.444631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.961111+0000","last_change":"2024-03-18T22:42:44.960540+0000","last_active":"2024-03-18T22:42:44.961111+0000","last_peered":"2024-03-18T22:42:44.961111+0000","last_clean":"2024-03-18T22:42:44.961111+0000","last_became_active":"2024-03-18T22:42:44.952010+0000","last_became_peered":"2024-03-18T22:42:44.952010+0000","last_unstale":"2024-03-18T22:42:44.961111+0000","last_undegraded":"2024-03-18T22:42:44.961111+0000","last_fullsized":"2024-03-18T22:42:44.961111+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:46:48.257820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948217+0000","last_change":"2024-03-18T22:40:41.808959+0000","last_active":"2024-03-18T22:42:44.948217+0000","last_peered":"2024-03-18T22:42:44.948217+0000","last_clean":"2024-03-18T22:42:44.948217+0000","last_became_active":"2024-03-18T22:40:41.808671+0000","last_became_peered":"2024-03-18T22:40:41.808671+0000","last_unstale":"2024-03-18T22:42:44.948217+0000","last_undegraded":"2024-03-18T22:42:44.948217+0000","last_fullsized":"2024-03-18T22:42:44.948217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:37.115318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947087+0000","last_change":"2024-03-18T22:40:41.814240+0000","last_active":"2024-03-18T22:42:44.947087+0000","last_peered":"2024-03-18T22:42:44.947087+0000","last_clean":"2024-03-18T22:42:44.947087+0000","last_became_active":"2024-03-18T22:40:41.813925+0000","last_became_peered":"2024-03-18T22:40:41.813925+0000","last_unstale":"2024-03-18T22:42:44.947087+0000","last_undegraded":"2024-03-18T22:42:44.947087+0000","last_fullsized":"2024-03-18T22:42:44.947087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:38.999803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949347+0000","last_change":"2024-03-18T22:40:41.813381+0000","last_active":"2024-03-18T22:42:44.949347+0000","last_peered":"2024-03-18T22:42:44.949347+0000","last_clean":"2024-03-18T22:42:44.949347+0000","last_became_active":"2024-03-18T22:40:41.812927+0000","last_became_peered":"2024-03-18T22:40:41.812927+0000","last_unstale":"2024-03-18T22:42:44.949347+0000","last_undegraded":"2024-03-18T22:42:44.949347+0000","last_fullsized":"2024-03-18T22:42:44.949347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:23:09.178485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949543+0000","last_change":"2024-03-18T22:40:41.812513+0000","last_active":"2024-03-18T22:42:44.949543+0000","last_peered":"2024-03-18T22:42:44.949543+0000","last_clean":"2024-03-18T22:42:44.949543+0000","last_became_active":"2024-03-18T22:40:41.811882+0000","last_became_peered":"2024-03-18T22:40:41.811882+0000","last_unstale":"2024-03-18T22:42:44.949543+0000","last_undegraded":"2024-03-18T22:42:44.949543+0000","last_fullsized":"2024-03-18T22:42:44.949543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:45.596911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955001+0000","last_change":"2024-03-18T22:40:41.814960+0000","last_active":"2024-03-18T22:42:44.955001+0000","last_peered":"2024-03-18T22:42:44.955001+0000","last_clean":"2024-03-18T22:42:44.955001+0000","last_became_active":"2024-03-18T22:40:41.814620+0000","last_became_peered":"2024-03-18T22:40:41.814620+0000","last_unstale":"2024-03-18T22:42:44.955001+0000","last_undegraded":"2024-03-18T22:42:44.955001+0000","last_fullsized":"2024-03-18T22:42:44.955001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:42.205885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948057+0000","last_change":"2024-03-18T22:41:44.334913+0000","last_active":"2024-03-18T22:42:44.948057+0000","last_peered":"2024-03-18T22:42:44.948057+0000","last_clean":"2024-03-18T22:42:44.948057+0000","last_became_active":"2024-03-18T22:41:44.334554+0000","last_became_peered":"2024-03-18T22:41:44.334554+0000","last_unstale":"2024-03-18T22:42:44.948057+0000","last_undegraded":"2024-03-18T22:42:44.948057+0000","last_fullsized":"2024-03-18T22:42:44.948057+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:00:17.369015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947401+0000","last_change":"2024-03-18T22:40:41.808718+0000","last_active":"2024-03-18T22:42:44.947401+0000","last_peered":"2024-03-18T22:42:44.947401+0000","last_clean":"2024-03-18T22:42:44.947401+0000","last_became_active":"2024-03-18T22:40:41.808379+0000","last_became_peered":"2024-03-18T22:40:41.808379+0000","last_unstale":"2024-03-18T22:42:44.947401+0000","last_undegraded":"2024-03-18T22:42:44.947401+0000","last_fullsized":"2024-03-18T22:42:44.947401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:26.637531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950358+0000","last_change":"2024-03-18T22:40:41.818046+0000","last_active":"2024-03-18T22:42:44.950358+0000","last_peered":"2024-03-18T22:42:44.950358+0000","last_clean":"2024-03-18T22:42:44.950358+0000","last_became_active":"2024-03-18T22:40:41.817698+0000","last_became_peered":"2024-03-18T22:40:41.817698+0000","last_unstale":"2024-03-18T22:42:44.950358+0000","last_undegraded":"2024-03-18T22:42:44.950358+0000","last_fullsized":"2024-03-18T22:42:44.950358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:22.967673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951447+0000","last_change":"2024-03-18T22:40:41.819337+0000","last_active":"2024-03-18T22:42:44.951447+0000","last_peered":"2024-03-18T22:42:44.951447+0000","last_clean":"2024-03-18T22:42:44.951447+0000","last_became_active":"2024-03-18T22:40:41.818987+0000","last_became_peered":"2024-03-18T22:40:41.818987+0000","last_unstale":"2024-03-18T22:42:44.951447+0000","last_undegraded":"2024-03-18T22:42:44.951447+0000","last_fullsized":"2024-03-18T22:42:44.951447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:07:04.529454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945317+0000","last_change":"2024-03-18T22:40:41.813262+0000","last_active":"2024-03-18T22:42:44.945317+0000","last_peered":"2024-03-18T22:42:44.945317+0000","last_clean":"2024-03-18T22:42:44.945317+0000","last_became_active":"2024-03-18T22:40:41.812956+0000","last_became_peered":"2024-03-18T22:40:41.812956+0000","last_unstale":"2024-03-18T22:42:44.945317+0000","last_undegraded":"2024-03-18T22:42:44.945317+0000","last_fullsized":"2024-03-18T22:42:44.945317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:19.940097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951429+0000","last_change":"2024-03-18T22:40:41.811897+0000","last_active":"2024-03-18T22:42:44.951429+0000","last_peered":"2024-03-18T22:42:44.951429+0000","last_clean":"2024-03-18T22:42:44.951429+0000","last_became_active":"2024-03-18T22:40:41.811575+0000","last_became_peered":"2024-03-18T22:40:41.811575+0000","last_unstale":"2024-03-18T22:42:44.951429+0000","last_undegraded":"2024-03-18T22:42:44.951429+0000","last_fullsized":"2024-03-18T22:42:44.951429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:45:43.321057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957689+0000","last_change":"2024-03-18T22:40:41.811757+0000","last_active":"2024-03-18T22:42:44.957689+0000","last_peered":"2024-03-18T22:42:44.957689+0000","last_clean":"2024-03-18T22:42:44.957689+0000","last_became_active":"2024-03-18T22:40:41.811405+0000","last_became_peered":"2024-03-18T22:40:41.811405+0000","last_unstale":"2024-03-18T22:42:44.957689+0000","last_undegraded":"2024-03-18T22:42:44.957689+0000","last_fullsized":"2024-03-18T22:42:44.957689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:39.224604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947749+0000","last_change":"2024-03-18T22:40:41.809987+0000","last_active":"2024-03-18T22:42:44.947749+0000","last_peered":"2024-03-18T22:42:44.947749+0000","last_clean":"2024-03-18T22:42:44.947749+0000","last_became_active":"2024-03-18T22:40:41.809544+0000","last_became_peered":"2024-03-18T22:40:41.809544+0000","last_unstale":"2024-03-18T22:42:44.947749+0000","last_undegraded":"2024-03-18T22:42:44.947749+0000","last_fullsized":"2024-03-18T22:42:44.947749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:23.800949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947201+0000","last_change":"2024-03-18T22:40:41.810963+0000","last_active":"2024-03-18T22:42:44.947201+0000","last_peered":"2024-03-18T22:42:44.947201+0000","last_clean":"2024-03-18T22:42:44.947201+0000","last_became_active":"2024-03-18T22:40:41.810641+0000","last_became_peered":"2024-03-18T22:40:41.810641+0000","last_unstale":"2024-03-18T22:42:44.947201+0000","last_undegraded":"2024-03-18T22:42:44.947201+0000","last_fullsized":"2024-03-18T22:42:44.947201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:37:48.872961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74094,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1151,"num_read_kb":686,"num_write":268,"num_write_kb":163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":222282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":234,"ondisk_log_size":234,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383721,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":522642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37056,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708096,"statfs":{"total":95995035648,"available":95957090304,"internally_reserved":0,"allocated":1761280,"data_stored":1172442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808883,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":524526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":520808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28660,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716492,"statfs":{"total":95995035648,"available":95965687808,"internally_reserved":0,"allocated":1683456,"data_stored":1119992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":516960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36452,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708700,"statfs":{"total":95995035648,"available":95957708800,"internally_reserved":0,"allocated":1142784,"data_stored":582202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":517884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1146880,"data_stored":582692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":515642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1073152,"data_stored":521034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28660,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716492,"statfs":{"total":95995035648,"available":95965687808,"internally_reserved":0,"allocated":1683456,"data_stored":1111874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":66990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":67480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T22:43:08.290 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-18T22:43:08.290 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T22:43:08.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:08 smithi113 ceph-mon[66775]: pgmap v365: 129 pgs: 129 active+clean; 649 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1023 B/s wr, 10 op/s 2024-03-18T22:43:08.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:08 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/544968523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:43:08.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:08 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3816420584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:43:08.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:08 smithi103 ceph-mon[66377]: pgmap v365: 129 pgs: 129 active+clean; 649 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1023 B/s wr, 10 op/s 2024-03-18T22:43:08.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:08 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/544968523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:43:08.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:08 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3816420584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:43:08.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.427+0000 7f316e922640 1 -- 172.21.15.40:0/722522898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3168100740 msgr2=0x7f3168100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:08.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.428+0000 7f316e922640 1 --2- 172.21.15.40:0/722522898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3168100740 0x7f3168100b60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3158009860 tx=0x7f315802bb70 comp rx=0 tx=0).stop 2024-03-18T22:43:08.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.428+0000 7f316e922640 1 -- 172.21.15.40:0/722522898 shutdown_connections 2024-03-18T22:43:08.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.428+0000 7f316e922640 1 --2- 172.21.15.40:0/722522898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f31681022f0 0x7f3168108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.428+0000 7f316e922640 1 --2- 172.21.15.40:0/722522898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3168101930 0x7f3168101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.428+0000 7f316e922640 1 --2- 172.21.15.40:0/722522898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3168100740 0x7f3168100b60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.428+0000 7f316e922640 1 -- 172.21.15.40:0/722522898 >> 172.21.15.40:0/722522898 conn(0x7f31680fbcb0 msgr2=0x7f31680fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.428+0000 7f316e922640 1 -- 172.21.15.40:0/722522898 shutdown_connections 2024-03-18T22:43:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.428+0000 7f316e922640 1 -- 172.21.15.40:0/722522898 wait complete. 2024-03-18T22:43:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.428+0000 7f316e922640 1 Processor -- start 2024-03-18T22:43:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.428+0000 7f316e922640 1 -- start start 2024-03-18T22:43:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.428+0000 7f316e922640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3168100740 0x7f3168197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.428+0000 7f316e922640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3168101930 0x7f3168197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f316e922640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31681022f0 0x7f316819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f316e922640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f316810d390 con 0x7f3168101930 2024-03-18T22:43:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f316e922640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f316810d210 con 0x7f31681022f0 2024-03-18T22:43:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f316e922640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f316810d510 con 0x7f3168100740 2024-03-18T22:43:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f31677fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3168101930 0x7f3168197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f3167fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3168100740 0x7f3168197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f31677fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3168101930 0x7f3168197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37394/0 (socket says 172.21.15.40:37394) 2024-03-18T22:43:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f31677fe640 1 -- 172.21.15.40:0/1771503468 learned_addr learned my addr 172.21.15.40:0/1771503468 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f316cebf640 1 --2- 172.21.15.40:0/1771503468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31681022f0 0x7f316819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f31677fe640 1 -- 172.21.15.40:0/1771503468 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3168100740 msgr2=0x7f3168197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f31677fe640 1 --2- 172.21.15.40:0/1771503468 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3168100740 0x7f3168197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f31677fe640 1 -- 172.21.15.40:0/1771503468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31681022f0 msgr2=0x7f316819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f31677fe640 1 --2- 172.21.15.40:0/1771503468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31681022f0 0x7f316819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f31677fe640 1 -- 172.21.15.40:0/1771503468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f316819f6d0 con 0x7f3168101930 2024-03-18T22:43:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f3167fff640 1 --2- 172.21.15.40:0/1771503468 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3168100740 0x7f3168197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f31677fe640 1 --2- 172.21.15.40:0/1771503468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3168101930 0x7f3168197ff0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f315c00e9c0 tx=0x7f315c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:08.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f31657fa640 1 -- 172.21.15.40:0/1771503468 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c00cde0 con 0x7f3168101930 2024-03-18T22:43:08.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f31657fa640 1 -- 172.21.15.40:0/1771503468 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f315c0035f0 con 0x7f3168101930 2024-03-18T22:43:08.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f316e922640 1 -- 172.21.15.40:0/1771503468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f316819f9c0 con 0x7f3168101930 2024-03-18T22:43:08.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f31657fa640 1 -- 172.21.15.40:0/1771503468 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c010690 con 0x7f3168101930 2024-03-18T22:43:08.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.429+0000 7f316e922640 1 -- 172.21.15.40:0/1771503468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f316819ff10 con 0x7f3168101930 2024-03-18T22:43:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.430+0000 7f316e922640 1 -- 172.21.15.40:0/1771503468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3168107b80 con 0x7f3168101930 2024-03-18T22:43:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.431+0000 7f31657fa640 1 -- 172.21.15.40:0/1771503468 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f315c0037a0 con 0x7f3168101930 2024-03-18T22:43:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.432+0000 7f31657fa640 1 --2- 172.21.15.40:0/1771503468 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f31500a2800 0x7f31500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.432+0000 7f3167fff640 1 --2- 172.21.15.40:0/1771503468 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f31500a2800 0x7f31500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.432+0000 7f31657fa640 1 -- 172.21.15.40:0/1771503468 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f315c0cf170 con 0x7f3168101930 2024-03-18T22:43:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.432+0000 7f3167fff640 1 --2- 172.21.15.40:0/1771503468 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f31500a2800 0x7f31500a4cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f315802c180 tx=0x7f31580349a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:08.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.436+0000 7f31657fa640 1 -- 172.21.15.40:0/1771503468 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f315c009510 con 0x7f3168101930 2024-03-18T22:43:08.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.615+0000 7f316e922640 1 -- 172.21.15.40:0/1771503468 --> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3168061c60 con 0x7f31500a2800 2024-03-18T22:43:08.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.625+0000 7f31657fa640 1 -- 172.21.15.40:0/1771503468 <== mgr.14148 v2:172.21.15.40:6800/3109254132 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340540 (secure 0 0 0) 0x7f3168061c60 con 0x7f31500a2800 2024-03-18T22:43:08.629 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:43:08.631 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-18T22:43:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.628+0000 7f316e922640 1 -- 172.21.15.40:0/1771503468 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f31500a2800 msgr2=0x7f31500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.628+0000 7f316e922640 1 --2- 172.21.15.40:0/1771503468 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f31500a2800 0x7f31500a4cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f315802c180 tx=0x7f31580349a0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.628+0000 7f316e922640 1 -- 172.21.15.40:0/1771503468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3168101930 msgr2=0x7f3168197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.628+0000 7f316e922640 1 --2- 172.21.15.40:0/1771503468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3168101930 0x7f3168197ff0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f315c00e9c0 tx=0x7f315c00ee80 comp rx=0 tx=0).stop 2024-03-18T22:43:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.628+0000 7f316e922640 1 -- 172.21.15.40:0/1771503468 shutdown_connections 2024-03-18T22:43:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.628+0000 7f316e922640 1 --2- 172.21.15.40:0/1771503468 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f31500a2800 0x7f31500a4cc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.628+0000 7f316e922640 1 --2- 172.21.15.40:0/1771503468 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31681022f0 0x7f316819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.628+0000 7f316e922640 1 --2- 172.21.15.40:0/1771503468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3168101930 0x7f3168197ff0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.628+0000 7f316e922640 1 --2- 172.21.15.40:0/1771503468 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3168100740 0x7f3168197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.628+0000 7f316e922640 1 -- 172.21.15.40:0/1771503468 >> 172.21.15.40:0/1771503468 conn(0x7f31680fbcb0 msgr2=0x7f3168104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:08.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.629+0000 7f316e922640 1 -- 172.21.15.40:0/1771503468 shutdown_connections 2024-03-18T22:43:08.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.629+0000 7f316e922640 1 -- 172.21.15.40:0/1771503468 wait complete. 2024-03-18T22:43:08.640 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-03-18T22:43:06.844822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664696,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1277,"num_read_kb":795,"num_write":451,"num_write_kb":3103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1182002,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":873,"ondisk_log_size":873,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384576,"kb_used_data":14624,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557248,"statfs":{"total":1151940427776,"available":1151546621952,"internally_reserved":0,"allocated":14974976,"data_stored":8208698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2470,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":109,"num_read_kb":59,"num_write":14,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003309"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947520+0000","last_change":"2024-03-18T22:40:41.806379+0000","last_active":"2024-03-18T22:42:44.947520+0000","last_peered":"2024-03-18T22:42:44.947520+0000","last_clean":"2024-03-18T22:42:44.947520+0000","last_became_active":"2024-03-18T22:40:41.806058+0000","last_became_peered":"2024-03-18T22:40:41.806058+0000","last_unstale":"2024-03-18T22:42:44.947520+0000","last_undegraded":"2024-03-18T22:42:44.947520+0000","last_fullsized":"2024-03-18T22:42:44.947520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:42:45.834396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955707+0000","last_change":"2024-03-18T22:40:41.813722+0000","last_active":"2024-03-18T22:42:44.955707+0000","last_peered":"2024-03-18T22:42:44.955707+0000","last_clean":"2024-03-18T22:42:44.955707+0000","last_became_active":"2024-03-18T22:40:41.813284+0000","last_became_peered":"2024-03-18T22:40:41.813284+0000","last_unstale":"2024-03-18T22:42:44.955707+0000","last_undegraded":"2024-03-18T22:42:44.955707+0000","last_fullsized":"2024-03-18T22:42:44.955707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:46.374117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955331+0000","last_change":"2024-03-18T22:40:41.808340+0000","last_active":"2024-03-18T22:42:44.955331+0000","last_peered":"2024-03-18T22:42:44.955331+0000","last_clean":"2024-03-18T22:42:44.955331+0000","last_became_active":"2024-03-18T22:40:41.808002+0000","last_became_peered":"2024-03-18T22:40:41.808002+0000","last_unstale":"2024-03-18T22:42:44.955331+0000","last_undegraded":"2024-03-18T22:42:44.955331+0000","last_fullsized":"2024-03-18T22:42:44.955331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:40:32.122124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945721+0000","last_change":"2024-03-18T22:40:41.809959+0000","last_active":"2024-03-18T22:42:44.945721+0000","last_peered":"2024-03-18T22:42:44.945721+0000","last_clean":"2024-03-18T22:42:44.945721+0000","last_became_active":"2024-03-18T22:40:41.809676+0000","last_became_peered":"2024-03-18T22:40:41.809676+0000","last_unstale":"2024-03-18T22:42:44.945721+0000","last_undegraded":"2024-03-18T22:42:44.945721+0000","last_fullsized":"2024-03-18T22:42:44.945721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:29:45.263011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950820+0000","last_change":"2024-03-18T22:40:41.804583+0000","last_active":"2024-03-18T22:42:44.950820+0000","last_peered":"2024-03-18T22:42:44.950820+0000","last_clean":"2024-03-18T22:42:44.950820+0000","last_became_active":"2024-03-18T22:40:41.804064+0000","last_became_peered":"2024-03-18T22:40:41.804064+0000","last_unstale":"2024-03-18T22:42:44.950820+0000","last_undegraded":"2024-03-18T22:42:44.950820+0000","last_fullsized":"2024-03-18T22:42:44.950820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:46:23.455106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955919+0000","last_change":"2024-03-18T22:41:44.301195+0000","last_active":"2024-03-18T22:42:44.955919+0000","last_peered":"2024-03-18T22:42:44.955919+0000","last_clean":"2024-03-18T22:42:44.955919+0000","last_became_active":"2024-03-18T22:41:44.300801+0000","last_became_peered":"2024-03-18T22:41:44.300801+0000","last_unstale":"2024-03-18T22:42:44.955919+0000","last_undegraded":"2024-03-18T22:42:44.955919+0000","last_fullsized":"2024-03-18T22:42:44.955919+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:12:55.579977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947351+0000","last_change":"2024-03-18T22:40:41.814971+0000","last_active":"2024-03-18T22:42:44.947351+0000","last_peered":"2024-03-18T22:42:44.947351+0000","last_clean":"2024-03-18T22:42:44.947351+0000","last_became_active":"2024-03-18T22:40:41.814305+0000","last_became_peered":"2024-03-18T22:40:41.814305+0000","last_unstale":"2024-03-18T22:42:44.947351+0000","last_undegraded":"2024-03-18T22:42:44.947351+0000","last_fullsized":"2024-03-18T22:42:44.947351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:32:07.390157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950536+0000","last_change":"2024-03-18T22:40:41.815674+0000","last_active":"2024-03-18T22:42:44.950536+0000","last_peered":"2024-03-18T22:42:44.950536+0000","last_clean":"2024-03-18T22:42:44.950536+0000","last_became_active":"2024-03-18T22:40:41.813560+0000","last_became_peered":"2024-03-18T22:40:41.813560+0000","last_unstale":"2024-03-18T22:42:44.950536+0000","last_undegraded":"2024-03-18T22:42:44.950536+0000","last_fullsized":"2024-03-18T22:42:44.950536+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:42:32.409338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.958219+0000","last_change":"2024-03-18T22:40:41.817508+0000","last_active":"2024-03-18T22:42:44.958219+0000","last_peered":"2024-03-18T22:42:44.958219+0000","last_clean":"2024-03-18T22:42:44.958219+0000","last_became_active":"2024-03-18T22:40:41.817184+0000","last_became_peered":"2024-03-18T22:40:41.817184+0000","last_unstale":"2024-03-18T22:42:44.958219+0000","last_undegraded":"2024-03-18T22:42:44.958219+0000","last_fullsized":"2024-03-18T22:42:44.958219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:32:10.255238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956881+0000","last_change":"2024-03-18T22:40:41.806235+0000","last_active":"2024-03-18T22:42:44.956881+0000","last_peered":"2024-03-18T22:42:44.956881+0000","last_clean":"2024-03-18T22:42:44.956881+0000","last_became_active":"2024-03-18T22:40:41.805913+0000","last_became_peered":"2024-03-18T22:40:41.805913+0000","last_unstale":"2024-03-18T22:42:44.956881+0000","last_undegraded":"2024-03-18T22:42:44.956881+0000","last_fullsized":"2024-03-18T22:42:44.956881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:39:06.299989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947974+0000","last_change":"2024-03-18T22:40:41.814735+0000","last_active":"2024-03-18T22:42:44.947974+0000","last_peered":"2024-03-18T22:42:44.947974+0000","last_clean":"2024-03-18T22:42:44.947974+0000","last_became_active":"2024-03-18T22:40:41.814291+0000","last_became_peered":"2024-03-18T22:40:41.814291+0000","last_unstale":"2024-03-18T22:42:44.947974+0000","last_undegraded":"2024-03-18T22:42:44.947974+0000","last_fullsized":"2024-03-18T22:42:44.947974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:49.610674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945638+0000","last_change":"2024-03-18T22:40:41.815561+0000","last_active":"2024-03-18T22:42:44.945638+0000","last_peered":"2024-03-18T22:42:44.945638+0000","last_clean":"2024-03-18T22:42:44.945638+0000","last_became_active":"2024-03-18T22:40:41.815252+0000","last_became_peered":"2024-03-18T22:40:41.815252+0000","last_unstale":"2024-03-18T22:42:44.945638+0000","last_undegraded":"2024-03-18T22:42:44.945638+0000","last_fullsized":"2024-03-18T22:42:44.945638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:41:34.769168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947541+0000","last_change":"2024-03-18T22:40:41.803325+0000","last_active":"2024-03-18T22:42:44.947541+0000","last_peered":"2024-03-18T22:42:44.947541+0000","last_clean":"2024-03-18T22:42:44.947541+0000","last_became_active":"2024-03-18T22:40:41.803008+0000","last_became_peered":"2024-03-18T22:40:41.803008+0000","last_unstale":"2024-03-18T22:42:44.947541+0000","last_undegraded":"2024-03-18T22:42:44.947541+0000","last_fullsized":"2024-03-18T22:42:44.947541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:14:01.335548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.954563+0000","last_change":"2024-03-18T22:40:41.812541+0000","last_active":"2024-03-18T22:42:44.954563+0000","last_peered":"2024-03-18T22:42:44.954563+0000","last_clean":"2024-03-18T22:42:44.954563+0000","last_became_active":"2024-03-18T22:40:41.812153+0000","last_became_peered":"2024-03-18T22:40:41.812153+0000","last_unstale":"2024-03-18T22:42:44.954563+0000","last_undegraded":"2024-03-18T22:42:44.954563+0000","last_fullsized":"2024-03-18T22:42:44.954563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:04.927057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.954565+0000","last_change":"2024-03-18T22:40:41.815790+0000","last_active":"2024-03-18T22:42:44.954565+0000","last_peered":"2024-03-18T22:42:44.954565+0000","last_clean":"2024-03-18T22:42:44.954565+0000","last_became_active":"2024-03-18T22:40:41.814949+0000","last_became_peered":"2024-03-18T22:40:41.814949+0000","last_unstale":"2024-03-18T22:42:44.954565+0000","last_undegraded":"2024-03-18T22:42:44.954565+0000","last_fullsized":"2024-03-18T22:42:44.954565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:10:15.874285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957433+0000","last_change":"2024-03-18T22:40:41.809980+0000","last_active":"2024-03-18T22:42:44.957433+0000","last_peered":"2024-03-18T22:42:44.957433+0000","last_clean":"2024-03-18T22:42:44.957433+0000","last_became_active":"2024-03-18T22:40:41.809640+0000","last_became_peered":"2024-03-18T22:40:41.809640+0000","last_unstale":"2024-03-18T22:42:44.957433+0000","last_undegraded":"2024-03-18T22:42:44.957433+0000","last_fullsized":"2024-03-18T22:42:44.957433+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:25.389027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947447+0000","last_change":"2024-03-18T22:40:41.807088+0000","last_active":"2024-03-18T22:42:44.947447+0000","last_peered":"2024-03-18T22:42:44.947447+0000","last_clean":"2024-03-18T22:42:44.947447+0000","last_became_active":"2024-03-18T22:40:41.806770+0000","last_became_peered":"2024-03-18T22:40:41.806770+0000","last_unstale":"2024-03-18T22:42:44.947447+0000","last_undegraded":"2024-03-18T22:42:44.947447+0000","last_fullsized":"2024-03-18T22:42:44.947447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:53:14.001247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956723+0000","last_change":"2024-03-18T22:40:41.812685+0000","last_active":"2024-03-18T22:42:44.956723+0000","last_peered":"2024-03-18T22:42:44.956723+0000","last_clean":"2024-03-18T22:42:44.956723+0000","last_became_active":"2024-03-18T22:40:41.812346+0000","last_became_peered":"2024-03-18T22:40:41.812346+0000","last_unstale":"2024-03-18T22:42:44.956723+0000","last_undegraded":"2024-03-18T22:42:44.956723+0000","last_fullsized":"2024-03-18T22:42:44.956723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:25.033460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950634+0000","last_change":"2024-03-18T22:40:41.812627+0000","last_active":"2024-03-18T22:42:44.950634+0000","last_peered":"2024-03-18T22:42:44.950634+0000","last_clean":"2024-03-18T22:42:44.950634+0000","last_became_active":"2024-03-18T22:40:41.812327+0000","last_became_peered":"2024-03-18T22:40:41.812327+0000","last_unstale":"2024-03-18T22:42:44.950634+0000","last_undegraded":"2024-03-18T22:42:44.950634+0000","last_fullsized":"2024-03-18T22:42:44.950634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:58:22.088589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949400+0000","last_change":"2024-03-18T22:40:45.847822+0000","last_active":"2024-03-18T22:42:44.949400+0000","last_peered":"2024-03-18T22:42:44.949400+0000","last_clean":"2024-03-18T22:42:44.949400+0000","last_became_active":"2024-03-18T22:40:45.847407+0000","last_became_peered":"2024-03-18T22:40:45.847407+0000","last_unstale":"2024-03-18T22:42:44.949400+0000","last_undegraded":"2024-03-18T22:42:44.949400+0000","last_fullsized":"2024-03-18T22:42:44.949400+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:44.825997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'15","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:01.983401+0000","last_change":"2024-03-18T22:40:41.218797+0000","last_active":"2024-03-18T22:43:01.983401+0000","last_peered":"2024-03-18T22:43:01.983401+0000","last_clean":"2024-03-18T22:43:01.983401+0000","last_became_active":"2024-03-18T22:40:41.218472+0000","last_became_peered":"2024-03-18T22:40:41.218472+0000","last_unstale":"2024-03-18T22:43:01.983401+0000","last_undegraded":"2024-03-18T22:43:01.983401+0000","last_fullsized":"2024-03-18T22:43:01.983401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:10.230257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956484+0000","last_change":"2024-03-18T22:41:44.005444+0000","last_active":"2024-03-18T22:42:44.956484+0000","last_peered":"2024-03-18T22:42:44.956484+0000","last_clean":"2024-03-18T22:42:44.956484+0000","last_became_active":"2024-03-18T22:41:44.005157+0000","last_became_peered":"2024-03-18T22:41:44.005157+0000","last_unstale":"2024-03-18T22:42:44.956484+0000","last_undegraded":"2024-03-18T22:42:44.956484+0000","last_fullsized":"2024-03-18T22:42:44.956484+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:53:16.249875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945333+0000","last_change":"2024-03-18T22:40:45.851126+0000","last_active":"2024-03-18T22:42:44.945333+0000","last_peered":"2024-03-18T22:42:44.945333+0000","last_clean":"2024-03-18T22:42:44.945333+0000","last_became_active":"2024-03-18T22:40:45.850549+0000","last_became_peered":"2024-03-18T22:40:45.850549+0000","last_unstale":"2024-03-18T22:42:44.945333+0000","last_undegraded":"2024-03-18T22:42:44.945333+0000","last_fullsized":"2024-03-18T22:42:44.945333+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:39.359263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951119+0000","last_change":"2024-03-18T22:40:41.220944+0000","last_active":"2024-03-18T22:42:44.951119+0000","last_peered":"2024-03-18T22:42:44.951119+0000","last_clean":"2024-03-18T22:42:44.951119+0000","last_became_active":"2024-03-18T22:40:41.220612+0000","last_became_peered":"2024-03-18T22:40:41.220612+0000","last_unstale":"2024-03-18T22:42:44.951119+0000","last_undegraded":"2024-03-18T22:42:44.951119+0000","last_fullsized":"2024-03-18T22:42:44.951119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:55:16.083134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.954672+0000","last_change":"2024-03-18T22:40:41.818031+0000","last_active":"2024-03-18T22:42:44.954672+0000","last_peered":"2024-03-18T22:42:44.954672+0000","last_clean":"2024-03-18T22:42:44.954672+0000","last_became_active":"2024-03-18T22:40:41.817693+0000","last_became_peered":"2024-03-18T22:40:41.817693+0000","last_unstale":"2024-03-18T22:42:44.954672+0000","last_undegraded":"2024-03-18T22:42:44.954672+0000","last_fullsized":"2024-03-18T22:42:44.954672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:05.901203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945296+0000","last_change":"2024-03-18T22:40:45.850749+0000","last_active":"2024-03-18T22:42:44.945296+0000","last_peered":"2024-03-18T22:42:44.945296+0000","last_clean":"2024-03-18T22:42:44.945296+0000","last_became_active":"2024-03-18T22:40:45.850318+0000","last_became_peered":"2024-03-18T22:40:45.850318+0000","last_unstale":"2024-03-18T22:42:44.945296+0000","last_undegraded":"2024-03-18T22:42:44.945296+0000","last_fullsized":"2024-03-18T22:42:44.945296+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:56.047413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950968+0000","last_change":"2024-03-18T22:40:41.220775+0000","last_active":"2024-03-18T22:42:44.950968+0000","last_peered":"2024-03-18T22:42:44.950968+0000","last_clean":"2024-03-18T22:42:44.950968+0000","last_became_active":"2024-03-18T22:40:41.220313+0000","last_became_peered":"2024-03-18T22:40:41.220313+0000","last_unstale":"2024-03-18T22:42:44.950968+0000","last_undegraded":"2024-03-18T22:42:44.950968+0000","last_fullsized":"2024-03-18T22:42:44.950968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:24.180884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957754+0000","last_change":"2024-03-18T22:40:41.818448+0000","last_active":"2024-03-18T22:42:44.957754+0000","last_peered":"2024-03-18T22:42:44.957754+0000","last_clean":"2024-03-18T22:42:44.957754+0000","last_became_active":"2024-03-18T22:40:41.818088+0000","last_became_peered":"2024-03-18T22:40:41.818088+0000","last_unstale":"2024-03-18T22:42:44.957754+0000","last_undegraded":"2024-03-18T22:42:44.957754+0000","last_fullsized":"2024-03-18T22:42:44.957754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:34:54.298226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945216+0000","last_change":"2024-03-18T22:40:45.846250+0000","last_active":"2024-03-18T22:42:44.945216+0000","last_peered":"2024-03-18T22:42:44.945216+0000","last_clean":"2024-03-18T22:42:44.945216+0000","last_became_active":"2024-03-18T22:40:45.845859+0000","last_became_peered":"2024-03-18T22:40:45.845859+0000","last_unstale":"2024-03-18T22:42:44.945216+0000","last_undegraded":"2024-03-18T22:42:44.945216+0000","last_fullsized":"2024-03-18T22:42:44.945216+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:57.855301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951899+0000","last_change":"2024-03-18T22:40:41.216569+0000","last_active":"2024-03-18T22:42:44.951899+0000","last_peered":"2024-03-18T22:42:44.951899+0000","last_clean":"2024-03-18T22:42:44.951899+0000","last_became_active":"2024-03-18T22:40:41.216255+0000","last_became_peered":"2024-03-18T22:40:41.216255+0000","last_unstale":"2024-03-18T22:42:44.951899+0000","last_undegraded":"2024-03-18T22:42:44.951899+0000","last_fullsized":"2024-03-18T22:42:44.951899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:23.016346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945576+0000","last_change":"2024-03-18T22:41:44.301578+0000","last_active":"2024-03-18T22:42:44.945576+0000","last_peered":"2024-03-18T22:42:44.945576+0000","last_clean":"2024-03-18T22:42:44.945576+0000","last_became_active":"2024-03-18T22:41:44.301178+0000","last_became_peered":"2024-03-18T22:41:44.301178+0000","last_unstale":"2024-03-18T22:42:44.945576+0000","last_undegraded":"2024-03-18T22:42:44.945576+0000","last_fullsized":"2024-03-18T22:42:44.945576+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:25:50.635812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949131+0000","last_change":"2024-03-18T22:41:44.302390+0000","last_active":"2024-03-18T22:42:44.949131+0000","last_peered":"2024-03-18T22:42:44.949131+0000","last_clean":"2024-03-18T22:42:44.949131+0000","last_became_active":"2024-03-18T22:41:44.301920+0000","last_became_peered":"2024-03-18T22:41:44.301920+0000","last_unstale":"2024-03-18T22:42:44.949131+0000","last_undegraded":"2024-03-18T22:42:44.949131+0000","last_fullsized":"2024-03-18T22:42:44.949131+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:56:12.561520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,7,11],"acting":[2,8,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:02.973659+0000","last_change":"2024-03-18T22:42:44.962925+0000","last_active":"2024-03-18T22:43:02.973659+0000","last_peered":"2024-03-18T22:43:02.973659+0000","last_clean":"2024-03-18T22:43:02.973659+0000","last_became_active":"2024-03-18T22:42:44.951614+0000","last_became_peered":"2024-03-18T22:42:44.951614+0000","last_unstale":"2024-03-18T22:43:02.973659+0000","last_undegraded":"2024-03-18T22:43:02.973659+0000","last_fullsized":"2024-03-18T22:43:02.973659+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:18:00.200940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950735+0000","last_change":"2024-03-18T22:41:44.004814+0000","last_active":"2024-03-18T22:42:44.950735+0000","last_peered":"2024-03-18T22:42:44.950735+0000","last_clean":"2024-03-18T22:42:44.950735+0000","last_became_active":"2024-03-18T22:41:44.004219+0000","last_became_peered":"2024-03-18T22:41:44.004219+0000","last_unstale":"2024-03-18T22:42:44.950735+0000","last_undegraded":"2024-03-18T22:42:44.950735+0000","last_fullsized":"2024-03-18T22:42:44.950735+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:29:03.746508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949773+0000","last_change":"2024-03-18T22:41:44.005790+0000","last_active":"2024-03-18T22:42:44.949773+0000","last_peered":"2024-03-18T22:42:44.949773+0000","last_clean":"2024-03-18T22:42:44.949773+0000","last_became_active":"2024-03-18T22:41:44.005386+0000","last_became_peered":"2024-03-18T22:41:44.005386+0000","last_unstale":"2024-03-18T22:42:44.949773+0000","last_undegraded":"2024-03-18T22:42:44.949773+0000","last_fullsized":"2024-03-18T22:42:44.949773+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:49:08.212639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955868+0000","last_change":"2024-03-18T22:40:41.217449+0000","last_active":"2024-03-18T22:42:44.955868+0000","last_peered":"2024-03-18T22:42:44.955868+0000","last_clean":"2024-03-18T22:42:44.955868+0000","last_became_active":"2024-03-18T22:40:41.217127+0000","last_became_peered":"2024-03-18T22:40:41.217127+0000","last_unstale":"2024-03-18T22:42:44.955868+0000","last_undegraded":"2024-03-18T22:42:44.955868+0000","last_fullsized":"2024-03-18T22:42:44.955868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:08:00.178720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945729+0000","last_change":"2024-03-18T22:40:41.814154+0000","last_active":"2024-03-18T22:42:44.945729+0000","last_peered":"2024-03-18T22:42:44.945729+0000","last_clean":"2024-03-18T22:42:44.945729+0000","last_became_active":"2024-03-18T22:40:41.813824+0000","last_became_peered":"2024-03-18T22:40:41.813824+0000","last_unstale":"2024-03-18T22:42:44.945729+0000","last_undegraded":"2024-03-18T22:42:44.945729+0000","last_fullsized":"2024-03-18T22:42:44.945729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:32:12.412689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947414+0000","last_change":"2024-03-18T22:40:45.852711+0000","last_active":"2024-03-18T22:42:44.947414+0000","last_peered":"2024-03-18T22:42:44.947414+0000","last_clean":"2024-03-18T22:42:44.947414+0000","last_became_active":"2024-03-18T22:40:45.852247+0000","last_became_peered":"2024-03-18T22:40:45.852247+0000","last_unstale":"2024-03-18T22:42:44.947414+0000","last_undegraded":"2024-03-18T22:42:44.947414+0000","last_fullsized":"2024-03-18T22:42:44.947414+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:22:03.061689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947647+0000","last_change":"2024-03-18T22:40:41.212399+0000","last_active":"2024-03-18T22:42:44.947647+0000","last_peered":"2024-03-18T22:42:44.947647+0000","last_clean":"2024-03-18T22:42:44.947647+0000","last_became_active":"2024-03-18T22:40:41.207416+0000","last_became_peered":"2024-03-18T22:40:41.207416+0000","last_unstale":"2024-03-18T22:42:44.947647+0000","last_undegraded":"2024-03-18T22:42:44.947647+0000","last_fullsized":"2024-03-18T22:42:44.947647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:36:11.406342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950757+0000","last_change":"2024-03-18T22:40:41.807313+0000","last_active":"2024-03-18T22:42:44.950757+0000","last_peered":"2024-03-18T22:42:44.950757+0000","last_clean":"2024-03-18T22:42:44.950757+0000","last_became_active":"2024-03-18T22:40:41.806029+0000","last_became_peered":"2024-03-18T22:40:41.806029+0000","last_unstale":"2024-03-18T22:42:44.950757+0000","last_undegraded":"2024-03-18T22:42:44.950757+0000","last_fullsized":"2024-03-18T22:42:44.950757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:23:43.973443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955513+0000","last_change":"2024-03-18T22:40:45.835479+0000","last_active":"2024-03-18T22:42:44.955513+0000","last_peered":"2024-03-18T22:42:44.955513+0000","last_clean":"2024-03-18T22:42:44.955513+0000","last_became_active":"2024-03-18T22:40:45.834994+0000","last_became_peered":"2024-03-18T22:40:45.834994+0000","last_unstale":"2024-03-18T22:42:44.955513+0000","last_undegraded":"2024-03-18T22:42:44.955513+0000","last_fullsized":"2024-03-18T22:42:44.955513+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:31.317843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945288+0000","last_change":"2024-03-18T22:40:41.212110+0000","last_active":"2024-03-18T22:42:44.945288+0000","last_peered":"2024-03-18T22:42:44.945288+0000","last_clean":"2024-03-18T22:42:44.945288+0000","last_became_active":"2024-03-18T22:40:41.211803+0000","last_became_peered":"2024-03-18T22:40:41.211803+0000","last_unstale":"2024-03-18T22:42:44.945288+0000","last_undegraded":"2024-03-18T22:42:44.945288+0000","last_fullsized":"2024-03-18T22:42:44.945288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:40.482983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956612+0000","last_change":"2024-03-18T22:40:41.812585+0000","last_active":"2024-03-18T22:42:44.956612+0000","last_peered":"2024-03-18T22:42:44.956612+0000","last_clean":"2024-03-18T22:42:44.956612+0000","last_became_active":"2024-03-18T22:40:41.812260+0000","last_became_peered":"2024-03-18T22:40:41.812260+0000","last_unstale":"2024-03-18T22:42:44.956612+0000","last_undegraded":"2024-03-18T22:42:44.956612+0000","last_fullsized":"2024-03-18T22:42:44.956612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:12:59.999503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950731+0000","last_change":"2024-03-18T22:40:45.842252+0000","last_active":"2024-03-18T22:42:44.950731+0000","last_peered":"2024-03-18T22:42:44.950731+0000","last_clean":"2024-03-18T22:42:44.950731+0000","last_became_active":"2024-03-18T22:40:45.841849+0000","last_became_peered":"2024-03-18T22:40:45.841849+0000","last_unstale":"2024-03-18T22:42:44.950731+0000","last_undegraded":"2024-03-18T22:42:44.950731+0000","last_fullsized":"2024-03-18T22:42:44.950731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:21.253947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'7","reported_seq":270,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:03.974303+0000","last_change":"2024-03-18T22:40:41.214253+0000","last_active":"2024-03-18T22:43:03.974303+0000","last_peered":"2024-03-18T22:43:03.974303+0000","last_clean":"2024-03-18T22:43:03.974303+0000","last_became_active":"2024-03-18T22:40:41.213911+0000","last_became_peered":"2024-03-18T22:40:41.213911+0000","last_unstale":"2024-03-18T22:43:03.974303+0000","last_undegraded":"2024-03-18T22:43:03.974303+0000","last_fullsized":"2024-03-18T22:43:03.974303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:38:19.185714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":232,"num_read_kb":232,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.958079+0000","last_change":"2024-03-18T22:40:41.813281+0000","last_active":"2024-03-18T22:42:44.958079+0000","last_peered":"2024-03-18T22:42:44.958079+0000","last_clean":"2024-03-18T22:42:44.958079+0000","last_became_active":"2024-03-18T22:40:41.812909+0000","last_became_peered":"2024-03-18T22:40:41.812909+0000","last_unstale":"2024-03-18T22:42:44.958079+0000","last_undegraded":"2024-03-18T22:42:44.958079+0000","last_fullsized":"2024-03-18T22:42:44.958079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:17:26.619441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956423+0000","last_change":"2024-03-18T22:40:45.846956+0000","last_active":"2024-03-18T22:42:44.956423+0000","last_peered":"2024-03-18T22:42:44.956423+0000","last_clean":"2024-03-18T22:42:44.956423+0000","last_became_active":"2024-03-18T22:40:45.843061+0000","last_became_peered":"2024-03-18T22:40:45.843061+0000","last_unstale":"2024-03-18T22:42:44.956423+0000","last_undegraded":"2024-03-18T22:42:44.956423+0000","last_fullsized":"2024-03-18T22:42:44.956423+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:12:52.043424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947970+0000","last_change":"2024-03-18T22:40:41.214933+0000","last_active":"2024-03-18T22:42:44.947970+0000","last_peered":"2024-03-18T22:42:44.947970+0000","last_clean":"2024-03-18T22:42:44.947970+0000","last_became_active":"2024-03-18T22:40:41.214606+0000","last_became_peered":"2024-03-18T22:40:41.214606+0000","last_unstale":"2024-03-18T22:42:44.947970+0000","last_undegraded":"2024-03-18T22:42:44.947970+0000","last_fullsized":"2024-03-18T22:42:44.947970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:08:06.230288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955583+0000","last_change":"2024-03-18T22:40:41.817590+0000","last_active":"2024-03-18T22:42:44.955583+0000","last_peered":"2024-03-18T22:42:44.955583+0000","last_clean":"2024-03-18T22:42:44.955583+0000","last_became_active":"2024-03-18T22:40:41.815809+0000","last_became_peered":"2024-03-18T22:40:41.815809+0000","last_unstale":"2024-03-18T22:42:44.955583+0000","last_undegraded":"2024-03-18T22:42:44.955583+0000","last_fullsized":"2024-03-18T22:42:44.955583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:44:54.043647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945405+0000","last_change":"2024-03-18T22:40:45.844767+0000","last_active":"2024-03-18T22:42:44.945405+0000","last_peered":"2024-03-18T22:42:44.945405+0000","last_clean":"2024-03-18T22:42:44.945405+0000","last_became_active":"2024-03-18T22:40:45.844251+0000","last_became_peered":"2024-03-18T22:40:45.844251+0000","last_unstale":"2024-03-18T22:42:44.945405+0000","last_undegraded":"2024-03-18T22:42:44.945405+0000","last_fullsized":"2024-03-18T22:42:44.945405+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:04:43.756959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945512+0000","last_change":"2024-03-18T22:40:41.214310+0000","last_active":"2024-03-18T22:42:44.945512+0000","last_peered":"2024-03-18T22:42:44.945512+0000","last_clean":"2024-03-18T22:42:44.945512+0000","last_became_active":"2024-03-18T22:40:41.213963+0000","last_became_peered":"2024-03-18T22:40:41.213963+0000","last_unstale":"2024-03-18T22:42:44.945512+0000","last_undegraded":"2024-03-18T22:42:44.945512+0000","last_fullsized":"2024-03-18T22:42:44.945512+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:22:03.401220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957904+0000","last_change":"2024-03-18T22:40:41.810876+0000","last_active":"2024-03-18T22:42:44.957904+0000","last_peered":"2024-03-18T22:42:44.957904+0000","last_clean":"2024-03-18T22:42:44.957904+0000","last_became_active":"2024-03-18T22:40:41.810104+0000","last_became_peered":"2024-03-18T22:40:41.810104+0000","last_unstale":"2024-03-18T22:42:44.957904+0000","last_undegraded":"2024-03-18T22:42:44.957904+0000","last_fullsized":"2024-03-18T22:42:44.957904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:33:41.294286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955701+0000","last_change":"2024-03-18T22:41:44.266318+0000","last_active":"2024-03-18T22:42:44.955701+0000","last_peered":"2024-03-18T22:42:44.955701+0000","last_clean":"2024-03-18T22:42:44.955701+0000","last_became_active":"2024-03-18T22:41:44.265903+0000","last_became_peered":"2024-03-18T22:41:44.265903+0000","last_unstale":"2024-03-18T22:42:44.955701+0000","last_undegraded":"2024-03-18T22:42:44.955701+0000","last_fullsized":"2024-03-18T22:42:44.955701+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:44.307096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'40","reported_seq":317,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:03.972467+0000","last_change":"2024-03-18T22:40:41.209837+0000","last_active":"2024-03-18T22:43:03.972467+0000","last_peered":"2024-03-18T22:43:03.972467+0000","last_clean":"2024-03-18T22:43:03.972467+0000","last_became_active":"2024-03-18T22:40:41.209465+0000","last_became_peered":"2024-03-18T22:40:41.209465+0000","last_unstale":"2024-03-18T22:43:03.972467+0000","last_undegraded":"2024-03-18T22:43:03.972467+0000","last_fullsized":"2024-03-18T22:43:03.972467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:45:15.765475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64100,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":245,"num_read_kb":186,"num_write":42,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951123+0000","last_change":"2024-03-18T22:40:41.810058+0000","last_active":"2024-03-18T22:42:44.951123+0000","last_peered":"2024-03-18T22:42:44.951123+0000","last_clean":"2024-03-18T22:42:44.951123+0000","last_became_active":"2024-03-18T22:40:41.809677+0000","last_became_peered":"2024-03-18T22:40:41.809677+0000","last_unstale":"2024-03-18T22:42:44.951123+0000","last_undegraded":"2024-03-18T22:42:44.951123+0000","last_fullsized":"2024-03-18T22:42:44.951123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:34:03.935036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945802+0000","last_change":"2024-03-18T22:40:45.845638+0000","last_active":"2024-03-18T22:42:44.945802+0000","last_peered":"2024-03-18T22:42:44.945802+0000","last_clean":"2024-03-18T22:42:44.945802+0000","last_became_active":"2024-03-18T22:40:45.844994+0000","last_became_peered":"2024-03-18T22:40:45.844994+0000","last_unstale":"2024-03-18T22:42:44.945802+0000","last_undegraded":"2024-03-18T22:42:44.945802+0000","last_fullsized":"2024-03-18T22:42:44.945802+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:21.551271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950170+0000","last_change":"2024-03-18T22:40:41.218397+0000","last_active":"2024-03-18T22:42:44.950170+0000","last_peered":"2024-03-18T22:42:44.950170+0000","last_clean":"2024-03-18T22:42:44.950170+0000","last_became_active":"2024-03-18T22:40:41.217587+0000","last_became_peered":"2024-03-18T22:40:41.217587+0000","last_unstale":"2024-03-18T22:42:44.950170+0000","last_undegraded":"2024-03-18T22:42:44.950170+0000","last_fullsized":"2024-03-18T22:42:44.950170+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:36.524954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947698+0000","last_change":"2024-03-18T22:40:41.812026+0000","last_active":"2024-03-18T22:42:44.947698+0000","last_peered":"2024-03-18T22:42:44.947698+0000","last_clean":"2024-03-18T22:42:44.947698+0000","last_became_active":"2024-03-18T22:40:41.811668+0000","last_became_peered":"2024-03-18T22:40:41.811668+0000","last_unstale":"2024-03-18T22:42:44.947698+0000","last_undegraded":"2024-03-18T22:42:44.947698+0000","last_fullsized":"2024-03-18T22:42:44.947698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:17:54.775327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947145+0000","last_change":"2024-03-18T22:40:45.848466+0000","last_active":"2024-03-18T22:42:44.947145+0000","last_peered":"2024-03-18T22:42:44.947145+0000","last_clean":"2024-03-18T22:42:44.947145+0000","last_became_active":"2024-03-18T22:40:45.848078+0000","last_became_peered":"2024-03-18T22:40:45.848078+0000","last_unstale":"2024-03-18T22:42:44.947145+0000","last_undegraded":"2024-03-18T22:42:44.947145+0000","last_fullsized":"2024-03-18T22:42:44.947145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:31:28.515368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945811+0000","last_change":"2024-03-18T22:40:41.216434+0000","last_active":"2024-03-18T22:42:44.945811+0000","last_peered":"2024-03-18T22:42:44.945811+0000","last_clean":"2024-03-18T22:42:44.945811+0000","last_became_active":"2024-03-18T22:40:41.216103+0000","last_became_peered":"2024-03-18T22:40:41.216103+0000","last_unstale":"2024-03-18T22:42:44.945811+0000","last_undegraded":"2024-03-18T22:42:44.945811+0000","last_fullsized":"2024-03-18T22:42:44.945811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:39.516463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947732+0000","last_change":"2024-03-18T22:40:41.808405+0000","last_active":"2024-03-18T22:42:44.947732+0000","last_peered":"2024-03-18T22:42:44.947732+0000","last_clean":"2024-03-18T22:42:44.947732+0000","last_became_active":"2024-03-18T22:40:41.808111+0000","last_became_peered":"2024-03-18T22:40:41.808111+0000","last_unstale":"2024-03-18T22:42:44.947732+0000","last_undegraded":"2024-03-18T22:42:44.947732+0000","last_fullsized":"2024-03-18T22:42:44.947732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:37:20.808163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956599+0000","last_change":"2024-03-18T22:40:45.848409+0000","last_active":"2024-03-18T22:42:44.956599+0000","last_peered":"2024-03-18T22:42:44.956599+0000","last_clean":"2024-03-18T22:42:44.956599+0000","last_became_active":"2024-03-18T22:40:45.847687+0000","last_became_peered":"2024-03-18T22:40:45.847687+0000","last_unstale":"2024-03-18T22:42:44.956599+0000","last_undegraded":"2024-03-18T22:42:44.956599+0000","last_fullsized":"2024-03-18T22:42:44.956599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:07:32.894885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'8","reported_seq":160,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:01.950868+0000","last_change":"2024-03-18T22:40:41.208408+0000","last_active":"2024-03-18T22:43:01.950868+0000","last_peered":"2024-03-18T22:43:01.950868+0000","last_clean":"2024-03-18T22:43:01.950868+0000","last_became_active":"2024-03-18T22:40:41.207976+0000","last_became_peered":"2024-03-18T22:40:41.207976+0000","last_unstale":"2024-03-18T22:43:01.950868+0000","last_undegraded":"2024-03-18T22:43:01.950868+0000","last_fullsized":"2024-03-18T22:43:01.950868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:43:24.416974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1614,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947842+0000","last_change":"2024-03-18T22:40:41.813547+0000","last_active":"2024-03-18T22:42:44.947842+0000","last_peered":"2024-03-18T22:42:44.947842+0000","last_clean":"2024-03-18T22:42:44.947842+0000","last_became_active":"2024-03-18T22:40:41.813250+0000","last_became_peered":"2024-03-18T22:40:41.813250+0000","last_unstale":"2024-03-18T22:42:44.947842+0000","last_undegraded":"2024-03-18T22:42:44.947842+0000","last_fullsized":"2024-03-18T22:42:44.947842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:54.428485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948147+0000","last_change":"2024-03-18T22:40:45.845797+0000","last_active":"2024-03-18T22:42:44.948147+0000","last_peered":"2024-03-18T22:42:44.948147+0000","last_clean":"2024-03-18T22:42:44.948147+0000","last_became_active":"2024-03-18T22:40:45.845324+0000","last_became_peered":"2024-03-18T22:40:45.845324+0000","last_unstale":"2024-03-18T22:42:44.948147+0000","last_undegraded":"2024-03-18T22:42:44.948147+0000","last_fullsized":"2024-03-18T22:42:44.948147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:21.613203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":74,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.969073+0000","last_change":"2024-03-18T22:42:44.968575+0000","last_active":"2024-03-18T22:42:44.969073+0000","last_peered":"2024-03-18T22:42:44.969073+0000","last_clean":"2024-03-18T22:42:44.969073+0000","last_became_active":"2024-03-18T22:42:44.957518+0000","last_became_peered":"2024-03-18T22:42:44.957518+0000","last_unstale":"2024-03-18T22:42:44.969073+0000","last_undegraded":"2024-03-18T22:42:44.969073+0000","last_fullsized":"2024-03-18T22:42:44.969073+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:42:40.590291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,1],"acting":[8,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945444+0000","last_change":"2024-03-18T22:40:41.814606+0000","last_active":"2024-03-18T22:42:44.945444+0000","last_peered":"2024-03-18T22:42:44.945444+0000","last_clean":"2024-03-18T22:42:44.945444+0000","last_became_active":"2024-03-18T22:40:41.814309+0000","last_became_peered":"2024-03-18T22:40:41.814309+0000","last_unstale":"2024-03-18T22:42:44.945444+0000","last_undegraded":"2024-03-18T22:42:44.945444+0000","last_fullsized":"2024-03-18T22:42:44.945444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:33.086229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945376+0000","last_change":"2024-03-18T22:40:45.845711+0000","last_active":"2024-03-18T22:42:44.945376+0000","last_peered":"2024-03-18T22:42:44.945376+0000","last_clean":"2024-03-18T22:42:44.945376+0000","last_became_active":"2024-03-18T22:40:45.845094+0000","last_became_peered":"2024-03-18T22:40:45.845094+0000","last_unstale":"2024-03-18T22:42:44.945376+0000","last_undegraded":"2024-03-18T22:42:44.945376+0000","last_fullsized":"2024-03-18T22:42:44.945376+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:08.372875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:56.474783+0000","last_change":"2024-03-18T22:40:41.219076+0000","last_active":"2024-03-18T22:42:56.474783+0000","last_peered":"2024-03-18T22:42:56.474783+0000","last_clean":"2024-03-18T22:42:56.474783+0000","last_became_active":"2024-03-18T22:40:41.217580+0000","last_became_peered":"2024-03-18T22:40:41.217580+0000","last_unstale":"2024-03-18T22:42:56.474783+0000","last_undegraded":"2024-03-18T22:42:56.474783+0000","last_fullsized":"2024-03-18T22:42:56.474783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:38.112325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947271+0000","last_change":"2024-03-18T22:40:41.810267+0000","last_active":"2024-03-18T22:42:44.947271+0000","last_peered":"2024-03-18T22:42:44.947271+0000","last_clean":"2024-03-18T22:42:44.947271+0000","last_became_active":"2024-03-18T22:40:41.809960+0000","last_became_peered":"2024-03-18T22:40:41.809960+0000","last_unstale":"2024-03-18T22:42:44.947271+0000","last_undegraded":"2024-03-18T22:42:44.947271+0000","last_fullsized":"2024-03-18T22:42:44.947271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:33:00.973817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950675+0000","last_change":"2024-03-18T22:40:45.848461+0000","last_active":"2024-03-18T22:42:44.950675+0000","last_peered":"2024-03-18T22:42:44.950675+0000","last_clean":"2024-03-18T22:42:44.950675+0000","last_became_active":"2024-03-18T22:40:45.848043+0000","last_became_peered":"2024-03-18T22:40:45.848043+0000","last_unstale":"2024-03-18T22:42:44.950675+0000","last_undegraded":"2024-03-18T22:42:44.950675+0000","last_fullsized":"2024-03-18T22:42:44.950675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:38.865565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957138+0000","last_change":"2024-03-18T22:40:41.213069+0000","last_active":"2024-03-18T22:42:44.957138+0000","last_peered":"2024-03-18T22:42:44.957138+0000","last_clean":"2024-03-18T22:42:44.957138+0000","last_became_active":"2024-03-18T22:40:41.212725+0000","last_became_peered":"2024-03-18T22:40:41.212725+0000","last_unstale":"2024-03-18T22:42:44.957138+0000","last_undegraded":"2024-03-18T22:42:44.957138+0000","last_fullsized":"2024-03-18T22:42:44.957138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:43:50.219433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956533+0000","last_change":"2024-03-18T22:41:44.004772+0000","last_active":"2024-03-18T22:42:44.956533+0000","last_peered":"2024-03-18T22:42:44.956533+0000","last_clean":"2024-03-18T22:42:44.956533+0000","last_became_active":"2024-03-18T22:41:44.004342+0000","last_became_peered":"2024-03-18T22:41:44.004342+0000","last_unstale":"2024-03-18T22:42:44.956533+0000","last_undegraded":"2024-03-18T22:42:44.956533+0000","last_fullsized":"2024-03-18T22:42:44.956533+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:18:38.674911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947926+0000","last_change":"2024-03-18T22:40:45.840058+0000","last_active":"2024-03-18T22:42:44.947926+0000","last_peered":"2024-03-18T22:42:44.947926+0000","last_clean":"2024-03-18T22:42:44.947926+0000","last_became_active":"2024-03-18T22:40:45.839661+0000","last_became_peered":"2024-03-18T22:40:45.839661+0000","last_unstale":"2024-03-18T22:42:44.947926+0000","last_undegraded":"2024-03-18T22:42:44.947926+0000","last_fullsized":"2024-03-18T22:42:44.947926+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:35:14.323972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'8","reported_seq":156,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:01.983783+0000","last_change":"2024-03-18T22:40:41.215988+0000","last_active":"2024-03-18T22:43:01.983783+0000","last_peered":"2024-03-18T22:43:01.983783+0000","last_clean":"2024-03-18T22:43:01.983783+0000","last_became_active":"2024-03-18T22:40:41.215680+0000","last_became_peered":"2024-03-18T22:40:41.215680+0000","last_unstale":"2024-03-18T22:43:01.983783+0000","last_undegraded":"2024-03-18T22:43:01.983783+0000","last_fullsized":"2024-03-18T22:43:01.983783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:43.412362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1614,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.956119+0000","last_change":"2024-03-18T22:40:41.812521+0000","last_active":"2024-03-18T22:42:44.956119+0000","last_peered":"2024-03-18T22:42:44.956119+0000","last_clean":"2024-03-18T22:42:44.956119+0000","last_became_active":"2024-03-18T22:40:41.812223+0000","last_became_peered":"2024-03-18T22:40:41.812223+0000","last_unstale":"2024-03-18T22:42:44.956119+0000","last_undegraded":"2024-03-18T22:42:44.956119+0000","last_fullsized":"2024-03-18T22:42:44.956119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:35:37.945144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948016+0000","last_change":"2024-03-18T22:40:45.838372+0000","last_active":"2024-03-18T22:42:44.948016+0000","last_peered":"2024-03-18T22:42:44.948016+0000","last_clean":"2024-03-18T22:42:44.948016+0000","last_became_active":"2024-03-18T22:40:45.837962+0000","last_became_peered":"2024-03-18T22:40:45.837962+0000","last_unstale":"2024-03-18T22:42:44.948016+0000","last_undegraded":"2024-03-18T22:42:44.948016+0000","last_fullsized":"2024-03-18T22:42:44.948016+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:18:39.323023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945400+0000","last_change":"2024-03-18T22:40:41.812146+0000","last_active":"2024-03-18T22:42:44.945400+0000","last_peered":"2024-03-18T22:42:44.945400+0000","last_clean":"2024-03-18T22:42:44.945400+0000","last_became_active":"2024-03-18T22:40:41.811840+0000","last_became_peered":"2024-03-18T22:40:41.811840+0000","last_unstale":"2024-03-18T22:42:44.945400+0000","last_undegraded":"2024-03-18T22:42:44.945400+0000","last_fullsized":"2024-03-18T22:42:44.945400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:50:30.829187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'7","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:04.953826+0000","last_change":"2024-03-18T22:40:41.212133+0000","last_active":"2024-03-18T22:43:04.953826+0000","last_peered":"2024-03-18T22:43:04.953826+0000","last_clean":"2024-03-18T22:43:04.953826+0000","last_became_active":"2024-03-18T22:40:41.211581+0000","last_became_peered":"2024-03-18T22:40:41.211581+0000","last_unstale":"2024-03-18T22:43:04.953826+0000","last_undegraded":"2024-03-18T22:43:04.953826+0000","last_fullsized":"2024-03-18T22:43:04.953826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:55:44.287725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947800+0000","last_change":"2024-03-18T22:40:45.843660+0000","last_active":"2024-03-18T22:42:44.947800+0000","last_peered":"2024-03-18T22:42:44.947800+0000","last_clean":"2024-03-18T22:42:44.947800+0000","last_became_active":"2024-03-18T22:40:45.843233+0000","last_became_peered":"2024-03-18T22:40:45.843233+0000","last_unstale":"2024-03-18T22:42:44.947800+0000","last_undegraded":"2024-03-18T22:42:44.947800+0000","last_fullsized":"2024-03-18T22:42:44.947800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:08:19.758496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950814+0000","last_change":"2024-03-18T22:40:41.811798+0000","last_active":"2024-03-18T22:42:44.950814+0000","last_peered":"2024-03-18T22:42:44.950814+0000","last_clean":"2024-03-18T22:42:44.950814+0000","last_became_active":"2024-03-18T22:40:41.811422+0000","last_became_peered":"2024-03-18T22:40:41.811422+0000","last_unstale":"2024-03-18T22:42:44.950814+0000","last_undegraded":"2024-03-18T22:42:44.950814+0000","last_fullsized":"2024-03-18T22:42:44.950814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:53.944556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945349+0000","last_change":"2024-03-18T22:40:41.210052+0000","last_active":"2024-03-18T22:42:44.945349+0000","last_peered":"2024-03-18T22:42:44.945349+0000","last_clean":"2024-03-18T22:42:44.945349+0000","last_became_active":"2024-03-18T22:40:41.209647+0000","last_became_peered":"2024-03-18T22:40:41.209647+0000","last_unstale":"2024-03-18T22:42:44.945349+0000","last_undegraded":"2024-03-18T22:42:44.945349+0000","last_fullsized":"2024-03-18T22:42:44.945349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:57:35.187348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947306+0000","last_change":"2024-03-18T22:39:32.831758+0000","last_active":"2024-03-18T22:42:44.947306+0000","last_peered":"2024-03-18T22:42:44.947306+0000","last_clean":"2024-03-18T22:42:44.947306+0000","last_became_active":"2024-03-18T22:39:32.831287+0000","last_became_peered":"2024-03-18T22:39:32.831287+0000","last_unstale":"2024-03-18T22:42:44.947306+0000","last_undegraded":"2024-03-18T22:42:44.947306+0000","last_fullsized":"2024-03-18T22:42:44.947306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:36:07.806854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:36:07.806854+0000","last_clean_scrub_stamp":"2024-03-18T22:36:07.806854+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:36:42.449107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945891+0000","last_change":"2024-03-18T22:40:45.842008+0000","last_active":"2024-03-18T22:42:44.945891+0000","last_peered":"2024-03-18T22:42:44.945891+0000","last_clean":"2024-03-18T22:42:44.945891+0000","last_became_active":"2024-03-18T22:40:45.841231+0000","last_became_peered":"2024-03-18T22:40:45.841231+0000","last_unstale":"2024-03-18T22:42:44.945891+0000","last_undegraded":"2024-03-18T22:42:44.945891+0000","last_fullsized":"2024-03-18T22:42:44.945891+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:03:06.812049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947613+0000","last_change":"2024-03-18T22:40:41.807216+0000","last_active":"2024-03-18T22:42:44.947613+0000","last_peered":"2024-03-18T22:42:44.947613+0000","last_clean":"2024-03-18T22:42:44.947613+0000","last_became_active":"2024-03-18T22:40:41.806797+0000","last_became_peered":"2024-03-18T22:40:41.806797+0000","last_unstale":"2024-03-18T22:42:44.947613+0000","last_undegraded":"2024-03-18T22:42:44.947613+0000","last_fullsized":"2024-03-18T22:42:44.947613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:26.422329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955531+0000","last_change":"2024-03-18T22:42:44.955531+0000","last_active":"2024-03-18T22:42:44.955531+0000","last_peered":"2024-03-18T22:42:44.955531+0000","last_clean":"2024-03-18T22:42:44.955531+0000","last_became_active":"2024-03-18T22:42:44.949859+0000","last_became_peered":"2024-03-18T22:42:44.949859+0000","last_unstale":"2024-03-18T22:42:44.955531+0000","last_undegraded":"2024-03-18T22:42:44.955531+0000","last_fullsized":"2024-03-18T22:42:44.955531+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:19:50.522069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947630+0000","last_change":"2024-03-18T22:40:45.839889+0000","last_active":"2024-03-18T22:42:44.947630+0000","last_peered":"2024-03-18T22:42:44.947630+0000","last_clean":"2024-03-18T22:42:44.947630+0000","last_became_active":"2024-03-18T22:40:45.839414+0000","last_became_peered":"2024-03-18T22:40:45.839414+0000","last_unstale":"2024-03-18T22:42:44.947630+0000","last_undegraded":"2024-03-18T22:42:44.947630+0000","last_fullsized":"2024-03-18T22:42:44.947630+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:04:52.497378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951108+0000","last_change":"2024-03-18T22:40:41.221026+0000","last_active":"2024-03-18T22:42:44.951108+0000","last_peered":"2024-03-18T22:42:44.951108+0000","last_clean":"2024-03-18T22:42:44.951108+0000","last_became_active":"2024-03-18T22:40:41.220681+0000","last_became_peered":"2024-03-18T22:40:41.220681+0000","last_unstale":"2024-03-18T22:42:44.951108+0000","last_undegraded":"2024-03-18T22:42:44.951108+0000","last_fullsized":"2024-03-18T22:42:44.951108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:36:11.490445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948129+0000","last_change":"2024-03-18T22:40:41.810500+0000","last_active":"2024-03-18T22:42:44.948129+0000","last_peered":"2024-03-18T22:42:44.948129+0000","last_clean":"2024-03-18T22:42:44.948129+0000","last_became_active":"2024-03-18T22:40:41.809723+0000","last_became_peered":"2024-03-18T22:40:41.809723+0000","last_unstale":"2024-03-18T22:42:44.948129+0000","last_undegraded":"2024-03-18T22:42:44.948129+0000","last_fullsized":"2024-03-18T22:42:44.948129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:18:37.053809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950609+0000","last_change":"2024-03-18T22:40:45.844323+0000","last_active":"2024-03-18T22:42:44.950609+0000","last_peered":"2024-03-18T22:42:44.950609+0000","last_clean":"2024-03-18T22:42:44.950609+0000","last_became_active":"2024-03-18T22:40:45.843146+0000","last_became_peered":"2024-03-18T22:40:45.843146+0000","last_unstale":"2024-03-18T22:42:44.950609+0000","last_undegraded":"2024-03-18T22:42:44.950609+0000","last_fullsized":"2024-03-18T22:42:44.950609+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:14:59.881467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'14","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:53.026843+0000","last_change":"2024-03-18T22:40:41.212126+0000","last_active":"2024-03-18T22:42:53.026843+0000","last_peered":"2024-03-18T22:42:53.026843+0000","last_clean":"2024-03-18T22:42:53.026843+0000","last_became_active":"2024-03-18T22:40:41.211772+0000","last_became_peered":"2024-03-18T22:40:41.211772+0000","last_unstale":"2024-03-18T22:42:53.026843+0000","last_undegraded":"2024-03-18T22:42:53.026843+0000","last_fullsized":"2024-03-18T22:42:53.026843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:03.215751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951248+0000","last_change":"2024-03-18T22:40:41.820476+0000","last_active":"2024-03-18T22:42:44.951248+0000","last_peered":"2024-03-18T22:42:44.951248+0000","last_clean":"2024-03-18T22:42:44.951248+0000","last_became_active":"2024-03-18T22:40:41.820084+0000","last_became_peered":"2024-03-18T22:40:41.820084+0000","last_unstale":"2024-03-18T22:42:44.951248+0000","last_undegraded":"2024-03-18T22:42:44.951248+0000","last_fullsized":"2024-03-18T22:42:44.951248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:51:50.580208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949154+0000","last_change":"2024-03-18T22:40:45.838588+0000","last_active":"2024-03-18T22:42:44.949154+0000","last_peered":"2024-03-18T22:42:44.949154+0000","last_clean":"2024-03-18T22:42:44.949154+0000","last_became_active":"2024-03-18T22:40:45.838068+0000","last_became_peered":"2024-03-18T22:40:45.838068+0000","last_unstale":"2024-03-18T22:42:44.949154+0000","last_undegraded":"2024-03-18T22:42:44.949154+0000","last_fullsized":"2024-03-18T22:42:44.949154+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:11:02.162668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945922+0000","last_change":"2024-03-18T22:40:41.210108+0000","last_active":"2024-03-18T22:42:44.945922+0000","last_peered":"2024-03-18T22:42:44.945922+0000","last_clean":"2024-03-18T22:42:44.945922+0000","last_became_active":"2024-03-18T22:40:41.209781+0000","last_became_peered":"2024-03-18T22:40:41.209781+0000","last_unstale":"2024-03-18T22:42:44.945922+0000","last_undegraded":"2024-03-18T22:42:44.945922+0000","last_fullsized":"2024-03-18T22:42:44.945922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:10.670822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955691+0000","last_change":"2024-03-18T22:40:41.814637+0000","last_active":"2024-03-18T22:42:44.955691+0000","last_peered":"2024-03-18T22:42:44.955691+0000","last_clean":"2024-03-18T22:42:44.955691+0000","last_became_active":"2024-03-18T22:40:41.814279+0000","last_became_peered":"2024-03-18T22:40:41.814279+0000","last_unstale":"2024-03-18T22:42:44.955691+0000","last_undegraded":"2024-03-18T22:42:44.955691+0000","last_fullsized":"2024-03-18T22:42:44.955691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:14:12.496230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945752+0000","last_change":"2024-03-18T22:40:45.851560+0000","last_active":"2024-03-18T22:42:44.945752+0000","last_peered":"2024-03-18T22:42:44.945752+0000","last_clean":"2024-03-18T22:42:44.945752+0000","last_became_active":"2024-03-18T22:40:45.850884+0000","last_became_peered":"2024-03-18T22:40:45.850884+0000","last_unstale":"2024-03-18T22:42:44.945752+0000","last_undegraded":"2024-03-18T22:42:44.945752+0000","last_fullsized":"2024-03-18T22:42:44.945752+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:04.114468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957432+0000","last_change":"2024-03-18T22:40:41.210156+0000","last_active":"2024-03-18T22:42:44.957432+0000","last_peered":"2024-03-18T22:42:44.957432+0000","last_clean":"2024-03-18T22:42:44.957432+0000","last_became_active":"2024-03-18T22:40:41.209855+0000","last_became_peered":"2024-03-18T22:40:41.209855+0000","last_unstale":"2024-03-18T22:42:44.957432+0000","last_undegraded":"2024-03-18T22:42:44.957432+0000","last_fullsized":"2024-03-18T22:42:44.957432+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:47:02.752573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949647+0000","last_change":"2024-03-18T22:40:41.803681+0000","last_active":"2024-03-18T22:42:44.949647+0000","last_peered":"2024-03-18T22:42:44.949647+0000","last_clean":"2024-03-18T22:42:44.949647+0000","last_became_active":"2024-03-18T22:40:41.803244+0000","last_became_peered":"2024-03-18T22:40:41.803244+0000","last_unstale":"2024-03-18T22:42:44.949647+0000","last_undegraded":"2024-03-18T22:42:44.949647+0000","last_fullsized":"2024-03-18T22:42:44.949647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:12:55.944575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951408+0000","last_change":"2024-03-18T22:41:44.005707+0000","last_active":"2024-03-18T22:42:44.951408+0000","last_peered":"2024-03-18T22:42:44.951408+0000","last_clean":"2024-03-18T22:42:44.951408+0000","last_became_active":"2024-03-18T22:41:44.004338+0000","last_became_peered":"2024-03-18T22:41:44.004338+0000","last_unstale":"2024-03-18T22:42:44.951408+0000","last_undegraded":"2024-03-18T22:42:44.951408+0000","last_fullsized":"2024-03-18T22:42:44.951408+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:58:46.054087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951208+0000","last_change":"2024-03-18T22:40:41.210830+0000","last_active":"2024-03-18T22:42:44.951208+0000","last_peered":"2024-03-18T22:42:44.951208+0000","last_clean":"2024-03-18T22:42:44.951208+0000","last_became_active":"2024-03-18T22:40:41.210476+0000","last_became_peered":"2024-03-18T22:40:41.210476+0000","last_unstale":"2024-03-18T22:42:44.951208+0000","last_undegraded":"2024-03-18T22:42:44.951208+0000","last_fullsized":"2024-03-18T22:42:44.951208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:13.243788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947025+0000","last_change":"2024-03-18T22:40:41.807678+0000","last_active":"2024-03-18T22:42:44.947025+0000","last_peered":"2024-03-18T22:42:44.947025+0000","last_clean":"2024-03-18T22:42:44.947025+0000","last_became_active":"2024-03-18T22:40:41.807383+0000","last_became_peered":"2024-03-18T22:40:41.807383+0000","last_unstale":"2024-03-18T22:42:44.947025+0000","last_undegraded":"2024-03-18T22:42:44.947025+0000","last_fullsized":"2024-03-18T22:42:44.947025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:27:55.930293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945556+0000","last_change":"2024-03-18T22:40:45.850982+0000","last_active":"2024-03-18T22:42:44.945556+0000","last_peered":"2024-03-18T22:42:44.945556+0000","last_clean":"2024-03-18T22:42:44.945556+0000","last_became_active":"2024-03-18T22:40:45.850526+0000","last_became_peered":"2024-03-18T22:40:45.850526+0000","last_unstale":"2024-03-18T22:42:44.945556+0000","last_undegraded":"2024-03-18T22:42:44.945556+0000","last_fullsized":"2024-03-18T22:42:44.945556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:05:24.450986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949309+0000","last_change":"2024-03-18T22:40:41.219540+0000","last_active":"2024-03-18T22:42:44.949309+0000","last_peered":"2024-03-18T22:42:44.949309+0000","last_clean":"2024-03-18T22:42:44.949309+0000","last_became_active":"2024-03-18T22:40:41.219206+0000","last_became_peered":"2024-03-18T22:40:41.219206+0000","last_unstale":"2024-03-18T22:42:44.949309+0000","last_undegraded":"2024-03-18T22:42:44.949309+0000","last_fullsized":"2024-03-18T22:42:44.949309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:05:33.586368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951047+0000","last_change":"2024-03-18T22:40:41.815561+0000","last_active":"2024-03-18T22:42:44.951047+0000","last_peered":"2024-03-18T22:42:44.951047+0000","last_clean":"2024-03-18T22:42:44.951047+0000","last_became_active":"2024-03-18T22:40:41.815082+0000","last_became_peered":"2024-03-18T22:40:41.815082+0000","last_unstale":"2024-03-18T22:42:44.951047+0000","last_undegraded":"2024-03-18T22:42:44.951047+0000","last_fullsized":"2024-03-18T22:42:44.951047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:01.676907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945700+0000","last_change":"2024-03-18T22:40:45.845386+0000","last_active":"2024-03-18T22:42:44.945700+0000","last_peered":"2024-03-18T22:42:44.945700+0000","last_clean":"2024-03-18T22:42:44.945700+0000","last_became_active":"2024-03-18T22:40:45.845007+0000","last_became_peered":"2024-03-18T22:40:45.845007+0000","last_unstale":"2024-03-18T22:42:44.945700+0000","last_undegraded":"2024-03-18T22:42:44.945700+0000","last_fullsized":"2024-03-18T22:42:44.945700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:41.429195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.960602+0000","last_change":"2024-03-18T22:42:44.960602+0000","last_active":"2024-03-18T22:42:44.960602+0000","last_peered":"2024-03-18T22:42:44.960602+0000","last_clean":"2024-03-18T22:42:44.960602+0000","last_became_active":"2024-03-18T22:42:44.951902+0000","last_became_peered":"2024-03-18T22:42:44.951902+0000","last_unstale":"2024-03-18T22:42:44.960602+0000","last_undegraded":"2024-03-18T22:42:44.960602+0000","last_fullsized":"2024-03-18T22:42:44.960602+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:32:07.038253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,7],"acting":[11,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947336+0000","last_change":"2024-03-18T22:40:41.807533+0000","last_active":"2024-03-18T22:42:44.947336+0000","last_peered":"2024-03-18T22:42:44.947336+0000","last_clean":"2024-03-18T22:42:44.947336+0000","last_became_active":"2024-03-18T22:40:41.807219+0000","last_became_peered":"2024-03-18T22:40:41.807219+0000","last_unstale":"2024-03-18T22:42:44.947336+0000","last_undegraded":"2024-03-18T22:42:44.947336+0000","last_fullsized":"2024-03-18T22:42:44.947336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:15:24.214367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957534+0000","last_change":"2024-03-18T22:40:45.839670+0000","last_active":"2024-03-18T22:42:44.957534+0000","last_peered":"2024-03-18T22:42:44.957534+0000","last_clean":"2024-03-18T22:42:44.957534+0000","last_became_active":"2024-03-18T22:40:45.839263+0000","last_became_peered":"2024-03-18T22:40:45.839263+0000","last_unstale":"2024-03-18T22:42:44.957534+0000","last_undegraded":"2024-03-18T22:42:44.957534+0000","last_fullsized":"2024-03-18T22:42:44.957534+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:01:48.991959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'6","reported_seq":152,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:43:00.986309+0000","last_change":"2024-03-18T22:40:41.209617+0000","last_active":"2024-03-18T22:43:00.986309+0000","last_peered":"2024-03-18T22:43:00.986309+0000","last_clean":"2024-03-18T22:43:00.986309+0000","last_became_active":"2024-03-18T22:40:41.209311+0000","last_became_peered":"2024-03-18T22:40:41.209311+0000","last_unstale":"2024-03-18T22:43:00.986309+0000","last_undegraded":"2024-03-18T22:43:00.986309+0000","last_fullsized":"2024-03-18T22:43:00.986309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:44:06.019870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1664,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948110+0000","last_change":"2024-03-18T22:40:41.808210+0000","last_active":"2024-03-18T22:42:44.948110+0000","last_peered":"2024-03-18T22:42:44.948110+0000","last_clean":"2024-03-18T22:42:44.948110+0000","last_became_active":"2024-03-18T22:40:41.807898+0000","last_became_peered":"2024-03-18T22:40:41.807898+0000","last_unstale":"2024-03-18T22:42:44.948110+0000","last_undegraded":"2024-03-18T22:42:44.948110+0000","last_fullsized":"2024-03-18T22:42:44.948110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:20.401165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.954699+0000","last_change":"2024-03-18T22:40:45.841905+0000","last_active":"2024-03-18T22:42:44.954699+0000","last_peered":"2024-03-18T22:42:44.954699+0000","last_clean":"2024-03-18T22:42:44.954699+0000","last_became_active":"2024-03-18T22:40:45.841464+0000","last_became_peered":"2024-03-18T22:40:45.841464+0000","last_unstale":"2024-03-18T22:42:44.954699+0000","last_undegraded":"2024-03-18T22:42:44.954699+0000","last_fullsized":"2024-03-18T22:42:44.954699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:46:30.435788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945748+0000","last_change":"2024-03-18T22:40:41.216919+0000","last_active":"2024-03-18T22:42:44.945748+0000","last_peered":"2024-03-18T22:42:44.945748+0000","last_clean":"2024-03-18T22:42:44.945748+0000","last_became_active":"2024-03-18T22:40:41.216608+0000","last_became_peered":"2024-03-18T22:40:41.216608+0000","last_unstale":"2024-03-18T22:42:44.945748+0000","last_undegraded":"2024-03-18T22:42:44.945748+0000","last_fullsized":"2024-03-18T22:42:44.945748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:10:42.230268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947883+0000","last_change":"2024-03-18T22:40:41.813879+0000","last_active":"2024-03-18T22:42:44.947883+0000","last_peered":"2024-03-18T22:42:44.947883+0000","last_clean":"2024-03-18T22:42:44.947883+0000","last_became_active":"2024-03-18T22:40:41.813584+0000","last_became_peered":"2024-03-18T22:40:41.813584+0000","last_unstale":"2024-03-18T22:42:44.947883+0000","last_undegraded":"2024-03-18T22:42:44.947883+0000","last_fullsized":"2024-03-18T22:42:44.947883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:20:44.808087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945203+0000","last_change":"2024-03-18T22:40:45.838033+0000","last_active":"2024-03-18T22:42:44.945203+0000","last_peered":"2024-03-18T22:42:44.945203+0000","last_clean":"2024-03-18T22:42:44.945203+0000","last_became_active":"2024-03-18T22:40:45.837530+0000","last_became_peered":"2024-03-18T22:40:45.837530+0000","last_unstale":"2024-03-18T22:42:44.945203+0000","last_undegraded":"2024-03-18T22:42:44.945203+0000","last_fullsized":"2024-03-18T22:42:44.945203+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:44.817547+0000","last_clean_scrub_stamp":"2024-03-18T22:40:44.817547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:43:43.444631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.961111+0000","last_change":"2024-03-18T22:42:44.960540+0000","last_active":"2024-03-18T22:42:44.961111+0000","last_peered":"2024-03-18T22:42:44.961111+0000","last_clean":"2024-03-18T22:42:44.961111+0000","last_became_active":"2024-03-18T22:42:44.952010+0000","last_became_peered":"2024-03-18T22:42:44.952010+0000","last_unstale":"2024-03-18T22:42:44.961111+0000","last_undegraded":"2024-03-18T22:42:44.961111+0000","last_fullsized":"2024-03-18T22:42:44.961111+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:40.189859+0000","last_clean_scrub_stamp":"2024-03-18T22:40:40.189859+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:46:48.257820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948217+0000","last_change":"2024-03-18T22:40:41.808959+0000","last_active":"2024-03-18T22:42:44.948217+0000","last_peered":"2024-03-18T22:42:44.948217+0000","last_clean":"2024-03-18T22:42:44.948217+0000","last_became_active":"2024-03-18T22:40:41.808671+0000","last_became_peered":"2024-03-18T22:40:41.808671+0000","last_unstale":"2024-03-18T22:42:44.948217+0000","last_undegraded":"2024-03-18T22:42:44.948217+0000","last_fullsized":"2024-03-18T22:42:44.948217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:37.115318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947087+0000","last_change":"2024-03-18T22:40:41.814240+0000","last_active":"2024-03-18T22:42:44.947087+0000","last_peered":"2024-03-18T22:42:44.947087+0000","last_clean":"2024-03-18T22:42:44.947087+0000","last_became_active":"2024-03-18T22:40:41.813925+0000","last_became_peered":"2024-03-18T22:40:41.813925+0000","last_unstale":"2024-03-18T22:42:44.947087+0000","last_undegraded":"2024-03-18T22:42:44.947087+0000","last_fullsized":"2024-03-18T22:42:44.947087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:07:38.999803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949347+0000","last_change":"2024-03-18T22:40:41.813381+0000","last_active":"2024-03-18T22:42:44.949347+0000","last_peered":"2024-03-18T22:42:44.949347+0000","last_clean":"2024-03-18T22:42:44.949347+0000","last_became_active":"2024-03-18T22:40:41.812927+0000","last_became_peered":"2024-03-18T22:40:41.812927+0000","last_unstale":"2024-03-18T22:42:44.949347+0000","last_undegraded":"2024-03-18T22:42:44.949347+0000","last_fullsized":"2024-03-18T22:42:44.949347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:23:09.178485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.949543+0000","last_change":"2024-03-18T22:40:41.812513+0000","last_active":"2024-03-18T22:42:44.949543+0000","last_peered":"2024-03-18T22:42:44.949543+0000","last_clean":"2024-03-18T22:42:44.949543+0000","last_became_active":"2024-03-18T22:40:41.811882+0000","last_became_peered":"2024-03-18T22:40:41.811882+0000","last_unstale":"2024-03-18T22:42:44.949543+0000","last_undegraded":"2024-03-18T22:42:44.949543+0000","last_fullsized":"2024-03-18T22:42:44.949543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:45.596911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.955001+0000","last_change":"2024-03-18T22:40:41.814960+0000","last_active":"2024-03-18T22:42:44.955001+0000","last_peered":"2024-03-18T22:42:44.955001+0000","last_clean":"2024-03-18T22:42:44.955001+0000","last_became_active":"2024-03-18T22:40:41.814620+0000","last_became_peered":"2024-03-18T22:40:41.814620+0000","last_unstale":"2024-03-18T22:42:44.955001+0000","last_undegraded":"2024-03-18T22:42:44.955001+0000","last_fullsized":"2024-03-18T22:42:44.955001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:20:42.205885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.948057+0000","last_change":"2024-03-18T22:41:44.334913+0000","last_active":"2024-03-18T22:42:44.948057+0000","last_peered":"2024-03-18T22:42:44.948057+0000","last_clean":"2024-03-18T22:42:44.948057+0000","last_became_active":"2024-03-18T22:41:44.334554+0000","last_became_peered":"2024-03-18T22:41:44.334554+0000","last_unstale":"2024-03-18T22:42:44.948057+0000","last_undegraded":"2024-03-18T22:42:44.948057+0000","last_fullsized":"2024-03-18T22:42:44.948057+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:00:17.369015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947401+0000","last_change":"2024-03-18T22:40:41.808718+0000","last_active":"2024-03-18T22:42:44.947401+0000","last_peered":"2024-03-18T22:42:44.947401+0000","last_clean":"2024-03-18T22:42:44.947401+0000","last_became_active":"2024-03-18T22:40:41.808379+0000","last_became_peered":"2024-03-18T22:40:41.808379+0000","last_unstale":"2024-03-18T22:42:44.947401+0000","last_undegraded":"2024-03-18T22:42:44.947401+0000","last_fullsized":"2024-03-18T22:42:44.947401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:42:26.637531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.950358+0000","last_change":"2024-03-18T22:40:41.818046+0000","last_active":"2024-03-18T22:42:44.950358+0000","last_peered":"2024-03-18T22:42:44.950358+0000","last_clean":"2024-03-18T22:42:44.950358+0000","last_became_active":"2024-03-18T22:40:41.817698+0000","last_became_peered":"2024-03-18T22:40:41.817698+0000","last_unstale":"2024-03-18T22:42:44.950358+0000","last_undegraded":"2024-03-18T22:42:44.950358+0000","last_fullsized":"2024-03-18T22:42:44.950358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:22.967673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951447+0000","last_change":"2024-03-18T22:40:41.819337+0000","last_active":"2024-03-18T22:42:44.951447+0000","last_peered":"2024-03-18T22:42:44.951447+0000","last_clean":"2024-03-18T22:42:44.951447+0000","last_became_active":"2024-03-18T22:40:41.818987+0000","last_became_peered":"2024-03-18T22:40:41.818987+0000","last_unstale":"2024-03-18T22:42:44.951447+0000","last_undegraded":"2024-03-18T22:42:44.951447+0000","last_fullsized":"2024-03-18T22:42:44.951447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:07:04.529454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.945317+0000","last_change":"2024-03-18T22:40:41.813262+0000","last_active":"2024-03-18T22:42:44.945317+0000","last_peered":"2024-03-18T22:42:44.945317+0000","last_clean":"2024-03-18T22:42:44.945317+0000","last_became_active":"2024-03-18T22:40:41.812956+0000","last_became_peered":"2024-03-18T22:40:41.812956+0000","last_unstale":"2024-03-18T22:42:44.945317+0000","last_undegraded":"2024-03-18T22:42:44.945317+0000","last_fullsized":"2024-03-18T22:42:44.945317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:19.940097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.951429+0000","last_change":"2024-03-18T22:40:41.811897+0000","last_active":"2024-03-18T22:42:44.951429+0000","last_peered":"2024-03-18T22:42:44.951429+0000","last_clean":"2024-03-18T22:42:44.951429+0000","last_became_active":"2024-03-18T22:40:41.811575+0000","last_became_peered":"2024-03-18T22:40:41.811575+0000","last_unstale":"2024-03-18T22:42:44.951429+0000","last_undegraded":"2024-03-18T22:42:44.951429+0000","last_fullsized":"2024-03-18T22:42:44.951429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:45:43.321057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.957689+0000","last_change":"2024-03-18T22:40:41.811757+0000","last_active":"2024-03-18T22:42:44.957689+0000","last_peered":"2024-03-18T22:42:44.957689+0000","last_clean":"2024-03-18T22:42:44.957689+0000","last_became_active":"2024-03-18T22:40:41.811405+0000","last_became_peered":"2024-03-18T22:40:41.811405+0000","last_unstale":"2024-03-18T22:42:44.957689+0000","last_undegraded":"2024-03-18T22:42:44.957689+0000","last_fullsized":"2024-03-18T22:42:44.957689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:39.224604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947749+0000","last_change":"2024-03-18T22:40:41.809987+0000","last_active":"2024-03-18T22:42:44.947749+0000","last_peered":"2024-03-18T22:42:44.947749+0000","last_clean":"2024-03-18T22:42:44.947749+0000","last_became_active":"2024-03-18T22:40:41.809544+0000","last_became_peered":"2024-03-18T22:40:41.809544+0000","last_unstale":"2024-03-18T22:42:44.947749+0000","last_undegraded":"2024-03-18T22:42:44.947749+0000","last_fullsized":"2024-03-18T22:42:44.947749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:23.800949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T22:42:44.947201+0000","last_change":"2024-03-18T22:40:41.810963+0000","last_active":"2024-03-18T22:42:44.947201+0000","last_peered":"2024-03-18T22:42:44.947201+0000","last_clean":"2024-03-18T22:42:44.947201+0000","last_became_active":"2024-03-18T22:40:41.810641+0000","last_became_peered":"2024-03-18T22:40:41.810641+0000","last_unstale":"2024-03-18T22:42:44.947201+0000","last_undegraded":"2024-03-18T22:42:44.947201+0000","last_fullsized":"2024-03-18T22:42:44.947201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T22:40:41.192083+0000","last_clean_scrub_stamp":"2024-03-18T22:40:41.192083+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:37:48.872961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74094,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1151,"num_read_kb":686,"num_write":268,"num_write_kb":163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":222282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":234,"ondisk_log_size":234,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383721,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":522642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37056,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708096,"statfs":{"total":95995035648,"available":95957090304,"internally_reserved":0,"allocated":1761280,"data_stored":1172442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808883,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":524526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":520808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28660,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716492,"statfs":{"total":95995035648,"available":95965687808,"internally_reserved":0,"allocated":1683456,"data_stored":1119992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":516960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36452,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708700,"statfs":{"total":95995035648,"available":95957708800,"internally_reserved":0,"allocated":1142784,"data_stored":582202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":517884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1146880,"data_stored":582692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":515642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36384,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708768,"statfs":{"total":95995035648,"available":95957778432,"internally_reserved":0,"allocated":1073152,"data_stored":521034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28660,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716492,"statfs":{"total":95995035648,"available":95965687808,"internally_reserved":0,"allocated":1683456,"data_stored":1111874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":66990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":67480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T22:43:08.645 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:43:08.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.782+0000 7f005899e640 1 -- 172.21.15.40:0/798905183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0054102a50 msgr2=0x7f0054102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:08.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.782+0000 7f005899e640 1 --2- 172.21.15.40:0/798905183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0054102a50 0x7f0054102ed0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f003c009900 tx=0x7f003c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:43:08.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.783+0000 7f005899e640 1 -- 172.21.15.40:0/798905183 shutdown_connections 2024-03-18T22:43:08.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.783+0000 7f005899e640 1 --2- 172.21.15.40:0/798905183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0054103410 0x7f0054109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.783+0000 7f005899e640 1 --2- 172.21.15.40:0/798905183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0054102a50 0x7f0054102ed0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.783+0000 7f005899e640 1 --2- 172.21.15.40:0/798905183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0054101860 0x7f0054101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.783+0000 7f005899e640 1 -- 172.21.15.40:0/798905183 >> 172.21.15.40:0/798905183 conn(0x7f00540fcdd0 msgr2=0x7f00540ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:08.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.783+0000 7f005899e640 1 -- 172.21.15.40:0/798905183 shutdown_connections 2024-03-18T22:43:08.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.783+0000 7f005899e640 1 -- 172.21.15.40:0/798905183 wait complete. 2024-03-18T22:43:08.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.783+0000 7f005899e640 1 Processor -- start 2024-03-18T22:43:08.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.783+0000 7f005899e640 1 -- start start 2024-03-18T22:43:08.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.783+0000 7f005899e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0054101860 0x7f0054197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:08.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.783+0000 7f005899e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0054102a50 0x7f0054198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:08.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.783+0000 7f005899e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0054103410 0x7f005419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:08.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.783+0000 7f005899e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f005410d370 con 0x7f0054103410 2024-03-18T22:43:08.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.783+0000 7f005899e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f005410d1f0 con 0x7f0054102a50 2024-03-18T22:43:08.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.783+0000 7f005899e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f005410d4f0 con 0x7f0054101860 2024-03-18T22:43:08.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f005259c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0054101860 0x7f0054197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:08.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f005259c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0054101860 0x7f0054197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:32956/0 (socket says 172.21.15.40:32956) 2024-03-18T22:43:08.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f005259c640 1 -- 172.21.15.40:0/866615065 learned_addr learned my addr 172.21.15.40:0/866615065 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:08.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f004bfff640 1 --2- 172.21.15.40:0/866615065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0054102a50 0x7f0054198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:08.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f0052d9d640 1 --2- 172.21.15.40:0/866615065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0054103410 0x7f005419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:08.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f005259c640 1 -- 172.21.15.40:0/866615065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0054102a50 msgr2=0x7f0054198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:08.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f005259c640 1 --2- 172.21.15.40:0/866615065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0054102a50 0x7f0054198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f005259c640 1 -- 172.21.15.40:0/866615065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0054103410 msgr2=0x7f005419f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:08.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f005259c640 1 --2- 172.21.15.40:0/866615065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0054103410 0x7f005419f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f005259c640 1 -- 172.21.15.40:0/866615065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005419f710 con 0x7f0054101860 2024-03-18T22:43:08.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f004bfff640 1 --2- 172.21.15.40:0/866615065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0054102a50 0x7f0054198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:08.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f005259c640 1 --2- 172.21.15.40:0/866615065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0054101860 0x7f0054197af0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f004400e9c0 tx=0x7f004400edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:08.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f004b7fe640 1 -- 172.21.15.40:0/866615065 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004400ccc0 con 0x7f0054101860 2024-03-18T22:43:08.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f004b7fe640 1 -- 172.21.15.40:0/866615065 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00440035f0 con 0x7f0054101860 2024-03-18T22:43:08.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f005899e640 1 -- 172.21.15.40:0/866615065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f005419fa00 con 0x7f0054101860 2024-03-18T22:43:08.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f004b7fe640 1 -- 172.21.15.40:0/866615065 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004400ce60 con 0x7f0054101860 2024-03-18T22:43:08.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.784+0000 7f005899e640 1 -- 172.21.15.40:0/866615065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f005419ff20 con 0x7f0054101860 2024-03-18T22:43:08.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.785+0000 7f005899e640 1 -- 172.21.15.40:0/866615065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0054108ca0 con 0x7f0054101860 2024-03-18T22:43:08.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.787+0000 7f004b7fe640 1 -- 172.21.15.40:0/866615065 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f00440037a0 con 0x7f0054101860 2024-03-18T22:43:08.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.787+0000 7f004b7fe640 1 --2- 172.21.15.40:0/866615065 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00400a2800 0x7f00400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:08.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.787+0000 7f004bfff640 1 --2- 172.21.15.40:0/866615065 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00400a2800 0x7f00400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:08.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.787+0000 7f004b7fe640 1 -- 172.21.15.40:0/866615065 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f00440cf2e0 con 0x7f0054101860 2024-03-18T22:43:08.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.788+0000 7f004bfff640 1 --2- 172.21.15.40:0/866615065 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00400a2800 0x7f00400a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f003c009230 tx=0x7f003c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:08.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.794+0000 7f004b7fe640 1 -- 172.21.15.40:0/866615065 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f004409c020 con 0x7f0054101860 2024-03-18T22:43:08.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.979+0000 7f005899e640 1 -- 172.21.15.40:0/866615065 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f005402d220 con 0x7f0054101860 2024-03-18T22:43:08.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.981+0000 7f004b7fe640 1 -- 172.21.15.40:0/866615065 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24908 (secure 0 0 0) 0x7f004409b920 con 0x7f0054101860 2024-03-18T22:43:08.982 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:43:08.982 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":93,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:42:44.941873+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:43:08.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.983+0000 7f005899e640 1 -- 172.21.15.40:0/866615065 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00400a2800 msgr2=0x7f00400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:08.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.983+0000 7f005899e640 1 --2- 172.21.15.40:0/866615065 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00400a2800 0x7f00400a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f003c009230 tx=0x7f003c005fd0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.983+0000 7f005899e640 1 -- 172.21.15.40:0/866615065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0054101860 msgr2=0x7f0054197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:08.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.983+0000 7f005899e640 1 --2- 172.21.15.40:0/866615065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0054101860 0x7f0054197af0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f004400e9c0 tx=0x7f004400edc0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.983+0000 7f005899e640 1 -- 172.21.15.40:0/866615065 shutdown_connections 2024-03-18T22:43:08.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.983+0000 7f005899e640 1 --2- 172.21.15.40:0/866615065 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00400a2800 0x7f00400a4cc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.983+0000 7f005899e640 1 --2- 172.21.15.40:0/866615065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0054103410 0x7f005419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.983+0000 7f005899e640 1 --2- 172.21.15.40:0/866615065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0054102a50 0x7f0054198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.983+0000 7f005899e640 1 --2- 172.21.15.40:0/866615065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0054101860 0x7f0054197af0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:08.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.983+0000 7f005899e640 1 -- 172.21.15.40:0/866615065 >> 172.21.15.40:0/866615065 conn(0x7f00540fcdd0 msgr2=0x7f0054105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:08.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.983+0000 7f005899e640 1 -- 172.21.15.40:0/866615065 shutdown_connections 2024-03-18T22:43:08.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:08.983+0000 7f005899e640 1 -- 172.21.15.40:0/866615065 wait complete. 2024-03-18T22:43:08.995 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T22:36:07.060691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T22:40:39.925147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T22:40:40.773757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T22:40:44.068183+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T22:43:08.995 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T22:43:09.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.102+0000 7fae22f22640 1 -- 172.21.15.40:0/4167932061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae1c100720 msgr2=0x7fae1c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.102+0000 7fae22f22640 1 --2- 172.21.15.40:0/4167932061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae1c100720 0x7fae1c100b40 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fae14009ca0 tx=0x7fae1402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.102+0000 7fae22f22640 1 -- 172.21.15.40:0/4167932061 shutdown_connections 2024-03-18T22:43:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.102+0000 7fae22f22640 1 --2- 172.21.15.40:0/4167932061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae1c1022d0 0x7fae1c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.102+0000 7fae22f22640 1 --2- 172.21.15.40:0/4167932061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae1c101910 0x7fae1c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.102+0000 7fae22f22640 1 --2- 172.21.15.40:0/4167932061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae1c100720 0x7fae1c100b40 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.102+0000 7fae22f22640 1 -- 172.21.15.40:0/4167932061 >> 172.21.15.40:0/4167932061 conn(0x7fae1c0fbcb0 msgr2=0x7fae1c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.103+0000 7fae22f22640 1 -- 172.21.15.40:0/4167932061 shutdown_connections 2024-03-18T22:43:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.103+0000 7fae22f22640 1 -- 172.21.15.40:0/4167932061 wait complete. 2024-03-18T22:43:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.103+0000 7fae22f22640 1 Processor -- start 2024-03-18T22:43:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.103+0000 7fae22f22640 1 -- start start 2024-03-18T22:43:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.103+0000 7fae22f22640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae1c100720 0x7fae1c197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.103+0000 7fae22f22640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae1c101910 0x7fae1c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.103+0000 7fae22f22640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae1c1022d0 0x7fae1c19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.103+0000 7fae22f22640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c10c220 con 0x7fae1c101910 2024-03-18T22:43:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.103+0000 7fae22f22640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c10c0a0 con 0x7fae1c1022d0 2024-03-18T22:43:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.103+0000 7fae22f22640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fae1c10c3a0 con 0x7fae1c100720 2024-03-18T22:43:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.103+0000 7fae1bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae1c101910 0x7fae1c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.103+0000 7fae20cbe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae1c100720 0x7fae1c197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.103+0000 7fae1bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae1c101910 0x7fae1c198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39554/0 (socket says 172.21.15.40:39554) 2024-03-18T22:43:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.103+0000 7fae1bfff640 1 -- 172.21.15.40:0/1608956450 learned_addr learned my addr 172.21.15.40:0/1608956450 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.103+0000 7fae214bf640 1 --2- 172.21.15.40:0/1608956450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae1c1022d0 0x7fae1c19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.104+0000 7fae1bfff640 1 -- 172.21.15.40:0/1608956450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae1c100720 msgr2=0x7fae1c197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.104+0000 7fae1bfff640 1 --2- 172.21.15.40:0/1608956450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae1c100720 0x7fae1c197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.104+0000 7fae1bfff640 1 -- 172.21.15.40:0/1608956450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae1c1022d0 msgr2=0x7fae1c19f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:09.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.104+0000 7fae1bfff640 1 --2- 172.21.15.40:0/1608956450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae1c1022d0 0x7fae1c19f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.104+0000 7fae1bfff640 1 -- 172.21.15.40:0/1608956450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae1c19f710 con 0x7fae1c101910 2024-03-18T22:43:09.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.104+0000 7fae20cbe640 1 --2- 172.21.15.40:0/1608956450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae1c100720 0x7fae1c197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:09.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.104+0000 7fae1bfff640 1 --2- 172.21.15.40:0/1608956450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae1c101910 0x7fae1c198080 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fae0c00e9f0 tx=0x7fae0c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:09.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.104+0000 7fae19ffb640 1 -- 172.21.15.40:0/1608956450 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c00ccc0 con 0x7fae1c101910 2024-03-18T22:43:09.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.104+0000 7fae19ffb640 1 -- 172.21.15.40:0/1608956450 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae0c0108e0 con 0x7fae1c101910 2024-03-18T22:43:09.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.104+0000 7fae22f22640 1 -- 172.21.15.40:0/1608956450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae1c19fa00 con 0x7fae1c101910 2024-03-18T22:43:09.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.104+0000 7fae19ffb640 1 -- 172.21.15.40:0/1608956450 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c0187c0 con 0x7fae1c101910 2024-03-18T22:43:09.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.104+0000 7fae22f22640 1 -- 172.21.15.40:0/1608956450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae1c19ffa0 con 0x7fae1c101910 2024-03-18T22:43:09.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.105+0000 7fae22f22640 1 -- 172.21.15.40:0/1608956450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae1c1a01d0 con 0x7fae1c101910 2024-03-18T22:43:09.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.106+0000 7fae19ffb640 1 -- 172.21.15.40:0/1608956450 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fae0c010a90 con 0x7fae1c101910 2024-03-18T22:43:09.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.107+0000 7fae19ffb640 1 --2- 172.21.15.40:0/1608956450 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fae080a2880 0x7fae080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:09.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.107+0000 7fae19ffb640 1 -- 172.21.15.40:0/1608956450 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fae0c0cf650 con 0x7fae1c101910 2024-03-18T22:43:09.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.107+0000 7fae20cbe640 1 --2- 172.21.15.40:0/1608956450 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fae080a2880 0x7fae080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:09.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.107+0000 7fae20cbe640 1 --2- 172.21.15.40:0/1608956450 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fae080a2880 0x7fae080a4d40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fae14032380 tx=0x7fae14000f40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:09.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.110+0000 7fae19ffb640 1 -- 172.21.15.40:0/1608956450 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fae0c09c300 con 0x7fae1c101910 2024-03-18T22:43:09.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.291+0000 7fae22f22640 1 -- 172.21.15.40:0/1608956450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fae1c1a03a0 con 0x7fae1c101910 2024-03-18T22:43:09.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.291+0000 7fae19ffb640 1 -- 172.21.15.40:0/1608956450 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fae0c09bc00 con 0x7fae1c101910 2024-03-18T22:43:09.292 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-18T22:43:09.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.293+0000 7fae22f22640 1 -- 172.21.15.40:0/1608956450 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fae080a2880 msgr2=0x7fae080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:09.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.293+0000 7fae22f22640 1 --2- 172.21.15.40:0/1608956450 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fae080a2880 0x7fae080a4d40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fae14032380 tx=0x7fae14000f40 comp rx=0 tx=0).stop 2024-03-18T22:43:09.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.293+0000 7fae22f22640 1 -- 172.21.15.40:0/1608956450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae1c101910 msgr2=0x7fae1c198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:09.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.293+0000 7fae22f22640 1 --2- 172.21.15.40:0/1608956450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae1c101910 0x7fae1c198080 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fae0c00e9f0 tx=0x7fae0c00edf0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.293+0000 7fae22f22640 1 -- 172.21.15.40:0/1608956450 shutdown_connections 2024-03-18T22:43:09.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.293+0000 7fae22f22640 1 --2- 172.21.15.40:0/1608956450 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fae080a2880 0x7fae080a4d40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.293+0000 7fae22f22640 1 --2- 172.21.15.40:0/1608956450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae1c1022d0 0x7fae1c19f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.293+0000 7fae22f22640 1 --2- 172.21.15.40:0/1608956450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae1c101910 0x7fae1c198080 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.293+0000 7fae22f22640 1 --2- 172.21.15.40:0/1608956450 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fae1c100720 0x7fae1c197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.293+0000 7fae22f22640 1 -- 172.21.15.40:0/1608956450 >> 172.21.15.40:0/1608956450 conn(0x7fae1c0fbcb0 msgr2=0x7fae1c068a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:09.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.293+0000 7fae22f22640 1 -- 172.21.15.40:0/1608956450 shutdown_connections 2024-03-18T22:43:09.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.293+0000 7fae22f22640 1 -- 172.21.15.40:0/1608956450 wait complete. 2024-03-18T22:43:09.303 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:09 smithi040 ceph-mon[62643]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:43:09.303 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:09 smithi040 ceph-mon[62643]: from='client.15573 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:43:09.303 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:09 smithi040 ceph-mon[62643]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:43:09.304 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:09 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/866615065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:43:09.305 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T22:43:09.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:09 smithi113 ceph-mon[66775]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:43:09.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:09 smithi113 ceph-mon[66775]: from='client.15573 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:43:09.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:09 smithi113 ceph-mon[66775]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:43:09.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:09 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/866615065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:43:09.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:09 smithi103 ceph-mon[66377]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:43:09.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:09 smithi103 ceph-mon[66377]: from='client.15573 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:43:09.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:09 smithi103 ceph-mon[66377]: from='client.15579 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T22:43:09.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:09 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/866615065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:43:09.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.410+0000 7fdaf040b640 1 -- 172.21.15.40:0/233684412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdae8101830 msgr2=0x7fdae8101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:09.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.410+0000 7fdaf040b640 1 --2- 172.21.15.40:0/233684412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdae8101830 0x7fdae8101c50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdae4009900 tx=0x7fdae402bc30 comp rx=0 tx=0).stop 2024-03-18T22:43:09.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.410+0000 7fdaf040b640 1 -- 172.21.15.40:0/233684412 shutdown_connections 2024-03-18T22:43:09.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.410+0000 7fdaf040b640 1 --2- 172.21.15.40:0/233684412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdae81033e0 0x7fdae8109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.410+0000 7fdaf040b640 1 --2- 172.21.15.40:0/233684412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdae8102a20 0x7fdae8102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.410+0000 7fdaf040b640 1 --2- 172.21.15.40:0/233684412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdae8101830 0x7fdae8101c50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.410+0000 7fdaf040b640 1 -- 172.21.15.40:0/233684412 >> 172.21.15.40:0/233684412 conn(0x7fdae80fcdc0 msgr2=0x7fdae80ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:09.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.410+0000 7fdaf040b640 1 -- 172.21.15.40:0/233684412 shutdown_connections 2024-03-18T22:43:09.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.410+0000 7fdaf040b640 1 -- 172.21.15.40:0/233684412 wait complete. 2024-03-18T22:43:09.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaf040b640 1 Processor -- start 2024-03-18T22:43:09.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaf040b640 1 -- start start 2024-03-18T22:43:09.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaf040b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdae8101830 0x7fdae8197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:09.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaf040b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdae8102a20 0x7fdae8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:09.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaf040b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdae81033e0 0x7fdae819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:09.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaf040b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdae810d330 con 0x7fdae8101830 2024-03-18T22:43:09.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaf040b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdae810d1b0 con 0x7fdae81033e0 2024-03-18T22:43:09.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaf040b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdae810d4b0 con 0x7fdae8102a20 2024-03-18T22:43:09.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaee1a7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdae8101830 0x7fdae8197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:09.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaee1a7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdae8101830 0x7fdae8197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39578/0 (socket says 172.21.15.40:39578) 2024-03-18T22:43:09.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaee1a7640 1 -- 172.21.15.40:0/3480574914 learned_addr learned my addr 172.21.15.40:0/3480574914 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:09.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaed9a6640 1 --2- 172.21.15.40:0/3480574914 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdae8102a20 0x7fdae8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:09.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaee9a8640 1 --2- 172.21.15.40:0/3480574914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdae81033e0 0x7fdae819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:09.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaee1a7640 1 -- 172.21.15.40:0/3480574914 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdae8102a20 msgr2=0x7fdae8197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:09.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaee1a7640 1 --2- 172.21.15.40:0/3480574914 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdae8102a20 0x7fdae8197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaee1a7640 1 -- 172.21.15.40:0/3480574914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdae81033e0 msgr2=0x7fdae819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:09.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaee1a7640 1 --2- 172.21.15.40:0/3480574914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdae81033e0 0x7fdae819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaee1a7640 1 -- 172.21.15.40:0/3480574914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae819f6d0 con 0x7fdae8101830 2024-03-18T22:43:09.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.411+0000 7fdaee1a7640 1 --2- 172.21.15.40:0/3480574914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdae8101830 0x7fdae8197ab0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fdae4000c00 tx=0x7fdae4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:09.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.412+0000 7fdadf7fe640 1 -- 172.21.15.40:0/3480574914 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae4043020 con 0x7fdae8101830 2024-03-18T22:43:09.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.412+0000 7fdadf7fe640 1 -- 172.21.15.40:0/3480574914 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdae4004350 con 0x7fdae8101830 2024-03-18T22:43:09.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.412+0000 7fdaf040b640 1 -- 172.21.15.40:0/3480574914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae819f960 con 0x7fdae8101830 2024-03-18T22:43:09.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.412+0000 7fdadf7fe640 1 -- 172.21.15.40:0/3480574914 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae4003b80 con 0x7fdae8101830 2024-03-18T22:43:09.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.412+0000 7fdaf040b640 1 -- 172.21.15.40:0/3480574914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae819feb0 con 0x7fdae8101830 2024-03-18T22:43:09.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.413+0000 7fdaf040b640 1 -- 172.21.15.40:0/3480574914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdae8068a60 con 0x7fdae8101830 2024-03-18T22:43:09.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.414+0000 7fdadf7fe640 1 -- 172.21.15.40:0/3480574914 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdae4046020 con 0x7fdae8101830 2024-03-18T22:43:09.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.414+0000 7fdadf7fe640 1 --2- 172.21.15.40:0/3480574914 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdad40a2800 0x7fdad40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:09.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.414+0000 7fdaed9a6640 1 --2- 172.21.15.40:0/3480574914 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdad40a2800 0x7fdad40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:09.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.414+0000 7fdadf7fe640 1 -- 172.21.15.40:0/3480574914 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdae40f1330 con 0x7fdae8101830 2024-03-18T22:43:09.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.415+0000 7fdaed9a6640 1 --2- 172.21.15.40:0/3480574914 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdad40a2800 0x7fdad40a4cc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fdae8198fc0 tx=0x7fdad8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:09.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.418+0000 7fdadf7fe640 1 -- 172.21.15.40:0/3480574914 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdae404d020 con 0x7fdae8101830 2024-03-18T22:43:09.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.600+0000 7fdaf040b640 1 -- 172.21.15.40:0/3480574914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdae802d1f0 con 0x7fdae8101830 2024-03-18T22:43:09.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.600+0000 7fdadf7fe640 1 -- 172.21.15.40:0/3480574914 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fdae40bdfe0 con 0x7fdae8101830 2024-03-18T22:43:09.601 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:43:09.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.602+0000 7fdaf040b640 1 -- 172.21.15.40:0/3480574914 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdad40a2800 msgr2=0x7fdad40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:09.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.602+0000 7fdaf040b640 1 --2- 172.21.15.40:0/3480574914 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdad40a2800 0x7fdad40a4cc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fdae8198fc0 tx=0x7fdad8005f90 comp rx=0 tx=0).stop 2024-03-18T22:43:09.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.602+0000 7fdaf040b640 1 -- 172.21.15.40:0/3480574914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdae8101830 msgr2=0x7fdae8197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:09.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.602+0000 7fdaf040b640 1 --2- 172.21.15.40:0/3480574914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdae8101830 0x7fdae8197ab0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fdae4000c00 tx=0x7fdae4005dc0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.602+0000 7fdaf040b640 1 -- 172.21.15.40:0/3480574914 shutdown_connections 2024-03-18T22:43:09.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.602+0000 7fdaf040b640 1 --2- 172.21.15.40:0/3480574914 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdad40a2800 0x7fdad40a4cc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.602+0000 7fdaf040b640 1 --2- 172.21.15.40:0/3480574914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdae81033e0 0x7fdae819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.602+0000 7fdaf040b640 1 --2- 172.21.15.40:0/3480574914 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdae8102a20 0x7fdae8197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.602+0000 7fdaf040b640 1 --2- 172.21.15.40:0/3480574914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdae8101830 0x7fdae8197ab0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.602+0000 7fdaf040b640 1 -- 172.21.15.40:0/3480574914 >> 172.21.15.40:0/3480574914 conn(0x7fdae80fcdc0 msgr2=0x7fdae8105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:09.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.603+0000 7fdaf040b640 1 -- 172.21.15.40:0/3480574914 shutdown_connections 2024-03-18T22:43:09.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.603+0000 7fdaf040b640 1 -- 172.21.15.40:0/3480574914 wait complete. 2024-03-18T22:43:09.614 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T22:43:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.719+0000 7f25bdac6640 1 -- 172.21.15.40:0/733818240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25b81022f0 msgr2=0x7f25b8108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.719+0000 7f25bdac6640 1 --2- 172.21.15.40:0/733818240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25b81022f0 0x7f25b8108b80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009ca0 tx=0x7f25b002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.719+0000 7f25bdac6640 1 -- 172.21.15.40:0/733818240 shutdown_connections 2024-03-18T22:43:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.719+0000 7f25bdac6640 1 --2- 172.21.15.40:0/733818240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25b81022f0 0x7f25b8108b80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.719+0000 7f25bdac6640 1 --2- 172.21.15.40:0/733818240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b8101930 0x7f25b8101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.719+0000 7f25bdac6640 1 --2- 172.21.15.40:0/733818240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b8100740 0x7f25b8100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.719+0000 7f25bdac6640 1 -- 172.21.15.40:0/733818240 >> 172.21.15.40:0/733818240 conn(0x7f25b80fbcd0 msgr2=0x7f25b80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.719+0000 7f25bdac6640 1 -- 172.21.15.40:0/733818240 shutdown_connections 2024-03-18T22:43:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.719+0000 7f25bdac6640 1 -- 172.21.15.40:0/733818240 wait complete. 2024-03-18T22:43:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.719+0000 7f25bdac6640 1 Processor -- start 2024-03-18T22:43:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25bdac6640 1 -- start start 2024-03-18T22:43:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25bdac6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25b8100740 0x7f25b8197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25bdac6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b8101930 0x7f25b8197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25bdac6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81022f0 0x7f25b819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25bdac6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25b810c250 con 0x7f25b81022f0 2024-03-18T22:43:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25bdac6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f25b810c0d0 con 0x7f25b8101930 2024-03-18T22:43:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25bdac6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f25b810c3d0 con 0x7f25b8100740 2024-03-18T22:43:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25b7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81022f0 0x7f25b819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25b77fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25b8100740 0x7f25b8197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25b7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81022f0 0x7f25b819efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39608/0 (socket says 172.21.15.40:39608) 2024-03-18T22:43:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25b7fff640 1 -- 172.21.15.40:0/2339291981 learned_addr learned my addr 172.21.15.40:0/2339291981 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25b6ffd640 1 --2- 172.21.15.40:0/2339291981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b8101930 0x7f25b8197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25b7fff640 1 -- 172.21.15.40:0/2339291981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25b8100740 msgr2=0x7f25b8197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:09.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25b7fff640 1 --2- 172.21.15.40:0/2339291981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25b8100740 0x7f25b8197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25b7fff640 1 -- 172.21.15.40:0/2339291981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b8101930 msgr2=0x7f25b8197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:09.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25b7fff640 1 --2- 172.21.15.40:0/2339291981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b8101930 0x7f25b8197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25b7fff640 1 -- 172.21.15.40:0/2339291981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b819f620 con 0x7f25b81022f0 2024-03-18T22:43:09.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.720+0000 7f25b7fff640 1 --2- 172.21.15.40:0/2339291981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81022f0 0x7f25b819efb0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f25b0032380 tx=0x7f25b0032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:09.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.721+0000 7f25b4ff9640 1 -- 172.21.15.40:0/2339291981 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b0041020 con 0x7f25b81022f0 2024-03-18T22:43:09.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.721+0000 7f25bdac6640 1 -- 172.21.15.40:0/2339291981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b819f8b0 con 0x7f25b81022f0 2024-03-18T22:43:09.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.721+0000 7f25b4ff9640 1 -- 172.21.15.40:0/2339291981 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25b0003e20 con 0x7f25b81022f0 2024-03-18T22:43:09.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.721+0000 7f25b4ff9640 1 -- 172.21.15.40:0/2339291981 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b0039040 con 0x7f25b81022f0 2024-03-18T22:43:09.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.721+0000 7f25bdac6640 1 -- 172.21.15.40:0/2339291981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b819fda0 con 0x7f25b81022f0 2024-03-18T22:43:09.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.722+0000 7f25bdac6640 1 -- 172.21.15.40:0/2339291981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25b81a0000 con 0x7f25b81022f0 2024-03-18T22:43:09.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.723+0000 7f25b4ff9640 1 -- 172.21.15.40:0/2339291981 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f25b0044020 con 0x7f25b81022f0 2024-03-18T22:43:09.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.723+0000 7f25b4ff9640 1 --2- 172.21.15.40:0/2339291981 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25a00a2800 0x7f25a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:09.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.723+0000 7f25b4ff9640 1 -- 172.21.15.40:0/2339291981 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f25b00f0ee0 con 0x7f25b81022f0 2024-03-18T22:43:09.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.723+0000 7f25b77fe640 1 --2- 172.21.15.40:0/2339291981 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25a00a2800 0x7f25a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:09.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.724+0000 7f25b77fe640 1 --2- 172.21.15.40:0/2339291981 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25a00a2800 0x7f25a00a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f25b8101790 tx=0x7f25a8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:09.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.728+0000 7f25b4ff9640 1 -- 172.21.15.40:0/2339291981 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f25b004b020 con 0x7f25b81022f0 2024-03-18T22:43:09.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.914+0000 7f25bdac6640 1 -- 172.21.15.40:0/2339291981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f25b81a0390 con 0x7f25b81022f0 2024-03-18T22:43:09.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.915+0000 7f25b4ff9640 1 -- 172.21.15.40:0/2339291981 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f25b00bdb90 con 0x7f25b81022f0 2024-03-18T22:43:09.915 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-18T22:43:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.917+0000 7f25bdac6640 1 -- 172.21.15.40:0/2339291981 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25a00a2800 msgr2=0x7f25a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.917+0000 7f25bdac6640 1 --2- 172.21.15.40:0/2339291981 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25a00a2800 0x7f25a00a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f25b8101790 tx=0x7f25a8005f90 comp rx=0 tx=0).stop 2024-03-18T22:43:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.917+0000 7f25bdac6640 1 -- 172.21.15.40:0/2339291981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81022f0 msgr2=0x7f25b819efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:09.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.917+0000 7f25bdac6640 1 --2- 172.21.15.40:0/2339291981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81022f0 0x7f25b819efb0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f25b0032380 tx=0x7f25b0032820 comp rx=0 tx=0).stop 2024-03-18T22:43:09.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.917+0000 7f25bdac6640 1 -- 172.21.15.40:0/2339291981 shutdown_connections 2024-03-18T22:43:09.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.917+0000 7f25bdac6640 1 --2- 172.21.15.40:0/2339291981 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25a00a2800 0x7f25a00a4cc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.917+0000 7f25bdac6640 1 --2- 172.21.15.40:0/2339291981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25b81022f0 0x7f25b819efb0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.917+0000 7f25bdac6640 1 --2- 172.21.15.40:0/2339291981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25b8101930 0x7f25b8197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.917+0000 7f25bdac6640 1 --2- 172.21.15.40:0/2339291981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25b8100740 0x7f25b8197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:09.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.917+0000 7f25bdac6640 1 -- 172.21.15.40:0/2339291981 >> 172.21.15.40:0/2339291981 conn(0x7f25b80fbcd0 msgr2=0x7f25b8069dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:09.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.917+0000 7f25bdac6640 1 -- 172.21.15.40:0/2339291981 shutdown_connections 2024-03-18T22:43:09.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:09.917+0000 7f25bdac6640 1 -- 172.21.15.40:0/2339291981 wait complete. 2024-03-18T22:43:09.928 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T22:43:10.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.034+0000 7f9e1600b640 1 -- 172.21.15.40:0/46692473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e10101830 msgr2=0x7f9e10101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.034+0000 7f9e1600b640 1 --2- 172.21.15.40:0/46692473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e10101830 0x7f9e10101c50 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009900 tx=0x7f9e0002bc30 comp rx=0 tx=0).stop 2024-03-18T22:43:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.034+0000 7f9e1600b640 1 -- 172.21.15.40:0/46692473 shutdown_connections 2024-03-18T22:43:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.034+0000 7f9e1600b640 1 --2- 172.21.15.40:0/46692473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e101033e0 0x7f9e10109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.034+0000 7f9e1600b640 1 --2- 172.21.15.40:0/46692473 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e10102a20 0x7f9e10102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.034+0000 7f9e1600b640 1 --2- 172.21.15.40:0/46692473 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e10101830 0x7f9e10101c50 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.034+0000 7f9e1600b640 1 -- 172.21.15.40:0/46692473 >> 172.21.15.40:0/46692473 conn(0x7f9e100fcdc0 msgr2=0x7f9e100ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.034+0000 7f9e1600b640 1 -- 172.21.15.40:0/46692473 shutdown_connections 2024-03-18T22:43:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.034+0000 7f9e1600b640 1 -- 172.21.15.40:0/46692473 wait complete. 2024-03-18T22:43:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.034+0000 7f9e1600b640 1 Processor -- start 2024-03-18T22:43:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e1600b640 1 -- start start 2024-03-18T22:43:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e1600b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e10101830 0x7f9e10197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e1600b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e10102a20 0x7f9e10197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e1600b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e101033e0 0x7f9e1019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e1600b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1010d350 con 0x7f9e101033e0 2024-03-18T22:43:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e1600b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1010d1d0 con 0x7f9e10101830 2024-03-18T22:43:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e1600b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1010d4d0 con 0x7f9e10102a20 2024-03-18T22:43:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e0ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e101033e0 0x7f9e1019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e0ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e101033e0 0x7f9e1019f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39626/0 (socket says 172.21.15.40:39626) 2024-03-18T22:43:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e0ffff640 1 -- 172.21.15.40:0/606403657 learned_addr learned my addr 172.21.15.40:0/606403657 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e0effd640 1 --2- 172.21.15.40:0/606403657 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e10102a20 0x7f9e10197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e0f7fe640 1 --2- 172.21.15.40:0/606403657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e10101830 0x7f9e10197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e0ffff640 1 -- 172.21.15.40:0/606403657 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e10102a20 msgr2=0x7f9e10197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e0ffff640 1 --2- 172.21.15.40:0/606403657 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e10102a20 0x7f9e10197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e0ffff640 1 -- 172.21.15.40:0/606403657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e10101830 msgr2=0x7f9e10197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e0ffff640 1 --2- 172.21.15.40:0/606403657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e10101830 0x7f9e10197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e0ffff640 1 -- 172.21.15.40:0/606403657 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1019f6d0 con 0x7f9e101033e0 2024-03-18T22:43:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e0ffff640 1 --2- 172.21.15.40:0/606403657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e101033e0 0x7f9e1019f060 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9e0800eb10 tx=0x7f9e0800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:10.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.035+0000 7f9e0cff9640 1 -- 172.21.15.40:0/606403657 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08004b20 con 0x7f9e101033e0 2024-03-18T22:43:10.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.036+0000 7f9e0cff9640 1 -- 172.21.15.40:0/606403657 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e08004cc0 con 0x7f9e101033e0 2024-03-18T22:43:10.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.036+0000 7f9e1600b640 1 -- 172.21.15.40:0/606403657 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e1019f9c0 con 0x7f9e101033e0 2024-03-18T22:43:10.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.036+0000 7f9e0cff9640 1 -- 172.21.15.40:0/606403657 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08010690 con 0x7f9e101033e0 2024-03-18T22:43:10.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.036+0000 7f9e1600b640 1 -- 172.21.15.40:0/606403657 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e1019fec0 con 0x7f9e101033e0 2024-03-18T22:43:10.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.037+0000 7f9e1600b640 1 -- 172.21.15.40:0/606403657 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e10108c90 con 0x7f9e101033e0 2024-03-18T22:43:10.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.038+0000 7f9e0cff9640 1 -- 172.21.15.40:0/606403657 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9e08005000 con 0x7f9e101033e0 2024-03-18T22:43:10.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.039+0000 7f9e0cff9640 1 --2- 172.21.15.40:0/606403657 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9df80a2800 0x7f9df80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:10.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.039+0000 7f9e0f7fe640 1 --2- 172.21.15.40:0/606403657 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9df80a2800 0x7f9df80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:10.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.039+0000 7f9e0cff9640 1 -- 172.21.15.40:0/606403657 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9e080cf5e0 con 0x7f9e101033e0 2024-03-18T22:43:10.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.039+0000 7f9e0f7fe640 1 --2- 172.21.15.40:0/606403657 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9df80a2800 0x7f9df80a4cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9e00005ab0 tx=0x7f9e00005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:10.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.043+0000 7f9e0cff9640 1 -- 172.21.15.40:0/606403657 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9e08053740 con 0x7f9e101033e0 2024-03-18T22:43:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:10 smithi103 ceph-mon[66377]: pgmap v366: 129 pgs: 129 active+clean; 649 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-18T22:43:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:10 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1608956450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:43:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:10 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3480574914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:43:10.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:10 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2339291981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:43:10.206 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:10 smithi040 ceph-mon[62643]: pgmap v366: 129 pgs: 129 active+clean; 649 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-18T22:43:10.207 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:10 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1608956450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:43:10.207 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:10 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3480574914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:43:10.207 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:10 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2339291981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:43:10.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.228+0000 7f9e1600b640 1 -- 172.21.15.40:0/606403657 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9e1002d440 con 0x7f9e101033e0 2024-03-18T22:43:10.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.229+0000 7f9e0cff9640 1 -- 172.21.15.40:0/606403657 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f9e0809c290 con 0x7f9e101033e0 2024-03-18T22:43:10.229 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:43:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.231+0000 7f9e1600b640 1 -- 172.21.15.40:0/606403657 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9df80a2800 msgr2=0x7f9df80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.231+0000 7f9e1600b640 1 --2- 172.21.15.40:0/606403657 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9df80a2800 0x7f9df80a4cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9e00005ab0 tx=0x7f9e00005fd0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.231+0000 7f9e1600b640 1 -- 172.21.15.40:0/606403657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e101033e0 msgr2=0x7f9e1019f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.231+0000 7f9e1600b640 1 --2- 172.21.15.40:0/606403657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e101033e0 0x7f9e1019f060 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9e0800eb10 tx=0x7f9e0800c2d0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.231+0000 7f9e1600b640 1 -- 172.21.15.40:0/606403657 shutdown_connections 2024-03-18T22:43:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.231+0000 7f9e1600b640 1 --2- 172.21.15.40:0/606403657 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9df80a2800 0x7f9df80a4cc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.231+0000 7f9e1600b640 1 --2- 172.21.15.40:0/606403657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e101033e0 0x7f9e1019f060 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.231+0000 7f9e1600b640 1 --2- 172.21.15.40:0/606403657 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e10102a20 0x7f9e10197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.231+0000 7f9e1600b640 1 --2- 172.21.15.40:0/606403657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e10101830 0x7f9e10197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.231+0000 7f9e1600b640 1 -- 172.21.15.40:0/606403657 >> 172.21.15.40:0/606403657 conn(0x7f9e100fcdc0 msgr2=0x7f9e10105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.231+0000 7f9e1600b640 1 -- 172.21.15.40:0/606403657 shutdown_connections 2024-03-18T22:43:10.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.231+0000 7f9e1600b640 1 -- 172.21.15.40:0/606403657 wait complete. 2024-03-18T22:43:10.242 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:43:10.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:10 smithi113 ceph-mon[66775]: pgmap v366: 129 pgs: 129 active+clean; 649 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-18T22:43:10.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:10 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1608956450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:43:10.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:10 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3480574914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:43:10.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:10 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2339291981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:43:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.345+0000 7fcb371f4640 1 -- 172.21.15.40:0/4125648391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb30103420 msgr2=0x7fcb30109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.345+0000 7fcb371f4640 1 --2- 172.21.15.40:0/4125648391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb30103420 0x7fcb30109cb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fcb28009ca0 tx=0x7fcb2802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.345+0000 7fcb371f4640 1 -- 172.21.15.40:0/4125648391 shutdown_connections 2024-03-18T22:43:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.345+0000 7fcb371f4640 1 --2- 172.21.15.40:0/4125648391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb30103420 0x7fcb30109cb0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.345+0000 7fcb371f4640 1 --2- 172.21.15.40:0/4125648391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb30102a60 0x7fcb30102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.345+0000 7fcb371f4640 1 --2- 172.21.15.40:0/4125648391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb30101870 0x7fcb30101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.345+0000 7fcb371f4640 1 -- 172.21.15.40:0/4125648391 >> 172.21.15.40:0/4125648391 conn(0x7fcb300fcdc0 msgr2=0x7fcb300ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.345+0000 7fcb371f4640 1 -- 172.21.15.40:0/4125648391 shutdown_connections 2024-03-18T22:43:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.345+0000 7fcb371f4640 1 -- 172.21.15.40:0/4125648391 wait complete. 2024-03-18T22:43:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.345+0000 7fcb371f4640 1 Processor -- start 2024-03-18T22:43:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.345+0000 7fcb371f4640 1 -- start start 2024-03-18T22:43:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb371f4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb30101870 0x7fcb300755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb371f4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb30102a60 0x7fcb30073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb371f4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb30103420 0x7fcb30074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb371f4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3010d380 con 0x7fcb30103420 2024-03-18T22:43:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb371f4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3010d200 con 0x7fcb30102a60 2024-03-18T22:43:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb371f4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3010d500 con 0x7fcb30101870 2024-03-18T22:43:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb35791640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb30103420 0x7fcb30074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb35791640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb30103420 0x7fcb30074150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39644/0 (socket says 172.21.15.40:39644) 2024-03-18T22:43:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb35791640 1 -- 172.21.15.40:0/1503523836 learned_addr learned my addr 172.21.15.40:0/1503523836 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb2ffff640 1 --2- 172.21.15.40:0/1503523836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb30102a60 0x7fcb30073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb34f90640 1 --2- 172.21.15.40:0/1503523836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb30101870 0x7fcb300755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb35791640 1 -- 172.21.15.40:0/1503523836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb30101870 msgr2=0x7fcb300755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb35791640 1 --2- 172.21.15.40:0/1503523836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb30101870 0x7fcb300755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb35791640 1 -- 172.21.15.40:0/1503523836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb30102a60 msgr2=0x7fcb30073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb35791640 1 --2- 172.21.15.40:0/1503523836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb30102a60 0x7fcb30073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb35791640 1 -- 172.21.15.40:0/1503523836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb300749e0 con 0x7fcb30103420 2024-03-18T22:43:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb35791640 1 --2- 172.21.15.40:0/1503523836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb30103420 0x7fcb30074150 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fcb2802c700 tx=0x7fcb280328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb2dffb640 1 -- 172.21.15.40:0/1503523836 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb28041020 con 0x7fcb30103420 2024-03-18T22:43:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb2dffb640 1 -- 172.21.15.40:0/1503523836 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb28003e20 con 0x7fcb30103420 2024-03-18T22:43:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb371f4640 1 -- 172.21.15.40:0/1503523836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb301a1ae0 con 0x7fcb30103420 2024-03-18T22:43:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.346+0000 7fcb2dffb640 1 -- 172.21.15.40:0/1503523836 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb28039040 con 0x7fcb30103420 2024-03-18T22:43:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.347+0000 7fcb371f4640 1 -- 172.21.15.40:0/1503523836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb301a2080 con 0x7fcb30103420 2024-03-18T22:43:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.347+0000 7fcb371f4640 1 -- 172.21.15.40:0/1503523836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb30108c10 con 0x7fcb30103420 2024-03-18T22:43:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.348+0000 7fcb2dffb640 1 -- 172.21.15.40:0/1503523836 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcb28044020 con 0x7fcb30103420 2024-03-18T22:43:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.349+0000 7fcb2dffb640 1 --2- 172.21.15.40:0/1503523836 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcb200a2880 0x7fcb200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.349+0000 7fcb2dffb640 1 -- 172.21.15.40:0/1503523836 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fcb280f0f80 con 0x7fcb30103420 2024-03-18T22:43:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.349+0000 7fcb34f90640 1 --2- 172.21.15.40:0/1503523836 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcb200a2880 0x7fcb200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.350+0000 7fcb34f90640 1 --2- 172.21.15.40:0/1503523836 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcb200a2880 0x7fcb200a4d40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcb240095f0 tx=0x7fcb24005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:10.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.354+0000 7fcb2dffb640 1 -- 172.21.15.40:0/1503523836 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcb2804b020 con 0x7fcb30103420 2024-03-18T22:43:10.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.576+0000 7fcb371f4640 1 -- 172.21.15.40:0/1503523836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb300624c0 con 0x7fcb30103420 2024-03-18T22:43:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.576+0000 7fcb2dffb640 1 -- 172.21.15.40:0/1503523836 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11947 (secure 0 0 0) 0x7fcb280bdc30 con 0x7fcb30103420 2024-03-18T22:43:10.577 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:43:10.577 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:43:03.220450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24388,24401,24473,24481,24509]},"id":1}]} 2024-03-18T22:43:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 28 2024-03-18T22:43:10.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.578+0000 7fcb371f4640 1 -- 172.21.15.40:0/1503523836 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcb200a2880 msgr2=0x7fcb200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:10.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.578+0000 7fcb371f4640 1 --2- 172.21.15.40:0/1503523836 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcb200a2880 0x7fcb200a4d40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcb240095f0 tx=0x7fcb24005f90 comp rx=0 tx=0).stop 2024-03-18T22:43:10.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.578+0000 7fcb371f4640 1 -- 172.21.15.40:0/1503523836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb30103420 msgr2=0x7fcb30074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:10.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.578+0000 7fcb371f4640 1 --2- 172.21.15.40:0/1503523836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb30103420 0x7fcb30074150 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fcb2802c700 tx=0x7fcb280328a0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.578+0000 7fcb371f4640 1 -- 172.21.15.40:0/1503523836 shutdown_connections 2024-03-18T22:43:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.578+0000 7fcb371f4640 1 --2- 172.21.15.40:0/1503523836 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcb200a2880 0x7fcb200a4d40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.578+0000 7fcb371f4640 1 --2- 172.21.15.40:0/1503523836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb30103420 0x7fcb30074150 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.578+0000 7fcb371f4640 1 --2- 172.21.15.40:0/1503523836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb30102a60 0x7fcb30073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.578+0000 7fcb371f4640 1 --2- 172.21.15.40:0/1503523836 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb30101870 0x7fcb300755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.578+0000 7fcb371f4640 1 -- 172.21.15.40:0/1503523836 >> 172.21.15.40:0/1503523836 conn(0x7fcb300fcdc0 msgr2=0x7fcb30105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.578+0000 7fcb371f4640 1 -- 172.21.15.40:0/1503523836 shutdown_connections 2024-03-18T22:43:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.578+0000 7fcb371f4640 1 -- 172.21.15.40:0/1503523836 wait complete. 2024-03-18T22:43:10.590 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:43:10.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.691+0000 7fe06f31b640 1 -- 172.21.15.40:0/2306160396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe068100740 msgr2=0x7fe068100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:10.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.691+0000 7fe06f31b640 1 --2- 172.21.15.40:0/2306160396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe068100740 0x7fe068100b60 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe058009ca0 tx=0x7fe05802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06f31b640 1 -- 172.21.15.40:0/2306160396 shutdown_connections 2024-03-18T22:43:10.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06f31b640 1 --2- 172.21.15.40:0/2306160396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0681022f0 0x7fe068108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06f31b640 1 --2- 172.21.15.40:0/2306160396 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe068101930 0x7fe068101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06f31b640 1 --2- 172.21.15.40:0/2306160396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe068100740 0x7fe068100b60 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06f31b640 1 -- 172.21.15.40:0/2306160396 >> 172.21.15.40:0/2306160396 conn(0x7fe0680fbcb0 msgr2=0x7fe0680fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:10.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06f31b640 1 -- 172.21.15.40:0/2306160396 shutdown_connections 2024-03-18T22:43:10.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06f31b640 1 -- 172.21.15.40:0/2306160396 wait complete. 2024-03-18T22:43:10.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06f31b640 1 Processor -- start 2024-03-18T22:43:10.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06f31b640 1 -- start start 2024-03-18T22:43:10.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06f31b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe068100740 0x7fe068197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06f31b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe068101930 0x7fe068198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06f31b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0681022f0 0x7fe06819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06f31b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe06810c260 con 0x7fe068101930 2024-03-18T22:43:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06f31b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe06810c0e0 con 0x7fe068100740 2024-03-18T22:43:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06f31b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe06810c3e0 con 0x7fe0681022f0 2024-03-18T22:43:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06c8b6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe068101930 0x7fe068198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06c8b6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe068101930 0x7fe068198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39662/0 (socket says 172.21.15.40:39662) 2024-03-18T22:43:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06c8b6640 1 -- 172.21.15.40:0/1067196713 learned_addr learned my addr 172.21.15.40:0/1067196713 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.692+0000 7fe06d0b7640 1 --2- 172.21.15.40:0/1067196713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe068100740 0x7fe068197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.693+0000 7fe06d8b8640 1 --2- 172.21.15.40:0/1067196713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0681022f0 0x7fe06819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.693+0000 7fe06c8b6640 1 -- 172.21.15.40:0/1067196713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0681022f0 msgr2=0x7fe06819f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.693+0000 7fe06c8b6640 1 --2- 172.21.15.40:0/1067196713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0681022f0 0x7fe06819f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.693+0000 7fe06c8b6640 1 -- 172.21.15.40:0/1067196713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe068100740 msgr2=0x7fe068197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.693+0000 7fe06c8b6640 1 --2- 172.21.15.40:0/1067196713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe068100740 0x7fe068197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.693+0000 7fe06c8b6640 1 -- 172.21.15.40:0/1067196713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe06819f700 con 0x7fe068101930 2024-03-18T22:43:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.693+0000 7fe06c8b6640 1 --2- 172.21.15.40:0/1067196713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe068101930 0x7fe068198020 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00e9f0 tx=0x7fe05c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.693+0000 7fe0667fc640 1 -- 172.21.15.40:0/1067196713 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05c00cd80 con 0x7fe068101930 2024-03-18T22:43:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.693+0000 7fe0667fc640 1 -- 172.21.15.40:0/1067196713 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe05c0054d0 con 0x7fe068101930 2024-03-18T22:43:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.693+0000 7fe06f31b640 1 -- 172.21.15.40:0/1067196713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe06819f9f0 con 0x7fe068101930 2024-03-18T22:43:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.693+0000 7fe0667fc640 1 -- 172.21.15.40:0/1067196713 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05c010490 con 0x7fe068101930 2024-03-18T22:43:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.693+0000 7fe06f31b640 1 -- 172.21.15.40:0/1067196713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe06819ff40 con 0x7fe068101930 2024-03-18T22:43:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.694+0000 7fe06f31b640 1 -- 172.21.15.40:0/1067196713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0681a01d0 con 0x7fe068101930 2024-03-18T22:43:10.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.695+0000 7fe0667fc640 1 -- 172.21.15.40:0/1067196713 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe05c005000 con 0x7fe068101930 2024-03-18T22:43:10.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.696+0000 7fe0667fc640 1 --2- 172.21.15.40:0/1067196713 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe0500a2880 0x7fe0500a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:10.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.696+0000 7fe06d0b7640 1 --2- 172.21.15.40:0/1067196713 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe0500a2880 0x7fe0500a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:10.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.696+0000 7fe0667fc640 1 -- 172.21.15.40:0/1067196713 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe05c0cf3e0 con 0x7fe068101930 2024-03-18T22:43:10.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.696+0000 7fe06d0b7640 1 --2- 172.21.15.40:0/1067196713 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe0500a2880 0x7fe0500a4d40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe058032380 tx=0x7fe058000f40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:10.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.699+0000 7fe0667fc640 1 -- 172.21.15.40:0/1067196713 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe05c053740 con 0x7fe068101930 2024-03-18T22:43:10.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.883+0000 7fe06f31b640 1 -- 172.21.15.40:0/1067196713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe0681a03a0 con 0x7fe068101930 2024-03-18T22:43:10.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.885+0000 7fe0667fc640 1 -- 172.21.15.40:0/1067196713 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24908 (secure 0 0 0) 0x7fe05c0052f0 con 0x7fe068101930 2024-03-18T22:43:10.886 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:43:10.886 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":93,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:42:44.941873+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:43:10.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.887+0000 7fe06f31b640 1 -- 172.21.15.40:0/1067196713 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe0500a2880 msgr2=0x7fe0500a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:10.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.887+0000 7fe06f31b640 1 --2- 172.21.15.40:0/1067196713 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe0500a2880 0x7fe0500a4d40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe058032380 tx=0x7fe058000f40 comp rx=0 tx=0).stop 2024-03-18T22:43:10.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.887+0000 7fe06f31b640 1 -- 172.21.15.40:0/1067196713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe068101930 msgr2=0x7fe068198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:10.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.887+0000 7fe06f31b640 1 --2- 172.21.15.40:0/1067196713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe068101930 0x7fe068198020 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00e9f0 tx=0x7fe05c00edf0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.887+0000 7fe06f31b640 1 -- 172.21.15.40:0/1067196713 shutdown_connections 2024-03-18T22:43:10.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.887+0000 7fe06f31b640 1 --2- 172.21.15.40:0/1067196713 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe0500a2880 0x7fe0500a4d40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.887+0000 7fe06f31b640 1 --2- 172.21.15.40:0/1067196713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe0681022f0 0x7fe06819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.887+0000 7fe06f31b640 1 --2- 172.21.15.40:0/1067196713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe068101930 0x7fe068198020 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.887+0000 7fe06f31b640 1 --2- 172.21.15.40:0/1067196713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe068100740 0x7fe068197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:10.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.887+0000 7fe06f31b640 1 -- 172.21.15.40:0/1067196713 >> 172.21.15.40:0/1067196713 conn(0x7fe0680fbcb0 msgr2=0x7fe068068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:10.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.887+0000 7fe06f31b640 1 -- 172.21.15.40:0/1067196713 shutdown_connections 2024-03-18T22:43:10.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:10.887+0000 7fe06f31b640 1 -- 172.21.15.40:0/1067196713 wait complete. 2024-03-18T22:43:10.899 DEBUG:tasks.fwd_scrub:Yielding 2024-03-18T22:43:10.899 INFO:teuthology.run_tasks:Running task workunit... 2024-03-18T22:43:10.912 INFO:tasks.workunit:Pulling workunits from ref e48b1f6c0608b2615e1890cc46286c3877d6e910 2024-03-18T22:43:10.912 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-18T22:43:10.913 INFO:tasks.workunit:timeout=3h 2024-03-18T22:43:10.913 INFO:tasks.workunit:cleanup=True 2024-03-18T22:43:10.913 DEBUG:teuthology.orchestra.run.smithi040:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-18T22:43:10.913 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-18T22:43:10.913 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:43:10.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:43:10.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:43:10.930 INFO:teuthology.orchestra.run.smithi040.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-18T22:43:10.930 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-18T22:43:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 77h/119d Inode: 1 Links: 2 2024-03-18T22:43:10.930 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T22:43:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-18T22:43:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-03-18 22:41:10.208710249 +0000 2024-03-18T22:43:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-03-18 22:41:10.208710249 +0000 2024-03-18T22:43:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-03-18 22:43:02.841312167 +0000 2024-03-18T22:43:10.931 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2024-03-18T22:43:10.931 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-18T22:43:10.931 DEBUG:teuthology.orchestra.run.smithi040:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-18T22:43:11.003 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e48b1f6c0608b2615e1890cc46286c3877d6e910 2024-03-18T22:43:11.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.023+0000 7f253c84b640 1 -- 172.21.15.40:0/1457405063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2538102ad0 msgr2=0x7f2538102f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.023+0000 7f253c84b640 1 --2- 172.21.15.40:0/1457405063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2538102ad0 0x7f2538102f50 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f252c009930 tx=0x7f252c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:43:11.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.023+0000 7f253c84b640 1 -- 172.21.15.40:0/1457405063 shutdown_connections 2024-03-18T22:43:11.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.023+0000 7f253c84b640 1 --2- 172.21.15.40:0/1457405063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2538103490 0x7f2538109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.023+0000 7f253c84b640 1 --2- 172.21.15.40:0/1457405063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2538102ad0 0x7f2538102f50 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.023+0000 7f253c84b640 1 --2- 172.21.15.40:0/1457405063 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25380ffb00 0x7f25380fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.023+0000 7f253c84b640 1 -- 172.21.15.40:0/1457405063 >> 172.21.15.40:0/1457405063 conn(0x7f25380fb450 msgr2=0x7f25380fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:11.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.023+0000 7f253c84b640 1 -- 172.21.15.40:0/1457405063 shutdown_connections 2024-03-18T22:43:11.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.023+0000 7f253c84b640 1 -- 172.21.15.40:0/1457405063 wait complete. 2024-03-18T22:43:11.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.024+0000 7f253c84b640 1 Processor -- start 2024-03-18T22:43:11.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.024+0000 7f253c84b640 1 -- start start 2024-03-18T22:43:11.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.024+0000 7f253c84b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25380ffb00 0x7f2538197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.024+0000 7f253c84b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2538102ad0 0x7f2538198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.024+0000 7f253c84b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2538103490 0x7f253819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.024+0000 7f253c84b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f253810d390 con 0x7f2538103490 2024-03-18T22:43:11.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.024+0000 7f253c84b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f253810d210 con 0x7f2538102ad0 2024-03-18T22:43:11.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.024+0000 7f253c84b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f253810d510 con 0x7f25380ffb00 2024-03-18T22:43:11.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.024+0000 7f2536d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2538103490 0x7f253819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.024+0000 7f2536d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2538103490 0x7f253819f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39680/0 (socket says 172.21.15.40:39680) 2024-03-18T22:43:11.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.024+0000 7f2536d9d640 1 -- 172.21.15.40:0/1424116163 learned_addr learned my addr 172.21.15.40:0/1424116163 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:11.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.024+0000 7f253659c640 1 --2- 172.21.15.40:0/1424116163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25380ffb00 0x7f2538197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.024+0000 7f2535d9b640 1 --2- 172.21.15.40:0/1424116163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2538102ad0 0x7f2538198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.025+0000 7f253659c640 1 -- 172.21.15.40:0/1424116163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2538102ad0 msgr2=0x7f2538198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.025+0000 7f253659c640 1 --2- 172.21.15.40:0/1424116163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2538102ad0 0x7f2538198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.025+0000 7f253659c640 1 -- 172.21.15.40:0/1424116163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2538103490 msgr2=0x7f253819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.025+0000 7f253659c640 1 --2- 172.21.15.40:0/1424116163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2538103490 0x7f253819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.025+0000 7f253659c640 1 -- 172.21.15.40:0/1424116163 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253819f6e0 con 0x7f25380ffb00 2024-03-18T22:43:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.025+0000 7f2536d9d640 1 --2- 172.21.15.40:0/1424116163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2538103490 0x7f253819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.025+0000 7f253659c640 1 --2- 172.21.15.40:0/1424116163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25380ffb00 0x7f2538197ac0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f252800e9c0 tx=0x7f252800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.025+0000 7f25277fe640 1 -- 172.21.15.40:0/1424116163 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252800ccc0 con 0x7f25380ffb00 2024-03-18T22:43:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.025+0000 7f25277fe640 1 -- 172.21.15.40:0/1424116163 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25280035f0 con 0x7f25380ffb00 2024-03-18T22:43:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.025+0000 7f25277fe640 1 -- 172.21.15.40:0/1424116163 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252800ce60 con 0x7f25380ffb00 2024-03-18T22:43:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.025+0000 7f253c84b640 1 -- 172.21.15.40:0/1424116163 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253819f9d0 con 0x7f25380ffb00 2024-03-18T22:43:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.025+0000 7f253c84b640 1 -- 172.21.15.40:0/1424116163 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f253819fea0 con 0x7f25380ffb00 2024-03-18T22:43:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.027+0000 7f25277fe640 1 -- 172.21.15.40:0/1424116163 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f25280037a0 con 0x7f25380ffb00 2024-03-18T22:43:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.027+0000 7f253f436740 1 Processor -- start 2024-03-18T22:43:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f253f436740 1 -- start start 2024-03-18T22:43:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f253f436740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f3383a0 0x563a7f3387c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f253f436740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f338d90 0x563a7f33d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f253f436740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 0x563a7f340390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f253f436740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563a7f29d160 con 0x563a7f338d90 2024-03-18T22:43:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f253f436740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563a7f245e10 con 0x563a7f3383a0 2024-03-18T22:43:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f253f436740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563a7f299200 con 0x563a7f33df00 2024-03-18T22:43:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f253659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f338d90 0x563a7f33d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f2535d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 0x563a7f340390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f253659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f338d90 0x563a7f33d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39686/0 (socket says 172.21.15.40:39686) 2024-03-18T22:43:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f2536d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f3383a0 0x563a7f3387c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f253659c640 1 -- 172.21.15.40:0/2898999353 learned_addr learned my addr 172.21.15.40:0/2898999353 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f2535d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 0x563a7f340390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33074/0 (socket says 172.21.15.40:33074) 2024-03-18T22:43:11.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f25277fe640 1 --2- 172.21.15.40:0/1424116163 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25200a2800 0x7f25200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f2535d9b640 1 -- 172.21.15.40:0/2898999353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f3383a0 msgr2=0x563a7f3387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f2536d9d640 1 --2- 172.21.15.40:0/1424116163 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25200a2800 0x7f25200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f2535d9b640 1 --2- 172.21.15.40:0/2898999353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f3383a0 0x563a7f3387c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f2535d9b640 1 -- 172.21.15.40:0/2898999353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f338d90 msgr2=0x563a7f33d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f2535d9b640 1 --2- 172.21.15.40:0/2898999353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f338d90 0x563a7f33d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f2535d9b640 1 -- 172.21.15.40:0/2898999353 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a7f0797e0 con 0x563a7f33df00 2024-03-18T22:43:11.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.028+0000 7f25277fe640 1 -- 172.21.15.40:0/1424116163 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f25280cf230 con 0x7f25380ffb00 2024-03-18T22:43:11.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.029+0000 7f2535d9b640 1 --2- 172.21.15.40:0/2898999353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 0x563a7f340390 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f252c009230 tx=0x7f252c009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4cba2aab71df132 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:11.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.029+0000 7f2536d9d640 1 --2- 172.21.15.40:0/1424116163 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25200a2800 0x7f25200a4cc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x563a7f3395f0 tx=0x7f2530004ab0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:11.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.029+0000 7f2524ff9640 1 -- 172.21.15.40:0/2898999353 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c045020 con 0x563a7f33df00 2024-03-18T22:43:11.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.029+0000 7f2524ff9640 1 -- 172.21.15.40:0/2898999353 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f252c004040 con 0x563a7f33df00 2024-03-18T22:43:11.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.029+0000 7f2524ff9640 1 -- 172.21.15.40:0/2898999353 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c005560 con 0x563a7f33df00 2024-03-18T22:43:11.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 -- 172.21.15.40:0/2898999353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 msgr2=0x563a7f340390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 --2- 172.21.15.40:0/2898999353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 0x563a7f340390 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f252c009230 tx=0x7f252c009560 comp rx=0 tx=0).stop 2024-03-18T22:43:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 -- 172.21.15.40:0/2898999353 shutdown_connections 2024-03-18T22:43:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 --2- 172.21.15.40:0/2898999353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 0x563a7f340390 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 --2- 172.21.15.40:0/2898999353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f338d90 0x563a7f33d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 --2- 172.21.15.40:0/2898999353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f3383a0 0x563a7f3387c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 -- 172.21.15.40:0/2898999353 >> 172.21.15.40:0/2898999353 conn(0x563a7f333910 msgr2=0x563a7f333d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 -- 172.21.15.40:0/2898999353 shutdown_connections 2024-03-18T22:43:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 -- 172.21.15.40:0/2898999353 wait complete. 2024-03-18T22:43:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 -- start start 2024-03-18T22:43:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 Processor -- start 2024-03-18T22:43:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f3383a0 0x563a7f3427c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f338d90 0x563a7f342d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f33df00 0x563a7f349d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563a7f340e50 con 0x563a7f33df00 2024-03-18T22:43:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563a7f2591a0 con 0x563a7f338d90 2024-03-18T22:43:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563a7f340fd0 con 0x563a7f3383a0 2024-03-18T22:43:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253f436740 10 client.0 _ll_register_callbacks cb 0x563a7efb0c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f3383a0 0x563a7f3427c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f2536d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f33df00 0x563a7f349d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f3383a0 0x563a7f3427c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33086/0 (socket says 172.21.15.40:33086) 2024-03-18T22:43:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f253659c640 1 -- 172.21.15.40:0/1884461163 learned_addr learned my addr 172.21.15.40:0/1884461163 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.030+0000 7f2535d9b640 1 --2- 172.21.15.40:0/1884461163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f338d90 0x563a7f342d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.031+0000 7f253659c640 1 -- 172.21.15.40:0/1884461163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f338d90 msgr2=0x563a7f342d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.031+0000 7f253659c640 1 --2- 172.21.15.40:0/1884461163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f338d90 0x563a7f342d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.031+0000 7f253659c640 1 -- 172.21.15.40:0/1884461163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f33df00 msgr2=0x563a7f349d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.031+0000 7f253659c640 1 --2- 172.21.15.40:0/1884461163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f33df00 0x563a7f349d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.031+0000 7f253659c640 1 -- 172.21.15.40:0/1884461163 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a7f0d9f70 con 0x563a7f3383a0 2024-03-18T22:43:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.031+0000 7f2535d9b640 1 --2- 172.21.15.40:0/1884461163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f338d90 0x563a7f342d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.031+0000 7f253659c640 1 --2- 172.21.15.40:0/1884461163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f3383a0 0x563a7f3427c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f25280cf4d0 tx=0x7f25280045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4082e91fa323ca60 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.031+0000 7f251dffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:43:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.031+0000 7f251dffb640 1 -- 172.21.15.40:0/1884461163 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2528003b10 con 0x563a7f3383a0 2024-03-18T22:43:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.031+0000 7f253f436740 1 -- 172.21.15.40:0/1884461163 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f252c004040 con 0x563a7f3383a0 2024-03-18T22:43:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.031+0000 7f251dffb640 1 -- 172.21.15.40:0/1884461163 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f25280974d0 con 0x563a7f3383a0 2024-03-18T22:43:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.031+0000 7f251dffb640 1 -- 172.21.15.40:0/1884461163 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f252809be80 con 0x563a7f3383a0 2024-03-18T22:43:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.031+0000 7f253f436740 10 client.25507 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.031+0000 7f253f436740 1 -- 172.21.15.40:0/1884461163 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563a7ef71170 con 0x563a7f3383a0 2024-03-18T22:43:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.031+0000 7f251dffb640 1 -- 172.21.15.40:0/1884461163 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f252c004040 con 0x563a7f3383a0 2024-03-18T22:43:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.032+0000 7f251dffb640 1 -- 172.21.15.40:0/1884461163 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2528097040 con 0x563a7f3383a0 2024-03-18T22:43:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.032+0000 7f253f436740 10 client.25507 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.032+0000 7f253f436740 10 client.25507 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:43:11.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.032+0000 7f253f436740 20 client.25507 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:11.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.032+0000 7f253f436740 1 --2- 172.21.15.40:0/1884461163 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563a7f34e620 0x563a7f350ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.032+0000 7f253f436740 4 client.25507 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:43:11.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.032+0000 7f253f436740 1 -- 172.21.15.40:0/1884461163 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a7f2aa6a0 con 0x563a7f34e620 2024-03-18T22:43:11.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.032+0000 7f2535d9b640 1 --2- 172.21.15.40:0/1884461163 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563a7f34e620 0x563a7f350ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.032+0000 7f2535d9b640 1 --2- 172.21.15.40:0/1884461163 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563a7f34e620 0x563a7f350ab0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42a930dad8e939a2 server_cookie=c10701e2e200c975 in_seq=0 out_seq=0 2024-03-18T22:43:11.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.032+0000 7f251dffb640 10 client.25507 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:11.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.033+0000 7f251dffb640 1 -- 172.21.15.40:0/1884461163 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563a7f2aa6a0 con 0x563a7f34e620 2024-03-18T22:43:11.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.033+0000 7f251dffb640 10 client.25507 handle_command_reply: tid=0 2024-03-18T22:43:11.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.033+0000 7f253f436740 1 client.25507 shutdown 2024-03-18T22:43:11.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.033+0000 7f253f436740 1 -- 172.21.15.40:0/1884461163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f3383a0 msgr2=0x563a7f3427c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.033+0000 7f253f436740 1 --2- 172.21.15.40:0/1884461163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f3383a0 0x563a7f3427c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f25280cf4d0 tx=0x7f25280045e0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.033+0000 7f253f436740 1 -- 172.21.15.40:0/1884461163 shutdown_connections 2024-03-18T22:43:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.033+0000 7f253f436740 1 --2- 172.21.15.40:0/1884461163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f33df00 0x563a7f349d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.033+0000 7f253f436740 1 --2- 172.21.15.40:0/1884461163 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563a7f34e620 0x563a7f350ab0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.033+0000 7f253f436740 1 --2- 172.21.15.40:0/1884461163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f338d90 0x563a7f342d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.033+0000 7f253f436740 1 --2- 172.21.15.40:0/1884461163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f3383a0 0x563a7f3427c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.033+0000 7f251dffb640 0 client.25507 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.033+0000 7f253f436740 1 -- 172.21.15.40:0/1884461163 >> 172.21.15.40:0/1884461163 conn(0x563a7f333910 msgr2=0x563a7f332b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.033+0000 7f253f436740 1 -- 172.21.15.40:0/1884461163 shutdown_connections 2024-03-18T22:43:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.033+0000 7f253f436740 1 -- 172.21.15.40:0/1884461163 wait complete. 2024-03-18T22:43:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.034+0000 7f253f436740 20 client.25507 trim_cache size 0 max 16384 2024-03-18T22:43:11.043 INFO:tasks.workunit.client.0.smithi040.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-18T22:43:11.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.051+0000 7f253f436740 1 Processor -- start 2024-03-18T22:43:11.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.051+0000 7f253f436740 1 -- start start 2024-03-18T22:43:11.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.051+0000 7f253f436740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f3383a0 0x563a7f342ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.051+0000 7f253f436740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f338d90 0x563a7f349d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.051+0000 7f253f436740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 0x563a7f3322a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.051+0000 7f253f436740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25381a8200 con 0x563a7f338d90 2024-03-18T22:43:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.051+0000 7f253f436740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f25381a9d40 con 0x563a7f3383a0 2024-03-18T22:43:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.051+0000 7f253f436740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f25381a88d0 con 0x563a7f33df00 2024-03-18T22:43:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f2535d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f338d90 0x563a7f349d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f253659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f3383a0 0x563a7f342ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f253659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f3383a0 0x563a7f342ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:54296/0 (socket says 172.21.15.40:54296) 2024-03-18T22:43:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f2535d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f338d90 0x563a7f349d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39700/0 (socket says 172.21.15.40:39700) 2024-03-18T22:43:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f253659c640 1 -- 172.21.15.40:0/2627944051 learned_addr learned my addr 172.21.15.40:0/2627944051 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f2536d9d640 1 --2- 172.21.15.40:0/2627944051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 0x563a7f3322a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f253659c640 1 -- 172.21.15.40:0/2627944051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 msgr2=0x563a7f3322a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f253659c640 1 --2- 172.21.15.40:0/2627944051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 0x563a7f3322a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f253659c640 1 -- 172.21.15.40:0/2627944051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f338d90 msgr2=0x563a7f349d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f253659c640 1 --2- 172.21.15.40:0/2627944051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f338d90 0x563a7f349d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f253659c640 1 -- 172.21.15.40:0/2627944051 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a7f3334b0 con 0x563a7f3383a0 2024-03-18T22:43:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f2535d9b640 1 --2- 172.21.15.40:0/2627944051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f338d90 0x563a7f349d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f253659c640 1 --2- 172.21.15.40:0/2627944051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f3383a0 0x563a7f342ae0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f252800c6a0 tx=0x7f2528005e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93d8609975fcdc11 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f251ffff640 1 -- 172.21.15.40:0/2627944051 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2528003b10 con 0x563a7f3383a0 2024-03-18T22:43:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f251ffff640 1 -- 172.21.15.40:0/2627944051 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f252809b780 con 0x563a7f3383a0 2024-03-18T22:43:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.052+0000 7f251ffff640 1 -- 172.21.15.40:0/2627944051 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2528004000 con 0x563a7f3383a0 2024-03-18T22:43:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.053+0000 7f253f436740 1 -- 172.21.15.40:0/2627944051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f3383a0 msgr2=0x563a7f342ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.053+0000 7f253f436740 1 --2- 172.21.15.40:0/2627944051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f3383a0 0x563a7f342ae0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f252800c6a0 tx=0x7f2528005e30 comp rx=0 tx=0).stop 2024-03-18T22:43:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.053+0000 7f253f436740 1 -- 172.21.15.40:0/2627944051 shutdown_connections 2024-03-18T22:43:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.053+0000 7f253f436740 1 --2- 172.21.15.40:0/2627944051 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 0x563a7f3322a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.053+0000 7f253f436740 1 --2- 172.21.15.40:0/2627944051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f338d90 0x563a7f349d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.053+0000 7f253f436740 1 --2- 172.21.15.40:0/2627944051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f3383a0 0x563a7f342ae0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.053+0000 7f253f436740 1 -- 172.21.15.40:0/2627944051 >> 172.21.15.40:0/2627944051 conn(0x563a7f333910 msgr2=0x563a7f3423d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.053+0000 7f253f436740 1 -- 172.21.15.40:0/2627944051 shutdown_connections 2024-03-18T22:43:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.053+0000 7f253f436740 1 -- 172.21.15.40:0/2627944051 wait complete. 2024-03-18T22:43:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.053+0000 7f253f436740 1 -- start start 2024-03-18T22:43:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.053+0000 7f253f436740 1 Processor -- start 2024-03-18T22:43:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253f436740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f3383a0 0x563a7f3476b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253f436740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f338d90 0x563a7f347bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253f436740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 0x563a7f348c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253f436740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25381a9670 con 0x563a7f3383a0 2024-03-18T22:43:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253f436740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f25381a8fa0 con 0x563a7f338d90 2024-03-18T22:43:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253f436740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f25381a61e0 con 0x563a7f33df00 2024-03-18T22:43:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253f436740 10 client.0 _ll_register_callbacks cb 0x7f25280cc820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f3383a0 0x563a7f3476b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f3383a0 0x563a7f3476b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39716/0 (socket says 172.21.15.40:39716) 2024-03-18T22:43:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253659c640 1 -- 172.21.15.40:0/1788303909 learned_addr learned my addr 172.21.15.40:0/1788303909 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f2535d9b640 1 --2- 172.21.15.40:0/1788303909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f338d90 0x563a7f347bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f2536d9d640 1 --2- 172.21.15.40:0/1788303909 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 0x563a7f348c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253659c640 1 -- 172.21.15.40:0/1788303909 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 msgr2=0x563a7f348c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253659c640 1 --2- 172.21.15.40:0/1788303909 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 0x563a7f348c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253659c640 1 -- 172.21.15.40:0/1788303909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f338d90 msgr2=0x563a7f347bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253659c640 1 --2- 172.21.15.40:0/1788303909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f338d90 0x563a7f347bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253659c640 1 -- 172.21.15.40:0/1788303909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a7f334380 con 0x563a7f3383a0 2024-03-18T22:43:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253659c640 1 --2- 172.21.15.40:0/1788303909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f3383a0 0x563a7f3476b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f25280cf4d0 tx=0x7f25280862e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=135e6ca1eae2a259 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f251dffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:43:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f251dffb640 1 -- 172.21.15.40:0/1788303909 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2528097a20 con 0x563a7f3383a0 2024-03-18T22:43:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253f436740 1 -- 172.21.15.40:0/1788303909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f252809b780 con 0x563a7f3383a0 2024-03-18T22:43:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f251dffb640 1 -- 172.21.15.40:0/1788303909 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2528004000 con 0x563a7f3383a0 2024-03-18T22:43:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f251dffb640 1 -- 172.21.15.40:0/1788303909 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2528003b10 con 0x563a7f3383a0 2024-03-18T22:43:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253f436740 10 client.15639 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.054+0000 7f253f436740 1 -- 172.21.15.40:0/1788303909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563a7f3b4a70 con 0x563a7f3383a0 2024-03-18T22:43:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.055+0000 7f251dffb640 1 -- 172.21.15.40:0/1788303909 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f252809b780 con 0x563a7f3383a0 2024-03-18T22:43:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.055+0000 7f251dffb640 1 -- 172.21.15.40:0/1788303909 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2528097040 con 0x563a7f3383a0 2024-03-18T22:43:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.055+0000 7f253f436740 10 client.15639 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.055+0000 7f253f436740 10 client.15639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:43:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.055+0000 7f253f436740 20 client.15639 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.055+0000 7f253f436740 1 --2- 172.21.15.40:0/1788303909 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563a7f3b4f90 0x563a7f3b73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.055+0000 7f253f436740 4 client.15639 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:43:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.055+0000 7f253f436740 1 -- 172.21.15.40:0/1788303909 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563a7f2a49f0 con 0x563a7f3b4f90 2024-03-18T22:43:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.055+0000 7f2535d9b640 1 --2- 172.21.15.40:0/1788303909 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563a7f3b4f90 0x563a7f3b73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.055+0000 7f251dffb640 10 client.15639 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.055+0000 7f2535d9b640 1 --2- 172.21.15.40:0/1788303909 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563a7f3b4f90 0x563a7f3b73b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0839965dd8d5254 server_cookie=8a11a678f14c6325 in_seq=0 out_seq=0 2024-03-18T22:43:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.056+0000 7f251dffb640 1 -- 172.21.15.40:0/1788303909 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563a7f2a49f0 con 0x563a7f3b4f90 2024-03-18T22:43:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.056+0000 7f251dffb640 10 client.15639 handle_command_reply: tid=0 2024-03-18T22:43:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.056+0000 7f253f436740 1 client.15639 shutdown 2024-03-18T22:43:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253f436740 1 -- 172.21.15.40:0/1788303909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f3383a0 msgr2=0x563a7f3476b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253f436740 1 --2- 172.21.15.40:0/1788303909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f3383a0 0x563a7f3476b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f25280cf4d0 tx=0x7f25280862e0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253f436740 1 -- 172.21.15.40:0/1788303909 shutdown_connections 2024-03-18T22:43:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253f436740 1 --2- 172.21.15.40:0/1788303909 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563a7f33df00 0x563a7f348c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253f436740 1 --2- 172.21.15.40:0/1788303909 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563a7f3b4f90 0x563a7f3b73b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253f436740 1 --2- 172.21.15.40:0/1788303909 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563a7f338d90 0x563a7f347bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253f436740 1 --2- 172.21.15.40:0/1788303909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563a7f3383a0 0x563a7f3476b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f251dffb640 0 client.15639 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253f436740 1 -- 172.21.15.40:0/1788303909 >> 172.21.15.40:0/1788303909 conn(0x563a7f333910 msgr2=0x563a7f342bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253f436740 1 -- 172.21.15.40:0/1788303909 shutdown_connections 2024-03-18T22:43:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253f436740 1 -- 172.21.15.40:0/1788303909 wait complete. 2024-03-18T22:43:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253f436740 20 client.15639 trim_cache size 0 max 16384 2024-03-18T22:43:11.064 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:43:11.064 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:43:11.064 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "f75cb960-5553-46d0-b03a-98c03ed7b889", 2024-03-18T22:43:11.064 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:43:11.064 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:43:11.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253c84b640 1 -- 172.21.15.40:0/1424116163 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25200a2800 msgr2=0x7f25200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253c84b640 1 --2- 172.21.15.40:0/1424116163 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25200a2800 0x7f25200a4cc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x563a7f3395f0 tx=0x7f2530004ab0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253c84b640 1 -- 172.21.15.40:0/1424116163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25380ffb00 msgr2=0x7f2538197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253c84b640 1 --2- 172.21.15.40:0/1424116163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25380ffb00 0x7f2538197ac0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f252800e9c0 tx=0x7f252800ee80 comp rx=0 tx=0).stop 2024-03-18T22:43:11.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253c84b640 1 -- 172.21.15.40:0/1424116163 shutdown_connections 2024-03-18T22:43:11.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253c84b640 1 --2- 172.21.15.40:0/1424116163 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25200a2800 0x7f25200a4cc0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253c84b640 1 --2- 172.21.15.40:0/1424116163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2538103490 0x7f253819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253c84b640 1 --2- 172.21.15.40:0/1424116163 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2538102ad0 0x7f2538198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253c84b640 1 --2- 172.21.15.40:0/1424116163 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25380ffb00 0x7f2538197ac0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.057+0000 7f253c84b640 1 -- 172.21.15.40:0/1424116163 >> 172.21.15.40:0/1424116163 conn(0x7f25380fb450 msgr2=0x7f2538105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:11.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.058+0000 7f253c84b640 1 -- 172.21.15.40:0/1424116163 shutdown_connections 2024-03-18T22:43:11.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.058+0000 7f253c84b640 1 -- 172.21.15.40:0/1424116163 wait complete. 2024-03-18T22:43:11.069 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:43:11.069 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:43:11.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.215+0000 7f25758c0640 1 -- 172.21.15.40:0/3475687416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25701022d0 msgr2=0x7f2570108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.215+0000 7f25758c0640 1 --2- 172.21.15.40:0/3475687416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25701022d0 0x7f2570108b60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2568009ca0 tx=0x7f256802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.215+0000 7f25758c0640 1 -- 172.21.15.40:0/3475687416 shutdown_connections 2024-03-18T22:43:11.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.215+0000 7f25758c0640 1 --2- 172.21.15.40:0/3475687416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25701022d0 0x7f2570108b60 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.215+0000 7f25758c0640 1 --2- 172.21.15.40:0/3475687416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2570101910 0x7f2570101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.215+0000 7f25758c0640 1 --2- 172.21.15.40:0/3475687416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2570100720 0x7f2570100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.215+0000 7f25758c0640 1 -- 172.21.15.40:0/3475687416 >> 172.21.15.40:0/3475687416 conn(0x7f25700fbcb0 msgr2=0x7f25700fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:11.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.215+0000 7f25758c0640 1 -- 172.21.15.40:0/3475687416 shutdown_connections 2024-03-18T22:43:11.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.215+0000 7f25758c0640 1 -- 172.21.15.40:0/3475687416 wait complete. 2024-03-18T22:43:11.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.216+0000 7f25758c0640 1 Processor -- start 2024-03-18T22:43:11.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.216+0000 7f25758c0640 1 -- start start 2024-03-18T22:43:11.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.216+0000 7f25758c0640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2570100720 0x7f2570195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.216+0000 7f25758c0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2570101910 0x7f2570195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.216+0000 7f25758c0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25701022d0 0x7f257019cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.216+0000 7f25758c0640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f257010b150 con 0x7f2570101910 2024-03-18T22:43:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.216+0000 7f25758c0640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f257010afd0 con 0x7f2570100720 2024-03-18T22:43:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.216+0000 7f25758c0640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f257010b2d0 con 0x7f25701022d0 2024-03-18T22:43:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.216+0000 7f256e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2570101910 0x7f2570195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.216+0000 7f256e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2570101910 0x7f2570195d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39744/0 (socket says 172.21.15.40:39744) 2024-03-18T22:43:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.216+0000 7f256e7fc640 1 -- 172.21.15.40:0/3278488626 learned_addr learned my addr 172.21.15.40:0/3278488626 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.216+0000 7f256effd640 1 --2- 172.21.15.40:0/3278488626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2570100720 0x7f2570195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.217+0000 7f256f7fe640 1 --2- 172.21.15.40:0/3278488626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25701022d0 0x7f257019cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.217+0000 7f256e7fc640 1 -- 172.21.15.40:0/3278488626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25701022d0 msgr2=0x7f257019cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.217+0000 7f256e7fc640 1 --2- 172.21.15.40:0/3278488626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25701022d0 0x7f257019cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.217+0000 7f256e7fc640 1 -- 172.21.15.40:0/3278488626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2570100720 msgr2=0x7f2570195830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.217+0000 7f256e7fc640 1 --2- 172.21.15.40:0/3278488626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2570100720 0x7f2570195830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.217+0000 7f256e7fc640 1 -- 172.21.15.40:0/3278488626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f257019d450 con 0x7f2570101910 2024-03-18T22:43:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.217+0000 7f256e7fc640 1 --2- 172.21.15.40:0/3278488626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2570101910 0x7f2570195d70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f256400e9f0 tx=0x7f256400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.217+0000 7f25748be640 1 -- 172.21.15.40:0/3278488626 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256400cd80 con 0x7f2570101910 2024-03-18T22:43:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.217+0000 7f25748be640 1 -- 172.21.15.40:0/3278488626 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25640054d0 con 0x7f2570101910 2024-03-18T22:43:11.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.217+0000 7f25758c0640 1 -- 172.21.15.40:0/3278488626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f257019d740 con 0x7f2570101910 2024-03-18T22:43:11.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.217+0000 7f25748be640 1 -- 172.21.15.40:0/3278488626 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564010490 con 0x7f2570101910 2024-03-18T22:43:11.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.217+0000 7f25758c0640 1 -- 172.21.15.40:0/3278488626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f257019dc40 con 0x7f2570101910 2024-03-18T22:43:11.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f25748be640 1 -- 172.21.15.40:0/3278488626 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2564005020 con 0x7f2570101910 2024-03-18T22:43:11.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f2578587740 1 Processor -- start 2024-03-18T22:43:11.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f2578587740 1 -- start start 2024-03-18T22:43:11.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f2578587740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a91718a50 0x557a91718e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f2578587740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a9171df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f2578587740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a9171e5b0 0x557a91720a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f2578587740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557a9167d500 con 0x557a91718a50 2024-03-18T22:43:11.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f2578587740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557a91625be0 con 0x557a91719440 2024-03-18T22:43:11.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f2578587740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557a916795a0 con 0x557a9171e5b0 2024-03-18T22:43:11.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f256f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a91718a50 0x557a91718e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f256f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a91718a50 0x557a91718e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39752/0 (socket says 172.21.15.40:39752) 2024-03-18T22:43:11.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f256f7fe640 1 -- 172.21.15.40:0/2268091661 learned_addr learned my addr 172.21.15.40:0/2268091661 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:11.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f256effd640 1 --2- 172.21.15.40:0/2268091661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a9171df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f256e7fc640 1 --2- 172.21.15.40:0/2268091661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a9171e5b0 0x557a91720a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f25748be640 1 --2- 172.21.15.40:0/3278488626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25580a2800 0x7f25580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f256f7fe640 1 -- 172.21.15.40:0/2268091661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a9171e5b0 msgr2=0x557a91720a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f256f7fe640 1 --2- 172.21.15.40:0/2268091661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a9171e5b0 0x557a91720a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f256f7fe640 1 -- 172.21.15.40:0/2268091661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 msgr2=0x557a9171df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f256f7fe640 1 --2- 172.21.15.40:0/2268091661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a9171df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f256f7fe640 1 -- 172.21.15.40:0/2268091661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a91647000 con 0x557a91718a50 2024-03-18T22:43:11.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f25748be640 1 -- 172.21.15.40:0/3278488626 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f25640cf1d0 con 0x7f2570101910 2024-03-18T22:43:11.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.220+0000 7f256f7fe640 1 --2- 172.21.15.40:0/3278488626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25580a2800 0x7f25580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.221+0000 7f256f7fe640 1 --2- 172.21.15.40:0/2268091661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a91718a50 0x557a91718e70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2568005e30 tx=0x7f2568009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b87160f7561e50 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:11.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.221+0000 7f2555ffb640 1 -- 172.21.15.40:0/2268091661 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2568039040 con 0x557a91718a50 2024-03-18T22:43:11.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.221+0000 7f2555ffb640 1 -- 172.21.15.40:0/2268091661 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2568003c30 con 0x557a91718a50 2024-03-18T22:43:11.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.221+0000 7f2555ffb640 1 -- 172.21.15.40:0/2268091661 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256804a660 con 0x557a91718a50 2024-03-18T22:43:11.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.221+0000 7f2578587740 1 -- 172.21.15.40:0/2268091661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a91718a50 msgr2=0x557a91718e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.221+0000 7f256f7fe640 1 --2- 172.21.15.40:0/3278488626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25580a2800 0x7f25580a4cc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f256804c000 tx=0x7f2568045000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:11.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.221+0000 7f2578587740 1 --2- 172.21.15.40:0/2268091661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a91718a50 0x557a91718e70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2568005e30 tx=0x7f2568009880 comp rx=0 tx=0).stop 2024-03-18T22:43:11.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 1 -- 172.21.15.40:0/2268091661 shutdown_connections 2024-03-18T22:43:11.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 1 --2- 172.21.15.40:0/2268091661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a9171e5b0 0x557a91720a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 1 --2- 172.21.15.40:0/2268091661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a9171df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 1 --2- 172.21.15.40:0/2268091661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a91718a50 0x557a91718e70 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 1 -- 172.21.15.40:0/2268091661 >> 172.21.15.40:0/2268091661 conn(0x557a91713fc0 msgr2=0x557a917143c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:11.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 1 -- 172.21.15.40:0/2268091661 shutdown_connections 2024-03-18T22:43:11.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 1 -- 172.21.15.40:0/2268091661 wait complete. 2024-03-18T22:43:11.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 1 -- start start 2024-03-18T22:43:11.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 1 Processor -- start 2024-03-18T22:43:11.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 0x557a91722ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a91723430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a9172a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557a91721560 con 0x557a9171e5b0 2024-03-18T22:43:11.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557a91639160 con 0x557a91719440 2024-03-18T22:43:11.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557a917216e0 con 0x557a91718a50 2024-03-18T22:43:11.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.222+0000 7f2578587740 10 client.0 _ll_register_callbacks cb 0x557a91675a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:11.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f256effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 0x557a91722ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f256e7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a91723430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f256f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a9172a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f256effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 0x557a91722ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33168/0 (socket says 172.21.15.40:33168) 2024-03-18T22:43:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f256effd640 1 -- 172.21.15.40:0/2074715177 learned_addr learned my addr 172.21.15.40:0/2074715177 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f256effd640 1 -- 172.21.15.40:0/2074715177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 msgr2=0x557a91723430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f256effd640 1 --2- 172.21.15.40:0/2074715177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a91723430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f256effd640 1 -- 172.21.15.40:0/2074715177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 msgr2=0x557a9172a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f256effd640 1 --2- 172.21.15.40:0/2074715177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a9172a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f256effd640 1 -- 172.21.15.40:0/2074715177 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a9145b7d0 con 0x557a91718a50 2024-03-18T22:43:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f256f7fe640 1 --2- 172.21.15.40:0/2074715177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a9172a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f256effd640 1 --2- 172.21.15.40:0/2074715177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 0x557a91722ef0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2560016980 tx=0x7f2560016d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c76497b61cb959a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f254effd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:43:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f254effd640 1 -- 172.21.15.40:0/2074715177 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560004460 con 0x557a91718a50 2024-03-18T22:43:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f254effd640 1 -- 172.21.15.40:0/2074715177 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f25600147a0 con 0x557a91718a50 2024-03-18T22:43:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f2578587740 1 -- 172.21.15.40:0/2074715177 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2568003c30 con 0x557a91718a50 2024-03-18T22:43:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f254effd640 1 -- 172.21.15.40:0/2074715177 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f25600149a0 con 0x557a91718a50 2024-03-18T22:43:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f2578587740 10 client.25525 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.223+0000 7f2578587740 1 -- 172.21.15.40:0/2074715177 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557a913913f0 con 0x557a91718a50 2024-03-18T22:43:11.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.224+0000 7f254effd640 1 -- 172.21.15.40:0/2074715177 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2568003c30 con 0x557a91718a50 2024-03-18T22:43:11.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.224+0000 7f254effd640 1 -- 172.21.15.40:0/2074715177 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2560020400 con 0x557a91718a50 2024-03-18T22:43:11.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.224+0000 7f2578587740 10 client.25525 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:11.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.224+0000 7f2578587740 10 client.25525 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:43:11.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.224+0000 7f2578587740 20 client.25525 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:11.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.224+0000 7f2578587740 1 --2- 172.21.15.40:0/2074715177 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557a9172ee30 0x557a917312d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.224+0000 7f2578587740 4 client.25525 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:43:11.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.224+0000 7f2578587740 1 -- 172.21.15.40:0/2074715177 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a9168aa40 con 0x557a9172ee30 2024-03-18T22:43:11.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.224+0000 7f256e7fc640 1 --2- 172.21.15.40:0/2074715177 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557a9172ee30 0x557a917312d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.224+0000 7f256e7fc640 1 --2- 172.21.15.40:0/2074715177 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557a9172ee30 0x557a917312d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aacc6b90fa25074d server_cookie=24a21449875794 in_seq=0 out_seq=0 2024-03-18T22:43:11.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.224+0000 7f254effd640 10 client.25525 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:11.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.225+0000 7f254effd640 1 -- 172.21.15.40:0/2074715177 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557a9168aa40 con 0x557a9172ee30 2024-03-18T22:43:11.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.225+0000 7f254effd640 10 client.25525 handle_command_reply: tid=0 2024-03-18T22:43:11.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.225+0000 7f2578587740 1 client.25525 shutdown 2024-03-18T22:43:11.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.225+0000 7f2578587740 1 -- 172.21.15.40:0/2074715177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 msgr2=0x557a91722ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.225+0000 7f2578587740 1 --2- 172.21.15.40:0/2074715177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 0x557a91722ef0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2560016980 tx=0x7f2560016d80 comp rx=0 tx=0).stop 2024-03-18T22:43:11.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.225+0000 7f2578587740 1 -- 172.21.15.40:0/2074715177 shutdown_connections 2024-03-18T22:43:11.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.225+0000 7f2578587740 1 --2- 172.21.15.40:0/2074715177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a9172a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.225+0000 7f2578587740 1 --2- 172.21.15.40:0/2074715177 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557a9172ee30 0x557a917312d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.225+0000 7f2578587740 1 --2- 172.21.15.40:0/2074715177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a91723430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.225+0000 7f2578587740 1 --2- 172.21.15.40:0/2074715177 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 0x557a91722ef0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.225+0000 7f254effd640 0 client.25525 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:11.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.225+0000 7f2578587740 1 -- 172.21.15.40:0/2074715177 >> 172.21.15.40:0/2074715177 conn(0x557a91713fc0 msgr2=0x557a917131e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:11.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.225+0000 7f2578587740 1 -- 172.21.15.40:0/2074715177 shutdown_connections 2024-03-18T22:43:11.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.225+0000 7f2578587740 1 -- 172.21.15.40:0/2074715177 wait complete. 2024-03-18T22:43:11.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.225+0000 7f2578587740 20 client.25525 trim_cache size 0 max 16384 2024-03-18T22:43:11.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f2578587740 1 Processor -- start 2024-03-18T22:43:11.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f2578587740 1 -- start start 2024-03-18T22:43:11.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f2578587740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 0x557a91717930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f2578587740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a9172a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f2578587740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a91722510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f2578587740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25701a8340 con 0x557a9171e5b0 2024-03-18T22:43:11.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f2578587740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f25701a9e80 con 0x557a91719440 2024-03-18T22:43:11.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f2578587740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f25701a8a10 con 0x557a91718a50 2024-03-18T22:43:11.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f256f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a91722510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f256f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a91722510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39778/0 (socket says 172.21.15.40:39778) 2024-03-18T22:43:11.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f256f7fe640 1 -- 172.21.15.40:0/432795011 learned_addr learned my addr 172.21.15.40:0/432795011 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:11.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f256effd640 1 --2- 172.21.15.40:0/432795011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 0x557a91717930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f256e7fc640 1 --2- 172.21.15.40:0/432795011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a9172a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f256f7fe640 1 -- 172.21.15.40:0/432795011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 msgr2=0x557a91717930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f256f7fe640 1 --2- 172.21.15.40:0/432795011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 0x557a91717930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f256f7fe640 1 -- 172.21.15.40:0/432795011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 msgr2=0x557a9172a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f256f7fe640 1 --2- 172.21.15.40:0/432795011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a9172a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f256f7fe640 1 -- 172.21.15.40:0/432795011 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a913510f0 con 0x557a9171e5b0 2024-03-18T22:43:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f256f7fe640 1 --2- 172.21.15.40:0/432795011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a91722510 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2568006010 tx=0x7f2568052000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3526f8264e77cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f25557fa640 1 -- 172.21.15.40:0/432795011 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2568039050 con 0x557a9171e5b0 2024-03-18T22:43:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f25557fa640 1 -- 172.21.15.40:0/432795011 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f256804a980 con 0x557a9171e5b0 2024-03-18T22:43:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.243+0000 7f25557fa640 1 -- 172.21.15.40:0/432795011 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25680042a0 con 0x557a9171e5b0 2024-03-18T22:43:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.244+0000 7f2578587740 1 -- 172.21.15.40:0/432795011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 msgr2=0x557a91722510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.244+0000 7f2578587740 1 --2- 172.21.15.40:0/432795011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a91722510 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2568006010 tx=0x7f2568052000 comp rx=0 tx=0).stop 2024-03-18T22:43:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.244+0000 7f2578587740 1 -- 172.21.15.40:0/432795011 shutdown_connections 2024-03-18T22:43:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.244+0000 7f2578587740 1 --2- 172.21.15.40:0/432795011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a91722510 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.244+0000 7f2578587740 1 --2- 172.21.15.40:0/432795011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a9172a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.244+0000 7f2578587740 1 --2- 172.21.15.40:0/432795011 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 0x557a91717930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.244+0000 7f2578587740 1 -- 172.21.15.40:0/432795011 >> 172.21.15.40:0/432795011 conn(0x557a91713fc0 msgr2=0x557a91717220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.244+0000 7f2578587740 1 -- 172.21.15.40:0/432795011 shutdown_connections 2024-03-18T22:43:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.244+0000 7f2578587740 1 -- 172.21.15.40:0/432795011 wait complete. 2024-03-18T22:43:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.244+0000 7f2578587740 1 -- start start 2024-03-18T22:43:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f2578587740 1 Processor -- start 2024-03-18T22:43:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f2578587740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 0x557a91727610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f2578587740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a91727b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f2578587740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a91728b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f2578587740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25701a97b0 con 0x557a9171e5b0 2024-03-18T22:43:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f2578587740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f25701a90e0 con 0x557a91719440 2024-03-18T22:43:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f2578587740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f25701a6320 con 0x557a91718a50 2024-03-18T22:43:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f2578587740 10 client.0 _ll_register_callbacks cb 0x7f256000bb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f256f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a91728b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f256e7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a91727b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f256f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a91728b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39780/0 (socket says 172.21.15.40:39780) 2024-03-18T22:43:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f256f7fe640 1 -- 172.21.15.40:0/4194102874 learned_addr learned my addr 172.21.15.40:0/4194102874 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f256effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 0x557a91727610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f256f7fe640 1 -- 172.21.15.40:0/4194102874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 msgr2=0x557a91727610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f256f7fe640 1 --2- 172.21.15.40:0/4194102874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 0x557a91727610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f256f7fe640 1 -- 172.21.15.40:0/4194102874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 msgr2=0x557a91727b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f256f7fe640 1 --2- 172.21.15.40:0/4194102874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a91727b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f256f7fe640 1 -- 172.21.15.40:0/4194102874 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a91729150 con 0x557a9171e5b0 2024-03-18T22:43:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f256e7fc640 1 --2- 172.21.15.40:0/4194102874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a91727b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f256f7fe640 1 --2- 172.21.15.40:0/4194102874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a91728b60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2568005e30 tx=0x7f256804aea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c53b9402836192 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.245+0000 7f254effd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:43:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.246+0000 7f254effd640 1 -- 172.21.15.40:0/4194102874 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2568064060 con 0x557a9171e5b0 2024-03-18T22:43:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.246+0000 7f254effd640 1 -- 172.21.15.40:0/4194102874 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2568003c30 con 0x557a9171e5b0 2024-03-18T22:43:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.246+0000 7f2578587740 1 -- 172.21.15.40:0/4194102874 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f256804a980 con 0x557a9171e5b0 2024-03-18T22:43:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.246+0000 7f254effd640 1 -- 172.21.15.40:0/4194102874 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2568039020 con 0x557a9171e5b0 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.246+0000 7f2578587740 10 client.15657 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.246+0000 7f2578587740 1 -- 172.21.15.40:0/4194102874 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557a9172a460 con 0x557a9171e5b0 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.246+0000 7f254effd640 1 -- 172.21.15.40:0/4194102874 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f256804a980 con 0x557a9171e5b0 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.246+0000 7f254effd640 1 -- 172.21.15.40:0/4194102874 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f256803f040 con 0x557a9171e5b0 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.246+0000 7f2578587740 10 client.15657 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.246+0000 7f2578587740 10 client.15657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.246+0000 7f2578587740 20 client.15657 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.246+0000 7f2578587740 1 --2- 172.21.15.40:0/4194102874 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557a91795220 0x557a91797610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.246+0000 7f2578587740 4 client.15657 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.246+0000 7f2578587740 1 -- 172.21.15.40:0/4194102874 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557a91684d90 con 0x557a91795220 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.246+0000 7f256effd640 1 --2- 172.21.15.40:0/4194102874 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557a91795220 0x557a91797610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f254effd640 10 client.15657 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f256effd640 1 --2- 172.21.15.40:0/4194102874 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557a91795220 0x557a91797610 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7227374b10d59ed server_cookie=56385979421468a9 in_seq=0 out_seq=0 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f254effd640 1 -- 172.21.15.40:0/4194102874 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557a91684d90 con 0x557a91795220 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f254effd640 10 client.15657 handle_command_reply: tid=0 2024-03-18T22:43:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f2578587740 1 client.15657 shutdown 2024-03-18T22:43:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f2578587740 1 -- 172.21.15.40:0/4194102874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 msgr2=0x557a91728b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f2578587740 1 --2- 172.21.15.40:0/4194102874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a91728b60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2568005e30 tx=0x7f256804aea0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f2578587740 1 -- 172.21.15.40:0/4194102874 shutdown_connections 2024-03-18T22:43:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f2578587740 1 --2- 172.21.15.40:0/4194102874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557a9171e5b0 0x557a91728b60 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f2578587740 1 --2- 172.21.15.40:0/4194102874 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557a91795220 0x557a91797610 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f2578587740 1 --2- 172.21.15.40:0/4194102874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557a91719440 0x557a91727b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f2578587740 1 --2- 172.21.15.40:0/4194102874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557a91718a50 0x557a91727610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f254effd640 0 client.15657 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f2578587740 1 -- 172.21.15.40:0/4194102874 >> 172.21.15.40:0/4194102874 conn(0x557a91713fc0 msgr2=0x557a91717ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f2578587740 1 -- 172.21.15.40:0/4194102874 shutdown_connections 2024-03-18T22:43:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f2578587740 1 -- 172.21.15.40:0/4194102874 wait complete. 2024-03-18T22:43:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.247+0000 7f2578587740 20 client.15657 trim_cache size 0 max 16384 2024-03-18T22:43:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:43:11.255 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:43:11.255 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:43:11.255 INFO:teuthology.orchestra.run.smithi040.stdout: "f75cb960-5553-46d0-b03a-98c03ed7b889": { 2024-03-18T22:43:11.255 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:43:11.255 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "f75cb960-5553-46d0-b03a-98c03ed7b889", 2024-03-18T22:43:11.255 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:43:11.255 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:43:11.255 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:43:11.255 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:43:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.248+0000 7f25758c0640 1 -- 172.21.15.40:0/3278488626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25580a2800 msgr2=0x7f25580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.248+0000 7f25758c0640 1 --2- 172.21.15.40:0/3278488626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25580a2800 0x7f25580a4cc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f256804c000 tx=0x7f2568045000 comp rx=0 tx=0).stop 2024-03-18T22:43:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.248+0000 7f25758c0640 1 -- 172.21.15.40:0/3278488626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2570101910 msgr2=0x7f2570195d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.248+0000 7f25758c0640 1 --2- 172.21.15.40:0/3278488626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2570101910 0x7f2570195d70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f256400e9f0 tx=0x7f256400edf0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.248+0000 7f25758c0640 1 -- 172.21.15.40:0/3278488626 shutdown_connections 2024-03-18T22:43:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.248+0000 7f25758c0640 1 --2- 172.21.15.40:0/3278488626 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25580a2800 0x7f25580a4cc0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.248+0000 7f25758c0640 1 --2- 172.21.15.40:0/3278488626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25701022d0 0x7f257019cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.248+0000 7f25758c0640 1 --2- 172.21.15.40:0/3278488626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2570101910 0x7f2570195d70 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.248+0000 7f25758c0640 1 --2- 172.21.15.40:0/3278488626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2570100720 0x7f2570195830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.248+0000 7f25758c0640 1 -- 172.21.15.40:0/3278488626 >> 172.21.15.40:0/3278488626 conn(0x7f25700fbcb0 msgr2=0x7f2570104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.248+0000 7f25758c0640 1 -- 172.21.15.40:0/3278488626 shutdown_connections 2024-03-18T22:43:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:11.248+0000 7f25758c0640 1 -- 172.21.15.40:0/3278488626 wait complete. 2024-03-18T22:43:11.258 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:11 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/606403657' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:43:11.258 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:11 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1503523836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:43:11.258 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:11 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1067196713' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:43:11.259 INFO:tasks.cephfs.filesystem:scrub status for tag:f75cb960-5553-46d0-b03a-98c03ed7b889 - {'path': '/', 'tag': 'f75cb960-5553-46d0-b03a-98c03ed7b889', 'options': 'recursive,force'} 2024-03-18T22:43:11.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:11 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/606403657' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:43:11.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:11 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1503523836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:43:11.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:11 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1067196713' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:43:11.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:11 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/606403657' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:43:11.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:11 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1503523836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:43:11.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:11 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1067196713' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:43:12.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:12 smithi113 ceph-mon[66775]: pgmap v367: 129 pgs: 129 active+clean; 653 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-18T22:43:12.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:12 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:43:12.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:12 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:43:12.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:12 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:43:12.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:12 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:43:12.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:12 smithi103 ceph-mon[66377]: pgmap v367: 129 pgs: 129 active+clean; 653 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-18T22:43:12.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:12 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:43:12.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:12 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:43:12.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:12 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:43:12.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:12 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:43:12.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:12 smithi040 ceph-mon[62643]: pgmap v367: 129 pgs: 129 active+clean; 653 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-18T22:43:12.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:12 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:43:12.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:12 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:43:12.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:12 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:43:12.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:12 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:43:14.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:14 smithi113 ceph-mon[66775]: pgmap v368: 129 pgs: 129 active+clean; 653 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-03-18T22:43:14.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:14 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:43:14.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:14 smithi103 ceph-mon[66377]: pgmap v368: 129 pgs: 129 active+clean; 653 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-03-18T22:43:14.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:14 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:43:14.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:14 smithi040 ceph-mon[62643]: pgmap v368: 129 pgs: 129 active+clean; 653 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-03-18T22:43:14.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:14 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:43:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:16 smithi113 ceph-mon[66775]: pgmap v369: 129 pgs: 129 active+clean; 659 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-03-18T22:43:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:43:17.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:43:17.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:16 smithi103 ceph-mon[66377]: pgmap v369: 129 pgs: 129 active+clean; 659 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-03-18T22:43:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:43:17.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:43:17.110 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:16 smithi040 ceph-mon[62643]: pgmap v369: 129 pgs: 129 active+clean; 659 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-03-18T22:43:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:43:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:43:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:43:19.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:18 smithi113 ceph-mon[66775]: pgmap v370: 129 pgs: 129 active+clean; 659 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-18T22:43:19.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:18 smithi103 ceph-mon[66377]: pgmap v370: 129 pgs: 129 active+clean; 659 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-18T22:43:19.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:18 smithi040 ceph-mon[62643]: pgmap v370: 129 pgs: 129 active+clean; 659 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-18T22:43:21.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:20 smithi113 ceph-mon[66775]: pgmap v371: 129 pgs: 129 active+clean; 659 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T22:43:21.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:20 smithi103 ceph-mon[66377]: pgmap v371: 129 pgs: 129 active+clean; 659 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T22:43:21.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:20 smithi040 ceph-mon[62643]: pgmap v371: 129 pgs: 129 active+clean; 659 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T22:43:23.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:22 smithi113 ceph-mon[66775]: pgmap v372: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-18T22:43:23.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:22 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:43:23.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:22 smithi103 ceph-mon[66377]: pgmap v372: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-18T22:43:23.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:22 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:43:23.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:22 smithi040 ceph-mon[62643]: pgmap v372: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-18T22:43:23.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:22 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:43:25.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:24 smithi113 ceph-mon[66775]: pgmap v373: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-18T22:43:25.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:24 smithi103 ceph-mon[66377]: pgmap v373: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-18T22:43:25.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:24 smithi040 ceph-mon[62643]: pgmap v373: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-18T22:43:27.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:26 smithi113 ceph-mon[66775]: pgmap v374: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T22:43:27.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:26 smithi103 ceph-mon[66377]: pgmap v374: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T22:43:27.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:26 smithi040 ceph-mon[62643]: pgmap v374: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T22:43:29.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:28 smithi113 ceph-mon[66775]: pgmap v375: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T22:43:29.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:28 smithi103 ceph-mon[66377]: pgmap v375: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T22:43:29.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:28 smithi040 ceph-mon[62643]: pgmap v375: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T22:43:31.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:30 smithi113 ceph-mon[66775]: pgmap v376: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T22:43:31.108 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:30 smithi103 ceph-mon[66377]: pgmap v376: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T22:43:31.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:30 smithi040 ceph-mon[62643]: pgmap v376: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T22:43:33.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:32 smithi113 ceph-mon[66775]: pgmap v377: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-18T22:43:33.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:32 smithi103 ceph-mon[66377]: pgmap v377: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-18T22:43:33.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:32 smithi040 ceph-mon[62643]: pgmap v377: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-18T22:43:35.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:34 smithi113 ceph-mon[66775]: pgmap v378: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:43:35.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:34 smithi103 ceph-mon[66377]: pgmap v378: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:43:35.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:34 smithi040 ceph-mon[62643]: pgmap v378: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:43:37.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:36 smithi113 ceph-mon[66775]: pgmap v379: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T22:43:37.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:36 smithi103 ceph-mon[66377]: pgmap v379: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T22:43:37.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:36 smithi040 ceph-mon[62643]: pgmap v379: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T22:43:39.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:38 smithi113 ceph-mon[66775]: pgmap v380: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:43:39.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:38 smithi103 ceph-mon[66377]: pgmap v380: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:43:39.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:38 smithi040 ceph-mon[62643]: pgmap v380: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:43:41.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:40 smithi113 ceph-mon[66775]: pgmap v381: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:43:41.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:40 smithi103 ceph-mon[66377]: pgmap v381: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:43:41.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:40 smithi040 ceph-mon[62643]: pgmap v381: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:43:41.261 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:43:41.261 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:43:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.378+0000 7f7394359640 1 -- 172.21.15.40:0/2581925067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f738c1075a0 msgr2=0x7f738c109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.378+0000 7f7394359640 1 --2- 172.21.15.40:0/2581925067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f738c1075a0 0x7f738c109990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7384009be0 tx=0x7f738402bef0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.378+0000 7f7394359640 1 -- 172.21.15.40:0/2581925067 shutdown_connections 2024-03-18T22:43:41.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.378+0000 7f7394359640 1 --2- 172.21.15.40:0/2581925067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f738c1075a0 0x7f738c109990 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.378+0000 7f7394359640 1 --2- 172.21.15.40:0/2581925067 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f738c104c70 0x7f738c107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.378+0000 7f7394359640 1 --2- 172.21.15.40:0/2581925067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f738c102340 0x7f738c104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.378+0000 7f7394359640 1 -- 172.21.15.40:0/2581925067 >> 172.21.15.40:0/2581925067 conn(0x7f738c0fbcb0 msgr2=0x7f738c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.378+0000 7f7394359640 1 -- 172.21.15.40:0/2581925067 shutdown_connections 2024-03-18T22:43:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.378+0000 7f7394359640 1 -- 172.21.15.40:0/2581925067 wait complete. 2024-03-18T22:43:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.379+0000 7f7394359640 1 Processor -- start 2024-03-18T22:43:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.379+0000 7f7394359640 1 -- start start 2024-03-18T22:43:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.379+0000 7f7394359640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f738c102340 0x7f738c101cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.379+0000 7f7394359640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f738c104c70 0x7f738c1002f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.379+0000 7f7394359640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f738c1075a0 0x7f738c100830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.379+0000 7f7394359640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f738c10d370 con 0x7f738c1075a0 2024-03-18T22:43:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.379+0000 7f7394359640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f738c10d1f0 con 0x7f738c102340 2024-03-18T22:43:41.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.379+0000 7f7394359640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f738c10d4f0 con 0x7f738c104c70 2024-03-18T22:43:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.379+0000 7f73918f4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f738c104c70 0x7f738c1002f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.379+0000 7f73928f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f738c1075a0 0x7f738c100830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.379+0000 7f73918f4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f738c104c70 0x7f738c1002f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55786/0 (socket says 172.21.15.40:55786) 2024-03-18T22:43:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.379+0000 7f73918f4640 1 -- 172.21.15.40:0/3633711129 learned_addr learned my addr 172.21.15.40:0/3633711129 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.379+0000 7f73920f5640 1 --2- 172.21.15.40:0/3633711129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f738c102340 0x7f738c101cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.380+0000 7f73918f4640 1 -- 172.21.15.40:0/3633711129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f738c102340 msgr2=0x7f738c101cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.380+0000 7f73918f4640 1 --2- 172.21.15.40:0/3633711129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f738c102340 0x7f738c101cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.380+0000 7f73918f4640 1 -- 172.21.15.40:0/3633711129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f738c1075a0 msgr2=0x7f738c100830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.380+0000 7f73918f4640 1 --2- 172.21.15.40:0/3633711129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f738c1075a0 0x7f738c100830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.380+0000 7f73918f4640 1 -- 172.21.15.40:0/3633711129 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f738c1010c0 con 0x7f738c104c70 2024-03-18T22:43:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.380+0000 7f73918f4640 1 --2- 172.21.15.40:0/3633711129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f738c104c70 0x7f738c1002f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f737c00e9f0 tx=0x7f737c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.380+0000 7f73837fe640 1 -- 172.21.15.40:0/3633711129 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c00ccc0 con 0x7f738c104c70 2024-03-18T22:43:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.380+0000 7f7394359640 1 -- 172.21.15.40:0/3633711129 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f738c1a1ba0 con 0x7f738c104c70 2024-03-18T22:43:41.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.380+0000 7f7394359640 1 -- 172.21.15.40:0/3633711129 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f738c1a20f0 con 0x7f738c104c70 2024-03-18T22:43:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.380+0000 7f73837fe640 1 -- 172.21.15.40:0/3633711129 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f737c0054d0 con 0x7f738c104c70 2024-03-18T22:43:41.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.380+0000 7f73837fe640 1 -- 172.21.15.40:0/3633711129 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c010430 con 0x7f738c104c70 2024-03-18T22:43:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.383+0000 7f73837fe640 1 -- 172.21.15.40:0/3633711129 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f737c005020 con 0x7f738c104c70 2024-03-18T22:43:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.383+0000 7f7397016740 1 Processor -- start 2024-03-18T22:43:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.384+0000 7f7397016740 1 -- start start 2024-03-18T22:43:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.384+0000 7f7397016740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x55788508de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.384+0000 7f7397016740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55788508e440 0x557885092f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.384+0000 7f7397016740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5578850935b0 0x557885095a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.384+0000 7f7397016740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557884ff2500 con 0x55788508da50 2024-03-18T22:43:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.384+0000 7f7397016740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557884f9abe0 con 0x55788508e440 2024-03-18T22:43:41.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.384+0000 7f7397016740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557884fee5a0 con 0x5578850935b0 2024-03-18T22:43:41.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.384+0000 7f73928f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x55788508de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.384+0000 7f73928f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x55788508de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45780/0 (socket says 172.21.15.40:45780) 2024-03-18T22:43:41.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.384+0000 7f73928f6640 1 -- 172.21.15.40:0/1775457064 learned_addr learned my addr 172.21.15.40:0/1775457064 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:41.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.384+0000 7f73920f5640 1 --2- 172.21.15.40:0/1775457064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55788508e440 0x557885092f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.384+0000 7f73918f4640 1 --2- 172.21.15.40:0/1775457064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5578850935b0 0x557885095a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.385+0000 7f73837fe640 1 --2- 172.21.15.40:0/3633711129 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f73780a2800 0x7f73780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.385+0000 7f73928f6640 1 -- 172.21.15.40:0/1775457064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5578850935b0 msgr2=0x557885095a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.385+0000 7f73928f6640 1 --2- 172.21.15.40:0/1775457064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5578850935b0 0x557885095a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.385+0000 7f73928f6640 1 -- 172.21.15.40:0/1775457064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55788508e440 msgr2=0x557885092f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.385+0000 7f73928f6640 1 --2- 172.21.15.40:0/1775457064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55788508e440 0x557885092f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.385+0000 7f73928f6640 1 -- 172.21.15.40:0/1775457064 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557884fbc000 con 0x55788508da50 2024-03-18T22:43:41.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.385+0000 7f73837fe640 1 -- 172.21.15.40:0/3633711129 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f737c0cf290 con 0x7f738c104c70 2024-03-18T22:43:41.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.385+0000 7f73918f4640 1 --2- 172.21.15.40:0/1775457064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5578850935b0 0x557885095a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:43:41.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.385+0000 7f73920f5640 1 --2- 172.21.15.40:0/1775457064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55788508e440 0x557885092f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:43:41.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.385+0000 7f73928f6640 1 --2- 172.21.15.40:0/3633711129 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f73780a2800 0x7f73780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.385+0000 7f73928f6640 1 --2- 172.21.15.40:0/1775457064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x55788508de70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7384009be0 tx=0x7f7384005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ed0e51c2dd67f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:41.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.385+0000 7f7380bfe640 1 -- 172.21.15.40:0/1775457064 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7384043040 con 0x55788508da50 2024-03-18T22:43:41.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.385+0000 7f7380bfe640 1 -- 172.21.15.40:0/1775457064 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7384003970 con 0x55788508da50 2024-03-18T22:43:41.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.386+0000 7f7397016740 1 -- 172.21.15.40:0/1775457064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 msgr2=0x55788508de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.386+0000 7f73928f6640 1 --2- 172.21.15.40:0/3633711129 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f73780a2800 0x7f73780a4cc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7384005f00 tx=0x7f73840043a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:41.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.386+0000 7f7397016740 1 --2- 172.21.15.40:0/1775457064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x55788508de70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7384009be0 tx=0x7f7384005bd0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.386+0000 7f7397016740 1 -- 172.21.15.40:0/1775457064 shutdown_connections 2024-03-18T22:43:41.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.386+0000 7f7397016740 1 --2- 172.21.15.40:0/1775457064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5578850935b0 0x557885095a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.386+0000 7f7397016740 1 --2- 172.21.15.40:0/1775457064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55788508e440 0x557885092f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.386+0000 7f7397016740 1 --2- 172.21.15.40:0/1775457064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x55788508de70 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.386+0000 7f7397016740 1 -- 172.21.15.40:0/1775457064 >> 172.21.15.40:0/1775457064 conn(0x557885088fc0 msgr2=0x5578850893c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:41.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.387+0000 7f7397016740 1 -- 172.21.15.40:0/1775457064 shutdown_connections 2024-03-18T22:43:41.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.387+0000 7f7397016740 1 -- 172.21.15.40:0/1775457064 wait complete. 2024-03-18T22:43:41.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.387+0000 7f7397016740 1 -- start start 2024-03-18T22:43:41.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.387+0000 7f7397016740 1 Processor -- start 2024-03-18T22:43:41.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.387+0000 7f7397016740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508da50 0x557885097e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.387+0000 7f7397016740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508e440 0x5578850983a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.387+0000 7f7397016740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 0x55788509f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.387+0000 7f7397016740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5578850965b0 con 0x55788508e440 2024-03-18T22:43:41.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.387+0000 7f7397016740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557884fae160 con 0x5578850935b0 2024-03-18T22:43:41.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.387+0000 7f7397016740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557885096730 con 0x55788508da50 2024-03-18T22:43:41.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.388+0000 7f7397016740 10 client.0 _ll_register_callbacks cb 0x557884feaa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:41.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.388+0000 7f73920f5640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508da50 0x557885097e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.388+0000 7f73920f5640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508da50 0x557885097e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55802/0 (socket says 172.21.15.40:55802) 2024-03-18T22:43:41.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.388+0000 7f73920f5640 1 -- 172.21.15.40:0/3335230874 learned_addr learned my addr 172.21.15.40:0/3335230874 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:41.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.388+0000 7f73918f4640 1 --2- 172.21.15.40:0/3335230874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508e440 0x5578850983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.388+0000 7f73928f6640 1 --2- 172.21.15.40:0/3335230874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 0x55788509f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.388+0000 7f73920f5640 1 -- 172.21.15.40:0/3335230874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 msgr2=0x55788509f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.388+0000 7f73920f5640 1 --2- 172.21.15.40:0/3335230874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 0x55788509f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.388+0000 7f73920f5640 1 -- 172.21.15.40:0/3335230874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508e440 msgr2=0x5578850983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.388+0000 7f73920f5640 1 --2- 172.21.15.40:0/3335230874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508e440 0x5578850983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.388+0000 7f73920f5640 1 -- 172.21.15.40:0/3335230874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557884dd07d0 con 0x55788508da50 2024-03-18T22:43:41.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.388+0000 7f73920f5640 1 --2- 172.21.15.40:0/3335230874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508da50 0x557885097e60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f73880126d0 tx=0x7f7388012b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bbcab49f032bc45 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:41.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.388+0000 7f7375ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:43:41.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.388+0000 7f7375ffb640 1 -- 172.21.15.40:0/3335230874 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7388004020 con 0x55788508da50 2024-03-18T22:43:41.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.389+0000 7f7397016740 1 -- 172.21.15.40:0/3335230874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7384003970 con 0x55788508da50 2024-03-18T22:43:41.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.389+0000 7f7375ffb640 1 -- 172.21.15.40:0/3335230874 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f73880041c0 con 0x55788508da50 2024-03-18T22:43:41.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.389+0000 7f7375ffb640 1 -- 172.21.15.40:0/3335230874 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7388025050 con 0x55788508da50 2024-03-18T22:43:41.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.389+0000 7f7397016740 10 client.25540 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:41.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.389+0000 7f7397016740 1 -- 172.21.15.40:0/3335230874 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557884d063f0 con 0x55788508da50 2024-03-18T22:43:41.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.389+0000 7f7375ffb640 1 -- 172.21.15.40:0/3335230874 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7384003970 con 0x55788508da50 2024-03-18T22:43:41.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.389+0000 7f7375ffb640 1 -- 172.21.15.40:0/3335230874 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7388003890 con 0x55788508da50 2024-03-18T22:43:41.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.389+0000 7f7397016740 10 client.25540 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:41.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.389+0000 7f7397016740 10 client.25540 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:43:41.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.389+0000 7f7397016740 20 client.25540 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:41.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.389+0000 7f7397016740 1 --2- 172.21.15.40:0/3335230874 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5578850a40b0 0x5578850a6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.389+0000 7f7397016740 4 client.25540 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:43:41.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.389+0000 7f7397016740 1 -- 172.21.15.40:0/3335230874 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557884fffa40 con 0x5578850a40b0 2024-03-18T22:43:41.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.389+0000 7f73918f4640 1 --2- 172.21.15.40:0/3335230874 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5578850a40b0 0x5578850a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.390+0000 7f73918f4640 1 --2- 172.21.15.40:0/3335230874 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5578850a40b0 0x5578850a6550 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2983dae2792229c server_cookie=ed27a5e7d75f7276 in_seq=0 out_seq=0 2024-03-18T22:43:41.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.390+0000 7f7375ffb640 10 client.25540 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:41.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.390+0000 7f7375ffb640 1 -- 172.21.15.40:0/3335230874 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557884fffa40 con 0x5578850a40b0 2024-03-18T22:43:41.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.390+0000 7f7375ffb640 10 client.25540 handle_command_reply: tid=0 2024-03-18T22:43:41.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.391+0000 7f7397016740 1 client.25540 shutdown 2024-03-18T22:43:41.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.391+0000 7f7397016740 1 -- 172.21.15.40:0/3335230874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508da50 msgr2=0x557885097e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.391+0000 7f7397016740 1 --2- 172.21.15.40:0/3335230874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508da50 0x557885097e60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f73880126d0 tx=0x7f7388012b90 comp rx=0 tx=0).stop 2024-03-18T22:43:41.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.391+0000 7f7397016740 1 -- 172.21.15.40:0/3335230874 shutdown_connections 2024-03-18T22:43:41.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.391+0000 7f7397016740 1 --2- 172.21.15.40:0/3335230874 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5578850a40b0 0x5578850a6550 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.391+0000 7f7397016740 1 --2- 172.21.15.40:0/3335230874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 0x55788509f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.391+0000 7f7375ffb640 0 client.25540 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:41.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.391+0000 7f7397016740 1 --2- 172.21.15.40:0/3335230874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508e440 0x5578850983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.391+0000 7f7397016740 1 --2- 172.21.15.40:0/3335230874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508da50 0x557885097e60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.391+0000 7f7397016740 1 -- 172.21.15.40:0/3335230874 >> 172.21.15.40:0/3335230874 conn(0x557885088fc0 msgr2=0x5578850881e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:41.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.391+0000 7f7397016740 1 -- 172.21.15.40:0/3335230874 shutdown_connections 2024-03-18T22:43:41.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.391+0000 7f7397016740 1 -- 172.21.15.40:0/3335230874 wait complete. 2024-03-18T22:43:41.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.391+0000 7f7397016740 20 client.25540 trim_cache size 0 max 16384 2024-03-18T22:43:41.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.413+0000 7f7397016740 1 Processor -- start 2024-03-18T22:43:41.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.413+0000 7f7397016740 1 -- start start 2024-03-18T22:43:41.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.413+0000 7f7397016740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x557885087950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.413+0000 7f7397016740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508e440 0x55788509f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.413+0000 7f7397016740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 0x557885087e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.413+0000 7f7397016740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f738c1ac8a0 con 0x55788508da50 2024-03-18T22:43:41.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.413+0000 7f7397016740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f738c1ae3e0 con 0x5578850935b0 2024-03-18T22:43:41.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.413+0000 7f7397016740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f738c1acf70 con 0x55788508e440 2024-03-18T22:43:41.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.413+0000 7f73920f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x557885087950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.414+0000 7f73918f4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508e440 0x55788509f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.414+0000 7f73920f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x557885087950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45806/0 (socket says 172.21.15.40:45806) 2024-03-18T22:43:41.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.414+0000 7f73918f4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508e440 0x55788509f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55804/0 (socket says 172.21.15.40:55804) 2024-03-18T22:43:41.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.414+0000 7f73928f6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 0x557885087e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.414+0000 7f73920f5640 1 -- 172.21.15.40:0/1738414725 learned_addr learned my addr 172.21.15.40:0/1738414725 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:41.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.414+0000 7f73918f4640 1 -- 172.21.15.40:0/1738414725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 msgr2=0x557885087e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.414+0000 7f73918f4640 1 --2- 172.21.15.40:0/1738414725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 0x557885087e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.414+0000 7f73918f4640 1 -- 172.21.15.40:0/1738414725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 msgr2=0x557885087950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.414+0000 7f73918f4640 1 --2- 172.21.15.40:0/1738414725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x557885087950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.414+0000 7f73918f4640 1 -- 172.21.15.40:0/1738414725 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557884cc60f0 con 0x55788508e440 2024-03-18T22:43:41.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.414+0000 7f73920f5640 1 --2- 172.21.15.40:0/1738414725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x557885087950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:41.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.414+0000 7f73918f4640 1 --2- 172.21.15.40:0/1738414725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508e440 0x55788509f3a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f737c0cc9f0 tx=0x7f737c0d17a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=741f3a610be84265 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:41.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.414+0000 7f7377fff640 1 -- 172.21.15.40:0/1738414725 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c0cce60 con 0x55788508e440 2024-03-18T22:43:41.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.414+0000 7f7377fff640 1 -- 172.21.15.40:0/1738414725 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f737c005310 con 0x55788508e440 2024-03-18T22:43:41.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.414+0000 7f7377fff640 1 -- 172.21.15.40:0/1738414725 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737c0a3fe0 con 0x55788508e440 2024-03-18T22:43:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.415+0000 7f7397016740 1 -- 172.21.15.40:0/1738414725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508e440 msgr2=0x55788509f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.415+0000 7f7397016740 1 --2- 172.21.15.40:0/1738414725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508e440 0x55788509f3a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f737c0cc9f0 tx=0x7f737c0d17a0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.415+0000 7f7397016740 1 -- 172.21.15.40:0/1738414725 shutdown_connections 2024-03-18T22:43:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.415+0000 7f7397016740 1 --2- 172.21.15.40:0/1738414725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 0x557885087e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.415+0000 7f7397016740 1 --2- 172.21.15.40:0/1738414725 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508e440 0x55788509f3a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.415+0000 7f7397016740 1 --2- 172.21.15.40:0/1738414725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x557885087950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.415+0000 7f7397016740 1 -- 172.21.15.40:0/1738414725 >> 172.21.15.40:0/1738414725 conn(0x557885088fc0 msgr2=0x5578850893a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:41.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.415+0000 7f7397016740 1 -- 172.21.15.40:0/1738414725 shutdown_connections 2024-03-18T22:43:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.415+0000 7f7397016740 1 -- 172.21.15.40:0/1738414725 wait complete. 2024-03-18T22:43:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.415+0000 7f7397016740 1 -- start start 2024-03-18T22:43:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f7397016740 1 Processor -- start 2024-03-18T22:43:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f7397016740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x55788509c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f7397016740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508e440 0x55788509cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f7397016740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 0x55788509dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f7397016740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f738c1add10 con 0x55788508da50 2024-03-18T22:43:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f7397016740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f738c1ad640 con 0x5578850935b0 2024-03-18T22:43:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f7397016740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f738c1aa880 con 0x55788508e440 2024-03-18T22:43:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f7397016740 10 client.0 _ll_register_callbacks cb 0x7f7388009870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f73920f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x55788509c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f73918f4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508e440 0x55788509cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f73928f6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 0x55788509dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f73918f4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508e440 0x55788509cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55812/0 (socket says 172.21.15.40:55812) 2024-03-18T22:43:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f73918f4640 1 -- 172.21.15.40:0/2368626857 learned_addr learned my addr 172.21.15.40:0/2368626857 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f73928f6640 1 -- 172.21.15.40:0/2368626857 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508e440 msgr2=0x55788509cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f73928f6640 1 --2- 172.21.15.40:0/2368626857 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508e440 0x55788509cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f73928f6640 1 -- 172.21.15.40:0/2368626857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 msgr2=0x55788509c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f73928f6640 1 --2- 172.21.15.40:0/2368626857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x55788509c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f73928f6640 1 -- 172.21.15.40:0/2368626857 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55788509e190 con 0x5578850935b0 2024-03-18T22:43:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.416+0000 7f73918f4640 1 --2- 172.21.15.40:0/2368626857 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508e440 0x55788509cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f73928f6640 1 --2- 172.21.15.40:0/2368626857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 0x55788509dba0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7384004af0 tx=0x7f7384042000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfe055af42b6b42 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f7375ffb640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:43:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f7375ffb640 1 -- 172.21.15.40:0/2368626857 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73840041e0 con 0x5578850935b0 2024-03-18T22:43:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f7375ffb640 1 -- 172.21.15.40:0/2368626857 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f73840046f0 con 0x5578850935b0 2024-03-18T22:43:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f7397016740 1 -- 172.21.15.40:0/2368626857 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557884ff9d90 con 0x5578850935b0 2024-03-18T22:43:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f7375ffb640 1 -- 172.21.15.40:0/2368626857 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7384064050 con 0x5578850935b0 2024-03-18T22:43:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f7397016740 10 client.25565 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f7397016740 1 -- 172.21.15.40:0/2368626857 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55788509f4f0 con 0x5578850935b0 2024-03-18T22:43:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f7375ffb640 1 -- 172.21.15.40:0/2368626857 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x557884ff9d90 con 0x5578850935b0 2024-03-18T22:43:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f7375ffb640 1 -- 172.21.15.40:0/2368626857 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7384047020 con 0x5578850935b0 2024-03-18T22:43:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f7397016740 10 client.25565 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f7397016740 10 client.25565 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:43:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f7397016740 20 client.25565 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f7397016740 1 --2- 172.21.15.40:0/2368626857 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5578850a40b0 0x55788510c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f7397016740 4 client.25565 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:43:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.417+0000 7f7397016740 1 -- 172.21.15.40:0/2368626857 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557884f77540 con 0x5578850a40b0 2024-03-18T22:43:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f73920f5640 1 --2- 172.21.15.40:0/2368626857 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5578850a40b0 0x55788510c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f73920f5640 1 --2- 172.21.15.40:0/2368626857 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5578850a40b0 0x55788510c580 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2956ec526e0f350d server_cookie=ca0fc3acf9b376c3 in_seq=0 out_seq=0 2024-03-18T22:43:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f7375ffb640 10 client.25565 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f7375ffb640 1 -- 172.21.15.40:0/2368626857 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557884f77540 con 0x5578850a40b0 2024-03-18T22:43:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f7375ffb640 10 client.25565 handle_command_reply: tid=0 2024-03-18T22:43:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f7397016740 1 client.25565 shutdown 2024-03-18T22:43:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f7397016740 1 -- 172.21.15.40:0/2368626857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 msgr2=0x55788509dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f7397016740 1 --2- 172.21.15.40:0/2368626857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 0x55788509dba0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7384004af0 tx=0x7f7384042000 comp rx=0 tx=0).stop 2024-03-18T22:43:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f7397016740 1 -- 172.21.15.40:0/2368626857 shutdown_connections 2024-03-18T22:43:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f7397016740 1 --2- 172.21.15.40:0/2368626857 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5578850a40b0 0x55788510c580 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f7397016740 1 --2- 172.21.15.40:0/2368626857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5578850935b0 0x55788509dba0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f7397016740 1 --2- 172.21.15.40:0/2368626857 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55788508e440 0x55788509cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f7397016740 1 --2- 172.21.15.40:0/2368626857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55788508da50 0x55788509c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f7397016740 1 -- 172.21.15.40:0/2368626857 >> 172.21.15.40:0/2368626857 conn(0x557885088fc0 msgr2=0x55788508c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f7375ffb640 0 client.25565 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f7397016740 1 -- 172.21.15.40:0/2368626857 shutdown_connections 2024-03-18T22:43:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.418+0000 7f7397016740 1 -- 172.21.15.40:0/2368626857 wait complete. 2024-03-18T22:43:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.419+0000 7f7397016740 20 client.25565 trim_cache size 0 max 16384 2024-03-18T22:43:41.426 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:43:41.426 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:43:41.426 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:43:41.426 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:43:41.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.419+0000 7f7394359640 1 -- 172.21.15.40:0/3633711129 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f73780a2800 msgr2=0x7f73780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.419+0000 7f7394359640 1 --2- 172.21.15.40:0/3633711129 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f73780a2800 0x7f73780a4cc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7384005f00 tx=0x7f73840043a0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.419+0000 7f7394359640 1 -- 172.21.15.40:0/3633711129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f738c104c70 msgr2=0x7f738c1002f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.419+0000 7f7394359640 1 --2- 172.21.15.40:0/3633711129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f738c104c70 0x7f738c1002f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f737c00e9f0 tx=0x7f737c00eeb0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.419+0000 7f7394359640 1 -- 172.21.15.40:0/3633711129 shutdown_connections 2024-03-18T22:43:41.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.419+0000 7f7394359640 1 --2- 172.21.15.40:0/3633711129 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f73780a2800 0x7f73780a4cc0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.419+0000 7f7394359640 1 --2- 172.21.15.40:0/3633711129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f738c1075a0 0x7f738c100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.419+0000 7f7394359640 1 --2- 172.21.15.40:0/3633711129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f738c104c70 0x7f738c1002f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.419+0000 7f7394359640 1 --2- 172.21.15.40:0/3633711129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f738c102340 0x7f738c101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.419+0000 7f7394359640 1 -- 172.21.15.40:0/3633711129 >> 172.21.15.40:0/3633711129 conn(0x7f738c0fbcb0 msgr2=0x7f738c102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:41.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.419+0000 7f7394359640 1 -- 172.21.15.40:0/3633711129 shutdown_connections 2024-03-18T22:43:41.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.419+0000 7f7394359640 1 -- 172.21.15.40:0/3633711129 wait complete. 2024-03-18T22:43:41.431 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:43:41.431 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:43:41.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.535+0000 7f887a169640 1 -- 172.21.15.40:0/3969929182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8874101930 msgr2=0x7f8874101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.535+0000 7f887a169640 1 --2- 172.21.15.40:0/3969929182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8874101930 0x7f8874101db0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8868009900 tx=0x7f886802bc30 comp rx=0 tx=0).stop 2024-03-18T22:43:41.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.535+0000 7f887a169640 1 -- 172.21.15.40:0/3969929182 shutdown_connections 2024-03-18T22:43:41.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.535+0000 7f887a169640 1 --2- 172.21.15.40:0/3969929182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88741022f0 0x7f8874108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.535+0000 7f887a169640 1 --2- 172.21.15.40:0/3969929182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8874101930 0x7f8874101db0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.535+0000 7f887a169640 1 --2- 172.21.15.40:0/3969929182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8874100740 0x7f8874100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.535+0000 7f887a169640 1 -- 172.21.15.40:0/3969929182 >> 172.21.15.40:0/3969929182 conn(0x7f88740fbcb0 msgr2=0x7f88740fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:41.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.535+0000 7f887a169640 1 -- 172.21.15.40:0/3969929182 shutdown_connections 2024-03-18T22:43:41.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.535+0000 7f887a169640 1 -- 172.21.15.40:0/3969929182 wait complete. 2024-03-18T22:43:41.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.535+0000 7f887a169640 1 Processor -- start 2024-03-18T22:43:41.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.535+0000 7f887a169640 1 -- start start 2024-03-18T22:43:41.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f887a169640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8874100740 0x7f8874197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f887a169640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8874101930 0x7f8874197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f887a169640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88741022f0 0x7f887419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f887a169640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f887410d3f0 con 0x7f8874101930 2024-03-18T22:43:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f887a169640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f887410d270 con 0x7f88741022f0 2024-03-18T22:43:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f887a169640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f887410d570 con 0x7f8874100740 2024-03-18T22:43:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f8872ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8874101930 0x7f8874197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f8872ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8874101930 0x7f8874197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45836/0 (socket says 172.21.15.40:45836) 2024-03-18T22:43:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f8872ffd640 1 -- 172.21.15.40:0/1361834099 learned_addr learned my addr 172.21.15.40:0/1361834099 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f88737fe640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8874100740 0x7f8874197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f8873fff640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88741022f0 0x7f887419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f8872ffd640 1 -- 172.21.15.40:0/1361834099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8874100740 msgr2=0x7f8874197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f8872ffd640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8874100740 0x7f8874197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f8872ffd640 1 -- 172.21.15.40:0/1361834099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88741022f0 msgr2=0x7f887419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f8872ffd640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88741022f0 0x7f887419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f8872ffd640 1 -- 172.21.15.40:0/1361834099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8874109540 con 0x7f8874101930 2024-03-18T22:43:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f88737fe640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8874100740 0x7f8874197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:41.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f8873fff640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88741022f0 0x7f887419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:43:41.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.536+0000 7f8872ffd640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8874101930 0x7f8874197f90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8868000c00 tx=0x7f8868041a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:41.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.537+0000 7f8870ff9640 1 -- 172.21.15.40:0/1361834099 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868004450 con 0x7f8874101930 2024-03-18T22:43:41.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.537+0000 7f887a169640 1 -- 172.21.15.40:0/1361834099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88741097d0 con 0x7f8874101930 2024-03-18T22:43:41.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.537+0000 7f8870ff9640 1 -- 172.21.15.40:0/1361834099 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f88680052b0 con 0x7f8874101930 2024-03-18T22:43:41.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.537+0000 7f8870ff9640 1 -- 172.21.15.40:0/1361834099 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868003d90 con 0x7f8874101930 2024-03-18T22:43:41.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.537+0000 7f887a169640 1 -- 172.21.15.40:0/1361834099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8874109d10 con 0x7f8874101930 2024-03-18T22:43:41.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.538+0000 7f887a169640 1 -- 172.21.15.40:0/1361834099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f887402d1f0 con 0x7f8874101930 2024-03-18T22:43:41.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.540+0000 7f8870ff9640 1 -- 172.21.15.40:0/1361834099 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f88680035f0 con 0x7f8874101930 2024-03-18T22:43:41.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.541+0000 7f8870ff9640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f885c0a2800 0x7f885c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.541+0000 7f88737fe640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f885c0a2800 0x7f885c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.541+0000 7f8870ff9640 1 -- 172.21.15.40:0/1361834099 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f88680f9890 con 0x7f8874101930 2024-03-18T22:43:41.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.541+0000 7f88737fe640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f885c0a2800 0x7f885c0a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8874101790 tx=0x7f8864005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:41.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.544+0000 7f8870ff9640 1 -- 172.21.15.40:0/1361834099 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8868046070 con 0x7f8874101930 2024-03-18T22:43:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.767+0000 7f887a169640 1 -- 172.21.15.40:0/1361834099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f887402d3c0 con 0x7f8874101930 2024-03-18T22:43:41.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.768+0000 7f8870ff9640 1 -- 172.21.15.40:0/1361834099 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11947 (secure 0 0 0) 0x7f88680c6660 con 0x7f8874101930 2024-03-18T22:43:41.769 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:43:41.769 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:43:03.220450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24388,24401,24473,24481,24509]},"id":1}]} 2024-03-18T22:43:41.769 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 28 2024-03-18T22:43:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.770+0000 7f887a169640 1 -- 172.21.15.40:0/1361834099 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f885c0a2800 msgr2=0x7f885c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.770+0000 7f887a169640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f885c0a2800 0x7f885c0a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8874101790 tx=0x7f8864005f90 comp rx=0 tx=0).stop 2024-03-18T22:43:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.770+0000 7f887a169640 1 -- 172.21.15.40:0/1361834099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8874101930 msgr2=0x7f8874197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.770+0000 7f887a169640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8874101930 0x7f8874197f90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8868000c00 tx=0x7f8868041a60 comp rx=0 tx=0).stop 2024-03-18T22:43:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.770+0000 7f887a169640 1 -- 172.21.15.40:0/1361834099 shutdown_connections 2024-03-18T22:43:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.770+0000 7f887a169640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f885c0a2800 0x7f885c0a4cc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.770+0000 7f887a169640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88741022f0 0x7f887419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.770+0000 7f887a169640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8874101930 0x7f8874197f90 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.770+0000 7f887a169640 1 --2- 172.21.15.40:0/1361834099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8874100740 0x7f8874197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.770+0000 7f887a169640 1 -- 172.21.15.40:0/1361834099 >> 172.21.15.40:0/1361834099 conn(0x7f88740fbcb0 msgr2=0x7f8874104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.770+0000 7f887a169640 1 -- 172.21.15.40:0/1361834099 shutdown_connections 2024-03-18T22:43:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.770+0000 7f887a169640 1 -- 172.21.15.40:0/1361834099 wait complete. 2024-03-18T22:43:41.783 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:43:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.891+0000 7f44d4d10640 1 -- 172.21.15.40:0/2313988025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d01022d0 msgr2=0x7f44d0108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.891+0000 7f44d4d10640 1 --2- 172.21.15.40:0/2313988025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d01022d0 0x7f44d0108b60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009ca0 tx=0x7f44c802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.891+0000 7f44d4d10640 1 -- 172.21.15.40:0/2313988025 shutdown_connections 2024-03-18T22:43:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.891+0000 7f44d4d10640 1 --2- 172.21.15.40:0/2313988025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d01022d0 0x7f44d0108b60 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.891+0000 7f44d4d10640 1 --2- 172.21.15.40:0/2313988025 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d0101910 0x7f44d0101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.891+0000 7f44d4d10640 1 --2- 172.21.15.40:0/2313988025 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44d0100720 0x7f44d0100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.891+0000 7f44d4d10640 1 -- 172.21.15.40:0/2313988025 >> 172.21.15.40:0/2313988025 conn(0x7f44d00fbc90 msgr2=0x7f44d00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.891+0000 7f44d4d10640 1 -- 172.21.15.40:0/2313988025 shutdown_connections 2024-03-18T22:43:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.891+0000 7f44d4d10640 1 -- 172.21.15.40:0/2313988025 wait complete. 2024-03-18T22:43:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.892+0000 7f44d4d10640 1 Processor -- start 2024-03-18T22:43:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.892+0000 7f44d4d10640 1 -- start start 2024-03-18T22:43:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.892+0000 7f44d4d10640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d0100720 0x7f44d01979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.892+0000 7f44d4d10640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44d0101910 0x7f44d0197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.892+0000 7f44d4d10640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d01022d0 0x7f44d019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.892+0000 7f44d4d10640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f44d010d370 con 0x7f44d0100720 2024-03-18T22:43:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.892+0000 7f44d4d10640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f44d010d1f0 con 0x7f44d01022d0 2024-03-18T22:43:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.892+0000 7f44d4d10640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f44d010d4f0 con 0x7f44d0101910 2024-03-18T22:43:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.892+0000 7f44ce59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d0100720 0x7f44d01979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.892+0000 7f44cdd9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44d0101910 0x7f44d0197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.892+0000 7f44ce59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d0100720 0x7f44d01979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45856/0 (socket says 172.21.15.40:45856) 2024-03-18T22:43:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.892+0000 7f44ce59c640 1 -- 172.21.15.40:0/1876718307 learned_addr learned my addr 172.21.15.40:0/1876718307 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.892+0000 7f44ced9d640 1 --2- 172.21.15.40:0/1876718307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d01022d0 0x7f44d019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.893+0000 7f44cdd9b640 1 -- 172.21.15.40:0/1876718307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d01022d0 msgr2=0x7f44d019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.893+0000 7f44cdd9b640 1 --2- 172.21.15.40:0/1876718307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d01022d0 0x7f44d019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.893+0000 7f44cdd9b640 1 -- 172.21.15.40:0/1876718307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d0100720 msgr2=0x7f44d01979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.893+0000 7f44cdd9b640 1 --2- 172.21.15.40:0/1876718307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d0100720 0x7f44d01979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.893+0000 7f44cdd9b640 1 -- 172.21.15.40:0/1876718307 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d019f610 con 0x7f44d0101910 2024-03-18T22:43:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.893+0000 7f44ce59c640 1 --2- 172.21.15.40:0/1876718307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d0100720 0x7f44d01979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.893+0000 7f44cdd9b640 1 --2- 172.21.15.40:0/1876718307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44d0101910 0x7f44d0197f30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f44c400e9f0 tx=0x7f44c400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.893+0000 7f44bf7fe640 1 -- 172.21.15.40:0/1876718307 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c400cd80 con 0x7f44d0101910 2024-03-18T22:43:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.893+0000 7f44bf7fe640 1 -- 172.21.15.40:0/1876718307 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44c40035f0 con 0x7f44d0101910 2024-03-18T22:43:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.893+0000 7f44bf7fe640 1 -- 172.21.15.40:0/1876718307 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c4010670 con 0x7f44d0101910 2024-03-18T22:43:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.893+0000 7f44d4d10640 1 -- 172.21.15.40:0/1876718307 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d0109580 con 0x7f44d0101910 2024-03-18T22:43:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.893+0000 7f44d4d10640 1 -- 172.21.15.40:0/1876718307 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d0109a70 con 0x7f44d0101910 2024-03-18T22:43:41.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.896+0000 7f44d79db740 1 Processor -- start 2024-03-18T22:43:41.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.896+0000 7f44d79db740 1 -- start start 2024-03-18T22:43:41.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.896+0000 7f44bf7fe640 1 -- 172.21.15.40:0/1876718307 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f44c40037a0 con 0x7f44d0101910 2024-03-18T22:43:41.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.896+0000 7f44d79db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228dd6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.896+0000 7f44d79db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228dd74f0 0x557228ddbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.896+0000 7f44d79db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228ddc660 0x557228ddeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.896+0000 7f44d79db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557228d324b0 con 0x557228dd6b00 2024-03-18T22:43:41.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.896+0000 7f44d79db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557228cdad70 con 0x557228dd74f0 2024-03-18T22:43:41.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.896+0000 7f44d79db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557228d2e550 con 0x557228ddc660 2024-03-18T22:43:41.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.896+0000 7f44ced9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228dd6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.897+0000 7f44ce59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228dd74f0 0x557228ddbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.897+0000 7f44ced9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228dd6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45858/0 (socket says 172.21.15.40:45858) 2024-03-18T22:43:41.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.897+0000 7f44ced9d640 1 -- 172.21.15.40:0/2598142752 learned_addr learned my addr 172.21.15.40:0/2598142752 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:41.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.897+0000 7f44ce59c640 1 -- 172.21.15.40:0/2598142752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228ddc660 msgr2=0x557228ddeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.897+0000 7f44cdd9b640 1 --2- 172.21.15.40:0/2598142752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228ddc660 0x557228ddeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.897+0000 7f44ce59c640 1 --2- 172.21.15.40:0/2598142752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228ddc660 0x557228ddeaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.897+0000 7f44ce59c640 1 -- 172.21.15.40:0/2598142752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 msgr2=0x557228dd6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.897+0000 7f44ce59c640 1 --2- 172.21.15.40:0/2598142752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228dd6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.897+0000 7f44ce59c640 1 -- 172.21.15.40:0/2598142752 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557228cfd210 con 0x557228dd74f0 2024-03-18T22:43:41.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.897+0000 7f44ced9d640 1 --2- 172.21.15.40:0/2598142752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228dd6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:41.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.897+0000 7f44bf7fe640 1 --2- 172.21.15.40:0/1876718307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f44b80a2800 0x7f44b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.897+0000 7f44ce59c640 1 --2- 172.21.15.40:0/2598142752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228dd74f0 0x557228ddbff0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f44c0010550 tx=0x7f44c0010a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f2f77f0ae1b1b6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.897+0000 7f44ced9d640 1 --2- 172.21.15.40:0/1876718307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f44b80a2800 0x7f44b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.897+0000 7f44bf7fe640 1 -- 172.21.15.40:0/1876718307 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f44c40cf040 con 0x7f44d0101910 2024-03-18T22:43:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.897+0000 7f44bcff9640 1 -- 172.21.15.40:0/2598142752 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c000e7d0 con 0x557228dd74f0 2024-03-18T22:43:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.898+0000 7f44bcff9640 1 -- 172.21.15.40:0/2598142752 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44c000ee20 con 0x557228dd74f0 2024-03-18T22:43:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.898+0000 7f44bcff9640 1 -- 172.21.15.40:0/2598142752 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c00193d0 con 0x557228dd74f0 2024-03-18T22:43:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.898+0000 7f44d79db740 1 -- 172.21.15.40:0/2598142752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228dd74f0 msgr2=0x557228ddbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.898+0000 7f44ced9d640 1 --2- 172.21.15.40:0/1876718307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f44b80a2800 0x7f44b80a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x557228dd7d50 tx=0x7f44c8009360 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.898+0000 7f44d79db740 1 --2- 172.21.15.40:0/2598142752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228dd74f0 0x557228ddbff0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f44c0010550 tx=0x7f44c0010a10 comp rx=0 tx=0).stop 2024-03-18T22:43:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.898+0000 7f44d79db740 1 -- 172.21.15.40:0/2598142752 shutdown_connections 2024-03-18T22:43:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.898+0000 7f44d79db740 1 --2- 172.21.15.40:0/2598142752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228ddc660 0x557228ddeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.898+0000 7f44d79db740 1 --2- 172.21.15.40:0/2598142752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228dd74f0 0x557228ddbff0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.898+0000 7f44d79db740 1 --2- 172.21.15.40:0/2598142752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228dd6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.898+0000 7f44d79db740 1 -- 172.21.15.40:0/2598142752 >> 172.21.15.40:0/2598142752 conn(0x557228dd2070 msgr2=0x557228dd2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44d79db740 1 -- 172.21.15.40:0/2598142752 shutdown_connections 2024-03-18T22:43:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44d79db740 1 -- 172.21.15.40:0/2598142752 wait complete. 2024-03-18T22:43:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44d79db740 1 -- start start 2024-03-18T22:43:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44d79db740 1 Processor -- start 2024-03-18T22:43:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44d79db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228de0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44d79db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd74f0 0x557228de14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44d79db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228ddc660 0x557228de8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44d79db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557228ddf6b0 con 0x557228dd6b00 2024-03-18T22:43:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44d79db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557228cee310 con 0x557228ddc660 2024-03-18T22:43:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44d79db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557228ddf830 con 0x557228dd74f0 2024-03-18T22:43:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44d79db740 10 client.0 _ll_register_callbacks cb 0x557228ce22d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44ce59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228de0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44cdd9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd74f0 0x557228de14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44ced9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228ddc660 0x557228de8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44ce59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228de0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45870/0 (socket says 172.21.15.40:45870) 2024-03-18T22:43:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.899+0000 7f44ce59c640 1 -- 172.21.15.40:0/3602612324 learned_addr learned my addr 172.21.15.40:0/3602612324 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.900+0000 7f44ce59c640 1 -- 172.21.15.40:0/3602612324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd74f0 msgr2=0x557228de14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.900+0000 7f44ce59c640 1 --2- 172.21.15.40:0/3602612324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd74f0 0x557228de14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.900+0000 7f44ce59c640 1 -- 172.21.15.40:0/3602612324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228ddc660 msgr2=0x557228de8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.900+0000 7f44ce59c640 1 --2- 172.21.15.40:0/3602612324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228ddc660 0x557228de8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.900+0000 7f44ce59c640 1 -- 172.21.15.40:0/3602612324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557228a463f0 con 0x557228dd6b00 2024-03-18T22:43:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.900+0000 7f44ced9d640 1 --2- 172.21.15.40:0/3602612324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228ddc660 0x557228de8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.900+0000 7f44ce59c640 1 --2- 172.21.15.40:0/3602612324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228de0f90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f44c000f010 tx=0x7f44c0011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994c7f867561192b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.900+0000 7f44b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:43:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.900+0000 7f44b5ffb640 1 -- 172.21.15.40:0/3602612324 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c0027030 con 0x557228dd6b00 2024-03-18T22:43:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.900+0000 7f44b5ffb640 1 -- 172.21.15.40:0/3602612324 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f44c001f070 con 0x557228dd6b00 2024-03-18T22:43:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.900+0000 7f44d79db740 1 -- 172.21.15.40:0/3602612324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44c000ee20 con 0x557228dd6b00 2024-03-18T22:43:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.901+0000 7f44b5ffb640 1 -- 172.21.15.40:0/3602612324 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f44c0031050 con 0x557228dd6b00 2024-03-18T22:43:41.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.901+0000 7f44d79db740 10 client.15693 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:41.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.901+0000 7f44d79db740 1 -- 172.21.15.40:0/3602612324 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557228b107d0 con 0x557228dd6b00 2024-03-18T22:43:41.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.901+0000 7f44b5ffb640 1 -- 172.21.15.40:0/3602612324 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44c000ee20 con 0x557228dd6b00 2024-03-18T22:43:41.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.901+0000 7f44b5ffb640 1 -- 172.21.15.40:0/3602612324 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f44c000e660 con 0x557228dd6b00 2024-03-18T22:43:41.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.901+0000 7f44d79db740 10 client.15693 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.901+0000 7f44d79db740 10 client.15693 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:43:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.901+0000 7f44d79db740 20 client.15693 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.901+0000 7f44d79db740 1 --2- 172.21.15.40:0/3602612324 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557228ded170 0x557228def600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.901+0000 7f44d79db740 4 client.15693 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:43:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.901+0000 7f44d79db740 1 -- 172.21.15.40:0/3602612324 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557228d903e0 con 0x557228ded170 2024-03-18T22:43:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.902+0000 7f44cdd9b640 1 --2- 172.21.15.40:0/3602612324 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557228ded170 0x557228def600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.902+0000 7f44cdd9b640 1 --2- 172.21.15.40:0/3602612324 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557228ded170 0x557228def600 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2f330e714526f506 server_cookie=447e1810d2c4b31f in_seq=0 out_seq=0 2024-03-18T22:43:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.902+0000 7f44b5ffb640 10 client.15693 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:43:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.903+0000 7f44b5ffb640 1 -- 172.21.15.40:0/3602612324 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557228d903e0 con 0x557228ded170 2024-03-18T22:43:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.903+0000 7f44b5ffb640 10 client.15693 handle_command_reply: tid=0 2024-03-18T22:43:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.903+0000 7f44d79db740 1 client.15693 shutdown 2024-03-18T22:43:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.903+0000 7f44d79db740 1 -- 172.21.15.40:0/3602612324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 msgr2=0x557228de0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.903+0000 7f44d79db740 1 --2- 172.21.15.40:0/3602612324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228de0f90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f44c000f010 tx=0x7f44c0011d30 comp rx=0 tx=0).stop 2024-03-18T22:43:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.903+0000 7f44d79db740 1 -- 172.21.15.40:0/3602612324 shutdown_connections 2024-03-18T22:43:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.903+0000 7f44d79db740 1 --2- 172.21.15.40:0/3602612324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228ddc660 0x557228de8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.903+0000 7f44d79db740 1 --2- 172.21.15.40:0/3602612324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd74f0 0x557228de14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.903+0000 7f44d79db740 1 --2- 172.21.15.40:0/3602612324 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557228ded170 0x557228def600 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.903+0000 7f44d79db740 1 --2- 172.21.15.40:0/3602612324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228de0f90 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.903+0000 7f44b5ffb640 0 client.15693 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:43:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.903+0000 7f44d79db740 1 -- 172.21.15.40:0/3602612324 >> 172.21.15.40:0/3602612324 conn(0x557228dd2070 msgr2=0x557228dc8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.903+0000 7f44d79db740 1 -- 172.21.15.40:0/3602612324 shutdown_connections 2024-03-18T22:43:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.903+0000 7f44d79db740 1 -- 172.21.15.40:0/3602612324 wait complete. 2024-03-18T22:43:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.903+0000 7f44d79db740 20 client.15693 trim_cache size 0 max 16384 2024-03-18T22:43:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.930+0000 7f44d79db740 1 Processor -- start 2024-03-18T22:43:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.930+0000 7f44d79db740 1 -- start start 2024-03-18T22:43:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.930+0000 7f44d79db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228dc4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.930+0000 7f44d79db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd74f0 0x557228de84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.930+0000 7f44d79db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228ddc660 0x557228dd2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.930+0000 7f44d79db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f44d01a83d0 con 0x557228dd6b00 2024-03-18T22:43:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.930+0000 7f44d79db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f44d01a9f10 con 0x557228ddc660 2024-03-18T22:43:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.930+0000 7f44d79db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f44d01a8aa0 con 0x557228dd74f0 2024-03-18T22:43:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.930+0000 7f44ce59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228dc4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.930+0000 7f44cdd9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd74f0 0x557228de84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.930+0000 7f44ce59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228dc4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45886/0 (socket says 172.21.15.40:45886) 2024-03-18T22:43:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.930+0000 7f44ce59c640 1 -- 172.21.15.40:0/1616096494 learned_addr learned my addr 172.21.15.40:0/1616096494 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.930+0000 7f44ced9d640 1 --2- 172.21.15.40:0/1616096494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228ddc660 0x557228dd2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.931+0000 7f44cdd9b640 1 -- 172.21.15.40:0/1616096494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228ddc660 msgr2=0x557228dd2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.931+0000 7f44cdd9b640 1 --2- 172.21.15.40:0/1616096494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228ddc660 0x557228dd2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.931+0000 7f44cdd9b640 1 -- 172.21.15.40:0/1616096494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 msgr2=0x557228dc4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.931+0000 7f44cdd9b640 1 --2- 172.21.15.40:0/1616096494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228dc4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.931+0000 7f44cdd9b640 1 -- 172.21.15.40:0/1616096494 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557228d9f880 con 0x557228dd74f0 2024-03-18T22:43:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.931+0000 7f44ce59c640 1 --2- 172.21.15.40:0/1616096494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228dc4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.931+0000 7f44cdd9b640 1 --2- 172.21.15.40:0/1616096494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd74f0 0x557228de84d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f44c40cc7a0 tx=0x7f44c40cc620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3a19ed2010013bd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.931+0000 7f44b7fff640 1 -- 172.21.15.40:0/1616096494 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c4003b30 con 0x557228dd74f0 2024-03-18T22:43:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.931+0000 7f44b7fff640 1 -- 172.21.15.40:0/1616096494 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44c409bcf0 con 0x557228dd74f0 2024-03-18T22:43:41.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.931+0000 7f44b7fff640 1 -- 172.21.15.40:0/1616096494 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c4009310 con 0x557228dd74f0 2024-03-18T22:43:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.932+0000 7f44d79db740 1 -- 172.21.15.40:0/1616096494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd74f0 msgr2=0x557228de84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.932+0000 7f44d79db740 1 --2- 172.21.15.40:0/1616096494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd74f0 0x557228de84d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f44c40cc7a0 tx=0x7f44c40cc620 comp rx=0 tx=0).stop 2024-03-18T22:43:41.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.932+0000 7f44d79db740 1 -- 172.21.15.40:0/1616096494 shutdown_connections 2024-03-18T22:43:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.932+0000 7f44d79db740 1 --2- 172.21.15.40:0/1616096494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228ddc660 0x557228dd2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.932+0000 7f44d79db740 1 --2- 172.21.15.40:0/1616096494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd74f0 0x557228de84d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.932+0000 7f44d79db740 1 --2- 172.21.15.40:0/1616096494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228dd6b00 0x557228dc4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.932+0000 7f44d79db740 1 -- 172.21.15.40:0/1616096494 >> 172.21.15.40:0/1616096494 conn(0x557228dd2070 msgr2=0x557228dc8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.932+0000 7f44d79db740 1 -- 172.21.15.40:0/1616096494 shutdown_connections 2024-03-18T22:43:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.932+0000 7f44d79db740 1 -- 172.21.15.40:0/1616096494 wait complete. 2024-03-18T22:43:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.932+0000 7f44d79db740 1 -- start start 2024-03-18T22:43:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.932+0000 7f44d79db740 1 Processor -- start 2024-03-18T22:43:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44d79db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd6b00 0x557228de54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44d79db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228dd74f0 0x557228de59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44d79db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228ddc660 0x557228e51530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44d79db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f44d01a9840 con 0x557228ddc660 2024-03-18T22:43:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44d79db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f44d01a9170 con 0x557228dd74f0 2024-03-18T22:43:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44d79db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f44d01a63b0 con 0x557228dd6b00 2024-03-18T22:43:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44d79db740 10 client.0 _ll_register_callbacks cb 0x7f44c0011bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44cdd9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228dd74f0 0x557228de59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44ced9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228ddc660 0x557228e51530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44ce59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd6b00 0x557228de54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44cdd9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228dd74f0 0x557228de59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60256/0 (socket says 172.21.15.40:60256) 2024-03-18T22:43:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44cdd9b640 1 -- 172.21.15.40:0/3202070694 learned_addr learned my addr 172.21.15.40:0/3202070694 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44cdd9b640 1 -- 172.21.15.40:0/3202070694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd6b00 msgr2=0x557228de54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44cdd9b640 1 --2- 172.21.15.40:0/3202070694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd6b00 0x557228de54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44cdd9b640 1 -- 172.21.15.40:0/3202070694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228ddc660 msgr2=0x557228e51530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44cdd9b640 1 --2- 172.21.15.40:0/3202070694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228ddc660 0x557228e51530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44cdd9b640 1 -- 172.21.15.40:0/3202070694 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557228dd1c10 con 0x557228dd74f0 2024-03-18T22:43:41.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.933+0000 7f44ced9d640 1 --2- 172.21.15.40:0/3202070694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228ddc660 0x557228e51530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44cdd9b640 1 --2- 172.21.15.40:0/3202070694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228dd74f0 0x557228de59f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f44c4000c00 tx=0x7f44c4099790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b17480f369aca41a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:43:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44b5ffb640 1 -- 172.21.15.40:0/3202070694 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c409b5f0 con 0x557228dd74f0 2024-03-18T22:43:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44d79db740 1 -- 172.21.15.40:0/3202070694 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44c409bcf0 con 0x557228dd74f0 2024-03-18T22:43:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44b5ffb640 1 -- 172.21.15.40:0/3202070694 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f44c4097c60 con 0x557228dd74f0 2024-03-18T22:43:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44b5ffb640 1 -- 172.21.15.40:0/3202070694 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f44c409f740 con 0x557228dd74f0 2024-03-18T22:43:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44d79db740 10 client.25589 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44d79db740 1 -- 172.21.15.40:0/3202070694 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557228e52b40 con 0x557228dd74f0 2024-03-18T22:43:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44b5ffb640 1 -- 172.21.15.40:0/3202070694 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44c409bcf0 con 0x557228dd74f0 2024-03-18T22:43:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44b5ffb640 1 -- 172.21.15.40:0/3202070694 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f44c4084b80 con 0x557228dd74f0 2024-03-18T22:43:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44d79db740 10 client.25589 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44d79db740 10 client.25589 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:43:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44d79db740 20 client.25589 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44d79db740 1 --2- 172.21.15.40:0/3202070694 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557228ded170 0x557228e554c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44d79db740 4 client.25589 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:43:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.934+0000 7f44d79db740 1 -- 172.21.15.40:0/3202070694 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557228d3f9f0 con 0x557228ded170 2024-03-18T22:43:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44ce59c640 1 --2- 172.21.15.40:0/3202070694 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557228ded170 0x557228e554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44ce59c640 1 --2- 172.21.15.40:0/3202070694 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557228ded170 0x557228e554c0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2b3388b25bd98ab6 server_cookie=a009307bf679e78b in_seq=0 out_seq=0 2024-03-18T22:43:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44b5ffb640 10 client.25589 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:43:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44b5ffb640 1 -- 172.21.15.40:0/3202070694 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557228d3f9f0 con 0x557228ded170 2024-03-18T22:43:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44b5ffb640 10 client.25589 handle_command_reply: tid=0 2024-03-18T22:43:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44d79db740 1 client.25589 shutdown 2024-03-18T22:43:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44d79db740 1 -- 172.21.15.40:0/3202070694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228dd74f0 msgr2=0x557228de59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44d79db740 1 --2- 172.21.15.40:0/3202070694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228dd74f0 0x557228de59f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f44c4000c00 tx=0x7f44c4099790 comp rx=0 tx=0).stop 2024-03-18T22:43:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44d79db740 1 -- 172.21.15.40:0/3202070694 shutdown_connections 2024-03-18T22:43:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44d79db740 1 --2- 172.21.15.40:0/3202070694 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557228ded170 0x557228e554c0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44d79db740 1 --2- 172.21.15.40:0/3202070694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557228ddc660 0x557228e51530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44d79db740 1 --2- 172.21.15.40:0/3202070694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557228dd74f0 0x557228de59f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44d79db740 1 --2- 172.21.15.40:0/3202070694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557228dd6b00 0x557228de54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44d79db740 1 -- 172.21.15.40:0/3202070694 >> 172.21.15.40:0/3202070694 conn(0x557228dd2070 msgr2=0x557228dc8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44b5ffb640 0 client.25589 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:43:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44d79db740 1 -- 172.21.15.40:0/3202070694 shutdown_connections 2024-03-18T22:43:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44d79db740 1 -- 172.21.15.40:0/3202070694 wait complete. 2024-03-18T22:43:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.935+0000 7f44d79db740 20 client.25589 trim_cache size 0 max 16384 2024-03-18T22:43:41.943 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:43:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.936+0000 7f44d4d10640 1 -- 172.21.15.40:0/1876718307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f44b80a2800 msgr2=0x7f44b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.936+0000 7f44d4d10640 1 --2- 172.21.15.40:0/1876718307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f44b80a2800 0x7f44b80a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x557228dd7d50 tx=0x7f44c8009360 comp rx=0 tx=0).stop 2024-03-18T22:43:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.936+0000 7f44d4d10640 1 -- 172.21.15.40:0/1876718307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44d0101910 msgr2=0x7f44d0197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.936+0000 7f44d4d10640 1 --2- 172.21.15.40:0/1876718307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44d0101910 0x7f44d0197f30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f44c400e9f0 tx=0x7f44c400eeb0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.936+0000 7f44d4d10640 1 -- 172.21.15.40:0/1876718307 shutdown_connections 2024-03-18T22:43:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.936+0000 7f44d4d10640 1 --2- 172.21.15.40:0/1876718307 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f44b80a2800 0x7f44b80a4cc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.936+0000 7f44d4d10640 1 --2- 172.21.15.40:0/1876718307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d01022d0 0x7f44d019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.936+0000 7f44d4d10640 1 --2- 172.21.15.40:0/1876718307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f44d0101910 0x7f44d0197f30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.936+0000 7f44d4d10640 1 --2- 172.21.15.40:0/1876718307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44d0100720 0x7f44d01979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.936+0000 7f44d4d10640 1 -- 172.21.15.40:0/1876718307 >> 172.21.15.40:0/1876718307 conn(0x7f44d00fbc90 msgr2=0x7f44d0104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.936+0000 7f44d4d10640 1 -- 172.21.15.40:0/1876718307 shutdown_connections 2024-03-18T22:43:41.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:41.936+0000 7f44d4d10640 1 -- 172.21.15.40:0/1876718307 wait complete. 2024-03-18T22:43:41.947 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:43:42.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.057+0000 7f4b5968a640 1 -- 172.21.15.40:0/3141993613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b540ffad0 msgr2=0x7f4b540ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.057+0000 7f4b5968a640 1 --2- 172.21.15.40:0/3141993613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b540ffad0 0x7f4b540ffef0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4b44009890 tx=0x7f4b4402bb70 comp rx=0 tx=0).stop 2024-03-18T22:43:42.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.057+0000 7f4b5968a640 1 -- 172.21.15.40:0/3141993613 shutdown_connections 2024-03-18T22:43:42.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.057+0000 7f4b5968a640 1 --2- 172.21.15.40:0/3141993613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b54103460 0x7f4b54109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.057+0000 7f4b5968a640 1 --2- 172.21.15.40:0/3141993613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b54102aa0 0x7f4b54102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.057+0000 7f4b5968a640 1 --2- 172.21.15.40:0/3141993613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b540ffad0 0x7f4b540ffef0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.057+0000 7f4b5968a640 1 -- 172.21.15.40:0/3141993613 >> 172.21.15.40:0/3141993613 conn(0x7f4b540fb400 msgr2=0x7f4b540fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.057+0000 7f4b5968a640 1 -- 172.21.15.40:0/3141993613 shutdown_connections 2024-03-18T22:43:42.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.057+0000 7f4b5968a640 1 -- 172.21.15.40:0/3141993613 wait complete. 2024-03-18T22:43:42.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.057+0000 7f4b5968a640 1 Processor -- start 2024-03-18T22:43:42.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b5968a640 1 -- start start 2024-03-18T22:43:42.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b5968a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b54102aa0 0x7f4b54197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b5968a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b54103460 0x7f4b541981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b5968a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b5419d220 0x7f4b5419f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b5968a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5410d350 con 0x7f4b5419d220 2024-03-18T22:43:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b5968a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5410d1d0 con 0x7f4b54103460 2024-03-18T22:43:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b5968a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5410d4d0 con 0x7f4b54102aa0 2024-03-18T22:43:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b52ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b54102aa0 0x7f4b54197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b52ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b54102aa0 0x7f4b54197c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55884/0 (socket says 172.21.15.40:55884) 2024-03-18T22:43:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b52ffd640 1 -- 172.21.15.40:0/1662991792 learned_addr learned my addr 172.21.15.40:0/1662991792 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b527fc640 1 --2- 172.21.15.40:0/1662991792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b54103460 0x7f4b541981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b537fe640 1 --2- 172.21.15.40:0/1662991792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b5419d220 0x7f4b5419f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b52ffd640 1 -- 172.21.15.40:0/1662991792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b54103460 msgr2=0x7f4b541981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b52ffd640 1 --2- 172.21.15.40:0/1662991792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b54103460 0x7f4b541981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b52ffd640 1 -- 172.21.15.40:0/1662991792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b5419d220 msgr2=0x7f4b5419f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b52ffd640 1 --2- 172.21.15.40:0/1662991792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b5419d220 0x7f4b5419f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b52ffd640 1 -- 172.21.15.40:0/1662991792 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b5419fcb0 con 0x7f4b54102aa0 2024-03-18T22:43:42.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.058+0000 7f4b537fe640 1 --2- 172.21.15.40:0/1662991792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b5419d220 0x7f4b5419f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:43:42.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.059+0000 7f4b52ffd640 1 --2- 172.21.15.40:0/1662991792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b54102aa0 0x7f4b54197c90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4b44036000 tx=0x7f4b440040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.059+0000 7f4b3bfff640 1 -- 172.21.15.40:0/1662991792 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44041020 con 0x7f4b54102aa0 2024-03-18T22:43:42.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.059+0000 7f4b3bfff640 1 -- 172.21.15.40:0/1662991792 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b4402c9c0 con 0x7f4b54102aa0 2024-03-18T22:43:42.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.059+0000 7f4b3bfff640 1 -- 172.21.15.40:0/1662991792 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44039040 con 0x7f4b54102aa0 2024-03-18T22:43:42.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.059+0000 7f4b5968a640 1 -- 172.21.15.40:0/1662991792 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b5419ff40 con 0x7f4b54102aa0 2024-03-18T22:43:42.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.060+0000 7f4b5968a640 1 -- 172.21.15.40:0/1662991792 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b541a0350 con 0x7f4b54102aa0 2024-03-18T22:43:42.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.062+0000 7f4b5c270740 1 Processor -- start 2024-03-18T22:43:42.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.062+0000 7f4b3bfff640 1 -- 172.21.15.40:0/1662991792 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4b44044020 con 0x7f4b54102aa0 2024-03-18T22:43:42.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b5c270740 1 -- start start 2024-03-18T22:43:42.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b5c270740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574869882f0 0x557486988710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b5c270740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557486988ce0 0x55748698d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b5c270740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55748698de50 0x5574869902e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b5c270740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5574868ec4b0 con 0x5574869882f0 2024-03-18T22:43:42.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b5c270740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557486894d70 con 0x55748698de50 2024-03-18T22:43:42.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b5c270740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5574868e8550 con 0x557486988ce0 2024-03-18T22:43:42.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574869882f0 0x557486988710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b527fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55748698de50 0x5574869902e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574869882f0 0x557486988710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45932/0 (socket says 172.21.15.40:45932) 2024-03-18T22:43:42.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b537fe640 1 -- 172.21.15.40:0/1880477714 learned_addr learned my addr 172.21.15.40:0/1880477714 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b52ffd640 1 --2- 172.21.15.40:0/1880477714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557486988ce0 0x55748698d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b3bfff640 1 --2- 172.21.15.40:0/1662991792 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b3c0a2800 0x7f4b3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b527fc640 1 -- 172.21.15.40:0/1880477714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557486988ce0 msgr2=0x55748698d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b527fc640 1 --2- 172.21.15.40:0/1880477714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557486988ce0 0x55748698d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b527fc640 1 -- 172.21.15.40:0/1880477714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574869882f0 msgr2=0x557486988710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b527fc640 1 --2- 172.21.15.40:0/1880477714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574869882f0 0x557486988710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b527fc640 1 -- 172.21.15.40:0/1880477714 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574868b7210 con 0x55748698de50 2024-03-18T22:43:42.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.063+0000 7f4b537fe640 1 --2- 172.21.15.40:0/1880477714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574869882f0 0x557486988710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:42.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.064+0000 7f4b537fe640 1 --2- 172.21.15.40:0/1662991792 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b3c0a2800 0x7f4b3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.064+0000 7f4b527fc640 1 --2- 172.21.15.40:0/1880477714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55748698de50 0x5574869902e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009bf0 tx=0x7f4b48011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73fa2ffc8ce1d3b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.064+0000 7f4b397fa640 1 -- 172.21.15.40:0/1880477714 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b48004050 con 0x55748698de50 2024-03-18T22:43:42.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.064+0000 7f4b3bfff640 1 -- 172.21.15.40:0/1662991792 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4b440efbe0 con 0x7f4b54102aa0 2024-03-18T22:43:42.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.064+0000 7f4b397fa640 1 -- 172.21.15.40:0/1880477714 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b480054d0 con 0x55748698de50 2024-03-18T22:43:42.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.064+0000 7f4b397fa640 1 -- 172.21.15.40:0/1880477714 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b48004c70 con 0x55748698de50 2024-03-18T22:43:42.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.065+0000 7f4b5c270740 1 -- 172.21.15.40:0/1880477714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55748698de50 msgr2=0x5574869902e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.065+0000 7f4b5c270740 1 --2- 172.21.15.40:0/1880477714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55748698de50 0x5574869902e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009bf0 tx=0x7f4b48011810 comp rx=0 tx=0).stop 2024-03-18T22:43:42.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.065+0000 7f4b537fe640 1 --2- 172.21.15.40:0/1662991792 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b3c0a2800 0x7f4b3c0a4cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x557486989540 tx=0x7f4b4c011040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.065+0000 7f4b5c270740 1 -- 172.21.15.40:0/1880477714 shutdown_connections 2024-03-18T22:43:42.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.065+0000 7f4b5c270740 1 --2- 172.21.15.40:0/1880477714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55748698de50 0x5574869902e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.065+0000 7f4b5c270740 1 --2- 172.21.15.40:0/1880477714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557486988ce0 0x55748698d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.065+0000 7f4b5c270740 1 --2- 172.21.15.40:0/1880477714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5574869882f0 0x557486988710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.065+0000 7f4b5c270740 1 -- 172.21.15.40:0/1880477714 >> 172.21.15.40:0/1880477714 conn(0x557486983860 msgr2=0x557486983c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.065+0000 7f4b5c270740 1 -- 172.21.15.40:0/1880477714 shutdown_connections 2024-03-18T22:43:42.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.065+0000 7f4b5c270740 1 -- 172.21.15.40:0/1880477714 wait complete. 2024-03-18T22:43:42.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.065+0000 7f4b5c270740 1 -- start start 2024-03-18T22:43:42.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.065+0000 7f4b5c270740 1 Processor -- start 2024-03-18T22:43:42.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b5c270740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5574869882f0 0x5574869927f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b5c270740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557486988ce0 0x557486992d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b52ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5574869882f0 0x5574869927f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b52ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5574869882f0 0x5574869927f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55908/0 (socket says 172.21.15.40:55908) 2024-03-18T22:43:42.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b527fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557486988ce0 0x557486992d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b527fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557486988ce0 0x557486992d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45944/0 (socket says 172.21.15.40:45944) 2024-03-18T22:43:42.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b52ffd640 1 -- 172.21.15.40:0/3956720678 learned_addr learned my addr 172.21.15.40:0/3956720678 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b5c270740 1 --2- 172.21.15.40:0/3956720678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55748698de50 0x557486999d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b5c270740 1 -- 172.21.15.40:0/3956720678 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557486990e00 con 0x557486988ce0 2024-03-18T22:43:42.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b5c270740 1 -- 172.21.15.40:0/3956720678 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5574868a8310 con 0x55748698de50 2024-03-18T22:43:42.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b5c270740 1 -- 172.21.15.40:0/3956720678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557486990f80 con 0x5574869882f0 2024-03-18T22:43:42.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b537fe640 1 --2- 172.21.15.40:0/3956720678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55748698de50 0x557486999d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b52ffd640 1 -- 172.21.15.40:0/3956720678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55748698de50 msgr2=0x557486999d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b52ffd640 1 --2- 172.21.15.40:0/3956720678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55748698de50 0x557486999d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b52ffd640 1 -- 172.21.15.40:0/3956720678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557486988ce0 msgr2=0x557486992d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.066+0000 7f4b52ffd640 1 --2- 172.21.15.40:0/3956720678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557486988ce0 0x557486992d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.067+0000 7f4b52ffd640 1 -- 172.21.15.40:0/3956720678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574866003f0 con 0x5574869882f0 2024-03-18T22:43:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.067+0000 7f4b537fe640 1 --2- 172.21.15.40:0/3956720678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55748698de50 0x557486999d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:43:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.067+0000 7f4b527fc640 1 --2- 172.21.15.40:0/3956720678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557486988ce0 0x557486992d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.067+0000 7f4b5c270740 10 client.0 _ll_register_callbacks cb 0x55748689c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.067+0000 7f4b52ffd640 1 --2- 172.21.15.40:0/3956720678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5574869882f0 0x5574869927f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4b440b8b60 tx=0x7f4b44005b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dba993054543c1e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.067+0000 7f4b36ff5640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:43:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.067+0000 7f4b5c270740 1 -- 172.21.15.40:0/3956720678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b480054d0 con 0x5574869882f0 2024-03-18T22:43:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.067+0000 7f4b36ff5640 1 -- 172.21.15.40:0/3956720678 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b440bbf60 con 0x5574869882f0 2024-03-18T22:43:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.067+0000 7f4b36ff5640 1 -- 172.21.15.40:0/3956720678 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4b440ba490 con 0x5574869882f0 2024-03-18T22:43:42.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.067+0000 7f4b36ff5640 1 -- 172.21.15.40:0/3956720678 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4b440bbaf0 con 0x5574869882f0 2024-03-18T22:43:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.067+0000 7f4b5c270740 10 client.25579 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.067+0000 7f4b5c270740 1 -- 172.21.15.40:0/3956720678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({fsmap=28}) -- 0x5574866ca7d0 con 0x5574869882f0 2024-03-18T22:43:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.067+0000 7f4b36ff5640 1 -- 172.21.15.40:0/3956720678 <== mon.1 v2:172.21.15.113:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4b4403c030 con 0x5574869882f0 2024-03-18T22:43:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.067+0000 7f4b5c270740 10 client.25579 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.067+0000 7f4b5c270740 10 client.25579 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:43:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.068+0000 7f4b5c270740 20 client.25579 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.068+0000 7f4b5c270740 1 --2- 172.21.15.40:0/3956720678 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55748699e400 0x5574869a0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.068+0000 7f4b5c270740 4 client.25579 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:43:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.068+0000 7f4b5c270740 1 -- 172.21.15.40:0/3956720678 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55748694a3e0 con 0x55748699e400 2024-03-18T22:43:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.068+0000 7f4b527fc640 1 --2- 172.21.15.40:0/3956720678 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55748699e400 0x5574869a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.068+0000 7f4b36ff5640 10 client.25579 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:43:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.068+0000 7f4b527fc640 1 --2- 172.21.15.40:0/3956720678 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55748699e400 0x5574869a0890 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=45c234e70b36cded server_cookie=4a4bd7cb495a947c in_seq=0 out_seq=0 2024-03-18T22:43:42.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.069+0000 7f4b36ff5640 1 -- 172.21.15.40:0/3956720678 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55748694a3e0 con 0x55748699e400 2024-03-18T22:43:42.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.069+0000 7f4b36ff5640 10 client.25579 handle_command_reply: tid=0 2024-03-18T22:43:42.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.069+0000 7f4b5c270740 1 client.25579 shutdown 2024-03-18T22:43:42.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.069+0000 7f4b5c270740 1 -- 172.21.15.40:0/3956720678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5574869882f0 msgr2=0x5574869927f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.069+0000 7f4b5c270740 1 --2- 172.21.15.40:0/3956720678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5574869882f0 0x5574869927f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4b440b8b60 tx=0x7f4b44005b10 comp rx=0 tx=0).stop 2024-03-18T22:43:42.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.069+0000 7f4b5c270740 1 -- 172.21.15.40:0/3956720678 shutdown_connections 2024-03-18T22:43:42.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.069+0000 7f4b5c270740 1 --2- 172.21.15.40:0/3956720678 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55748699e400 0x5574869a0890 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.069+0000 7f4b36ff5640 0 client.25579 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:43:42.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.069+0000 7f4b5c270740 1 --2- 172.21.15.40:0/3956720678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55748698de50 0x557486999d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.069+0000 7f4b5c270740 1 --2- 172.21.15.40:0/3956720678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557486988ce0 0x557486992d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.069+0000 7f4b5c270740 1 --2- 172.21.15.40:0/3956720678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5574869882f0 0x5574869927f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.069+0000 7f4b5c270740 1 -- 172.21.15.40:0/3956720678 >> 172.21.15.40:0/3956720678 conn(0x557486983860 msgr2=0x557486982ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.069+0000 7f4b5c270740 1 -- 172.21.15.40:0/3956720678 shutdown_connections 2024-03-18T22:43:42.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.069+0000 7f4b5c270740 1 -- 172.21.15.40:0/3956720678 wait complete. 2024-03-18T22:43:42.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.069+0000 7f4b5c270740 20 client.25579 trim_cache size 0 max 16384 2024-03-18T22:43:42.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:41 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1361834099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:43:42.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b5c270740 1 Processor -- start 2024-03-18T22:43:42.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b5c270740 1 -- start start 2024-03-18T22:43:42.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b5c270740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5574869882f0 0x557486989700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b5c270740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557486988ce0 0x557486999d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b5c270740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55748698de50 0x557486983c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b5c270740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b541a8870 con 0x55748698de50 2024-03-18T22:43:42.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b5c270740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4b541aa3b0 con 0x5574869882f0 2024-03-18T22:43:42.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b5c270740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4b541a8f40 con 0x557486988ce0 2024-03-18T22:43:42.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b527fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557486988ce0 0x557486999d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b52ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5574869882f0 0x557486989700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b527fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557486988ce0 0x557486999d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55914/0 (socket says 172.21.15.40:55914) 2024-03-18T22:43:42.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b527fc640 1 -- 172.21.15.40:0/3058621832 learned_addr learned my addr 172.21.15.40:0/3058621832 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b52ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5574869882f0 0x557486989700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60304/0 (socket says 172.21.15.40:60304) 2024-03-18T22:43:42.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b52ffd640 1 -- 172.21.15.40:0/3058621832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557486988ce0 msgr2=0x557486999d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b52ffd640 1 --2- 172.21.15.40:0/3058621832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557486988ce0 0x557486999d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b52ffd640 1 -- 172.21.15.40:0/3058621832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55748698de50 msgr2=0x557486983c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b52ffd640 1 --2- 172.21.15.40:0/3058621832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55748698de50 0x557486983c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b52ffd640 1 -- 172.21.15.40:0/3058621832 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557486959880 con 0x5574869882f0 2024-03-18T22:43:42.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.083+0000 7f4b527fc640 1 --2- 172.21.15.40:0/3058621832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557486988ce0 0x557486999d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:42.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.084+0000 7f4b52ffd640 1 --2- 172.21.15.40:0/3058621832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5574869882f0 0x557486989700 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4b440dc2c0 tx=0x7f4b440f8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc09acbeee350dfe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.084+0000 7f4b38ff9640 1 -- 172.21.15.40:0/3058621832 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b440bbcd0 con 0x5574869882f0 2024-03-18T22:43:42.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.084+0000 7f4b38ff9640 1 -- 172.21.15.40:0/3058621832 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b480054d0 con 0x5574869882f0 2024-03-18T22:43:42.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.084+0000 7f4b38ff9640 1 -- 172.21.15.40:0/3058621832 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b440c02e0 con 0x5574869882f0 2024-03-18T22:43:42.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.084+0000 7f4b5c270740 1 -- 172.21.15.40:0/3058621832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5574869882f0 msgr2=0x557486989700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.084+0000 7f4b5c270740 1 --2- 172.21.15.40:0/3058621832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5574869882f0 0x557486989700 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4b440dc2c0 tx=0x7f4b440f8000 comp rx=0 tx=0).stop 2024-03-18T22:43:42.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.085+0000 7f4b5c270740 1 -- 172.21.15.40:0/3058621832 shutdown_connections 2024-03-18T22:43:42.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.085+0000 7f4b5c270740 1 --2- 172.21.15.40:0/3058621832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55748698de50 0x557486983c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.085+0000 7f4b5c270740 1 --2- 172.21.15.40:0/3058621832 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557486988ce0 0x557486999d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.085+0000 7f4b5c270740 1 --2- 172.21.15.40:0/3058621832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5574869882f0 0x557486989700 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.085+0000 7f4b5c270740 1 -- 172.21.15.40:0/3058621832 >> 172.21.15.40:0/3058621832 conn(0x557486983860 msgr2=0x557486987350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.085+0000 7f4b5c270740 1 -- 172.21.15.40:0/3058621832 shutdown_connections 2024-03-18T22:43:42.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.085+0000 7f4b5c270740 1 -- 172.21.15.40:0/3058621832 wait complete. 2024-03-18T22:43:42.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.085+0000 7f4b5c270740 1 -- start start 2024-03-18T22:43:42.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.085+0000 7f4b5c270740 1 Processor -- start 2024-03-18T22:43:42.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.085+0000 7f4b5c270740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5574869882f0 0x557486997580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.085+0000 7f4b5c270740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557486988ce0 0x557486997ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.085+0000 7f4b5c270740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55748698de50 0x557486998ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.085+0000 7f4b5c270740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b541a9ce0 con 0x557486988ce0 2024-03-18T22:43:42.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.085+0000 7f4b5c270740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4b541a9610 con 0x5574869882f0 2024-03-18T22:43:42.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.085+0000 7f4b5c270740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4b541a6850 con 0x55748698de50 2024-03-18T22:43:42.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b527fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557486988ce0 0x557486997ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b5c270740 10 client.0 _ll_register_callbacks cb 0x7f4b44005760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:42.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b52ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5574869882f0 0x557486997580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b527fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557486988ce0 0x557486997ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45960/0 (socket says 172.21.15.40:45960) 2024-03-18T22:43:42.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b527fc640 1 -- 172.21.15.40:0/1661849399 learned_addr learned my addr 172.21.15.40:0/1661849399 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b537fe640 1 --2- 172.21.15.40:0/1661849399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55748698de50 0x557486998ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b52ffd640 1 -- 172.21.15.40:0/1661849399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55748698de50 msgr2=0x557486998ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b52ffd640 1 --2- 172.21.15.40:0/1661849399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55748698de50 0x557486998ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b52ffd640 1 -- 172.21.15.40:0/1661849399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557486988ce0 msgr2=0x557486997ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b52ffd640 1 --2- 172.21.15.40:0/1661849399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557486988ce0 0x557486997ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b52ffd640 1 -- 172.21.15.40:0/1661849399 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574869990c0 con 0x5574869882f0 2024-03-18T22:43:42.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b527fc640 1 --2- 172.21.15.40:0/1661849399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557486988ce0 0x557486997ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:42.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b52ffd640 1 --2- 172.21.15.40:0/1661849399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5574869882f0 0x557486997580 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4b4402c150 tx=0x7f4b4402cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4df2ec1bd70beaa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b36ff5640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:43:42.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b36ff5640 1 -- 172.21.15.40:0/1661849399 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b440bbcd0 con 0x5574869882f0 2024-03-18T22:43:42.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b5c270740 1 -- 172.21.15.40:0/1661849399 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b480054d0 con 0x5574869882f0 2024-03-18T22:43:42.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b36ff5640 1 -- 172.21.15.40:0/1661849399 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4b440c46a0 con 0x5574869882f0 2024-03-18T22:43:42.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b36ff5640 1 -- 172.21.15.40:0/1661849399 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4b440c4930 con 0x5574869882f0 2024-03-18T22:43:42.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b5c270740 10 client.25604 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:42.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.086+0000 7f4b5c270740 1 -- 172.21.15.40:0/1661849399 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5574869990c0 con 0x5574869882f0 2024-03-18T22:43:42.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.087+0000 7f4b36ff5640 1 -- 172.21.15.40:0/1661849399 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b480054d0 con 0x5574869882f0 2024-03-18T22:43:42.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.087+0000 7f4b36ff5640 1 -- 172.21.15.40:0/1661849399 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4b440d7060 con 0x5574869882f0 2024-03-18T22:43:42.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.087+0000 7f4b5c270740 10 client.25604 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:42.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.087+0000 7f4b5c270740 10 client.25604 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:43:42.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.087+0000 7f4b5c270740 20 client.25604 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:42.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.087+0000 7f4b5c270740 1 --2- 172.21.15.40:0/1661849399 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x557486a04da0 0x557486a07190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.087+0000 7f4b5c270740 4 client.25604 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:43:42.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.087+0000 7f4b5c270740 1 -- 172.21.15.40:0/1661849399 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5574868f99f0 con 0x557486a04da0 2024-03-18T22:43:42.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.087+0000 7f4b527fc640 1 --2- 172.21.15.40:0/1661849399 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x557486a04da0 0x557486a07190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.087+0000 7f4b527fc640 1 --2- 172.21.15.40:0/1661849399 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x557486a04da0 0x557486a07190 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3a89de71f8af624c server_cookie=8af6275d214e51f7 in_seq=0 out_seq=0 2024-03-18T22:43:42.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.087+0000 7f4b36ff5640 10 client.25604 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:43:42.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b36ff5640 1 -- 172.21.15.40:0/1661849399 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5574868f99f0 con 0x557486a04da0 2024-03-18T22:43:42.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b36ff5640 10 client.25604 handle_command_reply: tid=0 2024-03-18T22:43:42.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b5c270740 1 client.25604 shutdown 2024-03-18T22:43:42.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b5c270740 1 -- 172.21.15.40:0/1661849399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5574869882f0 msgr2=0x557486997580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b5c270740 1 --2- 172.21.15.40:0/1661849399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5574869882f0 0x557486997580 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4b4402c150 tx=0x7f4b4402cc90 comp rx=0 tx=0).stop 2024-03-18T22:43:42.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b5c270740 1 -- 172.21.15.40:0/1661849399 shutdown_connections 2024-03-18T22:43:42.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b5c270740 1 --2- 172.21.15.40:0/1661849399 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x557486a04da0 0x557486a07190 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b36ff5640 0 client.25604 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:43:42.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b5c270740 1 --2- 172.21.15.40:0/1661849399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55748698de50 0x557486998ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b5c270740 1 --2- 172.21.15.40:0/1661849399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557486988ce0 0x557486997ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b5c270740 1 --2- 172.21.15.40:0/1661849399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5574869882f0 0x557486997580 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b5c270740 1 -- 172.21.15.40:0/1661849399 >> 172.21.15.40:0/1661849399 conn(0x557486983860 msgr2=0x557486987350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b5c270740 1 -- 172.21.15.40:0/1661849399 shutdown_connections 2024-03-18T22:43:42.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b5c270740 1 -- 172.21.15.40:0/1661849399 wait complete. 2024-03-18T22:43:42.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b5c270740 20 client.25604 trim_cache size 0 max 16384 2024-03-18T22:43:42.095 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:43:42.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.088+0000 7f4b5968a640 1 -- 172.21.15.40:0/1662991792 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b3c0a2800 msgr2=0x7f4b3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.089+0000 7f4b5968a640 1 --2- 172.21.15.40:0/1662991792 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b3c0a2800 0x7f4b3c0a4cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x557486989540 tx=0x7f4b4c011040 comp rx=0 tx=0).stop 2024-03-18T22:43:42.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.089+0000 7f4b5968a640 1 -- 172.21.15.40:0/1662991792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b54102aa0 msgr2=0x7f4b54197c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.089+0000 7f4b5968a640 1 --2- 172.21.15.40:0/1662991792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b54102aa0 0x7f4b54197c90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4b44036000 tx=0x7f4b440040c0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.089+0000 7f4b5968a640 1 -- 172.21.15.40:0/1662991792 shutdown_connections 2024-03-18T22:43:42.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.089+0000 7f4b5968a640 1 --2- 172.21.15.40:0/1662991792 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b3c0a2800 0x7f4b3c0a4cc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.089+0000 7f4b5968a640 1 --2- 172.21.15.40:0/1662991792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b5419d220 0x7f4b5419f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.089+0000 7f4b5968a640 1 --2- 172.21.15.40:0/1662991792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b54103460 0x7f4b541981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.089+0000 7f4b5968a640 1 --2- 172.21.15.40:0/1662991792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b54102aa0 0x7f4b54197c90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.089+0000 7f4b5968a640 1 -- 172.21.15.40:0/1662991792 >> 172.21.15.40:0/1662991792 conn(0x7f4b540fb400 msgr2=0x7f4b54105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.089+0000 7f4b5968a640 1 -- 172.21.15.40:0/1662991792 shutdown_connections 2024-03-18T22:43:42.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.089+0000 7f4b5968a640 1 -- 172.21.15.40:0/1662991792 wait complete. 2024-03-18T22:43:42.100 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:41 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1361834099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:43:42.101 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:43:42.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:41 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1361834099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:43:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.207+0000 7f6296636640 1 -- 172.21.15.40:0/1205166114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290101910 msgr2=0x7f6290101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.207+0000 7f6296636640 1 --2- 172.21.15.40:0/1205166114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290101910 0x7f6290101d90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6284009860 tx=0x7f628402bb70 comp rx=0 tx=0).stop 2024-03-18T22:43:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.207+0000 7f6296636640 1 -- 172.21.15.40:0/1205166114 shutdown_connections 2024-03-18T22:43:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.207+0000 7f6296636640 1 --2- 172.21.15.40:0/1205166114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62901022d0 0x7f6290108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.207+0000 7f6296636640 1 --2- 172.21.15.40:0/1205166114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290101910 0x7f6290101d90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.207+0000 7f6296636640 1 --2- 172.21.15.40:0/1205166114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6290100720 0x7f6290100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.207+0000 7f6296636640 1 -- 172.21.15.40:0/1205166114 >> 172.21.15.40:0/1205166114 conn(0x7f62900fbc90 msgr2=0x7f62900fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.207+0000 7f6296636640 1 -- 172.21.15.40:0/1205166114 shutdown_connections 2024-03-18T22:43:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.207+0000 7f6296636640 1 -- 172.21.15.40:0/1205166114 wait complete. 2024-03-18T22:43:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.208+0000 7f6296636640 1 Processor -- start 2024-03-18T22:43:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.208+0000 7f6296636640 1 -- start start 2024-03-18T22:43:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.208+0000 7f6296636640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290100720 0x7f629006a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.208+0000 7f6296636640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6290101910 0x7f629019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.208+0000 7f6296636640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62901022d0 0x7f629019ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.208+0000 7f6296636640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f629010c230 con 0x7f6290101910 2024-03-18T22:43:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.208+0000 7f6296636640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f629010c0b0 con 0x7f6290100720 2024-03-18T22:43:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.208+0000 7f6296636640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f629010c3b0 con 0x7f62901022d0 2024-03-18T22:43:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.208+0000 7f628f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6290101910 0x7f629019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.208+0000 7f628ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290100720 0x7f629006a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.208+0000 7f628ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290100720 0x7f629006a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60332/0 (socket says 172.21.15.40:60332) 2024-03-18T22:43:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.208+0000 7f628ffff640 1 -- 172.21.15.40:0/3693238243 learned_addr learned my addr 172.21.15.40:0/3693238243 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.209+0000 7f6294bd3640 1 --2- 172.21.15.40:0/3693238243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62901022d0 0x7f629019ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.209+0000 7f628ffff640 1 -- 172.21.15.40:0/3693238243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62901022d0 msgr2=0x7f629019ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.209+0000 7f628ffff640 1 --2- 172.21.15.40:0/3693238243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62901022d0 0x7f629019ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.209+0000 7f628ffff640 1 -- 172.21.15.40:0/3693238243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6290101910 msgr2=0x7f629019c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.209+0000 7f628ffff640 1 --2- 172.21.15.40:0/3693238243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6290101910 0x7f629019c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.209+0000 7f628ffff640 1 -- 172.21.15.40:0/3693238243 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f629019f650 con 0x7f6290100720 2024-03-18T22:43:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.209+0000 7f628ffff640 1 --2- 172.21.15.40:0/3693238243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290100720 0x7f629006a7d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f628000e9f0 tx=0x7f628000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.209+0000 7f628d7fa640 1 -- 172.21.15.40:0/3693238243 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628000ccc0 con 0x7f6290100720 2024-03-18T22:43:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.209+0000 7f6296636640 1 -- 172.21.15.40:0/3693238243 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f629019f940 con 0x7f6290100720 2024-03-18T22:43:42.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.209+0000 7f628d7fa640 1 -- 172.21.15.40:0/3693238243 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62800035f0 con 0x7f6290100720 2024-03-18T22:43:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.209+0000 7f628d7fa640 1 -- 172.21.15.40:0/3693238243 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628000ce60 con 0x7f6290100720 2024-03-18T22:43:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.209+0000 7f6296636640 1 -- 172.21.15.40:0/3693238243 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f629019feb0 con 0x7f6290100720 2024-03-18T22:43:42.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.212+0000 7f6299229740 1 Processor -- start 2024-03-18T22:43:42.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.212+0000 7f6299229740 1 -- start start 2024-03-18T22:43:42.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.212+0000 7f6299229740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b88e9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.212+0000 7f6299229740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88e9ce0 0x55f3b88ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.212+0000 7f6299229740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88eee50 0x55f3b88f12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.212+0000 7f6299229740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f3b884d4b0 con 0x55f3b88e92f0 2024-03-18T22:43:42.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.212+0000 7f6299229740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f3b87f5d70 con 0x55f3b88e9ce0 2024-03-18T22:43:42.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.212+0000 7f6299229740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f3b8849550 con 0x55f3b88eee50 2024-03-18T22:43:42.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.212+0000 7f6294bd3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b88e9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.212+0000 7f6294bd3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b88e9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45988/0 (socket says 172.21.15.40:45988) 2024-03-18T22:43:42.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.212+0000 7f6294bd3640 1 -- 172.21.15.40:0/3026740453 learned_addr learned my addr 172.21.15.40:0/3026740453 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.213+0000 7f628f7fe640 1 --2- 172.21.15.40:0/3026740453 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88eee50 0x55f3b88f12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.213+0000 7f628d7fa640 1 -- 172.21.15.40:0/3693238243 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6280010690 con 0x7f6290100720 2024-03-18T22:43:42.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.213+0000 7f6294bd3640 1 -- 172.21.15.40:0/3026740453 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88eee50 msgr2=0x55f3b88f12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.213+0000 7f6294bd3640 1 --2- 172.21.15.40:0/3026740453 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88eee50 0x55f3b88f12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.213+0000 7f6294bd3640 1 -- 172.21.15.40:0/3026740453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88e9ce0 msgr2=0x55f3b88ee7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:43:42.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.213+0000 7f6294bd3640 1 --2- 172.21.15.40:0/3026740453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88e9ce0 0x55f3b88ee7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.213+0000 7f6294bd3640 1 -- 172.21.15.40:0/3026740453 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3b8818210 con 0x55f3b88e92f0 2024-03-18T22:43:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.213+0000 7f6294bd3640 1 --2- 172.21.15.40:0/3026740453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b88e9710 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6288009fd0 tx=0x7f62880118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c98f5d1ca019defa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.213+0000 7f62767fc640 1 -- 172.21.15.40:0/3026740453 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6288004d20 con 0x55f3b88e92f0 2024-03-18T22:43:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.213+0000 7f62767fc640 1 -- 172.21.15.40:0/3026740453 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6288004050 con 0x55f3b88e92f0 2024-03-18T22:43:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.213+0000 7f628d7fa640 1 --2- 172.21.15.40:0/3693238243 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f62780a2800 0x7f62780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.213+0000 7f628d7fa640 1 -- 172.21.15.40:0/3693238243 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f62800cf0a0 con 0x7f6290100720 2024-03-18T22:43:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.214+0000 7f628ffff640 1 --2- 172.21.15.40:0/3693238243 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f62780a2800 0x7f62780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.214+0000 7f62767fc640 1 -- 172.21.15.40:0/3026740453 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6288005280 con 0x55f3b88e92f0 2024-03-18T22:43:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.214+0000 7f6299229740 1 -- 172.21.15.40:0/3026740453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 msgr2=0x55f3b88e9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.214+0000 7f628ffff640 1 --2- 172.21.15.40:0/3693238243 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f62780a2800 0x7f62780a4cc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6280097ab0 tx=0x7f6280097250 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.214+0000 7f6299229740 1 --2- 172.21.15.40:0/3026740453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b88e9710 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6288009fd0 tx=0x7f62880118d0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.214+0000 7f6299229740 1 -- 172.21.15.40:0/3026740453 shutdown_connections 2024-03-18T22:43:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.214+0000 7f6299229740 1 --2- 172.21.15.40:0/3026740453 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88eee50 0x55f3b88f12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.214+0000 7f6299229740 1 --2- 172.21.15.40:0/3026740453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88e9ce0 0x55f3b88ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.214+0000 7f6299229740 1 --2- 172.21.15.40:0/3026740453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b88e9710 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.214+0000 7f6299229740 1 -- 172.21.15.40:0/3026740453 >> 172.21.15.40:0/3026740453 conn(0x55f3b88e4860 msgr2=0x55f3b88e4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.214+0000 7f6299229740 1 -- 172.21.15.40:0/3026740453 shutdown_connections 2024-03-18T22:43:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.214+0000 7f6299229740 1 -- 172.21.15.40:0/3026740453 wait complete. 2024-03-18T22:43:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.215+0000 7f6299229740 1 -- start start 2024-03-18T22:43:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.215+0000 7f6299229740 1 Processor -- start 2024-03-18T22:43:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.215+0000 7f6299229740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e92f0 0x55f3b88ebc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.215+0000 7f6299229740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e9ce0 0x55f3b88ec160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.215+0000 7f6299229740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88eee50 0x55f3b88ed170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.215+0000 7f6299229740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f3b88f62b0 con 0x55f3b88e9ce0 2024-03-18T22:43:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.215+0000 7f6299229740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f3b8809310 con 0x55f3b88eee50 2024-03-18T22:43:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.215+0000 7f6299229740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f3b88f6430 con 0x55f3b88e92f0 2024-03-18T22:43:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.215+0000 7f6299229740 10 client.0 _ll_register_callbacks cb 0x55f3b87fd2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.215+0000 7f628f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e9ce0 0x55f3b88ec160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.215+0000 7f628f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e9ce0 0x55f3b88ec160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45992/0 (socket says 172.21.15.40:45992) 2024-03-18T22:43:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.215+0000 7f628f7fe640 1 -- 172.21.15.40:0/4033290574 learned_addr learned my addr 172.21.15.40:0/4033290574 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.215+0000 7f6294bd3640 1 --2- 172.21.15.40:0/4033290574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e92f0 0x55f3b88ebc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.215+0000 7f6294bd3640 1 --2- 172.21.15.40:0/4033290574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88eee50 0x55f3b88ed170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.216+0000 7f628f7fe640 1 -- 172.21.15.40:0/4033290574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e92f0 msgr2=0x55f3b88ebc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.216+0000 7f628f7fe640 1 --2- 172.21.15.40:0/4033290574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e92f0 0x55f3b88ebc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.216+0000 7f628f7fe640 1 -- 172.21.15.40:0/4033290574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88eee50 msgr2=0x55f3b88ed170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.216+0000 7f628f7fe640 1 --2- 172.21.15.40:0/4033290574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88eee50 0x55f3b88ed170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.216+0000 7f628f7fe640 1 -- 172.21.15.40:0/4033290574 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3b85613f0 con 0x55f3b88e9ce0 2024-03-18T22:43:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.216+0000 7f6294bd3640 1 --2- 172.21.15.40:0/4033290574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e92f0 0x55f3b88ebc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.216+0000 7f628f7fe640 1 --2- 172.21.15.40:0/4033290574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e9ce0 0x55f3b88ec160 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6284009860 tx=0x7f6284005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e804e2a7916b8b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.216+0000 7f626f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:43:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.216+0000 7f626f7fe640 1 -- 172.21.15.40:0/4033290574 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6284039040 con 0x55f3b88e9ce0 2024-03-18T22:43:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.216+0000 7f626f7fe640 1 -- 172.21.15.40:0/4033290574 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6284004240 con 0x55f3b88e9ce0 2024-03-18T22:43:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.216+0000 7f6299229740 1 -- 172.21.15.40:0/4033290574 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6288004050 con 0x55f3b88e9ce0 2024-03-18T22:43:42.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.216+0000 7f626f7fe640 1 -- 172.21.15.40:0/4033290574 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f628404e050 con 0x55f3b88e9ce0 2024-03-18T22:43:42.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.216+0000 7f6299229740 10 client.15726 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:42.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.216+0000 7f6299229740 1 -- 172.21.15.40:0/4033290574 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f3b862b7d0 con 0x55f3b88e9ce0 2024-03-18T22:43:42.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.216+0000 7f626f7fe640 1 -- 172.21.15.40:0/4033290574 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6288004050 con 0x55f3b88e9ce0 2024-03-18T22:43:42.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.217+0000 7f626f7fe640 1 -- 172.21.15.40:0/4033290574 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f62840038a0 con 0x55f3b88e9ce0 2024-03-18T22:43:42.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.217+0000 7f6299229740 10 client.15726 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:42.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.217+0000 7f6299229740 10 client.15726 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:43:42.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.217+0000 7f6299229740 20 client.15726 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:42.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.217+0000 7f6299229740 1 --2- 172.21.15.40:0/4033290574 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f3b8907ee0 0x55f3b890a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.217+0000 7f6299229740 4 client.15726 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:43:42.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.217+0000 7f6299229740 1 -- 172.21.15.40:0/4033290574 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3b88ab3e0 con 0x55f3b8907ee0 2024-03-18T22:43:42.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.217+0000 7f6294bd3640 1 --2- 172.21.15.40:0/4033290574 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f3b8907ee0 0x55f3b890a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.217+0000 7f6294bd3640 1 --2- 172.21.15.40:0/4033290574 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f3b8907ee0 0x55f3b890a320 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ea9c11ca2e3b452d server_cookie=f9e688deb4b17fef in_seq=0 out_seq=0 2024-03-18T22:43:42.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.217+0000 7f626f7fe640 10 client.15726 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:43:42.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.218+0000 7f626f7fe640 1 -- 172.21.15.40:0/4033290574 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f3b88ab3e0 con 0x55f3b8907ee0 2024-03-18T22:43:42.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.218+0000 7f626f7fe640 10 client.15726 handle_command_reply: tid=0 2024-03-18T22:43:42.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.218+0000 7f6299229740 1 client.15726 shutdown 2024-03-18T22:43:42.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.218+0000 7f6299229740 1 -- 172.21.15.40:0/4033290574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e9ce0 msgr2=0x55f3b88ec160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.218+0000 7f6299229740 1 --2- 172.21.15.40:0/4033290574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e9ce0 0x55f3b88ec160 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6284009860 tx=0x7f6284005b20 comp rx=0 tx=0).stop 2024-03-18T22:43:42.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.218+0000 7f6299229740 1 -- 172.21.15.40:0/4033290574 shutdown_connections 2024-03-18T22:43:42.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.218+0000 7f6299229740 1 --2- 172.21.15.40:0/4033290574 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f3b8907ee0 0x55f3b890a320 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.218+0000 7f6299229740 1 --2- 172.21.15.40:0/4033290574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88eee50 0x55f3b88ed170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.218+0000 7f6299229740 1 --2- 172.21.15.40:0/4033290574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e9ce0 0x55f3b88ec160 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.218+0000 7f6299229740 1 --2- 172.21.15.40:0/4033290574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e92f0 0x55f3b88ebc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.218+0000 7f6299229740 1 -- 172.21.15.40:0/4033290574 >> 172.21.15.40:0/4033290574 conn(0x55f3b88e4860 msgr2=0x55f3b88e3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.218+0000 7f626f7fe640 0 client.15726 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:43:42.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.218+0000 7f6299229740 1 -- 172.21.15.40:0/4033290574 shutdown_connections 2024-03-18T22:43:42.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.218+0000 7f6299229740 1 -- 172.21.15.40:0/4033290574 wait complete. 2024-03-18T22:43:42.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.219+0000 7f6299229740 20 client.15726 trim_cache size 0 max 16384 2024-03-18T22:43:42.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.245+0000 7f6299229740 1 Processor -- start 2024-03-18T22:43:42.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.245+0000 7f6299229740 1 -- start start 2024-03-18T22:43:42.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.245+0000 7f6299229740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88e4860 0x55f3b88f2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.245+0000 7f6299229740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b88f2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.245+0000 7f6299229740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e9ce0 0x55f3b88e3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.246+0000 7f6299229740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f62901a8320 con 0x55f3b88e92f0 2024-03-18T22:43:42.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.246+0000 7f6299229740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f62901a9e60 con 0x55f3b88e4860 2024-03-18T22:43:42.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.246+0000 7f628f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b88f2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.246+0000 7f628f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b88f2880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46000/0 (socket says 172.21.15.40:46000) 2024-03-18T22:43:42.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.246+0000 7f628f7fe640 1 -- 172.21.15.40:0/3020746146 learned_addr learned my addr 172.21.15.40:0/3020746146 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.246+0000 7f6294bd3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88e4860 0x55f3b88f2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.246+0000 7f6299229740 1 -- 172.21.15.40:0/3020746146 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f62901a89f0 con 0x55f3b88e9ce0 2024-03-18T22:43:42.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.246+0000 7f628f7fe640 1 -- 172.21.15.40:0/3020746146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e9ce0 msgr2=0x55f3b88e3310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:43:42.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.246+0000 7f628f7fe640 1 --2- 172.21.15.40:0/3020746146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e9ce0 0x55f3b88e3310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.246+0000 7f628f7fe640 1 -- 172.21.15.40:0/3020746146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88e4860 msgr2=0x55f3b88f2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.246+0000 7f628f7fe640 1 --2- 172.21.15.40:0/3020746146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88e4860 0x55f3b88f2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.246+0000 7f628f7fe640 1 -- 172.21.15.40:0/3020746146 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3b88e02c0 con 0x55f3b88e92f0 2024-03-18T22:43:42.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.246+0000 7f6294bd3640 1 --2- 172.21.15.40:0/3020746146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88e4860 0x55f3b88f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:43:42.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.246+0000 7f628f7fe640 1 --2- 172.21.15.40:0/3020746146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b88f2880 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f628402c0b0 tx=0x7f6284005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e67d62d049d2beaf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.247+0000 7f6275ffb640 1 -- 172.21.15.40:0/3020746146 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6284057070 con 0x55f3b88e92f0 2024-03-18T22:43:42.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.247+0000 7f6275ffb640 1 -- 172.21.15.40:0/3020746146 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62840096b0 con 0x55f3b88e92f0 2024-03-18T22:43:42.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.247+0000 7f6275ffb640 1 -- 172.21.15.40:0/3020746146 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6284039070 con 0x55f3b88e92f0 2024-03-18T22:43:42.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.248+0000 7f6299229740 1 -- 172.21.15.40:0/3020746146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 msgr2=0x55f3b88f2880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.248+0000 7f6299229740 1 --2- 172.21.15.40:0/3020746146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b88f2880 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f628402c0b0 tx=0x7f6284005f10 comp rx=0 tx=0).stop 2024-03-18T22:43:42.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.248+0000 7f6299229740 1 -- 172.21.15.40:0/3020746146 shutdown_connections 2024-03-18T22:43:42.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.248+0000 7f6299229740 1 --2- 172.21.15.40:0/3020746146 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e9ce0 0x55f3b88e3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.248+0000 7f6299229740 1 --2- 172.21.15.40:0/3020746146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b88f2880 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.248+0000 7f6299229740 1 --2- 172.21.15.40:0/3020746146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88e4860 0x55f3b88f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.248+0000 7f6299229740 1 -- 172.21.15.40:0/3020746146 >> 172.21.15.40:0/3020746146 conn(0x55f3b88eaaf0 msgr2=0x55f3b88ebc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.248+0000 7f6299229740 1 -- 172.21.15.40:0/3020746146 shutdown_connections 2024-03-18T22:43:42.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.248+0000 7f6299229740 1 -- 172.21.15.40:0/3020746146 wait complete. 2024-03-18T22:43:42.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.248+0000 7f6299229740 1 -- start start 2024-03-18T22:43:42.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.249+0000 7f6299229740 1 Processor -- start 2024-03-18T22:43:42.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.249+0000 7f6299229740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e4860 0x55f3b8902830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.249+0000 7f6299229740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b8902d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.249+0000 7f6294bd3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e4860 0x55f3b8902830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.249+0000 7f6294bd3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e4860 0x55f3b8902830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55970/0 (socket says 172.21.15.40:55970) 2024-03-18T22:43:42.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.249+0000 7f628f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b8902d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f628f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b8902d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46014/0 (socket says 172.21.15.40:46014) 2024-03-18T22:43:42.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f6294bd3640 1 -- 172.21.15.40:0/1397837732 learned_addr learned my addr 172.21.15.40:0/1397837732 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f6299229740 1 --2- 172.21.15.40:0/1397837732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88e9ce0 0x55f3b8903d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f6299229740 1 -- 172.21.15.40:0/1397837732 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f62901a9790 con 0x55f3b88e92f0 2024-03-18T22:43:42.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f6299229740 1 -- 172.21.15.40:0/1397837732 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f62901a90c0 con 0x55f3b88e9ce0 2024-03-18T22:43:42.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f6299229740 1 -- 172.21.15.40:0/1397837732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f62901a6300 con 0x55f3b88e4860 2024-03-18T22:43:42.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f6294bd3640 1 --2- 172.21.15.40:0/1397837732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88e9ce0 0x55f3b8903d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f6294bd3640 1 -- 172.21.15.40:0/1397837732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88e9ce0 msgr2=0x55f3b8903d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f6294bd3640 1 --2- 172.21.15.40:0/1397837732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88e9ce0 0x55f3b8903d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f6294bd3640 1 -- 172.21.15.40:0/1397837732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 msgr2=0x55f3b8902d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f6294bd3640 1 --2- 172.21.15.40:0/1397837732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b8902d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f6299229740 10 client.0 _ll_register_callbacks cb 0x7f62880093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f6294bd3640 1 -- 172.21.15.40:0/1397837732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3b8904320 con 0x55f3b88e4860 2024-03-18T22:43:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f6299229740 1 -- 172.21.15.40:0/1397837732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62840096b0 con 0x55f3b88e4860 2024-03-18T22:43:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f628f7fe640 1 --2- 172.21.15.40:0/1397837732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b8902d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f6294bd3640 1 --2- 172.21.15.40:0/1397837732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e4860 0x55f3b8902830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6288004750 tx=0x7f6288000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65c20e17b752ddc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.250+0000 7f626f7fe640 10 client.25600 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:43:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.251+0000 7f626f7fe640 1 -- 172.21.15.40:0/1397837732 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628802fa30 con 0x55f3b88e4860 2024-03-18T22:43:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.251+0000 7f626f7fe640 1 -- 172.21.15.40:0/1397837732 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f628802fbd0 con 0x55f3b88e4860 2024-03-18T22:43:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.251+0000 7f626f7fe640 1 -- 172.21.15.40:0/1397837732 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f628802fe10 con 0x55f3b88e4860 2024-03-18T22:43:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.251+0000 7f6299229740 10 client.25600 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.251+0000 7f6299229740 1 -- 172.21.15.40:0/1397837732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55f3b896e620 con 0x55f3b88e4860 2024-03-18T22:43:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.251+0000 7f626f7fe640 1 -- 172.21.15.40:0/1397837732 <== mon.1 v2:172.21.15.113:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6288025040 con 0x55f3b88e4860 2024-03-18T22:43:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.251+0000 7f6299229740 10 client.25600 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.251+0000 7f6299229740 10 client.25600 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:43:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.251+0000 7f6299229740 20 client.25600 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.251+0000 7f6299229740 1 --2- 172.21.15.40:0/1397837732 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f3b88eee50 0x55f3b8970c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.251+0000 7f6299229740 4 client.25600 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:43:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.251+0000 7f6299229740 1 -- 172.21.15.40:0/1397837732 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3b885a9f0 con 0x55f3b88eee50 2024-03-18T22:43:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f628f7fe640 1 --2- 172.21.15.40:0/1397837732 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f3b88eee50 0x55f3b8970c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f628f7fe640 1 --2- 172.21.15.40:0/1397837732 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f3b88eee50 0x55f3b8970c80 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1755be0e1f0a73a2 server_cookie=408e8f681c98f4e1 in_seq=0 out_seq=0 2024-03-18T22:43:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f626f7fe640 10 client.25600 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:43:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f626f7fe640 1 -- 172.21.15.40:0/1397837732 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3b885a9f0 con 0x55f3b88eee50 2024-03-18T22:43:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f626f7fe640 10 client.25600 handle_command_reply: tid=0 2024-03-18T22:43:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f6299229740 1 client.25600 shutdown 2024-03-18T22:43:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f6299229740 1 -- 172.21.15.40:0/1397837732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e4860 msgr2=0x55f3b8902830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f6299229740 1 --2- 172.21.15.40:0/1397837732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e4860 0x55f3b8902830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6288004750 tx=0x7f6288000f40 comp rx=0 tx=0).stop 2024-03-18T22:43:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f6299229740 1 -- 172.21.15.40:0/1397837732 shutdown_connections 2024-03-18T22:43:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f6299229740 1 --2- 172.21.15.40:0/1397837732 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f3b88eee50 0x55f3b8970c80 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f6299229740 1 --2- 172.21.15.40:0/1397837732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f3b88e9ce0 0x55f3b8903d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f626f7fe640 0 client.25600 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:43:42.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f6299229740 1 --2- 172.21.15.40:0/1397837732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f3b88e92f0 0x55f3b8902d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f6299229740 1 --2- 172.21.15.40:0/1397837732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f3b88e4860 0x55f3b8902830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f6299229740 1 -- 172.21.15.40:0/1397837732 >> 172.21.15.40:0/1397837732 conn(0x55f3b88eaaf0 msgr2=0x55f3b88e8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f6299229740 1 -- 172.21.15.40:0/1397837732 shutdown_connections 2024-03-18T22:43:42.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.252+0000 7f6299229740 1 -- 172.21.15.40:0/1397837732 wait complete. 2024-03-18T22:43:42.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.253+0000 7f6299229740 20 client.25600 trim_cache size 0 max 16384 2024-03-18T22:43:42.259 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:43:42.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.253+0000 7f6296636640 1 -- 172.21.15.40:0/3693238243 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f62780a2800 msgr2=0x7f62780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.253+0000 7f6296636640 1 --2- 172.21.15.40:0/3693238243 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f62780a2800 0x7f62780a4cc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6280097ab0 tx=0x7f6280097250 comp rx=0 tx=0).stop 2024-03-18T22:43:42.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.253+0000 7f6296636640 1 -- 172.21.15.40:0/3693238243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290100720 msgr2=0x7f629006a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.253+0000 7f6296636640 1 --2- 172.21.15.40:0/3693238243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290100720 0x7f629006a7d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f628000e9f0 tx=0x7f628000eeb0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.253+0000 7f6296636640 1 -- 172.21.15.40:0/3693238243 shutdown_connections 2024-03-18T22:43:42.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.253+0000 7f6296636640 1 --2- 172.21.15.40:0/3693238243 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f62780a2800 0x7f62780a4cc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.253+0000 7f6296636640 1 --2- 172.21.15.40:0/3693238243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f62901022d0 0x7f629019ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.253+0000 7f6296636640 1 --2- 172.21.15.40:0/3693238243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6290101910 0x7f629019c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.253+0000 7f6296636640 1 --2- 172.21.15.40:0/3693238243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6290100720 0x7f629006a7d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.253+0000 7f6296636640 1 -- 172.21.15.40:0/3693238243 >> 172.21.15.40:0/3693238243 conn(0x7f62900fbc90 msgr2=0x7f6290104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.253+0000 7f6296636640 1 -- 172.21.15.40:0/3693238243 shutdown_connections 2024-03-18T22:43:42.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.253+0000 7f6296636640 1 -- 172.21.15.40:0/3693238243 wait complete. 2024-03-18T22:43:42.265 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:43:42.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.370+0000 7f59e2da9640 1 -- 172.21.15.40:0/1446903508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59dc104c90 msgr2=0x7f59dc107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.370+0000 7f59e2da9640 1 --2- 172.21.15.40:0/1446903508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59dc104c90 0x7f59dc107080 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f59c8009900 tx=0x7f59c802bc30 comp rx=0 tx=0).stop 2024-03-18T22:43:42.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.371+0000 7f59e2da9640 1 -- 172.21.15.40:0/1446903508 shutdown_connections 2024-03-18T22:43:42.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.371+0000 7f59e2da9640 1 --2- 172.21.15.40:0/1446903508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59dc1075c0 0x7f59dc1099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.371+0000 7f59e2da9640 1 --2- 172.21.15.40:0/1446903508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59dc104c90 0x7f59dc107080 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.371+0000 7f59e2da9640 1 --2- 172.21.15.40:0/1446903508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc102360 0x7f59dc104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.371+0000 7f59e2da9640 1 -- 172.21.15.40:0/1446903508 >> 172.21.15.40:0/1446903508 conn(0x7f59dc0fbc90 msgr2=0x7f59dc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.371+0000 7f59e2da9640 1 -- 172.21.15.40:0/1446903508 shutdown_connections 2024-03-18T22:43:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.371+0000 7f59e2da9640 1 -- 172.21.15.40:0/1446903508 wait complete. 2024-03-18T22:43:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.371+0000 7f59e2da9640 1 Processor -- start 2024-03-18T22:43:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.371+0000 7f59e2da9640 1 -- start start 2024-03-18T22:43:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.371+0000 7f59e2da9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59dc102360 0x7f59dc197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.371+0000 7f59e2da9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59dc104c90 0x7f59dc197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.371+0000 7f59e2da9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc1075c0 0x7f59dc19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.371+0000 7f59e2da9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc10d350 con 0x7f59dc102360 2024-03-18T22:43:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.371+0000 7f59e2da9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc10d1d0 con 0x7f59dc1075c0 2024-03-18T22:43:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.371+0000 7f59e2da9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc10d4d0 con 0x7f59dc104c90 2024-03-18T22:43:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59e0b45640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59dc102360 0x7f59dc197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59dbfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59dc104c90 0x7f59dc197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59dbfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59dc104c90 0x7f59dc197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55982/0 (socket says 172.21.15.40:55982) 2024-03-18T22:43:42.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59e0b45640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59dc102360 0x7f59dc197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46030/0 (socket says 172.21.15.40:46030) 2024-03-18T22:43:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59dbfff640 1 -- 172.21.15.40:0/3223388852 learned_addr learned my addr 172.21.15.40:0/3223388852 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59e1346640 1 --2- 172.21.15.40:0/3223388852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc1075c0 0x7f59dc19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59dbfff640 1 -- 172.21.15.40:0/3223388852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc1075c0 msgr2=0x7f59dc19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59dbfff640 1 --2- 172.21.15.40:0/3223388852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc1075c0 0x7f59dc19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59dbfff640 1 -- 172.21.15.40:0/3223388852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59dc102360 msgr2=0x7f59dc197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59dbfff640 1 --2- 172.21.15.40:0/3223388852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59dc102360 0x7f59dc197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59dbfff640 1 -- 172.21.15.40:0/3223388852 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59dc19f6d0 con 0x7f59dc104c90 2024-03-18T22:43:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59e0b45640 1 --2- 172.21.15.40:0/3223388852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59dc102360 0x7f59dc197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59dbfff640 1 --2- 172.21.15.40:0/3223388852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59dc104c90 0x7f59dc197ff0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f59c8000c00 tx=0x7f59c8005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59d9ffb640 1 -- 172.21.15.40:0/3223388852 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c8043020 con 0x7f59dc104c90 2024-03-18T22:43:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59e2da9640 1 -- 172.21.15.40:0/3223388852 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59dc100150 con 0x7f59dc104c90 2024-03-18T22:43:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59d9ffb640 1 -- 172.21.15.40:0/3223388852 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59c8004350 con 0x7f59dc104c90 2024-03-18T22:43:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.372+0000 7f59d9ffb640 1 -- 172.21.15.40:0/3223388852 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c8003b80 con 0x7f59dc104c90 2024-03-18T22:43:42.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.373+0000 7f59e2da9640 1 -- 172.21.15.40:0/3223388852 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59dc1006f0 con 0x7f59dc104c90 2024-03-18T22:43:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.375+0000 7f59d9ffb640 1 -- 172.21.15.40:0/3223388852 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f59c8046020 con 0x7f59dc104c90 2024-03-18T22:43:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.375+0000 7f59d9ffb640 1 --2- 172.21.15.40:0/3223388852 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f59cc0a2880 0x7f59cc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.375+0000 7f59e0b45640 1 --2- 172.21.15.40:0/3223388852 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f59cc0a2880 0x7f59cc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.375+0000 7f59d9ffb640 1 -- 172.21.15.40:0/3223388852 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f59c80f0f00 con 0x7f59dc104c90 2024-03-18T22:43:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.376+0000 7f59e5993740 1 Processor -- start 2024-03-18T22:43:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.376+0000 7f59e0b45640 1 --2- 172.21.15.40:0/3223388852 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f59cc0a2880 0x7f59cc0a4d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009fc0 tx=0x7f59d0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.376+0000 7f59e5993740 1 -- start start 2024-03-18T22:43:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.376+0000 7f59e5993740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14ee7b00 0x556a14ee7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.376+0000 7f59e5993740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee84f0 0x556a14eecff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.376+0000 7f59e5993740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14eed660 0x556a14eefaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.376+0000 7f59e5993740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556a14e434b0 con 0x556a14ee84f0 2024-03-18T22:43:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.376+0000 7f59e5993740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556a14debd70 con 0x556a14eed660 2024-03-18T22:43:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.376+0000 7f59e5993740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556a14e3f550 con 0x556a14ee7b00 2024-03-18T22:43:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.376+0000 7f59e0b45640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14ee7b00 0x556a14ee7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.376+0000 7f59dbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee84f0 0x556a14eecff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.376+0000 7f59e0b45640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14ee7b00 0x556a14ee7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55988/0 (socket says 172.21.15.40:55988) 2024-03-18T22:43:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.376+0000 7f59e0b45640 1 -- 172.21.15.40:0/1190621180 learned_addr learned my addr 172.21.15.40:0/1190621180 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.376+0000 7f59e1346640 1 --2- 172.21.15.40:0/1190621180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14eed660 0x556a14eefaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.377+0000 7f59e0b45640 1 -- 172.21.15.40:0/1190621180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14eed660 msgr2=0x556a14eefaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.377+0000 7f59e0b45640 1 --2- 172.21.15.40:0/1190621180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14eed660 0x556a14eefaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.377+0000 7f59e0b45640 1 -- 172.21.15.40:0/1190621180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee84f0 msgr2=0x556a14eecff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.377+0000 7f59e0b45640 1 --2- 172.21.15.40:0/1190621180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee84f0 0x556a14eecff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.377+0000 7f59e0b45640 1 -- 172.21.15.40:0/1190621180 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a14e0e210 con 0x556a14ee7b00 2024-03-18T22:43:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.377+0000 7f59e0b45640 1 --2- 172.21.15.40:0/1190621180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14ee7b00 0x556a14ee7f20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009ba0 tx=0x7f59d0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa99ee85158fbda1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.377+0000 7f59c6ffd640 1 -- 172.21.15.40:0/1190621180 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0026480 con 0x556a14ee7b00 2024-03-18T22:43:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.377+0000 7f59c6ffd640 1 -- 172.21.15.40:0/1190621180 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59d0026ad0 con 0x556a14ee7b00 2024-03-18T22:43:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.377+0000 7f59c6ffd640 1 -- 172.21.15.40:0/1190621180 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d000fc40 con 0x556a14ee7b00 2024-03-18T22:43:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.378+0000 7f59e5993740 1 -- 172.21.15.40:0/1190621180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14ee7b00 msgr2=0x556a14ee7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.378+0000 7f59e5993740 1 --2- 172.21.15.40:0/1190621180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14ee7b00 0x556a14ee7f20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009ba0 tx=0x7f59d0014000 comp rx=0 tx=0).stop 2024-03-18T22:43:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.378+0000 7f59e5993740 1 -- 172.21.15.40:0/1190621180 shutdown_connections 2024-03-18T22:43:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.378+0000 7f59e5993740 1 --2- 172.21.15.40:0/1190621180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14eed660 0x556a14eefaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.378+0000 7f59e5993740 1 --2- 172.21.15.40:0/1190621180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee84f0 0x556a14eecff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.378+0000 7f59e5993740 1 --2- 172.21.15.40:0/1190621180 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14ee7b00 0x556a14ee7f20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.378+0000 7f59e5993740 1 -- 172.21.15.40:0/1190621180 >> 172.21.15.40:0/1190621180 conn(0x556a14eda860 msgr2=0x556a14edac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.378+0000 7f59e5993740 1 -- 172.21.15.40:0/1190621180 shutdown_connections 2024-03-18T22:43:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.378+0000 7f59e5993740 1 -- 172.21.15.40:0/1190621180 wait complete. 2024-03-18T22:43:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.379+0000 7f59e5993740 1 -- start start 2024-03-18T22:43:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.379+0000 7f59e5993740 1 Processor -- start 2024-03-18T22:43:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.379+0000 7f59e5993740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee7b00 0x556a14ef2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.379+0000 7f59e5993740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee84f0 0x556a14ef25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.379+0000 7f59e5993740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14eed660 0x556a14ef95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.379+0000 7f59e5993740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556a14ef0610 con 0x556a14ee7b00 2024-03-18T22:43:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.379+0000 7f59e0b45640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee7b00 0x556a14ef2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59e5993740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556a14dff310 con 0x556a14ee84f0 2024-03-18T22:43:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59e1346640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14eed660 0x556a14ef95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59dbfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee84f0 0x556a14ef25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59e0b45640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee7b00 0x556a14ef2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46040/0 (socket says 172.21.15.40:46040) 2024-03-18T22:43:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59e0b45640 1 -- 172.21.15.40:0/1605579349 learned_addr learned my addr 172.21.15.40:0/1605579349 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59e5993740 1 -- 172.21.15.40:0/1605579349 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556a14ef0790 con 0x556a14eed660 2024-03-18T22:43:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59e5993740 10 client.0 _ll_register_callbacks cb 0x556a14df32d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59e1346640 1 -- 172.21.15.40:0/1605579349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee84f0 msgr2=0x556a14ef25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59e1346640 1 --2- 172.21.15.40:0/1605579349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee84f0 0x556a14ef25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59e1346640 1 -- 172.21.15.40:0/1605579349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee7b00 msgr2=0x556a14ef2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59e1346640 1 --2- 172.21.15.40:0/1605579349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee7b00 0x556a14ef2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59e1346640 1 -- 172.21.15.40:0/1605579349 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a14b573f0 con 0x556a14eed660 2024-03-18T22:43:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59e5993740 1 -- 172.21.15.40:0/1605579349 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59d0026ad0 con 0x556a14eed660 2024-03-18T22:43:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59dbfff640 1 --2- 172.21.15.40:0/1605579349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee84f0 0x556a14ef25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59e1346640 1 --2- 172.21.15.40:0/1605579349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14eed660 0x556a14ef95f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f59d4005fd0 tx=0x7f59d400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30609df66cc381ad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59bbfff640 10 client.25612 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:43:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59bbfff640 1 -- 172.21.15.40:0/1605579349 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4009e30 con 0x556a14eed660 2024-03-18T22:43:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.380+0000 7f59bbfff640 1 -- 172.21.15.40:0/1605579349 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f59d4004b20 con 0x556a14eed660 2024-03-18T22:43:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.381+0000 7f59bbfff640 1 -- 172.21.15.40:0/1605579349 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f59d4004d60 con 0x556a14eed660 2024-03-18T22:43:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.381+0000 7f59e5993740 10 client.25612 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.381+0000 7f59e5993740 1 -- 172.21.15.40:0/1605579349 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({fsmap=28}) -- 0x556a14c217d0 con 0x556a14eed660 2024-03-18T22:43:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.381+0000 7f59bbfff640 1 -- 172.21.15.40:0/1605579349 <== mon.1 v2:172.21.15.113:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f59d4019070 con 0x556a14eed660 2024-03-18T22:43:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.381+0000 7f59e5993740 10 client.25612 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.381+0000 7f59e5993740 10 client.25612 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:43:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.381+0000 7f59e5993740 20 client.25612 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.381+0000 7f59e5993740 1 --2- 172.21.15.40:0/1605579349 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556a14efde70 0x556a14f00280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.381+0000 7f59e5993740 4 client.25612 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:43:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.381+0000 7f59e5993740 1 -- 172.21.15.40:0/1605579349 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a14ea13e0 con 0x556a14efde70 2024-03-18T22:43:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.381+0000 7f59e0b45640 1 --2- 172.21.15.40:0/1605579349 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556a14efde70 0x556a14f00280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.382+0000 7f59bbfff640 10 client.25612 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:43:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.382+0000 7f59e0b45640 1 --2- 172.21.15.40:0/1605579349 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556a14efde70 0x556a14f00280 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=55f3d4039dfa7246 server_cookie=2e2aac9c2eaa6bbf in_seq=0 out_seq=0 2024-03-18T22:43:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.383+0000 7f59bbfff640 1 -- 172.21.15.40:0/1605579349 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556a14ea13e0 con 0x556a14efde70 2024-03-18T22:43:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.383+0000 7f59bbfff640 10 client.25612 handle_command_reply: tid=0 2024-03-18T22:43:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.383+0000 7f59e5993740 1 client.25612 shutdown 2024-03-18T22:43:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.383+0000 7f59e5993740 1 -- 172.21.15.40:0/1605579349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14eed660 msgr2=0x556a14ef95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.383+0000 7f59e5993740 1 --2- 172.21.15.40:0/1605579349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14eed660 0x556a14ef95f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f59d4005fd0 tx=0x7f59d400f790 comp rx=0 tx=0).stop 2024-03-18T22:43:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.383+0000 7f59e5993740 1 -- 172.21.15.40:0/1605579349 shutdown_connections 2024-03-18T22:43:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.383+0000 7f59e5993740 1 --2- 172.21.15.40:0/1605579349 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556a14efde70 0x556a14f00280 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.383+0000 7f59bbfff640 0 client.25612 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:43:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.383+0000 7f59e5993740 1 --2- 172.21.15.40:0/1605579349 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14eed660 0x556a14ef95f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.383+0000 7f59e5993740 1 --2- 172.21.15.40:0/1605579349 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee84f0 0x556a14ef25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.383+0000 7f59e5993740 1 --2- 172.21.15.40:0/1605579349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee7b00 0x556a14ef2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.383+0000 7f59e5993740 1 -- 172.21.15.40:0/1605579349 >> 172.21.15.40:0/1605579349 conn(0x556a14eda860 msgr2=0x556a14ed9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.383+0000 7f59e5993740 1 -- 172.21.15.40:0/1605579349 shutdown_connections 2024-03-18T22:43:42.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.383+0000 7f59e5993740 1 -- 172.21.15.40:0/1605579349 wait complete. 2024-03-18T22:43:42.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.384+0000 7f59e5993740 20 client.25612 trim_cache size 0 max 16384 2024-03-18T22:43:42.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.410+0000 7f59e5993740 1 Processor -- start 2024-03-18T22:43:42.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.410+0000 7f59e5993740 1 -- start start 2024-03-18T22:43:42.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.410+0000 7f59e5993740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee7b00 0x556a14ef2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.410+0000 7f59e5993740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee84f0 0x556a14ef95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59e5993740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14eed660 0x556a14ef25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59dbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee84f0 0x556a14ef95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59e0b45640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee7b00 0x556a14ef2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59dbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee84f0 0x556a14ef95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46042/0 (socket says 172.21.15.40:46042) 2024-03-18T22:43:42.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59e0b45640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee7b00 0x556a14ef2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60404/0 (socket says 172.21.15.40:60404) 2024-03-18T22:43:42.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59dbfff640 1 -- 172.21.15.40:0/3720219475 learned_addr learned my addr 172.21.15.40:0/3720219475 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59e5993740 1 -- 172.21.15.40:0/3720219475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc1a83e0 con 0x556a14ee84f0 2024-03-18T22:43:42.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59e5993740 1 -- 172.21.15.40:0/3720219475 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc1a9f20 con 0x556a14ee7b00 2024-03-18T22:43:42.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59e5993740 1 -- 172.21.15.40:0/3720219475 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc1a8ab0 con 0x556a14eed660 2024-03-18T22:43:42.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59e1346640 1 --2- 172.21.15.40:0/3720219475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14eed660 0x556a14ef25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59e0b45640 1 -- 172.21.15.40:0/3720219475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14eed660 msgr2=0x556a14ef25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59e0b45640 1 --2- 172.21.15.40:0/3720219475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14eed660 0x556a14ef25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59e0b45640 1 -- 172.21.15.40:0/3720219475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee84f0 msgr2=0x556a14ef95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59e0b45640 1 --2- 172.21.15.40:0/3720219475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee84f0 0x556a14ef95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59e0b45640 1 -- 172.21.15.40:0/3720219475 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a14ee88d0 con 0x556a14ee7b00 2024-03-18T22:43:42.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59dbfff640 1 --2- 172.21.15.40:0/3720219475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee84f0 0x556a14ef95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:42.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.411+0000 7f59e0b45640 1 --2- 172.21.15.40:0/3720219475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee7b00 0x556a14ef2070 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x556a14f00a90 tx=0x7f59d00043c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edfd43fd65042608 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.412+0000 7f59c67fc640 1 -- 172.21.15.40:0/3720219475 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0037030 con 0x556a14ee7b00 2024-03-18T22:43:42.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.412+0000 7f59c67fc640 1 -- 172.21.15.40:0/3720219475 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59d00054f0 con 0x556a14ee7b00 2024-03-18T22:43:42.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.412+0000 7f59c67fc640 1 -- 172.21.15.40:0/3720219475 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d000f980 con 0x556a14ee7b00 2024-03-18T22:43:42.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.413+0000 7f59e5993740 1 -- 172.21.15.40:0/3720219475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee7b00 msgr2=0x556a14ef2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.413+0000 7f59e5993740 1 --2- 172.21.15.40:0/3720219475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee7b00 0x556a14ef2070 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x556a14f00a90 tx=0x7f59d00043c0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.413+0000 7f59e5993740 1 -- 172.21.15.40:0/3720219475 shutdown_connections 2024-03-18T22:43:42.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.413+0000 7f59e5993740 1 --2- 172.21.15.40:0/3720219475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14eed660 0x556a14ef25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.413+0000 7f59e5993740 1 --2- 172.21.15.40:0/3720219475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14ee84f0 0x556a14ef95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.413+0000 7f59e5993740 1 --2- 172.21.15.40:0/3720219475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee7b00 0x556a14ef2070 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.413+0000 7f59e5993740 1 -- 172.21.15.40:0/3720219475 >> 172.21.15.40:0/3720219475 conn(0x556a14eda860 msgr2=0x556a14edac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.413+0000 7f59e5993740 1 -- 172.21.15.40:0/3720219475 shutdown_connections 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.413+0000 7f59e5993740 1 -- 172.21.15.40:0/3720219475 wait complete. 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.413+0000 7f59e5993740 1 -- start start 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.413+0000 7f59e5993740 1 Processor -- start 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e5993740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14ee7b00 0x556a14ef6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e5993740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee84f0 0x556a14ef6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e5993740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14eed660 0x556a14f62110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e5993740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc1a9850 con 0x556a14eed660 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e0b45640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14ee7b00 0x556a14ef6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e5993740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc1a9180 con 0x556a14ee84f0 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e0b45640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14ee7b00 0x556a14ef6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:56008/0 (socket says 172.21.15.40:56008) 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e5993740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f59dc1a63c0 con 0x556a14ee7b00 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59dbfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee84f0 0x556a14ef6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e0b45640 1 -- 172.21.15.40:0/1668157553 learned_addr learned my addr 172.21.15.40:0/1668157553 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e1346640 1 --2- 172.21.15.40:0/1668157553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14eed660 0x556a14f62110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e5993740 10 client.0 _ll_register_callbacks cb 0x7f59d4009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e0b45640 1 -- 172.21.15.40:0/1668157553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee84f0 msgr2=0x556a14ef6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e0b45640 1 --2- 172.21.15.40:0/1668157553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee84f0 0x556a14ef6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e0b45640 1 -- 172.21.15.40:0/1668157553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14eed660 msgr2=0x556a14f62110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e0b45640 1 --2- 172.21.15.40:0/1668157553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14eed660 0x556a14f62110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e0b45640 1 -- 172.21.15.40:0/1668157553 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a14f62650 con 0x556a14ee7b00 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e5993740 1 -- 172.21.15.40:0/1668157553 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59d00054f0 con 0x556a14ee7b00 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.414+0000 7f59e1346640 1 --2- 172.21.15.40:0/1668157553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14eed660 0x556a14f62110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.415+0000 7f59e0b45640 1 --2- 172.21.15.40:0/1668157553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14ee7b00 0x556a14ef6540 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009e50 tx=0x7f59d0005730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c842ec44a242b735 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.415+0000 7f59bbfff640 10 client.25618 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.415+0000 7f59bbfff640 1 -- 172.21.15.40:0/1668157553 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0037040 con 0x556a14ee7b00 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.415+0000 7f59bbfff640 1 -- 172.21.15.40:0/1668157553 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f59d0005030 con 0x556a14ee7b00 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.415+0000 7f59bbfff640 1 -- 172.21.15.40:0/1668157553 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f59d0005270 con 0x556a14ee7b00 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.415+0000 7f59e5993740 10 client.25618 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.415+0000 7f59e5993740 1 -- 172.21.15.40:0/1668157553 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({fsmap=28}) -- 0x556a14f63850 con 0x556a14ee7b00 2024-03-18T22:43:42.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.415+0000 7f59bbfff640 1 -- 172.21.15.40:0/1668157553 <== mon.1 v2:172.21.15.113:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f59d004c020 con 0x556a14ee7b00 2024-03-18T22:43:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.415+0000 7f59e5993740 10 client.25618 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.415+0000 7f59e5993740 10 client.25618 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:43:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.416+0000 7f59e5993740 20 client.25618 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.416+0000 7f59e5993740 1 --2- 172.21.15.40:0/1668157553 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556a14f63d40 0x556a14f66130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.416+0000 7f59e5993740 4 client.25618 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:43:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.416+0000 7f59e5993740 1 -- 172.21.15.40:0/1668157553 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556a14e509f0 con 0x556a14f63d40 2024-03-18T22:43:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.416+0000 7f59dbfff640 1 --2- 172.21.15.40:0/1668157553 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556a14f63d40 0x556a14f66130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.416+0000 7f59dbfff640 1 --2- 172.21.15.40:0/1668157553 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556a14f63d40 0x556a14f66130 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2fb907ae7a6b3831 server_cookie=35430de3b57e69d7 in_seq=0 out_seq=0 2024-03-18T22:43:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.416+0000 7f59bbfff640 10 client.25618 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:43:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.416+0000 7f59bbfff640 1 -- 172.21.15.40:0/1668157553 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556a14e509f0 con 0x556a14f63d40 2024-03-18T22:43:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.416+0000 7f59bbfff640 10 client.25618 handle_command_reply: tid=0 2024-03-18T22:43:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.416+0000 7f59e5993740 1 client.25618 shutdown 2024-03-18T22:43:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.417+0000 7f59e5993740 1 -- 172.21.15.40:0/1668157553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14ee7b00 msgr2=0x556a14ef6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.417+0000 7f59e5993740 1 --2- 172.21.15.40:0/1668157553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14ee7b00 0x556a14ef6540 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009e50 tx=0x7f59d0005730 comp rx=0 tx=0).stop 2024-03-18T22:43:42.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.417+0000 7f59e5993740 1 -- 172.21.15.40:0/1668157553 shutdown_connections 2024-03-18T22:43:42.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.417+0000 7f59e5993740 1 --2- 172.21.15.40:0/1668157553 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556a14f63d40 0x556a14f66130 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.417+0000 7f59e5993740 1 --2- 172.21.15.40:0/1668157553 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556a14eed660 0x556a14f62110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.417+0000 7f59bbfff640 0 client.25618 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:43:42.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.417+0000 7f59e5993740 1 --2- 172.21.15.40:0/1668157553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a14ee84f0 0x556a14ef6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.417+0000 7f59e5993740 1 --2- 172.21.15.40:0/1668157553 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556a14ee7b00 0x556a14ef6540 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.417+0000 7f59e5993740 1 -- 172.21.15.40:0/1668157553 >> 172.21.15.40:0/1668157553 conn(0x556a14eda860 msgr2=0x556a14ef1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.417+0000 7f59e5993740 1 -- 172.21.15.40:0/1668157553 shutdown_connections 2024-03-18T22:43:42.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.417+0000 7f59e5993740 1 -- 172.21.15.40:0/1668157553 wait complete. 2024-03-18T22:43:42.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.417+0000 7f59e5993740 20 client.25618 trim_cache size 0 max 16384 2024-03-18T22:43:42.422 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:43:42.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.418+0000 7f59e2da9640 1 -- 172.21.15.40:0/3223388852 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f59cc0a2880 msgr2=0x7f59cc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.418+0000 7f59e2da9640 1 --2- 172.21.15.40:0/3223388852 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f59cc0a2880 0x7f59cc0a4d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009fc0 tx=0x7f59d0005f90 comp rx=0 tx=0).stop 2024-03-18T22:43:42.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.418+0000 7f59e2da9640 1 -- 172.21.15.40:0/3223388852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59dc104c90 msgr2=0x7f59dc197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.418+0000 7f59e2da9640 1 --2- 172.21.15.40:0/3223388852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59dc104c90 0x7f59dc197ff0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f59c8000c00 tx=0x7f59c8005dc0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.418+0000 7f59e2da9640 1 -- 172.21.15.40:0/3223388852 shutdown_connections 2024-03-18T22:43:42.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.418+0000 7f59e2da9640 1 --2- 172.21.15.40:0/3223388852 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f59cc0a2880 0x7f59cc0a4d40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.418+0000 7f59e2da9640 1 --2- 172.21.15.40:0/3223388852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f59dc1075c0 0x7f59dc19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.418+0000 7f59e2da9640 1 --2- 172.21.15.40:0/3223388852 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f59dc104c90 0x7f59dc197ff0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.418+0000 7f59e2da9640 1 --2- 172.21.15.40:0/3223388852 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59dc102360 0x7f59dc197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.418+0000 7f59e2da9640 1 -- 172.21.15.40:0/3223388852 >> 172.21.15.40:0/3223388852 conn(0x7f59dc0fbc90 msgr2=0x7f59dc102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.418+0000 7f59e2da9640 1 -- 172.21.15.40:0/3223388852 shutdown_connections 2024-03-18T22:43:42.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.418+0000 7f59e2da9640 1 -- 172.21.15.40:0/3223388852 wait complete. 2024-03-18T22:43:42.433 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:43:42.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.548+0000 7fd88d873640 1 -- 172.21.15.40:0/1988734650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd888102b70 msgr2=0x7fd888102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.548+0000 7fd88d873640 1 --2- 172.21.15.40:0/1988734650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd888102b70 0x7fd888102ff0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009900 tx=0x7fd87c02bc80 comp rx=0 tx=0).stop 2024-03-18T22:43:42.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.548+0000 7fd88d873640 1 -- 172.21.15.40:0/1988734650 shutdown_connections 2024-03-18T22:43:42.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.548+0000 7fd88d873640 1 --2- 172.21.15.40:0/1988734650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd888103530 0x7fd888109dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.548+0000 7fd88d873640 1 --2- 172.21.15.40:0/1988734650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd888102b70 0x7fd888102ff0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.548+0000 7fd88d873640 1 --2- 172.21.15.40:0/1988734650 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8880ffb00 0x7fd8880fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.548+0000 7fd88d873640 1 -- 172.21.15.40:0/1988734650 >> 172.21.15.40:0/1988734650 conn(0x7fd8880fb430 msgr2=0x7fd8880fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.548+0000 7fd88d873640 1 -- 172.21.15.40:0/1988734650 shutdown_connections 2024-03-18T22:43:42.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.548+0000 7fd88d873640 1 -- 172.21.15.40:0/1988734650 wait complete. 2024-03-18T22:43:42.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.549+0000 7fd88d873640 1 Processor -- start 2024-03-18T22:43:42.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.549+0000 7fd88d873640 1 -- start start 2024-03-18T22:43:42.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.549+0000 7fd88d873640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8880ffb00 0x7fd8881024e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.549+0000 7fd88d873640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd888102b70 0x7fd888100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.549+0000 7fd88d873640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd888103530 0x7fd888101060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.549+0000 7fd88d873640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd88810d370 con 0x7fd888102b70 2024-03-18T22:43:42.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.549+0000 7fd88d873640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd88810d1f0 con 0x7fd888103530 2024-03-18T22:43:42.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.549+0000 7fd88d873640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd88810d4f0 con 0x7fd8880ffb00 2024-03-18T22:43:42.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.549+0000 7fd886ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8880ffb00 0x7fd8881024e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.549+0000 7fd8867fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd888102b70 0x7fd888100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.549+0000 7fd8877fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd888103530 0x7fd888101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.549+0000 7fd8867fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd888102b70 0x7fd888100b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46068/0 (socket says 172.21.15.40:46068) 2024-03-18T22:43:42.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.549+0000 7fd8867fc640 1 -- 172.21.15.40:0/420155241 learned_addr learned my addr 172.21.15.40:0/420155241 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.549+0000 7fd8877fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd888103530 0x7fd888101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60442/0 (socket says 172.21.15.40:60442) 2024-03-18T22:43:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.550+0000 7fd8877fe640 1 -- 172.21.15.40:0/420155241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8880ffb00 msgr2=0x7fd8881024e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.550+0000 7fd8877fe640 1 --2- 172.21.15.40:0/420155241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8880ffb00 0x7fd8881024e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.550+0000 7fd8877fe640 1 -- 172.21.15.40:0/420155241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd888102b70 msgr2=0x7fd888100b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.550+0000 7fd8877fe640 1 --2- 172.21.15.40:0/420155241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd888102b70 0x7fd888100b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.550+0000 7fd8877fe640 1 -- 172.21.15.40:0/420155241 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8881018f0 con 0x7fd888103530 2024-03-18T22:43:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.550+0000 7fd8867fc640 1 --2- 172.21.15.40:0/420155241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd888102b70 0x7fd888100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.550+0000 7fd886ffd640 1 --2- 172.21.15.40:0/420155241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8880ffb00 0x7fd8881024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.550+0000 7fd8877fe640 1 --2- 172.21.15.40:0/420155241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd888103530 0x7fd888101060 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd880009fd0 tx=0x7fd88000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.550+0000 7fd88c871640 1 -- 172.21.15.40:0/420155241 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88000ce20 con 0x7fd888103530 2024-03-18T22:43:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.550+0000 7fd88c871640 1 -- 172.21.15.40:0/420155241 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd880005020 con 0x7fd888103530 2024-03-18T22:43:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.551+0000 7fd88c871640 1 -- 172.21.15.40:0/420155241 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd880010690 con 0x7fd888103530 2024-03-18T22:43:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.551+0000 7fd88d873640 1 -- 172.21.15.40:0/420155241 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8881a1b70 con 0x7fd888103530 2024-03-18T22:43:42.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.551+0000 7fd88d873640 1 -- 172.21.15.40:0/420155241 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8881a1fe0 con 0x7fd888103530 2024-03-18T22:43:42.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.554+0000 7fd88c871640 1 -- 172.21.15.40:0/420155241 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd8800051d0 con 0x7fd888103530 2024-03-18T22:43:42.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.555+0000 7fd89045d740 1 Processor -- start 2024-03-18T22:43:42.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.555+0000 7fd89045d740 1 -- start start 2024-03-18T22:43:42.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.555+0000 7fd89045d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d77322f0 0x5648d7732710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.555+0000 7fd89045d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7732ce0 0x5648d77377e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.555+0000 7fd89045d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d7737e50 0x5648d773a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.555+0000 7fd89045d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5648d76964b0 con 0x5648d77322f0 2024-03-18T22:43:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.555+0000 7fd89045d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5648d763ed70 con 0x5648d7732ce0 2024-03-18T22:43:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.555+0000 7fd89045d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5648d7692550 con 0x5648d7737e50 2024-03-18T22:43:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.555+0000 7fd886ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7732ce0 0x5648d77377e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.555+0000 7fd8867fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d77322f0 0x5648d7732710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.555+0000 7fd88c871640 1 --2- 172.21.15.40:0/420155241 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd8700a2760 0x7fd8700a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.555+0000 7fd886ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7732ce0 0x5648d77377e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60458/0 (socket says 172.21.15.40:60458) 2024-03-18T22:43:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.555+0000 7fd8867fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d77322f0 0x5648d7732710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46074/0 (socket says 172.21.15.40:46074) 2024-03-18T22:43:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.555+0000 7fd886ffd640 1 -- 172.21.15.40:0/3982423250 learned_addr learned my addr 172.21.15.40:0/3982423250 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.555+0000 7fd8877fe640 1 --2- 172.21.15.40:0/420155241 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd8700a2760 0x7fd8700a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.556+0000 7fd8867fc640 1 -- 172.21.15.40:0/3982423250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d7737e50 msgr2=0x5648d773a2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:43:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.556+0000 7fd8867fc640 1 --2- 172.21.15.40:0/3982423250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d7737e50 0x5648d773a2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.556+0000 7fd8867fc640 1 -- 172.21.15.40:0/3982423250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7732ce0 msgr2=0x5648d77377e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.556+0000 7fd8867fc640 1 --2- 172.21.15.40:0/3982423250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7732ce0 0x5648d77377e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.556+0000 7fd8867fc640 1 -- 172.21.15.40:0/3982423250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648d7661210 con 0x5648d77322f0 2024-03-18T22:43:42.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.556+0000 7fd88c871640 1 -- 172.21.15.40:0/420155241 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd8800cf0f0 con 0x7fd888103530 2024-03-18T22:43:42.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.556+0000 7fd886ffd640 1 --2- 172.21.15.40:0/3982423250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7732ce0 0x5648d77377e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:43:42.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.556+0000 7fd8877fe640 1 --2- 172.21.15.40:0/420155241 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd8700a2760 0x7fd8700a4c20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd88009b5d0 tx=0x7fd8800979d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.556+0000 7fd8867fc640 1 --2- 172.21.15.40:0/3982423250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d77322f0 0x5648d7732710 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009230 tx=0x7fd87c02c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f3395f8eaa826a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.556+0000 7fd86dffb640 1 -- 172.21.15.40:0/3982423250 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c045020 con 0x5648d77322f0 2024-03-18T22:43:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.556+0000 7fd86dffb640 1 -- 172.21.15.40:0/3982423250 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd87c005040 con 0x5648d77322f0 2024-03-18T22:43:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.556+0000 7fd86dffb640 1 -- 172.21.15.40:0/3982423250 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c005d50 con 0x5648d77322f0 2024-03-18T22:43:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.557+0000 7fd89045d740 1 -- 172.21.15.40:0/3982423250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d77322f0 msgr2=0x5648d7732710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.557+0000 7fd89045d740 1 --2- 172.21.15.40:0/3982423250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d77322f0 0x5648d7732710 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009230 tx=0x7fd87c02c720 comp rx=0 tx=0).stop 2024-03-18T22:43:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.557+0000 7fd89045d740 1 -- 172.21.15.40:0/3982423250 shutdown_connections 2024-03-18T22:43:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.557+0000 7fd89045d740 1 --2- 172.21.15.40:0/3982423250 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d7737e50 0x5648d773a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.557+0000 7fd89045d740 1 --2- 172.21.15.40:0/3982423250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7732ce0 0x5648d77377e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.557+0000 7fd89045d740 1 --2- 172.21.15.40:0/3982423250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d77322f0 0x5648d7732710 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.557+0000 7fd89045d740 1 -- 172.21.15.40:0/3982423250 >> 172.21.15.40:0/3982423250 conn(0x5648d772d860 msgr2=0x5648d772dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.557+0000 7fd89045d740 1 -- 172.21.15.40:0/3982423250 shutdown_connections 2024-03-18T22:43:42.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.557+0000 7fd89045d740 1 -- 172.21.15.40:0/3982423250 wait complete. 2024-03-18T22:43:42.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.558+0000 7fd89045d740 1 -- start start 2024-03-18T22:43:42.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.558+0000 7fd89045d740 1 Processor -- start 2024-03-18T22:43:42.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.558+0000 7fd89045d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d77322f0 0x5648d773c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.558+0000 7fd89045d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7732ce0 0x5648d773cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.558+0000 7fd89045d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7737e50 0x5648d7743d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.558+0000 7fd89045d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5648d773ada0 con 0x5648d7737e50 2024-03-18T22:43:42.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.558+0000 7fd89045d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5648d7652310 con 0x5648d7732ce0 2024-03-18T22:43:42.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.558+0000 7fd89045d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5648d773af20 con 0x5648d77322f0 2024-03-18T22:43:42.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.559+0000 7fd89045d740 10 client.0 _ll_register_callbacks cb 0x5648d76462d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:42.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.559+0000 7fd8867fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7737e50 0x5648d7743d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.559+0000 7fd8867fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7737e50 0x5648d7743d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46082/0 (socket says 172.21.15.40:46082) 2024-03-18T22:43:42.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.559+0000 7fd8867fc640 1 -- 172.21.15.40:0/1382988283 learned_addr learned my addr 172.21.15.40:0/1382988283 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.559+0000 7fd886ffd640 1 --2- 172.21.15.40:0/1382988283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7732ce0 0x5648d773cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.559+0000 7fd8867fc640 1 --2- 172.21.15.40:0/1382988283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d77322f0 0x5648d773c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.559+0000 7fd8867fc640 1 -- 172.21.15.40:0/1382988283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7732ce0 msgr2=0x5648d773cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.559+0000 7fd8867fc640 1 --2- 172.21.15.40:0/1382988283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7732ce0 0x5648d773cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.559+0000 7fd8867fc640 1 -- 172.21.15.40:0/1382988283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7737e50 msgr2=0x5648d7743d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.559+0000 7fd8867fc640 1 --2- 172.21.15.40:0/1382988283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7737e50 0x5648d7743d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.559+0000 7fd8867fc640 1 -- 172.21.15.40:0/1382988283 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648d73aa3f0 con 0x5648d77322f0 2024-03-18T22:43:42.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.559+0000 7fd886ffd640 1 --2- 172.21.15.40:0/1382988283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7732ce0 0x5648d773cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:42.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.559+0000 7fd8867fc640 1 --2- 172.21.15.40:0/1382988283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d77322f0 0x5648d773c7f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0096c0 tx=0x7fd87c02cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=650f7f012479e932 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.559+0000 7fd866ffd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:43:42.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.559+0000 7fd866ffd640 1 -- 172.21.15.40:0/1382988283 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c045040 con 0x5648d77322f0 2024-03-18T22:43:42.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.560+0000 7fd866ffd640 1 -- 172.21.15.40:0/1382988283 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd87c004d90 con 0x5648d77322f0 2024-03-18T22:43:42.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.560+0000 7fd89045d740 1 -- 172.21.15.40:0/1382988283 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd87c005040 con 0x5648d77322f0 2024-03-18T22:43:42.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.560+0000 7fd866ffd640 1 -- 172.21.15.40:0/1382988283 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd87c056770 con 0x5648d77322f0 2024-03-18T22:43:42.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.560+0000 7fd89045d740 10 client.25627 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:42.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.560+0000 7fd89045d740 1 -- 172.21.15.40:0/1382988283 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5648d74747d0 con 0x5648d77322f0 2024-03-18T22:43:42.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.560+0000 7fd866ffd640 1 -- 172.21.15.40:0/1382988283 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd87c005040 con 0x5648d77322f0 2024-03-18T22:43:42.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.560+0000 7fd866ffd640 1 -- 172.21.15.40:0/1382988283 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd87c041480 con 0x5648d77322f0 2024-03-18T22:43:42.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.561+0000 7fd89045d740 10 client.25627 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:42.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.561+0000 7fd89045d740 10 client.25627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:43:42.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.561+0000 7fd89045d740 20 client.25627 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:42.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.561+0000 7fd89045d740 1 --2- 172.21.15.40:0/1382988283 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5648d77481b0 0x5648d774a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.561+0000 7fd89045d740 4 client.25627 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:43:42.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.561+0000 7fd89045d740 1 -- 172.21.15.40:0/1382988283 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648d76f43e0 con 0x5648d77481b0 2024-03-18T22:43:42.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.561+0000 7fd886ffd640 1 --2- 172.21.15.40:0/1382988283 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5648d77481b0 0x5648d774a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.562+0000 7fd866ffd640 10 client.25627 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:42.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.562+0000 7fd886ffd640 1 --2- 172.21.15.40:0/1382988283 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5648d77481b0 0x5648d774a5a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da8a61da7c168951 server_cookie=9bc440097381ca91 in_seq=0 out_seq=0 2024-03-18T22:43:42.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.562+0000 7fd866ffd640 1 -- 172.21.15.40:0/1382988283 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5648d76f43e0 con 0x5648d77481b0 2024-03-18T22:43:42.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.562+0000 7fd866ffd640 10 client.25627 handle_command_reply: tid=0 2024-03-18T22:43:42.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.563+0000 7fd89045d740 1 client.25627 shutdown 2024-03-18T22:43:42.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.563+0000 7fd89045d740 1 -- 172.21.15.40:0/1382988283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d77322f0 msgr2=0x5648d773c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.563+0000 7fd89045d740 1 --2- 172.21.15.40:0/1382988283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d77322f0 0x5648d773c7f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0096c0 tx=0x7fd87c02cd80 comp rx=0 tx=0).stop 2024-03-18T22:43:42.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.563+0000 7fd89045d740 1 -- 172.21.15.40:0/1382988283 shutdown_connections 2024-03-18T22:43:42.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.563+0000 7fd89045d740 1 --2- 172.21.15.40:0/1382988283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7737e50 0x5648d7743d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.563+0000 7fd89045d740 1 --2- 172.21.15.40:0/1382988283 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5648d77481b0 0x5648d774a5a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.563+0000 7fd866ffd640 0 client.25627 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:42.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.563+0000 7fd89045d740 1 --2- 172.21.15.40:0/1382988283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7732ce0 0x5648d773cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.563+0000 7fd89045d740 1 --2- 172.21.15.40:0/1382988283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d77322f0 0x5648d773c7f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.563+0000 7fd89045d740 1 -- 172.21.15.40:0/1382988283 >> 172.21.15.40:0/1382988283 conn(0x5648d772d860 msgr2=0x5648d772cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.563+0000 7fd89045d740 1 -- 172.21.15.40:0/1382988283 shutdown_connections 2024-03-18T22:43:42.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.563+0000 7fd89045d740 1 -- 172.21.15.40:0/1382988283 wait complete. 2024-03-18T22:43:42.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.563+0000 7fd89045d740 20 client.25627 trim_cache size 0 max 16384 2024-03-18T22:43:42.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.589+0000 7fd89045d740 1 Processor -- start 2024-03-18T22:43:42.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd89045d740 1 -- start start 2024-03-18T22:43:42.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd89045d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d77322f0 0x5648d773cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd89045d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d7732ce0 0x5648d7743d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd89045d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7737e50 0x5648d772c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd89045d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd8881ac540 con 0x5648d7737e50 2024-03-18T22:43:42.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd89045d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd8881ae080 con 0x5648d77322f0 2024-03-18T22:43:42.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd89045d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd8881acc10 con 0x5648d7732ce0 2024-03-18T22:43:42.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd886ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d7732ce0 0x5648d7743d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd8867fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d77322f0 0x5648d773cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd886ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d7732ce0 0x5648d7743d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:56046/0 (socket says 172.21.15.40:56046) 2024-03-18T22:43:42.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd886ffd640 1 -- 172.21.15.40:0/1935644762 learned_addr learned my addr 172.21.15.40:0/1935644762 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd8867fc640 1 --2- 172.21.15.40:0/1935644762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7737e50 0x5648d772c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd886ffd640 1 -- 172.21.15.40:0/1935644762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d77322f0 msgr2=0x5648d773cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd886ffd640 1 --2- 172.21.15.40:0/1935644762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d77322f0 0x5648d773cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd886ffd640 1 -- 172.21.15.40:0/1935644762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7737e50 msgr2=0x5648d772c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd886ffd640 1 --2- 172.21.15.40:0/1935644762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7737e50 0x5648d772c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd886ffd640 1 -- 172.21.15.40:0/1935644762 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648d7703880 con 0x5648d7732ce0 2024-03-18T22:43:42.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.590+0000 7fd8867fc640 1 --2- 172.21.15.40:0/1935644762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d77322f0 0x5648d773cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:43:42.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.591+0000 7fd886ffd640 1 --2- 172.21.15.40:0/1935644762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d7732ce0 0x5648d7743d30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x5648d772e7d0 tx=0x7fd878015b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69077da3b176afbd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.591+0000 7fd86d7fa640 1 -- 172.21.15.40:0/1935644762 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87802c020 con 0x5648d7732ce0 2024-03-18T22:43:42.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.591+0000 7fd86d7fa640 1 -- 172.21.15.40:0/1935644762 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8780150d0 con 0x5648d7732ce0 2024-03-18T22:43:42.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.591+0000 7fd86d7fa640 1 -- 172.21.15.40:0/1935644762 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd878028c70 con 0x5648d7732ce0 2024-03-18T22:43:42.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.592+0000 7fd89045d740 1 -- 172.21.15.40:0/1935644762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d7732ce0 msgr2=0x5648d7743d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.592+0000 7fd89045d740 1 --2- 172.21.15.40:0/1935644762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d7732ce0 0x5648d7743d30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x5648d772e7d0 tx=0x7fd878015b60 comp rx=0 tx=0).stop 2024-03-18T22:43:42.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.592+0000 7fd89045d740 1 -- 172.21.15.40:0/1935644762 shutdown_connections 2024-03-18T22:43:42.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.592+0000 7fd89045d740 1 --2- 172.21.15.40:0/1935644762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7737e50 0x5648d772c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.592+0000 7fd89045d740 1 --2- 172.21.15.40:0/1935644762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d7732ce0 0x5648d7743d30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.592+0000 7fd89045d740 1 --2- 172.21.15.40:0/1935644762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d77322f0 0x5648d773cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.592+0000 7fd89045d740 1 -- 172.21.15.40:0/1935644762 >> 172.21.15.40:0/1935644762 conn(0x5648d772d860 msgr2=0x5648d773c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.592+0000 7fd89045d740 1 -- 172.21.15.40:0/1935644762 shutdown_connections 2024-03-18T22:43:42.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.592+0000 7fd89045d740 1 -- 172.21.15.40:0/1935644762 wait complete. 2024-03-18T22:43:42.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.592+0000 7fd89045d740 1 -- start start 2024-03-18T22:43:42.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.593+0000 7fd89045d740 1 Processor -- start 2024-03-18T22:43:42.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.593+0000 7fd89045d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d77322f0 0x5648d7741570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.593+0000 7fd89045d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7732ce0 0x5648d7741ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.593+0000 7fd89045d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7737e50 0x5648d7742ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.593+0000 7fd8867fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d77322f0 0x5648d7741570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.593+0000 7fd8867fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d77322f0 0x5648d7741570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:56062/0 (socket says 172.21.15.40:56062) 2024-03-18T22:43:42.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd886ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7732ce0 0x5648d7741ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd886ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7732ce0 0x5648d7741ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46098/0 (socket says 172.21.15.40:46098) 2024-03-18T22:43:42.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd8867fc640 1 -- 172.21.15.40:0/1052449931 learned_addr learned my addr 172.21.15.40:0/1052449931 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:42.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd89045d740 1 -- 172.21.15.40:0/1052449931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd8881ad9b0 con 0x5648d7732ce0 2024-03-18T22:43:42.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd89045d740 1 -- 172.21.15.40:0/1052449931 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd8881ad2e0 con 0x5648d7737e50 2024-03-18T22:43:42.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd89045d740 1 -- 172.21.15.40:0/1052449931 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd8881aa520 con 0x5648d77322f0 2024-03-18T22:43:42.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd89045d740 10 client.0 _ll_register_callbacks cb 0x7fd87c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:42.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd8867fc640 1 --2- 172.21.15.40:0/1052449931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7737e50 0x5648d7742ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd886ffd640 1 -- 172.21.15.40:0/1052449931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d77322f0 msgr2=0x5648d7741570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd886ffd640 1 --2- 172.21.15.40:0/1052449931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d77322f0 0x5648d7741570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd886ffd640 1 -- 172.21.15.40:0/1052449931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7737e50 msgr2=0x5648d7742ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:42.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd886ffd640 1 --2- 172.21.15.40:0/1052449931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7737e50 0x5648d7742ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd886ffd640 1 -- 172.21.15.40:0/1052449931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648d77430b0 con 0x5648d7732ce0 2024-03-18T22:43:42.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd8867fc640 1 --2- 172.21.15.40:0/1052449931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d77322f0 0x5648d7741570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:42.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd886ffd640 1 --2- 172.21.15.40:0/1052449931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7732ce0 0x5648d7741ab0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x5648d772d830 tx=0x7fd8780044b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fbcc9084772dec2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:42.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.594+0000 7fd866ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:43:42.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.595+0000 7fd866ffd640 1 -- 172.21.15.40:0/1052449931 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87802c030 con 0x5648d7732ce0 2024-03-18T22:43:42.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.595+0000 7fd866ffd640 1 -- 172.21.15.40:0/1052449931 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd878003b20 con 0x5648d7732ce0 2024-03-18T22:43:42.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.595+0000 7fd89045d740 1 -- 172.21.15.40:0/1052449931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8780150d0 con 0x5648d7732ce0 2024-03-18T22:43:42.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.595+0000 7fd866ffd640 1 -- 172.21.15.40:0/1052449931 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd878045050 con 0x5648d7732ce0 2024-03-18T22:43:42.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.595+0000 7fd89045d740 10 client.15768 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:42.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.595+0000 7fd89045d740 1 -- 172.21.15.40:0/1052449931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5648d77ae370 con 0x5648d7732ce0 2024-03-18T22:43:42.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.595+0000 7fd866ffd640 1 -- 172.21.15.40:0/1052449931 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8780150d0 con 0x5648d7732ce0 2024-03-18T22:43:42.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.596+0000 7fd866ffd640 1 -- 172.21.15.40:0/1052449931 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd8780138f0 con 0x5648d7732ce0 2024-03-18T22:43:42.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.596+0000 7fd89045d740 10 client.15768 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:42.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.596+0000 7fd89045d740 10 client.15768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:43:42.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.596+0000 7fd89045d740 20 client.15768 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:42.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.596+0000 7fd89045d740 1 --2- 172.21.15.40:0/1052449931 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5648d77481b0 0x5648d77b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:42.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.596+0000 7fd89045d740 4 client.15768 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:43:42.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.596+0000 7fd89045d740 1 -- 172.21.15.40:0/1052449931 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5648d76a39f0 con 0x5648d77481b0 2024-03-18T22:43:42.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.597+0000 7fd8867fc640 1 --2- 172.21.15.40:0/1052449931 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5648d77481b0 0x5648d77b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:42.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.597+0000 7fd866ffd640 10 client.15768 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:42.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.597+0000 7fd8867fc640 1 --2- 172.21.15.40:0/1052449931 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5648d77481b0 0x5648d77b0cb0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d58cc784c4f20dd server_cookie=360bbfa7a27cb07 in_seq=0 out_seq=0 2024-03-18T22:43:42.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.597+0000 7fd866ffd640 1 -- 172.21.15.40:0/1052449931 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5648d76a39f0 con 0x5648d77481b0 2024-03-18T22:43:42.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.597+0000 7fd866ffd640 10 client.15768 handle_command_reply: tid=0 2024-03-18T22:43:42.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.597+0000 7fd89045d740 1 client.15768 shutdown 2024-03-18T22:43:42.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.598+0000 7fd89045d740 1 -- 172.21.15.40:0/1052449931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7732ce0 msgr2=0x5648d7741ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.598+0000 7fd89045d740 1 --2- 172.21.15.40:0/1052449931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7732ce0 0x5648d7741ab0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x5648d772d830 tx=0x7fd8780044b0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.598+0000 7fd89045d740 1 -- 172.21.15.40:0/1052449931 shutdown_connections 2024-03-18T22:43:42.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.598+0000 7fd89045d740 1 --2- 172.21.15.40:0/1052449931 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5648d77481b0 0x5648d77b0cb0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.598+0000 7fd89045d740 1 --2- 172.21.15.40:0/1052449931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5648d7737e50 0x5648d7742ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.598+0000 7fd89045d740 1 --2- 172.21.15.40:0/1052449931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5648d7732ce0 0x5648d7741ab0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.598+0000 7fd89045d740 1 --2- 172.21.15.40:0/1052449931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5648d77322f0 0x5648d7741570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.598+0000 7fd89045d740 1 -- 172.21.15.40:0/1052449931 >> 172.21.15.40:0/1052449931 conn(0x5648d772d860 msgr2=0x5648d7730ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.598+0000 7fd866ffd640 0 client.15768 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:42.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.598+0000 7fd89045d740 1 -- 172.21.15.40:0/1052449931 shutdown_connections 2024-03-18T22:43:42.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.598+0000 7fd89045d740 1 -- 172.21.15.40:0/1052449931 wait complete. 2024-03-18T22:43:42.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.598+0000 7fd89045d740 20 client.15768 trim_cache size 0 max 16384 2024-03-18T22:43:42.603 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:43:42.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.599+0000 7fd88d873640 1 -- 172.21.15.40:0/420155241 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd8700a2760 msgr2=0x7fd8700a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.599+0000 7fd88d873640 1 --2- 172.21.15.40:0/420155241 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd8700a2760 0x7fd8700a4c20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd88009b5d0 tx=0x7fd8800979d0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.599+0000 7fd88d873640 1 -- 172.21.15.40:0/420155241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd888103530 msgr2=0x7fd888101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:42.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.599+0000 7fd88d873640 1 --2- 172.21.15.40:0/420155241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd888103530 0x7fd888101060 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd880009fd0 tx=0x7fd88000ef10 comp rx=0 tx=0).stop 2024-03-18T22:43:42.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.599+0000 7fd88d873640 1 -- 172.21.15.40:0/420155241 shutdown_connections 2024-03-18T22:43:42.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.599+0000 7fd88d873640 1 --2- 172.21.15.40:0/420155241 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd8700a2760 0x7fd8700a4c20 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.599+0000 7fd88d873640 1 --2- 172.21.15.40:0/420155241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd888103530 0x7fd888101060 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.599+0000 7fd88d873640 1 --2- 172.21.15.40:0/420155241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd888102b70 0x7fd888100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.599+0000 7fd88d873640 1 --2- 172.21.15.40:0/420155241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8880ffb00 0x7fd8881024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:42.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.599+0000 7fd88d873640 1 -- 172.21.15.40:0/420155241 >> 172.21.15.40:0/420155241 conn(0x7fd8880fb430 msgr2=0x7fd888105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:42.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.599+0000 7fd88d873640 1 -- 172.21.15.40:0/420155241 shutdown_connections 2024-03-18T22:43:42.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:42.599+0000 7fd88d873640 1 -- 172.21.15.40:0/420155241 wait complete. 2024-03-18T22:43:43.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:42 smithi113 ceph-mon[66775]: pgmap v382: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:43:43.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:42 smithi103 ceph-mon[66377]: pgmap v382: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:43:43.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:42 smithi040 ceph-mon[62643]: pgmap v382: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:43:43.615 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:43:43.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:43:43.616 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:43:43.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.729+0000 7fc2d1960640 1 -- 172.21.15.40:0/1400455953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2cc075800 msgr2=0x7fc2cc073c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.729+0000 7fc2d1960640 1 --2- 172.21.15.40:0/1400455953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2cc075800 0x7fc2cc073c50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc009900 tx=0x7fc2bc02bcd0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.729+0000 7fc2d1960640 1 -- 172.21.15.40:0/1400455953 shutdown_connections 2024-03-18T22:43:43.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.729+0000 7fc2d1960640 1 --2- 172.21.15.40:0/1400455953 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2cc1096e0 0x7fc2cc10bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.729+0000 7fc2d1960640 1 --2- 172.21.15.40:0/1400455953 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2cc074220 0x7fc2cc074680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.729+0000 7fc2d1960640 1 --2- 172.21.15.40:0/1400455953 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2cc075800 0x7fc2cc073c50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.729+0000 7fc2d1960640 1 -- 172.21.15.40:0/1400455953 >> 172.21.15.40:0/1400455953 conn(0x7fc2cc0fcdc0 msgr2=0x7fc2cc0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:43.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.729+0000 7fc2d1960640 1 -- 172.21.15.40:0/1400455953 shutdown_connections 2024-03-18T22:43:43.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.729+0000 7fc2d1960640 1 -- 172.21.15.40:0/1400455953 wait complete. 2024-03-18T22:43:43.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.730+0000 7fc2d1960640 1 Processor -- start 2024-03-18T22:43:43.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.730+0000 7fc2d1960640 1 -- start start 2024-03-18T22:43:43.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.730+0000 7fc2d1960640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2cc074220 0x7fc2cc199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.730+0000 7fc2d1960640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2cc075800 0x7fc2cc19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.730+0000 7fc2ca7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2cc075800 0x7fc2cc19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.730+0000 7fc2d1960640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2cc1096e0 0x7fc2cc1a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.730+0000 7fc2ca7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2cc075800 0x7fc2cc19a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46124/0 (socket says 172.21.15.40:46124) 2024-03-18T22:43:43.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.730+0000 7fc2d1960640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc2cc10f550 con 0x7fc2cc075800 2024-03-18T22:43:43.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.730+0000 7fc2ca7fc640 1 -- 172.21.15.40:0/2261054140 learned_addr learned my addr 172.21.15.40:0/2261054140 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:43.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.730+0000 7fc2d1960640 1 -- 172.21.15.40:0/2261054140 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc2cc10f3d0 con 0x7fc2cc074220 2024-03-18T22:43:43.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.730+0000 7fc2caffd640 1 --2- 172.21.15.40:0/2261054140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2cc074220 0x7fc2cc199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.730+0000 7fc2d1960640 1 -- 172.21.15.40:0/2261054140 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc2cc10f6d0 con 0x7fc2cc1096e0 2024-03-18T22:43:43.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.731+0000 7fc2cb7fe640 1 --2- 172.21.15.40:0/2261054140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2cc1096e0 0x7fc2cc1a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.731+0000 7fc2caffd640 1 -- 172.21.15.40:0/2261054140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2cc1096e0 msgr2=0x7fc2cc1a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.731+0000 7fc2caffd640 1 --2- 172.21.15.40:0/2261054140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2cc1096e0 0x7fc2cc1a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.731+0000 7fc2caffd640 1 -- 172.21.15.40:0/2261054140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2cc075800 msgr2=0x7fc2cc19a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.731+0000 7fc2caffd640 1 --2- 172.21.15.40:0/2261054140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2cc075800 0x7fc2cc19a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.731+0000 7fc2caffd640 1 -- 172.21.15.40:0/2261054140 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2cc1a18f0 con 0x7fc2cc074220 2024-03-18T22:43:43.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.731+0000 7fc2ca7fc640 1 --2- 172.21.15.40:0/2261054140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2cc075800 0x7fc2cc19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:43.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.731+0000 7fc2caffd640 1 --2- 172.21.15.40:0/2261054140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2cc074220 0x7fc2cc199cb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc000c00 tx=0x7fc2bc004600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:43.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.731+0000 7fc2d095e640 1 -- 172.21.15.40:0/2261054140 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc043020 con 0x7fc2cc074220 2024-03-18T22:43:43.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.731+0000 7fc2d095e640 1 -- 172.21.15.40:0/2261054140 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2bc02dc20 con 0x7fc2cc074220 2024-03-18T22:43:43.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.731+0000 7fc2d1960640 1 -- 172.21.15.40:0/2261054140 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2cc1a1b80 con 0x7fc2cc074220 2024-03-18T22:43:43.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.731+0000 7fc2d095e640 1 -- 172.21.15.40:0/2261054140 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc003ae0 con 0x7fc2cc074220 2024-03-18T22:43:43.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.731+0000 7fc2d1960640 1 -- 172.21.15.40:0/2261054140 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2cc1a20f0 con 0x7fc2cc074220 2024-03-18T22:43:43.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.734+0000 7fc2d095e640 1 -- 172.21.15.40:0/2261054140 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc2bc046020 con 0x7fc2cc074220 2024-03-18T22:43:43.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.735+0000 7fc2d4620740 1 Processor -- start 2024-03-18T22:43:43.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.735+0000 7fc2d4620740 1 -- start start 2024-03-18T22:43:43.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.735+0000 7fc2d095e640 1 --2- 172.21.15.40:0/2261054140 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2b40a2880 0x7fc2b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.735+0000 7fc2d095e640 1 -- 172.21.15.40:0/2261054140 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc2bc0f0d20 con 0x7fc2cc074220 2024-03-18T22:43:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.735+0000 7fc2cb7fe640 1 --2- 172.21.15.40:0/2261054140 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2b40a2880 0x7fc2b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.735+0000 7fc2d4620740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbce37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.735+0000 7fc2d4620740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce3d90 0x5568bbce8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.735+0000 7fc2d4620740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce8f00 0x5568bbceb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.735+0000 7fc2d4620740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5568bbc48160 con 0x5568bbce33a0 2024-03-18T22:43:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.735+0000 7fc2d4620740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5568bbbf0e10 con 0x5568bbce3d90 2024-03-18T22:43:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.735+0000 7fc2d4620740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5568bbc44200 con 0x5568bbce8f00 2024-03-18T22:43:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.735+0000 7fc2caffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbce37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.735+0000 7fc2ca7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce3d90 0x5568bbce8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.735+0000 7fc2ca7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce3d90 0x5568bbce8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60520/0 (socket says 172.21.15.40:60520) 2024-03-18T22:43:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.735+0000 7fc2ca7fc640 1 -- 172.21.15.40:0/912916203 learned_addr learned my addr 172.21.15.40:0/912916203 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.736+0000 7fc2ca7fc640 1 -- 172.21.15.40:0/912916203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce8f00 msgr2=0x5568bbceb390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:43:43.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.736+0000 7fc2ca7fc640 1 --2- 172.21.15.40:0/912916203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce8f00 0x5568bbceb390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.736+0000 7fc2ca7fc640 1 -- 172.21.15.40:0/912916203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 msgr2=0x5568bbce37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:43.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.736+0000 7fc2ca7fc640 1 --2- 172.21.15.40:0/912916203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbce37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.736+0000 7fc2ca7fc640 1 -- 172.21.15.40:0/912916203 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568bba247e0 con 0x5568bbce3d90 2024-03-18T22:43:43.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.736+0000 7fc2ca7fc640 1 --2- 172.21.15.40:0/912916203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce3d90 0x5568bbce8890 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00115f0 tx=0x7fc2c0011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b42e9657ec694373 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:43.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.736+0000 7fc2cb7fe640 1 --2- 172.21.15.40:0/2261054140 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2b40a2880 0x7fc2b40a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4005fd0 tx=0x7fc2c4009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:43.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.736+0000 7fc2b1ffb640 1 -- 172.21.15.40:0/912916203 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c001e070 con 0x5568bbce3d90 2024-03-18T22:43:43.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.736+0000 7fc2b1ffb640 1 -- 172.21.15.40:0/912916203 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2c000f660 con 0x5568bbce3d90 2024-03-18T22:43:43.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.736+0000 7fc2b1ffb640 1 -- 172.21.15.40:0/912916203 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c0003850 con 0x5568bbce3d90 2024-03-18T22:43:43.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.737+0000 7fc2d4620740 1 -- 172.21.15.40:0/912916203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce3d90 msgr2=0x5568bbce8890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.737+0000 7fc2d4620740 1 --2- 172.21.15.40:0/912916203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce3d90 0x5568bbce8890 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc2c00115f0 tx=0x7fc2c0011ab0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.737+0000 7fc2d4620740 1 -- 172.21.15.40:0/912916203 shutdown_connections 2024-03-18T22:43:43.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.737+0000 7fc2d4620740 1 --2- 172.21.15.40:0/912916203 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce8f00 0x5568bbceb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.737+0000 7fc2d4620740 1 --2- 172.21.15.40:0/912916203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce3d90 0x5568bbce8890 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.737+0000 7fc2d4620740 1 --2- 172.21.15.40:0/912916203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbce37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.737+0000 7fc2d4620740 1 -- 172.21.15.40:0/912916203 >> 172.21.15.40:0/912916203 conn(0x5568bbcde910 msgr2=0x5568bbcded10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:43.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.737+0000 7fc2d4620740 1 -- 172.21.15.40:0/912916203 shutdown_connections 2024-03-18T22:43:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.737+0000 7fc2d4620740 1 -- 172.21.15.40:0/912916203 wait complete. 2024-03-18T22:43:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.737+0000 7fc2d4620740 1 -- start start 2024-03-18T22:43:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.737+0000 7fc2d4620740 1 Processor -- start 2024-03-18T22:43:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2d4620740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbced820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2d4620740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 0x5568bbcedd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2d4620740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 0x5568bbcf4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2d4620740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5568bbcebeb0 con 0x5568bbce33a0 2024-03-18T22:43:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2d4620740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5568bbc041a0 con 0x5568bbce8f00 2024-03-18T22:43:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2d4620740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5568bbcec030 con 0x5568bbce3d90 2024-03-18T22:43:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2d4620740 10 client.0 _ll_register_callbacks cb 0x5568bb95bc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2caffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbced820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2caffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbced820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46152/0 (socket says 172.21.15.40:46152) 2024-03-18T22:43:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2caffd640 1 -- 172.21.15.40:0/4137889078 learned_addr learned my addr 172.21.15.40:0/4137889078 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2cb7fe640 1 --2- 172.21.15.40:0/4137889078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 0x5568bbcf4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2ca7fc640 1 --2- 172.21.15.40:0/4137889078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 0x5568bbcedd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2caffd640 1 -- 172.21.15.40:0/4137889078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 msgr2=0x5568bbcedd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2caffd640 1 --2- 172.21.15.40:0/4137889078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 0x5568bbcedd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2caffd640 1 -- 172.21.15.40:0/4137889078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 msgr2=0x5568bbcf4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2caffd640 1 --2- 172.21.15.40:0/4137889078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 0x5568bbcf4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2caffd640 1 -- 172.21.15.40:0/4137889078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568bba84f70 con 0x5568bbce33a0 2024-03-18T22:43:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2cb7fe640 1 --2- 172.21.15.40:0/4137889078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 0x5568bbcf4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2ca7fc640 1 --2- 172.21.15.40:0/4137889078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 0x5568bbcedd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:43:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2caffd640 1 --2- 172.21.15.40:0/4137889078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbced820 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0ee7a0 tx=0x7fc2bc02bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d9d6cdf3e394e24 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.738+0000 7fc2aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:43:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.739+0000 7fc2aaffd640 1 -- 172.21.15.40:0/4137889078 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc0d6060 con 0x5568bbce33a0 2024-03-18T22:43:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.739+0000 7fc2aaffd640 1 -- 172.21.15.40:0/4137889078 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc2bc0bb5d0 con 0x5568bbce33a0 2024-03-18T22:43:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.739+0000 7fc2d4620740 1 -- 172.21.15.40:0/4137889078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2c000f660 con 0x5568bbce33a0 2024-03-18T22:43:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.739+0000 7fc2aaffd640 1 -- 172.21.15.40:0/4137889078 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc2bc0c5a70 con 0x5568bbce33a0 2024-03-18T22:43:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.739+0000 7fc2d4620740 10 client.15780 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.739+0000 7fc2d4620740 1 -- 172.21.15.40:0/4137889078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5568bb91c170 con 0x5568bbce33a0 2024-03-18T22:43:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.739+0000 7fc2aaffd640 1 -- 172.21.15.40:0/4137889078 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2c000f660 con 0x5568bbce33a0 2024-03-18T22:43:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.739+0000 7fc2aaffd640 1 -- 172.21.15.40:0/4137889078 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc2bc0bd530 con 0x5568bbce33a0 2024-03-18T22:43:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.739+0000 7fc2d4620740 10 client.15780 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.739+0000 7fc2d4620740 10 client.15780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:43:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.739+0000 7fc2d4620740 20 client.15780 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.739+0000 7fc2d4620740 1 --2- 172.21.15.40:0/4137889078 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5568bbcf9660 0x5568bbcfbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.739+0000 7fc2d4620740 4 client.15780 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:43:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.739+0000 7fc2d4620740 1 -- 172.21.15.40:0/4137889078 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568bbc556a0 con 0x5568bbcf9660 2024-03-18T22:43:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.740+0000 7fc2ca7fc640 1 --2- 172.21.15.40:0/4137889078 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5568bbcf9660 0x5568bbcfbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.740+0000 7fc2aaffd640 10 client.15780 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.740+0000 7fc2ca7fc640 1 --2- 172.21.15.40:0/4137889078 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5568bbcf9660 0x5568bbcfbaf0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fefbe78a1c50a539 server_cookie=de466194c44ae88f in_seq=0 out_seq=0 2024-03-18T22:43:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.741+0000 7fc2aaffd640 1 -- 172.21.15.40:0/4137889078 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5568bbc556a0 con 0x5568bbcf9660 2024-03-18T22:43:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.741+0000 7fc2aaffd640 10 client.15780 handle_command_reply: tid=0 2024-03-18T22:43:43.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.741+0000 7fc2d4620740 1 client.15780 shutdown 2024-03-18T22:43:43.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.741+0000 7fc2d4620740 1 -- 172.21.15.40:0/4137889078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 msgr2=0x5568bbced820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.741+0000 7fc2d4620740 1 --2- 172.21.15.40:0/4137889078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbced820 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0ee7a0 tx=0x7fc2bc02bcd0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.741+0000 7fc2d4620740 1 -- 172.21.15.40:0/4137889078 shutdown_connections 2024-03-18T22:43:43.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.741+0000 7fc2d4620740 1 --2- 172.21.15.40:0/4137889078 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5568bbcf9660 0x5568bbcfbaf0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.741+0000 7fc2d4620740 1 --2- 172.21.15.40:0/4137889078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 0x5568bbcf4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.741+0000 7fc2d4620740 1 --2- 172.21.15.40:0/4137889078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 0x5568bbcedd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.741+0000 7fc2aaffd640 0 client.15780 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:43.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.741+0000 7fc2d4620740 1 --2- 172.21.15.40:0/4137889078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbced820 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.741+0000 7fc2d4620740 1 -- 172.21.15.40:0/4137889078 >> 172.21.15.40:0/4137889078 conn(0x5568bbcde910 msgr2=0x5568bbcddb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:43.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.741+0000 7fc2d4620740 1 -- 172.21.15.40:0/4137889078 shutdown_connections 2024-03-18T22:43:43.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.741+0000 7fc2d4620740 1 -- 172.21.15.40:0/4137889078 wait complete. 2024-03-18T22:43:43.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.741+0000 7fc2d4620740 20 client.15780 trim_cache size 0 max 16384 2024-03-18T22:43:43.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.759+0000 7fc2d4620740 1 Processor -- start 2024-03-18T22:43:43.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.759+0000 7fc2d4620740 1 -- start start 2024-03-18T22:43:43.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.759+0000 7fc2d4620740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbcedb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.759+0000 7fc2d4620740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 0x5568bbcf4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.759+0000 7fc2d4620740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 0x5568bbcdd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.759+0000 7fc2d4620740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc2cc1ac790 con 0x5568bbce33a0 2024-03-18T22:43:43.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.759+0000 7fc2d4620740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc2cc1ae2d0 con 0x5568bbce8f00 2024-03-18T22:43:43.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.759+0000 7fc2d4620740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc2cc1ace60 con 0x5568bbce3d90 2024-03-18T22:43:43.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.759+0000 7fc2caffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbcedb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.759+0000 7fc2caffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbcedb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46168/0 (socket says 172.21.15.40:46168) 2024-03-18T22:43:43.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.759+0000 7fc2caffd640 1 -- 172.21.15.40:0/600806234 learned_addr learned my addr 172.21.15.40:0/600806234 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:43.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.759+0000 7fc2ca7fc640 1 --2- 172.21.15.40:0/600806234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 0x5568bbcf4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.759+0000 7fc2cb7fe640 1 --2- 172.21.15.40:0/600806234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 0x5568bbcdd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.760+0000 7fc2ca7fc640 1 -- 172.21.15.40:0/600806234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 msgr2=0x5568bbcdd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:43.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.760+0000 7fc2ca7fc640 1 --2- 172.21.15.40:0/600806234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 0x5568bbcdd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.760+0000 7fc2ca7fc640 1 -- 172.21.15.40:0/600806234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 msgr2=0x5568bbcedb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:43.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.760+0000 7fc2ca7fc640 1 --2- 172.21.15.40:0/600806234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbcedb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.760+0000 7fc2ca7fc640 1 -- 172.21.15.40:0/600806234 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568bbcdf680 con 0x5568bbce3d90 2024-03-18T22:43:43.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.760+0000 7fc2caffd640 1 --2- 172.21.15.40:0/600806234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbcedb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:43.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.760+0000 7fc2ca7fc640 1 --2- 172.21.15.40:0/600806234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 0x5568bbcf4d60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x5568bbcdf880 tx=0x7fc2c000fce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd324ac32ccf1a65 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:43.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.760+0000 7fc2b17fa640 1 -- 172.21.15.40:0/600806234 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c001e040 con 0x5568bbce3d90 2024-03-18T22:43:43.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.760+0000 7fc2b17fa640 1 -- 172.21.15.40:0/600806234 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2c000fde0 con 0x5568bbce3d90 2024-03-18T22:43:43.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.760+0000 7fc2b17fa640 1 -- 172.21.15.40:0/600806234 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c00283f0 con 0x5568bbce3d90 2024-03-18T22:43:43.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.761+0000 7fc2d4620740 1 -- 172.21.15.40:0/600806234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 msgr2=0x5568bbcf4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.761+0000 7fc2d4620740 1 --2- 172.21.15.40:0/600806234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 0x5568bbcf4d60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x5568bbcdf880 tx=0x7fc2c000fce0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.761+0000 7fc2d4620740 1 -- 172.21.15.40:0/600806234 shutdown_connections 2024-03-18T22:43:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.761+0000 7fc2d4620740 1 --2- 172.21.15.40:0/600806234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 0x5568bbcdd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.761+0000 7fc2d4620740 1 --2- 172.21.15.40:0/600806234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 0x5568bbcf4d60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.761+0000 7fc2d4620740 1 --2- 172.21.15.40:0/600806234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbcedb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.761+0000 7fc2d4620740 1 -- 172.21.15.40:0/600806234 >> 172.21.15.40:0/600806234 conn(0x5568bbcde910 msgr2=0x5568bbced430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.761+0000 7fc2d4620740 1 -- 172.21.15.40:0/600806234 shutdown_connections 2024-03-18T22:43:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.761+0000 7fc2d4620740 1 -- 172.21.15.40:0/600806234 wait complete. 2024-03-18T22:43:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.761+0000 7fc2d4620740 1 -- start start 2024-03-18T22:43:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2d4620740 1 Processor -- start 2024-03-18T22:43:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2d4620740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbcf2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2d4620740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 0x5568bbcf2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2d4620740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 0x5568bbcf3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2d4620740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc2cc1adc00 con 0x5568bbce33a0 2024-03-18T22:43:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2d4620740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc2cc1ad530 con 0x5568bbce8f00 2024-03-18T22:43:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2d4620740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc2cc1aa770 con 0x5568bbce3d90 2024-03-18T22:43:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2d4620740 10 client.0 _ll_register_callbacks cb 0x7fc2bc009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2caffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbcf2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2ca7fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 0x5568bbcf2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2caffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbcf2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46178/0 (socket says 172.21.15.40:46178) 2024-03-18T22:43:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2caffd640 1 -- 172.21.15.40:0/3236566848 learned_addr learned my addr 172.21.15.40:0/3236566848 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2cb7fe640 1 --2- 172.21.15.40:0/3236566848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 0x5568bbcf3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2ca7fc640 1 -- 172.21.15.40:0/3236566848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 msgr2=0x5568bbcf3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2ca7fc640 1 --2- 172.21.15.40:0/3236566848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 0x5568bbcf3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2ca7fc640 1 -- 172.21.15.40:0/3236566848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 msgr2=0x5568bbcf2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2ca7fc640 1 --2- 172.21.15.40:0/3236566848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbcf2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2ca7fc640 1 -- 172.21.15.40:0/3236566848 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568bbcf40f0 con 0x5568bbce3d90 2024-03-18T22:43:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.762+0000 7fc2caffd640 1 --2- 172.21.15.40:0/3236566848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbcf2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.763+0000 7fc2ca7fc640 1 --2- 172.21.15.40:0/3236566848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 0x5568bbcf2b60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0000c00 tx=0x7fc2c000fc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=421b026cee0d8344 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.763+0000 7fc2aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:43:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.763+0000 7fc2aaffd640 1 -- 172.21.15.40:0/3236566848 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c001e040 con 0x5568bbce3d90 2024-03-18T22:43:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.763+0000 7fc2d4620740 1 -- 172.21.15.40:0/3236566848 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2c000fde0 con 0x5568bbce3d90 2024-03-18T22:43:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.763+0000 7fc2aaffd640 1 -- 172.21.15.40:0/3236566848 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc2c00050e0 con 0x5568bbce3d90 2024-03-18T22:43:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.763+0000 7fc2aaffd640 1 -- 172.21.15.40:0/3236566848 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc2c00052e0 con 0x5568bbce3d90 2024-03-18T22:43:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.763+0000 7fc2d4620740 10 client.25645 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.763+0000 7fc2d4620740 1 -- 172.21.15.40:0/3236566848 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5568bbd5fbe0 con 0x5568bbce3d90 2024-03-18T22:43:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.763+0000 7fc2aaffd640 1 -- 172.21.15.40:0/3236566848 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2c000fde0 con 0x5568bbce3d90 2024-03-18T22:43:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.764+0000 7fc2aaffd640 1 -- 172.21.15.40:0/3236566848 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc2c0003710 con 0x5568bbce3d90 2024-03-18T22:43:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.764+0000 7fc2d4620740 10 client.25645 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.764+0000 7fc2d4620740 10 client.25645 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:43:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.764+0000 7fc2d4620740 20 client.25645 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.764+0000 7fc2d4620740 1 --2- 172.21.15.40:0/3236566848 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5568bbcf9660 0x5568bbd623e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.764+0000 7fc2d4620740 4 client.25645 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:43:43.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.764+0000 7fc2d4620740 1 -- 172.21.15.40:0/3236566848 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5568bbc4f9f0 con 0x5568bbcf9660 2024-03-18T22:43:43.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.764+0000 7fc2caffd640 1 --2- 172.21.15.40:0/3236566848 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5568bbcf9660 0x5568bbd623e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.764+0000 7fc2caffd640 1 --2- 172.21.15.40:0/3236566848 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5568bbcf9660 0x5568bbd623e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd092746d3c28536 server_cookie=2045f1849ecf4234 in_seq=0 out_seq=0 2024-03-18T22:43:43.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.764+0000 7fc2aaffd640 10 client.25645 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:43.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.765+0000 7fc2aaffd640 1 -- 172.21.15.40:0/3236566848 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5568bbc4f9f0 con 0x5568bbcf9660 2024-03-18T22:43:43.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.765+0000 7fc2aaffd640 10 client.25645 handle_command_reply: tid=0 2024-03-18T22:43:43.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.765+0000 7fc2d4620740 1 client.25645 shutdown 2024-03-18T22:43:43.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.765+0000 7fc2d4620740 1 -- 172.21.15.40:0/3236566848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 msgr2=0x5568bbcf2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.765+0000 7fc2d4620740 1 --2- 172.21.15.40:0/3236566848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 0x5568bbcf2b60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0000c00 tx=0x7fc2c000fc40 comp rx=0 tx=0).stop 2024-03-18T22:43:43.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d4620740 1 -- 172.21.15.40:0/3236566848 shutdown_connections 2024-03-18T22:43:43.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d4620740 1 --2- 172.21.15.40:0/3236566848 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5568bbcf9660 0x5568bbd623e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d4620740 1 --2- 172.21.15.40:0/3236566848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568bbce8f00 0x5568bbcf3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d4620740 1 --2- 172.21.15.40:0/3236566848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5568bbce3d90 0x5568bbcf2b60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d4620740 1 --2- 172.21.15.40:0/3236566848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5568bbce33a0 0x5568bbcf2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d4620740 1 -- 172.21.15.40:0/3236566848 >> 172.21.15.40:0/3236566848 conn(0x5568bbcde910 msgr2=0x5568bbce1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:43.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2aaffd640 0 client.25645 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:43.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d4620740 1 -- 172.21.15.40:0/3236566848 shutdown_connections 2024-03-18T22:43:43.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d4620740 1 -- 172.21.15.40:0/3236566848 wait complete. 2024-03-18T22:43:43.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d4620740 20 client.25645 trim_cache size 0 max 16384 2024-03-18T22:43:43.771 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:43:43.771 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:43:43.771 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "749b40b8-4491-4d1f-b091-4500595dbdc2", 2024-03-18T22:43:43.771 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:43:43.771 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:43:43.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d1960640 1 -- 172.21.15.40:0/2261054140 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2b40a2880 msgr2=0x7fc2b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d1960640 1 --2- 172.21.15.40:0/2261054140 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2b40a2880 0x7fc2b40a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc2c4005fd0 tx=0x7fc2c4009500 comp rx=0 tx=0).stop 2024-03-18T22:43:43.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d1960640 1 -- 172.21.15.40:0/2261054140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2cc074220 msgr2=0x7fc2cc199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d1960640 1 --2- 172.21.15.40:0/2261054140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2cc074220 0x7fc2cc199cb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc000c00 tx=0x7fc2bc004600 comp rx=0 tx=0).stop 2024-03-18T22:43:43.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d1960640 1 -- 172.21.15.40:0/2261054140 shutdown_connections 2024-03-18T22:43:43.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d1960640 1 --2- 172.21.15.40:0/2261054140 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2b40a2880 0x7fc2b40a4d40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d1960640 1 --2- 172.21.15.40:0/2261054140 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2cc1096e0 0x7fc2cc1a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d1960640 1 --2- 172.21.15.40:0/2261054140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2cc075800 0x7fc2cc19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d1960640 1 --2- 172.21.15.40:0/2261054140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2cc074220 0x7fc2cc199cb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d1960640 1 -- 172.21.15.40:0/2261054140 >> 172.21.15.40:0/2261054140 conn(0x7fc2cc0fcdc0 msgr2=0x7fc2cc105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:43.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.766+0000 7fc2d1960640 1 -- 172.21.15.40:0/2261054140 shutdown_connections 2024-03-18T22:43:43.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.767+0000 7fc2d1960640 1 -- 172.21.15.40:0/2261054140 wait complete. 2024-03-18T22:43:43.778 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:43:43.778 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:43:43.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.903+0000 7f7cbe675640 1 -- 172.21.15.40:0/3527817595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb8101910 msgr2=0x7f7cb8101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.903+0000 7f7cbe675640 1 --2- 172.21.15.40:0/3527817595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb8101910 0x7f7cb8101d90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009900 tx=0x7f7cac02bc30 comp rx=0 tx=0).stop 2024-03-18T22:43:43.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.903+0000 7f7cbe675640 1 -- 172.21.15.40:0/3527817595 shutdown_connections 2024-03-18T22:43:43.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.903+0000 7f7cbe675640 1 --2- 172.21.15.40:0/3527817595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cb81022d0 0x7f7cb8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.903+0000 7f7cbe675640 1 --2- 172.21.15.40:0/3527817595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb8101910 0x7f7cb8101d90 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.903+0000 7f7cbe675640 1 --2- 172.21.15.40:0/3527817595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7cb8100720 0x7f7cb8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.903+0000 7f7cbe675640 1 -- 172.21.15.40:0/3527817595 >> 172.21.15.40:0/3527817595 conn(0x7f7cb80fbcb0 msgr2=0x7f7cb80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:43.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.903+0000 7f7cbe675640 1 -- 172.21.15.40:0/3527817595 shutdown_connections 2024-03-18T22:43:43.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.903+0000 7f7cbe675640 1 -- 172.21.15.40:0/3527817595 wait complete. 2024-03-18T22:43:43.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.904+0000 7f7cbe675640 1 Processor -- start 2024-03-18T22:43:43.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.904+0000 7f7cbe675640 1 -- start start 2024-03-18T22:43:43.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.904+0000 7f7cbe675640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb8100720 0x7f7cb81046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.904+0000 7f7cbe675640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cb8101910 0x7f7cb8102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.904+0000 7f7cbe675640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7cb81022d0 0x7f7cb8103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.904+0000 7f7cbe675640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb810c230 con 0x7f7cb8100720 2024-03-18T22:43:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.904+0000 7f7cbe675640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb810c0b0 con 0x7f7cb8101910 2024-03-18T22:43:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.904+0000 7f7cbe675640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb810c3b0 con 0x7f7cb81022d0 2024-03-18T22:43:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.904+0000 7f7cb7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb8100720 0x7f7cb81046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.904+0000 7f7cbcc12640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7cb81022d0 0x7f7cb8103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.904+0000 7f7cbcc12640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7cb81022d0 0x7f7cb8103230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:56148/0 (socket says 172.21.15.40:56148) 2024-03-18T22:43:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.904+0000 7f7cb77fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cb8101910 0x7f7cb8102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.904+0000 7f7cb7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb8100720 0x7f7cb81046b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46194/0 (socket says 172.21.15.40:46194) 2024-03-18T22:43:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.904+0000 7f7cbcc12640 1 -- 172.21.15.40:0/2632911564 learned_addr learned my addr 172.21.15.40:0/2632911564 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.905+0000 7f7cb7fff640 1 -- 172.21.15.40:0/2632911564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7cb81022d0 msgr2=0x7f7cb8103230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.905+0000 7f7cb7fff640 1 --2- 172.21.15.40:0/2632911564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7cb81022d0 0x7f7cb8103230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.905+0000 7f7cb7fff640 1 -- 172.21.15.40:0/2632911564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cb8101910 msgr2=0x7f7cb8102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.905+0000 7f7cb7fff640 1 --2- 172.21.15.40:0/2632911564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cb8101910 0x7f7cb8102cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.905+0000 7f7cb7fff640 1 -- 172.21.15.40:0/2632911564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb8103ac0 con 0x7f7cb8100720 2024-03-18T22:43:43.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.905+0000 7f7cbcc12640 1 --2- 172.21.15.40:0/2632911564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7cb81022d0 0x7f7cb8103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:43.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.905+0000 7f7cb7fff640 1 --2- 172.21.15.40:0/2632911564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb8100720 0x7f7cb81046b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800e9c0 tx=0x7f7ca800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:43.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.905+0000 7f7cb57fa640 1 -- 172.21.15.40:0/2632911564 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca800ccc0 con 0x7f7cb8100720 2024-03-18T22:43:43.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.905+0000 7f7cb57fa640 1 -- 172.21.15.40:0/2632911564 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ca80035f0 con 0x7f7cb8100720 2024-03-18T22:43:43.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.905+0000 7f7cbe675640 1 -- 172.21.15.40:0/2632911564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb81a1c00 con 0x7f7cb8100720 2024-03-18T22:43:43.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.905+0000 7f7cb57fa640 1 -- 172.21.15.40:0/2632911564 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca800ce60 con 0x7f7cb8100720 2024-03-18T22:43:43.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.905+0000 7f7cbe675640 1 -- 172.21.15.40:0/2632911564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb81a2170 con 0x7f7cb8100720 2024-03-18T22:43:43.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.908+0000 7f7cb57fa640 1 -- 172.21.15.40:0/2632911564 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7ca80037a0 con 0x7f7cb8100720 2024-03-18T22:43:43.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.908+0000 7f7cc125b740 1 Processor -- start 2024-03-18T22:43:43.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.908+0000 7f7cc125b740 1 -- start start 2024-03-18T22:43:43.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cc125b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa12a50 0x55727aa12e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cc125b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa13440 0x55727aa17f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cc125b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa185b0 0x55727aa1aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cc125b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55727a977500 con 0x55727aa13440 2024-03-18T22:43:43.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cc125b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55727a91fbe0 con 0x55727aa185b0 2024-03-18T22:43:43.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cc125b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55727a9735a0 con 0x55727aa12a50 2024-03-18T22:43:43.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cb57fa640 1 --2- 172.21.15.40:0/2632911564 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ca00a2880 0x7f7ca00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cbcc12640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa12a50 0x55727aa12e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cbcc12640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa12a50 0x55727aa12e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:56160/0 (socket says 172.21.15.40:56160) 2024-03-18T22:43:43.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cbcc12640 1 -- 172.21.15.40:0/1176148280 learned_addr learned my addr 172.21.15.40:0/1176148280 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:43.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cbcc12640 1 --2- 172.21.15.40:0/2632911564 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ca00a2880 0x7f7ca00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cb57fa640 1 -- 172.21.15.40:0/2632911564 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7ca80cf240 con 0x7f7cb8100720 2024-03-18T22:43:43.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cb7fff640 1 --2- 172.21.15.40:0/1176148280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa13440 0x55727aa17f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cb77fe640 1 --2- 172.21.15.40:0/1176148280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa185b0 0x55727aa1aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cbcc12640 1 -- 172.21.15.40:0/1176148280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa185b0 msgr2=0x55727aa1aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:43.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cbcc12640 1 --2- 172.21.15.40:0/1176148280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa185b0 0x55727aa1aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cbcc12640 1 -- 172.21.15.40:0/1176148280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa13440 msgr2=0x55727aa17f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:43.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cbcc12640 1 --2- 172.21.15.40:0/1176148280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa13440 0x55727aa17f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.909+0000 7f7cbcc12640 1 -- 172.21.15.40:0/1176148280 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55727a941000 con 0x55727aa12a50 2024-03-18T22:43:43.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.910+0000 7f7cb7fff640 1 --2- 172.21.15.40:0/1176148280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa13440 0x55727aa17f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:43.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.910+0000 7f7cbcc12640 1 --2- 172.21.15.40:0/1176148280 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa12a50 0x55727aa12e70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00136f0 tx=0x7f7cb0013bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=422247fda8c0d236 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:43.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.910+0000 7f7cbcc12640 1 --2- 172.21.15.40:0/2632911564 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ca00a2880 0x7f7ca00a4d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7cb000e720 tx=0x7f7cb0018000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:43.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.910+0000 7f7c9e7fc640 1 -- 172.21.15.40:0/1176148280 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0025070 con 0x55727aa12a50 2024-03-18T22:43:43.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.910+0000 7f7c9e7fc640 1 -- 172.21.15.40:0/1176148280 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cb0011700 con 0x55727aa12a50 2024-03-18T22:43:43.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.910+0000 7f7c9e7fc640 1 -- 172.21.15.40:0/1176148280 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb000cd80 con 0x55727aa12a50 2024-03-18T22:43:43.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.911+0000 7f7cc125b740 1 -- 172.21.15.40:0/1176148280 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa12a50 msgr2=0x55727aa12e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.911+0000 7f7cc125b740 1 --2- 172.21.15.40:0/1176148280 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa12a50 0x55727aa12e70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00136f0 tx=0x7f7cb0013bb0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.911+0000 7f7cc125b740 1 -- 172.21.15.40:0/1176148280 shutdown_connections 2024-03-18T22:43:43.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.911+0000 7f7cc125b740 1 --2- 172.21.15.40:0/1176148280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa185b0 0x55727aa1aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.911+0000 7f7cc125b740 1 --2- 172.21.15.40:0/1176148280 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa13440 0x55727aa17f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.911+0000 7f7cc125b740 1 --2- 172.21.15.40:0/1176148280 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa12a50 0x55727aa12e70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.911+0000 7f7cc125b740 1 -- 172.21.15.40:0/1176148280 >> 172.21.15.40:0/1176148280 conn(0x55727aa0dfc0 msgr2=0x55727aa0e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:43.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.911+0000 7f7cc125b740 1 -- 172.21.15.40:0/1176148280 shutdown_connections 2024-03-18T22:43:43.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.911+0000 7f7cc125b740 1 -- 172.21.15.40:0/1176148280 wait complete. 2024-03-18T22:43:43.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.911+0000 7f7cc125b740 1 -- start start 2024-03-18T22:43:43.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.911+0000 7f7cc125b740 1 Processor -- start 2024-03-18T22:43:43.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cc125b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa12a50 0x55727aa1ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cc125b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa13440 0x55727aa1d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cc125b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa185b0 0x55727aa24430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cc125b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55727aa1b500 con 0x55727aa13440 2024-03-18T22:43:43.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cc125b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55727a933160 con 0x55727aa185b0 2024-03-18T22:43:43.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cc125b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55727aa1b680 con 0x55727aa12a50 2024-03-18T22:43:43.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cc125b740 10 client.0 _ll_register_callbacks cb 0x55727a96fa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:43.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cb77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa13440 0x55727aa1d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cb77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa13440 0x55727aa1d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46214/0 (socket says 172.21.15.40:46214) 2024-03-18T22:43:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cb77fe640 1 -- 172.21.15.40:0/3178403885 learned_addr learned my addr 172.21.15.40:0/3178403885 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cbcc12640 1 --2- 172.21.15.40:0/3178403885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa185b0 0x55727aa24430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cb7fff640 1 --2- 172.21.15.40:0/3178403885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa12a50 0x55727aa1ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cb7fff640 1 -- 172.21.15.40:0/3178403885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa185b0 msgr2=0x55727aa24430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cb7fff640 1 --2- 172.21.15.40:0/3178403885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa185b0 0x55727aa24430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cb7fff640 1 -- 172.21.15.40:0/3178403885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa13440 msgr2=0x55727aa1d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cb7fff640 1 --2- 172.21.15.40:0/3178403885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa13440 0x55727aa1d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cb7fff640 1 -- 172.21.15.40:0/3178403885 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55727a7557d0 con 0x55727aa12a50 2024-03-18T22:43:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cbcc12640 1 --2- 172.21.15.40:0/3178403885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa185b0 0x55727aa24430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cb77fe640 1 --2- 172.21.15.40:0/3178403885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa13440 0x55727aa1d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7cb7fff640 1 --2- 172.21.15.40:0/3178403885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa12a50 0x55727aa1ceb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80cc9a0 tx=0x7f7ca809b420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79283e299d733aaa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7c977fe640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:43:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.912+0000 7f7c977fe640 1 -- 172.21.15.40:0/3178403885 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca809b7f0 con 0x55727aa12a50 2024-03-18T22:43:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.913+0000 7f7cc125b740 1 -- 172.21.15.40:0/3178403885 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cb0011700 con 0x55727aa12a50 2024-03-18T22:43:43.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.913+0000 7f7c977fe640 1 -- 172.21.15.40:0/3178403885 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7ca809ac40 con 0x55727aa12a50 2024-03-18T22:43:43.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.913+0000 7f7c977fe640 1 -- 172.21.15.40:0/3178403885 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7ca809f940 con 0x55727aa12a50 2024-03-18T22:43:43.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.913+0000 7f7cc125b740 10 client.25657 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:43.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.913+0000 7f7cc125b740 1 -- 172.21.15.40:0/3178403885 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55727a68b3f0 con 0x55727aa12a50 2024-03-18T22:43:43.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.913+0000 7f7c977fe640 1 -- 172.21.15.40:0/3178403885 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cb0011700 con 0x55727aa12a50 2024-03-18T22:43:43.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.913+0000 7f7c977fe640 1 -- 172.21.15.40:0/3178403885 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7ca80bb040 con 0x55727aa12a50 2024-03-18T22:43:43.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.913+0000 7f7cc125b740 10 client.25657 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:43.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.913+0000 7f7cc125b740 10 client.25657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:43:43.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.913+0000 7f7cc125b740 20 client.25657 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:43.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.913+0000 7f7cc125b740 1 --2- 172.21.15.40:0/3178403885 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55727aa28d10 0x55727aa2b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.913+0000 7f7cc125b740 4 client.25657 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:43:43.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.913+0000 7f7cc125b740 1 -- 172.21.15.40:0/3178403885 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55727a984a40 con 0x55727aa28d10 2024-03-18T22:43:43.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.913+0000 7f7cb77fe640 1 --2- 172.21.15.40:0/3178403885 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55727aa28d10 0x55727aa2b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.914+0000 7f7c977fe640 10 client.25657 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:43.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.914+0000 7f7cb77fe640 1 --2- 172.21.15.40:0/3178403885 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55727aa28d10 0x55727aa2b180 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b5a1504e0cdfcee server_cookie=a4ea45a00d5bb69a in_seq=0 out_seq=0 2024-03-18T22:43:43.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.914+0000 7f7c977fe640 1 -- 172.21.15.40:0/3178403885 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55727a984a40 con 0x55727aa28d10 2024-03-18T22:43:43.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.914+0000 7f7c977fe640 10 client.25657 handle_command_reply: tid=0 2024-03-18T22:43:43.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.914+0000 7f7cc125b740 1 client.25657 shutdown 2024-03-18T22:43:43.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.915+0000 7f7cc125b740 1 -- 172.21.15.40:0/3178403885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa12a50 msgr2=0x55727aa1ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.915+0000 7f7cc125b740 1 --2- 172.21.15.40:0/3178403885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa12a50 0x55727aa1ceb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80cc9a0 tx=0x7f7ca809b420 comp rx=0 tx=0).stop 2024-03-18T22:43:43.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.915+0000 7f7cc125b740 1 -- 172.21.15.40:0/3178403885 shutdown_connections 2024-03-18T22:43:43.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.915+0000 7f7cc125b740 1 --2- 172.21.15.40:0/3178403885 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55727aa28d10 0x55727aa2b180 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.915+0000 7f7cc125b740 1 --2- 172.21.15.40:0/3178403885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa185b0 0x55727aa24430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.915+0000 7f7c977fe640 0 client.25657 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:43.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.915+0000 7f7cc125b740 1 --2- 172.21.15.40:0/3178403885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa13440 0x55727aa1d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.915+0000 7f7cc125b740 1 --2- 172.21.15.40:0/3178403885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa12a50 0x55727aa1ceb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.915+0000 7f7cc125b740 1 -- 172.21.15.40:0/3178403885 >> 172.21.15.40:0/3178403885 conn(0x55727aa0dfc0 msgr2=0x55727aa0d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:43.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.915+0000 7f7cc125b740 1 -- 172.21.15.40:0/3178403885 shutdown_connections 2024-03-18T22:43:43.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.915+0000 7f7cc125b740 1 -- 172.21.15.40:0/3178403885 wait complete. 2024-03-18T22:43:43.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.915+0000 7f7cc125b740 20 client.25657 trim_cache size 0 max 16384 2024-03-18T22:43:43.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.949+0000 7f7cc125b740 1 Processor -- start 2024-03-18T22:43:43.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.949+0000 7f7cc125b740 1 -- start start 2024-03-18T22:43:43.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.949+0000 7f7cc125b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa0dfc0 0x55727aa091c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.949+0000 7f7cc125b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa12a50 0x55727aa243f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.949+0000 7f7cc125b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa13440 0x55727aa1d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.949+0000 7f7cc125b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb81ac780 con 0x55727aa0dfc0 2024-03-18T22:43:43.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.949+0000 7f7cc125b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb81ae2c0 con 0x55727aa12a50 2024-03-18T22:43:43.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.949+0000 7f7cc125b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb81ace50 con 0x55727aa13440 2024-03-18T22:43:43.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.949+0000 7f7cb77fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa12a50 0x55727aa243f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.949+0000 7f7cb7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa0dfc0 0x55727aa091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.949+0000 7f7cb77fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa12a50 0x55727aa243f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60614/0 (socket says 172.21.15.40:60614) 2024-03-18T22:43:43.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.949+0000 7f7cb77fe640 1 -- 172.21.15.40:0/2844458858 learned_addr learned my addr 172.21.15.40:0/2844458858 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:43.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.949+0000 7f7cbcc12640 1 --2- 172.21.15.40:0/2844458858 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa13440 0x55727aa1d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.950+0000 7f7cb77fe640 1 -- 172.21.15.40:0/2844458858 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa13440 msgr2=0x55727aa1d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:43.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.950+0000 7f7cb77fe640 1 --2- 172.21.15.40:0/2844458858 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa13440 0x55727aa1d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.950+0000 7f7cb77fe640 1 -- 172.21.15.40:0/2844458858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa0dfc0 msgr2=0x55727aa091c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:43.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.950+0000 7f7cb77fe640 1 --2- 172.21.15.40:0/2844458858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa0dfc0 0x55727aa091c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.950+0000 7f7cb77fe640 1 -- 172.21.15.40:0/2844458858 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55727a64b0f0 con 0x55727aa12a50 2024-03-18T22:43:43.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.950+0000 7f7cb7fff640 1 --2- 172.21.15.40:0/2844458858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa0dfc0 0x55727aa091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:43.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.950+0000 7f7cbcc12640 1 --2- 172.21.15.40:0/2844458858 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa13440 0x55727aa1d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:43.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.950+0000 7f7cb77fe640 1 --2- 172.21.15.40:0/2844458858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa12a50 0x55727aa243f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7cac005ab0 tx=0x7f7cac005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8528b14850b8247 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:43.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.950+0000 7f7c9dffb640 1 -- 172.21.15.40:0/2844458858 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac04e020 con 0x55727aa12a50 2024-03-18T22:43:43.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.950+0000 7f7c9dffb640 1 -- 172.21.15.40:0/2844458858 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cac04aab0 con 0x55727aa12a50 2024-03-18T22:43:43.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.950+0000 7f7c9dffb640 1 -- 172.21.15.40:0/2844458858 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac048070 con 0x55727aa12a50 2024-03-18T22:43:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.951+0000 7f7cc125b740 1 -- 172.21.15.40:0/2844458858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa12a50 msgr2=0x55727aa243f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.951+0000 7f7cc125b740 1 --2- 172.21.15.40:0/2844458858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa12a50 0x55727aa243f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7cac005ab0 tx=0x7f7cac005fd0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.951+0000 7f7cc125b740 1 -- 172.21.15.40:0/2844458858 shutdown_connections 2024-03-18T22:43:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.951+0000 7f7cc125b740 1 --2- 172.21.15.40:0/2844458858 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa13440 0x55727aa1d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.951+0000 7f7cc125b740 1 --2- 172.21.15.40:0/2844458858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa12a50 0x55727aa243f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.951+0000 7f7cc125b740 1 --2- 172.21.15.40:0/2844458858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa0dfc0 0x55727aa091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.951+0000 7f7cc125b740 1 -- 172.21.15.40:0/2844458858 >> 172.21.15.40:0/2844458858 conn(0x55727aa14220 msgr2=0x55727aa11220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.951+0000 7f7cc125b740 1 -- 172.21.15.40:0/2844458858 shutdown_connections 2024-03-18T22:43:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.951+0000 7f7cc125b740 1 -- 172.21.15.40:0/2844458858 wait complete. 2024-03-18T22:43:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.951+0000 7f7cc125b740 1 -- start start 2024-03-18T22:43:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cc125b740 1 Processor -- start 2024-03-18T22:43:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cc125b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa0dfc0 0x55727aa21920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cc125b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa12a50 0x55727aa21e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cc125b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa13440 0x55727aa22e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cc125b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb81adbf0 con 0x55727aa12a50 2024-03-18T22:43:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cc125b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb81ad520 con 0x55727aa0dfc0 2024-03-18T22:43:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cc125b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb81aa760 con 0x55727aa13440 2024-03-18T22:43:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cc125b740 10 client.0 _ll_register_callbacks cb 0x7f7ca8097a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:43:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cb77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa12a50 0x55727aa21e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cb7fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa0dfc0 0x55727aa21920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cb77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa12a50 0x55727aa21e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46228/0 (socket says 172.21.15.40:46228) 2024-03-18T22:43:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cb77fe640 1 -- 172.21.15.40:0/3918650188 learned_addr learned my addr 172.21.15.40:0/3918650188 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:43:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cbcc12640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa13440 0x55727aa22e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cb77fe640 1 -- 172.21.15.40:0/3918650188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa13440 msgr2=0x55727aa22e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cb77fe640 1 --2- 172.21.15.40:0/3918650188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa13440 0x55727aa22e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.952+0000 7f7cb77fe640 1 -- 172.21.15.40:0/3918650188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa0dfc0 msgr2=0x55727aa21920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:43:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.953+0000 7f7cb77fe640 1 --2- 172.21.15.40:0/3918650188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa0dfc0 0x55727aa21920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.953+0000 7f7cb77fe640 1 -- 172.21.15.40:0/3918650188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55727aa233f0 con 0x55727aa12a50 2024-03-18T22:43:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.953+0000 7f7cbcc12640 1 --2- 172.21.15.40:0/3918650188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa13440 0x55727aa22e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.953+0000 7f7cb7fff640 1 --2- 172.21.15.40:0/3918650188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa0dfc0 0x55727aa21920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:43:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.953+0000 7f7cb77fe640 1 --2- 172.21.15.40:0/3918650188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa12a50 0x55727aa21e60 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7cac036040 tx=0x7f7cac0095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbec2b066c4c3574 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:43:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.953+0000 7f7c977fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:43:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.953+0000 7f7c977fe640 1 -- 172.21.15.40:0/3918650188 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac04e040 con 0x55727aa12a50 2024-03-18T22:43:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.953+0000 7f7c977fe640 1 -- 172.21.15.40:0/3918650188 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7cac005040 con 0x55727aa12a50 2024-03-18T22:43:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.953+0000 7f7cc125b740 1 -- 172.21.15.40:0/3918650188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cac04aab0 con 0x55727aa12a50 2024-03-18T22:43:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.953+0000 7f7c977fe640 1 -- 172.21.15.40:0/3918650188 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7cac05f050 con 0x55727aa12a50 2024-03-18T22:43:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.953+0000 7f7cc125b740 10 client.15804 fetch_fsmap learned FSMap version 28 2024-03-18T22:43:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.953+0000 7f7cc125b740 1 -- 172.21.15.40:0/3918650188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55727aa24730 con 0x55727aa12a50 2024-03-18T22:43:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.953+0000 7f7c977fe640 1 -- 172.21.15.40:0/3918650188 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cac04aab0 con 0x55727aa12a50 2024-03-18T22:43:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.954+0000 7f7c977fe640 1 -- 172.21.15.40:0/3918650188 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7cac05a030 con 0x55727aa12a50 2024-03-18T22:43:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.954+0000 7f7cc125b740 10 client.15804 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T22:43:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.954+0000 7f7cc125b740 10 client.15804 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:43:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.954+0000 7f7cc125b740 20 client.15804 populate_metadata read hostname 'smithi040' 2024-03-18T22:43:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.954+0000 7f7cc125b740 1 --2- 172.21.15.40:0/3918650188 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55727aa185b0 0x55727aa91840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:43:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.954+0000 7f7cc125b740 4 client.15804 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:43:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.954+0000 7f7cc125b740 1 -- 172.21.15.40:0/3918650188 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55727a97ed90 con 0x55727aa185b0 2024-03-18T22:43:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.954+0000 7f7cb7fff640 1 --2- 172.21.15.40:0/3918650188 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55727aa185b0 0x55727aa91840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:43:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.954+0000 7f7c977fe640 10 client.15804 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.954+0000 7f7cb7fff640 1 --2- 172.21.15.40:0/3918650188 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55727aa185b0 0x55727aa91840 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f992192fe724f8e server_cookie=b63791dc9bb6c756 in_seq=0 out_seq=0 2024-03-18T22:43:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.955+0000 7f7c977fe640 1 -- 172.21.15.40:0/3918650188 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55727a97ed90 con 0x55727aa185b0 2024-03-18T22:43:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.955+0000 7f7c977fe640 10 client.15804 handle_command_reply: tid=0 2024-03-18T22:43:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.955+0000 7f7cc125b740 1 client.15804 shutdown 2024-03-18T22:43:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.955+0000 7f7cc125b740 1 -- 172.21.15.40:0/3918650188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa12a50 msgr2=0x55727aa21e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.955+0000 7f7cc125b740 1 --2- 172.21.15.40:0/3918650188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa12a50 0x55727aa21e60 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7cac036040 tx=0x7f7cac0095a0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.955+0000 7f7cc125b740 1 -- 172.21.15.40:0/3918650188 shutdown_connections 2024-03-18T22:43:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.955+0000 7f7cc125b740 1 --2- 172.21.15.40:0/3918650188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55727aa13440 0x55727aa22e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.955+0000 7f7cc125b740 1 --2- 172.21.15.40:0/3918650188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55727aa12a50 0x55727aa21e60 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.955+0000 7f7cc125b740 1 --2- 172.21.15.40:0/3918650188 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55727aa185b0 0x55727aa91840 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.955+0000 7f7c977fe640 0 client.15804 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:43:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.955+0000 7f7cc125b740 1 --2- 172.21.15.40:0/3918650188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55727aa0dfc0 0x55727aa21920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.955+0000 7f7cc125b740 1 -- 172.21.15.40:0/3918650188 >> 172.21.15.40:0/3918650188 conn(0x55727aa14220 msgr2=0x55727aa11ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.955+0000 7f7cc125b740 1 -- 172.21.15.40:0/3918650188 shutdown_connections 2024-03-18T22:43:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.955+0000 7f7cc125b740 1 -- 172.21.15.40:0/3918650188 wait complete. 2024-03-18T22:43:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.955+0000 7f7cc125b740 20 client.15804 trim_cache size 0 max 16384 2024-03-18T22:43:43.962 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:43:43.962 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:43:43.962 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:43:43.962 INFO:teuthology.orchestra.run.smithi040.stdout: "749b40b8-4491-4d1f-b091-4500595dbdc2": { 2024-03-18T22:43:43.963 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:43:43.963 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "749b40b8-4491-4d1f-b091-4500595dbdc2", 2024-03-18T22:43:43.963 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:43:43.963 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:43:43.963 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:43:43.963 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:43:43.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.956+0000 7f7cbe675640 1 -- 172.21.15.40:0/2632911564 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ca00a2880 msgr2=0x7f7ca00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.956+0000 7f7cbe675640 1 --2- 172.21.15.40:0/2632911564 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ca00a2880 0x7f7ca00a4d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7cb000e720 tx=0x7f7cb0018000 comp rx=0 tx=0).stop 2024-03-18T22:43:43.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.956+0000 7f7cbe675640 1 -- 172.21.15.40:0/2632911564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb8100720 msgr2=0x7f7cb81046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:43:43.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.956+0000 7f7cbe675640 1 --2- 172.21.15.40:0/2632911564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb8100720 0x7f7cb81046b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800e9c0 tx=0x7f7ca800ee80 comp rx=0 tx=0).stop 2024-03-18T22:43:43.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.956+0000 7f7cbe675640 1 -- 172.21.15.40:0/2632911564 shutdown_connections 2024-03-18T22:43:43.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.956+0000 7f7cbe675640 1 --2- 172.21.15.40:0/2632911564 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7ca00a2880 0x7f7ca00a4d40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.956+0000 7f7cbe675640 1 --2- 172.21.15.40:0/2632911564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7cb81022d0 0x7f7cb8103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.956+0000 7f7cbe675640 1 --2- 172.21.15.40:0/2632911564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7cb8101910 0x7f7cb8102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.956+0000 7f7cbe675640 1 --2- 172.21.15.40:0/2632911564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb8100720 0x7f7cb81046b0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:43:43.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.956+0000 7f7cbe675640 1 -- 172.21.15.40:0/2632911564 >> 172.21.15.40:0/2632911564 conn(0x7f7cb80fbcb0 msgr2=0x7f7cb80689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:43:43.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.956+0000 7f7cbe675640 1 -- 172.21.15.40:0/2632911564 shutdown_connections 2024-03-18T22:43:43.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:43:43.956+0000 7f7cbe675640 1 -- 172.21.15.40:0/2632911564 wait complete. 2024-03-18T22:43:43.969 INFO:tasks.cephfs.filesystem:scrub status for tag:749b40b8-4491-4d1f-b091-4500595dbdc2 - {'path': '/', 'tag': '749b40b8-4491-4d1f-b091-4500595dbdc2', 'options': 'recursive,force'} 2024-03-18T22:43:45.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:44 smithi113 ceph-mon[66775]: pgmap v383: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:43:45.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:44 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:43:45.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:44 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:43:45.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:44 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:43:45.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:44 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:43:45.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:44 smithi103 ceph-mon[66377]: pgmap v383: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:43:45.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:44 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:43:45.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:44 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:43:45.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:44 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:43:45.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:44 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:43:45.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:44 smithi040 ceph-mon[62643]: pgmap v383: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T22:43:45.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:44 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:43:45.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:44 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:43:45.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:44 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:43:45.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:44 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:43:47.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:46 smithi113 ceph-mon[66775]: pgmap v384: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-03-18T22:43:47.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:46 smithi103 ceph-mon[66377]: pgmap v384: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-03-18T22:43:47.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:46 smithi040 ceph-mon[62643]: pgmap v384: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-03-18T22:43:48.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:47 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:43:48.108 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:47 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:43:48.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:47 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:43:49.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:48 smithi113 ceph-mon[66775]: pgmap v385: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-18T22:43:49.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:48 smithi103 ceph-mon[66377]: pgmap v385: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-18T22:43:49.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:48 smithi040 ceph-mon[62643]: pgmap v385: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-18T22:43:50.747 INFO:tasks.workunit.client.0.smithi040.stderr:Note: switching to 'e48b1f6c0608b2615e1890cc46286c3877d6e910'. 2024-03-18T22:43:50.747 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-03-18T22:43:50.748 INFO:tasks.workunit.client.0.smithi040.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-18T22:43:50.748 INFO:tasks.workunit.client.0.smithi040.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-18T22:43:50.748 INFO:tasks.workunit.client.0.smithi040.stderr:state without impacting any branches by switching back to a branch. 2024-03-18T22:43:50.748 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-03-18T22:43:50.748 INFO:tasks.workunit.client.0.smithi040.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-18T22:43:50.748 INFO:tasks.workunit.client.0.smithi040.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-18T22:43:50.748 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-03-18T22:43:50.748 INFO:tasks.workunit.client.0.smithi040.stderr: git switch -c 2024-03-18T22:43:50.748 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-03-18T22:43:50.748 INFO:tasks.workunit.client.0.smithi040.stderr:Or undo this operation with: 2024-03-18T22:43:50.748 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-03-18T22:43:50.748 INFO:tasks.workunit.client.0.smithi040.stderr: git switch - 2024-03-18T22:43:50.748 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-03-18T22:43:50.748 INFO:tasks.workunit.client.0.smithi040.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-18T22:43:50.748 INFO:tasks.workunit.client.0.smithi040.stderr: 2024-03-18T22:43:50.749 INFO:tasks.workunit.client.0.smithi040.stderr:HEAD is now at e48b1f6c060 testfix 2024-03-18T22:43:50.752 DEBUG:teuthology.orchestra.run.smithi040:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-18T22:43:50.772 INFO:tasks.workunit.client.0.smithi040.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-18T22:43:50.774 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T22:43:50.775 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-18T22:43:51.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:50 smithi113 ceph-mon[66775]: pgmap v386: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-18T22:43:51.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:50 smithi103 ceph-mon[66377]: pgmap v386: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-18T22:43:51.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:50 smithi040 ceph-mon[62643]: pgmap v386: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-18T22:43:51.471 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-18T22:43:51.538 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-18T22:43:51.577 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T22:43:51.578 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T22:43:51.579 INFO:tasks.workunit.client.0.smithi040.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-18T22:43:51.617 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T22:43:51.620 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:43:51.620 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-18T22:43:51.676 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-03-18T22:43:51.677 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-03-18T22:43:51.677 DEBUG:teuthology.orchestra.run.smithi040:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-18T22:43:51.739 INFO:tasks.workunit.client.0.smithi040.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-18T22:43:51.740 INFO:tasks.workunit.client.0.smithi040.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-03-18T22:43:51.740 INFO:tasks.workunit.client.0.smithi040.stderr:+ trap -- retry EXIT 2024-03-18T22:43:51.740 INFO:tasks.workunit.client.0.smithi040.stderr:+ rm -rf ffsb 2024-03-18T22:43:51.740 INFO:tasks.workunit.client.0.smithi040.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-03-18T22:43:51.745 INFO:tasks.workunit.client.0.smithi040.stderr:Cloning into 'ffsb'... 2024-03-18T22:43:53.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:52 smithi113 ceph-mon[66775]: pgmap v387: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-03-18T22:43:53.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:52 smithi103 ceph-mon[66377]: pgmap v387: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-03-18T22:43:53.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:52 smithi040 ceph-mon[62643]: pgmap v387: 129 pgs: 129 active+clean; 664 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-03-18T22:43:53.751 INFO:tasks.workunit.client.0.smithi040.stderr:+ trap - EXIT 2024-03-18T22:43:53.751 INFO:tasks.workunit.client.0.smithi040.stderr:+ cd ffsb 2024-03-18T22:43:53.752 INFO:tasks.workunit.client.0.smithi040.stderr:+ ./configure 2024-03-18T22:43:53.880 INFO:tasks.workunit.client.0.smithi040.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-18T22:43:54.891 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether build environment is sane... yes 2024-03-18T22:43:54.909 INFO:tasks.workunit.client.0.smithi040.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-18T22:43:54.910 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gawk... gawk 2024-03-18T22:43:54.934 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether make sets $(MAKE)... yes 2024-03-18T22:43:54.979 INFO:tasks.workunit.client.0.smithi040.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-18T22:43:54.979 INFO:tasks.workunit.client.0.smithi040.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-18T22:43:55.001 INFO:tasks.workunit.client.0.smithi040.stdout:checking for style of include used by make... GNU 2024-03-18T22:43:55.009 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gcc... gcc 2024-03-18T22:43:55.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:54 smithi113 ceph-mon[66775]: pgmap v388: 129 pgs: 129 active+clean; 699 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.4 KiB/s wr, 10 op/s 2024-03-18T22:43:55.097 INFO:tasks.workunit.client.0.smithi040.stdout:checking for C compiler default output file name... a.out 2024-03-18T22:43:55.100 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether the C compiler works... yes 2024-03-18T22:43:55.104 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether we are cross compiling... no 2024-03-18T22:43:55.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:54 smithi103 ceph-mon[66377]: pgmap v388: 129 pgs: 129 active+clean; 699 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.4 KiB/s wr, 10 op/s 2024-03-18T22:43:55.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:54 smithi040 ceph-mon[62643]: pgmap v388: 129 pgs: 129 active+clean; 699 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.4 KiB/s wr, 10 op/s 2024-03-18T22:43:55.156 INFO:tasks.workunit.client.0.smithi040.stdout:checking for suffix of executables... 2024-03-18T22:43:55.198 INFO:tasks.workunit.client.0.smithi040.stdout:checking for suffix of object files... o 2024-03-18T22:43:55.244 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether we are using the GNU C compiler... yes 2024-03-18T22:43:55.291 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether gcc accepts -g... yes 2024-03-18T22:43:55.393 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-18T22:43:55.569 INFO:tasks.workunit.client.0.smithi040.stdout:checking dependency style of gcc... gcc3 2024-03-18T22:43:55.657 INFO:tasks.workunit.client.0.smithi040.stdout:checking how to run the C preprocessor... gcc -E 2024-03-18T22:43:55.718 INFO:tasks.workunit.client.0.smithi040.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-18T22:43:55.720 INFO:tasks.workunit.client.0.smithi040.stdout:checking for egrep... /usr/bin/grep -E 2024-03-18T22:43:55.733 INFO:tasks.workunit.client.0.smithi040.stdout:checking for AIX... no 2024-03-18T22:43:55.765 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gcc... (cached) gcc 2024-03-18T22:43:55.776 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-18T22:43:55.778 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-18T22:43:55.778 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-18T22:43:55.778 INFO:tasks.workunit.client.0.smithi040.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-18T22:43:55.861 INFO:tasks.workunit.client.0.smithi040.stdout:checking for main in -lm... yes 2024-03-18T22:43:55.937 INFO:tasks.workunit.client.0.smithi040.stdout:checking for main in -lpthread... yes 2024-03-18T22:43:56.156 INFO:tasks.workunit.client.0.smithi040.stdout:checking for ANSI C header files... yes 2024-03-18T22:43:56.241 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-18T22:43:56.310 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/types.h... yes 2024-03-18T22:43:56.380 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/stat.h... yes 2024-03-18T22:43:56.450 INFO:tasks.workunit.client.0.smithi040.stdout:checking for stdlib.h... yes 2024-03-18T22:43:56.524 INFO:tasks.workunit.client.0.smithi040.stdout:checking for string.h... yes 2024-03-18T22:43:56.601 INFO:tasks.workunit.client.0.smithi040.stdout:checking for memory.h... yes 2024-03-18T22:43:56.675 INFO:tasks.workunit.client.0.smithi040.stdout:checking for strings.h... yes 2024-03-18T22:43:56.746 INFO:tasks.workunit.client.0.smithi040.stdout:checking for inttypes.h... yes 2024-03-18T22:43:56.817 INFO:tasks.workunit.client.0.smithi040.stdout:checking for stdint.h... yes 2024-03-18T22:43:56.890 INFO:tasks.workunit.client.0.smithi040.stdout:checking for unistd.h... yes 2024-03-18T22:43:56.971 INFO:tasks.workunit.client.0.smithi040.stdout:checking pthread.h usability... yes 2024-03-18T22:43:57.002 INFO:tasks.workunit.client.0.smithi040.stdout:checking pthread.h presence... yes 2024-03-18T22:43:57.003 INFO:tasks.workunit.client.0.smithi040.stdout:checking for pthread.h... yes 2024-03-18T22:43:57.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:56 smithi113 ceph-mon[66775]: pgmap v389: 129 pgs: 129 active+clean; 2.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 154 KiB/s wr, 28 op/s 2024-03-18T22:43:57.083 INFO:tasks.workunit.client.0.smithi040.stdout:checking fcntl.h usability... yes 2024-03-18T22:43:57.108 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:56 smithi103 ceph-mon[66377]: pgmap v389: 129 pgs: 129 active+clean; 2.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 154 KiB/s wr, 28 op/s 2024-03-18T22:43:57.115 INFO:tasks.workunit.client.0.smithi040.stdout:checking fcntl.h presence... yes 2024-03-18T22:43:57.115 INFO:tasks.workunit.client.0.smithi040.stdout:checking for fcntl.h... yes 2024-03-18T22:43:57.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:56 smithi040 ceph-mon[62643]: pgmap v389: 129 pgs: 129 active+clean; 2.4 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 154 KiB/s wr, 28 op/s 2024-03-18T22:43:57.191 INFO:tasks.workunit.client.0.smithi040.stdout:checking limits.h usability... yes 2024-03-18T22:43:57.228 INFO:tasks.workunit.client.0.smithi040.stdout:checking limits.h presence... yes 2024-03-18T22:43:57.229 INFO:tasks.workunit.client.0.smithi040.stdout:checking for limits.h... yes 2024-03-18T22:43:57.257 INFO:tasks.workunit.client.0.smithi040.stdout:checking for stdint.h... (cached) yes 2024-03-18T22:43:57.338 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/time.h usability... yes 2024-03-18T22:43:57.367 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/time.h presence... yes 2024-03-18T22:43:57.368 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/time.h... yes 2024-03-18T22:43:57.404 INFO:tasks.workunit.client.0.smithi040.stdout:checking for unistd.h... (cached) yes 2024-03-18T22:43:57.472 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/vfs.h usability... yes 2024-03-18T22:43:57.503 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/vfs.h presence... yes 2024-03-18T22:43:57.504 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/vfs.h... yes 2024-03-18T22:43:57.579 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/limits.h usability... no 2024-03-18T22:43:57.612 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/limits.h presence... no 2024-03-18T22:43:57.612 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/limits.h... no 2024-03-18T22:43:57.657 INFO:tasks.workunit.client.0.smithi040.stdout:checking for an ANSI C-conforming const... yes 2024-03-18T22:43:57.705 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-18T22:43:57.708 INFO:tasks.workunit.client.0.smithi040.stdout:checking for function prototypes... yes 2024-03-18T22:43:57.775 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether setvbuf arguments are reversed... no 2024-03-18T22:43:57.868 INFO:tasks.workunit.client.0.smithi040.stdout:checking for system... yes 2024-03-18T22:43:57.959 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gettimeofday... yes 2024-03-18T22:43:58.052 INFO:tasks.workunit.client.0.smithi040.stdout:checking for mkdir... yes 2024-03-18T22:43:58.141 INFO:tasks.workunit.client.0.smithi040.stdout:checking for strerror... yes 2024-03-18T22:43:58.237 INFO:tasks.workunit.client.0.smithi040.stdout:checking for open64... yes 2024-03-18T22:43:58.342 INFO:tasks.workunit.client.0.smithi040.stdout:checking for stat64... yes 2024-03-18T22:43:58.442 INFO:tasks.workunit.client.0.smithi040.stdout:checking for fseeko64... yes 2024-03-18T22:43:58.537 INFO:tasks.workunit.client.0.smithi040.stdout:checking for lrand48_r... yes 2024-03-18T22:43:58.630 INFO:tasks.workunit.client.0.smithi040.stdout:checking for srand48_r... yes 2024-03-18T22:43:58.664 INFO:tasks.workunit.client.0.smithi040.stdout:configure: creating ./config.status 2024-03-18T22:43:58.806 INFO:tasks.workunit.client.0.smithi040.stdout:config.status: creating Makefile 2024-03-18T22:43:58.824 INFO:tasks.workunit.client.0.smithi040.stdout:config.status: creating config.h 2024-03-18T22:43:58.852 INFO:tasks.workunit.client.0.smithi040.stdout:config.status: executing depfiles commands 2024-03-18T22:43:58.932 INFO:tasks.workunit.client.0.smithi040.stdout:config.status: executing default commands 2024-03-18T22:43:59.021 INFO:tasks.workunit.client.0.smithi040.stderr:+ make 2024-03-18T22:43:59.057 INFO:tasks.workunit.client.0.smithi040.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-03-18T22:43:59.064 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-03-18T22:43:59.065 INFO:tasks.workunit.client.0.smithi040.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-03-18T22:43:59.065 INFO:tasks.workunit.client.0.smithi040.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-03-18T22:43:59.065 INFO:tasks.workunit.client.0.smithi040.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-03-18T22:43:59.065 INFO:tasks.workunit.client.0.smithi040.stderr: any GNU archive site. 2024-03-18T22:43:59.069 INFO:tasks.workunit.client.0.smithi040.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-03-18T22:43:59.073 INFO:tasks.workunit.client.0.smithi040.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-03-18T22:43:59.074 INFO:tasks.workunit.client.0.smithi040.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-03-18T22:43:59.074 INFO:tasks.workunit.client.0.smithi040.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-03-18T22:43:59.075 INFO:tasks.workunit.client.0.smithi040.stderr: You might want to install the `Automake' and `Perl' packages. 2024-03-18T22:43:59.075 INFO:tasks.workunit.client.0.smithi040.stderr: Grab them from any GNU archive site. 2024-03-18T22:43:59.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:43:58 smithi113 ceph-mon[66775]: pgmap v390: 129 pgs: 129 active+clean; 2.7 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 191 KiB/s wr, 40 op/s 2024-03-18T22:43:59.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:43:58 smithi103 ceph-mon[66377]: pgmap v390: 129 pgs: 129 active+clean; 2.7 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 191 KiB/s wr, 40 op/s 2024-03-18T22:43:59.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:43:58 smithi040 ceph-mon[62643]: pgmap v390: 129 pgs: 129 active+clean; 2.7 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 191 KiB/s wr, 40 op/s 2024-03-18T22:43:59.437 INFO:tasks.workunit.client.0.smithi040.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-03-18T22:43:59.769 INFO:tasks.workunit.client.0.smithi040.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-03-18T22:43:59.770 INFO:tasks.workunit.client.0.smithi040.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-03-18T22:43:59.770 INFO:tasks.workunit.client.0.smithi040.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-03-18T22:43:59.770 INFO:tasks.workunit.client.0.smithi040.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-03-18T22:43:59.908 INFO:tasks.workunit.client.0.smithi040.stdout:/bin/sh ./config.status --recheck 2024-03-18T22:43:59.971 INFO:tasks.workunit.client.0.smithi040.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-03-18T22:44:00.130 INFO:tasks.workunit.client.0.smithi040.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-18T22:44:01.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:00 smithi113 ceph-mon[66775]: pgmap v391: 129 pgs: 129 active+clean; 3.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 240 KiB/s wr, 56 op/s 2024-03-18T22:44:01.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:00 smithi103 ceph-mon[66377]: pgmap v391: 129 pgs: 129 active+clean; 3.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 240 KiB/s wr, 56 op/s 2024-03-18T22:44:01.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:00 smithi040 ceph-mon[62643]: pgmap v391: 129 pgs: 129 active+clean; 3.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 240 KiB/s wr, 56 op/s 2024-03-18T22:44:01.144 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether build environment is sane... yes 2024-03-18T22:44:01.153 INFO:tasks.workunit.client.0.smithi040.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-18T22:44:01.154 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gawk... gawk 2024-03-18T22:44:01.184 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether make sets $(MAKE)... yes 2024-03-18T22:44:01.233 INFO:tasks.workunit.client.0.smithi040.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-18T22:44:01.233 INFO:tasks.workunit.client.0.smithi040.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-18T22:44:01.257 INFO:tasks.workunit.client.0.smithi040.stdout:checking for style of include used by make... GNU 2024-03-18T22:44:01.262 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gcc... gcc 2024-03-18T22:44:01.398 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether the C compiler works... yes 2024-03-18T22:44:01.399 INFO:tasks.workunit.client.0.smithi040.stdout:checking for C compiler default output file name... a.out 2024-03-18T22:44:01.455 INFO:tasks.workunit.client.0.smithi040.stdout:checking for suffix of executables... 2024-03-18T22:44:01.507 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether we are cross compiling... no 2024-03-18T22:44:01.562 INFO:tasks.workunit.client.0.smithi040.stdout:checking for suffix of object files... o 2024-03-18T22:44:01.599 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether we are using the GNU C compiler... yes 2024-03-18T22:44:01.636 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether gcc accepts -g... yes 2024-03-18T22:44:01.685 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-18T22:44:01.865 INFO:tasks.workunit.client.0.smithi040.stdout:checking dependency style of gcc... gcc3 2024-03-18T22:44:01.935 INFO:tasks.workunit.client.0.smithi040.stdout:checking how to run the C preprocessor... gcc -E 2024-03-18T22:44:02.005 INFO:tasks.workunit.client.0.smithi040.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-18T22:44:02.006 INFO:tasks.workunit.client.0.smithi040.stdout:checking for egrep... /usr/bin/grep -E 2024-03-18T22:44:02.191 INFO:tasks.workunit.client.0.smithi040.stdout:checking for ANSI C header files... yes 2024-03-18T22:44:02.243 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/types.h... yes 2024-03-18T22:44:02.309 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/stat.h... yes 2024-03-18T22:44:02.363 INFO:tasks.workunit.client.0.smithi040.stdout:checking for stdlib.h... yes 2024-03-18T22:44:02.422 INFO:tasks.workunit.client.0.smithi040.stdout:checking for string.h... yes 2024-03-18T22:44:02.487 INFO:tasks.workunit.client.0.smithi040.stdout:checking for memory.h... yes 2024-03-18T22:44:02.551 INFO:tasks.workunit.client.0.smithi040.stdout:checking for strings.h... yes 2024-03-18T22:44:02.610 INFO:tasks.workunit.client.0.smithi040.stdout:checking for inttypes.h... yes 2024-03-18T22:44:02.670 INFO:tasks.workunit.client.0.smithi040.stdout:checking for stdint.h... yes 2024-03-18T22:44:02.738 INFO:tasks.workunit.client.0.smithi040.stdout:checking for unistd.h... yes 2024-03-18T22:44:02.802 INFO:tasks.workunit.client.0.smithi040.stdout:checking minix/config.h usability... no 2024-03-18T22:44:02.849 INFO:tasks.workunit.client.0.smithi040.stdout:checking minix/config.h presence... no 2024-03-18T22:44:02.849 INFO:tasks.workunit.client.0.smithi040.stdout:checking for minix/config.h... no 2024-03-18T22:44:02.897 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-03-18T22:44:02.900 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gcc... (cached) gcc 2024-03-18T22:44:02.959 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-18T22:44:02.959 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-18T22:44:02.959 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-18T22:44:02.960 INFO:tasks.workunit.client.0.smithi040.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-18T22:44:03.021 INFO:tasks.workunit.client.0.smithi040.stdout:checking for main in -lm... yes 2024-03-18T22:44:03.089 INFO:tasks.workunit.client.0.smithi040.stdout:checking for main in -lpthread... yes 2024-03-18T22:44:03.096 INFO:tasks.workunit.client.0.smithi040.stdout:checking for ANSI C header files... (cached) yes 2024-03-18T22:44:03.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:02 smithi103 ceph-mon[66377]: pgmap v392: 129 pgs: 129 active+clean; 5.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 469 KiB/s wr, 80 op/s 2024-03-18T22:44:03.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:02 smithi040 ceph-mon[62643]: pgmap v392: 129 pgs: 129 active+clean; 5.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 469 KiB/s wr, 80 op/s 2024-03-18T22:44:03.148 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-18T22:44:03.213 INFO:tasks.workunit.client.0.smithi040.stdout:checking pthread.h usability... yes 2024-03-18T22:44:03.242 INFO:tasks.workunit.client.0.smithi040.stdout:checking pthread.h presence... yes 2024-03-18T22:44:03.243 INFO:tasks.workunit.client.0.smithi040.stdout:checking for pthread.h... yes 2024-03-18T22:44:03.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:02 smithi113 ceph-mon[66775]: pgmap v392: 129 pgs: 129 active+clean; 5.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 469 KiB/s wr, 80 op/s 2024-03-18T22:44:03.337 INFO:tasks.workunit.client.0.smithi040.stdout:checking fcntl.h usability... yes 2024-03-18T22:44:03.365 INFO:tasks.workunit.client.0.smithi040.stdout:checking fcntl.h presence... yes 2024-03-18T22:44:03.366 INFO:tasks.workunit.client.0.smithi040.stdout:checking for fcntl.h... yes 2024-03-18T22:44:03.434 INFO:tasks.workunit.client.0.smithi040.stdout:checking limits.h usability... yes 2024-03-18T22:44:03.461 INFO:tasks.workunit.client.0.smithi040.stdout:checking limits.h presence... yes 2024-03-18T22:44:03.461 INFO:tasks.workunit.client.0.smithi040.stdout:checking for limits.h... yes 2024-03-18T22:44:03.490 INFO:tasks.workunit.client.0.smithi040.stdout:checking for stdint.h... (cached) yes 2024-03-18T22:44:03.550 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/time.h usability... yes 2024-03-18T22:44:03.577 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/time.h presence... yes 2024-03-18T22:44:03.578 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/time.h... yes 2024-03-18T22:44:03.601 INFO:tasks.workunit.client.0.smithi040.stdout:checking for unistd.h... (cached) yes 2024-03-18T22:44:03.657 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/vfs.h usability... yes 2024-03-18T22:44:03.678 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/vfs.h presence... yes 2024-03-18T22:44:03.679 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/vfs.h... yes 2024-03-18T22:44:03.760 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/limits.h usability... no 2024-03-18T22:44:03.806 INFO:tasks.workunit.client.0.smithi040.stdout:checking sys/limits.h presence... no 2024-03-18T22:44:03.806 INFO:tasks.workunit.client.0.smithi040.stdout:checking for sys/limits.h... no 2024-03-18T22:44:03.843 INFO:tasks.workunit.client.0.smithi040.stdout:checking for an ANSI C-conforming const... yes 2024-03-18T22:44:03.883 INFO:tasks.workunit.client.0.smithi040.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-18T22:44:03.971 INFO:tasks.workunit.client.0.smithi040.stdout:checking for system... yes 2024-03-18T22:44:04.054 INFO:tasks.workunit.client.0.smithi040.stdout:checking for gettimeofday... yes 2024-03-18T22:44:04.138 INFO:tasks.workunit.client.0.smithi040.stdout:checking for mkdir... yes 2024-03-18T22:44:04.219 INFO:tasks.workunit.client.0.smithi040.stdout:checking for strerror... yes 2024-03-18T22:44:04.324 INFO:tasks.workunit.client.0.smithi040.stdout:checking for open64... yes 2024-03-18T22:44:04.414 INFO:tasks.workunit.client.0.smithi040.stdout:checking for stat64... yes 2024-03-18T22:44:04.497 INFO:tasks.workunit.client.0.smithi040.stdout:checking for fseeko64... yes 2024-03-18T22:44:04.578 INFO:tasks.workunit.client.0.smithi040.stdout:checking for lrand48_r... yes 2024-03-18T22:44:04.661 INFO:tasks.workunit.client.0.smithi040.stdout:checking for srand48_r... yes 2024-03-18T22:44:04.692 INFO:tasks.workunit.client.0.smithi040.stdout:configure: creating ./config.status 2024-03-18T22:44:04.801 INFO:tasks.workunit.client.0.smithi040.stdout: /bin/sh ./config.status 2024-03-18T22:44:04.871 INFO:tasks.workunit.client.0.smithi040.stdout:config.status: creating Makefile 2024-03-18T22:44:04.891 INFO:tasks.workunit.client.0.smithi040.stdout:config.status: creating config.h 2024-03-18T22:44:04.921 INFO:tasks.workunit.client.0.smithi040.stdout:config.status: executing depfiles commands 2024-03-18T22:44:04.939 INFO:tasks.workunit.client.0.smithi040.stdout:config.status: executing default commands 2024-03-18T22:44:05.008 INFO:tasks.workunit.client.0.smithi040.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-03-18T22:44:05.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:04 smithi103 ceph-mon[66377]: pgmap v393: 129 pgs: 129 active+clean; 5.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 540 KiB/s wr, 109 op/s 2024-03-18T22:44:05.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:04 smithi040 ceph-mon[62643]: pgmap v393: 129 pgs: 129 active+clean; 5.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 540 KiB/s wr, 109 op/s 2024-03-18T22:44:05.176 INFO:tasks.workunit.client.0.smithi040.stdout:rm -f stamp-h1 2024-03-18T22:44:05.180 INFO:tasks.workunit.client.0.smithi040.stdout:touch config.h.in 2024-03-18T22:44:05.182 INFO:tasks.workunit.client.0.smithi040.stdout:cd . && /bin/sh ./config.status config.h 2024-03-18T22:44:05.244 INFO:tasks.workunit.client.0.smithi040.stdout:config.status: creating config.h 2024-03-18T22:44:05.306 INFO:tasks.workunit.client.0.smithi040.stdout:make all-am 2024-03-18T22:44:05.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:04 smithi113 ceph-mon[66775]: pgmap v393: 129 pgs: 129 active+clean; 5.8 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 540 KiB/s wr, 109 op/s 2024-03-18T22:44:05.349 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-18T22:44:05.349 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-03-18T22:44:05.531 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-03-18T22:44:05.537 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-03-18T22:44:05.635 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-03-18T22:44:05.642 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-03-18T22:44:05.746 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-03-18T22:44:05.753 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-03-18T22:44:05.862 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-03-18T22:44:05.869 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-03-18T22:44:06.031 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-03-18T22:44:06.038 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-03-18T22:44:06.109 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-03-18T22:44:06.115 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-03-18T22:44:06.168 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-03-18T22:44:06.175 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-03-18T22:44:06.239 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-03-18T22:44:06.247 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-03-18T22:44:06.479 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-03-18T22:44:06.484 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-03-18T22:44:06.641 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-03-18T22:44:06.648 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-03-18T22:44:06.824 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-03-18T22:44:06.831 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-03-18T22:44:06.918 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-03-18T22:44:06.925 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-03-18T22:44:07.018 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-03-18T22:44:07.023 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-03-18T22:44:07.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:06 smithi103 ceph-mon[66377]: pgmap v394: 129 pgs: 129 active+clean; 7.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 713 KiB/s wr, 127 op/s 2024-03-18T22:44:07.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:06 smithi040 ceph-mon[62643]: pgmap v394: 129 pgs: 129 active+clean; 7.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 713 KiB/s wr, 127 op/s 2024-03-18T22:44:07.155 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-03-18T22:44:07.161 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-03-18T22:44:07.207 INFO:tasks.workunit.client.0.smithi040.stderr:In file included from parser.h:22, 2024-03-18T22:44:07.207 INFO:tasks.workunit.client.0.smithi040.stderr: from parser.c:25: 2024-03-18T22:44:07.207 INFO:tasks.workunit.client.0.smithi040.stderr:parser.c: In function ‘init_filesys’: 2024-03-18T22:44:07.209 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T22:44:07.210 INFO:tasks.workunit.client.0.smithi040.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T22:44:07.210 INFO:tasks.workunit.client.0.smithi040.stderr: | ^ 2024-03-18T22:44:07.210 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T22:44:07.210 INFO:tasks.workunit.client.0.smithi040.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T22:44:07.210 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~ 2024-03-18T22:44:07.210 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T22:44:07.210 INFO:tasks.workunit.client.0.smithi040.stderr: 36 | container_of(ptr, type, member) 2024-03-18T22:44:07.210 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~ 2024-03-18T22:44:07.210 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-18T22:44:07.210 INFO:tasks.workunit.client.0.smithi040.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-18T22:44:07.211 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~ 2024-03-18T22:44:07.211 INFO:tasks.workunit.client.0.smithi040.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T22:44:07.211 INFO:tasks.workunit.client.0.smithi040.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-18T22:44:07.211 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T22:44:07.211 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T22:44:07.211 INFO:tasks.workunit.client.0.smithi040.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T22:44:07.211 INFO:tasks.workunit.client.0.smithi040.stderr: | ^ 2024-03-18T22:44:07.211 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T22:44:07.211 INFO:tasks.workunit.client.0.smithi040.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T22:44:07.211 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~ 2024-03-18T22:44:07.211 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T22:44:07.211 INFO:tasks.workunit.client.0.smithi040.stderr: 36 | container_of(ptr, type, member) 2024-03-18T22:44:07.211 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~ 2024-03-18T22:44:07.212 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-18T22:44:07.212 INFO:tasks.workunit.client.0.smithi040.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-18T22:44:07.212 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~ 2024-03-18T22:44:07.212 INFO:tasks.workunit.client.0.smithi040.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T22:44:07.212 INFO:tasks.workunit.client.0.smithi040.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-18T22:44:07.212 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T22:44:07.212 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T22:44:07.212 INFO:tasks.workunit.client.0.smithi040.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T22:44:07.212 INFO:tasks.workunit.client.0.smithi040.stderr: | ^ 2024-03-18T22:44:07.212 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T22:44:07.212 INFO:tasks.workunit.client.0.smithi040.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T22:44:07.212 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~ 2024-03-18T22:44:07.212 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T22:44:07.212 INFO:tasks.workunit.client.0.smithi040.stderr: 36 | container_of(ptr, type, member) 2024-03-18T22:44:07.213 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~ 2024-03-18T22:44:07.213 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-18T22:44:07.213 INFO:tasks.workunit.client.0.smithi040.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-18T22:44:07.213 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~ 2024-03-18T22:44:07.213 INFO:tasks.workunit.client.0.smithi040.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T22:44:07.213 INFO:tasks.workunit.client.0.smithi040.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-18T22:44:07.213 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T22:44:07.213 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T22:44:07.213 INFO:tasks.workunit.client.0.smithi040.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T22:44:07.213 INFO:tasks.workunit.client.0.smithi040.stderr: | ^ 2024-03-18T22:44:07.213 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T22:44:07.213 INFO:tasks.workunit.client.0.smithi040.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T22:44:07.214 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~ 2024-03-18T22:44:07.214 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T22:44:07.214 INFO:tasks.workunit.client.0.smithi040.stderr: 36 | container_of(ptr, type, member) 2024-03-18T22:44:07.214 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~ 2024-03-18T22:44:07.214 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-18T22:44:07.214 INFO:tasks.workunit.client.0.smithi040.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-18T22:44:07.214 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~ 2024-03-18T22:44:07.214 INFO:tasks.workunit.client.0.smithi040.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T22:44:07.214 INFO:tasks.workunit.client.0.smithi040.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-18T22:44:07.214 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T22:44:07.214 INFO:tasks.workunit.client.0.smithi040.stderr:parser.c: In function ‘init_tg_stats’: 2024-03-18T22:44:07.214 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T22:44:07.215 INFO:tasks.workunit.client.0.smithi040.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T22:44:07.215 INFO:tasks.workunit.client.0.smithi040.stderr: | ^ 2024-03-18T22:44:07.215 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T22:44:07.215 INFO:tasks.workunit.client.0.smithi040.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T22:44:07.215 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~ 2024-03-18T22:44:07.215 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T22:44:07.215 INFO:tasks.workunit.client.0.smithi040.stderr: 36 | container_of(ptr, type, member) 2024-03-18T22:44:07.215 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~ 2024-03-18T22:44:07.215 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-18T22:44:07.215 INFO:tasks.workunit.client.0.smithi040.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-18T22:44:07.215 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~ 2024-03-18T22:44:07.215 INFO:tasks.workunit.client.0.smithi040.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T22:44:07.216 INFO:tasks.workunit.client.0.smithi040.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-18T22:44:07.216 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T22:44:07.216 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T22:44:07.216 INFO:tasks.workunit.client.0.smithi040.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T22:44:07.216 INFO:tasks.workunit.client.0.smithi040.stderr: | ^ 2024-03-18T22:44:07.216 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T22:44:07.216 INFO:tasks.workunit.client.0.smithi040.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T22:44:07.216 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~ 2024-03-18T22:44:07.216 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T22:44:07.216 INFO:tasks.workunit.client.0.smithi040.stderr: 36 | container_of(ptr, type, member) 2024-03-18T22:44:07.216 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~ 2024-03-18T22:44:07.216 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-18T22:44:07.216 INFO:tasks.workunit.client.0.smithi040.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-18T22:44:07.217 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~ 2024-03-18T22:44:07.217 INFO:tasks.workunit.client.0.smithi040.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T22:44:07.217 INFO:tasks.workunit.client.0.smithi040.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-18T22:44:07.217 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T22:44:07.217 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T22:44:07.217 INFO:tasks.workunit.client.0.smithi040.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T22:44:07.217 INFO:tasks.workunit.client.0.smithi040.stderr: | ^ 2024-03-18T22:44:07.217 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T22:44:07.217 INFO:tasks.workunit.client.0.smithi040.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T22:44:07.217 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~ 2024-03-18T22:44:07.217 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T22:44:07.217 INFO:tasks.workunit.client.0.smithi040.stderr: 36 | container_of(ptr, type, member) 2024-03-18T22:44:07.217 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~ 2024-03-18T22:44:07.218 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-18T22:44:07.218 INFO:tasks.workunit.client.0.smithi040.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-18T22:44:07.218 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~ 2024-03-18T22:44:07.218 INFO:tasks.workunit.client.0.smithi040.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T22:44:07.218 INFO:tasks.workunit.client.0.smithi040.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-18T22:44:07.218 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T22:44:07.218 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T22:44:07.218 INFO:tasks.workunit.client.0.smithi040.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T22:44:07.218 INFO:tasks.workunit.client.0.smithi040.stderr: | ^ 2024-03-18T22:44:07.218 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T22:44:07.218 INFO:tasks.workunit.client.0.smithi040.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T22:44:07.218 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~ 2024-03-18T22:44:07.219 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T22:44:07.219 INFO:tasks.workunit.client.0.smithi040.stderr: 36 | container_of(ptr, type, member) 2024-03-18T22:44:07.219 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~ 2024-03-18T22:44:07.219 INFO:tasks.workunit.client.0.smithi040.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-18T22:44:07.219 INFO:tasks.workunit.client.0.smithi040.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-18T22:44:07.219 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~ 2024-03-18T22:44:07.219 INFO:tasks.workunit.client.0.smithi040.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T22:44:07.219 INFO:tasks.workunit.client.0.smithi040.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-18T22:44:07.219 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T22:44:07.271 INFO:tasks.workunit.client.0.smithi040.stderr:parser.c: In function ‘get_optstr’: 2024-03-18T22:44:07.271 INFO:tasks.workunit.client.0.smithi040.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-03-18T22:44:07.271 INFO:tasks.workunit.client.0.smithi040.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-03-18T22:44:07.271 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T22:44:07.271 INFO:tasks.workunit.client.0.smithi040.stderr:parser.c:200:14: note: source object declared here 2024-03-18T22:44:07.271 INFO:tasks.workunit.client.0.smithi040.stderr: 200 | char temp[BUFSIZE]; 2024-03-18T22:44:07.271 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~ 2024-03-18T22:44:07.278 INFO:tasks.workunit.client.0.smithi040.stderr:parser.c: In function ‘size64_convert’: 2024-03-18T22:44:07.278 INFO:tasks.workunit.client.0.smithi040.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-03-18T22:44:07.278 INFO:tasks.workunit.client.0.smithi040.stderr: 119 | memcpy(unit, "\0", 3); 2024-03-18T22:44:07.278 INFO:tasks.workunit.client.0.smithi040.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-03-18T22:44:07.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:06 smithi113 ceph-mon[66775]: pgmap v394: 129 pgs: 129 active+clean; 7.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 713 KiB/s wr, 127 op/s 2024-03-18T22:44:07.592 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-03-18T22:44:07.598 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-03-18T22:44:07.678 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-03-18T22:44:07.685 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-03-18T22:44:07.796 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-03-18T22:44:07.808 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-03-18T22:44:07.855 INFO:tasks.workunit.client.0.smithi040.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-03-18T22:44:07.869 INFO:tasks.workunit.client.0.smithi040.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-03-18T22:44:07.975 INFO:tasks.workunit.client.0.smithi040.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-18T22:44:07.976 INFO:tasks.workunit.client.0.smithi040.stderr:+ cd .. 2024-03-18T22:44:07.977 INFO:tasks.workunit.client.0.smithi040.stderr:+ mkdir tmp 2024-03-18T22:44:07.980 INFO:tasks.workunit.client.0.smithi040.stderr:+ cd tmp 2024-03-18T22:44:07.981 INFO:tasks.workunit.client.0.smithi040.stderr:+ for f in $mydir/*.ffsb 2024-03-18T22:44:07.981 INFO:tasks.workunit.client.0.smithi040.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-03-18T22:44:07.981 INFO:tasks.workunit.client.0.smithi040.stdout:FFSB version 6.0-RC2 started 2024-03-18T22:44:07.982 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:44:07.985 INFO:tasks.workunit.client.0.smithi040.stdout:benchmark time = 300 2024-03-18T22:44:07.985 INFO:tasks.workunit.client.0.smithi040.stdout:ThreadGroup 0 2024-03-18T22:44:07.985 INFO:tasks.workunit.client.0.smithi040.stdout:================ 2024-03-18T22:44:07.985 INFO:tasks.workunit.client.0.smithi040.stdout: num_threads = 32 2024-03-18T22:44:07.985 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:44:07.985 INFO:tasks.workunit.client.0.smithi040.stdout: read_random = off 2024-03-18T22:44:07.985 INFO:tasks.workunit.client.0.smithi040.stdout: read_size = 0 (0B) 2024-03-18T22:44:07.985 INFO:tasks.workunit.client.0.smithi040.stdout: read_blocksize = 0 (0B) 2024-03-18T22:44:07.985 INFO:tasks.workunit.client.0.smithi040.stdout: read_skip = off 2024-03-18T22:44:07.985 INFO:tasks.workunit.client.0.smithi040.stdout: read_skipsize = 0 (0B) 2024-03-18T22:44:07.985 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:44:07.985 INFO:tasks.workunit.client.0.smithi040.stdout: write_random = on 2024-03-18T22:44:07.985 INFO:tasks.workunit.client.0.smithi040.stdout: write_size = 5242880 (5MB) 2024-03-18T22:44:07.986 INFO:tasks.workunit.client.0.smithi040.stdout: fsync_file = 0 2024-03-18T22:44:07.986 INFO:tasks.workunit.client.0.smithi040.stdout: write_blocksize = 4096 (4KB) 2024-03-18T22:44:07.986 INFO:tasks.workunit.client.0.smithi040.stdout: wait time = 0 2024-03-18T22:44:07.986 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:44:07.986 INFO:tasks.workunit.client.0.smithi040.stdout: op weights 2024-03-18T22:44:07.986 INFO:tasks.workunit.client.0.smithi040.stdout: read = 0 (0.00%) 2024-03-18T22:44:07.986 INFO:tasks.workunit.client.0.smithi040.stdout: readall = 0 (0.00%) 2024-03-18T22:44:07.986 INFO:tasks.workunit.client.0.smithi040.stdout: write = 1 (100.00%) 2024-03-18T22:44:07.986 INFO:tasks.workunit.client.0.smithi040.stdout: create = 0 (0.00%) 2024-03-18T22:44:07.986 INFO:tasks.workunit.client.0.smithi040.stdout: append = 0 (0.00%) 2024-03-18T22:44:07.986 INFO:tasks.workunit.client.0.smithi040.stdout: delete = 0 (0.00%) 2024-03-18T22:44:07.986 INFO:tasks.workunit.client.0.smithi040.stdout: metaop = 0 (0.00%) 2024-03-18T22:44:07.987 INFO:tasks.workunit.client.0.smithi040.stdout: createdir = 0 (0.00%) 2024-03-18T22:44:07.987 INFO:tasks.workunit.client.0.smithi040.stdout: stat = 0 (0.00%) 2024-03-18T22:44:07.987 INFO:tasks.workunit.client.0.smithi040.stdout: writeall = 0 (0.00%) 2024-03-18T22:44:07.987 INFO:tasks.workunit.client.0.smithi040.stdout: writeall_fsync = 0 (0.00%) 2024-03-18T22:44:07.987 INFO:tasks.workunit.client.0.smithi040.stdout: open_close = 0 (0.00%) 2024-03-18T22:44:07.987 INFO:tasks.workunit.client.0.smithi040.stdout: write_fsync = 0 (0.00%) 2024-03-18T22:44:07.987 INFO:tasks.workunit.client.0.smithi040.stdout: create_fsync = 0 (0.00%) 2024-03-18T22:44:07.987 INFO:tasks.workunit.client.0.smithi040.stdout: append_fsync = 0 (0.00%) 2024-03-18T22:44:07.987 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:44:07.987 INFO:tasks.workunit.client.0.smithi040.stdout:FileSystem . 2024-03-18T22:44:07.987 INFO:tasks.workunit.client.0.smithi040.stdout:========== 2024-03-18T22:44:07.987 INFO:tasks.workunit.client.0.smithi040.stdout: num_dirs = 0 2024-03-18T22:44:07.987 INFO:tasks.workunit.client.0.smithi040.stdout: starting files = 128 2024-03-18T22:44:07.988 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:44:07.988 INFO:tasks.workunit.client.0.smithi040.stdout: min file size = 104857600 (100MB) 2024-03-18T22:44:07.988 INFO:tasks.workunit.client.0.smithi040.stdout: max file size = 104857600 (100MB) 2024-03-18T22:44:07.988 INFO:tasks.workunit.client.0.smithi040.stdout: directio = off 2024-03-18T22:44:07.988 INFO:tasks.workunit.client.0.smithi040.stdout: alignedio = on 2024-03-18T22:44:07.988 INFO:tasks.workunit.client.0.smithi040.stdout: bufferedio = off 2024-03-18T22:44:07.988 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:44:07.988 INFO:tasks.workunit.client.0.smithi040.stdout: aging is off 2024-03-18T22:44:07.988 INFO:tasks.workunit.client.0.smithi040.stdout: current utilization = 0.04% 2024-03-18T22:44:07.988 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:44:07.988 INFO:tasks.workunit.client.0.smithi040.stdout:checking existing fs: . 2024-03-18T22:44:07.988 INFO:tasks.workunit.client.0.smithi040.stdout:recreating new fileset 2024-03-18T22:44:07.988 INFO:tasks.workunit.client.0.smithi040.stderr:opendir: No such file or directory 2024-03-18T22:44:09.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:08 smithi103 ceph-mon[66377]: pgmap v395: 129 pgs: 129 active+clean; 7.8 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 607 KiB/s wr, 138 op/s 2024-03-18T22:44:09.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:08 smithi040 ceph-mon[62643]: pgmap v395: 129 pgs: 129 active+clean; 7.8 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 607 KiB/s wr, 138 op/s 2024-03-18T22:44:09.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:08 smithi113 ceph-mon[66775]: pgmap v395: 129 pgs: 129 active+clean; 7.8 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 607 KiB/s wr, 138 op/s 2024-03-18T22:44:11.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:10 smithi103 ceph-mon[66377]: pgmap v396: 129 pgs: 129 active+clean; 8.0 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 607 KiB/s wr, 142 op/s 2024-03-18T22:44:11.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:10 smithi040 ceph-mon[62643]: pgmap v396: 129 pgs: 129 active+clean; 8.0 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 607 KiB/s wr, 142 op/s 2024-03-18T22:44:11.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:10 smithi113 ceph-mon[66775]: pgmap v396: 129 pgs: 129 active+clean; 8.0 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 607 KiB/s wr, 142 op/s 2024-03-18T22:44:13.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:12 smithi103 ceph-mon[66377]: pgmap v397: 129 pgs: 129 active+clean; 17 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-03-18T22:44:13.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:12 smithi040 ceph-mon[62643]: pgmap v397: 129 pgs: 129 active+clean; 17 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-03-18T22:44:13.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:12 smithi113 ceph-mon[66775]: pgmap v397: 129 pgs: 129 active+clean; 17 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-03-18T22:44:13.970 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:44:13.971 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:44:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.090+0000 7fc81d137640 1 -- 172.21.15.40:0/3231607354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8181022d0 msgr2=0x7fc818108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.090+0000 7fc81d137640 1 --2- 172.21.15.40:0/3231607354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8181022d0 0x7fc818108b60 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc810009ca0 tx=0x7fc81002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.090+0000 7fc81d137640 1 -- 172.21.15.40:0/3231607354 shutdown_connections 2024-03-18T22:44:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.090+0000 7fc81d137640 1 --2- 172.21.15.40:0/3231607354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8181022d0 0x7fc818108b60 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.090+0000 7fc81d137640 1 --2- 172.21.15.40:0/3231607354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc818101910 0x7fc818101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.090+0000 7fc81d137640 1 --2- 172.21.15.40:0/3231607354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc818100720 0x7fc818100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.090+0000 7fc81d137640 1 -- 172.21.15.40:0/3231607354 >> 172.21.15.40:0/3231607354 conn(0x7fc8180fbcb0 msgr2=0x7fc8180fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.090+0000 7fc81d137640 1 -- 172.21.15.40:0/3231607354 shutdown_connections 2024-03-18T22:44:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.090+0000 7fc81d137640 1 -- 172.21.15.40:0/3231607354 wait complete. 2024-03-18T22:44:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.090+0000 7fc81d137640 1 Processor -- start 2024-03-18T22:44:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc81d137640 1 -- start start 2024-03-18T22:44:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc81d137640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc818100720 0x7fc8180694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc81d137640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc818101910 0x7fc81819c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc81d137640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8181022d0 0x7fc81819ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc81d137640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc81810c250 con 0x7fc818100720 2024-03-18T22:44:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc81d137640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc81810c0d0 con 0x7fc8181022d0 2024-03-18T22:44:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc81d137640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc81810c3d0 con 0x7fc818101910 2024-03-18T22:44:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc816d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc818100720 0x7fc8180694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc81759e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8181022d0 0x7fc81819ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc81659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc818101910 0x7fc81819c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc816d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc818100720 0x7fc8180694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38796/0 (socket says 172.21.15.40:38796) 2024-03-18T22:44:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc816d9d640 1 -- 172.21.15.40:0/4150417292 learned_addr learned my addr 172.21.15.40:0/4150417292 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc81759e640 1 -- 172.21.15.40:0/4150417292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc818101910 msgr2=0x7fc81819c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc81759e640 1 --2- 172.21.15.40:0/4150417292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc818101910 0x7fc81819c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc81759e640 1 -- 172.21.15.40:0/4150417292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc818100720 msgr2=0x7fc8180694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc81759e640 1 --2- 172.21.15.40:0/4150417292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc818100720 0x7fc8180694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc81759e640 1 -- 172.21.15.40:0/4150417292 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81819f6a0 con 0x7fc8181022d0 2024-03-18T22:44:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc816d9d640 1 --2- 172.21.15.40:0/4150417292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc818100720 0x7fc8180694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.091+0000 7fc81659c640 1 --2- 172.21.15.40:0/4150417292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc818101910 0x7fc81819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.092+0000 7fc81759e640 1 --2- 172.21.15.40:0/4150417292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8181022d0 0x7fc81819ed90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc81002c630 tx=0x7fc8100328a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.092+0000 7fc7fffff640 1 -- 172.21.15.40:0/4150417292 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc810041020 con 0x7fc8181022d0 2024-03-18T22:44:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.092+0000 7fc81d137640 1 -- 172.21.15.40:0/4150417292 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc81819f8d0 con 0x7fc8181022d0 2024-03-18T22:44:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.092+0000 7fc7fffff640 1 -- 172.21.15.40:0/4150417292 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc810003e20 con 0x7fc8181022d0 2024-03-18T22:44:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.092+0000 7fc7fffff640 1 -- 172.21.15.40:0/4150417292 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc810039040 con 0x7fc8181022d0 2024-03-18T22:44:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.092+0000 7fc81d137640 1 -- 172.21.15.40:0/4150417292 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc81819fdc0 con 0x7fc8181022d0 2024-03-18T22:44:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.094+0000 7fc81fdf9740 1 Processor -- start 2024-03-18T22:44:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.094+0000 7fc7fffff640 1 -- 172.21.15.40:0/4150417292 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc810044020 con 0x7fc8181022d0 2024-03-18T22:44:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.094+0000 7fc81fdf9740 1 -- start start 2024-03-18T22:44:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81fdf9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e23730e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81fdf9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e23731440 0x555e23735f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81fdf9740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e237365b0 0x555e23738a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81fdf9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555e23695500 con 0x555e23731440 2024-03-18T22:44:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81fdf9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555e2363dbe0 con 0x555e23730a50 2024-03-18T22:44:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81fdf9740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x555e236915a0 con 0x555e237365b0 2024-03-18T22:44:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc816d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e23731440 0x555e23735f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e23730e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc816d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e23731440 0x555e23735f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38802/0 (socket says 172.21.15.40:38802) 2024-03-18T22:44:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e23730e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:33658/0 (socket says 172.21.15.40:33658) 2024-03-18T22:44:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc816d9d640 1 -- 172.21.15.40:0/2731790362 learned_addr learned my addr 172.21.15.40:0/2731790362 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81659c640 1 --2- 172.21.15.40:0/2731790362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e237365b0 0x555e23738a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc7fffff640 1 --2- 172.21.15.40:0/4150417292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8000a2800 0x7fc8000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81659c640 1 -- 172.21.15.40:0/2731790362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e237365b0 msgr2=0x555e23738a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81659c640 1 --2- 172.21.15.40:0/2731790362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e237365b0 0x555e23738a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81659c640 1 -- 172.21.15.40:0/2731790362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e23731440 msgr2=0x555e23735f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81659c640 1 --2- 172.21.15.40:0/2731790362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e23731440 0x555e23735f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81659c640 1 -- 172.21.15.40:0/2731790362 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e2365f000 con 0x555e23730a50 2024-03-18T22:44:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81759e640 1 --2- 172.21.15.40:0/4150417292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8000a2800 0x7fc8000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc7fffff640 1 -- 172.21.15.40:0/4150417292 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc8100f0fe0 con 0x7fc8181022d0 2024-03-18T22:44:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc81659c640 1 --2- 172.21.15.40:0/2731790362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e23730e70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc818069260 tx=0x7fc80c005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=962feff8f1aa2e93 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.095+0000 7fc7fd7fa640 1 -- 172.21.15.40:0/2731790362 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c004050 con 0x555e23730a50 2024-03-18T22:44:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.096+0000 7fc7fd7fa640 1 -- 172.21.15.40:0/2731790362 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc80c00fd70 con 0x555e23730a50 2024-03-18T22:44:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.096+0000 7fc7fd7fa640 1 -- 172.21.15.40:0/2731790362 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c003a70 con 0x555e23730a50 2024-03-18T22:44:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.096+0000 7fc81fdf9740 1 -- 172.21.15.40:0/2731790362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 msgr2=0x555e23730e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.096+0000 7fc81759e640 1 --2- 172.21.15.40:0/4150417292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8000a2800 0x7fc8000a4cc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc810032380 tx=0x7fc8100887f0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.096+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/2731790362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e23730e70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc818069260 tx=0x7fc80c005850 comp rx=0 tx=0).stop 2024-03-18T22:44:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.096+0000 7fc81fdf9740 1 -- 172.21.15.40:0/2731790362 shutdown_connections 2024-03-18T22:44:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.096+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/2731790362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e237365b0 0x555e23738a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.096+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/2731790362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e23731440 0x555e23735f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.096+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/2731790362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e23730e70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.096+0000 7fc81fdf9740 1 -- 172.21.15.40:0/2731790362 >> 172.21.15.40:0/2731790362 conn(0x555e2372bfc0 msgr2=0x555e2372c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.096+0000 7fc81fdf9740 1 -- 172.21.15.40:0/2731790362 shutdown_connections 2024-03-18T22:44:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.096+0000 7fc81fdf9740 1 -- 172.21.15.40:0/2731790362 wait complete. 2024-03-18T22:44:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81fdf9740 1 -- start start 2024-03-18T22:44:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81fdf9740 1 Processor -- start 2024-03-18T22:44:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81fdf9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e2373af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81fdf9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e23731440 0x555e2373b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81fdf9740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e237365b0 0x555e23742450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81fdf9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555e23739560 con 0x555e23731440 2024-03-18T22:44:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81fdf9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555e23651160 con 0x555e23730a50 2024-03-18T22:44:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81fdf9740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x555e237396e0 con 0x555e237365b0 2024-03-18T22:44:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81fdf9740 10 client.0 _ll_register_callbacks cb 0x555e2368da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc816d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e23731440 0x555e2373b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e2373af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e2373af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:33664/0 (socket says 172.21.15.40:33664) 2024-03-18T22:44:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81659c640 1 -- 172.21.15.40:0/819223114 learned_addr learned my addr 172.21.15.40:0/819223114 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81659c640 1 --2- 172.21.15.40:0/819223114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e237365b0 0x555e23742450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81659c640 1 -- 172.21.15.40:0/819223114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e237365b0 msgr2=0x555e23742450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81659c640 1 --2- 172.21.15.40:0/819223114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e237365b0 0x555e23742450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81659c640 1 -- 172.21.15.40:0/819223114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e23731440 msgr2=0x555e2373b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81659c640 1 --2- 172.21.15.40:0/819223114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e23731440 0x555e2373b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc81659c640 1 -- 172.21.15.40:0/819223114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e234737d0 con 0x555e23730a50 2024-03-18T22:44:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.097+0000 7fc816d9d640 1 --2- 172.21.15.40:0/819223114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e23731440 0x555e2373b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc81659c640 1 --2- 172.21.15.40:0/819223114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e2373af40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc80c0057c0 tx=0x7fc80c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50c047113bf39757 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc7faff5640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:44:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc7faff5640 1 -- 172.21.15.40:0/819223114 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c028040 con 0x555e23730a50 2024-03-18T22:44:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc7faff5640 1 -- 172.21.15.40:0/819223114 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc80c005460 con 0x555e23730a50 2024-03-18T22:44:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc81fdf9740 1 -- 172.21.15.40:0/819223114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc80c00fd70 con 0x555e23730a50 2024-03-18T22:44:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc7faff5640 1 -- 172.21.15.40:0/819223114 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc80c034050 con 0x555e23730a50 2024-03-18T22:44:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc81fdf9740 10 client.25697 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc81fdf9740 1 -- 172.21.15.40:0/819223114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555e233a93f0 con 0x555e23730a50 2024-03-18T22:44:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc7faff5640 1 -- 172.21.15.40:0/819223114 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc80c00fd70 con 0x555e23730a50 2024-03-18T22:44:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc7faff5640 1 -- 172.21.15.40:0/819223114 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fc80c0048f0 con 0x555e23730a50 2024-03-18T22:44:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc81fdf9740 10 client.25697 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc81fdf9740 10 client.25697 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc81fdf9740 20 client.25697 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/819223114 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x555e23746aa0 0x555e23748ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc81fdf9740 4 client.25697 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.098+0000 7fc81fdf9740 1 -- 172.21.15.40:0/819223114 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e236a2a40 con 0x555e23746aa0 2024-03-18T22:44:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.099+0000 7fc816d9d640 1 --2- 172.21.15.40:0/819223114 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x555e23746aa0 0x555e23748ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.099+0000 7fc816d9d640 1 --2- 172.21.15.40:0/819223114 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x555e23746aa0 0x555e23748ee0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7486056b43855358 server_cookie=cc48243177632cd7 in_seq=0 out_seq=0 2024-03-18T22:44:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.099+0000 7fc7faff5640 10 client.25697 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.100+0000 7fc7faff5640 1 -- 172.21.15.40:0/819223114 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555e236a2a40 con 0x555e23746aa0 2024-03-18T22:44:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.100+0000 7fc7faff5640 10 client.25697 handle_command_reply: tid=0 2024-03-18T22:44:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.100+0000 7fc81fdf9740 1 client.25697 shutdown 2024-03-18T22:44:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.100+0000 7fc81fdf9740 1 -- 172.21.15.40:0/819223114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 msgr2=0x555e2373af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.100+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/819223114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e2373af40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc80c0057c0 tx=0x7fc80c005f90 comp rx=0 tx=0).stop 2024-03-18T22:44:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.100+0000 7fc81fdf9740 1 -- 172.21.15.40:0/819223114 shutdown_connections 2024-03-18T22:44:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.100+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/819223114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e237365b0 0x555e23742450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.100+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/819223114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e23731440 0x555e2373b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.100+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/819223114 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x555e23746aa0 0x555e23748ee0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.100+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/819223114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e2373af40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.100+0000 7fc7faff5640 0 client.25697 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.100+0000 7fc81fdf9740 1 -- 172.21.15.40:0/819223114 >> 172.21.15.40:0/819223114 conn(0x555e2372bfc0 msgr2=0x555e2372b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.100+0000 7fc81fdf9740 1 -- 172.21.15.40:0/819223114 shutdown_connections 2024-03-18T22:44:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.100+0000 7fc81fdf9740 1 -- 172.21.15.40:0/819223114 wait complete. 2024-03-18T22:44:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.100+0000 7fc81fdf9740 20 client.25697 trim_cache size 0 max 16384 2024-03-18T22:44:14.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:13 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:44:14.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.118+0000 7fc81fdf9740 1 Processor -- start 2024-03-18T22:44:14.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.118+0000 7fc81fdf9740 1 -- start start 2024-03-18T22:44:14.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.118+0000 7fc81fdf9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e2373b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc81fdf9740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e23731440 0x555e237423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc81fdf9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e237365b0 0x555e2372a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc81fdf9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc8181aa510 con 0x555e237365b0 2024-03-18T22:44:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc81fdf9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc8181ac080 con 0x555e23730a50 2024-03-18T22:44:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc81fdf9740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc8181aabe0 con 0x555e23731440 2024-03-18T22:44:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc81659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e2373b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc816d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e23731440 0x555e237423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc81659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e237365b0 0x555e2372a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc816d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e23731440 0x555e237423e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:45932/0 (socket says 172.21.15.40:45932) 2024-03-18T22:44:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc816d9d640 1 -- 172.21.15.40:0/3449155920 learned_addr learned my addr 172.21.15.40:0/3449155920 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc81659c640 1 -- 172.21.15.40:0/3449155920 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e23731440 msgr2=0x555e237423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc81659c640 1 --2- 172.21.15.40:0/3449155920 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e23731440 0x555e237423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc81659c640 1 -- 172.21.15.40:0/3449155920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 msgr2=0x555e2373b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc81659c640 1 --2- 172.21.15.40:0/3449155920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e2373b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc81659c640 1 -- 172.21.15.40:0/3449155920 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e233690f0 con 0x555e237365b0 2024-03-18T22:44:14.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc816d9d640 1 --2- 172.21.15.40:0/3449155920 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e23731440 0x555e237423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:14.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc81659c640 1 --2- 172.21.15.40:0/3449155920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e237365b0 0x555e2372a950 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc80c013fd0 tx=0x7fc80c0138d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f83cd6b4e382806 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc7fcff9640 1 -- 172.21.15.40:0/3449155920 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c005460 con 0x555e237365b0 2024-03-18T22:44:14.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc7fcff9640 1 -- 172.21.15.40:0/3449155920 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc80c00fe80 con 0x555e237365b0 2024-03-18T22:44:14.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.119+0000 7fc7fcff9640 1 -- 172.21.15.40:0/3449155920 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c01f910 con 0x555e237365b0 2024-03-18T22:44:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.120+0000 7fc81fdf9740 1 -- 172.21.15.40:0/3449155920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e237365b0 msgr2=0x555e2372a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.120+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/3449155920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e237365b0 0x555e2372a950 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc80c013fd0 tx=0x7fc80c0138d0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.120+0000 7fc81fdf9740 1 -- 172.21.15.40:0/3449155920 shutdown_connections 2024-03-18T22:44:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.120+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/3449155920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e237365b0 0x555e2372a950 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.120+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/3449155920 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e23731440 0x555e237423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.120+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/3449155920 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23730a50 0x555e2373b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.120+0000 7fc81fdf9740 1 -- 172.21.15.40:0/3449155920 >> 172.21.15.40:0/3449155920 conn(0x555e2372bfc0 msgr2=0x555e2372c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.120+0000 7fc81fdf9740 1 -- 172.21.15.40:0/3449155920 shutdown_connections 2024-03-18T22:44:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.120+0000 7fc81fdf9740 1 -- 172.21.15.40:0/3449155920 wait complete. 2024-03-18T22:44:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc81fdf9740 1 -- start start 2024-03-18T22:44:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc81fdf9740 1 Processor -- start 2024-03-18T22:44:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc81fdf9740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e23730a50 0x555e2373f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc81fdf9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23731440 0x555e2373fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc81fdf9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e237365b0 0x555e23740d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc81fdf9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc8181ab980 con 0x555e237365b0 2024-03-18T22:44:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc81fdf9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc8181ab2b0 con 0x555e23731440 2024-03-18T22:44:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc81fdf9740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc8181a84f0 con 0x555e23730a50 2024-03-18T22:44:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc81fdf9740 10 client.0 _ll_register_callbacks cb 0x7fc808009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc81659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e237365b0 0x555e23740d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc816d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23731440 0x555e2373fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc81659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e237365b0 0x555e23740d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38820/0 (socket says 172.21.15.40:38820) 2024-03-18T22:44:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc81659c640 1 -- 172.21.15.40:0/2025654752 learned_addr learned my addr 172.21.15.40:0/2025654752 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc81659c640 1 --2- 172.21.15.40:0/2025654752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e23730a50 0x555e2373f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc816d9d640 1 -- 172.21.15.40:0/2025654752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e23730a50 msgr2=0x555e2373f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc816d9d640 1 --2- 172.21.15.40:0/2025654752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e23730a50 0x555e2373f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc816d9d640 1 -- 172.21.15.40:0/2025654752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e237365b0 msgr2=0x555e23740d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc816d9d640 1 --2- 172.21.15.40:0/2025654752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e237365b0 0x555e23740d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.121+0000 7fc816d9d640 1 -- 172.21.15.40:0/2025654752 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e23741300 con 0x555e23731440 2024-03-18T22:44:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.122+0000 7fc816d9d640 1 --2- 172.21.15.40:0/2025654752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23731440 0x555e2373fd00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc808012b10 tx=0x7fc80801f9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4f0694f5365ecdd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.122+0000 7fc7faff5640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:44:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.122+0000 7fc7faff5640 1 -- 172.21.15.40:0/2025654752 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc808022410 con 0x555e23731440 2024-03-18T22:44:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.122+0000 7fc7faff5640 1 -- 172.21.15.40:0/2025654752 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc8080225b0 con 0x555e23731440 2024-03-18T22:44:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.122+0000 7fc81fdf9740 1 -- 172.21.15.40:0/2025654752 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555e2369cd90 con 0x555e23731440 2024-03-18T22:44:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.122+0000 7fc7faff5640 1 -- 172.21.15.40:0/2025654752 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc808032050 con 0x555e23731440 2024-03-18T22:44:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.122+0000 7fc81fdf9740 10 client.25703 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.122+0000 7fc81fdf9740 1 -- 172.21.15.40:0/2025654752 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555e23742680 con 0x555e23731440 2024-03-18T22:44:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.122+0000 7fc7faff5640 1 -- 172.21.15.40:0/2025654752 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x555e2369cd90 con 0x555e23731440 2024-03-18T22:44:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.123+0000 7fc7faff5640 1 -- 172.21.15.40:0/2025654752 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fc808004ad0 con 0x555e23731440 2024-03-18T22:44:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.123+0000 7fc81fdf9740 10 client.25703 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.123+0000 7fc81fdf9740 10 client.25703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.123+0000 7fc81fdf9740 20 client.25703 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.123+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/2025654752 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x555e237ad430 0x555e237af850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.123+0000 7fc81fdf9740 4 client.25703 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:44:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.123+0000 7fc81fdf9740 1 -- 172.21.15.40:0/2025654752 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555e2361a540 con 0x555e237ad430 2024-03-18T22:44:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.123+0000 7fc81659c640 1 --2- 172.21.15.40:0/2025654752 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x555e237ad430 0x555e237af850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.123+0000 7fc81659c640 1 --2- 172.21.15.40:0/2025654752 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x555e237ad430 0x555e237af850 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc690f6739443f77 server_cookie=cecc0e553255082e in_seq=0 out_seq=0 2024-03-18T22:44:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.123+0000 7fc7faff5640 10 client.25703 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.123+0000 7fc7faff5640 1 -- 172.21.15.40:0/2025654752 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555e2361a540 con 0x555e237ad430 2024-03-18T22:44:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.123+0000 7fc7faff5640 10 client.25703 handle_command_reply: tid=0 2024-03-18T22:44:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.123+0000 7fc81fdf9740 1 client.25703 shutdown 2024-03-18T22:44:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.123+0000 7fc81fdf9740 1 -- 172.21.15.40:0/2025654752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23731440 msgr2=0x555e2373fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.123+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/2025654752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23731440 0x555e2373fd00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc808012b10 tx=0x7fc80801f9e0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81fdf9740 1 -- 172.21.15.40:0/2025654752 shutdown_connections 2024-03-18T22:44:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/2025654752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555e237365b0 0x555e23740d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/2025654752 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x555e237ad430 0x555e237af850 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/2025654752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e23731440 0x555e2373fd00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81fdf9740 1 --2- 172.21.15.40:0/2025654752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555e23730a50 0x555e2373f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81fdf9740 1 -- 172.21.15.40:0/2025654752 >> 172.21.15.40:0/2025654752 conn(0x555e2372bfc0 msgr2=0x555e2372f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc7faff5640 0 client.25703 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81fdf9740 1 -- 172.21.15.40:0/2025654752 shutdown_connections 2024-03-18T22:44:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81fdf9740 1 -- 172.21.15.40:0/2025654752 wait complete. 2024-03-18T22:44:14.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81fdf9740 20 client.25703 trim_cache size 0 max 16384 2024-03-18T22:44:14.132 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:44:14.132 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:44:14.132 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:44:14.132 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:44:14.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81d137640 1 -- 172.21.15.40:0/4150417292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8000a2800 msgr2=0x7fc8000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81d137640 1 --2- 172.21.15.40:0/4150417292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8000a2800 0x7fc8000a4cc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc810032380 tx=0x7fc8100887f0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81d137640 1 -- 172.21.15.40:0/4150417292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8181022d0 msgr2=0x7fc81819ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81d137640 1 --2- 172.21.15.40:0/4150417292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8181022d0 0x7fc81819ed90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc81002c630 tx=0x7fc8100328a0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81d137640 1 -- 172.21.15.40:0/4150417292 shutdown_connections 2024-03-18T22:44:14.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81d137640 1 --2- 172.21.15.40:0/4150417292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8000a2800 0x7fc8000a4cc0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81d137640 1 --2- 172.21.15.40:0/4150417292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc8181022d0 0x7fc81819ed90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81d137640 1 --2- 172.21.15.40:0/4150417292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc818101910 0x7fc81819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81d137640 1 --2- 172.21.15.40:0/4150417292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc818100720 0x7fc8180694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81d137640 1 -- 172.21.15.40:0/4150417292 >> 172.21.15.40:0/4150417292 conn(0x7fc8180fbcb0 msgr2=0x7fc818104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81d137640 1 -- 172.21.15.40:0/4150417292 shutdown_connections 2024-03-18T22:44:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.124+0000 7fc81d137640 1 -- 172.21.15.40:0/4150417292 wait complete. 2024-03-18T22:44:14.136 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:13 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:44:14.137 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:44:14.137 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:44:14.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.268+0000 7f40e447e640 1 -- 172.21.15.40:0/273633475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40dc101930 msgr2=0x7f40dc101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.268+0000 7f40e447e640 1 --2- 172.21.15.40:0/273633475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40dc101930 0x7f40dc101db0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f40cc009900 tx=0x7f40cc02bc30 comp rx=0 tx=0).stop 2024-03-18T22:44:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.268+0000 7f40e447e640 1 -- 172.21.15.40:0/273633475 shutdown_connections 2024-03-18T22:44:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.268+0000 7f40e447e640 1 --2- 172.21.15.40:0/273633475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40dc1022f0 0x7f40dc108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.268+0000 7f40e447e640 1 --2- 172.21.15.40:0/273633475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40dc101930 0x7f40dc101db0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.268+0000 7f40e447e640 1 --2- 172.21.15.40:0/273633475 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40dc100740 0x7f40dc100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.268+0000 7f40e447e640 1 -- 172.21.15.40:0/273633475 >> 172.21.15.40:0/273633475 conn(0x7f40dc0fbcb0 msgr2=0x7f40dc0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.268+0000 7f40e447e640 1 -- 172.21.15.40:0/273633475 shutdown_connections 2024-03-18T22:44:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.268+0000 7f40e447e640 1 -- 172.21.15.40:0/273633475 wait complete. 2024-03-18T22:44:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.268+0000 7f40e447e640 1 Processor -- start 2024-03-18T22:44:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.268+0000 7f40e447e640 1 -- start start 2024-03-18T22:44:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e447e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40dc100740 0x7f40dc197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e447e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40dc101930 0x7f40dc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e447e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40dc1022f0 0x7f40dc19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e447e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f40dc10c250 con 0x7f40dc1022f0 2024-03-18T22:44:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e447e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f40dc10c0d0 con 0x7f40dc101930 2024-03-18T22:44:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e447e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f40dc10c3d0 con 0x7f40dc100740 2024-03-18T22:44:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e2a1b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40dc1022f0 0x7f40dc19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e2a1b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40dc1022f0 0x7f40dc19f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38850/0 (socket says 172.21.15.40:38850) 2024-03-18T22:44:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e2a1b640 1 -- 172.21.15.40:0/2615318075 learned_addr learned my addr 172.21.15.40:0/2615318075 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e221a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40dc100740 0x7f40dc197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e1a19640 1 --2- 172.21.15.40:0/2615318075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40dc101930 0x7f40dc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e2a1b640 1 -- 172.21.15.40:0/2615318075 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40dc100740 msgr2=0x7f40dc197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e2a1b640 1 --2- 172.21.15.40:0/2615318075 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40dc100740 0x7f40dc197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e2a1b640 1 -- 172.21.15.40:0/2615318075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40dc101930 msgr2=0x7f40dc198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e2a1b640 1 --2- 172.21.15.40:0/2615318075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40dc101930 0x7f40dc198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e2a1b640 1 -- 172.21.15.40:0/2615318075 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40dc19f6f0 con 0x7f40dc1022f0 2024-03-18T22:44:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e221a640 1 --2- 172.21.15.40:0/2615318075 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40dc100740 0x7f40dc197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40e2a1b640 1 --2- 172.21.15.40:0/2615318075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40dc1022f0 0x7f40dc19f080 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f40d400eb10 tx=0x7f40d400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.269+0000 7f40d37fe640 1 -- 172.21.15.40:0/2615318075 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d4004b20 con 0x7f40dc1022f0 2024-03-18T22:44:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.270+0000 7f40d37fe640 1 -- 172.21.15.40:0/2615318075 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f40d4004cc0 con 0x7f40dc1022f0 2024-03-18T22:44:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.270+0000 7f40e447e640 1 -- 172.21.15.40:0/2615318075 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40dc102bb0 con 0x7f40dc1022f0 2024-03-18T22:44:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.270+0000 7f40d37fe640 1 -- 172.21.15.40:0/2615318075 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d4010690 con 0x7f40dc1022f0 2024-03-18T22:44:14.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.270+0000 7f40e447e640 1 -- 172.21.15.40:0/2615318075 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40dc1030d0 con 0x7f40dc1022f0 2024-03-18T22:44:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.271+0000 7f40e447e640 1 -- 172.21.15.40:0/2615318075 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40dc103390 con 0x7f40dc1022f0 2024-03-18T22:44:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.272+0000 7f40d37fe640 1 -- 172.21.15.40:0/2615318075 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f40d4005020 con 0x7f40dc1022f0 2024-03-18T22:44:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.272+0000 7f40d37fe640 1 --2- 172.21.15.40:0/2615318075 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f40c80a2880 0x7f40c80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.272+0000 7f40e221a640 1 --2- 172.21.15.40:0/2615318075 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f40c80a2880 0x7f40c80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.272+0000 7f40d37fe640 1 -- 172.21.15.40:0/2615318075 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f40d40cf360 con 0x7f40dc1022f0 2024-03-18T22:44:14.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.273+0000 7f40e221a640 1 --2- 172.21.15.40:0/2615318075 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f40c80a2880 0x7f40c80a4d40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f40dc101790 tx=0x7f40d8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.278+0000 7f40d37fe640 1 -- 172.21.15.40:0/2615318075 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f40d409c010 con 0x7f40dc1022f0 2024-03-18T22:44:14.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:13 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:44:14.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.542+0000 7f40e447e640 1 -- 172.21.15.40:0/2615318075 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40dc103af0 con 0x7f40dc1022f0 2024-03-18T22:44:14.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.543+0000 7f40d37fe640 1 -- 172.21.15.40:0/2615318075 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11948 (secure 0 0 0) 0x7f40d409b910 con 0x7f40dc1022f0 2024-03-18T22:44:14.544 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:44:14.545 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:44:12.723992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24509,24481,24473,24388,24401]},"id":1}]} 2024-03-18T22:44:14.545 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 29 2024-03-18T22:44:14.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.545+0000 7f40e447e640 1 -- 172.21.15.40:0/2615318075 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f40c80a2880 msgr2=0x7f40c80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.546+0000 7f40e447e640 1 --2- 172.21.15.40:0/2615318075 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f40c80a2880 0x7f40c80a4d40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f40dc101790 tx=0x7f40d8005f90 comp rx=0 tx=0).stop 2024-03-18T22:44:14.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.546+0000 7f40e447e640 1 -- 172.21.15.40:0/2615318075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40dc1022f0 msgr2=0x7f40dc19f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.546+0000 7f40e447e640 1 --2- 172.21.15.40:0/2615318075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40dc1022f0 0x7f40dc19f080 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f40d400eb10 tx=0x7f40d400c2d0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.546+0000 7f40e447e640 1 -- 172.21.15.40:0/2615318075 shutdown_connections 2024-03-18T22:44:14.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.546+0000 7f40e447e640 1 --2- 172.21.15.40:0/2615318075 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f40c80a2880 0x7f40c80a4d40 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.546+0000 7f40e447e640 1 --2- 172.21.15.40:0/2615318075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40dc1022f0 0x7f40dc19f080 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.546+0000 7f40e447e640 1 --2- 172.21.15.40:0/2615318075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40dc101930 0x7f40dc198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.546+0000 7f40e447e640 1 --2- 172.21.15.40:0/2615318075 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f40dc100740 0x7f40dc197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.546+0000 7f40e447e640 1 -- 172.21.15.40:0/2615318075 >> 172.21.15.40:0/2615318075 conn(0x7f40dc0fbcb0 msgr2=0x7f40dc068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.546+0000 7f40e447e640 1 -- 172.21.15.40:0/2615318075 shutdown_connections 2024-03-18T22:44:14.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.546+0000 7f40e447e640 1 -- 172.21.15.40:0/2615318075 wait complete. 2024-03-18T22:44:14.558 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:44:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.669+0000 7f7304753640 1 -- 172.21.15.40:0/528672501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72fc100720 msgr2=0x7f72fc100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.669+0000 7f7304753640 1 --2- 172.21.15.40:0/528672501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72fc100720 0x7f72fc100b40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f72f8009900 tx=0x7f72f802bc30 comp rx=0 tx=0).stop 2024-03-18T22:44:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.669+0000 7f7304753640 1 -- 172.21.15.40:0/528672501 shutdown_connections 2024-03-18T22:44:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.669+0000 7f7304753640 1 --2- 172.21.15.40:0/528672501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72fc1022d0 0x7f72fc108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.669+0000 7f7304753640 1 --2- 172.21.15.40:0/528672501 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72fc101910 0x7f72fc101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.669+0000 7f7304753640 1 --2- 172.21.15.40:0/528672501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72fc100720 0x7f72fc100b40 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.669+0000 7f7304753640 1 -- 172.21.15.40:0/528672501 >> 172.21.15.40:0/528672501 conn(0x7f72fc0fbc90 msgr2=0x7f72fc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.669+0000 7f7304753640 1 -- 172.21.15.40:0/528672501 shutdown_connections 2024-03-18T22:44:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.669+0000 7f7304753640 1 -- 172.21.15.40:0/528672501 wait complete. 2024-03-18T22:44:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.670+0000 7f7304753640 1 Processor -- start 2024-03-18T22:44:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.670+0000 7f7304753640 1 -- start start 2024-03-18T22:44:14.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.670+0000 7f7304753640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72fc100720 0x7f72fc197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.670+0000 7f7304753640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72fc101910 0x7f72fc197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.670+0000 7f7304753640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72fc1022d0 0x7f72fc19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.670+0000 7f7304753640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f72fc10c220 con 0x7f72fc100720 2024-03-18T22:44:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.670+0000 7f7304753640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72fc10c0a0 con 0x7f72fc101910 2024-03-18T22:44:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.670+0000 7f7304753640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f72fc10c3a0 con 0x7f72fc1022d0 2024-03-18T22:44:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.670+0000 7f73024ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72fc100720 0x7f72fc197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.670+0000 7f7302cf0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72fc1022d0 0x7f72fc19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.670+0000 7f73024ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72fc100720 0x7f72fc197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38864/0 (socket says 172.21.15.40:38864) 2024-03-18T22:44:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.670+0000 7f73024ef640 1 -- 172.21.15.40:0/4123441544 learned_addr learned my addr 172.21.15.40:0/4123441544 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.670+0000 7f7301cee640 1 --2- 172.21.15.40:0/4123441544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72fc101910 0x7f72fc197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.671+0000 7f73024ef640 1 -- 172.21.15.40:0/4123441544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72fc1022d0 msgr2=0x7f72fc19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.671+0000 7f73024ef640 1 --2- 172.21.15.40:0/4123441544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72fc1022d0 0x7f72fc19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.671+0000 7f73024ef640 1 -- 172.21.15.40:0/4123441544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72fc101910 msgr2=0x7f72fc197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.671+0000 7f73024ef640 1 --2- 172.21.15.40:0/4123441544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72fc101910 0x7f72fc197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.671+0000 7f73024ef640 1 -- 172.21.15.40:0/4123441544 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72fc19f630 con 0x7f72fc100720 2024-03-18T22:44:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.671+0000 7f7302cf0640 1 --2- 172.21.15.40:0/4123441544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72fc1022d0 0x7f72fc19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.671+0000 7f73024ef640 1 --2- 172.21.15.40:0/4123441544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72fc100720 0x7f72fc197a10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f72f8000c00 tx=0x7f72f8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.671+0000 7f72f37fe640 1 -- 172.21.15.40:0/4123441544 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f8043020 con 0x7f72fc100720 2024-03-18T22:44:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.671+0000 7f7304753640 1 -- 172.21.15.40:0/4123441544 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72fc19f8c0 con 0x7f72fc100720 2024-03-18T22:44:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.671+0000 7f72f37fe640 1 -- 172.21.15.40:0/4123441544 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72f8004aa0 con 0x7f72fc100720 2024-03-18T22:44:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.671+0000 7f72f37fe640 1 -- 172.21.15.40:0/4123441544 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f8003d00 con 0x7f72fc100720 2024-03-18T22:44:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.671+0000 7f7304753640 1 -- 172.21.15.40:0/4123441544 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72fc19fe10 con 0x7f72fc100720 2024-03-18T22:44:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.674+0000 7f730741b740 1 Processor -- start 2024-03-18T22:44:14.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.674+0000 7f730741b740 1 -- start start 2024-03-18T22:44:14.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f72f37fe640 1 -- 172.21.15.40:0/4123441544 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f72f8046020 con 0x7f72fc100720 2024-03-18T22:44:14.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f730741b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882ac700 0x561d882acb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f730741b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f730741b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 0x561d882b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f730741b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561d8820c4b0 con 0x561d882ad0f0 2024-03-18T22:44:14.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f730741b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561d881b4d70 con 0x561d882ac700 2024-03-18T22:44:14.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f730741b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x561d88208550 con 0x561d882b2260 2024-03-18T22:44:14.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f73024ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f7301cee640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 0x561d882b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f7302cf0640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882ac700 0x561d882acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f73024ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882b1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38878/0 (socket says 172.21.15.40:38878) 2024-03-18T22:44:14.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f73024ef640 1 -- 172.21.15.40:0/4094600189 learned_addr learned my addr 172.21.15.40:0/4094600189 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f7301cee640 1 -- 172.21.15.40:0/4094600189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882ac700 msgr2=0x561d882acb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f7301cee640 1 --2- 172.21.15.40:0/4094600189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882ac700 0x561d882acb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f7301cee640 1 -- 172.21.15.40:0/4094600189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 msgr2=0x561d882b1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f7301cee640 1 --2- 172.21.15.40:0/4094600189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882b1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f7301cee640 1 -- 172.21.15.40:0/4094600189 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d881d7210 con 0x561d882b2260 2024-03-18T22:44:14.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f72f37fe640 1 --2- 172.21.15.40:0/4123441544 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72e80a2800 0x7f72e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.675+0000 7f73024ef640 1 --2- 172.21.15.40:0/4094600189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882b1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:14.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.676+0000 7f7302cf0640 1 --2- 172.21.15.40:0/4123441544 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72e80a2800 0x7f72e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.676+0000 7f72f37fe640 1 -- 172.21.15.40:0/4123441544 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f72f80f13e0 con 0x7f72fc100720 2024-03-18T22:44:14.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.676+0000 7f7301cee640 1 --2- 172.21.15.40:0/4094600189 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 0x561d882b46f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f72ec00bbc0 tx=0x7f72ec0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28a0607dcfbcc9bf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.676+0000 7f72f0ff9640 1 -- 172.21.15.40:0/4094600189 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec004030 con 0x561d882b2260 2024-03-18T22:44:14.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.676+0000 7f72f0ff9640 1 -- 172.21.15.40:0/4094600189 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72ec0035f0 con 0x561d882b2260 2024-03-18T22:44:14.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.676+0000 7f72f0ff9640 1 -- 172.21.15.40:0/4094600189 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec0116a0 con 0x561d882b2260 2024-03-18T22:44:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f730741b740 1 -- 172.21.15.40:0/4094600189 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 msgr2=0x561d882b46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f7302cf0640 1 --2- 172.21.15.40:0/4123441544 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72e80a2800 0x7f72e80a4cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x561d882ad950 tx=0x7f72f4007520 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f730741b740 1 --2- 172.21.15.40:0/4094600189 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 0x561d882b46f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f72ec00bbc0 tx=0x7f72ec0107a0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f730741b740 1 -- 172.21.15.40:0/4094600189 shutdown_connections 2024-03-18T22:44:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f730741b740 1 --2- 172.21.15.40:0/4094600189 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 0x561d882b46f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f730741b740 1 --2- 172.21.15.40:0/4094600189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882b1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f730741b740 1 --2- 172.21.15.40:0/4094600189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882ac700 0x561d882acb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f730741b740 1 -- 172.21.15.40:0/4094600189 >> 172.21.15.40:0/4094600189 conn(0x561d882a3860 msgr2=0x561d882a3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f730741b740 1 -- 172.21.15.40:0/4094600189 shutdown_connections 2024-03-18T22:44:14.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f730741b740 1 -- 172.21.15.40:0/4094600189 wait complete. 2024-03-18T22:44:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f730741b740 1 -- start start 2024-03-18T22:44:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f730741b740 1 Processor -- start 2024-03-18T22:44:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f730741b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ac700 0x561d882b6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f730741b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882ad0f0 0x561d882b7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f730741b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 0x561d882be180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.677+0000 7f730741b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561d882b5210 con 0x561d882ac700 2024-03-18T22:44:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f730741b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561d881c8310 con 0x561d882ad0f0 2024-03-18T22:44:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f730741b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x561d882b5390 con 0x561d882b2260 2024-03-18T22:44:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f73024ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ac700 0x561d882b6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f73024ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ac700 0x561d882b6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38880/0 (socket says 172.21.15.40:38880) 2024-03-18T22:44:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f730741b740 10 client.0 _ll_register_callbacks cb 0x561d881bc2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f73024ef640 1 -- 172.21.15.40:0/4034546171 learned_addr learned my addr 172.21.15.40:0/4034546171 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f7301cee640 1 --2- 172.21.15.40:0/4034546171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882ad0f0 0x561d882b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f7302cf0640 1 --2- 172.21.15.40:0/4034546171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 0x561d882be180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f73024ef640 1 -- 172.21.15.40:0/4034546171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 msgr2=0x561d882be180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f73024ef640 1 --2- 172.21.15.40:0/4034546171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 0x561d882be180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f73024ef640 1 -- 172.21.15.40:0/4034546171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882ad0f0 msgr2=0x561d882b7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f73024ef640 1 --2- 172.21.15.40:0/4034546171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882ad0f0 0x561d882b7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f73024ef640 1 -- 172.21.15.40:0/4034546171 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d87f203f0 con 0x561d882ac700 2024-03-18T22:44:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f73024ef640 1 --2- 172.21.15.40:0/4034546171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ac700 0x561d882b6c00 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f72f80ba300 tx=0x7f72f8005b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a8184f8707cfcc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f72e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:44:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f72e5ffb640 1 -- 172.21.15.40:0/4034546171 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f80b9bd0 con 0x561d882ac700 2024-03-18T22:44:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f72e5ffb640 1 -- 172.21.15.40:0/4034546171 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f72f80eeb10 con 0x561d882ac700 2024-03-18T22:44:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f730741b740 1 -- 172.21.15.40:0/4034546171 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72ec0035f0 con 0x561d882ac700 2024-03-18T22:44:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f72e5ffb640 1 -- 172.21.15.40:0/4034546171 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f72f80d6050 con 0x561d882ac700 2024-03-18T22:44:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f730741b740 10 client.15840 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.678+0000 7f730741b740 1 -- 172.21.15.40:0/4034546171 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561d87fea7d0 con 0x561d882ac700 2024-03-18T22:44:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.679+0000 7f72e5ffb640 1 -- 172.21.15.40:0/4034546171 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72ec0035f0 con 0x561d882ac700 2024-03-18T22:44:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.679+0000 7f72e5ffb640 1 -- 172.21.15.40:0/4034546171 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f72f80c5c40 con 0x561d882ac700 2024-03-18T22:44:14.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.679+0000 7f730741b740 10 client.15840 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:14.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.679+0000 7f730741b740 10 client.15840 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:44:14.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.679+0000 7f730741b740 20 client.15840 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:14.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.679+0000 7f730741b740 1 --2- 172.21.15.40:0/4034546171 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x561d882c2a90 0x561d882c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.679+0000 7f730741b740 4 client.15840 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:14.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.679+0000 7f730741b740 1 -- 172.21.15.40:0/4034546171 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d8826a3e0 con 0x561d882c2a90 2024-03-18T22:44:14.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.679+0000 7f7301cee640 1 --2- 172.21.15.40:0/4034546171 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x561d882c2a90 0x561d882c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.679+0000 7f7301cee640 1 --2- 172.21.15.40:0/4034546171 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x561d882c2a90 0x561d882c4f30 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f1535ddd6b254434 server_cookie=4870cfb0aae4ab6c in_seq=0 out_seq=0 2024-03-18T22:44:14.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.679+0000 7f72e5ffb640 10 client.15840 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:44:14.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.680+0000 7f72e5ffb640 1 -- 172.21.15.40:0/4034546171 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561d8826a3e0 con 0x561d882c2a90 2024-03-18T22:44:14.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.680+0000 7f72e5ffb640 10 client.15840 handle_command_reply: tid=0 2024-03-18T22:44:14.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.680+0000 7f730741b740 1 client.15840 shutdown 2024-03-18T22:44:14.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.680+0000 7f730741b740 1 -- 172.21.15.40:0/4034546171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ac700 msgr2=0x561d882b6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.680+0000 7f730741b740 1 --2- 172.21.15.40:0/4034546171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ac700 0x561d882b6c00 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f72f80ba300 tx=0x7f72f8005b70 comp rx=0 tx=0).stop 2024-03-18T22:44:14.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.680+0000 7f730741b740 1 -- 172.21.15.40:0/4034546171 shutdown_connections 2024-03-18T22:44:14.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.680+0000 7f730741b740 1 --2- 172.21.15.40:0/4034546171 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 0x561d882be180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.680+0000 7f730741b740 1 --2- 172.21.15.40:0/4034546171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882ad0f0 0x561d882b7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.680+0000 7f730741b740 1 --2- 172.21.15.40:0/4034546171 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x561d882c2a90 0x561d882c4f30 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.680+0000 7f730741b740 1 --2- 172.21.15.40:0/4034546171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ac700 0x561d882b6c00 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.680+0000 7f72e5ffb640 0 client.15840 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:44:14.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.680+0000 7f730741b740 1 -- 172.21.15.40:0/4034546171 >> 172.21.15.40:0/4034546171 conn(0x561d882a3860 msgr2=0x561d882a2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.680+0000 7f730741b740 1 -- 172.21.15.40:0/4034546171 shutdown_connections 2024-03-18T22:44:14.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.680+0000 7f730741b740 1 -- 172.21.15.40:0/4034546171 wait complete. 2024-03-18T22:44:14.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.681+0000 7f730741b740 20 client.15840 trim_cache size 0 max 16384 2024-03-18T22:44:14.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.696+0000 7f730741b740 1 Processor -- start 2024-03-18T22:44:14.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.696+0000 7f730741b740 1 -- start start 2024-03-18T22:44:14.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.696+0000 7f730741b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882ac700 0x561d882b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.696+0000 7f730741b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882be110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.696+0000 7f730741b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882b2260 0x561d882b6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.696+0000 7f730741b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f72fc1a82d0 con 0x561d882ad0f0 2024-03-18T22:44:14.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.696+0000 7f730741b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72fc1a9e40 con 0x561d882b2260 2024-03-18T22:44:14.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.696+0000 7f730741b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f72fc1a89a0 con 0x561d882ac700 2024-03-18T22:44:14.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.696+0000 7f7301cee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882be110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.697+0000 7f73024ef640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882ac700 0x561d882b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.697+0000 7f7301cee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882be110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38896/0 (socket says 172.21.15.40:38896) 2024-03-18T22:44:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.697+0000 7f7302cf0640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882b2260 0x561d882b6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.697+0000 7f7301cee640 1 -- 172.21.15.40:0/2490985195 learned_addr learned my addr 172.21.15.40:0/2490985195 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.697+0000 7f7301cee640 1 -- 172.21.15.40:0/2490985195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882ac700 msgr2=0x561d882b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.697+0000 7f7301cee640 1 --2- 172.21.15.40:0/2490985195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882ac700 0x561d882b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.697+0000 7f7301cee640 1 -- 172.21.15.40:0/2490985195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882b2260 msgr2=0x561d882b6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.697+0000 7f7301cee640 1 --2- 172.21.15.40:0/2490985195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882b2260 0x561d882b6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.697+0000 7f7301cee640 1 -- 172.21.15.40:0/2490985195 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d882c55b0 con 0x561d882ad0f0 2024-03-18T22:44:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.697+0000 7f7302cf0640 1 --2- 172.21.15.40:0/2490985195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882b2260 0x561d882b6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.697+0000 7f7301cee640 1 --2- 172.21.15.40:0/2490985195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882be110 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f72ec000be0 tx=0x7f72ec005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c33dd03f1171c7f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.697+0000 7f72e7fff640 1 -- 172.21.15.40:0/2490985195 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec02e070 con 0x561d882ad0f0 2024-03-18T22:44:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.697+0000 7f72e7fff640 1 -- 172.21.15.40:0/2490985195 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72ec0290d0 con 0x561d882ad0f0 2024-03-18T22:44:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.697+0000 7f72e7fff640 1 -- 172.21.15.40:0/2490985195 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec003850 con 0x561d882ad0f0 2024-03-18T22:44:14.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.698+0000 7f730741b740 1 -- 172.21.15.40:0/2490985195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 msgr2=0x561d882be110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.698+0000 7f730741b740 1 --2- 172.21.15.40:0/2490985195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882be110 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f72ec000be0 tx=0x7f72ec005900 comp rx=0 tx=0).stop 2024-03-18T22:44:14.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.698+0000 7f730741b740 1 -- 172.21.15.40:0/2490985195 shutdown_connections 2024-03-18T22:44:14.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.698+0000 7f730741b740 1 --2- 172.21.15.40:0/2490985195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882b2260 0x561d882b6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.698+0000 7f730741b740 1 --2- 172.21.15.40:0/2490985195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882be110 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.698+0000 7f730741b740 1 --2- 172.21.15.40:0/2490985195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882ac700 0x561d882b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.698+0000 7f730741b740 1 -- 172.21.15.40:0/2490985195 >> 172.21.15.40:0/2490985195 conn(0x561d882a3860 msgr2=0x561d882a3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.698+0000 7f730741b740 1 -- 172.21.15.40:0/2490985195 shutdown_connections 2024-03-18T22:44:14.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.698+0000 7f730741b740 1 -- 172.21.15.40:0/2490985195 wait complete. 2024-03-18T22:44:14.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.698+0000 7f730741b740 1 -- start start 2024-03-18T22:44:14.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.698+0000 7f730741b740 1 Processor -- start 2024-03-18T22:44:14.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f730741b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882ac700 0x561d882bb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f730741b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882bbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f730741b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 0x561d882bcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f730741b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f72fc1a9740 con 0x561d882ad0f0 2024-03-18T22:44:14.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f730741b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72fc1a9070 con 0x561d882ac700 2024-03-18T22:44:14.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f730741b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f72fc1a62b0 con 0x561d882b2260 2024-03-18T22:44:14.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f730741b740 10 client.0 _ll_register_callbacks cb 0x7f72f80057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:14.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f7301cee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882bbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f7301cee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882bbb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38910/0 (socket says 172.21.15.40:38910) 2024-03-18T22:44:14.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f7301cee640 1 -- 172.21.15.40:0/2732797842 learned_addr learned my addr 172.21.15.40:0/2732797842 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f7302cf0640 1 --2- 172.21.15.40:0/2732797842 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 0x561d882bcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f73024ef640 1 --2- 172.21.15.40:0/2732797842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882ac700 0x561d882bb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f7301cee640 1 -- 172.21.15.40:0/2732797842 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 msgr2=0x561d882bcb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f7301cee640 1 --2- 172.21.15.40:0/2732797842 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 0x561d882bcb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f7301cee640 1 -- 172.21.15.40:0/2732797842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882ac700 msgr2=0x561d882bb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f7301cee640 1 --2- 172.21.15.40:0/2732797842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882ac700 0x561d882bb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f7301cee640 1 -- 172.21.15.40:0/2732797842 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d88279880 con 0x561d882ad0f0 2024-03-18T22:44:14.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f7301cee640 1 --2- 172.21.15.40:0/2732797842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882bbb30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f72ec02c040 tx=0x7f72ec010e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7fc9744506f45c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f72e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:44:14.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f72e5ffb640 1 -- 172.21.15.40:0/2732797842 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec036060 con 0x561d882ad0f0 2024-03-18T22:44:14.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f72e5ffb640 1 -- 172.21.15.40:0/2732797842 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f72ec0050b0 con 0x561d882ad0f0 2024-03-18T22:44:14.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.699+0000 7f730741b740 1 -- 172.21.15.40:0/2732797842 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72ec0290d0 con 0x561d882ad0f0 2024-03-18T22:44:14.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.700+0000 7f72e5ffb640 1 -- 172.21.15.40:0/2732797842 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f72ec03d050 con 0x561d882ad0f0 2024-03-18T22:44:14.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.700+0000 7f730741b740 10 client.15846 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:14.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.700+0000 7f730741b740 1 -- 172.21.15.40:0/2732797842 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561d882be260 con 0x561d882ad0f0 2024-03-18T22:44:14.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.700+0000 7f72e5ffb640 1 -- 172.21.15.40:0/2732797842 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72ec0290d0 con 0x561d882ad0f0 2024-03-18T22:44:14.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.700+0000 7f72e5ffb640 1 -- 172.21.15.40:0/2732797842 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f72ec027470 con 0x561d882ad0f0 2024-03-18T22:44:14.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.700+0000 7f730741b740 10 client.15846 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:14.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.700+0000 7f730741b740 10 client.15846 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:44:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.700+0000 7f730741b740 20 client.15846 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.700+0000 7f730741b740 1 --2- 172.21.15.40:0/2732797842 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x561d88329250 0x561d8832b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.700+0000 7f730741b740 4 client.15846 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:44:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.700+0000 7f730741b740 1 -- 172.21.15.40:0/2732797842 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561d882199f0 con 0x561d88329250 2024-03-18T22:44:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.700+0000 7f73024ef640 1 --2- 172.21.15.40:0/2732797842 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x561d88329250 0x561d8832b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f73024ef640 1 --2- 172.21.15.40:0/2732797842 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x561d88329250 0x561d8832b670 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6d63e752be41cdcc server_cookie=52820b2d6550dee2 in_seq=0 out_seq=0 2024-03-18T22:44:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f72e5ffb640 10 client.15846 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:44:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f72e5ffb640 1 -- 172.21.15.40:0/2732797842 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561d882199f0 con 0x561d88329250 2024-03-18T22:44:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f72e5ffb640 10 client.15846 handle_command_reply: tid=0 2024-03-18T22:44:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f730741b740 1 client.15846 shutdown 2024-03-18T22:44:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f730741b740 1 -- 172.21.15.40:0/2732797842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 msgr2=0x561d882bbb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f730741b740 1 --2- 172.21.15.40:0/2732797842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882bbb30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f72ec02c040 tx=0x7f72ec010e40 comp rx=0 tx=0).stop 2024-03-18T22:44:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f730741b740 1 -- 172.21.15.40:0/2732797842 shutdown_connections 2024-03-18T22:44:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f730741b740 1 --2- 172.21.15.40:0/2732797842 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561d882b2260 0x561d882bcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f730741b740 1 --2- 172.21.15.40:0/2732797842 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x561d88329250 0x561d8832b670 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f730741b740 1 --2- 172.21.15.40:0/2732797842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561d882ad0f0 0x561d882bbb30 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f72e5ffb640 0 client.15846 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:44:14.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f730741b740 1 --2- 172.21.15.40:0/2732797842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561d882ac700 0x561d882bb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f730741b740 1 -- 172.21.15.40:0/2732797842 >> 172.21.15.40:0/2732797842 conn(0x561d882a3860 msgr2=0x561d882b67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f730741b740 1 -- 172.21.15.40:0/2732797842 shutdown_connections 2024-03-18T22:44:14.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f730741b740 1 -- 172.21.15.40:0/2732797842 wait complete. 2024-03-18T22:44:14.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.701+0000 7f730741b740 20 client.15846 trim_cache size 0 max 16384 2024-03-18T22:44:14.709 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:44:14.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.702+0000 7f7304753640 1 -- 172.21.15.40:0/4123441544 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72e80a2800 msgr2=0x7f72e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.702+0000 7f7304753640 1 --2- 172.21.15.40:0/4123441544 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72e80a2800 0x7f72e80a4cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x561d882ad950 tx=0x7f72f4007520 comp rx=0 tx=0).stop 2024-03-18T22:44:14.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.702+0000 7f7304753640 1 -- 172.21.15.40:0/4123441544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72fc100720 msgr2=0x7f72fc197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.702+0000 7f7304753640 1 --2- 172.21.15.40:0/4123441544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72fc100720 0x7f72fc197a10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f72f8000c00 tx=0x7f72f8004270 comp rx=0 tx=0).stop 2024-03-18T22:44:14.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.702+0000 7f7304753640 1 -- 172.21.15.40:0/4123441544 shutdown_connections 2024-03-18T22:44:14.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.702+0000 7f7304753640 1 --2- 172.21.15.40:0/4123441544 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72e80a2800 0x7f72e80a4cc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.702+0000 7f7304753640 1 --2- 172.21.15.40:0/4123441544 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72fc1022d0 0x7f72fc19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.702+0000 7f7304753640 1 --2- 172.21.15.40:0/4123441544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72fc101910 0x7f72fc197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.702+0000 7f7304753640 1 --2- 172.21.15.40:0/4123441544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72fc100720 0x7f72fc197a10 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.702+0000 7f7304753640 1 -- 172.21.15.40:0/4123441544 >> 172.21.15.40:0/4123441544 conn(0x7f72fc0fbc90 msgr2=0x7f72fc0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.702+0000 7f7304753640 1 -- 172.21.15.40:0/4123441544 shutdown_connections 2024-03-18T22:44:14.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.702+0000 7f7304753640 1 -- 172.21.15.40:0/4123441544 wait complete. 2024-03-18T22:44:14.714 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:44:14.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.833+0000 7fade22de640 1 -- 172.21.15.40:0/3826585716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc1022d0 msgr2=0x7faddc108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.833+0000 7fade22de640 1 --2- 172.21.15.40:0/3826585716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc1022d0 0x7faddc108b60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009ca0 tx=0x7fadd402c100 comp rx=0 tx=0).stop 2024-03-18T22:44:14.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.833+0000 7fade22de640 1 -- 172.21.15.40:0/3826585716 shutdown_connections 2024-03-18T22:44:14.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.833+0000 7fade22de640 1 --2- 172.21.15.40:0/3826585716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc1022d0 0x7faddc108b60 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.833+0000 7fade22de640 1 --2- 172.21.15.40:0/3826585716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faddc101910 0x7faddc101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.833+0000 7fade22de640 1 --2- 172.21.15.40:0/3826585716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc100720 0x7faddc100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.833+0000 7fade22de640 1 -- 172.21.15.40:0/3826585716 >> 172.21.15.40:0/3826585716 conn(0x7faddc0fbc90 msgr2=0x7faddc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.833+0000 7fade22de640 1 -- 172.21.15.40:0/3826585716 shutdown_connections 2024-03-18T22:44:14.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.833+0000 7fade22de640 1 -- 172.21.15.40:0/3826585716 wait complete. 2024-03-18T22:44:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.834+0000 7fade22de640 1 Processor -- start 2024-03-18T22:44:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.834+0000 7fade22de640 1 -- start start 2024-03-18T22:44:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7fade22de640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc100720 0x7faddc199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7fade22de640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faddc101910 0x7faddc19a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7fade22de640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc1022d0 0x7faddc1a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7fade22de640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faddc10c230 con 0x7faddc100720 2024-03-18T22:44:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7fade22de640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faddc10c0b0 con 0x7faddc1022d0 2024-03-18T22:44:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7fade22de640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7faddc10c3b0 con 0x7faddc101910 2024-03-18T22:44:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7faddbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc100720 0x7faddc199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7faddbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc100720 0x7faddc199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38922/0 (socket says 172.21.15.40:38922) 2024-03-18T22:44:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7fade087b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc1022d0 0x7faddc1a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7faddbfff640 1 -- 172.21.15.40:0/1241838145 learned_addr learned my addr 172.21.15.40:0/1241838145 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7faddb7fe640 1 --2- 172.21.15.40:0/1241838145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faddc101910 0x7faddc19a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7fade087b640 1 -- 172.21.15.40:0/1241838145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faddc101910 msgr2=0x7faddc19a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7fade087b640 1 --2- 172.21.15.40:0/1241838145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faddc101910 0x7faddc19a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7fade087b640 1 -- 172.21.15.40:0/1241838145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc100720 msgr2=0x7faddc199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7fade087b640 1 --2- 172.21.15.40:0/1241838145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc100720 0x7faddc199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7fade087b640 1 -- 172.21.15.40:0/1241838145 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faddc1a1790 con 0x7faddc1022d0 2024-03-18T22:44:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.835+0000 7faddb7fe640 1 --2- 172.21.15.40:0/1241838145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faddc101910 0x7faddc19a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:44:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.836+0000 7fade087b640 1 --2- 172.21.15.40:0/1241838145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc1022d0 0x7faddc1a1120 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fadd402c640 tx=0x7fadd4035d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.836+0000 7fadd97fa640 1 -- 172.21.15.40:0/1241838145 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd4039040 con 0x7faddc1022d0 2024-03-18T22:44:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.836+0000 7fade22de640 1 -- 172.21.15.40:0/1241838145 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faddc1a1a20 con 0x7faddc1022d0 2024-03-18T22:44:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.836+0000 7fadd97fa640 1 -- 172.21.15.40:0/1241838145 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadd4003970 con 0x7faddc1022d0 2024-03-18T22:44:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.836+0000 7fadd97fa640 1 -- 172.21.15.40:0/1241838145 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd4004410 con 0x7faddc1022d0 2024-03-18T22:44:14.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.836+0000 7fade22de640 1 -- 172.21.15.40:0/1241838145 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faddc1a1f60 con 0x7faddc1022d0 2024-03-18T22:44:14.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.839+0000 7fadd97fa640 1 -- 172.21.15.40:0/1241838145 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fadd4046030 con 0x7faddc1022d0 2024-03-18T22:44:14.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.839+0000 7fade4f9b740 1 Processor -- start 2024-03-18T22:44:14.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.839+0000 7fade4f9b740 1 -- start start 2024-03-18T22:44:14.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7fade4f9b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19bae52f0 0x55a19bae5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7fade4f9b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baea7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7fade4f9b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19baeae50 0x55a19baed2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7fadd97fa640 1 --2- 172.21.15.40:0/1241838145 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadc40a2900 0x7fadc40a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7fade4f9b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a19ba494b0 con 0x55a19bae5ce0 2024-03-18T22:44:14.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7fade4f9b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a19b9f1d70 con 0x55a19bae52f0 2024-03-18T22:44:14.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7fade4f9b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55a19ba45550 con 0x55a19baeae50 2024-03-18T22:44:14.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7faddbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baea7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7faddb7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19bae52f0 0x55a19bae5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7faddbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baea7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38938/0 (socket says 172.21.15.40:38938) 2024-03-18T22:44:14.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7faddbfff640 1 -- 172.21.15.40:0/2267783359 learned_addr learned my addr 172.21.15.40:0/2267783359 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7faddb7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19bae52f0 0x55a19bae5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:33778/0 (socket says 172.21.15.40:33778) 2024-03-18T22:44:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7fade087b640 1 --2- 172.21.15.40:0/1241838145 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadc40a2900 0x7fadc40a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7fadd97fa640 1 -- 172.21.15.40:0/1241838145 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fadd40f0a10 con 0x7faddc1022d0 2024-03-18T22:44:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7faddb7fe640 1 --2- 172.21.15.40:0/2267783359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19baeae50 0x55a19baed2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7faddb7fe640 1 -- 172.21.15.40:0/2267783359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19baeae50 msgr2=0x55a19baed2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7faddb7fe640 1 --2- 172.21.15.40:0/2267783359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19baeae50 0x55a19baed2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7faddb7fe640 1 -- 172.21.15.40:0/2267783359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 msgr2=0x55a19baea7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7faddb7fe640 1 --2- 172.21.15.40:0/2267783359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baea7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7faddb7fe640 1 -- 172.21.15.40:0/2267783359 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a19ba14210 con 0x55a19bae52f0 2024-03-18T22:44:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7faddbfff640 1 --2- 172.21.15.40:0/2267783359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baea7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7fade087b640 1 --2- 172.21.15.40:0/1241838145 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadc40a2900 0x7fadc40a4dc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fadd4005e30 tx=0x7fadd40095c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.840+0000 7faddb7fe640 1 --2- 172.21.15.40:0/2267783359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19bae52f0 0x55a19bae5710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fadd00127f0 tx=0x7fadd000d3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1ed9e11e69558d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.841+0000 7fadc27fc640 1 -- 172.21.15.40:0/2267783359 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd000aa60 con 0x55a19bae52f0 2024-03-18T22:44:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.841+0000 7fadc27fc640 1 -- 172.21.15.40:0/2267783359 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadd0012e30 con 0x55a19bae52f0 2024-03-18T22:44:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.841+0000 7fadc27fc640 1 -- 172.21.15.40:0/2267783359 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd0004ad0 con 0x55a19bae52f0 2024-03-18T22:44:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.841+0000 7fade4f9b740 1 -- 172.21.15.40:0/2267783359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19bae52f0 msgr2=0x55a19bae5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.841+0000 7fade4f9b740 1 --2- 172.21.15.40:0/2267783359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19bae52f0 0x55a19bae5710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fadd00127f0 tx=0x7fadd000d3c0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.842+0000 7fade4f9b740 1 -- 172.21.15.40:0/2267783359 shutdown_connections 2024-03-18T22:44:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.842+0000 7fade4f9b740 1 --2- 172.21.15.40:0/2267783359 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19baeae50 0x55a19baed2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.842+0000 7fade4f9b740 1 --2- 172.21.15.40:0/2267783359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baea7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.842+0000 7fade4f9b740 1 --2- 172.21.15.40:0/2267783359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19bae52f0 0x55a19bae5710 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.842+0000 7fade4f9b740 1 -- 172.21.15.40:0/2267783359 >> 172.21.15.40:0/2267783359 conn(0x55a19bae0860 msgr2=0x55a19bae0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.842+0000 7fade4f9b740 1 -- 172.21.15.40:0/2267783359 shutdown_connections 2024-03-18T22:44:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.842+0000 7fade4f9b740 1 -- 172.21.15.40:0/2267783359 wait complete. 2024-03-18T22:44:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.842+0000 7fade4f9b740 1 -- start start 2024-03-18T22:44:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.842+0000 7fade4f9b740 1 Processor -- start 2024-03-18T22:44:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.842+0000 7fade4f9b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 0x55a19baef890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.842+0000 7fade4f9b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baefdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.842+0000 7fade4f9b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19baeae50 0x55a19baf6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.842+0000 7fade4f9b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a19baede00 con 0x55a19bae5ce0 2024-03-18T22:44:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.842+0000 7fade4f9b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a19ba05310 con 0x55a19baeae50 2024-03-18T22:44:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.842+0000 7fade4f9b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55a19baedf80 con 0x55a19bae52f0 2024-03-18T22:44:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7fade4f9b740 10 client.0 _ll_register_callbacks cb 0x55a19b9f92d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7faddbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baefdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7faddb7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 0x55a19baef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7faddb7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 0x55a19baef890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:46028/0 (socket says 172.21.15.40:46028) 2024-03-18T22:44:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7faddb7fe640 1 -- 172.21.15.40:0/1343074656 learned_addr learned my addr 172.21.15.40:0/1343074656 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7faddbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baefdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38948/0 (socket says 172.21.15.40:38948) 2024-03-18T22:44:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7faddb7fe640 1 --2- 172.21.15.40:0/1343074656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19baeae50 0x55a19baf6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7faddbfff640 1 -- 172.21.15.40:0/1343074656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 msgr2=0x55a19baef890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7faddbfff640 1 --2- 172.21.15.40:0/1343074656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 0x55a19baef890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7faddbfff640 1 -- 172.21.15.40:0/1343074656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19baeae50 msgr2=0x55a19baf6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7faddbfff640 1 --2- 172.21.15.40:0/1343074656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19baeae50 0x55a19baf6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7faddbfff640 1 -- 172.21.15.40:0/1343074656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a19b75d3f0 con 0x55a19bae5ce0 2024-03-18T22:44:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7faddb7fe640 1 --2- 172.21.15.40:0/1343074656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 0x55a19baef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7faddbfff640 1 --2- 172.21.15.40:0/1343074656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baefdd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x55a19bae17d0 tx=0x7fadcc012ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5646cecf2c4d5594 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7fadbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:44:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7fadbb7fe640 1 -- 172.21.15.40:0/1343074656 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc004020 con 0x55a19bae5ce0 2024-03-18T22:44:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7fadbb7fe640 1 -- 172.21.15.40:0/1343074656 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fadcc0041c0 con 0x55a19bae5ce0 2024-03-18T22:44:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.843+0000 7fade4f9b740 1 -- 172.21.15.40:0/1343074656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadd0012e30 con 0x55a19bae5ce0 2024-03-18T22:44:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.844+0000 7fadbb7fe640 1 -- 172.21.15.40:0/1343074656 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fadcc0043c0 con 0x55a19bae5ce0 2024-03-18T22:44:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.844+0000 7fade4f9b740 10 client.15858 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.844+0000 7fade4f9b740 1 -- 172.21.15.40:0/1343074656 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a19b8277d0 con 0x55a19bae5ce0 2024-03-18T22:44:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.844+0000 7fadbb7fe640 1 -- 172.21.15.40:0/1343074656 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadd0012e30 con 0x55a19bae5ce0 2024-03-18T22:44:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.844+0000 7fadbb7fe640 1 -- 172.21.15.40:0/1343074656 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fadcc0039c0 con 0x55a19bae5ce0 2024-03-18T22:44:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.844+0000 7fade4f9b740 10 client.15858 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.844+0000 7fade4f9b740 10 client.15858 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:44:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.844+0000 7fade4f9b740 20 client.15858 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:14.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.844+0000 7fade4f9b740 1 --2- 172.21.15.40:0/1343074656 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a19bafb850 0x55a19bafdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.844+0000 7fade4f9b740 4 client.15858 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:14.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.844+0000 7fade4f9b740 1 -- 172.21.15.40:0/1343074656 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a19baa73e0 con 0x55a19bafb850 2024-03-18T22:44:14.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.844+0000 7faddb7fe640 1 --2- 172.21.15.40:0/1343074656 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a19bafb850 0x55a19bafdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.845+0000 7fadbb7fe640 10 client.15858 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:44:14.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.845+0000 7faddb7fe640 1 --2- 172.21.15.40:0/1343074656 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a19bafb850 0x55a19bafdcb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=769e1d3832aa1dc0 server_cookie=ac17bbe2e4043655 in_seq=0 out_seq=0 2024-03-18T22:44:14.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.845+0000 7fadbb7fe640 1 -- 172.21.15.40:0/1343074656 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a19baa73e0 con 0x55a19bafb850 2024-03-18T22:44:14.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.845+0000 7fadbb7fe640 10 client.15858 handle_command_reply: tid=0 2024-03-18T22:44:14.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.845+0000 7fade4f9b740 1 client.15858 shutdown 2024-03-18T22:44:14.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.845+0000 7fade4f9b740 1 -- 172.21.15.40:0/1343074656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 msgr2=0x55a19baefdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.846+0000 7fade4f9b740 1 --2- 172.21.15.40:0/1343074656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baefdd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x55a19bae17d0 tx=0x7fadcc012ca0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.846+0000 7fade4f9b740 1 -- 172.21.15.40:0/1343074656 shutdown_connections 2024-03-18T22:44:14.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.846+0000 7fade4f9b740 1 --2- 172.21.15.40:0/1343074656 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a19bafb850 0x55a19bafdcb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.846+0000 7fade4f9b740 1 --2- 172.21.15.40:0/1343074656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19baeae50 0x55a19baf6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.846+0000 7fade4f9b740 1 --2- 172.21.15.40:0/1343074656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baefdd0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.846+0000 7fade4f9b740 1 --2- 172.21.15.40:0/1343074656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 0x55a19baef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.846+0000 7fadbb7fe640 0 client.15858 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:44:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.846+0000 7fade4f9b740 1 -- 172.21.15.40:0/1343074656 >> 172.21.15.40:0/1343074656 conn(0x55a19bae0860 msgr2=0x55a19badfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.846+0000 7fade4f9b740 1 -- 172.21.15.40:0/1343074656 shutdown_connections 2024-03-18T22:44:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.846+0000 7fade4f9b740 1 -- 172.21.15.40:0/1343074656 wait complete. 2024-03-18T22:44:14.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.846+0000 7fade4f9b740 20 client.15858 trim_cache size 0 max 16384 2024-03-18T22:44:14.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7fade4f9b740 1 Processor -- start 2024-03-18T22:44:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7fade4f9b740 1 -- start start 2024-03-18T22:44:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7fade4f9b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 0x55a19baef270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7fade4f9b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19bae5ce0 0x55a19baf6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7fade4f9b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19baeae50 0x55a19badf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7fade4f9b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fadc800fd20 con 0x55a19baeae50 2024-03-18T22:44:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7fade4f9b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fadc8011890 con 0x55a19bae5ce0 2024-03-18T22:44:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7fade4f9b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fadc80103f0 con 0x55a19bae52f0 2024-03-18T22:44:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7faddb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19baeae50 0x55a19badf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7faddbfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19bae5ce0 0x55a19baf6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7faddb7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 0x55a19baef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7faddbfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19bae5ce0 0x55a19baf6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:33784/0 (socket says 172.21.15.40:33784) 2024-03-18T22:44:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7faddbfff640 1 -- 172.21.15.40:0/2976517241 learned_addr learned my addr 172.21.15.40:0/2976517241 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7faddbfff640 1 -- 172.21.15.40:0/2976517241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 msgr2=0x55a19baef270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7faddbfff640 1 --2- 172.21.15.40:0/2976517241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 0x55a19baef270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7faddbfff640 1 -- 172.21.15.40:0/2976517241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19baeae50 msgr2=0x55a19badf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7faddbfff640 1 --2- 172.21.15.40:0/2976517241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19baeae50 0x55a19badf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7faddbfff640 1 -- 172.21.15.40:0/2976517241 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a19bab6880 con 0x55a19bae5ce0 2024-03-18T22:44:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.866+0000 7faddb7fe640 1 --2- 172.21.15.40:0/2976517241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 0x55a19baef270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.867+0000 7faddbfff640 1 --2- 172.21.15.40:0/2976517241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19bae5ce0 0x55a19baf6dd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fadcc003e00 tx=0x7fadcc005710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=493d16b06e46323e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.867+0000 7fadc1ffb640 1 -- 172.21.15.40:0/2976517241 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc012360 con 0x55a19bae5ce0 2024-03-18T22:44:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.867+0000 7fadc1ffb640 1 -- 172.21.15.40:0/2976517241 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadcc0054e0 con 0x55a19bae5ce0 2024-03-18T22:44:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.867+0000 7fadc1ffb640 1 -- 172.21.15.40:0/2976517241 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc00f790 con 0x55a19bae5ce0 2024-03-18T22:44:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.868+0000 7fade4f9b740 1 -- 172.21.15.40:0/2976517241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19bae5ce0 msgr2=0x55a19baf6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.868+0000 7fade4f9b740 1 --2- 172.21.15.40:0/2976517241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19bae5ce0 0x55a19baf6dd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fadcc003e00 tx=0x7fadcc005710 comp rx=0 tx=0).stop 2024-03-18T22:44:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.868+0000 7fade4f9b740 1 -- 172.21.15.40:0/2976517241 shutdown_connections 2024-03-18T22:44:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.868+0000 7fade4f9b740 1 --2- 172.21.15.40:0/2976517241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19baeae50 0x55a19badf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.868+0000 7fade4f9b740 1 --2- 172.21.15.40:0/2976517241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19bae5ce0 0x55a19baf6dd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.868+0000 7fade4f9b740 1 --2- 172.21.15.40:0/2976517241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 0x55a19baef270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.868+0000 7fade4f9b740 1 -- 172.21.15.40:0/2976517241 >> 172.21.15.40:0/2976517241 conn(0x55a19bae0860 msgr2=0x55a19baeeb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.868+0000 7fade4f9b740 1 -- 172.21.15.40:0/2976517241 shutdown_connections 2024-03-18T22:44:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.868+0000 7fade4f9b740 1 -- 172.21.15.40:0/2976517241 wait complete. 2024-03-18T22:44:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.868+0000 7fade4f9b740 1 -- start start 2024-03-18T22:44:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.868+0000 7fade4f9b740 1 Processor -- start 2024-03-18T22:44:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.868+0000 7fade4f9b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 0x55a19baf4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7fade4f9b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baf4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7fade4f9b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19baeae50 0x55a19baf5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7fade4f9b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fadc8011190 con 0x55a19bae5ce0 2024-03-18T22:44:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7fade4f9b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fadc8010ac0 con 0x55a19baeae50 2024-03-18T22:44:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7fade4f9b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fadc800dd00 con 0x55a19bae52f0 2024-03-18T22:44:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7fade4f9b740 10 client.0 _ll_register_callbacks cb 0x7fadcc009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7faddbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baf4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7faddbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baf4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38970/0 (socket says 172.21.15.40:38970) 2024-03-18T22:44:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7faddbfff640 1 -- 172.21.15.40:0/2444954348 learned_addr learned my addr 172.21.15.40:0/2444954348 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7faddb7fe640 1 --2- 172.21.15.40:0/2444954348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 0x55a19baf4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7faddb7fe640 1 --2- 172.21.15.40:0/2444954348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19baeae50 0x55a19baf5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7faddbfff640 1 -- 172.21.15.40:0/2444954348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 msgr2=0x55a19baf4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7faddbfff640 1 --2- 172.21.15.40:0/2444954348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 0x55a19baf4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7faddbfff640 1 -- 172.21.15.40:0/2444954348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19baeae50 msgr2=0x55a19baf5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7faddbfff640 1 --2- 172.21.15.40:0/2444954348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19baeae50 0x55a19baf5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7faddbfff640 1 -- 172.21.15.40:0/2444954348 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a19baf5d10 con 0x55a19bae5ce0 2024-03-18T22:44:14.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7faddbfff640 1 --2- 172.21.15.40:0/2444954348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baf4760 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0058c0 tx=0x7fadcc003700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44094183e73486e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:14.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7fadbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:44:14.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7fadbb7fe640 1 -- 172.21.15.40:0/2444954348 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc003b80 con 0x55a19bae5ce0 2024-03-18T22:44:14.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7fadbb7fe640 1 -- 172.21.15.40:0/2444954348 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fadcc00f9a0 con 0x55a19bae5ce0 2024-03-18T22:44:14.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7fade4f9b740 1 -- 172.21.15.40:0/2444954348 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadcc0054e0 con 0x55a19bae5ce0 2024-03-18T22:44:14.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.869+0000 7fadbb7fe640 1 -- 172.21.15.40:0/2444954348 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fadcc00fba0 con 0x55a19bae5ce0 2024-03-18T22:44:14.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.870+0000 7fade4f9b740 10 client.15864 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:14.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.870+0000 7fade4f9b740 1 -- 172.21.15.40:0/2444954348 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a19baf7010 con 0x55a19bae5ce0 2024-03-18T22:44:14.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.870+0000 7fadbb7fe640 1 -- 172.21.15.40:0/2444954348 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadcc0054e0 con 0x55a19bae5ce0 2024-03-18T22:44:14.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.870+0000 7fadbb7fe640 1 -- 172.21.15.40:0/2444954348 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fadcc033450 con 0x55a19bae5ce0 2024-03-18T22:44:14.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.870+0000 7fade4f9b740 10 client.15864 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:14.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.870+0000 7fade4f9b740 10 client.15864 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:44:14.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.870+0000 7fade4f9b740 20 client.15864 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:14.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.870+0000 7fade4f9b740 1 --2- 172.21.15.40:0/2444954348 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a19bb62240 0x55a19bb64660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:14.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.870+0000 7fade4f9b740 4 client.15864 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:44:14.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.870+0000 7fade4f9b740 1 -- 172.21.15.40:0/2444954348 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a19ba569f0 con 0x55a19bb62240 2024-03-18T22:44:14.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.870+0000 7faddb7fe640 1 --2- 172.21.15.40:0/2444954348 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a19bb62240 0x55a19bb64660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:14.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fadbb7fe640 10 client.15864 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:44:14.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7faddb7fe640 1 --2- 172.21.15.40:0/2444954348 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a19bb62240 0x55a19bb64660 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=420123f0a9b9bf52 server_cookie=4a45ffedca33d5ef in_seq=0 out_seq=0 2024-03-18T22:44:14.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fadbb7fe640 1 -- 172.21.15.40:0/2444954348 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a19ba569f0 con 0x55a19bb62240 2024-03-18T22:44:14.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fadbb7fe640 10 client.15864 handle_command_reply: tid=0 2024-03-18T22:44:14.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fade4f9b740 1 client.15864 shutdown 2024-03-18T22:44:14.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fade4f9b740 1 -- 172.21.15.40:0/2444954348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 msgr2=0x55a19baf4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fade4f9b740 1 --2- 172.21.15.40:0/2444954348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baf4760 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0058c0 tx=0x7fadcc003700 comp rx=0 tx=0).stop 2024-03-18T22:44:14.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fade4f9b740 1 -- 172.21.15.40:0/2444954348 shutdown_connections 2024-03-18T22:44:14.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fade4f9b740 1 --2- 172.21.15.40:0/2444954348 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a19bb62240 0x55a19bb64660 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fade4f9b740 1 --2- 172.21.15.40:0/2444954348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a19baeae50 0x55a19baf5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fade4f9b740 1 --2- 172.21.15.40:0/2444954348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a19bae5ce0 0x55a19baf4760 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fade4f9b740 1 --2- 172.21.15.40:0/2444954348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a19bae52f0 0x55a19baf4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fade4f9b740 1 -- 172.21.15.40:0/2444954348 >> 172.21.15.40:0/2444954348 conn(0x55a19bae0860 msgr2=0x55a19bae3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fadbb7fe640 0 client.15864 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:44:14.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fade4f9b740 1 -- 172.21.15.40:0/2444954348 shutdown_connections 2024-03-18T22:44:14.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fade4f9b740 1 -- 172.21.15.40:0/2444954348 wait complete. 2024-03-18T22:44:14.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.871+0000 7fade4f9b740 20 client.15864 trim_cache size 0 max 16384 2024-03-18T22:44:14.879 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:44:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.872+0000 7fade22de640 1 -- 172.21.15.40:0/1241838145 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadc40a2900 msgr2=0x7fadc40a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.872+0000 7fade22de640 1 --2- 172.21.15.40:0/1241838145 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadc40a2900 0x7fadc40a4dc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fadd4005e30 tx=0x7fadd40095c0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.872+0000 7fade22de640 1 -- 172.21.15.40:0/1241838145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc1022d0 msgr2=0x7faddc1a1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.872+0000 7fade22de640 1 --2- 172.21.15.40:0/1241838145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc1022d0 0x7faddc1a1120 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fadd402c640 tx=0x7fadd4035d60 comp rx=0 tx=0).stop 2024-03-18T22:44:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.872+0000 7fade22de640 1 -- 172.21.15.40:0/1241838145 shutdown_connections 2024-03-18T22:44:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.872+0000 7fade22de640 1 --2- 172.21.15.40:0/1241838145 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadc40a2900 0x7fadc40a4dc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.872+0000 7fade22de640 1 --2- 172.21.15.40:0/1241838145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc1022d0 0x7faddc1a1120 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.872+0000 7fade22de640 1 --2- 172.21.15.40:0/1241838145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faddc101910 0x7faddc19a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.872+0000 7fade22de640 1 --2- 172.21.15.40:0/1241838145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc100720 0x7faddc199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.872+0000 7fade22de640 1 -- 172.21.15.40:0/1241838145 >> 172.21.15.40:0/1241838145 conn(0x7faddc0fbc90 msgr2=0x7faddc0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.872+0000 7fade22de640 1 -- 172.21.15.40:0/1241838145 shutdown_connections 2024-03-18T22:44:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.872+0000 7fade22de640 1 -- 172.21.15.40:0/1241838145 wait complete. 2024-03-18T22:44:14.885 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:44:15.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.997+0000 7f486caf1640 1 -- 172.21.15.40:0/4096821760 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48681013e0 msgr2=0x7f4868108c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.997+0000 7f486caf1640 1 --2- 172.21.15.40:0/4096821760 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48681013e0 0x7f4868108c80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4860009be0 tx=0x7f486002bef0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.998+0000 7f486caf1640 1 -- 172.21.15.40:0/4096821760 shutdown_connections 2024-03-18T22:44:15.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.998+0000 7f486caf1640 1 --2- 172.21.15.40:0/4096821760 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f48681013e0 0x7f4868108c80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.998+0000 7f486caf1640 1 --2- 172.21.15.40:0/4096821760 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4868100a20 0x7f4868100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.998+0000 7f486caf1640 1 --2- 172.21.15.40:0/4096821760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4868101cf0 0x7f4868102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.998+0000 7f486caf1640 1 -- 172.21.15.40:0/4096821760 >> 172.21.15.40:0/4096821760 conn(0x7f48680fb450 msgr2=0x7f48680fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.998+0000 7f486caf1640 1 -- 172.21.15.40:0/4096821760 shutdown_connections 2024-03-18T22:44:15.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.998+0000 7f486caf1640 1 -- 172.21.15.40:0/4096821760 wait complete. 2024-03-18T22:44:15.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.998+0000 7f486caf1640 1 Processor -- start 2024-03-18T22:44:15.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.998+0000 7f486caf1640 1 -- start start 2024-03-18T22:44:15.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f486caf1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4868100a20 0x7f4868197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f486caf1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48681013e0 0x7f4868197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f486caf1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4868101cf0 0x7f486819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f486caf1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f486810c2a0 con 0x7f48681013e0 2024-03-18T22:44:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f486caf1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f486810c120 con 0x7f4868101cf0 2024-03-18T22:44:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f486caf1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f486810c420 con 0x7f4868100a20 2024-03-18T22:44:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f4865d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48681013e0 0x7f4868197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f4865d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48681013e0 0x7f4868197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38980/0 (socket says 172.21.15.40:38980) 2024-03-18T22:44:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f4865d9b640 1 -- 172.21.15.40:0/1971393869 learned_addr learned my addr 172.21.15.40:0/1971393869 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f4866d9d640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4868101cf0 0x7f486819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f486659c640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4868100a20 0x7f4868197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f4865d9b640 1 -- 172.21.15.40:0/1971393869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4868100a20 msgr2=0x7f4868197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f4865d9b640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4868100a20 0x7f4868197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f4865d9b640 1 -- 172.21.15.40:0/1971393869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4868101cf0 msgr2=0x7f486819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f4865d9b640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4868101cf0 0x7f486819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f4865d9b640 1 -- 172.21.15.40:0/1971393869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f486819f650 con 0x7f48681013e0 2024-03-18T22:44:15.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f486659c640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4868100a20 0x7f4868197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:15.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f4866d9d640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4868101cf0 0x7f486819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:15.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:14.999+0000 7f4865d9b640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48681013e0 0x7f4868197f70 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f485c00e9f0 tx=0x7f485c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.000+0000 7f48577fe640 1 -- 172.21.15.40:0/1971393869 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f485c00cd80 con 0x7f48681013e0 2024-03-18T22:44:15.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.000+0000 7f486caf1640 1 -- 172.21.15.40:0/1971393869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f486819f940 con 0x7f48681013e0 2024-03-18T22:44:15.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.000+0000 7f48577fe640 1 -- 172.21.15.40:0/1971393869 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f485c0054d0 con 0x7f48681013e0 2024-03-18T22:44:15.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.000+0000 7f48577fe640 1 -- 172.21.15.40:0/1971393869 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f485c010490 con 0x7f48681013e0 2024-03-18T22:44:15.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.000+0000 7f486caf1640 1 -- 172.21.15.40:0/1971393869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f486819fe90 con 0x7f48681013e0 2024-03-18T22:44:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.002+0000 7f48577fe640 1 -- 172.21.15.40:0/1971393869 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f485c005020 con 0x7f48681013e0 2024-03-18T22:44:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.003+0000 7f486f7b0740 1 Processor -- start 2024-03-18T22:44:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.003+0000 7f486f7b0740 1 -- start start 2024-03-18T22:44:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.003+0000 7f486f7b0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db22f0 0x55f591db2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.003+0000 7f486f7b0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db2ce0 0x55f591db77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.003+0000 7f486f7b0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db7e50 0x55f591dba2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.003+0000 7f486f7b0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f591d164b0 con 0x55f591db22f0 2024-03-18T22:44:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.003+0000 7f486f7b0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f591cbed70 con 0x55f591db7e50 2024-03-18T22:44:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.003+0000 7f486f7b0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f591d12550 con 0x55f591db2ce0 2024-03-18T22:44:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.003+0000 7f48577fe640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f48500a2800 0x7f48500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.003+0000 7f486659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db2ce0 0x55f591db77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.003+0000 7f486659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db2ce0 0x55f591db77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:46064/0 (socket says 172.21.15.40:46064) 2024-03-18T22:44:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.003+0000 7f486659c640 1 -- 172.21.15.40:0/202399966 learned_addr learned my addr 172.21.15.40:0/202399966 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.003+0000 7f4865d9b640 1 --2- 172.21.15.40:0/202399966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db7e50 0x55f591dba2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.003+0000 7f4866d9d640 1 --2- 172.21.15.40:0/202399966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db22f0 0x55f591db2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.004+0000 7f4866d9d640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f48500a2800 0x7f48500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.004+0000 7f486659c640 1 -- 172.21.15.40:0/202399966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db7e50 msgr2=0x55f591dba2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.004+0000 7f486659c640 1 --2- 172.21.15.40:0/202399966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db7e50 0x55f591dba2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.004+0000 7f486659c640 1 -- 172.21.15.40:0/202399966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db22f0 msgr2=0x55f591db2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.004+0000 7f486659c640 1 --2- 172.21.15.40:0/202399966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db22f0 0x55f591db2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.004+0000 7f48577fe640 1 -- 172.21.15.40:0/1971393869 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f485c0cf1d0 con 0x7f48681013e0 2024-03-18T22:44:15.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.004+0000 7f486659c640 1 -- 172.21.15.40:0/202399966 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f591ce1210 con 0x55f591db2ce0 2024-03-18T22:44:15.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.004+0000 7f4866d9d640 1 --2- 172.21.15.40:0/202399966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db22f0 0x55f591db2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:44:15.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.004+0000 7f486659c640 1 --2- 172.21.15.40:0/202399966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db2ce0 0x55f591db77e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f48580114f0 tx=0x7f48580119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c5410e254f179be server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.004+0000 7f4866d9d640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f48500a2800 0x7f48500a4cc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4860004af0 tx=0x7f4860005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.004+0000 7f4854ff9640 1 -- 172.21.15.40:0/202399966 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4858005020 con 0x55f591db2ce0 2024-03-18T22:44:15.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.004+0000 7f4854ff9640 1 -- 172.21.15.40:0/202399966 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48580047a0 con 0x55f591db2ce0 2024-03-18T22:44:15.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.004+0000 7f4854ff9640 1 -- 172.21.15.40:0/202399966 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f485800f920 con 0x55f591db2ce0 2024-03-18T22:44:15.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.005+0000 7f486f7b0740 1 -- 172.21.15.40:0/202399966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db2ce0 msgr2=0x55f591db77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.005+0000 7f486f7b0740 1 --2- 172.21.15.40:0/202399966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db2ce0 0x55f591db77e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f48580114f0 tx=0x7f48580119b0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.005+0000 7f486f7b0740 1 -- 172.21.15.40:0/202399966 shutdown_connections 2024-03-18T22:44:15.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.005+0000 7f486f7b0740 1 --2- 172.21.15.40:0/202399966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db7e50 0x55f591dba2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.005+0000 7f486f7b0740 1 --2- 172.21.15.40:0/202399966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db2ce0 0x55f591db77e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.005+0000 7f486f7b0740 1 --2- 172.21.15.40:0/202399966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db22f0 0x55f591db2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.005+0000 7f486f7b0740 1 -- 172.21.15.40:0/202399966 >> 172.21.15.40:0/202399966 conn(0x55f591dad860 msgr2=0x55f591dadc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.005+0000 7f486f7b0740 1 -- 172.21.15.40:0/202399966 shutdown_connections 2024-03-18T22:44:15.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.005+0000 7f486f7b0740 1 -- 172.21.15.40:0/202399966 wait complete. 2024-03-18T22:44:15.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.005+0000 7f486f7b0740 1 -- start start 2024-03-18T22:44:15.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.005+0000 7f486f7b0740 1 Processor -- start 2024-03-18T22:44:15.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f486f7b0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db22f0 0x55f591dbc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f486f7b0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db2ce0 0x55f591dbcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f486f7b0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db7e50 0x55f591dc3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f486f7b0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f591dbae00 con 0x55f591db2ce0 2024-03-18T22:44:15.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f486f7b0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f591cd2310 con 0x55f591db7e50 2024-03-18T22:44:15.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f486f7b0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f591dbaf80 con 0x55f591db22f0 2024-03-18T22:44:15.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f486f7b0740 10 client.0 _ll_register_callbacks cb 0x55f591cc62d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f4865d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db2ce0 0x55f591dbcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f486659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db22f0 0x55f591dbc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f4865d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db2ce0 0x55f591dbcd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39006/0 (socket says 172.21.15.40:39006) 2024-03-18T22:44:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f4865d9b640 1 -- 172.21.15.40:0/3696347362 learned_addr learned my addr 172.21.15.40:0/3696347362 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f4866d9d640 1 --2- 172.21.15.40:0/3696347362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db7e50 0x55f591dc3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f4865d9b640 1 -- 172.21.15.40:0/3696347362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db22f0 msgr2=0x55f591dbc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f4865d9b640 1 --2- 172.21.15.40:0/3696347362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db22f0 0x55f591dbc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f4865d9b640 1 -- 172.21.15.40:0/3696347362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db7e50 msgr2=0x55f591dc3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f4865d9b640 1 --2- 172.21.15.40:0/3696347362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db7e50 0x55f591dc3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f4865d9b640 1 -- 172.21.15.40:0/3696347362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f591a2a3f0 con 0x55f591db2ce0 2024-03-18T22:44:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f486659c640 1 --2- 172.21.15.40:0/3696347362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db22f0 0x55f591dbc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f4865d9b640 1 --2- 172.21.15.40:0/3696347362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db2ce0 0x55f591dbcd90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f485c0cc8e0 tx=0x7f485c09b1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e137ae00982646cc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.006+0000 7f484dffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:44:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.007+0000 7f484dffb640 1 -- 172.21.15.40:0/3696347362 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f485c09b780 con 0x55f591db2ce0 2024-03-18T22:44:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.007+0000 7f484dffb640 1 -- 172.21.15.40:0/3696347362 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f485c0860b0 con 0x55f591db2ce0 2024-03-18T22:44:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.007+0000 7f486f7b0740 1 -- 172.21.15.40:0/3696347362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48580047a0 con 0x55f591db2ce0 2024-03-18T22:44:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.007+0000 7f484dffb640 1 -- 172.21.15.40:0/3696347362 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f485c09f8d0 con 0x55f591db2ce0 2024-03-18T22:44:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.007+0000 7f486f7b0740 10 client.15876 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.007+0000 7f486f7b0740 1 -- 172.21.15.40:0/3696347362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f591af47d0 con 0x55f591db2ce0 2024-03-18T22:44:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.007+0000 7f484dffb640 1 -- 172.21.15.40:0/3696347362 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48580047a0 con 0x55f591db2ce0 2024-03-18T22:44:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.007+0000 7f484dffb640 1 -- 172.21.15.40:0/3696347362 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f485c0afb60 con 0x55f591db2ce0 2024-03-18T22:44:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.007+0000 7f486f7b0740 10 client.15876 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.007+0000 7f486f7b0740 10 client.15876 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:44:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.007+0000 7f486f7b0740 20 client.15876 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.007+0000 7f486f7b0740 1 --2- 172.21.15.40:0/3696347362 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f591dc86b0 0x55f591dcab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.007+0000 7f486f7b0740 4 client.15876 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.007+0000 7f486f7b0740 1 -- 172.21.15.40:0/3696347362 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f591d743e0 con 0x55f591dc86b0 2024-03-18T22:44:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.007+0000 7f486659c640 1 --2- 172.21.15.40:0/3696347362 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f591dc86b0 0x55f591dcab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.008+0000 7f484dffb640 10 client.15876 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:44:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.008+0000 7f486659c640 1 --2- 172.21.15.40:0/3696347362 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f591dc86b0 0x55f591dcab50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8918b01ec344dc7c server_cookie=b6c057624053729f in_seq=0 out_seq=0 2024-03-18T22:44:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.008+0000 7f484dffb640 1 -- 172.21.15.40:0/3696347362 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f591d743e0 con 0x55f591dc86b0 2024-03-18T22:44:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.009+0000 7f484dffb640 10 client.15876 handle_command_reply: tid=0 2024-03-18T22:44:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.009+0000 7f486f7b0740 1 client.15876 shutdown 2024-03-18T22:44:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.009+0000 7f486f7b0740 1 -- 172.21.15.40:0/3696347362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db2ce0 msgr2=0x55f591dbcd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.009+0000 7f486f7b0740 1 --2- 172.21.15.40:0/3696347362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db2ce0 0x55f591dbcd90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f485c0cc8e0 tx=0x7f485c09b1a0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.009+0000 7f486f7b0740 1 -- 172.21.15.40:0/3696347362 shutdown_connections 2024-03-18T22:44:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.009+0000 7f486f7b0740 1 --2- 172.21.15.40:0/3696347362 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f591dc86b0 0x55f591dcab50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.009+0000 7f486f7b0740 1 --2- 172.21.15.40:0/3696347362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db7e50 0x55f591dc3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.009+0000 7f486f7b0740 1 --2- 172.21.15.40:0/3696347362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db2ce0 0x55f591dbcd90 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.009+0000 7f484dffb640 0 client.15876 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:44:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.009+0000 7f486f7b0740 1 --2- 172.21.15.40:0/3696347362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db22f0 0x55f591dbc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.009+0000 7f486f7b0740 1 -- 172.21.15.40:0/3696347362 >> 172.21.15.40:0/3696347362 conn(0x55f591dad860 msgr2=0x55f591dacba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.009+0000 7f486f7b0740 1 -- 172.21.15.40:0/3696347362 shutdown_connections 2024-03-18T22:44:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.009+0000 7f486f7b0740 1 -- 172.21.15.40:0/3696347362 wait complete. 2024-03-18T22:44:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.009+0000 7f486f7b0740 20 client.15876 trim_cache size 0 max 16384 2024-03-18T22:44:15.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.023+0000 7f486f7b0740 1 Processor -- start 2024-03-18T22:44:15.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.023+0000 7f486f7b0740 1 -- start start 2024-03-18T22:44:15.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.023+0000 7f486f7b0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db22f0 0x55f591dac310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.023+0000 7f486f7b0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db2ce0 0x55f591dc3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.023+0000 7f486f7b0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db7e50 0x55f591dac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.023+0000 7f486f7b0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f48681aa410 con 0x55f591db22f0 2024-03-18T22:44:15.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.023+0000 7f486f7b0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f48681abf80 con 0x55f591db2ce0 2024-03-18T22:44:15.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.023+0000 7f486f7b0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f48681aaae0 con 0x55f591db7e50 2024-03-18T22:44:15.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.023+0000 7f486659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db22f0 0x55f591dac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.023+0000 7f4866d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db7e50 0x55f591dac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.023+0000 7f486659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db22f0 0x55f591dac310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39014/0 (socket says 172.21.15.40:39014) 2024-03-18T22:44:15.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.023+0000 7f486659c640 1 -- 172.21.15.40:0/4074980103 learned_addr learned my addr 172.21.15.40:0/4074980103 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:15.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.023+0000 7f4865d9b640 1 --2- 172.21.15.40:0/4074980103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db2ce0 0x55f591dc3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.023+0000 7f486659c640 1 -- 172.21.15.40:0/4074980103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db7e50 msgr2=0x55f591dac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.024+0000 7f486659c640 1 --2- 172.21.15.40:0/4074980103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db7e50 0x55f591dac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.024+0000 7f486659c640 1 -- 172.21.15.40:0/4074980103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db2ce0 msgr2=0x55f591dc3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.024+0000 7f486659c640 1 --2- 172.21.15.40:0/4074980103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db2ce0 0x55f591dc3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.024+0000 7f486659c640 1 -- 172.21.15.40:0/4074980103 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f591d83880 con 0x55f591db22f0 2024-03-18T22:44:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.024+0000 7f4866d9d640 1 --2- 172.21.15.40:0/4074980103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db7e50 0x55f591dac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.024+0000 7f4865d9b640 1 --2- 172.21.15.40:0/4074980103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db2ce0 0x55f591dc3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.024+0000 7f486659c640 1 --2- 172.21.15.40:0/4074980103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db22f0 0x55f591dac310 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4858011eb0 tx=0x7f48580119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18174c9362ebf780 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.024+0000 7f484ffff640 1 -- 172.21.15.40:0/4074980103 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f485802d070 con 0x55f591db22f0 2024-03-18T22:44:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.024+0000 7f484ffff640 1 -- 172.21.15.40:0/4074980103 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4858004730 con 0x55f591db22f0 2024-03-18T22:44:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.024+0000 7f484ffff640 1 -- 172.21.15.40:0/4074980103 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4858005110 con 0x55f591db22f0 2024-03-18T22:44:15.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 -- 172.21.15.40:0/4074980103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db22f0 msgr2=0x55f591dac310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 --2- 172.21.15.40:0/4074980103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db22f0 0x55f591dac310 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4858011eb0 tx=0x7f48580119b0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 -- 172.21.15.40:0/4074980103 shutdown_connections 2024-03-18T22:44:15.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 --2- 172.21.15.40:0/4074980103 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db7e50 0x55f591dac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 --2- 172.21.15.40:0/4074980103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db2ce0 0x55f591dc3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 --2- 172.21.15.40:0/4074980103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db22f0 0x55f591dac310 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 -- 172.21.15.40:0/4074980103 >> 172.21.15.40:0/4074980103 conn(0x55f591dad860 msgr2=0x55f591dbc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 -- 172.21.15.40:0/4074980103 shutdown_connections 2024-03-18T22:44:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 -- 172.21.15.40:0/4074980103 wait complete. 2024-03-18T22:44:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 -- start start 2024-03-18T22:44:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 Processor -- start 2024-03-18T22:44:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db22f0 0x55f591dc1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db2ce0 0x55f591dc1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db7e50 0x55f591dc2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f48681ab880 con 0x55f591db2ce0 2024-03-18T22:44:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f48681ab1b0 con 0x55f591db22f0 2024-03-18T22:44:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.025+0000 7f486f7b0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f48681a83f0 con 0x55f591db7e50 2024-03-18T22:44:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f486f7b0740 10 client.0 _ll_register_callbacks cb 0x7f485c00cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f4865d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db2ce0 0x55f591dc1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f486659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db22f0 0x55f591dc1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f486659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db22f0 0x55f591dc1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:33852/0 (socket says 172.21.15.40:33852) 2024-03-18T22:44:15.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f4865d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db2ce0 0x55f591dc1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39018/0 (socket says 172.21.15.40:39018) 2024-03-18T22:44:15.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f486659c640 1 -- 172.21.15.40:0/2859556700 learned_addr learned my addr 172.21.15.40:0/2859556700 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:15.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f4866d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db7e50 0x55f591dc2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f486659c640 1 -- 172.21.15.40:0/2859556700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db7e50 msgr2=0x55f591dc2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f486659c640 1 --2- 172.21.15.40:0/2859556700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db7e50 0x55f591dc2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f486659c640 1 -- 172.21.15.40:0/2859556700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db2ce0 msgr2=0x55f591dc1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f486659c640 1 --2- 172.21.15.40:0/2859556700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db2ce0 0x55f591dc1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f486659c640 1 -- 172.21.15.40:0/2859556700 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f591dc2b80 con 0x55f591db22f0 2024-03-18T22:44:15.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f486659c640 1 --2- 172.21.15.40:0/2859556700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db22f0 0x55f591dc1040 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f485801b040 tx=0x7f48580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d88e05c7461865e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f484dffb640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:44:15.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f484dffb640 1 -- 172.21.15.40:0/2859556700 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f485802d050 con 0x55f591db22f0 2024-03-18T22:44:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f484dffb640 1 -- 172.21.15.40:0/2859556700 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4858029850 con 0x55f591db22f0 2024-03-18T22:44:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f486f7b0740 1 -- 172.21.15.40:0/2859556700 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4858004730 con 0x55f591db22f0 2024-03-18T22:44:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.026+0000 7f484dffb640 1 -- 172.21.15.40:0/2859556700 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4858045050 con 0x55f591db22f0 2024-03-18T22:44:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.027+0000 7f486f7b0740 10 client.25760 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.027+0000 7f486f7b0740 1 -- 172.21.15.40:0/2859556700 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f591dc3e80 con 0x55f591db22f0 2024-03-18T22:44:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.027+0000 7f484dffb640 1 -- 172.21.15.40:0/2859556700 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4858004730 con 0x55f591db22f0 2024-03-18T22:44:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.027+0000 7f484dffb640 1 -- 172.21.15.40:0/2859556700 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f485800f5c0 con 0x55f591db22f0 2024-03-18T22:44:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.027+0000 7f486f7b0740 10 client.25760 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.027+0000 7f486f7b0740 10 client.25760 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:44:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.027+0000 7f486f7b0740 20 client.25760 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.027+0000 7f486f7b0740 1 --2- 172.21.15.40:0/2859556700 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f591e2ee90 0x55f591e312b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.027+0000 7f486f7b0740 4 client.25760 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:44:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.027+0000 7f486f7b0740 1 -- 172.21.15.40:0/2859556700 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f591d239f0 con 0x55f591e2ee90 2024-03-18T22:44:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.027+0000 7f4865d9b640 1 --2- 172.21.15.40:0/2859556700 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f591e2ee90 0x55f591e312b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f484dffb640 10 client.25760 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:44:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f4865d9b640 1 --2- 172.21.15.40:0/2859556700 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f591e2ee90 0x55f591e312b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f734f1548ccfcfa2 server_cookie=d81ef32ecf66c6c3 in_seq=0 out_seq=0 2024-03-18T22:44:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f484dffb640 1 -- 172.21.15.40:0/2859556700 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f591d239f0 con 0x55f591e2ee90 2024-03-18T22:44:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f484dffb640 10 client.25760 handle_command_reply: tid=0 2024-03-18T22:44:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f486f7b0740 1 client.25760 shutdown 2024-03-18T22:44:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f486f7b0740 1 -- 172.21.15.40:0/2859556700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db22f0 msgr2=0x55f591dc1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f486f7b0740 1 --2- 172.21.15.40:0/2859556700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db22f0 0x55f591dc1040 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f485801b040 tx=0x7f48580046c0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f486f7b0740 1 -- 172.21.15.40:0/2859556700 shutdown_connections 2024-03-18T22:44:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f486f7b0740 1 --2- 172.21.15.40:0/2859556700 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f591e2ee90 0x55f591e312b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f486f7b0740 1 --2- 172.21.15.40:0/2859556700 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f591db7e50 0x55f591dc2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f486f7b0740 1 --2- 172.21.15.40:0/2859556700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f591db2ce0 0x55f591dc1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f484dffb640 0 client.25760 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:44:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f486f7b0740 1 --2- 172.21.15.40:0/2859556700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f591db22f0 0x55f591dc1040 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f486f7b0740 1 -- 172.21.15.40:0/2859556700 >> 172.21.15.40:0/2859556700 conn(0x55f591dad860 msgr2=0x55f591db0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f486f7b0740 1 -- 172.21.15.40:0/2859556700 shutdown_connections 2024-03-18T22:44:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f486f7b0740 1 -- 172.21.15.40:0/2859556700 wait complete. 2024-03-18T22:44:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.028+0000 7f486f7b0740 20 client.25760 trim_cache size 0 max 16384 2024-03-18T22:44:15.036 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:44:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.029+0000 7f486caf1640 1 -- 172.21.15.40:0/1971393869 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f48500a2800 msgr2=0x7f48500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.029+0000 7f486caf1640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f48500a2800 0x7f48500a4cc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4860004af0 tx=0x7f4860005fd0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.029+0000 7f486caf1640 1 -- 172.21.15.40:0/1971393869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48681013e0 msgr2=0x7f4868197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.029+0000 7f486caf1640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48681013e0 0x7f4868197f70 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f485c00e9f0 tx=0x7f485c00edf0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.029+0000 7f486caf1640 1 -- 172.21.15.40:0/1971393869 shutdown_connections 2024-03-18T22:44:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.029+0000 7f486caf1640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f48500a2800 0x7f48500a4cc0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.029+0000 7f486caf1640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4868101cf0 0x7f486819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.029+0000 7f486caf1640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48681013e0 0x7f4868197f70 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.029+0000 7f486caf1640 1 --2- 172.21.15.40:0/1971393869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4868100a20 0x7f4868197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.029+0000 7f486caf1640 1 -- 172.21.15.40:0/1971393869 >> 172.21.15.40:0/1971393869 conn(0x7f48680fb450 msgr2=0x7f48680fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.029+0000 7f486caf1640 1 -- 172.21.15.40:0/1971393869 shutdown_connections 2024-03-18T22:44:15.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.029+0000 7f486caf1640 1 -- 172.21.15.40:0/1971393869 wait complete. 2024-03-18T22:44:15.041 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:44:15.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:14 smithi103 ceph-mon[66377]: pgmap v398: 129 pgs: 129 active+clean; 78 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 6.1 MiB/s wr, 123 op/s 2024-03-18T22:44:15.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:14 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2615318075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:44:15.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:14 smithi040 ceph-mon[62643]: pgmap v398: 129 pgs: 129 active+clean; 78 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 6.1 MiB/s wr, 123 op/s 2024-03-18T22:44:15.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:14 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2615318075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:44:15.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.158+0000 7f2b18f1b640 1 -- 172.21.15.40:0/1263507487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b141022d0 msgr2=0x7f2b14108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.158+0000 7f2b18f1b640 1 --2- 172.21.15.40:0/1263507487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b141022d0 0x7f2b14108b60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c009ca0 tx=0x7f2b0c02c100 comp rx=0 tx=0).stop 2024-03-18T22:44:15.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.159+0000 7f2b18f1b640 1 -- 172.21.15.40:0/1263507487 shutdown_connections 2024-03-18T22:44:15.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.159+0000 7f2b18f1b640 1 --2- 172.21.15.40:0/1263507487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b141022d0 0x7f2b14108b60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.159+0000 7f2b18f1b640 1 --2- 172.21.15.40:0/1263507487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14101910 0x7f2b14101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.159+0000 7f2b18f1b640 1 --2- 172.21.15.40:0/1263507487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b14100720 0x7f2b14100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.159+0000 7f2b18f1b640 1 -- 172.21.15.40:0/1263507487 >> 172.21.15.40:0/1263507487 conn(0x7f2b140fbc90 msgr2=0x7f2b140fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.159+0000 7f2b18f1b640 1 -- 172.21.15.40:0/1263507487 shutdown_connections 2024-03-18T22:44:15.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.159+0000 7f2b18f1b640 1 -- 172.21.15.40:0/1263507487 wait complete. 2024-03-18T22:44:15.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b18f1b640 1 Processor -- start 2024-03-18T22:44:15.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b18f1b640 1 -- start start 2024-03-18T22:44:15.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b18f1b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14100720 0x7f2b140694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b18f1b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b14101910 0x7f2b1419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b18f1b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b141022d0 0x7f2b141a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b18f1b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1410c2c0 con 0x7f2b14100720 2024-03-18T22:44:15.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b18f1b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1410c140 con 0x7f2b14101910 2024-03-18T22:44:15.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b18f1b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1410c440 con 0x7f2b141022d0 2024-03-18T22:44:15.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b1259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14100720 0x7f2b140694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b12d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b141022d0 0x7f2b141a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b11d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b14101910 0x7f2b1419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b12d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b141022d0 0x7f2b141a1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:46092/0 (socket says 172.21.15.40:46092) 2024-03-18T22:44:15.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b12d9d640 1 -- 172.21.15.40:0/2645693942 learned_addr learned my addr 172.21.15.40:0/2645693942 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:15.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b11d9b640 1 -- 172.21.15.40:0/2645693942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b141022d0 msgr2=0x7f2b141a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b11d9b640 1 --2- 172.21.15.40:0/2645693942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b141022d0 0x7f2b141a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b11d9b640 1 -- 172.21.15.40:0/2645693942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14100720 msgr2=0x7f2b140694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b11d9b640 1 --2- 172.21.15.40:0/2645693942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14100720 0x7f2b140694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b11d9b640 1 -- 172.21.15.40:0/2645693942 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b141a1900 con 0x7f2b14101910 2024-03-18T22:44:15.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b12d9d640 1 --2- 172.21.15.40:0/2645693942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b141022d0 0x7f2b141a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:15.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.160+0000 7f2b1259c640 1 --2- 172.21.15.40:0/2645693942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14100720 0x7f2b140694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:15.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.161+0000 7f2b11d9b640 1 --2- 172.21.15.40:0/2645693942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b14101910 0x7f2b1419ebc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2b0800e9f0 tx=0x7f2b0800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.161+0000 7f2b037fe640 1 -- 172.21.15.40:0/2645693942 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0800ccc0 con 0x7f2b14101910 2024-03-18T22:44:15.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.161+0000 7f2b037fe640 1 -- 172.21.15.40:0/2645693942 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b080035f0 con 0x7f2b14101910 2024-03-18T22:44:15.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.161+0000 7f2b037fe640 1 -- 172.21.15.40:0/2645693942 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0800ce60 con 0x7f2b14101910 2024-03-18T22:44:15.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.161+0000 7f2b18f1b640 1 -- 172.21.15.40:0/2645693942 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b141a1b90 con 0x7f2b14101910 2024-03-18T22:44:15.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.161+0000 7f2b18f1b640 1 -- 172.21.15.40:0/2645693942 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b141a2090 con 0x7f2b14101910 2024-03-18T22:44:15.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.164+0000 7f2b037fe640 1 -- 172.21.15.40:0/2645693942 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2b080037a0 con 0x7f2b14101910 2024-03-18T22:44:15.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.164+0000 7f2b1bbe1740 1 Processor -- start 2024-03-18T22:44:15.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.164+0000 7f2b1bbe1740 1 -- start start 2024-03-18T22:44:15.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.164+0000 7f2b1bbe1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c316442f0 0x557c31644710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.164+0000 7f2b1bbe1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c31644ce0 0x557c316497e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.164+0000 7f2b1bbe1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c31649e50 0x557c3164c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.164+0000 7f2b1bbe1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557c315a84b0 con 0x557c316442f0 2024-03-18T22:44:15.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.164+0000 7f2b1bbe1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557c31550d70 con 0x557c31644ce0 2024-03-18T22:44:15.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.164+0000 7f2b1bbe1740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557c315a4550 con 0x557c31649e50 2024-03-18T22:44:15.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b12d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c316442f0 0x557c31644710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b1259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c31644ce0 0x557c316497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b1259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c31644ce0 0x557c316497e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:33886/0 (socket says 172.21.15.40:33886) 2024-03-18T22:44:15.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b1259c640 1 -- 172.21.15.40:0/2273162414 learned_addr learned my addr 172.21.15.40:0/2273162414 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:15.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b11d9b640 1 --2- 172.21.15.40:0/2273162414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c31649e50 0x557c3164c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b037fe640 1 --2- 172.21.15.40:0/2645693942 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2afc0a2830 0x7f2afc0a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b12d9d640 1 --2- 172.21.15.40:0/2645693942 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2afc0a2830 0x7f2afc0a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b1259c640 1 -- 172.21.15.40:0/2273162414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c31649e50 msgr2=0x557c3164c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b1259c640 1 --2- 172.21.15.40:0/2273162414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c31649e50 0x557c3164c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b1259c640 1 -- 172.21.15.40:0/2273162414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c316442f0 msgr2=0x557c31644710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b1259c640 1 --2- 172.21.15.40:0/2273162414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c316442f0 0x557c31644710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b1259c640 1 -- 172.21.15.40:0/2273162414 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c31573210 con 0x557c31644ce0 2024-03-18T22:44:15.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b037fe640 1 -- 172.21.15.40:0/2645693942 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2b080ceff0 con 0x7f2b14101910 2024-03-18T22:44:15.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b1259c640 1 --2- 172.21.15.40:0/2273162414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c31644ce0 0x557c316497e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009e00 tx=0x7f2b0400e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aa7a2b15cb72efc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b00ff9640 1 -- 172.21.15.40:0/2273162414 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04017070 con 0x557c31644ce0 2024-03-18T22:44:15.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b12d9d640 1 --2- 172.21.15.40:0/2645693942 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2afc0a2830 0x7f2afc0a4cf0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x557c31645540 tx=0x7f2b0c02c100 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b00ff9640 1 -- 172.21.15.40:0/2273162414 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b04013920 con 0x557c31644ce0 2024-03-18T22:44:15.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.165+0000 7f2b00ff9640 1 -- 172.21.15.40:0/2273162414 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04012830 con 0x557c31644ce0 2024-03-18T22:44:15.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.166+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/2273162414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c31644ce0 msgr2=0x557c316497e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.166+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/2273162414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c31644ce0 0x557c316497e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009e00 tx=0x7f2b0400e5a0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.166+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/2273162414 shutdown_connections 2024-03-18T22:44:15.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.166+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/2273162414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c31649e50 0x557c3164c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.166+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/2273162414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c31644ce0 0x557c316497e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.166+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/2273162414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c316442f0 0x557c31644710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.166+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/2273162414 >> 172.21.15.40:0/2273162414 conn(0x557c3163f860 msgr2=0x557c3163fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.166+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/2273162414 shutdown_connections 2024-03-18T22:44:15.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.166+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/2273162414 wait complete. 2024-03-18T22:44:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1bbe1740 1 -- start start 2024-03-18T22:44:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1bbe1740 1 Processor -- start 2024-03-18T22:44:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1bbe1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c316442f0 0x557c3164e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1bbe1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 0x557c3164ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1bbe1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c31649e50 0x557c31655db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1bbe1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557c3164ce00 con 0x557c31644ce0 2024-03-18T22:44:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1bbe1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557c31564310 con 0x557c31649e50 2024-03-18T22:44:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1bbe1740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557c3164cf80 con 0x557c316442f0 2024-03-18T22:44:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1bbe1740 10 client.0 _ll_register_callbacks cb 0x557c315582d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1259c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c316442f0 0x557c3164e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b11d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 0x557c3164ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1259c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c316442f0 0x557c3164e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:46114/0 (socket says 172.21.15.40:46114) 2024-03-18T22:44:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1259c640 1 -- 172.21.15.40:0/3647452823 learned_addr learned my addr 172.21.15.40:0/3647452823 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b12d9d640 1 --2- 172.21.15.40:0/3647452823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c31649e50 0x557c31655db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1259c640 1 -- 172.21.15.40:0/3647452823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c31649e50 msgr2=0x557c31655db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1259c640 1 --2- 172.21.15.40:0/3647452823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c31649e50 0x557c31655db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1259c640 1 -- 172.21.15.40:0/3647452823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 msgr2=0x557c3164ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1259c640 1 --2- 172.21.15.40:0/3647452823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 0x557c3164ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.167+0000 7f2b1259c640 1 -- 172.21.15.40:0/3647452823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c312bc3f0 con 0x557c316442f0 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.168+0000 7f2b1259c640 1 --2- 172.21.15.40:0/3647452823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c316442f0 0x557c3164e830 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2b040092c0 tx=0x7f2b04013d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=814b74a27c812ea4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.168+0000 7f2af9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.168+0000 7f2af9ffb640 1 -- 172.21.15.40:0/3647452823 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04017050 con 0x557c316442f0 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.168+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3647452823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b04013920 con 0x557c316442f0 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.168+0000 7f2af9ffb640 1 -- 172.21.15.40:0/3647452823 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2b04012980 con 0x557c316442f0 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.168+0000 7f2af9ffb640 1 -- 172.21.15.40:0/3647452823 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2b04013df0 con 0x557c316442f0 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.168+0000 7f2b1bbe1740 10 client.25744 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.168+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3647452823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557c313867d0 con 0x557c316442f0 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.168+0000 7f2af9ffb640 1 -- 172.21.15.40:0/3647452823 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b04013920 con 0x557c316442f0 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.168+0000 7f2af9ffb640 1 -- 172.21.15.40:0/3647452823 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f2b0401cc40 con 0x557c316442f0 2024-03-18T22:44:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.168+0000 7f2b1bbe1740 10 client.25744 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.168+0000 7f2b1bbe1740 10 client.25744 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:44:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.168+0000 7f2b1bbe1740 20 client.25744 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.169+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/3647452823 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557c3165a280 0x557c3165c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.169+0000 7f2b1bbe1740 4 client.25744 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.169+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3647452823 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c316063e0 con 0x557c3165a280 2024-03-18T22:44:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.169+0000 7f2b11d9b640 1 --2- 172.21.15.40:0/3647452823 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557c3165a280 0x557c3165c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.169+0000 7f2b11d9b640 1 --2- 172.21.15.40:0/3647452823 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557c3165a280 0x557c3165c710 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=67da9eed64872bed server_cookie=2a1e5b5235820a34 in_seq=0 out_seq=0 2024-03-18T22:44:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.169+0000 7f2af9ffb640 10 client.25744 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:44:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.170+0000 7f2af9ffb640 1 -- 172.21.15.40:0/3647452823 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557c316063e0 con 0x557c3165a280 2024-03-18T22:44:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.170+0000 7f2af9ffb640 10 client.25744 handle_command_reply: tid=0 2024-03-18T22:44:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.170+0000 7f2b1bbe1740 1 client.25744 shutdown 2024-03-18T22:44:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.170+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3647452823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c316442f0 msgr2=0x557c3164e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.170+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/3647452823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c316442f0 0x557c3164e830 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2b040092c0 tx=0x7f2b04013d00 comp rx=0 tx=0).stop 2024-03-18T22:44:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.170+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3647452823 shutdown_connections 2024-03-18T22:44:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.170+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/3647452823 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557c3165a280 0x557c3165c710 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.170+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/3647452823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c31649e50 0x557c31655db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.170+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/3647452823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 0x557c3164ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.170+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/3647452823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c316442f0 0x557c3164e830 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.170+0000 7f2af9ffb640 0 client.25744 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:44:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.170+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3647452823 >> 172.21.15.40:0/3647452823 conn(0x557c3163f860 msgr2=0x557c3163eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.170+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3647452823 shutdown_connections 2024-03-18T22:44:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.170+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3647452823 wait complete. 2024-03-18T22:44:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.170+0000 7f2b1bbe1740 20 client.25744 trim_cache size 0 max 16384 2024-03-18T22:44:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b1bbe1740 1 Processor -- start 2024-03-18T22:44:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b1bbe1740 1 -- start start 2024-03-18T22:44:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b1bbe1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c3163f860 0x557c316431d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b1bbe1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c316442f0 0x557c31655d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b1bbe1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 0x557c3164de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b1bbe1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2b141ac850 con 0x557c31644ce0 2024-03-18T22:44:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b1bbe1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2b141ae3c0 con 0x557c3163f860 2024-03-18T22:44:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b1bbe1740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2b141acf20 con 0x557c316442f0 2024-03-18T22:44:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b12d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 0x557c3164de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b1259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c3163f860 0x557c316431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b12d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 0x557c3164de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39056/0 (socket says 172.21.15.40:39056) 2024-03-18T22:44:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b1259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c3163f860 0x557c316431d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:33906/0 (socket says 172.21.15.40:33906) 2024-03-18T22:44:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b11d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c316442f0 0x557c31655d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b12d9d640 1 -- 172.21.15.40:0/531035205 learned_addr learned my addr 172.21.15.40:0/531035205 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b1259c640 1 -- 172.21.15.40:0/531035205 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c316442f0 msgr2=0x557c31655d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b1259c640 1 --2- 172.21.15.40:0/531035205 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c316442f0 0x557c31655d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b1259c640 1 -- 172.21.15.40:0/531035205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 msgr2=0x557c3164de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b1259c640 1 --2- 172.21.15.40:0/531035205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 0x557c3164de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.184+0000 7f2b1259c640 1 -- 172.21.15.40:0/531035205 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c31615880 con 0x557c3163f860 2024-03-18T22:44:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.185+0000 7f2b1259c640 1 --2- 172.21.15.40:0/531035205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c3163f860 0x557c316431d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2b04026510 tx=0x7f2b04009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47bebbc35739e98 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.185+0000 7f2afbfff640 1 -- 172.21.15.40:0/531035205 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04009750 con 0x557c3163f860 2024-03-18T22:44:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.185+0000 7f2afbfff640 1 -- 172.21.15.40:0/531035205 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b04026700 con 0x557c3163f860 2024-03-18T22:44:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.185+0000 7f2afbfff640 1 -- 172.21.15.40:0/531035205 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04026e30 con 0x557c3163f860 2024-03-18T22:44:15.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/531035205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c3163f860 msgr2=0x557c316431d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/531035205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c3163f860 0x557c316431d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2b04026510 tx=0x7f2b04009b20 comp rx=0 tx=0).stop 2024-03-18T22:44:15.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/531035205 shutdown_connections 2024-03-18T22:44:15.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/531035205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 0x557c3164de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/531035205 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c316442f0 0x557c31655d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/531035205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c3163f860 0x557c316431d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/531035205 >> 172.21.15.40:0/531035205 conn(0x557c31645ac0 msgr2=0x557c31642ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/531035205 shutdown_connections 2024-03-18T22:44:15.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/531035205 wait complete. 2024-03-18T22:44:15.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 -- start start 2024-03-18T22:44:15.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 Processor -- start 2024-03-18T22:44:15.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c3163f860 0x557c31653010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c316442f0 0x557c31653550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 0x557c31654510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2b141adcc0 con 0x557c31644ce0 2024-03-18T22:44:15.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2b141ad5f0 con 0x557c316442f0 2024-03-18T22:44:15.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2b141aa830 con 0x557c3163f860 2024-03-18T22:44:15.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1259c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c3163f860 0x557c31653010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b11d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c316442f0 0x557c31653550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1bbe1740 10 client.0 _ll_register_callbacks cb 0x7f2b08005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:15.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b12d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 0x557c31654510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b1259c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c3163f860 0x557c31653010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:46136/0 (socket says 172.21.15.40:46136) 2024-03-18T22:44:15.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.186+0000 7f2b11d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c316442f0 0x557c31653550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:33912/0 (socket says 172.21.15.40:33912) 2024-03-18T22:44:15.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2b1259c640 1 -- 172.21.15.40:0/3516071951 learned_addr learned my addr 172.21.15.40:0/3516071951 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:15.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2b1259c640 1 -- 172.21.15.40:0/3516071951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c316442f0 msgr2=0x557c31653550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2b1259c640 1 --2- 172.21.15.40:0/3516071951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c316442f0 0x557c31653550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2b1259c640 1 -- 172.21.15.40:0/3516071951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 msgr2=0x557c31654510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2b1259c640 1 --2- 172.21.15.40:0/3516071951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 0x557c31654510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2b1259c640 1 -- 172.21.15.40:0/3516071951 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c31654b00 con 0x557c3163f860 2024-03-18T22:44:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2b11d9b640 1 --2- 172.21.15.40:0/3516071951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c316442f0 0x557c31653550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2b12d9d640 1 --2- 172.21.15.40:0/3516071951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 0x557c31654510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2b1259c640 1 --2- 172.21.15.40:0/3516071951 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c3163f860 0x557c31653010 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009330 tx=0x7f2b04026f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cf5094ae297c8b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2af9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:44:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2af9ffb640 1 -- 172.21.15.40:0/3516071951 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04037020 con 0x557c3163f860 2024-03-18T22:44:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3516071951 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b04026700 con 0x557c3163f860 2024-03-18T22:44:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2af9ffb640 1 -- 172.21.15.40:0/3516071951 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2b04012780 con 0x557c3163f860 2024-03-18T22:44:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2af9ffb640 1 -- 172.21.15.40:0/3516071951 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2b04042050 con 0x557c3163f860 2024-03-18T22:44:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2b1bbe1740 10 client.25750 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.187+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3516071951 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557c31655e00 con 0x557c3163f860 2024-03-18T22:44:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.188+0000 7f2af9ffb640 1 -- 172.21.15.40:0/3516071951 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b04026700 con 0x557c3163f860 2024-03-18T22:44:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.188+0000 7f2af9ffb640 1 -- 172.21.15.40:0/3516071951 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f2b04033a20 con 0x557c3163f860 2024-03-18T22:44:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.188+0000 7f2b1bbe1740 10 client.25750 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.188+0000 7f2b1bbe1740 10 client.25750 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:44:15.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.188+0000 7f2b1bbe1740 20 client.25750 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:15.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.188+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/3516071951 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557c31649e50 0x557c316c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.188+0000 7f2b1bbe1740 4 client.25750 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:44:15.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.188+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3516071951 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557c315b59f0 con 0x557c31649e50 2024-03-18T22:44:15.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.188+0000 7f2b11d9b640 1 --2- 172.21.15.40:0/3516071951 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557c31649e50 0x557c316c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.188+0000 7f2af9ffb640 10 client.25750 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:44:15.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.188+0000 7f2b11d9b640 1 --2- 172.21.15.40:0/3516071951 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557c31649e50 0x557c316c2b30 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1e385ac212bc2460 server_cookie=613f621d0b88df61 in_seq=0 out_seq=0 2024-03-18T22:44:15.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2af9ffb640 1 -- 172.21.15.40:0/3516071951 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557c315b59f0 con 0x557c31649e50 2024-03-18T22:44:15.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2af9ffb640 10 client.25750 handle_command_reply: tid=0 2024-03-18T22:44:15.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b1bbe1740 1 client.25750 shutdown 2024-03-18T22:44:15.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3516071951 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c3163f860 msgr2=0x557c31653010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/3516071951 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c3163f860 0x557c31653010 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009330 tx=0x7f2b04026f90 comp rx=0 tx=0).stop 2024-03-18T22:44:15.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3516071951 shutdown_connections 2024-03-18T22:44:15.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/3516071951 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557c31649e50 0x557c316c2b30 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/3516071951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557c31644ce0 0x557c31654510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/3516071951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557c316442f0 0x557c31653550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b1bbe1740 1 --2- 172.21.15.40:0/3516071951 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557c3163f860 0x557c31653010 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3516071951 >> 172.21.15.40:0/3516071951 conn(0x557c31645ac0 msgr2=0x557c31643350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2af9ffb640 0 client.25750 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:44:15.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3516071951 shutdown_connections 2024-03-18T22:44:15.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b1bbe1740 1 -- 172.21.15.40:0/3516071951 wait complete. 2024-03-18T22:44:15.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b1bbe1740 20 client.25750 trim_cache size 0 max 16384 2024-03-18T22:44:15.197 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:44:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b18f1b640 1 -- 172.21.15.40:0/2645693942 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2afc0a2830 msgr2=0x7f2afc0a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b18f1b640 1 --2- 172.21.15.40:0/2645693942 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2afc0a2830 0x7f2afc0a4cf0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x557c31645540 tx=0x7f2b0c02c100 comp rx=0 tx=0).stop 2024-03-18T22:44:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b18f1b640 1 -- 172.21.15.40:0/2645693942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b14101910 msgr2=0x7f2b1419ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b18f1b640 1 --2- 172.21.15.40:0/2645693942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b14101910 0x7f2b1419ebc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2b0800e9f0 tx=0x7f2b0800edf0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b18f1b640 1 -- 172.21.15.40:0/2645693942 shutdown_connections 2024-03-18T22:44:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.189+0000 7f2b18f1b640 1 --2- 172.21.15.40:0/2645693942 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2afc0a2830 0x7f2afc0a4cf0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.190+0000 7f2b18f1b640 1 --2- 172.21.15.40:0/2645693942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2b141022d0 0x7f2b141a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.190+0000 7f2b18f1b640 1 --2- 172.21.15.40:0/2645693942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2b14101910 0x7f2b1419ebc0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.190+0000 7f2b18f1b640 1 --2- 172.21.15.40:0/2645693942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b14100720 0x7f2b140694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.190+0000 7f2b18f1b640 1 -- 172.21.15.40:0/2645693942 >> 172.21.15.40:0/2645693942 conn(0x7f2b140fbc90 msgr2=0x7f2b14104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.190+0000 7f2b18f1b640 1 -- 172.21.15.40:0/2645693942 shutdown_connections 2024-03-18T22:44:15.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.190+0000 7f2b18f1b640 1 -- 172.21.15.40:0/2645693942 wait complete. 2024-03-18T22:44:15.202 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:44:15.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.312+0000 7f63d27df640 1 -- 172.21.15.40:0/3881944121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63cc102380 msgr2=0x7f63cc104770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.312+0000 7f63d27df640 1 --2- 172.21.15.40:0/3881944121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63cc102380 0x7f63cc104770 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009900 tx=0x7f63bc02bc30 comp rx=0 tx=0).stop 2024-03-18T22:44:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.312+0000 7f63d27df640 1 -- 172.21.15.40:0/3881944121 shutdown_connections 2024-03-18T22:44:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.312+0000 7f63d27df640 1 --2- 172.21.15.40:0/3881944121 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63cc1075e0 0x7f63cc1099d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.312+0000 7f63d27df640 1 --2- 172.21.15.40:0/3881944121 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63cc104cb0 0x7f63cc1070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.312+0000 7f63d27df640 1 --2- 172.21.15.40:0/3881944121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63cc102380 0x7f63cc104770 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.312+0000 7f63d27df640 1 -- 172.21.15.40:0/3881944121 >> 172.21.15.40:0/3881944121 conn(0x7f63cc0fbcf0 msgr2=0x7f63cc0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.312+0000 7f63d27df640 1 -- 172.21.15.40:0/3881944121 shutdown_connections 2024-03-18T22:44:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.313+0000 7f63d27df640 1 -- 172.21.15.40:0/3881944121 wait complete. 2024-03-18T22:44:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.313+0000 7f63d27df640 1 Processor -- start 2024-03-18T22:44:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.313+0000 7f63d27df640 1 -- start start 2024-03-18T22:44:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63d27df640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63cc102380 0x7f63cc101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63d27df640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63cc104cb0 0x7f63cc100330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63d27df640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63cc1075e0 0x7f63cc100870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63d27df640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc10d3f0 con 0x7f63cc1075e0 2024-03-18T22:44:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63d27df640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc10d270 con 0x7f63cc102380 2024-03-18T22:44:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63d27df640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc10d570 con 0x7f63cc104cb0 2024-03-18T22:44:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63d0d7c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63cc1075e0 0x7f63cc100870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63d0d7c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63cc1075e0 0x7f63cc100870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39088/0 (socket says 172.21.15.40:39088) 2024-03-18T22:44:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63d0d7c640 1 -- 172.21.15.40:0/2151956429 learned_addr learned my addr 172.21.15.40:0/2151956429 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63cb7fe640 1 --2- 172.21.15.40:0/2151956429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63cc104cb0 0x7f63cc100330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63d0d7c640 1 -- 172.21.15.40:0/2151956429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63cc104cb0 msgr2=0x7f63cc100330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63d0d7c640 1 --2- 172.21.15.40:0/2151956429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63cc104cb0 0x7f63cc100330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63d0d7c640 1 -- 172.21.15.40:0/2151956429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63cc102380 msgr2=0x7f63cc101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63d0d7c640 1 --2- 172.21.15.40:0/2151956429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63cc102380 0x7f63cc101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63d0d7c640 1 -- 172.21.15.40:0/2151956429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63cc101100 con 0x7f63cc1075e0 2024-03-18T22:44:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63cb7fe640 1 --2- 172.21.15.40:0/2151956429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63cc104cb0 0x7f63cc100330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.314+0000 7f63d0d7c640 1 --2- 172.21.15.40:0/2151956429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63cc1075e0 0x7f63cc100870 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f63c400eb10 tx=0x7f63c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.315+0000 7f63c97fa640 1 -- 172.21.15.40:0/2151956429 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c4004b20 con 0x7f63cc1075e0 2024-03-18T22:44:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.315+0000 7f63d27df640 1 -- 172.21.15.40:0/2151956429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63cc1a1b90 con 0x7f63cc1075e0 2024-03-18T22:44:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.315+0000 7f63c97fa640 1 -- 172.21.15.40:0/2151956429 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63c4004cc0 con 0x7f63cc1075e0 2024-03-18T22:44:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.315+0000 7f63c97fa640 1 -- 172.21.15.40:0/2151956429 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c4010690 con 0x7f63cc1075e0 2024-03-18T22:44:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.315+0000 7f63d27df640 1 -- 172.21.15.40:0/2151956429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63cc1a20e0 con 0x7f63cc1075e0 2024-03-18T22:44:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.318+0000 7f63c97fa640 1 -- 172.21.15.40:0/2151956429 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f63c4005020 con 0x7f63cc1075e0 2024-03-18T22:44:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.318+0000 7f63d53ca740 1 Processor -- start 2024-03-18T22:44:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.318+0000 7f63d53ca740 1 -- start start 2024-03-18T22:44:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.318+0000 7f63d53ca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22ea2f0 0x55d2e22ea710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.318+0000 7f63d53ca740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22eace0 0x55d2e22ef7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.318+0000 7f63d53ca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22efe50 0x55d2e22f22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.318+0000 7f63c97fa640 1 --2- 172.21.15.40:0/2151956429 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63b40a2800 0x7f63b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.318+0000 7f63d53ca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d2e224e4b0 con 0x55d2e22ea2f0 2024-03-18T22:44:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.318+0000 7f63d53ca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d2e21f6d70 con 0x55d2e22efe50 2024-03-18T22:44:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.318+0000 7f63d53ca740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55d2e224a550 con 0x55d2e22eace0 2024-03-18T22:44:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.319+0000 7f63cb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22ea2f0 0x55d2e22ea710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.318+0000 7f63d0d7c640 1 --2- 172.21.15.40:0/2151956429 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63b40a2800 0x7f63b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.319+0000 7f63cb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22ea2f0 0x55d2e22ea710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39090/0 (socket says 172.21.15.40:39090) 2024-03-18T22:44:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.319+0000 7f63cb7fe640 1 -- 172.21.15.40:0/1963568757 learned_addr learned my addr 172.21.15.40:0/1963568757 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.319+0000 7f63cbfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22eace0 0x55d2e22ef7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.319+0000 7f63c97fa640 1 -- 172.21.15.40:0/2151956429 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f63c40cf650 con 0x7f63cc1075e0 2024-03-18T22:44:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.319+0000 7f63cbfff640 1 -- 172.21.15.40:0/1963568757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22efe50 msgr2=0x55d2e22f22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.319+0000 7f63cbfff640 1 --2- 172.21.15.40:0/1963568757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22efe50 0x55d2e22f22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.319+0000 7f63cbfff640 1 -- 172.21.15.40:0/1963568757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22ea2f0 msgr2=0x55d2e22ea710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.319+0000 7f63cbfff640 1 --2- 172.21.15.40:0/1963568757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22ea2f0 0x55d2e22ea710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.319+0000 7f63cbfff640 1 -- 172.21.15.40:0/1963568757 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2e2219210 con 0x55d2e22eace0 2024-03-18T22:44:15.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.319+0000 7f63cbfff640 1 --2- 172.21.15.40:0/1963568757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22eace0 0x55d2e22ef7e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009230 tx=0x7f63bc005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=603f7eb450332c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.319+0000 7f63b27fc640 1 -- 172.21.15.40:0/1963568757 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc042020 con 0x55d2e22eace0 2024-03-18T22:44:15.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.320+0000 7f63d0d7c640 1 --2- 172.21.15.40:0/2151956429 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63b40a2800 0x7f63b40a4cc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f63c409bb90 tx=0x7f63c400cee0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.320+0000 7f63b27fc640 1 -- 172.21.15.40:0/1963568757 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63bc02cae0 con 0x55d2e22eace0 2024-03-18T22:44:15.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.320+0000 7f63b27fc640 1 -- 172.21.15.40:0/1963568757 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc003c20 con 0x55d2e22eace0 2024-03-18T22:44:15.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.321+0000 7f63d53ca740 1 -- 172.21.15.40:0/1963568757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22eace0 msgr2=0x55d2e22ef7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.321+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1963568757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22eace0 0x55d2e22ef7e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009230 tx=0x7f63bc005780 comp rx=0 tx=0).stop 2024-03-18T22:44:15.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.321+0000 7f63d53ca740 1 -- 172.21.15.40:0/1963568757 shutdown_connections 2024-03-18T22:44:15.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.321+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1963568757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22efe50 0x55d2e22f22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.321+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1963568757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22eace0 0x55d2e22ef7e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.321+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1963568757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22ea2f0 0x55d2e22ea710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.321+0000 7f63d53ca740 1 -- 172.21.15.40:0/1963568757 >> 172.21.15.40:0/1963568757 conn(0x55d2e22e5860 msgr2=0x55d2e22e5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.321+0000 7f63d53ca740 1 -- 172.21.15.40:0/1963568757 shutdown_connections 2024-03-18T22:44:15.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.321+0000 7f63d53ca740 1 -- 172.21.15.40:0/1963568757 wait complete. 2024-03-18T22:44:15.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.321+0000 7f63d53ca740 1 -- start start 2024-03-18T22:44:15.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.322+0000 7f63d53ca740 1 Processor -- start 2024-03-18T22:44:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.322+0000 7f63d53ca740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22ea2f0 0x55d2e22f4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.322+0000 7f63d53ca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22eace0 0x55d2e22f4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.322+0000 7f63d53ca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22efe50 0x55d2e22fbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.322+0000 7f63d53ca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d2e22f2da0 con 0x55d2e22eace0 2024-03-18T22:44:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.322+0000 7f63d53ca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d2e220a310 con 0x55d2e22efe50 2024-03-18T22:44:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.322+0000 7f63d53ca740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55d2e22f2f20 con 0x55d2e22ea2f0 2024-03-18T22:44:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.322+0000 7f63cbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22eace0 0x55d2e22f4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.322+0000 7f63cb7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22ea2f0 0x55d2e22f4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.322+0000 7f63cbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22eace0 0x55d2e22f4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39098/0 (socket says 172.21.15.40:39098) 2024-03-18T22:44:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.322+0000 7f63cbfff640 1 -- 172.21.15.40:0/1565585777 learned_addr learned my addr 172.21.15.40:0/1565585777 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.323+0000 7f63d53ca740 10 client.0 _ll_register_callbacks cb 0x55d2e21fe2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.323+0000 7f63cb7fe640 1 -- 172.21.15.40:0/1565585777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22efe50 msgr2=0x55d2e22fbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.323+0000 7f63cb7fe640 1 --2- 172.21.15.40:0/1565585777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22efe50 0x55d2e22fbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.323+0000 7f63cb7fe640 1 -- 172.21.15.40:0/1565585777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22eace0 msgr2=0x55d2e22f4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.323+0000 7f63cb7fe640 1 --2- 172.21.15.40:0/1565585777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22eace0 0x55d2e22f4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.323+0000 7f63cb7fe640 1 -- 172.21.15.40:0/1565585777 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2e1f623f0 con 0x55d2e22ea2f0 2024-03-18T22:44:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.323+0000 7f63cbfff640 1 --2- 172.21.15.40:0/1565585777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22eace0 0x55d2e22f4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.323+0000 7f63cb7fe640 1 --2- 172.21.15.40:0/1565585777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22ea2f0 0x55d2e22f4770 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f63c0000b60 tx=0x7f63c000f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1aece8d303c8ecb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.323+0000 7f63ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:44:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.323+0000 7f63ab7fe640 1 -- 172.21.15.40:0/1565585777 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0005020 con 0x55d2e22ea2f0 2024-03-18T22:44:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.323+0000 7f63ab7fe640 1 -- 172.21.15.40:0/1565585777 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f63c00051c0 con 0x55d2e22ea2f0 2024-03-18T22:44:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.323+0000 7f63d53ca740 1 -- 172.21.15.40:0/1565585777 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63bc02cae0 con 0x55d2e22ea2f0 2024-03-18T22:44:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.324+0000 7f63ab7fe640 1 -- 172.21.15.40:0/1565585777 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f63c00053c0 con 0x55d2e22ea2f0 2024-03-18T22:44:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.324+0000 7f63d53ca740 10 client.25762 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.324+0000 7f63d53ca740 1 -- 172.21.15.40:0/1565585777 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d2e202c7d0 con 0x55d2e22ea2f0 2024-03-18T22:44:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.324+0000 7f63ab7fe640 1 -- 172.21.15.40:0/1565585777 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63bc02cae0 con 0x55d2e22ea2f0 2024-03-18T22:44:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.324+0000 7f63ab7fe640 1 -- 172.21.15.40:0/1565585777 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f63c0004b70 con 0x55d2e22ea2f0 2024-03-18T22:44:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.324+0000 7f63d53ca740 10 client.25762 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.324+0000 7f63d53ca740 10 client.25762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.324+0000 7f63d53ca740 20 client.25762 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.324+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1565585777 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2e2311650 0x55d2e2313b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.324+0000 7f63d53ca740 4 client.25762 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.324+0000 7f63d53ca740 1 -- 172.21.15.40:0/1565585777 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2e22ac3e0 con 0x55d2e2311650 2024-03-18T22:44:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.324+0000 7f63cbfff640 1 --2- 172.21.15.40:0/1565585777 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2e2311650 0x55d2e2313b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.325+0000 7f63cbfff640 1 --2- 172.21.15.40:0/1565585777 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2e2311650 0x55d2e2313b10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e70e6ba82b45d5b4 server_cookie=83346c07c07162cb in_seq=0 out_seq=0 2024-03-18T22:44:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.325+0000 7f63ab7fe640 10 client.25762 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.325+0000 7f63ab7fe640 1 -- 172.21.15.40:0/1565585777 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d2e22ac3e0 con 0x55d2e2311650 2024-03-18T22:44:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.325+0000 7f63ab7fe640 10 client.25762 handle_command_reply: tid=0 2024-03-18T22:44:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.325+0000 7f63d53ca740 1 client.25762 shutdown 2024-03-18T22:44:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.326+0000 7f63d53ca740 1 -- 172.21.15.40:0/1565585777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22ea2f0 msgr2=0x55d2e22f4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.326+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1565585777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22ea2f0 0x55d2e22f4770 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f63c0000b60 tx=0x7f63c000f7f0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.326+0000 7f63d53ca740 1 -- 172.21.15.40:0/1565585777 shutdown_connections 2024-03-18T22:44:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.326+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1565585777 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2e2311650 0x55d2e2313b10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.326+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1565585777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22efe50 0x55d2e22fbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.326+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1565585777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22eace0 0x55d2e22f4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.326+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1565585777 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22ea2f0 0x55d2e22f4770 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.326+0000 7f63ab7fe640 0 client.25762 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.326+0000 7f63d53ca740 1 -- 172.21.15.40:0/1565585777 >> 172.21.15.40:0/1565585777 conn(0x55d2e22e5860 msgr2=0x55d2e22e4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.326+0000 7f63d53ca740 1 -- 172.21.15.40:0/1565585777 shutdown_connections 2024-03-18T22:44:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.326+0000 7f63d53ca740 1 -- 172.21.15.40:0/1565585777 wait complete. 2024-03-18T22:44:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.326+0000 7f63d53ca740 20 client.25762 trim_cache size 0 max 16384 2024-03-18T22:44:15.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:14 smithi113 ceph-mon[66775]: pgmap v398: 129 pgs: 129 active+clean; 78 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 6.1 MiB/s wr, 123 op/s 2024-03-18T22:44:15.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:14 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2615318075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:44:15.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.340+0000 7f63d53ca740 1 Processor -- start 2024-03-18T22:44:15.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.340+0000 7f63d53ca740 1 -- start start 2024-03-18T22:44:15.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.340+0000 7f63d53ca740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22e5860 0x55d2e22f4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.340+0000 7f63d53ca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22ea2f0 0x55d2e22fbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.340+0000 7f63d53ca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22eace0 0x55d2e22e4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.340+0000 7f63d53ca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc1a8400 con 0x55d2e22ea2f0 2024-03-18T22:44:15.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.340+0000 7f63d53ca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc1a9f70 con 0x55d2e22eace0 2024-03-18T22:44:15.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.340+0000 7f63d53ca740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc1a8ad0 con 0x55d2e22e5860 2024-03-18T22:44:15.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.341+0000 7f63cbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22ea2f0 0x55d2e22fbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.341+0000 7f63cbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22ea2f0 0x55d2e22fbcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39112/0 (socket says 172.21.15.40:39112) 2024-03-18T22:44:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.341+0000 7f63cbfff640 1 -- 172.21.15.40:0/2453259337 learned_addr learned my addr 172.21.15.40:0/2453259337 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.341+0000 7f63cb7fe640 1 --2- 172.21.15.40:0/2453259337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22eace0 0x55d2e22e4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.341+0000 7f63cb7fe640 1 --2- 172.21.15.40:0/2453259337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22e5860 0x55d2e22f4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.341+0000 7f63cbfff640 1 -- 172.21.15.40:0/2453259337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22e5860 msgr2=0x55d2e22f4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.341+0000 7f63cbfff640 1 --2- 172.21.15.40:0/2453259337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22e5860 0x55d2e22f4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.341+0000 7f63cbfff640 1 -- 172.21.15.40:0/2453259337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22eace0 msgr2=0x55d2e22e4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.341+0000 7f63cbfff640 1 --2- 172.21.15.40:0/2453259337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22eace0 0x55d2e22e4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.341+0000 7f63cbfff640 1 -- 172.21.15.40:0/2453259337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2e22bb880 con 0x55d2e22ea2f0 2024-03-18T22:44:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.341+0000 7f63cbfff640 1 --2- 172.21.15.40:0/2453259337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22ea2f0 0x55d2e22fbcb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009230 tx=0x7f63bc049040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d9e65b15dd502e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.341+0000 7f63b1ffb640 1 -- 172.21.15.40:0/2453259337 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc042040 con 0x55d2e22ea2f0 2024-03-18T22:44:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.341+0000 7f63b1ffb640 1 -- 172.21.15.40:0/2453259337 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63bc055800 con 0x55d2e22ea2f0 2024-03-18T22:44:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.341+0000 7f63b1ffb640 1 -- 172.21.15.40:0/2453259337 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0038e0 con 0x55d2e22ea2f0 2024-03-18T22:44:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.342+0000 7f63d53ca740 1 -- 172.21.15.40:0/2453259337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22ea2f0 msgr2=0x55d2e22fbcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.342+0000 7f63d53ca740 1 --2- 172.21.15.40:0/2453259337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22ea2f0 0x55d2e22fbcb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009230 tx=0x7f63bc049040 comp rx=0 tx=0).stop 2024-03-18T22:44:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.342+0000 7f63d53ca740 1 -- 172.21.15.40:0/2453259337 shutdown_connections 2024-03-18T22:44:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.342+0000 7f63d53ca740 1 --2- 172.21.15.40:0/2453259337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22eace0 0x55d2e22e4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.342+0000 7f63d53ca740 1 --2- 172.21.15.40:0/2453259337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22ea2f0 0x55d2e22fbcb0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.342+0000 7f63d53ca740 1 --2- 172.21.15.40:0/2453259337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22e5860 0x55d2e22f4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.342+0000 7f63d53ca740 1 -- 172.21.15.40:0/2453259337 >> 172.21.15.40:0/2453259337 conn(0x55d2e22ebac0 msgr2=0x55d2e22f4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.342+0000 7f63d53ca740 1 -- 172.21.15.40:0/2453259337 shutdown_connections 2024-03-18T22:44:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.342+0000 7f63d53ca740 1 -- 172.21.15.40:0/2453259337 wait complete. 2024-03-18T22:44:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.343+0000 7f63d53ca740 1 -- start start 2024-03-18T22:44:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.343+0000 7f63d53ca740 1 Processor -- start 2024-03-18T22:44:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.343+0000 7f63d53ca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22e5860 0x55d2e22f93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.343+0000 7f63d53ca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22ea2f0 0x55d2e22f9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.343+0000 7f63d53ca740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22eace0 0x55d2e22fa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.343+0000 7f63d53ca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc1a9870 con 0x55d2e22e5860 2024-03-18T22:44:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.343+0000 7f63d53ca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc1a91a0 con 0x55d2e22ea2f0 2024-03-18T22:44:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.343+0000 7f63d53ca740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc1a63e0 con 0x55d2e22eace0 2024-03-18T22:44:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.343+0000 7f63d53ca740 10 client.0 _ll_register_callbacks cb 0x7f63c000ab30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.343+0000 7f63cb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22e5860 0x55d2e22f93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.343+0000 7f63cbfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22ea2f0 0x55d2e22f9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.343+0000 7f63cb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22e5860 0x55d2e22f93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39116/0 (socket says 172.21.15.40:39116) 2024-03-18T22:44:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.343+0000 7f63cb7fe640 1 -- 172.21.15.40:0/1650685312 learned_addr learned my addr 172.21.15.40:0/1650685312 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.343+0000 7f63cb7fe640 1 --2- 172.21.15.40:0/1650685312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22eace0 0x55d2e22fa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63cb7fe640 1 -- 172.21.15.40:0/1650685312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22eace0 msgr2=0x55d2e22fa940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63cb7fe640 1 --2- 172.21.15.40:0/1650685312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22eace0 0x55d2e22fa940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63cb7fe640 1 -- 172.21.15.40:0/1650685312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22ea2f0 msgr2=0x55d2e22f9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63cb7fe640 1 --2- 172.21.15.40:0/1650685312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22ea2f0 0x55d2e22f9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63cb7fe640 1 -- 172.21.15.40:0/1650685312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2e22faf30 con 0x55d2e22e5860 2024-03-18T22:44:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63cbfff640 1 --2- 172.21.15.40:0/1650685312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22ea2f0 0x55d2e22f9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63cb7fe640 1 --2- 172.21.15.40:0/1650685312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22e5860 0x55d2e22f93f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f63c000a520 tx=0x7f63c0010c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779e42240f994d59 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:44:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63ab7fe640 1 -- 172.21.15.40:0/1650685312 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0004b00 con 0x55d2e22e5860 2024-03-18T22:44:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63ab7fe640 1 -- 172.21.15.40:0/1650685312 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f63c0037040 con 0x55d2e22e5860 2024-03-18T22:44:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63d53ca740 1 -- 172.21.15.40:0/1650685312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d2e225b9f0 con 0x55d2e22e5860 2024-03-18T22:44:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63ab7fe640 1 -- 172.21.15.40:0/1650685312 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f63c0004ca0 con 0x55d2e22e5860 2024-03-18T22:44:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63d53ca740 10 client.15918 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63d53ca740 1 -- 172.21.15.40:0/1650685312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d2e2377a10 con 0x55d2e22e5860 2024-03-18T22:44:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63ab7fe640 1 -- 172.21.15.40:0/1650685312 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55d2e225b9f0 con 0x55d2e22e5860 2024-03-18T22:44:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.344+0000 7f63ab7fe640 1 -- 172.21.15.40:0/1650685312 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f63c003a070 con 0x55d2e22e5860 2024-03-18T22:44:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.345+0000 7f63d53ca740 10 client.15918 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.345+0000 7f63d53ca740 10 client.15918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.345+0000 7f63d53ca740 20 client.15918 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.345+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1650685312 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2e22efe50 0x55d2e237a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.345+0000 7f63d53ca740 4 client.15918 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:44:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.345+0000 7f63d53ca740 1 -- 172.21.15.40:0/1650685312 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d2e2255d40 con 0x55d2e22efe50 2024-03-18T22:44:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.345+0000 7f63cbfff640 1 --2- 172.21.15.40:0/1650685312 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2e22efe50 0x55d2e237a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.345+0000 7f63ab7fe640 10 client.15918 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.345+0000 7f63cbfff640 1 --2- 172.21.15.40:0/1650685312 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2e22efe50 0x55d2e237a420 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c07701a2885bea1 server_cookie=b9dc8a229472614f in_seq=0 out_seq=0 2024-03-18T22:44:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.345+0000 7f63ab7fe640 1 -- 172.21.15.40:0/1650685312 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d2e2255d40 con 0x55d2e22efe50 2024-03-18T22:44:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.345+0000 7f63ab7fe640 10 client.15918 handle_command_reply: tid=0 2024-03-18T22:44:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.345+0000 7f63d53ca740 1 client.15918 shutdown 2024-03-18T22:44:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.345+0000 7f63d53ca740 1 -- 172.21.15.40:0/1650685312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22e5860 msgr2=0x55d2e22f93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1650685312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22e5860 0x55d2e22f93f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f63c000a520 tx=0x7f63c0010c90 comp rx=0 tx=0).stop 2024-03-18T22:44:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d53ca740 1 -- 172.21.15.40:0/1650685312 shutdown_connections 2024-03-18T22:44:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1650685312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2e22eace0 0x55d2e22fa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1650685312 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2e22efe50 0x55d2e237a420 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1650685312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2e22ea2f0 0x55d2e22f9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d53ca740 1 --2- 172.21.15.40:0/1650685312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2e22e5860 0x55d2e22f93f0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63ab7fe640 0 client.15918 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d53ca740 1 -- 172.21.15.40:0/1650685312 >> 172.21.15.40:0/1650685312 conn(0x55d2e22ebac0 msgr2=0x55d2e22e8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d53ca740 1 -- 172.21.15.40:0/1650685312 shutdown_connections 2024-03-18T22:44:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d53ca740 1 -- 172.21.15.40:0/1650685312 wait complete. 2024-03-18T22:44:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d53ca740 20 client.15918 trim_cache size 0 max 16384 2024-03-18T22:44:15.353 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:44:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d27df640 1 -- 172.21.15.40:0/2151956429 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63b40a2800 msgr2=0x7f63b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d27df640 1 --2- 172.21.15.40:0/2151956429 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63b40a2800 0x7f63b40a4cc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f63c409bb90 tx=0x7f63c400cee0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d27df640 1 -- 172.21.15.40:0/2151956429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63cc1075e0 msgr2=0x7f63cc100870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d27df640 1 --2- 172.21.15.40:0/2151956429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63cc1075e0 0x7f63cc100870 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f63c400eb10 tx=0x7f63c400c2d0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d27df640 1 -- 172.21.15.40:0/2151956429 shutdown_connections 2024-03-18T22:44:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d27df640 1 --2- 172.21.15.40:0/2151956429 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63b40a2800 0x7f63b40a4cc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d27df640 1 --2- 172.21.15.40:0/2151956429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63cc1075e0 0x7f63cc100870 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d27df640 1 --2- 172.21.15.40:0/2151956429 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63cc104cb0 0x7f63cc100330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d27df640 1 --2- 172.21.15.40:0/2151956429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63cc102380 0x7f63cc101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.346+0000 7f63d27df640 1 -- 172.21.15.40:0/2151956429 >> 172.21.15.40:0/2151956429 conn(0x7f63cc0fbcf0 msgr2=0x7f63cc102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.347+0000 7f63d27df640 1 -- 172.21.15.40:0/2151956429 shutdown_connections 2024-03-18T22:44:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:15.347+0000 7f63d27df640 1 -- 172.21.15.40:0/2151956429 wait complete. 2024-03-18T22:44:16.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:15 smithi040 ceph-mon[62643]: pgmap v399: 129 pgs: 129 active+clean; 114 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 9.1 MiB/s wr, 98 op/s 2024-03-18T22:44:16.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:15 smithi113 ceph-mon[66775]: pgmap v399: 129 pgs: 129 active+clean; 114 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 9.1 MiB/s wr, 98 op/s 2024-03-18T22:44:16.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:15 smithi103 ceph-mon[66377]: pgmap v399: 129 pgs: 129 active+clean; 114 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 9.1 MiB/s wr, 98 op/s 2024-03-18T22:44:16.360 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:44:16.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:44:16.360 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:44:16.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.474+0000 7f217b7f3640 1 -- 172.21.15.40:0/2178654149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2174100720 msgr2=0x7f2174100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.474+0000 7f217b7f3640 1 --2- 172.21.15.40:0/2178654149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2174100720 0x7f2174100b40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2170009860 tx=0x7f217002bb70 comp rx=0 tx=0).stop 2024-03-18T22:44:16.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.474+0000 7f217b7f3640 1 -- 172.21.15.40:0/2178654149 shutdown_connections 2024-03-18T22:44:16.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.474+0000 7f217b7f3640 1 --2- 172.21.15.40:0/2178654149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f21741022d0 0x7f2174108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.474+0000 7f217b7f3640 1 --2- 172.21.15.40:0/2178654149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2174101910 0x7f2174101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.474+0000 7f217b7f3640 1 --2- 172.21.15.40:0/2178654149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2174100720 0x7f2174100b40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.474+0000 7f217b7f3640 1 -- 172.21.15.40:0/2178654149 >> 172.21.15.40:0/2178654149 conn(0x7f21740fbcb0 msgr2=0x7f21740fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:16.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.474+0000 7f217b7f3640 1 -- 172.21.15.40:0/2178654149 shutdown_connections 2024-03-18T22:44:16.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.474+0000 7f217b7f3640 1 -- 172.21.15.40:0/2178654149 wait complete. 2024-03-18T22:44:16.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.474+0000 7f217b7f3640 1 Processor -- start 2024-03-18T22:44:16.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f217b7f3640 1 -- start start 2024-03-18T22:44:16.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f217b7f3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2174100720 0x7f21740694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f217b7f3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2174101910 0x7f217419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f217b7f3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21741022d0 0x7f217419ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f217b7f3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f217410c220 con 0x7f21741022d0 2024-03-18T22:44:16.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f217b7f3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f217410c0a0 con 0x7f2174101910 2024-03-18T22:44:16.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f217b7f3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f217410c3a0 con 0x7f2174100720 2024-03-18T22:44:16.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f217958f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2174100720 0x7f21740694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f217958f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2174100720 0x7f21740694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:46222/0 (socket says 172.21.15.40:46222) 2024-03-18T22:44:16.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f217958f640 1 -- 172.21.15.40:0/3641016189 learned_addr learned my addr 172.21.15.40:0/3641016189 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:16.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f2178d8e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2174101910 0x7f217419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f2179d90640 1 --2- 172.21.15.40:0/3641016189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21741022d0 0x7f217419ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f217958f640 1 -- 172.21.15.40:0/3641016189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2174101910 msgr2=0x7f217419c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f217958f640 1 --2- 172.21.15.40:0/3641016189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2174101910 0x7f217419c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f217958f640 1 -- 172.21.15.40:0/3641016189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21741022d0 msgr2=0x7f217419ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f217958f640 1 --2- 172.21.15.40:0/3641016189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21741022d0 0x7f217419ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f217958f640 1 -- 172.21.15.40:0/3641016189 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f217419f710 con 0x7f2174100720 2024-03-18T22:44:16.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.475+0000 7f2178d8e640 1 --2- 172.21.15.40:0/3641016189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2174101910 0x7f217419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:16.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.476+0000 7f217958f640 1 --2- 172.21.15.40:0/3641016189 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2174100720 0x7f21740694f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2170005940 tx=0x7f21700042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:16.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.476+0000 7f216a7fc640 1 -- 172.21.15.40:0/3641016189 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2170041020 con 0x7f2174100720 2024-03-18T22:44:16.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.476+0000 7f217b7f3640 1 -- 172.21.15.40:0/3641016189 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f217419f940 con 0x7f2174100720 2024-03-18T22:44:16.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.476+0000 7f216a7fc640 1 -- 172.21.15.40:0/3641016189 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21700035f0 con 0x7f2174100720 2024-03-18T22:44:16.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.476+0000 7f216a7fc640 1 -- 172.21.15.40:0/3641016189 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2170039040 con 0x7f2174100720 2024-03-18T22:44:16.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.476+0000 7f217b7f3640 1 -- 172.21.15.40:0/3641016189 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f217419fe30 con 0x7f2174100720 2024-03-18T22:44:16.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.478+0000 7f216a7fc640 1 -- 172.21.15.40:0/3641016189 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2170044020 con 0x7f2174100720 2024-03-18T22:44:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.479+0000 7f216a7fc640 1 --2- 172.21.15.40:0/3641016189 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21600a2800 0x7f21600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.479+0000 7f217e3d6740 1 Processor -- start 2024-03-18T22:44:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.480+0000 7f216a7fc640 1 -- 172.21.15.40:0/3641016189 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f21700f0800 con 0x7f2174100720 2024-03-18T22:44:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.480+0000 7f217e3d6740 1 -- start start 2024-03-18T22:44:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.480+0000 7f217e3d6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7c940 0x5579fce7cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.480+0000 7f217e3d6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce7d330 0x5579fce818b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.480+0000 7f217e3d6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce81f20 0x5579fce843b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.480+0000 7f217e3d6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5579fcde2160 con 0x5579fce7c940 2024-03-18T22:44:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.480+0000 7f217e3d6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5579fcd8ae10 con 0x5579fce81f20 2024-03-18T22:44:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.480+0000 7f217e3d6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5579fcdde200 con 0x5579fce7d330 2024-03-18T22:44:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.480+0000 7f2178d8e640 1 --2- 172.21.15.40:0/3641016189 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21600a2800 0x7f21600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.480+0000 7f217958f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7c940 0x5579fce7cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.480+0000 7f217958f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7c940 0x5579fce7cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39142/0 (socket says 172.21.15.40:39142) 2024-03-18T22:44:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.480+0000 7f217958f640 1 -- 172.21.15.40:0/34976559 learned_addr learned my addr 172.21.15.40:0/34976559 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.481+0000 7f217958f640 1 -- 172.21.15.40:0/34976559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce7d330 msgr2=0x5579fce818b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:44:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.481+0000 7f217958f640 1 --2- 172.21.15.40:0/34976559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce7d330 0x5579fce818b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.481+0000 7f217958f640 1 -- 172.21.15.40:0/34976559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce81f20 msgr2=0x5579fce843b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:44:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.481+0000 7f217958f640 1 --2- 172.21.15.40:0/34976559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce81f20 0x5579fce843b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.481+0000 7f217958f640 1 -- 172.21.15.40:0/34976559 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579fcbbe7e0 con 0x5579fce7c940 2024-03-18T22:44:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.481+0000 7f2178d8e640 1 --2- 172.21.15.40:0/3641016189 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21600a2800 0x7f21600a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2174069260 tx=0x7f2164009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.481+0000 7f217958f640 1 --2- 172.21.15.40:0/34976559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7c940 0x5579fce7cd60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f21700bcd40 tx=0x7f21700bc9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d705c1e819e7321c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:16.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.481+0000 7f215f7fe640 1 -- 172.21.15.40:0/34976559 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21700bd4b0 con 0x5579fce7c940 2024-03-18T22:44:16.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.481+0000 7f215f7fe640 1 -- 172.21.15.40:0/34976559 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21700c0030 con 0x5579fce7c940 2024-03-18T22:44:16.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.481+0000 7f215f7fe640 1 -- 172.21.15.40:0/34976559 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2170003b10 con 0x5579fce7c940 2024-03-18T22:44:16.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.482+0000 7f217e3d6740 1 -- 172.21.15.40:0/34976559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7c940 msgr2=0x5579fce7cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.482+0000 7f217e3d6740 1 --2- 172.21.15.40:0/34976559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7c940 0x5579fce7cd60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f21700bcd40 tx=0x7f21700bc9b0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.482+0000 7f217e3d6740 1 -- 172.21.15.40:0/34976559 shutdown_connections 2024-03-18T22:44:16.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.482+0000 7f217e3d6740 1 --2- 172.21.15.40:0/34976559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce81f20 0x5579fce843b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.482+0000 7f217e3d6740 1 --2- 172.21.15.40:0/34976559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce7d330 0x5579fce818b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.482+0000 7f217e3d6740 1 --2- 172.21.15.40:0/34976559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7c940 0x5579fce7cd60 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.482+0000 7f217e3d6740 1 -- 172.21.15.40:0/34976559 >> 172.21.15.40:0/34976559 conn(0x5579fce78810 msgr2=0x5579fce78c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:16.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.482+0000 7f217e3d6740 1 -- 172.21.15.40:0/34976559 shutdown_connections 2024-03-18T22:44:16.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.482+0000 7f217e3d6740 1 -- 172.21.15.40:0/34976559 wait complete. 2024-03-18T22:44:16.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.482+0000 7f217e3d6740 1 -- start start 2024-03-18T22:44:16.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f217e3d6740 1 Processor -- start 2024-03-18T22:44:16.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f217e3d6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce7c940 0x5579fce875f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f217e3d6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce7d330 0x5579fce87b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f217e3d6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce81f20 0x5579fce8eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f217e3d6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5579fce84c70 con 0x5579fce81f20 2024-03-18T22:44:16.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f217e3d6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5579fcd9e1a0 con 0x5579fce7c940 2024-03-18T22:44:16.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f217e3d6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5579fce85f70 con 0x5579fce7d330 2024-03-18T22:44:16.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f217e3d6740 10 client.0 _ll_register_callbacks cb 0x5579fcaf5c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:16.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f2179d90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce81f20 0x5579fce8eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f2179d90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce81f20 0x5579fce8eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39152/0 (socket says 172.21.15.40:39152) 2024-03-18T22:44:16.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f2179d90640 1 -- 172.21.15.40:0/713009003 learned_addr learned my addr 172.21.15.40:0/713009003 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:16.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f2178d8e640 1 --2- 172.21.15.40:0/713009003 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce7d330 0x5579fce87b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f2179d90640 1 -- 172.21.15.40:0/713009003 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce7d330 msgr2=0x5579fce87b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:16.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f2179d90640 1 --2- 172.21.15.40:0/713009003 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce7d330 0x5579fce87b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f2179d90640 1 -- 172.21.15.40:0/713009003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce7c940 msgr2=0x5579fce875f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:16.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f2179d90640 1 --2- 172.21.15.40:0/713009003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce7c940 0x5579fce875f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.483+0000 7f2179d90640 1 -- 172.21.15.40:0/713009003 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579fcc1ef70 con 0x5579fce81f20 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.484+0000 7f2179d90640 1 --2- 172.21.15.40:0/713009003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce81f20 0x5579fce8eb70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f216c0118c0 tx=0x7f216c011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbaa2955525980e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.484+0000 7f215cff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.484+0000 7f215cff9640 1 -- 172.21.15.40:0/713009003 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c003e40 con 0x5579fce81f20 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.484+0000 7f217e3d6740 1 -- 172.21.15.40:0/713009003 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21700c0030 con 0x5579fce81f20 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.484+0000 7f215cff9640 1 -- 172.21.15.40:0/713009003 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f216c004b20 con 0x5579fce81f20 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.484+0000 7f215cff9640 1 -- 172.21.15.40:0/713009003 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f216c004d60 con 0x5579fce81f20 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.484+0000 7f217e3d6740 10 client.15930 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.484+0000 7f217e3d6740 1 -- 172.21.15.40:0/713009003 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5579fcab6170 con 0x5579fce81f20 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.484+0000 7f215cff9640 1 -- 172.21.15.40:0/713009003 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21700c0030 con 0x5579fce81f20 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.484+0000 7f215cff9640 1 -- 172.21.15.40:0/713009003 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f216c018070 con 0x5579fce81f20 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.485+0000 7f217e3d6740 10 client.15930 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.485+0000 7f217e3d6740 10 client.15930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.485+0000 7f217e3d6740 20 client.15930 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.485+0000 7f217e3d6740 1 --2- 172.21.15.40:0/713009003 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5579fce935f0 0x5579fce95a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.485+0000 7f217e3d6740 4 client.15930 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:16.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.485+0000 7f217e3d6740 1 -- 172.21.15.40:0/713009003 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579fcdef6a0 con 0x5579fce935f0 2024-03-18T22:44:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.485+0000 7f217958f640 1 --2- 172.21.15.40:0/713009003 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5579fce935f0 0x5579fce95a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.485+0000 7f215cff9640 10 client.15930 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.485+0000 7f217958f640 1 --2- 172.21.15.40:0/713009003 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5579fce935f0 0x5579fce95a90 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5107bef662f66ee server_cookie=f6b89481c55d42ca in_seq=0 out_seq=0 2024-03-18T22:44:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.486+0000 7f215cff9640 1 -- 172.21.15.40:0/713009003 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5579fcdef6a0 con 0x5579fce935f0 2024-03-18T22:44:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.486+0000 7f215cff9640 10 client.15930 handle_command_reply: tid=0 2024-03-18T22:44:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.486+0000 7f217e3d6740 1 client.15930 shutdown 2024-03-18T22:44:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.486+0000 7f217e3d6740 1 -- 172.21.15.40:0/713009003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce81f20 msgr2=0x5579fce8eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.486+0000 7f217e3d6740 1 --2- 172.21.15.40:0/713009003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce81f20 0x5579fce8eb70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f216c0118c0 tx=0x7f216c011d80 comp rx=0 tx=0).stop 2024-03-18T22:44:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.486+0000 7f217e3d6740 1 -- 172.21.15.40:0/713009003 shutdown_connections 2024-03-18T22:44:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.486+0000 7f217e3d6740 1 --2- 172.21.15.40:0/713009003 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce81f20 0x5579fce8eb70 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.486+0000 7f217e3d6740 1 --2- 172.21.15.40:0/713009003 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce7d330 0x5579fce87b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.486+0000 7f217e3d6740 1 --2- 172.21.15.40:0/713009003 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5579fce935f0 0x5579fce95a90 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.486+0000 7f215cff9640 0 client.15930 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.486+0000 7f217e3d6740 1 --2- 172.21.15.40:0/713009003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce7c940 0x5579fce875f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.486+0000 7f217e3d6740 1 -- 172.21.15.40:0/713009003 >> 172.21.15.40:0/713009003 conn(0x5579fce78810 msgr2=0x5579fce857f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.487+0000 7f217e3d6740 1 -- 172.21.15.40:0/713009003 shutdown_connections 2024-03-18T22:44:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.487+0000 7f217e3d6740 1 -- 172.21.15.40:0/713009003 wait complete. 2024-03-18T22:44:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.487+0000 7f217e3d6740 20 client.15930 trim_cache size 0 max 16384 2024-03-18T22:44:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.515+0000 7f217e3d6740 1 Processor -- start 2024-03-18T22:44:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.515+0000 7f217e3d6740 1 -- start start 2024-03-18T22:44:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.515+0000 7f217e3d6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce78810 0x5579fce85670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.515+0000 7f217e3d6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce7c940 0x5579fce8eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.515+0000 7f217e3d6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7d330 0x5579fce9a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.515+0000 7f217e3d6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21741a8360 con 0x5579fce7d330 2024-03-18T22:44:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.515+0000 7f217e3d6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f21741a9ed0 con 0x5579fce7c940 2024-03-18T22:44:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.515+0000 7f217e3d6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f21741a8a30 con 0x5579fce78810 2024-03-18T22:44:16.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.515+0000 7f2179d90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7d330 0x5579fce9a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.515+0000 7f2178d8e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce7c940 0x5579fce8eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.515+0000 7f2179d90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7d330 0x5579fce9a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39164/0 (socket says 172.21.15.40:39164) 2024-03-18T22:44:16.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.515+0000 7f2179d90640 1 -- 172.21.15.40:0/4141559167 learned_addr learned my addr 172.21.15.40:0/4141559167 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:16.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.515+0000 7f217958f640 1 --2- 172.21.15.40:0/4141559167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce78810 0x5579fce85670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.516+0000 7f2179d90640 1 -- 172.21.15.40:0/4141559167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce78810 msgr2=0x5579fce85670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:16.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.516+0000 7f2179d90640 1 --2- 172.21.15.40:0/4141559167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce78810 0x5579fce85670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.516+0000 7f2179d90640 1 -- 172.21.15.40:0/4141559167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce7c940 msgr2=0x5579fce8eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:16.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.516+0000 7f2179d90640 1 --2- 172.21.15.40:0/4141559167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce7c940 0x5579fce8eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.516+0000 7f2179d90640 1 -- 172.21.15.40:0/4141559167 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579fce87cd0 con 0x5579fce7d330 2024-03-18T22:44:16.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.516+0000 7f2179d90640 1 --2- 172.21.15.40:0/4141559167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7d330 0x5579fce9a350 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f216c011890 tx=0x7f216c004ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbbecd22e824097b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:16.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.516+0000 7f215effd640 1 -- 172.21.15.40:0/4141559167 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c02b020 con 0x5579fce7d330 2024-03-18T22:44:16.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.516+0000 7f215effd640 1 -- 172.21.15.40:0/4141559167 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f216c019d40 con 0x5579fce7d330 2024-03-18T22:44:16.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.516+0000 7f215effd640 1 -- 172.21.15.40:0/4141559167 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c027480 con 0x5579fce7d330 2024-03-18T22:44:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.517+0000 7f217e3d6740 1 -- 172.21.15.40:0/4141559167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7d330 msgr2=0x5579fce9a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.517+0000 7f217e3d6740 1 --2- 172.21.15.40:0/4141559167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7d330 0x5579fce9a350 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f216c011890 tx=0x7f216c004ca0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.517+0000 7f217e3d6740 1 -- 172.21.15.40:0/4141559167 shutdown_connections 2024-03-18T22:44:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.517+0000 7f217e3d6740 1 --2- 172.21.15.40:0/4141559167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7d330 0x5579fce9a350 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.517+0000 7f217e3d6740 1 --2- 172.21.15.40:0/4141559167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce7c940 0x5579fce8eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.517+0000 7f217e3d6740 1 --2- 172.21.15.40:0/4141559167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce78810 0x5579fce85670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.517+0000 7f217e3d6740 1 -- 172.21.15.40:0/4141559167 >> 172.21.15.40:0/4141559167 conn(0x5579fce7db90 msgr2=0x5579fce84f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.517+0000 7f217e3d6740 1 -- 172.21.15.40:0/4141559167 shutdown_connections 2024-03-18T22:44:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.517+0000 7f217e3d6740 1 -- 172.21.15.40:0/4141559167 wait complete. 2024-03-18T22:44:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.517+0000 7f217e3d6740 1 -- start start 2024-03-18T22:44:16.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.517+0000 7f217e3d6740 1 Processor -- start 2024-03-18T22:44:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f217e3d6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce78810 0x5579fce8e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f217e3d6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce7c940 0x5579fce8cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f217e3d6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7d330 0x5579fce8d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f217e3d6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21741a97d0 con 0x5579fce7d330 2024-03-18T22:44:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f217e3d6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f21741a9100 con 0x5579fce7c940 2024-03-18T22:44:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f217e3d6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f21741a6340 con 0x5579fce78810 2024-03-18T22:44:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f217e3d6740 10 client.0 _ll_register_callbacks cb 0x7f216c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f2179d90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7d330 0x5579fce8d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f217958f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce78810 0x5579fce8e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f2179d90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7d330 0x5579fce8d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39178/0 (socket says 172.21.15.40:39178) 2024-03-18T22:44:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f2179d90640 1 -- 172.21.15.40:0/3411123041 learned_addr learned my addr 172.21.15.40:0/3411123041 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f2178d8e640 1 --2- 172.21.15.40:0/3411123041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce7c940 0x5579fce8cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f217958f640 1 -- 172.21.15.40:0/3411123041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce7c940 msgr2=0x5579fce8cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:16.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f217958f640 1 --2- 172.21.15.40:0/3411123041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce7c940 0x5579fce8cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f217958f640 1 -- 172.21.15.40:0/3411123041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7d330 msgr2=0x5579fce8d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f217958f640 1 --2- 172.21.15.40:0/3411123041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7d330 0x5579fce8d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f217958f640 1 -- 172.21.15.40:0/3411123041 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579fce74160 con 0x5579fce78810 2024-03-18T22:44:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f217958f640 1 --2- 172.21.15.40:0/3411123041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce78810 0x5579fce8e920 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f21700b9780 tx=0x7f217002bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a492ce5b320aed4e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.518+0000 7f215cff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:44:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.519+0000 7f215cff9640 1 -- 172.21.15.40:0/3411123041 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21700c0f00 con 0x5579fce78810 2024-03-18T22:44:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.519+0000 7f217e3d6740 1 -- 172.21.15.40:0/3411123041 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f216c019d40 con 0x5579fce78810 2024-03-18T22:44:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.519+0000 7f215cff9640 1 -- 172.21.15.40:0/3411123041 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2170003b10 con 0x5579fce78810 2024-03-18T22:44:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.519+0000 7f215cff9640 1 -- 172.21.15.40:0/3411123041 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f21700c5550 con 0x5579fce78810 2024-03-18T22:44:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.519+0000 7f217e3d6740 10 client.25780 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.519+0000 7f217e3d6740 1 -- 172.21.15.40:0/3411123041 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5579fcefa740 con 0x5579fce78810 2024-03-18T22:44:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.519+0000 7f215cff9640 1 -- 172.21.15.40:0/3411123041 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f216c019d40 con 0x5579fce78810 2024-03-18T22:44:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.519+0000 7f215cff9640 1 -- 172.21.15.40:0/3411123041 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f217003c030 con 0x5579fce78810 2024-03-18T22:44:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.519+0000 7f217e3d6740 10 client.25780 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:16.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.519+0000 7f217e3d6740 10 client.25780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.519+0000 7f217e3d6740 20 client.25780 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.519+0000 7f217e3d6740 1 --2- 172.21.15.40:0/3411123041 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5579fce81f20 0x5579fcefcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.519+0000 7f217e3d6740 4 client.25780 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:44:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.519+0000 7f217e3d6740 1 -- 172.21.15.40:0/3411123041 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5579fcde99f0 con 0x5579fce81f20 2024-03-18T22:44:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.520+0000 7f2178d8e640 1 --2- 172.21.15.40:0/3411123041 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5579fce81f20 0x5579fcefcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.520+0000 7f215cff9640 10 client.25780 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.520+0000 7f2178d8e640 1 --2- 172.21.15.40:0/3411123041 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5579fce81f20 0x5579fcefcdd0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53adea6e2f3d6a3a server_cookie=5d89e94e1c4202bd in_seq=0 out_seq=0 2024-03-18T22:44:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.521+0000 7f215cff9640 1 -- 172.21.15.40:0/3411123041 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5579fcde99f0 con 0x5579fce81f20 2024-03-18T22:44:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.521+0000 7f215cff9640 10 client.25780 handle_command_reply: tid=0 2024-03-18T22:44:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.521+0000 7f217e3d6740 1 client.25780 shutdown 2024-03-18T22:44:16.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.521+0000 7f217e3d6740 1 -- 172.21.15.40:0/3411123041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce78810 msgr2=0x5579fce8e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.521+0000 7f217e3d6740 1 --2- 172.21.15.40:0/3411123041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce78810 0x5579fce8e920 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f21700b9780 tx=0x7f217002bb70 comp rx=0 tx=0).stop 2024-03-18T22:44:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.521+0000 7f217e3d6740 1 -- 172.21.15.40:0/3411123041 shutdown_connections 2024-03-18T22:44:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.521+0000 7f217e3d6740 1 --2- 172.21.15.40:0/3411123041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579fce7d330 0x5579fce8d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.521+0000 7f217e3d6740 1 --2- 172.21.15.40:0/3411123041 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5579fce81f20 0x5579fcefcdd0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.521+0000 7f217e3d6740 1 --2- 172.21.15.40:0/3411123041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579fce7c940 0x5579fce8cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.521+0000 7f217e3d6740 1 --2- 172.21.15.40:0/3411123041 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5579fce78810 0x5579fce8e920 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.521+0000 7f215cff9640 0 client.25780 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.521+0000 7f217e3d6740 1 -- 172.21.15.40:0/3411123041 >> 172.21.15.40:0/3411123041 conn(0x5579fce7db90 msgr2=0x5579fce77b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.521+0000 7f217e3d6740 1 -- 172.21.15.40:0/3411123041 shutdown_connections 2024-03-18T22:44:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.521+0000 7f217e3d6740 1 -- 172.21.15.40:0/3411123041 wait complete. 2024-03-18T22:44:16.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.521+0000 7f217e3d6740 20 client.25780 trim_cache size 0 max 16384 2024-03-18T22:44:16.529 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:44:16.529 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:44:16.529 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "3bf386c0-f31e-4218-ac60-84ae9d174304", 2024-03-18T22:44:16.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:44:16.529 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:44:16.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.522+0000 7f217b7f3640 1 -- 172.21.15.40:0/3641016189 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21600a2800 msgr2=0x7f21600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.522+0000 7f217b7f3640 1 --2- 172.21.15.40:0/3641016189 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21600a2800 0x7f21600a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2174069260 tx=0x7f2164009260 comp rx=0 tx=0).stop 2024-03-18T22:44:16.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.522+0000 7f217b7f3640 1 -- 172.21.15.40:0/3641016189 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2174100720 msgr2=0x7f21740694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.522+0000 7f217b7f3640 1 --2- 172.21.15.40:0/3641016189 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2174100720 0x7f21740694f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2170005940 tx=0x7f21700042f0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.522+0000 7f217b7f3640 1 -- 172.21.15.40:0/3641016189 shutdown_connections 2024-03-18T22:44:16.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.522+0000 7f217b7f3640 1 --2- 172.21.15.40:0/3641016189 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21600a2800 0x7f21600a4cc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.522+0000 7f217b7f3640 1 --2- 172.21.15.40:0/3641016189 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21741022d0 0x7f217419ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.522+0000 7f217b7f3640 1 --2- 172.21.15.40:0/3641016189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2174101910 0x7f217419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.522+0000 7f217b7f3640 1 --2- 172.21.15.40:0/3641016189 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2174100720 0x7f21740694f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.522+0000 7f217b7f3640 1 -- 172.21.15.40:0/3641016189 >> 172.21.15.40:0/3641016189 conn(0x7f21740fbcb0 msgr2=0x7f2174104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:16.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.522+0000 7f217b7f3640 1 -- 172.21.15.40:0/3641016189 shutdown_connections 2024-03-18T22:44:16.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.522+0000 7f217b7f3640 1 -- 172.21.15.40:0/3641016189 wait complete. 2024-03-18T22:44:16.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:44:16.536 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:44:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.653+0000 7f6b64d87640 1 -- 172.21.15.40:0/2313837419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b60100720 msgr2=0x7f6b60100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.653+0000 7f6b64d87640 1 --2- 172.21.15.40:0/2313837419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b60100720 0x7f6b60100b40 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6b50009900 tx=0x7f6b5002bc30 comp rx=0 tx=0).stop 2024-03-18T22:44:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.653+0000 7f6b64d87640 1 -- 172.21.15.40:0/2313837419 shutdown_connections 2024-03-18T22:44:16.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.653+0000 7f6b64d87640 1 --2- 172.21.15.40:0/2313837419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b601022d0 0x7f6b60108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.653+0000 7f6b64d87640 1 --2- 172.21.15.40:0/2313837419 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b60101910 0x7f6b60101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.653+0000 7f6b64d87640 1 --2- 172.21.15.40:0/2313837419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b60100720 0x7f6b60100b40 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.653+0000 7f6b64d87640 1 -- 172.21.15.40:0/2313837419 >> 172.21.15.40:0/2313837419 conn(0x7f6b600fbcb0 msgr2=0x7f6b600fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.653+0000 7f6b64d87640 1 -- 172.21.15.40:0/2313837419 shutdown_connections 2024-03-18T22:44:16.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.653+0000 7f6b64d87640 1 -- 172.21.15.40:0/2313837419 wait complete. 2024-03-18T22:44:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.654+0000 7f6b64d87640 1 Processor -- start 2024-03-18T22:44:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.654+0000 7f6b64d87640 1 -- start start 2024-03-18T22:44:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.654+0000 7f6b64d87640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b60100720 0x7f6b600694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.654+0000 7f6b64d87640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b60101910 0x7f6b6019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b64d87640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b601022d0 0x7f6b6019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b64d87640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6010c240 con 0x7f6b60101910 2024-03-18T22:44:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b64d87640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6010c0c0 con 0x7f6b601022d0 2024-03-18T22:44:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b64d87640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6010c3c0 con 0x7f6b60100720 2024-03-18T22:44:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b57fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b60101910 0x7f6b6019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b57fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b60101910 0x7f6b6019c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39188/0 (socket says 172.21.15.40:39188) 2024-03-18T22:44:16.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b57fff640 1 -- 172.21.15.40:0/2627667716 learned_addr learned my addr 172.21.15.40:0/2627667716 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:16.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b5e59c640 1 --2- 172.21.15.40:0/2627667716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b60100720 0x7f6b600694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b5ed9d640 1 --2- 172.21.15.40:0/2627667716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b601022d0 0x7f6b6019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b57fff640 1 -- 172.21.15.40:0/2627667716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b60100720 msgr2=0x7f6b600694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b57fff640 1 --2- 172.21.15.40:0/2627667716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b60100720 0x7f6b600694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b57fff640 1 -- 172.21.15.40:0/2627667716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b601022d0 msgr2=0x7f6b6019ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b57fff640 1 --2- 172.21.15.40:0/2627667716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b601022d0 0x7f6b6019ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b57fff640 1 -- 172.21.15.40:0/2627667716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6019f730 con 0x7f6b60101910 2024-03-18T22:44:16.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b5e59c640 1 --2- 172.21.15.40:0/2627667716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b60100720 0x7f6b600694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:16.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b57fff640 1 --2- 172.21.15.40:0/2627667716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b60101910 0x7f6b6019c800 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6b4800e9f0 tx=0x7f6b4800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:16.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b577fe640 1 -- 172.21.15.40:0/2627667716 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4800cd80 con 0x7f6b60101910 2024-03-18T22:44:16.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b64d87640 1 -- 172.21.15.40:0/2627667716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b6019f9c0 con 0x7f6b60101910 2024-03-18T22:44:16.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b577fe640 1 -- 172.21.15.40:0/2627667716 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b480054d0 con 0x7f6b60101910 2024-03-18T22:44:16.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.655+0000 7f6b577fe640 1 -- 172.21.15.40:0/2627667716 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b48010490 con 0x7f6b60101910 2024-03-18T22:44:16.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.656+0000 7f6b64d87640 1 -- 172.21.15.40:0/2627667716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b6019fec0 con 0x7f6b60101910 2024-03-18T22:44:16.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.658+0000 7f6b577fe640 1 -- 172.21.15.40:0/2627667716 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6b48005020 con 0x7f6b60101910 2024-03-18T22:44:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.658+0000 7f6b577fe640 1 --2- 172.21.15.40:0/2627667716 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6b4c0a2800 0x7f6b4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b67970740 1 Processor -- start 2024-03-18T22:44:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b5e59c640 1 --2- 172.21.15.40:0/2627667716 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6b4c0a2800 0x7f6b4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b577fe640 1 -- 172.21.15.40:0/2627667716 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6b480cf1b0 con 0x7f6b60101910 2024-03-18T22:44:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b67970740 1 -- start start 2024-03-18T22:44:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b67970740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3ea50 0x558a23b3ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b67970740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b3f440 0x558a23b43f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b67970740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b445b0 0x558a23b46a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b67970740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558a23aa3500 con 0x558a23b3f440 2024-03-18T22:44:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b67970740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558a23a4bbe0 con 0x558a23b3ea50 2024-03-18T22:44:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b67970740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x558a23a9f5a0 con 0x558a23b445b0 2024-03-18T22:44:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b57fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b3f440 0x558a23b43f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b5e59c640 1 --2- 172.21.15.40:0/2627667716 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6b4c0a2800 0x7f6b4c0a4cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6b50005ab0 tx=0x7f6b50005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:16.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b57fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b3f440 0x558a23b43f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39190/0 (socket says 172.21.15.40:39190) 2024-03-18T22:44:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b57fff640 1 -- 172.21.15.40:0/837656068 learned_addr learned my addr 172.21.15.40:0/837656068 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b5e59c640 1 --2- 172.21.15.40:0/837656068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3ea50 0x558a23b3ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.659+0000 7f6b5ed9d640 1 --2- 172.21.15.40:0/837656068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b445b0 0x558a23b46a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.660+0000 7f6b57fff640 1 -- 172.21.15.40:0/837656068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b445b0 msgr2=0x558a23b46a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.660+0000 7f6b57fff640 1 --2- 172.21.15.40:0/837656068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b445b0 0x558a23b46a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.660+0000 7f6b57fff640 1 -- 172.21.15.40:0/837656068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3ea50 msgr2=0x558a23b3ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.660+0000 7f6b57fff640 1 --2- 172.21.15.40:0/837656068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3ea50 0x558a23b3ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.660+0000 7f6b57fff640 1 -- 172.21.15.40:0/837656068 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a23a6d000 con 0x558a23b3f440 2024-03-18T22:44:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.660+0000 7f6b5e59c640 1 --2- 172.21.15.40:0/837656068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3ea50 0x558a23b3ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.660+0000 7f6b57fff640 1 --2- 172.21.15.40:0/837656068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b3f440 0x558a23b43f40 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6b48097bc0 tx=0x7f6b4809b180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7a0ab9bac6c22d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.660+0000 7f6b54ff9640 1 -- 172.21.15.40:0/837656068 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b480b4070 con 0x558a23b3f440 2024-03-18T22:44:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.660+0000 7f6b54ff9640 1 -- 172.21.15.40:0/837656068 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b4809be60 con 0x558a23b3f440 2024-03-18T22:44:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.660+0000 7f6b54ff9640 1 -- 172.21.15.40:0/837656068 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b480046c0 con 0x558a23b3f440 2024-03-18T22:44:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.661+0000 7f6b67970740 1 -- 172.21.15.40:0/837656068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b3f440 msgr2=0x558a23b43f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.661+0000 7f6b67970740 1 --2- 172.21.15.40:0/837656068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b3f440 0x558a23b43f40 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6b48097bc0 tx=0x7f6b4809b180 comp rx=0 tx=0).stop 2024-03-18T22:44:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.661+0000 7f6b67970740 1 -- 172.21.15.40:0/837656068 shutdown_connections 2024-03-18T22:44:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.661+0000 7f6b67970740 1 --2- 172.21.15.40:0/837656068 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b445b0 0x558a23b46a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.661+0000 7f6b67970740 1 --2- 172.21.15.40:0/837656068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b3f440 0x558a23b43f40 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.661+0000 7f6b67970740 1 --2- 172.21.15.40:0/837656068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3ea50 0x558a23b3ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.661+0000 7f6b67970740 1 -- 172.21.15.40:0/837656068 >> 172.21.15.40:0/837656068 conn(0x558a23b39fc0 msgr2=0x558a23b3a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.661+0000 7f6b67970740 1 -- 172.21.15.40:0/837656068 shutdown_connections 2024-03-18T22:44:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.661+0000 7f6b67970740 1 -- 172.21.15.40:0/837656068 wait complete. 2024-03-18T22:44:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.661+0000 7f6b67970740 1 -- start start 2024-03-18T22:44:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.661+0000 7f6b67970740 1 Processor -- start 2024-03-18T22:44:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.661+0000 7f6b67970740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b3ea50 0x558a23b48f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.661+0000 7f6b67970740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3f440 0x558a23b49450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.661+0000 7f6b67970740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b445b0 0x558a23b50490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.661+0000 7f6b67970740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558a23b47560 con 0x558a23b445b0 2024-03-18T22:44:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b67970740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558a23a5f160 con 0x558a23b3f440 2024-03-18T22:44:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b67970740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x558a23b476e0 con 0x558a23b3ea50 2024-03-18T22:44:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b67970740 10 client.0 _ll_register_callbacks cb 0x558a23a9ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b5ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b445b0 0x558a23b50490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b5e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b3ea50 0x558a23b48f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b5e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b3ea50 0x558a23b48f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:46298/0 (socket says 172.21.15.40:46298) 2024-03-18T22:44:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b5ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b445b0 0x558a23b50490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39206/0 (socket says 172.21.15.40:39206) 2024-03-18T22:44:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b5e59c640 1 -- 172.21.15.40:0/209726793 learned_addr learned my addr 172.21.15.40:0/209726793 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b57fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3f440 0x558a23b49450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b5e59c640 1 -- 172.21.15.40:0/209726793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3f440 msgr2=0x558a23b49450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b5e59c640 1 --2- 172.21.15.40:0/209726793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3f440 0x558a23b49450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b5e59c640 1 -- 172.21.15.40:0/209726793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b445b0 msgr2=0x558a23b50490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b5e59c640 1 --2- 172.21.15.40:0/209726793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b445b0 0x558a23b50490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b5e59c640 1 -- 172.21.15.40:0/209726793 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a238817d0 con 0x558a23b3ea50 2024-03-18T22:44:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b5e59c640 1 --2- 172.21.15.40:0/209726793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b3ea50 0x558a23b48f10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6b60101770 tx=0x7f6b50009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=263a4c0461c0afaa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:44:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b45ffb640 1 -- 172.21.15.40:0/209726793 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5004f070 con 0x558a23b3ea50 2024-03-18T22:44:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b67970740 1 -- 172.21.15.40:0/209726793 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b4809be60 con 0x558a23b3ea50 2024-03-18T22:44:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.662+0000 7f6b45ffb640 1 -- 172.21.15.40:0/209726793 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6b50004990 con 0x558a23b3ea50 2024-03-18T22:44:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.663+0000 7f6b45ffb640 1 -- 172.21.15.40:0/209726793 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6b5005b050 con 0x558a23b3ea50 2024-03-18T22:44:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.663+0000 7f6b67970740 10 client.25792 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.663+0000 7f6b67970740 1 -- 172.21.15.40:0/209726793 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558a237b73f0 con 0x558a23b3ea50 2024-03-18T22:44:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.663+0000 7f6b45ffb640 1 -- 172.21.15.40:0/209726793 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b4809be60 con 0x558a23b3ea50 2024-03-18T22:44:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.663+0000 7f6b45ffb640 1 -- 172.21.15.40:0/209726793 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f6b5004a070 con 0x558a23b3ea50 2024-03-18T22:44:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.663+0000 7f6b67970740 10 client.25792 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.663+0000 7f6b67970740 10 client.25792 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.663+0000 7f6b67970740 20 client.25792 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.663+0000 7f6b67970740 1 --2- 172.21.15.40:0/209726793 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a23b54d10 0x558a23b571b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.663+0000 7f6b67970740 4 client.25792 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.663+0000 7f6b67970740 1 -- 172.21.15.40:0/209726793 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a23ab0a40 con 0x558a23b54d10 2024-03-18T22:44:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.663+0000 7f6b57fff640 1 --2- 172.21.15.40:0/209726793 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a23b54d10 0x558a23b571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b57fff640 1 --2- 172.21.15.40:0/209726793 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a23b54d10 0x558a23b571b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c42967cfe433ed6 server_cookie=f1e203ad933aa2e6 in_seq=0 out_seq=0 2024-03-18T22:44:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b45ffb640 10 client.25792 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b45ffb640 1 -- 172.21.15.40:0/209726793 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558a23ab0a40 con 0x558a23b54d10 2024-03-18T22:44:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b45ffb640 10 client.25792 handle_command_reply: tid=0 2024-03-18T22:44:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b67970740 1 client.25792 shutdown 2024-03-18T22:44:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b67970740 1 -- 172.21.15.40:0/209726793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b3ea50 msgr2=0x558a23b48f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b67970740 1 --2- 172.21.15.40:0/209726793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b3ea50 0x558a23b48f10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6b60101770 tx=0x7f6b50009260 comp rx=0 tx=0).stop 2024-03-18T22:44:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b67970740 1 -- 172.21.15.40:0/209726793 shutdown_connections 2024-03-18T22:44:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b67970740 1 --2- 172.21.15.40:0/209726793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b445b0 0x558a23b50490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b67970740 1 --2- 172.21.15.40:0/209726793 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a23b54d10 0x558a23b571b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b67970740 1 --2- 172.21.15.40:0/209726793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3f440 0x558a23b49450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b67970740 1 --2- 172.21.15.40:0/209726793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b3ea50 0x558a23b48f10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b45ffb640 0 client.25792 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b67970740 1 -- 172.21.15.40:0/209726793 >> 172.21.15.40:0/209726793 conn(0x558a23b39fc0 msgr2=0x558a23b391e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b67970740 1 -- 172.21.15.40:0/209726793 shutdown_connections 2024-03-18T22:44:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.664+0000 7f6b67970740 1 -- 172.21.15.40:0/209726793 wait complete. 2024-03-18T22:44:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.665+0000 7f6b67970740 20 client.25792 trim_cache size 0 max 16384 2024-03-18T22:44:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.692+0000 7f6b67970740 1 Processor -- start 2024-03-18T22:44:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.692+0000 7f6b67970740 1 -- start start 2024-03-18T22:44:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.692+0000 7f6b67970740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b39fc0 0x558a23b3d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.692+0000 7f6b67970740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b3ea50 0x558a23b5ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b67970740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3f440 0x558a23b5bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b67970740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601a8400 con 0x558a23b39fc0 2024-03-18T22:44:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b67970740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601a9f70 con 0x558a23b3f440 2024-03-18T22:44:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b67970740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601a8ad0 con 0x558a23b3ea50 2024-03-18T22:44:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b5e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b39fc0 0x558a23b3d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b5e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b39fc0 0x558a23b3d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39208/0 (socket says 172.21.15.40:39208) 2024-03-18T22:44:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b57fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b3ea50 0x558a23b5ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b5e59c640 1 -- 172.21.15.40:0/2571070808 learned_addr learned my addr 172.21.15.40:0/2571070808 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b5ed9d640 1 --2- 172.21.15.40:0/2571070808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3f440 0x558a23b5bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b57fff640 1 -- 172.21.15.40:0/2571070808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3f440 msgr2=0x558a23b5bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b57fff640 1 --2- 172.21.15.40:0/2571070808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3f440 0x558a23b5bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b57fff640 1 -- 172.21.15.40:0/2571070808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b39fc0 msgr2=0x558a23b3d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b57fff640 1 --2- 172.21.15.40:0/2571070808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b39fc0 0x558a23b3d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b57fff640 1 -- 172.21.15.40:0/2571070808 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a237770f0 con 0x558a23b3ea50 2024-03-18T22:44:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b57fff640 1 --2- 172.21.15.40:0/2571070808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b3ea50 0x558a23b5ba70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009380 tx=0x7f6b4809b5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c605f06047df5ae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b47fff640 1 -- 172.21.15.40:0/2571070808 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b480b4040 con 0x558a23b3ea50 2024-03-18T22:44:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b47fff640 1 -- 172.21.15.40:0/2571070808 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b4809be60 con 0x558a23b3ea50 2024-03-18T22:44:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.693+0000 7f6b47fff640 1 -- 172.21.15.40:0/2571070808 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b480046c0 con 0x558a23b3ea50 2024-03-18T22:44:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.694+0000 7f6b67970740 1 -- 172.21.15.40:0/2571070808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b3ea50 msgr2=0x558a23b5ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.694+0000 7f6b67970740 1 --2- 172.21.15.40:0/2571070808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b3ea50 0x558a23b5ba70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009380 tx=0x7f6b4809b5c0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.694+0000 7f6b67970740 1 -- 172.21.15.40:0/2571070808 shutdown_connections 2024-03-18T22:44:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.694+0000 7f6b67970740 1 --2- 172.21.15.40:0/2571070808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3f440 0x558a23b5bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.694+0000 7f6b67970740 1 --2- 172.21.15.40:0/2571070808 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b3ea50 0x558a23b5ba70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.694+0000 7f6b67970740 1 --2- 172.21.15.40:0/2571070808 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b39fc0 0x558a23b3d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.694+0000 7f6b67970740 1 -- 172.21.15.40:0/2571070808 >> 172.21.15.40:0/2571070808 conn(0x558a23b40220 msgr2=0x558a23b3d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.694+0000 7f6b67970740 1 -- 172.21.15.40:0/2571070808 shutdown_connections 2024-03-18T22:44:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.694+0000 7f6b67970740 1 -- 172.21.15.40:0/2571070808 wait complete. 2024-03-18T22:44:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b67970740 1 -- start start 2024-03-18T22:44:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b67970740 1 Processor -- start 2024-03-18T22:44:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b67970740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b39fc0 0x558a23b4e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b67970740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3ea50 0x558a23b4e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b67970740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b3f440 0x558a23b4f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b67970740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601a9870 con 0x558a23b3f440 2024-03-18T22:44:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b67970740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601a91a0 con 0x558a23b3ea50 2024-03-18T22:44:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b67970740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601a63e0 con 0x558a23b39fc0 2024-03-18T22:44:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b67970740 10 client.0 _ll_register_callbacks cb 0x7f6b50009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b5e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b39fc0 0x558a23b4e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b5ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b3f440 0x558a23b4f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b5e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b39fc0 0x558a23b4e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:46316/0 (socket says 172.21.15.40:46316) 2024-03-18T22:44:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b5e59c640 1 -- 172.21.15.40:0/2095615823 learned_addr learned my addr 172.21.15.40:0/2095615823 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b57fff640 1 --2- 172.21.15.40:0/2095615823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3ea50 0x558a23b4e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b5e59c640 1 -- 172.21.15.40:0/2095615823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3ea50 msgr2=0x558a23b4e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b5e59c640 1 --2- 172.21.15.40:0/2095615823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3ea50 0x558a23b4e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b5e59c640 1 -- 172.21.15.40:0/2095615823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b3f440 msgr2=0x558a23b4f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b5e59c640 1 --2- 172.21.15.40:0/2095615823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b3f440 0x558a23b4f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b5e59c640 1 -- 172.21.15.40:0/2095615823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a23b4fbf0 con 0x558a23b39fc0 2024-03-18T22:44:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.695+0000 7f6b57fff640 1 --2- 172.21.15.40:0/2095615823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3ea50 0x558a23b4e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b5e59c640 1 --2- 172.21.15.40:0/2095615823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b39fc0 0x558a23b4e0b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6b50009850 tx=0x7f6b50004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25ed9c2b0d99cfa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:44:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b45ffb640 1 -- 172.21.15.40:0/2095615823 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5004f040 con 0x558a23b39fc0 2024-03-18T22:44:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b67970740 1 -- 172.21.15.40:0/2095615823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b4809be60 con 0x558a23b39fc0 2024-03-18T22:44:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b45ffb640 1 -- 172.21.15.40:0/2095615823 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6b50041d40 con 0x558a23b39fc0 2024-03-18T22:44:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b45ffb640 1 -- 172.21.15.40:0/2095615823 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6b5006e050 con 0x558a23b39fc0 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b67970740 10 client.25798 fetch_fsmap learned FSMap version 29 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b67970740 1 -- 172.21.15.40:0/2095615823 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558a23bbb970 con 0x558a23b39fc0 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b45ffb640 1 -- 172.21.15.40:0/2095615823 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b4809be60 con 0x558a23b39fc0 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b45ffb640 1 -- 172.21.15.40:0/2095615823 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f6b5004a040 con 0x558a23b39fc0 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b67970740 10 client.25798 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b67970740 10 client.25798 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b67970740 20 client.25798 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b67970740 1 --2- 172.21.15.40:0/2095615823 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a23b445b0 0x558a23bbe3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b67970740 4 client.25798 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.696+0000 7f6b67970740 1 -- 172.21.15.40:0/2095615823 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558a23aaad90 con 0x558a23b445b0 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b57fff640 1 --2- 172.21.15.40:0/2095615823 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a23b445b0 0x558a23bbe3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b45ffb640 10 client.25798 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b57fff640 1 --2- 172.21.15.40:0/2095615823 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a23b445b0 0x558a23bbe3d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62aecafff8695b78 server_cookie=5ecb0142c8550e33 in_seq=0 out_seq=0 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b45ffb640 1 -- 172.21.15.40:0/2095615823 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558a23aaad90 con 0x558a23b445b0 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b45ffb640 10 client.25798 handle_command_reply: tid=0 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b67970740 1 client.25798 shutdown 2024-03-18T22:44:16.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b67970740 1 -- 172.21.15.40:0/2095615823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b39fc0 msgr2=0x558a23b4e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b67970740 1 --2- 172.21.15.40:0/2095615823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b39fc0 0x558a23b4e0b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6b50009850 tx=0x7f6b50004610 comp rx=0 tx=0).stop 2024-03-18T22:44:16.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b67970740 1 -- 172.21.15.40:0/2095615823 shutdown_connections 2024-03-18T22:44:16.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b67970740 1 --2- 172.21.15.40:0/2095615823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a23b3f440 0x558a23b4f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b67970740 1 --2- 172.21.15.40:0/2095615823 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a23b445b0 0x558a23bbe3d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b67970740 1 --2- 172.21.15.40:0/2095615823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a23b3ea50 0x558a23b4e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b67970740 1 --2- 172.21.15.40:0/2095615823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a23b39fc0 0x558a23b4e0b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b45ffb640 0 client.25798 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:16.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b67970740 1 -- 172.21.15.40:0/2095615823 >> 172.21.15.40:0/2095615823 conn(0x558a23b40220 msgr2=0x558a23b3dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:16.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b67970740 1 -- 172.21.15.40:0/2095615823 shutdown_connections 2024-03-18T22:44:16.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b67970740 1 -- 172.21.15.40:0/2095615823 wait complete. 2024-03-18T22:44:16.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.697+0000 7f6b67970740 20 client.25798 trim_cache size 0 max 16384 2024-03-18T22:44:16.704 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:44:16.705 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:44:16.705 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:44:16.705 INFO:teuthology.orchestra.run.smithi040.stdout: "3bf386c0-f31e-4218-ac60-84ae9d174304": { 2024-03-18T22:44:16.705 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:44:16.705 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "3bf386c0-f31e-4218-ac60-84ae9d174304", 2024-03-18T22:44:16.705 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:44:16.705 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:44:16.705 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:44:16.705 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:44:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.698+0000 7f6b64d87640 1 -- 172.21.15.40:0/2627667716 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6b4c0a2800 msgr2=0x7f6b4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.698+0000 7f6b64d87640 1 --2- 172.21.15.40:0/2627667716 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6b4c0a2800 0x7f6b4c0a4cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6b50005ab0 tx=0x7f6b50005fd0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.698+0000 7f6b64d87640 1 -- 172.21.15.40:0/2627667716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b60101910 msgr2=0x7f6b6019c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.698+0000 7f6b64d87640 1 --2- 172.21.15.40:0/2627667716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b60101910 0x7f6b6019c800 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6b4800e9f0 tx=0x7f6b4800edf0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.698+0000 7f6b64d87640 1 -- 172.21.15.40:0/2627667716 shutdown_connections 2024-03-18T22:44:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.698+0000 7f6b64d87640 1 --2- 172.21.15.40:0/2627667716 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6b4c0a2800 0x7f6b4c0a4cc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.698+0000 7f6b64d87640 1 --2- 172.21.15.40:0/2627667716 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b601022d0 0x7f6b6019ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.698+0000 7f6b64d87640 1 --2- 172.21.15.40:0/2627667716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b60101910 0x7f6b6019c800 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.698+0000 7f6b64d87640 1 --2- 172.21.15.40:0/2627667716 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6b60100720 0x7f6b600694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.698+0000 7f6b64d87640 1 -- 172.21.15.40:0/2627667716 >> 172.21.15.40:0/2627667716 conn(0x7f6b600fbcb0 msgr2=0x7f6b60104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.698+0000 7f6b64d87640 1 -- 172.21.15.40:0/2627667716 shutdown_connections 2024-03-18T22:44:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:16.698+0000 7f6b64d87640 1 -- 172.21.15.40:0/2627667716 wait complete. 2024-03-18T22:44:16.711 INFO:tasks.cephfs.filesystem:scrub status for tag:3bf386c0-f31e-4218-ac60-84ae9d174304 - {'path': '/', 'tag': '3bf386c0-f31e-4218-ac60-84ae9d174304', 'options': 'recursive,force'} 2024-03-18T22:44:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:16 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:44:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:16 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:16 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:44:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:16 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:17.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:16 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:44:17.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:16 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:44:17.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:16 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:17.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:16 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:44:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:16 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:17.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:16 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:44:17.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:16 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:44:17.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:16 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:17.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:16 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:44:17.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:16 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:17.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:16 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:44:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:17 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:44:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:17 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:17 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:44:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:17 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:17 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:44:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:17 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:17 smithi040 ceph-mon[62643]: pgmap v400: 129 pgs: 129 active+clean; 202 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 16 MiB/s wr, 81 op/s 2024-03-18T22:44:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:17 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:17 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:44:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:17 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:17 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:44:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:17 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:17 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:44:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:17 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:17 smithi113 ceph-mon[66775]: pgmap v400: 129 pgs: 129 active+clean; 202 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 16 MiB/s wr, 81 op/s 2024-03-18T22:44:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:17 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:17 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:17 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:17 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:17 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:17 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:17 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:44:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:17 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:17 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:44:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:17 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:17 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:44:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:17 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:17 smithi103 ceph-mon[66377]: pgmap v400: 129 pgs: 129 active+clean; 202 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 16 MiB/s wr, 81 op/s 2024-03-18T22:44:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:17 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:17 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:17 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:17 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:17 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:18.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:17 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:19.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:18 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:44:19.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:18 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:44:19.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:18 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:19.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:18 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:44:19.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:18 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:44:19.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:18 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:44:19.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:18 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:19.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:18 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:44:19.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:18 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:44:19.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:18 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:44:19.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:18 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:44:19.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:18 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:44:20.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:20 smithi103 ceph-mon[66377]: pgmap v401: 129 pgs: 129 active+clean; 234 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 19 MiB/s wr, 52 op/s 2024-03-18T22:44:20.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:20 smithi040 ceph-mon[62643]: pgmap v401: 129 pgs: 129 active+clean; 234 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 19 MiB/s wr, 52 op/s 2024-03-18T22:44:20.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:20 smithi113 ceph-mon[66775]: pgmap v401: 129 pgs: 129 active+clean; 234 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 19 MiB/s wr, 52 op/s 2024-03-18T22:44:22.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:22 smithi040 ceph-mon[62643]: pgmap v402: 129 pgs: 129 active+clean; 286 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 23 MiB/s wr, 40 op/s 2024-03-18T22:44:22.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:22 smithi113 ceph-mon[66775]: pgmap v402: 129 pgs: 129 active+clean; 286 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 23 MiB/s wr, 40 op/s 2024-03-18T22:44:22.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:22 smithi103 ceph-mon[66377]: pgmap v402: 129 pgs: 129 active+clean; 286 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 23 MiB/s wr, 40 op/s 2024-03-18T22:44:23.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:23 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:44:23.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:23 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:44:23.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:23 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:44:23.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:23 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:44:23.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:23 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:44:23.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:23 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:44:24.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:24 smithi040 ceph-mon[62643]: pgmap v403: 129 pgs: 129 active+clean; 362 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 29 MiB/s wr, 28 op/s 2024-03-18T22:44:24.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:24 smithi113 ceph-mon[66775]: pgmap v403: 129 pgs: 129 active+clean; 362 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 29 MiB/s wr, 28 op/s 2024-03-18T22:44:24.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:24 smithi103 ceph-mon[66377]: pgmap v403: 129 pgs: 129 active+clean; 362 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 29 MiB/s wr, 28 op/s 2024-03-18T22:44:26.632 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:26 smithi113 ceph-mon[66775]: pgmap v404: 129 pgs: 129 active+clean; 398 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 27 MiB/s wr, 26 op/s 2024-03-18T22:44:26.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:26 smithi040 ceph-mon[62643]: pgmap v404: 129 pgs: 129 active+clean; 398 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 27 MiB/s wr, 26 op/s 2024-03-18T22:44:26.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:26 smithi103 ceph-mon[66377]: pgmap v404: 129 pgs: 129 active+clean; 398 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 27 MiB/s wr, 26 op/s 2024-03-18T22:44:28.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:28 smithi040 ceph-mon[62643]: pgmap v405: 129 pgs: 129 active+clean; 454 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:44:28.660 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:28 smithi103 ceph-mon[66377]: pgmap v405: 129 pgs: 129 active+clean; 454 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:44:28.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:28 smithi113 ceph-mon[66775]: pgmap v405: 129 pgs: 129 active+clean; 454 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:44:30.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:30 smithi040 ceph-mon[62643]: pgmap v406: 129 pgs: 129 active+clean; 502 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-03-18T22:44:30.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:30 smithi113 ceph-mon[66775]: pgmap v406: 129 pgs: 129 active+clean; 502 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-03-18T22:44:30.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:30 smithi103 ceph-mon[66377]: pgmap v406: 129 pgs: 129 active+clean; 502 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 MiB/s wr, 21 op/s 2024-03-18T22:44:32.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:32 smithi040 ceph-mon[62643]: pgmap v407: 129 pgs: 129 active+clean; 554 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 MiB/s wr, 24 op/s 2024-03-18T22:44:32.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:32 smithi113 ceph-mon[66775]: pgmap v407: 129 pgs: 129 active+clean; 554 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 MiB/s wr, 24 op/s 2024-03-18T22:44:32.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:32 smithi103 ceph-mon[66377]: pgmap v407: 129 pgs: 129 active+clean; 554 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 MiB/s wr, 24 op/s 2024-03-18T22:44:34.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:34 smithi040 ceph-mon[62643]: pgmap v408: 129 pgs: 129 active+clean; 642 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:44:34.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:34 smithi113 ceph-mon[66775]: pgmap v408: 129 pgs: 129 active+clean; 642 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:44:34.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:34 smithi103 ceph-mon[66377]: pgmap v408: 129 pgs: 129 active+clean; 642 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:44:36.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:36 smithi040 ceph-mon[62643]: pgmap v409: 129 pgs: 129 active+clean; 674 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 26 MiB/s wr, 24 op/s 2024-03-18T22:44:36.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:36 smithi113 ceph-mon[66775]: pgmap v409: 129 pgs: 129 active+clean; 674 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 26 MiB/s wr, 24 op/s 2024-03-18T22:44:36.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:36 smithi103 ceph-mon[66377]: pgmap v409: 129 pgs: 129 active+clean; 674 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 26 MiB/s wr, 24 op/s 2024-03-18T22:44:38.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:38 smithi040 ceph-mon[62643]: pgmap v410: 129 pgs: 129 active+clean; 742 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:44:38.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:38 smithi113 ceph-mon[66775]: pgmap v410: 129 pgs: 129 active+clean; 742 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:44:38.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:38 smithi103 ceph-mon[66377]: pgmap v410: 129 pgs: 129 active+clean; 742 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:44:40.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:40 smithi040 ceph-mon[62643]: pgmap v411: 129 pgs: 129 active+clean; 778 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 27 MiB/s wr, 21 op/s 2024-03-18T22:44:40.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:40 smithi113 ceph-mon[66775]: pgmap v411: 129 pgs: 129 active+clean; 778 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 27 MiB/s wr, 21 op/s 2024-03-18T22:44:40.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:40 smithi103 ceph-mon[66377]: pgmap v411: 129 pgs: 129 active+clean; 778 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 27 MiB/s wr, 21 op/s 2024-03-18T22:44:42.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:42 smithi040 ceph-mon[62643]: pgmap v412: 129 pgs: 129 active+clean; 830 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:44:42.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:42 smithi113 ceph-mon[66775]: pgmap v412: 129 pgs: 129 active+clean; 830 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:44:42.862 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:42 smithi103 ceph-mon[66377]: pgmap v412: 129 pgs: 129 active+clean; 830 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:44:44.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:44 smithi040 ceph-mon[62643]: pgmap v413: 129 pgs: 129 active+clean; 934 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 32 MiB/s wr, 22 op/s 2024-03-18T22:44:44.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:44 smithi113 ceph-mon[66775]: pgmap v413: 129 pgs: 129 active+clean; 934 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 32 MiB/s wr, 22 op/s 2024-03-18T22:44:44.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:44 smithi103 ceph-mon[66377]: pgmap v413: 129 pgs: 129 active+clean; 934 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 32 MiB/s wr, 22 op/s 2024-03-18T22:44:46.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:46 smithi040 ceph-mon[62643]: pgmap v414: 129 pgs: 129 active+clean; 946 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:44:46.712 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:44:46.712 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:44:46.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.826+0000 7f495eb5b640 1 -- 172.21.15.40:0/633537826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4958102a40 msgr2=0x7f4958102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:46.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.826+0000 7f495eb5b640 1 --2- 172.21.15.40:0/633537826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4958102a40 0x7f4958102ec0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4948009900 tx=0x7f494802bc30 comp rx=0 tx=0).stop 2024-03-18T22:44:46.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.826+0000 7f495eb5b640 1 -- 172.21.15.40:0/633537826 shutdown_connections 2024-03-18T22:44:46.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.826+0000 7f495eb5b640 1 --2- 172.21.15.40:0/633537826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4958103400 0x7f4958109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.826+0000 7f495eb5b640 1 --2- 172.21.15.40:0/633537826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4958102a40 0x7f4958102ec0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.826+0000 7f495eb5b640 1 --2- 172.21.15.40:0/633537826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4958101850 0x7f4958101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.826+0000 7f495eb5b640 1 -- 172.21.15.40:0/633537826 >> 172.21.15.40:0/633537826 conn(0x7f49580fcdc0 msgr2=0x7f49580ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:46.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.826+0000 7f495eb5b640 1 -- 172.21.15.40:0/633537826 shutdown_connections 2024-03-18T22:44:46.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.826+0000 7f495eb5b640 1 -- 172.21.15.40:0/633537826 wait complete. 2024-03-18T22:44:46.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.826+0000 7f495eb5b640 1 Processor -- start 2024-03-18T22:44:46.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.826+0000 7f495eb5b640 1 -- start start 2024-03-18T22:44:46.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f495eb5b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4958101850 0x7f4958197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f495eb5b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4958102a40 0x7f4958197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f495eb5b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4958103400 0x7f495819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f495eb5b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f495810d360 con 0x7f4958102a40 2024-03-18T22:44:46.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f495eb5b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f495810d1e0 con 0x7f4958103400 2024-03-18T22:44:46.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f495eb5b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f495810d4e0 con 0x7f4958101850 2024-03-18T22:44:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f494ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4958102a40 0x7f4958197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f494ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4958102a40 0x7f4958197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33616/0 (socket says 172.21.15.40:33616) 2024-03-18T22:44:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f494ffff640 1 -- 172.21.15.40:0/1725269183 learned_addr learned my addr 172.21.15.40:0/1725269183 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f495d0f8640 1 --2- 172.21.15.40:0/1725269183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4958103400 0x7f495819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f495c8f7640 1 --2- 172.21.15.40:0/1725269183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4958101850 0x7f4958197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f494ffff640 1 -- 172.21.15.40:0/1725269183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4958101850 msgr2=0x7f4958197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f494ffff640 1 --2- 172.21.15.40:0/1725269183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4958101850 0x7f4958197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f494ffff640 1 -- 172.21.15.40:0/1725269183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4958103400 msgr2=0x7f495819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f494ffff640 1 --2- 172.21.15.40:0/1725269183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4958103400 0x7f495819f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f494ffff640 1 -- 172.21.15.40:0/1725269183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495819f6b0 con 0x7f4958102a40 2024-03-18T22:44:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f495c8f7640 1 --2- 172.21.15.40:0/1725269183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4958101850 0x7f4958197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:44:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f494ffff640 1 --2- 172.21.15.40:0/1725269183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4958102a40 0x7f4958197fd0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4948006010 tx=0x7f4948005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f494dffb640 1 -- 172.21.15.40:0/1725269183 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4948043020 con 0x7f4958102a40 2024-03-18T22:44:46.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f494dffb640 1 -- 172.21.15.40:0/1725269183 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49480043d0 con 0x7f4958102a40 2024-03-18T22:44:46.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f495eb5b640 1 -- 172.21.15.40:0/1725269183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f495819f940 con 0x7f4958102a40 2024-03-18T22:44:46.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.827+0000 7f494dffb640 1 -- 172.21.15.40:0/1725269183 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4948003b80 con 0x7f4958102a40 2024-03-18T22:44:46.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.828+0000 7f495eb5b640 1 -- 172.21.15.40:0/1725269183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f495819fee0 con 0x7f4958102a40 2024-03-18T22:44:46.832 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:46 smithi113 ceph-mon[66775]: pgmap v414: 129 pgs: 129 active+clean; 946 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:44:46.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.830+0000 7f494dffb640 1 -- 172.21.15.40:0/1725269183 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4948046020 con 0x7f4958102a40 2024-03-18T22:44:46.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.830+0000 7f494dffb640 1 --2- 172.21.15.40:0/1725269183 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f49400a2880 0x7f49400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.830+0000 7f4961824740 1 Processor -- start 2024-03-18T22:44:46.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f4961824740 1 -- start start 2024-03-18T22:44:46.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f494dffb640 1 -- 172.21.15.40:0/1725269183 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f49480f1060 con 0x7f4958102a40 2024-03-18T22:44:46.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f495d0f8640 1 --2- 172.21.15.40:0/1725269183 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f49400a2880 0x7f49400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f4961824740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 0x563ba6d97e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f4961824740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d98440 0x563ba6d9cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f4961824740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d9d5b0 0x563ba6d9fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f4961824740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563ba6cfc500 con 0x563ba6d9d5b0 2024-03-18T22:44:46.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f4961824740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563ba6ca4be0 con 0x563ba6d97a50 2024-03-18T22:44:46.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f4961824740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563ba6cf85a0 con 0x563ba6d98440 2024-03-18T22:44:46.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f494ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d98440 0x563ba6d9cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f495c8f7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 0x563ba6d97e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f494ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d98440 0x563ba6d9cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41370/0 (socket says 172.21.15.40:41370) 2024-03-18T22:44:46.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f494ffff640 1 -- 172.21.15.40:0/3425451301 learned_addr learned my addr 172.21.15.40:0/3425451301 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:46.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f495d0f8640 1 --2- 172.21.15.40:0/3425451301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d9d5b0 0x563ba6d9fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f495d0f8640 1 --2- 172.21.15.40:0/1725269183 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f49400a2880 0x7f49400a4d40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4950005fd0 tx=0x7f4950011810 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:46.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f494ffff640 1 -- 172.21.15.40:0/3425451301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 msgr2=0x563ba6d97e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:46.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f494ffff640 1 --2- 172.21.15.40:0/3425451301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 0x563ba6d97e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f494ffff640 1 -- 172.21.15.40:0/3425451301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d9d5b0 msgr2=0x563ba6d9fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f494ffff640 1 --2- 172.21.15.40:0/3425451301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d9d5b0 0x563ba6d9fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.831+0000 7f494ffff640 1 -- 172.21.15.40:0/3425451301 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ba6cc6000 con 0x563ba6d98440 2024-03-18T22:44:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.832+0000 7f494ffff640 1 --2- 172.21.15.40:0/3425451301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d98440 0x563ba6d9cf40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4948005ab0 tx=0x7f49480b9980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b59d7428120f715 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.832+0000 7f493effd640 1 -- 172.21.15.40:0/3425451301 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49480d6070 con 0x563ba6d98440 2024-03-18T22:44:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.832+0000 7f493effd640 1 -- 172.21.15.40:0/3425451301 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49480b9040 con 0x563ba6d98440 2024-03-18T22:44:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.832+0000 7f493effd640 1 -- 172.21.15.40:0/3425451301 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4948003d20 con 0x563ba6d98440 2024-03-18T22:44:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 -- 172.21.15.40:0/3425451301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d98440 msgr2=0x563ba6d9cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 --2- 172.21.15.40:0/3425451301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d98440 0x563ba6d9cf40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4948005ab0 tx=0x7f49480b9980 comp rx=0 tx=0).stop 2024-03-18T22:44:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 -- 172.21.15.40:0/3425451301 shutdown_connections 2024-03-18T22:44:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 --2- 172.21.15.40:0/3425451301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d9d5b0 0x563ba6d9fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 --2- 172.21.15.40:0/3425451301 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d98440 0x563ba6d9cf40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 --2- 172.21.15.40:0/3425451301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 0x563ba6d97e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 -- 172.21.15.40:0/3425451301 >> 172.21.15.40:0/3425451301 conn(0x563ba6d92fc0 msgr2=0x563ba6d933c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:46.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 -- 172.21.15.40:0/3425451301 shutdown_connections 2024-03-18T22:44:46.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 -- 172.21.15.40:0/3425451301 wait complete. 2024-03-18T22:44:46.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 -- start start 2024-03-18T22:44:46.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 Processor -- start 2024-03-18T22:44:46.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 0x563ba6da1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d98440 0x563ba6da2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 0x563ba6da9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563ba6da0320 con 0x563ba6d98440 2024-03-18T22:44:46.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563ba6cb8160 con 0x563ba6d97a50 2024-03-18T22:44:46.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563ba6da04a0 con 0x563ba6d9d5b0 2024-03-18T22:44:46.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f4961824740 10 client.0 _ll_register_callbacks cb 0x563ba6cf4a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:46.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.833+0000 7f494ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d98440 0x563ba6da2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f494ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d98440 0x563ba6da2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33626/0 (socket says 172.21.15.40:33626) 2024-03-18T22:44:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f494ffff640 1 -- 172.21.15.40:0/4054957823 learned_addr learned my addr 172.21.15.40:0/4054957823 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f495c8f7640 1 --2- 172.21.15.40:0/4054957823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 0x563ba6da1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f495d0f8640 1 --2- 172.21.15.40:0/4054957823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 0x563ba6da9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f494ffff640 1 -- 172.21.15.40:0/4054957823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 msgr2=0x563ba6da9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f494ffff640 1 --2- 172.21.15.40:0/4054957823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 0x563ba6da9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f494ffff640 1 -- 172.21.15.40:0/4054957823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 msgr2=0x563ba6da1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f494ffff640 1 --2- 172.21.15.40:0/4054957823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 0x563ba6da1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f494ffff640 1 -- 172.21.15.40:0/4054957823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ba6ada7d0 con 0x563ba6d98440 2024-03-18T22:44:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f494ffff640 1 --2- 172.21.15.40:0/4054957823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d98440 0x563ba6da2130 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f49480c1920 tx=0x7f4948004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=334e7733fbaec8f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f4937fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:44:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f4937fff640 1 -- 172.21.15.40:0/4054957823 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49480db060 con 0x563ba6d98440 2024-03-18T22:44:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f4937fff640 1 -- 172.21.15.40:0/4054957823 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f49480c5920 con 0x563ba6d98440 2024-03-18T22:44:46.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f4961824740 1 -- 172.21.15.40:0/4054957823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49480b9040 con 0x563ba6d98440 2024-03-18T22:44:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f4937fff640 1 -- 172.21.15.40:0/4054957823 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f49480d6040 con 0x563ba6d98440 2024-03-18T22:44:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f4961824740 10 client.15966 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.834+0000 7f4961824740 1 -- 172.21.15.40:0/4054957823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563ba6a103f0 con 0x563ba6d98440 2024-03-18T22:44:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.835+0000 7f4937fff640 1 -- 172.21.15.40:0/4054957823 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f49480b9040 con 0x563ba6d98440 2024-03-18T22:44:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.835+0000 7f4937fff640 1 -- 172.21.15.40:0/4054957823 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f49480a6ca0 con 0x563ba6d98440 2024-03-18T22:44:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.835+0000 7f4961824740 10 client.15966 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.835+0000 7f4961824740 10 client.15966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.835+0000 7f4961824740 20 client.15966 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.835+0000 7f4961824740 1 --2- 172.21.15.40:0/4054957823 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba6dad660 0x563ba6dafa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.835+0000 7f4961824740 4 client.15966 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.835+0000 7f4961824740 1 -- 172.21.15.40:0/4054957823 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ba6d09a40 con 0x563ba6dad660 2024-03-18T22:44:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.835+0000 7f495c8f7640 1 --2- 172.21.15.40:0/4054957823 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba6dad660 0x563ba6dafa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.835+0000 7f495c8f7640 1 --2- 172.21.15.40:0/4054957823 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba6dad660 0x563ba6dafa50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=985a80ee5a25b843 server_cookie=cee1b6a7cdba2943 in_seq=0 out_seq=0 2024-03-18T22:44:46.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.835+0000 7f4937fff640 10 client.15966 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.836+0000 7f4937fff640 1 -- 172.21.15.40:0/4054957823 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563ba6d09a40 con 0x563ba6dad660 2024-03-18T22:44:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.836+0000 7f4937fff640 10 client.15966 handle_command_reply: tid=0 2024-03-18T22:44:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.836+0000 7f4961824740 1 client.15966 shutdown 2024-03-18T22:44:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.836+0000 7f4961824740 1 -- 172.21.15.40:0/4054957823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d98440 msgr2=0x563ba6da2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.836+0000 7f4961824740 1 --2- 172.21.15.40:0/4054957823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d98440 0x563ba6da2130 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f49480c1920 tx=0x7f4948004600 comp rx=0 tx=0).stop 2024-03-18T22:44:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.836+0000 7f4961824740 1 -- 172.21.15.40:0/4054957823 shutdown_connections 2024-03-18T22:44:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.836+0000 7f4961824740 1 --2- 172.21.15.40:0/4054957823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 0x563ba6da9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.836+0000 7f4961824740 1 --2- 172.21.15.40:0/4054957823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d98440 0x563ba6da2130 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.836+0000 7f4961824740 1 --2- 172.21.15.40:0/4054957823 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba6dad660 0x563ba6dafa50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.836+0000 7f4961824740 1 --2- 172.21.15.40:0/4054957823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 0x563ba6da1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.836+0000 7f4937fff640 0 client.15966 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:46.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.836+0000 7f4961824740 1 -- 172.21.15.40:0/4054957823 >> 172.21.15.40:0/4054957823 conn(0x563ba6d92fc0 msgr2=0x563ba6d921e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:46.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.836+0000 7f4961824740 1 -- 172.21.15.40:0/4054957823 shutdown_connections 2024-03-18T22:44:46.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.836+0000 7f4961824740 1 -- 172.21.15.40:0/4054957823 wait complete. 2024-03-18T22:44:46.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.836+0000 7f4961824740 20 client.15966 trim_cache size 0 max 16384 2024-03-18T22:44:46.861 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:46 smithi103 ceph-mon[66377]: pgmap v414: 129 pgs: 129 active+clean; 946 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:44:46.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.855+0000 7f4961824740 1 Processor -- start 2024-03-18T22:44:46.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.855+0000 7f4961824740 1 -- start start 2024-03-18T22:44:46.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f4961824740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 0x563ba6d8e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f4961824740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d98440 0x563ba6da9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f4961824740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 0x563ba6d96930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f4961824740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f49581aa5b0 con 0x563ba6d98440 2024-03-18T22:44:46.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f4961824740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f49581ac1b0 con 0x563ba6d97a50 2024-03-18T22:44:46.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f4961824740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f49581aac80 con 0x563ba6d9d5b0 2024-03-18T22:44:46.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f494ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d98440 0x563ba6da9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f495c8f7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 0x563ba6d8e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f495d0f8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 0x563ba6d96930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f495c8f7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 0x563ba6d8e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:50940/0 (socket says 172.21.15.40:50940) 2024-03-18T22:44:46.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f495c8f7640 1 -- 172.21.15.40:0/98673145 learned_addr learned my addr 172.21.15.40:0/98673145 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:46.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f495d0f8640 1 -- 172.21.15.40:0/98673145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 msgr2=0x563ba6d8e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:46.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f495d0f8640 1 --2- 172.21.15.40:0/98673145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 0x563ba6d8e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f495d0f8640 1 -- 172.21.15.40:0/98673145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d98440 msgr2=0x563ba6da9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:46.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f495d0f8640 1 --2- 172.21.15.40:0/98673145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d98440 0x563ba6da9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f495d0f8640 1 -- 172.21.15.40:0/98673145 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ba69d00f0 con 0x563ba6d9d5b0 2024-03-18T22:44:46.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f495c8f7640 1 --2- 172.21.15.40:0/98673145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 0x563ba6d8e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:46.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f494ffff640 1 --2- 172.21.15.40:0/98673145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d98440 0x563ba6da9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:46.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.856+0000 7f495d0f8640 1 --2- 172.21.15.40:0/98673145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 0x563ba6d96930 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4950009fd0 tx=0x7f495001d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1826b272c883179 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:46.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.857+0000 7f493e7fc640 1 -- 172.21.15.40:0/98673145 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495002d070 con 0x563ba6d9d5b0 2024-03-18T22:44:46.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.857+0000 7f493e7fc640 1 -- 172.21.15.40:0/98673145 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4950019050 con 0x563ba6d9d5b0 2024-03-18T22:44:46.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.857+0000 7f493e7fc640 1 -- 172.21.15.40:0/98673145 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4950017040 con 0x563ba6d9d5b0 2024-03-18T22:44:46.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.857+0000 7f4961824740 1 -- 172.21.15.40:0/98673145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 msgr2=0x563ba6d96930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:46.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.857+0000 7f4961824740 1 --2- 172.21.15.40:0/98673145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 0x563ba6d96930 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4950009fd0 tx=0x7f495001d000 comp rx=0 tx=0).stop 2024-03-18T22:44:46.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.857+0000 7f4961824740 1 -- 172.21.15.40:0/98673145 shutdown_connections 2024-03-18T22:44:46.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.857+0000 7f4961824740 1 --2- 172.21.15.40:0/98673145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 0x563ba6d96930 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.857+0000 7f4961824740 1 --2- 172.21.15.40:0/98673145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d98440 0x563ba6da9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.857+0000 7f4961824740 1 --2- 172.21.15.40:0/98673145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d97a50 0x563ba6d8e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.857+0000 7f4961824740 1 -- 172.21.15.40:0/98673145 >> 172.21.15.40:0/98673145 conn(0x563ba6d92fc0 msgr2=0x563ba6d96220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:46.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.857+0000 7f4961824740 1 -- 172.21.15.40:0/98673145 shutdown_connections 2024-03-18T22:44:46.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.857+0000 7f4961824740 1 -- 172.21.15.40:0/98673145 wait complete. 2024-03-18T22:44:46.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.858+0000 7f4961824740 1 -- start start 2024-03-18T22:44:46.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.858+0000 7f4961824740 1 Processor -- start 2024-03-18T22:44:46.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.858+0000 7f4961824740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d97a50 0x563ba6da68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.858+0000 7f4961824740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d98440 0x563ba6da6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.858+0000 7f4961824740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 0x563ba6da7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.858+0000 7f4961824740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f49581aba20 con 0x563ba6d97a50 2024-03-18T22:44:46.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.858+0000 7f4961824740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f49581ab350 con 0x563ba6d98440 2024-03-18T22:44:46.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.858+0000 7f4961824740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f49581a8590 con 0x563ba6d9d5b0 2024-03-18T22:44:46.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.858+0000 7f4961824740 10 client.0 _ll_register_callbacks cb 0x7f49480d4070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:46.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.858+0000 7f495c8f7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d97a50 0x563ba6da68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.858+0000 7f495c8f7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d97a50 0x563ba6da68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33636/0 (socket says 172.21.15.40:33636) 2024-03-18T22:44:46.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.858+0000 7f495c8f7640 1 -- 172.21.15.40:0/3590199620 learned_addr learned my addr 172.21.15.40:0/3590199620 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:46.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.858+0000 7f494ffff640 1 --2- 172.21.15.40:0/3590199620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d98440 0x563ba6da6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.858+0000 7f495d0f8640 1 --2- 172.21.15.40:0/3590199620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 0x563ba6da7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.859+0000 7f495c8f7640 1 -- 172.21.15.40:0/3590199620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 msgr2=0x563ba6da7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:46.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.859+0000 7f495c8f7640 1 --2- 172.21.15.40:0/3590199620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 0x563ba6da7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.859+0000 7f495c8f7640 1 -- 172.21.15.40:0/3590199620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d98440 msgr2=0x563ba6da6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:46.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.859+0000 7f495c8f7640 1 --2- 172.21.15.40:0/3590199620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d98440 0x563ba6da6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.859+0000 7f495c8f7640 1 -- 172.21.15.40:0/3590199620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ba6da83d0 con 0x563ba6d97a50 2024-03-18T22:44:46.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.859+0000 7f495c8f7640 1 --2- 172.21.15.40:0/3590199620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d97a50 0x563ba6da68e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x563ba6d98ca0 tx=0x7f49540216b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=969070066f161b5d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:46.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.859+0000 7f4937fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:44:46.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.859+0000 7f4937fff640 1 -- 172.21.15.40:0/3590199620 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4954012cb0 con 0x563ba6d97a50 2024-03-18T22:44:46.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.859+0000 7f4937fff640 1 -- 172.21.15.40:0/3590199620 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4954005030 con 0x563ba6d97a50 2024-03-18T22:44:46.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.859+0000 7f4961824740 1 -- 172.21.15.40:0/3590199620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4950019050 con 0x563ba6d97a50 2024-03-18T22:44:46.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.859+0000 7f4937fff640 1 -- 172.21.15.40:0/3590199620 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4954032050 con 0x563ba6d97a50 2024-03-18T22:44:46.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.859+0000 7f4961824740 10 client.15972 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:46.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.859+0000 7f4961824740 1 -- 172.21.15.40:0/3590199620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563ba6e13ac0 con 0x563ba6d97a50 2024-03-18T22:44:46.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.859+0000 7f4937fff640 1 -- 172.21.15.40:0/3590199620 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4950019050 con 0x563ba6d97a50 2024-03-18T22:44:46.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4937fff640 1 -- 172.21.15.40:0/3590199620 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f495402a430 con 0x563ba6d97a50 2024-03-18T22:44:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4961824740 10 client.15972 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4961824740 10 client.15972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4961824740 20 client.15972 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4961824740 1 --2- 172.21.15.40:0/3590199620 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba6e14600 0x563ba6e16a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4961824740 4 client.15972 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:44:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4961824740 1 -- 172.21.15.40:0/3590199620 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563ba6d03d90 con 0x563ba6e14600 2024-03-18T22:44:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f494ffff640 1 --2- 172.21.15.40:0/3590199620 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba6e14600 0x563ba6e16a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4937fff640 10 client.15972 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f494ffff640 1 --2- 172.21.15.40:0/3590199620 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba6e14600 0x563ba6e16a20 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=449957becc3b2587 server_cookie=160d3bb8a563051a in_seq=0 out_seq=0 2024-03-18T22:44:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4937fff640 1 -- 172.21.15.40:0/3590199620 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563ba6d03d90 con 0x563ba6e14600 2024-03-18T22:44:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4937fff640 10 client.15972 handle_command_reply: tid=0 2024-03-18T22:44:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4961824740 1 client.15972 shutdown 2024-03-18T22:44:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4961824740 1 -- 172.21.15.40:0/3590199620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d97a50 msgr2=0x563ba6da68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:46.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4961824740 1 --2- 172.21.15.40:0/3590199620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d97a50 0x563ba6da68e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x563ba6d98ca0 tx=0x7f49540216b0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4961824740 1 -- 172.21.15.40:0/3590199620 shutdown_connections 2024-03-18T22:44:46.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4961824740 1 --2- 172.21.15.40:0/3590199620 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba6d9d5b0 0x563ba6da7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4961824740 1 --2- 172.21.15.40:0/3590199620 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba6e14600 0x563ba6e16a20 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f4937fff640 0 client.15972 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:46.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.860+0000 7f4961824740 1 --2- 172.21.15.40:0/3590199620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba6d98440 0x563ba6da6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f4961824740 1 --2- 172.21.15.40:0/3590199620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba6d97a50 0x563ba6da68e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f4961824740 1 -- 172.21.15.40:0/3590199620 >> 172.21.15.40:0/3590199620 conn(0x563ba6d92fc0 msgr2=0x563ba6d96ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:46.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f4961824740 1 -- 172.21.15.40:0/3590199620 shutdown_connections 2024-03-18T22:44:46.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f4961824740 1 -- 172.21.15.40:0/3590199620 wait complete. 2024-03-18T22:44:46.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f4961824740 20 client.15972 trim_cache size 0 max 16384 2024-03-18T22:44:46.868 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:44:46.869 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:44:46.869 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:44:46.869 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:44:46.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f495eb5b640 1 -- 172.21.15.40:0/1725269183 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f49400a2880 msgr2=0x7f49400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:46.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f495eb5b640 1 --2- 172.21.15.40:0/1725269183 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f49400a2880 0x7f49400a4d40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4950005fd0 tx=0x7f4950011810 comp rx=0 tx=0).stop 2024-03-18T22:44:46.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f495eb5b640 1 -- 172.21.15.40:0/1725269183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4958102a40 msgr2=0x7f4958197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:46.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f495eb5b640 1 --2- 172.21.15.40:0/1725269183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4958102a40 0x7f4958197fd0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4948006010 tx=0x7f4948005e40 comp rx=0 tx=0).stop 2024-03-18T22:44:46.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f495eb5b640 1 -- 172.21.15.40:0/1725269183 shutdown_connections 2024-03-18T22:44:46.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f495eb5b640 1 --2- 172.21.15.40:0/1725269183 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f49400a2880 0x7f49400a4d40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f495eb5b640 1 --2- 172.21.15.40:0/1725269183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4958103400 0x7f495819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f495eb5b640 1 --2- 172.21.15.40:0/1725269183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4958102a40 0x7f4958197fd0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f495eb5b640 1 --2- 172.21.15.40:0/1725269183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4958101850 0x7f4958197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.861+0000 7f495eb5b640 1 -- 172.21.15.40:0/1725269183 >> 172.21.15.40:0/1725269183 conn(0x7f49580fcdc0 msgr2=0x7f4958105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:46.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.862+0000 7f495eb5b640 1 -- 172.21.15.40:0/1725269183 shutdown_connections 2024-03-18T22:44:46.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.862+0000 7f495eb5b640 1 -- 172.21.15.40:0/1725269183 wait complete. 2024-03-18T22:44:46.873 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:44:46.874 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:44:46.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.993+0000 7fec89fb9640 1 -- 172.21.15.40:0/3442996954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec84103400 msgr2=0x7fec84109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:46.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.993+0000 7fec89fb9640 1 --2- 172.21.15.40:0/3442996954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec84103400 0x7fec84109c90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009ca0 tx=0x7fec7c02c100 comp rx=0 tx=0).stop 2024-03-18T22:44:46.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.993+0000 7fec89fb9640 1 -- 172.21.15.40:0/3442996954 shutdown_connections 2024-03-18T22:44:46.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.993+0000 7fec89fb9640 1 --2- 172.21.15.40:0/3442996954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec84103400 0x7fec84109c90 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.993+0000 7fec89fb9640 1 --2- 172.21.15.40:0/3442996954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec84102a40 0x7fec84102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.993+0000 7fec89fb9640 1 --2- 172.21.15.40:0/3442996954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec84101850 0x7fec84101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:46.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.993+0000 7fec89fb9640 1 -- 172.21.15.40:0/3442996954 >> 172.21.15.40:0/3442996954 conn(0x7fec840fcdc0 msgr2=0x7fec840ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.993+0000 7fec89fb9640 1 -- 172.21.15.40:0/3442996954 shutdown_connections 2024-03-18T22:44:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.993+0000 7fec89fb9640 1 -- 172.21.15.40:0/3442996954 wait complete. 2024-03-18T22:44:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.994+0000 7fec89fb9640 1 Processor -- start 2024-03-18T22:44:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.994+0000 7fec89fb9640 1 -- start start 2024-03-18T22:44:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.994+0000 7fec89fb9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec84101850 0x7fec84197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.994+0000 7fec89fb9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec84102a40 0x7fec84197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.994+0000 7fec89fb9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec84103400 0x7fec8419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.994+0000 7fec89fb9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fec8410d380 con 0x7fec84102a40 2024-03-18T22:44:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.994+0000 7fec89fb9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fec8410d200 con 0x7fec84101850 2024-03-18T22:44:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.994+0000 7fec89fb9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fec8410d500 con 0x7fec84103400 2024-03-18T22:44:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.994+0000 7fec82ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec84102a40 0x7fec84197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.994+0000 7fec82ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec84102a40 0x7fec84197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33664/0 (socket says 172.21.15.40:33664) 2024-03-18T22:44:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.994+0000 7fec82ffd640 1 -- 172.21.15.40:0/499241322 learned_addr learned my addr 172.21.15.40:0/499241322 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.994+0000 7fec837fe640 1 --2- 172.21.15.40:0/499241322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec84101850 0x7fec84197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.994+0000 7fec83fff640 1 --2- 172.21.15.40:0/499241322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec84103400 0x7fec8419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.995+0000 7fec82ffd640 1 -- 172.21.15.40:0/499241322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec84103400 msgr2=0x7fec8419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.995+0000 7fec82ffd640 1 --2- 172.21.15.40:0/499241322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec84103400 0x7fec8419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.995+0000 7fec82ffd640 1 -- 172.21.15.40:0/499241322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec84101850 msgr2=0x7fec84197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.995+0000 7fec82ffd640 1 --2- 172.21.15.40:0/499241322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec84101850 0x7fec84197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.995+0000 7fec82ffd640 1 -- 172.21.15.40:0/499241322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8419f670 con 0x7fec84102a40 2024-03-18T22:44:47.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.995+0000 7fec837fe640 1 --2- 172.21.15.40:0/499241322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec84101850 0x7fec84197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.995+0000 7fec82ffd640 1 --2- 172.21.15.40:0/499241322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec84102a40 0x7fec84197f90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fec7800ea60 tx=0x7fec7800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.995+0000 7fec80ff9640 1 -- 172.21.15.40:0/499241322 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7800cde0 con 0x7fec84102a40 2024-03-18T22:44:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.995+0000 7fec89fb9640 1 -- 172.21.15.40:0/499241322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec8419f960 con 0x7fec84102a40 2024-03-18T22:44:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.995+0000 7fec80ff9640 1 -- 172.21.15.40:0/499241322 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec780035f0 con 0x7fec84102a40 2024-03-18T22:44:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.995+0000 7fec80ff9640 1 -- 172.21.15.40:0/499241322 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78010690 con 0x7fec84102a40 2024-03-18T22:44:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.995+0000 7fec89fb9640 1 -- 172.21.15.40:0/499241322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec8419feb0 con 0x7fec84102a40 2024-03-18T22:44:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.996+0000 7fec89fb9640 1 -- 172.21.15.40:0/499241322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec84108be0 con 0x7fec84102a40 2024-03-18T22:44:47.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.998+0000 7fec80ff9640 1 -- 172.21.15.40:0/499241322 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fec780037a0 con 0x7fec84102a40 2024-03-18T22:44:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.999+0000 7fec80ff9640 1 --2- 172.21.15.40:0/499241322 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec6c0a2800 0x7fec6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.999+0000 7fec80ff9640 1 -- 172.21.15.40:0/499241322 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fec780cf500 con 0x7fec84102a40 2024-03-18T22:44:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.999+0000 7fec837fe640 1 --2- 172.21.15.40:0/499241322 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec6c0a2800 0x7fec6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:46.999+0000 7fec837fe640 1 --2- 172.21.15.40:0/499241322 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec6c0a2800 0x7fec6c0a4cc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fec841028a0 tx=0x7fec74005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.003+0000 7fec80ff9640 1 -- 172.21.15.40:0/499241322 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fec7809c1b0 con 0x7fec84102a40 2024-03-18T22:44:47.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.252+0000 7fec89fb9640 1 -- 172.21.15.40:0/499241322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec8402d1f0 con 0x7fec84102a40 2024-03-18T22:44:47.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.254+0000 7fec80ff9640 1 -- 172.21.15.40:0/499241322 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11954 (secure 0 0 0) 0x7fec7809bab0 con 0x7fec84102a40 2024-03-18T22:44:47.254 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:44:47.255 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:44:17.725012+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24401,24388,24473,24509,24481]},"id":1}]} 2024-03-18T22:44:47.255 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 30 2024-03-18T22:44:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.256+0000 7fec89fb9640 1 -- 172.21.15.40:0/499241322 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec6c0a2800 msgr2=0x7fec6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.256+0000 7fec89fb9640 1 --2- 172.21.15.40:0/499241322 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec6c0a2800 0x7fec6c0a4cc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fec841028a0 tx=0x7fec74005f90 comp rx=0 tx=0).stop 2024-03-18T22:44:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.256+0000 7fec89fb9640 1 -- 172.21.15.40:0/499241322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec84102a40 msgr2=0x7fec84197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.257+0000 7fec89fb9640 1 --2- 172.21.15.40:0/499241322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec84102a40 0x7fec84197f90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fec7800ea60 tx=0x7fec7800ef20 comp rx=0 tx=0).stop 2024-03-18T22:44:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.257+0000 7fec89fb9640 1 -- 172.21.15.40:0/499241322 shutdown_connections 2024-03-18T22:44:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.257+0000 7fec89fb9640 1 --2- 172.21.15.40:0/499241322 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fec6c0a2800 0x7fec6c0a4cc0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.257+0000 7fec89fb9640 1 --2- 172.21.15.40:0/499241322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fec84103400 0x7fec8419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.257+0000 7fec89fb9640 1 --2- 172.21.15.40:0/499241322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec84102a40 0x7fec84197f90 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.257+0000 7fec89fb9640 1 --2- 172.21.15.40:0/499241322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec84101850 0x7fec84197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.257+0000 7fec89fb9640 1 -- 172.21.15.40:0/499241322 >> 172.21.15.40:0/499241322 conn(0x7fec840fcdc0 msgr2=0x7fec84105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.257+0000 7fec89fb9640 1 -- 172.21.15.40:0/499241322 shutdown_connections 2024-03-18T22:44:47.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.257+0000 7fec89fb9640 1 -- 172.21.15.40:0/499241322 wait complete. 2024-03-18T22:44:47.270 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:44:47.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.384+0000 7f95fa80b640 1 -- 172.21.15.40:0/2545108736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95f40ffe90 msgr2=0x7f95f41002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.384+0000 7f95fa80b640 1 --2- 172.21.15.40:0/2545108736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95f40ffe90 0x7f95f41002b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f95e4009900 tx=0x7f95e402bc30 comp rx=0 tx=0).stop 2024-03-18T22:44:47.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.384+0000 7f95fa80b640 1 -- 172.21.15.40:0/2545108736 shutdown_connections 2024-03-18T22:44:47.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.384+0000 7f95fa80b640 1 --2- 172.21.15.40:0/2545108736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95f41019a0 0x7f95f41082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.384+0000 7f95fa80b640 1 --2- 172.21.15.40:0/2545108736 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f4100fe0 0x7f95f4101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.384+0000 7f95fa80b640 1 --2- 172.21.15.40:0/2545108736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95f40ffe90 0x7f95f41002b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.384+0000 7f95fa80b640 1 -- 172.21.15.40:0/2545108736 >> 172.21.15.40:0/2545108736 conn(0x7f95f40fb400 msgr2=0x7f95f40fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.384+0000 7f95fa80b640 1 -- 172.21.15.40:0/2545108736 shutdown_connections 2024-03-18T22:44:47.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.384+0000 7f95fa80b640 1 -- 172.21.15.40:0/2545108736 wait complete. 2024-03-18T22:44:47.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.385+0000 7f95fa80b640 1 Processor -- start 2024-03-18T22:44:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.385+0000 7f95fa80b640 1 -- start start 2024-03-18T22:44:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.385+0000 7f95fa80b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f40ffe90 0x7f95f4109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.385+0000 7f95fa80b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95f4100fe0 0x7f95f419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.385+0000 7f95fa80b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95f41019a0 0x7f95f419ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.385+0000 7f95fa80b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f95f410c2a0 con 0x7f95f4100fe0 2024-03-18T22:44:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.385+0000 7f95fa80b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f95f410c120 con 0x7f95f41019a0 2024-03-18T22:44:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.385+0000 7f95fa80b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f95f410c420 con 0x7f95f40ffe90 2024-03-18T22:44:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.385+0000 7f95f37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95f4100fe0 0x7f95f419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.385+0000 7f95f3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f40ffe90 0x7f95f4109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.385+0000 7f95f37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95f4100fe0 0x7f95f419c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33688/0 (socket says 172.21.15.40:33688) 2024-03-18T22:44:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.385+0000 7f95f37fe640 1 -- 172.21.15.40:0/4097826703 learned_addr learned my addr 172.21.15.40:0/4097826703 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.385+0000 7f95f8da8640 1 --2- 172.21.15.40:0/4097826703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95f41019a0 0x7f95f419ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.386+0000 7f95f3fff640 1 -- 172.21.15.40:0/4097826703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95f41019a0 msgr2=0x7f95f419ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.386+0000 7f95f3fff640 1 --2- 172.21.15.40:0/4097826703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95f41019a0 0x7f95f419ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.386+0000 7f95f3fff640 1 -- 172.21.15.40:0/4097826703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95f4100fe0 msgr2=0x7f95f419c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.386+0000 7f95f3fff640 1 --2- 172.21.15.40:0/4097826703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95f4100fe0 0x7f95f419c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.386+0000 7f95f3fff640 1 -- 172.21.15.40:0/4097826703 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f419f6a0 con 0x7f95f40ffe90 2024-03-18T22:44:47.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.386+0000 7f95f3fff640 1 --2- 172.21.15.40:0/4097826703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f40ffe90 0x7f95f4109510 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f95e4000c00 tx=0x7f95e4005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.386+0000 7f95f17fa640 1 -- 172.21.15.40:0/4097826703 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e4043020 con 0x7f95f40ffe90 2024-03-18T22:44:47.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.386+0000 7f95f17fa640 1 -- 172.21.15.40:0/4097826703 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f95e4004350 con 0x7f95f40ffe90 2024-03-18T22:44:47.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.386+0000 7f95fa80b640 1 -- 172.21.15.40:0/4097826703 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95f419f930 con 0x7f95f40ffe90 2024-03-18T22:44:47.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.386+0000 7f95f17fa640 1 -- 172.21.15.40:0/4097826703 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e4003b50 con 0x7f95f40ffe90 2024-03-18T22:44:47.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.386+0000 7f95fa80b640 1 -- 172.21.15.40:0/4097826703 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95f419fe80 con 0x7f95f40ffe90 2024-03-18T22:44:47.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.388+0000 7f95f17fa640 1 -- 172.21.15.40:0/4097826703 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f95e4046020 con 0x7f95f40ffe90 2024-03-18T22:44:47.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.389+0000 7f95f17fa640 1 --2- 172.21.15.40:0/4097826703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f95dc0a2800 0x7f95dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.389+0000 7f95fd3f8740 1 Processor -- start 2024-03-18T22:44:47.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.389+0000 7f95fd3f8740 1 -- start start 2024-03-18T22:44:47.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.389+0000 7f95f37fe640 1 --2- 172.21.15.40:0/4097826703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f95dc0a2800 0x7f95dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.389+0000 7f95f17fa640 1 -- 172.21.15.40:0/4097826703 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f95e40f0eb0 con 0x7f95f40ffe90 2024-03-18T22:44:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.389+0000 7f95fd3f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acb72f0 0x559c2acb7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.389+0000 7f95fd3f8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acb7ce0 0x559c2acbc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.389+0000 7f95fd3f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acbce50 0x559c2acbf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.389+0000 7f95fd3f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559c2ac1b4b0 con 0x559c2acbce50 2024-03-18T22:44:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.389+0000 7f95fd3f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559c2abc3d70 con 0x559c2acb72f0 2024-03-18T22:44:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.389+0000 7f95fd3f8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559c2ac17550 con 0x559c2acb7ce0 2024-03-18T22:44:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.390+0000 7f95f8da8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acbce50 0x559c2acbf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.390+0000 7f95f8da8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acbce50 0x559c2acbf2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33702/0 (socket says 172.21.15.40:33702) 2024-03-18T22:44:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.390+0000 7f95f8da8640 1 -- 172.21.15.40:0/3365830591 learned_addr learned my addr 172.21.15.40:0/3365830591 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.390+0000 7f95f3fff640 1 --2- 172.21.15.40:0/3365830591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acb72f0 0x559c2acb7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.390+0000 7f95f37fe640 1 --2- 172.21.15.40:0/3365830591 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acb7ce0 0x559c2acbc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.390+0000 7f95f8da8640 1 -- 172.21.15.40:0/3365830591 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acb7ce0 msgr2=0x559c2acbc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.390+0000 7f95f8da8640 1 --2- 172.21.15.40:0/3365830591 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acb7ce0 0x559c2acbc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.390+0000 7f95f8da8640 1 -- 172.21.15.40:0/3365830591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acb72f0 msgr2=0x559c2acb7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.390+0000 7f95f8da8640 1 --2- 172.21.15.40:0/3365830591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acb72f0 0x559c2acb7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.390+0000 7f95f8da8640 1 -- 172.21.15.40:0/3365830591 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c2abe6210 con 0x559c2acbce50 2024-03-18T22:44:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.390+0000 7f95f3fff640 1 --2- 172.21.15.40:0/3365830591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acb72f0 0x559c2acb7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.390+0000 7f95f37fe640 1 --2- 172.21.15.40:0/4097826703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f95dc0a2800 0x7f95dc0a4cc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f95f4109280 tx=0x7f95e8011a00 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.390+0000 7f95f8da8640 1 --2- 172.21.15.40:0/3365830591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acbce50 0x559c2acbf2e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f95ec009fd0 tx=0x7f95ec011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65cbeec6c824498a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.390+0000 7f95da7fc640 1 -- 172.21.15.40:0/3365830591 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec004d20 con 0x559c2acbce50 2024-03-18T22:44:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.390+0000 7f95da7fc640 1 -- 172.21.15.40:0/3365830591 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f95ec004050 con 0x559c2acbce50 2024-03-18T22:44:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.391+0000 7f95da7fc640 1 -- 172.21.15.40:0/3365830591 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec005280 con 0x559c2acbce50 2024-03-18T22:44:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.391+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3365830591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acbce50 msgr2=0x559c2acbf2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.391+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/3365830591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acbce50 0x559c2acbf2e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f95ec009fd0 tx=0x7f95ec011810 comp rx=0 tx=0).stop 2024-03-18T22:44:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.391+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3365830591 shutdown_connections 2024-03-18T22:44:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.391+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/3365830591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acbce50 0x559c2acbf2e0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.391+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/3365830591 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acb7ce0 0x559c2acbc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.391+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/3365830591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acb72f0 0x559c2acb7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.391+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3365830591 >> 172.21.15.40:0/3365830591 conn(0x559c2acb2860 msgr2=0x559c2acb2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.391+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3365830591 shutdown_connections 2024-03-18T22:44:47.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.391+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3365830591 wait complete. 2024-03-18T22:44:47.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.392+0000 7f95fd3f8740 1 -- start start 2024-03-18T22:44:47.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.392+0000 7f95fd3f8740 1 Processor -- start 2024-03-18T22:44:47.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.392+0000 7f95fd3f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb72f0 0x559c2acc17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.392+0000 7f95fd3f8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acb7ce0 0x559c2acc1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.392+0000 7f95fd3f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acbce50 0x559c2acc8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.392+0000 7f95fd3f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559c2acbfe00 con 0x559c2acb72f0 2024-03-18T22:44:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.392+0000 7f95fd3f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559c2abd7310 con 0x559c2acbce50 2024-03-18T22:44:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.392+0000 7f95fd3f8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559c2acbff80 con 0x559c2acb7ce0 2024-03-18T22:44:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95f3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb72f0 0x559c2acc17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95f37fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acb7ce0 0x559c2acc1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95fd3f8740 10 client.0 _ll_register_callbacks cb 0x559c2abcb2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95f37fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acb7ce0 0x559c2acc1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41444/0 (socket says 172.21.15.40:41444) 2024-03-18T22:44:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95f37fe640 1 -- 172.21.15.40:0/1310825924 learned_addr learned my addr 172.21.15.40:0/1310825924 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95f8da8640 1 --2- 172.21.15.40:0/1310825924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acbce50 0x559c2acc8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95f37fe640 1 -- 172.21.15.40:0/1310825924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acbce50 msgr2=0x559c2acc8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95f37fe640 1 --2- 172.21.15.40:0/1310825924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acbce50 0x559c2acc8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95f37fe640 1 -- 172.21.15.40:0/1310825924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb72f0 msgr2=0x559c2acc17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95f37fe640 1 --2- 172.21.15.40:0/1310825924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb72f0 0x559c2acc17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95f37fe640 1 -- 172.21.15.40:0/1310825924 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c2a92f3f0 con 0x559c2acb7ce0 2024-03-18T22:44:47.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95f3fff640 1 --2- 172.21.15.40:0/1310825924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb72f0 0x559c2acc17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:47.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95f37fe640 1 --2- 172.21.15.40:0/1310825924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acb7ce0 0x559c2acc1d20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f95e80044f0 tx=0x7f95e8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d27199dcf6a9769 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:44:47.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95d37fe640 1 -- 172.21.15.40:0/1310825924 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e801a040 con 0x559c2acb7ce0 2024-03-18T22:44:47.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95d37fe640 1 -- 172.21.15.40:0/1310825924 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f95e8003a00 con 0x559c2acb7ce0 2024-03-18T22:44:47.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.393+0000 7f95fd3f8740 1 -- 172.21.15.40:0/1310825924 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95ec004050 con 0x559c2acb7ce0 2024-03-18T22:44:47.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.394+0000 7f95d37fe640 1 -- 172.21.15.40:0/1310825924 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f95e8003c00 con 0x559c2acb7ce0 2024-03-18T22:44:47.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.394+0000 7f95fd3f8740 10 client.25834 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:47.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.394+0000 7f95fd3f8740 1 -- 172.21.15.40:0/1310825924 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559c2a9f97d0 con 0x559c2acb7ce0 2024-03-18T22:44:47.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.394+0000 7f95d37fe640 1 -- 172.21.15.40:0/1310825924 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f95ec004050 con 0x559c2acb7ce0 2024-03-18T22:44:47.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.394+0000 7f95d37fe640 1 -- 172.21.15.40:0/1310825924 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f95e801d030 con 0x559c2acb7ce0 2024-03-18T22:44:47.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.394+0000 7f95fd3f8740 10 client.25834 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:47.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.394+0000 7f95fd3f8740 10 client.25834 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:44:47.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.395+0000 7f95fd3f8740 20 client.25834 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:47.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.395+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/1310825924 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559c2accd730 0x559c2accfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.395+0000 7f95fd3f8740 4 client.25834 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:47.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.395+0000 7f95fd3f8740 1 -- 172.21.15.40:0/1310825924 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c2ac793e0 con 0x559c2accd730 2024-03-18T22:44:47.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.395+0000 7f95f3fff640 1 --2- 172.21.15.40:0/1310825924 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559c2accd730 0x559c2accfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.395+0000 7f95d37fe640 10 client.25834 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:44:47.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.395+0000 7f95f3fff640 1 --2- 172.21.15.40:0/1310825924 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559c2accd730 0x559c2accfbc0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9ce413e0901c4e84 server_cookie=833946c6bdc85b06 in_seq=0 out_seq=0 2024-03-18T22:44:47.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.396+0000 7f95d37fe640 1 -- 172.21.15.40:0/1310825924 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559c2ac793e0 con 0x559c2accd730 2024-03-18T22:44:47.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.396+0000 7f95d37fe640 10 client.25834 handle_command_reply: tid=0 2024-03-18T22:44:47.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.396+0000 7f95fd3f8740 1 client.25834 shutdown 2024-03-18T22:44:47.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.396+0000 7f95fd3f8740 1 -- 172.21.15.40:0/1310825924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acb7ce0 msgr2=0x559c2acc1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.396+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/1310825924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acb7ce0 0x559c2acc1d20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f95e80044f0 tx=0x7f95e8004730 comp rx=0 tx=0).stop 2024-03-18T22:44:47.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.396+0000 7f95fd3f8740 1 -- 172.21.15.40:0/1310825924 shutdown_connections 2024-03-18T22:44:47.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.396+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/1310825924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acbce50 0x559c2acc8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.396+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/1310825924 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acb7ce0 0x559c2acc1d20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.396+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/1310825924 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559c2accd730 0x559c2accfbc0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.396+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/1310825924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb72f0 0x559c2acc17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.396+0000 7f95d37fe640 0 client.25834 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:44:47.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.396+0000 7f95fd3f8740 1 -- 172.21.15.40:0/1310825924 >> 172.21.15.40:0/1310825924 conn(0x559c2acb2860 msgr2=0x559c2acb1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.396+0000 7f95fd3f8740 1 -- 172.21.15.40:0/1310825924 shutdown_connections 2024-03-18T22:44:47.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.396+0000 7f95fd3f8740 1 -- 172.21.15.40:0/1310825924 wait complete. 2024-03-18T22:44:47.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.396+0000 7f95fd3f8740 20 client.25834 trim_cache size 0 max 16384 2024-03-18T22:44:47.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95fd3f8740 1 Processor -- start 2024-03-18T22:44:47.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95fd3f8740 1 -- start start 2024-03-18T22:44:47.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95fd3f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acb72f0 0x559c2acb2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95fd3f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb7ce0 0x559c2acc8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95fd3f8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acbce50 0x559c2acb1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95fd3f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f95f41aa460 con 0x559c2acb7ce0 2024-03-18T22:44:47.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95fd3f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f95f41ac060 con 0x559c2acb72f0 2024-03-18T22:44:47.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95fd3f8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f95f41aab30 con 0x559c2acbce50 2024-03-18T22:44:47.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95f37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb7ce0 0x559c2acc8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95f37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb7ce0 0x559c2acc8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33728/0 (socket says 172.21.15.40:33728) 2024-03-18T22:44:47.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95f37fe640 1 -- 172.21.15.40:0/3000667039 learned_addr learned my addr 172.21.15.40:0/3000667039 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95f8da8640 1 --2- 172.21.15.40:0/3000667039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acbce50 0x559c2acb1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95f37fe640 1 -- 172.21.15.40:0/3000667039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acbce50 msgr2=0x559c2acb1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95f37fe640 1 --2- 172.21.15.40:0/3000667039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acbce50 0x559c2acb1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95f37fe640 1 -- 172.21.15.40:0/3000667039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acb72f0 msgr2=0x559c2acb2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95f37fe640 1 --2- 172.21.15.40:0/3000667039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acb72f0 0x559c2acb2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.415+0000 7f95f37fe640 1 -- 172.21.15.40:0/3000667039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c2ac88880 con 0x559c2acb7ce0 2024-03-18T22:44:47.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.416+0000 7f95f37fe640 1 --2- 172.21.15.40:0/3000667039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb7ce0 0x559c2acc8d20 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f95e8004700 tx=0x7f95e8005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219b5592b3b1599e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.416+0000 7f95d9ffb640 1 -- 172.21.15.40:0/3000667039 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e803b070 con 0x559c2acb7ce0 2024-03-18T22:44:47.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.416+0000 7f95d9ffb640 1 -- 172.21.15.40:0/3000667039 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f95e8028e20 con 0x559c2acb7ce0 2024-03-18T22:44:47.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.416+0000 7f95d9ffb640 1 -- 172.21.15.40:0/3000667039 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e801a040 con 0x559c2acb7ce0 2024-03-18T22:44:47.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.417+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3000667039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb7ce0 msgr2=0x559c2acc8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.417+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/3000667039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb7ce0 0x559c2acc8d20 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f95e8004700 tx=0x7f95e8005f10 comp rx=0 tx=0).stop 2024-03-18T22:44:47.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.417+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3000667039 shutdown_connections 2024-03-18T22:44:47.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.417+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/3000667039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acbce50 0x559c2acb1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.417+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/3000667039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb7ce0 0x559c2acc8d20 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.417+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/3000667039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acb72f0 0x559c2acb2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.417+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3000667039 >> 172.21.15.40:0/3000667039 conn(0x559c2acb2860 msgr2=0x559c2acb86e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.417+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3000667039 shutdown_connections 2024-03-18T22:44:47.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.417+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3000667039 wait complete. 2024-03-18T22:44:47.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.418+0000 7f95fd3f8740 1 -- start start 2024-03-18T22:44:47.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.418+0000 7f95fd3f8740 1 Processor -- start 2024-03-18T22:44:47.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.418+0000 7f95fd3f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb72f0 0x559c2acc6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.418+0000 7f95fd3f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acb7ce0 0x559c2acc7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.418+0000 7f95fd3f8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acbce50 0x559c2acc8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.418+0000 7f95fd3f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f95f41ab8d0 con 0x559c2acb72f0 2024-03-18T22:44:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.418+0000 7f95fd3f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f95f41ab200 con 0x559c2acb7ce0 2024-03-18T22:44:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.418+0000 7f95fd3f8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f95f41a8440 con 0x559c2acbce50 2024-03-18T22:44:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.418+0000 7f95fd3f8740 10 client.0 _ll_register_callbacks cb 0x7f95e8009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.419+0000 7f95f3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb72f0 0x559c2acc6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.419+0000 7f95f3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb72f0 0x559c2acc6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33732/0 (socket says 172.21.15.40:33732) 2024-03-18T22:44:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.419+0000 7f95f3fff640 1 -- 172.21.15.40:0/3642667779 learned_addr learned my addr 172.21.15.40:0/3642667779 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.419+0000 7f95f37fe640 1 --2- 172.21.15.40:0/3642667779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acb7ce0 0x559c2acc7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.419+0000 7f95f8da8640 1 --2- 172.21.15.40:0/3642667779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acbce50 0x559c2acc8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.419+0000 7f95f3fff640 1 -- 172.21.15.40:0/3642667779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acbce50 msgr2=0x559c2acc8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.419+0000 7f95f3fff640 1 --2- 172.21.15.40:0/3642667779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acbce50 0x559c2acc8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.419+0000 7f95f3fff640 1 -- 172.21.15.40:0/3642667779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acb7ce0 msgr2=0x559c2acc7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.419+0000 7f95f3fff640 1 --2- 172.21.15.40:0/3642667779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acb7ce0 0x559c2acc7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.419+0000 7f95f3fff640 1 -- 172.21.15.40:0/3642667779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c2acc8640 con 0x559c2acb72f0 2024-03-18T22:44:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.419+0000 7f95f3fff640 1 --2- 172.21.15.40:0/3642667779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb72f0 0x559c2acc6b00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f95e40b9e30 tx=0x7f95e40bdc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d56ab7c74d6692 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.419+0000 7f95d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:44:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.420+0000 7f95d37fe640 1 -- 172.21.15.40:0/3642667779 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e40d6060 con 0x559c2acb72f0 2024-03-18T22:44:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.420+0000 7f95d37fe640 1 -- 172.21.15.40:0/3642667779 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f95e40054d0 con 0x559c2acb72f0 2024-03-18T22:44:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.420+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3642667779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95e8028e20 con 0x559c2acb72f0 2024-03-18T22:44:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.420+0000 7f95d37fe640 1 -- 172.21.15.40:0/3642667779 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f95e40a6070 con 0x559c2acb72f0 2024-03-18T22:44:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.420+0000 7f95fd3f8740 10 client.15996 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.420+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3642667779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559c2ad34250 con 0x559c2acb72f0 2024-03-18T22:44:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.420+0000 7f95d37fe640 1 -- 172.21.15.40:0/3642667779 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f95e8028e20 con 0x559c2acb72f0 2024-03-18T22:44:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.420+0000 7f95d37fe640 1 -- 172.21.15.40:0/3642667779 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f95e40c1170 con 0x559c2acb72f0 2024-03-18T22:44:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.420+0000 7f95fd3f8740 10 client.15996 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.420+0000 7f95fd3f8740 10 client.15996 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:44:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.420+0000 7f95fd3f8740 20 client.15996 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.420+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/3642667779 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559c2ad34b30 0x559c2ad36f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.420+0000 7f95fd3f8740 4 client.15996 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:44:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.420+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3642667779 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559c2ac289f0 con 0x559c2ad34b30 2024-03-18T22:44:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.420+0000 7f95f37fe640 1 --2- 172.21.15.40:0/3642667779 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559c2ad34b30 0x559c2ad36f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.421+0000 7f95d37fe640 10 client.15996 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:44:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.421+0000 7f95f37fe640 1 --2- 172.21.15.40:0/3642667779 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559c2ad34b30 0x559c2ad36f50 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fee317d762dc0261 server_cookie=b4a9fb6334c95bc5 in_seq=0 out_seq=0 2024-03-18T22:44:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.421+0000 7f95d37fe640 1 -- 172.21.15.40:0/3642667779 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559c2ac289f0 con 0x559c2ad34b30 2024-03-18T22:44:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.421+0000 7f95d37fe640 10 client.15996 handle_command_reply: tid=0 2024-03-18T22:44:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.421+0000 7f95fd3f8740 1 client.15996 shutdown 2024-03-18T22:44:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.421+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3642667779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb72f0 msgr2=0x559c2acc6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.421+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/3642667779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb72f0 0x559c2acc6b00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f95e40b9e30 tx=0x7f95e40bdc20 comp rx=0 tx=0).stop 2024-03-18T22:44:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.421+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3642667779 shutdown_connections 2024-03-18T22:44:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.421+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/3642667779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559c2acbce50 0x559c2acc8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.421+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/3642667779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559c2acb7ce0 0x559c2acc7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.421+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/3642667779 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559c2ad34b30 0x559c2ad36f50 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.421+0000 7f95fd3f8740 1 --2- 172.21.15.40:0/3642667779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559c2acb72f0 0x559c2acc6b00 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.421+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3642667779 >> 172.21.15.40:0/3642667779 conn(0x559c2acb2860 msgr2=0x559c2acb2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.421+0000 7f95d37fe640 0 client.15996 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:44:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.422+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3642667779 shutdown_connections 2024-03-18T22:44:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.422+0000 7f95fd3f8740 1 -- 172.21.15.40:0/3642667779 wait complete. 2024-03-18T22:44:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.422+0000 7f95fd3f8740 20 client.15996 trim_cache size 0 max 16384 2024-03-18T22:44:47.427 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:44:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.422+0000 7f95fa80b640 1 -- 172.21.15.40:0/4097826703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f95dc0a2800 msgr2=0x7f95dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.422+0000 7f95fa80b640 1 --2- 172.21.15.40:0/4097826703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f95dc0a2800 0x7f95dc0a4cc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f95f4109280 tx=0x7f95e8011a00 comp rx=0 tx=0).stop 2024-03-18T22:44:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.422+0000 7f95fa80b640 1 -- 172.21.15.40:0/4097826703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f40ffe90 msgr2=0x7f95f4109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.422+0000 7f95fa80b640 1 --2- 172.21.15.40:0/4097826703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f40ffe90 0x7f95f4109510 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f95e4000c00 tx=0x7f95e4005dc0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.422+0000 7f95fa80b640 1 -- 172.21.15.40:0/4097826703 shutdown_connections 2024-03-18T22:44:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.422+0000 7f95fa80b640 1 --2- 172.21.15.40:0/4097826703 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f95dc0a2800 0x7f95dc0a4cc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.422+0000 7f95fa80b640 1 --2- 172.21.15.40:0/4097826703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f95f41019a0 0x7f95f419ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.422+0000 7f95fa80b640 1 --2- 172.21.15.40:0/4097826703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95f4100fe0 0x7f95f419c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.422+0000 7f95fa80b640 1 --2- 172.21.15.40:0/4097826703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f95f40ffe90 0x7f95f4109510 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.422+0000 7f95fa80b640 1 -- 172.21.15.40:0/4097826703 >> 172.21.15.40:0/4097826703 conn(0x7f95f40fb400 msgr2=0x7f95f41043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.423+0000 7f95fa80b640 1 -- 172.21.15.40:0/4097826703 shutdown_connections 2024-03-18T22:44:47.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.423+0000 7f95fa80b640 1 -- 172.21.15.40:0/4097826703 wait complete. 2024-03-18T22:44:47.436 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:44:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.551+0000 7fd965753640 1 -- 172.21.15.40:0/1416288667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9601022d0 msgr2=0x7fd960108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.551+0000 7fd965753640 1 --2- 172.21.15.40:0/1416288667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9601022d0 0x7fd960108b60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd958009be0 tx=0x7fd95802bef0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.551+0000 7fd965753640 1 -- 172.21.15.40:0/1416288667 shutdown_connections 2024-03-18T22:44:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.551+0000 7fd965753640 1 --2- 172.21.15.40:0/1416288667 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9601022d0 0x7fd960108b60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.551+0000 7fd965753640 1 --2- 172.21.15.40:0/1416288667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960101910 0x7fd960101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.551+0000 7fd965753640 1 --2- 172.21.15.40:0/1416288667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd960100720 0x7fd960100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.551+0000 7fd965753640 1 -- 172.21.15.40:0/1416288667 >> 172.21.15.40:0/1416288667 conn(0x7fd9600fbc90 msgr2=0x7fd9600fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.551+0000 7fd965753640 1 -- 172.21.15.40:0/1416288667 shutdown_connections 2024-03-18T22:44:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.551+0000 7fd965753640 1 -- 172.21.15.40:0/1416288667 wait complete. 2024-03-18T22:44:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.552+0000 7fd965753640 1 Processor -- start 2024-03-18T22:44:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.552+0000 7fd965753640 1 -- start start 2024-03-18T22:44:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.552+0000 7fd965753640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd960100720 0x7fd960197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.552+0000 7fd965753640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960101910 0x7fd960197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.552+0000 7fd965753640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9601022d0 0x7fd96019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.552+0000 7fd965753640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd96010c230 con 0x7fd960101910 2024-03-18T22:44:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.552+0000 7fd965753640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd96010c0b0 con 0x7fd9601022d0 2024-03-18T22:44:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.552+0000 7fd965753640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd96010c3b0 con 0x7fd960100720 2024-03-18T22:44:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.552+0000 7fd957fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960101910 0x7fd960197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.552+0000 7fd95effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd960100720 0x7fd960197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.552+0000 7fd95effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd960100720 0x7fd960197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41484/0 (socket says 172.21.15.40:41484) 2024-03-18T22:44:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.552+0000 7fd957fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960101910 0x7fd960197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33750/0 (socket says 172.21.15.40:33750) 2024-03-18T22:44:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.552+0000 7fd95effd640 1 -- 172.21.15.40:0/2658693383 learned_addr learned my addr 172.21.15.40:0/2658693383 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.552+0000 7fd95f7fe640 1 --2- 172.21.15.40:0/2658693383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9601022d0 0x7fd96019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.553+0000 7fd95effd640 1 -- 172.21.15.40:0/2658693383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9601022d0 msgr2=0x7fd96019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.553+0000 7fd95effd640 1 --2- 172.21.15.40:0/2658693383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9601022d0 0x7fd96019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.553+0000 7fd95effd640 1 -- 172.21.15.40:0/2658693383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960101910 msgr2=0x7fd960197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.553+0000 7fd95effd640 1 --2- 172.21.15.40:0/2658693383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960101910 0x7fd960197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.553+0000 7fd95effd640 1 -- 172.21.15.40:0/2658693383 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96019f620 con 0x7fd960100720 2024-03-18T22:44:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.553+0000 7fd957fff640 1 --2- 172.21.15.40:0/2658693383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960101910 0x7fd960197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.553+0000 7fd95effd640 1 --2- 172.21.15.40:0/2658693383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd960100720 0x7fd960197a00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd95000e9c0 tx=0x7fd95000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.553+0000 7fd95cff9640 1 -- 172.21.15.40:0/2658693383 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95000cd80 con 0x7fd960100720 2024-03-18T22:44:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.553+0000 7fd95cff9640 1 -- 172.21.15.40:0/2658693383 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9500035f0 con 0x7fd960100720 2024-03-18T22:44:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.553+0000 7fd95cff9640 1 -- 172.21.15.40:0/2658693383 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd950010690 con 0x7fd960100720 2024-03-18T22:44:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.553+0000 7fd965753640 1 -- 172.21.15.40:0/2658693383 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96019f910 con 0x7fd960100720 2024-03-18T22:44:47.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.553+0000 7fd965753640 1 -- 172.21.15.40:0/2658693383 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd96019fe60 con 0x7fd960100720 2024-03-18T22:44:47.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.556+0000 7fd96841e740 1 Processor -- start 2024-03-18T22:44:47.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.556+0000 7fd96841e740 1 -- start start 2024-03-18T22:44:47.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.556+0000 7fd95cff9640 1 -- 172.21.15.40:0/2658693383 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd9500037a0 con 0x7fd960100720 2024-03-18T22:44:47.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.556+0000 7fd96841e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 0x56271d091710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.556+0000 7fd96841e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d0967e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.556+0000 7fd96841e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d0992e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.556+0000 7fd96841e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56271cff54b0 con 0x56271d096e50 2024-03-18T22:44:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.556+0000 7fd96841e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56271cf9dd70 con 0x56271d0912f0 2024-03-18T22:44:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.556+0000 7fd96841e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56271cff1550 con 0x56271d091ce0 2024-03-18T22:44:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd95effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d0967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd95f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 0x56271d091710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd957fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d0992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd95effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d0967e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41488/0 (socket says 172.21.15.40:41488) 2024-03-18T22:44:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd95f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 0x56271d091710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:51038/0 (socket says 172.21.15.40:51038) 2024-03-18T22:44:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd95effd640 1 -- 172.21.15.40:0/2311772335 learned_addr learned my addr 172.21.15.40:0/2311772335 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd95effd640 1 -- 172.21.15.40:0/2311772335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 msgr2=0x56271d091710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd95effd640 1 --2- 172.21.15.40:0/2311772335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 0x56271d091710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd95effd640 1 -- 172.21.15.40:0/2311772335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 msgr2=0x56271d0992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd95effd640 1 --2- 172.21.15.40:0/2311772335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d0992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd95effd640 1 -- 172.21.15.40:0/2311772335 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56271cfc0210 con 0x56271d091ce0 2024-03-18T22:44:47.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd95cff9640 1 --2- 172.21.15.40:0/2658693383 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd94c0a2800 0x7fd94c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd95effd640 1 --2- 172.21.15.40:0/2311772335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d0967e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd9500d7260 tx=0x7fd9500d7350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c0daed4dc6c4978 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd95f7fe640 1 --2- 172.21.15.40:0/2658693383 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd94c0a2800 0x7fd94c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd95cff9640 1 -- 172.21.15.40:0/2658693383 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd9500cf1e0 con 0x7fd960100720 2024-03-18T22:44:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd9557fa640 1 -- 172.21.15.40:0/2311772335 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd950097a30 con 0x56271d091ce0 2024-03-18T22:44:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd9557fa640 1 -- 172.21.15.40:0/2311772335 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd950003b30 con 0x56271d091ce0 2024-03-18T22:44:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.557+0000 7fd9557fa640 1 -- 172.21.15.40:0/2311772335 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95009b0f0 con 0x56271d091ce0 2024-03-18T22:44:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.558+0000 7fd96841e740 1 -- 172.21.15.40:0/2311772335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 msgr2=0x56271d0967e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.558+0000 7fd96841e740 1 --2- 172.21.15.40:0/2311772335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d0967e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd9500d7260 tx=0x7fd9500d7350 comp rx=0 tx=0).stop 2024-03-18T22:44:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.558+0000 7fd95f7fe640 1 --2- 172.21.15.40:0/2658693383 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd94c0a2800 0x7fd94c0a4cc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd958004af0 tx=0x7fd958005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.558+0000 7fd96841e740 1 -- 172.21.15.40:0/2311772335 shutdown_connections 2024-03-18T22:44:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.558+0000 7fd96841e740 1 --2- 172.21.15.40:0/2311772335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d0992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.558+0000 7fd96841e740 1 --2- 172.21.15.40:0/2311772335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d0967e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.558+0000 7fd96841e740 1 --2- 172.21.15.40:0/2311772335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 0x56271d091710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.558+0000 7fd96841e740 1 -- 172.21.15.40:0/2311772335 >> 172.21.15.40:0/2311772335 conn(0x56271d08c860 msgr2=0x56271d08cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.558+0000 7fd96841e740 1 -- 172.21.15.40:0/2311772335 shutdown_connections 2024-03-18T22:44:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.558+0000 7fd96841e740 1 -- 172.21.15.40:0/2311772335 wait complete. 2024-03-18T22:44:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd96841e740 1 -- start start 2024-03-18T22:44:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd96841e740 1 Processor -- start 2024-03-18T22:44:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd96841e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 0x56271d09b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd96841e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d09bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd96841e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d0a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd96841e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56271d099e00 con 0x56271d096e50 2024-03-18T22:44:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd96841e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56271cfb1310 con 0x56271d0912f0 2024-03-18T22:44:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd96841e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56271d099f80 con 0x56271d091ce0 2024-03-18T22:44:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd96841e740 10 client.0 _ll_register_callbacks cb 0x56271cfa52d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd95f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d0a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd95effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 0x56271d09b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd957fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d09bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd95effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 0x56271d09b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:51050/0 (socket says 172.21.15.40:51050) 2024-03-18T22:44:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd95effd640 1 -- 172.21.15.40:0/530914594 learned_addr learned my addr 172.21.15.40:0/530914594 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd95f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d0a2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33756/0 (socket says 172.21.15.40:33756) 2024-03-18T22:44:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd95effd640 1 -- 172.21.15.40:0/530914594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 msgr2=0x56271d09bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.559+0000 7fd95effd640 1 --2- 172.21.15.40:0/530914594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d09bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.560+0000 7fd95effd640 1 -- 172.21.15.40:0/530914594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 msgr2=0x56271d0a2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.560+0000 7fd95effd640 1 --2- 172.21.15.40:0/530914594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d0a2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.560+0000 7fd95effd640 1 -- 172.21.15.40:0/530914594 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56271cd093f0 con 0x56271d0912f0 2024-03-18T22:44:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.560+0000 7fd957fff640 1 --2- 172.21.15.40:0/530914594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d09bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.560+0000 7fd95f7fe640 1 --2- 172.21.15.40:0/530914594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d0a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.560+0000 7fd95effd640 1 --2- 172.21.15.40:0/530914594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 0x56271d09b860 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd9500ccc60 tx=0x7fd950003d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a05e8cbf3d5d1de6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.560+0000 7fd9467fc640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:44:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.560+0000 7fd9467fc640 1 -- 172.21.15.40:0/530914594 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9500b4060 con 0x56271d0912f0 2024-03-18T22:44:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.560+0000 7fd96841e740 1 -- 172.21.15.40:0/530914594 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56271d0533e0 con 0x56271d0912f0 2024-03-18T22:44:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.560+0000 7fd9467fc640 1 -- 172.21.15.40:0/530914594 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd9500860b0 con 0x56271d0912f0 2024-03-18T22:44:47.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.560+0000 7fd9467fc640 1 -- 172.21.15.40:0/530914594 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd95009b2d0 con 0x56271d0912f0 2024-03-18T22:44:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.560+0000 7fd96841e740 10 client.25865 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.560+0000 7fd96841e740 1 -- 172.21.15.40:0/530914594 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56271cdd37d0 con 0x56271d0912f0 2024-03-18T22:44:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.560+0000 7fd9467fc640 1 -- 172.21.15.40:0/530914594 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56271d0533e0 con 0x56271d0912f0 2024-03-18T22:44:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.561+0000 7fd9467fc640 1 -- 172.21.15.40:0/530914594 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fd9500a3aa0 con 0x56271d0912f0 2024-03-18T22:44:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.561+0000 7fd96841e740 10 client.25865 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.561+0000 7fd96841e740 10 client.25865 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:44:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.561+0000 7fd96841e740 20 client.25865 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.561+0000 7fd96841e740 1 --2- 172.21.15.40:0/530914594 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56271d0a7260 0x56271d0a9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.561+0000 7fd96841e740 4 client.25865 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.561+0000 7fd96841e740 1 -- 172.21.15.40:0/530914594 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56271d0029f0 con 0x56271d0a7260 2024-03-18T22:44:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.561+0000 7fd957fff640 1 --2- 172.21.15.40:0/530914594 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56271d0a7260 0x56271d0a9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.561+0000 7fd9467fc640 10 client.25865 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:44:47.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.561+0000 7fd957fff640 1 --2- 172.21.15.40:0/530914594 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56271d0a7260 0x56271d0a9650 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=72e590e72f83d5df server_cookie=43e9b55bb95d8b35 in_seq=0 out_seq=0 2024-03-18T22:44:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.562+0000 7fd9467fc640 1 -- 172.21.15.40:0/530914594 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56271d0029f0 con 0x56271d0a7260 2024-03-18T22:44:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.562+0000 7fd9467fc640 10 client.25865 handle_command_reply: tid=0 2024-03-18T22:44:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.562+0000 7fd96841e740 1 client.25865 shutdown 2024-03-18T22:44:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.562+0000 7fd96841e740 1 -- 172.21.15.40:0/530914594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 msgr2=0x56271d09b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.562+0000 7fd96841e740 1 --2- 172.21.15.40:0/530914594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 0x56271d09b860 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd9500ccc60 tx=0x7fd950003d80 comp rx=0 tx=0).stop 2024-03-18T22:44:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.562+0000 7fd96841e740 1 -- 172.21.15.40:0/530914594 shutdown_connections 2024-03-18T22:44:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.562+0000 7fd96841e740 1 --2- 172.21.15.40:0/530914594 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56271d0a7260 0x56271d0a9650 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.562+0000 7fd96841e740 1 --2- 172.21.15.40:0/530914594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d0a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.562+0000 7fd96841e740 1 --2- 172.21.15.40:0/530914594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d09bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.562+0000 7fd9467fc640 0 client.25865 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:44:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.562+0000 7fd96841e740 1 --2- 172.21.15.40:0/530914594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 0x56271d09b860 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.562+0000 7fd96841e740 1 -- 172.21.15.40:0/530914594 >> 172.21.15.40:0/530914594 conn(0x56271d08c860 msgr2=0x56271d08bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.562+0000 7fd96841e740 1 -- 172.21.15.40:0/530914594 shutdown_connections 2024-03-18T22:44:47.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.562+0000 7fd96841e740 1 -- 172.21.15.40:0/530914594 wait complete. 2024-03-18T22:44:47.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.562+0000 7fd96841e740 20 client.25865 trim_cache size 0 max 16384 2024-03-18T22:44:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.579+0000 7fd96841e740 1 Processor -- start 2024-03-18T22:44:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.579+0000 7fd96841e740 1 -- start start 2024-03-18T22:44:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd96841e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d0912f0 0x56271d087a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd96841e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d091ce0 0x56271d0a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd96841e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d096e50 0x56271d09b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd96841e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd9601aa4e0 con 0x56271d091ce0 2024-03-18T22:44:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd96841e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd9601ac0e0 con 0x56271d096e50 2024-03-18T22:44:47.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd96841e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd9601aabb0 con 0x56271d0912f0 2024-03-18T22:44:47.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd95effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d0912f0 0x56271d087a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd95effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d0912f0 0x56271d087a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41502/0 (socket says 172.21.15.40:41502) 2024-03-18T22:44:47.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd957fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d091ce0 0x56271d0a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd95effd640 1 -- 172.21.15.40:0/1268302326 learned_addr learned my addr 172.21.15.40:0/1268302326 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd95effd640 1 -- 172.21.15.40:0/1268302326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d096e50 msgr2=0x56271d09b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd95effd640 1 --2- 172.21.15.40:0/1268302326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d096e50 0x56271d09b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd95effd640 1 -- 172.21.15.40:0/1268302326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d091ce0 msgr2=0x56271d0a2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd95effd640 1 --2- 172.21.15.40:0/1268302326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d091ce0 0x56271d0a2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd95effd640 1 -- 172.21.15.40:0/1268302326 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56271d062880 con 0x56271d0912f0 2024-03-18T22:44:47.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd95effd640 1 --2- 172.21.15.40:0/1268302326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d0912f0 0x56271d087a80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd950097bf0 tx=0x7fd9500d7350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=516b4baa70bd78e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.580+0000 7fd954ff9640 1 -- 172.21.15.40:0/1268302326 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9500b4040 con 0x56271d0912f0 2024-03-18T22:44:47.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.581+0000 7fd954ff9640 1 -- 172.21.15.40:0/1268302326 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9500d7850 con 0x56271d0912f0 2024-03-18T22:44:47.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.581+0000 7fd954ff9640 1 -- 172.21.15.40:0/1268302326 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95009b560 con 0x56271d0912f0 2024-03-18T22:44:47.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.582+0000 7fd96841e740 1 -- 172.21.15.40:0/1268302326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d0912f0 msgr2=0x56271d087a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.582+0000 7fd96841e740 1 --2- 172.21.15.40:0/1268302326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d0912f0 0x56271d087a80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd950097bf0 tx=0x7fd9500d7350 comp rx=0 tx=0).stop 2024-03-18T22:44:47.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.582+0000 7fd96841e740 1 -- 172.21.15.40:0/1268302326 shutdown_connections 2024-03-18T22:44:47.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.582+0000 7fd96841e740 1 --2- 172.21.15.40:0/1268302326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d096e50 0x56271d09b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.582+0000 7fd96841e740 1 --2- 172.21.15.40:0/1268302326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d091ce0 0x56271d0a2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.582+0000 7fd96841e740 1 --2- 172.21.15.40:0/1268302326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d0912f0 0x56271d087a80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.582+0000 7fd96841e740 1 -- 172.21.15.40:0/1268302326 >> 172.21.15.40:0/1268302326 conn(0x56271d08c860 msgr2=0x56271d08cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.582+0000 7fd96841e740 1 -- 172.21.15.40:0/1268302326 shutdown_connections 2024-03-18T22:44:47.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.582+0000 7fd96841e740 1 -- 172.21.15.40:0/1268302326 wait complete. 2024-03-18T22:44:47.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.582+0000 7fd96841e740 1 -- start start 2024-03-18T22:44:47.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.582+0000 7fd96841e740 1 Processor -- start 2024-03-18T22:44:47.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd96841e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 0x56271d0a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd96841e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d09f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd96841e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d09fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd96841e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd9601ab950 con 0x56271d096e50 2024-03-18T22:44:47.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd96841e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd9601ab280 con 0x56271d0912f0 2024-03-18T22:44:47.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd96841e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd9601a84c0 con 0x56271d091ce0 2024-03-18T22:44:47.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd96841e740 10 client.0 _ll_register_callbacks cb 0x7fd948009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:47.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd957fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d09f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd95f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d09fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd957fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d09f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41508/0 (socket says 172.21.15.40:41508) 2024-03-18T22:44:47.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd95f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d09fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33778/0 (socket says 172.21.15.40:33778) 2024-03-18T22:44:47.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd957fff640 1 -- 172.21.15.40:0/1836250732 learned_addr learned my addr 172.21.15.40:0/1836250732 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd95effd640 1 --2- 172.21.15.40:0/1836250732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 0x56271d0a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd957fff640 1 -- 172.21.15.40:0/1836250732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 msgr2=0x56271d0a2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd957fff640 1 --2- 172.21.15.40:0/1836250732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 0x56271d0a2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd957fff640 1 -- 172.21.15.40:0/1836250732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 msgr2=0x56271d09fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd957fff640 1 --2- 172.21.15.40:0/1836250732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d09fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd957fff640 1 -- 172.21.15.40:0/1836250732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56271d10b3b0 con 0x56271d091ce0 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd95f7fe640 1 --2- 172.21.15.40:0/1836250732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d09fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd957fff640 1 --2- 172.21.15.40:0/1836250732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d09f640 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd948004700 tx=0x7fd94801f7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7f9d884c8081379 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.583+0000 7fd9467fc640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.584+0000 7fd9467fc640 1 -- 172.21.15.40:0/1836250732 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd948005020 con 0x56271d091ce0 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.584+0000 7fd9467fc640 1 -- 172.21.15.40:0/1836250732 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd9480051c0 con 0x56271d091ce0 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.584+0000 7fd96841e740 1 -- 172.21.15.40:0/1836250732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9500d7850 con 0x56271d091ce0 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.584+0000 7fd9467fc640 1 -- 172.21.15.40:0/1836250732 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd948031050 con 0x56271d091ce0 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.584+0000 7fd96841e740 10 client.25858 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.584+0000 7fd96841e740 1 -- 172.21.15.40:0/1836250732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56271d10c6b0 con 0x56271d091ce0 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.584+0000 7fd9467fc640 1 -- 172.21.15.40:0/1836250732 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9500d7850 con 0x56271d091ce0 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.585+0000 7fd9467fc640 1 -- 172.21.15.40:0/1836250732 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fd948022150 con 0x56271d091ce0 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.585+0000 7fd96841e740 10 client.25858 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:47.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.585+0000 7fd96841e740 10 client.25858 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:44:47.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.585+0000 7fd96841e740 20 client.25858 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:47.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.585+0000 7fd96841e740 1 --2- 172.21.15.40:0/1836250732 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56271d10d040 0x56271d10f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.585+0000 7fd96841e740 4 client.25858 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:44:47.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.585+0000 7fd96841e740 1 -- 172.21.15.40:0/1836250732 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56271cffcd40 con 0x56271d10d040 2024-03-18T22:44:47.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.585+0000 7fd95effd640 1 --2- 172.21.15.40:0/1836250732 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56271d10d040 0x56271d10f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.585+0000 7fd95effd640 1 --2- 172.21.15.40:0/1836250732 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56271d10d040 0x56271d10f460 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3022d8823eddd040 server_cookie=801cca1a43c00878 in_seq=0 out_seq=0 2024-03-18T22:44:47.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.585+0000 7fd9467fc640 10 client.25858 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:44:47.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.586+0000 7fd9467fc640 1 -- 172.21.15.40:0/1836250732 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56271cffcd40 con 0x56271d10d040 2024-03-18T22:44:47.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.586+0000 7fd9467fc640 10 client.25858 handle_command_reply: tid=0 2024-03-18T22:44:47.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.586+0000 7fd96841e740 1 client.25858 shutdown 2024-03-18T22:44:47.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.586+0000 7fd96841e740 1 -- 172.21.15.40:0/1836250732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 msgr2=0x56271d09f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.586+0000 7fd96841e740 1 --2- 172.21.15.40:0/1836250732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d09f640 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd948004700 tx=0x7fd94801f7c0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.586+0000 7fd96841e740 1 -- 172.21.15.40:0/1836250732 shutdown_connections 2024-03-18T22:44:47.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.586+0000 7fd96841e740 1 --2- 172.21.15.40:0/1836250732 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56271d10d040 0x56271d10f460 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.586+0000 7fd96841e740 1 --2- 172.21.15.40:0/1836250732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56271d096e50 0x56271d09fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.586+0000 7fd96841e740 1 --2- 172.21.15.40:0/1836250732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56271d091ce0 0x56271d09f640 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.586+0000 7fd96841e740 1 --2- 172.21.15.40:0/1836250732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56271d0912f0 0x56271d0a2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.586+0000 7fd9467fc640 0 client.25858 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:44:47.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.586+0000 7fd96841e740 1 -- 172.21.15.40:0/1836250732 >> 172.21.15.40:0/1836250732 conn(0x56271d08c860 msgr2=0x56271d09b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.586+0000 7fd96841e740 1 -- 172.21.15.40:0/1836250732 shutdown_connections 2024-03-18T22:44:47.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.586+0000 7fd96841e740 1 -- 172.21.15.40:0/1836250732 wait complete. 2024-03-18T22:44:47.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.586+0000 7fd96841e740 20 client.25858 trim_cache size 0 max 16384 2024-03-18T22:44:47.591 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:44:47.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.587+0000 7fd965753640 1 -- 172.21.15.40:0/2658693383 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd94c0a2800 msgr2=0x7fd94c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.587+0000 7fd965753640 1 --2- 172.21.15.40:0/2658693383 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd94c0a2800 0x7fd94c0a4cc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd958004af0 tx=0x7fd958005fd0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.587+0000 7fd965753640 1 -- 172.21.15.40:0/2658693383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd960100720 msgr2=0x7fd960197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.587+0000 7fd965753640 1 --2- 172.21.15.40:0/2658693383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd960100720 0x7fd960197a00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd95000e9c0 tx=0x7fd95000ee80 comp rx=0 tx=0).stop 2024-03-18T22:44:47.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.587+0000 7fd965753640 1 -- 172.21.15.40:0/2658693383 shutdown_connections 2024-03-18T22:44:47.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.587+0000 7fd965753640 1 --2- 172.21.15.40:0/2658693383 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd94c0a2800 0x7fd94c0a4cc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.587+0000 7fd965753640 1 --2- 172.21.15.40:0/2658693383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9601022d0 0x7fd96019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.587+0000 7fd965753640 1 --2- 172.21.15.40:0/2658693383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd960101910 0x7fd960197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.587+0000 7fd965753640 1 --2- 172.21.15.40:0/2658693383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd960100720 0x7fd960197a00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.587+0000 7fd965753640 1 -- 172.21.15.40:0/2658693383 >> 172.21.15.40:0/2658693383 conn(0x7fd9600fbc90 msgr2=0x7fd9600fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.587+0000 7fd965753640 1 -- 172.21.15.40:0/2658693383 shutdown_connections 2024-03-18T22:44:47.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.587+0000 7fd965753640 1 -- 172.21.15.40:0/2658693383 wait complete. 2024-03-18T22:44:47.598 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:47 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/499241322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:44:47.599 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:44:47.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.714+0000 7f7347777640 1 -- 172.21.15.40:0/223863254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7340101910 msgr2=0x7f7340101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.714+0000 7f7347777640 1 --2- 172.21.15.40:0/223863254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7340101910 0x7f7340101d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7330009900 tx=0x7f733002bc30 comp rx=0 tx=0).stop 2024-03-18T22:44:47.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.714+0000 7f7347777640 1 -- 172.21.15.40:0/223863254 shutdown_connections 2024-03-18T22:44:47.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.714+0000 7f7347777640 1 --2- 172.21.15.40:0/223863254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73401022d0 0x7f7340108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.714+0000 7f7347777640 1 --2- 172.21.15.40:0/223863254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7340101910 0x7f7340101d90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.714+0000 7f7347777640 1 --2- 172.21.15.40:0/223863254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7340100720 0x7f7340100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.714+0000 7f7347777640 1 -- 172.21.15.40:0/223863254 >> 172.21.15.40:0/223863254 conn(0x7f73400fbc90 msgr2=0x7f73400fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.714+0000 7f7347777640 1 -- 172.21.15.40:0/223863254 shutdown_connections 2024-03-18T22:44:47.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.714+0000 7f7347777640 1 -- 172.21.15.40:0/223863254 wait complete. 2024-03-18T22:44:47.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.714+0000 7f7347777640 1 Processor -- start 2024-03-18T22:44:47.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.714+0000 7f7347777640 1 -- start start 2024-03-18T22:44:47.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.714+0000 7f7347777640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7340100720 0x7f7340197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.714+0000 7f7347777640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7340101910 0x7f7340197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7347777640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73401022d0 0x7f734019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7347777640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f734010c230 con 0x7f73401022d0 2024-03-18T22:44:47.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7347777640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f734010c0b0 con 0x7f7340100720 2024-03-18T22:44:47.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7347777640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f734010c3b0 con 0x7f7340101910 2024-03-18T22:44:47.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7345d14640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73401022d0 0x7f734019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7345513640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7340100720 0x7f7340197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7344d12640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7340101910 0x7f7340197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7345d14640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73401022d0 0x7f734019f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33806/0 (socket says 172.21.15.40:33806) 2024-03-18T22:44:47.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7345d14640 1 -- 172.21.15.40:0/3982809110 learned_addr learned my addr 172.21.15.40:0/3982809110 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7344d12640 1 -- 172.21.15.40:0/3982809110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7340100720 msgr2=0x7f7340197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7344d12640 1 --2- 172.21.15.40:0/3982809110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7340100720 0x7f7340197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7344d12640 1 -- 172.21.15.40:0/3982809110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73401022d0 msgr2=0x7f734019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7344d12640 1 --2- 172.21.15.40:0/3982809110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73401022d0 0x7f734019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7344d12640 1 -- 172.21.15.40:0/3982809110 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f734019f6a0 con 0x7f7340101910 2024-03-18T22:44:47.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7345d14640 1 --2- 172.21.15.40:0/3982809110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73401022d0 0x7f734019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:47.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7344d12640 1 --2- 172.21.15.40:0/3982809110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7340101910 0x7f7340197fc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7330000c00 tx=0x7f7330005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f73367fc640 1 -- 172.21.15.40:0/3982809110 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7330043020 con 0x7f7340101910 2024-03-18T22:44:47.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f73367fc640 1 -- 172.21.15.40:0/3982809110 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7330004350 con 0x7f7340101910 2024-03-18T22:44:47.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f73367fc640 1 -- 172.21.15.40:0/3982809110 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7330003b50 con 0x7f7340101910 2024-03-18T22:44:47.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.715+0000 7f7347777640 1 -- 172.21.15.40:0/3982809110 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f734019f930 con 0x7f7340101910 2024-03-18T22:44:47.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.716+0000 7f7347777640 1 -- 172.21.15.40:0/3982809110 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f734019fed0 con 0x7f7340101910 2024-03-18T22:44:47.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.718+0000 7f734a365740 1 Processor -- start 2024-03-18T22:44:47.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.718+0000 7f734a365740 1 -- start start 2024-03-18T22:44:47.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.718+0000 7f734a365740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef22f0 0x562a89ef2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.718+0000 7f73367fc640 1 -- 172.21.15.40:0/3982809110 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7330046020 con 0x7f7340101910 2024-03-18T22:44:47.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.718+0000 7f734a365740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef2ce0 0x562a89ef77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.718+0000 7f734a365740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef7e50 0x562a89efa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.718+0000 7f734a365740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562a89e564b0 con 0x562a89ef22f0 2024-03-18T22:44:47.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.718+0000 7f734a365740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562a89dfed70 con 0x562a89ef7e50 2024-03-18T22:44:47.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.718+0000 7f734a365740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x562a89e52550 con 0x562a89ef2ce0 2024-03-18T22:44:47.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f7345d14640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef22f0 0x562a89ef2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f7344d12640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef7e50 0x562a89efa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f7345d14640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef22f0 0x562a89ef2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33812/0 (socket says 172.21.15.40:33812) 2024-03-18T22:44:47.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f7345d14640 1 -- 172.21.15.40:0/2272709465 learned_addr learned my addr 172.21.15.40:0/2272709465 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f7345513640 1 --2- 172.21.15.40:0/2272709465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef2ce0 0x562a89ef77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f7345d14640 1 -- 172.21.15.40:0/2272709465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef2ce0 msgr2=0x562a89ef77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f7345d14640 1 --2- 172.21.15.40:0/2272709465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef2ce0 0x562a89ef77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f7345d14640 1 -- 172.21.15.40:0/2272709465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef7e50 msgr2=0x562a89efa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f7345d14640 1 --2- 172.21.15.40:0/2272709465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef7e50 0x562a89efa2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f7345d14640 1 -- 172.21.15.40:0/2272709465 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a89e21210 con 0x562a89ef22f0 2024-03-18T22:44:47.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f7344d12640 1 --2- 172.21.15.40:0/2272709465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef7e50 0x562a89efa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:47.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f7345513640 1 --2- 172.21.15.40:0/2272709465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef2ce0 0x562a89ef77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:47.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f73367fc640 1 --2- 172.21.15.40:0/3982809110 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f732c0a2880 0x7f732c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f7345d14640 1 --2- 172.21.15.40:0/2272709465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef22f0 0x562a89ef2710 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f73380118b0 tx=0x7f7338011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb7fa5b724d9d3f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f7345513640 1 --2- 172.21.15.40:0/3982809110 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f732c0a2880 0x7f732c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f73367fc640 1 -- 172.21.15.40:0/3982809110 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f73300f0dc0 con 0x7f7340101910 2024-03-18T22:44:47.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f732b7fe640 1 -- 172.21.15.40:0/2272709465 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733801e070 con 0x562a89ef22f0 2024-03-18T22:44:47.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f732b7fe640 1 -- 172.21.15.40:0/2272709465 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f733800fe70 con 0x562a89ef22f0 2024-03-18T22:44:47.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.719+0000 7f732b7fe640 1 -- 172.21.15.40:0/2272709465 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7338019d90 con 0x562a89ef22f0 2024-03-18T22:44:47.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.720+0000 7f734a365740 1 -- 172.21.15.40:0/2272709465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef22f0 msgr2=0x562a89ef2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.720+0000 7f7345513640 1 --2- 172.21.15.40:0/3982809110 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f732c0a2880 0x7f732c0a4d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x562a89ef30e0 tx=0x7f733c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.720+0000 7f734a365740 1 --2- 172.21.15.40:0/2272709465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef22f0 0x562a89ef2710 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f73380118b0 tx=0x7f7338011d70 comp rx=0 tx=0).stop 2024-03-18T22:44:47.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.720+0000 7f734a365740 1 -- 172.21.15.40:0/2272709465 shutdown_connections 2024-03-18T22:44:47.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.720+0000 7f734a365740 1 --2- 172.21.15.40:0/2272709465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef7e50 0x562a89efa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.720+0000 7f734a365740 1 --2- 172.21.15.40:0/2272709465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef2ce0 0x562a89ef77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.720+0000 7f734a365740 1 --2- 172.21.15.40:0/2272709465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef22f0 0x562a89ef2710 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.720+0000 7f734a365740 1 -- 172.21.15.40:0/2272709465 >> 172.21.15.40:0/2272709465 conn(0x562a89eed860 msgr2=0x562a89eedc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.720+0000 7f734a365740 1 -- 172.21.15.40:0/2272709465 shutdown_connections 2024-03-18T22:44:47.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.720+0000 7f734a365740 1 -- 172.21.15.40:0/2272709465 wait complete. 2024-03-18T22:44:47.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f734a365740 1 -- start start 2024-03-18T22:44:47.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f734a365740 1 Processor -- start 2024-03-18T22:44:47.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f734a365740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef22f0 0x562a89efc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f734a365740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef2ce0 0x562a89efccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f734a365740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef7e50 0x562a89f03d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f734a365740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562a89efae00 con 0x562a89ef7e50 2024-03-18T22:44:47.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f734a365740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562a89e12310 con 0x562a89ef22f0 2024-03-18T22:44:47.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f734a365740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x562a89efaf80 con 0x562a89ef2ce0 2024-03-18T22:44:47.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f734a365740 10 client.0 _ll_register_callbacks cb 0x562a89e062d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:47.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f7345d14640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef7e50 0x562a89f03d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f7345d14640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef7e50 0x562a89f03d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33826/0 (socket says 172.21.15.40:33826) 2024-03-18T22:44:47.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f7344d12640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef2ce0 0x562a89efccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f7345d14640 1 -- 172.21.15.40:0/2149573948 learned_addr learned my addr 172.21.15.40:0/2149573948 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f7345513640 1 --2- 172.21.15.40:0/2149573948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef22f0 0x562a89efc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f7345d14640 1 -- 172.21.15.40:0/2149573948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef2ce0 msgr2=0x562a89efccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.721+0000 7f7345d14640 1 --2- 172.21.15.40:0/2149573948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef2ce0 0x562a89efccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.722+0000 7f7345d14640 1 -- 172.21.15.40:0/2149573948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef22f0 msgr2=0x562a89efc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.722+0000 7f7345d14640 1 --2- 172.21.15.40:0/2149573948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef22f0 0x562a89efc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.722+0000 7f7345d14640 1 -- 172.21.15.40:0/2149573948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a89b6a3f0 con 0x562a89ef7e50 2024-03-18T22:44:47.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.722+0000 7f7345513640 1 --2- 172.21.15.40:0/2149573948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef22f0 0x562a89efc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:47.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.722+0000 7f7344d12640 1 --2- 172.21.15.40:0/2149573948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef2ce0 0x562a89efccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:47.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.722+0000 7f7345d14640 1 --2- 172.21.15.40:0/2149573948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef7e50 0x562a89f03d30 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7338012fd0 tx=0x7f73380196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7b23775cc0320d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.722+0000 7f7328ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:44:47.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.722+0000 7f7328ff9640 1 -- 172.21.15.40:0/2149573948 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7338028020 con 0x562a89ef7e50 2024-03-18T22:44:47.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.722+0000 7f7328ff9640 1 -- 172.21.15.40:0/2149573948 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7338004a40 con 0x562a89ef7e50 2024-03-18T22:44:47.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.722+0000 7f734a365740 1 -- 172.21.15.40:0/2149573948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f733800fe70 con 0x562a89ef7e50 2024-03-18T22:44:47.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.722+0000 7f7328ff9640 1 -- 172.21.15.40:0/2149573948 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7338033050 con 0x562a89ef7e50 2024-03-18T22:44:47.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.722+0000 7f734a365740 10 client.16026 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:47.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.722+0000 7f734a365740 1 -- 172.21.15.40:0/2149573948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562a89c347d0 con 0x562a89ef7e50 2024-03-18T22:44:47.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.722+0000 7f7328ff9640 1 -- 172.21.15.40:0/2149573948 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f733800fe70 con 0x562a89ef7e50 2024-03-18T22:44:47.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.723+0000 7f7328ff9640 1 -- 172.21.15.40:0/2149573948 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f73380052e0 con 0x562a89ef7e50 2024-03-18T22:44:47.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.723+0000 7f734a365740 10 client.16026 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:47.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.723+0000 7f734a365740 10 client.16026 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:44:47.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.723+0000 7f734a365740 20 client.16026 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:47.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.723+0000 7f734a365740 1 --2- 172.21.15.40:0/2149573948 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x562a89f08410 0x562a89f0a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.723+0000 7f734a365740 4 client.16026 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:47.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.723+0000 7f734a365740 1 -- 172.21.15.40:0/2149573948 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a89eb43e0 con 0x562a89f08410 2024-03-18T22:44:47.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.723+0000 7f7345513640 1 --2- 172.21.15.40:0/2149573948 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x562a89f08410 0x562a89f0a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.723+0000 7f7345513640 1 --2- 172.21.15.40:0/2149573948 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x562a89f08410 0x562a89f0a820 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e98beece1daa333e server_cookie=bb1c6d8e224a63d0 in_seq=0 out_seq=0 2024-03-18T22:44:47.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.723+0000 7f7328ff9640 10 client.16026 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:44:47.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.724+0000 7f7328ff9640 1 -- 172.21.15.40:0/2149573948 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562a89eb43e0 con 0x562a89f08410 2024-03-18T22:44:47.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.724+0000 7f7328ff9640 10 client.16026 handle_command_reply: tid=0 2024-03-18T22:44:47.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.724+0000 7f734a365740 1 client.16026 shutdown 2024-03-18T22:44:47.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.724+0000 7f734a365740 1 -- 172.21.15.40:0/2149573948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef7e50 msgr2=0x562a89f03d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.724+0000 7f734a365740 1 --2- 172.21.15.40:0/2149573948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef7e50 0x562a89f03d30 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7338012fd0 tx=0x7f73380196c0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.724+0000 7f734a365740 1 -- 172.21.15.40:0/2149573948 shutdown_connections 2024-03-18T22:44:47.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.724+0000 7f734a365740 1 --2- 172.21.15.40:0/2149573948 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x562a89f08410 0x562a89f0a820 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.725+0000 7f734a365740 1 --2- 172.21.15.40:0/2149573948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef7e50 0x562a89f03d30 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.725+0000 7f734a365740 1 --2- 172.21.15.40:0/2149573948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef2ce0 0x562a89efccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.725+0000 7f734a365740 1 --2- 172.21.15.40:0/2149573948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef22f0 0x562a89efc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.725+0000 7f7328ff9640 0 client.16026 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:44:47.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.725+0000 7f734a365740 1 -- 172.21.15.40:0/2149573948 >> 172.21.15.40:0/2149573948 conn(0x562a89eed860 msgr2=0x562a89eecba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.725+0000 7f734a365740 1 -- 172.21.15.40:0/2149573948 shutdown_connections 2024-03-18T22:44:47.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.725+0000 7f734a365740 1 -- 172.21.15.40:0/2149573948 wait complete. 2024-03-18T22:44:47.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.725+0000 7f734a365740 20 client.16026 trim_cache size 0 max 16384 2024-03-18T22:44:47.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.743+0000 7f734a365740 1 Processor -- start 2024-03-18T22:44:47.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.743+0000 7f734a365740 1 -- start start 2024-03-18T22:44:47.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.743+0000 7f734a365740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef22f0 0x562a89eec310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.743+0000 7f734a365740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef2ce0 0x562a89f03cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.743+0000 7f734a365740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef7e50 0x562a89eec850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.743+0000 7f734a365740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f73401aa4a0 con 0x562a89ef2ce0 2024-03-18T22:44:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.743+0000 7f734a365740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f73401ac0a0 con 0x562a89ef7e50 2024-03-18T22:44:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.743+0000 7f734a365740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f73401aab70 con 0x562a89ef22f0 2024-03-18T22:44:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.743+0000 7f7344d12640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef2ce0 0x562a89f03cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.743+0000 7f7344d12640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef2ce0 0x562a89f03cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33830/0 (socket says 172.21.15.40:33830) 2024-03-18T22:44:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.743+0000 7f7344d12640 1 -- 172.21.15.40:0/2205374802 learned_addr learned my addr 172.21.15.40:0/2205374802 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.744+0000 7f7344d12640 1 -- 172.21.15.40:0/2205374802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef22f0 msgr2=0x562a89eec310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.744+0000 7f7344d12640 1 --2- 172.21.15.40:0/2205374802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef22f0 0x562a89eec310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.744+0000 7f7344d12640 1 -- 172.21.15.40:0/2205374802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef7e50 msgr2=0x562a89eec850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.744+0000 7f7344d12640 1 --2- 172.21.15.40:0/2205374802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef7e50 0x562a89eec850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.744+0000 7f7344d12640 1 -- 172.21.15.40:0/2205374802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a89eed400 con 0x562a89ef2ce0 2024-03-18T22:44:47.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.744+0000 7f7344d12640 1 --2- 172.21.15.40:0/2205374802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef2ce0 0x562a89f03cf0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f73300ee4d0 tx=0x7f73300f48a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24afcf4491e2a03b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.744+0000 7f732affd640 1 -- 172.21.15.40:0/2205374802 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73300b9610 con 0x562a89ef2ce0 2024-03-18T22:44:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.744+0000 7f732affd640 1 -- 172.21.15.40:0/2205374802 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73300bda70 con 0x562a89ef2ce0 2024-03-18T22:44:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.744+0000 7f732affd640 1 -- 172.21.15.40:0/2205374802 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73300bccd0 con 0x562a89ef2ce0 2024-03-18T22:44:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.745+0000 7f734a365740 1 -- 172.21.15.40:0/2205374802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef2ce0 msgr2=0x562a89f03cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.745+0000 7f734a365740 1 --2- 172.21.15.40:0/2205374802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef2ce0 0x562a89f03cf0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f73300ee4d0 tx=0x7f73300f48a0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.745+0000 7f734a365740 1 -- 172.21.15.40:0/2205374802 shutdown_connections 2024-03-18T22:44:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.745+0000 7f734a365740 1 --2- 172.21.15.40:0/2205374802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef7e50 0x562a89eec850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.745+0000 7f734a365740 1 --2- 172.21.15.40:0/2205374802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef2ce0 0x562a89f03cf0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.745+0000 7f734a365740 1 --2- 172.21.15.40:0/2205374802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef22f0 0x562a89eec310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.745+0000 7f734a365740 1 -- 172.21.15.40:0/2205374802 >> 172.21.15.40:0/2205374802 conn(0x562a89eed860 msgr2=0x562a89efc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.746+0000 7f734a365740 1 -- 172.21.15.40:0/2205374802 shutdown_connections 2024-03-18T22:44:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.746+0000 7f734a365740 1 -- 172.21.15.40:0/2205374802 wait complete. 2024-03-18T22:44:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.746+0000 7f734a365740 1 -- start start 2024-03-18T22:44:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.746+0000 7f734a365740 1 Processor -- start 2024-03-18T22:44:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f734a365740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef22f0 0x562a89f00f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f734a365740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef2ce0 0x562a89f01460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f734a365740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef7e50 0x562a89f02470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f734a365740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f73401ab910 con 0x562a89ef2ce0 2024-03-18T22:44:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f734a365740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f73401ab240 con 0x562a89ef7e50 2024-03-18T22:44:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f734a365740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f73401a8480 con 0x562a89ef22f0 2024-03-18T22:44:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f734a365740 10 client.0 _ll_register_callbacks cb 0x7f733c009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f7344d12640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef2ce0 0x562a89f01460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f7344d12640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef2ce0 0x562a89f01460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33844/0 (socket says 172.21.15.40:33844) 2024-03-18T22:44:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f7344d12640 1 -- 172.21.15.40:0/3271361195 learned_addr learned my addr 172.21.15.40:0/3271361195 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f7345513640 1 --2- 172.21.15.40:0/3271361195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef22f0 0x562a89f00f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f7345d14640 1 --2- 172.21.15.40:0/3271361195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef7e50 0x562a89f02470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f7344d12640 1 -- 172.21.15.40:0/3271361195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef22f0 msgr2=0x562a89f00f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f7344d12640 1 --2- 172.21.15.40:0/3271361195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef22f0 0x562a89f00f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f7344d12640 1 -- 172.21.15.40:0/3271361195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef7e50 msgr2=0x562a89f02470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f7344d12640 1 --2- 172.21.15.40:0/3271361195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef7e50 0x562a89f02470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.747+0000 7f7344d12640 1 -- 172.21.15.40:0/3271361195 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a89ec3880 con 0x562a89ef2ce0 2024-03-18T22:44:47.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.748+0000 7f7344d12640 1 --2- 172.21.15.40:0/3271361195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef2ce0 0x562a89f01460 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f73300ee4a0 tx=0x7f73300045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efcc5d1664b9341d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.748+0000 7f7328ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:44:47.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.748+0000 7f7328ff9640 1 -- 172.21.15.40:0/3271361195 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73300bceb0 con 0x562a89ef2ce0 2024-03-18T22:44:47.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.748+0000 7f7328ff9640 1 -- 172.21.15.40:0/3271361195 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f73300c1170 con 0x562a89ef2ce0 2024-03-18T22:44:47.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.748+0000 7f734a365740 1 -- 172.21.15.40:0/3271361195 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73300bda70 con 0x562a89ef2ce0 2024-03-18T22:44:47.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.748+0000 7f7328ff9640 1 -- 172.21.15.40:0/3271361195 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f73300bd140 con 0x562a89ef2ce0 2024-03-18T22:44:47.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.748+0000 7f734a365740 10 client.16032 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:47.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.748+0000 7f734a365740 1 -- 172.21.15.40:0/3271361195 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562a89f03b90 con 0x562a89ef2ce0 2024-03-18T22:44:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.749+0000 7f7328ff9640 1 -- 172.21.15.40:0/3271361195 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f73300bd370 con 0x562a89ef2ce0 2024-03-18T22:44:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.749+0000 7f7328ff9640 1 -- 172.21.15.40:0/3271361195 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f73300c5680 con 0x562a89ef2ce0 2024-03-18T22:44:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.749+0000 7f734a365740 10 client.16032 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.749+0000 7f734a365740 10 client.16032 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:44:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.749+0000 7f734a365740 20 client.16032 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.749+0000 7f734a365740 1 --2- 172.21.15.40:0/3271361195 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x562a89f6e870 0x562a89f70c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.749+0000 7f734a365740 4 client.16032 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:44:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.749+0000 7f734a365740 1 -- 172.21.15.40:0/3271361195 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f73300bda70 con 0x562a89f6e870 2024-03-18T22:44:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.750+0000 7f7345513640 1 --2- 172.21.15.40:0/3271361195 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x562a89f6e870 0x562a89f70c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.750+0000 7f7345513640 1 --2- 172.21.15.40:0/3271361195 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x562a89f6e870 0x562a89f70c90 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ccace387a26d49b server_cookie=c5d2a0a6f71b40ae in_seq=0 out_seq=0 2024-03-18T22:44:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.750+0000 7f7328ff9640 10 client.16032 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:44:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.750+0000 7f7328ff9640 1 -- 172.21.15.40:0/3271361195 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f73300bda70 con 0x562a89f6e870 2024-03-18T22:44:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.750+0000 7f7328ff9640 10 client.16032 handle_command_reply: tid=0 2024-03-18T22:44:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.750+0000 7f734a365740 1 client.16032 shutdown 2024-03-18T22:44:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.750+0000 7f734a365740 1 -- 172.21.15.40:0/3271361195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef2ce0 msgr2=0x562a89f01460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.750+0000 7f734a365740 1 --2- 172.21.15.40:0/3271361195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef2ce0 0x562a89f01460 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f73300ee4a0 tx=0x7f73300045c0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.750+0000 7f734a365740 1 -- 172.21.15.40:0/3271361195 shutdown_connections 2024-03-18T22:44:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.750+0000 7f734a365740 1 --2- 172.21.15.40:0/3271361195 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x562a89f6e870 0x562a89f70c90 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f734a365740 1 --2- 172.21.15.40:0/3271361195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a89ef7e50 0x562a89f02470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f734a365740 1 --2- 172.21.15.40:0/3271361195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a89ef2ce0 0x562a89f01460 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f7328ff9640 0 client.16032 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:44:47.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f734a365740 1 --2- 172.21.15.40:0/3271361195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a89ef22f0 0x562a89f00f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f734a365740 1 -- 172.21.15.40:0/3271361195 >> 172.21.15.40:0/3271361195 conn(0x562a89eed860 msgr2=0x562a89ef0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f734a365740 1 -- 172.21.15.40:0/3271361195 shutdown_connections 2024-03-18T22:44:47.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f734a365740 1 -- 172.21.15.40:0/3271361195 wait complete. 2024-03-18T22:44:47.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f734a365740 20 client.16032 trim_cache size 0 max 16384 2024-03-18T22:44:47.756 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:44:47.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f7347777640 1 -- 172.21.15.40:0/3982809110 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f732c0a2880 msgr2=0x7f732c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f7347777640 1 --2- 172.21.15.40:0/3982809110 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f732c0a2880 0x7f732c0a4d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x562a89ef30e0 tx=0x7f733c005f90 comp rx=0 tx=0).stop 2024-03-18T22:44:47.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f7347777640 1 -- 172.21.15.40:0/3982809110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7340101910 msgr2=0x7f7340197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f7347777640 1 --2- 172.21.15.40:0/3982809110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7340101910 0x7f7340197fc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7330000c00 tx=0x7f7330005dc0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f7347777640 1 -- 172.21.15.40:0/3982809110 shutdown_connections 2024-03-18T22:44:47.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f7347777640 1 --2- 172.21.15.40:0/3982809110 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f732c0a2880 0x7f732c0a4d40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f7347777640 1 --2- 172.21.15.40:0/3982809110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73401022d0 0x7f734019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f7347777640 1 --2- 172.21.15.40:0/3982809110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7340101910 0x7f7340197fc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f7347777640 1 --2- 172.21.15.40:0/3982809110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7340100720 0x7f7340197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.751+0000 7f7347777640 1 -- 172.21.15.40:0/3982809110 >> 172.21.15.40:0/3982809110 conn(0x7f73400fbc90 msgr2=0x7f7340068a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.752+0000 7f7347777640 1 -- 172.21.15.40:0/3982809110 shutdown_connections 2024-03-18T22:44:47.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.752+0000 7f7347777640 1 -- 172.21.15.40:0/3982809110 wait complete. 2024-03-18T22:44:47.765 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:44:47.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:47 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/499241322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:44:47.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:47 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/499241322' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:44:47.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.876+0000 7fee67d52640 1 -- 172.21.15.40:0/1590798717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee60101910 msgr2=0x7fee60101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.876+0000 7fee67d52640 1 --2- 172.21.15.40:0/1590798717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee60101910 0x7fee60101d90 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fee50009900 tx=0x7fee5002bc30 comp rx=0 tx=0).stop 2024-03-18T22:44:47.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.876+0000 7fee67d52640 1 -- 172.21.15.40:0/1590798717 shutdown_connections 2024-03-18T22:44:47.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.876+0000 7fee67d52640 1 --2- 172.21.15.40:0/1590798717 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee601022d0 0x7fee60108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.876+0000 7fee67d52640 1 --2- 172.21.15.40:0/1590798717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee60101910 0x7fee60101d90 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.876+0000 7fee67d52640 1 --2- 172.21.15.40:0/1590798717 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee60100720 0x7fee60100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.876+0000 7fee67d52640 1 -- 172.21.15.40:0/1590798717 >> 172.21.15.40:0/1590798717 conn(0x7fee600fbc90 msgr2=0x7fee600fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.876+0000 7fee67d52640 1 -- 172.21.15.40:0/1590798717 shutdown_connections 2024-03-18T22:44:47.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.876+0000 7fee67d52640 1 -- 172.21.15.40:0/1590798717 wait complete. 2024-03-18T22:44:47.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee67d52640 1 Processor -- start 2024-03-18T22:44:47.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee67d52640 1 -- start start 2024-03-18T22:44:47.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee67d52640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee60100720 0x7fee600694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee67d52640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee60101910 0x7fee6019c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee67d52640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee601022d0 0x7fee6019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee67d52640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee6010c250 con 0x7fee60101910 2024-03-18T22:44:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee67d52640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee6010c0d0 con 0x7fee601022d0 2024-03-18T22:44:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee67d52640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fee6010c3d0 con 0x7fee60100720 2024-03-18T22:44:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee652ed640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee60101910 0x7fee6019c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee652ed640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee60101910 0x7fee6019c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33864/0 (socket says 172.21.15.40:33864) 2024-03-18T22:44:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee652ed640 1 -- 172.21.15.40:0/1229646030 learned_addr learned my addr 172.21.15.40:0/1229646030 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee65aee640 1 --2- 172.21.15.40:0/1229646030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee60100720 0x7fee600694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee662ef640 1 --2- 172.21.15.40:0/1229646030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee601022d0 0x7fee6019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee652ed640 1 -- 172.21.15.40:0/1229646030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee60100720 msgr2=0x7fee600694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee652ed640 1 --2- 172.21.15.40:0/1229646030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee60100720 0x7fee600694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee652ed640 1 -- 172.21.15.40:0/1229646030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee601022d0 msgr2=0x7fee6019ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee652ed640 1 --2- 172.21.15.40:0/1229646030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee601022d0 0x7fee6019ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee652ed640 1 -- 172.21.15.40:0/1229646030 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6019f680 con 0x7fee60101910 2024-03-18T22:44:47.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.877+0000 7fee652ed640 1 --2- 172.21.15.40:0/1229646030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee60101910 0x7fee6019c750 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fee50006010 tx=0x7fee50005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.878+0000 7fee56ffd640 1 -- 172.21.15.40:0/1229646030 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee50043020 con 0x7fee60101910 2024-03-18T22:44:47.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.878+0000 7fee67d52640 1 -- 172.21.15.40:0/1229646030 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee6019f8b0 con 0x7fee60101910 2024-03-18T22:44:47.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.878+0000 7fee56ffd640 1 -- 172.21.15.40:0/1229646030 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee500043d0 con 0x7fee60101910 2024-03-18T22:44:47.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.878+0000 7fee56ffd640 1 -- 172.21.15.40:0/1229646030 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee50003b80 con 0x7fee60101910 2024-03-18T22:44:47.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.878+0000 7fee67d52640 1 -- 172.21.15.40:0/1229646030 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee6019fe50 con 0x7fee60101910 2024-03-18T22:44:47.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.880+0000 7fee56ffd640 1 -- 172.21.15.40:0/1229646030 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fee50046020 con 0x7fee60101910 2024-03-18T22:44:47.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.880+0000 7fee6aa0f740 1 Processor -- start 2024-03-18T22:44:47.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee56ffd640 1 --2- 172.21.15.40:0/1229646030 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fee4c0a2800 0x7fee4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee6aa0f740 1 -- start start 2024-03-18T22:44:47.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee6aa0f740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac92f0 0x556c6cac9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee662ef640 1 --2- 172.21.15.40:0/1229646030 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fee4c0a2800 0x7fee4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee6aa0f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cac9ce0 0x556c6cace7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee6aa0f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cacee50 0x556c6cad12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee6aa0f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556c6ca2d4b0 con 0x556c6cac9ce0 2024-03-18T22:44:47.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee6aa0f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556c6c9d5d70 con 0x556c6cacee50 2024-03-18T22:44:47.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee6aa0f740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556c6ca29550 con 0x556c6cac92f0 2024-03-18T22:44:47.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee56ffd640 1 -- 172.21.15.40:0/1229646030 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fee500f1000 con 0x7fee60101910 2024-03-18T22:44:47.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee65aee640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac92f0 0x556c6cac9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee65aee640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac92f0 0x556c6cac9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41586/0 (socket says 172.21.15.40:41586) 2024-03-18T22:44:47.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee65aee640 1 -- 172.21.15.40:0/1532771761 learned_addr learned my addr 172.21.15.40:0/1532771761 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee652ed640 1 --2- 172.21.15.40:0/1532771761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cac9ce0 0x556c6cace7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee662ef640 1 --2- 172.21.15.40:0/1532771761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cacee50 0x556c6cad12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee65aee640 1 -- 172.21.15.40:0/1532771761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cacee50 msgr2=0x556c6cad12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee65aee640 1 --2- 172.21.15.40:0/1532771761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cacee50 0x556c6cad12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee65aee640 1 -- 172.21.15.40:0/1532771761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cac9ce0 msgr2=0x556c6cace7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee65aee640 1 --2- 172.21.15.40:0/1532771761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cac9ce0 0x556c6cace7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.881+0000 7fee65aee640 1 -- 172.21.15.40:0/1532771761 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c6c9f8210 con 0x556c6cac92f0 2024-03-18T22:44:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.882+0000 7fee662ef640 1 --2- 172.21.15.40:0/1229646030 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fee4c0a2800 0x7fee4c0a4cc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fee58005fd0 tx=0x7fee58011940 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.882+0000 7fee65aee640 1 --2- 172.21.15.40:0/1532771761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac92f0 0x556c6cac9710 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009ca0 tx=0x7fee5c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f350cc28add788ec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.882+0000 7fee4bfff640 1 -- 172.21.15.40:0/1532771761 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c004220 con 0x556c6cac92f0 2024-03-18T22:44:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.882+0000 7fee4bfff640 1 -- 172.21.15.40:0/1532771761 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee5c0043c0 con 0x556c6cac92f0 2024-03-18T22:44:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.882+0000 7fee4bfff640 1 -- 172.21.15.40:0/1532771761 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c019810 con 0x556c6cac92f0 2024-03-18T22:44:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.882+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1532771761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac92f0 msgr2=0x556c6cac9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/1532771761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac92f0 0x556c6cac9710 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009ca0 tx=0x7fee5c011810 comp rx=0 tx=0).stop 2024-03-18T22:44:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1532771761 shutdown_connections 2024-03-18T22:44:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/1532771761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cacee50 0x556c6cad12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/1532771761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cac9ce0 0x556c6cace7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/1532771761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac92f0 0x556c6cac9710 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1532771761 >> 172.21.15.40:0/1532771761 conn(0x556c6cac4860 msgr2=0x556c6cac4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1532771761 shutdown_connections 2024-03-18T22:44:47.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1532771761 wait complete. 2024-03-18T22:44:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 -- start start 2024-03-18T22:44:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 Processor -- start 2024-03-18T22:44:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac92f0 0x556c6cad3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac9ce0 0x556c6cad3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cacee50 0x556c6cadac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556c6cad1e00 con 0x556c6cacee50 2024-03-18T22:44:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556c6c9e9310 con 0x556c6cac9ce0 2024-03-18T22:44:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.883+0000 7fee6aa0f740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556c6cad1f80 con 0x556c6cac92f0 2024-03-18T22:44:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee6aa0f740 10 client.0 _ll_register_callbacks cb 0x556c6c9dd2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee65aee640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac92f0 0x556c6cad3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee662ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cacee50 0x556c6cadac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee65aee640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac92f0 0x556c6cad3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41588/0 (socket says 172.21.15.40:41588) 2024-03-18T22:44:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee65aee640 1 -- 172.21.15.40:0/1071560584 learned_addr learned my addr 172.21.15.40:0/1071560584 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee652ed640 1 --2- 172.21.15.40:0/1071560584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac9ce0 0x556c6cad3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee65aee640 1 -- 172.21.15.40:0/1071560584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac9ce0 msgr2=0x556c6cad3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee65aee640 1 --2- 172.21.15.40:0/1071560584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac9ce0 0x556c6cad3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee65aee640 1 -- 172.21.15.40:0/1071560584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cacee50 msgr2=0x556c6cadac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee65aee640 1 --2- 172.21.15.40:0/1071560584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cacee50 0x556c6cadac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee65aee640 1 -- 172.21.15.40:0/1071560584 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c6c7413f0 con 0x556c6cac92f0 2024-03-18T22:44:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee65aee640 1 --2- 172.21.15.40:0/1071560584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac92f0 0x556c6cad3710 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009c70 tx=0x7fee5c003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab596440c626b918 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee497fa640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:44:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee497fa640 1 -- 172.21.15.40:0/1071560584 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c028070 con 0x556c6cac92f0 2024-03-18T22:44:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1071560584 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee5c0043c0 con 0x556c6cac92f0 2024-03-18T22:44:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee497fa640 1 -- 172.21.15.40:0/1071560584 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fee5c005040 con 0x556c6cac92f0 2024-03-18T22:44:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee497fa640 1 -- 172.21.15.40:0/1071560584 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fee5c034050 con 0x556c6cac92f0 2024-03-18T22:44:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee6aa0f740 10 client.25885 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.884+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1071560584 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556c6c80b7d0 con 0x556c6cac92f0 2024-03-18T22:44:47.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.885+0000 7fee497fa640 1 -- 172.21.15.40:0/1071560584 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee5c0043c0 con 0x556c6cac92f0 2024-03-18T22:44:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.885+0000 7fee497fa640 1 -- 172.21.15.40:0/1071560584 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fee5c004bd0 con 0x556c6cac92f0 2024-03-18T22:44:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.885+0000 7fee6aa0f740 10 client.25885 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.885+0000 7fee6aa0f740 10 client.25885 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:44:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.885+0000 7fee6aa0f740 20 client.25885 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.885+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/1071560584 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556c6cadf330 0x556c6cae17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.885+0000 7fee6aa0f740 4 client.25885 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.885+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1071560584 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c6ca8b3e0 con 0x556c6cadf330 2024-03-18T22:44:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.885+0000 7fee652ed640 1 --2- 172.21.15.40:0/1071560584 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556c6cadf330 0x556c6cae17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.885+0000 7fee652ed640 1 --2- 172.21.15.40:0/1071560584 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556c6cadf330 0x556c6cae17d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=26b7d793c4757626 server_cookie=e15001e3121b6dde in_seq=0 out_seq=0 2024-03-18T22:44:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.885+0000 7fee497fa640 10 client.25885 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:44:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.886+0000 7fee497fa640 1 -- 172.21.15.40:0/1071560584 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556c6ca8b3e0 con 0x556c6cadf330 2024-03-18T22:44:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.886+0000 7fee497fa640 10 client.25885 handle_command_reply: tid=0 2024-03-18T22:44:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.886+0000 7fee6aa0f740 1 client.25885 shutdown 2024-03-18T22:44:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.886+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1071560584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac92f0 msgr2=0x556c6cad3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.886+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/1071560584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac92f0 0x556c6cad3710 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009c70 tx=0x7fee5c003820 comp rx=0 tx=0).stop 2024-03-18T22:44:47.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.886+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1071560584 shutdown_connections 2024-03-18T22:44:47.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.886+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/1071560584 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556c6cadf330 0x556c6cae17d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.886+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/1071560584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cacee50 0x556c6cadac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.886+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/1071560584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac9ce0 0x556c6cad3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.886+0000 7fee497fa640 0 client.25885 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:44:47.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.886+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/1071560584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac92f0 0x556c6cad3710 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.886+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1071560584 >> 172.21.15.40:0/1071560584 conn(0x556c6cac4860 msgr2=0x556c6cac3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.886+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1071560584 shutdown_connections 2024-03-18T22:44:47.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.886+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1071560584 wait complete. 2024-03-18T22:44:47.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.886+0000 7fee6aa0f740 20 client.25885 trim_cache size 0 max 16384 2024-03-18T22:44:47.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.901+0000 7fee6aa0f740 1 Processor -- start 2024-03-18T22:44:47.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.901+0000 7fee6aa0f740 1 -- start start 2024-03-18T22:44:47.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.901+0000 7fee6aa0f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac92f0 0x556c6cae60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.901+0000 7fee6aa0f740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac9ce0 0x556c6cadac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.901+0000 7fee6aa0f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cacee50 0x556c6cae6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.901+0000 7fee6aa0f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee601aa4b0 con 0x556c6cacee50 2024-03-18T22:44:47.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.901+0000 7fee6aa0f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee601ac0b0 con 0x556c6cac92f0 2024-03-18T22:44:47.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.901+0000 7fee6aa0f740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fee601aab80 con 0x556c6cac9ce0 2024-03-18T22:44:47.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.901+0000 7fee662ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cacee50 0x556c6cae6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.901+0000 7fee652ed640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac9ce0 0x556c6cadac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.901+0000 7fee652ed640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac9ce0 0x556c6cadac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41598/0 (socket says 172.21.15.40:41598) 2024-03-18T22:44:47.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.901+0000 7fee652ed640 1 -- 172.21.15.40:0/1650909731 learned_addr learned my addr 172.21.15.40:0/1650909731 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.901+0000 7fee662ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cacee50 0x556c6cae6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33892/0 (socket says 172.21.15.40:33892) 2024-03-18T22:44:47.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.901+0000 7fee65aee640 1 --2- 172.21.15.40:0/1650909731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac92f0 0x556c6cae60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.902+0000 7fee652ed640 1 -- 172.21.15.40:0/1650909731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac92f0 msgr2=0x556c6cae60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.902+0000 7fee652ed640 1 --2- 172.21.15.40:0/1650909731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac92f0 0x556c6cae60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.902+0000 7fee652ed640 1 -- 172.21.15.40:0/1650909731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cacee50 msgr2=0x556c6cae6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.902+0000 7fee652ed640 1 --2- 172.21.15.40:0/1650909731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cacee50 0x556c6cae6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.902+0000 7fee652ed640 1 -- 172.21.15.40:0/1650909731 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c6ca9a880 con 0x556c6cac9ce0 2024-03-18T22:44:47.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.902+0000 7fee662ef640 1 --2- 172.21.15.40:0/1650909731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cacee50 0x556c6cae6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:47.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.902+0000 7fee65aee640 1 --2- 172.21.15.40:0/1650909731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac92f0 0x556c6cae60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:47.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.902+0000 7fee652ed640 1 --2- 172.21.15.40:0/1650909731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac9ce0 0x556c6cadac50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fee500b9a10 tx=0x7fee500ee5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a12a2af32a2d75bf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.902+0000 7fee4b7fe640 1 -- 172.21.15.40:0/1650909731 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee500d5070 con 0x556c6cac9ce0 2024-03-18T22:44:47.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.902+0000 7fee4b7fe640 1 -- 172.21.15.40:0/1650909731 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee500c1700 con 0x556c6cac9ce0 2024-03-18T22:44:47.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.902+0000 7fee4b7fe640 1 -- 172.21.15.40:0/1650909731 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee500b9c20 con 0x556c6cac9ce0 2024-03-18T22:44:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.903+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1650909731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac9ce0 msgr2=0x556c6cadac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.903+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/1650909731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac9ce0 0x556c6cadac50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fee500b9a10 tx=0x7fee500ee5e0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.903+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1650909731 shutdown_connections 2024-03-18T22:44:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.903+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/1650909731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cacee50 0x556c6cae6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.903+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/1650909731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cac9ce0 0x556c6cadac50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.903+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/1650909731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac92f0 0x556c6cae60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.903+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1650909731 >> 172.21.15.40:0/1650909731 conn(0x556c6cac4860 msgr2=0x556c6cac8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.903+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1650909731 shutdown_connections 2024-03-18T22:44:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.903+0000 7fee6aa0f740 1 -- 172.21.15.40:0/1650909731 wait complete. 2024-03-18T22:44:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.903+0000 7fee6aa0f740 1 -- start start 2024-03-18T22:44:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.903+0000 7fee6aa0f740 1 Processor -- start 2024-03-18T22:44:47.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee6aa0f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac92f0 0x556c6cad81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee6aa0f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cac9ce0 0x556c6cad8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee6aa0f740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cacee50 0x556c6cad9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee6aa0f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee601ab920 con 0x556c6cac9ce0 2024-03-18T22:44:47.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee6aa0f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee601ab250 con 0x556c6cac92f0 2024-03-18T22:44:47.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee6aa0f740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fee601a8490 con 0x556c6cacee50 2024-03-18T22:44:47.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee6aa0f740 10 client.0 _ll_register_callbacks cb 0x7fee50004580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:47.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee652ed640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cac9ce0 0x556c6cad8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee65aee640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac92f0 0x556c6cad81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee662ef640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cacee50 0x556c6cad9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee652ed640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cac9ce0 0x556c6cad8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33898/0 (socket says 172.21.15.40:33898) 2024-03-18T22:44:47.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee65aee640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac92f0 0x556c6cad81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:51192/0 (socket says 172.21.15.40:51192) 2024-03-18T22:44:47.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee652ed640 1 -- 172.21.15.40:0/2972764729 learned_addr learned my addr 172.21.15.40:0/2972764729 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:47.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee662ef640 1 -- 172.21.15.40:0/2972764729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac92f0 msgr2=0x556c6cad81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee662ef640 1 --2- 172.21.15.40:0/2972764729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac92f0 0x556c6cad81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee662ef640 1 -- 172.21.15.40:0/2972764729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cac9ce0 msgr2=0x556c6cad8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:47.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee662ef640 1 --2- 172.21.15.40:0/2972764729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cac9ce0 0x556c6cad8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee662ef640 1 -- 172.21.15.40:0/2972764729 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c6cad9d00 con 0x556c6cacee50 2024-03-18T22:44:47.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee652ed640 1 --2- 172.21.15.40:0/2972764729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cac9ce0 0x556c6cad8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:47.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee662ef640 1 --2- 172.21.15.40:0/2972764729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cacee50 0x556c6cad9710 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fee58011e40 tx=0x7fee58018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=340aaf6702d92174 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:47.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.904+0000 7fee497fa640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:44:47.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.905+0000 7fee497fa640 1 -- 172.21.15.40:0/2972764729 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee58020d30 con 0x556c6cacee50 2024-03-18T22:44:47.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.905+0000 7fee497fa640 1 -- 172.21.15.40:0/2972764729 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fee58004b20 con 0x556c6cacee50 2024-03-18T22:44:47.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.905+0000 7fee6aa0f740 1 -- 172.21.15.40:0/2972764729 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee500c1700 con 0x556c6cacee50 2024-03-18T22:44:47.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.905+0000 7fee497fa640 1 -- 172.21.15.40:0/2972764729 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fee58032050 con 0x556c6cacee50 2024-03-18T22:44:47.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.905+0000 7fee6aa0f740 10 client.25891 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:47.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.905+0000 7fee6aa0f740 1 -- 172.21.15.40:0/2972764729 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556c6cb45720 con 0x556c6cacee50 2024-03-18T22:44:47.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.905+0000 7fee497fa640 1 -- 172.21.15.40:0/2972764729 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee500c1700 con 0x556c6cacee50 2024-03-18T22:44:47.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.905+0000 7fee497fa640 1 -- 172.21.15.40:0/2972764729 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fee58005820 con 0x556c6cacee50 2024-03-18T22:44:47.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.905+0000 7fee6aa0f740 10 client.25891 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:47.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.905+0000 7fee6aa0f740 10 client.25891 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:44:47.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.905+0000 7fee6aa0f740 20 client.25891 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:47.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.905+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/2972764729 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556c6cb45fe0 0x556c6cb48400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:47.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.905+0000 7fee6aa0f740 4 client.25891 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:44:47.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.905+0000 7fee6aa0f740 1 -- 172.21.15.40:0/2972764729 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c6ca3a9f0 con 0x556c6cb45fe0 2024-03-18T22:44:47.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.905+0000 7fee65aee640 1 --2- 172.21.15.40:0/2972764729 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556c6cb45fe0 0x556c6cb48400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:47.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee497fa640 10 client.25891 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:44:47.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee65aee640 1 --2- 172.21.15.40:0/2972764729 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556c6cb45fe0 0x556c6cb48400 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8cb5acad23e2cce2 server_cookie=b388b035d7b156f5 in_seq=0 out_seq=0 2024-03-18T22:44:47.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee497fa640 1 -- 172.21.15.40:0/2972764729 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c6ca3a9f0 con 0x556c6cb45fe0 2024-03-18T22:44:47.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee497fa640 10 client.25891 handle_command_reply: tid=0 2024-03-18T22:44:47.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee6aa0f740 1 client.25891 shutdown 2024-03-18T22:44:47.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee6aa0f740 1 -- 172.21.15.40:0/2972764729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cacee50 msgr2=0x556c6cad9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/2972764729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cacee50 0x556c6cad9710 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fee58011e40 tx=0x7fee58018000 comp rx=0 tx=0).stop 2024-03-18T22:44:47.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee6aa0f740 1 -- 172.21.15.40:0/2972764729 shutdown_connections 2024-03-18T22:44:47.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/2972764729 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x556c6cb45fe0 0x556c6cb48400 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/2972764729 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c6cacee50 0x556c6cad9710 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/2972764729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c6cac9ce0 0x556c6cad8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee497fa640 0 client.25891 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:44:47.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee6aa0f740 1 --2- 172.21.15.40:0/2972764729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c6cac92f0 0x556c6cad81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee6aa0f740 1 -- 172.21.15.40:0/2972764729 >> 172.21.15.40:0/2972764729 conn(0x556c6cac4860 msgr2=0x556c6cac3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee6aa0f740 1 -- 172.21.15.40:0/2972764729 shutdown_connections 2024-03-18T22:44:47.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee6aa0f740 1 -- 172.21.15.40:0/2972764729 wait complete. 2024-03-18T22:44:47.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.906+0000 7fee6aa0f740 20 client.25891 trim_cache size 0 max 16384 2024-03-18T22:44:47.914 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:44:47.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.907+0000 7fee67d52640 1 -- 172.21.15.40:0/1229646030 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fee4c0a2800 msgr2=0x7fee4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.907+0000 7fee67d52640 1 --2- 172.21.15.40:0/1229646030 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fee4c0a2800 0x7fee4c0a4cc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fee58005fd0 tx=0x7fee58011940 comp rx=0 tx=0).stop 2024-03-18T22:44:47.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.907+0000 7fee67d52640 1 -- 172.21.15.40:0/1229646030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee60101910 msgr2=0x7fee6019c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:47.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.907+0000 7fee67d52640 1 --2- 172.21.15.40:0/1229646030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee60101910 0x7fee6019c750 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fee50006010 tx=0x7fee50005e40 comp rx=0 tx=0).stop 2024-03-18T22:44:47.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.907+0000 7fee67d52640 1 -- 172.21.15.40:0/1229646030 shutdown_connections 2024-03-18T22:44:47.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.907+0000 7fee67d52640 1 --2- 172.21.15.40:0/1229646030 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fee4c0a2800 0x7fee4c0a4cc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.907+0000 7fee67d52640 1 --2- 172.21.15.40:0/1229646030 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee601022d0 0x7fee6019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.907+0000 7fee67d52640 1 --2- 172.21.15.40:0/1229646030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee60101910 0x7fee6019c750 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.907+0000 7fee67d52640 1 --2- 172.21.15.40:0/1229646030 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee60100720 0x7fee600694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:47.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.907+0000 7fee67d52640 1 -- 172.21.15.40:0/1229646030 >> 172.21.15.40:0/1229646030 conn(0x7fee600fbc90 msgr2=0x7fee60104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:47.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.907+0000 7fee67d52640 1 -- 172.21.15.40:0/1229646030 shutdown_connections 2024-03-18T22:44:47.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:47.907+0000 7fee67d52640 1 -- 172.21.15.40:0/1229646030 wait complete. 2024-03-18T22:44:47.919 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:44:48.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.029+0000 7f61941e6640 1 -- 172.21.15.40:0/3868633374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f618c100720 msgr2=0x7f618c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:48.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.029+0000 7f61941e6640 1 --2- 172.21.15.40:0/3868633374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f618c100720 0x7f618c100b40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6188009900 tx=0x7f618802bc30 comp rx=0 tx=0).stop 2024-03-18T22:44:48.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.029+0000 7f61941e6640 1 -- 172.21.15.40:0/3868633374 shutdown_connections 2024-03-18T22:44:48.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.029+0000 7f61941e6640 1 --2- 172.21.15.40:0/3868633374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f618c1022d0 0x7f618c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.029+0000 7f61941e6640 1 --2- 172.21.15.40:0/3868633374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f618c101910 0x7f618c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.029+0000 7f61941e6640 1 --2- 172.21.15.40:0/3868633374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f618c100720 0x7f618c100b40 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.029+0000 7f61941e6640 1 -- 172.21.15.40:0/3868633374 >> 172.21.15.40:0/3868633374 conn(0x7f618c0fbc90 msgr2=0x7f618c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:48.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.029+0000 7f61941e6640 1 -- 172.21.15.40:0/3868633374 shutdown_connections 2024-03-18T22:44:48.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.029+0000 7f61941e6640 1 -- 172.21.15.40:0/3868633374 wait complete. 2024-03-18T22:44:48.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f61941e6640 1 Processor -- start 2024-03-18T22:44:48.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f61941e6640 1 -- start start 2024-03-18T22:44:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f61941e6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f618c100720 0x7f618c0694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f61941e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f618c101910 0x7f618c19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f61941e6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f618c1022d0 0x7f618c19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f61941e6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f618c10c220 con 0x7f618c101910 2024-03-18T22:44:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f61941e6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f618c10c0a0 con 0x7f618c1022d0 2024-03-18T22:44:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f61941e6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f618c10c3a0 con 0x7f618c100720 2024-03-18T22:44:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f6191781640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f618c101910 0x7f618c19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f6191781640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f618c101910 0x7f618c19c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33912/0 (socket says 172.21.15.40:33912) 2024-03-18T22:44:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f6191781640 1 -- 172.21.15.40:0/4184295910 learned_addr learned my addr 172.21.15.40:0/4184295910 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f6191f82640 1 --2- 172.21.15.40:0/4184295910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f618c100720 0x7f618c0694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f6192783640 1 --2- 172.21.15.40:0/4184295910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f618c1022d0 0x7f618c19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f6191781640 1 -- 172.21.15.40:0/4184295910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f618c100720 msgr2=0x7f618c0694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f6191781640 1 --2- 172.21.15.40:0/4184295910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f618c100720 0x7f618c0694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f6191781640 1 -- 172.21.15.40:0/4184295910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f618c1022d0 msgr2=0x7f618c19edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:48.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f6191781640 1 --2- 172.21.15.40:0/4184295910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f618c1022d0 0x7f618c19edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f6191781640 1 -- 172.21.15.40:0/4184295910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618c19f6c0 con 0x7f618c101910 2024-03-18T22:44:48.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.030+0000 7f6191781640 1 --2- 172.21.15.40:0/4184295910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f618c101910 0x7f618c19c790 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f617c00ea60 tx=0x7f617c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:48.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.031+0000 7f6182ffd640 1 -- 172.21.15.40:0/4184295910 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c00cde0 con 0x7f618c101910 2024-03-18T22:44:48.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.031+0000 7f61941e6640 1 -- 172.21.15.40:0/4184295910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618c19f950 con 0x7f618c101910 2024-03-18T22:44:48.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.031+0000 7f6182ffd640 1 -- 172.21.15.40:0/4184295910 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f617c0035f0 con 0x7f618c101910 2024-03-18T22:44:48.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.031+0000 7f6182ffd640 1 -- 172.21.15.40:0/4184295910 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c010690 con 0x7f618c101910 2024-03-18T22:44:48.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.031+0000 7f61941e6640 1 -- 172.21.15.40:0/4184295910 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f618c19fef0 con 0x7f618c101910 2024-03-18T22:44:48.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.033+0000 7f6182ffd640 1 -- 172.21.15.40:0/4184295910 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f617c0037a0 con 0x7f618c101910 2024-03-18T22:44:48.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.033+0000 7f6196dd3740 1 Processor -- start 2024-03-18T22:44:48.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6196dd3740 1 -- start start 2024-03-18T22:44:48.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6196dd3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579835662f0 0x557983566710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6196dd3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x55798356b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6196dd3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55798356be50 0x55798356e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6196dd3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5579834ca4b0 con 0x5579835662f0 2024-03-18T22:44:48.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6196dd3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557983472d70 con 0x55798356be50 2024-03-18T22:44:48.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6196dd3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5579834c6550 con 0x557983566ce0 2024-03-18T22:44:48.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6191781640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55798356be50 0x55798356e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6191f82640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x55798356b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6192783640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579835662f0 0x557983566710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6182ffd640 1 --2- 172.21.15.40:0/4184295910 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f61780a2800 0x7f61780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6191781640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55798356be50 0x55798356e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:51206/0 (socket says 172.21.15.40:51206) 2024-03-18T22:44:48.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6191781640 1 -- 172.21.15.40:0/3205285236 learned_addr learned my addr 172.21.15.40:0/3205285236 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:48.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6191f82640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x55798356b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41630/0 (socket says 172.21.15.40:41630) 2024-03-18T22:44:48.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6182ffd640 1 -- 172.21.15.40:0/4184295910 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f617c0cf0c0 con 0x7f618c101910 2024-03-18T22:44:48.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6191f82640 1 -- 172.21.15.40:0/3205285236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55798356be50 msgr2=0x55798356e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:48.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6191f82640 1 --2- 172.21.15.40:0/3205285236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55798356be50 0x55798356e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6191f82640 1 -- 172.21.15.40:0/3205285236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579835662f0 msgr2=0x557983566710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:48.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6191f82640 1 --2- 172.21.15.40:0/3205285236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579835662f0 0x557983566710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6192783640 1 --2- 172.21.15.40:0/4184295910 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f61780a2800 0x7f61780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.034+0000 7f6191f82640 1 -- 172.21.15.40:0/3205285236 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557983495210 con 0x557983566ce0 2024-03-18T22:44:48.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.035+0000 7f6191f82640 1 --2- 172.21.15.40:0/3205285236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x55798356b7e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6188036040 tx=0x7f6188009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5125ca1dcb8c5d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:48.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.035+0000 7f6177fff640 1 -- 172.21.15.40:0/3205285236 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618804c070 con 0x557983566ce0 2024-03-18T22:44:48.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.035+0000 7f6177fff640 1 -- 172.21.15.40:0/3205285236 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6188004040 con 0x557983566ce0 2024-03-18T22:44:48.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.035+0000 7f6177fff640 1 -- 172.21.15.40:0/3205285236 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6188005480 con 0x557983566ce0 2024-03-18T22:44:48.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.035+0000 7f6192783640 1 --2- 172.21.15.40:0/4184295910 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f61780a2800 0x7f61780a4cc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x557983567540 tx=0x7f6184004ab0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:48.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.036+0000 7f6196dd3740 1 -- 172.21.15.40:0/3205285236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 msgr2=0x55798356b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:48.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.036+0000 7f6196dd3740 1 --2- 172.21.15.40:0/3205285236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x55798356b7e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6188036040 tx=0x7f6188009560 comp rx=0 tx=0).stop 2024-03-18T22:44:48.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.036+0000 7f6196dd3740 1 -- 172.21.15.40:0/3205285236 shutdown_connections 2024-03-18T22:44:48.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.036+0000 7f6196dd3740 1 --2- 172.21.15.40:0/3205285236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55798356be50 0x55798356e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.036+0000 7f6196dd3740 1 --2- 172.21.15.40:0/3205285236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x55798356b7e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.036+0000 7f6196dd3740 1 --2- 172.21.15.40:0/3205285236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5579835662f0 0x557983566710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.036+0000 7f6196dd3740 1 -- 172.21.15.40:0/3205285236 >> 172.21.15.40:0/3205285236 conn(0x557983561860 msgr2=0x557983561c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:48.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.036+0000 7f6196dd3740 1 -- 172.21.15.40:0/3205285236 shutdown_connections 2024-03-18T22:44:48.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.036+0000 7f6196dd3740 1 -- 172.21.15.40:0/3205285236 wait complete. 2024-03-18T22:44:48.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.036+0000 7f6196dd3740 1 -- start start 2024-03-18T22:44:48.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.036+0000 7f6196dd3740 1 Processor -- start 2024-03-18T22:44:48.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6196dd3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 0x557983570780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6196dd3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557983566ce0 0x557983570cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6196dd3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55798356be50 0x557983577d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6196dd3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55798356eda0 con 0x557983566ce0 2024-03-18T22:44:48.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6196dd3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557983486310 con 0x5579835662f0 2024-03-18T22:44:48.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6196dd3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55798356ef20 con 0x55798356be50 2024-03-18T22:44:48.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6196dd3740 10 client.0 _ll_register_callbacks cb 0x55798347a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:48.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6191781640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557983566ce0 0x557983570cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6191f82640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 0x557983570780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6191f82640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 0x557983570780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:51214/0 (socket says 172.21.15.40:51214) 2024-03-18T22:44:48.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6191f82640 1 -- 172.21.15.40:0/4216696833 learned_addr learned my addr 172.21.15.40:0/4216696833 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:48.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6192783640 1 --2- 172.21.15.40:0/4216696833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55798356be50 0x557983577d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6191f82640 1 -- 172.21.15.40:0/4216696833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55798356be50 msgr2=0x557983577d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:48.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6191f82640 1 --2- 172.21.15.40:0/4216696833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55798356be50 0x557983577d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6191f82640 1 -- 172.21.15.40:0/4216696833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557983566ce0 msgr2=0x557983570cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:48.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6191f82640 1 --2- 172.21.15.40:0/4216696833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557983566ce0 0x557983570cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6191f82640 1 -- 172.21.15.40:0/4216696833 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579831de3f0 con 0x5579835662f0 2024-03-18T22:44:48.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6192783640 1 --2- 172.21.15.40:0/4216696833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55798356be50 0x557983577d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:48.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f6191f82640 1 --2- 172.21.15.40:0/4216696833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 0x557983570780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f618802c170 tx=0x7f6188004420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=664bbd0953a1f0c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:48.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f61757fa640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:44:48.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.037+0000 7f61757fa640 1 -- 172.21.15.40:0/4216696833 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618804c030 con 0x5579835662f0 2024-03-18T22:44:48.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.038+0000 7f61757fa640 1 -- 172.21.15.40:0/4216696833 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6188044040 con 0x5579835662f0 2024-03-18T22:44:48.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.038+0000 7f6196dd3740 1 -- 172.21.15.40:0/4216696833 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6188004040 con 0x5579835662f0 2024-03-18T22:44:48.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.038+0000 7f61757fa640 1 -- 172.21.15.40:0/4216696833 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f618805d050 con 0x5579835662f0 2024-03-18T22:44:48.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.038+0000 7f6196dd3740 10 client.25910 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:48.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.038+0000 7f6196dd3740 1 -- 172.21.15.40:0/4216696833 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5579832a87d0 con 0x5579835662f0 2024-03-18T22:44:48.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.038+0000 7f61757fa640 1 -- 172.21.15.40:0/4216696833 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6188004040 con 0x5579835662f0 2024-03-18T22:44:48.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.038+0000 7f61757fa640 1 -- 172.21.15.40:0/4216696833 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f61880035f0 con 0x5579835662f0 2024-03-18T22:44:48.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.038+0000 7f6196dd3740 10 client.25910 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:48.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.038+0000 7f6196dd3740 10 client.25910 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:48.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.038+0000 7f6196dd3740 20 client.25910 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:48.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.038+0000 7f6196dd3740 1 --2- 172.21.15.40:0/4216696833 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55798357c1f0 0x55798357e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.038+0000 7f6196dd3740 4 client.25910 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:48.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.038+0000 7f6196dd3740 1 -- 172.21.15.40:0/4216696833 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579835283e0 con 0x55798357c1f0 2024-03-18T22:44:48.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.038+0000 7f6191781640 1 --2- 172.21.15.40:0/4216696833 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55798357c1f0 0x55798357e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.039+0000 7f6191781640 1 --2- 172.21.15.40:0/4216696833 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55798357c1f0 0x55798357e5e0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21eac6bf7191d64c server_cookie=37407d4cf8c35b3b in_seq=0 out_seq=0 2024-03-18T22:44:48.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.039+0000 7f61757fa640 10 client.25910 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:48.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.040+0000 7f61757fa640 1 -- 172.21.15.40:0/4216696833 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5579835283e0 con 0x55798357c1f0 2024-03-18T22:44:48.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.040+0000 7f61757fa640 10 client.25910 handle_command_reply: tid=0 2024-03-18T22:44:48.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.040+0000 7f6196dd3740 1 client.25910 shutdown 2024-03-18T22:44:48.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.040+0000 7f6196dd3740 1 -- 172.21.15.40:0/4216696833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 msgr2=0x557983570780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:48.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.040+0000 7f6196dd3740 1 --2- 172.21.15.40:0/4216696833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 0x557983570780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f618802c170 tx=0x7f6188004420 comp rx=0 tx=0).stop 2024-03-18T22:44:48.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.040+0000 7f6196dd3740 1 -- 172.21.15.40:0/4216696833 shutdown_connections 2024-03-18T22:44:48.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.040+0000 7f6196dd3740 1 --2- 172.21.15.40:0/4216696833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55798356be50 0x557983577d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.040+0000 7f6196dd3740 1 --2- 172.21.15.40:0/4216696833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557983566ce0 0x557983570cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.040+0000 7f6196dd3740 1 --2- 172.21.15.40:0/4216696833 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55798357c1f0 0x55798357e5e0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.040+0000 7f6196dd3740 1 --2- 172.21.15.40:0/4216696833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 0x557983570780 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.040+0000 7f61757fa640 0 client.25910 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:48.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.040+0000 7f6196dd3740 1 -- 172.21.15.40:0/4216696833 >> 172.21.15.40:0/4216696833 conn(0x557983561860 msgr2=0x557983560ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:48.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.040+0000 7f6196dd3740 1 -- 172.21.15.40:0/4216696833 shutdown_connections 2024-03-18T22:44:48.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.040+0000 7f6196dd3740 1 -- 172.21.15.40:0/4216696833 wait complete. 2024-03-18T22:44:48.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.040+0000 7f6196dd3740 20 client.25910 trim_cache size 0 max 16384 2024-03-18T22:44:48.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6196dd3740 1 Processor -- start 2024-03-18T22:44:48.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6196dd3740 1 -- start start 2024-03-18T22:44:48.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6196dd3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 0x55798355ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6196dd3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x557983577cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6196dd3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798356be50 0x557983561c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6196dd3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f618c1a82d0 con 0x55798356be50 2024-03-18T22:44:48.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6196dd3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f618c1a9ed0 con 0x5579835662f0 2024-03-18T22:44:48.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6196dd3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f618c1a89a0 con 0x557983566ce0 2024-03-18T22:44:48.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6192783640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798356be50 0x557983561c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6192783640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798356be50 0x557983561c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33934/0 (socket says 172.21.15.40:33934) 2024-03-18T22:44:48.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6192783640 1 -- 172.21.15.40:0/1688099696 learned_addr learned my addr 172.21.15.40:0/1688099696 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:48.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6191781640 1 --2- 172.21.15.40:0/1688099696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x557983577cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6191f82640 1 --2- 172.21.15.40:0/1688099696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 0x55798355ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6191781640 1 -- 172.21.15.40:0/1688099696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 msgr2=0x55798355ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:48.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6191781640 1 --2- 172.21.15.40:0/1688099696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 0x55798355ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6191781640 1 -- 172.21.15.40:0/1688099696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798356be50 msgr2=0x557983561c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:48.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6191781640 1 --2- 172.21.15.40:0/1688099696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798356be50 0x557983561c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6191781640 1 -- 172.21.15.40:0/1688099696 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557983537880 con 0x557983566ce0 2024-03-18T22:44:48.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6192783640 1 --2- 172.21.15.40:0/1688099696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798356be50 0x557983561c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:48.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.054+0000 7f6191781640 1 --2- 172.21.15.40:0/1688099696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x557983577cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f617c09b600 tx=0x7f617c0d15d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc97f26d84fd8eea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:48.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.055+0000 7f61777fe640 1 -- 172.21.15.40:0/1688099696 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c0bb040 con 0x557983566ce0 2024-03-18T22:44:48.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.055+0000 7f61777fe640 1 -- 172.21.15.40:0/1688099696 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f617c003b30 con 0x557983566ce0 2024-03-18T22:44:48.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.055+0000 7f61777fe640 1 -- 172.21.15.40:0/1688099696 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c099970 con 0x557983566ce0 2024-03-18T22:44:48.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.055+0000 7f6196dd3740 1 -- 172.21.15.40:0/1688099696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 msgr2=0x557983577cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:48.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.055+0000 7f6196dd3740 1 --2- 172.21.15.40:0/1688099696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x557983577cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f617c09b600 tx=0x7f617c0d15d0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.055+0000 7f6196dd3740 1 -- 172.21.15.40:0/1688099696 shutdown_connections 2024-03-18T22:44:48.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.055+0000 7f6196dd3740 1 --2- 172.21.15.40:0/1688099696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798356be50 0x557983561c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.055+0000 7f6196dd3740 1 --2- 172.21.15.40:0/1688099696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x557983577cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.055+0000 7f6196dd3740 1 --2- 172.21.15.40:0/1688099696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 0x55798355ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.055+0000 7f6196dd3740 1 -- 172.21.15.40:0/1688099696 >> 172.21.15.40:0/1688099696 conn(0x557983561860 msgr2=0x557983565350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:48.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6196dd3740 1 -- 172.21.15.40:0/1688099696 shutdown_connections 2024-03-18T22:44:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6196dd3740 1 -- 172.21.15.40:0/1688099696 wait complete. 2024-03-18T22:44:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6196dd3740 1 -- start start 2024-03-18T22:44:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6196dd3740 1 Processor -- start 2024-03-18T22:44:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6196dd3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 0x5579835750b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6196dd3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x5579835755f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6196dd3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798356be50 0x557983576600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6196dd3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f618c1a9740 con 0x55798356be50 2024-03-18T22:44:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6196dd3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f618c1a9070 con 0x5579835662f0 2024-03-18T22:44:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6196dd3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f618c1a62b0 con 0x557983566ce0 2024-03-18T22:44:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6196dd3740 10 client.0 _ll_register_callbacks cb 0x7f618804a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6192783640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798356be50 0x557983576600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6191781640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x5579835755f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6191781640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x5579835755f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41644/0 (socket says 172.21.15.40:41644) 2024-03-18T22:44:48.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6191781640 1 -- 172.21.15.40:0/294987854 learned_addr learned my addr 172.21.15.40:0/294987854 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.056+0000 7f6192783640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798356be50 0x557983576600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33946/0 (socket says 172.21.15.40:33946) 2024-03-18T22:44:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f6191f82640 1 --2- 172.21.15.40:0/294987854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 0x5579835750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f6191781640 1 -- 172.21.15.40:0/294987854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 msgr2=0x5579835750b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f6191781640 1 --2- 172.21.15.40:0/294987854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 0x5579835750b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f6191781640 1 -- 172.21.15.40:0/294987854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798356be50 msgr2=0x557983576600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f6191781640 1 --2- 172.21.15.40:0/294987854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798356be50 0x557983576600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f6191781640 1 -- 172.21.15.40:0/294987854 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557983576bf0 con 0x557983566ce0 2024-03-18T22:44:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f6192783640 1 --2- 172.21.15.40:0/294987854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798356be50 0x557983576600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f6191781640 1 --2- 172.21.15.40:0/294987854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x5579835755f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f617c009920 tx=0x7f617c004560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e026f7a13bfeaf99 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f61757fa640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:44:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f61757fa640 1 -- 172.21.15.40:0/294987854 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c0bb040 con 0x557983566ce0 2024-03-18T22:44:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f61757fa640 1 -- 172.21.15.40:0/294987854 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f617c099710 con 0x557983566ce0 2024-03-18T22:44:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f6196dd3740 1 -- 172.21.15.40:0/294987854 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5579834d79f0 con 0x557983566ce0 2024-03-18T22:44:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f61757fa640 1 -- 172.21.15.40:0/294987854 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f617c09b670 con 0x557983566ce0 2024-03-18T22:44:48.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f6196dd3740 10 client.25909 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:48.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f6196dd3740 1 -- 172.21.15.40:0/294987854 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557983577ef0 con 0x557983566ce0 2024-03-18T22:44:48.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.057+0000 7f61757fa640 1 -- 172.21.15.40:0/294987854 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5579834d79f0 con 0x557983566ce0 2024-03-18T22:44:48.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.058+0000 7f61757fa640 1 -- 172.21.15.40:0/294987854 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f617c0b4270 con 0x557983566ce0 2024-03-18T22:44:48.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.058+0000 7f6196dd3740 10 client.25909 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:48.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.058+0000 7f6196dd3740 10 client.25909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:48.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.058+0000 7f6196dd3740 20 client.25909 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:48.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.058+0000 7f6196dd3740 1 --2- 172.21.15.40:0/294987854 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5579835e2c20 0x5579835e5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:48.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.058+0000 7f6196dd3740 4 client.25909 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:44:48.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.058+0000 7f6196dd3740 1 -- 172.21.15.40:0/294987854 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5579834d1d40 con 0x5579835e2c20 2024-03-18T22:44:48.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.058+0000 7f6191f82640 1 --2- 172.21.15.40:0/294987854 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5579835e2c20 0x5579835e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:48.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.061+0000 7f6191f82640 1 --2- 172.21.15.40:0/294987854 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5579835e2c20 0x5579835e5040 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30d287bad850a26f server_cookie=265c45e74e72146b in_seq=0 out_seq=0 2024-03-18T22:44:48.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.061+0000 7f61757fa640 10 client.25909 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:48.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.061+0000 7f61757fa640 1 -- 172.21.15.40:0/294987854 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5579834d1d40 con 0x5579835e2c20 2024-03-18T22:44:48.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.061+0000 7f61757fa640 10 client.25909 handle_command_reply: tid=0 2024-03-18T22:44:48.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.061+0000 7f6196dd3740 1 client.25909 shutdown 2024-03-18T22:44:48.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.061+0000 7f6196dd3740 1 -- 172.21.15.40:0/294987854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 msgr2=0x5579835755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:48.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.061+0000 7f6196dd3740 1 --2- 172.21.15.40:0/294987854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x5579835755f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f617c009920 tx=0x7f617c004560 comp rx=0 tx=0).stop 2024-03-18T22:44:48.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f6196dd3740 1 -- 172.21.15.40:0/294987854 shutdown_connections 2024-03-18T22:44:48.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f6196dd3740 1 --2- 172.21.15.40:0/294987854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55798356be50 0x557983576600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f6196dd3740 1 --2- 172.21.15.40:0/294987854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557983566ce0 0x5579835755f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f6196dd3740 1 --2- 172.21.15.40:0/294987854 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5579835e2c20 0x5579835e5040 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f6196dd3740 1 --2- 172.21.15.40:0/294987854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5579835662f0 0x5579835750b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f6196dd3740 1 -- 172.21.15.40:0/294987854 >> 172.21.15.40:0/294987854 conn(0x557983561860 msgr2=0x557983565350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:48.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f61757fa640 0 client.25909 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:48.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f6196dd3740 1 -- 172.21.15.40:0/294987854 shutdown_connections 2024-03-18T22:44:48.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f6196dd3740 1 -- 172.21.15.40:0/294987854 wait complete. 2024-03-18T22:44:48.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f6196dd3740 20 client.25909 trim_cache size 0 max 16384 2024-03-18T22:44:48.067 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:44:48.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f61941e6640 1 -- 172.21.15.40:0/4184295910 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f61780a2800 msgr2=0x7f61780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:48.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f61941e6640 1 --2- 172.21.15.40:0/4184295910 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f61780a2800 0x7f61780a4cc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x557983567540 tx=0x7f6184004ab0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f61941e6640 1 -- 172.21.15.40:0/4184295910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f618c101910 msgr2=0x7f618c19c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:48.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f61941e6640 1 --2- 172.21.15.40:0/4184295910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f618c101910 0x7f618c19c790 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f617c00ea60 tx=0x7f617c00ef20 comp rx=0 tx=0).stop 2024-03-18T22:44:48.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f61941e6640 1 -- 172.21.15.40:0/4184295910 shutdown_connections 2024-03-18T22:44:48.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f61941e6640 1 --2- 172.21.15.40:0/4184295910 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f61780a2800 0x7f61780a4cc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f61941e6640 1 --2- 172.21.15.40:0/4184295910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f618c1022d0 0x7f618c19edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f61941e6640 1 --2- 172.21.15.40:0/4184295910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f618c101910 0x7f618c19c790 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f61941e6640 1 --2- 172.21.15.40:0/4184295910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f618c100720 0x7f618c0694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:48.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f61941e6640 1 -- 172.21.15.40:0/4184295910 >> 172.21.15.40:0/4184295910 conn(0x7f618c0fbc90 msgr2=0x7f618c104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:48.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f61941e6640 1 -- 172.21.15.40:0/4184295910 shutdown_connections 2024-03-18T22:44:48.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:48.062+0000 7f61941e6640 1 -- 172.21.15.40:0/4184295910 wait complete. 2024-03-18T22:44:48.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:48 smithi113 ceph-mon[66775]: pgmap v415: 129 pgs: 129 active+clean; 1014 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:44:48.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:48 smithi103 ceph-mon[66377]: pgmap v415: 129 pgs: 129 active+clean; 1014 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:44:48.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:48 smithi040 ceph-mon[62643]: pgmap v415: 129 pgs: 129 active+clean; 1014 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:44:49.076 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:44:49.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:44:49.077 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:44:49.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.191+0000 7f7e8bf09640 1 -- 172.21.15.40:0/2075628494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e84100720 msgr2=0x7f7e84100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.191+0000 7f7e8bf09640 1 --2- 172.21.15.40:0/2075628494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e84100720 0x7f7e84100b40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009860 tx=0x7f7e8002bb70 comp rx=0 tx=0).stop 2024-03-18T22:44:49.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.191+0000 7f7e8bf09640 1 -- 172.21.15.40:0/2075628494 shutdown_connections 2024-03-18T22:44:49.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.191+0000 7f7e8bf09640 1 --2- 172.21.15.40:0/2075628494 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e841022d0 0x7f7e84108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.191+0000 7f7e8bf09640 1 --2- 172.21.15.40:0/2075628494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e84101910 0x7f7e84101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.191+0000 7f7e8bf09640 1 --2- 172.21.15.40:0/2075628494 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e84100720 0x7f7e84100b40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.191+0000 7f7e8bf09640 1 -- 172.21.15.40:0/2075628494 >> 172.21.15.40:0/2075628494 conn(0x7f7e840fbcb0 msgr2=0x7f7e840fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:49.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.191+0000 7f7e8bf09640 1 -- 172.21.15.40:0/2075628494 shutdown_connections 2024-03-18T22:44:49.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.191+0000 7f7e8bf09640 1 -- 172.21.15.40:0/2075628494 wait complete. 2024-03-18T22:44:49.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.191+0000 7f7e8bf09640 1 Processor -- start 2024-03-18T22:44:49.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.191+0000 7f7e8bf09640 1 -- start start 2024-03-18T22:44:49.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e8bf09640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e84100720 0x7f7e84197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e8bf09640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e84101910 0x7f7e84198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e8bf09640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e841022d0 0x7f7e8419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e8bf09640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8410c250 con 0x7f7e84100720 2024-03-18T22:44:49.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e8bf09640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8410c0d0 con 0x7f7e841022d0 2024-03-18T22:44:49.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e8bf09640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8410c3d0 con 0x7f7e84101910 2024-03-18T22:44:49.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e89ca5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e84100720 0x7f7e84197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e89ca5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e84100720 0x7f7e84197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37898/0 (socket says 172.21.15.40:37898) 2024-03-18T22:44:49.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e89ca5640 1 -- 172.21.15.40:0/880486275 learned_addr learned my addr 172.21.15.40:0/880486275 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:49.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e8a4a6640 1 --2- 172.21.15.40:0/880486275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e841022d0 0x7f7e8419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e894a4640 1 --2- 172.21.15.40:0/880486275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e84101910 0x7f7e84198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e89ca5640 1 -- 172.21.15.40:0/880486275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e84101910 msgr2=0x7f7e84198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e89ca5640 1 --2- 172.21.15.40:0/880486275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e84101910 0x7f7e84198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e89ca5640 1 -- 172.21.15.40:0/880486275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e841022d0 msgr2=0x7f7e8419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e89ca5640 1 --2- 172.21.15.40:0/880486275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e841022d0 0x7f7e8419f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e89ca5640 1 -- 172.21.15.40:0/880486275 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e8419f700 con 0x7f7e84100720 2024-03-18T22:44:49.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e8a4a6640 1 --2- 172.21.15.40:0/880486275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e841022d0 0x7f7e8419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:49.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e89ca5640 1 --2- 172.21.15.40:0/880486275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e84100720 0x7f7e84197ae0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7e800057c0 tx=0x7f7e800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:49.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.192+0000 7f7e7affd640 1 -- 172.21.15.40:0/880486275 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e80041030 con 0x7f7e84100720 2024-03-18T22:44:49.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.193+0000 7f7e7affd640 1 -- 172.21.15.40:0/880486275 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e8002c9c0 con 0x7f7e84100720 2024-03-18T22:44:49.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.193+0000 7f7e7affd640 1 -- 172.21.15.40:0/880486275 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e80039040 con 0x7f7e84100720 2024-03-18T22:44:49.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.193+0000 7f7e8bf09640 1 -- 172.21.15.40:0/880486275 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e8419f990 con 0x7f7e84100720 2024-03-18T22:44:49.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.193+0000 7f7e8bf09640 1 -- 172.21.15.40:0/880486275 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e8419fed0 con 0x7f7e84100720 2024-03-18T22:44:49.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.195+0000 7f7e7affd640 1 -- 172.21.15.40:0/880486275 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7e80044070 con 0x7f7e84100720 2024-03-18T22:44:49.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.195+0000 7f7e8ebd4740 1 Processor -- start 2024-03-18T22:44:49.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.196+0000 7f7e8ebd4740 1 -- start start 2024-03-18T22:44:49.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.196+0000 7f7e8ebd4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c4443a0 0x55db9c4447c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.196+0000 7f7e8ebd4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c444d90 0x55db9c449890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.196+0000 7f7e7affd640 1 --2- 172.21.15.40:0/880486275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7e700a2880 0x7f7e700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.196+0000 7f7e8ebd4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c449f00 0x55db9c44c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.196+0000 7f7e8ebd4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55db9c3a9160 con 0x55db9c449f00 2024-03-18T22:44:49.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.196+0000 7f7e8ebd4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55db9c351e10 con 0x55db9c444d90 2024-03-18T22:44:49.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.196+0000 7f7e8ebd4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55db9c3a5200 con 0x55db9c4443a0 2024-03-18T22:44:49.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.196+0000 7f7e894a4640 1 --2- 172.21.15.40:0/880486275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7e700a2880 0x7f7e700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.196+0000 7f7e8a4a6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c449f00 0x55db9c44c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.196+0000 7f7e7affd640 1 -- 172.21.15.40:0/880486275 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7e800f0310 con 0x7f7e84100720 2024-03-18T22:44:49.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.196+0000 7f7e8a4a6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c449f00 0x55db9c44c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37902/0 (socket says 172.21.15.40:37902) 2024-03-18T22:44:49.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.196+0000 7f7e8a4a6640 1 -- 172.21.15.40:0/3735587136 learned_addr learned my addr 172.21.15.40:0/3735587136 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:49.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.196+0000 7f7e8a4a6640 1 --2- 172.21.15.40:0/3735587136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c4443a0 0x55db9c4447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.197+0000 7f7e89ca5640 1 --2- 172.21.15.40:0/3735587136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c444d90 0x55db9c449890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.197+0000 7f7e8a4a6640 1 -- 172.21.15.40:0/3735587136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c4443a0 msgr2=0x55db9c4447c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.197+0000 7f7e8a4a6640 1 --2- 172.21.15.40:0/3735587136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c4443a0 0x55db9c4447c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.197+0000 7f7e8a4a6640 1 -- 172.21.15.40:0/3735587136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c444d90 msgr2=0x55db9c449890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.197+0000 7f7e8a4a6640 1 --2- 172.21.15.40:0/3735587136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c444d90 0x55db9c449890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.197+0000 7f7e8a4a6640 1 -- 172.21.15.40:0/3735587136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db9c1857e0 con 0x55db9c449f00 2024-03-18T22:44:49.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.197+0000 7f7e894a4640 1 --2- 172.21.15.40:0/880486275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7e700a2880 0x7f7e700a4d40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7e84198ff0 tx=0x7f7e74009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:49.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.197+0000 7f7e8a4a6640 1 --2- 172.21.15.40:0/3735587136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c449f00 0x55db9c44c390 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0099e0 tx=0x7f7e7c009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b22155848f26ad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:49.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.197+0000 7f7e6ffff640 1 -- 172.21.15.40:0/3735587136 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c00fc50 con 0x55db9c449f00 2024-03-18T22:44:49.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.197+0000 7f7e6ffff640 1 -- 172.21.15.40:0/3735587136 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e7c003e20 con 0x55db9c449f00 2024-03-18T22:44:49.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.197+0000 7f7e6ffff640 1 -- 172.21.15.40:0/3735587136 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0046e0 con 0x55db9c449f00 2024-03-18T22:44:49.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.198+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/3735587136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c449f00 msgr2=0x55db9c44c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.198+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/3735587136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c449f00 0x55db9c44c390 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0099e0 tx=0x7f7e7c009f30 comp rx=0 tx=0).stop 2024-03-18T22:44:49.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.198+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/3735587136 shutdown_connections 2024-03-18T22:44:49.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.198+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/3735587136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c449f00 0x55db9c44c390 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.198+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/3735587136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c444d90 0x55db9c449890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.198+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/3735587136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c4443a0 0x55db9c4447c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.198+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/3735587136 >> 172.21.15.40:0/3735587136 conn(0x55db9c43f910 msgr2=0x55db9c43fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:49.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.198+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/3735587136 shutdown_connections 2024-03-18T22:44:49.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.198+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/3735587136 wait complete. 2024-03-18T22:44:49.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.199+0000 7f7e8ebd4740 1 -- start start 2024-03-18T22:44:49.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.199+0000 7f7e8ebd4740 1 Processor -- start 2024-03-18T22:44:49.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.199+0000 7f7e8ebd4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c4443a0 0x55db9c44e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.199+0000 7f7e8ebd4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c444d90 0x55db9c44ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.199+0000 7f7e8ebd4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c449f00 0x55db9c455d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.199+0000 7f7e8ebd4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55db9c44ceb0 con 0x55db9c444d90 2024-03-18T22:44:49.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.199+0000 7f7e8ebd4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55db9c3651a0 con 0x55db9c449f00 2024-03-18T22:44:49.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.199+0000 7f7e8ebd4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55db9c44d030 con 0x55db9c4443a0 2024-03-18T22:44:49.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.199+0000 7f7e894a4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c444d90 0x55db9c44ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.199+0000 7f7e8ebd4740 10 client.0 _ll_register_callbacks cb 0x55db9c0bcc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:49.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e894a4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c444d90 0x55db9c44ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37916/0 (socket says 172.21.15.40:37916) 2024-03-18T22:44:49.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e894a4640 1 -- 172.21.15.40:0/2644593691 learned_addr learned my addr 172.21.15.40:0/2644593691 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:49.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e89ca5640 1 --2- 172.21.15.40:0/2644593691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c4443a0 0x55db9c44e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e8a4a6640 1 --2- 172.21.15.40:0/2644593691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c449f00 0x55db9c455d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e894a4640 1 -- 172.21.15.40:0/2644593691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c4443a0 msgr2=0x55db9c44e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e894a4640 1 --2- 172.21.15.40:0/2644593691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c4443a0 0x55db9c44e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e894a4640 1 -- 172.21.15.40:0/2644593691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c449f00 msgr2=0x55db9c455d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e894a4640 1 --2- 172.21.15.40:0/2644593691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c449f00 0x55db9c455d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e894a4640 1 -- 172.21.15.40:0/2644593691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db9c1e5f70 con 0x55db9c444d90 2024-03-18T22:44:49.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e8a4a6640 1 --2- 172.21.15.40:0/2644593691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c449f00 0x55db9c455d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:44:49.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e894a4640 1 --2- 172.21.15.40:0/2644593691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c444d90 0x55db9c44ed00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7e7401c990 tx=0x7f7e74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7820161924c141b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:49.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:44:49.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e6d7fa640 1 -- 172.21.15.40:0/2644593691 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7400e8b0 con 0x55db9c444d90 2024-03-18T22:44:49.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e6d7fa640 1 -- 172.21.15.40:0/2644593691 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7e7400ea50 con 0x55db9c444d90 2024-03-18T22:44:49.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/2644593691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e7c003e20 con 0x55db9c444d90 2024-03-18T22:44:49.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e6d7fa640 1 -- 172.21.15.40:0/2644593691 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7e7400ec50 con 0x55db9c444d90 2024-03-18T22:44:49.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e8ebd4740 10 client.16080 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:49.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.200+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/2644593691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55db9c07d170 con 0x55db9c444d90 2024-03-18T22:44:49.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.201+0000 7f7e6d7fa640 1 -- 172.21.15.40:0/2644593691 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e7c003e20 con 0x55db9c444d90 2024-03-18T22:44:49.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.201+0000 7f7e6d7fa640 1 -- 172.21.15.40:0/2644593691 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f7e7401d420 con 0x55db9c444d90 2024-03-18T22:44:49.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.201+0000 7f7e8ebd4740 10 client.16080 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:49.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.201+0000 7f7e8ebd4740 10 client.16080 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:49.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.201+0000 7f7e8ebd4740 20 client.16080 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:49.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.201+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/2644593691 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55db9c45a690 0x55db9c45cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.201+0000 7f7e8ebd4740 4 client.16080 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:49.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.201+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/2644593691 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db9c3b66a0 con 0x55db9c45a690 2024-03-18T22:44:49.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.201+0000 7f7e89ca5640 1 --2- 172.21.15.40:0/2644593691 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55db9c45a690 0x55db9c45cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.201+0000 7f7e6d7fa640 10 client.16080 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:49.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.201+0000 7f7e89ca5640 1 --2- 172.21.15.40:0/2644593691 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55db9c45a690 0x55db9c45cad0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=365beb6dc00c12db server_cookie=c6ac433e9437c7d8 in_seq=0 out_seq=0 2024-03-18T22:44:49.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.202+0000 7f7e6d7fa640 1 -- 172.21.15.40:0/2644593691 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55db9c3b66a0 con 0x55db9c45a690 2024-03-18T22:44:49.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.202+0000 7f7e6d7fa640 10 client.16080 handle_command_reply: tid=0 2024-03-18T22:44:49.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.202+0000 7f7e8ebd4740 1 client.16080 shutdown 2024-03-18T22:44:49.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.202+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/2644593691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c444d90 msgr2=0x55db9c44ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.202+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/2644593691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c444d90 0x55db9c44ed00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7e7401c990 tx=0x7f7e74004730 comp rx=0 tx=0).stop 2024-03-18T22:44:49.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.202+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/2644593691 shutdown_connections 2024-03-18T22:44:49.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.202+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/2644593691 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55db9c45a690 0x55db9c45cad0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.202+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/2644593691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c449f00 0x55db9c455d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.202+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/2644593691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c444d90 0x55db9c44ed00 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.202+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/2644593691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c4443a0 0x55db9c44e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.202+0000 7f7e6d7fa640 0 client.16080 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:49.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.202+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/2644593691 >> 172.21.15.40:0/2644593691 conn(0x55db9c43f910 msgr2=0x55db9c43eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:49.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.203+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/2644593691 shutdown_connections 2024-03-18T22:44:49.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.203+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/2644593691 wait complete. 2024-03-18T22:44:49.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.203+0000 7f7e8ebd4740 20 client.16080 trim_cache size 0 max 16384 2024-03-18T22:44:49.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.223+0000 7f7e8ebd4740 1 Processor -- start 2024-03-18T22:44:49.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e8ebd4740 1 -- start start 2024-03-18T22:44:49.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e8ebd4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c4443a0 0x55db9c43e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e8ebd4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c444d90 0x55db9c455d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e8ebd4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c449f00 0x55db9c43e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e8ebd4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7e841aa570 con 0x55db9c449f00 2024-03-18T22:44:49.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e8ebd4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7e841ac170 con 0x55db9c444d90 2024-03-18T22:44:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e8ebd4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7e841aac40 con 0x55db9c4443a0 2024-03-18T22:44:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e8a4a6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c449f00 0x55db9c43e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e8a4a6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c449f00 0x55db9c43e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37918/0 (socket says 172.21.15.40:37918) 2024-03-18T22:44:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e894a4640 1 --2- 172.21.15.40:0/3951828036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c444d90 0x55db9c455d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e8a4a6640 1 -- 172.21.15.40:0/3951828036 learned_addr learned my addr 172.21.15.40:0/3951828036 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e89ca5640 1 --2- 172.21.15.40:0/3951828036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c4443a0 0x55db9c43e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e894a4640 1 -- 172.21.15.40:0/3951828036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c4443a0 msgr2=0x55db9c43e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e894a4640 1 --2- 172.21.15.40:0/3951828036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c4443a0 0x55db9c43e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e894a4640 1 -- 172.21.15.40:0/3951828036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c449f00 msgr2=0x55db9c43e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e894a4640 1 --2- 172.21.15.40:0/3951828036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c449f00 0x55db9c43e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.224+0000 7f7e894a4640 1 -- 172.21.15.40:0/3951828036 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db9c440680 con 0x55db9c444d90 2024-03-18T22:44:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.225+0000 7f7e894a4640 1 --2- 172.21.15.40:0/3951828036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c444d90 0x55db9c455d00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7e7401d010 tx=0x7f7e74009f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba692a0a32ec52f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:49.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.225+0000 7f7e6f7fe640 1 -- 172.21.15.40:0/3951828036 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7403b020 con 0x55db9c444d90 2024-03-18T22:44:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.225+0000 7f7e6f7fe640 1 -- 172.21.15.40:0/3951828036 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e7401c750 con 0x55db9c444d90 2024-03-18T22:44:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.225+0000 7f7e6f7fe640 1 -- 172.21.15.40:0/3951828036 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e740286e0 con 0x55db9c444d90 2024-03-18T22:44:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.225+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/3951828036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c444d90 msgr2=0x55db9c455d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.225+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/3951828036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c444d90 0x55db9c455d00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7e7401d010 tx=0x7f7e74009f00 comp rx=0 tx=0).stop 2024-03-18T22:44:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/3951828036 shutdown_connections 2024-03-18T22:44:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/3951828036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c449f00 0x55db9c43e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/3951828036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c444d90 0x55db9c455d00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/3951828036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c4443a0 0x55db9c43e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/3951828036 >> 172.21.15.40:0/3951828036 conn(0x55db9c43f910 msgr2=0x55db9c44e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:49.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/3951828036 shutdown_connections 2024-03-18T22:44:49.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/3951828036 wait complete. 2024-03-18T22:44:49.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 1 -- start start 2024-03-18T22:44:49.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 1 Processor -- start 2024-03-18T22:44:49.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c4443a0 0x55db9c4537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c444d90 0x55db9c453cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c449f00 0x55db9c454d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7e841ab9e0 con 0x55db9c4443a0 2024-03-18T22:44:49.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7e841ab310 con 0x55db9c449f00 2024-03-18T22:44:49.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7e841a8550 con 0x55db9c444d90 2024-03-18T22:44:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.226+0000 7f7e8ebd4740 10 client.0 _ll_register_callbacks cb 0x7f7e74009c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e89ca5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c4443a0 0x55db9c4537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e8a4a6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c449f00 0x55db9c454d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e89ca5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c4443a0 0x55db9c4537b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37924/0 (socket says 172.21.15.40:37924) 2024-03-18T22:44:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e8a4a6640 1 --2- 172.21.15.40:0/192171491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c449f00 0x55db9c454d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:53686/0 (socket says 172.21.15.40:53686) 2024-03-18T22:44:49.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e894a4640 1 --2- 172.21.15.40:0/192171491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c444d90 0x55db9c453cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e89ca5640 1 -- 172.21.15.40:0/192171491 learned_addr learned my addr 172.21.15.40:0/192171491 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:49.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e8a4a6640 1 -- 172.21.15.40:0/192171491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c444d90 msgr2=0x55db9c453cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e8a4a6640 1 --2- 172.21.15.40:0/192171491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c444d90 0x55db9c453cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e8a4a6640 1 -- 172.21.15.40:0/192171491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c4443a0 msgr2=0x55db9c4537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e8a4a6640 1 --2- 172.21.15.40:0/192171491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c4443a0 0x55db9c4537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e8a4a6640 1 -- 172.21.15.40:0/192171491 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db9c4552f0 con 0x55db9c449f00 2024-03-18T22:44:49.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e8a4a6640 1 --2- 172.21.15.40:0/192171491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c449f00 0x55db9c454d00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x55db9c45a610 tx=0x7f7e7c011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48e7805a899abb0a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:49.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:44:49.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e6d7fa640 1 -- 172.21.15.40:0/192171491 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c003c40 con 0x55db9c449f00 2024-03-18T22:44:49.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/192171491 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55db9c44e5b0 con 0x55db9c449f00 2024-03-18T22:44:49.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.227+0000 7f7e6d7fa640 1 -- 172.21.15.40:0/192171491 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7e7c003de0 con 0x55db9c449f00 2024-03-18T22:44:49.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.228+0000 7f7e6d7fa640 1 -- 172.21.15.40:0/192171491 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7e7c02a3f0 con 0x55db9c449f00 2024-03-18T22:44:49.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.228+0000 7f7e8ebd4740 10 client.25934 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:49.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.228+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/192171491 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55db9c4c0e50 con 0x55db9c449f00 2024-03-18T22:44:49.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.228+0000 7f7e6d7fa640 1 -- 172.21.15.40:0/192171491 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55db9c44e5b0 con 0x55db9c449f00 2024-03-18T22:44:49.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.228+0000 7f7e6d7fa640 1 -- 172.21.15.40:0/192171491 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f7e7c028070 con 0x55db9c449f00 2024-03-18T22:44:49.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.228+0000 7f7e8ebd4740 10 client.25934 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:49.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.228+0000 7f7e8ebd4740 10 client.25934 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:49.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.228+0000 7f7e8ebd4740 20 client.25934 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:49.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.228+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/192171491 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55db9c4c1630 0x55db9c4c3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.228+0000 7f7e8ebd4740 4 client.25934 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:44:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.228+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/192171491 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55db9c3b09f0 con 0x55db9c4c1630 2024-03-18T22:44:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.228+0000 7f7e89ca5640 1 --2- 172.21.15.40:0/192171491 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55db9c4c1630 0x55db9c4c3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.229+0000 7f7e6d7fa640 10 client.25934 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.229+0000 7f7e89ca5640 1 --2- 172.21.15.40:0/192171491 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55db9c4c1630 0x55db9c4c3a50 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9c07a9b9064c55b server_cookie=a0952736be30eb29 in_seq=0 out_seq=0 2024-03-18T22:44:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.230+0000 7f7e6d7fa640 1 -- 172.21.15.40:0/192171491 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55db9c3b09f0 con 0x55db9c4c1630 2024-03-18T22:44:49.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.230+0000 7f7e6d7fa640 10 client.25934 handle_command_reply: tid=0 2024-03-18T22:44:49.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.230+0000 7f7e8ebd4740 1 client.25934 shutdown 2024-03-18T22:44:49.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.230+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/192171491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c449f00 msgr2=0x55db9c454d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.230+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/192171491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c449f00 0x55db9c454d00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x55db9c45a610 tx=0x7f7e7c011900 comp rx=0 tx=0).stop 2024-03-18T22:44:49.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.230+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/192171491 shutdown_connections 2024-03-18T22:44:49.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.230+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/192171491 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55db9c4c1630 0x55db9c4c3a50 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.230+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/192171491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db9c449f00 0x55db9c454d00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.230+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/192171491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55db9c444d90 0x55db9c453cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.230+0000 7f7e6d7fa640 0 client.25934 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:49.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.230+0000 7f7e8ebd4740 1 --2- 172.21.15.40:0/192171491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55db9c4443a0 0x55db9c4537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.230+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/192171491 >> 172.21.15.40:0/192171491 conn(0x55db9c43f910 msgr2=0x55db9c442b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:49.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.230+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/192171491 shutdown_connections 2024-03-18T22:44:49.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.230+0000 7f7e8ebd4740 1 -- 172.21.15.40:0/192171491 wait complete. 2024-03-18T22:44:49.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.230+0000 7f7e8ebd4740 20 client.25934 trim_cache size 0 max 16384 2024-03-18T22:44:49.248 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:44:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:44:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "9451b9e5-448b-4323-969d-83ccf36e994f", 2024-03-18T22:44:49.249 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:44:49.249 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:44:49.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.231+0000 7f7e8bf09640 1 -- 172.21.15.40:0/880486275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7e700a2880 msgr2=0x7f7e700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.231+0000 7f7e8bf09640 1 --2- 172.21.15.40:0/880486275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7e700a2880 0x7f7e700a4d40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7e84198ff0 tx=0x7f7e74009260 comp rx=0 tx=0).stop 2024-03-18T22:44:49.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.231+0000 7f7e8bf09640 1 -- 172.21.15.40:0/880486275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e84100720 msgr2=0x7f7e84197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.231+0000 7f7e8bf09640 1 --2- 172.21.15.40:0/880486275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e84100720 0x7f7e84197ae0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7e800057c0 tx=0x7f7e800042f0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.231+0000 7f7e8bf09640 1 -- 172.21.15.40:0/880486275 shutdown_connections 2024-03-18T22:44:49.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.231+0000 7f7e8bf09640 1 --2- 172.21.15.40:0/880486275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7e700a2880 0x7f7e700a4d40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.231+0000 7f7e8bf09640 1 --2- 172.21.15.40:0/880486275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e841022d0 0x7f7e8419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.231+0000 7f7e8bf09640 1 --2- 172.21.15.40:0/880486275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7e84101910 0x7f7e84198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.231+0000 7f7e8bf09640 1 --2- 172.21.15.40:0/880486275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e84100720 0x7f7e84197ae0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.231+0000 7f7e8bf09640 1 -- 172.21.15.40:0/880486275 >> 172.21.15.40:0/880486275 conn(0x7f7e840fbcb0 msgr2=0x7f7e84068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:49.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.231+0000 7f7e8bf09640 1 -- 172.21.15.40:0/880486275 shutdown_connections 2024-03-18T22:44:49.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.231+0000 7f7e8bf09640 1 -- 172.21.15.40:0/880486275 wait complete. 2024-03-18T22:44:49.253 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:44:49.253 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:44:49.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.374+0000 7fa195285640 1 -- 172.21.15.40:0/1925217298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1901000d0 msgr2=0x7fa1901004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.374+0000 7fa195285640 1 --2- 172.21.15.40:0/1925217298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1901000d0 0x7fa1901004f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa180009ca0 tx=0x7fa18002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.375+0000 7fa195285640 1 -- 172.21.15.40:0/1925217298 shutdown_connections 2024-03-18T22:44:49.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.375+0000 7fa195285640 1 --2- 172.21.15.40:0/1925217298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1901013d0 0x7fa190101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.375+0000 7fa195285640 1 --2- 172.21.15.40:0/1925217298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa190100a30 0x7fa190100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.375+0000 7fa195285640 1 --2- 172.21.15.40:0/1925217298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1901000d0 0x7fa1901004f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.375+0000 7fa195285640 1 -- 172.21.15.40:0/1925217298 >> 172.21.15.40:0/1925217298 conn(0x7fa1900fbcd0 msgr2=0x7fa1900fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:49.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.375+0000 7fa195285640 1 -- 172.21.15.40:0/1925217298 shutdown_connections 2024-03-18T22:44:49.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.375+0000 7fa195285640 1 -- 172.21.15.40:0/1925217298 wait complete. 2024-03-18T22:44:49.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.375+0000 7fa195285640 1 Processor -- start 2024-03-18T22:44:49.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.375+0000 7fa195285640 1 -- start start 2024-03-18T22:44:49.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa195285640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1901000d0 0x7fa190199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa195285640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa190100a30 0x7fa19019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa195285640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1901013d0 0x7fa1901a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa195285640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa19010f570 con 0x7fa1901013d0 2024-03-18T22:44:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa195285640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa19010f3f0 con 0x7fa190100a30 2024-03-18T22:44:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa195285640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa19010f6f0 con 0x7fa1901000d0 2024-03-18T22:44:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa18f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1901013d0 0x7fa1901a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa18e7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa190100a30 0x7fa19019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa18f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1901013d0 0x7fa1901a1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37956/0 (socket says 172.21.15.40:37956) 2024-03-18T22:44:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa18effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1901000d0 0x7fa190199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa18f7fe640 1 -- 172.21.15.40:0/887214247 learned_addr learned my addr 172.21.15.40:0/887214247 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa18e7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa190100a30 0x7fa19019a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:53706/0 (socket says 172.21.15.40:53706) 2024-03-18T22:44:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa18e7fc640 1 -- 172.21.15.40:0/887214247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1901000d0 msgr2=0x7fa190199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/887214247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1901000d0 0x7fa190199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa18e7fc640 1 -- 172.21.15.40:0/887214247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1901013d0 msgr2=0x7fa1901a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/887214247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1901013d0 0x7fa1901a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa18e7fc640 1 -- 172.21.15.40:0/887214247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1901a1900 con 0x7fa190100a30 2024-03-18T22:44:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa18effd640 1 --2- 172.21.15.40:0/887214247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1901000d0 0x7fa190199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa18f7fe640 1 --2- 172.21.15.40:0/887214247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1901013d0 0x7fa1901a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.376+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/887214247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa190100a30 0x7fa19019a250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa18400e9f0 tx=0x7fa18400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.377+0000 7fa177fff640 1 -- 172.21.15.40:0/887214247 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18400ccc0 con 0x7fa190100a30 2024-03-18T22:44:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.377+0000 7fa195285640 1 -- 172.21.15.40:0/887214247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1901a1bf0 con 0x7fa190100a30 2024-03-18T22:44:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.377+0000 7fa177fff640 1 -- 172.21.15.40:0/887214247 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1840108e0 con 0x7fa190100a30 2024-03-18T22:44:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.377+0000 7fa177fff640 1 -- 172.21.15.40:0/887214247 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184018880 con 0x7fa190100a30 2024-03-18T22:44:49.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.377+0000 7fa195285640 1 -- 172.21.15.40:0/887214247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1901a2190 con 0x7fa190100a30 2024-03-18T22:44:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.379+0000 7fa197e74740 1 Processor -- start 2024-03-18T22:44:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.379+0000 7fa197e74740 1 -- start start 2024-03-18T22:44:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.379+0000 7fa177fff640 1 -- 172.21.15.40:0/887214247 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa184018a20 con 0x7fa190100a30 2024-03-18T22:44:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.379+0000 7fa197e74740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a1a50 0x5575986a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.379+0000 7fa197e74740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a2440 0x5575986a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.379+0000 7fa197e74740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a75b0 0x5575986a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.379+0000 7fa197e74740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557598606500 con 0x5575986a75b0 2024-03-18T22:44:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.379+0000 7fa197e74740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5575985aebe0 con 0x5575986a2440 2024-03-18T22:44:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.379+0000 7fa197e74740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5575986025a0 con 0x5575986a1a50 2024-03-18T22:44:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.379+0000 7fa18effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a2440 0x5575986a6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.379+0000 7fa18e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a75b0 0x5575986a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.379+0000 7fa18effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a2440 0x5575986a6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:53718/0 (socket says 172.21.15.40:53718) 2024-03-18T22:44:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.379+0000 7fa18effd640 1 -- 172.21.15.40:0/346973514 learned_addr learned my addr 172.21.15.40:0/346973514 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.380+0000 7fa18f7fe640 1 --2- 172.21.15.40:0/346973514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a1a50 0x5575986a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.380+0000 7fa18e7fc640 1 -- 172.21.15.40:0/346973514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a1a50 msgr2=0x5575986a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.380+0000 7fa177fff640 1 --2- 172.21.15.40:0/887214247 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa1780a2800 0x7fa1780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.380+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/346973514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a1a50 0x5575986a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.380+0000 7fa18e7fc640 1 -- 172.21.15.40:0/346973514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a2440 msgr2=0x5575986a6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.380+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/346973514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a2440 0x5575986a6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.380+0000 7fa18e7fc640 1 -- 172.21.15.40:0/346973514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575985d0000 con 0x5575986a75b0 2024-03-18T22:44:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.380+0000 7fa18f7fe640 1 --2- 172.21.15.40:0/346973514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a1a50 0x5575986a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.380+0000 7fa18f7fe640 1 --2- 172.21.15.40:0/887214247 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa1780a2800 0x7fa1780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.380+0000 7fa177fff640 1 -- 172.21.15.40:0/887214247 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa1840cf190 con 0x7fa190100a30 2024-03-18T22:44:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.380+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/346973514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a75b0 0x5575986a9a40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa1840cc870 tx=0x7fa1840d2f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f788170fb127aaab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.380+0000 7fa1757fa640 1 -- 172.21.15.40:0/346973514 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184026030 con 0x5575986a75b0 2024-03-18T22:44:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.380+0000 7fa1757fa640 1 -- 172.21.15.40:0/346973514 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa184010e00 con 0x5575986a75b0 2024-03-18T22:44:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.380+0000 7fa1757fa640 1 -- 172.21.15.40:0/346973514 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18409b7f0 con 0x5575986a75b0 2024-03-18T22:44:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.381+0000 7fa197e74740 1 -- 172.21.15.40:0/346973514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a75b0 msgr2=0x5575986a9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.381+0000 7fa197e74740 1 --2- 172.21.15.40:0/346973514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a75b0 0x5575986a9a40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa1840cc870 tx=0x7fa1840d2f80 comp rx=0 tx=0).stop 2024-03-18T22:44:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.381+0000 7fa18f7fe640 1 --2- 172.21.15.40:0/887214247 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa1780a2800 0x7fa1780a4cc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x5575986a2ca0 tx=0x7fa1880046c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:49.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.381+0000 7fa197e74740 1 -- 172.21.15.40:0/346973514 shutdown_connections 2024-03-18T22:44:49.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.381+0000 7fa197e74740 1 --2- 172.21.15.40:0/346973514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a75b0 0x5575986a9a40 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.381+0000 7fa197e74740 1 --2- 172.21.15.40:0/346973514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a2440 0x5575986a6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.381+0000 7fa197e74740 1 --2- 172.21.15.40:0/346973514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a1a50 0x5575986a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.381+0000 7fa197e74740 1 -- 172.21.15.40:0/346973514 >> 172.21.15.40:0/346973514 conn(0x55759869cfc0 msgr2=0x55759869d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:49.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.381+0000 7fa197e74740 1 -- 172.21.15.40:0/346973514 shutdown_connections 2024-03-18T22:44:49.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.381+0000 7fa197e74740 1 -- 172.21.15.40:0/346973514 wait complete. 2024-03-18T22:44:49.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.381+0000 7fa197e74740 1 -- start start 2024-03-18T22:44:49.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa197e74740 1 Processor -- start 2024-03-18T22:44:49.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa197e74740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a1a50 0x5575986abe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa197e74740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a2440 0x5575986ac3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa197e74740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a75b0 0x5575986b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa197e74740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5575986aa560 con 0x5575986a1a50 2024-03-18T22:44:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa197e74740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5575985c2160 con 0x5575986a2440 2024-03-18T22:44:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa197e74740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5575986aa6e0 con 0x5575986a75b0 2024-03-18T22:44:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa197e74740 10 client.0 _ll_register_callbacks cb 0x5575985fea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa18effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a1a50 0x5575986abe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa18effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a1a50 0x5575986abe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37966/0 (socket says 172.21.15.40:37966) 2024-03-18T22:44:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa18effd640 1 -- 172.21.15.40:0/2326814752 learned_addr learned my addr 172.21.15.40:0/2326814752 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/2326814752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a2440 0x5575986ac3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa18f7fe640 1 --2- 172.21.15.40:0/2326814752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a75b0 0x5575986b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa18e7fc640 1 -- 172.21.15.40:0/2326814752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a75b0 msgr2=0x5575986b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/2326814752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a75b0 0x5575986b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa18e7fc640 1 -- 172.21.15.40:0/2326814752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a1a50 msgr2=0x5575986abe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/2326814752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a1a50 0x5575986abe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa18e7fc640 1 -- 172.21.15.40:0/2326814752 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575983e47d0 con 0x5575986a2440 2024-03-18T22:44:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa18effd640 1 --2- 172.21.15.40:0/2326814752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a1a50 0x5575986abe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/2326814752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a2440 0x5575986ac3b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa1840cc8f0 tx=0x7fa1840cf490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0ae5e4bacaa9655 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.382+0000 7fa172ff5640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:44:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.383+0000 7fa172ff5640 1 -- 172.21.15.40:0/2326814752 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18409b5c0 con 0x5575986a2440 2024-03-18T22:44:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.383+0000 7fa172ff5640 1 -- 172.21.15.40:0/2326814752 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa1840b4580 con 0x5575986a2440 2024-03-18T22:44:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.383+0000 7fa197e74740 1 -- 172.21.15.40:0/2326814752 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557598613a40 con 0x5575986a2440 2024-03-18T22:44:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.383+0000 7fa172ff5640 1 -- 172.21.15.40:0/2326814752 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa18409f890 con 0x5575986a2440 2024-03-18T22:44:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.383+0000 7fa197e74740 10 client.25946 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.383+0000 7fa197e74740 1 -- 172.21.15.40:0/2326814752 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55759831a3f0 con 0x5575986a2440 2024-03-18T22:44:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.383+0000 7fa172ff5640 1 -- 172.21.15.40:0/2326814752 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x557598613a40 con 0x5575986a2440 2024-03-18T22:44:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.383+0000 7fa172ff5640 1 -- 172.21.15.40:0/2326814752 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fa184010a90 con 0x5575986a2440 2024-03-18T22:44:49.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.383+0000 7fa197e74740 10 client.25946 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:49.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.383+0000 7fa197e74740 10 client.25946 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:49.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.383+0000 7fa197e74740 20 client.25946 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:49.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.383+0000 7fa197e74740 1 --2- 172.21.15.40:0/2326814752 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5575986b78e0 0x5575986b9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.383+0000 7fa197e74740 4 client.25946 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:44:49.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.383+0000 7fa197e74740 1 -- 172.21.15.40:0/2326814752 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55759860dd90 con 0x5575986b78e0 2024-03-18T22:44:49.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.384+0000 7fa18effd640 1 --2- 172.21.15.40:0/2326814752 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5575986b78e0 0x5575986b9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.384+0000 7fa172ff5640 10 client.25946 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:49.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.384+0000 7fa18effd640 1 --2- 172.21.15.40:0/2326814752 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5575986b78e0 0x5575986b9cd0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b6fb3dff75a069 server_cookie=8c88d959a6c9fc11 in_seq=0 out_seq=0 2024-03-18T22:44:49.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.385+0000 7fa172ff5640 1 -- 172.21.15.40:0/2326814752 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55759860dd90 con 0x5575986b78e0 2024-03-18T22:44:49.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.385+0000 7fa172ff5640 10 client.25946 handle_command_reply: tid=0 2024-03-18T22:44:49.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.385+0000 7fa197e74740 1 client.25946 shutdown 2024-03-18T22:44:49.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.385+0000 7fa197e74740 1 -- 172.21.15.40:0/2326814752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a2440 msgr2=0x5575986ac3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.385+0000 7fa197e74740 1 --2- 172.21.15.40:0/2326814752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a2440 0x5575986ac3b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa1840cc8f0 tx=0x7fa1840cf490 comp rx=0 tx=0).stop 2024-03-18T22:44:49.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.385+0000 7fa197e74740 1 -- 172.21.15.40:0/2326814752 shutdown_connections 2024-03-18T22:44:49.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.385+0000 7fa197e74740 1 --2- 172.21.15.40:0/2326814752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a75b0 0x5575986b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.385+0000 7fa197e74740 1 --2- 172.21.15.40:0/2326814752 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5575986b78e0 0x5575986b9cd0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.385+0000 7fa197e74740 1 --2- 172.21.15.40:0/2326814752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a2440 0x5575986ac3b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.385+0000 7fa197e74740 1 --2- 172.21.15.40:0/2326814752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a1a50 0x5575986abe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.385+0000 7fa172ff5640 0 client.25946 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:49.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.385+0000 7fa197e74740 1 -- 172.21.15.40:0/2326814752 >> 172.21.15.40:0/2326814752 conn(0x55759869cfc0 msgr2=0x55759869c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:49.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.385+0000 7fa197e74740 1 -- 172.21.15.40:0/2326814752 shutdown_connections 2024-03-18T22:44:49.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.385+0000 7fa197e74740 1 -- 172.21.15.40:0/2326814752 wait complete. 2024-03-18T22:44:49.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.385+0000 7fa197e74740 20 client.25946 trim_cache size 0 max 16384 2024-03-18T22:44:49.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.405+0000 7fa197e74740 1 Processor -- start 2024-03-18T22:44:49.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.405+0000 7fa197e74740 1 -- start start 2024-03-18T22:44:49.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.405+0000 7fa197e74740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a1a50 0x55759869b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.405+0000 7fa197e74740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a2440 0x5575986b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.405+0000 7fa197e74740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a75b0 0x55759869be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.405+0000 7fa197e74740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa1901ac7a0 con 0x5575986a2440 2024-03-18T22:44:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.405+0000 7fa197e74740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa1901ae3a0 con 0x5575986a1a50 2024-03-18T22:44:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.405+0000 7fa197e74740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa1901ace70 con 0x5575986a75b0 2024-03-18T22:44:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.405+0000 7fa18e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a2440 0x5575986b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.405+0000 7fa18e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a2440 0x5575986b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37976/0 (socket says 172.21.15.40:37976) 2024-03-18T22:44:49.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.406+0000 7fa18e7fc640 1 -- 172.21.15.40:0/2327612599 learned_addr learned my addr 172.21.15.40:0/2327612599 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:49.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.406+0000 7fa18effd640 1 --2- 172.21.15.40:0/2327612599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a1a50 0x55759869b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.406+0000 7fa18f7fe640 1 --2- 172.21.15.40:0/2327612599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a75b0 0x55759869be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.406+0000 7fa18e7fc640 1 -- 172.21.15.40:0/2327612599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a75b0 msgr2=0x55759869be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.406+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/2327612599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a75b0 0x55759869be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.406+0000 7fa18e7fc640 1 -- 172.21.15.40:0/2327612599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a1a50 msgr2=0x55759869b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.406+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/2327612599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a1a50 0x55759869b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.406+0000 7fa18e7fc640 1 -- 172.21.15.40:0/2327612599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575982da0f0 con 0x5575986a2440 2024-03-18T22:44:49.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.406+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/2327612599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a2440 0x5575986b33b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa18400e9c0 tx=0x7fa1840af9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d848a3ebb0e4115 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:49.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.406+0000 7fa174ff9640 1 -- 172.21.15.40:0/2327612599 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18409b5c0 con 0x5575986a2440 2024-03-18T22:44:49.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.406+0000 7fa174ff9640 1 -- 172.21.15.40:0/2327612599 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa18409b7f0 con 0x5575986a2440 2024-03-18T22:44:49.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.406+0000 7fa174ff9640 1 -- 172.21.15.40:0/2327612599 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184010a90 con 0x5575986a2440 2024-03-18T22:44:49.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.407+0000 7fa197e74740 1 -- 172.21.15.40:0/2327612599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a2440 msgr2=0x5575986b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.407+0000 7fa197e74740 1 --2- 172.21.15.40:0/2327612599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a2440 0x5575986b33b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa18400e9c0 tx=0x7fa1840af9f0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.407+0000 7fa197e74740 1 -- 172.21.15.40:0/2327612599 shutdown_connections 2024-03-18T22:44:49.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.407+0000 7fa197e74740 1 --2- 172.21.15.40:0/2327612599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a75b0 0x55759869be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.407+0000 7fa197e74740 1 --2- 172.21.15.40:0/2327612599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a2440 0x5575986b33b0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.407+0000 7fa197e74740 1 --2- 172.21.15.40:0/2327612599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a1a50 0x55759869b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.407+0000 7fa197e74740 1 -- 172.21.15.40:0/2327612599 >> 172.21.15.40:0/2327612599 conn(0x55759869cfc0 msgr2=0x5575986abe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.407+0000 7fa197e74740 1 -- 172.21.15.40:0/2327612599 shutdown_connections 2024-03-18T22:44:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.407+0000 7fa197e74740 1 -- 172.21.15.40:0/2327612599 wait complete. 2024-03-18T22:44:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.407+0000 7fa197e74740 1 -- start start 2024-03-18T22:44:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.407+0000 7fa197e74740 1 Processor -- start 2024-03-18T22:44:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa197e74740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a1a50 0x5575986b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa197e74740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a2440 0x5575986b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa197e74740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a75b0 0x5575986b1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa197e74740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa1901adc10 con 0x5575986a2440 2024-03-18T22:44:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa197e74740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa1901ad540 con 0x5575986a75b0 2024-03-18T22:44:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa197e74740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa1901aa780 con 0x5575986a1a50 2024-03-18T22:44:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa197e74740 10 client.0 _ll_register_callbacks cb 0x7fa184005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:44:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa18effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a1a50 0x5575986b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa18effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a1a50 0x5575986b0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:40086/0 (socket says 172.21.15.40:40086) 2024-03-18T22:44:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa18e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a2440 0x5575986b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa18effd640 1 -- 172.21.15.40:0/2794741732 learned_addr learned my addr 172.21.15.40:0/2794741732 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:44:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa18f7fe640 1 --2- 172.21.15.40:0/2794741732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a75b0 0x5575986b1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa18effd640 1 -- 172.21.15.40:0/2794741732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a75b0 msgr2=0x5575986b1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa18effd640 1 --2- 172.21.15.40:0/2794741732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a75b0 0x5575986b1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa18effd640 1 -- 172.21.15.40:0/2794741732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a2440 msgr2=0x5575986b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:44:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa18effd640 1 --2- 172.21.15.40:0/2794741732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a2440 0x5575986b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa18effd640 1 -- 172.21.15.40:0/2794741732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575986b24a0 con 0x5575986a1a50 2024-03-18T22:44:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/2794741732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a2440 0x5575986b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:44:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa18effd640 1 --2- 172.21.15.40:0/2794741732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a1a50 0x5575986b0a70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa18002e010 tx=0x7fa180032900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=928ba0c34fa2e4e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:44:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.408+0000 7fa172ff5640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:44:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.409+0000 7fa172ff5640 1 -- 172.21.15.40:0/2794741732 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa180039040 con 0x5575986a1a50 2024-03-18T22:44:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.409+0000 7fa172ff5640 1 -- 172.21.15.40:0/2794741732 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa180048a30 con 0x5575986a1a50 2024-03-18T22:44:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.409+0000 7fa197e74740 1 -- 172.21.15.40:0/2794741732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa18409b7f0 con 0x5575986a1a50 2024-03-18T22:44:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.409+0000 7fa172ff5640 1 -- 172.21.15.40:0/2794741732 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa180059050 con 0x5575986a1a50 2024-03-18T22:44:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.409+0000 7fa197e74740 10 client.25945 fetch_fsmap learned FSMap version 30 2024-03-18T22:44:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.409+0000 7fa197e74740 1 -- 172.21.15.40:0/2794741732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55759871db50 con 0x5575986a1a50 2024-03-18T22:44:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.409+0000 7fa172ff5640 1 -- 172.21.15.40:0/2794741732 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa18409b7f0 con 0x5575986a1a50 2024-03-18T22:44:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.409+0000 7fa172ff5640 1 -- 172.21.15.40:0/2794741732 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fa1800485c0 con 0x5575986a1a50 2024-03-18T22:44:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.409+0000 7fa197e74740 10 client.25945 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:44:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.409+0000 7fa197e74740 10 client.25945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:44:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.409+0000 7fa197e74740 20 client.25945 populate_metadata read hostname 'smithi040' 2024-03-18T22:44:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.409+0000 7fa197e74740 1 --2- 172.21.15.40:0/2794741732 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55759871e580 0x5575987209a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:44:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.409+0000 7fa197e74740 4 client.25945 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:44:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.409+0000 7fa197e74740 1 -- 172.21.15.40:0/2794741732 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55759858b540 con 0x55759871e580 2024-03-18T22:44:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.409+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/2794741732 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55759871e580 0x5575987209a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:44:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa172ff5640 10 client.25945 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa18e7fc640 1 --2- 172.21.15.40:0/2794741732 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55759871e580 0x5575987209a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51b55ec46ded4cb6 server_cookie=bdb60842fcdb7c27 in_seq=0 out_seq=0 2024-03-18T22:44:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa172ff5640 1 -- 172.21.15.40:0/2794741732 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55759858b540 con 0x55759871e580 2024-03-18T22:44:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa172ff5640 10 client.25945 handle_command_reply: tid=0 2024-03-18T22:44:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa197e74740 1 client.25945 shutdown 2024-03-18T22:44:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa197e74740 1 -- 172.21.15.40:0/2794741732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a1a50 msgr2=0x5575986b0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa197e74740 1 --2- 172.21.15.40:0/2794741732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a1a50 0x5575986b0a70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa18002e010 tx=0x7fa180032900 comp rx=0 tx=0).stop 2024-03-18T22:44:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa197e74740 1 -- 172.21.15.40:0/2794741732 shutdown_connections 2024-03-18T22:44:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa197e74740 1 --2- 172.21.15.40:0/2794741732 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55759871e580 0x5575987209a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa197e74740 1 --2- 172.21.15.40:0/2794741732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5575986a75b0 0x5575986b1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa197e74740 1 --2- 172.21.15.40:0/2794741732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5575986a2440 0x5575986b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa172ff5640 0 client.25945 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:44:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa197e74740 1 --2- 172.21.15.40:0/2794741732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5575986a1a50 0x5575986b0a70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa197e74740 1 -- 172.21.15.40:0/2794741732 >> 172.21.15.40:0/2794741732 conn(0x55759869cfc0 msgr2=0x5575986a0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa197e74740 1 -- 172.21.15.40:0/2794741732 shutdown_connections 2024-03-18T22:44:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa197e74740 1 -- 172.21.15.40:0/2794741732 wait complete. 2024-03-18T22:44:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.410+0000 7fa197e74740 20 client.25945 trim_cache size 0 max 16384 2024-03-18T22:44:49.420 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:44:49.420 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:44:49.420 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:44:49.421 INFO:teuthology.orchestra.run.smithi040.stdout: "9451b9e5-448b-4323-969d-83ccf36e994f": { 2024-03-18T22:44:49.421 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:44:49.421 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "9451b9e5-448b-4323-969d-83ccf36e994f", 2024-03-18T22:44:49.421 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:44:49.421 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:44:49.421 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:44:49.421 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:44:49.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.411+0000 7fa195285640 1 -- 172.21.15.40:0/887214247 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa1780a2800 msgr2=0x7fa1780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.411+0000 7fa195285640 1 --2- 172.21.15.40:0/887214247 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa1780a2800 0x7fa1780a4cc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x5575986a2ca0 tx=0x7fa1880046c0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.411+0000 7fa195285640 1 -- 172.21.15.40:0/887214247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa190100a30 msgr2=0x7fa19019a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:44:49.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.411+0000 7fa195285640 1 --2- 172.21.15.40:0/887214247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa190100a30 0x7fa19019a250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa18400e9f0 tx=0x7fa18400eeb0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.411+0000 7fa195285640 1 -- 172.21.15.40:0/887214247 shutdown_connections 2024-03-18T22:44:49.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.411+0000 7fa195285640 1 --2- 172.21.15.40:0/887214247 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa1780a2800 0x7fa1780a4cc0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.411+0000 7fa195285640 1 --2- 172.21.15.40:0/887214247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1901013d0 0x7fa1901a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.411+0000 7fa195285640 1 --2- 172.21.15.40:0/887214247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa190100a30 0x7fa19019a250 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.411+0000 7fa195285640 1 --2- 172.21.15.40:0/887214247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa1901000d0 0x7fa190199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:44:49.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.411+0000 7fa195285640 1 -- 172.21.15.40:0/887214247 >> 172.21.15.40:0/887214247 conn(0x7fa1900fbcd0 msgr2=0x7fa1900fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:44:49.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.411+0000 7fa195285640 1 -- 172.21.15.40:0/887214247 shutdown_connections 2024-03-18T22:44:49.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:44:49.411+0000 7fa195285640 1 -- 172.21.15.40:0/887214247 wait complete. 2024-03-18T22:44:49.424 INFO:tasks.cephfs.filesystem:scrub status for tag:9451b9e5-448b-4323-969d-83ccf36e994f - {'path': '/', 'tag': '9451b9e5-448b-4323-969d-83ccf36e994f', 'options': 'recursive,force'} 2024-03-18T22:44:50.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:50 smithi113 ceph-mon[66775]: pgmap v416: 129 pgs: 129 active+clean; 1.0 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:44:50.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:50 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:44:50.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:50 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:50 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:44:50.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:50 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:44:50.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:50 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:50 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:44:50.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:50 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:50 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:44:50.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:50 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:50 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:44:50.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:50 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:50 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:50 smithi103 ceph-mon[66377]: pgmap v416: 129 pgs: 129 active+clean; 1.0 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:44:50.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:50 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:44:50.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:50 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:50 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:44:50.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:50 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:44:50.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:50 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:50 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:44:50.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:50 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:50 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:44:50.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:50 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:50 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:44:50.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:50 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:50 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:50 smithi040 ceph-mon[62643]: pgmap v416: 129 pgs: 129 active+clean; 1.0 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:44:50.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:50 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:44:50.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:50 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:50 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:44:50.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:50 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:44:50.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:50 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:50 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:44:50.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:50 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:50 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:44:50.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:50 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:50 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:44:50.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:50 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:50.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:50 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:44:52.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:52 smithi113 ceph-mon[66775]: pgmap v417: 129 pgs: 129 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 29 MiB/s wr, 23 op/s 2024-03-18T22:44:52.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:52 smithi103 ceph-mon[66377]: pgmap v417: 129 pgs: 129 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 29 MiB/s wr, 23 op/s 2024-03-18T22:44:52.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:52 smithi040 ceph-mon[62643]: pgmap v417: 129 pgs: 129 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 29 MiB/s wr, 23 op/s 2024-03-18T22:44:54.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:53 smithi040 ceph-mon[62643]: pgmap v418: 129 pgs: 129 active+clean; 1.2 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:44:54.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:53 smithi113 ceph-mon[66775]: pgmap v418: 129 pgs: 129 active+clean; 1.2 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:44:54.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:53 smithi103 ceph-mon[66377]: pgmap v418: 129 pgs: 129 active+clean; 1.2 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:44:56.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:55 smithi113 ceph-mon[66775]: pgmap v419: 129 pgs: 129 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 MiB/s wr, 22 op/s 2024-03-18T22:44:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:55 smithi103 ceph-mon[66377]: pgmap v419: 129 pgs: 129 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 MiB/s wr, 22 op/s 2024-03-18T22:44:56.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:55 smithi040 ceph-mon[62643]: pgmap v419: 129 pgs: 129 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 MiB/s wr, 22 op/s 2024-03-18T22:44:58.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:57 smithi113 ceph-mon[66775]: pgmap v420: 129 pgs: 129 active+clean; 1.3 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:44:58.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:57 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:44:58.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:57 smithi103 ceph-mon[66377]: pgmap v420: 129 pgs: 129 active+clean; 1.3 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:44:58.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:57 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:44:58.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:57 smithi040 ceph-mon[62643]: pgmap v420: 129 pgs: 129 active+clean; 1.3 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:44:58.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:57 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:44:59.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:58 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:44:59.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:58 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:44:59.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:58 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:45:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:44:59 smithi113 ceph-mon[66775]: pgmap v421: 129 pgs: 129 active+clean; 1.3 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:45:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:44:59 smithi103 ceph-mon[66377]: pgmap v421: 129 pgs: 129 active+clean; 1.3 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:45:00.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:44:59 smithi040 ceph-mon[62643]: pgmap v421: 129 pgs: 129 active+clean; 1.3 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:45:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:01 smithi113 ceph-mon[66775]: pgmap v422: 129 pgs: 129 active+clean; 1.4 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:45:02.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:01 smithi103 ceph-mon[66377]: pgmap v422: 129 pgs: 129 active+clean; 1.4 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:45:02.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:01 smithi040 ceph-mon[62643]: pgmap v422: 129 pgs: 129 active+clean; 1.4 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:45:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:03 smithi113 ceph-mon[66775]: pgmap v423: 129 pgs: 129 active+clean; 1.4 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:45:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:03 smithi103 ceph-mon[66377]: pgmap v423: 129 pgs: 129 active+clean; 1.4 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:45:04.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:03 smithi040 ceph-mon[62643]: pgmap v423: 129 pgs: 129 active+clean; 1.4 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:45:06.195 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:05 smithi040 ceph-mon[62643]: pgmap v424: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:45:06.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:05 smithi113 ceph-mon[66775]: pgmap v424: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:45:06.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:05 smithi103 ceph-mon[66377]: pgmap v424: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:45:08.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:07 smithi113 ceph-mon[66775]: pgmap v425: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 31 MiB/s wr, 21 op/s 2024-03-18T22:45:08.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:07 smithi103 ceph-mon[66377]: pgmap v425: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 31 MiB/s wr, 21 op/s 2024-03-18T22:45:08.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:07 smithi040 ceph-mon[62643]: pgmap v425: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 31 MiB/s wr, 21 op/s 2024-03-18T22:45:10.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:09 smithi113 ceph-mon[66775]: pgmap v426: 129 pgs: 129 active+clean; 1.6 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:45:10.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:09 smithi103 ceph-mon[66377]: pgmap v426: 129 pgs: 129 active+clean; 1.6 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:45:10.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:09 smithi040 ceph-mon[62643]: pgmap v426: 129 pgs: 129 active+clean; 1.6 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:45:12.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:11 smithi113 ceph-mon[66775]: pgmap v427: 129 pgs: 129 active+clean; 1.6 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 29 MiB/s wr, 22 op/s 2024-03-18T22:45:12.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:11 smithi103 ceph-mon[66377]: pgmap v427: 129 pgs: 129 active+clean; 1.6 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 29 MiB/s wr, 22 op/s 2024-03-18T22:45:12.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:11 smithi040 ceph-mon[62643]: pgmap v427: 129 pgs: 129 active+clean; 1.6 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 29 MiB/s wr, 22 op/s 2024-03-18T22:45:14.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:13 smithi113 ceph-mon[66775]: pgmap v428: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:45:14.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:13 smithi103 ceph-mon[66377]: pgmap v428: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:45:14.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:13 smithi040 ceph-mon[62643]: pgmap v428: 129 pgs: 129 active+clean; 1.7 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:45:16.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:15 smithi113 ceph-mon[66775]: pgmap v429: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:45:16.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:15 smithi103 ceph-mon[66377]: pgmap v429: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:45:16.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:15 smithi040 ceph-mon[62643]: pgmap v429: 129 pgs: 129 active+clean; 1.7 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:45:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:17 smithi113 ceph-mon[66775]: pgmap v430: 129 pgs: 129 active+clean; 1.8 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:45:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:17 smithi103 ceph-mon[66377]: pgmap v430: 129 pgs: 129 active+clean; 1.8 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:45:18.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:17 smithi040 ceph-mon[62643]: pgmap v430: 129 pgs: 129 active+clean; 1.8 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:45:19.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:18 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:45:19.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:18 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:45:19.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:18 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:45:19.425 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:45:19.425 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:45:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.543+0000 7f60a8990640 1 -- 172.21.15.40:0/3234234316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60a4102310 msgr2=0x7f60a4108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.543+0000 7f60a8990640 1 --2- 172.21.15.40:0/3234234316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60a4102310 0x7f60a4108ba0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f609c009ca0 tx=0x7f609c02c0f0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.543+0000 7f60a8990640 1 -- 172.21.15.40:0/3234234316 shutdown_connections 2024-03-18T22:45:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.543+0000 7f60a8990640 1 --2- 172.21.15.40:0/3234234316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60a4102310 0x7f60a4108ba0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.543+0000 7f60a8990640 1 --2- 172.21.15.40:0/3234234316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60a4101950 0x7f60a4101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.543+0000 7f60a8990640 1 --2- 172.21.15.40:0/3234234316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60a4100760 0x7f60a4100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.543+0000 7f60a8990640 1 -- 172.21.15.40:0/3234234316 >> 172.21.15.40:0/3234234316 conn(0x7f60a40fbcd0 msgr2=0x7f60a40fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.543+0000 7f60a8990640 1 -- 172.21.15.40:0/3234234316 shutdown_connections 2024-03-18T22:45:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.543+0000 7f60a8990640 1 -- 172.21.15.40:0/3234234316 wait complete. 2024-03-18T22:45:19.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.544+0000 7f60a8990640 1 Processor -- start 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.544+0000 7f60a8990640 1 -- start start 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.544+0000 7f60a8990640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60a4100760 0x7f60a4197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.544+0000 7f60a8990640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60a4101950 0x7f60a4198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.544+0000 7f60a8990640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60a4102310 0x7f60a419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.544+0000 7f60a8990640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f60a410c290 con 0x7f60a4101950 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.544+0000 7f60a8990640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60a410c110 con 0x7f60a4102310 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.544+0000 7f60a8990640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f60a410c410 con 0x7f60a4100760 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.544+0000 7f60a1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60a4101950 0x7f60a4198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.544+0000 7f60a1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60a4101950 0x7f60a4198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58194/0 (socket says 172.21.15.40:58194) 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.544+0000 7f60a1d9b640 1 -- 172.21.15.40:0/276159840 learned_addr learned my addr 172.21.15.40:0/276159840 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.544+0000 7f60a259c640 1 --2- 172.21.15.40:0/276159840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60a4100760 0x7f60a4197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.544+0000 7f60a2d9d640 1 --2- 172.21.15.40:0/276159840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60a4102310 0x7f60a419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.545+0000 7f60a1d9b640 1 -- 172.21.15.40:0/276159840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60a4100760 msgr2=0x7f60a4197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.545+0000 7f60a1d9b640 1 --2- 172.21.15.40:0/276159840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60a4100760 0x7f60a4197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.545+0000 7f60a1d9b640 1 -- 172.21.15.40:0/276159840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60a4102310 msgr2=0x7f60a419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:19.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.545+0000 7f60a1d9b640 1 --2- 172.21.15.40:0/276159840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60a4102310 0x7f60a419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.545+0000 7f60a1d9b640 1 -- 172.21.15.40:0/276159840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60a419f690 con 0x7f60a4101950 2024-03-18T22:45:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.545+0000 7f60a1d9b640 1 --2- 172.21.15.40:0/276159840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60a4101950 0x7f60a4198000 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f609800ea60 tx=0x7f609800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.545+0000 7f60937fe640 1 -- 172.21.15.40:0/276159840 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609800cde0 con 0x7f60a4101950 2024-03-18T22:45:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.545+0000 7f60a8990640 1 -- 172.21.15.40:0/276159840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60a419f980 con 0x7f60a4101950 2024-03-18T22:45:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.545+0000 7f60937fe640 1 -- 172.21.15.40:0/276159840 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60980035f0 con 0x7f60a4101950 2024-03-18T22:45:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.545+0000 7f60937fe640 1 -- 172.21.15.40:0/276159840 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6098010690 con 0x7f60a4101950 2024-03-18T22:45:19.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.545+0000 7f60a8990640 1 -- 172.21.15.40:0/276159840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60a419fed0 con 0x7f60a4101950 2024-03-18T22:45:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.547+0000 7f60937fe640 1 -- 172.21.15.40:0/276159840 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f60980037a0 con 0x7f60a4101950 2024-03-18T22:45:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.548+0000 7f60ab57e740 1 Processor -- start 2024-03-18T22:45:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.548+0000 7f60ab57e740 1 -- start start 2024-03-18T22:45:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.548+0000 7f60ab57e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 0x55f132d10e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.548+0000 7f60ab57e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d11440 0x55f132d15f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.548+0000 7f60ab57e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d165b0 0x55f132d18a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.548+0000 7f60937fe640 1 --2- 172.21.15.40:0/276159840 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f608c0a2800 0x7f608c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.548+0000 7f60ab57e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f132c75500 con 0x55f132d165b0 2024-03-18T22:45:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.548+0000 7f60ab57e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f132c1dbe0 con 0x55f132d11440 2024-03-18T22:45:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.548+0000 7f60ab57e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f132c715a0 con 0x55f132d10a50 2024-03-18T22:45:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.548+0000 7f60a1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d165b0 0x55f132d18a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.548+0000 7f60a259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d11440 0x55f132d15f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.548+0000 7f60a259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d11440 0x55f132d15f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:34092/0 (socket says 172.21.15.40:34092) 2024-03-18T22:45:19.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.548+0000 7f60a1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d165b0 0x55f132d18a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58202/0 (socket says 172.21.15.40:58202) 2024-03-18T22:45:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.548+0000 7f60a2d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 0x55f132d10e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.548+0000 7f60a259c640 1 -- 172.21.15.40:0/3578669283 learned_addr learned my addr 172.21.15.40:0/3578669283 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.549+0000 7f60a2d9d640 1 --2- 172.21.15.40:0/276159840 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f608c0a2800 0x7f608c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.549+0000 7f60a259c640 1 -- 172.21.15.40:0/3578669283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 msgr2=0x55f132d10e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.549+0000 7f60a259c640 1 --2- 172.21.15.40:0/3578669283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 0x55f132d10e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.549+0000 7f60937fe640 1 -- 172.21.15.40:0/276159840 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f60980cf080 con 0x7f60a4101950 2024-03-18T22:45:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.549+0000 7f60a259c640 1 -- 172.21.15.40:0/3578669283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d165b0 msgr2=0x55f132d18a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.549+0000 7f60a259c640 1 --2- 172.21.15.40:0/3578669283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d165b0 0x55f132d18a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.549+0000 7f60a259c640 1 -- 172.21.15.40:0/3578669283 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f132c3f000 con 0x55f132d11440 2024-03-18T22:45:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.549+0000 7f60a2d9d640 1 --2- 172.21.15.40:0/276159840 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f608c0a2800 0x7f608c0a4cc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f609c006010 tx=0x7f609c0324f0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.549+0000 7f60a259c640 1 --2- 172.21.15.40:0/3578669283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d11440 0x55f132d15f40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6094009ca0 tx=0x7f6094011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f359e3c9d7900225 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.549+0000 7f6090ff9640 1 -- 172.21.15.40:0/3578669283 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6094004220 con 0x55f132d11440 2024-03-18T22:45:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.549+0000 7f6090ff9640 1 -- 172.21.15.40:0/3578669283 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60940043c0 con 0x55f132d11440 2024-03-18T22:45:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.549+0000 7f6090ff9640 1 -- 172.21.15.40:0/3578669283 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60940197a0 con 0x55f132d11440 2024-03-18T22:45:19.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.550+0000 7f60ab57e740 1 -- 172.21.15.40:0/3578669283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d11440 msgr2=0x55f132d15f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.550+0000 7f60ab57e740 1 --2- 172.21.15.40:0/3578669283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d11440 0x55f132d15f40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6094009ca0 tx=0x7f6094011810 comp rx=0 tx=0).stop 2024-03-18T22:45:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.550+0000 7f60ab57e740 1 -- 172.21.15.40:0/3578669283 shutdown_connections 2024-03-18T22:45:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.550+0000 7f60ab57e740 1 --2- 172.21.15.40:0/3578669283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d165b0 0x55f132d18a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.550+0000 7f60ab57e740 1 --2- 172.21.15.40:0/3578669283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d11440 0x55f132d15f40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.550+0000 7f60ab57e740 1 --2- 172.21.15.40:0/3578669283 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 0x55f132d10e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.550+0000 7f60ab57e740 1 -- 172.21.15.40:0/3578669283 >> 172.21.15.40:0/3578669283 conn(0x55f132d0bfc0 msgr2=0x55f132d0c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:19.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.550+0000 7f60ab57e740 1 -- 172.21.15.40:0/3578669283 shutdown_connections 2024-03-18T22:45:19.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.550+0000 7f60ab57e740 1 -- 172.21.15.40:0/3578669283 wait complete. 2024-03-18T22:45:19.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.550+0000 7f60ab57e740 1 -- start start 2024-03-18T22:45:19.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.550+0000 7f60ab57e740 1 Processor -- start 2024-03-18T22:45:19.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60ab57e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 0x55f132d1aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60ab57e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d11440 0x55f132d1b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60ab57e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 0x55f132d22440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60ab57e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f132d19560 con 0x55f132d11440 2024-03-18T22:45:19.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60ab57e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f132c31160 con 0x55f132d165b0 2024-03-18T22:45:19.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60ab57e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f132d196e0 con 0x55f132d10a50 2024-03-18T22:45:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60a1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d11440 0x55f132d1b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60a259c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 0x55f132d1aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60a1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d11440 0x55f132d1b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58214/0 (socket says 172.21.15.40:58214) 2024-03-18T22:45:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60a1d9b640 1 -- 172.21.15.40:0/2837784305 learned_addr learned my addr 172.21.15.40:0/2837784305 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60a2d9d640 1 --2- 172.21.15.40:0/2837784305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 0x55f132d22440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60ab57e740 10 client.0 _ll_register_callbacks cb 0x55f132c6da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60a1d9b640 1 -- 172.21.15.40:0/2837784305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 msgr2=0x55f132d1aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60a1d9b640 1 --2- 172.21.15.40:0/2837784305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 0x55f132d1aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60a1d9b640 1 -- 172.21.15.40:0/2837784305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 msgr2=0x55f132d22440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60a1d9b640 1 --2- 172.21.15.40:0/2837784305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 0x55f132d22440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60a1d9b640 1 -- 172.21.15.40:0/2837784305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f132a537d0 con 0x55f132d11440 2024-03-18T22:45:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f60a1d9b640 1 --2- 172.21.15.40:0/2837784305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d11440 0x55f132d1b400 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f60980cc790 tx=0x7f6098009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4ba4044cce75c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.551+0000 7f6089ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:19.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.552+0000 7f6089ffb640 1 -- 172.21.15.40:0/2837784305 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6098003b30 con 0x55f132d11440 2024-03-18T22:45:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.552+0000 7f6089ffb640 1 -- 172.21.15.40:0/2837784305 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6098097ca0 con 0x55f132d11440 2024-03-18T22:45:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.552+0000 7f60ab57e740 1 -- 172.21.15.40:0/2837784305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60940043c0 con 0x55f132d11440 2024-03-18T22:45:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.552+0000 7f6089ffb640 1 -- 172.21.15.40:0/2837784305 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f609809bd30 con 0x55f132d11440 2024-03-18T22:45:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.552+0000 7f60ab57e740 10 client.16116 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.552+0000 7f60ab57e740 1 -- 172.21.15.40:0/2837784305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f1329893f0 con 0x55f132d11440 2024-03-18T22:45:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.552+0000 7f6089ffb640 1 -- 172.21.15.40:0/2837784305 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60940043c0 con 0x55f132d11440 2024-03-18T22:45:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.552+0000 7f6089ffb640 1 -- 172.21.15.40:0/2837784305 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f6098084bc0 con 0x55f132d11440 2024-03-18T22:45:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.552+0000 7f60ab57e740 10 client.16116 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.552+0000 7f60ab57e740 10 client.16116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.552+0000 7f60ab57e740 20 client.16116 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.552+0000 7f60ab57e740 1 --2- 172.21.15.40:0/2837784305 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f132d26cb0 0x55f132d29110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.552+0000 7f60ab57e740 4 client.16116 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.552+0000 7f60ab57e740 1 -- 172.21.15.40:0/2837784305 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f132c82a40 con 0x55f132d26cb0 2024-03-18T22:45:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.553+0000 7f60a259c640 1 --2- 172.21.15.40:0/2837784305 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f132d26cb0 0x55f132d29110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.553+0000 7f60a259c640 1 --2- 172.21.15.40:0/2837784305 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f132d26cb0 0x55f132d29110 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f03aa40972f3dba server_cookie=36f678d203131e61 in_seq=0 out_seq=0 2024-03-18T22:45:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.553+0000 7f6089ffb640 10 client.16116 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.554+0000 7f6089ffb640 1 -- 172.21.15.40:0/2837784305 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f132c82a40 con 0x55f132d26cb0 2024-03-18T22:45:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.554+0000 7f6089ffb640 10 client.16116 handle_command_reply: tid=0 2024-03-18T22:45:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.554+0000 7f60ab57e740 1 client.16116 shutdown 2024-03-18T22:45:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.554+0000 7f60ab57e740 1 -- 172.21.15.40:0/2837784305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d11440 msgr2=0x55f132d1b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.554+0000 7f60ab57e740 1 --2- 172.21.15.40:0/2837784305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d11440 0x55f132d1b400 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f60980cc790 tx=0x7f6098009490 comp rx=0 tx=0).stop 2024-03-18T22:45:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.554+0000 7f60ab57e740 1 -- 172.21.15.40:0/2837784305 shutdown_connections 2024-03-18T22:45:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.554+0000 7f60ab57e740 1 --2- 172.21.15.40:0/2837784305 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f132d26cb0 0x55f132d29110 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.554+0000 7f60ab57e740 1 --2- 172.21.15.40:0/2837784305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 0x55f132d22440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.554+0000 7f60ab57e740 1 --2- 172.21.15.40:0/2837784305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d11440 0x55f132d1b400 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.554+0000 7f6089ffb640 0 client.16116 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.554+0000 7f60ab57e740 1 --2- 172.21.15.40:0/2837784305 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 0x55f132d1aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.554+0000 7f60ab57e740 1 -- 172.21.15.40:0/2837784305 >> 172.21.15.40:0/2837784305 conn(0x55f132d0bfc0 msgr2=0x55f132d0b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:19.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.554+0000 7f60ab57e740 1 -- 172.21.15.40:0/2837784305 shutdown_connections 2024-03-18T22:45:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.554+0000 7f60ab57e740 1 -- 172.21.15.40:0/2837784305 wait complete. 2024-03-18T22:45:19.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.554+0000 7f60ab57e740 20 client.16116 trim_cache size 0 max 16384 2024-03-18T22:45:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.575+0000 7f60ab57e740 1 Processor -- start 2024-03-18T22:45:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.575+0000 7f60ab57e740 1 -- start start 2024-03-18T22:45:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.575+0000 7f60ab57e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d10a50 0x55f132d19a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.575+0000 7f60ab57e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d11440 0x55f132d22400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.575+0000 7f60ab57e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 0x55f132d0f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.575+0000 7f60ab57e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f60a41aa590 con 0x55f132d10a50 2024-03-18T22:45:19.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.575+0000 7f60ab57e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60a41ac190 con 0x55f132d165b0 2024-03-18T22:45:19.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.575+0000 7f60ab57e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f60a41aac60 con 0x55f132d11440 2024-03-18T22:45:19.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.575+0000 7f60a259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d10a50 0x55f132d19a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.576+0000 7f60a259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d10a50 0x55f132d19a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58224/0 (socket says 172.21.15.40:58224) 2024-03-18T22:45:19.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.576+0000 7f60a259c640 1 -- 172.21.15.40:0/3887581467 learned_addr learned my addr 172.21.15.40:0/3887581467 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.576+0000 7f60a2d9d640 1 --2- 172.21.15.40:0/3887581467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 0x55f132d0f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.576+0000 7f60a259c640 1 -- 172.21.15.40:0/3887581467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d11440 msgr2=0x55f132d22400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:19.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.576+0000 7f60a259c640 1 --2- 172.21.15.40:0/3887581467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d11440 0x55f132d22400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.576+0000 7f60a259c640 1 -- 172.21.15.40:0/3887581467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 msgr2=0x55f132d0f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.576+0000 7f60a259c640 1 --2- 172.21.15.40:0/3887581467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 0x55f132d0f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.576+0000 7f60a259c640 1 -- 172.21.15.40:0/3887581467 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1329490f0 con 0x55f132d10a50 2024-03-18T22:45:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.576+0000 7f60a259c640 1 --2- 172.21.15.40:0/3887581467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d10a50 0x55f132d19a80 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6094009ca0 tx=0x7f6094009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3164aa36ccc592 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.576+0000 7f608bfff640 1 -- 172.21.15.40:0/3887581467 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609402f020 con 0x55f132d10a50 2024-03-18T22:45:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.577+0000 7f608bfff640 1 -- 172.21.15.40:0/3887581467 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60940042d0 con 0x55f132d10a50 2024-03-18T22:45:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.577+0000 7f608bfff640 1 -- 172.21.15.40:0/3887581467 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609402b780 con 0x55f132d10a50 2024-03-18T22:45:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.577+0000 7f60ab57e740 1 -- 172.21.15.40:0/3887581467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d10a50 msgr2=0x55f132d19a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.577+0000 7f60ab57e740 1 --2- 172.21.15.40:0/3887581467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d10a50 0x55f132d19a80 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6094009ca0 tx=0x7f6094009680 comp rx=0 tx=0).stop 2024-03-18T22:45:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.577+0000 7f60ab57e740 1 -- 172.21.15.40:0/3887581467 shutdown_connections 2024-03-18T22:45:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.577+0000 7f60ab57e740 1 --2- 172.21.15.40:0/3887581467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 0x55f132d0f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.577+0000 7f60ab57e740 1 --2- 172.21.15.40:0/3887581467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d11440 0x55f132d22400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.577+0000 7f60ab57e740 1 --2- 172.21.15.40:0/3887581467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d10a50 0x55f132d19a80 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.577+0000 7f60ab57e740 1 -- 172.21.15.40:0/3887581467 >> 172.21.15.40:0/3887581467 conn(0x55f132d0bfc0 msgr2=0x55f132d0f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.577+0000 7f60ab57e740 1 -- 172.21.15.40:0/3887581467 shutdown_connections 2024-03-18T22:45:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.578+0000 7f60ab57e740 1 -- 172.21.15.40:0/3887581467 wait complete. 2024-03-18T22:45:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.578+0000 7f60ab57e740 1 -- start start 2024-03-18T22:45:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.578+0000 7f60ab57e740 1 Processor -- start 2024-03-18T22:45:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.578+0000 7f60ab57e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 0x55f132d20380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.578+0000 7f60ab57e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d11440 0x55f132d208c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.578+0000 7f60ab57e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 0x55f132d21880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.578+0000 7f60ab57e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f60a41aba00 con 0x55f132d11440 2024-03-18T22:45:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.578+0000 7f60ab57e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60a41ab330 con 0x55f132d165b0 2024-03-18T22:45:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.578+0000 7f60ab57e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f60a41a8570 con 0x55f132d10a50 2024-03-18T22:45:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.578+0000 7f60ab57e740 10 client.0 _ll_register_callbacks cb 0x7f6098003d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.578+0000 7f60a1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d11440 0x55f132d208c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.578+0000 7f60a1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d11440 0x55f132d208c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58226/0 (socket says 172.21.15.40:58226) 2024-03-18T22:45:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.578+0000 7f60a1d9b640 1 -- 172.21.15.40:0/4004436527 learned_addr learned my addr 172.21.15.40:0/4004436527 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f60a1d9b640 1 -- 172.21.15.40:0/4004436527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 msgr2=0x55f132d20380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f60a259c640 1 --2- 172.21.15.40:0/4004436527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 0x55f132d20380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f60a2d9d640 1 --2- 172.21.15.40:0/4004436527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 0x55f132d21880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f60a1d9b640 1 --2- 172.21.15.40:0/4004436527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 0x55f132d20380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f60a1d9b640 1 -- 172.21.15.40:0/4004436527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 msgr2=0x55f132d21880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f60a1d9b640 1 --2- 172.21.15.40:0/4004436527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 0x55f132d21880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f60a1d9b640 1 -- 172.21.15.40:0/4004436527 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f132d21e70 con 0x55f132d11440 2024-03-18T22:45:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f60a259c640 1 --2- 172.21.15.40:0/4004436527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 0x55f132d20380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:45:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f60a2d9d640 1 --2- 172.21.15.40:0/4004436527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 0x55f132d21880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:45:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f60a1d9b640 1 --2- 172.21.15.40:0/4004436527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d11440 0x55f132d208c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f609809b330 tx=0x7f6098003e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ec3741d490835c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f6089ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f6089ffb640 1 -- 172.21.15.40:0/4004436527 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60980b8060 con 0x55f132d11440 2024-03-18T22:45:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f6089ffb640 1 -- 172.21.15.40:0/4004436527 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f609809ac80 con 0x55f132d11440 2024-03-18T22:45:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f60ab57e740 1 -- 172.21.15.40:0/4004436527 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60940042d0 con 0x55f132d11440 2024-03-18T22:45:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f6089ffb640 1 -- 172.21.15.40:0/4004436527 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f609809af90 con 0x55f132d11440 2024-03-18T22:45:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f60ab57e740 10 client.16122 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f60ab57e740 1 -- 172.21.15.40:0/4004436527 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f132d8d8a0 con 0x55f132d11440 2024-03-18T22:45:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.579+0000 7f6089ffb640 1 -- 172.21.15.40:0/4004436527 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60940042d0 con 0x55f132d11440 2024-03-18T22:45:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.580+0000 7f6089ffb640 1 -- 172.21.15.40:0/4004436527 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f6098004000 con 0x55f132d11440 2024-03-18T22:45:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.580+0000 7f60ab57e740 10 client.16122 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.580+0000 7f60ab57e740 10 client.16122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.580+0000 7f60ab57e740 20 client.16122 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.580+0000 7f60ab57e740 1 --2- 172.21.15.40:0/4004436527 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f132d8e360 0x55f132d90780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.580+0000 7f60ab57e740 4 client.16122 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:45:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.580+0000 7f60ab57e740 1 -- 172.21.15.40:0/4004436527 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f132c7cd90 con 0x55f132d8e360 2024-03-18T22:45:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.580+0000 7f60a259c640 1 --2- 172.21.15.40:0/4004436527 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f132d8e360 0x55f132d90780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f6089ffb640 10 client.16122 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f60a259c640 1 --2- 172.21.15.40:0/4004436527 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f132d8e360 0x55f132d90780 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dabdbe385335a59 server_cookie=5876e80feb20a5f7 in_seq=0 out_seq=0 2024-03-18T22:45:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f6089ffb640 1 -- 172.21.15.40:0/4004436527 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f132c7cd90 con 0x55f132d8e360 2024-03-18T22:45:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f6089ffb640 10 client.16122 handle_command_reply: tid=0 2024-03-18T22:45:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f60ab57e740 1 client.16122 shutdown 2024-03-18T22:45:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f60ab57e740 1 -- 172.21.15.40:0/4004436527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d11440 msgr2=0x55f132d208c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f60ab57e740 1 --2- 172.21.15.40:0/4004436527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d11440 0x55f132d208c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f609809b330 tx=0x7f6098003e50 comp rx=0 tx=0).stop 2024-03-18T22:45:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f60ab57e740 1 -- 172.21.15.40:0/4004436527 shutdown_connections 2024-03-18T22:45:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f60ab57e740 1 --2- 172.21.15.40:0/4004436527 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f132d8e360 0x55f132d90780 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f60ab57e740 1 --2- 172.21.15.40:0/4004436527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f132d165b0 0x55f132d21880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f60ab57e740 1 --2- 172.21.15.40:0/4004436527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f132d11440 0x55f132d208c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f60ab57e740 1 --2- 172.21.15.40:0/4004436527 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f132d10a50 0x55f132d20380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f60ab57e740 1 -- 172.21.15.40:0/4004436527 >> 172.21.15.40:0/4004436527 conn(0x55f132d0bfc0 msgr2=0x55f132d0fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:19.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f6089ffb640 0 client.16122 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:19.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f60ab57e740 1 -- 172.21.15.40:0/4004436527 shutdown_connections 2024-03-18T22:45:19.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f60ab57e740 1 -- 172.21.15.40:0/4004436527 wait complete. 2024-03-18T22:45:19.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.581+0000 7f60ab57e740 20 client.16122 trim_cache size 0 max 16384 2024-03-18T22:45:19.588 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:45:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:45:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:45:19.588 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:45:19.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.582+0000 7f60a8990640 1 -- 172.21.15.40:0/276159840 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f608c0a2800 msgr2=0x7f608c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:19.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.582+0000 7f60a8990640 1 --2- 172.21.15.40:0/276159840 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f608c0a2800 0x7f608c0a4cc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f609c006010 tx=0x7f609c0324f0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.582+0000 7f60a8990640 1 -- 172.21.15.40:0/276159840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60a4101950 msgr2=0x7f60a4198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:19.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.582+0000 7f60a8990640 1 --2- 172.21.15.40:0/276159840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60a4101950 0x7f60a4198000 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f609800ea60 tx=0x7f609800ef20 comp rx=0 tx=0).stop 2024-03-18T22:45:19.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.582+0000 7f60a8990640 1 -- 172.21.15.40:0/276159840 shutdown_connections 2024-03-18T22:45:19.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.582+0000 7f60a8990640 1 --2- 172.21.15.40:0/276159840 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f608c0a2800 0x7f608c0a4cc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.582+0000 7f60a8990640 1 --2- 172.21.15.40:0/276159840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60a4102310 0x7f60a419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.582+0000 7f60a8990640 1 --2- 172.21.15.40:0/276159840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60a4101950 0x7f60a4198000 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.582+0000 7f60a8990640 1 --2- 172.21.15.40:0/276159840 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60a4100760 0x7f60a4197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.582+0000 7f60a8990640 1 -- 172.21.15.40:0/276159840 >> 172.21.15.40:0/276159840 conn(0x7f60a40fbcd0 msgr2=0x7f60a4069b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:19.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.582+0000 7f60a8990640 1 -- 172.21.15.40:0/276159840 shutdown_connections 2024-03-18T22:45:19.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.582+0000 7f60a8990640 1 -- 172.21.15.40:0/276159840 wait complete. 2024-03-18T22:45:19.595 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:45:19.595 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:45:19.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 -- 172.21.15.40:0/3983973485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac100760 msgr2=0x7f6cac100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:19.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 --2- 172.21.15.40:0/3983973485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac100760 0x7f6cac100b80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009900 tx=0x7f6ca002bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:19.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 -- 172.21.15.40:0/3983973485 shutdown_connections 2024-03-18T22:45:19.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 --2- 172.21.15.40:0/3983973485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cac102310 0x7f6cac108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 --2- 172.21.15.40:0/3983973485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cac101950 0x7f6cac101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 --2- 172.21.15.40:0/3983973485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac100760 0x7f6cac100b80 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 -- 172.21.15.40:0/3983973485 >> 172.21.15.40:0/3983973485 conn(0x7f6cac0fbcd0 msgr2=0x7f6cac0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:19.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 -- 172.21.15.40:0/3983973485 shutdown_connections 2024-03-18T22:45:19.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 -- 172.21.15.40:0/3983973485 wait complete. 2024-03-18T22:45:19.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 Processor -- start 2024-03-18T22:45:19.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 -- start start 2024-03-18T22:45:19.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cac100760 0x7f6cac197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac101950 0x7f6cac198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cac102310 0x7f6cac19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac10c260 con 0x7f6cac101950 2024-03-18T22:45:19.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac10c0e0 con 0x7f6cac102310 2024-03-18T22:45:19.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.728+0000 7f6cb2d77640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac10c3e0 con 0x7f6cac100760 2024-03-18T22:45:19.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6cabfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac101950 0x7f6cac198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6cb0b13640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cac100760 0x7f6cac197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6cabfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac101950 0x7f6cac198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58254/0 (socket says 172.21.15.40:58254) 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6cabfff640 1 -- 172.21.15.40:0/989253640 learned_addr learned my addr 172.21.15.40:0/989253640 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6cb1314640 1 --2- 172.21.15.40:0/989253640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cac102310 0x7f6cac19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6cb0b13640 1 -- 172.21.15.40:0/989253640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cac102310 msgr2=0x7f6cac19f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6cb0b13640 1 --2- 172.21.15.40:0/989253640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cac102310 0x7f6cac19f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6cb0b13640 1 -- 172.21.15.40:0/989253640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac101950 msgr2=0x7f6cac198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6cb0b13640 1 --2- 172.21.15.40:0/989253640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac101950 0x7f6cac198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6cb0b13640 1 -- 172.21.15.40:0/989253640 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cac19f710 con 0x7f6cac100760 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6cabfff640 1 --2- 172.21.15.40:0/989253640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac101950 0x7f6cac198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6cb0b13640 1 --2- 172.21.15.40:0/989253640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cac100760 0x7f6cac197af0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0036040 tx=0x7f6ca0005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6ca9ffb640 1 -- 172.21.15.40:0/989253640 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0043020 con 0x7f6cac100760 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6cb2d77640 1 -- 172.21.15.40:0/989253640 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cac19f9a0 con 0x7f6cac100760 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6ca9ffb640 1 -- 172.21.15.40:0/989253640 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ca0004350 con 0x7f6cac100760 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.729+0000 7f6ca9ffb640 1 -- 172.21.15.40:0/989253640 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0003ab0 con 0x7f6cac100760 2024-03-18T22:45:19.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.730+0000 7f6cb2d77640 1 -- 172.21.15.40:0/989253640 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cac19ff10 con 0x7f6cac100760 2024-03-18T22:45:19.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.731+0000 7f6cb2d77640 1 -- 172.21.15.40:0/989253640 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cac068a60 con 0x7f6cac100760 2024-03-18T22:45:19.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.732+0000 7f6ca9ffb640 1 -- 172.21.15.40:0/989253640 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6ca0046020 con 0x7f6cac100760 2024-03-18T22:45:19.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.732+0000 7f6ca9ffb640 1 --2- 172.21.15.40:0/989253640 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6c9c0a2800 0x7f6c9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:19.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.733+0000 7f6cabfff640 1 --2- 172.21.15.40:0/989253640 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6c9c0a2800 0x7f6c9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:19.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.733+0000 7f6ca9ffb640 1 -- 172.21.15.40:0/989253640 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6ca00f0f20 con 0x7f6cac100760 2024-03-18T22:45:19.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.733+0000 7f6cabfff640 1 --2- 172.21.15.40:0/989253640 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6c9c0a2800 0x7f6c9c0a4cc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6cac199000 tx=0x7f6c98005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:19.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:19.741+0000 7f6ca9ffb640 1 -- 172.21.15.40:0/989253640 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6ca004d020 con 0x7f6cac100760 2024-03-18T22:45:20.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.028+0000 7f6cb2d77640 1 -- 172.21.15.40:0/989253640 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6cac1a00e0 con 0x7f6cac100760 2024-03-18T22:45:20.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.028+0000 7f6ca9ffb640 1 -- 172.21.15.40:0/989253640 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11954 (secure 0 0 0) 0x7f6ca00bdbd0 con 0x7f6cac100760 2024-03-18T22:45:20.030 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:45:20.030 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:44:17.725012+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24401,24388,24473,24509,24481]},"id":1}]} 2024-03-18T22:45:20.031 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 30 2024-03-18T22:45:20.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.031+0000 7f6cb2d77640 1 -- 172.21.15.40:0/989253640 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6c9c0a2800 msgr2=0x7f6c9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.031+0000 7f6cb2d77640 1 --2- 172.21.15.40:0/989253640 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6c9c0a2800 0x7f6c9c0a4cc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6cac199000 tx=0x7f6c98005f90 comp rx=0 tx=0).stop 2024-03-18T22:45:20.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.031+0000 7f6cb2d77640 1 -- 172.21.15.40:0/989253640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cac100760 msgr2=0x7f6cac197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.031+0000 7f6cb2d77640 1 --2- 172.21.15.40:0/989253640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cac100760 0x7f6cac197af0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0036040 tx=0x7f6ca0005dc0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.031+0000 7f6cb2d77640 1 -- 172.21.15.40:0/989253640 shutdown_connections 2024-03-18T22:45:20.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.031+0000 7f6cb2d77640 1 --2- 172.21.15.40:0/989253640 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6c9c0a2800 0x7f6c9c0a4cc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.031+0000 7f6cb2d77640 1 --2- 172.21.15.40:0/989253640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cac102310 0x7f6cac19f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.031+0000 7f6cb2d77640 1 --2- 172.21.15.40:0/989253640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6cac101950 0x7f6cac198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.031+0000 7f6cb2d77640 1 --2- 172.21.15.40:0/989253640 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6cac100760 0x7f6cac197af0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.031+0000 7f6cb2d77640 1 -- 172.21.15.40:0/989253640 >> 172.21.15.40:0/989253640 conn(0x7f6cac0fbcd0 msgr2=0x7f6cac069f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.031+0000 7f6cb2d77640 1 -- 172.21.15.40:0/989253640 shutdown_connections 2024-03-18T22:45:20.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.031+0000 7f6cb2d77640 1 -- 172.21.15.40:0/989253640 wait complete. 2024-03-18T22:45:20.044 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:45:20.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.153+0000 7eff56c61640 1 -- 172.21.15.40:0/3984832261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff501022d0 msgr2=0x7eff50108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.153+0000 7eff56c61640 1 --2- 172.21.15.40:0/3984832261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff501022d0 0x7eff50108b60 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7eff48009ca0 tx=0x7eff4802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.153+0000 7eff56c61640 1 -- 172.21.15.40:0/3984832261 shutdown_connections 2024-03-18T22:45:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.153+0000 7eff56c61640 1 --2- 172.21.15.40:0/3984832261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff501022d0 0x7eff50108b60 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.153+0000 7eff56c61640 1 --2- 172.21.15.40:0/3984832261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff50101910 0x7eff50101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.153+0000 7eff56c61640 1 --2- 172.21.15.40:0/3984832261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff50100720 0x7eff50100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.153+0000 7eff56c61640 1 -- 172.21.15.40:0/3984832261 >> 172.21.15.40:0/3984832261 conn(0x7eff500fbc90 msgr2=0x7eff500fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.153+0000 7eff56c61640 1 -- 172.21.15.40:0/3984832261 shutdown_connections 2024-03-18T22:45:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.153+0000 7eff56c61640 1 -- 172.21.15.40:0/3984832261 wait complete. 2024-03-18T22:45:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff56c61640 1 Processor -- start 2024-03-18T22:45:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff56c61640 1 -- start start 2024-03-18T22:45:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff56c61640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff50100720 0x7eff501046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff56c61640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff50101910 0x7eff50102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff56c61640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff501022d0 0x7eff50103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff56c61640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7eff5010c250 con 0x7eff50100720 2024-03-18T22:45:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff56c61640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7eff5010c0d0 con 0x7eff501022d0 2024-03-18T22:45:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff56c61640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7eff5010c3d0 con 0x7eff50101910 2024-03-18T22:45:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff549fd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff50100720 0x7eff501046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff549fd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff50100720 0x7eff501046b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58266/0 (socket says 172.21.15.40:58266) 2024-03-18T22:45:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff549fd640 1 -- 172.21.15.40:0/2955759099 learned_addr learned my addr 172.21.15.40:0/2955759099 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff4ffff640 1 --2- 172.21.15.40:0/2955759099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff50101910 0x7eff50102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff551fe640 1 --2- 172.21.15.40:0/2955759099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff501022d0 0x7eff50103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff549fd640 1 -- 172.21.15.40:0/2955759099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff50101910 msgr2=0x7eff50102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff549fd640 1 --2- 172.21.15.40:0/2955759099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff50101910 0x7eff50102cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff549fd640 1 -- 172.21.15.40:0/2955759099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff501022d0 msgr2=0x7eff50103230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff549fd640 1 --2- 172.21.15.40:0/2955759099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff501022d0 0x7eff50103230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff549fd640 1 -- 172.21.15.40:0/2955759099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff50103ac0 con 0x7eff50100720 2024-03-18T22:45:20.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.154+0000 7eff549fd640 1 --2- 172.21.15.40:0/2955759099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff50100720 0x7eff501046b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7eff4400e9c0 tx=0x7eff4400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.155+0000 7eff4dffb640 1 -- 172.21.15.40:0/2955759099 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4400cd80 con 0x7eff50100720 2024-03-18T22:45:20.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.155+0000 7eff4dffb640 1 -- 172.21.15.40:0/2955759099 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff440035f0 con 0x7eff50100720 2024-03-18T22:45:20.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.155+0000 7eff56c61640 1 -- 172.21.15.40:0/2955759099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff501a1b50 con 0x7eff50100720 2024-03-18T22:45:20.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.155+0000 7eff4dffb640 1 -- 172.21.15.40:0/2955759099 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44010690 con 0x7eff50100720 2024-03-18T22:45:20.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.155+0000 7eff56c61640 1 -- 172.21.15.40:0/2955759099 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff501a2050 con 0x7eff50100720 2024-03-18T22:45:20.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.157+0000 7eff4dffb640 1 -- 172.21.15.40:0/2955759099 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7eff440037a0 con 0x7eff50100720 2024-03-18T22:45:20.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.157+0000 7eff59851740 1 Processor -- start 2024-03-18T22:45:20.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff59851740 1 -- start start 2024-03-18T22:45:20.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff59851740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4ca710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff59851740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4cace0 0x55d6ac4cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff59851740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff59851740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d6ac42e4b0 con 0x55d6ac4cace0 2024-03-18T22:45:20.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff59851740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d6ac3d6d70 con 0x55d6ac4cfe50 2024-03-18T22:45:20.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff59851740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55d6ac42a550 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff4dffb640 1 --2- 172.21.15.40:0/2955759099 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7eff400a2800 0x7eff400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff4ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff549fd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4cace0 0x55d6ac4cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff4ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4d22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:34186/0 (socket says 172.21.15.40:34186) 2024-03-18T22:45:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff4ffff640 1 -- 172.21.15.40:0/86984941 learned_addr learned my addr 172.21.15.40:0/86984941 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff551fe640 1 --2- 172.21.15.40:0/86984941 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff551fe640 1 --2- 172.21.15.40:0/2955759099 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7eff400a2800 0x7eff400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff4dffb640 1 -- 172.21.15.40:0/2955759099 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7eff440cf3a0 con 0x7eff50100720 2024-03-18T22:45:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff4ffff640 1 -- 172.21.15.40:0/86984941 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4ca2f0 msgr2=0x55d6ac4ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff4ffff640 1 --2- 172.21.15.40:0/86984941 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4ca710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff4ffff640 1 -- 172.21.15.40:0/86984941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4cace0 msgr2=0x55d6ac4cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff4ffff640 1 --2- 172.21.15.40:0/86984941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4cace0 0x55d6ac4cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff4ffff640 1 -- 172.21.15.40:0/86984941 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6ac3f9210 con 0x55d6ac4cfe50 2024-03-18T22:45:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.158+0000 7eff549fd640 1 --2- 172.21.15.40:0/86984941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4cace0 0x55d6ac4cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.159+0000 7eff4ffff640 1 --2- 172.21.15.40:0/86984941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4d22e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7eff3c009c90 tx=0x7eff3c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bf21172b2e8aba0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.159+0000 7eff551fe640 1 --2- 172.21.15.40:0/2955759099 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7eff400a2800 0x7eff400a4cc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7eff48006010 tx=0x7eff480324f0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.159+0000 7eff3affd640 1 -- 172.21.15.40:0/86984941 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3c01e070 con 0x55d6ac4cfe50 2024-03-18T22:45:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.159+0000 7eff3affd640 1 -- 172.21.15.40:0/86984941 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff3c019ae0 con 0x55d6ac4cfe50 2024-03-18T22:45:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.159+0000 7eff3affd640 1 -- 172.21.15.40:0/86984941 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3c00ea00 con 0x55d6ac4cfe50 2024-03-18T22:45:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.159+0000 7eff59851740 1 -- 172.21.15.40:0/86984941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cfe50 msgr2=0x55d6ac4d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.159+0000 7eff59851740 1 --2- 172.21.15.40:0/86984941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4d22e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7eff3c009c90 tx=0x7eff3c011810 comp rx=0 tx=0).stop 2024-03-18T22:45:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.159+0000 7eff59851740 1 -- 172.21.15.40:0/86984941 shutdown_connections 2024-03-18T22:45:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.159+0000 7eff59851740 1 --2- 172.21.15.40:0/86984941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4d22e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.159+0000 7eff59851740 1 --2- 172.21.15.40:0/86984941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4cace0 0x55d6ac4cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.159+0000 7eff59851740 1 --2- 172.21.15.40:0/86984941 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.159+0000 7eff59851740 1 -- 172.21.15.40:0/86984941 >> 172.21.15.40:0/86984941 conn(0x55d6ac4c5860 msgr2=0x55d6ac4c5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff59851740 1 -- 172.21.15.40:0/86984941 shutdown_connections 2024-03-18T22:45:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff59851740 1 -- 172.21.15.40:0/86984941 wait complete. 2024-03-18T22:45:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff59851740 1 -- start start 2024-03-18T22:45:20.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff59851740 1 Processor -- start 2024-03-18T22:45:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff59851740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff59851740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cace0 0x55d6ac4d4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff59851740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4dbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff59851740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d6ac4d2e00 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff59851740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d6ac3ea310 con 0x55d6ac4cace0 2024-03-18T22:45:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff59851740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55d6ac4d2f80 con 0x55d6ac4cfe50 2024-03-18T22:45:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff59851740 10 client.0 _ll_register_callbacks cb 0x55d6ac3de2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff549fd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff549fd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58284/0 (socket says 172.21.15.40:58284) 2024-03-18T22:45:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff549fd640 1 -- 172.21.15.40:0/1175858467 learned_addr learned my addr 172.21.15.40:0/1175858467 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff4ffff640 1 --2- 172.21.15.40:0/1175858467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cace0 0x55d6ac4d4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.160+0000 7eff551fe640 1 --2- 172.21.15.40:0/1175858467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4dbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.161+0000 7eff549fd640 1 -- 172.21.15.40:0/1175858467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4cfe50 msgr2=0x55d6ac4dbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.161+0000 7eff549fd640 1 --2- 172.21.15.40:0/1175858467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4dbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.161+0000 7eff549fd640 1 -- 172.21.15.40:0/1175858467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cace0 msgr2=0x55d6ac4d4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.161+0000 7eff549fd640 1 --2- 172.21.15.40:0/1175858467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cace0 0x55d6ac4d4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.161+0000 7eff549fd640 1 -- 172.21.15.40:0/1175858467 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6ac1423f0 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.161+0000 7eff549fd640 1 --2- 172.21.15.40:0/1175858467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d47b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7eff4400cfd0 tx=0x7eff4406d5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad161286542887 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.161+0000 7eff2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.161+0000 7eff2ffff640 1 -- 172.21.15.40:0/1175858467 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44003b10 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.161+0000 7eff2ffff640 1 -- 172.21.15.40:0/1175858467 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7eff44098000 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.161+0000 7eff59851740 1 -- 172.21.15.40:0/1175858467 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff3c019ae0 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.161+0000 7eff2ffff640 1 -- 172.21.15.40:0/1175858467 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7eff4409c050 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.161+0000 7eff59851740 10 client.16140 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.161+0000 7eff59851740 1 -- 172.21.15.40:0/1175858467 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d6ac20c7d0 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.161+0000 7eff2ffff640 1 -- 172.21.15.40:0/1175858467 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff3c019ae0 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.162+0000 7eff2ffff640 1 -- 172.21.15.40:0/1175858467 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7eff440a3cf0 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.162+0000 7eff59851740 10 client.16140 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:20.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.162+0000 7eff59851740 10 client.16140 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:45:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.162+0000 7eff59851740 20 client.16140 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.162+0000 7eff59851740 1 --2- 172.21.15.40:0/1175858467 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55d6ac4e0680 0x55d6ac4e2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.162+0000 7eff59851740 4 client.16140 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.162+0000 7eff59851740 1 -- 172.21.15.40:0/1175858467 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6ac48c3e0 con 0x55d6ac4e0680 2024-03-18T22:45:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.162+0000 7eff4ffff640 1 --2- 172.21.15.40:0/1175858467 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55d6ac4e0680 0x55d6ac4e2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.162+0000 7eff2ffff640 10 client.16140 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:45:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.162+0000 7eff4ffff640 1 --2- 172.21.15.40:0/1175858467 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55d6ac4e0680 0x55d6ac4e2b10 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=15e049d0189b1e37 server_cookie=687ee0432170cfcc in_seq=0 out_seq=0 2024-03-18T22:45:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.163+0000 7eff2ffff640 1 -- 172.21.15.40:0/1175858467 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d6ac48c3e0 con 0x55d6ac4e0680 2024-03-18T22:45:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.163+0000 7eff2ffff640 10 client.16140 handle_command_reply: tid=0 2024-03-18T22:45:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.163+0000 7eff59851740 1 client.16140 shutdown 2024-03-18T22:45:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.163+0000 7eff59851740 1 -- 172.21.15.40:0/1175858467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4ca2f0 msgr2=0x55d6ac4d47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.163+0000 7eff59851740 1 --2- 172.21.15.40:0/1175858467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d47b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7eff4400cfd0 tx=0x7eff4406d5d0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.163+0000 7eff59851740 1 -- 172.21.15.40:0/1175858467 shutdown_connections 2024-03-18T22:45:20.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.163+0000 7eff59851740 1 --2- 172.21.15.40:0/1175858467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4dbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.163+0000 7eff59851740 1 --2- 172.21.15.40:0/1175858467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cace0 0x55d6ac4d4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.163+0000 7eff59851740 1 --2- 172.21.15.40:0/1175858467 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55d6ac4e0680 0x55d6ac4e2b10 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.163+0000 7eff59851740 1 --2- 172.21.15.40:0/1175858467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d47b0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.163+0000 7eff2ffff640 0 client.16140 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:45:20.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.163+0000 7eff59851740 1 -- 172.21.15.40:0/1175858467 >> 172.21.15.40:0/1175858467 conn(0x55d6ac4c5860 msgr2=0x55d6ac4c4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.163+0000 7eff59851740 1 -- 172.21.15.40:0/1175858467 shutdown_connections 2024-03-18T22:45:20.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.163+0000 7eff59851740 1 -- 172.21.15.40:0/1175858467 wait complete. 2024-03-18T22:45:20.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.163+0000 7eff59851740 20 client.16140 trim_cache size 0 max 16384 2024-03-18T22:45:20.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.181+0000 7eff59851740 1 Processor -- start 2024-03-18T22:45:20.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.181+0000 7eff59851740 1 -- start start 2024-03-18T22:45:20.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.181+0000 7eff59851740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.181+0000 7eff59851740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4cace0 0x55d6ac4dbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.181+0000 7eff59851740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4c4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.181+0000 7eff59851740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7eff501ac790 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.181+0000 7eff59851740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7eff501ae390 con 0x55d6ac4cfe50 2024-03-18T22:45:20.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.181+0000 7eff59851740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7eff501ace60 con 0x55d6ac4cace0 2024-03-18T22:45:20.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.182+0000 7eff549fd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.182+0000 7eff549fd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58292/0 (socket says 172.21.15.40:58292) 2024-03-18T22:45:20.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.182+0000 7eff549fd640 1 -- 172.21.15.40:0/2271366666 learned_addr learned my addr 172.21.15.40:0/2271366666 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.182+0000 7eff4ffff640 1 --2- 172.21.15.40:0/2271366666 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4cace0 0x55d6ac4dbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.182+0000 7eff551fe640 1 --2- 172.21.15.40:0/2271366666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4c4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.182+0000 7eff549fd640 1 -- 172.21.15.40:0/2271366666 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4cace0 msgr2=0x55d6ac4dbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.182+0000 7eff549fd640 1 --2- 172.21.15.40:0/2271366666 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4cace0 0x55d6ac4dbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.182+0000 7eff549fd640 1 -- 172.21.15.40:0/2271366666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cfe50 msgr2=0x55d6ac4c4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.182+0000 7eff549fd640 1 --2- 172.21.15.40:0/2271366666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4c4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.182+0000 7eff549fd640 1 -- 172.21.15.40:0/2271366666 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6ac4cbcd0 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.182+0000 7eff549fd640 1 --2- 172.21.15.40:0/2271366666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d4c50 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7eff440cce20 tx=0x7eff44009740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ca55632e31bf4c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.182+0000 7eff3a7fc640 1 -- 172.21.15.40:0/2271366666 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44003b10 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.182+0000 7eff3a7fc640 1 -- 172.21.15.40:0/2271366666 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff4409faa0 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.182+0000 7eff3a7fc640 1 -- 172.21.15.40:0/2271366666 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff440a3c60 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.183+0000 7eff59851740 1 -- 172.21.15.40:0/2271366666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4ca2f0 msgr2=0x55d6ac4d4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.183+0000 7eff59851740 1 --2- 172.21.15.40:0/2271366666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d4c50 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7eff440cce20 tx=0x7eff44009740 comp rx=0 tx=0).stop 2024-03-18T22:45:20.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.183+0000 7eff59851740 1 -- 172.21.15.40:0/2271366666 shutdown_connections 2024-03-18T22:45:20.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.183+0000 7eff59851740 1 --2- 172.21.15.40:0/2271366666 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4c4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.183+0000 7eff59851740 1 --2- 172.21.15.40:0/2271366666 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4cace0 0x55d6ac4dbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.183+0000 7eff59851740 1 --2- 172.21.15.40:0/2271366666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d4c50 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.183+0000 7eff59851740 1 -- 172.21.15.40:0/2271366666 >> 172.21.15.40:0/2271366666 conn(0x55d6ac4c5860 msgr2=0x55d6ac4c5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.183+0000 7eff59851740 1 -- 172.21.15.40:0/2271366666 shutdown_connections 2024-03-18T22:45:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.183+0000 7eff59851740 1 -- 172.21.15.40:0/2271366666 wait complete. 2024-03-18T22:45:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.183+0000 7eff59851740 1 -- start start 2024-03-18T22:45:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff59851740 1 Processor -- start 2024-03-18T22:45:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff59851740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff59851740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4cace0 0x55d6ac4d9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff59851740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4dadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff59851740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7eff501adc00 con 0x55d6ac4cfe50 2024-03-18T22:45:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff59851740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7eff501ad530 con 0x55d6ac4ca2f0 2024-03-18T22:45:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff59851740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7eff501aa770 con 0x55d6ac4cace0 2024-03-18T22:45:20.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff59851740 10 client.0 _ll_register_callbacks cb 0x7eff44003e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff551fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4dadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff551fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4dadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58302/0 (socket says 172.21.15.40:58302) 2024-03-18T22:45:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff551fe640 1 -- 172.21.15.40:0/4192653639 learned_addr learned my addr 172.21.15.40:0/4192653639 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff549fd640 1 --2- 172.21.15.40:0/4192653639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff4ffff640 1 --2- 172.21.15.40:0/4192653639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4cace0 0x55d6ac4d9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff551fe640 1 -- 172.21.15.40:0/4192653639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4cace0 msgr2=0x55d6ac4d9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff551fe640 1 --2- 172.21.15.40:0/4192653639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4cace0 0x55d6ac4d9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff551fe640 1 -- 172.21.15.40:0/4192653639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4ca2f0 msgr2=0x55d6ac4d9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.184+0000 7eff551fe640 1 --2- 172.21.15.40:0/4192653639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.185+0000 7eff551fe640 1 -- 172.21.15.40:0/4192653639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6ac4db3b0 con 0x55d6ac4cfe50 2024-03-18T22:45:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.185+0000 7eff551fe640 1 --2- 172.21.15.40:0/4192653639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4dadc0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7eff48032420 tx=0x7eff4802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cfb205e82286841 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.185+0000 7eff2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.185+0000 7eff2ffff640 1 -- 172.21.15.40:0/4192653639 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff48056060 con 0x55d6ac4cfe50 2024-03-18T22:45:20.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.185+0000 7eff2ffff640 1 -- 172.21.15.40:0/4192653639 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7eff48004690 con 0x55d6ac4cfe50 2024-03-18T22:45:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.185+0000 7eff59851740 1 -- 172.21.15.40:0/4192653639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff4409faa0 con 0x55d6ac4cfe50 2024-03-18T22:45:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.185+0000 7eff2ffff640 1 -- 172.21.15.40:0/4192653639 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7eff48039040 con 0x55d6ac4cfe50 2024-03-18T22:45:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.185+0000 7eff59851740 10 client.16146 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.185+0000 7eff59851740 1 -- 172.21.15.40:0/4192653639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d6ac546f30 con 0x55d6ac4cfe50 2024-03-18T22:45:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.185+0000 7eff2ffff640 1 -- 172.21.15.40:0/4192653639 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff4409faa0 con 0x55d6ac4cfe50 2024-03-18T22:45:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.185+0000 7eff2ffff640 1 -- 172.21.15.40:0/4192653639 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7eff48051440 con 0x55d6ac4cfe50 2024-03-18T22:45:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.186+0000 7eff59851740 10 client.16146 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.186+0000 7eff59851740 10 client.16146 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:45:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.186+0000 7eff59851740 20 client.16146 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.186+0000 7eff59851740 1 --2- 172.21.15.40:0/4192653639 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55d6ac547960 0x55d6ac549d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.186+0000 7eff59851740 4 client.16146 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:45:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.186+0000 7eff59851740 1 -- 172.21.15.40:0/4192653639 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d6ac43b9f0 con 0x55d6ac547960 2024-03-18T22:45:20.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.186+0000 7eff549fd640 1 --2- 172.21.15.40:0/4192653639 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55d6ac547960 0x55d6ac549d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.186+0000 7eff2ffff640 10 client.16146 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:45:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.186+0000 7eff549fd640 1 --2- 172.21.15.40:0/4192653639 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55d6ac547960 0x55d6ac549d80 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=412cc7fd41498f95 server_cookie=5db90f4eca554092 in_seq=0 out_seq=0 2024-03-18T22:45:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.186+0000 7eff2ffff640 1 -- 172.21.15.40:0/4192653639 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d6ac43b9f0 con 0x55d6ac547960 2024-03-18T22:45:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.186+0000 7eff2ffff640 10 client.16146 handle_command_reply: tid=0 2024-03-18T22:45:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.186+0000 7eff59851740 1 client.16146 shutdown 2024-03-18T22:45:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.187+0000 7eff59851740 1 -- 172.21.15.40:0/4192653639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4cfe50 msgr2=0x55d6ac4dadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.187+0000 7eff59851740 1 --2- 172.21.15.40:0/4192653639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4dadc0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7eff48032420 tx=0x7eff4802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.187+0000 7eff59851740 1 -- 172.21.15.40:0/4192653639 shutdown_connections 2024-03-18T22:45:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.187+0000 7eff59851740 1 --2- 172.21.15.40:0/4192653639 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55d6ac547960 0x55d6ac549d80 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.187+0000 7eff2ffff640 0 client.16146 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:45:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.187+0000 7eff59851740 1 --2- 172.21.15.40:0/4192653639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d6ac4cfe50 0x55d6ac4dadc0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.187+0000 7eff59851740 1 --2- 172.21.15.40:0/4192653639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d6ac4cace0 0x55d6ac4d9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.187+0000 7eff59851740 1 --2- 172.21.15.40:0/4192653639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d6ac4ca2f0 0x55d6ac4d9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.187+0000 7eff59851740 1 -- 172.21.15.40:0/4192653639 >> 172.21.15.40:0/4192653639 conn(0x55d6ac4c5860 msgr2=0x55d6ac4d4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.187+0000 7eff59851740 1 -- 172.21.15.40:0/4192653639 shutdown_connections 2024-03-18T22:45:20.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.187+0000 7eff59851740 1 -- 172.21.15.40:0/4192653639 wait complete. 2024-03-18T22:45:20.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.187+0000 7eff59851740 20 client.16146 trim_cache size 0 max 16384 2024-03-18T22:45:20.196 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:45:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.188+0000 7eff56c61640 1 -- 172.21.15.40:0/2955759099 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7eff400a2800 msgr2=0x7eff400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.188+0000 7eff56c61640 1 --2- 172.21.15.40:0/2955759099 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7eff400a2800 0x7eff400a4cc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7eff48006010 tx=0x7eff480324f0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.188+0000 7eff56c61640 1 -- 172.21.15.40:0/2955759099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff50100720 msgr2=0x7eff501046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.188+0000 7eff56c61640 1 --2- 172.21.15.40:0/2955759099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff50100720 0x7eff501046b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7eff4400e9c0 tx=0x7eff4400ee80 comp rx=0 tx=0).stop 2024-03-18T22:45:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.188+0000 7eff56c61640 1 -- 172.21.15.40:0/2955759099 shutdown_connections 2024-03-18T22:45:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.188+0000 7eff56c61640 1 --2- 172.21.15.40:0/2955759099 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7eff400a2800 0x7eff400a4cc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.188+0000 7eff56c61640 1 --2- 172.21.15.40:0/2955759099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff501022d0 0x7eff50103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.188+0000 7eff56c61640 1 --2- 172.21.15.40:0/2955759099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7eff50101910 0x7eff50102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.188+0000 7eff56c61640 1 --2- 172.21.15.40:0/2955759099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7eff50100720 0x7eff501046b0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.188+0000 7eff56c61640 1 -- 172.21.15.40:0/2955759099 >> 172.21.15.40:0/2955759099 conn(0x7eff500fbc90 msgr2=0x7eff500fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.188+0000 7eff56c61640 1 -- 172.21.15.40:0/2955759099 shutdown_connections 2024-03-18T22:45:20.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.188+0000 7eff56c61640 1 -- 172.21.15.40:0/2955759099 wait complete. 2024-03-18T22:45:20.200 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:45:20.225 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:19 smithi040 ceph-mon[62643]: pgmap v431: 129 pgs: 129 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 26 MiB/s wr, 19 op/s 2024-03-18T22:45:20.225 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:19 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:45:20.225 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:19 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:45:20.225 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:19 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:45:20.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.308+0000 7fad7887e640 1 -- 172.21.15.40:0/2873543185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad74101950 msgr2=0x7fad74101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.308+0000 7fad7887e640 1 --2- 172.21.15.40:0/2873543185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad74101950 0x7fad74101dd0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fad68009900 tx=0x7fad6802bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:20.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.308+0000 7fad7887e640 1 -- 172.21.15.40:0/2873543185 shutdown_connections 2024-03-18T22:45:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.308+0000 7fad7887e640 1 --2- 172.21.15.40:0/2873543185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad74102310 0x7fad74108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.308+0000 7fad7887e640 1 --2- 172.21.15.40:0/2873543185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad74101950 0x7fad74101dd0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.308+0000 7fad7887e640 1 --2- 172.21.15.40:0/2873543185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad74100760 0x7fad74100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.308+0000 7fad7887e640 1 -- 172.21.15.40:0/2873543185 >> 172.21.15.40:0/2873543185 conn(0x7fad740fbcf0 msgr2=0x7fad740fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.308+0000 7fad7887e640 1 -- 172.21.15.40:0/2873543185 shutdown_connections 2024-03-18T22:45:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.308+0000 7fad7887e640 1 -- 172.21.15.40:0/2873543185 wait complete. 2024-03-18T22:45:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.309+0000 7fad7887e640 1 Processor -- start 2024-03-18T22:45:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.309+0000 7fad7887e640 1 -- start start 2024-03-18T22:45:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.309+0000 7fad7887e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad74100760 0x7fad741046f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.309+0000 7fad7887e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad74101950 0x7fad74102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.309+0000 7fad7887e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad74102310 0x7fad74103270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.309+0000 7fad7887e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fad7410c270 con 0x7fad74102310 2024-03-18T22:45:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.309+0000 7fad7887e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fad7410c0f0 con 0x7fad74100760 2024-03-18T22:45:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.309+0000 7fad7887e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fad7410c3f0 con 0x7fad74101950 2024-03-18T22:45:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad72d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad74102310 0x7fad74103270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad7259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad74100760 0x7fad741046f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad71d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad74101950 0x7fad74102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad7259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad74100760 0x7fad741046f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:34250/0 (socket says 172.21.15.40:34250) 2024-03-18T22:45:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad7259c640 1 -- 172.21.15.40:0/1785865511 learned_addr learned my addr 172.21.15.40:0/1785865511 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad7259c640 1 -- 172.21.15.40:0/1785865511 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad74101950 msgr2=0x7fad74102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad7259c640 1 --2- 172.21.15.40:0/1785865511 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad74101950 0x7fad74102d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad7259c640 1 -- 172.21.15.40:0/1785865511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad74102310 msgr2=0x7fad74103270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad7259c640 1 --2- 172.21.15.40:0/1785865511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad74102310 0x7fad74103270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad7259c640 1 -- 172.21.15.40:0/1785865511 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad74103b00 con 0x7fad74100760 2024-03-18T22:45:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad7259c640 1 --2- 172.21.15.40:0/1785865511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad74100760 0x7fad741046f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fad6400e9c0 tx=0x7fad6400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad637fe640 1 -- 172.21.15.40:0/1785865511 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6400cd80 con 0x7fad74100760 2024-03-18T22:45:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad7887e640 1 -- 172.21.15.40:0/1785865511 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad741a1b80 con 0x7fad74100760 2024-03-18T22:45:20.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad637fe640 1 -- 172.21.15.40:0/1785865511 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad640035f0 con 0x7fad74100760 2024-03-18T22:45:20.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad637fe640 1 -- 172.21.15.40:0/1785865511 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad64010690 con 0x7fad74100760 2024-03-18T22:45:20.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.310+0000 7fad7887e640 1 -- 172.21.15.40:0/1785865511 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad741a20f0 con 0x7fad74100760 2024-03-18T22:45:20.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.313+0000 7fad637fe640 1 -- 172.21.15.40:0/1785865511 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fad640037a0 con 0x7fad74100760 2024-03-18T22:45:20.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.313+0000 7fad7b463740 1 Processor -- start 2024-03-18T22:45:20.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.313+0000 7fad7b463740 1 -- start start 2024-03-18T22:45:20.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad637fe640 1 --2- 172.21.15.40:0/1785865511 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fad5c0a2800 0x7fad5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad7b463740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca58e2f0 0x55e8ca58e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad7b463740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca5937e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad7b463740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca593e50 0x55e8ca5962e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad7b463740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e8ca4f24b0 con 0x55e8ca58e2f0 2024-03-18T22:45:20.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad7b463740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e8ca49ad70 con 0x55e8ca593e50 2024-03-18T22:45:20.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad7b463740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e8ca4ee550 con 0x55e8ca58ece0 2024-03-18T22:45:20.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad637fe640 1 -- 172.21.15.40:0/1785865511 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fad640cf200 con 0x7fad74100760 2024-03-18T22:45:20.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad72d9d640 1 --2- 172.21.15.40:0/1785865511 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fad5c0a2800 0x7fad5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad72d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca593e50 0x55e8ca5962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad72d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca593e50 0x55e8ca5962e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:34254/0 (socket says 172.21.15.40:34254) 2024-03-18T22:45:20.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad72d9d640 1 -- 172.21.15.40:0/1119539060 learned_addr learned my addr 172.21.15.40:0/1119539060 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad7259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca58e2f0 0x55e8ca58e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad71d9b640 1 --2- 172.21.15.40:0/1119539060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca5937e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad72d9d640 1 -- 172.21.15.40:0/1119539060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 msgr2=0x55e8ca5937e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad72d9d640 1 --2- 172.21.15.40:0/1119539060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca5937e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad72d9d640 1 -- 172.21.15.40:0/1119539060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca58e2f0 msgr2=0x55e8ca58e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad72d9d640 1 --2- 172.21.15.40:0/1119539060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca58e2f0 0x55e8ca58e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad72d9d640 1 -- 172.21.15.40:0/1119539060 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8ca4bd210 con 0x55e8ca593e50 2024-03-18T22:45:20.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad71d9b640 1 --2- 172.21.15.40:0/1119539060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca5937e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad7259c640 1 --2- 172.21.15.40:0/1119539060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca58e2f0 0x55e8ca58e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad72d9d640 1 --2- 172.21.15.40:0/1785865511 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fad5c0a2800 0x7fad5c0a4cc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fad6c013d10 tx=0x7fad6c009ea0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.314+0000 7fad72d9d640 1 --2- 172.21.15.40:0/1119539060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca593e50 0x55e8ca5962e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fad6c00fc20 tx=0x7fad6c01d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=300c614a395db1f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.315+0000 7fad60ff9640 1 -- 172.21.15.40:0/1119539060 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6c013170 con 0x55e8ca593e50 2024-03-18T22:45:20.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.315+0000 7fad60ff9640 1 -- 172.21.15.40:0/1119539060 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad6c00ec40 con 0x55e8ca593e50 2024-03-18T22:45:20.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.315+0000 7fad60ff9640 1 -- 172.21.15.40:0/1119539060 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6c004020 con 0x55e8ca593e50 2024-03-18T22:45:20.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.315+0000 7fad7b463740 1 -- 172.21.15.40:0/1119539060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca593e50 msgr2=0x55e8ca5962e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.315+0000 7fad7b463740 1 --2- 172.21.15.40:0/1119539060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca593e50 0x55e8ca5962e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fad6c00fc20 tx=0x7fad6c01d000 comp rx=0 tx=0).stop 2024-03-18T22:45:20.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.315+0000 7fad7b463740 1 -- 172.21.15.40:0/1119539060 shutdown_connections 2024-03-18T22:45:20.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.315+0000 7fad7b463740 1 --2- 172.21.15.40:0/1119539060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca593e50 0x55e8ca5962e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.315+0000 7fad7b463740 1 --2- 172.21.15.40:0/1119539060 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca5937e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.315+0000 7fad7b463740 1 --2- 172.21.15.40:0/1119539060 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca58e2f0 0x55e8ca58e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.315+0000 7fad7b463740 1 -- 172.21.15.40:0/1119539060 >> 172.21.15.40:0/1119539060 conn(0x55e8ca589860 msgr2=0x55e8ca589c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad7b463740 1 -- 172.21.15.40:0/1119539060 shutdown_connections 2024-03-18T22:45:20.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad7b463740 1 -- 172.21.15.40:0/1119539060 wait complete. 2024-03-18T22:45:20.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad7b463740 1 -- start start 2024-03-18T22:45:20.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad7b463740 1 Processor -- start 2024-03-18T22:45:20.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad7b463740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58e2f0 0x55e8ca598790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad7b463740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca58ece0 0x55e8ca598cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad7b463740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca593e50 0x55e8ca59fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad7b463740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e8ca596d40 con 0x55e8ca58ece0 2024-03-18T22:45:20.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad7b463740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e8ca4ae310 con 0x55e8ca593e50 2024-03-18T22:45:20.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad7b463740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e8ca596ec0 con 0x55e8ca58e2f0 2024-03-18T22:45:20.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad7b463740 10 client.0 _ll_register_callbacks cb 0x55e8ca4a22d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:20.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad71d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca58ece0 0x55e8ca598cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad71d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca58ece0 0x55e8ca598cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58320/0 (socket says 172.21.15.40:58320) 2024-03-18T22:45:20.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad71d9b640 1 -- 172.21.15.40:0/3452279043 learned_addr learned my addr 172.21.15.40:0/3452279043 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad7259c640 1 --2- 172.21.15.40:0/3452279043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58e2f0 0x55e8ca598790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.316+0000 7fad72d9d640 1 --2- 172.21.15.40:0/3452279043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca593e50 0x55e8ca59fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.317+0000 7fad71d9b640 1 -- 172.21.15.40:0/3452279043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58e2f0 msgr2=0x55e8ca598790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.317+0000 7fad71d9b640 1 --2- 172.21.15.40:0/3452279043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58e2f0 0x55e8ca598790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.317+0000 7fad71d9b640 1 -- 172.21.15.40:0/3452279043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca593e50 msgr2=0x55e8ca59fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.317+0000 7fad71d9b640 1 --2- 172.21.15.40:0/3452279043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca593e50 0x55e8ca59fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.317+0000 7fad71d9b640 1 -- 172.21.15.40:0/3452279043 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8ca2063f0 con 0x55e8ca58ece0 2024-03-18T22:45:20.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.317+0000 7fad7259c640 1 --2- 172.21.15.40:0/3452279043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58e2f0 0x55e8ca598790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.317+0000 7fad71d9b640 1 --2- 172.21.15.40:0/3452279043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca58ece0 0x55e8ca598cd0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fad6802c170 tx=0x7fad68004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65e052633d2a6d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.317+0000 7fad59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:20.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.317+0000 7fad59ffb640 1 -- 172.21.15.40:0/3452279043 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad68005030 con 0x55e8ca58ece0 2024-03-18T22:45:20.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.317+0000 7fad59ffb640 1 -- 172.21.15.40:0/3452279043 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fad680051d0 con 0x55e8ca58ece0 2024-03-18T22:45:20.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.317+0000 7fad7b463740 1 -- 172.21.15.40:0/3452279043 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad6c00ec40 con 0x55e8ca58ece0 2024-03-18T22:45:20.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.317+0000 7fad59ffb640 1 -- 172.21.15.40:0/3452279043 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fad680053d0 con 0x55e8ca58ece0 2024-03-18T22:45:20.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.317+0000 7fad7b463740 10 client.16158 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:20.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.317+0000 7fad7b463740 1 -- 172.21.15.40:0/3452279043 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e8ca2d07d0 con 0x55e8ca58ece0 2024-03-18T22:45:20.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.317+0000 7fad59ffb640 1 -- 172.21.15.40:0/3452279043 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad6c00ec40 con 0x55e8ca58ece0 2024-03-18T22:45:20.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.318+0000 7fad59ffb640 1 -- 172.21.15.40:0/3452279043 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fad68003710 con 0x55e8ca58ece0 2024-03-18T22:45:20.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.318+0000 7fad7b463740 10 client.16158 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:20.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.318+0000 7fad7b463740 10 client.16158 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:45:20.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.318+0000 7fad7b463740 20 client.16158 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:20.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.318+0000 7fad7b463740 1 --2- 172.21.15.40:0/3452279043 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e8ca5a4530 0x55e8ca5a69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.318+0000 7fad7b463740 4 client.16158 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:20.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.318+0000 7fad7b463740 1 -- 172.21.15.40:0/3452279043 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8ca5503e0 con 0x55e8ca5a4530 2024-03-18T22:45:20.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.318+0000 7fad7259c640 1 --2- 172.21.15.40:0/3452279043 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e8ca5a4530 0x55e8ca5a69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.318+0000 7fad59ffb640 10 client.16158 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:45:20.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.318+0000 7fad7259c640 1 --2- 172.21.15.40:0/3452279043 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e8ca5a4530 0x55e8ca5a69d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f213033d342af113 server_cookie=ed004784a6b13f0a in_seq=0 out_seq=0 2024-03-18T22:45:20.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.319+0000 7fad59ffb640 1 -- 172.21.15.40:0/3452279043 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e8ca5503e0 con 0x55e8ca5a4530 2024-03-18T22:45:20.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.319+0000 7fad59ffb640 10 client.16158 handle_command_reply: tid=0 2024-03-18T22:45:20.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.319+0000 7fad7b463740 1 client.16158 shutdown 2024-03-18T22:45:20.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.319+0000 7fad7b463740 1 -- 172.21.15.40:0/3452279043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca58ece0 msgr2=0x55e8ca598cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.319+0000 7fad7b463740 1 --2- 172.21.15.40:0/3452279043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca58ece0 0x55e8ca598cd0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fad6802c170 tx=0x7fad68004270 comp rx=0 tx=0).stop 2024-03-18T22:45:20.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.319+0000 7fad7b463740 1 -- 172.21.15.40:0/3452279043 shutdown_connections 2024-03-18T22:45:20.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.320+0000 7fad7b463740 1 --2- 172.21.15.40:0/3452279043 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e8ca5a4530 0x55e8ca5a69d0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.320+0000 7fad59ffb640 0 client.16158 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:45:20.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.320+0000 7fad7b463740 1 --2- 172.21.15.40:0/3452279043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca593e50 0x55e8ca59fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.320+0000 7fad7b463740 1 --2- 172.21.15.40:0/3452279043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca58ece0 0x55e8ca598cd0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.320+0000 7fad7b463740 1 --2- 172.21.15.40:0/3452279043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58e2f0 0x55e8ca598790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.320+0000 7fad7b463740 1 -- 172.21.15.40:0/3452279043 >> 172.21.15.40:0/3452279043 conn(0x55e8ca589860 msgr2=0x55e8ca588ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.320+0000 7fad7b463740 1 -- 172.21.15.40:0/3452279043 shutdown_connections 2024-03-18T22:45:20.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.320+0000 7fad7b463740 1 -- 172.21.15.40:0/3452279043 wait complete. 2024-03-18T22:45:20.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.320+0000 7fad7b463740 20 client.16158 trim_cache size 0 max 16384 2024-03-18T22:45:20.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:19 smithi113 ceph-mon[66775]: pgmap v431: 129 pgs: 129 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 26 MiB/s wr, 19 op/s 2024-03-18T22:45:20.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:19 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:45:20.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:19 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:45:20.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:19 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:45:20.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.333+0000 7fad7b463740 1 Processor -- start 2024-03-18T22:45:20.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.333+0000 7fad7b463740 1 -- start start 2024-03-18T22:45:20.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.333+0000 7fad7b463740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca58e2f0 0x55e8ca588310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.333+0000 7fad7b463740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca59fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.333+0000 7fad7b463740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca593e50 0x55e8ca588850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.333+0000 7fad7b463740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fad741ac7a0 con 0x55e8ca593e50 2024-03-18T22:45:20.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.333+0000 7fad7b463740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fad741ae3a0 con 0x55e8ca58e2f0 2024-03-18T22:45:20.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.333+0000 7fad7b463740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fad741ace70 con 0x55e8ca58ece0 2024-03-18T22:45:20.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.333+0000 7fad71d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca59fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.333+0000 7fad7259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca58e2f0 0x55e8ca588310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.333+0000 7fad72d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca593e50 0x55e8ca588850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.333+0000 7fad71d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca59fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33500/0 (socket says 172.21.15.40:33500) 2024-03-18T22:45:20.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.333+0000 7fad7259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca58e2f0 0x55e8ca588310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:34268/0 (socket says 172.21.15.40:34268) 2024-03-18T22:45:20.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.333+0000 7fad71d9b640 1 -- 172.21.15.40:0/700852769 learned_addr learned my addr 172.21.15.40:0/700852769 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.333+0000 7fad72d9d640 1 -- 172.21.15.40:0/700852769 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 msgr2=0x55e8ca59fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.334+0000 7fad72d9d640 1 --2- 172.21.15.40:0/700852769 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca59fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.334+0000 7fad72d9d640 1 -- 172.21.15.40:0/700852769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca58e2f0 msgr2=0x55e8ca588310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.334+0000 7fad72d9d640 1 --2- 172.21.15.40:0/700852769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca58e2f0 0x55e8ca588310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.334+0000 7fad72d9d640 1 -- 172.21.15.40:0/700852769 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8ca58f4d0 con 0x55e8ca593e50 2024-03-18T22:45:20.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.334+0000 7fad7259c640 1 --2- 172.21.15.40:0/700852769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca58e2f0 0x55e8ca588310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.334+0000 7fad71d9b640 1 --2- 172.21.15.40:0/700852769 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca59fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.334+0000 7fad72d9d640 1 --2- 172.21.15.40:0/700852769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca593e50 0x55e8ca588850 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fad6c023040 tx=0x7fad6c009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2afd852e286fb8b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.334+0000 7fad5bfff640 1 -- 172.21.15.40:0/700852769 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6c025040 con 0x55e8ca593e50 2024-03-18T22:45:20.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.334+0000 7fad5bfff640 1 -- 172.21.15.40:0/700852769 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad6c00ec40 con 0x55e8ca593e50 2024-03-18T22:45:20.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.334+0000 7fad5bfff640 1 -- 172.21.15.40:0/700852769 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6c003d90 con 0x55e8ca593e50 2024-03-18T22:45:20.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 -- 172.21.15.40:0/700852769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca593e50 msgr2=0x55e8ca588850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 --2- 172.21.15.40:0/700852769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca593e50 0x55e8ca588850 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fad6c023040 tx=0x7fad6c009dc0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 -- 172.21.15.40:0/700852769 shutdown_connections 2024-03-18T22:45:20.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 --2- 172.21.15.40:0/700852769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca593e50 0x55e8ca588850 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 --2- 172.21.15.40:0/700852769 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca59fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 --2- 172.21.15.40:0/700852769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca58e2f0 0x55e8ca588310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 -- 172.21.15.40:0/700852769 >> 172.21.15.40:0/700852769 conn(0x55e8ca589860 msgr2=0x55e8ca598790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 -- 172.21.15.40:0/700852769 shutdown_connections 2024-03-18T22:45:20.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 -- 172.21.15.40:0/700852769 wait complete. 2024-03-18T22:45:20.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 -- start start 2024-03-18T22:45:20.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 Processor -- start 2024-03-18T22:45:20.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca58e2f0 0x55e8ca59d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca59d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca593e50 0x55e8ca59e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fad741adc10 con 0x55e8ca593e50 2024-03-18T22:45:20.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fad741ad540 con 0x55e8ca58e2f0 2024-03-18T22:45:20.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fad741aa780 con 0x55e8ca58ece0 2024-03-18T22:45:20.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.335+0000 7fad7b463740 10 client.0 _ll_register_callbacks cb 0x7fad68009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:20.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad7259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca58e2f0 0x55e8ca59d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad72d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca593e50 0x55e8ca59e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad7259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca58e2f0 0x55e8ca59d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:34284/0 (socket says 172.21.15.40:34284) 2024-03-18T22:45:20.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad72d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca593e50 0x55e8ca59e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58328/0 (socket says 172.21.15.40:58328) 2024-03-18T22:45:20.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad7259c640 1 -- 172.21.15.40:0/1236780658 learned_addr learned my addr 172.21.15.40:0/1236780658 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad71d9b640 1 --2- 172.21.15.40:0/1236780658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca59d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad7259c640 1 -- 172.21.15.40:0/1236780658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 msgr2=0x55e8ca59d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad7259c640 1 --2- 172.21.15.40:0/1236780658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca59d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad7259c640 1 -- 172.21.15.40:0/1236780658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca593e50 msgr2=0x55e8ca59e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad7259c640 1 --2- 172.21.15.40:0/1236780658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca593e50 0x55e8ca59e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad7259c640 1 -- 172.21.15.40:0/1236780658 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8ca55f880 con 0x55e8ca58e2f0 2024-03-18T22:45:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad71d9b640 1 --2- 172.21.15.40:0/1236780658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca59d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad7259c640 1 --2- 172.21.15.40:0/1236780658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca58e2f0 0x55e8ca59d340 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fad64098130 tx=0x7fad64009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39f0b8f4ee3b948d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:45:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad59ffb640 1 -- 172.21.15.40:0/1236780658 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad640a3f50 con 0x55e8ca58e2f0 2024-03-18T22:45:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad7b463740 1 -- 172.21.15.40:0/1236780658 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad6c00ec40 con 0x55e8ca58e2f0 2024-03-18T22:45:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad59ffb640 1 -- 172.21.15.40:0/1236780658 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fad640b4580 con 0x55e8ca58e2f0 2024-03-18T22:45:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad59ffb640 1 -- 172.21.15.40:0/1236780658 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fad640860b0 con 0x55e8ca58e2f0 2024-03-18T22:45:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad7b463740 10 client.26009 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.336+0000 7fad7b463740 1 -- 172.21.15.40:0/1236780658 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e8ca5a0020 con 0x55e8ca58e2f0 2024-03-18T22:45:20.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.337+0000 7fad59ffb640 1 -- 172.21.15.40:0/1236780658 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad6c00ec40 con 0x55e8ca58e2f0 2024-03-18T22:45:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.337+0000 7fad59ffb640 1 -- 172.21.15.40:0/1236780658 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fad640a3ac0 con 0x55e8ca58e2f0 2024-03-18T22:45:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.337+0000 7fad7b463740 10 client.26009 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.337+0000 7fad7b463740 10 client.26009 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:45:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.337+0000 7fad7b463740 20 client.26009 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.337+0000 7fad7b463740 1 --2- 172.21.15.40:0/1236780658 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e8ca5a4530 0x55e8ca60d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.337+0000 7fad7b463740 4 client.26009 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:45:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.337+0000 7fad7b463740 1 -- 172.21.15.40:0/1236780658 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e8ca4ff9f0 con 0x55e8ca5a4530 2024-03-18T22:45:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.337+0000 7fad71d9b640 1 --2- 172.21.15.40:0/1236780658 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e8ca5a4530 0x55e8ca60d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.337+0000 7fad71d9b640 1 --2- 172.21.15.40:0/1236780658 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e8ca5a4530 0x55e8ca60d050 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=16f4f625fcc5b73 server_cookie=e4f62df1a8af994d in_seq=0 out_seq=0 2024-03-18T22:45:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.337+0000 7fad59ffb640 10 client.26009 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:45:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad59ffb640 1 -- 172.21.15.40:0/1236780658 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e8ca4ff9f0 con 0x55e8ca5a4530 2024-03-18T22:45:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad59ffb640 10 client.26009 handle_command_reply: tid=0 2024-03-18T22:45:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad7b463740 1 client.26009 shutdown 2024-03-18T22:45:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad7b463740 1 -- 172.21.15.40:0/1236780658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca58e2f0 msgr2=0x55e8ca59d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad7b463740 1 --2- 172.21.15.40:0/1236780658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca58e2f0 0x55e8ca59d340 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fad64098130 tx=0x7fad64009780 comp rx=0 tx=0).stop 2024-03-18T22:45:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad7b463740 1 -- 172.21.15.40:0/1236780658 shutdown_connections 2024-03-18T22:45:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad7b463740 1 --2- 172.21.15.40:0/1236780658 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e8ca5a4530 0x55e8ca60d050 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad7b463740 1 --2- 172.21.15.40:0/1236780658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8ca593e50 0x55e8ca59e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad7b463740 1 --2- 172.21.15.40:0/1236780658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8ca58ece0 0x55e8ca59d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad59ffb640 0 client.26009 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:45:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad7b463740 1 --2- 172.21.15.40:0/1236780658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8ca58e2f0 0x55e8ca59d340 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad7b463740 1 -- 172.21.15.40:0/1236780658 >> 172.21.15.40:0/1236780658 conn(0x55e8ca589860 msgr2=0x55e8ca58cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad7b463740 1 -- 172.21.15.40:0/1236780658 shutdown_connections 2024-03-18T22:45:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad7b463740 1 -- 172.21.15.40:0/1236780658 wait complete. 2024-03-18T22:45:20.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad7b463740 20 client.26009 trim_cache size 0 max 16384 2024-03-18T22:45:20.346 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:45:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad7887e640 1 -- 172.21.15.40:0/1785865511 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fad5c0a2800 msgr2=0x7fad5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad7887e640 1 --2- 172.21.15.40:0/1785865511 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fad5c0a2800 0x7fad5c0a4cc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fad6c013d10 tx=0x7fad6c009ea0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.338+0000 7fad7887e640 1 -- 172.21.15.40:0/1785865511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad74100760 msgr2=0x7fad741046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.339+0000 7fad7887e640 1 --2- 172.21.15.40:0/1785865511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad74100760 0x7fad741046f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fad6400e9c0 tx=0x7fad6400ee80 comp rx=0 tx=0).stop 2024-03-18T22:45:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.339+0000 7fad7887e640 1 -- 172.21.15.40:0/1785865511 shutdown_connections 2024-03-18T22:45:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.339+0000 7fad7887e640 1 --2- 172.21.15.40:0/1785865511 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fad5c0a2800 0x7fad5c0a4cc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.339+0000 7fad7887e640 1 --2- 172.21.15.40:0/1785865511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad74102310 0x7fad74103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.339+0000 7fad7887e640 1 --2- 172.21.15.40:0/1785865511 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fad74101950 0x7fad74102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.339+0000 7fad7887e640 1 --2- 172.21.15.40:0/1785865511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad74100760 0x7fad741046f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.339+0000 7fad7887e640 1 -- 172.21.15.40:0/1785865511 >> 172.21.15.40:0/1785865511 conn(0x7fad740fbcf0 msgr2=0x7fad740fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.339+0000 7fad7887e640 1 -- 172.21.15.40:0/1785865511 shutdown_connections 2024-03-18T22:45:20.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.339+0000 7fad7887e640 1 -- 172.21.15.40:0/1785865511 wait complete. 2024-03-18T22:45:20.353 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:45:20.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:19 smithi103 ceph-mon[66377]: pgmap v431: 129 pgs: 129 active+clean; 1.9 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 26 MiB/s wr, 19 op/s 2024-03-18T22:45:20.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:19 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:45:20.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:19 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:45:20.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:19 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:45:20.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.500+0000 7fdf8703d640 1 -- 172.21.15.40:0/3661037431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf80101950 msgr2=0x7fdf80101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.500+0000 7fdf8703d640 1 --2- 172.21.15.40:0/3661037431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf80101950 0x7fdf80101dd0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009900 tx=0x7fdf6c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:20.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.500+0000 7fdf8703d640 1 -- 172.21.15.40:0/3661037431 shutdown_connections 2024-03-18T22:45:20.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.500+0000 7fdf8703d640 1 --2- 172.21.15.40:0/3661037431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf80102310 0x7fdf80108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.500+0000 7fdf8703d640 1 --2- 172.21.15.40:0/3661037431 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf80101950 0x7fdf80101dd0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.500+0000 7fdf8703d640 1 --2- 172.21.15.40:0/3661037431 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf80100760 0x7fdf80100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.500+0000 7fdf8703d640 1 -- 172.21.15.40:0/3661037431 >> 172.21.15.40:0/3661037431 conn(0x7fdf800fbcd0 msgr2=0x7fdf800fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.500+0000 7fdf8703d640 1 -- 172.21.15.40:0/3661037431 shutdown_connections 2024-03-18T22:45:20.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.500+0000 7fdf8703d640 1 -- 172.21.15.40:0/3661037431 wait complete. 2024-03-18T22:45:20.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.501+0000 7fdf8703d640 1 Processor -- start 2024-03-18T22:45:20.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.501+0000 7fdf8703d640 1 -- start start 2024-03-18T22:45:20.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.501+0000 7fdf8703d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf80100760 0x7fdf80197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.501+0000 7fdf8703d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf80101950 0x7fdf80197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.501+0000 7fdf8703d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf80102310 0x7fdf8019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.501+0000 7fdf8703d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8010c290 con 0x7fdf80100760 2024-03-18T22:45:20.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.501+0000 7fdf8703d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8010c110 con 0x7fdf80102310 2024-03-18T22:45:20.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.501+0000 7fdf8703d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8010c410 con 0x7fdf80101950 2024-03-18T22:45:20.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.501+0000 7fdf84dd9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf80100760 0x7fdf80197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.501+0000 7fdf84dd9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf80100760 0x7fdf80197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58340/0 (socket says 172.21.15.40:58340) 2024-03-18T22:45:20.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.501+0000 7fdf84dd9640 1 -- 172.21.15.40:0/2903431752 learned_addr learned my addr 172.21.15.40:0/2903431752 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.501+0000 7fdf7ffff640 1 --2- 172.21.15.40:0/2903431752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf80101950 0x7fdf80197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.501+0000 7fdf855da640 1 --2- 172.21.15.40:0/2903431752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf80102310 0x7fdf8019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.502+0000 7fdf84dd9640 1 -- 172.21.15.40:0/2903431752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf80101950 msgr2=0x7fdf80197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.502+0000 7fdf84dd9640 1 --2- 172.21.15.40:0/2903431752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf80101950 0x7fdf80197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.502+0000 7fdf84dd9640 1 -- 172.21.15.40:0/2903431752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf80102310 msgr2=0x7fdf8019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.502+0000 7fdf84dd9640 1 --2- 172.21.15.40:0/2903431752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf80102310 0x7fdf8019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.502+0000 7fdf84dd9640 1 -- 172.21.15.40:0/2903431752 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf8019f6d0 con 0x7fdf80100760 2024-03-18T22:45:20.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.502+0000 7fdf84dd9640 1 --2- 172.21.15.40:0/2903431752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf80100760 0x7fdf80197ab0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400e9c0 tx=0x7fdf7400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.502+0000 7fdf7dffb640 1 -- 172.21.15.40:0/2903431752 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7400ccc0 con 0x7fdf80100760 2024-03-18T22:45:20.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.502+0000 7fdf8703d640 1 -- 172.21.15.40:0/2903431752 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf8019f9c0 con 0x7fdf80100760 2024-03-18T22:45:20.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.502+0000 7fdf7dffb640 1 -- 172.21.15.40:0/2903431752 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf740035f0 con 0x7fdf80100760 2024-03-18T22:45:20.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.502+0000 7fdf7dffb640 1 -- 172.21.15.40:0/2903431752 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7400ce60 con 0x7fdf80100760 2024-03-18T22:45:20.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.502+0000 7fdf8703d640 1 -- 172.21.15.40:0/2903431752 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf8019fe90 con 0x7fdf80100760 2024-03-18T22:45:20.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.504+0000 7fdf7dffb640 1 -- 172.21.15.40:0/2903431752 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdf740037a0 con 0x7fdf80100760 2024-03-18T22:45:20.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.505+0000 7fdf7dffb640 1 --2- 172.21.15.40:0/2903431752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf700a2800 0x7fdf700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.505+0000 7fdf7ffff640 1 --2- 172.21.15.40:0/2903431752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf700a2800 0x7fdf700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.505+0000 7fdf89c21740 1 Processor -- start 2024-03-18T22:45:20.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.505+0000 7fdf7dffb640 1 -- 172.21.15.40:0/2903431752 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdf740cf320 con 0x7fdf80100760 2024-03-18T22:45:20.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.505+0000 7fdf89c21740 1 -- start start 2024-03-18T22:45:20.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.505+0000 7fdf89c21740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f080ffc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.505+0000 7fdf89c21740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 0x55f0810017e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.505+0000 7fdf89c21740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f0810042e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.505+0000 7fdf89c21740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f080f604b0 con 0x55f080ffc2f0 2024-03-18T22:45:20.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.505+0000 7fdf89c21740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f080f08d70 con 0x55f081001e50 2024-03-18T22:45:20.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.505+0000 7fdf89c21740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f080f5c550 con 0x55f080ffcce0 2024-03-18T22:45:20.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf84dd9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f080ffc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf7ffff640 1 --2- 172.21.15.40:0/2903431752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf700a2800 0x7fdf700a4cc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009230 tx=0x7fdf6c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf855da640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f0810042e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf7ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 0x55f0810017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf855da640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f0810042e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:34316/0 (socket says 172.21.15.40:34316) 2024-03-18T22:45:20.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf84dd9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f080ffc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58354/0 (socket says 172.21.15.40:58354) 2024-03-18T22:45:20.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf855da640 1 -- 172.21.15.40:0/3031377933 learned_addr learned my addr 172.21.15.40:0/3031377933 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf855da640 1 -- 172.21.15.40:0/3031377933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 msgr2=0x55f0810017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf855da640 1 --2- 172.21.15.40:0/3031377933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 0x55f0810017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf855da640 1 -- 172.21.15.40:0/3031377933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 msgr2=0x55f080ffc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf855da640 1 --2- 172.21.15.40:0/3031377933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f080ffc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf855da640 1 -- 172.21.15.40:0/3031377933 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f080f2b210 con 0x55f081001e50 2024-03-18T22:45:20.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf7ffff640 1 --2- 172.21.15.40:0/3031377933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 0x55f0810017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf855da640 1 --2- 172.21.15.40:0/3031377933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f0810042e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdf78011440 tx=0x7fdf78011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d522d821f0af4af1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf6affd640 1 -- 172.21.15.40:0/3031377933 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78003c30 con 0x55f081001e50 2024-03-18T22:45:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf6affd640 1 -- 172.21.15.40:0/3031377933 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf78003dd0 con 0x55f081001e50 2024-03-18T22:45:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.506+0000 7fdf6affd640 1 -- 172.21.15.40:0/3031377933 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7800fa40 con 0x55f081001e50 2024-03-18T22:45:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.507+0000 7fdf89c21740 1 -- 172.21.15.40:0/3031377933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 msgr2=0x55f0810042e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.507+0000 7fdf89c21740 1 --2- 172.21.15.40:0/3031377933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f0810042e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdf78011440 tx=0x7fdf78011900 comp rx=0 tx=0).stop 2024-03-18T22:45:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.507+0000 7fdf89c21740 1 -- 172.21.15.40:0/3031377933 shutdown_connections 2024-03-18T22:45:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.507+0000 7fdf89c21740 1 --2- 172.21.15.40:0/3031377933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f0810042e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.507+0000 7fdf89c21740 1 --2- 172.21.15.40:0/3031377933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 0x55f0810017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.507+0000 7fdf89c21740 1 --2- 172.21.15.40:0/3031377933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f080ffc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.507+0000 7fdf89c21740 1 -- 172.21.15.40:0/3031377933 >> 172.21.15.40:0/3031377933 conn(0x55f080ff7860 msgr2=0x55f080ff7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.507+0000 7fdf89c21740 1 -- 172.21.15.40:0/3031377933 shutdown_connections 2024-03-18T22:45:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.507+0000 7fdf89c21740 1 -- 172.21.15.40:0/3031377933 wait complete. 2024-03-18T22:45:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.507+0000 7fdf89c21740 1 -- start start 2024-03-18T22:45:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.507+0000 7fdf89c21740 1 Processor -- start 2024-03-18T22:45:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf89c21740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f081006830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf89c21740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 0x55f081006d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf89c21740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f08100ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf89c21740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f081004e00 con 0x55f080ffc2f0 2024-03-18T22:45:20.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf89c21740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f080f1c310 con 0x55f081001e50 2024-03-18T22:45:20.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf89c21740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f081004f80 con 0x55f080ffcce0 2024-03-18T22:45:20.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf89c21740 10 client.0 _ll_register_callbacks cb 0x55f080f102d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:20.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf84dd9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f081006830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf84dd9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f081006830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58358/0 (socket says 172.21.15.40:58358) 2024-03-18T22:45:20.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf84dd9640 1 -- 172.21.15.40:0/3730087959 learned_addr learned my addr 172.21.15.40:0/3730087959 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf7ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 0x55f081006d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf855da640 1 --2- 172.21.15.40:0/3730087959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f08100ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf84dd9640 1 -- 172.21.15.40:0/3730087959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 msgr2=0x55f081006d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf84dd9640 1 --2- 172.21.15.40:0/3730087959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 0x55f081006d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf84dd9640 1 -- 172.21.15.40:0/3730087959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 msgr2=0x55f08100ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf84dd9640 1 --2- 172.21.15.40:0/3730087959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f08100ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf84dd9640 1 -- 172.21.15.40:0/3730087959 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f080c743f0 con 0x55f080ffc2f0 2024-03-18T22:45:20.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf7ffff640 1 --2- 172.21.15.40:0/3730087959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 0x55f081006d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.508+0000 7fdf855da640 1 --2- 172.21.15.40:0/3730087959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f08100ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf84dd9640 1 --2- 172.21.15.40:0/3730087959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f081006830 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009fd0 tx=0x7fdf74003bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a679ea7e8259a5d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:20.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf5ffff640 1 -- 172.21.15.40:0/3730087959 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf74084e60 con 0x55f080ffc2f0 2024-03-18T22:45:20.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf5ffff640 1 -- 172.21.15.40:0/3730087959 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdf740a3be0 con 0x55f080ffc2f0 2024-03-18T22:45:20.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf89c21740 1 -- 172.21.15.40:0/3730087959 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf78003dd0 con 0x55f080ffc2f0 2024-03-18T22:45:20.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf5ffff640 1 -- 172.21.15.40:0/3730087959 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdf740ba170 con 0x55f080ffc2f0 2024-03-18T22:45:20.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf89c21740 10 client.16176 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:20.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf89c21740 1 -- 172.21.15.40:0/3730087959 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f080d3e7d0 con 0x55f080ffc2f0 2024-03-18T22:45:20.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf5ffff640 1 -- 172.21.15.40:0/3730087959 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf78003dd0 con 0x55f080ffc2f0 2024-03-18T22:45:20.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf5ffff640 1 -- 172.21.15.40:0/3730087959 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fdf74097620 con 0x55f080ffc2f0 2024-03-18T22:45:20.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf89c21740 10 client.16176 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:20.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf89c21740 10 client.16176 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:45:20.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf89c21740 20 client.16176 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:20.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf89c21740 1 --2- 172.21.15.40:0/3730087959 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f081012630 0x55f081014ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf89c21740 4 client.16176 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:20.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.509+0000 7fdf89c21740 1 -- 172.21.15.40:0/3730087959 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f080fbe3e0 con 0x55f081012630 2024-03-18T22:45:20.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.510+0000 7fdf7ffff640 1 --2- 172.21.15.40:0/3730087959 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f081012630 0x55f081014ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.510+0000 7fdf5ffff640 10 client.16176 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:45:20.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.510+0000 7fdf7ffff640 1 --2- 172.21.15.40:0/3730087959 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f081012630 0x55f081014ac0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a3c56fce5f4b70b9 server_cookie=a3eb988cf12cf517 in_seq=0 out_seq=0 2024-03-18T22:45:20.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.511+0000 7fdf5ffff640 1 -- 172.21.15.40:0/3730087959 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f080fbe3e0 con 0x55f081012630 2024-03-18T22:45:20.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.511+0000 7fdf5ffff640 10 client.16176 handle_command_reply: tid=0 2024-03-18T22:45:20.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.511+0000 7fdf89c21740 1 client.16176 shutdown 2024-03-18T22:45:20.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.511+0000 7fdf89c21740 1 -- 172.21.15.40:0/3730087959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 msgr2=0x55f081006830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.511+0000 7fdf89c21740 1 --2- 172.21.15.40:0/3730087959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f081006830 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009fd0 tx=0x7fdf74003bd0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.511+0000 7fdf89c21740 1 -- 172.21.15.40:0/3730087959 shutdown_connections 2024-03-18T22:45:20.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.511+0000 7fdf89c21740 1 --2- 172.21.15.40:0/3730087959 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f081012630 0x55f081014ac0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.511+0000 7fdf89c21740 1 --2- 172.21.15.40:0/3730087959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f08100ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.511+0000 7fdf89c21740 1 --2- 172.21.15.40:0/3730087959 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 0x55f081006d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.511+0000 7fdf5ffff640 0 client.16176 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:45:20.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.511+0000 7fdf89c21740 1 --2- 172.21.15.40:0/3730087959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f081006830 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.511+0000 7fdf89c21740 1 -- 172.21.15.40:0/3730087959 >> 172.21.15.40:0/3730087959 conn(0x55f080ff7860 msgr2=0x55f080ff6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.511+0000 7fdf89c21740 1 -- 172.21.15.40:0/3730087959 shutdown_connections 2024-03-18T22:45:20.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.511+0000 7fdf89c21740 1 -- 172.21.15.40:0/3730087959 wait complete. 2024-03-18T22:45:20.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.511+0000 7fdf89c21740 20 client.16176 trim_cache size 0 max 16384 2024-03-18T22:45:20.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.525+0000 7fdf89c21740 1 Processor -- start 2024-03-18T22:45:20.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.525+0000 7fdf89c21740 1 -- start start 2024-03-18T22:45:20.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.525+0000 7fdf89c21740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f080ff7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.525+0000 7fdf89c21740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 0x55f08100dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf89c21740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f081006440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf89c21740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf801aa530 con 0x55f080ffc2f0 2024-03-18T22:45:20.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf89c21740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf801ac130 con 0x55f081001e50 2024-03-18T22:45:20.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf89c21740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdf801aac00 con 0x55f080ffcce0 2024-03-18T22:45:20.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf84dd9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f080ff7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf7ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 0x55f08100dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf855da640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f081006440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf84dd9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f080ff7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58364/0 (socket says 172.21.15.40:58364) 2024-03-18T22:45:20.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf84dd9640 1 -- 172.21.15.40:0/3837062874 learned_addr learned my addr 172.21.15.40:0/3837062874 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf84dd9640 1 -- 172.21.15.40:0/3837062874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 msgr2=0x55f08100dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf84dd9640 1 --2- 172.21.15.40:0/3837062874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 0x55f08100dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf84dd9640 1 -- 172.21.15.40:0/3837062874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 msgr2=0x55f081006440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf84dd9640 1 --2- 172.21.15.40:0/3837062874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f081006440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf84dd9640 1 -- 172.21.15.40:0/3837062874 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f080ff32c0 con 0x55f080ffc2f0 2024-03-18T22:45:20.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf855da640 1 --2- 172.21.15.40:0/3837062874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f081006440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf84dd9640 1 --2- 172.21.15.40:0/3837062874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f080ff7c40 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400c6a0 tx=0x7fdf74099de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96dd5ef7e398c1fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf6a7fc640 1 -- 172.21.15.40:0/3837062874 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf74084e60 con 0x55f080ffc2f0 2024-03-18T22:45:20.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf6a7fc640 1 -- 172.21.15.40:0/3837062874 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf7409b8d0 con 0x55f080ffc2f0 2024-03-18T22:45:20.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.526+0000 7fdf6a7fc640 1 -- 172.21.15.40:0/3837062874 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf74097620 con 0x55f080ffc2f0 2024-03-18T22:45:20.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.527+0000 7fdf89c21740 1 -- 172.21.15.40:0/3837062874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 msgr2=0x55f080ff7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.527+0000 7fdf89c21740 1 --2- 172.21.15.40:0/3837062874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f080ff7c40 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400c6a0 tx=0x7fdf74099de0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.527+0000 7fdf89c21740 1 -- 172.21.15.40:0/3837062874 shutdown_connections 2024-03-18T22:45:20.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.527+0000 7fdf89c21740 1 --2- 172.21.15.40:0/3837062874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f081001e50 0x55f081006440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.527+0000 7fdf89c21740 1 --2- 172.21.15.40:0/3837062874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffcce0 0x55f08100dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.527+0000 7fdf89c21740 1 --2- 172.21.15.40:0/3837062874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f080ffc2f0 0x55f080ff7c40 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.527+0000 7fdf89c21740 1 -- 172.21.15.40:0/3837062874 >> 172.21.15.40:0/3837062874 conn(0x55f080ff7860 msgr2=0x55f080ff2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.527+0000 7fdf89c21740 1 -- 172.21.15.40:0/3837062874 shutdown_connections 2024-03-18T22:45:20.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.527+0000 7fdf89c21740 1 -- 172.21.15.40:0/3837062874 wait complete. 2024-03-18T22:45:20.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.527+0000 7fdf89c21740 1 -- start start 2024-03-18T22:45:20.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf89c21740 1 Processor -- start 2024-03-18T22:45:20.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf89c21740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffc2f0 0x55f08100ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf89c21740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f080ffcce0 0x55f08100b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf89c21740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f081001e50 0x55f08100c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf89c21740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf801ab9a0 con 0x55f081001e50 2024-03-18T22:45:20.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf89c21740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf801ab2d0 con 0x55f080ffcce0 2024-03-18T22:45:20.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf89c21740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdf801a8510 con 0x55f080ffc2f0 2024-03-18T22:45:20.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf89c21740 10 client.0 _ll_register_callbacks cb 0x7fdf74003d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:20.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf855da640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f081001e50 0x55f08100c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf84dd9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffc2f0 0x55f08100ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf7ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f080ffcce0 0x55f08100b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf855da640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f081001e50 0x55f08100c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58378/0 (socket says 172.21.15.40:58378) 2024-03-18T22:45:20.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf84dd9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffc2f0 0x55f08100ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33556/0 (socket says 172.21.15.40:33556) 2024-03-18T22:45:20.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf855da640 1 -- 172.21.15.40:0/1937315713 learned_addr learned my addr 172.21.15.40:0/1937315713 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf855da640 1 -- 172.21.15.40:0/1937315713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffc2f0 msgr2=0x55f08100ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf855da640 1 --2- 172.21.15.40:0/1937315713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffc2f0 0x55f08100ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf855da640 1 -- 172.21.15.40:0/1937315713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f080ffcce0 msgr2=0x55f08100b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf855da640 1 --2- 172.21.15.40:0/1937315713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f080ffcce0 0x55f08100b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.528+0000 7fdf855da640 1 -- 172.21.15.40:0/1937315713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f08100c920 con 0x55f081001e50 2024-03-18T22:45:20.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf855da640 1 --2- 172.21.15.40:0/1937315713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f081001e50 0x55f08100c330 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009fc0 tx=0x7fdf78024ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7e89b3130cc15f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:20.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf5ffff640 1 -- 172.21.15.40:0/1937315713 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78033060 con 0x55f081001e50 2024-03-18T22:45:20.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf89c21740 1 -- 172.21.15.40:0/1937315713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf7409b8d0 con 0x55f081001e50 2024-03-18T22:45:20.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf5ffff640 1 -- 172.21.15.40:0/1937315713 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdf7800f540 con 0x55f081001e50 2024-03-18T22:45:20.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf5ffff640 1 -- 172.21.15.40:0/1937315713 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdf7803a050 con 0x55f081001e50 2024-03-18T22:45:20.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf89c21740 10 client.16182 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:20.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf89c21740 1 -- 172.21.15.40:0/1937315713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f08100dc20 con 0x55f081001e50 2024-03-18T22:45:20.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf5ffff640 1 -- 172.21.15.40:0/1937315713 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf7409b8d0 con 0x55f081001e50 2024-03-18T22:45:20.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf5ffff640 1 -- 172.21.15.40:0/1937315713 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fdf7802d070 con 0x55f081001e50 2024-03-18T22:45:20.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf89c21740 10 client.16182 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:20.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf89c21740 10 client.16182 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:45:20.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf89c21740 20 client.16182 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:20.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf89c21740 1 --2- 172.21.15.40:0/1937315713 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f081078b80 0x55f08107af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf89c21740 4 client.16182 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:45:20.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.529+0000 7fdf89c21740 1 -- 172.21.15.40:0/1937315713 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f080f6d9f0 con 0x55f081078b80 2024-03-18T22:45:20.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf84dd9640 1 --2- 172.21.15.40:0/1937315713 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f081078b80 0x55f08107af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf84dd9640 1 --2- 172.21.15.40:0/1937315713 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f081078b80 0x55f08107af70 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fcc1e078c5eca361 server_cookie=a8b8a98044203941 in_seq=0 out_seq=0 2024-03-18T22:45:20.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf5ffff640 10 client.16182 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:45:20.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf5ffff640 1 -- 172.21.15.40:0/1937315713 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f080f6d9f0 con 0x55f081078b80 2024-03-18T22:45:20.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf5ffff640 10 client.16182 handle_command_reply: tid=0 2024-03-18T22:45:20.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf89c21740 1 client.16182 shutdown 2024-03-18T22:45:20.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf89c21740 1 -- 172.21.15.40:0/1937315713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f081001e50 msgr2=0x55f08100c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf89c21740 1 --2- 172.21.15.40:0/1937315713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f081001e50 0x55f08100c330 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009fc0 tx=0x7fdf78024ee0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf89c21740 1 -- 172.21.15.40:0/1937315713 shutdown_connections 2024-03-18T22:45:20.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf89c21740 1 --2- 172.21.15.40:0/1937315713 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55f081078b80 0x55f08107af70 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf89c21740 1 --2- 172.21.15.40:0/1937315713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f081001e50 0x55f08100c330 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf89c21740 1 --2- 172.21.15.40:0/1937315713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f080ffcce0 0x55f08100b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf89c21740 1 --2- 172.21.15.40:0/1937315713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f080ffc2f0 0x55f08100ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf5ffff640 0 client.16182 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:45:20.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf89c21740 1 -- 172.21.15.40:0/1937315713 >> 172.21.15.40:0/1937315713 conn(0x55f080ff7860 msgr2=0x55f080ff7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.530+0000 7fdf89c21740 1 -- 172.21.15.40:0/1937315713 shutdown_connections 2024-03-18T22:45:20.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.531+0000 7fdf89c21740 1 -- 172.21.15.40:0/1937315713 wait complete. 2024-03-18T22:45:20.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.531+0000 7fdf89c21740 20 client.16182 trim_cache size 0 max 16384 2024-03-18T22:45:20.539 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:45:20.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.531+0000 7fdf8703d640 1 -- 172.21.15.40:0/2903431752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf700a2800 msgr2=0x7fdf700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.531+0000 7fdf8703d640 1 --2- 172.21.15.40:0/2903431752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf700a2800 0x7fdf700a4cc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009230 tx=0x7fdf6c005fd0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.531+0000 7fdf8703d640 1 -- 172.21.15.40:0/2903431752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf80100760 msgr2=0x7fdf80197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.531+0000 7fdf8703d640 1 --2- 172.21.15.40:0/2903431752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf80100760 0x7fdf80197ab0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400e9c0 tx=0x7fdf7400edc0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.531+0000 7fdf8703d640 1 -- 172.21.15.40:0/2903431752 shutdown_connections 2024-03-18T22:45:20.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.531+0000 7fdf8703d640 1 --2- 172.21.15.40:0/2903431752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf700a2800 0x7fdf700a4cc0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.531+0000 7fdf8703d640 1 --2- 172.21.15.40:0/2903431752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf80102310 0x7fdf8019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.531+0000 7fdf8703d640 1 --2- 172.21.15.40:0/2903431752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf80101950 0x7fdf80197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.531+0000 7fdf8703d640 1 --2- 172.21.15.40:0/2903431752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf80100760 0x7fdf80197ab0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.531+0000 7fdf8703d640 1 -- 172.21.15.40:0/2903431752 >> 172.21.15.40:0/2903431752 conn(0x7fdf800fbcd0 msgr2=0x7fdf800fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.531+0000 7fdf8703d640 1 -- 172.21.15.40:0/2903431752 shutdown_connections 2024-03-18T22:45:20.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.531+0000 7fdf8703d640 1 -- 172.21.15.40:0/2903431752 wait complete. 2024-03-18T22:45:20.545 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:45:20.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.661+0000 7f5b6ccac640 1 -- 172.21.15.40:0/460559263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b68102320 msgr2=0x7f5b68104710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.661+0000 7f5b6ccac640 1 --2- 172.21.15.40:0/460559263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b68102320 0x7f5b68104710 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009900 tx=0x7f5b5802bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:20.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.661+0000 7f5b6ccac640 1 -- 172.21.15.40:0/460559263 shutdown_connections 2024-03-18T22:45:20.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.661+0000 7f5b6ccac640 1 --2- 172.21.15.40:0/460559263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b68107580 0x7f5b68109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.661+0000 7f5b6ccac640 1 --2- 172.21.15.40:0/460559263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b68104c50 0x7f5b68107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.661+0000 7f5b6ccac640 1 --2- 172.21.15.40:0/460559263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b68102320 0x7f5b68104710 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.661+0000 7f5b6ccac640 1 -- 172.21.15.40:0/460559263 >> 172.21.15.40:0/460559263 conn(0x7f5b680fbc90 msgr2=0x7f5b680fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.661+0000 7f5b6ccac640 1 -- 172.21.15.40:0/460559263 shutdown_connections 2024-03-18T22:45:20.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.661+0000 7f5b6ccac640 1 -- 172.21.15.40:0/460559263 wait complete. 2024-03-18T22:45:20.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.662+0000 7f5b6ccac640 1 Processor -- start 2024-03-18T22:45:20.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.662+0000 7f5b6ccac640 1 -- start start 2024-03-18T22:45:20.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.662+0000 7f5b6ccac640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b68102320 0x7f5b68197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.662+0000 7f5b6ccac640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b68104c50 0x7f5b68197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.662+0000 7f5b6ccac640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b68107580 0x7f5b6819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.662+0000 7f5b6ccac640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6810d340 con 0x7f5b68102320 2024-03-18T22:45:20.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.662+0000 7f5b6ccac640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6810d1c0 con 0x7f5b68107580 2024-03-18T22:45:20.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.662+0000 7f5b6ccac640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6810d4c0 con 0x7f5b68104c50 2024-03-18T22:45:20.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.662+0000 7f5b6659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b68102320 0x7f5b68197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.662+0000 7f5b6659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b68102320 0x7f5b68197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58410/0 (socket says 172.21.15.40:58410) 2024-03-18T22:45:20.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.662+0000 7f5b6659c640 1 -- 172.21.15.40:0/316883141 learned_addr learned my addr 172.21.15.40:0/316883141 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.662+0000 7f5b5ffff640 1 --2- 172.21.15.40:0/316883141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b68104c50 0x7f5b68197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.663+0000 7f5b66d9d640 1 --2- 172.21.15.40:0/316883141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b68107580 0x7f5b6819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.663+0000 7f5b6659c640 1 -- 172.21.15.40:0/316883141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b68104c50 msgr2=0x7f5b68197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.663+0000 7f5b6659c640 1 --2- 172.21.15.40:0/316883141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b68104c50 0x7f5b68197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.663+0000 7f5b6659c640 1 -- 172.21.15.40:0/316883141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b68107580 msgr2=0x7f5b6819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.663+0000 7f5b6659c640 1 --2- 172.21.15.40:0/316883141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b68107580 0x7f5b6819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.663+0000 7f5b6659c640 1 -- 172.21.15.40:0/316883141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b6819f6d0 con 0x7f5b68102320 2024-03-18T22:45:20.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.663+0000 7f5b5ffff640 1 --2- 172.21.15.40:0/316883141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b68104c50 0x7f5b68197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.663+0000 7f5b6659c640 1 --2- 172.21.15.40:0/316883141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b68102320 0x7f5b68197ab0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5b58006010 tx=0x7f5b58005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.663+0000 7f5b5f7fe640 1 -- 172.21.15.40:0/316883141 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b58043020 con 0x7f5b68102320 2024-03-18T22:45:20.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.663+0000 7f5b6ccac640 1 -- 172.21.15.40:0/316883141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b6819f960 con 0x7f5b68102320 2024-03-18T22:45:20.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.663+0000 7f5b5f7fe640 1 -- 172.21.15.40:0/316883141 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b580043d0 con 0x7f5b68102320 2024-03-18T22:45:20.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.663+0000 7f5b5f7fe640 1 -- 172.21.15.40:0/316883141 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b58003a80 con 0x7f5b68102320 2024-03-18T22:45:20.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.663+0000 7f5b6ccac640 1 -- 172.21.15.40:0/316883141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b6819feb0 con 0x7f5b68102320 2024-03-18T22:45:20.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.666+0000 7f5b5f7fe640 1 -- 172.21.15.40:0/316883141 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5b58046020 con 0x7f5b68102320 2024-03-18T22:45:20.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.666+0000 7f5b6f972740 1 Processor -- start 2024-03-18T22:45:20.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.666+0000 7f5b6f972740 1 -- start start 2024-03-18T22:45:20.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.666+0000 7f5b6f972740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a12f0 0x5624a63a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.666+0000 7f5b6f972740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a1ce0 0x5624a63a67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.666+0000 7f5b6f972740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a6e50 0x5624a63a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.666+0000 7f5b6f972740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5624a63054b0 con 0x5624a63a1ce0 2024-03-18T22:45:20.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.666+0000 7f5b6f972740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5624a62add70 con 0x5624a63a12f0 2024-03-18T22:45:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.666+0000 7f5b6f972740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5624a6301550 con 0x5624a63a6e50 2024-03-18T22:45:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b6659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a1ce0 0x5624a63a67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b66d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a12f0 0x5624a63a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b66d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a12f0 0x5624a63a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:34382/0 (socket says 172.21.15.40:34382) 2024-03-18T22:45:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b6659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a1ce0 0x5624a63a67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58412/0 (socket says 172.21.15.40:58412) 2024-03-18T22:45:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b66d9d640 1 -- 172.21.15.40:0/3580133517 learned_addr learned my addr 172.21.15.40:0/3580133517 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b5ffff640 1 --2- 172.21.15.40:0/3580133517 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a6e50 0x5624a63a92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b5f7fe640 1 --2- 172.21.15.40:0/316883141 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b540a2800 0x7f5b540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b66d9d640 1 --2- 172.21.15.40:0/316883141 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b540a2800 0x7f5b540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b5ffff640 1 -- 172.21.15.40:0/3580133517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a12f0 msgr2=0x5624a63a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b5ffff640 1 --2- 172.21.15.40:0/3580133517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a12f0 0x5624a63a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b5ffff640 1 -- 172.21.15.40:0/3580133517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a1ce0 msgr2=0x5624a63a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b5ffff640 1 --2- 172.21.15.40:0/3580133517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a1ce0 0x5624a63a67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b5ffff640 1 -- 172.21.15.40:0/3580133517 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624a62d0210 con 0x5624a63a6e50 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b5f7fe640 1 -- 172.21.15.40:0/316883141 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5b580f1020 con 0x7f5b68102320 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b5ffff640 1 --2- 172.21.15.40:0/3580133517 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a6e50 0x5624a63a92e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5b68069460 tx=0x7f5b500118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d90feef3e60ace6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b66d9d640 1 --2- 172.21.15.40:0/316883141 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b540a2800 0x7f5b540a4cc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x5624a63a2540 tx=0x7f5b60004ab0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b5cff9640 1 -- 172.21.15.40:0/3580133517 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b50004020 con 0x5624a63a6e50 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b5cff9640 1 -- 172.21.15.40:0/3580133517 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b500054d0 con 0x5624a63a6e50 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.667+0000 7f5b5cff9640 1 -- 172.21.15.40:0/3580133517 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b50004c10 con 0x5624a63a6e50 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.668+0000 7f5b6f972740 1 -- 172.21.15.40:0/3580133517 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a6e50 msgr2=0x5624a63a92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.668+0000 7f5b6f972740 1 --2- 172.21.15.40:0/3580133517 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a6e50 0x5624a63a92e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5b68069460 tx=0x7f5b500118d0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.668+0000 7f5b6f972740 1 -- 172.21.15.40:0/3580133517 shutdown_connections 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.668+0000 7f5b6f972740 1 --2- 172.21.15.40:0/3580133517 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a6e50 0x5624a63a92e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.668+0000 7f5b6f972740 1 --2- 172.21.15.40:0/3580133517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a1ce0 0x5624a63a67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.668+0000 7f5b6f972740 1 --2- 172.21.15.40:0/3580133517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a12f0 0x5624a63a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.668+0000 7f5b6f972740 1 -- 172.21.15.40:0/3580133517 >> 172.21.15.40:0/3580133517 conn(0x5624a639c860 msgr2=0x5624a639cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.668+0000 7f5b6f972740 1 -- 172.21.15.40:0/3580133517 shutdown_connections 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.668+0000 7f5b6f972740 1 -- 172.21.15.40:0/3580133517 wait complete. 2024-03-18T22:45:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6f972740 1 -- start start 2024-03-18T22:45:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6f972740 1 Processor -- start 2024-03-18T22:45:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6f972740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a12f0 0x5624a63ab7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6f972740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a1ce0 0x5624a63abcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6f972740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a6e50 0x5624a63b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6f972740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5624a63a9e00 con 0x5624a63a1ce0 2024-03-18T22:45:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6f972740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5624a62c1310 con 0x5624a63a6e50 2024-03-18T22:45:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6f972740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5624a63a9f80 con 0x5624a63a12f0 2024-03-18T22:45:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6f972740 10 client.0 _ll_register_callbacks cb 0x5624a62b52d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a12f0 0x5624a63ab7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b5ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a1ce0 0x5624a63abcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a12f0 0x5624a63ab7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33600/0 (socket says 172.21.15.40:33600) 2024-03-18T22:45:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6659c640 1 -- 172.21.15.40:0/1126005818 learned_addr learned my addr 172.21.15.40:0/1126005818 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b66d9d640 1 --2- 172.21.15.40:0/1126005818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a6e50 0x5624a63b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6659c640 1 -- 172.21.15.40:0/1126005818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a6e50 msgr2=0x5624a63b2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6659c640 1 --2- 172.21.15.40:0/1126005818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a6e50 0x5624a63b2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6659c640 1 -- 172.21.15.40:0/1126005818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a1ce0 msgr2=0x5624a63abcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6659c640 1 --2- 172.21.15.40:0/1126005818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a1ce0 0x5624a63abcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b6659c640 1 -- 172.21.15.40:0/1126005818 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624a60193f0 con 0x5624a63a12f0 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.669+0000 7f5b5ffff640 1 --2- 172.21.15.40:0/1126005818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a1ce0 0x5624a63abcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b6659c640 1 --2- 172.21.15.40:0/1126005818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a12f0 0x5624a63ab7b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5b58005ab0 tx=0x7f5b58005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd1372ca9b60e569 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b4dffb640 1 -- 172.21.15.40:0/1126005818 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b580bd110 con 0x5624a63a12f0 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b4dffb640 1 -- 172.21.15.40:0/1126005818 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5b580053b0 con 0x5624a63a12f0 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b6f972740 1 -- 172.21.15.40:0/1126005818 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b500054d0 con 0x5624a63a12f0 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b4dffb640 1 -- 172.21.15.40:0/1126005818 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5b580bcf30 con 0x5624a63a12f0 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b6f972740 10 client.26029 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b6f972740 1 -- 172.21.15.40:0/1126005818 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5624a60e37d0 con 0x5624a63a12f0 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b4dffb640 1 -- 172.21.15.40:0/1126005818 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b500054d0 con 0x5624a63a12f0 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b4dffb640 1 -- 172.21.15.40:0/1126005818 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f5b580f30e0 con 0x5624a63a12f0 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b6f972740 10 client.26029 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b6f972740 10 client.26029 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b6f972740 20 client.26029 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b6f972740 1 --2- 172.21.15.40:0/1126005818 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5624a63b75b0 0x5624a63b9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b6f972740 4 client.26029 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.670+0000 7f5b6f972740 1 -- 172.21.15.40:0/1126005818 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624a63633e0 con 0x5624a63b75b0 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.671+0000 7f5b5ffff640 1 --2- 172.21.15.40:0/1126005818 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5624a63b75b0 0x5624a63b9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.671+0000 7f5b4dffb640 10 client.26029 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.671+0000 7f5b5ffff640 1 --2- 172.21.15.40:0/1126005818 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5624a63b75b0 0x5624a63b9a40 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=429443b81d840614 server_cookie=6627d323e890ea7 in_seq=0 out_seq=0 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.672+0000 7f5b4dffb640 1 -- 172.21.15.40:0/1126005818 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5624a63633e0 con 0x5624a63b75b0 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.672+0000 7f5b4dffb640 10 client.26029 handle_command_reply: tid=0 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.672+0000 7f5b6f972740 1 client.26029 shutdown 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.672+0000 7f5b6f972740 1 -- 172.21.15.40:0/1126005818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a12f0 msgr2=0x5624a63ab7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.672+0000 7f5b6f972740 1 --2- 172.21.15.40:0/1126005818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a12f0 0x5624a63ab7b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5b58005ab0 tx=0x7f5b58005b50 comp rx=0 tx=0).stop 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.672+0000 7f5b6f972740 1 -- 172.21.15.40:0/1126005818 shutdown_connections 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.672+0000 7f5b6f972740 1 --2- 172.21.15.40:0/1126005818 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5624a63b75b0 0x5624a63b9a40 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.672+0000 7f5b6f972740 1 --2- 172.21.15.40:0/1126005818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a6e50 0x5624a63b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.672+0000 7f5b4dffb640 0 client.26029 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:45:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.672+0000 7f5b6f972740 1 --2- 172.21.15.40:0/1126005818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a1ce0 0x5624a63abcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.672+0000 7f5b6f972740 1 --2- 172.21.15.40:0/1126005818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a12f0 0x5624a63ab7b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.672+0000 7f5b6f972740 1 -- 172.21.15.40:0/1126005818 >> 172.21.15.40:0/1126005818 conn(0x5624a639c860 msgr2=0x5624a639bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.672+0000 7f5b6f972740 1 -- 172.21.15.40:0/1126005818 shutdown_connections 2024-03-18T22:45:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.672+0000 7f5b6f972740 1 -- 172.21.15.40:0/1126005818 wait complete. 2024-03-18T22:45:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.672+0000 7f5b6f972740 20 client.26029 trim_cache size 0 max 16384 2024-03-18T22:45:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.685+0000 7f5b6f972740 1 Processor -- start 2024-03-18T22:45:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.685+0000 7f5b6f972740 1 -- start start 2024-03-18T22:45:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b6f972740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a12f0 0x5624a63abc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b6f972740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a1ce0 0x5624a63b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b6f972740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a6e50 0x5624a639b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b6f972740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b681aa5a0 con 0x5624a63a12f0 2024-03-18T22:45:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b6f972740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5b681ac1a0 con 0x5624a63a6e50 2024-03-18T22:45:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b6f972740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5b681aac70 con 0x5624a63a1ce0 2024-03-18T22:45:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b6659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a12f0 0x5624a63abc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b6659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a12f0 0x5624a63abc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58438/0 (socket says 172.21.15.40:58438) 2024-03-18T22:45:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b6659c640 1 -- 172.21.15.40:0/2845195583 learned_addr learned my addr 172.21.15.40:0/2845195583 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b5ffff640 1 --2- 172.21.15.40:0/2845195583 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a1ce0 0x5624a63b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b66d9d640 1 --2- 172.21.15.40:0/2845195583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a6e50 0x5624a639b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b6659c640 1 -- 172.21.15.40:0/2845195583 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a1ce0 msgr2=0x5624a63b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b6659c640 1 --2- 172.21.15.40:0/2845195583 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a1ce0 0x5624a63b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b6659c640 1 -- 172.21.15.40:0/2845195583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a6e50 msgr2=0x5624a639b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b6659c640 1 --2- 172.21.15.40:0/2845195583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a6e50 0x5624a639b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b6659c640 1 -- 172.21.15.40:0/2845195583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624a6372880 con 0x5624a63a12f0 2024-03-18T22:45:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b6659c640 1 --2- 172.21.15.40:0/2845195583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a12f0 0x5624a63abc50 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5b580b9fa0 tx=0x7f5b58005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d440e788915a111 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b4ffff640 1 -- 172.21.15.40:0/2845195583 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b580bd110 con 0x5624a63a12f0 2024-03-18T22:45:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b4ffff640 1 -- 172.21.15.40:0/2845195583 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b580bdcd0 con 0x5624a63a12f0 2024-03-18T22:45:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.686+0000 7f5b4ffff640 1 -- 172.21.15.40:0/2845195583 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b580053b0 con 0x5624a63a12f0 2024-03-18T22:45:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.687+0000 7f5b6f972740 1 -- 172.21.15.40:0/2845195583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a12f0 msgr2=0x5624a63abc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.687+0000 7f5b6f972740 1 --2- 172.21.15.40:0/2845195583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a12f0 0x5624a63abc50 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5b580b9fa0 tx=0x7f5b58005770 comp rx=0 tx=0).stop 2024-03-18T22:45:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.687+0000 7f5b6f972740 1 -- 172.21.15.40:0/2845195583 shutdown_connections 2024-03-18T22:45:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.687+0000 7f5b6f972740 1 --2- 172.21.15.40:0/2845195583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a6e50 0x5624a639b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.687+0000 7f5b6f972740 1 --2- 172.21.15.40:0/2845195583 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a1ce0 0x5624a63b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.687+0000 7f5b6f972740 1 --2- 172.21.15.40:0/2845195583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a12f0 0x5624a63abc50 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.687+0000 7f5b6f972740 1 -- 172.21.15.40:0/2845195583 >> 172.21.15.40:0/2845195583 conn(0x5624a639c860 msgr2=0x5624a639cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.687+0000 7f5b6f972740 1 -- 172.21.15.40:0/2845195583 shutdown_connections 2024-03-18T22:45:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.687+0000 7f5b6f972740 1 -- 172.21.15.40:0/2845195583 wait complete. 2024-03-18T22:45:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6f972740 1 -- start start 2024-03-18T22:45:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6f972740 1 Processor -- start 2024-03-18T22:45:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6f972740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a12f0 0x5624a63b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6f972740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a1ce0 0x5624a63b1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6f972740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a6e50 0x5624a63b2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6f972740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5b681aba10 con 0x5624a63a12f0 2024-03-18T22:45:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6f972740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5b681ab340 con 0x5624a63a1ce0 2024-03-18T22:45:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6f972740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5b681a8580 con 0x5624a63a6e50 2024-03-18T22:45:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6f972740 10 client.0 _ll_register_callbacks cb 0x7f5b580b7fb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a12f0 0x5624a63b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a12f0 0x5624a63b0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58440/0 (socket says 172.21.15.40:58440) 2024-03-18T22:45:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6659c640 1 -- 172.21.15.40:0/3531638183 learned_addr learned my addr 172.21.15.40:0/3531638183 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b66d9d640 1 --2- 172.21.15.40:0/3531638183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a6e50 0x5624a63b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b5ffff640 1 --2- 172.21.15.40:0/3531638183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a1ce0 0x5624a63b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6659c640 1 -- 172.21.15.40:0/3531638183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a6e50 msgr2=0x5624a63b2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6659c640 1 --2- 172.21.15.40:0/3531638183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a6e50 0x5624a63b2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6659c640 1 -- 172.21.15.40:0/3531638183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a1ce0 msgr2=0x5624a63b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6659c640 1 --2- 172.21.15.40:0/3531638183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a1ce0 0x5624a63b1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6659c640 1 -- 172.21.15.40:0/3531638183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624a63b25f0 con 0x5624a63a12f0 2024-03-18T22:45:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b6659c640 1 --2- 172.21.15.40:0/3531638183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a12f0 0x5624a63b0b20 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5b580b9a30 tx=0x7f5b580bbae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8919904a363beb90 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.688+0000 7f5b4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.689+0000 7f5b4dffb640 1 -- 172.21.15.40:0/3531638183 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b580bd110 con 0x5624a63a12f0 2024-03-18T22:45:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.689+0000 7f5b4dffb640 1 -- 172.21.15.40:0/3531638183 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5b580d6420 con 0x5624a63a12f0 2024-03-18T22:45:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.689+0000 7f5b6f972740 1 -- 172.21.15.40:0/3531638183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b580bdcd0 con 0x5624a63a12f0 2024-03-18T22:45:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.689+0000 7f5b4dffb640 1 -- 172.21.15.40:0/3531638183 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5b580bda90 con 0x5624a63a12f0 2024-03-18T22:45:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.689+0000 7f5b6f972740 10 client.16200 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.689+0000 7f5b6f972740 1 -- 172.21.15.40:0/3531638183 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5624a63b25f0 con 0x5624a63a12f0 2024-03-18T22:45:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.689+0000 7f5b4dffb640 1 -- 172.21.15.40:0/3531638183 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b580bdcd0 con 0x5624a63a12f0 2024-03-18T22:45:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.689+0000 7f5b4dffb640 1 -- 172.21.15.40:0/3531638183 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f5b580c2640 con 0x5624a63a12f0 2024-03-18T22:45:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.689+0000 7f5b6f972740 10 client.16200 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.689+0000 7f5b6f972740 10 client.16200 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:45:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.689+0000 7f5b6f972740 20 client.16200 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.689+0000 7f5b6f972740 1 --2- 172.21.15.40:0/3531638183 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5624a641e390 0x5624a6420780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.689+0000 7f5b6f972740 4 client.16200 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:45:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.689+0000 7f5b6f972740 1 -- 172.21.15.40:0/3531638183 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5624a63129f0 con 0x5624a641e390 2024-03-18T22:45:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.689+0000 7f5b5ffff640 1 --2- 172.21.15.40:0/3531638183 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5624a641e390 0x5624a6420780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.690+0000 7f5b4dffb640 10 client.16200 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:45:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.690+0000 7f5b5ffff640 1 --2- 172.21.15.40:0/3531638183 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5624a641e390 0x5624a6420780 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1804fb3c7e00ef47 server_cookie=8986951e1f37cd48 in_seq=0 out_seq=0 2024-03-18T22:45:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.690+0000 7f5b4dffb640 1 -- 172.21.15.40:0/3531638183 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5624a63129f0 con 0x5624a641e390 2024-03-18T22:45:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.690+0000 7f5b4dffb640 10 client.16200 handle_command_reply: tid=0 2024-03-18T22:45:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.690+0000 7f5b6f972740 1 client.16200 shutdown 2024-03-18T22:45:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.690+0000 7f5b6f972740 1 -- 172.21.15.40:0/3531638183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a12f0 msgr2=0x5624a63b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.690+0000 7f5b6f972740 1 --2- 172.21.15.40:0/3531638183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a12f0 0x5624a63b0b20 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5b580b9a30 tx=0x7f5b580bbae0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.690+0000 7f5b6f972740 1 -- 172.21.15.40:0/3531638183 shutdown_connections 2024-03-18T22:45:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.690+0000 7f5b6f972740 1 --2- 172.21.15.40:0/3531638183 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5624a641e390 0x5624a6420780 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.690+0000 7f5b6f972740 1 --2- 172.21.15.40:0/3531638183 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624a63a6e50 0x5624a63b2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.690+0000 7f5b4dffb640 0 client.16200 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:45:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.690+0000 7f5b6f972740 1 --2- 172.21.15.40:0/3531638183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624a63a1ce0 0x5624a63b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.690+0000 7f5b6f972740 1 --2- 172.21.15.40:0/3531638183 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624a63a12f0 0x5624a63b0b20 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.690+0000 7f5b6f972740 1 -- 172.21.15.40:0/3531638183 >> 172.21.15.40:0/3531638183 conn(0x5624a639c860 msgr2=0x5624a639fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.690+0000 7f5b6f972740 1 -- 172.21.15.40:0/3531638183 shutdown_connections 2024-03-18T22:45:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.691+0000 7f5b6f972740 1 -- 172.21.15.40:0/3531638183 wait complete. 2024-03-18T22:45:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.691+0000 7f5b6f972740 20 client.16200 trim_cache size 0 max 16384 2024-03-18T22:45:20.699 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:45:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.691+0000 7f5b6ccac640 1 -- 172.21.15.40:0/316883141 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b540a2800 msgr2=0x7f5b540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.691+0000 7f5b6ccac640 1 --2- 172.21.15.40:0/316883141 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b540a2800 0x7f5b540a4cc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x5624a63a2540 tx=0x7f5b60004ab0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.691+0000 7f5b6ccac640 1 -- 172.21.15.40:0/316883141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b68102320 msgr2=0x7f5b68197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.691+0000 7f5b6ccac640 1 --2- 172.21.15.40:0/316883141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b68102320 0x7f5b68197ab0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5b58006010 tx=0x7f5b58005e40 comp rx=0 tx=0).stop 2024-03-18T22:45:20.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.691+0000 7f5b6ccac640 1 -- 172.21.15.40:0/316883141 shutdown_connections 2024-03-18T22:45:20.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.691+0000 7f5b6ccac640 1 --2- 172.21.15.40:0/316883141 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5b540a2800 0x7f5b540a4cc0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.691+0000 7f5b6ccac640 1 --2- 172.21.15.40:0/316883141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b68107580 0x7f5b6819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.691+0000 7f5b6ccac640 1 --2- 172.21.15.40:0/316883141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5b68104c50 0x7f5b68197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.691+0000 7f5b6ccac640 1 --2- 172.21.15.40:0/316883141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b68102320 0x7f5b68197ab0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.691+0000 7f5b6ccac640 1 -- 172.21.15.40:0/316883141 >> 172.21.15.40:0/316883141 conn(0x7f5b680fbc90 msgr2=0x7f5b68102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.691+0000 7f5b6ccac640 1 -- 172.21.15.40:0/316883141 shutdown_connections 2024-03-18T22:45:20.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.691+0000 7f5b6ccac640 1 -- 172.21.15.40:0/316883141 wait complete. 2024-03-18T22:45:20.704 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:45:20.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.814+0000 7feb45fd4640 1 -- 172.21.15.40:0/1868980566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40101950 msgr2=0x7feb40101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.814+0000 7feb45fd4640 1 --2- 172.21.15.40:0/1868980566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40101950 0x7feb40101dd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7feb34009900 tx=0x7feb3402bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:20.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.814+0000 7feb45fd4640 1 -- 172.21.15.40:0/1868980566 shutdown_connections 2024-03-18T22:45:20.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.814+0000 7feb45fd4640 1 --2- 172.21.15.40:0/1868980566 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb40102310 0x7feb40108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.814+0000 7feb45fd4640 1 --2- 172.21.15.40:0/1868980566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40101950 0x7feb40101dd0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.814+0000 7feb45fd4640 1 --2- 172.21.15.40:0/1868980566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40100760 0x7feb40100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.814+0000 7feb45fd4640 1 -- 172.21.15.40:0/1868980566 >> 172.21.15.40:0/1868980566 conn(0x7feb400fbcf0 msgr2=0x7feb400fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.814+0000 7feb45fd4640 1 -- 172.21.15.40:0/1868980566 shutdown_connections 2024-03-18T22:45:20.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.814+0000 7feb45fd4640 1 -- 172.21.15.40:0/1868980566 wait complete. 2024-03-18T22:45:20.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.814+0000 7feb45fd4640 1 Processor -- start 2024-03-18T22:45:20.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb45fd4640 1 -- start start 2024-03-18T22:45:20.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb45fd4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40100760 0x7feb40197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb45fd4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb40101950 0x7feb40197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb45fd4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40102310 0x7feb4019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb45fd4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb4010c290 con 0x7feb40102310 2024-03-18T22:45:20.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb45fd4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feb4010c110 con 0x7feb40100760 2024-03-18T22:45:20.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb45fd4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7feb4010c410 con 0x7feb40101950 2024-03-18T22:45:20.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb3ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40102310 0x7feb4019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb3ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40102310 0x7feb4019f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58452/0 (socket says 172.21.15.40:58452) 2024-03-18T22:45:20.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb3f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40100760 0x7feb40197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb3ffff640 1 -- 172.21.15.40:0/3037842684 learned_addr learned my addr 172.21.15.40:0/3037842684 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb3effd640 1 --2- 172.21.15.40:0/3037842684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb40101950 0x7feb40197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb3ffff640 1 -- 172.21.15.40:0/3037842684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb40101950 msgr2=0x7feb40197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb3ffff640 1 --2- 172.21.15.40:0/3037842684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb40101950 0x7feb40197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb3ffff640 1 -- 172.21.15.40:0/3037842684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40100760 msgr2=0x7feb40197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb3ffff640 1 --2- 172.21.15.40:0/3037842684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40100760 0x7feb40197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb3ffff640 1 -- 172.21.15.40:0/3037842684 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4019f680 con 0x7feb40102310 2024-03-18T22:45:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb3f7fe640 1 --2- 172.21.15.40:0/3037842684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40100760 0x7feb40197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.815+0000 7feb3effd640 1 --2- 172.21.15.40:0/3037842684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb40101950 0x7feb40197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.816+0000 7feb3ffff640 1 --2- 172.21.15.40:0/3037842684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40102310 0x7feb4019f010 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7feb3800eb10 tx=0x7feb3800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.816+0000 7feb3cff9640 1 -- 172.21.15.40:0/3037842684 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb38004b20 con 0x7feb40102310 2024-03-18T22:45:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.816+0000 7feb45fd4640 1 -- 172.21.15.40:0/3037842684 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb4019f970 con 0x7feb40102310 2024-03-18T22:45:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.816+0000 7feb3cff9640 1 -- 172.21.15.40:0/3037842684 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb38004cc0 con 0x7feb40102310 2024-03-18T22:45:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.816+0000 7feb45fd4640 1 -- 172.21.15.40:0/3037842684 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb4019fee0 con 0x7feb40102310 2024-03-18T22:45:20.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.816+0000 7feb3cff9640 1 -- 172.21.15.40:0/3037842684 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb38010690 con 0x7feb40102310 2024-03-18T22:45:20.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.818+0000 7feb3cff9640 1 -- 172.21.15.40:0/3037842684 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7feb38005020 con 0x7feb40102310 2024-03-18T22:45:20.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb48bc0740 1 Processor -- start 2024-03-18T22:45:20.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb3cff9640 1 --2- 172.21.15.40:0/3037842684 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb280a2800 0x7feb280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb48bc0740 1 -- start start 2024-03-18T22:45:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb3effd640 1 --2- 172.21.15.40:0/3037842684 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb280a2800 0x7feb280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb3cff9640 1 -- 172.21.15.40:0/3037842684 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7feb380cf540 con 0x7feb40102310 2024-03-18T22:45:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb48bc0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb48bc0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6bce0 0x558a8bb707e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb48bc0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb75260 0x558a8bb776f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb48bc0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558a8bacf4b0 con 0x558a8bb6b2f0 2024-03-18T22:45:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb48bc0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558a8ba77d70 con 0x558a8bb6bce0 2024-03-18T22:45:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb48bc0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x558a8bacb550 con 0x558a8bb75260 2024-03-18T22:45:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb3f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb3f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58456/0 (socket says 172.21.15.40:58456) 2024-03-18T22:45:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb3f7fe640 1 -- 172.21.15.40:0/353157587 learned_addr learned my addr 172.21.15.40:0/353157587 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb3ffff640 1 --2- 172.21.15.40:0/353157587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb75260 0x558a8bb776f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb3effd640 1 --2- 172.21.15.40:0/353157587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6bce0 0x558a8bb707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb3f7fe640 1 -- 172.21.15.40:0/353157587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb75260 msgr2=0x558a8bb776f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb3f7fe640 1 --2- 172.21.15.40:0/353157587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb75260 0x558a8bb776f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb3f7fe640 1 -- 172.21.15.40:0/353157587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6bce0 msgr2=0x558a8bb707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb3f7fe640 1 --2- 172.21.15.40:0/353157587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6bce0 0x558a8bb707e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb3f7fe640 1 -- 172.21.15.40:0/353157587 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a8ba9a210 con 0x558a8bb6b2f0 2024-03-18T22:45:20.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb3effd640 1 --2- 172.21.15.40:0/353157587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6bce0 0x558a8bb707e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:45:20.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.819+0000 7feb3ffff640 1 --2- 172.21.15.40:0/353157587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb75260 0x558a8bb776f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.820+0000 7feb3effd640 1 --2- 172.21.15.40:0/3037842684 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb280a2800 0x7feb280a4cc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7feb34009230 tx=0x7feb340041a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.820+0000 7feb3f7fe640 1 --2- 172.21.15.40:0/353157587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6b710 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7feb300114c0 tx=0x7feb30011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6d1a06b5dc2f65 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.820+0000 7feb25ffb640 1 -- 172.21.15.40:0/353157587 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30004020 con 0x558a8bb6b2f0 2024-03-18T22:45:20.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.820+0000 7feb25ffb640 1 -- 172.21.15.40:0/353157587 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb300054d0 con 0x558a8bb6b2f0 2024-03-18T22:45:20.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.820+0000 7feb25ffb640 1 -- 172.21.15.40:0/353157587 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30004cb0 con 0x558a8bb6b2f0 2024-03-18T22:45:20.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.821+0000 7feb48bc0740 1 -- 172.21.15.40:0/353157587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb6b2f0 msgr2=0x558a8bb6b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.821+0000 7feb48bc0740 1 --2- 172.21.15.40:0/353157587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6b710 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7feb300114c0 tx=0x7feb30011980 comp rx=0 tx=0).stop 2024-03-18T22:45:20.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.821+0000 7feb48bc0740 1 -- 172.21.15.40:0/353157587 shutdown_connections 2024-03-18T22:45:20.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.821+0000 7feb48bc0740 1 --2- 172.21.15.40:0/353157587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb75260 0x558a8bb776f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.821+0000 7feb48bc0740 1 --2- 172.21.15.40:0/353157587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6bce0 0x558a8bb707e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.821+0000 7feb48bc0740 1 --2- 172.21.15.40:0/353157587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6b710 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.821+0000 7feb48bc0740 1 -- 172.21.15.40:0/353157587 >> 172.21.15.40:0/353157587 conn(0x558a8bb66860 msgr2=0x558a8bb66c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.821+0000 7feb48bc0740 1 -- 172.21.15.40:0/353157587 shutdown_connections 2024-03-18T22:45:20.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.821+0000 7feb48bc0740 1 -- 172.21.15.40:0/353157587 wait complete. 2024-03-18T22:45:20.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.821+0000 7feb48bc0740 1 -- start start 2024-03-18T22:45:20.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.821+0000 7feb48bc0740 1 Processor -- start 2024-03-18T22:45:20.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb48bc0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb48bc0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb6bce0 0x558a8bb6eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb48bc0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb75260 0x558a8bb6fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb48bc0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558a8bb71180 con 0x558a8bb6bce0 2024-03-18T22:45:20.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb48bc0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558a8ba8b310 con 0x558a8bb6b2f0 2024-03-18T22:45:20.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb48bc0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x558a8bb71300 con 0x558a8bb75260 2024-03-18T22:45:20.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb48bc0740 10 client.0 _ll_register_callbacks cb 0x558a8ba7f2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:20.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb3f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb3effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb6bce0 0x558a8bb6eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb3f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:34450/0 (socket says 172.21.15.40:34450) 2024-03-18T22:45:20.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb3f7fe640 1 -- 172.21.15.40:0/715759787 learned_addr learned my addr 172.21.15.40:0/715759787 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb3ffff640 1 --2- 172.21.15.40:0/715759787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb75260 0x558a8bb6fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb3f7fe640 1 -- 172.21.15.40:0/715759787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb75260 msgr2=0x558a8bb6fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb3f7fe640 1 --2- 172.21.15.40:0/715759787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb75260 0x558a8bb6fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb3f7fe640 1 -- 172.21.15.40:0/715759787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb6bce0 msgr2=0x558a8bb6eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb3f7fe640 1 --2- 172.21.15.40:0/715759787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb6bce0 0x558a8bb6eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb3f7fe640 1 -- 172.21.15.40:0/715759787 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a8b7e33f0 con 0x558a8bb6b2f0 2024-03-18T22:45:20.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb3f7fe640 1 --2- 172.21.15.40:0/715759787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6e660 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7feb30009530 tx=0x7feb3000fbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=415479a69caca73d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb1effd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:45:20.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb1effd640 1 -- 172.21.15.40:0/715759787 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30028020 con 0x558a8bb6b2f0 2024-03-18T22:45:20.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb48bc0740 1 -- 172.21.15.40:0/715759787 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb300054d0 con 0x558a8bb6b2f0 2024-03-18T22:45:20.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.822+0000 7feb1effd640 1 -- 172.21.15.40:0/715759787 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7feb3000fda0 con 0x558a8bb6b2f0 2024-03-18T22:45:20.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.823+0000 7feb1effd640 1 -- 172.21.15.40:0/715759787 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7feb3001d040 con 0x558a8bb6b2f0 2024-03-18T22:45:20.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.823+0000 7feb48bc0740 10 client.26057 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:20.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.823+0000 7feb48bc0740 1 -- 172.21.15.40:0/715759787 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558a8b8ad7d0 con 0x558a8bb6b2f0 2024-03-18T22:45:20.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.823+0000 7feb1effd640 1 -- 172.21.15.40:0/715759787 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb300054d0 con 0x558a8bb6b2f0 2024-03-18T22:45:20.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.823+0000 7feb1effd640 1 -- 172.21.15.40:0/715759787 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7feb30004f10 con 0x558a8bb6b2f0 2024-03-18T22:45:20.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.823+0000 7feb48bc0740 10 client.26057 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:20.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.823+0000 7feb48bc0740 10 client.26057 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:20.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.823+0000 7feb48bc0740 20 client.26057 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:20.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.823+0000 7feb48bc0740 1 --2- 172.21.15.40:0/715759787 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a8bb89cb0 0x558a8bb8c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.823+0000 7feb48bc0740 4 client.26057 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:20.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.823+0000 7feb48bc0740 1 -- 172.21.15.40:0/715759787 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a8bb2d3e0 con 0x558a8bb89cb0 2024-03-18T22:45:20.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.823+0000 7feb3effd640 1 --2- 172.21.15.40:0/715759787 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a8bb89cb0 0x558a8bb8c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.824+0000 7feb1effd640 10 client.26057 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:20.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.824+0000 7feb3effd640 1 --2- 172.21.15.40:0/715759787 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a8bb89cb0 0x558a8bb8c0f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=313c7f0ab07fce56 server_cookie=886ceec12dac1e45 in_seq=0 out_seq=0 2024-03-18T22:45:20.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.824+0000 7feb1effd640 1 -- 172.21.15.40:0/715759787 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558a8bb2d3e0 con 0x558a8bb89cb0 2024-03-18T22:45:20.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.824+0000 7feb1effd640 10 client.26057 handle_command_reply: tid=0 2024-03-18T22:45:20.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.824+0000 7feb48bc0740 1 client.26057 shutdown 2024-03-18T22:45:20.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.824+0000 7feb48bc0740 1 -- 172.21.15.40:0/715759787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6b2f0 msgr2=0x558a8bb6e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.824+0000 7feb48bc0740 1 --2- 172.21.15.40:0/715759787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6e660 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7feb30009530 tx=0x7feb3000fbb0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.824+0000 7feb48bc0740 1 -- 172.21.15.40:0/715759787 shutdown_connections 2024-03-18T22:45:20.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.824+0000 7feb48bc0740 1 --2- 172.21.15.40:0/715759787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb75260 0x558a8bb6fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.824+0000 7feb48bc0740 1 --2- 172.21.15.40:0/715759787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb6bce0 0x558a8bb6eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.824+0000 7feb48bc0740 1 --2- 172.21.15.40:0/715759787 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a8bb89cb0 0x558a8bb8c0f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.825+0000 7feb48bc0740 1 --2- 172.21.15.40:0/715759787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6e660 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.825+0000 7feb1effd640 0 client.26057 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:20.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.825+0000 7feb48bc0740 1 -- 172.21.15.40:0/715759787 >> 172.21.15.40:0/715759787 conn(0x558a8bb66860 msgr2=0x558a8bb65ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.825+0000 7feb48bc0740 1 -- 172.21.15.40:0/715759787 shutdown_connections 2024-03-18T22:45:20.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.825+0000 7feb48bc0740 1 -- 172.21.15.40:0/715759787 wait complete. 2024-03-18T22:45:20.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.825+0000 7feb48bc0740 20 client.26057 trim_cache size 0 max 16384 2024-03-18T22:45:20.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.839+0000 7feb48bc0740 1 Processor -- start 2024-03-18T22:45:20.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.839+0000 7feb48bc0740 1 -- start start 2024-03-18T22:45:20.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.839+0000 7feb48bc0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.839+0000 7feb48bc0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb6bce0 0x558a8bb6d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.839+0000 7feb48bc0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb75260 0x558a8bb6de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.839+0000 7feb48bc0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb401a8380 con 0x558a8bb75260 2024-03-18T22:45:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.839+0000 7feb48bc0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feb401a9f80 con 0x558a8bb6b2f0 2024-03-18T22:45:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.839+0000 7feb48bc0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7feb401a8a50 con 0x558a8bb6bce0 2024-03-18T22:45:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.839+0000 7feb3ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb75260 0x558a8bb6de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.839+0000 7feb3ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb75260 0x558a8bb6de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58472/0 (socket says 172.21.15.40:58472) 2024-03-18T22:45:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.839+0000 7feb3ffff640 1 -- 172.21.15.40:0/4110992663 learned_addr learned my addr 172.21.15.40:0/4110992663 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.839+0000 7feb3f7fe640 1 --2- 172.21.15.40:0/4110992663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.839+0000 7feb3effd640 1 --2- 172.21.15.40:0/4110992663 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb6bce0 0x558a8bb6d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.840+0000 7feb3f7fe640 1 -- 172.21.15.40:0/4110992663 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb6bce0 msgr2=0x558a8bb6d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.840+0000 7feb3f7fe640 1 --2- 172.21.15.40:0/4110992663 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb6bce0 0x558a8bb6d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.840+0000 7feb3f7fe640 1 -- 172.21.15.40:0/4110992663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb75260 msgr2=0x558a8bb6de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.840+0000 7feb3f7fe640 1 --2- 172.21.15.40:0/4110992663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb75260 0x558a8bb6de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.840+0000 7feb3f7fe640 1 -- 172.21.15.40:0/4110992663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a8bb71060 con 0x558a8bb6b2f0 2024-03-18T22:45:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.840+0000 7feb3ffff640 1 --2- 172.21.15.40:0/4110992663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb75260 0x558a8bb6de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.840+0000 7feb3effd640 1 --2- 172.21.15.40:0/4110992663 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb6bce0 0x558a8bb6d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.840+0000 7feb3f7fe640 1 --2- 172.21.15.40:0/4110992663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6a1d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7feb30009580 tx=0x7feb30001e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=652bc9a4a6d92712 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.840+0000 7feb257fa640 1 -- 172.21.15.40:0/4110992663 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30028070 con 0x558a8bb6b2f0 2024-03-18T22:45:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.840+0000 7feb257fa640 1 -- 172.21.15.40:0/4110992663 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb300235b0 con 0x558a8bb6b2f0 2024-03-18T22:45:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.840+0000 7feb257fa640 1 -- 172.21.15.40:0/4110992663 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30023a50 con 0x558a8bb6b2f0 2024-03-18T22:45:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 -- 172.21.15.40:0/4110992663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6b2f0 msgr2=0x558a8bb6a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 --2- 172.21.15.40:0/4110992663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6a1d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7feb30009580 tx=0x7feb30001e50 comp rx=0 tx=0).stop 2024-03-18T22:45:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 -- 172.21.15.40:0/4110992663 shutdown_connections 2024-03-18T22:45:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 --2- 172.21.15.40:0/4110992663 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb75260 0x558a8bb6de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 --2- 172.21.15.40:0/4110992663 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb6bce0 0x558a8bb6d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 --2- 172.21.15.40:0/4110992663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6b2f0 0x558a8bb6a1d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 -- 172.21.15.40:0/4110992663 >> 172.21.15.40:0/4110992663 conn(0x558a8bb66860 msgr2=0x558a8bb69ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 -- 172.21.15.40:0/4110992663 shutdown_connections 2024-03-18T22:45:20.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 -- 172.21.15.40:0/4110992663 wait complete. 2024-03-18T22:45:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 -- start start 2024-03-18T22:45:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 Processor -- start 2024-03-18T22:45:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb6b2f0 0x558a8bb830d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6bce0 0x558a8bb83610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb75260 0x558a8bb84620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb401a97f0 con 0x558a8bb75260 2024-03-18T22:45:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feb401a9120 con 0x558a8bb6bce0 2024-03-18T22:45:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7feb401a6360 con 0x558a8bb6b2f0 2024-03-18T22:45:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.841+0000 7feb48bc0740 10 client.0 _ll_register_callbacks cb 0x7feb38086040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb3ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb75260 0x558a8bb84620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb3effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6bce0 0x558a8bb83610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb3effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6bce0 0x558a8bb83610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:34466/0 (socket says 172.21.15.40:34466) 2024-03-18T22:45:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb3effd640 1 -- 172.21.15.40:0/2959866835 learned_addr learned my addr 172.21.15.40:0/2959866835 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb3f7fe640 1 --2- 172.21.15.40:0/2959866835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb6b2f0 0x558a8bb830d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb3effd640 1 -- 172.21.15.40:0/2959866835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb6b2f0 msgr2=0x558a8bb830d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb3effd640 1 --2- 172.21.15.40:0/2959866835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb6b2f0 0x558a8bb830d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb3effd640 1 -- 172.21.15.40:0/2959866835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb75260 msgr2=0x558a8bb84620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:20.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb3effd640 1 --2- 172.21.15.40:0/2959866835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb75260 0x558a8bb84620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb3effd640 1 -- 172.21.15.40:0/2959866835 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a8bb84c10 con 0x558a8bb6bce0 2024-03-18T22:45:20.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb3ffff640 1 --2- 172.21.15.40:0/2959866835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb75260 0x558a8bb84620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:20.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb3effd640 1 --2- 172.21.15.40:0/2959866835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6bce0 0x558a8bb83610 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7feb34009900 tx=0x7feb3402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8b91c87df1f461b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:20.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb1effd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:45:20.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb1effd640 1 -- 172.21.15.40:0/2959866835 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34049070 con 0x558a8bb6bce0 2024-03-18T22:45:20.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb1effd640 1 -- 172.21.15.40:0/2959866835 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7feb34053350 con 0x558a8bb6bce0 2024-03-18T22:45:20.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.842+0000 7feb48bc0740 1 -- 172.21.15.40:0/2959866835 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb300235b0 con 0x558a8bb6bce0 2024-03-18T22:45:20.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.843+0000 7feb1effd640 1 -- 172.21.15.40:0/2959866835 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7feb34053550 con 0x558a8bb6bce0 2024-03-18T22:45:20.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.843+0000 7feb48bc0740 10 client.26063 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:20.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.843+0000 7feb48bc0740 1 -- 172.21.15.40:0/2959866835 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558a8bbefff0 con 0x558a8bb6bce0 2024-03-18T22:45:20.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.843+0000 7feb1effd640 1 -- 172.21.15.40:0/2959866835 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb300235b0 con 0x558a8bb6bce0 2024-03-18T22:45:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.843+0000 7feb1effd640 1 -- 172.21.15.40:0/2959866835 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7feb34040040 con 0x558a8bb6bce0 2024-03-18T22:45:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.843+0000 7feb48bc0740 10 client.26063 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.843+0000 7feb48bc0740 10 client.26063 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.843+0000 7feb48bc0740 20 client.26063 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.843+0000 7feb48bc0740 1 --2- 172.21.15.40:0/2959866835 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a8bbf0970 0x558a8bbf2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.843+0000 7feb48bc0740 4 client.26063 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:45:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.843+0000 7feb48bc0740 1 -- 172.21.15.40:0/2959866835 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558a8badc9f0 con 0x558a8bbf0970 2024-03-18T22:45:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.843+0000 7feb3f7fe640 1 --2- 172.21.15.40:0/2959866835 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a8bbf0970 0x558a8bbf2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb1effd640 10 client.26063 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb3f7fe640 1 --2- 172.21.15.40:0/2959866835 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a8bbf0970 0x558a8bbf2d90 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bde9c34d8489426 server_cookie=5a4f6198b9e4e476 in_seq=0 out_seq=0 2024-03-18T22:45:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb1effd640 1 -- 172.21.15.40:0/2959866835 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558a8badc9f0 con 0x558a8bbf0970 2024-03-18T22:45:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb1effd640 10 client.26063 handle_command_reply: tid=0 2024-03-18T22:45:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb48bc0740 1 client.26063 shutdown 2024-03-18T22:45:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb48bc0740 1 -- 172.21.15.40:0/2959866835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6bce0 msgr2=0x558a8bb83610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb48bc0740 1 --2- 172.21.15.40:0/2959866835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6bce0 0x558a8bb83610 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7feb34009900 tx=0x7feb3402bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb48bc0740 1 -- 172.21.15.40:0/2959866835 shutdown_connections 2024-03-18T22:45:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb48bc0740 1 --2- 172.21.15.40:0/2959866835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558a8bb75260 0x558a8bb84620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb48bc0740 1 --2- 172.21.15.40:0/2959866835 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558a8bbf0970 0x558a8bbf2d90 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb48bc0740 1 --2- 172.21.15.40:0/2959866835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558a8bb6bce0 0x558a8bb83610 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb1effd640 0 client.26063 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb48bc0740 1 --2- 172.21.15.40:0/2959866835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558a8bb6b2f0 0x558a8bb830d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb48bc0740 1 -- 172.21.15.40:0/2959866835 >> 172.21.15.40:0/2959866835 conn(0x558a8bb66860 msgr2=0x558a8bb6a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb48bc0740 1 -- 172.21.15.40:0/2959866835 shutdown_connections 2024-03-18T22:45:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb48bc0740 1 -- 172.21.15.40:0/2959866835 wait complete. 2024-03-18T22:45:20.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.844+0000 7feb48bc0740 20 client.26063 trim_cache size 0 max 16384 2024-03-18T22:45:20.852 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:45:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.845+0000 7feb45fd4640 1 -- 172.21.15.40:0/3037842684 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb280a2800 msgr2=0x7feb280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.845+0000 7feb45fd4640 1 --2- 172.21.15.40:0/3037842684 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb280a2800 0x7feb280a4cc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7feb34009230 tx=0x7feb340041a0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.845+0000 7feb45fd4640 1 -- 172.21.15.40:0/3037842684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40102310 msgr2=0x7feb4019f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.845+0000 7feb45fd4640 1 --2- 172.21.15.40:0/3037842684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40102310 0x7feb4019f010 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7feb3800eb10 tx=0x7feb3800c2d0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.845+0000 7feb45fd4640 1 -- 172.21.15.40:0/3037842684 shutdown_connections 2024-03-18T22:45:20.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.845+0000 7feb45fd4640 1 --2- 172.21.15.40:0/3037842684 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb280a2800 0x7feb280a4cc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.845+0000 7feb45fd4640 1 --2- 172.21.15.40:0/3037842684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40102310 0x7feb4019f010 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.845+0000 7feb45fd4640 1 --2- 172.21.15.40:0/3037842684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb40101950 0x7feb40197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.845+0000 7feb45fd4640 1 --2- 172.21.15.40:0/3037842684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40100760 0x7feb40197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.845+0000 7feb45fd4640 1 -- 172.21.15.40:0/3037842684 >> 172.21.15.40:0/3037842684 conn(0x7feb400fbcf0 msgr2=0x7feb400fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.845+0000 7feb45fd4640 1 -- 172.21.15.40:0/3037842684 shutdown_connections 2024-03-18T22:45:20.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:20.845+0000 7feb45fd4640 1 -- 172.21.15.40:0/3037842684 wait complete. 2024-03-18T22:45:21.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:20 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/989253640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:45:21.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:20 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/989253640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:45:21.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:20 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/989253640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:45:21.860 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:45:21.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:45:21.860 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:45:21.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.975+0000 7f6aff73b640 1 -- 172.21.15.40:0/3565339746 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af8100740 msgr2=0x7f6af8100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:21.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.975+0000 7f6aff73b640 1 --2- 172.21.15.40:0/3565339746 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af8100740 0x7f6af8100b60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6af4009900 tx=0x7f6af402bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:21.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.975+0000 7f6aff73b640 1 -- 172.21.15.40:0/3565339746 shutdown_connections 2024-03-18T22:45:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.975+0000 7f6aff73b640 1 --2- 172.21.15.40:0/3565339746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6af81022f0 0x7f6af8108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.975+0000 7f6aff73b640 1 --2- 172.21.15.40:0/3565339746 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6af8101930 0x7f6af8101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.975+0000 7f6aff73b640 1 --2- 172.21.15.40:0/3565339746 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af8100740 0x7f6af8100b60 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.975+0000 7f6aff73b640 1 -- 172.21.15.40:0/3565339746 >> 172.21.15.40:0/3565339746 conn(0x7f6af80fbcd0 msgr2=0x7f6af80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.976+0000 7f6aff73b640 1 -- 172.21.15.40:0/3565339746 shutdown_connections 2024-03-18T22:45:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.976+0000 7f6aff73b640 1 -- 172.21.15.40:0/3565339746 wait complete. 2024-03-18T22:45:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.976+0000 7f6aff73b640 1 Processor -- start 2024-03-18T22:45:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.976+0000 7f6aff73b640 1 -- start start 2024-03-18T22:45:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.976+0000 7f6aff73b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af8100740 0x7f6af8197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.976+0000 7f6aff73b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6af8101930 0x7f6af8198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.976+0000 7f6aff73b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6af81022f0 0x7f6af819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.976+0000 7f6aff73b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6af810c260 con 0x7f6af8100740 2024-03-18T22:45:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.976+0000 7f6aff73b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6af810c0e0 con 0x7f6af8101930 2024-03-18T22:45:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.976+0000 7f6aff73b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6af810c3e0 con 0x7f6af81022f0 2024-03-18T22:45:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.976+0000 7f6afd4d7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af8100740 0x7f6af8197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.976+0000 7f6afccd6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6af8101930 0x7f6af8198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.976+0000 7f6afd4d7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af8100740 0x7f6af8197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58484/0 (socket says 172.21.15.40:58484) 2024-03-18T22:45:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.976+0000 7f6afd4d7640 1 -- 172.21.15.40:0/1049403621 learned_addr learned my addr 172.21.15.40:0/1049403621 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.976+0000 7f6afdcd8640 1 --2- 172.21.15.40:0/1049403621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6af81022f0 0x7f6af819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.977+0000 7f6afccd6640 1 -- 172.21.15.40:0/1049403621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6af81022f0 msgr2=0x7f6af819f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.977+0000 7f6afccd6640 1 --2- 172.21.15.40:0/1049403621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6af81022f0 0x7f6af819f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.977+0000 7f6afccd6640 1 -- 172.21.15.40:0/1049403621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af8100740 msgr2=0x7f6af8197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.977+0000 7f6afccd6640 1 --2- 172.21.15.40:0/1049403621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af8100740 0x7f6af8197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.977+0000 7f6afccd6640 1 -- 172.21.15.40:0/1049403621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af819f730 con 0x7f6af8101930 2024-03-18T22:45:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.977+0000 7f6afd4d7640 1 --2- 172.21.15.40:0/1049403621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af8100740 0x7f6af8197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:21.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.977+0000 7f6afccd6640 1 --2- 172.21.15.40:0/1049403621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6af8101930 0x7f6af8198050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800e9f0 tx=0x7f6ae800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:21.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.977+0000 7f6aee7fc640 1 -- 172.21.15.40:0/1049403621 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae800ccc0 con 0x7f6af8101930 2024-03-18T22:45:21.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.977+0000 7f6aee7fc640 1 -- 172.21.15.40:0/1049403621 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ae8005020 con 0x7f6af8101930 2024-03-18T22:45:21.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.977+0000 7f6aff73b640 1 -- 172.21.15.40:0/1049403621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6af819fa20 con 0x7f6af8101930 2024-03-18T22:45:21.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.977+0000 7f6aee7fc640 1 -- 172.21.15.40:0/1049403621 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8004a00 con 0x7f6af8101930 2024-03-18T22:45:21.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.977+0000 7f6aff73b640 1 -- 172.21.15.40:0/1049403621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6af819ff70 con 0x7f6af8101930 2024-03-18T22:45:21.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.980+0000 7f6aee7fc640 1 -- 172.21.15.40:0/1049403621 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6ae8004ba0 con 0x7f6af8101930 2024-03-18T22:45:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.980+0000 7f6b023fe740 1 Processor -- start 2024-03-18T22:45:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.980+0000 7f6b023fe740 1 -- start start 2024-03-18T22:45:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.980+0000 7f6b023fe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603e13a0 0x562a603e17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.980+0000 7f6b023fe740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e1d90 0x562a603e6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.980+0000 7f6b023fe740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e6f00 0x562a603e9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.980+0000 7f6b023fe740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562a60346160 con 0x562a603e13a0 2024-03-18T22:45:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.980+0000 7f6b023fe740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562a602eee10 con 0x562a603e1d90 2024-03-18T22:45:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.980+0000 7f6b023fe740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x562a60342200 con 0x562a603e6f00 2024-03-18T22:45:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.980+0000 7f6afdcd8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603e13a0 0x562a603e17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.980+0000 7f6afd4d7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e1d90 0x562a603e6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.981+0000 7f6afd4d7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e1d90 0x562a603e6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:34488/0 (socket says 172.21.15.40:34488) 2024-03-18T22:45:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.981+0000 7f6afd4d7640 1 -- 172.21.15.40:0/3121819478 learned_addr learned my addr 172.21.15.40:0/3121819478 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.981+0000 7f6afccd6640 1 --2- 172.21.15.40:0/3121819478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e6f00 0x562a603e9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.981+0000 7f6aee7fc640 1 --2- 172.21.15.40:0/1049403621 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ae40a2880 0x7f6ae40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.981+0000 7f6afdcd8640 1 --2- 172.21.15.40:0/1049403621 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ae40a2880 0x7f6ae40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.981+0000 7f6afd4d7640 1 -- 172.21.15.40:0/3121819478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e6f00 msgr2=0x562a603e9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.981+0000 7f6afd4d7640 1 --2- 172.21.15.40:0/3121819478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e6f00 0x562a603e9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.981+0000 7f6afd4d7640 1 -- 172.21.15.40:0/3121819478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603e13a0 msgr2=0x562a603e17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.981+0000 7f6afd4d7640 1 --2- 172.21.15.40:0/3121819478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603e13a0 0x562a603e17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.981+0000 7f6afd4d7640 1 -- 172.21.15.40:0/3121819478 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a601227e0 con 0x562a603e1d90 2024-03-18T22:45:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.981+0000 7f6afccd6640 1 --2- 172.21.15.40:0/3121819478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e6f00 0x562a603e9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.981+0000 7f6aee7fc640 1 -- 172.21.15.40:0/1049403621 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6ae80ced50 con 0x7f6af8101930 2024-03-18T22:45:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.981+0000 7f6afdcd8640 1 --2- 172.21.15.40:0/1049403621 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ae40a2880 0x7f6ae40a4d40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x562a603e25f0 tx=0x7f6af0004ab0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.981+0000 7f6afd4d7640 1 --2- 172.21.15.40:0/3121819478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e1d90 0x562a603e6890 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6af4036040 tx=0x7f6af4009660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=685dd7cdb0903d33 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.981+0000 7f6ae37fe640 1 -- 172.21.15.40:0/3121819478 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af4045020 con 0x562a603e1d90 2024-03-18T22:45:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.982+0000 7f6ae37fe640 1 -- 172.21.15.40:0/3121819478 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6af402cc90 con 0x562a603e1d90 2024-03-18T22:45:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.982+0000 7f6ae37fe640 1 -- 172.21.15.40:0/3121819478 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af4004970 con 0x562a603e1d90 2024-03-18T22:45:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.982+0000 7f6b023fe740 1 -- 172.21.15.40:0/3121819478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e1d90 msgr2=0x562a603e6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.982+0000 7f6b023fe740 1 --2- 172.21.15.40:0/3121819478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e1d90 0x562a603e6890 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6af4036040 tx=0x7f6af4009660 comp rx=0 tx=0).stop 2024-03-18T22:45:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.982+0000 7f6b023fe740 1 -- 172.21.15.40:0/3121819478 shutdown_connections 2024-03-18T22:45:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.982+0000 7f6b023fe740 1 --2- 172.21.15.40:0/3121819478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e6f00 0x562a603e9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.982+0000 7f6b023fe740 1 --2- 172.21.15.40:0/3121819478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e1d90 0x562a603e6890 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.982+0000 7f6b023fe740 1 --2- 172.21.15.40:0/3121819478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603e13a0 0x562a603e17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.982+0000 7f6b023fe740 1 -- 172.21.15.40:0/3121819478 >> 172.21.15.40:0/3121819478 conn(0x562a603dc910 msgr2=0x562a603dcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.982+0000 7f6b023fe740 1 -- 172.21.15.40:0/3121819478 shutdown_connections 2024-03-18T22:45:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.982+0000 7f6b023fe740 1 -- 172.21.15.40:0/3121819478 wait complete. 2024-03-18T22:45:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.983+0000 7f6b023fe740 1 -- start start 2024-03-18T22:45:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.983+0000 7f6b023fe740 1 Processor -- start 2024-03-18T22:45:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.983+0000 7f6b023fe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603e13a0 0x562a603eb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.983+0000 7f6b023fe740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e1d90 0x562a603ebd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:21.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.983+0000 7f6b023fe740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e6f00 0x562a603f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.983+0000 7f6b023fe740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562a603e9e50 con 0x562a603e13a0 2024-03-18T22:45:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.983+0000 7f6b023fe740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562a603021a0 con 0x562a603e6f00 2024-03-18T22:45:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.983+0000 7f6b023fe740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x562a603e9fd0 con 0x562a603e1d90 2024-03-18T22:45:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.983+0000 7f6b023fe740 10 client.0 _ll_register_callbacks cb 0x562a60059c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.983+0000 7f6afd4d7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603e13a0 0x562a603eb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.983+0000 7f6afd4d7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603e13a0 0x562a603eb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58490/0 (socket says 172.21.15.40:58490) 2024-03-18T22:45:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.983+0000 7f6afdcd8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e6f00 0x562a603f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.983+0000 7f6afccd6640 1 --2- 172.21.15.40:0/4145601088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e1d90 0x562a603ebd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.983+0000 7f6afd4d7640 1 -- 172.21.15.40:0/4145601088 learned_addr learned my addr 172.21.15.40:0/4145601088 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6afd4d7640 1 -- 172.21.15.40:0/4145601088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e1d90 msgr2=0x562a603ebd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6afd4d7640 1 --2- 172.21.15.40:0/4145601088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e1d90 0x562a603ebd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6afd4d7640 1 -- 172.21.15.40:0/4145601088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e6f00 msgr2=0x562a603f2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6afd4d7640 1 --2- 172.21.15.40:0/4145601088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e6f00 0x562a603f2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6afd4d7640 1 -- 172.21.15.40:0/4145601088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a60182f70 con 0x562a603e13a0 2024-03-18T22:45:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6afccd6640 1 --2- 172.21.15.40:0/4145601088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e1d90 0x562a603ebd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6afdcd8640 1 --2- 172.21.15.40:0/4145601088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e6f00 0x562a603f2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6afd4d7640 1 --2- 172.21.15.40:0/4145601088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603e13a0 0x562a603eb800 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6af4005ab0 tx=0x7f6af4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6623f12deb698e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6ae0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6ae0ff9640 1 -- 172.21.15.40:0/4145601088 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af4050070 con 0x562a603e13a0 2024-03-18T22:45:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6ae0ff9640 1 -- 172.21.15.40:0/4145601088 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6af4003a00 con 0x562a603e13a0 2024-03-18T22:45:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6b023fe740 1 -- 172.21.15.40:0/4145601088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6af402cc90 con 0x562a603e13a0 2024-03-18T22:45:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6ae0ff9640 1 -- 172.21.15.40:0/4145601088 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6af4059050 con 0x562a603e13a0 2024-03-18T22:45:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6b023fe740 10 client.16230 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6b023fe740 1 -- 172.21.15.40:0/4145601088 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562a6001a170 con 0x562a603e13a0 2024-03-18T22:45:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.984+0000 7f6ae0ff9640 1 -- 172.21.15.40:0/4145601088 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6af402cc90 con 0x562a603e13a0 2024-03-18T22:45:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.985+0000 7f6ae0ff9640 1 -- 172.21.15.40:0/4145601088 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f6af4053020 con 0x562a603e13a0 2024-03-18T22:45:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.985+0000 7f6b023fe740 10 client.16230 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.985+0000 7f6b023fe740 10 client.16230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.985+0000 7f6b023fe740 20 client.16230 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.985+0000 7f6b023fe740 1 --2- 172.21.15.40:0/4145601088 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a603f7200 0x562a603f9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.985+0000 7f6b023fe740 4 client.16230 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.985+0000 7f6b023fe740 1 -- 172.21.15.40:0/4145601088 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a603536a0 con 0x562a603f7200 2024-03-18T22:45:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.985+0000 7f6afccd6640 1 --2- 172.21.15.40:0/4145601088 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a603f7200 0x562a603f9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.985+0000 7f6afccd6640 1 --2- 172.21.15.40:0/4145601088 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a603f7200 0x562a603f9640 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d2a2a7dc1f08b0e server_cookie=4ce801067397f4dd in_seq=0 out_seq=0 2024-03-18T22:45:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.985+0000 7f6ae0ff9640 10 client.16230 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.986+0000 7f6ae0ff9640 1 -- 172.21.15.40:0/4145601088 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562a603536a0 con 0x562a603f7200 2024-03-18T22:45:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.986+0000 7f6ae0ff9640 10 client.16230 handle_command_reply: tid=0 2024-03-18T22:45:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.986+0000 7f6b023fe740 1 client.16230 shutdown 2024-03-18T22:45:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.986+0000 7f6b023fe740 1 -- 172.21.15.40:0/4145601088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603e13a0 msgr2=0x562a603eb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.986+0000 7f6b023fe740 1 --2- 172.21.15.40:0/4145601088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603e13a0 0x562a603eb800 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6af4005ab0 tx=0x7f6af4000bc0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.986+0000 7f6b023fe740 1 -- 172.21.15.40:0/4145601088 shutdown_connections 2024-03-18T22:45:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.986+0000 7f6b023fe740 1 --2- 172.21.15.40:0/4145601088 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a603f7200 0x562a603f9640 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.986+0000 7f6b023fe740 1 --2- 172.21.15.40:0/4145601088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e6f00 0x562a603f2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.986+0000 7f6b023fe740 1 --2- 172.21.15.40:0/4145601088 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e1d90 0x562a603ebd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.986+0000 7f6b023fe740 1 --2- 172.21.15.40:0/4145601088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603e13a0 0x562a603eb800 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.986+0000 7f6ae0ff9640 0 client.16230 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.986+0000 7f6b023fe740 1 -- 172.21.15.40:0/4145601088 >> 172.21.15.40:0/4145601088 conn(0x562a603dc910 msgr2=0x562a603dbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.986+0000 7f6b023fe740 1 -- 172.21.15.40:0/4145601088 shutdown_connections 2024-03-18T22:45:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.986+0000 7f6b023fe740 1 -- 172.21.15.40:0/4145601088 wait complete. 2024-03-18T22:45:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:21.986+0000 7f6b023fe740 20 client.16230 trim_cache size 0 max 16384 2024-03-18T22:45:22.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.008+0000 7f6b023fe740 1 Processor -- start 2024-03-18T22:45:22.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.008+0000 7f6b023fe740 1 -- start start 2024-03-18T22:45:22.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6b023fe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603dc910 0x562a603eaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6b023fe740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e13a0 0x562a603f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6b023fe740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e1d90 0x562a603eb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6b023fe740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6af81aa460 con 0x562a603dc910 2024-03-18T22:45:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6b023fe740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6af81ac060 con 0x562a603e13a0 2024-03-18T22:45:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6b023fe740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6af81aab30 con 0x562a603e1d90 2024-03-18T22:45:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6afd4d7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603dc910 0x562a603eaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6afd4d7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603dc910 0x562a603eaaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58502/0 (socket says 172.21.15.40:58502) 2024-03-18T22:45:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6afd4d7640 1 -- 172.21.15.40:0/1504602508 learned_addr learned my addr 172.21.15.40:0/1504602508 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6afccd6640 1 --2- 172.21.15.40:0/1504602508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e13a0 0x562a603f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6afdcd8640 1 --2- 172.21.15.40:0/1504602508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e1d90 0x562a603eb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6afd4d7640 1 -- 172.21.15.40:0/1504602508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e1d90 msgr2=0x562a603eb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6afd4d7640 1 --2- 172.21.15.40:0/1504602508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e1d90 0x562a603eb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6afd4d7640 1 -- 172.21.15.40:0/1504602508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e13a0 msgr2=0x562a603f2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6afd4d7640 1 --2- 172.21.15.40:0/1504602508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e13a0 0x562a603f2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6afd4d7640 1 -- 172.21.15.40:0/1504602508 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a603d8380 con 0x562a603dc910 2024-03-18T22:45:22.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6afccd6640 1 --2- 172.21.15.40:0/1504602508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e13a0 0x562a603f2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:22.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6afd4d7640 1 --2- 172.21.15.40:0/1504602508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603dc910 0x562a603eaaf0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6af402c840 tx=0x7f6af402c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=543cfd028f1ac73a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:22.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6ae2ffd640 1 -- 172.21.15.40:0/1504602508 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af4050030 con 0x562a603dc910 2024-03-18T22:45:22.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.009+0000 7f6ae2ffd640 1 -- 172.21.15.40:0/1504602508 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6af4005030 con 0x562a603dc910 2024-03-18T22:45:22.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.010+0000 7f6ae2ffd640 1 -- 172.21.15.40:0/1504602508 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af4045040 con 0x562a603dc910 2024-03-18T22:45:22.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.010+0000 7f6b023fe740 1 -- 172.21.15.40:0/1504602508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603dc910 msgr2=0x562a603eaaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:22.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.010+0000 7f6b023fe740 1 --2- 172.21.15.40:0/1504602508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603dc910 0x562a603eaaf0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6af402c840 tx=0x7f6af402c3b0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.010+0000 7f6b023fe740 1 -- 172.21.15.40:0/1504602508 shutdown_connections 2024-03-18T22:45:22.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.010+0000 7f6b023fe740 1 --2- 172.21.15.40:0/1504602508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e1d90 0x562a603eb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.010+0000 7f6b023fe740 1 --2- 172.21.15.40:0/1504602508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e13a0 0x562a603f2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.010+0000 7f6b023fe740 1 --2- 172.21.15.40:0/1504602508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603dc910 0x562a603eaaf0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.010+0000 7f6b023fe740 1 -- 172.21.15.40:0/1504602508 >> 172.21.15.40:0/1504602508 conn(0x562a603e2b70 msgr2=0x562a603e0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:22.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.010+0000 7f6b023fe740 1 -- 172.21.15.40:0/1504602508 shutdown_connections 2024-03-18T22:45:22.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.010+0000 7f6b023fe740 1 -- 172.21.15.40:0/1504602508 wait complete. 2024-03-18T22:45:22.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.011+0000 7f6b023fe740 1 -- start start 2024-03-18T22:45:22.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.011+0000 7f6b023fe740 1 Processor -- start 2024-03-18T22:45:22.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.011+0000 7f6b023fe740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603dc910 0x562a603f0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.011+0000 7f6b023fe740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e13a0 0x562a603f0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.011+0000 7f6b023fe740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e1d90 0x562a603f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.011+0000 7f6b023fe740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6af81ab8d0 con 0x562a603dc910 2024-03-18T22:45:22.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.011+0000 7f6b023fe740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6af81ab200 con 0x562a603e1d90 2024-03-18T22:45:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.011+0000 7f6b023fe740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6af81a8440 con 0x562a603e13a0 2024-03-18T22:45:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.011+0000 7f6b023fe740 10 client.0 _ll_register_callbacks cb 0x7f6af0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.011+0000 7f6afd4d7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603dc910 0x562a603f0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.011+0000 7f6afd4d7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603dc910 0x562a603f0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58508/0 (socket says 172.21.15.40:58508) 2024-03-18T22:45:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.011+0000 7f6afd4d7640 1 -- 172.21.15.40:0/1215430850 learned_addr learned my addr 172.21.15.40:0/1215430850 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.011+0000 7f6afccd6640 1 --2- 172.21.15.40:0/1215430850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e13a0 0x562a603f0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.011+0000 7f6afdcd8640 1 --2- 172.21.15.40:0/1215430850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e1d90 0x562a603f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.012+0000 7f6afd4d7640 1 -- 172.21.15.40:0/1215430850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e13a0 msgr2=0x562a603f0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.012+0000 7f6afd4d7640 1 --2- 172.21.15.40:0/1215430850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e13a0 0x562a603f0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.012+0000 7f6afd4d7640 1 -- 172.21.15.40:0/1215430850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e1d90 msgr2=0x562a603f1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.012+0000 7f6afd4d7640 1 --2- 172.21.15.40:0/1215430850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e1d90 0x562a603f1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.012+0000 7f6afd4d7640 1 -- 172.21.15.40:0/1215430850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a603f23b0 con 0x562a603dc910 2024-03-18T22:45:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.012+0000 7f6afccd6640 1 --2- 172.21.15.40:0/1215430850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e13a0 0x562a603f0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.012+0000 7f6afdcd8640 1 --2- 172.21.15.40:0/1215430850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e1d90 0x562a603f1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:22.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.012+0000 7f6afd4d7640 1 --2- 172.21.15.40:0/1215430850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603dc910 0x562a603f0910 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6af404c480 tx=0x7f6af402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd19c384988306bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.012+0000 7f6ae0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.012+0000 7f6ae0ff9640 1 -- 172.21.15.40:0/1215430850 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af4045070 con 0x562a603dc910 2024-03-18T22:45:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.012+0000 7f6ae0ff9640 1 -- 172.21.15.40:0/1215430850 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6af4003c10 con 0x562a603dc910 2024-03-18T22:45:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.012+0000 7f6b023fe740 1 -- 172.21.15.40:0/1215430850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6af4005030 con 0x562a603dc910 2024-03-18T22:45:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.012+0000 7f6ae0ff9640 1 -- 172.21.15.40:0/1215430850 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6af406b050 con 0x562a603dc910 2024-03-18T22:45:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.012+0000 7f6b023fe740 10 client.16236 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.012+0000 7f6b023fe740 1 -- 172.21.15.40:0/1215430850 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562a6045d970 con 0x562a603dc910 2024-03-18T22:45:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.013+0000 7f6ae0ff9640 1 -- 172.21.15.40:0/1215430850 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6af4005030 con 0x562a603dc910 2024-03-18T22:45:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.013+0000 7f6ae0ff9640 1 -- 172.21.15.40:0/1215430850 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f6af4066030 con 0x562a603dc910 2024-03-18T22:45:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.013+0000 7f6b023fe740 10 client.16236 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.013+0000 7f6b023fe740 10 client.16236 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.013+0000 7f6b023fe740 20 client.16236 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.013+0000 7f6b023fe740 1 --2- 172.21.15.40:0/1215430850 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a603e6f00 0x562a604603e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.013+0000 7f6b023fe740 4 client.16236 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:45:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.013+0000 7f6b023fe740 1 -- 172.21.15.40:0/1215430850 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562a6034d9f0 con 0x562a603e6f00 2024-03-18T22:45:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.013+0000 7f6afccd6640 1 --2- 172.21.15.40:0/1215430850 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a603e6f00 0x562a604603e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.013+0000 7f6afccd6640 1 --2- 172.21.15.40:0/1215430850 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a603e6f00 0x562a604603e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7bcb5372d4b98cc server_cookie=b88a728130a0286b in_seq=0 out_seq=0 2024-03-18T22:45:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.013+0000 7f6ae0ff9640 10 client.16236 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.015+0000 7f6ae0ff9640 1 -- 172.21.15.40:0/1215430850 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562a6034d9f0 con 0x562a603e6f00 2024-03-18T22:45:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.015+0000 7f6ae0ff9640 10 client.16236 handle_command_reply: tid=0 2024-03-18T22:45:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.015+0000 7f6b023fe740 1 client.16236 shutdown 2024-03-18T22:45:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.015+0000 7f6b023fe740 1 -- 172.21.15.40:0/1215430850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603dc910 msgr2=0x562a603f0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.015+0000 7f6b023fe740 1 --2- 172.21.15.40:0/1215430850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603dc910 0x562a603f0910 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6af404c480 tx=0x7f6af402bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.015+0000 7f6b023fe740 1 -- 172.21.15.40:0/1215430850 shutdown_connections 2024-03-18T22:45:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.015+0000 7f6b023fe740 1 --2- 172.21.15.40:0/1215430850 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a603e6f00 0x562a604603e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.015+0000 7f6b023fe740 1 --2- 172.21.15.40:0/1215430850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a603e1d90 0x562a603f1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.015+0000 7f6ae0ff9640 0 client.16236 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.015+0000 7f6b023fe740 1 --2- 172.21.15.40:0/1215430850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a603e13a0 0x562a603f0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.015+0000 7f6b023fe740 1 --2- 172.21.15.40:0/1215430850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a603dc910 0x562a603f0910 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.015+0000 7f6b023fe740 1 -- 172.21.15.40:0/1215430850 >> 172.21.15.40:0/1215430850 conn(0x562a603e2b70 msgr2=0x562a603db2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.015+0000 7f6b023fe740 1 -- 172.21.15.40:0/1215430850 shutdown_connections 2024-03-18T22:45:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.015+0000 7f6b023fe740 1 -- 172.21.15.40:0/1215430850 wait complete. 2024-03-18T22:45:22.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.015+0000 7f6b023fe740 20 client.16236 trim_cache size 0 max 16384 2024-03-18T22:45:22.022 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:45:22.022 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:45:22.022 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "f6400119-cfa1-4064-ad42-76739338bd06", 2024-03-18T22:45:22.022 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:45:22.022 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:45:22.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.016+0000 7f6aff73b640 1 -- 172.21.15.40:0/1049403621 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ae40a2880 msgr2=0x7f6ae40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:22.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.016+0000 7f6aff73b640 1 --2- 172.21.15.40:0/1049403621 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ae40a2880 0x7f6ae40a4d40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x562a603e25f0 tx=0x7f6af0004ab0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.016+0000 7f6aff73b640 1 -- 172.21.15.40:0/1049403621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6af8101930 msgr2=0x7f6af8198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:22.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.016+0000 7f6aff73b640 1 --2- 172.21.15.40:0/1049403621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6af8101930 0x7f6af8198050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800e9f0 tx=0x7f6ae800eeb0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.016+0000 7f6aff73b640 1 -- 172.21.15.40:0/1049403621 shutdown_connections 2024-03-18T22:45:22.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.016+0000 7f6aff73b640 1 --2- 172.21.15.40:0/1049403621 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ae40a2880 0x7f6ae40a4d40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.016+0000 7f6aff73b640 1 --2- 172.21.15.40:0/1049403621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6af81022f0 0x7f6af819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.016+0000 7f6aff73b640 1 --2- 172.21.15.40:0/1049403621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6af8101930 0x7f6af8198050 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.016+0000 7f6aff73b640 1 --2- 172.21.15.40:0/1049403621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6af8100740 0x7f6af8197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.016+0000 7f6aff73b640 1 -- 172.21.15.40:0/1049403621 >> 172.21.15.40:0/1049403621 conn(0x7f6af80fbcd0 msgr2=0x7f6af8069ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:22.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.016+0000 7f6aff73b640 1 -- 172.21.15.40:0/1049403621 shutdown_connections 2024-03-18T22:45:22.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.016+0000 7f6aff73b640 1 -- 172.21.15.40:0/1049403621 wait complete. 2024-03-18T22:45:22.029 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:45:22.029 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:45:22.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:21 smithi040 ceph-mon[62643]: pgmap v432: 129 pgs: 129 active+clean; 1.9 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:45:22.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.145+0000 7f15e85f1640 1 -- 172.21.15.40:0/216654392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15e0101910 msgr2=0x7f15e0101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:22.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.145+0000 7f15e85f1640 1 --2- 172.21.15.40:0/216654392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15e0101910 0x7f15e0101d90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f15d0009900 tx=0x7f15d002bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:22.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.145+0000 7f15e85f1640 1 -- 172.21.15.40:0/216654392 shutdown_connections 2024-03-18T22:45:22.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.145+0000 7f15e85f1640 1 --2- 172.21.15.40:0/216654392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e01022d0 0x7f15e0108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.145+0000 7f15e85f1640 1 --2- 172.21.15.40:0/216654392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15e0101910 0x7f15e0101d90 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.145+0000 7f15e85f1640 1 --2- 172.21.15.40:0/216654392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15e0100720 0x7f15e0100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.145+0000 7f15e85f1640 1 -- 172.21.15.40:0/216654392 >> 172.21.15.40:0/216654392 conn(0x7f15e00fbcb0 msgr2=0x7f15e00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:22.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.145+0000 7f15e85f1640 1 -- 172.21.15.40:0/216654392 shutdown_connections 2024-03-18T22:45:22.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.145+0000 7f15e85f1640 1 -- 172.21.15.40:0/216654392 wait complete. 2024-03-18T22:45:22.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.146+0000 7f15e85f1640 1 Processor -- start 2024-03-18T22:45:22.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.146+0000 7f15e85f1640 1 -- start start 2024-03-18T22:45:22.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.146+0000 7f15e85f1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15e0100720 0x7f15e0069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.146+0000 7f15e85f1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e0101910 0x7f15e019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.146+0000 7f15e85f1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15e01022d0 0x7f15e019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.146+0000 7f15e85f1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f15e010c280 con 0x7f15e01022d0 2024-03-18T22:45:22.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.146+0000 7f15e85f1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f15e010c100 con 0x7f15e0101910 2024-03-18T22:45:22.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.146+0000 7f15e85f1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f15e010c400 con 0x7f15e0100720 2024-03-18T22:45:22.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.146+0000 7f15e6b8e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15e01022d0 0x7f15e019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.146+0000 7f15e6b8e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15e01022d0 0x7f15e019ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58532/0 (socket says 172.21.15.40:58532) 2024-03-18T22:45:22.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.146+0000 7f15e638d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15e0100720 0x7f15e0069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.146+0000 7f15e6b8e640 1 -- 172.21.15.40:0/2396754560 learned_addr learned my addr 172.21.15.40:0/2396754560 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.146+0000 7f15e5b8c640 1 --2- 172.21.15.40:0/2396754560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e0101910 0x7f15e019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.147+0000 7f15e6b8e640 1 -- 172.21.15.40:0/2396754560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15e0100720 msgr2=0x7f15e0069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.147+0000 7f15e6b8e640 1 --2- 172.21.15.40:0/2396754560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15e0100720 0x7f15e0069510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.147+0000 7f15e6b8e640 1 -- 172.21.15.40:0/2396754560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e0101910 msgr2=0x7f15e019c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.147+0000 7f15e6b8e640 1 --2- 172.21.15.40:0/2396754560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e0101910 0x7f15e019c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.147+0000 7f15e6b8e640 1 -- 172.21.15.40:0/2396754560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e019f630 con 0x7f15e01022d0 2024-03-18T22:45:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.147+0000 7f15e5b8c640 1 --2- 172.21.15.40:0/2396754560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e0101910 0x7f15e019c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.147+0000 7f15e6b8e640 1 --2- 172.21.15.40:0/2396754560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15e01022d0 0x7f15e019ed70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f15d800eb10 tx=0x7f15d800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.147+0000 7f15d77fe640 1 -- 172.21.15.40:0/2396754560 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d8004b20 con 0x7f15e01022d0 2024-03-18T22:45:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.147+0000 7f15e85f1640 1 -- 172.21.15.40:0/2396754560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e019f920 con 0x7f15e01022d0 2024-03-18T22:45:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.147+0000 7f15d77fe640 1 -- 172.21.15.40:0/2396754560 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15d8004cc0 con 0x7f15e01022d0 2024-03-18T22:45:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.147+0000 7f15d77fe640 1 -- 172.21.15.40:0/2396754560 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d8010690 con 0x7f15e01022d0 2024-03-18T22:45:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.147+0000 7f15e85f1640 1 -- 172.21.15.40:0/2396754560 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e019fdf0 con 0x7f15e01022d0 2024-03-18T22:45:22.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.149+0000 7f15d77fe640 1 -- 172.21.15.40:0/2396754560 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f15d8005020 con 0x7f15e01022d0 2024-03-18T22:45:22.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.150+0000 7f15d77fe640 1 --2- 172.21.15.40:0/2396754560 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f15cc0a2800 0x7f15cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.150+0000 7f15e638d640 1 --2- 172.21.15.40:0/2396754560 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f15cc0a2800 0x7f15cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.150+0000 7f15d77fe640 1 -- 172.21.15.40:0/2396754560 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f15d80cf3e0 con 0x7f15e01022d0 2024-03-18T22:45:22.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15eb1d8740 1 Processor -- start 2024-03-18T22:45:22.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15e638d640 1 --2- 172.21.15.40:0/2396754560 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f15cc0a2800 0x7f15cc0a4cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f15e0101770 tx=0x7f15dc005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:22.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15eb1d8740 1 -- start start 2024-03-18T22:45:22.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15eb1d8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4efa50 0x55f0ad4efe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15eb1d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f0440 0x55f0ad4f4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15eb1d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f55b0 0x55f0ad4f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15eb1d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f0ad454500 con 0x55f0ad4f0440 2024-03-18T22:45:22.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15eb1d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f0ad3fcbe0 con 0x55f0ad4f55b0 2024-03-18T22:45:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15eb1d8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f0ad4505a0 con 0x55f0ad4efa50 2024-03-18T22:45:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15e5b8c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f0440 0x55f0ad4f4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15e638d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4efa50 0x55f0ad4efe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15e5b8c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f0440 0x55f0ad4f4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58538/0 (socket says 172.21.15.40:58538) 2024-03-18T22:45:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15e5b8c640 1 -- 172.21.15.40:0/1042655956 learned_addr learned my addr 172.21.15.40:0/1042655956 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15e6b8e640 1 --2- 172.21.15.40:0/1042655956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f55b0 0x55f0ad4f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15e5b8c640 1 -- 172.21.15.40:0/1042655956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4efa50 msgr2=0x55f0ad4efe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15e5b8c640 1 --2- 172.21.15.40:0/1042655956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4efa50 0x55f0ad4efe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15e5b8c640 1 -- 172.21.15.40:0/1042655956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f55b0 msgr2=0x55f0ad4f7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15e5b8c640 1 --2- 172.21.15.40:0/1042655956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f55b0 0x55f0ad4f7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.151+0000 7f15e5b8c640 1 -- 172.21.15.40:0/1042655956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0ad41e000 con 0x55f0ad4f0440 2024-03-18T22:45:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.152+0000 7f15e5b8c640 1 --2- 172.21.15.40:0/1042655956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f0440 0x55f0ad4f4f40 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f15d0005ab0 tx=0x7f15d0005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e8bd2342109369 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.152+0000 7f15d4bfe640 1 -- 172.21.15.40:0/1042655956 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0045020 con 0x55f0ad4f0440 2024-03-18T22:45:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.152+0000 7f15d4bfe640 1 -- 172.21.15.40:0/1042655956 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15d0005040 con 0x55f0ad4f0440 2024-03-18T22:45:22.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.152+0000 7f15d4bfe640 1 -- 172.21.15.40:0/1042655956 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0004b20 con 0x55f0ad4f0440 2024-03-18T22:45:22.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.153+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1042655956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f0440 msgr2=0x55f0ad4f4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:22.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.153+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/1042655956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f0440 0x55f0ad4f4f40 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f15d0005ab0 tx=0x7f15d0005e80 comp rx=0 tx=0).stop 2024-03-18T22:45:22.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.153+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1042655956 shutdown_connections 2024-03-18T22:45:22.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.153+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/1042655956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f55b0 0x55f0ad4f7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.153+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/1042655956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f0440 0x55f0ad4f4f40 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.153+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/1042655956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4efa50 0x55f0ad4efe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.153+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1042655956 >> 172.21.15.40:0/1042655956 conn(0x55f0ad4eafc0 msgr2=0x55f0ad4eb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:22.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.153+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1042655956 shutdown_connections 2024-03-18T22:45:22.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.153+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1042655956 wait complete. 2024-03-18T22:45:22.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.153+0000 7f15eb1d8740 1 -- start start 2024-03-18T22:45:22.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.153+0000 7f15eb1d8740 1 Processor -- start 2024-03-18T22:45:22.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15eb1d8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4efa50 0x55f0ad4f9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15eb1d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f0440 0x55f0ad4fa470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15eb1d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f55b0 0x55f0ad501440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15eb1d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f0ad4f8500 con 0x55f0ad4f0440 2024-03-18T22:45:22.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15eb1d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f0ad410160 con 0x55f0ad4f55b0 2024-03-18T22:45:22.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15eb1d8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f0ad4f8680 con 0x55f0ad4efa50 2024-03-18T22:45:22.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15eb1d8740 10 client.0 _ll_register_callbacks cb 0x55f0ad44ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:22.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15e5b8c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f0440 0x55f0ad4fa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15e5b8c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f0440 0x55f0ad4fa470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58542/0 (socket says 172.21.15.40:58542) 2024-03-18T22:45:22.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15e5b8c640 1 -- 172.21.15.40:0/2133473215 learned_addr learned my addr 172.21.15.40:0/2133473215 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:22.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15e638d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4efa50 0x55f0ad4f9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15e6b8e640 1 --2- 172.21.15.40:0/2133473215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f55b0 0x55f0ad501440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15e5b8c640 1 -- 172.21.15.40:0/2133473215 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4efa50 msgr2=0x55f0ad4f9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15e5b8c640 1 --2- 172.21.15.40:0/2133473215 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4efa50 0x55f0ad4f9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15e5b8c640 1 -- 172.21.15.40:0/2133473215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f55b0 msgr2=0x55f0ad501440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15e5b8c640 1 --2- 172.21.15.40:0/2133473215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f55b0 0x55f0ad501440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15e5b8c640 1 -- 172.21.15.40:0/2133473215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0ad2327d0 con 0x55f0ad4f0440 2024-03-18T22:45:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15e5b8c640 1 --2- 172.21.15.40:0/2133473215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f0440 0x55f0ad4fa470 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f15d0006010 tx=0x7f15d0005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17fdc15bc94c52db server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.154+0000 7f15c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.155+0000 7f15c9ffb640 1 -- 172.21.15.40:0/2133473215 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0045040 con 0x55f0ad4f0440 2024-03-18T22:45:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.155+0000 7f15c9ffb640 1 -- 172.21.15.40:0/2133473215 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f15d0004c70 con 0x55f0ad4f0440 2024-03-18T22:45:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.155+0000 7f15eb1d8740 1 -- 172.21.15.40:0/2133473215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15d0005040 con 0x55f0ad4f0440 2024-03-18T22:45:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.155+0000 7f15c9ffb640 1 -- 172.21.15.40:0/2133473215 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f15d005b050 con 0x55f0ad4f0440 2024-03-18T22:45:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.155+0000 7f15eb1d8740 10 client.16248 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.155+0000 7f15eb1d8740 1 -- 172.21.15.40:0/2133473215 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f0ad1683f0 con 0x55f0ad4f0440 2024-03-18T22:45:22.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.155+0000 7f15c9ffb640 1 -- 172.21.15.40:0/2133473215 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15d0005040 con 0x55f0ad4f0440 2024-03-18T22:45:22.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.155+0000 7f15c9ffb640 1 -- 172.21.15.40:0/2133473215 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f15d0041480 con 0x55f0ad4f0440 2024-03-18T22:45:22.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.155+0000 7f15eb1d8740 10 client.16248 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:22.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.155+0000 7f15eb1d8740 10 client.16248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:22.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.155+0000 7f15eb1d8740 20 client.16248 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:22.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.155+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/2133473215 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f0ad5059d0 0x55f0ad507dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.155+0000 7f15eb1d8740 4 client.16248 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:22.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.155+0000 7f15eb1d8740 1 -- 172.21.15.40:0/2133473215 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0ad461a40 con 0x55f0ad5059d0 2024-03-18T22:45:22.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.155+0000 7f15e638d640 1 --2- 172.21.15.40:0/2133473215 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f0ad5059d0 0x55f0ad507dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.156+0000 7f15e638d640 1 --2- 172.21.15.40:0/2133473215 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f0ad5059d0 0x55f0ad507dc0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64680d3ac4da71a5 server_cookie=2b2bea2ff388f175 in_seq=0 out_seq=0 2024-03-18T22:45:22.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.156+0000 7f15c9ffb640 10 client.16248 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:22.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.157+0000 7f15c9ffb640 1 -- 172.21.15.40:0/2133473215 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f0ad461a40 con 0x55f0ad5059d0 2024-03-18T22:45:22.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.157+0000 7f15c9ffb640 10 client.16248 handle_command_reply: tid=0 2024-03-18T22:45:22.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.157+0000 7f15eb1d8740 1 client.16248 shutdown 2024-03-18T22:45:22.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.157+0000 7f15eb1d8740 1 -- 172.21.15.40:0/2133473215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f0440 msgr2=0x55f0ad4fa470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:22.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.157+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/2133473215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f0440 0x55f0ad4fa470 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f15d0006010 tx=0x7f15d0005420 comp rx=0 tx=0).stop 2024-03-18T22:45:22.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.157+0000 7f15eb1d8740 1 -- 172.21.15.40:0/2133473215 shutdown_connections 2024-03-18T22:45:22.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.157+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/2133473215 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f0ad5059d0 0x55f0ad507dc0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.157+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/2133473215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f55b0 0x55f0ad501440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.157+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/2133473215 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f0440 0x55f0ad4fa470 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.157+0000 7f15c9ffb640 0 client.16248 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:22.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.157+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/2133473215 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4efa50 0x55f0ad4f9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.157+0000 7f15eb1d8740 1 -- 172.21.15.40:0/2133473215 >> 172.21.15.40:0/2133473215 conn(0x55f0ad4eafc0 msgr2=0x55f0ad4ea1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:22.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.157+0000 7f15eb1d8740 1 -- 172.21.15.40:0/2133473215 shutdown_connections 2024-03-18T22:45:22.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.157+0000 7f15eb1d8740 1 -- 172.21.15.40:0/2133473215 wait complete. 2024-03-18T22:45:22.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.157+0000 7f15eb1d8740 20 client.16248 trim_cache size 0 max 16384 2024-03-18T22:45:22.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15eb1d8740 1 Processor -- start 2024-03-18T22:45:22.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15eb1d8740 1 -- start start 2024-03-18T22:45:22.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15eb1d8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4efa50 0x55f0ad4f9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15eb1d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f0440 0x55f0ad501400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15eb1d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f55b0 0x55f0ad4e9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15eb1d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f15e01aa3a0 con 0x55f0ad4f55b0 2024-03-18T22:45:22.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15eb1d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f15e01abfa0 con 0x55f0ad4f0440 2024-03-18T22:45:22.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15eb1d8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f15e01aaa70 con 0x55f0ad4efa50 2024-03-18T22:45:22.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15e5b8c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f0440 0x55f0ad501400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15e638d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4efa50 0x55f0ad4f9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15e6b8e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f55b0 0x55f0ad4e9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15e5b8c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f0440 0x55f0ad501400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:34568/0 (socket says 172.21.15.40:34568) 2024-03-18T22:45:22.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15e5b8c640 1 -- 172.21.15.40:0/1840179581 learned_addr learned my addr 172.21.15.40:0/1840179581 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:22.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15e5b8c640 1 -- 172.21.15.40:0/1840179581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4efa50 msgr2=0x55f0ad4f9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:22.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15e5b8c640 1 --2- 172.21.15.40:0/1840179581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4efa50 0x55f0ad4f9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15e5b8c640 1 -- 172.21.15.40:0/1840179581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f55b0 msgr2=0x55f0ad4e9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:22.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15e5b8c640 1 --2- 172.21.15.40:0/1840179581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f55b0 0x55f0ad4e9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.178+0000 7f15e5b8c640 1 -- 172.21.15.40:0/1840179581 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0ad1280f0 con 0x55f0ad4f0440 2024-03-18T22:45:22.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.179+0000 7f15e5b8c640 1 --2- 172.21.15.40:0/1840179581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f0440 0x55f0ad501400 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f15d0041850 tx=0x7f15d0005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d49a0605c59e3a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:22.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.179+0000 7f15cbfff640 1 -- 172.21.15.40:0/1840179581 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0045040 con 0x55f0ad4f0440 2024-03-18T22:45:22.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.179+0000 7f15cbfff640 1 -- 172.21.15.40:0/1840179581 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15d0041ac0 con 0x55f0ad4f0440 2024-03-18T22:45:22.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.179+0000 7f15cbfff640 1 -- 172.21.15.40:0/1840179581 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0004f00 con 0x55f0ad4f0440 2024-03-18T22:45:22.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.179+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1840179581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f0440 msgr2=0x55f0ad501400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.179+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/1840179581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f0440 0x55f0ad501400 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f15d0041850 tx=0x7f15d0005e80 comp rx=0 tx=0).stop 2024-03-18T22:45:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.179+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1840179581 shutdown_connections 2024-03-18T22:45:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.179+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/1840179581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f55b0 0x55f0ad4e9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/1840179581 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4f0440 0x55f0ad501400 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/1840179581 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4efa50 0x55f0ad4f9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1840179581 >> 172.21.15.40:0/1840179581 conn(0x55f0ad4eafc0 msgr2=0x55f0ad4f1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1840179581 shutdown_connections 2024-03-18T22:45:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1840179581 wait complete. 2024-03-18T22:45:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15eb1d8740 1 -- start start 2024-03-18T22:45:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15eb1d8740 1 Processor -- start 2024-03-18T22:45:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15eb1d8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4efa50 0x55f0ad4fed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15eb1d8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4f0440 0x55f0ad4ff280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15eb1d8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f55b0 0x55f0ad500290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15eb1d8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f15e01ab810 con 0x55f0ad4f55b0 2024-03-18T22:45:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15eb1d8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f15e01ab140 con 0x55f0ad4efa50 2024-03-18T22:45:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15eb1d8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f15e01a8380 con 0x55f0ad4f0440 2024-03-18T22:45:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15eb1d8740 10 client.0 _ll_register_callbacks cb 0x7f15d80057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15e6b8e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f55b0 0x55f0ad500290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15e5b8c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4f0440 0x55f0ad4ff280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15e638d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4efa50 0x55f0ad4fed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15e5b8c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4f0440 0x55f0ad4ff280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33780/0 (socket says 172.21.15.40:33780) 2024-03-18T22:45:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15e6b8e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f55b0 0x55f0ad500290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:58564/0 (socket says 172.21.15.40:58564) 2024-03-18T22:45:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.180+0000 7f15e5b8c640 1 -- 172.21.15.40:0/1855740289 learned_addr learned my addr 172.21.15.40:0/1855740289 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.181+0000 7f15e638d640 1 -- 172.21.15.40:0/1855740289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4f0440 msgr2=0x55f0ad4ff280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.181+0000 7f15e638d640 1 --2- 172.21.15.40:0/1855740289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4f0440 0x55f0ad4ff280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.181+0000 7f15e638d640 1 -- 172.21.15.40:0/1855740289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f55b0 msgr2=0x55f0ad500290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.181+0000 7f15e638d640 1 --2- 172.21.15.40:0/1855740289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f55b0 0x55f0ad500290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.181+0000 7f15e638d640 1 -- 172.21.15.40:0/1855740289 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0ad500880 con 0x55f0ad4efa50 2024-03-18T22:45:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.181+0000 7f15e638d640 1 --2- 172.21.15.40:0/1855740289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4efa50 0x55f0ad4fed40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x55f0ad5083a0 tx=0x7f15dc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4c55572bbc752dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:22.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.181+0000 7f15c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:45:22.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.181+0000 7f15c9ffb640 1 -- 172.21.15.40:0/1855740289 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc038060 con 0x55f0ad4efa50 2024-03-18T22:45:22.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.181+0000 7f15c9ffb640 1 -- 172.21.15.40:0/1855740289 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f15dc010b80 con 0x55f0ad4efa50 2024-03-18T22:45:22.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.181+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1855740289 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15d0041ac0 con 0x55f0ad4efa50 2024-03-18T22:45:22.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.181+0000 7f15c9ffb640 1 -- 172.21.15.40:0/1855740289 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f15dc03f050 con 0x55f0ad4efa50 2024-03-18T22:45:22.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.181+0000 7f15eb1d8740 10 client.26099 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:22.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.181+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1855740289 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f0ad56bfa0 con 0x55f0ad4efa50 2024-03-18T22:45:22.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.181+0000 7f15c9ffb640 1 -- 172.21.15.40:0/1855740289 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15d0041ac0 con 0x55f0ad4efa50 2024-03-18T22:45:22.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.182+0000 7f15c9ffb640 1 -- 172.21.15.40:0/1855740289 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f15dc032030 con 0x55f0ad4efa50 2024-03-18T22:45:22.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.182+0000 7f15eb1d8740 10 client.26099 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:22.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.182+0000 7f15eb1d8740 10 client.26099 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:22.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.182+0000 7f15eb1d8740 20 client.26099 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:22.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.182+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/1855740289 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f0ad56ca10 0x55f0ad56ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:22.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.182+0000 7f15eb1d8740 4 client.26099 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:45:22.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.182+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1855740289 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f0ad45bd90 con 0x55f0ad56ca10 2024-03-18T22:45:22.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.182+0000 7f15e5b8c640 1 --2- 172.21.15.40:0/1855740289 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f0ad56ca10 0x55f0ad56ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:22.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.182+0000 7f15e5b8c640 1 --2- 172.21.15.40:0/1855740289 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f0ad56ca10 0x55f0ad56ee30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48492a8355e8a39c server_cookie=c338523cc06013b9 in_seq=0 out_seq=0 2024-03-18T22:45:22.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.182+0000 7f15c9ffb640 10 client.26099 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:22.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.182+0000 7f15c9ffb640 1 -- 172.21.15.40:0/1855740289 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f0ad45bd90 con 0x55f0ad56ca10 2024-03-18T22:45:22.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.182+0000 7f15c9ffb640 10 client.26099 handle_command_reply: tid=0 2024-03-18T22:45:22.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.182+0000 7f15eb1d8740 1 client.26099 shutdown 2024-03-18T22:45:22.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1855740289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4efa50 msgr2=0x55f0ad4fed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:22.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/1855740289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4efa50 0x55f0ad4fed40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x55f0ad5083a0 tx=0x7f15dc004730 comp rx=0 tx=0).stop 2024-03-18T22:45:22.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1855740289 shutdown_connections 2024-03-18T22:45:22.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/1855740289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f0ad4f55b0 0x55f0ad500290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/1855740289 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f0ad4f0440 0x55f0ad4ff280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/1855740289 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55f0ad56ca10 0x55f0ad56ee30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15eb1d8740 1 --2- 172.21.15.40:0/1855740289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f0ad4efa50 0x55f0ad4fed40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15c9ffb640 0 client.26099 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:22.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1855740289 >> 172.21.15.40:0/1855740289 conn(0x55f0ad4eafc0 msgr2=0x55f0ad4ee220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:22.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1855740289 shutdown_connections 2024-03-18T22:45:22.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15eb1d8740 1 -- 172.21.15.40:0/1855740289 wait complete. 2024-03-18T22:45:22.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15eb1d8740 20 client.26099 trim_cache size 0 max 16384 2024-03-18T22:45:22.191 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:45:22.191 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:45:22.191 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:45:22.191 INFO:teuthology.orchestra.run.smithi040.stdout: "f6400119-cfa1-4064-ad42-76739338bd06": { 2024-03-18T22:45:22.191 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:45:22.191 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "f6400119-cfa1-4064-ad42-76739338bd06", 2024-03-18T22:45:22.192 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:45:22.192 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:45:22.192 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:45:22.192 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:45:22.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15e85f1640 1 -- 172.21.15.40:0/2396754560 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f15cc0a2800 msgr2=0x7f15cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:22.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15e85f1640 1 --2- 172.21.15.40:0/2396754560 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f15cc0a2800 0x7f15cc0a4cc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f15e0101770 tx=0x7f15dc005f90 comp rx=0 tx=0).stop 2024-03-18T22:45:22.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15e85f1640 1 -- 172.21.15.40:0/2396754560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15e01022d0 msgr2=0x7f15e019ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:22.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15e85f1640 1 --2- 172.21.15.40:0/2396754560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15e01022d0 0x7f15e019ed70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f15d800eb10 tx=0x7f15d800c2d0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15e85f1640 1 -- 172.21.15.40:0/2396754560 shutdown_connections 2024-03-18T22:45:22.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15e85f1640 1 --2- 172.21.15.40:0/2396754560 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f15cc0a2800 0x7f15cc0a4cc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15e85f1640 1 --2- 172.21.15.40:0/2396754560 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15e01022d0 0x7f15e019ed70 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15e85f1640 1 --2- 172.21.15.40:0/2396754560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15e0101910 0x7f15e019c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15e85f1640 1 --2- 172.21.15.40:0/2396754560 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f15e0100720 0x7f15e0069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:22.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15e85f1640 1 -- 172.21.15.40:0/2396754560 >> 172.21.15.40:0/2396754560 conn(0x7f15e00fbcb0 msgr2=0x7f15e0104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:22.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15e85f1640 1 -- 172.21.15.40:0/2396754560 shutdown_connections 2024-03-18T22:45:22.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:22.183+0000 7f15e85f1640 1 -- 172.21.15.40:0/2396754560 wait complete. 2024-03-18T22:45:22.196 INFO:tasks.cephfs.filesystem:scrub status for tag:f6400119-cfa1-4064-ad42-76739338bd06 - {'path': '/', 'tag': 'f6400119-cfa1-4064-ad42-76739338bd06', 'options': 'recursive,force'} 2024-03-18T22:45:22.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:21 smithi113 ceph-mon[66775]: pgmap v432: 129 pgs: 129 active+clean; 1.9 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:45:22.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:21 smithi103 ceph-mon[66377]: pgmap v432: 129 pgs: 129 active+clean; 1.9 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:45:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:22 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:45:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:22 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:22 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:45:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:22 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:45:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:22 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:22 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:45:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:22 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:22 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:45:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:22 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:23.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:22 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:22 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:45:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:22 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:22 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:45:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:22 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:45:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:22 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:22 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:45:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:22 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:22 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:45:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:22 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:22 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:23.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:22 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:45:23.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:22 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:23.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:22 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:45:23.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:22 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:45:23.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:22 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:22 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:45:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:22 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:22 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:45:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:22 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:22 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:24.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:23 smithi113 ceph-mon[66775]: pgmap v433: 129 pgs: 129 active+clean; 2.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:45:24.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:23 smithi103 ceph-mon[66377]: pgmap v433: 129 pgs: 129 active+clean; 2.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:45:24.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:23 smithi040 ceph-mon[62643]: pgmap v433: 129 pgs: 129 active+clean; 2.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:45:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:25 smithi113 ceph-mon[66775]: pgmap v434: 129 pgs: 129 active+clean; 2.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 23 op/s 2024-03-18T22:45:26.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:25 smithi103 ceph-mon[66377]: pgmap v434: 129 pgs: 129 active+clean; 2.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 23 op/s 2024-03-18T22:45:26.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:25 smithi040 ceph-mon[62643]: pgmap v434: 129 pgs: 129 active+clean; 2.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 23 op/s 2024-03-18T22:45:28.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:27 smithi113 ceph-mon[66775]: pgmap v435: 129 pgs: 129 active+clean; 2.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:45:28.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:27 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:45:28.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:27 smithi103 ceph-mon[66377]: pgmap v435: 129 pgs: 129 active+clean; 2.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:45:28.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:27 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:45:28.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:27 smithi040 ceph-mon[62643]: pgmap v435: 129 pgs: 129 active+clean; 2.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:45:28.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:27 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:45:29.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:28 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:45:29.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:28 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:45:29.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:28 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:45:30.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:29 smithi113 ceph-mon[66775]: pgmap v436: 129 pgs: 129 active+clean; 2.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:45:30.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:29 smithi103 ceph-mon[66377]: pgmap v436: 129 pgs: 129 active+clean; 2.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:45:30.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:29 smithi040 ceph-mon[62643]: pgmap v436: 129 pgs: 129 active+clean; 2.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:45:32.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:31 smithi113 ceph-mon[66775]: pgmap v437: 129 pgs: 129 active+clean; 2.2 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:45:32.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:31 smithi103 ceph-mon[66377]: pgmap v437: 129 pgs: 129 active+clean; 2.2 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:45:32.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:31 smithi040 ceph-mon[62643]: pgmap v437: 129 pgs: 129 active+clean; 2.2 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:45:34.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:33 smithi113 ceph-mon[66775]: pgmap v438: 129 pgs: 129 active+clean; 2.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:45:34.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:33 smithi103 ceph-mon[66377]: pgmap v438: 129 pgs: 129 active+clean; 2.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:45:34.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:33 smithi040 ceph-mon[62643]: pgmap v438: 129 pgs: 129 active+clean; 2.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:45:36.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:35 smithi113 ceph-mon[66775]: pgmap v439: 129 pgs: 129 active+clean; 2.3 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:45:36.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:35 smithi103 ceph-mon[66377]: pgmap v439: 129 pgs: 129 active+clean; 2.3 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:45:36.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:35 smithi040 ceph-mon[62643]: pgmap v439: 129 pgs: 129 active+clean; 2.3 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:45:38.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:37 smithi113 ceph-mon[66775]: pgmap v440: 129 pgs: 129 active+clean; 2.3 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:45:38.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:37 smithi103 ceph-mon[66377]: pgmap v440: 129 pgs: 129 active+clean; 2.3 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:45:38.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:37 smithi040 ceph-mon[62643]: pgmap v440: 129 pgs: 129 active+clean; 2.3 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:45:40.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:39 smithi113 ceph-mon[66775]: pgmap v441: 129 pgs: 129 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 27 MiB/s wr, 19 op/s 2024-03-18T22:45:40.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:39 smithi103 ceph-mon[66377]: pgmap v441: 129 pgs: 129 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 27 MiB/s wr, 19 op/s 2024-03-18T22:45:40.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:39 smithi040 ceph-mon[62643]: pgmap v441: 129 pgs: 129 active+clean; 2.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 27 MiB/s wr, 19 op/s 2024-03-18T22:45:42.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:41 smithi113 ceph-mon[66775]: pgmap v442: 129 pgs: 129 active+clean; 2.5 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:45:42.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:41 smithi103 ceph-mon[66377]: pgmap v442: 129 pgs: 129 active+clean; 2.5 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:45:42.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:41 smithi040 ceph-mon[62643]: pgmap v442: 129 pgs: 129 active+clean; 2.5 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:45:44.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:43 smithi113 ceph-mon[66775]: pgmap v443: 129 pgs: 129 active+clean; 2.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:45:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:43 smithi103 ceph-mon[66377]: pgmap v443: 129 pgs: 129 active+clean; 2.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:45:44.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:43 smithi040 ceph-mon[62643]: pgmap v443: 129 pgs: 129 active+clean; 2.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:45:46.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:45 smithi113 ceph-mon[66775]: pgmap v444: 129 pgs: 129 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:45:46.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:45 smithi103 ceph-mon[66377]: pgmap v444: 129 pgs: 129 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:45:46.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:45 smithi040 ceph-mon[62643]: pgmap v444: 129 pgs: 129 active+clean; 2.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:45:48.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:48 smithi113 ceph-mon[66775]: pgmap v445: 129 pgs: 129 active+clean; 2.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:45:48.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:48 smithi103 ceph-mon[66377]: pgmap v445: 129 pgs: 129 active+clean; 2.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:45:48.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:48 smithi040 ceph-mon[62643]: pgmap v445: 129 pgs: 129 active+clean; 2.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:45:50.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:50 smithi113 ceph-mon[66775]: pgmap v446: 129 pgs: 129 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:45:50.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:50 smithi103 ceph-mon[66377]: pgmap v446: 129 pgs: 129 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:45:50.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:50 smithi040 ceph-mon[62643]: pgmap v446: 129 pgs: 129 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:45:52.197 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:45:52.198 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:45:52.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.306+0000 7feb4869a640 1 -- 172.21.15.40:0/3430489831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40104c30 msgr2=0x7feb40107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.306+0000 7feb4869a640 1 --2- 172.21.15.40:0/3430489831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40104c30 0x7feb40107020 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7feb30009900 tx=0x7feb3002bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:52.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.306+0000 7feb4869a640 1 -- 172.21.15.40:0/3430489831 shutdown_connections 2024-03-18T22:45:52.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.306+0000 7feb4869a640 1 --2- 172.21.15.40:0/3430489831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40107560 0x7feb40109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.306+0000 7feb4869a640 1 --2- 172.21.15.40:0/3430489831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40104c30 0x7feb40107020 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.306+0000 7feb4869a640 1 --2- 172.21.15.40:0/3430489831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb40102300 0x7feb401046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.306+0000 7feb4869a640 1 -- 172.21.15.40:0/3430489831 >> 172.21.15.40:0/3430489831 conn(0x7feb400fbcd0 msgr2=0x7feb400fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:52.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.306+0000 7feb4869a640 1 -- 172.21.15.40:0/3430489831 shutdown_connections 2024-03-18T22:45:52.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.306+0000 7feb4869a640 1 -- 172.21.15.40:0/3430489831 wait complete. 2024-03-18T22:45:52.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.307+0000 7feb4869a640 1 Processor -- start 2024-03-18T22:45:52.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.307+0000 7feb4869a640 1 -- start start 2024-03-18T22:45:52.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.307+0000 7feb4869a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb40102300 0x7feb40197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.307+0000 7feb4869a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40104c30 0x7feb40198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.307+0000 7feb4869a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40107560 0x7feb4019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.307+0000 7feb4869a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb4010d340 con 0x7feb40104c30 2024-03-18T22:45:52.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.307+0000 7feb4869a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feb4010d1c0 con 0x7feb40107560 2024-03-18T22:45:52.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.307+0000 7feb4869a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7feb4010d4c0 con 0x7feb40102300 2024-03-18T22:45:52.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.307+0000 7feb45c35640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40104c30 0x7feb40198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.307+0000 7feb45c35640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40104c30 0x7feb40198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50620/0 (socket says 172.21.15.40:50620) 2024-03-18T22:45:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.307+0000 7feb45c35640 1 -- 172.21.15.40:0/2966896148 learned_addr learned my addr 172.21.15.40:0/2966896148 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.307+0000 7feb46c37640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40107560 0x7feb4019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.307+0000 7feb46436640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb40102300 0x7feb40197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.308+0000 7feb45c35640 1 -- 172.21.15.40:0/2966896148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb40102300 msgr2=0x7feb40197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.308+0000 7feb45c35640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb40102300 0x7feb40197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.308+0000 7feb45c35640 1 -- 172.21.15.40:0/2966896148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40107560 msgr2=0x7feb4019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.308+0000 7feb45c35640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40107560 0x7feb4019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.308+0000 7feb45c35640 1 -- 172.21.15.40:0/2966896148 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4019f6e0 con 0x7feb40104c30 2024-03-18T22:45:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.308+0000 7feb46436640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb40102300 0x7feb40197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.308+0000 7feb46c37640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40107560 0x7feb4019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:52.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.308+0000 7feb45c35640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40104c30 0x7feb40198000 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7feb30000c00 tx=0x7feb30005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:52.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.308+0000 7feb377fe640 1 -- 172.21.15.40:0/2966896148 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30043020 con 0x7feb40104c30 2024-03-18T22:45:52.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.308+0000 7feb377fe640 1 -- 172.21.15.40:0/2966896148 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb30004350 con 0x7feb40104c30 2024-03-18T22:45:52.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.308+0000 7feb4869a640 1 -- 172.21.15.40:0/2966896148 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb4019f970 con 0x7feb40104c30 2024-03-18T22:45:52.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.308+0000 7feb377fe640 1 -- 172.21.15.40:0/2966896148 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30003b10 con 0x7feb40104c30 2024-03-18T22:45:52.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.308+0000 7feb4869a640 1 -- 172.21.15.40:0/2966896148 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb4019fec0 con 0x7feb40104c30 2024-03-18T22:45:52.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.310+0000 7feb377fe640 1 -- 172.21.15.40:0/2966896148 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7feb30046020 con 0x7feb40104c30 2024-03-18T22:45:52.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.311+0000 7feb4b364740 1 Processor -- start 2024-03-18T22:45:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.311+0000 7feb4b364740 1 -- start start 2024-03-18T22:45:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.311+0000 7feb377fe640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb2c0a2800 0x7feb2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.311+0000 7feb4b364740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace8a50 0x56498ace8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.311+0000 7feb4b364740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498ace9440 0x56498acedf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.311+0000 7feb4b364740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498acee5b0 0x56498acf0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.311+0000 7feb4b364740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56498ac4d500 con 0x56498acee5b0 2024-03-18T22:45:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.311+0000 7feb4b364740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56498abf5be0 con 0x56498ace9440 2024-03-18T22:45:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.311+0000 7feb4b364740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56498ac495a0 con 0x56498ace8a50 2024-03-18T22:45:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.311+0000 7feb46c37640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb2c0a2800 0x7feb2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.311+0000 7feb377fe640 1 -- 172.21.15.40:0/2966896148 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7feb300f0fb0 con 0x7feb40104c30 2024-03-18T22:45:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.311+0000 7feb46436640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace8a50 0x56498ace8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.312+0000 7feb45c35640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498ace9440 0x56498acedf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.312+0000 7feb46436640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace8a50 0x56498ace8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:36412/0 (socket says 172.21.15.40:36412) 2024-03-18T22:45:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.312+0000 7feb46436640 1 -- 172.21.15.40:0/2098915157 learned_addr learned my addr 172.21.15.40:0/2098915157 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.312+0000 7feb46c37640 1 --2- 172.21.15.40:0/2098915157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498acee5b0 0x56498acf0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.312+0000 7feb45c35640 1 -- 172.21.15.40:0/2098915157 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace8a50 msgr2=0x56498ace8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.312+0000 7feb45c35640 1 --2- 172.21.15.40:0/2098915157 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace8a50 0x56498ace8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.312+0000 7feb45c35640 1 -- 172.21.15.40:0/2098915157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498acee5b0 msgr2=0x56498acf0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.312+0000 7feb45c35640 1 --2- 172.21.15.40:0/2098915157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498acee5b0 0x56498acf0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.312+0000 7feb45c35640 1 -- 172.21.15.40:0/2098915157 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56498ac17000 con 0x56498ace9440 2024-03-18T22:45:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.312+0000 7feb46c37640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb2c0a2800 0x7feb2c0a4cc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7feb38009b80 tx=0x7feb38011f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.312+0000 7feb45c35640 1 --2- 172.21.15.40:0/2098915157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498ace9440 0x56498acedf40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feb30005ab0 tx=0x7feb30009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16aa143c1584e63e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.312+0000 7feb34ff9640 1 -- 172.21.15.40:0/2098915157 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb300bcec0 con 0x56498ace9440 2024-03-18T22:45:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.312+0000 7feb34ff9640 1 -- 172.21.15.40:0/2098915157 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb300bdc60 con 0x56498ace9440 2024-03-18T22:45:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.312+0000 7feb34ff9640 1 -- 172.21.15.40:0/2098915157 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30046310 con 0x56498ace9440 2024-03-18T22:45:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.313+0000 7feb4b364740 1 -- 172.21.15.40:0/2098915157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498ace9440 msgr2=0x56498acedf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.313+0000 7feb4b364740 1 --2- 172.21.15.40:0/2098915157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498ace9440 0x56498acedf40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feb30005ab0 tx=0x7feb30009260 comp rx=0 tx=0).stop 2024-03-18T22:45:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.313+0000 7feb4b364740 1 -- 172.21.15.40:0/2098915157 shutdown_connections 2024-03-18T22:45:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.313+0000 7feb4b364740 1 --2- 172.21.15.40:0/2098915157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498acee5b0 0x56498acf0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.313+0000 7feb4b364740 1 --2- 172.21.15.40:0/2098915157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498ace9440 0x56498acedf40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.313+0000 7feb4b364740 1 --2- 172.21.15.40:0/2098915157 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace8a50 0x56498ace8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.313+0000 7feb4b364740 1 -- 172.21.15.40:0/2098915157 >> 172.21.15.40:0/2098915157 conn(0x56498ace3fc0 msgr2=0x56498ace43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.313+0000 7feb4b364740 1 -- 172.21.15.40:0/2098915157 shutdown_connections 2024-03-18T22:45:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.313+0000 7feb4b364740 1 -- 172.21.15.40:0/2098915157 wait complete. 2024-03-18T22:45:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.313+0000 7feb4b364740 1 -- start start 2024-03-18T22:45:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb4b364740 1 Processor -- start 2024-03-18T22:45:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb4b364740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace8a50 0x56498acf7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb4b364740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498ace9440 0x56498acf75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb4b364740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498acee5b0 0x56498acfe5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb4b364740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56498acf5730 con 0x56498ace8a50 2024-03-18T22:45:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb4b364740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56498ac09160 con 0x56498ace9440 2024-03-18T22:45:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb4b364740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56498acf58b0 con 0x56498acee5b0 2024-03-18T22:45:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb4b364740 10 client.0 _ll_register_callbacks cb 0x56498ac45a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb46436640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace8a50 0x56498acf7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb46436640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace8a50 0x56498acf7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50632/0 (socket says 172.21.15.40:50632) 2024-03-18T22:45:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb46436640 1 -- 172.21.15.40:0/3070358439 learned_addr learned my addr 172.21.15.40:0/3070358439 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb46c37640 1 --2- 172.21.15.40:0/3070358439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498acee5b0 0x56498acfe5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb45c35640 1 --2- 172.21.15.40:0/3070358439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498ace9440 0x56498acf75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb46436640 1 -- 172.21.15.40:0/3070358439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498acee5b0 msgr2=0x56498acfe5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb46436640 1 --2- 172.21.15.40:0/3070358439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498acee5b0 0x56498acfe5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb46436640 1 -- 172.21.15.40:0/3070358439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498ace9440 msgr2=0x56498acf75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb46436640 1 --2- 172.21.15.40:0/3070358439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498ace9440 0x56498acf75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb46436640 1 -- 172.21.15.40:0/3070358439 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56498aa2b7d0 con 0x56498ace8a50 2024-03-18T22:45:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb46c37640 1 --2- 172.21.15.40:0/3070358439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498acee5b0 0x56498acfe5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.314+0000 7feb45c35640 1 --2- 172.21.15.40:0/3070358439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498ace9440 0x56498acf75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.315+0000 7feb46436640 1 --2- 172.21.15.40:0/3070358439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace8a50 0x56498acf7090 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7feb3c012d20 tx=0x7feb3c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b636f5d94bb42578 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.315+0000 7feb29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.315+0000 7feb29ffb640 1 -- 172.21.15.40:0/3070358439 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c0120d0 con 0x56498ace8a50 2024-03-18T22:45:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.315+0000 7feb29ffb640 1 -- 172.21.15.40:0/3070358439 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7feb3c0047a0 con 0x56498ace8a50 2024-03-18T22:45:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.315+0000 7feb4b364740 1 -- 172.21.15.40:0/3070358439 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb300bdc60 con 0x56498ace8a50 2024-03-18T22:45:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.315+0000 7feb29ffb640 1 -- 172.21.15.40:0/3070358439 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7feb3c012270 con 0x56498ace8a50 2024-03-18T22:45:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.315+0000 7feb4b364740 10 client.16266 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.315+0000 7feb4b364740 1 -- 172.21.15.40:0/3070358439 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56498a9613f0 con 0x56498ace8a50 2024-03-18T22:45:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.315+0000 7feb29ffb640 1 -- 172.21.15.40:0/3070358439 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb300bdc60 con 0x56498ace8a50 2024-03-18T22:45:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.315+0000 7feb29ffb640 1 -- 172.21.15.40:0/3070358439 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7feb3c01c470 con 0x56498ace8a50 2024-03-18T22:45:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.315+0000 7feb4b364740 10 client.16266 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.315+0000 7feb4b364740 10 client.16266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.315+0000 7feb4b364740 20 client.16266 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.315+0000 7feb4b364740 1 --2- 172.21.15.40:0/3070358439 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56498ad02f30 0x56498ad053d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.315+0000 7feb4b364740 4 client.16266 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.316+0000 7feb4b364740 1 -- 172.21.15.40:0/3070358439 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56498ac5aa40 con 0x56498ad02f30 2024-03-18T22:45:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.316+0000 7feb45c35640 1 --2- 172.21.15.40:0/3070358439 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56498ad02f30 0x56498ad053d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.316+0000 7feb45c35640 1 --2- 172.21.15.40:0/3070358439 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56498ad02f30 0x56498ad053d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=667cbf63f6013047 server_cookie=31f1d6e5b0e0455e in_seq=0 out_seq=0 2024-03-18T22:45:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.316+0000 7feb29ffb640 10 client.16266 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.317+0000 7feb29ffb640 1 -- 172.21.15.40:0/3070358439 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56498ac5aa40 con 0x56498ad02f30 2024-03-18T22:45:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.317+0000 7feb29ffb640 10 client.16266 handle_command_reply: tid=0 2024-03-18T22:45:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.317+0000 7feb4b364740 1 client.16266 shutdown 2024-03-18T22:45:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.317+0000 7feb4b364740 1 -- 172.21.15.40:0/3070358439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace8a50 msgr2=0x56498acf7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.317+0000 7feb4b364740 1 --2- 172.21.15.40:0/3070358439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace8a50 0x56498acf7090 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7feb3c012d20 tx=0x7feb3c00f790 comp rx=0 tx=0).stop 2024-03-18T22:45:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.317+0000 7feb4b364740 1 -- 172.21.15.40:0/3070358439 shutdown_connections 2024-03-18T22:45:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.317+0000 7feb4b364740 1 --2- 172.21.15.40:0/3070358439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498acee5b0 0x56498acfe5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.317+0000 7feb4b364740 1 --2- 172.21.15.40:0/3070358439 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56498ad02f30 0x56498ad053d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.317+0000 7feb4b364740 1 --2- 172.21.15.40:0/3070358439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498ace9440 0x56498acf75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.317+0000 7feb4b364740 1 --2- 172.21.15.40:0/3070358439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace8a50 0x56498acf7090 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.317+0000 7feb29ffb640 0 client.16266 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.317+0000 7feb4b364740 1 -- 172.21.15.40:0/3070358439 >> 172.21.15.40:0/3070358439 conn(0x56498ace3fc0 msgr2=0x56498ace31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.317+0000 7feb4b364740 1 -- 172.21.15.40:0/3070358439 shutdown_connections 2024-03-18T22:45:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.317+0000 7feb4b364740 1 -- 172.21.15.40:0/3070358439 wait complete. 2024-03-18T22:45:52.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.317+0000 7feb4b364740 20 client.16266 trim_cache size 0 max 16384 2024-03-18T22:45:52.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:52 smithi113 ceph-mon[66775]: pgmap v447: 129 pgs: 129 active+clean; 2.7 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:45:52.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.336+0000 7feb4b364740 1 Processor -- start 2024-03-18T22:45:52.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.336+0000 7feb4b364740 1 -- start start 2024-03-18T22:45:52.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.336+0000 7feb4b364740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace8a50 0x56498ace7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.336+0000 7feb4b364740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace9440 0x56498acfe560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.336+0000 7feb4b364740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498acee5b0 0x56498ad0ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.336+0000 7feb4b364740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb401a8310 con 0x56498ace9440 2024-03-18T22:45:52.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.336+0000 7feb4b364740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feb401a9f10 con 0x56498acee5b0 2024-03-18T22:45:52.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.336+0000 7feb4b364740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7feb401a89e0 con 0x56498ace8a50 2024-03-18T22:45:52.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.337+0000 7feb45c35640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace9440 0x56498acfe560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.337+0000 7feb45c35640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace9440 0x56498acfe560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50646/0 (socket says 172.21.15.40:50646) 2024-03-18T22:45:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.337+0000 7feb45c35640 1 -- 172.21.15.40:0/2144865977 learned_addr learned my addr 172.21.15.40:0/2144865977 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.337+0000 7feb46436640 1 --2- 172.21.15.40:0/2144865977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace8a50 0x56498ace7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.337+0000 7feb46c37640 1 --2- 172.21.15.40:0/2144865977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498acee5b0 0x56498ad0ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.337+0000 7feb45c35640 1 -- 172.21.15.40:0/2144865977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace8a50 msgr2=0x56498ace7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.337+0000 7feb45c35640 1 --2- 172.21.15.40:0/2144865977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace8a50 0x56498ace7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.337+0000 7feb45c35640 1 -- 172.21.15.40:0/2144865977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498acee5b0 msgr2=0x56498ad0ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.337+0000 7feb45c35640 1 --2- 172.21.15.40:0/2144865977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498acee5b0 0x56498ad0ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.337+0000 7feb45c35640 1 -- 172.21.15.40:0/2144865977 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56498a9210f0 con 0x56498ace9440 2024-03-18T22:45:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.337+0000 7feb46c37640 1 --2- 172.21.15.40:0/2144865977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498acee5b0 0x56498ad0ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:45:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.337+0000 7feb45c35640 1 --2- 172.21.15.40:0/2144865977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace9440 0x56498acfe560 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7feb30005ab0 tx=0x7feb300f0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc435f74257094b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:52.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.337+0000 7feb2bfff640 1 -- 172.21.15.40:0/2144865977 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30087070 con 0x56498ace9440 2024-03-18T22:45:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.338+0000 7feb2bfff640 1 -- 172.21.15.40:0/2144865977 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb300bdc60 con 0x56498ace9440 2024-03-18T22:45:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.338+0000 7feb2bfff640 1 -- 172.21.15.40:0/2144865977 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb300bd560 con 0x56498ace9440 2024-03-18T22:45:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 -- 172.21.15.40:0/2144865977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace9440 msgr2=0x56498acfe560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 --2- 172.21.15.40:0/2144865977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace9440 0x56498acfe560 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7feb30005ab0 tx=0x7feb300f0040 comp rx=0 tx=0).stop 2024-03-18T22:45:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 -- 172.21.15.40:0/2144865977 shutdown_connections 2024-03-18T22:45:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 --2- 172.21.15.40:0/2144865977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498acee5b0 0x56498ad0ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 --2- 172.21.15.40:0/2144865977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace9440 0x56498acfe560 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 --2- 172.21.15.40:0/2144865977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace8a50 0x56498ace7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 -- 172.21.15.40:0/2144865977 >> 172.21.15.40:0/2144865977 conn(0x56498ace3fc0 msgr2=0x56498ace2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 -- 172.21.15.40:0/2144865977 shutdown_connections 2024-03-18T22:45:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 -- 172.21.15.40:0/2144865977 wait complete. 2024-03-18T22:45:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 -- start start 2024-03-18T22:45:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 Processor -- start 2024-03-18T22:45:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace8a50 0x56498acfb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace9440 0x56498acfbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498acee5b0 0x56498ad67300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feb401a9780 con 0x56498ace8a50 2024-03-18T22:45:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feb401a90b0 con 0x56498acee5b0 2024-03-18T22:45:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7feb401a62f0 con 0x56498ace9440 2024-03-18T22:45:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb4b364740 10 client.0 _ll_register_callbacks cb 0x7feb3c009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb46436640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace8a50 0x56498acfb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb46436640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace8a50 0x56498acfb560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50656/0 (socket says 172.21.15.40:50656) 2024-03-18T22:45:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb46436640 1 -- 172.21.15.40:0/750403293 learned_addr learned my addr 172.21.15.40:0/750403293 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb45c35640 1 --2- 172.21.15.40:0/750403293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace9440 0x56498acfbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.339+0000 7feb46c37640 1 --2- 172.21.15.40:0/750403293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498acee5b0 0x56498ad67300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.340+0000 7feb46436640 1 -- 172.21.15.40:0/750403293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace9440 msgr2=0x56498acfbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.340+0000 7feb46436640 1 --2- 172.21.15.40:0/750403293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace9440 0x56498acfbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.340+0000 7feb46436640 1 -- 172.21.15.40:0/750403293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498acee5b0 msgr2=0x56498ad67300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.340+0000 7feb46436640 1 --2- 172.21.15.40:0/750403293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498acee5b0 0x56498ad67300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.340+0000 7feb46436640 1 -- 172.21.15.40:0/750403293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56498ad678f0 con 0x56498ace8a50 2024-03-18T22:45:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.340+0000 7feb46436640 1 --2- 172.21.15.40:0/750403293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace8a50 0x56498acfb560 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7feb3c01c960 tx=0x7feb3c012a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d033503670e6c83 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.340+0000 7feb29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.340+0000 7feb29ffb640 1 -- 172.21.15.40:0/750403293 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c02f060 con 0x56498ace8a50 2024-03-18T22:45:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.340+0000 7feb29ffb640 1 -- 172.21.15.40:0/750403293 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7feb3c01c680 con 0x56498ace8a50 2024-03-18T22:45:52.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.340+0000 7feb4b364740 1 -- 172.21.15.40:0/750403293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb300bdc60 con 0x56498ace8a50 2024-03-18T22:45:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.340+0000 7feb29ffb640 1 -- 172.21.15.40:0/750403293 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7feb3c005110 con 0x56498ace8a50 2024-03-18T22:45:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.340+0000 7feb4b364740 10 client.16272 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.340+0000 7feb4b364740 1 -- 172.21.15.40:0/750403293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56498ad68c70 con 0x56498ace8a50 2024-03-18T22:45:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.340+0000 7feb29ffb640 1 -- 172.21.15.40:0/750403293 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb300bdc60 con 0x56498ace8a50 2024-03-18T22:45:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.341+0000 7feb29ffb640 1 -- 172.21.15.40:0/750403293 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7feb3c003ca0 con 0x56498ace8a50 2024-03-18T22:45:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.341+0000 7feb4b364740 10 client.16272 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.341+0000 7feb4b364740 10 client.16272 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.341+0000 7feb4b364740 20 client.16272 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.341+0000 7feb4b364740 1 --2- 172.21.15.40:0/750403293 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56498ad69760 0x56498ad6bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.341+0000 7feb4b364740 4 client.16272 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:45:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.341+0000 7feb4b364740 1 -- 172.21.15.40:0/750403293 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56498ac54d90 con 0x56498ad69760 2024-03-18T22:45:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.341+0000 7feb45c35640 1 --2- 172.21.15.40:0/750403293 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56498ad69760 0x56498ad6bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.341+0000 7feb29ffb640 10 client.16272 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:52.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.341+0000 7feb45c35640 1 --2- 172.21.15.40:0/750403293 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56498ad69760 0x56498ad6bb80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ea6b7438bfae1e1 server_cookie=86e493bdf5af44d5 in_seq=0 out_seq=0 2024-03-18T22:45:52.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.341+0000 7feb29ffb640 1 -- 172.21.15.40:0/750403293 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56498ac54d90 con 0x56498ad69760 2024-03-18T22:45:52.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.341+0000 7feb29ffb640 10 client.16272 handle_command_reply: tid=0 2024-03-18T22:45:52.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.341+0000 7feb4b364740 1 client.16272 shutdown 2024-03-18T22:45:52.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4b364740 1 -- 172.21.15.40:0/750403293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace8a50 msgr2=0x56498acfb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4b364740 1 --2- 172.21.15.40:0/750403293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace8a50 0x56498acfb560 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7feb3c01c960 tx=0x7feb3c012a00 comp rx=0 tx=0).stop 2024-03-18T22:45:52.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4b364740 1 -- 172.21.15.40:0/750403293 shutdown_connections 2024-03-18T22:45:52.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4b364740 1 --2- 172.21.15.40:0/750403293 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56498ad69760 0x56498ad6bb80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4b364740 1 --2- 172.21.15.40:0/750403293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56498acee5b0 0x56498ad67300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4b364740 1 --2- 172.21.15.40:0/750403293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56498ace9440 0x56498acfbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb29ffb640 0 client.16272 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:52.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4b364740 1 --2- 172.21.15.40:0/750403293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56498ace8a50 0x56498acfb560 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4b364740 1 -- 172.21.15.40:0/750403293 >> 172.21.15.40:0/750403293 conn(0x56498ace3fc0 msgr2=0x56498ace31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:52.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4b364740 1 -- 172.21.15.40:0/750403293 shutdown_connections 2024-03-18T22:45:52.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4b364740 1 -- 172.21.15.40:0/750403293 wait complete. 2024-03-18T22:45:52.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4b364740 20 client.16272 trim_cache size 0 max 16384 2024-03-18T22:45:52.349 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:45:52.349 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:45:52.349 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:45:52.349 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:45:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4869a640 1 -- 172.21.15.40:0/2966896148 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb2c0a2800 msgr2=0x7feb2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4869a640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb2c0a2800 0x7feb2c0a4cc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7feb38009b80 tx=0x7feb38011f90 comp rx=0 tx=0).stop 2024-03-18T22:45:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4869a640 1 -- 172.21.15.40:0/2966896148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40104c30 msgr2=0x7feb40198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4869a640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40104c30 0x7feb40198000 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7feb30000c00 tx=0x7feb30005dc0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4869a640 1 -- 172.21.15.40:0/2966896148 shutdown_connections 2024-03-18T22:45:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4869a640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feb2c0a2800 0x7feb2c0a4cc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4869a640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feb40107560 0x7feb4019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4869a640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb40104c30 0x7feb40198000 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.342+0000 7feb4869a640 1 --2- 172.21.15.40:0/2966896148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feb40102300 0x7feb40197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.343+0000 7feb4869a640 1 -- 172.21.15.40:0/2966896148 >> 172.21.15.40:0/2966896148 conn(0x7feb400fbcd0 msgr2=0x7feb40102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:52.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.343+0000 7feb4869a640 1 -- 172.21.15.40:0/2966896148 shutdown_connections 2024-03-18T22:45:52.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.343+0000 7feb4869a640 1 -- 172.21.15.40:0/2966896148 wait complete. 2024-03-18T22:45:52.356 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:52 smithi040 ceph-mon[62643]: pgmap v447: 129 pgs: 129 active+clean; 2.7 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:45:52.357 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:45:52.358 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:45:52.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:52 smithi103 ceph-mon[66377]: pgmap v447: 129 pgs: 129 active+clean; 2.7 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:45:52.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.462+0000 7f127ab1b640 1 -- 172.21.15.40:0/4098063374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1274100740 msgr2=0x7f1274100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.462+0000 7f127ab1b640 1 --2- 172.21.15.40:0/4098063374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1274100740 0x7f1274100b60 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1268009900 tx=0x7f126802bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.462+0000 7f127ab1b640 1 -- 172.21.15.40:0/4098063374 shutdown_connections 2024-03-18T22:45:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.462+0000 7f127ab1b640 1 --2- 172.21.15.40:0/4098063374 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12741022f0 0x7f1274108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.462+0000 7f127ab1b640 1 --2- 172.21.15.40:0/4098063374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1274101930 0x7f1274101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.462+0000 7f127ab1b640 1 --2- 172.21.15.40:0/4098063374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1274100740 0x7f1274100b60 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.462+0000 7f127ab1b640 1 -- 172.21.15.40:0/4098063374 >> 172.21.15.40:0/4098063374 conn(0x7f12740fbcb0 msgr2=0x7f12740fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.462+0000 7f127ab1b640 1 -- 172.21.15.40:0/4098063374 shutdown_connections 2024-03-18T22:45:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.462+0000 7f127ab1b640 1 -- 172.21.15.40:0/4098063374 wait complete. 2024-03-18T22:45:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.462+0000 7f127ab1b640 1 Processor -- start 2024-03-18T22:45:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.462+0000 7f127ab1b640 1 -- start start 2024-03-18T22:45:52.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.462+0000 7f127ab1b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1274100740 0x7f1274199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.462+0000 7f127ab1b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1274101930 0x7f127419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.462+0000 7f127ab1b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12741022f0 0x7f12741a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f127ab1b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f127410d3f0 con 0x7f1274100740 2024-03-18T22:45:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f127ab1b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f127410d270 con 0x7f1274101930 2024-03-18T22:45:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f127ab1b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f127410d570 con 0x7f12741022f0 2024-03-18T22:45:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f12788b7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1274100740 0x7f1274199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f12788b7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1274100740 0x7f1274199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50676/0 (socket says 172.21.15.40:50676) 2024-03-18T22:45:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f1273fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1274101930 0x7f127419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f12788b7640 1 -- 172.21.15.40:0/2530381819 learned_addr learned my addr 172.21.15.40:0/2530381819 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f12790b8640 1 --2- 172.21.15.40:0/2530381819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12741022f0 0x7f12741a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f12788b7640 1 -- 172.21.15.40:0/2530381819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12741022f0 msgr2=0x7f12741a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f12788b7640 1 --2- 172.21.15.40:0/2530381819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12741022f0 0x7f12741a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f12788b7640 1 -- 172.21.15.40:0/2530381819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1274101930 msgr2=0x7f127419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f12788b7640 1 --2- 172.21.15.40:0/2530381819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1274101930 0x7f127419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f12788b7640 1 -- 172.21.15.40:0/2530381819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12741a1920 con 0x7f1274100740 2024-03-18T22:45:52.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f1273fff640 1 --2- 172.21.15.40:0/2530381819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1274101930 0x7f127419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:52.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f12788b7640 1 --2- 172.21.15.40:0/2530381819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1274100740 0x7f1274199d10 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1268000c00 tx=0x7f1268005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:52.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f1271ffb640 1 -- 172.21.15.40:0/2530381819 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1268043020 con 0x7f1274100740 2024-03-18T22:45:52.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f127ab1b640 1 -- 172.21.15.40:0/2530381819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12741a1bb0 con 0x7f1274100740 2024-03-18T22:45:52.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f1271ffb640 1 -- 172.21.15.40:0/2530381819 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12680043d0 con 0x7f1274100740 2024-03-18T22:45:52.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.463+0000 7f1271ffb640 1 -- 172.21.15.40:0/2530381819 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1268003bb0 con 0x7f1274100740 2024-03-18T22:45:52.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.464+0000 7f127ab1b640 1 -- 172.21.15.40:0/2530381819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12741a2100 con 0x7f1274100740 2024-03-18T22:45:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.465+0000 7f127ab1b640 1 -- 172.21.15.40:0/2530381819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12741a2330 con 0x7f1274100740 2024-03-18T22:45:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.466+0000 7f1271ffb640 1 -- 172.21.15.40:0/2530381819 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1268046020 con 0x7f1274100740 2024-03-18T22:45:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.467+0000 7f1271ffb640 1 --2- 172.21.15.40:0/2530381819 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f12640a2800 0x7f12640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.467+0000 7f1271ffb640 1 -- 172.21.15.40:0/2530381819 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f12680f1000 con 0x7f1274100740 2024-03-18T22:45:52.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.467+0000 7f1273fff640 1 --2- 172.21.15.40:0/2530381819 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f12640a2800 0x7f12640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.468+0000 7f1273fff640 1 --2- 172.21.15.40:0/2530381819 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f12640a2800 0x7f12640a4cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1260009fc0 tx=0x7f1260005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:52.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.472+0000 7f1271ffb640 1 -- 172.21.15.40:0/2530381819 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f126804d020 con 0x7f1274100740 2024-03-18T22:45:52.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.775+0000 7f127ab1b640 1 -- 172.21.15.40:0/2530381819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12741a26c0 con 0x7f1274100740 2024-03-18T22:45:52.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.775+0000 7f1271ffb640 1 -- 172.21.15.40:0/2530381819 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11954 (secure 0 0 0) 0x7f12680bdcb0 con 0x7f1274100740 2024-03-18T22:45:52.776 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:45:52.776 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:44:17.725012+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24401,24388,24473,24509,24481]},"id":1}]} 2024-03-18T22:45:52.777 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 30 2024-03-18T22:45:52.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.778+0000 7f127ab1b640 1 -- 172.21.15.40:0/2530381819 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f12640a2800 msgr2=0x7f12640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.778+0000 7f127ab1b640 1 --2- 172.21.15.40:0/2530381819 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f12640a2800 0x7f12640a4cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1260009fc0 tx=0x7f1260005f90 comp rx=0 tx=0).stop 2024-03-18T22:45:52.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.778+0000 7f127ab1b640 1 -- 172.21.15.40:0/2530381819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1274100740 msgr2=0x7f1274199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.778+0000 7f127ab1b640 1 --2- 172.21.15.40:0/2530381819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1274100740 0x7f1274199d10 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1268000c00 tx=0x7f1268005e60 comp rx=0 tx=0).stop 2024-03-18T22:45:52.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.778+0000 7f127ab1b640 1 -- 172.21.15.40:0/2530381819 shutdown_connections 2024-03-18T22:45:52.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.778+0000 7f127ab1b640 1 --2- 172.21.15.40:0/2530381819 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f12640a2800 0x7f12640a4cc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.778+0000 7f127ab1b640 1 --2- 172.21.15.40:0/2530381819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f12741022f0 0x7f12741a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.778+0000 7f127ab1b640 1 --2- 172.21.15.40:0/2530381819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1274101930 0x7f127419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.778+0000 7f127ab1b640 1 --2- 172.21.15.40:0/2530381819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1274100740 0x7f1274199d10 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.778+0000 7f127ab1b640 1 -- 172.21.15.40:0/2530381819 >> 172.21.15.40:0/2530381819 conn(0x7f12740fbcb0 msgr2=0x7f1274109540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.778+0000 7f127ab1b640 1 -- 172.21.15.40:0/2530381819 shutdown_connections 2024-03-18T22:45:52.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.778+0000 7f127ab1b640 1 -- 172.21.15.40:0/2530381819 wait complete. 2024-03-18T22:45:52.791 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:45:52.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.905+0000 7f82e1297640 1 -- 172.21.15.40:0/1620737192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82dc103460 msgr2=0x7f82dc109cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.905+0000 7f82e1297640 1 --2- 172.21.15.40:0/1620737192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82dc103460 0x7f82dc109cf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009be0 tx=0x7f82d402bef0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.905+0000 7f82e1297640 1 -- 172.21.15.40:0/1620737192 shutdown_connections 2024-03-18T22:45:52.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.905+0000 7f82e1297640 1 --2- 172.21.15.40:0/1620737192 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82dc103460 0x7f82dc109cf0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.905+0000 7f82e1297640 1 --2- 172.21.15.40:0/1620737192 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82dc102aa0 0x7f82dc102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.905+0000 7f82e1297640 1 --2- 172.21.15.40:0/1620737192 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc1018b0 0x7f82dc101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.906+0000 7f82e1297640 1 -- 172.21.15.40:0/1620737192 >> 172.21.15.40:0/1620737192 conn(0x7f82dc0fce40 msgr2=0x7f82dc0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:52.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.906+0000 7f82e1297640 1 -- 172.21.15.40:0/1620737192 shutdown_connections 2024-03-18T22:45:52.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.906+0000 7f82e1297640 1 -- 172.21.15.40:0/1620737192 wait complete. 2024-03-18T22:45:52.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.906+0000 7f82e1297640 1 Processor -- start 2024-03-18T22:45:52.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.906+0000 7f82e1297640 1 -- start start 2024-03-18T22:45:52.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82e1297640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82dc1018b0 0x7f82dc0755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82e1297640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc102aa0 0x7f82dc073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82e1297640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82dc103460 0x7f82dc074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82e1297640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc10d370 con 0x7f82dc102aa0 2024-03-18T22:45:52.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82e1297640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc10d1f0 con 0x7f82dc103460 2024-03-18T22:45:52.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82e1297640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc10d4f0 con 0x7f82dc1018b0 2024-03-18T22:45:52.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82da7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc102aa0 0x7f82dc073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82daffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82dc1018b0 0x7f82dc0755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82da7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc102aa0 0x7f82dc073c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50700/0 (socket says 172.21.15.40:50700) 2024-03-18T22:45:52.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82db7fe640 1 --2- 172.21.15.40:0/1237291865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82dc103460 0x7f82dc074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82da7fc640 1 -- 172.21.15.40:0/1237291865 learned_addr learned my addr 172.21.15.40:0/1237291865 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:52.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82daffd640 1 -- 172.21.15.40:0/1237291865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82dc103460 msgr2=0x7f82dc074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82daffd640 1 --2- 172.21.15.40:0/1237291865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82dc103460 0x7f82dc074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82daffd640 1 -- 172.21.15.40:0/1237291865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc102aa0 msgr2=0x7f82dc073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82daffd640 1 --2- 172.21.15.40:0/1237291865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc102aa0 0x7f82dc073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82daffd640 1 -- 172.21.15.40:0/1237291865 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82dc0749e0 con 0x7f82dc1018b0 2024-03-18T22:45:52.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82db7fe640 1 --2- 172.21.15.40:0/1237291865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82dc103460 0x7f82dc074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:52.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82da7fc640 1 --2- 172.21.15.40:0/1237291865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc102aa0 0x7f82dc073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:52.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82daffd640 1 --2- 172.21.15.40:0/1237291865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82dc1018b0 0x7f82dc0755d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f82dc102900 tx=0x7f82cc00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:52.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82c3fff640 1 -- 172.21.15.40:0/1237291865 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82cc004440 con 0x7f82dc1018b0 2024-03-18T22:45:52.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82c3fff640 1 -- 172.21.15.40:0/1237291865 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82cc00cd70 con 0x7f82dc1018b0 2024-03-18T22:45:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82c3fff640 1 -- 172.21.15.40:0/1237291865 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82cc019480 con 0x7f82dc1018b0 2024-03-18T22:45:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.907+0000 7f82e1297640 1 -- 172.21.15.40:0/1237291865 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82dc1a1b00 con 0x7f82dc1018b0 2024-03-18T22:45:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.908+0000 7f82e1297640 1 -- 172.21.15.40:0/1237291865 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82dc1a20a0 con 0x7f82dc1018b0 2024-03-18T22:45:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.910+0000 7f82c3fff640 1 -- 172.21.15.40:0/1237291865 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f82cc003710 con 0x7f82dc1018b0 2024-03-18T22:45:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.910+0000 7f82e3e84740 1 Processor -- start 2024-03-18T22:45:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.910+0000 7f82e3e84740 1 -- start start 2024-03-18T22:45:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.910+0000 7f82e3e84740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e42f0 0x560ea42e4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.910+0000 7f82e3e84740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e4ce0 0x560ea42e97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.910+0000 7f82e3e84740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e9e50 0x560ea42ec2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.910+0000 7f82e3e84740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560ea42484b0 con 0x560ea42e42f0 2024-03-18T22:45:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.910+0000 7f82e3e84740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560ea41f0d70 con 0x560ea42e4ce0 2024-03-18T22:45:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.910+0000 7f82e3e84740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560ea4244550 con 0x560ea42e9e50 2024-03-18T22:45:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.910+0000 7f82db7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e42f0 0x560ea42e4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.910+0000 7f82db7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e42f0 0x560ea42e4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50710/0 (socket says 172.21.15.40:50710) 2024-03-18T22:45:52.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.910+0000 7f82db7fe640 1 -- 172.21.15.40:0/2428841894 learned_addr learned my addr 172.21.15.40:0/2428841894 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:52.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.911+0000 7f82daffd640 1 --2- 172.21.15.40:0/2428841894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e4ce0 0x560ea42e97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.911+0000 7f82da7fc640 1 --2- 172.21.15.40:0/2428841894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e9e50 0x560ea42ec2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.911+0000 7f82db7fe640 1 -- 172.21.15.40:0/2428841894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e9e50 msgr2=0x560ea42ec2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.911+0000 7f82db7fe640 1 --2- 172.21.15.40:0/2428841894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e9e50 0x560ea42ec2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.911+0000 7f82db7fe640 1 -- 172.21.15.40:0/2428841894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e4ce0 msgr2=0x560ea42e97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.911+0000 7f82db7fe640 1 --2- 172.21.15.40:0/2428841894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e4ce0 0x560ea42e97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.911+0000 7f82db7fe640 1 -- 172.21.15.40:0/2428841894 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ea4213210 con 0x560ea42e42f0 2024-03-18T22:45:52.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.911+0000 7f82daffd640 1 --2- 172.21.15.40:0/2428841894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e4ce0 0x560ea42e97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:45:52.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.911+0000 7f82c3fff640 1 --2- 172.21.15.40:0/1237291865 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f82c40a2880 0x7f82c40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.911+0000 7f82daffd640 1 --2- 172.21.15.40:0/1237291865 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f82c40a2880 0x7f82c40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.911+0000 7f82db7fe640 1 --2- 172.21.15.40:0/2428841894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e42f0 0x560ea42e4710 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009be0 tx=0x7f82d4005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343951ac6d5d9a3b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.911+0000 7f82c3fff640 1 -- 172.21.15.40:0/1237291865 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f82cc0cf1b0 con 0x7f82dc1018b0 2024-03-18T22:45:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.911+0000 7f82c17fa640 1 -- 172.21.15.40:0/2428841894 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4043030 con 0x560ea42e42f0 2024-03-18T22:45:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.911+0000 7f82c17fa640 1 -- 172.21.15.40:0/2428841894 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82d4003970 con 0x560ea42e42f0 2024-03-18T22:45:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.911+0000 7f82c17fa640 1 -- 172.21.15.40:0/2428841894 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d402cba0 con 0x560ea42e42f0 2024-03-18T22:45:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.912+0000 7f82e3e84740 1 -- 172.21.15.40:0/2428841894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e42f0 msgr2=0x560ea42e4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.912+0000 7f82daffd640 1 --2- 172.21.15.40:0/1237291865 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f82c40a2880 0x7f82c40a4d40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f82cc0ccc30 tx=0x7f82cc086040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.912+0000 7f82e3e84740 1 --2- 172.21.15.40:0/2428841894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e42f0 0x560ea42e4710 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009be0 tx=0x7f82d4005e30 comp rx=0 tx=0).stop 2024-03-18T22:45:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.912+0000 7f82e3e84740 1 -- 172.21.15.40:0/2428841894 shutdown_connections 2024-03-18T22:45:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.912+0000 7f82e3e84740 1 --2- 172.21.15.40:0/2428841894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e9e50 0x560ea42ec2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.912+0000 7f82e3e84740 1 --2- 172.21.15.40:0/2428841894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e4ce0 0x560ea42e97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.912+0000 7f82e3e84740 1 --2- 172.21.15.40:0/2428841894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e42f0 0x560ea42e4710 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.912+0000 7f82e3e84740 1 -- 172.21.15.40:0/2428841894 >> 172.21.15.40:0/2428841894 conn(0x560ea42df860 msgr2=0x560ea42dfc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:52.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.912+0000 7f82e3e84740 1 -- 172.21.15.40:0/2428841894 shutdown_connections 2024-03-18T22:45:52.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.912+0000 7f82e3e84740 1 -- 172.21.15.40:0/2428841894 wait complete. 2024-03-18T22:45:52.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.912+0000 7f82e3e84740 1 -- start start 2024-03-18T22:45:52.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82e3e84740 1 Processor -- start 2024-03-18T22:45:52.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82e3e84740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e42f0 0x560ea42ee800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82e3e84740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e4ce0 0x560ea42eed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82e3e84740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e9e50 0x560ea42f5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82e3e84740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560ea42ecda0 con 0x560ea42e9e50 2024-03-18T22:45:52.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82e3e84740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560ea4204310 con 0x560ea42e4ce0 2024-03-18T22:45:52.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82e3e84740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560ea42ecf20 con 0x560ea42e42f0 2024-03-18T22:45:52.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82e3e84740 10 client.0 _ll_register_callbacks cb 0x560ea41f82d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:52.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82db7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e9e50 0x560ea42f5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82db7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e42f0 0x560ea42ee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82db7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e42f0 0x560ea42ee800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:36466/0 (socket says 172.21.15.40:36466) 2024-03-18T22:45:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82db7fe640 1 -- 172.21.15.40:0/813937188 learned_addr learned my addr 172.21.15.40:0/813937188 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82da7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e4ce0 0x560ea42eed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82db7fe640 1 -- 172.21.15.40:0/813937188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e4ce0 msgr2=0x560ea42eed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82db7fe640 1 --2- 172.21.15.40:0/813937188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e4ce0 0x560ea42eed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82db7fe640 1 -- 172.21.15.40:0/813937188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e9e50 msgr2=0x560ea42f5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82db7fe640 1 --2- 172.21.15.40:0/813937188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e9e50 0x560ea42f5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82db7fe640 1 -- 172.21.15.40:0/813937188 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ea3f5c3f0 con 0x560ea42e42f0 2024-03-18T22:45:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82da7fc640 1 --2- 172.21.15.40:0/813937188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e4ce0 0x560ea42eed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82db7fe640 1 --2- 172.21.15.40:0/813937188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e42f0 0x560ea42ee800 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f82d40099a0 tx=0x7f82d402bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=652975df754f211e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.913+0000 7f82beff5640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:45:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.914+0000 7f82beff5640 1 -- 172.21.15.40:0/813937188 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d404d060 con 0x560ea42e42f0 2024-03-18T22:45:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.914+0000 7f82beff5640 1 -- 172.21.15.40:0/813937188 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f82d4033de0 con 0x560ea42e42f0 2024-03-18T22:45:52.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.914+0000 7f82e3e84740 1 -- 172.21.15.40:0/813937188 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82d4003970 con 0x560ea42e42f0 2024-03-18T22:45:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.914+0000 7f82beff5640 1 -- 172.21.15.40:0/813937188 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f82d4043040 con 0x560ea42e42f0 2024-03-18T22:45:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.914+0000 7f82e3e84740 10 client.26122 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.914+0000 7f82e3e84740 1 -- 172.21.15.40:0/813937188 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560ea40267d0 con 0x560ea42e42f0 2024-03-18T22:45:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.914+0000 7f82beff5640 1 -- 172.21.15.40:0/813937188 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82d4003970 con 0x560ea42e42f0 2024-03-18T22:45:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.914+0000 7f82beff5640 1 -- 172.21.15.40:0/813937188 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f82d4034430 con 0x560ea42e42f0 2024-03-18T22:45:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.914+0000 7f82e3e84740 10 client.26122 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.914+0000 7f82e3e84740 10 client.26122 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:45:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.914+0000 7f82e3e84740 20 client.26122 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.914+0000 7f82e3e84740 1 --2- 172.21.15.40:0/813937188 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560ea42fa250 0x560ea42fc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.914+0000 7f82e3e84740 4 client.26122 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.914+0000 7f82e3e84740 1 -- 172.21.15.40:0/813937188 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ea42a63e0 con 0x560ea42fa250 2024-03-18T22:45:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.914+0000 7f82da7fc640 1 --2- 172.21.15.40:0/813937188 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560ea42fa250 0x560ea42fc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.915+0000 7f82da7fc640 1 --2- 172.21.15.40:0/813937188 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560ea42fa250 0x560ea42fc640 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5d8129561f7e9d04 server_cookie=61e0b046c528b485 in_seq=0 out_seq=0 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.915+0000 7f82beff5640 10 client.26122 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.915+0000 7f82beff5640 1 -- 172.21.15.40:0/813937188 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560ea42a63e0 con 0x560ea42fa250 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.915+0000 7f82beff5640 10 client.26122 handle_command_reply: tid=0 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.915+0000 7f82e3e84740 1 client.26122 shutdown 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.916+0000 7f82e3e84740 1 -- 172.21.15.40:0/813937188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e42f0 msgr2=0x560ea42ee800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.916+0000 7f82e3e84740 1 --2- 172.21.15.40:0/813937188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e42f0 0x560ea42ee800 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f82d40099a0 tx=0x7f82d402bef0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.916+0000 7f82e3e84740 1 -- 172.21.15.40:0/813937188 shutdown_connections 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.916+0000 7f82e3e84740 1 --2- 172.21.15.40:0/813937188 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560ea42fa250 0x560ea42fc640 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.916+0000 7f82e3e84740 1 --2- 172.21.15.40:0/813937188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e9e50 0x560ea42f5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.916+0000 7f82e3e84740 1 --2- 172.21.15.40:0/813937188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e4ce0 0x560ea42eed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.916+0000 7f82beff5640 0 client.26122 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.916+0000 7f82e3e84740 1 --2- 172.21.15.40:0/813937188 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e42f0 0x560ea42ee800 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.916+0000 7f82e3e84740 1 -- 172.21.15.40:0/813937188 >> 172.21.15.40:0/813937188 conn(0x560ea42df860 msgr2=0x560ea42deba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.916+0000 7f82e3e84740 1 -- 172.21.15.40:0/813937188 shutdown_connections 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.916+0000 7f82e3e84740 1 -- 172.21.15.40:0/813937188 wait complete. 2024-03-18T22:45:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.916+0000 7f82e3e84740 20 client.26122 trim_cache size 0 max 16384 2024-03-18T22:45:52.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82e3e84740 1 Processor -- start 2024-03-18T22:45:52.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82e3e84740 1 -- start start 2024-03-18T22:45:52.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82e3e84740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e42f0 0x560ea42eeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82e3e84740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e4ce0 0x560ea42f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82e3e84740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e9e50 0x560ea42de310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82e3e84740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1ac5b0 con 0x560ea42e9e50 2024-03-18T22:45:52.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82e3e84740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1ae1b0 con 0x560ea42e42f0 2024-03-18T22:45:52.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82e3e84740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1acc80 con 0x560ea42e4ce0 2024-03-18T22:45:52.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82db7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e9e50 0x560ea42de310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82da7fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e4ce0 0x560ea42f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82da7fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e4ce0 0x560ea42f5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:36478/0 (socket says 172.21.15.40:36478) 2024-03-18T22:45:52.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82db7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e9e50 0x560ea42de310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50728/0 (socket says 172.21.15.40:50728) 2024-03-18T22:45:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82da7fc640 1 -- 172.21.15.40:0/3546842842 learned_addr learned my addr 172.21.15.40:0/3546842842 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82db7fe640 1 --2- 172.21.15.40:0/3546842842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e42f0 0x560ea42eeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82da7fc640 1 -- 172.21.15.40:0/3546842842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e42f0 msgr2=0x560ea42eeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82da7fc640 1 --2- 172.21.15.40:0/3546842842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e42f0 0x560ea42eeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82da7fc640 1 -- 172.21.15.40:0/3546842842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e9e50 msgr2=0x560ea42de310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82da7fc640 1 --2- 172.21.15.40:0/3546842842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e9e50 0x560ea42de310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.937+0000 7f82da7fc640 1 -- 172.21.15.40:0/3546842842 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ea42e3f30 con 0x560ea42e4ce0 2024-03-18T22:45:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.938+0000 7f82da7fc640 1 --2- 172.21.15.40:0/3546842842 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e4ce0 0x560ea42f5d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f82dc075340 tx=0x7f82d0009420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4a9ed094d6d79d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.938+0000 7f82c0ff9640 1 -- 172.21.15.40:0/3546842842 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d000fe80 con 0x560ea42e4ce0 2024-03-18T22:45:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.938+0000 7f82c0ff9640 1 -- 172.21.15.40:0/3546842842 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82d00054d0 con 0x560ea42e4ce0 2024-03-18T22:45:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.938+0000 7f82c0ff9640 1 -- 172.21.15.40:0/3546842842 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d00047a0 con 0x560ea42e4ce0 2024-03-18T22:45:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.938+0000 7f82e3e84740 1 -- 172.21.15.40:0/3546842842 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e4ce0 msgr2=0x560ea42f5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.938+0000 7f82e3e84740 1 --2- 172.21.15.40:0/3546842842 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e4ce0 0x560ea42f5d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f82dc075340 tx=0x7f82d0009420 comp rx=0 tx=0).stop 2024-03-18T22:45:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 1 -- 172.21.15.40:0/3546842842 shutdown_connections 2024-03-18T22:45:52.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 1 --2- 172.21.15.40:0/3546842842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e9e50 0x560ea42de310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 1 --2- 172.21.15.40:0/3546842842 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e4ce0 0x560ea42f5d40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 1 --2- 172.21.15.40:0/3546842842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e42f0 0x560ea42eeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 1 -- 172.21.15.40:0/3546842842 >> 172.21.15.40:0/3546842842 conn(0x560ea42df860 msgr2=0x560ea42ee410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 1 -- 172.21.15.40:0/3546842842 shutdown_connections 2024-03-18T22:45:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 1 -- 172.21.15.40:0/3546842842 wait complete. 2024-03-18T22:45:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 1 -- start start 2024-03-18T22:45:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 1 Processor -- start 2024-03-18T22:45:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e42f0 0x560ea42f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e4ce0 0x560ea42f3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e9e50 0x560ea42f4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1ada20 con 0x560ea42e4ce0 2024-03-18T22:45:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1ad350 con 0x560ea42e9e50 2024-03-18T22:45:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1aa590 con 0x560ea42e42f0 2024-03-18T22:45:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82e3e84740 10 client.0 _ll_register_callbacks cb 0x7f82d0009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:52.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82da7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e4ce0 0x560ea42f3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82db7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e42f0 0x560ea42f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82da7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e4ce0 0x560ea42f3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50744/0 (socket says 172.21.15.40:50744) 2024-03-18T22:45:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82da7fc640 1 -- 172.21.15.40:0/199498636 learned_addr learned my addr 172.21.15.40:0/199498636 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.939+0000 7f82db7fe640 1 --2- 172.21.15.40:0/199498636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e9e50 0x560ea42f4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.940+0000 7f82da7fc640 1 -- 172.21.15.40:0/199498636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e42f0 msgr2=0x560ea42f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.940+0000 7f82da7fc640 1 --2- 172.21.15.40:0/199498636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e42f0 0x560ea42f2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.940+0000 7f82da7fc640 1 -- 172.21.15.40:0/199498636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e9e50 msgr2=0x560ea42f4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.940+0000 7f82da7fc640 1 --2- 172.21.15.40:0/199498636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e9e50 0x560ea42f4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.940+0000 7f82da7fc640 1 -- 172.21.15.40:0/199498636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ea42f4a80 con 0x560ea42e4ce0 2024-03-18T22:45:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.940+0000 7f82db7fe640 1 --2- 172.21.15.40:0/199498636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e42f0 0x560ea42f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.940+0000 7f82da7fc640 1 --2- 172.21.15.40:0/199498636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e4ce0 0x560ea42f3520 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f82d00057a0 tx=0x7f82d000fe80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a02d2e476bd467a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.940+0000 7f82beff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.940+0000 7f82beff5640 1 -- 172.21.15.40:0/199498636 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d0012110 con 0x560ea42e4ce0 2024-03-18T22:45:52.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.940+0000 7f82beff5640 1 -- 172.21.15.40:0/199498636 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f82d0022170 con 0x560ea42e4ce0 2024-03-18T22:45:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.940+0000 7f82e3e84740 1 -- 172.21.15.40:0/199498636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82d00054d0 con 0x560ea42e4ce0 2024-03-18T22:45:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.940+0000 7f82beff5640 1 -- 172.21.15.40:0/199498636 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f82d0039050 con 0x560ea42e4ce0 2024-03-18T22:45:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.940+0000 7f82e3e84740 10 client.16296 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.940+0000 7f82e3e84740 1 -- 172.21.15.40:0/199498636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560ea42f5df0 con 0x560ea42e4ce0 2024-03-18T22:45:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.941+0000 7f82beff5640 1 -- 172.21.15.40:0/199498636 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82d00054d0 con 0x560ea42e4ce0 2024-03-18T22:45:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.941+0000 7f82beff5640 1 -- 172.21.15.40:0/199498636 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f82d0010430 con 0x560ea42e4ce0 2024-03-18T22:45:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.941+0000 7f82e3e84740 10 client.16296 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.941+0000 7f82e3e84740 10 client.16296 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:45:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.941+0000 7f82e3e84740 20 client.16296 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.941+0000 7f82e3e84740 1 --2- 172.21.15.40:0/199498636 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560ea4360930 0x560ea4362d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.941+0000 7f82e3e84740 4 client.16296 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:45:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.941+0000 7f82e3e84740 1 -- 172.21.15.40:0/199498636 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560ea42559f0 con 0x560ea4360930 2024-03-18T22:45:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.941+0000 7f82db7fe640 1 --2- 172.21.15.40:0/199498636 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560ea4360930 0x560ea4362d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:52.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.941+0000 7f82db7fe640 1 --2- 172.21.15.40:0/199498636 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560ea4360930 0x560ea4362d50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=df8a20d44cbcebd server_cookie=1c0a717f73196bc5 in_seq=0 out_seq=0 2024-03-18T22:45:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.941+0000 7f82beff5640 10 client.16296 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:45:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.941+0000 7f82beff5640 1 -- 172.21.15.40:0/199498636 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560ea42559f0 con 0x560ea4360930 2024-03-18T22:45:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.941+0000 7f82beff5640 10 client.16296 handle_command_reply: tid=0 2024-03-18T22:45:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.941+0000 7f82e3e84740 1 client.16296 shutdown 2024-03-18T22:45:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e3e84740 1 -- 172.21.15.40:0/199498636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e4ce0 msgr2=0x560ea42f3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e3e84740 1 --2- 172.21.15.40:0/199498636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e4ce0 0x560ea42f3520 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f82d00057a0 tx=0x7f82d000fe80 comp rx=0 tx=0).stop 2024-03-18T22:45:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e3e84740 1 -- 172.21.15.40:0/199498636 shutdown_connections 2024-03-18T22:45:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e3e84740 1 --2- 172.21.15.40:0/199498636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560ea42e9e50 0x560ea42f4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e3e84740 1 --2- 172.21.15.40:0/199498636 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560ea4360930 0x560ea4362d50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e3e84740 1 --2- 172.21.15.40:0/199498636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560ea42e4ce0 0x560ea42f3520 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e3e84740 1 --2- 172.21.15.40:0/199498636 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560ea42e42f0 0x560ea42f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82beff5640 0 client.16296 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:45:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e3e84740 1 -- 172.21.15.40:0/199498636 >> 172.21.15.40:0/199498636 conn(0x560ea42df860 msgr2=0x560ea42e2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e3e84740 1 -- 172.21.15.40:0/199498636 shutdown_connections 2024-03-18T22:45:52.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e3e84740 1 -- 172.21.15.40:0/199498636 wait complete. 2024-03-18T22:45:52.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e3e84740 20 client.16296 trim_cache size 0 max 16384 2024-03-18T22:45:52.952 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:45:52.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e1297640 1 -- 172.21.15.40:0/1237291865 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f82c40a2880 msgr2=0x7f82c40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e1297640 1 --2- 172.21.15.40:0/1237291865 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f82c40a2880 0x7f82c40a4d40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f82cc0ccc30 tx=0x7f82cc086040 comp rx=0 tx=0).stop 2024-03-18T22:45:52.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e1297640 1 -- 172.21.15.40:0/1237291865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82dc1018b0 msgr2=0x7f82dc0755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:52.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e1297640 1 --2- 172.21.15.40:0/1237291865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82dc1018b0 0x7f82dc0755d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f82dc102900 tx=0x7f82cc00ee50 comp rx=0 tx=0).stop 2024-03-18T22:45:52.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e1297640 1 -- 172.21.15.40:0/1237291865 shutdown_connections 2024-03-18T22:45:52.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e1297640 1 --2- 172.21.15.40:0/1237291865 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f82c40a2880 0x7f82c40a4d40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e1297640 1 --2- 172.21.15.40:0/1237291865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82dc103460 0x7f82dc074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e1297640 1 --2- 172.21.15.40:0/1237291865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82dc102aa0 0x7f82dc073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e1297640 1 --2- 172.21.15.40:0/1237291865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f82dc1018b0 0x7f82dc0755d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:52.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.942+0000 7f82e1297640 1 -- 172.21.15.40:0/1237291865 >> 172.21.15.40:0/1237291865 conn(0x7f82dc0fce40 msgr2=0x7f82dc105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:52.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.943+0000 7f82e1297640 1 -- 172.21.15.40:0/1237291865 shutdown_connections 2024-03-18T22:45:52.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:52.943+0000 7f82e1297640 1 -- 172.21.15.40:0/1237291865 wait complete. 2024-03-18T22:45:52.955 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:45:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.068+0000 7fd638adc640 1 -- 172.21.15.40:0/69542545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd634104c00 msgr2=0x7fd634106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.068+0000 7fd638adc640 1 --2- 172.21.15.40:0/69542545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd634104c00 0x7fd634106ff0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd624009900 tx=0x7fd62402bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.068+0000 7fd638adc640 1 -- 172.21.15.40:0/69542545 shutdown_connections 2024-03-18T22:45:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.068+0000 7fd638adc640 1 --2- 172.21.15.40:0/69542545 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd634107530 0x7fd634109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.068+0000 7fd638adc640 1 --2- 172.21.15.40:0/69542545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd634104c00 0x7fd634106ff0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.068+0000 7fd638adc640 1 --2- 172.21.15.40:0/69542545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6341022d0 0x7fd6341046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.068+0000 7fd638adc640 1 -- 172.21.15.40:0/69542545 >> 172.21.15.40:0/69542545 conn(0x7fd6340fbcc0 msgr2=0x7fd6340fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.068+0000 7fd638adc640 1 -- 172.21.15.40:0/69542545 shutdown_connections 2024-03-18T22:45:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.068+0000 7fd638adc640 1 -- 172.21.15.40:0/69542545 wait complete. 2024-03-18T22:45:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.069+0000 7fd638adc640 1 Processor -- start 2024-03-18T22:45:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.069+0000 7fd638adc640 1 -- start start 2024-03-18T22:45:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.069+0000 7fd638adc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6341022d0 0x7fd634197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.069+0000 7fd638adc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd634104c00 0x7fd634197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.069+0000 7fd638adc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd634107530 0x7fd63419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.069+0000 7fd638adc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd63410d340 con 0x7fd6341022d0 2024-03-18T22:45:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.069+0000 7fd638adc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd63410d1c0 con 0x7fd634104c00 2024-03-18T22:45:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.069+0000 7fd638adc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd63410d4c0 con 0x7fd634107530 2024-03-18T22:45:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.069+0000 7fd63259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6341022d0 0x7fd634197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.069+0000 7fd631d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd634104c00 0x7fd634197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.069+0000 7fd63259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6341022d0 0x7fd634197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50770/0 (socket says 172.21.15.40:50770) 2024-03-18T22:45:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.069+0000 7fd63259c640 1 -- 172.21.15.40:0/688353772 learned_addr learned my addr 172.21.15.40:0/688353772 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.069+0000 7fd632d9d640 1 --2- 172.21.15.40:0/688353772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd634107530 0x7fd63419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.070+0000 7fd63259c640 1 -- 172.21.15.40:0/688353772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd634107530 msgr2=0x7fd63419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.070+0000 7fd63259c640 1 --2- 172.21.15.40:0/688353772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd634107530 0x7fd63419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.070+0000 7fd63259c640 1 -- 172.21.15.40:0/688353772 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd634104c00 msgr2=0x7fd634197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.070+0000 7fd63259c640 1 --2- 172.21.15.40:0/688353772 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd634104c00 0x7fd634197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.070+0000 7fd63259c640 1 -- 172.21.15.40:0/688353772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63419f670 con 0x7fd6341022d0 2024-03-18T22:45:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.070+0000 7fd63259c640 1 --2- 172.21.15.40:0/688353772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6341022d0 0x7fd634197a50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd62800ea60 tx=0x7fd62800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.070+0000 7fd6237fe640 1 -- 172.21.15.40:0/688353772 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62800cde0 con 0x7fd6341022d0 2024-03-18T22:45:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.070+0000 7fd638adc640 1 -- 172.21.15.40:0/688353772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd63419f960 con 0x7fd6341022d0 2024-03-18T22:45:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.070+0000 7fd6237fe640 1 -- 172.21.15.40:0/688353772 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6280035f0 con 0x7fd6341022d0 2024-03-18T22:45:53.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.070+0000 7fd6237fe640 1 -- 172.21.15.40:0/688353772 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd628010690 con 0x7fd6341022d0 2024-03-18T22:45:53.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.070+0000 7fd638adc640 1 -- 172.21.15.40:0/688353772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd63419fed0 con 0x7fd6341022d0 2024-03-18T22:45:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.072+0000 7fd6237fe640 1 -- 172.21.15.40:0/688353772 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd6280037a0 con 0x7fd6341022d0 2024-03-18T22:45:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.073+0000 7fd6237fe640 1 --2- 172.21.15.40:0/688353772 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd61c0a2800 0x7fd61c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.073+0000 7fd631d9b640 1 --2- 172.21.15.40:0/688353772 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd61c0a2800 0x7fd61c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.073+0000 7fd6237fe640 1 -- 172.21.15.40:0/688353772 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd6280cf150 con 0x7fd6341022d0 2024-03-18T22:45:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.073+0000 7fd63b7a0740 1 Processor -- start 2024-03-18T22:45:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.073+0000 7fd63b7a0740 1 -- start start 2024-03-18T22:45:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.073+0000 7fd631d9b640 1 --2- 172.21.15.40:0/688353772 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd61c0a2800 0x7fd61c0a4cc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd624005ab0 tx=0x7fd624005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.073+0000 7fd63b7a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdc6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.073+0000 7fd63b7a0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdcb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd63b7a0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdcbe50 0x55a3fcdce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd63b7a0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a3fcd2a4b0 con 0x55a3fcdc6ce0 2024-03-18T22:45:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd63b7a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a3fccd2d70 con 0x55a3fcdc62f0 2024-03-18T22:45:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd63b7a0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55a3fcd26550 con 0x55a3fcdcbe50 2024-03-18T22:45:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd631d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdcb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd63259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdc6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd631d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdcb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50778/0 (socket says 172.21.15.40:50778) 2024-03-18T22:45:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd63259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdc6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36738/0 (socket says 172.21.15.40:36738) 2024-03-18T22:45:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd631d9b640 1 -- 172.21.15.40:0/931134887 learned_addr learned my addr 172.21.15.40:0/931134887 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd632d9d640 1 --2- 172.21.15.40:0/931134887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdcbe50 0x55a3fcdce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd63259c640 1 -- 172.21.15.40:0/931134887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdcbe50 msgr2=0x55a3fcdce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd63259c640 1 --2- 172.21.15.40:0/931134887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdcbe50 0x55a3fcdce2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd63259c640 1 -- 172.21.15.40:0/931134887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc6ce0 msgr2=0x55a3fcdcb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd63259c640 1 --2- 172.21.15.40:0/931134887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdcb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd63259c640 1 -- 172.21.15.40:0/931134887 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3fccf5210 con 0x55a3fcdc62f0 2024-03-18T22:45:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd631d9b640 1 --2- 172.21.15.40:0/931134887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdcb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd63259c640 1 --2- 172.21.15.40:0/931134887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdc6710 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd62800c760 tx=0x7fd62809b240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b35fdf22e8e82e4d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd620ff9640 1 -- 172.21.15.40:0/931134887 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6280b4070 con 0x55a3fcdc62f0 2024-03-18T22:45:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd620ff9640 1 -- 172.21.15.40:0/931134887 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd62809be00 con 0x55a3fcdc62f0 2024-03-18T22:45:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.074+0000 7fd620ff9640 1 -- 172.21.15.40:0/931134887 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd628097d70 con 0x55a3fcdc62f0 2024-03-18T22:45:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.075+0000 7fd63b7a0740 1 -- 172.21.15.40:0/931134887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc62f0 msgr2=0x55a3fcdc6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.075+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/931134887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdc6710 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd62800c760 tx=0x7fd62809b240 comp rx=0 tx=0).stop 2024-03-18T22:45:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.075+0000 7fd63b7a0740 1 -- 172.21.15.40:0/931134887 shutdown_connections 2024-03-18T22:45:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.075+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/931134887 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdcbe50 0x55a3fcdce2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.075+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/931134887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdcb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.075+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/931134887 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdc6710 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.075+0000 7fd63b7a0740 1 -- 172.21.15.40:0/931134887 >> 172.21.15.40:0/931134887 conn(0x55a3fcdc1860 msgr2=0x55a3fcdc1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.075+0000 7fd63b7a0740 1 -- 172.21.15.40:0/931134887 shutdown_connections 2024-03-18T22:45:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.075+0000 7fd63b7a0740 1 -- 172.21.15.40:0/931134887 wait complete. 2024-03-18T22:45:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.076+0000 7fd63b7a0740 1 -- start start 2024-03-18T22:45:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.076+0000 7fd63b7a0740 1 Processor -- start 2024-03-18T22:45:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.076+0000 7fd63b7a0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdd0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.076+0000 7fd63b7a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.076+0000 7fd63b7a0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdcbe50 0x55a3fcdd7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.076+0000 7fd63b7a0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a3fcdcee00 con 0x55a3fcdc62f0 2024-03-18T22:45:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.076+0000 7fd63b7a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a3fcce6310 con 0x55a3fcdc6ce0 2024-03-18T22:45:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.076+0000 7fd63b7a0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55a3fcdcef80 con 0x55a3fcdcbe50 2024-03-18T22:45:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.076+0000 7fd63b7a0740 10 client.0 _ll_register_callbacks cb 0x55a3fccda2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.076+0000 7fd63259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdd0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.076+0000 7fd63259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdd0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50780/0 (socket says 172.21.15.40:50780) 2024-03-18T22:45:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.076+0000 7fd632d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdcbe50 0x55a3fcdd7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.076+0000 7fd63259c640 1 -- 172.21.15.40:0/68407855 learned_addr learned my addr 172.21.15.40:0/68407855 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.076+0000 7fd631d9b640 1 --2- 172.21.15.40:0/68407855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.076+0000 7fd63259c640 1 -- 172.21.15.40:0/68407855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdcbe50 msgr2=0x55a3fcdd7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.077+0000 7fd63259c640 1 --2- 172.21.15.40:0/68407855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdcbe50 0x55a3fcdd7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.077+0000 7fd63259c640 1 -- 172.21.15.40:0/68407855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc6ce0 msgr2=0x55a3fcdd0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.077+0000 7fd63259c640 1 --2- 172.21.15.40:0/68407855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.077+0000 7fd63259c640 1 -- 172.21.15.40:0/68407855 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3fca3e3f0 con 0x55a3fcdc62f0 2024-03-18T22:45:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.077+0000 7fd631d9b640 1 --2- 172.21.15.40:0/68407855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.077+0000 7fd63259c640 1 --2- 172.21.15.40:0/68407855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdd0920 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd6280ccbd0 tx=0x7fd62809b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d1e205b7545e51f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.077+0000 7fd619ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.077+0000 7fd619ffb640 1 -- 172.21.15.40:0/68407855 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6280b4050 con 0x55a3fcdc62f0 2024-03-18T22:45:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.077+0000 7fd619ffb640 1 -- 172.21.15.40:0/68407855 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd62809ab50 con 0x55a3fcdc62f0 2024-03-18T22:45:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.077+0000 7fd63b7a0740 1 -- 172.21.15.40:0/68407855 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd62809be00 con 0x55a3fcdc62f0 2024-03-18T22:45:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.077+0000 7fd619ffb640 1 -- 172.21.15.40:0/68407855 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd628082050 con 0x55a3fcdc62f0 2024-03-18T22:45:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.077+0000 7fd63b7a0740 10 client.16308 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.077+0000 7fd63b7a0740 1 -- 172.21.15.40:0/68407855 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a3fcb087d0 con 0x55a3fcdc62f0 2024-03-18T22:45:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.077+0000 7fd619ffb640 1 -- 172.21.15.40:0/68407855 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6280a3ea0 con 0x55a3fcdc62f0 2024-03-18T22:45:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.078+0000 7fd619ffb640 1 -- 172.21.15.40:0/68407855 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fd628097450 con 0x55a3fcdc62f0 2024-03-18T22:45:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.078+0000 7fd63b7a0740 10 client.16308 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.078+0000 7fd63b7a0740 10 client.16308 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:45:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.078+0000 7fd63b7a0740 20 client.16308 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.078+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/68407855 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a3fcddc320 0x55a3fcdde710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.078+0000 7fd63b7a0740 4 client.16308 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.078+0000 7fd63b7a0740 1 -- 172.21.15.40:0/68407855 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd62809be00 con 0x55a3fcddc320 2024-03-18T22:45:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.078+0000 7fd631d9b640 1 --2- 172.21.15.40:0/68407855 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a3fcddc320 0x55a3fcdde710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.078+0000 7fd631d9b640 1 --2- 172.21.15.40:0/68407855 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a3fcddc320 0x55a3fcdde710 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=82df7538ad961d9e server_cookie=242119fd61dea27d in_seq=0 out_seq=0 2024-03-18T22:45:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.078+0000 7fd619ffb640 10 client.16308 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:45:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.079+0000 7fd619ffb640 1 -- 172.21.15.40:0/68407855 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fd62809be00 con 0x55a3fcddc320 2024-03-18T22:45:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.079+0000 7fd619ffb640 10 client.16308 handle_command_reply: tid=0 2024-03-18T22:45:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.079+0000 7fd63b7a0740 1 client.16308 shutdown 2024-03-18T22:45:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.079+0000 7fd63b7a0740 1 -- 172.21.15.40:0/68407855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc62f0 msgr2=0x55a3fcdd0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.079+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/68407855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdd0920 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd6280ccbd0 tx=0x7fd62809b770 comp rx=0 tx=0).stop 2024-03-18T22:45:53.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.079+0000 7fd63b7a0740 1 -- 172.21.15.40:0/68407855 shutdown_connections 2024-03-18T22:45:53.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.079+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/68407855 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a3fcddc320 0x55a3fcdde710 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.079+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/68407855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdcbe50 0x55a3fcdd7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.079+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/68407855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.079+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/68407855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdd0920 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.079+0000 7fd63b7a0740 1 -- 172.21.15.40:0/68407855 >> 172.21.15.40:0/68407855 conn(0x55a3fcdc1860 msgr2=0x55a3fcdc0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.079+0000 7fd619ffb640 0 client.16308 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:45:53.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.079+0000 7fd63b7a0740 1 -- 172.21.15.40:0/68407855 shutdown_connections 2024-03-18T22:45:53.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.079+0000 7fd63b7a0740 1 -- 172.21.15.40:0/68407855 wait complete. 2024-03-18T22:45:53.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.079+0000 7fd63b7a0740 20 client.16308 trim_cache size 0 max 16384 2024-03-18T22:45:53.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.093+0000 7fd63b7a0740 1 Processor -- start 2024-03-18T22:45:53.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.093+0000 7fd63b7a0740 1 -- start start 2024-03-18T22:45:53.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.093+0000 7fd63b7a0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdd0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.093+0000 7fd63b7a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.093+0000 7fd63b7a0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdcbe50 0x55a3fcdc0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.093+0000 7fd63b7a0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341aa560 con 0x55a3fcdcbe50 2024-03-18T22:45:53.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.093+0000 7fd63b7a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341ac160 con 0x55a3fcdc6ce0 2024-03-18T22:45:53.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.093+0000 7fd63b7a0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341aac30 con 0x55a3fcdc62f0 2024-03-18T22:45:53.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.093+0000 7fd632d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdcbe50 0x55a3fcdc0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.094+0000 7fd632d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdcbe50 0x55a3fcdc0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50788/0 (socket says 172.21.15.40:50788) 2024-03-18T22:45:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.094+0000 7fd632d9d640 1 -- 172.21.15.40:0/3355101318 learned_addr learned my addr 172.21.15.40:0/3355101318 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.094+0000 7fd63259c640 1 --2- 172.21.15.40:0/3355101318 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdd0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.094+0000 7fd631d9b640 1 --2- 172.21.15.40:0/3355101318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.094+0000 7fd632d9d640 1 -- 172.21.15.40:0/3355101318 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdc62f0 msgr2=0x55a3fcdd0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.094+0000 7fd632d9d640 1 --2- 172.21.15.40:0/3355101318 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdd0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.094+0000 7fd632d9d640 1 -- 172.21.15.40:0/3355101318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc6ce0 msgr2=0x55a3fcdd7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.094+0000 7fd632d9d640 1 --2- 172.21.15.40:0/3355101318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.094+0000 7fd632d9d640 1 -- 172.21.15.40:0/3355101318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3fcdcf0b0 con 0x55a3fcdcbe50 2024-03-18T22:45:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.094+0000 7fd631d9b640 1 --2- 172.21.15.40:0/3355101318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:45:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.094+0000 7fd632d9d640 1 --2- 172.21.15.40:0/3355101318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdcbe50 0x55a3fcdc0310 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd62c003b80 tx=0x7fd62c004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab138c2ff465d28c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.094+0000 7fd61bfff640 1 -- 172.21.15.40:0/3355101318 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c010430 con 0x55a3fcdcbe50 2024-03-18T22:45:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.094+0000 7fd61bfff640 1 -- 172.21.15.40:0/3355101318 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd62c010a80 con 0x55a3fcdcbe50 2024-03-18T22:45:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.094+0000 7fd61bfff640 1 -- 172.21.15.40:0/3355101318 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c01ba70 con 0x55a3fcdcbe50 2024-03-18T22:45:53.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.095+0000 7fd63b7a0740 1 -- 172.21.15.40:0/3355101318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdcbe50 msgr2=0x55a3fcdc0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.095+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/3355101318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdcbe50 0x55a3fcdc0310 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd62c003b80 tx=0x7fd62c004410 comp rx=0 tx=0).stop 2024-03-18T22:45:53.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.095+0000 7fd63b7a0740 1 -- 172.21.15.40:0/3355101318 shutdown_connections 2024-03-18T22:45:53.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.095+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/3355101318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdcbe50 0x55a3fcdc0310 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.095+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/3355101318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.095+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/3355101318 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdd0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.095+0000 7fd63b7a0740 1 -- 172.21.15.40:0/3355101318 >> 172.21.15.40:0/3355101318 conn(0x55a3fcdc1860 msgr2=0x55a3fcdd0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.095+0000 7fd63b7a0740 1 -- 172.21.15.40:0/3355101318 shutdown_connections 2024-03-18T22:45:53.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.095+0000 7fd63b7a0740 1 -- 172.21.15.40:0/3355101318 wait complete. 2024-03-18T22:45:53.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.095+0000 7fd63b7a0740 1 -- start start 2024-03-18T22:45:53.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd63b7a0740 1 Processor -- start 2024-03-18T22:45:53.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd63b7a0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdd4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd63b7a0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd63b7a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdcbe50 0x55a3fce40510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd63b7a0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341ab9d0 con 0x55a3fcdc6ce0 2024-03-18T22:45:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd63b7a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341ab300 con 0x55a3fcdcbe50 2024-03-18T22:45:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd63b7a0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341a8540 con 0x55a3fcdc62f0 2024-03-18T22:45:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd63b7a0740 10 client.0 _ll_register_callbacks cb 0x7fd6280b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd631d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd631d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50802/0 (socket says 172.21.15.40:50802) 2024-03-18T22:45:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd631d9b640 1 -- 172.21.15.40:0/107998686 learned_addr learned my addr 172.21.15.40:0/107998686 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd632d9d640 1 --2- 172.21.15.40:0/107998686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdcbe50 0x55a3fce40510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd63259c640 1 --2- 172.21.15.40:0/107998686 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdd4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd631d9b640 1 -- 172.21.15.40:0/107998686 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdc62f0 msgr2=0x55a3fcdd4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd631d9b640 1 --2- 172.21.15.40:0/107998686 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdd4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd631d9b640 1 -- 172.21.15.40:0/107998686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdcbe50 msgr2=0x55a3fce40510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd631d9b640 1 --2- 172.21.15.40:0/107998686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdcbe50 0x55a3fce40510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd631d9b640 1 -- 172.21.15.40:0/107998686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3fcdd8140 con 0x55a3fcdc6ce0 2024-03-18T22:45:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd632d9d640 1 --2- 172.21.15.40:0/107998686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdcbe50 0x55a3fce40510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:45:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd63259c640 1 --2- 172.21.15.40:0/107998686 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdd4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:45:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd631d9b640 1 --2- 172.21.15.40:0/107998686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd5130 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd62402c170 tx=0x7fd62402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=747892ac5dea6b14 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.096+0000 7fd619ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.097+0000 7fd619ffb640 1 -- 172.21.15.40:0/107998686 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd624005450 con 0x55a3fcdc6ce0 2024-03-18T22:45:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.097+0000 7fd619ffb640 1 -- 172.21.15.40:0/107998686 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd624053070 con 0x55a3fcdc6ce0 2024-03-18T22:45:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.097+0000 7fd63b7a0740 1 -- 172.21.15.40:0/107998686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd62c010a80 con 0x55a3fcdc6ce0 2024-03-18T22:45:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.097+0000 7fd619ffb640 1 -- 172.21.15.40:0/107998686 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd624061050 con 0x55a3fcdc6ce0 2024-03-18T22:45:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.097+0000 7fd63b7a0740 10 client.16314 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.097+0000 7fd63b7a0740 1 -- 172.21.15.40:0/107998686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a3fce41a90 con 0x55a3fcdc6ce0 2024-03-18T22:45:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.097+0000 7fd619ffb640 1 -- 172.21.15.40:0/107998686 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd62c010a80 con 0x55a3fcdc6ce0 2024-03-18T22:45:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.097+0000 7fd619ffb640 1 -- 172.21.15.40:0/107998686 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fd624004be0 con 0x55a3fcdc6ce0 2024-03-18T22:45:53.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.097+0000 7fd63b7a0740 10 client.16314 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.097+0000 7fd63b7a0740 10 client.16314 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:45:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.097+0000 7fd63b7a0740 20 client.16314 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.097+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/107998686 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a3fce42440 0x55a3fce44860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.097+0000 7fd63b7a0740 4 client.16314 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:45:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.097+0000 7fd63b7a0740 1 -- 172.21.15.40:0/107998686 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a3fcd883e0 con 0x55a3fce42440 2024-03-18T22:45:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd63259c640 1 --2- 172.21.15.40:0/107998686 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a3fce42440 0x55a3fce44860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd63259c640 1 --2- 172.21.15.40:0/107998686 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a3fce42440 0x55a3fce44860 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=31edc45f9a38589e server_cookie=8d645be53a0b1bc5 in_seq=0 out_seq=0 2024-03-18T22:45:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd619ffb640 10 client.16314 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:45:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd619ffb640 1 -- 172.21.15.40:0/107998686 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a3fcd883e0 con 0x55a3fce42440 2024-03-18T22:45:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd619ffb640 10 client.16314 handle_command_reply: tid=0 2024-03-18T22:45:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd63b7a0740 1 client.16314 shutdown 2024-03-18T22:45:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd63b7a0740 1 -- 172.21.15.40:0/107998686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc6ce0 msgr2=0x55a3fcdd5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/107998686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd5130 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd62402c170 tx=0x7fd62402bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:53.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd63b7a0740 1 -- 172.21.15.40:0/107998686 shutdown_connections 2024-03-18T22:45:53.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/107998686 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55a3fce42440 0x55a3fce44860 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/107998686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a3fcdcbe50 0x55a3fce40510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/107998686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3fcdc6ce0 0x55a3fcdd5130 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd619ffb640 0 client.16314 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:45:53.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd63b7a0740 1 --2- 172.21.15.40:0/107998686 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a3fcdc62f0 0x55a3fcdd4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd63b7a0740 1 -- 172.21.15.40:0/107998686 >> 172.21.15.40:0/107998686 conn(0x55a3fcdc1860 msgr2=0x55a3fcdc5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd63b7a0740 1 -- 172.21.15.40:0/107998686 shutdown_connections 2024-03-18T22:45:53.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd63b7a0740 1 -- 172.21.15.40:0/107998686 wait complete. 2024-03-18T22:45:53.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.098+0000 7fd63b7a0740 20 client.16314 trim_cache size 0 max 16384 2024-03-18T22:45:53.105 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:45:53.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.099+0000 7fd638adc640 1 -- 172.21.15.40:0/688353772 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd61c0a2800 msgr2=0x7fd61c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.099+0000 7fd638adc640 1 --2- 172.21.15.40:0/688353772 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd61c0a2800 0x7fd61c0a4cc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd624005ab0 tx=0x7fd624005fd0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.099+0000 7fd638adc640 1 -- 172.21.15.40:0/688353772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6341022d0 msgr2=0x7fd634197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.099+0000 7fd638adc640 1 --2- 172.21.15.40:0/688353772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6341022d0 0x7fd634197a50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd62800ea60 tx=0x7fd62800ef20 comp rx=0 tx=0).stop 2024-03-18T22:45:53.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.099+0000 7fd638adc640 1 -- 172.21.15.40:0/688353772 shutdown_connections 2024-03-18T22:45:53.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.099+0000 7fd638adc640 1 --2- 172.21.15.40:0/688353772 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd61c0a2800 0x7fd61c0a4cc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.099+0000 7fd638adc640 1 --2- 172.21.15.40:0/688353772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd634107530 0x7fd63419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.099+0000 7fd638adc640 1 --2- 172.21.15.40:0/688353772 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd634104c00 0x7fd634197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.099+0000 7fd638adc640 1 --2- 172.21.15.40:0/688353772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6341022d0 0x7fd634197a50 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.099+0000 7fd638adc640 1 -- 172.21.15.40:0/688353772 >> 172.21.15.40:0/688353772 conn(0x7fd6340fbcc0 msgr2=0x7fd634102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.099+0000 7fd638adc640 1 -- 172.21.15.40:0/688353772 shutdown_connections 2024-03-18T22:45:53.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.099+0000 7fd638adc640 1 -- 172.21.15.40:0/688353772 wait complete. 2024-03-18T22:45:53.112 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:53 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2530381819' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:45:53.113 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:45:53.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.232+0000 7fb9df092640 1 -- 172.21.15.40:0/3407488622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9d8107560 msgr2=0x7fb9d8109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.232+0000 7fb9df092640 1 --2- 172.21.15.40:0/3407488622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9d8107560 0x7fb9d8109950 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009ca0 tx=0x7fb9d002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.232+0000 7fb9df092640 1 -- 172.21.15.40:0/3407488622 shutdown_connections 2024-03-18T22:45:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.232+0000 7fb9df092640 1 --2- 172.21.15.40:0/3407488622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9d8107560 0x7fb9d8109950 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.232+0000 7fb9df092640 1 --2- 172.21.15.40:0/3407488622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9d8104c30 0x7fb9d8107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.232+0000 7fb9df092640 1 --2- 172.21.15.40:0/3407488622 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9d8102300 0x7fb9d81046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.232+0000 7fb9df092640 1 -- 172.21.15.40:0/3407488622 >> 172.21.15.40:0/3407488622 conn(0x7fb9d80fbc90 msgr2=0x7fb9d80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.232+0000 7fb9df092640 1 -- 172.21.15.40:0/3407488622 shutdown_connections 2024-03-18T22:45:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.232+0000 7fb9df092640 1 -- 172.21.15.40:0/3407488622 wait complete. 2024-03-18T22:45:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9df092640 1 Processor -- start 2024-03-18T22:45:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9df092640 1 -- start start 2024-03-18T22:45:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9df092640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9d8102300 0x7fb9d8197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9df092640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9d8104c30 0x7fb9d8197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9df092640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9d8107560 0x7fb9d819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9df092640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d810d350 con 0x7fb9d8107560 2024-03-18T22:45:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9df092640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d810d1d0 con 0x7fb9d8104c30 2024-03-18T22:45:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9df092640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d810d4d0 con 0x7fb9d8102300 2024-03-18T22:45:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9dce2e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9d8102300 0x7fb9d8197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9d7fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9d8104c30 0x7fb9d8197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9dd62f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9d8107560 0x7fb9d819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9d7fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9d8104c30 0x7fb9d8197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36788/0 (socket says 172.21.15.40:36788) 2024-03-18T22:45:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9dce2e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9d8102300 0x7fb9d8197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:36540/0 (socket says 172.21.15.40:36540) 2024-03-18T22:45:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9d7fff640 1 -- 172.21.15.40:0/123216632 learned_addr learned my addr 172.21.15.40:0/123216632 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9d7fff640 1 -- 172.21.15.40:0/123216632 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9d8102300 msgr2=0x7fb9d8197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9d7fff640 1 --2- 172.21.15.40:0/123216632 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9d8102300 0x7fb9d8197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9d7fff640 1 -- 172.21.15.40:0/123216632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9d8107560 msgr2=0x7fb9d819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9d7fff640 1 --2- 172.21.15.40:0/123216632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9d8107560 0x7fb9d819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.233+0000 7fb9d7fff640 1 -- 172.21.15.40:0/123216632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d819f630 con 0x7fb9d8104c30 2024-03-18T22:45:53.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.234+0000 7fb9d7fff640 1 --2- 172.21.15.40:0/123216632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9d8104c30 0x7fb9d8197fa0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400e9c0 tx=0x7fb9c400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.234+0000 7fb9d5ffb640 1 -- 172.21.15.40:0/123216632 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c400cd80 con 0x7fb9d8104c30 2024-03-18T22:45:53.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.234+0000 7fb9df092640 1 -- 172.21.15.40:0/123216632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9d819f920 con 0x7fb9d8104c30 2024-03-18T22:45:53.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.234+0000 7fb9d5ffb640 1 -- 172.21.15.40:0/123216632 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9c40054d0 con 0x7fb9d8104c30 2024-03-18T22:45:53.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.234+0000 7fb9d5ffb640 1 -- 172.21.15.40:0/123216632 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4010430 con 0x7fb9d8104c30 2024-03-18T22:45:53.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.234+0000 7fb9df092640 1 -- 172.21.15.40:0/123216632 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9d819fec0 con 0x7fb9d8104c30 2024-03-18T22:45:53.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9d5ffb640 1 -- 172.21.15.40:0/123216632 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb9c4004fd0 con 0x7fb9d8104c30 2024-03-18T22:45:53.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9e1c7d740 1 Processor -- start 2024-03-18T22:45:53.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9e1c7d740 1 -- start start 2024-03-18T22:45:53.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9d5ffb640 1 --2- 172.21.15.40:0/123216632 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9c80a2800 0x7fb9c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9e1c7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234adbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9e1c7d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234adc0f0 0x563234ae0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9e1c7d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234ae1260 0x563234ae36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9e1c7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563234a3b4b0 con 0x563234adb700 2024-03-18T22:45:53.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9e1c7d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5632349e3d70 con 0x563234ae1260 2024-03-18T22:45:53.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9e1c7d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563234a37550 con 0x563234adc0f0 2024-03-18T22:45:53.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9dd62f640 1 --2- 172.21.15.40:0/123216632 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9c80a2800 0x7fb9c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9d5ffb640 1 -- 172.21.15.40:0/123216632 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb9c40cf220 con 0x7fb9d8104c30 2024-03-18T22:45:53.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9d7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234adc0f0 0x563234ae0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9dce2e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234adbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9d7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234adc0f0 0x563234ae0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:36544/0 (socket says 172.21.15.40:36544) 2024-03-18T22:45:53.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9dd62f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234ae1260 0x563234ae36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9dce2e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234adbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50832/0 (socket says 172.21.15.40:50832) 2024-03-18T22:45:53.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.237+0000 7fb9d7fff640 1 -- 172.21.15.40:0/3377789865 learned_addr learned my addr 172.21.15.40:0/3377789865 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.238+0000 7fb9d7fff640 1 -- 172.21.15.40:0/3377789865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234ae1260 msgr2=0x563234ae36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.238+0000 7fb9d7fff640 1 --2- 172.21.15.40:0/3377789865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234ae1260 0x563234ae36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.238+0000 7fb9d7fff640 1 -- 172.21.15.40:0/3377789865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 msgr2=0x563234adbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.238+0000 7fb9d7fff640 1 --2- 172.21.15.40:0/3377789865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234adbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.238+0000 7fb9d7fff640 1 -- 172.21.15.40:0/3377789865 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563234a06210 con 0x563234adc0f0 2024-03-18T22:45:53.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.238+0000 7fb9dd62f640 1 --2- 172.21.15.40:0/123216632 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9c80a2800 0x7fb9c80a4cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0006010 tx=0x7fb9d0005f60 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.238+0000 7fb9d7fff640 1 --2- 172.21.15.40:0/3377789865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234adc0f0 0x563234ae0bf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4097c30 tx=0x7fb9c4004780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d584f32440dd618f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.238+0000 7fb9c2ffd640 1 -- 172.21.15.40:0/3377789865 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c409b310 con 0x563234adc0f0 2024-03-18T22:45:53.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.238+0000 7fb9c2ffd640 1 -- 172.21.15.40:0/3377789865 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9c409bed0 con 0x563234adc0f0 2024-03-18T22:45:53.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.238+0000 7fb9c2ffd640 1 -- 172.21.15.40:0/3377789865 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c409b130 con 0x563234adc0f0 2024-03-18T22:45:53.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.239+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3377789865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234adc0f0 msgr2=0x563234ae0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.239+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3377789865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234adc0f0 0x563234ae0bf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb9c4097c30 tx=0x7fb9c4004780 comp rx=0 tx=0).stop 2024-03-18T22:45:53.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.239+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3377789865 shutdown_connections 2024-03-18T22:45:53.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.239+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3377789865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234ae1260 0x563234ae36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.239+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3377789865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234adc0f0 0x563234ae0bf0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.239+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3377789865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234adbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.239+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3377789865 >> 172.21.15.40:0/3377789865 conn(0x563234ad2860 msgr2=0x563234ad2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.239+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3377789865 shutdown_connections 2024-03-18T22:45:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.239+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3377789865 wait complete. 2024-03-18T22:45:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.239+0000 7fb9e1c7d740 1 -- start start 2024-03-18T22:45:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.239+0000 7fb9e1c7d740 1 Processor -- start 2024-03-18T22:45:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9e1c7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234ae5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9e1c7d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adc0f0 0x563234ae6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9e1c7d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234ae1260 0x563234aed0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9e1c7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563234ae4210 con 0x563234adb700 2024-03-18T22:45:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9e1c7d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5632349f7310 con 0x563234adc0f0 2024-03-18T22:45:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9e1c7d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563234ae4390 con 0x563234ae1260 2024-03-18T22:45:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9dce2e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234ae5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9e1c7d740 10 client.0 _ll_register_callbacks cb 0x5632349eb2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:53.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9dce2e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234ae5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50848/0 (socket says 172.21.15.40:50848) 2024-03-18T22:45:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9dce2e640 1 -- 172.21.15.40:0/3638673901 learned_addr learned my addr 172.21.15.40:0/3638673901 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9d7fff640 1 --2- 172.21.15.40:0/3638673901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adc0f0 0x563234ae6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9dd62f640 1 --2- 172.21.15.40:0/3638673901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234ae1260 0x563234aed0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9dce2e640 1 -- 172.21.15.40:0/3638673901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234ae1260 msgr2=0x563234aed0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9dce2e640 1 --2- 172.21.15.40:0/3638673901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234ae1260 0x563234aed0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9dce2e640 1 -- 172.21.15.40:0/3638673901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adc0f0 msgr2=0x563234ae6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9dce2e640 1 --2- 172.21.15.40:0/3638673901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adc0f0 0x563234ae6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9dce2e640 1 -- 172.21.15.40:0/3638673901 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56323474f3f0 con 0x563234adb700 2024-03-18T22:45:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9d7fff640 1 --2- 172.21.15.40:0/3638673901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adc0f0 0x563234ae6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.240+0000 7fb9dce2e640 1 --2- 172.21.15.40:0/3638673901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234ae5b50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc012700 tx=0x7fb9cc012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ab2d3b73a13bf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.241+0000 7fb9b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.241+0000 7fb9b7fff640 1 -- 172.21.15.40:0/3638673901 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc00f8c0 con 0x563234adb700 2024-03-18T22:45:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.241+0000 7fb9b7fff640 1 -- 172.21.15.40:0/3638673901 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb9cc00fa60 con 0x563234adb700 2024-03-18T22:45:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.241+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3638673901 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9c409bed0 con 0x563234adb700 2024-03-18T22:45:53.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.241+0000 7fb9b7fff640 1 -- 172.21.15.40:0/3638673901 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb9cc025050 con 0x563234adb700 2024-03-18T22:45:53.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.241+0000 7fb9e1c7d740 10 client.16326 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:53.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.241+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3638673901 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5632348197d0 con 0x563234adb700 2024-03-18T22:45:53.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.241+0000 7fb9b7fff640 1 -- 172.21.15.40:0/3638673901 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9c409bed0 con 0x563234adb700 2024-03-18T22:45:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.241+0000 7fb9b7fff640 1 -- 172.21.15.40:0/3638673901 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fb9cc010430 con 0x563234adb700 2024-03-18T22:45:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.241+0000 7fb9e1c7d740 10 client.16326 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.241+0000 7fb9e1c7d740 10 client.16326 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:45:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.241+0000 7fb9e1c7d740 20 client.16326 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.241+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3638673901 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x563234af1a60 0x563234af3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.241+0000 7fb9e1c7d740 4 client.16326 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.242+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3638673901 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563234a993e0 con 0x563234af1a60 2024-03-18T22:45:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.242+0000 7fb9d7fff640 1 --2- 172.21.15.40:0/3638673901 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x563234af1a60 0x563234af3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.242+0000 7fb9b7fff640 10 client.16326 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:45:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.242+0000 7fb9d7fff640 1 --2- 172.21.15.40:0/3638673901 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x563234af1a60 0x563234af3ee0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cc889c8d879d0bb1 server_cookie=8b1b13a690e43bdc in_seq=0 out_seq=0 2024-03-18T22:45:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.243+0000 7fb9b7fff640 1 -- 172.21.15.40:0/3638673901 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563234a993e0 con 0x563234af1a60 2024-03-18T22:45:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.243+0000 7fb9b7fff640 10 client.16326 handle_command_reply: tid=0 2024-03-18T22:45:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.243+0000 7fb9e1c7d740 1 client.16326 shutdown 2024-03-18T22:45:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.243+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3638673901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 msgr2=0x563234ae5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.243+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3638673901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234ae5b50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc012700 tx=0x7fb9cc012bc0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.243+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3638673901 shutdown_connections 2024-03-18T22:45:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.243+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3638673901 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x563234af1a60 0x563234af3ee0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.243+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3638673901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234ae1260 0x563234aed0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.243+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3638673901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adc0f0 0x563234ae6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.243+0000 7fb9b7fff640 0 client.16326 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:45:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.243+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3638673901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234ae5b50 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.243+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3638673901 >> 172.21.15.40:0/3638673901 conn(0x563234ad2860 msgr2=0x563234ad1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.243+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3638673901 shutdown_connections 2024-03-18T22:45:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.243+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3638673901 wait complete. 2024-03-18T22:45:53.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.244+0000 7fb9e1c7d740 20 client.16326 trim_cache size 0 max 16384 2024-03-18T22:45:53.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.258+0000 7fb9e1c7d740 1 Processor -- start 2024-03-18T22:45:53.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.258+0000 7fb9e1c7d740 1 -- start start 2024-03-18T22:45:53.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9e1c7d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adb700 0x563234ae5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9e1c7d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234adc0f0 0x563234aed090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9e1c7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234ae1260 0x563234ad1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9e1c7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d81a8390 con 0x563234ae1260 2024-03-18T22:45:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9e1c7d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d81a9f90 con 0x563234adb700 2024-03-18T22:45:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9e1c7d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d81a8a60 con 0x563234adc0f0 2024-03-18T22:45:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9dce2e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adb700 0x563234ae5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9dd62f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234ae1260 0x563234ad1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9d7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234adc0f0 0x563234aed090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9dce2e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adb700 0x563234ae5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36822/0 (socket says 172.21.15.40:36822) 2024-03-18T22:45:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9dd62f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234ae1260 0x563234ad1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50852/0 (socket says 172.21.15.40:50852) 2024-03-18T22:45:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9dce2e640 1 -- 172.21.15.40:0/511269224 learned_addr learned my addr 172.21.15.40:0/511269224 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9dce2e640 1 -- 172.21.15.40:0/511269224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234adc0f0 msgr2=0x563234aed090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9dce2e640 1 --2- 172.21.15.40:0/511269224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234adc0f0 0x563234aed090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9dce2e640 1 -- 172.21.15.40:0/511269224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234ae1260 msgr2=0x563234ad1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9dce2e640 1 --2- 172.21.15.40:0/511269224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234ae1260 0x563234ad1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9dce2e640 1 -- 172.21.15.40:0/511269224 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563234adc4d0 con 0x563234adb700 2024-03-18T22:45:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9dce2e640 1 --2- 172.21.15.40:0/511269224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adb700 0x563234ae5e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc000c00 tx=0x7fb9cc011ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca8bf70e2be4f44c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.259+0000 7fb9c27fc640 1 -- 172.21.15.40:0/511269224 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc010570 con 0x563234adb700 2024-03-18T22:45:53.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.260+0000 7fb9c27fc640 1 -- 172.21.15.40:0/511269224 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9cc00f790 con 0x563234adb700 2024-03-18T22:45:53.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.260+0000 7fb9c27fc640 1 -- 172.21.15.40:0/511269224 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc0035f0 con 0x563234adb700 2024-03-18T22:45:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.260+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/511269224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adb700 msgr2=0x563234ae5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.260+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/511269224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adb700 0x563234ae5e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc000c00 tx=0x7fb9cc011ef0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.260+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/511269224 shutdown_connections 2024-03-18T22:45:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.260+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/511269224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234ae1260 0x563234ad1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.260+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/511269224 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234adc0f0 0x563234aed090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.260+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/511269224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adb700 0x563234ae5e30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.260+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/511269224 >> 172.21.15.40:0/511269224 conn(0x563234ad2860 msgr2=0x563234ad1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.260+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/511269224 shutdown_connections 2024-03-18T22:45:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.260+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/511269224 wait complete. 2024-03-18T22:45:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.261+0000 7fb9e1c7d740 1 -- start start 2024-03-18T22:45:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.261+0000 7fb9e1c7d740 1 Processor -- start 2024-03-18T22:45:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.261+0000 7fb9e1c7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234aea5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.261+0000 7fb9e1c7d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adc0f0 0x563234aeaae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.261+0000 7fb9e1c7d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234ae1260 0x563234aebaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.261+0000 7fb9e1c7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d81a9800 con 0x563234adb700 2024-03-18T22:45:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.261+0000 7fb9e1c7d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d81a9130 con 0x563234adc0f0 2024-03-18T22:45:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.261+0000 7fb9e1c7d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d81a6370 con 0x563234ae1260 2024-03-18T22:45:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.261+0000 7fb9e1c7d740 10 client.0 _ll_register_callbacks cb 0x7fb9cc009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9dce2e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234aea5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9dce2e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234aea5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50856/0 (socket says 172.21.15.40:50856) 2024-03-18T22:45:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9dce2e640 1 -- 172.21.15.40:0/3686523239 learned_addr learned my addr 172.21.15.40:0/3686523239 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9dd62f640 1 --2- 172.21.15.40:0/3686523239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234ae1260 0x563234aebaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9d7fff640 1 --2- 172.21.15.40:0/3686523239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adc0f0 0x563234aeaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9dce2e640 1 -- 172.21.15.40:0/3686523239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234ae1260 msgr2=0x563234aebaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9dce2e640 1 --2- 172.21.15.40:0/3686523239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234ae1260 0x563234aebaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9dce2e640 1 -- 172.21.15.40:0/3686523239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adc0f0 msgr2=0x563234aeaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9dce2e640 1 --2- 172.21.15.40:0/3686523239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adc0f0 0x563234aeaae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9dce2e640 1 -- 172.21.15.40:0/3686523239 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563234aec0e0 con 0x563234adb700 2024-03-18T22:45:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9dce2e640 1 --2- 172.21.15.40:0/3686523239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234aea5a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc010940 tx=0x7fb9cc011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520e48110702e2ae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9b7fff640 1 -- 172.21.15.40:0/3686523239 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc022040 con 0x563234adb700 2024-03-18T22:45:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9b7fff640 1 -- 172.21.15.40:0/3686523239 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb9cc00fb50 con 0x563234adb700 2024-03-18T22:45:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3686523239 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9cc00f790 con 0x563234adb700 2024-03-18T22:45:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9b7fff640 1 -- 172.21.15.40:0/3686523239 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb9cc00fd50 con 0x563234adb700 2024-03-18T22:45:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9e1c7d740 10 client.16332 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.262+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3686523239 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563234aed3e0 con 0x563234adb700 2024-03-18T22:45:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.263+0000 7fb9b7fff640 1 -- 172.21.15.40:0/3686523239 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9cc00f790 con 0x563234adb700 2024-03-18T22:45:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.263+0000 7fb9b7fff640 1 -- 172.21.15.40:0/3686523239 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fb9cc003a40 con 0x563234adb700 2024-03-18T22:45:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.263+0000 7fb9e1c7d740 10 client.16332 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.263+0000 7fb9e1c7d740 10 client.16332 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:45:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.263+0000 7fb9e1c7d740 20 client.16332 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.263+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3686523239 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x563234b58500 0x563234b5a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.263+0000 7fb9e1c7d740 4 client.16332 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:45:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.263+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3686523239 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563234a489f0 con 0x563234b58500 2024-03-18T22:45:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.263+0000 7fb9d7fff640 1 --2- 172.21.15.40:0/3686523239 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x563234b58500 0x563234b5a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.264+0000 7fb9d7fff640 1 --2- 172.21.15.40:0/3686523239 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x563234b58500 0x563234b5a920 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7d7fbfe5a621f376 server_cookie=3474ec90fedb4e05 in_seq=0 out_seq=0 2024-03-18T22:45:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.264+0000 7fb9b7fff640 10 client.16332 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:45:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.264+0000 7fb9b7fff640 1 -- 172.21.15.40:0/3686523239 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563234a489f0 con 0x563234b58500 2024-03-18T22:45:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.264+0000 7fb9b7fff640 10 client.16332 handle_command_reply: tid=0 2024-03-18T22:45:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.264+0000 7fb9e1c7d740 1 client.16332 shutdown 2024-03-18T22:45:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.264+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3686523239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 msgr2=0x563234aea5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.265+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3686523239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234aea5a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc010940 tx=0x7fb9cc011cd0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.265+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3686523239 shutdown_connections 2024-03-18T22:45:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.265+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3686523239 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x563234b58500 0x563234b5a920 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.265+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3686523239 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563234ae1260 0x563234aebaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.265+0000 7fb9b7fff640 0 client.16332 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:45:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.265+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3686523239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563234adc0f0 0x563234aeaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.265+0000 7fb9e1c7d740 1 --2- 172.21.15.40:0/3686523239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563234adb700 0x563234aea5a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.265+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3686523239 >> 172.21.15.40:0/3686523239 conn(0x563234ad2860 msgr2=0x563234ad1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.265+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3686523239 shutdown_connections 2024-03-18T22:45:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.265+0000 7fb9e1c7d740 1 -- 172.21.15.40:0/3686523239 wait complete. 2024-03-18T22:45:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.265+0000 7fb9e1c7d740 20 client.16332 trim_cache size 0 max 16384 2024-03-18T22:45:53.270 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:45:53.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.265+0000 7fb9df092640 1 -- 172.21.15.40:0/123216632 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9c80a2800 msgr2=0x7fb9c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.265+0000 7fb9df092640 1 --2- 172.21.15.40:0/123216632 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9c80a2800 0x7fb9c80a4cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0006010 tx=0x7fb9d0005f60 comp rx=0 tx=0).stop 2024-03-18T22:45:53.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.265+0000 7fb9df092640 1 -- 172.21.15.40:0/123216632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9d8104c30 msgr2=0x7fb9d8197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.265+0000 7fb9df092640 1 --2- 172.21.15.40:0/123216632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9d8104c30 0x7fb9d8197fa0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400e9c0 tx=0x7fb9c400ee80 comp rx=0 tx=0).stop 2024-03-18T22:45:53.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.266+0000 7fb9df092640 1 -- 172.21.15.40:0/123216632 shutdown_connections 2024-03-18T22:45:53.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.266+0000 7fb9df092640 1 --2- 172.21.15.40:0/123216632 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9c80a2800 0x7fb9c80a4cc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.266+0000 7fb9df092640 1 --2- 172.21.15.40:0/123216632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb9d8107560 0x7fb9d819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.266+0000 7fb9df092640 1 --2- 172.21.15.40:0/123216632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9d8104c30 0x7fb9d8197fa0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.266+0000 7fb9df092640 1 --2- 172.21.15.40:0/123216632 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9d8102300 0x7fb9d8197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.266+0000 7fb9df092640 1 -- 172.21.15.40:0/123216632 >> 172.21.15.40:0/123216632 conn(0x7fb9d80fbc90 msgr2=0x7fb9d8102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.266+0000 7fb9df092640 1 -- 172.21.15.40:0/123216632 shutdown_connections 2024-03-18T22:45:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.266+0000 7fb9df092640 1 -- 172.21.15.40:0/123216632 wait complete. 2024-03-18T22:45:53.278 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:45:53.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:53 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2530381819' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:45:53.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:53 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2530381819' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:45:53.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.401+0000 7fb25d745640 1 -- 172.21.15.40:0/3275837001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb258101900 msgr2=0x7fb258101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.401+0000 7fb25d745640 1 --2- 172.21.15.40:0/3275837001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb258101900 0x7fb258101d80 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009900 tx=0x7fb24c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:53.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.401+0000 7fb25d745640 1 -- 172.21.15.40:0/3275837001 shutdown_connections 2024-03-18T22:45:53.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.401+0000 7fb25d745640 1 --2- 172.21.15.40:0/3275837001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2581022c0 0x7fb258108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.401+0000 7fb25d745640 1 --2- 172.21.15.40:0/3275837001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb258101900 0x7fb258101d80 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.401+0000 7fb25d745640 1 --2- 172.21.15.40:0/3275837001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb258100710 0x7fb258100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.401+0000 7fb25d745640 1 -- 172.21.15.40:0/3275837001 >> 172.21.15.40:0/3275837001 conn(0x7fb2580fbcc0 msgr2=0x7fb2580fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.401+0000 7fb25d745640 1 -- 172.21.15.40:0/3275837001 shutdown_connections 2024-03-18T22:45:53.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.401+0000 7fb25d745640 1 -- 172.21.15.40:0/3275837001 wait complete. 2024-03-18T22:45:53.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb25d745640 1 Processor -- start 2024-03-18T22:45:53.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb25d745640 1 -- start start 2024-03-18T22:45:53.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb25d745640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb258100710 0x7fb258199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb25d745640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb258101900 0x7fb25819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb25d745640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2581022c0 0x7fb2581a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb25d745640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb25810c240 con 0x7fb258101900 2024-03-18T22:45:53.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb25d745640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb25810c0c0 con 0x7fb2581022c0 2024-03-18T22:45:53.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb25d745640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb25810c3c0 con 0x7fb258100710 2024-03-18T22:45:53.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb2567fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb258101900 0x7fb25819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb2567fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb258101900 0x7fb25819a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50862/0 (socket says 172.21.15.40:50862) 2024-03-18T22:45:53.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb2567fc640 1 -- 172.21.15.40:0/665245618 learned_addr learned my addr 172.21.15.40:0/665245618 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb256ffd640 1 --2- 172.21.15.40:0/665245618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb258100710 0x7fb258199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb2577fe640 1 --2- 172.21.15.40:0/665245618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2581022c0 0x7fb2581a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb2567fc640 1 -- 172.21.15.40:0/665245618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb258100710 msgr2=0x7fb258199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb2567fc640 1 --2- 172.21.15.40:0/665245618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb258100710 0x7fb258199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb2567fc640 1 -- 172.21.15.40:0/665245618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2581022c0 msgr2=0x7fb2581a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.402+0000 7fb2567fc640 1 --2- 172.21.15.40:0/665245618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2581022c0 0x7fb2581a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.403+0000 7fb2567fc640 1 -- 172.21.15.40:0/665245618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2581a18b0 con 0x7fb258101900 2024-03-18T22:45:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.403+0000 7fb2567fc640 1 --2- 172.21.15.40:0/665245618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb258101900 0x7fb25819a1b0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb24c006010 tx=0x7fb24c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.403+0000 7fb23ffff640 1 -- 172.21.15.40:0/665245618 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24c043020 con 0x7fb258101900 2024-03-18T22:45:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.403+0000 7fb23ffff640 1 -- 172.21.15.40:0/665245618 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb24c0043d0 con 0x7fb258101900 2024-03-18T22:45:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.403+0000 7fb25d745640 1 -- 172.21.15.40:0/665245618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2581a1b40 con 0x7fb258101900 2024-03-18T22:45:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.403+0000 7fb23ffff640 1 -- 172.21.15.40:0/665245618 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24c003b80 con 0x7fb258101900 2024-03-18T22:45:53.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.403+0000 7fb25d745640 1 -- 172.21.15.40:0/665245618 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2581a2090 con 0x7fb258101900 2024-03-18T22:45:53.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.405+0000 7fb23ffff640 1 -- 172.21.15.40:0/665245618 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb24c046020 con 0x7fb258101900 2024-03-18T22:45:53.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb26040d740 1 Processor -- start 2024-03-18T22:45:53.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb26040d740 1 -- start start 2024-03-18T22:45:53.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb26040d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbed2f0 0x55aafbbed710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb26040d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbf27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb26040d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbf2e50 0x55aafbbf52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb26040d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55aafbb514b0 con 0x55aafbbedce0 2024-03-18T22:45:53.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb26040d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55aafbaf9d70 con 0x55aafbbed2f0 2024-03-18T22:45:53.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb26040d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55aafbb4d550 con 0x55aafbbf2e50 2024-03-18T22:45:53.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb256ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbf27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb23ffff640 1 --2- 172.21.15.40:0/665245618 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb2400a2800 0x7fb2400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb256ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbf27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50876/0 (socket says 172.21.15.40:50876) 2024-03-18T22:45:53.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb256ffd640 1 -- 172.21.15.40:0/2649703073 learned_addr learned my addr 172.21.15.40:0/2649703073 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb2577fe640 1 --2- 172.21.15.40:0/2649703073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbed2f0 0x55aafbbed710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb2567fc640 1 --2- 172.21.15.40:0/2649703073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbf2e50 0x55aafbbf52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb2577fe640 1 --2- 172.21.15.40:0/665245618 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb2400a2800 0x7fb2400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb256ffd640 1 -- 172.21.15.40:0/2649703073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbf2e50 msgr2=0x55aafbbf52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb256ffd640 1 --2- 172.21.15.40:0/2649703073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbf2e50 0x55aafbbf52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb256ffd640 1 -- 172.21.15.40:0/2649703073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbed2f0 msgr2=0x55aafbbed710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb23ffff640 1 -- 172.21.15.40:0/665245618 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb24c0f1040 con 0x7fb258101900 2024-03-18T22:45:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb256ffd640 1 --2- 172.21.15.40:0/2649703073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbed2f0 0x55aafbbed710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb256ffd640 1 -- 172.21.15.40:0/2649703073 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aafbb1c210 con 0x55aafbbedce0 2024-03-18T22:45:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.406+0000 7fb2567fc640 1 --2- 172.21.15.40:0/2649703073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbf2e50 0x55aafbbf52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:45:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.407+0000 7fb256ffd640 1 --2- 172.21.15.40:0/2649703073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbf27e0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb248009c60 tx=0x7fb248011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3f9fb02f1c3a98 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.407+0000 7fb2577fe640 1 --2- 172.21.15.40:0/665245618 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb2400a2800 0x7fb2400a4cc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x55aafbbee540 tx=0x7fb250009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.407+0000 7fb23d7fa640 1 -- 172.21.15.40:0/2649703073 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb248004220 con 0x55aafbbedce0 2024-03-18T22:45:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.407+0000 7fb23d7fa640 1 -- 172.21.15.40:0/2649703073 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb248009e10 con 0x55aafbbedce0 2024-03-18T22:45:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.407+0000 7fb23d7fa640 1 -- 172.21.15.40:0/2649703073 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb248019690 con 0x55aafbbedce0 2024-03-18T22:45:53.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 -- 172.21.15.40:0/2649703073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 msgr2=0x55aafbbf27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 --2- 172.21.15.40:0/2649703073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbf27e0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb248009c60 tx=0x7fb248011810 comp rx=0 tx=0).stop 2024-03-18T22:45:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 -- 172.21.15.40:0/2649703073 shutdown_connections 2024-03-18T22:45:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 --2- 172.21.15.40:0/2649703073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbf2e50 0x55aafbbf52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 --2- 172.21.15.40:0/2649703073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbf27e0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 --2- 172.21.15.40:0/2649703073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbed2f0 0x55aafbbed710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 -- 172.21.15.40:0/2649703073 >> 172.21.15.40:0/2649703073 conn(0x55aafbbe8860 msgr2=0x55aafbbe8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 -- 172.21.15.40:0/2649703073 shutdown_connections 2024-03-18T22:45:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 -- 172.21.15.40:0/2649703073 wait complete. 2024-03-18T22:45:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 -- start start 2024-03-18T22:45:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 Processor -- start 2024-03-18T22:45:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbed2f0 0x55aafbbf77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbedce0 0x55aafbbf7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbf2e50 0x55aafbbfed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55aafbbf5e00 con 0x55aafbbf2e50 2024-03-18T22:45:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55aafbb0d310 con 0x55aafbbedce0 2024-03-18T22:45:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55aafbbf5f80 con 0x55aafbbed2f0 2024-03-18T22:45:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.408+0000 7fb26040d740 10 client.0 _ll_register_callbacks cb 0x55aafbb012d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb2577fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbf2e50 0x55aafbbfed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb2567fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbedce0 0x55aafbbf7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb2577fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbf2e50 0x55aafbbfed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50878/0 (socket says 172.21.15.40:50878) 2024-03-18T22:45:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb256ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbed2f0 0x55aafbbf77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb2577fe640 1 -- 172.21.15.40:0/4093312986 learned_addr learned my addr 172.21.15.40:0/4093312986 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb2577fe640 1 -- 172.21.15.40:0/4093312986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbed2f0 msgr2=0x55aafbbf77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb2577fe640 1 --2- 172.21.15.40:0/4093312986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbed2f0 0x55aafbbf77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb2577fe640 1 -- 172.21.15.40:0/4093312986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbedce0 msgr2=0x55aafbbf7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb2577fe640 1 --2- 172.21.15.40:0/4093312986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbedce0 0x55aafbbf7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb2577fe640 1 -- 172.21.15.40:0/4093312986 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aafb8653f0 con 0x55aafbbf2e50 2024-03-18T22:45:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb256ffd640 1 --2- 172.21.15.40:0/4093312986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbed2f0 0x55aafbbf77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb2577fe640 1 --2- 172.21.15.40:0/4093312986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbf2e50 0x55aafbbfed50 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb250011ab0 tx=0x7fb250004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af9e12baa74f80b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb23aff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb23aff5640 1 -- 172.21.15.40:0/4093312986 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb250004780 con 0x55aafbbf2e50 2024-03-18T22:45:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb23aff5640 1 -- 172.21.15.40:0/4093312986 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb250004b20 con 0x55aafbbf2e50 2024-03-18T22:45:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb26040d740 1 -- 172.21.15.40:0/4093312986 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb248009e10 con 0x55aafbbf2e50 2024-03-18T22:45:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb23aff5640 1 -- 172.21.15.40:0/4093312986 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb250033050 con 0x55aafbbf2e50 2024-03-18T22:45:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb26040d740 10 client.16344 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.409+0000 7fb26040d740 1 -- 172.21.15.40:0/4093312986 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55aafb92f7d0 con 0x55aafbbf2e50 2024-03-18T22:45:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.410+0000 7fb23aff5640 1 -- 172.21.15.40:0/4093312986 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb248009e10 con 0x55aafbbf2e50 2024-03-18T22:45:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.410+0000 7fb23aff5640 1 -- 172.21.15.40:0/4093312986 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fb250027030 con 0x55aafbbf2e50 2024-03-18T22:45:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.410+0000 7fb26040d740 10 client.16344 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.410+0000 7fb26040d740 10 client.16344 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:45:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.410+0000 7fb26040d740 20 client.16344 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:53.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.410+0000 7fb26040d740 1 --2- 172.21.15.40:0/4093312986 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55aafbc03740 0x55aafbc05bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.410+0000 7fb26040d740 4 client.16344 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.410+0000 7fb26040d740 1 -- 172.21.15.40:0/4093312986 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aafbbaf3e0 con 0x55aafbc03740 2024-03-18T22:45:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.410+0000 7fb256ffd640 1 --2- 172.21.15.40:0/4093312986 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55aafbc03740 0x55aafbc05bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.410+0000 7fb256ffd640 1 --2- 172.21.15.40:0/4093312986 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55aafbc03740 0x55aafbc05bc0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ecd400d3f1b87e58 server_cookie=1223b35e031b476e in_seq=0 out_seq=0 2024-03-18T22:45:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.410+0000 7fb23aff5640 10 client.16344 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:45:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.411+0000 7fb23aff5640 1 -- 172.21.15.40:0/4093312986 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55aafbbaf3e0 con 0x55aafbc03740 2024-03-18T22:45:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.411+0000 7fb23aff5640 10 client.16344 handle_command_reply: tid=0 2024-03-18T22:45:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.411+0000 7fb26040d740 1 client.16344 shutdown 2024-03-18T22:45:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.412+0000 7fb26040d740 1 -- 172.21.15.40:0/4093312986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbf2e50 msgr2=0x55aafbbfed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.412+0000 7fb26040d740 1 --2- 172.21.15.40:0/4093312986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbf2e50 0x55aafbbfed50 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb250011ab0 tx=0x7fb250004a40 comp rx=0 tx=0).stop 2024-03-18T22:45:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.412+0000 7fb26040d740 1 -- 172.21.15.40:0/4093312986 shutdown_connections 2024-03-18T22:45:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.412+0000 7fb26040d740 1 --2- 172.21.15.40:0/4093312986 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55aafbc03740 0x55aafbc05bc0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.412+0000 7fb26040d740 1 --2- 172.21.15.40:0/4093312986 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbf2e50 0x55aafbbfed50 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.412+0000 7fb26040d740 1 --2- 172.21.15.40:0/4093312986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbedce0 0x55aafbbf7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.412+0000 7fb23aff5640 0 client.16344 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:45:53.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.412+0000 7fb26040d740 1 --2- 172.21.15.40:0/4093312986 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbed2f0 0x55aafbbf77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.412+0000 7fb26040d740 1 -- 172.21.15.40:0/4093312986 >> 172.21.15.40:0/4093312986 conn(0x55aafbbe8860 msgr2=0x55aafbbe7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.412+0000 7fb26040d740 1 -- 172.21.15.40:0/4093312986 shutdown_connections 2024-03-18T22:45:53.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.412+0000 7fb26040d740 1 -- 172.21.15.40:0/4093312986 wait complete. 2024-03-18T22:45:53.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.412+0000 7fb26040d740 20 client.16344 trim_cache size 0 max 16384 2024-03-18T22:45:53.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb26040d740 1 Processor -- start 2024-03-18T22:45:53.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb26040d740 1 -- start start 2024-03-18T22:45:53.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb26040d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbed2f0 0x55aafbbe3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb26040d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbfed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb26040d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbf2e50 0x55aafbbe7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb26040d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2581ac650 con 0x55aafbbedce0 2024-03-18T22:45:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb26040d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2581ae250 con 0x55aafbbed2f0 2024-03-18T22:45:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb26040d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb2581acd20 con 0x55aafbbf2e50 2024-03-18T22:45:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb2567fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbfed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb256ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbed2f0 0x55aafbbe3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb256ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbed2f0 0x55aafbbe3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36862/0 (socket says 172.21.15.40:36862) 2024-03-18T22:45:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb256ffd640 1 -- 172.21.15.40:0/306724904 learned_addr learned my addr 172.21.15.40:0/306724904 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb2577fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbf2e50 0x55aafbbe7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb256ffd640 1 -- 172.21.15.40:0/306724904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbf2e50 msgr2=0x55aafbbe7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb256ffd640 1 --2- 172.21.15.40:0/306724904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbf2e50 0x55aafbbe7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb256ffd640 1 -- 172.21.15.40:0/306724904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 msgr2=0x55aafbbfed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb256ffd640 1 --2- 172.21.15.40:0/306724904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbfed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb256ffd640 1 -- 172.21.15.40:0/306724904 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aafbbf2410 con 0x55aafbbed2f0 2024-03-18T22:45:53.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.427+0000 7fb2577fe640 1 --2- 172.21.15.40:0/306724904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbf2e50 0x55aafbbe7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:53.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.428+0000 7fb256ffd640 1 --2- 172.21.15.40:0/306724904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbed2f0 0x55aafbbe3a30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb248013010 tx=0x7fb248009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62b633a3c5194542 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.428+0000 7fb23cff9640 1 -- 172.21.15.40:0/306724904 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb248003a40 con 0x55aafbbed2f0 2024-03-18T22:45:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.428+0000 7fb23cff9640 1 -- 172.21.15.40:0/306724904 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb248030030 con 0x55aafbbed2f0 2024-03-18T22:45:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.428+0000 7fb23cff9640 1 -- 172.21.15.40:0/306724904 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb248004180 con 0x55aafbbed2f0 2024-03-18T22:45:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.429+0000 7fb26040d740 1 -- 172.21.15.40:0/306724904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbed2f0 msgr2=0x55aafbbe3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.429+0000 7fb26040d740 1 --2- 172.21.15.40:0/306724904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbed2f0 0x55aafbbe3a30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb248013010 tx=0x7fb248009310 comp rx=0 tx=0).stop 2024-03-18T22:45:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.429+0000 7fb26040d740 1 -- 172.21.15.40:0/306724904 shutdown_connections 2024-03-18T22:45:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.429+0000 7fb26040d740 1 --2- 172.21.15.40:0/306724904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbf2e50 0x55aafbbe7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.429+0000 7fb26040d740 1 --2- 172.21.15.40:0/306724904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbfed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.429+0000 7fb26040d740 1 --2- 172.21.15.40:0/306724904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbed2f0 0x55aafbbe3a30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.429+0000 7fb26040d740 1 -- 172.21.15.40:0/306724904 >> 172.21.15.40:0/306724904 conn(0x55aafbbe8860 msgr2=0x55aafbbe8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.429+0000 7fb26040d740 1 -- 172.21.15.40:0/306724904 shutdown_connections 2024-03-18T22:45:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.429+0000 7fb26040d740 1 -- 172.21.15.40:0/306724904 wait complete. 2024-03-18T22:45:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.429+0000 7fb26040d740 1 -- start start 2024-03-18T22:45:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.429+0000 7fb26040d740 1 Processor -- start 2024-03-18T22:45:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.429+0000 7fb26040d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbed2f0 0x55aafbbfbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.429+0000 7fb26040d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbfc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb26040d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbf2e50 0x55aafbc67af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb26040d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb2581adac0 con 0x55aafbbedce0 2024-03-18T22:45:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb26040d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2581ad3f0 con 0x55aafbbf2e50 2024-03-18T22:45:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb26040d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb2581aa630 con 0x55aafbbed2f0 2024-03-18T22:45:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb26040d740 10 client.0 _ll_register_callbacks cb 0x7fb250011860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb2567fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbfc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb2567fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbfc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50886/0 (socket says 172.21.15.40:50886) 2024-03-18T22:45:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb2567fc640 1 -- 172.21.15.40:0/716585500 learned_addr learned my addr 172.21.15.40:0/716585500 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb256ffd640 1 --2- 172.21.15.40:0/716585500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbed2f0 0x55aafbbfbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb2567fc640 1 -- 172.21.15.40:0/716585500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbed2f0 msgr2=0x55aafbbfbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb2577fe640 1 --2- 172.21.15.40:0/716585500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbf2e50 0x55aafbc67af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb2567fc640 1 --2- 172.21.15.40:0/716585500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbed2f0 0x55aafbbfbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb2567fc640 1 -- 172.21.15.40:0/716585500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbf2e50 msgr2=0x55aafbc67af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb2567fc640 1 --2- 172.21.15.40:0/716585500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbf2e50 0x55aafbc67af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb2567fc640 1 -- 172.21.15.40:0/716585500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aafbc68030 con 0x55aafbbedce0 2024-03-18T22:45:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb256ffd640 1 --2- 172.21.15.40:0/716585500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbed2f0 0x55aafbbfbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:45:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb2567fc640 1 --2- 172.21.15.40:0/716585500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbfc0d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb24c0b9fc0 tx=0x7fb24c08d110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc4346e298bc190b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb23aff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.430+0000 7fb23aff5640 1 -- 172.21.15.40:0/716585500 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24c0bd5f0 con 0x55aafbbedce0 2024-03-18T22:45:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.431+0000 7fb23aff5640 1 -- 172.21.15.40:0/716585500 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb24c003d20 con 0x55aafbbedce0 2024-03-18T22:45:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.431+0000 7fb26040d740 1 -- 172.21.15.40:0/716585500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb248030030 con 0x55aafbbedce0 2024-03-18T22:45:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.431+0000 7fb23aff5640 1 -- 172.21.15.40:0/716585500 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb24c0c1740 con 0x55aafbbedce0 2024-03-18T22:45:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.431+0000 7fb26040d740 10 client.16350 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.431+0000 7fb26040d740 1 -- 172.21.15.40:0/716585500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55aafbc69280 con 0x55aafbbedce0 2024-03-18T22:45:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.431+0000 7fb23aff5640 1 -- 172.21.15.40:0/716585500 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb248030030 con 0x55aafbbedce0 2024-03-18T22:45:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.431+0000 7fb23aff5640 1 -- 172.21.15.40:0/716585500 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fb24c0c5990 con 0x55aafbbedce0 2024-03-18T22:45:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.431+0000 7fb26040d740 10 client.16350 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.431+0000 7fb26040d740 10 client.16350 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:45:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.431+0000 7fb26040d740 20 client.16350 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.431+0000 7fb26040d740 1 --2- 172.21.15.40:0/716585500 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55aafbc69c30 0x55aafbc6c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.431+0000 7fb26040d740 4 client.16350 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:45:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.431+0000 7fb26040d740 1 -- 172.21.15.40:0/716585500 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aafbb5e9f0 con 0x55aafbc69c30 2024-03-18T22:45:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.432+0000 7fb256ffd640 1 --2- 172.21.15.40:0/716585500 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55aafbc69c30 0x55aafbc6c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.432+0000 7fb23aff5640 10 client.16350 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:45:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.432+0000 7fb256ffd640 1 --2- 172.21.15.40:0/716585500 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55aafbc69c30 0x55aafbc6c050 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4d659c3a5e526e66 server_cookie=1308d72a06a90551 in_seq=0 out_seq=0 2024-03-18T22:45:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.432+0000 7fb23aff5640 1 -- 172.21.15.40:0/716585500 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aafbb5e9f0 con 0x55aafbc69c30 2024-03-18T22:45:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.432+0000 7fb23aff5640 10 client.16350 handle_command_reply: tid=0 2024-03-18T22:45:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.432+0000 7fb26040d740 1 client.16350 shutdown 2024-03-18T22:45:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.432+0000 7fb26040d740 1 -- 172.21.15.40:0/716585500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 msgr2=0x55aafbbfc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.432+0000 7fb26040d740 1 --2- 172.21.15.40:0/716585500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbfc0d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb24c0b9fc0 tx=0x7fb24c08d110 comp rx=0 tx=0).stop 2024-03-18T22:45:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.432+0000 7fb26040d740 1 -- 172.21.15.40:0/716585500 shutdown_connections 2024-03-18T22:45:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.432+0000 7fb26040d740 1 --2- 172.21.15.40:0/716585500 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55aafbc69c30 0x55aafbc6c050 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.432+0000 7fb26040d740 1 --2- 172.21.15.40:0/716585500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55aafbbf2e50 0x55aafbc67af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.432+0000 7fb26040d740 1 --2- 172.21.15.40:0/716585500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55aafbbedce0 0x55aafbbfc0d0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.432+0000 7fb23aff5640 0 client.16350 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:45:53.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.432+0000 7fb26040d740 1 --2- 172.21.15.40:0/716585500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55aafbbed2f0 0x55aafbbfbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.432+0000 7fb26040d740 1 -- 172.21.15.40:0/716585500 >> 172.21.15.40:0/716585500 conn(0x55aafbbe8860 msgr2=0x55aafbbe8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.433+0000 7fb26040d740 1 -- 172.21.15.40:0/716585500 shutdown_connections 2024-03-18T22:45:53.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.433+0000 7fb26040d740 1 -- 172.21.15.40:0/716585500 wait complete. 2024-03-18T22:45:53.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.433+0000 7fb26040d740 20 client.16350 trim_cache size 0 max 16384 2024-03-18T22:45:53.439 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:45:53.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.433+0000 7fb25d745640 1 -- 172.21.15.40:0/665245618 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb2400a2800 msgr2=0x7fb2400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.433+0000 7fb25d745640 1 --2- 172.21.15.40:0/665245618 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb2400a2800 0x7fb2400a4cc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x55aafbbee540 tx=0x7fb250009500 comp rx=0 tx=0).stop 2024-03-18T22:45:53.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.433+0000 7fb25d745640 1 -- 172.21.15.40:0/665245618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb258101900 msgr2=0x7fb25819a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.433+0000 7fb25d745640 1 --2- 172.21.15.40:0/665245618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb258101900 0x7fb25819a1b0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb24c006010 tx=0x7fb24c005e40 comp rx=0 tx=0).stop 2024-03-18T22:45:53.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.433+0000 7fb25d745640 1 -- 172.21.15.40:0/665245618 shutdown_connections 2024-03-18T22:45:53.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.433+0000 7fb25d745640 1 --2- 172.21.15.40:0/665245618 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb2400a2800 0x7fb2400a4cc0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.433+0000 7fb25d745640 1 --2- 172.21.15.40:0/665245618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2581022c0 0x7fb2581a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.433+0000 7fb25d745640 1 --2- 172.21.15.40:0/665245618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb258101900 0x7fb25819a1b0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.433+0000 7fb25d745640 1 --2- 172.21.15.40:0/665245618 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb258100710 0x7fb258199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.433+0000 7fb25d745640 1 -- 172.21.15.40:0/665245618 >> 172.21.15.40:0/665245618 conn(0x7fb2580fbcc0 msgr2=0x7fb2580fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.433+0000 7fb25d745640 1 -- 172.21.15.40:0/665245618 shutdown_connections 2024-03-18T22:45:53.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.433+0000 7fb25d745640 1 -- 172.21.15.40:0/665245618 wait complete. 2024-03-18T22:45:53.447 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:45:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.553+0000 7f765adf4640 1 -- 172.21.15.40:0/828430800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7654100720 msgr2=0x7f7654100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.553+0000 7f765adf4640 1 --2- 172.21.15.40:0/828430800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7654100720 0x7f7654100b40 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7648009900 tx=0x7f764802bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.553+0000 7f765adf4640 1 -- 172.21.15.40:0/828430800 shutdown_connections 2024-03-18T22:45:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.553+0000 7f765adf4640 1 --2- 172.21.15.40:0/828430800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76541022d0 0x7f7654108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.553+0000 7f765adf4640 1 --2- 172.21.15.40:0/828430800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7654101910 0x7f7654101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.553+0000 7f765adf4640 1 --2- 172.21.15.40:0/828430800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7654100720 0x7f7654100b40 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.553+0000 7f765adf4640 1 -- 172.21.15.40:0/828430800 >> 172.21.15.40:0/828430800 conn(0x7f76540fbc90 msgr2=0x7f76540fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.553+0000 7f765adf4640 1 -- 172.21.15.40:0/828430800 shutdown_connections 2024-03-18T22:45:53.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.553+0000 7f765adf4640 1 -- 172.21.15.40:0/828430800 wait complete. 2024-03-18T22:45:53.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.554+0000 7f765adf4640 1 Processor -- start 2024-03-18T22:45:53.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.554+0000 7f765adf4640 1 -- start start 2024-03-18T22:45:53.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.554+0000 7f765adf4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7654100720 0x7f7654197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.554+0000 7f765adf4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7654101910 0x7f7654197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.554+0000 7f765adf4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76541022d0 0x7f765419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.554+0000 7f765adf4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f765410c220 con 0x7f7654101910 2024-03-18T22:45:53.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.554+0000 7f765adf4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f765410c0a0 con 0x7f76541022d0 2024-03-18T22:45:53.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.554+0000 7f765adf4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f765410c3a0 con 0x7f7654100720 2024-03-18T22:45:53.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.554+0000 7f7653fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7654101910 0x7f7654197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.554+0000 7f7658b90640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7654100720 0x7f7654197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.554+0000 7f7653fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7654101910 0x7f7654197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50902/0 (socket says 172.21.15.40:50902) 2024-03-18T22:45:53.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.554+0000 7f7653fff640 1 -- 172.21.15.40:0/3709606092 learned_addr learned my addr 172.21.15.40:0/3709606092 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.554+0000 7f7659391640 1 --2- 172.21.15.40:0/3709606092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76541022d0 0x7f765419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.555+0000 7f7653fff640 1 -- 172.21.15.40:0/3709606092 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7654100720 msgr2=0x7f7654197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.555+0000 7f7653fff640 1 --2- 172.21.15.40:0/3709606092 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7654100720 0x7f7654197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.555+0000 7f7653fff640 1 -- 172.21.15.40:0/3709606092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76541022d0 msgr2=0x7f765419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.555+0000 7f7653fff640 1 --2- 172.21.15.40:0/3709606092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76541022d0 0x7f765419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.555+0000 7f7653fff640 1 -- 172.21.15.40:0/3709606092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765419f690 con 0x7f7654101910 2024-03-18T22:45:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.555+0000 7f7658b90640 1 --2- 172.21.15.40:0/3709606092 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7654100720 0x7f7654197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.555+0000 7f7653fff640 1 --2- 172.21.15.40:0/3709606092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7654101910 0x7f7654197fb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f764000e9f0 tx=0x7f764000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.555+0000 7f7651ffb640 1 -- 172.21.15.40:0/3709606092 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764000cd80 con 0x7f7654101910 2024-03-18T22:45:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.555+0000 7f7651ffb640 1 -- 172.21.15.40:0/3709606092 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76400108e0 con 0x7f7654101910 2024-03-18T22:45:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.555+0000 7f765adf4640 1 -- 172.21.15.40:0/3709606092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f765419f980 con 0x7f7654101910 2024-03-18T22:45:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.555+0000 7f7651ffb640 1 -- 172.21.15.40:0/3709606092 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7640018920 con 0x7f7654101910 2024-03-18T22:45:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.555+0000 7f765adf4640 1 -- 172.21.15.40:0/3709606092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f765419fed0 con 0x7f7654101910 2024-03-18T22:45:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.558+0000 7f7651ffb640 1 -- 172.21.15.40:0/3709606092 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7640010a90 con 0x7f7654101910 2024-03-18T22:45:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.558+0000 7f765d9dd740 1 Processor -- start 2024-03-18T22:45:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.558+0000 7f765d9dd740 1 -- start start 2024-03-18T22:45:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.558+0000 7f7651ffb640 1 --2- 172.21.15.40:0/3709606092 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f76440a2800 0x7f76440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.558+0000 7f765d9dd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda22f0 0x556f8bda2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.558+0000 7f765d9dd740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 0x556f8bda77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.558+0000 7f765d9dd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda7e50 0x556f8bdaa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.558+0000 7f765d9dd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556f8bd064b0 con 0x556f8bda22f0 2024-03-18T22:45:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.558+0000 7f765d9dd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556f8bcaed70 con 0x556f8bda7e50 2024-03-18T22:45:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.558+0000 7f765d9dd740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556f8bd02550 con 0x556f8bda2ce0 2024-03-18T22:45:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f7658b90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda22f0 0x556f8bda2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f7651ffb640 1 -- 172.21.15.40:0/3709606092 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f76400cf140 con 0x7f7654101910 2024-03-18T22:45:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f7659391640 1 --2- 172.21.15.40:0/3709606092 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f76440a2800 0x7f76440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f7658b90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda22f0 0x556f8bda2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50908/0 (socket says 172.21.15.40:50908) 2024-03-18T22:45:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f7653fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 0x556f8bda77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f7658b90640 1 -- 172.21.15.40:0/3628361630 learned_addr learned my addr 172.21.15.40:0/3628361630 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f7659391640 1 --2- 172.21.15.40:0/3628361630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda7e50 0x556f8bdaa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f7653fff640 1 -- 172.21.15.40:0/3628361630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda7e50 msgr2=0x556f8bdaa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f7653fff640 1 --2- 172.21.15.40:0/3628361630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda7e50 0x556f8bdaa2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f7653fff640 1 -- 172.21.15.40:0/3628361630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda22f0 msgr2=0x556f8bda2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f7653fff640 1 --2- 172.21.15.40:0/3628361630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda22f0 0x556f8bda2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f7653fff640 1 -- 172.21.15.40:0/3628361630 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f8bcd1210 con 0x556f8bda2ce0 2024-03-18T22:45:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f7659391640 1 --2- 172.21.15.40:0/3628361630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda7e50 0x556f8bdaa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:45:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f7653fff640 1 --2- 172.21.15.40:0/3628361630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 0x556f8bda77e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f76400ccbc0 tx=0x7f7640097b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20bceded245db263 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f7659391640 1 --2- 172.21.15.40:0/3709606092 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f76440a2800 0x7f76440a4cc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f764c005fd0 tx=0x7f764c009d90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f763effd640 1 -- 172.21.15.40:0/3628361630 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764001f690 con 0x556f8bda2ce0 2024-03-18T22:45:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f763effd640 1 -- 172.21.15.40:0/3628361630 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7640010de0 con 0x556f8bda2ce0 2024-03-18T22:45:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.559+0000 7f763effd640 1 -- 172.21.15.40:0/3628361630 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7640097e10 con 0x556f8bda2ce0 2024-03-18T22:45:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.560+0000 7f765d9dd740 1 -- 172.21.15.40:0/3628361630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 msgr2=0x556f8bda77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.560+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3628361630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 0x556f8bda77e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f76400ccbc0 tx=0x7f7640097b70 comp rx=0 tx=0).stop 2024-03-18T22:45:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.560+0000 7f765d9dd740 1 -- 172.21.15.40:0/3628361630 shutdown_connections 2024-03-18T22:45:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.560+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3628361630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda7e50 0x556f8bdaa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.560+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3628361630 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 0x556f8bda77e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.560+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3628361630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda22f0 0x556f8bda2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.560+0000 7f765d9dd740 1 -- 172.21.15.40:0/3628361630 >> 172.21.15.40:0/3628361630 conn(0x556f8bd9d860 msgr2=0x556f8bd9dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.560+0000 7f765d9dd740 1 -- 172.21.15.40:0/3628361630 shutdown_connections 2024-03-18T22:45:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.560+0000 7f765d9dd740 1 -- 172.21.15.40:0/3628361630 wait complete. 2024-03-18T22:45:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.560+0000 7f765d9dd740 1 -- start start 2024-03-18T22:45:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f765d9dd740 1 Processor -- start 2024-03-18T22:45:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f765d9dd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda22f0 0x556f8bdac840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f765d9dd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda2ce0 0x556f8bdacd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f765d9dd740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda7e50 0x556f8bdb3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f765d9dd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556f8bdaad40 con 0x556f8bda2ce0 2024-03-18T22:45:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f765d9dd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556f8bcc2310 con 0x556f8bda22f0 2024-03-18T22:45:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f765d9dd740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556f8bdaaec0 con 0x556f8bda7e50 2024-03-18T22:45:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f7653fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda2ce0 0x556f8bdacd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f765d9dd740 10 client.0 _ll_register_callbacks cb 0x556f8bcb62d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f7659391640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda7e50 0x556f8bdb3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f7653fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda2ce0 0x556f8bdacd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50910/0 (socket says 172.21.15.40:50910) 2024-03-18T22:45:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f7653fff640 1 -- 172.21.15.40:0/3122064674 learned_addr learned my addr 172.21.15.40:0/3122064674 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f7658b90640 1 --2- 172.21.15.40:0/3122064674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda22f0 0x556f8bdac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f7653fff640 1 -- 172.21.15.40:0/3122064674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda7e50 msgr2=0x556f8bdb3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f7653fff640 1 --2- 172.21.15.40:0/3122064674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda7e50 0x556f8bdb3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f7653fff640 1 -- 172.21.15.40:0/3122064674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda22f0 msgr2=0x556f8bdac840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f7653fff640 1 --2- 172.21.15.40:0/3122064674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda22f0 0x556f8bdac840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f7653fff640 1 -- 172.21.15.40:0/3122064674 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f8ba1a3f0 con 0x556f8bda2ce0 2024-03-18T22:45:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.561+0000 7f7659391640 1 --2- 172.21.15.40:0/3122064674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda7e50 0x556f8bdb3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f7653fff640 1 --2- 172.21.15.40:0/3122064674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda2ce0 0x556f8bdacd80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f764009b730 tx=0x7f764001ff20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f5a2cb694f8c94 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f7633fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f7633fff640 1 -- 172.21.15.40:0/3122064674 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76400b3020 con 0x556f8bda2ce0 2024-03-18T22:45:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f7633fff640 1 -- 172.21.15.40:0/3122064674 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7640010430 con 0x556f8bda2ce0 2024-03-18T22:45:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f765d9dd740 1 -- 172.21.15.40:0/3122064674 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556f8bd643e0 con 0x556f8bda2ce0 2024-03-18T22:45:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f7633fff640 1 -- 172.21.15.40:0/3122064674 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7640099b80 con 0x556f8bda2ce0 2024-03-18T22:45:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f765d9dd740 10 client.16362 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f765d9dd740 1 -- 172.21.15.40:0/3122064674 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556f8bae47d0 con 0x556f8bda2ce0 2024-03-18T22:45:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f7633fff640 1 -- 172.21.15.40:0/3122064674 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x556f8bd643e0 con 0x556f8bda2ce0 2024-03-18T22:45:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f7633fff640 1 -- 172.21.15.40:0/3122064674 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f76400b4250 con 0x556f8bda2ce0 2024-03-18T22:45:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f765d9dd740 10 client.16362 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f765d9dd740 10 client.16362 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f765d9dd740 20 client.16362 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3122064674 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556f8bdb8140 0x556f8bdba530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f765d9dd740 4 client.16362 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.562+0000 7f765d9dd740 1 -- 172.21.15.40:0/3122064674 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f8bd139f0 con 0x556f8bdb8140 2024-03-18T22:45:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.563+0000 7f7658b90640 1 --2- 172.21.15.40:0/3122064674 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556f8bdb8140 0x556f8bdba530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.563+0000 7f7658b90640 1 --2- 172.21.15.40:0/3122064674 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556f8bdb8140 0x556f8bdba530 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a38f387b919b5f server_cookie=9b99ae219e8a26fb in_seq=0 out_seq=0 2024-03-18T22:45:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.563+0000 7f7633fff640 10 client.16362 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.564+0000 7f7633fff640 1 -- 172.21.15.40:0/3122064674 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556f8bd139f0 con 0x556f8bdb8140 2024-03-18T22:45:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.564+0000 7f7633fff640 10 client.16362 handle_command_reply: tid=0 2024-03-18T22:45:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.564+0000 7f765d9dd740 1 client.16362 shutdown 2024-03-18T22:45:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.564+0000 7f765d9dd740 1 -- 172.21.15.40:0/3122064674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda2ce0 msgr2=0x556f8bdacd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.564+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3122064674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda2ce0 0x556f8bdacd80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f764009b730 tx=0x7f764001ff20 comp rx=0 tx=0).stop 2024-03-18T22:45:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.564+0000 7f765d9dd740 1 -- 172.21.15.40:0/3122064674 shutdown_connections 2024-03-18T22:45:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.564+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3122064674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda7e50 0x556f8bdb3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.564+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3122064674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda2ce0 0x556f8bdacd80 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.564+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3122064674 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556f8bdb8140 0x556f8bdba530 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.564+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3122064674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda22f0 0x556f8bdac840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.564+0000 7f7633fff640 0 client.16362 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:53.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.564+0000 7f765d9dd740 1 -- 172.21.15.40:0/3122064674 >> 172.21.15.40:0/3122064674 conn(0x556f8bd9d860 msgr2=0x556f8bd9cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.564+0000 7f765d9dd740 1 -- 172.21.15.40:0/3122064674 shutdown_connections 2024-03-18T22:45:53.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.564+0000 7f765d9dd740 1 -- 172.21.15.40:0/3122064674 wait complete. 2024-03-18T22:45:53.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.564+0000 7f765d9dd740 20 client.16362 trim_cache size 0 max 16384 2024-03-18T22:45:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.579+0000 7f765d9dd740 1 Processor -- start 2024-03-18T22:45:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.579+0000 7f765d9dd740 1 -- start start 2024-03-18T22:45:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.579+0000 7f765d9dd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda22f0 0x556f8bd9c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f765d9dd740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 0x556f8bd9c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f765d9dd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda7e50 0x556f8bd9cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f765d9dd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f76541aa5b0 con 0x556f8bda22f0 2024-03-18T22:45:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f765d9dd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f76541ac1b0 con 0x556f8bda7e50 2024-03-18T22:45:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f765d9dd740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f76541aac80 con 0x556f8bda2ce0 2024-03-18T22:45:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f7658b90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda22f0 0x556f8bd9c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f7658b90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda22f0 0x556f8bd9c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50914/0 (socket says 172.21.15.40:50914) 2024-03-18T22:45:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f7658b90640 1 -- 172.21.15.40:0/2862653402 learned_addr learned my addr 172.21.15.40:0/2862653402 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f7659391640 1 --2- 172.21.15.40:0/2862653402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda7e50 0x556f8bd9cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f7653fff640 1 --2- 172.21.15.40:0/2862653402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 0x556f8bd9c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f7658b90640 1 -- 172.21.15.40:0/2862653402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 msgr2=0x556f8bd9c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f7658b90640 1 --2- 172.21.15.40:0/2862653402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 0x556f8bd9c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f7658b90640 1 -- 172.21.15.40:0/2862653402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda7e50 msgr2=0x556f8bd9cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f7658b90640 1 --2- 172.21.15.40:0/2862653402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda7e50 0x556f8bd9cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f7658b90640 1 -- 172.21.15.40:0/2862653402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f8bd73880 con 0x556f8bda22f0 2024-03-18T22:45:53.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f7658b90640 1 --2- 172.21.15.40:0/2862653402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda22f0 0x556f8bd9c310 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7648005ab0 tx=0x7f7648005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a930b9fbd7442fef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f763e7fc640 1 -- 172.21.15.40:0/2862653402 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648049050 con 0x556f8bda22f0 2024-03-18T22:45:53.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f763e7fc640 1 -- 172.21.15.40:0/2862653402 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76480041e0 con 0x556f8bda22f0 2024-03-18T22:45:53.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.580+0000 7f763e7fc640 1 -- 172.21.15.40:0/2862653402 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764804b820 con 0x556f8bda22f0 2024-03-18T22:45:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.581+0000 7f765d9dd740 1 -- 172.21.15.40:0/2862653402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda22f0 msgr2=0x556f8bd9c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.581+0000 7f765d9dd740 1 --2- 172.21.15.40:0/2862653402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda22f0 0x556f8bd9c310 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7648005ab0 tx=0x7f7648005fd0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.581+0000 7f765d9dd740 1 -- 172.21.15.40:0/2862653402 shutdown_connections 2024-03-18T22:45:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.581+0000 7f765d9dd740 1 --2- 172.21.15.40:0/2862653402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda7e50 0x556f8bd9cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.581+0000 7f765d9dd740 1 --2- 172.21.15.40:0/2862653402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 0x556f8bd9c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.581+0000 7f765d9dd740 1 --2- 172.21.15.40:0/2862653402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda22f0 0x556f8bd9c310 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.581+0000 7f765d9dd740 1 -- 172.21.15.40:0/2862653402 >> 172.21.15.40:0/2862653402 conn(0x556f8bd9d860 msgr2=0x556f8bdac840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.581+0000 7f765d9dd740 1 -- 172.21.15.40:0/2862653402 shutdown_connections 2024-03-18T22:45:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.581+0000 7f765d9dd740 1 -- 172.21.15.40:0/2862653402 wait complete. 2024-03-18T22:45:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.582+0000 7f765d9dd740 1 -- start start 2024-03-18T22:45:53.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.582+0000 7f765d9dd740 1 Processor -- start 2024-03-18T22:45:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.582+0000 7f765d9dd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda22f0 0x556f8be19420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.582+0000 7f765d9dd740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 0x556f8be19960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.582+0000 7f765d9dd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda7e50 0x556f8be209a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.582+0000 7f765d9dd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f76541aba20 con 0x556f8bda7e50 2024-03-18T22:45:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.582+0000 7f765d9dd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f76541ab350 con 0x556f8bda22f0 2024-03-18T22:45:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.582+0000 7f765d9dd740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f76541a8590 con 0x556f8bda2ce0 2024-03-18T22:45:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.582+0000 7f765d9dd740 10 client.0 _ll_register_callbacks cb 0x7f7648009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.582+0000 7f7653fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 0x556f8be19960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.582+0000 7f7659391640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda7e50 0x556f8be209a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.582+0000 7f7658b90640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda22f0 0x556f8be19420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f7653fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 0x556f8be19960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:36658/0 (socket says 172.21.15.40:36658) 2024-03-18T22:45:53.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f7659391640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda7e50 0x556f8be209a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50916/0 (socket says 172.21.15.40:50916) 2024-03-18T22:45:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f7658b90640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda22f0 0x556f8be19420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36928/0 (socket says 172.21.15.40:36928) 2024-03-18T22:45:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f7659391640 1 -- 172.21.15.40:0/3473095343 learned_addr learned my addr 172.21.15.40:0/3473095343 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f7653fff640 1 -- 172.21.15.40:0/3473095343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda22f0 msgr2=0x556f8be19420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f7653fff640 1 --2- 172.21.15.40:0/3473095343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda22f0 0x556f8be19420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f7653fff640 1 -- 172.21.15.40:0/3473095343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda7e50 msgr2=0x556f8be209a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f7653fff640 1 --2- 172.21.15.40:0/3473095343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda7e50 0x556f8be209a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f7653fff640 1 -- 172.21.15.40:0/3473095343 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f8be20f90 con 0x556f8bda2ce0 2024-03-18T22:45:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f7653fff640 1 --2- 172.21.15.40:0/3473095343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 0x556f8be19960 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f76400b4640 tx=0x7f7640005b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ca0f072219284ce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f7633fff640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:45:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f7633fff640 1 -- 172.21.15.40:0/3473095343 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76400b3070 con 0x556f8bda2ce0 2024-03-18T22:45:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f7633fff640 1 -- 172.21.15.40:0/3473095343 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7640097e10 con 0x556f8bda2ce0 2024-03-18T22:45:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f765d9dd740 1 -- 172.21.15.40:0/3473095343 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76480041e0 con 0x556f8bda2ce0 2024-03-18T22:45:53.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f7633fff640 1 -- 172.21.15.40:0/3473095343 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f764009b7a0 con 0x556f8bda2ce0 2024-03-18T22:45:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f765d9dd740 10 client.26191 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.583+0000 7f765d9dd740 1 -- 172.21.15.40:0/3473095343 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556f8be22290 con 0x556f8bda2ce0 2024-03-18T22:45:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.584+0000 7f7633fff640 1 -- 172.21.15.40:0/3473095343 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76480041e0 con 0x556f8bda2ce0 2024-03-18T22:45:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.584+0000 7f7633fff640 1 -- 172.21.15.40:0/3473095343 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f764009abf0 con 0x556f8bda2ce0 2024-03-18T22:45:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.584+0000 7f765d9dd740 10 client.26191 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.584+0000 7f765d9dd740 10 client.26191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.584+0000 7f765d9dd740 20 client.26191 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.584+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3473095343 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556f8be22b80 0x556f8be24fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.584+0000 7f765d9dd740 4 client.26191 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:45:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.584+0000 7f765d9dd740 1 -- 172.21.15.40:0/3473095343 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556f8bd0dd40 con 0x556f8be22b80 2024-03-18T22:45:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.584+0000 7f7658b90640 1 --2- 172.21.15.40:0/3473095343 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556f8be22b80 0x556f8be24fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:53.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f7658b90640 1 --2- 172.21.15.40:0/3473095343 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556f8be22b80 0x556f8be24fa0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f7d9ac1608128fb server_cookie=68a6da38a1e7b16c in_seq=0 out_seq=0 2024-03-18T22:45:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f7633fff640 10 client.26191 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f7633fff640 1 -- 172.21.15.40:0/3473095343 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556f8bd0dd40 con 0x556f8be22b80 2024-03-18T22:45:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f7633fff640 10 client.26191 handle_command_reply: tid=0 2024-03-18T22:45:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f765d9dd740 1 client.26191 shutdown 2024-03-18T22:45:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f765d9dd740 1 -- 172.21.15.40:0/3473095343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 msgr2=0x556f8be19960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3473095343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 0x556f8be19960 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f76400b4640 tx=0x7f7640005b10 comp rx=0 tx=0).stop 2024-03-18T22:45:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f765d9dd740 1 -- 172.21.15.40:0/3473095343 shutdown_connections 2024-03-18T22:45:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3473095343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556f8bda7e50 0x556f8be209a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3473095343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556f8bda2ce0 0x556f8be19960 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3473095343 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556f8be22b80 0x556f8be24fa0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f765d9dd740 1 --2- 172.21.15.40:0/3473095343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f8bda22f0 0x556f8be19420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f7633fff640 0 client.26191 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:53.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f765d9dd740 1 -- 172.21.15.40:0/3473095343 >> 172.21.15.40:0/3473095343 conn(0x556f8bd9d860 msgr2=0x556f8bd9c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f765d9dd740 1 -- 172.21.15.40:0/3473095343 shutdown_connections 2024-03-18T22:45:53.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f765d9dd740 1 -- 172.21.15.40:0/3473095343 wait complete. 2024-03-18T22:45:53.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.585+0000 7f765d9dd740 20 client.26191 trim_cache size 0 max 16384 2024-03-18T22:45:53.593 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:45:53.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.586+0000 7f765adf4640 1 -- 172.21.15.40:0/3709606092 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f76440a2800 msgr2=0x7f76440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.586+0000 7f765adf4640 1 --2- 172.21.15.40:0/3709606092 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f76440a2800 0x7f76440a4cc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f764c005fd0 tx=0x7f764c009d90 comp rx=0 tx=0).stop 2024-03-18T22:45:53.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.586+0000 7f765adf4640 1 -- 172.21.15.40:0/3709606092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7654101910 msgr2=0x7f7654197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:53.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.586+0000 7f765adf4640 1 --2- 172.21.15.40:0/3709606092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7654101910 0x7f7654197fb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f764000e9f0 tx=0x7f764000eeb0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.586+0000 7f765adf4640 1 -- 172.21.15.40:0/3709606092 shutdown_connections 2024-03-18T22:45:53.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.586+0000 7f765adf4640 1 --2- 172.21.15.40:0/3709606092 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f76440a2800 0x7f76440a4cc0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.586+0000 7f765adf4640 1 --2- 172.21.15.40:0/3709606092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76541022d0 0x7f765419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.586+0000 7f765adf4640 1 --2- 172.21.15.40:0/3709606092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7654101910 0x7f7654197fb0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.586+0000 7f765adf4640 1 --2- 172.21.15.40:0/3709606092 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7654100720 0x7f7654197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:53.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.586+0000 7f765adf4640 1 -- 172.21.15.40:0/3709606092 >> 172.21.15.40:0/3709606092 conn(0x7f76540fbc90 msgr2=0x7f7654068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:53.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.586+0000 7f765adf4640 1 -- 172.21.15.40:0/3709606092 shutdown_connections 2024-03-18T22:45:53.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:53.586+0000 7f765adf4640 1 -- 172.21.15.40:0/3709606092 wait complete. 2024-03-18T22:45:54.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:54 smithi113 ceph-mon[66775]: pgmap v448: 129 pgs: 129 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:45:54.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:54 smithi103 ceph-mon[66377]: pgmap v448: 129 pgs: 129 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:45:54.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:54 smithi040 ceph-mon[62643]: pgmap v448: 129 pgs: 129 active+clean; 2.8 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:45:54.602 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:45:54.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:45:54.603 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:45:54.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.716+0000 7fb961c27640 1 -- 172.21.15.40:0/787002187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c100720 msgr2=0x7fb95c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.716+0000 7fb961c27640 1 --2- 172.21.15.40:0/787002187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c100720 0x7fb95c100b40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009900 tx=0x7fb94c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:54.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.716+0000 7fb961c27640 1 -- 172.21.15.40:0/787002187 shutdown_connections 2024-03-18T22:45:54.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.716+0000 7fb961c27640 1 --2- 172.21.15.40:0/787002187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb95c1022d0 0x7fb95c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.716+0000 7fb961c27640 1 --2- 172.21.15.40:0/787002187 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95c101910 0x7fb95c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.716+0000 7fb961c27640 1 --2- 172.21.15.40:0/787002187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c100720 0x7fb95c100b40 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.716+0000 7fb961c27640 1 -- 172.21.15.40:0/787002187 >> 172.21.15.40:0/787002187 conn(0x7fb95c0fbcb0 msgr2=0x7fb95c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.716+0000 7fb961c27640 1 -- 172.21.15.40:0/787002187 shutdown_connections 2024-03-18T22:45:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.716+0000 7fb961c27640 1 -- 172.21.15.40:0/787002187 wait complete. 2024-03-18T22:45:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.716+0000 7fb961c27640 1 Processor -- start 2024-03-18T22:45:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb961c27640 1 -- start start 2024-03-18T22:45:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb961c27640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb95c100720 0x7fb95c1046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb961c27640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95c101910 0x7fb95c102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb961c27640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c1022d0 0x7fb95c103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb961c27640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c10c240 con 0x7fb95c1022d0 2024-03-18T22:45:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb961c27640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c10c0c0 con 0x7fb95c100720 2024-03-18T22:45:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb961c27640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c10c3c0 con 0x7fb95c101910 2024-03-18T22:45:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb95bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c1022d0 0x7fb95c103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb95affd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95c101910 0x7fb95c102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb95bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c1022d0 0x7fb95c103230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50940/0 (socket says 172.21.15.40:50940) 2024-03-18T22:45:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb95affd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95c101910 0x7fb95c102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:36672/0 (socket says 172.21.15.40:36672) 2024-03-18T22:45:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb95bfff640 1 -- 172.21.15.40:0/3361069921 learned_addr learned my addr 172.21.15.40:0/3361069921 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb95b7fe640 1 --2- 172.21.15.40:0/3361069921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb95c100720 0x7fb95c1046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb95bfff640 1 -- 172.21.15.40:0/3361069921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95c101910 msgr2=0x7fb95c102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb95bfff640 1 --2- 172.21.15.40:0/3361069921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95c101910 0x7fb95c102cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb95bfff640 1 -- 172.21.15.40:0/3361069921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb95c100720 msgr2=0x7fb95c1046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb95bfff640 1 --2- 172.21.15.40:0/3361069921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb95c100720 0x7fb95c1046b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb95bfff640 1 -- 172.21.15.40:0/3361069921 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95c103ac0 con 0x7fb95c1022d0 2024-03-18T22:45:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb95affd640 1 --2- 172.21.15.40:0/3361069921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95c101910 0x7fb95c102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb95b7fe640 1 --2- 172.21.15.40:0/3361069921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb95c100720 0x7fb95c1046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.717+0000 7fb95bfff640 1 --2- 172.21.15.40:0/3361069921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c1022d0 0x7fb95c103230 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb95400eb10 tx=0x7fb95400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.718+0000 7fb958ff9640 1 -- 172.21.15.40:0/3361069921 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954004b20 con 0x7fb95c1022d0 2024-03-18T22:45:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.718+0000 7fb958ff9640 1 -- 172.21.15.40:0/3361069921 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb954004cc0 con 0x7fb95c1022d0 2024-03-18T22:45:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.718+0000 7fb961c27640 1 -- 172.21.15.40:0/3361069921 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb95c1a1c10 con 0x7fb95c1022d0 2024-03-18T22:45:54.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.718+0000 7fb958ff9640 1 -- 172.21.15.40:0/3361069921 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954010690 con 0x7fb95c1022d0 2024-03-18T22:45:54.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.718+0000 7fb961c27640 1 -- 172.21.15.40:0/3361069921 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb95c1a2160 con 0x7fb95c1022d0 2024-03-18T22:45:54.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb958ff9640 1 -- 172.21.15.40:0/3361069921 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb954005020 con 0x7fb95c1022d0 2024-03-18T22:45:54.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb964813740 1 Processor -- start 2024-03-18T22:45:54.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb964813740 1 -- start start 2024-03-18T22:45:54.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb964813740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1e9f3a0 0x5618b1e9f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb964813740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 0x5618b1ea4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb964813740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1ea4f00 0x5618b1ea7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb964813740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5618b1e04160 con 0x5618b1ea4f00 2024-03-18T22:45:54.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb964813740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5618b1dace10 con 0x5618b1e9fd90 2024-03-18T22:45:54.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb964813740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5618b1e00200 con 0x5618b1e9f3a0 2024-03-18T22:45:54.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb958ff9640 1 --2- 172.21.15.40:0/3361069921 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9440a2880 0x7fb9440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb95b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 0x5618b1ea4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb95affd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1e9f3a0 0x5618b1e9f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb95b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 0x5618b1ea4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36938/0 (socket says 172.21.15.40:36938) 2024-03-18T22:45:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb95affd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1e9f3a0 0x5618b1e9f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:36676/0 (socket says 172.21.15.40:36676) 2024-03-18T22:45:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb95b7fe640 1 -- 172.21.15.40:0/1080186635 learned_addr learned my addr 172.21.15.40:0/1080186635 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.721+0000 7fb95affd640 1 --2- 172.21.15.40:0/1080186635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1ea4f00 0x5618b1ea7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.722+0000 7fb95bfff640 1 --2- 172.21.15.40:0/3361069921 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9440a2880 0x7fb9440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.722+0000 7fb958ff9640 1 -- 172.21.15.40:0/3361069921 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb9540cf3d0 con 0x7fb95c1022d0 2024-03-18T22:45:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.722+0000 7fb95affd640 1 -- 172.21.15.40:0/1080186635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 msgr2=0x5618b1ea4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.722+0000 7fb95affd640 1 --2- 172.21.15.40:0/1080186635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 0x5618b1ea4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.722+0000 7fb95affd640 1 -- 172.21.15.40:0/1080186635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1ea4f00 msgr2=0x5618b1ea7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.722+0000 7fb95affd640 1 --2- 172.21.15.40:0/1080186635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1ea4f00 0x5618b1ea7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.722+0000 7fb95affd640 1 -- 172.21.15.40:0/1080186635 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618b1be07e0 con 0x5618b1e9f3a0 2024-03-18T22:45:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.722+0000 7fb95bfff640 1 --2- 172.21.15.40:0/3361069921 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9440a2880 0x7fb9440a4d40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb95409b680 tx=0x7fb95401d000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.722+0000 7fb95affd640 1 --2- 172.21.15.40:0/1080186635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1e9f3a0 0x5618b1e9f7c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb95c104420 tx=0x7fb950005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d0b603e578a7f12 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.722+0000 7fb941ffb640 1 -- 172.21.15.40:0/1080186635 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb950011050 con 0x5618b1e9f3a0 2024-03-18T22:45:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.722+0000 7fb941ffb640 1 -- 172.21.15.40:0/1080186635 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb95000fd30 con 0x5618b1e9f3a0 2024-03-18T22:45:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.722+0000 7fb941ffb640 1 -- 172.21.15.40:0/1080186635 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9500047a0 con 0x5618b1e9f3a0 2024-03-18T22:45:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.723+0000 7fb964813740 1 -- 172.21.15.40:0/1080186635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1e9f3a0 msgr2=0x5618b1e9f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.723+0000 7fb964813740 1 --2- 172.21.15.40:0/1080186635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1e9f3a0 0x5618b1e9f7c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb95c104420 tx=0x7fb950005850 comp rx=0 tx=0).stop 2024-03-18T22:45:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.723+0000 7fb964813740 1 -- 172.21.15.40:0/1080186635 shutdown_connections 2024-03-18T22:45:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.723+0000 7fb964813740 1 --2- 172.21.15.40:0/1080186635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1ea4f00 0x5618b1ea7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.723+0000 7fb964813740 1 --2- 172.21.15.40:0/1080186635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 0x5618b1ea4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.723+0000 7fb964813740 1 --2- 172.21.15.40:0/1080186635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1e9f3a0 0x5618b1e9f7c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.723+0000 7fb964813740 1 -- 172.21.15.40:0/1080186635 >> 172.21.15.40:0/1080186635 conn(0x5618b1e9a910 msgr2=0x5618b1e9ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.723+0000 7fb964813740 1 -- 172.21.15.40:0/1080186635 shutdown_connections 2024-03-18T22:45:54.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.723+0000 7fb964813740 1 -- 172.21.15.40:0/1080186635 wait complete. 2024-03-18T22:45:54.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.723+0000 7fb964813740 1 -- start start 2024-03-18T22:45:54.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb964813740 1 Processor -- start 2024-03-18T22:45:54.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb964813740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9f3a0 0x5618b1ea9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb964813740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1e9fd90 0x5618b1ea9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb964813740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1ea4f00 0x5618b1eb0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb964813740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5618b1ea7eb0 con 0x5618b1e9fd90 2024-03-18T22:45:54.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb964813740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5618b1dc01a0 con 0x5618b1e9f3a0 2024-03-18T22:45:54.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb964813740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5618b1ea8030 con 0x5618b1ea4f00 2024-03-18T22:45:54.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb964813740 10 client.0 _ll_register_callbacks cb 0x5618b1b17c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:54.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb95b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1e9fd90 0x5618b1ea9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb95affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9f3a0 0x5618b1ea9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb95affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9f3a0 0x5618b1ea9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36944/0 (socket says 172.21.15.40:36944) 2024-03-18T22:45:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb95affd640 1 -- 172.21.15.40:0/3895300269 learned_addr learned my addr 172.21.15.40:0/3895300269 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb95b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1e9fd90 0x5618b1ea9e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50954/0 (socket says 172.21.15.40:50954) 2024-03-18T22:45:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb95affd640 1 --2- 172.21.15.40:0/3895300269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1ea4f00 0x5618b1eb0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb95b7fe640 1 -- 172.21.15.40:0/3895300269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1ea4f00 msgr2=0x5618b1eb0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb95b7fe640 1 --2- 172.21.15.40:0/3895300269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1ea4f00 0x5618b1eb0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb95b7fe640 1 -- 172.21.15.40:0/3895300269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9f3a0 msgr2=0x5618b1ea9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb95b7fe640 1 --2- 172.21.15.40:0/3895300269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9f3a0 0x5618b1ea9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb95b7fe640 1 -- 172.21.15.40:0/3895300269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618b1c40f70 con 0x5618b1e9fd90 2024-03-18T22:45:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.724+0000 7fb95affd640 1 --2- 172.21.15.40:0/3895300269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9f3a0 0x5618b1ea9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.725+0000 7fb95b7fe640 1 --2- 172.21.15.40:0/3895300269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1e9fd90 0x5618b1ea9e50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x5618b1ea0800 tx=0x7fb94c005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9922ac3102270d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.725+0000 7fb93affd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.725+0000 7fb93affd640 1 -- 172.21.15.40:0/3895300269 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c005d50 con 0x5618b1e9fd90 2024-03-18T22:45:54.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.725+0000 7fb93affd640 1 -- 172.21.15.40:0/3895300269 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb94c0046c0 con 0x5618b1e9fd90 2024-03-18T22:45:54.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.725+0000 7fb964813740 1 -- 172.21.15.40:0/3895300269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb95000fd30 con 0x5618b1e9fd90 2024-03-18T22:45:54.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.725+0000 7fb93affd640 1 -- 172.21.15.40:0/3895300269 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb94c02cc90 con 0x5618b1e9fd90 2024-03-18T22:45:54.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.725+0000 7fb964813740 10 client.16380 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:54.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.725+0000 7fb964813740 1 -- 172.21.15.40:0/3895300269 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5618b1ad8170 con 0x5618b1e9fd90 2024-03-18T22:45:54.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.725+0000 7fb93affd640 1 -- 172.21.15.40:0/3895300269 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb95000fd30 con 0x5618b1e9fd90 2024-03-18T22:45:54.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.725+0000 7fb93affd640 1 -- 172.21.15.40:0/3895300269 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fb94c0038f0 con 0x5618b1e9fd90 2024-03-18T22:45:54.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.726+0000 7fb964813740 10 client.16380 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:54.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.726+0000 7fb964813740 10 client.16380 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:54.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.726+0000 7fb964813740 20 client.16380 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:54.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.726+0000 7fb964813740 1 --2- 172.21.15.40:0/3895300269 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5618b1eb56e0 0x5618b1eb7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.726+0000 7fb964813740 4 client.16380 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:54.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.726+0000 7fb964813740 1 -- 172.21.15.40:0/3895300269 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618b1e116a0 con 0x5618b1eb56e0 2024-03-18T22:45:54.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.726+0000 7fb95affd640 1 --2- 172.21.15.40:0/3895300269 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5618b1eb56e0 0x5618b1eb7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.726+0000 7fb93affd640 10 client.16380 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:54.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.726+0000 7fb95affd640 1 --2- 172.21.15.40:0/3895300269 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5618b1eb56e0 0x5618b1eb7b60 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b08591fe60543153 server_cookie=53c748cb31dc89dc in_seq=0 out_seq=0 2024-03-18T22:45:54.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.727+0000 7fb93affd640 1 -- 172.21.15.40:0/3895300269 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5618b1e116a0 con 0x5618b1eb56e0 2024-03-18T22:45:54.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.727+0000 7fb93affd640 10 client.16380 handle_command_reply: tid=0 2024-03-18T22:45:54.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.727+0000 7fb964813740 1 client.16380 shutdown 2024-03-18T22:45:54.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.727+0000 7fb964813740 1 -- 172.21.15.40:0/3895300269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1e9fd90 msgr2=0x5618b1ea9e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.727+0000 7fb964813740 1 --2- 172.21.15.40:0/3895300269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1e9fd90 0x5618b1ea9e50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x5618b1ea0800 tx=0x7fb94c005270 comp rx=0 tx=0).stop 2024-03-18T22:45:54.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.727+0000 7fb964813740 1 -- 172.21.15.40:0/3895300269 shutdown_connections 2024-03-18T22:45:54.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.727+0000 7fb964813740 1 --2- 172.21.15.40:0/3895300269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1ea4f00 0x5618b1eb0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.727+0000 7fb964813740 1 --2- 172.21.15.40:0/3895300269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1e9fd90 0x5618b1ea9e50 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.727+0000 7fb964813740 1 --2- 172.21.15.40:0/3895300269 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5618b1eb56e0 0x5618b1eb7b60 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.727+0000 7fb964813740 1 --2- 172.21.15.40:0/3895300269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9f3a0 0x5618b1ea9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.727+0000 7fb964813740 1 -- 172.21.15.40:0/3895300269 >> 172.21.15.40:0/3895300269 conn(0x5618b1e9a910 msgr2=0x5618b1e99b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:54.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.727+0000 7fb93affd640 0 client.16380 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:54.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.727+0000 7fb964813740 1 -- 172.21.15.40:0/3895300269 shutdown_connections 2024-03-18T22:45:54.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.727+0000 7fb964813740 1 -- 172.21.15.40:0/3895300269 wait complete. 2024-03-18T22:45:54.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.727+0000 7fb964813740 20 client.16380 trim_cache size 0 max 16384 2024-03-18T22:45:54.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.750+0000 7fb964813740 1 Processor -- start 2024-03-18T22:45:54.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.750+0000 7fb964813740 1 -- start start 2024-03-18T22:45:54.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.750+0000 7fb964813740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1e9f3a0 0x5618b1ea12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.750+0000 7fb964813740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 0x5618b1eb0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.750+0000 7fb964813740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1ea4f00 0x5618b1ea1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.750+0000 7fb964813740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c1ac770 con 0x5618b1e9f3a0 2024-03-18T22:45:54.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.750+0000 7fb964813740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c1ae370 con 0x5618b1e9fd90 2024-03-18T22:45:54.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.750+0000 7fb964813740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c1ace40 con 0x5618b1ea4f00 2024-03-18T22:45:54.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.750+0000 7fb95affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1e9f3a0 0x5618b1ea12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.750+0000 7fb95affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1e9f3a0 0x5618b1ea12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50960/0 (socket says 172.21.15.40:50960) 2024-03-18T22:45:54.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.750+0000 7fb95affd640 1 -- 172.21.15.40:0/2889589611 learned_addr learned my addr 172.21.15.40:0/2889589611 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:54.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.750+0000 7fb95b7fe640 1 --2- 172.21.15.40:0/2889589611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 0x5618b1eb0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.751+0000 7fb95affd640 1 --2- 172.21.15.40:0/2889589611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1ea4f00 0x5618b1ea1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.751+0000 7fb95affd640 1 -- 172.21.15.40:0/2889589611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1ea4f00 msgr2=0x5618b1ea1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.751+0000 7fb95affd640 1 --2- 172.21.15.40:0/2889589611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1ea4f00 0x5618b1ea1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.751+0000 7fb95affd640 1 -- 172.21.15.40:0/2889589611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 msgr2=0x5618b1eb0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.751+0000 7fb95affd640 1 --2- 172.21.15.40:0/2889589611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 0x5618b1eb0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.751+0000 7fb95affd640 1 -- 172.21.15.40:0/2889589611 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618b1ea7750 con 0x5618b1e9f3a0 2024-03-18T22:45:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.751+0000 7fb95affd640 1 --2- 172.21.15.40:0/2889589611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1e9f3a0 0x5618b1ea12d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb950011af0 tx=0x7fb950009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=868d538d369da351 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.751+0000 7fb9417fa640 1 -- 172.21.15.40:0/2889589611 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb95000fb90 con 0x5618b1e9f3a0 2024-03-18T22:45:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.751+0000 7fb9417fa640 1 -- 172.21.15.40:0/2889589611 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb95000fd30 con 0x5618b1e9f3a0 2024-03-18T22:45:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.751+0000 7fb9417fa640 1 -- 172.21.15.40:0/2889589611 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9500039e0 con 0x5618b1e9f3a0 2024-03-18T22:45:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.752+0000 7fb964813740 1 -- 172.21.15.40:0/2889589611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1e9f3a0 msgr2=0x5618b1ea12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.752+0000 7fb964813740 1 --2- 172.21.15.40:0/2889589611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1e9f3a0 0x5618b1ea12d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb950011af0 tx=0x7fb950009eb0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.752+0000 7fb964813740 1 -- 172.21.15.40:0/2889589611 shutdown_connections 2024-03-18T22:45:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.752+0000 7fb964813740 1 --2- 172.21.15.40:0/2889589611 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1ea4f00 0x5618b1ea1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.752+0000 7fb964813740 1 --2- 172.21.15.40:0/2889589611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 0x5618b1eb0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.752+0000 7fb964813740 1 --2- 172.21.15.40:0/2889589611 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1e9f3a0 0x5618b1ea12d0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.752+0000 7fb964813740 1 -- 172.21.15.40:0/2889589611 >> 172.21.15.40:0/2889589611 conn(0x5618b1e9a910 msgr2=0x5618b1ea8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.752+0000 7fb964813740 1 -- 172.21.15.40:0/2889589611 shutdown_connections 2024-03-18T22:45:54.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.752+0000 7fb964813740 1 -- 172.21.15.40:0/2889589611 wait complete. 2024-03-18T22:45:54.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.752+0000 7fb964813740 1 -- start start 2024-03-18T22:45:54.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.753+0000 7fb964813740 1 Processor -- start 2024-03-18T22:45:54.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.753+0000 7fb964813740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1e9f3a0 0x5618b1eae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.753+0000 7fb964813740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 0x5618b1eae820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.753+0000 7fb964813740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1ea4f00 0x5618b1eaf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.753+0000 7fb964813740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c1adbe0 con 0x5618b1ea4f00 2024-03-18T22:45:54.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.753+0000 7fb964813740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c1ad510 con 0x5618b1e9fd90 2024-03-18T22:45:54.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.753+0000 7fb964813740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c1aa750 con 0x5618b1e9f3a0 2024-03-18T22:45:54.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.753+0000 7fb964813740 10 client.0 _ll_register_callbacks cb 0x7fb94c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:54.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.753+0000 7fb95affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1ea4f00 0x5618b1eaf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.753+0000 7fb95affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1ea4f00 0x5618b1eaf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50972/0 (socket says 172.21.15.40:50972) 2024-03-18T22:45:54.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.753+0000 7fb95affd640 1 -- 172.21.15.40:0/1641235348 learned_addr learned my addr 172.21.15.40:0/1641235348 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:54.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.753+0000 7fb95b7fe640 1 --2- 172.21.15.40:0/1641235348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 0x5618b1eae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.754+0000 7fb95affd640 1 --2- 172.21.15.40:0/1641235348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1e9f3a0 0x5618b1eae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.754+0000 7fb95b7fe640 1 -- 172.21.15.40:0/1641235348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1e9f3a0 msgr2=0x5618b1eae2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.754+0000 7fb95b7fe640 1 --2- 172.21.15.40:0/1641235348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1e9f3a0 0x5618b1eae2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.754+0000 7fb95b7fe640 1 -- 172.21.15.40:0/1641235348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1ea4f00 msgr2=0x5618b1eaf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.754+0000 7fb95b7fe640 1 --2- 172.21.15.40:0/1641235348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1ea4f00 0x5618b1eaf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.754+0000 7fb95b7fe640 1 -- 172.21.15.40:0/1641235348 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618b1eafd80 con 0x5618b1e9fd90 2024-03-18T22:45:54.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.754+0000 7fb95affd640 1 --2- 172.21.15.40:0/1641235348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1ea4f00 0x5618b1eaf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:54.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.754+0000 7fb95b7fe640 1 --2- 172.21.15.40:0/1641235348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 0x5618b1eae820 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb94c02c170 tx=0x7fb94c005820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46eaa2e79130dcc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:54.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.754+0000 7fb93affd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:45:54.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.754+0000 7fb93affd640 1 -- 172.21.15.40:0/1641235348 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c05b070 con 0x5618b1e9fd90 2024-03-18T22:45:54.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.754+0000 7fb93affd640 1 -- 172.21.15.40:0/1641235348 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb94c034710 con 0x5618b1e9fd90 2024-03-18T22:45:54.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.754+0000 7fb964813740 1 -- 172.21.15.40:0/1641235348 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb95000fd30 con 0x5618b1e9fd90 2024-03-18T22:45:54.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.755+0000 7fb93affd640 1 -- 172.21.15.40:0/1641235348 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb94c064050 con 0x5618b1e9fd90 2024-03-18T22:45:54.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.755+0000 7fb964813740 10 client.26219 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:54.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.755+0000 7fb964813740 1 -- 172.21.15.40:0/1641235348 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5618b1eb1130 con 0x5618b1e9fd90 2024-03-18T22:45:54.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.755+0000 7fb93affd640 1 -- 172.21.15.40:0/1641235348 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb95000fd30 con 0x5618b1e9fd90 2024-03-18T22:45:54.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.755+0000 7fb93affd640 1 -- 172.21.15.40:0/1641235348 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7fb94c05e020 con 0x5618b1e9fd90 2024-03-18T22:45:54.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.755+0000 7fb964813740 10 client.26219 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:54.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.755+0000 7fb964813740 10 client.26219 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:54.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.755+0000 7fb964813740 20 client.26219 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:54.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.755+0000 7fb964813740 1 --2- 172.21.15.40:0/1641235348 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5618b1f1c1c0 0x5618b1f1e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.755+0000 7fb964813740 4 client.26219 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:45:54.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.755+0000 7fb964813740 1 -- 172.21.15.40:0/1641235348 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5618b1e0b9f0 con 0x5618b1f1c1c0 2024-03-18T22:45:54.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.756+0000 7fb95affd640 1 --2- 172.21.15.40:0/1641235348 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5618b1f1c1c0 0x5618b1f1e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.756+0000 7fb95affd640 1 --2- 172.21.15.40:0/1641235348 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5618b1f1c1c0 0x5618b1f1e5e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=186b74c60bd16def server_cookie=30af6d77d7be591 in_seq=0 out_seq=0 2024-03-18T22:45:54.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.756+0000 7fb93affd640 10 client.26219 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.757+0000 7fb93affd640 1 -- 172.21.15.40:0/1641235348 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5618b1e0b9f0 con 0x5618b1f1c1c0 2024-03-18T22:45:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.757+0000 7fb93affd640 10 client.26219 handle_command_reply: tid=0 2024-03-18T22:45:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.757+0000 7fb964813740 1 client.26219 shutdown 2024-03-18T22:45:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.757+0000 7fb964813740 1 -- 172.21.15.40:0/1641235348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 msgr2=0x5618b1eae820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.757+0000 7fb964813740 1 --2- 172.21.15.40:0/1641235348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 0x5618b1eae820 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb94c02c170 tx=0x7fb94c005820 comp rx=0 tx=0).stop 2024-03-18T22:45:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.757+0000 7fb964813740 1 -- 172.21.15.40:0/1641235348 shutdown_connections 2024-03-18T22:45:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.757+0000 7fb964813740 1 --2- 172.21.15.40:0/1641235348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5618b1ea4f00 0x5618b1eaf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.757+0000 7fb964813740 1 --2- 172.21.15.40:0/1641235348 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5618b1f1c1c0 0x5618b1f1e5e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.757+0000 7fb964813740 1 --2- 172.21.15.40:0/1641235348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5618b1e9fd90 0x5618b1eae820 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.758+0000 7fb93affd640 0 client.26219 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.758+0000 7fb964813740 1 --2- 172.21.15.40:0/1641235348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5618b1e9f3a0 0x5618b1eae2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.758+0000 7fb964813740 1 -- 172.21.15.40:0/1641235348 >> 172.21.15.40:0/1641235348 conn(0x5618b1e9a910 msgr2=0x5618b1ea98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.758+0000 7fb964813740 1 -- 172.21.15.40:0/1641235348 shutdown_connections 2024-03-18T22:45:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.758+0000 7fb964813740 1 -- 172.21.15.40:0/1641235348 wait complete. 2024-03-18T22:45:54.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.758+0000 7fb964813740 20 client.26219 trim_cache size 0 max 16384 2024-03-18T22:45:54.764 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:45:54.764 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:45:54.765 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "1ba372cc-bdb2-407f-81d2-da1c828b5e36", 2024-03-18T22:45:54.765 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:45:54.765 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:45:54.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.758+0000 7fb961c27640 1 -- 172.21.15.40:0/3361069921 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9440a2880 msgr2=0x7fb9440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.758+0000 7fb961c27640 1 --2- 172.21.15.40:0/3361069921 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9440a2880 0x7fb9440a4d40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb95409b680 tx=0x7fb95401d000 comp rx=0 tx=0).stop 2024-03-18T22:45:54.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.759+0000 7fb961c27640 1 -- 172.21.15.40:0/3361069921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c1022d0 msgr2=0x7fb95c103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.759+0000 7fb961c27640 1 --2- 172.21.15.40:0/3361069921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c1022d0 0x7fb95c103230 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb95400eb10 tx=0x7fb95400c2d0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.759+0000 7fb961c27640 1 -- 172.21.15.40:0/3361069921 shutdown_connections 2024-03-18T22:45:54.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.759+0000 7fb961c27640 1 --2- 172.21.15.40:0/3361069921 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9440a2880 0x7fb9440a4d40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.759+0000 7fb961c27640 1 --2- 172.21.15.40:0/3361069921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb95c1022d0 0x7fb95c103230 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.759+0000 7fb961c27640 1 --2- 172.21.15.40:0/3361069921 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb95c101910 0x7fb95c102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.759+0000 7fb961c27640 1 --2- 172.21.15.40:0/3361069921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb95c100720 0x7fb95c1046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.759+0000 7fb961c27640 1 -- 172.21.15.40:0/3361069921 >> 172.21.15.40:0/3361069921 conn(0x7fb95c0fbcb0 msgr2=0x7fb95c0689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:54.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.759+0000 7fb961c27640 1 -- 172.21.15.40:0/3361069921 shutdown_connections 2024-03-18T22:45:54.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.759+0000 7fb961c27640 1 -- 172.21.15.40:0/3361069921 wait complete. 2024-03-18T22:45:54.772 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:45:54.773 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:45:54.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.893+0000 7f4b3d543640 1 -- 172.21.15.40:0/1376032569 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2802cbe0 con 0x7f4b38101910 2024-03-18T22:45:54.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.893+0000 7f4b407a9640 1 -- 172.21.15.40:0/1376032569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38101910 msgr2=0x7f4b38101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.893+0000 7f4b407a9640 1 --2- 172.21.15.40:0/1376032569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38101910 0x7f4b38101d90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4b28009900 tx=0x7f4b2802bc30 comp rx=0 tx=0).stop 2024-03-18T22:45:54.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.894+0000 7f4b407a9640 1 -- 172.21.15.40:0/1376032569 shutdown_connections 2024-03-18T22:45:54.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.894+0000 7f4b407a9640 1 --2- 172.21.15.40:0/1376032569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b381022d0 0x7f4b38108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.894+0000 7f4b407a9640 1 --2- 172.21.15.40:0/1376032569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38101910 0x7f4b38101d90 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.894+0000 7f4b407a9640 1 --2- 172.21.15.40:0/1376032569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b38100720 0x7f4b38100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.894+0000 7f4b407a9640 1 -- 172.21.15.40:0/1376032569 >> 172.21.15.40:0/1376032569 conn(0x7f4b380fbcb0 msgr2=0x7f4b380fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:54.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.894+0000 7f4b407a9640 1 -- 172.21.15.40:0/1376032569 shutdown_connections 2024-03-18T22:45:54.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.894+0000 7f4b407a9640 1 -- 172.21.15.40:0/1376032569 wait complete. 2024-03-18T22:45:54.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.894+0000 7f4b407a9640 1 Processor -- start 2024-03-18T22:45:54.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.894+0000 7f4b407a9640 1 -- start start 2024-03-18T22:45:54.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b407a9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b38100720 0x7f4b38197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b407a9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38101910 0x7f4b38197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b407a9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b381022d0 0x7f4b3819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b407a9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3810d380 con 0x7f4b38101910 2024-03-18T22:45:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b407a9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3810d200 con 0x7f4b38100720 2024-03-18T22:45:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b407a9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3810d500 con 0x7f4b381022d0 2024-03-18T22:45:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b3dd44640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38101910 0x7f4b38197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b3dd44640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38101910 0x7f4b38197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50990/0 (socket says 172.21.15.40:50990) 2024-03-18T22:45:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b3dd44640 1 -- 172.21.15.40:0/2826141013 learned_addr learned my addr 172.21.15.40:0/2826141013 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b3e545640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b38100720 0x7f4b38197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b3ed46640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b381022d0 0x7f4b3819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b3e545640 1 -- 172.21.15.40:0/2826141013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b381022d0 msgr2=0x7f4b3819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b3e545640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b381022d0 0x7f4b3819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b3e545640 1 -- 172.21.15.40:0/2826141013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38101910 msgr2=0x7f4b38197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b3e545640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38101910 0x7f4b38197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b3e545640 1 -- 172.21.15.40:0/2826141013 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b3819f6d0 con 0x7f4b38100720 2024-03-18T22:45:54.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b3dd44640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38101910 0x7f4b38197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:54.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b3ed46640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b381022d0 0x7f4b3819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:54.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b3e545640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b38100720 0x7f4b38197ab0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400e9c0 tx=0x7f4b3400edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:54.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.895+0000 7f4b2f7fe640 1 -- 172.21.15.40:0/2826141013 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3400ccc0 con 0x7f4b38100720 2024-03-18T22:45:54.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.896+0000 7f4b407a9640 1 -- 172.21.15.40:0/2826141013 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b3819f9c0 con 0x7f4b38100720 2024-03-18T22:45:54.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.896+0000 7f4b2f7fe640 1 -- 172.21.15.40:0/2826141013 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b340035f0 con 0x7f4b38100720 2024-03-18T22:45:54.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.896+0000 7f4b2f7fe640 1 -- 172.21.15.40:0/2826141013 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3400ce60 con 0x7f4b38100720 2024-03-18T22:45:54.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.896+0000 7f4b407a9640 1 -- 172.21.15.40:0/2826141013 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b3819ff60 con 0x7f4b38100720 2024-03-18T22:45:54.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.898+0000 7f4b2f7fe640 1 -- 172.21.15.40:0/2826141013 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4b340037a0 con 0x7f4b38100720 2024-03-18T22:45:54.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.898+0000 7f4b43390740 1 Processor -- start 2024-03-18T22:45:54.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b43390740 1 -- start start 2024-03-18T22:45:54.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b43390740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3ba50 0x56525bd3be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b43390740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd3c440 0x56525bd40f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b43390740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd415b0 0x56525bd43a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b43390740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56525bca0500 con 0x56525bd415b0 2024-03-18T22:45:54.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b43390740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56525bc48be0 con 0x56525bd3c440 2024-03-18T22:45:54.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b43390740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56525bc9c5a0 con 0x56525bd3ba50 2024-03-18T22:45:54.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b3e545640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd3c440 0x56525bd40f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b3dd44640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd415b0 0x56525bd43a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b3ed46640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3ba50 0x56525bd3be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b2f7fe640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b240a2800 0x7f4b240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b3e545640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd3c440 0x56525bd40f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36986/0 (socket says 172.21.15.40:36986) 2024-03-18T22:45:54.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b3e545640 1 -- 172.21.15.40:0/2628924514 learned_addr learned my addr 172.21.15.40:0/2628924514 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:54.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b3ed46640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3ba50 0x56525bd3be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:36720/0 (socket says 172.21.15.40:36720) 2024-03-18T22:45:54.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b3dd44640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd415b0 0x56525bd43a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51002/0 (socket says 172.21.15.40:51002) 2024-03-18T22:45:54.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b3ed46640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b240a2800 0x7f4b240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b2f7fe640 1 -- 172.21.15.40:0/2826141013 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4b340cf200 con 0x7f4b38100720 2024-03-18T22:45:54.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b3e545640 1 -- 172.21.15.40:0/2628924514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3ba50 msgr2=0x56525bd3be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b3e545640 1 --2- 172.21.15.40:0/2628924514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3ba50 0x56525bd3be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b3e545640 1 -- 172.21.15.40:0/2628924514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd415b0 msgr2=0x56525bd43a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b3e545640 1 --2- 172.21.15.40:0/2628924514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd415b0 0x56525bd43a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b3e545640 1 -- 172.21.15.40:0/2628924514 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56525bc6a000 con 0x56525bd3c440 2024-03-18T22:45:54.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.899+0000 7f4b3ed46640 1 --2- 172.21.15.40:0/2628924514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3ba50 0x56525bd3be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:54.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.900+0000 7f4b3e545640 1 --2- 172.21.15.40:0/2628924514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd3c440 0x56525bd40f40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400c6a0 tx=0x7f4b340046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2dbcd36cd8f8d9e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:54.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.900+0000 7f4b3ed46640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b240a2800 0x7f4b240a4cc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x56525bd3cca0 tx=0x7f4b30004ab0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:54.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.900+0000 7f4b2cff9640 1 -- 172.21.15.40:0/2628924514 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b340ace60 con 0x56525bd3c440 2024-03-18T22:45:54.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.900+0000 7f4b2cff9640 1 -- 172.21.15.40:0/2628924514 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b34003b10 con 0x56525bd3c440 2024-03-18T22:45:54.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.900+0000 7f4b2cff9640 1 -- 172.21.15.40:0/2628924514 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b34086070 con 0x56525bd3c440 2024-03-18T22:45:54.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.900+0000 7f4b43390740 1 -- 172.21.15.40:0/2628924514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd3c440 msgr2=0x56525bd40f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.900+0000 7f4b43390740 1 --2- 172.21.15.40:0/2628924514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd3c440 0x56525bd40f40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400c6a0 tx=0x7f4b340046c0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.900+0000 7f4b43390740 1 -- 172.21.15.40:0/2628924514 shutdown_connections 2024-03-18T22:45:54.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.900+0000 7f4b43390740 1 --2- 172.21.15.40:0/2628924514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd415b0 0x56525bd43a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.900+0000 7f4b43390740 1 --2- 172.21.15.40:0/2628924514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd3c440 0x56525bd40f40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.900+0000 7f4b43390740 1 --2- 172.21.15.40:0/2628924514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3ba50 0x56525bd3be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b43390740 1 -- 172.21.15.40:0/2628924514 >> 172.21.15.40:0/2628924514 conn(0x56525bd36fc0 msgr2=0x56525bd373c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:54.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b43390740 1 -- 172.21.15.40:0/2628924514 shutdown_connections 2024-03-18T22:45:54.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b43390740 1 -- 172.21.15.40:0/2628924514 wait complete. 2024-03-18T22:45:54.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b43390740 1 -- start start 2024-03-18T22:45:54.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b43390740 1 Processor -- start 2024-03-18T22:45:54.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b43390740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3ba50 0x56525bd45eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b43390740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3c440 0x56525bd463f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b43390740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 0x56525bd4d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b43390740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56525bd44500 con 0x56525bd3ba50 2024-03-18T22:45:54.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b43390740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56525bc5c160 con 0x56525bd415b0 2024-03-18T22:45:54.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b43390740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56525bd44680 con 0x56525bd3c440 2024-03-18T22:45:54.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b3e545640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3ba50 0x56525bd45eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b43390740 10 client.0 _ll_register_callbacks cb 0x56525bc98a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:54.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b3e545640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3ba50 0x56525bd45eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51004/0 (socket says 172.21.15.40:51004) 2024-03-18T22:45:54.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b3e545640 1 -- 172.21.15.40:0/3970518241 learned_addr learned my addr 172.21.15.40:0/3970518241 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:54.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b3ed46640 1 --2- 172.21.15.40:0/3970518241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 0x56525bd4d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.901+0000 7f4b3dd44640 1 --2- 172.21.15.40:0/3970518241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3c440 0x56525bd463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.902+0000 7f4b3e545640 1 -- 172.21.15.40:0/3970518241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3c440 msgr2=0x56525bd463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.902+0000 7f4b3e545640 1 --2- 172.21.15.40:0/3970518241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3c440 0x56525bd463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.902+0000 7f4b3e545640 1 -- 172.21.15.40:0/3970518241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 msgr2=0x56525bd4d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.902+0000 7f4b3e545640 1 --2- 172.21.15.40:0/3970518241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 0x56525bd4d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.902+0000 7f4b3e545640 1 -- 172.21.15.40:0/3970518241 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56525ba7e7d0 con 0x56525bd3ba50 2024-03-18T22:45:54.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.902+0000 7f4b3ed46640 1 --2- 172.21.15.40:0/3970518241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 0x56525bd4d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:54.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.902+0000 7f4b3e545640 1 --2- 172.21.15.40:0/3970518241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3ba50 0x56525bd45eb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4b34097c10 tx=0x7f4b3409b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648c85ba790ec6cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:54.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.902+0000 7f4b21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:45:54.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.902+0000 7f4b21ffb640 1 -- 172.21.15.40:0/3970518241 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b340afb90 con 0x56525bd3ba50 2024-03-18T22:45:54.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.902+0000 7f4b21ffb640 1 -- 172.21.15.40:0/3970518241 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4b340afd30 con 0x56525bd3ba50 2024-03-18T22:45:54.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.902+0000 7f4b43390740 1 -- 172.21.15.40:0/3970518241 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56525bcada40 con 0x56525bd3ba50 2024-03-18T22:45:54.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.902+0000 7f4b21ffb640 1 -- 172.21.15.40:0/3970518241 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4b3409ac00 con 0x56525bd3ba50 2024-03-18T22:45:54.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.902+0000 7f4b43390740 10 client.16398 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:54.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.902+0000 7f4b43390740 1 -- 172.21.15.40:0/3970518241 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56525b9b43f0 con 0x56525bd3ba50 2024-03-18T22:45:54.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.902+0000 7f4b21ffb640 1 -- 172.21.15.40:0/3970518241 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56525bcada40 con 0x56525bd3ba50 2024-03-18T22:45:54.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.903+0000 7f4b21ffb640 1 -- 172.21.15.40:0/3970518241 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f4b34097700 con 0x56525bd3ba50 2024-03-18T22:45:54.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.903+0000 7f4b43390740 10 client.16398 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:54.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.903+0000 7f4b43390740 10 client.16398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:54.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.903+0000 7f4b43390740 20 client.16398 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:54.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.903+0000 7f4b43390740 1 --2- 172.21.15.40:0/3970518241 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56525bd51920 0x56525bd53d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.903+0000 7f4b43390740 4 client.16398 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:45:54.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.903+0000 7f4b43390740 1 -- 172.21.15.40:0/3970518241 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56525bca7d90 con 0x56525bd51920 2024-03-18T22:45:54.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.903+0000 7f4b3dd44640 1 --2- 172.21.15.40:0/3970518241 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56525bd51920 0x56525bd53d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.903+0000 7f4b3dd44640 1 --2- 172.21.15.40:0/3970518241 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56525bd51920 0x56525bd53d10 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2273e0c1c182a312 server_cookie=e8d8f3fbd393fa2d in_seq=0 out_seq=0 2024-03-18T22:45:54.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.903+0000 7f4b21ffb640 10 client.16398 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:54.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.904+0000 7f4b21ffb640 1 -- 172.21.15.40:0/3970518241 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56525bca7d90 con 0x56525bd51920 2024-03-18T22:45:54.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.904+0000 7f4b21ffb640 10 client.16398 handle_command_reply: tid=0 2024-03-18T22:45:54.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.904+0000 7f4b43390740 1 client.16398 shutdown 2024-03-18T22:45:54.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.904+0000 7f4b43390740 1 -- 172.21.15.40:0/3970518241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3ba50 msgr2=0x56525bd45eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.904+0000 7f4b43390740 1 --2- 172.21.15.40:0/3970518241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3ba50 0x56525bd45eb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4b34097c10 tx=0x7f4b3409b7b0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.904+0000 7f4b43390740 1 -- 172.21.15.40:0/3970518241 shutdown_connections 2024-03-18T22:45:54.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.904+0000 7f4b43390740 1 --2- 172.21.15.40:0/3970518241 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56525bd51920 0x56525bd53d10 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.904+0000 7f4b43390740 1 --2- 172.21.15.40:0/3970518241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 0x56525bd4d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.904+0000 7f4b43390740 1 --2- 172.21.15.40:0/3970518241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3c440 0x56525bd463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.904+0000 7f4b43390740 1 --2- 172.21.15.40:0/3970518241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3ba50 0x56525bd45eb0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.904+0000 7f4b43390740 1 -- 172.21.15.40:0/3970518241 >> 172.21.15.40:0/3970518241 conn(0x56525bd36fc0 msgr2=0x56525bd361e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:54.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.904+0000 7f4b21ffb640 0 client.16398 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:54.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.904+0000 7f4b43390740 1 -- 172.21.15.40:0/3970518241 shutdown_connections 2024-03-18T22:45:54.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.904+0000 7f4b43390740 1 -- 172.21.15.40:0/3970518241 wait complete. 2024-03-18T22:45:54.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.904+0000 7f4b43390740 20 client.16398 trim_cache size 0 max 16384 2024-03-18T22:45:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.929+0000 7f4b43390740 1 Processor -- start 2024-03-18T22:45:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b43390740 1 -- start start 2024-03-18T22:45:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b43390740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3ba50 0x56525bd461d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b43390740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3c440 0x56525bd4d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b43390740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 0x56525bd35950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b43390740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b381aa5d0 con 0x56525bd3ba50 2024-03-18T22:45:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b43390740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4b381ac1d0 con 0x56525bd415b0 2024-03-18T22:45:54.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b43390740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4b381aaca0 con 0x56525bd3c440 2024-03-18T22:45:54.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b3e545640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3ba50 0x56525bd461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b3e545640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3ba50 0x56525bd461d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51014/0 (socket says 172.21.15.40:51014) 2024-03-18T22:45:54.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b3e545640 1 -- 172.21.15.40:0/3088042307 learned_addr learned my addr 172.21.15.40:0/3088042307 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:54.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b3dd44640 1 --2- 172.21.15.40:0/3088042307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3c440 0x56525bd4d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b3ed46640 1 --2- 172.21.15.40:0/3088042307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 0x56525bd35950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b3e545640 1 -- 172.21.15.40:0/3088042307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3c440 msgr2=0x56525bd4d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b3e545640 1 --2- 172.21.15.40:0/3088042307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3c440 0x56525bd4d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b3e545640 1 -- 172.21.15.40:0/3088042307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 msgr2=0x56525bd35950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b3e545640 1 --2- 172.21.15.40:0/3088042307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 0x56525bd35950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b3e545640 1 -- 172.21.15.40:0/3088042307 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56525b9740f0 con 0x56525bd3ba50 2024-03-18T22:45:54.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.930+0000 7f4b3e545640 1 --2- 172.21.15.40:0/3088042307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3ba50 0x56525bd461d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4b340cf1d0 tx=0x7f4b3409a9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1d8d5b3fe10d71 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:54.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.931+0000 7f4b23fff640 1 -- 172.21.15.40:0/3088042307 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3409b580 con 0x56525bd3ba50 2024-03-18T22:45:54.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.931+0000 7f4b23fff640 1 -- 172.21.15.40:0/3088042307 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b340a3f50 con 0x56525bd3ba50 2024-03-18T22:45:54.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.931+0000 7f4b23fff640 1 -- 172.21.15.40:0/3088042307 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3409b2f0 con 0x56525bd3ba50 2024-03-18T22:45:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.931+0000 7f4b43390740 1 -- 172.21.15.40:0/3088042307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3ba50 msgr2=0x56525bd461d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.931+0000 7f4b43390740 1 --2- 172.21.15.40:0/3088042307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3ba50 0x56525bd461d0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4b340cf1d0 tx=0x7f4b3409a9d0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.931+0000 7f4b43390740 1 -- 172.21.15.40:0/3088042307 shutdown_connections 2024-03-18T22:45:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.931+0000 7f4b43390740 1 --2- 172.21.15.40:0/3088042307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 0x56525bd35950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.931+0000 7f4b43390740 1 --2- 172.21.15.40:0/3088042307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3c440 0x56525bd4d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.931+0000 7f4b43390740 1 --2- 172.21.15.40:0/3088042307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3ba50 0x56525bd461d0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.931+0000 7f4b43390740 1 -- 172.21.15.40:0/3088042307 >> 172.21.15.40:0/3088042307 conn(0x56525bd36fc0 msgr2=0x56525bd45ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.931+0000 7f4b43390740 1 -- 172.21.15.40:0/3088042307 shutdown_connections 2024-03-18T22:45:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.931+0000 7f4b43390740 1 -- 172.21.15.40:0/3088042307 wait complete. 2024-03-18T22:45:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b43390740 1 -- start start 2024-03-18T22:45:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b43390740 1 Processor -- start 2024-03-18T22:45:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b43390740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3ba50 0x56525bd4a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b43390740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3c440 0x56525bd4ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b43390740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 0x56525bd4bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b43390740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4b381aba40 con 0x56525bd3c440 2024-03-18T22:45:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b43390740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4b381ab370 con 0x56525bd415b0 2024-03-18T22:45:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b43390740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4b381a85b0 con 0x56525bd3ba50 2024-03-18T22:45:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b43390740 10 client.0 _ll_register_callbacks cb 0x7f4b340d2040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:45:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b3dd44640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3c440 0x56525bd4ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b3e545640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3ba50 0x56525bd4a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b3ed46640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 0x56525bd4bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b3e545640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3ba50 0x56525bd4a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:36750/0 (socket says 172.21.15.40:36750) 2024-03-18T22:45:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b3dd44640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3c440 0x56525bd4ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51024/0 (socket says 172.21.15.40:51024) 2024-03-18T22:45:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b3ed46640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 0x56525bd4bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:37008/0 (socket says 172.21.15.40:37008) 2024-03-18T22:45:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b3e545640 1 -- 172.21.15.40:0/141331801 learned_addr learned my addr 172.21.15.40:0/141331801 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:45:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b3e545640 1 -- 172.21.15.40:0/141331801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 msgr2=0x56525bd4bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b3e545640 1 --2- 172.21.15.40:0/141331801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 0x56525bd4bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b3e545640 1 -- 172.21.15.40:0/141331801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3c440 msgr2=0x56525bd4ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:45:54.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.932+0000 7f4b3e545640 1 --2- 172.21.15.40:0/141331801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3c440 0x56525bd4ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.933+0000 7f4b3e545640 1 -- 172.21.15.40:0/141331801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56525bd4c290 con 0x56525bd3ba50 2024-03-18T22:45:54.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.933+0000 7f4b3ed46640 1 --2- 172.21.15.40:0/141331801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 0x56525bd4bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:54.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.933+0000 7f4b3dd44640 1 --2- 172.21.15.40:0/141331801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3c440 0x56525bd4ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:45:54.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.933+0000 7f4b3e545640 1 --2- 172.21.15.40:0/141331801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3ba50 0x56525bd4a750 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4b34097490 tx=0x7f4b3409b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1daca751ed87b3cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:45:54.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.933+0000 7f4b21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:45:54.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.933+0000 7f4b21ffb640 1 -- 172.21.15.40:0/141331801 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3409b2f0 con 0x56525bd3ba50 2024-03-18T22:45:54.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.933+0000 7f4b43390740 1 -- 172.21.15.40:0/141331801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b340a3f50 con 0x56525bd3ba50 2024-03-18T22:45:54.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.933+0000 7f4b21ffb640 1 -- 172.21.15.40:0/141331801 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4b34097500 con 0x56525bd3ba50 2024-03-18T22:45:54.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.933+0000 7f4b21ffb640 1 -- 172.21.15.40:0/141331801 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4b3409b580 con 0x56525bd3ba50 2024-03-18T22:45:54.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.933+0000 7f4b43390740 10 client.26227 fetch_fsmap learned FSMap version 30 2024-03-18T22:45:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.933+0000 7f4b43390740 1 -- 172.21.15.40:0/141331801 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56525bd4c290 con 0x56525bd3ba50 2024-03-18T22:45:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.933+0000 7f4b21ffb640 1 -- 172.21.15.40:0/141331801 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b340a3f50 con 0x56525bd3ba50 2024-03-18T22:45:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.934+0000 7f4b21ffb640 1 -- 172.21.15.40:0/141331801 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 30) ==== 8378+0+0 (secure 0 0 0) 0x7f4b340bb320 con 0x56525bd3ba50 2024-03-18T22:45:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.934+0000 7f4b43390740 10 client.26227 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T22:45:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.934+0000 7f4b43390740 10 client.26227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:45:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.934+0000 7f4b43390740 20 client.26227 populate_metadata read hostname 'smithi040' 2024-03-18T22:45:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.934+0000 7f4b43390740 1 --2- 172.21.15.40:0/141331801 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56525bdb7d60 0x56525bdba180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:45:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.934+0000 7f4b43390740 4 client.26227 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:45:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.934+0000 7f4b43390740 1 -- 172.21.15.40:0/141331801 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56525bc25540 con 0x56525bdb7d60 2024-03-18T22:45:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.934+0000 7f4b3dd44640 1 --2- 172.21.15.40:0/141331801 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56525bdb7d60 0x56525bdba180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:45:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.934+0000 7f4b3dd44640 1 --2- 172.21.15.40:0/141331801 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56525bdb7d60 0x56525bdba180 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daae607cf1cb4bef server_cookie=207f58750f141574 in_seq=0 out_seq=0 2024-03-18T22:45:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.934+0000 7f4b21ffb640 10 client.26227 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.934+0000 7f4b21ffb640 1 -- 172.21.15.40:0/141331801 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56525bc25540 con 0x56525bdb7d60 2024-03-18T22:45:54.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.934+0000 7f4b21ffb640 10 client.26227 handle_command_reply: tid=0 2024-03-18T22:45:54.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.934+0000 7f4b43390740 1 client.26227 shutdown 2024-03-18T22:45:54.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b43390740 1 -- 172.21.15.40:0/141331801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3ba50 msgr2=0x56525bd4a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b43390740 1 --2- 172.21.15.40:0/141331801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3ba50 0x56525bd4a750 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4b34097490 tx=0x7f4b3409b7b0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b43390740 1 -- 172.21.15.40:0/141331801 shutdown_connections 2024-03-18T22:45:54.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b43390740 1 --2- 172.21.15.40:0/141331801 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56525bdb7d60 0x56525bdba180 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b43390740 1 --2- 172.21.15.40:0/141331801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56525bd415b0 0x56525bd4bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b43390740 1 --2- 172.21.15.40:0/141331801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56525bd3c440 0x56525bd4ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b21ffb640 0 client.26227 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:45:54.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b43390740 1 --2- 172.21.15.40:0/141331801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56525bd3ba50 0x56525bd4a750 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b43390740 1 -- 172.21.15.40:0/141331801 >> 172.21.15.40:0/141331801 conn(0x56525bd36fc0 msgr2=0x56525bd3a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:54.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b43390740 1 -- 172.21.15.40:0/141331801 shutdown_connections 2024-03-18T22:45:54.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b43390740 1 -- 172.21.15.40:0/141331801 wait complete. 2024-03-18T22:45:54.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b43390740 20 client.26227 trim_cache size 0 max 16384 2024-03-18T22:45:54.943 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:45:54.943 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:45:54.943 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:45:54.943 INFO:teuthology.orchestra.run.smithi040.stdout: "1ba372cc-bdb2-407f-81d2-da1c828b5e36": { 2024-03-18T22:45:54.943 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:45:54.943 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "1ba372cc-bdb2-407f-81d2-da1c828b5e36", 2024-03-18T22:45:54.944 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:45:54.944 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:45:54.944 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:45:54.944 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:45:54.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b407a9640 1 -- 172.21.15.40:0/2826141013 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b240a2800 msgr2=0x7f4b240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b407a9640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b240a2800 0x7f4b240a4cc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x56525bd3cca0 tx=0x7f4b30004ab0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b407a9640 1 -- 172.21.15.40:0/2826141013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b38100720 msgr2=0x7f4b38197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:45:54.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b407a9640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b38100720 0x7f4b38197ab0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400e9c0 tx=0x7f4b3400edc0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b407a9640 1 -- 172.21.15.40:0/2826141013 shutdown_connections 2024-03-18T22:45:54.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b407a9640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4b240a2800 0x7f4b240a4cc0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b407a9640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4b381022d0 0x7f4b3819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b407a9640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4b38101910 0x7f4b38197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b407a9640 1 --2- 172.21.15.40:0/2826141013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b38100720 0x7f4b38197ab0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:45:54.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.935+0000 7f4b407a9640 1 -- 172.21.15.40:0/2826141013 >> 172.21.15.40:0/2826141013 conn(0x7f4b380fbcb0 msgr2=0x7f4b38104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:45:54.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.936+0000 7f4b407a9640 1 -- 172.21.15.40:0/2826141013 shutdown_connections 2024-03-18T22:45:54.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:45:54.936+0000 7f4b407a9640 1 -- 172.21.15.40:0/2826141013 wait complete. 2024-03-18T22:45:54.947 INFO:tasks.cephfs.filesystem:scrub status for tag:1ba372cc-bdb2-407f-81d2-da1c828b5e36 - {'path': '/', 'tag': '1ba372cc-bdb2-407f-81d2-da1c828b5e36', 'options': 'recursive,force'} 2024-03-18T22:45:56.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:56 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:45:56.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:56 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:56.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:56 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:45:56.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:56 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:45:56.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:56 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:56.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:56 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:45:56.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:56 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:56.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:56 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:45:56.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:56 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:56.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:56 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:56.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:56 smithi113 ceph-mon[66775]: pgmap v449: 129 pgs: 129 active+clean; 2.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:45:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:56 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:45:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:56 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:56 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:45:56.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:56 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:45:56.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:56 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:56.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:56 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:45:56.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:56 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:56.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:56 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:45:56.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:56 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:56.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:56 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:56.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:56 smithi103 ceph-mon[66377]: pgmap v449: 129 pgs: 129 active+clean; 2.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:45:56.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:56 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:45:56.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:56 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:56.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:56 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:45:56.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:56 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:45:56.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:56 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:56.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:56 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:45:56.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:56 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:56.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:56 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:45:56.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:56 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:56.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:56 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:45:56.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:56 smithi040 ceph-mon[62643]: pgmap v449: 129 pgs: 129 active+clean; 2.8 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:45:58.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:45:58 smithi113 ceph-mon[66775]: pgmap v450: 129 pgs: 129 active+clean; 2.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 31 MiB/s wr, 21 op/s 2024-03-18T22:45:58.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:45:58 smithi103 ceph-mon[66377]: pgmap v450: 129 pgs: 129 active+clean; 2.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 31 MiB/s wr, 21 op/s 2024-03-18T22:45:58.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:45:58 smithi040 ceph-mon[62643]: pgmap v450: 129 pgs: 129 active+clean; 2.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 31 MiB/s wr, 21 op/s 2024-03-18T22:46:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:00 smithi113 ceph-mon[66775]: pgmap v451: 129 pgs: 129 active+clean; 2.9 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:46:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:00 smithi103 ceph-mon[66377]: pgmap v451: 129 pgs: 129 active+clean; 2.9 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:46:00.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:00 smithi040 ceph-mon[62643]: pgmap v451: 129 pgs: 129 active+clean; 2.9 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:46:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:02 smithi113 ceph-mon[66775]: pgmap v452: 129 pgs: 129 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:46:02.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:02 smithi103 ceph-mon[66377]: pgmap v452: 129 pgs: 129 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:46:02.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:02 smithi040 ceph-mon[62643]: pgmap v452: 129 pgs: 129 active+clean; 3.0 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:46:03.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:03 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:46:03.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:03 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:46:03.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:03 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:46:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:04 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:46:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:04 smithi113 ceph-mon[66775]: pgmap v453: 129 pgs: 129 active+clean; 3.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:46:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:04 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:46:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:04 smithi103 ceph-mon[66377]: pgmap v453: 129 pgs: 129 active+clean; 3.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:46:04.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:04 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:46:04.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:04 smithi040 ceph-mon[62643]: pgmap v453: 129 pgs: 129 active+clean; 3.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:46:06.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:06 smithi113 ceph-mon[66775]: pgmap v454: 129 pgs: 129 active+clean; 3.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:46:06.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:06 smithi103 ceph-mon[66377]: pgmap v454: 129 pgs: 129 active+clean; 3.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:46:06.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:06 smithi040 ceph-mon[62643]: pgmap v454: 129 pgs: 129 active+clean; 3.1 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:46:08.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:08 smithi113 ceph-mon[66775]: pgmap v455: 129 pgs: 129 active+clean; 3.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:46:08.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:08 smithi103 ceph-mon[66377]: pgmap v455: 129 pgs: 129 active+clean; 3.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:46:08.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:08 smithi040 ceph-mon[62643]: pgmap v455: 129 pgs: 129 active+clean; 3.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:46:10.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:10 smithi113 ceph-mon[66775]: pgmap v456: 129 pgs: 129 active+clean; 3.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:46:10.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:10 smithi103 ceph-mon[66377]: pgmap v456: 129 pgs: 129 active+clean; 3.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:46:10.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:10 smithi040 ceph-mon[62643]: pgmap v456: 129 pgs: 129 active+clean; 3.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:46:12.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:12 smithi113 ceph-mon[66775]: pgmap v457: 129 pgs: 129 active+clean; 3.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:46:12.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:12 smithi103 ceph-mon[66377]: pgmap v457: 129 pgs: 129 active+clean; 3.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:46:12.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:12 smithi040 ceph-mon[62643]: pgmap v457: 129 pgs: 129 active+clean; 3.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:46:14.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:14 smithi113 ceph-mon[66775]: pgmap v458: 129 pgs: 129 active+clean; 3.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:46:14.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:14 smithi103 ceph-mon[66377]: pgmap v458: 129 pgs: 129 active+clean; 3.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:46:14.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:14 smithi040 ceph-mon[62643]: pgmap v458: 129 pgs: 129 active+clean; 3.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:46:16.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:16 smithi113 ceph-mon[66775]: pgmap v459: 129 pgs: 129 active+clean; 3.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:46:16.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:16 smithi103 ceph-mon[66377]: pgmap v459: 129 pgs: 129 active+clean; 3.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:46:16.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:16 smithi040 ceph-mon[62643]: pgmap v459: 129 pgs: 129 active+clean; 3.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:46:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:18 smithi113 ceph-mon[66775]: pgmap v460: 129 pgs: 129 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:46:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:18 smithi103 ceph-mon[66377]: pgmap v460: 129 pgs: 129 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:46:18.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:18 smithi040 ceph-mon[62643]: pgmap v460: 129 pgs: 129 active+clean; 3.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:46:20.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:20 smithi113 ceph-mon[66775]: pgmap v461: 129 pgs: 129 active+clean; 3.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:46:20.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:46:20.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:46:20.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:46:20.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:20 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:46:20.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:20 smithi103 ceph-mon[66377]: pgmap v461: 129 pgs: 129 active+clean; 3.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:46:20.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:46:20.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:46:20.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:46:20.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:20 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:46:20.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:20 smithi040 ceph-mon[62643]: pgmap v461: 129 pgs: 129 active+clean; 3.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:46:20.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:46:20.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:46:20.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:46:20.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:20 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:46:22.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:22 smithi113 ceph-mon[66775]: pgmap v462: 129 pgs: 129 active+clean; 3.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 29 MiB/s wr, 22 op/s 2024-03-18T22:46:22.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:22 smithi103 ceph-mon[66377]: pgmap v462: 129 pgs: 129 active+clean; 3.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 29 MiB/s wr, 22 op/s 2024-03-18T22:46:22.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:22 smithi040 ceph-mon[62643]: pgmap v462: 129 pgs: 129 active+clean; 3.5 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 29 MiB/s wr, 22 op/s 2024-03-18T22:46:24.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:24 smithi113 ceph-mon[66775]: pgmap v463: 129 pgs: 129 active+clean; 3.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:46:24.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:24 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:46:24.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:24 smithi103 ceph-mon[66377]: pgmap v463: 129 pgs: 129 active+clean; 3.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:46:24.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:24 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:46:24.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:24 smithi040 ceph-mon[62643]: pgmap v463: 129 pgs: 129 active+clean; 3.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:46:24.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:24 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:46:24.948 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:46:24.948 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:46:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.069+0000 7f7063c59640 1 -- 172.21.15.40:0/2886182530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705c1022d0 msgr2=0x7f705c108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.069+0000 7f7063c59640 1 --2- 172.21.15.40:0/2886182530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705c1022d0 0x7f705c108b60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7054009be0 tx=0x7f705402bef0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.069+0000 7f7063c59640 1 -- 172.21.15.40:0/2886182530 shutdown_connections 2024-03-18T22:46:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.069+0000 7f7063c59640 1 --2- 172.21.15.40:0/2886182530 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705c1022d0 0x7f705c108b60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.069+0000 7f7063c59640 1 --2- 172.21.15.40:0/2886182530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f705c101910 0x7f705c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.069+0000 7f7063c59640 1 --2- 172.21.15.40:0/2886182530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f705c100720 0x7f705c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.069+0000 7f7063c59640 1 -- 172.21.15.40:0/2886182530 >> 172.21.15.40:0/2886182530 conn(0x7f705c0fbcb0 msgr2=0x7f705c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.069+0000 7f7063c59640 1 -- 172.21.15.40:0/2886182530 shutdown_connections 2024-03-18T22:46:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.069+0000 7f7063c59640 1 -- 172.21.15.40:0/2886182530 wait complete. 2024-03-18T22:46:25.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f7063c59640 1 Processor -- start 2024-03-18T22:46:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f7063c59640 1 -- start start 2024-03-18T22:46:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f7063c59640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705c100720 0x7f705c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f7063c59640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f705c101910 0x7f705c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f7063c59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f705c1022d0 0x7f705c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f7063c59640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f705c10c230 con 0x7f705c1022d0 2024-03-18T22:46:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f7063c59640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f705c10c0b0 con 0x7f705c101910 2024-03-18T22:46:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f7063c59640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f705c10c3b0 con 0x7f705c100720 2024-03-18T22:46:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f70621f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f705c1022d0 0x7f705c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f70619f5640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705c100720 0x7f705c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f70621f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f705c1022d0 0x7f705c19efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43964/0 (socket says 172.21.15.40:43964) 2024-03-18T22:46:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f70619f5640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705c100720 0x7f705c197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55644/0 (socket says 172.21.15.40:55644) 2024-03-18T22:46:25.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f70621f6640 1 -- 172.21.15.40:0/488574416 learned_addr learned my addr 172.21.15.40:0/488574416 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f70611f4640 1 --2- 172.21.15.40:0/488574416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f705c101910 0x7f705c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f70621f6640 1 -- 172.21.15.40:0/488574416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705c100720 msgr2=0x7f705c197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f70621f6640 1 --2- 172.21.15.40:0/488574416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705c100720 0x7f705c197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f70621f6640 1 -- 172.21.15.40:0/488574416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f705c101910 msgr2=0x7f705c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f70621f6640 1 --2- 172.21.15.40:0/488574416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f705c101910 0x7f705c197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f70621f6640 1 -- 172.21.15.40:0/488574416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705c19f620 con 0x7f705c1022d0 2024-03-18T22:46:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f70619f5640 1 --2- 172.21.15.40:0/488574416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705c100720 0x7f705c197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f70611f4640 1 --2- 172.21.15.40:0/488574416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f705c101910 0x7f705c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.070+0000 7f70621f6640 1 --2- 172.21.15.40:0/488574416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f705c1022d0 0x7f705c19efb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7054004af0 tx=0x7f7054033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.071+0000 7f7052ffd640 1 -- 172.21.15.40:0/488574416 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7054043020 con 0x7f705c1022d0 2024-03-18T22:46:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.071+0000 7f7052ffd640 1 -- 172.21.15.40:0/488574416 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f705402ca60 con 0x7f705c1022d0 2024-03-18T22:46:25.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.071+0000 7f7063c59640 1 -- 172.21.15.40:0/488574416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f705c19f8b0 con 0x7f705c1022d0 2024-03-18T22:46:25.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.071+0000 7f7052ffd640 1 -- 172.21.15.40:0/488574416 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7054034690 con 0x7f705c1022d0 2024-03-18T22:46:25.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.071+0000 7f7063c59640 1 -- 172.21.15.40:0/488574416 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f705c19fe00 con 0x7f705c1022d0 2024-03-18T22:46:25.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.073+0000 7f7052ffd640 1 -- 172.21.15.40:0/488574416 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7054046020 con 0x7f705c1022d0 2024-03-18T22:46:25.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.073+0000 7f7052ffd640 1 --2- 172.21.15.40:0/488574416 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70480a2800 0x7f70480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.074+0000 7f7052ffd640 1 -- 172.21.15.40:0/488574416 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f70540f1510 con 0x7f705c1022d0 2024-03-18T22:46:25.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.074+0000 7f70619f5640 1 --2- 172.21.15.40:0/488574416 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70480a2800 0x7f70480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.074+0000 7f7066843740 1 Processor -- start 2024-03-18T22:46:25.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.074+0000 7f7066843740 1 -- start start 2024-03-18T22:46:25.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.074+0000 7f70619f5640 1 --2- 172.21.15.40:0/488574416 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70480a2800 0x7f70480a4cc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f705c101770 tx=0x7f7058005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.074+0000 7f7066843740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9023a50 0x55a4a9023e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.074+0000 7f7066843740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9024440 0x55a4a9028f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.074+0000 7f7066843740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 0x55a4a902ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.074+0000 7f7066843740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a4a8f88500 con 0x55a4a9024440 2024-03-18T22:46:25.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.074+0000 7f7066843740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a4a8f30be0 con 0x55a4a9023a50 2024-03-18T22:46:25.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.074+0000 7f7066843740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55a4a8f845a0 con 0x55a4a90295b0 2024-03-18T22:46:25.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.074+0000 7f70611f4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9024440 0x55a4a9028f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.074+0000 7f70619f5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9023a50 0x55a4a9023e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.074+0000 7f70611f4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9024440 0x55a4a9028f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43978/0 (socket says 172.21.15.40:43978) 2024-03-18T22:46:25.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.074+0000 7f70611f4640 1 -- 172.21.15.40:0/2190254155 learned_addr learned my addr 172.21.15.40:0/2190254155 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.075+0000 7f70621f6640 1 --2- 172.21.15.40:0/2190254155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 0x55a4a902ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.075+0000 7f70619f5640 1 -- 172.21.15.40:0/2190254155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 msgr2=0x55a4a902ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.075+0000 7f70619f5640 1 --2- 172.21.15.40:0/2190254155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 0x55a4a902ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.075+0000 7f70619f5640 1 -- 172.21.15.40:0/2190254155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9024440 msgr2=0x55a4a9028f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.075+0000 7f70619f5640 1 --2- 172.21.15.40:0/2190254155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9024440 0x55a4a9028f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.075+0000 7f70619f5640 1 -- 172.21.15.40:0/2190254155 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4a8f52000 con 0x55a4a9023a50 2024-03-18T22:46:25.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.075+0000 7f70611f4640 1 --2- 172.21.15.40:0/2190254155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9024440 0x55a4a9028f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:25.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.075+0000 7f70619f5640 1 --2- 172.21.15.40:0/2190254155 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9023a50 0x55a4a9023e70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f70580041f0 tx=0x7f7058014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45a939e63b8cbc89 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.075+0000 7f7047fff640 1 -- 172.21.15.40:0/2190254155 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7058026480 con 0x55a4a9023a50 2024-03-18T22:46:25.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.075+0000 7f7047fff640 1 -- 172.21.15.40:0/2190254155 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f705800f930 con 0x55a4a9023a50 2024-03-18T22:46:25.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.075+0000 7f7047fff640 1 -- 172.21.15.40:0/2190254155 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705801c830 con 0x55a4a9023a50 2024-03-18T22:46:25.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.076+0000 7f7066843740 1 -- 172.21.15.40:0/2190254155 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9023a50 msgr2=0x55a4a9023e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.076+0000 7f7066843740 1 --2- 172.21.15.40:0/2190254155 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9023a50 0x55a4a9023e70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f70580041f0 tx=0x7f7058014000 comp rx=0 tx=0).stop 2024-03-18T22:46:25.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.076+0000 7f7066843740 1 -- 172.21.15.40:0/2190254155 shutdown_connections 2024-03-18T22:46:25.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.076+0000 7f7066843740 1 --2- 172.21.15.40:0/2190254155 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 0x55a4a902ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.076+0000 7f7066843740 1 --2- 172.21.15.40:0/2190254155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9024440 0x55a4a9028f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.076+0000 7f7066843740 1 --2- 172.21.15.40:0/2190254155 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9023a50 0x55a4a9023e70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.076+0000 7f7066843740 1 -- 172.21.15.40:0/2190254155 >> 172.21.15.40:0/2190254155 conn(0x55a4a901efc0 msgr2=0x55a4a901f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.076+0000 7f7066843740 1 -- 172.21.15.40:0/2190254155 shutdown_connections 2024-03-18T22:46:25.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.076+0000 7f7066843740 1 -- 172.21.15.40:0/2190254155 wait complete. 2024-03-18T22:46:25.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.076+0000 7f7066843740 1 -- start start 2024-03-18T22:46:25.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.076+0000 7f7066843740 1 Processor -- start 2024-03-18T22:46:25.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f7066843740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9023a50 0x55a4a902df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f7066843740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9024440 0x55a4a902e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f7066843740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 0x55a4a90354b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f7066843740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a4a902c560 con 0x55a4a9023a50 2024-03-18T22:46:25.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f7066843740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a4a8f44160 con 0x55a4a9024440 2024-03-18T22:46:25.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f7066843740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55a4a902c6e0 con 0x55a4a90295b0 2024-03-18T22:46:25.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f7066843740 10 client.0 _ll_register_callbacks cb 0x55a4a8f80a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:25.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f70619f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9023a50 0x55a4a902df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f70611f4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9024440 0x55a4a902e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f70619f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9023a50 0x55a4a902df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43984/0 (socket says 172.21.15.40:43984) 2024-03-18T22:46:25.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f70621f6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 0x55a4a90354b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f70611f4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9024440 0x55a4a902e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47192/0 (socket says 172.21.15.40:47192) 2024-03-18T22:46:25.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f70619f5640 1 -- 172.21.15.40:0/2132504447 learned_addr learned my addr 172.21.15.40:0/2132504447 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f70611f4640 1 -- 172.21.15.40:0/2132504447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 msgr2=0x55a4a90354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f70611f4640 1 --2- 172.21.15.40:0/2132504447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 0x55a4a90354b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f70611f4640 1 -- 172.21.15.40:0/2132504447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9023a50 msgr2=0x55a4a902df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f70611f4640 1 --2- 172.21.15.40:0/2132504447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9023a50 0x55a4a902df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f70611f4640 1 -- 172.21.15.40:0/2132504447 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4a8d667d0 con 0x55a4a9024440 2024-03-18T22:46:25.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f70619f5640 1 --2- 172.21.15.40:0/2132504447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9023a50 0x55a4a902df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:25.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.077+0000 7f70621f6640 1 --2- 172.21.15.40:0/2132504447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 0x55a4a90354b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:25.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f70611f4640 1 --2- 172.21.15.40:0/2132504447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9024440 0x55a4a902e470 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55a4a9024840 tx=0x7f704c00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d230835995402c76 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f70457fa640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f70457fa640 1 -- 172.21.15.40:0/2132504447 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704c0120d0 con 0x55a4a9024440 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f7066843740 1 -- 172.21.15.40:0/2132504447 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f705800f930 con 0x55a4a9024440 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f70457fa640 1 -- 172.21.15.40:0/2132504447 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f704c010430 con 0x55a4a9024440 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f70457fa640 1 -- 172.21.15.40:0/2132504447 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f704c012270 con 0x55a4a9024440 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f7066843740 10 client.26249 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f7066843740 1 -- 172.21.15.40:0/2132504447 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a4a8c9c3f0 con 0x55a4a9024440 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f70457fa640 1 -- 172.21.15.40:0/2132504447 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f705800f930 con 0x55a4a9024440 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f70457fa640 1 -- 172.21.15.40:0/2132504447 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f704c01c7c0 con 0x55a4a9024440 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f7066843740 10 client.26249 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f7066843740 10 client.26249 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f7066843740 20 client.26249 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f7066843740 1 --2- 172.21.15.40:0/2132504447 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a4a9039e00 0x55a4a903c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f7066843740 4 client.26249 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.078+0000 7f7066843740 1 -- 172.21.15.40:0/2132504447 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4a8f95a40 con 0x55a4a9039e00 2024-03-18T22:46:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.079+0000 7f70619f5640 1 --2- 172.21.15.40:0/2132504447 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a4a9039e00 0x55a4a903c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.079+0000 7f70619f5640 1 --2- 172.21.15.40:0/2132504447 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a4a9039e00 0x55a4a903c280 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2358ce42a96ce04 server_cookie=304968500599cbf4 in_seq=0 out_seq=0 2024-03-18T22:46:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.079+0000 7f70457fa640 10 client.26249 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.080+0000 7f70457fa640 1 -- 172.21.15.40:0/2132504447 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a4a8f95a40 con 0x55a4a9039e00 2024-03-18T22:46:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.080+0000 7f70457fa640 10 client.26249 handle_command_reply: tid=0 2024-03-18T22:46:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.080+0000 7f7066843740 1 client.26249 shutdown 2024-03-18T22:46:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.080+0000 7f7066843740 1 -- 172.21.15.40:0/2132504447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9024440 msgr2=0x55a4a902e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.080+0000 7f7066843740 1 --2- 172.21.15.40:0/2132504447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9024440 0x55a4a902e470 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55a4a9024840 tx=0x7f704c00f790 comp rx=0 tx=0).stop 2024-03-18T22:46:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.080+0000 7f7066843740 1 -- 172.21.15.40:0/2132504447 shutdown_connections 2024-03-18T22:46:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.080+0000 7f7066843740 1 --2- 172.21.15.40:0/2132504447 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 0x55a4a90354b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.080+0000 7f7066843740 1 --2- 172.21.15.40:0/2132504447 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a4a9039e00 0x55a4a903c280 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.080+0000 7f7066843740 1 --2- 172.21.15.40:0/2132504447 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9024440 0x55a4a902e470 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.080+0000 7f70457fa640 0 client.26249 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.080+0000 7f7066843740 1 --2- 172.21.15.40:0/2132504447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9023a50 0x55a4a902df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.080+0000 7f7066843740 1 -- 172.21.15.40:0/2132504447 >> 172.21.15.40:0/2132504447 conn(0x55a4a901efc0 msgr2=0x55a4a901e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.080+0000 7f7066843740 1 -- 172.21.15.40:0/2132504447 shutdown_connections 2024-03-18T22:46:25.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.080+0000 7f7066843740 1 -- 172.21.15.40:0/2132504447 wait complete. 2024-03-18T22:46:25.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.081+0000 7f7066843740 20 client.26249 trim_cache size 0 max 16384 2024-03-18T22:46:25.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f7066843740 1 Processor -- start 2024-03-18T22:46:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f7066843740 1 -- start start 2024-03-18T22:46:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f7066843740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a9023a50 0x55a4a9035f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f7066843740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9024440 0x55a4a9035470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f7066843740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a90295b0 0x55a4a90258a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f7066843740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f705c1aa490 con 0x55a4a90295b0 2024-03-18T22:46:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f7066843740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f705c1ac000 con 0x55a4a9024440 2024-03-18T22:46:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f7066843740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f705c1aab60 con 0x55a4a9023a50 2024-03-18T22:46:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f70621f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a90295b0 0x55a4a90258a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f70619f5640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a9023a50 0x55a4a9035f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f70619f5640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a9023a50 0x55a4a9035f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55672/0 (socket says 172.21.15.40:55672) 2024-03-18T22:46:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f70621f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a90295b0 0x55a4a90258a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43990/0 (socket says 172.21.15.40:43990) 2024-03-18T22:46:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f70619f5640 1 -- 172.21.15.40:0/385047370 learned_addr learned my addr 172.21.15.40:0/385047370 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f70611f4640 1 --2- 172.21.15.40:0/385047370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9024440 0x55a4a9035470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f70619f5640 1 -- 172.21.15.40:0/385047370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9024440 msgr2=0x55a4a9035470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f70619f5640 1 --2- 172.21.15.40:0/385047370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9024440 0x55a4a9035470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f70619f5640 1 -- 172.21.15.40:0/385047370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a90295b0 msgr2=0x55a4a90258a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f70619f5640 1 --2- 172.21.15.40:0/385047370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a90295b0 0x55a4a90258a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f70619f5640 1 -- 172.21.15.40:0/385047370 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4a8c5c0f0 con 0x55a4a9023a50 2024-03-18T22:46:25.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.099+0000 7f70611f4640 1 --2- 172.21.15.40:0/385047370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9024440 0x55a4a9035470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:25.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.100+0000 7f70619f5640 1 --2- 172.21.15.40:0/385047370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a9023a50 0x55a4a9035f00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f70580058c0 tx=0x7f70580096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54371f100cd5879b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.100+0000 7f70477fe640 1 -- 172.21.15.40:0/385047370 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7058038020 con 0x55a4a9023a50 2024-03-18T22:46:25.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.100+0000 7f70477fe640 1 -- 172.21.15.40:0/385047370 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f705801c8d0 con 0x55a4a9023a50 2024-03-18T22:46:25.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.100+0000 7f70477fe640 1 -- 172.21.15.40:0/385047370 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7058034160 con 0x55a4a9023a50 2024-03-18T22:46:25.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 -- 172.21.15.40:0/385047370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a9023a50 msgr2=0x55a4a9035f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 --2- 172.21.15.40:0/385047370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a9023a50 0x55a4a9035f00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f70580058c0 tx=0x7f70580096c0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 -- 172.21.15.40:0/385047370 shutdown_connections 2024-03-18T22:46:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 --2- 172.21.15.40:0/385047370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a90295b0 0x55a4a90258a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 --2- 172.21.15.40:0/385047370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9024440 0x55a4a9035470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 --2- 172.21.15.40:0/385047370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a9023a50 0x55a4a9035f00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 -- 172.21.15.40:0/385047370 >> 172.21.15.40:0/385047370 conn(0x55a4a901efc0 msgr2=0x55a4a902df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 -- 172.21.15.40:0/385047370 shutdown_connections 2024-03-18T22:46:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 -- 172.21.15.40:0/385047370 wait complete. 2024-03-18T22:46:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 -- start start 2024-03-18T22:46:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 Processor -- start 2024-03-18T22:46:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9023a50 0x55a4a90322b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9024440 0x55a4a90327f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 0x55a4a909e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f705c1ab900 con 0x55a4a9024440 2024-03-18T22:46:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f705c1ab230 con 0x55a4a9023a50 2024-03-18T22:46:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f705c1a8470 con 0x55a4a90295b0 2024-03-18T22:46:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f7066843740 10 client.0 _ll_register_callbacks cb 0x7f704c012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.101+0000 7f70611f4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9024440 0x55a4a90327f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f70611f4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9024440 0x55a4a90327f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44006/0 (socket says 172.21.15.40:44006) 2024-03-18T22:46:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f70621f6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 0x55a4a909e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f70611f4640 1 -- 172.21.15.40:0/3201308981 learned_addr learned my addr 172.21.15.40:0/3201308981 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f70619f5640 1 --2- 172.21.15.40:0/3201308981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9023a50 0x55a4a90322b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f70621f6640 1 -- 172.21.15.40:0/3201308981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9023a50 msgr2=0x55a4a90322b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f70621f6640 1 --2- 172.21.15.40:0/3201308981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9023a50 0x55a4a90322b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f70621f6640 1 -- 172.21.15.40:0/3201308981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9024440 msgr2=0x55a4a90327f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f70621f6640 1 --2- 172.21.15.40:0/3201308981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9024440 0x55a4a90327f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f70621f6640 1 -- 172.21.15.40:0/3201308981 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4a909e6f0 con 0x55a4a90295b0 2024-03-18T22:46:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f70611f4640 1 --2- 172.21.15.40:0/3201308981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9024440 0x55a4a90327f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f70621f6640 1 --2- 172.21.15.40:0/3201308981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 0x55a4a909e1b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f70540eef90 tx=0x7f70540bd720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=875c6ab26b52857d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f70457fa640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:46:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f70457fa640 1 -- 172.21.15.40:0/3201308981 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70540db060 con 0x55a4a90295b0 2024-03-18T22:46:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f70457fa640 1 -- 172.21.15.40:0/3201308981 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f70540bbdc0 con 0x55a4a90295b0 2024-03-18T22:46:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f7066843740 1 -- 172.21.15.40:0/3201308981 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a4a8f8fd90 con 0x55a4a90295b0 2024-03-18T22:46:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f70457fa640 1 -- 172.21.15.40:0/3201308981 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f70540d5030 con 0x55a4a90295b0 2024-03-18T22:46:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f7066843740 10 client.26245 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.102+0000 7f7066843740 1 -- 172.21.15.40:0/3201308981 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a4a909f9d0 con 0x55a4a90295b0 2024-03-18T22:46:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.103+0000 7f70457fa640 1 -- 172.21.15.40:0/3201308981 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a4a8f8fd90 con 0x55a4a90295b0 2024-03-18T22:46:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.103+0000 7f70457fa640 1 -- 172.21.15.40:0/3201308981 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f70540d6b80 con 0x55a4a90295b0 2024-03-18T22:46:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.103+0000 7f7066843740 10 client.26245 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.103+0000 7f7066843740 10 client.26245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:46:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.103+0000 7f7066843740 20 client.26245 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.103+0000 7f7066843740 1 --2- 172.21.15.40:0/3201308981 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a4a9039e00 0x55a4a90a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.103+0000 7f7066843740 4 client.26245 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:46:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.103+0000 7f7066843740 1 -- 172.21.15.40:0/3201308981 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a4a8f0d540 con 0x55a4a9039e00 2024-03-18T22:46:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.103+0000 7f70619f5640 1 --2- 172.21.15.40:0/3201308981 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a4a9039e00 0x55a4a90a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.103+0000 7f70457fa640 10 client.26245 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.103+0000 7f70619f5640 1 --2- 172.21.15.40:0/3201308981 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a4a9039e00 0x55a4a90a23a0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40622b60d8943408 server_cookie=96b33108c1e62dba in_seq=0 out_seq=0 2024-03-18T22:46:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f70457fa640 1 -- 172.21.15.40:0/3201308981 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a4a8f0d540 con 0x55a4a9039e00 2024-03-18T22:46:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f70457fa640 10 client.26245 handle_command_reply: tid=0 2024-03-18T22:46:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7066843740 1 client.26245 shutdown 2024-03-18T22:46:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7066843740 1 -- 172.21.15.40:0/3201308981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 msgr2=0x55a4a909e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7066843740 1 --2- 172.21.15.40:0/3201308981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 0x55a4a909e1b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f70540eef90 tx=0x7f70540bd720 comp rx=0 tx=0).stop 2024-03-18T22:46:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7066843740 1 -- 172.21.15.40:0/3201308981 shutdown_connections 2024-03-18T22:46:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7066843740 1 --2- 172.21.15.40:0/3201308981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a4a90295b0 0x55a4a909e1b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7066843740 1 --2- 172.21.15.40:0/3201308981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a4a9024440 0x55a4a90327f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7066843740 1 --2- 172.21.15.40:0/3201308981 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a4a9039e00 0x55a4a90a23a0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7066843740 1 --2- 172.21.15.40:0/3201308981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4a9023a50 0x55a4a90322b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f70457fa640 0 client.26245 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:25.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7066843740 1 -- 172.21.15.40:0/3201308981 >> 172.21.15.40:0/3201308981 conn(0x55a4a901efc0 msgr2=0x55a4a902df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7066843740 1 -- 172.21.15.40:0/3201308981 shutdown_connections 2024-03-18T22:46:25.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7066843740 1 -- 172.21.15.40:0/3201308981 wait complete. 2024-03-18T22:46:25.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7066843740 20 client.26245 trim_cache size 0 max 16384 2024-03-18T22:46:25.111 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:46:25.112 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:46:25.112 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:46:25.112 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:46:25.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7063c59640 1 -- 172.21.15.40:0/488574416 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70480a2800 msgr2=0x7f70480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7063c59640 1 --2- 172.21.15.40:0/488574416 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70480a2800 0x7f70480a4cc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f705c101770 tx=0x7f7058005f90 comp rx=0 tx=0).stop 2024-03-18T22:46:25.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7063c59640 1 -- 172.21.15.40:0/488574416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f705c1022d0 msgr2=0x7f705c19efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.104+0000 7f7063c59640 1 --2- 172.21.15.40:0/488574416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f705c1022d0 0x7f705c19efb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7054004af0 tx=0x7f7054033c90 comp rx=0 tx=0).stop 2024-03-18T22:46:25.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.105+0000 7f7063c59640 1 -- 172.21.15.40:0/488574416 shutdown_connections 2024-03-18T22:46:25.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.105+0000 7f7063c59640 1 --2- 172.21.15.40:0/488574416 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70480a2800 0x7f70480a4cc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.105+0000 7f7063c59640 1 --2- 172.21.15.40:0/488574416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f705c1022d0 0x7f705c19efb0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.105+0000 7f7063c59640 1 --2- 172.21.15.40:0/488574416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f705c101910 0x7f705c197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.105+0000 7f7063c59640 1 --2- 172.21.15.40:0/488574416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f705c100720 0x7f705c197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.105+0000 7f7063c59640 1 -- 172.21.15.40:0/488574416 >> 172.21.15.40:0/488574416 conn(0x7f705c0fbcb0 msgr2=0x7f705c0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.105+0000 7f7063c59640 1 -- 172.21.15.40:0/488574416 shutdown_connections 2024-03-18T22:46:25.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.105+0000 7f7063c59640 1 -- 172.21.15.40:0/488574416 wait complete. 2024-03-18T22:46:25.116 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:46:25.116 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:46:25.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.235+0000 7fc821910640 1 -- 172.21.15.40:0/2028606263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc81c101930 msgr2=0x7fc81c101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.235+0000 7fc821910640 1 --2- 172.21.15.40:0/2028606263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc81c101930 0x7fc81c101db0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc804009900 tx=0x7fc80402bc30 comp rx=0 tx=0).stop 2024-03-18T22:46:25.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.235+0000 7fc821910640 1 -- 172.21.15.40:0/2028606263 shutdown_connections 2024-03-18T22:46:25.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.235+0000 7fc821910640 1 --2- 172.21.15.40:0/2028606263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc81c1022f0 0x7fc81c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.235+0000 7fc821910640 1 --2- 172.21.15.40:0/2028606263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc81c101930 0x7fc81c101db0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.235+0000 7fc821910640 1 --2- 172.21.15.40:0/2028606263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc81c100740 0x7fc81c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.235+0000 7fc821910640 1 -- 172.21.15.40:0/2028606263 >> 172.21.15.40:0/2028606263 conn(0x7fc81c0fbcb0 msgr2=0x7fc81c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.235+0000 7fc821910640 1 -- 172.21.15.40:0/2028606263 shutdown_connections 2024-03-18T22:46:25.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.235+0000 7fc821910640 1 -- 172.21.15.40:0/2028606263 wait complete. 2024-03-18T22:46:25.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.236+0000 7fc821910640 1 Processor -- start 2024-03-18T22:46:25.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.236+0000 7fc821910640 1 -- start start 2024-03-18T22:46:25.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.236+0000 7fc821910640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc81c100740 0x7fc81c1969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.236+0000 7fc821910640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc81c101930 0x7fc81c196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.236+0000 7fc821910640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc81c1022f0 0x7fc81c19df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.236+0000 7fc821910640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c10c270 con 0x7fc81c1022f0 2024-03-18T22:46:25.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.236+0000 7fc821910640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c10c0f0 con 0x7fc81c101930 2024-03-18T22:46:25.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.236+0000 7fc821910640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c10c3f0 con 0x7fc81c100740 2024-03-18T22:46:25.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.236+0000 7fc81b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc81c1022f0 0x7fc81c19df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.236+0000 7fc81affd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc81c100740 0x7fc81c1969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.236+0000 7fc81affd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc81c100740 0x7fc81c1969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55692/0 (socket says 172.21.15.40:55692) 2024-03-18T22:46:25.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.236+0000 7fc81affd640 1 -- 172.21.15.40:0/3113350217 learned_addr learned my addr 172.21.15.40:0/3113350217 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.236+0000 7fc8127fc640 1 --2- 172.21.15.40:0/3113350217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc81c101930 0x7fc81c196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.237+0000 7fc81affd640 1 -- 172.21.15.40:0/3113350217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc81c101930 msgr2=0x7fc81c196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.237+0000 7fc81affd640 1 --2- 172.21.15.40:0/3113350217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc81c101930 0x7fc81c196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.237+0000 7fc81affd640 1 -- 172.21.15.40:0/3113350217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc81c1022f0 msgr2=0x7fc81c19df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.237+0000 7fc81affd640 1 --2- 172.21.15.40:0/3113350217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc81c1022f0 0x7fc81c19df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.237+0000 7fc81affd640 1 -- 172.21.15.40:0/3113350217 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81c19e5d0 con 0x7fc81c100740 2024-03-18T22:46:25.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.237+0000 7fc81b7fe640 1 --2- 172.21.15.40:0/3113350217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc81c1022f0 0x7fc81c19df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:25.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.237+0000 7fc81affd640 1 --2- 172.21.15.40:0/3113350217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc81c100740 0x7fc81c1969b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc80c00e9c0 tx=0x7fc80c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.237+0000 7fc82090e640 1 -- 172.21.15.40:0/3113350217 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c00cd80 con 0x7fc81c100740 2024-03-18T22:46:25.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.237+0000 7fc82090e640 1 -- 172.21.15.40:0/3113350217 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc80c0035f0 con 0x7fc81c100740 2024-03-18T22:46:25.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.237+0000 7fc821910640 1 -- 172.21.15.40:0/3113350217 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc81c19e8c0 con 0x7fc81c100740 2024-03-18T22:46:25.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.237+0000 7fc82090e640 1 -- 172.21.15.40:0/3113350217 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c010690 con 0x7fc81c100740 2024-03-18T22:46:25.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.237+0000 7fc821910640 1 -- 172.21.15.40:0/3113350217 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc81c19ee30 con 0x7fc81c100740 2024-03-18T22:46:25.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.238+0000 7fc821910640 1 -- 172.21.15.40:0/3113350217 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc81c107ad0 con 0x7fc81c100740 2024-03-18T22:46:25.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.240+0000 7fc82090e640 1 -- 172.21.15.40:0/3113350217 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc80c0037a0 con 0x7fc81c100740 2024-03-18T22:46:25.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.240+0000 7fc82090e640 1 --2- 172.21.15.40:0/3113350217 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8080a2880 0x7fc8080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.240+0000 7fc8127fc640 1 --2- 172.21.15.40:0/3113350217 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8080a2880 0x7fc8080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.240+0000 7fc82090e640 1 -- 172.21.15.40:0/3113350217 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc80c0cf1c0 con 0x7fc81c100740 2024-03-18T22:46:25.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.241+0000 7fc8127fc640 1 --2- 172.21.15.40:0/3113350217 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8080a2880 0x7fc8080a4d40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc804009230 tx=0x7fc804005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.243+0000 7fc82090e640 1 -- 172.21.15.40:0/3113350217 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc80c09be70 con 0x7fc81c100740 2024-03-18T22:46:25.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.547+0000 7fc821910640 1 -- 172.21.15.40:0/3113350217 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc81c02d1f0 con 0x7fc81c100740 2024-03-18T22:46:25.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.548+0000 7fc82090e640 1 -- 172.21.15.40:0/3113350217 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11948 (secure 0 0 0) 0x7fc80c09b770 con 0x7fc81c100740 2024-03-18T22:46:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:46:25.549 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:46:22.724760+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24481,24509,24473,24401,24388]},"id":1}]} 2024-03-18T22:46:25.549 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 31 2024-03-18T22:46:25.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.550+0000 7fc821910640 1 -- 172.21.15.40:0/3113350217 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8080a2880 msgr2=0x7fc8080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.550+0000 7fc821910640 1 --2- 172.21.15.40:0/3113350217 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8080a2880 0x7fc8080a4d40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc804009230 tx=0x7fc804005fd0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.550+0000 7fc821910640 1 -- 172.21.15.40:0/3113350217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc81c100740 msgr2=0x7fc81c1969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.550+0000 7fc821910640 1 --2- 172.21.15.40:0/3113350217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc81c100740 0x7fc81c1969b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc80c00e9c0 tx=0x7fc80c00ee80 comp rx=0 tx=0).stop 2024-03-18T22:46:25.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.550+0000 7fc821910640 1 -- 172.21.15.40:0/3113350217 shutdown_connections 2024-03-18T22:46:25.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.550+0000 7fc821910640 1 --2- 172.21.15.40:0/3113350217 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc8080a2880 0x7fc8080a4d40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.550+0000 7fc821910640 1 --2- 172.21.15.40:0/3113350217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc81c1022f0 0x7fc81c19df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.550+0000 7fc821910640 1 --2- 172.21.15.40:0/3113350217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc81c101930 0x7fc81c196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.550+0000 7fc821910640 1 --2- 172.21.15.40:0/3113350217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc81c100740 0x7fc81c1969b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.550+0000 7fc821910640 1 -- 172.21.15.40:0/3113350217 >> 172.21.15.40:0/3113350217 conn(0x7fc81c0fbcb0 msgr2=0x7fc81c104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.550+0000 7fc821910640 1 -- 172.21.15.40:0/3113350217 shutdown_connections 2024-03-18T22:46:25.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.550+0000 7fc821910640 1 -- 172.21.15.40:0/3113350217 wait complete. 2024-03-18T22:46:25.562 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:46:25.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.677+0000 7fa4107f4640 1 -- 172.21.15.40:0/3325864226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4081022d0 msgr2=0x7fa408108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.677+0000 7fa4107f4640 1 --2- 172.21.15.40:0/3325864226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4081022d0 0x7fa408108b60 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa400009ca0 tx=0x7fa40002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.677+0000 7fa4107f4640 1 -- 172.21.15.40:0/3325864226 shutdown_connections 2024-03-18T22:46:25.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.677+0000 7fa4107f4640 1 --2- 172.21.15.40:0/3325864226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4081022d0 0x7fa408108b60 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.677+0000 7fa4107f4640 1 --2- 172.21.15.40:0/3325864226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa408101910 0x7fa408101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.677+0000 7fa4107f4640 1 --2- 172.21.15.40:0/3325864226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa408100720 0x7fa408100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.677+0000 7fa4107f4640 1 -- 172.21.15.40:0/3325864226 >> 172.21.15.40:0/3325864226 conn(0x7fa4080fbc90 msgr2=0x7fa4080fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.677+0000 7fa4107f4640 1 -- 172.21.15.40:0/3325864226 shutdown_connections 2024-03-18T22:46:25.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.677+0000 7fa4107f4640 1 -- 172.21.15.40:0/3325864226 wait complete. 2024-03-18T22:46:25.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.677+0000 7fa4107f4640 1 Processor -- start 2024-03-18T22:46:25.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.677+0000 7fa4107f4640 1 -- start start 2024-03-18T22:46:25.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa4107f4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408100720 0x7fa408197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa4107f4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa408101910 0x7fa408197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa4107f4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4081022d0 0x7fa40819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa4107f4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa40810c250 con 0x7fa408100720 2024-03-18T22:46:25.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa4107f4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa40810c0d0 con 0x7fa408101910 2024-03-18T22:46:25.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa4107f4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa40810c3d0 con 0x7fa4081022d0 2024-03-18T22:46:25.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa40e590640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408100720 0x7fa408197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa40e590640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408100720 0x7fa408197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44038/0 (socket says 172.21.15.40:44038) 2024-03-18T22:46:25.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa40e590640 1 -- 172.21.15.40:0/1566771811 learned_addr learned my addr 172.21.15.40:0/1566771811 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa40ed91640 1 --2- 172.21.15.40:0/1566771811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4081022d0 0x7fa40819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa40dd8f640 1 --2- 172.21.15.40:0/1566771811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa408101910 0x7fa408197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa40ed91640 1 -- 172.21.15.40:0/1566771811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa408101910 msgr2=0x7fa408197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa40ed91640 1 --2- 172.21.15.40:0/1566771811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa408101910 0x7fa408197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa40ed91640 1 -- 172.21.15.40:0/1566771811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408100720 msgr2=0x7fa408197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa40ed91640 1 --2- 172.21.15.40:0/1566771811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408100720 0x7fa408197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa40ed91640 1 -- 172.21.15.40:0/1566771811 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa40819f650 con 0x7fa4081022d0 2024-03-18T22:46:25.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa40e590640 1 --2- 172.21.15.40:0/1566771811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408100720 0x7fa408197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:25.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.678+0000 7fa40ed91640 1 --2- 172.21.15.40:0/1566771811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4081022d0 0x7fa40819efe0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa40002c630 tx=0x7fa4000328a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.679+0000 7fa3ff7fe640 1 -- 172.21.15.40:0/1566771811 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa400041020 con 0x7fa4081022d0 2024-03-18T22:46:25.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.679+0000 7fa3ff7fe640 1 -- 172.21.15.40:0/1566771811 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa400003e20 con 0x7fa4081022d0 2024-03-18T22:46:25.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.679+0000 7fa4107f4640 1 -- 172.21.15.40:0/1566771811 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa40819f8e0 con 0x7fa4081022d0 2024-03-18T22:46:25.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.679+0000 7fa3ff7fe640 1 -- 172.21.15.40:0/1566771811 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa400039040 con 0x7fa4081022d0 2024-03-18T22:46:25.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.679+0000 7fa4107f4640 1 -- 172.21.15.40:0/1566771811 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa40819fdd0 con 0x7fa4081022d0 2024-03-18T22:46:25.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.681+0000 7fa3ff7fe640 1 -- 172.21.15.40:0/1566771811 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa400044020 con 0x7fa4081022d0 2024-03-18T22:46:25.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa4133e6740 1 Processor -- start 2024-03-18T22:46:25.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa4133e6740 1 -- start start 2024-03-18T22:46:25.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa4133e6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277c700 0x55b4e277cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa4133e6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277d0f0 0x55b4e2781bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa4133e6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e2782260 0x55b4e27846f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa4133e6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b4e26dc4b0 con 0x55b4e277c700 2024-03-18T22:46:25.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa4133e6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b4e2684d70 con 0x55b4e2782260 2024-03-18T22:46:25.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa4133e6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55b4e26d8550 con 0x55b4e277d0f0 2024-03-18T22:46:25.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa3ff7fe640 1 --2- 172.21.15.40:0/1566771811 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa3f40a2800 0x7fa3f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa40dd8f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277c700 0x55b4e277cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa40e590640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277d0f0 0x55b4e2781bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa40ed91640 1 --2- 172.21.15.40:0/1566771811 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa3f40a2800 0x7fa3f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa40dd8f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277c700 0x55b4e277cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44042/0 (socket says 172.21.15.40:44042) 2024-03-18T22:46:25.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa40dd8f640 1 -- 172.21.15.40:0/2454763872 learned_addr learned my addr 172.21.15.40:0/2454763872 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa40dd8f640 1 --2- 172.21.15.40:0/2454763872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e2782260 0x55b4e27846f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.682+0000 7fa3ff7fe640 1 -- 172.21.15.40:0/1566771811 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa4000f0fe0 con 0x7fa4081022d0 2024-03-18T22:46:25.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.683+0000 7fa40dd8f640 1 -- 172.21.15.40:0/2454763872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277d0f0 msgr2=0x55b4e2781bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.683+0000 7fa40dd8f640 1 --2- 172.21.15.40:0/2454763872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277d0f0 0x55b4e2781bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.683+0000 7fa40dd8f640 1 -- 172.21.15.40:0/2454763872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e2782260 msgr2=0x55b4e27846f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.683+0000 7fa40dd8f640 1 --2- 172.21.15.40:0/2454763872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e2782260 0x55b4e27846f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.683+0000 7fa40dd8f640 1 -- 172.21.15.40:0/2454763872 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4e26a7210 con 0x55b4e277c700 2024-03-18T22:46:25.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.683+0000 7fa40e590640 1 --2- 172.21.15.40:0/2454763872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277d0f0 0x55b4e2781bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:25.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.683+0000 7fa40dd8f640 1 --2- 172.21.15.40:0/2454763872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277c700 0x55b4e277cb20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80137f0 tx=0x7fa3f80097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e45d9581e6923850 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.683+0000 7fa40ed91640 1 --2- 172.21.15.40:0/1566771811 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa3f40a2800 0x7fa3f40a4cc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa4000bd520 tx=0x7fa4000887f0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.683+0000 7fa3fcff9640 1 -- 172.21.15.40:0/2454763872 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8013d70 con 0x55b4e277c700 2024-03-18T22:46:25.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.683+0000 7fa3fcff9640 1 -- 172.21.15.40:0/2454763872 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3f8005020 con 0x55b4e277c700 2024-03-18T22:46:25.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.683+0000 7fa3fcff9640 1 -- 172.21.15.40:0/2454763872 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8004a00 con 0x55b4e277c700 2024-03-18T22:46:25.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.684+0000 7fa4133e6740 1 -- 172.21.15.40:0/2454763872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277c700 msgr2=0x55b4e277cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.684+0000 7fa4133e6740 1 --2- 172.21.15.40:0/2454763872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277c700 0x55b4e277cb20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80137f0 tx=0x7fa3f80097a0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.684+0000 7fa4133e6740 1 -- 172.21.15.40:0/2454763872 shutdown_connections 2024-03-18T22:46:25.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.684+0000 7fa4133e6740 1 --2- 172.21.15.40:0/2454763872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e2782260 0x55b4e27846f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.684+0000 7fa4133e6740 1 --2- 172.21.15.40:0/2454763872 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277d0f0 0x55b4e2781bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.684+0000 7fa4133e6740 1 --2- 172.21.15.40:0/2454763872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277c700 0x55b4e277cb20 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.684+0000 7fa4133e6740 1 -- 172.21.15.40:0/2454763872 >> 172.21.15.40:0/2454763872 conn(0x55b4e2773860 msgr2=0x55b4e2773c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.684+0000 7fa4133e6740 1 -- 172.21.15.40:0/2454763872 shutdown_connections 2024-03-18T22:46:25.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.684+0000 7fa4133e6740 1 -- 172.21.15.40:0/2454763872 wait complete. 2024-03-18T22:46:25.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.684+0000 7fa4133e6740 1 -- start start 2024-03-18T22:46:25.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.684+0000 7fa4133e6740 1 Processor -- start 2024-03-18T22:46:25.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa4133e6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277c700 0x55b4e2786ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa4133e6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277d0f0 0x55b4e27870e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa4133e6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e2782260 0x55b4e278e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa4133e6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b4e27851b0 con 0x55b4e277d0f0 2024-03-18T22:46:25.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa4133e6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b4e2698310 con 0x55b4e2782260 2024-03-18T22:46:25.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa4133e6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55b4e2785330 con 0x55b4e277c700 2024-03-18T22:46:25.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa4133e6740 10 client.0 _ll_register_callbacks cb 0x55b4e268c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:25.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa40e590640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277d0f0 0x55b4e27870e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa40e590640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277d0f0 0x55b4e27870e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44052/0 (socket says 172.21.15.40:44052) 2024-03-18T22:46:25.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa40e590640 1 -- 172.21.15.40:0/222541077 learned_addr learned my addr 172.21.15.40:0/222541077 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa40dd8f640 1 --2- 172.21.15.40:0/222541077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e2782260 0x55b4e278e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa40dd8f640 1 --2- 172.21.15.40:0/222541077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277c700 0x55b4e2786ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa40e590640 1 -- 172.21.15.40:0/222541077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277c700 msgr2=0x55b4e2786ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa40e590640 1 --2- 172.21.15.40:0/222541077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277c700 0x55b4e2786ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa40e590640 1 -- 172.21.15.40:0/222541077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e2782260 msgr2=0x55b4e278e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa40e590640 1 --2- 172.21.15.40:0/222541077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e2782260 0x55b4e278e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa40e590640 1 -- 172.21.15.40:0/222541077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4e23f03f0 con 0x55b4e277d0f0 2024-03-18T22:46:25.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa40e590640 1 --2- 172.21.15.40:0/222541077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277d0f0 0x55b4e27870e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa404012cf0 tx=0x7fa40400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=781b08560d162e4d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa3f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:25.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa3f1ffb640 1 -- 172.21.15.40:0/222541077 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4040120d0 con 0x55b4e277d0f0 2024-03-18T22:46:25.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa3f1ffb640 1 -- 172.21.15.40:0/222541077 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa404004020 con 0x55b4e277d0f0 2024-03-18T22:46:25.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.685+0000 7fa4133e6740 1 -- 172.21.15.40:0/222541077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3f8005020 con 0x55b4e277d0f0 2024-03-18T22:46:25.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.686+0000 7fa3f1ffb640 1 -- 172.21.15.40:0/222541077 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa404025050 con 0x55b4e277d0f0 2024-03-18T22:46:25.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.686+0000 7fa4133e6740 10 client.16440 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:25.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.686+0000 7fa4133e6740 1 -- 172.21.15.40:0/222541077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b4e24ba7d0 con 0x55b4e277d0f0 2024-03-18T22:46:25.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.686+0000 7fa3f1ffb640 1 -- 172.21.15.40:0/222541077 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3f8005020 con 0x55b4e277d0f0 2024-03-18T22:46:25.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.686+0000 7fa3f1ffb640 1 -- 172.21.15.40:0/222541077 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fa40401c470 con 0x55b4e277d0f0 2024-03-18T22:46:25.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.686+0000 7fa4133e6740 10 client.16440 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:25.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.686+0000 7fa4133e6740 10 client.16440 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:46:25.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.686+0000 7fa4133e6740 20 client.16440 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:25.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.686+0000 7fa4133e6740 1 --2- 172.21.15.40:0/222541077 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55b4e2792b00 0x55b4e2794f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.686+0000 7fa4133e6740 4 client.16440 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:46:25.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.686+0000 7fa4133e6740 1 -- 172.21.15.40:0/222541077 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4e273a3e0 con 0x55b4e2792b00 2024-03-18T22:46:25.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.686+0000 7fa40dd8f640 1 --2- 172.21.15.40:0/222541077 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55b4e2792b00 0x55b4e2794f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.687+0000 7fa3f1ffb640 10 client.16440 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:46:25.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.687+0000 7fa40dd8f640 1 --2- 172.21.15.40:0/222541077 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55b4e2792b00 0x55b4e2794f40 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b7d9cd0d98e5f82d server_cookie=c0cc4765c398ea17 in_seq=0 out_seq=0 2024-03-18T22:46:25.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.688+0000 7fa3f1ffb640 1 -- 172.21.15.40:0/222541077 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b4e273a3e0 con 0x55b4e2792b00 2024-03-18T22:46:25.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.688+0000 7fa3f1ffb640 10 client.16440 handle_command_reply: tid=0 2024-03-18T22:46:25.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.688+0000 7fa4133e6740 1 client.16440 shutdown 2024-03-18T22:46:25.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.688+0000 7fa4133e6740 1 -- 172.21.15.40:0/222541077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277d0f0 msgr2=0x55b4e27870e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.688+0000 7fa4133e6740 1 --2- 172.21.15.40:0/222541077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277d0f0 0x55b4e27870e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa404012cf0 tx=0x7fa40400f790 comp rx=0 tx=0).stop 2024-03-18T22:46:25.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.688+0000 7fa4133e6740 1 -- 172.21.15.40:0/222541077 shutdown_connections 2024-03-18T22:46:25.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.688+0000 7fa4133e6740 1 --2- 172.21.15.40:0/222541077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e2782260 0x55b4e278e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.688+0000 7fa4133e6740 1 --2- 172.21.15.40:0/222541077 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55b4e2792b00 0x55b4e2794f40 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.688+0000 7fa4133e6740 1 --2- 172.21.15.40:0/222541077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277d0f0 0x55b4e27870e0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.688+0000 7fa4133e6740 1 --2- 172.21.15.40:0/222541077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277c700 0x55b4e2786ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.688+0000 7fa4133e6740 1 -- 172.21.15.40:0/222541077 >> 172.21.15.40:0/222541077 conn(0x55b4e2773860 msgr2=0x55b4e2772ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.688+0000 7fa3f1ffb640 0 client.16440 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:46:25.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.688+0000 7fa4133e6740 1 -- 172.21.15.40:0/222541077 shutdown_connections 2024-03-18T22:46:25.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.688+0000 7fa4133e6740 1 -- 172.21.15.40:0/222541077 wait complete. 2024-03-18T22:46:25.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.688+0000 7fa4133e6740 20 client.16440 trim_cache size 0 max 16384 2024-03-18T22:46:25.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa4133e6740 1 Processor -- start 2024-03-18T22:46:25.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa4133e6740 1 -- start start 2024-03-18T22:46:25.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa4133e6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277c700 0x55b4e2785700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa4133e6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e277d0f0 0x55b4e2785c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa4133e6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e2782260 0x55b4e278e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa4133e6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa4081a82b0 con 0x55b4e277c700 2024-03-18T22:46:25.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa4133e6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa4081a9e20 con 0x55b4e277d0f0 2024-03-18T22:46:25.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa4133e6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa4081a8980 con 0x55b4e2782260 2024-03-18T22:46:25.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa40dd8f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277c700 0x55b4e2785700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa40dd8f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277c700 0x55b4e2785700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44062/0 (socket says 172.21.15.40:44062) 2024-03-18T22:46:25.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa40dd8f640 1 -- 172.21.15.40:0/3505040087 learned_addr learned my addr 172.21.15.40:0/3505040087 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa40e590640 1 --2- 172.21.15.40:0/3505040087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e277d0f0 0x55b4e2785c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa40dd8f640 1 --2- 172.21.15.40:0/3505040087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e2782260 0x55b4e278e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa40e590640 1 -- 172.21.15.40:0/3505040087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e2782260 msgr2=0x55b4e278e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa40e590640 1 --2- 172.21.15.40:0/3505040087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e2782260 0x55b4e278e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa40e590640 1 -- 172.21.15.40:0/3505040087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277c700 msgr2=0x55b4e2785700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa40e590640 1 --2- 172.21.15.40:0/3505040087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277c700 0x55b4e2785700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa40e590640 1 -- 172.21.15.40:0/3505040087 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4e27818a0 con 0x55b4e277d0f0 2024-03-18T22:46:25.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa40dd8f640 1 --2- 172.21.15.40:0/3505040087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277c700 0x55b4e2785700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:25.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.703+0000 7fa40e590640 1 --2- 172.21.15.40:0/3505040087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e277d0f0 0x55b4e2785c40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa404009600 tx=0x7fa4040122e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4993a7edd3d5046 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.704+0000 7fa3f3fff640 1 -- 172.21.15.40:0/3505040087 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa404022040 con 0x55b4e277d0f0 2024-03-18T22:46:25.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.704+0000 7fa3f3fff640 1 -- 172.21.15.40:0/3505040087 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa404005580 con 0x55b4e277d0f0 2024-03-18T22:46:25.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.704+0000 7fa3f3fff640 1 -- 172.21.15.40:0/3505040087 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa404010430 con 0x55b4e277d0f0 2024-03-18T22:46:25.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.704+0000 7fa4133e6740 1 -- 172.21.15.40:0/3505040087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e277d0f0 msgr2=0x55b4e2785c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.704+0000 7fa4133e6740 1 --2- 172.21.15.40:0/3505040087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e277d0f0 0x55b4e2785c40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa404009600 tx=0x7fa4040122e0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.704+0000 7fa4133e6740 1 -- 172.21.15.40:0/3505040087 shutdown_connections 2024-03-18T22:46:25.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.704+0000 7fa4133e6740 1 --2- 172.21.15.40:0/3505040087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e2782260 0x55b4e278e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.704+0000 7fa4133e6740 1 --2- 172.21.15.40:0/3505040087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e277d0f0 0x55b4e2785c40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.704+0000 7fa4133e6740 1 --2- 172.21.15.40:0/3505040087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e277c700 0x55b4e2785700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.704+0000 7fa4133e6740 1 -- 172.21.15.40:0/3505040087 >> 172.21.15.40:0/3505040087 conn(0x55b4e2773860 msgr2=0x55b4e2773c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.704+0000 7fa4133e6740 1 -- 172.21.15.40:0/3505040087 shutdown_connections 2024-03-18T22:46:25.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa4133e6740 1 -- 172.21.15.40:0/3505040087 wait complete. 2024-03-18T22:46:25.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa4133e6740 1 -- start start 2024-03-18T22:46:25.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa4133e6740 1 Processor -- start 2024-03-18T22:46:25.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa4133e6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e277c700 0x55b4e278add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa4133e6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277d0f0 0x55b4e278b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa4133e6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e2782260 0x55b4e27f6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa4133e6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa4081a9720 con 0x55b4e2782260 2024-03-18T22:46:25.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa4133e6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa4081a9050 con 0x55b4e277c700 2024-03-18T22:46:25.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa4133e6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa4081a6290 con 0x55b4e277d0f0 2024-03-18T22:46:25.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa4133e6740 10 client.0 _ll_register_callbacks cb 0x7fa404009cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:25.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa40dd8f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e2782260 0x55b4e27f6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa40dd8f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e2782260 0x55b4e27f6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44066/0 (socket says 172.21.15.40:44066) 2024-03-18T22:46:25.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa40dd8f640 1 -- 172.21.15.40:0/2050642957 learned_addr learned my addr 172.21.15.40:0/2050642957 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa40e590640 1 --2- 172.21.15.40:0/2050642957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277d0f0 0x55b4e278b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa40dd8f640 1 --2- 172.21.15.40:0/2050642957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e277c700 0x55b4e278add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa40dd8f640 1 -- 172.21.15.40:0/2050642957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277d0f0 msgr2=0x55b4e278b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa40dd8f640 1 --2- 172.21.15.40:0/2050642957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277d0f0 0x55b4e278b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa40dd8f640 1 -- 172.21.15.40:0/2050642957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e277c700 msgr2=0x55b4e278add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.705+0000 7fa40dd8f640 1 --2- 172.21.15.40:0/2050642957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e277c700 0x55b4e278add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.706+0000 7fa40dd8f640 1 -- 172.21.15.40:0/2050642957 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4e278e3e0 con 0x55b4e2782260 2024-03-18T22:46:25.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.706+0000 7fa40dd8f640 1 --2- 172.21.15.40:0/2050642957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e2782260 0x55b4e27f6e70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009b90 tx=0x7fa3f8009bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d05aa46a9f1b67 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.706+0000 7fa3f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:25.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.706+0000 7fa3f1ffb640 1 -- 172.21.15.40:0/2050642957 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f80313d0 con 0x55b4e2782260 2024-03-18T22:46:25.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.706+0000 7fa4133e6740 1 -- 172.21.15.40:0/2050642957 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa404005580 con 0x55b4e2782260 2024-03-18T22:46:25.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.706+0000 7fa3f1ffb640 1 -- 172.21.15.40:0/2050642957 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa3f8031570 con 0x55b4e2782260 2024-03-18T22:46:25.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.706+0000 7fa3f1ffb640 1 -- 172.21.15.40:0/2050642957 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa3f8031770 con 0x55b4e2782260 2024-03-18T22:46:25.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.706+0000 7fa4133e6740 10 client.16446 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:25.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.706+0000 7fa4133e6740 1 -- 172.21.15.40:0/2050642957 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b4e27f83b0 con 0x55b4e2782260 2024-03-18T22:46:25.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.706+0000 7fa3f1ffb640 1 -- 172.21.15.40:0/2050642957 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa404005580 con 0x55b4e2782260 2024-03-18T22:46:25.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.706+0000 7fa3f1ffb640 1 -- 172.21.15.40:0/2050642957 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fa3f8030040 con 0x55b4e2782260 2024-03-18T22:46:25.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.706+0000 7fa4133e6740 10 client.16446 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:25.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.706+0000 7fa4133e6740 10 client.16446 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:46:25.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.706+0000 7fa4133e6740 20 client.16446 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:25.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.706+0000 7fa4133e6740 1 --2- 172.21.15.40:0/2050642957 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55b4e27f8c00 0x55b4e27fb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa4133e6740 4 client.16446 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:46:25.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa4133e6740 1 -- 172.21.15.40:0/2050642957 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b4e26e99f0 con 0x55b4e27f8c00 2024-03-18T22:46:25.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa40e590640 1 --2- 172.21.15.40:0/2050642957 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55b4e27f8c00 0x55b4e27fb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa40e590640 1 --2- 172.21.15.40:0/2050642957 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55b4e27f8c00 0x55b4e27fb020 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d12f067687e9e8d0 server_cookie=cae34e1af6afc568 in_seq=0 out_seq=0 2024-03-18T22:46:25.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa3f1ffb640 10 client.16446 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:46:25.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa3f1ffb640 1 -- 172.21.15.40:0/2050642957 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b4e26e99f0 con 0x55b4e27f8c00 2024-03-18T22:46:25.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa3f1ffb640 10 client.16446 handle_command_reply: tid=0 2024-03-18T22:46:25.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa4133e6740 1 client.16446 shutdown 2024-03-18T22:46:25.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa4133e6740 1 -- 172.21.15.40:0/2050642957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e2782260 msgr2=0x55b4e27f6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa4133e6740 1 --2- 172.21.15.40:0/2050642957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e2782260 0x55b4e27f6e70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009b90 tx=0x7fa3f8009bc0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa4133e6740 1 -- 172.21.15.40:0/2050642957 shutdown_connections 2024-03-18T22:46:25.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa4133e6740 1 --2- 172.21.15.40:0/2050642957 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55b4e27f8c00 0x55b4e27fb020 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa4133e6740 1 --2- 172.21.15.40:0/2050642957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b4e2782260 0x55b4e27f6e70 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa3f1ffb640 0 client.16446 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:46:25.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa4133e6740 1 --2- 172.21.15.40:0/2050642957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b4e277d0f0 0x55b4e278b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa4133e6740 1 --2- 172.21.15.40:0/2050642957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b4e277c700 0x55b4e278add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.707+0000 7fa4133e6740 1 -- 172.21.15.40:0/2050642957 >> 172.21.15.40:0/2050642957 conn(0x55b4e2773860 msgr2=0x55b4e2785700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.708+0000 7fa4133e6740 1 -- 172.21.15.40:0/2050642957 shutdown_connections 2024-03-18T22:46:25.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.708+0000 7fa4133e6740 1 -- 172.21.15.40:0/2050642957 wait complete. 2024-03-18T22:46:25.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.708+0000 7fa4133e6740 20 client.16446 trim_cache size 0 max 16384 2024-03-18T22:46:25.717 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:46:25.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.708+0000 7fa4107f4640 1 -- 172.21.15.40:0/1566771811 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa3f40a2800 msgr2=0x7fa3f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.708+0000 7fa4107f4640 1 --2- 172.21.15.40:0/1566771811 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa3f40a2800 0x7fa3f40a4cc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa4000bd520 tx=0x7fa4000887f0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.708+0000 7fa4107f4640 1 -- 172.21.15.40:0/1566771811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4081022d0 msgr2=0x7fa40819efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.708+0000 7fa4107f4640 1 --2- 172.21.15.40:0/1566771811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4081022d0 0x7fa40819efe0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa40002c630 tx=0x7fa4000328a0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.708+0000 7fa4107f4640 1 -- 172.21.15.40:0/1566771811 shutdown_connections 2024-03-18T22:46:25.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.708+0000 7fa4107f4640 1 --2- 172.21.15.40:0/1566771811 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa3f40a2800 0x7fa3f40a4cc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.708+0000 7fa4107f4640 1 --2- 172.21.15.40:0/1566771811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4081022d0 0x7fa40819efe0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.708+0000 7fa4107f4640 1 --2- 172.21.15.40:0/1566771811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa408101910 0x7fa408197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.708+0000 7fa4107f4640 1 --2- 172.21.15.40:0/1566771811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa408100720 0x7fa408197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.708+0000 7fa4107f4640 1 -- 172.21.15.40:0/1566771811 >> 172.21.15.40:0/1566771811 conn(0x7fa4080fbc90 msgr2=0x7fa4080fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.708+0000 7fa4107f4640 1 -- 172.21.15.40:0/1566771811 shutdown_connections 2024-03-18T22:46:25.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.708+0000 7fa4107f4640 1 -- 172.21.15.40:0/1566771811 wait complete. 2024-03-18T22:46:25.720 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:46:25.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.842+0000 7f57cedd4640 1 -- 172.21.15.40:0/3711261599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c8101920 msgr2=0x7f57c8101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.842+0000 7f57cedd4640 1 --2- 172.21.15.40:0/3711261599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c8101920 0x7f57c8101da0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009900 tx=0x7f57b402bc30 comp rx=0 tx=0).stop 2024-03-18T22:46:25.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.842+0000 7f57cedd4640 1 -- 172.21.15.40:0/3711261599 shutdown_connections 2024-03-18T22:46:25.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.842+0000 7f57cedd4640 1 --2- 172.21.15.40:0/3711261599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57c81022e0 0x7f57c8108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.842+0000 7f57cedd4640 1 --2- 172.21.15.40:0/3711261599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c8101920 0x7f57c8101da0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.842+0000 7f57cedd4640 1 --2- 172.21.15.40:0/3711261599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57c8100730 0x7f57c8100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.842+0000 7f57cedd4640 1 -- 172.21.15.40:0/3711261599 >> 172.21.15.40:0/3711261599 conn(0x7f57c80fbca0 msgr2=0x7f57c80fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.842+0000 7f57cedd4640 1 -- 172.21.15.40:0/3711261599 shutdown_connections 2024-03-18T22:46:25.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.842+0000 7f57cedd4640 1 -- 172.21.15.40:0/3711261599 wait complete. 2024-03-18T22:46:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57cedd4640 1 Processor -- start 2024-03-18T22:46:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57cedd4640 1 -- start start 2024-03-18T22:46:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57cedd4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57c8100730 0x7f57c806a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57cedd4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57c8101920 0x7f57c819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57cedd4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c81022e0 0x7f57c819ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57cedd4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f57c810c260 con 0x7f57c81022e0 2024-03-18T22:46:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57cedd4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57c810c0e0 con 0x7f57c8100730 2024-03-18T22:46:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57cedd4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f57c810c3e0 con 0x7f57c8101920 2024-03-18T22:46:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57cd371640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c81022e0 0x7f57c819ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57ccb70640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57c8100730 0x7f57c806a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57c7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57c8101920 0x7f57c819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57cd371640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c81022e0 0x7f57c819ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44076/0 (socket says 172.21.15.40:44076) 2024-03-18T22:46:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57cd371640 1 -- 172.21.15.40:0/515456026 learned_addr learned my addr 172.21.15.40:0/515456026 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57ccb70640 1 -- 172.21.15.40:0/515456026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57c8101920 msgr2=0x7f57c819c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57ccb70640 1 --2- 172.21.15.40:0/515456026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57c8101920 0x7f57c819c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57ccb70640 1 -- 172.21.15.40:0/515456026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c81022e0 msgr2=0x7f57c819ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57ccb70640 1 --2- 172.21.15.40:0/515456026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c81022e0 0x7f57c819ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57ccb70640 1 -- 172.21.15.40:0/515456026 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c819f640 con 0x7f57c8100730 2024-03-18T22:46:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.843+0000 7f57cd371640 1 --2- 172.21.15.40:0/515456026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c81022e0 0x7f57c819ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.844+0000 7f57ccb70640 1 --2- 172.21.15.40:0/515456026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57c8100730 0x7f57c806a980 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00e9c0 tx=0x7f57bc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.844+0000 7f57c5ffb640 1 -- 172.21.15.40:0/515456026 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc00ccc0 con 0x7f57c8100730 2024-03-18T22:46:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.844+0000 7f57cedd4640 1 -- 172.21.15.40:0/515456026 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c819f930 con 0x7f57c8100730 2024-03-18T22:46:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.844+0000 7f57c5ffb640 1 -- 172.21.15.40:0/515456026 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57bc0035f0 con 0x7f57c8100730 2024-03-18T22:46:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.844+0000 7f57c5ffb640 1 -- 172.21.15.40:0/515456026 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc00ce60 con 0x7f57c8100730 2024-03-18T22:46:25.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.844+0000 7f57cedd4640 1 -- 172.21.15.40:0/515456026 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c819fe00 con 0x7f57c8100730 2024-03-18T22:46:25.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.846+0000 7f57c5ffb640 1 -- 172.21.15.40:0/515456026 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f57bc0037a0 con 0x7f57c8100730 2024-03-18T22:46:25.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57c5ffb640 1 --2- 172.21.15.40:0/515456026 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57b80a2800 0x7f57b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57c7fff640 1 --2- 172.21.15.40:0/515456026 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57b80a2800 0x7f57b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57c5ffb640 1 -- 172.21.15.40:0/515456026 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f57bc0cf1d0 con 0x7f57c8100730 2024-03-18T22:46:25.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57d19c4740 1 Processor -- start 2024-03-18T22:46:25.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57d19c4740 1 -- start start 2024-03-18T22:46:25.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57d19c4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac82f0 0x561f9bac8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57d19c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac8ce0 0x561f9bacd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57d19c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bacde50 0x561f9bad02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57d19c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561f9ba2c4b0 con 0x561f9bac8ce0 2024-03-18T22:46:25.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57d19c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561f9b9d4d70 con 0x561f9bacde50 2024-03-18T22:46:25.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57d19c4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x561f9ba28550 con 0x561f9bac82f0 2024-03-18T22:46:25.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57ccb70640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac82f0 0x561f9bac8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57cd371640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bacde50 0x561f9bad02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57c7fff640 1 --2- 172.21.15.40:0/515456026 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57b80a2800 0x7f57b80a4cc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009230 tx=0x7f57b4005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57ccb70640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac82f0 0x561f9bac8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55762/0 (socket says 172.21.15.40:55762) 2024-03-18T22:46:25.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57ccb70640 1 -- 172.21.15.40:0/3832769579 learned_addr learned my addr 172.21.15.40:0/3832769579 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57c7fff640 1 --2- 172.21.15.40:0/3832769579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac8ce0 0x561f9bacd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.847+0000 7f57cd371640 1 --2- 172.21.15.40:0/3832769579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bacde50 0x561f9bad02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47330/0 (socket says 172.21.15.40:47330) 2024-03-18T22:46:25.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.848+0000 7f57ccb70640 1 -- 172.21.15.40:0/3832769579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bacde50 msgr2=0x561f9bad02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.848+0000 7f57ccb70640 1 --2- 172.21.15.40:0/3832769579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bacde50 0x561f9bad02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.848+0000 7f57ccb70640 1 -- 172.21.15.40:0/3832769579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac8ce0 msgr2=0x561f9bacd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.848+0000 7f57ccb70640 1 --2- 172.21.15.40:0/3832769579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac8ce0 0x561f9bacd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.848+0000 7f57ccb70640 1 -- 172.21.15.40:0/3832769579 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f9b9f7210 con 0x561f9bac82f0 2024-03-18T22:46:25.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.848+0000 7f57ccb70640 1 --2- 172.21.15.40:0/3832769579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac82f0 0x561f9bac8710 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0cc880 tx=0x7f57bc09b340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64240197df3aab41 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.848+0000 7f57b2ffd640 1 -- 172.21.15.40:0/3832769579 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0b4070 con 0x561f9bac82f0 2024-03-18T22:46:25.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.848+0000 7f57b2ffd640 1 -- 172.21.15.40:0/3832769579 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57bc09b780 con 0x561f9bac82f0 2024-03-18T22:46:25.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.848+0000 7f57b2ffd640 1 -- 172.21.15.40:0/3832769579 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc097df0 con 0x561f9bac82f0 2024-03-18T22:46:25.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.849+0000 7f57d19c4740 1 -- 172.21.15.40:0/3832769579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac82f0 msgr2=0x561f9bac8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.849+0000 7f57d19c4740 1 --2- 172.21.15.40:0/3832769579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac82f0 0x561f9bac8710 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0cc880 tx=0x7f57bc09b340 comp rx=0 tx=0).stop 2024-03-18T22:46:25.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.849+0000 7f57d19c4740 1 -- 172.21.15.40:0/3832769579 shutdown_connections 2024-03-18T22:46:25.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.849+0000 7f57d19c4740 1 --2- 172.21.15.40:0/3832769579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bacde50 0x561f9bad02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.849+0000 7f57d19c4740 1 --2- 172.21.15.40:0/3832769579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac8ce0 0x561f9bacd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.849+0000 7f57d19c4740 1 --2- 172.21.15.40:0/3832769579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac82f0 0x561f9bac8710 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.849+0000 7f57d19c4740 1 -- 172.21.15.40:0/3832769579 >> 172.21.15.40:0/3832769579 conn(0x561f9bac3860 msgr2=0x561f9bac3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.849+0000 7f57d19c4740 1 -- 172.21.15.40:0/3832769579 shutdown_connections 2024-03-18T22:46:25.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.849+0000 7f57d19c4740 1 -- 172.21.15.40:0/3832769579 wait complete. 2024-03-18T22:46:25.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.849+0000 7f57d19c4740 1 -- start start 2024-03-18T22:46:25.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.849+0000 7f57d19c4740 1 Processor -- start 2024-03-18T22:46:25.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57d19c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bac82f0 0x561f9bad27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57d19c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac8ce0 0x561f9bad2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57d19c4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bacde50 0x561f9bad9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57d19c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561f9bad0e00 con 0x561f9bac8ce0 2024-03-18T22:46:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57d19c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561f9b9e8310 con 0x561f9bac82f0 2024-03-18T22:46:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57d19c4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x561f9bad0f80 con 0x561f9bacde50 2024-03-18T22:46:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57d19c4740 10 client.0 _ll_register_callbacks cb 0x561f9b9dc2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57c7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac8ce0 0x561f9bad2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57cd371640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bacde50 0x561f9bad9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57c7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac8ce0 0x561f9bad2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44096/0 (socket says 172.21.15.40:44096) 2024-03-18T22:46:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57cd371640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bacde50 0x561f9bad9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55776/0 (socket says 172.21.15.40:55776) 2024-03-18T22:46:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57c7fff640 1 -- 172.21.15.40:0/3376602348 learned_addr learned my addr 172.21.15.40:0/3376602348 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57ccb70640 1 --2- 172.21.15.40:0/3376602348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bac82f0 0x561f9bad27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57c7fff640 1 -- 172.21.15.40:0/3376602348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bacde50 msgr2=0x561f9bad9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57c7fff640 1 --2- 172.21.15.40:0/3376602348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bacde50 0x561f9bad9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57c7fff640 1 -- 172.21.15.40:0/3376602348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bac82f0 msgr2=0x561f9bad27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57c7fff640 1 --2- 172.21.15.40:0/3376602348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bac82f0 0x561f9bad27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57c7fff640 1 -- 172.21.15.40:0/3376602348 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f9b7403f0 con 0x561f9bac8ce0 2024-03-18T22:46:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57cd371640 1 --2- 172.21.15.40:0/3376602348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bacde50 0x561f9bad9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57ccb70640 1 --2- 172.21.15.40:0/3376602348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bac82f0 0x561f9bad27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57c7fff640 1 --2- 172.21.15.40:0/3376602348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac8ce0 0x561f9bad2d10 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009900 tx=0x7f57b4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92796429755a9249 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.850+0000 7f57a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.851+0000 7f57a7fff640 1 -- 172.21.15.40:0/3376602348 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4050070 con 0x561f9bac8ce0 2024-03-18T22:46:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.851+0000 7f57a7fff640 1 -- 172.21.15.40:0/3376602348 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f57b40048d0 con 0x561f9bac8ce0 2024-03-18T22:46:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.851+0000 7f57d19c4740 1 -- 172.21.15.40:0/3376602348 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57bc09b780 con 0x561f9bac8ce0 2024-03-18T22:46:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.851+0000 7f57a7fff640 1 -- 172.21.15.40:0/3376602348 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f57b405c050 con 0x561f9bac8ce0 2024-03-18T22:46:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.851+0000 7f57d19c4740 10 client.16458 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.851+0000 7f57d19c4740 1 -- 172.21.15.40:0/3376602348 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561f9b80a7d0 con 0x561f9bac8ce0 2024-03-18T22:46:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.851+0000 7f57a7fff640 1 -- 172.21.15.40:0/3376602348 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57bc09b780 con 0x561f9bac8ce0 2024-03-18T22:46:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.851+0000 7f57a7fff640 1 -- 172.21.15.40:0/3376602348 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f57b4003c70 con 0x561f9bac8ce0 2024-03-18T22:46:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.851+0000 7f57d19c4740 10 client.16458 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.851+0000 7f57d19c4740 10 client.16458 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:46:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.851+0000 7f57d19c4740 20 client.16458 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.851+0000 7f57d19c4740 1 --2- 172.21.15.40:0/3376602348 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x561f9bae29b0 0x561f9bae4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.851+0000 7f57d19c4740 4 client.16458 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:46:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.851+0000 7f57d19c4740 1 -- 172.21.15.40:0/3376602348 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f9ba8a3e0 con 0x561f9bae29b0 2024-03-18T22:46:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.851+0000 7f57ccb70640 1 --2- 172.21.15.40:0/3376602348 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x561f9bae29b0 0x561f9bae4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.852+0000 7f57a7fff640 10 client.16458 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:46:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.852+0000 7f57ccb70640 1 --2- 172.21.15.40:0/3376602348 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x561f9bae29b0 0x561f9bae4e50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1d07a182cbdb184c server_cookie=15dbc76cc894c495 in_seq=0 out_seq=0 2024-03-18T22:46:25.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.853+0000 7f57a7fff640 1 -- 172.21.15.40:0/3376602348 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561f9ba8a3e0 con 0x561f9bae29b0 2024-03-18T22:46:25.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.853+0000 7f57a7fff640 10 client.16458 handle_command_reply: tid=0 2024-03-18T22:46:25.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.853+0000 7f57d19c4740 1 client.16458 shutdown 2024-03-18T22:46:25.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.853+0000 7f57d19c4740 1 -- 172.21.15.40:0/3376602348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac8ce0 msgr2=0x561f9bad2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.853+0000 7f57d19c4740 1 --2- 172.21.15.40:0/3376602348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac8ce0 0x561f9bad2d10 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009900 tx=0x7f57b4009260 comp rx=0 tx=0).stop 2024-03-18T22:46:25.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.853+0000 7f57d19c4740 1 -- 172.21.15.40:0/3376602348 shutdown_connections 2024-03-18T22:46:25.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.853+0000 7f57d19c4740 1 --2- 172.21.15.40:0/3376602348 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x561f9bae29b0 0x561f9bae4e50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.853+0000 7f57d19c4740 1 --2- 172.21.15.40:0/3376602348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bacde50 0x561f9bad9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.853+0000 7f57d19c4740 1 --2- 172.21.15.40:0/3376602348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac8ce0 0x561f9bad2d10 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.853+0000 7f57a7fff640 0 client.16458 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:46:25.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.853+0000 7f57d19c4740 1 --2- 172.21.15.40:0/3376602348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bac82f0 0x561f9bad27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.853+0000 7f57d19c4740 1 -- 172.21.15.40:0/3376602348 >> 172.21.15.40:0/3376602348 conn(0x561f9bac3860 msgr2=0x561f9bac2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.853+0000 7f57d19c4740 1 -- 172.21.15.40:0/3376602348 shutdown_connections 2024-03-18T22:46:25.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.853+0000 7f57d19c4740 1 -- 172.21.15.40:0/3376602348 wait complete. 2024-03-18T22:46:25.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.853+0000 7f57d19c4740 20 client.16458 trim_cache size 0 max 16384 2024-03-18T22:46:25.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57d19c4740 1 Processor -- start 2024-03-18T22:46:25.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57d19c4740 1 -- start start 2024-03-18T22:46:25.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57d19c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bac82f0 0x561f9bad27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57d19c4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac8ce0 0x561f9bad2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57d19c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bacde50 0x561f9bad9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57d19c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f57c81a8200 con 0x561f9bacde50 2024-03-18T22:46:25.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57d19c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57c81a9d70 con 0x561f9bac82f0 2024-03-18T22:46:25.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57d19c4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f57c81a88d0 con 0x561f9bac8ce0 2024-03-18T22:46:25.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57cd371640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bacde50 0x561f9bad9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57cd371640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bacde50 0x561f9bad9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44112/0 (socket says 172.21.15.40:44112) 2024-03-18T22:46:25.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57cd371640 1 -- 172.21.15.40:0/3808319938 learned_addr learned my addr 172.21.15.40:0/3808319938 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57ccb70640 1 --2- 172.21.15.40:0/3808319938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bac82f0 0x561f9bad27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57c7fff640 1 --2- 172.21.15.40:0/3808319938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac8ce0 0x561f9bad2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57cd371640 1 -- 172.21.15.40:0/3808319938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac8ce0 msgr2=0x561f9bad2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57cd371640 1 --2- 172.21.15.40:0/3808319938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac8ce0 0x561f9bad2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57cd371640 1 -- 172.21.15.40:0/3808319938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bac82f0 msgr2=0x561f9bad27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57cd371640 1 --2- 172.21.15.40:0/3808319938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bac82f0 0x561f9bad27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.871+0000 7f57cd371640 1 -- 172.21.15.40:0/3808319938 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f9bac7f30 con 0x561f9bacde50 2024-03-18T22:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.872+0000 7f57cd371640 1 --2- 172.21.15.40:0/3808319938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bacde50 0x561f9bad9d10 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f57c000fe70 tx=0x7f57c0009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ea14e6365d7993 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.872+0000 7f57b27fc640 1 -- 172.21.15.40:0/3808319938 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c00120d0 con 0x561f9bacde50 2024-03-18T22:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.872+0000 7f57b27fc640 1 -- 172.21.15.40:0/3808319938 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57c0012270 con 0x561f9bacde50 2024-03-18T22:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.872+0000 7f57b27fc640 1 -- 172.21.15.40:0/3808319938 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c00059d0 con 0x561f9bacde50 2024-03-18T22:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 -- 172.21.15.40:0/3808319938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bacde50 msgr2=0x561f9bad9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 --2- 172.21.15.40:0/3808319938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bacde50 0x561f9bad9d10 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f57c000fe70 tx=0x7f57c0009500 comp rx=0 tx=0).stop 2024-03-18T22:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 -- 172.21.15.40:0/3808319938 shutdown_connections 2024-03-18T22:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 --2- 172.21.15.40:0/3808319938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bacde50 0x561f9bad9d10 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 --2- 172.21.15.40:0/3808319938 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac8ce0 0x561f9bad2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 --2- 172.21.15.40:0/3808319938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bac82f0 0x561f9bad27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 -- 172.21.15.40:0/3808319938 >> 172.21.15.40:0/3808319938 conn(0x561f9bac3860 msgr2=0x561f9babea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 -- 172.21.15.40:0/3808319938 shutdown_connections 2024-03-18T22:46:25.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 -- 172.21.15.40:0/3808319938 wait complete. 2024-03-18T22:46:25.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 -- start start 2024-03-18T22:46:25.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 Processor -- start 2024-03-18T22:46:25.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac82f0 0x561f9bad7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac8ce0 0x561f9bad7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bacde50 0x561f9bad8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f57c81a9670 con 0x561f9bac82f0 2024-03-18T22:46:25.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57c81a8fa0 con 0x561f9bacde50 2024-03-18T22:46:25.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.873+0000 7f57d19c4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f57c81a61e0 con 0x561f9bac8ce0 2024-03-18T22:46:25.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57d19c4740 10 client.0 _ll_register_callbacks cb 0x7f57b4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:25.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57ccb70640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac82f0 0x561f9bad7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57ccb70640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac82f0 0x561f9bad7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44126/0 (socket says 172.21.15.40:44126) 2024-03-18T22:46:25.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57ccb70640 1 -- 172.21.15.40:0/564891965 learned_addr learned my addr 172.21.15.40:0/564891965 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:25.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57cd371640 1 --2- 172.21.15.40:0/564891965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bacde50 0x561f9bad8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57c7fff640 1 --2- 172.21.15.40:0/564891965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac8ce0 0x561f9bad7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57ccb70640 1 -- 172.21.15.40:0/564891965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac8ce0 msgr2=0x561f9bad7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57ccb70640 1 --2- 172.21.15.40:0/564891965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac8ce0 0x561f9bad7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57ccb70640 1 -- 172.21.15.40:0/564891965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bacde50 msgr2=0x561f9bad8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57ccb70640 1 --2- 172.21.15.40:0/564891965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bacde50 0x561f9bad8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57ccb70640 1 -- 172.21.15.40:0/564891965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f9ba99880 con 0x561f9bac82f0 2024-03-18T22:46:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57ccb70640 1 --2- 172.21.15.40:0/564891965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac82f0 0x561f9bad7440 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f57bc097be0 tx=0x7f57bc099a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=503131a0da9f1465 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57a7fff640 1 -- 172.21.15.40:0/564891965 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc07f060 con 0x561f9bac82f0 2024-03-18T22:46:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57a7fff640 1 -- 172.21.15.40:0/564891965 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f57bc0860b0 con 0x561f9bac82f0 2024-03-18T22:46:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57d19c4740 1 -- 172.21.15.40:0/564891965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57c0012270 con 0x561f9bac82f0 2024-03-18T22:46:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57a7fff640 1 -- 172.21.15.40:0/564891965 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f57bc0a5020 con 0x561f9bac82f0 2024-03-18T22:46:25.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57d19c4740 10 client.16464 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.874+0000 7f57d19c4740 1 -- 172.21.15.40:0/564891965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561f9bada120 con 0x561f9bac82f0 2024-03-18T22:46:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.875+0000 7f57a7fff640 1 -- 172.21.15.40:0/564891965 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57c0012270 con 0x561f9bac82f0 2024-03-18T22:46:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.875+0000 7f57a7fff640 1 -- 172.21.15.40:0/564891965 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f57bc097040 con 0x561f9bac82f0 2024-03-18T22:46:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.875+0000 7f57d19c4740 10 client.16464 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.875+0000 7f57d19c4740 10 client.16464 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:46:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.875+0000 7f57d19c4740 20 client.16464 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.875+0000 7f57d19c4740 1 --2- 172.21.15.40:0/564891965 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x561f9bb49120 0x561f9bb4b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.875+0000 7f57d19c4740 4 client.16464 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:46:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.875+0000 7f57d19c4740 1 -- 172.21.15.40:0/564891965 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561f9ba399f0 con 0x561f9bb49120 2024-03-18T22:46:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.875+0000 7f57c7fff640 1 --2- 172.21.15.40:0/564891965 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x561f9bb49120 0x561f9bb4b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.875+0000 7f57c7fff640 1 --2- 172.21.15.40:0/564891965 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x561f9bb49120 0x561f9bb4b510 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7c98868910f1084c server_cookie=e1145118756271f2 in_seq=0 out_seq=0 2024-03-18T22:46:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.875+0000 7f57a7fff640 10 client.16464 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:46:25.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.876+0000 7f57a7fff640 1 -- 172.21.15.40:0/564891965 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561f9ba399f0 con 0x561f9bb49120 2024-03-18T22:46:25.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.876+0000 7f57a7fff640 10 client.16464 handle_command_reply: tid=0 2024-03-18T22:46:25.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.876+0000 7f57d19c4740 1 client.16464 shutdown 2024-03-18T22:46:25.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.876+0000 7f57d19c4740 1 -- 172.21.15.40:0/564891965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac82f0 msgr2=0x561f9bad7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.876+0000 7f57d19c4740 1 --2- 172.21.15.40:0/564891965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac82f0 0x561f9bad7440 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f57bc097be0 tx=0x7f57bc099a80 comp rx=0 tx=0).stop 2024-03-18T22:46:25.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.876+0000 7f57d19c4740 1 -- 172.21.15.40:0/564891965 shutdown_connections 2024-03-18T22:46:25.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.876+0000 7f57d19c4740 1 --2- 172.21.15.40:0/564891965 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x561f9bb49120 0x561f9bb4b510 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.876+0000 7f57d19c4740 1 --2- 172.21.15.40:0/564891965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561f9bacde50 0x561f9bad8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.876+0000 7f57d19c4740 1 --2- 172.21.15.40:0/564891965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561f9bac8ce0 0x561f9bad7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.876+0000 7f57d19c4740 1 --2- 172.21.15.40:0/564891965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561f9bac82f0 0x561f9bad7440 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.876+0000 7f57d19c4740 1 -- 172.21.15.40:0/564891965 >> 172.21.15.40:0/564891965 conn(0x561f9bac3860 msgr2=0x561f9bad2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.876+0000 7f57a7fff640 0 client.16464 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:46:25.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.876+0000 7f57d19c4740 1 -- 172.21.15.40:0/564891965 shutdown_connections 2024-03-18T22:46:25.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.876+0000 7f57d19c4740 1 -- 172.21.15.40:0/564891965 wait complete. 2024-03-18T22:46:25.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.876+0000 7f57d19c4740 20 client.16464 trim_cache size 0 max 16384 2024-03-18T22:46:25.884 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:46:25.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.877+0000 7f57cedd4640 1 -- 172.21.15.40:0/515456026 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57b80a2800 msgr2=0x7f57b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.877+0000 7f57cedd4640 1 --2- 172.21.15.40:0/515456026 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57b80a2800 0x7f57b80a4cc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009230 tx=0x7f57b4005fd0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.877+0000 7f57cedd4640 1 -- 172.21.15.40:0/515456026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57c8100730 msgr2=0x7f57c806a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:25.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.877+0000 7f57cedd4640 1 --2- 172.21.15.40:0/515456026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57c8100730 0x7f57c806a980 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00e9c0 tx=0x7f57bc00ee80 comp rx=0 tx=0).stop 2024-03-18T22:46:25.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.877+0000 7f57cedd4640 1 -- 172.21.15.40:0/515456026 shutdown_connections 2024-03-18T22:46:25.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.877+0000 7f57cedd4640 1 --2- 172.21.15.40:0/515456026 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57b80a2800 0x7f57b80a4cc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.877+0000 7f57cedd4640 1 --2- 172.21.15.40:0/515456026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57c81022e0 0x7f57c819ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.877+0000 7f57cedd4640 1 --2- 172.21.15.40:0/515456026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57c8101920 0x7f57c819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.877+0000 7f57cedd4640 1 --2- 172.21.15.40:0/515456026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57c8100730 0x7f57c806a980 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.877+0000 7f57cedd4640 1 -- 172.21.15.40:0/515456026 >> 172.21.15.40:0/515456026 conn(0x7f57c80fbca0 msgr2=0x7f57c8104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.877+0000 7f57cedd4640 1 -- 172.21.15.40:0/515456026 shutdown_connections 2024-03-18T22:46:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:25.877+0000 7f57cedd4640 1 -- 172.21.15.40:0/515456026 wait complete. 2024-03-18T22:46:25.889 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:46:26.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.013+0000 7fc6c35f4640 1 -- 172.21.15.40:0/1373818572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6bc100720 msgr2=0x7fc6bc100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.013+0000 7fc6c35f4640 1 --2- 172.21.15.40:0/1373818572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6bc100720 0x7fc6bc100b40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8009860 tx=0x7fc6b802bb70 comp rx=0 tx=0).stop 2024-03-18T22:46:26.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.013+0000 7fc6c35f4640 1 -- 172.21.15.40:0/1373818572 shutdown_connections 2024-03-18T22:46:26.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.013+0000 7fc6c35f4640 1 --2- 172.21.15.40:0/1373818572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6bc1022d0 0x7fc6bc108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.013+0000 7fc6c35f4640 1 --2- 172.21.15.40:0/1373818572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6bc101910 0x7fc6bc101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.013+0000 7fc6c35f4640 1 --2- 172.21.15.40:0/1373818572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6bc100720 0x7fc6bc100b40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.013+0000 7fc6c35f4640 1 -- 172.21.15.40:0/1373818572 >> 172.21.15.40:0/1373818572 conn(0x7fc6bc0fbc90 msgr2=0x7fc6bc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.014+0000 7fc6c35f4640 1 -- 172.21.15.40:0/1373818572 shutdown_connections 2024-03-18T22:46:26.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.014+0000 7fc6c35f4640 1 -- 172.21.15.40:0/1373818572 wait complete. 2024-03-18T22:46:26.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.014+0000 7fc6c35f4640 1 Processor -- start 2024-03-18T22:46:26.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.014+0000 7fc6c35f4640 1 -- start start 2024-03-18T22:46:26.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.014+0000 7fc6c35f4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6bc100720 0x7fc6bc0694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.014+0000 7fc6c35f4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6bc101910 0x7fc6bc19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.014+0000 7fc6c35f4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6bc1022d0 0x7fc6bc19eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.014+0000 7fc6c35f4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc10c250 con 0x7fc6bc100720 2024-03-18T22:46:26.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.014+0000 7fc6c35f4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc10c0d0 con 0x7fc6bc101910 2024-03-18T22:46:26.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.014+0000 7fc6c35f4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc10c3d0 con 0x7fc6bc1022d0 2024-03-18T22:46:26.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.014+0000 7fc6c1390640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6bc100720 0x7fc6bc0694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.014+0000 7fc6c1390640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6bc100720 0x7fc6bc0694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44150/0 (socket says 172.21.15.40:44150) 2024-03-18T22:46:26.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.014+0000 7fc6c1390640 1 -- 172.21.15.40:0/524924074 learned_addr learned my addr 172.21.15.40:0/524924074 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:26.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.014+0000 7fc6c0b8f640 1 --2- 172.21.15.40:0/524924074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6bc101910 0x7fc6bc19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.015+0000 7fc6c1b91640 1 --2- 172.21.15.40:0/524924074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6bc1022d0 0x7fc6bc19eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.015+0000 7fc6c1390640 1 -- 172.21.15.40:0/524924074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6bc1022d0 msgr2=0x7fc6bc19eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.015+0000 7fc6c1390640 1 --2- 172.21.15.40:0/524924074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6bc1022d0 0x7fc6bc19eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.015+0000 7fc6c1390640 1 -- 172.21.15.40:0/524924074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6bc101910 msgr2=0x7fc6bc19c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.015+0000 7fc6c1390640 1 --2- 172.21.15.40:0/524924074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6bc101910 0x7fc6bc19c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.015+0000 7fc6c1390640 1 -- 172.21.15.40:0/524924074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6bc19f6b0 con 0x7fc6bc100720 2024-03-18T22:46:26.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.015+0000 7fc6c1390640 1 --2- 172.21.15.40:0/524924074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6bc100720 0x7fc6bc0694f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc6b80057c0 tx=0x7fc6b80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.015+0000 7fc6b27fc640 1 -- 172.21.15.40:0/524924074 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8041030 con 0x7fc6bc100720 2024-03-18T22:46:26.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.015+0000 7fc6c35f4640 1 -- 172.21.15.40:0/524924074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6bc19f8e0 con 0x7fc6bc100720 2024-03-18T22:46:26.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.015+0000 7fc6b27fc640 1 -- 172.21.15.40:0/524924074 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6b802c9c0 con 0x7fc6bc100720 2024-03-18T22:46:26.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.015+0000 7fc6b27fc640 1 -- 172.21.15.40:0/524924074 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8039040 con 0x7fc6bc100720 2024-03-18T22:46:26.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.015+0000 7fc6c35f4640 1 -- 172.21.15.40:0/524924074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6bc19fdd0 con 0x7fc6bc100720 2024-03-18T22:46:26.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.018+0000 7fc6b27fc640 1 -- 172.21.15.40:0/524924074 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc6b8044070 con 0x7fc6bc100720 2024-03-18T22:46:26.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.018+0000 7fc6c61e7740 1 Processor -- start 2024-03-18T22:46:26.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.018+0000 7fc6c61e7740 1 -- start start 2024-03-18T22:46:26.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c61e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97a700 0x55c2de97ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c61e7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97b0f0 0x55c2de97fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c61e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de980260 0x55c2de9826f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c61e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c2de8da4b0 con 0x55c2de980260 2024-03-18T22:46:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c61e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c2de882d70 con 0x55c2de97a700 2024-03-18T22:46:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c61e7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c2de8d6550 con 0x55c2de97b0f0 2024-03-18T22:46:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c1b91640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97a700 0x55c2de97ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c1390640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97b0f0 0x55c2de97fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c1b91640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97a700 0x55c2de97ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47392/0 (socket says 172.21.15.40:47392) 2024-03-18T22:46:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c1b91640 1 -- 172.21.15.40:0/1432086426 learned_addr learned my addr 172.21.15.40:0/1432086426 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6b27fc640 1 --2- 172.21.15.40:0/524924074 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc6a80a2800 0x7fc6a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c0b8f640 1 --2- 172.21.15.40:0/1432086426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de980260 0x55c2de9826f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c1b91640 1 --2- 172.21.15.40:0/524924074 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc6a80a2800 0x7fc6a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6b27fc640 1 -- 172.21.15.40:0/524924074 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc6b80f0370 con 0x7fc6bc100720 2024-03-18T22:46:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c1b91640 1 -- 172.21.15.40:0/1432086426 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97b0f0 msgr2=0x55c2de97fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c1b91640 1 --2- 172.21.15.40:0/1432086426 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97b0f0 0x55c2de97fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c1b91640 1 -- 172.21.15.40:0/1432086426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de980260 msgr2=0x55c2de9826f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c1b91640 1 --2- 172.21.15.40:0/1432086426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de980260 0x55c2de9826f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c1b91640 1 -- 172.21.15.40:0/1432086426 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2de8a5210 con 0x55c2de97a700 2024-03-18T22:46:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c0b8f640 1 --2- 172.21.15.40:0/1432086426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de980260 0x55c2de9826f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.019+0000 7fc6c1b91640 1 --2- 172.21.15.40:0/1432086426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97a700 0x55c2de97ab20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4005c40 tx=0x7fc6b400f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfc97c2ccacf75e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.020+0000 7fc6c1b91640 1 --2- 172.21.15.40:0/524924074 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc6a80a2800 0x7fc6a80a4cc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4008000 tx=0x7fc6b401f000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.020+0000 7fc6a77fe640 1 -- 172.21.15.40:0/1432086426 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b400fd30 con 0x55c2de97a700 2024-03-18T22:46:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.020+0000 7fc6a77fe640 1 -- 172.21.15.40:0/1432086426 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6b4022030 con 0x55c2de97a700 2024-03-18T22:46:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.020+0000 7fc6a77fe640 1 -- 172.21.15.40:0/1432086426 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4010840 con 0x55c2de97a700 2024-03-18T22:46:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.020+0000 7fc6c61e7740 1 -- 172.21.15.40:0/1432086426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97a700 msgr2=0x55c2de97ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.020+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/1432086426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97a700 0x55c2de97ab20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4005c40 tx=0x7fc6b400f3c0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.020+0000 7fc6c61e7740 1 -- 172.21.15.40:0/1432086426 shutdown_connections 2024-03-18T22:46:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.020+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/1432086426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de980260 0x55c2de9826f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.020+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/1432086426 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97b0f0 0x55c2de97fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.020+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/1432086426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97a700 0x55c2de97ab20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.020+0000 7fc6c61e7740 1 -- 172.21.15.40:0/1432086426 >> 172.21.15.40:0/1432086426 conn(0x55c2de971860 msgr2=0x55c2de971c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.020+0000 7fc6c61e7740 1 -- 172.21.15.40:0/1432086426 shutdown_connections 2024-03-18T22:46:26.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c61e7740 1 -- 172.21.15.40:0/1432086426 wait complete. 2024-03-18T22:46:26.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c61e7740 1 -- start start 2024-03-18T22:46:26.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c61e7740 1 Processor -- start 2024-03-18T22:46:26.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c61e7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97a700 0x55c2de984c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c61e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97b0f0 0x55c2de985180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c61e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de980260 0x55c2de98c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c61e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c2de9831b0 con 0x55c2de980260 2024-03-18T22:46:26.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c61e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c2de896310 con 0x55c2de97b0f0 2024-03-18T22:46:26.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c61e7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c2de983330 con 0x55c2de97a700 2024-03-18T22:46:26.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c61e7740 10 client.0 _ll_register_callbacks cb 0x55c2de88a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:26.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c1b91640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de980260 0x55c2de98c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c0b8f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97b0f0 0x55c2de985180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c1390640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97a700 0x55c2de984c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c0b8f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97b0f0 0x55c2de985180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47400/0 (socket says 172.21.15.40:47400) 2024-03-18T22:46:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c0b8f640 1 -- 172.21.15.40:0/258947392 learned_addr learned my addr 172.21.15.40:0/258947392 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c1390640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97a700 0x55c2de984c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55836/0 (socket says 172.21.15.40:55836) 2024-03-18T22:46:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.021+0000 7fc6c1b91640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de980260 0x55c2de98c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44168/0 (socket says 172.21.15.40:44168) 2024-03-18T22:46:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.022+0000 7fc6c0b8f640 1 -- 172.21.15.40:0/258947392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97a700 msgr2=0x55c2de984c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.022+0000 7fc6c0b8f640 1 --2- 172.21.15.40:0/258947392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97a700 0x55c2de984c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.022+0000 7fc6c0b8f640 1 -- 172.21.15.40:0/258947392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de980260 msgr2=0x55c2de98c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.022+0000 7fc6c0b8f640 1 --2- 172.21.15.40:0/258947392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de980260 0x55c2de98c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.022+0000 7fc6c0b8f640 1 -- 172.21.15.40:0/258947392 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2de5ee3f0 con 0x55c2de97b0f0 2024-03-18T22:46:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.022+0000 7fc6c1390640 1 --2- 172.21.15.40:0/258947392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97a700 0x55c2de984c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.022+0000 7fc6c1b91640 1 --2- 172.21.15.40:0/258947392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de980260 0x55c2de98c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.022+0000 7fc6c0b8f640 1 --2- 172.21.15.40:0/258947392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97b0f0 0x55c2de985180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac011760 tx=0x7fc6ac011b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1ee182845337f81 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.022+0000 7fc6a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:46:26.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.022+0000 7fc6a4ff9640 1 -- 172.21.15.40:0/258947392 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac004300 con 0x55c2de97b0f0 2024-03-18T22:46:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.022+0000 7fc6a4ff9640 1 -- 172.21.15.40:0/258947392 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc6ac00f8c0 con 0x55c2de97b0f0 2024-03-18T22:46:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.022+0000 7fc6c61e7740 1 -- 172.21.15.40:0/258947392 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6b4022030 con 0x55c2de97b0f0 2024-03-18T22:46:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.022+0000 7fc6a4ff9640 1 -- 172.21.15.40:0/258947392 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc6ac025050 con 0x55c2de97b0f0 2024-03-18T22:46:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.022+0000 7fc6c61e7740 10 client.26309 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.022+0000 7fc6c61e7740 1 -- 172.21.15.40:0/258947392 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c2de6b87d0 con 0x55c2de97b0f0 2024-03-18T22:46:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.023+0000 7fc6a4ff9640 1 -- 172.21.15.40:0/258947392 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6b4022030 con 0x55c2de97b0f0 2024-03-18T22:46:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.023+0000 7fc6a4ff9640 1 -- 172.21.15.40:0/258947392 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fc6ac019030 con 0x55c2de97b0f0 2024-03-18T22:46:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.023+0000 7fc6c61e7740 10 client.26309 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.023+0000 7fc6c61e7740 10 client.26309 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:46:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.023+0000 7fc6c61e7740 20 client.26309 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.023+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/258947392 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c2de990b00 0x55c2de992f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.023+0000 7fc6c61e7740 4 client.26309 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:46:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.023+0000 7fc6c61e7740 1 -- 172.21.15.40:0/258947392 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2de9383e0 con 0x55c2de990b00 2024-03-18T22:46:26.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.023+0000 7fc6c1390640 1 --2- 172.21.15.40:0/258947392 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c2de990b00 0x55c2de992f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.023+0000 7fc6c1390640 1 --2- 172.21.15.40:0/258947392 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c2de990b00 0x55c2de992f40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2cae3d2813a9d6b1 server_cookie=a0f972f62df4e34 in_seq=0 out_seq=0 2024-03-18T22:46:26.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.023+0000 7fc6a4ff9640 10 client.26309 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:46:26.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.024+0000 7fc6a4ff9640 1 -- 172.21.15.40:0/258947392 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c2de9383e0 con 0x55c2de990b00 2024-03-18T22:46:26.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.024+0000 7fc6a4ff9640 10 client.26309 handle_command_reply: tid=0 2024-03-18T22:46:26.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.024+0000 7fc6c61e7740 1 client.26309 shutdown 2024-03-18T22:46:26.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.024+0000 7fc6c61e7740 1 -- 172.21.15.40:0/258947392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97b0f0 msgr2=0x55c2de985180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.024+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/258947392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97b0f0 0x55c2de985180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac011760 tx=0x7fc6ac011b60 comp rx=0 tx=0).stop 2024-03-18T22:46:26.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.024+0000 7fc6c61e7740 1 -- 172.21.15.40:0/258947392 shutdown_connections 2024-03-18T22:46:26.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.024+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/258947392 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c2de990b00 0x55c2de992f40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.024+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/258947392 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de980260 0x55c2de98c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.024+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/258947392 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97b0f0 0x55c2de985180 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.024+0000 7fc6a4ff9640 0 client.26309 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:46:26.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.024+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/258947392 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97a700 0x55c2de984c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.024+0000 7fc6c61e7740 1 -- 172.21.15.40:0/258947392 >> 172.21.15.40:0/258947392 conn(0x55c2de971860 msgr2=0x55c2de970ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.024+0000 7fc6c61e7740 1 -- 172.21.15.40:0/258947392 shutdown_connections 2024-03-18T22:46:26.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.024+0000 7fc6c61e7740 1 -- 172.21.15.40:0/258947392 wait complete. 2024-03-18T22:46:26.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.024+0000 7fc6c61e7740 20 client.26309 trim_cache size 0 max 16384 2024-03-18T22:46:26.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.038+0000 7fc6c61e7740 1 Processor -- start 2024-03-18T22:46:26.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.038+0000 7fc6c61e7740 1 -- start start 2024-03-18T22:46:26.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.038+0000 7fc6c61e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de97a700 0x55c2de96ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.038+0000 7fc6c61e7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97b0f0 0x55c2de98c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.038+0000 7fc6c61e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de980260 0x55c2de971c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.038+0000 7fc6c61e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc1a8310 con 0x55c2de97a700 2024-03-18T22:46:26.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6c61e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc1a9e80 con 0x55c2de980260 2024-03-18T22:46:26.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6c61e7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc1a89e0 con 0x55c2de97b0f0 2024-03-18T22:46:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6c1390640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de97a700 0x55c2de96ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6c1390640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de97a700 0x55c2de96ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44180/0 (socket says 172.21.15.40:44180) 2024-03-18T22:46:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6c1390640 1 -- 172.21.15.40:0/16471956 learned_addr learned my addr 172.21.15.40:0/16471956 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6c0b8f640 1 --2- 172.21.15.40:0/16471956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97b0f0 0x55c2de98c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6c1b91640 1 --2- 172.21.15.40:0/16471956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de980260 0x55c2de971c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6c1390640 1 -- 172.21.15.40:0/16471956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97b0f0 msgr2=0x55c2de98c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6c1390640 1 --2- 172.21.15.40:0/16471956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97b0f0 0x55c2de98c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6c1390640 1 -- 172.21.15.40:0/16471956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de980260 msgr2=0x55c2de971c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6c1390640 1 --2- 172.21.15.40:0/16471956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de980260 0x55c2de971c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6c1390640 1 -- 172.21.15.40:0/16471956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2de947880 con 0x55c2de97a700 2024-03-18T22:46:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6c0b8f640 1 --2- 172.21.15.40:0/16471956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97b0f0 0x55c2de98c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6c1b91640 1 --2- 172.21.15.40:0/16471956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de980260 0x55c2de971c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6c1390640 1 --2- 172.21.15.40:0/16471956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de97a700 0x55c2de96ca30 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fc6b80eddf0 tx=0x7fc6b80ed950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf333ea91e21888 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6a6ffd640 1 -- 172.21.15.40:0/16471956 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b80bc280 con 0x55c2de97a700 2024-03-18T22:46:26.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6a6ffd640 1 -- 172.21.15.40:0/16471956 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6b80bd020 con 0x55c2de97a700 2024-03-18T22:46:26.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.039+0000 7fc6a6ffd640 1 -- 172.21.15.40:0/16471956 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b80f0160 con 0x55c2de97a700 2024-03-18T22:46:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.040+0000 7fc6c61e7740 1 -- 172.21.15.40:0/16471956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de97a700 msgr2=0x55c2de96ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.040+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/16471956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de97a700 0x55c2de96ca30 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fc6b80eddf0 tx=0x7fc6b80ed950 comp rx=0 tx=0).stop 2024-03-18T22:46:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.040+0000 7fc6c61e7740 1 -- 172.21.15.40:0/16471956 shutdown_connections 2024-03-18T22:46:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.040+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/16471956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de980260 0x55c2de971c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.040+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/16471956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de97b0f0 0x55c2de98c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.040+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/16471956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de97a700 0x55c2de96ca30 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.040+0000 7fc6c61e7740 1 -- 172.21.15.40:0/16471956 >> 172.21.15.40:0/16471956 conn(0x55c2de971860 msgr2=0x55c2de97bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.040+0000 7fc6c61e7740 1 -- 172.21.15.40:0/16471956 shutdown_connections 2024-03-18T22:46:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.040+0000 7fc6c61e7740 1 -- 172.21.15.40:0/16471956 wait complete. 2024-03-18T22:46:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.041+0000 7fc6c61e7740 1 -- start start 2024-03-18T22:46:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.041+0000 7fc6c61e7740 1 Processor -- start 2024-03-18T22:46:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.041+0000 7fc6c61e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de97a700 0x55c2de9890e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.041+0000 7fc6c61e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97b0f0 0x55c2de989620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.041+0000 7fc6c61e7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de980260 0x55c2de9f4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.041+0000 7fc6c61e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc1a9780 con 0x55c2de97a700 2024-03-18T22:46:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.041+0000 7fc6c61e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc1a90b0 con 0x55c2de97b0f0 2024-03-18T22:46:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.041+0000 7fc6c61e7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc1a62f0 con 0x55c2de980260 2024-03-18T22:46:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.041+0000 7fc6c61e7740 10 client.0 _ll_register_callbacks cb 0x7fc6ac009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.041+0000 7fc6c1390640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de97a700 0x55c2de9890e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.041+0000 7fc6c1390640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de97a700 0x55c2de9890e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44192/0 (socket says 172.21.15.40:44192) 2024-03-18T22:46:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.041+0000 7fc6c1390640 1 -- 172.21.15.40:0/138532442 learned_addr learned my addr 172.21.15.40:0/138532442 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:26.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.041+0000 7fc6c0b8f640 1 --2- 172.21.15.40:0/138532442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97b0f0 0x55c2de989620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.041+0000 7fc6c1b91640 1 --2- 172.21.15.40:0/138532442 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de980260 0x55c2de9f4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.042+0000 7fc6c1390640 1 -- 172.21.15.40:0/138532442 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de980260 msgr2=0x55c2de9f4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.042+0000 7fc6c1390640 1 --2- 172.21.15.40:0/138532442 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de980260 0x55c2de9f4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.042+0000 7fc6c1390640 1 -- 172.21.15.40:0/138532442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97b0f0 msgr2=0x55c2de989620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.042+0000 7fc6c1390640 1 --2- 172.21.15.40:0/138532442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97b0f0 0x55c2de989620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.042+0000 7fc6c1390640 1 -- 172.21.15.40:0/138532442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2de9f53b0 con 0x55c2de97a700 2024-03-18T22:46:26.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.042+0000 7fc6c0b8f640 1 --2- 172.21.15.40:0/138532442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97b0f0 0x55c2de989620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.042+0000 7fc6c1390640 1 --2- 172.21.15.40:0/138532442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de97a700 0x55c2de9890e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fc6b80bc8b0 tx=0x7fc6b8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5411ff93833b854 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.042+0000 7fc6a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:26.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.042+0000 7fc6a4ff9640 1 -- 172.21.15.40:0/138532442 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b80c50c0 con 0x55c2de97a700 2024-03-18T22:46:26.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.042+0000 7fc6a4ff9640 1 -- 172.21.15.40:0/138532442 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc6b80ba9c0 con 0x55c2de97a700 2024-03-18T22:46:26.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.042+0000 7fc6c61e7740 1 -- 172.21.15.40:0/138532442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6b80bd020 con 0x55c2de97a700 2024-03-18T22:46:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.042+0000 7fc6a4ff9640 1 -- 172.21.15.40:0/138532442 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc6b80bc920 con 0x55c2de97a700 2024-03-18T22:46:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.042+0000 7fc6c61e7740 10 client.16482 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.042+0000 7fc6c61e7740 1 -- 172.21.15.40:0/138532442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c2de9f6660 con 0x55c2de97a700 2024-03-18T22:46:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.042+0000 7fc6a4ff9640 1 -- 172.21.15.40:0/138532442 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6b80bd020 con 0x55c2de97a700 2024-03-18T22:46:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.043+0000 7fc6a4ff9640 1 -- 172.21.15.40:0/138532442 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fc6b804e060 con 0x55c2de97a700 2024-03-18T22:46:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.043+0000 7fc6c61e7740 10 client.16482 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.043+0000 7fc6c61e7740 10 client.16482 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:46:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.043+0000 7fc6c61e7740 20 client.16482 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.043+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/138532442 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c2de9f6e50 0x55c2de9f9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.043+0000 7fc6c61e7740 4 client.16482 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:46:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.043+0000 7fc6c61e7740 1 -- 172.21.15.40:0/138532442 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c2de8e79f0 con 0x55c2de9f6e50 2024-03-18T22:46:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.043+0000 7fc6c0b8f640 1 --2- 172.21.15.40:0/138532442 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c2de9f6e50 0x55c2de9f9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.043+0000 7fc6c0b8f640 1 --2- 172.21.15.40:0/138532442 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c2de9f6e50 0x55c2de9f9270 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=93c2619bae252c8e server_cookie=9bb588ef678e8efe in_seq=0 out_seq=0 2024-03-18T22:46:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.043+0000 7fc6a4ff9640 10 client.16482 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:46:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.043+0000 7fc6a4ff9640 1 -- 172.21.15.40:0/138532442 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c2de8e79f0 con 0x55c2de9f6e50 2024-03-18T22:46:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.043+0000 7fc6a4ff9640 10 client.16482 handle_command_reply: tid=0 2024-03-18T22:46:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.043+0000 7fc6c61e7740 1 client.16482 shutdown 2024-03-18T22:46:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c61e7740 1 -- 172.21.15.40:0/138532442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de97a700 msgr2=0x55c2de9890e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/138532442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de97a700 0x55c2de9890e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fc6b80bc8b0 tx=0x7fc6b8005f90 comp rx=0 tx=0).stop 2024-03-18T22:46:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c61e7740 1 -- 172.21.15.40:0/138532442 shutdown_connections 2024-03-18T22:46:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/138532442 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c2de9f6e50 0x55c2de9f9270 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/138532442 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c2de980260 0x55c2de9f4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6a4ff9640 0 client.16482 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:46:26.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/138532442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c2de97b0f0 0x55c2de989620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c61e7740 1 --2- 172.21.15.40:0/138532442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c2de97a700 0x55c2de9890e0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c61e7740 1 -- 172.21.15.40:0/138532442 >> 172.21.15.40:0/138532442 conn(0x55c2de971860 msgr2=0x55c2de97c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c61e7740 1 -- 172.21.15.40:0/138532442 shutdown_connections 2024-03-18T22:46:26.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c61e7740 1 -- 172.21.15.40:0/138532442 wait complete. 2024-03-18T22:46:26.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c61e7740 20 client.16482 trim_cache size 0 max 16384 2024-03-18T22:46:26.051 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:46:26.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c35f4640 1 -- 172.21.15.40:0/524924074 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc6a80a2800 msgr2=0x7fc6a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c35f4640 1 --2- 172.21.15.40:0/524924074 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc6a80a2800 0x7fc6a80a4cc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4008000 tx=0x7fc6b401f000 comp rx=0 tx=0).stop 2024-03-18T22:46:26.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c35f4640 1 -- 172.21.15.40:0/524924074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6bc100720 msgr2=0x7fc6bc0694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c35f4640 1 --2- 172.21.15.40:0/524924074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6bc100720 0x7fc6bc0694f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc6b80057c0 tx=0x7fc6b80042f0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c35f4640 1 -- 172.21.15.40:0/524924074 shutdown_connections 2024-03-18T22:46:26.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c35f4640 1 --2- 172.21.15.40:0/524924074 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc6a80a2800 0x7fc6a80a4cc0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c35f4640 1 --2- 172.21.15.40:0/524924074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc6bc1022d0 0x7fc6bc19eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c35f4640 1 --2- 172.21.15.40:0/524924074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6bc101910 0x7fc6bc19c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c35f4640 1 --2- 172.21.15.40:0/524924074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6bc100720 0x7fc6bc0694f0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.044+0000 7fc6c35f4640 1 -- 172.21.15.40:0/524924074 >> 172.21.15.40:0/524924074 conn(0x7fc6bc0fbc90 msgr2=0x7fc6bc104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.045+0000 7fc6c35f4640 1 -- 172.21.15.40:0/524924074 shutdown_connections 2024-03-18T22:46:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.045+0000 7fc6c35f4640 1 -- 172.21.15.40:0/524924074 wait complete. 2024-03-18T22:46:26.057 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:46:26.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:26 smithi040 ceph-mon[62643]: pgmap v464: 129 pgs: 129 active+clean; 3.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:46:26.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:26 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3113350217' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:46:26.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.170+0000 7f722fbe8640 1 -- 172.21.15.40:0/1040409864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72281055f0 msgr2=0x7f722810be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.170+0000 7f722fbe8640 1 --2- 172.21.15.40:0/1040409864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72281055f0 0x7f722810be80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7220009ca0 tx=0x7f722002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.170+0000 7f722fbe8640 1 -- 172.21.15.40:0/1040409864 shutdown_connections 2024-03-18T22:46:26.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.170+0000 7f722fbe8640 1 --2- 172.21.15.40:0/1040409864 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72281055f0 0x7f722810be80 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.170+0000 7f722fbe8640 1 --2- 172.21.15.40:0/1040409864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7228104c30 0x7f72281050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.170+0000 7f722fbe8640 1 --2- 172.21.15.40:0/1040409864 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7228103a40 0x7f7228103e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.170+0000 7f722fbe8640 1 -- 172.21.15.40:0/1040409864 >> 172.21.15.40:0/1040409864 conn(0x7f72280fefd0 msgr2=0x7f72281013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.170+0000 7f722fbe8640 1 -- 172.21.15.40:0/1040409864 shutdown_connections 2024-03-18T22:46:26.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.170+0000 7f722fbe8640 1 -- 172.21.15.40:0/1040409864 wait complete. 2024-03-18T22:46:26.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.170+0000 7f722fbe8640 1 Processor -- start 2024-03-18T22:46:26.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.170+0000 7f722fbe8640 1 -- start start 2024-03-18T22:46:26.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722fbe8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7228103a40 0x7f7228199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722fbe8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7228104c30 0x7f722819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722fbe8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72281055f0 0x7f72281a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722fbe8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f722810f550 con 0x7f7228103a40 2024-03-18T22:46:26.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722fbe8640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f722810f3d0 con 0x7f7228104c30 2024-03-18T22:46:26.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722fbe8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f722810f6d0 con 0x7f72281055f0 2024-03-18T22:46:26.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722d984640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7228103a40 0x7f7228199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722d984640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7228103a40 0x7f7228199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44202/0 (socket says 172.21.15.40:44202) 2024-03-18T22:46:26.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722d984640 1 -- 172.21.15.40:0/2605699648 learned_addr learned my addr 172.21.15.40:0/2605699648 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:26.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722d183640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7228104c30 0x7f722819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722e185640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72281055f0 0x7f72281a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722d984640 1 -- 172.21.15.40:0/2605699648 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72281055f0 msgr2=0x7f72281a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722d984640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72281055f0 0x7f72281a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722d984640 1 -- 172.21.15.40:0/2605699648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7228104c30 msgr2=0x7f722819a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722d984640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7228104c30 0x7f722819a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722d984640 1 -- 172.21.15.40:0/2605699648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72281a17f0 con 0x7f7228103a40 2024-03-18T22:46:26.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722e185640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72281055f0 0x7f72281a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722d183640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7228104c30 0x7f722819a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.171+0000 7f722d984640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7228103a40 0x7f7228199bd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f722400e9c0 tx=0x7f722400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.172+0000 7f721effd640 1 -- 172.21.15.40:0/2605699648 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722400cd80 con 0x7f7228103a40 2024-03-18T22:46:26.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.172+0000 7f722fbe8640 1 -- 172.21.15.40:0/2605699648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72281a1ae0 con 0x7f7228103a40 2024-03-18T22:46:26.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.172+0000 7f721effd640 1 -- 172.21.15.40:0/2605699648 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72240035f0 con 0x7f7228103a40 2024-03-18T22:46:26.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.172+0000 7f721effd640 1 -- 172.21.15.40:0/2605699648 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7224010690 con 0x7f7228103a40 2024-03-18T22:46:26.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.172+0000 7f722fbe8640 1 -- 172.21.15.40:0/2605699648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72281a2080 con 0x7f7228103a40 2024-03-18T22:46:26.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.174+0000 7f721effd640 1 -- 172.21.15.40:0/2605699648 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f72240037a0 con 0x7f7228103a40 2024-03-18T22:46:26.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.174+0000 7f72327cb740 1 Processor -- start 2024-03-18T22:46:26.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f72327cb740 1 -- start start 2024-03-18T22:46:26.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f72327cb740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804b2f0 0x557ff804b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f72327cb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff804bce0 0x557ff80507e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f72327cb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff8050e50 0x557ff80532e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f72327cb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557ff7faf4b0 con 0x557ff8050e50 2024-03-18T22:46:26.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f72327cb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557ff7f57d70 con 0x557ff804bce0 2024-03-18T22:46:26.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f72327cb740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557ff7fab550 con 0x557ff804b2f0 2024-03-18T22:46:26.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f722d984640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff804bce0 0x557ff80507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f722e185640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804b2f0 0x557ff804b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f722d984640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff804bce0 0x557ff80507e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47430/0 (socket says 172.21.15.40:47430) 2024-03-18T22:46:26.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f722e185640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804b2f0 0x557ff804b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55874/0 (socket says 172.21.15.40:55874) 2024-03-18T22:46:26.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f722d984640 1 -- 172.21.15.40:0/3108470439 learned_addr learned my addr 172.21.15.40:0/3108470439 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:26.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f722d183640 1 --2- 172.21.15.40:0/3108470439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff8050e50 0x557ff80532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f721effd640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72140a2800 0x7f72140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f722d984640 1 -- 172.21.15.40:0/3108470439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804b2f0 msgr2=0x557ff804b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f722d984640 1 --2- 172.21.15.40:0/3108470439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804b2f0 0x557ff804b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f722d984640 1 -- 172.21.15.40:0/3108470439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff8050e50 msgr2=0x557ff80532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f722d984640 1 --2- 172.21.15.40:0/3108470439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff8050e50 0x557ff80532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f722d984640 1 -- 172.21.15.40:0/3108470439 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ff7f7a210 con 0x557ff804bce0 2024-03-18T22:46:26.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.175+0000 7f722e185640 1 --2- 172.21.15.40:0/3108470439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804b2f0 0x557ff804b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.176+0000 7f722e185640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72140a2800 0x7f72140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.176+0000 7f721effd640 1 -- 172.21.15.40:0/2605699648 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f72240cf1e0 con 0x7f7228103a40 2024-03-18T22:46:26.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.176+0000 7f722d984640 1 --2- 172.21.15.40:0/3108470439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff804bce0 0x557ff80507e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f722401ca80 tx=0x7f7224014f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b432834fefceab27 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.176+0000 7f7213fff640 1 -- 172.21.15.40:0/3108470439 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722409b560 con 0x557ff804bce0 2024-03-18T22:46:26.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.176+0000 7f722e185640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72140a2800 0x7f72140a4cc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7220006010 tx=0x7f722002e040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.176+0000 7f7213fff640 1 -- 172.21.15.40:0/3108470439 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f722407f0d0 con 0x557ff804bce0 2024-03-18T22:46:26.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.176+0000 7f7213fff640 1 -- 172.21.15.40:0/3108470439 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722409b0f0 con 0x557ff804bce0 2024-03-18T22:46:26.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.177+0000 7f72327cb740 1 -- 172.21.15.40:0/3108470439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff804bce0 msgr2=0x557ff80507e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.177+0000 7f72327cb740 1 --2- 172.21.15.40:0/3108470439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff804bce0 0x557ff80507e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f722401ca80 tx=0x7f7224014f40 comp rx=0 tx=0).stop 2024-03-18T22:46:26.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.177+0000 7f72327cb740 1 -- 172.21.15.40:0/3108470439 shutdown_connections 2024-03-18T22:46:26.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.177+0000 7f72327cb740 1 --2- 172.21.15.40:0/3108470439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff8050e50 0x557ff80532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.177+0000 7f72327cb740 1 --2- 172.21.15.40:0/3108470439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff804bce0 0x557ff80507e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.177+0000 7f72327cb740 1 --2- 172.21.15.40:0/3108470439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804b2f0 0x557ff804b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.177+0000 7f72327cb740 1 -- 172.21.15.40:0/3108470439 >> 172.21.15.40:0/3108470439 conn(0x557ff8046860 msgr2=0x557ff8046c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.177+0000 7f72327cb740 1 -- 172.21.15.40:0/3108470439 shutdown_connections 2024-03-18T22:46:26.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.177+0000 7f72327cb740 1 -- 172.21.15.40:0/3108470439 wait complete. 2024-03-18T22:46:26.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.177+0000 7f72327cb740 1 -- start start 2024-03-18T22:46:26.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.177+0000 7f72327cb740 1 Processor -- start 2024-03-18T22:46:26.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f72327cb740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804b2f0 0x557ff8055660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f72327cb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff804bce0 0x557ff8055ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f72327cb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff8050e50 0x557ff805cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f72327cb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557ff8053d40 con 0x557ff8050e50 2024-03-18T22:46:26.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f72327cb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557ff7f6b310 con 0x557ff804bce0 2024-03-18T22:46:26.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f72327cb740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557ff8053ec0 con 0x557ff804b2f0 2024-03-18T22:46:26.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f72327cb740 10 client.0 _ll_register_callbacks cb 0x557ff7f5f2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:26.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f722e185640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff8050e50 0x557ff805cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f722d984640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804b2f0 0x557ff8055660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f722d984640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804b2f0 0x557ff8055660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55876/0 (socket says 172.21.15.40:55876) 2024-03-18T22:46:26.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f722d183640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff804bce0 0x557ff8055ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f722e185640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff8050e50 0x557ff805cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44230/0 (socket says 172.21.15.40:44230) 2024-03-18T22:46:26.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f722e185640 1 -- 172.21.15.40:0/1131868875 learned_addr learned my addr 172.21.15.40:0/1131868875 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:26.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f722d984640 1 -- 172.21.15.40:0/1131868875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff804bce0 msgr2=0x557ff8055ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f722d984640 1 --2- 172.21.15.40:0/1131868875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff804bce0 0x557ff8055ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f722d984640 1 -- 172.21.15.40:0/1131868875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff8050e50 msgr2=0x557ff805cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f722d984640 1 --2- 172.21.15.40:0/1131868875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff8050e50 0x557ff805cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f722d984640 1 -- 172.21.15.40:0/1131868875 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ff7cc33f0 con 0x557ff804b2f0 2024-03-18T22:46:26.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f722e185640 1 --2- 172.21.15.40:0/1131868875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff8050e50 0x557ff805cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f722d984640 1 --2- 172.21.15.40:0/1131868875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804b2f0 0x557ff8055660 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f722409b720 tx=0x7f72240d2e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd34a1ffa01fa345 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.178+0000 7f72117fa640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:46:26.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.179+0000 7f72117fa640 1 -- 172.21.15.40:0/1131868875 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722409b2d0 con 0x557ff804b2f0 2024-03-18T22:46:26.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.179+0000 7f72117fa640 1 -- 172.21.15.40:0/1131868875 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f72240af830 con 0x557ff804b2f0 2024-03-18T22:46:26.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.179+0000 7f72327cb740 1 -- 172.21.15.40:0/1131868875 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f722407f0d0 con 0x557ff804b2f0 2024-03-18T22:46:26.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.179+0000 7f72117fa640 1 -- 172.21.15.40:0/1131868875 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f722409b0f0 con 0x557ff804b2f0 2024-03-18T22:46:26.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.179+0000 7f72327cb740 10 client.26317 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:26.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.179+0000 7f72327cb740 1 -- 172.21.15.40:0/1131868875 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557ff7d8d7d0 con 0x557ff804b2f0 2024-03-18T22:46:26.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.179+0000 7f72117fa640 1 -- 172.21.15.40:0/1131868875 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f722407f0d0 con 0x557ff804b2f0 2024-03-18T22:46:26.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.179+0000 7f72117fa640 1 -- 172.21.15.40:0/1131868875 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f722409b9f0 con 0x557ff804b2f0 2024-03-18T22:46:26.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.179+0000 7f72327cb740 10 client.26317 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:26.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.179+0000 7f72327cb740 10 client.26317 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:46:26.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.179+0000 7f72327cb740 20 client.26317 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:26.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.179+0000 7f72327cb740 1 --2- 172.21.15.40:0/1131868875 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557ff80610d0 0x557ff80634c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.179+0000 7f72327cb740 4 client.26317 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:46:26.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.179+0000 7f72327cb740 1 -- 172.21.15.40:0/1131868875 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ff800d3e0 con 0x557ff80610d0 2024-03-18T22:46:26.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.179+0000 7f722d183640 1 --2- 172.21.15.40:0/1131868875 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557ff80610d0 0x557ff80634c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.180+0000 7f722d183640 1 --2- 172.21.15.40:0/1131868875 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557ff80610d0 0x557ff80634c0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e132b0b80918f652 server_cookie=7f933c0c384b8c38 in_seq=0 out_seq=0 2024-03-18T22:46:26.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.180+0000 7f72117fa640 10 client.26317 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:46:26.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.181+0000 7f72117fa640 1 -- 172.21.15.40:0/1131868875 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557ff800d3e0 con 0x557ff80610d0 2024-03-18T22:46:26.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.181+0000 7f72117fa640 10 client.26317 handle_command_reply: tid=0 2024-03-18T22:46:26.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.181+0000 7f72327cb740 1 client.26317 shutdown 2024-03-18T22:46:26.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.181+0000 7f72327cb740 1 -- 172.21.15.40:0/1131868875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804b2f0 msgr2=0x557ff8055660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.181+0000 7f72327cb740 1 --2- 172.21.15.40:0/1131868875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804b2f0 0x557ff8055660 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f722409b720 tx=0x7f72240d2e60 comp rx=0 tx=0).stop 2024-03-18T22:46:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.181+0000 7f72327cb740 1 -- 172.21.15.40:0/1131868875 shutdown_connections 2024-03-18T22:46:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.181+0000 7f72327cb740 1 --2- 172.21.15.40:0/1131868875 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557ff80610d0 0x557ff80634c0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.181+0000 7f72327cb740 1 --2- 172.21.15.40:0/1131868875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff8050e50 0x557ff805cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.181+0000 7f72327cb740 1 --2- 172.21.15.40:0/1131868875 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff804bce0 0x557ff8055ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.181+0000 7f72327cb740 1 --2- 172.21.15.40:0/1131868875 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804b2f0 0x557ff8055660 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.181+0000 7f72117fa640 0 client.26317 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:46:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.181+0000 7f72327cb740 1 -- 172.21.15.40:0/1131868875 >> 172.21.15.40:0/1131868875 conn(0x557ff8046860 msgr2=0x557ff8045ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.181+0000 7f72327cb740 1 -- 172.21.15.40:0/1131868875 shutdown_connections 2024-03-18T22:46:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.181+0000 7f72327cb740 1 -- 172.21.15.40:0/1131868875 wait complete. 2024-03-18T22:46:26.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.181+0000 7f72327cb740 20 client.26317 trim_cache size 0 max 16384 2024-03-18T22:46:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f72327cb740 1 Processor -- start 2024-03-18T22:46:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f72327cb740 1 -- start start 2024-03-18T22:46:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f72327cb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff804b2f0 0x557ff8068cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f72327cb740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804bce0 0x557ff8045a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f72327cb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff8050e50 0x557ff8045f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f72327cb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f72281ac680 con 0x557ff804b2f0 2024-03-18T22:46:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f72327cb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72281ae1f0 con 0x557ff8050e50 2024-03-18T22:46:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f72327cb740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f72281acd50 con 0x557ff804bce0 2024-03-18T22:46:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f722d984640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff804b2f0 0x557ff8068cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f722d183640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804bce0 0x557ff8045a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f722d984640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff804b2f0 0x557ff8068cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44246/0 (socket says 172.21.15.40:44246) 2024-03-18T22:46:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f722e185640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff8050e50 0x557ff8045f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f722d984640 1 -- 172.21.15.40:0/3051255508 learned_addr learned my addr 172.21.15.40:0/3051255508 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f722d183640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804bce0 0x557ff8045a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55878/0 (socket says 172.21.15.40:55878) 2024-03-18T22:46:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f722d183640 1 -- 172.21.15.40:0/3051255508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff8050e50 msgr2=0x557ff8045f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f722d183640 1 --2- 172.21.15.40:0/3051255508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff8050e50 0x557ff8045f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f722d183640 1 -- 172.21.15.40:0/3051255508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff804b2f0 msgr2=0x557ff8068cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f722d183640 1 --2- 172.21.15.40:0/3051255508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff804b2f0 0x557ff8068cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f722d183640 1 -- 172.21.15.40:0/3051255508 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ff801c880 con 0x557ff804bce0 2024-03-18T22:46:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.202+0000 7f722d984640 1 --2- 172.21.15.40:0/3051255508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff804b2f0 0x557ff8068cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.203+0000 7f722d183640 1 --2- 172.21.15.40:0/3051255508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804bce0 0x557ff8045a20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x557ff8063c60 tx=0x7f7218005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53e68a873988be54 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.203+0000 7f72137fe640 1 -- 172.21.15.40:0/3051255508 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7218027070 con 0x557ff804bce0 2024-03-18T22:46:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.203+0000 7f72137fe640 1 -- 172.21.15.40:0/3051255508 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7218010900 con 0x557ff804bce0 2024-03-18T22:46:26.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.203+0000 7f72137fe640 1 -- 172.21.15.40:0/3051255508 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72180214d0 con 0x557ff804bce0 2024-03-18T22:46:26.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.204+0000 7f72327cb740 1 -- 172.21.15.40:0/3051255508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804bce0 msgr2=0x557ff8045a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.204+0000 7f72327cb740 1 --2- 172.21.15.40:0/3051255508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804bce0 0x557ff8045a20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x557ff8063c60 tx=0x7f7218005f90 comp rx=0 tx=0).stop 2024-03-18T22:46:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.204+0000 7f72327cb740 1 -- 172.21.15.40:0/3051255508 shutdown_connections 2024-03-18T22:46:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.204+0000 7f72327cb740 1 --2- 172.21.15.40:0/3051255508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff8050e50 0x557ff8045f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.204+0000 7f72327cb740 1 --2- 172.21.15.40:0/3051255508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804bce0 0x557ff8045a20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.204+0000 7f72327cb740 1 --2- 172.21.15.40:0/3051255508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff804b2f0 0x557ff8068cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.204+0000 7f72327cb740 1 -- 172.21.15.40:0/3051255508 >> 172.21.15.40:0/3051255508 conn(0x557ff8046860 msgr2=0x557ff8045310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.204+0000 7f72327cb740 1 -- 172.21.15.40:0/3051255508 shutdown_connections 2024-03-18T22:46:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.204+0000 7f72327cb740 1 -- 172.21.15.40:0/3051255508 wait complete. 2024-03-18T22:46:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.204+0000 7f72327cb740 1 -- start start 2024-03-18T22:46:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.204+0000 7f72327cb740 1 Processor -- start 2024-03-18T22:46:26.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f72327cb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff804b2f0 0x557ff8059fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f72327cb740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804bce0 0x557ff805a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f72327cb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff8050e50 0x557ff805b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f72327cb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f72281adaf0 con 0x557ff804b2f0 2024-03-18T22:46:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f72327cb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72281ad420 con 0x557ff8050e50 2024-03-18T22:46:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f72327cb740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f72281aa660 con 0x557ff804bce0 2024-03-18T22:46:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f72327cb740 10 client.0 _ll_register_callbacks cb 0x7f7218009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f722d984640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff804b2f0 0x557ff8059fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f722d984640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff804b2f0 0x557ff8059fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44254/0 (socket says 172.21.15.40:44254) 2024-03-18T22:46:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f722d984640 1 -- 172.21.15.40:0/3673268940 learned_addr learned my addr 172.21.15.40:0/3673268940 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f722e185640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff8050e50 0x557ff805b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f722d183640 1 --2- 172.21.15.40:0/3673268940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804bce0 0x557ff805a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f722d984640 1 -- 172.21.15.40:0/3673268940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804bce0 msgr2=0x557ff805a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f722d984640 1 --2- 172.21.15.40:0/3673268940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804bce0 0x557ff805a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f722d984640 1 -- 172.21.15.40:0/3673268940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff8050e50 msgr2=0x557ff805b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f722d984640 1 --2- 172.21.15.40:0/3673268940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff8050e50 0x557ff805b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f722d984640 1 -- 172.21.15.40:0/3673268940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ff80464a0 con 0x557ff804b2f0 2024-03-18T22:46:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f722d984640 1 --2- 172.21.15.40:0/3673268940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff804b2f0 0x557ff8059fc0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7224098160 tx=0x7f72240d2df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d7f7aae22c788c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f72117fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f72117fa640 1 -- 172.21.15.40:0/3673268940 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722409b560 con 0x557ff804b2f0 2024-03-18T22:46:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f72327cb740 1 -- 172.21.15.40:0/3673268940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7218010900 con 0x557ff804b2f0 2024-03-18T22:46:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.205+0000 7f72117fa640 1 -- 172.21.15.40:0/3673268940 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f722409b9f0 con 0x557ff804b2f0 2024-03-18T22:46:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.206+0000 7f72117fa640 1 -- 172.21.15.40:0/3673268940 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f722409b2d0 con 0x557ff804b2f0 2024-03-18T22:46:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.206+0000 7f72327cb740 10 client.16500 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.206+0000 7f72327cb740 1 -- 172.21.15.40:0/3673268940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557ff805cbe0 con 0x557ff804b2f0 2024-03-18T22:46:26.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.206+0000 7f72117fa640 1 -- 172.21.15.40:0/3673268940 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7218010900 con 0x557ff804b2f0 2024-03-18T22:46:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.206+0000 7f72117fa640 1 -- 172.21.15.40:0/3673268940 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f72240974e0 con 0x557ff804b2f0 2024-03-18T22:46:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.206+0000 7f72327cb740 10 client.16500 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.206+0000 7f72327cb740 10 client.16500 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:46:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.206+0000 7f72327cb740 20 client.16500 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.206+0000 7f72327cb740 1 --2- 172.21.15.40:0/3673268940 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557ff80c7930 0x557ff80c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.206+0000 7f72327cb740 4 client.16500 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:46:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.206+0000 7f72327cb740 1 -- 172.21.15.40:0/3673268940 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557ff7fbc9f0 con 0x557ff80c7930 2024-03-18T22:46:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.206+0000 7f722d183640 1 --2- 172.21.15.40:0/3673268940 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557ff80c7930 0x557ff80c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72117fa640 10 client.16500 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:46:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f722d183640 1 --2- 172.21.15.40:0/3673268940 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557ff80c7930 0x557ff80c9d50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b9ad3eead3493c94 server_cookie=ff467a815b1385eb in_seq=0 out_seq=0 2024-03-18T22:46:26.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72117fa640 1 -- 172.21.15.40:0/3673268940 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557ff7fbc9f0 con 0x557ff80c7930 2024-03-18T22:46:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72117fa640 10 client.16500 handle_command_reply: tid=0 2024-03-18T22:46:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72327cb740 1 client.16500 shutdown 2024-03-18T22:46:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72327cb740 1 -- 172.21.15.40:0/3673268940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff804b2f0 msgr2=0x557ff8059fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72327cb740 1 --2- 172.21.15.40:0/3673268940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff804b2f0 0x557ff8059fc0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7224098160 tx=0x7f72240d2df0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72327cb740 1 -- 172.21.15.40:0/3673268940 shutdown_connections 2024-03-18T22:46:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72327cb740 1 --2- 172.21.15.40:0/3673268940 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x557ff80c7930 0x557ff80c9d50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72327cb740 1 --2- 172.21.15.40:0/3673268940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ff8050e50 0x557ff805b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72327cb740 1 --2- 172.21.15.40:0/3673268940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ff804bce0 0x557ff805a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72327cb740 1 --2- 172.21.15.40:0/3673268940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ff804b2f0 0x557ff8059fc0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72117fa640 0 client.16500 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:46:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72327cb740 1 -- 172.21.15.40:0/3673268940 >> 172.21.15.40:0/3673268940 conn(0x557ff8046860 msgr2=0x557ff8045ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72327cb740 1 -- 172.21.15.40:0/3673268940 shutdown_connections 2024-03-18T22:46:26.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72327cb740 1 -- 172.21.15.40:0/3673268940 wait complete. 2024-03-18T22:46:26.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.207+0000 7f72327cb740 20 client.16500 trim_cache size 0 max 16384 2024-03-18T22:46:26.216 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:46:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.208+0000 7f722fbe8640 1 -- 172.21.15.40:0/2605699648 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72140a2800 msgr2=0x7f72140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.208+0000 7f722fbe8640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72140a2800 0x7f72140a4cc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7220006010 tx=0x7f722002e040 comp rx=0 tx=0).stop 2024-03-18T22:46:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.208+0000 7f722fbe8640 1 -- 172.21.15.40:0/2605699648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7228103a40 msgr2=0x7f7228199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.208+0000 7f722fbe8640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7228103a40 0x7f7228199bd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f722400e9c0 tx=0x7f722400ee80 comp rx=0 tx=0).stop 2024-03-18T22:46:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.208+0000 7f722fbe8640 1 -- 172.21.15.40:0/2605699648 shutdown_connections 2024-03-18T22:46:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.208+0000 7f722fbe8640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f72140a2800 0x7f72140a4cc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.208+0000 7f722fbe8640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f72281055f0 0x7f72281a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.208+0000 7f722fbe8640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7228104c30 0x7f722819a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.208+0000 7f722fbe8640 1 --2- 172.21.15.40:0/2605699648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7228103a40 0x7f7228199bd0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.208+0000 7f722fbe8640 1 -- 172.21.15.40:0/2605699648 >> 172.21.15.40:0/2605699648 conn(0x7f72280fefd0 msgr2=0x7f7228107f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.208+0000 7f722fbe8640 1 -- 172.21.15.40:0/2605699648 shutdown_connections 2024-03-18T22:46:26.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.208+0000 7f722fbe8640 1 -- 172.21.15.40:0/2605699648 wait complete. 2024-03-18T22:46:26.219 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:46:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:26 smithi113 ceph-mon[66775]: pgmap v464: 129 pgs: 129 active+clean; 3.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:46:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:26 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3113350217' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:46:26.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.340+0000 7fecc5e8c640 1 -- 172.21.15.40:0/2289994884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0100720 msgr2=0x7fecc0100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.340+0000 7fecc5e8c640 1 --2- 172.21.15.40:0/2289994884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0100720 0x7fecc0100b40 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fecb0009900 tx=0x7fecb002bc30 comp rx=0 tx=0).stop 2024-03-18T22:46:26.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.340+0000 7fecc5e8c640 1 -- 172.21.15.40:0/2289994884 shutdown_connections 2024-03-18T22:46:26.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.340+0000 7fecc5e8c640 1 --2- 172.21.15.40:0/2289994884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc01022d0 0x7fecc0108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.340+0000 7fecc5e8c640 1 --2- 172.21.15.40:0/2289994884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecc0101910 0x7fecc0101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.340+0000 7fecc5e8c640 1 --2- 172.21.15.40:0/2289994884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0100720 0x7fecc0100b40 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.340+0000 7fecc5e8c640 1 -- 172.21.15.40:0/2289994884 >> 172.21.15.40:0/2289994884 conn(0x7fecc00fbc90 msgr2=0x7fecc00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.340+0000 7fecc5e8c640 1 -- 172.21.15.40:0/2289994884 shutdown_connections 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.340+0000 7fecc5e8c640 1 -- 172.21.15.40:0/2289994884 wait complete. 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.341+0000 7fecc5e8c640 1 Processor -- start 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.341+0000 7fecc5e8c640 1 -- start start 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.341+0000 7fecc5e8c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc0100720 0x7fecc0104850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.341+0000 7fecc5e8c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0101910 0x7fecc0102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.341+0000 7fecc5e8c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecc01022d0 0x7fecc01033d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.341+0000 7fecc5e8c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fecc010c240 con 0x7fecc0101910 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.341+0000 7fecc5e8c640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fecc010c0c0 con 0x7fecc0100720 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.341+0000 7fecc5e8c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fecc010c3c0 con 0x7fecc01022d0 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.341+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0101910 0x7fecc0102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.341+0000 7fecbffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecc01022d0 0x7fecc01033d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.341+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0101910 0x7fecc0102e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44260/0 (socket says 172.21.15.40:44260) 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.341+0000 7fecbeffd640 1 -- 172.21.15.40:0/4043672574 learned_addr learned my addr 172.21.15.40:0/4043672574 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.341+0000 7fecbf7fe640 1 --2- 172.21.15.40:0/4043672574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc0100720 0x7fecc0104850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.342+0000 7fecbeffd640 1 -- 172.21.15.40:0/4043672574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecc01022d0 msgr2=0x7fecc01033d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.342+0000 7fecbeffd640 1 --2- 172.21.15.40:0/4043672574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecc01022d0 0x7fecc01033d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.342+0000 7fecbeffd640 1 -- 172.21.15.40:0/4043672574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc0100720 msgr2=0x7fecc0104850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.342+0000 7fecbeffd640 1 --2- 172.21.15.40:0/4043672574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc0100720 0x7fecc0104850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.342+0000 7fecbeffd640 1 -- 172.21.15.40:0/4043672574 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc0103c60 con 0x7fecc0101910 2024-03-18T22:46:26.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.342+0000 7fecbffff640 1 --2- 172.21.15.40:0/4043672574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecc01022d0 0x7fecc01033d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.342+0000 7fecbeffd640 1 --2- 172.21.15.40:0/4043672574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0101910 0x7fecc0102e90 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fecb400e9f0 tx=0x7fecb400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.342+0000 7fecbcff9640 1 -- 172.21.15.40:0/4043672574 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb400cd80 con 0x7fecc0101910 2024-03-18T22:46:26.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.342+0000 7fecbcff9640 1 -- 172.21.15.40:0/4043672574 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fecb40054d0 con 0x7fecc0101910 2024-03-18T22:46:26.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.342+0000 7fecc5e8c640 1 -- 172.21.15.40:0/4043672574 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecc01a1b60 con 0x7fecc0101910 2024-03-18T22:46:26.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.342+0000 7fecbcff9640 1 -- 172.21.15.40:0/4043672574 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4010490 con 0x7fecc0101910 2024-03-18T22:46:26.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.342+0000 7fecc5e8c640 1 -- 172.21.15.40:0/4043672574 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc01a2060 con 0x7fecc0101910 2024-03-18T22:46:26.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.344+0000 7fecbcff9640 1 -- 172.21.15.40:0/4043672574 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fecb4005020 con 0x7fecc0101910 2024-03-18T22:46:26.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecc8a72740 1 Processor -- start 2024-03-18T22:46:26.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecc8a72740 1 -- start start 2024-03-18T22:46:26.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecc8a72740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c220262f0 0x562c22026710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecbcff9640 1 --2- 172.21.15.40:0/4043672574 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feca80a2800 0x7feca80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecc8a72740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c22026ce0 0x562c2202b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecc8a72740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c2202be50 0x562c2202e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecc8a72740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562c21f8a4b0 con 0x562c22026ce0 2024-03-18T22:46:26.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecc8a72740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562c21f32d70 con 0x562c2202be50 2024-03-18T22:46:26.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecc8a72740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x562c21f86550 con 0x562c220262f0 2024-03-18T22:46:26.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecbf7fe640 1 --2- 172.21.15.40:0/4043672574 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feca80a2800 0x7feca80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c22026ce0 0x562c2202b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c22026ce0 0x562c2202b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44270/0 (socket says 172.21.15.40:44270) 2024-03-18T22:46:26.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecbeffd640 1 -- 172.21.15.40:0/1729292254 learned_addr learned my addr 172.21.15.40:0/1729292254 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:26.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecbcff9640 1 -- 172.21.15.40:0/4043672574 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fecb40cf240 con 0x7fecc0101910 2024-03-18T22:46:26.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecbffff640 1 --2- 172.21.15.40:0/1729292254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c220262f0 0x562c22026710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.345+0000 7fecbffff640 1 --2- 172.21.15.40:0/1729292254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c2202be50 0x562c2202e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.346+0000 7fecbeffd640 1 -- 172.21.15.40:0/1729292254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c220262f0 msgr2=0x562c22026710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.346+0000 7fecbeffd640 1 --2- 172.21.15.40:0/1729292254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c220262f0 0x562c22026710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.346+0000 7fecbeffd640 1 -- 172.21.15.40:0/1729292254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c2202be50 msgr2=0x562c2202e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.346+0000 7fecbeffd640 1 --2- 172.21.15.40:0/1729292254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c2202be50 0x562c2202e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.346+0000 7fecbeffd640 1 -- 172.21.15.40:0/1729292254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c21f55210 con 0x562c22026ce0 2024-03-18T22:46:26.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.346+0000 7fecbffff640 1 --2- 172.21.15.40:0/1729292254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c220262f0 0x562c22026710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.346+0000 7fecbf7fe640 1 --2- 172.21.15.40:0/4043672574 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feca80a2800 0x7feca80a4cc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fecb0005ab0 tx=0x7fecb0005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.346+0000 7fecbeffd640 1 --2- 172.21.15.40:0/1729292254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c22026ce0 0x562c2202b7e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fecb400c6a0 tx=0x7fecb406d530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=645ffa4920d3634e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.346+0000 7feca5ffb640 1 -- 172.21.15.40:0/1729292254 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb40a0860 con 0x562c22026ce0 2024-03-18T22:46:26.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.346+0000 7feca5ffb640 1 -- 172.21.15.40:0/1729292254 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fecb4005310 con 0x562c22026ce0 2024-03-18T22:46:26.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.346+0000 7feca5ffb640 1 -- 172.21.15.40:0/1729292254 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb409bef0 con 0x562c22026ce0 2024-03-18T22:46:26.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.347+0000 7fecc8a72740 1 -- 172.21.15.40:0/1729292254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c22026ce0 msgr2=0x562c2202b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.347+0000 7fecc8a72740 1 --2- 172.21.15.40:0/1729292254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c22026ce0 0x562c2202b7e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fecb400c6a0 tx=0x7fecb406d530 comp rx=0 tx=0).stop 2024-03-18T22:46:26.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.347+0000 7fecc8a72740 1 -- 172.21.15.40:0/1729292254 shutdown_connections 2024-03-18T22:46:26.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.347+0000 7fecc8a72740 1 --2- 172.21.15.40:0/1729292254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c2202be50 0x562c2202e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.347+0000 7fecc8a72740 1 --2- 172.21.15.40:0/1729292254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c22026ce0 0x562c2202b7e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.347+0000 7fecc8a72740 1 --2- 172.21.15.40:0/1729292254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c220262f0 0x562c22026710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.347+0000 7fecc8a72740 1 -- 172.21.15.40:0/1729292254 >> 172.21.15.40:0/1729292254 conn(0x562c22021860 msgr2=0x562c22021c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.347+0000 7fecc8a72740 1 -- 172.21.15.40:0/1729292254 shutdown_connections 2024-03-18T22:46:26.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.347+0000 7fecc8a72740 1 -- 172.21.15.40:0/1729292254 wait complete. 2024-03-18T22:46:26.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.347+0000 7fecc8a72740 1 -- start start 2024-03-18T22:46:26.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecc8a72740 1 Processor -- start 2024-03-18T22:46:26.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecc8a72740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c220262f0 0x562c22030810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecc8a72740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c22026ce0 0x562c22030d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecc8a72740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c2202be50 0x562c22037d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecc8a72740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562c2202eda0 con 0x562c2202be50 2024-03-18T22:46:26.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecc8a72740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562c21f46310 con 0x562c220262f0 2024-03-18T22:46:26.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecc8a72740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x562c2202ef20 con 0x562c22026ce0 2024-03-18T22:46:26.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecc8a72740 10 client.0 _ll_register_callbacks cb 0x562c21f3a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:26.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecbffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c2202be50 0x562c22037d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c22026ce0 0x562c22030d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecbf7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c220262f0 0x562c22030810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecbf7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c220262f0 0x562c22030810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47480/0 (socket says 172.21.15.40:47480) 2024-03-18T22:46:26.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c22026ce0 0x562c22030d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55920/0 (socket says 172.21.15.40:55920) 2024-03-18T22:46:26.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecbf7fe640 1 -- 172.21.15.40:0/3251657182 learned_addr learned my addr 172.21.15.40:0/3251657182 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:26.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecbffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c2202be50 0x562c22037d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44274/0 (socket says 172.21.15.40:44274) 2024-03-18T22:46:26.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecbffff640 1 -- 172.21.15.40:0/3251657182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c22026ce0 msgr2=0x562c22030d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecbffff640 1 --2- 172.21.15.40:0/3251657182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c22026ce0 0x562c22030d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecbffff640 1 -- 172.21.15.40:0/3251657182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c220262f0 msgr2=0x562c22030810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecbffff640 1 --2- 172.21.15.40:0/3251657182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c220262f0 0x562c22030810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecbffff640 1 -- 172.21.15.40:0/3251657182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c21c9e3f0 con 0x562c2202be50 2024-03-18T22:46:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecbf7fe640 1 --2- 172.21.15.40:0/3251657182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c220262f0 0x562c22030810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.348+0000 7fecbeffd640 1 --2- 172.21.15.40:0/3251657182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c22026ce0 0x562c22030d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fecbffff640 1 --2- 172.21.15.40:0/3251657182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c2202be50 0x562c22037d90 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fecb8005c40 tx=0x7fecb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd6a2e9616c47b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fec9effd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fec9effd640 1 -- 172.21.15.40:0/3251657182 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8023060 con 0x562c2202be50 2024-03-18T22:46:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fec9effd640 1 -- 172.21.15.40:0/3251657182 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fecb801e430 con 0x562c2202be50 2024-03-18T22:46:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fecc8a72740 1 -- 172.21.15.40:0/3251657182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562c21fe83e0 con 0x562c2202be50 2024-03-18T22:46:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fec9effd640 1 -- 172.21.15.40:0/3251657182 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fecb802a050 con 0x562c2202be50 2024-03-18T22:46:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fecc8a72740 10 client.16512 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fecc8a72740 1 -- 172.21.15.40:0/3251657182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562c21d687d0 con 0x562c2202be50 2024-03-18T22:46:26.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fec9effd640 1 -- 172.21.15.40:0/3251657182 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x562c21fe83e0 con 0x562c2202be50 2024-03-18T22:46:26.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fec9effd640 1 -- 172.21.15.40:0/3251657182 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fecb801d070 con 0x562c2202be50 2024-03-18T22:46:26.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fecc8a72740 10 client.16512 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:26.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fecc8a72740 10 client.16512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:46:26.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fecc8a72740 20 client.16512 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:26.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fecc8a72740 1 --2- 172.21.15.40:0/3251657182 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c2203c7f0 0x562c2203ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fecc8a72740 4 client.16512 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:46:26.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.349+0000 7fecc8a72740 1 -- 172.21.15.40:0/3251657182 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c21f979f0 con 0x562c2203c7f0 2024-03-18T22:46:26.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.350+0000 7fecbf7fe640 1 --2- 172.21.15.40:0/3251657182 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c2203c7f0 0x562c2203ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.350+0000 7fec9effd640 10 client.16512 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:26.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.350+0000 7fecbf7fe640 1 --2- 172.21.15.40:0/3251657182 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c2203c7f0 0x562c2203ec50 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d13cfc46b0bc66fe server_cookie=54a71e107cb27fb3 in_seq=0 out_seq=0 2024-03-18T22:46:26.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.351+0000 7fec9effd640 1 -- 172.21.15.40:0/3251657182 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562c21f979f0 con 0x562c2203c7f0 2024-03-18T22:46:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.351+0000 7fec9effd640 10 client.16512 handle_command_reply: tid=0 2024-03-18T22:46:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.351+0000 7fecc8a72740 1 client.16512 shutdown 2024-03-18T22:46:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.351+0000 7fecc8a72740 1 -- 172.21.15.40:0/3251657182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c2202be50 msgr2=0x562c22037d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.351+0000 7fecc8a72740 1 --2- 172.21.15.40:0/3251657182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c2202be50 0x562c22037d90 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fecb8005c40 tx=0x7fecb8004a40 comp rx=0 tx=0).stop 2024-03-18T22:46:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.351+0000 7fecc8a72740 1 -- 172.21.15.40:0/3251657182 shutdown_connections 2024-03-18T22:46:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.351+0000 7fecc8a72740 1 --2- 172.21.15.40:0/3251657182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c2202be50 0x562c22037d90 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.351+0000 7fecc8a72740 1 --2- 172.21.15.40:0/3251657182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c22026ce0 0x562c22030d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.351+0000 7fecc8a72740 1 --2- 172.21.15.40:0/3251657182 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c2203c7f0 0x562c2203ec50 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.351+0000 7fecc8a72740 1 --2- 172.21.15.40:0/3251657182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c220262f0 0x562c22030810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.351+0000 7fec9effd640 0 client.16512 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.351+0000 7fecc8a72740 1 -- 172.21.15.40:0/3251657182 >> 172.21.15.40:0/3251657182 conn(0x562c22021860 msgr2=0x562c22020ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.351+0000 7fecc8a72740 1 -- 172.21.15.40:0/3251657182 shutdown_connections 2024-03-18T22:46:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.351+0000 7fecc8a72740 1 -- 172.21.15.40:0/3251657182 wait complete. 2024-03-18T22:46:26.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.351+0000 7fecc8a72740 20 client.16512 trim_cache size 0 max 16384 2024-03-18T22:46:26.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:26 smithi103 ceph-mon[66377]: pgmap v464: 129 pgs: 129 active+clean; 3.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:46:26.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:26 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3113350217' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:46:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.366+0000 7fecc8a72740 1 Processor -- start 2024-03-18T22:46:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.366+0000 7fecc8a72740 1 -- start start 2024-03-18T22:46:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.366+0000 7fecc8a72740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c220262f0 0x562c2201ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.366+0000 7fecc8a72740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c22026ce0 0x562c22037d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.366+0000 7fecc8a72740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c2202be50 0x562c2202f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.366+0000 7fecc8a72740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fecc01ac730 con 0x562c220262f0 2024-03-18T22:46:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.366+0000 7fecc8a72740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fecc01ae2a0 con 0x562c2202be50 2024-03-18T22:46:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.366+0000 7fecc8a72740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fecc01ace00 con 0x562c22026ce0 2024-03-18T22:46:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.366+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c22026ce0 0x562c22037d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.366+0000 7fecbf7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c220262f0 0x562c2201ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.366+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c22026ce0 0x562c22037d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55924/0 (socket says 172.21.15.40:55924) 2024-03-18T22:46:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.366+0000 7fecbeffd640 1 -- 172.21.15.40:0/3448085498 learned_addr learned my addr 172.21.15.40:0/3448085498 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.366+0000 7fecbffff640 1 --2- 172.21.15.40:0/3448085498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c2202be50 0x562c2202f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.367+0000 7fecbf7fe640 1 -- 172.21.15.40:0/3448085498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c22026ce0 msgr2=0x562c22037d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.367+0000 7fecbf7fe640 1 --2- 172.21.15.40:0/3448085498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c22026ce0 0x562c22037d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.367+0000 7fecbf7fe640 1 -- 172.21.15.40:0/3448085498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c2202be50 msgr2=0x562c2202f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.367+0000 7fecbf7fe640 1 --2- 172.21.15.40:0/3448085498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c2202be50 0x562c2202f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.367+0000 7fecbf7fe640 1 -- 172.21.15.40:0/3448085498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c21ff7880 con 0x562c220262f0 2024-03-18T22:46:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.367+0000 7fecbffff640 1 --2- 172.21.15.40:0/3448085498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c2202be50 0x562c2202f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.367+0000 7fecbf7fe640 1 --2- 172.21.15.40:0/3448085498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c220262f0 0x562c2201ca30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fecb0005ec0 tx=0x7fecb003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8012f1a737249e30 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.367+0000 7feca57fa640 1 -- 172.21.15.40:0/3448085498 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb0059070 con 0x562c220262f0 2024-03-18T22:46:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.367+0000 7feca57fa640 1 -- 172.21.15.40:0/3448085498 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fecb00055d0 con 0x562c220262f0 2024-03-18T22:46:26.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.367+0000 7feca57fa640 1 -- 172.21.15.40:0/3448085498 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb0003850 con 0x562c220262f0 2024-03-18T22:46:26.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.368+0000 7fecc8a72740 1 -- 172.21.15.40:0/3448085498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c220262f0 msgr2=0x562c2201ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.368+0000 7fecc8a72740 1 --2- 172.21.15.40:0/3448085498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c220262f0 0x562c2201ca30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fecb0005ec0 tx=0x7fecb003e000 comp rx=0 tx=0).stop 2024-03-18T22:46:26.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.368+0000 7fecc8a72740 1 -- 172.21.15.40:0/3448085498 shutdown_connections 2024-03-18T22:46:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.368+0000 7fecc8a72740 1 --2- 172.21.15.40:0/3448085498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c2202be50 0x562c2202f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.368+0000 7fecc8a72740 1 --2- 172.21.15.40:0/3448085498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c22026ce0 0x562c22037d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.368+0000 7fecc8a72740 1 --2- 172.21.15.40:0/3448085498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c220262f0 0x562c2201ca30 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.368+0000 7fecc8a72740 1 -- 172.21.15.40:0/3448085498 >> 172.21.15.40:0/3448085498 conn(0x562c22021860 msgr2=0x562c22027780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.368+0000 7fecc8a72740 1 -- 172.21.15.40:0/3448085498 shutdown_connections 2024-03-18T22:46:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.368+0000 7fecc8a72740 1 -- 172.21.15.40:0/3448085498 wait complete. 2024-03-18T22:46:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.368+0000 7fecc8a72740 1 -- start start 2024-03-18T22:46:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.368+0000 7fecc8a72740 1 Processor -- start 2024-03-18T22:46:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecc8a72740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c220262f0 0x562c22034980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecc8a72740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c22026ce0 0x562c22034ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecc8a72740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c2202be50 0x562c220a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecc8a72740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fecc01adba0 con 0x562c2202be50 2024-03-18T22:46:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecc8a72740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fecc01ad4d0 con 0x562c22026ce0 2024-03-18T22:46:26.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecc8a72740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fecc01aa710 con 0x562c220262f0 2024-03-18T22:46:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecc8a72740 10 client.0 _ll_register_callbacks cb 0x7fecb8003a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecbffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c2202be50 0x562c220a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecbf7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c220262f0 0x562c22034980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecbffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c2202be50 0x562c220a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44286/0 (socket says 172.21.15.40:44286) 2024-03-18T22:46:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecbffff640 1 -- 172.21.15.40:0/345961674 learned_addr learned my addr 172.21.15.40:0/345961674 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecbeffd640 1 --2- 172.21.15.40:0/345961674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c22026ce0 0x562c22034ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecbffff640 1 -- 172.21.15.40:0/345961674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c220262f0 msgr2=0x562c22034980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecbffff640 1 --2- 172.21.15.40:0/345961674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c220262f0 0x562c22034980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecbffff640 1 -- 172.21.15.40:0/345961674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c22026ce0 msgr2=0x562c22034ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecbffff640 1 --2- 172.21.15.40:0/345961674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c22026ce0 0x562c22034ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecbffff640 1 -- 172.21.15.40:0/345961674 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c22037f90 con 0x562c2202be50 2024-03-18T22:46:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecbeffd640 1 --2- 172.21.15.40:0/345961674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c22026ce0 0x562c22034ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecbf7fe640 1 --2- 172.21.15.40:0/345961674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c220262f0 0x562c22034980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecbffff640 1 --2- 172.21.15.40:0/345961674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c2202be50 0x562c220a0b80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fecb8009540 tx=0x7fecb8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f949655a9e3e618 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fec9effd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fec9effd640 1 -- 172.21.15.40:0/345961674 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8004cb0 con 0x562c2202be50 2024-03-18T22:46:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fec9effd640 1 -- 172.21.15.40:0/345961674 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fecb800f980 con 0x562c2202be50 2024-03-18T22:46:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.369+0000 7fecc8a72740 1 -- 172.21.15.40:0/345961674 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecb00055d0 con 0x562c2202be50 2024-03-18T22:46:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.370+0000 7fec9effd640 1 -- 172.21.15.40:0/345961674 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fecb8004e50 con 0x562c2202be50 2024-03-18T22:46:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.370+0000 7fecc8a72740 10 client.16518 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.370+0000 7fecc8a72740 1 -- 172.21.15.40:0/345961674 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562c220a2070 con 0x562c2202be50 2024-03-18T22:46:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.370+0000 7fec9effd640 1 -- 172.21.15.40:0/345961674 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fecb00055d0 con 0x562c2202be50 2024-03-18T22:46:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.370+0000 7fec9effd640 1 -- 172.21.15.40:0/345961674 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7fecb801d040 con 0x562c2202be50 2024-03-18T22:46:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.370+0000 7fecc8a72740 10 client.16518 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.370+0000 7fecc8a72740 10 client.16518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:46:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.370+0000 7fecc8a72740 20 client.16518 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.370+0000 7fecc8a72740 1 --2- 172.21.15.40:0/345961674 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c2203c7f0 0x562c220a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:26.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.370+0000 7fecc8a72740 4 client.16518 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:46:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.370+0000 7fecc8a72740 1 -- 172.21.15.40:0/345961674 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562c21f91d40 con 0x562c2203c7f0 2024-03-18T22:46:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.370+0000 7fecbf7fe640 1 --2- 172.21.15.40:0/345961674 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c2203c7f0 0x562c220a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fec9effd640 10 client.16518 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fecbf7fe640 1 --2- 172.21.15.40:0/345961674 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c2203c7f0 0x562c220a4a50 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ca1fed82c3ee04b server_cookie=b90d3b744342982f in_seq=0 out_seq=0 2024-03-18T22:46:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fec9effd640 1 -- 172.21.15.40:0/345961674 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562c21f91d40 con 0x562c2203c7f0 2024-03-18T22:46:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fec9effd640 10 client.16518 handle_command_reply: tid=0 2024-03-18T22:46:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fecc8a72740 1 client.16518 shutdown 2024-03-18T22:46:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fecc8a72740 1 -- 172.21.15.40:0/345961674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c2202be50 msgr2=0x562c220a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fecc8a72740 1 --2- 172.21.15.40:0/345961674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c2202be50 0x562c220a0b80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fecb8009540 tx=0x7fecb8009f90 comp rx=0 tx=0).stop 2024-03-18T22:46:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fecc8a72740 1 -- 172.21.15.40:0/345961674 shutdown_connections 2024-03-18T22:46:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fecc8a72740 1 --2- 172.21.15.40:0/345961674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c2202be50 0x562c220a0b80 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fecc8a72740 1 --2- 172.21.15.40:0/345961674 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c2203c7f0 0x562c220a4a50 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fecc8a72740 1 --2- 172.21.15.40:0/345961674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c22026ce0 0x562c22034ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fecc8a72740 1 --2- 172.21.15.40:0/345961674 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c220262f0 0x562c22034980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fec9effd640 0 client.16518 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:26.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fecc8a72740 1 -- 172.21.15.40:0/345961674 >> 172.21.15.40:0/345961674 conn(0x562c22021860 msgr2=0x562c2201ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fecc8a72740 1 -- 172.21.15.40:0/345961674 shutdown_connections 2024-03-18T22:46:26.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fecc8a72740 1 -- 172.21.15.40:0/345961674 wait complete. 2024-03-18T22:46:26.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.371+0000 7fecc8a72740 20 client.16518 trim_cache size 0 max 16384 2024-03-18T22:46:26.378 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:46:26.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.372+0000 7fecc5e8c640 1 -- 172.21.15.40:0/4043672574 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feca80a2800 msgr2=0x7feca80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.372+0000 7fecc5e8c640 1 --2- 172.21.15.40:0/4043672574 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feca80a2800 0x7feca80a4cc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fecb0005ab0 tx=0x7fecb0005fd0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.372+0000 7fecc5e8c640 1 -- 172.21.15.40:0/4043672574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0101910 msgr2=0x7fecc0102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:26.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.372+0000 7fecc5e8c640 1 --2- 172.21.15.40:0/4043672574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0101910 0x7fecc0102e90 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fecb400e9f0 tx=0x7fecb400edf0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.372+0000 7fecc5e8c640 1 -- 172.21.15.40:0/4043672574 shutdown_connections 2024-03-18T22:46:26.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.372+0000 7fecc5e8c640 1 --2- 172.21.15.40:0/4043672574 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feca80a2800 0x7feca80a4cc0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.372+0000 7fecc5e8c640 1 --2- 172.21.15.40:0/4043672574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fecc01022d0 0x7fecc01033d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.372+0000 7fecc5e8c640 1 --2- 172.21.15.40:0/4043672574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fecc0101910 0x7fecc0102e90 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.372+0000 7fecc5e8c640 1 --2- 172.21.15.40:0/4043672574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecc0100720 0x7fecc0104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:26.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.372+0000 7fecc5e8c640 1 -- 172.21.15.40:0/4043672574 >> 172.21.15.40:0/4043672574 conn(0x7fecc00fbc90 msgr2=0x7fecc0068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:26.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.372+0000 7fecc5e8c640 1 -- 172.21.15.40:0/4043672574 shutdown_connections 2024-03-18T22:46:26.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:26.372+0000 7fecc5e8c640 1 -- 172.21.15.40:0/4043672574 wait complete. 2024-03-18T22:46:27.384 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:46:27.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:46:27.385 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:46:27.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.496+0000 7faa278dc640 1 -- 172.21.15.40:0/977899144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa200ffae0 msgr2=0x7faa200fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.496+0000 7faa278dc640 1 --2- 172.21.15.40:0/977899144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa200ffae0 0x7faa200fff00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009930 tx=0x7faa1c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:46:27.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.496+0000 7faa278dc640 1 -- 172.21.15.40:0/977899144 shutdown_connections 2024-03-18T22:46:27.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.496+0000 7faa278dc640 1 --2- 172.21.15.40:0/977899144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa20103470 0x7faa20109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.496+0000 7faa278dc640 1 --2- 172.21.15.40:0/977899144 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa20102ab0 0x7faa20102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.496+0000 7faa278dc640 1 --2- 172.21.15.40:0/977899144 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa200ffae0 0x7faa200fff00 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.496+0000 7faa278dc640 1 -- 172.21.15.40:0/977899144 >> 172.21.15.40:0/977899144 conn(0x7faa200fb410 msgr2=0x7faa200fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:27.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.496+0000 7faa278dc640 1 -- 172.21.15.40:0/977899144 shutdown_connections 2024-03-18T22:46:27.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.496+0000 7faa278dc640 1 -- 172.21.15.40:0/977899144 wait complete. 2024-03-18T22:46:27.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa278dc640 1 Processor -- start 2024-03-18T22:46:27.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa278dc640 1 -- start start 2024-03-18T22:46:27.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa278dc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa200ffae0 0x7faa20197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa278dc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa20102ab0 0x7faa20198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa278dc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa20103470 0x7faa2019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa278dc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa2010d360 con 0x7faa20103470 2024-03-18T22:46:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa278dc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faa2010d1e0 con 0x7faa200ffae0 2024-03-18T22:46:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa278dc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7faa2010d4e0 con 0x7faa20102ab0 2024-03-18T22:46:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa25e79640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa20103470 0x7faa2019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa25e79640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa20103470 0x7faa2019f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44312/0 (socket says 172.21.15.40:44312) 2024-03-18T22:46:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa25e79640 1 -- 172.21.15.40:0/2095770727 learned_addr learned my addr 172.21.15.40:0/2095770727 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa24e77640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa20102ab0 0x7faa20198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa25678640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa200ffae0 0x7faa20197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa25e79640 1 -- 172.21.15.40:0/2095770727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa20102ab0 msgr2=0x7faa20198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa25e79640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa20102ab0 0x7faa20198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa25e79640 1 -- 172.21.15.40:0/2095770727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa200ffae0 msgr2=0x7faa20197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa25e79640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa200ffae0 0x7faa20197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa25e79640 1 -- 172.21.15.40:0/2095770727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa2019f730 con 0x7faa20103470 2024-03-18T22:46:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa25678640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa200ffae0 0x7faa20197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.497+0000 7faa24e77640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa20102ab0 0x7faa20198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.498+0000 7faa25e79640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa20103470 0x7faa2019f0c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7faa1800eb10 tx=0x7faa1800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.498+0000 7faa167fc640 1 -- 172.21.15.40:0/2095770727 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa18004b20 con 0x7faa20103470 2024-03-18T22:46:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.498+0000 7faa278dc640 1 -- 172.21.15.40:0/2095770727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa2019fa20 con 0x7faa20103470 2024-03-18T22:46:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.498+0000 7faa167fc640 1 -- 172.21.15.40:0/2095770727 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa18004cc0 con 0x7faa20103470 2024-03-18T22:46:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.498+0000 7faa167fc640 1 -- 172.21.15.40:0/2095770727 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa18010690 con 0x7faa20103470 2024-03-18T22:46:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.498+0000 7faa278dc640 1 -- 172.21.15.40:0/2095770727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa2019ff20 con 0x7faa20103470 2024-03-18T22:46:27.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.500+0000 7faa2a5a5740 1 Processor -- start 2024-03-18T22:46:27.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa167fc640 1 -- 172.21.15.40:0/2095770727 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7faa18005020 con 0x7faa20103470 2024-03-18T22:46:27.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa2a5a5740 1 -- start start 2024-03-18T22:46:27.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa2a5a5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33df3a0 0x557ed33df7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa2a5a5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33dfd90 0x557ed33e4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa2a5a5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33e4f00 0x557ed33e7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa2a5a5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557ed3344160 con 0x557ed33dfd90 2024-03-18T22:46:27.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa2a5a5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557ed32ece10 con 0x557ed33e4f00 2024-03-18T22:46:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa2a5a5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557ed3340200 con 0x557ed33df3a0 2024-03-18T22:46:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa25678640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33dfd90 0x557ed33e4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa24e77640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33df3a0 0x557ed33df7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa25678640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33dfd90 0x557ed33e4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44318/0 (socket says 172.21.15.40:44318) 2024-03-18T22:46:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa25678640 1 -- 172.21.15.40:0/4264763145 learned_addr learned my addr 172.21.15.40:0/4264763145 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa24e77640 1 --2- 172.21.15.40:0/4264763145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33e4f00 0x557ed33e7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa167fc640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa0c0a2800 0x7faa0c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa25678640 1 -- 172.21.15.40:0/4264763145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33df3a0 msgr2=0x557ed33df7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa25678640 1 --2- 172.21.15.40:0/4264763145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33df3a0 0x557ed33df7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa25678640 1 -- 172.21.15.40:0/4264763145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33e4f00 msgr2=0x557ed33e7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa25678640 1 --2- 172.21.15.40:0/4264763145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33e4f00 0x557ed33e7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa25678640 1 -- 172.21.15.40:0/4264763145 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ed31207e0 con 0x557ed33dfd90 2024-03-18T22:46:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa24e77640 1 --2- 172.21.15.40:0/4264763145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33df3a0 0x557ed33df7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:27.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.501+0000 7faa25e79640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa0c0a2800 0x7faa0c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.502+0000 7faa167fc640 1 -- 172.21.15.40:0/2095770727 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7faa180cf3c0 con 0x7faa20103470 2024-03-18T22:46:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.502+0000 7faa25678640 1 --2- 172.21.15.40:0/4264763145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33dfd90 0x557ed33e4890 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009930 tx=0x7faa1c02cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f14012e7863e6f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.502+0000 7faa0b7fe640 1 -- 172.21.15.40:0/4264763145 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c045020 con 0x557ed33dfd90 2024-03-18T22:46:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.502+0000 7faa25e79640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa0c0a2800 0x7faa0c0a4cc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7faa18097dd0 tx=0x7faa1806e6a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.502+0000 7faa0b7fe640 1 -- 172.21.15.40:0/4264763145 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa1c005040 con 0x557ed33dfd90 2024-03-18T22:46:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.502+0000 7faa0b7fe640 1 -- 172.21.15.40:0/4264763145 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c004920 con 0x557ed33dfd90 2024-03-18T22:46:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.503+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4264763145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33dfd90 msgr2=0x557ed33e4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.503+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4264763145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33dfd90 0x557ed33e4890 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009930 tx=0x7faa1c02cc90 comp rx=0 tx=0).stop 2024-03-18T22:46:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.503+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4264763145 shutdown_connections 2024-03-18T22:46:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.503+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4264763145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33e4f00 0x557ed33e7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.503+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4264763145 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33dfd90 0x557ed33e4890 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.503+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4264763145 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33df3a0 0x557ed33df7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.503+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4264763145 >> 172.21.15.40:0/4264763145 conn(0x557ed33da910 msgr2=0x557ed33dad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:27.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.503+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4264763145 shutdown_connections 2024-03-18T22:46:27.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.503+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4264763145 wait complete. 2024-03-18T22:46:27.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.503+0000 7faa2a5a5740 1 -- start start 2024-03-18T22:46:27.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.503+0000 7faa2a5a5740 1 Processor -- start 2024-03-18T22:46:27.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa2a5a5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33df3a0 0x557ed33e9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa2a5a5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33e9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa2a5a5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33e4f00 0x557ed33f0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa2a5a5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557ed33e7e50 con 0x557ed33e4f00 2024-03-18T22:46:27.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa2a5a5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557ed33001a0 con 0x557ed33df3a0 2024-03-18T22:46:27.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa2a5a5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557ed33e7fd0 con 0x557ed33dfd90 2024-03-18T22:46:27.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa2a5a5740 10 client.0 _ll_register_callbacks cb 0x557ed3057c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:27.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa25678640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33e9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa24e77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33e4f00 0x557ed33f0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa25678640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33e9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55978/0 (socket says 172.21.15.40:55978) 2024-03-18T22:46:27.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa25678640 1 -- 172.21.15.40:0/4142814594 learned_addr learned my addr 172.21.15.40:0/4142814594 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:27.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa24e77640 1 --2- 172.21.15.40:0/4142814594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33df3a0 0x557ed33e9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa25678640 1 -- 172.21.15.40:0/4142814594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33df3a0 msgr2=0x557ed33e9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:27.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa25678640 1 --2- 172.21.15.40:0/4142814594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33df3a0 0x557ed33e9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa25678640 1 -- 172.21.15.40:0/4142814594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33e4f00 msgr2=0x557ed33f0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:27.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa25678640 1 --2- 172.21.15.40:0/4142814594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33e4f00 0x557ed33f0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa25678640 1 -- 172.21.15.40:0/4142814594 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ed3180f70 con 0x557ed33dfd90 2024-03-18T22:46:27.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa24e77640 1 --2- 172.21.15.40:0/4142814594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33e4f00 0x557ed33f0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:27.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa25678640 1 --2- 172.21.15.40:0/4142814594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33e9d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009900 tx=0x7faa1c0048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5790211c909c8827 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:27.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.504+0000 7faa08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:46:27.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.505+0000 7faa08ff9640 1 -- 172.21.15.40:0/4142814594 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c045040 con 0x557ed33dfd90 2024-03-18T22:46:27.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.505+0000 7faa08ff9640 1 -- 172.21.15.40:0/4142814594 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7faa1c041480 con 0x557ed33dfd90 2024-03-18T22:46:27.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.505+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4142814594 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa1c005040 con 0x557ed33dfd90 2024-03-18T22:46:27.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.505+0000 7faa08ff9640 1 -- 172.21.15.40:0/4142814594 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faa1c05b050 con 0x557ed33dfd90 2024-03-18T22:46:27.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.505+0000 7faa2a5a5740 10 client.26353 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:27.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.505+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4142814594 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557ed3018170 con 0x557ed33dfd90 2024-03-18T22:46:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.505+0000 7faa08ff9640 1 -- 172.21.15.40:0/4142814594 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa1c005040 con 0x557ed33dfd90 2024-03-18T22:46:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.505+0000 7faa08ff9640 1 -- 172.21.15.40:0/4142814594 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7faa1c04a450 con 0x557ed33dfd90 2024-03-18T22:46:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.505+0000 7faa2a5a5740 10 client.26353 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.505+0000 7faa2a5a5740 10 client.26353 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:46:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.505+0000 7faa2a5a5740 20 client.26353 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.505+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4142814594 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557ed33f5270 0x557ed33f7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.505+0000 7faa2a5a5740 4 client.26353 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:46:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.505+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4142814594 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ed33516a0 con 0x557ed33f5270 2024-03-18T22:46:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.506+0000 7faa24e77640 1 --2- 172.21.15.40:0/4142814594 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557ed33f5270 0x557ed33f7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.506+0000 7faa24e77640 1 --2- 172.21.15.40:0/4142814594 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557ed33f5270 0x557ed33f7660 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20868e5ac1be1a52 server_cookie=3f23568cd581efd2 in_seq=0 out_seq=0 2024-03-18T22:46:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.506+0000 7faa08ff9640 10 client.26353 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.507+0000 7faa08ff9640 1 -- 172.21.15.40:0/4142814594 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557ed33516a0 con 0x557ed33f5270 2024-03-18T22:46:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.507+0000 7faa08ff9640 10 client.26353 handle_command_reply: tid=0 2024-03-18T22:46:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.507+0000 7faa2a5a5740 1 client.26353 shutdown 2024-03-18T22:46:27.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.507+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4142814594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 msgr2=0x557ed33e9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.507+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4142814594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33e9d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009900 tx=0x7faa1c0048f0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.507+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4142814594 shutdown_connections 2024-03-18T22:46:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.507+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4142814594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33e4f00 0x557ed33f0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.507+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4142814594 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33e9d40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.507+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4142814594 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557ed33f5270 0x557ed33f7660 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.507+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4142814594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33df3a0 0x557ed33e9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.507+0000 7faa08ff9640 0 client.26353 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.507+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4142814594 >> 172.21.15.40:0/4142814594 conn(0x557ed33da910 msgr2=0x557ed33d9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.507+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4142814594 shutdown_connections 2024-03-18T22:46:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.507+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4142814594 wait complete. 2024-03-18T22:46:27.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.507+0000 7faa2a5a5740 20 client.26353 trim_cache size 0 max 16384 2024-03-18T22:46:27.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.530+0000 7faa2a5a5740 1 Processor -- start 2024-03-18T22:46:27.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.530+0000 7faa2a5a5740 1 -- start start 2024-03-18T22:46:27.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.530+0000 7faa2a5a5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33df3a0 0x557ed33de280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.530+0000 7faa2a5a5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33f0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.530+0000 7faa2a5a5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33e4f00 0x557ed33fbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.530+0000 7faa2a5a5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa201a82d0 con 0x557ed33df3a0 2024-03-18T22:46:27.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.530+0000 7faa2a5a5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faa201a9e40 con 0x557ed33e4f00 2024-03-18T22:46:27.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.530+0000 7faa2a5a5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7faa201a89a0 con 0x557ed33dfd90 2024-03-18T22:46:27.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.530+0000 7faa24e77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33df3a0 0x557ed33de280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.530+0000 7faa25678640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33f0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.530+0000 7faa24e77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33df3a0 0x557ed33de280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44336/0 (socket says 172.21.15.40:44336) 2024-03-18T22:46:27.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.530+0000 7faa25678640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33f0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55992/0 (socket says 172.21.15.40:55992) 2024-03-18T22:46:27.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.530+0000 7faa24e77640 1 -- 172.21.15.40:0/3923108606 learned_addr learned my addr 172.21.15.40:0/3923108606 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:27.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.530+0000 7faa24e77640 1 --2- 172.21.15.40:0/3923108606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33e4f00 0x557ed33fbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.531+0000 7faa25678640 1 -- 172.21.15.40:0/3923108606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33e4f00 msgr2=0x557ed33fbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:27.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.531+0000 7faa25678640 1 --2- 172.21.15.40:0/3923108606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33e4f00 0x557ed33fbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.531+0000 7faa25678640 1 -- 172.21.15.40:0/3923108606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33df3a0 msgr2=0x557ed33de280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:27.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.531+0000 7faa25678640 1 --2- 172.21.15.40:0/3923108606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33df3a0 0x557ed33de280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.531+0000 7faa25678640 1 -- 172.21.15.40:0/3923108606 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ed33e18d0 con 0x557ed33dfd90 2024-03-18T22:46:27.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.531+0000 7faa24e77640 1 --2- 172.21.15.40:0/3923108606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33df3a0 0x557ed33de280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:27.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.531+0000 7faa25678640 1 --2- 172.21.15.40:0/3923108606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33f0d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faa1c04a820 tx=0x7faa1c004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc0a2928ce0459b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:27.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.531+0000 7faa0affd640 1 -- 172.21.15.40:0/3923108606 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c045040 con 0x557ed33dfd90 2024-03-18T22:46:27.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.531+0000 7faa0affd640 1 -- 172.21.15.40:0/3923108606 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa1c04a450 con 0x557ed33dfd90 2024-03-18T22:46:27.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.531+0000 7faa0affd640 1 -- 172.21.15.40:0/3923108606 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c003850 con 0x557ed33dfd90 2024-03-18T22:46:27.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.532+0000 7faa2a5a5740 1 -- 172.21.15.40:0/3923108606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 msgr2=0x557ed33f0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.532+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/3923108606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33f0d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faa1c04a820 tx=0x7faa1c004960 comp rx=0 tx=0).stop 2024-03-18T22:46:27.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.532+0000 7faa2a5a5740 1 -- 172.21.15.40:0/3923108606 shutdown_connections 2024-03-18T22:46:27.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.532+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/3923108606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33e4f00 0x557ed33fbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.532+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/3923108606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33f0d40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.532+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/3923108606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33df3a0 0x557ed33de280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.532+0000 7faa2a5a5740 1 -- 172.21.15.40:0/3923108606 >> 172.21.15.40:0/3923108606 conn(0x557ed33da910 msgr2=0x557ed33ddb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.532+0000 7faa2a5a5740 1 -- 172.21.15.40:0/3923108606 shutdown_connections 2024-03-18T22:46:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.532+0000 7faa2a5a5740 1 -- 172.21.15.40:0/3923108606 wait complete. 2024-03-18T22:46:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.532+0000 7faa2a5a5740 1 -- start start 2024-03-18T22:46:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.532+0000 7faa2a5a5740 1 Processor -- start 2024-03-18T22:46:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa2a5a5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33df3a0 0x557ed33eeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa2a5a5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33ef050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa2a5a5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33e4f00 0x557ed33f0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa2a5a5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa201a9740 con 0x557ed33df3a0 2024-03-18T22:46:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa2a5a5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faa201a9070 con 0x557ed33e4f00 2024-03-18T22:46:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa2a5a5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7faa201a62b0 con 0x557ed33dfd90 2024-03-18T22:46:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa2a5a5740 10 client.0 _ll_register_callbacks cb 0x7faa10015a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa24e77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33df3a0 0x557ed33eeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa24e77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33df3a0 0x557ed33eeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44344/0 (socket says 172.21.15.40:44344) 2024-03-18T22:46:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa24e77640 1 -- 172.21.15.40:0/4195110185 learned_addr learned my addr 172.21.15.40:0/4195110185 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa25678640 1 --2- 172.21.15.40:0/4195110185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33ef050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa24e77640 1 --2- 172.21.15.40:0/4195110185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33e4f00 0x557ed33f0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa24e77640 1 -- 172.21.15.40:0/4195110185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 msgr2=0x557ed33ef050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa24e77640 1 --2- 172.21.15.40:0/4195110185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33ef050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa24e77640 1 -- 172.21.15.40:0/4195110185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33e4f00 msgr2=0x557ed33f0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa24e77640 1 --2- 172.21.15.40:0/4195110185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33e4f00 0x557ed33f0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa24e77640 1 -- 172.21.15.40:0/4195110185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ed33f0600 con 0x557ed33df3a0 2024-03-18T22:46:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa25678640 1 --2- 172.21.15.40:0/4195110185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33ef050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa24e77640 1 --2- 172.21.15.40:0/4195110185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33df3a0 0x557ed33eeb10 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7faa100058c0 tx=0x7faa10005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618ef6b35b92950e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.533+0000 7faa08ff9640 1 -- 172.21.15.40:0/4195110185 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1002c710 con 0x557ed33df3a0 2024-03-18T22:46:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.534+0000 7faa08ff9640 1 -- 172.21.15.40:0/4195110185 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7faa10014840 con 0x557ed33df3a0 2024-03-18T22:46:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.534+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4195110185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa1c04a450 con 0x557ed33df3a0 2024-03-18T22:46:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.534+0000 7faa08ff9640 1 -- 172.21.15.40:0/4195110185 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faa1003c050 con 0x557ed33df3a0 2024-03-18T22:46:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.534+0000 7faa2a5a5740 10 client.16536 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.534+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4195110185 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557ed345bb90 con 0x557ed33df3a0 2024-03-18T22:46:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.534+0000 7faa08ff9640 1 -- 172.21.15.40:0/4195110185 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa1c04a450 con 0x557ed33df3a0 2024-03-18T22:46:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.534+0000 7faa08ff9640 1 -- 172.21.15.40:0/4195110185 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7faa10024040 con 0x557ed33df3a0 2024-03-18T22:46:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.534+0000 7faa2a5a5740 10 client.16536 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.534+0000 7faa2a5a5740 10 client.16536 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:46:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.534+0000 7faa2a5a5740 20 client.16536 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.534+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4195110185 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557ed345c5c0 0x557ed345e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.534+0000 7faa2a5a5740 4 client.16536 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:46:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.534+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4195110185 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557ed334b9f0 con 0x557ed345c5c0 2024-03-18T22:46:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.534+0000 7faa25678640 1 --2- 172.21.15.40:0/4195110185 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557ed345c5c0 0x557ed345e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.535+0000 7faa25678640 1 --2- 172.21.15.40:0/4195110185 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557ed345c5c0 0x557ed345e9e0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af6510ebeae26952 server_cookie=279231b4bfb26790 in_seq=0 out_seq=0 2024-03-18T22:46:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.535+0000 7faa08ff9640 10 client.16536 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:27.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.536+0000 7faa08ff9640 1 -- 172.21.15.40:0/4195110185 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557ed334b9f0 con 0x557ed345c5c0 2024-03-18T22:46:27.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.536+0000 7faa08ff9640 10 client.16536 handle_command_reply: tid=0 2024-03-18T22:46:27.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.536+0000 7faa2a5a5740 1 client.16536 shutdown 2024-03-18T22:46:27.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.536+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4195110185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33df3a0 msgr2=0x557ed33eeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.536+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4195110185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33df3a0 0x557ed33eeb10 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7faa100058c0 tx=0x7faa10005f90 comp rx=0 tx=0).stop 2024-03-18T22:46:27.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.536+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4195110185 shutdown_connections 2024-03-18T22:46:27.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.536+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4195110185 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557ed345c5c0 0x557ed345e9e0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.536+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4195110185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557ed33e4f00 0x557ed33f0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.536+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4195110185 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557ed33dfd90 0x557ed33ef050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.536+0000 7faa08ff9640 0 client.16536 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:27.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.536+0000 7faa2a5a5740 1 --2- 172.21.15.40:0/4195110185 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ed33df3a0 0x557ed33eeb10 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.536+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4195110185 >> 172.21.15.40:0/4195110185 conn(0x557ed33da910 msgr2=0x557ed33de400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:27.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.536+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4195110185 shutdown_connections 2024-03-18T22:46:27.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.536+0000 7faa2a5a5740 1 -- 172.21.15.40:0/4195110185 wait complete. 2024-03-18T22:46:27.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.536+0000 7faa2a5a5740 20 client.16536 trim_cache size 0 max 16384 2024-03-18T22:46:27.543 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:46:27.543 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:46:27.543 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "6cbeb0ee-8c3c-4662-bf50-44dc9054ed3c", 2024-03-18T22:46:27.543 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:46:27.543 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:46:27.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.537+0000 7faa278dc640 1 -- 172.21.15.40:0/2095770727 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa0c0a2800 msgr2=0x7faa0c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.537+0000 7faa278dc640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa0c0a2800 0x7faa0c0a4cc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7faa18097dd0 tx=0x7faa1806e6a0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.537+0000 7faa278dc640 1 -- 172.21.15.40:0/2095770727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa20103470 msgr2=0x7faa2019f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.537+0000 7faa278dc640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa20103470 0x7faa2019f0c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7faa1800eb10 tx=0x7faa1800c2d0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.537+0000 7faa278dc640 1 -- 172.21.15.40:0/2095770727 shutdown_connections 2024-03-18T22:46:27.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.537+0000 7faa278dc640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa0c0a2800 0x7faa0c0a4cc0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.537+0000 7faa278dc640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa20103470 0x7faa2019f0c0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.537+0000 7faa278dc640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa20102ab0 0x7faa20198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.537+0000 7faa278dc640 1 --2- 172.21.15.40:0/2095770727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa200ffae0 0x7faa20197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.537+0000 7faa278dc640 1 -- 172.21.15.40:0/2095770727 >> 172.21.15.40:0/2095770727 conn(0x7faa200fb410 msgr2=0x7faa20105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:27.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.537+0000 7faa278dc640 1 -- 172.21.15.40:0/2095770727 shutdown_connections 2024-03-18T22:46:27.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.537+0000 7faa278dc640 1 -- 172.21.15.40:0/2095770727 wait complete. 2024-03-18T22:46:27.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:46:27.550 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:46:27.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.666+0000 7f09d69d4640 1 -- 172.21.15.40:0/2603149441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09d0100720 msgr2=0x7f09d0100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.666+0000 7f09d69d4640 1 --2- 172.21.15.40:0/2603149441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09d0100720 0x7f09d0100b40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009900 tx=0x7f09c002bc30 comp rx=0 tx=0).stop 2024-03-18T22:46:27.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.666+0000 7f09d69d4640 1 -- 172.21.15.40:0/2603149441 shutdown_connections 2024-03-18T22:46:27.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.666+0000 7f09d69d4640 1 --2- 172.21.15.40:0/2603149441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09d01022d0 0x7f09d0108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.666+0000 7f09d69d4640 1 --2- 172.21.15.40:0/2603149441 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d0101910 0x7f09d0101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.666+0000 7f09d69d4640 1 --2- 172.21.15.40:0/2603149441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09d0100720 0x7f09d0100b40 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.666+0000 7f09d69d4640 1 -- 172.21.15.40:0/2603149441 >> 172.21.15.40:0/2603149441 conn(0x7f09d00fbcb0 msgr2=0x7f09d00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:27.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.666+0000 7f09d69d4640 1 -- 172.21.15.40:0/2603149441 shutdown_connections 2024-03-18T22:46:27.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.666+0000 7f09d69d4640 1 -- 172.21.15.40:0/2603149441 wait complete. 2024-03-18T22:46:27.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.667+0000 7f09d69d4640 1 Processor -- start 2024-03-18T22:46:27.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.667+0000 7f09d69d4640 1 -- start start 2024-03-18T22:46:27.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.667+0000 7f09d69d4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d0100720 0x7f09d0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.667+0000 7f09d69d4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09d0101910 0x7f09d0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.667+0000 7f09d69d4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09d01022d0 0x7f09d019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.667+0000 7f09d69d4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f09d010c240 con 0x7f09d0101910 2024-03-18T22:46:27.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.667+0000 7f09d69d4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f09d010c0c0 con 0x7f09d01022d0 2024-03-18T22:46:27.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.667+0000 7f09d69d4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f09d010c3c0 con 0x7f09d0100720 2024-03-18T22:46:27.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.667+0000 7f09cf7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09d0101910 0x7f09d0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.667+0000 7f09cffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d0100720 0x7f09d0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.667+0000 7f09cf7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09d0101910 0x7f09d0198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44354/0 (socket says 172.21.15.40:44354) 2024-03-18T22:46:27.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.667+0000 7f09cf7fe640 1 -- 172.21.15.40:0/1593807538 learned_addr learned my addr 172.21.15.40:0/1593807538 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:27.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.667+0000 7f09d4f71640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09d01022d0 0x7f09d019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.668+0000 7f09cffff640 1 -- 172.21.15.40:0/1593807538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09d01022d0 msgr2=0x7f09d019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.668+0000 7f09cffff640 1 --2- 172.21.15.40:0/1593807538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09d01022d0 0x7f09d019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.668+0000 7f09cffff640 1 -- 172.21.15.40:0/1593807538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09d0101910 msgr2=0x7f09d0198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.668+0000 7f09cffff640 1 --2- 172.21.15.40:0/1593807538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09d0101910 0x7f09d0198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.668+0000 7f09cffff640 1 -- 172.21.15.40:0/1593807538 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d019f6e0 con 0x7f09d0100720 2024-03-18T22:46:27.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.668+0000 7f09d4f71640 1 --2- 172.21.15.40:0/1593807538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09d01022d0 0x7f09d019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:27.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.668+0000 7f09cf7fe640 1 --2- 172.21.15.40:0/1593807538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09d0101910 0x7f09d0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:27.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.668+0000 7f09cffff640 1 --2- 172.21.15.40:0/1593807538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d0100720 0x7f09d0197ac0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f09c0036040 tx=0x7f09c0005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:27.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.668+0000 7f09cd7fa640 1 -- 172.21.15.40:0/1593807538 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0043020 con 0x7f09d0100720 2024-03-18T22:46:27.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.668+0000 7f09d69d4640 1 -- 172.21.15.40:0/1593807538 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09d019f970 con 0x7f09d0100720 2024-03-18T22:46:27.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.668+0000 7f09cd7fa640 1 -- 172.21.15.40:0/1593807538 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09c0004350 con 0x7f09d0100720 2024-03-18T22:46:27.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.668+0000 7f09cd7fa640 1 -- 172.21.15.40:0/1593807538 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0003ab0 con 0x7f09d0100720 2024-03-18T22:46:27.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.668+0000 7f09d69d4640 1 -- 172.21.15.40:0/1593807538 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09d019fe90 con 0x7f09d0100720 2024-03-18T22:46:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.671+0000 7f09cd7fa640 1 -- 172.21.15.40:0/1593807538 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f09c0046020 con 0x7f09d0100720 2024-03-18T22:46:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.671+0000 7f09d95c4740 1 Processor -- start 2024-03-18T22:46:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.671+0000 7f09d95c4740 1 -- start start 2024-03-18T22:46:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.671+0000 7f09cd7fa640 1 --2- 172.21.15.40:0/1593807538 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09b80a2800 0x7f09b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.671+0000 7f09d95c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dca50 0x55c90e5dce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.671+0000 7f09d95c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dd440 0x55c90e5e1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.671+0000 7f09d95c4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5e25b0 0x55c90e5e4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.671+0000 7f09d95c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c90e541500 con 0x55c90e5dd440 2024-03-18T22:46:27.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.671+0000 7f09d95c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c90e4e9be0 con 0x55c90e5dca50 2024-03-18T22:46:27.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.671+0000 7f09d95c4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c90e53d5a0 con 0x55c90e5e25b0 2024-03-18T22:46:27.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.671+0000 7f09d4f71640 1 --2- 172.21.15.40:0/1593807538 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09b80a2800 0x7f09b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.671+0000 7f09cd7fa640 1 -- 172.21.15.40:0/1593807538 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f09c00f0f30 con 0x7f09d0100720 2024-03-18T22:46:27.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.672+0000 7f09cf7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dd440 0x55c90e5e1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.672+0000 7f09cffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dca50 0x55c90e5dce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.672+0000 7f09cf7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dd440 0x55c90e5e1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44364/0 (socket says 172.21.15.40:44364) 2024-03-18T22:46:27.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.672+0000 7f09cf7fe640 1 -- 172.21.15.40:0/3014832801 learned_addr learned my addr 172.21.15.40:0/3014832801 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:27.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.672+0000 7f09cffff640 1 -- 172.21.15.40:0/3014832801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5e25b0 msgr2=0x55c90e5e4a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:46:27.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.672+0000 7f09cffff640 1 --2- 172.21.15.40:0/3014832801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5e25b0 0x55c90e5e4a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.672+0000 7f09cffff640 1 -- 172.21.15.40:0/3014832801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dd440 msgr2=0x55c90e5e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:27.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.672+0000 7f09cffff640 1 --2- 172.21.15.40:0/3014832801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dd440 0x55c90e5e1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.672+0000 7f09cffff640 1 -- 172.21.15.40:0/3014832801 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c90e50b000 con 0x55c90e5dca50 2024-03-18T22:46:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.672+0000 7f09cf7fe640 1 --2- 172.21.15.40:0/3014832801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dd440 0x55c90e5e1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.672+0000 7f09d4f71640 1 --2- 172.21.15.40:0/1593807538 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09b80a2800 0x7f09b80a4cc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f09c8005fd0 tx=0x7f09c8009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.672+0000 7f09cffff640 1 --2- 172.21.15.40:0/3014832801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dca50 0x55c90e5dce70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f09c002c170 tx=0x7f09c00b9120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc82fb6a44879ef7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.672+0000 7f09b67fc640 1 -- 172.21.15.40:0/3014832801 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c00bce40 con 0x55c90e5dca50 2024-03-18T22:46:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.672+0000 7f09b67fc640 1 -- 172.21.15.40:0/3014832801 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09c000b0a0 con 0x55c90e5dca50 2024-03-18T22:46:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.672+0000 7f09b67fc640 1 -- 172.21.15.40:0/3014832801 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c000aec0 con 0x55c90e5dca50 2024-03-18T22:46:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.673+0000 7f09d95c4740 1 -- 172.21.15.40:0/3014832801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dca50 msgr2=0x55c90e5dce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.673+0000 7f09d95c4740 1 --2- 172.21.15.40:0/3014832801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dca50 0x55c90e5dce70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f09c002c170 tx=0x7f09c00b9120 comp rx=0 tx=0).stop 2024-03-18T22:46:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.673+0000 7f09d95c4740 1 -- 172.21.15.40:0/3014832801 shutdown_connections 2024-03-18T22:46:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.673+0000 7f09d95c4740 1 --2- 172.21.15.40:0/3014832801 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5e25b0 0x55c90e5e4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.673+0000 7f09d95c4740 1 --2- 172.21.15.40:0/3014832801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dd440 0x55c90e5e1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.673+0000 7f09d95c4740 1 --2- 172.21.15.40:0/3014832801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dca50 0x55c90e5dce70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.673+0000 7f09d95c4740 1 -- 172.21.15.40:0/3014832801 >> 172.21.15.40:0/3014832801 conn(0x55c90e5d7fc0 msgr2=0x55c90e5d83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.673+0000 7f09d95c4740 1 -- 172.21.15.40:0/3014832801 shutdown_connections 2024-03-18T22:46:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.673+0000 7f09d95c4740 1 -- 172.21.15.40:0/3014832801 wait complete. 2024-03-18T22:46:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.673+0000 7f09d95c4740 1 -- start start 2024-03-18T22:46:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09d95c4740 1 Processor -- start 2024-03-18T22:46:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09d95c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dca50 0x55c90e5e6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09d95c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 0x55c90e5e70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09d95c4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5e25b0 0x55c90e5ee110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09d95c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c90e5e5320 con 0x55c90e5dca50 2024-03-18T22:46:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09d95c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c90e4fd160 con 0x55c90e5dd440 2024-03-18T22:46:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09d95c4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c90e5e54a0 con 0x55c90e5e25b0 2024-03-18T22:46:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09d95c4740 10 client.0 _ll_register_callbacks cb 0x55c90e539a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09cffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dca50 0x55c90e5e6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09cffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dca50 0x55c90e5e6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44378/0 (socket says 172.21.15.40:44378) 2024-03-18T22:46:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09cffff640 1 -- 172.21.15.40:0/1400376134 learned_addr learned my addr 172.21.15.40:0/1400376134 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09cf7fe640 1 --2- 172.21.15.40:0/1400376134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 0x55c90e5e70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09d4f71640 1 --2- 172.21.15.40:0/1400376134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5e25b0 0x55c90e5ee110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09cffff640 1 -- 172.21.15.40:0/1400376134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5e25b0 msgr2=0x55c90e5ee110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09cffff640 1 --2- 172.21.15.40:0/1400376134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5e25b0 0x55c90e5ee110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09cffff640 1 -- 172.21.15.40:0/1400376134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 msgr2=0x55c90e5e70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09cffff640 1 --2- 172.21.15.40:0/1400376134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 0x55c90e5e70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.674+0000 7f09cffff640 1 -- 172.21.15.40:0/1400376134 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c90e31f7d0 con 0x55c90e5dca50 2024-03-18T22:46:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09cffff640 1 --2- 172.21.15.40:0/1400376134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dca50 0x55c90e5e6b90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f09c00bd470 tx=0x7f09c00bced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5756d8252fb70b87 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09af7fe640 1 -- 172.21.15.40:0/1400376134 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c00c1630 con 0x55c90e5dca50 2024-03-18T22:46:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09af7fe640 1 -- 172.21.15.40:0/1400376134 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f09c0003c50 con 0x55c90e5dca50 2024-03-18T22:46:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09d95c4740 1 -- 172.21.15.40:0/1400376134 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c90e54ea40 con 0x55c90e5dca50 2024-03-18T22:46:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09af7fe640 1 -- 172.21.15.40:0/1400376134 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f09c0003e50 con 0x55c90e5dca50 2024-03-18T22:46:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09d95c4740 10 client.16548 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:27.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09d95c4740 1 -- 172.21.15.40:0/1400376134 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c90e2553f0 con 0x55c90e5dca50 2024-03-18T22:46:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09af7fe640 1 -- 172.21.15.40:0/1400376134 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55c90e54ea40 con 0x55c90e5dca50 2024-03-18T22:46:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09af7fe640 1 -- 172.21.15.40:0/1400376134 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f09c00c5880 con 0x55c90e5dca50 2024-03-18T22:46:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09d95c4740 10 client.16548 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09d95c4740 10 client.16548 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:46:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09d95c4740 20 client.16548 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09d95c4740 1 --2- 172.21.15.40:0/1400376134 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c90e5f26f0 0x55c90e5f4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09d95c4740 4 client.16548 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:46:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.675+0000 7f09d95c4740 1 -- 172.21.15.40:0/1400376134 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c90e548d90 con 0x55c90e5f26f0 2024-03-18T22:46:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.676+0000 7f09cf7fe640 1 --2- 172.21.15.40:0/1400376134 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c90e5f26f0 0x55c90e5f4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.676+0000 7f09af7fe640 10 client.16548 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.676+0000 7f09cf7fe640 1 --2- 172.21.15.40:0/1400376134 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c90e5f26f0 0x55c90e5f4b40 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca5b3a6c2d478746 server_cookie=3af70292f1b8b746 in_seq=0 out_seq=0 2024-03-18T22:46:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.677+0000 7f09af7fe640 1 -- 172.21.15.40:0/1400376134 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c90e548d90 con 0x55c90e5f26f0 2024-03-18T22:46:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.677+0000 7f09af7fe640 10 client.16548 handle_command_reply: tid=0 2024-03-18T22:46:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.677+0000 7f09d95c4740 1 client.16548 shutdown 2024-03-18T22:46:27.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.677+0000 7f09d95c4740 1 -- 172.21.15.40:0/1400376134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dca50 msgr2=0x55c90e5e6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.677+0000 7f09d95c4740 1 --2- 172.21.15.40:0/1400376134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dca50 0x55c90e5e6b90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f09c00bd470 tx=0x7f09c00bced0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.677+0000 7f09d95c4740 1 -- 172.21.15.40:0/1400376134 shutdown_connections 2024-03-18T22:46:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.677+0000 7f09d95c4740 1 --2- 172.21.15.40:0/1400376134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5e25b0 0x55c90e5ee110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.677+0000 7f09d95c4740 1 --2- 172.21.15.40:0/1400376134 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c90e5f26f0 0x55c90e5f4b40 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.677+0000 7f09af7fe640 0 client.16548 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.677+0000 7f09d95c4740 1 --2- 172.21.15.40:0/1400376134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 0x55c90e5e70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.677+0000 7f09d95c4740 1 --2- 172.21.15.40:0/1400376134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dca50 0x55c90e5e6b90 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.677+0000 7f09d95c4740 1 -- 172.21.15.40:0/1400376134 >> 172.21.15.40:0/1400376134 conn(0x55c90e5d7fc0 msgr2=0x55c90e5d71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.677+0000 7f09d95c4740 1 -- 172.21.15.40:0/1400376134 shutdown_connections 2024-03-18T22:46:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.677+0000 7f09d95c4740 1 -- 172.21.15.40:0/1400376134 wait complete. 2024-03-18T22:46:27.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.677+0000 7f09d95c4740 20 client.16548 trim_cache size 0 max 16384 2024-03-18T22:46:27.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.696+0000 7f09d95c4740 1 Processor -- start 2024-03-18T22:46:27.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.696+0000 7f09d95c4740 1 -- start start 2024-03-18T22:46:27.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.696+0000 7f09d95c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dca50 0x55c90e5de800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.696+0000 7f09d95c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 0x55c90e5ee0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.696+0000 7f09d95c4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5e25b0 0x55c90e5e6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.696+0000 7f09d95c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01aa570 con 0x55c90e5dca50 2024-03-18T22:46:27.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.696+0000 7f09d95c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01ac0e0 con 0x55c90e5dd440 2024-03-18T22:46:27.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.696+0000 7f09d95c4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01aac40 con 0x55c90e5e25b0 2024-03-18T22:46:27.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.696+0000 7f09cffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dca50 0x55c90e5de800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.696+0000 7f09cffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dca50 0x55c90e5de800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44382/0 (socket says 172.21.15.40:44382) 2024-03-18T22:46:27.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.696+0000 7f09cffff640 1 -- 172.21.15.40:0/3465279436 learned_addr learned my addr 172.21.15.40:0/3465279436 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:27.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.696+0000 7f09cf7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 0x55c90e5ee0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.696+0000 7f09d4f71640 1 --2- 172.21.15.40:0/3465279436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5e25b0 0x55c90e5e6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.697+0000 7f09cf7fe640 1 -- 172.21.15.40:0/3465279436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5e25b0 msgr2=0x55c90e5e6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:27.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.697+0000 7f09cf7fe640 1 --2- 172.21.15.40:0/3465279436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5e25b0 0x55c90e5e6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.697+0000 7f09cf7fe640 1 -- 172.21.15.40:0/3465279436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dca50 msgr2=0x55c90e5de800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:27.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.697+0000 7f09cf7fe640 1 --2- 172.21.15.40:0/3465279436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dca50 0x55c90e5de800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.697+0000 7f09cf7fe640 1 -- 172.21.15.40:0/3465279436 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c90e2150f0 con 0x55c90e5dd440 2024-03-18T22:46:27.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.697+0000 7f09cffff640 1 --2- 172.21.15.40:0/3465279436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dca50 0x55c90e5de800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:27.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.697+0000 7f09d4f71640 1 --2- 172.21.15.40:0/3465279436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5e25b0 0x55c90e5e6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:27.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.697+0000 7f09cf7fe640 1 --2- 172.21.15.40:0/3465279436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 0x55c90e5ee0d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55c90e5f53b0 tx=0x7f09c4005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dfead341e41661a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:27.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.697+0000 7f09b5ffb640 1 -- 172.21.15.40:0/3465279436 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c400f8f0 con 0x55c90e5dd440 2024-03-18T22:46:27.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.697+0000 7f09b5ffb640 1 -- 172.21.15.40:0/3465279436 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09c4026030 con 0x55c90e5dd440 2024-03-18T22:46:27.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.697+0000 7f09b5ffb640 1 -- 172.21.15.40:0/3465279436 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c4004a00 con 0x55c90e5dd440 2024-03-18T22:46:27.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.698+0000 7f09d95c4740 1 -- 172.21.15.40:0/3465279436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 msgr2=0x55c90e5ee0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.698+0000 7f09d95c4740 1 --2- 172.21.15.40:0/3465279436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 0x55c90e5ee0d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55c90e5f53b0 tx=0x7f09c4005f90 comp rx=0 tx=0).stop 2024-03-18T22:46:27.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.698+0000 7f09d95c4740 1 -- 172.21.15.40:0/3465279436 shutdown_connections 2024-03-18T22:46:27.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.698+0000 7f09d95c4740 1 --2- 172.21.15.40:0/3465279436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5e25b0 0x55c90e5e6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.698+0000 7f09d95c4740 1 --2- 172.21.15.40:0/3465279436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 0x55c90e5ee0d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.698+0000 7f09d95c4740 1 --2- 172.21.15.40:0/3465279436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5dca50 0x55c90e5de800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.698+0000 7f09d95c4740 1 -- 172.21.15.40:0/3465279436 >> 172.21.15.40:0/3465279436 conn(0x55c90e5d7fc0 msgr2=0x55c90e5dbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:27.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.698+0000 7f09d95c4740 1 -- 172.21.15.40:0/3465279436 shutdown_connections 2024-03-18T22:46:27.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.698+0000 7f09d95c4740 1 -- 172.21.15.40:0/3465279436 wait complete. 2024-03-18T22:46:27.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.698+0000 7f09d95c4740 1 -- start start 2024-03-18T22:46:27.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.698+0000 7f09d95c4740 1 Processor -- start 2024-03-18T22:46:27.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d95c4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5dca50 0x55c90e5eba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d95c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 0x55c90e5ebf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d95c4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5e25b0 0x55c90e5ecfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d95c4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01ab9e0 con 0x55c90e5e25b0 2024-03-18T22:46:27.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d95c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01ab310 con 0x55c90e5dd440 2024-03-18T22:46:27.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d95c4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f09d01a8550 con 0x55c90e5dca50 2024-03-18T22:46:27.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d95c4740 10 client.0 _ll_register_callbacks cb 0x7f09c00bd0e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:27.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d4f71640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5e25b0 0x55c90e5ecfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d4f71640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5e25b0 0x55c90e5ecfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44388/0 (socket says 172.21.15.40:44388) 2024-03-18T22:46:27.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09cf7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 0x55c90e5ebf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09cffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5dca50 0x55c90e5eba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d4f71640 1 -- 172.21.15.40:0/956415719 learned_addr learned my addr 172.21.15.40:0/956415719 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:27.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d4f71640 1 -- 172.21.15.40:0/956415719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5dca50 msgr2=0x55c90e5eba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:27.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d4f71640 1 --2- 172.21.15.40:0/956415719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5dca50 0x55c90e5eba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d4f71640 1 -- 172.21.15.40:0/956415719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 msgr2=0x55c90e5ebf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:27.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d4f71640 1 --2- 172.21.15.40:0/956415719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 0x55c90e5ebf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d4f71640 1 -- 172.21.15.40:0/956415719 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c90e5ed590 con 0x55c90e5e25b0 2024-03-18T22:46:27.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09cf7fe640 1 --2- 172.21.15.40:0/956415719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 0x55c90e5ebf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:27.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09cffff640 1 --2- 172.21.15.40:0/956415719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5dca50 0x55c90e5eba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:27.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09d4f71640 1 --2- 172.21.15.40:0/956415719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5e25b0 0x55c90e5ecfa0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f09c8003ed0 tx=0x7f09c8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94ad3af09f3e7f11 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:27.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.699+0000 7f09af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:27.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.700+0000 7f09af7fe640 1 -- 172.21.15.40:0/956415719 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c801e600 con 0x55c90e5e25b0 2024-03-18T22:46:27.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.700+0000 7f09af7fe640 1 -- 172.21.15.40:0/956415719 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f09c801e7a0 con 0x55c90e5e25b0 2024-03-18T22:46:27.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.700+0000 7f09d95c4740 1 -- 172.21.15.40:0/956415719 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09c4026030 con 0x55c90e5e25b0 2024-03-18T22:46:27.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.700+0000 7f09af7fe640 1 -- 172.21.15.40:0/956415719 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f09c801e9a0 con 0x55c90e5e25b0 2024-03-18T22:46:27.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.700+0000 7f09d95c4740 10 client.16554 fetch_fsmap learned FSMap version 31 2024-03-18T22:46:27.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.700+0000 7f09d95c4740 1 -- 172.21.15.40:0/956415719 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c90e658d70 con 0x55c90e5e25b0 2024-03-18T22:46:27.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.700+0000 7f09af7fe640 1 -- 172.21.15.40:0/956415719 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09c4026030 con 0x55c90e5e25b0 2024-03-18T22:46:27.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.700+0000 7f09af7fe640 1 -- 172.21.15.40:0/956415719 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f09c8029c90 con 0x55c90e5e25b0 2024-03-18T22:46:27.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.700+0000 7f09d95c4740 10 client.16554 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T22:46:27.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.700+0000 7f09d95c4740 10 client.16554 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:46:27.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.700+0000 7f09d95c4740 20 client.16554 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:27.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.700+0000 7f09d95c4740 1 --2- 172.21.15.40:0/956415719 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c90e5f26f0 0x55c90e65b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:27.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.700+0000 7f09d95c4740 4 client.16554 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:46:27.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.700+0000 7f09d95c4740 1 -- 172.21.15.40:0/956415719 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c90e4c6540 con 0x55c90e5f26f0 2024-03-18T22:46:27.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09cffff640 1 --2- 172.21.15.40:0/956415719 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c90e5f26f0 0x55c90e65b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:27.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09af7fe640 10 client.16554 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:27.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09cffff640 1 --2- 172.21.15.40:0/956415719 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c90e5f26f0 0x55c90e65b890 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=304b753e12cf52c2 server_cookie=f5767e2196cf681a in_seq=0 out_seq=0 2024-03-18T22:46:27.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09af7fe640 1 -- 172.21.15.40:0/956415719 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c90e4c6540 con 0x55c90e5f26f0 2024-03-18T22:46:27.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09af7fe640 10 client.16554 handle_command_reply: tid=0 2024-03-18T22:46:27.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09d95c4740 1 client.16554 shutdown 2024-03-18T22:46:27.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09d95c4740 1 -- 172.21.15.40:0/956415719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5e25b0 msgr2=0x55c90e5ecfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09d95c4740 1 --2- 172.21.15.40:0/956415719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5e25b0 0x55c90e5ecfa0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f09c8003ed0 tx=0x7f09c8004ab0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09d95c4740 1 -- 172.21.15.40:0/956415719 shutdown_connections 2024-03-18T22:46:27.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09d95c4740 1 --2- 172.21.15.40:0/956415719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c90e5e25b0 0x55c90e5ecfa0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09d95c4740 1 --2- 172.21.15.40:0/956415719 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c90e5f26f0 0x55c90e65b890 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09d95c4740 1 --2- 172.21.15.40:0/956415719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c90e5dd440 0x55c90e5ebf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09d95c4740 1 --2- 172.21.15.40:0/956415719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c90e5dca50 0x55c90e5eba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09af7fe640 0 client.16554 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:27.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09d95c4740 1 -- 172.21.15.40:0/956415719 >> 172.21.15.40:0/956415719 conn(0x55c90e5d7fc0 msgr2=0x55c90e5dbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:27.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.701+0000 7f09d95c4740 1 -- 172.21.15.40:0/956415719 shutdown_connections 2024-03-18T22:46:27.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.702+0000 7f09d95c4740 1 -- 172.21.15.40:0/956415719 wait complete. 2024-03-18T22:46:27.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.702+0000 7f09d95c4740 20 client.16554 trim_cache size 0 max 16384 2024-03-18T22:46:27.709 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:46:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:46:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:46:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "6cbeb0ee-8c3c-4662-bf50-44dc9054ed3c": { 2024-03-18T22:46:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:46:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "6cbeb0ee-8c3c-4662-bf50-44dc9054ed3c", 2024-03-18T22:46:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:46:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:46:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:46:27.709 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:46:27.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.702+0000 7f09d69d4640 1 -- 172.21.15.40:0/1593807538 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09b80a2800 msgr2=0x7f09b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.702+0000 7f09d69d4640 1 --2- 172.21.15.40:0/1593807538 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09b80a2800 0x7f09b80a4cc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f09c8005fd0 tx=0x7f09c8009500 comp rx=0 tx=0).stop 2024-03-18T22:46:27.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.702+0000 7f09d69d4640 1 -- 172.21.15.40:0/1593807538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d0100720 msgr2=0x7f09d0197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:27.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.702+0000 7f09d69d4640 1 --2- 172.21.15.40:0/1593807538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d0100720 0x7f09d0197ac0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f09c0036040 tx=0x7f09c0005dc0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.702+0000 7f09d69d4640 1 -- 172.21.15.40:0/1593807538 shutdown_connections 2024-03-18T22:46:27.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.702+0000 7f09d69d4640 1 --2- 172.21.15.40:0/1593807538 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09b80a2800 0x7f09b80a4cc0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.702+0000 7f09d69d4640 1 --2- 172.21.15.40:0/1593807538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09d01022d0 0x7f09d019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.702+0000 7f09d69d4640 1 --2- 172.21.15.40:0/1593807538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09d0101910 0x7f09d0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.702+0000 7f09d69d4640 1 --2- 172.21.15.40:0/1593807538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f09d0100720 0x7f09d0197ac0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:27.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.702+0000 7f09d69d4640 1 -- 172.21.15.40:0/1593807538 >> 172.21.15.40:0/1593807538 conn(0x7f09d00fbcb0 msgr2=0x7f09d00689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:27.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.702+0000 7f09d69d4640 1 -- 172.21.15.40:0/1593807538 shutdown_connections 2024-03-18T22:46:27.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:27.702+0000 7f09d69d4640 1 -- 172.21.15.40:0/1593807538 wait complete. 2024-03-18T22:46:27.715 INFO:tasks.cephfs.filesystem:scrub status for tag:6cbeb0ee-8c3c-4662-bf50-44dc9054ed3c - {'path': '/', 'tag': '6cbeb0ee-8c3c-4662-bf50-44dc9054ed3c', 'options': 'recursive,force'} 2024-03-18T22:46:28.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:28 smithi113 ceph-mon[66775]: pgmap v465: 129 pgs: 129 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 20 op/s 2024-03-18T22:46:28.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:28 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:46:28.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:28 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:46:28.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:28 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:46:28.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:28 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:46:28.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:28 smithi103 ceph-mon[66377]: pgmap v465: 129 pgs: 129 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 20 op/s 2024-03-18T22:46:28.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:28 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:46:28.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:28 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:46:28.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:28 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:46:28.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:28 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:46:28.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:28 smithi040 ceph-mon[62643]: pgmap v465: 129 pgs: 129 active+clean; 3.7 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 20 op/s 2024-03-18T22:46:28.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:28 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:46:28.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:28 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:46:28.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:28 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:46:28.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:28 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:46:29.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:29 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:46:29.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:29 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:46:29.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:29 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:46:29.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:29 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:46:29.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:29 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:46:29.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:29 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:46:29.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:29 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:46:29.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:29 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:46:29.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:29 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:46:29.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:29 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:46:29.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:29 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:46:29.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:29 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:46:29.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:29 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:46:29.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:29 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:46:29.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:29 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:46:29.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:29 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:46:29.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:29 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:46:29.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:29 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:46:30.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:30 smithi113 ceph-mon[66775]: pgmap v466: 129 pgs: 129 active+clean; 3.7 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-03-18T22:46:30.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:30 smithi103 ceph-mon[66377]: pgmap v466: 129 pgs: 129 active+clean; 3.7 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-03-18T22:46:30.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:30 smithi040 ceph-mon[62643]: pgmap v466: 129 pgs: 129 active+clean; 3.7 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 25 MiB/s wr, 19 op/s 2024-03-18T22:46:32.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:32 smithi113 ceph-mon[66775]: pgmap v467: 129 pgs: 129 active+clean; 3.8 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:46:32.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:32 smithi103 ceph-mon[66377]: pgmap v467: 129 pgs: 129 active+clean; 3.8 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:46:32.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:32 smithi040 ceph-mon[62643]: pgmap v467: 129 pgs: 129 active+clean; 3.8 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:46:33.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:33 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:46:33.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:33 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:46:33.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:33 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:46:34.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:34 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:46:34.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:34 smithi113 ceph-mon[66775]: pgmap v468: 129 pgs: 129 active+clean; 3.9 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 21 op/s 2024-03-18T22:46:34.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:34 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:46:34.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:34 smithi103 ceph-mon[66377]: pgmap v468: 129 pgs: 129 active+clean; 3.9 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 21 op/s 2024-03-18T22:46:34.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:34 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:46:34.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:34 smithi040 ceph-mon[62643]: pgmap v468: 129 pgs: 129 active+clean; 3.9 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 21 op/s 2024-03-18T22:46:36.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:36 smithi113 ceph-mon[66775]: pgmap v469: 129 pgs: 129 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:46:36.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:36 smithi103 ceph-mon[66377]: pgmap v469: 129 pgs: 129 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:46:36.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:36 smithi040 ceph-mon[62643]: pgmap v469: 129 pgs: 129 active+clean; 3.9 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:46:38.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:38 smithi113 ceph-mon[66775]: pgmap v470: 129 pgs: 129 active+clean; 4.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:46:38.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:38 smithi103 ceph-mon[66377]: pgmap v470: 129 pgs: 129 active+clean; 4.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:46:38.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:38 smithi040 ceph-mon[62643]: pgmap v470: 129 pgs: 129 active+clean; 4.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:46:40.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:40 smithi113 ceph-mon[66775]: pgmap v471: 129 pgs: 129 active+clean; 4.0 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:46:40.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:40 smithi103 ceph-mon[66377]: pgmap v471: 129 pgs: 129 active+clean; 4.0 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:46:40.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:40 smithi040 ceph-mon[62643]: pgmap v471: 129 pgs: 129 active+clean; 4.0 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:46:42.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:42 smithi113 ceph-mon[66775]: pgmap v472: 129 pgs: 129 active+clean; 4.1 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:46:42.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:42 smithi103 ceph-mon[66377]: pgmap v472: 129 pgs: 129 active+clean; 4.1 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:46:42.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:42 smithi040 ceph-mon[62643]: pgmap v472: 129 pgs: 129 active+clean; 4.1 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:46:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:44 smithi103 ceph-mon[66377]: pgmap v473: 129 pgs: 129 active+clean; 4.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:46:44.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:44 smithi040 ceph-mon[62643]: pgmap v473: 129 pgs: 129 active+clean; 4.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:46:44.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:44 smithi113 ceph-mon[66775]: pgmap v473: 129 pgs: 129 active+clean; 4.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:46:46.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:46 smithi103 ceph-mon[66377]: pgmap v474: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:46:46.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:46 smithi040 ceph-mon[62643]: pgmap v474: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:46:46.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:46 smithi113 ceph-mon[66775]: pgmap v474: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:46:48.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:48 smithi103 ceph-mon[66377]: pgmap v475: 129 pgs: 129 active+clean; 4.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 31 MiB/s wr, 21 op/s 2024-03-18T22:46:48.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:48 smithi040 ceph-mon[62643]: pgmap v475: 129 pgs: 129 active+clean; 4.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 31 MiB/s wr, 21 op/s 2024-03-18T22:46:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:48 smithi113 ceph-mon[66775]: pgmap v475: 129 pgs: 129 active+clean; 4.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 31 MiB/s wr, 21 op/s 2024-03-18T22:46:50.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:50 smithi103 ceph-mon[66377]: pgmap v476: 129 pgs: 129 active+clean; 4.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:46:50.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:50 smithi040 ceph-mon[62643]: pgmap v476: 129 pgs: 129 active+clean; 4.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:46:50.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:50 smithi113 ceph-mon[66775]: pgmap v476: 129 pgs: 129 active+clean; 4.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:46:52.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:52 smithi103 ceph-mon[66377]: pgmap v477: 129 pgs: 129 active+clean; 4.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:46:52.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:52 smithi040 ceph-mon[62643]: pgmap v477: 129 pgs: 129 active+clean; 4.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:46:52.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:52 smithi113 ceph-mon[66775]: pgmap v477: 129 pgs: 129 active+clean; 4.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:46:54.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:54 smithi103 ceph-mon[66377]: pgmap v478: 129 pgs: 129 active+clean; 4.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 31 MiB/s wr, 22 op/s 2024-03-18T22:46:54.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:54 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:46:54.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:54 smithi040 ceph-mon[62643]: pgmap v478: 129 pgs: 129 active+clean; 4.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 31 MiB/s wr, 22 op/s 2024-03-18T22:46:54.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:54 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:46:54.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:54 smithi113 ceph-mon[66775]: pgmap v478: 129 pgs: 129 active+clean; 4.4 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 31 MiB/s wr, 22 op/s 2024-03-18T22:46:54.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:54 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:46:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:56 smithi103 ceph-mon[66377]: pgmap v479: 129 pgs: 129 active+clean; 4.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:46:56.382 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:56 smithi113 ceph-mon[66775]: pgmap v479: 129 pgs: 129 active+clean; 4.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:46:56.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:56 smithi040 ceph-mon[62643]: pgmap v479: 129 pgs: 129 active+clean; 4.4 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:46:57.716 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:46:57.716 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:46:57.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.834+0000 7fe08c3d3640 1 -- 172.21.15.40:0/4217677805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe084101910 msgr2=0x7fe084101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:57.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.834+0000 7fe08c3d3640 1 --2- 172.21.15.40:0/4217677805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe084101910 0x7fe084101d90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe074009900 tx=0x7fe07402bc30 comp rx=0 tx=0).stop 2024-03-18T22:46:57.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.834+0000 7fe08c3d3640 1 -- 172.21.15.40:0/4217677805 shutdown_connections 2024-03-18T22:46:57.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.834+0000 7fe08c3d3640 1 --2- 172.21.15.40:0/4217677805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0841022d0 0x7fe084108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.834+0000 7fe08c3d3640 1 --2- 172.21.15.40:0/4217677805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe084101910 0x7fe084101d90 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.834+0000 7fe08c3d3640 1 --2- 172.21.15.40:0/4217677805 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe084100720 0x7fe084100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.834+0000 7fe08c3d3640 1 -- 172.21.15.40:0/4217677805 >> 172.21.15.40:0/4217677805 conn(0x7fe0840fbcb0 msgr2=0x7fe0840fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:57.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.834+0000 7fe08c3d3640 1 -- 172.21.15.40:0/4217677805 shutdown_connections 2024-03-18T22:46:57.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.834+0000 7fe08c3d3640 1 -- 172.21.15.40:0/4217677805 wait complete. 2024-03-18T22:46:57.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.834+0000 7fe08c3d3640 1 Processor -- start 2024-03-18T22:46:57.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.834+0000 7fe08c3d3640 1 -- start start 2024-03-18T22:46:57.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08c3d3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe084100720 0x7fe084197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08c3d3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe084101910 0x7fe084198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08c3d3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0841022d0 0x7fe08419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08c3d3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe08410c250 con 0x7fe0841022d0 2024-03-18T22:46:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08c3d3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe08410c0d0 con 0x7fe084100720 2024-03-18T22:46:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08c3d3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe08410c3d0 con 0x7fe084101910 2024-03-18T22:46:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08a970640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0841022d0 0x7fe08419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08a16f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe084100720 0x7fe084197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08a970640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0841022d0 0x7fe08419f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48844/0 (socket says 172.21.15.40:48844) 2024-03-18T22:46:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08a16f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe084100720 0x7fe084197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:59692/0 (socket says 172.21.15.40:59692) 2024-03-18T22:46:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08a970640 1 -- 172.21.15.40:0/3210594478 learned_addr learned my addr 172.21.15.40:0/3210594478 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08996e640 1 --2- 172.21.15.40:0/3210594478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe084101910 0x7fe084198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08a16f640 1 -- 172.21.15.40:0/3210594478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe084101910 msgr2=0x7fe084198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08a16f640 1 --2- 172.21.15.40:0/3210594478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe084101910 0x7fe084198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08a16f640 1 -- 172.21.15.40:0/3210594478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0841022d0 msgr2=0x7fe08419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08a16f640 1 --2- 172.21.15.40:0/3210594478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0841022d0 0x7fe08419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08a16f640 1 -- 172.21.15.40:0/3210594478 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe08419f6f0 con 0x7fe084100720 2024-03-18T22:46:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08996e640 1 --2- 172.21.15.40:0/3210594478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe084101910 0x7fe084198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08a970640 1 --2- 172.21.15.40:0/3210594478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0841022d0 0x7fe08419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.835+0000 7fe08a16f640 1 --2- 172.21.15.40:0/3210594478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe084100720 0x7fe084197ad0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe08000e9c0 tx=0x7fe08000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.836+0000 7fe07b7fe640 1 -- 172.21.15.40:0/3210594478 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08000ccc0 con 0x7fe084100720 2024-03-18T22:46:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.836+0000 7fe07b7fe640 1 -- 172.21.15.40:0/3210594478 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe0800035f0 con 0x7fe084100720 2024-03-18T22:46:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.836+0000 7fe08c3d3640 1 -- 172.21.15.40:0/3210594478 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe08419f9e0 con 0x7fe084100720 2024-03-18T22:46:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.836+0000 7fe07b7fe640 1 -- 172.21.15.40:0/3210594478 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08000ce60 con 0x7fe084100720 2024-03-18T22:46:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.836+0000 7fe08c3d3640 1 -- 172.21.15.40:0/3210594478 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe08419feb0 con 0x7fe084100720 2024-03-18T22:46:57.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.838+0000 7fe07b7fe640 1 -- 172.21.15.40:0/3210594478 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe0800037a0 con 0x7fe084100720 2024-03-18T22:46:57.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.839+0000 7fe08efb6740 1 Processor -- start 2024-03-18T22:46:57.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.839+0000 7fe08efb6740 1 -- start start 2024-03-18T22:46:57.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.839+0000 7fe08efb6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf776a50 0x55bdaf776e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.839+0000 7fe08efb6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 0x55bdaf77bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.839+0000 7fe08efb6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf77c5b0 0x55bdaf77ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.839+0000 7fe08efb6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bdaf6db500 con 0x55bdaf776a50 2024-03-18T22:46:57.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.839+0000 7fe08efb6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bdaf683be0 con 0x55bdaf77c5b0 2024-03-18T22:46:57.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.839+0000 7fe08efb6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55bdaf6d75a0 con 0x55bdaf777440 2024-03-18T22:46:57.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.839+0000 7fe08a970640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf776a50 0x55bdaf776e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.839+0000 7fe07b7fe640 1 --2- 172.21.15.40:0/3210594478 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe0700a2800 0x7fe0700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.839+0000 7fe08a970640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf776a50 0x55bdaf776e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48854/0 (socket says 172.21.15.40:48854) 2024-03-18T22:46:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.839+0000 7fe08a970640 1 -- 172.21.15.40:0/1645553312 learned_addr learned my addr 172.21.15.40:0/1645553312 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.839+0000 7fe08a16f640 1 --2- 172.21.15.40:0/1645553312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 0x55bdaf77bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.839+0000 7fe08996e640 1 --2- 172.21.15.40:0/1645553312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf77c5b0 0x55bdaf77ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.840+0000 7fe08a970640 1 --2- 172.21.15.40:0/3210594478 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe0700a2800 0x7fe0700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.840+0000 7fe08a970640 1 -- 172.21.15.40:0/1645553312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 msgr2=0x55bdaf77bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.840+0000 7fe07b7fe640 1 -- 172.21.15.40:0/3210594478 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe0800cf1d0 con 0x7fe084100720 2024-03-18T22:46:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.840+0000 7fe08a970640 1 --2- 172.21.15.40:0/1645553312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 0x55bdaf77bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.840+0000 7fe08a970640 1 -- 172.21.15.40:0/1645553312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf77c5b0 msgr2=0x55bdaf77ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.840+0000 7fe08a970640 1 --2- 172.21.15.40:0/1645553312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf77c5b0 0x55bdaf77ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.840+0000 7fe08a970640 1 -- 172.21.15.40:0/1645553312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdaf6a5000 con 0x55bdaf776a50 2024-03-18T22:46:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.840+0000 7fe08a16f640 1 --2- 172.21.15.40:0/1645553312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 0x55bdaf77bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.840+0000 7fe08a970640 1 --2- 172.21.15.40:0/1645553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf776a50 0x55bdaf776e70 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00ff20 tx=0x7fe07c0138f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35945ecbe966853a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.840+0000 7fe08a970640 1 --2- 172.21.15.40:0/3210594478 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe0700a2800 0x7fe0700a4cc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe07c013df0 tx=0x7fe07c020000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.840+0000 7fe078bfe640 1 -- 172.21.15.40:0/1645553312 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c00f470 con 0x55bdaf776a50 2024-03-18T22:46:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.840+0000 7fe078bfe640 1 -- 172.21.15.40:0/1645553312 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe07c023030 con 0x55bdaf776a50 2024-03-18T22:46:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.840+0000 7fe078bfe640 1 -- 172.21.15.40:0/1645553312 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c010690 con 0x55bdaf776a50 2024-03-18T22:46:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.841+0000 7fe08efb6740 1 -- 172.21.15.40:0/1645553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf776a50 msgr2=0x55bdaf776e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.841+0000 7fe08efb6740 1 --2- 172.21.15.40:0/1645553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf776a50 0x55bdaf776e70 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00ff20 tx=0x7fe07c0138f0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.841+0000 7fe08efb6740 1 -- 172.21.15.40:0/1645553312 shutdown_connections 2024-03-18T22:46:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.841+0000 7fe08efb6740 1 --2- 172.21.15.40:0/1645553312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf77c5b0 0x55bdaf77ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.841+0000 7fe08efb6740 1 --2- 172.21.15.40:0/1645553312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 0x55bdaf77bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.841+0000 7fe08efb6740 1 --2- 172.21.15.40:0/1645553312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf776a50 0x55bdaf776e70 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.841+0000 7fe08efb6740 1 -- 172.21.15.40:0/1645553312 >> 172.21.15.40:0/1645553312 conn(0x55bdaf771fc0 msgr2=0x55bdaf7723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.841+0000 7fe08efb6740 1 -- 172.21.15.40:0/1645553312 shutdown_connections 2024-03-18T22:46:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.841+0000 7fe08efb6740 1 -- 172.21.15.40:0/1645553312 wait complete. 2024-03-18T22:46:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08efb6740 1 -- start start 2024-03-18T22:46:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08efb6740 1 Processor -- start 2024-03-18T22:46:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08efb6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf776a50 0x55bdaf780ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08efb6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 0x55bdaf781400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08efb6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf77c5b0 0x55bdaf7883d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08efb6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bdaf77f500 con 0x55bdaf776a50 2024-03-18T22:46:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08efb6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bdaf697160 con 0x55bdaf77c5b0 2024-03-18T22:46:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08efb6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55bdaf77f680 con 0x55bdaf777440 2024-03-18T22:46:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08efb6740 10 client.0 _ll_register_callbacks cb 0x55bdaf6d3a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08a16f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf776a50 0x55bdaf780ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08996e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 0x55bdaf781400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08996e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 0x55bdaf781400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:39056/0 (socket says 172.21.15.40:39056) 2024-03-18T22:46:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08996e640 1 -- 172.21.15.40:0/3655910829 learned_addr learned my addr 172.21.15.40:0/3655910829 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08a970640 1 --2- 172.21.15.40:0/3655910829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf77c5b0 0x55bdaf7883d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08a16f640 1 -- 172.21.15.40:0/3655910829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 msgr2=0x55bdaf781400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08a16f640 1 --2- 172.21.15.40:0/3655910829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 0x55bdaf781400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.842+0000 7fe08a16f640 1 -- 172.21.15.40:0/3655910829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf77c5b0 msgr2=0x55bdaf7883d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.843+0000 7fe08a16f640 1 --2- 172.21.15.40:0/3655910829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf77c5b0 0x55bdaf7883d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.843+0000 7fe08a16f640 1 -- 172.21.15.40:0/3655910829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdaf4b97d0 con 0x55bdaf776a50 2024-03-18T22:46:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.843+0000 7fe08a16f640 1 --2- 172.21.15.40:0/3655910829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf776a50 0x55bdaf780ec0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe0800a39f0 tx=0x7fe08009b380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf82cf80ac2b4a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.843+0000 7fe06dffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.843+0000 7fe06dffb640 1 -- 172.21.15.40:0/3655910829 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08009be80 con 0x55bdaf776a50 2024-03-18T22:46:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.843+0000 7fe06dffb640 1 -- 172.21.15.40:0/3655910829 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe0800a3a90 con 0x55bdaf776a50 2024-03-18T22:46:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.843+0000 7fe08efb6740 1 -- 172.21.15.40:0/3655910829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe07c023030 con 0x55bdaf776a50 2024-03-18T22:46:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.843+0000 7fe06dffb640 1 -- 172.21.15.40:0/3655910829 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe08009b780 con 0x55bdaf776a50 2024-03-18T22:46:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.843+0000 7fe08efb6740 10 client.16566 fetch_fsmap learned FSMap version 32 2024-03-18T22:46:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.843+0000 7fe08efb6740 1 -- 172.21.15.40:0/3655910829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bdaf3ef3f0 con 0x55bdaf776a50 2024-03-18T22:46:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.843+0000 7fe06dffb640 1 -- 172.21.15.40:0/3655910829 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe07c023030 con 0x55bdaf776a50 2024-03-18T22:46:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.843+0000 7fe06dffb640 1 -- 172.21.15.40:0/3655910829 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe0800afb60 con 0x55bdaf776a50 2024-03-18T22:46:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.843+0000 7fe08efb6740 10 client.16566 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:46:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.843+0000 7fe08efb6740 10 client.16566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:46:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.843+0000 7fe08efb6740 20 client.16566 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.844+0000 7fe08efb6740 1 --2- 172.21.15.40:0/3655910829 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdaf78cd00 0x55bdaf78f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.844+0000 7fe08efb6740 4 client.16566 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:46:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.844+0000 7fe08efb6740 1 -- 172.21.15.40:0/3655910829 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdaf6e8a40 con 0x55bdaf78cd00 2024-03-18T22:46:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.844+0000 7fe08996e640 1 --2- 172.21.15.40:0/3655910829 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdaf78cd00 0x55bdaf78f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.844+0000 7fe08996e640 1 --2- 172.21.15.40:0/3655910829 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdaf78cd00 0x55bdaf78f170 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=349601ed7659d064 server_cookie=f58d3c85c0cfdebf in_seq=0 out_seq=0 2024-03-18T22:46:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.844+0000 7fe06dffb640 10 client.16566 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:57.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.845+0000 7fe06dffb640 1 -- 172.21.15.40:0/3655910829 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bdaf6e8a40 con 0x55bdaf78cd00 2024-03-18T22:46:57.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.845+0000 7fe06dffb640 10 client.16566 handle_command_reply: tid=0 2024-03-18T22:46:57.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.845+0000 7fe08efb6740 1 client.16566 shutdown 2024-03-18T22:46:57.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.845+0000 7fe08efb6740 1 -- 172.21.15.40:0/3655910829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf776a50 msgr2=0x55bdaf780ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:57.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.845+0000 7fe08efb6740 1 --2- 172.21.15.40:0/3655910829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf776a50 0x55bdaf780ec0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe0800a39f0 tx=0x7fe08009b380 comp rx=0 tx=0).stop 2024-03-18T22:46:57.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.845+0000 7fe08efb6740 1 -- 172.21.15.40:0/3655910829 shutdown_connections 2024-03-18T22:46:57.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.845+0000 7fe08efb6740 1 --2- 172.21.15.40:0/3655910829 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdaf78cd00 0x55bdaf78f170 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.845+0000 7fe08efb6740 1 --2- 172.21.15.40:0/3655910829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf77c5b0 0x55bdaf7883d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.845+0000 7fe08efb6740 1 --2- 172.21.15.40:0/3655910829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 0x55bdaf781400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.845+0000 7fe08efb6740 1 --2- 172.21.15.40:0/3655910829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf776a50 0x55bdaf780ec0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.845+0000 7fe08efb6740 1 -- 172.21.15.40:0/3655910829 >> 172.21.15.40:0/3655910829 conn(0x55bdaf771fc0 msgr2=0x55bdaf7711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:57.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.845+0000 7fe06dffb640 0 client.16566 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:57.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.845+0000 7fe08efb6740 1 -- 172.21.15.40:0/3655910829 shutdown_connections 2024-03-18T22:46:57.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.845+0000 7fe08efb6740 1 -- 172.21.15.40:0/3655910829 wait complete. 2024-03-18T22:46:57.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.846+0000 7fe08efb6740 20 client.16566 trim_cache size 0 max 16384 2024-03-18T22:46:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.866+0000 7fe08efb6740 1 Processor -- start 2024-03-18T22:46:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.866+0000 7fe08efb6740 1 -- start start 2024-03-18T22:46:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.866+0000 7fe08efb6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf776a50 0x55bdaf78ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.866+0000 7fe08efb6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf777440 0x55bdaf788390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.866+0000 7fe08efb6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf77c5b0 0x55bdaf7811e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.866+0000 7fe08efb6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe0841aa560 con 0x55bdaf777440 2024-03-18T22:46:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.866+0000 7fe08efb6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe0841ac0a0 con 0x55bdaf776a50 2024-03-18T22:46:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.866+0000 7fe08efb6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe0841aac30 con 0x55bdaf77c5b0 2024-03-18T22:46:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.866+0000 7fe08996e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf777440 0x55bdaf788390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.866+0000 7fe08996e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf777440 0x55bdaf788390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48870/0 (socket says 172.21.15.40:48870) 2024-03-18T22:46:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.866+0000 7fe08996e640 1 -- 172.21.15.40:0/2758188306 learned_addr learned my addr 172.21.15.40:0/2758188306 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:57.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.866+0000 7fe08a16f640 1 --2- 172.21.15.40:0/2758188306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf776a50 0x55bdaf78ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.866+0000 7fe08a970640 1 --2- 172.21.15.40:0/2758188306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf77c5b0 0x55bdaf7811e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.867+0000 7fe08a16f640 1 -- 172.21.15.40:0/2758188306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf77c5b0 msgr2=0x55bdaf7811e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.867+0000 7fe08a16f640 1 --2- 172.21.15.40:0/2758188306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf77c5b0 0x55bdaf7811e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.867+0000 7fe08a16f640 1 -- 172.21.15.40:0/2758188306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf777440 msgr2=0x55bdaf788390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.867+0000 7fe08a16f640 1 --2- 172.21.15.40:0/2758188306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf777440 0x55bdaf788390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.867+0000 7fe08a16f640 1 -- 172.21.15.40:0/2758188306 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdaf78f4e0 con 0x55bdaf776a50 2024-03-18T22:46:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.867+0000 7fe08996e640 1 --2- 172.21.15.40:0/2758188306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf777440 0x55bdaf788390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.867+0000 7fe08a16f640 1 --2- 172.21.15.40:0/2758188306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf776a50 0x55bdaf78ed10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe080098150 tx=0x7fe080003e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f0a09eb3bfad27a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.867+0000 7fe06ffff640 1 -- 172.21.15.40:0/2758188306 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08009be80 con 0x55bdaf776a50 2024-03-18T22:46:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.867+0000 7fe06ffff640 1 -- 172.21.15.40:0/2758188306 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe08009f8d0 con 0x55bdaf776a50 2024-03-18T22:46:57.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.867+0000 7fe06ffff640 1 -- 172.21.15.40:0/2758188306 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0800a3cd0 con 0x55bdaf776a50 2024-03-18T22:46:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.868+0000 7fe08efb6740 1 -- 172.21.15.40:0/2758188306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf776a50 msgr2=0x55bdaf78ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.868+0000 7fe08efb6740 1 --2- 172.21.15.40:0/2758188306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf776a50 0x55bdaf78ed10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe080098150 tx=0x7fe080003e10 comp rx=0 tx=0).stop 2024-03-18T22:46:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.868+0000 7fe08efb6740 1 -- 172.21.15.40:0/2758188306 shutdown_connections 2024-03-18T22:46:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.868+0000 7fe08efb6740 1 --2- 172.21.15.40:0/2758188306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf77c5b0 0x55bdaf7811e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.868+0000 7fe08efb6740 1 --2- 172.21.15.40:0/2758188306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf777440 0x55bdaf788390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.868+0000 7fe08efb6740 1 --2- 172.21.15.40:0/2758188306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf776a50 0x55bdaf78ed10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.868+0000 7fe08efb6740 1 -- 172.21.15.40:0/2758188306 >> 172.21.15.40:0/2758188306 conn(0x55bdaf771fc0 msgr2=0x55bdaf780ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.868+0000 7fe08efb6740 1 -- 172.21.15.40:0/2758188306 shutdown_connections 2024-03-18T22:46:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.868+0000 7fe08efb6740 1 -- 172.21.15.40:0/2758188306 wait complete. 2024-03-18T22:46:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.868+0000 7fe08efb6740 1 -- start start 2024-03-18T22:46:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.868+0000 7fe08efb6740 1 Processor -- start 2024-03-18T22:46:57.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08efb6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf776a50 0x55bdaf785820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08efb6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 0x55bdaf785d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08efb6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf77c5b0 0x55bdaf786d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08efb6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe0841ab9d0 con 0x55bdaf77c5b0 2024-03-18T22:46:57.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08efb6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe0841ab300 con 0x55bdaf776a50 2024-03-18T22:46:57.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08efb6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe0841a8540 con 0x55bdaf777440 2024-03-18T22:46:57.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08efb6740 10 client.0 _ll_register_callbacks cb 0x7fe080004540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:57.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08a16f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf776a50 0x55bdaf785820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08a970640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf77c5b0 0x55bdaf786d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08a16f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf776a50 0x55bdaf785820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:59732/0 (socket says 172.21.15.40:59732) 2024-03-18T22:46:57.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08a16f640 1 -- 172.21.15.40:0/2836396800 learned_addr learned my addr 172.21.15.40:0/2836396800 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:57.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08996e640 1 --2- 172.21.15.40:0/2836396800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 0x55bdaf785d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08a16f640 1 -- 172.21.15.40:0/2836396800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 msgr2=0x55bdaf785d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:57.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08a16f640 1 --2- 172.21.15.40:0/2836396800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 0x55bdaf785d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08a16f640 1 -- 172.21.15.40:0/2836396800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf77c5b0 msgr2=0x55bdaf786d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08a16f640 1 --2- 172.21.15.40:0/2836396800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf77c5b0 0x55bdaf786d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08a16f640 1 -- 172.21.15.40:0/2836396800 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdaf3af0f0 con 0x55bdaf776a50 2024-03-18T22:46:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08996e640 1 --2- 172.21.15.40:0/2836396800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 0x55bdaf785d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08a970640 1 --2- 172.21.15.40:0/2836396800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf77c5b0 0x55bdaf786d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08a16f640 1 --2- 172.21.15.40:0/2836396800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf776a50 0x55bdaf785820 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe080097b90 tx=0x7fe0800a3ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe89f7e62dc0735c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe06dffb640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:46:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe06dffb640 1 -- 172.21.15.40:0/2836396800 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0800a3f20 con 0x55bdaf776a50 2024-03-18T22:46:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe06dffb640 1 -- 172.21.15.40:0/2836396800 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe080004000 con 0x55bdaf776a50 2024-03-18T22:46:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.869+0000 7fe08efb6740 1 -- 172.21.15.40:0/2836396800 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe08009f8d0 con 0x55bdaf776a50 2024-03-18T22:46:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.870+0000 7fe06dffb640 1 -- 172.21.15.40:0/2836396800 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe08009be80 con 0x55bdaf776a50 2024-03-18T22:46:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.870+0000 7fe08efb6740 10 client.26405 fetch_fsmap learned FSMap version 32 2024-03-18T22:46:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.870+0000 7fe08efb6740 1 -- 172.21.15.40:0/2836396800 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bdaf7884a0 con 0x55bdaf776a50 2024-03-18T22:46:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.870+0000 7fe06dffb640 1 -- 172.21.15.40:0/2836396800 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe08009f8d0 con 0x55bdaf776a50 2024-03-18T22:46:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.870+0000 7fe06dffb640 1 -- 172.21.15.40:0/2836396800 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe080097050 con 0x55bdaf776a50 2024-03-18T22:46:57.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.870+0000 7fe08efb6740 10 client.26405 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:46:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.870+0000 7fe08efb6740 10 client.26405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:46:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.870+0000 7fe08efb6740 20 client.26405 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.870+0000 7fe08efb6740 1 --2- 172.21.15.40:0/2836396800 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdaf7f32e0 0x55bdaf7f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.870+0000 7fe08efb6740 4 client.26405 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:46:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.870+0000 7fe08efb6740 1 -- 172.21.15.40:0/2836396800 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bdaf6e2d90 con 0x55bdaf7f32e0 2024-03-18T22:46:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.870+0000 7fe08996e640 1 --2- 172.21.15.40:0/2836396800 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdaf7f32e0 0x55bdaf7f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe06dffb640 10 client.26405 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08996e640 1 --2- 172.21.15.40:0/2836396800 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdaf7f32e0 0x55bdaf7f56d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5b97c6352485b64 server_cookie=9aa8d52719e33e1b in_seq=0 out_seq=0 2024-03-18T22:46:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe06dffb640 1 -- 172.21.15.40:0/2836396800 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bdaf6e2d90 con 0x55bdaf7f32e0 2024-03-18T22:46:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe06dffb640 10 client.26405 handle_command_reply: tid=0 2024-03-18T22:46:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08efb6740 1 client.26405 shutdown 2024-03-18T22:46:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08efb6740 1 -- 172.21.15.40:0/2836396800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf776a50 msgr2=0x55bdaf785820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08efb6740 1 --2- 172.21.15.40:0/2836396800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf776a50 0x55bdaf785820 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe080097b90 tx=0x7fe0800a3ca0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08efb6740 1 -- 172.21.15.40:0/2836396800 shutdown_connections 2024-03-18T22:46:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08efb6740 1 --2- 172.21.15.40:0/2836396800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdaf77c5b0 0x55bdaf786d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08efb6740 1 --2- 172.21.15.40:0/2836396800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdaf777440 0x55bdaf785d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08efb6740 1 --2- 172.21.15.40:0/2836396800 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdaf7f32e0 0x55bdaf7f56d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08efb6740 1 --2- 172.21.15.40:0/2836396800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdaf776a50 0x55bdaf785820 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe06dffb640 0 client.26405 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08efb6740 1 -- 172.21.15.40:0/2836396800 >> 172.21.15.40:0/2836396800 conn(0x55bdaf771fc0 msgr2=0x55bdaf775220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08efb6740 1 -- 172.21.15.40:0/2836396800 shutdown_connections 2024-03-18T22:46:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08efb6740 1 -- 172.21.15.40:0/2836396800 wait complete. 2024-03-18T22:46:57.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08efb6740 20 client.26405 trim_cache size 0 max 16384 2024-03-18T22:46:57.879 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:46:57.880 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:46:57.880 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:46:57.880 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:46:57.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08c3d3640 1 -- 172.21.15.40:0/3210594478 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe0700a2800 msgr2=0x7fe0700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:57.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08c3d3640 1 --2- 172.21.15.40:0/3210594478 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe0700a2800 0x7fe0700a4cc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe07c013df0 tx=0x7fe07c020000 comp rx=0 tx=0).stop 2024-03-18T22:46:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.871+0000 7fe08c3d3640 1 -- 172.21.15.40:0/3210594478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe084100720 msgr2=0x7fe084197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.872+0000 7fe08c3d3640 1 --2- 172.21.15.40:0/3210594478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe084100720 0x7fe084197ad0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe08000e9c0 tx=0x7fe08000ee80 comp rx=0 tx=0).stop 2024-03-18T22:46:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.872+0000 7fe08c3d3640 1 -- 172.21.15.40:0/3210594478 shutdown_connections 2024-03-18T22:46:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.872+0000 7fe08c3d3640 1 --2- 172.21.15.40:0/3210594478 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe0700a2800 0x7fe0700a4cc0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.872+0000 7fe08c3d3640 1 --2- 172.21.15.40:0/3210594478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0841022d0 0x7fe08419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.872+0000 7fe08c3d3640 1 --2- 172.21.15.40:0/3210594478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe084101910 0x7fe084198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.872+0000 7fe08c3d3640 1 --2- 172.21.15.40:0/3210594478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe084100720 0x7fe084197ad0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.872+0000 7fe08c3d3640 1 -- 172.21.15.40:0/3210594478 >> 172.21.15.40:0/3210594478 conn(0x7fe0840fbcb0 msgr2=0x7fe0840689d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.872+0000 7fe08c3d3640 1 -- 172.21.15.40:0/3210594478 shutdown_connections 2024-03-18T22:46:57.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:57.872+0000 7fe08c3d3640 1 -- 172.21.15.40:0/3210594478 wait complete. 2024-03-18T22:46:57.884 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:46:57.885 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:46:58.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.008+0000 7f6edccd3640 1 -- 172.21.15.40:0/3542215539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ed8100740 msgr2=0x7f6ed8100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.008+0000 7f6edccd3640 1 --2- 172.21.15.40:0/3542215539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ed8100740 0x7f6ed8100b60 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8009900 tx=0x7f6ec802bc30 comp rx=0 tx=0).stop 2024-03-18T22:46:58.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.008+0000 7f6edccd3640 1 -- 172.21.15.40:0/3542215539 shutdown_connections 2024-03-18T22:46:58.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.008+0000 7f6edccd3640 1 --2- 172.21.15.40:0/3542215539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ed81022f0 0x7f6ed8108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.008+0000 7f6edccd3640 1 --2- 172.21.15.40:0/3542215539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ed8101930 0x7f6ed8101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.008+0000 7f6edccd3640 1 --2- 172.21.15.40:0/3542215539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ed8100740 0x7f6ed8100b60 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.008+0000 7f6edccd3640 1 -- 172.21.15.40:0/3542215539 >> 172.21.15.40:0/3542215539 conn(0x7f6ed80fbcb0 msgr2=0x7f6ed80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.008+0000 7f6edccd3640 1 -- 172.21.15.40:0/3542215539 shutdown_connections 2024-03-18T22:46:58.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.008+0000 7f6edccd3640 1 -- 172.21.15.40:0/3542215539 wait complete. 2024-03-18T22:46:58.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.009+0000 7f6edccd3640 1 Processor -- start 2024-03-18T22:46:58.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.009+0000 7f6edccd3640 1 -- start start 2024-03-18T22:46:58.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.009+0000 7f6edccd3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ed8100740 0x7f6ed8196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.009+0000 7f6edccd3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ed8101930 0x7f6ed8196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.009+0000 7f6edccd3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ed81022f0 0x7f6ed819dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.009+0000 7f6edccd3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed810c260 con 0x7f6ed8100740 2024-03-18T22:46:58.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.009+0000 7f6edccd3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed810c0e0 con 0x7f6ed8101930 2024-03-18T22:46:58.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.009+0000 7f6edccd3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed810c3e0 con 0x7f6ed81022f0 2024-03-18T22:46:58.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.009+0000 7f6ed659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ed8100740 0x7f6ed8196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.009+0000 7f6ed659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ed8100740 0x7f6ed8196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48888/0 (socket says 172.21.15.40:48888) 2024-03-18T22:46:58.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.009+0000 7f6ed659c640 1 -- 172.21.15.40:0/2355180436 learned_addr learned my addr 172.21.15.40:0/2355180436 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.010+0000 7f6ed6d9d640 1 --2- 172.21.15.40:0/2355180436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ed81022f0 0x7f6ed819dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.009+0000 7f6ecffff640 1 --2- 172.21.15.40:0/2355180436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ed8101930 0x7f6ed8196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.010+0000 7f6ed659c640 1 -- 172.21.15.40:0/2355180436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ed81022f0 msgr2=0x7f6ed819dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.010+0000 7f6ed659c640 1 --2- 172.21.15.40:0/2355180436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ed81022f0 0x7f6ed819dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.010+0000 7f6ed659c640 1 -- 172.21.15.40:0/2355180436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ed8101930 msgr2=0x7f6ed8196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.010+0000 7f6ed659c640 1 --2- 172.21.15.40:0/2355180436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ed8101930 0x7f6ed8196f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.010+0000 7f6ed659c640 1 -- 172.21.15.40:0/2355180436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed819e630 con 0x7f6ed8100740 2024-03-18T22:46:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.010+0000 7f6ed6d9d640 1 --2- 172.21.15.40:0/2355180436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ed81022f0 0x7f6ed819dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.010+0000 7f6ed659c640 1 --2- 172.21.15.40:0/2355180436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ed8100740 0x7f6ed8196a10 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8000c00 tx=0x7f6ec8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.010+0000 7f6ecf7fe640 1 -- 172.21.15.40:0/2355180436 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8043020 con 0x7f6ed8100740 2024-03-18T22:46:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.010+0000 7f6ecf7fe640 1 -- 172.21.15.40:0/2355180436 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ec8004350 con 0x7f6ed8100740 2024-03-18T22:46:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.010+0000 7f6edccd3640 1 -- 172.21.15.40:0/2355180436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ed8068980 con 0x7f6ed8100740 2024-03-18T22:46:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.010+0000 7f6ecf7fe640 1 -- 172.21.15.40:0/2355180436 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8003b10 con 0x7f6ed8100740 2024-03-18T22:46:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.010+0000 7f6edccd3640 1 -- 172.21.15.40:0/2355180436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ed8068ed0 con 0x7f6ed8100740 2024-03-18T22:46:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.011+0000 7f6edccd3640 1 -- 172.21.15.40:0/2355180436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ed8068610 con 0x7f6ed8100740 2024-03-18T22:46:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.013+0000 7f6ecf7fe640 1 -- 172.21.15.40:0/2355180436 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6ec8046020 con 0x7f6ed8100740 2024-03-18T22:46:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.013+0000 7f6ecf7fe640 1 --2- 172.21.15.40:0/2355180436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ec40a2800 0x7f6ec40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.014+0000 7f6ecf7fe640 1 -- 172.21.15.40:0/2355180436 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6ec80f1360 con 0x7f6ed8100740 2024-03-18T22:46:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.014+0000 7f6ecffff640 1 --2- 172.21.15.40:0/2355180436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ec40a2800 0x7f6ec40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.014+0000 7f6ecffff640 1 --2- 172.21.15.40:0/2355180436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ec40a2800 0x7f6ec40a4cc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8197f20 tx=0x7f6ec0012040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.018+0000 7f6ecf7fe640 1 -- 172.21.15.40:0/2355180436 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6ec804d020 con 0x7f6ed8100740 2024-03-18T22:46:58.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.285+0000 7f6edccd3640 1 -- 172.21.15.40:0/2355180436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ed81075f0 con 0x7f6ed8100740 2024-03-18T22:46:58.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.286+0000 7f6ecf7fe640 1 -- 172.21.15.40:0/2355180436 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11947 (secure 0 0 0) 0x7f6ed81075f0 con 0x7f6ed8100740 2024-03-18T22:46:58.288 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:46:58.288 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:46:52.724919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24388,24401,24473,24481,24509]},"id":1}]} 2024-03-18T22:46:58.288 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 32 2024-03-18T22:46:58.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.288+0000 7f6edccd3640 1 -- 172.21.15.40:0/2355180436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ec40a2800 msgr2=0x7f6ec40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.288+0000 7f6edccd3640 1 --2- 172.21.15.40:0/2355180436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ec40a2800 0x7f6ec40a4cc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8197f20 tx=0x7f6ec0012040 comp rx=0 tx=0).stop 2024-03-18T22:46:58.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.288+0000 7f6edccd3640 1 -- 172.21.15.40:0/2355180436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ed8100740 msgr2=0x7f6ed8196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.288+0000 7f6edccd3640 1 --2- 172.21.15.40:0/2355180436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ed8100740 0x7f6ed8196a10 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8000c00 tx=0x7f6ec8005dc0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.288+0000 7f6edccd3640 1 -- 172.21.15.40:0/2355180436 shutdown_connections 2024-03-18T22:46:58.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.288+0000 7f6edccd3640 1 --2- 172.21.15.40:0/2355180436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ec40a2800 0x7f6ec40a4cc0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.288+0000 7f6edccd3640 1 --2- 172.21.15.40:0/2355180436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6ed81022f0 0x7f6ed819dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.288+0000 7f6edccd3640 1 --2- 172.21.15.40:0/2355180436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6ed8101930 0x7f6ed8196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.288+0000 7f6edccd3640 1 --2- 172.21.15.40:0/2355180436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ed8100740 0x7f6ed8196a10 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.288+0000 7f6edccd3640 1 -- 172.21.15.40:0/2355180436 >> 172.21.15.40:0/2355180436 conn(0x7f6ed80fbcb0 msgr2=0x7f6ed8104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.288+0000 7f6edccd3640 1 -- 172.21.15.40:0/2355180436 shutdown_connections 2024-03-18T22:46:58.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.288+0000 7f6edccd3640 1 -- 172.21.15.40:0/2355180436 wait complete. 2024-03-18T22:46:58.302 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:46:58.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:58 smithi103 ceph-mon[66377]: pgmap v480: 129 pgs: 129 active+clean; 4.5 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:46:58.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:58 smithi040 ceph-mon[62643]: pgmap v480: 129 pgs: 129 active+clean; 4.5 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:46:58.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.415+0000 7f4f23f48640 1 -- 172.21.15.40:0/662741514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f1c101820 msgr2=0x7f4f1c101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 --2- 172.21.15.40:0/662741514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f1c101820 0x7f4f1c101c40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f4f18009900 tx=0x7f4f1802bc30 comp rx=0 tx=0).stop 2024-03-18T22:46:58.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 -- 172.21.15.40:0/662741514 shutdown_connections 2024-03-18T22:46:58.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 --2- 172.21.15.40:0/662741514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c1033d0 0x7f4f1c109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 --2- 172.21.15.40:0/662741514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f1c102a10 0x7f4f1c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 --2- 172.21.15.40:0/662741514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f1c101820 0x7f4f1c101c40 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 -- 172.21.15.40:0/662741514 >> 172.21.15.40:0/662741514 conn(0x7f4f1c0fcd90 msgr2=0x7f4f1c0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 -- 172.21.15.40:0/662741514 shutdown_connections 2024-03-18T22:46:58.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 -- 172.21.15.40:0/662741514 wait complete. 2024-03-18T22:46:58.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 Processor -- start 2024-03-18T22:46:58.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 -- start start 2024-03-18T22:46:58.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c101820 0x7f4f1c197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f1c102a10 0x7f4f1c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f1c1033d0 0x7f4f1c19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1c10d320 con 0x7f4f1c102a10 2024-03-18T22:46:58.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1c10d1a0 con 0x7f4f1c1033d0 2024-03-18T22:46:58.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.416+0000 7f4f23f48640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1c10d4a0 con 0x7f4f1c101820 2024-03-18T22:46:58.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f214e3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f1c102a10 0x7f4f1c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f214e3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f1c102a10 0x7f4f1c197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48900/0 (socket says 172.21.15.40:48900) 2024-03-18T22:46:58.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f214e3640 1 -- 172.21.15.40:0/992972404 learned_addr learned my addr 172.21.15.40:0/992972404 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f21ce4640 1 --2- 172.21.15.40:0/992972404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c101820 0x7f4f1c197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f224e5640 1 --2- 172.21.15.40:0/992972404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f1c1033d0 0x7f4f1c19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f214e3640 1 -- 172.21.15.40:0/992972404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c101820 msgr2=0x7f4f1c197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f214e3640 1 --2- 172.21.15.40:0/992972404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c101820 0x7f4f1c197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f214e3640 1 -- 172.21.15.40:0/992972404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f1c1033d0 msgr2=0x7f4f1c19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f214e3640 1 --2- 172.21.15.40:0/992972404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f1c1033d0 0x7f4f1c19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f214e3640 1 -- 172.21.15.40:0/992972404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f1c19f690 con 0x7f4f1c102a10 2024-03-18T22:46:58.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f224e5640 1 --2- 172.21.15.40:0/992972404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f1c1033d0 0x7f4f1c19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f214e3640 1 --2- 172.21.15.40:0/992972404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f1c102a10 0x7f4f1c197fb0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00e9f0 tx=0x7f4f0c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f12ffd640 1 -- 172.21.15.40:0/992972404 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c00ccc0 con 0x7f4f1c102a10 2024-03-18T22:46:58.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f12ffd640 1 -- 172.21.15.40:0/992972404 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f0c0108e0 con 0x7f4f1c102a10 2024-03-18T22:46:58.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f12ffd640 1 -- 172.21.15.40:0/992972404 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c0187c0 con 0x7f4f1c102a10 2024-03-18T22:46:58.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f23f48640 1 -- 172.21.15.40:0/992972404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f1c19f980 con 0x7f4f1c102a10 2024-03-18T22:46:58.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.417+0000 7f4f23f48640 1 -- 172.21.15.40:0/992972404 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f1c19fed0 con 0x7f4f1c102a10 2024-03-18T22:46:58.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.420+0000 7f4f12ffd640 1 -- 172.21.15.40:0/992972404 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4f0c010a90 con 0x7f4f1c102a10 2024-03-18T22:46:58.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f26c0b740 1 Processor -- start 2024-03-18T22:46:58.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f26c0b740 1 -- start start 2024-03-18T22:46:58.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f26c0b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1bc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f26c0b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1c17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f26c0b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f26c0b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ae9f1204b0 con 0x55ae9f1bc2f0 2024-03-18T22:46:58.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f26c0b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ae9f0c8d70 con 0x55ae9f1bcce0 2024-03-18T22:46:58.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f26c0b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55ae9f11c550 con 0x55ae9f1c1e50 2024-03-18T22:46:58.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f224e5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1bc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f214e3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1c42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f21ce4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1c17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f214e3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1c42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:39144/0 (socket says 172.21.15.40:39144) 2024-03-18T22:46:58.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f224e5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1bc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48910/0 (socket says 172.21.15.40:48910) 2024-03-18T22:46:58.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f214e3640 1 -- 172.21.15.40:0/2710236399 learned_addr learned my addr 172.21.15.40:0/2710236399 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f12ffd640 1 --2- 172.21.15.40:0/992972404 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4f080a2800 0x7f4f080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f214e3640 1 -- 172.21.15.40:0/2710236399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bcce0 msgr2=0x55ae9f1c17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f214e3640 1 --2- 172.21.15.40:0/2710236399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1c17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f214e3640 1 -- 172.21.15.40:0/2710236399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1bc2f0 msgr2=0x55ae9f1bc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f214e3640 1 --2- 172.21.15.40:0/2710236399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1bc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.421+0000 7f4f214e3640 1 -- 172.21.15.40:0/2710236399 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae9f0eb210 con 0x55ae9f1c1e50 2024-03-18T22:46:58.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.422+0000 7f4f224e5640 1 --2- 172.21.15.40:0/992972404 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4f080a2800 0x7f4f080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.422+0000 7f4f214e3640 1 --2- 172.21.15.40:0/2710236399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1c42e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0ca3c0 tx=0x7f4f0c0c6240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de4d4c32d0599442 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.422+0000 7f4f12ffd640 1 -- 172.21.15.40:0/992972404 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4f0c0cf240 con 0x7f4f1c102a10 2024-03-18T22:46:58.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.422+0000 7f4f07fff640 1 -- 172.21.15.40:0/2710236399 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c026030 con 0x55ae9f1c1e50 2024-03-18T22:46:58.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.422+0000 7f4f07fff640 1 -- 172.21.15.40:0/2710236399 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f0c0d2e50 con 0x55ae9f1c1e50 2024-03-18T22:46:58.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.422+0000 7f4f07fff640 1 -- 172.21.15.40:0/2710236399 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c010e00 con 0x55ae9f1c1e50 2024-03-18T22:46:58.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.422+0000 7f4f224e5640 1 --2- 172.21.15.40:0/992972404 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4f080a2800 0x7f4f080a4cc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x55ae9f1bd540 tx=0x7f4f14004a40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.423+0000 7f4f26c0b740 1 -- 172.21.15.40:0/2710236399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1c1e50 msgr2=0x55ae9f1c42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.423+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/2710236399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1c42e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0ca3c0 tx=0x7f4f0c0c6240 comp rx=0 tx=0).stop 2024-03-18T22:46:58.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.423+0000 7f4f26c0b740 1 -- 172.21.15.40:0/2710236399 shutdown_connections 2024-03-18T22:46:58.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.423+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/2710236399 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1c42e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.423+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/2710236399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1c17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.423+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/2710236399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1bc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.423+0000 7f4f26c0b740 1 -- 172.21.15.40:0/2710236399 >> 172.21.15.40:0/2710236399 conn(0x55ae9f1b7860 msgr2=0x55ae9f1b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.423+0000 7f4f26c0b740 1 -- 172.21.15.40:0/2710236399 shutdown_connections 2024-03-18T22:46:58.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.423+0000 7f4f26c0b740 1 -- 172.21.15.40:0/2710236399 wait complete. 2024-03-18T22:46:58.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.423+0000 7f4f26c0b740 1 -- start start 2024-03-18T22:46:58.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.423+0000 7f4f26c0b740 1 Processor -- start 2024-03-18T22:46:58.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f26c0b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1c6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f26c0b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f26c0b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f26c0b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ae9f1c4da0 con 0x55ae9f1c1e50 2024-03-18T22:46:58.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f26c0b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ae9f0dc310 con 0x55ae9f1bc2f0 2024-03-18T22:46:58.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f26c0b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55ae9f1c4f20 con 0x55ae9f1bcce0 2024-03-18T22:46:58.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f26c0b740 10 client.0 _ll_register_callbacks cb 0x55ae9f0d02d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:58.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f224e5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f214e3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f214e3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1c6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:39160/0 (socket says 172.21.15.40:39160) 2024-03-18T22:46:58.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f214e3640 1 -- 172.21.15.40:0/337201750 learned_addr learned my addr 172.21.15.40:0/337201750 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f224e5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1cdc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48920/0 (socket says 172.21.15.40:48920) 2024-03-18T22:46:58.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f21ce4640 1 --2- 172.21.15.40:0/337201750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f214e3640 1 -- 172.21.15.40:0/337201750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bc2f0 msgr2=0x55ae9f1c6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f214e3640 1 --2- 172.21.15.40:0/337201750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1c6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f214e3640 1 -- 172.21.15.40:0/337201750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 msgr2=0x55ae9f1cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f214e3640 1 --2- 172.21.15.40:0/337201750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1cdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f214e3640 1 -- 172.21.15.40:0/337201750 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae9ee343f0 con 0x55ae9f1bcce0 2024-03-18T22:46:58.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f214e3640 1 --2- 172.21.15.40:0/337201750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1c6c50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0cc9a0 tx=0x7f4f0c0973d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59480c002cf91bd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.424+0000 7f4f057fa640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:46:58.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.425+0000 7f4f057fa640 1 -- 172.21.15.40:0/337201750 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c010e00 con 0x55ae9f1bcce0 2024-03-18T22:46:58.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.425+0000 7f4f26c0b740 1 -- 172.21.15.40:0/337201750 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f0c0d2e50 con 0x55ae9f1bcce0 2024-03-18T22:46:58.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.425+0000 7f4f057fa640 1 -- 172.21.15.40:0/337201750 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4f0c0af8d0 con 0x55ae9f1bcce0 2024-03-18T22:46:58.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.425+0000 7f4f057fa640 1 -- 172.21.15.40:0/337201750 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4f0c09bef0 con 0x55ae9f1bcce0 2024-03-18T22:46:58.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.425+0000 7f4f26c0b740 10 client.26410 fetch_fsmap learned FSMap version 32 2024-03-18T22:46:58.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.425+0000 7f4f26c0b740 1 -- 172.21.15.40:0/337201750 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ae9eefe7d0 con 0x55ae9f1bcce0 2024-03-18T22:46:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.425+0000 7f4f057fa640 1 -- 172.21.15.40:0/337201750 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f0c0d2e50 con 0x55ae9f1bcce0 2024-03-18T22:46:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.425+0000 7f4f057fa640 1 -- 172.21.15.40:0/337201750 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4f0c0bb070 con 0x55ae9f1bcce0 2024-03-18T22:46:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.425+0000 7f4f26c0b740 10 client.26410 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:46:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.425+0000 7f4f26c0b740 10 client.26410 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:46:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.425+0000 7f4f26c0b740 20 client.26410 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.425+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/337201750 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55ae9f1d2180 0x55ae9f1d4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.425+0000 7f4f26c0b740 4 client.26410 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:46:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.425+0000 7f4f26c0b740 1 -- 172.21.15.40:0/337201750 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae9f17e3e0 con 0x55ae9f1d2180 2024-03-18T22:46:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.425+0000 7f4f21ce4640 1 --2- 172.21.15.40:0/337201750 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55ae9f1d2180 0x55ae9f1d4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.426+0000 7f4f21ce4640 1 --2- 172.21.15.40:0/337201750 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55ae9f1d2180 0x55ae9f1d4570 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=47512c7ca73be13d server_cookie=e55856a4ffe33815 in_seq=0 out_seq=0 2024-03-18T22:46:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.426+0000 7f4f057fa640 10 client.26410 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:46:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.426+0000 7f4f057fa640 1 -- 172.21.15.40:0/337201750 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ae9f17e3e0 con 0x55ae9f1d2180 2024-03-18T22:46:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.426+0000 7f4f057fa640 10 client.26410 handle_command_reply: tid=0 2024-03-18T22:46:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.426+0000 7f4f26c0b740 1 client.26410 shutdown 2024-03-18T22:46:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.427+0000 7f4f26c0b740 1 -- 172.21.15.40:0/337201750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bcce0 msgr2=0x55ae9f1c6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.427+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/337201750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1c6c50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0cc9a0 tx=0x7f4f0c0973d0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.427+0000 7f4f26c0b740 1 -- 172.21.15.40:0/337201750 shutdown_connections 2024-03-18T22:46:58.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.427+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/337201750 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55ae9f1d2180 0x55ae9f1d4570 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.427+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/337201750 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.427+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/337201750 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1c6c50 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.427+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/337201750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1c6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.427+0000 7f4f057fa640 0 client.26410 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:46:58.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.427+0000 7f4f26c0b740 1 -- 172.21.15.40:0/337201750 >> 172.21.15.40:0/337201750 conn(0x55ae9f1b7860 msgr2=0x55ae9f1b6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.427+0000 7f4f26c0b740 1 -- 172.21.15.40:0/337201750 shutdown_connections 2024-03-18T22:46:58.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.427+0000 7f4f26c0b740 1 -- 172.21.15.40:0/337201750 wait complete. 2024-03-18T22:46:58.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.427+0000 7f4f26c0b740 20 client.26410 trim_cache size 0 max 16384 2024-03-18T22:46:58.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.441+0000 7f4f26c0b740 1 Processor -- start 2024-03-18T22:46:58.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.441+0000 7f4f26c0b740 1 -- start start 2024-03-18T22:46:58.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.441+0000 7f4f26c0b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1be140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.441+0000 7f4f26c0b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.441+0000 7f4f26c0b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1be680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.441+0000 7f4f26c0b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1c1aa5b0 con 0x55ae9f1c1e50 2024-03-18T22:46:58.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.441+0000 7f4f26c0b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1c1ac0f0 con 0x55ae9f1bcce0 2024-03-18T22:46:58.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.441+0000 7f4f26c0b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1c1aac80 con 0x55ae9f1bc2f0 2024-03-18T22:46:58.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.441+0000 7f4f224e5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1be680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.441+0000 7f4f21ce4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1be140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.441+0000 7f4f224e5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1be680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48924/0 (socket says 172.21.15.40:48924) 2024-03-18T22:46:58.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.441+0000 7f4f224e5640 1 -- 172.21.15.40:0/3000949111 learned_addr learned my addr 172.21.15.40:0/3000949111 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.441+0000 7f4f214e3640 1 --2- 172.21.15.40:0/3000949111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.442+0000 7f4f21ce4640 1 -- 172.21.15.40:0/3000949111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bcce0 msgr2=0x55ae9f1cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.442+0000 7f4f21ce4640 1 --2- 172.21.15.40:0/3000949111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.442+0000 7f4f21ce4640 1 -- 172.21.15.40:0/3000949111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 msgr2=0x55ae9f1be680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.442+0000 7f4f21ce4640 1 --2- 172.21.15.40:0/3000949111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1be680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.442+0000 7f4f21ce4640 1 -- 172.21.15.40:0/3000949111 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae9f1b7400 con 0x55ae9f1bc2f0 2024-03-18T22:46:58.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.442+0000 7f4f21ce4640 1 --2- 172.21.15.40:0/3000949111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1be140 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4f18036040 tx=0x7f4f18005e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e30f42f871fc1c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.442+0000 7f4f077fe640 1 -- 172.21.15.40:0/3000949111 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f1802cc00 con 0x55ae9f1bc2f0 2024-03-18T22:46:58.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.442+0000 7f4f077fe640 1 -- 172.21.15.40:0/3000949111 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f18004040 con 0x55ae9f1bc2f0 2024-03-18T22:46:58.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.442+0000 7f4f077fe640 1 -- 172.21.15.40:0/3000949111 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f180055a0 con 0x55ae9f1bc2f0 2024-03-18T22:46:58.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.443+0000 7f4f26c0b740 1 -- 172.21.15.40:0/3000949111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bc2f0 msgr2=0x55ae9f1be140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.443+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/3000949111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1be140 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4f18036040 tx=0x7f4f18005e70 comp rx=0 tx=0).stop 2024-03-18T22:46:58.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.443+0000 7f4f26c0b740 1 -- 172.21.15.40:0/3000949111 shutdown_connections 2024-03-18T22:46:58.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.443+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/3000949111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1be680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.443+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/3000949111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.443+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/3000949111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1be140 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.443+0000 7f4f26c0b740 1 -- 172.21.15.40:0/3000949111 >> 172.21.15.40:0/3000949111 conn(0x55ae9f1b7860 msgr2=0x55ae9f1c6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.443+0000 7f4f26c0b740 1 -- 172.21.15.40:0/3000949111 shutdown_connections 2024-03-18T22:46:58.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.443+0000 7f4f26c0b740 1 -- 172.21.15.40:0/3000949111 wait complete. 2024-03-18T22:46:58.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.443+0000 7f4f26c0b740 1 -- start start 2024-03-18T22:46:58.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.443+0000 7f4f26c0b740 1 Processor -- start 2024-03-18T22:46:58.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f26c0b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1cb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f26c0b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f26c0b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f26c0b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1c1aba20 con 0x55ae9f1c1e50 2024-03-18T22:46:58.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f26c0b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1c1ab350 con 0x55ae9f1bc2f0 2024-03-18T22:46:58.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f26c0b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1c1a8590 con 0x55ae9f1bcce0 2024-03-18T22:46:58.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f26c0b740 10 client.0 _ll_register_callbacks cb 0x7f4f14009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:58.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f214e3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f224e5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f214e3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1cb5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:39178/0 (socket says 172.21.15.40:39178) 2024-03-18T22:46:58.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f214e3640 1 -- 172.21.15.40:0/3476381497 learned_addr learned my addr 172.21.15.40:0/3476381497 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f21ce4640 1 --2- 172.21.15.40:0/3476381497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f214e3640 1 -- 172.21.15.40:0/3476381497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bc2f0 msgr2=0x55ae9f1cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f214e3640 1 --2- 172.21.15.40:0/3476381497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1cb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f214e3640 1 -- 172.21.15.40:0/3476381497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 msgr2=0x55ae9f1cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f214e3640 1 --2- 172.21.15.40:0/3476381497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f214e3640 1 -- 172.21.15.40:0/3476381497 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae9f18d880 con 0x55ae9f1bcce0 2024-03-18T22:46:58.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f214e3640 1 --2- 172.21.15.40:0/3476381497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1cb5b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c009250 tx=0x7f4f0c0cf170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eedcf4df9d30ee41 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f057fa640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:46:58.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f057fa640 1 -- 172.21.15.40:0/3476381497 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c09b150 con 0x55ae9f1bcce0 2024-03-18T22:46:58.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f26c0b740 1 -- 172.21.15.40:0/3476381497 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f18004040 con 0x55ae9f1bcce0 2024-03-18T22:46:58.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.444+0000 7f4f057fa640 1 -- 172.21.15.40:0/3476381497 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4f0c09ba50 con 0x55ae9f1bcce0 2024-03-18T22:46:58.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.445+0000 7f4f057fa640 1 -- 172.21.15.40:0/3476381497 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4f0c010e00 con 0x55ae9f1bcce0 2024-03-18T22:46:58.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.445+0000 7f4f26c0b740 10 client.26416 fetch_fsmap learned FSMap version 32 2024-03-18T22:46:58.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.445+0000 7f4f26c0b740 1 -- 172.21.15.40:0/3476381497 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ae9f1cdce0 con 0x55ae9f1bcce0 2024-03-18T22:46:58.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.445+0000 7f4f057fa640 1 -- 172.21.15.40:0/3476381497 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f18004040 con 0x55ae9f1bcce0 2024-03-18T22:46:58.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.445+0000 7f4f057fa640 1 -- 172.21.15.40:0/3476381497 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4f0c0bb040 con 0x55ae9f1bcce0 2024-03-18T22:46:58.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.445+0000 7f4f26c0b740 10 client.26416 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:46:58.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.445+0000 7f4f26c0b740 10 client.26416 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:46:58.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.445+0000 7f4f26c0b740 20 client.26416 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:58.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.445+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/3476381497 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55ae9f238650 0x55ae9f23aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.445+0000 7f4f26c0b740 4 client.26416 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:46:58.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.445+0000 7f4f26c0b740 1 -- 172.21.15.40:0/3476381497 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ae9f12d9f0 con 0x55ae9f238650 2024-03-18T22:46:58.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.445+0000 7f4f21ce4640 1 --2- 172.21.15.40:0/3476381497 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55ae9f238650 0x55ae9f23aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.445+0000 7f4f21ce4640 1 --2- 172.21.15.40:0/3476381497 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55ae9f238650 0x55ae9f23aa40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c3b27f2c68af9ff1 server_cookie=9348df825d72c5d8 in_seq=0 out_seq=0 2024-03-18T22:46:58.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.445+0000 7f4f057fa640 10 client.26416 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:46:58.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.446+0000 7f4f057fa640 1 -- 172.21.15.40:0/3476381497 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ae9f12d9f0 con 0x55ae9f238650 2024-03-18T22:46:58.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.446+0000 7f4f057fa640 10 client.26416 handle_command_reply: tid=0 2024-03-18T22:46:58.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.446+0000 7f4f26c0b740 1 client.26416 shutdown 2024-03-18T22:46:58.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.446+0000 7f4f26c0b740 1 -- 172.21.15.40:0/3476381497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bcce0 msgr2=0x55ae9f1cb5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.446+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/3476381497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1cb5b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c009250 tx=0x7f4f0c0cf170 comp rx=0 tx=0).stop 2024-03-18T22:46:58.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.446+0000 7f4f26c0b740 1 -- 172.21.15.40:0/3476381497 shutdown_connections 2024-03-18T22:46:58.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.446+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/3476381497 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55ae9f238650 0x55ae9f23aa40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.446+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/3476381497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ae9f1c1e50 0x55ae9f1cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.446+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/3476381497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ae9f1bcce0 0x55ae9f1cb5b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.446+0000 7f4f057fa640 0 client.26416 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:46:58.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.446+0000 7f4f26c0b740 1 --2- 172.21.15.40:0/3476381497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae9f1bc2f0 0x55ae9f1cb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.446+0000 7f4f26c0b740 1 -- 172.21.15.40:0/3476381497 >> 172.21.15.40:0/3476381497 conn(0x55ae9f1b7860 msgr2=0x55ae9f1c65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.446+0000 7f4f26c0b740 1 -- 172.21.15.40:0/3476381497 shutdown_connections 2024-03-18T22:46:58.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.446+0000 7f4f26c0b740 1 -- 172.21.15.40:0/3476381497 wait complete. 2024-03-18T22:46:58.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.446+0000 7f4f26c0b740 20 client.26416 trim_cache size 0 max 16384 2024-03-18T22:46:58.454 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:46:58.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.447+0000 7f4f23f48640 1 -- 172.21.15.40:0/992972404 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4f080a2800 msgr2=0x7f4f080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.447+0000 7f4f23f48640 1 --2- 172.21.15.40:0/992972404 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4f080a2800 0x7f4f080a4cc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x55ae9f1bd540 tx=0x7f4f14004a40 comp rx=0 tx=0).stop 2024-03-18T22:46:58.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.447+0000 7f4f23f48640 1 -- 172.21.15.40:0/992972404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f1c102a10 msgr2=0x7f4f1c197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.447+0000 7f4f23f48640 1 --2- 172.21.15.40:0/992972404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f1c102a10 0x7f4f1c197fb0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00e9f0 tx=0x7f4f0c00edf0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.447+0000 7f4f23f48640 1 -- 172.21.15.40:0/992972404 shutdown_connections 2024-03-18T22:46:58.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.447+0000 7f4f23f48640 1 --2- 172.21.15.40:0/992972404 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4f080a2800 0x7f4f080a4cc0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.447+0000 7f4f23f48640 1 --2- 172.21.15.40:0/992972404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4f1c1033d0 0x7f4f1c19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.447+0000 7f4f23f48640 1 --2- 172.21.15.40:0/992972404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4f1c102a10 0x7f4f1c197fb0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.447+0000 7f4f23f48640 1 --2- 172.21.15.40:0/992972404 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4f1c101820 0x7f4f1c197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.447+0000 7f4f23f48640 1 -- 172.21.15.40:0/992972404 >> 172.21.15.40:0/992972404 conn(0x7f4f1c0fcd90 msgr2=0x7f4f1c105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.447+0000 7f4f23f48640 1 -- 172.21.15.40:0/992972404 shutdown_connections 2024-03-18T22:46:58.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.447+0000 7f4f23f48640 1 -- 172.21.15.40:0/992972404 wait complete. 2024-03-18T22:46:58.460 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:46:58.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.577+0000 7f25ca7f3640 1 -- 172.21.15.40:0/1285536610 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4100730 msgr2=0x7f25c4100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.577+0000 7f25ca7f3640 1 --2- 172.21.15.40:0/1285536610 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4100730 0x7f25c4100b50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f25b4009900 tx=0x7f25b402bc30 comp rx=0 tx=0).stop 2024-03-18T22:46:58.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.577+0000 7f25ca7f3640 1 -- 172.21.15.40:0/1285536610 shutdown_connections 2024-03-18T22:46:58.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.577+0000 7f25ca7f3640 1 --2- 172.21.15.40:0/1285536610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25c41022e0 0x7f25c4108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.577+0000 7f25ca7f3640 1 --2- 172.21.15.40:0/1285536610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25c4101920 0x7f25c4101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.577+0000 7f25ca7f3640 1 --2- 172.21.15.40:0/1285536610 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4100730 0x7f25c4100b50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.577+0000 7f25ca7f3640 1 -- 172.21.15.40:0/1285536610 >> 172.21.15.40:0/1285536610 conn(0x7f25c40fbcc0 msgr2=0x7f25c40fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.577+0000 7f25ca7f3640 1 -- 172.21.15.40:0/1285536610 shutdown_connections 2024-03-18T22:46:58.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.577+0000 7f25ca7f3640 1 -- 172.21.15.40:0/1285536610 wait complete. 2024-03-18T22:46:58.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.577+0000 7f25ca7f3640 1 Processor -- start 2024-03-18T22:46:58.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.577+0000 7f25ca7f3640 1 -- start start 2024-03-18T22:46:58.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:58 smithi113 ceph-mon[66775]: pgmap v480: 129 pgs: 129 active+clean; 4.5 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:46:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25ca7f3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4100730 0x7f25c4197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25ca7f3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25c4101920 0x7f25c4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25ca7f3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25c41022e0 0x7f25c419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25ca7f3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25c410c230 con 0x7f25c41022e0 2024-03-18T22:46:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25ca7f3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f25c410c0b0 con 0x7f25c4101920 2024-03-18T22:46:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25ca7f3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f25c410c3b0 con 0x7f25c4100730 2024-03-18T22:46:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25c8d90640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25c41022e0 0x7f25c419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25c3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4100730 0x7f25c4197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25c3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4100730 0x7f25c4197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:39210/0 (socket says 172.21.15.40:39210) 2024-03-18T22:46:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25c37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25c4101920 0x7f25c4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25c3fff640 1 -- 172.21.15.40:0/1892684996 learned_addr learned my addr 172.21.15.40:0/1892684996 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25c8d90640 1 -- 172.21.15.40:0/1892684996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4100730 msgr2=0x7f25c4197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25c8d90640 1 --2- 172.21.15.40:0/1892684996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4100730 0x7f25c4197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25c8d90640 1 -- 172.21.15.40:0/1892684996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25c4101920 msgr2=0x7f25c4197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25c8d90640 1 --2- 172.21.15.40:0/1892684996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25c4101920 0x7f25c4197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25c8d90640 1 -- 172.21.15.40:0/1892684996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c419f6a0 con 0x7f25c41022e0 2024-03-18T22:46:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25c37fe640 1 --2- 172.21.15.40:0/1892684996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25c4101920 0x7f25c4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25c3fff640 1 --2- 172.21.15.40:0/1892684996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4100730 0x7f25c4197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.578+0000 7f25c8d90640 1 --2- 172.21.15.40:0/1892684996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25c41022e0 0x7f25c419f030 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f25bc009fd0 tx=0x7f25bc00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.579+0000 7f25c17fa640 1 -- 172.21.15.40:0/1892684996 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25bc00ce20 con 0x7f25c41022e0 2024-03-18T22:46:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.579+0000 7f25c17fa640 1 -- 172.21.15.40:0/1892684996 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25bc005020 con 0x7f25c41022e0 2024-03-18T22:46:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.579+0000 7f25ca7f3640 1 -- 172.21.15.40:0/1892684996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25c419f990 con 0x7f25c41022e0 2024-03-18T22:46:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.579+0000 7f25c17fa640 1 -- 172.21.15.40:0/1892684996 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25bc010690 con 0x7f25c41022e0 2024-03-18T22:46:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.579+0000 7f25ca7f3640 1 -- 172.21.15.40:0/1892684996 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25c419fe90 con 0x7f25c41022e0 2024-03-18T22:46:58.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.581+0000 7f25c17fa640 1 -- 172.21.15.40:0/1892684996 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f25bc0051d0 con 0x7f25c41022e0 2024-03-18T22:46:58.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25c17fa640 1 --2- 172.21.15.40:0/1892684996 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25ac0a2800 0x7f25ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25cd3d7740 1 Processor -- start 2024-03-18T22:46:58.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25cd3d7740 1 -- start start 2024-03-18T22:46:58.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25c3fff640 1 --2- 172.21.15.40:0/1892684996 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25ac0a2800 0x7f25ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25c17fa640 1 -- 172.21.15.40:0/1892684996 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f25bc0cf4f0 con 0x7f25c41022e0 2024-03-18T22:46:58.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25cd3d7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0aaf2f0 0x563da0aaf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25cd3d7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0ab47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25cd3d7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0ab4e50 0x563da0ab72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25cd3d7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563da0a134b0 con 0x563da0aafce0 2024-03-18T22:46:58.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25cd3d7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563da09bbd70 con 0x563da0aaf2f0 2024-03-18T22:46:58.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25cd3d7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563da0a0f550 con 0x563da0ab4e50 2024-03-18T22:46:58.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25c37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0ab47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25c8d90640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0ab4e50 0x563da0ab72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25c8d90640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0ab4e50 0x563da0ab72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:39212/0 (socket says 172.21.15.40:39212) 2024-03-18T22:46:58.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25c37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0ab47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48964/0 (socket says 172.21.15.40:48964) 2024-03-18T22:46:58.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25c8d90640 1 -- 172.21.15.40:0/989343889 learned_addr learned my addr 172.21.15.40:0/989343889 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.582+0000 7f25c3fff640 1 --2- 172.21.15.40:0/989343889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0aaf2f0 0x563da0aaf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.583+0000 7f25c3fff640 1 --2- 172.21.15.40:0/1892684996 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25ac0a2800 0x7f25ac0a4cc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f25b402c840 tx=0x7f25b402ccf0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.583+0000 7f25c8d90640 1 -- 172.21.15.40:0/989343889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0aaf2f0 msgr2=0x563da0aaf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.583+0000 7f25c8d90640 1 --2- 172.21.15.40:0/989343889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0aaf2f0 0x563da0aaf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.583+0000 7f25c8d90640 1 -- 172.21.15.40:0/989343889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 msgr2=0x563da0ab47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.583+0000 7f25c8d90640 1 --2- 172.21.15.40:0/989343889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0ab47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.583+0000 7f25c8d90640 1 -- 172.21.15.40:0/989343889 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563da09de210 con 0x563da0ab4e50 2024-03-18T22:46:58.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.583+0000 7f25c8d90640 1 --2- 172.21.15.40:0/989343889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0ab4e50 0x563da0ab72e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f25bc09ba30 tx=0x7f25bc09c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=decf99252363ea61 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.583+0000 7f25aa7fc640 1 -- 172.21.15.40:0/989343889 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25bc099da0 con 0x563da0ab4e50 2024-03-18T22:46:58.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.583+0000 7f25aa7fc640 1 -- 172.21.15.40:0/989343889 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25bc005540 con 0x563da0ab4e50 2024-03-18T22:46:58.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.583+0000 7f25aa7fc640 1 -- 172.21.15.40:0/989343889 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25bc0653a0 con 0x563da0ab4e50 2024-03-18T22:46:58.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.584+0000 7f25cd3d7740 1 -- 172.21.15.40:0/989343889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0ab4e50 msgr2=0x563da0ab72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.584+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/989343889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0ab4e50 0x563da0ab72e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f25bc09ba30 tx=0x7f25bc09c300 comp rx=0 tx=0).stop 2024-03-18T22:46:58.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.584+0000 7f25cd3d7740 1 -- 172.21.15.40:0/989343889 shutdown_connections 2024-03-18T22:46:58.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.584+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/989343889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0ab4e50 0x563da0ab72e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.584+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/989343889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0ab47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.584+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/989343889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0aaf2f0 0x563da0aaf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.584+0000 7f25cd3d7740 1 -- 172.21.15.40:0/989343889 >> 172.21.15.40:0/989343889 conn(0x563da0aaa860 msgr2=0x563da0aaac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.584+0000 7f25cd3d7740 1 -- 172.21.15.40:0/989343889 shutdown_connections 2024-03-18T22:46:58.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.584+0000 7f25cd3d7740 1 -- 172.21.15.40:0/989343889 wait complete. 2024-03-18T22:46:58.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.584+0000 7f25cd3d7740 1 -- start start 2024-03-18T22:46:58.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25cd3d7740 1 Processor -- start 2024-03-18T22:46:58.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25cd3d7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0aaf2f0 0x563da0ab9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25cd3d7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0ab9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25cd3d7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0ab4e50 0x563da0ac0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25cd3d7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563da0ab7e00 con 0x563da0aafce0 2024-03-18T22:46:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25cd3d7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563da09cf310 con 0x563da0aaf2f0 2024-03-18T22:46:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25cd3d7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563da0ab7f80 con 0x563da0ab4e50 2024-03-18T22:46:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25cd3d7740 10 client.0 _ll_register_callbacks cb 0x563da09c32d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25c3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0aaf2f0 0x563da0ab9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25c37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0ab9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25c3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0aaf2f0 0x563da0ab9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:59830/0 (socket says 172.21.15.40:59830) 2024-03-18T22:46:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25c3fff640 1 -- 172.21.15.40:0/449005099 learned_addr learned my addr 172.21.15.40:0/449005099 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25c8d90640 1 --2- 172.21.15.40:0/449005099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0ab4e50 0x563da0ac0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25c3fff640 1 -- 172.21.15.40:0/449005099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0ab4e50 msgr2=0x563da0ac0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25c3fff640 1 --2- 172.21.15.40:0/449005099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0ab4e50 0x563da0ac0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25c3fff640 1 -- 172.21.15.40:0/449005099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 msgr2=0x563da0ab9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25c3fff640 1 --2- 172.21.15.40:0/449005099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0ab9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.585+0000 7f25c3fff640 1 -- 172.21.15.40:0/449005099 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563da07273f0 con 0x563da0aaf2f0 2024-03-18T22:46:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.586+0000 7f25c3fff640 1 --2- 172.21.15.40:0/449005099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0aaf2f0 0x563da0ab9830 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f25b4036040 tx=0x7f25b402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3002550d1260137 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.586+0000 7f25a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:46:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.586+0000 7f25a37fe640 1 -- 172.21.15.40:0/449005099 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b4050070 con 0x563da0aaf2f0 2024-03-18T22:46:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.586+0000 7f25a37fe640 1 -- 172.21.15.40:0/449005099 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f25b40048d0 con 0x563da0aaf2f0 2024-03-18T22:46:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.586+0000 7f25cd3d7740 1 -- 172.21.15.40:0/449005099 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25bc005540 con 0x563da0aaf2f0 2024-03-18T22:46:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.586+0000 7f25a37fe640 1 -- 172.21.15.40:0/449005099 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f25b405c050 con 0x563da0aaf2f0 2024-03-18T22:46:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.586+0000 7f25cd3d7740 10 client.26441 fetch_fsmap learned FSMap version 32 2024-03-18T22:46:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.586+0000 7f25cd3d7740 1 -- 172.21.15.40:0/449005099 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563da07f17d0 con 0x563da0aaf2f0 2024-03-18T22:46:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.586+0000 7f25a37fe640 1 -- 172.21.15.40:0/449005099 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25b40035f0 con 0x563da0aaf2f0 2024-03-18T22:46:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.586+0000 7f25a37fe640 1 -- 172.21.15.40:0/449005099 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f25b4034440 con 0x563da0aaf2f0 2024-03-18T22:46:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.587+0000 7f25cd3d7740 10 client.26441 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:46:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.587+0000 7f25cd3d7740 10 client.26441 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:46:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.587+0000 7f25cd3d7740 20 client.26441 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.587+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/449005099 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x563da0ac9aa0 0x563da0acbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.587+0000 7f25cd3d7740 4 client.26441 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:46:58.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.587+0000 7f25cd3d7740 1 -- 172.21.15.40:0/449005099 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f25bc005540 con 0x563da0ac9aa0 2024-03-18T22:46:58.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.587+0000 7f25c37fe640 1 --2- 172.21.15.40:0/449005099 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x563da0ac9aa0 0x563da0acbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.587+0000 7f25a37fe640 10 client.26441 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:46:58.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.587+0000 7f25c37fe640 1 --2- 172.21.15.40:0/449005099 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x563da0ac9aa0 0x563da0acbf20 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=258fff780b5aea84 server_cookie=26e14bd486ee541b in_seq=0 out_seq=0 2024-03-18T22:46:58.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.588+0000 7f25a37fe640 1 -- 172.21.15.40:0/449005099 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f25bc005540 con 0x563da0ac9aa0 2024-03-18T22:46:58.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.588+0000 7f25a37fe640 10 client.26441 handle_command_reply: tid=0 2024-03-18T22:46:58.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.588+0000 7f25cd3d7740 1 client.26441 shutdown 2024-03-18T22:46:58.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.588+0000 7f25cd3d7740 1 -- 172.21.15.40:0/449005099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0aaf2f0 msgr2=0x563da0ab9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.588+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/449005099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0aaf2f0 0x563da0ab9830 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f25b4036040 tx=0x7f25b402bc30 comp rx=0 tx=0).stop 2024-03-18T22:46:58.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.588+0000 7f25cd3d7740 1 -- 172.21.15.40:0/449005099 shutdown_connections 2024-03-18T22:46:58.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.588+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/449005099 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x563da0ac9aa0 0x563da0acbf20 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.588+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/449005099 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0ab4e50 0x563da0ac0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.588+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/449005099 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0ab9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.588+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/449005099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0aaf2f0 0x563da0ab9830 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.588+0000 7f25a37fe640 0 client.26441 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:46:58.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.588+0000 7f25cd3d7740 1 -- 172.21.15.40:0/449005099 >> 172.21.15.40:0/449005099 conn(0x563da0aaa860 msgr2=0x563da0aa9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.588+0000 7f25cd3d7740 1 -- 172.21.15.40:0/449005099 shutdown_connections 2024-03-18T22:46:58.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.588+0000 7f25cd3d7740 1 -- 172.21.15.40:0/449005099 wait complete. 2024-03-18T22:46:58.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.588+0000 7f25cd3d7740 20 client.26441 trim_cache size 0 max 16384 2024-03-18T22:46:58.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25cd3d7740 1 Processor -- start 2024-03-18T22:46:58.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25cd3d7740 1 -- start start 2024-03-18T22:46:58.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25cd3d7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0aaf2f0 0x563da0ab9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25cd3d7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0ac0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25cd3d7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0ab4e50 0x563da0aa9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25cd3d7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25c41a8260 con 0x563da0aafce0 2024-03-18T22:46:58.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25cd3d7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f25c41a9da0 con 0x563da0ab4e50 2024-03-18T22:46:58.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25cd3d7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f25c41a8930 con 0x563da0aaf2f0 2024-03-18T22:46:58.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25c37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0ac0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25c3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0aaf2f0 0x563da0ab9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25c37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0ac0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48982/0 (socket says 172.21.15.40:48982) 2024-03-18T22:46:58.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25c37fe640 1 -- 172.21.15.40:0/2069048656 learned_addr learned my addr 172.21.15.40:0/2069048656 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25c8d90640 1 --2- 172.21.15.40:0/2069048656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0ab4e50 0x563da0aa9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25c3fff640 1 -- 172.21.15.40:0/2069048656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0ab4e50 msgr2=0x563da0aa9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25c3fff640 1 --2- 172.21.15.40:0/2069048656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0ab4e50 0x563da0aa9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25c3fff640 1 -- 172.21.15.40:0/2069048656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 msgr2=0x563da0ac0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25c3fff640 1 --2- 172.21.15.40:0/2069048656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0ac0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25c3fff640 1 -- 172.21.15.40:0/2069048656 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563da0a80880 con 0x563da0aaf2f0 2024-03-18T22:46:58.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25c37fe640 1 --2- 172.21.15.40:0/2069048656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0ac0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.604+0000 7f25c8d90640 1 --2- 172.21.15.40:0/2069048656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0ab4e50 0x563da0aa9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.605+0000 7f25c3fff640 1 --2- 172.21.15.40:0/2069048656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0aaf2f0 0x563da0ab9cd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f25b40348b0 tx=0x7f25b4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b172a860f76305b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.605+0000 7f25a9ffb640 1 -- 172.21.15.40:0/2069048656 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b4059020 con 0x563da0aaf2f0 2024-03-18T22:46:58.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.605+0000 7f25a9ffb640 1 -- 172.21.15.40:0/2069048656 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25b4034440 con 0x563da0aaf2f0 2024-03-18T22:46:58.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.605+0000 7f25a9ffb640 1 -- 172.21.15.40:0/2069048656 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b4050070 con 0x563da0aaf2f0 2024-03-18T22:46:58.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.605+0000 7f25cd3d7740 1 -- 172.21.15.40:0/2069048656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0aaf2f0 msgr2=0x563da0ab9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.605+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/2069048656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0aaf2f0 0x563da0ab9cd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f25b40348b0 tx=0x7f25b4000bc0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.605+0000 7f25cd3d7740 1 -- 172.21.15.40:0/2069048656 shutdown_connections 2024-03-18T22:46:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.605+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/2069048656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0ab4e50 0x563da0aa9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.605+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/2069048656 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0ac0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.605+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/2069048656 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0aaf2f0 0x563da0ab9cd0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.605+0000 7f25cd3d7740 1 -- 172.21.15.40:0/2069048656 >> 172.21.15.40:0/2069048656 conn(0x563da0aaa860 msgr2=0x563da0aaac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25cd3d7740 1 -- 172.21.15.40:0/2069048656 shutdown_connections 2024-03-18T22:46:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25cd3d7740 1 -- 172.21.15.40:0/2069048656 wait complete. 2024-03-18T22:46:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25cd3d7740 1 -- start start 2024-03-18T22:46:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25cd3d7740 1 Processor -- start 2024-03-18T22:46:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25cd3d7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0aaf2f0 0x563da0abed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25cd3d7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0abf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25cd3d7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0ab4e50 0x563da0ac0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25cd3d7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f25c41a96d0 con 0x563da0aafce0 2024-03-18T22:46:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25cd3d7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f25c41a9000 con 0x563da0ab4e50 2024-03-18T22:46:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25cd3d7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f25c41a6240 con 0x563da0aaf2f0 2024-03-18T22:46:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25cd3d7740 10 client.0 _ll_register_callbacks cb 0x7f25b4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25c37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0abf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25c3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0aaf2f0 0x563da0abed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25c8d90640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0ab4e50 0x563da0ac0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25c3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0aaf2f0 0x563da0abed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:39244/0 (socket says 172.21.15.40:39244) 2024-03-18T22:46:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.606+0000 7f25c3fff640 1 -- 172.21.15.40:0/2860186698 learned_addr learned my addr 172.21.15.40:0/2860186698 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.607+0000 7f25c3fff640 1 -- 172.21.15.40:0/2860186698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0ab4e50 msgr2=0x563da0ac0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.607+0000 7f25c3fff640 1 --2- 172.21.15.40:0/2860186698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0ab4e50 0x563da0ac0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.607+0000 7f25c3fff640 1 -- 172.21.15.40:0/2860186698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 msgr2=0x563da0abf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.607+0000 7f25c3fff640 1 --2- 172.21.15.40:0/2860186698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0abf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.607+0000 7f25c3fff640 1 -- 172.21.15.40:0/2860186698 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563da0ac0880 con 0x563da0aaf2f0 2024-03-18T22:46:58.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.607+0000 7f25c37fe640 1 --2- 172.21.15.40:0/2860186698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0abf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.607+0000 7f25c8d90640 1 --2- 172.21.15.40:0/2860186698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0ab4e50 0x563da0ac0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.607+0000 7f25c3fff640 1 --2- 172.21.15.40:0/2860186698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0aaf2f0 0x563da0abed40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f25b4034830 tx=0x7f25b4004f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93f220f82aded2fc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.607+0000 7f25a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:46:58.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.607+0000 7f25a37fe640 1 -- 172.21.15.40:0/2860186698 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b406d060 con 0x563da0aaf2f0 2024-03-18T22:46:58.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.607+0000 7f25cd3d7740 1 -- 172.21.15.40:0/2860186698 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25b4034440 con 0x563da0aaf2f0 2024-03-18T22:46:58.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.607+0000 7f25a37fe640 1 -- 172.21.15.40:0/2860186698 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f25b4034c60 con 0x563da0aaf2f0 2024-03-18T22:46:58.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.607+0000 7f25a37fe640 1 -- 172.21.15.40:0/2860186698 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f25b4059030 con 0x563da0aaf2f0 2024-03-18T22:46:58.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.607+0000 7f25cd3d7740 10 client.26434 fetch_fsmap learned FSMap version 32 2024-03-18T22:46:58.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.607+0000 7f25cd3d7740 1 -- 172.21.15.40:0/2860186698 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563da0b306b0 con 0x563da0aaf2f0 2024-03-18T22:46:58.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.608+0000 7f25a37fe640 1 -- 172.21.15.40:0/2860186698 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25b4034440 con 0x563da0aaf2f0 2024-03-18T22:46:58.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.608+0000 7f25a37fe640 1 -- 172.21.15.40:0/2860186698 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f25b404d450 con 0x563da0aaf2f0 2024-03-18T22:46:58.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.608+0000 7f25cd3d7740 10 client.26434 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:46:58.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.608+0000 7f25cd3d7740 10 client.26434 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:46:58.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.608+0000 7f25cd3d7740 20 client.26434 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:58.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.608+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/2860186698 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x563da0b30fd0 0x563da0b333f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.608+0000 7f25cd3d7740 4 client.26434 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:46:58.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.608+0000 7f25cd3d7740 1 -- 172.21.15.40:0/2860186698 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563da0a713e0 con 0x563da0b30fd0 2024-03-18T22:46:58.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.608+0000 7f25c37fe640 1 --2- 172.21.15.40:0/2860186698 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x563da0b30fd0 0x563da0b333f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.608+0000 7f25c37fe640 1 --2- 172.21.15.40:0/2860186698 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x563da0b30fd0 0x563da0b333f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=83f63089a002b726 server_cookie=f6ebfd6da2ea1991 in_seq=0 out_seq=0 2024-03-18T22:46:58.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.608+0000 7f25a37fe640 10 client.26434 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:46:58.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25a37fe640 1 -- 172.21.15.40:0/2860186698 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563da0a713e0 con 0x563da0b30fd0 2024-03-18T22:46:58.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25a37fe640 10 client.26434 handle_command_reply: tid=0 2024-03-18T22:46:58.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25cd3d7740 1 client.26434 shutdown 2024-03-18T22:46:58.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25cd3d7740 1 -- 172.21.15.40:0/2860186698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0aaf2f0 msgr2=0x563da0abed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/2860186698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0aaf2f0 0x563da0abed40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f25b4034830 tx=0x7f25b4004f10 comp rx=0 tx=0).stop 2024-03-18T22:46:58.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25cd3d7740 1 -- 172.21.15.40:0/2860186698 shutdown_connections 2024-03-18T22:46:58.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/2860186698 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x563da0b30fd0 0x563da0b333f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/2860186698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563da0ab4e50 0x563da0ac0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/2860186698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563da0aafce0 0x563da0abf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25a37fe640 0 client.26434 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:46:58.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25cd3d7740 1 --2- 172.21.15.40:0/2860186698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563da0aaf2f0 0x563da0abed40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25cd3d7740 1 -- 172.21.15.40:0/2860186698 >> 172.21.15.40:0/2860186698 conn(0x563da0aaa860 msgr2=0x563da0aadac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25cd3d7740 1 -- 172.21.15.40:0/2860186698 shutdown_connections 2024-03-18T22:46:58.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25cd3d7740 1 -- 172.21.15.40:0/2860186698 wait complete. 2024-03-18T22:46:58.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25cd3d7740 20 client.26434 trim_cache size 0 max 16384 2024-03-18T22:46:58.618 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:46:58.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25ca7f3640 1 -- 172.21.15.40:0/1892684996 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25ac0a2800 msgr2=0x7f25ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25ca7f3640 1 --2- 172.21.15.40:0/1892684996 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25ac0a2800 0x7f25ac0a4cc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f25b402c840 tx=0x7f25b402ccf0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25ca7f3640 1 -- 172.21.15.40:0/1892684996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25c41022e0 msgr2=0x7f25c419f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.609+0000 7f25ca7f3640 1 --2- 172.21.15.40:0/1892684996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25c41022e0 0x7f25c419f030 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f25bc009fd0 tx=0x7f25bc00ef10 comp rx=0 tx=0).stop 2024-03-18T22:46:58.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.610+0000 7f25ca7f3640 1 -- 172.21.15.40:0/1892684996 shutdown_connections 2024-03-18T22:46:58.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.610+0000 7f25ca7f3640 1 --2- 172.21.15.40:0/1892684996 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f25ac0a2800 0x7f25ac0a4cc0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.610+0000 7f25ca7f3640 1 --2- 172.21.15.40:0/1892684996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f25c41022e0 0x7f25c419f030 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.610+0000 7f25ca7f3640 1 --2- 172.21.15.40:0/1892684996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25c4101920 0x7f25c4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.610+0000 7f25ca7f3640 1 --2- 172.21.15.40:0/1892684996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f25c4100730 0x7f25c4197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.610+0000 7f25ca7f3640 1 -- 172.21.15.40:0/1892684996 >> 172.21.15.40:0/1892684996 conn(0x7f25c40fbcc0 msgr2=0x7f25c40fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.610+0000 7f25ca7f3640 1 -- 172.21.15.40:0/1892684996 shutdown_connections 2024-03-18T22:46:58.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.610+0000 7f25ca7f3640 1 -- 172.21.15.40:0/1892684996 wait complete. 2024-03-18T22:46:58.623 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:46:58.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.737+0000 7f701d0ac640 1 -- 172.21.15.40:0/96367478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70181022d0 msgr2=0x7f7018108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.737+0000 7f701d0ac640 1 --2- 172.21.15.40:0/96367478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70181022d0 0x7f7018108b60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f7010009ca0 tx=0x7f701002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.737+0000 7f701d0ac640 1 -- 172.21.15.40:0/96367478 shutdown_connections 2024-03-18T22:46:58.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.737+0000 7f701d0ac640 1 --2- 172.21.15.40:0/96367478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70181022d0 0x7f7018108b60 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.737+0000 7f701d0ac640 1 --2- 172.21.15.40:0/96367478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7018101910 0x7f7018101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.737+0000 7f701d0ac640 1 --2- 172.21.15.40:0/96367478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7018100720 0x7f7018100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.737+0000 7f701d0ac640 1 -- 172.21.15.40:0/96367478 >> 172.21.15.40:0/96367478 conn(0x7f70180fbc90 msgr2=0x7f70180fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.737+0000 7f701d0ac640 1 -- 172.21.15.40:0/96367478 shutdown_connections 2024-03-18T22:46:58.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.737+0000 7f701d0ac640 1 -- 172.21.15.40:0/96367478 wait complete. 2024-03-18T22:46:58.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701d0ac640 1 Processor -- start 2024-03-18T22:46:58.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701d0ac640 1 -- start start 2024-03-18T22:46:58.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701d0ac640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7018100720 0x7f70180694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701d0ac640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7018101910 0x7f701819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701d0ac640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70181022d0 0x7f701819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701d0ac640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f701810c250 con 0x7f70181022d0 2024-03-18T22:46:58.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701d0ac640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f701810c0d0 con 0x7f7018100720 2024-03-18T22:46:58.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701d0ac640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f701810c3d0 con 0x7f7018101910 2024-03-18T22:46:58.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701759e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70181022d0 0x7f701819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7018101910 0x7f701819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701759e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70181022d0 0x7f701819ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35216/0 (socket says 172.21.15.40:35216) 2024-03-18T22:46:58.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701759e640 1 -- 172.21.15.40:0/1090293461 learned_addr learned my addr 172.21.15.40:0/1090293461 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f7016d9d640 1 --2- 172.21.15.40:0/1090293461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7018100720 0x7f70180694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701759e640 1 -- 172.21.15.40:0/1090293461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7018101910 msgr2=0x7f701819c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701759e640 1 --2- 172.21.15.40:0/1090293461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7018101910 0x7f701819c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701759e640 1 -- 172.21.15.40:0/1090293461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7018100720 msgr2=0x7f70180694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701759e640 1 --2- 172.21.15.40:0/1090293461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7018100720 0x7f70180694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701759e640 1 -- 172.21.15.40:0/1090293461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701819f670 con 0x7f70181022d0 2024-03-18T22:46:58.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f701659c640 1 --2- 172.21.15.40:0/1090293461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7018101910 0x7f701819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.738+0000 7f7016d9d640 1 --2- 172.21.15.40:0/1090293461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7018100720 0x7f70180694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.739+0000 7f701759e640 1 --2- 172.21.15.40:0/1090293461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70181022d0 0x7f701819ed60 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f701002c700 tx=0x7f70100328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.739+0000 7f6ffffff640 1 -- 172.21.15.40:0/1090293461 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7010041020 con 0x7f70181022d0 2024-03-18T22:46:58.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.739+0000 7f701d0ac640 1 -- 172.21.15.40:0/1090293461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f701819f8a0 con 0x7f70181022d0 2024-03-18T22:46:58.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.739+0000 7f6ffffff640 1 -- 172.21.15.40:0/1090293461 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7010003e20 con 0x7f70181022d0 2024-03-18T22:46:58.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.739+0000 7f6ffffff640 1 -- 172.21.15.40:0/1090293461 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7010039040 con 0x7f70181022d0 2024-03-18T22:46:58.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.739+0000 7f701d0ac640 1 -- 172.21.15.40:0/1090293461 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f701819fd90 con 0x7f70181022d0 2024-03-18T22:46:58.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.741+0000 7f6ffffff640 1 -- 172.21.15.40:0/1090293461 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7010044020 con 0x7f70181022d0 2024-03-18T22:46:58.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.742+0000 7f6ffffff640 1 --2- 172.21.15.40:0/1090293461 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70000a2800 0x7f70000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.742+0000 7f7016d9d640 1 --2- 172.21.15.40:0/1090293461 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70000a2800 0x7f70000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.742+0000 7f6ffffff640 1 -- 172.21.15.40:0/1090293461 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f70100f0fe0 con 0x7f70181022d0 2024-03-18T22:46:58.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.742+0000 7f701fd74740 1 Processor -- start 2024-03-18T22:46:58.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.742+0000 7f701fd74740 1 -- start start 2024-03-18T22:46:58.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.742+0000 7f701fd74740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01a2f0 0x5637ca01a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.742+0000 7f701fd74740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01ace0 0x5637ca01f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.742+0000 7f701fd74740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01fe50 0x5637ca0222e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.742+0000 7f701fd74740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5637c9f7e4b0 con 0x5637ca01a2f0 2024-03-18T22:46:58.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.742+0000 7f701fd74740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5637c9f26d70 con 0x5637ca01ace0 2024-03-18T22:46:58.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.742+0000 7f701fd74740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5637c9f7a550 con 0x5637ca01fe50 2024-03-18T22:46:58.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f7016d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01a2f0 0x5637ca01a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f7016d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01a2f0 0x5637ca01a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35218/0 (socket says 172.21.15.40:35218) 2024-03-18T22:46:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f701659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01ace0 0x5637ca01f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f7016d9d640 1 -- 172.21.15.40:0/3838836917 learned_addr learned my addr 172.21.15.40:0/3838836917 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f7016d9d640 1 --2- 172.21.15.40:0/1090293461 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70000a2800 0x7f70000a4cc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7018101770 tx=0x7f7008005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f701759e640 1 --2- 172.21.15.40:0/3838836917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01fe50 0x5637ca0222e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f7016d9d640 1 -- 172.21.15.40:0/3838836917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01fe50 msgr2=0x5637ca0222e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f7016d9d640 1 --2- 172.21.15.40:0/3838836917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01fe50 0x5637ca0222e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f7016d9d640 1 -- 172.21.15.40:0/3838836917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01ace0 msgr2=0x5637ca01f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f7016d9d640 1 --2- 172.21.15.40:0/3838836917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01ace0 0x5637ca01f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f7016d9d640 1 -- 172.21.15.40:0/3838836917 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637c9f49210 con 0x5637ca01a2f0 2024-03-18T22:46:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f701659c640 1 --2- 172.21.15.40:0/3838836917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01ace0 0x5637ca01f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f7016d9d640 1 --2- 172.21.15.40:0/3838836917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01a2f0 0x5637ca01a710 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f700800f680 tx=0x7f7008014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c856d2e858cef3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f6ffd7fa640 1 -- 172.21.15.40:0/3838836917 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7008005220 con 0x5637ca01a2f0 2024-03-18T22:46:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f6ffd7fa640 1 -- 172.21.15.40:0/3838836917 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7008004c50 con 0x5637ca01a2f0 2024-03-18T22:46:58.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.743+0000 7f6ffd7fa640 1 -- 172.21.15.40:0/3838836917 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f700801cb10 con 0x5637ca01a2f0 2024-03-18T22:46:58.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.744+0000 7f701fd74740 1 -- 172.21.15.40:0/3838836917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01a2f0 msgr2=0x5637ca01a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.744+0000 7f701fd74740 1 --2- 172.21.15.40:0/3838836917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01a2f0 0x5637ca01a710 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f700800f680 tx=0x7f7008014000 comp rx=0 tx=0).stop 2024-03-18T22:46:58.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.744+0000 7f701fd74740 1 -- 172.21.15.40:0/3838836917 shutdown_connections 2024-03-18T22:46:58.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.744+0000 7f701fd74740 1 --2- 172.21.15.40:0/3838836917 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01fe50 0x5637ca0222e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.744+0000 7f701fd74740 1 --2- 172.21.15.40:0/3838836917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01ace0 0x5637ca01f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.744+0000 7f701fd74740 1 --2- 172.21.15.40:0/3838836917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01a2f0 0x5637ca01a710 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.744+0000 7f701fd74740 1 -- 172.21.15.40:0/3838836917 >> 172.21.15.40:0/3838836917 conn(0x5637ca015860 msgr2=0x5637ca015c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.744+0000 7f701fd74740 1 -- 172.21.15.40:0/3838836917 shutdown_connections 2024-03-18T22:46:58.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.744+0000 7f701fd74740 1 -- 172.21.15.40:0/3838836917 wait complete. 2024-03-18T22:46:58.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.744+0000 7f701fd74740 1 -- start start 2024-03-18T22:46:58.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.744+0000 7f701fd74740 1 Processor -- start 2024-03-18T22:46:58.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701fd74740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01a2f0 0x5637ca024840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701fd74740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01ace0 0x5637ca024d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701fd74740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01fe50 0x5637ca02bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701fd74740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5637ca022e00 con 0x5637ca01ace0 2024-03-18T22:46:58.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701fd74740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5637c9f3a310 con 0x5637ca01fe50 2024-03-18T22:46:58.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701fd74740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5637ca022f80 con 0x5637ca01a2f0 2024-03-18T22:46:58.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701fd74740 10 client.0 _ll_register_callbacks cb 0x5637c9f2e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:58.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01ace0 0x5637ca024d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01ace0 0x5637ca024d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35222/0 (socket says 172.21.15.40:35222) 2024-03-18T22:46:58.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f7016d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01a2f0 0x5637ca024840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701659c640 1 -- 172.21.15.40:0/407792529 learned_addr learned my addr 172.21.15.40:0/407792529 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701759e640 1 --2- 172.21.15.40:0/407792529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01fe50 0x5637ca02bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701659c640 1 -- 172.21.15.40:0/407792529 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01a2f0 msgr2=0x5637ca024840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701659c640 1 --2- 172.21.15.40:0/407792529 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01a2f0 0x5637ca024840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701659c640 1 -- 172.21.15.40:0/407792529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01fe50 msgr2=0x5637ca02bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701659c640 1 --2- 172.21.15.40:0/407792529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01fe50 0x5637ca02bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701659c640 1 -- 172.21.15.40:0/407792529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637c9c923f0 con 0x5637ca01ace0 2024-03-18T22:46:58.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f7016d9d640 1 --2- 172.21.15.40:0/407792529 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01a2f0 0x5637ca024840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f701659c640 1 --2- 172.21.15.40:0/407792529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01ace0 0x5637ca024d80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f700c012d20 tx=0x7f700c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8a883f6dc440cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.745+0000 7f6ffaff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:58.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.746+0000 7f6ffaff5640 1 -- 172.21.15.40:0/407792529 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f700c012110 con 0x5637ca01ace0 2024-03-18T22:46:58.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.746+0000 7f6ffaff5640 1 -- 172.21.15.40:0/407792529 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f700c004050 con 0x5637ca01ace0 2024-03-18T22:46:58.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.746+0000 7f701fd74740 1 -- 172.21.15.40:0/407792529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7008004c50 con 0x5637ca01ace0 2024-03-18T22:46:58.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.746+0000 7f6ffaff5640 1 -- 172.21.15.40:0/407792529 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f700c0122b0 con 0x5637ca01ace0 2024-03-18T22:46:58.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.746+0000 7f701fd74740 10 client.16626 fetch_fsmap learned FSMap version 32 2024-03-18T22:46:58.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.746+0000 7f701fd74740 1 -- 172.21.15.40:0/407792529 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5637c9d5c7d0 con 0x5637ca01ace0 2024-03-18T22:46:58.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.746+0000 7f6ffaff5640 1 -- 172.21.15.40:0/407792529 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7008004c50 con 0x5637ca01ace0 2024-03-18T22:46:58.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.746+0000 7f6ffaff5640 1 -- 172.21.15.40:0/407792529 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f700c010430 con 0x5637ca01ace0 2024-03-18T22:46:58.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.746+0000 7f701fd74740 10 client.16626 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:46:58.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.746+0000 7f701fd74740 10 client.16626 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:46:58.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.746+0000 7f701fd74740 20 client.16626 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:58.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.746+0000 7f701fd74740 1 --2- 172.21.15.40:0/407792529 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5637ca0305b0 0x5637ca032a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.746+0000 7f701fd74740 4 client.16626 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:46:58.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.746+0000 7f701fd74740 1 -- 172.21.15.40:0/407792529 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637c9fdc3e0 con 0x5637ca0305b0 2024-03-18T22:46:58.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.746+0000 7f7016d9d640 1 --2- 172.21.15.40:0/407792529 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5637ca0305b0 0x5637ca032a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.747+0000 7f6ffaff5640 10 client.16626 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:46:58.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.747+0000 7f7016d9d640 1 --2- 172.21.15.40:0/407792529 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5637ca0305b0 0x5637ca032a50 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a30d5cf4f870f243 server_cookie=f9349c47753194ed in_seq=0 out_seq=0 2024-03-18T22:46:58.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.748+0000 7f6ffaff5640 1 -- 172.21.15.40:0/407792529 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5637c9fdc3e0 con 0x5637ca0305b0 2024-03-18T22:46:58.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.748+0000 7f6ffaff5640 10 client.16626 handle_command_reply: tid=0 2024-03-18T22:46:58.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.748+0000 7f701fd74740 1 client.16626 shutdown 2024-03-18T22:46:58.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.748+0000 7f701fd74740 1 -- 172.21.15.40:0/407792529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01ace0 msgr2=0x5637ca024d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.748+0000 7f701fd74740 1 --2- 172.21.15.40:0/407792529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01ace0 0x5637ca024d80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f700c012d20 tx=0x7f700c00f790 comp rx=0 tx=0).stop 2024-03-18T22:46:58.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.748+0000 7f701fd74740 1 -- 172.21.15.40:0/407792529 shutdown_connections 2024-03-18T22:46:58.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.748+0000 7f701fd74740 1 --2- 172.21.15.40:0/407792529 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5637ca0305b0 0x5637ca032a50 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.748+0000 7f701fd74740 1 --2- 172.21.15.40:0/407792529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01fe50 0x5637ca02bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.748+0000 7f701fd74740 1 --2- 172.21.15.40:0/407792529 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01ace0 0x5637ca024d80 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.748+0000 7f6ffaff5640 0 client.16626 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:46:58.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.748+0000 7f701fd74740 1 --2- 172.21.15.40:0/407792529 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01a2f0 0x5637ca024840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.748+0000 7f701fd74740 1 -- 172.21.15.40:0/407792529 >> 172.21.15.40:0/407792529 conn(0x5637ca015860 msgr2=0x5637ca014ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.748+0000 7f701fd74740 1 -- 172.21.15.40:0/407792529 shutdown_connections 2024-03-18T22:46:58.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.748+0000 7f701fd74740 1 -- 172.21.15.40:0/407792529 wait complete. 2024-03-18T22:46:58.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.748+0000 7f701fd74740 20 client.16626 trim_cache size 0 max 16384 2024-03-18T22:46:58.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f701fd74740 1 Processor -- start 2024-03-18T22:46:58.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f701fd74740 1 -- start start 2024-03-18T22:46:58.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f701fd74740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01a2f0 0x5637ca010a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f701fd74740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01ace0 0x5637ca02bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f701fd74740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01fe50 0x5637ca01c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f701fd74740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f70181aa570 con 0x5637ca01fe50 2024-03-18T22:46:58.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f701fd74740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f70181ac0b0 con 0x5637ca01a2f0 2024-03-18T22:46:58.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f701fd74740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f70181aac40 con 0x5637ca01ace0 2024-03-18T22:46:58.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f701759e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01fe50 0x5637ca01c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f701759e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01fe50 0x5637ca01c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35236/0 (socket says 172.21.15.40:35236) 2024-03-18T22:46:58.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f7016d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01a2f0 0x5637ca010a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f701759e640 1 -- 172.21.15.40:0/1091280570 learned_addr learned my addr 172.21.15.40:0/1091280570 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f701659c640 1 --2- 172.21.15.40:0/1091280570 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01ace0 0x5637ca02bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f7016d9d640 1 -- 172.21.15.40:0/1091280570 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01ace0 msgr2=0x5637ca02bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f7016d9d640 1 --2- 172.21.15.40:0/1091280570 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01ace0 0x5637ca02bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f7016d9d640 1 -- 172.21.15.40:0/1091280570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01fe50 msgr2=0x5637ca01c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f7016d9d640 1 --2- 172.21.15.40:0/1091280570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01fe50 0x5637ca01c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f7016d9d640 1 -- 172.21.15.40:0/1091280570 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637ca022ce0 con 0x5637ca01a2f0 2024-03-18T22:46:58.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f701659c640 1 --2- 172.21.15.40:0/1091280570 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01ace0 0x5637ca02bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.763+0000 7f701759e640 1 --2- 172.21.15.40:0/1091280570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01fe50 0x5637ca01c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.764+0000 7f7016d9d640 1 --2- 172.21.15.40:0/1091280570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01a2f0 0x5637ca010a30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f700800f650 tx=0x7f7008005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9126542c12293926 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.764+0000 7f6ffcff9640 1 -- 172.21.15.40:0/1091280570 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f700801c550 con 0x5637ca01a2f0 2024-03-18T22:46:58.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.764+0000 7f6ffcff9640 1 -- 172.21.15.40:0/1091280570 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70080335a0 con 0x5637ca01a2f0 2024-03-18T22:46:58.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.764+0000 7f6ffcff9640 1 -- 172.21.15.40:0/1091280570 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7008026a50 con 0x5637ca01a2f0 2024-03-18T22:46:58.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 -- 172.21.15.40:0/1091280570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01a2f0 msgr2=0x5637ca010a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 --2- 172.21.15.40:0/1091280570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01a2f0 0x5637ca010a30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f700800f650 tx=0x7f7008005850 comp rx=0 tx=0).stop 2024-03-18T22:46:58.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 -- 172.21.15.40:0/1091280570 shutdown_connections 2024-03-18T22:46:58.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 --2- 172.21.15.40:0/1091280570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01fe50 0x5637ca01c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 --2- 172.21.15.40:0/1091280570 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01ace0 0x5637ca02bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 --2- 172.21.15.40:0/1091280570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01a2f0 0x5637ca010a30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 -- 172.21.15.40:0/1091280570 >> 172.21.15.40:0/1091280570 conn(0x5637ca015860 msgr2=0x5637ca015c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 -- 172.21.15.40:0/1091280570 shutdown_connections 2024-03-18T22:46:58.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 -- 172.21.15.40:0/1091280570 wait complete. 2024-03-18T22:46:58.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 -- start start 2024-03-18T22:46:58.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 Processor -- start 2024-03-18T22:46:58.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01a2f0 0x5637ca029100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01ace0 0x5637ca029640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01fe50 0x5637ca02a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f70181ab9e0 con 0x5637ca01a2f0 2024-03-18T22:46:58.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f70181ab310 con 0x5637ca01fe50 2024-03-18T22:46:58.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f70181a8550 con 0x5637ca01ace0 2024-03-18T22:46:58.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701fd74740 10 client.0 _ll_register_callbacks cb 0x7f700c009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:58.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f7016d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01a2f0 0x5637ca029100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701759e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01fe50 0x5637ca02a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f701659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01ace0 0x5637ca029640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f7016d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01a2f0 0x5637ca029100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35246/0 (socket says 172.21.15.40:35246) 2024-03-18T22:46:58.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.765+0000 7f7016d9d640 1 -- 172.21.15.40:0/2280060772 learned_addr learned my addr 172.21.15.40:0/2280060772 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.766+0000 7f7016d9d640 1 -- 172.21.15.40:0/2280060772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01ace0 msgr2=0x5637ca029640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.766+0000 7f7016d9d640 1 --2- 172.21.15.40:0/2280060772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01ace0 0x5637ca029640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.766+0000 7f7016d9d640 1 -- 172.21.15.40:0/2280060772 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01fe50 msgr2=0x5637ca02a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.766+0000 7f7016d9d640 1 --2- 172.21.15.40:0/2280060772 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01fe50 0x5637ca02a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.766+0000 7f7016d9d640 1 -- 172.21.15.40:0/2280060772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637ca02ac40 con 0x5637ca01a2f0 2024-03-18T22:46:58.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.766+0000 7f701759e640 1 --2- 172.21.15.40:0/2280060772 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01fe50 0x5637ca02a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.766+0000 7f7016d9d640 1 --2- 172.21.15.40:0/2280060772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01a2f0 0x5637ca029100 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7008013010 tx=0x7f7008011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3042d188248d754 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.766+0000 7f6ffaff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:58.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.766+0000 7f6ffaff5640 1 -- 172.21.15.40:0/2280060772 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f700801c550 con 0x5637ca01a2f0 2024-03-18T22:46:58.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.766+0000 7f6ffaff5640 1 -- 172.21.15.40:0/2280060772 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f700801c040 con 0x5637ca01a2f0 2024-03-18T22:46:58.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.766+0000 7f701fd74740 1 -- 172.21.15.40:0/2280060772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70080335a0 con 0x5637ca01a2f0 2024-03-18T22:46:58.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.766+0000 7f6ffaff5640 1 -- 172.21.15.40:0/2280060772 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f700804d050 con 0x5637ca01a2f0 2024-03-18T22:46:58.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.766+0000 7f701fd74740 10 client.16632 fetch_fsmap learned FSMap version 32 2024-03-18T22:46:58.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.766+0000 7f701fd74740 1 -- 172.21.15.40:0/2280060772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5637ca02bf40 con 0x5637ca01a2f0 2024-03-18T22:46:58.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.767+0000 7f6ffaff5640 1 -- 172.21.15.40:0/2280060772 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70080335a0 con 0x5637ca01a2f0 2024-03-18T22:46:58.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.767+0000 7f6ffaff5640 1 -- 172.21.15.40:0/2280060772 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f700803c420 con 0x5637ca01a2f0 2024-03-18T22:46:58.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.767+0000 7f701fd74740 10 client.16632 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:46:58.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.767+0000 7f701fd74740 10 client.16632 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:46:58.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.767+0000 7f701fd74740 20 client.16632 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:58.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.767+0000 7f701fd74740 1 --2- 172.21.15.40:0/2280060772 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5637ca0305b0 0x5637ca098c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.767+0000 7f701fd74740 4 client.16632 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:46:58.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.767+0000 7f701fd74740 1 -- 172.21.15.40:0/2280060772 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5637c9f8b9f0 con 0x5637ca0305b0 2024-03-18T22:46:58.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.767+0000 7f701659c640 1 --2- 172.21.15.40:0/2280060772 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5637ca0305b0 0x5637ca098c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.767+0000 7f701659c640 1 --2- 172.21.15.40:0/2280060772 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5637ca0305b0 0x5637ca098c20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=93169c2d9676d012 server_cookie=adf448374994062b in_seq=0 out_seq=0 2024-03-18T22:46:58.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.767+0000 7f6ffaff5640 10 client.16632 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:46:58.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f6ffaff5640 1 -- 172.21.15.40:0/2280060772 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5637c9f8b9f0 con 0x5637ca0305b0 2024-03-18T22:46:58.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f6ffaff5640 10 client.16632 handle_command_reply: tid=0 2024-03-18T22:46:58.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701fd74740 1 client.16632 shutdown 2024-03-18T22:46:58.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701fd74740 1 -- 172.21.15.40:0/2280060772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01a2f0 msgr2=0x5637ca029100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701fd74740 1 --2- 172.21.15.40:0/2280060772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01a2f0 0x5637ca029100 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7008013010 tx=0x7f7008011de0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701fd74740 1 -- 172.21.15.40:0/2280060772 shutdown_connections 2024-03-18T22:46:58.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701fd74740 1 --2- 172.21.15.40:0/2280060772 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5637ca0305b0 0x5637ca098c20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701fd74740 1 --2- 172.21.15.40:0/2280060772 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5637ca01fe50 0x5637ca02a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701fd74740 1 --2- 172.21.15.40:0/2280060772 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5637ca01ace0 0x5637ca029640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701fd74740 1 --2- 172.21.15.40:0/2280060772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5637ca01a2f0 0x5637ca029100 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f6ffaff5640 0 client.16632 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:46:58.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701fd74740 1 -- 172.21.15.40:0/2280060772 >> 172.21.15.40:0/2280060772 conn(0x5637ca015860 msgr2=0x5637ca023320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701fd74740 1 -- 172.21.15.40:0/2280060772 shutdown_connections 2024-03-18T22:46:58.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701fd74740 1 -- 172.21.15.40:0/2280060772 wait complete. 2024-03-18T22:46:58.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701fd74740 20 client.16632 trim_cache size 0 max 16384 2024-03-18T22:46:58.776 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:46:58.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701d0ac640 1 -- 172.21.15.40:0/1090293461 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70000a2800 msgr2=0x7f70000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701d0ac640 1 --2- 172.21.15.40:0/1090293461 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70000a2800 0x7f70000a4cc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7018101770 tx=0x7f7008005f90 comp rx=0 tx=0).stop 2024-03-18T22:46:58.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701d0ac640 1 -- 172.21.15.40:0/1090293461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70181022d0 msgr2=0x7f701819ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.768+0000 7f701d0ac640 1 --2- 172.21.15.40:0/1090293461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70181022d0 0x7f701819ed60 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f701002c700 tx=0x7f70100328a0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.769+0000 7f701d0ac640 1 -- 172.21.15.40:0/1090293461 shutdown_connections 2024-03-18T22:46:58.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.769+0000 7f701d0ac640 1 --2- 172.21.15.40:0/1090293461 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70000a2800 0x7f70000a4cc0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.769+0000 7f701d0ac640 1 --2- 172.21.15.40:0/1090293461 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70181022d0 0x7f701819ed60 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.769+0000 7f701d0ac640 1 --2- 172.21.15.40:0/1090293461 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7018101910 0x7f701819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.769+0000 7f701d0ac640 1 --2- 172.21.15.40:0/1090293461 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7018100720 0x7f70180694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.769+0000 7f701d0ac640 1 -- 172.21.15.40:0/1090293461 >> 172.21.15.40:0/1090293461 conn(0x7f70180fbc90 msgr2=0x7f7018104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.769+0000 7f701d0ac640 1 -- 172.21.15.40:0/1090293461 shutdown_connections 2024-03-18T22:46:58.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.769+0000 7f701d0ac640 1 -- 172.21.15.40:0/1090293461 wait complete. 2024-03-18T22:46:58.781 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:46:58.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.898+0000 7f8aa5eff640 1 -- 172.21.15.40:0/4005432074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa0104c50 msgr2=0x7f8aa01050d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.898+0000 7f8aa5eff640 1 --2- 172.21.15.40:0/4005432074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa0104c50 0x7f8aa01050d0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009900 tx=0x7f8a9402bc30 comp rx=0 tx=0).stop 2024-03-18T22:46:58.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.898+0000 7f8aa5eff640 1 -- 172.21.15.40:0/4005432074 shutdown_connections 2024-03-18T22:46:58.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.898+0000 7f8aa5eff640 1 --2- 172.21.15.40:0/4005432074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8aa0105610 0x7f8aa010bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.898+0000 7f8aa5eff640 1 --2- 172.21.15.40:0/4005432074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa0104c50 0x7f8aa01050d0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.898+0000 7f8aa5eff640 1 --2- 172.21.15.40:0/4005432074 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8aa0103a60 0x7f8aa0103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.898+0000 7f8aa5eff640 1 -- 172.21.15.40:0/4005432074 >> 172.21.15.40:0/4005432074 conn(0x7f8aa00fefd0 msgr2=0x7f8aa0101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.898+0000 7f8aa5eff640 1 -- 172.21.15.40:0/4005432074 shutdown_connections 2024-03-18T22:46:58.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.898+0000 7f8aa5eff640 1 -- 172.21.15.40:0/4005432074 wait complete. 2024-03-18T22:46:58.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.899+0000 7f8aa5eff640 1 Processor -- start 2024-03-18T22:46:58.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.899+0000 7f8aa5eff640 1 -- start start 2024-03-18T22:46:58.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.899+0000 7f8aa5eff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa0103a60 0x7f8aa0199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.899+0000 7f8aa5eff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8aa0104c50 0x7f8aa019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.899+0000 7f8aa5eff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8aa0105610 0x7f8aa01a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.899+0000 7f8aa5eff640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa010f590 con 0x7f8aa0103a60 2024-03-18T22:46:58.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.899+0000 7f8aa5eff640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa010f410 con 0x7f8aa0104c50 2024-03-18T22:46:58.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.899+0000 7f8aa5eff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa010f710 con 0x7f8aa0105610 2024-03-18T22:46:58.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.899+0000 7f8a9f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa0103a60 0x7f8aa0199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.899+0000 7f8a9f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa0103a60 0x7f8aa0199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35274/0 (socket says 172.21.15.40:35274) 2024-03-18T22:46:58.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.899+0000 7f8a9f7fe640 1 -- 172.21.15.40:0/3303418715 learned_addr learned my addr 172.21.15.40:0/3303418715 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.899+0000 7f8a9effd640 1 --2- 172.21.15.40:0/3303418715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8aa0104c50 0x7f8aa019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.899+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/3303418715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8aa0105610 0x7f8aa01a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.899+0000 7f8a9f7fe640 1 -- 172.21.15.40:0/3303418715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8aa0105610 msgr2=0x7f8aa01a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.900+0000 7f8a9f7fe640 1 --2- 172.21.15.40:0/3303418715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8aa0105610 0x7f8aa01a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.900+0000 7f8a9f7fe640 1 -- 172.21.15.40:0/3303418715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8aa0104c50 msgr2=0x7f8aa019a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.900+0000 7f8a9f7fe640 1 --2- 172.21.15.40:0/3303418715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8aa0104c50 0x7f8aa019a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.900+0000 7f8a9f7fe640 1 -- 172.21.15.40:0/3303418715 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa01a18f0 con 0x7f8aa0103a60 2024-03-18T22:46:58.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.900+0000 7f8a9effd640 1 --2- 172.21.15.40:0/3303418715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8aa0104c50 0x7f8aa019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.900+0000 7f8a9f7fe640 1 --2- 172.21.15.40:0/3303418715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa0103a60 0x7f8aa0199cb0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000e9c0 tx=0x7f8a9000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.900+0000 7f8a9cff9640 1 -- 172.21.15.40:0/3303418715 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9000cd80 con 0x7f8aa0103a60 2024-03-18T22:46:58.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.900+0000 7f8a9cff9640 1 -- 172.21.15.40:0/3303418715 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a900035f0 con 0x7f8aa0103a60 2024-03-18T22:46:58.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.900+0000 7f8aa5eff640 1 -- 172.21.15.40:0/3303418715 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aa01a1be0 con 0x7f8aa0103a60 2024-03-18T22:46:58.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.900+0000 7f8a9cff9640 1 -- 172.21.15.40:0/3303418715 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a90010690 con 0x7f8aa0103a60 2024-03-18T22:46:58.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.900+0000 7f8aa5eff640 1 -- 172.21.15.40:0/3303418715 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aa01a2100 con 0x7f8aa0103a60 2024-03-18T22:46:58.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8a9cff9640 1 -- 172.21.15.40:0/3303418715 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8a900037a0 con 0x7f8aa0103a60 2024-03-18T22:46:58.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8aa8bc7740 1 Processor -- start 2024-03-18T22:46:58.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8aa8bc7740 1 -- start start 2024-03-18T22:46:58.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8aa8bc7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6ab710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8aa8bc7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6abce0 0x563daf6b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8aa8bc7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6b0e50 0x563daf6b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8aa8bc7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563daf60f4b0 con 0x563daf6b0e50 2024-03-18T22:46:58.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8aa8bc7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563daf5b7d70 con 0x563daf6abce0 2024-03-18T22:46:58.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8aa8bc7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563daf60b550 con 0x563daf6ab2f0 2024-03-18T22:46:58.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8a9ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6ab710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8a9effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6b0e50 0x563daf6b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8a9ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6ab710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:56988/0 (socket says 172.21.15.40:56988) 2024-03-18T22:46:58.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8a9ffff640 1 -- 172.21.15.40:0/2591481612 learned_addr learned my addr 172.21.15.40:0/2591481612 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8a9f7fe640 1 --2- 172.21.15.40:0/2591481612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6abce0 0x563daf6b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8a9ffff640 1 -- 172.21.15.40:0/2591481612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6abce0 msgr2=0x563daf6b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/2591481612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6abce0 0x563daf6b07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8a9ffff640 1 -- 172.21.15.40:0/2591481612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6b0e50 msgr2=0x563daf6b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.903+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/2591481612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6b0e50 0x563daf6b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.904+0000 7f8a9ffff640 1 -- 172.21.15.40:0/2591481612 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563daf5da210 con 0x563daf6ab2f0 2024-03-18T22:46:58.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.904+0000 7f8a9cff9640 1 --2- 172.21.15.40:0/3303418715 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8a880a2880 0x7f8a880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.904+0000 7f8a9f7fe640 1 --2- 172.21.15.40:0/3303418715 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8a880a2880 0x7f8a880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.904+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/2591481612 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6ab710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8a98009fd0 tx=0x7f8a980118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5369a0c0a9cc554d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.904+0000 7f8a9cff9640 1 -- 172.21.15.40:0/3303418715 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8a900cf2f0 con 0x7f8aa0103a60 2024-03-18T22:46:58.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.904+0000 7f8a85ffb640 1 -- 172.21.15.40:0/2591481612 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a98004d20 con 0x563daf6ab2f0 2024-03-18T22:46:58.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.904+0000 7f8a85ffb640 1 -- 172.21.15.40:0/2591481612 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a98004050 con 0x563daf6ab2f0 2024-03-18T22:46:58.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.904+0000 7f8a85ffb640 1 -- 172.21.15.40:0/2591481612 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a98005280 con 0x563daf6ab2f0 2024-03-18T22:46:58.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.905+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/2591481612 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 msgr2=0x563daf6ab710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.905+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/2591481612 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6ab710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8a98009fd0 tx=0x7f8a980118d0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.905+0000 7f8a9f7fe640 1 --2- 172.21.15.40:0/3303418715 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8a880a2880 0x7f8a880a4d40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8a900ccd70 tx=0x7f8a900d2dd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.905+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/2591481612 shutdown_connections 2024-03-18T22:46:58.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.905+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/2591481612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6b0e50 0x563daf6b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.905+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/2591481612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6abce0 0x563daf6b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.905+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/2591481612 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6ab710 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.905+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/2591481612 >> 172.21.15.40:0/2591481612 conn(0x563daf6a6860 msgr2=0x563daf6a6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.905+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/2591481612 shutdown_connections 2024-03-18T22:46:58.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.905+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/2591481612 wait complete. 2024-03-18T22:46:58.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.905+0000 7f8aa8bc7740 1 -- start start 2024-03-18T22:46:58.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.905+0000 7f8aa8bc7740 1 Processor -- start 2024-03-18T22:46:58.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8aa8bc7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8aa8bc7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6abce0 0x563daf6b5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8aa8bc7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6b0e50 0x563daf6bcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8aa8bc7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563daf6b3e00 con 0x563daf6b0e50 2024-03-18T22:46:58.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8aa8bc7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563daf5cb310 con 0x563daf6abce0 2024-03-18T22:46:58.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8aa8bc7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563daf6b3f80 con 0x563daf6ab2f0 2024-03-18T22:46:58.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8aa8bc7740 10 client.0 _ll_register_callbacks cb 0x563daf5bf2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:58.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8a9ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8a9ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6b57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:56998/0 (socket says 172.21.15.40:56998) 2024-03-18T22:46:58.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8a9ffff640 1 -- 172.21.15.40:0/661594467 learned_addr learned my addr 172.21.15.40:0/661594467 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8a9effd640 1 --2- 172.21.15.40:0/661594467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6abce0 0x563daf6b5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/661594467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6b0e50 0x563daf6bcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8a9ffff640 1 -- 172.21.15.40:0/661594467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6abce0 msgr2=0x563daf6b5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/661594467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6abce0 0x563daf6b5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8a9ffff640 1 -- 172.21.15.40:0/661594467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6b0e50 msgr2=0x563daf6bcd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/661594467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6b0e50 0x563daf6bcd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8a9ffff640 1 -- 172.21.15.40:0/661594467 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563daf3233f0 con 0x563daf6ab2f0 2024-03-18T22:46:58.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8a9effd640 1 --2- 172.21.15.40:0/661594467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6abce0 0x563daf6b5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/661594467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6b57c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8a9801b040 tx=0x7f8a980118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e95f18979f1663cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.906+0000 7f8a7effd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:46:58.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.907+0000 7f8a7effd640 1 -- 172.21.15.40:0/661594467 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a98004a40 con 0x563daf6ab2f0 2024-03-18T22:46:58.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.907+0000 7f8a7effd640 1 -- 172.21.15.40:0/661594467 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8a98004be0 con 0x563daf6ab2f0 2024-03-18T22:46:58.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.907+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/661594467 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a98004050 con 0x563daf6ab2f0 2024-03-18T22:46:58.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.907+0000 7f8a7effd640 1 -- 172.21.15.40:0/661594467 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8a98036050 con 0x563daf6ab2f0 2024-03-18T22:46:58.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.907+0000 7f8aa8bc7740 10 client.26464 fetch_fsmap learned FSMap version 32 2024-03-18T22:46:58.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.907+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/661594467 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563daf3ed7d0 con 0x563daf6ab2f0 2024-03-18T22:46:58.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.907+0000 7f8a7effd640 1 -- 172.21.15.40:0/661594467 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a98004050 con 0x563daf6ab2f0 2024-03-18T22:46:58.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.907+0000 7f8a7effd640 1 -- 172.21.15.40:0/661594467 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8a9800f4e0 con 0x563daf6ab2f0 2024-03-18T22:46:58.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.907+0000 7f8aa8bc7740 10 client.26464 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:46:58.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.907+0000 7f8aa8bc7740 10 client.26464 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:46:58.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.907+0000 7f8aa8bc7740 20 client.26464 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:58.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.907+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/661594467 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x563daf6c1400 0x563daf6c3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.907+0000 7f8aa8bc7740 4 client.26464 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:46:58.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.907+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/661594467 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563daf66d3e0 con 0x563daf6c1400 2024-03-18T22:46:58.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.907+0000 7f8a9effd640 1 --2- 172.21.15.40:0/661594467 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x563daf6c1400 0x563daf6c3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.908+0000 7f8a9effd640 1 --2- 172.21.15.40:0/661594467 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x563daf6c1400 0x563daf6c3810 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7cbe9c5299197132 server_cookie=3474bbcee5d9eff1 in_seq=0 out_seq=0 2024-03-18T22:46:58.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.908+0000 7f8a7effd640 10 client.26464 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:46:58.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.909+0000 7f8a7effd640 1 -- 172.21.15.40:0/661594467 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563daf66d3e0 con 0x563daf6c1400 2024-03-18T22:46:58.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.909+0000 7f8a7effd640 10 client.26464 handle_command_reply: tid=0 2024-03-18T22:46:58.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.909+0000 7f8aa8bc7740 1 client.26464 shutdown 2024-03-18T22:46:58.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.909+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/661594467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 msgr2=0x563daf6b57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.909+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/661594467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6b57c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8a9801b040 tx=0x7f8a980118d0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.909+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/661594467 shutdown_connections 2024-03-18T22:46:58.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.909+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/661594467 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x563daf6c1400 0x563daf6c3810 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.909+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/661594467 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6b0e50 0x563daf6bcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.909+0000 7f8a7effd640 0 client.26464 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:46:58.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.909+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/661594467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6abce0 0x563daf6b5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.909+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/661594467 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6b57c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.909+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/661594467 >> 172.21.15.40:0/661594467 conn(0x563daf6a6860 msgr2=0x563daf6a5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.909+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/661594467 shutdown_connections 2024-03-18T22:46:58.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.909+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/661594467 wait complete. 2024-03-18T22:46:58.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.909+0000 7f8aa8bc7740 20 client.26464 trim_cache size 0 max 16384 2024-03-18T22:46:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.929+0000 7f8aa8bc7740 1 Processor -- start 2024-03-18T22:46:58.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.929+0000 7f8aa8bc7740 1 -- start start 2024-03-18T22:46:58.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.929+0000 7f8aa8bc7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6ab2f0 0x563daf6b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.929+0000 7f8aa8bc7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6abce0 0x563daf6bcd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.929+0000 7f8aa8bc7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6b0e50 0x563daf6aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.929+0000 7f8aa8bc7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa01ac770 con 0x563daf6abce0 2024-03-18T22:46:58.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.929+0000 7f8aa8bc7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa01ae2b0 con 0x563daf6ab2f0 2024-03-18T22:46:58.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.929+0000 7f8aa8bc7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa01ace40 con 0x563daf6b0e50 2024-03-18T22:46:58.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.929+0000 7f8a9effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6abce0 0x563daf6bcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.929+0000 7f8a9effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6abce0 0x563daf6bcd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35292/0 (socket says 172.21.15.40:35292) 2024-03-18T22:46:58.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.929+0000 7f8a9effd640 1 -- 172.21.15.40:0/3531281223 learned_addr learned my addr 172.21.15.40:0/3531281223 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.929+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/3531281223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6ab2f0 0x563daf6b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.929+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/3531281223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6b0e50 0x563daf6aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.930+0000 7f8a9effd640 1 -- 172.21.15.40:0/3531281223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6b0e50 msgr2=0x563daf6aa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.930+0000 7f8a9effd640 1 --2- 172.21.15.40:0/3531281223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6b0e50 0x563daf6aa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.930+0000 7f8a9effd640 1 -- 172.21.15.40:0/3531281223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6ab2f0 msgr2=0x563daf6b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.930+0000 7f8a9effd640 1 --2- 172.21.15.40:0/3531281223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6ab2f0 0x563daf6b5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.930+0000 7f8a9effd640 1 -- 172.21.15.40:0/3531281223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563daf67c880 con 0x563daf6abce0 2024-03-18T22:46:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.930+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/3531281223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6ab2f0 0x563daf6b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.930+0000 7f8a9effd640 1 --2- 172.21.15.40:0/3531281223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6abce0 0x563daf6bcd00 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8a9402ce10 tx=0x7f8a94005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec8e61cb3d4c335 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.930+0000 7f8a857fa640 1 -- 172.21.15.40:0/3531281223 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a94004490 con 0x563daf6abce0 2024-03-18T22:46:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.930+0000 7f8a857fa640 1 -- 172.21.15.40:0/3531281223 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a9404b400 con 0x563daf6abce0 2024-03-18T22:46:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.930+0000 7f8a857fa640 1 -- 172.21.15.40:0/3531281223 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a940055c0 con 0x563daf6abce0 2024-03-18T22:46:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.931+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/3531281223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6abce0 msgr2=0x563daf6bcd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.931+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/3531281223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6abce0 0x563daf6bcd00 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8a9402ce10 tx=0x7f8a94005fd0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.931+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/3531281223 shutdown_connections 2024-03-18T22:46:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.931+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/3531281223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6b0e50 0x563daf6aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.931+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/3531281223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6abce0 0x563daf6bcd00 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.931+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/3531281223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6ab2f0 0x563daf6b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.931+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/3531281223 >> 172.21.15.40:0/3531281223 conn(0x563daf6a6860 msgr2=0x563daf6a9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.931+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/3531281223 shutdown_connections 2024-03-18T22:46:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.931+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/3531281223 wait complete. 2024-03-18T22:46:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.931+0000 7f8aa8bc7740 1 -- start start 2024-03-18T22:46:58.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8aa8bc7740 1 Processor -- start 2024-03-18T22:46:58.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8aa8bc7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6b9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8aa8bc7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6abce0 0x563daf6ba1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8aa8bc7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6b0e50 0x563daf725740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8aa8bc7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa01adbe0 con 0x563daf6abce0 2024-03-18T22:46:58.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8aa8bc7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa01ad510 con 0x563daf6b0e50 2024-03-18T22:46:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8aa8bc7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa01aa750 con 0x563daf6ab2f0 2024-03-18T22:46:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8aa8bc7740 10 client.0 _ll_register_callbacks cb 0x7f8a90005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8a9effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6abce0 0x563daf6ba1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8a9effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6abce0 0x563daf6ba1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35298/0 (socket says 172.21.15.40:35298) 2024-03-18T22:46:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8a9effd640 1 -- 172.21.15.40:0/1809763218 learned_addr learned my addr 172.21.15.40:0/1809763218 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/1809763218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6b0e50 0x563daf725740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/1809763218 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6b9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8a9effd640 1 -- 172.21.15.40:0/1809763218 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 msgr2=0x563daf6b9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8a9effd640 1 --2- 172.21.15.40:0/1809763218 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6b9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8a9effd640 1 -- 172.21.15.40:0/1809763218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6b0e50 msgr2=0x563daf725740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8a9effd640 1 --2- 172.21.15.40:0/1809763218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6b0e50 0x563daf725740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8a9effd640 1 -- 172.21.15.40:0/1809763218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563daf725c80 con 0x563daf6abce0 2024-03-18T22:46:58.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/1809763218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6b0e50 0x563daf725740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8a9effd640 1 --2- 172.21.15.40:0/1809763218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6abce0 0x563daf6ba1c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8a9402c840 tx=0x7f8a94004490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7eff2c174bf783 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.932+0000 7f8a7effd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.933+0000 7f8a7effd640 1 -- 172.21.15.40:0/1809763218 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a94005340 con 0x563daf6abce0 2024-03-18T22:46:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.933+0000 7f8a7effd640 1 -- 172.21.15.40:0/1809763218 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8a940054e0 con 0x563daf6abce0 2024-03-18T22:46:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.933+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/1809763218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a9404b400 con 0x563daf6abce0 2024-03-18T22:46:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.933+0000 7f8a7effd640 1 -- 172.21.15.40:0/1809763218 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8a9404bab0 con 0x563daf6abce0 2024-03-18T22:46:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.933+0000 7f8aa8bc7740 10 client.16650 fetch_fsmap learned FSMap version 32 2024-03-18T22:46:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.933+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/1809763218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563daf726e50 con 0x563daf6abce0 2024-03-18T22:46:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.933+0000 7f8a7effd640 1 -- 172.21.15.40:0/1809763218 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a9404b400 con 0x563daf6abce0 2024-03-18T22:46:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.933+0000 7f8a7effd640 1 -- 172.21.15.40:0/1809763218 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8a94054070 con 0x563daf6abce0 2024-03-18T22:46:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.933+0000 7f8aa8bc7740 10 client.16650 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:46:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.933+0000 7f8aa8bc7740 10 client.16650 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:46:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.933+0000 7f8aa8bc7740 20 client.16650 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:58.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.933+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/1809763218 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x563daf7277e0 0x563daf729c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.933+0000 7f8aa8bc7740 4 client.16650 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:46:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.933+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/1809763218 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563daf61c9f0 con 0x563daf7277e0 2024-03-18T22:46:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.933+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/1809763218 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x563daf7277e0 0x563daf729c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8a9ffff640 1 --2- 172.21.15.40:0/1809763218 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x563daf7277e0 0x563daf729c00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=efeeb5e00215faff server_cookie=8e75c5e1f2503700 in_seq=0 out_seq=0 2024-03-18T22:46:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8a7effd640 10 client.16650 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:46:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8a7effd640 1 -- 172.21.15.40:0/1809763218 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563daf61c9f0 con 0x563daf7277e0 2024-03-18T22:46:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8a7effd640 10 client.16650 handle_command_reply: tid=0 2024-03-18T22:46:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8aa8bc7740 1 client.16650 shutdown 2024-03-18T22:46:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/1809763218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6abce0 msgr2=0x563daf6ba1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/1809763218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6abce0 0x563daf6ba1c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8a9402c840 tx=0x7f8a94004490 comp rx=0 tx=0).stop 2024-03-18T22:46:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/1809763218 shutdown_connections 2024-03-18T22:46:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/1809763218 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x563daf7277e0 0x563daf729c00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/1809763218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563daf6b0e50 0x563daf725740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/1809763218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563daf6abce0 0x563daf6ba1c0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8a7effd640 0 client.16650 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:46:58.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8aa8bc7740 1 --2- 172.21.15.40:0/1809763218 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563daf6ab2f0 0x563daf6b9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/1809763218 >> 172.21.15.40:0/1809763218 conn(0x563daf6a6860 msgr2=0x563daf6aa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/1809763218 shutdown_connections 2024-03-18T22:46:58.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8aa8bc7740 1 -- 172.21.15.40:0/1809763218 wait complete. 2024-03-18T22:46:58.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.934+0000 7f8aa8bc7740 20 client.16650 trim_cache size 0 max 16384 2024-03-18T22:46:58.946 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:46:58.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.935+0000 7f8aa5eff640 1 -- 172.21.15.40:0/3303418715 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8a880a2880 msgr2=0x7f8a880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.935+0000 7f8aa5eff640 1 --2- 172.21.15.40:0/3303418715 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8a880a2880 0x7f8a880a4d40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8a900ccd70 tx=0x7f8a900d2dd0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.935+0000 7f8aa5eff640 1 -- 172.21.15.40:0/3303418715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa0103a60 msgr2=0x7f8aa0199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:58.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.935+0000 7f8aa5eff640 1 --2- 172.21.15.40:0/3303418715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa0103a60 0x7f8aa0199cb0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000e9c0 tx=0x7f8a9000ee80 comp rx=0 tx=0).stop 2024-03-18T22:46:58.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.935+0000 7f8aa5eff640 1 -- 172.21.15.40:0/3303418715 shutdown_connections 2024-03-18T22:46:58.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.935+0000 7f8aa5eff640 1 --2- 172.21.15.40:0/3303418715 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8a880a2880 0x7f8a880a4d40 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.935+0000 7f8aa5eff640 1 --2- 172.21.15.40:0/3303418715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8aa0105610 0x7f8aa01a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.935+0000 7f8aa5eff640 1 --2- 172.21.15.40:0/3303418715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8aa0104c50 0x7f8aa019a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.935+0000 7f8aa5eff640 1 --2- 172.21.15.40:0/3303418715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8aa0103a60 0x7f8aa0199cb0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:58.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.935+0000 7f8aa5eff640 1 -- 172.21.15.40:0/3303418715 >> 172.21.15.40:0/3303418715 conn(0x7f8aa00fefd0 msgr2=0x7f8aa0108070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:58.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.935+0000 7f8aa5eff640 1 -- 172.21.15.40:0/3303418715 shutdown_connections 2024-03-18T22:46:58.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:58.935+0000 7f8aa5eff640 1 -- 172.21.15.40:0/3303418715 wait complete. 2024-03-18T22:46:58.950 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:46:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.067+0000 7f17441e6640 1 -- 172.21.15.40:0/571207249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f173c101910 msgr2=0x7f173c101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.067+0000 7f17441e6640 1 --2- 172.21.15.40:0/571207249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f173c101910 0x7f173c101d90 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f172c009900 tx=0x7f172c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:46:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.067+0000 7f17441e6640 1 -- 172.21.15.40:0/571207249 shutdown_connections 2024-03-18T22:46:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.067+0000 7f17441e6640 1 --2- 172.21.15.40:0/571207249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f173c1022d0 0x7f173c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.067+0000 7f17441e6640 1 --2- 172.21.15.40:0/571207249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f173c101910 0x7f173c101d90 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.067+0000 7f17441e6640 1 --2- 172.21.15.40:0/571207249 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f173c100720 0x7f173c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.067+0000 7f17441e6640 1 -- 172.21.15.40:0/571207249 >> 172.21.15.40:0/571207249 conn(0x7f173c0fbc90 msgr2=0x7f173c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.067+0000 7f17441e6640 1 -- 172.21.15.40:0/571207249 shutdown_connections 2024-03-18T22:46:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.067+0000 7f17441e6640 1 -- 172.21.15.40:0/571207249 wait complete. 2024-03-18T22:46:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f17441e6640 1 Processor -- start 2024-03-18T22:46:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f17441e6640 1 -- start start 2024-03-18T22:46:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f17441e6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f173c100720 0x7f173c196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f17441e6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f173c101910 0x7f173c196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f17441e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f173c1022d0 0x7f173c19dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f17441e6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f173c10c230 con 0x7f173c1022d0 2024-03-18T22:46:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f17441e6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f173c10c0b0 con 0x7f173c100720 2024-03-18T22:46:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f17441e6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f173c10c3b0 con 0x7f173c101910 2024-03-18T22:46:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f1742783640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f173c1022d0 0x7f173c19dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f1742783640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f173c1022d0 0x7f173c19dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35318/0 (socket says 172.21.15.40:35318) 2024-03-18T22:46:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f1742783640 1 -- 172.21.15.40:0/1284011361 learned_addr learned my addr 172.21.15.40:0/1284011361 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f1741781640 1 --2- 172.21.15.40:0/1284011361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f173c101910 0x7f173c196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f1741f82640 1 --2- 172.21.15.40:0/1284011361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f173c100720 0x7f173c196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f1742783640 1 -- 172.21.15.40:0/1284011361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f173c101910 msgr2=0x7f173c196e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f1742783640 1 --2- 172.21.15.40:0/1284011361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f173c101910 0x7f173c196e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f1742783640 1 -- 172.21.15.40:0/1284011361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f173c100720 msgr2=0x7f173c196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f1742783640 1 --2- 172.21.15.40:0/1284011361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f173c100720 0x7f173c196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.068+0000 7f1742783640 1 -- 172.21.15.40:0/1284011361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f173c19e530 con 0x7f173c1022d0 2024-03-18T22:46:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.069+0000 7f1742783640 1 --2- 172.21.15.40:0/1284011361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f173c1022d0 0x7f173c19dec0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f173400eb10 tx=0x7f173400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.069+0000 7f1732ffd640 1 -- 172.21.15.40:0/1284011361 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1734004b20 con 0x7f173c1022d0 2024-03-18T22:46:59.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.069+0000 7f1732ffd640 1 -- 172.21.15.40:0/1284011361 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1734004cc0 con 0x7f173c1022d0 2024-03-18T22:46:59.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.069+0000 7f1732ffd640 1 -- 172.21.15.40:0/1284011361 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1734010690 con 0x7f173c1022d0 2024-03-18T22:46:59.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.069+0000 7f17441e6640 1 -- 172.21.15.40:0/1284011361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f173c19e820 con 0x7f173c1022d0 2024-03-18T22:46:59.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.069+0000 7f17441e6640 1 -- 172.21.15.40:0/1284011361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f173c19ed40 con 0x7f173c1022d0 2024-03-18T22:46:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.071+0000 7f1732ffd640 1 -- 172.21.15.40:0/1284011361 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1734005020 con 0x7f173c1022d0 2024-03-18T22:46:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.071+0000 7f1746dd1740 1 Processor -- start 2024-03-18T22:46:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1732ffd640 1 --2- 172.21.15.40:0/1284011361 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17280a2880 0x7f17280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1746dd1740 1 -- start start 2024-03-18T22:46:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1732ffd640 1 -- 172.21.15.40:0/1284011361 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f17340cf360 con 0x7f173c1022d0 2024-03-18T22:46:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1741781640 1 --2- 172.21.15.40:0/1284011361 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17280a2880 0x7f17280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1746dd1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7c500 0x560d1ca7c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1746dd1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7cef0 0x560d1ca819f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1746dd1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca82060 0x560d1ca844f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1746dd1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560d1c9de4b0 con 0x560d1ca7c500 2024-03-18T22:46:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1746dd1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560d1c986d70 con 0x560d1ca82060 2024-03-18T22:46:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1746dd1740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560d1c9da550 con 0x560d1ca7cef0 2024-03-18T22:46:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1741f82640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7c500 0x560d1ca7c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1741781640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7cef0 0x560d1ca819f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1742783640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca82060 0x560d1ca844f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1741f82640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7c500 0x560d1ca7c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35330/0 (socket says 172.21.15.40:35330) 2024-03-18T22:46:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1741f82640 1 -- 172.21.15.40:0/674512433 learned_addr learned my addr 172.21.15.40:0/674512433 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1741781640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7cef0 0x560d1ca819f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57054/0 (socket says 172.21.15.40:57054) 2024-03-18T22:46:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1742783640 1 -- 172.21.15.40:0/674512433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7cef0 msgr2=0x560d1ca819f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1742783640 1 --2- 172.21.15.40:0/674512433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7cef0 0x560d1ca819f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1742783640 1 -- 172.21.15.40:0/674512433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7c500 msgr2=0x560d1ca7c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1742783640 1 --2- 172.21.15.40:0/674512433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7c500 0x560d1ca7c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.072+0000 7f1741781640 1 --2- 172.21.15.40:0/1284011361 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17280a2880 0x7f17280a4d40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f172c009230 tx=0x7f172c02cb90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.073+0000 7f1742783640 1 -- 172.21.15.40:0/674512433 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d1c9a9210 con 0x560d1ca82060 2024-03-18T22:46:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.073+0000 7f1742783640 1 --2- 172.21.15.40:0/674512433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca82060 0x560d1ca844f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f17340ccac0 tx=0x7f173409b450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9932571f0b5fb17 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.073+0000 7f1727fff640 1 -- 172.21.15.40:0/674512433 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17340b4070 con 0x560d1ca82060 2024-03-18T22:46:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.073+0000 7f1727fff640 1 -- 172.21.15.40:0/674512433 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f173409b910 con 0x560d1ca82060 2024-03-18T22:46:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.073+0000 7f1727fff640 1 -- 172.21.15.40:0/674512433 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17340a3c20 con 0x560d1ca82060 2024-03-18T22:46:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.074+0000 7f1746dd1740 1 -- 172.21.15.40:0/674512433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca82060 msgr2=0x560d1ca844f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.074+0000 7f1746dd1740 1 --2- 172.21.15.40:0/674512433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca82060 0x560d1ca844f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f17340ccac0 tx=0x7f173409b450 comp rx=0 tx=0).stop 2024-03-18T22:46:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.074+0000 7f1746dd1740 1 -- 172.21.15.40:0/674512433 shutdown_connections 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.074+0000 7f1746dd1740 1 --2- 172.21.15.40:0/674512433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca82060 0x560d1ca844f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.074+0000 7f1746dd1740 1 --2- 172.21.15.40:0/674512433 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7cef0 0x560d1ca819f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.074+0000 7f1746dd1740 1 --2- 172.21.15.40:0/674512433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7c500 0x560d1ca7c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.074+0000 7f1746dd1740 1 -- 172.21.15.40:0/674512433 >> 172.21.15.40:0/674512433 conn(0x560d1ca75860 msgr2=0x560d1ca75c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.074+0000 7f1746dd1740 1 -- 172.21.15.40:0/674512433 shutdown_connections 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.074+0000 7f1746dd1740 1 -- 172.21.15.40:0/674512433 wait complete. 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.074+0000 7f1746dd1740 1 -- start start 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.074+0000 7f1746dd1740 1 Processor -- start 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1746dd1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7c500 0x560d1ca86a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1746dd1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7cef0 0x560d1ca86f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1746dd1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca82060 0x560d1ca8df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1746dd1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560d1ca85010 con 0x560d1ca7cef0 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1746dd1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560d1c99a310 con 0x560d1ca82060 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1746dd1740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560d1ca85190 con 0x560d1ca7c500 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1746dd1740 10 client.0 _ll_register_callbacks cb 0x560d1c98e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1741781640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7cef0 0x560d1ca86f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1741781640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7cef0 0x560d1ca86f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35332/0 (socket says 172.21.15.40:35332) 2024-03-18T22:46:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1741781640 1 -- 172.21.15.40:0/3233928785 learned_addr learned my addr 172.21.15.40:0/3233928785 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1741f82640 1 --2- 172.21.15.40:0/3233928785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7c500 0x560d1ca86a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1742783640 1 --2- 172.21.15.40:0/3233928785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca82060 0x560d1ca8df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1741781640 1 -- 172.21.15.40:0/3233928785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7c500 msgr2=0x560d1ca86a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1741781640 1 --2- 172.21.15.40:0/3233928785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7c500 0x560d1ca86a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1741781640 1 -- 172.21.15.40:0/3233928785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca82060 msgr2=0x560d1ca8df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1741781640 1 --2- 172.21.15.40:0/3233928785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca82060 0x560d1ca8df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1741781640 1 -- 172.21.15.40:0/3233928785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d1c6f23f0 con 0x560d1ca7cef0 2024-03-18T22:46:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f1741781640 1 --2- 172.21.15.40:0/3233928785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7cef0 0x560d1ca86f50 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f172c02c170 tx=0x7f172c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ac6866a9466d36 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f17257fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.075+0000 7f17257fa640 1 -- 172.21.15.40:0/3233928785 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c051030 con 0x560d1ca7cef0 2024-03-18T22:46:59.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.076+0000 7f17257fa640 1 -- 172.21.15.40:0/3233928785 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f172c03a040 con 0x560d1ca7cef0 2024-03-18T22:46:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.076+0000 7f1746dd1740 1 -- 172.21.15.40:0/3233928785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f173409b910 con 0x560d1ca7cef0 2024-03-18T22:46:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.076+0000 7f17257fa640 1 -- 172.21.15.40:0/3233928785 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f172c05e050 con 0x560d1ca7cef0 2024-03-18T22:46:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.076+0000 7f1746dd1740 10 client.16662 fetch_fsmap learned FSMap version 32 2024-03-18T22:46:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.076+0000 7f1746dd1740 1 -- 172.21.15.40:0/3233928785 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560d1c7bc7d0 con 0x560d1ca7cef0 2024-03-18T22:46:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.076+0000 7f17257fa640 1 -- 172.21.15.40:0/3233928785 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f173409b910 con 0x560d1ca7cef0 2024-03-18T22:46:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.076+0000 7f17257fa640 1 -- 172.21.15.40:0/3233928785 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f172c040040 con 0x560d1ca7cef0 2024-03-18T22:46:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.076+0000 7f1746dd1740 10 client.16662 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:46:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.076+0000 7f1746dd1740 10 client.16662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:46:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.076+0000 7f1746dd1740 20 client.16662 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.076+0000 7f1746dd1740 1 --2- 172.21.15.40:0/3233928785 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d1ca92930 0x560d1ca94d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.076+0000 7f1746dd1740 4 client.16662 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:46:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.076+0000 7f1746dd1740 1 -- 172.21.15.40:0/3233928785 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d1ca3c3e0 con 0x560d1ca92930 2024-03-18T22:46:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.076+0000 7f1741f82640 1 --2- 172.21.15.40:0/3233928785 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d1ca92930 0x560d1ca94d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.077+0000 7f17257fa640 10 client.16662 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:59.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.077+0000 7f1741f82640 1 --2- 172.21.15.40:0/3233928785 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d1ca92930 0x560d1ca94d90 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec0dbc6243625e98 server_cookie=7eef238a1051effc in_seq=0 out_seq=0 2024-03-18T22:46:59.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.077+0000 7f17257fa640 1 -- 172.21.15.40:0/3233928785 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560d1ca3c3e0 con 0x560d1ca92930 2024-03-18T22:46:59.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.077+0000 7f17257fa640 10 client.16662 handle_command_reply: tid=0 2024-03-18T22:46:59.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.077+0000 7f1746dd1740 1 client.16662 shutdown 2024-03-18T22:46:59.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.078+0000 7f1746dd1740 1 -- 172.21.15.40:0/3233928785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7cef0 msgr2=0x560d1ca86f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:59.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.078+0000 7f1746dd1740 1 --2- 172.21.15.40:0/3233928785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7cef0 0x560d1ca86f50 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f172c02c170 tx=0x7f172c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:46:59.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.078+0000 7f1746dd1740 1 -- 172.21.15.40:0/3233928785 shutdown_connections 2024-03-18T22:46:59.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.078+0000 7f1746dd1740 1 --2- 172.21.15.40:0/3233928785 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d1ca92930 0x560d1ca94d90 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.078+0000 7f1746dd1740 1 --2- 172.21.15.40:0/3233928785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca82060 0x560d1ca8df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.078+0000 7f17257fa640 0 client.16662 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:59.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.078+0000 7f1746dd1740 1 --2- 172.21.15.40:0/3233928785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7cef0 0x560d1ca86f50 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.078+0000 7f1746dd1740 1 --2- 172.21.15.40:0/3233928785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7c500 0x560d1ca86a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.078+0000 7f1746dd1740 1 -- 172.21.15.40:0/3233928785 >> 172.21.15.40:0/3233928785 conn(0x560d1ca75860 msgr2=0x560d1ca74ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:59.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.078+0000 7f1746dd1740 1 -- 172.21.15.40:0/3233928785 shutdown_connections 2024-03-18T22:46:59.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.078+0000 7f1746dd1740 1 -- 172.21.15.40:0/3233928785 wait complete. 2024-03-18T22:46:59.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.078+0000 7f1746dd1740 20 client.16662 trim_cache size 0 max 16384 2024-03-18T22:46:59.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.096+0000 7f1746dd1740 1 Processor -- start 2024-03-18T22:46:59.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.096+0000 7f1746dd1740 1 -- start start 2024-03-18T22:46:59.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1746dd1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7c500 0x560d1ca791d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1746dd1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca7cef0 0x560d1ca99650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1746dd1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca82060 0x560d1ca99b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1746dd1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f173c1a8300 con 0x560d1ca82060 2024-03-18T22:46:59.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1746dd1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f173c1a9e40 con 0x560d1ca7cef0 2024-03-18T22:46:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1746dd1740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f173c1a89d0 con 0x560d1ca7c500 2024-03-18T22:46:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1742783640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca82060 0x560d1ca99b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1742783640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca82060 0x560d1ca99b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35338/0 (socket says 172.21.15.40:35338) 2024-03-18T22:46:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1741781640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca7cef0 0x560d1ca99650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1742783640 1 -- 172.21.15.40:0/2746419794 learned_addr learned my addr 172.21.15.40:0/2746419794 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1741f82640 1 --2- 172.21.15.40:0/2746419794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7c500 0x560d1ca791d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1741781640 1 -- 172.21.15.40:0/2746419794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7c500 msgr2=0x560d1ca791d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1741781640 1 --2- 172.21.15.40:0/2746419794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7c500 0x560d1ca791d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1741781640 1 -- 172.21.15.40:0/2746419794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca82060 msgr2=0x560d1ca99b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1741781640 1 --2- 172.21.15.40:0/2746419794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca82060 0x560d1ca99b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1741781640 1 -- 172.21.15.40:0/2746419794 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d1ca7d2d0 con 0x560d1ca7cef0 2024-03-18T22:46:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1741f82640 1 --2- 172.21.15.40:0/2746419794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7c500 0x560d1ca791d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:46:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.097+0000 7f1741781640 1 --2- 172.21.15.40:0/2746419794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca7cef0 0x560d1ca99650 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f172c003b00 tx=0x7f172c02c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7804fc413ec1a41 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:59.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.098+0000 7f17277fe640 1 -- 172.21.15.40:0/2746419794 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c051020 con 0x560d1ca7cef0 2024-03-18T22:46:59.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.098+0000 7f17277fe640 1 -- 172.21.15.40:0/2746419794 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f172c003d70 con 0x560d1ca7cef0 2024-03-18T22:46:59.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.098+0000 7f17277fe640 1 -- 172.21.15.40:0/2746419794 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c049070 con 0x560d1ca7cef0 2024-03-18T22:46:59.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.099+0000 7f1746dd1740 1 -- 172.21.15.40:0/2746419794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca7cef0 msgr2=0x560d1ca99650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:59.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.099+0000 7f1746dd1740 1 --2- 172.21.15.40:0/2746419794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca7cef0 0x560d1ca99650 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f172c003b00 tx=0x7f172c02c330 comp rx=0 tx=0).stop 2024-03-18T22:46:59.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.099+0000 7f1746dd1740 1 -- 172.21.15.40:0/2746419794 shutdown_connections 2024-03-18T22:46:59.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.099+0000 7f1746dd1740 1 --2- 172.21.15.40:0/2746419794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca82060 0x560d1ca99b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.099+0000 7f1746dd1740 1 --2- 172.21.15.40:0/2746419794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca7cef0 0x560d1ca99650 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.099+0000 7f1746dd1740 1 --2- 172.21.15.40:0/2746419794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca7c500 0x560d1ca791d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.099+0000 7f1746dd1740 1 -- 172.21.15.40:0/2746419794 >> 172.21.15.40:0/2746419794 conn(0x560d1ca75860 msgr2=0x560d1ca78ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:59.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.099+0000 7f1746dd1740 1 -- 172.21.15.40:0/2746419794 shutdown_connections 2024-03-18T22:46:59.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.099+0000 7f1746dd1740 1 -- 172.21.15.40:0/2746419794 wait complete. 2024-03-18T22:46:59.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.099+0000 7f1746dd1740 1 -- start start 2024-03-18T22:46:59.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.099+0000 7f1746dd1740 1 Processor -- start 2024-03-18T22:46:59.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1746dd1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca7c500 0x560d1caf4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1746dd1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7cef0 0x560d1caf71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1746dd1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca82060 0x560d1caf9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1746dd1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f173c1a9770 con 0x560d1ca7cef0 2024-03-18T22:46:59.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1746dd1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f173c1a90a0 con 0x560d1ca7c500 2024-03-18T22:46:59.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1746dd1740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f173c1a62e0 con 0x560d1ca82060 2024-03-18T22:46:59.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1746dd1740 10 client.0 _ll_register_callbacks cb 0x7f172c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:46:59.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1741781640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7cef0 0x560d1caf71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1741781640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7cef0 0x560d1caf71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35354/0 (socket says 172.21.15.40:35354) 2024-03-18T22:46:59.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1741781640 1 -- 172.21.15.40:0/4219241792 learned_addr learned my addr 172.21.15.40:0/4219241792 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:46:59.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1741f82640 1 --2- 172.21.15.40:0/4219241792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca7c500 0x560d1caf4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1741781640 1 -- 172.21.15.40:0/4219241792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca82060 msgr2=0x560d1caf9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:59.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1741781640 1 --2- 172.21.15.40:0/4219241792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca82060 0x560d1caf9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1741781640 1 -- 172.21.15.40:0/4219241792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca7c500 msgr2=0x560d1caf4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:46:59.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1741781640 1 --2- 172.21.15.40:0/4219241792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca7c500 0x560d1caf4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1741781640 1 -- 172.21.15.40:0/4219241792 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d1caf9e00 con 0x560d1ca7cef0 2024-03-18T22:46:59.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f1741781640 1 --2- 172.21.15.40:0/4219241792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7cef0 0x560d1caf71c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f172c0095a0 tx=0x7f172c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b996390b84bc1f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:46:59.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.100+0000 7f17257fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:46:59.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.102+0000 7f17257fa640 1 -- 172.21.15.40:0/4219241792 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c051030 con 0x560d1ca7cef0 2024-03-18T22:46:59.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.102+0000 7f1746dd1740 1 -- 172.21.15.40:0/4219241792 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f172c003d70 con 0x560d1ca7cef0 2024-03-18T22:46:59.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.102+0000 7f17257fa640 1 -- 172.21.15.40:0/4219241792 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f172c0035f0 con 0x560d1ca7cef0 2024-03-18T22:46:59.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.102+0000 7f17257fa640 1 -- 172.21.15.40:0/4219241792 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f172c071050 con 0x560d1ca7cef0 2024-03-18T22:46:59.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.102+0000 7f1746dd1740 10 client.16668 fetch_fsmap learned FSMap version 32 2024-03-18T22:46:59.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.102+0000 7f1746dd1740 1 -- 172.21.15.40:0/4219241792 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560d1cafb0e0 con 0x560d1ca7cef0 2024-03-18T22:46:59.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.103+0000 7f17257fa640 1 -- 172.21.15.40:0/4219241792 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f172c003d70 con 0x560d1ca7cef0 2024-03-18T22:46:59.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.103+0000 7f17257fa640 1 -- 172.21.15.40:0/4219241792 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f172c06d4f0 con 0x560d1ca7cef0 2024-03-18T22:46:59.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.103+0000 7f1746dd1740 10 client.16668 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:46:59.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.103+0000 7f1746dd1740 10 client.16668 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:46:59.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.103+0000 7f1746dd1740 20 client.16668 populate_metadata read hostname 'smithi040' 2024-03-18T22:46:59.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.103+0000 7f1746dd1740 1 --2- 172.21.15.40:0/4219241792 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d1cafba30 0x560d1cafde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:46:59.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.103+0000 7f1746dd1740 4 client.16668 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:46:59.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.103+0000 7f1746dd1740 1 -- 172.21.15.40:0/4219241792 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d1c9eb9f0 con 0x560d1cafba30 2024-03-18T22:46:59.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.103+0000 7f1741f82640 1 --2- 172.21.15.40:0/4219241792 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d1cafba30 0x560d1cafde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:46:59.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f1741f82640 1 --2- 172.21.15.40:0/4219241792 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d1cafba30 0x560d1cafde50 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df15afc1153847c1 server_cookie=2e4c5e7b766d34fd in_seq=0 out_seq=0 2024-03-18T22:46:59.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f17257fa640 10 client.16668 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:59.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f17257fa640 1 -- 172.21.15.40:0/4219241792 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d1c9eb9f0 con 0x560d1cafba30 2024-03-18T22:46:59.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f17257fa640 10 client.16668 handle_command_reply: tid=0 2024-03-18T22:46:59.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f1746dd1740 1 client.16668 shutdown 2024-03-18T22:46:59.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f1746dd1740 1 -- 172.21.15.40:0/4219241792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7cef0 msgr2=0x560d1caf71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:59.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f1746dd1740 1 --2- 172.21.15.40:0/4219241792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7cef0 0x560d1caf71c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f172c0095a0 tx=0x7f172c005fd0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f1746dd1740 1 -- 172.21.15.40:0/4219241792 shutdown_connections 2024-03-18T22:46:59.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f1746dd1740 1 --2- 172.21.15.40:0/4219241792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d1ca82060 0x560d1caf9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f1746dd1740 1 --2- 172.21.15.40:0/4219241792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d1ca7cef0 0x560d1caf71c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f1746dd1740 1 --2- 172.21.15.40:0/4219241792 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d1cafba30 0x560d1cafde50 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f1746dd1740 1 --2- 172.21.15.40:0/4219241792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d1ca7c500 0x560d1caf4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f17257fa640 0 client.16668 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:46:59.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f1746dd1740 1 -- 172.21.15.40:0/4219241792 >> 172.21.15.40:0/4219241792 conn(0x560d1ca75860 msgr2=0x560d1ca79350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:59.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f1746dd1740 1 -- 172.21.15.40:0/4219241792 shutdown_connections 2024-03-18T22:46:59.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f1746dd1740 1 -- 172.21.15.40:0/4219241792 wait complete. 2024-03-18T22:46:59.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.104+0000 7f1746dd1740 20 client.16668 trim_cache size 0 max 16384 2024-03-18T22:46:59.111 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:46:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.105+0000 7f17441e6640 1 -- 172.21.15.40:0/1284011361 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17280a2880 msgr2=0x7f17280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.105+0000 7f17441e6640 1 --2- 172.21.15.40:0/1284011361 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17280a2880 0x7f17280a4d40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f172c009230 tx=0x7f172c02cb90 comp rx=0 tx=0).stop 2024-03-18T22:46:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.105+0000 7f17441e6640 1 -- 172.21.15.40:0/1284011361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f173c1022d0 msgr2=0x7f173c19dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:46:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.105+0000 7f17441e6640 1 --2- 172.21.15.40:0/1284011361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f173c1022d0 0x7f173c19dec0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f173400eb10 tx=0x7f173400c2d0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.105+0000 7f17441e6640 1 -- 172.21.15.40:0/1284011361 shutdown_connections 2024-03-18T22:46:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.105+0000 7f17441e6640 1 --2- 172.21.15.40:0/1284011361 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17280a2880 0x7f17280a4d40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.105+0000 7f17441e6640 1 --2- 172.21.15.40:0/1284011361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f173c1022d0 0x7f173c19dec0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.105+0000 7f17441e6640 1 --2- 172.21.15.40:0/1284011361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f173c101910 0x7f173c196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.105+0000 7f17441e6640 1 --2- 172.21.15.40:0/1284011361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f173c100720 0x7f173c196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:46:59.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.105+0000 7f17441e6640 1 -- 172.21.15.40:0/1284011361 >> 172.21.15.40:0/1284011361 conn(0x7f173c0fbc90 msgr2=0x7f173c104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:46:59.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.105+0000 7f17441e6640 1 -- 172.21.15.40:0/1284011361 shutdown_connections 2024-03-18T22:46:59.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:46:59.105+0000 7f17441e6640 1 -- 172.21.15.40:0/1284011361 wait complete. 2024-03-18T22:46:59.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:46:59 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2355180436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:46:59.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:46:59 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2355180436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:46:59.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:46:59 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2355180436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:47:00.121 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:47:00.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:47:00.121 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:47:00.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.235+0000 7fe394b53640 1 -- 172.21.15.40:0/3748632294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe390102a20 msgr2=0x7fe390102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.235+0000 7fe394b53640 1 --2- 172.21.15.40:0/3748632294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe390102a20 0x7fe390102ea0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe384009900 tx=0x7fe38402bc30 comp rx=0 tx=0).stop 2024-03-18T22:47:00.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.236+0000 7fe394b53640 1 -- 172.21.15.40:0/3748632294 shutdown_connections 2024-03-18T22:47:00.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.236+0000 7fe394b53640 1 --2- 172.21.15.40:0/3748632294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3901033e0 0x7fe390109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.236+0000 7fe394b53640 1 --2- 172.21.15.40:0/3748632294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe390102a20 0x7fe390102ea0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.236+0000 7fe394b53640 1 --2- 172.21.15.40:0/3748632294 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe390101830 0x7fe390101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.236+0000 7fe394b53640 1 -- 172.21.15.40:0/3748632294 >> 172.21.15.40:0/3748632294 conn(0x7fe3900fcda0 msgr2=0x7fe3900ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:00.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.236+0000 7fe394b53640 1 -- 172.21.15.40:0/3748632294 shutdown_connections 2024-03-18T22:47:00.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.236+0000 7fe394b53640 1 -- 172.21.15.40:0/3748632294 wait complete. 2024-03-18T22:47:00.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.236+0000 7fe394b53640 1 Processor -- start 2024-03-18T22:47:00.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe394b53640 1 -- start start 2024-03-18T22:47:00.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe394b53640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe390101830 0x7fe390197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe394b53640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe390102a20 0x7fe390197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe394b53640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3901033e0 0x7fe39019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe394b53640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe39010d360 con 0x7fe390101830 2024-03-18T22:47:00.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe394b53640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe39010d1e0 con 0x7fe390102a20 2024-03-18T22:47:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe394b53640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe39010d4e0 con 0x7fe3901033e0 2024-03-18T22:47:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe38e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe390101830 0x7fe390197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe38e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe390101830 0x7fe390197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35374/0 (socket says 172.21.15.40:35374) 2024-03-18T22:47:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe38e59c640 1 -- 172.21.15.40:0/3195418992 learned_addr learned my addr 172.21.15.40:0/3195418992 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe38ed9d640 1 --2- 172.21.15.40:0/3195418992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3901033e0 0x7fe39019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe38dd9b640 1 --2- 172.21.15.40:0/3195418992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe390102a20 0x7fe390197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe38e59c640 1 -- 172.21.15.40:0/3195418992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3901033e0 msgr2=0x7fe39019f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe38e59c640 1 --2- 172.21.15.40:0/3195418992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3901033e0 0x7fe39019f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe38e59c640 1 -- 172.21.15.40:0/3195418992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe390102a20 msgr2=0x7fe390197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe38e59c640 1 --2- 172.21.15.40:0/3195418992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe390102a20 0x7fe390197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe38e59c640 1 -- 172.21.15.40:0/3195418992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe39019f6b0 con 0x7fe390101830 2024-03-18T22:47:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.237+0000 7fe38e59c640 1 --2- 172.21.15.40:0/3195418992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe390101830 0x7fe390197a90 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe38000e9c0 tx=0x7fe38000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.238+0000 7fe37f7fe640 1 -- 172.21.15.40:0/3195418992 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38000cd80 con 0x7fe390101830 2024-03-18T22:47:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.238+0000 7fe37f7fe640 1 -- 172.21.15.40:0/3195418992 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3800035f0 con 0x7fe390101830 2024-03-18T22:47:00.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.238+0000 7fe394b53640 1 -- 172.21.15.40:0/3195418992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe39019f9a0 con 0x7fe390101830 2024-03-18T22:47:00.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.238+0000 7fe37f7fe640 1 -- 172.21.15.40:0/3195418992 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe380010690 con 0x7fe390101830 2024-03-18T22:47:00.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.238+0000 7fe394b53640 1 -- 172.21.15.40:0/3195418992 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe39019fe70 con 0x7fe390101830 2024-03-18T22:47:00.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.240+0000 7fe37f7fe640 1 -- 172.21.15.40:0/3195418992 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe3800037a0 con 0x7fe390101830 2024-03-18T22:47:00.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.241+0000 7fe37f7fe640 1 --2- 172.21.15.40:0/3195418992 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe3780a2880 0x7fe3780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.241+0000 7fe39781e740 1 Processor -- start 2024-03-18T22:47:00.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.241+0000 7fe38dd9b640 1 --2- 172.21.15.40:0/3195418992 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe3780a2880 0x7fe3780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.241+0000 7fe37f7fe640 1 -- 172.21.15.40:0/3195418992 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe3800cf2f0 con 0x7fe390101830 2024-03-18T22:47:00.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.241+0000 7fe39781e740 1 -- start start 2024-03-18T22:47:00.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.241+0000 7fe39781e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40b3a0 0x561c0b40b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.241+0000 7fe39781e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40bd90 0x561c0b410890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.241+0000 7fe39781e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b410f00 0x561c0b413390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.241+0000 7fe39781e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561c0b370160 con 0x561c0b40bd90 2024-03-18T22:47:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.241+0000 7fe39781e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561c0b318e10 con 0x561c0b40b3a0 2024-03-18T22:47:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.241+0000 7fe39781e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x561c0b36c200 con 0x561c0b410f00 2024-03-18T22:47:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.242+0000 7fe38e59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40b3a0 0x561c0b40b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.242+0000 7fe38dd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40bd90 0x561c0b410890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.242+0000 7fe38e59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40b3a0 0x561c0b40b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:39054/0 (socket says 172.21.15.40:39054) 2024-03-18T22:47:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.242+0000 7fe38e59c640 1 -- 172.21.15.40:0/1554561782 learned_addr learned my addr 172.21.15.40:0/1554561782 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.242+0000 7fe38ed9d640 1 --2- 172.21.15.40:0/1554561782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b410f00 0x561c0b413390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.242+0000 7fe38dd9b640 1 --2- 172.21.15.40:0/3195418992 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe3780a2880 0x7fe3780a4d40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe384009230 tx=0x7fe384005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.242+0000 7fe38e59c640 1 -- 172.21.15.40:0/1554561782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b410f00 msgr2=0x561c0b413390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.242+0000 7fe38e59c640 1 --2- 172.21.15.40:0/1554561782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b410f00 0x561c0b413390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.242+0000 7fe38e59c640 1 -- 172.21.15.40:0/1554561782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40bd90 msgr2=0x561c0b410890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.242+0000 7fe38e59c640 1 --2- 172.21.15.40:0/1554561782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40bd90 0x561c0b410890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.242+0000 7fe38e59c640 1 -- 172.21.15.40:0/1554561782 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c0b14c7e0 con 0x561c0b40b3a0 2024-03-18T22:47:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.242+0000 7fe38e59c640 1 --2- 172.21.15.40:0/1554561782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40b3a0 0x561c0b40b7c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe3800a3b10 tx=0x7fe38009b200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff978b01edf40ce3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:00.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.242+0000 7fe37cff9640 1 -- 172.21.15.40:0/1554561782 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38009b8a0 con 0x561c0b40b3a0 2024-03-18T22:47:00.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.242+0000 7fe37cff9640 1 -- 172.21.15.40:0/1554561782 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe38009f9f0 con 0x561c0b40b3a0 2024-03-18T22:47:00.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.242+0000 7fe37cff9640 1 -- 172.21.15.40:0/1554561782 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3800980a0 con 0x561c0b40b3a0 2024-03-18T22:47:00.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.243+0000 7fe39781e740 1 -- 172.21.15.40:0/1554561782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40b3a0 msgr2=0x561c0b40b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.243+0000 7fe39781e740 1 --2- 172.21.15.40:0/1554561782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40b3a0 0x561c0b40b7c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe3800a3b10 tx=0x7fe38009b200 comp rx=0 tx=0).stop 2024-03-18T22:47:00.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.243+0000 7fe39781e740 1 -- 172.21.15.40:0/1554561782 shutdown_connections 2024-03-18T22:47:00.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.243+0000 7fe39781e740 1 --2- 172.21.15.40:0/1554561782 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b410f00 0x561c0b413390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.243+0000 7fe39781e740 1 --2- 172.21.15.40:0/1554561782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40bd90 0x561c0b410890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.243+0000 7fe39781e740 1 --2- 172.21.15.40:0/1554561782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40b3a0 0x561c0b40b7c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.243+0000 7fe39781e740 1 -- 172.21.15.40:0/1554561782 >> 172.21.15.40:0/1554561782 conn(0x561c0b406910 msgr2=0x561c0b406d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:00.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.243+0000 7fe39781e740 1 -- 172.21.15.40:0/1554561782 shutdown_connections 2024-03-18T22:47:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.243+0000 7fe39781e740 1 -- 172.21.15.40:0/1554561782 wait complete. 2024-03-18T22:47:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.243+0000 7fe39781e740 1 -- start start 2024-03-18T22:47:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe39781e740 1 Processor -- start 2024-03-18T22:47:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe39781e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40b3a0 0x561c0b4157d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe39781e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b40bd90 0x561c0b415d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe39781e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b410f00 0x561c0b41cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe39781e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561c0b413e50 con 0x561c0b410f00 2024-03-18T22:47:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe39781e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561c0b32c1a0 con 0x561c0b40b3a0 2024-03-18T22:47:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe39781e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x561c0b413fd0 con 0x561c0b40bd90 2024-03-18T22:47:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe39781e740 10 client.0 _ll_register_callbacks cb 0x561c0b083c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe38dd9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b40bd90 0x561c0b415d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe38ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b410f00 0x561c0b41cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe38e59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40b3a0 0x561c0b4157d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe38dd9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b40bd90 0x561c0b415d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57138/0 (socket says 172.21.15.40:57138) 2024-03-18T22:47:00.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe38dd9b640 1 -- 172.21.15.40:0/2737534417 learned_addr learned my addr 172.21.15.40:0/2737534417 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe38ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b410f00 0x561c0b41cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35398/0 (socket says 172.21.15.40:35398) 2024-03-18T22:47:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe38dd9b640 1 -- 172.21.15.40:0/2737534417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40b3a0 msgr2=0x561c0b4157d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe38dd9b640 1 --2- 172.21.15.40:0/2737534417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40b3a0 0x561c0b4157d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe38dd9b640 1 -- 172.21.15.40:0/2737534417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b410f00 msgr2=0x561c0b41cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe38dd9b640 1 --2- 172.21.15.40:0/2737534417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b410f00 0x561c0b41cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe38dd9b640 1 -- 172.21.15.40:0/2737534417 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c0b1acf70 con 0x561c0b40bd90 2024-03-18T22:47:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.244+0000 7fe38ed9d640 1 --2- 172.21.15.40:0/2737534417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b410f00 0x561c0b41cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe38dd9b640 1 --2- 172.21.15.40:0/2737534417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b40bd90 0x561c0b415d10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe38402c170 tx=0x7fe384009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ceed2fe3302af67 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe375ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:47:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe375ffb640 1 -- 172.21.15.40:0/2737534417 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38404f030 con 0x561c0b40bd90 2024-03-18T22:47:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe375ffb640 1 -- 172.21.15.40:0/2737534417 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe38403a040 con 0x561c0b40bd90 2024-03-18T22:47:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe39781e740 1 -- 172.21.15.40:0/2737534417 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe38009f9f0 con 0x561c0b40bd90 2024-03-18T22:47:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe375ffb640 1 -- 172.21.15.40:0/2737534417 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe38405c050 con 0x561c0b40bd90 2024-03-18T22:47:00.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe39781e740 10 client.26497 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe39781e740 1 -- 172.21.15.40:0/2737534417 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561c0b044170 con 0x561c0b40bd90 2024-03-18T22:47:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe375ffb640 1 -- 172.21.15.40:0/2737534417 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe38009f9f0 con 0x561c0b40bd90 2024-03-18T22:47:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe375ffb640 1 -- 172.21.15.40:0/2737534417 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe38404a040 con 0x561c0b40bd90 2024-03-18T22:47:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe39781e740 10 client.26497 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe39781e740 10 client.26497 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:47:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe39781e740 20 client.26497 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe39781e740 1 --2- 172.21.15.40:0/2737534417 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561c0b4216f0 0x561c0b423b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe39781e740 4 client.26497 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:47:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.245+0000 7fe39781e740 1 -- 172.21.15.40:0/2737534417 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c0b37d6a0 con 0x561c0b4216f0 2024-03-18T22:47:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.246+0000 7fe38e59c640 1 --2- 172.21.15.40:0/2737534417 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561c0b4216f0 0x561c0b423b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.246+0000 7fe38e59c640 1 --2- 172.21.15.40:0/2737534417 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561c0b4216f0 0x561c0b423b70 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ecd73db29af908 server_cookie=ec32f412d9a40ae9 in_seq=0 out_seq=0 2024-03-18T22:47:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.246+0000 7fe375ffb640 10 client.26497 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.247+0000 7fe375ffb640 1 -- 172.21.15.40:0/2737534417 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561c0b37d6a0 con 0x561c0b4216f0 2024-03-18T22:47:00.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.247+0000 7fe375ffb640 10 client.26497 handle_command_reply: tid=0 2024-03-18T22:47:00.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.247+0000 7fe39781e740 1 client.26497 shutdown 2024-03-18T22:47:00.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.247+0000 7fe39781e740 1 -- 172.21.15.40:0/2737534417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b40bd90 msgr2=0x561c0b415d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.247+0000 7fe39781e740 1 --2- 172.21.15.40:0/2737534417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b40bd90 0x561c0b415d10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe38402c170 tx=0x7fe384009260 comp rx=0 tx=0).stop 2024-03-18T22:47:00.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.247+0000 7fe39781e740 1 -- 172.21.15.40:0/2737534417 shutdown_connections 2024-03-18T22:47:00.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.247+0000 7fe39781e740 1 --2- 172.21.15.40:0/2737534417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b410f00 0x561c0b41cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.247+0000 7fe39781e740 1 --2- 172.21.15.40:0/2737534417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b40bd90 0x561c0b415d10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.247+0000 7fe39781e740 1 --2- 172.21.15.40:0/2737534417 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561c0b4216f0 0x561c0b423b70 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.247+0000 7fe39781e740 1 --2- 172.21.15.40:0/2737534417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40b3a0 0x561c0b4157d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.247+0000 7fe39781e740 1 -- 172.21.15.40:0/2737534417 >> 172.21.15.40:0/2737534417 conn(0x561c0b406910 msgr2=0x561c0b405b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:00.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.247+0000 7fe375ffb640 0 client.26497 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:00.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.247+0000 7fe39781e740 1 -- 172.21.15.40:0/2737534417 shutdown_connections 2024-03-18T22:47:00.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.247+0000 7fe39781e740 1 -- 172.21.15.40:0/2737534417 wait complete. 2024-03-18T22:47:00.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.247+0000 7fe39781e740 20 client.26497 trim_cache size 0 max 16384 2024-03-18T22:47:00.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.269+0000 7fe39781e740 1 Processor -- start 2024-03-18T22:47:00.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.269+0000 7fe39781e740 1 -- start start 2024-03-18T22:47:00.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.269+0000 7fe39781e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b40b3a0 0x561c0b4157d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.269+0000 7fe39781e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40bd90 0x561c0b415d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.269+0000 7fe39781e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b410f00 0x561c0b4052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.269+0000 7fe39781e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe3901aa4d0 con 0x561c0b40bd90 2024-03-18T22:47:00.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.269+0000 7fe39781e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe3901ac010 con 0x561c0b410f00 2024-03-18T22:47:00.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.269+0000 7fe39781e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe3901aaba0 con 0x561c0b40b3a0 2024-03-18T22:47:00.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.270+0000 7fe38e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b40b3a0 0x561c0b4157d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.270+0000 7fe38e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b40b3a0 0x561c0b4157d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57142/0 (socket says 172.21.15.40:57142) 2024-03-18T22:47:00.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.270+0000 7fe38e59c640 1 -- 172.21.15.40:0/284401720 learned_addr learned my addr 172.21.15.40:0/284401720 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:00.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.270+0000 7fe38dd9b640 1 --2- 172.21.15.40:0/284401720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40bd90 0x561c0b415d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.270+0000 7fe38e59c640 1 -- 172.21.15.40:0/284401720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b410f00 msgr2=0x561c0b4052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.270+0000 7fe38e59c640 1 --2- 172.21.15.40:0/284401720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b410f00 0x561c0b4052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.270+0000 7fe38e59c640 1 -- 172.21.15.40:0/284401720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40bd90 msgr2=0x561c0b415d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.270+0000 7fe38e59c640 1 --2- 172.21.15.40:0/284401720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40bd90 0x561c0b415d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.270+0000 7fe38e59c640 1 -- 172.21.15.40:0/284401720 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c0b402160 con 0x561c0b40b3a0 2024-03-18T22:47:00.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.270+0000 7fe38dd9b640 1 --2- 172.21.15.40:0/284401720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40bd90 0x561c0b415d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:00.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.270+0000 7fe38e59c640 1 --2- 172.21.15.40:0/284401720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b40b3a0 0x561c0b4157d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe38000c6a0 tx=0x7fe38009b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90b6d74165031d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:00.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.270+0000 7fe377fff640 1 -- 172.21.15.40:0/284401720 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3800b9070 con 0x561c0b40b3a0 2024-03-18T22:47:00.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.270+0000 7fe377fff640 1 -- 172.21.15.40:0/284401720 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe38009b8a0 con 0x561c0b40b3a0 2024-03-18T22:47:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.270+0000 7fe377fff640 1 -- 172.21.15.40:0/284401720 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38009cbd0 con 0x561c0b40b3a0 2024-03-18T22:47:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.271+0000 7fe39781e740 1 -- 172.21.15.40:0/284401720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b40b3a0 msgr2=0x561c0b4157d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.271+0000 7fe39781e740 1 --2- 172.21.15.40:0/284401720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b40b3a0 0x561c0b4157d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe38000c6a0 tx=0x7fe38009b330 comp rx=0 tx=0).stop 2024-03-18T22:47:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.271+0000 7fe39781e740 1 -- 172.21.15.40:0/284401720 shutdown_connections 2024-03-18T22:47:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.271+0000 7fe39781e740 1 --2- 172.21.15.40:0/284401720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b410f00 0x561c0b4052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.271+0000 7fe39781e740 1 --2- 172.21.15.40:0/284401720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40bd90 0x561c0b415d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.271+0000 7fe39781e740 1 --2- 172.21.15.40:0/284401720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b40b3a0 0x561c0b4157d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.271+0000 7fe39781e740 1 -- 172.21.15.40:0/284401720 >> 172.21.15.40:0/284401720 conn(0x561c0b406910 msgr2=0x561c0b40d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.271+0000 7fe39781e740 1 -- 172.21.15.40:0/284401720 shutdown_connections 2024-03-18T22:47:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.271+0000 7fe39781e740 1 -- 172.21.15.40:0/284401720 wait complete. 2024-03-18T22:47:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.272+0000 7fe39781e740 1 -- start start 2024-03-18T22:47:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.272+0000 7fe39781e740 1 Processor -- start 2024-03-18T22:47:00.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.272+0000 7fe39781e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40b3a0 0x561c0b482ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.272+0000 7fe39781e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40bd90 0x561c0b483220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.272+0000 7fe39781e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b410f00 0x561c0b48a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.272+0000 7fe39781e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe3901ab940 con 0x561c0b40b3a0 2024-03-18T22:47:00.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.272+0000 7fe39781e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe3901ab270 con 0x561c0b40bd90 2024-03-18T22:47:00.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.272+0000 7fe39781e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe3901a84b0 con 0x561c0b410f00 2024-03-18T22:47:00.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.272+0000 7fe39781e740 10 client.0 _ll_register_callbacks cb 0x7fe384009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:00.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.272+0000 7fe38e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40b3a0 0x561c0b482ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.272+0000 7fe38e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40b3a0 0x561c0b482ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35416/0 (socket says 172.21.15.40:35416) 2024-03-18T22:47:00.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.272+0000 7fe38e59c640 1 -- 172.21.15.40:0/1588686268 learned_addr learned my addr 172.21.15.40:0/1588686268 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:00.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.272+0000 7fe38ed9d640 1 --2- 172.21.15.40:0/1588686268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b410f00 0x561c0b48a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.273+0000 7fe38e59c640 1 -- 172.21.15.40:0/1588686268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b410f00 msgr2=0x561c0b48a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.273+0000 7fe38e59c640 1 --2- 172.21.15.40:0/1588686268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b410f00 0x561c0b48a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.273+0000 7fe38e59c640 1 -- 172.21.15.40:0/1588686268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40bd90 msgr2=0x561c0b483220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.273+0000 7fe38e59c640 1 --2- 172.21.15.40:0/1588686268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40bd90 0x561c0b483220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.273+0000 7fe38e59c640 1 -- 172.21.15.40:0/1588686268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c0b48a850 con 0x561c0b40b3a0 2024-03-18T22:47:00.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.273+0000 7fe38e59c640 1 --2- 172.21.15.40:0/1588686268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40b3a0 0x561c0b482ce0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe3800cca50 tx=0x7fe380003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3f1ce7d20925c7d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:00.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.273+0000 7fe375ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:47:00.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.273+0000 7fe375ffb640 1 -- 172.21.15.40:0/1588686268 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe380080060 con 0x561c0b40b3a0 2024-03-18T22:47:00.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.273+0000 7fe375ffb640 1 -- 172.21.15.40:0/1588686268 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe3800860b0 con 0x561c0b40b3a0 2024-03-18T22:47:00.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.273+0000 7fe39781e740 1 -- 172.21.15.40:0/1588686268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe38009b8a0 con 0x561c0b40b3a0 2024-03-18T22:47:00.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.273+0000 7fe375ffb640 1 -- 172.21.15.40:0/1588686268 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe3800b9070 con 0x561c0b40b3a0 2024-03-18T22:47:00.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.273+0000 7fe39781e740 10 client.16686 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:00.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.273+0000 7fe39781e740 1 -- 172.21.15.40:0/1588686268 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561c0b48bb90 con 0x561c0b40b3a0 2024-03-18T22:47:00.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.273+0000 7fe375ffb640 1 -- 172.21.15.40:0/1588686268 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe38009b8a0 con 0x561c0b40b3a0 2024-03-18T22:47:00.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.274+0000 7fe375ffb640 1 -- 172.21.15.40:0/1588686268 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe380097710 con 0x561c0b40b3a0 2024-03-18T22:47:00.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.274+0000 7fe39781e740 10 client.16686 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:00.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.274+0000 7fe39781e740 10 client.16686 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:47:00.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.274+0000 7fe39781e740 20 client.16686 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:00.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.274+0000 7fe39781e740 1 --2- 172.21.15.40:0/1588686268 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561c0b48be10 0x561c0b48e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.274+0000 7fe39781e740 4 client.16686 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:47:00.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.274+0000 7fe39781e740 1 -- 172.21.15.40:0/1588686268 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561c0b3779f0 con 0x561c0b48be10 2024-03-18T22:47:00.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.274+0000 7fe38dd9b640 1 --2- 172.21.15.40:0/1588686268 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561c0b48be10 0x561c0b48e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.274+0000 7fe375ffb640 10 client.16686 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:00.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.274+0000 7fe38dd9b640 1 --2- 172.21.15.40:0/1588686268 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561c0b48be10 0x561c0b48e230 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=122058ee4adc14b0 server_cookie=932e05a19831a7b2 in_seq=0 out_seq=0 2024-03-18T22:47:00.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.275+0000 7fe375ffb640 1 -- 172.21.15.40:0/1588686268 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561c0b3779f0 con 0x561c0b48be10 2024-03-18T22:47:00.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.275+0000 7fe375ffb640 10 client.16686 handle_command_reply: tid=0 2024-03-18T22:47:00.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.275+0000 7fe39781e740 1 client.16686 shutdown 2024-03-18T22:47:00.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.276+0000 7fe39781e740 1 -- 172.21.15.40:0/1588686268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40b3a0 msgr2=0x561c0b482ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.276+0000 7fe39781e740 1 --2- 172.21.15.40:0/1588686268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40b3a0 0x561c0b482ce0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe3800cca50 tx=0x7fe380003c00 comp rx=0 tx=0).stop 2024-03-18T22:47:00.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.276+0000 7fe39781e740 1 -- 172.21.15.40:0/1588686268 shutdown_connections 2024-03-18T22:47:00.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.276+0000 7fe39781e740 1 --2- 172.21.15.40:0/1588686268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561c0b410f00 0x561c0b48a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.276+0000 7fe39781e740 1 --2- 172.21.15.40:0/1588686268 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561c0b48be10 0x561c0b48e230 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.276+0000 7fe39781e740 1 --2- 172.21.15.40:0/1588686268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561c0b40bd90 0x561c0b483220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.276+0000 7fe39781e740 1 --2- 172.21.15.40:0/1588686268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561c0b40b3a0 0x561c0b482ce0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.276+0000 7fe375ffb640 0 client.16686 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:00.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.276+0000 7fe39781e740 1 -- 172.21.15.40:0/1588686268 >> 172.21.15.40:0/1588686268 conn(0x561c0b406910 msgr2=0x561c0b415740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:00.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.276+0000 7fe39781e740 1 -- 172.21.15.40:0/1588686268 shutdown_connections 2024-03-18T22:47:00.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.276+0000 7fe39781e740 1 -- 172.21.15.40:0/1588686268 wait complete. 2024-03-18T22:47:00.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.276+0000 7fe39781e740 20 client.16686 trim_cache size 0 max 16384 2024-03-18T22:47:00.283 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:47:00.283 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:47:00.283 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "3630afb1-6156-42e1-a0af-8dbbc6fec421", 2024-03-18T22:47:00.283 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:47:00.283 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:47:00.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.277+0000 7fe394b53640 1 -- 172.21.15.40:0/3195418992 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe3780a2880 msgr2=0x7fe3780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.277+0000 7fe394b53640 1 --2- 172.21.15.40:0/3195418992 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe3780a2880 0x7fe3780a4d40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe384009230 tx=0x7fe384005fd0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.277+0000 7fe394b53640 1 -- 172.21.15.40:0/3195418992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe390101830 msgr2=0x7fe390197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.277+0000 7fe394b53640 1 --2- 172.21.15.40:0/3195418992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe390101830 0x7fe390197a90 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe38000e9c0 tx=0x7fe38000ee80 comp rx=0 tx=0).stop 2024-03-18T22:47:00.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.277+0000 7fe394b53640 1 -- 172.21.15.40:0/3195418992 shutdown_connections 2024-03-18T22:47:00.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.277+0000 7fe394b53640 1 --2- 172.21.15.40:0/3195418992 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe3780a2880 0x7fe3780a4d40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.277+0000 7fe394b53640 1 --2- 172.21.15.40:0/3195418992 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe3901033e0 0x7fe39019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.277+0000 7fe394b53640 1 --2- 172.21.15.40:0/3195418992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe390102a20 0x7fe390197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.277+0000 7fe394b53640 1 --2- 172.21.15.40:0/3195418992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe390101830 0x7fe390197a90 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.277+0000 7fe394b53640 1 -- 172.21.15.40:0/3195418992 >> 172.21.15.40:0/3195418992 conn(0x7fe3900fcda0 msgr2=0x7fe390105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:00.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.277+0000 7fe394b53640 1 -- 172.21.15.40:0/3195418992 shutdown_connections 2024-03-18T22:47:00.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.277+0000 7fe394b53640 1 -- 172.21.15.40:0/3195418992 wait complete. 2024-03-18T22:47:00.292 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:47:00.292 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:47:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:00 smithi103 ceph-mon[66377]: pgmap v481: 129 pgs: 129 active+clean; 4.6 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 26 MiB/s wr, 19 op/s 2024-03-18T22:47:00.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:00 smithi040 ceph-mon[62643]: pgmap v481: 129 pgs: 129 active+clean; 4.6 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 26 MiB/s wr, 19 op/s 2024-03-18T22:47:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.402+0000 7fd98b44b640 1 -- 172.21.15.40:0/1638070378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd984101910 msgr2=0x7fd984101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.402+0000 7fd98b44b640 1 --2- 172.21.15.40:0/1638070378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd984101910 0x7fd984101d90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd974009860 tx=0x7fd97402bb70 comp rx=0 tx=0).stop 2024-03-18T22:47:00.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.403+0000 7fd98b44b640 1 -- 172.21.15.40:0/1638070378 shutdown_connections 2024-03-18T22:47:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.403+0000 7fd98b44b640 1 --2- 172.21.15.40:0/1638070378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9841022d0 0x7fd984108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.403+0000 7fd98b44b640 1 --2- 172.21.15.40:0/1638070378 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd984101910 0x7fd984101d90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.403+0000 7fd98b44b640 1 --2- 172.21.15.40:0/1638070378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd984100720 0x7fd984100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.403+0000 7fd98b44b640 1 -- 172.21.15.40:0/1638070378 >> 172.21.15.40:0/1638070378 conn(0x7fd9840fbcb0 msgr2=0x7fd9840fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.403+0000 7fd98b44b640 1 -- 172.21.15.40:0/1638070378 shutdown_connections 2024-03-18T22:47:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.403+0000 7fd98b44b640 1 -- 172.21.15.40:0/1638070378 wait complete. 2024-03-18T22:47:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.403+0000 7fd98b44b640 1 Processor -- start 2024-03-18T22:47:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.403+0000 7fd98b44b640 1 -- start start 2024-03-18T22:47:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.403+0000 7fd98b44b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd984100720 0x7fd984104660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.403+0000 7fd98b44b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd984101910 0x7fd984102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd98b44b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9841022d0 0x7fd984103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd98b44b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd98410c250 con 0x7fd9841022d0 2024-03-18T22:47:00.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd98b44b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd98410c0d0 con 0x7fd984101910 2024-03-18T22:47:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd98b44b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd98410c3d0 con 0x7fd984100720 2024-03-18T22:47:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd9891e7640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd984100720 0x7fd984104660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd9899e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9841022d0 0x7fd984103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd9891e7640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd984100720 0x7fd984104660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57168/0 (socket says 172.21.15.40:57168) 2024-03-18T22:47:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd9891e7640 1 -- 172.21.15.40:0/505988439 learned_addr learned my addr 172.21.15.40:0/505988439 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd9899e8640 1 --2- 172.21.15.40:0/505988439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9841022d0 0x7fd984103230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35436/0 (socket says 172.21.15.40:35436) 2024-03-18T22:47:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd9889e6640 1 --2- 172.21.15.40:0/505988439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd984101910 0x7fd984102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd9891e7640 1 -- 172.21.15.40:0/505988439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd984101910 msgr2=0x7fd984102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd9891e7640 1 --2- 172.21.15.40:0/505988439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd984101910 0x7fd984102cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd9891e7640 1 -- 172.21.15.40:0/505988439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9841022d0 msgr2=0x7fd984103230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd9891e7640 1 --2- 172.21.15.40:0/505988439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9841022d0 0x7fd984103230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd9891e7640 1 -- 172.21.15.40:0/505988439 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd984103af0 con 0x7fd984100720 2024-03-18T22:47:00.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd9891e7640 1 --2- 172.21.15.40:0/505988439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd984100720 0x7fd984104660 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd98000e9f0 tx=0x7fd98000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:00.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd97a7fc640 1 -- 172.21.15.40:0/505988439 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98000ccc0 con 0x7fd984100720 2024-03-18T22:47:00.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd97a7fc640 1 -- 172.21.15.40:0/505988439 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9800035f0 con 0x7fd984100720 2024-03-18T22:47:00.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd98b44b640 1 -- 172.21.15.40:0/505988439 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9841a1c00 con 0x7fd984100720 2024-03-18T22:47:00.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd97a7fc640 1 -- 172.21.15.40:0/505988439 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98000ce60 con 0x7fd984100720 2024-03-18T22:47:00.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.404+0000 7fd98b44b640 1 -- 172.21.15.40:0/505988439 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9841a2170 con 0x7fd984100720 2024-03-18T22:47:00.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.407+0000 7fd97a7fc640 1 -- 172.21.15.40:0/505988439 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd9800037a0 con 0x7fd984100720 2024-03-18T22:47:00.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.407+0000 7fd98e03a740 1 Processor -- start 2024-03-18T22:47:00.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.407+0000 7fd98e03a740 1 -- start start 2024-03-18T22:47:00.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.407+0000 7fd98e03a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd7790ea50 0x55fd7790ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.407+0000 7fd98e03a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790f440 0x55fd77913f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.407+0000 7fd98e03a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd779145b0 0x55fd77916a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.407+0000 7fd98e03a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fd77873500 con 0x55fd779145b0 2024-03-18T22:47:00.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.407+0000 7fd98e03a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fd7781bbe0 con 0x55fd7790f440 2024-03-18T22:47:00.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.407+0000 7fd98e03a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55fd7786f5a0 con 0x55fd7790ea50 2024-03-18T22:47:00.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.407+0000 7fd9899e8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd7790ea50 0x55fd7790ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.407+0000 7fd9891e7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790f440 0x55fd77913f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.407+0000 7fd9899e8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd7790ea50 0x55fd7790ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57180/0 (socket says 172.21.15.40:57180) 2024-03-18T22:47:00.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.407+0000 7fd9899e8640 1 -- 172.21.15.40:0/2144356269 learned_addr learned my addr 172.21.15.40:0/2144356269 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:00.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.407+0000 7fd9889e6640 1 --2- 172.21.15.40:0/2144356269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd779145b0 0x55fd77916a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.408+0000 7fd97a7fc640 1 --2- 172.21.15.40:0/505988439 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd9700a2880 0x7fd9700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.408+0000 7fd9899e8640 1 -- 172.21.15.40:0/2144356269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790f440 msgr2=0x55fd77913f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.408+0000 7fd9899e8640 1 --2- 172.21.15.40:0/2144356269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790f440 0x55fd77913f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.408+0000 7fd9899e8640 1 -- 172.21.15.40:0/2144356269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd779145b0 msgr2=0x55fd77916a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.408+0000 7fd9899e8640 1 --2- 172.21.15.40:0/2144356269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd779145b0 0x55fd77916a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.408+0000 7fd9899e8640 1 -- 172.21.15.40:0/2144356269 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd7783d000 con 0x55fd7790ea50 2024-03-18T22:47:00.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.408+0000 7fd9899e8640 1 --2- 172.21.15.40:0/505988439 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd9700a2880 0x7fd9700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.408+0000 7fd97a7fc640 1 -- 172.21.15.40:0/505988439 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd9800cf000 con 0x7fd984100720 2024-03-18T22:47:00.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.408+0000 7fd9899e8640 1 --2- 172.21.15.40:0/2144356269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd7790ea50 0x55fd7790ee70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd97c011590 tx=0x7fd97c011a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cb62513a1a169e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:00.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.408+0000 7fd96f7fe640 1 -- 172.21.15.40:0/2144356269 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c017050 con 0x55fd7790ea50 2024-03-18T22:47:00.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.408+0000 7fd96f7fe640 1 -- 172.21.15.40:0/2144356269 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd97c00fcf0 con 0x55fd7790ea50 2024-03-18T22:47:00.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.408+0000 7fd9899e8640 1 --2- 172.21.15.40:0/505988439 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd9700a2880 0x7fd9700a4d40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd97c022000 tx=0x7fd97c023000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:00.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.408+0000 7fd96f7fe640 1 -- 172.21.15.40:0/2144356269 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c01b9c0 con 0x55fd7790ea50 2024-03-18T22:47:00.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.409+0000 7fd98e03a740 1 -- 172.21.15.40:0/2144356269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd7790ea50 msgr2=0x55fd7790ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.409+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2144356269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd7790ea50 0x55fd7790ee70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd97c011590 tx=0x7fd97c011a50 comp rx=0 tx=0).stop 2024-03-18T22:47:00.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.409+0000 7fd98e03a740 1 -- 172.21.15.40:0/2144356269 shutdown_connections 2024-03-18T22:47:00.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.409+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2144356269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd779145b0 0x55fd77916a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.409+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2144356269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790f440 0x55fd77913f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.409+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2144356269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd7790ea50 0x55fd7790ee70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.409+0000 7fd98e03a740 1 -- 172.21.15.40:0/2144356269 >> 172.21.15.40:0/2144356269 conn(0x55fd77909fc0 msgr2=0x55fd7790a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:00.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.409+0000 7fd98e03a740 1 -- 172.21.15.40:0/2144356269 shutdown_connections 2024-03-18T22:47:00.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.409+0000 7fd98e03a740 1 -- 172.21.15.40:0/2144356269 wait complete. 2024-03-18T22:47:00.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.409+0000 7fd98e03a740 1 -- start start 2024-03-18T22:47:00.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.409+0000 7fd98e03a740 1 Processor -- start 2024-03-18T22:47:00.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd98e03a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790ea50 0x55fd77918ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd98e03a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd7790f440 0x55fd77919420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd98e03a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd779145b0 0x55fd77920460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd98e03a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fd77917560 con 0x55fd779145b0 2024-03-18T22:47:00.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd98e03a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fd7782f160 con 0x55fd7790ea50 2024-03-18T22:47:00.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd98e03a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55fd779176e0 con 0x55fd7790f440 2024-03-18T22:47:00.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd98e03a740 10 client.0 _ll_register_callbacks cb 0x55fd7786ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:00.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd9889e6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd7790f440 0x55fd77919420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd9891e7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790ea50 0x55fd77918ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd9889e6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd7790f440 0x55fd77919420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57188/0 (socket says 172.21.15.40:57188) 2024-03-18T22:47:00.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd9899e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd779145b0 0x55fd77920460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd9891e7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790ea50 0x55fd77918ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:39098/0 (socket says 172.21.15.40:39098) 2024-03-18T22:47:00.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd9889e6640 1 -- 172.21.15.40:0/2206482965 learned_addr learned my addr 172.21.15.40:0/2206482965 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:00.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd9889e6640 1 -- 172.21.15.40:0/2206482965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790ea50 msgr2=0x55fd77918ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd9889e6640 1 --2- 172.21.15.40:0/2206482965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790ea50 0x55fd77918ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd9889e6640 1 -- 172.21.15.40:0/2206482965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd779145b0 msgr2=0x55fd77920460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd9889e6640 1 --2- 172.21.15.40:0/2206482965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd779145b0 0x55fd77920460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd9889e6640 1 -- 172.21.15.40:0/2206482965 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd776517d0 con 0x55fd7790f440 2024-03-18T22:47:00.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd9891e7640 1 --2- 172.21.15.40:0/2206482965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790ea50 0x55fd77918ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:00.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.410+0000 7fd9899e8640 1 --2- 172.21.15.40:0/2206482965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd779145b0 0x55fd77920460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:00.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd9889e6640 1 --2- 172.21.15.40:0/2206482965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd7790f440 0x55fd77919420 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd97402c0b0 tx=0x7fd974005b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8374ec4c54b9845a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:00.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd96cff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:47:00.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd96cff9640 1 -- 172.21.15.40:0/2206482965 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd974039040 con 0x55fd7790f440 2024-03-18T22:47:00.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd98e03a740 1 -- 172.21.15.40:0/2206482965 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd97c00fcf0 con 0x55fd7790f440 2024-03-18T22:47:00.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd96cff9640 1 -- 172.21.15.40:0/2206482965 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd974004240 con 0x55fd7790f440 2024-03-18T22:47:00.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd96cff9640 1 -- 172.21.15.40:0/2206482965 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd97404e050 con 0x55fd7790f440 2024-03-18T22:47:00.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd98e03a740 10 client.26515 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:00.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd98e03a740 1 -- 172.21.15.40:0/2206482965 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fd775873f0 con 0x55fd7790f440 2024-03-18T22:47:00.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd96cff9640 1 -- 172.21.15.40:0/2206482965 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd97c00fcf0 con 0x55fd7790f440 2024-03-18T22:47:00.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd96cff9640 1 -- 172.21.15.40:0/2206482965 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd974003970 con 0x55fd7790f440 2024-03-18T22:47:00.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd98e03a740 10 client.26515 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:00.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd98e03a740 10 client.26515 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:47:00.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd98e03a740 20 client.26515 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:00.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2206482965 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55fd77924da0 0x55fd77927200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd98e03a740 4 client.26515 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:47:00.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.411+0000 7fd98e03a740 1 -- 172.21.15.40:0/2206482965 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd77880a40 con 0x55fd77924da0 2024-03-18T22:47:00.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.412+0000 7fd9891e7640 1 --2- 172.21.15.40:0/2206482965 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55fd77924da0 0x55fd77927200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.412+0000 7fd9891e7640 1 --2- 172.21.15.40:0/2206482965 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55fd77924da0 0x55fd77927200 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f78b269ec3005704 server_cookie=45f65ba2ca1bf2d5 in_seq=0 out_seq=0 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.412+0000 7fd96cff9640 10 client.26515 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.413+0000 7fd96cff9640 1 -- 172.21.15.40:0/2206482965 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fd77880a40 con 0x55fd77924da0 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.413+0000 7fd96cff9640 10 client.26515 handle_command_reply: tid=0 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.413+0000 7fd98e03a740 1 client.26515 shutdown 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.413+0000 7fd98e03a740 1 -- 172.21.15.40:0/2206482965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd7790f440 msgr2=0x55fd77919420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.413+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2206482965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd7790f440 0x55fd77919420 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd97402c0b0 tx=0x7fd974005b10 comp rx=0 tx=0).stop 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.413+0000 7fd98e03a740 1 -- 172.21.15.40:0/2206482965 shutdown_connections 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.413+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2206482965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd779145b0 0x55fd77920460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.413+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2206482965 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd7790f440 0x55fd77919420 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.413+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2206482965 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55fd77924da0 0x55fd77927200 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.413+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2206482965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790ea50 0x55fd77918ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.413+0000 7fd96cff9640 0 client.26515 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.413+0000 7fd98e03a740 1 -- 172.21.15.40:0/2206482965 >> 172.21.15.40:0/2206482965 conn(0x55fd77909fc0 msgr2=0x55fd779091e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.413+0000 7fd98e03a740 1 -- 172.21.15.40:0/2206482965 shutdown_connections 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.413+0000 7fd98e03a740 1 -- 172.21.15.40:0/2206482965 wait complete. 2024-03-18T22:47:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.413+0000 7fd98e03a740 20 client.26515 trim_cache size 0 max 16384 2024-03-18T22:47:00.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.431+0000 7fd98e03a740 1 Processor -- start 2024-03-18T22:47:00.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.431+0000 7fd98e03a740 1 -- start start 2024-03-18T22:47:00.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd98e03a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd77909fc0 0x55fd77908950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd98e03a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd7790ea50 0x55fd77920420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd98e03a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790f440 0x55fd77908e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd98e03a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd9841ac700 con 0x55fd7790ea50 2024-03-18T22:47:00.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd98e03a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd9841ae240 con 0x55fd7790f440 2024-03-18T22:47:00.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd98e03a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd9841acdd0 con 0x55fd77909fc0 2024-03-18T22:47:00.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd9889e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd7790ea50 0x55fd77920420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd9899e8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790f440 0x55fd77908e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd9891e7640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd77909fc0 0x55fd77908950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd9899e8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790f440 0x55fd77908e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:39102/0 (socket says 172.21.15.40:39102) 2024-03-18T22:47:00.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd9891e7640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd77909fc0 0x55fd77908950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57200/0 (socket says 172.21.15.40:57200) 2024-03-18T22:47:00.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd9899e8640 1 -- 172.21.15.40:0/3049128841 learned_addr learned my addr 172.21.15.40:0/3049128841 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:00.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd9889e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd7790ea50 0x55fd77920420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35462/0 (socket says 172.21.15.40:35462) 2024-03-18T22:47:00.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd9899e8640 1 -- 172.21.15.40:0/3049128841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd77909fc0 msgr2=0x55fd77908950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd9899e8640 1 --2- 172.21.15.40:0/3049128841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd77909fc0 0x55fd77908950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd9899e8640 1 -- 172.21.15.40:0/3049128841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd7790ea50 msgr2=0x55fd77920420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd9899e8640 1 --2- 172.21.15.40:0/3049128841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd7790ea50 0x55fd77920420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.432+0000 7fd9899e8640 1 -- 172.21.15.40:0/3049128841 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd775470f0 con 0x55fd7790f440 2024-03-18T22:47:00.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.433+0000 7fd9899e8640 1 --2- 172.21.15.40:0/3049128841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790f440 0x55fd77908e90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x55fd7790fca0 tx=0x7fd97c003f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8635a997414d9fdb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:00.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.433+0000 7fd96effd640 1 -- 172.21.15.40:0/3049128841 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c036070 con 0x55fd7790f440 2024-03-18T22:47:00.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.433+0000 7fd96effd640 1 -- 172.21.15.40:0/3049128841 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd97c01ba50 con 0x55fd7790f440 2024-03-18T22:47:00.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.433+0000 7fd96effd640 1 -- 172.21.15.40:0/3049128841 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c00f8a0 con 0x55fd7790f440 2024-03-18T22:47:00.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.434+0000 7fd98e03a740 1 -- 172.21.15.40:0/3049128841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790f440 msgr2=0x55fd77908e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.434+0000 7fd98e03a740 1 --2- 172.21.15.40:0/3049128841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790f440 0x55fd77908e90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x55fd7790fca0 tx=0x7fd97c003f80 comp rx=0 tx=0).stop 2024-03-18T22:47:00.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.434+0000 7fd98e03a740 1 -- 172.21.15.40:0/3049128841 shutdown_connections 2024-03-18T22:47:00.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.434+0000 7fd98e03a740 1 --2- 172.21.15.40:0/3049128841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790f440 0x55fd77908e90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.434+0000 7fd98e03a740 1 --2- 172.21.15.40:0/3049128841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd7790ea50 0x55fd77920420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.434+0000 7fd98e03a740 1 --2- 172.21.15.40:0/3049128841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd77909fc0 0x55fd77908950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.434+0000 7fd98e03a740 1 -- 172.21.15.40:0/3049128841 >> 172.21.15.40:0/3049128841 conn(0x55fd7790a710 msgr2=0x55fd77918ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:00.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.434+0000 7fd98e03a740 1 -- 172.21.15.40:0/3049128841 shutdown_connections 2024-03-18T22:47:00.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.434+0000 7fd98e03a740 1 -- 172.21.15.40:0/3049128841 wait complete. 2024-03-18T22:47:00.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.435+0000 7fd98e03a740 1 -- start start 2024-03-18T22:47:00.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.435+0000 7fd98e03a740 1 Processor -- start 2024-03-18T22:47:00.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd98e03a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd77909fc0 0x55fd7791e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd98e03a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790ea50 0x55fd7791e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd98e03a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd7790f440 0x55fd7791f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd98e03a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd9841adb70 con 0x55fd7790f440 2024-03-18T22:47:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd98e03a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd9841ad4a0 con 0x55fd7790ea50 2024-03-18T22:47:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd98e03a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd9841aa6e0 con 0x55fd77909fc0 2024-03-18T22:47:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd98e03a740 10 client.0 _ll_register_callbacks cb 0x7fd97c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd9899e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd7790f440 0x55fd7791f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd9891e7640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd77909fc0 0x55fd7791e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd9899e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd7790f440 0x55fd7791f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35472/0 (socket says 172.21.15.40:35472) 2024-03-18T22:47:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd9899e8640 1 -- 172.21.15.40:0/2733531441 learned_addr learned my addr 172.21.15.40:0/2733531441 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd9889e6640 1 --2- 172.21.15.40:0/2733531441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790ea50 0x55fd7791e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd9899e8640 1 -- 172.21.15.40:0/2733531441 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd77909fc0 msgr2=0x55fd7791e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd9899e8640 1 --2- 172.21.15.40:0/2733531441 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd77909fc0 0x55fd7791e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd9899e8640 1 -- 172.21.15.40:0/2733531441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790ea50 msgr2=0x55fd7791e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd9899e8640 1 --2- 172.21.15.40:0/2733531441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790ea50 0x55fd7791e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd9899e8640 1 -- 172.21.15.40:0/2733531441 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd7791fc00 con 0x55fd7790f440 2024-03-18T22:47:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.436+0000 7fd9889e6640 1 --2- 172.21.15.40:0/2733531441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790ea50 0x55fd7791e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:47:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.437+0000 7fd9899e8640 1 --2- 172.21.15.40:0/2733531441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd7790f440 0x55fd7791f610 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fd97c034040 tx=0x7fd97c005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95bfeb6c7ef4f81 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.437+0000 7fd96cff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:47:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.437+0000 7fd96cff9640 1 -- 172.21.15.40:0/2733531441 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c03f060 con 0x55fd7790f440 2024-03-18T22:47:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.437+0000 7fd96cff9640 1 -- 172.21.15.40:0/2733531441 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd97c00f840 con 0x55fd7790f440 2024-03-18T22:47:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.437+0000 7fd98e03a740 1 -- 172.21.15.40:0/2733531441 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd97c01ba50 con 0x55fd7790f440 2024-03-18T22:47:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.437+0000 7fd96cff9640 1 -- 172.21.15.40:0/2733531441 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd97c046050 con 0x55fd7790f440 2024-03-18T22:47:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.437+0000 7fd98e03a740 10 client.16704 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.437+0000 7fd98e03a740 1 -- 172.21.15.40:0/2733531441 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fd7798b660 con 0x55fd7790f440 2024-03-18T22:47:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.437+0000 7fd96cff9640 1 -- 172.21.15.40:0/2733531441 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd97c01ba50 con 0x55fd7790f440 2024-03-18T22:47:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.438+0000 7fd96cff9640 1 -- 172.21.15.40:0/2733531441 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd97c01a070 con 0x55fd7790f440 2024-03-18T22:47:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.438+0000 7fd98e03a740 10 client.16704 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:00.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.438+0000 7fd98e03a740 10 client.16704 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:47:00.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.438+0000 7fd98e03a740 20 client.16704 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:00.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.438+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2733531441 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55fd779145b0 0x55fd7798dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:00.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.438+0000 7fd98e03a740 4 client.16704 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:47:00.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.438+0000 7fd98e03a740 1 -- 172.21.15.40:0/2733531441 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fd7787ad90 con 0x55fd779145b0 2024-03-18T22:47:00.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.438+0000 7fd9891e7640 1 --2- 172.21.15.40:0/2733531441 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55fd779145b0 0x55fd7798dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:00.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.438+0000 7fd9891e7640 1 --2- 172.21.15.40:0/2733531441 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55fd779145b0 0x55fd7798dda0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3facf7986ab36032 server_cookie=badaddd782cc33a6 in_seq=0 out_seq=0 2024-03-18T22:47:00.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.438+0000 7fd96cff9640 10 client.16704 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:00.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.439+0000 7fd96cff9640 1 -- 172.21.15.40:0/2733531441 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fd7787ad90 con 0x55fd779145b0 2024-03-18T22:47:00.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.439+0000 7fd96cff9640 10 client.16704 handle_command_reply: tid=0 2024-03-18T22:47:00.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.439+0000 7fd98e03a740 1 client.16704 shutdown 2024-03-18T22:47:00.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.439+0000 7fd98e03a740 1 -- 172.21.15.40:0/2733531441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd7790f440 msgr2=0x55fd7791f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.439+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2733531441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd7790f440 0x55fd7791f610 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fd97c034040 tx=0x7fd97c005b50 comp rx=0 tx=0).stop 2024-03-18T22:47:00.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.439+0000 7fd98e03a740 1 -- 172.21.15.40:0/2733531441 shutdown_connections 2024-03-18T22:47:00.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.439+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2733531441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fd7790f440 0x55fd7791f610 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.439+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2733531441 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55fd779145b0 0x55fd7798dda0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.439+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2733531441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd7790ea50 0x55fd7791e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.439+0000 7fd98e03a740 1 --2- 172.21.15.40:0/2733531441 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fd77909fc0 0x55fd7791e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.439+0000 7fd96cff9640 0 client.16704 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:00.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.439+0000 7fd98e03a740 1 -- 172.21.15.40:0/2733531441 >> 172.21.15.40:0/2733531441 conn(0x55fd7790a710 msgr2=0x55fd7790d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:00.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.439+0000 7fd98e03a740 1 -- 172.21.15.40:0/2733531441 shutdown_connections 2024-03-18T22:47:00.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.439+0000 7fd98e03a740 1 -- 172.21.15.40:0/2733531441 wait complete. 2024-03-18T22:47:00.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.439+0000 7fd98e03a740 20 client.16704 trim_cache size 0 max 16384 2024-03-18T22:47:00.445 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:47:00.445 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:47:00.446 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:47:00.446 INFO:teuthology.orchestra.run.smithi040.stdout: "3630afb1-6156-42e1-a0af-8dbbc6fec421": { 2024-03-18T22:47:00.446 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:47:00.446 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "3630afb1-6156-42e1-a0af-8dbbc6fec421", 2024-03-18T22:47:00.446 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:47:00.446 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:47:00.446 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:47:00.446 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:47:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.440+0000 7fd98b44b640 1 -- 172.21.15.40:0/505988439 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd9700a2880 msgr2=0x7fd9700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.440+0000 7fd98b44b640 1 --2- 172.21.15.40:0/505988439 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd9700a2880 0x7fd9700a4d40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd97c022000 tx=0x7fd97c023000 comp rx=0 tx=0).stop 2024-03-18T22:47:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.440+0000 7fd98b44b640 1 -- 172.21.15.40:0/505988439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd984100720 msgr2=0x7fd984104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.440+0000 7fd98b44b640 1 --2- 172.21.15.40:0/505988439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd984100720 0x7fd984104660 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd98000e9f0 tx=0x7fd98000eeb0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.440+0000 7fd98b44b640 1 -- 172.21.15.40:0/505988439 shutdown_connections 2024-03-18T22:47:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.440+0000 7fd98b44b640 1 --2- 172.21.15.40:0/505988439 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd9700a2880 0x7fd9700a4d40 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.440+0000 7fd98b44b640 1 --2- 172.21.15.40:0/505988439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9841022d0 0x7fd984103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.440+0000 7fd98b44b640 1 --2- 172.21.15.40:0/505988439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd984101910 0x7fd984102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.440+0000 7fd98b44b640 1 --2- 172.21.15.40:0/505988439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd984100720 0x7fd984104660 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.440+0000 7fd98b44b640 1 -- 172.21.15.40:0/505988439 >> 172.21.15.40:0/505988439 conn(0x7fd9840fbcb0 msgr2=0x7fd984068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.440+0000 7fd98b44b640 1 -- 172.21.15.40:0/505988439 shutdown_connections 2024-03-18T22:47:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:00.440+0000 7fd98b44b640 1 -- 172.21.15.40:0/505988439 wait complete. 2024-03-18T22:47:00.454 INFO:tasks.cephfs.filesystem:scrub status for tag:3630afb1-6156-42e1-a0af-8dbbc6fec421 - {'path': '/', 'tag': '3630afb1-6156-42e1-a0af-8dbbc6fec421', 'options': 'recursive,force'} 2024-03-18T22:47:00.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:00 smithi113 ceph-mon[66775]: pgmap v481: 129 pgs: 129 active+clean; 4.6 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 26 MiB/s wr, 19 op/s 2024-03-18T22:47:01.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:01 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:47:01.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:01 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:01.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:01 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:47:01.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:01 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:01.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:01 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:47:01.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:01 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:01.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:01 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:47:01.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:01 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:01.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:01 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:47:01.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:01 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:01.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:01 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:47:01.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:01 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:02.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:02 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:47:02.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:02 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:02.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:02 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:47:02.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:02 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:02.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:02 smithi040 ceph-mon[62643]: pgmap v482: 129 pgs: 129 active+clean; 4.6 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:47:02.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:02 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:47:02.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:02 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:02.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:02 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:47:02.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:02 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:02.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:02 smithi113 ceph-mon[66775]: pgmap v482: 129 pgs: 129 active+clean; 4.6 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:47:02.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:02 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:47:02.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:02 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:02.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:02 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:47:02.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:02 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:02.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:02 smithi103 ceph-mon[66377]: pgmap v482: 129 pgs: 129 active+clean; 4.6 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:47:04.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:04 smithi040 ceph-mon[62643]: pgmap v483: 129 pgs: 129 active+clean; 4.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:47:04.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:04 smithi113 ceph-mon[66775]: pgmap v483: 129 pgs: 129 active+clean; 4.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:47:04.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:04 smithi103 ceph-mon[66377]: pgmap v483: 129 pgs: 129 active+clean; 4.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:47:06.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:06 smithi040 ceph-mon[62643]: pgmap v484: 129 pgs: 129 active+clean; 4.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:47:06.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:06 smithi113 ceph-mon[66775]: pgmap v484: 129 pgs: 129 active+clean; 4.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:47:06.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:06 smithi103 ceph-mon[66377]: pgmap v484: 129 pgs: 129 active+clean; 4.7 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:47:08.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:08 smithi040 ceph-mon[62643]: pgmap v485: 129 pgs: 129 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:47:08.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:08 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:47:08.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:08 smithi113 ceph-mon[66775]: pgmap v485: 129 pgs: 129 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:47:08.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:08 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:47:08.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:08 smithi103 ceph-mon[66377]: pgmap v485: 129 pgs: 129 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:47:08.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:08 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:47:09.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:09 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:47:09.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:09 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:47:09.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:09 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:47:10.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:10 smithi040 ceph-mon[62643]: pgmap v486: 129 pgs: 129 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-03-18T22:47:10.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:10 smithi113 ceph-mon[66775]: pgmap v486: 129 pgs: 129 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-03-18T22:47:10.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:10 smithi103 ceph-mon[66377]: pgmap v486: 129 pgs: 129 active+clean; 4.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-03-18T22:47:12.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:12 smithi040 ceph-mon[62643]: pgmap v487: 129 pgs: 129 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:47:12.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:12 smithi113 ceph-mon[66775]: pgmap v487: 129 pgs: 129 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:47:12.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:12 smithi103 ceph-mon[66377]: pgmap v487: 129 pgs: 129 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:47:14.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:14 smithi040 ceph-mon[62643]: pgmap v488: 129 pgs: 129 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:47:14.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:14 smithi113 ceph-mon[66775]: pgmap v488: 129 pgs: 129 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:47:14.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:14 smithi103 ceph-mon[66377]: pgmap v488: 129 pgs: 129 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:47:16.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:16 smithi040 ceph-mon[62643]: pgmap v489: 129 pgs: 129 active+clean; 5.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:47:16.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:16 smithi113 ceph-mon[66775]: pgmap v489: 129 pgs: 129 active+clean; 5.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:47:16.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:16 smithi103 ceph-mon[66377]: pgmap v489: 129 pgs: 129 active+clean; 5.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:47:18.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:18 smithi040 ceph-mon[62643]: pgmap v490: 129 pgs: 129 active+clean; 5.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 30 MiB/s wr, 20 op/s 2024-03-18T22:47:18.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:18 smithi113 ceph-mon[66775]: pgmap v490: 129 pgs: 129 active+clean; 5.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 30 MiB/s wr, 20 op/s 2024-03-18T22:47:18.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:18 smithi103 ceph-mon[66377]: pgmap v490: 129 pgs: 129 active+clean; 5.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 30 MiB/s wr, 20 op/s 2024-03-18T22:47:20.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:20 smithi040 ceph-mon[62643]: pgmap v491: 129 pgs: 129 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:47:20.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:20 smithi113 ceph-mon[66775]: pgmap v491: 129 pgs: 129 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:47:20.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:20 smithi103 ceph-mon[66377]: pgmap v491: 129 pgs: 129 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:47:21.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:21 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:47:21.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:21 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:47:21.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:21 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:47:21.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:21 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:47:21.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:21 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:47:21.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:21 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:47:21.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:21 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:47:21.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:21 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:47:21.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:21 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:47:21.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:21 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:47:21.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:21 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:47:21.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:21 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:47:22.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:22 smithi040 ceph-mon[62643]: pgmap v492: 129 pgs: 129 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:47:22.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:22 smithi113 ceph-mon[66775]: pgmap v492: 129 pgs: 129 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:47:22.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:22 smithi103 ceph-mon[66377]: pgmap v492: 129 pgs: 129 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:47:24.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:24 smithi113 ceph-mon[66775]: pgmap v493: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:47:24.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:24 smithi103 ceph-mon[66377]: pgmap v493: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:47:24.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:24 smithi040 ceph-mon[62643]: pgmap v493: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:47:26.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:26 smithi113 ceph-mon[66775]: pgmap v494: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:47:26.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:26 smithi103 ceph-mon[66377]: pgmap v494: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:47:26.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:26 smithi040 ceph-mon[62643]: pgmap v494: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:47:28.411 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:28 smithi103 ceph-mon[66377]: pgmap v495: 129 pgs: 129 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:47:28.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:28 smithi113 ceph-mon[66775]: pgmap v495: 129 pgs: 129 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:47:28.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:28 smithi040 ceph-mon[62643]: pgmap v495: 129 pgs: 129 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:47:30.454 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:47:30.455 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:47:30.482 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:30 smithi040 ceph-mon[62643]: pgmap v496: 129 pgs: 129 active+clean; 5.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:47:30.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 -- 172.21.15.40:0/2802293788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22b4100720 msgr2=0x7f22b4100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 --2- 172.21.15.40:0/2802293788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22b4100720 0x7f22b4100b40 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f22a8009900 tx=0x7f22a802bc30 comp rx=0 tx=0).stop 2024-03-18T22:47:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 -- 172.21.15.40:0/2802293788 shutdown_connections 2024-03-18T22:47:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 --2- 172.21.15.40:0/2802293788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22b41022d0 0x7f22b4108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 --2- 172.21.15.40:0/2802293788 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22b4101910 0x7f22b4101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 --2- 172.21.15.40:0/2802293788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22b4100720 0x7f22b4100b40 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 -- 172.21.15.40:0/2802293788 >> 172.21.15.40:0/2802293788 conn(0x7f22b40fbcb0 msgr2=0x7f22b40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 -- 172.21.15.40:0/2802293788 shutdown_connections 2024-03-18T22:47:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 -- 172.21.15.40:0/2802293788 wait complete. 2024-03-18T22:47:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 Processor -- start 2024-03-18T22:47:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 -- start start 2024-03-18T22:47:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22b4100720 0x7f22b4197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22b4101910 0x7f22b4198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22b41022d0 0x7f22b419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f22b410c220 con 0x7f22b4101910 2024-03-18T22:47:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f22b410c0a0 con 0x7f22b4100720 2024-03-18T22:47:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.574+0000 7f22baeb2640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f22b410c3a0 con 0x7f22b41022d0 2024-03-18T22:47:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22b3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22b4101910 0x7f22b4198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22b3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22b4101910 0x7f22b4198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55402/0 (socket says 172.21.15.40:55402) 2024-03-18T22:47:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22b3fff640 1 -- 172.21.15.40:0/3316700122 learned_addr learned my addr 172.21.15.40:0/3316700122 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22b8c4e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22b4100720 0x7f22b4197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22b944f640 1 --2- 172.21.15.40:0/3316700122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22b41022d0 0x7f22b419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22b3fff640 1 -- 172.21.15.40:0/3316700122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22b41022d0 msgr2=0x7f22b419f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22b3fff640 1 --2- 172.21.15.40:0/3316700122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22b41022d0 0x7f22b419f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22b3fff640 1 -- 172.21.15.40:0/3316700122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22b4100720 msgr2=0x7f22b4197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22b3fff640 1 --2- 172.21.15.40:0/3316700122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22b4100720 0x7f22b4197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22b3fff640 1 -- 172.21.15.40:0/3316700122 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b419f720 con 0x7f22b4101910 2024-03-18T22:47:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22b8c4e640 1 --2- 172.21.15.40:0/3316700122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22b4100720 0x7f22b4197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22b3fff640 1 --2- 172.21.15.40:0/3316700122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22b4101910 0x7f22b4198040 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f22a000e9f0 tx=0x7f22a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22b1ffb640 1 -- 172.21.15.40:0/3316700122 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a000cd80 con 0x7f22b4101910 2024-03-18T22:47:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22b1ffb640 1 -- 172.21.15.40:0/3316700122 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22a00108e0 con 0x7f22b4101910 2024-03-18T22:47:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22baeb2640 1 -- 172.21.15.40:0/3316700122 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b419fa10 con 0x7f22b4101910 2024-03-18T22:47:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.575+0000 7f22b1ffb640 1 -- 172.21.15.40:0/3316700122 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a0018920 con 0x7f22b4101910 2024-03-18T22:47:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.576+0000 7f22baeb2640 1 -- 172.21.15.40:0/3316700122 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b419ff60 con 0x7f22b4101910 2024-03-18T22:47:30.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:30 smithi113 ceph-mon[66775]: pgmap v496: 129 pgs: 129 active+clean; 5.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:47:30.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.578+0000 7f22bdb80740 1 Processor -- start 2024-03-18T22:47:30.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.578+0000 7f22b1ffb640 1 -- 172.21.15.40:0/3316700122 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f22a0010a90 con 0x7f22b4101910 2024-03-18T22:47:30.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.578+0000 7f22bdb80740 1 -- start start 2024-03-18T22:47:30.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.578+0000 7f22bdb80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0535a50 0x55c5b0535e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.578+0000 7f22bdb80740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0536440 0x55c5b053af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.578+0000 7f22bdb80740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b053b5b0 0x55c5b053da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.578+0000 7f22bdb80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c5b049a500 con 0x55c5b0535a50 2024-03-18T22:47:30.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.578+0000 7f22bdb80740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c5b0442be0 con 0x55c5b0536440 2024-03-18T22:47:30.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.578+0000 7f22bdb80740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c5b04965a0 con 0x55c5b053b5b0 2024-03-18T22:47:30.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f22b944f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0535a50 0x55c5b0535e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f22b8c4e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0536440 0x55c5b053af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f22b8c4e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0536440 0x55c5b053af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56640/0 (socket says 172.21.15.40:56640) 2024-03-18T22:47:30.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f22b8c4e640 1 -- 172.21.15.40:0/820427793 learned_addr learned my addr 172.21.15.40:0/820427793 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:30.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f22b3fff640 1 --2- 172.21.15.40:0/820427793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b053b5b0 0x55c5b053da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f22b8c4e640 1 -- 172.21.15.40:0/820427793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b053b5b0 msgr2=0x55c5b053da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:30.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f22b8c4e640 1 --2- 172.21.15.40:0/820427793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b053b5b0 0x55c5b053da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f22b8c4e640 1 -- 172.21.15.40:0/820427793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0535a50 msgr2=0x55c5b0535e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:30.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f22b8c4e640 1 --2- 172.21.15.40:0/820427793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0535a50 0x55c5b0535e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f22b8c4e640 1 -- 172.21.15.40:0/820427793 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5b0464000 con 0x55c5b0536440 2024-03-18T22:47:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f22b1ffb640 1 --2- 172.21.15.40:0/3316700122 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f22a40a2800 0x7f22a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f22b944f640 1 --2- 172.21.15.40:0/820427793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0535a50 0x55c5b0535e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f22b1ffb640 1 -- 172.21.15.40:0/3316700122 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f22a00cf140 con 0x7f22b4101910 2024-03-18T22:47:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f22b944f640 1 --2- 172.21.15.40:0/3316700122 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f22a40a2800 0x7f22a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f22b8c4e640 1 --2- 172.21.15.40:0/820427793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0536440 0x55c5b053af40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f22a8009900 tx=0x7f22a80095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e4965f55066042 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.579+0000 7f229effd640 1 -- 172.21.15.40:0/820427793 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a8045020 con 0x55c5b0536440 2024-03-18T22:47:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.580+0000 7f229effd640 1 -- 172.21.15.40:0/820427793 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22a802cc90 con 0x55c5b0536440 2024-03-18T22:47:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.580+0000 7f229effd640 1 -- 172.21.15.40:0/820427793 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a8004920 con 0x55c5b0536440 2024-03-18T22:47:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.580+0000 7f22bdb80740 1 -- 172.21.15.40:0/820427793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0536440 msgr2=0x55c5b053af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.580+0000 7f22bdb80740 1 --2- 172.21.15.40:0/820427793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0536440 0x55c5b053af40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f22a8009900 tx=0x7f22a80095f0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.580+0000 7f22b944f640 1 --2- 172.21.15.40:0/3316700122 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f22a40a2800 0x7f22a40a4cc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x55c5b0536ca0 tx=0x7f22ac009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.580+0000 7f22bdb80740 1 -- 172.21.15.40:0/820427793 shutdown_connections 2024-03-18T22:47:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.580+0000 7f22bdb80740 1 --2- 172.21.15.40:0/820427793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b053b5b0 0x55c5b053da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.580+0000 7f22bdb80740 1 --2- 172.21.15.40:0/820427793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0536440 0x55c5b053af40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.580+0000 7f22bdb80740 1 --2- 172.21.15.40:0/820427793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0535a50 0x55c5b0535e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.580+0000 7f22bdb80740 1 -- 172.21.15.40:0/820427793 >> 172.21.15.40:0/820427793 conn(0x55c5b0530fc0 msgr2=0x55c5b05313c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.580+0000 7f22bdb80740 1 -- 172.21.15.40:0/820427793 shutdown_connections 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.580+0000 7f22bdb80740 1 -- 172.21.15.40:0/820427793 wait complete. 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.581+0000 7f22bdb80740 1 -- start start 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.581+0000 7f22bdb80740 1 Processor -- start 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.581+0000 7f22bdb80740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0535a50 0x55c5b053fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.581+0000 7f22bdb80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0536440 0x55c5b0540380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.581+0000 7f22bdb80740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b053b5b0 0x55c5b05473c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.581+0000 7f22bdb80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c5b053e500 con 0x55c5b0536440 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.581+0000 7f22bdb80740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c5b0456160 con 0x55c5b0535a50 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.581+0000 7f22bdb80740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c5b053e680 con 0x55c5b053b5b0 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.581+0000 7f22b3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0536440 0x55c5b0540380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.581+0000 7f22b8c4e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0535a50 0x55c5b053fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.581+0000 7f22b3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0536440 0x55c5b0540380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55418/0 (socket says 172.21.15.40:55418) 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.581+0000 7f22bdb80740 10 client.0 _ll_register_callbacks cb 0x55c5b0492a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:30.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.581+0000 7f22b3fff640 1 -- 172.21.15.40:0/765179314 learned_addr learned my addr 172.21.15.40:0/765179314 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:30.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.581+0000 7f22b944f640 1 --2- 172.21.15.40:0/765179314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b053b5b0 0x55c5b05473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.582+0000 7f22b8c4e640 1 -- 172.21.15.40:0/765179314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b053b5b0 msgr2=0x55c5b05473c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:30.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.582+0000 7f22b8c4e640 1 --2- 172.21.15.40:0/765179314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b053b5b0 0x55c5b05473c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.582+0000 7f22b8c4e640 1 -- 172.21.15.40:0/765179314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0536440 msgr2=0x55c5b0540380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:30.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.582+0000 7f22b8c4e640 1 --2- 172.21.15.40:0/765179314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0536440 0x55c5b0540380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.582+0000 7f22b8c4e640 1 -- 172.21.15.40:0/765179314 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5b02787d0 con 0x55c5b0535a50 2024-03-18T22:47:30.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.582+0000 7f22b8c4e640 1 --2- 172.21.15.40:0/765179314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0535a50 0x55c5b053fe40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55c5b0531f30 tx=0x7f22a802c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d0a561daf0fa4b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:30.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.582+0000 7f2293fff640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:47:30.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.582+0000 7f2293fff640 1 -- 172.21.15.40:0/765179314 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a8050020 con 0x55c5b0535a50 2024-03-18T22:47:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.582+0000 7f22bdb80740 1 -- 172.21.15.40:0/765179314 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22a802cc90 con 0x55c5b0535a50 2024-03-18T22:47:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.582+0000 7f2293fff640 1 -- 172.21.15.40:0/765179314 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f22a8003b80 con 0x55c5b0535a50 2024-03-18T22:47:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.582+0000 7f2293fff640 1 -- 172.21.15.40:0/765179314 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f22a805b050 con 0x55c5b0535a50 2024-03-18T22:47:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.582+0000 7f22bdb80740 10 client.26543 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.582+0000 7f22bdb80740 1 -- 172.21.15.40:0/765179314 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c5b01ae3f0 con 0x55c5b0535a50 2024-03-18T22:47:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.583+0000 7f2293fff640 1 -- 172.21.15.40:0/765179314 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22a802cc90 con 0x55c5b0535a50 2024-03-18T22:47:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.583+0000 7f2293fff640 1 -- 172.21.15.40:0/765179314 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f22a804c460 con 0x55c5b0535a50 2024-03-18T22:47:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.583+0000 7f22bdb80740 10 client.26543 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.583+0000 7f22bdb80740 10 client.26543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:47:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.583+0000 7f22bdb80740 20 client.26543 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.583+0000 7f22bdb80740 1 --2- 172.21.15.40:0/765179314 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c5b054b8b0 0x55c5b054dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.583+0000 7f22bdb80740 4 client.26543 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:47:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.583+0000 7f22bdb80740 1 -- 172.21.15.40:0/765179314 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5b04a7a40 con 0x55c5b054b8b0 2024-03-18T22:47:30.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.583+0000 7f22b3fff640 1 --2- 172.21.15.40:0/765179314 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c5b054b8b0 0x55c5b054dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.583+0000 7f2293fff640 10 client.26543 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:30.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.583+0000 7f22b3fff640 1 --2- 172.21.15.40:0/765179314 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c5b054b8b0 0x55c5b054dca0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7605facf2d146fca server_cookie=25ae50b017ae0cd3 in_seq=0 out_seq=0 2024-03-18T22:47:30.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.584+0000 7f2293fff640 1 -- 172.21.15.40:0/765179314 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c5b04a7a40 con 0x55c5b054b8b0 2024-03-18T22:47:30.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.584+0000 7f2293fff640 10 client.26543 handle_command_reply: tid=0 2024-03-18T22:47:30.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.584+0000 7f22bdb80740 1 client.26543 shutdown 2024-03-18T22:47:30.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.585+0000 7f22bdb80740 1 -- 172.21.15.40:0/765179314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0535a50 msgr2=0x55c5b053fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:30.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.585+0000 7f22bdb80740 1 --2- 172.21.15.40:0/765179314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0535a50 0x55c5b053fe40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55c5b0531f30 tx=0x7f22a802c3b0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.585+0000 7f22bdb80740 1 -- 172.21.15.40:0/765179314 shutdown_connections 2024-03-18T22:47:30.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.585+0000 7f22bdb80740 1 --2- 172.21.15.40:0/765179314 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b053b5b0 0x55c5b05473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.585+0000 7f22bdb80740 1 --2- 172.21.15.40:0/765179314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0536440 0x55c5b0540380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.585+0000 7f22bdb80740 1 --2- 172.21.15.40:0/765179314 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c5b054b8b0 0x55c5b054dca0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.585+0000 7f22bdb80740 1 --2- 172.21.15.40:0/765179314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0535a50 0x55c5b053fe40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.585+0000 7f22bdb80740 1 -- 172.21.15.40:0/765179314 >> 172.21.15.40:0/765179314 conn(0x55c5b0530fc0 msgr2=0x55c5b05301e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.585+0000 7f2293fff640 0 client.26543 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.585+0000 7f22bdb80740 1 -- 172.21.15.40:0/765179314 shutdown_connections 2024-03-18T22:47:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.585+0000 7f22bdb80740 1 -- 172.21.15.40:0/765179314 wait complete. 2024-03-18T22:47:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.585+0000 7f22bdb80740 20 client.26543 trim_cache size 0 max 16384 2024-03-18T22:47:30.613 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:30 smithi103 ceph-mon[66377]: pgmap v496: 129 pgs: 129 active+clean; 5.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:47:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.606+0000 7f22bdb80740 1 Processor -- start 2024-03-18T22:47:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22bdb80740 1 -- start start 2024-03-18T22:47:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22bdb80740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b0535a50 0x55c5b0530060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22bdb80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0536440 0x55c5b05305a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22bdb80740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b053b5b0 0x55c5b0547380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22bdb80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f22b41aa560 con 0x55c5b0536440 2024-03-18T22:47:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22bdb80740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f22b41ac0a0 con 0x55c5b053b5b0 2024-03-18T22:47:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22bdb80740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f22b41aac30 con 0x55c5b0535a50 2024-03-18T22:47:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22b3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0536440 0x55c5b05305a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22b944f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b053b5b0 0x55c5b0547380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22b8c4e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b0535a50 0x55c5b0530060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22b3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0536440 0x55c5b05305a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55430/0 (socket says 172.21.15.40:55430) 2024-03-18T22:47:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22b944f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b053b5b0 0x55c5b0547380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56654/0 (socket says 172.21.15.40:56654) 2024-03-18T22:47:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22b8c4e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b0535a50 0x55c5b0530060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33146/0 (socket says 172.21.15.40:33146) 2024-03-18T22:47:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22b3fff640 1 -- 172.21.15.40:0/1890056364 learned_addr learned my addr 172.21.15.40:0/1890056364 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22b8c4e640 1 -- 172.21.15.40:0/1890056364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b053b5b0 msgr2=0x55c5b0547380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22b8c4e640 1 --2- 172.21.15.40:0/1890056364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b053b5b0 0x55c5b0547380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22b8c4e640 1 -- 172.21.15.40:0/1890056364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0536440 msgr2=0x55c5b05305a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22b8c4e640 1 --2- 172.21.15.40:0/1890056364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0536440 0x55c5b05305a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22b8c4e640 1 -- 172.21.15.40:0/1890056364 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5b054e010 con 0x55c5b0535a50 2024-03-18T22:47:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.607+0000 7f22b8c4e640 1 --2- 172.21.15.40:0/1890056364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b0535a50 0x55c5b0530060 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f22a804c900 tx=0x7f22a8005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f712015821e83b01 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.608+0000 7f229e7fc640 1 -- 172.21.15.40:0/1890056364 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a8050070 con 0x55c5b0535a50 2024-03-18T22:47:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.608+0000 7f229e7fc640 1 -- 172.21.15.40:0/1890056364 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22a802cc90 con 0x55c5b0535a50 2024-03-18T22:47:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.608+0000 7f229e7fc640 1 -- 172.21.15.40:0/1890056364 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a8045040 con 0x55c5b0535a50 2024-03-18T22:47:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.608+0000 7f22bdb80740 1 -- 172.21.15.40:0/1890056364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b0535a50 msgr2=0x55c5b0530060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.608+0000 7f22bdb80740 1 --2- 172.21.15.40:0/1890056364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b0535a50 0x55c5b0530060 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f22a804c900 tx=0x7f22a8005bc0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.609+0000 7f22bdb80740 1 -- 172.21.15.40:0/1890056364 shutdown_connections 2024-03-18T22:47:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.609+0000 7f22bdb80740 1 --2- 172.21.15.40:0/1890056364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b053b5b0 0x55c5b0547380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.609+0000 7f22bdb80740 1 --2- 172.21.15.40:0/1890056364 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b0536440 0x55c5b05305a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.609+0000 7f22bdb80740 1 --2- 172.21.15.40:0/1890056364 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b0535a50 0x55c5b0530060 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.609+0000 7f22bdb80740 1 -- 172.21.15.40:0/1890056364 >> 172.21.15.40:0/1890056364 conn(0x55c5b0530fc0 msgr2=0x55c5b052f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.609+0000 7f22bdb80740 1 -- 172.21.15.40:0/1890056364 shutdown_connections 2024-03-18T22:47:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.609+0000 7f22bdb80740 1 -- 172.21.15.40:0/1890056364 wait complete. 2024-03-18T22:47:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.609+0000 7f22bdb80740 1 -- start start 2024-03-18T22:47:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.609+0000 7f22bdb80740 1 Processor -- start 2024-03-18T22:47:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.609+0000 7f22bdb80740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0535a50 0x55c5b0544930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.609+0000 7f22bdb80740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b0536440 0x55c5b0544e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.609+0000 7f22bdb80740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b053b5b0 0x55c5b0545e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.609+0000 7f22bdb80740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f22b41ab9d0 con 0x55c5b053b5b0 2024-03-18T22:47:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.609+0000 7f22bdb80740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f22b41ab300 con 0x55c5b0535a50 2024-03-18T22:47:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.609+0000 7f22bdb80740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f22b41a8540 con 0x55c5b0536440 2024-03-18T22:47:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22bdb80740 10 client.0 _ll_register_callbacks cb 0x7f22ac009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22b8c4e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0535a50 0x55c5b0544930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22b3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b0536440 0x55c5b0544e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22b944f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b053b5b0 0x55c5b0545e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22b8c4e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0535a50 0x55c5b0544930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56670/0 (socket says 172.21.15.40:56670) 2024-03-18T22:47:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22b8c4e640 1 -- 172.21.15.40:0/662538417 learned_addr learned my addr 172.21.15.40:0/662538417 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22b3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b0536440 0x55c5b0544e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33156/0 (socket says 172.21.15.40:33156) 2024-03-18T22:47:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22b3fff640 1 -- 172.21.15.40:0/662538417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0535a50 msgr2=0x55c5b0544930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22b3fff640 1 --2- 172.21.15.40:0/662538417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0535a50 0x55c5b0544930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22b3fff640 1 -- 172.21.15.40:0/662538417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b053b5b0 msgr2=0x55c5b0545e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22b3fff640 1 --2- 172.21.15.40:0/662538417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b053b5b0 0x55c5b0545e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22b3fff640 1 -- 172.21.15.40:0/662538417 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5b016e0f0 con 0x55c5b0536440 2024-03-18T22:47:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22b8c4e640 1 --2- 172.21.15.40:0/662538417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0535a50 0x55c5b0544930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22b944f640 1 --2- 172.21.15.40:0/662538417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b053b5b0 0x55c5b0545e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22b3fff640 1 --2- 172.21.15.40:0/662538417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b0536440 0x55c5b0544e70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f22a0098170 tx=0x7f22a00cf530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d29192bb5c2dc52e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f2293fff640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:47:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f2293fff640 1 -- 172.21.15.40:0/662538417 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a009b100 con 0x55c5b0536440 2024-03-18T22:47:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f2293fff640 1 -- 172.21.15.40:0/662538417 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f22a00860b0 con 0x55c5b0536440 2024-03-18T22:47:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22bdb80740 1 -- 172.21.15.40:0/662538417 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22a802cc90 con 0x55c5b0536440 2024-03-18T22:47:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f2293fff640 1 -- 172.21.15.40:0/662538417 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f22a0010de0 con 0x55c5b0536440 2024-03-18T22:47:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22bdb80740 10 client.26539 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.610+0000 7f22bdb80740 1 -- 172.21.15.40:0/662538417 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c5b0547610 con 0x55c5b0536440 2024-03-18T22:47:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.611+0000 7f2293fff640 1 -- 172.21.15.40:0/662538417 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22a802cc90 con 0x55c5b0536440 2024-03-18T22:47:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.611+0000 7f2293fff640 1 -- 172.21.15.40:0/662538417 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f22a00d2c90 con 0x55c5b0536440 2024-03-18T22:47:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.611+0000 7f22bdb80740 10 client.26539 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:30.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.611+0000 7f22bdb80740 10 client.26539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:47:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.611+0000 7f22bdb80740 20 client.26539 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.611+0000 7f22bdb80740 1 --2- 172.21.15.40:0/662538417 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c5b05b21b0 0x55c5b05b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.611+0000 7f22bdb80740 4 client.26539 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:47:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.611+0000 7f22bdb80740 1 -- 172.21.15.40:0/662538417 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c5b04a1d90 con 0x55c5b05b21b0 2024-03-18T22:47:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.611+0000 7f22b8c4e640 1 --2- 172.21.15.40:0/662538417 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c5b05b21b0 0x55c5b05b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.611+0000 7f2293fff640 10 client.26539 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.611+0000 7f22b8c4e640 1 --2- 172.21.15.40:0/662538417 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c5b05b21b0 0x55c5b05b45d0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d384db7720b8beda server_cookie=39f920a5bdb920c0 in_seq=0 out_seq=0 2024-03-18T22:47:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.612+0000 7f2293fff640 1 -- 172.21.15.40:0/662538417 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c5b04a1d90 con 0x55c5b05b21b0 2024-03-18T22:47:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.612+0000 7f2293fff640 10 client.26539 handle_command_reply: tid=0 2024-03-18T22:47:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.612+0000 7f22bdb80740 1 client.26539 shutdown 2024-03-18T22:47:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.612+0000 7f22bdb80740 1 -- 172.21.15.40:0/662538417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b0536440 msgr2=0x55c5b0544e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.612+0000 7f22bdb80740 1 --2- 172.21.15.40:0/662538417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b0536440 0x55c5b0544e70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f22a0098170 tx=0x7f22a00cf530 comp rx=0 tx=0).stop 2024-03-18T22:47:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.612+0000 7f22bdb80740 1 -- 172.21.15.40:0/662538417 shutdown_connections 2024-03-18T22:47:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.612+0000 7f22bdb80740 1 --2- 172.21.15.40:0/662538417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c5b053b5b0 0x55c5b0545e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.612+0000 7f22bdb80740 1 --2- 172.21.15.40:0/662538417 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c5b0536440 0x55c5b0544e70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.612+0000 7f22bdb80740 1 --2- 172.21.15.40:0/662538417 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c5b05b21b0 0x55c5b05b45d0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.612+0000 7f22bdb80740 1 --2- 172.21.15.40:0/662538417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5b0535a50 0x55c5b0544930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.612+0000 7f2293fff640 0 client.26539 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.612+0000 7f22bdb80740 1 -- 172.21.15.40:0/662538417 >> 172.21.15.40:0/662538417 conn(0x55c5b0530fc0 msgr2=0x55c5b05301e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.612+0000 7f22bdb80740 1 -- 172.21.15.40:0/662538417 shutdown_connections 2024-03-18T22:47:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.612+0000 7f22bdb80740 1 -- 172.21.15.40:0/662538417 wait complete. 2024-03-18T22:47:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.612+0000 7f22bdb80740 20 client.26539 trim_cache size 0 max 16384 2024-03-18T22:47:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:47:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:47:30.632 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:47:30.632 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:47:30.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.613+0000 7f22baeb2640 1 -- 172.21.15.40:0/3316700122 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f22a40a2800 msgr2=0x7f22a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.613+0000 7f22baeb2640 1 --2- 172.21.15.40:0/3316700122 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f22a40a2800 0x7f22a40a4cc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x55c5b0536ca0 tx=0x7f22ac009500 comp rx=0 tx=0).stop 2024-03-18T22:47:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.613+0000 7f22baeb2640 1 -- 172.21.15.40:0/3316700122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22b4101910 msgr2=0x7f22b4198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.613+0000 7f22baeb2640 1 --2- 172.21.15.40:0/3316700122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22b4101910 0x7f22b4198040 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f22a000e9f0 tx=0x7f22a000eeb0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.613+0000 7f22baeb2640 1 -- 172.21.15.40:0/3316700122 shutdown_connections 2024-03-18T22:47:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.613+0000 7f22baeb2640 1 --2- 172.21.15.40:0/3316700122 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f22a40a2800 0x7f22a40a4cc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.613+0000 7f22baeb2640 1 --2- 172.21.15.40:0/3316700122 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f22b41022d0 0x7f22b419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.613+0000 7f22baeb2640 1 --2- 172.21.15.40:0/3316700122 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22b4101910 0x7f22b4198040 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.613+0000 7f22baeb2640 1 --2- 172.21.15.40:0/3316700122 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22b4100720 0x7f22b4197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.613+0000 7f22baeb2640 1 -- 172.21.15.40:0/3316700122 >> 172.21.15.40:0/3316700122 conn(0x7f22b40fbcb0 msgr2=0x7f22b4068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.613+0000 7f22baeb2640 1 -- 172.21.15.40:0/3316700122 shutdown_connections 2024-03-18T22:47:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.613+0000 7f22baeb2640 1 -- 172.21.15.40:0/3316700122 wait complete. 2024-03-18T22:47:30.636 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:47:30.636 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:47:30.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.787+0000 7f60c7f6f640 1 -- 172.21.15.40:0/2597107979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60c0100740 msgr2=0x7f60c0100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:30.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.787+0000 7f60c7f6f640 1 --2- 172.21.15.40:0/2597107979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60c0100740 0x7f60c0100b60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f60bc009900 tx=0x7f60bc02bc30 comp rx=0 tx=0).stop 2024-03-18T22:47:30.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.787+0000 7f60c7f6f640 1 -- 172.21.15.40:0/2597107979 shutdown_connections 2024-03-18T22:47:30.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.787+0000 7f60c7f6f640 1 --2- 172.21.15.40:0/2597107979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c01022f0 0x7f60c0108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.787+0000 7f60c7f6f640 1 --2- 172.21.15.40:0/2597107979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60c0101930 0x7f60c0101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.787+0000 7f60c7f6f640 1 --2- 172.21.15.40:0/2597107979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60c0100740 0x7f60c0100b60 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.787+0000 7f60c7f6f640 1 -- 172.21.15.40:0/2597107979 >> 172.21.15.40:0/2597107979 conn(0x7f60c00fbcb0 msgr2=0x7f60c00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:30.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.787+0000 7f60c7f6f640 1 -- 172.21.15.40:0/2597107979 shutdown_connections 2024-03-18T22:47:30.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.787+0000 7f60c7f6f640 1 -- 172.21.15.40:0/2597107979 wait complete. 2024-03-18T22:47:30.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.788+0000 7f60c7f6f640 1 Processor -- start 2024-03-18T22:47:30.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.788+0000 7f60c7f6f640 1 -- start start 2024-03-18T22:47:30.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.788+0000 7f60c7f6f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0100740 0x7f60c01046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.788+0000 7f60c7f6f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60c0101930 0x7f60c0102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.788+0000 7f60c7f6f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60c01022f0 0x7f60c0103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.788+0000 7f60c7f6f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f60c010c240 con 0x7f60c0101930 2024-03-18T22:47:30.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.788+0000 7f60c7f6f640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60c010c0c0 con 0x7f60c0100740 2024-03-18T22:47:30.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.788+0000 7f60c7f6f640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f60c010c3c0 con 0x7f60c01022f0 2024-03-18T22:47:30.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.788+0000 7f60c5d0b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0100740 0x7f60c01046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.788+0000 7f60c5d0b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0100740 0x7f60c01046d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56702/0 (socket says 172.21.15.40:56702) 2024-03-18T22:47:30.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.788+0000 7f60c5d0b640 1 -- 172.21.15.40:0/2506967098 learned_addr learned my addr 172.21.15.40:0/2506967098 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:30.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.788+0000 7f60c550a640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60c0101930 0x7f60c0102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.789+0000 7f60c650c640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60c01022f0 0x7f60c0103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.789+0000 7f60c5d0b640 1 -- 172.21.15.40:0/2506967098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60c01022f0 msgr2=0x7f60c0103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:30.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.789+0000 7f60c5d0b640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60c01022f0 0x7f60c0103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.789+0000 7f60c5d0b640 1 -- 172.21.15.40:0/2506967098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60c0101930 msgr2=0x7f60c0102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:30.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.789+0000 7f60c5d0b640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60c0101930 0x7f60c0102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:30.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.789+0000 7f60c5d0b640 1 -- 172.21.15.40:0/2506967098 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c0103ae0 con 0x7f60c0100740 2024-03-18T22:47:30.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.789+0000 7f60c650c640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60c01022f0 0x7f60c0103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:30.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.789+0000 7f60c550a640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60c0101930 0x7f60c0102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:30.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.790+0000 7f60c5d0b640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0100740 0x7f60c01046d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f60bc000c00 tx=0x7f60bc005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:30.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.790+0000 7f60b6ffd640 1 -- 172.21.15.40:0/2506967098 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc043020 con 0x7f60c0100740 2024-03-18T22:47:30.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.790+0000 7f60c7f6f640 1 -- 172.21.15.40:0/2506967098 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c01a1c10 con 0x7f60c0100740 2024-03-18T22:47:30.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.790+0000 7f60b6ffd640 1 -- 172.21.15.40:0/2506967098 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60bc004350 con 0x7f60c0100740 2024-03-18T22:47:30.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.790+0000 7f60b6ffd640 1 -- 172.21.15.40:0/2506967098 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc003b80 con 0x7f60c0100740 2024-03-18T22:47:30.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.790+0000 7f60c7f6f640 1 -- 172.21.15.40:0/2506967098 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c01a2160 con 0x7f60c0100740 2024-03-18T22:47:30.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.791+0000 7f60c7f6f640 1 -- 172.21.15.40:0/2506967098 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60c01a2330 con 0x7f60c0100740 2024-03-18T22:47:30.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.793+0000 7f60b6ffd640 1 -- 172.21.15.40:0/2506967098 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f60bc046020 con 0x7f60c0100740 2024-03-18T22:47:30.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.794+0000 7f60b6ffd640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f60ac0a2800 0x7f60ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:30.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.794+0000 7f60c550a640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f60ac0a2800 0x7f60ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:30.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.794+0000 7f60b6ffd640 1 -- 172.21.15.40:0/2506967098 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f60bc0f14d0 con 0x7f60c0100740 2024-03-18T22:47:30.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.794+0000 7f60c550a640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f60ac0a2800 0x7f60ac0a4cc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f60c0104440 tx=0x7f60b0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:30.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:30.797+0000 7f60b6ffd640 1 -- 172.21.15.40:0/2506967098 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f60bc04d020 con 0x7f60c0100740 2024-03-18T22:47:31.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.065+0000 7f60c7f6f640 1 -- 172.21.15.40:0/2506967098 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60c01a2500 con 0x7f60c0100740 2024-03-18T22:47:31.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.066+0000 7f60b6ffd640 1 -- 172.21.15.40:0/2506967098 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11947 (secure 0 0 0) 0x7f60bc0be180 con 0x7f60c0100740 2024-03-18T22:47:31.066 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:47:31.067 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:46:52.724919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24388,24401,24473,24481,24509]},"id":1}]} 2024-03-18T22:47:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 32 2024-03-18T22:47:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.068+0000 7f60c7f6f640 1 -- 172.21.15.40:0/2506967098 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f60ac0a2800 msgr2=0x7f60ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.068+0000 7f60c7f6f640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f60ac0a2800 0x7f60ac0a4cc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f60c0104440 tx=0x7f60b0005f90 comp rx=0 tx=0).stop 2024-03-18T22:47:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.068+0000 7f60c7f6f640 1 -- 172.21.15.40:0/2506967098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0100740 msgr2=0x7f60c01046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.068+0000 7f60c7f6f640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0100740 0x7f60c01046d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f60bc000c00 tx=0x7f60bc005dc0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.068+0000 7f60c7f6f640 1 -- 172.21.15.40:0/2506967098 shutdown_connections 2024-03-18T22:47:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.068+0000 7f60c7f6f640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f60ac0a2800 0x7f60ac0a4cc0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.068+0000 7f60c7f6f640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f60c01022f0 0x7f60c0103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.068+0000 7f60c7f6f640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60c0101930 0x7f60c0102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.068+0000 7f60c7f6f640 1 --2- 172.21.15.40:0/2506967098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0100740 0x7f60c01046d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.068+0000 7f60c7f6f640 1 -- 172.21.15.40:0/2506967098 >> 172.21.15.40:0/2506967098 conn(0x7f60c00fbcb0 msgr2=0x7f60c0068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.068+0000 7f60c7f6f640 1 -- 172.21.15.40:0/2506967098 shutdown_connections 2024-03-18T22:47:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.068+0000 7f60c7f6f640 1 -- 172.21.15.40:0/2506967098 wait complete. 2024-03-18T22:47:31.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:47:31.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.197+0000 7f561759e640 1 -- 172.21.15.40:0/566019971 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5610101910 msgr2=0x7f5610101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.197+0000 7f561759e640 1 --2- 172.21.15.40:0/566019971 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5610101910 0x7f5610101d90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5600009900 tx=0x7f560002bc30 comp rx=0 tx=0).stop 2024-03-18T22:47:31.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.198+0000 7f561759e640 1 -- 172.21.15.40:0/566019971 shutdown_connections 2024-03-18T22:47:31.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.198+0000 7f561759e640 1 --2- 172.21.15.40:0/566019971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f56101022d0 0x7f5610108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.198+0000 7f561759e640 1 --2- 172.21.15.40:0/566019971 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5610101910 0x7f5610101d90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.198+0000 7f561759e640 1 --2- 172.21.15.40:0/566019971 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5610100720 0x7f5610100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.198+0000 7f561759e640 1 -- 172.21.15.40:0/566019971 >> 172.21.15.40:0/566019971 conn(0x7f56100fbc90 msgr2=0x7f56100fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.198+0000 7f561759e640 1 -- 172.21.15.40:0/566019971 shutdown_connections 2024-03-18T22:47:31.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.198+0000 7f561759e640 1 -- 172.21.15.40:0/566019971 wait complete. 2024-03-18T22:47:31.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.198+0000 7f561759e640 1 Processor -- start 2024-03-18T22:47:31.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.198+0000 7f561759e640 1 -- start start 2024-03-18T22:47:31.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.198+0000 7f561759e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5610100720 0x7f5610197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.198+0000 7f561759e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5610101910 0x7f5610197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.198+0000 7f561759e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56101022d0 0x7f561019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.198+0000 7f561759e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f561010c250 con 0x7f56101022d0 2024-03-18T22:47:31.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.198+0000 7f561759e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f561010c0d0 con 0x7f5610100720 2024-03-18T22:47:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.198+0000 7f561759e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f561010c3d0 con 0x7f5610101910 2024-03-18T22:47:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f561533a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5610100720 0x7f5610197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f5615b3b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56101022d0 0x7f561019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f5614b39640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5610101910 0x7f5610197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f561533a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5610100720 0x7f5610197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56734/0 (socket says 172.21.15.40:56734) 2024-03-18T22:47:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f5615b3b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56101022d0 0x7f561019f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55468/0 (socket says 172.21.15.40:55468) 2024-03-18T22:47:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f561533a640 1 -- 172.21.15.40:0/477167712 learned_addr learned my addr 172.21.15.40:0/477167712 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f561533a640 1 -- 172.21.15.40:0/477167712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5610101910 msgr2=0x7f5610197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f561533a640 1 --2- 172.21.15.40:0/477167712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5610101910 0x7f5610197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f561533a640 1 -- 172.21.15.40:0/477167712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56101022d0 msgr2=0x7f561019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f561533a640 1 --2- 172.21.15.40:0/477167712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56101022d0 0x7f561019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f561533a640 1 -- 172.21.15.40:0/477167712 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561019f6c0 con 0x7f5610100720 2024-03-18T22:47:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f561533a640 1 --2- 172.21.15.40:0/477167712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5610100720 0x7f5610197aa0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f560c00e9c0 tx=0x7f560c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f56067fc640 1 -- 172.21.15.40:0/477167712 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c00cd80 con 0x7f5610100720 2024-03-18T22:47:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f56067fc640 1 -- 172.21.15.40:0/477167712 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f560c0035f0 con 0x7f5610100720 2024-03-18T22:47:31.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f56067fc640 1 -- 172.21.15.40:0/477167712 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c010690 con 0x7f5610100720 2024-03-18T22:47:31.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f561759e640 1 -- 172.21.15.40:0/477167712 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f561019f9b0 con 0x7f5610100720 2024-03-18T22:47:31.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.199+0000 7f561759e640 1 -- 172.21.15.40:0/477167712 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f561019fed0 con 0x7f5610100720 2024-03-18T22:47:31.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.201+0000 7f56067fc640 1 -- 172.21.15.40:0/477167712 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f560c0037a0 con 0x7f5610100720 2024-03-18T22:47:31.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.202+0000 7f56067fc640 1 --2- 172.21.15.40:0/477167712 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f55fc0a2800 0x7f55fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.202+0000 7f5614b39640 1 --2- 172.21.15.40:0/477167712 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f55fc0a2800 0x7f55fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.202+0000 7f56067fc640 1 -- 172.21.15.40:0/477167712 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f560c0cf1e0 con 0x7f5610100720 2024-03-18T22:47:31.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.202+0000 7f561a189740 1 Processor -- start 2024-03-18T22:47:31.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.202+0000 7f561a189740 1 -- start start 2024-03-18T22:47:31.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f561a189740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496e2f0 0x564ef496e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f561a189740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496ece0 0x564ef49737e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f561a189740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef4973e50 0x564ef49762e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f561a189740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564ef48d24b0 con 0x564ef4973e50 2024-03-18T22:47:31.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f561a189740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564ef487ad70 con 0x564ef496ece0 2024-03-18T22:47:31.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f561a189740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x564ef48ce550 con 0x564ef496e2f0 2024-03-18T22:47:31.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f5614b39640 1 --2- 172.21.15.40:0/477167712 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f55fc0a2800 0x7f55fc0a4cc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5600009230 tx=0x7f5600005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f5615b3b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef4973e50 0x564ef49762e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f5615b3b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef4973e50 0x564ef49762e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55478/0 (socket says 172.21.15.40:55478) 2024-03-18T22:47:31.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f5615b3b640 1 -- 172.21.15.40:0/3975030035 learned_addr learned my addr 172.21.15.40:0/3975030035 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f561533a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496e2f0 0x564ef496e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f5614b39640 1 --2- 172.21.15.40:0/3975030035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496ece0 0x564ef49737e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f5615b3b640 1 -- 172.21.15.40:0/3975030035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496e2f0 msgr2=0x564ef496e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f5615b3b640 1 --2- 172.21.15.40:0/3975030035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496e2f0 0x564ef496e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f5615b3b640 1 -- 172.21.15.40:0/3975030035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496ece0 msgr2=0x564ef49737e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f5615b3b640 1 --2- 172.21.15.40:0/3975030035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496ece0 0x564ef49737e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f5615b3b640 1 -- 172.21.15.40:0/3975030035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ef489d210 con 0x564ef4973e50 2024-03-18T22:47:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f5615b3b640 1 --2- 172.21.15.40:0/3975030035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef4973e50 0x564ef49762e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5608009fd0 tx=0x7f56080118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e9a97b91467f89 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.203+0000 7f55fb7fe640 1 -- 172.21.15.40:0/3975030035 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5608004d20 con 0x564ef4973e50 2024-03-18T22:47:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.204+0000 7f55fb7fe640 1 -- 172.21.15.40:0/3975030035 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5608004050 con 0x564ef4973e50 2024-03-18T22:47:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.204+0000 7f55fb7fe640 1 -- 172.21.15.40:0/3975030035 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5608005280 con 0x564ef4973e50 2024-03-18T22:47:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.204+0000 7f561a189740 1 -- 172.21.15.40:0/3975030035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef4973e50 msgr2=0x564ef49762e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.204+0000 7f561a189740 1 --2- 172.21.15.40:0/3975030035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef4973e50 0x564ef49762e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5608009fd0 tx=0x7f56080118d0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.204+0000 7f561a189740 1 -- 172.21.15.40:0/3975030035 shutdown_connections 2024-03-18T22:47:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.204+0000 7f561a189740 1 --2- 172.21.15.40:0/3975030035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef4973e50 0x564ef49762e0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.204+0000 7f561a189740 1 --2- 172.21.15.40:0/3975030035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496ece0 0x564ef49737e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.204+0000 7f561a189740 1 --2- 172.21.15.40:0/3975030035 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496e2f0 0x564ef496e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.204+0000 7f561a189740 1 -- 172.21.15.40:0/3975030035 >> 172.21.15.40:0/3975030035 conn(0x564ef4969860 msgr2=0x564ef4969c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.204+0000 7f561a189740 1 -- 172.21.15.40:0/3975030035 shutdown_connections 2024-03-18T22:47:31.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.204+0000 7f561a189740 1 -- 172.21.15.40:0/3975030035 wait complete. 2024-03-18T22:47:31.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.205+0000 7f561a189740 1 -- start start 2024-03-18T22:47:31.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.205+0000 7f561a189740 1 Processor -- start 2024-03-18T22:47:31.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.205+0000 7f561a189740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496e2f0 0x564ef4978780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.205+0000 7f561a189740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef496ece0 0x564ef4978cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.205+0000 7f561a189740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef4973e50 0x564ef497fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.205+0000 7f561a189740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564ef4976e00 con 0x564ef496ece0 2024-03-18T22:47:31.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.205+0000 7f561a189740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564ef488e310 con 0x564ef496e2f0 2024-03-18T22:47:31.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.205+0000 7f561a189740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x564ef4976f80 con 0x564ef4973e50 2024-03-18T22:47:31.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.205+0000 7f561a189740 10 client.0 _ll_register_callbacks cb 0x564ef48822d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:31.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.205+0000 7f5614b39640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef496ece0 0x564ef4978cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.205+0000 7f561533a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496e2f0 0x564ef4978780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.205+0000 7f5614b39640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef496ece0 0x564ef4978cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55492/0 (socket says 172.21.15.40:55492) 2024-03-18T22:47:31.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.205+0000 7f5615b3b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef4973e50 0x564ef497fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.205+0000 7f5614b39640 1 -- 172.21.15.40:0/2665937851 learned_addr learned my addr 172.21.15.40:0/2665937851 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.205+0000 7f561533a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496e2f0 0x564ef4978780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56752/0 (socket says 172.21.15.40:56752) 2024-03-18T22:47:31.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.206+0000 7f561533a640 1 -- 172.21.15.40:0/2665937851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef4973e50 msgr2=0x564ef497fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.206+0000 7f561533a640 1 --2- 172.21.15.40:0/2665937851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef4973e50 0x564ef497fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.206+0000 7f561533a640 1 -- 172.21.15.40:0/2665937851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef496ece0 msgr2=0x564ef4978cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.206+0000 7f561533a640 1 --2- 172.21.15.40:0/2665937851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef496ece0 0x564ef4978cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.206+0000 7f561533a640 1 -- 172.21.15.40:0/2665937851 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ef45e63f0 con 0x564ef496e2f0 2024-03-18T22:47:31.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.206+0000 7f5614b39640 1 --2- 172.21.15.40:0/2665937851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef496ece0 0x564ef4978cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.206+0000 7f561533a640 1 --2- 172.21.15.40:0/2665937851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496e2f0 0x564ef4978780 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f560c0cc940 tx=0x7f560c09b1b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bd2878ac638b3f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.206+0000 7f55f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:47:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.206+0000 7f55f8ff9640 1 -- 172.21.15.40:0/2665937851 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c0b3070 con 0x564ef496e2f0 2024-03-18T22:47:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.206+0000 7f55f8ff9640 1 -- 172.21.15.40:0/2665937851 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f560c0a3aa0 con 0x564ef496e2f0 2024-03-18T22:47:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.206+0000 7f561a189740 1 -- 172.21.15.40:0/2665937851 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5608004050 con 0x564ef496e2f0 2024-03-18T22:47:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.206+0000 7f55f8ff9640 1 -- 172.21.15.40:0/2665937851 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f560c09b790 con 0x564ef496e2f0 2024-03-18T22:47:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.206+0000 7f561a189740 10 client.26567 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.206+0000 7f561a189740 1 -- 172.21.15.40:0/2665937851 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564ef46b07d0 con 0x564ef496e2f0 2024-03-18T22:47:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.207+0000 7f55f8ff9640 1 -- 172.21.15.40:0/2665937851 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5608004050 con 0x564ef496e2f0 2024-03-18T22:47:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.207+0000 7f55f8ff9640 1 -- 172.21.15.40:0/2665937851 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f560c0afb70 con 0x564ef496e2f0 2024-03-18T22:47:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.207+0000 7f561a189740 10 client.26567 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.207+0000 7f561a189740 10 client.26567 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:47:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.207+0000 7f561a189740 20 client.26567 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.207+0000 7f561a189740 1 --2- 172.21.15.40:0/2665937851 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x564ef4984590 0x564ef4986a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.207+0000 7f561a189740 4 client.26567 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:47:31.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.207+0000 7f561a189740 1 -- 172.21.15.40:0/2665937851 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ef49303e0 con 0x564ef4984590 2024-03-18T22:47:31.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.207+0000 7f5614b39640 1 --2- 172.21.15.40:0/2665937851 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x564ef4984590 0x564ef4986a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.207+0000 7f55f8ff9640 10 client.26567 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:47:31.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.207+0000 7f5614b39640 1 --2- 172.21.15.40:0/2665937851 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x564ef4984590 0x564ef4986a50 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b651c4358b0ce02b server_cookie=ba04bcbab1f50b0e in_seq=0 out_seq=0 2024-03-18T22:47:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.208+0000 7f55f8ff9640 1 -- 172.21.15.40:0/2665937851 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564ef49303e0 con 0x564ef4984590 2024-03-18T22:47:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.208+0000 7f55f8ff9640 10 client.26567 handle_command_reply: tid=0 2024-03-18T22:47:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.208+0000 7f561a189740 1 client.26567 shutdown 2024-03-18T22:47:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.208+0000 7f561a189740 1 -- 172.21.15.40:0/2665937851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496e2f0 msgr2=0x564ef4978780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.208+0000 7f561a189740 1 --2- 172.21.15.40:0/2665937851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496e2f0 0x564ef4978780 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f560c0cc940 tx=0x7f560c09b1b0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.208+0000 7f561a189740 1 -- 172.21.15.40:0/2665937851 shutdown_connections 2024-03-18T22:47:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.208+0000 7f561a189740 1 --2- 172.21.15.40:0/2665937851 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef4973e50 0x564ef497fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.208+0000 7f561a189740 1 --2- 172.21.15.40:0/2665937851 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x564ef4984590 0x564ef4986a50 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.208+0000 7f561a189740 1 --2- 172.21.15.40:0/2665937851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef496ece0 0x564ef4978cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.208+0000 7f561a189740 1 --2- 172.21.15.40:0/2665937851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496e2f0 0x564ef4978780 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.208+0000 7f55f8ff9640 0 client.26567 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:47:31.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.208+0000 7f561a189740 1 -- 172.21.15.40:0/2665937851 >> 172.21.15.40:0/2665937851 conn(0x564ef4969860 msgr2=0x564ef4968ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.208+0000 7f561a189740 1 -- 172.21.15.40:0/2665937851 shutdown_connections 2024-03-18T22:47:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.208+0000 7f561a189740 1 -- 172.21.15.40:0/2665937851 wait complete. 2024-03-18T22:47:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.209+0000 7f561a189740 20 client.26567 trim_cache size 0 max 16384 2024-03-18T22:47:31.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f561a189740 1 Processor -- start 2024-03-18T22:47:31.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f561a189740 1 -- start start 2024-03-18T22:47:31.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f561a189740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496e2f0 0x564ef4978390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f561a189740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496ece0 0x564ef497fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f561a189740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef4973e50 0x564ef49788d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f561a189740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f56101aa4a0 con 0x564ef4973e50 2024-03-18T22:47:31.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f561a189740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f56101abfe0 con 0x564ef496ece0 2024-03-18T22:47:31.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f561a189740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f56101aab70 con 0x564ef496e2f0 2024-03-18T22:47:31.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f5615b3b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef4973e50 0x564ef49788d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f561533a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496e2f0 0x564ef4978390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f5615b3b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef4973e50 0x564ef49788d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55494/0 (socket says 172.21.15.40:55494) 2024-03-18T22:47:31.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f5615b3b640 1 -- 172.21.15.40:0/1266958488 learned_addr learned my addr 172.21.15.40:0/1266958488 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f561533a640 1 --2- 172.21.15.40:0/1266958488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496e2f0 0x564ef4978390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33216/0 (socket says 172.21.15.40:33216) 2024-03-18T22:47:31.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f5614b39640 1 --2- 172.21.15.40:0/1266958488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496ece0 0x564ef497fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f5615b3b640 1 -- 172.21.15.40:0/1266958488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496e2f0 msgr2=0x564ef4978390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f5615b3b640 1 --2- 172.21.15.40:0/1266958488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496e2f0 0x564ef4978390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f5615b3b640 1 -- 172.21.15.40:0/1266958488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496ece0 msgr2=0x564ef497fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f5615b3b640 1 --2- 172.21.15.40:0/1266958488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496ece0 0x564ef497fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f5615b3b640 1 -- 172.21.15.40:0/1266958488 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ef4969400 con 0x564ef4973e50 2024-03-18T22:47:31.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f561533a640 1 --2- 172.21.15.40:0/1266958488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496e2f0 0x564ef4978390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.227+0000 7f5614b39640 1 --2- 172.21.15.40:0/1266958488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496ece0 0x564ef497fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.228+0000 7f5615b3b640 1 --2- 172.21.15.40:0/1266958488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef4973e50 0x564ef49788d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f5608011dd0 tx=0x7f5608009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa3814028676e305 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.228+0000 7f55faffd640 1 -- 172.21.15.40:0/1266958488 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56080039f0 con 0x564ef4973e50 2024-03-18T22:47:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.228+0000 7f55faffd640 1 -- 172.21.15.40:0/1266958488 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5608004500 con 0x564ef4973e50 2024-03-18T22:47:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.228+0000 7f55faffd640 1 -- 172.21.15.40:0/1266958488 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5608005030 con 0x564ef4973e50 2024-03-18T22:47:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 -- 172.21.15.40:0/1266958488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef4973e50 msgr2=0x564ef49788d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 --2- 172.21.15.40:0/1266958488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef4973e50 0x564ef49788d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f5608011dd0 tx=0x7f5608009cc0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 -- 172.21.15.40:0/1266958488 shutdown_connections 2024-03-18T22:47:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 --2- 172.21.15.40:0/1266958488 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef4973e50 0x564ef49788d0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 --2- 172.21.15.40:0/1266958488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef496ece0 0x564ef497fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 --2- 172.21.15.40:0/1266958488 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496e2f0 0x564ef4978390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 -- 172.21.15.40:0/1266958488 >> 172.21.15.40:0/1266958488 conn(0x564ef4969860 msgr2=0x564ef4969c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 -- 172.21.15.40:0/1266958488 shutdown_connections 2024-03-18T22:47:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 -- 172.21.15.40:0/1266958488 wait complete. 2024-03-18T22:47:31.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 -- start start 2024-03-18T22:47:31.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 Processor -- start 2024-03-18T22:47:31.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef496e2f0 0x564ef497d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496ece0 0x564ef497da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef4973e50 0x564ef497ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f56101ab910 con 0x564ef496e2f0 2024-03-18T22:47:31.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f56101ab240 con 0x564ef4973e50 2024-03-18T22:47:31.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.229+0000 7f561a189740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f56101a8480 con 0x564ef496ece0 2024-03-18T22:47:31.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f561a189740 10 client.0 _ll_register_callbacks cb 0x7f560c097ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:31.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f561533a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef496e2f0 0x564ef497d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f561533a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef496e2f0 0x564ef497d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55496/0 (socket says 172.21.15.40:55496) 2024-03-18T22:47:31.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f5615b3b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef4973e50 0x564ef497ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f5614b39640 1 --2- 172.21.15.40:0/4293550407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496ece0 0x564ef497da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f561533a640 1 -- 172.21.15.40:0/4293550407 learned_addr learned my addr 172.21.15.40:0/4293550407 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f561533a640 1 -- 172.21.15.40:0/4293550407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496ece0 msgr2=0x564ef497da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f561533a640 1 --2- 172.21.15.40:0/4293550407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496ece0 0x564ef497da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f561533a640 1 -- 172.21.15.40:0/4293550407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef4973e50 msgr2=0x564ef497ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f561533a640 1 --2- 172.21.15.40:0/4293550407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef4973e50 0x564ef497ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f561533a640 1 -- 172.21.15.40:0/4293550407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ef493f880 con 0x564ef496e2f0 2024-03-18T22:47:31.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f5614b39640 1 --2- 172.21.15.40:0/4293550407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496ece0 0x564ef497da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f5615b3b640 1 --2- 172.21.15.40:0/4293550407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef4973e50 0x564ef497ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f561533a640 1 --2- 172.21.15.40:0/4293550407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef496e2f0 0x564ef497d540 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f560c09b950 tx=0x7f560c099ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c93e22d0f9a135 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f55f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:47:31.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f55f8ff9640 1 -- 172.21.15.40:0/4293550407 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560c0b3070 con 0x564ef496e2f0 2024-03-18T22:47:31.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f55f8ff9640 1 -- 172.21.15.40:0/4293550407 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f560c0a3ca0 con 0x564ef496e2f0 2024-03-18T22:47:31.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f561a189740 1 -- 172.21.15.40:0/4293550407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5608004500 con 0x564ef496e2f0 2024-03-18T22:47:31.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f55f8ff9640 1 -- 172.21.15.40:0/4293550407 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f560c0ab050 con 0x564ef496e2f0 2024-03-18T22:47:31.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f561a189740 10 client.16746 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:31.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.230+0000 7f561a189740 1 -- 172.21.15.40:0/4293550407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564ef49ea950 con 0x564ef496e2f0 2024-03-18T22:47:31.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.231+0000 7f55f8ff9640 1 -- 172.21.15.40:0/4293550407 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f560c0a5020 con 0x564ef496e2f0 2024-03-18T22:47:31.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.231+0000 7f55f8ff9640 1 -- 172.21.15.40:0/4293550407 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f560c083020 con 0x564ef496e2f0 2024-03-18T22:47:31.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.231+0000 7f561a189740 10 client.16746 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:31.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.231+0000 7f561a189740 10 client.16746 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:47:31.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.231+0000 7f561a189740 20 client.16746 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:31.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.231+0000 7f561a189740 1 --2- 172.21.15.40:0/4293550407 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x564ef49eb2f0 0x564ef49ed6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.231+0000 7f561a189740 4 client.16746 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:47:31.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.231+0000 7f561a189740 1 -- 172.21.15.40:0/4293550407 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564ef48df9f0 con 0x564ef49eb2f0 2024-03-18T22:47:31.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.231+0000 7f5614b39640 1 --2- 172.21.15.40:0/4293550407 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x564ef49eb2f0 0x564ef49ed6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.231+0000 7f55f8ff9640 10 client.16746 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:47:31.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.231+0000 7f5614b39640 1 --2- 172.21.15.40:0/4293550407 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x564ef49eb2f0 0x564ef49ed6e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e3e722bf279b841e server_cookie=35ebcd6d404924c3 in_seq=0 out_seq=0 2024-03-18T22:47:31.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.232+0000 7f55f8ff9640 1 -- 172.21.15.40:0/4293550407 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564ef48df9f0 con 0x564ef49eb2f0 2024-03-18T22:47:31.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.232+0000 7f55f8ff9640 10 client.16746 handle_command_reply: tid=0 2024-03-18T22:47:31.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.232+0000 7f561a189740 1 client.16746 shutdown 2024-03-18T22:47:31.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.232+0000 7f561a189740 1 -- 172.21.15.40:0/4293550407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef496e2f0 msgr2=0x564ef497d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.232+0000 7f561a189740 1 --2- 172.21.15.40:0/4293550407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef496e2f0 0x564ef497d540 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f560c09b950 tx=0x7f560c099ca0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.232+0000 7f561a189740 1 -- 172.21.15.40:0/4293550407 shutdown_connections 2024-03-18T22:47:31.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.232+0000 7f561a189740 1 --2- 172.21.15.40:0/4293550407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ef4973e50 0x564ef497ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.232+0000 7f561a189740 1 --2- 172.21.15.40:0/4293550407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564ef496ece0 0x564ef497da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.232+0000 7f561a189740 1 --2- 172.21.15.40:0/4293550407 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x564ef49eb2f0 0x564ef49ed6e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.232+0000 7f561a189740 1 --2- 172.21.15.40:0/4293550407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564ef496e2f0 0x564ef497d540 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.232+0000 7f55f8ff9640 0 client.16746 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:47:31.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.232+0000 7f561a189740 1 -- 172.21.15.40:0/4293550407 >> 172.21.15.40:0/4293550407 conn(0x564ef4969860 msgr2=0x564ef4978390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.232+0000 7f561a189740 1 -- 172.21.15.40:0/4293550407 shutdown_connections 2024-03-18T22:47:31.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.232+0000 7f561a189740 1 -- 172.21.15.40:0/4293550407 wait complete. 2024-03-18T22:47:31.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.232+0000 7f561a189740 20 client.16746 trim_cache size 0 max 16384 2024-03-18T22:47:31.240 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:47:31.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.233+0000 7f561759e640 1 -- 172.21.15.40:0/477167712 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f55fc0a2800 msgr2=0x7f55fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.233+0000 7f561759e640 1 --2- 172.21.15.40:0/477167712 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f55fc0a2800 0x7f55fc0a4cc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5600009230 tx=0x7f5600005fd0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.233+0000 7f561759e640 1 -- 172.21.15.40:0/477167712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5610100720 msgr2=0x7f5610197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.233+0000 7f561759e640 1 --2- 172.21.15.40:0/477167712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5610100720 0x7f5610197aa0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f560c00e9c0 tx=0x7f560c00ee80 comp rx=0 tx=0).stop 2024-03-18T22:47:31.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.233+0000 7f561759e640 1 -- 172.21.15.40:0/477167712 shutdown_connections 2024-03-18T22:47:31.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.233+0000 7f561759e640 1 --2- 172.21.15.40:0/477167712 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f55fc0a2800 0x7f55fc0a4cc0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.233+0000 7f561759e640 1 --2- 172.21.15.40:0/477167712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56101022d0 0x7f561019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.233+0000 7f561759e640 1 --2- 172.21.15.40:0/477167712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5610101910 0x7f5610197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.233+0000 7f561759e640 1 --2- 172.21.15.40:0/477167712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5610100720 0x7f5610197aa0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.233+0000 7f561759e640 1 -- 172.21.15.40:0/477167712 >> 172.21.15.40:0/477167712 conn(0x7f56100fbc90 msgr2=0x7f5610068a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.233+0000 7f561759e640 1 -- 172.21.15.40:0/477167712 shutdown_connections 2024-03-18T22:47:31.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.233+0000 7f561759e640 1 -- 172.21.15.40:0/477167712 wait complete. 2024-03-18T22:47:31.244 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:47:31.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.356+0000 7f186d685640 1 -- 172.21.15.40:0/1037041733 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1868102340 msgr2=0x7f1868104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.356+0000 7f186d685640 1 --2- 172.21.15.40:0/1037041733 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1868102340 0x7f1868104730 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1858009900 tx=0x7f185802bc30 comp rx=0 tx=0).stop 2024-03-18T22:47:31.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.356+0000 7f186d685640 1 -- 172.21.15.40:0/1037041733 shutdown_connections 2024-03-18T22:47:31.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.356+0000 7f186d685640 1 --2- 172.21.15.40:0/1037041733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18681075a0 0x7f1868109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.356+0000 7f186d685640 1 --2- 172.21.15.40:0/1037041733 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1868104c70 0x7f1868107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.356+0000 7f186d685640 1 --2- 172.21.15.40:0/1037041733 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1868102340 0x7f1868104730 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.356+0000 7f186d685640 1 -- 172.21.15.40:0/1037041733 >> 172.21.15.40:0/1037041733 conn(0x7f18680fbc90 msgr2=0x7f18680fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.356+0000 7f186d685640 1 -- 172.21.15.40:0/1037041733 shutdown_connections 2024-03-18T22:47:31.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.356+0000 7f186d685640 1 -- 172.21.15.40:0/1037041733 wait complete. 2024-03-18T22:47:31.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.357+0000 7f186d685640 1 Processor -- start 2024-03-18T22:47:31.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.357+0000 7f186d685640 1 -- start start 2024-03-18T22:47:31.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.357+0000 7f186d685640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1868102340 0x7f1868197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.357+0000 7f186d685640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1868104c70 0x7f1868197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.357+0000 7f186d685640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18681075a0 0x7f186819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.357+0000 7f186d685640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f186810d360 con 0x7f1868102340 2024-03-18T22:47:31.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.357+0000 7f186d685640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f186810d1e0 con 0x7f18681075a0 2024-03-18T22:47:31.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.357+0000 7f186d685640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f186810d4e0 con 0x7f1868104c70 2024-03-18T22:47:31.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.357+0000 7f1866ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1868102340 0x7f1868197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.357+0000 7f185ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1868104c70 0x7f1868197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.357+0000 7f1866ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1868102340 0x7f1868197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55514/0 (socket says 172.21.15.40:55514) 2024-03-18T22:47:31.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.357+0000 7f1866ffd640 1 -- 172.21.15.40:0/3277800197 learned_addr learned my addr 172.21.15.40:0/3277800197 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.357+0000 7f18677fe640 1 --2- 172.21.15.40:0/3277800197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18681075a0 0x7f186819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.358+0000 7f185ffff640 1 -- 172.21.15.40:0/3277800197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18681075a0 msgr2=0x7f186819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.358+0000 7f185ffff640 1 --2- 172.21.15.40:0/3277800197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18681075a0 0x7f186819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.358+0000 7f185ffff640 1 -- 172.21.15.40:0/3277800197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1868102340 msgr2=0x7f1868197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.358+0000 7f185ffff640 1 --2- 172.21.15.40:0/3277800197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1868102340 0x7f1868197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.358+0000 7f185ffff640 1 -- 172.21.15.40:0/3277800197 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f186819f650 con 0x7f1868104c70 2024-03-18T22:47:31.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.358+0000 7f1866ffd640 1 --2- 172.21.15.40:0/3277800197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1868102340 0x7f1868197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.358+0000 7f185ffff640 1 --2- 172.21.15.40:0/3277800197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1868104c70 0x7f1868197f70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f185000e9c0 tx=0x7f185000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.358+0000 7f1864ff9640 1 -- 172.21.15.40:0/3277800197 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185000ccc0 con 0x7f1868104c70 2024-03-18T22:47:31.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.358+0000 7f1864ff9640 1 -- 172.21.15.40:0/3277800197 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18500108e0 con 0x7f1868104c70 2024-03-18T22:47:31.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.358+0000 7f186d685640 1 -- 172.21.15.40:0/3277800197 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f186819f940 con 0x7f1868104c70 2024-03-18T22:47:31.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.358+0000 7f1864ff9640 1 -- 172.21.15.40:0/3277800197 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18500187e0 con 0x7f1868104c70 2024-03-18T22:47:31.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.358+0000 7f186d685640 1 -- 172.21.15.40:0/3277800197 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f186819fe90 con 0x7f1868104c70 2024-03-18T22:47:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.360+0000 7f1864ff9640 1 -- 172.21.15.40:0/3277800197 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1850010a90 con 0x7f1868104c70 2024-03-18T22:47:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.361+0000 7f1864ff9640 1 --2- 172.21.15.40:0/3277800197 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18540a2800 0x7f18540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.361+0000 7f1870277740 1 Processor -- start 2024-03-18T22:47:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.361+0000 7f1870277740 1 -- start start 2024-03-18T22:47:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.361+0000 7f1866ffd640 1 --2- 172.21.15.40:0/3277800197 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18540a2800 0x7f18540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.361+0000 7f1864ff9640 1 -- 172.21.15.40:0/3277800197 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f18500cf1f0 con 0x7f1868104c70 2024-03-18T22:47:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.361+0000 7f1870277740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 0x565147629710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.361+0000 7f1870277740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565147629ce0 0x56514762e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.361+0000 7f1870277740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56514762ee50 0x5651476312e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.361+0000 7f1870277740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56514758d4b0 con 0x56514762ee50 2024-03-18T22:47:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.361+0000 7f1870277740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x565147535d70 con 0x565147629ce0 2024-03-18T22:47:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.361+0000 7f1870277740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x565147589550 con 0x5651476292f0 2024-03-18T22:47:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f18677fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56514762ee50 0x5651476312e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f185ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565147629ce0 0x56514762e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f18677fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56514762ee50 0x5651476312e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55528/0 (socket says 172.21.15.40:55528) 2024-03-18T22:47:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f18677fe640 1 -- 172.21.15.40:0/651375173 learned_addr learned my addr 172.21.15.40:0/651375173 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f1866ffd640 1 --2- 172.21.15.40:0/651375173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 0x565147629710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f185ffff640 1 -- 172.21.15.40:0/651375173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 msgr2=0x565147629710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f185ffff640 1 --2- 172.21.15.40:0/651375173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 0x565147629710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f185ffff640 1 -- 172.21.15.40:0/651375173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56514762ee50 msgr2=0x5651476312e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f1866ffd640 1 --2- 172.21.15.40:0/3277800197 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18540a2800 0x7f18540a4cc0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f18580096c0 tx=0x7f185802cc90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f185ffff640 1 --2- 172.21.15.40:0/651375173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56514762ee50 0x5651476312e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f185ffff640 1 -- 172.21.15.40:0/651375173 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565147558210 con 0x565147629ce0 2024-03-18T22:47:31.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f18677fe640 1 --2- 172.21.15.40:0/651375173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56514762ee50 0x5651476312e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f185ffff640 1 --2- 172.21.15.40:0/651375173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565147629ce0 0x56514762e7e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f18500cc8a0 tx=0x7f185000efb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7856aef3a4973237 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f185d7fa640 1 -- 172.21.15.40:0/651375173 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1850099af0 con 0x565147629ce0 2024-03-18T22:47:31.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f185d7fa640 1 -- 172.21.15.40:0/651375173 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1850097e60 con 0x565147629ce0 2024-03-18T22:47:31.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.362+0000 7f185d7fa640 1 -- 172.21.15.40:0/651375173 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185009af30 con 0x565147629ce0 2024-03-18T22:47:31.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.363+0000 7f1870277740 1 -- 172.21.15.40:0/651375173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565147629ce0 msgr2=0x56514762e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.363+0000 7f1870277740 1 --2- 172.21.15.40:0/651375173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565147629ce0 0x56514762e7e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f18500cc8a0 tx=0x7f185000efb0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.363+0000 7f1870277740 1 -- 172.21.15.40:0/651375173 shutdown_connections 2024-03-18T22:47:31.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.363+0000 7f1870277740 1 --2- 172.21.15.40:0/651375173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56514762ee50 0x5651476312e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.363+0000 7f1870277740 1 --2- 172.21.15.40:0/651375173 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565147629ce0 0x56514762e7e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.363+0000 7f1870277740 1 --2- 172.21.15.40:0/651375173 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 0x565147629710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.363+0000 7f1870277740 1 -- 172.21.15.40:0/651375173 >> 172.21.15.40:0/651375173 conn(0x565147624860 msgr2=0x565147624c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.363+0000 7f1870277740 1 -- 172.21.15.40:0/651375173 shutdown_connections 2024-03-18T22:47:31.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.363+0000 7f1870277740 1 -- 172.21.15.40:0/651375173 wait complete. 2024-03-18T22:47:31.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f1870277740 1 -- start start 2024-03-18T22:47:31.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f1870277740 1 Processor -- start 2024-03-18T22:47:31.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f1870277740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651476292f0 0x565147637c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f1870277740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x565147638160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f1870277740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56514762ee50 0x56514763f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f1870277740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x565147636150 con 0x565147629ce0 2024-03-18T22:47:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f1870277740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x565147549310 con 0x5651476292f0 2024-03-18T22:47:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f1870277740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5651476362d0 con 0x56514762ee50 2024-03-18T22:47:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f1870277740 10 client.0 _ll_register_callbacks cb 0x56514753d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f185ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x565147638160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f1866ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651476292f0 0x565147637c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f185ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x565147638160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55542/0 (socket says 172.21.15.40:55542) 2024-03-18T22:47:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f185ffff640 1 -- 172.21.15.40:0/1999046520 learned_addr learned my addr 172.21.15.40:0/1999046520 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f18677fe640 1 --2- 172.21.15.40:0/1999046520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56514762ee50 0x56514763f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f18677fe640 1 -- 172.21.15.40:0/1999046520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651476292f0 msgr2=0x565147637c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f18677fe640 1 --2- 172.21.15.40:0/1999046520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651476292f0 0x565147637c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f18677fe640 1 -- 172.21.15.40:0/1999046520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 msgr2=0x565147638160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f18677fe640 1 --2- 172.21.15.40:0/1999046520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x565147638160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f18677fe640 1 -- 172.21.15.40:0/1999046520 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651472a13f0 con 0x56514762ee50 2024-03-18T22:47:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f185ffff640 1 --2- 172.21.15.40:0/1999046520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x565147638160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.364+0000 7f1866ffd640 1 --2- 172.21.15.40:0/1999046520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651476292f0 0x565147637c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.365+0000 7f18677fe640 1 --2- 172.21.15.40:0/1999046520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56514762ee50 0x56514763f1a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x5651476257d0 tx=0x7f186000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2e5426d874600d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.365+0000 7f184e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:47:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.365+0000 7f184e7fc640 1 -- 172.21.15.40:0/1999046520 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1860009e30 con 0x56514762ee50 2024-03-18T22:47:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.365+0000 7f184e7fc640 1 -- 172.21.15.40:0/1999046520 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1860003970 con 0x56514762ee50 2024-03-18T22:47:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.365+0000 7f1870277740 1 -- 172.21.15.40:0/1999046520 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1850097e60 con 0x56514762ee50 2024-03-18T22:47:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.365+0000 7f184e7fc640 1 -- 172.21.15.40:0/1999046520 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1860025050 con 0x56514762ee50 2024-03-18T22:47:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.365+0000 7f1870277740 10 client.26575 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.365+0000 7f1870277740 1 -- 172.21.15.40:0/1999046520 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56514736b7d0 con 0x56514762ee50 2024-03-18T22:47:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.365+0000 7f184e7fc640 1 -- 172.21.15.40:0/1999046520 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1850097e60 con 0x56514762ee50 2024-03-18T22:47:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.365+0000 7f184e7fc640 1 -- 172.21.15.40:0/1999046520 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f186001a410 con 0x56514762ee50 2024-03-18T22:47:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.365+0000 7f1870277740 10 client.26575 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.365+0000 7f1870277740 10 client.26575 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:47:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.365+0000 7f1870277740 20 client.26575 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.365+0000 7f1870277740 1 --2- 172.21.15.40:0/1999046520 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x565147643b40 0x565147645f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.366+0000 7f1870277740 4 client.26575 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:47:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.366+0000 7f1870277740 1 -- 172.21.15.40:0/1999046520 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651475eb3e0 con 0x565147643b40 2024-03-18T22:47:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.366+0000 7f1866ffd640 1 --2- 172.21.15.40:0/1999046520 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x565147643b40 0x565147645f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.366+0000 7f184e7fc640 10 client.26575 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:47:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.366+0000 7f1866ffd640 1 --2- 172.21.15.40:0/1999046520 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x565147643b40 0x565147645f80 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9fda71537d24ec26 server_cookie=a41b6b6177e2a435 in_seq=0 out_seq=0 2024-03-18T22:47:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.367+0000 7f184e7fc640 1 -- 172.21.15.40:0/1999046520 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5651475eb3e0 con 0x565147643b40 2024-03-18T22:47:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.367+0000 7f184e7fc640 10 client.26575 handle_command_reply: tid=0 2024-03-18T22:47:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.367+0000 7f1870277740 1 client.26575 shutdown 2024-03-18T22:47:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.367+0000 7f1870277740 1 -- 172.21.15.40:0/1999046520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56514762ee50 msgr2=0x56514763f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.367+0000 7f1870277740 1 --2- 172.21.15.40:0/1999046520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56514762ee50 0x56514763f1a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x5651476257d0 tx=0x7f186000f790 comp rx=0 tx=0).stop 2024-03-18T22:47:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.367+0000 7f1870277740 1 -- 172.21.15.40:0/1999046520 shutdown_connections 2024-03-18T22:47:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.367+0000 7f1870277740 1 --2- 172.21.15.40:0/1999046520 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x565147643b40 0x565147645f80 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.367+0000 7f1870277740 1 --2- 172.21.15.40:0/1999046520 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56514762ee50 0x56514763f1a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.367+0000 7f1870277740 1 --2- 172.21.15.40:0/1999046520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x565147638160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.367+0000 7f184e7fc640 0 client.26575 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:47:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.367+0000 7f1870277740 1 --2- 172.21.15.40:0/1999046520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651476292f0 0x565147637c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.367+0000 7f1870277740 1 -- 172.21.15.40:0/1999046520 >> 172.21.15.40:0/1999046520 conn(0x565147624860 msgr2=0x565147623ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.367+0000 7f1870277740 1 -- 172.21.15.40:0/1999046520 shutdown_connections 2024-03-18T22:47:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.367+0000 7f1870277740 1 -- 172.21.15.40:0/1999046520 wait complete. 2024-03-18T22:47:31.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.367+0000 7f1870277740 20 client.26575 trim_cache size 0 max 16384 2024-03-18T22:47:31.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.382+0000 7f1870277740 1 Processor -- start 2024-03-18T22:47:31.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.382+0000 7f1870277740 1 -- start start 2024-03-18T22:47:31.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f1870277740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 0x565147636670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f1870277740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x56514763f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f1870277740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56514762ee50 0x565147637c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f1870277740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f18681a8390 con 0x565147629ce0 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f1870277740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18681a9ed0 con 0x56514762ee50 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f1870277740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f18681a8a60 con 0x5651476292f0 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f185ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x56514763f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f1866ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 0x565147636670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f185ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x56514763f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55554/0 (socket says 172.21.15.40:55554) 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f185ffff640 1 -- 172.21.15.40:0/1203973754 learned_addr learned my addr 172.21.15.40:0/1203973754 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f18677fe640 1 --2- 172.21.15.40:0/1203973754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56514762ee50 0x565147637c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f1866ffd640 1 -- 172.21.15.40:0/1203973754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56514762ee50 msgr2=0x565147637c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f1866ffd640 1 --2- 172.21.15.40:0/1203973754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56514762ee50 0x565147637c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f1866ffd640 1 -- 172.21.15.40:0/1203973754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 msgr2=0x56514763f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f1866ffd640 1 --2- 172.21.15.40:0/1203973754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x56514763f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f1866ffd640 1 -- 172.21.15.40:0/1203973754 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651475fa880 con 0x5651476292f0 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f185ffff640 1 --2- 172.21.15.40:0/1203973754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x56514763f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.383+0000 7f1866ffd640 1 --2- 172.21.15.40:0/1203973754 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 0x565147636670 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1858009230 tx=0x7f1858042040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=521050c0120af246 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.384+0000 7f185cff9640 1 -- 172.21.15.40:0/1203973754 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1858047070 con 0x5651476292f0 2024-03-18T22:47:31.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.384+0000 7f185cff9640 1 -- 172.21.15.40:0/1203973754 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1858005490 con 0x5651476292f0 2024-03-18T22:47:31.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.384+0000 7f185cff9640 1 -- 172.21.15.40:0/1203973754 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18580038b0 con 0x5651476292f0 2024-03-18T22:47:31.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.384+0000 7f1870277740 1 -- 172.21.15.40:0/1203973754 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 msgr2=0x565147636670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.384+0000 7f1870277740 1 --2- 172.21.15.40:0/1203973754 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 0x565147636670 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1858009230 tx=0x7f1858042040 comp rx=0 tx=0).stop 2024-03-18T22:47:31.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.384+0000 7f1870277740 1 -- 172.21.15.40:0/1203973754 shutdown_connections 2024-03-18T22:47:31.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.384+0000 7f1870277740 1 --2- 172.21.15.40:0/1203973754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56514762ee50 0x565147637c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.384+0000 7f1870277740 1 --2- 172.21.15.40:0/1203973754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x56514763f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.384+0000 7f1870277740 1 --2- 172.21.15.40:0/1203973754 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 0x565147636670 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.384+0000 7f1870277740 1 -- 172.21.15.40:0/1203973754 >> 172.21.15.40:0/1203973754 conn(0x565147624860 msgr2=0x56514761fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.384+0000 7f1870277740 1 -- 172.21.15.40:0/1203973754 shutdown_connections 2024-03-18T22:47:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.384+0000 7f1870277740 1 -- 172.21.15.40:0/1203973754 wait complete. 2024-03-18T22:47:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1870277740 1 -- start start 2024-03-18T22:47:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1870277740 1 Processor -- start 2024-03-18T22:47:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1870277740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 0x56514763f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1870277740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x56514763bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1870277740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56514762ee50 0x56514763c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1870277740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f18681a9800 con 0x565147629ce0 2024-03-18T22:47:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1870277740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18681a9130 con 0x56514762ee50 2024-03-18T22:47:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1870277740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f18681a6370 con 0x5651476292f0 2024-03-18T22:47:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1870277740 10 client.0 _ll_register_callbacks cb 0x7f1860009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1866ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 0x56514763f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f185ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x56514763bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1866ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 0x56514763f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33292/0 (socket says 172.21.15.40:33292) 2024-03-18T22:47:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1866ffd640 1 -- 172.21.15.40:0/3853279376 learned_addr learned my addr 172.21.15.40:0/3853279376 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f18677fe640 1 --2- 172.21.15.40:0/3853279376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56514762ee50 0x56514763c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1866ffd640 1 -- 172.21.15.40:0/3853279376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56514762ee50 msgr2=0x56514763c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1866ffd640 1 --2- 172.21.15.40:0/3853279376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56514762ee50 0x56514763c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1866ffd640 1 -- 172.21.15.40:0/3853279376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 msgr2=0x56514763bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1866ffd640 1 --2- 172.21.15.40:0/3853279376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x56514763bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f1866ffd640 1 -- 172.21.15.40:0/3853279376 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651476a7ef0 con 0x5651476292f0 2024-03-18T22:47:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.385+0000 7f185ffff640 1 --2- 172.21.15.40:0/3853279376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x56514763bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f1866ffd640 1 --2- 172.21.15.40:0/3853279376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 0x56514763f140 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1858003630 tx=0x7f1858004570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e046c05008e03f50 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f184e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:47:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f184e7fc640 1 -- 172.21.15.40:0/3853279376 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1858058030 con 0x5651476292f0 2024-03-18T22:47:31.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f184e7fc640 1 -- 172.21.15.40:0/3853279376 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1858053070 con 0x5651476292f0 2024-03-18T22:47:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f1870277740 1 -- 172.21.15.40:0/3853279376 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1858005490 con 0x5651476292f0 2024-03-18T22:47:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f184e7fc640 1 -- 172.21.15.40:0/3853279376 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1858047040 con 0x5651476292f0 2024-03-18T22:47:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f1870277740 10 client.26581 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f1870277740 1 -- 172.21.15.40:0/3853279376 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5651476a9290 con 0x5651476292f0 2024-03-18T22:47:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f184e7fc640 1 -- 172.21.15.40:0/3853279376 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1858005490 con 0x5651476292f0 2024-03-18T22:47:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f184e7fc640 1 -- 172.21.15.40:0/3853279376 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1858034440 con 0x5651476292f0 2024-03-18T22:47:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f1870277740 10 client.26581 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f1870277740 10 client.26581 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:47:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f1870277740 20 client.26581 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f1870277740 1 --2- 172.21.15.40:0/3853279376 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x5651476a9c40 0x5651476ac060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f1870277740 4 client.26581 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:47:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.386+0000 7f1870277740 1 -- 172.21.15.40:0/3853279376 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56514759a9f0 con 0x5651476a9c40 2024-03-18T22:47:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f185ffff640 1 --2- 172.21.15.40:0/3853279376 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x5651476a9c40 0x5651476ac060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f185ffff640 1 --2- 172.21.15.40:0/3853279376 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x5651476a9c40 0x5651476ac060 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=50b15cbf95bc2739 server_cookie=d8f4c85e794db5d9 in_seq=0 out_seq=0 2024-03-18T22:47:31.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f184e7fc640 10 client.26581 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:47:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f184e7fc640 1 -- 172.21.15.40:0/3853279376 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56514759a9f0 con 0x5651476a9c40 2024-03-18T22:47:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f184e7fc640 10 client.26581 handle_command_reply: tid=0 2024-03-18T22:47:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f1870277740 1 client.26581 shutdown 2024-03-18T22:47:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f1870277740 1 -- 172.21.15.40:0/3853279376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 msgr2=0x56514763f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f1870277740 1 --2- 172.21.15.40:0/3853279376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 0x56514763f140 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1858003630 tx=0x7f1858004570 comp rx=0 tx=0).stop 2024-03-18T22:47:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f1870277740 1 -- 172.21.15.40:0/3853279376 shutdown_connections 2024-03-18T22:47:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f1870277740 1 --2- 172.21.15.40:0/3853279376 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x5651476a9c40 0x5651476ac060 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f1870277740 1 --2- 172.21.15.40:0/3853279376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56514762ee50 0x56514763c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f1870277740 1 --2- 172.21.15.40:0/3853279376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565147629ce0 0x56514763bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f184e7fc640 0 client.26581 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:47:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f1870277740 1 --2- 172.21.15.40:0/3853279376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651476292f0 0x56514763f140 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f1870277740 1 -- 172.21.15.40:0/3853279376 >> 172.21.15.40:0/3853279376 conn(0x565147624860 msgr2=0x565147636670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f1870277740 1 -- 172.21.15.40:0/3853279376 shutdown_connections 2024-03-18T22:47:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.387+0000 7f1870277740 1 -- 172.21.15.40:0/3853279376 wait complete. 2024-03-18T22:47:31.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.388+0000 7f1870277740 20 client.26581 trim_cache size 0 max 16384 2024-03-18T22:47:31.395 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:47:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.388+0000 7f186d685640 1 -- 172.21.15.40:0/3277800197 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18540a2800 msgr2=0x7f18540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.388+0000 7f186d685640 1 --2- 172.21.15.40:0/3277800197 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18540a2800 0x7f18540a4cc0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f18580096c0 tx=0x7f185802cc90 comp rx=0 tx=0).stop 2024-03-18T22:47:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.388+0000 7f186d685640 1 -- 172.21.15.40:0/3277800197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1868104c70 msgr2=0x7f1868197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.388+0000 7f186d685640 1 --2- 172.21.15.40:0/3277800197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1868104c70 0x7f1868197f70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f185000e9c0 tx=0x7f185000ee80 comp rx=0 tx=0).stop 2024-03-18T22:47:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.388+0000 7f186d685640 1 -- 172.21.15.40:0/3277800197 shutdown_connections 2024-03-18T22:47:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.388+0000 7f186d685640 1 --2- 172.21.15.40:0/3277800197 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f18540a2800 0x7f18540a4cc0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.388+0000 7f186d685640 1 --2- 172.21.15.40:0/3277800197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18681075a0 0x7f186819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.388+0000 7f186d685640 1 --2- 172.21.15.40:0/3277800197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1868104c70 0x7f1868197f70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.388+0000 7f186d685640 1 --2- 172.21.15.40:0/3277800197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1868102340 0x7f1868197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.388+0000 7f186d685640 1 -- 172.21.15.40:0/3277800197 >> 172.21.15.40:0/3277800197 conn(0x7f18680fbc90 msgr2=0x7f1868102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.388+0000 7f186d685640 1 -- 172.21.15.40:0/3277800197 shutdown_connections 2024-03-18T22:47:31.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.388+0000 7f186d685640 1 -- 172.21.15.40:0/3277800197 wait complete. 2024-03-18T22:47:31.397 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:31 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2506967098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:47:31.399 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:47:31.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.517+0000 7fef764c6640 1 -- 172.21.15.40:0/3140629319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef70104c70 msgr2=0x7fef70107060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.517+0000 7fef764c6640 1 --2- 172.21.15.40:0/3140629319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef70104c70 0x7fef70107060 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fef64009900 tx=0x7fef6402bc30 comp rx=0 tx=0).stop 2024-03-18T22:47:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.517+0000 7fef764c6640 1 -- 172.21.15.40:0/3140629319 shutdown_connections 2024-03-18T22:47:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.517+0000 7fef764c6640 1 --2- 172.21.15.40:0/3140629319 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef701075a0 0x7fef70109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.517+0000 7fef764c6640 1 --2- 172.21.15.40:0/3140629319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef70104c70 0x7fef70107060 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.517+0000 7fef764c6640 1 --2- 172.21.15.40:0/3140629319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef70102340 0x7fef70104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.517+0000 7fef764c6640 1 -- 172.21.15.40:0/3140629319 >> 172.21.15.40:0/3140629319 conn(0x7fef700fbc90 msgr2=0x7fef700fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.517+0000 7fef764c6640 1 -- 172.21.15.40:0/3140629319 shutdown_connections 2024-03-18T22:47:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.517+0000 7fef764c6640 1 -- 172.21.15.40:0/3140629319 wait complete. 2024-03-18T22:47:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.518+0000 7fef764c6640 1 Processor -- start 2024-03-18T22:47:31.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.518+0000 7fef764c6640 1 -- start start 2024-03-18T22:47:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.518+0000 7fef764c6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef70102340 0x7fef70197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.518+0000 7fef764c6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef70104c70 0x7fef70197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.518+0000 7fef764c6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef701075a0 0x7fef7019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.518+0000 7fef764c6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fef7010d3a0 con 0x7fef70104c70 2024-03-18T22:47:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.518+0000 7fef764c6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fef7010d220 con 0x7fef701075a0 2024-03-18T22:47:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.518+0000 7fef764c6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fef7010d520 con 0x7fef70102340 2024-03-18T22:47:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.518+0000 7fef6f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef70104c70 0x7fef70197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.518+0000 7fef6f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef70104c70 0x7fef70197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55590/0 (socket says 172.21.15.40:55590) 2024-03-18T22:47:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.518+0000 7fef6f7fe640 1 -- 172.21.15.40:0/2553198149 learned_addr learned my addr 172.21.15.40:0/2553198149 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.518+0000 7fef6ffff640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef70102340 0x7fef70197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.519+0000 7fef74a63640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef701075a0 0x7fef7019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.519+0000 7fef6f7fe640 1 -- 172.21.15.40:0/2553198149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef70102340 msgr2=0x7fef70197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.519+0000 7fef6f7fe640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef70102340 0x7fef70197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.519+0000 7fef6f7fe640 1 -- 172.21.15.40:0/2553198149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef701075a0 msgr2=0x7fef7019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.519+0000 7fef6f7fe640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef701075a0 0x7fef7019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.519+0000 7fef6f7fe640 1 -- 172.21.15.40:0/2553198149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef7019f6a0 con 0x7fef70104c70 2024-03-18T22:47:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.519+0000 7fef6ffff640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef70102340 0x7fef70197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.519+0000 7fef74a63640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef701075a0 0x7fef7019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.519+0000 7fef6f7fe640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef70104c70 0x7fef70197fc0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fef64000c00 tx=0x7fef64005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.519+0000 7fef6d7fa640 1 -- 172.21.15.40:0/2553198149 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef64043020 con 0x7fef70104c70 2024-03-18T22:47:31.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.519+0000 7fef764c6640 1 -- 172.21.15.40:0/2553198149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef70100130 con 0x7fef70104c70 2024-03-18T22:47:31.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.519+0000 7fef6d7fa640 1 -- 172.21.15.40:0/2553198149 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef640043d0 con 0x7fef70104c70 2024-03-18T22:47:31.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.519+0000 7fef6d7fa640 1 -- 172.21.15.40:0/2553198149 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef64003bb0 con 0x7fef70104c70 2024-03-18T22:47:31.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.519+0000 7fef764c6640 1 -- 172.21.15.40:0/2553198149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef70100680 con 0x7fef70104c70 2024-03-18T22:47:31.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.522+0000 7fef6d7fa640 1 -- 172.21.15.40:0/2553198149 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fef64046020 con 0x7fef70104c70 2024-03-18T22:47:31.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.522+0000 7fef79191740 1 Processor -- start 2024-03-18T22:47:31.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.522+0000 7fef79191740 1 -- start start 2024-03-18T22:47:31.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.522+0000 7fef79191740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd446902f0 0x55dd44690710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.522+0000 7fef79191740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd44690ce0 0x55dd446957e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.522+0000 7fef79191740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44695e50 0x55dd446982e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.522+0000 7fef79191740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dd445f44b0 con 0x55dd44695e50 2024-03-18T22:47:31.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.522+0000 7fef79191740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dd4459cd70 con 0x55dd446902f0 2024-03-18T22:47:31.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.522+0000 7fef79191740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55dd445f0550 con 0x55dd44690ce0 2024-03-18T22:47:31.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.523+0000 7fef74a63640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd446902f0 0x55dd44690710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.523+0000 7fef6d7fa640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fef580a2800 0x7fef580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.523+0000 7fef6f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44695e50 0x55dd446982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.523+0000 7fef74a63640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd446902f0 0x55dd44690710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56838/0 (socket says 172.21.15.40:56838) 2024-03-18T22:47:31.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.523+0000 7fef74a63640 1 -- 172.21.15.40:0/3831720232 learned_addr learned my addr 172.21.15.40:0/3831720232 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.523+0000 7fef6ffff640 1 --2- 172.21.15.40:0/3831720232 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd44690ce0 0x55dd446957e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.523+0000 7fef74a63640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fef580a2800 0x7fef580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.523+0000 7fef6d7fa640 1 -- 172.21.15.40:0/2553198149 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fef640f0f70 con 0x7fef70104c70 2024-03-18T22:47:31.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.523+0000 7fef74a63640 1 -- 172.21.15.40:0/3831720232 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd44690ce0 msgr2=0x55dd446957e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.523+0000 7fef74a63640 1 --2- 172.21.15.40:0/3831720232 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd44690ce0 0x55dd446957e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.523+0000 7fef74a63640 1 -- 172.21.15.40:0/3831720232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44695e50 msgr2=0x55dd446982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.523+0000 7fef74a63640 1 --2- 172.21.15.40:0/3831720232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44695e50 0x55dd446982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.523+0000 7fef74a63640 1 -- 172.21.15.40:0/3831720232 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd445bf210 con 0x55dd446902f0 2024-03-18T22:47:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.523+0000 7fef74a63640 1 --2- 172.21.15.40:0/3831720232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd446902f0 0x55dd44690710 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fef68009fd0 tx=0x7fef6800b750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c063808207890169 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.523+0000 7fef74a63640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fef580a2800 0x7fef580a4cc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fef68020000 tx=0x7fef68022000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.524+0000 7fef567fc640 1 -- 172.21.15.40:0/3831720232 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef68023040 con 0x55dd446902f0 2024-03-18T22:47:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.524+0000 7fef567fc640 1 -- 172.21.15.40:0/3831720232 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef68010b20 con 0x55dd446902f0 2024-03-18T22:47:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.524+0000 7fef567fc640 1 -- 172.21.15.40:0/3831720232 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6801fcf0 con 0x55dd446902f0 2024-03-18T22:47:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.524+0000 7fef79191740 1 -- 172.21.15.40:0/3831720232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd446902f0 msgr2=0x55dd44690710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.524+0000 7fef79191740 1 --2- 172.21.15.40:0/3831720232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd446902f0 0x55dd44690710 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fef68009fd0 tx=0x7fef6800b750 comp rx=0 tx=0).stop 2024-03-18T22:47:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.524+0000 7fef79191740 1 -- 172.21.15.40:0/3831720232 shutdown_connections 2024-03-18T22:47:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.524+0000 7fef79191740 1 --2- 172.21.15.40:0/3831720232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44695e50 0x55dd446982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.524+0000 7fef79191740 1 --2- 172.21.15.40:0/3831720232 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd44690ce0 0x55dd446957e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.524+0000 7fef79191740 1 --2- 172.21.15.40:0/3831720232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd446902f0 0x55dd44690710 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.524+0000 7fef79191740 1 -- 172.21.15.40:0/3831720232 >> 172.21.15.40:0/3831720232 conn(0x55dd4468b860 msgr2=0x55dd4468bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.524+0000 7fef79191740 1 -- 172.21.15.40:0/3831720232 shutdown_connections 2024-03-18T22:47:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.524+0000 7fef79191740 1 -- 172.21.15.40:0/3831720232 wait complete. 2024-03-18T22:47:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.525+0000 7fef79191740 1 -- start start 2024-03-18T22:47:31.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.525+0000 7fef79191740 1 Processor -- start 2024-03-18T22:47:31.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.525+0000 7fef79191740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd446902f0 0x55dd4469a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.525+0000 7fef79191740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44690ce0 0x55dd4469ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.525+0000 7fef79191740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44695e50 0x55dd446a1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.525+0000 7fef79191740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dd44698da0 con 0x55dd44690ce0 2024-03-18T22:47:31.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.525+0000 7fef79191740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dd445b0310 con 0x55dd44695e50 2024-03-18T22:47:31.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.525+0000 7fef79191740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55dd44698f20 con 0x55dd446902f0 2024-03-18T22:47:31.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.525+0000 7fef79191740 10 client.0 _ll_register_callbacks cb 0x55dd445a42d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:31.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.525+0000 7fef6f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44690ce0 0x55dd4469ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.525+0000 7fef6f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44690ce0 0x55dd4469ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55602/0 (socket says 172.21.15.40:55602) 2024-03-18T22:47:31.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.525+0000 7fef6f7fe640 1 -- 172.21.15.40:0/1457103129 learned_addr learned my addr 172.21.15.40:0/1457103129 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.525+0000 7fef6ffff640 1 --2- 172.21.15.40:0/1457103129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd446902f0 0x55dd4469a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.525+0000 7fef74a63640 1 --2- 172.21.15.40:0/1457103129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44695e50 0x55dd446a1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.526+0000 7fef6f7fe640 1 -- 172.21.15.40:0/1457103129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd446902f0 msgr2=0x55dd4469a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.526+0000 7fef6f7fe640 1 --2- 172.21.15.40:0/1457103129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd446902f0 0x55dd4469a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.526+0000 7fef6f7fe640 1 -- 172.21.15.40:0/1457103129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44695e50 msgr2=0x55dd446a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.526+0000 7fef6f7fe640 1 --2- 172.21.15.40:0/1457103129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44695e50 0x55dd446a1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.526+0000 7fef6f7fe640 1 -- 172.21.15.40:0/1457103129 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd443083f0 con 0x55dd44690ce0 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.526+0000 7fef6f7fe640 1 --2- 172.21.15.40:0/1457103129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44690ce0 0x55dd4469ad50 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fef640ee680 tx=0x7fef64009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c0faec102f7a200 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.526+0000 7fef4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.526+0000 7fef4f7fe640 1 -- 172.21.15.40:0/1457103129 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef640bdc20 con 0x55dd44690ce0 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.526+0000 7fef4f7fe640 1 -- 172.21.15.40:0/1457103129 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fef64003d50 con 0x55dd44690ce0 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.526+0000 7fef79191740 1 -- 172.21.15.40:0/1457103129 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef68010b20 con 0x55dd44690ce0 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.526+0000 7fef4f7fe640 1 -- 172.21.15.40:0/1457103129 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fef640bd520 con 0x55dd44690ce0 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.526+0000 7fef79191740 10 client.16776 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.526+0000 7fef79191740 1 -- 172.21.15.40:0/1457103129 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dd443d27d0 con 0x55dd44690ce0 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.526+0000 7fef4f7fe640 1 -- 172.21.15.40:0/1457103129 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef68010b20 con 0x55dd44690ce0 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.527+0000 7fef4f7fe640 1 -- 172.21.15.40:0/1457103129 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fef640d6200 con 0x55dd44690ce0 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.527+0000 7fef79191740 10 client.16776 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:31.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.527+0000 7fef79191740 10 client.16776 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:47:31.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.527+0000 7fef79191740 20 client.16776 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:31.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.527+0000 7fef79191740 1 --2- 172.21.15.40:0/1457103129 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dd446a65e0 0x55dd446a8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.527+0000 7fef79191740 4 client.16776 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:47:31.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.527+0000 7fef79191740 1 -- 172.21.15.40:0/1457103129 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd446523e0 con 0x55dd446a65e0 2024-03-18T22:47:31.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.527+0000 7fef6ffff640 1 --2- 172.21.15.40:0/1457103129 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dd446a65e0 0x55dd446a8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.527+0000 7fef4f7fe640 10 client.16776 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:47:31.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.527+0000 7fef6ffff640 1 --2- 172.21.15.40:0/1457103129 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dd446a65e0 0x55dd446a8a60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=84815fb98163eb95 server_cookie=a9288c30a579ddd in_seq=0 out_seq=0 2024-03-18T22:47:31.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.528+0000 7fef4f7fe640 1 -- 172.21.15.40:0/1457103129 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dd446523e0 con 0x55dd446a65e0 2024-03-18T22:47:31.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.528+0000 7fef4f7fe640 10 client.16776 handle_command_reply: tid=0 2024-03-18T22:47:31.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.528+0000 7fef79191740 1 client.16776 shutdown 2024-03-18T22:47:31.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.528+0000 7fef79191740 1 -- 172.21.15.40:0/1457103129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44690ce0 msgr2=0x55dd4469ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.528+0000 7fef79191740 1 --2- 172.21.15.40:0/1457103129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44690ce0 0x55dd4469ad50 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fef640ee680 tx=0x7fef64009260 comp rx=0 tx=0).stop 2024-03-18T22:47:31.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.529+0000 7fef79191740 1 -- 172.21.15.40:0/1457103129 shutdown_connections 2024-03-18T22:47:31.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.529+0000 7fef79191740 1 --2- 172.21.15.40:0/1457103129 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dd446a65e0 0x55dd446a8a60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.529+0000 7fef79191740 1 --2- 172.21.15.40:0/1457103129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44695e50 0x55dd446a1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.529+0000 7fef79191740 1 --2- 172.21.15.40:0/1457103129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44690ce0 0x55dd4469ad50 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.529+0000 7fef79191740 1 --2- 172.21.15.40:0/1457103129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd446902f0 0x55dd4469a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.529+0000 7fef79191740 1 -- 172.21.15.40:0/1457103129 >> 172.21.15.40:0/1457103129 conn(0x55dd4468b860 msgr2=0x55dd4468aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.529+0000 7fef4f7fe640 0 client.16776 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:47:31.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.529+0000 7fef79191740 1 -- 172.21.15.40:0/1457103129 shutdown_connections 2024-03-18T22:47:31.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.529+0000 7fef79191740 1 -- 172.21.15.40:0/1457103129 wait complete. 2024-03-18T22:47:31.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.529+0000 7fef79191740 20 client.16776 trim_cache size 0 max 16384 2024-03-18T22:47:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.543+0000 7fef79191740 1 Processor -- start 2024-03-18T22:47:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.543+0000 7fef79191740 1 -- start start 2024-03-18T22:47:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.543+0000 7fef79191740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd446902f0 0x55dd4468a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.543+0000 7fef79191740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44690ce0 0x55dd446a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.543+0000 7fef79191740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44695e50 0x55dd4468a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.543+0000 7fef79191740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fef701a8360 con 0x55dd44690ce0 2024-03-18T22:47:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.543+0000 7fef79191740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fef701a9ea0 con 0x55dd44695e50 2024-03-18T22:47:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.543+0000 7fef79191740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fef701a8a30 con 0x55dd446902f0 2024-03-18T22:47:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.543+0000 7fef6f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44690ce0 0x55dd446a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.543+0000 7fef6f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44690ce0 0x55dd446a1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55610/0 (socket says 172.21.15.40:55610) 2024-03-18T22:47:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.543+0000 7fef6f7fe640 1 -- 172.21.15.40:0/2035336727 learned_addr learned my addr 172.21.15.40:0/2035336727 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.543+0000 7fef6ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd446902f0 0x55dd4468a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.543+0000 7fef74a63640 1 --2- 172.21.15.40:0/2035336727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44695e50 0x55dd4468a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.544+0000 7fef6ffff640 1 -- 172.21.15.40:0/2035336727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44695e50 msgr2=0x55dd4468a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.544+0000 7fef6ffff640 1 --2- 172.21.15.40:0/2035336727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44695e50 0x55dd4468a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.544+0000 7fef6ffff640 1 -- 172.21.15.40:0/2035336727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44690ce0 msgr2=0x55dd446a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.544+0000 7fef6ffff640 1 --2- 172.21.15.40:0/2035336727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44690ce0 0x55dd446a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.544+0000 7fef6ffff640 1 -- 172.21.15.40:0/2035336727 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd44661880 con 0x55dd446902f0 2024-03-18T22:47:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.544+0000 7fef6f7fe640 1 --2- 172.21.15.40:0/2035336727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44690ce0 0x55dd446a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.544+0000 7fef6ffff640 1 --2- 172.21.15.40:0/2035336727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd446902f0 0x55dd4468a310 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x55dd446a9310 tx=0x7fef60005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80bfa8a2cab43614 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.544+0000 7fef55ffb640 1 -- 172.21.15.40:0/2035336727 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60012110 con 0x55dd446902f0 2024-03-18T22:47:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.544+0000 7fef55ffb640 1 -- 172.21.15.40:0/2035336727 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef6000f870 con 0x55dd446902f0 2024-03-18T22:47:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.544+0000 7fef55ffb640 1 -- 172.21.15.40:0/2035336727 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60020370 con 0x55dd446902f0 2024-03-18T22:47:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.545+0000 7fef79191740 1 -- 172.21.15.40:0/2035336727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd446902f0 msgr2=0x55dd4468a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.545+0000 7fef79191740 1 --2- 172.21.15.40:0/2035336727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd446902f0 0x55dd4468a310 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x55dd446a9310 tx=0x7fef60005f90 comp rx=0 tx=0).stop 2024-03-18T22:47:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.545+0000 7fef79191740 1 -- 172.21.15.40:0/2035336727 shutdown_connections 2024-03-18T22:47:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.545+0000 7fef79191740 1 --2- 172.21.15.40:0/2035336727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44695e50 0x55dd4468a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.545+0000 7fef79191740 1 --2- 172.21.15.40:0/2035336727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd44690ce0 0x55dd446a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.545+0000 7fef79191740 1 --2- 172.21.15.40:0/2035336727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd446902f0 0x55dd4468a310 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.545+0000 7fef79191740 1 -- 172.21.15.40:0/2035336727 >> 172.21.15.40:0/2035336727 conn(0x55dd4468b860 msgr2=0x55dd4469a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.545+0000 7fef79191740 1 -- 172.21.15.40:0/2035336727 shutdown_connections 2024-03-18T22:47:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.545+0000 7fef79191740 1 -- 172.21.15.40:0/2035336727 wait complete. 2024-03-18T22:47:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.545+0000 7fef79191740 1 -- start start 2024-03-18T22:47:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.545+0000 7fef79191740 1 Processor -- start 2024-03-18T22:47:31.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef79191740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd446902f0 0x55dd4469f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef79191740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44690ce0 0x55dd4469f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef79191740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd44695e50 0x55dd446a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef79191740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fef701a97d0 con 0x55dd446902f0 2024-03-18T22:47:31.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef79191740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fef701a9100 con 0x55dd44690ce0 2024-03-18T22:47:31.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef79191740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fef701a6340 con 0x55dd44695e50 2024-03-18T22:47:31.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef6ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd446902f0 0x55dd4469f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef79191740 10 client.0 _ll_register_callbacks cb 0x7fef64005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:31.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef6f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44690ce0 0x55dd4469f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef6ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd446902f0 0x55dd4469f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55620/0 (socket says 172.21.15.40:55620) 2024-03-18T22:47:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef6ffff640 1 -- 172.21.15.40:0/2391015646 learned_addr learned my addr 172.21.15.40:0/2391015646 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef74a63640 1 --2- 172.21.15.40:0/2391015646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd44695e50 0x55dd446a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef6ffff640 1 -- 172.21.15.40:0/2391015646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd44695e50 msgr2=0x55dd446a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef6ffff640 1 --2- 172.21.15.40:0/2391015646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd44695e50 0x55dd446a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef6ffff640 1 -- 172.21.15.40:0/2391015646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44690ce0 msgr2=0x55dd4469f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef6ffff640 1 --2- 172.21.15.40:0/2391015646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44690ce0 0x55dd4469f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef6ffff640 1 -- 172.21.15.40:0/2391015646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd4468ff30 con 0x55dd446902f0 2024-03-18T22:47:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef6f7fe640 1 --2- 172.21.15.40:0/2391015646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44690ce0 0x55dd4469f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.546+0000 7fef74a63640 1 --2- 172.21.15.40:0/2391015646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd44695e50 0x55dd446a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.547+0000 7fef6ffff640 1 --2- 172.21.15.40:0/2391015646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd446902f0 0x55dd4469f490 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fef60009460 tx=0x7fef60012200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36def9fbf6627c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.547+0000 7fef4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:47:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.547+0000 7fef4f7fe640 1 -- 172.21.15.40:0/2391015646 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60025040 con 0x55dd446902f0 2024-03-18T22:47:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.547+0000 7fef79191740 1 -- 172.21.15.40:0/2391015646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef6000f870 con 0x55dd446902f0 2024-03-18T22:47:31.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.547+0000 7fef4f7fe640 1 -- 172.21.15.40:0/2391015646 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fef60004250 con 0x55dd446902f0 2024-03-18T22:47:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.547+0000 7fef4f7fe640 1 -- 172.21.15.40:0/2391015646 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fef60037050 con 0x55dd446902f0 2024-03-18T22:47:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.547+0000 7fef79191740 10 client.16782 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.547+0000 7fef79191740 1 -- 172.21.15.40:0/2391015646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dd4470c9a0 con 0x55dd446902f0 2024-03-18T22:47:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.547+0000 7fef4f7fe640 1 -- 172.21.15.40:0/2391015646 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef6000f870 con 0x55dd446902f0 2024-03-18T22:47:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.547+0000 7fef4f7fe640 1 -- 172.21.15.40:0/2391015646 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fef60028070 con 0x55dd446902f0 2024-03-18T22:47:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.547+0000 7fef79191740 10 client.16782 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.547+0000 7fef79191740 10 client.16782 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:47:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef79191740 20 client.16782 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef79191740 1 --2- 172.21.15.40:0/2391015646 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dd4470d210 0x55dd4470f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef79191740 4 client.16782 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:47:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef79191740 1 -- 172.21.15.40:0/2391015646 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dd446019f0 con 0x55dd4470d210 2024-03-18T22:47:31.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef6f7fe640 1 --2- 172.21.15.40:0/2391015646 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dd4470d210 0x55dd4470f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef6f7fe640 1 --2- 172.21.15.40:0/2391015646 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dd4470d210 0x55dd4470f630 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d02e2b37cd798c38 server_cookie=bf3aa7a9cac3467c in_seq=0 out_seq=0 2024-03-18T22:47:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef4f7fe640 10 client.16782 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:47:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef4f7fe640 1 -- 172.21.15.40:0/2391015646 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dd446019f0 con 0x55dd4470d210 2024-03-18T22:47:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef4f7fe640 10 client.16782 handle_command_reply: tid=0 2024-03-18T22:47:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef79191740 1 client.16782 shutdown 2024-03-18T22:47:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef79191740 1 -- 172.21.15.40:0/2391015646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd446902f0 msgr2=0x55dd4469f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef79191740 1 --2- 172.21.15.40:0/2391015646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd446902f0 0x55dd4469f490 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fef60009460 tx=0x7fef60012200 comp rx=0 tx=0).stop 2024-03-18T22:47:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef79191740 1 -- 172.21.15.40:0/2391015646 shutdown_connections 2024-03-18T22:47:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef79191740 1 --2- 172.21.15.40:0/2391015646 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dd4470d210 0x55dd4470f630 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef79191740 1 --2- 172.21.15.40:0/2391015646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dd44695e50 0x55dd446a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef79191740 1 --2- 172.21.15.40:0/2391015646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd44690ce0 0x55dd4469f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef79191740 1 --2- 172.21.15.40:0/2391015646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dd446902f0 0x55dd4469f490 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef79191740 1 -- 172.21.15.40:0/2391015646 >> 172.21.15.40:0/2391015646 conn(0x55dd4468b860 msgr2=0x55dd4468eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.548+0000 7fef4f7fe640 0 client.16782 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:47:31.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.549+0000 7fef79191740 1 -- 172.21.15.40:0/2391015646 shutdown_connections 2024-03-18T22:47:31.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.549+0000 7fef79191740 1 -- 172.21.15.40:0/2391015646 wait complete. 2024-03-18T22:47:31.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.549+0000 7fef79191740 20 client.16782 trim_cache size 0 max 16384 2024-03-18T22:47:31.556 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:47:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.549+0000 7fef764c6640 1 -- 172.21.15.40:0/2553198149 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fef580a2800 msgr2=0x7fef580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.549+0000 7fef764c6640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fef580a2800 0x7fef580a4cc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fef68020000 tx=0x7fef68022000 comp rx=0 tx=0).stop 2024-03-18T22:47:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.549+0000 7fef764c6640 1 -- 172.21.15.40:0/2553198149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef70104c70 msgr2=0x7fef70197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.549+0000 7fef764c6640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef70104c70 0x7fef70197fc0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fef64000c00 tx=0x7fef64005e60 comp rx=0 tx=0).stop 2024-03-18T22:47:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.549+0000 7fef764c6640 1 -- 172.21.15.40:0/2553198149 shutdown_connections 2024-03-18T22:47:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.549+0000 7fef764c6640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fef580a2800 0x7fef580a4cc0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.549+0000 7fef764c6640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef701075a0 0x7fef7019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.549+0000 7fef764c6640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef70104c70 0x7fef70197fc0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.549+0000 7fef764c6640 1 --2- 172.21.15.40:0/2553198149 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fef70102340 0x7fef70197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.549+0000 7fef764c6640 1 -- 172.21.15.40:0/2553198149 >> 172.21.15.40:0/2553198149 conn(0x7fef700fbc90 msgr2=0x7fef70102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.549+0000 7fef764c6640 1 -- 172.21.15.40:0/2553198149 shutdown_connections 2024-03-18T22:47:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.549+0000 7fef764c6640 1 -- 172.21.15.40:0/2553198149 wait complete. 2024-03-18T22:47:31.561 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:47:31.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:31 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2506967098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:47:31.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:31 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2506967098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:47:31.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.671+0000 7fe48995b640 1 -- 172.21.15.40:0/1421073347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4841033d0 msgr2=0x7fe484109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.671+0000 7fe48995b640 1 --2- 172.21.15.40:0/1421073347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4841033d0 0x7fe484109c60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009ca0 tx=0x7fe47c02c0f0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.671+0000 7fe48995b640 1 -- 172.21.15.40:0/1421073347 shutdown_connections 2024-03-18T22:47:31.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.671+0000 7fe48995b640 1 --2- 172.21.15.40:0/1421073347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4841033d0 0x7fe484109c60 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.671+0000 7fe48995b640 1 --2- 172.21.15.40:0/1421073347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe484102a10 0x7fe484102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.671+0000 7fe48995b640 1 --2- 172.21.15.40:0/1421073347 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe484101820 0x7fe484101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.671+0000 7fe48995b640 1 -- 172.21.15.40:0/1421073347 >> 172.21.15.40:0/1421073347 conn(0x7fe4840fcd90 msgr2=0x7fe4840ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.671+0000 7fe48995b640 1 -- 172.21.15.40:0/1421073347 shutdown_connections 2024-03-18T22:47:31.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.671+0000 7fe48995b640 1 -- 172.21.15.40:0/1421073347 wait complete. 2024-03-18T22:47:31.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.671+0000 7fe48995b640 1 Processor -- start 2024-03-18T22:47:31.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe48995b640 1 -- start start 2024-03-18T22:47:31.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe48995b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe484101820 0x7fe4840755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe48995b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe484102a10 0x7fe484073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe48995b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4841033d0 0x7fe484074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe48995b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe48410d330 con 0x7fe4841033d0 2024-03-18T22:47:31.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe48995b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe48410d1b0 con 0x7fe484101820 2024-03-18T22:47:31.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe48995b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe48410d4b0 con 0x7fe484102a10 2024-03-18T22:47:31.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe4837fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4841033d0 0x7fe484074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe4837fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4841033d0 0x7fe484074150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55634/0 (socket says 172.21.15.40:55634) 2024-03-18T22:47:31.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe4827fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe484102a10 0x7fe484073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe4837fe640 1 -- 172.21.15.40:0/3675415281 learned_addr learned my addr 172.21.15.40:0/3675415281 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe482ffd640 1 --2- 172.21.15.40:0/3675415281 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe484101820 0x7fe4840755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe4837fe640 1 -- 172.21.15.40:0/3675415281 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe484102a10 msgr2=0x7fe484073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe4837fe640 1 --2- 172.21.15.40:0/3675415281 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe484102a10 0x7fe484073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe4837fe640 1 -- 172.21.15.40:0/3675415281 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe484101820 msgr2=0x7fe4840755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe4837fe640 1 --2- 172.21.15.40:0/3675415281 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe484101820 0x7fe4840755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe4837fe640 1 -- 172.21.15.40:0/3675415281 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4840749e0 con 0x7fe4841033d0 2024-03-18T22:47:31.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe4837fe640 1 --2- 172.21.15.40:0/3675415281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4841033d0 0x7fe484074150 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fe47c02c700 tx=0x7fe47c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.672+0000 7fe488959640 1 -- 172.21.15.40:0/3675415281 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c041020 con 0x7fe4841033d0 2024-03-18T22:47:31.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.673+0000 7fe488959640 1 -- 172.21.15.40:0/3675415281 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe47c003e20 con 0x7fe4841033d0 2024-03-18T22:47:31.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.673+0000 7fe48995b640 1 -- 172.21.15.40:0/3675415281 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4841a1a90 con 0x7fe4841033d0 2024-03-18T22:47:31.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.673+0000 7fe48995b640 1 -- 172.21.15.40:0/3675415281 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4841a1fe0 con 0x7fe4841033d0 2024-03-18T22:47:31.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.673+0000 7fe488959640 1 -- 172.21.15.40:0/3675415281 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c039040 con 0x7fe4841033d0 2024-03-18T22:47:31.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.675+0000 7fe488959640 1 -- 172.21.15.40:0/3675415281 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe47c044020 con 0x7fe4841033d0 2024-03-18T22:47:31.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.675+0000 7fe48c623740 1 Processor -- start 2024-03-18T22:47:31.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.675+0000 7fe48c623740 1 -- start start 2024-03-18T22:47:31.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.675+0000 7fe48c623740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd0012f0 0x5622fd001710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.675+0000 7fe48c623740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd001ce0 0x5622fd0067e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.675+0000 7fe48c623740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 0x5622fd0092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.675+0000 7fe48c623740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5622fcf654b0 con 0x5622fd0012f0 2024-03-18T22:47:31.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.675+0000 7fe48c623740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5622fcf0dd70 con 0x5622fd001ce0 2024-03-18T22:47:31.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.675+0000 7fe48c623740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5622fcf61550 con 0x5622fd006e50 2024-03-18T22:47:31.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.675+0000 7fe488959640 1 --2- 172.21.15.40:0/3675415281 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe46c0a2800 0x7fe46c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe482ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd001ce0 0x5622fd0067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe4827fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd0012f0 0x5622fd001710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe482ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd001ce0 0x5622fd0067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56892/0 (socket says 172.21.15.40:56892) 2024-03-18T22:47:31.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe482ffd640 1 -- 172.21.15.40:0/3518645302 learned_addr learned my addr 172.21.15.40:0/3518645302 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe4837fe640 1 --2- 172.21.15.40:0/3675415281 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe46c0a2800 0x7fe46c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe4827fc640 1 --2- 172.21.15.40:0/3518645302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 0x5622fd0092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe488959640 1 -- 172.21.15.40:0/3675415281 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe47c0f0fb0 con 0x7fe4841033d0 2024-03-18T22:47:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe482ffd640 1 -- 172.21.15.40:0/3518645302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 msgr2=0x5622fd0092e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe482ffd640 1 --2- 172.21.15.40:0/3518645302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 0x5622fd0092e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe482ffd640 1 -- 172.21.15.40:0/3518645302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd0012f0 msgr2=0x5622fd001710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe482ffd640 1 --2- 172.21.15.40:0/3518645302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd0012f0 0x5622fd001710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe482ffd640 1 -- 172.21.15.40:0/3518645302 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622fcf30210 con 0x5622fd001ce0 2024-03-18T22:47:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe4827fc640 1 --2- 172.21.15.40:0/3518645302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 0x5622fd0092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:47:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe482ffd640 1 --2- 172.21.15.40:0/3518645302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd001ce0 0x5622fd0067e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe474009ca0 tx=0x7fe474011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f38666422b312102 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe4837fe640 1 --2- 172.21.15.40:0/3675415281 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe46c0a2800 0x7fe46c0a4cc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0bd4f0 tx=0x7fe47c0a8040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe469ffb640 1 -- 172.21.15.40:0/3518645302 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe474004220 con 0x5622fd001ce0 2024-03-18T22:47:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe469ffb640 1 -- 172.21.15.40:0/3518645302 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4740043c0 con 0x5622fd001ce0 2024-03-18T22:47:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.676+0000 7fe469ffb640 1 -- 172.21.15.40:0/3518645302 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4740198b0 con 0x5622fd001ce0 2024-03-18T22:47:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.677+0000 7fe48c623740 1 -- 172.21.15.40:0/3518645302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd001ce0 msgr2=0x5622fd0067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.677+0000 7fe48c623740 1 --2- 172.21.15.40:0/3518645302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd001ce0 0x5622fd0067e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe474009ca0 tx=0x7fe474011810 comp rx=0 tx=0).stop 2024-03-18T22:47:31.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.677+0000 7fe48c623740 1 -- 172.21.15.40:0/3518645302 shutdown_connections 2024-03-18T22:47:31.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.677+0000 7fe48c623740 1 --2- 172.21.15.40:0/3518645302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 0x5622fd0092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.677+0000 7fe48c623740 1 --2- 172.21.15.40:0/3518645302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd001ce0 0x5622fd0067e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.677+0000 7fe48c623740 1 --2- 172.21.15.40:0/3518645302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd0012f0 0x5622fd001710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.677+0000 7fe48c623740 1 -- 172.21.15.40:0/3518645302 >> 172.21.15.40:0/3518645302 conn(0x5622fcffc860 msgr2=0x5622fcffcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.677+0000 7fe48c623740 1 -- 172.21.15.40:0/3518645302 shutdown_connections 2024-03-18T22:47:31.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.677+0000 7fe48c623740 1 -- 172.21.15.40:0/3518645302 wait complete. 2024-03-18T22:47:31.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.678+0000 7fe48c623740 1 -- start start 2024-03-18T22:47:31.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.678+0000 7fe48c623740 1 Processor -- start 2024-03-18T22:47:31.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.678+0000 7fe48c623740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd0012f0 0x5622fd00b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.678+0000 7fe48c623740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd001ce0 0x5622fd00bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.678+0000 7fe48c623740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 0x5622fd012d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.678+0000 7fe48c623740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5622fd009e00 con 0x5622fd001ce0 2024-03-18T22:47:31.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.678+0000 7fe48c623740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5622fcf21310 con 0x5622fd0012f0 2024-03-18T22:47:31.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.678+0000 7fe48c623740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5622fd009f80 con 0x5622fd006e50 2024-03-18T22:47:31.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.678+0000 7fe48c623740 10 client.0 _ll_register_callbacks cb 0x5622fcf152d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:31.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.678+0000 7fe482ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd001ce0 0x5622fd00bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.678+0000 7fe482ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd001ce0 0x5622fd00bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55644/0 (socket says 172.21.15.40:55644) 2024-03-18T22:47:31.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.678+0000 7fe4827fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd0012f0 0x5622fd00b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.678+0000 7fe482ffd640 1 -- 172.21.15.40:0/183242006 learned_addr learned my addr 172.21.15.40:0/183242006 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.678+0000 7fe4827fc640 1 --2- 172.21.15.40:0/183242006 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 0x5622fd012d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.679+0000 7fe482ffd640 1 -- 172.21.15.40:0/183242006 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 msgr2=0x5622fd012d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.679+0000 7fe482ffd640 1 --2- 172.21.15.40:0/183242006 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 0x5622fd012d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.679+0000 7fe482ffd640 1 -- 172.21.15.40:0/183242006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd0012f0 msgr2=0x5622fd00b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.679+0000 7fe482ffd640 1 --2- 172.21.15.40:0/183242006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd0012f0 0x5622fd00b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.679+0000 7fe482ffd640 1 -- 172.21.15.40:0/183242006 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622fcc793f0 con 0x5622fd001ce0 2024-03-18T22:47:31.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.679+0000 7fe4827fc640 1 --2- 172.21.15.40:0/183242006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd0012f0 0x5622fd00b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.679+0000 7fe482ffd640 1 --2- 172.21.15.40:0/183242006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd001ce0 0x5622fd00bd80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe474009380 tx=0x7fe4740038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61183bc37797bb34 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.679+0000 7fe462ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:47:31.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.679+0000 7fe462ffd640 1 -- 172.21.15.40:0/183242006 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe474028070 con 0x5622fd001ce0 2024-03-18T22:47:31.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.679+0000 7fe462ffd640 1 -- 172.21.15.40:0/183242006 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe474005040 con 0x5622fd001ce0 2024-03-18T22:47:31.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.679+0000 7fe48c623740 1 -- 172.21.15.40:0/183242006 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4740043c0 con 0x5622fd001ce0 2024-03-18T22:47:31.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.679+0000 7fe462ffd640 1 -- 172.21.15.40:0/183242006 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe474034050 con 0x5622fd001ce0 2024-03-18T22:47:31.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.679+0000 7fe48c623740 10 client.16794 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:31.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.679+0000 7fe48c623740 1 -- 172.21.15.40:0/183242006 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5622fcd437d0 con 0x5622fd001ce0 2024-03-18T22:47:31.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.679+0000 7fe462ffd640 1 -- 172.21.15.40:0/183242006 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4740043c0 con 0x5622fd001ce0 2024-03-18T22:47:31.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.680+0000 7fe462ffd640 1 -- 172.21.15.40:0/183242006 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe4740049f0 con 0x5622fd001ce0 2024-03-18T22:47:31.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.680+0000 7fe48c623740 10 client.16794 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:31.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.680+0000 7fe48c623740 10 client.16794 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:47:31.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.680+0000 7fe48c623740 20 client.16794 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:31.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.680+0000 7fe48c623740 1 --2- 172.21.15.40:0/183242006 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5622fd0173d0 0x5622fd019850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.680+0000 7fe48c623740 4 client.16794 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:47:31.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.680+0000 7fe48c623740 1 -- 172.21.15.40:0/183242006 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622fcfc33e0 con 0x5622fd0173d0 2024-03-18T22:47:31.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.680+0000 7fe4827fc640 1 --2- 172.21.15.40:0/183242006 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5622fd0173d0 0x5622fd019850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.680+0000 7fe4827fc640 1 --2- 172.21.15.40:0/183242006 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5622fd0173d0 0x5622fd019850 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3416953f1ce719f0 server_cookie=957e32b24a6e63fc in_seq=0 out_seq=0 2024-03-18T22:47:31.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.680+0000 7fe462ffd640 10 client.16794 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:47:31.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.681+0000 7fe462ffd640 1 -- 172.21.15.40:0/183242006 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5622fcfc33e0 con 0x5622fd0173d0 2024-03-18T22:47:31.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.681+0000 7fe462ffd640 10 client.16794 handle_command_reply: tid=0 2024-03-18T22:47:31.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.681+0000 7fe48c623740 1 client.16794 shutdown 2024-03-18T22:47:31.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.681+0000 7fe48c623740 1 -- 172.21.15.40:0/183242006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd001ce0 msgr2=0x5622fd00bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.681+0000 7fe48c623740 1 --2- 172.21.15.40:0/183242006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd001ce0 0x5622fd00bd80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe474009380 tx=0x7fe4740038a0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.681+0000 7fe48c623740 1 -- 172.21.15.40:0/183242006 shutdown_connections 2024-03-18T22:47:31.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.681+0000 7fe48c623740 1 --2- 172.21.15.40:0/183242006 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5622fd0173d0 0x5622fd019850 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.681+0000 7fe48c623740 1 --2- 172.21.15.40:0/183242006 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 0x5622fd012d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.681+0000 7fe48c623740 1 --2- 172.21.15.40:0/183242006 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd001ce0 0x5622fd00bd80 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.681+0000 7fe48c623740 1 --2- 172.21.15.40:0/183242006 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd0012f0 0x5622fd00b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.681+0000 7fe48c623740 1 -- 172.21.15.40:0/183242006 >> 172.21.15.40:0/183242006 conn(0x5622fcffc860 msgr2=0x5622fcffbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.681+0000 7fe462ffd640 0 client.16794 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:47:31.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.681+0000 7fe48c623740 1 -- 172.21.15.40:0/183242006 shutdown_connections 2024-03-18T22:47:31.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.681+0000 7fe48c623740 1 -- 172.21.15.40:0/183242006 wait complete. 2024-03-18T22:47:31.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.681+0000 7fe48c623740 20 client.16794 trim_cache size 0 max 16384 2024-03-18T22:47:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.697+0000 7fe48c623740 1 Processor -- start 2024-03-18T22:47:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.697+0000 7fe48c623740 1 -- start start 2024-03-18T22:47:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.697+0000 7fe48c623740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd0012f0 0x5622fd00b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.697+0000 7fe48c623740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd001ce0 0x5622fd00bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.697+0000 7fe48c623740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 0x5622fd012d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.697+0000 7fe48c623740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe4841ac720 con 0x5622fd001ce0 2024-03-18T22:47:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.697+0000 7fe48c623740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe4841ae260 con 0x5622fd0012f0 2024-03-18T22:47:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.697+0000 7fe48c623740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe4841acdf0 con 0x5622fd006e50 2024-03-18T22:47:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.697+0000 7fe482ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd001ce0 0x5622fd00bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.697+0000 7fe482ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd001ce0 0x5622fd00bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55650/0 (socket says 172.21.15.40:55650) 2024-03-18T22:47:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.698+0000 7fe482ffd640 1 -- 172.21.15.40:0/3064976696 learned_addr learned my addr 172.21.15.40:0/3064976696 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.698+0000 7fe4827fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd0012f0 0x5622fd00b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.698+0000 7fe4827fc640 1 --2- 172.21.15.40:0/3064976696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 0x5622fd012d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.698+0000 7fe482ffd640 1 -- 172.21.15.40:0/3064976696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 msgr2=0x5622fd012d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.698+0000 7fe482ffd640 1 --2- 172.21.15.40:0/3064976696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 0x5622fd012d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.698+0000 7fe482ffd640 1 -- 172.21.15.40:0/3064976696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd0012f0 msgr2=0x5622fd00b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.698+0000 7fe482ffd640 1 --2- 172.21.15.40:0/3064976696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd0012f0 0x5622fd00b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.698+0000 7fe482ffd640 1 -- 172.21.15.40:0/3064976696 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622fcfd2880 con 0x5622fd001ce0 2024-03-18T22:47:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.698+0000 7fe482ffd640 1 --2- 172.21.15.40:0/3064976696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd001ce0 0x5622fd00bd80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe474004e90 tx=0x7fe474003800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2bac4875c913c41 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.698+0000 7fe4697fa640 1 -- 172.21.15.40:0/3064976696 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe474028070 con 0x5622fd001ce0 2024-03-18T22:47:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.698+0000 7fe4697fa640 1 -- 172.21.15.40:0/3064976696 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe474019800 con 0x5622fd001ce0 2024-03-18T22:47:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.698+0000 7fe4697fa640 1 -- 172.21.15.40:0/3064976696 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4740052a0 con 0x5622fd001ce0 2024-03-18T22:47:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.699+0000 7fe48c623740 1 -- 172.21.15.40:0/3064976696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd001ce0 msgr2=0x5622fd00bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.699+0000 7fe48c623740 1 --2- 172.21.15.40:0/3064976696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd001ce0 0x5622fd00bd80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe474004e90 tx=0x7fe474003800 comp rx=0 tx=0).stop 2024-03-18T22:47:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.699+0000 7fe48c623740 1 -- 172.21.15.40:0/3064976696 shutdown_connections 2024-03-18T22:47:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.699+0000 7fe48c623740 1 --2- 172.21.15.40:0/3064976696 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd006e50 0x5622fd012d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.699+0000 7fe48c623740 1 --2- 172.21.15.40:0/3064976696 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd001ce0 0x5622fd00bd80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.699+0000 7fe48c623740 1 --2- 172.21.15.40:0/3064976696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd0012f0 0x5622fd00b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.699+0000 7fe48c623740 1 -- 172.21.15.40:0/3064976696 >> 172.21.15.40:0/3064976696 conn(0x5622fcffc860 msgr2=0x5622fd00a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.699+0000 7fe48c623740 1 -- 172.21.15.40:0/3064976696 shutdown_connections 2024-03-18T22:47:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.699+0000 7fe48c623740 1 -- 172.21.15.40:0/3064976696 wait complete. 2024-03-18T22:47:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.699+0000 7fe48c623740 1 -- start start 2024-03-18T22:47:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.699+0000 7fe48c623740 1 Processor -- start 2024-03-18T22:47:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe48c623740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd0012f0 0x5622fd00fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe48c623740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd001ce0 0x5622fd00ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe48c623740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd006e50 0x5622fd07b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe48c623740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe4841adb90 con 0x5622fd006e50 2024-03-18T22:47:31.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe48c623740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe4841ad4c0 con 0x5622fd001ce0 2024-03-18T22:47:31.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe48c623740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe4841aa700 con 0x5622fd0012f0 2024-03-18T22:47:31.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe48c623740 10 client.0 _ll_register_callbacks cb 0x7fe478009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:31.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe4827fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd0012f0 0x5622fd00fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe482ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd001ce0 0x5622fd00ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe4827fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd0012f0 0x5622fd00fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33404/0 (socket says 172.21.15.40:33404) 2024-03-18T22:47:31.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe4827fc640 1 -- 172.21.15.40:0/741515779 learned_addr learned my addr 172.21.15.40:0/741515779 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe4827fc640 1 --2- 172.21.15.40:0/741515779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd006e50 0x5622fd07b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe4827fc640 1 -- 172.21.15.40:0/741515779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd001ce0 msgr2=0x5622fd00ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe4827fc640 1 --2- 172.21.15.40:0/741515779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd001ce0 0x5622fd00ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe4827fc640 1 -- 172.21.15.40:0/741515779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd006e50 msgr2=0x5622fd07b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe4827fc640 1 --2- 172.21.15.40:0/741515779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd006e50 0x5622fd07b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe4827fc640 1 -- 172.21.15.40:0/741515779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622fd012f60 con 0x5622fd0012f0 2024-03-18T22:47:31.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe4827fc640 1 --2- 172.21.15.40:0/741515779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd0012f0 0x5622fd00fa10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe478000c00 tx=0x7fe478004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40b5e71849d75de9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.700+0000 7fe462ffd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:47:31.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.701+0000 7fe462ffd640 1 -- 172.21.15.40:0/741515779 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47801c040 con 0x5622fd0012f0 2024-03-18T22:47:31.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.701+0000 7fe462ffd640 1 -- 172.21.15.40:0/741515779 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe47800fa40 con 0x5622fd0012f0 2024-03-18T22:47:31.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.701+0000 7fe48c623740 1 -- 172.21.15.40:0/741515779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe474019800 con 0x5622fd0012f0 2024-03-18T22:47:31.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.701+0000 7fe462ffd640 1 -- 172.21.15.40:0/741515779 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe47800fc40 con 0x5622fd0012f0 2024-03-18T22:47:31.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.701+0000 7fe48c623740 10 client.26617 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:31.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.701+0000 7fe48c623740 1 -- 172.21.15.40:0/741515779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5622fd07cc10 con 0x5622fd0012f0 2024-03-18T22:47:31.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.701+0000 7fe462ffd640 1 -- 172.21.15.40:0/741515779 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe474019800 con 0x5622fd0012f0 2024-03-18T22:47:31.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.701+0000 7fe462ffd640 1 -- 172.21.15.40:0/741515779 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe478027ce0 con 0x5622fd0012f0 2024-03-18T22:47:31.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.701+0000 7fe48c623740 10 client.26617 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:31.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.701+0000 7fe48c623740 10 client.26617 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:47:31.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.701+0000 7fe48c623740 20 client.26617 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:31.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.701+0000 7fe48c623740 1 --2- 172.21.15.40:0/741515779 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5622fd07d640 0x5622fd07fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.701+0000 7fe48c623740 4 client.26617 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:47:31.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.701+0000 7fe48c623740 1 -- 172.21.15.40:0/741515779 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5622fcf729f0 con 0x5622fd07d640 2024-03-18T22:47:31.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.701+0000 7fe482ffd640 1 --2- 172.21.15.40:0/741515779 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5622fd07d640 0x5622fd07fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe462ffd640 10 client.26617 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:47:31.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe482ffd640 1 --2- 172.21.15.40:0/741515779 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5622fd07d640 0x5622fd07fa60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=83340df210313014 server_cookie=f31a6e934c3d3ca3 in_seq=0 out_seq=0 2024-03-18T22:47:31.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe462ffd640 1 -- 172.21.15.40:0/741515779 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5622fcf729f0 con 0x5622fd07d640 2024-03-18T22:47:31.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe462ffd640 10 client.26617 handle_command_reply: tid=0 2024-03-18T22:47:31.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe48c623740 1 client.26617 shutdown 2024-03-18T22:47:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe48c623740 1 -- 172.21.15.40:0/741515779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd0012f0 msgr2=0x5622fd00fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe48c623740 1 --2- 172.21.15.40:0/741515779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd0012f0 0x5622fd00fa10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe478000c00 tx=0x7fe478004730 comp rx=0 tx=0).stop 2024-03-18T22:47:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe48c623740 1 -- 172.21.15.40:0/741515779 shutdown_connections 2024-03-18T22:47:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe48c623740 1 --2- 172.21.15.40:0/741515779 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5622fd07d640 0x5622fd07fa60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe48c623740 1 --2- 172.21.15.40:0/741515779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5622fd006e50 0x5622fd07b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe48c623740 1 --2- 172.21.15.40:0/741515779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5622fd001ce0 0x5622fd00ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe462ffd640 0 client.26617 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:47:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe48c623740 1 --2- 172.21.15.40:0/741515779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5622fd0012f0 0x5622fd00fa10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe48c623740 1 -- 172.21.15.40:0/741515779 >> 172.21.15.40:0/741515779 conn(0x5622fcffc860 msgr2=0x5622fd00a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe48c623740 1 -- 172.21.15.40:0/741515779 shutdown_connections 2024-03-18T22:47:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe48c623740 1 -- 172.21.15.40:0/741515779 wait complete. 2024-03-18T22:47:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.702+0000 7fe48c623740 20 client.26617 trim_cache size 0 max 16384 2024-03-18T22:47:31.711 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:47:31.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.703+0000 7fe48995b640 1 -- 172.21.15.40:0/3675415281 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe46c0a2800 msgr2=0x7fe46c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.703+0000 7fe48995b640 1 --2- 172.21.15.40:0/3675415281 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe46c0a2800 0x7fe46c0a4cc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0bd4f0 tx=0x7fe47c0a8040 comp rx=0 tx=0).stop 2024-03-18T22:47:31.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.703+0000 7fe48995b640 1 -- 172.21.15.40:0/3675415281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4841033d0 msgr2=0x7fe484074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.703+0000 7fe48995b640 1 --2- 172.21.15.40:0/3675415281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4841033d0 0x7fe484074150 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fe47c02c700 tx=0x7fe47c0328a0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.703+0000 7fe48995b640 1 -- 172.21.15.40:0/3675415281 shutdown_connections 2024-03-18T22:47:31.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.703+0000 7fe48995b640 1 --2- 172.21.15.40:0/3675415281 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe46c0a2800 0x7fe46c0a4cc0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.703+0000 7fe48995b640 1 --2- 172.21.15.40:0/3675415281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4841033d0 0x7fe484074150 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.703+0000 7fe48995b640 1 --2- 172.21.15.40:0/3675415281 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe484102a10 0x7fe484073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.703+0000 7fe48995b640 1 --2- 172.21.15.40:0/3675415281 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe484101820 0x7fe4840755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.703+0000 7fe48995b640 1 -- 172.21.15.40:0/3675415281 >> 172.21.15.40:0/3675415281 conn(0x7fe4840fcd90 msgr2=0x7fe484105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.703+0000 7fe48995b640 1 -- 172.21.15.40:0/3675415281 shutdown_connections 2024-03-18T22:47:31.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.703+0000 7fe48995b640 1 -- 172.21.15.40:0/3675415281 wait complete. 2024-03-18T22:47:31.716 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:47:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.831+0000 7fb01e085640 1 -- 172.21.15.40:0/3287695213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb018101850 msgr2=0x7fb018101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.831+0000 7fb01e085640 1 --2- 172.21.15.40:0/3287695213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb018101850 0x7fb018101c70 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb008009900 tx=0x7fb00802bc30 comp rx=0 tx=0).stop 2024-03-18T22:47:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.831+0000 7fb01e085640 1 -- 172.21.15.40:0/3287695213 shutdown_connections 2024-03-18T22:47:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.831+0000 7fb01e085640 1 --2- 172.21.15.40:0/3287695213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb018103400 0x7fb018109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.831+0000 7fb01e085640 1 --2- 172.21.15.40:0/3287695213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb018102a40 0x7fb018102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.831+0000 7fb01e085640 1 --2- 172.21.15.40:0/3287695213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb018101850 0x7fb018101c70 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.831+0000 7fb01e085640 1 -- 172.21.15.40:0/3287695213 >> 172.21.15.40:0/3287695213 conn(0x7fb0180fcde0 msgr2=0x7fb0180ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.831+0000 7fb01e085640 1 -- 172.21.15.40:0/3287695213 shutdown_connections 2024-03-18T22:47:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.831+0000 7fb01e085640 1 -- 172.21.15.40:0/3287695213 wait complete. 2024-03-18T22:47:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.831+0000 7fb01e085640 1 Processor -- start 2024-03-18T22:47:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.831+0000 7fb01e085640 1 -- start start 2024-03-18T22:47:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb01e085640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb018101850 0x7fb0180755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb01e085640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb018102a40 0x7fb018073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb01e085640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb018103400 0x7fb018074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb01e085640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb01810d350 con 0x7fb018103400 2024-03-18T22:47:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb01e085640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb01810d1d0 con 0x7fb018102a40 2024-03-18T22:47:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb01e085640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb01810d4d0 con 0x7fb018101850 2024-03-18T22:47:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb0177fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb018101850 0x7fb0180755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb017fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb018103400 0x7fb018074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb0177fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb018101850 0x7fb0180755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33420/0 (socket says 172.21.15.40:33420) 2024-03-18T22:47:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb0177fe640 1 -- 172.21.15.40:0/3025151926 learned_addr learned my addr 172.21.15.40:0/3025151926 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb016ffd640 1 --2- 172.21.15.40:0/3025151926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb018102a40 0x7fb018073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb017fff640 1 -- 172.21.15.40:0/3025151926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb018101850 msgr2=0x7fb0180755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb017fff640 1 --2- 172.21.15.40:0/3025151926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb018101850 0x7fb0180755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb017fff640 1 -- 172.21.15.40:0/3025151926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb018102a40 msgr2=0x7fb018073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb017fff640 1 --2- 172.21.15.40:0/3025151926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb018102a40 0x7fb018073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb017fff640 1 -- 172.21.15.40:0/3025151926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0180749e0 con 0x7fb018103400 2024-03-18T22:47:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb016ffd640 1 --2- 172.21.15.40:0/3025151926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb018102a40 0x7fb018073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb0177fe640 1 --2- 172.21.15.40:0/3025151926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb018101850 0x7fb0180755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.832+0000 7fb017fff640 1 --2- 172.21.15.40:0/3025151926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb018103400 0x7fb018074150 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb010009fd0 tx=0x7fb01000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.833+0000 7fb014ff9640 1 -- 172.21.15.40:0/3025151926 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01000ce20 con 0x7fb018103400 2024-03-18T22:47:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.833+0000 7fb01e085640 1 -- 172.21.15.40:0/3025151926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0181a1b50 con 0x7fb018103400 2024-03-18T22:47:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.833+0000 7fb014ff9640 1 -- 172.21.15.40:0/3025151926 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb010005020 con 0x7fb018103400 2024-03-18T22:47:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.833+0000 7fb014ff9640 1 -- 172.21.15.40:0/3025151926 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb010010690 con 0x7fb018103400 2024-03-18T22:47:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.833+0000 7fb01e085640 1 -- 172.21.15.40:0/3025151926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0181a20a0 con 0x7fb018103400 2024-03-18T22:47:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.835+0000 7fb014ff9640 1 -- 172.21.15.40:0/3025151926 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb0100051d0 con 0x7fb018103400 2024-03-18T22:47:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb020c7c740 1 Processor -- start 2024-03-18T22:47:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb020c7c740 1 -- start start 2024-03-18T22:47:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb020c7c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d0655752f0 0x55d065575710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb020c7c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d065575ce0 0x55d06557a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb020c7c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d06557ae50 0x55d06557d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb020c7c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d0654d94b0 con 0x55d06557ae50 2024-03-18T22:47:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb020c7c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d065481d70 con 0x55d065575ce0 2024-03-18T22:47:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb020c7c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55d0654d5550 con 0x55d0655752f0 2024-03-18T22:47:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb014ff9640 1 --2- 172.21.15.40:0/3025151926 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0000a2800 0x7fb0000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb016ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d06557ae50 0x55d06557d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb016ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d0655752f0 0x55d065575710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb0177fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d065575ce0 0x55d06557a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb016ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d0655752f0 0x55d065575710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33426/0 (socket says 172.21.15.40:33426) 2024-03-18T22:47:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb0177fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d065575ce0 0x55d06557a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56942/0 (socket says 172.21.15.40:56942) 2024-03-18T22:47:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb016ffd640 1 -- 172.21.15.40:0/772269659 learned_addr learned my addr 172.21.15.40:0/772269659 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb017fff640 1 --2- 172.21.15.40:0/3025151926 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0000a2800 0x7fb0000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb014ff9640 1 -- 172.21.15.40:0/3025151926 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb0100cf560 con 0x7fb018103400 2024-03-18T22:47:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.836+0000 7fb016ffd640 1 -- 172.21.15.40:0/772269659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d065575ce0 msgr2=0x55d06557a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.837+0000 7fb016ffd640 1 --2- 172.21.15.40:0/772269659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d065575ce0 0x55d06557a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.837+0000 7fb016ffd640 1 -- 172.21.15.40:0/772269659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d06557ae50 msgr2=0x55d06557d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.837+0000 7fb016ffd640 1 --2- 172.21.15.40:0/772269659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d06557ae50 0x55d06557d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.837+0000 7fb0177fe640 1 --2- 172.21.15.40:0/772269659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d065575ce0 0x55d06557a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:47:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.837+0000 7fb016ffd640 1 -- 172.21.15.40:0/772269659 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0654a4210 con 0x55d0655752f0 2024-03-18T22:47:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.837+0000 7fb016ffd640 1 --2- 172.21.15.40:0/772269659 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d0655752f0 0x55d065575710 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009560 tx=0x7fb00c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ded87e333351fcd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.837+0000 7fb017fff640 1 --2- 172.21.15.40:0/3025151926 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0000a2800 0x7fb0000a4cc0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb010097f70 tx=0x7fb01006e840 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.837+0000 7faffdffb640 1 -- 172.21.15.40:0/772269659 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c016070 con 0x55d0655752f0 2024-03-18T22:47:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.837+0000 7faffdffb640 1 -- 172.21.15.40:0/772269659 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb00c00fe50 con 0x55d0655752f0 2024-03-18T22:47:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.837+0000 7faffdffb640 1 -- 172.21.15.40:0/772269659 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c019040 con 0x55d0655752f0 2024-03-18T22:47:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.838+0000 7fb020c7c740 1 -- 172.21.15.40:0/772269659 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d0655752f0 msgr2=0x55d065575710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.838+0000 7fb020c7c740 1 --2- 172.21.15.40:0/772269659 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d0655752f0 0x55d065575710 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009560 tx=0x7fb00c005f90 comp rx=0 tx=0).stop 2024-03-18T22:47:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.838+0000 7fb020c7c740 1 -- 172.21.15.40:0/772269659 shutdown_connections 2024-03-18T22:47:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.838+0000 7fb020c7c740 1 --2- 172.21.15.40:0/772269659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d06557ae50 0x55d06557d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.838+0000 7fb020c7c740 1 --2- 172.21.15.40:0/772269659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d065575ce0 0x55d06557a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.838+0000 7fb020c7c740 1 --2- 172.21.15.40:0/772269659 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d0655752f0 0x55d065575710 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.838+0000 7fb020c7c740 1 -- 172.21.15.40:0/772269659 >> 172.21.15.40:0/772269659 conn(0x55d065570860 msgr2=0x55d065570c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.838+0000 7fb020c7c740 1 -- 172.21.15.40:0/772269659 shutdown_connections 2024-03-18T22:47:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.838+0000 7fb020c7c740 1 -- 172.21.15.40:0/772269659 wait complete. 2024-03-18T22:47:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.838+0000 7fb020c7c740 1 -- start start 2024-03-18T22:47:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.838+0000 7fb020c7c740 1 Processor -- start 2024-03-18T22:47:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb020c7c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0655752f0 0x55d06557f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb020c7c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d06557fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb020c7c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d06557ae50 0x55d065586d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb020c7c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d06557de00 con 0x55d06557ae50 2024-03-18T22:47:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb020c7c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d065495310 con 0x55d0655752f0 2024-03-18T22:47:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb020c7c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55d06557df80 con 0x55d065575ce0 2024-03-18T22:47:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb020c7c740 10 client.0 _ll_register_callbacks cb 0x55d0654892d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb0177fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d06557fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb016ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0655752f0 0x55d06557f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb0177fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d06557fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33438/0 (socket says 172.21.15.40:33438) 2024-03-18T22:47:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb0177fe640 1 -- 172.21.15.40:0/2097042510 learned_addr learned my addr 172.21.15.40:0/2097042510 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb016ffd640 1 --2- 172.21.15.40:0/2097042510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d06557ae50 0x55d065586d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb0177fe640 1 -- 172.21.15.40:0/2097042510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0655752f0 msgr2=0x55d06557f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb0177fe640 1 --2- 172.21.15.40:0/2097042510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0655752f0 0x55d06557f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb0177fe640 1 -- 172.21.15.40:0/2097042510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d06557ae50 msgr2=0x55d065586d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb0177fe640 1 --2- 172.21.15.40:0/2097042510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d06557ae50 0x55d065586d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb0177fe640 1 -- 172.21.15.40:0/2097042510 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0651ed3f0 con 0x55d065575ce0 2024-03-18T22:47:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7fb0177fe640 1 --2- 172.21.15.40:0/2097042510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d06557fd20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb008005ab0 tx=0x7fb008005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc4a9cec5be1d451 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.839+0000 7faff6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:47:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.840+0000 7faff6ffd640 1 -- 172.21.15.40:0/2097042510 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb008005d50 con 0x55d065575ce0 2024-03-18T22:47:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.840+0000 7fb020c7c740 1 -- 172.21.15.40:0/2097042510 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb00c00fe50 con 0x55d065575ce0 2024-03-18T22:47:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.840+0000 7faff6ffd640 1 -- 172.21.15.40:0/2097042510 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb0080046c0 con 0x55d065575ce0 2024-03-18T22:47:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.840+0000 7faff6ffd640 1 -- 172.21.15.40:0/2097042510 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb00802cc90 con 0x55d065575ce0 2024-03-18T22:47:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.840+0000 7fb020c7c740 10 client.26629 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.840+0000 7fb020c7c740 1 -- 172.21.15.40:0/2097042510 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d0652b77d0 con 0x55d065575ce0 2024-03-18T22:47:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.840+0000 7faff6ffd640 1 -- 172.21.15.40:0/2097042510 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb00c00fe50 con 0x55d065575ce0 2024-03-18T22:47:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.840+0000 7faff6ffd640 1 -- 172.21.15.40:0/2097042510 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb008003a30 con 0x55d065575ce0 2024-03-18T22:47:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.840+0000 7fb020c7c740 10 client.26629 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.840+0000 7fb020c7c740 10 client.26629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.840+0000 7fb020c7c740 20 client.26629 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.840+0000 7fb020c7c740 1 --2- 172.21.15.40:0/2097042510 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d06558b650 0x55d06558da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.840+0000 7fb020c7c740 4 client.26629 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.840+0000 7fb020c7c740 1 -- 172.21.15.40:0/2097042510 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0655373e0 con 0x55d06558b650 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.841+0000 7fb016ffd640 1 --2- 172.21.15.40:0/2097042510 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d06558b650 0x55d06558da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.841+0000 7fb016ffd640 1 --2- 172.21.15.40:0/2097042510 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d06558b650 0x55d06558da90 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6aa9cff9baae8ea server_cookie=294bea8148f279f2 in_seq=0 out_seq=0 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.841+0000 7faff6ffd640 10 client.26629 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.841+0000 7faff6ffd640 1 -- 172.21.15.40:0/2097042510 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d0655373e0 con 0x55d06558b650 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.841+0000 7faff6ffd640 10 client.26629 handle_command_reply: tid=0 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.842+0000 7fb020c7c740 1 client.26629 shutdown 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.842+0000 7fb020c7c740 1 -- 172.21.15.40:0/2097042510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 msgr2=0x55d06557fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.842+0000 7fb020c7c740 1 --2- 172.21.15.40:0/2097042510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d06557fd20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb008005ab0 tx=0x7fb008005270 comp rx=0 tx=0).stop 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.842+0000 7fb020c7c740 1 -- 172.21.15.40:0/2097042510 shutdown_connections 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.842+0000 7fb020c7c740 1 --2- 172.21.15.40:0/2097042510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d06557ae50 0x55d065586d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.842+0000 7fb020c7c740 1 --2- 172.21.15.40:0/2097042510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d06557fd20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.842+0000 7fb020c7c740 1 --2- 172.21.15.40:0/2097042510 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d06558b650 0x55d06558da90 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.842+0000 7fb020c7c740 1 --2- 172.21.15.40:0/2097042510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0655752f0 0x55d06557f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.842+0000 7fb020c7c740 1 -- 172.21.15.40:0/2097042510 >> 172.21.15.40:0/2097042510 conn(0x55d065570860 msgr2=0x55d06556fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.842+0000 7faff6ffd640 0 client.26629 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.842+0000 7fb020c7c740 1 -- 172.21.15.40:0/2097042510 shutdown_connections 2024-03-18T22:47:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.842+0000 7fb020c7c740 1 -- 172.21.15.40:0/2097042510 wait complete. 2024-03-18T22:47:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.842+0000 7fb020c7c740 20 client.26629 trim_cache size 0 max 16384 2024-03-18T22:47:31.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb020c7c740 1 Processor -- start 2024-03-18T22:47:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb020c7c740 1 -- start start 2024-03-18T22:47:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb020c7c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0655752f0 0x55d06557f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb020c7c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d06557fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb020c7c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d06557ae50 0x55d065586d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb020c7c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb0181ac6c0 con 0x55d06557ae50 2024-03-18T22:47:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb020c7c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb0181ae200 con 0x55d0655752f0 2024-03-18T22:47:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb020c7c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb0181acd90 con 0x55d065575ce0 2024-03-18T22:47:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb016ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0655752f0 0x55d06557f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb0177fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d06557fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb0177fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d06557fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33446/0 (socket says 172.21.15.40:33446) 2024-03-18T22:47:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb016ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0655752f0 0x55d06557f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56952/0 (socket says 172.21.15.40:56952) 2024-03-18T22:47:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb0177fe640 1 -- 172.21.15.40:0/1818396121 learned_addr learned my addr 172.21.15.40:0/1818396121 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb016ffd640 1 --2- 172.21.15.40:0/1818396121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d06557ae50 0x55d065586d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb0177fe640 1 -- 172.21.15.40:0/1818396121 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0655752f0 msgr2=0x55d06557f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.861+0000 7fb0177fe640 1 --2- 172.21.15.40:0/1818396121 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0655752f0 0x55d06557f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.862+0000 7fb0177fe640 1 -- 172.21.15.40:0/1818396121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d06557ae50 msgr2=0x55d065586d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.862+0000 7fb0177fe640 1 --2- 172.21.15.40:0/1818396121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d06557ae50 0x55d065586d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.862+0000 7fb0177fe640 1 -- 172.21.15.40:0/1818396121 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d065546880 con 0x55d065575ce0 2024-03-18T22:47:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.862+0000 7fb0177fe640 1 --2- 172.21.15.40:0/1818396121 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d06557fd20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb008036040 tx=0x7fb00802bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abd562025cd859ac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.862+0000 7faffd7fa640 1 -- 172.21.15.40:0/1818396121 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00805a070 con 0x55d065575ce0 2024-03-18T22:47:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.862+0000 7faffd7fa640 1 -- 172.21.15.40:0/1818396121 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb00802cc90 con 0x55d065575ce0 2024-03-18T22:47:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.862+0000 7faffd7fa640 1 -- 172.21.15.40:0/1818396121 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb008003a50 con 0x55d065575ce0 2024-03-18T22:47:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.863+0000 7fb020c7c740 1 -- 172.21.15.40:0/1818396121 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 msgr2=0x55d06557fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.863+0000 7fb020c7c740 1 --2- 172.21.15.40:0/1818396121 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d06557fd20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb008036040 tx=0x7fb00802bc30 comp rx=0 tx=0).stop 2024-03-18T22:47:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.863+0000 7fb020c7c740 1 -- 172.21.15.40:0/1818396121 shutdown_connections 2024-03-18T22:47:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.863+0000 7fb020c7c740 1 --2- 172.21.15.40:0/1818396121 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d06557ae50 0x55d065586d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.863+0000 7fb020c7c740 1 --2- 172.21.15.40:0/1818396121 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d06557fd20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.863+0000 7fb020c7c740 1 --2- 172.21.15.40:0/1818396121 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0655752f0 0x55d06557f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.863+0000 7fb020c7c740 1 -- 172.21.15.40:0/1818396121 >> 172.21.15.40:0/1818396121 conn(0x55d065570860 msgr2=0x55d065570c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.863+0000 7fb020c7c740 1 -- 172.21.15.40:0/1818396121 shutdown_connections 2024-03-18T22:47:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.863+0000 7fb020c7c740 1 -- 172.21.15.40:0/1818396121 wait complete. 2024-03-18T22:47:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.863+0000 7fb020c7c740 1 -- start start 2024-03-18T22:47:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.863+0000 7fb020c7c740 1 Processor -- start 2024-03-18T22:47:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb020c7c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d0655752f0 0x55d065584840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb020c7c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d065584d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb020c7c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d06557ae50 0x55d065585d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb020c7c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb0181adb30 con 0x55d0655752f0 2024-03-18T22:47:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb020c7c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb0181ad460 con 0x55d06557ae50 2024-03-18T22:47:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb020c7c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb0181aa6a0 con 0x55d065575ce0 2024-03-18T22:47:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb020c7c740 10 client.0 _ll_register_callbacks cb 0x7fb008009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb016ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d0655752f0 0x55d065584840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb016ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d0655752f0 0x55d065584840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55712/0 (socket says 172.21.15.40:55712) 2024-03-18T22:47:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb016ffd640 1 -- 172.21.15.40:0/1330554373 learned_addr learned my addr 172.21.15.40:0/1330554373 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb0177fe640 1 --2- 172.21.15.40:0/1330554373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d065584d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb016ffd640 1 --2- 172.21.15.40:0/1330554373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d06557ae50 0x55d065585d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb016ffd640 1 -- 172.21.15.40:0/1330554373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 msgr2=0x55d065584d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb016ffd640 1 --2- 172.21.15.40:0/1330554373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d065584d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb016ffd640 1 -- 172.21.15.40:0/1330554373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d06557ae50 msgr2=0x55d065585d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb016ffd640 1 --2- 172.21.15.40:0/1330554373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d06557ae50 0x55d065585d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb016ffd640 1 -- 172.21.15.40:0/1330554373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d065586380 con 0x55d0655752f0 2024-03-18T22:47:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb016ffd640 1 --2- 172.21.15.40:0/1330554373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d0655752f0 0x55d065584840 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009700 tx=0x7fb00c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=730ec301d29e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7faff6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:47:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7faff6ffd640 1 -- 172.21.15.40:0/1330554373 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c016040 con 0x55d0655752f0 2024-03-18T22:47:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7faff6ffd640 1 -- 172.21.15.40:0/1330554373 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb00c0313e0 con 0x55d0655752f0 2024-03-18T22:47:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.864+0000 7fb020c7c740 1 -- 172.21.15.40:0/1330554373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb00802cc90 con 0x55d0655752f0 2024-03-18T22:47:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.865+0000 7faff6ffd640 1 -- 172.21.15.40:0/1330554373 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb00c040050 con 0x55d0655752f0 2024-03-18T22:47:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.865+0000 7fb020c7c740 10 client.16818 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.865+0000 7fb020c7c740 1 -- 172.21.15.40:0/1330554373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d0655f1d50 con 0x55d0655752f0 2024-03-18T22:47:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.865+0000 7faff6ffd640 1 -- 172.21.15.40:0/1330554373 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb00802cc90 con 0x55d0655752f0 2024-03-18T22:47:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.865+0000 7faff6ffd640 1 -- 172.21.15.40:0/1330554373 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fb00c030040 con 0x55d0655752f0 2024-03-18T22:47:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.865+0000 7fb020c7c740 10 client.16818 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.865+0000 7fb020c7c740 10 client.16818 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:47:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.865+0000 7fb020c7c740 20 client.16818 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.865+0000 7fb020c7c740 1 --2- 172.21.15.40:0/1330554373 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d0655f2540 0x55d0655f4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.865+0000 7fb020c7c740 4 client.16818 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:47:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.865+0000 7fb020c7c740 1 -- 172.21.15.40:0/1330554373 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d0654e69f0 con 0x55d0655f2540 2024-03-18T22:47:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.865+0000 7fb0177fe640 1 --2- 172.21.15.40:0/1330554373 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d0655f2540 0x55d0655f4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7faff6ffd640 10 client.16818 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7fb0177fe640 1 --2- 172.21.15.40:0/1330554373 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d0655f2540 0x55d0655f4960 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bf4ff35736c917b server_cookie=50b5260c06d7838 in_seq=0 out_seq=0 2024-03-18T22:47:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7faff6ffd640 1 -- 172.21.15.40:0/1330554373 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d0654e69f0 con 0x55d0655f2540 2024-03-18T22:47:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7faff6ffd640 10 client.16818 handle_command_reply: tid=0 2024-03-18T22:47:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7fb020c7c740 1 client.16818 shutdown 2024-03-18T22:47:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7fb020c7c740 1 -- 172.21.15.40:0/1330554373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d0655752f0 msgr2=0x55d065584840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7fb020c7c740 1 --2- 172.21.15.40:0/1330554373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d0655752f0 0x55d065584840 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009700 tx=0x7fb00c005f90 comp rx=0 tx=0).stop 2024-03-18T22:47:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7fb020c7c740 1 -- 172.21.15.40:0/1330554373 shutdown_connections 2024-03-18T22:47:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7fb020c7c740 1 --2- 172.21.15.40:0/1330554373 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d0655f2540 0x55d0655f4960 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7fb020c7c740 1 --2- 172.21.15.40:0/1330554373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d06557ae50 0x55d065585d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7fb020c7c740 1 --2- 172.21.15.40:0/1330554373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d065575ce0 0x55d065584d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7faff6ffd640 0 client.16818 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:31.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7fb020c7c740 1 --2- 172.21.15.40:0/1330554373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d0655752f0 0x55d065584840 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7fb020c7c740 1 -- 172.21.15.40:0/1330554373 >> 172.21.15.40:0/1330554373 conn(0x55d065570860 msgr2=0x55d06557f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7fb020c7c740 1 -- 172.21.15.40:0/1330554373 shutdown_connections 2024-03-18T22:47:31.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7fb020c7c740 1 -- 172.21.15.40:0/1330554373 wait complete. 2024-03-18T22:47:31.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.866+0000 7fb020c7c740 20 client.16818 trim_cache size 0 max 16384 2024-03-18T22:47:31.873 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:47:31.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.867+0000 7fb01e085640 1 -- 172.21.15.40:0/3025151926 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0000a2800 msgr2=0x7fb0000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.867+0000 7fb01e085640 1 --2- 172.21.15.40:0/3025151926 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0000a2800 0x7fb0000a4cc0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb010097f70 tx=0x7fb01006e840 comp rx=0 tx=0).stop 2024-03-18T22:47:31.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.867+0000 7fb01e085640 1 -- 172.21.15.40:0/3025151926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb018103400 msgr2=0x7fb018074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:31.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.867+0000 7fb01e085640 1 --2- 172.21.15.40:0/3025151926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb018103400 0x7fb018074150 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb010009fd0 tx=0x7fb01000ef10 comp rx=0 tx=0).stop 2024-03-18T22:47:31.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.867+0000 7fb01e085640 1 -- 172.21.15.40:0/3025151926 shutdown_connections 2024-03-18T22:47:31.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.867+0000 7fb01e085640 1 --2- 172.21.15.40:0/3025151926 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0000a2800 0x7fb0000a4cc0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.867+0000 7fb01e085640 1 --2- 172.21.15.40:0/3025151926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb018103400 0x7fb018074150 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.867+0000 7fb01e085640 1 --2- 172.21.15.40:0/3025151926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb018102a40 0x7fb018073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.867+0000 7fb01e085640 1 --2- 172.21.15.40:0/3025151926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb018101850 0x7fb0180755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:31.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.867+0000 7fb01e085640 1 -- 172.21.15.40:0/3025151926 >> 172.21.15.40:0/3025151926 conn(0x7fb0180fcde0 msgr2=0x7fb018105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:31.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.867+0000 7fb01e085640 1 -- 172.21.15.40:0/3025151926 shutdown_connections 2024-03-18T22:47:31.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:31.867+0000 7fb01e085640 1 -- 172.21.15.40:0/3025151926 wait complete. 2024-03-18T22:47:32.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:32 smithi113 ceph-mon[66775]: pgmap v497: 129 pgs: 129 active+clean; 5.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:47:32.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:32 smithi103 ceph-mon[66377]: pgmap v497: 129 pgs: 129 active+clean; 5.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:47:32.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:32 smithi040 ceph-mon[62643]: pgmap v497: 129 pgs: 129 active+clean; 5.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:47:32.879 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:47:32.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:47:32.880 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:47:33.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.997+0000 7f1d140f1640 1 -- 172.21.15.40:0/2051033829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d0c1013d0 msgr2=0x7f1d0c101850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.997+0000 7f1d140f1640 1 --2- 172.21.15.40:0/2051033829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d0c1013d0 0x7f1d0c101850 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009ca0 tx=0x7f1d0402c140 comp rx=0 tx=0).stop 2024-03-18T22:47:33.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.997+0000 7f1d140f1640 1 -- 172.21.15.40:0/2051033829 shutdown_connections 2024-03-18T22:47:33.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.997+0000 7f1d140f1640 1 --2- 172.21.15.40:0/2051033829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d0c1013d0 0x7f1d0c101850 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.997+0000 7f1d140f1640 1 --2- 172.21.15.40:0/2051033829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0c100a30 0x7f1d0c100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.997+0000 7f1d140f1640 1 --2- 172.21.15.40:0/2051033829 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d0c1000d0 0x7f1d0c1004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.997+0000 7f1d140f1640 1 -- 172.21.15.40:0/2051033829 >> 172.21.15.40:0/2051033829 conn(0x7f1d0c0fbcb0 msgr2=0x7f1d0c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:33.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.997+0000 7f1d140f1640 1 -- 172.21.15.40:0/2051033829 shutdown_connections 2024-03-18T22:47:33.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.998+0000 7f1d140f1640 1 -- 172.21.15.40:0/2051033829 wait complete. 2024-03-18T22:47:33.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.998+0000 7f1d140f1640 1 Processor -- start 2024-03-18T22:47:33.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.998+0000 7f1d140f1640 1 -- start start 2024-03-18T22:47:33.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.998+0000 7f1d140f1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d0c1000d0 0x7f1d0c199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.998+0000 7f1d140f1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0c100a30 0x7f1d0c19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.998+0000 7f1d140f1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d0c1013d0 0x7f1d0c1a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.998+0000 7f1d140f1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c10f530 con 0x7f1d0c1013d0 2024-03-18T22:47:33.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.998+0000 7f1d140f1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c10f3b0 con 0x7f1d0c100a30 2024-03-18T22:47:33.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.998+0000 7f1d140f1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c10f6b0 con 0x7f1d0c1000d0 2024-03-18T22:47:33.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d11e8d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d0c1000d0 0x7f1d0c199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d1168c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0c100a30 0x7f1d0c19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d1268e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d0c1013d0 0x7f1d0c1a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d11e8d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d0c1000d0 0x7f1d0c199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33470/0 (socket says 172.21.15.40:33470) 2024-03-18T22:47:33.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d1168c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0c100a30 0x7f1d0c19a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56984/0 (socket says 172.21.15.40:56984) 2024-03-18T22:47:33.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d11e8d640 1 -- 172.21.15.40:0/3995131017 learned_addr learned my addr 172.21.15.40:0/3995131017 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:33.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d1268e640 1 -- 172.21.15.40:0/3995131017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d0c1000d0 msgr2=0x7f1d0c199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d1268e640 1 --2- 172.21.15.40:0/3995131017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d0c1000d0 0x7f1d0c199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d1268e640 1 -- 172.21.15.40:0/3995131017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0c100a30 msgr2=0x7f1d0c19a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d1268e640 1 --2- 172.21.15.40:0/3995131017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0c100a30 0x7f1d0c19a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d1268e640 1 -- 172.21.15.40:0/3995131017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d0c1a1860 con 0x7f1d0c1013d0 2024-03-18T22:47:33.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d1168c640 1 --2- 172.21.15.40:0/3995131017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0c100a30 0x7f1d0c19a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:33.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d11e8d640 1 --2- 172.21.15.40:0/3995131017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d0c1000d0 0x7f1d0c199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:33.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d1268e640 1 --2- 172.21.15.40:0/3995131017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d0c1013d0 0x7f1d0c1a11c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1d0402c680 tx=0x7f1d04035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:33.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d02ffd640 1 -- 172.21.15.40:0/3995131017 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d04039040 con 0x7f1d0c1013d0 2024-03-18T22:47:33.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d02ffd640 1 -- 172.21.15.40:0/3995131017 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d04034700 con 0x7f1d0c1013d0 2024-03-18T22:47:33.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d140f1640 1 -- 172.21.15.40:0/3995131017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d0c1a1af0 con 0x7f1d0c1013d0 2024-03-18T22:47:33.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d02ffd640 1 -- 172.21.15.40:0/3995131017 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d040043c0 con 0x7f1d0c1013d0 2024-03-18T22:47:33.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:32.999+0000 7f1d140f1640 1 -- 172.21.15.40:0/3995131017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d0c1a1fe0 con 0x7f1d0c1013d0 2024-03-18T22:47:33.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.002+0000 7f1d02ffd640 1 -- 172.21.15.40:0/3995131017 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1d04046030 con 0x7f1d0c1013d0 2024-03-18T22:47:33.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.002+0000 7f1d16db8740 1 Processor -- start 2024-03-18T22:47:33.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d16db8740 1 -- start start 2024-03-18T22:47:33.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d16db8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84eb3a0 0x55ecb84eb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d16db8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84ebd90 0x55ecb84f0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d16db8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84f0f00 0x55ecb84f3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d16db8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ecb8450160 con 0x55ecb84ebd90 2024-03-18T22:47:33.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d16db8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ecb83f8e10 con 0x55ecb84eb3a0 2024-03-18T22:47:33.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d16db8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55ecb844c200 con 0x55ecb84f0f00 2024-03-18T22:47:33.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d11e8d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84ebd90 0x55ecb84f0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d11e8d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84ebd90 0x55ecb84f0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55736/0 (socket says 172.21.15.40:55736) 2024-03-18T22:47:33.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d11e8d640 1 -- 172.21.15.40:0/933331898 learned_addr learned my addr 172.21.15.40:0/933331898 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:33.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d1168c640 1 --2- 172.21.15.40:0/933331898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84eb3a0 0x55ecb84eb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d02ffd640 1 --2- 172.21.15.40:0/3995131017 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1cf80a2800 0x7f1cf80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d1168c640 1 --2- 172.21.15.40:0/933331898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84f0f00 0x55ecb84f3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d1268e640 1 --2- 172.21.15.40:0/3995131017 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1cf80a2800 0x7f1cf80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d1168c640 1 -- 172.21.15.40:0/933331898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84f0f00 msgr2=0x55ecb84f3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d1168c640 1 --2- 172.21.15.40:0/933331898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84f0f00 0x55ecb84f3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d1168c640 1 -- 172.21.15.40:0/933331898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84ebd90 msgr2=0x55ecb84f0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d1168c640 1 --2- 172.21.15.40:0/933331898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84ebd90 0x55ecb84f0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d1168c640 1 -- 172.21.15.40:0/933331898 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecb822c7e0 con 0x55ecb84eb3a0 2024-03-18T22:47:33.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d02ffd640 1 -- 172.21.15.40:0/3995131017 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1d040f0a70 con 0x7f1d0c1013d0 2024-03-18T22:47:33.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.003+0000 7f1d11e8d640 1 --2- 172.21.15.40:0/933331898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84ebd90 0x55ecb84f0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.004+0000 7f1d1168c640 1 --2- 172.21.15.40:0/933331898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84eb3a0 0x55ecb84eb7c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc011b20 tx=0x7f1cfc005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7e3df2c6315a5e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.004+0000 7f1d1268e640 1 --2- 172.21.15.40:0/3995131017 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1cf80a2800 0x7f1cf80a4cc0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1d0402c720 tx=0x7f1d040dd040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.004+0000 7f1cf7fff640 1 -- 172.21.15.40:0/933331898 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc011050 con 0x55ecb84eb3a0 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.004+0000 7f1cf7fff640 1 -- 172.21.15.40:0/933331898 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cfc00fdf0 con 0x55ecb84eb3a0 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.004+0000 7f1cf7fff640 1 -- 172.21.15.40:0/933331898 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc0055b0 con 0x55ecb84eb3a0 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.005+0000 7f1d16db8740 1 -- 172.21.15.40:0/933331898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84eb3a0 msgr2=0x55ecb84eb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.005+0000 7f1d16db8740 1 --2- 172.21.15.40:0/933331898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84eb3a0 0x55ecb84eb7c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc011b20 tx=0x7f1cfc005850 comp rx=0 tx=0).stop 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.005+0000 7f1d16db8740 1 -- 172.21.15.40:0/933331898 shutdown_connections 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.005+0000 7f1d16db8740 1 --2- 172.21.15.40:0/933331898 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84f0f00 0x55ecb84f3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.005+0000 7f1d16db8740 1 --2- 172.21.15.40:0/933331898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84ebd90 0x55ecb84f0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.005+0000 7f1d16db8740 1 --2- 172.21.15.40:0/933331898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84eb3a0 0x55ecb84eb7c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.005+0000 7f1d16db8740 1 -- 172.21.15.40:0/933331898 >> 172.21.15.40:0/933331898 conn(0x55ecb84e6910 msgr2=0x55ecb84e6d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.005+0000 7f1d16db8740 1 -- 172.21.15.40:0/933331898 shutdown_connections 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.005+0000 7f1d16db8740 1 -- 172.21.15.40:0/933331898 wait complete. 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.005+0000 7f1d16db8740 1 -- start start 2024-03-18T22:47:33.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.005+0000 7f1d16db8740 1 Processor -- start 2024-03-18T22:47:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.005+0000 7f1d16db8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84eb3a0 0x55ecb84f5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d16db8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84ebd90 0x55ecb84f5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d16db8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84f0f00 0x55ecb84fcd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d16db8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ecb84f3e50 con 0x55ecb84eb3a0 2024-03-18T22:47:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d16db8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ecb840c1a0 con 0x55ecb84f0f00 2024-03-18T22:47:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d16db8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55ecb84f3fd0 con 0x55ecb84ebd90 2024-03-18T22:47:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d16db8740 10 client.0 _ll_register_callbacks cb 0x55ecb8163c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d1168c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84eb3a0 0x55ecb84f5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d1168c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84eb3a0 0x55ecb84f5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55740/0 (socket says 172.21.15.40:55740) 2024-03-18T22:47:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d1168c640 1 -- 172.21.15.40:0/459464089 learned_addr learned my addr 172.21.15.40:0/459464089 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d11e8d640 1 --2- 172.21.15.40:0/459464089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84ebd90 0x55ecb84f5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d1168c640 1 --2- 172.21.15.40:0/459464089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84f0f00 0x55ecb84fcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d1168c640 1 -- 172.21.15.40:0/459464089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84ebd90 msgr2=0x55ecb84f5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d1168c640 1 --2- 172.21.15.40:0/459464089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84ebd90 0x55ecb84f5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d1168c640 1 -- 172.21.15.40:0/459464089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84f0f00 msgr2=0x55ecb84fcd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d1168c640 1 --2- 172.21.15.40:0/459464089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84f0f00 0x55ecb84fcd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d1168c640 1 -- 172.21.15.40:0/459464089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecb828cf70 con 0x55ecb84eb3a0 2024-03-18T22:47:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d1168c640 1 --2- 172.21.15.40:0/459464089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84eb3a0 0x55ecb84f5860 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc006010 tx=0x7f1cfc0096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70412cf9cd0fb1c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1cf57fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:47:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1cf57fa640 1 -- 172.21.15.40:0/459464089 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc011270 con 0x55ecb84eb3a0 2024-03-18T22:47:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1cf57fa640 1 -- 172.21.15.40:0/459464089 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1cfc0046c0 con 0x55ecb84eb3a0 2024-03-18T22:47:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.006+0000 7f1d16db8740 1 -- 172.21.15.40:0/459464089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cfc00fdf0 con 0x55ecb84eb3a0 2024-03-18T22:47:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.007+0000 7f1cf57fa640 1 -- 172.21.15.40:0/459464089 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1cfc032050 con 0x55ecb84eb3a0 2024-03-18T22:47:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.007+0000 7f1d16db8740 10 client.16830 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.007+0000 7f1d16db8740 1 -- 172.21.15.40:0/459464089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ecb8124170 con 0x55ecb84eb3a0 2024-03-18T22:47:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.007+0000 7f1cf57fa640 1 -- 172.21.15.40:0/459464089 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cfc00fdf0 con 0x55ecb84eb3a0 2024-03-18T22:47:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.007+0000 7f1cf57fa640 1 -- 172.21.15.40:0/459464089 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1cfc01f670 con 0x55ecb84eb3a0 2024-03-18T22:47:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.007+0000 7f1d16db8740 10 client.16830 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:33.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.007+0000 7f1d16db8740 10 client.16830 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:47:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.007+0000 7f1d16db8740 20 client.16830 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.007+0000 7f1d16db8740 1 --2- 172.21.15.40:0/459464089 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ecb8509cc0 0x55ecb850c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.007+0000 7f1d16db8740 4 client.16830 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:47:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.007+0000 7f1d16db8740 1 -- 172.21.15.40:0/459464089 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecb845d6a0 con 0x55ecb8509cc0 2024-03-18T22:47:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.007+0000 7f1d11e8d640 1 --2- 172.21.15.40:0/459464089 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ecb8509cc0 0x55ecb850c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.008+0000 7f1d11e8d640 1 --2- 172.21.15.40:0/459464089 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ecb8509cc0 0x55ecb850c0d0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aef366f4200d5f2d server_cookie=72e42592f52ed396 in_seq=0 out_seq=0 2024-03-18T22:47:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.008+0000 7f1cf57fa640 10 client.16830 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.008+0000 7f1cf57fa640 1 -- 172.21.15.40:0/459464089 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ecb845d6a0 con 0x55ecb8509cc0 2024-03-18T22:47:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.008+0000 7f1cf57fa640 10 client.16830 handle_command_reply: tid=0 2024-03-18T22:47:33.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.008+0000 7f1d16db8740 1 client.16830 shutdown 2024-03-18T22:47:33.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.008+0000 7f1d16db8740 1 -- 172.21.15.40:0/459464089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84eb3a0 msgr2=0x55ecb84f5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.008+0000 7f1d16db8740 1 --2- 172.21.15.40:0/459464089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84eb3a0 0x55ecb84f5860 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc006010 tx=0x7f1cfc0096c0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.009+0000 7f1d16db8740 1 -- 172.21.15.40:0/459464089 shutdown_connections 2024-03-18T22:47:33.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.009+0000 7f1d16db8740 1 --2- 172.21.15.40:0/459464089 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ecb8509cc0 0x55ecb850c0d0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.009+0000 7f1d16db8740 1 --2- 172.21.15.40:0/459464089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84f0f00 0x55ecb84fcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.009+0000 7f1d16db8740 1 --2- 172.21.15.40:0/459464089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84ebd90 0x55ecb84f5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.009+0000 7f1d16db8740 1 --2- 172.21.15.40:0/459464089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84eb3a0 0x55ecb84f5860 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.009+0000 7f1cf57fa640 0 client.16830 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:33.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.009+0000 7f1d16db8740 1 -- 172.21.15.40:0/459464089 >> 172.21.15.40:0/459464089 conn(0x55ecb84e6910 msgr2=0x55ecb84e5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:33.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.009+0000 7f1d16db8740 1 -- 172.21.15.40:0/459464089 shutdown_connections 2024-03-18T22:47:33.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.009+0000 7f1d16db8740 1 -- 172.21.15.40:0/459464089 wait complete. 2024-03-18T22:47:33.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.009+0000 7f1d16db8740 20 client.16830 trim_cache size 0 max 16384 2024-03-18T22:47:33.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d16db8740 1 Processor -- start 2024-03-18T22:47:33.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d16db8740 1 -- start start 2024-03-18T22:47:33.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d16db8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84eb3a0 0x55ecb84f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d16db8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84ebd90 0x55ecb84fcd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d16db8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84f0f00 0x55ecb8510990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d16db8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c1a81d0 con 0x55ecb84eb3a0 2024-03-18T22:47:33.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d16db8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c1a9d10 con 0x55ecb84ebd90 2024-03-18T22:47:33.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d16db8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c1a88a0 con 0x55ecb84f0f00 2024-03-18T22:47:33.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d1168c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84eb3a0 0x55ecb84f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d1168c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84eb3a0 0x55ecb84f4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55754/0 (socket says 172.21.15.40:55754) 2024-03-18T22:47:33.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d1168c640 1 -- 172.21.15.40:0/917872989 learned_addr learned my addr 172.21.15.40:0/917872989 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:33.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d11e8d640 1 --2- 172.21.15.40:0/917872989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84ebd90 0x55ecb84fcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d1168c640 1 --2- 172.21.15.40:0/917872989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84f0f00 0x55ecb8510990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d1168c640 1 -- 172.21.15.40:0/917872989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84f0f00 msgr2=0x55ecb8510990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d1168c640 1 --2- 172.21.15.40:0/917872989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84f0f00 0x55ecb8510990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d1168c640 1 -- 172.21.15.40:0/917872989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84ebd90 msgr2=0x55ecb84fcd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d1168c640 1 --2- 172.21.15.40:0/917872989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84ebd90 0x55ecb84fcd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.030+0000 7f1d1168c640 1 -- 172.21.15.40:0/917872989 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecb84e6740 con 0x55ecb84eb3a0 2024-03-18T22:47:33.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.031+0000 7f1d1168c640 1 --2- 172.21.15.40:0/917872989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84eb3a0 0x55ecb84f4ae0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc000c00 tx=0x7f1cfc001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc63776e9b150f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:33.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.031+0000 7f1cf77fe640 1 -- 172.21.15.40:0/917872989 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc005310 con 0x55ecb84eb3a0 2024-03-18T22:47:33.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.031+0000 7f1cf77fe640 1 -- 172.21.15.40:0/917872989 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cfc011030 con 0x55ecb84eb3a0 2024-03-18T22:47:33.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.031+0000 7f1cf77fe640 1 -- 172.21.15.40:0/917872989 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc003950 con 0x55ecb84eb3a0 2024-03-18T22:47:33.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.032+0000 7f1d16db8740 1 -- 172.21.15.40:0/917872989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84eb3a0 msgr2=0x55ecb84f4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.032+0000 7f1d16db8740 1 --2- 172.21.15.40:0/917872989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84eb3a0 0x55ecb84f4ae0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc000c00 tx=0x7f1cfc001e50 comp rx=0 tx=0).stop 2024-03-18T22:47:33.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.032+0000 7f1d16db8740 1 -- 172.21.15.40:0/917872989 shutdown_connections 2024-03-18T22:47:33.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.032+0000 7f1d16db8740 1 --2- 172.21.15.40:0/917872989 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84f0f00 0x55ecb8510990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.032+0000 7f1d16db8740 1 --2- 172.21.15.40:0/917872989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84ebd90 0x55ecb84fcd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.032+0000 7f1d16db8740 1 --2- 172.21.15.40:0/917872989 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84eb3a0 0x55ecb84f4ae0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.032+0000 7f1d16db8740 1 -- 172.21.15.40:0/917872989 >> 172.21.15.40:0/917872989 conn(0x55ecb84e6910 msgr2=0x55ecb84f43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:33.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.032+0000 7f1d16db8740 1 -- 172.21.15.40:0/917872989 shutdown_connections 2024-03-18T22:47:33.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.032+0000 7f1d16db8740 1 -- 172.21.15.40:0/917872989 wait complete. 2024-03-18T22:47:33.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.032+0000 7f1d16db8740 1 -- start start 2024-03-18T22:47:33.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.032+0000 7f1d16db8740 1 Processor -- start 2024-03-18T22:47:33.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d16db8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84eb3a0 0x55ecb84fa900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d16db8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84ebd90 0x55ecb84fae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d16db8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84f0f00 0x55ecb84fbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d16db8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c1a9640 con 0x55ecb84ebd90 2024-03-18T22:47:33.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d16db8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c1a8f70 con 0x55ecb84eb3a0 2024-03-18T22:47:33.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d16db8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c1a61b0 con 0x55ecb84f0f00 2024-03-18T22:47:33.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d16db8740 10 client.0 _ll_register_callbacks cb 0x7f1d08009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:33.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d11e8d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84ebd90 0x55ecb84fae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d11e8d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84ebd90 0x55ecb84fae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55756/0 (socket says 172.21.15.40:55756) 2024-03-18T22:47:33.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d11e8d640 1 -- 172.21.15.40:0/19303046 learned_addr learned my addr 172.21.15.40:0/19303046 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:33.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d1168c640 1 --2- 172.21.15.40:0/19303046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84eb3a0 0x55ecb84fa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d1168c640 1 --2- 172.21.15.40:0/19303046 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84f0f00 0x55ecb84fbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d11e8d640 1 -- 172.21.15.40:0/19303046 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84f0f00 msgr2=0x55ecb84fbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d11e8d640 1 --2- 172.21.15.40:0/19303046 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84f0f00 0x55ecb84fbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d11e8d640 1 -- 172.21.15.40:0/19303046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84eb3a0 msgr2=0x55ecb84fa900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d11e8d640 1 --2- 172.21.15.40:0/19303046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84eb3a0 0x55ecb84fa900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d11e8d640 1 -- 172.21.15.40:0/19303046 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecb84fc440 con 0x55ecb84ebd90 2024-03-18T22:47:33.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d11e8d640 1 --2- 172.21.15.40:0/19303046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84ebd90 0x55ecb84fae40 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1d08000c00 tx=0x7f1d08012ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ddb66c42703050d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:33.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1cf57fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:47:33.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1cf57fa640 1 -- 172.21.15.40:0/19303046 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d08004050 con 0x55ecb84ebd90 2024-03-18T22:47:33.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1cf57fa640 1 -- 172.21.15.40:0/19303046 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1d080041f0 con 0x55ecb84ebd90 2024-03-18T22:47:33.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.033+0000 7f1d16db8740 1 -- 172.21.15.40:0/19303046 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cfc011030 con 0x55ecb84ebd90 2024-03-18T22:47:33.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.034+0000 7f1cf57fa640 1 -- 172.21.15.40:0/19303046 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1d08030050 con 0x55ecb84ebd90 2024-03-18T22:47:33.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.034+0000 7f1d16db8740 10 client.16836 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:33.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.034+0000 7f1d16db8740 1 -- 172.21.15.40:0/19303046 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ecb85704f0 con 0x55ecb84ebd90 2024-03-18T22:47:33.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.034+0000 7f1cf57fa640 1 -- 172.21.15.40:0/19303046 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1cfc011030 con 0x55ecb84ebd90 2024-03-18T22:47:33.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.034+0000 7f1cf57fa640 1 -- 172.21.15.40:0/19303046 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1d08005450 con 0x55ecb84ebd90 2024-03-18T22:47:33.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.034+0000 7f1d16db8740 10 client.16836 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:33.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.034+0000 7f1d16db8740 10 client.16836 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:47:33.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.034+0000 7f1d16db8740 20 client.16836 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:33.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.034+0000 7f1d16db8740 1 --2- 172.21.15.40:0/19303046 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ecb8570f20 0x55ecb8573340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.034+0000 7f1d16db8740 4 client.16836 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:47:33.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.034+0000 7f1d16db8740 1 -- 172.21.15.40:0/19303046 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ecb84579f0 con 0x55ecb8570f20 2024-03-18T22:47:33.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.034+0000 7f1d1168c640 1 --2- 172.21.15.40:0/19303046 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ecb8570f20 0x55ecb8573340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.035+0000 7f1d1168c640 1 --2- 172.21.15.40:0/19303046 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ecb8570f20 0x55ecb8573340 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee34c637dcb3baaa server_cookie=f152c7361de20148 in_seq=0 out_seq=0 2024-03-18T22:47:33.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.035+0000 7f1cf57fa640 10 client.16836 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:33.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.036+0000 7f1cf57fa640 1 -- 172.21.15.40:0/19303046 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ecb84579f0 con 0x55ecb8570f20 2024-03-18T22:47:33.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.036+0000 7f1cf57fa640 10 client.16836 handle_command_reply: tid=0 2024-03-18T22:47:33.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.036+0000 7f1d16db8740 1 client.16836 shutdown 2024-03-18T22:47:33.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.036+0000 7f1d16db8740 1 -- 172.21.15.40:0/19303046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84ebd90 msgr2=0x55ecb84fae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.036+0000 7f1d16db8740 1 --2- 172.21.15.40:0/19303046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84ebd90 0x55ecb84fae40 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1d08000c00 tx=0x7f1d08012ef0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.036+0000 7f1d16db8740 1 -- 172.21.15.40:0/19303046 shutdown_connections 2024-03-18T22:47:33.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.036+0000 7f1d16db8740 1 --2- 172.21.15.40:0/19303046 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ecb84f0f00 0x55ecb84fbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.036+0000 7f1d16db8740 1 --2- 172.21.15.40:0/19303046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ecb84ebd90 0x55ecb84fae40 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.036+0000 7f1d16db8740 1 --2- 172.21.15.40:0/19303046 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ecb8570f20 0x55ecb8573340 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.036+0000 7f1d16db8740 1 --2- 172.21.15.40:0/19303046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ecb84eb3a0 0x55ecb84fa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.036+0000 7f1d16db8740 1 -- 172.21.15.40:0/19303046 >> 172.21.15.40:0/19303046 conn(0x55ecb84e6910 msgr2=0x55ecb84f4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:33.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.036+0000 7f1cf57fa640 0 client.16836 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:33.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.036+0000 7f1d16db8740 1 -- 172.21.15.40:0/19303046 shutdown_connections 2024-03-18T22:47:33.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.036+0000 7f1d16db8740 1 -- 172.21.15.40:0/19303046 wait complete. 2024-03-18T22:47:33.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.036+0000 7f1d16db8740 20 client.16836 trim_cache size 0 max 16384 2024-03-18T22:47:33.051 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:47:33.051 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:47:33.052 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "f9532611-4682-49a3-8a0d-3c09e93e4fb8", 2024-03-18T22:47:33.052 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:47:33.052 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:47:33.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.037+0000 7f1d140f1640 1 -- 172.21.15.40:0/3995131017 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1cf80a2800 msgr2=0x7f1cf80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.037+0000 7f1d140f1640 1 --2- 172.21.15.40:0/3995131017 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1cf80a2800 0x7f1cf80a4cc0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1d0402c720 tx=0x7f1d040dd040 comp rx=0 tx=0).stop 2024-03-18T22:47:33.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.037+0000 7f1d140f1640 1 -- 172.21.15.40:0/3995131017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d0c1013d0 msgr2=0x7f1d0c1a11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.037+0000 7f1d140f1640 1 --2- 172.21.15.40:0/3995131017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d0c1013d0 0x7f1d0c1a11c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1d0402c680 tx=0x7f1d04035d60 comp rx=0 tx=0).stop 2024-03-18T22:47:33.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.037+0000 7f1d140f1640 1 -- 172.21.15.40:0/3995131017 shutdown_connections 2024-03-18T22:47:33.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.037+0000 7f1d140f1640 1 --2- 172.21.15.40:0/3995131017 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1cf80a2800 0x7f1cf80a4cc0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.037+0000 7f1d140f1640 1 --2- 172.21.15.40:0/3995131017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d0c1013d0 0x7f1d0c1a11c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.037+0000 7f1d140f1640 1 --2- 172.21.15.40:0/3995131017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d0c100a30 0x7f1d0c19a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.037+0000 7f1d140f1640 1 --2- 172.21.15.40:0/3995131017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d0c1000d0 0x7f1d0c199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.037+0000 7f1d140f1640 1 -- 172.21.15.40:0/3995131017 >> 172.21.15.40:0/3995131017 conn(0x7f1d0c0fbcb0 msgr2=0x7f1d0c0fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:33.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.037+0000 7f1d140f1640 1 -- 172.21.15.40:0/3995131017 shutdown_connections 2024-03-18T22:47:33.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.037+0000 7f1d140f1640 1 -- 172.21.15.40:0/3995131017 wait complete. 2024-03-18T22:47:33.054 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:47:33.054 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:47:33.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.175+0000 7f1e151f4640 1 -- 172.21.15.40:0/3781805705 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e10102a40 msgr2=0x7f1e10102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.175+0000 7f1e151f4640 1 --2- 172.21.15.40:0/3781805705 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e10102a40 0x7f1e10102ec0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1e04009900 tx=0x7f1e0402bc30 comp rx=0 tx=0).stop 2024-03-18T22:47:33.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.175+0000 7f1e151f4640 1 -- 172.21.15.40:0/3781805705 shutdown_connections 2024-03-18T22:47:33.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.175+0000 7f1e151f4640 1 --2- 172.21.15.40:0/3781805705 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e10103400 0x7f1e10109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.175+0000 7f1e151f4640 1 --2- 172.21.15.40:0/3781805705 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e10102a40 0x7f1e10102ec0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.175+0000 7f1e151f4640 1 --2- 172.21.15.40:0/3781805705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e10101850 0x7f1e10101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.175+0000 7f1e151f4640 1 -- 172.21.15.40:0/3781805705 >> 172.21.15.40:0/3781805705 conn(0x7f1e100fcdc0 msgr2=0x7f1e100ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:33.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.175+0000 7f1e151f4640 1 -- 172.21.15.40:0/3781805705 shutdown_connections 2024-03-18T22:47:33.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.175+0000 7f1e151f4640 1 -- 172.21.15.40:0/3781805705 wait complete. 2024-03-18T22:47:33.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.176+0000 7f1e151f4640 1 Processor -- start 2024-03-18T22:47:33.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.176+0000 7f1e151f4640 1 -- start start 2024-03-18T22:47:33.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.176+0000 7f1e151f4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e10101850 0x7f1e100755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.176+0000 7f1e151f4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e10102a40 0x7f1e10073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.176+0000 7f1e151f4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e10103400 0x7f1e10074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.176+0000 7f1e151f4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1010d380 con 0x7f1e10101850 2024-03-18T22:47:33.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.176+0000 7f1e151f4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1010d200 con 0x7f1e10103400 2024-03-18T22:47:33.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.176+0000 7f1e151f4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1010d500 con 0x7f1e10102a40 2024-03-18T22:47:33.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.176+0000 7f1e0ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e10101850 0x7f1e100755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.176+0000 7f1e0ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e10101850 0x7f1e100755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55772/0 (socket says 172.21.15.40:55772) 2024-03-18T22:47:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.176+0000 7f1e0ed9d640 1 -- 172.21.15.40:0/393243974 learned_addr learned my addr 172.21.15.40:0/393243974 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.176+0000 7f1e0e59c640 1 --2- 172.21.15.40:0/393243974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e10102a40 0x7f1e10073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.176+0000 7f1e0f59e640 1 --2- 172.21.15.40:0/393243974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e10103400 0x7f1e10074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.177+0000 7f1e0e59c640 1 -- 172.21.15.40:0/393243974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e10103400 msgr2=0x7f1e10074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.177+0000 7f1e0e59c640 1 --2- 172.21.15.40:0/393243974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e10103400 0x7f1e10074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.177+0000 7f1e0e59c640 1 -- 172.21.15.40:0/393243974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e10101850 msgr2=0x7f1e100755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.177+0000 7f1e0e59c640 1 --2- 172.21.15.40:0/393243974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e10101850 0x7f1e100755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.177+0000 7f1e0e59c640 1 -- 172.21.15.40:0/393243974 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e100749e0 con 0x7f1e10102a40 2024-03-18T22:47:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.177+0000 7f1e0ed9d640 1 --2- 172.21.15.40:0/393243974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e10101850 0x7f1e100755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.177+0000 7f1e0e59c640 1 --2- 172.21.15.40:0/393243974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e10102a40 0x7f1e10073c10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1e04000c00 tx=0x7f1e04005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.177+0000 7f1df7fff640 1 -- 172.21.15.40:0/393243974 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e04043020 con 0x7f1e10102a40 2024-03-18T22:47:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.177+0000 7f1df7fff640 1 -- 172.21.15.40:0/393243974 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e04004350 con 0x7f1e10102a40 2024-03-18T22:47:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.177+0000 7f1e151f4640 1 -- 172.21.15.40:0/393243974 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e101a1bf0 con 0x7f1e10102a40 2024-03-18T22:47:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.177+0000 7f1df7fff640 1 -- 172.21.15.40:0/393243974 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e04003b50 con 0x7f1e10102a40 2024-03-18T22:47:33.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.177+0000 7f1e151f4640 1 -- 172.21.15.40:0/393243974 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e101a2140 con 0x7f1e10102a40 2024-03-18T22:47:33.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.179+0000 7f1df7fff640 1 -- 172.21.15.40:0/393243974 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1e04046020 con 0x7f1e10102a40 2024-03-18T22:47:33.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.180+0000 7f1df7fff640 1 --2- 172.21.15.40:0/393243974 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1df80a2800 0x7f1df80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.180+0000 7f1e0ed9d640 1 --2- 172.21.15.40:0/393243974 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1df80a2800 0x7f1df80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.180+0000 7f1df7fff640 1 -- 172.21.15.40:0/393243974 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1e040f0eb0 con 0x7f1e10102a40 2024-03-18T22:47:33.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.180+0000 7f1e17de3740 1 Processor -- start 2024-03-18T22:47:33.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.180+0000 7f1e17de3740 1 -- start start 2024-03-18T22:47:33.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.180+0000 7f1e0ed9d640 1 --2- 172.21.15.40:0/393243974 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1df80a2800 0x7f1df80a4cc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1e000058c0 tx=0x7f1e00005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:33.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.180+0000 7f1e17de3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 0x5610737d1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.180+0000 7f1e17de3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d2440 0x5610737d6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.180+0000 7f1e17de3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d75b0 0x5610737d9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.180+0000 7f1e17de3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561073736500 con 0x5610737d75b0 2024-03-18T22:47:33.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.180+0000 7f1e17de3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5610736debe0 con 0x5610737d1a50 2024-03-18T22:47:33.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.180+0000 7f1e17de3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5610737325a0 con 0x5610737d2440 2024-03-18T22:47:33.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1e0ed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 0x5610737d1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1e0f59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d75b0 0x5610737d9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1e0ed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 0x5610737d1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:57042/0 (socket says 172.21.15.40:57042) 2024-03-18T22:47:33.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1e0ed9d640 1 -- 172.21.15.40:0/3610826259 learned_addr learned my addr 172.21.15.40:0/3610826259 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:33.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1e0f59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d75b0 0x5610737d9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55780/0 (socket says 172.21.15.40:55780) 2024-03-18T22:47:33.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1e0e59c640 1 --2- 172.21.15.40:0/3610826259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d2440 0x5610737d6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1e0f59e640 1 -- 172.21.15.40:0/3610826259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d2440 msgr2=0x5610737d6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1e0f59e640 1 --2- 172.21.15.40:0/3610826259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d2440 0x5610737d6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1e0f59e640 1 -- 172.21.15.40:0/3610826259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 msgr2=0x5610737d1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1e0f59e640 1 --2- 172.21.15.40:0/3610826259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 0x5610737d1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1e0f59e640 1 -- 172.21.15.40:0/3610826259 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561073700000 con 0x5610737d75b0 2024-03-18T22:47:33.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1e0ed9d640 1 --2- 172.21.15.40:0/3610826259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 0x5610737d1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:33.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1e0f59e640 1 --2- 172.21.15.40:0/3610826259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d75b0 0x5610737d9a40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009f00 tx=0x7f1e08011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e788fd9d8764a7a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:33.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1df57fa640 1 -- 172.21.15.40:0/3610826259 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e08004d20 con 0x5610737d75b0 2024-03-18T22:47:33.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1df57fa640 1 -- 172.21.15.40:0/3610826259 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e08004030 con 0x5610737d75b0 2024-03-18T22:47:33.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.181+0000 7f1df57fa640 1 -- 172.21.15.40:0/3610826259 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e08005260 con 0x5610737d75b0 2024-03-18T22:47:33.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.182+0000 7f1e17de3740 1 -- 172.21.15.40:0/3610826259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d75b0 msgr2=0x5610737d9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.182+0000 7f1e17de3740 1 --2- 172.21.15.40:0/3610826259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d75b0 0x5610737d9a40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009f00 tx=0x7f1e08011810 comp rx=0 tx=0).stop 2024-03-18T22:47:33.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.182+0000 7f1e17de3740 1 -- 172.21.15.40:0/3610826259 shutdown_connections 2024-03-18T22:47:33.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.182+0000 7f1e17de3740 1 --2- 172.21.15.40:0/3610826259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d75b0 0x5610737d9a40 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.182+0000 7f1e17de3740 1 --2- 172.21.15.40:0/3610826259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d2440 0x5610737d6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.182+0000 7f1e17de3740 1 --2- 172.21.15.40:0/3610826259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 0x5610737d1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.182+0000 7f1e17de3740 1 -- 172.21.15.40:0/3610826259 >> 172.21.15.40:0/3610826259 conn(0x5610737ccfc0 msgr2=0x5610737cd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:33.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.182+0000 7f1e17de3740 1 -- 172.21.15.40:0/3610826259 shutdown_connections 2024-03-18T22:47:33.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.182+0000 7f1e17de3740 1 -- 172.21.15.40:0/3610826259 wait complete. 2024-03-18T22:47:33.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.182+0000 7f1e17de3740 1 -- start start 2024-03-18T22:47:33.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.182+0000 7f1e17de3740 1 Processor -- start 2024-03-18T22:47:33.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e17de3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 0x5610737dbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e17de3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d2440 0x5610737dc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e17de3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d75b0 0x5610737e33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e17de3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5610737da560 con 0x5610737d2440 2024-03-18T22:47:33.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e17de3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5610736f2160 con 0x5610737d1a50 2024-03-18T22:47:33.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e17de3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5610737da6e0 con 0x5610737d75b0 2024-03-18T22:47:33.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e17de3740 10 client.0 _ll_register_callbacks cb 0x56107372ea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:33.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e0e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d2440 0x5610737dc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e0e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d2440 0x5610737dc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55794/0 (socket says 172.21.15.40:55794) 2024-03-18T22:47:33.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e0e59c640 1 -- 172.21.15.40:0/3291543199 learned_addr learned my addr 172.21.15.40:0/3291543199 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:33.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e0f59e640 1 --2- 172.21.15.40:0/3291543199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d75b0 0x5610737e33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e0ed9d640 1 --2- 172.21.15.40:0/3291543199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 0x5610737dbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e0e59c640 1 -- 172.21.15.40:0/3291543199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d75b0 msgr2=0x5610737e33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e0e59c640 1 --2- 172.21.15.40:0/3291543199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d75b0 0x5610737e33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e0e59c640 1 -- 172.21.15.40:0/3291543199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 msgr2=0x5610737dbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e0e59c640 1 --2- 172.21.15.40:0/3291543199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 0x5610737dbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e0e59c640 1 -- 172.21.15.40:0/3291543199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610735147d0 con 0x5610737d2440 2024-03-18T22:47:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1e0e59c640 1 --2- 172.21.15.40:0/3291543199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d2440 0x5610737dc3e0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1e040bd160 tx=0x7f1e040bd060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=156b9c53421a4243 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1df2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:47:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.183+0000 7f1df2ff5640 1 -- 172.21.15.40:0/3291543199 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e040c15b0 con 0x5610737d2440 2024-03-18T22:47:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.184+0000 7f1df2ff5640 1 -- 172.21.15.40:0/3291543199 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1e040b9ad0 con 0x5610737d2440 2024-03-18T22:47:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.184+0000 7f1e17de3740 1 -- 172.21.15.40:0/3291543199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e08004030 con 0x5610737d2440 2024-03-18T22:47:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.184+0000 7f1df2ff5640 1 -- 172.21.15.40:0/3291543199 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1e040c5c00 con 0x5610737d2440 2024-03-18T22:47:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.184+0000 7f1e17de3740 10 client.16848 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.184+0000 7f1e17de3740 1 -- 172.21.15.40:0/3291543199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56107344a3f0 con 0x5610737d2440 2024-03-18T22:47:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.184+0000 7f1df2ff5640 1 -- 172.21.15.40:0/3291543199 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e08004030 con 0x5610737d2440 2024-03-18T22:47:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.184+0000 7f1df2ff5640 1 -- 172.21.15.40:0/3291543199 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1e040c5770 con 0x5610737d2440 2024-03-18T22:47:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.184+0000 7f1e17de3740 10 client.16848 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.184+0000 7f1e17de3740 10 client.16848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:47:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.184+0000 7f1e17de3740 20 client.16848 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.184+0000 7f1e17de3740 1 --2- 172.21.15.40:0/3291543199 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5610737e7cc0 0x5610737ea130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.184+0000 7f1e17de3740 4 client.16848 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:47:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.184+0000 7f1e17de3740 1 -- 172.21.15.40:0/3291543199 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561073743a40 con 0x5610737e7cc0 2024-03-18T22:47:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.184+0000 7f1e0ed9d640 1 --2- 172.21.15.40:0/3291543199 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5610737e7cc0 0x5610737ea130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.185+0000 7f1df2ff5640 10 client.16848 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.185+0000 7f1e0ed9d640 1 --2- 172.21.15.40:0/3291543199 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5610737e7cc0 0x5610737ea130 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adc9e2c00b668af4 server_cookie=5ee7bb1403b8b7ba in_seq=0 out_seq=0 2024-03-18T22:47:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.186+0000 7f1df2ff5640 1 -- 172.21.15.40:0/3291543199 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561073743a40 con 0x5610737e7cc0 2024-03-18T22:47:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.186+0000 7f1df2ff5640 10 client.16848 handle_command_reply: tid=0 2024-03-18T22:47:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.186+0000 7f1e17de3740 1 client.16848 shutdown 2024-03-18T22:47:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.186+0000 7f1e17de3740 1 -- 172.21.15.40:0/3291543199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d2440 msgr2=0x5610737dc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.186+0000 7f1e17de3740 1 --2- 172.21.15.40:0/3291543199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d2440 0x5610737dc3e0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1e040bd160 tx=0x7f1e040bd060 comp rx=0 tx=0).stop 2024-03-18T22:47:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.186+0000 7f1e17de3740 1 -- 172.21.15.40:0/3291543199 shutdown_connections 2024-03-18T22:47:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.186+0000 7f1e17de3740 1 --2- 172.21.15.40:0/3291543199 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d75b0 0x5610737e33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.186+0000 7f1e17de3740 1 --2- 172.21.15.40:0/3291543199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d2440 0x5610737dc3e0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.186+0000 7f1e17de3740 1 --2- 172.21.15.40:0/3291543199 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5610737e7cc0 0x5610737ea130 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.186+0000 7f1e17de3740 1 --2- 172.21.15.40:0/3291543199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 0x5610737dbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.186+0000 7f1df2ff5640 0 client.16848 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.186+0000 7f1e17de3740 1 -- 172.21.15.40:0/3291543199 >> 172.21.15.40:0/3291543199 conn(0x5610737ccfc0 msgr2=0x5610737cc1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.186+0000 7f1e17de3740 1 -- 172.21.15.40:0/3291543199 shutdown_connections 2024-03-18T22:47:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.186+0000 7f1e17de3740 1 -- 172.21.15.40:0/3291543199 wait complete. 2024-03-18T22:47:33.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.186+0000 7f1e17de3740 20 client.16848 trim_cache size 0 max 16384 2024-03-18T22:47:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.204+0000 7f1e17de3740 1 Processor -- start 2024-03-18T22:47:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.204+0000 7f1e17de3740 1 -- start start 2024-03-18T22:47:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e17de3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d1a50 0x5610737cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e17de3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d2440 0x5610737e3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e17de3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d75b0 0x5610737eea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e17de3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1e101ac770 con 0x5610737d1a50 2024-03-18T22:47:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e17de3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e101ae2b0 con 0x5610737d75b0 2024-03-18T22:47:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e17de3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1e101ace40 con 0x5610737d2440 2024-03-18T22:47:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e0ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d1a50 0x5610737cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e0ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d1a50 0x5610737cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55796/0 (socket says 172.21.15.40:55796) 2024-03-18T22:47:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e0ed9d640 1 -- 172.21.15.40:0/482482243 learned_addr learned my addr 172.21.15.40:0/482482243 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e0e59c640 1 --2- 172.21.15.40:0/482482243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d2440 0x5610737e3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e0f59e640 1 --2- 172.21.15.40:0/482482243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d75b0 0x5610737eea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e0ed9d640 1 -- 172.21.15.40:0/482482243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d2440 msgr2=0x5610737e3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e0ed9d640 1 --2- 172.21.15.40:0/482482243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d2440 0x5610737e3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e0ed9d640 1 -- 172.21.15.40:0/482482243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d75b0 msgr2=0x5610737eea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e0ed9d640 1 --2- 172.21.15.40:0/482482243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d75b0 0x5610737eea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e0ed9d640 1 -- 172.21.15.40:0/482482243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56107340a0f0 con 0x5610737d1a50 2024-03-18T22:47:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1e0ed9d640 1 --2- 172.21.15.40:0/482482243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d1a50 0x5610737cc060 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1e00011db0 tx=0x7f1e00014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee967f8dfaca4fea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1df4ff9640 1 -- 172.21.15.40:0/482482243 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e00030070 con 0x5610737d1a50 2024-03-18T22:47:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1df4ff9640 1 -- 172.21.15.40:0/482482243 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e0000fe80 con 0x5610737d1a50 2024-03-18T22:47:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.205+0000 7f1df4ff9640 1 -- 172.21.15.40:0/482482243 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e00028670 con 0x5610737d1a50 2024-03-18T22:47:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.206+0000 7f1e17de3740 1 -- 172.21.15.40:0/482482243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d1a50 msgr2=0x5610737cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.206+0000 7f1e17de3740 1 --2- 172.21.15.40:0/482482243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d1a50 0x5610737cc060 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1e00011db0 tx=0x7f1e00014000 comp rx=0 tx=0).stop 2024-03-18T22:47:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.206+0000 7f1e17de3740 1 -- 172.21.15.40:0/482482243 shutdown_connections 2024-03-18T22:47:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.206+0000 7f1e17de3740 1 --2- 172.21.15.40:0/482482243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d75b0 0x5610737eea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.206+0000 7f1e17de3740 1 --2- 172.21.15.40:0/482482243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d2440 0x5610737e3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.206+0000 7f1e17de3740 1 --2- 172.21.15.40:0/482482243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d1a50 0x5610737cc060 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.206+0000 7f1e17de3740 1 -- 172.21.15.40:0/482482243 >> 172.21.15.40:0/482482243 conn(0x5610737ccfc0 msgr2=0x5610737cb950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.206+0000 7f1e17de3740 1 -- 172.21.15.40:0/482482243 shutdown_connections 2024-03-18T22:47:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.206+0000 7f1e17de3740 1 -- 172.21.15.40:0/482482243 wait complete. 2024-03-18T22:47:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.206+0000 7f1e17de3740 1 -- start start 2024-03-18T22:47:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e17de3740 1 Processor -- start 2024-03-18T22:47:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e17de3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 0x5610737e0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e17de3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d2440 0x5610737e1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e17de3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d75b0 0x5610737e2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e17de3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1e101adbe0 con 0x5610737d75b0 2024-03-18T22:47:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e17de3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e101ad510 con 0x5610737d1a50 2024-03-18T22:47:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e17de3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1e101aa750 con 0x5610737d2440 2024-03-18T22:47:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e17de3740 10 client.0 _ll_register_callbacks cb 0x7f1e040095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:47:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e0f59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d75b0 0x5610737e2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e0f59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d75b0 0x5610737e2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55806/0 (socket says 172.21.15.40:55806) 2024-03-18T22:47:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e0f59e640 1 -- 172.21.15.40:0/2355623799 learned_addr learned my addr 172.21.15.40:0/2355623799 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:47:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e0ed9d640 1 --2- 172.21.15.40:0/2355623799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 0x5610737e0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e0e59c640 1 --2- 172.21.15.40:0/2355623799 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d2440 0x5610737e1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e0ed9d640 1 -- 172.21.15.40:0/2355623799 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d2440 msgr2=0x5610737e1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e0ed9d640 1 --2- 172.21.15.40:0/2355623799 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d2440 0x5610737e1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e0ed9d640 1 -- 172.21.15.40:0/2355623799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d75b0 msgr2=0x5610737e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:47:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e0ed9d640 1 --2- 172.21.15.40:0/2355623799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d75b0 0x5610737e2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e0ed9d640 1 -- 172.21.15.40:0/2355623799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610737e2760 con 0x5610737d1a50 2024-03-18T22:47:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.207+0000 7f1e0f59e640 1 --2- 172.21.15.40:0/2355623799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d75b0 0x5610737e2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:47:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1e0ed9d640 1 --2- 172.21.15.40:0/2355623799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 0x5610737e0c20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1e00009f40 tx=0x7f1e0002b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61e9f8fc880cdb2b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:47:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1df2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:47:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1df2ff5640 1 -- 172.21.15.40:0/2355623799 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e00030040 con 0x5610737d1a50 2024-03-18T22:47:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1e17de3740 1 -- 172.21.15.40:0/2355623799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56107373dd90 con 0x5610737d1a50 2024-03-18T22:47:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1df2ff5640 1 -- 172.21.15.40:0/2355623799 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1e0002b730 con 0x5610737d1a50 2024-03-18T22:47:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1df2ff5640 1 -- 172.21.15.40:0/2355623799 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1e0002b160 con 0x5610737d1a50 2024-03-18T22:47:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1e17de3740 10 client.26681 fetch_fsmap learned FSMap version 32 2024-03-18T22:47:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1e17de3740 1 -- 172.21.15.40:0/2355623799 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56107384e2b0 con 0x5610737d1a50 2024-03-18T22:47:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1df2ff5640 1 -- 172.21.15.40:0/2355623799 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56107373dd90 con 0x5610737d1a50 2024-03-18T22:47:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1df2ff5640 1 -- 172.21.15.40:0/2355623799 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1e000348e0 con 0x5610737d1a50 2024-03-18T22:47:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1e17de3740 10 client.26681 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:47:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1e17de3740 10 client.26681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:47:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1e17de3740 20 client.26681 populate_metadata read hostname 'smithi040' 2024-03-18T22:47:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1e17de3740 1 --2- 172.21.15.40:0/2355623799 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56107384eba0 0x561073850fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:47:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1e17de3740 4 client.26681 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:47:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.208+0000 7f1e17de3740 1 -- 172.21.15.40:0/2355623799 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5610736bb540 con 0x56107384eba0 2024-03-18T22:47:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1e0e59c640 1 --2- 172.21.15.40:0/2355623799 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56107384eba0 0x561073850fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:47:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1df2ff5640 10 client.26681 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1e0e59c640 1 --2- 172.21.15.40:0/2355623799 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56107384eba0 0x561073850fc0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a825abbae1c70eb9 server_cookie=e8905965977c8887 in_seq=0 out_seq=0 2024-03-18T22:47:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1df2ff5640 1 -- 172.21.15.40:0/2355623799 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5610736bb540 con 0x56107384eba0 2024-03-18T22:47:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1df2ff5640 10 client.26681 handle_command_reply: tid=0 2024-03-18T22:47:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1e17de3740 1 client.26681 shutdown 2024-03-18T22:47:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1e17de3740 1 -- 172.21.15.40:0/2355623799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 msgr2=0x5610737e0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1e17de3740 1 --2- 172.21.15.40:0/2355623799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 0x5610737e0c20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1e00009f40 tx=0x7f1e0002b480 comp rx=0 tx=0).stop 2024-03-18T22:47:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1e17de3740 1 -- 172.21.15.40:0/2355623799 shutdown_connections 2024-03-18T22:47:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1e17de3740 1 --2- 172.21.15.40:0/2355623799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5610737d75b0 0x5610737e2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1e17de3740 1 --2- 172.21.15.40:0/2355623799 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5610737d2440 0x5610737e1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1e17de3740 1 --2- 172.21.15.40:0/2355623799 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56107384eba0 0x561073850fc0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1e17de3740 1 --2- 172.21.15.40:0/2355623799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5610737d1a50 0x5610737e0c20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1df2ff5640 0 client.26681 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:47:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1e17de3740 1 -- 172.21.15.40:0/2355623799 >> 172.21.15.40:0/2355623799 conn(0x5610737ccfc0 msgr2=0x5610737d0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1e17de3740 1 -- 172.21.15.40:0/2355623799 shutdown_connections 2024-03-18T22:47:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1e17de3740 1 -- 172.21.15.40:0/2355623799 wait complete. 2024-03-18T22:47:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.209+0000 7f1e17de3740 20 client.26681 trim_cache size 0 max 16384 2024-03-18T22:47:33.221 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:47:33.221 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:47:33.221 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:47:33.221 INFO:teuthology.orchestra.run.smithi040.stdout: "f9532611-4682-49a3-8a0d-3c09e93e4fb8": { 2024-03-18T22:47:33.221 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:47:33.221 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "f9532611-4682-49a3-8a0d-3c09e93e4fb8", 2024-03-18T22:47:33.221 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:47:33.221 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:47:33.221 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:47:33.221 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:47:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.210+0000 7f1e151f4640 1 -- 172.21.15.40:0/393243974 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1df80a2800 msgr2=0x7f1df80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.210+0000 7f1e151f4640 1 --2- 172.21.15.40:0/393243974 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1df80a2800 0x7f1df80a4cc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1e000058c0 tx=0x7f1e00005f90 comp rx=0 tx=0).stop 2024-03-18T22:47:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.210+0000 7f1e151f4640 1 -- 172.21.15.40:0/393243974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e10102a40 msgr2=0x7f1e10073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:47:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.210+0000 7f1e151f4640 1 --2- 172.21.15.40:0/393243974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e10102a40 0x7f1e10073c10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1e04000c00 tx=0x7f1e04005dc0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.210+0000 7f1e151f4640 1 -- 172.21.15.40:0/393243974 shutdown_connections 2024-03-18T22:47:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.210+0000 7f1e151f4640 1 --2- 172.21.15.40:0/393243974 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1df80a2800 0x7f1df80a4cc0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.210+0000 7f1e151f4640 1 --2- 172.21.15.40:0/393243974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e10103400 0x7f1e10074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.210+0000 7f1e151f4640 1 --2- 172.21.15.40:0/393243974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e10102a40 0x7f1e10073c10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.210+0000 7f1e151f4640 1 --2- 172.21.15.40:0/393243974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e10101850 0x7f1e100755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:47:33.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.210+0000 7f1e151f4640 1 -- 172.21.15.40:0/393243974 >> 172.21.15.40:0/393243974 conn(0x7f1e100fcdc0 msgr2=0x7f1e10105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:47:33.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.210+0000 7f1e151f4640 1 -- 172.21.15.40:0/393243974 shutdown_connections 2024-03-18T22:47:33.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:47:33.210+0000 7f1e151f4640 1 -- 172.21.15.40:0/393243974 wait complete. 2024-03-18T22:47:33.224 INFO:tasks.cephfs.filesystem:scrub status for tag:f9532611-4682-49a3-8a0d-3c09e93e4fb8 - {'path': '/', 'tag': 'f9532611-4682-49a3-8a0d-3c09e93e4fb8', 'options': 'recursive,force'} 2024-03-18T22:47:34.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:34 smithi113 ceph-mon[66775]: pgmap v498: 129 pgs: 129 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:47:34.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:34 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:47:34.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:34 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:34.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:34 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:47:34.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:34 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:47:34.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:34 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:34.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:34 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:47:34.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:34 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:34.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:34 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:34.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:34 smithi103 ceph-mon[66377]: pgmap v498: 129 pgs: 129 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:47:34.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:34 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:47:34.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:34 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:34.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:34 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:47:34.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:34 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:47:34.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:34 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:34.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:34 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:47:34.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:34 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:34.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:34 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:34.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:34 smithi040 ceph-mon[62643]: pgmap v498: 129 pgs: 129 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:47:34.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:34 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:47:34.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:34 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:34.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:34 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:47:34.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:34 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:47:34.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:34 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:34.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:34 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:47:34.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:34 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:34.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:34 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:47:36.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:36 smithi040 ceph-mon[62643]: pgmap v499: 129 pgs: 129 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:47:36.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:36 smithi113 ceph-mon[66775]: pgmap v499: 129 pgs: 129 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:47:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:36 smithi103 ceph-mon[66377]: pgmap v499: 129 pgs: 129 active+clean; 5.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:47:38.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:38 smithi113 ceph-mon[66775]: pgmap v500: 129 pgs: 129 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:47:38.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:38 smithi103 ceph-mon[66377]: pgmap v500: 129 pgs: 129 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:47:38.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:38 smithi040 ceph-mon[62643]: pgmap v500: 129 pgs: 129 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:47:40.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:40 smithi113 ceph-mon[66775]: pgmap v501: 129 pgs: 129 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:47:40.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:40 smithi103 ceph-mon[66377]: pgmap v501: 129 pgs: 129 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:47:40.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:40 smithi040 ceph-mon[62643]: pgmap v501: 129 pgs: 129 active+clean; 5.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:47:42.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:42 smithi113 ceph-mon[66775]: pgmap v502: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:47:42.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:42 smithi103 ceph-mon[66377]: pgmap v502: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:47:42.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:42 smithi040 ceph-mon[62643]: pgmap v502: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:47:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:43 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:47:43.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:43 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:47:43.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:43 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:47:44.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:44 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:47:44.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:44 smithi113 ceph-mon[66775]: pgmap v503: 129 pgs: 129 active+clean; 5.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 33 MiB/s wr, 22 op/s 2024-03-18T22:47:44.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:44 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:47:44.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:44 smithi103 ceph-mon[66377]: pgmap v503: 129 pgs: 129 active+clean; 5.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 33 MiB/s wr, 22 op/s 2024-03-18T22:47:44.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:44 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:47:44.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:44 smithi040 ceph-mon[62643]: pgmap v503: 129 pgs: 129 active+clean; 5.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 33 MiB/s wr, 22 op/s 2024-03-18T22:47:46.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:46 smithi113 ceph-mon[66775]: pgmap v504: 129 pgs: 129 active+clean; 5.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:47:46.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:46 smithi103 ceph-mon[66377]: pgmap v504: 129 pgs: 129 active+clean; 5.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:47:46.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:46 smithi040 ceph-mon[62643]: pgmap v504: 129 pgs: 129 active+clean; 5.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:47:48.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:48 smithi113 ceph-mon[66775]: pgmap v505: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:47:48.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:48 smithi103 ceph-mon[66377]: pgmap v505: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:47:48.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:48 smithi040 ceph-mon[62643]: pgmap v505: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:47:50.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:50 smithi113 ceph-mon[66775]: pgmap v506: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:47:50.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:50 smithi103 ceph-mon[66377]: pgmap v506: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:47:50.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:50 smithi040 ceph-mon[62643]: pgmap v506: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:47:52.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:52 smithi113 ceph-mon[66775]: pgmap v507: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:47:52.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:52 smithi103 ceph-mon[66377]: pgmap v507: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:47:52.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:52 smithi040 ceph-mon[62643]: pgmap v507: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:47:54.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:54 smithi113 ceph-mon[66775]: pgmap v508: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:47:54.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:54 smithi103 ceph-mon[66377]: pgmap v508: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:47:54.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:54 smithi040 ceph-mon[62643]: pgmap v508: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:47:56.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:56 smithi113 ceph-mon[66775]: pgmap v509: 129 pgs: 129 active+clean; 6.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:47:56.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:56 smithi103 ceph-mon[66377]: pgmap v509: 129 pgs: 129 active+clean; 6.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:47:56.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:56 smithi040 ceph-mon[62643]: pgmap v509: 129 pgs: 129 active+clean; 6.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:47:58.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:47:58 smithi113 ceph-mon[66775]: pgmap v510: 129 pgs: 129 active+clean; 6.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:47:58.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:47:58 smithi103 ceph-mon[66377]: pgmap v510: 129 pgs: 129 active+clean; 6.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:47:58.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:47:58 smithi040 ceph-mon[62643]: pgmap v510: 129 pgs: 129 active+clean; 6.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:48:00.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:00 smithi113 ceph-mon[66775]: pgmap v511: 129 pgs: 129 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:48:00.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:00 smithi103 ceph-mon[66377]: pgmap v511: 129 pgs: 129 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:48:00.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:00 smithi040 ceph-mon[62643]: pgmap v511: 129 pgs: 129 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:48:02.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:02 smithi113 ceph-mon[66775]: pgmap v512: 129 pgs: 129 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:48:02.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:02 smithi103 ceph-mon[66377]: pgmap v512: 129 pgs: 129 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:48:02.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:02 smithi040 ceph-mon[62643]: pgmap v512: 129 pgs: 129 active+clean; 6.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:48:03.225 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:48:03.225 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:48:03.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.348+0000 7f459919d640 1 -- 172.21.15.40:0/1562681526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4594100720 msgr2=0x7f4594100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.348+0000 7f459919d640 1 --2- 172.21.15.40:0/1562681526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4594100720 0x7f4594100b40 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f4584009900 tx=0x7f458402bc30 comp rx=0 tx=0).stop 2024-03-18T22:48:03.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.348+0000 7f459919d640 1 -- 172.21.15.40:0/1562681526 shutdown_connections 2024-03-18T22:48:03.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.348+0000 7f459919d640 1 --2- 172.21.15.40:0/1562681526 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f45941022d0 0x7f4594108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.348+0000 7f459919d640 1 --2- 172.21.15.40:0/1562681526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4594101910 0x7f4594101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.348+0000 7f459919d640 1 --2- 172.21.15.40:0/1562681526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4594100720 0x7f4594100b40 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.348+0000 7f459919d640 1 -- 172.21.15.40:0/1562681526 >> 172.21.15.40:0/1562681526 conn(0x7f45940fbcb0 msgr2=0x7f45940fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:03.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.348+0000 7f459919d640 1 -- 172.21.15.40:0/1562681526 shutdown_connections 2024-03-18T22:48:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.348+0000 7f459919d640 1 -- 172.21.15.40:0/1562681526 wait complete. 2024-03-18T22:48:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.349+0000 7f459919d640 1 Processor -- start 2024-03-18T22:48:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.349+0000 7f459919d640 1 -- start start 2024-03-18T22:48:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.349+0000 7f459919d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4594100720 0x7f45941048f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.349+0000 7f459919d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4594101910 0x7f4594102f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.349+0000 7f459919d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45941022d0 0x7f4594103470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.349+0000 7f459919d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f459410c250 con 0x7f45941022d0 2024-03-18T22:48:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.349+0000 7f459919d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f459410c0d0 con 0x7f4594101910 2024-03-18T22:48:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.349+0000 7f459919d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f459410c3d0 con 0x7f4594100720 2024-03-18T22:48:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.349+0000 7f459359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45941022d0 0x7f4594103470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.349+0000 7f459359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45941022d0 0x7f4594103470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46662/0 (socket says 172.21.15.40:46662) 2024-03-18T22:48:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.349+0000 7f459359e640 1 -- 172.21.15.40:0/2932280401 learned_addr learned my addr 172.21.15.40:0/2932280401 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.349+0000 7f459259c640 1 --2- 172.21.15.40:0/2932280401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4594101910 0x7f4594102f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.350+0000 7f459359e640 1 -- 172.21.15.40:0/2932280401 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4594100720 msgr2=0x7f45941048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.350+0000 7f459359e640 1 --2- 172.21.15.40:0/2932280401 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4594100720 0x7f45941048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.350+0000 7f459359e640 1 -- 172.21.15.40:0/2932280401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4594101910 msgr2=0x7f4594102f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.350+0000 7f459359e640 1 --2- 172.21.15.40:0/2932280401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4594101910 0x7f4594102f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.350+0000 7f459359e640 1 -- 172.21.15.40:0/2932280401 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4594103d00 con 0x7f45941022d0 2024-03-18T22:48:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.350+0000 7f459359e640 1 --2- 172.21.15.40:0/2932280401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45941022d0 0x7f4594103470 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f458c00eb10 tx=0x7f458c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.350+0000 7f457bfff640 1 -- 172.21.15.40:0/2932280401 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c004b20 con 0x7f45941022d0 2024-03-18T22:48:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.350+0000 7f457bfff640 1 -- 172.21.15.40:0/2932280401 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f458c004cc0 con 0x7f45941022d0 2024-03-18T22:48:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.350+0000 7f459919d640 1 -- 172.21.15.40:0/2932280401 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45941a1be0 con 0x7f45941022d0 2024-03-18T22:48:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.350+0000 7f457bfff640 1 -- 172.21.15.40:0/2932280401 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c010690 con 0x7f45941022d0 2024-03-18T22:48:03.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.350+0000 7f459919d640 1 -- 172.21.15.40:0/2932280401 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45941a2130 con 0x7f45941022d0 2024-03-18T22:48:03.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.353+0000 7f459bd81740 1 Processor -- start 2024-03-18T22:48:03.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.353+0000 7f457bfff640 1 -- 172.21.15.40:0/2932280401 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f458c005020 con 0x7f45941022d0 2024-03-18T22:48:03.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.354+0000 7f459bd81740 1 -- start start 2024-03-18T22:48:03.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.354+0000 7f459bd81740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8ede70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.354+0000 7f459bd81740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8ee440 0x563eae8f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.354+0000 7f459bd81740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8f35b0 0x563eae8f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.354+0000 7f459bd81740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563eae852500 con 0x563eae8eda50 2024-03-18T22:48:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.354+0000 7f459bd81740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563eae7fabe0 con 0x563eae8ee440 2024-03-18T22:48:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.354+0000 7f459bd81740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563eae84e5a0 con 0x563eae8f35b0 2024-03-18T22:48:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.354+0000 7f459259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8ede70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.354+0000 7f457bfff640 1 --2- 172.21.15.40:0/2932280401 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f457c0a2800 0x7f457c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.354+0000 7f459259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8ede70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46674/0 (socket says 172.21.15.40:46674) 2024-03-18T22:48:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.354+0000 7f459259c640 1 -- 172.21.15.40:0/1600929874 learned_addr learned my addr 172.21.15.40:0/1600929874 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.354+0000 7f459259c640 1 --2- 172.21.15.40:0/1600929874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8f35b0 0x563eae8f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.355+0000 7f457bfff640 1 -- 172.21.15.40:0/2932280401 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f458c0cf650 con 0x7f45941022d0 2024-03-18T22:48:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.355+0000 7f4592d9d640 1 --2- 172.21.15.40:0/1600929874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8ee440 0x563eae8f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.355+0000 7f459259c640 1 -- 172.21.15.40:0/1600929874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8f35b0 msgr2=0x563eae8f5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.355+0000 7f459259c640 1 --2- 172.21.15.40:0/1600929874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8f35b0 0x563eae8f5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.355+0000 7f459259c640 1 -- 172.21.15.40:0/1600929874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8ee440 msgr2=0x563eae8f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.355+0000 7f459259c640 1 --2- 172.21.15.40:0/1600929874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8ee440 0x563eae8f2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.355+0000 7f459259c640 1 -- 172.21.15.40:0/1600929874 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563eae81c000 con 0x563eae8eda50 2024-03-18T22:48:03.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.355+0000 7f4592d9d640 1 --2- 172.21.15.40:0/1600929874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8ee440 0x563eae8f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:48:03.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.355+0000 7f459259c640 1 --2- 172.21.15.40:0/1600929874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8ede70 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f45880137e0 tx=0x7f4588009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2632b3e0fcb4cc85 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:03.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.355+0000 7f459359e640 1 --2- 172.21.15.40:0/2932280401 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f457c0a2800 0x7f457c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.355+0000 7f45797fa640 1 -- 172.21.15.40:0/1600929874 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4588004020 con 0x563eae8eda50 2024-03-18T22:48:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.356+0000 7f45797fa640 1 -- 172.21.15.40:0/1600929874 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4588011c90 con 0x563eae8eda50 2024-03-18T22:48:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.356+0000 7f45797fa640 1 -- 172.21.15.40:0/1600929874 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45880038c0 con 0x563eae8eda50 2024-03-18T22:48:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.356+0000 7f459bd81740 1 -- 172.21.15.40:0/1600929874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 msgr2=0x563eae8ede70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.356+0000 7f459bd81740 1 --2- 172.21.15.40:0/1600929874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8ede70 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f45880137e0 tx=0x7f4588009850 comp rx=0 tx=0).stop 2024-03-18T22:48:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.356+0000 7f459359e640 1 --2- 172.21.15.40:0/2932280401 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f457c0a2800 0x7f457c0a4cc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f458c000f80 tx=0x7f458c00efd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.357+0000 7f459bd81740 1 -- 172.21.15.40:0/1600929874 shutdown_connections 2024-03-18T22:48:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.357+0000 7f459bd81740 1 --2- 172.21.15.40:0/1600929874 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8f35b0 0x563eae8f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.357+0000 7f459bd81740 1 --2- 172.21.15.40:0/1600929874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8ee440 0x563eae8f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.357+0000 7f459bd81740 1 --2- 172.21.15.40:0/1600929874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8ede70 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.357+0000 7f459bd81740 1 -- 172.21.15.40:0/1600929874 >> 172.21.15.40:0/1600929874 conn(0x563eae8e8fc0 msgr2=0x563eae8e93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:03.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.357+0000 7f459bd81740 1 -- 172.21.15.40:0/1600929874 shutdown_connections 2024-03-18T22:48:03.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.357+0000 7f459bd81740 1 -- 172.21.15.40:0/1600929874 wait complete. 2024-03-18T22:48:03.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.357+0000 7f459bd81740 1 -- start start 2024-03-18T22:48:03.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.357+0000 7f459bd81740 1 Processor -- start 2024-03-18T22:48:03.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.357+0000 7f459bd81740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8f7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f459bd81740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8ee440 0x563eae8f83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f459bd81740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8f35b0 0x563eae8ff3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f459bd81740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563eae8f6500 con 0x563eae8eda50 2024-03-18T22:48:03.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f459bd81740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563eae80e160 con 0x563eae8ee440 2024-03-18T22:48:03.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f459bd81740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563eae8f6680 con 0x563eae8f35b0 2024-03-18T22:48:03.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f459bd81740 10 client.0 _ll_register_callbacks cb 0x563eae84aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:03.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f4592d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8ee440 0x563eae8f83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f459259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8f7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f4592d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8ee440 0x563eae8f83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52338/0 (socket says 172.21.15.40:52338) 2024-03-18T22:48:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f459259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8f7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46686/0 (socket says 172.21.15.40:46686) 2024-03-18T22:48:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f4592d9d640 1 -- 172.21.15.40:0/2026720773 learned_addr learned my addr 172.21.15.40:0/2026720773 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f459259c640 1 --2- 172.21.15.40:0/2026720773 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8f35b0 0x563eae8ff3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f4592d9d640 1 -- 172.21.15.40:0/2026720773 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8f35b0 msgr2=0x563eae8ff3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f4592d9d640 1 --2- 172.21.15.40:0/2026720773 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8f35b0 0x563eae8ff3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f4592d9d640 1 -- 172.21.15.40:0/2026720773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 msgr2=0x563eae8f7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f4592d9d640 1 --2- 172.21.15.40:0/2026720773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8f7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f4592d9d640 1 -- 172.21.15.40:0/2026720773 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563eae6307d0 con 0x563eae8ee440 2024-03-18T22:48:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f4592d9d640 1 --2- 172.21.15.40:0/2026720773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8ee440 0x563eae8f83b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4584036040 tx=0x7f4584004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a3662e168556a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.358+0000 7f4576ff5640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:48:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.359+0000 7f4576ff5640 1 -- 172.21.15.40:0/2026720773 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4584043020 con 0x563eae8ee440 2024-03-18T22:48:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.359+0000 7f4576ff5640 1 -- 172.21.15.40:0/2026720773 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4584005040 con 0x563eae8ee440 2024-03-18T22:48:03.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.359+0000 7f459bd81740 1 -- 172.21.15.40:0/2026720773 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4588011c90 con 0x563eae8ee440 2024-03-18T22:48:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.359+0000 7f4576ff5640 1 -- 172.21.15.40:0/2026720773 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4584005d50 con 0x563eae8ee440 2024-03-18T22:48:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.359+0000 7f459bd81740 10 client.26690 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.359+0000 7f459bd81740 1 -- 172.21.15.40:0/2026720773 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563eae5663f0 con 0x563eae8ee440 2024-03-18T22:48:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.359+0000 7f4576ff5640 1 -- 172.21.15.40:0/2026720773 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4588011c90 con 0x563eae8ee440 2024-03-18T22:48:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.359+0000 7f4576ff5640 1 -- 172.21.15.40:0/2026720773 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f4584004bf0 con 0x563eae8ee440 2024-03-18T22:48:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.359+0000 7f459bd81740 10 client.26690 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.359+0000 7f459bd81740 10 client.26690 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.359+0000 7f459bd81740 20 client.26690 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.359+0000 7f459bd81740 1 --2- 172.21.15.40:0/2026720773 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563eae903b70 0x563eae905ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.359+0000 7f459bd81740 4 client.26690 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.359+0000 7f459bd81740 1 -- 172.21.15.40:0/2026720773 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563eae85fa40 con 0x563eae903b70 2024-03-18T22:48:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.360+0000 7f459259c640 1 --2- 172.21.15.40:0/2026720773 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563eae903b70 0x563eae905ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.360+0000 7f4576ff5640 10 client.26690 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.360+0000 7f459259c640 1 --2- 172.21.15.40:0/2026720773 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563eae903b70 0x563eae905ff0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13806ee85be4f58 server_cookie=d9a6ab29b3c767e3 in_seq=0 out_seq=0 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.361+0000 7f4576ff5640 1 -- 172.21.15.40:0/2026720773 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563eae85fa40 con 0x563eae903b70 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.361+0000 7f4576ff5640 10 client.26690 handle_command_reply: tid=0 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.361+0000 7f459bd81740 1 client.26690 shutdown 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.361+0000 7f459bd81740 1 -- 172.21.15.40:0/2026720773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8ee440 msgr2=0x563eae8f83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.361+0000 7f459bd81740 1 --2- 172.21.15.40:0/2026720773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8ee440 0x563eae8f83b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4584036040 tx=0x7f4584004270 comp rx=0 tx=0).stop 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.361+0000 7f459bd81740 1 -- 172.21.15.40:0/2026720773 shutdown_connections 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.361+0000 7f459bd81740 1 --2- 172.21.15.40:0/2026720773 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8f35b0 0x563eae8ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.361+0000 7f459bd81740 1 --2- 172.21.15.40:0/2026720773 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563eae903b70 0x563eae905ff0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.361+0000 7f459bd81740 1 --2- 172.21.15.40:0/2026720773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8ee440 0x563eae8f83b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.361+0000 7f4576ff5640 0 client.26690 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.361+0000 7f459bd81740 1 --2- 172.21.15.40:0/2026720773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8f7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.361+0000 7f459bd81740 1 -- 172.21.15.40:0/2026720773 >> 172.21.15.40:0/2026720773 conn(0x563eae8e8fc0 msgr2=0x563eae8e81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.361+0000 7f459bd81740 1 -- 172.21.15.40:0/2026720773 shutdown_connections 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.361+0000 7f459bd81740 1 -- 172.21.15.40:0/2026720773 wait complete. 2024-03-18T22:48:03.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.361+0000 7f459bd81740 20 client.26690 trim_cache size 0 max 16384 2024-03-18T22:48:03.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.385+0000 7f459bd81740 1 Processor -- start 2024-03-18T22:48:03.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.385+0000 7f459bd81740 1 -- start start 2024-03-18T22:48:03.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.385+0000 7f459bd81740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8eda50 0x563eae8eecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.385+0000 7f459bd81740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8ee440 0x563eae8ff3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.385+0000 7f459bd81740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8f35b0 0x563eae8e7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.385+0000 7f459bd81740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f45941a4040 con 0x563eae8ee440 2024-03-18T22:48:03.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.385+0000 7f459bd81740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f45941a5b80 con 0x563eae8f35b0 2024-03-18T22:48:03.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.385+0000 7f459bd81740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f45941a4710 con 0x563eae8eda50 2024-03-18T22:48:03.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.385+0000 7f4592d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8ee440 0x563eae8ff3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.385+0000 7f459259c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8eda50 0x563eae8eecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.385+0000 7f4592d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8ee440 0x563eae8ff3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46702/0 (socket says 172.21.15.40:46702) 2024-03-18T22:48:03.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.385+0000 7f4592d9d640 1 -- 172.21.15.40:0/366835645 learned_addr learned my addr 172.21.15.40:0/366835645 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:03.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.385+0000 7f459259c640 1 --2- 172.21.15.40:0/366835645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8f35b0 0x563eae8e7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.385+0000 7f459259c640 1 -- 172.21.15.40:0/366835645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8f35b0 msgr2=0x563eae8e7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.386+0000 7f459259c640 1 --2- 172.21.15.40:0/366835645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8f35b0 0x563eae8e7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.386+0000 7f459259c640 1 -- 172.21.15.40:0/366835645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8ee440 msgr2=0x563eae8ff3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.386+0000 7f459259c640 1 --2- 172.21.15.40:0/366835645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8ee440 0x563eae8ff3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.386+0000 7f459259c640 1 -- 172.21.15.40:0/366835645 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563eae5260f0 con 0x563eae8eda50 2024-03-18T22:48:03.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.386+0000 7f4592d9d640 1 --2- 172.21.15.40:0/366835645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8ee440 0x563eae8ff3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:03.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.386+0000 7f459259c640 1 --2- 172.21.15.40:0/366835645 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8eda50 0x563eae8eecd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4588009fd0 tx=0x7f4588009c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1563afa3b1b20900 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:03.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.386+0000 7f4578ff9640 1 -- 172.21.15.40:0/366835645 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4588029070 con 0x563eae8eda50 2024-03-18T22:48:03.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.386+0000 7f4578ff9640 1 -- 172.21.15.40:0/366835645 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f458802a040 con 0x563eae8eda50 2024-03-18T22:48:03.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.386+0000 7f4578ff9640 1 -- 172.21.15.40:0/366835645 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4588005320 con 0x563eae8eda50 2024-03-18T22:48:03.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.387+0000 7f459bd81740 1 -- 172.21.15.40:0/366835645 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8eda50 msgr2=0x563eae8eecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.387+0000 7f459bd81740 1 --2- 172.21.15.40:0/366835645 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8eda50 0x563eae8eecd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4588009fd0 tx=0x7f4588009c70 comp rx=0 tx=0).stop 2024-03-18T22:48:03.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.387+0000 7f459bd81740 1 -- 172.21.15.40:0/366835645 shutdown_connections 2024-03-18T22:48:03.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.387+0000 7f459bd81740 1 --2- 172.21.15.40:0/366835645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8f35b0 0x563eae8e7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.387+0000 7f459bd81740 1 --2- 172.21.15.40:0/366835645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8ee440 0x563eae8ff3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.387+0000 7f459bd81740 1 --2- 172.21.15.40:0/366835645 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8eda50 0x563eae8eecd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.387+0000 7f459bd81740 1 -- 172.21.15.40:0/366835645 >> 172.21.15.40:0/366835645 conn(0x563eae8e8fc0 msgr2=0x563eae8ef8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:03.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.387+0000 7f459bd81740 1 -- 172.21.15.40:0/366835645 shutdown_connections 2024-03-18T22:48:03.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.387+0000 7f459bd81740 1 -- 172.21.15.40:0/366835645 wait complete. 2024-03-18T22:48:03.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.387+0000 7f459bd81740 1 -- start start 2024-03-18T22:48:03.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.387+0000 7f459bd81740 1 Processor -- start 2024-03-18T22:48:03.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459bd81740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8fcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459bd81740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8ee440 0x563eae8fcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459bd81740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8f35b0 0x563eae8fdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459bd81740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f45941a54b0 con 0x563eae8eda50 2024-03-18T22:48:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459bd81740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f45941a4de0 con 0x563eae8f35b0 2024-03-18T22:48:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459bd81740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4594073710 con 0x563eae8ee440 2024-03-18T22:48:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459bd81740 10 client.0 _ll_register_callbacks cb 0x7f4588005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8fcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8fcaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46706/0 (socket says 172.21.15.40:46706) 2024-03-18T22:48:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459259c640 1 -- 172.21.15.40:0/4149797260 learned_addr learned my addr 172.21.15.40:0/4149797260 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f4592d9d640 1 --2- 172.21.15.40:0/4149797260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8ee440 0x563eae8fcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459259c640 1 --2- 172.21.15.40:0/4149797260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8f35b0 0x563eae8fdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459259c640 1 -- 172.21.15.40:0/4149797260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8ee440 msgr2=0x563eae8fcfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459259c640 1 --2- 172.21.15.40:0/4149797260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8ee440 0x563eae8fcfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459259c640 1 -- 172.21.15.40:0/4149797260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8f35b0 msgr2=0x563eae8fdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459259c640 1 --2- 172.21.15.40:0/4149797260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8f35b0 0x563eae8fdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459259c640 1 -- 172.21.15.40:0/4149797260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563eae8fe590 con 0x563eae8eda50 2024-03-18T22:48:03.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459259c640 1 --2- 172.21.15.40:0/4149797260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8fcaa0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f45880134c0 tx=0x7f4588009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c0cdb83b035a80 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f4576ff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:48:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f4576ff5640 1 -- 172.21.15.40:0/4149797260 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4588029040 con 0x563eae8eda50 2024-03-18T22:48:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f4576ff5640 1 -- 172.21.15.40:0/4149797260 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4588032600 con 0x563eae8eda50 2024-03-18T22:48:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.388+0000 7f459bd81740 1 -- 172.21.15.40:0/4149797260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f458802a040 con 0x563eae8eda50 2024-03-18T22:48:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.389+0000 7f4576ff5640 1 -- 172.21.15.40:0/4149797260 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4588048050 con 0x563eae8eda50 2024-03-18T22:48:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.389+0000 7f459bd81740 10 client.16872 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.389+0000 7f459bd81740 1 -- 172.21.15.40:0/4149797260 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563eae969f30 con 0x563eae8eda50 2024-03-18T22:48:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.389+0000 7f4576ff5640 1 -- 172.21.15.40:0/4149797260 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f458802a040 con 0x563eae8eda50 2024-03-18T22:48:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.389+0000 7f4576ff5640 1 -- 172.21.15.40:0/4149797260 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f45880049e0 con 0x563eae8eda50 2024-03-18T22:48:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.389+0000 7f459bd81740 10 client.16872 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.389+0000 7f459bd81740 10 client.16872 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.389+0000 7f459bd81740 20 client.16872 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.389+0000 7f459bd81740 1 --2- 172.21.15.40:0/4149797260 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563eae96a630 0x563eae96ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.389+0000 7f459bd81740 4 client.16872 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:48:03.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.389+0000 7f459bd81740 1 -- 172.21.15.40:0/4149797260 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563eae859d90 con 0x563eae96a630 2024-03-18T22:48:03.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.389+0000 7f4592d9d640 1 --2- 172.21.15.40:0/4149797260 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563eae96a630 0x563eae96ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f4576ff5640 10 client.16872 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:03.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f4592d9d640 1 --2- 172.21.15.40:0/4149797260 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563eae96a630 0x563eae96ca50 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbc6ad1b12063c51 server_cookie=5c59016a3d754d9c in_seq=0 out_seq=0 2024-03-18T22:48:03.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f4576ff5640 1 -- 172.21.15.40:0/4149797260 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563eae859d90 con 0x563eae96a630 2024-03-18T22:48:03.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f4576ff5640 10 client.16872 handle_command_reply: tid=0 2024-03-18T22:48:03.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f459bd81740 1 client.16872 shutdown 2024-03-18T22:48:03.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f459bd81740 1 -- 172.21.15.40:0/4149797260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 msgr2=0x563eae8fcaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f459bd81740 1 --2- 172.21.15.40:0/4149797260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8fcaa0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f45880134c0 tx=0x7f4588009ec0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f459bd81740 1 -- 172.21.15.40:0/4149797260 shutdown_connections 2024-03-18T22:48:03.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f459bd81740 1 --2- 172.21.15.40:0/4149797260 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563eae96a630 0x563eae96ca50 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f459bd81740 1 --2- 172.21.15.40:0/4149797260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563eae8f35b0 0x563eae8fdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f459bd81740 1 --2- 172.21.15.40:0/4149797260 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563eae8ee440 0x563eae8fcfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f4576ff5640 0 client.16872 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:03.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f459bd81740 1 --2- 172.21.15.40:0/4149797260 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563eae8eda50 0x563eae8fcaa0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f459bd81740 1 -- 172.21.15.40:0/4149797260 >> 172.21.15.40:0/4149797260 conn(0x563eae8e8fc0 msgr2=0x563eae8ec220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:03.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f459bd81740 1 -- 172.21.15.40:0/4149797260 shutdown_connections 2024-03-18T22:48:03.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f459bd81740 1 -- 172.21.15.40:0/4149797260 wait complete. 2024-03-18T22:48:03.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.390+0000 7f459bd81740 20 client.16872 trim_cache size 0 max 16384 2024-03-18T22:48:03.398 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:48:03.398 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:48:03.398 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:48:03.398 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:48:03.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.391+0000 7f459919d640 1 -- 172.21.15.40:0/2932280401 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f457c0a2800 msgr2=0x7f457c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.391+0000 7f459919d640 1 --2- 172.21.15.40:0/2932280401 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f457c0a2800 0x7f457c0a4cc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f458c000f80 tx=0x7f458c00efd0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.391+0000 7f459919d640 1 -- 172.21.15.40:0/2932280401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45941022d0 msgr2=0x7f4594103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.391+0000 7f459919d640 1 --2- 172.21.15.40:0/2932280401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45941022d0 0x7f4594103470 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f458c00eb10 tx=0x7f458c00c2d0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.391+0000 7f459919d640 1 -- 172.21.15.40:0/2932280401 shutdown_connections 2024-03-18T22:48:03.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.391+0000 7f459919d640 1 --2- 172.21.15.40:0/2932280401 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f457c0a2800 0x7f457c0a4cc0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.391+0000 7f459919d640 1 --2- 172.21.15.40:0/2932280401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45941022d0 0x7f4594103470 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.391+0000 7f459919d640 1 --2- 172.21.15.40:0/2932280401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4594101910 0x7f4594102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.391+0000 7f459919d640 1 --2- 172.21.15.40:0/2932280401 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4594100720 0x7f45941048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.391+0000 7f459919d640 1 -- 172.21.15.40:0/2932280401 >> 172.21.15.40:0/2932280401 conn(0x7f45940fbcb0 msgr2=0x7f4594068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:03.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.391+0000 7f459919d640 1 -- 172.21.15.40:0/2932280401 shutdown_connections 2024-03-18T22:48:03.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.391+0000 7f459919d640 1 -- 172.21.15.40:0/2932280401 wait complete. 2024-03-18T22:48:03.403 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:48:03.403 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:48:03.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.515+0000 7fceef0b2640 1 -- 172.21.15.40:0/4044637065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcee8101930 msgr2=0x7fcee8101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.515+0000 7fceef0b2640 1 --2- 172.21.15.40:0/4044637065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcee8101930 0x7fcee8101db0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fced4009900 tx=0x7fced402bc30 comp rx=0 tx=0).stop 2024-03-18T22:48:03.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.515+0000 7fceef0b2640 1 -- 172.21.15.40:0/4044637065 shutdown_connections 2024-03-18T22:48:03.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.515+0000 7fceef0b2640 1 --2- 172.21.15.40:0/4044637065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee81022f0 0x7fcee8108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.515+0000 7fceef0b2640 1 --2- 172.21.15.40:0/4044637065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcee8101930 0x7fcee8101db0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.515+0000 7fceef0b2640 1 --2- 172.21.15.40:0/4044637065 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcee8100740 0x7fcee8100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.515+0000 7fceef0b2640 1 -- 172.21.15.40:0/4044637065 >> 172.21.15.40:0/4044637065 conn(0x7fcee80fbcb0 msgr2=0x7fcee80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:03.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.515+0000 7fceef0b2640 1 -- 172.21.15.40:0/4044637065 shutdown_connections 2024-03-18T22:48:03.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.515+0000 7fceef0b2640 1 -- 172.21.15.40:0/4044637065 wait complete. 2024-03-18T22:48:03.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.516+0000 7fceef0b2640 1 Processor -- start 2024-03-18T22:48:03.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.516+0000 7fceef0b2640 1 -- start start 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fceef0b2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcee8100740 0x7fcee80694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fceef0b2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcee8101930 0x7fcee819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fceef0b2640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee81022f0 0x7fcee81a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fceef0b2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcee810c250 con 0x7fcee8101930 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fceef0b2640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcee810c0d0 con 0x7fcee81022f0 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fceef0b2640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fcee810c3d0 con 0x7fcee8100740 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fcee7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcee8101930 0x7fcee819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fceece4e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcee8100740 0x7fcee80694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fceece4e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcee8100740 0x7fcee80694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:47912/0 (socket says 172.21.15.40:47912) 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fceece4e640 1 -- 172.21.15.40:0/2898329856 learned_addr learned my addr 172.21.15.40:0/2898329856 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fceece4e640 1 -- 172.21.15.40:0/2898329856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee81022f0 msgr2=0x7fcee81a1110 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fceece4e640 1 --2- 172.21.15.40:0/2898329856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee81022f0 0x7fcee81a1110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fceece4e640 1 -- 172.21.15.40:0/2898329856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcee8101930 msgr2=0x7fcee819ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fceece4e640 1 --2- 172.21.15.40:0/2898329856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcee8101930 0x7fcee819ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fceece4e640 1 -- 172.21.15.40:0/2898329856 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee81a1900 con 0x7fcee8100740 2024-03-18T22:48:03.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.517+0000 7fcee7fff640 1 --2- 172.21.15.40:0/2898329856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcee8101930 0x7fcee819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:03.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.518+0000 7fceece4e640 1 --2- 172.21.15.40:0/2898329856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcee8100740 0x7fcee80694f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcedc00e9c0 tx=0x7fcedc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:03.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.518+0000 7fcee5ffb640 1 -- 172.21.15.40:0/2898329856 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc00cde0 con 0x7fcee8100740 2024-03-18T22:48:03.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.518+0000 7fcee5ffb640 1 -- 172.21.15.40:0/2898329856 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcedc0108e0 con 0x7fcee8100740 2024-03-18T22:48:03.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.518+0000 7fceef0b2640 1 -- 172.21.15.40:0/2898329856 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcee81a1b90 con 0x7fcee8100740 2024-03-18T22:48:03.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.518+0000 7fcee5ffb640 1 -- 172.21.15.40:0/2898329856 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcedc018990 con 0x7fcee8100740 2024-03-18T22:48:03.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.518+0000 7fceef0b2640 1 -- 172.21.15.40:0/2898329856 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcee81a2100 con 0x7fcee8100740 2024-03-18T22:48:03.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.520+0000 7fceef0b2640 1 -- 172.21.15.40:0/2898329856 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcee8107b30 con 0x7fcee8100740 2024-03-18T22:48:03.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.521+0000 7fcee5ffb640 1 -- 172.21.15.40:0/2898329856 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcedc01f020 con 0x7fcee8100740 2024-03-18T22:48:03.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.522+0000 7fcee5ffb640 1 --2- 172.21.15.40:0/2898329856 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fced80a28b0 0x7fced80a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.522+0000 7fcee5ffb640 1 -- 172.21.15.40:0/2898329856 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fcedc0cf050 con 0x7fcee8100740 2024-03-18T22:48:03.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.522+0000 7fcee7fff640 1 --2- 172.21.15.40:0/2898329856 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fced80a28b0 0x7fced80a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.523+0000 7fcee7fff640 1 --2- 172.21.15.40:0/2898329856 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fced80a28b0 0x7fced80a4d70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fced4009230 tx=0x7fced4005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:03.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.526+0000 7fcee5ffb640 1 -- 172.21.15.40:0/2898329856 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcedc09bd00 con 0x7fcee8100740 2024-03-18T22:48:03.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.793+0000 7fceef0b2640 1 -- 172.21.15.40:0/2898329856 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcee8062420 con 0x7fcee8100740 2024-03-18T22:48:03.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.793+0000 7fcee5ffb640 1 -- 172.21.15.40:0/2898329856 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11947 (secure 0 0 0) 0x7fcedc09b600 con 0x7fcee8100740 2024-03-18T22:48:03.795 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:48:03.795 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:46:52.724919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24388,24401,24473,24481,24509]},"id":1}]} 2024-03-18T22:48:03.795 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 32 2024-03-18T22:48:03.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.796+0000 7fceef0b2640 1 -- 172.21.15.40:0/2898329856 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fced80a28b0 msgr2=0x7fced80a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.796+0000 7fceef0b2640 1 --2- 172.21.15.40:0/2898329856 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fced80a28b0 0x7fced80a4d70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fced4009230 tx=0x7fced4005fd0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.796+0000 7fceef0b2640 1 -- 172.21.15.40:0/2898329856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcee8100740 msgr2=0x7fcee80694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.796+0000 7fceef0b2640 1 --2- 172.21.15.40:0/2898329856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcee8100740 0x7fcee80694f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcedc00e9c0 tx=0x7fcedc00ee80 comp rx=0 tx=0).stop 2024-03-18T22:48:03.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.796+0000 7fceef0b2640 1 -- 172.21.15.40:0/2898329856 shutdown_connections 2024-03-18T22:48:03.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.796+0000 7fceef0b2640 1 --2- 172.21.15.40:0/2898329856 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fced80a28b0 0x7fced80a4d70 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.796+0000 7fceef0b2640 1 --2- 172.21.15.40:0/2898329856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee81022f0 0x7fcee81a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.796+0000 7fceef0b2640 1 --2- 172.21.15.40:0/2898329856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcee8101930 0x7fcee819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.796+0000 7fceef0b2640 1 --2- 172.21.15.40:0/2898329856 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcee8100740 0x7fcee80694f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.796+0000 7fceef0b2640 1 -- 172.21.15.40:0/2898329856 >> 172.21.15.40:0/2898329856 conn(0x7fcee80fbcb0 msgr2=0x7fcee8104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.796+0000 7fceef0b2640 1 -- 172.21.15.40:0/2898329856 shutdown_connections 2024-03-18T22:48:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.796+0000 7fceef0b2640 1 -- 172.21.15.40:0/2898329856 wait complete. 2024-03-18T22:48:03.811 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:48:03.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.923+0000 7f0a0d577640 1 -- 172.21.15.40:0/2760051053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a08102aa0 msgr2=0x7f0a08102f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.923+0000 7f0a0d577640 1 --2- 172.21.15.40:0/2760051053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a08102aa0 0x7f0a08102f20 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f09fc009930 tx=0x7f09fc02bc30 comp rx=0 tx=0).stop 2024-03-18T22:48:03.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.923+0000 7f0a0d577640 1 -- 172.21.15.40:0/2760051053 shutdown_connections 2024-03-18T22:48:03.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.923+0000 7f0a0d577640 1 --2- 172.21.15.40:0/2760051053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a08103460 0x7f0a08109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.923+0000 7f0a0d577640 1 --2- 172.21.15.40:0/2760051053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a08102aa0 0x7f0a08102f20 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.923+0000 7f0a0d577640 1 --2- 172.21.15.40:0/2760051053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a080ffad0 0x7f0a080ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.923+0000 7f0a0d577640 1 -- 172.21.15.40:0/2760051053 >> 172.21.15.40:0/2760051053 conn(0x7f0a080fb400 msgr2=0x7f0a080fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:03.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.923+0000 7f0a0d577640 1 -- 172.21.15.40:0/2760051053 shutdown_connections 2024-03-18T22:48:03.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.923+0000 7f0a0d577640 1 -- 172.21.15.40:0/2760051053 wait complete. 2024-03-18T22:48:03.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.923+0000 7f0a0d577640 1 Processor -- start 2024-03-18T22:48:03.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a0d577640 1 -- start start 2024-03-18T22:48:03.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a0d577640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a080ffad0 0x7f0a08197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a0d577640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a08102aa0 0x7f0a08197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a0d577640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a08103460 0x7f0a0819f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a0d577640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0810d360 con 0x7f0a08102aa0 2024-03-18T22:48:03.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a0d577640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0810d1e0 con 0x7f0a08103460 2024-03-18T22:48:03.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a0d577640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0810d4e0 con 0x7f0a080ffad0 2024-03-18T22:48:03.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a067fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a08102aa0 0x7f0a08197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a067fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a08102aa0 0x7f0a08197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46734/0 (socket says 172.21.15.40:46734) 2024-03-18T22:48:03.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a067fc640 1 -- 172.21.15.40:0/4053522444 learned_addr learned my addr 172.21.15.40:0/4053522444 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:03.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a06ffd640 1 --2- 172.21.15.40:0/4053522444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a080ffad0 0x7f0a08197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a077fe640 1 --2- 172.21.15.40:0/4053522444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a08103460 0x7f0a0819f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a067fc640 1 -- 172.21.15.40:0/4053522444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a080ffad0 msgr2=0x7f0a08197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a067fc640 1 --2- 172.21.15.40:0/4053522444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a080ffad0 0x7f0a08197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a067fc640 1 -- 172.21.15.40:0/4053522444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a08103460 msgr2=0x7f0a0819f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a067fc640 1 --2- 172.21.15.40:0/4053522444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a08103460 0x7f0a0819f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a067fc640 1 -- 172.21.15.40:0/4053522444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a0819f6b0 con 0x7f0a08102aa0 2024-03-18T22:48:03.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.924+0000 7f0a067fc640 1 --2- 172.21.15.40:0/4053522444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a08102aa0 0x7f0a08197fd0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f09fc009230 tx=0x7f09fc005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:03.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.925+0000 7f09effff640 1 -- 172.21.15.40:0/4053522444 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09fc043020 con 0x7f0a08102aa0 2024-03-18T22:48:03.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.925+0000 7f09effff640 1 -- 172.21.15.40:0/4053522444 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09fc0043d0 con 0x7f0a08102aa0 2024-03-18T22:48:03.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.925+0000 7f0a0d577640 1 -- 172.21.15.40:0/4053522444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a0819f940 con 0x7f0a08102aa0 2024-03-18T22:48:03.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.925+0000 7f09effff640 1 -- 172.21.15.40:0/4053522444 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09fc003af0 con 0x7f0a08102aa0 2024-03-18T22:48:03.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.925+0000 7f0a0d577640 1 -- 172.21.15.40:0/4053522444 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a0819fe90 con 0x7f0a08102aa0 2024-03-18T22:48:03.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.927+0000 7f09effff640 1 -- 172.21.15.40:0/4053522444 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f09fc046020 con 0x7f0a08102aa0 2024-03-18T22:48:03.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.927+0000 7f0a10163740 1 Processor -- start 2024-03-18T22:48:03.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.927+0000 7f0a10163740 1 -- start start 2024-03-18T22:48:03.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.927+0000 7f09effff640 1 --2- 172.21.15.40:0/4053522444 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09f00a2800 0x7f09f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.927+0000 7f0a10163740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd72f0 0x562864bd7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a10163740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd7ce0 0x562864bdc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a10163740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bdf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a10163740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562864b3b4b0 con 0x562864bd7ce0 2024-03-18T22:48:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a10163740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562864ae3d70 con 0x562864bdce50 2024-03-18T22:48:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a10163740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x562864b37550 con 0x562864bd72f0 2024-03-18T22:48:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a077fe640 1 --2- 172.21.15.40:0/4053522444 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09f00a2800 0x7f09f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f09effff640 1 -- 172.21.15.40:0/4053522444 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f09fc0f1030 con 0x7f0a08102aa0 2024-03-18T22:48:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a067fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd7ce0 0x562864bdc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a067fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd7ce0 0x562864bdc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46736/0 (socket says 172.21.15.40:46736) 2024-03-18T22:48:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a06ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd72f0 0x562864bd7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a067fc640 1 -- 172.21.15.40:0/3819516522 learned_addr learned my addr 172.21.15.40:0/3819516522 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a077fe640 1 --2- 172.21.15.40:0/3819516522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bdf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a06ffd640 1 -- 172.21.15.40:0/3819516522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 msgr2=0x562864bdf2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a06ffd640 1 --2- 172.21.15.40:0/3819516522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bdf2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a06ffd640 1 -- 172.21.15.40:0/3819516522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd7ce0 msgr2=0x562864bdc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a06ffd640 1 --2- 172.21.15.40:0/3819516522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd7ce0 0x562864bdc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a06ffd640 1 -- 172.21.15.40:0/3819516522 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562864b06210 con 0x562864bd72f0 2024-03-18T22:48:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a067fc640 1 --2- 172.21.15.40:0/3819516522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd7ce0 0x562864bdc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a077fe640 1 --2- 172.21.15.40:0/4053522444 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09f00a2800 0x7f09f00a4cc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0a00005fd0 tx=0x7f0a00011bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f0a06ffd640 1 --2- 172.21.15.40:0/3819516522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd72f0 0x562864bd7710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009bd0 tx=0x7f09f8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6d3917d9c9e0f80 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.928+0000 7f09ed7fa640 1 -- 172.21.15.40:0/3819516522 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f8004220 con 0x562864bd72f0 2024-03-18T22:48:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.929+0000 7f09ed7fa640 1 -- 172.21.15.40:0/3819516522 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09f8009e40 con 0x562864bd72f0 2024-03-18T22:48:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.929+0000 7f09ed7fa640 1 -- 172.21.15.40:0/3819516522 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f80185f0 con 0x562864bd72f0 2024-03-18T22:48:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.929+0000 7f0a10163740 1 -- 172.21.15.40:0/3819516522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd72f0 msgr2=0x562864bd7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.929+0000 7f0a10163740 1 --2- 172.21.15.40:0/3819516522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd72f0 0x562864bd7710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009bd0 tx=0x7f09f8011810 comp rx=0 tx=0).stop 2024-03-18T22:48:03.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.929+0000 7f0a10163740 1 -- 172.21.15.40:0/3819516522 shutdown_connections 2024-03-18T22:48:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.929+0000 7f0a10163740 1 --2- 172.21.15.40:0/3819516522 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bdf2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.929+0000 7f0a10163740 1 --2- 172.21.15.40:0/3819516522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd7ce0 0x562864bdc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.929+0000 7f0a10163740 1 --2- 172.21.15.40:0/3819516522 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd72f0 0x562864bd7710 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.929+0000 7f0a10163740 1 -- 172.21.15.40:0/3819516522 >> 172.21.15.40:0/3819516522 conn(0x562864bd2860 msgr2=0x562864bd2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.929+0000 7f0a10163740 1 -- 172.21.15.40:0/3819516522 shutdown_connections 2024-03-18T22:48:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.929+0000 7f0a10163740 1 -- 172.21.15.40:0/3819516522 wait complete. 2024-03-18T22:48:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a10163740 1 -- start start 2024-03-18T22:48:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a10163740 1 Processor -- start 2024-03-18T22:48:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a10163740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd72f0 0x562864be5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a10163740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd7ce0 0x562864be60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a10163740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bed110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a10163740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562864be4210 con 0x562864bd72f0 2024-03-18T22:48:03.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a10163740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562864af7310 con 0x562864bdce50 2024-03-18T22:48:03.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a10163740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x562864be4390 con 0x562864bd7ce0 2024-03-18T22:48:03.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a10163740 10 client.0 _ll_register_callbacks cb 0x562864aeb2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:03.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a06ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd72f0 0x562864be5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a06ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd72f0 0x562864be5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46752/0 (socket says 172.21.15.40:46752) 2024-03-18T22:48:03.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a06ffd640 1 -- 172.21.15.40:0/2895676602 learned_addr learned my addr 172.21.15.40:0/2895676602 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:03.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a067fc640 1 --2- 172.21.15.40:0/2895676602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd7ce0 0x562864be60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a077fe640 1 --2- 172.21.15.40:0/2895676602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bed110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a06ffd640 1 -- 172.21.15.40:0/2895676602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd7ce0 msgr2=0x562864be60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a06ffd640 1 --2- 172.21.15.40:0/2895676602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd7ce0 0x562864be60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a06ffd640 1 -- 172.21.15.40:0/2895676602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 msgr2=0x562864bed110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a06ffd640 1 --2- 172.21.15.40:0/2895676602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bed110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.930+0000 7f0a06ffd640 1 -- 172.21.15.40:0/2895676602 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56286484f3f0 con 0x562864bd72f0 2024-03-18T22:48:03.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f0a06ffd640 1 --2- 172.21.15.40:0/2895676602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd72f0 0x562864be5b90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f09f8005270 tx=0x7f09f80052a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bfcb6872874da80 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:03.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f09eaff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:48:03.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f09eaff5640 1 -- 172.21.15.40:0/2895676602 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f8027020 con 0x562864bd72f0 2024-03-18T22:48:03.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f0a10163740 1 -- 172.21.15.40:0/2895676602 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562864b993e0 con 0x562864bd72f0 2024-03-18T22:48:03.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f09eaff5640 1 -- 172.21.15.40:0/2895676602 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f09f80035f0 con 0x562864bd72f0 2024-03-18T22:48:03.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f09eaff5640 1 -- 172.21.15.40:0/2895676602 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f09f8032050 con 0x562864bd72f0 2024-03-18T22:48:03.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f0a10163740 10 client.16890 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:03.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f0a10163740 1 -- 172.21.15.40:0/2895676602 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5628649197d0 con 0x562864bd72f0 2024-03-18T22:48:03.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f09eaff5640 1 -- 172.21.15.40:0/2895676602 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x562864b993e0 con 0x562864bd72f0 2024-03-18T22:48:03.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f09eaff5640 1 -- 172.21.15.40:0/2895676602 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f09f8004c90 con 0x562864bd72f0 2024-03-18T22:48:03.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f0a10163740 10 client.16890 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:03.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f0a10163740 10 client.16890 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:48:03.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f0a10163740 20 client.16890 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:03.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f0a10163740 1 --2- 172.21.15.40:0/2895676602 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x562864bf1770 0x562864bf3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f0a10163740 4 client.16890 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:03.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.931+0000 7f0a10163740 1 -- 172.21.15.40:0/2895676602 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562864b489f0 con 0x562864bf1770 2024-03-18T22:48:03.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.932+0000 7f0a067fc640 1 --2- 172.21.15.40:0/2895676602 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x562864bf1770 0x562864bf3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.932+0000 7f09eaff5640 10 client.16890 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:48:03.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.932+0000 7f0a067fc640 1 --2- 172.21.15.40:0/2895676602 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x562864bf1770 0x562864bf3be0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=97aae88ccc69c6e0 server_cookie=55a31aaab7cc94df in_seq=0 out_seq=0 2024-03-18T22:48:03.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.933+0000 7f09eaff5640 1 -- 172.21.15.40:0/2895676602 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562864b489f0 con 0x562864bf1770 2024-03-18T22:48:03.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.933+0000 7f09eaff5640 10 client.16890 handle_command_reply: tid=0 2024-03-18T22:48:03.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.933+0000 7f0a10163740 1 client.16890 shutdown 2024-03-18T22:48:03.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.933+0000 7f0a10163740 1 -- 172.21.15.40:0/2895676602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd72f0 msgr2=0x562864be5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.933+0000 7f0a10163740 1 --2- 172.21.15.40:0/2895676602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd72f0 0x562864be5b90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f09f8005270 tx=0x7f09f80052a0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.933+0000 7f0a10163740 1 -- 172.21.15.40:0/2895676602 shutdown_connections 2024-03-18T22:48:03.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.933+0000 7f0a10163740 1 --2- 172.21.15.40:0/2895676602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bed110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.933+0000 7f0a10163740 1 --2- 172.21.15.40:0/2895676602 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd7ce0 0x562864be60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.933+0000 7f0a10163740 1 --2- 172.21.15.40:0/2895676602 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x562864bf1770 0x562864bf3be0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.933+0000 7f0a10163740 1 --2- 172.21.15.40:0/2895676602 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd72f0 0x562864be5b90 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.933+0000 7f0a10163740 1 -- 172.21.15.40:0/2895676602 >> 172.21.15.40:0/2895676602 conn(0x562864bd2860 msgr2=0x562864bd1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:03.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.933+0000 7f09eaff5640 0 client.16890 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:48:03.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.933+0000 7f0a10163740 1 -- 172.21.15.40:0/2895676602 shutdown_connections 2024-03-18T22:48:03.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.933+0000 7f0a10163740 1 -- 172.21.15.40:0/2895676602 wait complete. 2024-03-18T22:48:03.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.933+0000 7f0a10163740 20 client.16890 trim_cache size 0 max 16384 2024-03-18T22:48:03.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.948+0000 7f0a10163740 1 Processor -- start 2024-03-18T22:48:03.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.948+0000 7f0a10163740 1 -- start start 2024-03-18T22:48:03.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.948+0000 7f0a10163740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd72f0 0x562864bd1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.948+0000 7f0a10163740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd7ce0 0x562864bed0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.948+0000 7f0a10163740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bd1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.948+0000 7f0a10163740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0a081a8270 con 0x562864bd72f0 2024-03-18T22:48:03.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.948+0000 7f0a10163740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0a081a9db0 con 0x562864bdce50 2024-03-18T22:48:03.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.948+0000 7f0a10163740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0a081a8940 con 0x562864bd7ce0 2024-03-18T22:48:03.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.948+0000 7f0a06ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd72f0 0x562864bd1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.948+0000 7f0a06ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd72f0 0x562864bd1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46758/0 (socket says 172.21.15.40:46758) 2024-03-18T22:48:03.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.948+0000 7f0a06ffd640 1 -- 172.21.15.40:0/1058170776 learned_addr learned my addr 172.21.15.40:0/1058170776 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:03.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.948+0000 7f0a067fc640 1 --2- 172.21.15.40:0/1058170776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd7ce0 0x562864bed0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.948+0000 7f0a077fe640 1 --2- 172.21.15.40:0/1058170776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bd1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.949+0000 7f0a06ffd640 1 -- 172.21.15.40:0/1058170776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd7ce0 msgr2=0x562864bed0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.949+0000 7f0a06ffd640 1 --2- 172.21.15.40:0/1058170776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd7ce0 0x562864bed0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.949+0000 7f0a06ffd640 1 -- 172.21.15.40:0/1058170776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 msgr2=0x562864bd1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.949+0000 7f0a06ffd640 1 --2- 172.21.15.40:0/1058170776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bd1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.949+0000 7f0a06ffd640 1 -- 172.21.15.40:0/1058170776 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562864bf3f50 con 0x562864bd72f0 2024-03-18T22:48:03.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.949+0000 7f0a067fc640 1 --2- 172.21.15.40:0/1058170776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd7ce0 0x562864bed0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:03.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.949+0000 7f0a06ffd640 1 --2- 172.21.15.40:0/1058170776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd72f0 0x562864bd1310 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f09f8011d10 tx=0x7f09f8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33455ebd51c035a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:03.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.949+0000 7f09ecff9640 1 -- 172.21.15.40:0/1058170776 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f8027040 con 0x562864bd72f0 2024-03-18T22:48:03.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.949+0000 7f09ecff9640 1 -- 172.21.15.40:0/1058170776 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09f80035f0 con 0x562864bd72f0 2024-03-18T22:48:03.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.949+0000 7f09ecff9640 1 -- 172.21.15.40:0/1058170776 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f801e040 con 0x562864bd72f0 2024-03-18T22:48:03.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 -- 172.21.15.40:0/1058170776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd72f0 msgr2=0x562864bd1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 --2- 172.21.15.40:0/1058170776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd72f0 0x562864bd1310 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f09f8011d10 tx=0x7f09f8005fd0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 -- 172.21.15.40:0/1058170776 shutdown_connections 2024-03-18T22:48:03.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 --2- 172.21.15.40:0/1058170776 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bd1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 --2- 172.21.15.40:0/1058170776 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd7ce0 0x562864bed0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 --2- 172.21.15.40:0/1058170776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd72f0 0x562864bd1310 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 -- 172.21.15.40:0/1058170776 >> 172.21.15.40:0/1058170776 conn(0x562864bd2860 msgr2=0x562864bd91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:03.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 -- 172.21.15.40:0/1058170776 shutdown_connections 2024-03-18T22:48:03.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 -- 172.21.15.40:0/1058170776 wait complete. 2024-03-18T22:48:03.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 -- start start 2024-03-18T22:48:03.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 Processor -- start 2024-03-18T22:48:03.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd72f0 0x562864bed0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd7ce0 0x562864be9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bea190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0a081a96e0 con 0x562864bd7ce0 2024-03-18T22:48:03.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0a081a9010 con 0x562864bdce50 2024-03-18T22:48:03.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.950+0000 7f0a10163740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0a081a6250 con 0x562864bd72f0 2024-03-18T22:48:03.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f0a10163740 10 client.0 _ll_register_callbacks cb 0x7f09fc004580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:03.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f0a067fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd7ce0 0x562864be9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f0a067fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd7ce0 0x562864be9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46760/0 (socket says 172.21.15.40:46760) 2024-03-18T22:48:03.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f0a06ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd72f0 0x562864bed0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f0a067fc640 1 -- 172.21.15.40:0/770563784 learned_addr learned my addr 172.21.15.40:0/770563784 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:03.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f0a077fe640 1 --2- 172.21.15.40:0/770563784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bea190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f0a067fc640 1 -- 172.21.15.40:0/770563784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd72f0 msgr2=0x562864bed0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f0a067fc640 1 --2- 172.21.15.40:0/770563784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd72f0 0x562864bed0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f0a067fc640 1 -- 172.21.15.40:0/770563784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 msgr2=0x562864bea190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:03.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f0a067fc640 1 --2- 172.21.15.40:0/770563784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bea190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f0a067fc640 1 -- 172.21.15.40:0/770563784 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562864c55bc0 con 0x562864bd7ce0 2024-03-18T22:48:03.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f0a067fc640 1 --2- 172.21.15.40:0/770563784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd7ce0 0x562864be9bb0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f09fc0b9a40 tx=0x7f09fc02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8557a8c166b3df1b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:03.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f09eaff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:48:03.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f09eaff5640 1 -- 172.21.15.40:0/770563784 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09fc0c1730 con 0x562864bd7ce0 2024-03-18T22:48:03.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f09eaff5640 1 -- 172.21.15.40:0/770563784 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f09fc0a6c90 con 0x562864bd7ce0 2024-03-18T22:48:03.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f0a10163740 1 -- 172.21.15.40:0/770563784 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09f80035f0 con 0x562864bd7ce0 2024-03-18T22:48:03.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f09eaff5640 1 -- 172.21.15.40:0/770563784 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f09fc0c5d80 con 0x562864bd7ce0 2024-03-18T22:48:03.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f0a10163740 10 client.16896 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:03.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.951+0000 7f0a10163740 1 -- 172.21.15.40:0/770563784 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562864c56ed0 con 0x562864bd7ce0 2024-03-18T22:48:03.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.952+0000 7f09eaff5640 1 -- 172.21.15.40:0/770563784 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09f80035f0 con 0x562864bd7ce0 2024-03-18T22:48:03.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.952+0000 7f09eaff5640 1 -- 172.21.15.40:0/770563784 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f09fc0a9040 con 0x562864bd7ce0 2024-03-18T22:48:03.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.952+0000 7f0a10163740 10 client.16896 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:03.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.952+0000 7f0a10163740 10 client.16896 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:48:03.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.952+0000 7f0a10163740 20 client.16896 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:03.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.952+0000 7f0a10163740 1 --2- 172.21.15.40:0/770563784 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x562864c578b0 0x562864c59cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:03.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.952+0000 7f0a10163740 4 client.16896 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:48:03.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.952+0000 7f0a10163740 1 -- 172.21.15.40:0/770563784 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562864b42d40 con 0x562864c578b0 2024-03-18T22:48:03.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.952+0000 7f0a06ffd640 1 --2- 172.21.15.40:0/770563784 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x562864c578b0 0x562864c59cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:03.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.952+0000 7f09eaff5640 10 client.16896 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:48:03.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.952+0000 7f0a06ffd640 1 --2- 172.21.15.40:0/770563784 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x562864c578b0 0x562864c59cd0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5ff7208533152020 server_cookie=d1a220360f5c5921 in_seq=0 out_seq=0 2024-03-18T22:48:03.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f09eaff5640 1 -- 172.21.15.40:0/770563784 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562864b42d40 con 0x562864c578b0 2024-03-18T22:48:03.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f09eaff5640 10 client.16896 handle_command_reply: tid=0 2024-03-18T22:48:03.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a10163740 1 client.16896 shutdown 2024-03-18T22:48:03.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a10163740 1 -- 172.21.15.40:0/770563784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd7ce0 msgr2=0x562864be9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a10163740 1 --2- 172.21.15.40:0/770563784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd7ce0 0x562864be9bb0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f09fc0b9a40 tx=0x7f09fc02bc30 comp rx=0 tx=0).stop 2024-03-18T22:48:03.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a10163740 1 -- 172.21.15.40:0/770563784 shutdown_connections 2024-03-18T22:48:03.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a10163740 1 --2- 172.21.15.40:0/770563784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562864bdce50 0x562864bea190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a10163740 1 --2- 172.21.15.40:0/770563784 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x562864c578b0 0x562864c59cd0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f09eaff5640 0 client.16896 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:48:03.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a10163740 1 --2- 172.21.15.40:0/770563784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562864bd7ce0 0x562864be9bb0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a10163740 1 --2- 172.21.15.40:0/770563784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562864bd72f0 0x562864bed0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a10163740 1 -- 172.21.15.40:0/770563784 >> 172.21.15.40:0/770563784 conn(0x562864bd2860 msgr2=0x562864bd6350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:03.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a10163740 1 -- 172.21.15.40:0/770563784 shutdown_connections 2024-03-18T22:48:03.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a10163740 1 -- 172.21.15.40:0/770563784 wait complete. 2024-03-18T22:48:03.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a10163740 20 client.16896 trim_cache size 0 max 16384 2024-03-18T22:48:03.963 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:48:03.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a0d577640 1 -- 172.21.15.40:0/4053522444 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09f00a2800 msgr2=0x7f09f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a0d577640 1 --2- 172.21.15.40:0/4053522444 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09f00a2800 0x7f09f00a4cc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0a00005fd0 tx=0x7f0a00011bc0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a0d577640 1 -- 172.21.15.40:0/4053522444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a08102aa0 msgr2=0x7f0a08197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:03.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.953+0000 7f0a0d577640 1 --2- 172.21.15.40:0/4053522444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a08102aa0 0x7f0a08197fd0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f09fc009230 tx=0x7f09fc005e40 comp rx=0 tx=0).stop 2024-03-18T22:48:03.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.954+0000 7f0a0d577640 1 -- 172.21.15.40:0/4053522444 shutdown_connections 2024-03-18T22:48:03.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.954+0000 7f0a0d577640 1 --2- 172.21.15.40:0/4053522444 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f09f00a2800 0x7f09f00a4cc0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.954+0000 7f0a0d577640 1 --2- 172.21.15.40:0/4053522444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0a08103460 0x7f0a0819f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.954+0000 7f0a0d577640 1 --2- 172.21.15.40:0/4053522444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a08102aa0 0x7f0a08197fd0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.954+0000 7f0a0d577640 1 --2- 172.21.15.40:0/4053522444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0a080ffad0 0x7f0a08197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:03.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.954+0000 7f0a0d577640 1 -- 172.21.15.40:0/4053522444 >> 172.21.15.40:0/4053522444 conn(0x7f0a080fb400 msgr2=0x7f0a08105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:03.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.954+0000 7f0a0d577640 1 -- 172.21.15.40:0/4053522444 shutdown_connections 2024-03-18T22:48:03.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:03.954+0000 7f0a0d577640 1 -- 172.21.15.40:0/4053522444 wait complete. 2024-03-18T22:48:03.966 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:48:04.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.084+0000 7f2d5a643640 1 -- 172.21.15.40:0/209645154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d54101910 msgr2=0x7f2d54101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.084+0000 7f2d5a643640 1 --2- 172.21.15.40:0/209645154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d54101910 0x7f2d54101d90 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009900 tx=0x7f2d4802bc30 comp rx=0 tx=0).stop 2024-03-18T22:48:04.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.084+0000 7f2d5a643640 1 -- 172.21.15.40:0/209645154 shutdown_connections 2024-03-18T22:48:04.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.084+0000 7f2d5a643640 1 --2- 172.21.15.40:0/209645154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d541022d0 0x7f2d54108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.084+0000 7f2d5a643640 1 --2- 172.21.15.40:0/209645154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d54101910 0x7f2d54101d90 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.084+0000 7f2d5a643640 1 --2- 172.21.15.40:0/209645154 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d54100720 0x7f2d54100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.084+0000 7f2d5a643640 1 -- 172.21.15.40:0/209645154 >> 172.21.15.40:0/209645154 conn(0x7f2d540fbc90 msgr2=0x7f2d540fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.084+0000 7f2d5a643640 1 -- 172.21.15.40:0/209645154 shutdown_connections 2024-03-18T22:48:04.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.084+0000 7f2d5a643640 1 -- 172.21.15.40:0/209645154 wait complete. 2024-03-18T22:48:04.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d5a643640 1 Processor -- start 2024-03-18T22:48:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d5a643640 1 -- start start 2024-03-18T22:48:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d5a643640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d54100720 0x7f2d54196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d5a643640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d54101910 0x7f2d54196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d5a643640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d541022d0 0x7f2d5419df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d5a643640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5410c250 con 0x7f2d54100720 2024-03-18T22:48:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d5a643640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5410c0d0 con 0x7f2d541022d0 2024-03-18T22:48:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d5a643640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5410c3d0 con 0x7f2d54101910 2024-03-18T22:48:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d53fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d54100720 0x7f2d54196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d53fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d54100720 0x7f2d54196950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46776/0 (socket says 172.21.15.40:46776) 2024-03-18T22:48:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d53fff640 1 -- 172.21.15.40:0/3438476571 learned_addr learned my addr 172.21.15.40:0/3438476571 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d537fe640 1 --2- 172.21.15.40:0/3438476571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d54101910 0x7f2d54196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d58be0640 1 --2- 172.21.15.40:0/3438476571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d541022d0 0x7f2d5419df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d53fff640 1 -- 172.21.15.40:0/3438476571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d54101910 msgr2=0x7f2d54196e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d53fff640 1 --2- 172.21.15.40:0/3438476571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d54101910 0x7f2d54196e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d53fff640 1 -- 172.21.15.40:0/3438476571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d541022d0 msgr2=0x7f2d5419df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d53fff640 1 --2- 172.21.15.40:0/3438476571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d541022d0 0x7f2d5419df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.085+0000 7f2d53fff640 1 -- 172.21.15.40:0/3438476571 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d5419e570 con 0x7f2d54100720 2024-03-18T22:48:04.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.086+0000 7f2d53fff640 1 --2- 172.21.15.40:0/3438476571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d54100720 0x7f2d54196950 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400e9c0 tx=0x7f2d4400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.086+0000 7f2d517fa640 1 -- 172.21.15.40:0/3438476571 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4400cd80 con 0x7f2d54100720 2024-03-18T22:48:04.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.086+0000 7f2d5a643640 1 -- 172.21.15.40:0/3438476571 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d5419e860 con 0x7f2d54100720 2024-03-18T22:48:04.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.086+0000 7f2d517fa640 1 -- 172.21.15.40:0/3438476571 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d440035f0 con 0x7f2d54100720 2024-03-18T22:48:04.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.086+0000 7f2d5a643640 1 -- 172.21.15.40:0/3438476571 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d5419ed80 con 0x7f2d54100720 2024-03-18T22:48:04.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.086+0000 7f2d517fa640 1 -- 172.21.15.40:0/3438476571 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44010690 con 0x7f2d54100720 2024-03-18T22:48:04.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.088+0000 7f2d517fa640 1 -- 172.21.15.40:0/3438476571 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2d440037a0 con 0x7f2d54100720 2024-03-18T22:48:04.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.088+0000 7f2d5d22a740 1 Processor -- start 2024-03-18T22:48:04.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d5d22a740 1 -- start start 2024-03-18T22:48:04.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d5d22a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4500 0x556c1edf4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d5d22a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4ef0 0x556c1edf99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d5d22a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edfa060 0x556c1edfc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d5d22a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556c1ed564b0 con 0x556c1edfa060 2024-03-18T22:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d5d22a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556c1ecfed70 con 0x556c1edf4500 2024-03-18T22:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d5d22a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556c1ed52550 con 0x556c1edf4ef0 2024-03-18T22:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d517fa640 1 --2- 172.21.15.40:0/3438476571 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d3c0a2800 0x7f2d3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edfa060 0x556c1edfc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d53fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4ef0 0x556c1edf99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edfa060 0x556c1edfc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46782/0 (socket says 172.21.15.40:46782) 2024-03-18T22:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d537fe640 1 -- 172.21.15.40:0/2104317886 learned_addr learned my addr 172.21.15.40:0/2104317886 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d58be0640 1 --2- 172.21.15.40:0/2104317886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4500 0x556c1edf4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d58be0640 1 --2- 172.21.15.40:0/3438476571 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d3c0a2800 0x7f2d3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d517fa640 1 -- 172.21.15.40:0/3438476571 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2d440cf330 con 0x7f2d54100720 2024-03-18T22:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d537fe640 1 -- 172.21.15.40:0/2104317886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4ef0 msgr2=0x556c1edf99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d537fe640 1 --2- 172.21.15.40:0/2104317886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4ef0 0x556c1edf99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d537fe640 1 -- 172.21.15.40:0/2104317886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4500 msgr2=0x556c1edf4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d537fe640 1 --2- 172.21.15.40:0/2104317886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4500 0x556c1edf4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.089+0000 7f2d537fe640 1 -- 172.21.15.40:0/2104317886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c1ed21210 con 0x556c1edfa060 2024-03-18T22:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.090+0000 7f2d537fe640 1 --2- 172.21.15.40:0/2104317886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edfa060 0x556c1edfc4f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009900 tx=0x7f2d48009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e019843dd636e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.090+0000 7f2d58be0640 1 --2- 172.21.15.40:0/3438476571 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d3c0a2800 0x7f2d3c0a4cc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x556c1edf5750 tx=0x7f2d4c009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.090+0000 7f2d3a7fc640 1 -- 172.21.15.40:0/2104317886 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48045020 con 0x556c1edfa060 2024-03-18T22:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.090+0000 7f2d3a7fc640 1 -- 172.21.15.40:0/2104317886 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d48004040 con 0x556c1edfa060 2024-03-18T22:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.090+0000 7f2d3a7fc640 1 -- 172.21.15.40:0/2104317886 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48005460 con 0x556c1edfa060 2024-03-18T22:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.090+0000 7f2d5d22a740 1 -- 172.21.15.40:0/2104317886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edfa060 msgr2=0x556c1edfc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.090+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/2104317886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edfa060 0x556c1edfc4f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009900 tx=0x7f2d48009560 comp rx=0 tx=0).stop 2024-03-18T22:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.090+0000 7f2d5d22a740 1 -- 172.21.15.40:0/2104317886 shutdown_connections 2024-03-18T22:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.090+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/2104317886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edfa060 0x556c1edfc4f0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.090+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/2104317886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4ef0 0x556c1edf99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.090+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/2104317886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4500 0x556c1edf4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.090+0000 7f2d5d22a740 1 -- 172.21.15.40:0/2104317886 >> 172.21.15.40:0/2104317886 conn(0x556c1eded860 msgr2=0x556c1ededc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.090+0000 7f2d5d22a740 1 -- 172.21.15.40:0/2104317886 shutdown_connections 2024-03-18T22:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.091+0000 7f2d5d22a740 1 -- 172.21.15.40:0/2104317886 wait complete. 2024-03-18T22:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.091+0000 7f2d5d22a740 1 -- start start 2024-03-18T22:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.091+0000 7f2d5d22a740 1 Processor -- start 2024-03-18T22:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.091+0000 7f2d5d22a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4500 0x556c1edfefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.091+0000 7f2d5d22a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edf4ef0 0x556c1ee01520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.091+0000 7f2d5d22a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edfa060 0x556c1ee03a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.091+0000 7f2d5d22a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556c1edf2150 con 0x556c1edf4ef0 2024-03-18T22:48:04.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.091+0000 7f2d5d22a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556c1ed12310 con 0x556c1edf4500 2024-03-18T22:48:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.091+0000 7f2d5d22a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556c1edf22d0 con 0x556c1edfa060 2024-03-18T22:48:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.091+0000 7f2d5d22a740 10 client.0 _ll_register_callbacks cb 0x556c1ed062d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.091+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edf4ef0 0x556c1ee01520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.091+0000 7f2d53fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4500 0x556c1edfefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.091+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edf4ef0 0x556c1ee01520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46792/0 (socket says 172.21.15.40:46792) 2024-03-18T22:48:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.091+0000 7f2d537fe640 1 -- 172.21.15.40:0/488634328 learned_addr learned my addr 172.21.15.40:0/488634328 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.091+0000 7f2d58be0640 1 --2- 172.21.15.40:0/488634328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edfa060 0x556c1ee03a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.092+0000 7f2d53fff640 1 -- 172.21.15.40:0/488634328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edfa060 msgr2=0x556c1ee03a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.092+0000 7f2d53fff640 1 --2- 172.21.15.40:0/488634328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edfa060 0x556c1ee03a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.092+0000 7f2d53fff640 1 -- 172.21.15.40:0/488634328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edf4ef0 msgr2=0x556c1ee01520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.092+0000 7f2d53fff640 1 --2- 172.21.15.40:0/488634328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edf4ef0 0x556c1ee01520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.092+0000 7f2d53fff640 1 -- 172.21.15.40:0/488634328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c1ea6a3f0 con 0x556c1edf4500 2024-03-18T22:48:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.092+0000 7f2d53fff640 1 --2- 172.21.15.40:0/488634328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4500 0x556c1edfefd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2d44097d40 tx=0x7f2d4406b400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=632f9f478a474f27 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.092+0000 7f2d337fe640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:48:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.092+0000 7f2d337fe640 1 -- 172.21.15.40:0/488634328 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4409bfe0 con 0x556c1edf4500 2024-03-18T22:48:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.092+0000 7f2d337fe640 1 -- 172.21.15.40:0/488634328 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2d44014230 con 0x556c1edf4500 2024-03-18T22:48:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.092+0000 7f2d5d22a740 1 -- 172.21.15.40:0/488634328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d48004040 con 0x556c1edf4500 2024-03-18T22:48:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.092+0000 7f2d337fe640 1 -- 172.21.15.40:0/488634328 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2d44014430 con 0x556c1edf4500 2024-03-18T22:48:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.092+0000 7f2d5d22a740 10 client.26729 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.092+0000 7f2d5d22a740 1 -- 172.21.15.40:0/488634328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556c1eb347d0 con 0x556c1edf4500 2024-03-18T22:48:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.092+0000 7f2d337fe640 1 -- 172.21.15.40:0/488634328 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d48004040 con 0x556c1edf4500 2024-03-18T22:48:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.093+0000 7f2d337fe640 1 -- 172.21.15.40:0/488634328 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2d4407a250 con 0x556c1edf4500 2024-03-18T22:48:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.093+0000 7f2d5d22a740 10 client.26729 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.093+0000 7f2d5d22a740 10 client.26729 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:48:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.093+0000 7f2d5d22a740 20 client.26729 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.093+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/488634328 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x556c1ee08600 0x556c1ee0aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.093+0000 7f2d5d22a740 4 client.26729 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:04.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.093+0000 7f2d5d22a740 1 -- 172.21.15.40:0/488634328 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c1edb43e0 con 0x556c1ee08600 2024-03-18T22:48:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.093+0000 7f2d537fe640 1 --2- 172.21.15.40:0/488634328 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x556c1ee08600 0x556c1ee0aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.093+0000 7f2d537fe640 1 --2- 172.21.15.40:0/488634328 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x556c1ee08600 0x556c1ee0aa90 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8bcbecb6c3162b31 server_cookie=11301f921811297c in_seq=0 out_seq=0 2024-03-18T22:48:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.093+0000 7f2d337fe640 10 client.26729 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:48:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.094+0000 7f2d337fe640 1 -- 172.21.15.40:0/488634328 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556c1edb43e0 con 0x556c1ee08600 2024-03-18T22:48:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.094+0000 7f2d337fe640 10 client.26729 handle_command_reply: tid=0 2024-03-18T22:48:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.094+0000 7f2d5d22a740 1 client.26729 shutdown 2024-03-18T22:48:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.095+0000 7f2d5d22a740 1 -- 172.21.15.40:0/488634328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4500 msgr2=0x556c1edfefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.095+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/488634328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4500 0x556c1edfefd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2d44097d40 tx=0x7f2d4406b400 comp rx=0 tx=0).stop 2024-03-18T22:48:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.095+0000 7f2d5d22a740 1 -- 172.21.15.40:0/488634328 shutdown_connections 2024-03-18T22:48:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.095+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/488634328 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x556c1ee08600 0x556c1ee0aa90 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.095+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/488634328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edfa060 0x556c1ee03a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.095+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/488634328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edf4ef0 0x556c1ee01520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.095+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/488634328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4500 0x556c1edfefd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.095+0000 7f2d337fe640 0 client.26729 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:48:04.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.095+0000 7f2d5d22a740 1 -- 172.21.15.40:0/488634328 >> 172.21.15.40:0/488634328 conn(0x556c1eded860 msgr2=0x556c1edecba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.095+0000 7f2d5d22a740 1 -- 172.21.15.40:0/488634328 shutdown_connections 2024-03-18T22:48:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.095+0000 7f2d5d22a740 1 -- 172.21.15.40:0/488634328 wait complete. 2024-03-18T22:48:04.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.095+0000 7f2d5d22a740 20 client.26729 trim_cache size 0 max 16384 2024-03-18T22:48:04.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.114+0000 7f2d5d22a740 1 Processor -- start 2024-03-18T22:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.114+0000 7f2d5d22a740 1 -- start start 2024-03-18T22:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.114+0000 7f2d5d22a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edf4500 0x556c1ee014a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.114+0000 7f2d5d22a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4ef0 0x556c1edeeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.114+0000 7f2d5d22a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edfa060 0x556c1edeeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.114+0000 7f2d5d22a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2d541a8320 con 0x556c1edf4500 2024-03-18T22:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.114+0000 7f2d5d22a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d541a9e60 con 0x556c1edfa060 2024-03-18T22:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.114+0000 7f2d5d22a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2d541a89f0 con 0x556c1edf4ef0 2024-03-18T22:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.114+0000 7f2d53fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edf4500 0x556c1ee014a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.114+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4ef0 0x556c1edeeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.114+0000 7f2d53fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edf4500 0x556c1ee014a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46796/0 (socket says 172.21.15.40:46796) 2024-03-18T22:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.114+0000 7f2d58be0640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edfa060 0x556c1edeeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.114+0000 7f2d53fff640 1 -- 172.21.15.40:0/2377989569 learned_addr learned my addr 172.21.15.40:0/2377989569 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.115+0000 7f2d537fe640 1 -- 172.21.15.40:0/2377989569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edfa060 msgr2=0x556c1edeeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.115+0000 7f2d537fe640 1 --2- 172.21.15.40:0/2377989569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edfa060 0x556c1edeeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.115+0000 7f2d537fe640 1 -- 172.21.15.40:0/2377989569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edf4500 msgr2=0x556c1ee014a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.115+0000 7f2d537fe640 1 --2- 172.21.15.40:0/2377989569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edf4500 0x556c1ee014a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.115+0000 7f2d537fe640 1 -- 172.21.15.40:0/2377989569 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c1edf52d0 con 0x556c1edf4ef0 2024-03-18T22:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.115+0000 7f2d53fff640 1 --2- 172.21.15.40:0/2377989569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edf4500 0x556c1ee014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.115+0000 7f2d537fe640 1 --2- 172.21.15.40:0/2377989569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4ef0 0x556c1edeeab0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x556c1ee0b2d0 tx=0x7f2d48000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21df6d989fab5da0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.115+0000 7f2d39ffb640 1 -- 172.21.15.40:0/2377989569 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48045040 con 0x556c1edf4ef0 2024-03-18T22:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.115+0000 7f2d39ffb640 1 -- 172.21.15.40:0/2377989569 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d4802cb60 con 0x556c1edf4ef0 2024-03-18T22:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.115+0000 7f2d39ffb640 1 -- 172.21.15.40:0/2377989569 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d48004ab0 con 0x556c1edf4ef0 2024-03-18T22:48:04.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.116+0000 7f2d5d22a740 1 -- 172.21.15.40:0/2377989569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4ef0 msgr2=0x556c1edeeab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.116+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/2377989569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4ef0 0x556c1edeeab0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x556c1ee0b2d0 tx=0x7f2d48000bc0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.116+0000 7f2d5d22a740 1 -- 172.21.15.40:0/2377989569 shutdown_connections 2024-03-18T22:48:04.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.116+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/2377989569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edfa060 0x556c1edeeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.116+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/2377989569 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4ef0 0x556c1edeeab0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.116+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/2377989569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edf4500 0x556c1ee014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.116+0000 7f2d5d22a740 1 -- 172.21.15.40:0/2377989569 >> 172.21.15.40:0/2377989569 conn(0x556c1eded860 msgr2=0x556c1ee00d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.116+0000 7f2d5d22a740 1 -- 172.21.15.40:0/2377989569 shutdown_connections 2024-03-18T22:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.116+0000 7f2d5d22a740 1 -- 172.21.15.40:0/2377989569 wait complete. 2024-03-18T22:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.116+0000 7f2d5d22a740 1 -- start start 2024-03-18T22:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.116+0000 7f2d5d22a740 1 Processor -- start 2024-03-18T22:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d5d22a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4500 0x556c1ee19b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d5d22a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4ef0 0x556c1ee1a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d5d22a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edfa060 0x556c1ee1b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d5d22a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2d541a9790 con 0x556c1edfa060 2024-03-18T22:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d5d22a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d541a90c0 con 0x556c1edf4ef0 2024-03-18T22:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d5d22a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2d541a6300 con 0x556c1edf4500 2024-03-18T22:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d5d22a740 10 client.0 _ll_register_callbacks cb 0x7f2d44097b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4ef0 0x556c1ee1a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d53fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4500 0x556c1ee19b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d58be0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edfa060 0x556c1ee1b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4ef0 0x556c1ee1a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52444/0 (socket says 172.21.15.40:52444) 2024-03-18T22:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d53fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4500 0x556c1ee19b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:48018/0 (socket says 172.21.15.40:48018) 2024-03-18T22:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d537fe640 1 -- 172.21.15.40:0/354786015 learned_addr learned my addr 172.21.15.40:0/354786015 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d58be0640 1 -- 172.21.15.40:0/354786015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4500 msgr2=0x556c1ee19b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d58be0640 1 --2- 172.21.15.40:0/354786015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4500 0x556c1ee19b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d58be0640 1 -- 172.21.15.40:0/354786015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4ef0 msgr2=0x556c1ee1a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d58be0640 1 --2- 172.21.15.40:0/354786015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4ef0 0x556c1ee1a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d58be0640 1 -- 172.21.15.40:0/354786015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c1ee1b630 con 0x556c1edfa060 2024-03-18T22:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d537fe640 1 --2- 172.21.15.40:0/354786015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4ef0 0x556c1ee1a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d53fff640 1 --2- 172.21.15.40:0/354786015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4500 0x556c1ee19b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d58be0640 1 --2- 172.21.15.40:0/354786015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edfa060 0x556c1ee1b040 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c009270 tx=0x7f2d4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d3e2ba8045058ae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d337fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.117+0000 7f2d337fe640 1 -- 172.21.15.40:0/354786015 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c00fe60 con 0x556c1edfa060 2024-03-18T22:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.118+0000 7f2d337fe640 1 -- 172.21.15.40:0/354786015 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2d4c004b20 con 0x556c1edfa060 2024-03-18T22:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.118+0000 7f2d5d22a740 1 -- 172.21.15.40:0/354786015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d4802cb60 con 0x556c1edfa060 2024-03-18T22:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.118+0000 7f2d337fe640 1 -- 172.21.15.40:0/354786015 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2d4c004d20 con 0x556c1edfa060 2024-03-18T22:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.118+0000 7f2d5d22a740 10 client.16914 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.118+0000 7f2d5d22a740 1 -- 172.21.15.40:0/354786015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556c1ee6ec70 con 0x556c1edfa060 2024-03-18T22:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.118+0000 7f2d337fe640 1 -- 172.21.15.40:0/354786015 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d4802cb60 con 0x556c1edfa060 2024-03-18T22:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.118+0000 7f2d337fe640 1 -- 172.21.15.40:0/354786015 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f2d4c02c070 con 0x556c1edfa060 2024-03-18T22:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.118+0000 7f2d5d22a740 10 client.16914 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.118+0000 7f2d5d22a740 10 client.16914 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.118+0000 7f2d5d22a740 20 client.16914 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.118+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/354786015 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x556c1ee6f6f0 0x556c1ee71b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.118+0000 7f2d5d22a740 4 client.16914 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.118+0000 7f2d5d22a740 1 -- 172.21.15.40:0/354786015 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c1ed639f0 con 0x556c1ee6f6f0 2024-03-18T22:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.118+0000 7f2d53fff640 1 --2- 172.21.15.40:0/354786015 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x556c1ee6f6f0 0x556c1ee71b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d337fe640 10 client.16914 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d53fff640 1 --2- 172.21.15.40:0/354786015 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x556c1ee6f6f0 0x556c1ee71b10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2b8bdf59a9ba0c56 server_cookie=b5b772d9e6cf889a in_seq=0 out_seq=0 2024-03-18T22:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d337fe640 1 -- 172.21.15.40:0/354786015 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c1ed639f0 con 0x556c1ee6f6f0 2024-03-18T22:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d337fe640 10 client.16914 handle_command_reply: tid=0 2024-03-18T22:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d5d22a740 1 client.16914 shutdown 2024-03-18T22:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d5d22a740 1 -- 172.21.15.40:0/354786015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edfa060 msgr2=0x556c1ee1b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/354786015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edfa060 0x556c1ee1b040 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c009270 tx=0x7f2d4c004a40 comp rx=0 tx=0).stop 2024-03-18T22:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d5d22a740 1 -- 172.21.15.40:0/354786015 shutdown_connections 2024-03-18T22:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/354786015 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x556c1ee6f6f0 0x556c1ee71b10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/354786015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c1edfa060 0x556c1ee1b040 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/354786015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c1edf4ef0 0x556c1ee1a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d337fe640 0 client.16914 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d5d22a740 1 --2- 172.21.15.40:0/354786015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c1edf4500 0x556c1ee19b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d5d22a740 1 -- 172.21.15.40:0/354786015 >> 172.21.15.40:0/354786015 conn(0x556c1eded860 msgr2=0x556c1edec310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d5d22a740 1 -- 172.21.15.40:0/354786015 shutdown_connections 2024-03-18T22:48:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d5d22a740 1 -- 172.21.15.40:0/354786015 wait complete. 2024-03-18T22:48:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.119+0000 7f2d5d22a740 20 client.16914 trim_cache size 0 max 16384 2024-03-18T22:48:04.127 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:48:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.120+0000 7f2d5a643640 1 -- 172.21.15.40:0/3438476571 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d3c0a2800 msgr2=0x7f2d3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.120+0000 7f2d5a643640 1 --2- 172.21.15.40:0/3438476571 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d3c0a2800 0x7f2d3c0a4cc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x556c1edf5750 tx=0x7f2d4c009500 comp rx=0 tx=0).stop 2024-03-18T22:48:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.120+0000 7f2d5a643640 1 -- 172.21.15.40:0/3438476571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d54100720 msgr2=0x7f2d54196950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.120+0000 7f2d5a643640 1 --2- 172.21.15.40:0/3438476571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d54100720 0x7f2d54196950 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400e9c0 tx=0x7f2d4400ee80 comp rx=0 tx=0).stop 2024-03-18T22:48:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.120+0000 7f2d5a643640 1 -- 172.21.15.40:0/3438476571 shutdown_connections 2024-03-18T22:48:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.120+0000 7f2d5a643640 1 --2- 172.21.15.40:0/3438476571 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d3c0a2800 0x7f2d3c0a4cc0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.120+0000 7f2d5a643640 1 --2- 172.21.15.40:0/3438476571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d541022d0 0x7f2d5419df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.120+0000 7f2d5a643640 1 --2- 172.21.15.40:0/3438476571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d54101910 0x7f2d54196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.120+0000 7f2d5a643640 1 --2- 172.21.15.40:0/3438476571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d54100720 0x7f2d54196950 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.120+0000 7f2d5a643640 1 -- 172.21.15.40:0/3438476571 >> 172.21.15.40:0/3438476571 conn(0x7f2d540fbc90 msgr2=0x7f2d54104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.120+0000 7f2d5a643640 1 -- 172.21.15.40:0/3438476571 shutdown_connections 2024-03-18T22:48:04.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.120+0000 7f2d5a643640 1 -- 172.21.15.40:0/3438476571 wait complete. 2024-03-18T22:48:04.134 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:48:04.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.254+0000 7f8eca027640 1 -- 172.21.15.40:0/1764309646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec4102300 msgr2=0x7f8ec41046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.254+0000 7f8eca027640 1 --2- 172.21.15.40:0/1764309646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec4102300 0x7f8ec41046f0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4009900 tx=0x7f8eb402bc30 comp rx=0 tx=0).stop 2024-03-18T22:48:04.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.254+0000 7f8eca027640 1 -- 172.21.15.40:0/1764309646 shutdown_connections 2024-03-18T22:48:04.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.254+0000 7f8eca027640 1 --2- 172.21.15.40:0/1764309646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ec4107560 0x7f8ec4109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.254+0000 7f8eca027640 1 --2- 172.21.15.40:0/1764309646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ec4104c30 0x7f8ec4107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.254+0000 7f8eca027640 1 --2- 172.21.15.40:0/1764309646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec4102300 0x7f8ec41046f0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.254+0000 7f8eca027640 1 -- 172.21.15.40:0/1764309646 >> 172.21.15.40:0/1764309646 conn(0x7f8ec40fbc90 msgr2=0x7f8ec40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.254+0000 7f8eca027640 1 -- 172.21.15.40:0/1764309646 shutdown_connections 2024-03-18T22:48:04.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.254+0000 7f8eca027640 1 -- 172.21.15.40:0/1764309646 wait complete. 2024-03-18T22:48:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8eca027640 1 Processor -- start 2024-03-18T22:48:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8eca027640 1 -- start start 2024-03-18T22:48:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8eca027640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ec4102300 0x7f8ec4197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8eca027640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec4104c30 0x7f8ec4197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8eca027640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ec4107560 0x7f8ec419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8eca027640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec410d370 con 0x7f8ec4104c30 2024-03-18T22:48:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8eca027640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec410d1f0 con 0x7f8ec4102300 2024-03-18T22:48:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8eca027640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec410d4f0 con 0x7f8ec4107560 2024-03-18T22:48:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8ec3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ec4107560 0x7f8ec419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8ec37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ec4102300 0x7f8ec4197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8ec2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec4104c30 0x7f8ec4197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8ec3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ec4107560 0x7f8ec419f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:48034/0 (socket says 172.21.15.40:48034) 2024-03-18T22:48:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8ec37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ec4102300 0x7f8ec4197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52466/0 (socket says 172.21.15.40:52466) 2024-03-18T22:48:04.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8ec3fff640 1 -- 172.21.15.40:0/4164315478 learned_addr learned my addr 172.21.15.40:0/4164315478 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8ec3fff640 1 -- 172.21.15.40:0/4164315478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ec4102300 msgr2=0x7f8ec4197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8ec3fff640 1 --2- 172.21.15.40:0/4164315478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ec4102300 0x7f8ec4197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8ec3fff640 1 -- 172.21.15.40:0/4164315478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec4104c30 msgr2=0x7f8ec4197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8ec3fff640 1 --2- 172.21.15.40:0/4164315478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec4104c30 0x7f8ec4197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8ec3fff640 1 -- 172.21.15.40:0/4164315478 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec419f690 con 0x7f8ec4107560 2024-03-18T22:48:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.255+0000 7f8ec2ffd640 1 --2- 172.21.15.40:0/4164315478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec4104c30 0x7f8ec4197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.256+0000 7f8ec3fff640 1 --2- 172.21.15.40:0/4164315478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ec4107560 0x7f8ec419f020 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00eb10 tx=0x7f8ebc00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.256+0000 7f8ec0ff9640 1 -- 172.21.15.40:0/4164315478 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc004b20 con 0x7f8ec4107560 2024-03-18T22:48:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.256+0000 7f8ec0ff9640 1 -- 172.21.15.40:0/4164315478 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ebc004cc0 con 0x7f8ec4107560 2024-03-18T22:48:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.256+0000 7f8eca027640 1 -- 172.21.15.40:0/4164315478 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ec419f980 con 0x7f8ec4107560 2024-03-18T22:48:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.256+0000 7f8ec0ff9640 1 -- 172.21.15.40:0/4164315478 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc010690 con 0x7f8ec4107560 2024-03-18T22:48:04.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.256+0000 7f8eca027640 1 -- 172.21.15.40:0/4164315478 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ec419fed0 con 0x7f8ec4107560 2024-03-18T22:48:04.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.259+0000 7f8ec0ff9640 1 -- 172.21.15.40:0/4164315478 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8ebc005020 con 0x7f8ec4107560 2024-03-18T22:48:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.259+0000 7f8eccc11740 1 Processor -- start 2024-03-18T22:48:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.259+0000 7f8eccc11740 1 -- start start 2024-03-18T22:48:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.259+0000 7f8eccc11740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c812f0 0x5582d0c81710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.259+0000 7f8eccc11740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c81ce0 0x5582d0c867e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.259+0000 7f8eccc11740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c86e50 0x5582d0c892e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.259+0000 7f8eccc11740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5582d0be54b0 con 0x5582d0c812f0 2024-03-18T22:48:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.259+0000 7f8eccc11740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5582d0b8dd70 con 0x5582d0c86e50 2024-03-18T22:48:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.259+0000 7f8eccc11740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5582d0be1550 con 0x5582d0c81ce0 2024-03-18T22:48:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.259+0000 7f8ec2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c812f0 0x5582d0c81710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.259+0000 7f8ec2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c812f0 0x5582d0c81710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46850/0 (socket says 172.21.15.40:46850) 2024-03-18T22:48:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.259+0000 7f8ec37fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c81ce0 0x5582d0c867e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.259+0000 7f8ec2ffd640 1 -- 172.21.15.40:0/3824829761 learned_addr learned my addr 172.21.15.40:0/3824829761 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.259+0000 7f8ec2ffd640 1 --2- 172.21.15.40:0/3824829761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c86e50 0x5582d0c892e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.260+0000 7f8ec37fe640 1 -- 172.21.15.40:0/3824829761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c86e50 msgr2=0x5582d0c892e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.260+0000 7f8ec37fe640 1 --2- 172.21.15.40:0/3824829761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c86e50 0x5582d0c892e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.260+0000 7f8ec0ff9640 1 --2- 172.21.15.40:0/4164315478 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8eac0a2800 0x7f8eac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.260+0000 7f8ec37fe640 1 -- 172.21.15.40:0/3824829761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c812f0 msgr2=0x5582d0c81710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.260+0000 7f8ec37fe640 1 --2- 172.21.15.40:0/3824829761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c812f0 0x5582d0c81710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.260+0000 7f8ec37fe640 1 -- 172.21.15.40:0/3824829761 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582d0bb0210 con 0x5582d0c81ce0 2024-03-18T22:48:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.260+0000 7f8ec3fff640 1 --2- 172.21.15.40:0/4164315478 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8eac0a2800 0x7f8eac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.260+0000 7f8ec37fe640 1 --2- 172.21.15.40:0/3824829761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c81ce0 0x5582d0c867e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4000c00 tx=0x7f8eb4009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc3bcbc702939163 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.260+0000 7f8ec0ff9640 1 -- 172.21.15.40:0/4164315478 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8ebc0cf510 con 0x7f8ec4107560 2024-03-18T22:48:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.260+0000 7f8ea9ffb640 1 -- 172.21.15.40:0/3824829761 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4045020 con 0x5582d0c81ce0 2024-03-18T22:48:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.260+0000 7f8ea9ffb640 1 -- 172.21.15.40:0/3824829761 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8eb4004040 con 0x5582d0c81ce0 2024-03-18T22:48:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.260+0000 7f8ea9ffb640 1 -- 172.21.15.40:0/3824829761 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4005530 con 0x5582d0c81ce0 2024-03-18T22:48:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.261+0000 7f8eccc11740 1 -- 172.21.15.40:0/3824829761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c81ce0 msgr2=0x5582d0c867e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.261+0000 7f8ec3fff640 1 --2- 172.21.15.40:0/4164315478 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8eac0a2800 0x7f8eac0a4cc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc09b7c0 tx=0x7f8ebc0c6510 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.261+0000 7f8eccc11740 1 --2- 172.21.15.40:0/3824829761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c81ce0 0x5582d0c867e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8eb4000c00 tx=0x7f8eb4009560 comp rx=0 tx=0).stop 2024-03-18T22:48:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.261+0000 7f8eccc11740 1 -- 172.21.15.40:0/3824829761 shutdown_connections 2024-03-18T22:48:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.261+0000 7f8eccc11740 1 --2- 172.21.15.40:0/3824829761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c86e50 0x5582d0c892e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.261+0000 7f8eccc11740 1 --2- 172.21.15.40:0/3824829761 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c81ce0 0x5582d0c867e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.261+0000 7f8eccc11740 1 --2- 172.21.15.40:0/3824829761 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c812f0 0x5582d0c81710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.261+0000 7f8eccc11740 1 -- 172.21.15.40:0/3824829761 >> 172.21.15.40:0/3824829761 conn(0x5582d0c7c860 msgr2=0x5582d0c7cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.261+0000 7f8eccc11740 1 -- 172.21.15.40:0/3824829761 shutdown_connections 2024-03-18T22:48:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.261+0000 7f8eccc11740 1 -- 172.21.15.40:0/3824829761 wait complete. 2024-03-18T22:48:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.261+0000 7f8eccc11740 1 -- start start 2024-03-18T22:48:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.261+0000 7f8eccc11740 1 Processor -- start 2024-03-18T22:48:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8eccc11740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c812f0 0x5582d0c8fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8eccc11740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c81ce0 0x5582d0c90090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8eccc11740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c86e50 0x5582d0c970d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8eccc11740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5582d0c8e120 con 0x5582d0c81ce0 2024-03-18T22:48:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8eccc11740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5582d0ba1310 con 0x5582d0c86e50 2024-03-18T22:48:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8eccc11740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5582d0c8e2a0 con 0x5582d0c812f0 2024-03-18T22:48:04.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8eccc11740 10 client.0 _ll_register_callbacks cb 0x5582d0b952d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8ec37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c81ce0 0x5582d0c90090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8ec2ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c812f0 0x5582d0c8fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8ec37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c81ce0 0x5582d0c90090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46862/0 (socket says 172.21.15.40:46862) 2024-03-18T22:48:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8ec2ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c812f0 0x5582d0c8fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:48056/0 (socket says 172.21.15.40:48056) 2024-03-18T22:48:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8ec37fe640 1 -- 172.21.15.40:0/3414280802 learned_addr learned my addr 172.21.15.40:0/3414280802 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8ec2ffd640 1 --2- 172.21.15.40:0/3414280802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c86e50 0x5582d0c970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8ec2ffd640 1 -- 172.21.15.40:0/3414280802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c86e50 msgr2=0x5582d0c970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8ec2ffd640 1 --2- 172.21.15.40:0/3414280802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c86e50 0x5582d0c970d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8ec2ffd640 1 -- 172.21.15.40:0/3414280802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c81ce0 msgr2=0x5582d0c90090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8ec2ffd640 1 --2- 172.21.15.40:0/3414280802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c81ce0 0x5582d0c90090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8ec2ffd640 1 -- 172.21.15.40:0/3414280802 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582d08f93f0 con 0x5582d0c812f0 2024-03-18T22:48:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8ec37fe640 1 --2- 172.21.15.40:0/3414280802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c81ce0 0x5582d0c90090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8ec2ffd640 1 --2- 172.21.15.40:0/3414280802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c812f0 0x5582d0c8fb50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x5582d0c82540 tx=0x7f8eb8013460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63aa1f8cdd5435f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.262+0000 7f8ea2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.263+0000 7f8ea2ffd640 1 -- 172.21.15.40:0/3414280802 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb800fc50 con 0x5582d0c812f0 2024-03-18T22:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.263+0000 7f8ea2ffd640 1 -- 172.21.15.40:0/3414280802 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8eb800fdf0 con 0x5582d0c812f0 2024-03-18T22:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.263+0000 7f8eccc11740 1 -- 172.21.15.40:0/3414280802 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8eb4004040 con 0x5582d0c812f0 2024-03-18T22:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.263+0000 7f8ea2ffd640 1 -- 172.21.15.40:0/3414280802 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8eb802b050 con 0x5582d0c812f0 2024-03-18T22:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.263+0000 7f8eccc11740 10 client.26740 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.263+0000 7f8eccc11740 1 -- 172.21.15.40:0/3414280802 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5582d09c37d0 con 0x5582d0c812f0 2024-03-18T22:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.263+0000 7f8ea2ffd640 1 -- 172.21.15.40:0/3414280802 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8eb4004040 con 0x5582d0c812f0 2024-03-18T22:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.263+0000 7f8ea2ffd640 1 -- 172.21.15.40:0/3414280802 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8eb80047a0 con 0x5582d0c812f0 2024-03-18T22:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.263+0000 7f8eccc11740 10 client.26740 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.263+0000 7f8eccc11740 10 client.26740 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.263+0000 7f8eccc11740 20 client.26740 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.263+0000 7f8eccc11740 1 --2- 172.21.15.40:0/3414280802 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5582d0c9b920 0x5582d0c9ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.263+0000 7f8eccc11740 4 client.26740 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.263+0000 7f8eccc11740 1 -- 172.21.15.40:0/3414280802 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582d0c433e0 con 0x5582d0c9b920 2024-03-18T22:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.263+0000 7f8ec37fe640 1 --2- 172.21.15.40:0/3414280802 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5582d0c9b920 0x5582d0c9ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.264+0000 7f8ea2ffd640 10 client.26740 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.264+0000 7f8ec37fe640 1 --2- 172.21.15.40:0/3414280802 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5582d0c9b920 0x5582d0c9ddc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d648f33fc5937ccb server_cookie=ffd5f805f1c89c4b in_seq=0 out_seq=0 2024-03-18T22:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.264+0000 7f8ea2ffd640 1 -- 172.21.15.40:0/3414280802 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5582d0c433e0 con 0x5582d0c9b920 2024-03-18T22:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.264+0000 7f8ea2ffd640 10 client.26740 handle_command_reply: tid=0 2024-03-18T22:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.264+0000 7f8eccc11740 1 client.26740 shutdown 2024-03-18T22:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.265+0000 7f8eccc11740 1 -- 172.21.15.40:0/3414280802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c812f0 msgr2=0x5582d0c8fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.265+0000 7f8eccc11740 1 --2- 172.21.15.40:0/3414280802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c812f0 0x5582d0c8fb50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x5582d0c82540 tx=0x7f8eb8013460 comp rx=0 tx=0).stop 2024-03-18T22:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.265+0000 7f8eccc11740 1 -- 172.21.15.40:0/3414280802 shutdown_connections 2024-03-18T22:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.265+0000 7f8eccc11740 1 --2- 172.21.15.40:0/3414280802 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5582d0c9b920 0x5582d0c9ddc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.265+0000 7f8eccc11740 1 --2- 172.21.15.40:0/3414280802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c86e50 0x5582d0c970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.265+0000 7f8ea2ffd640 0 client.26740 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.265+0000 7f8eccc11740 1 --2- 172.21.15.40:0/3414280802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c81ce0 0x5582d0c90090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.265+0000 7f8eccc11740 1 --2- 172.21.15.40:0/3414280802 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c812f0 0x5582d0c8fb50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.265+0000 7f8eccc11740 1 -- 172.21.15.40:0/3414280802 >> 172.21.15.40:0/3414280802 conn(0x5582d0c7c860 msgr2=0x5582d0c7bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.265+0000 7f8eccc11740 1 -- 172.21.15.40:0/3414280802 shutdown_connections 2024-03-18T22:48:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.265+0000 7f8eccc11740 1 -- 172.21.15.40:0/3414280802 wait complete. 2024-03-18T22:48:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.265+0000 7f8eccc11740 20 client.26740 trim_cache size 0 max 16384 2024-03-18T22:48:04.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.280+0000 7f8eccc11740 1 Processor -- start 2024-03-18T22:48:04.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.280+0000 7f8eccc11740 1 -- start start 2024-03-18T22:48:04.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.280+0000 7f8eccc11740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c812f0 0x5582d0c8fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.280+0000 7f8eccc11740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c81ce0 0x5582d0c97090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.280+0000 7f8eccc11740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c86e50 0x5582d0c7b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.280+0000 7f8eccc11740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec41a8320 con 0x5582d0c86e50 2024-03-18T22:48:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.280+0000 7f8eccc11740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec41a9e60 con 0x5582d0c81ce0 2024-03-18T22:48:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.280+0000 7f8eccc11740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec41a89f0 con 0x5582d0c812f0 2024-03-18T22:48:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.280+0000 7f8ec2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c86e50 0x5582d0c7b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.280+0000 7f8ec37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c81ce0 0x5582d0c97090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.280+0000 7f8ec2ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c812f0 0x5582d0c8fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.280+0000 7f8ec37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c81ce0 0x5582d0c97090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52482/0 (socket says 172.21.15.40:52482) 2024-03-18T22:48:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.280+0000 7f8ec37fe640 1 -- 172.21.15.40:0/1940193980 learned_addr learned my addr 172.21.15.40:0/1940193980 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.281+0000 7f8ec2ffd640 1 -- 172.21.15.40:0/1940193980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c81ce0 msgr2=0x5582d0c97090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.281+0000 7f8ec2ffd640 1 --2- 172.21.15.40:0/1940193980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c81ce0 0x5582d0c97090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.281+0000 7f8ec2ffd640 1 -- 172.21.15.40:0/1940193980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c86e50 msgr2=0x5582d0c7b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.281+0000 7f8ec2ffd640 1 --2- 172.21.15.40:0/1940193980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c86e50 0x5582d0c7b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.281+0000 7f8ec2ffd640 1 -- 172.21.15.40:0/1940193980 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582d0c52880 con 0x5582d0c812f0 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.281+0000 7f8ec37fe640 1 --2- 172.21.15.40:0/1940193980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c81ce0 0x5582d0c97090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.281+0000 7f8ec2ffd640 1 --2- 172.21.15.40:0/1940193980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c812f0 0x5582d0c8fff0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8013390 tx=0x7f8eb8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59c58f578b769879 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.281+0000 7f8ea97fa640 1 -- 172.21.15.40:0/1940193980 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb800fc20 con 0x5582d0c812f0 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.281+0000 7f8ea97fa640 1 -- 172.21.15.40:0/1940193980 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8eb800fdc0 con 0x5582d0c812f0 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.281+0000 7f8ea97fa640 1 -- 172.21.15.40:0/1940193980 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb8004920 con 0x5582d0c812f0 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.282+0000 7f8eccc11740 1 -- 172.21.15.40:0/1940193980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c812f0 msgr2=0x5582d0c8fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.282+0000 7f8eccc11740 1 --2- 172.21.15.40:0/1940193980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c812f0 0x5582d0c8fff0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8013390 tx=0x7f8eb8000bc0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.282+0000 7f8eccc11740 1 -- 172.21.15.40:0/1940193980 shutdown_connections 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.282+0000 7f8eccc11740 1 --2- 172.21.15.40:0/1940193980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c86e50 0x5582d0c7b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.282+0000 7f8eccc11740 1 --2- 172.21.15.40:0/1940193980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c81ce0 0x5582d0c97090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.282+0000 7f8eccc11740 1 --2- 172.21.15.40:0/1940193980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c812f0 0x5582d0c8fff0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.282+0000 7f8eccc11740 1 -- 172.21.15.40:0/1940193980 >> 172.21.15.40:0/1940193980 conn(0x5582d0c7c860 msgr2=0x5582d0c7cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.282+0000 7f8eccc11740 1 -- 172.21.15.40:0/1940193980 shutdown_connections 2024-03-18T22:48:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.282+0000 7f8eccc11740 1 -- 172.21.15.40:0/1940193980 wait complete. 2024-03-18T22:48:04.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.282+0000 7f8eccc11740 1 -- start start 2024-03-18T22:48:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.282+0000 7f8eccc11740 1 Processor -- start 2024-03-18T22:48:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8eccc11740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c812f0 0x5582d0c94bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8eccc11740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c81ce0 0x5582d0c95110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8eccc11740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c86e50 0x5582d0c960b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8eccc11740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec41a9790 con 0x5582d0c86e50 2024-03-18T22:48:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8eccc11740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec41a90c0 con 0x5582d0c812f0 2024-03-18T22:48:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8eccc11740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec41a6300 con 0x5582d0c81ce0 2024-03-18T22:48:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8eccc11740 10 client.0 _ll_register_callbacks cb 0x7f8eb8013a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8ec2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c86e50 0x5582d0c960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8ec2ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c812f0 0x5582d0c94bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8ec37fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c81ce0 0x5582d0c95110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8ec2ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c812f0 0x5582d0c94bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52484/0 (socket says 172.21.15.40:52484) 2024-03-18T22:48:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8ec2ffd640 1 -- 172.21.15.40:0/240406270 learned_addr learned my addr 172.21.15.40:0/240406270 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8ec37fe640 1 -- 172.21.15.40:0/240406270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c812f0 msgr2=0x5582d0c94bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8ec37fe640 1 --2- 172.21.15.40:0/240406270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c812f0 0x5582d0c94bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8ec37fe640 1 -- 172.21.15.40:0/240406270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c86e50 msgr2=0x5582d0c960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8ec37fe640 1 --2- 172.21.15.40:0/240406270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c86e50 0x5582d0c960b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8ec37fe640 1 -- 172.21.15.40:0/240406270 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582d0c966a0 con 0x5582d0c81ce0 2024-03-18T22:48:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8ec37fe640 1 --2- 172.21.15.40:0/240406270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c81ce0 0x5582d0c95110 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8eb402c170 tx=0x7f8eb402c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23d0771e073770b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.283+0000 7f8ea2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:48:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.284+0000 7f8ea2ffd640 1 -- 172.21.15.40:0/240406270 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb4058020 con 0x5582d0c81ce0 2024-03-18T22:48:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.284+0000 7f8ea2ffd640 1 -- 172.21.15.40:0/240406270 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8eb4054ab0 con 0x5582d0c81ce0 2024-03-18T22:48:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.284+0000 7f8eccc11740 1 -- 172.21.15.40:0/240406270 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8eb800fdc0 con 0x5582d0c81ce0 2024-03-18T22:48:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.284+0000 7f8ea2ffd640 1 -- 172.21.15.40:0/240406270 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8eb4060050 con 0x5582d0c81ce0 2024-03-18T22:48:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.284+0000 7f8eccc11740 10 client.26746 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.284+0000 7f8eccc11740 1 -- 172.21.15.40:0/240406270 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5582d0d02060 con 0x5582d0c81ce0 2024-03-18T22:48:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.284+0000 7f8ea2ffd640 1 -- 172.21.15.40:0/240406270 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8eb800fdc0 con 0x5582d0c81ce0 2024-03-18T22:48:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.284+0000 7f8ea2ffd640 1 -- 172.21.15.40:0/240406270 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8eb40542e0 con 0x5582d0c81ce0 2024-03-18T22:48:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.284+0000 7f8eccc11740 10 client.26746 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.284+0000 7f8eccc11740 10 client.26746 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:48:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.284+0000 7f8eccc11740 20 client.26746 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.284+0000 7f8eccc11740 1 --2- 172.21.15.40:0/240406270 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5582d0d025f0 0x5582d0d04a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.284+0000 7f8eccc11740 4 client.26746 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:48:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.284+0000 7f8eccc11740 1 -- 172.21.15.40:0/240406270 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5582d0bf29f0 con 0x5582d0d025f0 2024-03-18T22:48:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.284+0000 7f8ec2ffd640 1 --2- 172.21.15.40:0/240406270 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5582d0d025f0 0x5582d0d04a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8ea2ffd640 10 client.26746 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:48:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8ec2ffd640 1 --2- 172.21.15.40:0/240406270 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5582d0d025f0 0x5582d0d04a10 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=693b1131a4e100fe server_cookie=8f132b602acda7 in_seq=0 out_seq=0 2024-03-18T22:48:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8ea2ffd640 1 -- 172.21.15.40:0/240406270 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5582d0bf29f0 con 0x5582d0d025f0 2024-03-18T22:48:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8ea2ffd640 10 client.26746 handle_command_reply: tid=0 2024-03-18T22:48:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8eccc11740 1 client.26746 shutdown 2024-03-18T22:48:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8eccc11740 1 -- 172.21.15.40:0/240406270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c81ce0 msgr2=0x5582d0c95110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8eccc11740 1 --2- 172.21.15.40:0/240406270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c81ce0 0x5582d0c95110 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8eb402c170 tx=0x7f8eb402c330 comp rx=0 tx=0).stop 2024-03-18T22:48:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8eccc11740 1 -- 172.21.15.40:0/240406270 shutdown_connections 2024-03-18T22:48:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8eccc11740 1 --2- 172.21.15.40:0/240406270 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5582d0d025f0 0x5582d0d04a10 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8eccc11740 1 --2- 172.21.15.40:0/240406270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5582d0c86e50 0x5582d0c960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8eccc11740 1 --2- 172.21.15.40:0/240406270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5582d0c81ce0 0x5582d0c95110 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8ea2ffd640 0 client.26746 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:48:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8eccc11740 1 --2- 172.21.15.40:0/240406270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582d0c812f0 0x5582d0c94bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8eccc11740 1 -- 172.21.15.40:0/240406270 >> 172.21.15.40:0/240406270 conn(0x5582d0c7c860 msgr2=0x5582d0c7fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8eccc11740 1 -- 172.21.15.40:0/240406270 shutdown_connections 2024-03-18T22:48:04.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8eccc11740 1 -- 172.21.15.40:0/240406270 wait complete. 2024-03-18T22:48:04.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.285+0000 7f8eccc11740 20 client.26746 trim_cache size 0 max 16384 2024-03-18T22:48:04.293 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:48:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.286+0000 7f8eca027640 1 -- 172.21.15.40:0/4164315478 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8eac0a2800 msgr2=0x7f8eac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.286+0000 7f8eca027640 1 --2- 172.21.15.40:0/4164315478 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8eac0a2800 0x7f8eac0a4cc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc09b7c0 tx=0x7f8ebc0c6510 comp rx=0 tx=0).stop 2024-03-18T22:48:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.286+0000 7f8eca027640 1 -- 172.21.15.40:0/4164315478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ec4107560 msgr2=0x7f8ec419f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.286+0000 7f8eca027640 1 --2- 172.21.15.40:0/4164315478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ec4107560 0x7f8ec419f020 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00eb10 tx=0x7f8ebc00c2d0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.286+0000 7f8eca027640 1 -- 172.21.15.40:0/4164315478 shutdown_connections 2024-03-18T22:48:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.286+0000 7f8eca027640 1 --2- 172.21.15.40:0/4164315478 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8eac0a2800 0x7f8eac0a4cc0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.286+0000 7f8eca027640 1 --2- 172.21.15.40:0/4164315478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8ec4107560 0x7f8ec419f020 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.286+0000 7f8eca027640 1 --2- 172.21.15.40:0/4164315478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ec4104c30 0x7f8ec4197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.286+0000 7f8eca027640 1 --2- 172.21.15.40:0/4164315478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ec4102300 0x7f8ec4197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.286+0000 7f8eca027640 1 -- 172.21.15.40:0/4164315478 >> 172.21.15.40:0/4164315478 conn(0x7f8ec40fbc90 msgr2=0x7f8ec4102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.286+0000 7f8eca027640 1 -- 172.21.15.40:0/4164315478 shutdown_connections 2024-03-18T22:48:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.286+0000 7f8eca027640 1 -- 172.21.15.40:0/4164315478 wait complete. 2024-03-18T22:48:04.299 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:48:04.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:04 smithi040 ceph-mon[62643]: pgmap v513: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 31 MiB/s wr, 21 op/s 2024-03-18T22:48:04.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:04 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2898329856' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:48:04.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.417+0000 7fdbda80b640 1 -- 172.21.15.40:0/627309151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd4100720 msgr2=0x7fdbd4100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.417+0000 7fdbda80b640 1 --2- 172.21.15.40:0/627309151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd4100720 0x7fdbd4100b40 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009900 tx=0x7fdbc402bc30 comp rx=0 tx=0).stop 2024-03-18T22:48:04.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.417+0000 7fdbda80b640 1 -- 172.21.15.40:0/627309151 shutdown_connections 2024-03-18T22:48:04.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.417+0000 7fdbda80b640 1 --2- 172.21.15.40:0/627309151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdbd41022d0 0x7fdbd4108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.417+0000 7fdbda80b640 1 --2- 172.21.15.40:0/627309151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbd4101910 0x7fdbd4101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.417+0000 7fdbda80b640 1 --2- 172.21.15.40:0/627309151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd4100720 0x7fdbd4100b40 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.417+0000 7fdbda80b640 1 -- 172.21.15.40:0/627309151 >> 172.21.15.40:0/627309151 conn(0x7fdbd40fbc90 msgr2=0x7fdbd40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.417+0000 7fdbda80b640 1 -- 172.21.15.40:0/627309151 shutdown_connections 2024-03-18T22:48:04.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.417+0000 7fdbda80b640 1 -- 172.21.15.40:0/627309151 wait complete. 2024-03-18T22:48:04.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.417+0000 7fdbda80b640 1 Processor -- start 2024-03-18T22:48:04.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbda80b640 1 -- start start 2024-03-18T22:48:04.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbda80b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbd4100720 0x7fdbd4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbda80b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdbd4101910 0x7fdbd4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbda80b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd41022d0 0x7fdbd419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbda80b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd410c240 con 0x7fdbd41022d0 2024-03-18T22:48:04.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbda80b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd410c0c0 con 0x7fdbd4101910 2024-03-18T22:48:04.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbda80b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd410c3c0 con 0x7fdbd4100720 2024-03-18T22:48:04.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbd8da8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd41022d0 0x7fdbd419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbd8da8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd41022d0 0x7fdbd419f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46892/0 (socket says 172.21.15.40:46892) 2024-03-18T22:48:04.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbd8da8640 1 -- 172.21.15.40:0/862520552 learned_addr learned my addr 172.21.15.40:0/862520552 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbd3fff640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbd4100720 0x7fdbd4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbd37fe640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdbd4101910 0x7fdbd4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbd8da8640 1 -- 172.21.15.40:0/862520552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbd4100720 msgr2=0x7fdbd4197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbd8da8640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbd4100720 0x7fdbd4197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbd8da8640 1 -- 172.21.15.40:0/862520552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdbd4101910 msgr2=0x7fdbd4197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbd8da8640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdbd4101910 0x7fdbd4197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbd8da8640 1 -- 172.21.15.40:0/862520552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd419f6b0 con 0x7fdbd41022d0 2024-03-18T22:48:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbd3fff640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbd4100720 0x7fdbd4197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.418+0000 7fdbd37fe640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdbd4101910 0x7fdbd4197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.419+0000 7fdbd8da8640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd41022d0 0x7fdbd419f040 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00eb10 tx=0x7fdbcc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.419+0000 7fdbd17fa640 1 -- 172.21.15.40:0/862520552 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc004b20 con 0x7fdbd41022d0 2024-03-18T22:48:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.419+0000 7fdbda80b640 1 -- 172.21.15.40:0/862520552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbd419f9a0 con 0x7fdbd41022d0 2024-03-18T22:48:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.419+0000 7fdbd17fa640 1 -- 172.21.15.40:0/862520552 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdbcc004cc0 con 0x7fdbd41022d0 2024-03-18T22:48:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.419+0000 7fdbd17fa640 1 -- 172.21.15.40:0/862520552 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc010690 con 0x7fdbd41022d0 2024-03-18T22:48:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.419+0000 7fdbda80b640 1 -- 172.21.15.40:0/862520552 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbd419fef0 con 0x7fdbd41022d0 2024-03-18T22:48:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.421+0000 7fdbd17fa640 1 -- 172.21.15.40:0/862520552 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdbcc005020 con 0x7fdbd41022d0 2024-03-18T22:48:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.421+0000 7fdbdd3f9740 1 Processor -- start 2024-03-18T22:48:04.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.421+0000 7fdbdd3f9740 1 -- start start 2024-03-18T22:48:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbdd3f9740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444c2f0 0x55865444c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbdd3f9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55865444cce0 0x5586544517e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbdd3f9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558654451e50 0x5586544542e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbdd3f9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5586543b04b0 con 0x55865444cce0 2024-03-18T22:48:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbdd3f9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558654358d70 con 0x558654451e50 2024-03-18T22:48:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbdd3f9740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5586543ac550 con 0x55865444c2f0 2024-03-18T22:48:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbd3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55865444cce0 0x5586544517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbd3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55865444cce0 0x5586544517e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46904/0 (socket says 172.21.15.40:46904) 2024-03-18T22:48:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbd3fff640 1 -- 172.21.15.40:0/3363058402 learned_addr learned my addr 172.21.15.40:0/3363058402 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbd37fe640 1 --2- 172.21.15.40:0/3363058402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444c2f0 0x55865444c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbd37fe640 1 --2- 172.21.15.40:0/3363058402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558654451e50 0x5586544542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbd17fa640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdbbc0a2800 0x7fdbbc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbd3fff640 1 -- 172.21.15.40:0/3363058402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444c2f0 msgr2=0x55865444c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbd3fff640 1 --2- 172.21.15.40:0/3363058402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444c2f0 0x55865444c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbd8da8640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdbbc0a2800 0x7fdbbc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbd3fff640 1 -- 172.21.15.40:0/3363058402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558654451e50 msgr2=0x5586544542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbd3fff640 1 --2- 172.21.15.40:0/3363058402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558654451e50 0x5586544542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbd3fff640 1 -- 172.21.15.40:0/3363058402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55865437b210 con 0x55865444cce0 2024-03-18T22:48:04.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbd37fe640 1 --2- 172.21.15.40:0/3363058402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444c2f0 0x55865444c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:04.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbd17fa640 1 -- 172.21.15.40:0/862520552 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdbcc0cf3b0 con 0x7fdbd41022d0 2024-03-18T22:48:04.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.422+0000 7fdbd3fff640 1 --2- 172.21.15.40:0/3363058402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55865444cce0 0x5586544517e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009900 tx=0x7fdbc40095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c2f6093a43ba3c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.423+0000 7fdbd8da8640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdbbc0a2800 0x7fdbbc0a4cc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0a3bd0 tx=0x7fdbcc00efd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.423+0000 7fdbba7fc640 1 -- 172.21.15.40:0/3363058402 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4045020 con 0x55865444cce0 2024-03-18T22:48:04.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.423+0000 7fdbba7fc640 1 -- 172.21.15.40:0/3363058402 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdbc402cc90 con 0x55865444cce0 2024-03-18T22:48:04.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.423+0000 7fdbba7fc640 1 -- 172.21.15.40:0/3363058402 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc40048c0 con 0x55865444cce0 2024-03-18T22:48:04.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.423+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/3363058402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55865444cce0 msgr2=0x5586544517e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.423+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/3363058402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55865444cce0 0x5586544517e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009900 tx=0x7fdbc40095f0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.423+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/3363058402 shutdown_connections 2024-03-18T22:48:04.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.423+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/3363058402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558654451e50 0x5586544542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.423+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/3363058402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55865444cce0 0x5586544517e0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.423+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/3363058402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444c2f0 0x55865444c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.423+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/3363058402 >> 172.21.15.40:0/3363058402 conn(0x558654447860 msgr2=0x558654447c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.423+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/3363058402 shutdown_connections 2024-03-18T22:48:04.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.423+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/3363058402 wait complete. 2024-03-18T22:48:04.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.424+0000 7fdbdd3f9740 1 -- start start 2024-03-18T22:48:04.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.424+0000 7fdbdd3f9740 1 Processor -- start 2024-03-18T22:48:04.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.424+0000 7fdbdd3f9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55865444c2f0 0x558654456790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.424+0000 7fdbdd3f9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444cce0 0x558654456cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.424+0000 7fdbdd3f9740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558654451e50 0x55865445dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.424+0000 7fdbdd3f9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558654454da0 con 0x55865444c2f0 2024-03-18T22:48:04.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.424+0000 7fdbdd3f9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55865436c310 con 0x55865444cce0 2024-03-18T22:48:04.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.424+0000 7fdbdd3f9740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x558654454f20 con 0x558654451e50 2024-03-18T22:48:04.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.424+0000 7fdbdd3f9740 10 client.0 _ll_register_callbacks cb 0x5586543602d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:04.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.424+0000 7fdbd37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55865444c2f0 0x558654456790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.424+0000 7fdbd37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55865444c2f0 0x558654456790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46914/0 (socket says 172.21.15.40:46914) 2024-03-18T22:48:04.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.424+0000 7fdbd37fe640 1 -- 172.21.15.40:0/246287015 learned_addr learned my addr 172.21.15.40:0/246287015 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.424+0000 7fdbd3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444cce0 0x558654456cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.425+0000 7fdbd37fe640 1 --2- 172.21.15.40:0/246287015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558654451e50 0x55865445dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.425+0000 7fdbd3fff640 1 -- 172.21.15.40:0/246287015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558654451e50 msgr2=0x55865445dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.425+0000 7fdbd3fff640 1 --2- 172.21.15.40:0/246287015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558654451e50 0x55865445dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.425+0000 7fdbd3fff640 1 -- 172.21.15.40:0/246287015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55865444c2f0 msgr2=0x558654456790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.425+0000 7fdbd3fff640 1 --2- 172.21.15.40:0/246287015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55865444c2f0 0x558654456790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.425+0000 7fdbd3fff640 1 -- 172.21.15.40:0/246287015 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586540c43f0 con 0x55865444cce0 2024-03-18T22:48:04.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.425+0000 7fdbd37fe640 1 --2- 172.21.15.40:0/246287015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55865444c2f0 0x558654456790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:04.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.425+0000 7fdbd3fff640 1 --2- 172.21.15.40:0/246287015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444cce0 0x558654456cd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x5586544487d0 tx=0x7fdbc40038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3774a2c4b73b02d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.425+0000 7fdbb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:48:04.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.425+0000 7fdbb37fe640 1 -- 172.21.15.40:0/246287015 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4050020 con 0x55865444cce0 2024-03-18T22:48:04.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.425+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/246287015 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbc402cc90 con 0x55865444cce0 2024-03-18T22:48:04.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.425+0000 7fdbb37fe640 1 -- 172.21.15.40:0/246287015 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdbc4041480 con 0x55865444cce0 2024-03-18T22:48:04.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.425+0000 7fdbb37fe640 1 -- 172.21.15.40:0/246287015 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdbc405b050 con 0x55865444cce0 2024-03-18T22:48:04.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.425+0000 7fdbdd3f9740 10 client.26765 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:04.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.425+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/246287015 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55865418e7d0 con 0x55865444cce0 2024-03-18T22:48:04.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.426+0000 7fdbb37fe640 1 -- 172.21.15.40:0/246287015 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdbc402cc90 con 0x55865444cce0 2024-03-18T22:48:04.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.426+0000 7fdbb37fe640 1 -- 172.21.15.40:0/246287015 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fdbc404c460 con 0x55865444cce0 2024-03-18T22:48:04.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.426+0000 7fdbdd3f9740 10 client.26765 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:04.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.426+0000 7fdbdd3f9740 10 client.26765 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:48:04.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.426+0000 7fdbdd3f9740 20 client.26765 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:04.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.426+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/246287015 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x558654462200 0x5586544645f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.426+0000 7fdbdd3f9740 4 client.26765 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:04.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.426+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/246287015 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55865440e3e0 con 0x558654462200 2024-03-18T22:48:04.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.426+0000 7fdbd37fe640 1 --2- 172.21.15.40:0/246287015 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x558654462200 0x5586544645f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.426+0000 7fdbd37fe640 1 --2- 172.21.15.40:0/246287015 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x558654462200 0x5586544645f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=41cf084b771810c7 server_cookie=e94900124a26fca1 in_seq=0 out_seq=0 2024-03-18T22:48:04.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.426+0000 7fdbb37fe640 10 client.26765 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:48:04.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.427+0000 7fdbb37fe640 1 -- 172.21.15.40:0/246287015 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55865440e3e0 con 0x558654462200 2024-03-18T22:48:04.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.427+0000 7fdbb37fe640 10 client.26765 handle_command_reply: tid=0 2024-03-18T22:48:04.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.427+0000 7fdbdd3f9740 1 client.26765 shutdown 2024-03-18T22:48:04.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.427+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/246287015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444cce0 msgr2=0x558654456cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.427+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/246287015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444cce0 0x558654456cd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x5586544487d0 tx=0x7fdbc40038a0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.427+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/246287015 shutdown_connections 2024-03-18T22:48:04.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.427+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/246287015 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x558654462200 0x5586544645f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.427+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/246287015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558654451e50 0x55865445dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.427+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/246287015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444cce0 0x558654456cd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.427+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/246287015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55865444c2f0 0x558654456790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.427+0000 7fdbb37fe640 0 client.26765 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:48:04.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.427+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/246287015 >> 172.21.15.40:0/246287015 conn(0x558654447860 msgr2=0x558654446ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.427+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/246287015 shutdown_connections 2024-03-18T22:48:04.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.427+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/246287015 wait complete. 2024-03-18T22:48:04.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.427+0000 7fdbdd3f9740 20 client.26765 trim_cache size 0 max 16384 2024-03-18T22:48:04.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbdd3f9740 1 Processor -- start 2024-03-18T22:48:04.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbdd3f9740 1 -- start start 2024-03-18T22:48:04.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbdd3f9740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444c2f0 0x55865444b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbdd3f9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444cce0 0x55865445dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbdd3f9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558654451e50 0x558654468e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbdd3f9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41aa4c0 con 0x558654451e50 2024-03-18T22:48:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbdd3f9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41ac000 con 0x55865444cce0 2024-03-18T22:48:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbdd3f9740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41aab90 con 0x55865444c2f0 2024-03-18T22:48:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbd3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444cce0 0x55865445dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbd37fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444c2f0 0x55865444b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbd3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444cce0 0x55865445dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52528/0 (socket says 172.21.15.40:52528) 2024-03-18T22:48:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbd3fff640 1 -- 172.21.15.40:0/3743178835 learned_addr learned my addr 172.21.15.40:0/3743178835 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbd37fe640 1 --2- 172.21.15.40:0/3743178835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558654451e50 0x558654468e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbd3fff640 1 -- 172.21.15.40:0/3743178835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444c2f0 msgr2=0x55865444b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbd3fff640 1 --2- 172.21.15.40:0/3743178835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444c2f0 0x55865444b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbd3fff640 1 -- 172.21.15.40:0/3743178835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558654451e50 msgr2=0x558654468e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbd3fff640 1 --2- 172.21.15.40:0/3743178835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558654451e50 0x558654468e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbd3fff640 1 -- 172.21.15.40:0/3743178835 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558654455880 con 0x55865444cce0 2024-03-18T22:48:04.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.443+0000 7fdbd37fe640 1 --2- 172.21.15.40:0/3743178835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444c2f0 0x55865444b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:04.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.444+0000 7fdbd3fff640 1 --2- 172.21.15.40:0/3743178835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444cce0 0x55865445dcd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdbc402c170 tx=0x7fdbc4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a35764446f947c64 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.444+0000 7fdbb9ffb640 1 -- 172.21.15.40:0/3743178835 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4050070 con 0x55865444cce0 2024-03-18T22:48:04.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.444+0000 7fdbb9ffb640 1 -- 172.21.15.40:0/3743178835 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdbc402cc90 con 0x55865444cce0 2024-03-18T22:48:04.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.444+0000 7fdbb9ffb640 1 -- 172.21.15.40:0/3743178835 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4045040 con 0x55865444cce0 2024-03-18T22:48:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.444+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/3743178835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444cce0 msgr2=0x55865445dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.444+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/3743178835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444cce0 0x55865445dcd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdbc402c170 tx=0x7fdbc4000bc0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.444+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/3743178835 shutdown_connections 2024-03-18T22:48:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.444+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/3743178835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558654451e50 0x558654468e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.444+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/3743178835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444cce0 0x55865445dcd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.444+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/3743178835 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444c2f0 0x55865444b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.444+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/3743178835 >> 172.21.15.40:0/3743178835 conn(0x558654447860 msgr2=0x55865444aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.444+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/3743178835 shutdown_connections 2024-03-18T22:48:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.445+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/3743178835 wait complete. 2024-03-18T22:48:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.445+0000 7fdbdd3f9740 1 -- start start 2024-03-18T22:48:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.445+0000 7fdbdd3f9740 1 Processor -- start 2024-03-18T22:48:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.445+0000 7fdbdd3f9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444c2f0 0x55865445b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.445+0000 7fdbdd3f9740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444cce0 0x55865445bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.445+0000 7fdbdd3f9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558654451e50 0x55865445cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.445+0000 7fdbdd3f9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41ab930 con 0x558654451e50 2024-03-18T22:48:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.445+0000 7fdbdd3f9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41ab260 con 0x55865444c2f0 2024-03-18T22:48:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.445+0000 7fdbdd3f9740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd41a84a0 con 0x55865444cce0 2024-03-18T22:48:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.445+0000 7fdbdd3f9740 10 client.0 _ll_register_callbacks cb 0x7fdbc8011610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.445+0000 7fdbd37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444c2f0 0x55865445b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.445+0000 7fdbd37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558654451e50 0x55865445cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.445+0000 7fdbd37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558654451e50 0x55865445cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46918/0 (socket says 172.21.15.40:46918) 2024-03-18T22:48:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.445+0000 7fdbd37fe640 1 -- 172.21.15.40:0/2882235682 learned_addr learned my addr 172.21.15.40:0/2882235682 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.445+0000 7fdbd3fff640 1 --2- 172.21.15.40:0/2882235682 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444cce0 0x55865445bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.446+0000 7fdbd37fe640 1 -- 172.21.15.40:0/2882235682 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444cce0 msgr2=0x55865445bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.446+0000 7fdbd37fe640 1 --2- 172.21.15.40:0/2882235682 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444cce0 0x55865445bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.446+0000 7fdbd37fe640 1 -- 172.21.15.40:0/2882235682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444c2f0 msgr2=0x55865445b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.446+0000 7fdbd37fe640 1 --2- 172.21.15.40:0/2882235682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444c2f0 0x55865445b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.446+0000 7fdbd37fe640 1 -- 172.21.15.40:0/2882235682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55865445d210 con 0x558654451e50 2024-03-18T22:48:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.446+0000 7fdbd37fe640 1 --2- 172.21.15.40:0/2882235682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558654451e50 0x55865445cc20 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8004520 tx=0x7fdbc8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d23d4f52287c1c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.446+0000 7fdbb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:48:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.446+0000 7fdbb37fe640 1 -- 172.21.15.40:0/2882235682 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8012e50 con 0x558654451e50 2024-03-18T22:48:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.446+0000 7fdbb37fe640 1 -- 172.21.15.40:0/2882235682 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdbc8009040 con 0x558654451e50 2024-03-18T22:48:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.446+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/2882235682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbc402cc90 con 0x558654451e50 2024-03-18T22:48:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.446+0000 7fdbb37fe640 1 -- 172.21.15.40:0/2882235682 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdbc803c050 con 0x558654451e50 2024-03-18T22:48:04.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.446+0000 7fdbdd3f9740 10 client.16950 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:04.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.446+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/2882235682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5586544c8860 con 0x558654451e50 2024-03-18T22:48:04.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.447+0000 7fdbb37fe640 1 -- 172.21.15.40:0/2882235682 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdbc402cc90 con 0x558654451e50 2024-03-18T22:48:04.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.447+0000 7fdbb37fe640 1 -- 172.21.15.40:0/2882235682 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fdbc8033070 con 0x558654451e50 2024-03-18T22:48:04.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.447+0000 7fdbdd3f9740 10 client.16950 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:04.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.447+0000 7fdbdd3f9740 10 client.16950 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:48:04.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.447+0000 7fdbdd3f9740 20 client.16950 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:04.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.447+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/2882235682 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5586544c9240 0x5586544cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.447+0000 7fdbdd3f9740 4 client.16950 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:48:04.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.447+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/2882235682 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5586543bd9f0 con 0x5586544c9240 2024-03-18T22:48:04.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.447+0000 7fdbd3fff640 1 --2- 172.21.15.40:0/2882235682 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5586544c9240 0x5586544cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.447+0000 7fdbb37fe640 10 client.16950 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:48:04.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.447+0000 7fdbd3fff640 1 --2- 172.21.15.40:0/2882235682 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5586544c9240 0x5586544cb660 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8da6d654d1e9d06a server_cookie=5c02df5f1123d516 in_seq=0 out_seq=0 2024-03-18T22:48:04.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.447+0000 7fdbb37fe640 1 -- 172.21.15.40:0/2882235682 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5586543bd9f0 con 0x5586544c9240 2024-03-18T22:48:04.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.447+0000 7fdbb37fe640 10 client.16950 handle_command_reply: tid=0 2024-03-18T22:48:04.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.447+0000 7fdbdd3f9740 1 client.16950 shutdown 2024-03-18T22:48:04.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/2882235682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558654451e50 msgr2=0x55865445cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/2882235682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558654451e50 0x55865445cc20 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8004520 tx=0x7fdbc8005f90 comp rx=0 tx=0).stop 2024-03-18T22:48:04.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/2882235682 shutdown_connections 2024-03-18T22:48:04.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/2882235682 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5586544c9240 0x5586544cb660 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/2882235682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558654451e50 0x55865445cc20 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/2882235682 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55865444cce0 0x55865445bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbb37fe640 0 client.16950 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:48:04.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbdd3f9740 1 --2- 172.21.15.40:0/2882235682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55865444c2f0 0x55865445b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/2882235682 >> 172.21.15.40:0/2882235682 conn(0x558654447860 msgr2=0x55865444b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/2882235682 shutdown_connections 2024-03-18T22:48:04.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbdd3f9740 1 -- 172.21.15.40:0/2882235682 wait complete. 2024-03-18T22:48:04.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbdd3f9740 20 client.16950 trim_cache size 0 max 16384 2024-03-18T22:48:04.456 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:48:04.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbda80b640 1 -- 172.21.15.40:0/862520552 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdbbc0a2800 msgr2=0x7fdbbc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbda80b640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdbbc0a2800 0x7fdbbc0a4cc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0a3bd0 tx=0x7fdbcc00efd0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbda80b640 1 -- 172.21.15.40:0/862520552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd41022d0 msgr2=0x7fdbd419f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.448+0000 7fdbda80b640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd41022d0 0x7fdbd419f040 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc00eb10 tx=0x7fdbcc00c2d0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.449+0000 7fdbda80b640 1 -- 172.21.15.40:0/862520552 shutdown_connections 2024-03-18T22:48:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.449+0000 7fdbda80b640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdbbc0a2800 0x7fdbbc0a4cc0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.449+0000 7fdbda80b640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd41022d0 0x7fdbd419f040 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.449+0000 7fdbda80b640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdbd4101910 0x7fdbd4197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.449+0000 7fdbda80b640 1 --2- 172.21.15.40:0/862520552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdbd4100720 0x7fdbd4197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.449+0000 7fdbda80b640 1 -- 172.21.15.40:0/862520552 >> 172.21.15.40:0/862520552 conn(0x7fdbd40fbc90 msgr2=0x7fdbd40fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.449+0000 7fdbda80b640 1 -- 172.21.15.40:0/862520552 shutdown_connections 2024-03-18T22:48:04.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.449+0000 7fdbda80b640 1 -- 172.21.15.40:0/862520552 wait complete. 2024-03-18T22:48:04.461 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:48:04.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:04 smithi113 ceph-mon[66775]: pgmap v513: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 31 MiB/s wr, 21 op/s 2024-03-18T22:48:04.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:04 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2898329856' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:48:04.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.583+0000 7fd105dd4640 1 -- 172.21.15.40:0/1458192354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd100102aa0 msgr2=0x7fd100102f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.583+0000 7fd105dd4640 1 --2- 172.21.15.40:0/1458192354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd100102aa0 0x7fd100102f20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009860 tx=0x7fd0f402bb70 comp rx=0 tx=0).stop 2024-03-18T22:48:04.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.583+0000 7fd105dd4640 1 -- 172.21.15.40:0/1458192354 shutdown_connections 2024-03-18T22:48:04.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.583+0000 7fd105dd4640 1 --2- 172.21.15.40:0/1458192354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd100103460 0x7fd100109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.583+0000 7fd105dd4640 1 --2- 172.21.15.40:0/1458192354 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd100102aa0 0x7fd100102f20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.583+0000 7fd105dd4640 1 --2- 172.21.15.40:0/1458192354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1001018b0 0x7fd100101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.583+0000 7fd105dd4640 1 -- 172.21.15.40:0/1458192354 >> 172.21.15.40:0/1458192354 conn(0x7fd1000fce40 msgr2=0x7fd1000ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.583+0000 7fd105dd4640 1 -- 172.21.15.40:0/1458192354 shutdown_connections 2024-03-18T22:48:04.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.583+0000 7fd105dd4640 1 -- 172.21.15.40:0/1458192354 wait complete. 2024-03-18T22:48:04.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.583+0000 7fd105dd4640 1 Processor -- start 2024-03-18T22:48:04.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.583+0000 7fd105dd4640 1 -- start start 2024-03-18T22:48:04.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd105dd4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1001018b0 0x7fd1000755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd105dd4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd100102aa0 0x7fd100073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd105dd4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd100103460 0x7fd100074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd105dd4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd10010d370 con 0x7fd100103460 2024-03-18T22:48:04.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd105dd4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd10010d1f0 con 0x7fd100102aa0 2024-03-18T22:48:04.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd105dd4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd10010d4f0 con 0x7fd1001018b0 2024-03-18T22:48:04.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd0fffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd100103460 0x7fd100074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd0ff7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1001018b0 0x7fd1000755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd0fffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd100103460 0x7fd100074150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46940/0 (socket says 172.21.15.40:46940) 2024-03-18T22:48:04.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd0fffff640 1 -- 172.21.15.40:0/2480011635 learned_addr learned my addr 172.21.15.40:0/2480011635 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd0feffd640 1 --2- 172.21.15.40:0/2480011635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd100102aa0 0x7fd100073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd0ff7fe640 1 -- 172.21.15.40:0/2480011635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd100102aa0 msgr2=0x7fd100073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd0ff7fe640 1 --2- 172.21.15.40:0/2480011635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd100102aa0 0x7fd100073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd0ff7fe640 1 -- 172.21.15.40:0/2480011635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd100103460 msgr2=0x7fd100074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd0ff7fe640 1 --2- 172.21.15.40:0/2480011635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd100103460 0x7fd100074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd0ff7fe640 1 -- 172.21.15.40:0/2480011635 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1000749e0 con 0x7fd1001018b0 2024-03-18T22:48:04.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd0fffff640 1 --2- 172.21.15.40:0/2480011635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd100103460 0x7fd100074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:04.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd0feffd640 1 --2- 172.21.15.40:0/2480011635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd100102aa0 0x7fd100073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:04.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.584+0000 7fd0ff7fe640 1 --2- 172.21.15.40:0/2480011635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1001018b0 0x7fd1000755d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd0f000e9c0 tx=0x7fd0f000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.585+0000 7fd0fcff9640 1 -- 172.21.15.40:0/2480011635 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f000ccc0 con 0x7fd1001018b0 2024-03-18T22:48:04.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.585+0000 7fd0fcff9640 1 -- 172.21.15.40:0/2480011635 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0f00035f0 con 0x7fd1001018b0 2024-03-18T22:48:04.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.585+0000 7fd105dd4640 1 -- 172.21.15.40:0/2480011635 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1001a1bc0 con 0x7fd1001018b0 2024-03-18T22:48:04.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.585+0000 7fd0fcff9640 1 -- 172.21.15.40:0/2480011635 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f000ce60 con 0x7fd1001018b0 2024-03-18T22:48:04.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.585+0000 7fd105dd4640 1 -- 172.21.15.40:0/2480011635 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1001a20e0 con 0x7fd1001018b0 2024-03-18T22:48:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.587+0000 7fd0fcff9640 1 -- 172.21.15.40:0/2480011635 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd0f00037a0 con 0x7fd1001018b0 2024-03-18T22:48:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.587+0000 7fd1089c3740 1 Processor -- start 2024-03-18T22:48:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd1089c3740 1 -- start start 2024-03-18T22:48:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd0fcff9640 1 --2- 172.21.15.40:0/2480011635 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0e80a2880 0x7fd0e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd1089c3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e7d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd1089c3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e7dce0 0x565106e827e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd0fffff640 1 --2- 172.21.15.40:0/2480011635 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0e80a2880 0x7fd0e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd1089c3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e82e50 0x565106e852e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd1089c3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x565106de14b0 con 0x565106e82e50 2024-03-18T22:48:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd1089c3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x565106d89d70 con 0x565106e7d2f0 2024-03-18T22:48:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd1089c3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x565106ddd550 con 0x565106e7dce0 2024-03-18T22:48:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd0fcff9640 1 -- 172.21.15.40:0/2480011635 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd0f00cf1a0 con 0x7fd1001018b0 2024-03-18T22:48:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd0ff7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e7d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd0feffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e7dce0 0x565106e827e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd0fffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e82e50 0x565106e852e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd0ff7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e7d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52556/0 (socket says 172.21.15.40:52556) 2024-03-18T22:48:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd0ff7fe640 1 -- 172.21.15.40:0/53895456 learned_addr learned my addr 172.21.15.40:0/53895456 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd0ff7fe640 1 -- 172.21.15.40:0/53895456 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e7dce0 msgr2=0x565106e827e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd0ff7fe640 1 --2- 172.21.15.40:0/53895456 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e7dce0 0x565106e827e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd0ff7fe640 1 -- 172.21.15.40:0/53895456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e82e50 msgr2=0x565106e852e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd0ff7fe640 1 --2- 172.21.15.40:0/53895456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e82e50 0x565106e852e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd0ff7fe640 1 -- 172.21.15.40:0/53895456 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565106dac210 con 0x565106e7d2f0 2024-03-18T22:48:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.588+0000 7fd0fffff640 1 --2- 172.21.15.40:0/2480011635 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0e80a2880 0x7fd0e80a4d40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8009fd0 tx=0x7fd0f800c740 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.589+0000 7fd0ff7fe640 1 --2- 172.21.15.40:0/53895456 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e7d710 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0097b50 tx=0x7fd0f0097ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=462f4913fc11aa5f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.589+0000 7fd0e5ffb640 1 -- 172.21.15.40:0/53895456 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f009b050 con 0x565106e7d2f0 2024-03-18T22:48:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.589+0000 7fd0e5ffb640 1 -- 172.21.15.40:0/53895456 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0f0003b10 con 0x565106e7d2f0 2024-03-18T22:48:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.589+0000 7fd0e5ffb640 1 -- 172.21.15.40:0/53895456 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f001b160 con 0x565106e7d2f0 2024-03-18T22:48:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.589+0000 7fd1089c3740 1 -- 172.21.15.40:0/53895456 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 msgr2=0x565106e7d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.589+0000 7fd1089c3740 1 --2- 172.21.15.40:0/53895456 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e7d710 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0097b50 tx=0x7fd0f0097ac0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 1 -- 172.21.15.40:0/53895456 shutdown_connections 2024-03-18T22:48:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 1 --2- 172.21.15.40:0/53895456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e82e50 0x565106e852e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 1 --2- 172.21.15.40:0/53895456 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e7dce0 0x565106e827e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 1 --2- 172.21.15.40:0/53895456 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e7d710 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 1 -- 172.21.15.40:0/53895456 >> 172.21.15.40:0/53895456 conn(0x565106e78860 msgr2=0x565106e78c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 1 -- 172.21.15.40:0/53895456 shutdown_connections 2024-03-18T22:48:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 1 -- 172.21.15.40:0/53895456 wait complete. 2024-03-18T22:48:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 1 -- start start 2024-03-18T22:48:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 1 Processor -- start 2024-03-18T22:48:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e87810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e7dce0 0x565106e87d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e82e50 0x565106e8ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x565106e85e00 con 0x565106e82e50 2024-03-18T22:48:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x565106d9d310 con 0x565106e7d2f0 2024-03-18T22:48:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x565106e85f80 con 0x565106e7dce0 2024-03-18T22:48:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd1089c3740 10 client.0 _ll_register_callbacks cb 0x565106d912d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd0ff7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e87810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd0fffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e82e50 0x565106e8ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd0ff7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e87810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52560/0 (socket says 172.21.15.40:52560) 2024-03-18T22:48:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd0ff7fe640 1 -- 172.21.15.40:0/1278354803 learned_addr learned my addr 172.21.15.40:0/1278354803 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.590+0000 7fd0feffd640 1 --2- 172.21.15.40:0/1278354803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e7dce0 0x565106e87d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.591+0000 7fd0ff7fe640 1 -- 172.21.15.40:0/1278354803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e7dce0 msgr2=0x565106e87d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.591+0000 7fd0ff7fe640 1 --2- 172.21.15.40:0/1278354803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e7dce0 0x565106e87d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.591+0000 7fd0ff7fe640 1 -- 172.21.15.40:0/1278354803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e82e50 msgr2=0x565106e8ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.591+0000 7fd0ff7fe640 1 --2- 172.21.15.40:0/1278354803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e82e50 0x565106e8ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.591+0000 7fd0ff7fe640 1 -- 172.21.15.40:0/1278354803 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565106af53f0 con 0x565106e7d2f0 2024-03-18T22:48:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.591+0000 7fd0ff7fe640 1 --2- 172.21.15.40:0/1278354803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e87810 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd0f00980c0 tx=0x7fd0f0086180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c70fef54321686f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.591+0000 7fd0deffd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:48:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.591+0000 7fd0deffd640 1 -- 172.21.15.40:0/1278354803 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f00b9060 con 0x565106e7d2f0 2024-03-18T22:48:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.591+0000 7fd1089c3740 1 -- 172.21.15.40:0/1278354803 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x565106e3f3e0 con 0x565106e7d2f0 2024-03-18T22:48:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.591+0000 7fd0deffd640 1 -- 172.21.15.40:0/1278354803 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd0f00a3a00 con 0x565106e7d2f0 2024-03-18T22:48:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.591+0000 7fd0deffd640 1 -- 172.21.15.40:0/1278354803 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd0f0081050 con 0x565106e7d2f0 2024-03-18T22:48:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.591+0000 7fd1089c3740 10 client.26783 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.591+0000 7fd1089c3740 1 -- 172.21.15.40:0/1278354803 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x565106bbf7d0 con 0x565106e7d2f0 2024-03-18T22:48:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.591+0000 7fd0deffd640 1 -- 172.21.15.40:0/1278354803 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x565106e3f3e0 con 0x565106e7d2f0 2024-03-18T22:48:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.592+0000 7fd0deffd640 1 -- 172.21.15.40:0/1278354803 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd0f009b9e0 con 0x565106e7d2f0 2024-03-18T22:48:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.592+0000 7fd1089c3740 10 client.26783 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.592+0000 7fd1089c3740 10 client.26783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.592+0000 7fd1089c3740 20 client.26783 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.592+0000 7fd1089c3740 1 --2- 172.21.15.40:0/1278354803 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x565106e93210 0x565106e95600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.592+0000 7fd1089c3740 4 client.26783 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.592+0000 7fd1089c3740 1 -- 172.21.15.40:0/1278354803 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565106dee9f0 con 0x565106e93210 2024-03-18T22:48:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.592+0000 7fd0feffd640 1 --2- 172.21.15.40:0/1278354803 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x565106e93210 0x565106e95600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.592+0000 7fd0feffd640 1 --2- 172.21.15.40:0/1278354803 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x565106e93210 0x565106e95600 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c590eb22443a3c5 server_cookie=8ebddff9ec820a3a in_seq=0 out_seq=0 2024-03-18T22:48:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.592+0000 7fd0deffd640 10 client.26783 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.593+0000 7fd0deffd640 1 -- 172.21.15.40:0/1278354803 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x565106dee9f0 con 0x565106e93210 2024-03-18T22:48:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.593+0000 7fd0deffd640 10 client.26783 handle_command_reply: tid=0 2024-03-18T22:48:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.593+0000 7fd1089c3740 1 client.26783 shutdown 2024-03-18T22:48:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.593+0000 7fd1089c3740 1 -- 172.21.15.40:0/1278354803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 msgr2=0x565106e87810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.593+0000 7fd1089c3740 1 --2- 172.21.15.40:0/1278354803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e87810 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd0f00980c0 tx=0x7fd0f0086180 comp rx=0 tx=0).stop 2024-03-18T22:48:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.593+0000 7fd1089c3740 1 -- 172.21.15.40:0/1278354803 shutdown_connections 2024-03-18T22:48:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.593+0000 7fd1089c3740 1 --2- 172.21.15.40:0/1278354803 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e82e50 0x565106e8ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.593+0000 7fd1089c3740 1 --2- 172.21.15.40:0/1278354803 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e7dce0 0x565106e87d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.593+0000 7fd1089c3740 1 --2- 172.21.15.40:0/1278354803 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x565106e93210 0x565106e95600 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.593+0000 7fd1089c3740 1 --2- 172.21.15.40:0/1278354803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e87810 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.593+0000 7fd1089c3740 1 -- 172.21.15.40:0/1278354803 >> 172.21.15.40:0/1278354803 conn(0x565106e78860 msgr2=0x565106e77ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.593+0000 7fd0deffd640 0 client.26783 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.593+0000 7fd1089c3740 1 -- 172.21.15.40:0/1278354803 shutdown_connections 2024-03-18T22:48:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.593+0000 7fd1089c3740 1 -- 172.21.15.40:0/1278354803 wait complete. 2024-03-18T22:48:04.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.593+0000 7fd1089c3740 20 client.26783 trim_cache size 0 max 16384 2024-03-18T22:48:04.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:04 smithi103 ceph-mon[66377]: pgmap v513: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 31 MiB/s wr, 21 op/s 2024-03-18T22:48:04.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:04 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2898329856' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:48:04.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd1089c3740 1 Processor -- start 2024-03-18T22:48:04.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd1089c3740 1 -- start start 2024-03-18T22:48:04.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd1089c3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e87cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd1089c3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e7dce0 0x565106e8ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd1089c3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e82e50 0x565106e77a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd1089c3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd1001ac680 con 0x565106e7dce0 2024-03-18T22:48:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd1089c3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1001ae1c0 con 0x565106e7d2f0 2024-03-18T22:48:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd1089c3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd1001acd50 con 0x565106e82e50 2024-03-18T22:48:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd0feffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e7dce0 0x565106e8ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd0ff7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e87cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd0ff7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e87cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52564/0 (socket says 172.21.15.40:52564) 2024-03-18T22:48:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd0ff7fe640 1 -- 172.21.15.40:0/476545352 learned_addr learned my addr 172.21.15.40:0/476545352 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd0fffff640 1 --2- 172.21.15.40:0/476545352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e82e50 0x565106e77a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd0feffd640 1 -- 172.21.15.40:0/476545352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e82e50 msgr2=0x565106e77a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd0feffd640 1 --2- 172.21.15.40:0/476545352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e82e50 0x565106e77a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd0feffd640 1 -- 172.21.15.40:0/476545352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 msgr2=0x565106e87cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd0feffd640 1 --2- 172.21.15.40:0/476545352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e87cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd0feffd640 1 -- 172.21.15.40:0/476545352 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565106e7cf30 con 0x565106e7dce0 2024-03-18T22:48:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.612+0000 7fd0ff7fe640 1 --2- 172.21.15.40:0/476545352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e87cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.613+0000 7fd0feffd640 1 --2- 172.21.15.40:0/476545352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e7dce0 0x565106e8ece0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4037040 tx=0x7fd0f4034980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcfaa0eba0cebb61 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.613+0000 7fd0e57fa640 1 -- 172.21.15.40:0/476545352 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4039040 con 0x565106e7dce0 2024-03-18T22:48:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.613+0000 7fd0e57fa640 1 -- 172.21.15.40:0/476545352 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0f402c510 con 0x565106e7dce0 2024-03-18T22:48:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.613+0000 7fd0e57fa640 1 -- 172.21.15.40:0/476545352 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4003850 con 0x565106e7dce0 2024-03-18T22:48:04.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.614+0000 7fd1089c3740 1 -- 172.21.15.40:0/476545352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e7dce0 msgr2=0x565106e8ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.614+0000 7fd1089c3740 1 --2- 172.21.15.40:0/476545352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e7dce0 0x565106e8ece0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4037040 tx=0x7fd0f4034980 comp rx=0 tx=0).stop 2024-03-18T22:48:04.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.614+0000 7fd1089c3740 1 -- 172.21.15.40:0/476545352 shutdown_connections 2024-03-18T22:48:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.614+0000 7fd1089c3740 1 --2- 172.21.15.40:0/476545352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e82e50 0x565106e77a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.614+0000 7fd1089c3740 1 --2- 172.21.15.40:0/476545352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e7dce0 0x565106e8ece0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.614+0000 7fd1089c3740 1 --2- 172.21.15.40:0/476545352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e87cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.614+0000 7fd1089c3740 1 -- 172.21.15.40:0/476545352 >> 172.21.15.40:0/476545352 conn(0x565106e78860 msgr2=0x565106e77310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.614+0000 7fd1089c3740 1 -- 172.21.15.40:0/476545352 shutdown_connections 2024-03-18T22:48:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.614+0000 7fd1089c3740 1 -- 172.21.15.40:0/476545352 wait complete. 2024-03-18T22:48:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.614+0000 7fd1089c3740 1 -- start start 2024-03-18T22:48:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.614+0000 7fd1089c3740 1 Processor -- start 2024-03-18T22:48:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd1089c3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e8c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd1089c3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e7dce0 0x565106e8c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd1089c3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e82e50 0x565106e8d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd1089c3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd1001adaf0 con 0x565106e7dce0 2024-03-18T22:48:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd1089c3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1001ad420 con 0x565106e7d2f0 2024-03-18T22:48:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd1089c3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd1001aa660 con 0x565106e82e50 2024-03-18T22:48:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd1089c3740 10 client.0 _ll_register_callbacks cb 0x7fd0f8009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd0feffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e7dce0 0x565106e8c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd0feffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e7dce0 0x565106e8c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46984/0 (socket says 172.21.15.40:46984) 2024-03-18T22:48:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd0feffd640 1 -- 172.21.15.40:0/1915006977 learned_addr learned my addr 172.21.15.40:0/1915006977 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd0ff7fe640 1 --2- 172.21.15.40:0/1915006977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e8c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd0fffff640 1 --2- 172.21.15.40:0/1915006977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e82e50 0x565106e8d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd0feffd640 1 -- 172.21.15.40:0/1915006977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e82e50 msgr2=0x565106e8d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd0feffd640 1 --2- 172.21.15.40:0/1915006977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e82e50 0x565106e8d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd0feffd640 1 -- 172.21.15.40:0/1915006977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 msgr2=0x565106e8c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd0feffd640 1 --2- 172.21.15.40:0/1915006977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e8c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd0feffd640 1 -- 172.21.15.40:0/1915006977 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565106e8de80 con 0x565106e7dce0 2024-03-18T22:48:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd0ff7fe640 1 --2- 172.21.15.40:0/1915006977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e8c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd0feffd640 1 --2- 172.21.15.40:0/1915006977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e7dce0 0x565106e8c880 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fd0f402e010 tx=0x7fd0f4005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e39ee9902126dd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd0deffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:48:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd0deffd640 1 -- 172.21.15.40:0/1915006977 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4053030 con 0x565106e7dce0 2024-03-18T22:48:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd0deffd640 1 -- 172.21.15.40:0/1915006977 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd0f4046040 con 0x565106e7dce0 2024-03-18T22:48:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.615+0000 7fd1089c3740 1 -- 172.21.15.40:0/1915006977 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0f402c510 con 0x565106e7dce0 2024-03-18T22:48:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.616+0000 7fd0deffd640 1 -- 172.21.15.40:0/1915006977 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd0f4060050 con 0x565106e7dce0 2024-03-18T22:48:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.616+0000 7fd1089c3740 10 client.16968 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.616+0000 7fd1089c3740 1 -- 172.21.15.40:0/1915006977 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x565106ef92c0 con 0x565106e7dce0 2024-03-18T22:48:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.616+0000 7fd0deffd640 1 -- 172.21.15.40:0/1915006977 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0f402c510 con 0x565106e7dce0 2024-03-18T22:48:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.616+0000 7fd0deffd640 1 -- 172.21.15.40:0/1915006977 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd0f4003960 con 0x565106e7dce0 2024-03-18T22:48:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.616+0000 7fd1089c3740 10 client.16968 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.616+0000 7fd1089c3740 10 client.16968 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.616+0000 7fd1089c3740 20 client.16968 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.616+0000 7fd1089c3740 1 --2- 172.21.15.40:0/1915006977 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x565106ef9b70 0x565106efbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.616+0000 7fd1089c3740 4 client.16968 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:48:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.616+0000 7fd1089c3740 1 -- 172.21.15.40:0/1915006977 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x565106de8d40 con 0x565106ef9b70 2024-03-18T22:48:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.616+0000 7fd0ff7fe640 1 --2- 172.21.15.40:0/1915006977 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x565106ef9b70 0x565106efbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd0ff7fe640 1 --2- 172.21.15.40:0/1915006977 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x565106ef9b70 0x565106efbf90 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64d0e1d907f50362 server_cookie=6b82ec1492a871f3 in_seq=0 out_seq=0 2024-03-18T22:48:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd0deffd640 10 client.16968 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd0deffd640 1 -- 172.21.15.40:0/1915006977 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x565106de8d40 con 0x565106ef9b70 2024-03-18T22:48:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd0deffd640 10 client.16968 handle_command_reply: tid=0 2024-03-18T22:48:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd1089c3740 1 client.16968 shutdown 2024-03-18T22:48:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd1089c3740 1 -- 172.21.15.40:0/1915006977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e7dce0 msgr2=0x565106e8c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd1089c3740 1 --2- 172.21.15.40:0/1915006977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e7dce0 0x565106e8c880 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fd0f402e010 tx=0x7fd0f4005a90 comp rx=0 tx=0).stop 2024-03-18T22:48:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd1089c3740 1 -- 172.21.15.40:0/1915006977 shutdown_connections 2024-03-18T22:48:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd1089c3740 1 --2- 172.21.15.40:0/1915006977 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565106e82e50 0x565106e8d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd1089c3740 1 --2- 172.21.15.40:0/1915006977 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565106e7dce0 0x565106e8c880 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd1089c3740 1 --2- 172.21.15.40:0/1915006977 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x565106ef9b70 0x565106efbf90 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd1089c3740 1 --2- 172.21.15.40:0/1915006977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565106e7d2f0 0x565106e8c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd1089c3740 1 -- 172.21.15.40:0/1915006977 >> 172.21.15.40:0/1915006977 conn(0x565106e78860 msgr2=0x565106e77ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd0deffd640 0 client.16968 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd1089c3740 1 -- 172.21.15.40:0/1915006977 shutdown_connections 2024-03-18T22:48:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd1089c3740 1 -- 172.21.15.40:0/1915006977 wait complete. 2024-03-18T22:48:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.617+0000 7fd1089c3740 20 client.16968 trim_cache size 0 max 16384 2024-03-18T22:48:04.627 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:48:04.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.618+0000 7fd105dd4640 1 -- 172.21.15.40:0/2480011635 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0e80a2880 msgr2=0x7fd0e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.618+0000 7fd105dd4640 1 --2- 172.21.15.40:0/2480011635 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0e80a2880 0x7fd0e80a4d40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8009fd0 tx=0x7fd0f800c740 comp rx=0 tx=0).stop 2024-03-18T22:48:04.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.618+0000 7fd105dd4640 1 -- 172.21.15.40:0/2480011635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1001018b0 msgr2=0x7fd1000755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:04.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.618+0000 7fd105dd4640 1 --2- 172.21.15.40:0/2480011635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1001018b0 0x7fd1000755d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd0f000e9c0 tx=0x7fd0f000ee80 comp rx=0 tx=0).stop 2024-03-18T22:48:04.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.618+0000 7fd105dd4640 1 -- 172.21.15.40:0/2480011635 shutdown_connections 2024-03-18T22:48:04.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.618+0000 7fd105dd4640 1 --2- 172.21.15.40:0/2480011635 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0e80a2880 0x7fd0e80a4d40 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.618+0000 7fd105dd4640 1 --2- 172.21.15.40:0/2480011635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd100103460 0x7fd100074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.618+0000 7fd105dd4640 1 --2- 172.21.15.40:0/2480011635 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd100102aa0 0x7fd100073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.618+0000 7fd105dd4640 1 --2- 172.21.15.40:0/2480011635 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1001018b0 0x7fd1000755d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:04.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.618+0000 7fd105dd4640 1 -- 172.21.15.40:0/2480011635 >> 172.21.15.40:0/2480011635 conn(0x7fd1000fce40 msgr2=0x7fd100105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:04.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.618+0000 7fd105dd4640 1 -- 172.21.15.40:0/2480011635 shutdown_connections 2024-03-18T22:48:04.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:04.618+0000 7fd105dd4640 1 -- 172.21.15.40:0/2480011635 wait complete. 2024-03-18T22:48:05.631 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:48:05.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:48:05.632 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:48:05.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.757+0000 7fa44e07e640 1 -- 172.21.15.40:0/4278036153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4481022d0 msgr2=0x7fa448108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.757+0000 7fa44e07e640 1 --2- 172.21.15.40:0/4278036153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4481022d0 0x7fa448108b60 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa440009ca0 tx=0x7fa44002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.757+0000 7fa44e07e640 1 -- 172.21.15.40:0/4278036153 shutdown_connections 2024-03-18T22:48:05.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.757+0000 7fa44e07e640 1 --2- 172.21.15.40:0/4278036153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa4481022d0 0x7fa448108b60 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.757+0000 7fa44e07e640 1 --2- 172.21.15.40:0/4278036153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448101910 0x7fa448101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.757+0000 7fa44e07e640 1 --2- 172.21.15.40:0/4278036153 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa448100720 0x7fa448100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.757+0000 7fa44e07e640 1 -- 172.21.15.40:0/4278036153 >> 172.21.15.40:0/4278036153 conn(0x7fa4480fbcb0 msgr2=0x7fa4480fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:05.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.757+0000 7fa44e07e640 1 -- 172.21.15.40:0/4278036153 shutdown_connections 2024-03-18T22:48:05.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.757+0000 7fa44e07e640 1 -- 172.21.15.40:0/4278036153 wait complete. 2024-03-18T22:48:05.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.757+0000 7fa44e07e640 1 Processor -- start 2024-03-18T22:48:05.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa44e07e640 1 -- start start 2024-03-18T22:48:05.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa44e07e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448100720 0x7fa4480694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa44e07e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa448101910 0x7fa44819c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa44e07e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4481022d0 0x7fa44819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa44e07e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa44810c230 con 0x7fa448101910 2024-03-18T22:48:05.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa44e07e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa44810c0b0 con 0x7fa448100720 2024-03-18T22:48:05.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa44e07e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa44810c3b0 con 0x7fa4481022d0 2024-03-18T22:48:05.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa446ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa448101910 0x7fa44819c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa446ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa448101910 0x7fa44819c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47004/0 (socket says 172.21.15.40:47004) 2024-03-18T22:48:05.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa447fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4481022d0 0x7fa44819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448100720 0x7fa4480694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa446ffd640 1 -- 172.21.15.40:0/892248606 learned_addr learned my addr 172.21.15.40:0/892248606 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:05.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa446ffd640 1 -- 172.21.15.40:0/892248606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4481022d0 msgr2=0x7fa44819eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa446ffd640 1 --2- 172.21.15.40:0/892248606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4481022d0 0x7fa44819eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa446ffd640 1 -- 172.21.15.40:0/892248606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448100720 msgr2=0x7fa4480694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa446ffd640 1 --2- 172.21.15.40:0/892248606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448100720 0x7fa4480694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa446ffd640 1 -- 172.21.15.40:0/892248606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44819f6b0 con 0x7fa448101910 2024-03-18T22:48:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa4477fe640 1 --2- 172.21.15.40:0/892248606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448100720 0x7fa4480694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.758+0000 7fa447fff640 1 --2- 172.21.15.40:0/892248606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4481022d0 0x7fa44819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.759+0000 7fa446ffd640 1 --2- 172.21.15.40:0/892248606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa448101910 0x7fa44819c780 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa43c00e9f0 tx=0x7fa43c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.759+0000 7fa444ff9640 1 -- 172.21.15.40:0/892248606 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43c00cd80 con 0x7fa448101910 2024-03-18T22:48:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.759+0000 7fa444ff9640 1 -- 172.21.15.40:0/892248606 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa43c0108e0 con 0x7fa448101910 2024-03-18T22:48:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.759+0000 7fa44e07e640 1 -- 172.21.15.40:0/892248606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa44819f940 con 0x7fa448101910 2024-03-18T22:48:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.759+0000 7fa444ff9640 1 -- 172.21.15.40:0/892248606 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43c018920 con 0x7fa448101910 2024-03-18T22:48:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.759+0000 7fa44e07e640 1 -- 172.21.15.40:0/892248606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa44819fe90 con 0x7fa448101910 2024-03-18T22:48:05.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.761+0000 7fa444ff9640 1 -- 172.21.15.40:0/892248606 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa43c010a90 con 0x7fa448101910 2024-03-18T22:48:05.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa450c65740 1 Processor -- start 2024-03-18T22:48:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa444ff9640 1 --2- 172.21.15.40:0/892248606 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa4300a2800 0x7fa4300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa450c65740 1 -- start start 2024-03-18T22:48:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa4477fe640 1 --2- 172.21.15.40:0/892248606 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa4300a2800 0x7fa4300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa444ff9640 1 -- 172.21.15.40:0/892248606 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa43c0cf160 con 0x7fa448101910 2024-03-18T22:48:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa450c65740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa450c65740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 0x560f5c9d3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa450c65740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9d3f00 0x560f5c9d6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa450c65740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560f5c933160 con 0x560f5c9d3f00 2024-03-18T22:48:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa450c65740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560f5c8dbe10 con 0x560f5c9ced90 2024-03-18T22:48:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa450c65740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560f5c92f200 con 0x560f5c9ce3a0 2024-03-18T22:48:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa447fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9d3f00 0x560f5c9d6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa447fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9d3f00 0x560f5c9d6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47010/0 (socket says 172.21.15.40:47010) 2024-03-18T22:48:05.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa447fff640 1 -- 172.21.15.40:0/2783087548 learned_addr learned my addr 172.21.15.40:0/2783087548 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa446ffd640 1 --2- 172.21.15.40:0/2783087548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 0x560f5c9d3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa447fff640 1 -- 172.21.15.40:0/2783087548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 msgr2=0x560f5c9ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa4477fe640 1 --2- 172.21.15.40:0/892248606 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa4300a2800 0x7fa4300a4cc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa448101770 tx=0x7fa438011c80 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa447fff640 1 --2- 172.21.15.40:0/2783087548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa447fff640 1 -- 172.21.15.40:0/2783087548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 msgr2=0x560f5c9d3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa447fff640 1 --2- 172.21.15.40:0/2783087548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 0x560f5c9d3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa447fff640 1 -- 172.21.15.40:0/2783087548 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f5c70f7e0 con 0x560f5c9d3f00 2024-03-18T22:48:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.762+0000 7fa446ffd640 1 --2- 172.21.15.40:0/2783087548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 0x560f5c9d3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.763+0000 7fa447fff640 1 --2- 172.21.15.40:0/2783087548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9d3f00 0x560f5c9d6390 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fa440032380 tx=0x7fa440005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5b5ceba8f911b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.763+0000 7fa42dffb640 1 -- 172.21.15.40:0/2783087548 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440043070 con 0x560f5c9d3f00 2024-03-18T22:48:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.763+0000 7fa42dffb640 1 -- 172.21.15.40:0/2783087548 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa440003e20 con 0x560f5c9d3f00 2024-03-18T22:48:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.763+0000 7fa42dffb640 1 -- 172.21.15.40:0/2783087548 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440039040 con 0x560f5c9d3f00 2024-03-18T22:48:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.764+0000 7fa450c65740 1 -- 172.21.15.40:0/2783087548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9d3f00 msgr2=0x560f5c9d6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.764+0000 7fa450c65740 1 --2- 172.21.15.40:0/2783087548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9d3f00 0x560f5c9d6390 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fa440032380 tx=0x7fa440005ef0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.764+0000 7fa450c65740 1 -- 172.21.15.40:0/2783087548 shutdown_connections 2024-03-18T22:48:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.764+0000 7fa450c65740 1 --2- 172.21.15.40:0/2783087548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9d3f00 0x560f5c9d6390 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.764+0000 7fa450c65740 1 --2- 172.21.15.40:0/2783087548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 0x560f5c9d3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.764+0000 7fa450c65740 1 --2- 172.21.15.40:0/2783087548 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.764+0000 7fa450c65740 1 -- 172.21.15.40:0/2783087548 >> 172.21.15.40:0/2783087548 conn(0x560f5c9c9910 msgr2=0x560f5c9c9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.764+0000 7fa450c65740 1 -- 172.21.15.40:0/2783087548 shutdown_connections 2024-03-18T22:48:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.764+0000 7fa450c65740 1 -- 172.21.15.40:0/2783087548 wait complete. 2024-03-18T22:48:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.764+0000 7fa450c65740 1 -- start start 2024-03-18T22:48:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.764+0000 7fa450c65740 1 Processor -- start 2024-03-18T22:48:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa450c65740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9d8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa450c65740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 0x560f5c9d8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa450c65740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9d3f00 0x560f5c9dfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa450c65740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560f5c9d6e50 con 0x560f5c9d3f00 2024-03-18T22:48:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa450c65740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560f5c8ef1a0 con 0x560f5c9ced90 2024-03-18T22:48:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa450c65740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560f5c9d6fd0 con 0x560f5c9ce3a0 2024-03-18T22:48:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa450c65740 10 client.0 _ll_register_callbacks cb 0x560f5c646c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9d8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa447fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9d3f00 0x560f5c9dfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9d8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:48214/0 (socket says 172.21.15.40:48214) 2024-03-18T22:48:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa446ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 0x560f5c9d8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa4477fe640 1 -- 172.21.15.40:0/3919266096 learned_addr learned my addr 172.21.15.40:0/3919266096 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa4477fe640 1 -- 172.21.15.40:0/3919266096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 msgr2=0x560f5c9d8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa4477fe640 1 --2- 172.21.15.40:0/3919266096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 0x560f5c9d8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa4477fe640 1 -- 172.21.15.40:0/3919266096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9d3f00 msgr2=0x560f5c9dfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa4477fe640 1 --2- 172.21.15.40:0/3919266096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9d3f00 0x560f5c9dfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa4477fe640 1 -- 172.21.15.40:0/3919266096 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f5c76ff70 con 0x560f5c9ce3a0 2024-03-18T22:48:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa446ffd640 1 --2- 172.21.15.40:0/3919266096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 0x560f5c9d8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa447fff640 1 --2- 172.21.15.40:0/3919266096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9d3f00 0x560f5c9dfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa4477fe640 1 --2- 172.21.15.40:0/3919266096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9d8740 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa43800ff40 tx=0x7fa43801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c670b2a60801f206 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa426ffd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:48:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.765+0000 7fa426ffd640 1 -- 172.21.15.40:0/3919266096 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43801a040 con 0x560f5c9ce3a0 2024-03-18T22:48:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.766+0000 7fa450c65740 1 -- 172.21.15.40:0/3919266096 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa440003e20 con 0x560f5c9ce3a0 2024-03-18T22:48:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.766+0000 7fa426ffd640 1 -- 172.21.15.40:0/3919266096 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa438020450 con 0x560f5c9ce3a0 2024-03-18T22:48:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.766+0000 7fa426ffd640 1 -- 172.21.15.40:0/3919266096 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa438020690 con 0x560f5c9ce3a0 2024-03-18T22:48:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.766+0000 7fa450c65740 10 client.26794 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.766+0000 7fa450c65740 1 -- 172.21.15.40:0/3919266096 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560f5c607170 con 0x560f5c9ce3a0 2024-03-18T22:48:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.766+0000 7fa426ffd640 1 -- 172.21.15.40:0/3919266096 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa440003e20 con 0x560f5c9ce3a0 2024-03-18T22:48:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.766+0000 7fa426ffd640 1 -- 172.21.15.40:0/3919266096 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa43801d030 con 0x560f5c9ce3a0 2024-03-18T22:48:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.766+0000 7fa450c65740 10 client.26794 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.766+0000 7fa450c65740 10 client.26794 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.766+0000 7fa450c65740 20 client.26794 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.766+0000 7fa450c65740 1 --2- 172.21.15.40:0/3919266096 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560f5c9e45e0 0x560f5c9e6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.766+0000 7fa450c65740 4 client.26794 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.766+0000 7fa450c65740 1 -- 172.21.15.40:0/3919266096 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f5c9406a0 con 0x560f5c9e45e0 2024-03-18T22:48:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.766+0000 7fa446ffd640 1 --2- 172.21.15.40:0/3919266096 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560f5c9e45e0 0x560f5c9e6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.767+0000 7fa426ffd640 10 client.26794 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.767+0000 7fa446ffd640 1 --2- 172.21.15.40:0/3919266096 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560f5c9e45e0 0x560f5c9e6a70 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5203c86a24a7875d server_cookie=b4eea455eb1649a4 in_seq=0 out_seq=0 2024-03-18T22:48:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.767+0000 7fa426ffd640 1 -- 172.21.15.40:0/3919266096 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560f5c9406a0 con 0x560f5c9e45e0 2024-03-18T22:48:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.767+0000 7fa426ffd640 10 client.26794 handle_command_reply: tid=0 2024-03-18T22:48:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.767+0000 7fa450c65740 1 client.26794 shutdown 2024-03-18T22:48:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.767+0000 7fa450c65740 1 -- 172.21.15.40:0/3919266096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 msgr2=0x560f5c9d8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.768+0000 7fa450c65740 1 --2- 172.21.15.40:0/3919266096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9d8740 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa43800ff40 tx=0x7fa43801b000 comp rx=0 tx=0).stop 2024-03-18T22:48:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.768+0000 7fa450c65740 1 -- 172.21.15.40:0/3919266096 shutdown_connections 2024-03-18T22:48:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.768+0000 7fa450c65740 1 --2- 172.21.15.40:0/3919266096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9d3f00 0x560f5c9dfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.768+0000 7fa450c65740 1 --2- 172.21.15.40:0/3919266096 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560f5c9e45e0 0x560f5c9e6a70 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.768+0000 7fa450c65740 1 --2- 172.21.15.40:0/3919266096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 0x560f5c9d8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.768+0000 7fa426ffd640 0 client.26794 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.768+0000 7fa450c65740 1 --2- 172.21.15.40:0/3919266096 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9d8740 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.768+0000 7fa450c65740 1 -- 172.21.15.40:0/3919266096 >> 172.21.15.40:0/3919266096 conn(0x560f5c9c9910 msgr2=0x560f5c9c8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:05.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.768+0000 7fa450c65740 1 -- 172.21.15.40:0/3919266096 shutdown_connections 2024-03-18T22:48:05.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.768+0000 7fa450c65740 1 -- 172.21.15.40:0/3919266096 wait complete. 2024-03-18T22:48:05.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.768+0000 7fa450c65740 20 client.26794 trim_cache size 0 max 16384 2024-03-18T22:48:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.791+0000 7fa450c65740 1 Processor -- start 2024-03-18T22:48:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.791+0000 7fa450c65740 1 -- start start 2024-03-18T22:48:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.791+0000 7fa450c65740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9d8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.791+0000 7fa450c65740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9ced90 0x560f5c9dfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.791+0000 7fa450c65740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9d3f00 0x560f5c9c82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.791+0000 7fa450c65740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa4481aa590 con 0x560f5c9ced90 2024-03-18T22:48:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.791+0000 7fa450c65740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa4481ac0d0 con 0x560f5c9d3f00 2024-03-18T22:48:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.791+0000 7fa450c65740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa4481aac60 con 0x560f5c9ce3a0 2024-03-18T22:48:05.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.791+0000 7fa446ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9ced90 0x560f5c9dfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.791+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9d8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.791+0000 7fa447fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9d3f00 0x560f5c9c82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.791+0000 7fa446ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9ced90 0x560f5c9dfc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47022/0 (socket says 172.21.15.40:47022) 2024-03-18T22:48:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.791+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9d8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:48228/0 (socket says 172.21.15.40:48228) 2024-03-18T22:48:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.791+0000 7fa446ffd640 1 -- 172.21.15.40:0/4132813991 learned_addr learned my addr 172.21.15.40:0/4132813991 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.792+0000 7fa447fff640 1 -- 172.21.15.40:0/4132813991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 msgr2=0x560f5c9d8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.792+0000 7fa447fff640 1 --2- 172.21.15.40:0/4132813991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9d8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.792+0000 7fa447fff640 1 -- 172.21.15.40:0/4132813991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9ced90 msgr2=0x560f5c9dfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.792+0000 7fa447fff640 1 --2- 172.21.15.40:0/4132813991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9ced90 0x560f5c9dfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.792+0000 7fa447fff640 1 -- 172.21.15.40:0/4132813991 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f5c9c94b0 con 0x560f5c9d3f00 2024-03-18T22:48:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.792+0000 7fa4477fe640 1 --2- 172.21.15.40:0/4132813991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9d8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.792+0000 7fa447fff640 1 --2- 172.21.15.40:0/4132813991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9d3f00 0x560f5c9c82a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa440037040 tx=0x7fa440032f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f43f9e07284d02d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.792+0000 7fa42d7fa640 1 -- 172.21.15.40:0/4132813991 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440039040 con 0x560f5c9d3f00 2024-03-18T22:48:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.792+0000 7fa42d7fa640 1 -- 172.21.15.40:0/4132813991 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa440003b70 con 0x560f5c9d3f00 2024-03-18T22:48:05.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.792+0000 7fa42d7fa640 1 -- 172.21.15.40:0/4132813991 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440043040 con 0x560f5c9d3f00 2024-03-18T22:48:05.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.793+0000 7fa450c65740 1 -- 172.21.15.40:0/4132813991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9d3f00 msgr2=0x560f5c9c82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.793+0000 7fa450c65740 1 --2- 172.21.15.40:0/4132813991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9d3f00 0x560f5c9c82a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa440037040 tx=0x7fa440032f60 comp rx=0 tx=0).stop 2024-03-18T22:48:05.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.793+0000 7fa450c65740 1 -- 172.21.15.40:0/4132813991 shutdown_connections 2024-03-18T22:48:05.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.793+0000 7fa450c65740 1 --2- 172.21.15.40:0/4132813991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9d3f00 0x560f5c9c82a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.793+0000 7fa450c65740 1 --2- 172.21.15.40:0/4132813991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9ced90 0x560f5c9dfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.793+0000 7fa450c65740 1 --2- 172.21.15.40:0/4132813991 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9ce3a0 0x560f5c9d8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.793+0000 7fa450c65740 1 -- 172.21.15.40:0/4132813991 >> 172.21.15.40:0/4132813991 conn(0x560f5c9c9910 msgr2=0x560f5c9cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:05.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.793+0000 7fa450c65740 1 -- 172.21.15.40:0/4132813991 shutdown_connections 2024-03-18T22:48:05.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.793+0000 7fa450c65740 1 -- 172.21.15.40:0/4132813991 wait complete. 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.794+0000 7fa450c65740 1 -- start start 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.794+0000 7fa450c65740 1 Processor -- start 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.794+0000 7fa450c65740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9ce3a0 0x560f5c9dd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.794+0000 7fa450c65740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 0x560f5c9dde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.794+0000 7fa450c65740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9d3f00 0x560f5c9dee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.794+0000 7fa450c65740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa4481aba00 con 0x560f5c9ce3a0 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.794+0000 7fa450c65740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa4481ab330 con 0x560f5c9ced90 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.794+0000 7fa450c65740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa4481a8570 con 0x560f5c9d3f00 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.794+0000 7fa450c65740 10 client.0 _ll_register_callbacks cb 0x7fa43800fe20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.794+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9ce3a0 0x560f5c9dd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.794+0000 7fa4477fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9ce3a0 0x560f5c9dd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47024/0 (socket says 172.21.15.40:47024) 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.794+0000 7fa4477fe640 1 -- 172.21.15.40:0/2416198017 learned_addr learned my addr 172.21.15.40:0/2416198017 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.794+0000 7fa446ffd640 1 --2- 172.21.15.40:0/2416198017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 0x560f5c9dde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.795+0000 7fa4477fe640 1 -- 172.21.15.40:0/2416198017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9d3f00 msgr2=0x560f5c9dee70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.795+0000 7fa4477fe640 1 --2- 172.21.15.40:0/2416198017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9d3f00 0x560f5c9dee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.795+0000 7fa4477fe640 1 -- 172.21.15.40:0/2416198017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 msgr2=0x560f5c9dde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.795+0000 7fa4477fe640 1 --2- 172.21.15.40:0/2416198017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 0x560f5c9dde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.795+0000 7fa4477fe640 1 -- 172.21.15.40:0/2416198017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f5c9df460 con 0x560f5c9ce3a0 2024-03-18T22:48:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.795+0000 7fa446ffd640 1 --2- 172.21.15.40:0/2416198017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 0x560f5c9dde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.795+0000 7fa4477fe640 1 --2- 172.21.15.40:0/2416198017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9ce3a0 0x560f5c9dd920 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fa438028680 tx=0x7fa438009e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d365a1b21d3329a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.795+0000 7fa426ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:48:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.795+0000 7fa426ffd640 1 -- 172.21.15.40:0/2416198017 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43801a040 con 0x560f5c9ce3a0 2024-03-18T22:48:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.795+0000 7fa450c65740 1 -- 172.21.15.40:0/2416198017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa440003b70 con 0x560f5c9ce3a0 2024-03-18T22:48:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.795+0000 7fa426ffd640 1 -- 172.21.15.40:0/2416198017 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa4380288f0 con 0x560f5c9ce3a0 2024-03-18T22:48:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.795+0000 7fa426ffd640 1 -- 172.21.15.40:0/2416198017 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa438028af0 con 0x560f5c9ce3a0 2024-03-18T22:48:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.795+0000 7fa450c65740 10 client.16986 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.795+0000 7fa450c65740 1 -- 172.21.15.40:0/2416198017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560f5ca4aed0 con 0x560f5c9ce3a0 2024-03-18T22:48:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.796+0000 7fa426ffd640 1 -- 172.21.15.40:0/2416198017 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa440003b70 con 0x560f5c9ce3a0 2024-03-18T22:48:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.796+0000 7fa426ffd640 1 -- 172.21.15.40:0/2416198017 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa438020950 con 0x560f5c9ce3a0 2024-03-18T22:48:05.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.796+0000 7fa450c65740 10 client.16986 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.796+0000 7fa450c65740 10 client.16986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.796+0000 7fa450c65740 20 client.16986 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.796+0000 7fa450c65740 1 --2- 172.21.15.40:0/2416198017 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560f5ca4b950 0x560f5ca4dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.796+0000 7fa450c65740 4 client.16986 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:48:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.796+0000 7fa450c65740 1 -- 172.21.15.40:0/2416198017 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560f5c93a9f0 con 0x560f5ca4b950 2024-03-18T22:48:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.796+0000 7fa446ffd640 1 --2- 172.21.15.40:0/2416198017 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560f5ca4b950 0x560f5ca4dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.796+0000 7fa446ffd640 1 --2- 172.21.15.40:0/2416198017 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560f5ca4b950 0x560f5ca4dd70 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2703a462a42c974a server_cookie=39c2650f2be8e387 in_seq=0 out_seq=0 2024-03-18T22:48:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.796+0000 7fa426ffd640 10 client.16986 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.797+0000 7fa426ffd640 1 -- 172.21.15.40:0/2416198017 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560f5c93a9f0 con 0x560f5ca4b950 2024-03-18T22:48:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.797+0000 7fa426ffd640 10 client.16986 handle_command_reply: tid=0 2024-03-18T22:48:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.797+0000 7fa450c65740 1 client.16986 shutdown 2024-03-18T22:48:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa450c65740 1 -- 172.21.15.40:0/2416198017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9ce3a0 msgr2=0x560f5c9dd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa450c65740 1 --2- 172.21.15.40:0/2416198017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9ce3a0 0x560f5c9dd920 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fa438028680 tx=0x7fa438009e50 comp rx=0 tx=0).stop 2024-03-18T22:48:05.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa450c65740 1 -- 172.21.15.40:0/2416198017 shutdown_connections 2024-03-18T22:48:05.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa450c65740 1 --2- 172.21.15.40:0/2416198017 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560f5c9d3f00 0x560f5c9dee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa450c65740 1 --2- 172.21.15.40:0/2416198017 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560f5ca4b950 0x560f5ca4dd70 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa426ffd640 0 client.16986 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:05.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa450c65740 1 --2- 172.21.15.40:0/2416198017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f5c9ced90 0x560f5c9dde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa450c65740 1 --2- 172.21.15.40:0/2416198017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560f5c9ce3a0 0x560f5c9dd920 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa450c65740 1 -- 172.21.15.40:0/2416198017 >> 172.21.15.40:0/2416198017 conn(0x560f5c9c9910 msgr2=0x560f5c9ccb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:05.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa450c65740 1 -- 172.21.15.40:0/2416198017 shutdown_connections 2024-03-18T22:48:05.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa450c65740 1 -- 172.21.15.40:0/2416198017 wait complete. 2024-03-18T22:48:05.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa450c65740 20 client.16986 trim_cache size 0 max 16384 2024-03-18T22:48:05.804 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:48:05.804 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:48:05.804 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "419d4681-27e3-4e6d-9f86-cc86e518d421", 2024-03-18T22:48:05.805 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:48:05.805 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:48:05.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa44e07e640 1 -- 172.21.15.40:0/892248606 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa4300a2800 msgr2=0x7fa4300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa44e07e640 1 --2- 172.21.15.40:0/892248606 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa4300a2800 0x7fa4300a4cc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa448101770 tx=0x7fa438011c80 comp rx=0 tx=0).stop 2024-03-18T22:48:05.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa44e07e640 1 -- 172.21.15.40:0/892248606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa448101910 msgr2=0x7fa44819c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.798+0000 7fa44e07e640 1 --2- 172.21.15.40:0/892248606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa448101910 0x7fa44819c780 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa43c00e9f0 tx=0x7fa43c00eeb0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.799+0000 7fa44e07e640 1 -- 172.21.15.40:0/892248606 shutdown_connections 2024-03-18T22:48:05.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.799+0000 7fa44e07e640 1 --2- 172.21.15.40:0/892248606 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa4300a2800 0x7fa4300a4cc0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.799+0000 7fa44e07e640 1 --2- 172.21.15.40:0/892248606 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa4481022d0 0x7fa44819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.799+0000 7fa44e07e640 1 --2- 172.21.15.40:0/892248606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa448101910 0x7fa44819c780 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.799+0000 7fa44e07e640 1 --2- 172.21.15.40:0/892248606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa448100720 0x7fa4480694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.799+0000 7fa44e07e640 1 -- 172.21.15.40:0/892248606 >> 172.21.15.40:0/892248606 conn(0x7fa4480fbcb0 msgr2=0x7fa448104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:05.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.799+0000 7fa44e07e640 1 -- 172.21.15.40:0/892248606 shutdown_connections 2024-03-18T22:48:05.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.799+0000 7fa44e07e640 1 -- 172.21.15.40:0/892248606 wait complete. 2024-03-18T22:48:05.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:48:05.814 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:48:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.944+0000 7ffa8d15b640 1 -- 172.21.15.40:0/136608090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa881022d0 msgr2=0x7ffa88108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.944+0000 7ffa8d15b640 1 --2- 172.21.15.40:0/136608090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa881022d0 0x7ffa88108b60 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ffa80009ca0 tx=0x7ffa8002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.944+0000 7ffa8d15b640 1 -- 172.21.15.40:0/136608090 shutdown_connections 2024-03-18T22:48:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.944+0000 7ffa8d15b640 1 --2- 172.21.15.40:0/136608090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa881022d0 0x7ffa88108b60 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.944+0000 7ffa8d15b640 1 --2- 172.21.15.40:0/136608090 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa88101910 0x7ffa88101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.944+0000 7ffa8d15b640 1 --2- 172.21.15.40:0/136608090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa88100720 0x7ffa88100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.944+0000 7ffa8d15b640 1 -- 172.21.15.40:0/136608090 >> 172.21.15.40:0/136608090 conn(0x7ffa880fbcb0 msgr2=0x7ffa880fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.944+0000 7ffa8d15b640 1 -- 172.21.15.40:0/136608090 shutdown_connections 2024-03-18T22:48:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.944+0000 7ffa8d15b640 1 -- 172.21.15.40:0/136608090 wait complete. 2024-03-18T22:48:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa8d15b640 1 Processor -- start 2024-03-18T22:48:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa8d15b640 1 -- start start 2024-03-18T22:48:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa8d15b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa88100720 0x7ffa88197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa8d15b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa88101910 0x7ffa88197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa8d15b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa881022d0 0x7ffa8819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa8d15b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8810c250 con 0x7ffa88100720 2024-03-18T22:48:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa8d15b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8810c0d0 con 0x7ffa881022d0 2024-03-18T22:48:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa8d15b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8810c3d0 con 0x7ffa88101910 2024-03-18T22:48:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa86d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa88100720 0x7ffa88197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa7ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa88101910 0x7ffa88197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa7ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa88101910 0x7ffa88197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:48262/0 (socket says 172.21.15.40:48262) 2024-03-18T22:48:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa86d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa88100720 0x7ffa88197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47046/0 (socket says 172.21.15.40:47046) 2024-03-18T22:48:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa7ffff640 1 -- 172.21.15.40:0/2749688727 learned_addr learned my addr 172.21.15.40:0/2749688727 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa8759e640 1 --2- 172.21.15.40:0/2749688727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa881022d0 0x7ffa8819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa8759e640 1 -- 172.21.15.40:0/2749688727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa88101910 msgr2=0x7ffa88197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa8759e640 1 --2- 172.21.15.40:0/2749688727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa88101910 0x7ffa88197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa8759e640 1 -- 172.21.15.40:0/2749688727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa88100720 msgr2=0x7ffa88197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.945+0000 7ffa8759e640 1 --2- 172.21.15.40:0/2749688727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa88100720 0x7ffa88197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.946+0000 7ffa8759e640 1 -- 172.21.15.40:0/2749688727 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa8819f640 con 0x7ffa881022d0 2024-03-18T22:48:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.946+0000 7ffa7ffff640 1 --2- 172.21.15.40:0/2749688727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa88101910 0x7ffa88197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.946+0000 7ffa86d9d640 1 --2- 172.21.15.40:0/2749688727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa88100720 0x7ffa88197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.946+0000 7ffa8759e640 1 --2- 172.21.15.40:0/2749688727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa881022d0 0x7ffa8819efd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffa8002c630 tx=0x7ffa800328a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.946+0000 7ffa84d99640 1 -- 172.21.15.40:0/2749688727 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa80041020 con 0x7ffa881022d0 2024-03-18T22:48:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.946+0000 7ffa8d15b640 1 -- 172.21.15.40:0/2749688727 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa8819f8d0 con 0x7ffa881022d0 2024-03-18T22:48:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.946+0000 7ffa84d99640 1 -- 172.21.15.40:0/2749688727 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffa80003e20 con 0x7ffa881022d0 2024-03-18T22:48:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.946+0000 7ffa84d99640 1 -- 172.21.15.40:0/2749688727 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa80039040 con 0x7ffa881022d0 2024-03-18T22:48:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.946+0000 7ffa8d15b640 1 -- 172.21.15.40:0/2749688727 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa8819fe10 con 0x7ffa881022d0 2024-03-18T22:48:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.949+0000 7ffa84d99640 1 -- 172.21.15.40:0/2749688727 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ffa80044020 con 0x7ffa881022d0 2024-03-18T22:48:05.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.949+0000 7ffa8fe23740 1 Processor -- start 2024-03-18T22:48:05.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.949+0000 7ffa8fe23740 1 -- start start 2024-03-18T22:48:05.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa8fe23740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6e9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa8fe23740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e9850 0x55ac5f6ee350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa8fe23740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f6f0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa8fe23740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ac5f649500 con 0x55ac5f6ee9c0 2024-03-18T22:48:05.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa84d99640 1 --2- 172.21.15.40:0/2749688727 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ffa740a2800 0x7ffa740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa8fe23740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ac5f5f1be0 con 0x55ac5f6e9850 2024-03-18T22:48:05.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa8fe23740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55ac5f6455a0 con 0x55ac5f6e8e60 2024-03-18T22:48:05.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa8759e640 1 --2- 172.21.15.40:0/2749688727 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ffa740a2800 0x7ffa740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa7ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f6f0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa7ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f6f0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47056/0 (socket says 172.21.15.40:47056) 2024-03-18T22:48:05.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa7ffff640 1 -- 172.21.15.40:0/1502139082 learned_addr learned my addr 172.21.15.40:0/1502139082 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa84d99640 1 -- 172.21.15.40:0/2749688727 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ffa800f0fe0 con 0x7ffa881022d0 2024-03-18T22:48:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa7ffff640 1 --2- 172.21.15.40:0/1502139082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6e9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa86d9d640 1 --2- 172.21.15.40:0/1502139082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e9850 0x55ac5f6ee350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa7ffff640 1 -- 172.21.15.40:0/1502139082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6e8e60 msgr2=0x55ac5f6e9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa7ffff640 1 --2- 172.21.15.40:0/1502139082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6e9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa7ffff640 1 -- 172.21.15.40:0/1502139082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e9850 msgr2=0x55ac5f6ee350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa7ffff640 1 --2- 172.21.15.40:0/1502139082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e9850 0x55ac5f6ee350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa7ffff640 1 -- 172.21.15.40:0/1502139082 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac5f613000 con 0x55ac5f6ee9c0 2024-03-18T22:48:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.950+0000 7ffa8759e640 1 --2- 172.21.15.40:0/2749688727 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ffa740a2800 0x7ffa740a4cc0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ffa800bd290 tx=0x7ffa80088040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.951+0000 7ffa7ffff640 1 --2- 172.21.15.40:0/1502139082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f6f0e50 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ffa70009fd0 tx=0x7ffa70009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cf7394ce1ca5d37 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.951+0000 7ffa7d7fa640 1 -- 172.21.15.40:0/1502139082 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7000f940 con 0x55ac5f6ee9c0 2024-03-18T22:48:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.951+0000 7ffa7d7fa640 1 -- 172.21.15.40:0/1502139082 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffa70010430 con 0x55ac5f6ee9c0 2024-03-18T22:48:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.951+0000 7ffa7d7fa640 1 -- 172.21.15.40:0/1502139082 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7001e600 con 0x55ac5f6ee9c0 2024-03-18T22:48:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.952+0000 7ffa8fe23740 1 -- 172.21.15.40:0/1502139082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6ee9c0 msgr2=0x55ac5f6f0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.952+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/1502139082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f6f0e50 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ffa70009fd0 tx=0x7ffa70009d30 comp rx=0 tx=0).stop 2024-03-18T22:48:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.952+0000 7ffa8fe23740 1 -- 172.21.15.40:0/1502139082 shutdown_connections 2024-03-18T22:48:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.952+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/1502139082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f6f0e50 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.952+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/1502139082 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e9850 0x55ac5f6ee350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.952+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/1502139082 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6e9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.952+0000 7ffa8fe23740 1 -- 172.21.15.40:0/1502139082 >> 172.21.15.40:0/1502139082 conn(0x55ac5f6dffc0 msgr2=0x55ac5f6e03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.952+0000 7ffa8fe23740 1 -- 172.21.15.40:0/1502139082 shutdown_connections 2024-03-18T22:48:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.952+0000 7ffa8fe23740 1 -- 172.21.15.40:0/1502139082 wait complete. 2024-03-18T22:48:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.952+0000 7ffa8fe23740 1 -- start start 2024-03-18T22:48:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.952+0000 7ffa8fe23740 1 Processor -- start 2024-03-18T22:48:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa8fe23740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa8fe23740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6e9850 0x55ac5f6f3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa8fe23740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f6fa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa8fe23740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ac5f6f1970 con 0x55ac5f6e9850 2024-03-18T22:48:05.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa8fe23740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ac5f605160 con 0x55ac5f6e8e60 2024-03-18T22:48:05.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa8fe23740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55ac5f6f1af0 con 0x55ac5f6ee9c0 2024-03-18T22:48:05.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa8fe23740 10 client.0 _ll_register_callbacks cb 0x55ac5f641a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:05.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa7ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa7ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6f33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52686/0 (socket says 172.21.15.40:52686) 2024-03-18T22:48:05.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa7ffff640 1 -- 172.21.15.40:0/983304688 learned_addr learned my addr 172.21.15.40:0/983304688 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:05.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa86d9d640 1 --2- 172.21.15.40:0/983304688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6e9850 0x55ac5f6f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa7ffff640 1 --2- 172.21.15.40:0/983304688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f6fa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa7ffff640 1 -- 172.21.15.40:0/983304688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6ee9c0 msgr2=0x55ac5f6fa940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:05.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa7ffff640 1 --2- 172.21.15.40:0/983304688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f6fa940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa7ffff640 1 -- 172.21.15.40:0/983304688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6e9850 msgr2=0x55ac5f6f3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:05.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa7ffff640 1 --2- 172.21.15.40:0/983304688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6e9850 0x55ac5f6f3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa7ffff640 1 -- 172.21.15.40:0/983304688 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac5f4277d0 con 0x55ac5f6e8e60 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa86d9d640 1 --2- 172.21.15.40:0/983304688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6e9850 0x55ac5f6f3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa7ffff640 1 --2- 172.21.15.40:0/983304688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6f33c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ffa70001eb0 tx=0x7ffa70010b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac41fc250f1c8f94 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.953+0000 7ffa6e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.954+0000 7ffa6e7fc640 1 -- 172.21.15.40:0/983304688 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa70028040 con 0x55ac5f6e8e60 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.954+0000 7ffa8fe23740 1 -- 172.21.15.40:0/983304688 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa70010430 con 0x55ac5f6e8e60 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.954+0000 7ffa6e7fc640 1 -- 172.21.15.40:0/983304688 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ffa7000fd70 con 0x55ac5f6e8e60 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.954+0000 7ffa6e7fc640 1 -- 172.21.15.40:0/983304688 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffa7002f430 con 0x55ac5f6e8e60 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.954+0000 7ffa8fe23740 10 client.26819 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.954+0000 7ffa8fe23740 1 -- 172.21.15.40:0/983304688 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ac5f35d3f0 con 0x55ac5f6e8e60 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.954+0000 7ffa6e7fc640 1 -- 172.21.15.40:0/983304688 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffa70010430 con 0x55ac5f6e8e60 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.954+0000 7ffa6e7fc640 1 -- 172.21.15.40:0/983304688 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ffa7001f3f0 con 0x55ac5f6e8e60 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.954+0000 7ffa8fe23740 10 client.26819 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.954+0000 7ffa8fe23740 10 client.26819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.954+0000 7ffa8fe23740 20 client.26819 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.954+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/983304688 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ac5f6feed0 0x55ac5f701300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.954+0000 7ffa8fe23740 4 client.26819 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:05.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.954+0000 7ffa8fe23740 1 -- 172.21.15.40:0/983304688 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac5f656a40 con 0x55ac5f6feed0 2024-03-18T22:48:05.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.955+0000 7ffa86d9d640 1 --2- 172.21.15.40:0/983304688 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ac5f6feed0 0x55ac5f701300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.955+0000 7ffa86d9d640 1 --2- 172.21.15.40:0/983304688 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ac5f6feed0 0x55ac5f701300 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46059fa99e5531eb server_cookie=a3e452c6365df489 in_seq=0 out_seq=0 2024-03-18T22:48:05.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.955+0000 7ffa6e7fc640 10 client.26819 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:05.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.956+0000 7ffa6e7fc640 1 -- 172.21.15.40:0/983304688 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ac5f656a40 con 0x55ac5f6feed0 2024-03-18T22:48:05.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.956+0000 7ffa6e7fc640 10 client.26819 handle_command_reply: tid=0 2024-03-18T22:48:05.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.956+0000 7ffa8fe23740 1 client.26819 shutdown 2024-03-18T22:48:05.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.956+0000 7ffa8fe23740 1 -- 172.21.15.40:0/983304688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e8e60 msgr2=0x55ac5f6f33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.956+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/983304688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6f33c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ffa70001eb0 tx=0x7ffa70010b00 comp rx=0 tx=0).stop 2024-03-18T22:48:05.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.956+0000 7ffa8fe23740 1 -- 172.21.15.40:0/983304688 shutdown_connections 2024-03-18T22:48:05.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.956+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/983304688 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f6fa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.956+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/983304688 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6e9850 0x55ac5f6f3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.956+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/983304688 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ac5f6feed0 0x55ac5f701300 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.956+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/983304688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6f33c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.956+0000 7ffa6e7fc640 0 client.26819 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:05.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.956+0000 7ffa8fe23740 1 -- 172.21.15.40:0/983304688 >> 172.21.15.40:0/983304688 conn(0x55ac5f6dffc0 msgr2=0x55ac5f6df1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:05.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.956+0000 7ffa8fe23740 1 -- 172.21.15.40:0/983304688 shutdown_connections 2024-03-18T22:48:05.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.956+0000 7ffa8fe23740 1 -- 172.21.15.40:0/983304688 wait complete. 2024-03-18T22:48:05.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.956+0000 7ffa8fe23740 20 client.26819 trim_cache size 0 max 16384 2024-03-18T22:48:05.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.979+0000 7ffa8fe23740 1 Processor -- start 2024-03-18T22:48:05.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.979+0000 7ffa8fe23740 1 -- start start 2024-03-18T22:48:05.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.979+0000 7ffa8fe23740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6e3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.979+0000 7ffa8fe23740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6e9850 0x55ac5f6fa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.979+0000 7ffa8fe23740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f705bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.979+0000 7ffa8fe23740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ffa881a8270 con 0x55ac5f6ee9c0 2024-03-18T22:48:05.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.979+0000 7ffa8fe23740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa881a9db0 con 0x55ac5f6e8e60 2024-03-18T22:48:05.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.979+0000 7ffa8fe23740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ffa881a8940 con 0x55ac5f6e9850 2024-03-18T22:48:05.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.979+0000 7ffa7ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f705bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.979+0000 7ffa7ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f705bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47078/0 (socket says 172.21.15.40:47078) 2024-03-18T22:48:05.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.979+0000 7ffa86d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6e9850 0x55ac5f6fa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.979+0000 7ffa7ffff640 1 -- 172.21.15.40:0/2321249397 learned_addr learned my addr 172.21.15.40:0/2321249397 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:05.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.979+0000 7ffa7ffff640 1 --2- 172.21.15.40:0/2321249397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6e3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.980+0000 7ffa86d9d640 1 -- 172.21.15.40:0/2321249397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e8e60 msgr2=0x55ac5f6e3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:05.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.980+0000 7ffa86d9d640 1 --2- 172.21.15.40:0/2321249397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6e3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.980+0000 7ffa86d9d640 1 -- 172.21.15.40:0/2321249397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6ee9c0 msgr2=0x55ac5f705bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:05.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.980+0000 7ffa86d9d640 1 --2- 172.21.15.40:0/2321249397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f705bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.980+0000 7ffa86d9d640 1 -- 172.21.15.40:0/2321249397 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac5f31d0f0 con 0x55ac5f6e9850 2024-03-18T22:48:05.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.980+0000 7ffa7ffff640 1 --2- 172.21.15.40:0/2321249397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f705bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:05.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.980+0000 7ffa86d9d640 1 --2- 172.21.15.40:0/2321249397 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6e9850 0x55ac5f6fa8d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x55ac5f701b70 tx=0x7ffa78005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4db6c3d97d0e3faf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:05.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.980+0000 7ffa7cff9640 1 -- 172.21.15.40:0/2321249397 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa78024030 con 0x55ac5f6e9850 2024-03-18T22:48:05.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.980+0000 7ffa7cff9640 1 -- 172.21.15.40:0/2321249397 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffa780054d0 con 0x55ac5f6e9850 2024-03-18T22:48:05.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.980+0000 7ffa7cff9640 1 -- 172.21.15.40:0/2321249397 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa78020110 con 0x55ac5f6e9850 2024-03-18T22:48:05.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.981+0000 7ffa8fe23740 1 -- 172.21.15.40:0/2321249397 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6e9850 msgr2=0x55ac5f6fa8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.981+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/2321249397 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6e9850 0x55ac5f6fa8d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x55ac5f701b70 tx=0x7ffa78005f90 comp rx=0 tx=0).stop 2024-03-18T22:48:05.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.981+0000 7ffa8fe23740 1 -- 172.21.15.40:0/2321249397 shutdown_connections 2024-03-18T22:48:05.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.981+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/2321249397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f705bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.981+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/2321249397 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6e9850 0x55ac5f6fa8d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.981+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/2321249397 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6e3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.981+0000 7ffa8fe23740 1 -- 172.21.15.40:0/2321249397 >> 172.21.15.40:0/2321249397 conn(0x55ac5f6dffc0 msgr2=0x55ac5f6e3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:05.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.981+0000 7ffa8fe23740 1 -- 172.21.15.40:0/2321249397 shutdown_connections 2024-03-18T22:48:05.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.981+0000 7ffa8fe23740 1 -- 172.21.15.40:0/2321249397 wait complete. 2024-03-18T22:48:05.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.981+0000 7ffa8fe23740 1 -- start start 2024-03-18T22:48:05.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.981+0000 7ffa8fe23740 1 Processor -- start 2024-03-18T22:48:05.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa8fe23740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6f7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa8fe23740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e9850 0x55ac5f6f8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa8fe23740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f6f93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa8fe23740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ffa881a96e0 con 0x55ac5f6e8e60 2024-03-18T22:48:05.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa8fe23740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa881a9010 con 0x55ac5f6e9850 2024-03-18T22:48:05.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa8fe23740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ffa881a6250 con 0x55ac5f6ee9c0 2024-03-18T22:48:05.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa8fe23740 10 client.0 _ll_register_callbacks cb 0x7ffa70013970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:05.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa7ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6f7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa7ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6f7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47080/0 (socket says 172.21.15.40:47080) 2024-03-18T22:48:05.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa7ffff640 1 -- 172.21.15.40:0/2059776931 learned_addr learned my addr 172.21.15.40:0/2059776931 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:05.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa86d9d640 1 --2- 172.21.15.40:0/2059776931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e9850 0x55ac5f6f8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa7ffff640 1 --2- 172.21.15.40:0/2059776931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f6f93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa86d9d640 1 -- 172.21.15.40:0/2059776931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6ee9c0 msgr2=0x55ac5f6f93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:05.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa86d9d640 1 --2- 172.21.15.40:0/2059776931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f6f93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa86d9d640 1 -- 172.21.15.40:0/2059776931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6e8e60 msgr2=0x55ac5f6f7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:05.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa86d9d640 1 --2- 172.21.15.40:0/2059776931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6f7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa86d9d640 1 -- 172.21.15.40:0/2059776931 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac5f6f99a0 con 0x55ac5f6e9850 2024-03-18T22:48:05.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa7ffff640 1 --2- 172.21.15.40:0/2059776931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6f7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:05.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa86d9d640 1 --2- 172.21.15.40:0/2059776931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e9850 0x55ac5f6f8460 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55ac5f6f4380 tx=0x7ffa78009d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43bb12b7cea89b9d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:05.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa6e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:48:05.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa6e7fc640 1 -- 172.21.15.40:0/2059776931 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7802d070 con 0x55ac5f6e9850 2024-03-18T22:48:05.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.982+0000 7ffa8fe23740 1 -- 172.21.15.40:0/2059776931 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa780054d0 con 0x55ac5f6e9850 2024-03-18T22:48:05.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.983+0000 7ffa6e7fc640 1 -- 172.21.15.40:0/2059776931 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ffa78004ee0 con 0x55ac5f6e9850 2024-03-18T22:48:05.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.983+0000 7ffa6e7fc640 1 -- 172.21.15.40:0/2059776931 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffa7803b050 con 0x55ac5f6e9850 2024-03-18T22:48:05.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.983+0000 7ffa8fe23740 10 client.26825 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:05.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.983+0000 7ffa8fe23740 1 -- 172.21.15.40:0/2059776931 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ac5f765200 con 0x55ac5f6e9850 2024-03-18T22:48:05.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.983+0000 7ffa6e7fc640 1 -- 172.21.15.40:0/2059776931 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffa780054d0 con 0x55ac5f6e9850 2024-03-18T22:48:05.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.983+0000 7ffa6e7fc640 1 -- 172.21.15.40:0/2059776931 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7ffa78032460 con 0x55ac5f6e9850 2024-03-18T22:48:05.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.983+0000 7ffa8fe23740 10 client.26825 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:05.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.983+0000 7ffa8fe23740 10 client.26825 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:05.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.983+0000 7ffa8fe23740 20 client.26825 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:05.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.983+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/2059776931 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ac5f765a20 0x55ac5f767e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:05.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.983+0000 7ffa8fe23740 4 client.26825 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:48:05.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.983+0000 7ffa8fe23740 1 -- 172.21.15.40:0/2059776931 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ac5f650d90 con 0x55ac5f765a20 2024-03-18T22:48:05.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.983+0000 7ffa7ffff640 1 --2- 172.21.15.40:0/2059776931 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ac5f765a20 0x55ac5f767e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa6e7fc640 10 client.26825 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa7ffff640 1 --2- 172.21.15.40:0/2059776931 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ac5f765a20 0x55ac5f767e40 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc57cedda040d8ae server_cookie=fe665109e50f61fa in_seq=0 out_seq=0 2024-03-18T22:48:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa6e7fc640 1 -- 172.21.15.40:0/2059776931 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ac5f650d90 con 0x55ac5f765a20 2024-03-18T22:48:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa6e7fc640 10 client.26825 handle_command_reply: tid=0 2024-03-18T22:48:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa8fe23740 1 client.26825 shutdown 2024-03-18T22:48:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa8fe23740 1 -- 172.21.15.40:0/2059776931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e9850 msgr2=0x55ac5f6f8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/2059776931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e9850 0x55ac5f6f8460 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55ac5f6f4380 tx=0x7ffa78009d70 comp rx=0 tx=0).stop 2024-03-18T22:48:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa8fe23740 1 -- 172.21.15.40:0/2059776931 shutdown_connections 2024-03-18T22:48:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/2059776931 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ac5f6ee9c0 0x55ac5f6f93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/2059776931 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ac5f765a20 0x55ac5f767e40 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/2059776931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ac5f6e9850 0x55ac5f6f8460 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa8fe23740 1 --2- 172.21.15.40:0/2059776931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ac5f6e8e60 0x55ac5f6f7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa6e7fc640 0 client.26825 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa8fe23740 1 -- 172.21.15.40:0/2059776931 >> 172.21.15.40:0/2059776931 conn(0x55ac5f6dffc0 msgr2=0x55ac5f6e3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:05.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa8fe23740 1 -- 172.21.15.40:0/2059776931 shutdown_connections 2024-03-18T22:48:05.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa8fe23740 1 -- 172.21.15.40:0/2059776931 wait complete. 2024-03-18T22:48:05.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.984+0000 7ffa8fe23740 20 client.26825 trim_cache size 0 max 16384 2024-03-18T22:48:05.992 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:48:05.992 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:48:05.992 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:48:05.992 INFO:teuthology.orchestra.run.smithi040.stdout: "419d4681-27e3-4e6d-9f86-cc86e518d421": { 2024-03-18T22:48:05.992 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:48:05.992 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "419d4681-27e3-4e6d-9f86-cc86e518d421", 2024-03-18T22:48:05.992 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:48:05.992 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:48:05.992 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:48:05.992 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:48:05.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.985+0000 7ffa8d15b640 1 -- 172.21.15.40:0/2749688727 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ffa740a2800 msgr2=0x7ffa740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.985+0000 7ffa8d15b640 1 --2- 172.21.15.40:0/2749688727 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ffa740a2800 0x7ffa740a4cc0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ffa800bd290 tx=0x7ffa80088040 comp rx=0 tx=0).stop 2024-03-18T22:48:05.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.985+0000 7ffa8d15b640 1 -- 172.21.15.40:0/2749688727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa881022d0 msgr2=0x7ffa8819efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:05.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.985+0000 7ffa8d15b640 1 --2- 172.21.15.40:0/2749688727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa881022d0 0x7ffa8819efd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffa8002c630 tx=0x7ffa800328a0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.985+0000 7ffa8d15b640 1 -- 172.21.15.40:0/2749688727 shutdown_connections 2024-03-18T22:48:05.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.985+0000 7ffa8d15b640 1 --2- 172.21.15.40:0/2749688727 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ffa740a2800 0x7ffa740a4cc0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.985+0000 7ffa8d15b640 1 --2- 172.21.15.40:0/2749688727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa881022d0 0x7ffa8819efd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.985+0000 7ffa8d15b640 1 --2- 172.21.15.40:0/2749688727 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ffa88101910 0x7ffa88197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.985+0000 7ffa8d15b640 1 --2- 172.21.15.40:0/2749688727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa88100720 0x7ffa88197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:05.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.985+0000 7ffa8d15b640 1 -- 172.21.15.40:0/2749688727 >> 172.21.15.40:0/2749688727 conn(0x7ffa880fbcb0 msgr2=0x7ffa88068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:05.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.985+0000 7ffa8d15b640 1 -- 172.21.15.40:0/2749688727 shutdown_connections 2024-03-18T22:48:05.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:05.985+0000 7ffa8d15b640 1 -- 172.21.15.40:0/2749688727 wait complete. 2024-03-18T22:48:05.998 INFO:tasks.cephfs.filesystem:scrub status for tag:419d4681-27e3-4e6d-9f86-cc86e518d421 - {'path': '/', 'tag': '419d4681-27e3-4e6d-9f86-cc86e518d421', 'options': 'recursive,force'} 2024-03-18T22:48:06.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:06 smithi040 ceph-mon[62643]: pgmap v514: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:48:06.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:06 smithi113 ceph-mon[66775]: pgmap v514: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:48:06.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:06 smithi103 ceph-mon[66377]: pgmap v514: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:48:07.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:07 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:48:07.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:07 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:07.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:07 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:48:07.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:07 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:48:07.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:07 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:07.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:07 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:07.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:07 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:48:07.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:07 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:07.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:07 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:48:07.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:07 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:48:07.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:07 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:07.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:07 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:07.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:07 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:48:07.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:07 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:07.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:07 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:48:07.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:07 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:48:07.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:07 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:07.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:07 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:08.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:08 smithi113 ceph-mon[66775]: pgmap v515: 129 pgs: 129 active+clean; 6.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:48:08.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:08 smithi103 ceph-mon[66377]: pgmap v515: 129 pgs: 129 active+clean; 6.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:48:08.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:08 smithi040 ceph-mon[62643]: pgmap v515: 129 pgs: 129 active+clean; 6.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:48:10.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:10 smithi113 ceph-mon[66775]: pgmap v516: 129 pgs: 129 active+clean; 6.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:48:10.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:10 smithi103 ceph-mon[66377]: pgmap v516: 129 pgs: 129 active+clean; 6.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:48:10.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:10 smithi040 ceph-mon[62643]: pgmap v516: 129 pgs: 129 active+clean; 6.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:48:12.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:12 smithi113 ceph-mon[66775]: pgmap v517: 129 pgs: 129 active+clean; 6.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:48:12.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:12 smithi103 ceph-mon[66377]: pgmap v517: 129 pgs: 129 active+clean; 6.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:48:12.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:12 smithi040 ceph-mon[62643]: pgmap v517: 129 pgs: 129 active+clean; 6.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:48:13.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:13 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:48:13.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:13 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:48:13.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:13 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:48:14.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:14 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:48:14.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:14 smithi113 ceph-mon[66775]: pgmap v518: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:48:14.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:14 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:48:14.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:14 smithi103 ceph-mon[66377]: pgmap v518: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:48:14.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:14 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:48:14.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:14 smithi040 ceph-mon[62643]: pgmap v518: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:48:16.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:16 smithi113 ceph-mon[66775]: pgmap v519: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:48:16.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:16 smithi103 ceph-mon[66377]: pgmap v519: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:48:16.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:16 smithi040 ceph-mon[62643]: pgmap v519: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:48:18.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:18 smithi113 ceph-mon[66775]: pgmap v520: 129 pgs: 129 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:48:18.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:18 smithi103 ceph-mon[66377]: pgmap v520: 129 pgs: 129 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:48:18.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:18 smithi040 ceph-mon[62643]: pgmap v520: 129 pgs: 129 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:48:20.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:20 smithi113 ceph-mon[66775]: pgmap v521: 129 pgs: 129 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-03-18T22:48:20.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:20 smithi103 ceph-mon[66377]: pgmap v521: 129 pgs: 129 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-03-18T22:48:20.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:20 smithi040 ceph-mon[62643]: pgmap v521: 129 pgs: 129 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-03-18T22:48:21.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:21 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:48:21.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:21 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:48:21.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:21 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:48:22.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:22 smithi113 ceph-mon[66775]: pgmap v522: 129 pgs: 129 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:48:22.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:22 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:48:22.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:22 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:48:22.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:22 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:48:22.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:22 smithi103 ceph-mon[66377]: pgmap v522: 129 pgs: 129 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:48:22.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:22 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:48:22.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:22 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:48:22.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:22 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:48:22.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:22 smithi040 ceph-mon[62643]: pgmap v522: 129 pgs: 129 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:48:22.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:22 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:48:22.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:22 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:48:22.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:22 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:48:24.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:24 smithi113 ceph-mon[66775]: pgmap v523: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:48:24.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:24 smithi103 ceph-mon[66377]: pgmap v523: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:48:24.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:24 smithi040 ceph-mon[62643]: pgmap v523: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:48:26.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:26 smithi113 ceph-mon[66775]: pgmap v524: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:48:26.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:26 smithi103 ceph-mon[66377]: pgmap v524: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:48:26.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:26 smithi040 ceph-mon[62643]: pgmap v524: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:48:28.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:28 smithi113 ceph-mon[66775]: pgmap v525: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:48:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:28 smithi103 ceph-mon[66377]: pgmap v525: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:48:28.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:28 smithi040 ceph-mon[62643]: pgmap v525: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:48:30.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:30 smithi113 ceph-mon[66775]: pgmap v526: 129 pgs: 129 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-03-18T22:48:30.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:30 smithi103 ceph-mon[66377]: pgmap v526: 129 pgs: 129 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-03-18T22:48:30.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:30 smithi040 ceph-mon[62643]: pgmap v526: 129 pgs: 129 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 25 MiB/s wr, 20 op/s 2024-03-18T22:48:32.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:32 smithi113 ceph-mon[66775]: pgmap v527: 129 pgs: 129 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:48:32.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:32 smithi103 ceph-mon[66377]: pgmap v527: 129 pgs: 129 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:48:32.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:32 smithi040 ceph-mon[62643]: pgmap v527: 129 pgs: 129 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:48:34.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:34 smithi113 ceph-mon[66775]: pgmap v528: 129 pgs: 129 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:48:34.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:34 smithi103 ceph-mon[66377]: pgmap v528: 129 pgs: 129 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:48:34.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:34 smithi040 ceph-mon[62643]: pgmap v528: 129 pgs: 129 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:48:35.999 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:48:35.999 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:48:36.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.122+0000 7fa6336dc640 1 -- 172.21.15.40:0/1945551391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa62c100760 msgr2=0x7fa62c100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.122+0000 7fa6336dc640 1 --2- 172.21.15.40:0/1945551391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa62c100760 0x7fa62c100b80 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa628009900 tx=0x7fa62802bc30 comp rx=0 tx=0).stop 2024-03-18T22:48:36.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.122+0000 7fa6336dc640 1 -- 172.21.15.40:0/1945551391 shutdown_connections 2024-03-18T22:48:36.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.122+0000 7fa6336dc640 1 --2- 172.21.15.40:0/1945551391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa62c102310 0x7fa62c108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.122+0000 7fa6336dc640 1 --2- 172.21.15.40:0/1945551391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa62c101950 0x7fa62c101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.122+0000 7fa6336dc640 1 --2- 172.21.15.40:0/1945551391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa62c100760 0x7fa62c100b80 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.122+0000 7fa6336dc640 1 -- 172.21.15.40:0/1945551391 >> 172.21.15.40:0/1945551391 conn(0x7fa62c0fbcd0 msgr2=0x7fa62c0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.123+0000 7fa6336dc640 1 -- 172.21.15.40:0/1945551391 shutdown_connections 2024-03-18T22:48:36.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.123+0000 7fa6336dc640 1 -- 172.21.15.40:0/1945551391 wait complete. 2024-03-18T22:48:36.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.123+0000 7fa6336dc640 1 Processor -- start 2024-03-18T22:48:36.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.123+0000 7fa6336dc640 1 -- start start 2024-03-18T22:48:36.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.123+0000 7fa6336dc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa62c100760 0x7fa62c196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.123+0000 7fa6336dc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa62c101950 0x7fa62c196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.123+0000 7fa6336dc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa62c102310 0x7fa62c19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.123+0000 7fa6336dc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c10c260 con 0x7fa62c100760 2024-03-18T22:48:36.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.123+0000 7fa6336dc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c10c0e0 con 0x7fa62c101950 2024-03-18T22:48:36.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.123+0000 7fa6336dc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c10c3e0 con 0x7fa62c102310 2024-03-18T22:48:36.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa631478640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa62c100760 0x7fa62c196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa631478640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa62c100760 0x7fa62c196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45582/0 (socket says 172.21.15.40:45582) 2024-03-18T22:48:36.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa631c79640 1 --2- 172.21.15.40:0/2812065928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa62c102310 0x7fa62c19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa631478640 1 -- 172.21.15.40:0/2812065928 learned_addr learned my addr 172.21.15.40:0/2812065928 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa630c77640 1 --2- 172.21.15.40:0/2812065928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa62c101950 0x7fa62c196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa631478640 1 -- 172.21.15.40:0/2812065928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa62c102310 msgr2=0x7fa62c19dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa631478640 1 --2- 172.21.15.40:0/2812065928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa62c102310 0x7fa62c19dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa631478640 1 -- 172.21.15.40:0/2812065928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa62c101950 msgr2=0x7fa62c196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa631478640 1 --2- 172.21.15.40:0/2812065928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa62c101950 0x7fa62c196f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa631478640 1 -- 172.21.15.40:0/2812065928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c19e650 con 0x7fa62c100760 2024-03-18T22:48:36.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa630c77640 1 --2- 172.21.15.40:0/2812065928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa62c101950 0x7fa62c196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa631478640 1 --2- 172.21.15.40:0/2812065928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa62c100760 0x7fa62c196a30 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fa628000c00 tx=0x7fa628005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa6227fc640 1 -- 172.21.15.40:0/2812065928 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628043020 con 0x7fa62c100760 2024-03-18T22:48:36.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa6336dc640 1 -- 172.21.15.40:0/2812065928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62c19e8e0 con 0x7fa62c100760 2024-03-18T22:48:36.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa6227fc640 1 -- 172.21.15.40:0/2812065928 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6280042d0 con 0x7fa62c100760 2024-03-18T22:48:36.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa6227fc640 1 -- 172.21.15.40:0/2812065928 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628003b10 con 0x7fa62c100760 2024-03-18T22:48:36.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.124+0000 7fa6336dc640 1 -- 172.21.15.40:0/2812065928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa62c19ee30 con 0x7fa62c100760 2024-03-18T22:48:36.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.127+0000 7fa6227fc640 1 -- 172.21.15.40:0/2812065928 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa628046020 con 0x7fa62c100760 2024-03-18T22:48:36.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.127+0000 7fa63639f740 1 Processor -- start 2024-03-18T22:48:36.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.127+0000 7fa63639f740 1 -- start start 2024-03-18T22:48:36.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.127+0000 7fa63639f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b5c60 0x56070a1b6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.127+0000 7fa6227fc640 1 --2- 172.21.15.40:0/2812065928 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa6180a2800 0x7fa6180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.127+0000 7fa63639f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b6650 0x56070a1bb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa63639f740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1bb7c0 0x56070a1bdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa63639f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56070a118500 con 0x56070a1b6650 2024-03-18T22:48:36.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa63639f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56070a0c0be0 con 0x56070a1b5c60 2024-03-18T22:48:36.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa63639f740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56070a1145a0 con 0x56070a1bb7c0 2024-03-18T22:48:36.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa631c79640 1 --2- 172.21.15.40:0/2812065928 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa6180a2800 0x7fa6180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa630c77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b6650 0x56070a1bb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa631478640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b5c60 0x56070a1b6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa630c77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b6650 0x56070a1bb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45598/0 (socket says 172.21.15.40:45598) 2024-03-18T22:48:36.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa630c77640 1 -- 172.21.15.40:0/1053342833 learned_addr learned my addr 172.21.15.40:0/1053342833 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa631c79640 1 --2- 172.21.15.40:0/1053342833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1bb7c0 0x56070a1bdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa6227fc640 1 -- 172.21.15.40:0/2812065928 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa6280f0f20 con 0x7fa62c100760 2024-03-18T22:48:36.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa631478640 1 -- 172.21.15.40:0/1053342833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1bb7c0 msgr2=0x56070a1bdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa631478640 1 --2- 172.21.15.40:0/1053342833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1bb7c0 0x56070a1bdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa631478640 1 -- 172.21.15.40:0/1053342833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b6650 msgr2=0x56070a1bb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa631478640 1 --2- 172.21.15.40:0/1053342833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b6650 0x56070a1bb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa631478640 1 -- 172.21.15.40:0/1053342833 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56070a0e2000 con 0x56070a1b5c60 2024-03-18T22:48:36.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa630c77640 1 --2- 172.21.15.40:0/1053342833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b6650 0x56070a1bb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa631c79640 1 --2- 172.21.15.40:0/2812065928 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa6180a2800 0x7fa6180a4cc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa624005fd0 tx=0x7fa624009e60 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.128+0000 7fa631478640 1 --2- 172.21.15.40:0/1053342833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b5c60 0x56070a1b6080 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa6280bd1d0 tx=0x7fa62803ac10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8279b4ed7ec62688 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.129+0000 7fa6177fe640 1 -- 172.21.15.40:0/1053342833 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6280bce30 con 0x56070a1b5c60 2024-03-18T22:48:36.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.129+0000 7fa6177fe640 1 -- 172.21.15.40:0/1053342833 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6280bdbd0 con 0x56070a1b5c60 2024-03-18T22:48:36.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.129+0000 7fa6177fe640 1 -- 172.21.15.40:0/1053342833 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6280bb7d0 con 0x56070a1b5c60 2024-03-18T22:48:36.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.129+0000 7fa63639f740 1 -- 172.21.15.40:0/1053342833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b5c60 msgr2=0x56070a1b6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.129+0000 7fa63639f740 1 --2- 172.21.15.40:0/1053342833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b5c60 0x56070a1b6080 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa6280bd1d0 tx=0x7fa62803ac10 comp rx=0 tx=0).stop 2024-03-18T22:48:36.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.129+0000 7fa63639f740 1 -- 172.21.15.40:0/1053342833 shutdown_connections 2024-03-18T22:48:36.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.129+0000 7fa63639f740 1 --2- 172.21.15.40:0/1053342833 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1bb7c0 0x56070a1bdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.129+0000 7fa63639f740 1 --2- 172.21.15.40:0/1053342833 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b6650 0x56070a1bb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.129+0000 7fa63639f740 1 --2- 172.21.15.40:0/1053342833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b5c60 0x56070a1b6080 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.129+0000 7fa63639f740 1 -- 172.21.15.40:0/1053342833 >> 172.21.15.40:0/1053342833 conn(0x56070a1aefc0 msgr2=0x56070a1af3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.129+0000 7fa63639f740 1 -- 172.21.15.40:0/1053342833 shutdown_connections 2024-03-18T22:48:36.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.129+0000 7fa63639f740 1 -- 172.21.15.40:0/1053342833 wait complete. 2024-03-18T22:48:36.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.130+0000 7fa63639f740 1 -- start start 2024-03-18T22:48:36.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.130+0000 7fa63639f740 1 Processor -- start 2024-03-18T22:48:36.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.130+0000 7fa63639f740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1b5c60 0x56070a1bfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.130+0000 7fa63639f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b6650 0x56070a1c03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.130+0000 7fa63639f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1bb7c0 0x56070a1c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.130+0000 7fa63639f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56070a1be530 con 0x56070a1bb7c0 2024-03-18T22:48:36.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.130+0000 7fa63639f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56070a0d4160 con 0x56070a1b6650 2024-03-18T22:48:36.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.130+0000 7fa63639f740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56070a1be6b0 con 0x56070a1b5c60 2024-03-18T22:48:36.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.130+0000 7fa63639f740 10 client.0 _ll_register_callbacks cb 0x56070a110a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:36.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.130+0000 7fa631478640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1b5c60 0x56070a1bfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.130+0000 7fa630c77640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b6650 0x56070a1c03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.130+0000 7fa631478640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1b5c60 0x56070a1bfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38660/0 (socket says 172.21.15.40:38660) 2024-03-18T22:48:36.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.130+0000 7fa631c79640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1bb7c0 0x56070a1c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.130+0000 7fa631478640 1 -- 172.21.15.40:0/3126622926 learned_addr learned my addr 172.21.15.40:0/3126622926 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa631478640 1 -- 172.21.15.40:0/3126622926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b6650 msgr2=0x56070a1c03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa631478640 1 --2- 172.21.15.40:0/3126622926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b6650 0x56070a1c03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa631478640 1 -- 172.21.15.40:0/3126622926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1bb7c0 msgr2=0x56070a1c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa631478640 1 --2- 172.21.15.40:0/3126622926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1bb7c0 0x56070a1c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa631478640 1 -- 172.21.15.40:0/3126622926 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560709ef67d0 con 0x56070a1b5c60 2024-03-18T22:48:36.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa630c77640 1 --2- 172.21.15.40:0/3126622926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b6650 0x56070a1c03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa631c79640 1 --2- 172.21.15.40:0/3126622926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1bb7c0 0x56070a1c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa631478640 1 --2- 172.21.15.40:0/3126622926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1b5c60 0x56070a1bfe70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa6280ee9a0 tx=0x7fa628009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c962328df644a61b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa614ff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:48:36.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa614ff9640 1 -- 172.21.15.40:0/3126622926 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6280d6060 con 0x56070a1b5c60 2024-03-18T22:48:36.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa614ff9640 1 -- 172.21.15.40:0/3126622926 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa6280d6200 con 0x56070a1b5c60 2024-03-18T22:48:36.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa63639f740 1 -- 172.21.15.40:0/3126622926 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6280bdbd0 con 0x56070a1b5c60 2024-03-18T22:48:36.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa614ff9640 1 -- 172.21.15.40:0/3126622926 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa628086070 con 0x56070a1b5c60 2024-03-18T22:48:36.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa63639f740 10 client.26827 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:36.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa63639f740 1 -- 172.21.15.40:0/3126622926 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560709e2c3f0 con 0x56070a1b5c60 2024-03-18T22:48:36.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.131+0000 7fa614ff9640 1 -- 172.21.15.40:0/3126622926 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6280bdbd0 con 0x56070a1b5c60 2024-03-18T22:48:36.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.132+0000 7fa614ff9640 1 -- 172.21.15.40:0/3126622926 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa6280381c0 con 0x56070a1b5c60 2024-03-18T22:48:36.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.132+0000 7fa63639f740 10 client.26827 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:36.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.132+0000 7fa63639f740 10 client.26827 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:36.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.132+0000 7fa63639f740 20 client.26827 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:36.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.132+0000 7fa63639f740 1 --2- 172.21.15.40:0/3126622926 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56070a1cba60 0x56070a1cde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.132+0000 7fa63639f740 4 client.26827 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:36.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.132+0000 7fa63639f740 1 -- 172.21.15.40:0/3126622926 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56070a125a40 con 0x56070a1cba60 2024-03-18T22:48:36.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.132+0000 7fa630c77640 1 --2- 172.21.15.40:0/3126622926 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56070a1cba60 0x56070a1cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.132+0000 7fa630c77640 1 --2- 172.21.15.40:0/3126622926 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56070a1cba60 0x56070a1cde90 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=573e7313acf1dfdf server_cookie=257f6ac00a75c543 in_seq=0 out_seq=0 2024-03-18T22:48:36.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.132+0000 7fa614ff9640 10 client.26827 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:36.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.133+0000 7fa614ff9640 1 -- 172.21.15.40:0/3126622926 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56070a125a40 con 0x56070a1cba60 2024-03-18T22:48:36.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.133+0000 7fa614ff9640 10 client.26827 handle_command_reply: tid=0 2024-03-18T22:48:36.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.133+0000 7fa63639f740 1 client.26827 shutdown 2024-03-18T22:48:36.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.133+0000 7fa63639f740 1 -- 172.21.15.40:0/3126622926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1b5c60 msgr2=0x56070a1bfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.133+0000 7fa63639f740 1 --2- 172.21.15.40:0/3126622926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1b5c60 0x56070a1bfe70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa6280ee9a0 tx=0x7fa628009260 comp rx=0 tx=0).stop 2024-03-18T22:48:36.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.133+0000 7fa63639f740 1 -- 172.21.15.40:0/3126622926 shutdown_connections 2024-03-18T22:48:36.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.133+0000 7fa63639f740 1 --2- 172.21.15.40:0/3126622926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1bb7c0 0x56070a1c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.133+0000 7fa63639f740 1 --2- 172.21.15.40:0/3126622926 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56070a1cba60 0x56070a1cde90 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.134+0000 7fa63639f740 1 --2- 172.21.15.40:0/3126622926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b6650 0x56070a1c03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.134+0000 7fa63639f740 1 --2- 172.21.15.40:0/3126622926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1b5c60 0x56070a1bfe70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.134+0000 7fa614ff9640 0 client.26827 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:36.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.134+0000 7fa63639f740 1 -- 172.21.15.40:0/3126622926 >> 172.21.15.40:0/3126622926 conn(0x56070a1aefc0 msgr2=0x56070a1ae1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.134+0000 7fa63639f740 1 -- 172.21.15.40:0/3126622926 shutdown_connections 2024-03-18T22:48:36.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.134+0000 7fa63639f740 1 -- 172.21.15.40:0/3126622926 wait complete. 2024-03-18T22:48:36.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.134+0000 7fa63639f740 20 client.26827 trim_cache size 0 max 16384 2024-03-18T22:48:36.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.152+0000 7fa63639f740 1 Processor -- start 2024-03-18T22:48:36.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.152+0000 7fa63639f740 1 -- start start 2024-03-18T22:48:36.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.152+0000 7fa63639f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b5c60 0x56070a1c0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.152+0000 7fa63639f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b6650 0x56070a1ad950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.152+0000 7fa63639f740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1bb7c0 0x56070a1ade90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.152+0000 7fa63639f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1a83b0 con 0x56070a1b6650 2024-03-18T22:48:36.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.152+0000 7fa63639f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1a9ef0 con 0x56070a1b5c60 2024-03-18T22:48:36.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.152+0000 7fa63639f740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1a8a80 con 0x56070a1bb7c0 2024-03-18T22:48:36.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa630c77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b6650 0x56070a1ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa631478640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b5c60 0x56070a1c0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa630c77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b6650 0x56070a1ad950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45612/0 (socket says 172.21.15.40:45612) 2024-03-18T22:48:36.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa630c77640 1 -- 172.21.15.40:0/725344726 learned_addr learned my addr 172.21.15.40:0/725344726 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa631c79640 1 --2- 172.21.15.40:0/725344726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1bb7c0 0x56070a1ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa630c77640 1 -- 172.21.15.40:0/725344726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1bb7c0 msgr2=0x56070a1ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa630c77640 1 --2- 172.21.15.40:0/725344726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1bb7c0 0x56070a1ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa630c77640 1 -- 172.21.15.40:0/725344726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b5c60 msgr2=0x56070a1c0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa630c77640 1 --2- 172.21.15.40:0/725344726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b5c60 0x56070a1c0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa630c77640 1 -- 172.21.15.40:0/725344726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560709dec0f0 con 0x56070a1b6650 2024-03-18T22:48:36.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa631c79640 1 --2- 172.21.15.40:0/725344726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1bb7c0 0x56070a1ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:48:36.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa631478640 1 --2- 172.21.15.40:0/725344726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b5c60 0x56070a1c0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa630c77640 1 --2- 172.21.15.40:0/725344726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b6650 0x56070a1ad950 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x56070a1b6a50 tx=0x7fa61c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ea07b003959ef4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa616ffd640 1 -- 172.21.15.40:0/725344726 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c00fc80 con 0x56070a1b6650 2024-03-18T22:48:36.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa616ffd640 1 -- 172.21.15.40:0/725344726 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa61c00fe20 con 0x56070a1b6650 2024-03-18T22:48:36.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.153+0000 7fa616ffd640 1 -- 172.21.15.40:0/725344726 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c0227a0 con 0x56070a1b6650 2024-03-18T22:48:36.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.154+0000 7fa63639f740 1 -- 172.21.15.40:0/725344726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b6650 msgr2=0x56070a1ad950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.154+0000 7fa63639f740 1 --2- 172.21.15.40:0/725344726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b6650 0x56070a1ad950 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x56070a1b6a50 tx=0x7fa61c005f90 comp rx=0 tx=0).stop 2024-03-18T22:48:36.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.154+0000 7fa63639f740 1 -- 172.21.15.40:0/725344726 shutdown_connections 2024-03-18T22:48:36.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.154+0000 7fa63639f740 1 --2- 172.21.15.40:0/725344726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1bb7c0 0x56070a1ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.154+0000 7fa63639f740 1 --2- 172.21.15.40:0/725344726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b6650 0x56070a1ad950 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.154+0000 7fa63639f740 1 --2- 172.21.15.40:0/725344726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1b5c60 0x56070a1c0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.154+0000 7fa63639f740 1 -- 172.21.15.40:0/725344726 >> 172.21.15.40:0/725344726 conn(0x56070a1aefc0 msgr2=0x56070a1aa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.154+0000 7fa63639f740 1 -- 172.21.15.40:0/725344726 shutdown_connections 2024-03-18T22:48:36.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.154+0000 7fa63639f740 1 -- 172.21.15.40:0/725344726 wait complete. 2024-03-18T22:48:36.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.154+0000 7fa63639f740 1 -- start start 2024-03-18T22:48:36.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.154+0000 7fa63639f740 1 Processor -- start 2024-03-18T22:48:36.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa63639f740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b5c60 0x56070a1c46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa63639f740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1b6650 0x56070a1c4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa63639f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1bb7c0 0x56070a1c5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa63639f740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1a9820 con 0x56070a1b5c60 2024-03-18T22:48:36.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa63639f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1a9150 con 0x56070a1bb7c0 2024-03-18T22:48:36.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa63639f740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1a6390 con 0x56070a1b6650 2024-03-18T22:48:36.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa63639f740 10 client.0 _ll_register_callbacks cb 0x7fa61c009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:36.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa631478640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b5c60 0x56070a1c46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa631478640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b5c60 0x56070a1c46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45624/0 (socket says 172.21.15.40:45624) 2024-03-18T22:48:36.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa631478640 1 -- 172.21.15.40:0/2108973307 learned_addr learned my addr 172.21.15.40:0/2108973307 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa630c77640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1b6650 0x56070a1c4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa631c79640 1 --2- 172.21.15.40:0/2108973307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1bb7c0 0x56070a1c5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa631478640 1 -- 172.21.15.40:0/2108973307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1b6650 msgr2=0x56070a1c4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa631478640 1 --2- 172.21.15.40:0/2108973307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1b6650 0x56070a1c4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa631478640 1 -- 172.21.15.40:0/2108973307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1bb7c0 msgr2=0x56070a1c5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa631478640 1 --2- 172.21.15.40:0/2108973307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1bb7c0 0x56070a1c5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa631478640 1 -- 172.21.15.40:0/2108973307 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56070a1c61b0 con 0x56070a1b5c60 2024-03-18T22:48:36.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa631478640 1 --2- 172.21.15.40:0/2108973307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b5c60 0x56070a1c46c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa6280bd460 tx=0x7fa6280383f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e851ce35042e7ac2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.155+0000 7fa614ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:48:36.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.156+0000 7fa614ff9640 1 -- 172.21.15.40:0/2108973307 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6280d6040 con 0x56070a1b5c60 2024-03-18T22:48:36.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.156+0000 7fa614ff9640 1 -- 172.21.15.40:0/2108973307 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa6280c59f0 con 0x56070a1b5c60 2024-03-18T22:48:36.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.156+0000 7fa63639f740 1 -- 172.21.15.40:0/2108973307 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa61c00fe20 con 0x56070a1b5c60 2024-03-18T22:48:36.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.156+0000 7fa614ff9640 1 -- 172.21.15.40:0/2108973307 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa6280bce30 con 0x56070a1b5c60 2024-03-18T22:48:36.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.156+0000 7fa63639f740 10 client.17022 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:36.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.156+0000 7fa63639f740 1 -- 172.21.15.40:0/2108973307 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56070a1c61b0 con 0x56070a1b5c60 2024-03-18T22:48:36.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.156+0000 7fa614ff9640 1 -- 172.21.15.40:0/2108973307 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa61c00fe20 con 0x56070a1b5c60 2024-03-18T22:48:36.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.156+0000 7fa614ff9640 1 -- 172.21.15.40:0/2108973307 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fa6280bd730 con 0x56070a1b5c60 2024-03-18T22:48:36.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.156+0000 7fa63639f740 10 client.17022 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:36.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.156+0000 7fa63639f740 10 client.17022 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:36.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.156+0000 7fa63639f740 20 client.17022 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:36.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.156+0000 7fa63639f740 1 --2- 172.21.15.40:0/2108973307 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56070a232240 0x56070a234660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.156+0000 7fa63639f740 4 client.17022 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:48:36.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.156+0000 7fa63639f740 1 -- 172.21.15.40:0/2108973307 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56070a11fd90 con 0x56070a232240 2024-03-18T22:48:36.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.156+0000 7fa630c77640 1 --2- 172.21.15.40:0/2108973307 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56070a232240 0x56070a234660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa630c77640 1 --2- 172.21.15.40:0/2108973307 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56070a232240 0x56070a234660 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5e03501cdf326ec server_cookie=a6ef6e3cc6a18b69 in_seq=0 out_seq=0 2024-03-18T22:48:36.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa614ff9640 10 client.17022 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:36.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa614ff9640 1 -- 172.21.15.40:0/2108973307 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56070a11fd90 con 0x56070a232240 2024-03-18T22:48:36.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa614ff9640 10 client.17022 handle_command_reply: tid=0 2024-03-18T22:48:36.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa63639f740 1 client.17022 shutdown 2024-03-18T22:48:36.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa63639f740 1 -- 172.21.15.40:0/2108973307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b5c60 msgr2=0x56070a1c46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa63639f740 1 --2- 172.21.15.40:0/2108973307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b5c60 0x56070a1c46c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa6280bd460 tx=0x7fa6280383f0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa63639f740 1 -- 172.21.15.40:0/2108973307 shutdown_connections 2024-03-18T22:48:36.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa63639f740 1 --2- 172.21.15.40:0/2108973307 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56070a232240 0x56070a234660 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa63639f740 1 --2- 172.21.15.40:0/2108973307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56070a1bb7c0 0x56070a1c5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa63639f740 1 --2- 172.21.15.40:0/2108973307 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56070a1b6650 0x56070a1c4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa614ff9640 0 client.17022 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:36.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa63639f740 1 --2- 172.21.15.40:0/2108973307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56070a1b5c60 0x56070a1c46c0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa63639f740 1 -- 172.21.15.40:0/2108973307 >> 172.21.15.40:0/2108973307 conn(0x56070a1aefc0 msgr2=0x56070a1b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa63639f740 1 -- 172.21.15.40:0/2108973307 shutdown_connections 2024-03-18T22:48:36.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa63639f740 1 -- 172.21.15.40:0/2108973307 wait complete. 2024-03-18T22:48:36.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.157+0000 7fa63639f740 20 client.17022 trim_cache size 0 max 16384 2024-03-18T22:48:36.165 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:48:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:48:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:48:36.165 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:48:36.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.158+0000 7fa6336dc640 1 -- 172.21.15.40:0/2812065928 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa6180a2800 msgr2=0x7fa6180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.158+0000 7fa6336dc640 1 --2- 172.21.15.40:0/2812065928 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa6180a2800 0x7fa6180a4cc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa624005fd0 tx=0x7fa624009e60 comp rx=0 tx=0).stop 2024-03-18T22:48:36.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.158+0000 7fa6336dc640 1 -- 172.21.15.40:0/2812065928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa62c100760 msgr2=0x7fa62c196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.158+0000 7fa6336dc640 1 --2- 172.21.15.40:0/2812065928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa62c100760 0x7fa62c196a30 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fa628000c00 tx=0x7fa628005d50 comp rx=0 tx=0).stop 2024-03-18T22:48:36.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.158+0000 7fa6336dc640 1 -- 172.21.15.40:0/2812065928 shutdown_connections 2024-03-18T22:48:36.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.158+0000 7fa6336dc640 1 --2- 172.21.15.40:0/2812065928 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa6180a2800 0x7fa6180a4cc0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.158+0000 7fa6336dc640 1 --2- 172.21.15.40:0/2812065928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa62c102310 0x7fa62c19dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.158+0000 7fa6336dc640 1 --2- 172.21.15.40:0/2812065928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa62c101950 0x7fa62c196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.158+0000 7fa6336dc640 1 --2- 172.21.15.40:0/2812065928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa62c100760 0x7fa62c196a30 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.158+0000 7fa6336dc640 1 -- 172.21.15.40:0/2812065928 >> 172.21.15.40:0/2812065928 conn(0x7fa62c0fbcd0 msgr2=0x7fa62c104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.158+0000 7fa6336dc640 1 -- 172.21.15.40:0/2812065928 shutdown_connections 2024-03-18T22:48:36.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.158+0000 7fa6336dc640 1 -- 172.21.15.40:0/2812065928 wait complete. 2024-03-18T22:48:36.170 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:48:36.171 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:48:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.283+0000 7f7039190640 1 -- 172.21.15.40:0/1373573586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7034103410 msgr2=0x7f7034109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.283+0000 7f7039190640 1 --2- 172.21.15.40:0/1373573586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7034103410 0x7f7034109ca0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f702c009ca0 tx=0x7f702c02c0f0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.283+0000 7f7039190640 1 -- 172.21.15.40:0/1373573586 shutdown_connections 2024-03-18T22:48:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.283+0000 7f7039190640 1 --2- 172.21.15.40:0/1373573586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7034103410 0x7f7034109ca0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.283+0000 7f7039190640 1 --2- 172.21.15.40:0/1373573586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7034102a50 0x7f7034102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.283+0000 7f7039190640 1 --2- 172.21.15.40:0/1373573586 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7034101860 0x7f7034101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.283+0000 7f7039190640 1 -- 172.21.15.40:0/1373573586 >> 172.21.15.40:0/1373573586 conn(0x7f70340fcdd0 msgr2=0x7f70340ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.283+0000 7f7039190640 1 -- 172.21.15.40:0/1373573586 shutdown_connections 2024-03-18T22:48:36.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.283+0000 7f7039190640 1 -- 172.21.15.40:0/1373573586 wait complete. 2024-03-18T22:48:36.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f7039190640 1 Processor -- start 2024-03-18T22:48:36.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f7039190640 1 -- start start 2024-03-18T22:48:36.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f7039190640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7034101860 0x7f7034197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f7039190640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7034102a50 0x7f7034197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f7039190640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7034103410 0x7f703419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f7039190640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f703410d370 con 0x7f7034103410 2024-03-18T22:48:36.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f7039190640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f703410d1f0 con 0x7f7034101860 2024-03-18T22:48:36.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f7039190640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f703410d4f0 con 0x7f7034102a50 2024-03-18T22:48:36.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f703359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7034103410 0x7f703419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f703359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7034103410 0x7f703419efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45636/0 (socket says 172.21.15.40:45636) 2024-03-18T22:48:36.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f703359e640 1 -- 172.21.15.40:0/3739415895 learned_addr learned my addr 172.21.15.40:0/3739415895 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f702bfff640 1 --2- 172.21.15.40:0/3739415895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7034102a50 0x7f7034197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f7032d9d640 1 --2- 172.21.15.40:0/3739415895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7034101860 0x7f7034197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f703359e640 1 -- 172.21.15.40:0/3739415895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7034102a50 msgr2=0x7f7034197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f703359e640 1 --2- 172.21.15.40:0/3739415895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7034102a50 0x7f7034197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f703359e640 1 -- 172.21.15.40:0/3739415895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7034101860 msgr2=0x7f7034197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f703359e640 1 --2- 172.21.15.40:0/3739415895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7034101860 0x7f7034197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f703359e640 1 -- 172.21.15.40:0/3739415895 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f703419f640 con 0x7f7034103410 2024-03-18T22:48:36.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.284+0000 7f702bfff640 1 --2- 172.21.15.40:0/3739415895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7034102a50 0x7f7034197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.285+0000 7f703359e640 1 --2- 172.21.15.40:0/3739415895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7034103410 0x7f703419efd0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f702c02c700 tx=0x7f702c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.285+0000 7f7030d99640 1 -- 172.21.15.40:0/3739415895 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c041020 con 0x7f7034103410 2024-03-18T22:48:36.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.285+0000 7f7030d99640 1 -- 172.21.15.40:0/3739415895 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f702c003e20 con 0x7f7034103410 2024-03-18T22:48:36.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.285+0000 7f7039190640 1 -- 172.21.15.40:0/3739415895 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f703419f8d0 con 0x7f7034103410 2024-03-18T22:48:36.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.285+0000 7f7030d99640 1 -- 172.21.15.40:0/3739415895 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702c039040 con 0x7f7034103410 2024-03-18T22:48:36.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.285+0000 7f7039190640 1 -- 172.21.15.40:0/3739415895 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f703419fdc0 con 0x7f7034103410 2024-03-18T22:48:36.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.286+0000 7f7039190640 1 -- 172.21.15.40:0/3739415895 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7034108b80 con 0x7f7034103410 2024-03-18T22:48:36.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.287+0000 7f7030d99640 1 -- 172.21.15.40:0/3739415895 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f702c044020 con 0x7f7034103410 2024-03-18T22:48:36.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.288+0000 7f7030d99640 1 --2- 172.21.15.40:0/3739415895 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70200a2800 0x7f70200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.288+0000 7f7032d9d640 1 --2- 172.21.15.40:0/3739415895 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70200a2800 0x7f70200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.288+0000 7f7030d99640 1 -- 172.21.15.40:0/3739415895 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f702c0f1030 con 0x7f7034103410 2024-03-18T22:48:36.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.289+0000 7f7032d9d640 1 --2- 172.21.15.40:0/3739415895 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70200a2800 0x7f70200a4cc0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f70341028b0 tx=0x7f7024005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.295+0000 7f7030d99640 1 -- 172.21.15.40:0/3739415895 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f702c04b020 con 0x7f7034103410 2024-03-18T22:48:36.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:36 smithi040 ceph-mon[62643]: pgmap v529: 129 pgs: 129 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:48:36.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.564+0000 7f7039190640 1 -- 172.21.15.40:0/3739415895 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7034062420 con 0x7f7034103410 2024-03-18T22:48:36.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.565+0000 7f7030d99640 1 -- 172.21.15.40:0/3739415895 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11947 (secure 0 0 0) 0x7f702c0bdce0 con 0x7f7034103410 2024-03-18T22:48:36.566 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:48:36.566 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:46:52.724919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24388,24401,24473,24481,24509]},"id":1}]} 2024-03-18T22:48:36.566 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 32 2024-03-18T22:48:36.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.568+0000 7f7039190640 1 -- 172.21.15.40:0/3739415895 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70200a2800 msgr2=0x7f70200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.568+0000 7f7039190640 1 --2- 172.21.15.40:0/3739415895 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70200a2800 0x7f70200a4cc0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f70341028b0 tx=0x7f7024005f90 comp rx=0 tx=0).stop 2024-03-18T22:48:36.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.568+0000 7f7039190640 1 -- 172.21.15.40:0/3739415895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7034103410 msgr2=0x7f703419efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.568+0000 7f7039190640 1 --2- 172.21.15.40:0/3739415895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7034103410 0x7f703419efd0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f702c02c700 tx=0x7f702c0328a0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.568+0000 7f7039190640 1 -- 172.21.15.40:0/3739415895 shutdown_connections 2024-03-18T22:48:36.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.568+0000 7f7039190640 1 --2- 172.21.15.40:0/3739415895 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f70200a2800 0x7f70200a4cc0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.568+0000 7f7039190640 1 --2- 172.21.15.40:0/3739415895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7034103410 0x7f703419efd0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.568+0000 7f7039190640 1 --2- 172.21.15.40:0/3739415895 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7034102a50 0x7f7034197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.568+0000 7f7039190640 1 --2- 172.21.15.40:0/3739415895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7034101860 0x7f7034197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.568+0000 7f7039190640 1 -- 172.21.15.40:0/3739415895 >> 172.21.15.40:0/3739415895 conn(0x7f70340fcdd0 msgr2=0x7f7034105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.568+0000 7f7039190640 1 -- 172.21.15.40:0/3739415895 shutdown_connections 2024-03-18T22:48:36.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.568+0000 7f7039190640 1 -- 172.21.15.40:0/3739415895 wait complete. 2024-03-18T22:48:36.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:36 smithi113 ceph-mon[66775]: pgmap v529: 129 pgs: 129 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:48:36.581 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:48:36.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:36 smithi103 ceph-mon[66377]: pgmap v529: 129 pgs: 129 active+clean; 7.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:48:36.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.697+0000 7fbed126c640 1 -- 172.21.15.40:0/2835799234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbecc1034c0 msgr2=0x7fbecc109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.697+0000 7fbed126c640 1 --2- 172.21.15.40:0/2835799234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbecc1034c0 0x7fbecc109d50 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009cd0 tx=0x7fbec402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.697+0000 7fbed126c640 1 -- 172.21.15.40:0/2835799234 shutdown_connections 2024-03-18T22:48:36.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.697+0000 7fbed126c640 1 --2- 172.21.15.40:0/2835799234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbecc1034c0 0x7fbecc109d50 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.697+0000 7fbed126c640 1 --2- 172.21.15.40:0/2835799234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbecc102b00 0x7fbecc102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.697+0000 7fbed126c640 1 --2- 172.21.15.40:0/2835799234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbecc0ffb30 0x7fbecc0fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.697+0000 7fbed126c640 1 -- 172.21.15.40:0/2835799234 >> 172.21.15.40:0/2835799234 conn(0x7fbecc0fb460 msgr2=0x7fbecc0fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.698+0000 7fbed126c640 1 -- 172.21.15.40:0/2835799234 shutdown_connections 2024-03-18T22:48:36.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.698+0000 7fbed126c640 1 -- 172.21.15.40:0/2835799234 wait complete. 2024-03-18T22:48:36.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.698+0000 7fbed126c640 1 Processor -- start 2024-03-18T22:48:36.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.698+0000 7fbed126c640 1 -- start start 2024-03-18T22:48:36.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.698+0000 7fbed126c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbecc0ffb30 0x7fbecc102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.698+0000 7fbed126c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbecc102b00 0x7fbecc100ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.698+0000 7fbed126c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbecc1034c0 0x7fbecc100ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.698+0000 7fbed126c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc10d3b0 con 0x7fbecc1034c0 2024-03-18T22:48:36.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.698+0000 7fbed126c640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc10d230 con 0x7fbecc102b00 2024-03-18T22:48:36.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.698+0000 7fbed126c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc10d530 con 0x7fbecc0ffb30 2024-03-18T22:48:36.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbecb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbecc1034c0 0x7fbecc100ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbecb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbecc1034c0 0x7fbecc100ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45646/0 (socket says 172.21.15.40:45646) 2024-03-18T22:48:36.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbeca7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbecc102b00 0x7fbecc100ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbecb7fe640 1 -- 172.21.15.40:0/3334867292 learned_addr learned my addr 172.21.15.40:0/3334867292 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbecaffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbecc0ffb30 0x7fbecc102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbecb7fe640 1 -- 172.21.15.40:0/3334867292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbecc0ffb30 msgr2=0x7fbecc102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbecb7fe640 1 --2- 172.21.15.40:0/3334867292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbecc0ffb30 0x7fbecc102470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbecb7fe640 1 -- 172.21.15.40:0/3334867292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbecc102b00 msgr2=0x7fbecc100ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbecb7fe640 1 --2- 172.21.15.40:0/3334867292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbecc102b00 0x7fbecc100ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbecb7fe640 1 -- 172.21.15.40:0/3334867292 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbecc101880 con 0x7fbecc1034c0 2024-03-18T22:48:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbeca7fc640 1 --2- 172.21.15.40:0/3334867292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbecc102b00 0x7fbecc100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbecb7fe640 1 --2- 172.21.15.40:0/3334867292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbecc1034c0 0x7fbecc100ff0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fbec4032380 tx=0x7fbec4032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbec3fff640 1 -- 172.21.15.40:0/3334867292 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec4041020 con 0x7fbecc1034c0 2024-03-18T22:48:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbec3fff640 1 -- 172.21.15.40:0/3334867292 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbec4003e20 con 0x7fbecc1034c0 2024-03-18T22:48:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbed126c640 1 -- 172.21.15.40:0/3334867292 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbecc1a1ae0 con 0x7fbecc1034c0 2024-03-18T22:48:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbec3fff640 1 -- 172.21.15.40:0/3334867292 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec4039040 con 0x7fbecc1034c0 2024-03-18T22:48:36.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.699+0000 7fbed126c640 1 -- 172.21.15.40:0/3334867292 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbecc1a2080 con 0x7fbecc1034c0 2024-03-18T22:48:36.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.702+0000 7fbec3fff640 1 -- 172.21.15.40:0/3334867292 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbec4044020 con 0x7fbecc1034c0 2024-03-18T22:48:36.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.702+0000 7fbed3e58740 1 Processor -- start 2024-03-18T22:48:36.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.702+0000 7fbed3e58740 1 -- start start 2024-03-18T22:48:36.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.702+0000 7fbec3fff640 1 --2- 172.21.15.40:0/3334867292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbeb80a2880 0x7fbeb80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbec3fff640 1 -- 172.21.15.40:0/3334867292 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fbec40f0da0 con 0x7fbecc1034c0 2024-03-18T22:48:36.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbeca7fc640 1 --2- 172.21.15.40:0/3334867292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbeb80a2880 0x7fbeb80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbed3e58740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f9530572f0 0x55f953057710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbed3e58740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f95305c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbed3e58740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 0x55f95305f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbed3e58740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f952fbb4b0 con 0x55f953057ce0 2024-03-18T22:48:36.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbed3e58740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f952f63d70 con 0x55f9530572f0 2024-03-18T22:48:36.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbed3e58740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f952fb7550 con 0x55f95305ce50 2024-03-18T22:48:36.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbeca7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f95305c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbeca7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f95305c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45650/0 (socket says 172.21.15.40:45650) 2024-03-18T22:48:36.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbeca7fc640 1 -- 172.21.15.40:0/486196164 learned_addr learned my addr 172.21.15.40:0/486196164 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbecb7fe640 1 --2- 172.21.15.40:0/486196164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 0x55f95305f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbecaffd640 1 --2- 172.21.15.40:0/486196164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f9530572f0 0x55f953057710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbeca7fc640 1 --2- 172.21.15.40:0/3334867292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbeb80a2880 0x7fbeb80a4d40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbeb40138f0 tx=0x7fbeb4013cf0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbeca7fc640 1 -- 172.21.15.40:0/486196164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 msgr2=0x55f95305f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbeca7fc640 1 --2- 172.21.15.40:0/486196164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 0x55f95305f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbeca7fc640 1 -- 172.21.15.40:0/486196164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f9530572f0 msgr2=0x55f953057710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbeca7fc640 1 --2- 172.21.15.40:0/486196164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f9530572f0 0x55f953057710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbeca7fc640 1 -- 172.21.15.40:0/486196164 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f952f86210 con 0x55f953057ce0 2024-03-18T22:48:36.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.703+0000 7fbecaffd640 1 --2- 172.21.15.40:0/486196164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f9530572f0 0x55f953057710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.704+0000 7fbeca7fc640 1 --2- 172.21.15.40:0/486196164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f95305c7e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400fc80 tx=0x7fbeb4020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a7099f0a353e3f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.704+0000 7fbec17fa640 1 -- 172.21.15.40:0/486196164 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb400fcb0 con 0x55f953057ce0 2024-03-18T22:48:36.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.704+0000 7fbec17fa640 1 -- 172.21.15.40:0/486196164 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbeb4024070 con 0x55f953057ce0 2024-03-18T22:48:36.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.704+0000 7fbec17fa640 1 -- 172.21.15.40:0/486196164 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40054e0 con 0x55f953057ce0 2024-03-18T22:48:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.705+0000 7fbed3e58740 1 -- 172.21.15.40:0/486196164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 msgr2=0x55f95305c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.705+0000 7fbed3e58740 1 --2- 172.21.15.40:0/486196164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f95305c7e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbeb400fc80 tx=0x7fbeb4020000 comp rx=0 tx=0).stop 2024-03-18T22:48:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.705+0000 7fbed3e58740 1 -- 172.21.15.40:0/486196164 shutdown_connections 2024-03-18T22:48:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.705+0000 7fbed3e58740 1 --2- 172.21.15.40:0/486196164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 0x55f95305f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.705+0000 7fbed3e58740 1 --2- 172.21.15.40:0/486196164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f95305c7e0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.705+0000 7fbed3e58740 1 --2- 172.21.15.40:0/486196164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f9530572f0 0x55f953057710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.705+0000 7fbed3e58740 1 -- 172.21.15.40:0/486196164 >> 172.21.15.40:0/486196164 conn(0x55f953052860 msgr2=0x55f953052c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.705+0000 7fbed3e58740 1 -- 172.21.15.40:0/486196164 shutdown_connections 2024-03-18T22:48:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.705+0000 7fbed3e58740 1 -- 172.21.15.40:0/486196164 wait complete. 2024-03-18T22:48:36.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.705+0000 7fbed3e58740 1 -- start start 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.705+0000 7fbed3e58740 1 Processor -- start 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.705+0000 7fbed3e58740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f9530572f0 0x55f953061750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.705+0000 7fbed3e58740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f953061c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.705+0000 7fbed3e58740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f95305ce50 0x55f953068cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.705+0000 7fbed3e58740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f95305fe00 con 0x55f953057ce0 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbed3e58740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f952f77310 con 0x55f95305ce50 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbed3e58740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f95305ff80 con 0x55f9530572f0 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbed3e58740 10 client.0 _ll_register_callbacks cb 0x55f952f6b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbeca7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f953061c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbeca7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f953061c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45656/0 (socket says 172.21.15.40:45656) 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbecb7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f95305ce50 0x55f953068cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbeca7fc640 1 -- 172.21.15.40:0/2877131654 learned_addr learned my addr 172.21.15.40:0/2877131654 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbecaffd640 1 --2- 172.21.15.40:0/2877131654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f9530572f0 0x55f953061750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbecb7fe640 1 -- 172.21.15.40:0/2877131654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f9530572f0 msgr2=0x55f953061750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbecb7fe640 1 --2- 172.21.15.40:0/2877131654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f9530572f0 0x55f953061750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbecb7fe640 1 -- 172.21.15.40:0/2877131654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 msgr2=0x55f953061c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbecb7fe640 1 --2- 172.21.15.40:0/2877131654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f953061c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbecb7fe640 1 -- 172.21.15.40:0/2877131654 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f952ccf3f0 con 0x55f95305ce50 2024-03-18T22:48:36.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbeca7fc640 1 --2- 172.21.15.40:0/2877131654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f953061c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbecb7fe640 1 --2- 172.21.15.40:0/2877131654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f95305ce50 0x55f953068cd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbec40ee820 tx=0x7fbec40095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=215a4df9da01dc21 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbeb27fc640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:48:36.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbeb27fc640 1 -- 172.21.15.40:0/2877131654 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec40d6060 con 0x55f95305ce50 2024-03-18T22:48:36.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbeb27fc640 1 -- 172.21.15.40:0/2877131654 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fbec40b99c0 con 0x55f95305ce50 2024-03-18T22:48:36.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.706+0000 7fbed3e58740 1 -- 172.21.15.40:0/2877131654 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbeb4024070 con 0x55f95305ce50 2024-03-18T22:48:36.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.707+0000 7fbeb27fc640 1 -- 172.21.15.40:0/2877131654 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbec40d6200 con 0x55f95305ce50 2024-03-18T22:48:36.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.707+0000 7fbed3e58740 10 client.26861 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:36.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.707+0000 7fbed3e58740 1 -- 172.21.15.40:0/2877131654 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f952d997d0 con 0x55f95305ce50 2024-03-18T22:48:36.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.707+0000 7fbeb27fc640 1 -- 172.21.15.40:0/2877131654 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbeb4024070 con 0x55f95305ce50 2024-03-18T22:48:36.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.707+0000 7fbeb27fc640 1 -- 172.21.15.40:0/2877131654 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fbec403c030 con 0x55f95305ce50 2024-03-18T22:48:36.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.707+0000 7fbed3e58740 10 client.26861 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.707+0000 7fbed3e58740 10 client.26861 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:48:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.707+0000 7fbed3e58740 20 client.26861 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.707+0000 7fbed3e58740 1 --2- 172.21.15.40:0/2877131654 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f95306d5a0 0x55f95306f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.707+0000 7fbed3e58740 4 client.26861 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.707+0000 7fbed3e58740 1 -- 172.21.15.40:0/2877131654 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9530193e0 con 0x55f95306d5a0 2024-03-18T22:48:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.707+0000 7fbecaffd640 1 --2- 172.21.15.40:0/2877131654 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f95306d5a0 0x55f95306f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.708+0000 7fbecaffd640 1 --2- 172.21.15.40:0/2877131654 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f95306d5a0 0x55f95306f9e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d1e5a4092f556d64 server_cookie=68a85f0c695092b6 in_seq=0 out_seq=0 2024-03-18T22:48:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.708+0000 7fbeb27fc640 10 client.26861 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:48:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.708+0000 7fbeb27fc640 1 -- 172.21.15.40:0/2877131654 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f9530193e0 con 0x55f95306d5a0 2024-03-18T22:48:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.708+0000 7fbeb27fc640 10 client.26861 handle_command_reply: tid=0 2024-03-18T22:48:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.708+0000 7fbed3e58740 1 client.26861 shutdown 2024-03-18T22:48:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.708+0000 7fbed3e58740 1 -- 172.21.15.40:0/2877131654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f95305ce50 msgr2=0x55f953068cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.708+0000 7fbed3e58740 1 --2- 172.21.15.40:0/2877131654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f95305ce50 0x55f953068cd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbec40ee820 tx=0x7fbec40095c0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.709+0000 7fbed3e58740 1 -- 172.21.15.40:0/2877131654 shutdown_connections 2024-03-18T22:48:36.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.709+0000 7fbed3e58740 1 --2- 172.21.15.40:0/2877131654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f95305ce50 0x55f953068cd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.709+0000 7fbed3e58740 1 --2- 172.21.15.40:0/2877131654 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f95306d5a0 0x55f95306f9e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.709+0000 7fbed3e58740 1 --2- 172.21.15.40:0/2877131654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f953061c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.709+0000 7fbed3e58740 1 --2- 172.21.15.40:0/2877131654 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f9530572f0 0x55f953061750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.709+0000 7fbeb27fc640 0 client.26861 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:48:36.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.709+0000 7fbed3e58740 1 -- 172.21.15.40:0/2877131654 >> 172.21.15.40:0/2877131654 conn(0x55f953052860 msgr2=0x55f953051ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.709+0000 7fbed3e58740 1 -- 172.21.15.40:0/2877131654 shutdown_connections 2024-03-18T22:48:36.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.709+0000 7fbed3e58740 1 -- 172.21.15.40:0/2877131654 wait complete. 2024-03-18T22:48:36.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.709+0000 7fbed3e58740 20 client.26861 trim_cache size 0 max 16384 2024-03-18T22:48:36.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.724+0000 7fbed3e58740 1 Processor -- start 2024-03-18T22:48:36.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.724+0000 7fbed3e58740 1 -- start start 2024-03-18T22:48:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.724+0000 7fbed3e58740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f9530572f0 0x55f953061750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.724+0000 7fbed3e58740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f953068c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.724+0000 7fbed3e58740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 0x55f953061c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.724+0000 7fbed3e58740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1ac6a0 con 0x55f953057ce0 2024-03-18T22:48:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.724+0000 7fbed3e58740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1ae1e0 con 0x55f9530572f0 2024-03-18T22:48:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.724+0000 7fbed3e58740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1acd70 con 0x55f95305ce50 2024-03-18T22:48:36.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbeca7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f953068c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbecaffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f9530572f0 0x55f953061750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbecb7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 0x55f953061c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbecb7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 0x55f953061c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38744/0 (socket says 172.21.15.40:38744) 2024-03-18T22:48:36.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbeca7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f953068c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45658/0 (socket says 172.21.15.40:45658) 2024-03-18T22:48:36.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbecaffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f9530572f0 0x55f953061750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49628/0 (socket says 172.21.15.40:49628) 2024-03-18T22:48:36.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbeca7fc640 1 -- 172.21.15.40:0/2376277242 learned_addr learned my addr 172.21.15.40:0/2376277242 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbecaffd640 1 -- 172.21.15.40:0/2376277242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 msgr2=0x55f953061c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbecaffd640 1 --2- 172.21.15.40:0/2376277242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 0x55f953061c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbecaffd640 1 -- 172.21.15.40:0/2376277242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 msgr2=0x55f953068c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbecaffd640 1 --2- 172.21.15.40:0/2376277242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f953068c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbecaffd640 1 -- 172.21.15.40:0/2376277242 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f953070060 con 0x55f9530572f0 2024-03-18T22:48:36.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbeca7fc640 1 --2- 172.21.15.40:0/2376277242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f953068c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbecb7fe640 1 --2- 172.21.15.40:0/2376277242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 0x55f953061c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbecaffd640 1 --2- 172.21.15.40:0/2376277242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f9530572f0 0x55f953061750 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55f953070230 tx=0x7fbebc005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57cb040143bd4d96 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbec0ff9640 1 -- 172.21.15.40:0/2376277242 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc025070 con 0x55f9530572f0 2024-03-18T22:48:36.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbec0ff9640 1 -- 172.21.15.40:0/2376277242 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbebc004310 con 0x55f9530572f0 2024-03-18T22:48:36.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.725+0000 7fbec0ff9640 1 -- 172.21.15.40:0/2376277242 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc003b40 con 0x55f9530572f0 2024-03-18T22:48:36.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.726+0000 7fbed3e58740 1 -- 172.21.15.40:0/2376277242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f9530572f0 msgr2=0x55f953061750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.726+0000 7fbed3e58740 1 --2- 172.21.15.40:0/2376277242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f9530572f0 0x55f953061750 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55f953070230 tx=0x7fbebc005f90 comp rx=0 tx=0).stop 2024-03-18T22:48:36.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.726+0000 7fbed3e58740 1 -- 172.21.15.40:0/2376277242 shutdown_connections 2024-03-18T22:48:36.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.726+0000 7fbed3e58740 1 --2- 172.21.15.40:0/2376277242 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 0x55f953061c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.726+0000 7fbed3e58740 1 --2- 172.21.15.40:0/2376277242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f953057ce0 0x55f953068c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.726+0000 7fbed3e58740 1 --2- 172.21.15.40:0/2376277242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f9530572f0 0x55f953061750 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.726+0000 7fbed3e58740 1 -- 172.21.15.40:0/2376277242 >> 172.21.15.40:0/2376277242 conn(0x55f953052860 msgr2=0x55f953060350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.726+0000 7fbed3e58740 1 -- 172.21.15.40:0/2376277242 shutdown_connections 2024-03-18T22:48:36.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.726+0000 7fbed3e58740 1 -- 172.21.15.40:0/2376277242 wait complete. 2024-03-18T22:48:36.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbed3e58740 1 -- start start 2024-03-18T22:48:36.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbed3e58740 1 Processor -- start 2024-03-18T22:48:36.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbed3e58740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f9530572f0 0x55f953065dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbed3e58740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f953057ce0 0x55f953066300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbed3e58740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 0x55f953067310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbed3e58740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1adb10 con 0x55f9530572f0 2024-03-18T22:48:36.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbed3e58740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1ad440 con 0x55f953057ce0 2024-03-18T22:48:36.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbed3e58740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1aa680 con 0x55f95305ce50 2024-03-18T22:48:36.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbed3e58740 10 client.0 _ll_register_callbacks cb 0x7fbec40bd210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:36.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbecaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f9530572f0 0x55f953065dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbecaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f9530572f0 0x55f953065dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45666/0 (socket says 172.21.15.40:45666) 2024-03-18T22:48:36.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbecaffd640 1 -- 172.21.15.40:0/3242875809 learned_addr learned my addr 172.21.15.40:0/3242875809 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbecb7fe640 1 --2- 172.21.15.40:0/3242875809 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 0x55f953067310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbeca7fc640 1 --2- 172.21.15.40:0/3242875809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f953057ce0 0x55f953066300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbecaffd640 1 -- 172.21.15.40:0/3242875809 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 msgr2=0x55f953067310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbecaffd640 1 --2- 172.21.15.40:0/3242875809 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 0x55f953067310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbecaffd640 1 -- 172.21.15.40:0/3242875809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f953057ce0 msgr2=0x55f953066300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbecaffd640 1 --2- 172.21.15.40:0/3242875809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f953057ce0 0x55f953066300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.727+0000 7fbecaffd640 1 -- 172.21.15.40:0/3242875809 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f953028880 con 0x55f9530572f0 2024-03-18T22:48:36.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbecaffd640 1 --2- 172.21.15.40:0/3242875809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f9530572f0 0x55f953065dc0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009d10 tx=0x7fbebc010500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45543a2da41f675d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbeb27fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:48:36.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbeb27fc640 1 -- 172.21.15.40:0/3242875809 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc025070 con 0x55f9530572f0 2024-03-18T22:48:36.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbeb27fc640 1 -- 172.21.15.40:0/3242875809 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fbebc003ad0 con 0x55f9530572f0 2024-03-18T22:48:36.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbed3e58740 1 -- 172.21.15.40:0/3242875809 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbebc004310 con 0x55f9530572f0 2024-03-18T22:48:36.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbeb27fc640 1 -- 172.21.15.40:0/3242875809 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbebc038050 con 0x55f9530572f0 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbed3e58740 10 client.17046 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbed3e58740 1 -- 172.21.15.40:0/3242875809 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f953068a30 con 0x55f9530572f0 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbeb27fc640 1 -- 172.21.15.40:0/3242875809 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbebc004310 con 0x55f9530572f0 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbeb27fc640 1 -- 172.21.15.40:0/3242875809 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fbebc02a670 con 0x55f9530572f0 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbed3e58740 10 client.17046 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbed3e58740 10 client.17046 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbed3e58740 20 client.17046 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbed3e58740 1 --2- 172.21.15.40:0/3242875809 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f9530d3c10 0x55f9530d6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbed3e58740 4 client.17046 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.728+0000 7fbed3e58740 1 -- 172.21.15.40:0/3242875809 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f952fc89f0 con 0x55f9530d3c10 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbeca7fc640 1 --2- 172.21.15.40:0/3242875809 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f9530d3c10 0x55f9530d6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbeb27fc640 10 client.17046 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbeca7fc640 1 --2- 172.21.15.40:0/3242875809 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f9530d3c10 0x55f9530d6000 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3c6c94d1878afa56 server_cookie=b9641d08d1cecaf5 in_seq=0 out_seq=0 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbeb27fc640 1 -- 172.21.15.40:0/3242875809 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f952fc89f0 con 0x55f9530d3c10 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbeb27fc640 10 client.17046 handle_command_reply: tid=0 2024-03-18T22:48:36.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbed3e58740 1 client.17046 shutdown 2024-03-18T22:48:36.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbed3e58740 1 -- 172.21.15.40:0/3242875809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f9530572f0 msgr2=0x55f953065dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbed3e58740 1 --2- 172.21.15.40:0/3242875809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f9530572f0 0x55f953065dc0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009d10 tx=0x7fbebc010500 comp rx=0 tx=0).stop 2024-03-18T22:48:36.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbed3e58740 1 -- 172.21.15.40:0/3242875809 shutdown_connections 2024-03-18T22:48:36.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbed3e58740 1 --2- 172.21.15.40:0/3242875809 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f95305ce50 0x55f953067310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbed3e58740 1 --2- 172.21.15.40:0/3242875809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f953057ce0 0x55f953066300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbed3e58740 1 --2- 172.21.15.40:0/3242875809 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f9530d3c10 0x55f9530d6000 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbed3e58740 1 --2- 172.21.15.40:0/3242875809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f9530572f0 0x55f953065dc0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbeb27fc640 0 client.17046 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:48:36.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbed3e58740 1 -- 172.21.15.40:0/3242875809 >> 172.21.15.40:0/3242875809 conn(0x55f953052860 msgr2=0x55f953061750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbed3e58740 1 -- 172.21.15.40:0/3242875809 shutdown_connections 2024-03-18T22:48:36.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.729+0000 7fbed3e58740 1 -- 172.21.15.40:0/3242875809 wait complete. 2024-03-18T22:48:36.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.730+0000 7fbed3e58740 20 client.17046 trim_cache size 0 max 16384 2024-03-18T22:48:36.737 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:48:36.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.730+0000 7fbed126c640 1 -- 172.21.15.40:0/3334867292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbeb80a2880 msgr2=0x7fbeb80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.730+0000 7fbed126c640 1 --2- 172.21.15.40:0/3334867292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbeb80a2880 0x7fbeb80a4d40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbeb40138f0 tx=0x7fbeb4013cf0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.730+0000 7fbed126c640 1 -- 172.21.15.40:0/3334867292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbecc1034c0 msgr2=0x7fbecc100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.730+0000 7fbed126c640 1 --2- 172.21.15.40:0/3334867292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbecc1034c0 0x7fbecc100ff0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fbec4032380 tx=0x7fbec4032820 comp rx=0 tx=0).stop 2024-03-18T22:48:36.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.730+0000 7fbed126c640 1 -- 172.21.15.40:0/3334867292 shutdown_connections 2024-03-18T22:48:36.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.730+0000 7fbed126c640 1 --2- 172.21.15.40:0/3334867292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbeb80a2880 0x7fbeb80a4d40 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.730+0000 7fbed126c640 1 --2- 172.21.15.40:0/3334867292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbecc1034c0 0x7fbecc100ff0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.730+0000 7fbed126c640 1 --2- 172.21.15.40:0/3334867292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbecc102b00 0x7fbecc100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.730+0000 7fbed126c640 1 --2- 172.21.15.40:0/3334867292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbecc0ffb30 0x7fbecc102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.730+0000 7fbed126c640 1 -- 172.21.15.40:0/3334867292 >> 172.21.15.40:0/3334867292 conn(0x7fbecc0fb460 msgr2=0x7fbecc105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.730+0000 7fbed126c640 1 -- 172.21.15.40:0/3334867292 shutdown_connections 2024-03-18T22:48:36.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.730+0000 7fbed126c640 1 -- 172.21.15.40:0/3334867292 wait complete. 2024-03-18T22:48:36.742 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:48:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.864+0000 7f6a4b377640 1 -- 172.21.15.40:0/3940606342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a44100720 msgr2=0x7f6a44100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.864+0000 7f6a4b377640 1 --2- 172.21.15.40:0/3940606342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a44100720 0x7f6a44100b40 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f6a40009900 tx=0x7f6a4002bc30 comp rx=0 tx=0).stop 2024-03-18T22:48:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.864+0000 7f6a4b377640 1 -- 172.21.15.40:0/3940606342 shutdown_connections 2024-03-18T22:48:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.864+0000 7f6a4b377640 1 --2- 172.21.15.40:0/3940606342 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a441022d0 0x7f6a44108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.864+0000 7f6a4b377640 1 --2- 172.21.15.40:0/3940606342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a44101910 0x7f6a44101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.864+0000 7f6a4b377640 1 --2- 172.21.15.40:0/3940606342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a44100720 0x7f6a44100b40 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.864+0000 7f6a4b377640 1 -- 172.21.15.40:0/3940606342 >> 172.21.15.40:0/3940606342 conn(0x7f6a440fbc90 msgr2=0x7f6a440fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.864+0000 7f6a4b377640 1 -- 172.21.15.40:0/3940606342 shutdown_connections 2024-03-18T22:48:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.864+0000 7f6a4b377640 1 -- 172.21.15.40:0/3940606342 wait complete. 2024-03-18T22:48:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a4b377640 1 Processor -- start 2024-03-18T22:48:36.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a4b377640 1 -- start start 2024-03-18T22:48:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a4b377640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a44100720 0x7f6a44197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a4b377640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a44101910 0x7f6a44197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a4b377640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a441022d0 0x7f6a4419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a4b377640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4410c250 con 0x7f6a44101910 2024-03-18T22:48:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a4b377640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4410c0d0 con 0x7f6a44100720 2024-03-18T22:48:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a4b377640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4410c3d0 con 0x7f6a441022d0 2024-03-18T22:48:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a48912640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a44101910 0x7f6a44197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a49113640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a44100720 0x7f6a44197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a48912640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a44101910 0x7f6a44197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45672/0 (socket says 172.21.15.40:45672) 2024-03-18T22:48:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a48912640 1 -- 172.21.15.40:0/2539367027 learned_addr learned my addr 172.21.15.40:0/2539367027 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a49914640 1 --2- 172.21.15.40:0/2539367027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a441022d0 0x7f6a4419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a48912640 1 -- 172.21.15.40:0/2539367027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a441022d0 msgr2=0x7f6a4419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a48912640 1 --2- 172.21.15.40:0/2539367027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a441022d0 0x7f6a4419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a48912640 1 -- 172.21.15.40:0/2539367027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a44100720 msgr2=0x7f6a44197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a48912640 1 --2- 172.21.15.40:0/2539367027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a44100720 0x7f6a44197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a48912640 1 -- 172.21.15.40:0/2539367027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4419f680 con 0x7f6a44101910 2024-03-18T22:48:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.865+0000 7f6a49113640 1 --2- 172.21.15.40:0/2539367027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a44100720 0x7f6a44197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.866+0000 7f6a48912640 1 --2- 172.21.15.40:0/2539367027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a44101910 0x7f6a44197fa0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400e9f0 tx=0x7f6a3400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.866+0000 7f6a3a7fc640 1 -- 172.21.15.40:0/2539367027 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3400cd80 con 0x7f6a44101910 2024-03-18T22:48:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.866+0000 7f6a3a7fc640 1 -- 172.21.15.40:0/2539367027 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a340108e0 con 0x7f6a44101910 2024-03-18T22:48:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.866+0000 7f6a4b377640 1 -- 172.21.15.40:0/2539367027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a4419f970 con 0x7f6a44101910 2024-03-18T22:48:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.866+0000 7f6a3a7fc640 1 -- 172.21.15.40:0/2539367027 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34018920 con 0x7f6a44101910 2024-03-18T22:48:36.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.866+0000 7f6a4b377640 1 -- 172.21.15.40:0/2539367027 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a4419fec0 con 0x7f6a44101910 2024-03-18T22:48:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.868+0000 7f6a3a7fc640 1 -- 172.21.15.40:0/2539367027 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6a34010a90 con 0x7f6a44101910 2024-03-18T22:48:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.869+0000 7f6a3a7fc640 1 --2- 172.21.15.40:0/2539367027 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a300a2800 0x7f6a300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.869+0000 7f6a4df66740 1 Processor -- start 2024-03-18T22:48:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.869+0000 7f6a4df66740 1 -- start start 2024-03-18T22:48:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.869+0000 7f6a3a7fc640 1 -- 172.21.15.40:0/2539367027 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6a340cf140 con 0x7f6a44101910 2024-03-18T22:48:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.869+0000 7f6a49113640 1 --2- 172.21.15.40:0/2539367027 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a300a2800 0x7f6a300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.869+0000 7f6a4df66740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2d2f0 0x56483ee2d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.869+0000 7f6a4df66740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2dce0 0x56483ee327e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.869+0000 7f6a4df66740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee32e50 0x56483ee352e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.869+0000 7f6a4df66740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56483ed914b0 con 0x56483ee2d2f0 2024-03-18T22:48:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.869+0000 7f6a4df66740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56483ed39d70 con 0x56483ee2dce0 2024-03-18T22:48:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.869+0000 7f6a4df66740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56483ed8d550 con 0x56483ee32e50 2024-03-18T22:48:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a49113640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2d2f0 0x56483ee2d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a48912640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2dce0 0x56483ee327e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a48912640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2dce0 0x56483ee327e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49654/0 (socket says 172.21.15.40:49654) 2024-03-18T22:48:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a48912640 1 -- 172.21.15.40:0/3880215478 learned_addr learned my addr 172.21.15.40:0/3880215478 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a49914640 1 --2- 172.21.15.40:0/3880215478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee32e50 0x56483ee352e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a48912640 1 -- 172.21.15.40:0/3880215478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee32e50 msgr2=0x56483ee352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a48912640 1 --2- 172.21.15.40:0/3880215478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee32e50 0x56483ee352e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a48912640 1 -- 172.21.15.40:0/3880215478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2d2f0 msgr2=0x56483ee2d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a48912640 1 --2- 172.21.15.40:0/3880215478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2d2f0 0x56483ee2d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a48912640 1 -- 172.21.15.40:0/3880215478 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56483ed5c210 con 0x56483ee2dce0 2024-03-18T22:48:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a49914640 1 --2- 172.21.15.40:0/3880215478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee32e50 0x56483ee352e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a49113640 1 --2- 172.21.15.40:0/2539367027 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a300a2800 0x7f6a300a4cc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6a40005ab0 tx=0x7f6a400041a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a48912640 1 --2- 172.21.15.40:0/3880215478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2dce0 0x56483ee327e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6a3409b730 tx=0x7f6a34010ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b08743863eb41af server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a2f7fe640 1 -- 172.21.15.40:0/3880215478 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a340a3e90 con 0x56483ee2dce0 2024-03-18T22:48:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a2f7fe640 1 -- 172.21.15.40:0/3880215478 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a34099aa0 con 0x56483ee2dce0 2024-03-18T22:48:36.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.870+0000 7f6a2f7fe640 1 -- 172.21.15.40:0/3880215478 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a340974f0 con 0x56483ee2dce0 2024-03-18T22:48:36.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.871+0000 7f6a4df66740 1 -- 172.21.15.40:0/3880215478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2dce0 msgr2=0x56483ee327e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.871+0000 7f6a4df66740 1 --2- 172.21.15.40:0/3880215478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2dce0 0x56483ee327e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6a3409b730 tx=0x7f6a34010ea0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.871+0000 7f6a4df66740 1 -- 172.21.15.40:0/3880215478 shutdown_connections 2024-03-18T22:48:36.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.871+0000 7f6a4df66740 1 --2- 172.21.15.40:0/3880215478 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee32e50 0x56483ee352e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.871+0000 7f6a4df66740 1 --2- 172.21.15.40:0/3880215478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2dce0 0x56483ee327e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.871+0000 7f6a4df66740 1 --2- 172.21.15.40:0/3880215478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2d2f0 0x56483ee2d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.871+0000 7f6a4df66740 1 -- 172.21.15.40:0/3880215478 >> 172.21.15.40:0/3880215478 conn(0x56483ee28860 msgr2=0x56483ee28c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.871+0000 7f6a4df66740 1 -- 172.21.15.40:0/3880215478 shutdown_connections 2024-03-18T22:48:36.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.871+0000 7f6a4df66740 1 -- 172.21.15.40:0/3880215478 wait complete. 2024-03-18T22:48:36.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.872+0000 7f6a4df66740 1 -- start start 2024-03-18T22:48:36.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.872+0000 7f6a4df66740 1 Processor -- start 2024-03-18T22:48:36.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.872+0000 7f6a4df66740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee2d2f0 0x56483ee378c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.872+0000 7f6a4df66740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2dce0 0x56483ee37e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.872+0000 7f6a4df66740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee32e50 0x56483ee3ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.872+0000 7f6a4df66740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56483ee35e00 con 0x56483ee32e50 2024-03-18T22:48:36.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.872+0000 7f6a4df66740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56483ed4d310 con 0x56483ee2dce0 2024-03-18T22:48:36.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.872+0000 7f6a4df66740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56483ee35f80 con 0x56483ee2d2f0 2024-03-18T22:48:36.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.872+0000 7f6a4df66740 10 client.0 _ll_register_callbacks cb 0x56483ed412d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:36.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.872+0000 7f6a49914640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee32e50 0x56483ee3ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a49914640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee32e50 0x56483ee3ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45690/0 (socket says 172.21.15.40:45690) 2024-03-18T22:48:36.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a48912640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2dce0 0x56483ee37e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a49914640 1 -- 172.21.15.40:0/2520134116 learned_addr learned my addr 172.21.15.40:0/2520134116 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a49113640 1 --2- 172.21.15.40:0/2520134116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee2d2f0 0x56483ee378c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a49113640 1 -- 172.21.15.40:0/2520134116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2dce0 msgr2=0x56483ee37e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a49113640 1 --2- 172.21.15.40:0/2520134116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2dce0 0x56483ee37e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a49113640 1 -- 172.21.15.40:0/2520134116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee32e50 msgr2=0x56483ee3ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a49113640 1 --2- 172.21.15.40:0/2520134116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee32e50 0x56483ee3ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a49113640 1 -- 172.21.15.40:0/2520134116 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56483eaa53f0 con 0x56483ee2d2f0 2024-03-18T22:48:36.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a48912640 1 --2- 172.21.15.40:0/2520134116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2dce0 0x56483ee37e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a49914640 1 --2- 172.21.15.40:0/2520134116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee32e50 0x56483ee3ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a49113640 1 --2- 172.21.15.40:0/2520134116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee2d2f0 0x56483ee378c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6a40036040 tx=0x7f6a4002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b03a01159f5512ab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:48:36.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a2cff9640 1 -- 172.21.15.40:0/2520134116 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a40049070 con 0x56483ee2d2f0 2024-03-18T22:48:36.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a2cff9640 1 -- 172.21.15.40:0/2520134116 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6a40004990 con 0x56483ee2d2f0 2024-03-18T22:48:36.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a4df66740 1 -- 172.21.15.40:0/2520134116 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a34099aa0 con 0x56483ee2d2f0 2024-03-18T22:48:36.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.873+0000 7f6a2cff9640 1 -- 172.21.15.40:0/2520134116 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6a4005e050 con 0x56483ee2d2f0 2024-03-18T22:48:36.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.874+0000 7f6a4df66740 10 client.26869 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:36.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.874+0000 7f6a4df66740 1 -- 172.21.15.40:0/2520134116 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56483eb6f7d0 con 0x56483ee2d2f0 2024-03-18T22:48:36.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.874+0000 7f6a2cff9640 1 -- 172.21.15.40:0/2520134116 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a34099aa0 con 0x56483ee2d2f0 2024-03-18T22:48:36.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.874+0000 7f6a2cff9640 1 -- 172.21.15.40:0/2520134116 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6a40040040 con 0x56483ee2d2f0 2024-03-18T22:48:36.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.874+0000 7f6a4df66740 10 client.26869 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:36.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.874+0000 7f6a4df66740 10 client.26869 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:48:36.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.874+0000 7f6a4df66740 20 client.26869 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:36.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.874+0000 7f6a4df66740 1 --2- 172.21.15.40:0/2520134116 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56483ee43770 0x56483ee45bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.874+0000 7f6a4df66740 4 client.26869 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:36.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.874+0000 7f6a4df66740 1 -- 172.21.15.40:0/2520134116 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56483edef3e0 con 0x56483ee43770 2024-03-18T22:48:36.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.874+0000 7f6a48912640 1 --2- 172.21.15.40:0/2520134116 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56483ee43770 0x56483ee45bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.875+0000 7f6a2cff9640 10 client.26869 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.875+0000 7f6a48912640 1 --2- 172.21.15.40:0/2520134116 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56483ee43770 0x56483ee45bc0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e1d9c69db999bbd4 server_cookie=afadddf940db565e in_seq=0 out_seq=0 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.875+0000 7f6a2cff9640 1 -- 172.21.15.40:0/2520134116 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56483edef3e0 con 0x56483ee43770 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.875+0000 7f6a2cff9640 10 client.26869 handle_command_reply: tid=0 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.875+0000 7f6a4df66740 1 client.26869 shutdown 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.875+0000 7f6a4df66740 1 -- 172.21.15.40:0/2520134116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee2d2f0 msgr2=0x56483ee378c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.875+0000 7f6a4df66740 1 --2- 172.21.15.40:0/2520134116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee2d2f0 0x56483ee378c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6a40036040 tx=0x7f6a4002bc30 comp rx=0 tx=0).stop 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.875+0000 7f6a4df66740 1 -- 172.21.15.40:0/2520134116 shutdown_connections 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.875+0000 7f6a4df66740 1 --2- 172.21.15.40:0/2520134116 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56483ee43770 0x56483ee45bc0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.875+0000 7f6a4df66740 1 --2- 172.21.15.40:0/2520134116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee32e50 0x56483ee3ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.875+0000 7f6a4df66740 1 --2- 172.21.15.40:0/2520134116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2dce0 0x56483ee37e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.875+0000 7f6a4df66740 1 --2- 172.21.15.40:0/2520134116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee2d2f0 0x56483ee378c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.875+0000 7f6a2cff9640 0 client.26869 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.875+0000 7f6a4df66740 1 -- 172.21.15.40:0/2520134116 >> 172.21.15.40:0/2520134116 conn(0x56483ee28860 msgr2=0x56483ee27ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.876+0000 7f6a4df66740 1 -- 172.21.15.40:0/2520134116 shutdown_connections 2024-03-18T22:48:36.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.876+0000 7f6a4df66740 1 -- 172.21.15.40:0/2520134116 wait complete. 2024-03-18T22:48:36.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.876+0000 7f6a4df66740 20 client.26869 trim_cache size 0 max 16384 2024-03-18T22:48:36.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.892+0000 7f6a4df66740 1 Processor -- start 2024-03-18T22:48:36.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.892+0000 7f6a4df66740 1 -- start start 2024-03-18T22:48:36.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.892+0000 7f6a4df66740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2d2f0 0x56483ee374d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.892+0000 7f6a4df66740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2dce0 0x56483ee3ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.892+0000 7f6a4df66740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee32e50 0x56483ee37a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.892+0000 7f6a4df66740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441aa620 con 0x56483ee2dce0 2024-03-18T22:48:36.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.892+0000 7f6a4df66740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441ac160 con 0x56483ee2d2f0 2024-03-18T22:48:36.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.892+0000 7f6a4df66740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441aacf0 con 0x56483ee32e50 2024-03-18T22:48:36.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.892+0000 7f6a48912640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2dce0 0x56483ee3ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.892+0000 7f6a49113640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2d2f0 0x56483ee374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.892+0000 7f6a48912640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2dce0 0x56483ee3ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45698/0 (socket says 172.21.15.40:45698) 2024-03-18T22:48:36.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.892+0000 7f6a48912640 1 -- 172.21.15.40:0/2541900373 learned_addr learned my addr 172.21.15.40:0/2541900373 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.892+0000 7f6a49914640 1 --2- 172.21.15.40:0/2541900373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee32e50 0x56483ee37a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.892+0000 7f6a48912640 1 -- 172.21.15.40:0/2541900373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee32e50 msgr2=0x56483ee37a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.893+0000 7f6a48912640 1 --2- 172.21.15.40:0/2541900373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee32e50 0x56483ee37a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.893+0000 7f6a48912640 1 -- 172.21.15.40:0/2541900373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2d2f0 msgr2=0x56483ee374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.893+0000 7f6a48912640 1 --2- 172.21.15.40:0/2541900373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2d2f0 0x56483ee374d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.893+0000 7f6a48912640 1 -- 172.21.15.40:0/2541900373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56483edfe880 con 0x56483ee2dce0 2024-03-18T22:48:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.893+0000 7f6a49113640 1 --2- 172.21.15.40:0/2541900373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2d2f0 0x56483ee374d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.893+0000 7f6a48912640 1 --2- 172.21.15.40:0/2541900373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2dce0 0x56483ee3ee00 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6a340ccbc0 tx=0x7f6a340ce040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28fb902693b2ab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.893+0000 7f6a2effd640 1 -- 172.21.15.40:0/2541900373 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a340860d0 con 0x56483ee2dce0 2024-03-18T22:48:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.893+0000 7f6a2effd640 1 -- 172.21.15.40:0/2541900373 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a34097e10 con 0x56483ee2dce0 2024-03-18T22:48:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.893+0000 7f6a2effd640 1 -- 172.21.15.40:0/2541900373 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a340b44d0 con 0x56483ee2dce0 2024-03-18T22:48:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 -- 172.21.15.40:0/2541900373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2dce0 msgr2=0x56483ee3ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 --2- 172.21.15.40:0/2541900373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2dce0 0x56483ee3ee00 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6a340ccbc0 tx=0x7f6a340ce040 comp rx=0 tx=0).stop 2024-03-18T22:48:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 -- 172.21.15.40:0/2541900373 shutdown_connections 2024-03-18T22:48:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 --2- 172.21.15.40:0/2541900373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee32e50 0x56483ee37a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 --2- 172.21.15.40:0/2541900373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2dce0 0x56483ee3ee00 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 --2- 172.21.15.40:0/2541900373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee2d2f0 0x56483ee374d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 -- 172.21.15.40:0/2541900373 >> 172.21.15.40:0/2541900373 conn(0x56483ee28860 msgr2=0x56483ee2e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 -- 172.21.15.40:0/2541900373 shutdown_connections 2024-03-18T22:48:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 -- 172.21.15.40:0/2541900373 wait complete. 2024-03-18T22:48:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 -- start start 2024-03-18T22:48:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 Processor -- start 2024-03-18T22:48:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2d2f0 0x56483ee3bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee2dce0 0x56483ee3c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee32e50 0x56483eea7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441aba90 con 0x56483ee2d2f0 2024-03-18T22:48:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441ab3c0 con 0x56483ee32e50 2024-03-18T22:48:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.894+0000 7f6a4df66740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441a8600 con 0x56483ee2dce0 2024-03-18T22:48:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a4df66740 10 client.0 _ll_register_callbacks cb 0x7f6a40009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a49113640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2d2f0 0x56483ee3bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a49113640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2d2f0 0x56483ee3bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45702/0 (socket says 172.21.15.40:45702) 2024-03-18T22:48:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a49113640 1 -- 172.21.15.40:0/83972816 learned_addr learned my addr 172.21.15.40:0/83972816 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a48912640 1 --2- 172.21.15.40:0/83972816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee2dce0 0x56483ee3c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a49914640 1 --2- 172.21.15.40:0/83972816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee32e50 0x56483eea7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a49113640 1 -- 172.21.15.40:0/83972816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee2dce0 msgr2=0x56483ee3c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a49113640 1 --2- 172.21.15.40:0/83972816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee2dce0 0x56483ee3c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a49113640 1 -- 172.21.15.40:0/83972816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee32e50 msgr2=0x56483eea7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a49113640 1 --2- 172.21.15.40:0/83972816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee32e50 0x56483eea7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a49113640 1 -- 172.21.15.40:0/83972816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56483eea7fc0 con 0x56483ee2d2f0 2024-03-18T22:48:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a49113640 1 --2- 172.21.15.40:0/83972816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2d2f0 0x56483ee3bba0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6a40003dd0 tx=0x7f6a40009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94c401354d7ef85 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:48:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a2cff9640 1 -- 172.21.15.40:0/83972816 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a40067060 con 0x56483ee2d2f0 2024-03-18T22:48:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a2cff9640 1 -- 172.21.15.40:0/83972816 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6a40004b90 con 0x56483ee2d2f0 2024-03-18T22:48:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a4df66740 1 -- 172.21.15.40:0/83972816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a34097e10 con 0x56483ee2d2f0 2024-03-18T22:48:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a2cff9640 1 -- 172.21.15.40:0/83972816 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6a4006e050 con 0x56483ee2d2f0 2024-03-18T22:48:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a4df66740 10 client.17064 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.895+0000 7f6a4df66740 1 -- 172.21.15.40:0/83972816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56483eea91a0 con 0x56483ee2d2f0 2024-03-18T22:48:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.896+0000 7f6a2cff9640 1 -- 172.21.15.40:0/83972816 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a34097e10 con 0x56483ee2d2f0 2024-03-18T22:48:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.896+0000 7f6a2cff9640 1 -- 172.21.15.40:0/83972816 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f6a4004d900 con 0x56483ee2d2f0 2024-03-18T22:48:36.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.896+0000 7f6a4df66740 10 client.17064 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:36.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.896+0000 7f6a4df66740 10 client.17064 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:48:36.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.896+0000 7f6a4df66740 20 client.17064 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:36.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.896+0000 7f6a4df66740 1 --2- 172.21.15.40:0/83972816 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56483eea9b80 0x56483eeabfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:36.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.896+0000 7f6a4df66740 4 client.17064 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:48:36.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.896+0000 7f6a4df66740 1 -- 172.21.15.40:0/83972816 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56483ed9e9f0 con 0x56483eea9b80 2024-03-18T22:48:36.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.896+0000 7f6a48912640 1 --2- 172.21.15.40:0/83972816 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56483eea9b80 0x56483eeabfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:36.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.896+0000 7f6a2cff9640 10 client.17064 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:48:36.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.896+0000 7f6a48912640 1 --2- 172.21.15.40:0/83972816 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56483eea9b80 0x56483eeabfa0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3a20a7f70364b152 server_cookie=4f002fd26e87e1fb in_seq=0 out_seq=0 2024-03-18T22:48:36.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.897+0000 7f6a2cff9640 1 -- 172.21.15.40:0/83972816 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56483ed9e9f0 con 0x56483eea9b80 2024-03-18T22:48:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.897+0000 7f6a2cff9640 10 client.17064 handle_command_reply: tid=0 2024-03-18T22:48:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.897+0000 7f6a4df66740 1 client.17064 shutdown 2024-03-18T22:48:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.897+0000 7f6a4df66740 1 -- 172.21.15.40:0/83972816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2d2f0 msgr2=0x56483ee3bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.897+0000 7f6a4df66740 1 --2- 172.21.15.40:0/83972816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2d2f0 0x56483ee3bba0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6a40003dd0 tx=0x7f6a40009230 comp rx=0 tx=0).stop 2024-03-18T22:48:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.897+0000 7f6a4df66740 1 -- 172.21.15.40:0/83972816 shutdown_connections 2024-03-18T22:48:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.897+0000 7f6a4df66740 1 --2- 172.21.15.40:0/83972816 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x56483eea9b80 0x56483eeabfa0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.897+0000 7f6a4df66740 1 --2- 172.21.15.40:0/83972816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56483ee32e50 0x56483eea7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.897+0000 7f6a4df66740 1 --2- 172.21.15.40:0/83972816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56483ee2dce0 0x56483ee3c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.897+0000 7f6a2cff9640 0 client.17064 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:48:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.897+0000 7f6a4df66740 1 --2- 172.21.15.40:0/83972816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56483ee2d2f0 0x56483ee3bba0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.897+0000 7f6a4df66740 1 -- 172.21.15.40:0/83972816 >> 172.21.15.40:0/83972816 conn(0x56483ee28860 msgr2=0x56483ee374d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.897+0000 7f6a4df66740 1 -- 172.21.15.40:0/83972816 shutdown_connections 2024-03-18T22:48:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.897+0000 7f6a4df66740 1 -- 172.21.15.40:0/83972816 wait complete. 2024-03-18T22:48:36.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.897+0000 7f6a4df66740 20 client.17064 trim_cache size 0 max 16384 2024-03-18T22:48:36.904 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:48:36.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.898+0000 7f6a4b377640 1 -- 172.21.15.40:0/2539367027 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a300a2800 msgr2=0x7f6a300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.898+0000 7f6a4b377640 1 --2- 172.21.15.40:0/2539367027 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a300a2800 0x7f6a300a4cc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6a40005ab0 tx=0x7f6a400041a0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.898+0000 7f6a4b377640 1 -- 172.21.15.40:0/2539367027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a44101910 msgr2=0x7f6a44197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.898+0000 7f6a4b377640 1 --2- 172.21.15.40:0/2539367027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a44101910 0x7f6a44197fa0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400e9f0 tx=0x7f6a3400eeb0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.898+0000 7f6a4b377640 1 -- 172.21.15.40:0/2539367027 shutdown_connections 2024-03-18T22:48:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.898+0000 7f6a4b377640 1 --2- 172.21.15.40:0/2539367027 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a300a2800 0x7f6a300a4cc0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.898+0000 7f6a4b377640 1 --2- 172.21.15.40:0/2539367027 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a441022d0 0x7f6a4419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.898+0000 7f6a4b377640 1 --2- 172.21.15.40:0/2539367027 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a44101910 0x7f6a44197fa0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.898+0000 7f6a4b377640 1 --2- 172.21.15.40:0/2539367027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a44100720 0x7f6a44197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.898+0000 7f6a4b377640 1 -- 172.21.15.40:0/2539367027 >> 172.21.15.40:0/2539367027 conn(0x7f6a440fbc90 msgr2=0x7f6a44068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.898+0000 7f6a4b377640 1 -- 172.21.15.40:0/2539367027 shutdown_connections 2024-03-18T22:48:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:36.898+0000 7f6a4b377640 1 -- 172.21.15.40:0/2539367027 wait complete. 2024-03-18T22:48:36.911 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:48:37.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.023+0000 7f3239510640 1 -- 172.21.15.40:0/439797290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32341022d0 msgr2=0x7f3234108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.023+0000 7f3239510640 1 --2- 172.21.15.40:0/439797290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32341022d0 0x7f3234108b60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f322c009ca0 tx=0x7f322c02c0f0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.023+0000 7f3239510640 1 -- 172.21.15.40:0/439797290 shutdown_connections 2024-03-18T22:48:37.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.023+0000 7f3239510640 1 --2- 172.21.15.40:0/439797290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32341022d0 0x7f3234108b60 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.023+0000 7f3239510640 1 --2- 172.21.15.40:0/439797290 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3234101910 0x7f3234101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.023+0000 7f3239510640 1 --2- 172.21.15.40:0/439797290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3234100720 0x7f3234100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.023+0000 7f3239510640 1 -- 172.21.15.40:0/439797290 >> 172.21.15.40:0/439797290 conn(0x7f32340fbc90 msgr2=0x7f32340fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.023+0000 7f3239510640 1 -- 172.21.15.40:0/439797290 shutdown_connections 2024-03-18T22:48:37.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.023+0000 7f3239510640 1 -- 172.21.15.40:0/439797290 wait complete. 2024-03-18T22:48:37.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.023+0000 7f3239510640 1 Processor -- start 2024-03-18T22:48:37.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f3239510640 1 -- start start 2024-03-18T22:48:37.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f3239510640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3234100720 0x7f32341046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f3239510640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3234101910 0x7f3234102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f3239510640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32341022d0 0x7f3234103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f3239510640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f323410c230 con 0x7f32341022d0 2024-03-18T22:48:37.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f3239510640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f323410c0b0 con 0x7f3234100720 2024-03-18T22:48:37.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f3239510640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f323410c3b0 con 0x7f3234101910 2024-03-18T22:48:37.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f32337fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32341022d0 0x7f3234103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f32327fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3234101910 0x7f3234102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f3232ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3234100720 0x7f32341046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f32327fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3234101910 0x7f3234102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38850/0 (socket says 172.21.15.40:38850) 2024-03-18T22:48:37.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f32327fc640 1 -- 172.21.15.40:0/1904788508 learned_addr learned my addr 172.21.15.40:0/1904788508 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:37.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f32337fe640 1 -- 172.21.15.40:0/1904788508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3234101910 msgr2=0x7f3234102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f32337fe640 1 --2- 172.21.15.40:0/1904788508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3234101910 0x7f3234102cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f32337fe640 1 -- 172.21.15.40:0/1904788508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3234100720 msgr2=0x7f32341046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f32337fe640 1 --2- 172.21.15.40:0/1904788508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3234100720 0x7f32341046b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f32337fe640 1 -- 172.21.15.40:0/1904788508 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3234103ac0 con 0x7f32341022d0 2024-03-18T22:48:37.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f32327fc640 1 --2- 172.21.15.40:0/1904788508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3234101910 0x7f3234102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:37.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.024+0000 7f3232ffd640 1 --2- 172.21.15.40:0/1904788508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3234100720 0x7f32341046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:37.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.025+0000 7f32337fe640 1 --2- 172.21.15.40:0/1904788508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32341022d0 0x7f3234103230 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f322c02c630 tx=0x7f322c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.025+0000 7f321bfff640 1 -- 172.21.15.40:0/1904788508 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c041020 con 0x7f32341022d0 2024-03-18T22:48:37.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.025+0000 7f321bfff640 1 -- 172.21.15.40:0/1904788508 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f322c003e20 con 0x7f32341022d0 2024-03-18T22:48:37.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.025+0000 7f321bfff640 1 -- 172.21.15.40:0/1904788508 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c039040 con 0x7f32341022d0 2024-03-18T22:48:37.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.025+0000 7f3239510640 1 -- 172.21.15.40:0/1904788508 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32341a1ac0 con 0x7f32341022d0 2024-03-18T22:48:37.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.025+0000 7f3239510640 1 -- 172.21.15.40:0/1904788508 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32341a1fc0 con 0x7f32341022d0 2024-03-18T22:48:37.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.028+0000 7f321bfff640 1 -- 172.21.15.40:0/1904788508 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f322c044020 con 0x7f32341022d0 2024-03-18T22:48:37.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.028+0000 7f323c1db740 1 Processor -- start 2024-03-18T22:48:37.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.028+0000 7f323c1db740 1 -- start start 2024-03-18T22:48:37.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.028+0000 7f323c1db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b32f0 0x55d01b0b3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.028+0000 7f323c1db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b3ce0 0x55d01b0b87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.028+0000 7f323c1db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b8e50 0x55d01b0bb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.028+0000 7f323c1db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d01b0174b0 con 0x55d01b0b3ce0 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.028+0000 7f323c1db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d01afbfd70 con 0x55d01b0b8e50 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.028+0000 7f323c1db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55d01b013550 con 0x55d01b0b32f0 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.028+0000 7f32327fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b32f0 0x55d01b0b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.028+0000 7f3232ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b3ce0 0x55d01b0b87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.028+0000 7f32327fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b32f0 0x55d01b0b3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38852/0 (socket says 172.21.15.40:38852) 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.028+0000 7f32327fc640 1 -- 172.21.15.40:0/2296927558 learned_addr learned my addr 172.21.15.40:0/2296927558 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.028+0000 7f32327fc640 1 --2- 172.21.15.40:0/2296927558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b8e50 0x55d01b0bb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.028+0000 7f321bfff640 1 --2- 172.21.15.40:0/1904788508 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f321c0a2800 0x7f321c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.029+0000 7f32337fe640 1 --2- 172.21.15.40:0/1904788508 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f321c0a2800 0x7f321c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.029+0000 7f321bfff640 1 -- 172.21.15.40:0/1904788508 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f322c0f0fe0 con 0x7f32341022d0 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.029+0000 7f32327fc640 1 -- 172.21.15.40:0/2296927558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b8e50 msgr2=0x55d01b0bb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.029+0000 7f32327fc640 1 --2- 172.21.15.40:0/2296927558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b8e50 0x55d01b0bb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.029+0000 7f32327fc640 1 -- 172.21.15.40:0/2296927558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b3ce0 msgr2=0x55d01b0b87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.029+0000 7f32327fc640 1 --2- 172.21.15.40:0/2296927558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b3ce0 0x55d01b0b87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.029+0000 7f32327fc640 1 -- 172.21.15.40:0/2296927558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d01afe2210 con 0x55d01b0b32f0 2024-03-18T22:48:37.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.029+0000 7f32327fc640 1 --2- 172.21.15.40:0/2296927558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b32f0 0x55d01b0b3710 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3228011ca0 tx=0x7f3228009c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a7a6ce66d1f2398 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.029+0000 7f32337fe640 1 --2- 172.21.15.40:0/1904788508 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f321c0a2800 0x7f321c0a4cc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f322c0b99f0 tx=0x7f322c0425b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.029+0000 7f32197fa640 1 -- 172.21.15.40:0/2296927558 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322800fc50 con 0x55d01b0b32f0 2024-03-18T22:48:37.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.029+0000 7f32197fa640 1 -- 172.21.15.40:0/2296927558 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f322800fdf0 con 0x55d01b0b32f0 2024-03-18T22:48:37.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.029+0000 7f32197fa640 1 -- 172.21.15.40:0/2296927558 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322800e610 con 0x55d01b0b32f0 2024-03-18T22:48:37.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.030+0000 7f323c1db740 1 -- 172.21.15.40:0/2296927558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b32f0 msgr2=0x55d01b0b3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.030+0000 7f323c1db740 1 --2- 172.21.15.40:0/2296927558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b32f0 0x55d01b0b3710 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3228011ca0 tx=0x7f3228009c60 comp rx=0 tx=0).stop 2024-03-18T22:48:37.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.030+0000 7f323c1db740 1 -- 172.21.15.40:0/2296927558 shutdown_connections 2024-03-18T22:48:37.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.030+0000 7f323c1db740 1 --2- 172.21.15.40:0/2296927558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b8e50 0x55d01b0bb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.030+0000 7f323c1db740 1 --2- 172.21.15.40:0/2296927558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b3ce0 0x55d01b0b87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.030+0000 7f323c1db740 1 --2- 172.21.15.40:0/2296927558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b32f0 0x55d01b0b3710 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.030+0000 7f323c1db740 1 -- 172.21.15.40:0/2296927558 >> 172.21.15.40:0/2296927558 conn(0x55d01b0ae860 msgr2=0x55d01b0aec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.030+0000 7f323c1db740 1 -- 172.21.15.40:0/2296927558 shutdown_connections 2024-03-18T22:48:37.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.030+0000 7f323c1db740 1 -- 172.21.15.40:0/2296927558 wait complete. 2024-03-18T22:48:37.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.030+0000 7f323c1db740 1 -- start start 2024-03-18T22:48:37.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f323c1db740 1 Processor -- start 2024-03-18T22:48:37.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f323c1db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b32f0 0x55d01b0bd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f323c1db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b3ce0 0x55d01b0bdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f323c1db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b8e50 0x55d01b0c4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f323c1db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d01b0bbe00 con 0x55d01b0b8e50 2024-03-18T22:48:37.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f323c1db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d01afd3310 con 0x55d01b0b32f0 2024-03-18T22:48:37.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f323c1db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55d01b0bbf80 con 0x55d01b0b3ce0 2024-03-18T22:48:37.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f323c1db740 10 client.0 _ll_register_callbacks cb 0x55d01afc72d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:37.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f3232ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b3ce0 0x55d01b0bdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f3232ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b3ce0 0x55d01b0bdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38854/0 (socket says 172.21.15.40:38854) 2024-03-18T22:48:37.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f32327fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b8e50 0x55d01b0c4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f3232ffd640 1 -- 172.21.15.40:0/2549408930 learned_addr learned my addr 172.21.15.40:0/2549408930 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:37.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f32327fc640 1 --2- 172.21.15.40:0/2549408930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b32f0 0x55d01b0bd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f3232ffd640 1 -- 172.21.15.40:0/2549408930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b32f0 msgr2=0x55d01b0bd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f3232ffd640 1 --2- 172.21.15.40:0/2549408930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b32f0 0x55d01b0bd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f3232ffd640 1 -- 172.21.15.40:0/2549408930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b8e50 msgr2=0x55d01b0c4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f3232ffd640 1 --2- 172.21.15.40:0/2549408930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b8e50 0x55d01b0c4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.031+0000 7f3232ffd640 1 -- 172.21.15.40:0/2549408930 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d01ad2b3f0 con 0x55d01b0b3ce0 2024-03-18T22:48:37.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f3232ffd640 1 --2- 172.21.15.40:0/2549408930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b3ce0 0x55d01b0bdd90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3224012d00 tx=0x7f322400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea33dcf53d130030 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f3216ff5640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:48:37.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f3216ff5640 1 -- 172.21.15.40:0/2549408930 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3224012110 con 0x55d01b0b3ce0 2024-03-18T22:48:37.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f323c1db740 1 -- 172.21.15.40:0/2549408930 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f322800fdf0 con 0x55d01b0b3ce0 2024-03-18T22:48:37.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f3216ff5640 1 -- 172.21.15.40:0/2549408930 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f3224004030 con 0x55d01b0b3ce0 2024-03-18T22:48:37.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f3216ff5640 1 -- 172.21.15.40:0/2549408930 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3224025050 con 0x55d01b0b3ce0 2024-03-18T22:48:37.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f323c1db740 10 client.26887 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:37.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f323c1db740 1 -- 172.21.15.40:0/2549408930 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d01adf57d0 con 0x55d01b0b3ce0 2024-03-18T22:48:37.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f3216ff5640 1 -- 172.21.15.40:0/2549408930 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f322800fdf0 con 0x55d01b0b3ce0 2024-03-18T22:48:37.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f3216ff5640 1 -- 172.21.15.40:0/2549408930 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3224010430 con 0x55d01b0b3ce0 2024-03-18T22:48:37.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f323c1db740 10 client.26887 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:37.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f323c1db740 10 client.26887 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:48:37.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f323c1db740 20 client.26887 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:37.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f323c1db740 1 --2- 172.21.15.40:0/2549408930 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d01b0c9800 0x55d01b0cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f323c1db740 4 client.26887 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:37.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.032+0000 7f323c1db740 1 -- 172.21.15.40:0/2549408930 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d01b0753e0 con 0x55d01b0c9800 2024-03-18T22:48:37.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.033+0000 7f32327fc640 1 --2- 172.21.15.40:0/2549408930 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d01b0c9800 0x55d01b0cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.033+0000 7f32327fc640 1 --2- 172.21.15.40:0/2549408930 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d01b0c9800 0x55d01b0cbc40 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c788d4dd327dfc0b server_cookie=dead1587f0104636 in_seq=0 out_seq=0 2024-03-18T22:48:37.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.033+0000 7f3216ff5640 10 client.26887 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:48:37.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.033+0000 7f3216ff5640 1 -- 172.21.15.40:0/2549408930 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d01b0753e0 con 0x55d01b0c9800 2024-03-18T22:48:37.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.034+0000 7f3216ff5640 10 client.26887 handle_command_reply: tid=0 2024-03-18T22:48:37.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.034+0000 7f323c1db740 1 client.26887 shutdown 2024-03-18T22:48:37.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.034+0000 7f323c1db740 1 -- 172.21.15.40:0/2549408930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b3ce0 msgr2=0x55d01b0bdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.034+0000 7f323c1db740 1 --2- 172.21.15.40:0/2549408930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b3ce0 0x55d01b0bdd90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3224012d00 tx=0x7f322400f790 comp rx=0 tx=0).stop 2024-03-18T22:48:37.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.034+0000 7f323c1db740 1 -- 172.21.15.40:0/2549408930 shutdown_connections 2024-03-18T22:48:37.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.034+0000 7f323c1db740 1 --2- 172.21.15.40:0/2549408930 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d01b0c9800 0x55d01b0cbc40 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.034+0000 7f323c1db740 1 --2- 172.21.15.40:0/2549408930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b8e50 0x55d01b0c4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.034+0000 7f323c1db740 1 --2- 172.21.15.40:0/2549408930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b3ce0 0x55d01b0bdd90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.034+0000 7f3216ff5640 0 client.26887 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:48:37.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.034+0000 7f323c1db740 1 --2- 172.21.15.40:0/2549408930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b32f0 0x55d01b0bd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.034+0000 7f323c1db740 1 -- 172.21.15.40:0/2549408930 >> 172.21.15.40:0/2549408930 conn(0x55d01b0ae860 msgr2=0x55d01b0adba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.034+0000 7f323c1db740 1 -- 172.21.15.40:0/2549408930 shutdown_connections 2024-03-18T22:48:37.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.034+0000 7f323c1db740 1 -- 172.21.15.40:0/2549408930 wait complete. 2024-03-18T22:48:37.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.034+0000 7f323c1db740 20 client.26887 trim_cache size 0 max 16384 2024-03-18T22:48:37.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.048+0000 7f323c1db740 1 Processor -- start 2024-03-18T22:48:37.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.048+0000 7f323c1db740 1 -- start start 2024-03-18T22:48:37.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.048+0000 7f323c1db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b32f0 0x55d01b0ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.048+0000 7f323c1db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b3ce0 0x55d01b0c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.048+0000 7f323c1db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b8e50 0x55d01b0ad850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.048+0000 7f323c1db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32341ac6a0 con 0x55d01b0b3ce0 2024-03-18T22:48:37.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.048+0000 7f323c1db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32341ae1e0 con 0x55d01b0b32f0 2024-03-18T22:48:37.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.048+0000 7f323c1db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f32341acd70 con 0x55d01b0b8e50 2024-03-18T22:48:37.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.048+0000 7f3232ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b3ce0 0x55d01b0c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.048+0000 7f32327fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b32f0 0x55d01b0ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.048+0000 7f3232ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b3ce0 0x55d01b0c4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45756/0 (socket says 172.21.15.40:45756) 2024-03-18T22:48:37.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.048+0000 7f32327fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b32f0 0x55d01b0ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49708/0 (socket says 172.21.15.40:49708) 2024-03-18T22:48:37.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.048+0000 7f3232ffd640 1 -- 172.21.15.40:0/312971054 learned_addr learned my addr 172.21.15.40:0/312971054 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:37.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.049+0000 7f32327fc640 1 --2- 172.21.15.40:0/312971054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b8e50 0x55d01b0ad850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.049+0000 7f3232ffd640 1 -- 172.21.15.40:0/312971054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b8e50 msgr2=0x55d01b0ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.049+0000 7f3232ffd640 1 --2- 172.21.15.40:0/312971054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b8e50 0x55d01b0ad850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.049+0000 7f3232ffd640 1 -- 172.21.15.40:0/312971054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b32f0 msgr2=0x55d01b0ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.049+0000 7f3232ffd640 1 --2- 172.21.15.40:0/312971054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b32f0 0x55d01b0ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.049+0000 7f3232ffd640 1 -- 172.21.15.40:0/312971054 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d01b0af7d0 con 0x55d01b0b3ce0 2024-03-18T22:48:37.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.049+0000 7f32327fc640 1 --2- 172.21.15.40:0/312971054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b32f0 0x55d01b0ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:37.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.049+0000 7f3232ffd640 1 --2- 172.21.15.40:0/312971054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b3ce0 0x55d01b0c4d90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f3224000c00 tx=0x7f3224009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a44c7a9120d656a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.049+0000 7f3218ff9640 1 -- 172.21.15.40:0/312971054 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322402e070 con 0x55d01b0b3ce0 2024-03-18T22:48:37.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.049+0000 7f3218ff9640 1 -- 172.21.15.40:0/312971054 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3224012110 con 0x55d01b0b3ce0 2024-03-18T22:48:37.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.049+0000 7f3218ff9640 1 -- 172.21.15.40:0/312971054 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3224022040 con 0x55d01b0b3ce0 2024-03-18T22:48:37.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.050+0000 7f323c1db740 1 -- 172.21.15.40:0/312971054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b3ce0 msgr2=0x55d01b0c4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.050+0000 7f323c1db740 1 --2- 172.21.15.40:0/312971054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b3ce0 0x55d01b0c4d90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f3224000c00 tx=0x7f3224009c00 comp rx=0 tx=0).stop 2024-03-18T22:48:37.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.050+0000 7f323c1db740 1 -- 172.21.15.40:0/312971054 shutdown_connections 2024-03-18T22:48:37.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.050+0000 7f323c1db740 1 --2- 172.21.15.40:0/312971054 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b8e50 0x55d01b0ad850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.050+0000 7f323c1db740 1 --2- 172.21.15.40:0/312971054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b3ce0 0x55d01b0c4d90 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.050+0000 7f323c1db740 1 --2- 172.21.15.40:0/312971054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b32f0 0x55d01b0ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.050+0000 7f323c1db740 1 -- 172.21.15.40:0/312971054 >> 172.21.15.40:0/312971054 conn(0x55d01b0ae860 msgr2=0x55d01b0bd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.050+0000 7f323c1db740 1 -- 172.21.15.40:0/312971054 shutdown_connections 2024-03-18T22:48:37.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.050+0000 7f323c1db740 1 -- 172.21.15.40:0/312971054 wait complete. 2024-03-18T22:48:37.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.050+0000 7f323c1db740 1 -- start start 2024-03-18T22:48:37.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.050+0000 7f323c1db740 1 Processor -- start 2024-03-18T22:48:37.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.050+0000 7f323c1db740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b32f0 0x55d01b0c2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f323c1db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b3ce0 0x55d01b0c2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f323c1db740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b8e50 0x55d01b0c39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f323c1db740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f32341adb10 con 0x55d01b0b32f0 2024-03-18T22:48:37.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f323c1db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32341ad440 con 0x55d01b0b3ce0 2024-03-18T22:48:37.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f323c1db740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f32341aa680 con 0x55d01b0b8e50 2024-03-18T22:48:37.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f323c1db740 10 client.0 _ll_register_callbacks cb 0x7f3224009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:37.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f32327fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b32f0 0x55d01b0c2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f32327fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b32f0 0x55d01b0c2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45760/0 (socket says 172.21.15.40:45760) 2024-03-18T22:48:37.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f32327fc640 1 -- 172.21.15.40:0/3401997421 learned_addr learned my addr 172.21.15.40:0/3401997421 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:37.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f3232ffd640 1 --2- 172.21.15.40:0/3401997421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b3ce0 0x55d01b0c2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f32327fc640 1 --2- 172.21.15.40:0/3401997421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b8e50 0x55d01b0c39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f32327fc640 1 -- 172.21.15.40:0/3401997421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b8e50 msgr2=0x55d01b0c39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f32327fc640 1 --2- 172.21.15.40:0/3401997421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b8e50 0x55d01b0c39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f32327fc640 1 -- 172.21.15.40:0/3401997421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b3ce0 msgr2=0x55d01b0c2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f32327fc640 1 --2- 172.21.15.40:0/3401997421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b3ce0 0x55d01b0c2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f32327fc640 1 -- 172.21.15.40:0/3401997421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d01b0c3fc0 con 0x55d01b0b32f0 2024-03-18T22:48:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f32327fc640 1 --2- 172.21.15.40:0/3401997421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b32f0 0x55d01b0c2540 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f3228005e00 tx=0x7f3228009c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6fdf672da9eecf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f3216ff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:48:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f3216ff5640 1 -- 172.21.15.40:0/3401997421 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3228005040 con 0x55d01b0b32f0 2024-03-18T22:48:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f3216ff5640 1 -- 172.21.15.40:0/3401997421 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f3228019040 con 0x55d01b0b32f0 2024-03-18T22:48:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f323c1db740 1 -- 172.21.15.40:0/3401997421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3224012110 con 0x55d01b0b32f0 2024-03-18T22:48:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f3216ff5640 1 -- 172.21.15.40:0/3401997421 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3228045050 con 0x55d01b0b32f0 2024-03-18T22:48:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f323c1db740 10 client.17082 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.051+0000 7f323c1db740 1 -- 172.21.15.40:0/3401997421 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d01b12fb80 con 0x55d01b0b32f0 2024-03-18T22:48:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.052+0000 7f3216ff5640 1 -- 172.21.15.40:0/3401997421 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3224012110 con 0x55d01b0b32f0 2024-03-18T22:48:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.052+0000 7f3216ff5640 1 -- 172.21.15.40:0/3401997421 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f322801d020 con 0x55d01b0b32f0 2024-03-18T22:48:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.052+0000 7f323c1db740 10 client.17082 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.052+0000 7f323c1db740 10 client.17082 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:48:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.052+0000 7f323c1db740 20 client.17082 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.052+0000 7f323c1db740 1 --2- 172.21.15.40:0/3401997421 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d01b130260 0x55d01b132680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.052+0000 7f323c1db740 4 client.17082 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:48:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.052+0000 7f323c1db740 1 -- 172.21.15.40:0/3401997421 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d01b0249f0 con 0x55d01b130260 2024-03-18T22:48:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.052+0000 7f3232ffd640 1 --2- 172.21.15.40:0/3401997421 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d01b130260 0x55d01b132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.052+0000 7f3216ff5640 10 client.17082 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:48:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.052+0000 7f3232ffd640 1 --2- 172.21.15.40:0/3401997421 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d01b130260 0x55d01b132680 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=80f0d446823fbfaf server_cookie=3b3e58d5877bae48 in_seq=0 out_seq=0 2024-03-18T22:48:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f3216ff5640 1 -- 172.21.15.40:0/3401997421 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d01b0249f0 con 0x55d01b130260 2024-03-18T22:48:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f3216ff5640 10 client.17082 handle_command_reply: tid=0 2024-03-18T22:48:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f323c1db740 1 client.17082 shutdown 2024-03-18T22:48:37.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f323c1db740 1 -- 172.21.15.40:0/3401997421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b32f0 msgr2=0x55d01b0c2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f323c1db740 1 --2- 172.21.15.40:0/3401997421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b32f0 0x55d01b0c2540 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f3228005e00 tx=0x7f3228009c60 comp rx=0 tx=0).stop 2024-03-18T22:48:37.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f323c1db740 1 -- 172.21.15.40:0/3401997421 shutdown_connections 2024-03-18T22:48:37.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f323c1db740 1 --2- 172.21.15.40:0/3401997421 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d01b130260 0x55d01b132680 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f323c1db740 1 --2- 172.21.15.40:0/3401997421 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d01b0b8e50 0x55d01b0c39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f323c1db740 1 --2- 172.21.15.40:0/3401997421 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d01b0b3ce0 0x55d01b0c2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f3216ff5640 0 client.17082 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:48:37.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f323c1db740 1 --2- 172.21.15.40:0/3401997421 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d01b0b32f0 0x55d01b0c2540 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f323c1db740 1 -- 172.21.15.40:0/3401997421 >> 172.21.15.40:0/3401997421 conn(0x55d01b0ae860 msgr2=0x55d01b0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f323c1db740 1 -- 172.21.15.40:0/3401997421 shutdown_connections 2024-03-18T22:48:37.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f323c1db740 1 -- 172.21.15.40:0/3401997421 wait complete. 2024-03-18T22:48:37.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f323c1db740 20 client.17082 trim_cache size 0 max 16384 2024-03-18T22:48:37.061 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:48:37.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f3239510640 1 -- 172.21.15.40:0/1904788508 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f321c0a2800 msgr2=0x7f321c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f3239510640 1 --2- 172.21.15.40:0/1904788508 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f321c0a2800 0x7f321c0a4cc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f322c0b99f0 tx=0x7f322c0425b0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f3239510640 1 -- 172.21.15.40:0/1904788508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32341022d0 msgr2=0x7f3234103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.053+0000 7f3239510640 1 --2- 172.21.15.40:0/1904788508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32341022d0 0x7f3234103230 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f322c02c630 tx=0x7f322c0328a0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.054+0000 7f3239510640 1 -- 172.21.15.40:0/1904788508 shutdown_connections 2024-03-18T22:48:37.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.054+0000 7f3239510640 1 --2- 172.21.15.40:0/1904788508 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f321c0a2800 0x7f321c0a4cc0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.054+0000 7f3239510640 1 --2- 172.21.15.40:0/1904788508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f32341022d0 0x7f3234103230 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.054+0000 7f3239510640 1 --2- 172.21.15.40:0/1904788508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3234101910 0x7f3234102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.054+0000 7f3239510640 1 --2- 172.21.15.40:0/1904788508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3234100720 0x7f32341046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.054+0000 7f3239510640 1 -- 172.21.15.40:0/1904788508 >> 172.21.15.40:0/1904788508 conn(0x7f32340fbc90 msgr2=0x7f32340fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.054+0000 7f3239510640 1 -- 172.21.15.40:0/1904788508 shutdown_connections 2024-03-18T22:48:37.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.054+0000 7f3239510640 1 -- 172.21.15.40:0/1904788508 wait complete. 2024-03-18T22:48:37.066 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:48:37.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.177+0000 7fcd391f4640 1 -- 172.21.15.40:0/1058592814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd341022d0 msgr2=0x7fcd34108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.177+0000 7fcd391f4640 1 --2- 172.21.15.40:0/1058592814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd341022d0 0x7fcd34108b60 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c009ca0 tx=0x7fcd2c02c0f0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.177+0000 7fcd391f4640 1 -- 172.21.15.40:0/1058592814 shutdown_connections 2024-03-18T22:48:37.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.177+0000 7fcd391f4640 1 --2- 172.21.15.40:0/1058592814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd341022d0 0x7fcd34108b60 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.177+0000 7fcd391f4640 1 --2- 172.21.15.40:0/1058592814 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd34101910 0x7fcd34101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.177+0000 7fcd391f4640 1 --2- 172.21.15.40:0/1058592814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd34100720 0x7fcd34100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.177+0000 7fcd391f4640 1 -- 172.21.15.40:0/1058592814 >> 172.21.15.40:0/1058592814 conn(0x7fcd340fbc90 msgr2=0x7fcd340fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.178+0000 7fcd391f4640 1 -- 172.21.15.40:0/1058592814 shutdown_connections 2024-03-18T22:48:37.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.178+0000 7fcd391f4640 1 -- 172.21.15.40:0/1058592814 wait complete. 2024-03-18T22:48:37.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.178+0000 7fcd391f4640 1 Processor -- start 2024-03-18T22:48:37.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.178+0000 7fcd391f4640 1 -- start start 2024-03-18T22:48:37.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd391f4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd34100720 0x7fcd34069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd391f4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd34101910 0x7fcd3419c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd391f4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd341022d0 0x7fcd3419ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd391f4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3410c250 con 0x7fcd34101910 2024-03-18T22:48:37.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd391f4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3410c0d0 con 0x7fcd341022d0 2024-03-18T22:48:37.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd391f4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3410c3d0 con 0x7fcd34100720 2024-03-18T22:48:37.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd3259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd34101910 0x7fcd3419c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd3259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd34101910 0x7fcd3419c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45768/0 (socket says 172.21.15.40:45768) 2024-03-18T22:48:37.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd3259c640 1 -- 172.21.15.40:0/1636779726 learned_addr learned my addr 172.21.15.40:0/1636779726 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:37.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/1636779726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd34100720 0x7fcd34069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd3359e640 1 --2- 172.21.15.40:0/1636779726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd341022d0 0x7fcd3419ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd32d9d640 1 -- 172.21.15.40:0/1636779726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd341022d0 msgr2=0x7fcd3419ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/1636779726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd341022d0 0x7fcd3419ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd32d9d640 1 -- 172.21.15.40:0/1636779726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd34101910 msgr2=0x7fcd3419c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/1636779726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd34101910 0x7fcd3419c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd32d9d640 1 -- 172.21.15.40:0/1636779726 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd3419f650 con 0x7fcd34100720 2024-03-18T22:48:37.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.179+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/1636779726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd34100720 0x7fcd34069510 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcd2400e9c0 tx=0x7fcd2400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.180+0000 7fcd1bfff640 1 -- 172.21.15.40:0/1636779726 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2400cd80 con 0x7fcd34100720 2024-03-18T22:48:37.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.180+0000 7fcd391f4640 1 -- 172.21.15.40:0/1636779726 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd3419f940 con 0x7fcd34100720 2024-03-18T22:48:37.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.180+0000 7fcd1bfff640 1 -- 172.21.15.40:0/1636779726 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd240108e0 con 0x7fcd34100720 2024-03-18T22:48:37.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.180+0000 7fcd1bfff640 1 -- 172.21.15.40:0/1636779726 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd24018860 con 0x7fcd34100720 2024-03-18T22:48:37.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.180+0000 7fcd391f4640 1 -- 172.21.15.40:0/1636779726 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd3419fe90 con 0x7fcd34100720 2024-03-18T22:48:37.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.182+0000 7fcd1bfff640 1 -- 172.21.15.40:0/1636779726 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcd2401f020 con 0x7fcd34100720 2024-03-18T22:48:37.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.182+0000 7fcd3bde6740 1 Processor -- start 2024-03-18T22:48:37.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.182+0000 7fcd3bde6740 1 -- start start 2024-03-18T22:48:37.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd3bde6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a56700 0x5645d6a56b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd3bde6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a570f0 0x5645d6a5bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd3bde6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a5c260 0x5645d6a5e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd3bde6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5645d69b64b0 con 0x5645d6a56700 2024-03-18T22:48:37.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd3bde6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5645d695ed70 con 0x5645d6a5c260 2024-03-18T22:48:37.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd3bde6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5645d69b2550 con 0x5645d6a570f0 2024-03-18T22:48:37.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd1bfff640 1 --2- 172.21.15.40:0/1636779726 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcd1c0a2800 0x7fcd1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd3359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a56700 0x5645d6a56b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd32d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a570f0 0x5645d6a5bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd3359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a56700 0x5645d6a56b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45772/0 (socket says 172.21.15.40:45772) 2024-03-18T22:48:37.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd3359e640 1 -- 172.21.15.40:0/3791623141 learned_addr learned my addr 172.21.15.40:0/3791623141 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:37.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd32d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a570f0 0x5645d6a5bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38896/0 (socket says 172.21.15.40:38896) 2024-03-18T22:48:37.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd3259c640 1 --2- 172.21.15.40:0/3791623141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a5c260 0x5645d6a5e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd3359e640 1 --2- 172.21.15.40:0/1636779726 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcd1c0a2800 0x7fcd1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd1bfff640 1 -- 172.21.15.40:0/1636779726 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fcd240cee10 con 0x7fcd34100720 2024-03-18T22:48:37.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd32d9d640 1 -- 172.21.15.40:0/3791623141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a5c260 msgr2=0x5645d6a5e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/3791623141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a5c260 0x5645d6a5e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd32d9d640 1 -- 172.21.15.40:0/3791623141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a56700 msgr2=0x5645d6a56b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/3791623141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a56700 0x5645d6a56b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd32d9d640 1 -- 172.21.15.40:0/3791623141 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645d6981210 con 0x5645d6a570f0 2024-03-18T22:48:37.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd3259c640 1 --2- 172.21.15.40:0/3791623141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a5c260 0x5645d6a5e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:37.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/3791623141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a570f0 0x5645d6a5bbf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcd240cc890 tx=0x7fcd2400cf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc8c2de13c037cdf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.183+0000 7fcd3359e640 1 --2- 172.21.15.40:0/1636779726 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcd1c0a2800 0x7fcd1c0a4cc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c032380 tx=0x7fcd2c02c0f0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.184+0000 7fcd197fa640 1 -- 172.21.15.40:0/3791623141 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2409bac0 con 0x5645d6a570f0 2024-03-18T22:48:37.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.184+0000 7fcd197fa640 1 -- 172.21.15.40:0/3791623141 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd2409b3c0 con 0x5645d6a570f0 2024-03-18T22:48:37.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.184+0000 7fcd197fa640 1 -- 172.21.15.40:0/3791623141 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd24097a30 con 0x5645d6a570f0 2024-03-18T22:48:37.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.184+0000 7fcd3bde6740 1 -- 172.21.15.40:0/3791623141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a570f0 msgr2=0x5645d6a5bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.184+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/3791623141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a570f0 0x5645d6a5bbf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcd240cc890 tx=0x7fcd2400cf50 comp rx=0 tx=0).stop 2024-03-18T22:48:37.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.184+0000 7fcd3bde6740 1 -- 172.21.15.40:0/3791623141 shutdown_connections 2024-03-18T22:48:37.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.184+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/3791623141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a5c260 0x5645d6a5e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.184+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/3791623141 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a570f0 0x5645d6a5bbf0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.184+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/3791623141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a56700 0x5645d6a56b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.184+0000 7fcd3bde6740 1 -- 172.21.15.40:0/3791623141 >> 172.21.15.40:0/3791623141 conn(0x5645d6a4d860 msgr2=0x5645d6a4dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.184+0000 7fcd3bde6740 1 -- 172.21.15.40:0/3791623141 shutdown_connections 2024-03-18T22:48:37.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.184+0000 7fcd3bde6740 1 -- 172.21.15.40:0/3791623141 wait complete. 2024-03-18T22:48:37.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3bde6740 1 -- start start 2024-03-18T22:48:37.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3bde6740 1 Processor -- start 2024-03-18T22:48:37.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3bde6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a56700 0x5645d6a60cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3bde6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a570f0 0x5645d6a611f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3bde6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a5c260 0x5645d6a68230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3bde6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5645d6a5f210 con 0x5645d6a5c260 2024-03-18T22:48:37.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3bde6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5645d6972310 con 0x5645d6a56700 2024-03-18T22:48:37.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3bde6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5645d6a5f390 con 0x5645d6a570f0 2024-03-18T22:48:37.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3bde6740 10 client.0 _ll_register_callbacks cb 0x5645d69662d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:37.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a5c260 0x5645d6a68230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3259c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a570f0 0x5645d6a611f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a5c260 0x5645d6a68230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45774/0 (socket says 172.21.15.40:45774) 2024-03-18T22:48:37.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3359e640 1 -- 172.21.15.40:0/2163703211 learned_addr learned my addr 172.21.15.40:0/2163703211 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:37.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/2163703211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a56700 0x5645d6a60cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3259c640 1 -- 172.21.15.40:0/2163703211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a56700 msgr2=0x5645d6a60cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3259c640 1 --2- 172.21.15.40:0/2163703211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a56700 0x5645d6a60cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3259c640 1 -- 172.21.15.40:0/2163703211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a5c260 msgr2=0x5645d6a68230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3259c640 1 --2- 172.21.15.40:0/2163703211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a5c260 0x5645d6a68230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3259c640 1 -- 172.21.15.40:0/2163703211 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645d66ca3f0 con 0x5645d6a570f0 2024-03-18T22:48:37.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd3359e640 1 --2- 172.21.15.40:0/2163703211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a5c260 0x5645d6a68230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:37.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.185+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/2163703211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a56700 0x5645d6a60cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:37.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd3259c640 1 --2- 172.21.15.40:0/2163703211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a570f0 0x5645d6a611f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009640 tx=0x7fcd28012c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16b6dfbf255c04cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd16ff5640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:48:37.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd16ff5640 1 -- 172.21.15.40:0/2163703211 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2800f890 con 0x5645d6a570f0 2024-03-18T22:48:37.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd16ff5640 1 -- 172.21.15.40:0/2163703211 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fcd2800fa30 con 0x5645d6a570f0 2024-03-18T22:48:37.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2163703211 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd2409b3c0 con 0x5645d6a570f0 2024-03-18T22:48:37.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd16ff5640 1 -- 172.21.15.40:0/2163703211 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcd2800fc30 con 0x5645d6a570f0 2024-03-18T22:48:37.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd3bde6740 10 client.26905 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:37.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2163703211 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5645d67947d0 con 0x5645d6a570f0 2024-03-18T22:48:37.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd16ff5640 1 -- 172.21.15.40:0/2163703211 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd2409b3c0 con 0x5645d6a570f0 2024-03-18T22:48:37.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd16ff5640 1 -- 172.21.15.40:0/2163703211 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcd2801c420 con 0x5645d6a570f0 2024-03-18T22:48:37.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd3bde6740 10 client.26905 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:37.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd3bde6740 10 client.26905 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:48:37.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd3bde6740 20 client.26905 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:37.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2163703211 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5645d6a6cc60 0x5645d6a6f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd3bde6740 4 client.26905 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:37.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.186+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2163703211 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645d6a143e0 con 0x5645d6a6cc60 2024-03-18T22:48:37.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.187+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/2163703211 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5645d6a6cc60 0x5645d6a6f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.187+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/2163703211 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5645d6a6cc60 0x5645d6a6f0e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bb50c2b8f8ffc1ce server_cookie=b0bd8b85153225ed in_seq=0 out_seq=0 2024-03-18T22:48:37.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.187+0000 7fcd16ff5640 10 client.26905 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:48:37.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.188+0000 7fcd16ff5640 1 -- 172.21.15.40:0/2163703211 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5645d6a143e0 con 0x5645d6a6cc60 2024-03-18T22:48:37.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.188+0000 7fcd16ff5640 10 client.26905 handle_command_reply: tid=0 2024-03-18T22:48:37.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.188+0000 7fcd3bde6740 1 client.26905 shutdown 2024-03-18T22:48:37.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.188+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2163703211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a570f0 msgr2=0x5645d6a611f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.188+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2163703211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a570f0 0x5645d6a611f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009640 tx=0x7fcd28012c30 comp rx=0 tx=0).stop 2024-03-18T22:48:37.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.188+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2163703211 shutdown_connections 2024-03-18T22:48:37.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.188+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2163703211 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5645d6a6cc60 0x5645d6a6f0e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.188+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2163703211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a5c260 0x5645d6a68230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.188+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2163703211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a570f0 0x5645d6a611f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.188+0000 7fcd16ff5640 0 client.26905 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:48:37.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.188+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2163703211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a56700 0x5645d6a60cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.188+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2163703211 >> 172.21.15.40:0/2163703211 conn(0x5645d6a4d860 msgr2=0x5645d6a4cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.188+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2163703211 shutdown_connections 2024-03-18T22:48:37.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.188+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2163703211 wait complete. 2024-03-18T22:48:37.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.188+0000 7fcd3bde6740 20 client.26905 trim_cache size 0 max 16384 2024-03-18T22:48:37.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd3bde6740 1 Processor -- start 2024-03-18T22:48:37.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd3bde6740 1 -- start start 2024-03-18T22:48:37.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd3bde6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a56700 0x5645d6a5e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd3bde6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a570f0 0x5645d6a681f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd3bde6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a5c260 0x5645d6a5f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd3bde6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcd341a8370 con 0x5645d6a56700 2024-03-18T22:48:37.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd3bde6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcd341a9eb0 con 0x5645d6a570f0 2024-03-18T22:48:37.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd3bde6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fcd341a8a40 con 0x5645d6a5c260 2024-03-18T22:48:37.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd32d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a56700 0x5645d6a5e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd32d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a56700 0x5645d6a5e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45784/0 (socket says 172.21.15.40:45784) 2024-03-18T22:48:37.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd32d9d640 1 -- 172.21.15.40:0/2943925890 learned_addr learned my addr 172.21.15.40:0/2943925890 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:37.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd3359e640 1 --2- 172.21.15.40:0/2943925890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a5c260 0x5645d6a5f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd3259c640 1 --2- 172.21.15.40:0/2943925890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a570f0 0x5645d6a681f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd3259c640 1 -- 172.21.15.40:0/2943925890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a5c260 msgr2=0x5645d6a5f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd3259c640 1 --2- 172.21.15.40:0/2943925890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a5c260 0x5645d6a5f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd3259c640 1 -- 172.21.15.40:0/2943925890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a56700 msgr2=0x5645d6a5e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.209+0000 7fcd3259c640 1 --2- 172.21.15.40:0/2943925890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a56700 0x5645d6a5e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.210+0000 7fcd3259c640 1 -- 172.21.15.40:0/2943925890 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645d6a23880 con 0x5645d6a570f0 2024-03-18T22:48:37.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.210+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/2943925890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a56700 0x5645d6a5e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:37.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.210+0000 7fcd3359e640 1 --2- 172.21.15.40:0/2943925890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a5c260 0x5645d6a5f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:37.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.210+0000 7fcd3259c640 1 --2- 172.21.15.40:0/2943925890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a570f0 0x5645d6a681f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcd2801c8e0 tx=0x7fcd280097a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bce1865200ef98a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.210+0000 7fcd18ff9640 1 -- 172.21.15.40:0/2943925890 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2801cb50 con 0x5645d6a570f0 2024-03-18T22:48:37.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.210+0000 7fcd18ff9640 1 -- 172.21.15.40:0/2943925890 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd2801ccf0 con 0x5645d6a570f0 2024-03-18T22:48:37.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.210+0000 7fcd18ff9640 1 -- 172.21.15.40:0/2943925890 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd28004680 con 0x5645d6a570f0 2024-03-18T22:48:37.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2943925890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a570f0 msgr2=0x5645d6a681f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2943925890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a570f0 0x5645d6a681f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcd2801c8e0 tx=0x7fcd280097a0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2943925890 shutdown_connections 2024-03-18T22:48:37.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2943925890 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a5c260 0x5645d6a5f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2943925890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a570f0 0x5645d6a681f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2943925890 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a56700 0x5645d6a5e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2943925890 >> 172.21.15.40:0/2943925890 conn(0x5645d6a4d860 msgr2=0x5645d6a57b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2943925890 shutdown_connections 2024-03-18T22:48:37.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2943925890 wait complete. 2024-03-18T22:48:37.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 -- start start 2024-03-18T22:48:37.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 Processor -- start 2024-03-18T22:48:37.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a56700 0x5645d6a65280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a570f0 0x5645d6a657c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a5c260 0x5645d6a667d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcd341a97e0 con 0x5645d6a5c260 2024-03-18T22:48:37.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcd341a9110 con 0x5645d6a570f0 2024-03-18T22:48:37.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fcd341a6350 con 0x5645d6a56700 2024-03-18T22:48:37.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3bde6740 10 client.0 _ll_register_callbacks cb 0x7fcd28009960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.211+0000 7fcd3359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a5c260 0x5645d6a667d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd3259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a570f0 0x5645d6a657c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd3359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a5c260 0x5645d6a667d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45798/0 (socket says 172.21.15.40:45798) 2024-03-18T22:48:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd3359e640 1 -- 172.21.15.40:0/2358190484 learned_addr learned my addr 172.21.15.40:0/2358190484 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/2358190484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a56700 0x5645d6a65280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd3259c640 1 -- 172.21.15.40:0/2358190484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a56700 msgr2=0x5645d6a65280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd3259c640 1 --2- 172.21.15.40:0/2358190484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a56700 0x5645d6a65280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd3259c640 1 -- 172.21.15.40:0/2358190484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a5c260 msgr2=0x5645d6a667d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd3259c640 1 --2- 172.21.15.40:0/2358190484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a5c260 0x5645d6a667d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd3259c640 1 -- 172.21.15.40:0/2358190484 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645d6a66dc0 con 0x5645d6a570f0 2024-03-18T22:48:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd3359e640 1 --2- 172.21.15.40:0/2358190484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a5c260 0x5645d6a667d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/2358190484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a56700 0x5645d6a65280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd3259c640 1 --2- 172.21.15.40:0/2358190484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a570f0 0x5645d6a657c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcd2801c8e0 tx=0x7fcd28009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53f5a1f9f6e51cfc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd16ff5640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:48:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd16ff5640 1 -- 172.21.15.40:0/2358190484 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd28005050 con 0x5645d6a570f0 2024-03-18T22:48:37.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2358190484 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd2801ccf0 con 0x5645d6a570f0 2024-03-18T22:48:37.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd16ff5640 1 -- 172.21.15.40:0/2358190484 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fcd28018040 con 0x5645d6a570f0 2024-03-18T22:48:37.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd16ff5640 1 -- 172.21.15.40:0/2358190484 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcd2803f050 con 0x5645d6a570f0 2024-03-18T22:48:37.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.212+0000 7fcd3bde6740 10 client.26921 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:37.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.213+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2358190484 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5645d6a680c0 con 0x5645d6a570f0 2024-03-18T22:48:37.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.213+0000 7fcd16ff5640 1 -- 172.21.15.40:0/2358190484 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd2801ccf0 con 0x5645d6a570f0 2024-03-18T22:48:37.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.213+0000 7fcd16ff5640 1 -- 172.21.15.40:0/2358190484 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcd2803a030 con 0x5645d6a570f0 2024-03-18T22:48:37.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.213+0000 7fcd3bde6740 10 client.26921 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:37.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.213+0000 7fcd3bde6740 10 client.26921 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:48:37.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.213+0000 7fcd3bde6740 20 client.26921 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:37.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.213+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2358190484 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5645d6ad32a0 0x5645d6ad56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.213+0000 7fcd3bde6740 4 client.26921 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:48:37.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.213+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2358190484 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5645d69c39f0 con 0x5645d6ad32a0 2024-03-18T22:48:37.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.213+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/2358190484 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5645d6ad32a0 0x5645d6ad56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd16ff5640 10 client.26921 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:48:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd32d9d640 1 --2- 172.21.15.40:0/2358190484 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5645d6ad32a0 0x5645d6ad56c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9ca7d3efa9f97f9c server_cookie=8ee6196c0d87fb57 in_seq=0 out_seq=0 2024-03-18T22:48:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd16ff5640 1 -- 172.21.15.40:0/2358190484 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5645d69c39f0 con 0x5645d6ad32a0 2024-03-18T22:48:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd16ff5640 10 client.26921 handle_command_reply: tid=0 2024-03-18T22:48:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd3bde6740 1 client.26921 shutdown 2024-03-18T22:48:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2358190484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a570f0 msgr2=0x5645d6a657c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2358190484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a570f0 0x5645d6a657c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcd2801c8e0 tx=0x7fcd28009f70 comp rx=0 tx=0).stop 2024-03-18T22:48:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2358190484 shutdown_connections 2024-03-18T22:48:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2358190484 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5645d6ad32a0 0x5645d6ad56c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2358190484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5645d6a5c260 0x5645d6a667d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2358190484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645d6a570f0 0x5645d6a657c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd16ff5640 0 client.26921 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:48:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd3bde6740 1 --2- 172.21.15.40:0/2358190484 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5645d6a56700 0x5645d6a65280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2358190484 >> 172.21.15.40:0/2358190484 conn(0x5645d6a4d860 msgr2=0x5645d6a48a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2358190484 shutdown_connections 2024-03-18T22:48:37.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd3bde6740 1 -- 172.21.15.40:0/2358190484 wait complete. 2024-03-18T22:48:37.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.214+0000 7fcd3bde6740 20 client.26921 trim_cache size 0 max 16384 2024-03-18T22:48:37.222 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:48:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.215+0000 7fcd391f4640 1 -- 172.21.15.40:0/1636779726 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcd1c0a2800 msgr2=0x7fcd1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.215+0000 7fcd391f4640 1 --2- 172.21.15.40:0/1636779726 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcd1c0a2800 0x7fcd1c0a4cc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c032380 tx=0x7fcd2c02c0f0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.215+0000 7fcd391f4640 1 -- 172.21.15.40:0/1636779726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd34100720 msgr2=0x7fcd34069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.215+0000 7fcd391f4640 1 --2- 172.21.15.40:0/1636779726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd34100720 0x7fcd34069510 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcd2400e9c0 tx=0x7fcd2400ee80 comp rx=0 tx=0).stop 2024-03-18T22:48:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.215+0000 7fcd391f4640 1 -- 172.21.15.40:0/1636779726 shutdown_connections 2024-03-18T22:48:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.215+0000 7fcd391f4640 1 --2- 172.21.15.40:0/1636779726 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcd1c0a2800 0x7fcd1c0a4cc0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.215+0000 7fcd391f4640 1 --2- 172.21.15.40:0/1636779726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd341022d0 0x7fcd3419ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.215+0000 7fcd391f4640 1 --2- 172.21.15.40:0/1636779726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcd34101910 0x7fcd3419c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.215+0000 7fcd391f4640 1 --2- 172.21.15.40:0/1636779726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcd34100720 0x7fcd34069510 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.215+0000 7fcd391f4640 1 -- 172.21.15.40:0/1636779726 >> 172.21.15.40:0/1636779726 conn(0x7fcd340fbc90 msgr2=0x7fcd34104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.215+0000 7fcd391f4640 1 -- 172.21.15.40:0/1636779726 shutdown_connections 2024-03-18T22:48:37.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.215+0000 7fcd391f4640 1 -- 172.21.15.40:0/1636779726 wait complete. 2024-03-18T22:48:37.227 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:48:37.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.349+0000 7f3562fcd640 1 -- 172.21.15.40:0/2127660104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c1033d0 msgr2=0x7f355c109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.349+0000 7f3562fcd640 1 --2- 172.21.15.40:0/2127660104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c1033d0 0x7f355c109c60 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f3554009ca0 tx=0x7f355402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.349+0000 7f3562fcd640 1 -- 172.21.15.40:0/2127660104 shutdown_connections 2024-03-18T22:48:37.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.349+0000 7f3562fcd640 1 --2- 172.21.15.40:0/2127660104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c1033d0 0x7f355c109c60 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.349+0000 7f3562fcd640 1 --2- 172.21.15.40:0/2127660104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f355c102a10 0x7f355c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.349+0000 7f3562fcd640 1 --2- 172.21.15.40:0/2127660104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f355c101820 0x7f355c101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.349+0000 7f3562fcd640 1 -- 172.21.15.40:0/2127660104 >> 172.21.15.40:0/2127660104 conn(0x7f355c0fcd90 msgr2=0x7f355c0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.349+0000 7f3562fcd640 1 -- 172.21.15.40:0/2127660104 shutdown_connections 2024-03-18T22:48:37.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.349+0000 7f3562fcd640 1 -- 172.21.15.40:0/2127660104 wait complete. 2024-03-18T22:48:37.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3562fcd640 1 Processor -- start 2024-03-18T22:48:37.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3562fcd640 1 -- start start 2024-03-18T22:48:37.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3562fcd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f355c101820 0x7f355c1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3562fcd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f355c102a10 0x7f355c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3562fcd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c1033d0 0x7f355c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3562fcd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f355c10d350 con 0x7f355c1033d0 2024-03-18T22:48:37.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3562fcd640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f355c10d1d0 con 0x7f355c102a10 2024-03-18T22:48:37.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3562fcd640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f355c10d4d0 con 0x7f355c101820 2024-03-18T22:48:37.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f356156a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c1033d0 0x7f355c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3560d69640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f355c101820 0x7f355c1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3560d69640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f355c101820 0x7f355c1979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38924/0 (socket says 172.21.15.40:38924) 2024-03-18T22:48:37.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f355bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f355c102a10 0x7f355c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3560d69640 1 -- 172.21.15.40:0/2298576462 learned_addr learned my addr 172.21.15.40:0/2298576462 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:37.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3560d69640 1 -- 172.21.15.40:0/2298576462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f355c102a10 msgr2=0x7f355c197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3560d69640 1 --2- 172.21.15.40:0/2298576462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f355c102a10 0x7f355c197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3560d69640 1 -- 172.21.15.40:0/2298576462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c1033d0 msgr2=0x7f355c19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3560d69640 1 --2- 172.21.15.40:0/2298576462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c1033d0 0x7f355c19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f3560d69640 1 -- 172.21.15.40:0/2298576462 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355c19f5f0 con 0x7f355c101820 2024-03-18T22:48:37.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.350+0000 7f356156a640 1 --2- 172.21.15.40:0/2298576462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c1033d0 0x7f355c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:37.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.351+0000 7f3560d69640 1 --2- 172.21.15.40:0/2298576462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f355c101820 0x7f355c1979d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f355000e9c0 tx=0x7f355000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.351+0000 7f3559ffb640 1 -- 172.21.15.40:0/2298576462 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355000cd80 con 0x7f355c101820 2024-03-18T22:48:37.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.351+0000 7f3562fcd640 1 -- 172.21.15.40:0/2298576462 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f355c19f8e0 con 0x7f355c101820 2024-03-18T22:48:37.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.351+0000 7f3559ffb640 1 -- 172.21.15.40:0/2298576462 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35500035f0 con 0x7f355c101820 2024-03-18T22:48:37.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.351+0000 7f3559ffb640 1 -- 172.21.15.40:0/2298576462 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3550010690 con 0x7f355c101820 2024-03-18T22:48:37.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.351+0000 7f3562fcd640 1 -- 172.21.15.40:0/2298576462 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f355c19fe30 con 0x7f355c101820 2024-03-18T22:48:37.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.353+0000 7f3559ffb640 1 -- 172.21.15.40:0/2298576462 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f35500037a0 con 0x7f355c101820 2024-03-18T22:48:37.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.354+0000 7f3559ffb640 1 --2- 172.21.15.40:0/2298576462 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f354c0a2800 0x7f354c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.354+0000 7f3565bb8740 1 Processor -- start 2024-03-18T22:48:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.354+0000 7f355bfff640 1 --2- 172.21.15.40:0/2298576462 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f354c0a2800 0x7f354c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.354+0000 7f3559ffb640 1 -- 172.21.15.40:0/2298576462 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f35500cf3a0 con 0x7f355c101820 2024-03-18T22:48:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.354+0000 7f3565bb8740 1 -- start start 2024-03-18T22:48:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.354+0000 7f3565bb8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f1582f0 0x55b76f158710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.354+0000 7f3565bb8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f158ce0 0x55b76f15d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.354+0000 7f3565bb8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f15de50 0x55b76f1602e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.354+0000 7f3565bb8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b76f0bc4b0 con 0x55b76f1582f0 2024-03-18T22:48:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.354+0000 7f3565bb8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b76f064d70 con 0x55b76f158ce0 2024-03-18T22:48:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.354+0000 7f3565bb8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55b76f0b8550 con 0x55b76f15de50 2024-03-18T22:48:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.355+0000 7f3560d69640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f1582f0 0x55b76f158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.355+0000 7f3560d69640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f1582f0 0x55b76f158710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45834/0 (socket says 172.21.15.40:45834) 2024-03-18T22:48:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.355+0000 7f3560d69640 1 -- 172.21.15.40:0/645888226 learned_addr learned my addr 172.21.15.40:0/645888226 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.355+0000 7f356156a640 1 --2- 172.21.15.40:0/645888226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f15de50 0x55b76f1602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.355+0000 7f355bfff640 1 --2- 172.21.15.40:0/2298576462 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f354c0a2800 0x7f354c0a4cc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f355c198ee0 tx=0x7f3548005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.355+0000 7f355bfff640 1 --2- 172.21.15.40:0/645888226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f158ce0 0x55b76f15d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.355+0000 7f3560d69640 1 -- 172.21.15.40:0/645888226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f15de50 msgr2=0x55b76f1602e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.355+0000 7f3560d69640 1 --2- 172.21.15.40:0/645888226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f15de50 0x55b76f1602e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.355+0000 7f3560d69640 1 -- 172.21.15.40:0/645888226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f158ce0 msgr2=0x55b76f15d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.355+0000 7f3560d69640 1 --2- 172.21.15.40:0/645888226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f158ce0 0x55b76f15d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.355+0000 7f3560d69640 1 -- 172.21.15.40:0/645888226 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b76f087210 con 0x55b76f1582f0 2024-03-18T22:48:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.355+0000 7f3560d69640 1 --2- 172.21.15.40:0/645888226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f1582f0 0x55b76f158710 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f3550098320 tx=0x7f355009b2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd9655bcbb6c568 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.355+0000 7f3546ffd640 1 -- 172.21.15.40:0/645888226 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355009b950 con 0x55b76f1582f0 2024-03-18T22:48:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.355+0000 7f3546ffd640 1 -- 172.21.15.40:0/645888226 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f355009faa0 con 0x55b76f1582f0 2024-03-18T22:48:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.355+0000 7f3546ffd640 1 -- 172.21.15.40:0/645888226 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35500860d0 con 0x55b76f1582f0 2024-03-18T22:48:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.356+0000 7f3565bb8740 1 -- 172.21.15.40:0/645888226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f1582f0 msgr2=0x55b76f158710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.356+0000 7f3565bb8740 1 --2- 172.21.15.40:0/645888226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f1582f0 0x55b76f158710 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f3550098320 tx=0x7f355009b2b0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.356+0000 7f3565bb8740 1 -- 172.21.15.40:0/645888226 shutdown_connections 2024-03-18T22:48:37.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.356+0000 7f3565bb8740 1 --2- 172.21.15.40:0/645888226 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f15de50 0x55b76f1602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.356+0000 7f3565bb8740 1 --2- 172.21.15.40:0/645888226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f158ce0 0x55b76f15d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.356+0000 7f3565bb8740 1 --2- 172.21.15.40:0/645888226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f1582f0 0x55b76f158710 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.356+0000 7f3565bb8740 1 -- 172.21.15.40:0/645888226 >> 172.21.15.40:0/645888226 conn(0x55b76f153860 msgr2=0x55b76f153c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.356+0000 7f3565bb8740 1 -- 172.21.15.40:0/645888226 shutdown_connections 2024-03-18T22:48:37.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.356+0000 7f3565bb8740 1 -- 172.21.15.40:0/645888226 wait complete. 2024-03-18T22:48:37.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.357+0000 7f3565bb8740 1 -- start start 2024-03-18T22:48:37.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.357+0000 7f3565bb8740 1 Processor -- start 2024-03-18T22:48:37.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.357+0000 7f3565bb8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f1582f0 0x55b76f162780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.357+0000 7f3565bb8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f162cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.357+0000 7f3565bb8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f15de50 0x55b76f169d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.357+0000 7f3565bb8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55b76f160d40 con 0x55b76f158ce0 2024-03-18T22:48:37.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.357+0000 7f3565bb8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b76f078310 con 0x55b76f1582f0 2024-03-18T22:48:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.357+0000 7f3565bb8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55b76f160ec0 con 0x55b76f15de50 2024-03-18T22:48:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.357+0000 7f3565bb8740 10 client.0 _ll_register_callbacks cb 0x55b76f06c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.357+0000 7f355bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f162cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.357+0000 7f3560d69640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f1582f0 0x55b76f162780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.357+0000 7f355bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f162cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45850/0 (socket says 172.21.15.40:45850) 2024-03-18T22:48:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.357+0000 7f355bfff640 1 -- 172.21.15.40:0/1711651613 learned_addr learned my addr 172.21.15.40:0/1711651613 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.357+0000 7f356156a640 1 --2- 172.21.15.40:0/1711651613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f15de50 0x55b76f169d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.358+0000 7f355bfff640 1 -- 172.21.15.40:0/1711651613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f15de50 msgr2=0x55b76f169d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.358+0000 7f355bfff640 1 --2- 172.21.15.40:0/1711651613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f15de50 0x55b76f169d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.358+0000 7f355bfff640 1 -- 172.21.15.40:0/1711651613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f1582f0 msgr2=0x55b76f162780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.358+0000 7f355bfff640 1 --2- 172.21.15.40:0/1711651613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f1582f0 0x55b76f162780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.358+0000 7f355bfff640 1 -- 172.21.15.40:0/1711651613 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b76edd03f0 con 0x55b76f158ce0 2024-03-18T22:48:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.358+0000 7f3560d69640 1 --2- 172.21.15.40:0/1711651613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f1582f0 0x55b76f162780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:37.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.358+0000 7f355bfff640 1 --2- 172.21.15.40:0/1711651613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f162cc0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f354800ed30 tx=0x7f3548004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494c1e87a3a17b73 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.358+0000 7f353bfff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:48:37.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.358+0000 7f353bfff640 1 -- 172.21.15.40:0/1711651613 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354801db20 con 0x55b76f158ce0 2024-03-18T22:48:37.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.358+0000 7f353bfff640 1 -- 172.21.15.40:0/1711651613 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f354801dcc0 con 0x55b76f158ce0 2024-03-18T22:48:37.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.358+0000 7f3565bb8740 1 -- 172.21.15.40:0/1711651613 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f355009faa0 con 0x55b76f158ce0 2024-03-18T22:48:37.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.358+0000 7f353bfff640 1 -- 172.21.15.40:0/1711651613 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f35480047a0 con 0x55b76f158ce0 2024-03-18T22:48:37.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.358+0000 7f3565bb8740 10 client.17112 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:37.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.358+0000 7f3565bb8740 1 -- 172.21.15.40:0/1711651613 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b76ee9a7d0 con 0x55b76f158ce0 2024-03-18T22:48:37.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.359+0000 7f353bfff640 1 -- 172.21.15.40:0/1711651613 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f355009faa0 con 0x55b76f158ce0 2024-03-18T22:48:37.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.359+0000 7f353bfff640 1 -- 172.21.15.40:0/1711651613 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f3548026410 con 0x55b76f158ce0 2024-03-18T22:48:37.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.359+0000 7f3565bb8740 10 client.17112 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:37.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.359+0000 7f3565bb8740 10 client.17112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:37.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.359+0000 7f3565bb8740 20 client.17112 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:37.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.359+0000 7f3565bb8740 1 --2- 172.21.15.40:0/1711651613 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55b76f16e6d0 0x55b76f170b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.359+0000 7f3565bb8740 4 client.17112 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:37.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.359+0000 7f3565bb8740 1 -- 172.21.15.40:0/1711651613 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b76f11a3e0 con 0x55b76f16e6d0 2024-03-18T22:48:37.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.359+0000 7f3560d69640 1 --2- 172.21.15.40:0/1711651613 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55b76f16e6d0 0x55b76f170b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.359+0000 7f353bfff640 10 client.17112 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:37.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.359+0000 7f3560d69640 1 --2- 172.21.15.40:0/1711651613 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55b76f16e6d0 0x55b76f170b10 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e79b6c617b8d6fb server_cookie=dd4669137e22cd99 in_seq=0 out_seq=0 2024-03-18T22:48:37.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.360+0000 7f353bfff640 1 -- 172.21.15.40:0/1711651613 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b76f11a3e0 con 0x55b76f16e6d0 2024-03-18T22:48:37.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.360+0000 7f353bfff640 10 client.17112 handle_command_reply: tid=0 2024-03-18T22:48:37.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.360+0000 7f3565bb8740 1 client.17112 shutdown 2024-03-18T22:48:37.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.361+0000 7f3565bb8740 1 -- 172.21.15.40:0/1711651613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 msgr2=0x55b76f162cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.361+0000 7f3565bb8740 1 --2- 172.21.15.40:0/1711651613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f162cc0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f354800ed30 tx=0x7f3548004730 comp rx=0 tx=0).stop 2024-03-18T22:48:37.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.361+0000 7f3565bb8740 1 -- 172.21.15.40:0/1711651613 shutdown_connections 2024-03-18T22:48:37.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.361+0000 7f3565bb8740 1 --2- 172.21.15.40:0/1711651613 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f15de50 0x55b76f169d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.361+0000 7f3565bb8740 1 --2- 172.21.15.40:0/1711651613 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f162cc0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.361+0000 7f3565bb8740 1 --2- 172.21.15.40:0/1711651613 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55b76f16e6d0 0x55b76f170b10 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.361+0000 7f3565bb8740 1 --2- 172.21.15.40:0/1711651613 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f1582f0 0x55b76f162780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.361+0000 7f353bfff640 0 client.17112 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:37.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.361+0000 7f3565bb8740 1 -- 172.21.15.40:0/1711651613 >> 172.21.15.40:0/1711651613 conn(0x55b76f153860 msgr2=0x55b76f152ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.361+0000 7f3565bb8740 1 -- 172.21.15.40:0/1711651613 shutdown_connections 2024-03-18T22:48:37.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.361+0000 7f3565bb8740 1 -- 172.21.15.40:0/1711651613 wait complete. 2024-03-18T22:48:37.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.361+0000 7f3565bb8740 20 client.17112 trim_cache size 0 max 16384 2024-03-18T22:48:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.382+0000 7f3565bb8740 1 Processor -- start 2024-03-18T22:48:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.382+0000 7f3565bb8740 1 -- start start 2024-03-18T22:48:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.382+0000 7f3565bb8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f1582f0 0x55b76f15a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.382+0000 7f3565bb8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f15a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.382+0000 7f3565bb8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f15de50 0x55b76f169cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.382+0000 7f3565bb8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f355c1aa5b0 con 0x55b76f158ce0 2024-03-18T22:48:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.382+0000 7f3565bb8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f355c1ac0f0 con 0x55b76f15de50 2024-03-18T22:48:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.382+0000 7f3565bb8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f355c1aac80 con 0x55b76f1582f0 2024-03-18T22:48:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.382+0000 7f355bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f15a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.383+0000 7f355bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f15a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45866/0 (socket says 172.21.15.40:45866) 2024-03-18T22:48:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.383+0000 7f355bfff640 1 -- 172.21.15.40:0/32862267 learned_addr learned my addr 172.21.15.40:0/32862267 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:37.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.383+0000 7f356156a640 1 --2- 172.21.15.40:0/32862267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f15de50 0x55b76f169cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.383+0000 7f3560d69640 1 --2- 172.21.15.40:0/32862267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f1582f0 0x55b76f15a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.383+0000 7f355bfff640 1 -- 172.21.15.40:0/32862267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f1582f0 msgr2=0x55b76f15a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.383+0000 7f355bfff640 1 --2- 172.21.15.40:0/32862267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f1582f0 0x55b76f15a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.383+0000 7f355bfff640 1 -- 172.21.15.40:0/32862267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f15de50 msgr2=0x55b76f169cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.383+0000 7f355bfff640 1 --2- 172.21.15.40:0/32862267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f15de50 0x55b76f169cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.383+0000 7f355bfff640 1 -- 172.21.15.40:0/32862267 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b76f129880 con 0x55b76f158ce0 2024-03-18T22:48:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.383+0000 7f355bfff640 1 --2- 172.21.15.40:0/32862267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f15a760 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3548016040 tx=0x7f3548004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6be85ecb19e7aaf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.383+0000 7f35467fc640 1 -- 172.21.15.40:0/32862267 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354803e070 con 0x55b76f158ce0 2024-03-18T22:48:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.383+0000 7f35467fc640 1 -- 172.21.15.40:0/32862267 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3548026b40 con 0x55b76f158ce0 2024-03-18T22:48:37.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.383+0000 7f35467fc640 1 -- 172.21.15.40:0/32862267 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35480264d0 con 0x55b76f158ce0 2024-03-18T22:48:37.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.384+0000 7f3565bb8740 1 -- 172.21.15.40:0/32862267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 msgr2=0x55b76f15a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.384+0000 7f3565bb8740 1 --2- 172.21.15.40:0/32862267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f15a760 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3548016040 tx=0x7f3548004680 comp rx=0 tx=0).stop 2024-03-18T22:48:37.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.384+0000 7f3565bb8740 1 -- 172.21.15.40:0/32862267 shutdown_connections 2024-03-18T22:48:37.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.384+0000 7f3565bb8740 1 --2- 172.21.15.40:0/32862267 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f15de50 0x55b76f169cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.384+0000 7f3565bb8740 1 --2- 172.21.15.40:0/32862267 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f15a760 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.384+0000 7f3565bb8740 1 --2- 172.21.15.40:0/32862267 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f1582f0 0x55b76f15a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.384+0000 7f3565bb8740 1 -- 172.21.15.40:0/32862267 >> 172.21.15.40:0/32862267 conn(0x55b76f153860 msgr2=0x55b76f153c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.384+0000 7f3565bb8740 1 -- 172.21.15.40:0/32862267 shutdown_connections 2024-03-18T22:48:37.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.384+0000 7f3565bb8740 1 -- 172.21.15.40:0/32862267 wait complete. 2024-03-18T22:48:37.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.384+0000 7f3565bb8740 1 -- start start 2024-03-18T22:48:37.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.384+0000 7f3565bb8740 1 Processor -- start 2024-03-18T22:48:37.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f3565bb8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f1582f0 0x55b76f167360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f3565bb8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f1678a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f3565bb8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f15de50 0x55b76f168860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f3565bb8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f355c1aba20 con 0x55b76f158ce0 2024-03-18T22:48:37.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f3565bb8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f355c1ab350 con 0x55b76f15de50 2024-03-18T22:48:37.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f3565bb8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f355c1a8590 con 0x55b76f1582f0 2024-03-18T22:48:37.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f3565bb8740 10 client.0 _ll_register_callbacks cb 0x7f354800e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:37.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f355bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f1678a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f3560d69640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f1582f0 0x55b76f167360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f355bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f1678a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45876/0 (socket says 172.21.15.40:45876) 2024-03-18T22:48:37.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f356156a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f15de50 0x55b76f168860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f355bfff640 1 -- 172.21.15.40:0/273531357 learned_addr learned my addr 172.21.15.40:0/273531357 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:37.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f3560d69640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f1582f0 0x55b76f167360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38960/0 (socket says 172.21.15.40:38960) 2024-03-18T22:48:37.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f3560d69640 1 -- 172.21.15.40:0/273531357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f15de50 msgr2=0x55b76f168860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f3560d69640 1 --2- 172.21.15.40:0/273531357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f15de50 0x55b76f168860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f3560d69640 1 -- 172.21.15.40:0/273531357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 msgr2=0x55b76f1678a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:37.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f3560d69640 1 --2- 172.21.15.40:0/273531357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f1678a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f3560d69640 1 -- 172.21.15.40:0/273531357 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b76f168e50 con 0x55b76f1582f0 2024-03-18T22:48:37.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f355bfff640 1 --2- 172.21.15.40:0/273531357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f1678a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:37.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f3560d69640 1 --2- 172.21.15.40:0/273531357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f1582f0 0x55b76f167360 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3550098320 tx=0x7f3550097d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8ef09bc471710e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:37.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.385+0000 7f353bfff640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:48:37.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.386+0000 7f353bfff640 1 -- 172.21.15.40:0/273531357 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355009b950 con 0x55b76f1582f0 2024-03-18T22:48:37.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.386+0000 7f353bfff640 1 -- 172.21.15.40:0/273531357 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f3550097fc0 con 0x55b76f1582f0 2024-03-18T22:48:37.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.386+0000 7f3565bb8740 1 -- 172.21.15.40:0/273531357 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3548026b40 con 0x55b76f1582f0 2024-03-18T22:48:37.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.386+0000 7f353bfff640 1 -- 172.21.15.40:0/273531357 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f35500a40f0 con 0x55b76f1582f0 2024-03-18T22:48:37.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.386+0000 7f3565bb8740 10 client.26929 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:37.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.386+0000 7f3565bb8740 1 -- 172.21.15.40:0/273531357 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b76f1d4a10 con 0x55b76f1582f0 2024-03-18T22:48:37.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.386+0000 7f353bfff640 1 -- 172.21.15.40:0/273531357 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3548026b40 con 0x55b76f1582f0 2024-03-18T22:48:37.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.386+0000 7f353bfff640 1 -- 172.21.15.40:0/273531357 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f35500976a0 con 0x55b76f1582f0 2024-03-18T22:48:37.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.386+0000 7f3565bb8740 10 client.26929 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:37.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.386+0000 7f3565bb8740 10 client.26929 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:37.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.386+0000 7f3565bb8740 20 client.26929 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:37.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.386+0000 7f3565bb8740 1 --2- 172.21.15.40:0/273531357 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55b76f1d4ee0 0x55b76f1d7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:37.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.386+0000 7f3565bb8740 4 client.26929 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:48:37.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.386+0000 7f3565bb8740 1 -- 172.21.15.40:0/273531357 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b76f0c99f0 con 0x55b76f1d4ee0 2024-03-18T22:48:37.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.386+0000 7f355bfff640 1 --2- 172.21.15.40:0/273531357 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55b76f1d4ee0 0x55b76f1d7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:37.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f353bfff640 10 client.26929 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:37.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f355bfff640 1 --2- 172.21.15.40:0/273531357 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55b76f1d4ee0 0x55b76f1d7300 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=111ac1b022d96012 server_cookie=1701600f5a8e0c3b in_seq=0 out_seq=0 2024-03-18T22:48:37.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f353bfff640 1 -- 172.21.15.40:0/273531357 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b76f0c99f0 con 0x55b76f1d4ee0 2024-03-18T22:48:37.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f353bfff640 10 client.26929 handle_command_reply: tid=0 2024-03-18T22:48:37.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f3565bb8740 1 client.26929 shutdown 2024-03-18T22:48:37.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f3565bb8740 1 -- 172.21.15.40:0/273531357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f1582f0 msgr2=0x55b76f167360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f3565bb8740 1 --2- 172.21.15.40:0/273531357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f1582f0 0x55b76f167360 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3550098320 tx=0x7f3550097d20 comp rx=0 tx=0).stop 2024-03-18T22:48:37.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f3565bb8740 1 -- 172.21.15.40:0/273531357 shutdown_connections 2024-03-18T22:48:37.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f3565bb8740 1 --2- 172.21.15.40:0/273531357 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55b76f1d4ee0 0x55b76f1d7300 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f3565bb8740 1 --2- 172.21.15.40:0/273531357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b76f15de50 0x55b76f168860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f3565bb8740 1 --2- 172.21.15.40:0/273531357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b76f158ce0 0x55b76f1678a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f353bfff640 0 client.26929 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:37.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f3565bb8740 1 --2- 172.21.15.40:0/273531357 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55b76f1582f0 0x55b76f167360 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f3565bb8740 1 -- 172.21.15.40:0/273531357 >> 172.21.15.40:0/273531357 conn(0x55b76f153860 msgr2=0x55b76f162720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f3565bb8740 1 -- 172.21.15.40:0/273531357 shutdown_connections 2024-03-18T22:48:37.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f3565bb8740 1 -- 172.21.15.40:0/273531357 wait complete. 2024-03-18T22:48:37.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.387+0000 7f3565bb8740 20 client.26929 trim_cache size 0 max 16384 2024-03-18T22:48:37.394 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:48:37.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.388+0000 7f3562fcd640 1 -- 172.21.15.40:0/2298576462 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f354c0a2800 msgr2=0x7f354c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.388+0000 7f3562fcd640 1 --2- 172.21.15.40:0/2298576462 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f354c0a2800 0x7f354c0a4cc0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f355c198ee0 tx=0x7f3548005f90 comp rx=0 tx=0).stop 2024-03-18T22:48:37.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.388+0000 7f3562fcd640 1 -- 172.21.15.40:0/2298576462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f355c101820 msgr2=0x7f355c1979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:37.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.388+0000 7f3562fcd640 1 --2- 172.21.15.40:0/2298576462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f355c101820 0x7f355c1979d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f355000e9c0 tx=0x7f355000ee80 comp rx=0 tx=0).stop 2024-03-18T22:48:37.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.388+0000 7f3562fcd640 1 -- 172.21.15.40:0/2298576462 shutdown_connections 2024-03-18T22:48:37.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.388+0000 7f3562fcd640 1 --2- 172.21.15.40:0/2298576462 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f354c0a2800 0x7f354c0a4cc0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.388+0000 7f3562fcd640 1 --2- 172.21.15.40:0/2298576462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355c1033d0 0x7f355c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.388+0000 7f3562fcd640 1 --2- 172.21.15.40:0/2298576462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f355c102a10 0x7f355c197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.388+0000 7f3562fcd640 1 --2- 172.21.15.40:0/2298576462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f355c101820 0x7f355c1979d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:37.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.388+0000 7f3562fcd640 1 -- 172.21.15.40:0/2298576462 >> 172.21.15.40:0/2298576462 conn(0x7f355c0fcd90 msgr2=0x7f355c105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:37.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.388+0000 7f3562fcd640 1 -- 172.21.15.40:0/2298576462 shutdown_connections 2024-03-18T22:48:37.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:37.388+0000 7f3562fcd640 1 -- 172.21.15.40:0/2298576462 wait complete. 2024-03-18T22:48:37.397 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:37 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3739415895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:48:37.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:37 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3739415895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:48:37.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:37 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3739415895' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:48:38.400 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:48:38.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:48:38.401 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:48:38.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.515+0000 7f804ea9a640 1 -- 172.21.15.40:0/535850263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8048101910 msgr2=0x7f8048101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.515+0000 7f804ea9a640 1 --2- 172.21.15.40:0/535850263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8048101910 0x7f8048101d90 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8034009900 tx=0x7f803402bc30 comp rx=0 tx=0).stop 2024-03-18T22:48:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.515+0000 7f804ea9a640 1 -- 172.21.15.40:0/535850263 shutdown_connections 2024-03-18T22:48:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.515+0000 7f804ea9a640 1 --2- 172.21.15.40:0/535850263 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f80481022d0 0x7f8048108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.515+0000 7f804ea9a640 1 --2- 172.21.15.40:0/535850263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8048101910 0x7f8048101d90 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.515+0000 7f804ea9a640 1 --2- 172.21.15.40:0/535850263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8048100720 0x7f8048100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.515+0000 7f804ea9a640 1 -- 172.21.15.40:0/535850263 >> 172.21.15.40:0/535850263 conn(0x7f80480fbcb0 msgr2=0x7f80480fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.515+0000 7f804ea9a640 1 -- 172.21.15.40:0/535850263 shutdown_connections 2024-03-18T22:48:38.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.515+0000 7f804ea9a640 1 -- 172.21.15.40:0/535850263 wait complete. 2024-03-18T22:48:38.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804ea9a640 1 Processor -- start 2024-03-18T22:48:38.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804ea9a640 1 -- start start 2024-03-18T22:48:38.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804ea9a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8048100720 0x7f8048197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804ea9a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8048101910 0x7f8048198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804ea9a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80481022d0 0x7f804819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804ea9a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f804810c250 con 0x7f80481022d0 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804ea9a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f804810c0d0 con 0x7f8048101910 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804ea9a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f804810c3d0 con 0x7f8048100720 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804c836640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8048100720 0x7f8048197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804d037640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80481022d0 0x7f804819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804c836640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8048100720 0x7f8048197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38970/0 (socket says 172.21.15.40:38970) 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804c836640 1 -- 172.21.15.40:0/3864480804 learned_addr learned my addr 172.21.15.40:0/3864480804 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804d037640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80481022d0 0x7f804819f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45894/0 (socket says 172.21.15.40:45894) 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f8047fff640 1 --2- 172.21.15.40:0/3864480804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8048101910 0x7f8048198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804c836640 1 -- 172.21.15.40:0/3864480804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8048101910 msgr2=0x7f8048198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804c836640 1 --2- 172.21.15.40:0/3864480804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8048101910 0x7f8048198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804c836640 1 -- 172.21.15.40:0/3864480804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80481022d0 msgr2=0x7f804819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804c836640 1 --2- 172.21.15.40:0/3864480804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80481022d0 0x7f804819f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804c836640 1 -- 172.21.15.40:0/3864480804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f804819f6f0 con 0x7f8048100720 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.516+0000 7f804d037640 1 --2- 172.21.15.40:0/3864480804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80481022d0 0x7f804819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.517+0000 7f804c836640 1 --2- 172.21.15.40:0/3864480804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8048100720 0x7f8048197ad0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f803c00e9c0 tx=0x7f803c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:38.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.517+0000 7f8045ffb640 1 -- 172.21.15.40:0/3864480804 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803c00cd80 con 0x7f8048100720 2024-03-18T22:48:38.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.517+0000 7f804ea9a640 1 -- 172.21.15.40:0/3864480804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f804819f9e0 con 0x7f8048100720 2024-03-18T22:48:38.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.517+0000 7f8045ffb640 1 -- 172.21.15.40:0/3864480804 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f803c0035f0 con 0x7f8048100720 2024-03-18T22:48:38.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.517+0000 7f8045ffb640 1 -- 172.21.15.40:0/3864480804 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803c010690 con 0x7f8048100720 2024-03-18T22:48:38.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.517+0000 7f804ea9a640 1 -- 172.21.15.40:0/3864480804 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f804819ff00 con 0x7f8048100720 2024-03-18T22:48:38.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.519+0000 7f8045ffb640 1 -- 172.21.15.40:0/3864480804 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f803c0037a0 con 0x7f8048100720 2024-03-18T22:48:38.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.519+0000 7f8051764740 1 Processor -- start 2024-03-18T22:48:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f8051764740 1 -- start start 2024-03-18T22:48:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f8051764740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee587c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f8051764740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee58d90 0x564cbee5d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f8051764740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee5df00 0x564cbee60390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f8051764740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564cbedbd160 con 0x564cbee583a0 2024-03-18T22:48:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f8051764740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564cbed65e10 con 0x564cbee5df00 2024-03-18T22:48:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f8051764740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x564cbedb9200 con 0x564cbee58d90 2024-03-18T22:48:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f804d037640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f804c836640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee58d90 0x564cbee5d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f8047fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee5df00 0x564cbee60390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f804c836640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee58d90 0x564cbee5d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38980/0 (socket says 172.21.15.40:38980) 2024-03-18T22:48:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f804c836640 1 -- 172.21.15.40:0/3236272975 learned_addr learned my addr 172.21.15.40:0/3236272975 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f804d037640 1 --2- 172.21.15.40:0/3236272975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee587c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45898/0 (socket says 172.21.15.40:45898) 2024-03-18T22:48:38.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f8045ffb640 1 --2- 172.21.15.40:0/3864480804 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f80380a2800 0x7f80380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f804c836640 1 -- 172.21.15.40:0/3236272975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee5df00 msgr2=0x564cbee60390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f804c836640 1 --2- 172.21.15.40:0/3236272975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee5df00 0x564cbee60390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f804c836640 1 -- 172.21.15.40:0/3236272975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 msgr2=0x564cbee587c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f804c836640 1 --2- 172.21.15.40:0/3236272975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee587c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f804c836640 1 -- 172.21.15.40:0/3236272975 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cbeb997e0 con 0x564cbee58d90 2024-03-18T22:48:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f804d037640 1 --2- 172.21.15.40:0/3864480804 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f80380a2800 0x7f80380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.520+0000 7f8045ffb640 1 -- 172.21.15.40:0/3864480804 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f803c0cf350 con 0x7f8048100720 2024-03-18T22:48:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.521+0000 7f804c836640 1 --2- 172.21.15.40:0/3236272975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee58d90 0x564cbee5d890 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f803c056d20 tx=0x7f803c022000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=703c1ac1b3127bcd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.521+0000 7f8032ffd640 1 -- 172.21.15.40:0/3236272975 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803c09b6d0 con 0x564cbee58d90 2024-03-18T22:48:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.521+0000 7f804d037640 1 --2- 172.21.15.40:0/3864480804 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f80380a2800 0x7f80380a4cc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x564cbee595f0 tx=0x7f8040004ab0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.521+0000 7f8032ffd640 1 -- 172.21.15.40:0/3236272975 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f803c003b10 con 0x564cbee58d90 2024-03-18T22:48:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.521+0000 7f8032ffd640 1 -- 172.21.15.40:0/3236272975 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803c053a40 con 0x564cbee58d90 2024-03-18T22:48:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.521+0000 7f8051764740 1 -- 172.21.15.40:0/3236272975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee58d90 msgr2=0x564cbee5d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.521+0000 7f8051764740 1 --2- 172.21.15.40:0/3236272975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee58d90 0x564cbee5d890 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f803c056d20 tx=0x7f803c022000 comp rx=0 tx=0).stop 2024-03-18T22:48:38.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.522+0000 7f8051764740 1 -- 172.21.15.40:0/3236272975 shutdown_connections 2024-03-18T22:48:38.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.522+0000 7f8051764740 1 --2- 172.21.15.40:0/3236272975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee5df00 0x564cbee60390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.522+0000 7f8051764740 1 --2- 172.21.15.40:0/3236272975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee58d90 0x564cbee5d890 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.522+0000 7f8051764740 1 --2- 172.21.15.40:0/3236272975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee587c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.522+0000 7f8051764740 1 -- 172.21.15.40:0/3236272975 >> 172.21.15.40:0/3236272975 conn(0x564cbee53910 msgr2=0x564cbee53d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:38.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.522+0000 7f8051764740 1 -- 172.21.15.40:0/3236272975 shutdown_connections 2024-03-18T22:48:38.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.522+0000 7f8051764740 1 -- 172.21.15.40:0/3236272975 wait complete. 2024-03-18T22:48:38.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.522+0000 7f8051764740 1 -- start start 2024-03-18T22:48:38.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.522+0000 7f8051764740 1 Processor -- start 2024-03-18T22:48:38.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.522+0000 7f8051764740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee626a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.522+0000 7f8051764740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee58d90 0x564cbee62be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.522+0000 7f8051764740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee5df00 0x564cbee69c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.522+0000 7f8051764740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x564cbee60e50 con 0x564cbee583a0 2024-03-18T22:48:38.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.522+0000 7f8051764740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564cbed791a0 con 0x564cbee58d90 2024-03-18T22:48:38.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.522+0000 7f8051764740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x564cbee60fd0 con 0x564cbee5df00 2024-03-18T22:48:38.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f8051764740 10 client.0 _ll_register_callbacks cb 0x564cbead0c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:38.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f804c836640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee626a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f804c836640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee626a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45906/0 (socket says 172.21.15.40:45906) 2024-03-18T22:48:38.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f804c836640 1 -- 172.21.15.40:0/2021020783 learned_addr learned my addr 172.21.15.40:0/2021020783 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:38.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f8047fff640 1 --2- 172.21.15.40:0/2021020783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee58d90 0x564cbee62be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f804d037640 1 --2- 172.21.15.40:0/2021020783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee5df00 0x564cbee69c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f804c836640 1 -- 172.21.15.40:0/2021020783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee5df00 msgr2=0x564cbee69c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f804c836640 1 --2- 172.21.15.40:0/2021020783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee5df00 0x564cbee69c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f804c836640 1 -- 172.21.15.40:0/2021020783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee58d90 msgr2=0x564cbee62be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f804c836640 1 --2- 172.21.15.40:0/2021020783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee58d90 0x564cbee62be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f804c836640 1 -- 172.21.15.40:0/2021020783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cbebf9f70 con 0x564cbee583a0 2024-03-18T22:48:38.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f8047fff640 1 --2- 172.21.15.40:0/2021020783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee58d90 0x564cbee62be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:38.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f804c836640 1 --2- 172.21.15.40:0/2021020783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee626a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f803c0058c0 tx=0x7f803c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce18a8b8b17a92b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:38.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f8027fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:48:38.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f8027fff640 1 -- 172.21.15.40:0/2021020783 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803c09b260 con 0x564cbee583a0 2024-03-18T22:48:38.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f8027fff640 1 -- 172.21.15.40:0/2021020783 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f803c099c00 con 0x564cbee583a0 2024-03-18T22:48:38.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f8051764740 1 -- 172.21.15.40:0/2021020783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564cbedca6a0 con 0x564cbee583a0 2024-03-18T22:48:38.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f8027fff640 1 -- 172.21.15.40:0/2021020783 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f803c09b440 con 0x564cbee583a0 2024-03-18T22:48:38.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f8051764740 10 client.17130 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:38.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.523+0000 7f8051764740 1 -- 172.21.15.40:0/2021020783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564cbea91170 con 0x564cbee583a0 2024-03-18T22:48:38.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.524+0000 7f8027fff640 1 -- 172.21.15.40:0/2021020783 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x564cbedca6a0 con 0x564cbee583a0 2024-03-18T22:48:38.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.524+0000 7f8027fff640 1 -- 172.21.15.40:0/2021020783 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f803c0afce0 con 0x564cbee583a0 2024-03-18T22:48:38.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.524+0000 7f8051764740 10 client.17130 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:38.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.524+0000 7f8051764740 10 client.17130 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:38.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.524+0000 7f8051764740 20 client.17130 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:38.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.524+0000 7f8051764740 1 --2- 172.21.15.40:0/2021020783 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x564cbee6e110 0x564cbee70500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.524+0000 7f8051764740 4 client.17130 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:38.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.524+0000 7f8051764740 1 -- 172.21.15.40:0/2021020783 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cbedc49f0 con 0x564cbee6e110 2024-03-18T22:48:38.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.524+0000 7f8047fff640 1 --2- 172.21.15.40:0/2021020783 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x564cbee6e110 0x564cbee70500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.525+0000 7f8047fff640 1 --2- 172.21.15.40:0/2021020783 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x564cbee6e110 0x564cbee70500 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df92b923b52a663a server_cookie=298738a333127603 in_seq=0 out_seq=0 2024-03-18T22:48:38.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.525+0000 7f8027fff640 10 client.17130 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:38.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.525+0000 7f8027fff640 1 -- 172.21.15.40:0/2021020783 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564cbedc49f0 con 0x564cbee6e110 2024-03-18T22:48:38.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.525+0000 7f8027fff640 10 client.17130 handle_command_reply: tid=0 2024-03-18T22:48:38.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.525+0000 7f8051764740 1 client.17130 shutdown 2024-03-18T22:48:38.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.525+0000 7f8051764740 1 -- 172.21.15.40:0/2021020783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 msgr2=0x564cbee626a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.525+0000 7f8051764740 1 --2- 172.21.15.40:0/2021020783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee626a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f803c0058c0 tx=0x7f803c005740 comp rx=0 tx=0).stop 2024-03-18T22:48:38.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.525+0000 7f8051764740 1 -- 172.21.15.40:0/2021020783 shutdown_connections 2024-03-18T22:48:38.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.525+0000 7f8051764740 1 --2- 172.21.15.40:0/2021020783 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee5df00 0x564cbee69c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.525+0000 7f8051764740 1 --2- 172.21.15.40:0/2021020783 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x564cbee6e110 0x564cbee70500 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.525+0000 7f8051764740 1 --2- 172.21.15.40:0/2021020783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee58d90 0x564cbee62be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.525+0000 7f8051764740 1 --2- 172.21.15.40:0/2021020783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee626a0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.525+0000 7f8051764740 1 -- 172.21.15.40:0/2021020783 >> 172.21.15.40:0/2021020783 conn(0x564cbee53910 msgr2=0x564cbee52b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:38.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.525+0000 7f8027fff640 0 client.17130 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:38.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.526+0000 7f8051764740 1 -- 172.21.15.40:0/2021020783 shutdown_connections 2024-03-18T22:48:38.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.526+0000 7f8051764740 1 -- 172.21.15.40:0/2021020783 wait complete. 2024-03-18T22:48:38.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.526+0000 7f8051764740 20 client.17130 trim_cache size 0 max 16384 2024-03-18T22:48:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.554+0000 7f8051764740 1 Processor -- start 2024-03-18T22:48:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.554+0000 7f8051764740 1 -- start start 2024-03-18T22:48:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.554+0000 7f8051764740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.554+0000 7f8051764740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee58d90 0x564cbee69be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.554+0000 7f8051764740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee5df00 0x564cbee522a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.554+0000 7f8051764740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f80481aa560 con 0x564cbee583a0 2024-03-18T22:48:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.554+0000 7f8051764740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f80481ac0a0 con 0x564cbee5df00 2024-03-18T22:48:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.554+0000 7f8051764740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f80481aac30 con 0x564cbee58d90 2024-03-18T22:48:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.554+0000 7f804c836640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.554+0000 7f804d037640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee5df00 0x564cbee522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.554+0000 7f8047fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee58d90 0x564cbee69be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.554+0000 7f804d037640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee5df00 0x564cbee522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49850/0 (socket says 172.21.15.40:49850) 2024-03-18T22:48:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.554+0000 7f804d037640 1 -- 172.21.15.40:0/968621479 learned_addr learned my addr 172.21.15.40:0/968621479 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.554+0000 7f804c836640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee629c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45920/0 (socket says 172.21.15.40:45920) 2024-03-18T22:48:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.555+0000 7f804d037640 1 -- 172.21.15.40:0/968621479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee58d90 msgr2=0x564cbee69be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.555+0000 7f804d037640 1 --2- 172.21.15.40:0/968621479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee58d90 0x564cbee69be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.555+0000 7f804d037640 1 -- 172.21.15.40:0/968621479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 msgr2=0x564cbee629c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.555+0000 7f804d037640 1 --2- 172.21.15.40:0/968621479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee629c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.555+0000 7f804d037640 1 -- 172.21.15.40:0/968621479 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cbee534b0 con 0x564cbee5df00 2024-03-18T22:48:38.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.555+0000 7f804c836640 1 --2- 172.21.15.40:0/968621479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee629c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:38.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.555+0000 7f804d037640 1 --2- 172.21.15.40:0/968621479 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee5df00 0x564cbee522a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8040003ec0 tx=0x7f8040011490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7be35ac32763727 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:38.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.555+0000 7f80327fc640 1 -- 172.21.15.40:0/968621479 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804001a070 con 0x564cbee5df00 2024-03-18T22:48:38.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.555+0000 7f80327fc640 1 -- 172.21.15.40:0/968621479 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8040022a00 con 0x564cbee5df00 2024-03-18T22:48:38.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.555+0000 7f80327fc640 1 -- 172.21.15.40:0/968621479 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8040005640 con 0x564cbee5df00 2024-03-18T22:48:38.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.556+0000 7f8051764740 1 -- 172.21.15.40:0/968621479 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee5df00 msgr2=0x564cbee522a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.556+0000 7f8051764740 1 --2- 172.21.15.40:0/968621479 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee5df00 0x564cbee522a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8040003ec0 tx=0x7f8040011490 comp rx=0 tx=0).stop 2024-03-18T22:48:38.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.556+0000 7f8051764740 1 -- 172.21.15.40:0/968621479 shutdown_connections 2024-03-18T22:48:38.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.556+0000 7f8051764740 1 --2- 172.21.15.40:0/968621479 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee5df00 0x564cbee522a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.556+0000 7f8051764740 1 --2- 172.21.15.40:0/968621479 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee58d90 0x564cbee69be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.556+0000 7f8051764740 1 --2- 172.21.15.40:0/968621479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee583a0 0x564cbee629c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.556+0000 7f8051764740 1 -- 172.21.15.40:0/968621479 >> 172.21.15.40:0/968621479 conn(0x564cbee53910 msgr2=0x564cbee622b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:38.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.556+0000 7f8051764740 1 -- 172.21.15.40:0/968621479 shutdown_connections 2024-03-18T22:48:38.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.556+0000 7f8051764740 1 -- 172.21.15.40:0/968621479 wait complete. 2024-03-18T22:48:38.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.556+0000 7f8051764740 1 -- start start 2024-03-18T22:48:38.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.556+0000 7f8051764740 1 Processor -- start 2024-03-18T22:48:38.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8051764740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee583a0 0x564cbee67480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8051764740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee58d90 0x564cbee679c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8051764740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee5df00 0x564cbee689d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8051764740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f80481ab9d0 con 0x564cbee58d90 2024-03-18T22:48:38.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8051764740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f80481ab300 con 0x564cbee583a0 2024-03-18T22:48:38.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8051764740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f80481a8540 con 0x564cbee5df00 2024-03-18T22:48:38.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8051764740 10 client.0 _ll_register_callbacks cb 0x7f8040009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:38.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8047fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee58d90 0x564cbee679c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8047fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee58d90 0x564cbee679c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45930/0 (socket says 172.21.15.40:45930) 2024-03-18T22:48:38.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8047fff640 1 -- 172.21.15.40:0/362433800 learned_addr learned my addr 172.21.15.40:0/362433800 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:38.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f804c836640 1 --2- 172.21.15.40:0/362433800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee583a0 0x564cbee67480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f804d037640 1 --2- 172.21.15.40:0/362433800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee5df00 0x564cbee689d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8047fff640 1 -- 172.21.15.40:0/362433800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee5df00 msgr2=0x564cbee689d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8047fff640 1 --2- 172.21.15.40:0/362433800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee5df00 0x564cbee689d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8047fff640 1 -- 172.21.15.40:0/362433800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee583a0 msgr2=0x564cbee67480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8047fff640 1 --2- 172.21.15.40:0/362433800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee583a0 0x564cbee67480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8047fff640 1 -- 172.21.15.40:0/362433800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cbee60d90 con 0x564cbee58d90 2024-03-18T22:48:38.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8047fff640 1 --2- 172.21.15.40:0/362433800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee58d90 0x564cbee679c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f803402c170 tx=0x7f8034004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3551eaf854ebcf53 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.557+0000 7f8027fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:48:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.558+0000 7f8027fff640 1 -- 172.21.15.40:0/362433800 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803404f020 con 0x564cbee58d90 2024-03-18T22:48:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.558+0000 7f8027fff640 1 -- 172.21.15.40:0/362433800 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f803404bab0 con 0x564cbee58d90 2024-03-18T22:48:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.558+0000 7f8051764740 1 -- 172.21.15.40:0/362433800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8040022a00 con 0x564cbee58d90 2024-03-18T22:48:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.558+0000 7f8027fff640 1 -- 172.21.15.40:0/362433800 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f803405a050 con 0x564cbee58d90 2024-03-18T22:48:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.558+0000 7f8051764740 10 client.17136 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.558+0000 7f8051764740 1 -- 172.21.15.40:0/362433800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564cbeed4310 con 0x564cbee58d90 2024-03-18T22:48:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.558+0000 7f8027fff640 1 -- 172.21.15.40:0/362433800 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8040022a00 con 0x564cbee58d90 2024-03-18T22:48:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.558+0000 7f8027fff640 1 -- 172.21.15.40:0/362433800 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f80340052a0 con 0x564cbee58d90 2024-03-18T22:48:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.558+0000 7f8051764740 10 client.17136 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.558+0000 7f8051764740 10 client.17136 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.558+0000 7f8051764740 20 client.17136 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.558+0000 7f8051764740 1 --2- 172.21.15.40:0/362433800 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x564cbeed4d10 0x564cbeed7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.558+0000 7f8051764740 4 client.17136 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:48:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.558+0000 7f8051764740 1 -- 172.21.15.40:0/362433800 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564cbed42be0 con 0x564cbeed4d10 2024-03-18T22:48:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.558+0000 7f804c836640 1 --2- 172.21.15.40:0/362433800 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x564cbeed4d10 0x564cbeed7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.559+0000 7f804c836640 1 --2- 172.21.15.40:0/362433800 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x564cbeed4d10 0x564cbeed7100 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=184f94613fef2346 server_cookie=aa4a34e1328527e7 in_seq=0 out_seq=0 2024-03-18T22:48:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.559+0000 7f8027fff640 10 client.17136 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.560+0000 7f8027fff640 1 -- 172.21.15.40:0/362433800 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564cbed42be0 con 0x564cbeed4d10 2024-03-18T22:48:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.560+0000 7f8027fff640 10 client.17136 handle_command_reply: tid=0 2024-03-18T22:48:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.560+0000 7f8051764740 1 client.17136 shutdown 2024-03-18T22:48:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.560+0000 7f8051764740 1 -- 172.21.15.40:0/362433800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee58d90 msgr2=0x564cbee679c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.560+0000 7f8051764740 1 --2- 172.21.15.40:0/362433800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee58d90 0x564cbee679c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f803402c170 tx=0x7f8034004270 comp rx=0 tx=0).stop 2024-03-18T22:48:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.560+0000 7f8051764740 1 -- 172.21.15.40:0/362433800 shutdown_connections 2024-03-18T22:48:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.560+0000 7f8051764740 1 --2- 172.21.15.40:0/362433800 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564cbee5df00 0x564cbee689d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.560+0000 7f8051764740 1 --2- 172.21.15.40:0/362433800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564cbee58d90 0x564cbee679c0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.560+0000 7f8051764740 1 --2- 172.21.15.40:0/362433800 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x564cbeed4d10 0x564cbeed7100 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.560+0000 7f8051764740 1 --2- 172.21.15.40:0/362433800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564cbee583a0 0x564cbee67480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.560+0000 7f8051764740 1 -- 172.21.15.40:0/362433800 >> 172.21.15.40:0/362433800 conn(0x564cbee53910 msgr2=0x564cbee56b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:38.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.560+0000 7f8027fff640 0 client.17136 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:38.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.560+0000 7f8051764740 1 -- 172.21.15.40:0/362433800 shutdown_connections 2024-03-18T22:48:38.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.560+0000 7f8051764740 1 -- 172.21.15.40:0/362433800 wait complete. 2024-03-18T22:48:38.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.560+0000 7f8051764740 20 client.17136 trim_cache size 0 max 16384 2024-03-18T22:48:38.566 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:48:38.566 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:48:38.566 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "bef703c0-7028-41e6-bea9-8456fefb3d7e", 2024-03-18T22:48:38.566 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:48:38.566 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:48:38.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.561+0000 7f804ea9a640 1 -- 172.21.15.40:0/3864480804 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f80380a2800 msgr2=0x7f80380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.561+0000 7f804ea9a640 1 --2- 172.21.15.40:0/3864480804 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f80380a2800 0x7f80380a4cc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x564cbee595f0 tx=0x7f8040004ab0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.561+0000 7f804ea9a640 1 -- 172.21.15.40:0/3864480804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8048100720 msgr2=0x7f8048197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.561+0000 7f804ea9a640 1 --2- 172.21.15.40:0/3864480804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8048100720 0x7f8048197ad0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f803c00e9c0 tx=0x7f803c00ee80 comp rx=0 tx=0).stop 2024-03-18T22:48:38.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.561+0000 7f804ea9a640 1 -- 172.21.15.40:0/3864480804 shutdown_connections 2024-03-18T22:48:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.561+0000 7f804ea9a640 1 --2- 172.21.15.40:0/3864480804 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f80380a2800 0x7f80380a4cc0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.561+0000 7f804ea9a640 1 --2- 172.21.15.40:0/3864480804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f80481022d0 0x7f804819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.561+0000 7f804ea9a640 1 --2- 172.21.15.40:0/3864480804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8048101910 0x7f8048198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.561+0000 7f804ea9a640 1 --2- 172.21.15.40:0/3864480804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8048100720 0x7f8048197ad0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.561+0000 7f804ea9a640 1 -- 172.21.15.40:0/3864480804 >> 172.21.15.40:0/3864480804 conn(0x7f80480fbcb0 msgr2=0x7f80480689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.561+0000 7f804ea9a640 1 -- 172.21.15.40:0/3864480804 shutdown_connections 2024-03-18T22:48:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.561+0000 7f804ea9a640 1 -- 172.21.15.40:0/3864480804 wait complete. 2024-03-18T22:48:38.573 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:38 smithi040 ceph-mon[62643]: pgmap v530: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:48:38.574 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:48:38.574 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:48:38.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:38 smithi113 ceph-mon[66775]: pgmap v530: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:48:38.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:38 smithi103 ceph-mon[66377]: pgmap v530: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:48:38.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.697+0000 7fce5fdb9640 1 -- 172.21.15.40:0/2749306036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce581007c0 msgr2=0x7fce58100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.697+0000 7fce5fdb9640 1 --2- 172.21.15.40:0/2749306036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce581007c0 0x7fce58100be0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fce54009900 tx=0x7fce5402bc30 comp rx=0 tx=0).stop 2024-03-18T22:48:38.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.697+0000 7fce5fdb9640 1 -- 172.21.15.40:0/2749306036 shutdown_connections 2024-03-18T22:48:38.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.697+0000 7fce5fdb9640 1 --2- 172.21.15.40:0/2749306036 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce58102370 0x7fce58108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.697+0000 7fce5fdb9640 1 --2- 172.21.15.40:0/2749306036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce581019b0 0x7fce58101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.697+0000 7fce5fdb9640 1 --2- 172.21.15.40:0/2749306036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce581007c0 0x7fce58100be0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.697+0000 7fce5fdb9640 1 -- 172.21.15.40:0/2749306036 >> 172.21.15.40:0/2749306036 conn(0x7fce580fbd30 msgr2=0x7fce580fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:38.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.697+0000 7fce5fdb9640 1 -- 172.21.15.40:0/2749306036 shutdown_connections 2024-03-18T22:48:38.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.697+0000 7fce5fdb9640 1 -- 172.21.15.40:0/2749306036 wait complete. 2024-03-18T22:48:38.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5fdb9640 1 Processor -- start 2024-03-18T22:48:38.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5fdb9640 1 -- start start 2024-03-18T22:48:38.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5fdb9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce581007c0 0x7fce5806a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5fdb9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce581019b0 0x7fce5819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5fdb9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce58102370 0x7fce5819eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5fdb9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fce5810c2f0 con 0x7fce581007c0 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5fdb9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fce5810c170 con 0x7fce581019b0 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5fdb9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fce5810c470 con 0x7fce58102370 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5db55640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce581007c0 0x7fce5806a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5db55640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce581007c0 0x7fce5806a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38460/0 (socket says 172.21.15.40:38460) 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5db55640 1 -- 172.21.15.40:0/304041885 learned_addr learned my addr 172.21.15.40:0/304041885 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5d354640 1 --2- 172.21.15.40:0/304041885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce581019b0 0x7fce5819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5e356640 1 --2- 172.21.15.40:0/304041885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce58102370 0x7fce5819eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5db55640 1 -- 172.21.15.40:0/304041885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce58102370 msgr2=0x7fce5819eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5db55640 1 --2- 172.21.15.40:0/304041885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce58102370 0x7fce5819eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5db55640 1 -- 172.21.15.40:0/304041885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce581019b0 msgr2=0x7fce5819c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5db55640 1 --2- 172.21.15.40:0/304041885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce581019b0 0x7fce5819c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.698+0000 7fce5db55640 1 -- 172.21.15.40:0/304041885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5819f770 con 0x7fce581007c0 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.699+0000 7fce5db55640 1 --2- 172.21.15.40:0/304041885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce581007c0 0x7fce5806a9e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fce54000c00 tx=0x7fce54005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:38.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.699+0000 7fce4effd640 1 -- 172.21.15.40:0/304041885 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54043020 con 0x7fce581007c0 2024-03-18T22:48:38.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.699+0000 7fce4effd640 1 -- 172.21.15.40:0/304041885 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce54004350 con 0x7fce581007c0 2024-03-18T22:48:38.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.699+0000 7fce4effd640 1 -- 172.21.15.40:0/304041885 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54003b80 con 0x7fce581007c0 2024-03-18T22:48:38.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.699+0000 7fce5fdb9640 1 -- 172.21.15.40:0/304041885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce5819fa00 con 0x7fce581007c0 2024-03-18T22:48:38.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.699+0000 7fce5fdb9640 1 -- 172.21.15.40:0/304041885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce5819ffa0 con 0x7fce581007c0 2024-03-18T22:48:38.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.702+0000 7fce4effd640 1 -- 172.21.15.40:0/304041885 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fce54046020 con 0x7fce581007c0 2024-03-18T22:48:38.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.702+0000 7fce629a5740 1 Processor -- start 2024-03-18T22:48:38.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.702+0000 7fce629a5740 1 -- start start 2024-03-18T22:48:38.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.702+0000 7fce629a5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e1a50 0x556d585e1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.702+0000 7fce629a5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e2440 0x556d585e6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.702+0000 7fce629a5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e75b0 0x556d585e9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.702+0000 7fce629a5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556d58546500 con 0x556d585e1a50 2024-03-18T22:48:38.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.702+0000 7fce629a5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556d584eebe0 con 0x556d585e2440 2024-03-18T22:48:38.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.702+0000 7fce629a5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556d585425a0 con 0x556d585e75b0 2024-03-18T22:48:38.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce4effd640 1 --2- 172.21.15.40:0/304041885 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fce440a6c90 0x7fce440a9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce5db55640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e2440 0x556d585e6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce5d354640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e75b0 0x556d585e9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce5e356640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e1a50 0x556d585e1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce5d354640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e75b0 0x556d585e9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:34846/0 (socket says 172.21.15.40:34846) 2024-03-18T22:48:38.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce5e356640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e1a50 0x556d585e1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38462/0 (socket says 172.21.15.40:38462) 2024-03-18T22:48:38.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce5d354640 1 -- 172.21.15.40:0/1211416535 learned_addr learned my addr 172.21.15.40:0/1211416535 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:38.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce5e356640 1 --2- 172.21.15.40:0/304041885 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fce440a6c90 0x7fce440a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce5d354640 1 -- 172.21.15.40:0/1211416535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e2440 msgr2=0x556d585e6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce5d354640 1 --2- 172.21.15.40:0/1211416535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e2440 0x556d585e6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce5d354640 1 -- 172.21.15.40:0/1211416535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e1a50 msgr2=0x556d585e1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce5d354640 1 --2- 172.21.15.40:0/1211416535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e1a50 0x556d585e1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce5d354640 1 -- 172.21.15.40:0/1211416535 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d58510000 con 0x556d585e75b0 2024-03-18T22:48:38.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce4effd640 1 -- 172.21.15.40:0/304041885 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fce540f0f10 con 0x7fce581007c0 2024-03-18T22:48:38.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce5d354640 1 --2- 172.21.15.40:0/1211416535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e75b0 0x556d585e9a40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fce48009c90 tx=0x7fce48011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48fbca05712a878d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:38.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce5e356640 1 --2- 172.21.15.40:0/304041885 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fce440a6c90 0x7fce440a9150 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x556d585e2ca0 tx=0x7fce50009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:38.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.703+0000 7fce43fff640 1 -- 172.21.15.40:0/1211416535 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4801e070 con 0x556d585e75b0 2024-03-18T22:48:38.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.704+0000 7fce43fff640 1 -- 172.21.15.40:0/1211416535 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce48009e40 con 0x556d585e75b0 2024-03-18T22:48:38.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.704+0000 7fce43fff640 1 -- 172.21.15.40:0/1211416535 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4800e960 con 0x556d585e75b0 2024-03-18T22:48:38.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.704+0000 7fce629a5740 1 -- 172.21.15.40:0/1211416535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e75b0 msgr2=0x556d585e9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.704+0000 7fce629a5740 1 --2- 172.21.15.40:0/1211416535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e75b0 0x556d585e9a40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fce48009c90 tx=0x7fce48011810 comp rx=0 tx=0).stop 2024-03-18T22:48:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.704+0000 7fce629a5740 1 -- 172.21.15.40:0/1211416535 shutdown_connections 2024-03-18T22:48:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.704+0000 7fce629a5740 1 --2- 172.21.15.40:0/1211416535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e75b0 0x556d585e9a40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.704+0000 7fce629a5740 1 --2- 172.21.15.40:0/1211416535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e2440 0x556d585e6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.704+0000 7fce629a5740 1 --2- 172.21.15.40:0/1211416535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e1a50 0x556d585e1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.704+0000 7fce629a5740 1 -- 172.21.15.40:0/1211416535 >> 172.21.15.40:0/1211416535 conn(0x556d585dcfc0 msgr2=0x556d585dd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.705+0000 7fce629a5740 1 -- 172.21.15.40:0/1211416535 shutdown_connections 2024-03-18T22:48:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.705+0000 7fce629a5740 1 -- 172.21.15.40:0/1211416535 wait complete. 2024-03-18T22:48:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.705+0000 7fce629a5740 1 -- start start 2024-03-18T22:48:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.705+0000 7fce629a5740 1 Processor -- start 2024-03-18T22:48:38.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.705+0000 7fce629a5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e1a50 0x556d585ebed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.705+0000 7fce629a5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e2440 0x556d585ec410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.705+0000 7fce629a5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e75b0 0x556d585f3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.705+0000 7fce629a5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556d585ea560 con 0x556d585e75b0 2024-03-18T22:48:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.705+0000 7fce629a5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556d58502160 con 0x556d585e2440 2024-03-18T22:48:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.705+0000 7fce629a5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556d585ea6e0 con 0x556d585e1a50 2024-03-18T22:48:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.705+0000 7fce629a5740 10 client.0 _ll_register_callbacks cb 0x556d5853ea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce5e356640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e75b0 0x556d585f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce5d354640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e2440 0x556d585ec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce5e356640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e75b0 0x556d585f3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38478/0 (socket says 172.21.15.40:38478) 2024-03-18T22:48:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce5e356640 1 -- 172.21.15.40:0/1567351905 learned_addr learned my addr 172.21.15.40:0/1567351905 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce5db55640 1 --2- 172.21.15.40:0/1567351905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e1a50 0x556d585ebed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce5d354640 1 -- 172.21.15.40:0/1567351905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e1a50 msgr2=0x556d585ebed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce5d354640 1 --2- 172.21.15.40:0/1567351905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e1a50 0x556d585ebed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce5d354640 1 -- 172.21.15.40:0/1567351905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e75b0 msgr2=0x556d585f3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce5d354640 1 --2- 172.21.15.40:0/1567351905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e75b0 0x556d585f3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce5d354640 1 -- 172.21.15.40:0/1567351905 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d583247d0 con 0x556d585e2440 2024-03-18T22:48:38.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce5e356640 1 --2- 172.21.15.40:0/1567351905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e75b0 0x556d585f3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:38.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce5db55640 1 --2- 172.21.15.40:0/1567351905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e1a50 0x556d585ebed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:38.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce5d354640 1 --2- 172.21.15.40:0/1567351905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e2440 0x556d585ec410 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fce48011d10 tx=0x7fce48005720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ce835958e8d59cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:38.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce417fa640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:48:38.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce417fa640 1 -- 172.21.15.40:0/1567351905 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4801e050 con 0x556d585e2440 2024-03-18T22:48:38.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce629a5740 1 -- 172.21.15.40:0/1567351905 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556d58553a40 con 0x556d585e2440 2024-03-18T22:48:38.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.706+0000 7fce417fa640 1 -- 172.21.15.40:0/1567351905 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fce48004230 con 0x556d585e2440 2024-03-18T22:48:38.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.707+0000 7fce417fa640 1 -- 172.21.15.40:0/1567351905 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fce48032050 con 0x556d585e2440 2024-03-18T22:48:38.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.707+0000 7fce629a5740 10 client.26969 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:38.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.707+0000 7fce629a5740 1 -- 172.21.15.40:0/1567351905 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556d5825a3f0 con 0x556d585e2440 2024-03-18T22:48:38.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.707+0000 7fce417fa640 1 -- 172.21.15.40:0/1567351905 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x556d58553a40 con 0x556d585e2440 2024-03-18T22:48:38.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.707+0000 7fce417fa640 1 -- 172.21.15.40:0/1567351905 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fce4802d030 con 0x556d585e2440 2024-03-18T22:48:38.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.707+0000 7fce629a5740 10 client.26969 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:38.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.707+0000 7fce629a5740 10 client.26969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:38.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.707+0000 7fce629a5740 20 client.26969 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:38.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.707+0000 7fce629a5740 1 --2- 172.21.15.40:0/1567351905 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556d585f7ae0 0x556d585f9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.707+0000 7fce629a5740 4 client.26969 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:48:38.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.707+0000 7fce629a5740 1 -- 172.21.15.40:0/1567351905 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d5854dd90 con 0x556d585f7ae0 2024-03-18T22:48:38.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.707+0000 7fce5db55640 1 --2- 172.21.15.40:0/1567351905 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556d585f7ae0 0x556d585f9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.708+0000 7fce417fa640 10 client.26969 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:38.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.708+0000 7fce5db55640 1 --2- 172.21.15.40:0/1567351905 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556d585f7ae0 0x556d585f9f40 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e8497d16c04da7c server_cookie=140d1fef0322de4e in_seq=0 out_seq=0 2024-03-18T22:48:38.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.709+0000 7fce417fa640 1 -- 172.21.15.40:0/1567351905 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556d5854dd90 con 0x556d585f7ae0 2024-03-18T22:48:38.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.709+0000 7fce417fa640 10 client.26969 handle_command_reply: tid=0 2024-03-18T22:48:38.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.709+0000 7fce629a5740 1 client.26969 shutdown 2024-03-18T22:48:38.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.709+0000 7fce629a5740 1 -- 172.21.15.40:0/1567351905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e2440 msgr2=0x556d585ec410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.709+0000 7fce629a5740 1 --2- 172.21.15.40:0/1567351905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e2440 0x556d585ec410 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fce48011d10 tx=0x7fce48005720 comp rx=0 tx=0).stop 2024-03-18T22:48:38.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.709+0000 7fce629a5740 1 -- 172.21.15.40:0/1567351905 shutdown_connections 2024-03-18T22:48:38.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.709+0000 7fce629a5740 1 --2- 172.21.15.40:0/1567351905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e75b0 0x556d585f3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.709+0000 7fce629a5740 1 --2- 172.21.15.40:0/1567351905 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556d585f7ae0 0x556d585f9f40 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.709+0000 7fce629a5740 1 --2- 172.21.15.40:0/1567351905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e2440 0x556d585ec410 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.709+0000 7fce629a5740 1 --2- 172.21.15.40:0/1567351905 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e1a50 0x556d585ebed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.709+0000 7fce417fa640 0 client.26969 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:38.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.709+0000 7fce629a5740 1 -- 172.21.15.40:0/1567351905 >> 172.21.15.40:0/1567351905 conn(0x556d585dcfc0 msgr2=0x556d585dc1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:38.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.709+0000 7fce629a5740 1 -- 172.21.15.40:0/1567351905 shutdown_connections 2024-03-18T22:48:38.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.709+0000 7fce629a5740 1 -- 172.21.15.40:0/1567351905 wait complete. 2024-03-18T22:48:38.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.709+0000 7fce629a5740 20 client.26969 trim_cache size 0 max 16384 2024-03-18T22:48:38.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.736+0000 7fce629a5740 1 Processor -- start 2024-03-18T22:48:38.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.736+0000 7fce629a5740 1 -- start start 2024-03-18T22:48:38.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce629a5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e1a50 0x556d585dd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce629a5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e2440 0x556d585f3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce629a5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e75b0 0x556d585db950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce629a5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fce581a8390 con 0x556d585e1a50 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce629a5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fce581a9ed0 con 0x556d585e75b0 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce629a5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fce581a8a60 con 0x556d585e2440 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce5db55640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e1a50 0x556d585dd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce5db55640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e1a50 0x556d585dd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38482/0 (socket says 172.21.15.40:38482) 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce5e356640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e75b0 0x556d585db950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce5db55640 1 -- 172.21.15.40:0/2942216273 learned_addr learned my addr 172.21.15.40:0/2942216273 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce5d354640 1 --2- 172.21.15.40:0/2942216273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e2440 0x556d585f3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce5db55640 1 -- 172.21.15.40:0/2942216273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e2440 msgr2=0x556d585f3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce5db55640 1 --2- 172.21.15.40:0/2942216273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e2440 0x556d585f3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce5db55640 1 -- 172.21.15.40:0/2942216273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e75b0 msgr2=0x556d585db950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce5db55640 1 --2- 172.21.15.40:0/2942216273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e75b0 0x556d585db950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce5db55640 1 -- 172.21.15.40:0/2942216273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d5821a0f0 con 0x556d585e1a50 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce5d354640 1 --2- 172.21.15.40:0/2942216273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e2440 0x556d585f3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce5e356640 1 --2- 172.21.15.40:0/2942216273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e75b0 0x556d585db950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.737+0000 7fce5db55640 1 --2- 172.21.15.40:0/2942216273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e1a50 0x556d585dd3a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fce540ee670 tx=0x7fce54041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f70a556b32b6831d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:38.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.738+0000 7fce437fe640 1 -- 172.21.15.40:0/2942216273 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5408e040 con 0x556d585e1a50 2024-03-18T22:48:38.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.738+0000 7fce437fe640 1 -- 172.21.15.40:0/2942216273 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce540bdbc0 con 0x556d585e1a50 2024-03-18T22:48:38.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.738+0000 7fce437fe640 1 -- 172.21.15.40:0/2942216273 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce540c5c60 con 0x556d585e1a50 2024-03-18T22:48:38.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.738+0000 7fce629a5740 1 -- 172.21.15.40:0/2942216273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e1a50 msgr2=0x556d585dd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.738+0000 7fce629a5740 1 --2- 172.21.15.40:0/2942216273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e1a50 0x556d585dd3a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fce540ee670 tx=0x7fce54041040 comp rx=0 tx=0).stop 2024-03-18T22:48:38.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.738+0000 7fce629a5740 1 -- 172.21.15.40:0/2942216273 shutdown_connections 2024-03-18T22:48:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.738+0000 7fce629a5740 1 --2- 172.21.15.40:0/2942216273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e75b0 0x556d585db950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.738+0000 7fce629a5740 1 --2- 172.21.15.40:0/2942216273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e2440 0x556d585f3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.738+0000 7fce629a5740 1 --2- 172.21.15.40:0/2942216273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e1a50 0x556d585dd3a0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.738+0000 7fce629a5740 1 -- 172.21.15.40:0/2942216273 >> 172.21.15.40:0/2942216273 conn(0x556d585dcfc0 msgr2=0x556d585e38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce629a5740 1 -- 172.21.15.40:0/2942216273 shutdown_connections 2024-03-18T22:48:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce629a5740 1 -- 172.21.15.40:0/2942216273 wait complete. 2024-03-18T22:48:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce629a5740 1 -- start start 2024-03-18T22:48:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce629a5740 1 Processor -- start 2024-03-18T22:48:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce629a5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e1a50 0x556d585f07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce629a5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e2440 0x556d585f0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce629a5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e75b0 0x556d585f1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce629a5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fce581a9800 con 0x556d585e75b0 2024-03-18T22:48:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce629a5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fce581a9130 con 0x556d585e1a50 2024-03-18T22:48:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce629a5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fce581a6370 con 0x556d585e2440 2024-03-18T22:48:38.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce629a5740 10 client.0 _ll_register_callbacks cb 0x7fce4801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce5e356640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e75b0 0x556d585f1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce5d354640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e2440 0x556d585f0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce5e356640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e75b0 0x556d585f1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38496/0 (socket says 172.21.15.40:38496) 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce5db55640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e1a50 0x556d585f07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.739+0000 7fce5e356640 1 -- 172.21.15.40:0/3946966896 learned_addr learned my addr 172.21.15.40:0/3946966896 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.740+0000 7fce5e356640 1 -- 172.21.15.40:0/3946966896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e2440 msgr2=0x556d585f0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.740+0000 7fce5e356640 1 --2- 172.21.15.40:0/3946966896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e2440 0x556d585f0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.740+0000 7fce5e356640 1 -- 172.21.15.40:0/3946966896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e1a50 msgr2=0x556d585f07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.740+0000 7fce5e356640 1 --2- 172.21.15.40:0/3946966896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e1a50 0x556d585f07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.740+0000 7fce5e356640 1 -- 172.21.15.40:0/3946966896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d585f22f0 con 0x556d585e75b0 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.740+0000 7fce5d354640 1 --2- 172.21.15.40:0/3946966896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e2440 0x556d585f0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.740+0000 7fce5e356640 1 --2- 172.21.15.40:0/3946966896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e75b0 0x556d585f1d00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fce50011600 tx=0x7fce50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5dc6737da8a45ec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.740+0000 7fce417fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.740+0000 7fce417fa640 1 -- 172.21.15.40:0/3946966896 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50004d20 con 0x556d585e75b0 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.740+0000 7fce417fa640 1 -- 172.21.15.40:0/3946966896 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fce50020660 con 0x556d585e75b0 2024-03-18T22:48:38.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.740+0000 7fce629a5740 1 -- 172.21.15.40:0/3946966896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce540bdbc0 con 0x556d585e75b0 2024-03-18T22:48:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.740+0000 7fce417fa640 1 -- 172.21.15.40:0/3946966896 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fce50035050 con 0x556d585e75b0 2024-03-18T22:48:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.740+0000 7fce629a5740 10 client.17154 fetch_fsmap learned FSMap version 32 2024-03-18T22:48:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.740+0000 7fce629a5740 1 -- 172.21.15.40:0/3946966896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556d585f3670 con 0x556d585e75b0 2024-03-18T22:48:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.741+0000 7fce417fa640 1 -- 172.21.15.40:0/3946966896 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce540bdbc0 con 0x556d585e75b0 2024-03-18T22:48:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.741+0000 7fce417fa640 1 -- 172.21.15.40:0/3946966896 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fce5001d040 con 0x556d585e75b0 2024-03-18T22:48:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.741+0000 7fce629a5740 10 client.17154 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:48:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.741+0000 7fce629a5740 10 client.17154 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:48:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.741+0000 7fce629a5740 20 client.17154 populate_metadata read hostname 'smithi040' 2024-03-18T22:48:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.741+0000 7fce629a5740 1 --2- 172.21.15.40:0/3946966896 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556d5865e5b0 0x556d586609d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:48:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.741+0000 7fce629a5740 4 client.17154 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:48:38.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.741+0000 7fce629a5740 1 -- 172.21.15.40:0/3946966896 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556d584cb540 con 0x556d5865e5b0 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.741+0000 7fce5db55640 1 --2- 172.21.15.40:0/3946966896 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556d5865e5b0 0x556d586609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.741+0000 7fce5db55640 1 --2- 172.21.15.40:0/3946966896 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556d5865e5b0 0x556d586609d0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50bbc4c76bfda71 server_cookie=3f019ce009c253d in_seq=0 out_seq=0 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.741+0000 7fce417fa640 10 client.17154 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce417fa640 1 -- 172.21.15.40:0/3946966896 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556d584cb540 con 0x556d5865e5b0 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce417fa640 10 client.17154 handle_command_reply: tid=0 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce629a5740 1 client.17154 shutdown 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce629a5740 1 -- 172.21.15.40:0/3946966896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e75b0 msgr2=0x556d585f1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce629a5740 1 --2- 172.21.15.40:0/3946966896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e75b0 0x556d585f1d00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fce50011600 tx=0x7fce50004a40 comp rx=0 tx=0).stop 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce629a5740 1 -- 172.21.15.40:0/3946966896 shutdown_connections 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce629a5740 1 --2- 172.21.15.40:0/3946966896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556d585e75b0 0x556d585f1d00 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce629a5740 1 --2- 172.21.15.40:0/3946966896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556d585e2440 0x556d585f0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce629a5740 1 --2- 172.21.15.40:0/3946966896 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556d5865e5b0 0x556d586609d0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce629a5740 1 --2- 172.21.15.40:0/3946966896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d585e1a50 0x556d585f07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce417fa640 0 client.17154 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce629a5740 1 -- 172.21.15.40:0/3946966896 >> 172.21.15.40:0/3946966896 conn(0x556d585dcfc0 msgr2=0x556d585e0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:38.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce629a5740 1 -- 172.21.15.40:0/3946966896 shutdown_connections 2024-03-18T22:48:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce629a5740 1 -- 172.21.15.40:0/3946966896 wait complete. 2024-03-18T22:48:38.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce629a5740 20 client.17154 trim_cache size 0 max 16384 2024-03-18T22:48:38.748 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:48:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:48:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:48:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: "bef703c0-7028-41e6-bea9-8456fefb3d7e": { 2024-03-18T22:48:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:48:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "bef703c0-7028-41e6-bea9-8456fefb3d7e", 2024-03-18T22:48:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:48:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:48:38.748 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:48:38.749 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:48:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce5fdb9640 1 -- 172.21.15.40:0/304041885 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fce440a6c90 msgr2=0x7fce440a9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce5fdb9640 1 --2- 172.21.15.40:0/304041885 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fce440a6c90 0x7fce440a9150 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x556d585e2ca0 tx=0x7fce50009500 comp rx=0 tx=0).stop 2024-03-18T22:48:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce5fdb9640 1 -- 172.21.15.40:0/304041885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce581007c0 msgr2=0x7fce5806a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:48:38.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.742+0000 7fce5fdb9640 1 --2- 172.21.15.40:0/304041885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce581007c0 0x7fce5806a9e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fce54000c00 tx=0x7fce54005dc0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.743+0000 7fce5fdb9640 1 -- 172.21.15.40:0/304041885 shutdown_connections 2024-03-18T22:48:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.743+0000 7fce5fdb9640 1 --2- 172.21.15.40:0/304041885 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fce440a6c90 0x7fce440a9150 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.743+0000 7fce5fdb9640 1 --2- 172.21.15.40:0/304041885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fce58102370 0x7fce5819eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.743+0000 7fce5fdb9640 1 --2- 172.21.15.40:0/304041885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce581019b0 0x7fce5819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.743+0000 7fce5fdb9640 1 --2- 172.21.15.40:0/304041885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce581007c0 0x7fce5806a9e0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:48:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.743+0000 7fce5fdb9640 1 -- 172.21.15.40:0/304041885 >> 172.21.15.40:0/304041885 conn(0x7fce580fbd30 msgr2=0x7fce58104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:48:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.743+0000 7fce5fdb9640 1 -- 172.21.15.40:0/304041885 shutdown_connections 2024-03-18T22:48:38.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:48:38.743+0000 7fce5fdb9640 1 -- 172.21.15.40:0/304041885 wait complete. 2024-03-18T22:48:38.755 INFO:tasks.cephfs.filesystem:scrub status for tag:bef703c0-7028-41e6-bea9-8456fefb3d7e - {'path': '/', 'tag': 'bef703c0-7028-41e6-bea9-8456fefb3d7e', 'options': 'recursive,force'} 2024-03-18T22:48:39.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:39 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:48:39.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:39 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:39.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:39 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:48:39.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:39 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:48:39.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:39 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:39.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:39 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:48:39.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:39 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:39.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:39 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:39.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:39 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:48:39.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:39 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:39.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:39 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:48:39.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:39 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:48:39.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:39 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:39.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:39 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:48:39.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:39 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:39.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:39 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:39.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:39 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:48:39.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:39 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:39.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:39 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:48:39.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:39 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:48:39.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:39 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:39.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:39 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:48:39.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:39 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:39.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:39 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:48:40.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:40 smithi113 ceph-mon[66775]: pgmap v531: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:48:40.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:40 smithi103 ceph-mon[66377]: pgmap v531: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:48:40.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:40 smithi040 ceph-mon[62643]: pgmap v531: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:48:42.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:42 smithi113 ceph-mon[66775]: pgmap v532: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:48:42.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:42 smithi103 ceph-mon[66377]: pgmap v532: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:48:42.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:42 smithi040 ceph-mon[62643]: pgmap v532: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:48:44.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:44 smithi113 ceph-mon[66775]: pgmap v533: 129 pgs: 129 active+clean; 7.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 31 MiB/s wr, 22 op/s 2024-03-18T22:48:44.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:44 smithi103 ceph-mon[66377]: pgmap v533: 129 pgs: 129 active+clean; 7.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 31 MiB/s wr, 22 op/s 2024-03-18T22:48:44.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:44 smithi040 ceph-mon[62643]: pgmap v533: 129 pgs: 129 active+clean; 7.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 31 MiB/s wr, 22 op/s 2024-03-18T22:48:46.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:46 smithi113 ceph-mon[66775]: pgmap v534: 129 pgs: 129 active+clean; 7.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:48:46.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:46 smithi103 ceph-mon[66377]: pgmap v534: 129 pgs: 129 active+clean; 7.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:48:46.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:46 smithi040 ceph-mon[62643]: pgmap v534: 129 pgs: 129 active+clean; 7.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:48:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:48 smithi113 ceph-mon[66775]: pgmap v535: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:48:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:48 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:48:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:48 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:48:48.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:48 smithi103 ceph-mon[66377]: pgmap v535: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:48:48.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:48 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:48:48.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:48 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:48:48.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:48 smithi040 ceph-mon[62643]: pgmap v535: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:48:48.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:48 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:48:48.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:48 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:48:50.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:50 smithi113 ceph-mon[66775]: pgmap v536: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:48:50.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:50 smithi103 ceph-mon[66377]: pgmap v536: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:48:50.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:50 smithi040 ceph-mon[62643]: pgmap v536: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:48:52.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:52 smithi113 ceph-mon[66775]: pgmap v537: 129 pgs: 129 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:48:52.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:52 smithi103 ceph-mon[66377]: pgmap v537: 129 pgs: 129 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:48:52.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:52 smithi040 ceph-mon[62643]: pgmap v537: 129 pgs: 129 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:48:54.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:53 smithi113 ceph-mon[66775]: pgmap v538: 129 pgs: 129 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 31 MiB/s wr, 22 op/s 2024-03-18T22:48:54.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:53 smithi103 ceph-mon[66377]: pgmap v538: 129 pgs: 129 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 31 MiB/s wr, 22 op/s 2024-03-18T22:48:54.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:53 smithi040 ceph-mon[62643]: pgmap v538: 129 pgs: 129 active+clean; 7.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 31 MiB/s wr, 22 op/s 2024-03-18T22:48:56.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:55 smithi113 ceph-mon[66775]: pgmap v539: 129 pgs: 129 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:48:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:55 smithi103 ceph-mon[66377]: pgmap v539: 129 pgs: 129 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:48:56.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:55 smithi040 ceph-mon[62643]: pgmap v539: 129 pgs: 129 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:48:58.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:57 smithi113 ceph-mon[66775]: pgmap v540: 129 pgs: 129 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:48:58.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:57 smithi103 ceph-mon[66377]: pgmap v540: 129 pgs: 129 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:48:58.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:57 smithi040 ceph-mon[62643]: pgmap v540: 129 pgs: 129 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:49:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:48:59 smithi113 ceph-mon[66775]: pgmap v541: 129 pgs: 129 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 26 MiB/s wr, 19 op/s 2024-03-18T22:49:00.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:48:59 smithi103 ceph-mon[66377]: pgmap v541: 129 pgs: 129 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 26 MiB/s wr, 19 op/s 2024-03-18T22:49:00.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:48:59 smithi040 ceph-mon[62643]: pgmap v541: 129 pgs: 129 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 26 MiB/s wr, 19 op/s 2024-03-18T22:49:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:01 smithi113 ceph-mon[66775]: pgmap v542: 129 pgs: 129 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:49:02.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:01 smithi103 ceph-mon[66377]: pgmap v542: 129 pgs: 129 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:49:02.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:01 smithi040 ceph-mon[62643]: pgmap v542: 129 pgs: 129 active+clean; 7.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:49:04.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:03 smithi113 ceph-mon[66775]: pgmap v543: 129 pgs: 129 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:49:04.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:03 smithi103 ceph-mon[66377]: pgmap v543: 129 pgs: 129 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:49:04.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:03 smithi040 ceph-mon[62643]: pgmap v543: 129 pgs: 129 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:49:06.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:05 smithi113 ceph-mon[66775]: pgmap v544: 129 pgs: 129 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:49:06.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:05 smithi103 ceph-mon[66377]: pgmap v544: 129 pgs: 129 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:49:06.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:05 smithi040 ceph-mon[62643]: pgmap v544: 129 pgs: 129 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:49:08.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:07 smithi113 ceph-mon[66775]: pgmap v545: 129 pgs: 129 active+clean; 8.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:49:08.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:07 smithi103 ceph-mon[66377]: pgmap v545: 129 pgs: 129 active+clean; 8.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:49:08.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:07 smithi040 ceph-mon[62643]: pgmap v545: 129 pgs: 129 active+clean; 8.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:49:08.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:49:08.756 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:49:08.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 -- 172.21.15.40:0/2607002996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7e0100720 msgr2=0x7fe7e0100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:08.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 --2- 172.21.15.40:0/2607002996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7e0100720 0x7fe7e0100b40 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4009900 tx=0x7fe7d402bc30 comp rx=0 tx=0).stop 2024-03-18T22:49:08.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 -- 172.21.15.40:0/2607002996 shutdown_connections 2024-03-18T22:49:08.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 --2- 172.21.15.40:0/2607002996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7e01022d0 0x7fe7e0108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 --2- 172.21.15.40:0/2607002996 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7e0101910 0x7fe7e0101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 --2- 172.21.15.40:0/2607002996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7e0100720 0x7fe7e0100b40 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 -- 172.21.15.40:0/2607002996 >> 172.21.15.40:0/2607002996 conn(0x7fe7e00fbcb0 msgr2=0x7fe7e00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:08.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 -- 172.21.15.40:0/2607002996 shutdown_connections 2024-03-18T22:49:08.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 -- 172.21.15.40:0/2607002996 wait complete. 2024-03-18T22:49:08.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 Processor -- start 2024-03-18T22:49:08.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 -- start start 2024-03-18T22:49:08.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7e0100720 0x7fe7e0197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7e0101910 0x7fe7e0198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7e01022d0 0x7fe7e019f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e010c240 con 0x7fe7e01022d0 2024-03-18T22:49:08.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e010c0c0 con 0x7fe7e0100720 2024-03-18T22:49:08.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.873+0000 7fe7e6b37640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e010c3c0 con 0x7fe7e0101910 2024-03-18T22:49:08.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7e50d4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7e01022d0 0x7fe7e019f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7dffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7e0101910 0x7fe7e0198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7e50d4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7e01022d0 0x7fe7e019f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51618/0 (socket says 172.21.15.40:51618) 2024-03-18T22:49:08.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7e50d4640 1 -- 172.21.15.40:0/3197991752 learned_addr learned my addr 172.21.15.40:0/3197991752 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:08.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7e48d3640 1 --2- 172.21.15.40:0/3197991752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7e0100720 0x7fe7e0197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7e50d4640 1 -- 172.21.15.40:0/3197991752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7e0101910 msgr2=0x7fe7e0198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:08.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7e50d4640 1 --2- 172.21.15.40:0/3197991752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7e0101910 0x7fe7e0198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7e50d4640 1 -- 172.21.15.40:0/3197991752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7e0100720 msgr2=0x7fe7e0197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:08.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7e50d4640 1 --2- 172.21.15.40:0/3197991752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7e0100720 0x7fe7e0197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7e50d4640 1 -- 172.21.15.40:0/3197991752 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e019f740 con 0x7fe7e01022d0 2024-03-18T22:49:08.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7dffff640 1 --2- 172.21.15.40:0/3197991752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7e0101910 0x7fe7e0198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:08.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7e50d4640 1 --2- 172.21.15.40:0/3197991752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7e01022d0 0x7fe7e019f0d0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8009fd0 tx=0x7fe7d800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:08.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7ddffb640 1 -- 172.21.15.40:0/3197991752 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d8017070 con 0x7fe7e01022d0 2024-03-18T22:49:08.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7e6b37640 1 -- 172.21.15.40:0/3197991752 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7e019fa30 con 0x7fe7e01022d0 2024-03-18T22:49:08.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7ddffb640 1 -- 172.21.15.40:0/3197991752 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7d8003e20 con 0x7fe7e01022d0 2024-03-18T22:49:08.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.874+0000 7fe7ddffb640 1 -- 172.21.15.40:0/3197991752 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d8010460 con 0x7fe7e01022d0 2024-03-18T22:49:08.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.875+0000 7fe7e6b37640 1 -- 172.21.15.40:0/3197991752 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7e019ff30 con 0x7fe7e01022d0 2024-03-18T22:49:08.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.877+0000 7fe7ddffb640 1 -- 172.21.15.40:0/3197991752 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe7d8003970 con 0x7fe7e01022d0 2024-03-18T22:49:08.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.877+0000 7fe7e97f6740 1 Processor -- start 2024-03-18T22:49:08.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.877+0000 7fe7e97f6740 1 -- start start 2024-03-18T22:49:08.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.877+0000 7fe7e97f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 0x55c573ff3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.877+0000 7fe7e97f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c573ff8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.877+0000 7fe7ddffb640 1 --2- 172.21.15.40:0/3197991752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe7d00a2800 0x7fe7d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.877+0000 7fe7e97f6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 0x55c573ffba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.877+0000 7fe7e97f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c573f58500 con 0x55c573ff4440 2024-03-18T22:49:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.877+0000 7fe7e97f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c573f00be0 con 0x55c573ff3a50 2024-03-18T22:49:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.877+0000 7fe7e97f6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c573f545a0 con 0x55c573ff95b0 2024-03-18T22:49:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7e48d3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c573ff8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7dffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 0x55c573ff3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7e48d3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c573ff8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51628/0 (socket says 172.21.15.40:51628) 2024-03-18T22:49:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7e48d3640 1 -- 172.21.15.40:0/3732968343 learned_addr learned my addr 172.21.15.40:0/3732968343 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7dffff640 1 --2- 172.21.15.40:0/3197991752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe7d00a2800 0x7fe7d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7ddffb640 1 -- 172.21.15.40:0/3197991752 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe7d80ceb90 con 0x7fe7e01022d0 2024-03-18T22:49:08.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7e50d4640 1 --2- 172.21.15.40:0/3732968343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 0x55c573ffba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7e48d3640 1 -- 172.21.15.40:0/3732968343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 msgr2=0x55c573ffba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:08.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7e48d3640 1 --2- 172.21.15.40:0/3732968343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 0x55c573ffba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7e48d3640 1 -- 172.21.15.40:0/3732968343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 msgr2=0x55c573ff3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:08.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7e48d3640 1 --2- 172.21.15.40:0/3732968343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 0x55c573ff3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7e48d3640 1 -- 172.21.15.40:0/3732968343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c573f22000 con 0x55c573ff4440 2024-03-18T22:49:08.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7e50d4640 1 --2- 172.21.15.40:0/3732968343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 0x55c573ffba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:49:08.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7dffff640 1 --2- 172.21.15.40:0/3732968343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 0x55c573ff3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:08.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7e48d3640 1 --2- 172.21.15.40:0/3732968343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c573ff8f40 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4009900 tx=0x7fe7d4009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba37168804aa314 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:08.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7dffff640 1 --2- 172.21.15.40:0/3197991752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe7d00a2800 0x7fe7d00a4cc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x55c573ff4ca0 tx=0x7fe7cc005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:08.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7caffd640 1 -- 172.21.15.40:0/3732968343 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d404c070 con 0x55c573ff4440 2024-03-18T22:49:08.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7caffd640 1 -- 172.21.15.40:0/3732968343 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7d402c9a0 con 0x55c573ff4440 2024-03-18T22:49:08.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.878+0000 7fe7caffd640 1 -- 172.21.15.40:0/3732968343 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4005410 con 0x55c573ff4440 2024-03-18T22:49:08.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.879+0000 7fe7e97f6740 1 -- 172.21.15.40:0/3732968343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 msgr2=0x55c573ff8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:08.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.879+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/3732968343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c573ff8f40 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4009900 tx=0x7fe7d4009610 comp rx=0 tx=0).stop 2024-03-18T22:49:08.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.879+0000 7fe7e97f6740 1 -- 172.21.15.40:0/3732968343 shutdown_connections 2024-03-18T22:49:08.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.879+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/3732968343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 0x55c573ffba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.879+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/3732968343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c573ff8f40 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.879+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/3732968343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 0x55c573ff3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.879+0000 7fe7e97f6740 1 -- 172.21.15.40:0/3732968343 >> 172.21.15.40:0/3732968343 conn(0x55c573feefc0 msgr2=0x55c573fef3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:08.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.879+0000 7fe7e97f6740 1 -- 172.21.15.40:0/3732968343 shutdown_connections 2024-03-18T22:49:08.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.879+0000 7fe7e97f6740 1 -- 172.21.15.40:0/3732968343 wait complete. 2024-03-18T22:49:08.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.880+0000 7fe7e97f6740 1 -- start start 2024-03-18T22:49:08.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.880+0000 7fe7e97f6740 1 Processor -- start 2024-03-18T22:49:08.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.880+0000 7fe7e97f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 0x55c573ffde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.880+0000 7fe7e97f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c573ffe3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.880+0000 7fe7e97f6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 0x55c5740053f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.880+0000 7fe7e97f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c573ffc500 con 0x55c573ff4440 2024-03-18T22:49:08.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.880+0000 7fe7e97f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c573f14160 con 0x55c573ff3a50 2024-03-18T22:49:08.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.880+0000 7fe7e97f6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c573ffc680 con 0x55c573ff95b0 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.880+0000 7fe7e97f6740 10 client.0 _ll_register_callbacks cb 0x55c573f50a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.880+0000 7fe7dffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c573ffe3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.880+0000 7fe7dffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c573ffe3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51644/0 (socket says 172.21.15.40:51644) 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.880+0000 7fe7dffff640 1 -- 172.21.15.40:0/3667792333 learned_addr learned my addr 172.21.15.40:0/3667792333 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.880+0000 7fe7e48d3640 1 --2- 172.21.15.40:0/3667792333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 0x55c573ffde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.880+0000 7fe7e50d4640 1 --2- 172.21.15.40:0/3667792333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 0x55c5740053f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7dffff640 1 -- 172.21.15.40:0/3667792333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 msgr2=0x55c5740053f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7dffff640 1 --2- 172.21.15.40:0/3667792333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 0x55c5740053f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7dffff640 1 -- 172.21.15.40:0/3667792333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 msgr2=0x55c573ffde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7dffff640 1 --2- 172.21.15.40:0/3667792333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 0x55c573ffde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7dffff640 1 -- 172.21.15.40:0/3667792333 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c573d367d0 con 0x55c573ff4440 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7e50d4640 1 --2- 172.21.15.40:0/3667792333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 0x55c5740053f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7e48d3640 1 --2- 172.21.15.40:0/3667792333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 0x55c573ffde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7dffff640 1 --2- 172.21.15.40:0/3667792333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c573ffe3b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc011b20 tx=0x7fe7cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f90f1ff8bc77dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7bffff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7bffff640 1 -- 172.21.15.40:0/3667792333 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc0044d0 con 0x55c573ff4440 2024-03-18T22:49:08.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7bffff640 1 -- 172.21.15.40:0/3667792333 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe7cc020420 con 0x55c573ff4440 2024-03-18T22:49:08.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7e97f6740 1 -- 172.21.15.40:0/3667792333 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7d402c9a0 con 0x55c573ff4440 2024-03-18T22:49:08.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7bffff640 1 -- 172.21.15.40:0/3667792333 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe7cc020620 con 0x55c573ff4440 2024-03-18T22:49:08.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7e97f6740 10 client.17166 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:08.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7e97f6740 1 -- 172.21.15.40:0/3667792333 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c573c6c3f0 con 0x55c573ff4440 2024-03-18T22:49:08.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.881+0000 7fe7bffff640 1 -- 172.21.15.40:0/3667792333 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7d402c9a0 con 0x55c573ff4440 2024-03-18T22:49:08.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.882+0000 7fe7bffff640 1 -- 172.21.15.40:0/3667792333 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe7cc028070 con 0x55c573ff4440 2024-03-18T22:49:08.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.882+0000 7fe7e97f6740 10 client.17166 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:08.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.882+0000 7fe7e97f6740 10 client.17166 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:08.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.882+0000 7fe7e97f6740 20 client.17166 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:08.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.882+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/3667792333 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c574009d70 0x55c57400c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.882+0000 7fe7e97f6740 4 client.17166 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:08.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.882+0000 7fe7e97f6740 1 -- 172.21.15.40:0/3667792333 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c573f65a40 con 0x55c574009d70 2024-03-18T22:49:08.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.882+0000 7fe7e48d3640 1 --2- 172.21.15.40:0/3667792333 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c574009d70 0x55c57400c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.882+0000 7fe7bffff640 10 client.17166 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:08.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.882+0000 7fe7e48d3640 1 --2- 172.21.15.40:0/3667792333 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c574009d70 0x55c57400c200 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=155916a94259a10c server_cookie=4d73d7f92b022f05 in_seq=0 out_seq=0 2024-03-18T22:49:08.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.883+0000 7fe7bffff640 1 -- 172.21.15.40:0/3667792333 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c573f65a40 con 0x55c574009d70 2024-03-18T22:49:08.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.883+0000 7fe7bffff640 10 client.17166 handle_command_reply: tid=0 2024-03-18T22:49:08.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.883+0000 7fe7e97f6740 1 client.17166 shutdown 2024-03-18T22:49:08.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.883+0000 7fe7e97f6740 1 -- 172.21.15.40:0/3667792333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 msgr2=0x55c573ffe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:08.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.883+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/3667792333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c573ffe3b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc011b20 tx=0x7fe7cc004730 comp rx=0 tx=0).stop 2024-03-18T22:49:08.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.883+0000 7fe7e97f6740 1 -- 172.21.15.40:0/3667792333 shutdown_connections 2024-03-18T22:49:08.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.883+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/3667792333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 0x55c5740053f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.883+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/3667792333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c573ffe3b0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.883+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/3667792333 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c574009d70 0x55c57400c200 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.883+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/3667792333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 0x55c573ffde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.883+0000 7fe7bffff640 0 client.17166 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:08.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.883+0000 7fe7e97f6740 1 -- 172.21.15.40:0/3667792333 >> 172.21.15.40:0/3667792333 conn(0x55c573feefc0 msgr2=0x55c573fee1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:08.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.883+0000 7fe7e97f6740 1 -- 172.21.15.40:0/3667792333 shutdown_connections 2024-03-18T22:49:08.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.883+0000 7fe7e97f6740 1 -- 172.21.15.40:0/3667792333 wait complete. 2024-03-18T22:49:08.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.883+0000 7fe7e97f6740 20 client.17166 trim_cache size 0 max 16384 2024-03-18T22:49:08.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7e97f6740 1 Processor -- start 2024-03-18T22:49:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7e97f6740 1 -- start start 2024-03-18T22:49:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7e97f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 0x55c573ff2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7e97f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c5740053b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7e97f6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 0x55c574010ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7e97f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e01a8320 con 0x55c573ff4440 2024-03-18T22:49:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7e97f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e01a9e60 con 0x55c573ff3a50 2024-03-18T22:49:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7e97f6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e01a89f0 con 0x55c573ff95b0 2024-03-18T22:49:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7dffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c5740053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7dffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c5740053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51650/0 (socket says 172.21.15.40:51650) 2024-03-18T22:49:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7dffff640 1 -- 172.21.15.40:0/926107589 learned_addr learned my addr 172.21.15.40:0/926107589 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7e48d3640 1 --2- 172.21.15.40:0/926107589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 0x55c573ff2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7e50d4640 1 --2- 172.21.15.40:0/926107589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 0x55c574010ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7dffff640 1 -- 172.21.15.40:0/926107589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 msgr2=0x55c574010ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7dffff640 1 --2- 172.21.15.40:0/926107589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 0x55c574010ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7dffff640 1 -- 172.21.15.40:0/926107589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 msgr2=0x55c573ff2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7dffff640 1 --2- 172.21.15.40:0/926107589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 0x55c573ff2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7dffff640 1 -- 172.21.15.40:0/926107589 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c573c2c0f0 con 0x55c573ff4440 2024-03-18T22:49:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.903+0000 7fe7e50d4640 1 --2- 172.21.15.40:0/926107589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 0x55c574010ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.904+0000 7fe7dffff640 1 --2- 172.21.15.40:0/926107589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c5740053b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00fc90 tx=0x7fe7cc0118e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5472818981642b77 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.904+0000 7fe7ca7fc640 1 -- 172.21.15.40:0/926107589 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc0044d0 con 0x55c573ff4440 2024-03-18T22:49:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.904+0000 7fe7ca7fc640 1 -- 172.21.15.40:0/926107589 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7cc009cf0 con 0x55c573ff4440 2024-03-18T22:49:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.904+0000 7fe7ca7fc640 1 -- 172.21.15.40:0/926107589 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc011280 con 0x55c573ff4440 2024-03-18T22:49:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 -- 172.21.15.40:0/926107589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 msgr2=0x55c5740053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/926107589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c5740053b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00fc90 tx=0x7fe7cc0118e0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 -- 172.21.15.40:0/926107589 shutdown_connections 2024-03-18T22:49:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/926107589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff95b0 0x55c574010ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/926107589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff4440 0x55c5740053b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/926107589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff3a50 0x55c573ff2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 -- 172.21.15.40:0/926107589 >> 172.21.15.40:0/926107589 conn(0x55c573feefc0 msgr2=0x55c573ff2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 -- 172.21.15.40:0/926107589 shutdown_connections 2024-03-18T22:49:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 -- 172.21.15.40:0/926107589 wait complete. 2024-03-18T22:49:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 -- start start 2024-03-18T22:49:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 Processor -- start 2024-03-18T22:49:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff3a50 0x55c5740027d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff4440 0x55c574002d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff95b0 0x55c574003cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e01a9790 con 0x55c573ff95b0 2024-03-18T22:49:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e01a90c0 con 0x55c573ff4440 2024-03-18T22:49:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.905+0000 7fe7e97f6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e01a6300 con 0x55c573ff3a50 2024-03-18T22:49:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7e97f6740 10 client.0 _ll_register_callbacks cb 0x7fe7cc019040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7e50d4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff95b0 0x55c574003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7e48d3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff3a50 0x55c5740027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7dffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff4440 0x55c574002d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7e48d3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff3a50 0x55c5740027d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:60860/0 (socket says 172.21.15.40:60860) 2024-03-18T22:49:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7e48d3640 1 -- 172.21.15.40:0/753211046 learned_addr learned my addr 172.21.15.40:0/753211046 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7e48d3640 1 -- 172.21.15.40:0/753211046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff4440 msgr2=0x55c574002d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7e48d3640 1 --2- 172.21.15.40:0/753211046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff4440 0x55c574002d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7e48d3640 1 -- 172.21.15.40:0/753211046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff95b0 msgr2=0x55c574003cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7e48d3640 1 --2- 172.21.15.40:0/753211046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff95b0 0x55c574003cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7e48d3640 1 -- 172.21.15.40:0/753211046 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5740042c0 con 0x55c573ff3a50 2024-03-18T22:49:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7dffff640 1 --2- 172.21.15.40:0/753211046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff4440 0x55c574002d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7e50d4640 1 --2- 172.21.15.40:0/753211046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff95b0 0x55c574003cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7e48d3640 1 --2- 172.21.15.40:0/753211046 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff3a50 0x55c5740027d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe7d402c840 tx=0x7fe7d402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f588eba60a31337 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7bffff640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:49:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7bffff640 1 -- 172.21.15.40:0/753211046 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d404c040 con 0x55c573ff3a50 2024-03-18T22:49:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7e97f6740 1 -- 172.21.15.40:0/753211046 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7cc009cf0 con 0x55c573ff3a50 2024-03-18T22:49:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.906+0000 7fe7bffff640 1 -- 172.21.15.40:0/753211046 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe7d402c9a0 con 0x55c573ff3a50 2024-03-18T22:49:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.907+0000 7fe7bffff640 1 -- 172.21.15.40:0/753211046 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe7d4063050 con 0x55c573ff3a50 2024-03-18T22:49:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.907+0000 7fe7e97f6740 10 client.26983 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.907+0000 7fe7e97f6740 1 -- 172.21.15.40:0/753211046 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c574005640 con 0x55c573ff3a50 2024-03-18T22:49:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.907+0000 7fe7bffff640 1 -- 172.21.15.40:0/753211046 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe7cc009cf0 con 0x55c573ff3a50 2024-03-18T22:49:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.907+0000 7fe7bffff640 1 -- 172.21.15.40:0/753211046 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe7d4052500 con 0x55c573ff3a50 2024-03-18T22:49:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.907+0000 7fe7e97f6740 10 client.26983 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.907+0000 7fe7e97f6740 10 client.26983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.907+0000 7fe7e97f6740 20 client.26983 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.907+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/753211046 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c574070490 0x55c574072880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.907+0000 7fe7e97f6740 4 client.26983 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:49:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.907+0000 7fe7e97f6740 1 -- 172.21.15.40:0/753211046 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c573f5fd90 con 0x55c574070490 2024-03-18T22:49:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.907+0000 7fe7dffff640 1 --2- 172.21.15.40:0/753211046 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c574070490 0x55c574072880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7bffff640 10 client.26983 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7dffff640 1 --2- 172.21.15.40:0/753211046 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c574070490 0x55c574072880 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4847fa6439f0dda5 server_cookie=fb41bdedc53584a9 in_seq=0 out_seq=0 2024-03-18T22:49:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7bffff640 1 -- 172.21.15.40:0/753211046 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c573f5fd90 con 0x55c574070490 2024-03-18T22:49:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7bffff640 10 client.26983 handle_command_reply: tid=0 2024-03-18T22:49:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7e97f6740 1 client.26983 shutdown 2024-03-18T22:49:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7e97f6740 1 -- 172.21.15.40:0/753211046 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff3a50 msgr2=0x55c5740027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/753211046 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff3a50 0x55c5740027d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe7d402c840 tx=0x7fe7d402bc30 comp rx=0 tx=0).stop 2024-03-18T22:49:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7e97f6740 1 -- 172.21.15.40:0/753211046 shutdown_connections 2024-03-18T22:49:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/753211046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c573ff95b0 0x55c574003cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/753211046 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c574070490 0x55c574072880 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/753211046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c573ff4440 0x55c574002d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7e97f6740 1 --2- 172.21.15.40:0/753211046 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c573ff3a50 0x55c5740027d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7bffff640 0 client.26983 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7e97f6740 1 -- 172.21.15.40:0/753211046 >> 172.21.15.40:0/753211046 conn(0x55c573feefc0 msgr2=0x55c573ff2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7e97f6740 1 -- 172.21.15.40:0/753211046 shutdown_connections 2024-03-18T22:49:08.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7e97f6740 1 -- 172.21.15.40:0/753211046 wait complete. 2024-03-18T22:49:08.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.908+0000 7fe7e97f6740 20 client.26983 trim_cache size 0 max 16384 2024-03-18T22:49:08.916 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:49:08.916 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:49:08.916 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:49:08.916 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:49:08.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.909+0000 7fe7e6b37640 1 -- 172.21.15.40:0/3197991752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe7d00a2800 msgr2=0x7fe7d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:08.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.909+0000 7fe7e6b37640 1 --2- 172.21.15.40:0/3197991752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe7d00a2800 0x7fe7d00a4cc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x55c573ff4ca0 tx=0x7fe7cc005f90 comp rx=0 tx=0).stop 2024-03-18T22:49:08.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.909+0000 7fe7e6b37640 1 -- 172.21.15.40:0/3197991752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7e01022d0 msgr2=0x7fe7e019f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:08.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.909+0000 7fe7e6b37640 1 --2- 172.21.15.40:0/3197991752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7e01022d0 0x7fe7e019f0d0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8009fd0 tx=0x7fe7d800b260 comp rx=0 tx=0).stop 2024-03-18T22:49:08.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.909+0000 7fe7e6b37640 1 -- 172.21.15.40:0/3197991752 shutdown_connections 2024-03-18T22:49:08.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.909+0000 7fe7e6b37640 1 --2- 172.21.15.40:0/3197991752 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe7d00a2800 0x7fe7d00a4cc0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.909+0000 7fe7e6b37640 1 --2- 172.21.15.40:0/3197991752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7e01022d0 0x7fe7e019f0d0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.909+0000 7fe7e6b37640 1 --2- 172.21.15.40:0/3197991752 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe7e0101910 0x7fe7e0198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.909+0000 7fe7e6b37640 1 --2- 172.21.15.40:0/3197991752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7e0100720 0x7fe7e0197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:08.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.909+0000 7fe7e6b37640 1 -- 172.21.15.40:0/3197991752 >> 172.21.15.40:0/3197991752 conn(0x7fe7e00fbcb0 msgr2=0x7fe7e00689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:08.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.909+0000 7fe7e6b37640 1 -- 172.21.15.40:0/3197991752 shutdown_connections 2024-03-18T22:49:08.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:08.909+0000 7fe7e6b37640 1 -- 172.21.15.40:0/3197991752 wait complete. 2024-03-18T22:49:08.922 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:49:08.922 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:49:09.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.041+0000 7f9b06bc4640 1 -- 172.21.15.40:0/3918284510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b00103410 msgr2=0x7f9b00109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.041+0000 7f9b06bc4640 1 --2- 172.21.15.40:0/3918284510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b00103410 0x7f9b00109ca0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9af8009ca0 tx=0x7f9af802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.041+0000 7f9b06bc4640 1 -- 172.21.15.40:0/3918284510 shutdown_connections 2024-03-18T22:49:09.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.041+0000 7f9b06bc4640 1 --2- 172.21.15.40:0/3918284510 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b00103410 0x7f9b00109ca0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.041+0000 7f9b06bc4640 1 --2- 172.21.15.40:0/3918284510 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b00102a50 0x7f9b00102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.041+0000 7f9b06bc4640 1 --2- 172.21.15.40:0/3918284510 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b00101860 0x7f9b00101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.041+0000 7f9b06bc4640 1 -- 172.21.15.40:0/3918284510 >> 172.21.15.40:0/3918284510 conn(0x7f9b000fcdd0 msgr2=0x7f9b000ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.041+0000 7f9b06bc4640 1 -- 172.21.15.40:0/3918284510 shutdown_connections 2024-03-18T22:49:09.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.041+0000 7f9b06bc4640 1 -- 172.21.15.40:0/3918284510 wait complete. 2024-03-18T22:49:09.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.042+0000 7f9b06bc4640 1 Processor -- start 2024-03-18T22:49:09.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.042+0000 7f9b06bc4640 1 -- start start 2024-03-18T22:49:09.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.042+0000 7f9b06bc4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b00101860 0x7f9b00197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.042+0000 7f9b06bc4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b00102a50 0x7f9b00197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.042+0000 7f9b06bc4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b00103410 0x7f9b0019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.042+0000 7f9b06bc4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0010d390 con 0x7f9b00102a50 2024-03-18T22:49:09.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.042+0000 7f9b06bc4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0010d210 con 0x7f9b00101860 2024-03-18T22:49:09.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.042+0000 7f9b06bc4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0010d510 con 0x7f9b00103410 2024-03-18T22:49:09.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.042+0000 7f9afffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b00102a50 0x7f9b00197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.042+0000 7f9afffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b00102a50 0x7f9b00197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51682/0 (socket says 172.21.15.40:51682) 2024-03-18T22:49:09.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.042+0000 7f9afffff640 1 -- 172.21.15.40:0/3404868223 learned_addr learned my addr 172.21.15.40:0/3404868223 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.042+0000 7f9b05161640 1 --2- 172.21.15.40:0/3404868223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b00103410 0x7f9b0019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.042+0000 7f9b04960640 1 --2- 172.21.15.40:0/3404868223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b00101860 0x7f9b00197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.043+0000 7f9afffff640 1 -- 172.21.15.40:0/3404868223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b00103410 msgr2=0x7f9b0019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.043+0000 7f9afffff640 1 --2- 172.21.15.40:0/3404868223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b00103410 0x7f9b0019f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.043+0000 7f9afffff640 1 -- 172.21.15.40:0/3404868223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b00101860 msgr2=0x7f9b00197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.043+0000 7f9afffff640 1 --2- 172.21.15.40:0/3404868223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b00101860 0x7f9b00197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.043+0000 7f9afffff640 1 -- 172.21.15.40:0/3404868223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b0019f690 con 0x7f9b00102a50 2024-03-18T22:49:09.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.043+0000 7f9b05161640 1 --2- 172.21.15.40:0/3404868223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b00103410 0x7f9b0019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.043+0000 7f9afffff640 1 --2- 172.21.15.40:0/3404868223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b00102a50 0x7f9b00197fb0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00e9f0 tx=0x7f9aec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.043+0000 7f9afdffb640 1 -- 172.21.15.40:0/3404868223 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aec00cd80 con 0x7f9b00102a50 2024-03-18T22:49:09.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.043+0000 7f9afdffb640 1 -- 172.21.15.40:0/3404868223 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9aec0108e0 con 0x7f9b00102a50 2024-03-18T22:49:09.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.043+0000 7f9b06bc4640 1 -- 172.21.15.40:0/3404868223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b0019f980 con 0x7f9b00102a50 2024-03-18T22:49:09.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.043+0000 7f9afdffb640 1 -- 172.21.15.40:0/3404868223 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aec018920 con 0x7f9b00102a50 2024-03-18T22:49:09.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.043+0000 7f9b06bc4640 1 -- 172.21.15.40:0/3404868223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b0019ff20 con 0x7f9b00102a50 2024-03-18T22:49:09.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.044+0000 7f9b06bc4640 1 -- 172.21.15.40:0/3404868223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b00108c00 con 0x7f9b00102a50 2024-03-18T22:49:09.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.046+0000 7f9afdffb640 1 -- 172.21.15.40:0/3404868223 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9aec010a90 con 0x7f9b00102a50 2024-03-18T22:49:09.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.047+0000 7f9afdffb640 1 --2- 172.21.15.40:0/3404868223 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9af00a2880 0x7f9af00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.047+0000 7f9b04960640 1 --2- 172.21.15.40:0/3404868223 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9af00a2880 0x7f9af00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.047+0000 7f9afdffb640 1 -- 172.21.15.40:0/3404868223 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9aec0cf590 con 0x7f9b00102a50 2024-03-18T22:49:09.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.047+0000 7f9b04960640 1 --2- 172.21.15.40:0/3404868223 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9af00a2880 0x7f9af00a4d40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9b001028b0 tx=0x7f9af4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.051+0000 7f9afdffb640 1 -- 172.21.15.40:0/3404868223 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9aec09bb90 con 0x7f9b00102a50 2024-03-18T22:49:09.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.292+0000 7f9b06bc4640 1 -- 172.21.15.40:0/3404868223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b0002d1f0 con 0x7f9b00102a50 2024-03-18T22:49:09.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.293+0000 7f9afdffb640 1 -- 172.21.15.40:0/3404868223 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11947 (secure 0 0 0) 0x7f9aec09fc30 con 0x7f9b00102a50 2024-03-18T22:49:09.294 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:49:09.294 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:46:52.724919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24388,24401,24473,24481,24509]},"id":1}]} 2024-03-18T22:49:09.294 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 32 2024-03-18T22:49:09.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.295+0000 7f9b06bc4640 1 -- 172.21.15.40:0/3404868223 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9af00a2880 msgr2=0x7f9af00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.295+0000 7f9b06bc4640 1 --2- 172.21.15.40:0/3404868223 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9af00a2880 0x7f9af00a4d40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9b001028b0 tx=0x7f9af4005f90 comp rx=0 tx=0).stop 2024-03-18T22:49:09.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.295+0000 7f9b06bc4640 1 -- 172.21.15.40:0/3404868223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b00102a50 msgr2=0x7f9b00197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.295+0000 7f9b06bc4640 1 --2- 172.21.15.40:0/3404868223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b00102a50 0x7f9b00197fb0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00e9f0 tx=0x7f9aec00eeb0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.295+0000 7f9b06bc4640 1 -- 172.21.15.40:0/3404868223 shutdown_connections 2024-03-18T22:49:09.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.295+0000 7f9b06bc4640 1 --2- 172.21.15.40:0/3404868223 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9af00a2880 0x7f9af00a4d40 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.295+0000 7f9b06bc4640 1 --2- 172.21.15.40:0/3404868223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9b00103410 0x7f9b0019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.295+0000 7f9b06bc4640 1 --2- 172.21.15.40:0/3404868223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b00102a50 0x7f9b00197fb0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.295+0000 7f9b06bc4640 1 --2- 172.21.15.40:0/3404868223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b00101860 0x7f9b00197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.295+0000 7f9b06bc4640 1 -- 172.21.15.40:0/3404868223 >> 172.21.15.40:0/3404868223 conn(0x7f9b000fcdd0 msgr2=0x7f9b00105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.295+0000 7f9b06bc4640 1 -- 172.21.15.40:0/3404868223 shutdown_connections 2024-03-18T22:49:09.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.296+0000 7f9b06bc4640 1 -- 172.21.15.40:0/3404868223 wait complete. 2024-03-18T22:49:09.307 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:49:09.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.427+0000 7efdaa7ba640 1 -- 172.21.15.40:0/2609431579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efda4101910 msgr2=0x7efda4101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.427+0000 7efdaa7ba640 1 --2- 172.21.15.40:0/2609431579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efda4101910 0x7efda4101d90 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7efd98009900 tx=0x7efd9802bc30 comp rx=0 tx=0).stop 2024-03-18T22:49:09.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.427+0000 7efdaa7ba640 1 -- 172.21.15.40:0/2609431579 shutdown_connections 2024-03-18T22:49:09.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.427+0000 7efdaa7ba640 1 --2- 172.21.15.40:0/2609431579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efda41022d0 0x7efda4108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.427+0000 7efdaa7ba640 1 --2- 172.21.15.40:0/2609431579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efda4101910 0x7efda4101d90 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.427+0000 7efdaa7ba640 1 --2- 172.21.15.40:0/2609431579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efda4100720 0x7efda4100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.427+0000 7efdaa7ba640 1 -- 172.21.15.40:0/2609431579 >> 172.21.15.40:0/2609431579 conn(0x7efda40fbc90 msgr2=0x7efda40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.428+0000 7efdaa7ba640 1 -- 172.21.15.40:0/2609431579 shutdown_connections 2024-03-18T22:49:09.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.428+0000 7efdaa7ba640 1 -- 172.21.15.40:0/2609431579 wait complete. 2024-03-18T22:49:09.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.428+0000 7efdaa7ba640 1 Processor -- start 2024-03-18T22:49:09.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.428+0000 7efdaa7ba640 1 -- start start 2024-03-18T22:49:09.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.428+0000 7efdaa7ba640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efda4100720 0x7efda4197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.428+0000 7efdaa7ba640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efda4101910 0x7efda4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.428+0000 7efdaa7ba640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efda41022d0 0x7efda419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.428+0000 7efdaa7ba640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efda410c230 con 0x7efda41022d0 2024-03-18T22:49:09.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efdaa7ba640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efda410c0b0 con 0x7efda4101910 2024-03-18T22:49:09.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efdaa7ba640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7efda410c3b0 con 0x7efda4100720 2024-03-18T22:49:09.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efda8d57640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efda41022d0 0x7efda419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efda8d57640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efda41022d0 0x7efda419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51712/0 (socket says 172.21.15.40:51712) 2024-03-18T22:49:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efda8d57640 1 -- 172.21.15.40:0/2091150891 learned_addr learned my addr 172.21.15.40:0/2091150891 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efda37fe640 1 --2- 172.21.15.40:0/2091150891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efda4101910 0x7efda4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efda3fff640 1 --2- 172.21.15.40:0/2091150891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efda4100720 0x7efda4197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efda8d57640 1 -- 172.21.15.40:0/2091150891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efda4100720 msgr2=0x7efda4197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efda8d57640 1 --2- 172.21.15.40:0/2091150891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efda4100720 0x7efda4197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efda8d57640 1 -- 172.21.15.40:0/2091150891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efda4101910 msgr2=0x7efda4197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efda8d57640 1 --2- 172.21.15.40:0/2091150891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efda4101910 0x7efda4197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efda8d57640 1 -- 172.21.15.40:0/2091150891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda419f6d0 con 0x7efda41022d0 2024-03-18T22:49:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efda37fe640 1 --2- 172.21.15.40:0/2091150891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efda4101910 0x7efda4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efda8d57640 1 --2- 172.21.15.40:0/2091150891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efda41022d0 0x7efda419f060 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7efd9c00eb10 tx=0x7efd9c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efda17fa640 1 -- 172.21.15.40:0/2091150891 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c004b20 con 0x7efda41022d0 2024-03-18T22:49:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efda17fa640 1 -- 172.21.15.40:0/2091150891 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd9c004cc0 con 0x7efda41022d0 2024-03-18T22:49:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efdaa7ba640 1 -- 172.21.15.40:0/2091150891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efda419f9c0 con 0x7efda41022d0 2024-03-18T22:49:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.429+0000 7efda17fa640 1 -- 172.21.15.40:0/2091150891 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c010690 con 0x7efda41022d0 2024-03-18T22:49:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.430+0000 7efdaa7ba640 1 -- 172.21.15.40:0/2091150891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efda419fee0 con 0x7efda41022d0 2024-03-18T22:49:09.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.432+0000 7efda17fa640 1 -- 172.21.15.40:0/2091150891 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7efd9c005020 con 0x7efda41022d0 2024-03-18T22:49:09.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.432+0000 7efda17fa640 1 --2- 172.21.15.40:0/2091150891 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efd8c0a2880 0x7efd8c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.432+0000 7efda17fa640 1 -- 172.21.15.40:0/2091150891 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7efd9c0cf330 con 0x7efda41022d0 2024-03-18T22:49:09.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.432+0000 7efdad39d740 1 Processor -- start 2024-03-18T22:49:09.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.432+0000 7efda3fff640 1 --2- 172.21.15.40:0/2091150891 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efd8c0a2880 0x7efd8c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efdad39d740 1 -- start start 2024-03-18T22:49:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efdad39d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359a700 0x5611c359ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efdad39d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359b0f0 0x5611c359fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efdad39d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c35a0260 0x5611c35a26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efdad39d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5611c34fa4b0 con 0x5611c359a700 2024-03-18T22:49:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efdad39d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5611c34a2d70 con 0x5611c359b0f0 2024-03-18T22:49:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efdad39d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5611c34f6550 con 0x5611c35a0260 2024-03-18T22:49:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efda8d57640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c35a0260 0x5611c35a26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efda8d57640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c35a0260 0x5611c35a26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:60912/0 (socket says 172.21.15.40:60912) 2024-03-18T22:49:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efda8d57640 1 -- 172.21.15.40:0/425409265 learned_addr learned my addr 172.21.15.40:0/425409265 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efda37fe640 1 --2- 172.21.15.40:0/425409265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359b0f0 0x5611c359fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efda3fff640 1 --2- 172.21.15.40:0/425409265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359a700 0x5611c359ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efda3fff640 1 --2- 172.21.15.40:0/2091150891 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efd8c0a2880 0x7efd8c0a4d40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efd940095f0 tx=0x7efd94005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efda8d57640 1 -- 172.21.15.40:0/425409265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359b0f0 msgr2=0x5611c359fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efda8d57640 1 --2- 172.21.15.40:0/425409265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359b0f0 0x5611c359fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efda8d57640 1 -- 172.21.15.40:0/425409265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359a700 msgr2=0x5611c359ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efda8d57640 1 --2- 172.21.15.40:0/425409265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359a700 0x5611c359ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efda8d57640 1 -- 172.21.15.40:0/425409265 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611c34c5210 con 0x5611c35a0260 2024-03-18T22:49:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efda3fff640 1 --2- 172.21.15.40:0/425409265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359a700 0x5611c359ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:49:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.433+0000 7efda8d57640 1 --2- 172.21.15.40:0/425409265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c35a0260 0x5611c35a26f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efd9c0a3b50 tx=0x7efd9c09b4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df159d370bb64ddd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.434+0000 7efd8a7fc640 1 -- 172.21.15.40:0/425409265 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c0b4070 con 0x5611c35a0260 2024-03-18T22:49:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.434+0000 7efd8a7fc640 1 -- 172.21.15.40:0/425409265 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd9c0cf170 con 0x5611c35a0260 2024-03-18T22:49:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.434+0000 7efd8a7fc640 1 -- 172.21.15.40:0/425409265 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c0860c0 con 0x5611c35a0260 2024-03-18T22:49:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.434+0000 7efdad39d740 1 -- 172.21.15.40:0/425409265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c35a0260 msgr2=0x5611c35a26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.434+0000 7efdad39d740 1 --2- 172.21.15.40:0/425409265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c35a0260 0x5611c35a26f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efd9c0a3b50 tx=0x7efd9c09b4e0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.435+0000 7efdad39d740 1 -- 172.21.15.40:0/425409265 shutdown_connections 2024-03-18T22:49:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.435+0000 7efdad39d740 1 --2- 172.21.15.40:0/425409265 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c35a0260 0x5611c35a26f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.435+0000 7efdad39d740 1 --2- 172.21.15.40:0/425409265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359b0f0 0x5611c359fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.435+0000 7efdad39d740 1 --2- 172.21.15.40:0/425409265 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359a700 0x5611c359ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.435+0000 7efdad39d740 1 -- 172.21.15.40:0/425409265 >> 172.21.15.40:0/425409265 conn(0x5611c3591860 msgr2=0x5611c3591c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.435+0000 7efdad39d740 1 -- 172.21.15.40:0/425409265 shutdown_connections 2024-03-18T22:49:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.435+0000 7efdad39d740 1 -- 172.21.15.40:0/425409265 wait complete. 2024-03-18T22:49:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.435+0000 7efdad39d740 1 -- start start 2024-03-18T22:49:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.435+0000 7efdad39d740 1 Processor -- start 2024-03-18T22:49:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.435+0000 7efdad39d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c359a700 0x5611c35a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.435+0000 7efdad39d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359b0f0 0x5611c35a5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.435+0000 7efdad39d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c35a0260 0x5611c35ac190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.435+0000 7efdad39d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5611c35a3210 con 0x5611c359b0f0 2024-03-18T22:49:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.435+0000 7efdad39d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5611c34b6310 con 0x5611c35a0260 2024-03-18T22:49:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.435+0000 7efdad39d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5611c35a3390 con 0x5611c359a700 2024-03-18T22:49:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efdad39d740 10 client.0 _ll_register_callbacks cb 0x5611c34aa2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efda37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359b0f0 0x5611c35a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efda3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c359a700 0x5611c35a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efda37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359b0f0 0x5611c35a5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51744/0 (socket says 172.21.15.40:51744) 2024-03-18T22:49:09.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efda37fe640 1 -- 172.21.15.40:0/357776706 learned_addr learned my addr 172.21.15.40:0/357776706 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efda8d57640 1 --2- 172.21.15.40:0/357776706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c35a0260 0x5611c35ac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efda8d57640 1 -- 172.21.15.40:0/357776706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c359a700 msgr2=0x5611c35a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efda8d57640 1 --2- 172.21.15.40:0/357776706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c359a700 0x5611c35a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efda8d57640 1 -- 172.21.15.40:0/357776706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359b0f0 msgr2=0x5611c35a5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efda8d57640 1 --2- 172.21.15.40:0/357776706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359b0f0 0x5611c35a5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efda8d57640 1 -- 172.21.15.40:0/357776706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611c320e3f0 con 0x5611c35a0260 2024-03-18T22:49:09.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efda3fff640 1 --2- 172.21.15.40:0/357776706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c359a700 0x5611c35a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efda8d57640 1 --2- 172.21.15.40:0/357776706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c35a0260 0x5611c35ac190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efd9c097d40 tx=0x7efd9c0a3ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1abb309203f71d46 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efd837fe640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:49:09.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efd837fe640 1 -- 172.21.15.40:0/357776706 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c080060 con 0x5611c35a0260 2024-03-18T22:49:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efd837fe640 1 -- 172.21.15.40:0/357776706 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7efd9c005310 con 0x5611c35a0260 2024-03-18T22:49:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efdad39d740 1 -- 172.21.15.40:0/357776706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5611c35583e0 con 0x5611c35a0260 2024-03-18T22:49:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.436+0000 7efd837fe640 1 -- 172.21.15.40:0/357776706 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efd9c0b4040 con 0x5611c35a0260 2024-03-18T22:49:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.437+0000 7efdad39d740 10 client.27011 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.437+0000 7efdad39d740 1 -- 172.21.15.40:0/357776706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5611c32d87d0 con 0x5611c35a0260 2024-03-18T22:49:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.437+0000 7efd837fe640 1 -- 172.21.15.40:0/357776706 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5611c35583e0 con 0x5611c35a0260 2024-03-18T22:49:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.437+0000 7efd837fe640 1 -- 172.21.15.40:0/357776706 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7efd9c097630 con 0x5611c35a0260 2024-03-18T22:49:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.437+0000 7efdad39d740 10 client.27011 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.437+0000 7efdad39d740 10 client.27011 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:49:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.437+0000 7efdad39d740 20 client.27011 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.437+0000 7efdad39d740 1 --2- 172.21.15.40:0/357776706 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x5611c35b0680 0x5611c35b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.437+0000 7efdad39d740 4 client.27011 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:09.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.437+0000 7efdad39d740 1 -- 172.21.15.40:0/357776706 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5611c35079f0 con 0x5611c35b0680 2024-03-18T22:49:09.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.437+0000 7efda3fff640 1 --2- 172.21.15.40:0/357776706 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x5611c35b0680 0x5611c35b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efda3fff640 1 --2- 172.21.15.40:0/357776706 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x5611c35b0680 0x5611c35b2a70 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1eebba2390649bc7 server_cookie=d9c1794372fe07dd in_seq=0 out_seq=0 2024-03-18T22:49:09.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efd837fe640 10 client.27011 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:49:09.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efd837fe640 1 -- 172.21.15.40:0/357776706 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5611c35079f0 con 0x5611c35b0680 2024-03-18T22:49:09.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efd837fe640 10 client.27011 handle_command_reply: tid=0 2024-03-18T22:49:09.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efdad39d740 1 client.27011 shutdown 2024-03-18T22:49:09.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efdad39d740 1 -- 172.21.15.40:0/357776706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c35a0260 msgr2=0x5611c35ac190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efdad39d740 1 --2- 172.21.15.40:0/357776706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c35a0260 0x5611c35ac190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efd9c097d40 tx=0x7efd9c0a3ea0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efdad39d740 1 -- 172.21.15.40:0/357776706 shutdown_connections 2024-03-18T22:49:09.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efdad39d740 1 --2- 172.21.15.40:0/357776706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c35a0260 0x5611c35ac190 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efdad39d740 1 --2- 172.21.15.40:0/357776706 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x5611c35b0680 0x5611c35b2a70 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efdad39d740 1 --2- 172.21.15.40:0/357776706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359b0f0 0x5611c35a5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efdad39d740 1 --2- 172.21.15.40:0/357776706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c359a700 0x5611c35a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efd837fe640 0 client.27011 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:49:09.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efdad39d740 1 -- 172.21.15.40:0/357776706 >> 172.21.15.40:0/357776706 conn(0x5611c3591860 msgr2=0x5611c3590ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efdad39d740 1 -- 172.21.15.40:0/357776706 shutdown_connections 2024-03-18T22:49:09.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.438+0000 7efdad39d740 1 -- 172.21.15.40:0/357776706 wait complete. 2024-03-18T22:49:09.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.439+0000 7efdad39d740 20 client.27011 trim_cache size 0 max 16384 2024-03-18T22:49:09.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efdad39d740 1 Processor -- start 2024-03-18T22:49:09.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efdad39d740 1 -- start start 2024-03-18T22:49:09.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efdad39d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c359a700 0x5611c3591c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efdad39d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359b0f0 0x5611c35ac150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efdad39d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c35a0260 0x5611c3590310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efdad39d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efda41a8350 con 0x5611c35a0260 2024-03-18T22:49:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efdad39d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efda41a9e90 con 0x5611c359b0f0 2024-03-18T22:49:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efdad39d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7efda41a8a20 con 0x5611c359a700 2024-03-18T22:49:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efda8d57640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c35a0260 0x5611c3590310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efda3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c359a700 0x5611c3591c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efda3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c359a700 0x5611c3591c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:60930/0 (socket says 172.21.15.40:60930) 2024-03-18T22:49:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efda37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359b0f0 0x5611c35ac150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efda8d57640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c35a0260 0x5611c3590310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51758/0 (socket says 172.21.15.40:51758) 2024-03-18T22:49:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efda3fff640 1 -- 172.21.15.40:0/3816526026 learned_addr learned my addr 172.21.15.40:0/3816526026 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efda3fff640 1 -- 172.21.15.40:0/3816526026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359b0f0 msgr2=0x5611c35ac150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efda3fff640 1 --2- 172.21.15.40:0/3816526026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359b0f0 0x5611c35ac150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efda3fff640 1 -- 172.21.15.40:0/3816526026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c35a0260 msgr2=0x5611c3590310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efda3fff640 1 --2- 172.21.15.40:0/3816526026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c35a0260 0x5611c3590310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.453+0000 7efda3fff640 1 -- 172.21.15.40:0/3816526026 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611c35a34d0 con 0x5611c359a700 2024-03-18T22:49:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.454+0000 7efda8d57640 1 --2- 172.21.15.40:0/3816526026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c35a0260 0x5611c3590310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.454+0000 7efda3fff640 1 --2- 172.21.15.40:0/3816526026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c359a700 0x5611c3591c40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x5611c359b950 tx=0x7efd94014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=464cba20156d7bf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.454+0000 7efd89ffb640 1 -- 172.21.15.40:0/3816526026 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9401fc90 con 0x5611c359a700 2024-03-18T22:49:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.454+0000 7efd89ffb640 1 -- 172.21.15.40:0/3816526026 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd9401fe30 con 0x5611c359a700 2024-03-18T22:49:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.454+0000 7efd89ffb640 1 -- 172.21.15.40:0/3816526026 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd94003850 con 0x5611c359a700 2024-03-18T22:49:09.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 -- 172.21.15.40:0/3816526026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c359a700 msgr2=0x5611c3591c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 --2- 172.21.15.40:0/3816526026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c359a700 0x5611c3591c40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x5611c359b950 tx=0x7efd94014000 comp rx=0 tx=0).stop 2024-03-18T22:49:09.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 -- 172.21.15.40:0/3816526026 shutdown_connections 2024-03-18T22:49:09.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 --2- 172.21.15.40:0/3816526026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c35a0260 0x5611c3590310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 --2- 172.21.15.40:0/3816526026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359b0f0 0x5611c35ac150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 --2- 172.21.15.40:0/3816526026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c359a700 0x5611c3591c40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 -- 172.21.15.40:0/3816526026 >> 172.21.15.40:0/3816526026 conn(0x5611c3591860 msgr2=0x5611c358ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 -- 172.21.15.40:0/3816526026 shutdown_connections 2024-03-18T22:49:09.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 -- 172.21.15.40:0/3816526026 wait complete. 2024-03-18T22:49:09.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 -- start start 2024-03-18T22:49:09.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 Processor -- start 2024-03-18T22:49:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359a700 0x5611c35a9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359b0f0 0x5611c35a98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c35a0260 0x5611c35aa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efda41a97c0 con 0x5611c359b0f0 2024-03-18T22:49:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efda41a90f0 con 0x5611c359a700 2024-03-18T22:49:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7efda41a6330 con 0x5611c35a0260 2024-03-18T22:49:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.455+0000 7efdad39d740 10 client.0 _ll_register_callbacks cb 0x7efd9c0b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efda37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359b0f0 0x5611c35a98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efda3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359a700 0x5611c35a9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efda37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359b0f0 0x5611c35a98c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51764/0 (socket says 172.21.15.40:51764) 2024-03-18T22:49:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efda3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359a700 0x5611c35a9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:53736/0 (socket says 172.21.15.40:53736) 2024-03-18T22:49:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efda37fe640 1 -- 172.21.15.40:0/1066477288 learned_addr learned my addr 172.21.15.40:0/1066477288 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efda8d57640 1 --2- 172.21.15.40:0/1066477288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c35a0260 0x5611c35aa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efda37fe640 1 -- 172.21.15.40:0/1066477288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c35a0260 msgr2=0x5611c35aa8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efda37fe640 1 --2- 172.21.15.40:0/1066477288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c35a0260 0x5611c35aa8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efda37fe640 1 -- 172.21.15.40:0/1066477288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359a700 msgr2=0x5611c35a9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efda37fe640 1 --2- 172.21.15.40:0/1066477288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359a700 0x5611c35a9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efda37fe640 1 -- 172.21.15.40:0/1066477288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611c35aaec0 con 0x5611c359b0f0 2024-03-18T22:49:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efda3fff640 1 --2- 172.21.15.40:0/1066477288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359a700 0x5611c35a9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efda37fe640 1 --2- 172.21.15.40:0/1066477288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359b0f0 0x5611c35a98c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7efd98036040 tx=0x7efd98005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cebf63fdb39b6e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efd837fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efd837fe640 1 -- 172.21.15.40:0/1066477288 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98005550 con 0x5611c359b0f0 2024-03-18T22:49:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efd837fe640 1 -- 172.21.15.40:0/1066477288 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7efd9803a040 con 0x5611c359b0f0 2024-03-18T22:49:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.456+0000 7efdad39d740 1 -- 172.21.15.40:0/1066477288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd9401fe30 con 0x5611c359b0f0 2024-03-18T22:49:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.457+0000 7efd837fe640 1 -- 172.21.15.40:0/1066477288 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efd980046c0 con 0x5611c359b0f0 2024-03-18T22:49:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.457+0000 7efdad39d740 10 client.17196 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.457+0000 7efdad39d740 1 -- 172.21.15.40:0/1066477288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5611c35ac260 con 0x5611c359b0f0 2024-03-18T22:49:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.457+0000 7efd837fe640 1 -- 172.21.15.40:0/1066477288 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd9401fe30 con 0x5611c359b0f0 2024-03-18T22:49:09.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.457+0000 7efd837fe640 1 -- 172.21.15.40:0/1066477288 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7efd98052030 con 0x5611c359b0f0 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.457+0000 7efdad39d740 10 client.17196 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.457+0000 7efdad39d740 10 client.17196 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.457+0000 7efdad39d740 20 client.17196 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.457+0000 7efdad39d740 1 --2- 172.21.15.40:0/1066477288 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x5611c3616e80 0x5611c36192a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.457+0000 7efdad39d740 4 client.17196 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.457+0000 7efdad39d740 1 -- 172.21.15.40:0/1066477288 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5611c3501d40 con 0x5611c3616e80 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.457+0000 7efda3fff640 1 --2- 172.21.15.40:0/1066477288 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x5611c3616e80 0x5611c36192a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efd837fe640 10 client.17196 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efda3fff640 1 --2- 172.21.15.40:0/1066477288 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x5611c3616e80 0x5611c36192a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a6a2eb03895d71a1 server_cookie=99d9980fd1b88dfb in_seq=0 out_seq=0 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efd837fe640 1 -- 172.21.15.40:0/1066477288 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5611c3501d40 con 0x5611c3616e80 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efd837fe640 10 client.17196 handle_command_reply: tid=0 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efdad39d740 1 client.17196 shutdown 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efdad39d740 1 -- 172.21.15.40:0/1066477288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359b0f0 msgr2=0x5611c35a98c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efdad39d740 1 --2- 172.21.15.40:0/1066477288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359b0f0 0x5611c35a98c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7efd98036040 tx=0x7efd98005250 comp rx=0 tx=0).stop 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efdad39d740 1 -- 172.21.15.40:0/1066477288 shutdown_connections 2024-03-18T22:49:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efdad39d740 1 --2- 172.21.15.40:0/1066477288 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5611c35a0260 0x5611c35aa8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efdad39d740 1 --2- 172.21.15.40:0/1066477288 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x5611c3616e80 0x5611c36192a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efdad39d740 1 --2- 172.21.15.40:0/1066477288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5611c359b0f0 0x5611c35a98c0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efdad39d740 1 --2- 172.21.15.40:0/1066477288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5611c359a700 0x5611c35a9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efdad39d740 1 -- 172.21.15.40:0/1066477288 >> 172.21.15.40:0/1066477288 conn(0x5611c3591860 msgr2=0x5611c3591c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efd837fe640 0 client.17196 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:49:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efdad39d740 1 -- 172.21.15.40:0/1066477288 shutdown_connections 2024-03-18T22:49:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efdad39d740 1 -- 172.21.15.40:0/1066477288 wait complete. 2024-03-18T22:49:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.458+0000 7efdad39d740 20 client.17196 trim_cache size 0 max 16384 2024-03-18T22:49:09.464 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:49:09.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.459+0000 7efdaa7ba640 1 -- 172.21.15.40:0/2091150891 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efd8c0a2880 msgr2=0x7efd8c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.459+0000 7efdaa7ba640 1 --2- 172.21.15.40:0/2091150891 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efd8c0a2880 0x7efd8c0a4d40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efd940095f0 tx=0x7efd94005f90 comp rx=0 tx=0).stop 2024-03-18T22:49:09.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.459+0000 7efdaa7ba640 1 -- 172.21.15.40:0/2091150891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efda41022d0 msgr2=0x7efda419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.459+0000 7efdaa7ba640 1 --2- 172.21.15.40:0/2091150891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efda41022d0 0x7efda419f060 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7efd9c00eb10 tx=0x7efd9c00c2d0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.459+0000 7efdaa7ba640 1 -- 172.21.15.40:0/2091150891 shutdown_connections 2024-03-18T22:49:09.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.459+0000 7efdaa7ba640 1 --2- 172.21.15.40:0/2091150891 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efd8c0a2880 0x7efd8c0a4d40 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.459+0000 7efdaa7ba640 1 --2- 172.21.15.40:0/2091150891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efda41022d0 0x7efda419f060 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.459+0000 7efdaa7ba640 1 --2- 172.21.15.40:0/2091150891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efda4101910 0x7efda4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.459+0000 7efdaa7ba640 1 --2- 172.21.15.40:0/2091150891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efda4100720 0x7efda4197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.459+0000 7efdaa7ba640 1 -- 172.21.15.40:0/2091150891 >> 172.21.15.40:0/2091150891 conn(0x7efda40fbc90 msgr2=0x7efda4068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.459+0000 7efdaa7ba640 1 -- 172.21.15.40:0/2091150891 shutdown_connections 2024-03-18T22:49:09.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.459+0000 7efdaa7ba640 1 -- 172.21.15.40:0/2091150891 wait complete. 2024-03-18T22:49:09.474 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:49:09.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.594+0000 7fda6d434640 1 -- 172.21.15.40:0/1043529455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda681022d0 msgr2=0x7fda68108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.594+0000 7fda6d434640 1 --2- 172.21.15.40:0/1043529455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda681022d0 0x7fda68108b60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fda60009ca0 tx=0x7fda6002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.594+0000 7fda6d434640 1 -- 172.21.15.40:0/1043529455 shutdown_connections 2024-03-18T22:49:09.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.594+0000 7fda6d434640 1 --2- 172.21.15.40:0/1043529455 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda681022d0 0x7fda68108b60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.594+0000 7fda6d434640 1 --2- 172.21.15.40:0/1043529455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda68101910 0x7fda68101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.594+0000 7fda6d434640 1 --2- 172.21.15.40:0/1043529455 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda68100720 0x7fda68100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.594+0000 7fda6d434640 1 -- 172.21.15.40:0/1043529455 >> 172.21.15.40:0/1043529455 conn(0x7fda680fbc90 msgr2=0x7fda680fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.594+0000 7fda6d434640 1 -- 172.21.15.40:0/1043529455 shutdown_connections 2024-03-18T22:49:09.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.594+0000 7fda6d434640 1 -- 172.21.15.40:0/1043529455 wait complete. 2024-03-18T22:49:09.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.594+0000 7fda6d434640 1 Processor -- start 2024-03-18T22:49:09.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.594+0000 7fda6d434640 1 -- start start 2024-03-18T22:49:09.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda6d434640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda68100720 0x7fda681046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda6d434640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda68101910 0x7fda68102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda6d434640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda681022d0 0x7fda68103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda6d434640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fda6810c230 con 0x7fda68101910 2024-03-18T22:49:09.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda6d434640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda6810c0b0 con 0x7fda68100720 2024-03-18T22:49:09.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda6d434640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fda6810c3b0 con 0x7fda681022d0 2024-03-18T22:49:09.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda5ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda68101910 0x7fda68102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda5ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda68101910 0x7fda68102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51784/0 (socket says 172.21.15.40:51784) 2024-03-18T22:49:09.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda5ffff640 1 -- 172.21.15.40:0/2764360831 learned_addr learned my addr 172.21.15.40:0/2764360831 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda66ffd640 1 --2- 172.21.15.40:0/2764360831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda68100720 0x7fda681046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda677fe640 1 --2- 172.21.15.40:0/2764360831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda681022d0 0x7fda68103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda5ffff640 1 -- 172.21.15.40:0/2764360831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda681022d0 msgr2=0x7fda68103230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda5ffff640 1 --2- 172.21.15.40:0/2764360831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda681022d0 0x7fda68103230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda5ffff640 1 -- 172.21.15.40:0/2764360831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda68100720 msgr2=0x7fda681046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda5ffff640 1 --2- 172.21.15.40:0/2764360831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda68100720 0x7fda681046b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda5ffff640 1 -- 172.21.15.40:0/2764360831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda68103ac0 con 0x7fda68101910 2024-03-18T22:49:09.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda66ffd640 1 --2- 172.21.15.40:0/2764360831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda68100720 0x7fda681046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda5ffff640 1 --2- 172.21.15.40:0/2764360831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda68101910 0x7fda68102cf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fda5000e9f0 tx=0x7fda5000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda64ff9640 1 -- 172.21.15.40:0/2764360831 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5000cd80 con 0x7fda68101910 2024-03-18T22:49:09.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.595+0000 7fda64ff9640 1 -- 172.21.15.40:0/2764360831 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fda500108e0 con 0x7fda68101910 2024-03-18T22:49:09.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.596+0000 7fda6d434640 1 -- 172.21.15.40:0/2764360831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda681a1b40 con 0x7fda68101910 2024-03-18T22:49:09.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.596+0000 7fda64ff9640 1 -- 172.21.15.40:0/2764360831 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50018920 con 0x7fda68101910 2024-03-18T22:49:09.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.596+0000 7fda6d434640 1 -- 172.21.15.40:0/2764360831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda681a2090 con 0x7fda68101910 2024-03-18T22:49:09.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.598+0000 7fda64ff9640 1 -- 172.21.15.40:0/2764360831 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fda50010a90 con 0x7fda68101910 2024-03-18T22:49:09.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda7001a740 1 Processor -- start 2024-03-18T22:49:09.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda7001a740 1 -- start start 2024-03-18T22:49:09.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda64ff9640 1 --2- 172.21.15.40:0/2764360831 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fda540a2800 0x7fda540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda677fe640 1 --2- 172.21.15.40:0/2764360831 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fda540a2800 0x7fda540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda64ff9640 1 -- 172.21.15.40:0/2764360831 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fda500cf160 con 0x7fda68101910 2024-03-18T22:49:09.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda7001a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153be2f0 0x559f153be710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda7001a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153bece0 0x559f153c37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda7001a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153c3e50 0x559f153c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda7001a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559f153224b0 con 0x559f153c3e50 2024-03-18T22:49:09.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda7001a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559f152cad70 con 0x559f153be2f0 2024-03-18T22:49:09.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda7001a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559f1531e550 con 0x559f153bece0 2024-03-18T22:49:09.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda5ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153bece0 0x559f153c37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda5ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153bece0 0x559f153c37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:60978/0 (socket says 172.21.15.40:60978) 2024-03-18T22:49:09.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda5ffff640 1 -- 172.21.15.40:0/3963157248 learned_addr learned my addr 172.21.15.40:0/3963157248 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda66ffd640 1 --2- 172.21.15.40:0/3963157248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153be2f0 0x559f153be710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda677fe640 1 --2- 172.21.15.40:0/3963157248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153c3e50 0x559f153c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda5ffff640 1 -- 172.21.15.40:0/3963157248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153be2f0 msgr2=0x559f153be710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda5ffff640 1 --2- 172.21.15.40:0/3963157248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153be2f0 0x559f153be710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.599+0000 7fda677fe640 1 --2- 172.21.15.40:0/2764360831 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fda540a2800 0x7fda540a4cc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fda6002c630 tx=0x7fda60032bb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.600+0000 7fda5ffff640 1 -- 172.21.15.40:0/3963157248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153c3e50 msgr2=0x559f153c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.600+0000 7fda5ffff640 1 --2- 172.21.15.40:0/3963157248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153c3e50 0x559f153c62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.600+0000 7fda5ffff640 1 -- 172.21.15.40:0/3963157248 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f152ed210 con 0x559f153bece0 2024-03-18T22:49:09.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.600+0000 7fda5ffff640 1 --2- 172.21.15.40:0/3963157248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153bece0 0x559f153c37e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fda5009b750 tx=0x7fda50010ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ae90b2a88155da2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.600+0000 7fda5d7fa640 1 -- 172.21.15.40:0/3963157248 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda500a3eb0 con 0x559f153bece0 2024-03-18T22:49:09.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.600+0000 7fda5d7fa640 1 -- 172.21.15.40:0/3963157248 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fda50099ac0 con 0x559f153bece0 2024-03-18T22:49:09.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.600+0000 7fda5d7fa640 1 -- 172.21.15.40:0/3963157248 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50097510 con 0x559f153bece0 2024-03-18T22:49:09.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 -- 172.21.15.40:0/3963157248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153bece0 msgr2=0x559f153c37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 --2- 172.21.15.40:0/3963157248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153bece0 0x559f153c37e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fda5009b750 tx=0x7fda50010ea0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 -- 172.21.15.40:0/3963157248 shutdown_connections 2024-03-18T22:49:09.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 --2- 172.21.15.40:0/3963157248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153c3e50 0x559f153c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 --2- 172.21.15.40:0/3963157248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153bece0 0x559f153c37e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 --2- 172.21.15.40:0/3963157248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153be2f0 0x559f153be710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 -- 172.21.15.40:0/3963157248 >> 172.21.15.40:0/3963157248 conn(0x559f153b9860 msgr2=0x559f153b9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 -- 172.21.15.40:0/3963157248 shutdown_connections 2024-03-18T22:49:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 -- 172.21.15.40:0/3963157248 wait complete. 2024-03-18T22:49:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 -- start start 2024-03-18T22:49:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 Processor -- start 2024-03-18T22:49:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153be2f0 0x559f153c8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153bece0 0x559f153c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 0x559f153cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559f153c6e00 con 0x559f153be2f0 2024-03-18T22:49:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559f152de310 con 0x559f153c3e50 2024-03-18T22:49:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.601+0000 7fda7001a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559f153c6f80 con 0x559f153bece0 2024-03-18T22:49:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda7001a740 10 client.0 _ll_register_callbacks cb 0x559f152d22d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda66ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153be2f0 0x559f153c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda66ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153be2f0 0x559f153c8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51794/0 (socket says 172.21.15.40:51794) 2024-03-18T22:49:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda5ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153bece0 0x559f153c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda66ffd640 1 -- 172.21.15.40:0/2041272053 learned_addr learned my addr 172.21.15.40:0/2041272053 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda677fe640 1 --2- 172.21.15.40:0/2041272053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 0x559f153cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda66ffd640 1 -- 172.21.15.40:0/2041272053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153bece0 msgr2=0x559f153c8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda66ffd640 1 --2- 172.21.15.40:0/2041272053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153bece0 0x559f153c8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda66ffd640 1 -- 172.21.15.40:0/2041272053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 msgr2=0x559f153cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda66ffd640 1 --2- 172.21.15.40:0/2041272053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 0x559f153cfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda66ffd640 1 -- 172.21.15.40:0/2041272053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f150363f0 con 0x559f153be2f0 2024-03-18T22:49:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda5ffff640 1 --2- 172.21.15.40:0/2041272053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153bece0 0x559f153c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda66ffd640 1 --2- 172.21.15.40:0/2041272053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153be2f0 0x559f153c8820 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fda58012d20 tx=0x7fda5800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150600c8a6232aac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda4e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda4e7fc640 1 -- 172.21.15.40:0/2041272053 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda58012110 con 0x559f153be2f0 2024-03-18T22:49:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda4e7fc640 1 -- 172.21.15.40:0/2041272053 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fda58004030 con 0x559f153be2f0 2024-03-18T22:49:09.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda7001a740 1 -- 172.21.15.40:0/2041272053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda50099ac0 con 0x559f153be2f0 2024-03-18T22:49:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda4e7fc640 1 -- 172.21.15.40:0/2041272053 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fda58025050 con 0x559f153be2f0 2024-03-18T22:49:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda7001a740 10 client.17208 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.602+0000 7fda7001a740 1 -- 172.21.15.40:0/2041272053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559f151007d0 con 0x559f153be2f0 2024-03-18T22:49:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.603+0000 7fda4e7fc640 1 -- 172.21.15.40:0/2041272053 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fda50099ac0 con 0x559f153be2f0 2024-03-18T22:49:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.603+0000 7fda4e7fc640 1 -- 172.21.15.40:0/2041272053 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fda58003c30 con 0x559f153be2f0 2024-03-18T22:49:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.603+0000 7fda7001a740 10 client.17208 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.603+0000 7fda7001a740 10 client.17208 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:49:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.603+0000 7fda7001a740 20 client.17208 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.603+0000 7fda7001a740 1 --2- 172.21.15.40:0/2041272053 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559f153d4730 0x559f153d6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.603+0000 7fda7001a740 4 client.17208 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.603+0000 7fda7001a740 1 -- 172.21.15.40:0/2041272053 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f153803e0 con 0x559f153d4730 2024-03-18T22:49:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.603+0000 7fda5ffff640 1 --2- 172.21.15.40:0/2041272053 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559f153d4730 0x559f153d6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.603+0000 7fda5ffff640 1 --2- 172.21.15.40:0/2041272053 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559f153d4730 0x559f153d6bb0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ee87787e45d5153a server_cookie=440c7a248e1d4873 in_seq=0 out_seq=0 2024-03-18T22:49:09.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.603+0000 7fda4e7fc640 10 client.17208 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:49:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.604+0000 7fda4e7fc640 1 -- 172.21.15.40:0/2041272053 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559f153803e0 con 0x559f153d4730 2024-03-18T22:49:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.604+0000 7fda4e7fc640 10 client.17208 handle_command_reply: tid=0 2024-03-18T22:49:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.604+0000 7fda7001a740 1 client.17208 shutdown 2024-03-18T22:49:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.605+0000 7fda7001a740 1 -- 172.21.15.40:0/2041272053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153be2f0 msgr2=0x559f153c8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.605+0000 7fda7001a740 1 --2- 172.21.15.40:0/2041272053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153be2f0 0x559f153c8820 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fda58012d20 tx=0x7fda5800f790 comp rx=0 tx=0).stop 2024-03-18T22:49:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.605+0000 7fda7001a740 1 -- 172.21.15.40:0/2041272053 shutdown_connections 2024-03-18T22:49:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.605+0000 7fda7001a740 1 --2- 172.21.15.40:0/2041272053 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559f153d4730 0x559f153d6bb0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.605+0000 7fda7001a740 1 --2- 172.21.15.40:0/2041272053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 0x559f153cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.605+0000 7fda7001a740 1 --2- 172.21.15.40:0/2041272053 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153bece0 0x559f153c8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.605+0000 7fda4e7fc640 0 client.17208 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:49:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.605+0000 7fda7001a740 1 --2- 172.21.15.40:0/2041272053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153be2f0 0x559f153c8820 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.605+0000 7fda7001a740 1 -- 172.21.15.40:0/2041272053 >> 172.21.15.40:0/2041272053 conn(0x559f153b9860 msgr2=0x559f153b8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.605+0000 7fda7001a740 1 -- 172.21.15.40:0/2041272053 shutdown_connections 2024-03-18T22:49:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.605+0000 7fda7001a740 1 -- 172.21.15.40:0/2041272053 wait complete. 2024-03-18T22:49:09.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.605+0000 7fda7001a740 20 client.17208 trim_cache size 0 max 16384 2024-03-18T22:49:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.620+0000 7fda7001a740 1 Processor -- start 2024-03-18T22:49:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.620+0000 7fda7001a740 1 -- start start 2024-03-18T22:49:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.620+0000 7fda7001a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153be2f0 0x559f153c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.620+0000 7fda7001a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153bece0 0x559f153cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.620+0000 7fda7001a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 0x559f153c8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.620+0000 7fda7001a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fda681ac7f0 con 0x559f153bece0 2024-03-18T22:49:09.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.620+0000 7fda7001a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda681ae330 con 0x559f153c3e50 2024-03-18T22:49:09.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.620+0000 7fda7001a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fda681acec0 con 0x559f153be2f0 2024-03-18T22:49:09.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.620+0000 7fda677fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 0x559f153c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.620+0000 7fda5ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153bece0 0x559f153cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.620+0000 7fda66ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153be2f0 0x559f153c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.620+0000 7fda677fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 0x559f153c8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:53790/0 (socket says 172.21.15.40:53790) 2024-03-18T22:49:09.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.620+0000 7fda677fe640 1 -- 172.21.15.40:0/4048383273 learned_addr learned my addr 172.21.15.40:0/4048383273 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.621+0000 7fda66ffd640 1 -- 172.21.15.40:0/4048383273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 msgr2=0x559f153c8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.621+0000 7fda66ffd640 1 --2- 172.21.15.40:0/4048383273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 0x559f153c8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.621+0000 7fda66ffd640 1 -- 172.21.15.40:0/4048383273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153bece0 msgr2=0x559f153cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.621+0000 7fda66ffd640 1 --2- 172.21.15.40:0/4048383273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153bece0 0x559f153cfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.621+0000 7fda66ffd640 1 -- 172.21.15.40:0/4048383273 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f153bf0c0 con 0x559f153be2f0 2024-03-18T22:49:09.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.621+0000 7fda66ffd640 1 --2- 172.21.15.40:0/4048383273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153be2f0 0x559f153c7e40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fda58000c00 tx=0x7fda580123c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4eac37685e23d501 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.621+0000 7fda5cff9640 1 -- 172.21.15.40:0/4048383273 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda58022040 con 0x559f153be2f0 2024-03-18T22:49:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.621+0000 7fda5cff9640 1 -- 172.21.15.40:0/4048383273 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fda5800fe80 con 0x559f153be2f0 2024-03-18T22:49:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.621+0000 7fda5cff9640 1 -- 172.21.15.40:0/4048383273 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda58010be0 con 0x559f153be2f0 2024-03-18T22:49:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.622+0000 7fda7001a740 1 -- 172.21.15.40:0/4048383273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153be2f0 msgr2=0x559f153c7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.622+0000 7fda7001a740 1 --2- 172.21.15.40:0/4048383273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153be2f0 0x559f153c7e40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fda58000c00 tx=0x7fda580123c0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.622+0000 7fda7001a740 1 -- 172.21.15.40:0/4048383273 shutdown_connections 2024-03-18T22:49:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.622+0000 7fda7001a740 1 --2- 172.21.15.40:0/4048383273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 0x559f153c8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.622+0000 7fda7001a740 1 --2- 172.21.15.40:0/4048383273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153bece0 0x559f153cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.622+0000 7fda7001a740 1 --2- 172.21.15.40:0/4048383273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153be2f0 0x559f153c7e40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.622+0000 7fda7001a740 1 -- 172.21.15.40:0/4048383273 >> 172.21.15.40:0/4048383273 conn(0x559f153b9860 msgr2=0x559f153bd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.622+0000 7fda7001a740 1 -- 172.21.15.40:0/4048383273 shutdown_connections 2024-03-18T22:49:09.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.622+0000 7fda7001a740 1 -- 172.21.15.40:0/4048383273 wait complete. 2024-03-18T22:49:09.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.622+0000 7fda7001a740 1 -- start start 2024-03-18T22:49:09.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.622+0000 7fda7001a740 1 Processor -- start 2024-03-18T22:49:09.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda7001a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153be2f0 0x559f153cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda7001a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153bece0 0x559f153cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda7001a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 0x559f153ce570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda7001a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fda681adc60 con 0x559f153bece0 2024-03-18T22:49:09.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda7001a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda681ad590 con 0x559f153c3e50 2024-03-18T22:49:09.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda7001a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fda681aa7d0 con 0x559f153be2f0 2024-03-18T22:49:09.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda7001a740 10 client.0 _ll_register_callbacks cb 0x7fda58009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:09.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda5ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153bece0 0x559f153cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda66ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153be2f0 0x559f153cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda66ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153be2f0 0x559f153cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:32778/0 (socket says 172.21.15.40:32778) 2024-03-18T22:49:09.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda66ffd640 1 -- 172.21.15.40:0/589404818 learned_addr learned my addr 172.21.15.40:0/589404818 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda677fe640 1 --2- 172.21.15.40:0/589404818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 0x559f153ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda5ffff640 1 -- 172.21.15.40:0/589404818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153be2f0 msgr2=0x559f153cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda5ffff640 1 --2- 172.21.15.40:0/589404818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153be2f0 0x559f153cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda5ffff640 1 -- 172.21.15.40:0/589404818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 msgr2=0x559f153ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda5ffff640 1 --2- 172.21.15.40:0/589404818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 0x559f153ce570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda5ffff640 1 -- 172.21.15.40:0/589404818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f153ceb60 con 0x559f153bece0 2024-03-18T22:49:09.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda66ffd640 1 --2- 172.21.15.40:0/589404818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153be2f0 0x559f153cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda5ffff640 1 --2- 172.21.15.40:0/589404818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153bece0 0x559f153cd560 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x559f153c6ad0 tx=0x7fda5009f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=724499d8a28fff89 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda4e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:09.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda4e7fc640 1 -- 172.21.15.40:0/589404818 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50010430 con 0x559f153bece0 2024-03-18T22:49:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda4e7fc640 1 -- 172.21.15.40:0/589404818 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fda500105d0 con 0x559f153bece0 2024-03-18T22:49:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.623+0000 7fda7001a740 1 -- 172.21.15.40:0/589404818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559f1532f9f0 con 0x559f153bece0 2024-03-18T22:49:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.624+0000 7fda4e7fc640 1 -- 172.21.15.40:0/589404818 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fda5009ac10 con 0x559f153bece0 2024-03-18T22:49:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.624+0000 7fda7001a740 10 client.17214 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.624+0000 7fda7001a740 1 -- 172.21.15.40:0/589404818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559f153cfed0 con 0x559f153bece0 2024-03-18T22:49:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.624+0000 7fda4e7fc640 1 -- 172.21.15.40:0/589404818 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x559f1532f9f0 con 0x559f153bece0 2024-03-18T22:49:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.624+0000 7fda4e7fc640 1 -- 172.21.15.40:0/589404818 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fda500afaf0 con 0x559f153bece0 2024-03-18T22:49:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.624+0000 7fda7001a740 10 client.17214 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.624+0000 7fda7001a740 10 client.17214 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:49:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.624+0000 7fda7001a740 20 client.17214 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.624+0000 7fda7001a740 1 --2- 172.21.15.40:0/589404818 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559f1543ae30 0x559f1543d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.624+0000 7fda7001a740 4 client.17214 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:49:09.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.624+0000 7fda7001a740 1 -- 172.21.15.40:0/589404818 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559f15329d40 con 0x559f1543ae30 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.624+0000 7fda66ffd640 1 --2- 172.21.15.40:0/589404818 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559f1543ae30 0x559f1543d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda4e7fc640 10 client.17214 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda66ffd640 1 --2- 172.21.15.40:0/589404818 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559f1543ae30 0x559f1543d220 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a174ded979d42887 server_cookie=c595e9f45410d7b1 in_seq=0 out_seq=0 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda4e7fc640 1 -- 172.21.15.40:0/589404818 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559f15329d40 con 0x559f1543ae30 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda4e7fc640 10 client.17214 handle_command_reply: tid=0 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda7001a740 1 client.17214 shutdown 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda7001a740 1 -- 172.21.15.40:0/589404818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153bece0 msgr2=0x559f153cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda7001a740 1 --2- 172.21.15.40:0/589404818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153bece0 0x559f153cd560 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x559f153c6ad0 tx=0x7fda5009f9e0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda7001a740 1 -- 172.21.15.40:0/589404818 shutdown_connections 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda7001a740 1 --2- 172.21.15.40:0/589404818 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559f1543ae30 0x559f1543d220 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda7001a740 1 --2- 172.21.15.40:0/589404818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559f153c3e50 0x559f153ce570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda7001a740 1 --2- 172.21.15.40:0/589404818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559f153bece0 0x559f153cd560 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda4e7fc640 0 client.17214 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda7001a740 1 --2- 172.21.15.40:0/589404818 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559f153be2f0 0x559f153cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda7001a740 1 -- 172.21.15.40:0/589404818 >> 172.21.15.40:0/589404818 conn(0x559f153b9860 msgr2=0x559f153b8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda7001a740 1 -- 172.21.15.40:0/589404818 shutdown_connections 2024-03-18T22:49:09.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda7001a740 1 -- 172.21.15.40:0/589404818 wait complete. 2024-03-18T22:49:09.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.625+0000 7fda7001a740 20 client.17214 trim_cache size 0 max 16384 2024-03-18T22:49:09.632 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:49:09.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.626+0000 7fda6d434640 1 -- 172.21.15.40:0/2764360831 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fda540a2800 msgr2=0x7fda540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.626+0000 7fda6d434640 1 --2- 172.21.15.40:0/2764360831 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fda540a2800 0x7fda540a4cc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fda6002c630 tx=0x7fda60032bb0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.626+0000 7fda6d434640 1 -- 172.21.15.40:0/2764360831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda68101910 msgr2=0x7fda68102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.626+0000 7fda6d434640 1 --2- 172.21.15.40:0/2764360831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda68101910 0x7fda68102cf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fda5000e9f0 tx=0x7fda5000eeb0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.626+0000 7fda6d434640 1 -- 172.21.15.40:0/2764360831 shutdown_connections 2024-03-18T22:49:09.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.626+0000 7fda6d434640 1 --2- 172.21.15.40:0/2764360831 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fda540a2800 0x7fda540a4cc0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.626+0000 7fda6d434640 1 --2- 172.21.15.40:0/2764360831 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fda681022d0 0x7fda68103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.626+0000 7fda6d434640 1 --2- 172.21.15.40:0/2764360831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda68101910 0x7fda68102cf0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.626+0000 7fda6d434640 1 --2- 172.21.15.40:0/2764360831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda68100720 0x7fda681046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.626+0000 7fda6d434640 1 -- 172.21.15.40:0/2764360831 >> 172.21.15.40:0/2764360831 conn(0x7fda680fbc90 msgr2=0x7fda680fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.626+0000 7fda6d434640 1 -- 172.21.15.40:0/2764360831 shutdown_connections 2024-03-18T22:49:09.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.626+0000 7fda6d434640 1 -- 172.21.15.40:0/2764360831 wait complete. 2024-03-18T22:49:09.639 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:49:09.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.751+0000 7f38f1f53640 1 -- 172.21.15.40:0/4188688508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38ec1022d0 msgr2=0x7f38ec108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.751+0000 7f38f1f53640 1 --2- 172.21.15.40:0/4188688508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38ec1022d0 0x7f38ec108b60 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f38e4009ca0 tx=0x7f38e402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.751+0000 7f38f1f53640 1 -- 172.21.15.40:0/4188688508 shutdown_connections 2024-03-18T22:49:09.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.751+0000 7f38f1f53640 1 --2- 172.21.15.40:0/4188688508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38ec1022d0 0x7f38ec108b60 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.751+0000 7f38f1f53640 1 --2- 172.21.15.40:0/4188688508 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38ec101910 0x7f38ec101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.751+0000 7f38f1f53640 1 --2- 172.21.15.40:0/4188688508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38ec100720 0x7f38ec100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.751+0000 7f38f1f53640 1 -- 172.21.15.40:0/4188688508 >> 172.21.15.40:0/4188688508 conn(0x7f38ec0fbc90 msgr2=0x7f38ec0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.751+0000 7f38f1f53640 1 -- 172.21.15.40:0/4188688508 shutdown_connections 2024-03-18T22:49:09.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.751+0000 7f38f1f53640 1 -- 172.21.15.40:0/4188688508 wait complete. 2024-03-18T22:49:09.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.752+0000 7f38f1f53640 1 Processor -- start 2024-03-18T22:49:09.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.752+0000 7f38f1f53640 1 -- start start 2024-03-18T22:49:09.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.752+0000 7f38f1f53640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38ec100720 0x7f38ec104780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.752+0000 7f38f1f53640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38ec101910 0x7f38ec102dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.752+0000 7f38f1f53640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38ec1022d0 0x7f38ec103300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.752+0000 7f38f1f53640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f38ec10c230 con 0x7f38ec100720 2024-03-18T22:49:09.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.752+0000 7f38f1f53640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f38ec10c0b0 con 0x7f38ec101910 2024-03-18T22:49:09.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.752+0000 7f38f1f53640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f38ec10c3b0 con 0x7f38ec1022d0 2024-03-18T22:49:09.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.752+0000 7f38eb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38ec100720 0x7f38ec104780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.752+0000 7f38eb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38ec100720 0x7f38ec104780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51834/0 (socket says 172.21.15.40:51834) 2024-03-18T22:49:09.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.752+0000 7f38eb7fe640 1 -- 172.21.15.40:0/2899778248 learned_addr learned my addr 172.21.15.40:0/2899778248 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.752+0000 7f38eaffd640 1 --2- 172.21.15.40:0/2899778248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38ec101910 0x7f38ec102dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.752+0000 7f38ebfff640 1 --2- 172.21.15.40:0/2899778248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38ec1022d0 0x7f38ec103300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.753+0000 7f38eb7fe640 1 -- 172.21.15.40:0/2899778248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38ec1022d0 msgr2=0x7f38ec103300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.753+0000 7f38eb7fe640 1 --2- 172.21.15.40:0/2899778248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38ec1022d0 0x7f38ec103300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.753+0000 7f38eb7fe640 1 -- 172.21.15.40:0/2899778248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38ec101910 msgr2=0x7f38ec102dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.753+0000 7f38eb7fe640 1 --2- 172.21.15.40:0/2899778248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38ec101910 0x7f38ec102dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.753+0000 7f38eb7fe640 1 -- 172.21.15.40:0/2899778248 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38ec103b90 con 0x7f38ec100720 2024-03-18T22:49:09.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.753+0000 7f38eaffd640 1 --2- 172.21.15.40:0/2899778248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38ec101910 0x7f38ec102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.753+0000 7f38eb7fe640 1 --2- 172.21.15.40:0/2899778248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38ec100720 0x7f38ec104780 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f38dc00e9c0 tx=0x7f38dc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.753+0000 7f38e8ff9640 1 -- 172.21.15.40:0/2899778248 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38dc00cd80 con 0x7f38ec100720 2024-03-18T22:49:09.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.753+0000 7f38e8ff9640 1 -- 172.21.15.40:0/2899778248 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f38dc0035f0 con 0x7f38ec100720 2024-03-18T22:49:09.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.753+0000 7f38f1f53640 1 -- 172.21.15.40:0/2899778248 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38ec1a1b10 con 0x7f38ec100720 2024-03-18T22:49:09.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.753+0000 7f38e8ff9640 1 -- 172.21.15.40:0/2899778248 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38dc010690 con 0x7f38ec100720 2024-03-18T22:49:09.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.753+0000 7f38f1f53640 1 -- 172.21.15.40:0/2899778248 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38ec1a2060 con 0x7f38ec100720 2024-03-18T22:49:09.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.755+0000 7f38e8ff9640 1 -- 172.21.15.40:0/2899778248 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f38dc0037a0 con 0x7f38ec100720 2024-03-18T22:49:09.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.756+0000 7f38f4c1d740 1 Processor -- start 2024-03-18T22:49:09.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.756+0000 7f38f4c1d740 1 -- start start 2024-03-18T22:49:09.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.756+0000 7f38f4c1d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b91c2f0 0x555d2b91c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.756+0000 7f38f4c1d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91cce0 0x555d2b9217e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.756+0000 7f38e8ff9640 1 --2- 172.21.15.40:0/2899778248 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f38d40a2800 0x7f38d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.756+0000 7f38f4c1d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b921e50 0x555d2b9242e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.756+0000 7f38f4c1d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555d2b8804b0 con 0x555d2b91cce0 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.756+0000 7f38f4c1d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555d2b828d70 con 0x555d2b921e50 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.756+0000 7f38f4c1d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x555d2b87c550 con 0x555d2b91c2f0 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.756+0000 7f38eaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91cce0 0x555d2b9217e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.756+0000 7f38ebfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b91c2f0 0x555d2b91c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.756+0000 7f38eaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91cce0 0x555d2b9217e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51842/0 (socket says 172.21.15.40:51842) 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.756+0000 7f38eaffd640 1 -- 172.21.15.40:0/1702228436 learned_addr learned my addr 172.21.15.40:0/1702228436 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.756+0000 7f38eb7fe640 1 --2- 172.21.15.40:0/2899778248 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f38d40a2800 0x7f38d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.757+0000 7f38ebfff640 1 --2- 172.21.15.40:0/1702228436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b921e50 0x555d2b9242e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.757+0000 7f38eaffd640 1 -- 172.21.15.40:0/1702228436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b91c2f0 msgr2=0x555d2b91c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.757+0000 7f38eaffd640 1 --2- 172.21.15.40:0/1702228436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b91c2f0 0x555d2b91c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.757+0000 7f38eaffd640 1 -- 172.21.15.40:0/1702228436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b921e50 msgr2=0x555d2b9242e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.757+0000 7f38eaffd640 1 --2- 172.21.15.40:0/1702228436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b921e50 0x555d2b9242e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.757+0000 7f38eaffd640 1 -- 172.21.15.40:0/1702228436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d2b84b210 con 0x555d2b91cce0 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.757+0000 7f38ebfff640 1 --2- 172.21.15.40:0/1702228436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b91c2f0 0x555d2b91c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.757+0000 7f38e8ff9640 1 -- 172.21.15.40:0/2899778248 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f38dc0cf240 con 0x7f38ec100720 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.757+0000 7f38eaffd640 1 --2- 172.21.15.40:0/1702228436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91cce0 0x555d2b9217e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f38e00114f0 tx=0x7f38e00119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d736b6a33560327 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.757+0000 7f38eb7fe640 1 --2- 172.21.15.40:0/2899778248 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f38d40a2800 0x7f38d40a4cc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f38dc0cc9a0 tx=0x7f38dc097b90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.757+0000 7f38d1ffb640 1 -- 172.21.15.40:0/1702228436 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e001e070 con 0x555d2b91cce0 2024-03-18T22:49:09.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.757+0000 7f38d1ffb640 1 -- 172.21.15.40:0/1702228436 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f38e0011250 con 0x555d2b91cce0 2024-03-18T22:49:09.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.757+0000 7f38d1ffb640 1 -- 172.21.15.40:0/1702228436 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e0019970 con 0x555d2b91cce0 2024-03-18T22:49:09.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.758+0000 7f38f4c1d740 1 -- 172.21.15.40:0/1702228436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91cce0 msgr2=0x555d2b9217e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.758+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/1702228436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91cce0 0x555d2b9217e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f38e00114f0 tx=0x7f38e00119b0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.758+0000 7f38f4c1d740 1 -- 172.21.15.40:0/1702228436 shutdown_connections 2024-03-18T22:49:09.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.758+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/1702228436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b921e50 0x555d2b9242e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.758+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/1702228436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91cce0 0x555d2b9217e0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.758+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/1702228436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b91c2f0 0x555d2b91c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.758+0000 7f38f4c1d740 1 -- 172.21.15.40:0/1702228436 >> 172.21.15.40:0/1702228436 conn(0x555d2b917860 msgr2=0x555d2b917c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.758+0000 7f38f4c1d740 1 -- 172.21.15.40:0/1702228436 shutdown_connections 2024-03-18T22:49:09.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.758+0000 7f38f4c1d740 1 -- 172.21.15.40:0/1702228436 wait complete. 2024-03-18T22:49:09.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.758+0000 7f38f4c1d740 1 -- start start 2024-03-18T22:49:09.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38f4c1d740 1 Processor -- start 2024-03-18T22:49:09.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38f4c1d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91c2f0 0x555d2b926800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38f4c1d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b91cce0 0x555d2b926d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38f4c1d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b921e50 0x555d2b92dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38f4c1d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555d2b924e00 con 0x555d2b91c2f0 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38f4c1d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555d2b83c310 con 0x555d2b921e50 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38f4c1d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x555d2b924f80 con 0x555d2b91cce0 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38f4c1d740 10 client.0 _ll_register_callbacks cb 0x555d2b8302d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38ebfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91c2f0 0x555d2b926800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38ebfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91c2f0 0x555d2b926800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51858/0 (socket says 172.21.15.40:51858) 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38ebfff640 1 -- 172.21.15.40:0/3219609853 learned_addr learned my addr 172.21.15.40:0/3219609853 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38eaffd640 1 --2- 172.21.15.40:0/3219609853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b91cce0 0x555d2b926d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38ebfff640 1 --2- 172.21.15.40:0/3219609853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b921e50 0x555d2b92dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38ebfff640 1 -- 172.21.15.40:0/3219609853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b91cce0 msgr2=0x555d2b926d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38ebfff640 1 --2- 172.21.15.40:0/3219609853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b91cce0 0x555d2b926d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38ebfff640 1 -- 172.21.15.40:0/3219609853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b921e50 msgr2=0x555d2b92dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38ebfff640 1 --2- 172.21.15.40:0/3219609853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b921e50 0x555d2b92dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38ebfff640 1 -- 172.21.15.40:0/3219609853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d2b5943f0 con 0x555d2b91c2f0 2024-03-18T22:49:09.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38eaffd640 1 --2- 172.21.15.40:0/3219609853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b91cce0 0x555d2b926d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38ebfff640 1 --2- 172.21.15.40:0/3219609853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91c2f0 0x555d2b926800 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f38e4032420 tx=0x7f38e40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ee0de6a7ee56de server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.759+0000 7f38caffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.760+0000 7f38caffd640 1 -- 172.21.15.40:0/3219609853 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e4039040 con 0x555d2b91c2f0 2024-03-18T22:49:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.760+0000 7f38caffd640 1 -- 172.21.15.40:0/3219609853 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f38e4004040 con 0x555d2b91c2f0 2024-03-18T22:49:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.760+0000 7f38f4c1d740 1 -- 172.21.15.40:0/3219609853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38e0011250 con 0x555d2b91c2f0 2024-03-18T22:49:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.760+0000 7f38caffd640 1 -- 172.21.15.40:0/3219609853 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f38e4054050 con 0x555d2b91c2f0 2024-03-18T22:49:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.760+0000 7f38f4c1d740 10 client.17226 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.760+0000 7f38f4c1d740 1 -- 172.21.15.40:0/3219609853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555d2b65e7d0 con 0x555d2b91c2f0 2024-03-18T22:49:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.760+0000 7f38caffd640 1 -- 172.21.15.40:0/3219609853 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f38e404b070 con 0x555d2b91c2f0 2024-03-18T22:49:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.760+0000 7f38caffd640 1 -- 172.21.15.40:0/3219609853 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f38e403f070 con 0x555d2b91c2f0 2024-03-18T22:49:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.760+0000 7f38f4c1d740 10 client.17226 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.760+0000 7f38f4c1d740 10 client.17226 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:49:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.760+0000 7f38f4c1d740 20 client.17226 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:09.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.760+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/3219609853 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d2b9326b0 0x555d2b934b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.760+0000 7f38f4c1d740 4 client.17226 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:09.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.760+0000 7f38f4c1d740 1 -- 172.21.15.40:0/3219609853 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f38e0011250 con 0x555d2b9326b0 2024-03-18T22:49:09.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.761+0000 7f38eaffd640 1 --2- 172.21.15.40:0/3219609853 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d2b9326b0 0x555d2b934b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.761+0000 7f38eaffd640 1 --2- 172.21.15.40:0/3219609853 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d2b9326b0 0x555d2b934b00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c6286db9f7d94184 server_cookie=79a2b19369e79426 in_seq=0 out_seq=0 2024-03-18T22:49:09.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.761+0000 7f38caffd640 10 client.17226 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:49:09.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.762+0000 7f38caffd640 1 -- 172.21.15.40:0/3219609853 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f38e0011250 con 0x555d2b9326b0 2024-03-18T22:49:09.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.762+0000 7f38caffd640 10 client.17226 handle_command_reply: tid=0 2024-03-18T22:49:09.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.762+0000 7f38f4c1d740 1 client.17226 shutdown 2024-03-18T22:49:09.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.762+0000 7f38f4c1d740 1 -- 172.21.15.40:0/3219609853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91c2f0 msgr2=0x555d2b926800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.762+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/3219609853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91c2f0 0x555d2b926800 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f38e4032420 tx=0x7f38e40095c0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.762+0000 7f38f4c1d740 1 -- 172.21.15.40:0/3219609853 shutdown_connections 2024-03-18T22:49:09.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.762+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/3219609853 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d2b9326b0 0x555d2b934b00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.762+0000 7f38caffd640 0 client.17226 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:49:09.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.762+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/3219609853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b921e50 0x555d2b92dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.762+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/3219609853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b91cce0 0x555d2b926d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.762+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/3219609853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91c2f0 0x555d2b926800 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.762+0000 7f38f4c1d740 1 -- 172.21.15.40:0/3219609853 >> 172.21.15.40:0/3219609853 conn(0x555d2b917860 msgr2=0x555d2b916ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.762+0000 7f38f4c1d740 1 -- 172.21.15.40:0/3219609853 shutdown_connections 2024-03-18T22:49:09.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.762+0000 7f38f4c1d740 1 -- 172.21.15.40:0/3219609853 wait complete. 2024-03-18T22:49:09.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.762+0000 7f38f4c1d740 20 client.17226 trim_cache size 0 max 16384 2024-03-18T22:49:09.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.776+0000 7f38f4c1d740 1 Processor -- start 2024-03-18T22:49:09.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.776+0000 7f38f4c1d740 1 -- start start 2024-03-18T22:49:09.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.776+0000 7f38f4c1d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91c2f0 0x555d2b926b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.776+0000 7f38f4c1d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b91cce0 0x555d2b92dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.776+0000 7f38f4c1d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b921e50 0x555d2b916310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.776+0000 7f38f4c1d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f38ec1ac6b0 con 0x555d2b91c2f0 2024-03-18T22:49:09.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.776+0000 7f38f4c1d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f38ec1ae1f0 con 0x555d2b91cce0 2024-03-18T22:49:09.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.776+0000 7f38f4c1d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f38ec1acd80 con 0x555d2b921e50 2024-03-18T22:49:09.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.776+0000 7f38ebfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91c2f0 0x555d2b926b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.776+0000 7f38ebfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91c2f0 0x555d2b926b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51870/0 (socket says 172.21.15.40:51870) 2024-03-18T22:49:09.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.776+0000 7f38eaffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b91cce0 0x555d2b92dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.776+0000 7f38ebfff640 1 -- 172.21.15.40:0/2161893312 learned_addr learned my addr 172.21.15.40:0/2161893312 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.777+0000 7f38ebfff640 1 --2- 172.21.15.40:0/2161893312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b921e50 0x555d2b916310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.777+0000 7f38ebfff640 1 -- 172.21.15.40:0/2161893312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b921e50 msgr2=0x555d2b916310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.777+0000 7f38ebfff640 1 --2- 172.21.15.40:0/2161893312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b921e50 0x555d2b916310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.777+0000 7f38ebfff640 1 -- 172.21.15.40:0/2161893312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b91cce0 msgr2=0x555d2b92dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.777+0000 7f38ebfff640 1 --2- 172.21.15.40:0/2161893312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b91cce0 0x555d2b92dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.777+0000 7f38ebfff640 1 -- 172.21.15.40:0/2161893312 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d2b8ed880 con 0x555d2b91c2f0 2024-03-18T22:49:09.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.777+0000 7f38eaffd640 1 --2- 172.21.15.40:0/2161893312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b91cce0 0x555d2b92dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.777+0000 7f38ebfff640 1 --2- 172.21.15.40:0/2161893312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91c2f0 0x555d2b926b20 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f38e4000f80 tx=0x7f38e40324f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1687afd9f60f3b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.777+0000 7f38d17fa640 1 -- 172.21.15.40:0/2161893312 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e405d070 con 0x555d2b91c2f0 2024-03-18T22:49:09.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.777+0000 7f38d17fa640 1 -- 172.21.15.40:0/2161893312 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f38e404b040 con 0x555d2b91c2f0 2024-03-18T22:49:09.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.777+0000 7f38d17fa640 1 -- 172.21.15.40:0/2161893312 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e4039070 con 0x555d2b91c2f0 2024-03-18T22:49:09.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.778+0000 7f38f4c1d740 1 -- 172.21.15.40:0/2161893312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91c2f0 msgr2=0x555d2b926b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.778+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/2161893312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91c2f0 0x555d2b926b20 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f38e4000f80 tx=0x7f38e40324f0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.778+0000 7f38f4c1d740 1 -- 172.21.15.40:0/2161893312 shutdown_connections 2024-03-18T22:49:09.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.778+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/2161893312 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b921e50 0x555d2b916310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.778+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/2161893312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b91cce0 0x555d2b92dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.778+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/2161893312 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91c2f0 0x555d2b926b20 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.778+0000 7f38f4c1d740 1 -- 172.21.15.40:0/2161893312 >> 172.21.15.40:0/2161893312 conn(0x555d2b917860 msgr2=0x555d2b926410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.778+0000 7f38f4c1d740 1 -- 172.21.15.40:0/2161893312 shutdown_connections 2024-03-18T22:49:09.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.778+0000 7f38f4c1d740 1 -- 172.21.15.40:0/2161893312 wait complete. 2024-03-18T22:49:09.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.778+0000 7f38f4c1d740 1 -- start start 2024-03-18T22:49:09.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38f4c1d740 1 Processor -- start 2024-03-18T22:49:09.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38f4c1d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b91c2f0 0x555d2b92b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38f4c1d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91cce0 0x555d2b92b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38f4c1d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b921e50 0x555d2b92c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38f4c1d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f38ec1adb20 con 0x555d2b91cce0 2024-03-18T22:49:09.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38f4c1d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f38ec1ad450 con 0x555d2b91c2f0 2024-03-18T22:49:09.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38f4c1d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f38ec1aa690 con 0x555d2b921e50 2024-03-18T22:49:09.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38f4c1d740 10 client.0 _ll_register_callbacks cb 0x7f38e4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:09.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38eaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91cce0 0x555d2b92b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38ebfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b91c2f0 0x555d2b92b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38eaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91cce0 0x555d2b92b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51872/0 (socket says 172.21.15.40:51872) 2024-03-18T22:49:09.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38eaffd640 1 -- 172.21.15.40:0/2787930981 learned_addr learned my addr 172.21.15.40:0/2787930981 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38ebfff640 1 --2- 172.21.15.40:0/2787930981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b921e50 0x555d2b92c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38eaffd640 1 -- 172.21.15.40:0/2787930981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b921e50 msgr2=0x555d2b92c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38eaffd640 1 --2- 172.21.15.40:0/2787930981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b921e50 0x555d2b92c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38eaffd640 1 -- 172.21.15.40:0/2787930981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b91c2f0 msgr2=0x555d2b92b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38eaffd640 1 --2- 172.21.15.40:0/2787930981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b91c2f0 0x555d2b92b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38eaffd640 1 -- 172.21.15.40:0/2787930981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d2b92cd80 con 0x555d2b91cce0 2024-03-18T22:49:09.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38ebfff640 1 --2- 172.21.15.40:0/2787930981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b91c2f0 0x555d2b92b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38eaffd640 1 --2- 172.21.15.40:0/2787930981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91cce0 0x555d2b92b7d0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f38e0014040 tx=0x7f38e00110c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=329306985ae0a8c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.779+0000 7f38caffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:09.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.780+0000 7f38caffd640 1 -- 172.21.15.40:0/2787930981 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e001e040 con 0x555d2b91cce0 2024-03-18T22:49:09.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.780+0000 7f38caffd640 1 -- 172.21.15.40:0/2787930981 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f38e0005240 con 0x555d2b91cce0 2024-03-18T22:49:09.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.780+0000 7f38f4c1d740 1 -- 172.21.15.40:0/2787930981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38e404b040 con 0x555d2b91cce0 2024-03-18T22:49:09.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.780+0000 7f38caffd640 1 -- 172.21.15.40:0/2787930981 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f38e003b050 con 0x555d2b91cce0 2024-03-18T22:49:09.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.780+0000 7f38f4c1d740 10 client.17232 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:09.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.780+0000 7f38f4c1d740 1 -- 172.21.15.40:0/2787930981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555d2b998a00 con 0x555d2b91cce0 2024-03-18T22:49:09.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.780+0000 7f38caffd640 1 -- 172.21.15.40:0/2787930981 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f38e404b040 con 0x555d2b91cce0 2024-03-18T22:49:09.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.780+0000 7f38caffd640 1 -- 172.21.15.40:0/2787930981 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f38e00035f0 con 0x555d2b91cce0 2024-03-18T22:49:09.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.780+0000 7f38f4c1d740 10 client.17232 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:09.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.780+0000 7f38f4c1d740 10 client.17232 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:49:09.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.780+0000 7f38f4c1d740 20 client.17232 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:09.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.780+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/2787930981 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d2b9326b0 0x555d2b99b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.780+0000 7f38f4c1d740 4 client.17232 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:49:09.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.780+0000 7f38f4c1d740 1 -- 172.21.15.40:0/2787930981 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555d2b8de3e0 con 0x555d2b9326b0 2024-03-18T22:49:09.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.780+0000 7f38ebfff640 1 --2- 172.21.15.40:0/2787930981 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d2b9326b0 0x555d2b99b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.781+0000 7f38caffd640 10 client.17232 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:49:09.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.781+0000 7f38ebfff640 1 --2- 172.21.15.40:0/2787930981 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d2b9326b0 0x555d2b99b110 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1ed3d0dd4d1c19cd server_cookie=385be0304e485ca1 in_seq=0 out_seq=0 2024-03-18T22:49:09.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.781+0000 7f38caffd640 1 -- 172.21.15.40:0/2787930981 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555d2b8de3e0 con 0x555d2b9326b0 2024-03-18T22:49:09.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.781+0000 7f38caffd640 10 client.17232 handle_command_reply: tid=0 2024-03-18T22:49:09.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.781+0000 7f38f4c1d740 1 client.17232 shutdown 2024-03-18T22:49:09.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.781+0000 7f38f4c1d740 1 -- 172.21.15.40:0/2787930981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91cce0 msgr2=0x555d2b92b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.781+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/2787930981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91cce0 0x555d2b92b7d0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f38e0014040 tx=0x7f38e00110c0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.781+0000 7f38f4c1d740 1 -- 172.21.15.40:0/2787930981 shutdown_connections 2024-03-18T22:49:09.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.781+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/2787930981 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d2b9326b0 0x555d2b99b110 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.781+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/2787930981 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d2b921e50 0x555d2b92c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.781+0000 7f38caffd640 0 client.17232 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:49:09.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.781+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/2787930981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d2b91cce0 0x555d2b92b7d0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.781+0000 7f38f4c1d740 1 --2- 172.21.15.40:0/2787930981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d2b91c2f0 0x555d2b92b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.781+0000 7f38f4c1d740 1 -- 172.21.15.40:0/2787930981 >> 172.21.15.40:0/2787930981 conn(0x555d2b917860 msgr2=0x555d2b91aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.781+0000 7f38f4c1d740 1 -- 172.21.15.40:0/2787930981 shutdown_connections 2024-03-18T22:49:09.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.782+0000 7f38f4c1d740 1 -- 172.21.15.40:0/2787930981 wait complete. 2024-03-18T22:49:09.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.782+0000 7f38f4c1d740 20 client.17232 trim_cache size 0 max 16384 2024-03-18T22:49:09.788 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:49:09.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.782+0000 7f38f1f53640 1 -- 172.21.15.40:0/2899778248 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f38d40a2800 msgr2=0x7f38d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.782+0000 7f38f1f53640 1 --2- 172.21.15.40:0/2899778248 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f38d40a2800 0x7f38d40a4cc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f38dc0cc9a0 tx=0x7f38dc097b90 comp rx=0 tx=0).stop 2024-03-18T22:49:09.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.782+0000 7f38f1f53640 1 -- 172.21.15.40:0/2899778248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38ec100720 msgr2=0x7f38ec104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.782+0000 7f38f1f53640 1 --2- 172.21.15.40:0/2899778248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38ec100720 0x7f38ec104780 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f38dc00e9c0 tx=0x7f38dc00ee80 comp rx=0 tx=0).stop 2024-03-18T22:49:09.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.782+0000 7f38f1f53640 1 -- 172.21.15.40:0/2899778248 shutdown_connections 2024-03-18T22:49:09.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.782+0000 7f38f1f53640 1 --2- 172.21.15.40:0/2899778248 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f38d40a2800 0x7f38d40a4cc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.782+0000 7f38f1f53640 1 --2- 172.21.15.40:0/2899778248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f38ec1022d0 0x7f38ec103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.782+0000 7f38f1f53640 1 --2- 172.21.15.40:0/2899778248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38ec101910 0x7f38ec102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.782+0000 7f38f1f53640 1 --2- 172.21.15.40:0/2899778248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38ec100720 0x7f38ec104780 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.782+0000 7f38f1f53640 1 -- 172.21.15.40:0/2899778248 >> 172.21.15.40:0/2899778248 conn(0x7f38ec0fbc90 msgr2=0x7f38ec068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.782+0000 7f38f1f53640 1 -- 172.21.15.40:0/2899778248 shutdown_connections 2024-03-18T22:49:09.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.782+0000 7f38f1f53640 1 -- 172.21.15.40:0/2899778248 wait complete. 2024-03-18T22:49:09.794 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:49:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.912+0000 7f9cccfdf640 1 -- 172.21.15.40:0/4042282699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cc81009a0 msgr2=0x7f9cc8100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.912+0000 7f9cccfdf640 1 --2- 172.21.15.40:0/4042282699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cc81009a0 0x7f9cc8100e20 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc009900 tx=0x7f9cbc02bc80 comp rx=0 tx=0).stop 2024-03-18T22:49:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.912+0000 7f9cccfdf640 1 -- 172.21.15.40:0/4042282699 shutdown_connections 2024-03-18T22:49:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.912+0000 7f9cccfdf640 1 --2- 172.21.15.40:0/4042282699 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cc8101360 0x7f9cc8108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.912+0000 7f9cccfdf640 1 --2- 172.21.15.40:0/4042282699 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cc81009a0 0x7f9cc8100e20 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.912+0000 7f9cccfdf640 1 --2- 172.21.15.40:0/4042282699 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc8101de0 0x7f9cc8102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.912+0000 7f9cccfdf640 1 -- 172.21.15.40:0/4042282699 >> 172.21.15.40:0/4042282699 conn(0x7f9cc80fb450 msgr2=0x7f9cc80fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.912+0000 7f9cccfdf640 1 -- 172.21.15.40:0/4042282699 shutdown_connections 2024-03-18T22:49:09.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.912+0000 7f9cccfdf640 1 -- 172.21.15.40:0/4042282699 wait complete. 2024-03-18T22:49:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.913+0000 7f9cccfdf640 1 Processor -- start 2024-03-18T22:49:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.913+0000 7f9cccfdf640 1 -- start start 2024-03-18T22:49:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.913+0000 7f9cccfdf640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cc81009a0 0x7f9cc8104750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.913+0000 7f9cccfdf640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc8101360 0x7f9cc8102d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.913+0000 7f9cccfdf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cc8101de0 0x7f9cc81032d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.913+0000 7f9cccfdf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc810c2b0 con 0x7f9cc8101de0 2024-03-18T22:49:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.913+0000 7f9cccfdf640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc810c130 con 0x7f9cc8101360 2024-03-18T22:49:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.913+0000 7f9cccfdf640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc810c430 con 0x7f9cc81009a0 2024-03-18T22:49:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.913+0000 7f9cc6d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cc8101de0 0x7f9cc81032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.913+0000 7f9cc6d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cc8101de0 0x7f9cc81032d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51894/0 (socket says 172.21.15.40:51894) 2024-03-18T22:49:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.913+0000 7f9cc659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cc81009a0 0x7f9cc8104750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.913+0000 7f9cc6d9d640 1 -- 172.21.15.40:0/4111330713 learned_addr learned my addr 172.21.15.40:0/4111330713 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.913+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/4111330713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc8101360 0x7f9cc8102d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.914+0000 7f9cc6d9d640 1 -- 172.21.15.40:0/4111330713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cc81009a0 msgr2=0x7f9cc8104750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.914+0000 7f9cc6d9d640 1 --2- 172.21.15.40:0/4111330713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cc81009a0 0x7f9cc8104750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.914+0000 7f9cc6d9d640 1 -- 172.21.15.40:0/4111330713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc8101360 msgr2=0x7f9cc8102d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.914+0000 7f9cc6d9d640 1 --2- 172.21.15.40:0/4111330713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc8101360 0x7f9cc8102d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.914+0000 7f9cc6d9d640 1 -- 172.21.15.40:0/4111330713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc8103810 con 0x7f9cc8101de0 2024-03-18T22:49:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.914+0000 7f9cc659c640 1 --2- 172.21.15.40:0/4111330713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cc81009a0 0x7f9cc8104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.914+0000 7f9cc6d9d640 1 --2- 172.21.15.40:0/4111330713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cc8101de0 0x7f9cc81032d0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f9cc000eb10 tx=0x7f9cc000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.914+0000 7f9cb77fe640 1 -- 172.21.15.40:0/4111330713 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc0004b20 con 0x7f9cc8101de0 2024-03-18T22:49:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.914+0000 7f9cb77fe640 1 -- 172.21.15.40:0/4111330713 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9cc0004cc0 con 0x7f9cc8101de0 2024-03-18T22:49:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.914+0000 7f9cccfdf640 1 -- 172.21.15.40:0/4111330713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc8103b00 con 0x7f9cc8101de0 2024-03-18T22:49:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.914+0000 7f9cb77fe640 1 -- 172.21.15.40:0/4111330713 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc0010690 con 0x7f9cc8101de0 2024-03-18T22:49:09.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.914+0000 7f9cccfdf640 1 -- 172.21.15.40:0/4111330713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc81a6510 con 0x7f9cc8101de0 2024-03-18T22:49:09.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.916+0000 7f9cb77fe640 1 -- 172.21.15.40:0/4111330713 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9cc0005020 con 0x7f9cc8101de0 2024-03-18T22:49:09.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.917+0000 7f9ccfbc7740 1 Processor -- start 2024-03-18T22:49:09.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.917+0000 7f9ccfbc7740 1 -- start start 2024-03-18T22:49:09.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.917+0000 7f9cb77fe640 1 --2- 172.21.15.40:0/4111330713 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9cb00a2800 0x7f9cb00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.917+0000 7f9ccfbc7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4eb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.917+0000 7f9ccfbc7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4f07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.917+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/4111330713 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9cb00a2800 0x7f9cb00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.917+0000 7f9cb77fe640 1 -- 172.21.15.40:0/4111330713 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9cc00cf500 con 0x7f9cc8101de0 2024-03-18T22:49:09.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cc659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4eb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9ccfbc7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cc659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4eb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51900/0 (socket says 172.21.15.40:51900) 2024-03-18T22:49:09.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/2446043704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cf3f44f4b0 con 0x55cf3f4eb2f0 2024-03-18T22:49:09.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cc659c640 1 -- 172.21.15.40:0/2446043704 learned_addr learned my addr 172.21.15.40:0/2446043704 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/2446043704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4f07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/2446043704 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cf3f3f7d70 con 0x55cf3f4ebce0 2024-03-18T22:49:09.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/2446043704 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55cf3f44b550 con 0x55cf3f4f0e50 2024-03-18T22:49:09.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cc6d9d640 1 --2- 172.21.15.40:0/2446043704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cc5d9b640 1 -- 172.21.15.40:0/2446043704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 msgr2=0x55cf3f4f32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/2446043704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4f32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cc5d9b640 1 -- 172.21.15.40:0/2446043704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4eb2f0 msgr2=0x55cf3f4eb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/2446043704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4eb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cc5d9b640 1 -- 172.21.15.40:0/2446043704 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf3f41a210 con 0x55cf3f4ebce0 2024-03-18T22:49:09.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cc6d9d640 1 --2- 172.21.15.40:0/2446043704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cc659c640 1 --2- 172.21.15.40:0/2446043704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4eb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/4111330713 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9cb00a2800 0x7f9cb00a4cc0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc005ab0 tx=0x7f9cbc005de0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/2446043704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4f07e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc036040 tx=0x7f9cbc047000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73e7b69e936fa473 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cb4ff9640 1 -- 172.21.15.40:0/2446043704 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc005100 con 0x55cf3f4ebce0 2024-03-18T22:49:09.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cb4ff9640 1 -- 172.21.15.40:0/2446043704 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9cbc0046c0 con 0x55cf3f4ebce0 2024-03-18T22:49:09.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.918+0000 7f9cb4ff9640 1 -- 172.21.15.40:0/2446043704 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc003830 con 0x55cf3f4ebce0 2024-03-18T22:49:09.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.919+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/2446043704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4ebce0 msgr2=0x55cf3f4f07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.919+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/2446043704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4f07e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc036040 tx=0x7f9cbc047000 comp rx=0 tx=0).stop 2024-03-18T22:49:09.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.919+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/2446043704 shutdown_connections 2024-03-18T22:49:09.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.919+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/2446043704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.919+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/2446043704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4f07e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.919+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/2446043704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4eb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.919+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/2446043704 >> 172.21.15.40:0/2446043704 conn(0x55cf3f4e6860 msgr2=0x55cf3f4e6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.919+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/2446043704 shutdown_connections 2024-03-18T22:49:09.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.919+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/2446043704 wait complete. 2024-03-18T22:49:09.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9ccfbc7740 1 -- start start 2024-03-18T22:49:09.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9ccfbc7740 1 Processor -- start 2024-03-18T22:49:09.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9ccfbc7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4f5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9ccfbc7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9ccfbc7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4fcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9ccfbc7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cf3f4f3da0 con 0x55cf3f4ebce0 2024-03-18T22:49:09.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9ccfbc7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cf3f40b310 con 0x55cf3f4eb2f0 2024-03-18T22:49:09.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9ccfbc7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55cf3f4f3f20 con 0x55cf3f4f0e50 2024-03-18T22:49:09.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9ccfbc7740 10 client.0 _ll_register_callbacks cb 0x55cf3f3ff2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:09.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9cc5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9cc5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4f5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51908/0 (socket says 172.21.15.40:51908) 2024-03-18T22:49:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9cc5d9b640 1 -- 172.21.15.40:0/3859536037 learned_addr learned my addr 172.21.15.40:0/3859536037 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9cc6d9d640 1 --2- 172.21.15.40:0/3859536037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4fcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9cc659c640 1 --2- 172.21.15.40:0/3859536037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4f5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9cc5d9b640 1 -- 172.21.15.40:0/3859536037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 msgr2=0x55cf3f4fcd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/3859536037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4fcd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9cc5d9b640 1 -- 172.21.15.40:0/3859536037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4eb2f0 msgr2=0x55cf3f4f5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/3859536037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4f5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.920+0000 7f9cc5d9b640 1 -- 172.21.15.40:0/3859536037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf3f1633f0 con 0x55cf3f4ebce0 2024-03-18T22:49:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.921+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/3859536037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4f5d40 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc02c830 tx=0x7f9cbc004160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e806f1049f5ab8f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.921+0000 7f9cadffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.921+0000 7f9cadffb640 1 -- 172.21.15.40:0/3859536037 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc004a80 con 0x55cf3f4ebce0 2024-03-18T22:49:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.921+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/3859536037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cbc0046c0 con 0x55cf3f4ebce0 2024-03-18T22:49:09.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.921+0000 7f9cadffb640 1 -- 172.21.15.40:0/3859536037 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9cbc005100 con 0x55cf3f4ebce0 2024-03-18T22:49:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.921+0000 7f9cadffb640 1 -- 172.21.15.40:0/3859536037 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9cbc066050 con 0x55cf3f4ebce0 2024-03-18T22:49:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.921+0000 7f9ccfbc7740 10 client.17244 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.921+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/3859536037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cf3f22d7d0 con 0x55cf3f4ebce0 2024-03-18T22:49:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.921+0000 7f9cadffb640 1 -- 172.21.15.40:0/3859536037 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9cbc0046c0 con 0x55cf3f4ebce0 2024-03-18T22:49:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.921+0000 7f9cadffb640 1 -- 172.21.15.40:0/3859536037 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9cbc003ca0 con 0x55cf3f4ebce0 2024-03-18T22:49:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.921+0000 7f9ccfbc7740 10 client.17244 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.921+0000 7f9ccfbc7740 10 client.17244 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:49:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.922+0000 7f9ccfbc7740 20 client.17244 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.922+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/3859536037 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55cf3f501270 0x55cf3f503660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.922+0000 7f9ccfbc7740 4 client.17244 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.922+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/3859536037 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf3f4ad3e0 con 0x55cf3f501270 2024-03-18T22:49:09.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.922+0000 7f9cc659c640 1 --2- 172.21.15.40:0/3859536037 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55cf3f501270 0x55cf3f503660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.922+0000 7f9cc659c640 1 --2- 172.21.15.40:0/3859536037 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55cf3f501270 0x55cf3f503660 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=992ecb3b39889a5f server_cookie=c4ca2a2d5608925a in_seq=0 out_seq=0 2024-03-18T22:49:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.922+0000 7f9cadffb640 10 client.17244 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:49:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.923+0000 7f9cadffb640 1 -- 172.21.15.40:0/3859536037 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cf3f4ad3e0 con 0x55cf3f501270 2024-03-18T22:49:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.923+0000 7f9cadffb640 10 client.17244 handle_command_reply: tid=0 2024-03-18T22:49:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.923+0000 7f9ccfbc7740 1 client.17244 shutdown 2024-03-18T22:49:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.923+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/3859536037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4ebce0 msgr2=0x55cf3f4f5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.923+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/3859536037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4f5d40 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc02c830 tx=0x7f9cbc004160 comp rx=0 tx=0).stop 2024-03-18T22:49:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.923+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/3859536037 shutdown_connections 2024-03-18T22:49:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.923+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/3859536037 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55cf3f501270 0x55cf3f503660 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.923+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/3859536037 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.923+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/3859536037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4f5d40 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.923+0000 7f9cadffb640 0 client.17244 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:49:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.923+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/3859536037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4f5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.923+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/3859536037 >> 172.21.15.40:0/3859536037 conn(0x55cf3f4e6860 msgr2=0x55cf3f4e5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.923+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/3859536037 shutdown_connections 2024-03-18T22:49:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.923+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/3859536037 wait complete. 2024-03-18T22:49:09.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.924+0000 7f9ccfbc7740 20 client.17244 trim_cache size 0 max 16384 2024-03-18T22:49:09.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9ccfbc7740 1 Processor -- start 2024-03-18T22:49:09.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9ccfbc7740 1 -- start start 2024-03-18T22:49:09.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9ccfbc7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4ed240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9ccfbc7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4ed780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9ccfbc7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4fcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9ccfbc7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc81b0b10 con 0x55cf3f4ebce0 2024-03-18T22:49:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9ccfbc7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc81b2650 con 0x55cf3f4eb2f0 2024-03-18T22:49:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9ccfbc7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc81b11e0 con 0x55cf3f4f0e50 2024-03-18T22:49:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9cc5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4ed780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9cc5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4ed780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51916/0 (socket says 172.21.15.40:51916) 2024-03-18T22:49:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9cc5d9b640 1 -- 172.21.15.40:0/3208148843 learned_addr learned my addr 172.21.15.40:0/3208148843 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9cc6d9d640 1 --2- 172.21.15.40:0/3208148843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4fcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9cc659c640 1 --2- 172.21.15.40:0/3208148843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4ed240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9cc5d9b640 1 -- 172.21.15.40:0/3208148843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 msgr2=0x55cf3f4fcd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/3208148843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4fcd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9cc5d9b640 1 -- 172.21.15.40:0/3208148843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4eb2f0 msgr2=0x55cf3f4ed240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/3208148843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4ed240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9cc5d9b640 1 -- 172.21.15.40:0/3208148843 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf3f503c40 con 0x55cf3f4ebce0 2024-03-18T22:49:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.939+0000 7f9cc6d9d640 1 --2- 172.21.15.40:0/3208148843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4fcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.940+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/3208148843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4ed780 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc02c1c0 tx=0x7f9cbc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f226ccafd9fa4db server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.940+0000 7f9caffff640 1 -- 172.21.15.40:0/3208148843 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc02c4a0 con 0x55cf3f4ebce0 2024-03-18T22:49:09.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.940+0000 7f9caffff640 1 -- 172.21.15.40:0/3208148843 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9cbc05d070 con 0x55cf3f4ebce0 2024-03-18T22:49:09.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.940+0000 7f9caffff640 1 -- 172.21.15.40:0/3208148843 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc058970 con 0x55cf3f4ebce0 2024-03-18T22:49:09.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/3208148843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4ebce0 msgr2=0x55cf3f4ed780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/3208148843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4ed780 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc02c1c0 tx=0x7f9cbc009780 comp rx=0 tx=0).stop 2024-03-18T22:49:09.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/3208148843 shutdown_connections 2024-03-18T22:49:09.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/3208148843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4fcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/3208148843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4ed780 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/3208148843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4ed240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/3208148843 >> 172.21.15.40:0/3208148843 conn(0x55cf3f4e6860 msgr2=0x55cf3f4e6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/3208148843 shutdown_connections 2024-03-18T22:49:09.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/3208148843 wait complete. 2024-03-18T22:49:09.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 -- start start 2024-03-18T22:49:09.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 Processor -- start 2024-03-18T22:49:09.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4fa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4fae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4fbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc81b1f80 con 0x55cf3f4eb2f0 2024-03-18T22:49:09.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc81b18b0 con 0x55cf3f4f0e50 2024-03-18T22:49:09.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc81aeaf0 con 0x55cf3f4ebce0 2024-03-18T22:49:09.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9ccfbc7740 10 client.0 _ll_register_callbacks cb 0x7f9cb8009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:09.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9cc659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4fa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9cc659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4fa8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51926/0 (socket says 172.21.15.40:51926) 2024-03-18T22:49:09.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9cc659c640 1 -- 172.21.15.40:0/1306926020 learned_addr learned my addr 172.21.15.40:0/1306926020 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:09.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.941+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/1306926020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4fae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.942+0000 7f9cc6d9d640 1 --2- 172.21.15.40:0/1306926020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4fbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.942+0000 7f9cc659c640 1 -- 172.21.15.40:0/1306926020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4ebce0 msgr2=0x55cf3f4fae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.942+0000 7f9cc659c640 1 --2- 172.21.15.40:0/1306926020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4fae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.942+0000 7f9cc659c640 1 -- 172.21.15.40:0/1306926020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4f0e50 msgr2=0x55cf3f4fbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:09.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.942+0000 7f9cc659c640 1 --2- 172.21.15.40:0/1306926020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4fbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.942+0000 7f9cc659c640 1 -- 172.21.15.40:0/1306926020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf3f4bc880 con 0x55cf3f4eb2f0 2024-03-18T22:49:09.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.942+0000 7f9cc6d9d640 1 --2- 172.21.15.40:0/1306926020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4fbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:09.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.942+0000 7f9cc659c640 1 --2- 172.21.15.40:0/1306926020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4fa8e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x55cf3f4e77d0 tx=0x7f9cb801f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd9f09d93205f511 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:09.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.942+0000 7f9cadffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:09.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.942+0000 7f9cadffb640 1 -- 172.21.15.40:0/1306926020 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8028440 con 0x55cf3f4eb2f0 2024-03-18T22:49:09.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.942+0000 7f9cadffb640 1 -- 172.21.15.40:0/1306926020 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9cb8010430 con 0x55cf3f4eb2f0 2024-03-18T22:49:09.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.942+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/1306926020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cbc05d070 con 0x55cf3f4eb2f0 2024-03-18T22:49:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.942+0000 7f9cadffb640 1 -- 172.21.15.40:0/1306926020 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9cb8010630 con 0x55cf3f4eb2f0 2024-03-18T22:49:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.942+0000 7f9ccfbc7740 10 client.17250 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.942+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/1306926020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cf3f567750 con 0x55cf3f4eb2f0 2024-03-18T22:49:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.943+0000 7f9cadffb640 1 -- 172.21.15.40:0/1306926020 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9cbc05d070 con 0x55cf3f4eb2f0 2024-03-18T22:49:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.943+0000 7f9cadffb640 1 -- 172.21.15.40:0/1306926020 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f9cb8026070 con 0x55cf3f4eb2f0 2024-03-18T22:49:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.943+0000 7f9ccfbc7740 10 client.17250 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.943+0000 7f9ccfbc7740 10 client.17250 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:49:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.943+0000 7f9ccfbc7740 20 client.17250 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.943+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/1306926020 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55cf3f501270 0x55cf3f56a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.943+0000 7f9ccfbc7740 4 client.17250 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:49:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.943+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/1306926020 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cf3f45c9f0 con 0x55cf3f501270 2024-03-18T22:49:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.943+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/1306926020 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55cf3f501270 0x55cf3f56a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.943+0000 7f9cadffb640 10 client.17250 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:49:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.943+0000 7f9cc5d9b640 1 --2- 172.21.15.40:0/1306926020 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55cf3f501270 0x55cf3f56a210 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=334acf459bb13280 server_cookie=62275ecb04e4f49b in_seq=0 out_seq=0 2024-03-18T22:49:09.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.943+0000 7f9cadffb640 1 -- 172.21.15.40:0/1306926020 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cf3f45c9f0 con 0x55cf3f501270 2024-03-18T22:49:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.943+0000 7f9cadffb640 10 client.17250 handle_command_reply: tid=0 2024-03-18T22:49:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.943+0000 7f9ccfbc7740 1 client.17250 shutdown 2024-03-18T22:49:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/1306926020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4eb2f0 msgr2=0x55cf3f4fa8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/1306926020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4fa8e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x55cf3f4e77d0 tx=0x7f9cb801f770 comp rx=0 tx=0).stop 2024-03-18T22:49:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/1306926020 shutdown_connections 2024-03-18T22:49:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/1306926020 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55cf3f501270 0x55cf3f56a210 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/1306926020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf3f4f0e50 0x55cf3f4fbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/1306926020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf3f4ebce0 0x55cf3f4fae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9cadffb640 0 client.17250 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:49:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9ccfbc7740 1 --2- 172.21.15.40:0/1306926020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf3f4eb2f0 0x55cf3f4fa8e0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/1306926020 >> 172.21.15.40:0/1306926020 conn(0x55cf3f4e6860 msgr2=0x55cf3f4ed240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/1306926020 shutdown_connections 2024-03-18T22:49:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9ccfbc7740 1 -- 172.21.15.40:0/1306926020 wait complete. 2024-03-18T22:49:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9ccfbc7740 20 client.17250 trim_cache size 0 max 16384 2024-03-18T22:49:09.951 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:49:09.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9cccfdf640 1 -- 172.21.15.40:0/4111330713 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9cb00a2800 msgr2=0x7f9cb00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9cccfdf640 1 --2- 172.21.15.40:0/4111330713 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9cb00a2800 0x7f9cb00a4cc0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc005ab0 tx=0x7f9cbc005de0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9cccfdf640 1 -- 172.21.15.40:0/4111330713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cc8101de0 msgr2=0x7f9cc81032d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:09.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9cccfdf640 1 --2- 172.21.15.40:0/4111330713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cc8101de0 0x7f9cc81032d0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f9cc000eb10 tx=0x7f9cc000c2d0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9cccfdf640 1 -- 172.21.15.40:0/4111330713 shutdown_connections 2024-03-18T22:49:09.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9cccfdf640 1 --2- 172.21.15.40:0/4111330713 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9cb00a2800 0x7f9cb00a4cc0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9cccfdf640 1 --2- 172.21.15.40:0/4111330713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cc8101de0 0x7f9cc81032d0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9cccfdf640 1 --2- 172.21.15.40:0/4111330713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cc8101360 0x7f9cc8102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9cccfdf640 1 --2- 172.21.15.40:0/4111330713 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9cc81009a0 0x7f9cc8104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:09.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.944+0000 7f9cccfdf640 1 -- 172.21.15.40:0/4111330713 >> 172.21.15.40:0/4111330713 conn(0x7f9cc80fb450 msgr2=0x7f9cc80fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:09.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.945+0000 7f9cccfdf640 1 -- 172.21.15.40:0/4111330713 shutdown_connections 2024-03-18T22:49:09.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:09.945+0000 7f9cccfdf640 1 -- 172.21.15.40:0/4111330713 wait complete. 2024-03-18T22:49:09.956 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:49:10.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.079+0000 7f1a6031d640 1 -- 172.21.15.40:0/4020264117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a58100760 msgr2=0x7f1a58100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:10.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.079+0000 7f1a6031d640 1 --2- 172.21.15.40:0/4020264117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a58100760 0x7f1a58100b80 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f1a54009900 tx=0x7f1a5402bc30 comp rx=0 tx=0).stop 2024-03-18T22:49:10.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.079+0000 7f1a6031d640 1 -- 172.21.15.40:0/4020264117 shutdown_connections 2024-03-18T22:49:10.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.079+0000 7f1a6031d640 1 --2- 172.21.15.40:0/4020264117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a58102310 0x7f1a58108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.079+0000 7f1a6031d640 1 --2- 172.21.15.40:0/4020264117 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a58101950 0x7f1a58101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.079+0000 7f1a6031d640 1 --2- 172.21.15.40:0/4020264117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a58100760 0x7f1a58100b80 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.079+0000 7f1a6031d640 1 -- 172.21.15.40:0/4020264117 >> 172.21.15.40:0/4020264117 conn(0x7f1a580fbcf0 msgr2=0x7f1a580fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:10.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.079+0000 7f1a6031d640 1 -- 172.21.15.40:0/4020264117 shutdown_connections 2024-03-18T22:49:10.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.079+0000 7f1a6031d640 1 -- 172.21.15.40:0/4020264117 wait complete. 2024-03-18T22:49:10.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.079+0000 7f1a6031d640 1 Processor -- start 2024-03-18T22:49:10.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.079+0000 7f1a6031d640 1 -- start start 2024-03-18T22:49:10.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a6031d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a58100760 0x7f1a581046a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a6031d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a58101950 0x7f1a58102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a6031d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a58102310 0x7f1a58103270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a6031d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5810c280 con 0x7f1a58102310 2024-03-18T22:49:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a6031d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5810c100 con 0x7f1a58100760 2024-03-18T22:49:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a6031d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5810c400 con 0x7f1a58101950 2024-03-18T22:49:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a5e8ba640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a58102310 0x7f1a58103270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a5d8b8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a58101950 0x7f1a58102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a5e8ba640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a58102310 0x7f1a58103270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51952/0 (socket says 172.21.15.40:51952) 2024-03-18T22:49:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a5e8ba640 1 -- 172.21.15.40:0/2460325862 learned_addr learned my addr 172.21.15.40:0/2460325862 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a5e0b9640 1 --2- 172.21.15.40:0/2460325862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a58100760 0x7f1a581046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a5e8ba640 1 -- 172.21.15.40:0/2460325862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a58101950 msgr2=0x7f1a58102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a5e8ba640 1 --2- 172.21.15.40:0/2460325862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a58101950 0x7f1a58102d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a5e8ba640 1 -- 172.21.15.40:0/2460325862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a58100760 msgr2=0x7f1a581046a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a5e8ba640 1 --2- 172.21.15.40:0/2460325862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a58100760 0x7f1a581046a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a5e8ba640 1 -- 172.21.15.40:0/2460325862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a58103b30 con 0x7f1a58102310 2024-03-18T22:49:10.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a5e8ba640 1 --2- 172.21.15.40:0/2460325862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a58102310 0x7f1a58103270 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000efd0 tx=0x7f1a5000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:10.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a4f7fe640 1 -- 172.21.15.40:0/2460325862 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a50003b70 con 0x7f1a58102310 2024-03-18T22:49:10.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a4f7fe640 1 -- 172.21.15.40:0/2460325862 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a50003d10 con 0x7f1a58102310 2024-03-18T22:49:10.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a6031d640 1 -- 172.21.15.40:0/2460325862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a581a1c50 con 0x7f1a58102310 2024-03-18T22:49:10.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a4f7fe640 1 -- 172.21.15.40:0/2460325862 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a50010980 con 0x7f1a58102310 2024-03-18T22:49:10.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.080+0000 7f1a6031d640 1 -- 172.21.15.40:0/2460325862 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a581a2170 con 0x7f1a58102310 2024-03-18T22:49:10.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.083+0000 7f1a4f7fe640 1 -- 172.21.15.40:0/2460325862 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1a50004020 con 0x7f1a58102310 2024-03-18T22:49:10.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.083+0000 7f1a62fe0740 1 Processor -- start 2024-03-18T22:49:10.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.083+0000 7f1a62fe0740 1 -- start start 2024-03-18T22:49:10.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a62fe0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ee8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a62fe0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1eed7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a62fe0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1eede50 0x55e6f1ef02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a62fe0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e6f1e4c4b0 con 0x55e6f1ee82f0 2024-03-18T22:49:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a62fe0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e6f1df4d70 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a62fe0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e6f1e48550 con 0x55e6f1eede50 2024-03-18T22:49:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a5d8b8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ee8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a5e0b9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1eed7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a5d8b8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ee8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51966/0 (socket says 172.21.15.40:51966) 2024-03-18T22:49:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a5d8b8640 1 -- 172.21.15.40:0/1020025270 learned_addr learned my addr 172.21.15.40:0/1020025270 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a5d8b8640 1 --2- 172.21.15.40:0/1020025270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1eede50 0x55e6f1ef02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a4f7fe640 1 --2- 172.21.15.40:0/2460325862 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a440a2880 0x7f1a440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a5e0b9640 1 -- 172.21.15.40:0/1020025270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1eede50 msgr2=0x55e6f1ef02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:10.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a5e0b9640 1 --2- 172.21.15.40:0/1020025270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1eede50 0x55e6f1ef02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a5e0b9640 1 -- 172.21.15.40:0/1020025270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee82f0 msgr2=0x55e6f1ee8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:10.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a5e0b9640 1 --2- 172.21.15.40:0/1020025270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ee8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a5e0b9640 1 -- 172.21.15.40:0/1020025270 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6f1e17210 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a5e8ba640 1 --2- 172.21.15.40:0/2460325862 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a440a2880 0x7f1a440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a4f7fe640 1 -- 172.21.15.40:0/2460325862 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1a500cf230 con 0x7f1a58102310 2024-03-18T22:49:10.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.084+0000 7f1a5e0b9640 1 --2- 172.21.15.40:0/1020025270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1eed7e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1a54036040 tx=0x7f1a54009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37ac81afde3b4948 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:10.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.085+0000 7f1a4cbfe640 1 -- 172.21.15.40:0/1020025270 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a54045020 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.085+0000 7f1a4cbfe640 1 -- 172.21.15.40:0/1020025270 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a54004040 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.085+0000 7f1a4cbfe640 1 -- 172.21.15.40:0/1020025270 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a540046c0 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.085+0000 7f1a62fe0740 1 -- 172.21.15.40:0/1020025270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee8ce0 msgr2=0x55e6f1eed7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.085+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/1020025270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1eed7e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1a54036040 tx=0x7f1a54009560 comp rx=0 tx=0).stop 2024-03-18T22:49:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.085+0000 7f1a5e8ba640 1 --2- 172.21.15.40:0/2460325862 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a440a2880 0x7f1a440a4d40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1a500981b0 tx=0x7f1a50009f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 1 -- 172.21.15.40:0/1020025270 shutdown_connections 2024-03-18T22:49:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/1020025270 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1eede50 0x55e6f1ef02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/1020025270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1eed7e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/1020025270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ee8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 1 -- 172.21.15.40:0/1020025270 >> 172.21.15.40:0/1020025270 conn(0x55e6f1ee3860 msgr2=0x55e6f1ee3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:10.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 1 -- 172.21.15.40:0/1020025270 shutdown_connections 2024-03-18T22:49:10.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 1 -- 172.21.15.40:0/1020025270 wait complete. 2024-03-18T22:49:10.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 1 -- start start 2024-03-18T22:49:10.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 1 Processor -- start 2024-03-18T22:49:10.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ef2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1eede50 0x55e6f1ef9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e6f1ef0da0 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e6f1e08310 con 0x55e6f1ee82f0 2024-03-18T22:49:10.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e6f1ef0f20 con 0x55e6f1eede50 2024-03-18T22:49:10.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a62fe0740 10 client.0 _ll_register_callbacks cb 0x55e6f1dfc2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:10.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a5e0b9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a5e0b9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51972/0 (socket says 172.21.15.40:51972) 2024-03-18T22:49:10.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.086+0000 7f1a5e0b9640 1 -- 172.21.15.40:0/3996101222 learned_addr learned my addr 172.21.15.40:0/3996101222 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:10.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.087+0000 7f1a5d8b8640 1 --2- 172.21.15.40:0/3996101222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ef2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.087+0000 7f1a5d8b8640 1 --2- 172.21.15.40:0/3996101222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1eede50 0x55e6f1ef9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.087+0000 7f1a5e0b9640 1 -- 172.21.15.40:0/3996101222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1eede50 msgr2=0x55e6f1ef9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:10.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.087+0000 7f1a5e0b9640 1 --2- 172.21.15.40:0/3996101222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1eede50 0x55e6f1ef9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.087+0000 7f1a5e0b9640 1 -- 172.21.15.40:0/3996101222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee82f0 msgr2=0x55e6f1ef2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:10.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.087+0000 7f1a5e0b9640 1 --2- 172.21.15.40:0/3996101222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ef2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.087+0000 7f1a5e0b9640 1 -- 172.21.15.40:0/3996101222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6f1b603f0 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.087+0000 7f1a5e0b9640 1 --2- 172.21.15.40:0/3996101222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef2cc0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1a5402c170 tx=0x7f1a54004490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178cc3e4d6ef2901 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:10.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.087+0000 7f1a41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:10.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.087+0000 7f1a41ffb640 1 -- 172.21.15.40:0/3996101222 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a540035f0 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.087+0000 7f1a41ffb640 1 -- 172.21.15.40:0/3996101222 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1a54003790 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.087+0000 7f1a62fe0740 1 -- 172.21.15.40:0/3996101222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a54004040 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.087+0000 7f1a41ffb640 1 -- 172.21.15.40:0/3996101222 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1a5405b050 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.088+0000 7f1a62fe0740 10 client.17262 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.088+0000 7f1a62fe0740 1 -- 172.21.15.40:0/3996101222 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e6f1c2a7d0 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.088+0000 7f1a41ffb640 1 -- 172.21.15.40:0/3996101222 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a54004040 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.088+0000 7f1a41ffb640 1 -- 172.21.15.40:0/3996101222 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1a54004ca0 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.088+0000 7f1a62fe0740 10 client.17262 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.088+0000 7f1a62fe0740 10 client.17262 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.088+0000 7f1a62fe0740 20 client.17262 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.088+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/3996101222 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e6f1efe750 0x55e6f1f00bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.088+0000 7f1a62fe0740 4 client.17262 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.088+0000 7f1a62fe0740 1 -- 172.21.15.40:0/3996101222 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6f1eaa3e0 con 0x55e6f1efe750 2024-03-18T22:49:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.089+0000 7f1a5d8b8640 1 --2- 172.21.15.40:0/3996101222 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e6f1efe750 0x55e6f1f00bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.089+0000 7f1a41ffb640 10 client.17262 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.089+0000 7f1a5d8b8640 1 --2- 172.21.15.40:0/3996101222 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e6f1efe750 0x55e6f1f00bb0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=619c132a94edf627 server_cookie=7d43f723fd1ea524 in_seq=0 out_seq=0 2024-03-18T22:49:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.090+0000 7f1a41ffb640 1 -- 172.21.15.40:0/3996101222 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e6f1eaa3e0 con 0x55e6f1efe750 2024-03-18T22:49:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.090+0000 7f1a41ffb640 10 client.17262 handle_command_reply: tid=0 2024-03-18T22:49:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.090+0000 7f1a62fe0740 1 client.17262 shutdown 2024-03-18T22:49:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.090+0000 7f1a62fe0740 1 -- 172.21.15.40:0/3996101222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee8ce0 msgr2=0x55e6f1ef2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.090+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/3996101222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef2cc0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1a5402c170 tx=0x7f1a54004490 comp rx=0 tx=0).stop 2024-03-18T22:49:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.090+0000 7f1a62fe0740 1 -- 172.21.15.40:0/3996101222 shutdown_connections 2024-03-18T22:49:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.090+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/3996101222 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1eede50 0x55e6f1ef9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.090+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/3996101222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef2cc0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.090+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/3996101222 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e6f1efe750 0x55e6f1f00bb0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.090+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/3996101222 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ef2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.090+0000 7f1a41ffb640 0 client.17262 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.090+0000 7f1a62fe0740 1 -- 172.21.15.40:0/3996101222 >> 172.21.15.40:0/3996101222 conn(0x55e6f1ee3860 msgr2=0x55e6f1ee2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.090+0000 7f1a62fe0740 1 -- 172.21.15.40:0/3996101222 shutdown_connections 2024-03-18T22:49:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.090+0000 7f1a62fe0740 1 -- 172.21.15.40:0/3996101222 wait complete. 2024-03-18T22:49:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.090+0000 7f1a62fe0740 20 client.17262 trim_cache size 0 max 16384 2024-03-18T22:49:10.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a62fe0740 1 Processor -- start 2024-03-18T22:49:10.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a62fe0740 1 -- start start 2024-03-18T22:49:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a62fe0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ee3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a62fe0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a62fe0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1eede50 0x55e6f1ee2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a62fe0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a581ac5f0 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a62fe0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a581ae130 con 0x55e6f1eede50 2024-03-18T22:49:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a62fe0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1a581accc0 con 0x55e6f1ee82f0 2024-03-18T22:49:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a5e0b9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a5d8b8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ee3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a5d8b8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ee3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:32938/0 (socket says 172.21.15.40:32938) 2024-03-18T22:49:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a5d8b8640 1 -- 172.21.15.40:0/2852789061 learned_addr learned my addr 172.21.15.40:0/2852789061 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a5d8b8640 1 --2- 172.21.15.40:0/2852789061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1eede50 0x55e6f1ee2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a5e0b9640 1 -- 172.21.15.40:0/2852789061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1ee82f0 msgr2=0x55e6f1ee3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a5e0b9640 1 --2- 172.21.15.40:0/2852789061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ee3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a5e0b9640 1 -- 172.21.15.40:0/2852789061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1eede50 msgr2=0x55e6f1ee2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:10.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a5e0b9640 1 --2- 172.21.15.40:0/2852789061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1eede50 0x55e6f1ee2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a5e0b9640 1 -- 172.21.15.40:0/2852789061 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6f1ef1860 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.104+0000 7f1a5d8b8640 1 --2- 172.21.15.40:0/2852789061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ee3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:10.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.105+0000 7f1a5e0b9640 1 --2- 172.21.15.40:0/2852789061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef9cc0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f1a5402c420 tx=0x7f1a54005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b5387fc64bd4b3c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:10.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.105+0000 7f1a43fff640 1 -- 172.21.15.40:0/2852789061 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a54045040 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.105+0000 7f1a43fff640 1 -- 172.21.15.40:0/2852789061 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a54009520 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.105+0000 7f1a43fff640 1 -- 172.21.15.40:0/2852789061 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a54004920 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.105+0000 7f1a62fe0740 1 -- 172.21.15.40:0/2852789061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee8ce0 msgr2=0x55e6f1ef9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:10.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.105+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/2852789061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef9cc0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f1a5402c420 tx=0x7f1a54005fd0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 1 -- 172.21.15.40:0/2852789061 shutdown_connections 2024-03-18T22:49:10.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/2852789061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1eede50 0x55e6f1ee2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/2852789061 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef9cc0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/2852789061 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ee3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 1 -- 172.21.15.40:0/2852789061 >> 172.21.15.40:0/2852789061 conn(0x55e6f1ee3860 msgr2=0x55e6f1ee9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 1 -- 172.21.15.40:0/2852789061 shutdown_connections 2024-03-18T22:49:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 1 -- 172.21.15.40:0/2852789061 wait complete. 2024-03-18T22:49:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 1 -- start start 2024-03-18T22:49:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 1 Processor -- start 2024-03-18T22:49:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ef7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1eede50 0x55e6f1ef86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1a581ada60 con 0x55e6f1eede50 2024-03-18T22:49:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a581ad390 con 0x55e6f1ee82f0 2024-03-18T22:49:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1a581aa5d0 con 0x55e6f1ee8ce0 2024-03-18T22:49:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a62fe0740 10 client.0 _ll_register_callbacks cb 0x7f1a48009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a5d8b8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1eede50 0x55e6f1ef86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a5d8b8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1eede50 0x55e6f1ef86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51998/0 (socket says 172.21.15.40:51998) 2024-03-18T22:49:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a5d8b8640 1 -- 172.21.15.40:0/2796311580 learned_addr learned my addr 172.21.15.40:0/2796311580 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.106+0000 7f1a5e0b9640 1 --2- 172.21.15.40:0/2796311580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.107+0000 7f1a5d8b8640 1 --2- 172.21.15.40:0/2796311580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ef7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.107+0000 7f1a5d8b8640 1 -- 172.21.15.40:0/2796311580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1ee8ce0 msgr2=0x55e6f1ef76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.107+0000 7f1a5d8b8640 1 --2- 172.21.15.40:0/2796311580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.107+0000 7f1a5d8b8640 1 -- 172.21.15.40:0/2796311580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee82f0 msgr2=0x55e6f1ef7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.107+0000 7f1a5d8b8640 1 --2- 172.21.15.40:0/2796311580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ef7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.107+0000 7f1a5d8b8640 1 -- 172.21.15.40:0/2796311580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6f1ef8ca0 con 0x55e6f1eede50 2024-03-18T22:49:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.107+0000 7f1a5e0b9640 1 --2- 172.21.15.40:0/2796311580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.107+0000 7f1a5d8b8640 1 --2- 172.21.15.40:0/2796311580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1eede50 0x55e6f1ef86b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1a48026530 tx=0x7f1a54005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d895792c983a3b03 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.107+0000 7f1a41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.107+0000 7f1a41ffb640 1 -- 172.21.15.40:0/2796311580 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4801c040 con 0x55e6f1eede50 2024-03-18T22:49:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.107+0000 7f1a41ffb640 1 -- 172.21.15.40:0/2796311580 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1a48005220 con 0x55e6f1eede50 2024-03-18T22:49:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.107+0000 7f1a62fe0740 1 -- 172.21.15.40:0/2796311580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a54009520 con 0x55e6f1eede50 2024-03-18T22:49:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.107+0000 7f1a41ffb640 1 -- 172.21.15.40:0/2796311580 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1a48039050 con 0x55e6f1eede50 2024-03-18T22:49:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.107+0000 7f1a62fe0740 10 client.17268 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.107+0000 7f1a62fe0740 1 -- 172.21.15.40:0/2796311580 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e6f1efa010 con 0x55e6f1eede50 2024-03-18T22:49:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.108+0000 7f1a41ffb640 1 -- 172.21.15.40:0/2796311580 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a54009520 con 0x55e6f1eede50 2024-03-18T22:49:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.108+0000 7f1a41ffb640 1 -- 172.21.15.40:0/2796311580 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f1a48027ce0 con 0x55e6f1eede50 2024-03-18T22:49:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.108+0000 7f1a62fe0740 10 client.17268 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.108+0000 7f1a62fe0740 10 client.17268 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.108+0000 7f1a62fe0740 20 client.17268 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.108+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/2796311580 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e6f1f652d0 0x55e6f1f676f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.108+0000 7f1a62fe0740 4 client.17268 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:49:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.108+0000 7f1a62fe0740 1 -- 172.21.15.40:0/2796311580 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e6f1e599f0 con 0x55e6f1f652d0 2024-03-18T22:49:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.108+0000 7f1a5e0b9640 1 --2- 172.21.15.40:0/2796311580 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e6f1f652d0 0x55e6f1f676f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.108+0000 7f1a41ffb640 10 client.17268 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.108+0000 7f1a5e0b9640 1 --2- 172.21.15.40:0/2796311580 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e6f1f652d0 0x55e6f1f676f0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=317cb2729d5ce2a8 server_cookie=93d6974e8b774b61 in_seq=0 out_seq=0 2024-03-18T22:49:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.108+0000 7f1a41ffb640 1 -- 172.21.15.40:0/2796311580 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e6f1e599f0 con 0x55e6f1f652d0 2024-03-18T22:49:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.108+0000 7f1a41ffb640 10 client.17268 handle_command_reply: tid=0 2024-03-18T22:49:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.108+0000 7f1a62fe0740 1 client.17268 shutdown 2024-03-18T22:49:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a62fe0740 1 -- 172.21.15.40:0/2796311580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1eede50 msgr2=0x55e6f1ef86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/2796311580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1eede50 0x55e6f1ef86b0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1a48026530 tx=0x7f1a54005710 comp rx=0 tx=0).stop 2024-03-18T22:49:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a62fe0740 1 -- 172.21.15.40:0/2796311580 shutdown_connections 2024-03-18T22:49:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/2796311580 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6f1eede50 0x55e6f1ef86b0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/2796311580 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6f1ee8ce0 0x55e6f1ef76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/2796311580 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e6f1f652d0 0x55e6f1f676f0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a41ffb640 0 client.17268 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a62fe0740 1 --2- 172.21.15.40:0/2796311580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6f1ee82f0 0x55e6f1ef7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a62fe0740 1 -- 172.21.15.40:0/2796311580 >> 172.21.15.40:0/2796311580 conn(0x55e6f1ee3860 msgr2=0x55e6f1eea240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a62fe0740 1 -- 172.21.15.40:0/2796311580 shutdown_connections 2024-03-18T22:49:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a62fe0740 1 -- 172.21.15.40:0/2796311580 wait complete. 2024-03-18T22:49:10.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a62fe0740 20 client.17268 trim_cache size 0 max 16384 2024-03-18T22:49:10.116 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:49:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a6031d640 1 -- 172.21.15.40:0/2460325862 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a440a2880 msgr2=0x7f1a440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a6031d640 1 --2- 172.21.15.40:0/2460325862 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a440a2880 0x7f1a440a4d40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1a500981b0 tx=0x7f1a50009f90 comp rx=0 tx=0).stop 2024-03-18T22:49:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a6031d640 1 -- 172.21.15.40:0/2460325862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a58102310 msgr2=0x7f1a58103270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a6031d640 1 --2- 172.21.15.40:0/2460325862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a58102310 0x7f1a58103270 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000efd0 tx=0x7f1a5000c6a0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a6031d640 1 -- 172.21.15.40:0/2460325862 shutdown_connections 2024-03-18T22:49:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a6031d640 1 --2- 172.21.15.40:0/2460325862 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1a440a2880 0x7f1a440a4d40 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.109+0000 7f1a6031d640 1 --2- 172.21.15.40:0/2460325862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a58102310 0x7f1a58103270 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.110+0000 7f1a6031d640 1 --2- 172.21.15.40:0/2460325862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1a58101950 0x7f1a58102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.110+0000 7f1a6031d640 1 --2- 172.21.15.40:0/2460325862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a58100760 0x7f1a581046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.110+0000 7f1a6031d640 1 -- 172.21.15.40:0/2460325862 >> 172.21.15.40:0/2460325862 conn(0x7f1a580fbcf0 msgr2=0x7f1a58069c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.110+0000 7f1a6031d640 1 -- 172.21.15.40:0/2460325862 shutdown_connections 2024-03-18T22:49:10.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:10.110+0000 7f1a6031d640 1 -- 172.21.15.40:0/2460325862 wait complete. 2024-03-18T22:49:10.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:10 smithi113 ceph-mon[66775]: pgmap v546: 129 pgs: 129 active+clean; 8.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:49:10.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:10 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3404868223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:49:10.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:10 smithi103 ceph-mon[66377]: pgmap v546: 129 pgs: 129 active+clean; 8.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:49:10.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:10 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3404868223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:49:10.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:10 smithi040 ceph-mon[62643]: pgmap v546: 129 pgs: 129 active+clean; 8.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:49:10.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:10 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3404868223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:49:11.122 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:49:11.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:49:11.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:49:11.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.238+0000 7fd5f5e75640 1 -- 172.21.15.40:0/627629091 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f01000f0 msgr2=0x7fd5f0100510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.238+0000 7fd5f5e75640 1 --2- 172.21.15.40:0/627629091 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f01000f0 0x7fd5f0100510 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009900 tx=0x7fd5e002bc30 comp rx=0 tx=0).stop 2024-03-18T22:49:11.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.238+0000 7fd5f5e75640 1 -- 172.21.15.40:0/627629091 shutdown_connections 2024-03-18T22:49:11.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.238+0000 7fd5f5e75640 1 --2- 172.21.15.40:0/627629091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5f01013f0 0x7fd5f0101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.238+0000 7fd5f5e75640 1 --2- 172.21.15.40:0/627629091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5f0100a50 0x7fd5f0100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.238+0000 7fd5f5e75640 1 --2- 172.21.15.40:0/627629091 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f01000f0 0x7fd5f0100510 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.238+0000 7fd5f5e75640 1 -- 172.21.15.40:0/627629091 >> 172.21.15.40:0/627629091 conn(0x7fd5f00fbcb0 msgr2=0x7fd5f00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:11.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.238+0000 7fd5f5e75640 1 -- 172.21.15.40:0/627629091 shutdown_connections 2024-03-18T22:49:11.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.238+0000 7fd5f5e75640 1 -- 172.21.15.40:0/627629091 wait complete. 2024-03-18T22:49:11.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.239+0000 7fd5f5e75640 1 Processor -- start 2024-03-18T22:49:11.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.239+0000 7fd5f5e75640 1 -- start start 2024-03-18T22:49:11.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5f5e75640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5f01000f0 0x7fd5f0199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5f5e75640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f0100a50 0x7fd5f019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5f5e75640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5f01013f0 0x7fd5f01a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5f5e75640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f010f560 con 0x7fd5f01000f0 2024-03-18T22:49:11.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5f5e75640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f010f3e0 con 0x7fd5f01013f0 2024-03-18T22:49:11.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5f5e75640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f010f6e0 con 0x7fd5f0100a50 2024-03-18T22:49:11.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5ef7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5f01000f0 0x7fd5f0199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5ef7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5f01000f0 0x7fd5f0199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52020/0 (socket says 172.21.15.40:52020) 2024-03-18T22:49:11.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5ef7fe640 1 -- 172.21.15.40:0/1015415398 learned_addr learned my addr 172.21.15.40:0/1015415398 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:11.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5effff640 1 --2- 172.21.15.40:0/1015415398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5f01013f0 0x7fd5f01a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5ef7fe640 1 -- 172.21.15.40:0/1015415398 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f0100a50 msgr2=0x7fd5f019a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5ef7fe640 1 --2- 172.21.15.40:0/1015415398 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f0100a50 0x7fd5f019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5ef7fe640 1 -- 172.21.15.40:0/1015415398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5f01013f0 msgr2=0x7fd5f01a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5ef7fe640 1 --2- 172.21.15.40:0/1015415398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5f01013f0 0x7fd5f01a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5ef7fe640 1 -- 172.21.15.40:0/1015415398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f01a1890 con 0x7fd5f01000f0 2024-03-18T22:49:11.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.240+0000 7fd5effff640 1 --2- 172.21.15.40:0/1015415398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5f01013f0 0x7fd5f01a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:49:11.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.241+0000 7fd5ef7fe640 1 --2- 172.21.15.40:0/1015415398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5f01000f0 0x7fd5f0199c80 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0000c00 tx=0x7fd5e00040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:11.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.241+0000 7fd5ecff9640 1 -- 172.21.15.40:0/1015415398 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0043020 con 0x7fd5f01000f0 2024-03-18T22:49:11.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.241+0000 7fd5ecff9640 1 -- 172.21.15.40:0/1015415398 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5e00044c0 con 0x7fd5f01000f0 2024-03-18T22:49:11.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.241+0000 7fd5f5e75640 1 -- 172.21.15.40:0/1015415398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f01a1b20 con 0x7fd5f01000f0 2024-03-18T22:49:11.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.241+0000 7fd5ecff9640 1 -- 172.21.15.40:0/1015415398 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0003c30 con 0x7fd5f01000f0 2024-03-18T22:49:11.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.241+0000 7fd5f5e75640 1 -- 172.21.15.40:0/1015415398 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f01a2070 con 0x7fd5f01000f0 2024-03-18T22:49:11.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5ecff9640 1 -- 172.21.15.40:0/1015415398 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd5e0046020 con 0x7fd5f01000f0 2024-03-18T22:49:11.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5f8a5a740 1 Processor -- start 2024-03-18T22:49:11.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5f8a5a740 1 -- start start 2024-03-18T22:49:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5f8a5a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd343a0 0x55bdadd347c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5f8a5a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd34d90 0x55bdadd39890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5f8a5a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd39f00 0x55bdadd3c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5f8a5a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bdadc99160 con 0x55bdadd343a0 2024-03-18T22:49:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5f8a5a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bdadc41e10 con 0x55bdadd39f00 2024-03-18T22:49:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5f8a5a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55bdadc95200 con 0x55bdadd34d90 2024-03-18T22:49:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5effff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd343a0 0x55bdadd347c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5effff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd343a0 0x55bdadd347c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52024/0 (socket says 172.21.15.40:52024) 2024-03-18T22:49:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5effff640 1 -- 172.21.15.40:0/2106323879 learned_addr learned my addr 172.21.15.40:0/2106323879 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5ef7fe640 1 --2- 172.21.15.40:0/2106323879 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd34d90 0x55bdadd39890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5eeffd640 1 --2- 172.21.15.40:0/2106323879 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd39f00 0x55bdadd3c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5effff640 1 -- 172.21.15.40:0/2106323879 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd34d90 msgr2=0x55bdadd39890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5effff640 1 --2- 172.21.15.40:0/2106323879 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd34d90 0x55bdadd39890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5effff640 1 -- 172.21.15.40:0/2106323879 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd39f00 msgr2=0x55bdadd3c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5effff640 1 --2- 172.21.15.40:0/2106323879 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd39f00 0x55bdadd3c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.244+0000 7fd5effff640 1 -- 172.21.15.40:0/2106323879 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdada757e0 con 0x55bdadd343a0 2024-03-18T22:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.245+0000 7fd5ecff9640 1 --2- 172.21.15.40:0/1015415398 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5d80a2800 0x7fd5d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.245+0000 7fd5ef7fe640 1 --2- 172.21.15.40:0/1015415398 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5d80a2800 0x7fd5d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.245+0000 7fd5effff640 1 --2- 172.21.15.40:0/2106323879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd343a0 0x55bdadd347c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009f90 tx=0x7fd5e800f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963450ceb07bae74 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.245+0000 7fd5ecff9640 1 -- 172.21.15.40:0/1015415398 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd5e00f12c0 con 0x7fd5f01000f0 2024-03-18T22:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.245+0000 7fd5d5ffb640 1 -- 172.21.15.40:0/2106323879 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8003b70 con 0x55bdadd343a0 2024-03-18T22:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.245+0000 7fd5d5ffb640 1 -- 172.21.15.40:0/2106323879 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5e8003d10 con 0x55bdadd343a0 2024-03-18T22:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.245+0000 7fd5d5ffb640 1 -- 172.21.15.40:0/2106323879 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e80119c0 con 0x55bdadd343a0 2024-03-18T22:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.246+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/2106323879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd343a0 msgr2=0x55bdadd347c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.246+0000 7fd5ef7fe640 1 --2- 172.21.15.40:0/1015415398 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5d80a2800 0x7fd5d80a4cc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00c5a80 tx=0x7fd5e00f3040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.246+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/2106323879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd343a0 0x55bdadd347c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009f90 tx=0x7fd5e800f7f0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.246+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/2106323879 shutdown_connections 2024-03-18T22:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.246+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/2106323879 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd39f00 0x55bdadd3c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.246+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/2106323879 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd34d90 0x55bdadd39890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.246+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/2106323879 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd343a0 0x55bdadd347c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.246+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/2106323879 >> 172.21.15.40:0/2106323879 conn(0x55bdadd2f910 msgr2=0x55bdadd2fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.246+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/2106323879 shutdown_connections 2024-03-18T22:49:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.246+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/2106323879 wait complete. 2024-03-18T22:49:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.246+0000 7fd5f8a5a740 1 -- start start 2024-03-18T22:49:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.247+0000 7fd5f8a5a740 1 Processor -- start 2024-03-18T22:49:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.247+0000 7fd5f8a5a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd343a0 0x55bdadd3e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.247+0000 7fd5f8a5a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd34d90 0x55bdadd3edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.247+0000 7fd5f8a5a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd39f00 0x55bdadd45e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.247+0000 7fd5f8a5a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bdadd3ceb0 con 0x55bdadd39f00 2024-03-18T22:49:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.247+0000 7fd5f8a5a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bdadc551a0 con 0x55bdadd343a0 2024-03-18T22:49:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.247+0000 7fd5f8a5a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55bdadd3d030 con 0x55bdadd34d90 2024-03-18T22:49:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.247+0000 7fd5f8a5a740 10 client.0 _ll_register_callbacks cb 0x55bdad9acc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:11.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.247+0000 7fd5effff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd39f00 0x55bdadd45e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.247+0000 7fd5eeffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd34d90 0x55bdadd3edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.247+0000 7fd5effff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd39f00 0x55bdadd45e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52038/0 (socket says 172.21.15.40:52038) 2024-03-18T22:49:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.247+0000 7fd5effff640 1 -- 172.21.15.40:0/1466374768 learned_addr learned my addr 172.21.15.40:0/1466374768 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.247+0000 7fd5eeffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd34d90 0x55bdadd3edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:32982/0 (socket says 172.21.15.40:32982) 2024-03-18T22:49:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.247+0000 7fd5effff640 1 --2- 172.21.15.40:0/1466374768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd343a0 0x55bdadd3e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.248+0000 7fd5eeffd640 1 -- 172.21.15.40:0/1466374768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd343a0 msgr2=0x55bdadd3e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.248+0000 7fd5eeffd640 1 --2- 172.21.15.40:0/1466374768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd343a0 0x55bdadd3e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.248+0000 7fd5eeffd640 1 -- 172.21.15.40:0/1466374768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd39f00 msgr2=0x55bdadd45e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.248+0000 7fd5eeffd640 1 --2- 172.21.15.40:0/1466374768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd39f00 0x55bdadd45e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.248+0000 7fd5eeffd640 1 -- 172.21.15.40:0/1466374768 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdadad5f70 con 0x55bdadd34d90 2024-03-18T22:49:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.248+0000 7fd5eeffd640 1 --2- 172.21.15.40:0/1466374768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd34d90 0x55bdadd3edc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40095f0 tx=0x7fd5e4011b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d50b4e425e710d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.248+0000 7fd5ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:49:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.248+0000 7fd5ceffd640 1 -- 172.21.15.40:0/1466374768 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e400fcd0 con 0x55bdadd34d90 2024-03-18T22:49:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.248+0000 7fd5ceffd640 1 -- 172.21.15.40:0/1466374768 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd5e4004030 con 0x55bdadd34d90 2024-03-18T22:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.248+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1466374768 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5e8003d10 con 0x55bdadd34d90 2024-03-18T22:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.248+0000 7fd5ceffd640 1 -- 172.21.15.40:0/1466374768 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd5e400fe70 con 0x55bdadd34d90 2024-03-18T22:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.248+0000 7fd5f8a5a740 10 client.27088 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.248+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1466374768 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bdad96d170 con 0x55bdadd34d90 2024-03-18T22:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.248+0000 7fd5ceffd640 1 -- 172.21.15.40:0/1466374768 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5e8003d10 con 0x55bdadd34d90 2024-03-18T22:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.248+0000 7fd5ceffd640 1 -- 172.21.15.40:0/1466374768 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd5e4004950 con 0x55bdadd34d90 2024-03-18T22:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.249+0000 7fd5f8a5a740 10 client.27088 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.249+0000 7fd5f8a5a740 10 client.27088 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.249+0000 7fd5f8a5a740 20 client.27088 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.249+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1466374768 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdadd4a8d0 0x55bdadd4cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.249+0000 7fd5f8a5a740 4 client.27088 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.249+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1466374768 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdadca66a0 con 0x55bdadd4a8d0 2024-03-18T22:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.249+0000 7fd5effff640 1 --2- 172.21.15.40:0/1466374768 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdadd4a8d0 0x55bdadd4cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.249+0000 7fd5ceffd640 10 client.27088 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.249+0000 7fd5effff640 1 --2- 172.21.15.40:0/1466374768 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdadd4a8d0 0x55bdadd4cd10 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f5fd1ab59fe2261 server_cookie=8699e21ca7324759 in_seq=0 out_seq=0 2024-03-18T22:49:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.250+0000 7fd5ceffd640 1 -- 172.21.15.40:0/1466374768 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bdadca66a0 con 0x55bdadd4a8d0 2024-03-18T22:49:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.250+0000 7fd5ceffd640 10 client.27088 handle_command_reply: tid=0 2024-03-18T22:49:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.250+0000 7fd5f8a5a740 1 client.27088 shutdown 2024-03-18T22:49:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.250+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1466374768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd34d90 msgr2=0x55bdadd3edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.250+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1466374768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd34d90 0x55bdadd3edc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40095f0 tx=0x7fd5e4011b50 comp rx=0 tx=0).stop 2024-03-18T22:49:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.250+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1466374768 shutdown_connections 2024-03-18T22:49:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.250+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1466374768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd39f00 0x55bdadd45e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.250+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1466374768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd34d90 0x55bdadd3edc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.250+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1466374768 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdadd4a8d0 0x55bdadd4cd10 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.250+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1466374768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd343a0 0x55bdadd3e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.250+0000 7fd5ceffd640 0 client.27088 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.250+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1466374768 >> 172.21.15.40:0/1466374768 conn(0x55bdadd2f910 msgr2=0x55bdadd2eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.250+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1466374768 shutdown_connections 2024-03-18T22:49:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.251+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1466374768 wait complete. 2024-03-18T22:49:11.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.251+0000 7fd5f8a5a740 20 client.27088 trim_cache size 0 max 16384 2024-03-18T22:49:11.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.271+0000 7fd5f8a5a740 1 Processor -- start 2024-03-18T22:49:11.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.271+0000 7fd5f8a5a740 1 -- start start 2024-03-18T22:49:11.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.271+0000 7fd5f8a5a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd343a0 0x55bdadd33280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.271+0000 7fd5f8a5a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd34d90 0x55bdadd45dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.271+0000 7fd5f8a5a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd39f00 0x55bdadd51610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.271+0000 7fd5f8a5a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f01ac660 con 0x55bdadd39f00 2024-03-18T22:49:11.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.271+0000 7fd5f8a5a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f01ae1a0 con 0x55bdadd34d90 2024-03-18T22:49:11.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.271+0000 7fd5f8a5a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f01acd30 con 0x55bdadd343a0 2024-03-18T22:49:11.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.271+0000 7fd5effff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd39f00 0x55bdadd51610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.271+0000 7fd5eeffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd34d90 0x55bdadd45dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.271+0000 7fd5effff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd343a0 0x55bdadd33280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.271+0000 7fd5eeffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd34d90 0x55bdadd45dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:54010/0 (socket says 172.21.15.40:54010) 2024-03-18T22:49:11.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.271+0000 7fd5eeffd640 1 -- 172.21.15.40:0/1073044262 learned_addr learned my addr 172.21.15.40:0/1073044262 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:11.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.271+0000 7fd5effff640 1 -- 172.21.15.40:0/1073044262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd34d90 msgr2=0x55bdadd45dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.271+0000 7fd5effff640 1 --2- 172.21.15.40:0/1073044262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd34d90 0x55bdadd45dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.272+0000 7fd5effff640 1 -- 172.21.15.40:0/1073044262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd39f00 msgr2=0x55bdadd51610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.272+0000 7fd5effff640 1 --2- 172.21.15.40:0/1073044262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd39f00 0x55bdadd51610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.272+0000 7fd5effff640 1 -- 172.21.15.40:0/1073044262 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdadd4d390 con 0x55bdadd343a0 2024-03-18T22:49:11.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.272+0000 7fd5effff640 1 --2- 172.21.15.40:0/1073044262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd343a0 0x55bdadd33280 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x55bdadd4d5c0 tx=0x7fd5e800f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2703c8b436cccb59 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:11.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.272+0000 7fd5d57fa640 1 -- 172.21.15.40:0/1073044262 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8024040 con 0x55bdadd343a0 2024-03-18T22:49:11.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.272+0000 7fd5d57fa640 1 -- 172.21.15.40:0/1073044262 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5e8004df0 con 0x55bdadd343a0 2024-03-18T22:49:11.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.272+0000 7fd5d57fa640 1 -- 172.21.15.40:0/1073044262 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e80057d0 con 0x55bdadd343a0 2024-03-18T22:49:11.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1073044262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd343a0 msgr2=0x55bdadd33280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1073044262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd343a0 0x55bdadd33280 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x55bdadd4d5c0 tx=0x7fd5e800f7f0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1073044262 shutdown_connections 2024-03-18T22:49:11.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1073044262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd39f00 0x55bdadd51610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1073044262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd34d90 0x55bdadd45dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1073044262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd343a0 0x55bdadd33280 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1073044262 >> 172.21.15.40:0/1073044262 conn(0x55bdadd2f910 msgr2=0x55bdadd32b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:11.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1073044262 shutdown_connections 2024-03-18T22:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1073044262 wait complete. 2024-03-18T22:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 -- start start 2024-03-18T22:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 Processor -- start 2024-03-18T22:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd343a0 0x55bdadd45a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd34d90 0x55bdadd44060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd39f00 0x55bdadd445a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f01adad0 con 0x55bdadd34d90 2024-03-18T22:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f01ad400 con 0x55bdadd343a0 2024-03-18T22:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f01aa640 con 0x55bdadd39f00 2024-03-18T22:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.273+0000 7fd5f8a5a740 10 client.0 _ll_register_callbacks cb 0x7fd5e40097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5eeffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd34d90 0x55bdadd44060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5effff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd343a0 0x55bdadd45a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5eeffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd34d90 0x55bdadd44060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52054/0 (socket says 172.21.15.40:52054) 2024-03-18T22:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5eeffd640 1 -- 172.21.15.40:0/1588208337 learned_addr learned my addr 172.21.15.40:0/1588208337 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5effff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd343a0 0x55bdadd45a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:54012/0 (socket says 172.21.15.40:54012) 2024-03-18T22:49:11.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5effff640 1 --2- 172.21.15.40:0/1588208337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd39f00 0x55bdadd445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5eeffd640 1 -- 172.21.15.40:0/1588208337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd39f00 msgr2=0x55bdadd445a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5eeffd640 1 --2- 172.21.15.40:0/1588208337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd39f00 0x55bdadd445a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5eeffd640 1 -- 172.21.15.40:0/1588208337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd343a0 msgr2=0x55bdadd45a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5eeffd640 1 --2- 172.21.15.40:0/1588208337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd343a0 0x55bdadd45a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5eeffd640 1 -- 172.21.15.40:0/1588208337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdadd30680 con 0x55bdadd34d90 2024-03-18T22:49:11.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5eeffd640 1 --2- 172.21.15.40:0/1588208337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd34d90 0x55bdadd44060 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4004e90 tx=0x7fd5e40049b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc6068866978530 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:11.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:11.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5ceffd640 1 -- 172.21.15.40:0/1588208337 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4004240 con 0x55bdadd34d90 2024-03-18T22:49:11.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5ceffd640 1 -- 172.21.15.40:0/1588208337 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd5e40043e0 con 0x55bdadd34d90 2024-03-18T22:49:11.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1588208337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5e8004df0 con 0x55bdadd34d90 2024-03-18T22:49:11.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5ceffd640 1 -- 172.21.15.40:0/1588208337 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd5e4009ce0 con 0x55bdadd34d90 2024-03-18T22:49:11.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5f8a5a740 10 client.17286 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:11.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.274+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1588208337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bdaddb1850 con 0x55bdadd34d90 2024-03-18T22:49:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.275+0000 7fd5ceffd640 1 -- 172.21.15.40:0/1588208337 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5e8004df0 con 0x55bdadd34d90 2024-03-18T22:49:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.275+0000 7fd5ceffd640 1 -- 172.21.15.40:0/1588208337 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fd5e4003970 con 0x55bdadd34d90 2024-03-18T22:49:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.275+0000 7fd5f8a5a740 10 client.17286 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.275+0000 7fd5f8a5a740 10 client.17286 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.275+0000 7fd5f8a5a740 20 client.17286 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.275+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1588208337 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdaddb2280 0x55bdaddb46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.275+0000 7fd5f8a5a740 4 client.17286 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:49:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.275+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1588208337 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bdadca09f0 con 0x55bdaddb2280 2024-03-18T22:49:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.275+0000 7fd5effff640 1 --2- 172.21.15.40:0/1588208337 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdaddb2280 0x55bdaddb46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.275+0000 7fd5ceffd640 10 client.17286 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.275+0000 7fd5effff640 1 --2- 172.21.15.40:0/1588208337 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdaddb2280 0x55bdaddb46a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14266ccefcc9b728 server_cookie=c6f2bc76b89e7a50 in_seq=0 out_seq=0 2024-03-18T22:49:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.276+0000 7fd5ceffd640 1 -- 172.21.15.40:0/1588208337 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bdadca09f0 con 0x55bdaddb2280 2024-03-18T22:49:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.276+0000 7fd5ceffd640 10 client.17286 handle_command_reply: tid=0 2024-03-18T22:49:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.276+0000 7fd5f8a5a740 1 client.17286 shutdown 2024-03-18T22:49:11.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.277+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1588208337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd34d90 msgr2=0x55bdadd44060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.277+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1588208337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd34d90 0x55bdadd44060 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4004e90 tx=0x7fd5e40049b0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.277+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1588208337 shutdown_connections 2024-03-18T22:49:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.277+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1588208337 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bdadd39f00 0x55bdadd445a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.277+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1588208337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bdadd34d90 0x55bdadd44060 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.277+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1588208337 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55bdaddb2280 0x55bdaddb46a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.277+0000 7fd5f8a5a740 1 --2- 172.21.15.40:0/1588208337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bdadd343a0 0x55bdadd45a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.277+0000 7fd5ceffd640 0 client.17286 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.277+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1588208337 >> 172.21.15.40:0/1588208337 conn(0x55bdadd2f910 msgr2=0x55bdadd33400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.277+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1588208337 shutdown_connections 2024-03-18T22:49:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.277+0000 7fd5f8a5a740 1 -- 172.21.15.40:0/1588208337 wait complete. 2024-03-18T22:49:11.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.277+0000 7fd5f8a5a740 20 client.17286 trim_cache size 0 max 16384 2024-03-18T22:49:11.283 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:49:11.283 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:49:11.283 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "08c4afe1-367c-4b26-a212-50159d3b5875", 2024-03-18T22:49:11.283 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:49:11.283 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:49:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.278+0000 7fd5f5e75640 1 -- 172.21.15.40:0/1015415398 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5d80a2800 msgr2=0x7fd5d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.278+0000 7fd5f5e75640 1 --2- 172.21.15.40:0/1015415398 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5d80a2800 0x7fd5d80a4cc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00c5a80 tx=0x7fd5e00f3040 comp rx=0 tx=0).stop 2024-03-18T22:49:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.278+0000 7fd5f5e75640 1 -- 172.21.15.40:0/1015415398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5f01000f0 msgr2=0x7fd5f0199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.278+0000 7fd5f5e75640 1 --2- 172.21.15.40:0/1015415398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5f01000f0 0x7fd5f0199c80 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0000c00 tx=0x7fd5e00040c0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.278+0000 7fd5f5e75640 1 -- 172.21.15.40:0/1015415398 shutdown_connections 2024-03-18T22:49:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.278+0000 7fd5f5e75640 1 --2- 172.21.15.40:0/1015415398 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd5d80a2800 0x7fd5d80a4cc0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.278+0000 7fd5f5e75640 1 --2- 172.21.15.40:0/1015415398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5f01013f0 0x7fd5f01a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.278+0000 7fd5f5e75640 1 --2- 172.21.15.40:0/1015415398 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd5f0100a50 0x7fd5f019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.278+0000 7fd5f5e75640 1 --2- 172.21.15.40:0/1015415398 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5f01000f0 0x7fd5f0199c80 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.278+0000 7fd5f5e75640 1 -- 172.21.15.40:0/1015415398 >> 172.21.15.40:0/1015415398 conn(0x7fd5f00fbcb0 msgr2=0x7fd5f00fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:11.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.278+0000 7fd5f5e75640 1 -- 172.21.15.40:0/1015415398 shutdown_connections 2024-03-18T22:49:11.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.278+0000 7fd5f5e75640 1 -- 172.21.15.40:0/1015415398 wait complete. 2024-03-18T22:49:11.291 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:49:11.291 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:49:11.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.413+0000 7f8b01f60640 1 -- 172.21.15.40:0/1613837543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8afc102e40 msgr2=0x7f8afc103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.413+0000 7f8b01f60640 1 --2- 172.21.15.40:0/1613837543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8afc102e40 0x7f8afc103260 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8aec009900 tx=0x7f8aec02bc30 comp rx=0 tx=0).stop 2024-03-18T22:49:11.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.413+0000 7f8b01f60640 1 -- 172.21.15.40:0/1613837543 shutdown_connections 2024-03-18T22:49:11.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.413+0000 7f8b01f60640 1 --2- 172.21.15.40:0/1613837543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8afc100270 0x7f8afc109e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.413+0000 7f8b01f60640 1 --2- 172.21.15.40:0/1613837543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8afc0ff8b0 0x7f8afc0ffd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.413+0000 7f8b01f60640 1 --2- 172.21.15.40:0/1613837543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8afc102e40 0x7f8afc103260 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.413+0000 7f8b01f60640 1 -- 172.21.15.40:0/1613837543 >> 172.21.15.40:0/1613837543 conn(0x7f8afc0fb410 msgr2=0x7f8afc0fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:11.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.413+0000 7f8b01f60640 1 -- 172.21.15.40:0/1613837543 shutdown_connections 2024-03-18T22:49:11.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.413+0000 7f8b01f60640 1 -- 172.21.15.40:0/1613837543 wait complete. 2024-03-18T22:49:11.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.413+0000 7f8b01f60640 1 Processor -- start 2024-03-18T22:49:11.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8b01f60640 1 -- start start 2024-03-18T22:49:11.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8b01f60640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8afc0ff8b0 0x7f8afc197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8b01f60640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8afc100270 0x7f8afc198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8b01f60640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8afc102e40 0x7f8afc19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8b01f60640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc10d340 con 0x7f8afc0ff8b0 2024-03-18T22:49:11.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8b01f60640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc10d1c0 con 0x7f8afc102e40 2024-03-18T22:49:11.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8b01f60640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc10d4c0 con 0x7f8afc100270 2024-03-18T22:49:11.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8afb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8afc0ff8b0 0x7f8afc197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8afb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8afc0ff8b0 0x7f8afc197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52080/0 (socket says 172.21.15.40:52080) 2024-03-18T22:49:11.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8afb7fe640 1 -- 172.21.15.40:0/2968122680 learned_addr learned my addr 172.21.15.40:0/2968122680 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:11.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8afaffd640 1 --2- 172.21.15.40:0/2968122680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8afc100270 0x7f8afc198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8afbfff640 1 --2- 172.21.15.40:0/2968122680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8afc102e40 0x7f8afc19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8afaffd640 1 -- 172.21.15.40:0/2968122680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8afc102e40 msgr2=0x7f8afc19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8afaffd640 1 --2- 172.21.15.40:0/2968122680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8afc102e40 0x7f8afc19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8afaffd640 1 -- 172.21.15.40:0/2968122680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8afc0ff8b0 msgr2=0x7f8afc197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8afaffd640 1 --2- 172.21.15.40:0/2968122680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8afc0ff8b0 0x7f8afc197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8afaffd640 1 -- 172.21.15.40:0/2968122680 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8afc19f6e0 con 0x7f8afc100270 2024-03-18T22:49:11.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.414+0000 7f8afb7fe640 1 --2- 172.21.15.40:0/2968122680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8afc0ff8b0 0x7f8afc197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:11.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.415+0000 7f8afaffd640 1 --2- 172.21.15.40:0/2968122680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8afc100270 0x7f8afc198000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8af000e9f0 tx=0x7f8af000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:11.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.415+0000 7f8af8ff9640 1 -- 172.21.15.40:0/2968122680 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af000ccc0 con 0x7f8afc100270 2024-03-18T22:49:11.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.415+0000 7f8b01f60640 1 -- 172.21.15.40:0/2968122680 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8afc19f9d0 con 0x7f8afc100270 2024-03-18T22:49:11.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.415+0000 7f8af8ff9640 1 -- 172.21.15.40:0/2968122680 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8af00108e0 con 0x7f8afc100270 2024-03-18T22:49:11.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.415+0000 7f8af8ff9640 1 -- 172.21.15.40:0/2968122680 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af0018880 con 0x7f8afc100270 2024-03-18T22:49:11.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.415+0000 7f8b01f60640 1 -- 172.21.15.40:0/2968122680 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8afc19ff70 con 0x7f8afc100270 2024-03-18T22:49:11.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.417+0000 7f8b04c20740 1 Processor -- start 2024-03-18T22:49:11.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.417+0000 7f8af8ff9640 1 -- 172.21.15.40:0/2968122680 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8af0010a90 con 0x7f8afc100270 2024-03-18T22:49:11.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8b04c20740 1 -- start start 2024-03-18T22:49:11.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8b04c20740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a110a50 0x55905a110e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8b04c20740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a111440 0x55905a115f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8b04c20740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a1165b0 0x55905a118a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8b04c20740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55905a075500 con 0x55905a111440 2024-03-18T22:49:11.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8b04c20740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55905a01dbe0 con 0x55905a110a50 2024-03-18T22:49:11.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8b04c20740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55905a0715a0 con 0x55905a1165b0 2024-03-18T22:49:11.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8afb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a111440 0x55905a115f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8afaffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a1165b0 0x55905a118a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8afb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a111440 0x55905a115f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52084/0 (socket says 172.21.15.40:52084) 2024-03-18T22:49:11.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8afbfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a110a50 0x55905a110e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8afb7fe640 1 -- 172.21.15.40:0/3604199432 learned_addr learned my addr 172.21.15.40:0/3604199432 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:11.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8afb7fe640 1 -- 172.21.15.40:0/3604199432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a1165b0 msgr2=0x55905a118a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8afb7fe640 1 --2- 172.21.15.40:0/3604199432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a1165b0 0x55905a118a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8afb7fe640 1 -- 172.21.15.40:0/3604199432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a110a50 msgr2=0x55905a110e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8afb7fe640 1 --2- 172.21.15.40:0/3604199432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a110a50 0x55905a110e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8afb7fe640 1 -- 172.21.15.40:0/3604199432 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55905a03f000 con 0x55905a111440 2024-03-18T22:49:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8afbfff640 1 --2- 172.21.15.40:0/3604199432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a110a50 0x55905a110e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8af8ff9640 1 --2- 172.21.15.40:0/2968122680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8ae40a2800 0x7f8ae40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.418+0000 7f8afb7fe640 1 --2- 172.21.15.40:0/3604199432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a111440 0x55905a115f40 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f8aec036040 tx=0x7f8aec009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ca914708e8c15c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.419+0000 7f8afbfff640 1 --2- 172.21.15.40:0/2968122680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8ae40a2800 0x7f8ae40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.419+0000 7f8af8ff9640 1 -- 172.21.15.40:0/2968122680 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8af00cf190 con 0x7f8afc100270 2024-03-18T22:49:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.419+0000 7f8ae1ffb640 1 -- 172.21.15.40:0/3604199432 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec045020 con 0x55905a111440 2024-03-18T22:49:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.419+0000 7f8ae1ffb640 1 -- 172.21.15.40:0/3604199432 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8aec02cc90 con 0x55905a111440 2024-03-18T22:49:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.419+0000 7f8ae1ffb640 1 -- 172.21.15.40:0/3604199432 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec0049d0 con 0x55905a111440 2024-03-18T22:49:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.419+0000 7f8b04c20740 1 -- 172.21.15.40:0/3604199432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a111440 msgr2=0x55905a115f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.419+0000 7f8afbfff640 1 --2- 172.21.15.40:0/2968122680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8ae40a2800 0x7f8ae40a4cc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x55905a111ca0 tx=0x7f8af4009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:11.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.419+0000 7f8b04c20740 1 --2- 172.21.15.40:0/3604199432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a111440 0x55905a115f40 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f8aec036040 tx=0x7f8aec009660 comp rx=0 tx=0).stop 2024-03-18T22:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.419+0000 7f8b04c20740 1 -- 172.21.15.40:0/3604199432 shutdown_connections 2024-03-18T22:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.419+0000 7f8b04c20740 1 --2- 172.21.15.40:0/3604199432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a1165b0 0x55905a118a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.419+0000 7f8b04c20740 1 --2- 172.21.15.40:0/3604199432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a111440 0x55905a115f40 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.419+0000 7f8b04c20740 1 --2- 172.21.15.40:0/3604199432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a110a50 0x55905a110e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.419+0000 7f8b04c20740 1 -- 172.21.15.40:0/3604199432 >> 172.21.15.40:0/3604199432 conn(0x55905a10bfc0 msgr2=0x55905a10c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.420+0000 7f8b04c20740 1 -- 172.21.15.40:0/3604199432 shutdown_connections 2024-03-18T22:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.420+0000 7f8b04c20740 1 -- 172.21.15.40:0/3604199432 wait complete. 2024-03-18T22:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.420+0000 7f8b04c20740 1 -- start start 2024-03-18T22:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.420+0000 7f8b04c20740 1 Processor -- start 2024-03-18T22:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.420+0000 7f8b04c20740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a110a50 0x55905a11f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.420+0000 7f8b04c20740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a111440 0x55905a11f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.420+0000 7f8b04c20740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a1165b0 0x55905a1267d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.420+0000 7f8b04c20740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55905a11d910 con 0x55905a110a50 2024-03-18T22:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.420+0000 7f8b04c20740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55905a031160 con 0x55905a1165b0 2024-03-18T22:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.420+0000 7f8b04c20740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55905a11da90 con 0x55905a111440 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8afb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a110a50 0x55905a11f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8b04c20740 10 client.0 _ll_register_callbacks cb 0x55905a06da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8afbfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a1165b0 0x55905a1267d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8afb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a110a50 0x55905a11f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52096/0 (socket says 172.21.15.40:52096) 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8afb7fe640 1 -- 172.21.15.40:0/1583631576 learned_addr learned my addr 172.21.15.40:0/1583631576 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8afaffd640 1 --2- 172.21.15.40:0/1583631576 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a111440 0x55905a11f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8afbfff640 1 -- 172.21.15.40:0/1583631576 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a111440 msgr2=0x55905a11f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8afbfff640 1 --2- 172.21.15.40:0/1583631576 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a111440 0x55905a11f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8afbfff640 1 -- 172.21.15.40:0/1583631576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a110a50 msgr2=0x55905a11f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8afbfff640 1 --2- 172.21.15.40:0/1583631576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a110a50 0x55905a11f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8afbfff640 1 -- 172.21.15.40:0/1583631576 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559059e537d0 con 0x55905a1165b0 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8afbfff640 1 --2- 172.21.15.40:0/1583631576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a1165b0 0x55905a1267d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8af4003b80 tx=0x7f8af4004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cbdd938350a7c08 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8adaffd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8adaffd640 1 -- 172.21.15.40:0/1583631576 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4004250 con 0x55905a1165b0 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8adaffd640 1 -- 172.21.15.40:0/1583631576 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8af40043f0 con 0x55905a1165b0 2024-03-18T22:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8b04c20740 1 -- 172.21.15.40:0/1583631576 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8aec02cc90 con 0x55905a1165b0 2024-03-18T22:49:11.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8adaffd640 1 -- 172.21.15.40:0/1583631576 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8af40045f0 con 0x55905a1165b0 2024-03-18T22:49:11.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8b04c20740 10 client.27116 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:11.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.421+0000 7f8b04c20740 1 -- 172.21.15.40:0/1583631576 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559059d893f0 con 0x55905a1165b0 2024-03-18T22:49:11.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.422+0000 7f8adaffd640 1 -- 172.21.15.40:0/1583631576 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8aec02cc90 con 0x55905a1165b0 2024-03-18T22:49:11.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.422+0000 7f8adaffd640 1 -- 172.21.15.40:0/1583631576 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8af4026070 con 0x55905a1165b0 2024-03-18T22:49:11.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.422+0000 7f8b04c20740 10 client.27116 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:11.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.422+0000 7f8b04c20740 10 client.27116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:11.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.422+0000 7f8b04c20740 20 client.27116 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:11.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.422+0000 7f8b04c20740 1 --2- 172.21.15.40:0/1583631576 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55905a12b1a0 0x55905a12d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.422+0000 7f8b04c20740 4 client.27116 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:11.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.422+0000 7f8b04c20740 1 -- 172.21.15.40:0/1583631576 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55905a082a40 con 0x55905a12b1a0 2024-03-18T22:49:11.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.422+0000 7f8afb7fe640 1 --2- 172.21.15.40:0/1583631576 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55905a12b1a0 0x55905a12d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.422+0000 7f8adaffd640 10 client.27116 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:11.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.422+0000 7f8afb7fe640 1 --2- 172.21.15.40:0/1583631576 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55905a12b1a0 0x55905a12d600 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d04197a3c3ad564 server_cookie=dd00c7b7487014ba in_seq=0 out_seq=0 2024-03-18T22:49:11.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.423+0000 7f8adaffd640 1 -- 172.21.15.40:0/1583631576 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55905a082a40 con 0x55905a12b1a0 2024-03-18T22:49:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.423+0000 7f8adaffd640 10 client.27116 handle_command_reply: tid=0 2024-03-18T22:49:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.423+0000 7f8b04c20740 1 client.27116 shutdown 2024-03-18T22:49:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.423+0000 7f8b04c20740 1 -- 172.21.15.40:0/1583631576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a1165b0 msgr2=0x55905a1267d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.423+0000 7f8b04c20740 1 --2- 172.21.15.40:0/1583631576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a1165b0 0x55905a1267d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8af4003b80 tx=0x7f8af4004ab0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.423+0000 7f8b04c20740 1 -- 172.21.15.40:0/1583631576 shutdown_connections 2024-03-18T22:49:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.423+0000 7f8b04c20740 1 --2- 172.21.15.40:0/1583631576 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55905a12b1a0 0x55905a12d600 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.423+0000 7f8b04c20740 1 --2- 172.21.15.40:0/1583631576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a1165b0 0x55905a1267d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.423+0000 7f8b04c20740 1 --2- 172.21.15.40:0/1583631576 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a111440 0x55905a11f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.423+0000 7f8adaffd640 0 client.27116 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.423+0000 7f8b04c20740 1 --2- 172.21.15.40:0/1583631576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a110a50 0x55905a11f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.423+0000 7f8b04c20740 1 -- 172.21.15.40:0/1583631576 >> 172.21.15.40:0/1583631576 conn(0x55905a10bfc0 msgr2=0x55905a10b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.423+0000 7f8b04c20740 1 -- 172.21.15.40:0/1583631576 shutdown_connections 2024-03-18T22:49:11.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.423+0000 7f8b04c20740 1 -- 172.21.15.40:0/1583631576 wait complete. 2024-03-18T22:49:11.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.424+0000 7f8b04c20740 20 client.27116 trim_cache size 0 max 16384 2024-03-18T22:49:11.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.442+0000 7f8b04c20740 1 Processor -- start 2024-03-18T22:49:11.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.442+0000 7f8b04c20740 1 -- start start 2024-03-18T22:49:11.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.442+0000 7f8b04c20740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a110a50 0x55905a10a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.442+0000 7f8b04c20740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a111440 0x55905a126790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.442+0000 7f8b04c20740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a1165b0 0x55905a10ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.442+0000 7f8b04c20740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1a82e0 con 0x55905a111440 2024-03-18T22:49:11.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.442+0000 7f8b04c20740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1a9e20 con 0x55905a1165b0 2024-03-18T22:49:11.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.442+0000 7f8b04c20740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1a89b0 con 0x55905a110a50 2024-03-18T22:49:11.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.442+0000 7f8afaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a111440 0x55905a126790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.442+0000 7f8afb7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a110a50 0x55905a10a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.442+0000 7f8afaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a111440 0x55905a126790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52100/0 (socket says 172.21.15.40:52100) 2024-03-18T22:49:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.442+0000 7f8afb7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a110a50 0x55905a10a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:33052/0 (socket says 172.21.15.40:33052) 2024-03-18T22:49:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.442+0000 7f8afbfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a1165b0 0x55905a10ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.442+0000 7f8afaffd640 1 -- 172.21.15.40:0/3673265638 learned_addr learned my addr 172.21.15.40:0/3673265638 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.443+0000 7f8afbfff640 1 -- 172.21.15.40:0/3673265638 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a110a50 msgr2=0x55905a10a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.443+0000 7f8afbfff640 1 --2- 172.21.15.40:0/3673265638 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a110a50 0x55905a10a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.443+0000 7f8afbfff640 1 -- 172.21.15.40:0/3673265638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a111440 msgr2=0x55905a126790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.443+0000 7f8afbfff640 1 --2- 172.21.15.40:0/3673265638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a111440 0x55905a126790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.443+0000 7f8afbfff640 1 -- 172.21.15.40:0/3673265638 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559059d490f0 con 0x55905a1165b0 2024-03-18T22:49:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.443+0000 7f8afb7fe640 1 --2- 172.21.15.40:0/3673265638 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a110a50 0x55905a10a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.443+0000 7f8afbfff640 1 --2- 172.21.15.40:0/3673265638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a1165b0 0x55905a10ae90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8af4009d80 tx=0x7f8af4009320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=460029b75166024c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.443+0000 7f8ae17fa640 1 -- 172.21.15.40:0/3673265638 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4004250 con 0x55905a1165b0 2024-03-18T22:49:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.443+0000 7f8ae17fa640 1 -- 172.21.15.40:0/3673265638 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8af40043f0 con 0x55905a1165b0 2024-03-18T22:49:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.443+0000 7f8ae17fa640 1 -- 172.21.15.40:0/3673265638 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4028990 con 0x55905a1165b0 2024-03-18T22:49:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.444+0000 7f8b04c20740 1 -- 172.21.15.40:0/3673265638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a1165b0 msgr2=0x55905a10ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.444+0000 7f8b04c20740 1 --2- 172.21.15.40:0/3673265638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a1165b0 0x55905a10ae90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8af4009d80 tx=0x7f8af4009320 comp rx=0 tx=0).stop 2024-03-18T22:49:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.444+0000 7f8b04c20740 1 -- 172.21.15.40:0/3673265638 shutdown_connections 2024-03-18T22:49:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.444+0000 7f8b04c20740 1 --2- 172.21.15.40:0/3673265638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a1165b0 0x55905a10ae90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.444+0000 7f8b04c20740 1 --2- 172.21.15.40:0/3673265638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a111440 0x55905a126790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.444+0000 7f8b04c20740 1 --2- 172.21.15.40:0/3673265638 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a110a50 0x55905a10a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.444+0000 7f8b04c20740 1 -- 172.21.15.40:0/3673265638 >> 172.21.15.40:0/3673265638 conn(0x55905a10bfc0 msgr2=0x55905a11f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.444+0000 7f8b04c20740 1 -- 172.21.15.40:0/3673265638 shutdown_connections 2024-03-18T22:49:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.444+0000 7f8b04c20740 1 -- 172.21.15.40:0/3673265638 wait complete. 2024-03-18T22:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.444+0000 7f8b04c20740 1 -- start start 2024-03-18T22:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8b04c20740 1 Processor -- start 2024-03-18T22:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8b04c20740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a110a50 0x55905a124480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8b04c20740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a111440 0x55905a1249c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8b04c20740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a1165b0 0x55905a1259d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8b04c20740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1a9750 con 0x55905a110a50 2024-03-18T22:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8b04c20740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1a9080 con 0x55905a111440 2024-03-18T22:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8b04c20740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1a62c0 con 0x55905a1165b0 2024-03-18T22:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8b04c20740 10 client.0 _ll_register_callbacks cb 0x7f8af40114a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8afb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a110a50 0x55905a124480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8afb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a110a50 0x55905a124480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52116/0 (socket says 172.21.15.40:52116) 2024-03-18T22:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8afb7fe640 1 -- 172.21.15.40:0/1903772182 learned_addr learned my addr 172.21.15.40:0/1903772182 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8afaffd640 1 --2- 172.21.15.40:0/1903772182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a111440 0x55905a1249c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8afbfff640 1 --2- 172.21.15.40:0/1903772182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a1165b0 0x55905a1259d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8afb7fe640 1 -- 172.21.15.40:0/1903772182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a1165b0 msgr2=0x55905a1259d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8afb7fe640 1 --2- 172.21.15.40:0/1903772182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a1165b0 0x55905a1259d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8afb7fe640 1 -- 172.21.15.40:0/1903772182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a111440 msgr2=0x55905a1249c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8afb7fe640 1 --2- 172.21.15.40:0/1903772182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a111440 0x55905a1249c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8afb7fe640 1 -- 172.21.15.40:0/1903772182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55905a125fc0 con 0x55905a110a50 2024-03-18T22:49:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8afaffd640 1 --2- 172.21.15.40:0/1903772182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a111440 0x55905a1249c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8afb7fe640 1 --2- 172.21.15.40:0/1903772182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a110a50 0x55905a124480 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f8aec036040 tx=0x7f8aec02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d53e3c527cfb064 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.445+0000 7f8adaffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.446+0000 7f8adaffd640 1 -- 172.21.15.40:0/1903772182 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec045040 con 0x55905a110a50 2024-03-18T22:49:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.446+0000 7f8adaffd640 1 -- 172.21.15.40:0/1903772182 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8aec0046c0 con 0x55905a110a50 2024-03-18T22:49:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.446+0000 7f8b04c20740 1 -- 172.21.15.40:0/1903772182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8af40043f0 con 0x55905a110a50 2024-03-18T22:49:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.446+0000 7f8adaffd640 1 -- 172.21.15.40:0/1903772182 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8aec05e050 con 0x55905a110a50 2024-03-18T22:49:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.446+0000 7f8b04c20740 10 client.17304 fetch_fsmap learned FSMap version 32 2024-03-18T22:49:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.446+0000 7f8b04c20740 1 -- 172.21.15.40:0/1903772182 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55905a191ab0 con 0x55905a110a50 2024-03-18T22:49:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.446+0000 7f8adaffd640 1 -- 172.21.15.40:0/1903772182 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8af40043f0 con 0x55905a110a50 2024-03-18T22:49:11.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.446+0000 7f8adaffd640 1 -- 172.21.15.40:0/1903772182 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8af4011d20 con 0x55905a110a50 2024-03-18T22:49:11.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.446+0000 7f8b04c20740 10 client.17304 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T22:49:11.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.446+0000 7f8b04c20740 10 client.17304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:11.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.446+0000 7f8b04c20740 20 client.17304 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:11.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.446+0000 7f8b04c20740 1 --2- 172.21.15.40:0/1903772182 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55905a192070 0x55905a194460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:11.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.446+0000 7f8b04c20740 4 client.17304 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:49:11.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.446+0000 7f8b04c20740 1 -- 172.21.15.40:0/1903772182 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55905a07cd90 con 0x55905a192070 2024-03-18T22:49:11.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.446+0000 7f8afaffd640 1 --2- 172.21.15.40:0/1903772182 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55905a192070 0x55905a194460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:11.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8afaffd640 1 --2- 172.21.15.40:0/1903772182 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55905a192070 0x55905a194460 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fda4aaf11516d49f server_cookie=7a992c20d1dd6a8a in_seq=0 out_seq=0 2024-03-18T22:49:11.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8adaffd640 10 client.17304 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:11.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8adaffd640 1 -- 172.21.15.40:0/1903772182 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55905a07cd90 con 0x55905a192070 2024-03-18T22:49:11.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8adaffd640 10 client.17304 handle_command_reply: tid=0 2024-03-18T22:49:11.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8b04c20740 1 client.17304 shutdown 2024-03-18T22:49:11.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8b04c20740 1 -- 172.21.15.40:0/1903772182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a110a50 msgr2=0x55905a124480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8b04c20740 1 --2- 172.21.15.40:0/1903772182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a110a50 0x55905a124480 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f8aec036040 tx=0x7f8aec02bc30 comp rx=0 tx=0).stop 2024-03-18T22:49:11.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8b04c20740 1 -- 172.21.15.40:0/1903772182 shutdown_connections 2024-03-18T22:49:11.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8b04c20740 1 --2- 172.21.15.40:0/1903772182 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55905a1165b0 0x55905a1259d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8b04c20740 1 --2- 172.21.15.40:0/1903772182 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55905a192070 0x55905a194460 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8b04c20740 1 --2- 172.21.15.40:0/1903772182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55905a111440 0x55905a1249c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8b04c20740 1 --2- 172.21.15.40:0/1903772182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55905a110a50 0x55905a124480 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8adaffd640 0 client.17304 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:11.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8b04c20740 1 -- 172.21.15.40:0/1903772182 >> 172.21.15.40:0/1903772182 conn(0x55905a10bfc0 msgr2=0x55905a10f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:11.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8b04c20740 1 -- 172.21.15.40:0/1903772182 shutdown_connections 2024-03-18T22:49:11.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8b04c20740 1 -- 172.21.15.40:0/1903772182 wait complete. 2024-03-18T22:49:11.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.447+0000 7f8b04c20740 20 client.17304 trim_cache size 0 max 16384 2024-03-18T22:49:11.455 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:49:11.455 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:49:11.455 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:49:11.455 INFO:teuthology.orchestra.run.smithi040.stdout: "08c4afe1-367c-4b26-a212-50159d3b5875": { 2024-03-18T22:49:11.455 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:49:11.455 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "08c4afe1-367c-4b26-a212-50159d3b5875", 2024-03-18T22:49:11.455 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:49:11.455 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:49:11.455 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:49:11.455 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:49:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.448+0000 7f8b01f60640 1 -- 172.21.15.40:0/2968122680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8ae40a2800 msgr2=0x7f8ae40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.448+0000 7f8b01f60640 1 --2- 172.21.15.40:0/2968122680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8ae40a2800 0x7f8ae40a4cc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x55905a111ca0 tx=0x7f8af4009500 comp rx=0 tx=0).stop 2024-03-18T22:49:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.448+0000 7f8b01f60640 1 -- 172.21.15.40:0/2968122680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8afc100270 msgr2=0x7f8afc198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.448+0000 7f8b01f60640 1 --2- 172.21.15.40:0/2968122680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8afc100270 0x7f8afc198000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8af000e9f0 tx=0x7f8af000eeb0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.448+0000 7f8b01f60640 1 -- 172.21.15.40:0/2968122680 shutdown_connections 2024-03-18T22:49:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.448+0000 7f8b01f60640 1 --2- 172.21.15.40:0/2968122680 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8ae40a2800 0x7f8ae40a4cc0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.448+0000 7f8b01f60640 1 --2- 172.21.15.40:0/2968122680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8afc102e40 0x7f8afc19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.448+0000 7f8b01f60640 1 --2- 172.21.15.40:0/2968122680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8afc100270 0x7f8afc198000 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.448+0000 7f8b01f60640 1 --2- 172.21.15.40:0/2968122680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8afc0ff8b0 0x7f8afc197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.448+0000 7f8b01f60640 1 -- 172.21.15.40:0/2968122680 >> 172.21.15.40:0/2968122680 conn(0x7f8afc0fb410 msgr2=0x7f8afc105fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.448+0000 7f8b01f60640 1 -- 172.21.15.40:0/2968122680 shutdown_connections 2024-03-18T22:49:11.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:11.448+0000 7f8b01f60640 1 -- 172.21.15.40:0/2968122680 wait complete. 2024-03-18T22:49:11.459 INFO:tasks.cephfs.filesystem:scrub status for tag:08c4afe1-367c-4b26-a212-50159d3b5875 - {'path': '/', 'tag': '08c4afe1-367c-4b26-a212-50159d3b5875', 'options': 'recursive,force'} 2024-03-18T22:49:12.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:12 smithi113 ceph-mon[66775]: pgmap v547: 129 pgs: 129 active+clean; 8.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:49:12.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:12 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:49:12.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:12 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:12.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:12 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:49:12.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:12 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:12.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:12 smithi103 ceph-mon[66377]: pgmap v547: 129 pgs: 129 active+clean; 8.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:49:12.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:12 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:49:12.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:12 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:12.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:12 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:49:12.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:12 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:12.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:12 smithi040 ceph-mon[62643]: pgmap v547: 129 pgs: 129 active+clean; 8.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:49:12.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:12 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:49:12.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:12 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:12.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:12 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:49:12.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:12 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:13.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:13 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:49:13.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:13 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:13.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:13 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:49:13.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:13 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:13.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:13 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:49:13.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:13 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:14.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:14 smithi113 ceph-mon[66775]: pgmap v548: 129 pgs: 129 active+clean; 8.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:49:14.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:14 smithi103 ceph-mon[66377]: pgmap v548: 129 pgs: 129 active+clean; 8.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:49:14.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:14 smithi040 ceph-mon[62643]: pgmap v548: 129 pgs: 129 active+clean; 8.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:49:16.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:16 smithi113 ceph-mon[66775]: pgmap v549: 129 pgs: 129 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:49:16.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:16 smithi103 ceph-mon[66377]: pgmap v549: 129 pgs: 129 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:49:16.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:16 smithi040 ceph-mon[62643]: pgmap v549: 129 pgs: 129 active+clean; 8.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:49:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:18 smithi113 ceph-mon[66775]: pgmap v550: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:49:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:18 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:49:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:18 smithi103 ceph-mon[66377]: pgmap v550: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:49:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:18 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:49:18.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:18 smithi040 ceph-mon[62643]: pgmap v550: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:49:18.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:18 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:49:19.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:19 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:49:19.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:19 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:49:19.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:19 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:49:20.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:20 smithi113 ceph-mon[66775]: pgmap v551: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:49:20.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:20 smithi103 ceph-mon[66377]: pgmap v551: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:49:20.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:20 smithi040 ceph-mon[62643]: pgmap v551: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:49:21.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:21 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:49:21.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:21 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:49:21.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:21 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:49:22.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:22 smithi113 ceph-mon[66775]: pgmap v552: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:49:22.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:22 smithi103 ceph-mon[66377]: pgmap v552: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:49:22.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:22 smithi040 ceph-mon[62643]: pgmap v552: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:49:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:23 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:49:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:23 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:49:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:23 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:49:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:23 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:49:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:23 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:49:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:23 smithi113 ceph-mon[66775]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-03-18T22:49:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:23 smithi113 ceph-mon[66775]: Unable to set osd_memory_target on smithi040 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-03-18T22:49:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:23 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:49:23.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:23 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:49:23.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:23 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:49:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:23 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:49:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:23 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:49:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:23 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:49:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:23 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:49:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:23 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:49:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:23 smithi103 ceph-mon[66377]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-03-18T22:49:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:23 smithi103 ceph-mon[66377]: Unable to set osd_memory_target on smithi040 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-03-18T22:49:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:23 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:49:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:23 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:49:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:23 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:49:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:49:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:49:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:49:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:49:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:49:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:23 smithi040 ceph-mon[62643]: Adjusting osd_memory_target on smithi040 to 177.2M 2024-03-18T22:49:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:23 smithi040 ceph-mon[62643]: Unable to set osd_memory_target on smithi040 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-03-18T22:49:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:49:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:49:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:49:25.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:24 smithi040 ceph-mon[62643]: pgmap v553: 129 pgs: 129 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 22 op/s 2024-03-18T22:49:25.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:24 smithi113 ceph-mon[66775]: pgmap v553: 129 pgs: 129 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 22 op/s 2024-03-18T22:49:25.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:24 smithi103 ceph-mon[66377]: pgmap v553: 129 pgs: 129 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 22 op/s 2024-03-18T22:49:26.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:25 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:49:26.231 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:25 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:49:26.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:25 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:49:27.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:26 smithi040 ceph-mon[62643]: pgmap v554: 129 pgs: 129 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 MiB/s wr, 24 op/s 2024-03-18T22:49:27.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:26 smithi113 ceph-mon[66775]: pgmap v554: 129 pgs: 129 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 MiB/s wr, 24 op/s 2024-03-18T22:49:27.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:26 smithi103 ceph-mon[66377]: pgmap v554: 129 pgs: 129 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 MiB/s wr, 24 op/s 2024-03-18T22:49:29.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:28 smithi040 ceph-mon[62643]: pgmap v555: 129 pgs: 129 active+clean; 8.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:49:29.160 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:28 smithi103 ceph-mon[66377]: pgmap v555: 129 pgs: 129 active+clean; 8.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:49:29.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:28 smithi113 ceph-mon[66775]: pgmap v555: 129 pgs: 129 active+clean; 8.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:49:31.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:30 smithi040 ceph-mon[62643]: pgmap v556: 129 pgs: 129 active+clean; 8.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:49:31.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:30 smithi113 ceph-mon[66775]: pgmap v556: 129 pgs: 129 active+clean; 8.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:49:31.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:30 smithi103 ceph-mon[66377]: pgmap v556: 129 pgs: 129 active+clean; 8.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:49:32.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:31 smithi113 ceph-mon[66775]: pgmap v557: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:49:32.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:31 smithi103 ceph-mon[66377]: pgmap v557: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:49:32.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:31 smithi040 ceph-mon[62643]: pgmap v557: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:49:34.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:33 smithi113 ceph-mon[66775]: pgmap v558: 129 pgs: 129 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 30 MiB/s wr, 23 op/s 2024-03-18T22:49:34.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:33 smithi103 ceph-mon[66377]: pgmap v558: 129 pgs: 129 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 30 MiB/s wr, 23 op/s 2024-03-18T22:49:34.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:33 smithi040 ceph-mon[62643]: pgmap v558: 129 pgs: 129 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 30 MiB/s wr, 23 op/s 2024-03-18T22:49:36.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:35 smithi113 ceph-mon[66775]: pgmap v559: 129 pgs: 129 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 MiB/s wr, 23 op/s 2024-03-18T22:49:36.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:35 smithi103 ceph-mon[66377]: pgmap v559: 129 pgs: 129 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 MiB/s wr, 23 op/s 2024-03-18T22:49:36.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:35 smithi040 ceph-mon[62643]: pgmap v559: 129 pgs: 129 active+clean; 8.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 MiB/s wr, 23 op/s 2024-03-18T22:49:38.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:38 smithi113 ceph-mon[66775]: pgmap v560: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:49:38.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:38 smithi103 ceph-mon[66377]: pgmap v560: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:49:38.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:38 smithi040 ceph-mon[62643]: pgmap v560: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:49:40.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:40 smithi113 ceph-mon[66775]: pgmap v561: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:49:40.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:40 smithi103 ceph-mon[66377]: pgmap v561: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:49:40.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:40 smithi040 ceph-mon[62643]: pgmap v561: 129 pgs: 129 active+clean; 8.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:49:41.461 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:49:41.461 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:49:41.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.579+0000 7f63ee2dc640 1 -- 172.21.15.40:0/3943262370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63e81033e0 msgr2=0x7f63e8109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:41.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.579+0000 7f63ee2dc640 1 --2- 172.21.15.40:0/3943262370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63e81033e0 0x7f63e8109c70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f63e0009ca0 tx=0x7f63e002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.579+0000 7f63ee2dc640 1 -- 172.21.15.40:0/3943262370 shutdown_connections 2024-03-18T22:49:41.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.579+0000 7f63ee2dc640 1 --2- 172.21.15.40:0/3943262370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63e81033e0 0x7f63e8109c70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.579+0000 7f63ee2dc640 1 --2- 172.21.15.40:0/3943262370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e8102a20 0x7f63e8102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.579+0000 7f63ee2dc640 1 --2- 172.21.15.40:0/3943262370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e8101830 0x7f63e8101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.579+0000 7f63ee2dc640 1 -- 172.21.15.40:0/3943262370 >> 172.21.15.40:0/3943262370 conn(0x7f63e80fcda0 msgr2=0x7f63e80ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:41.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.579+0000 7f63ee2dc640 1 -- 172.21.15.40:0/3943262370 shutdown_connections 2024-03-18T22:49:41.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.579+0000 7f63ee2dc640 1 -- 172.21.15.40:0/3943262370 wait complete. 2024-03-18T22:49:41.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.579+0000 7f63ee2dc640 1 Processor -- start 2024-03-18T22:49:41.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.579+0000 7f63ee2dc640 1 -- start start 2024-03-18T22:49:41.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63ee2dc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63e8101830 0x7f63e8197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63ee2dc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e8102a20 0x7f63e8197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63ee2dc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e81033e0 0x7f63e819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63ee2dc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63e810d340 con 0x7f63e8102a20 2024-03-18T22:49:41.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63ee2dc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63e810d1c0 con 0x7f63e8101830 2024-03-18T22:49:41.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63ee2dc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f63e810d4c0 con 0x7f63e81033e0 2024-03-18T22:49:41.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63e77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e8102a20 0x7f63e8197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63e77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e8102a20 0x7f63e8197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33180/0 (socket says 172.21.15.40:33180) 2024-03-18T22:49:41.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63e77fe640 1 -- 172.21.15.40:0/2731044681 learned_addr learned my addr 172.21.15.40:0/2731044681 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:41.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63e7fff640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63e8101830 0x7f63e8197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63ec879640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e81033e0 0x7f63e819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63e77fe640 1 -- 172.21.15.40:0/2731044681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e81033e0 msgr2=0x7f63e819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:41.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63e77fe640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e81033e0 0x7f63e819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63e77fe640 1 -- 172.21.15.40:0/2731044681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63e8101830 msgr2=0x7f63e8197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:41.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63e77fe640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63e8101830 0x7f63e8197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63e77fe640 1 -- 172.21.15.40:0/2731044681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e819f610 con 0x7f63e8102a20 2024-03-18T22:49:41.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63ec879640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e81033e0 0x7f63e819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:41.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.580+0000 7f63e7fff640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63e8101830 0x7f63e8197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:41.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.581+0000 7f63e77fe640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e8102a20 0x7f63e8197f80 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f63d800ea60 tx=0x7f63d800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:41.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.581+0000 7f63e57fa640 1 -- 172.21.15.40:0/2731044681 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d800cde0 con 0x7f63e8102a20 2024-03-18T22:49:41.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.581+0000 7f63ee2dc640 1 -- 172.21.15.40:0/2731044681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63e819f900 con 0x7f63e8102a20 2024-03-18T22:49:41.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.581+0000 7f63e57fa640 1 -- 172.21.15.40:0/2731044681 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63d80035f0 con 0x7f63e8102a20 2024-03-18T22:49:41.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.581+0000 7f63e57fa640 1 -- 172.21.15.40:0/2731044681 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8010690 con 0x7f63e8102a20 2024-03-18T22:49:41.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.581+0000 7f63ee2dc640 1 -- 172.21.15.40:0/2731044681 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63e819fea0 con 0x7f63e8102a20 2024-03-18T22:49:41.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.583+0000 7f63f0fa6740 1 Processor -- start 2024-03-18T22:49:41.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63f0fa6740 1 -- start start 2024-03-18T22:49:41.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63e57fa640 1 -- 172.21.15.40:0/2731044681 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f63d80037a0 con 0x7f63e8102a20 2024-03-18T22:49:41.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63f0fa6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d77da50 0x556c7d77de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63f0fa6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d782f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63f0fa6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d7835b0 0x556c7d785a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63f0fa6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556c7d6e2500 con 0x556c7d77da50 2024-03-18T22:49:41.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63f0fa6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556c7d68abe0 con 0x556c7d7835b0 2024-03-18T22:49:41.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63f0fa6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556c7d6de5a0 con 0x556c7d77e440 2024-03-18T22:49:41.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63ec879640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d77da50 0x556c7d77de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63ec879640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d77da50 0x556c7d77de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33196/0 (socket says 172.21.15.40:33196) 2024-03-18T22:49:41.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63e7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d782f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63ec879640 1 -- 172.21.15.40:0/3942035884 learned_addr learned my addr 172.21.15.40:0/3942035884 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63e77fe640 1 --2- 172.21.15.40:0/3942035884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d7835b0 0x556c7d785a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63e7fff640 1 -- 172.21.15.40:0/3942035884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d7835b0 msgr2=0x556c7d785a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63e7fff640 1 --2- 172.21.15.40:0/3942035884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d7835b0 0x556c7d785a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63e7fff640 1 -- 172.21.15.40:0/3942035884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d77da50 msgr2=0x556c7d77de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63e7fff640 1 --2- 172.21.15.40:0/3942035884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d77da50 0x556c7d77de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63e7fff640 1 -- 172.21.15.40:0/3942035884 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c7d6ac000 con 0x556c7d77e440 2024-03-18T22:49:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63e57fa640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63d40a2800 0x7f63d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.584+0000 7f63e7fff640 1 --2- 172.21.15.40:0/3942035884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d782f40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009fa0 tx=0x7f63d00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f215bec99aee0d33 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.585+0000 7f63ec879640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63d40a2800 0x7f63d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.585+0000 7f63e57fa640 1 -- 172.21.15.40:0/2731044681 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f63d80cf0e0 con 0x7f63e8102a20 2024-03-18T22:49:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.585+0000 7f63de7fc640 1 -- 172.21.15.40:0/3942035884 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d000f5a0 con 0x556c7d77e440 2024-03-18T22:49:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.585+0000 7f63de7fc640 1 -- 172.21.15.40:0/3942035884 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63d000fbf0 con 0x556c7d77e440 2024-03-18T22:49:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.585+0000 7f63de7fc640 1 -- 172.21.15.40:0/3942035884 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0019b90 con 0x556c7d77e440 2024-03-18T22:49:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.585+0000 7f63f0fa6740 1 -- 172.21.15.40:0/3942035884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 msgr2=0x556c7d782f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.585+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/3942035884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d782f40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009fa0 tx=0x7f63d00118d0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.585+0000 7f63ec879640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63d40a2800 0x7f63d40a4cc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x556c7d77eca0 tx=0x7f63e00324f0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.585+0000 7f63f0fa6740 1 -- 172.21.15.40:0/3942035884 shutdown_connections 2024-03-18T22:49:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.585+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/3942035884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d7835b0 0x556c7d785a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.585+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/3942035884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d782f40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.585+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/3942035884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d77da50 0x556c7d77de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.585+0000 7f63f0fa6740 1 -- 172.21.15.40:0/3942035884 >> 172.21.15.40:0/3942035884 conn(0x556c7d778fc0 msgr2=0x556c7d7793c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63f0fa6740 1 -- 172.21.15.40:0/3942035884 shutdown_connections 2024-03-18T22:49:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63f0fa6740 1 -- 172.21.15.40:0/3942035884 wait complete. 2024-03-18T22:49:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63f0fa6740 1 -- start start 2024-03-18T22:49:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63f0fa6740 1 Processor -- start 2024-03-18T22:49:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63f0fa6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d77da50 0x556c7d787f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63f0fa6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d788440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63f0fa6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d7835b0 0x556c7d78f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63f0fa6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556c7d786560 con 0x556c7d7835b0 2024-03-18T22:49:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63f0fa6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556c7d69e160 con 0x556c7d77da50 2024-03-18T22:49:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63f0fa6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556c7d7866e0 con 0x556c7d77e440 2024-03-18T22:49:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63f0fa6740 10 client.0 _ll_register_callbacks cb 0x556c7d6daa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63ec879640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d7835b0 0x556c7d78f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63e7fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d77da50 0x556c7d787f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63e7fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d77da50 0x556c7d787f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:41836/0 (socket says 172.21.15.40:41836) 2024-03-18T22:49:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63e77fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d788440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63e7fff640 1 -- 172.21.15.40:0/454796968 learned_addr learned my addr 172.21.15.40:0/454796968 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.586+0000 7f63ec879640 1 --2- 172.21.15.40:0/454796968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d7835b0 0x556c7d78f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33206/0 (socket says 172.21.15.40:33206) 2024-03-18T22:49:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.587+0000 7f63e7fff640 1 -- 172.21.15.40:0/454796968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 msgr2=0x556c7d788440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.587+0000 7f63e7fff640 1 --2- 172.21.15.40:0/454796968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d788440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.587+0000 7f63e7fff640 1 -- 172.21.15.40:0/454796968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d7835b0 msgr2=0x556c7d78f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:41.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.587+0000 7f63e7fff640 1 --2- 172.21.15.40:0/454796968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d7835b0 0x556c7d78f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.587+0000 7f63e7fff640 1 -- 172.21.15.40:0/454796968 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c7d4c07d0 con 0x556c7d77da50 2024-03-18T22:49:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.587+0000 7f63e77fe640 1 --2- 172.21.15.40:0/454796968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d788440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.587+0000 7f63ec879640 1 --2- 172.21.15.40:0/454796968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d7835b0 0x556c7d78f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.587+0000 7f63e7fff640 1 --2- 172.21.15.40:0/454796968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d77da50 0x556c7d787f00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f63d001b040 tx=0x7f63d0009cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2cb625d02236e98 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.587+0000 7f63c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:49:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.587+0000 7f63c77fe640 1 -- 172.21.15.40:0/454796968 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0027020 con 0x556c7d77da50 2024-03-18T22:49:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.587+0000 7f63f0fa6740 1 -- 172.21.15.40:0/454796968 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63d000fbf0 con 0x556c7d77da50 2024-03-18T22:49:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.587+0000 7f63c77fe640 1 -- 172.21.15.40:0/454796968 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f63d000f5a0 con 0x556c7d77da50 2024-03-18T22:49:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.587+0000 7f63c77fe640 1 -- 172.21.15.40:0/454796968 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f63d0032050 con 0x556c7d77da50 2024-03-18T22:49:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.587+0000 7f63f0fa6740 10 client.27134 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.587+0000 7f63f0fa6740 1 -- 172.21.15.40:0/454796968 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x556c7d3f63f0 con 0x556c7d77da50 2024-03-18T22:49:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.588+0000 7f63c77fe640 1 -- 172.21.15.40:0/454796968 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63d000fbf0 con 0x556c7d77da50 2024-03-18T22:49:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.588+0000 7f63c77fe640 1 -- 172.21.15.40:0/454796968 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f63d0005160 con 0x556c7d77da50 2024-03-18T22:49:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.588+0000 7f63f0fa6740 10 client.27134 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.588+0000 7f63f0fa6740 10 client.27134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:41.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.588+0000 7f63f0fa6740 20 client.27134 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:41.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.588+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/454796968 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556c7d793ae0 0x556c7d795f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.588+0000 7f63f0fa6740 4 client.27134 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:41.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.588+0000 7f63f0fa6740 1 -- 172.21.15.40:0/454796968 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c7d6efa40 con 0x556c7d793ae0 2024-03-18T22:49:41.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.588+0000 7f63e77fe640 1 --2- 172.21.15.40:0/454796968 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556c7d793ae0 0x556c7d795f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.588+0000 7f63e77fe640 1 --2- 172.21.15.40:0/454796968 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556c7d793ae0 0x556c7d795f80 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bcb1db498eaa3e5 server_cookie=f8191bcc9082931 in_seq=0 out_seq=0 2024-03-18T22:49:41.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.588+0000 7f63c77fe640 10 client.27134 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:41.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.589+0000 7f63c77fe640 1 -- 172.21.15.40:0/454796968 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556c7d6efa40 con 0x556c7d793ae0 2024-03-18T22:49:41.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.589+0000 7f63c77fe640 10 client.27134 handle_command_reply: tid=0 2024-03-18T22:49:41.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.589+0000 7f63f0fa6740 1 client.27134 shutdown 2024-03-18T22:49:41.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.590+0000 7f63f0fa6740 1 -- 172.21.15.40:0/454796968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d77da50 msgr2=0x556c7d787f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:41.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.590+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/454796968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d77da50 0x556c7d787f00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f63d001b040 tx=0x7f63d0009cc0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.590+0000 7f63f0fa6740 1 -- 172.21.15.40:0/454796968 shutdown_connections 2024-03-18T22:49:41.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.590+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/454796968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d7835b0 0x556c7d78f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.590+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/454796968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d788440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.590+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/454796968 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556c7d793ae0 0x556c7d795f80 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.590+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/454796968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d77da50 0x556c7d787f00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.590+0000 7f63f0fa6740 1 -- 172.21.15.40:0/454796968 >> 172.21.15.40:0/454796968 conn(0x556c7d778fc0 msgr2=0x556c7d7781e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:41.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.590+0000 7f63c77fe640 0 client.27134 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:41.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.590+0000 7f63f0fa6740 1 -- 172.21.15.40:0/454796968 shutdown_connections 2024-03-18T22:49:41.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.590+0000 7f63f0fa6740 1 -- 172.21.15.40:0/454796968 wait complete. 2024-03-18T22:49:41.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.590+0000 7f63f0fa6740 20 client.27134 trim_cache size 0 max 16384 2024-03-18T22:49:41.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.610+0000 7f63f0fa6740 1 Processor -- start 2024-03-18T22:49:41.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.610+0000 7f63f0fa6740 1 -- start start 2024-03-18T22:49:41.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.610+0000 7f63f0fa6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77da50 0x556c7d773f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.610+0000 7f63f0fa6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d77e440 0x556c7d78f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.610+0000 7f63f0fa6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d7835b0 0x556c7d787520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.610+0000 7f63f0fa6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63e81aa520 con 0x556c7d77e440 2024-03-18T22:49:41.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.610+0000 7f63f0fa6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63e81ac120 con 0x556c7d7835b0 2024-03-18T22:49:41.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.610+0000 7f63f0fa6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f63e81aabf0 con 0x556c7d77da50 2024-03-18T22:49:41.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.610+0000 7f63e7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77da50 0x556c7d773f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.610+0000 7f63e77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d77e440 0x556c7d78f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.610+0000 7f63e7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77da50 0x556c7d773f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:60330/0 (socket says 172.21.15.40:60330) 2024-03-18T22:49:41.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.610+0000 7f63e7fff640 1 -- 172.21.15.40:0/1204640714 learned_addr learned my addr 172.21.15.40:0/1204640714 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:41.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.610+0000 7f63ec879640 1 --2- 172.21.15.40:0/1204640714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d7835b0 0x556c7d787520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.611+0000 7f63e7fff640 1 -- 172.21.15.40:0/1204640714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d7835b0 msgr2=0x556c7d787520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:41.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.611+0000 7f63e7fff640 1 --2- 172.21.15.40:0/1204640714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d7835b0 0x556c7d787520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.611+0000 7f63e7fff640 1 -- 172.21.15.40:0/1204640714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d77e440 msgr2=0x556c7d78f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:41.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.611+0000 7f63e7fff640 1 --2- 172.21.15.40:0/1204640714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d77e440 0x556c7d78f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.611+0000 7f63e7fff640 1 -- 172.21.15.40:0/1204640714 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c7d3b60f0 con 0x556c7d77da50 2024-03-18T22:49:41.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.611+0000 7f63e77fe640 1 --2- 172.21.15.40:0/1204640714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d77e440 0x556c7d78f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:41.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.611+0000 7f63e7fff640 1 --2- 172.21.15.40:0/1204640714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77da50 0x556c7d773f80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f63d0000c00 tx=0x7f63d0004810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c904e5a31cf028cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:41.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.611+0000 7f63ddffb640 1 -- 172.21.15.40:0/1204640714 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0027070 con 0x556c7d77da50 2024-03-18T22:49:41.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.611+0000 7f63ddffb640 1 -- 172.21.15.40:0/1204640714 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63d0005240 con 0x556c7d77da50 2024-03-18T22:49:41.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.611+0000 7f63ddffb640 1 -- 172.21.15.40:0/1204640714 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0019ca0 con 0x556c7d77da50 2024-03-18T22:49:41.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.612+0000 7f63f0fa6740 1 -- 172.21.15.40:0/1204640714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77da50 msgr2=0x556c7d773f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:41.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.612+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/1204640714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77da50 0x556c7d773f80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f63d0000c00 tx=0x7f63d0004810 comp rx=0 tx=0).stop 2024-03-18T22:49:41.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.612+0000 7f63f0fa6740 1 -- 172.21.15.40:0/1204640714 shutdown_connections 2024-03-18T22:49:41.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.612+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/1204640714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d7835b0 0x556c7d787520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.612+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/1204640714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d77e440 0x556c7d78f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.612+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/1204640714 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77da50 0x556c7d773f80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.612+0000 7f63f0fa6740 1 -- 172.21.15.40:0/1204640714 >> 172.21.15.40:0/1204640714 conn(0x556c7d778fc0 msgr2=0x556c7d7781e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:41.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.612+0000 7f63f0fa6740 1 -- 172.21.15.40:0/1204640714 shutdown_connections 2024-03-18T22:49:41.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.612+0000 7f63f0fa6740 1 -- 172.21.15.40:0/1204640714 wait complete. 2024-03-18T22:49:41.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.612+0000 7f63f0fa6740 1 -- start start 2024-03-18T22:49:41.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.612+0000 7f63f0fa6740 1 Processor -- start 2024-03-18T22:49:41.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63f0fa6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d77da50 0x556c7d78c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63f0fa6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d78c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63f0fa6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d7835b0 0x556c7d7f7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63f0fa6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63e81aba50 con 0x556c7d7835b0 2024-03-18T22:49:41.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63f0fa6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63e81ab2c0 con 0x556c7d77da50 2024-03-18T22:49:41.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63f0fa6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f63e81a8500 con 0x556c7d77e440 2024-03-18T22:49:41.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63f0fa6740 10 client.0 _ll_register_callbacks cb 0x7f63d8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:41.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63ec879640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d7835b0 0x556c7d7f7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63e77fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d78c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63ec879640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d7835b0 0x556c7d7f7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33214/0 (socket says 172.21.15.40:33214) 2024-03-18T22:49:41.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63e77fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d78c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:60340/0 (socket says 172.21.15.40:60340) 2024-03-18T22:49:41.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63ec879640 1 -- 172.21.15.40:0/1115760212 learned_addr learned my addr 172.21.15.40:0/1115760212 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:41.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63e7fff640 1 --2- 172.21.15.40:0/1115760212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d77da50 0x556c7d78c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63ec879640 1 -- 172.21.15.40:0/1115760212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 msgr2=0x556c7d78c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:41.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63ec879640 1 --2- 172.21.15.40:0/1115760212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d78c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63ec879640 1 -- 172.21.15.40:0/1115760212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d77da50 msgr2=0x556c7d78c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:41.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63ec879640 1 --2- 172.21.15.40:0/1115760212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d77da50 0x556c7d78c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63ec879640 1 -- 172.21.15.40:0/1115760212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c7d78f570 con 0x556c7d7835b0 2024-03-18T22:49:41.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63e7fff640 1 --2- 172.21.15.40:0/1115760212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d77da50 0x556c7d78c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:41.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63e77fe640 1 --2- 172.21.15.40:0/1115760212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d78c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:41.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63ec879640 1 --2- 172.21.15.40:0/1115760212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d7835b0 0x556c7d7f7e70 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f63e002c700 tx=0x7f63e00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9468abd87633d9db server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:41.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.613+0000 7f63c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:41.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.614+0000 7f63c77fe640 1 -- 172.21.15.40:0/1115760212 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e005c060 con 0x556c7d7835b0 2024-03-18T22:49:41.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.614+0000 7f63c77fe640 1 -- 172.21.15.40:0/1115760212 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f63e0034440 con 0x556c7d7835b0 2024-03-18T22:49:41.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.614+0000 7f63f0fa6740 1 -- 172.21.15.40:0/1115760212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63d0005240 con 0x556c7d7835b0 2024-03-18T22:49:41.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.614+0000 7f63c77fe640 1 -- 172.21.15.40:0/1115760212 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f63e0039040 con 0x556c7d7835b0 2024-03-18T22:49:41.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.614+0000 7f63f0fa6740 10 client.17322 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:41.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.614+0000 7f63f0fa6740 1 -- 172.21.15.40:0/1115760212 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x556c7d7f9320 con 0x556c7d7835b0 2024-03-18T22:49:41.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.614+0000 7f63c77fe640 1 -- 172.21.15.40:0/1115760212 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63d0005240 con 0x556c7d7835b0 2024-03-18T22:49:41.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.614+0000 7f63c77fe640 1 -- 172.21.15.40:0/1115760212 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f63e00405f0 con 0x556c7d7835b0 2024-03-18T22:49:41.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.614+0000 7f63f0fa6740 10 client.17322 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:41.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.614+0000 7f63f0fa6740 10 client.17322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:41.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.614+0000 7f63f0fa6740 20 client.17322 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:41.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.614+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/1115760212 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556c7d7f9d50 0x556c7d7fc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.614+0000 7f63f0fa6740 4 client.17322 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:49:41.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.614+0000 7f63f0fa6740 1 -- 172.21.15.40:0/1115760212 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556c7d6e9d90 con 0x556c7d7f9d50 2024-03-18T22:49:41.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.614+0000 7f63e7fff640 1 --2- 172.21.15.40:0/1115760212 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556c7d7f9d50 0x556c7d7fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63c77fe640 10 client.17322 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:41.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63e7fff640 1 --2- 172.21.15.40:0/1115760212 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556c7d7f9d50 0x556c7d7fc170 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6558514afd2290e1 server_cookie=bcedc918caa6bb36 in_seq=0 out_seq=0 2024-03-18T22:49:41.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63c77fe640 1 -- 172.21.15.40:0/1115760212 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556c7d6e9d90 con 0x556c7d7f9d50 2024-03-18T22:49:41.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63c77fe640 10 client.17322 handle_command_reply: tid=0 2024-03-18T22:49:41.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63f0fa6740 1 client.17322 shutdown 2024-03-18T22:49:41.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63f0fa6740 1 -- 172.21.15.40:0/1115760212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d7835b0 msgr2=0x556c7d7f7e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:41.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/1115760212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d7835b0 0x556c7d7f7e70 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f63e002c700 tx=0x7f63e00095c0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63f0fa6740 1 -- 172.21.15.40:0/1115760212 shutdown_connections 2024-03-18T22:49:41.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/1115760212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556c7d7835b0 0x556c7d7f7e70 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/1115760212 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556c7d77e440 0x556c7d78c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/1115760212 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556c7d7f9d50 0x556c7d7fc170 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63f0fa6740 1 --2- 172.21.15.40:0/1115760212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556c7d77da50 0x556c7d78c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63f0fa6740 1 -- 172.21.15.40:0/1115760212 >> 172.21.15.40:0/1115760212 conn(0x556c7d778fc0 msgr2=0x556c7d777950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:41.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63c77fe640 0 client.17322 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:41.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63f0fa6740 1 -- 172.21.15.40:0/1115760212 shutdown_connections 2024-03-18T22:49:41.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63f0fa6740 1 -- 172.21.15.40:0/1115760212 wait complete. 2024-03-18T22:49:41.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.615+0000 7f63f0fa6740 20 client.17322 trim_cache size 0 max 16384 2024-03-18T22:49:41.626 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:49:41.626 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:49:41.627 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:49:41.627 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:49:41.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.616+0000 7f63ee2dc640 1 -- 172.21.15.40:0/2731044681 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63d40a2800 msgr2=0x7f63d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:41.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.616+0000 7f63ee2dc640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63d40a2800 0x7f63d40a4cc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x556c7d77eca0 tx=0x7f63e00324f0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.616+0000 7f63ee2dc640 1 -- 172.21.15.40:0/2731044681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e8102a20 msgr2=0x7f63e8197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:41.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.616+0000 7f63ee2dc640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e8102a20 0x7f63e8197f80 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f63d800ea60 tx=0x7f63d800ef20 comp rx=0 tx=0).stop 2024-03-18T22:49:41.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.616+0000 7f63ee2dc640 1 -- 172.21.15.40:0/2731044681 shutdown_connections 2024-03-18T22:49:41.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.616+0000 7f63ee2dc640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63d40a2800 0x7f63d40a4cc0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.616+0000 7f63ee2dc640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f63e81033e0 0x7f63e819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.616+0000 7f63ee2dc640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e8102a20 0x7f63e8197f80 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.616+0000 7f63ee2dc640 1 --2- 172.21.15.40:0/2731044681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63e8101830 0x7f63e8197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.616+0000 7f63ee2dc640 1 -- 172.21.15.40:0/2731044681 >> 172.21.15.40:0/2731044681 conn(0x7f63e80fcda0 msgr2=0x7f63e8105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:41.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.616+0000 7f63ee2dc640 1 -- 172.21.15.40:0/2731044681 shutdown_connections 2024-03-18T22:49:41.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.616+0000 7f63ee2dc640 1 -- 172.21.15.40:0/2731044681 wait complete. 2024-03-18T22:49:41.630 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:49:41.630 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:49:41.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.740+0000 7f94cff48640 1 -- 172.21.15.40:0/4101977197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94c8101930 msgr2=0x7f94c8101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:41.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.740+0000 7f94cff48640 1 --2- 172.21.15.40:0/4101977197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94c8101930 0x7f94c8101db0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009900 tx=0x7f94b802bc30 comp rx=0 tx=0).stop 2024-03-18T22:49:41.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.740+0000 7f94cff48640 1 -- 172.21.15.40:0/4101977197 shutdown_connections 2024-03-18T22:49:41.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.740+0000 7f94cff48640 1 --2- 172.21.15.40:0/4101977197 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94c81022f0 0x7f94c8108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.740+0000 7f94cff48640 1 --2- 172.21.15.40:0/4101977197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94c8101930 0x7f94c8101db0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.740+0000 7f94cff48640 1 --2- 172.21.15.40:0/4101977197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94c8100740 0x7f94c8100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.740+0000 7f94cff48640 1 -- 172.21.15.40:0/4101977197 >> 172.21.15.40:0/4101977197 conn(0x7f94c80fbcb0 msgr2=0x7f94c80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:41.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.740+0000 7f94cff48640 1 -- 172.21.15.40:0/4101977197 shutdown_connections 2024-03-18T22:49:41.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.740+0000 7f94cff48640 1 -- 172.21.15.40:0/4101977197 wait complete. 2024-03-18T22:49:41.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.741+0000 7f94cff48640 1 Processor -- start 2024-03-18T22:49:41.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.741+0000 7f94cff48640 1 -- start start 2024-03-18T22:49:41.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.741+0000 7f94cff48640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94c8100740 0x7f94c80694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.741+0000 7f94cff48640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94c8101930 0x7f94c819c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.741+0000 7f94cff48640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94c81022f0 0x7f94c819edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.741+0000 7f94cff48640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f94c810c270 con 0x7f94c8100740 2024-03-18T22:49:41.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.741+0000 7f94cff48640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f94c810c0f0 con 0x7f94c8101930 2024-03-18T22:49:41.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.741+0000 7f94cff48640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f94c810c3f0 con 0x7f94c81022f0 2024-03-18T22:49:41.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.741+0000 7f94cdce4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94c8100740 0x7f94c80694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.741+0000 7f94cdce4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94c8100740 0x7f94c80694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33236/0 (socket says 172.21.15.40:33236) 2024-03-18T22:49:41.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.741+0000 7f94cdce4640 1 -- 172.21.15.40:0/2763010516 learned_addr learned my addr 172.21.15.40:0/2763010516 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:41.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.741+0000 7f94ce4e5640 1 --2- 172.21.15.40:0/2763010516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94c81022f0 0x7f94c819edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.741+0000 7f94cd4e3640 1 --2- 172.21.15.40:0/2763010516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94c8101930 0x7f94c819c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.742+0000 7f94cdce4640 1 -- 172.21.15.40:0/2763010516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94c81022f0 msgr2=0x7f94c819edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:41.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.742+0000 7f94cdce4640 1 --2- 172.21.15.40:0/2763010516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94c81022f0 0x7f94c819edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.742+0000 7f94cdce4640 1 -- 172.21.15.40:0/2763010516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94c8101930 msgr2=0x7f94c819c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:41.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.742+0000 7f94cdce4640 1 --2- 172.21.15.40:0/2763010516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94c8101930 0x7f94c819c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.742+0000 7f94cdce4640 1 -- 172.21.15.40:0/2763010516 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c819f6d0 con 0x7f94c8100740 2024-03-18T22:49:41.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.742+0000 7f94cdce4640 1 --2- 172.21.15.40:0/2763010516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94c8100740 0x7f94c80694f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f94c400e9c0 tx=0x7f94c400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:41.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.742+0000 7f94beffd640 1 -- 172.21.15.40:0/2763010516 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c400cd80 con 0x7f94c8100740 2024-03-18T22:49:41.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.742+0000 7f94beffd640 1 -- 172.21.15.40:0/2763010516 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94c40035f0 con 0x7f94c8100740 2024-03-18T22:49:41.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.742+0000 7f94cff48640 1 -- 172.21.15.40:0/2763010516 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94c819f960 con 0x7f94c8100740 2024-03-18T22:49:41.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.742+0000 7f94beffd640 1 -- 172.21.15.40:0/2763010516 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c4010690 con 0x7f94c8100740 2024-03-18T22:49:41.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.742+0000 7f94cff48640 1 -- 172.21.15.40:0/2763010516 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94c819fed0 con 0x7f94c8100740 2024-03-18T22:49:41.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.743+0000 7f94cff48640 1 -- 172.21.15.40:0/2763010516 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94c8068610 con 0x7f94c8100740 2024-03-18T22:49:41.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.744+0000 7f94beffd640 1 -- 172.21.15.40:0/2763010516 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f94c40037a0 con 0x7f94c8100740 2024-03-18T22:49:41.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.745+0000 7f94beffd640 1 --2- 172.21.15.40:0/2763010516 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f94b40a2880 0x7f94b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:41.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.745+0000 7f94cd4e3640 1 --2- 172.21.15.40:0/2763010516 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f94b40a2880 0x7f94b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:41.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.745+0000 7f94beffd640 1 -- 172.21.15.40:0/2763010516 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f94c40cf130 con 0x7f94c8100740 2024-03-18T22:49:41.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.745+0000 7f94cd4e3640 1 --2- 172.21.15.40:0/2763010516 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f94b40a2880 0x7f94b40a4d40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009230 tx=0x7f94b8005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:41.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.751+0000 7f94beffd640 1 -- 172.21.15.40:0/2763010516 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f94c409bde0 con 0x7f94c8100740 2024-03-18T22:49:41.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.990+0000 7f94cff48640 1 -- 172.21.15.40:0/2763010516 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94c8107ae0 con 0x7f94c8100740 2024-03-18T22:49:41.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.991+0000 7f94beffd640 1 -- 172.21.15.40:0/2763010516 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11954 (secure 0 0 0) 0x7f94c409b6e0 con 0x7f94c8100740 2024-03-18T22:49:41.993 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:49:41.993 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:49:24.214483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24401,24388,24473,24509,24481]},"id":1}]} 2024-03-18T22:49:41.993 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 34 2024-03-18T22:49:41.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.994+0000 7f94cff48640 1 -- 172.21.15.40:0/2763010516 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f94b40a2880 msgr2=0x7f94b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.994+0000 7f94cff48640 1 --2- 172.21.15.40:0/2763010516 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f94b40a2880 0x7f94b40a4d40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009230 tx=0x7f94b8005fd0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.994+0000 7f94cff48640 1 -- 172.21.15.40:0/2763010516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94c8100740 msgr2=0x7f94c80694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.994+0000 7f94cff48640 1 --2- 172.21.15.40:0/2763010516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94c8100740 0x7f94c80694f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f94c400e9c0 tx=0x7f94c400ee80 comp rx=0 tx=0).stop 2024-03-18T22:49:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.994+0000 7f94cff48640 1 -- 172.21.15.40:0/2763010516 shutdown_connections 2024-03-18T22:49:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.994+0000 7f94cff48640 1 --2- 172.21.15.40:0/2763010516 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f94b40a2880 0x7f94b40a4d40 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.994+0000 7f94cff48640 1 --2- 172.21.15.40:0/2763010516 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f94c81022f0 0x7f94c819edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.994+0000 7f94cff48640 1 --2- 172.21.15.40:0/2763010516 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f94c8101930 0x7f94c819c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.994+0000 7f94cff48640 1 --2- 172.21.15.40:0/2763010516 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94c8100740 0x7f94c80694f0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.994+0000 7f94cff48640 1 -- 172.21.15.40:0/2763010516 >> 172.21.15.40:0/2763010516 conn(0x7f94c80fbcb0 msgr2=0x7f94c8104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.994+0000 7f94cff48640 1 -- 172.21.15.40:0/2763010516 shutdown_connections 2024-03-18T22:49:41.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:41.994+0000 7f94cff48640 1 -- 172.21.15.40:0/2763010516 wait complete. 2024-03-18T22:49:42.008 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:49:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.122+0000 7f5dd15df640 1 -- 172.21.15.40:0/107163962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dcc1022d0 msgr2=0x7f5dcc108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.122+0000 7f5dd15df640 1 --2- 172.21.15.40:0/107163962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dcc1022d0 0x7f5dcc108b60 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4009ca0 tx=0x7f5dc402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.122+0000 7f5dd15df640 1 -- 172.21.15.40:0/107163962 shutdown_connections 2024-03-18T22:49:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.122+0000 7f5dd15df640 1 --2- 172.21.15.40:0/107163962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dcc1022d0 0x7f5dcc108b60 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.122+0000 7f5dd15df640 1 --2- 172.21.15.40:0/107163962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dcc101910 0x7f5dcc101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.122+0000 7f5dd15df640 1 --2- 172.21.15.40:0/107163962 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc100720 0x7f5dcc100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.122+0000 7f5dd15df640 1 -- 172.21.15.40:0/107163962 >> 172.21.15.40:0/107163962 conn(0x7f5dcc0fbc90 msgr2=0x7f5dcc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.123+0000 7f5dd15df640 1 -- 172.21.15.40:0/107163962 shutdown_connections 2024-03-18T22:49:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.123+0000 7f5dd15df640 1 -- 172.21.15.40:0/107163962 wait complete. 2024-03-18T22:49:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.123+0000 7f5dd15df640 1 Processor -- start 2024-03-18T22:49:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.123+0000 7f5dd15df640 1 -- start start 2024-03-18T22:49:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.123+0000 7f5dd15df640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dcc100720 0x7f5dcc1a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.123+0000 7f5dd15df640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc101910 0x7f5dcc1a2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.123+0000 7f5dd15df640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dcc1022d0 0x7f5dcc1a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.123+0000 7f5dd15df640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc10b160 con 0x7f5dcc1022d0 2024-03-18T22:49:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.123+0000 7f5dd15df640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc10afe0 con 0x7f5dcc100720 2024-03-18T22:49:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.123+0000 7f5dd15df640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc10b2e0 con 0x7f5dcc101910 2024-03-18T22:49:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.123+0000 7f5dcb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dcc1022d0 0x7f5dcc1a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.123+0000 7f5dcb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dcc1022d0 0x7f5dcc1a54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33268/0 (socket says 172.21.15.40:33268) 2024-03-18T22:49:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.123+0000 7f5dcb7fe640 1 -- 172.21.15.40:0/3877687927 learned_addr learned my addr 172.21.15.40:0/3877687927 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.123+0000 7f5dc3fff640 1 --2- 172.21.15.40:0/3877687927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc101910 0x7f5dcc1a2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.123+0000 7f5dcaffd640 1 --2- 172.21.15.40:0/3877687927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dcc100720 0x7f5dcc1a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.124+0000 7f5dcb7fe640 1 -- 172.21.15.40:0/3877687927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc101910 msgr2=0x7f5dcc1a2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.124+0000 7f5dcb7fe640 1 --2- 172.21.15.40:0/3877687927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc101910 0x7f5dcc1a2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.124+0000 7f5dcb7fe640 1 -- 172.21.15.40:0/3877687927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dcc100720 msgr2=0x7f5dcc1a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.124+0000 7f5dcb7fe640 1 --2- 172.21.15.40:0/3877687927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dcc100720 0x7f5dcc1a0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.124+0000 7f5dcb7fe640 1 -- 172.21.15.40:0/3877687927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dcc1a5c60 con 0x7f5dcc1022d0 2024-03-18T22:49:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.124+0000 7f5dcb7fe640 1 --2- 172.21.15.40:0/3877687927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dcc1022d0 0x7f5dcc1a54e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f5dc402c700 tx=0x7f5dc40328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.124+0000 7f5dc8ff9640 1 -- 172.21.15.40:0/3877687927 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4041020 con 0x7f5dcc1022d0 2024-03-18T22:49:42.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.124+0000 7f5dc8ff9640 1 -- 172.21.15.40:0/3877687927 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5dc4003e20 con 0x7f5dcc1022d0 2024-03-18T22:49:42.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.124+0000 7f5dd15df640 1 -- 172.21.15.40:0/3877687927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dcc1a5ef0 con 0x7f5dcc1022d0 2024-03-18T22:49:42.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.124+0000 7f5dc8ff9640 1 -- 172.21.15.40:0/3877687927 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4039040 con 0x7f5dcc1022d0 2024-03-18T22:49:42.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.124+0000 7f5dd15df640 1 -- 172.21.15.40:0/3877687927 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dcc1a6350 con 0x7f5dcc1022d0 2024-03-18T22:49:42.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.127+0000 7f5dd41c7740 1 Processor -- start 2024-03-18T22:49:42.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.127+0000 7f5dc8ff9640 1 -- 172.21.15.40:0/3877687927 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5dc4044020 con 0x7f5dcc1022d0 2024-03-18T22:49:42.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.127+0000 7f5dd41c7740 1 -- start start 2024-03-18T22:49:42.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.127+0000 7f5dd41c7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b02f0 0x559add3b0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.127+0000 7f5dd41c7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b0ce0 0x559add3b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.127+0000 7f5dd41c7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b5e50 0x559add3b82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.127+0000 7f5dd41c7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559add3144b0 con 0x559add3b0ce0 2024-03-18T22:49:42.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.127+0000 7f5dd41c7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559add2bcd70 con 0x559add3b5e50 2024-03-18T22:49:42.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.127+0000 7f5dd41c7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559add310550 con 0x559add3b02f0 2024-03-18T22:49:42.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dcaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b0ce0 0x559add3b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dc3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b02f0 0x559add3b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dcaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b0ce0 0x559add3b57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33276/0 (socket says 172.21.15.40:33276) 2024-03-18T22:49:42.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dcaffd640 1 -- 172.21.15.40:0/3302053596 learned_addr learned my addr 172.21.15.40:0/3302053596 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dc3fff640 1 --2- 172.21.15.40:0/3302053596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b5e50 0x559add3b82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dc8ff9640 1 --2- 172.21.15.40:0/3877687927 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5db80a2800 0x7f5db80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dc3fff640 1 -- 172.21.15.40:0/3302053596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b5e50 msgr2=0x559add3b82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dc3fff640 1 --2- 172.21.15.40:0/3302053596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b5e50 0x559add3b82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dc3fff640 1 -- 172.21.15.40:0/3302053596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b0ce0 msgr2=0x559add3b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dc3fff640 1 --2- 172.21.15.40:0/3302053596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b0ce0 0x559add3b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dc3fff640 1 -- 172.21.15.40:0/3302053596 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559add2df210 con 0x559add3b02f0 2024-03-18T22:49:42.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dcb7fe640 1 --2- 172.21.15.40:0/3877687927 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5db80a2800 0x7f5db80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dc8ff9640 1 -- 172.21.15.40:0/3877687927 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5dc40f0fc0 con 0x7f5dcc1022d0 2024-03-18T22:49:42.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dc3fff640 1 --2- 172.21.15.40:0/3302053596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b02f0 0x559add3b0710 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009730 tx=0x7f5db40096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6753a140a9b25060 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dc17fa640 1 -- 172.21.15.40:0/3302053596 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db4004050 con 0x559add3b02f0 2024-03-18T22:49:42.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dc17fa640 1 -- 172.21.15.40:0/3302053596 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5db40054d0 con 0x559add3b02f0 2024-03-18T22:49:42.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.128+0000 7f5dc17fa640 1 -- 172.21.15.40:0/3302053596 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db4004b50 con 0x559add3b02f0 2024-03-18T22:49:42.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.129+0000 7f5dd41c7740 1 -- 172.21.15.40:0/3302053596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b02f0 msgr2=0x559add3b0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.129+0000 7f5dcb7fe640 1 --2- 172.21.15.40:0/3877687927 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5db80a2800 0x7f5db80a4cc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40b99d0 tx=0x7f5dc4088330 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.129+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/3302053596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b02f0 0x559add3b0710 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009730 tx=0x7f5db40096c0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.129+0000 7f5dd41c7740 1 -- 172.21.15.40:0/3302053596 shutdown_connections 2024-03-18T22:49:42.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.129+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/3302053596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b5e50 0x559add3b82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.129+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/3302053596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b0ce0 0x559add3b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.129+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/3302053596 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b02f0 0x559add3b0710 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.129+0000 7f5dd41c7740 1 -- 172.21.15.40:0/3302053596 >> 172.21.15.40:0/3302053596 conn(0x559add3ab860 msgr2=0x559add3abc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.129+0000 7f5dd41c7740 1 -- 172.21.15.40:0/3302053596 shutdown_connections 2024-03-18T22:49:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.129+0000 7f5dd41c7740 1 -- 172.21.15.40:0/3302053596 wait complete. 2024-03-18T22:49:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.130+0000 7f5dd41c7740 1 -- start start 2024-03-18T22:49:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.130+0000 7f5dd41c7740 1 Processor -- start 2024-03-18T22:49:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.130+0000 7f5dd41c7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b02f0 0x559add3ba830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.130+0000 7f5dd41c7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b0ce0 0x559add3bad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.130+0000 7f5dd41c7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b5e50 0x559add3c1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.130+0000 7f5dd41c7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559add3b8e00 con 0x559add3b02f0 2024-03-18T22:49:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.130+0000 7f5dd41c7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559add2d0310 con 0x559add3b0ce0 2024-03-18T22:49:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.130+0000 7f5dd41c7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559add3b8f80 con 0x559add3b5e50 2024-03-18T22:49:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.130+0000 7f5dd41c7740 10 client.0 _ll_register_callbacks cb 0x559add2c42d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.130+0000 7f5dc3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b02f0 0x559add3ba830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.130+0000 7f5dc3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b02f0 0x559add3ba830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33278/0 (socket says 172.21.15.40:33278) 2024-03-18T22:49:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.130+0000 7f5dcaffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b0ce0 0x559add3bad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.130+0000 7f5dc3fff640 1 -- 172.21.15.40:0/523093123 learned_addr learned my addr 172.21.15.40:0/523093123 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.130+0000 7f5dc3fff640 1 --2- 172.21.15.40:0/523093123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b5e50 0x559add3c1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.131+0000 7f5dc3fff640 1 -- 172.21.15.40:0/523093123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b5e50 msgr2=0x559add3c1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.131+0000 7f5dc3fff640 1 --2- 172.21.15.40:0/523093123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b5e50 0x559add3c1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.131+0000 7f5dc3fff640 1 -- 172.21.15.40:0/523093123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b0ce0 msgr2=0x559add3bad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.131+0000 7f5dc3fff640 1 --2- 172.21.15.40:0/523093123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b0ce0 0x559add3bad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.131+0000 7f5dc3fff640 1 -- 172.21.15.40:0/523093123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559add0283f0 con 0x559add3b02f0 2024-03-18T22:49:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.131+0000 7f5dcaffd640 1 --2- 172.21.15.40:0/523093123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b0ce0 0x559add3bad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.131+0000 7f5dc3fff640 1 --2- 172.21.15.40:0/523093123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b02f0 0x559add3ba830 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f5db40045e0 tx=0x7f5db4005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4649eb84252ed1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.131+0000 7f5db27fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.131+0000 7f5db27fc640 1 -- 172.21.15.40:0/523093123 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db4004e10 con 0x559add3b02f0 2024-03-18T22:49:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.131+0000 7f5db27fc640 1 -- 172.21.15.40:0/523093123 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5db4005140 con 0x559add3b02f0 2024-03-18T22:49:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.131+0000 7f5dd41c7740 1 -- 172.21.15.40:0/523093123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5db40054d0 con 0x559add3b02f0 2024-03-18T22:49:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.131+0000 7f5db27fc640 1 -- 172.21.15.40:0/523093123 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f5db4011020 con 0x559add3b02f0 2024-03-18T22:49:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.131+0000 7f5dd41c7740 10 client.17340 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.131+0000 7f5dd41c7740 1 -- 172.21.15.40:0/523093123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559add0f27d0 con 0x559add3b02f0 2024-03-18T22:49:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.131+0000 7f5db27fc640 1 -- 172.21.15.40:0/523093123 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5db40054d0 con 0x559add3b02f0 2024-03-18T22:49:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.132+0000 7f5db27fc640 1 -- 172.21.15.40:0/523093123 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f5db40304c0 con 0x559add3b02f0 2024-03-18T22:49:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.132+0000 7f5dd41c7740 10 client.17340 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.132+0000 7f5dd41c7740 10 client.17340 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:49:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.132+0000 7f5dd41c7740 20 client.17340 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.132+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/523093123 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559add3c6360 0x559add3c8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.132+0000 7f5dd41c7740 4 client.17340 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.132+0000 7f5dd41c7740 1 -- 172.21.15.40:0/523093123 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559add3723e0 con 0x559add3c6360 2024-03-18T22:49:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.132+0000 7f5dcaffd640 1 --2- 172.21.15.40:0/523093123 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559add3c6360 0x559add3c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.132+0000 7f5dcaffd640 1 --2- 172.21.15.40:0/523093123 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559add3c6360 0x559add3c8770 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c4f57a7d0ca6b9a7 server_cookie=55cf796cf91d3f20 in_seq=0 out_seq=0 2024-03-18T22:49:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.132+0000 7f5db27fc640 10 client.17340 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:49:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.133+0000 7f5db27fc640 1 -- 172.21.15.40:0/523093123 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559add3723e0 con 0x559add3c6360 2024-03-18T22:49:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.133+0000 7f5db27fc640 10 client.17340 handle_command_reply: tid=0 2024-03-18T22:49:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.133+0000 7f5dd41c7740 1 client.17340 shutdown 2024-03-18T22:49:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.133+0000 7f5dd41c7740 1 -- 172.21.15.40:0/523093123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b02f0 msgr2=0x559add3ba830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.133+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/523093123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b02f0 0x559add3ba830 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f5db40045e0 tx=0x7f5db4005850 comp rx=0 tx=0).stop 2024-03-18T22:49:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.133+0000 7f5dd41c7740 1 -- 172.21.15.40:0/523093123 shutdown_connections 2024-03-18T22:49:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.133+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/523093123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b5e50 0x559add3c1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.133+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/523093123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b0ce0 0x559add3bad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.133+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/523093123 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559add3c6360 0x559add3c8770 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.133+0000 7f5db27fc640 0 client.17340 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:49:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.133+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/523093123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b02f0 0x559add3ba830 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.133+0000 7f5dd41c7740 1 -- 172.21.15.40:0/523093123 >> 172.21.15.40:0/523093123 conn(0x559add3ab860 msgr2=0x559add3aaba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.133+0000 7f5dd41c7740 1 -- 172.21.15.40:0/523093123 shutdown_connections 2024-03-18T22:49:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.134+0000 7f5dd41c7740 1 -- 172.21.15.40:0/523093123 wait complete. 2024-03-18T22:49:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.134+0000 7f5dd41c7740 20 client.17340 trim_cache size 0 max 16384 2024-03-18T22:49:42.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dd41c7740 1 Processor -- start 2024-03-18T22:49:42.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dd41c7740 1 -- start start 2024-03-18T22:49:42.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dd41c7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b02f0 0x559add3bacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dd41c7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b0ce0 0x559add3c1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dd41c7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b5e50 0x559add3af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dd41c7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc1b0ac0 con 0x559add3b0ce0 2024-03-18T22:49:42.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dd41c7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc1b26c0 con 0x559add3b02f0 2024-03-18T22:49:42.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dd41c7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc1b1190 con 0x559add3b5e50 2024-03-18T22:49:42.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dcaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b0ce0 0x559add3c1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dcaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b0ce0 0x559add3c1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33280/0 (socket says 172.21.15.40:33280) 2024-03-18T22:49:42.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dcaffd640 1 -- 172.21.15.40:0/2071021499 learned_addr learned my addr 172.21.15.40:0/2071021499 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dc3fff640 1 --2- 172.21.15.40:0/2071021499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b5e50 0x559add3af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dc3fff640 1 --2- 172.21.15.40:0/2071021499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b02f0 0x559add3bacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dcaffd640 1 -- 172.21.15.40:0/2071021499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b5e50 msgr2=0x559add3af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dcaffd640 1 --2- 172.21.15.40:0/2071021499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b5e50 0x559add3af1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dcaffd640 1 -- 172.21.15.40:0/2071021499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b02f0 msgr2=0x559add3bacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dcaffd640 1 --2- 172.21.15.40:0/2071021499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b02f0 0x559add3bacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.155+0000 7f5dcaffd640 1 -- 172.21.15.40:0/2071021499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559add3c8df0 con 0x559add3b0ce0 2024-03-18T22:49:42.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.156+0000 7f5dcaffd640 1 --2- 172.21.15.40:0/2071021499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b0ce0 0x559add3c1d70 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x559add3ac7d0 tx=0x7f5dbc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac46a87ef4a86f9d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.156+0000 7f5dc0ff9640 1 -- 172.21.15.40:0/2071021499 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc020310 con 0x559add3b0ce0 2024-03-18T22:49:42.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.156+0000 7f5dc0ff9640 1 -- 172.21.15.40:0/2071021499 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5dbc010430 con 0x559add3b0ce0 2024-03-18T22:49:42.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.156+0000 7f5dc0ff9640 1 -- 172.21.15.40:0/2071021499 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc01e680 con 0x559add3b0ce0 2024-03-18T22:49:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 -- 172.21.15.40:0/2071021499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b0ce0 msgr2=0x559add3c1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/2071021499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b0ce0 0x559add3c1d70 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x559add3ac7d0 tx=0x7f5dbc005f90 comp rx=0 tx=0).stop 2024-03-18T22:49:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 -- 172.21.15.40:0/2071021499 shutdown_connections 2024-03-18T22:49:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/2071021499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b5e50 0x559add3af1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/2071021499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b0ce0 0x559add3c1d70 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/2071021499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b02f0 0x559add3bacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 -- 172.21.15.40:0/2071021499 >> 172.21.15.40:0/2071021499 conn(0x559add3ab860 msgr2=0x559add3aeac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 -- 172.21.15.40:0/2071021499 shutdown_connections 2024-03-18T22:49:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 -- 172.21.15.40:0/2071021499 wait complete. 2024-03-18T22:49:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 -- start start 2024-03-18T22:49:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 Processor -- start 2024-03-18T22:49:42.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b02f0 0x559add3bf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b0ce0 0x559add3bfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b5e50 0x559add3c0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc1b1ff0 con 0x559add3b5e50 2024-03-18T22:49:42.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc1b1860 con 0x559add3b0ce0 2024-03-18T22:49:42.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5dcc1aeaa0 con 0x559add3b02f0 2024-03-18T22:49:42.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.157+0000 7f5dd41c7740 10 client.0 _ll_register_callbacks cb 0x7f5dbc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:42.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5dc3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b02f0 0x559add3bf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5dc3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b02f0 0x559add3bf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:60398/0 (socket says 172.21.15.40:60398) 2024-03-18T22:49:42.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5dc3fff640 1 -- 172.21.15.40:0/3107056998 learned_addr learned my addr 172.21.15.40:0/3107056998 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5dcaffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b0ce0 0x559add3bfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5dc3fff640 1 --2- 172.21.15.40:0/3107056998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b5e50 0x559add3c0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5dc3fff640 1 -- 172.21.15.40:0/3107056998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b0ce0 msgr2=0x559add3bfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5dc3fff640 1 --2- 172.21.15.40:0/3107056998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b0ce0 0x559add3bfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5dc3fff640 1 -- 172.21.15.40:0/3107056998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b5e50 msgr2=0x559add3c0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5dc3fff640 1 --2- 172.21.15.40:0/3107056998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b5e50 0x559add3c0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5dc3fff640 1 -- 172.21.15.40:0/3107056998 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559add381880 con 0x559add3b02f0 2024-03-18T22:49:42.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5dcaffd640 1 --2- 172.21.15.40:0/3107056998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b0ce0 0x559add3bfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5dc3fff640 1 --2- 172.21.15.40:0/3107056998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b02f0 0x559add3bf8d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5db40097c0 tx=0x7f5db40059b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68391deddad4e24d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5db27fc640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:49:42.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5db27fc640 1 -- 172.21.15.40:0/3107056998 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db4030d60 con 0x559add3b02f0 2024-03-18T22:49:42.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5db27fc640 1 -- 172.21.15.40:0/3107056998 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5db40050a0 con 0x559add3b02f0 2024-03-18T22:49:42.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5dd41c7740 1 -- 172.21.15.40:0/3107056998 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dbc010430 con 0x559add3b02f0 2024-03-18T22:49:42.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.158+0000 7f5db27fc640 1 -- 172.21.15.40:0/3107056998 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f5db40052a0 con 0x559add3b02f0 2024-03-18T22:49:42.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.159+0000 7f5dd41c7740 10 client.27154 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:42.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.159+0000 7f5dd41c7740 1 -- 172.21.15.40:0/3107056998 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559add42c850 con 0x559add3b02f0 2024-03-18T22:49:42.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.159+0000 7f5db27fc640 1 -- 172.21.15.40:0/3107056998 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5dbc010430 con 0x559add3b02f0 2024-03-18T22:49:42.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.159+0000 7f5db27fc640 1 -- 172.21.15.40:0/3107056998 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f5db400f8b0 con 0x559add3b02f0 2024-03-18T22:49:42.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.159+0000 7f5dd41c7740 10 client.27154 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:42.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.159+0000 7f5dd41c7740 10 client.27154 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:49:42.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.159+0000 7f5dd41c7740 20 client.27154 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:42.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.159+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/3107056998 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559add42d1c0 0x559add42f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.159+0000 7f5dd41c7740 4 client.27154 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:49:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.159+0000 7f5dd41c7740 1 -- 172.21.15.40:0/3107056998 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559add3219f0 con 0x559add42d1c0 2024-03-18T22:49:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.159+0000 7f5dcaffd640 1 --2- 172.21.15.40:0/3107056998 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559add42d1c0 0x559add42f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.159+0000 7f5dcaffd640 1 --2- 172.21.15.40:0/3107056998 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559add42d1c0 0x559add42f5e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ef7de2f32589bb09 server_cookie=11a440aa269202e3 in_seq=0 out_seq=0 2024-03-18T22:49:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.159+0000 7f5db27fc640 10 client.27154 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:49:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.160+0000 7f5db27fc640 1 -- 172.21.15.40:0/3107056998 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559add3219f0 con 0x559add42d1c0 2024-03-18T22:49:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.160+0000 7f5db27fc640 10 client.27154 handle_command_reply: tid=0 2024-03-18T22:49:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.160+0000 7f5dd41c7740 1 client.27154 shutdown 2024-03-18T22:49:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.160+0000 7f5dd41c7740 1 -- 172.21.15.40:0/3107056998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b02f0 msgr2=0x559add3bf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.160+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/3107056998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b02f0 0x559add3bf8d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5db40097c0 tx=0x7f5db40059b0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.160+0000 7f5dd41c7740 1 -- 172.21.15.40:0/3107056998 shutdown_connections 2024-03-18T22:49:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.160+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/3107056998 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x559add42d1c0 0x559add42f5e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.160+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/3107056998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559add3b5e50 0x559add3c0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.160+0000 7f5db27fc640 0 client.27154 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:49:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.160+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/3107056998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559add3b0ce0 0x559add3bfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.160+0000 7f5dd41c7740 1 --2- 172.21.15.40:0/3107056998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559add3b02f0 0x559add3bf8d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.160+0000 7f5dd41c7740 1 -- 172.21.15.40:0/3107056998 >> 172.21.15.40:0/3107056998 conn(0x559add3ab860 msgr2=0x559add3af350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.160+0000 7f5dd41c7740 1 -- 172.21.15.40:0/3107056998 shutdown_connections 2024-03-18T22:49:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.160+0000 7f5dd41c7740 1 -- 172.21.15.40:0/3107056998 wait complete. 2024-03-18T22:49:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.160+0000 7f5dd41c7740 20 client.27154 trim_cache size 0 max 16384 2024-03-18T22:49:42.170 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:49:42.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.161+0000 7f5dd15df640 1 -- 172.21.15.40:0/3877687927 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5db80a2800 msgr2=0x7f5db80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.161+0000 7f5dd15df640 1 --2- 172.21.15.40:0/3877687927 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5db80a2800 0x7f5db80a4cc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40b99d0 tx=0x7f5dc4088330 comp rx=0 tx=0).stop 2024-03-18T22:49:42.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.161+0000 7f5dd15df640 1 -- 172.21.15.40:0/3877687927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dcc1022d0 msgr2=0x7f5dcc1a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.161+0000 7f5dd15df640 1 --2- 172.21.15.40:0/3877687927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dcc1022d0 0x7f5dcc1a54e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f5dc402c700 tx=0x7f5dc40328a0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.161+0000 7f5dd15df640 1 -- 172.21.15.40:0/3877687927 shutdown_connections 2024-03-18T22:49:42.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.161+0000 7f5dd15df640 1 --2- 172.21.15.40:0/3877687927 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5db80a2800 0x7f5db80a4cc0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.161+0000 7f5dd15df640 1 --2- 172.21.15.40:0/3877687927 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5dcc1022d0 0x7f5dcc1a54e0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.161+0000 7f5dd15df640 1 --2- 172.21.15.40:0/3877687927 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5dcc101910 0x7f5dcc1a2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.161+0000 7f5dd15df640 1 --2- 172.21.15.40:0/3877687927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5dcc100720 0x7f5dcc1a0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.161+0000 7f5dd15df640 1 -- 172.21.15.40:0/3877687927 >> 172.21.15.40:0/3877687927 conn(0x7f5dcc0fbc90 msgr2=0x7f5dcc10dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.161+0000 7f5dd15df640 1 -- 172.21.15.40:0/3877687927 shutdown_connections 2024-03-18T22:49:42.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.161+0000 7f5dd15df640 1 -- 172.21.15.40:0/3877687927 wait complete. 2024-03-18T22:49:42.173 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:49:42.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.279+0000 7fdd37e12640 1 -- 172.21.15.40:0/417243781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30103a50 msgr2=0x7fdd30103e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.279+0000 7fdd37e12640 1 --2- 172.21.15.40:0/417243781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30103a50 0x7fdd30103e70 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009900 tx=0x7fdd2c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:49:42.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.280+0000 7fdd37e12640 1 -- 172.21.15.40:0/417243781 shutdown_connections 2024-03-18T22:49:42.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.280+0000 7fdd37e12640 1 --2- 172.21.15.40:0/417243781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30105600 0x7fdd3010be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.280+0000 7fdd37e12640 1 --2- 172.21.15.40:0/417243781 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd30104c40 0x7fdd301050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.280+0000 7fdd37e12640 1 --2- 172.21.15.40:0/417243781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30103a50 0x7fdd30103e70 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.280+0000 7fdd37e12640 1 -- 172.21.15.40:0/417243781 >> 172.21.15.40:0/417243781 conn(0x7fdd300fefe0 msgr2=0x7fdd30101400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.280+0000 7fdd37e12640 1 -- 172.21.15.40:0/417243781 shutdown_connections 2024-03-18T22:49:42.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.280+0000 7fdd37e12640 1 -- 172.21.15.40:0/417243781 wait complete. 2024-03-18T22:49:42.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.280+0000 7fdd37e12640 1 Processor -- start 2024-03-18T22:49:42.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.280+0000 7fdd37e12640 1 -- start start 2024-03-18T22:49:42.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.280+0000 7fdd37e12640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30103a50 0x7fdd30199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.280+0000 7fdd37e12640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd30104c40 0x7fdd3019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.280+0000 7fdd37e12640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30105600 0x7fdd301a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd37e12640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3010f570 con 0x7fdd30105600 2024-03-18T22:49:42.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd37e12640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3010f3f0 con 0x7fdd30103a50 2024-03-18T22:49:42.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd37e12640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3010f6f0 con 0x7fdd30104c40 2024-03-18T22:49:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd35bae640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30103a50 0x7fdd30199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd363af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30105600 0x7fdd301a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd35bae640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30103a50 0x7fdd30199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:41948/0 (socket says 172.21.15.40:41948) 2024-03-18T22:49:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd35bae640 1 -- 172.21.15.40:0/1714448452 learned_addr learned my addr 172.21.15.40:0/1714448452 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd353ad640 1 --2- 172.21.15.40:0/1714448452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd30104c40 0x7fdd3019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd35bae640 1 -- 172.21.15.40:0/1714448452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd30104c40 msgr2=0x7fdd3019a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd35bae640 1 --2- 172.21.15.40:0/1714448452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd30104c40 0x7fdd3019a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd35bae640 1 -- 172.21.15.40:0/1714448452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30105600 msgr2=0x7fdd301a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd35bae640 1 --2- 172.21.15.40:0/1714448452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30105600 0x7fdd301a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd35bae640 1 -- 172.21.15.40:0/1714448452 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd301a18d0 con 0x7fdd30103a50 2024-03-18T22:49:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd363af640 1 --2- 172.21.15.40:0/1714448452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30105600 0x7fdd301a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd35bae640 1 --2- 172.21.15.40:0/1714448452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30103a50 0x7fdd30199c90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c000c00 tx=0x7fdd2c005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd26ffd640 1 -- 172.21.15.40:0/1714448452 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c043020 con 0x7fdd30103a50 2024-03-18T22:49:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd37e12640 1 -- 172.21.15.40:0/1714448452 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd301a1b60 con 0x7fdd30103a50 2024-03-18T22:49:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.281+0000 7fdd26ffd640 1 -- 172.21.15.40:0/1714448452 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd2c004350 con 0x7fdd30103a50 2024-03-18T22:49:42.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.282+0000 7fdd26ffd640 1 -- 172.21.15.40:0/1714448452 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c003b80 con 0x7fdd30103a50 2024-03-18T22:49:42.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.282+0000 7fdd37e12640 1 -- 172.21.15.40:0/1714448452 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd301a2100 con 0x7fdd30103a50 2024-03-18T22:49:42.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.284+0000 7fdd3aa00740 1 Processor -- start 2024-03-18T22:49:42.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.284+0000 7fdd26ffd640 1 -- 172.21.15.40:0/1714448452 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdd2c046020 con 0x7fdd30103a50 2024-03-18T22:49:42.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.284+0000 7fdd3aa00740 1 -- start start 2024-03-18T22:49:42.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd3aa00740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc8048d2f0 0x55bc8048d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd3aa00740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 0x55bc804927e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd3aa00740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc80492e50 0x55bc804952e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd3aa00740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bc803f14b0 con 0x55bc8048d2f0 2024-03-18T22:49:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd3aa00740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bc80399d70 con 0x55bc8048dce0 2024-03-18T22:49:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd3aa00740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55bc803ed550 con 0x55bc80492e50 2024-03-18T22:49:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd363af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc8048d2f0 0x55bc8048d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd35bae640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 0x55bc804927e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd363af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc8048d2f0 0x55bc8048d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33326/0 (socket says 172.21.15.40:33326) 2024-03-18T22:49:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd363af640 1 -- 172.21.15.40:0/1665525432 learned_addr learned my addr 172.21.15.40:0/1665525432 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd353ad640 1 --2- 172.21.15.40:0/1665525432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc80492e50 0x55bc804952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd26ffd640 1 --2- 172.21.15.40:0/1714448452 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdd1c0a2800 0x7fdd1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd353ad640 1 -- 172.21.15.40:0/1665525432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 msgr2=0x55bc804927e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd353ad640 1 --2- 172.21.15.40:0/1665525432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 0x55bc804927e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd353ad640 1 -- 172.21.15.40:0/1665525432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc8048d2f0 msgr2=0x55bc8048d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd353ad640 1 --2- 172.21.15.40:0/1665525432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc8048d2f0 0x55bc8048d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd353ad640 1 -- 172.21.15.40:0/1665525432 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc803bc210 con 0x55bc80492e50 2024-03-18T22:49:42.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd35bae640 1 --2- 172.21.15.40:0/1665525432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 0x55bc804927e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd26ffd640 1 -- 172.21.15.40:0/1714448452 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdd2c0f0f70 con 0x7fdd30103a50 2024-03-18T22:49:42.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd363af640 1 --2- 172.21.15.40:0/1665525432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc8048d2f0 0x55bc8048d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.285+0000 7fdd363af640 1 --2- 172.21.15.40:0/1714448452 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdd1c0a2800 0x7fdd1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.286+0000 7fdd353ad640 1 --2- 172.21.15.40:0/1665525432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc80492e50 0x55bc804952e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009bd0 tx=0x7fdd20011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c127f9374a1a92c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.286+0000 7fdd1bfff640 1 -- 172.21.15.40:0/1665525432 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20004230 con 0x55bc80492e50 2024-03-18T22:49:42.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.286+0000 7fdd1bfff640 1 -- 172.21.15.40:0/1665525432 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd20009e40 con 0x55bc80492e50 2024-03-18T22:49:42.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.286+0000 7fdd1bfff640 1 -- 172.21.15.40:0/1665525432 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20004ca0 con 0x55bc80492e50 2024-03-18T22:49:42.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.286+0000 7fdd3aa00740 1 -- 172.21.15.40:0/1665525432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc80492e50 msgr2=0x55bc804952e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.286+0000 7fdd363af640 1 --2- 172.21.15.40:0/1714448452 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdd1c0a2800 0x7fdd1c0a4cc0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x55bc8048e540 tx=0x7fdd28004a40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.286+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/1665525432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc80492e50 0x55bc804952e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009bd0 tx=0x7fdd20011810 comp rx=0 tx=0).stop 2024-03-18T22:49:42.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.286+0000 7fdd3aa00740 1 -- 172.21.15.40:0/1665525432 shutdown_connections 2024-03-18T22:49:42.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.286+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/1665525432 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc80492e50 0x55bc804952e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.286+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/1665525432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 0x55bc804927e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.287+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/1665525432 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc8048d2f0 0x55bc8048d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.287+0000 7fdd3aa00740 1 -- 172.21.15.40:0/1665525432 >> 172.21.15.40:0/1665525432 conn(0x55bc80488860 msgr2=0x55bc80488c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.287+0000 7fdd3aa00740 1 -- 172.21.15.40:0/1665525432 shutdown_connections 2024-03-18T22:49:42.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.287+0000 7fdd3aa00740 1 -- 172.21.15.40:0/1665525432 wait complete. 2024-03-18T22:49:42.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.287+0000 7fdd3aa00740 1 -- start start 2024-03-18T22:49:42.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.287+0000 7fdd3aa00740 1 Processor -- start 2024-03-18T22:49:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.287+0000 7fdd3aa00740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc8048d2f0 0x55bc80497740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.287+0000 7fdd3aa00740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 0x55bc80497c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd3aa00740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc80492e50 0x55bc8049ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd3aa00740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55bc80495e00 con 0x55bc80492e50 2024-03-18T22:49:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd3aa00740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bc803ad310 con 0x55bc8048dce0 2024-03-18T22:49:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd3aa00740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55bc80495f80 con 0x55bc8048d2f0 2024-03-18T22:49:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd3aa00740 10 client.0 _ll_register_callbacks cb 0x55bc803a12d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd35bae640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc8048d2f0 0x55bc80497740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd353ad640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 0x55bc80497c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd35bae640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc8048d2f0 0x55bc80497740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:60436/0 (socket says 172.21.15.40:60436) 2024-03-18T22:49:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd363af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc80492e50 0x55bc8049ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd35bae640 1 -- 172.21.15.40:0/3838859766 learned_addr learned my addr 172.21.15.40:0/3838859766 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd35bae640 1 -- 172.21.15.40:0/3838859766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 msgr2=0x55bc80497c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd35bae640 1 --2- 172.21.15.40:0/3838859766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 0x55bc80497c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd35bae640 1 -- 172.21.15.40:0/3838859766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc80492e50 msgr2=0x55bc8049ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd35bae640 1 --2- 172.21.15.40:0/3838859766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc80492e50 0x55bc8049ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd35bae640 1 -- 172.21.15.40:0/3838859766 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc801053f0 con 0x55bc8048d2f0 2024-03-18T22:49:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd363af640 1 --2- 172.21.15.40:0/3838859766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc80492e50 0x55bc8049ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd35bae640 1 --2- 172.21.15.40:0/3838859766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc8048d2f0 0x55bc80497740 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c0bd220 tx=0x7fdd2c0f47b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13525458b6f64233 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd197fa640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:49:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd197fa640 1 -- 172.21.15.40:0/3838859766 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c0b97c0 con 0x55bc8048d2f0 2024-03-18T22:49:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd197fa640 1 -- 172.21.15.40:0/3838859766 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdd2c003d20 con 0x55bc8048d2f0 2024-03-18T22:49:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.288+0000 7fdd3aa00740 1 -- 172.21.15.40:0/3838859766 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bc8044f3e0 con 0x55bc8048d2f0 2024-03-18T22:49:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.289+0000 7fdd197fa640 1 -- 172.21.15.40:0/3838859766 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fdd2c0bce80 con 0x55bc8048d2f0 2024-03-18T22:49:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.289+0000 7fdd3aa00740 10 client.27166 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.289+0000 7fdd3aa00740 1 -- 172.21.15.40:0/3838859766 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55bc801cf7d0 con 0x55bc8048d2f0 2024-03-18T22:49:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.289+0000 7fdd197fa640 1 -- 172.21.15.40:0/3838859766 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55bc8044f3e0 con 0x55bc8048d2f0 2024-03-18T22:49:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.289+0000 7fdd197fa640 1 -- 172.21.15.40:0/3838859766 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fdd2c0f3110 con 0x55bc8048d2f0 2024-03-18T22:49:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.289+0000 7fdd3aa00740 10 client.27166 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.289+0000 7fdd3aa00740 10 client.27166 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:49:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.289+0000 7fdd3aa00740 20 client.27166 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.289+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/3838859766 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bc804a3590 0x55bc804a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.289+0000 7fdd3aa00740 4 client.27166 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.289+0000 7fdd3aa00740 1 -- 172.21.15.40:0/3838859766 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc803fe9f0 con 0x55bc804a3590 2024-03-18T22:49:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.289+0000 7fdd353ad640 1 --2- 172.21.15.40:0/3838859766 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bc804a3590 0x55bc804a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.290+0000 7fdd353ad640 1 --2- 172.21.15.40:0/3838859766 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bc804a3590 0x55bc804a5a20 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c06cd092ce3297b7 server_cookie=964f3262f9af9f57 in_seq=0 out_seq=0 2024-03-18T22:49:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.290+0000 7fdd197fa640 10 client.27166 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:49:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.291+0000 7fdd197fa640 1 -- 172.21.15.40:0/3838859766 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bc803fe9f0 con 0x55bc804a3590 2024-03-18T22:49:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.291+0000 7fdd197fa640 10 client.27166 handle_command_reply: tid=0 2024-03-18T22:49:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.291+0000 7fdd3aa00740 1 client.27166 shutdown 2024-03-18T22:49:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.291+0000 7fdd3aa00740 1 -- 172.21.15.40:0/3838859766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc8048d2f0 msgr2=0x55bc80497740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.291+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/3838859766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc8048d2f0 0x55bc80497740 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c0bd220 tx=0x7fdd2c0f47b0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.291+0000 7fdd3aa00740 1 -- 172.21.15.40:0/3838859766 shutdown_connections 2024-03-18T22:49:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.291+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/3838859766 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bc804a3590 0x55bc804a5a20 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.291+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/3838859766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc80492e50 0x55bc8049ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.291+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/3838859766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 0x55bc80497c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.291+0000 7fdd197fa640 0 client.27166 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:49:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.291+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/3838859766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc8048d2f0 0x55bc80497740 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.291+0000 7fdd3aa00740 1 -- 172.21.15.40:0/3838859766 >> 172.21.15.40:0/3838859766 conn(0x55bc80488860 msgr2=0x55bc80487ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.291+0000 7fdd3aa00740 1 -- 172.21.15.40:0/3838859766 shutdown_connections 2024-03-18T22:49:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.291+0000 7fdd3aa00740 1 -- 172.21.15.40:0/3838859766 wait complete. 2024-03-18T22:49:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.291+0000 7fdd3aa00740 20 client.27166 trim_cache size 0 max 16384 2024-03-18T22:49:42.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd3aa00740 1 Processor -- start 2024-03-18T22:49:42.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd3aa00740 1 -- start start 2024-03-18T22:49:42.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd3aa00740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc8048d2f0 0x55bc80497350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd3aa00740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 0x55bc8049ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd3aa00740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc80492e50 0x55bc80497890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd3aa00740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301ac6d0 con 0x55bc8048d2f0 2024-03-18T22:49:42.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd3aa00740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301ae2d0 con 0x55bc8048dce0 2024-03-18T22:49:42.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd3aa00740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301acda0 con 0x55bc80492e50 2024-03-18T22:49:42.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd35bae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc8048d2f0 0x55bc80497350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd35bae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc8048d2f0 0x55bc80497350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33340/0 (socket says 172.21.15.40:33340) 2024-03-18T22:49:42.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd353ad640 1 --2- 172.21.15.40:0/2850910339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 0x55bc8049ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd35bae640 1 -- 172.21.15.40:0/2850910339 learned_addr learned my addr 172.21.15.40:0/2850910339 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd363af640 1 --2- 172.21.15.40:0/2850910339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc80492e50 0x55bc80497890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd35bae640 1 -- 172.21.15.40:0/2850910339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc80492e50 msgr2=0x55bc80497890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd35bae640 1 --2- 172.21.15.40:0/2850910339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc80492e50 0x55bc80497890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd35bae640 1 -- 172.21.15.40:0/2850910339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 msgr2=0x55bc8049ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd35bae640 1 --2- 172.21.15.40:0/2850910339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 0x55bc8049ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd35bae640 1 -- 172.21.15.40:0/2850910339 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc8045e880 con 0x55bc8048d2f0 2024-03-18T22:49:42.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd353ad640 1 --2- 172.21.15.40:0/2850910339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 0x55bc8049ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd363af640 1 --2- 172.21.15.40:0/2850910339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc80492e50 0x55bc80497890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.309+0000 7fdd35bae640 1 --2- 172.21.15.40:0/2850910339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc8048d2f0 0x55bc80497350 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c0c5790 tx=0x7fdd2c0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7e301183138c8c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.310+0000 7fdd1b7fe640 1 -- 172.21.15.40:0/2850910339 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c0da070 con 0x55bc8048d2f0 2024-03-18T22:49:42.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.310+0000 7fdd1b7fe640 1 -- 172.21.15.40:0/2850910339 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd2c0bdc20 con 0x55bc8048d2f0 2024-03-18T22:49:42.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.310+0000 7fdd1b7fe640 1 -- 172.21.15.40:0/2850910339 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c0bd060 con 0x55bc8048d2f0 2024-03-18T22:49:42.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.311+0000 7fdd3aa00740 1 -- 172.21.15.40:0/2850910339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc8048d2f0 msgr2=0x55bc80497350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.311+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/2850910339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc8048d2f0 0x55bc80497350 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c0c5790 tx=0x7fdd2c0035f0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.311+0000 7fdd3aa00740 1 -- 172.21.15.40:0/2850910339 shutdown_connections 2024-03-18T22:49:42.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.311+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/2850910339 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc80492e50 0x55bc80497890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.311+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/2850910339 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048dce0 0x55bc8049ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.311+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/2850910339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc8048d2f0 0x55bc80497350 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.311+0000 7fdd3aa00740 1 -- 172.21.15.40:0/2850910339 >> 172.21.15.40:0/2850910339 conn(0x55bc80488860 msgr2=0x55bc80483a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.311+0000 7fdd3aa00740 1 -- 172.21.15.40:0/2850910339 shutdown_connections 2024-03-18T22:49:42.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.311+0000 7fdd3aa00740 1 -- 172.21.15.40:0/2850910339 wait complete. 2024-03-18T22:49:42.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.311+0000 7fdd3aa00740 1 -- start start 2024-03-18T22:49:42.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.311+0000 7fdd3aa00740 1 Processor -- start 2024-03-18T22:49:42.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd3aa00740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048d2f0 0x55bc8049bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd3aa00740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc8048dce0 0x55bc8049c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd3aa00740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc80492e50 0x55bc8049d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd3aa00740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301adc00 con 0x55bc80492e50 2024-03-18T22:49:42.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd3aa00740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301ad470 con 0x55bc8048d2f0 2024-03-18T22:49:42.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd3aa00740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301aa6b0 con 0x55bc8048dce0 2024-03-18T22:49:42.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd3aa00740 10 client.0 _ll_register_callbacks cb 0x7fdd2c0057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:42.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd35bae640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048d2f0 0x55bc8049bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd363af640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc80492e50 0x55bc8049d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd35bae640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048d2f0 0x55bc8049bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:41996/0 (socket says 172.21.15.40:41996) 2024-03-18T22:49:42.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd35bae640 1 -- 172.21.15.40:0/4112788896 learned_addr learned my addr 172.21.15.40:0/4112788896 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd353ad640 1 --2- 172.21.15.40:0/4112788896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc8048dce0 0x55bc8049c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd363af640 1 -- 172.21.15.40:0/4112788896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc8048dce0 msgr2=0x55bc8049c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd363af640 1 --2- 172.21.15.40:0/4112788896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc8048dce0 0x55bc8049c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd363af640 1 -- 172.21.15.40:0/4112788896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048d2f0 msgr2=0x55bc8049bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd363af640 1 --2- 172.21.15.40:0/4112788896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048d2f0 0x55bc8049bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd363af640 1 -- 172.21.15.40:0/4112788896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc8049da50 con 0x55bc80492e50 2024-03-18T22:49:42.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd35bae640 1 --2- 172.21.15.40:0/4112788896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048d2f0 0x55bc8049bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd353ad640 1 --2- 172.21.15.40:0/4112788896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc8048dce0 0x55bc8049c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd363af640 1 --2- 172.21.15.40:0/4112788896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc80492e50 0x55bc8049d460 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdd28005fd0 tx=0x7fdd28004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11df272a5bc032d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd197fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:42.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd197fa640 1 -- 172.21.15.40:0/4112788896 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2801c660 con 0x55bc80492e50 2024-03-18T22:49:42.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd197fa640 1 -- 172.21.15.40:0/4112788896 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdd2801c800 con 0x55bc80492e50 2024-03-18T22:49:42.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.312+0000 7fdd3aa00740 1 -- 172.21.15.40:0/4112788896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd2c0bdc20 con 0x55bc80492e50 2024-03-18T22:49:42.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.313+0000 7fdd197fa640 1 -- 172.21.15.40:0/4112788896 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fdd28039050 con 0x55bc80492e50 2024-03-18T22:49:42.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.313+0000 7fdd3aa00740 10 client.17364 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:42.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.313+0000 7fdd3aa00740 1 -- 172.21.15.40:0/4112788896 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55bc8049ee20 con 0x55bc80492e50 2024-03-18T22:49:42.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.313+0000 7fdd197fa640 1 -- 172.21.15.40:0/4112788896 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd2c0bdc20 con 0x55bc80492e50 2024-03-18T22:49:42.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.313+0000 7fdd197fa640 1 -- 172.21.15.40:0/4112788896 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fdd2802ba10 con 0x55bc80492e50 2024-03-18T22:49:42.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.313+0000 7fdd3aa00740 10 client.17364 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:42.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.313+0000 7fdd3aa00740 10 client.17364 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:49:42.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.313+0000 7fdd3aa00740 20 client.17364 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:42.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.313+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/4112788896 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bc8050a0a0 0x55bc8050c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.313+0000 7fdd3aa00740 4 client.17364 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:49:42.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.313+0000 7fdd3aa00740 1 -- 172.21.15.40:0/4112788896 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bc803f8d40 con 0x55bc8050a0a0 2024-03-18T22:49:42.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.313+0000 7fdd35bae640 1 --2- 172.21.15.40:0/4112788896 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bc8050a0a0 0x55bc8050c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd35bae640 1 --2- 172.21.15.40:0/4112788896 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bc8050a0a0 0x55bc8050c4c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a800d2857bf9e49c server_cookie=47d72ce1602b6c53 in_seq=0 out_seq=0 2024-03-18T22:49:42.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd197fa640 10 client.17364 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:49:42.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd197fa640 1 -- 172.21.15.40:0/4112788896 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bc803f8d40 con 0x55bc8050a0a0 2024-03-18T22:49:42.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd197fa640 10 client.17364 handle_command_reply: tid=0 2024-03-18T22:49:42.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd3aa00740 1 client.17364 shutdown 2024-03-18T22:49:42.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd3aa00740 1 -- 172.21.15.40:0/4112788896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc80492e50 msgr2=0x55bc8049d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/4112788896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc80492e50 0x55bc8049d460 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdd28005fd0 tx=0x7fdd28004ab0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd3aa00740 1 -- 172.21.15.40:0/4112788896 shutdown_connections 2024-03-18T22:49:42.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/4112788896 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55bc8050a0a0 0x55bc8050c4c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/4112788896 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55bc80492e50 0x55bc8049d460 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd197fa640 0 client.17364 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:49:42.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/4112788896 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55bc8048dce0 0x55bc8049c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd3aa00740 1 --2- 172.21.15.40:0/4112788896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bc8048d2f0 0x55bc8049bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd3aa00740 1 -- 172.21.15.40:0/4112788896 >> 172.21.15.40:0/4112788896 conn(0x55bc80488860 msgr2=0x55bc8048bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd3aa00740 1 -- 172.21.15.40:0/4112788896 shutdown_connections 2024-03-18T22:49:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd3aa00740 1 -- 172.21.15.40:0/4112788896 wait complete. 2024-03-18T22:49:42.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.314+0000 7fdd3aa00740 20 client.17364 trim_cache size 0 max 16384 2024-03-18T22:49:42.330 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:49:42.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.315+0000 7fdd37e12640 1 -- 172.21.15.40:0/1714448452 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdd1c0a2800 msgr2=0x7fdd1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.315+0000 7fdd37e12640 1 --2- 172.21.15.40:0/1714448452 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdd1c0a2800 0x7fdd1c0a4cc0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x55bc8048e540 tx=0x7fdd28004a40 comp rx=0 tx=0).stop 2024-03-18T22:49:42.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.315+0000 7fdd37e12640 1 -- 172.21.15.40:0/1714448452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30103a50 msgr2=0x7fdd30199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.315+0000 7fdd37e12640 1 --2- 172.21.15.40:0/1714448452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30103a50 0x7fdd30199c90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c000c00 tx=0x7fdd2c005dc0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.315+0000 7fdd37e12640 1 -- 172.21.15.40:0/1714448452 shutdown_connections 2024-03-18T22:49:42.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.315+0000 7fdd37e12640 1 --2- 172.21.15.40:0/1714448452 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdd1c0a2800 0x7fdd1c0a4cc0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.315+0000 7fdd37e12640 1 --2- 172.21.15.40:0/1714448452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd30105600 0x7fdd301a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.315+0000 7fdd37e12640 1 --2- 172.21.15.40:0/1714448452 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdd30104c40 0x7fdd3019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.315+0000 7fdd37e12640 1 --2- 172.21.15.40:0/1714448452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd30103a50 0x7fdd30199c90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.315+0000 7fdd37e12640 1 -- 172.21.15.40:0/1714448452 >> 172.21.15.40:0/1714448452 conn(0x7fdd300fefe0 msgr2=0x7fdd30108060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.315+0000 7fdd37e12640 1 -- 172.21.15.40:0/1714448452 shutdown_connections 2024-03-18T22:49:42.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.315+0000 7fdd37e12640 1 -- 172.21.15.40:0/1714448452 wait complete. 2024-03-18T22:49:42.334 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:42 smithi040 ceph-mon[62643]: pgmap v562: 129 pgs: 129 active+clean; 8.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:49:42.334 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:42 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2763010516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:49:42.335 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:42 smithi113 ceph-mon[66775]: pgmap v562: 129 pgs: 129 active+clean; 8.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:49:42.335 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:42 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2763010516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:49:42.336 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:49:42.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:42 smithi103 ceph-mon[66377]: pgmap v562: 129 pgs: 129 active+clean; 8.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:49:42.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:42 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2763010516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:49:42.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.454+0000 7fb8e49df640 1 -- 172.21.15.40:0/401721317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e0102310 msgr2=0x7fb8e0108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.454+0000 7fb8e49df640 1 --2- 172.21.15.40:0/401721317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e0102310 0x7fb8e0108ba0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8009ca0 tx=0x7fb8d802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.454+0000 7fb8e49df640 1 -- 172.21.15.40:0/401721317 shutdown_connections 2024-03-18T22:49:42.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.454+0000 7fb8e49df640 1 --2- 172.21.15.40:0/401721317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e0102310 0x7fb8e0108ba0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.454+0000 7fb8e49df640 1 --2- 172.21.15.40:0/401721317 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e0101950 0x7fb8e0101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.454+0000 7fb8e49df640 1 --2- 172.21.15.40:0/401721317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8e0100760 0x7fb8e0100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.454+0000 7fb8e49df640 1 -- 172.21.15.40:0/401721317 >> 172.21.15.40:0/401721317 conn(0x7fb8e00fbcf0 msgr2=0x7fb8e00fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.454+0000 7fb8e49df640 1 -- 172.21.15.40:0/401721317 shutdown_connections 2024-03-18T22:49:42.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.454+0000 7fb8e49df640 1 -- 172.21.15.40:0/401721317 wait complete. 2024-03-18T22:49:42.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.454+0000 7fb8e49df640 1 Processor -- start 2024-03-18T22:49:42.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.454+0000 7fb8e49df640 1 -- start start 2024-03-18T22:49:42.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8e49df640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e0100760 0x7fb8e0197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8e49df640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8e0101950 0x7fb8e0197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8e49df640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e0102310 0x7fb8e019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8e49df640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e010c290 con 0x7fb8e0102310 2024-03-18T22:49:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8e49df640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e010c110 con 0x7fb8e0101950 2024-03-18T22:49:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8e49df640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e010c410 con 0x7fb8e0100760 2024-03-18T22:49:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8ded9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e0102310 0x7fb8e019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8de59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e0100760 0x7fb8e0197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8ded9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e0102310 0x7fb8e019efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33356/0 (socket says 172.21.15.40:33356) 2024-03-18T22:49:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8ded9d640 1 -- 172.21.15.40:0/1258261292 learned_addr learned my addr 172.21.15.40:0/1258261292 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8d7fff640 1 --2- 172.21.15.40:0/1258261292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8e0101950 0x7fb8e0197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8de59c640 1 -- 172.21.15.40:0/1258261292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8e0101950 msgr2=0x7fb8e0197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8de59c640 1 --2- 172.21.15.40:0/1258261292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8e0101950 0x7fb8e0197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8de59c640 1 -- 172.21.15.40:0/1258261292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e0102310 msgr2=0x7fb8e019efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8de59c640 1 --2- 172.21.15.40:0/1258261292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e0102310 0x7fb8e019efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8de59c640 1 -- 172.21.15.40:0/1258261292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e019f640 con 0x7fb8e0100760 2024-03-18T22:49:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8ded9d640 1 --2- 172.21.15.40:0/1258261292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e0102310 0x7fb8e019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8d7fff640 1 --2- 172.21.15.40:0/1258261292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8e0101950 0x7fb8e0197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.455+0000 7fb8de59c640 1 --2- 172.21.15.40:0/1258261292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e0100760 0x7fb8e0197a20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb8d000e9c0 tx=0x7fb8d000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.456+0000 7fb8d77fe640 1 -- 172.21.15.40:0/1258261292 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d000ccc0 con 0x7fb8e0100760 2024-03-18T22:49:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.456+0000 7fb8e49df640 1 -- 172.21.15.40:0/1258261292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8e019f930 con 0x7fb8e0100760 2024-03-18T22:49:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.456+0000 7fb8d77fe640 1 -- 172.21.15.40:0/1258261292 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8d00035f0 con 0x7fb8e0100760 2024-03-18T22:49:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.456+0000 7fb8d77fe640 1 -- 172.21.15.40:0/1258261292 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d000ce60 con 0x7fb8e0100760 2024-03-18T22:49:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.456+0000 7fb8e49df640 1 -- 172.21.15.40:0/1258261292 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8e019fed0 con 0x7fb8e0100760 2024-03-18T22:49:42.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.458+0000 7fb8d77fe640 1 -- 172.21.15.40:0/1258261292 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb8d00037a0 con 0x7fb8e0100760 2024-03-18T22:49:42.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.459+0000 7fb8e75c6740 1 Processor -- start 2024-03-18T22:49:42.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.459+0000 7fb8e75c6740 1 -- start start 2024-03-18T22:49:42.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.459+0000 7fb8e75c6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ac700 0x55d4b36acb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.459+0000 7fb8e75c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36ad0f0 0x55d4b36b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.459+0000 7fb8e75c6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36b2260 0x55d4b36b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.459+0000 7fb8e75c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d4b360c4b0 con 0x55d4b36ad0f0 2024-03-18T22:49:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.459+0000 7fb8e75c6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d4b35b4d70 con 0x55d4b36b2260 2024-03-18T22:49:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.459+0000 7fb8e75c6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55d4b3608550 con 0x55d4b36ac700 2024-03-18T22:49:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.459+0000 7fb8de59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36ad0f0 0x55d4b36b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.459+0000 7fb8de59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36ad0f0 0x55d4b36b1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33366/0 (socket says 172.21.15.40:33366) 2024-03-18T22:49:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.459+0000 7fb8de59c640 1 -- 172.21.15.40:0/3870820652 learned_addr learned my addr 172.21.15.40:0/3870820652 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.459+0000 7fb8ded9d640 1 --2- 172.21.15.40:0/3870820652 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ac700 0x55d4b36acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.459+0000 7fb8d77fe640 1 --2- 172.21.15.40:0/1258261292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb8cc0a2800 0x7fb8cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.459+0000 7fb8d7fff640 1 --2- 172.21.15.40:0/3870820652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36b2260 0x55d4b36b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.459+0000 7fb8ded9d640 1 --2- 172.21.15.40:0/1258261292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb8cc0a2800 0x7fb8cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.460+0000 7fb8d77fe640 1 -- 172.21.15.40:0/1258261292 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb8d00cf290 con 0x7fb8e0100760 2024-03-18T22:49:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.460+0000 7fb8ded9d640 1 -- 172.21.15.40:0/3870820652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36b2260 msgr2=0x55d4b36b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.460+0000 7fb8ded9d640 1 --2- 172.21.15.40:0/3870820652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36b2260 0x55d4b36b46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.460+0000 7fb8ded9d640 1 -- 172.21.15.40:0/3870820652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36ad0f0 msgr2=0x55d4b36b1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.460+0000 7fb8ded9d640 1 --2- 172.21.15.40:0/3870820652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36ad0f0 0x55d4b36b1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.460+0000 7fb8ded9d640 1 -- 172.21.15.40:0/3870820652 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4b35d7210 con 0x55d4b36ac700 2024-03-18T22:49:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.460+0000 7fb8ded9d640 1 --2- 172.21.15.40:0/3870820652 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ac700 0x55d4b36acb20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8032380 tx=0x7fb8d80328e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da76aaf1dfcab088 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.460+0000 7fb8ded9d640 1 --2- 172.21.15.40:0/1258261292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb8cc0a2800 0x7fb8cc0a4cc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8047000 tx=0x7fb8d8048000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.460+0000 7fb8d4ff9640 1 -- 172.21.15.40:0/3870820652 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d8050070 con 0x55d4b36ac700 2024-03-18T22:49:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.460+0000 7fb8d4ff9640 1 -- 172.21.15.40:0/3870820652 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8d80046f0 con 0x55d4b36ac700 2024-03-18T22:49:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.460+0000 7fb8d4ff9640 1 -- 172.21.15.40:0/3870820652 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d8039040 con 0x55d4b36ac700 2024-03-18T22:49:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.461+0000 7fb8e75c6740 1 -- 172.21.15.40:0/3870820652 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ac700 msgr2=0x55d4b36acb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.461+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/3870820652 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ac700 0x55d4b36acb20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8032380 tx=0x7fb8d80328e0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.461+0000 7fb8e75c6740 1 -- 172.21.15.40:0/3870820652 shutdown_connections 2024-03-18T22:49:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.461+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/3870820652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36b2260 0x55d4b36b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.461+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/3870820652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36ad0f0 0x55d4b36b1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.461+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/3870820652 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ac700 0x55d4b36acb20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.461+0000 7fb8e75c6740 1 -- 172.21.15.40:0/3870820652 >> 172.21.15.40:0/3870820652 conn(0x55d4b36a3860 msgr2=0x55d4b36a3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.461+0000 7fb8e75c6740 1 -- 172.21.15.40:0/3870820652 shutdown_connections 2024-03-18T22:49:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.461+0000 7fb8e75c6740 1 -- 172.21.15.40:0/3870820652 wait complete. 2024-03-18T22:49:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.461+0000 7fb8e75c6740 1 -- start start 2024-03-18T22:49:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.461+0000 7fb8e75c6740 1 Processor -- start 2024-03-18T22:49:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8e75c6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ac700 0x55d4b36b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8e75c6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ad0f0 0x55d4b36b7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8e75c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36b2260 0x55d4b36be140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8e75c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d4b36b51b0 con 0x55d4b36b2260 2024-03-18T22:49:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8e75c6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d4b35c8310 con 0x55d4b36ad0f0 2024-03-18T22:49:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8e75c6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55d4b36b5330 con 0x55d4b36ac700 2024-03-18T22:49:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8e75c6740 10 client.0 _ll_register_callbacks cb 0x55d4b35bc2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8de59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ac700 0x55d4b36b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8ded9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36b2260 0x55d4b36be140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8de59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ac700 0x55d4b36b6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:60488/0 (socket says 172.21.15.40:60488) 2024-03-18T22:49:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8de59c640 1 -- 172.21.15.40:0/1817597362 learned_addr learned my addr 172.21.15.40:0/1817597362 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8d7fff640 1 --2- 172.21.15.40:0/1817597362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ad0f0 0x55d4b36b7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8ded9d640 1 -- 172.21.15.40:0/1817597362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ac700 msgr2=0x55d4b36b6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8ded9d640 1 --2- 172.21.15.40:0/1817597362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ac700 0x55d4b36b6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8ded9d640 1 -- 172.21.15.40:0/1817597362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ad0f0 msgr2=0x55d4b36b7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8ded9d640 1 --2- 172.21.15.40:0/1817597362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ad0f0 0x55d4b36b7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8ded9d640 1 -- 172.21.15.40:0/1817597362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4b33203f0 con 0x55d4b36b2260 2024-03-18T22:49:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8de59c640 1 --2- 172.21.15.40:0/1817597362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ac700 0x55d4b36b6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8d7fff640 1 --2- 172.21.15.40:0/1817597362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ad0f0 0x55d4b36b7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8ded9d640 1 --2- 172.21.15.40:0/1817597362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36b2260 0x55d4b36be140 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8006010 tx=0x7fb8d8004460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6feb8f5264706065 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8c5ffb640 1 -- 172.21.15.40:0/1817597362 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d8057020 con 0x55d4b36b2260 2024-03-18T22:49:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8c5ffb640 1 -- 172.21.15.40:0/1817597362 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb8d8043d10 con 0x55d4b36b2260 2024-03-18T22:49:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.462+0000 7fb8e75c6740 1 -- 172.21.15.40:0/1817597362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8d80046f0 con 0x55d4b36b2260 2024-03-18T22:49:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.463+0000 7fb8c5ffb640 1 -- 172.21.15.40:0/1817597362 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb8d8062050 con 0x55d4b36b2260 2024-03-18T22:49:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.463+0000 7fb8e75c6740 10 client.17376 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.463+0000 7fb8e75c6740 1 -- 172.21.15.40:0/1817597362 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55d4b33ea7d0 con 0x55d4b36b2260 2024-03-18T22:49:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.463+0000 7fb8c5ffb640 1 -- 172.21.15.40:0/1817597362 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8d80046f0 con 0x55d4b36b2260 2024-03-18T22:49:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.463+0000 7fb8c5ffb640 1 -- 172.21.15.40:0/1817597362 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fb8d803f070 con 0x55d4b36b2260 2024-03-18T22:49:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.463+0000 7fb8e75c6740 10 client.17376 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.463+0000 7fb8e75c6740 10 client.17376 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:49:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.463+0000 7fb8e75c6740 20 client.17376 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.463+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/1817597362 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d4b36c2680 0x55d4b36c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.463+0000 7fb8e75c6740 4 client.17376 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.463+0000 7fb8e75c6740 1 -- 172.21.15.40:0/1817597362 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4b366a3e0 con 0x55d4b36c2680 2024-03-18T22:49:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.463+0000 7fb8de59c640 1 --2- 172.21.15.40:0/1817597362 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d4b36c2680 0x55d4b36c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.464+0000 7fb8de59c640 1 --2- 172.21.15.40:0/1817597362 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d4b36c2680 0x55d4b36c4ab0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cbe048a834065a2d server_cookie=fe7a9388c8f197c5 in_seq=0 out_seq=0 2024-03-18T22:49:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.464+0000 7fb8c5ffb640 10 client.17376 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:49:42.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.465+0000 7fb8c5ffb640 1 -- 172.21.15.40:0/1817597362 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d4b366a3e0 con 0x55d4b36c2680 2024-03-18T22:49:42.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.465+0000 7fb8c5ffb640 10 client.17376 handle_command_reply: tid=0 2024-03-18T22:49:42.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.465+0000 7fb8e75c6740 1 client.17376 shutdown 2024-03-18T22:49:42.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.465+0000 7fb8e75c6740 1 -- 172.21.15.40:0/1817597362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36b2260 msgr2=0x55d4b36be140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.465+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/1817597362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36b2260 0x55d4b36be140 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8006010 tx=0x7fb8d8004460 comp rx=0 tx=0).stop 2024-03-18T22:49:42.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.465+0000 7fb8e75c6740 1 -- 172.21.15.40:0/1817597362 shutdown_connections 2024-03-18T22:49:42.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.465+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/1817597362 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d4b36c2680 0x55d4b36c4ab0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.465+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/1817597362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36b2260 0x55d4b36be140 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.465+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/1817597362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ad0f0 0x55d4b36b7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.465+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/1817597362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ac700 0x55d4b36b6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.465+0000 7fb8e75c6740 1 -- 172.21.15.40:0/1817597362 >> 172.21.15.40:0/1817597362 conn(0x55d4b36a3860 msgr2=0x55d4b36a2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.465+0000 7fb8c5ffb640 0 client.17376 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:49:42.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.465+0000 7fb8e75c6740 1 -- 172.21.15.40:0/1817597362 shutdown_connections 2024-03-18T22:49:42.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.465+0000 7fb8e75c6740 1 -- 172.21.15.40:0/1817597362 wait complete. 2024-03-18T22:49:42.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.465+0000 7fb8e75c6740 20 client.17376 trim_cache size 0 max 16384 2024-03-18T22:49:42.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.478+0000 7fb8e75c6740 1 Processor -- start 2024-03-18T22:49:42.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.478+0000 7fb8e75c6740 1 -- start start 2024-03-18T22:49:42.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.478+0000 7fb8e75c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36ac700 0x55d4b369ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.478+0000 7fb8e75c6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ad0f0 0x55d4b36be100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.478+0000 7fb8e75c6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36b2260 0x55d4b36ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.478+0000 7fb8e75c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e01a8370 con 0x55d4b36ac700 2024-03-18T22:49:42.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.478+0000 7fb8e75c6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e01a9f70 con 0x55d4b36ad0f0 2024-03-18T22:49:42.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.478+0000 7fb8e75c6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e01a8a40 con 0x55d4b36b2260 2024-03-18T22:49:42.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.478+0000 7fb8de59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36ac700 0x55d4b369ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.478+0000 7fb8de59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36ac700 0x55d4b369ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33386/0 (socket says 172.21.15.40:33386) 2024-03-18T22:49:42.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.478+0000 7fb8d7fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ad0f0 0x55d4b36be100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.479+0000 7fb8de59c640 1 -- 172.21.15.40:0/3771792891 learned_addr learned my addr 172.21.15.40:0/3771792891 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.479+0000 7fb8ded9d640 1 --2- 172.21.15.40:0/3771792891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36b2260 0x55d4b36ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.479+0000 7fb8de59c640 1 -- 172.21.15.40:0/3771792891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36b2260 msgr2=0x55d4b36ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.479+0000 7fb8de59c640 1 --2- 172.21.15.40:0/3771792891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36b2260 0x55d4b36ae650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.479+0000 7fb8de59c640 1 -- 172.21.15.40:0/3771792891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ad0f0 msgr2=0x55d4b36be100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.479+0000 7fb8de59c640 1 --2- 172.21.15.40:0/3771792891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ad0f0 0x55d4b36be100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.479+0000 7fb8de59c640 1 -- 172.21.15.40:0/3771792891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4b3679880 con 0x55d4b36ac700 2024-03-18T22:49:42.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.479+0000 7fb8d7fff640 1 --2- 172.21.15.40:0/3771792891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ad0f0 0x55d4b36be100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.479+0000 7fb8de59c640 1 --2- 172.21.15.40:0/3771792891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36ac700 0x55d4b369ea30 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb8d00981c0 tx=0x7fb8d00cc870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ce140599d2fb31 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.479+0000 7fb8c7fff640 1 -- 172.21.15.40:0/3771792891 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d00cce60 con 0x55d4b36ac700 2024-03-18T22:49:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.479+0000 7fb8c7fff640 1 -- 172.21.15.40:0/3771792891 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8d0003b10 con 0x55d4b36ac700 2024-03-18T22:49:42.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.479+0000 7fb8c7fff640 1 -- 172.21.15.40:0/3771792891 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d009f990 con 0x55d4b36ac700 2024-03-18T22:49:42.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.480+0000 7fb8e75c6740 1 -- 172.21.15.40:0/3771792891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36ac700 msgr2=0x55d4b369ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.480+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/3771792891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36ac700 0x55d4b369ea30 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb8d00981c0 tx=0x7fb8d00cc870 comp rx=0 tx=0).stop 2024-03-18T22:49:42.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.480+0000 7fb8e75c6740 1 -- 172.21.15.40:0/3771792891 shutdown_connections 2024-03-18T22:49:42.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.480+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/3771792891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36b2260 0x55d4b36ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.480+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/3771792891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ad0f0 0x55d4b36be100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.480+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/3771792891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36ac700 0x55d4b369ea30 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.480+0000 7fb8e75c6740 1 -- 172.21.15.40:0/3771792891 >> 172.21.15.40:0/3771792891 conn(0x55d4b36a3860 msgr2=0x55d4b36b56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.480+0000 7fb8e75c6740 1 -- 172.21.15.40:0/3771792891 shutdown_connections 2024-03-18T22:49:42.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.480+0000 7fb8e75c6740 1 -- 172.21.15.40:0/3771792891 wait complete. 2024-03-18T22:49:42.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.480+0000 7fb8e75c6740 1 -- start start 2024-03-18T22:49:42.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.480+0000 7fb8e75c6740 1 Processor -- start 2024-03-18T22:49:42.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8e75c6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ac700 0x55d4b36bb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8e75c6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ad0f0 0x55d4b36bb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8e75c6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36b2260 0x55d4b37269a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8e75c6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e01a98a0 con 0x55d4b36b2260 2024-03-18T22:49:42.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8e75c6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e01a9110 con 0x55d4b36ac700 2024-03-18T22:49:42.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8e75c6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e01a6350 con 0x55d4b36ad0f0 2024-03-18T22:49:42.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8e75c6740 10 client.0 _ll_register_callbacks cb 0x7fb8d804e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:42.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8ded9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36b2260 0x55d4b37269a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8ded9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36b2260 0x55d4b37269a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33396/0 (socket says 172.21.15.40:33396) 2024-03-18T22:49:42.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8ded9d640 1 -- 172.21.15.40:0/2985377823 learned_addr learned my addr 172.21.15.40:0/2985377823 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8d7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ad0f0 0x55d4b36bb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8de59c640 1 --2- 172.21.15.40:0/2985377823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ac700 0x55d4b36bb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8ded9d640 1 -- 172.21.15.40:0/2985377823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ad0f0 msgr2=0x55d4b36bb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8ded9d640 1 --2- 172.21.15.40:0/2985377823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ad0f0 0x55d4b36bb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8ded9d640 1 -- 172.21.15.40:0/2985377823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ac700 msgr2=0x55d4b36bb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8ded9d640 1 --2- 172.21.15.40:0/2985377823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ac700 0x55d4b36bb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8ded9d640 1 -- 172.21.15.40:0/2985377823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4b3726ee0 con 0x55d4b36b2260 2024-03-18T22:49:42.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8d7fff640 1 --2- 172.21.15.40:0/2985377823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ad0f0 0x55d4b36bb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8ded9d640 1 --2- 172.21.15.40:0/2985377823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36b2260 0x55d4b37269a0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8005f50 tx=0x7fb8d80324f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5231813d97732f2f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.481+0000 7fb8c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:42.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.482+0000 7fb8c5ffb640 1 -- 172.21.15.40:0/2985377823 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d806b060 con 0x55d4b36b2260 2024-03-18T22:49:42.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.482+0000 7fb8c5ffb640 1 -- 172.21.15.40:0/2985377823 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb8d8043400 con 0x55d4b36b2260 2024-03-18T22:49:42.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.482+0000 7fb8e75c6740 1 -- 172.21.15.40:0/2985377823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d4b36199f0 con 0x55d4b36b2260 2024-03-18T22:49:42.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.482+0000 7fb8c5ffb640 1 -- 172.21.15.40:0/2985377823 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb8d8072050 con 0x55d4b36b2260 2024-03-18T22:49:42.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.482+0000 7fb8e75c6740 10 client.17382 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:42.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.482+0000 7fb8e75c6740 1 -- 172.21.15.40:0/2985377823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55d4b3728110 con 0x55d4b36b2260 2024-03-18T22:49:42.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.482+0000 7fb8c5ffb640 1 -- 172.21.15.40:0/2985377823 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55d4b36199f0 con 0x55d4b36b2260 2024-03-18T22:49:42.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.482+0000 7fb8c5ffb640 1 -- 172.21.15.40:0/2985377823 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fb8d803f040 con 0x55d4b36b2260 2024-03-18T22:49:42.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.482+0000 7fb8e75c6740 10 client.17382 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:42.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.482+0000 7fb8e75c6740 10 client.17382 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:49:42.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.482+0000 7fb8e75c6740 20 client.17382 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:42.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.482+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/2985377823 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d4b3728ac0 0x55d4b372aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.482+0000 7fb8e75c6740 4 client.17382 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:49:42.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.482+0000 7fb8e75c6740 1 -- 172.21.15.40:0/2985377823 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d4b3613d40 con 0x55d4b3728ac0 2024-03-18T22:49:42.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.482+0000 7fb8de59c640 1 --2- 172.21.15.40:0/2985377823 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d4b3728ac0 0x55d4b372aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8c5ffb640 10 client.17382 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:49:42.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8de59c640 1 --2- 172.21.15.40:0/2985377823 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d4b3728ac0 0x55d4b372aee0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ba3234448344f57b server_cookie=7994420082a31a99 in_seq=0 out_seq=0 2024-03-18T22:49:42.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8c5ffb640 1 -- 172.21.15.40:0/2985377823 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d4b3613d40 con 0x55d4b3728ac0 2024-03-18T22:49:42.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8c5ffb640 10 client.17382 handle_command_reply: tid=0 2024-03-18T22:49:42.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8e75c6740 1 client.17382 shutdown 2024-03-18T22:49:42.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8e75c6740 1 -- 172.21.15.40:0/2985377823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36b2260 msgr2=0x55d4b37269a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/2985377823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36b2260 0x55d4b37269a0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8005f50 tx=0x7fb8d80324f0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8e75c6740 1 -- 172.21.15.40:0/2985377823 shutdown_connections 2024-03-18T22:49:42.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/2985377823 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55d4b3728ac0 0x55d4b372aee0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/2985377823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d4b36b2260 0x55d4b37269a0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/2985377823 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d4b36ad0f0 0x55d4b36bb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8c5ffb640 0 client.17382 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:49:42.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8e75c6740 1 --2- 172.21.15.40:0/2985377823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d4b36ac700 0x55d4b36bb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8e75c6740 1 -- 172.21.15.40:0/2985377823 >> 172.21.15.40:0/2985377823 conn(0x55d4b36a3860 msgr2=0x55d4b36b6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8e75c6740 1 -- 172.21.15.40:0/2985377823 shutdown_connections 2024-03-18T22:49:42.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.483+0000 7fb8e75c6740 1 -- 172.21.15.40:0/2985377823 wait complete. 2024-03-18T22:49:42.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.484+0000 7fb8e75c6740 20 client.17382 trim_cache size 0 max 16384 2024-03-18T22:49:42.492 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:49:42.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.484+0000 7fb8e49df640 1 -- 172.21.15.40:0/1258261292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb8cc0a2800 msgr2=0x7fb8cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.484+0000 7fb8e49df640 1 --2- 172.21.15.40:0/1258261292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb8cc0a2800 0x7fb8cc0a4cc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8047000 tx=0x7fb8d8048000 comp rx=0 tx=0).stop 2024-03-18T22:49:42.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.484+0000 7fb8e49df640 1 -- 172.21.15.40:0/1258261292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e0100760 msgr2=0x7fb8e0197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.484+0000 7fb8e49df640 1 --2- 172.21.15.40:0/1258261292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e0100760 0x7fb8e0197a20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb8d000e9c0 tx=0x7fb8d000ee80 comp rx=0 tx=0).stop 2024-03-18T22:49:42.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.484+0000 7fb8e49df640 1 -- 172.21.15.40:0/1258261292 shutdown_connections 2024-03-18T22:49:42.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.484+0000 7fb8e49df640 1 --2- 172.21.15.40:0/1258261292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb8cc0a2800 0x7fb8cc0a4cc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.484+0000 7fb8e49df640 1 --2- 172.21.15.40:0/1258261292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8e0102310 0x7fb8e019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.484+0000 7fb8e49df640 1 --2- 172.21.15.40:0/1258261292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8e0101950 0x7fb8e0197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.484+0000 7fb8e49df640 1 --2- 172.21.15.40:0/1258261292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb8e0100760 0x7fb8e0197a20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.484+0000 7fb8e49df640 1 -- 172.21.15.40:0/1258261292 >> 172.21.15.40:0/1258261292 conn(0x7fb8e00fbcf0 msgr2=0x7fb8e00fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.484+0000 7fb8e49df640 1 -- 172.21.15.40:0/1258261292 shutdown_connections 2024-03-18T22:49:42.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.484+0000 7fb8e49df640 1 -- 172.21.15.40:0/1258261292 wait complete. 2024-03-18T22:49:42.497 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:49:42.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.609+0000 7f7c813e1640 1 -- 172.21.15.40:0/1838311139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c7c075800 msgr2=0x7f7c7c073c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.609+0000 7f7c813e1640 1 --2- 172.21.15.40:0/1838311139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c7c075800 0x7f7c7c073c50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c009900 tx=0x7f7c6c02bcc0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.609+0000 7f7c813e1640 1 -- 172.21.15.40:0/1838311139 shutdown_connections 2024-03-18T22:49:42.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.609+0000 7f7c813e1640 1 --2- 172.21.15.40:0/1838311139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c7c109740 0x7f7c7c10bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.609+0000 7f7c813e1640 1 --2- 172.21.15.40:0/1838311139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c7c074220 0x7f7c7c074680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.609+0000 7f7c813e1640 1 --2- 172.21.15.40:0/1838311139 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c7c075800 0x7f7c7c073c50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.609+0000 7f7c813e1640 1 -- 172.21.15.40:0/1838311139 >> 172.21.15.40:0/1838311139 conn(0x7f7c7c0fce40 msgr2=0x7f7c7c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.609+0000 7f7c813e1640 1 -- 172.21.15.40:0/1838311139 shutdown_connections 2024-03-18T22:49:42.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.609+0000 7f7c813e1640 1 -- 172.21.15.40:0/1838311139 wait complete. 2024-03-18T22:49:42.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c813e1640 1 Processor -- start 2024-03-18T22:49:42.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c813e1640 1 -- start start 2024-03-18T22:49:42.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c813e1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c7c074220 0x7f7c7c199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c813e1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c7c075800 0x7f7c7c19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c813e1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c7c109740 0x7f7c7c1a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c813e1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c10f540 con 0x7f7c7c075800 2024-03-18T22:49:42.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c813e1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c10f3c0 con 0x7f7c7c074220 2024-03-18T22:49:42.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c813e1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c10f6c0 con 0x7f7c7c109740 2024-03-18T22:49:42.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c7a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c7c075800 0x7f7c7c19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c7affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c7c074220 0x7f7c7c199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c7a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c7c075800 0x7f7c7c19a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33424/0 (socket says 172.21.15.40:33424) 2024-03-18T22:49:42.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c7a7fc640 1 -- 172.21.15.40:0/2069130001 learned_addr learned my addr 172.21.15.40:0/2069130001 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c7b7fe640 1 --2- 172.21.15.40:0/2069130001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c7c109740 0x7f7c7c1a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c7affd640 1 -- 172.21.15.40:0/2069130001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c7c109740 msgr2=0x7f7c7c1a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c7affd640 1 --2- 172.21.15.40:0/2069130001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c7c109740 0x7f7c7c1a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c7affd640 1 -- 172.21.15.40:0/2069130001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c7c075800 msgr2=0x7f7c7c19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c7affd640 1 --2- 172.21.15.40:0/2069130001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c7c075800 0x7f7c7c19a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c7affd640 1 -- 172.21.15.40:0/2069130001 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c7c1a1880 con 0x7f7c7c074220 2024-03-18T22:49:42.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.610+0000 7f7c7a7fc640 1 --2- 172.21.15.40:0/2069130001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c7c075800 0x7f7c7c19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.611+0000 7f7c7affd640 1 --2- 172.21.15.40:0/2069130001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c7c074220 0x7f7c7c199cc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c000c00 tx=0x7f7c6c004600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.611+0000 7f7c63fff640 1 -- 172.21.15.40:0/2069130001 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c043020 con 0x7f7c7c074220 2024-03-18T22:49:42.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.611+0000 7f7c813e1640 1 -- 172.21.15.40:0/2069130001 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c7c1a1b10 con 0x7f7c7c074220 2024-03-18T22:49:42.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.611+0000 7f7c63fff640 1 -- 172.21.15.40:0/2069130001 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c6c02dc10 con 0x7f7c7c074220 2024-03-18T22:49:42.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.611+0000 7f7c63fff640 1 -- 172.21.15.40:0/2069130001 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c003ae0 con 0x7f7c7c074220 2024-03-18T22:49:42.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.611+0000 7f7c813e1640 1 -- 172.21.15.40:0/2069130001 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c7c1a2080 con 0x7f7c7c074220 2024-03-18T22:49:42.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.613+0000 7f7c63fff640 1 -- 172.21.15.40:0/2069130001 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7c6c046020 con 0x7f7c7c074220 2024-03-18T22:49:42.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.614+0000 7f7c83fc5740 1 Processor -- start 2024-03-18T22:49:42.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.614+0000 7f7c83fc5740 1 -- start start 2024-03-18T22:49:42.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.614+0000 7f7c63fff640 1 --2- 172.21.15.40:0/2069130001 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7c640a2800 0x7f7c640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.614+0000 7f7c83fc5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e59082f0 0x5647e5908710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.614+0000 7f7c83fc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e590d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.614+0000 7f7c7b7fe640 1 --2- 172.21.15.40:0/2069130001 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7c640a2800 0x7f7c640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.614+0000 7f7c83fc5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 0x5647e59102e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.614+0000 7f7c83fc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5647e586c4b0 con 0x5647e5908ce0 2024-03-18T22:49:42.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.614+0000 7f7c83fc5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5647e5814d70 con 0x5647e59082f0 2024-03-18T22:49:42.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.614+0000 7f7c83fc5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5647e5868550 con 0x5647e590de50 2024-03-18T22:49:42.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.614+0000 7f7c63fff640 1 -- 172.21.15.40:0/2069130001 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7c6c0f0e30 con 0x7f7c7c074220 2024-03-18T22:49:42.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.614+0000 7f7c7affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e59082f0 0x5647e5908710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.614+0000 7f7c7affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e59082f0 0x5647e5908710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:42068/0 (socket says 172.21.15.40:42068) 2024-03-18T22:49:42.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.614+0000 7f7c7affd640 1 -- 172.21.15.40:0/109989811 learned_addr learned my addr 172.21.15.40:0/109989811 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.615+0000 7f7c7a7fc640 1 --2- 172.21.15.40:0/109989811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e590d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.615+0000 7f7c7b7fe640 1 --2- 172.21.15.40:0/109989811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 0x5647e59102e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.615+0000 7f7c7affd640 1 -- 172.21.15.40:0/109989811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 msgr2=0x5647e59102e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.615+0000 7f7c7affd640 1 --2- 172.21.15.40:0/109989811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 0x5647e59102e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.615+0000 7f7c7affd640 1 -- 172.21.15.40:0/109989811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 msgr2=0x5647e590d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.615+0000 7f7c7affd640 1 --2- 172.21.15.40:0/109989811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e590d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.615+0000 7f7c7affd640 1 -- 172.21.15.40:0/109989811 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647e5837210 con 0x5647e59082f0 2024-03-18T22:49:42.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.615+0000 7f7c7b7fe640 1 --2- 172.21.15.40:0/109989811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 0x5647e59102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:49:42.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.615+0000 7f7c7b7fe640 1 --2- 172.21.15.40:0/2069130001 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7c640a2800 0x7f7c640a4cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7c74005fd0 tx=0x7f7c74009f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.615+0000 7f7c7affd640 1 --2- 172.21.15.40:0/109989811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e59082f0 0x5647e5908710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0b9d20 tx=0x7f7c6c02bcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14dc08d4201f6054 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.615+0000 7f7c617fa640 1 -- 172.21.15.40:0/109989811 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0bd3e0 con 0x5647e59082f0 2024-03-18T22:49:42.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.615+0000 7f7c617fa640 1 -- 172.21.15.40:0/109989811 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c6c0c1530 con 0x5647e59082f0 2024-03-18T22:49:42.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.615+0000 7f7c617fa640 1 -- 172.21.15.40:0/109989811 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0bd640 con 0x5647e59082f0 2024-03-18T22:49:42.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.616+0000 7f7c83fc5740 1 -- 172.21.15.40:0/109989811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e59082f0 msgr2=0x5647e5908710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.616+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/109989811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e59082f0 0x5647e5908710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0b9d20 tx=0x7f7c6c02bcc0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.616+0000 7f7c83fc5740 1 -- 172.21.15.40:0/109989811 shutdown_connections 2024-03-18T22:49:42.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.616+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/109989811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 0x5647e59102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.616+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/109989811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e590d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.616+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/109989811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e59082f0 0x5647e5908710 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.616+0000 7f7c83fc5740 1 -- 172.21.15.40:0/109989811 >> 172.21.15.40:0/109989811 conn(0x5647e5903860 msgr2=0x5647e5903c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.616+0000 7f7c83fc5740 1 -- 172.21.15.40:0/109989811 shutdown_connections 2024-03-18T22:49:42.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.616+0000 7f7c83fc5740 1 -- 172.21.15.40:0/109989811 wait complete. 2024-03-18T22:49:42.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.616+0000 7f7c83fc5740 1 -- start start 2024-03-18T22:49:42.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.616+0000 7f7c83fc5740 1 Processor -- start 2024-03-18T22:49:42.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c83fc5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e59082f0 0x5647e5912590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c83fc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e5912ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c83fc5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 0x5647e5919b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c83fc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5647e5910bc0 con 0x5647e5908ce0 2024-03-18T22:49:42.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c83fc5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5647e5828310 con 0x5647e59082f0 2024-03-18T22:49:42.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c83fc5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5647e5910d40 con 0x5647e590de50 2024-03-18T22:49:42.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c83fc5740 10 client.0 _ll_register_callbacks cb 0x5647e581c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:42.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c7a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e5912ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c7a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e5912ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33438/0 (socket says 172.21.15.40:33438) 2024-03-18T22:49:42.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c7a7fc640 1 -- 172.21.15.40:0/97053116 learned_addr learned my addr 172.21.15.40:0/97053116 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c7b7fe640 1 --2- 172.21.15.40:0/97053116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 0x5647e5919b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c7affd640 1 --2- 172.21.15.40:0/97053116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e59082f0 0x5647e5912590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c7a7fc640 1 -- 172.21.15.40:0/97053116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 msgr2=0x5647e5919b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c7a7fc640 1 --2- 172.21.15.40:0/97053116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 0x5647e5919b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c7a7fc640 1 -- 172.21.15.40:0/97053116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e59082f0 msgr2=0x5647e5912590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c7a7fc640 1 --2- 172.21.15.40:0/97053116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e59082f0 0x5647e5912590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c7a7fc640 1 -- 172.21.15.40:0/97053116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647e55803f0 con 0x5647e5908ce0 2024-03-18T22:49:42.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c7a7fc640 1 --2- 172.21.15.40:0/97053116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e5912ad0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7c70012d20 tx=0x7f7c7000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2986374a2f178748 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:42.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c5eff5640 1 -- 172.21.15.40:0/97053116 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c70004050 con 0x5647e5908ce0 2024-03-18T22:49:42.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c5eff5640 1 -- 172.21.15.40:0/97053116 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7c700041f0 con 0x5647e5908ce0 2024-03-18T22:49:42.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.617+0000 7f7c83fc5740 1 -- 172.21.15.40:0/97053116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c6c0c1530 con 0x5647e5908ce0 2024-03-18T22:49:42.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.618+0000 7f7c5eff5640 1 -- 172.21.15.40:0/97053116 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f7c70025050 con 0x5647e5908ce0 2024-03-18T22:49:42.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.618+0000 7f7c83fc5740 10 client.17394 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:42.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.618+0000 7f7c83fc5740 1 -- 172.21.15.40:0/97053116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5647e564a7d0 con 0x5647e5908ce0 2024-03-18T22:49:42.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.618+0000 7f7c5eff5640 1 -- 172.21.15.40:0/97053116 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c6c0c1530 con 0x5647e5908ce0 2024-03-18T22:49:42.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.618+0000 7f7c5eff5640 1 -- 172.21.15.40:0/97053116 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f7c70010430 con 0x5647e5908ce0 2024-03-18T22:49:42.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.618+0000 7f7c83fc5740 10 client.17394 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:42.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.618+0000 7f7c83fc5740 10 client.17394 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:49:42.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.618+0000 7f7c83fc5740 20 client.17394 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:42.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.618+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/97053116 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5647e591e520 0x5647e59209c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.618+0000 7f7c83fc5740 4 client.17394 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:42.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.618+0000 7f7c83fc5740 1 -- 172.21.15.40:0/97053116 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647e58ca3e0 con 0x5647e591e520 2024-03-18T22:49:42.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.618+0000 7f7c7affd640 1 --2- 172.21.15.40:0/97053116 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5647e591e520 0x5647e59209c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.619+0000 7f7c7affd640 1 --2- 172.21.15.40:0/97053116 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5647e591e520 0x5647e59209c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=62f81ae7117da989 server_cookie=b372f8f134a24f92 in_seq=0 out_seq=0 2024-03-18T22:49:42.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.619+0000 7f7c5eff5640 10 client.17394 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:49:42.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.619+0000 7f7c5eff5640 1 -- 172.21.15.40:0/97053116 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5647e58ca3e0 con 0x5647e591e520 2024-03-18T22:49:42.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.619+0000 7f7c5eff5640 10 client.17394 handle_command_reply: tid=0 2024-03-18T22:49:42.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.619+0000 7f7c83fc5740 1 client.17394 shutdown 2024-03-18T22:49:42.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.619+0000 7f7c83fc5740 1 -- 172.21.15.40:0/97053116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 msgr2=0x5647e5912ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.619+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/97053116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e5912ad0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7c70012d20 tx=0x7f7c7000f790 comp rx=0 tx=0).stop 2024-03-18T22:49:42.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.620+0000 7f7c83fc5740 1 -- 172.21.15.40:0/97053116 shutdown_connections 2024-03-18T22:49:42.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.620+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/97053116 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5647e591e520 0x5647e59209c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.620+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/97053116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 0x5647e5919b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.620+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/97053116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e5912ad0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.620+0000 7f7c5eff5640 0 client.17394 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:49:42.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.620+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/97053116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e59082f0 0x5647e5912590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.620+0000 7f7c83fc5740 1 -- 172.21.15.40:0/97053116 >> 172.21.15.40:0/97053116 conn(0x5647e5903860 msgr2=0x5647e5902ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.620+0000 7f7c83fc5740 1 -- 172.21.15.40:0/97053116 shutdown_connections 2024-03-18T22:49:42.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.620+0000 7f7c83fc5740 1 -- 172.21.15.40:0/97053116 wait complete. 2024-03-18T22:49:42.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.620+0000 7f7c83fc5740 20 client.17394 trim_cache size 0 max 16384 2024-03-18T22:49:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.641+0000 7f7c83fc5740 1 Processor -- start 2024-03-18T22:49:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.641+0000 7f7c83fc5740 1 -- start start 2024-03-18T22:49:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.641+0000 7f7c83fc5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e59082f0 0x5647e5902310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.641+0000 7f7c83fc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e5919ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.641+0000 7f7c83fc5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e590de50 0x5647e5902850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.641+0000 7f7c83fc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c1ac700 con 0x5647e5908ce0 2024-03-18T22:49:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.641+0000 7f7c83fc5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c1ae300 con 0x5647e590de50 2024-03-18T22:49:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.641+0000 7f7c83fc5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c1acdd0 con 0x5647e59082f0 2024-03-18T22:49:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.641+0000 7f7c7a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e5919ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.641+0000 7f7c7a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e5919ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33440/0 (socket says 172.21.15.40:33440) 2024-03-18T22:49:42.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.641+0000 7f7c7a7fc640 1 -- 172.21.15.40:0/2592410756 learned_addr learned my addr 172.21.15.40:0/2592410756 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.641+0000 7f7c7affd640 1 --2- 172.21.15.40:0/2592410756 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e59082f0 0x5647e5902310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.641+0000 7f7c7b7fe640 1 --2- 172.21.15.40:0/2592410756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e590de50 0x5647e5902850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.642+0000 7f7c7a7fc640 1 -- 172.21.15.40:0/2592410756 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e59082f0 msgr2=0x5647e5902310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.642+0000 7f7c7a7fc640 1 --2- 172.21.15.40:0/2592410756 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e59082f0 0x5647e5902310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.642+0000 7f7c7a7fc640 1 -- 172.21.15.40:0/2592410756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e590de50 msgr2=0x5647e5902850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.642+0000 7f7c7a7fc640 1 --2- 172.21.15.40:0/2592410756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e590de50 0x5647e5902850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.642+0000 7f7c7a7fc640 1 -- 172.21.15.40:0/2592410756 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647e5907f30 con 0x5647e5908ce0 2024-03-18T22:49:42.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.642+0000 7f7c7affd640 1 --2- 172.21.15.40:0/2592410756 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e59082f0 0x5647e5902310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.642+0000 7f7c7a7fc640 1 --2- 172.21.15.40:0/2592410756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e5919ad0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7c70010a40 tx=0x7f7c70010cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84caa5f015801b21 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.642+0000 7f7c60ff9640 1 -- 172.21.15.40:0/2592410756 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7000fe80 con 0x5647e5908ce0 2024-03-18T22:49:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.642+0000 7f7c60ff9640 1 -- 172.21.15.40:0/2592410756 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c70005560 con 0x5647e5908ce0 2024-03-18T22:49:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.642+0000 7f7c60ff9640 1 -- 172.21.15.40:0/2592410756 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c700036e0 con 0x5647e5908ce0 2024-03-18T22:49:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.643+0000 7f7c83fc5740 1 -- 172.21.15.40:0/2592410756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 msgr2=0x5647e5919ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.643+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/2592410756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e5919ad0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7c70010a40 tx=0x7f7c70010cd0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.643+0000 7f7c83fc5740 1 -- 172.21.15.40:0/2592410756 shutdown_connections 2024-03-18T22:49:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.643+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/2592410756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e590de50 0x5647e5902850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.643+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/2592410756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e5908ce0 0x5647e5919ad0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.643+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/2592410756 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e59082f0 0x5647e5902310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.643+0000 7f7c83fc5740 1 -- 172.21.15.40:0/2592410756 >> 172.21.15.40:0/2592410756 conn(0x5647e5903860 msgr2=0x5647e5903c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.643+0000 7f7c83fc5740 1 -- 172.21.15.40:0/2592410756 shutdown_connections 2024-03-18T22:49:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.643+0000 7f7c83fc5740 1 -- 172.21.15.40:0/2592410756 wait complete. 2024-03-18T22:49:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.643+0000 7f7c83fc5740 1 -- start start 2024-03-18T22:49:42.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.643+0000 7f7c83fc5740 1 Processor -- start 2024-03-18T22:49:42.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c83fc5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e59082f0 0x5647e5917250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c83fc5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e5908ce0 0x5647e5917790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c83fc5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 0x5647e59187a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c83fc5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c1adc30 con 0x5647e59082f0 2024-03-18T22:49:42.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c83fc5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c1ad4a0 con 0x5647e5908ce0 2024-03-18T22:49:42.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c83fc5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7c1aa6e0 con 0x5647e590de50 2024-03-18T22:49:42.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c83fc5740 10 client.0 _ll_register_callbacks cb 0x7f7c70009cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:42.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c7affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e59082f0 0x5647e5917250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c7affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e59082f0 0x5647e5917250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33454/0 (socket says 172.21.15.40:33454) 2024-03-18T22:49:42.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c7affd640 1 -- 172.21.15.40:0/3694111112 learned_addr learned my addr 172.21.15.40:0/3694111112 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c7b7fe640 1 --2- 172.21.15.40:0/3694111112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 0x5647e59187a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c7a7fc640 1 --2- 172.21.15.40:0/3694111112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e5908ce0 0x5647e5917790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c7affd640 1 -- 172.21.15.40:0/3694111112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 msgr2=0x5647e59187a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c7affd640 1 --2- 172.21.15.40:0/3694111112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 0x5647e59187a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c7affd640 1 -- 172.21.15.40:0/3694111112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e5908ce0 msgr2=0x5647e5917790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c7affd640 1 --2- 172.21.15.40:0/3694111112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e5908ce0 0x5647e5917790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c7affd640 1 -- 172.21.15.40:0/3694111112 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647e5918d90 con 0x5647e59082f0 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c7affd640 1 --2- 172.21.15.40:0/3694111112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e59082f0 0x5647e5917250 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0bd0e0 tx=0x7f7c6c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad01190f923068a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c5eff5640 1 -- 172.21.15.40:0/3694111112 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0b9b30 con 0x5647e59082f0 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c5eff5640 1 -- 172.21.15.40:0/3694111112 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7c6c003c80 con 0x5647e59082f0 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.644+0000 7f7c83fc5740 1 -- 172.21.15.40:0/3694111112 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c70005560 con 0x5647e59082f0 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.645+0000 7f7c5eff5640 1 -- 172.21.15.40:0/3694111112 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f7c6c0bb520 con 0x5647e59082f0 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.645+0000 7f7c83fc5740 10 client.17400 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.645+0000 7f7c83fc5740 1 -- 172.21.15.40:0/3694111112 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5647e59849c0 con 0x5647e59082f0 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.645+0000 7f7c5eff5640 1 -- 172.21.15.40:0/3694111112 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c70005560 con 0x5647e59082f0 2024-03-18T22:49:42.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.645+0000 7f7c5eff5640 1 -- 172.21.15.40:0/3694111112 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f7c6c0c1150 con 0x5647e59082f0 2024-03-18T22:49:42.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.645+0000 7f7c83fc5740 10 client.17400 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:42.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.645+0000 7f7c83fc5740 10 client.17400 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:49:42.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.645+0000 7f7c83fc5740 20 client.17400 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:42.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.645+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/3694111112 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5647e5985280 0x5647e59876a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.645+0000 7f7c83fc5740 4 client.17400 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:49:42.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.645+0000 7f7c83fc5740 1 -- 172.21.15.40:0/3694111112 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5647e58799f0 con 0x5647e5985280 2024-03-18T22:49:42.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.645+0000 7f7c7a7fc640 1 --2- 172.21.15.40:0/3694111112 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5647e5985280 0x5647e59876a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c7a7fc640 1 --2- 172.21.15.40:0/3694111112 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5647e5985280 0x5647e59876a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=326b72008e65eca9 server_cookie=2ad03583eb92b35f in_seq=0 out_seq=0 2024-03-18T22:49:42.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c5eff5640 10 client.17400 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:49:42.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c5eff5640 1 -- 172.21.15.40:0/3694111112 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5647e58799f0 con 0x5647e5985280 2024-03-18T22:49:42.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c5eff5640 10 client.17400 handle_command_reply: tid=0 2024-03-18T22:49:42.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c83fc5740 1 client.17400 shutdown 2024-03-18T22:49:42.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c83fc5740 1 -- 172.21.15.40:0/3694111112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e59082f0 msgr2=0x5647e5917250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/3694111112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e59082f0 0x5647e5917250 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0bd0e0 tx=0x7f7c6c005fd0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c83fc5740 1 -- 172.21.15.40:0/3694111112 shutdown_connections 2024-03-18T22:49:42.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/3694111112 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5647e5985280 0x5647e59876a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/3694111112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647e590de50 0x5647e59187a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/3694111112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647e5908ce0 0x5647e5917790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c83fc5740 1 --2- 172.21.15.40:0/3694111112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647e59082f0 0x5647e5917250 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c5eff5640 0 client.17400 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:49:42.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c83fc5740 1 -- 172.21.15.40:0/3694111112 >> 172.21.15.40:0/3694111112 conn(0x5647e5903860 msgr2=0x5647e5906ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c83fc5740 1 -- 172.21.15.40:0/3694111112 shutdown_connections 2024-03-18T22:49:42.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c83fc5740 1 -- 172.21.15.40:0/3694111112 wait complete. 2024-03-18T22:49:42.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.646+0000 7f7c83fc5740 20 client.17400 trim_cache size 0 max 16384 2024-03-18T22:49:42.658 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:49:42.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.647+0000 7f7c813e1640 1 -- 172.21.15.40:0/2069130001 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7c640a2800 msgr2=0x7f7c640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.647+0000 7f7c813e1640 1 --2- 172.21.15.40:0/2069130001 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7c640a2800 0x7f7c640a4cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7c74005fd0 tx=0x7f7c74009f90 comp rx=0 tx=0).stop 2024-03-18T22:49:42.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.647+0000 7f7c813e1640 1 -- 172.21.15.40:0/2069130001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c7c074220 msgr2=0x7f7c7c199cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.647+0000 7f7c813e1640 1 --2- 172.21.15.40:0/2069130001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c7c074220 0x7f7c7c199cc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c000c00 tx=0x7f7c6c004600 comp rx=0 tx=0).stop 2024-03-18T22:49:42.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.647+0000 7f7c813e1640 1 -- 172.21.15.40:0/2069130001 shutdown_connections 2024-03-18T22:49:42.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.647+0000 7f7c813e1640 1 --2- 172.21.15.40:0/2069130001 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7c640a2800 0x7f7c640a4cc0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.647+0000 7f7c813e1640 1 --2- 172.21.15.40:0/2069130001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7c7c109740 0x7f7c7c1a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.647+0000 7f7c813e1640 1 --2- 172.21.15.40:0/2069130001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c7c075800 0x7f7c7c19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.647+0000 7f7c813e1640 1 --2- 172.21.15.40:0/2069130001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c7c074220 0x7f7c7c199cc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.647+0000 7f7c813e1640 1 -- 172.21.15.40:0/2069130001 >> 172.21.15.40:0/2069130001 conn(0x7f7c7c0fce40 msgr2=0x7f7c7c1056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.647+0000 7f7c813e1640 1 -- 172.21.15.40:0/2069130001 shutdown_connections 2024-03-18T22:49:42.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.647+0000 7f7c813e1640 1 -- 172.21.15.40:0/2069130001 wait complete. 2024-03-18T22:49:42.660 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:49:42.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.774+0000 7f0828c9a640 1 -- 172.21.15.40:0/1068442768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0824100720 msgr2=0x7f0824100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.774+0000 7f0828c9a640 1 --2- 172.21.15.40:0/1068442768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0824100720 0x7f0824100b40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0814009860 tx=0x7f081402bb70 comp rx=0 tx=0).stop 2024-03-18T22:49:42.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.774+0000 7f0828c9a640 1 -- 172.21.15.40:0/1068442768 shutdown_connections 2024-03-18T22:49:42.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.774+0000 7f0828c9a640 1 --2- 172.21.15.40:0/1068442768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08241022d0 0x7f0824108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.774+0000 7f0828c9a640 1 --2- 172.21.15.40:0/1068442768 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0824101910 0x7f0824101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.774+0000 7f0828c9a640 1 --2- 172.21.15.40:0/1068442768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0824100720 0x7f0824100b40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.774+0000 7f0828c9a640 1 -- 172.21.15.40:0/1068442768 >> 172.21.15.40:0/1068442768 conn(0x7f08240fbc90 msgr2=0x7f08240fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.775+0000 7f0828c9a640 1 -- 172.21.15.40:0/1068442768 shutdown_connections 2024-03-18T22:49:42.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.775+0000 7f0828c9a640 1 -- 172.21.15.40:0/1068442768 wait complete. 2024-03-18T22:49:42.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.775+0000 7f0828c9a640 1 Processor -- start 2024-03-18T22:49:42.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.775+0000 7f0828c9a640 1 -- start start 2024-03-18T22:49:42.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f0828c9a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0824100720 0x7f0824197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f0828c9a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0824101910 0x7f08241980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f0828c9a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08241022d0 0x7f082419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f0828c9a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f082410c220 con 0x7f0824100720 2024-03-18T22:49:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f0828c9a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f082410c0a0 con 0x7f0824101910 2024-03-18T22:49:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f0828c9a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f082410c3a0 con 0x7f08241022d0 2024-03-18T22:49:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f082259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0824100720 0x7f0824197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f082259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0824100720 0x7f0824197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33482/0 (socket says 172.21.15.40:33482) 2024-03-18T22:49:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f082259c640 1 -- 172.21.15.40:0/442355471 learned_addr learned my addr 172.21.15.40:0/442355471 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f0822d9d640 1 --2- 172.21.15.40:0/442355471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08241022d0 0x7f082419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f0821d9b640 1 --2- 172.21.15.40:0/442355471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0824101910 0x7f08241980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f082259c640 1 -- 172.21.15.40:0/442355471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08241022d0 msgr2=0x7f082419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f082259c640 1 --2- 172.21.15.40:0/442355471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08241022d0 0x7f082419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f082259c640 1 -- 172.21.15.40:0/442355471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0824101910 msgr2=0x7f08241980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f082259c640 1 --2- 172.21.15.40:0/442355471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0824101910 0x7f08241980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f082259c640 1 -- 172.21.15.40:0/442355471 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f082419f6f0 con 0x7f0824100720 2024-03-18T22:49:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f0822d9d640 1 --2- 172.21.15.40:0/442355471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08241022d0 0x7f082419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f082259c640 1 --2- 172.21.15.40:0/442355471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0824100720 0x7f0824197b70 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f0814005940 tx=0x7f08140042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.776+0000 7f08137fe640 1 -- 172.21.15.40:0/442355471 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0814041020 con 0x7f0824100720 2024-03-18T22:49:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.777+0000 7f0828c9a640 1 -- 172.21.15.40:0/442355471 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f082419f980 con 0x7f0824100720 2024-03-18T22:49:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.777+0000 7f08137fe640 1 -- 172.21.15.40:0/442355471 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08140035f0 con 0x7f0824100720 2024-03-18T22:49:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.777+0000 7f08137fe640 1 -- 172.21.15.40:0/442355471 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0814039040 con 0x7f0824100720 2024-03-18T22:49:42.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.777+0000 7f0828c9a640 1 -- 172.21.15.40:0/442355471 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f082419fec0 con 0x7f0824100720 2024-03-18T22:49:42.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.779+0000 7f082b963740 1 Processor -- start 2024-03-18T22:49:42.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f082b963740 1 -- start start 2024-03-18T22:49:42.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f08137fe640 1 -- 172.21.15.40:0/442355471 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0814044020 con 0x7f0824100720 2024-03-18T22:49:42.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f082b963740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651268652f0 0x565126865710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f082b963740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565126865ce0 0x56512686a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f082b963740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56512686ae50 0x56512686d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f082b963740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5651267c94b0 con 0x56512686ae50 2024-03-18T22:49:42.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f082b963740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x565126771d70 con 0x565126865ce0 2024-03-18T22:49:42.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f082b963740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5651267c5550 con 0x5651268652f0 2024-03-18T22:49:42.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f0821d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56512686ae50 0x56512686d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651268652f0 0x565126865710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f0821d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56512686ae50 0x56512686d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33486/0 (socket says 172.21.15.40:33486) 2024-03-18T22:49:42.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f0821d9b640 1 -- 172.21.15.40:0/3709995275 learned_addr learned my addr 172.21.15.40:0/3709995275 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f082259c640 1 --2- 172.21.15.40:0/3709995275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565126865ce0 0x56512686a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f0822d9d640 1 -- 172.21.15.40:0/3709995275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565126865ce0 msgr2=0x56512686a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f0822d9d640 1 --2- 172.21.15.40:0/3709995275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565126865ce0 0x56512686a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f0822d9d640 1 -- 172.21.15.40:0/3709995275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56512686ae50 msgr2=0x56512686d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f0822d9d640 1 --2- 172.21.15.40:0/3709995275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56512686ae50 0x56512686d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.780+0000 7f0822d9d640 1 -- 172.21.15.40:0/3709995275 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565126794210 con 0x5651268652f0 2024-03-18T22:49:42.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.781+0000 7f08137fe640 1 --2- 172.21.15.40:0/442355471 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f080c0a6c10 0x7f080c0a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.781+0000 7f082259c640 1 --2- 172.21.15.40:0/442355471 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f080c0a6c10 0x7f080c0a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.781+0000 7f0822d9d640 1 --2- 172.21.15.40:0/3709995275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651268652f0 0x565126865710 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f081c009fc0 tx=0x7f081c00ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=535b7da569bf5e71 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.781+0000 7f08137fe640 1 -- 172.21.15.40:0/442355471 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f08140f07c0 con 0x7f0824100720 2024-03-18T22:49:42.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.781+0000 7f0810ff9640 1 -- 172.21.15.40:0/3709995275 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081c01d070 con 0x5651268652f0 2024-03-18T22:49:42.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.781+0000 7f0810ff9640 1 -- 172.21.15.40:0/3709995275 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f081c003e20 con 0x5651268652f0 2024-03-18T22:49:42.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.781+0000 7f0810ff9640 1 -- 172.21.15.40:0/3709995275 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081c004680 con 0x5651268652f0 2024-03-18T22:49:42.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.782+0000 7f082b963740 1 -- 172.21.15.40:0/3709995275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651268652f0 msgr2=0x565126865710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.782+0000 7f082b963740 1 --2- 172.21.15.40:0/3709995275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651268652f0 0x565126865710 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f081c009fc0 tx=0x7f081c00ec30 comp rx=0 tx=0).stop 2024-03-18T22:49:42.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.782+0000 7f082259c640 1 --2- 172.21.15.40:0/442355471 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f080c0a6c10 0x7f080c0a90d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f08140ee240 tx=0x7f08140f6550 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.782+0000 7f082b963740 1 -- 172.21.15.40:0/3709995275 shutdown_connections 2024-03-18T22:49:42.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.782+0000 7f082b963740 1 --2- 172.21.15.40:0/3709995275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56512686ae50 0x56512686d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.782+0000 7f082b963740 1 --2- 172.21.15.40:0/3709995275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565126865ce0 0x56512686a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.782+0000 7f082b963740 1 --2- 172.21.15.40:0/3709995275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651268652f0 0x565126865710 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.782+0000 7f082b963740 1 -- 172.21.15.40:0/3709995275 >> 172.21.15.40:0/3709995275 conn(0x565126860860 msgr2=0x565126860c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.782+0000 7f082b963740 1 -- 172.21.15.40:0/3709995275 shutdown_connections 2024-03-18T22:49:42.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.782+0000 7f082b963740 1 -- 172.21.15.40:0/3709995275 wait complete. 2024-03-18T22:49:42.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.782+0000 7f082b963740 1 -- start start 2024-03-18T22:49:42.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.782+0000 7f082b963740 1 Processor -- start 2024-03-18T22:49:42.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.782+0000 7f082b963740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651268652f0 0x56512686f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f082b963740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565126865ce0 0x56512686fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f082b963740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56512686ae50 0x565126876da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f082b963740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56512686de00 con 0x565126865ce0 2024-03-18T22:49:42.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f082b963740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x565126785310 con 0x56512686ae50 2024-03-18T22:49:42.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f082b963740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56512686df80 con 0x5651268652f0 2024-03-18T22:49:42.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f082b963740 10 client.0 _ll_register_callbacks cb 0x5651267792d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:42.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0821d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565126865ce0 0x56512686fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651268652f0 0x56512686f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651268652f0 0x56512686f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:60592/0 (socket says 172.21.15.40:60592) 2024-03-18T22:49:42.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0821d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565126865ce0 0x56512686fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33492/0 (socket says 172.21.15.40:33492) 2024-03-18T22:49:42.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0822d9d640 1 -- 172.21.15.40:0/3954827234 learned_addr learned my addr 172.21.15.40:0/3954827234 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0822d9d640 1 --2- 172.21.15.40:0/3954827234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56512686ae50 0x565126876da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0822d9d640 1 -- 172.21.15.40:0/3954827234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56512686ae50 msgr2=0x565126876da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0822d9d640 1 --2- 172.21.15.40:0/3954827234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56512686ae50 0x565126876da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0822d9d640 1 -- 172.21.15.40:0/3954827234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565126865ce0 msgr2=0x56512686fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0822d9d640 1 --2- 172.21.15.40:0/3954827234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565126865ce0 0x56512686fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0822d9d640 1 -- 172.21.15.40:0/3954827234 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651264dd3f0 con 0x5651268652f0 2024-03-18T22:49:42.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0821d9b640 1 --2- 172.21.15.40:0/3954827234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565126865ce0 0x56512686fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0822d9d640 1 --2- 172.21.15.40:0/3954827234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651268652f0 0x56512686f820 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f081c003b80 tx=0x7f081c010040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60a1fa3851dea78d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0809ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:49:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0809ffb640 1 -- 172.21.15.40:0/3954827234 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081c01d030 con 0x5651268652f0 2024-03-18T22:49:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f0809ffb640 1 -- 172.21.15.40:0/3954827234 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f081c00cc80 con 0x5651268652f0 2024-03-18T22:49:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.783+0000 7f082b963740 1 -- 172.21.15.40:0/3954827234 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f081c003e20 con 0x5651268652f0 2024-03-18T22:49:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.784+0000 7f0809ffb640 1 -- 172.21.15.40:0/3954827234 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f081c027400 con 0x5651268652f0 2024-03-18T22:49:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.784+0000 7f082b963740 10 client.27220 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.784+0000 7f082b963740 1 -- 172.21.15.40:0/3954827234 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5651265a77d0 con 0x5651268652f0 2024-03-18T22:49:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.784+0000 7f0809ffb640 1 -- 172.21.15.40:0/3954827234 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f081c003e20 con 0x5651268652f0 2024-03-18T22:49:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.784+0000 7f0809ffb640 1 -- 172.21.15.40:0/3954827234 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f081c00b040 con 0x5651268652f0 2024-03-18T22:49:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.784+0000 7f082b963740 10 client.27220 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.784+0000 7f082b963740 10 client.27220 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.784+0000 7f082b963740 20 client.27220 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.784+0000 7f082b963740 1 --2- 172.21.15.40:0/3954827234 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56512687b290 0x56512687d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.784+0000 7f082b963740 4 client.27220 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:42.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.784+0000 7f082b963740 1 -- 172.21.15.40:0/3954827234 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651268273e0 con 0x56512687b290 2024-03-18T22:49:42.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.784+0000 7f0821d9b640 1 --2- 172.21.15.40:0/3954827234 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56512687b290 0x56512687d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.785+0000 7f0809ffb640 10 client.27220 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:42.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.785+0000 7f0821d9b640 1 --2- 172.21.15.40:0/3954827234 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56512687b290 0x56512687d680 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b210a069faf8d0a server_cookie=1b2b678090830a2a in_seq=0 out_seq=0 2024-03-18T22:49:42.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.785+0000 7f0809ffb640 1 -- 172.21.15.40:0/3954827234 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5651268273e0 con 0x56512687b290 2024-03-18T22:49:42.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.785+0000 7f0809ffb640 10 client.27220 handle_command_reply: tid=0 2024-03-18T22:49:42.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.785+0000 7f082b963740 1 client.27220 shutdown 2024-03-18T22:49:42.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.785+0000 7f082b963740 1 -- 172.21.15.40:0/3954827234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651268652f0 msgr2=0x56512686f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.785+0000 7f082b963740 1 --2- 172.21.15.40:0/3954827234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651268652f0 0x56512686f820 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f081c003b80 tx=0x7f081c010040 comp rx=0 tx=0).stop 2024-03-18T22:49:42.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.785+0000 7f082b963740 1 -- 172.21.15.40:0/3954827234 shutdown_connections 2024-03-18T22:49:42.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.785+0000 7f082b963740 1 --2- 172.21.15.40:0/3954827234 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56512687b290 0x56512687d680 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.785+0000 7f082b963740 1 --2- 172.21.15.40:0/3954827234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56512686ae50 0x565126876da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.785+0000 7f082b963740 1 --2- 172.21.15.40:0/3954827234 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565126865ce0 0x56512686fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.785+0000 7f0809ffb640 0 client.27220 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:42.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.785+0000 7f082b963740 1 --2- 172.21.15.40:0/3954827234 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5651268652f0 0x56512686f820 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.786+0000 7f082b963740 1 -- 172.21.15.40:0/3954827234 >> 172.21.15.40:0/3954827234 conn(0x565126860860 msgr2=0x56512685fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.786+0000 7f082b963740 1 -- 172.21.15.40:0/3954827234 shutdown_connections 2024-03-18T22:49:42.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.786+0000 7f082b963740 1 -- 172.21.15.40:0/3954827234 wait complete. 2024-03-18T22:49:42.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.786+0000 7f082b963740 20 client.27220 trim_cache size 0 max 16384 2024-03-18T22:49:42.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.800+0000 7f082b963740 1 Processor -- start 2024-03-18T22:49:42.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.800+0000 7f082b963740 1 -- start start 2024-03-18T22:49:42.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.800+0000 7f082b963740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651268652f0 0x56512686fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.800+0000 7f082b963740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565126865ce0 0x565126876d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.800+0000 7f082b963740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56512686ae50 0x56512685f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.800+0000 7f082b963740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f08241a8380 con 0x5651268652f0 2024-03-18T22:49:42.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.800+0000 7f082b963740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f08241a9f80 con 0x56512686ae50 2024-03-18T22:49:42.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.800+0000 7f082b963740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f08241a8a50 con 0x565126865ce0 2024-03-18T22:49:42.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.800+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651268652f0 0x56512686fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.800+0000 7f0821d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565126865ce0 0x565126876d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.800+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651268652f0 0x56512686fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33508/0 (socket says 172.21.15.40:33508) 2024-03-18T22:49:42.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.800+0000 7f0822d9d640 1 -- 172.21.15.40:0/2229479371 learned_addr learned my addr 172.21.15.40:0/2229479371 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.801+0000 7f0822d9d640 1 --2- 172.21.15.40:0/2229479371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56512686ae50 0x56512685f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.801+0000 7f0821d9b640 1 -- 172.21.15.40:0/2229479371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56512686ae50 msgr2=0x56512685f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.801+0000 7f0821d9b640 1 --2- 172.21.15.40:0/2229479371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56512686ae50 0x56512685f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.801+0000 7f0821d9b640 1 -- 172.21.15.40:0/2229479371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651268652f0 msgr2=0x56512686fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.801+0000 7f0821d9b640 1 --2- 172.21.15.40:0/2229479371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651268652f0 0x56512686fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.801+0000 7f0821d9b640 1 -- 172.21.15.40:0/2229479371 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565126864f30 con 0x565126865ce0 2024-03-18T22:49:42.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.801+0000 7f0822d9d640 1 --2- 172.21.15.40:0/2229479371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651268652f0 0x56512686fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.801+0000 7f0821d9b640 1 --2- 172.21.15.40:0/2229479371 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565126865ce0 0x565126876d60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x56512687de30 tx=0x7f0818005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f0153c78443ac66 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.801+0000 7f080bfff640 1 -- 172.21.15.40:0/2229479371 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0818005220 con 0x565126865ce0 2024-03-18T22:49:42.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.801+0000 7f080bfff640 1 -- 172.21.15.40:0/2229479371 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0818004c50 con 0x565126865ce0 2024-03-18T22:49:42.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.801+0000 7f080bfff640 1 -- 172.21.15.40:0/2229479371 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0818020540 con 0x565126865ce0 2024-03-18T22:49:42.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 -- 172.21.15.40:0/2229479371 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565126865ce0 msgr2=0x565126876d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 --2- 172.21.15.40:0/2229479371 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565126865ce0 0x565126876d60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x56512687de30 tx=0x7f0818005f90 comp rx=0 tx=0).stop 2024-03-18T22:49:42.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 -- 172.21.15.40:0/2229479371 shutdown_connections 2024-03-18T22:49:42.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 --2- 172.21.15.40:0/2229479371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56512686ae50 0x56512685f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 --2- 172.21.15.40:0/2229479371 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565126865ce0 0x565126876d60 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 --2- 172.21.15.40:0/2229479371 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651268652f0 0x56512686fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 -- 172.21.15.40:0/2229479371 >> 172.21.15.40:0/2229479371 conn(0x565126860860 msgr2=0x56512686f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 -- 172.21.15.40:0/2229479371 shutdown_connections 2024-03-18T22:49:42.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 -- 172.21.15.40:0/2229479371 wait complete. 2024-03-18T22:49:42.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 -- start start 2024-03-18T22:49:42.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 Processor -- start 2024-03-18T22:49:42.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651268652f0 0x5651268744b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565126865ce0 0x5651268749f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56512686ae50 0x565126875a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f08241a98b0 con 0x5651268652f0 2024-03-18T22:49:42.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f08241a9120 con 0x565126865ce0 2024-03-18T22:49:42.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f08241a6360 con 0x56512686ae50 2024-03-18T22:49:42.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f082b963740 10 client.0 _ll_register_callbacks cb 0x7f081c01b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:42.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651268652f0 0x5651268744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f0822d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651268652f0 0x5651268744b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33516/0 (socket says 172.21.15.40:33516) 2024-03-18T22:49:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f0821d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565126865ce0 0x5651268749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.802+0000 7f0822d9d640 1 -- 172.21.15.40:0/2019545967 learned_addr learned my addr 172.21.15.40:0/2019545967 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.803+0000 7f0822d9d640 1 --2- 172.21.15.40:0/2019545967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56512686ae50 0x565126875a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.803+0000 7f0822d9d640 1 -- 172.21.15.40:0/2019545967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56512686ae50 msgr2=0x565126875a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.803+0000 7f0822d9d640 1 --2- 172.21.15.40:0/2019545967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56512686ae50 0x565126875a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.803+0000 7f0822d9d640 1 -- 172.21.15.40:0/2019545967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565126865ce0 msgr2=0x5651268749f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.803+0000 7f0822d9d640 1 --2- 172.21.15.40:0/2019545967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565126865ce0 0x5651268749f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.803+0000 7f0822d9d640 1 -- 172.21.15.40:0/2019545967 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565126836880 con 0x5651268652f0 2024-03-18T22:49:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.803+0000 7f0821d9b640 1 --2- 172.21.15.40:0/2019545967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565126865ce0 0x5651268749f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.803+0000 7f0822d9d640 1 --2- 172.21.15.40:0/2019545967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651268652f0 0x5651268744b0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f081c005cc0 tx=0x7f081c005d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5555f897a94b79f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.803+0000 7f0809ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.803+0000 7f0809ffb640 1 -- 172.21.15.40:0/2019545967 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081c01d040 con 0x5651268652f0 2024-03-18T22:49:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.803+0000 7f0809ffb640 1 -- 172.21.15.40:0/2019545967 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f081c00ca80 con 0x5651268652f0 2024-03-18T22:49:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.803+0000 7f082b963740 1 -- 172.21.15.40:0/2019545967 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0818004c50 con 0x5651268652f0 2024-03-18T22:49:42.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.803+0000 7f0809ffb640 1 -- 172.21.15.40:0/2019545967 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f081c045050 con 0x5651268652f0 2024-03-18T22:49:42.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.803+0000 7f082b963740 10 client.17418 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:42.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.803+0000 7f082b963740 1 -- 172.21.15.40:0/2019545967 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5651268e14c0 con 0x5651268652f0 2024-03-18T22:49:42.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.804+0000 7f0809ffb640 1 -- 172.21.15.40:0/2019545967 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0818004c50 con 0x5651268652f0 2024-03-18T22:49:42.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.804+0000 7f0809ffb640 1 -- 172.21.15.40:0/2019545967 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f081c00b040 con 0x5651268652f0 2024-03-18T22:49:42.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.804+0000 7f082b963740 10 client.17418 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:42.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.804+0000 7f082b963740 10 client.17418 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:42.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.804+0000 7f082b963740 20 client.17418 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:42.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.804+0000 7f082b963740 1 --2- 172.21.15.40:0/2019545967 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5651268e1c60 0x5651268e4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:42.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.804+0000 7f082b963740 4 client.17418 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:49:42.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.804+0000 7f082b963740 1 -- 172.21.15.40:0/2019545967 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5651267d69f0 con 0x5651268e1c60 2024-03-18T22:49:42.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.804+0000 7f0821d9b640 1 --2- 172.21.15.40:0/2019545967 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5651268e1c60 0x5651268e4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:42.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f0809ffb640 10 client.17418 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:42.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f0821d9b640 1 --2- 172.21.15.40:0/2019545967 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5651268e1c60 0x5651268e4080 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76ef0e3d35655885 server_cookie=2313f2b758f8752d in_seq=0 out_seq=0 2024-03-18T22:49:42.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f0809ffb640 1 -- 172.21.15.40:0/2019545967 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5651267d69f0 con 0x5651268e1c60 2024-03-18T22:49:42.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f0809ffb640 10 client.17418 handle_command_reply: tid=0 2024-03-18T22:49:42.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f082b963740 1 client.17418 shutdown 2024-03-18T22:49:42.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f082b963740 1 -- 172.21.15.40:0/2019545967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651268652f0 msgr2=0x5651268744b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f082b963740 1 --2- 172.21.15.40:0/2019545967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651268652f0 0x5651268744b0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f081c005cc0 tx=0x7f081c005d10 comp rx=0 tx=0).stop 2024-03-18T22:49:42.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f082b963740 1 -- 172.21.15.40:0/2019545967 shutdown_connections 2024-03-18T22:49:42.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f082b963740 1 --2- 172.21.15.40:0/2019545967 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56512686ae50 0x565126875a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f082b963740 1 --2- 172.21.15.40:0/2019545967 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5651268e1c60 0x5651268e4080 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f082b963740 1 --2- 172.21.15.40:0/2019545967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565126865ce0 0x5651268749f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f0809ffb640 0 client.17418 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:42.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f082b963740 1 --2- 172.21.15.40:0/2019545967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5651268652f0 0x5651268744b0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f082b963740 1 -- 172.21.15.40:0/2019545967 >> 172.21.15.40:0/2019545967 conn(0x565126860860 msgr2=0x565126863ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f082b963740 1 -- 172.21.15.40:0/2019545967 shutdown_connections 2024-03-18T22:49:42.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f082b963740 1 -- 172.21.15.40:0/2019545967 wait complete. 2024-03-18T22:49:42.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.805+0000 7f082b963740 20 client.17418 trim_cache size 0 max 16384 2024-03-18T22:49:42.813 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:49:42.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.806+0000 7f0828c9a640 1 -- 172.21.15.40:0/442355471 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f080c0a6c10 msgr2=0x7f080c0a90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.806+0000 7f0828c9a640 1 --2- 172.21.15.40:0/442355471 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f080c0a6c10 0x7f080c0a90d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f08140ee240 tx=0x7f08140f6550 comp rx=0 tx=0).stop 2024-03-18T22:49:42.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.806+0000 7f0828c9a640 1 -- 172.21.15.40:0/442355471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0824100720 msgr2=0x7f0824197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:42.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.806+0000 7f0828c9a640 1 --2- 172.21.15.40:0/442355471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0824100720 0x7f0824197b70 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f0814005940 tx=0x7f08140042f0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.806+0000 7f0828c9a640 1 -- 172.21.15.40:0/442355471 shutdown_connections 2024-03-18T22:49:42.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.806+0000 7f0828c9a640 1 --2- 172.21.15.40:0/442355471 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f080c0a6c10 0x7f080c0a90d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.806+0000 7f0828c9a640 1 --2- 172.21.15.40:0/442355471 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f08241022d0 0x7f082419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.806+0000 7f0828c9a640 1 --2- 172.21.15.40:0/442355471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0824101910 0x7f08241980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.806+0000 7f0828c9a640 1 --2- 172.21.15.40:0/442355471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0824100720 0x7f0824197b70 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:42.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.806+0000 7f0828c9a640 1 -- 172.21.15.40:0/442355471 >> 172.21.15.40:0/442355471 conn(0x7f08240fbc90 msgr2=0x7f0824068a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:42.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.806+0000 7f0828c9a640 1 -- 172.21.15.40:0/442355471 shutdown_connections 2024-03-18T22:49:42.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:42.806+0000 7f0828c9a640 1 -- 172.21.15.40:0/442355471 wait complete. 2024-03-18T22:49:43.823 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:49:43.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:49:43.823 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:49:43.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.942+0000 7faa37537640 1 -- 172.21.15.40:0/629550165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa30107560 msgr2=0x7faa30109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:43.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.942+0000 7faa37537640 1 --2- 172.21.15.40:0/629550165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa30107560 0x7faa30109950 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7faa28009ca0 tx=0x7faa2802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.943+0000 7faa37537640 1 -- 172.21.15.40:0/629550165 shutdown_connections 2024-03-18T22:49:43.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.943+0000 7faa37537640 1 --2- 172.21.15.40:0/629550165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa30107560 0x7faa30109950 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.943+0000 7faa37537640 1 --2- 172.21.15.40:0/629550165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa30104c30 0x7faa30107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.943+0000 7faa37537640 1 --2- 172.21.15.40:0/629550165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa30102300 0x7faa301046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.943+0000 7faa37537640 1 -- 172.21.15.40:0/629550165 >> 172.21.15.40:0/629550165 conn(0x7faa300fbcb0 msgr2=0x7faa300fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:43.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.943+0000 7faa37537640 1 -- 172.21.15.40:0/629550165 shutdown_connections 2024-03-18T22:49:43.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.943+0000 7faa37537640 1 -- 172.21.15.40:0/629550165 wait complete. 2024-03-18T22:49:43.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.943+0000 7faa37537640 1 Processor -- start 2024-03-18T22:49:43.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.943+0000 7faa37537640 1 -- start start 2024-03-18T22:49:43.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa37537640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa30102300 0x7faa30197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa37537640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa30104c30 0x7faa30198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa37537640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa30107560 0x7faa3019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa37537640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa3010d3c0 con 0x7faa30107560 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa37537640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faa3010d240 con 0x7faa30104c30 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa37537640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7faa3010d540 con 0x7faa30102300 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa35ad4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa30107560 0x7faa3019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa35ad4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa30107560 0x7faa3019f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33538/0 (socket says 172.21.15.40:33538) 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa352d3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa30102300 0x7faa30197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa34ad2640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa30104c30 0x7faa30198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa35ad4640 1 -- 172.21.15.40:0/1040959893 learned_addr learned my addr 172.21.15.40:0/1040959893 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa352d3640 1 -- 172.21.15.40:0/1040959893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa30104c30 msgr2=0x7faa30198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa352d3640 1 --2- 172.21.15.40:0/1040959893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa30104c30 0x7faa30198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa352d3640 1 -- 172.21.15.40:0/1040959893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa30107560 msgr2=0x7faa3019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa352d3640 1 --2- 172.21.15.40:0/1040959893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa30107560 0x7faa3019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa352d3640 1 -- 172.21.15.40:0/1040959893 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa3019f700 con 0x7faa30102300 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.944+0000 7faa35ad4640 1 --2- 172.21.15.40:0/1040959893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa30107560 0x7faa3019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.945+0000 7faa352d3640 1 --2- 172.21.15.40:0/1040959893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa30102300 0x7faa30197b30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7faa2c00e9f0 tx=0x7faa2c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.945+0000 7faa267fc640 1 -- 172.21.15.40:0/1040959893 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c00ccc0 con 0x7faa30102300 2024-03-18T22:49:43.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.945+0000 7faa37537640 1 -- 172.21.15.40:0/1040959893 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa3019f9f0 con 0x7faa30102300 2024-03-18T22:49:43.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.945+0000 7faa267fc640 1 -- 172.21.15.40:0/1040959893 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa2c0035f0 con 0x7faa30102300 2024-03-18T22:49:43.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.945+0000 7faa267fc640 1 -- 172.21.15.40:0/1040959893 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c00ce60 con 0x7faa30102300 2024-03-18T22:49:43.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.945+0000 7faa37537640 1 -- 172.21.15.40:0/1040959893 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa3019ff90 con 0x7faa30102300 2024-03-18T22:49:43.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.948+0000 7faa267fc640 1 -- 172.21.15.40:0/1040959893 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7faa2c0037a0 con 0x7faa30102300 2024-03-18T22:49:43.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.948+0000 7faa3a1f8740 1 Processor -- start 2024-03-18T22:49:43.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.948+0000 7faa3a1f8740 1 -- start start 2024-03-18T22:49:43.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.948+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb13a0 0x559280fb17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.948+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb1d90 0x559280fb6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.948+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb6f00 0x559280fb9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.948+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559280f16160 con 0x559280fb1d90 2024-03-18T22:49:43.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.948+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559280ebee10 con 0x559280fb6f00 2024-03-18T22:49:43.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.948+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559280f12200 con 0x559280fb13a0 2024-03-18T22:49:43.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.948+0000 7faa35ad4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb13a0 0x559280fb17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.948+0000 7faa34ad2640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb6f00 0x559280fb9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.948+0000 7faa35ad4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb13a0 0x559280fb17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:60636/0 (socket says 172.21.15.40:60636) 2024-03-18T22:49:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.948+0000 7faa35ad4640 1 -- 172.21.15.40:0/2227921565 learned_addr learned my addr 172.21.15.40:0/2227921565 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.948+0000 7faa352d3640 1 --2- 172.21.15.40:0/2227921565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb1d90 0x559280fb6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.948+0000 7faa267fc640 1 --2- 172.21.15.40:0/1040959893 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa1c0a2800 0x7faa1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.949+0000 7faa34ad2640 1 -- 172.21.15.40:0/2227921565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb13a0 msgr2=0x559280fb17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.949+0000 7faa34ad2640 1 --2- 172.21.15.40:0/2227921565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb13a0 0x559280fb17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.949+0000 7faa34ad2640 1 -- 172.21.15.40:0/2227921565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb1d90 msgr2=0x559280fb6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.949+0000 7faa34ad2640 1 --2- 172.21.15.40:0/2227921565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb1d90 0x559280fb6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.949+0000 7faa34ad2640 1 -- 172.21.15.40:0/2227921565 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559280cf27e0 con 0x559280fb6f00 2024-03-18T22:49:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.949+0000 7faa35ad4640 1 --2- 172.21.15.40:0/1040959893 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa1c0a2800 0x7faa1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.949+0000 7faa267fc640 1 -- 172.21.15.40:0/1040959893 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7faa2c0cf090 con 0x7faa30102300 2024-03-18T22:49:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.949+0000 7faa34ad2640 1 --2- 172.21.15.40:0/2227921565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb6f00 0x559280fb9390 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faa30069480 tx=0x7faa20009e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5793e0ddfcf8f53 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.949+0000 7faa1b7fe640 1 -- 172.21.15.40:0/2227921565 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20017070 con 0x559280fb6f00 2024-03-18T22:49:43.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.949+0000 7faa1b7fe640 1 -- 172.21.15.40:0/2227921565 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa20013470 con 0x559280fb6f00 2024-03-18T22:49:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.949+0000 7faa1b7fe640 1 -- 172.21.15.40:0/2227921565 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20012630 con 0x559280fb6f00 2024-03-18T22:49:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.950+0000 7faa3a1f8740 1 -- 172.21.15.40:0/2227921565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb6f00 msgr2=0x559280fb9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.950+0000 7faa35ad4640 1 --2- 172.21.15.40:0/1040959893 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa1c0a2800 0x7faa1c0a4cc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x559280fb25f0 tx=0x7faa2802c0f0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.950+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/2227921565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb6f00 0x559280fb9390 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faa30069480 tx=0x7faa20009e00 comp rx=0 tx=0).stop 2024-03-18T22:49:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.950+0000 7faa3a1f8740 1 -- 172.21.15.40:0/2227921565 shutdown_connections 2024-03-18T22:49:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.950+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/2227921565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb6f00 0x559280fb9390 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.950+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/2227921565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb1d90 0x559280fb6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.950+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/2227921565 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb13a0 0x559280fb17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.950+0000 7faa3a1f8740 1 -- 172.21.15.40:0/2227921565 >> 172.21.15.40:0/2227921565 conn(0x559280fac910 msgr2=0x559280facd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.950+0000 7faa3a1f8740 1 -- 172.21.15.40:0/2227921565 shutdown_connections 2024-03-18T22:49:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.950+0000 7faa3a1f8740 1 -- 172.21.15.40:0/2227921565 wait complete. 2024-03-18T22:49:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.950+0000 7faa3a1f8740 1 -- start start 2024-03-18T22:49:43.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.950+0000 7faa3a1f8740 1 Processor -- start 2024-03-18T22:49:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb13a0 0x559280fbb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb1d90 0x559280fbbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb6f00 0x559280fc2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559280fb9eb0 con 0x559280fb6f00 2024-03-18T22:49:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559280ed21a0 con 0x559280fb13a0 2024-03-18T22:49:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559280fba030 con 0x559280fb1d90 2024-03-18T22:49:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa3a1f8740 10 client.0 _ll_register_callbacks cb 0x559280c29c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa35ad4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb6f00 0x559280fc2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa34ad2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb1d90 0x559280fbbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa34ad2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb1d90 0x559280fbbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:60648/0 (socket says 172.21.15.40:60648) 2024-03-18T22:49:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa35ad4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb6f00 0x559280fc2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33562/0 (socket says 172.21.15.40:33562) 2024-03-18T22:49:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa34ad2640 1 -- 172.21.15.40:0/1364573651 learned_addr learned my addr 172.21.15.40:0/1364573651 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:43.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa352d3640 1 --2- 172.21.15.40:0/1364573651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb13a0 0x559280fbb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa34ad2640 1 -- 172.21.15.40:0/1364573651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb13a0 msgr2=0x559280fbb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa34ad2640 1 --2- 172.21.15.40:0/1364573651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb13a0 0x559280fbb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa34ad2640 1 -- 172.21.15.40:0/1364573651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb6f00 msgr2=0x559280fc2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa34ad2640 1 --2- 172.21.15.40:0/1364573651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb6f00 0x559280fc2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa34ad2640 1 -- 172.21.15.40:0/1364573651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559280d52f70 con 0x559280fb1d90 2024-03-18T22:49:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa35ad4640 1 --2- 172.21.15.40:0/1364573651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb6f00 0x559280fc2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa34ad2640 1 --2- 172.21.15.40:0/1364573651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb1d90 0x559280fbbd00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7faa20004810 tx=0x7faa200138d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4837c6c16ad5c91c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.951+0000 7faa18ff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:49:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.952+0000 7faa18ff9640 1 -- 172.21.15.40:0/1364573651 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20017050 con 0x559280fb1d90 2024-03-18T22:49:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.952+0000 7faa18ff9640 1 -- 172.21.15.40:0/1364573651 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7faa2001c860 con 0x559280fb1d90 2024-03-18T22:49:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.952+0000 7faa3a1f8740 1 -- 172.21.15.40:0/1364573651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa20013470 con 0x559280fb1d90 2024-03-18T22:49:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.952+0000 7faa18ff9640 1 -- 172.21.15.40:0/1364573651 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7faa2002f050 con 0x559280fb1d90 2024-03-18T22:49:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.952+0000 7faa3a1f8740 10 client.27238 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:43.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.952+0000 7faa3a1f8740 1 -- 172.21.15.40:0/1364573651 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559280bea170 con 0x559280fb1d90 2024-03-18T22:49:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.952+0000 7faa18ff9640 1 -- 172.21.15.40:0/1364573651 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa20013470 con 0x559280fb1d90 2024-03-18T22:49:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.952+0000 7faa18ff9640 1 -- 172.21.15.40:0/1364573651 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7faa20026720 con 0x559280fb1d90 2024-03-18T22:49:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.952+0000 7faa3a1f8740 10 client.27238 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.952+0000 7faa3a1f8740 10 client.27238 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.952+0000 7faa3a1f8740 20 client.27238 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.952+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/1364573651 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559280fc7230 0x559280fc9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.952+0000 7faa3a1f8740 4 client.27238 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.952+0000 7faa3a1f8740 1 -- 172.21.15.40:0/1364573651 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559280f236a0 con 0x559280fc7230 2024-03-18T22:49:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.952+0000 7faa352d3640 1 --2- 172.21.15.40:0/1364573651 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559280fc7230 0x559280fc9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.953+0000 7faa18ff9640 10 client.27238 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.953+0000 7faa352d3640 1 --2- 172.21.15.40:0/1364573651 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559280fc7230 0x559280fc9660 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ba9ae383337ade server_cookie=7a8b024b8e5d1244 in_seq=0 out_seq=0 2024-03-18T22:49:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.953+0000 7faa18ff9640 1 -- 172.21.15.40:0/1364573651 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559280f236a0 con 0x559280fc7230 2024-03-18T22:49:43.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.953+0000 7faa18ff9640 10 client.27238 handle_command_reply: tid=0 2024-03-18T22:49:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.953+0000 7faa3a1f8740 1 client.27238 shutdown 2024-03-18T22:49:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.954+0000 7faa3a1f8740 1 -- 172.21.15.40:0/1364573651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb1d90 msgr2=0x559280fbbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.954+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/1364573651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb1d90 0x559280fbbd00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7faa20004810 tx=0x7faa200138d0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.954+0000 7faa3a1f8740 1 -- 172.21.15.40:0/1364573651 shutdown_connections 2024-03-18T22:49:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.954+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/1364573651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb6f00 0x559280fc2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.954+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/1364573651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb1d90 0x559280fbbd00 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.954+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/1364573651 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559280fc7230 0x559280fc9660 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.954+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/1364573651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb13a0 0x559280fbb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.954+0000 7faa18ff9640 0 client.27238 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.954+0000 7faa3a1f8740 1 -- 172.21.15.40:0/1364573651 >> 172.21.15.40:0/1364573651 conn(0x559280fac910 msgr2=0x559280fabb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.954+0000 7faa3a1f8740 1 -- 172.21.15.40:0/1364573651 shutdown_connections 2024-03-18T22:49:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.954+0000 7faa3a1f8740 1 -- 172.21.15.40:0/1364573651 wait complete. 2024-03-18T22:49:43.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.954+0000 7faa3a1f8740 20 client.27238 trim_cache size 0 max 16384 2024-03-18T22:49:43.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.982+0000 7faa3a1f8740 1 Processor -- start 2024-03-18T22:49:43.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.982+0000 7faa3a1f8740 1 -- start start 2024-03-18T22:49:43.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.982+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb13a0 0x559280fb31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.982+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb1d90 0x559280fb36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.982+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb6f00 0x559280fc2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.982+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa301aa690 con 0x559280fb13a0 2024-03-18T22:49:43.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.982+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faa301ac290 con 0x559280fb1d90 2024-03-18T22:49:43.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.982+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7faa301aad60 con 0x559280fb6f00 2024-03-18T22:49:43.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.982+0000 7faa352d3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb13a0 0x559280fb31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.983+0000 7faa34ad2640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb1d90 0x559280fb36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.983+0000 7faa35ad4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb6f00 0x559280fc2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.983+0000 7faa352d3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb13a0 0x559280fb31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33568/0 (socket says 172.21.15.40:33568) 2024-03-18T22:49:43.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.983+0000 7faa34ad2640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb1d90 0x559280fb36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:42196/0 (socket says 172.21.15.40:42196) 2024-03-18T22:49:43.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.983+0000 7faa352d3640 1 -- 172.21.15.40:0/1431629865 learned_addr learned my addr 172.21.15.40:0/1431629865 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:43.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.983+0000 7faa352d3640 1 -- 172.21.15.40:0/1431629865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb6f00 msgr2=0x559280fc2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:43.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.983+0000 7faa352d3640 1 --2- 172.21.15.40:0/1431629865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb6f00 0x559280fc2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.983+0000 7faa352d3640 1 -- 172.21.15.40:0/1431629865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb1d90 msgr2=0x559280fb36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:43.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.983+0000 7faa352d3640 1 --2- 172.21.15.40:0/1431629865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb1d90 0x559280fb36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.983+0000 7faa352d3640 1 -- 172.21.15.40:0/1431629865 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559280fa8160 con 0x559280fb13a0 2024-03-18T22:49:43.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.983+0000 7faa34ad2640 1 --2- 172.21.15.40:0/1431629865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb1d90 0x559280fb36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:43.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.983+0000 7faa352d3640 1 --2- 172.21.15.40:0/1431629865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb13a0 0x559280fb31b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7faa2c0cc7a0 tx=0x7faa2c086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c4d1ef40555a83 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:43.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.983+0000 7faa1affd640 1 -- 172.21.15.40:0/1431629865 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c09afa0 con 0x559280fb13a0 2024-03-18T22:49:43.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.983+0000 7faa1affd640 1 -- 172.21.15.40:0/1431629865 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa2c09bd40 con 0x559280fb13a0 2024-03-18T22:49:43.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.983+0000 7faa1affd640 1 -- 172.21.15.40:0/1431629865 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c099940 con 0x559280fb13a0 2024-03-18T22:49:43.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.984+0000 7faa3a1f8740 1 -- 172.21.15.40:0/1431629865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb13a0 msgr2=0x559280fb31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:43.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.984+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/1431629865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb13a0 0x559280fb31b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7faa2c0cc7a0 tx=0x7faa2c086040 comp rx=0 tx=0).stop 2024-03-18T22:49:43.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.984+0000 7faa3a1f8740 1 -- 172.21.15.40:0/1431629865 shutdown_connections 2024-03-18T22:49:43.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.984+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/1431629865 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb6f00 0x559280fc2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.984+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/1431629865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb1d90 0x559280fb36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.984+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/1431629865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb13a0 0x559280fb31b0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.984+0000 7faa3a1f8740 1 -- 172.21.15.40:0/1431629865 >> 172.21.15.40:0/1431629865 conn(0x559280fac910 msgr2=0x559280fbb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:43.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.984+0000 7faa3a1f8740 1 -- 172.21.15.40:0/1431629865 shutdown_connections 2024-03-18T22:49:43.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.984+0000 7faa3a1f8740 1 -- 172.21.15.40:0/1431629865 wait complete. 2024-03-18T22:49:43.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.984+0000 7faa3a1f8740 1 -- start start 2024-03-18T22:49:43.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa3a1f8740 1 Processor -- start 2024-03-18T22:49:43.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb13a0 0x559280fc0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb1d90 0x559280fc07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb6f00 0x559280fc1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faa301abbc0 con 0x559280fb1d90 2024-03-18T22:49:43.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faa301ab430 con 0x559280fb13a0 2024-03-18T22:49:43.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7faa301a8670 con 0x559280fb6f00 2024-03-18T22:49:43.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa3a1f8740 10 client.0 _ll_register_callbacks cb 0x7faa2c097960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:43.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa34ad2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb1d90 0x559280fc07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa35ad4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb6f00 0x559280fc1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa34ad2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb1d90 0x559280fc07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33580/0 (socket says 172.21.15.40:33580) 2024-03-18T22:49:43.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa34ad2640 1 -- 172.21.15.40:0/2401143715 learned_addr learned my addr 172.21.15.40:0/2401143715 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:43.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa352d3640 1 --2- 172.21.15.40:0/2401143715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb13a0 0x559280fc0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa34ad2640 1 -- 172.21.15.40:0/2401143715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb6f00 msgr2=0x559280fc1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:43.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa34ad2640 1 --2- 172.21.15.40:0/2401143715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb6f00 0x559280fc1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa34ad2640 1 -- 172.21.15.40:0/2401143715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb13a0 msgr2=0x559280fc0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:43.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa34ad2640 1 --2- 172.21.15.40:0/2401143715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb13a0 0x559280fc0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa34ad2640 1 -- 172.21.15.40:0/2401143715 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559280fc1d30 con 0x559280fb1d90 2024-03-18T22:49:43.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.985+0000 7faa35ad4640 1 --2- 172.21.15.40:0/2401143715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb6f00 0x559280fc1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:43.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa34ad2640 1 --2- 172.21.15.40:0/2401143715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb1d90 0x559280fc07a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7faa20009150 tx=0x7faa20013b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db817c26b6017eb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:43.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa18ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:43.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa18ff9640 1 -- 172.21.15.40:0/2401143715 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20038060 con 0x559280fb1d90 2024-03-18T22:49:43.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa18ff9640 1 -- 172.21.15.40:0/2401143715 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7faa20026800 con 0x559280fb1d90 2024-03-18T22:49:43.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa3a1f8740 1 -- 172.21.15.40:0/2401143715 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa2c09bd40 con 0x559280fb1d90 2024-03-18T22:49:43.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa18ff9640 1 -- 172.21.15.40:0/2401143715 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7faa2003f050 con 0x559280fb1d90 2024-03-18T22:49:43.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa3a1f8740 10 client.17436 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:43.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa3a1f8740 1 -- 172.21.15.40:0/2401143715 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55928102d560 con 0x559280fb1d90 2024-03-18T22:49:43.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa18ff9640 1 -- 172.21.15.40:0/2401143715 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa2c09bd40 con 0x559280fb1d90 2024-03-18T22:49:43.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa18ff9640 1 -- 172.21.15.40:0/2401143715 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7faa2001cb10 con 0x559280fb1d90 2024-03-18T22:49:43.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa3a1f8740 10 client.17436 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:43.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa3a1f8740 10 client.17436 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:43.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa3a1f8740 20 client.17436 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:43.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/2401143715 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55928102dda0 0x5592810301c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa3a1f8740 4 client.17436 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:49:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.986+0000 7faa3a1f8740 1 -- 172.21.15.40:0/2401143715 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559280f1d9f0 con 0x55928102dda0 2024-03-18T22:49:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.987+0000 7faa352d3640 1 --2- 172.21.15.40:0/2401143715 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55928102dda0 0x5592810301c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.987+0000 7faa18ff9640 10 client.17436 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.987+0000 7faa352d3640 1 --2- 172.21.15.40:0/2401143715 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55928102dda0 0x5592810301c0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d073f8ef9a0e88a server_cookie=83745fb78ca0e3ed in_seq=0 out_seq=0 2024-03-18T22:49:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.988+0000 7faa18ff9640 1 -- 172.21.15.40:0/2401143715 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559280f1d9f0 con 0x55928102dda0 2024-03-18T22:49:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.988+0000 7faa18ff9640 10 client.17436 handle_command_reply: tid=0 2024-03-18T22:49:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.988+0000 7faa3a1f8740 1 client.17436 shutdown 2024-03-18T22:49:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.988+0000 7faa3a1f8740 1 -- 172.21.15.40:0/2401143715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb1d90 msgr2=0x559280fc07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.988+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/2401143715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb1d90 0x559280fc07a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7faa20009150 tx=0x7faa20013b70 comp rx=0 tx=0).stop 2024-03-18T22:49:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.988+0000 7faa3a1f8740 1 -- 172.21.15.40:0/2401143715 shutdown_connections 2024-03-18T22:49:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.988+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/2401143715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559280fb6f00 0x559280fc1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.988+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/2401143715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559280fb1d90 0x559280fc07a0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.988+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/2401143715 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55928102dda0 0x5592810301c0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.988+0000 7faa3a1f8740 1 --2- 172.21.15.40:0/2401143715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559280fb13a0 0x559280fc0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.988+0000 7faa3a1f8740 1 -- 172.21.15.40:0/2401143715 >> 172.21.15.40:0/2401143715 conn(0x559280fac910 msgr2=0x559280fbb760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:43.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.988+0000 7faa18ff9640 0 client.17436 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:43.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.989+0000 7faa3a1f8740 1 -- 172.21.15.40:0/2401143715 shutdown_connections 2024-03-18T22:49:43.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.989+0000 7faa3a1f8740 1 -- 172.21.15.40:0/2401143715 wait complete. 2024-03-18T22:49:43.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.989+0000 7faa3a1f8740 20 client.17436 trim_cache size 0 max 16384 2024-03-18T22:49:43.995 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:49:43.995 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:49:43.995 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "0947e7e2-240a-4a92-ab0d-e8d2d6001560", 2024-03-18T22:49:43.995 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:49:43.995 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:49:43.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.989+0000 7faa37537640 1 -- 172.21.15.40:0/1040959893 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa1c0a2800 msgr2=0x7faa1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:43.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.989+0000 7faa37537640 1 --2- 172.21.15.40:0/1040959893 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa1c0a2800 0x7faa1c0a4cc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x559280fb25f0 tx=0x7faa2802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.989+0000 7faa37537640 1 -- 172.21.15.40:0/1040959893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa30102300 msgr2=0x7faa30197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:43.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.989+0000 7faa37537640 1 --2- 172.21.15.40:0/1040959893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa30102300 0x7faa30197b30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7faa2c00e9f0 tx=0x7faa2c00edf0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.989+0000 7faa37537640 1 -- 172.21.15.40:0/1040959893 shutdown_connections 2024-03-18T22:49:43.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.989+0000 7faa37537640 1 --2- 172.21.15.40:0/1040959893 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7faa1c0a2800 0x7faa1c0a4cc0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.989+0000 7faa37537640 1 --2- 172.21.15.40:0/1040959893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faa30107560 0x7faa3019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.989+0000 7faa37537640 1 --2- 172.21.15.40:0/1040959893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa30104c30 0x7faa30198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.989+0000 7faa37537640 1 --2- 172.21.15.40:0/1040959893 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faa30102300 0x7faa30197b30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:43.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.989+0000 7faa37537640 1 -- 172.21.15.40:0/1040959893 >> 172.21.15.40:0/1040959893 conn(0x7faa300fbcb0 msgr2=0x7faa30102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:43.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.989+0000 7faa37537640 1 -- 172.21.15.40:0/1040959893 shutdown_connections 2024-03-18T22:49:43.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:43.990+0000 7faa37537640 1 -- 172.21.15.40:0/1040959893 wait complete. 2024-03-18T22:49:44.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:49:44.004 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:49:44.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.132+0000 7ff75933b640 1 -- 172.21.15.40:0/675069214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff754100720 msgr2=0x7ff754100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:44.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.132+0000 7ff75933b640 1 --2- 172.21.15.40:0/675069214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff754100720 0x7ff754100b40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff744009900 tx=0x7ff74402bc30 comp rx=0 tx=0).stop 2024-03-18T22:49:44.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.132+0000 7ff75933b640 1 -- 172.21.15.40:0/675069214 shutdown_connections 2024-03-18T22:49:44.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.132+0000 7ff75933b640 1 --2- 172.21.15.40:0/675069214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff7541022d0 0x7ff754108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.132+0000 7ff75933b640 1 --2- 172.21.15.40:0/675069214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff754101910 0x7ff754101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.132+0000 7ff75933b640 1 --2- 172.21.15.40:0/675069214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff754100720 0x7ff754100b40 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.132+0000 7ff75933b640 1 -- 172.21.15.40:0/675069214 >> 172.21.15.40:0/675069214 conn(0x7ff7540fbcb0 msgr2=0x7ff7540fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:44.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.132+0000 7ff75933b640 1 -- 172.21.15.40:0/675069214 shutdown_connections 2024-03-18T22:49:44.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.132+0000 7ff75933b640 1 -- 172.21.15.40:0/675069214 wait complete. 2024-03-18T22:49:44.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff75933b640 1 Processor -- start 2024-03-18T22:49:44.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff75933b640 1 -- start start 2024-03-18T22:49:44.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff75933b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff754100720 0x7ff754197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff75933b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff754101910 0x7ff754197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff75933b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7541022d0 0x7ff75419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff75933b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff75410c220 con 0x7ff7541022d0 2024-03-18T22:49:44.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff75933b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff75410c0a0 con 0x7ff754101910 2024-03-18T22:49:44.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff75933b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff75410c3a0 con 0x7ff754100720 2024-03-18T22:49:44.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff7537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7541022d0 0x7ff75419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff752ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff754100720 0x7ff754197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff7537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7541022d0 0x7ff75419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33586/0 (socket says 172.21.15.40:33586) 2024-03-18T22:49:44.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff7537fe640 1 -- 172.21.15.40:0/3038229600 learned_addr learned my addr 172.21.15.40:0/3038229600 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:44.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff752ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff754100720 0x7ff754197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:60672/0 (socket says 172.21.15.40:60672) 2024-03-18T22:49:44.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff74bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff754101910 0x7ff754197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff7537fe640 1 -- 172.21.15.40:0/3038229600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff754100720 msgr2=0x7ff754197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:44.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff7537fe640 1 --2- 172.21.15.40:0/3038229600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff754100720 0x7ff754197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff7537fe640 1 -- 172.21.15.40:0/3038229600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff754101910 msgr2=0x7ff754197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:44.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff7537fe640 1 --2- 172.21.15.40:0/3038229600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff754101910 0x7ff754197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff7537fe640 1 -- 172.21.15.40:0/3038229600 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff754102b30 con 0x7ff7541022d0 2024-03-18T22:49:44.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.133+0000 7ff752ffd640 1 --2- 172.21.15.40:0/3038229600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff754100720 0x7ff754197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:44.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.134+0000 7ff7537fe640 1 --2- 172.21.15.40:0/3038229600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7541022d0 0x7ff75419f060 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00eb10 tx=0x7ff74c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:44.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.134+0000 7ff750ff9640 1 -- 172.21.15.40:0/3038229600 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c004b20 con 0x7ff7541022d0 2024-03-18T22:49:44.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.134+0000 7ff750ff9640 1 -- 172.21.15.40:0/3038229600 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff74c004cc0 con 0x7ff7541022d0 2024-03-18T22:49:44.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.134+0000 7ff75933b640 1 -- 172.21.15.40:0/3038229600 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff754102e20 con 0x7ff7541022d0 2024-03-18T22:49:44.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.134+0000 7ff750ff9640 1 -- 172.21.15.40:0/3038229600 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c010690 con 0x7ff7541022d0 2024-03-18T22:49:44.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.134+0000 7ff75933b640 1 -- 172.21.15.40:0/3038229600 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff754103320 con 0x7ff7541022d0 2024-03-18T22:49:44.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.136+0000 7ff750ff9640 1 -- 172.21.15.40:0/3038229600 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff74c005020 con 0x7ff7541022d0 2024-03-18T22:49:44.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff75c009740 1 Processor -- start 2024-03-18T22:49:44.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff75c009740 1 -- start start 2024-03-18T22:49:44.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff750ff9640 1 --2- 172.21.15.40:0/3038229600 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff7400a2800 0x7ff7400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff74bfff640 1 --2- 172.21.15.40:0/3038229600 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff7400a2800 0x7ff7400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff750ff9640 1 -- 172.21.15.40:0/3038229600 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff74c0cf3b0 con 0x7ff7541022d0 2024-03-18T22:49:44.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff75c009740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a9084e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff75c009740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 0x5570a9089f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff75c009740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a908ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff75c009740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5570a8fe9500 con 0x5570a908a5b0 2024-03-18T22:49:44.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff75c009740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5570a8f91be0 con 0x5570a9084a50 2024-03-18T22:49:44.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff75c009740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5570a8fe55a0 con 0x5570a9085440 2024-03-18T22:49:44.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff752ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a9084e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff7537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a908ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff752ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a9084e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:42246/0 (socket says 172.21.15.40:42246) 2024-03-18T22:49:44.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff752ffd640 1 -- 172.21.15.40:0/4178374652 learned_addr learned my addr 172.21.15.40:0/4178374652 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:44.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.137+0000 7ff74bfff640 1 --2- 172.21.15.40:0/4178374652 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 0x5570a9089f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.138+0000 7ff74bfff640 1 --2- 172.21.15.40:0/3038229600 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff7400a2800 0x7ff7400a4cc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff754198fc0 tx=0x7ff73c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:44.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.138+0000 7ff74bfff640 1 -- 172.21.15.40:0/4178374652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 msgr2=0x5570a9084e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:44.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.138+0000 7ff74bfff640 1 --2- 172.21.15.40:0/4178374652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a9084e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.138+0000 7ff74bfff640 1 -- 172.21.15.40:0/4178374652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 msgr2=0x5570a908ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:44.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.138+0000 7ff74bfff640 1 --2- 172.21.15.40:0/4178374652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a908ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.138+0000 7ff74bfff640 1 -- 172.21.15.40:0/4178374652 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570a8fb3000 con 0x5570a9085440 2024-03-18T22:49:44.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.138+0000 7ff752ffd640 1 --2- 172.21.15.40:0/4178374652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a9084e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:44.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.138+0000 7ff74bfff640 1 --2- 172.21.15.40:0/4178374652 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 0x5570a9089f40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff73c011df0 tx=0x7ff73c01c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53ae1034429ce91c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:44.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.138+0000 7ff7497fa640 1 -- 172.21.15.40:0/4178374652 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff73c004230 con 0x5570a9085440 2024-03-18T22:49:44.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.138+0000 7ff7497fa640 1 -- 172.21.15.40:0/4178374652 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff73c020030 con 0x5570a9085440 2024-03-18T22:49:44.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.138+0000 7ff7497fa640 1 -- 172.21.15.40:0/4178374652 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff73c004d70 con 0x5570a9085440 2024-03-18T22:49:44.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.139+0000 7ff75c009740 1 -- 172.21.15.40:0/4178374652 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 msgr2=0x5570a9089f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:44.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.139+0000 7ff75c009740 1 --2- 172.21.15.40:0/4178374652 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 0x5570a9089f40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff73c011df0 tx=0x7ff73c01c000 comp rx=0 tx=0).stop 2024-03-18T22:49:44.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.139+0000 7ff75c009740 1 -- 172.21.15.40:0/4178374652 shutdown_connections 2024-03-18T22:49:44.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.139+0000 7ff75c009740 1 --2- 172.21.15.40:0/4178374652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a908ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.139+0000 7ff75c009740 1 --2- 172.21.15.40:0/4178374652 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 0x5570a9089f40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.139+0000 7ff75c009740 1 --2- 172.21.15.40:0/4178374652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a9084e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.139+0000 7ff75c009740 1 -- 172.21.15.40:0/4178374652 >> 172.21.15.40:0/4178374652 conn(0x5570a907ffc0 msgr2=0x5570a90803c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:44.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.139+0000 7ff75c009740 1 -- 172.21.15.40:0/4178374652 shutdown_connections 2024-03-18T22:49:44.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.139+0000 7ff75c009740 1 -- 172.21.15.40:0/4178374652 wait complete. 2024-03-18T22:49:44.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.139+0000 7ff75c009740 1 -- start start 2024-03-18T22:49:44.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff75c009740 1 Processor -- start 2024-03-18T22:49:44.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff75c009740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9084a50 0x5570a908ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff75c009740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9085440 0x5570a908f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff75c009740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a90964e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff75c009740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5570a908d560 con 0x5570a908a5b0 2024-03-18T22:49:44.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff75c009740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5570a8fa5160 con 0x5570a9085440 2024-03-18T22:49:44.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff75c009740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5570a908d6e0 con 0x5570a9084a50 2024-03-18T22:49:44.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff7537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a90964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff752ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9084a50 0x5570a908ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff75c009740 10 client.0 _ll_register_callbacks cb 0x5570a8fe1a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:44.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff752ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9084a50 0x5570a908ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:60686/0 (socket says 172.21.15.40:60686) 2024-03-18T22:49:44.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff752ffd640 1 -- 172.21.15.40:0/2429184680 learned_addr learned my addr 172.21.15.40:0/2429184680 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:44.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff74bfff640 1 --2- 172.21.15.40:0/2429184680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9085440 0x5570a908f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff752ffd640 1 -- 172.21.15.40:0/2429184680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9085440 msgr2=0x5570a908f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:44.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff752ffd640 1 --2- 172.21.15.40:0/2429184680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9085440 0x5570a908f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff752ffd640 1 -- 172.21.15.40:0/2429184680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 msgr2=0x5570a90964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:44.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff752ffd640 1 --2- 172.21.15.40:0/2429184680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a90964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff752ffd640 1 -- 172.21.15.40:0/2429184680 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570a8dc77d0 con 0x5570a9084a50 2024-03-18T22:49:44.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff752ffd640 1 --2- 172.21.15.40:0/2429184680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9084a50 0x5570a908ef60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff744009900 tx=0x7ff744005250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e12d5eb1ce95eceb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:44.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.140+0000 7ff73a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:49:44.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.141+0000 7ff73a7fc640 1 -- 172.21.15.40:0/2429184680 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff744046070 con 0x5570a9084a50 2024-03-18T22:49:44.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.141+0000 7ff73a7fc640 1 -- 172.21.15.40:0/2429184680 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff7440046c0 con 0x5570a9084a50 2024-03-18T22:49:44.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.141+0000 7ff75c009740 1 -- 172.21.15.40:0/2429184680 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff73c020030 con 0x5570a9084a50 2024-03-18T22:49:44.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.141+0000 7ff73a7fc640 1 -- 172.21.15.40:0/2429184680 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff744052050 con 0x5570a9084a50 2024-03-18T22:49:44.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.141+0000 7ff75c009740 10 client.27256 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:44.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.141+0000 7ff75c009740 1 -- 172.21.15.40:0/2429184680 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5570a8cfd3f0 con 0x5570a9084a50 2024-03-18T22:49:44.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.141+0000 7ff73a7fc640 1 -- 172.21.15.40:0/2429184680 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff73c020030 con 0x5570a9084a50 2024-03-18T22:49:44.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.141+0000 7ff73a7fc640 1 -- 172.21.15.40:0/2429184680 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7ff7440036f0 con 0x5570a9084a50 2024-03-18T22:49:44.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.141+0000 7ff75c009740 10 client.27256 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:44.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.141+0000 7ff75c009740 10 client.27256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:44.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.141+0000 7ff75c009740 20 client.27256 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:44.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.141+0000 7ff75c009740 1 --2- 172.21.15.40:0/2429184680 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5570a909ad80 0x5570a909d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.141+0000 7ff75c009740 4 client.27256 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:49:44.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.141+0000 7ff75c009740 1 -- 172.21.15.40:0/2429184680 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570a8ff6a40 con 0x5570a909ad80 2024-03-18T22:49:44.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.142+0000 7ff74bfff640 1 --2- 172.21.15.40:0/2429184680 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5570a909ad80 0x5570a909d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.149 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:44 smithi040 ceph-mon[62643]: pgmap v563: 129 pgs: 129 active+clean; 8.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:49:44.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.142+0000 7ff74bfff640 1 --2- 172.21.15.40:0/2429184680 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5570a909ad80 0x5570a909d1d0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dd52f67d6e29e60 server_cookie=8ff7b32f5e4d7973 in_seq=0 out_seq=0 2024-03-18T22:49:44.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.142+0000 7ff73a7fc640 10 client.27256 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:44.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.143+0000 7ff73a7fc640 1 -- 172.21.15.40:0/2429184680 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5570a8ff6a40 con 0x5570a909ad80 2024-03-18T22:49:44.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.143+0000 7ff73a7fc640 10 client.27256 handle_command_reply: tid=0 2024-03-18T22:49:44.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.143+0000 7ff75c009740 1 client.27256 shutdown 2024-03-18T22:49:44.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.143+0000 7ff75c009740 1 -- 172.21.15.40:0/2429184680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9084a50 msgr2=0x5570a908ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:44.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.143+0000 7ff75c009740 1 --2- 172.21.15.40:0/2429184680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9084a50 0x5570a908ef60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff744009900 tx=0x7ff744005250 comp rx=0 tx=0).stop 2024-03-18T22:49:44.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.143+0000 7ff75c009740 1 -- 172.21.15.40:0/2429184680 shutdown_connections 2024-03-18T22:49:44.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.143+0000 7ff75c009740 1 --2- 172.21.15.40:0/2429184680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a90964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.143+0000 7ff75c009740 1 --2- 172.21.15.40:0/2429184680 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5570a909ad80 0x5570a909d1d0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.143+0000 7ff75c009740 1 --2- 172.21.15.40:0/2429184680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9085440 0x5570a908f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.143+0000 7ff73a7fc640 0 client.27256 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:44.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.143+0000 7ff75c009740 1 --2- 172.21.15.40:0/2429184680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9084a50 0x5570a908ef60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.143+0000 7ff75c009740 1 -- 172.21.15.40:0/2429184680 >> 172.21.15.40:0/2429184680 conn(0x5570a907ffc0 msgr2=0x5570a907f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:44.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.143+0000 7ff75c009740 1 -- 172.21.15.40:0/2429184680 shutdown_connections 2024-03-18T22:49:44.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.143+0000 7ff75c009740 1 -- 172.21.15.40:0/2429184680 wait complete. 2024-03-18T22:49:44.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.143+0000 7ff75c009740 20 client.27256 trim_cache size 0 max 16384 2024-03-18T22:49:44.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.164+0000 7ff75c009740 1 Processor -- start 2024-03-18T22:49:44.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.164+0000 7ff75c009740 1 -- start start 2024-03-18T22:49:44.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.164+0000 7ff75c009740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a908e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.164+0000 7ff75c009740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 0x5570a90964a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.164+0000 7ff75c009740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a908eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.164+0000 7ff75c009740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff7541ac780 con 0x5570a908a5b0 2024-03-18T22:49:44.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.164+0000 7ff75c009740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff7541ae380 con 0x5570a9084a50 2024-03-18T22:49:44.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.164+0000 7ff75c009740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff7541ace50 con 0x5570a9085440 2024-03-18T22:49:44.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.164+0000 7ff7537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a908eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.164+0000 7ff752ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a908e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.165+0000 7ff7537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a908eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33608/0 (socket says 172.21.15.40:33608) 2024-03-18T22:49:44.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.165+0000 7ff74bfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 0x5570a90964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.165+0000 7ff7537fe640 1 -- 172.21.15.40:0/481388910 learned_addr learned my addr 172.21.15.40:0/481388910 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.165+0000 7ff752ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a908e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:42254/0 (socket says 172.21.15.40:42254) 2024-03-18T22:49:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.165+0000 7ff752ffd640 1 -- 172.21.15.40:0/481388910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 msgr2=0x5570a90964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.165+0000 7ff752ffd640 1 --2- 172.21.15.40:0/481388910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 0x5570a90964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.165+0000 7ff752ffd640 1 -- 172.21.15.40:0/481388910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 msgr2=0x5570a908eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.165+0000 7ff752ffd640 1 --2- 172.21.15.40:0/481388910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a908eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.165+0000 7ff752ffd640 1 -- 172.21.15.40:0/481388910 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570a8cbd0f0 con 0x5570a9084a50 2024-03-18T22:49:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.165+0000 7ff752ffd640 1 --2- 172.21.15.40:0/481388910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a908e580 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff744003ac0 tx=0x7ff74402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fe8d0f33c3a17ce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.165+0000 7ff748ff9640 1 -- 172.21.15.40:0/481388910 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff744046070 con 0x5570a9084a50 2024-03-18T22:49:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.165+0000 7ff748ff9640 1 -- 172.21.15.40:0/481388910 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff74402cc90 con 0x5570a9084a50 2024-03-18T22:49:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.165+0000 7ff748ff9640 1 -- 172.21.15.40:0/481388910 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff744004b70 con 0x5570a9084a50 2024-03-18T22:49:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.166+0000 7ff75c009740 1 -- 172.21.15.40:0/481388910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 msgr2=0x5570a908e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.166+0000 7ff75c009740 1 --2- 172.21.15.40:0/481388910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a908e580 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff744003ac0 tx=0x7ff74402bc30 comp rx=0 tx=0).stop 2024-03-18T22:49:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.166+0000 7ff75c009740 1 -- 172.21.15.40:0/481388910 shutdown_connections 2024-03-18T22:49:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.166+0000 7ff75c009740 1 --2- 172.21.15.40:0/481388910 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a908eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.166+0000 7ff75c009740 1 --2- 172.21.15.40:0/481388910 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 0x5570a90964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.166+0000 7ff75c009740 1 --2- 172.21.15.40:0/481388910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a908e580 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.166+0000 7ff75c009740 1 -- 172.21.15.40:0/481388910 >> 172.21.15.40:0/481388910 conn(0x5570a907ffc0 msgr2=0x5570a9083ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.166+0000 7ff75c009740 1 -- 172.21.15.40:0/481388910 shutdown_connections 2024-03-18T22:49:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.166+0000 7ff75c009740 1 -- 172.21.15.40:0/481388910 wait complete. 2024-03-18T22:49:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.166+0000 7ff75c009740 1 -- start start 2024-03-18T22:49:44.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.166+0000 7ff75c009740 1 Processor -- start 2024-03-18T22:49:44.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff75c009740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a9093a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff75c009740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 0x5570a9093f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff75c009740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a9094f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff75c009740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff7541adcb0 con 0x5570a908a5b0 2024-03-18T22:49:44.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff75c009740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff7541ad520 con 0x5570a9084a50 2024-03-18T22:49:44.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff75c009740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff7541aa760 con 0x5570a9085440 2024-03-18T22:49:44.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff75c009740 10 client.0 _ll_register_callbacks cb 0x7ff744009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:49:44.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff7537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a9094f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff7537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a9094f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33610/0 (socket says 172.21.15.40:33610) 2024-03-18T22:49:44.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff752ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a9093a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff7537fe640 1 -- 172.21.15.40:0/1949987999 learned_addr learned my addr 172.21.15.40:0/1949987999 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:49:44.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff74bfff640 1 --2- 172.21.15.40:0/1949987999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 0x5570a9093f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff7537fe640 1 -- 172.21.15.40:0/1949987999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 msgr2=0x5570a9093f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff7537fe640 1 --2- 172.21.15.40:0/1949987999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 0x5570a9093f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff7537fe640 1 -- 172.21.15.40:0/1949987999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 msgr2=0x5570a9093a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:49:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff7537fe640 1 --2- 172.21.15.40:0/1949987999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a9093a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff7537fe640 1 -- 172.21.15.40:0/1949987999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570a9095560 con 0x5570a908a5b0 2024-03-18T22:49:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff752ffd640 1 --2- 172.21.15.40:0/1949987999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a9093a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:49:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff7537fe640 1 --2- 172.21.15.40:0/1949987999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a9094f70 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff74c098330 tx=0x7ff74c09c060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d25f26b38185ec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:49:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff73a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:49:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff73a7fc640 1 -- 172.21.15.40:0/1949987999 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c0a4100 con 0x5570a908a5b0 2024-03-18T22:49:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff73a7fc640 1 -- 172.21.15.40:0/1949987999 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff74c0a3c70 con 0x5570a908a5b0 2024-03-18T22:49:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.167+0000 7ff75c009740 1 -- 172.21.15.40:0/1949987999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff74402cc90 con 0x5570a908a5b0 2024-03-18T22:49:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.168+0000 7ff73a7fc640 1 -- 172.21.15.40:0/1949987999 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff74c0a3e70 con 0x5570a908a5b0 2024-03-18T22:49:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.168+0000 7ff75c009740 10 client.17454 fetch_fsmap learned FSMap version 34 2024-03-18T22:49:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.168+0000 7ff75c009740 1 -- 172.21.15.40:0/1949987999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5570a91010d0 con 0x5570a908a5b0 2024-03-18T22:49:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.168+0000 7ff73a7fc640 1 -- 172.21.15.40:0/1949987999 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff74402cc90 con 0x5570a908a5b0 2024-03-18T22:49:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.168+0000 7ff73a7fc640 1 -- 172.21.15.40:0/1949987999 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7ff74c0861d0 con 0x5570a908a5b0 2024-03-18T22:49:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.168+0000 7ff75c009740 10 client.17454 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:49:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.168+0000 7ff75c009740 10 client.17454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:49:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.168+0000 7ff75c009740 20 client.17454 populate_metadata read hostname 'smithi040' 2024-03-18T22:49:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.168+0000 7ff75c009740 1 --2- 172.21.15.40:0/1949987999 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5570a9101a00 0x5570a9103e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:49:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.168+0000 7ff75c009740 4 client.17454 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:49:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.168+0000 7ff75c009740 1 -- 172.21.15.40:0/1949987999 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5570a8ff0d90 con 0x5570a9101a00 2024-03-18T22:49:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.168+0000 7ff752ffd640 1 --2- 172.21.15.40:0/1949987999 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5570a9101a00 0x5570a9103e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:49:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff752ffd640 1 --2- 172.21.15.40:0/1949987999 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5570a9101a00 0x5570a9103e20 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7c059366b446d1c server_cookie=572a59c25e2dd9bb in_seq=0 out_seq=0 2024-03-18T22:49:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff73a7fc640 10 client.17454 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff73a7fc640 1 -- 172.21.15.40:0/1949987999 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5570a8ff0d90 con 0x5570a9101a00 2024-03-18T22:49:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff73a7fc640 10 client.17454 handle_command_reply: tid=0 2024-03-18T22:49:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff75c009740 1 client.17454 shutdown 2024-03-18T22:49:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff75c009740 1 -- 172.21.15.40:0/1949987999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 msgr2=0x5570a9094f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff75c009740 1 --2- 172.21.15.40:0/1949987999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a9094f70 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff74c098330 tx=0x7ff74c09c060 comp rx=0 tx=0).stop 2024-03-18T22:49:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff75c009740 1 -- 172.21.15.40:0/1949987999 shutdown_connections 2024-03-18T22:49:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff75c009740 1 --2- 172.21.15.40:0/1949987999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5570a908a5b0 0x5570a9094f70 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff75c009740 1 --2- 172.21.15.40:0/1949987999 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5570a9085440 0x5570a9093f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff75c009740 1 --2- 172.21.15.40:0/1949987999 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5570a9101a00 0x5570a9103e20 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff75c009740 1 --2- 172.21.15.40:0/1949987999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570a9084a50 0x5570a9093a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff73a7fc640 0 client.17454 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:49:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff75c009740 1 -- 172.21.15.40:0/1949987999 >> 172.21.15.40:0/1949987999 conn(0x5570a907ffc0 msgr2=0x5570a907e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff75c009740 1 -- 172.21.15.40:0/1949987999 shutdown_connections 2024-03-18T22:49:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff75c009740 1 -- 172.21.15.40:0/1949987999 wait complete. 2024-03-18T22:49:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.169+0000 7ff75c009740 20 client.17454 trim_cache size 0 max 16384 2024-03-18T22:49:44.178 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:49:44.178 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:49:44.178 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:49:44.178 INFO:teuthology.orchestra.run.smithi040.stdout: "0947e7e2-240a-4a92-ab0d-e8d2d6001560": { 2024-03-18T22:49:44.178 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:49:44.178 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "0947e7e2-240a-4a92-ab0d-e8d2d6001560", 2024-03-18T22:49:44.178 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:49:44.178 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:49:44.178 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:49:44.178 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:49:44.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.170+0000 7ff75933b640 1 -- 172.21.15.40:0/3038229600 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff7400a2800 msgr2=0x7ff7400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:44.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.170+0000 7ff75933b640 1 --2- 172.21.15.40:0/3038229600 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff7400a2800 0x7ff7400a4cc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff754198fc0 tx=0x7ff73c005f90 comp rx=0 tx=0).stop 2024-03-18T22:49:44.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.170+0000 7ff75933b640 1 -- 172.21.15.40:0/3038229600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7541022d0 msgr2=0x7ff75419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:49:44.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.170+0000 7ff75933b640 1 --2- 172.21.15.40:0/3038229600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7541022d0 0x7ff75419f060 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00eb10 tx=0x7ff74c00c2d0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.170+0000 7ff75933b640 1 -- 172.21.15.40:0/3038229600 shutdown_connections 2024-03-18T22:49:44.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.170+0000 7ff75933b640 1 --2- 172.21.15.40:0/3038229600 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff7400a2800 0x7ff7400a4cc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.170+0000 7ff75933b640 1 --2- 172.21.15.40:0/3038229600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7541022d0 0x7ff75419f060 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.170+0000 7ff75933b640 1 --2- 172.21.15.40:0/3038229600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff754101910 0x7ff754197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.170+0000 7ff75933b640 1 --2- 172.21.15.40:0/3038229600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff754100720 0x7ff754197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:49:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.170+0000 7ff75933b640 1 -- 172.21.15.40:0/3038229600 >> 172.21.15.40:0/3038229600 conn(0x7ff7540fbcb0 msgr2=0x7ff754068a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:49:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.170+0000 7ff75933b640 1 -- 172.21.15.40:0/3038229600 shutdown_connections 2024-03-18T22:49:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:49:44.170+0000 7ff75933b640 1 -- 172.21.15.40:0/3038229600 wait complete. 2024-03-18T22:49:44.182 INFO:tasks.cephfs.filesystem:scrub status for tag:0947e7e2-240a-4a92-ab0d-e8d2d6001560 - {'path': '/', 'tag': '0947e7e2-240a-4a92-ab0d-e8d2d6001560', 'options': 'recursive,force'} 2024-03-18T22:49:44.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:44 smithi113 ceph-mon[66775]: pgmap v563: 129 pgs: 129 active+clean; 8.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:49:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:44 smithi103 ceph-mon[66377]: pgmap v563: 129 pgs: 129 active+clean; 8.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:49:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:45 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:49:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:45 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:45 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:49:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:45 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:49:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:45 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:45.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:45 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:45 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:49:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:45 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:45 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:49:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:45 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:49:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:45 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:45.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:45 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:45.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:45 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:49:45.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:45 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:45.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:45 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:49:45.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:45 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:49:45.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:45 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:45.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:45 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:49:46.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:46 smithi113 ceph-mon[66775]: pgmap v564: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:49:46.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:46 smithi103 ceph-mon[66377]: pgmap v564: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:49:46.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:46 smithi040 ceph-mon[62643]: pgmap v564: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:49:48.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:48 smithi113 ceph-mon[66775]: pgmap v565: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:49:48.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:48 smithi103 ceph-mon[66377]: pgmap v565: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:49:48.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:48 smithi040 ceph-mon[62643]: pgmap v565: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:49:50.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:50 smithi113 ceph-mon[66775]: pgmap v566: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:49:50.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:50 smithi103 ceph-mon[66377]: pgmap v566: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:49:50.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:50 smithi040 ceph-mon[62643]: pgmap v566: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 27 MiB/s wr, 20 op/s 2024-03-18T22:49:52.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:52 smithi113 ceph-mon[66775]: pgmap v567: 129 pgs: 129 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:49:52.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:52 smithi103 ceph-mon[66377]: pgmap v567: 129 pgs: 129 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:49:52.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:52 smithi040 ceph-mon[62643]: pgmap v567: 129 pgs: 129 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:49:53.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:53 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:49:53.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:53 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:49:53.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:53 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:49:54.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:54 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:49:54.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:54 smithi113 ceph-mon[66775]: pgmap v568: 129 pgs: 129 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:49:54.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:54 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:49:54.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:54 smithi103 ceph-mon[66377]: pgmap v568: 129 pgs: 129 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:49:54.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:54 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:49:54.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:54 smithi040 ceph-mon[62643]: pgmap v568: 129 pgs: 129 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:49:56.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:56 smithi113 ceph-mon[66775]: pgmap v569: 129 pgs: 129 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:49:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:56 smithi103 ceph-mon[66377]: pgmap v569: 129 pgs: 129 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:49:56.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:56 smithi040 ceph-mon[62643]: pgmap v569: 129 pgs: 129 active+clean; 9.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:49:58.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:49:58 smithi113 ceph-mon[66775]: pgmap v570: 129 pgs: 129 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:49:58.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:49:58 smithi103 ceph-mon[66377]: pgmap v570: 129 pgs: 129 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:49:58.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:49:58 smithi040 ceph-mon[62643]: pgmap v570: 129 pgs: 129 active+clean; 9.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 20 op/s 2024-03-18T22:50:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:00 smithi113 ceph-mon[66775]: pgmap v571: 129 pgs: 129 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:50:00.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:00 smithi113 ceph-mon[66775]: overall HEALTH_OK 2024-03-18T22:50:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:00 smithi103 ceph-mon[66377]: pgmap v571: 129 pgs: 129 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:50:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:00 smithi103 ceph-mon[66377]: overall HEALTH_OK 2024-03-18T22:50:00.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:00 smithi040 ceph-mon[62643]: pgmap v571: 129 pgs: 129 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:50:00.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:00 smithi040 ceph-mon[62643]: overall HEALTH_OK 2024-03-18T22:50:02.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:02 smithi113 ceph-mon[66775]: pgmap v572: 129 pgs: 129 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:50:02.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:02 smithi103 ceph-mon[66377]: pgmap v572: 129 pgs: 129 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:50:02.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:02 smithi040 ceph-mon[62643]: pgmap v572: 129 pgs: 129 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:50:04.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:04 smithi113 ceph-mon[66775]: pgmap v573: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:50:04.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:04 smithi103 ceph-mon[66377]: pgmap v573: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:50:04.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:04 smithi040 ceph-mon[62643]: pgmap v573: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:50:06.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:06 smithi113 ceph-mon[66775]: pgmap v574: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:50:06.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:06 smithi103 ceph-mon[66377]: pgmap v574: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:50:06.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:06 smithi040 ceph-mon[62643]: pgmap v574: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:50:08.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:08 smithi113 ceph-mon[66775]: pgmap v575: 129 pgs: 129 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 30 MiB/s wr, 20 op/s 2024-03-18T22:50:08.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:08 smithi103 ceph-mon[66377]: pgmap v575: 129 pgs: 129 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 30 MiB/s wr, 20 op/s 2024-03-18T22:50:08.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:08 smithi040 ceph-mon[62643]: pgmap v575: 129 pgs: 129 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 30 MiB/s wr, 20 op/s 2024-03-18T22:50:10.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:10 smithi113 ceph-mon[66775]: pgmap v576: 129 pgs: 129 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 26 MiB/s wr, 19 op/s 2024-03-18T22:50:10.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:10 smithi103 ceph-mon[66377]: pgmap v576: 129 pgs: 129 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 26 MiB/s wr, 19 op/s 2024-03-18T22:50:10.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:10 smithi040 ceph-mon[62643]: pgmap v576: 129 pgs: 129 active+clean; 9.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 26 MiB/s wr, 19 op/s 2024-03-18T22:50:12.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:12 smithi113 ceph-mon[66775]: pgmap v577: 129 pgs: 129 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:50:12.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:12 smithi103 ceph-mon[66377]: pgmap v577: 129 pgs: 129 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:50:12.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:12 smithi040 ceph-mon[62643]: pgmap v577: 129 pgs: 129 active+clean; 9.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:50:14.183 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:50:14.184 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:50:14.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.295+0000 7f4c49e31640 1 -- 172.21.15.40:0/1859839313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c44101910 msgr2=0x7f4c44101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.295+0000 7f4c49e31640 1 --2- 172.21.15.40:0/1859839313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c44101910 0x7f4c44101d90 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f4c38009900 tx=0x7f4c3802bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:14.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.295+0000 7f4c49e31640 1 -- 172.21.15.40:0/1859839313 shutdown_connections 2024-03-18T22:50:14.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.295+0000 7f4c49e31640 1 --2- 172.21.15.40:0/1859839313 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c441022d0 0x7f4c44108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.295+0000 7f4c49e31640 1 --2- 172.21.15.40:0/1859839313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c44101910 0x7f4c44101d90 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.295+0000 7f4c49e31640 1 --2- 172.21.15.40:0/1859839313 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44100720 0x7f4c44100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.295+0000 7f4c49e31640 1 -- 172.21.15.40:0/1859839313 >> 172.21.15.40:0/1859839313 conn(0x7f4c440fbcb0 msgr2=0x7f4c440fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:14.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.295+0000 7f4c49e31640 1 -- 172.21.15.40:0/1859839313 shutdown_connections 2024-03-18T22:50:14.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.295+0000 7f4c49e31640 1 -- 172.21.15.40:0/1859839313 wait complete. 2024-03-18T22:50:14.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.296+0000 7f4c49e31640 1 Processor -- start 2024-03-18T22:50:14.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.296+0000 7f4c49e31640 1 -- start start 2024-03-18T22:50:14.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.296+0000 7f4c49e31640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c44100720 0x7f4c440694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.296+0000 7f4c49e31640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c44101910 0x7f4c4419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.296+0000 7f4c49e31640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c441022d0 0x7f4c4419ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.296+0000 7f4c49e31640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4410c230 con 0x7f4c44100720 2024-03-18T22:50:14.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.296+0000 7f4c49e31640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4410c0b0 con 0x7f4c441022d0 2024-03-18T22:50:14.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.296+0000 7f4c49e31640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4410c3b0 con 0x7f4c44101910 2024-03-18T22:50:14.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c437fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c44100720 0x7f4c440694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c437fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c44100720 0x7f4c440694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45722/0 (socket says 172.21.15.40:45722) 2024-03-18T22:50:14.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c437fe640 1 -- 172.21.15.40:0/40148730 learned_addr learned my addr 172.21.15.40:0/40148730 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:14.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c42ffd640 1 --2- 172.21.15.40:0/40148730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c44101910 0x7f4c4419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c43fff640 1 --2- 172.21.15.40:0/40148730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c441022d0 0x7f4c4419ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c437fe640 1 -- 172.21.15.40:0/40148730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c44101910 msgr2=0x7f4c4419c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c437fe640 1 --2- 172.21.15.40:0/40148730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c44101910 0x7f4c4419c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c437fe640 1 -- 172.21.15.40:0/40148730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c441022d0 msgr2=0x7f4c4419ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c437fe640 1 --2- 172.21.15.40:0/40148730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c441022d0 0x7f4c4419ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c437fe640 1 -- 172.21.15.40:0/40148730 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4419f6f0 con 0x7f4c44100720 2024-03-18T22:50:14.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c437fe640 1 --2- 172.21.15.40:0/40148730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c44100720 0x7f4c440694f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f4c3400e9c0 tx=0x7f4c3400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:14.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c40ff9640 1 -- 172.21.15.40:0/40148730 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3400cd80 con 0x7f4c44100720 2024-03-18T22:50:14.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c40ff9640 1 -- 172.21.15.40:0/40148730 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c340035f0 con 0x7f4c44100720 2024-03-18T22:50:14.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c49e31640 1 -- 172.21.15.40:0/40148730 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c4419f980 con 0x7f4c44100720 2024-03-18T22:50:14.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c40ff9640 1 -- 172.21.15.40:0/40148730 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c34010690 con 0x7f4c44100720 2024-03-18T22:50:14.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.297+0000 7f4c49e31640 1 -- 172.21.15.40:0/40148730 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c4419fef0 con 0x7f4c44100720 2024-03-18T22:50:14.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c40ff9640 1 -- 172.21.15.40:0/40148730 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4c340037a0 con 0x7f4c44100720 2024-03-18T22:50:14.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c4ca18740 1 Processor -- start 2024-03-18T22:50:14.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c4ca18740 1 -- start start 2024-03-18T22:50:14.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c4ca18740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed382ea50 0x556ed382ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c4ca18740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382f440 0x556ed3833f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c4ca18740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed38345b0 0x556ed3836a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c4ca18740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556ed3793500 con 0x556ed382ea50 2024-03-18T22:50:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c4ca18740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556ed373bbe0 con 0x556ed38345b0 2024-03-18T22:50:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c4ca18740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556ed378f5a0 con 0x556ed382f440 2024-03-18T22:50:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c43fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed382ea50 0x556ed382ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c43fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed382ea50 0x556ed382ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45724/0 (socket says 172.21.15.40:45724) 2024-03-18T22:50:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c43fff640 1 -- 172.21.15.40:0/1579057118 learned_addr learned my addr 172.21.15.40:0/1579057118 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c42ffd640 1 --2- 172.21.15.40:0/1579057118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed38345b0 0x556ed3836a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c437fe640 1 --2- 172.21.15.40:0/1579057118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382f440 0x556ed3833f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c40ff9640 1 --2- 172.21.15.40:0/40148730 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4c2c0a2880 0x7f4c2c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c43fff640 1 -- 172.21.15.40:0/1579057118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382f440 msgr2=0x556ed3833f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c43fff640 1 --2- 172.21.15.40:0/1579057118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382f440 0x556ed3833f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c43fff640 1 -- 172.21.15.40:0/1579057118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed38345b0 msgr2=0x556ed3836a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c43fff640 1 --2- 172.21.15.40:0/1579057118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed38345b0 0x556ed3836a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.300+0000 7f4c43fff640 1 -- 172.21.15.40:0/1579057118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ed375d000 con 0x556ed382ea50 2024-03-18T22:50:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.301+0000 7f4c437fe640 1 --2- 172.21.15.40:0/1579057118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382f440 0x556ed3833f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:50:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.301+0000 7f4c42ffd640 1 --2- 172.21.15.40:0/1579057118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed38345b0 0x556ed3836a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.301+0000 7f4c40ff9640 1 -- 172.21.15.40:0/40148730 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4c340cf160 con 0x7f4c44100720 2024-03-18T22:50:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.301+0000 7f4c43fff640 1 --2- 172.21.15.40:0/40148730 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4c2c0a2880 0x7f4c2c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.301+0000 7f4c43fff640 1 --2- 172.21.15.40:0/1579057118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed382ea50 0x556ed382ee70 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c011590 tx=0x7f4c3c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44166659a4daaecf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.301+0000 7f4c29ffb640 1 -- 172.21.15.40:0/1579057118 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c017050 con 0x556ed382ea50 2024-03-18T22:50:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.301+0000 7f4c29ffb640 1 -- 172.21.15.40:0/1579057118 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c3c00fcf0 con 0x556ed382ea50 2024-03-18T22:50:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.301+0000 7f4c29ffb640 1 -- 172.21.15.40:0/1579057118 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c01ba90 con 0x556ed382ea50 2024-03-18T22:50:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.302+0000 7f4c4ca18740 1 -- 172.21.15.40:0/1579057118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed382ea50 msgr2=0x556ed382ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.302+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/1579057118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed382ea50 0x556ed382ee70 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c011590 tx=0x7f4c3c011a50 comp rx=0 tx=0).stop 2024-03-18T22:50:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.302+0000 7f4c43fff640 1 --2- 172.21.15.40:0/40148730 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4c2c0a2880 0x7f4c2c0a4d40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c022000 tx=0x7f4c3c023000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.302+0000 7f4c4ca18740 1 -- 172.21.15.40:0/1579057118 shutdown_connections 2024-03-18T22:50:14.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.302+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/1579057118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed38345b0 0x556ed3836a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.302+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/1579057118 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382f440 0x556ed3833f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.302+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/1579057118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed382ea50 0x556ed382ee70 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.302+0000 7f4c4ca18740 1 -- 172.21.15.40:0/1579057118 >> 172.21.15.40:0/1579057118 conn(0x556ed3829fc0 msgr2=0x556ed382a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:14.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.302+0000 7f4c4ca18740 1 -- 172.21.15.40:0/1579057118 shutdown_connections 2024-03-18T22:50:14.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.302+0000 7f4c4ca18740 1 -- 172.21.15.40:0/1579057118 wait complete. 2024-03-18T22:50:14.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.302+0000 7f4c4ca18740 1 -- start start 2024-03-18T22:50:14.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.302+0000 7f4c4ca18740 1 Processor -- start 2024-03-18T22:50:14.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c4ca18740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 0x556ed3838ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c4ca18740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed382f440 0x556ed38393e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c4ca18740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed38345b0 0x556ed3840420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c4ca18740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556ed3837560 con 0x556ed382f440 2024-03-18T22:50:14.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c4ca18740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556ed374f160 con 0x556ed38345b0 2024-03-18T22:50:14.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c4ca18740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556ed38376e0 con 0x556ed382ea50 2024-03-18T22:50:14.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c4ca18740 10 client.0 _ll_register_callbacks cb 0x556ed378ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:14.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c42ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed382f440 0x556ed38393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c437fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 0x556ed3838ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c437fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 0x556ed3838ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53812/0 (socket says 172.21.15.40:53812) 2024-03-18T22:50:14.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c437fe640 1 -- 172.21.15.40:0/1931747353 learned_addr learned my addr 172.21.15.40:0/1931747353 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:14.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c43fff640 1 --2- 172.21.15.40:0/1931747353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed38345b0 0x556ed3840420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c43fff640 1 -- 172.21.15.40:0/1931747353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 msgr2=0x556ed3838ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c43fff640 1 --2- 172.21.15.40:0/1931747353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 0x556ed3838ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c43fff640 1 -- 172.21.15.40:0/1931747353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed382f440 msgr2=0x556ed38393e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c43fff640 1 --2- 172.21.15.40:0/1931747353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed382f440 0x556ed38393e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c43fff640 1 -- 172.21.15.40:0/1931747353 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ed35717d0 con 0x556ed38345b0 2024-03-18T22:50:14.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c43fff640 1 --2- 172.21.15.40:0/1931747353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed38345b0 0x556ed3840420 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c011560 tx=0x7f4c3c003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12e52871b9f05bfd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:14.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.303+0000 7f4c22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:50:14.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.304+0000 7f4c22ffd640 1 -- 172.21.15.40:0/1931747353 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c035060 con 0x556ed38345b0 2024-03-18T22:50:14.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.304+0000 7f4c4ca18740 1 -- 172.21.15.40:0/1931747353 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c3c00fcf0 con 0x556ed38345b0 2024-03-18T22:50:14.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.304+0000 7f4c22ffd640 1 -- 172.21.15.40:0/1931747353 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4c3c004bf0 con 0x556ed38345b0 2024-03-18T22:50:14.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.304+0000 7f4c22ffd640 1 -- 172.21.15.40:0/1931747353 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4c3c017020 con 0x556ed38345b0 2024-03-18T22:50:14.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.304+0000 7f4c4ca18740 10 client.27281 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:14.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.304+0000 7f4c4ca18740 1 -- 172.21.15.40:0/1931747353 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x556ed34a73f0 con 0x556ed38345b0 2024-03-18T22:50:14.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.304+0000 7f4c22ffd640 1 -- 172.21.15.40:0/1931747353 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c3c00fcf0 con 0x556ed38345b0 2024-03-18T22:50:14.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.304+0000 7f4c22ffd640 1 -- 172.21.15.40:0/1931747353 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f4c3c01a070 con 0x556ed38345b0 2024-03-18T22:50:14.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.304+0000 7f4c4ca18740 10 client.27281 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:14.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.304+0000 7f4c4ca18740 10 client.27281 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:14.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.304+0000 7f4c4ca18740 20 client.27281 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:14.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.304+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/1931747353 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed3844a90 0x556ed3846ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.304+0000 7f4c4ca18740 4 client.27281 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:14.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.304+0000 7f4c4ca18740 1 -- 172.21.15.40:0/1931747353 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ed37a0a40 con 0x556ed3844a90 2024-03-18T22:50:14.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.305+0000 7f4c437fe640 1 --2- 172.21.15.40:0/1931747353 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed3844a90 0x556ed3846ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.305+0000 7f4c437fe640 1 --2- 172.21.15.40:0/1931747353 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed3844a90 0x556ed3846ee0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3273a458c287c3c7 server_cookie=348a752409f80199 in_seq=0 out_seq=0 2024-03-18T22:50:14.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.305+0000 7f4c22ffd640 10 client.27281 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:14.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.306+0000 7f4c22ffd640 1 -- 172.21.15.40:0/1931747353 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556ed37a0a40 con 0x556ed3844a90 2024-03-18T22:50:14.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.306+0000 7f4c22ffd640 10 client.27281 handle_command_reply: tid=0 2024-03-18T22:50:14.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.306+0000 7f4c4ca18740 1 client.27281 shutdown 2024-03-18T22:50:14.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.306+0000 7f4c4ca18740 1 -- 172.21.15.40:0/1931747353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed38345b0 msgr2=0x556ed3840420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.306+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/1931747353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed38345b0 0x556ed3840420 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c011560 tx=0x7f4c3c003970 comp rx=0 tx=0).stop 2024-03-18T22:50:14.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.306+0000 7f4c4ca18740 1 -- 172.21.15.40:0/1931747353 shutdown_connections 2024-03-18T22:50:14.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.306+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/1931747353 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed3844a90 0x556ed3846ee0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.306+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/1931747353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed38345b0 0x556ed3840420 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.306+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/1931747353 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed382f440 0x556ed38393e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.306+0000 7f4c22ffd640 0 client.27281 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:14.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.306+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/1931747353 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 0x556ed3838ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.306+0000 7f4c4ca18740 1 -- 172.21.15.40:0/1931747353 >> 172.21.15.40:0/1931747353 conn(0x556ed3829fc0 msgr2=0x556ed38291e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:14.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.306+0000 7f4c4ca18740 1 -- 172.21.15.40:0/1931747353 shutdown_connections 2024-03-18T22:50:14.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.306+0000 7f4c4ca18740 1 -- 172.21.15.40:0/1931747353 wait complete. 2024-03-18T22:50:14.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.306+0000 7f4c4ca18740 20 client.27281 trim_cache size 0 max 16384 2024-03-18T22:50:14.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c4ca18740 1 Processor -- start 2024-03-18T22:50:14.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c4ca18740 1 -- start start 2024-03-18T22:50:14.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c4ca18740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 0x556ed3824f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c4ca18740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed382f440 0x556ed38403e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c4ca18740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed38345b0 0x556ed382d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c4ca18740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4c441aa5b0 con 0x556ed38345b0 2024-03-18T22:50:14.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c4ca18740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4c441ac1b0 con 0x556ed382f440 2024-03-18T22:50:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c4ca18740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4c441aac80 con 0x556ed382ea50 2024-03-18T22:50:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c43fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed38345b0 0x556ed382d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c42ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed382f440 0x556ed38403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c42ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed382f440 0x556ed38403e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43560/0 (socket says 172.21.15.40:43560) 2024-03-18T22:50:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c42ffd640 1 -- 172.21.15.40:0/2759389742 learned_addr learned my addr 172.21.15.40:0/2759389742 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c437fe640 1 --2- 172.21.15.40:0/2759389742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 0x556ed3824f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c42ffd640 1 -- 172.21.15.40:0/2759389742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 msgr2=0x556ed3824f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c42ffd640 1 --2- 172.21.15.40:0/2759389742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 0x556ed3824f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c42ffd640 1 -- 172.21.15.40:0/2759389742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed38345b0 msgr2=0x556ed382d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c42ffd640 1 --2- 172.21.15.40:0/2759389742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed38345b0 0x556ed382d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c42ffd640 1 -- 172.21.15.40:0/2759389742 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ed3830f40 con 0x556ed382f440 2024-03-18T22:50:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.324+0000 7f4c43fff640 1 --2- 172.21.15.40:0/2759389742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed38345b0 0x556ed382d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.325+0000 7f4c42ffd640 1 --2- 172.21.15.40:0/2759389742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed382f440 0x556ed38403e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4c38009900 tx=0x7f4c3802c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c389f56c1e909c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.325+0000 7f4c297fa640 1 -- 172.21.15.40:0/2759389742 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3802c9e0 con 0x556ed382f440 2024-03-18T22:50:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.325+0000 7f4c297fa640 1 -- 172.21.15.40:0/2759389742 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c3802cb80 con 0x556ed382f440 2024-03-18T22:50:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.325+0000 7f4c297fa640 1 -- 172.21.15.40:0/2759389742 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c38004920 con 0x556ed382f440 2024-03-18T22:50:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.326+0000 7f4c4ca18740 1 -- 172.21.15.40:0/2759389742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed382f440 msgr2=0x556ed38403e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.326+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/2759389742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed382f440 0x556ed38403e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4c38009900 tx=0x7f4c3802c800 comp rx=0 tx=0).stop 2024-03-18T22:50:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.326+0000 7f4c4ca18740 1 -- 172.21.15.40:0/2759389742 shutdown_connections 2024-03-18T22:50:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.326+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/2759389742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed38345b0 0x556ed382d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.326+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/2759389742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed382f440 0x556ed38403e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.326+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/2759389742 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 0x556ed3824f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.326+0000 7f4c4ca18740 1 -- 172.21.15.40:0/2759389742 >> 172.21.15.40:0/2759389742 conn(0x556ed3829fc0 msgr2=0x556ed382d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.326+0000 7f4c4ca18740 1 -- 172.21.15.40:0/2759389742 shutdown_connections 2024-03-18T22:50:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.326+0000 7f4c4ca18740 1 -- 172.21.15.40:0/2759389742 wait complete. 2024-03-18T22:50:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.326+0000 7f4c4ca18740 1 -- start start 2024-03-18T22:50:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.326+0000 7f4c4ca18740 1 Processor -- start 2024-03-18T22:50:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.326+0000 7f4c4ca18740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 0x556ed383e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.326+0000 7f4c4ca18740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed382f440 0x556ed383e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c4ca18740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed38345b0 0x556ed383f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c4ca18740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4c441abae0 con 0x556ed38345b0 2024-03-18T22:50:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c4ca18740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4c441ab350 con 0x556ed382f440 2024-03-18T22:50:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c4ca18740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4c441a8590 con 0x556ed382ea50 2024-03-18T22:50:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c437fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 0x556ed383e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c43fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed38345b0 0x556ed383f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c42ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed382f440 0x556ed383e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c437fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 0x556ed383e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53820/0 (socket says 172.21.15.40:53820) 2024-03-18T22:50:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c437fe640 1 -- 172.21.15.40:0/3137787143 learned_addr learned my addr 172.21.15.40:0/3137787143 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c4ca18740 10 client.0 _ll_register_callbacks cb 0x7f4c3c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c437fe640 1 -- 172.21.15.40:0/3137787143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed382f440 msgr2=0x556ed383e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c437fe640 1 --2- 172.21.15.40:0/3137787143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed382f440 0x556ed383e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c437fe640 1 -- 172.21.15.40:0/3137787143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed38345b0 msgr2=0x556ed383f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c437fe640 1 --2- 172.21.15.40:0/3137787143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed38345b0 0x556ed383f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c437fe640 1 -- 172.21.15.40:0/3137787143 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ed34670f0 con 0x556ed382ea50 2024-03-18T22:50:14.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c42ffd640 1 --2- 172.21.15.40:0/3137787143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed382f440 0x556ed383e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:14.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c43fff640 1 --2- 172.21.15.40:0/3137787143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed38345b0 0x556ed383f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:14.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c437fe640 1 --2- 172.21.15.40:0/3137787143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 0x556ed383e000 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4c340cc8c0 tx=0x7f4c34003e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b54325f5c1dbb61 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:14.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:50:14.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c22ffd640 1 -- 172.21.15.40:0/3137787143 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c340b3020 con 0x556ed382ea50 2024-03-18T22:50:14.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c22ffd640 1 -- 172.21.15.40:0/3137787143 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4c34099a10 con 0x556ed382ea50 2024-03-18T22:50:14.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.327+0000 7f4c4ca18740 1 -- 172.21.15.40:0/3137787143 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c3802cb80 con 0x556ed382ea50 2024-03-18T22:50:14.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.328+0000 7f4c22ffd640 1 -- 172.21.15.40:0/3137787143 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4c3409be10 con 0x556ed382ea50 2024-03-18T22:50:14.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.328+0000 7f4c4ca18740 10 client.27280 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:14.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.328+0000 7f4c4ca18740 1 -- 172.21.15.40:0/3137787143 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x556ed38af560 con 0x556ed382ea50 2024-03-18T22:50:14.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.328+0000 7f4c22ffd640 1 -- 172.21.15.40:0/3137787143 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c3802cb80 con 0x556ed382ea50 2024-03-18T22:50:14.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.328+0000 7f4c22ffd640 1 -- 172.21.15.40:0/3137787143 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f4c340a3a20 con 0x556ed382ea50 2024-03-18T22:50:14.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.328+0000 7f4c4ca18740 10 client.27280 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:14.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.328+0000 7f4c4ca18740 10 client.27280 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:14.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.328+0000 7f4c4ca18740 20 client.27280 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:14.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.328+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/3137787143 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed38afee0 0x556ed38b2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.328+0000 7f4c4ca18740 4 client.27280 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:50:14.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.328+0000 7f4c4ca18740 1 -- 172.21.15.40:0/3137787143 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556ed379ad90 con 0x556ed38afee0 2024-03-18T22:50:14.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.328+0000 7f4c42ffd640 1 --2- 172.21.15.40:0/3137787143 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed38afee0 0x556ed38b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.328+0000 7f4c22ffd640 10 client.27280 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:14.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.328+0000 7f4c42ffd640 1 --2- 172.21.15.40:0/3137787143 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed38afee0 0x556ed38b2300 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13b6760411227f07 server_cookie=5538651cfe0c6f65 in_seq=0 out_seq=0 2024-03-18T22:50:14.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.329+0000 7f4c22ffd640 1 -- 172.21.15.40:0/3137787143 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556ed379ad90 con 0x556ed38afee0 2024-03-18T22:50:14.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.329+0000 7f4c22ffd640 10 client.27280 handle_command_reply: tid=0 2024-03-18T22:50:14.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.329+0000 7f4c4ca18740 1 client.27280 shutdown 2024-03-18T22:50:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.329+0000 7f4c4ca18740 1 -- 172.21.15.40:0/3137787143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 msgr2=0x556ed383e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.329+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/3137787143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 0x556ed383e000 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4c340cc8c0 tx=0x7f4c34003e30 comp rx=0 tx=0).stop 2024-03-18T22:50:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.329+0000 7f4c4ca18740 1 -- 172.21.15.40:0/3137787143 shutdown_connections 2024-03-18T22:50:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.329+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/3137787143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed38345b0 0x556ed383f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.329+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/3137787143 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed38afee0 0x556ed38b2300 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.329+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/3137787143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed382f440 0x556ed383e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.329+0000 7f4c4ca18740 1 --2- 172.21.15.40:0/3137787143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed382ea50 0x556ed383e000 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.329+0000 7f4c22ffd640 0 client.27280 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.329+0000 7f4c4ca18740 1 -- 172.21.15.40:0/3137787143 >> 172.21.15.40:0/3137787143 conn(0x556ed3829fc0 msgr2=0x556ed382dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.329+0000 7f4c4ca18740 1 -- 172.21.15.40:0/3137787143 shutdown_connections 2024-03-18T22:50:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.329+0000 7f4c4ca18740 1 -- 172.21.15.40:0/3137787143 wait complete. 2024-03-18T22:50:14.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.329+0000 7f4c4ca18740 20 client.27280 trim_cache size 0 max 16384 2024-03-18T22:50:14.337 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:50:14.337 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:50:14.337 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:50:14.337 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:50:14.337 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:14 smithi113 ceph-mon[66775]: pgmap v578: 129 pgs: 129 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:50:14.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.330+0000 7f4c49e31640 1 -- 172.21.15.40:0/40148730 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4c2c0a2880 msgr2=0x7f4c2c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.330+0000 7f4c49e31640 1 --2- 172.21.15.40:0/40148730 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4c2c0a2880 0x7f4c2c0a4d40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c022000 tx=0x7f4c3c023000 comp rx=0 tx=0).stop 2024-03-18T22:50:14.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.330+0000 7f4c49e31640 1 -- 172.21.15.40:0/40148730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c44100720 msgr2=0x7f4c440694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.330+0000 7f4c49e31640 1 --2- 172.21.15.40:0/40148730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c44100720 0x7f4c440694f0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f4c3400e9c0 tx=0x7f4c3400ee80 comp rx=0 tx=0).stop 2024-03-18T22:50:14.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.330+0000 7f4c49e31640 1 -- 172.21.15.40:0/40148730 shutdown_connections 2024-03-18T22:50:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.330+0000 7f4c49e31640 1 --2- 172.21.15.40:0/40148730 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4c2c0a2880 0x7f4c2c0a4d40 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.330+0000 7f4c49e31640 1 --2- 172.21.15.40:0/40148730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c441022d0 0x7f4c4419ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.330+0000 7f4c49e31640 1 --2- 172.21.15.40:0/40148730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c44101910 0x7f4c4419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.330+0000 7f4c49e31640 1 --2- 172.21.15.40:0/40148730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c44100720 0x7f4c440694f0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.330+0000 7f4c49e31640 1 -- 172.21.15.40:0/40148730 >> 172.21.15.40:0/40148730 conn(0x7f4c440fbcb0 msgr2=0x7f4c44104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.330+0000 7f4c49e31640 1 -- 172.21.15.40:0/40148730 shutdown_connections 2024-03-18T22:50:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.330+0000 7f4c49e31640 1 -- 172.21.15.40:0/40148730 wait complete. 2024-03-18T22:50:14.342 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:14 smithi040 ceph-mon[62643]: pgmap v578: 129 pgs: 129 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:50:14.343 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:50:14.343 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:50:14.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:14 smithi103 ceph-mon[66377]: pgmap v578: 129 pgs: 129 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:50:14.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.457+0000 7f244d8c6640 1 -- 172.21.15.40:0/238758106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2448101860 msgr2=0x7f2448101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.457+0000 7f244d8c6640 1 --2- 172.21.15.40:0/238758106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2448101860 0x7f2448101c80 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2438009900 tx=0x7f243802bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:14.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.457+0000 7f244d8c6640 1 -- 172.21.15.40:0/238758106 shutdown_connections 2024-03-18T22:50:14.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.457+0000 7f244d8c6640 1 --2- 172.21.15.40:0/238758106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2448103410 0x7f2448109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.457+0000 7f244d8c6640 1 --2- 172.21.15.40:0/238758106 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2448102a50 0x7f2448102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.457+0000 7f244d8c6640 1 --2- 172.21.15.40:0/238758106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2448101860 0x7f2448101c80 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.457+0000 7f244d8c6640 1 -- 172.21.15.40:0/238758106 >> 172.21.15.40:0/238758106 conn(0x7f24480fcdd0 msgr2=0x7f24480ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:14.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.457+0000 7f244d8c6640 1 -- 172.21.15.40:0/238758106 shutdown_connections 2024-03-18T22:50:14.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.457+0000 7f244d8c6640 1 -- 172.21.15.40:0/238758106 wait complete. 2024-03-18T22:50:14.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.458+0000 7f244d8c6640 1 Processor -- start 2024-03-18T22:50:14.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.458+0000 7f244d8c6640 1 -- start start 2024-03-18T22:50:14.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.458+0000 7f244d8c6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2448101860 0x7f2448197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.458+0000 7f244d8c6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2448102a50 0x7f2448198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.458+0000 7f244d8c6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2448103410 0x7f244819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.458+0000 7f244d8c6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f244810d380 con 0x7f2448101860 2024-03-18T22:50:14.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.458+0000 7f244d8c6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f244810d200 con 0x7f2448102a50 2024-03-18T22:50:14.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.458+0000 7f244d8c6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f244810d500 con 0x7f2448103410 2024-03-18T22:50:14.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.458+0000 7f2446ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2448101860 0x7f2448197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.458+0000 7f2446ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2448101860 0x7f2448197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45748/0 (socket says 172.21.15.40:45748) 2024-03-18T22:50:14.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.458+0000 7f2446ffd640 1 -- 172.21.15.40:0/690563411 learned_addr learned my addr 172.21.15.40:0/690563411 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:14.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.459+0000 7f24477fe640 1 --2- 172.21.15.40:0/690563411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2448103410 0x7f244819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.459+0000 7f243e7fc640 1 --2- 172.21.15.40:0/690563411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2448102a50 0x7f2448198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.459+0000 7f2446ffd640 1 -- 172.21.15.40:0/690563411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2448103410 msgr2=0x7f244819f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.459+0000 7f2446ffd640 1 --2- 172.21.15.40:0/690563411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2448103410 0x7f244819f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.459+0000 7f2446ffd640 1 -- 172.21.15.40:0/690563411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2448102a50 msgr2=0x7f2448198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.459+0000 7f2446ffd640 1 --2- 172.21.15.40:0/690563411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2448102a50 0x7f2448198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.459+0000 7f2446ffd640 1 -- 172.21.15.40:0/690563411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244819f760 con 0x7f2448101860 2024-03-18T22:50:14.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.459+0000 7f24477fe640 1 --2- 172.21.15.40:0/690563411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2448103410 0x7f244819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:14.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.459+0000 7f2446ffd640 1 --2- 172.21.15.40:0/690563411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2448101860 0x7f2448197b40 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2438000c00 tx=0x7f2438005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:14.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.459+0000 7f244c8c4640 1 -- 172.21.15.40:0/690563411 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438043020 con 0x7f2448101860 2024-03-18T22:50:14.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.459+0000 7f244d8c6640 1 -- 172.21.15.40:0/690563411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f244819f9f0 con 0x7f2448101860 2024-03-18T22:50:14.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.459+0000 7f244c8c4640 1 -- 172.21.15.40:0/690563411 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2438004350 con 0x7f2448101860 2024-03-18T22:50:14.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.459+0000 7f244c8c4640 1 -- 172.21.15.40:0/690563411 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438003b10 con 0x7f2448101860 2024-03-18T22:50:14.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.459+0000 7f244d8c6640 1 -- 172.21.15.40:0/690563411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f244819ff40 con 0x7f2448101860 2024-03-18T22:50:14.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.461+0000 7f244d8c6640 1 -- 172.21.15.40:0/690563411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2448108ca0 con 0x7f2448101860 2024-03-18T22:50:14.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.462+0000 7f244c8c4640 1 -- 172.21.15.40:0/690563411 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2438046020 con 0x7f2448101860 2024-03-18T22:50:14.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.462+0000 7f244c8c4640 1 --2- 172.21.15.40:0/690563411 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f24340a2800 0x7f24340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.462+0000 7f243e7fc640 1 --2- 172.21.15.40:0/690563411 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f24340a2800 0x7f24340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.462+0000 7f244c8c4640 1 -- 172.21.15.40:0/690563411 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f24380f0fb0 con 0x7f2448101860 2024-03-18T22:50:14.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.463+0000 7f243e7fc640 1 --2- 172.21.15.40:0/690563411 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f24340a2800 0x7f24340a4cc0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2448199050 tx=0x7f2430005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:14.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.469+0000 7f244c8c4640 1 -- 172.21.15.40:0/690563411 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f243804d020 con 0x7f2448101860 2024-03-18T22:50:14.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.716+0000 7f244d8c6640 1 -- 172.21.15.40:0/690563411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2448062420 con 0x7f2448101860 2024-03-18T22:50:14.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.717+0000 7f244c8c4640 1 -- 172.21.15.40:0/690563411 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11954 (secure 0 0 0) 0x7f24380bdc60 con 0x7f2448101860 2024-03-18T22:50:14.718 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:50:14.718 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:49:24.214483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24401,24388,24473,24509,24481]},"id":1}]} 2024-03-18T22:50:14.719 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 34 2024-03-18T22:50:14.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.719+0000 7f244d8c6640 1 -- 172.21.15.40:0/690563411 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f24340a2800 msgr2=0x7f24340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.719+0000 7f244d8c6640 1 --2- 172.21.15.40:0/690563411 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f24340a2800 0x7f24340a4cc0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2448199050 tx=0x7f2430005f90 comp rx=0 tx=0).stop 2024-03-18T22:50:14.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.719+0000 7f244d8c6640 1 -- 172.21.15.40:0/690563411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2448101860 msgr2=0x7f2448197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.719+0000 7f244d8c6640 1 --2- 172.21.15.40:0/690563411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2448101860 0x7f2448197b40 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2438000c00 tx=0x7f2438005dc0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.720+0000 7f244d8c6640 1 -- 172.21.15.40:0/690563411 shutdown_connections 2024-03-18T22:50:14.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.720+0000 7f244d8c6640 1 --2- 172.21.15.40:0/690563411 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f24340a2800 0x7f24340a4cc0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.720+0000 7f244d8c6640 1 --2- 172.21.15.40:0/690563411 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2448103410 0x7f244819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.720+0000 7f244d8c6640 1 --2- 172.21.15.40:0/690563411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2448102a50 0x7f2448198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.720+0000 7f244d8c6640 1 --2- 172.21.15.40:0/690563411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2448101860 0x7f2448197b40 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.720+0000 7f244d8c6640 1 -- 172.21.15.40:0/690563411 >> 172.21.15.40:0/690563411 conn(0x7f24480fcdd0 msgr2=0x7f2448105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:14.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.720+0000 7f244d8c6640 1 -- 172.21.15.40:0/690563411 shutdown_connections 2024-03-18T22:50:14.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.720+0000 7f244d8c6640 1 -- 172.21.15.40:0/690563411 wait complete. 2024-03-18T22:50:14.732 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:50:14.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.843+0000 7f0108153640 1 -- 172.21.15.40:0/1805148990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0100101de0 msgr2=0x7f0100102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.843+0000 7f0108153640 1 --2- 172.21.15.40:0/1805148990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0100101de0 0x7f0100102200 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f00fc009900 tx=0x7f00fc02bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.843+0000 7f0108153640 1 -- 172.21.15.40:0/1805148990 shutdown_connections 2024-03-18T22:50:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.843+0000 7f0108153640 1 --2- 172.21.15.40:0/1805148990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0100101360 0x7f0100108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.843+0000 7f0108153640 1 --2- 172.21.15.40:0/1805148990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f01001009a0 0x7f0100100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.843+0000 7f0108153640 1 --2- 172.21.15.40:0/1805148990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0100101de0 0x7f0100102200 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.843+0000 7f0108153640 1 -- 172.21.15.40:0/1805148990 >> 172.21.15.40:0/1805148990 conn(0x7f01000fb430 msgr2=0x7f01000fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.843+0000 7f0108153640 1 -- 172.21.15.40:0/1805148990 shutdown_connections 2024-03-18T22:50:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.843+0000 7f0108153640 1 -- 172.21.15.40:0/1805148990 wait complete. 2024-03-18T22:50:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.844+0000 7f0108153640 1 Processor -- start 2024-03-18T22:50:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.844+0000 7f0108153640 1 -- start start 2024-03-18T22:50:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.844+0000 7f0108153640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01001009a0 0x7f0100197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.844+0000 7f0108153640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0100101360 0x7f0100197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.844+0000 7f0108153640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0100101de0 0x7f010019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.844+0000 7f0108153640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f010010c230 con 0x7f01001009a0 2024-03-18T22:50:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.844+0000 7f0108153640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f010010c0b0 con 0x7f0100101360 2024-03-18T22:50:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.844+0000 7f0108153640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f010010c3b0 con 0x7f0100101de0 2024-03-18T22:50:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.844+0000 7f0105eef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01001009a0 0x7f0100197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.844+0000 7f0105eef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01001009a0 0x7f0100197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45754/0 (socket says 172.21.15.40:45754) 2024-03-18T22:50:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.844+0000 7f0105eef640 1 -- 172.21.15.40:0/1953676600 learned_addr learned my addr 172.21.15.40:0/1953676600 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.844+0000 7f01056ee640 1 --2- 172.21.15.40:0/1953676600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0100101360 0x7f0100197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.844+0000 7f01066f0640 1 --2- 172.21.15.40:0/1953676600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0100101de0 0x7f010019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.845+0000 7f0105eef640 1 -- 172.21.15.40:0/1953676600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0100101de0 msgr2=0x7f010019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.845+0000 7f0105eef640 1 --2- 172.21.15.40:0/1953676600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0100101de0 0x7f010019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.845+0000 7f0105eef640 1 -- 172.21.15.40:0/1953676600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0100101360 msgr2=0x7f0100197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.845+0000 7f0105eef640 1 --2- 172.21.15.40:0/1953676600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0100101360 0x7f0100197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.845+0000 7f0105eef640 1 -- 172.21.15.40:0/1953676600 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f010019f6a0 con 0x7f01001009a0 2024-03-18T22:50:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.845+0000 7f0105eef640 1 --2- 172.21.15.40:0/1953676600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01001009a0 0x7f0100197a80 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f00fc009a20 tx=0x7f00fc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.845+0000 7f00f6ffd640 1 -- 172.21.15.40:0/1953676600 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc043020 con 0x7f01001009a0 2024-03-18T22:50:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.845+0000 7f0108153640 1 -- 172.21.15.40:0/1953676600 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f010019f930 con 0x7f01001009a0 2024-03-18T22:50:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.845+0000 7f00f6ffd640 1 -- 172.21.15.40:0/1953676600 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00fc004440 con 0x7f01001009a0 2024-03-18T22:50:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.845+0000 7f00f6ffd640 1 -- 172.21.15.40:0/1953676600 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc003b30 con 0x7f01001009a0 2024-03-18T22:50:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.845+0000 7f0108153640 1 -- 172.21.15.40:0/1953676600 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f010019fed0 con 0x7f01001009a0 2024-03-18T22:50:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.847+0000 7f010ae19740 1 Processor -- start 2024-03-18T22:50:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.847+0000 7f00f6ffd640 1 -- 172.21.15.40:0/1953676600 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f00fc046020 con 0x7f01001009a0 2024-03-18T22:50:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f010ae19740 1 -- start start 2024-03-18T22:50:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f010ae19740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6b2f0 0x560517e6b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f010ae19740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e6bce0 0x560517e707e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f010ae19740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e70e50 0x560517e732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f010ae19740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560517dcf4b0 con 0x560517e6b2f0 2024-03-18T22:50:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f010ae19740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560517d77d70 con 0x560517e6bce0 2024-03-18T22:50:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f010ae19740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560517dcb550 con 0x560517e70e50 2024-03-18T22:50:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f01066f0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6b2f0 0x560517e6b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f0105eef640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e6bce0 0x560517e707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f0105eef640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e6bce0 0x560517e707e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43610/0 (socket says 172.21.15.40:43610) 2024-03-18T22:50:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f01056ee640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e70e50 0x560517e732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f0105eef640 1 -- 172.21.15.40:0/3026326412 learned_addr learned my addr 172.21.15.40:0/3026326412 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f01066f0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6b2f0 0x560517e6b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45762/0 (socket says 172.21.15.40:45762) 2024-03-18T22:50:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f0105eef640 1 -- 172.21.15.40:0/3026326412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e70e50 msgr2=0x560517e732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f0105eef640 1 --2- 172.21.15.40:0/3026326412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e70e50 0x560517e732e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f0105eef640 1 -- 172.21.15.40:0/3026326412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6b2f0 msgr2=0x560517e6b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f0105eef640 1 --2- 172.21.15.40:0/3026326412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6b2f0 0x560517e6b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f0105eef640 1 -- 172.21.15.40:0/3026326412 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560517d9a210 con 0x560517e6bce0 2024-03-18T22:50:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.848+0000 7f00f6ffd640 1 --2- 172.21.15.40:0/1953676600 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00ec0a2800 0x7f00ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.849+0000 7f01056ee640 1 --2- 172.21.15.40:0/1953676600 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00ec0a2800 0x7f00ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.849+0000 7f0105eef640 1 --2- 172.21.15.40:0/3026326412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e6bce0 0x560517e707e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f00fc0f4f80 tx=0x7f00fc0b9850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d3448c1925a68d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.849+0000 7f00f6ffd640 1 -- 172.21.15.40:0/1953676600 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f00fc0f1000 con 0x7f01001009a0 2024-03-18T22:50:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.849+0000 7f00ebfff640 1 -- 172.21.15.40:0/3026326412 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc0bd380 con 0x560517e6bce0 2024-03-18T22:50:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.849+0000 7f00ebfff640 1 -- 172.21.15.40:0/3026326412 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00fc0bdcb0 con 0x560517e6bce0 2024-03-18T22:50:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.849+0000 7f00ebfff640 1 -- 172.21.15.40:0/3026326412 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc0bcf10 con 0x560517e6bce0 2024-03-18T22:50:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.850+0000 7f010ae19740 1 -- 172.21.15.40:0/3026326412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e6bce0 msgr2=0x560517e707e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.850+0000 7f01056ee640 1 --2- 172.21.15.40:0/1953676600 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00ec0a2800 0x7f00ec0a4cc0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0100198f90 tx=0x7f00f0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.850+0000 7f010ae19740 1 --2- 172.21.15.40:0/3026326412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e6bce0 0x560517e707e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f00fc0f4f80 tx=0x7f00fc0b9850 comp rx=0 tx=0).stop 2024-03-18T22:50:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.850+0000 7f010ae19740 1 -- 172.21.15.40:0/3026326412 shutdown_connections 2024-03-18T22:50:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.850+0000 7f010ae19740 1 --2- 172.21.15.40:0/3026326412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e70e50 0x560517e732e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.850+0000 7f010ae19740 1 --2- 172.21.15.40:0/3026326412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e6bce0 0x560517e707e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.850+0000 7f010ae19740 1 --2- 172.21.15.40:0/3026326412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6b2f0 0x560517e6b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.850+0000 7f010ae19740 1 -- 172.21.15.40:0/3026326412 >> 172.21.15.40:0/3026326412 conn(0x560517e66860 msgr2=0x560517e66c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.850+0000 7f010ae19740 1 -- 172.21.15.40:0/3026326412 shutdown_connections 2024-03-18T22:50:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.850+0000 7f010ae19740 1 -- 172.21.15.40:0/3026326412 wait complete. 2024-03-18T22:50:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.850+0000 7f010ae19740 1 -- start start 2024-03-18T22:50:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.850+0000 7f010ae19740 1 Processor -- start 2024-03-18T22:50:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f010ae19740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e6b2f0 0x560517e75520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f010ae19740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6bce0 0x560517e75a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f010ae19740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e70e50 0x560517e7caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f010ae19740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560517e73bc0 con 0x560517e6bce0 2024-03-18T22:50:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f010ae19740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560517d8b310 con 0x560517e70e50 2024-03-18T22:50:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f010ae19740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560517e73d40 con 0x560517e6b2f0 2024-03-18T22:50:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f010ae19740 10 client.0 _ll_register_callbacks cb 0x560517d7f2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f01056ee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6bce0 0x560517e75a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f01056ee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6bce0 0x560517e75a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45776/0 (socket says 172.21.15.40:45776) 2024-03-18T22:50:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f01056ee640 1 -- 172.21.15.40:0/633379538 learned_addr learned my addr 172.21.15.40:0/633379538 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f01066f0640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e70e50 0x560517e7caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f0105eef640 1 --2- 172.21.15.40:0/633379538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e6b2f0 0x560517e75520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f01056ee640 1 -- 172.21.15.40:0/633379538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e6b2f0 msgr2=0x560517e75520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f01056ee640 1 --2- 172.21.15.40:0/633379538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e6b2f0 0x560517e75520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f01056ee640 1 -- 172.21.15.40:0/633379538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e70e50 msgr2=0x560517e7caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f01056ee640 1 --2- 172.21.15.40:0/633379538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e70e50 0x560517e7caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f01056ee640 1 -- 172.21.15.40:0/633379538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560517ae33f0 con 0x560517e6bce0 2024-03-18T22:50:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.851+0000 7f01066f0640 1 --2- 172.21.15.40:0/633379538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e70e50 0x560517e7caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.852+0000 7f01056ee640 1 --2- 172.21.15.40:0/633379538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6bce0 0x560517e75a60 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f00f0013010 tx=0x7f00f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=462ee649c9677c55 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.852+0000 7f00e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.852+0000 7f00e97fa640 1 -- 172.21.15.40:0/633379538 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f001e680 con 0x560517e6bce0 2024-03-18T22:50:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.852+0000 7f00e97fa640 1 -- 172.21.15.40:0/633379538 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f00f001e820 con 0x560517e6bce0 2024-03-18T22:50:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.852+0000 7f010ae19740 1 -- 172.21.15.40:0/633379538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00fc0bdcb0 con 0x560517e6bce0 2024-03-18T22:50:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.852+0000 7f00e97fa640 1 -- 172.21.15.40:0/633379538 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f00f001ea20 con 0x560517e6bce0 2024-03-18T22:50:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.852+0000 7f010ae19740 10 client.17490 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.852+0000 7f010ae19740 1 -- 172.21.15.40:0/633379538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560517bad7d0 con 0x560517e6bce0 2024-03-18T22:50:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.852+0000 7f00e97fa640 1 -- 172.21.15.40:0/633379538 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00fc0bdcb0 con 0x560517e6bce0 2024-03-18T22:50:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.852+0000 7f00e97fa640 1 -- 172.21.15.40:0/633379538 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f00f0027070 con 0x560517e6bce0 2024-03-18T22:50:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.852+0000 7f010ae19740 10 client.17490 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.852+0000 7f010ae19740 10 client.17490 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:50:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.852+0000 7f010ae19740 20 client.17490 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.853+0000 7f010ae19740 1 --2- 172.21.15.40:0/633379538 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560517e89d50 0x560517e8c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.853+0000 7f010ae19740 4 client.17490 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.853+0000 7f010ae19740 1 -- 172.21.15.40:0/633379538 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560517e2d3e0 con 0x560517e89d50 2024-03-18T22:50:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.853+0000 7f0105eef640 1 --2- 172.21.15.40:0/633379538 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560517e89d50 0x560517e8c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.853+0000 7f00e97fa640 10 client.17490 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:50:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.853+0000 7f0105eef640 1 --2- 172.21.15.40:0/633379538 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560517e89d50 0x560517e8c1b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=762e5820607fdc5d server_cookie=6e35fad57af77b9 in_seq=0 out_seq=0 2024-03-18T22:50:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.854+0000 7f00e97fa640 1 -- 172.21.15.40:0/633379538 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560517e2d3e0 con 0x560517e89d50 2024-03-18T22:50:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.854+0000 7f00e97fa640 10 client.17490 handle_command_reply: tid=0 2024-03-18T22:50:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.854+0000 7f010ae19740 1 client.17490 shutdown 2024-03-18T22:50:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.854+0000 7f010ae19740 1 -- 172.21.15.40:0/633379538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6bce0 msgr2=0x560517e75a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.854+0000 7f010ae19740 1 --2- 172.21.15.40:0/633379538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6bce0 0x560517e75a60 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f00f0013010 tx=0x7f00f0004730 comp rx=0 tx=0).stop 2024-03-18T22:50:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.854+0000 7f010ae19740 1 -- 172.21.15.40:0/633379538 shutdown_connections 2024-03-18T22:50:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.854+0000 7f010ae19740 1 --2- 172.21.15.40:0/633379538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e70e50 0x560517e7caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.854+0000 7f010ae19740 1 --2- 172.21.15.40:0/633379538 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560517e89d50 0x560517e8c1b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.854+0000 7f010ae19740 1 --2- 172.21.15.40:0/633379538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6bce0 0x560517e75a60 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.854+0000 7f010ae19740 1 --2- 172.21.15.40:0/633379538 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e6b2f0 0x560517e75520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.854+0000 7f00e97fa640 0 client.17490 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:50:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.854+0000 7f010ae19740 1 -- 172.21.15.40:0/633379538 >> 172.21.15.40:0/633379538 conn(0x560517e66860 msgr2=0x560517e65ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.854+0000 7f010ae19740 1 -- 172.21.15.40:0/633379538 shutdown_connections 2024-03-18T22:50:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.854+0000 7f010ae19740 1 -- 172.21.15.40:0/633379538 wait complete. 2024-03-18T22:50:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.854+0000 7f010ae19740 20 client.17490 trim_cache size 0 max 16384 2024-03-18T22:50:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.870+0000 7f010ae19740 1 Processor -- start 2024-03-18T22:50:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.870+0000 7f010ae19740 1 -- start start 2024-03-18T22:50:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.870+0000 7f010ae19740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6b2f0 0x560517e6a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.870+0000 7f010ae19740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e6bce0 0x560517e7ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.870+0000 7f010ae19740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e70e50 0x560517e90ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f010ae19740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f01001a8400 con 0x560517e6b2f0 2024-03-18T22:50:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f010ae19740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f01001aa000 con 0x560517e70e50 2024-03-18T22:50:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f010ae19740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f01001a8ad0 con 0x560517e6bce0 2024-03-18T22:50:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f0105eef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6b2f0 0x560517e6a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f0105eef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6b2f0 0x560517e6a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45782/0 (socket says 172.21.15.40:45782) 2024-03-18T22:50:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f0105eef640 1 -- 172.21.15.40:0/1745630126 learned_addr learned my addr 172.21.15.40:0/1745630126 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f01056ee640 1 --2- 172.21.15.40:0/1745630126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e6bce0 0x560517e7ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f01066f0640 1 --2- 172.21.15.40:0/1745630126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e70e50 0x560517e90ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f0105eef640 1 -- 172.21.15.40:0/1745630126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e6bce0 msgr2=0x560517e7ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f0105eef640 1 --2- 172.21.15.40:0/1745630126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e6bce0 0x560517e7ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f0105eef640 1 -- 172.21.15.40:0/1745630126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e70e50 msgr2=0x560517e90ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f0105eef640 1 --2- 172.21.15.40:0/1745630126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e70e50 0x560517e90ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f0105eef640 1 -- 172.21.15.40:0/1745630126 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560517e6c0c0 con 0x560517e6b2f0 2024-03-18T22:50:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f01056ee640 1 --2- 172.21.15.40:0/1745630126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e6bce0 0x560517e7ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f01066f0640 1 --2- 172.21.15.40:0/1745630126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e70e50 0x560517e90ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f0105eef640 1 --2- 172.21.15.40:0/1745630126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6b2f0 0x560517e6a1d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f00fc0b9a10 tx=0x7f00fc0c0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f520c950a3a78f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f00eb7fe640 1 -- 172.21.15.40:0/1745630126 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc0bd0f0 con 0x560517e6b2f0 2024-03-18T22:50:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f00eb7fe640 1 -- 172.21.15.40:0/1745630126 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00fc0bdcb0 con 0x560517e6b2f0 2024-03-18T22:50:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.871+0000 7f00eb7fe640 1 -- 172.21.15.40:0/1745630126 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc0bd5b0 con 0x560517e6b2f0 2024-03-18T22:50:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.872+0000 7f010ae19740 1 -- 172.21.15.40:0/1745630126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6b2f0 msgr2=0x560517e6a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.872+0000 7f010ae19740 1 --2- 172.21.15.40:0/1745630126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6b2f0 0x560517e6a1d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f00fc0b9a10 tx=0x7f00fc0c0040 comp rx=0 tx=0).stop 2024-03-18T22:50:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.872+0000 7f010ae19740 1 -- 172.21.15.40:0/1745630126 shutdown_connections 2024-03-18T22:50:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.872+0000 7f010ae19740 1 --2- 172.21.15.40:0/1745630126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e70e50 0x560517e90ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.872+0000 7f010ae19740 1 --2- 172.21.15.40:0/1745630126 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e6bce0 0x560517e7ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.872+0000 7f010ae19740 1 --2- 172.21.15.40:0/1745630126 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6b2f0 0x560517e6a1d0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.872+0000 7f010ae19740 1 -- 172.21.15.40:0/1745630126 >> 172.21.15.40:0/1745630126 conn(0x560517e66860 msgr2=0x560517e69ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.872+0000 7f010ae19740 1 -- 172.21.15.40:0/1745630126 shutdown_connections 2024-03-18T22:50:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.872+0000 7f010ae19740 1 -- 172.21.15.40:0/1745630126 wait complete. 2024-03-18T22:50:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.873+0000 7f010ae19740 1 -- start start 2024-03-18T22:50:14.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.873+0000 7f010ae19740 1 Processor -- start 2024-03-18T22:50:14.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.873+0000 7f010ae19740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e6b2f0 0x560517e7a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.873+0000 7f010ae19740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6bce0 0x560517e7a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.873+0000 7f010ae19740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e70e50 0x560517e7b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.873+0000 7f010ae19740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f01001a9930 con 0x560517e6bce0 2024-03-18T22:50:14.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.873+0000 7f010ae19740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f01001a91a0 con 0x560517e6b2f0 2024-03-18T22:50:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.873+0000 7f010ae19740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f01001a63e0 con 0x560517e70e50 2024-03-18T22:50:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.873+0000 7f010ae19740 10 client.0 _ll_register_callbacks cb 0x7f00f0009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.873+0000 7f01056ee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6bce0 0x560517e7a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.873+0000 7f01056ee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6bce0 0x560517e7a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45784/0 (socket says 172.21.15.40:45784) 2024-03-18T22:50:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.873+0000 7f01056ee640 1 -- 172.21.15.40:0/3384162462 learned_addr learned my addr 172.21.15.40:0/3384162462 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.873+0000 7f0105eef640 1 --2- 172.21.15.40:0/3384162462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e6b2f0 0x560517e7a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.873+0000 7f01066f0640 1 --2- 172.21.15.40:0/3384162462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e70e50 0x560517e7b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.874+0000 7f01056ee640 1 -- 172.21.15.40:0/3384162462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e70e50 msgr2=0x560517e7b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.874+0000 7f01056ee640 1 --2- 172.21.15.40:0/3384162462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e70e50 0x560517e7b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.874+0000 7f01056ee640 1 -- 172.21.15.40:0/3384162462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e6b2f0 msgr2=0x560517e7a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.874+0000 7f01056ee640 1 --2- 172.21.15.40:0/3384162462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e6b2f0 0x560517e7a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.874+0000 7f01056ee640 1 -- 172.21.15.40:0/3384162462 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560517e7bf80 con 0x560517e6bce0 2024-03-18T22:50:14.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.874+0000 7f01056ee640 1 --2- 172.21.15.40:0/3384162462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6bce0 0x560517e7a980 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f00f0013010 tx=0x7f00f001e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbaa96f5ca92f6ec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.874+0000 7f00e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.874+0000 7f00e97fa640 1 -- 172.21.15.40:0/3384162462 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f003b020 con 0x560517e6bce0 2024-03-18T22:50:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.874+0000 7f00e97fa640 1 -- 172.21.15.40:0/3384162462 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f00f001edd0 con 0x560517e6bce0 2024-03-18T22:50:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.874+0000 7f010ae19740 1 -- 172.21.15.40:0/3384162462 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00fc0bdcb0 con 0x560517e6bce0 2024-03-18T22:50:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.874+0000 7f00e97fa640 1 -- 172.21.15.40:0/3384162462 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f00f00043d0 con 0x560517e6bce0 2024-03-18T22:50:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.874+0000 7f010ae19740 10 client.17496 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.874+0000 7f010ae19740 1 -- 172.21.15.40:0/3384162462 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560517ef0480 con 0x560517e6bce0 2024-03-18T22:50:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.874+0000 7f00e97fa640 1 -- 172.21.15.40:0/3384162462 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00fc0bdcb0 con 0x560517e6bce0 2024-03-18T22:50:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.875+0000 7f00e97fa640 1 -- 172.21.15.40:0/3384162462 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f00f0027040 con 0x560517e6bce0 2024-03-18T22:50:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.875+0000 7f010ae19740 10 client.17496 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.875+0000 7f010ae19740 10 client.17496 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:50:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.875+0000 7f010ae19740 20 client.17496 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.875+0000 7f010ae19740 1 --2- 172.21.15.40:0/3384162462 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560517ef0f30 0x560517ef3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.875+0000 7f010ae19740 4 client.17496 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:50:14.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.875+0000 7f010ae19740 1 -- 172.21.15.40:0/3384162462 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560517ddc9f0 con 0x560517ef0f30 2024-03-18T22:50:14.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.875+0000 7f0105eef640 1 --2- 172.21.15.40:0/3384162462 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560517ef0f30 0x560517ef3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:14.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.875+0000 7f00e97fa640 10 client.17496 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:50:14.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.875+0000 7f0105eef640 1 --2- 172.21.15.40:0/3384162462 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560517ef0f30 0x560517ef3350 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8e6211a65763ab39 server_cookie=553f9630ae3ddb35 in_seq=0 out_seq=0 2024-03-18T22:50:14.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.875+0000 7f00e97fa640 1 -- 172.21.15.40:0/3384162462 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560517ddc9f0 con 0x560517ef0f30 2024-03-18T22:50:14.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.875+0000 7f00e97fa640 10 client.17496 handle_command_reply: tid=0 2024-03-18T22:50:14.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.875+0000 7f010ae19740 1 client.17496 shutdown 2024-03-18T22:50:14.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.875+0000 7f010ae19740 1 -- 172.21.15.40:0/3384162462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6bce0 msgr2=0x560517e7a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.875+0000 7f010ae19740 1 --2- 172.21.15.40:0/3384162462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6bce0 0x560517e7a980 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f00f0013010 tx=0x7f00f001e8b0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f010ae19740 1 -- 172.21.15.40:0/3384162462 shutdown_connections 2024-03-18T22:50:14.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f010ae19740 1 --2- 172.21.15.40:0/3384162462 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560517e70e50 0x560517e7b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f010ae19740 1 --2- 172.21.15.40:0/3384162462 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x560517ef0f30 0x560517ef3350 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f010ae19740 1 --2- 172.21.15.40:0/3384162462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560517e6bce0 0x560517e7a980 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f010ae19740 1 --2- 172.21.15.40:0/3384162462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560517e6b2f0 0x560517e7a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f00e97fa640 0 client.17496 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:50:14.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f010ae19740 1 -- 172.21.15.40:0/3384162462 >> 172.21.15.40:0/3384162462 conn(0x560517e66860 msgr2=0x560517e6a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:14.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f010ae19740 1 -- 172.21.15.40:0/3384162462 shutdown_connections 2024-03-18T22:50:14.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f010ae19740 1 -- 172.21.15.40:0/3384162462 wait complete. 2024-03-18T22:50:14.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f010ae19740 20 client.17496 trim_cache size 0 max 16384 2024-03-18T22:50:14.883 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:50:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f0108153640 1 -- 172.21.15.40:0/1953676600 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00ec0a2800 msgr2=0x7f00ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f0108153640 1 --2- 172.21.15.40:0/1953676600 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00ec0a2800 0x7f00ec0a4cc0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0100198f90 tx=0x7f00f0005f90 comp rx=0 tx=0).stop 2024-03-18T22:50:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f0108153640 1 -- 172.21.15.40:0/1953676600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01001009a0 msgr2=0x7f0100197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f0108153640 1 --2- 172.21.15.40:0/1953676600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01001009a0 0x7f0100197a80 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f00fc009a20 tx=0x7f00fc004040 comp rx=0 tx=0).stop 2024-03-18T22:50:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f0108153640 1 -- 172.21.15.40:0/1953676600 shutdown_connections 2024-03-18T22:50:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f0108153640 1 --2- 172.21.15.40:0/1953676600 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00ec0a2800 0x7f00ec0a4cc0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f0108153640 1 --2- 172.21.15.40:0/1953676600 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0100101de0 0x7f010019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f0108153640 1 --2- 172.21.15.40:0/1953676600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0100101360 0x7f0100197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f0108153640 1 --2- 172.21.15.40:0/1953676600 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01001009a0 0x7f0100197a80 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:14.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f0108153640 1 -- 172.21.15.40:0/1953676600 >> 172.21.15.40:0/1953676600 conn(0x7f01000fb430 msgr2=0x7f01000fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:14.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f0108153640 1 -- 172.21.15.40:0/1953676600 shutdown_connections 2024-03-18T22:50:14.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:14.876+0000 7f0108153640 1 -- 172.21.15.40:0/1953676600 wait complete. 2024-03-18T22:50:14.888 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:50:15.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.007+0000 7f4eda1d3640 1 -- 172.21.15.40:0/346383285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed40ffad0 msgr2=0x7f4ed40ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.007+0000 7f4eda1d3640 1 --2- 172.21.15.40:0/346383285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed40ffad0 0x7f4ed40ffef0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4009930 tx=0x7f4ec402bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:15.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.007+0000 7f4eda1d3640 1 -- 172.21.15.40:0/346383285 shutdown_connections 2024-03-18T22:50:15.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.007+0000 7f4eda1d3640 1 --2- 172.21.15.40:0/346383285 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ed4103460 0x7f4ed4109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.007+0000 7f4eda1d3640 1 --2- 172.21.15.40:0/346383285 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed4102aa0 0x7f4ed4102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.007+0000 7f4eda1d3640 1 --2- 172.21.15.40:0/346383285 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed40ffad0 0x7f4ed40ffef0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.007+0000 7f4eda1d3640 1 -- 172.21.15.40:0/346383285 >> 172.21.15.40:0/346383285 conn(0x7f4ed40fb400 msgr2=0x7f4ed40fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.007+0000 7f4eda1d3640 1 -- 172.21.15.40:0/346383285 shutdown_connections 2024-03-18T22:50:15.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.007+0000 7f4eda1d3640 1 -- 172.21.15.40:0/346383285 wait complete. 2024-03-18T22:50:15.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4eda1d3640 1 Processor -- start 2024-03-18T22:50:15.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4eda1d3640 1 -- start start 2024-03-18T22:50:15.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4eda1d3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ed40ffad0 0x7f4ed4197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4eda1d3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed4102aa0 0x7f4ed4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4eda1d3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed4103460 0x7f4ed419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4eda1d3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed410d350 con 0x7f4ed4103460 2024-03-18T22:50:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4eda1d3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed410d1d0 con 0x7f4ed4102aa0 2024-03-18T22:50:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4eda1d3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed410d4d0 con 0x7f4ed40ffad0 2024-03-18T22:50:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4ed3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed4103460 0x7f4ed419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4ed37fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ed40ffad0 0x7f4ed4197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4ed3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed4103460 0x7f4ed419f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45800/0 (socket says 172.21.15.40:45800) 2024-03-18T22:50:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4ed37fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ed40ffad0 0x7f4ed4197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53920/0 (socket says 172.21.15.40:53920) 2024-03-18T22:50:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4ed3fff640 1 -- 172.21.15.40:0/3956802698 learned_addr learned my addr 172.21.15.40:0/3956802698 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4ed2ffd640 1 --2- 172.21.15.40:0/3956802698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed4102aa0 0x7f4ed4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4ed37fe640 1 -- 172.21.15.40:0/3956802698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed4102aa0 msgr2=0x7f4ed4197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4ed37fe640 1 --2- 172.21.15.40:0/3956802698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed4102aa0 0x7f4ed4197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4ed37fe640 1 -- 172.21.15.40:0/3956802698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed4103460 msgr2=0x7f4ed419f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4ed37fe640 1 --2- 172.21.15.40:0/3956802698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed4103460 0x7f4ed419f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4ed37fe640 1 -- 172.21.15.40:0/3956802698 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed419f6a0 con 0x7f4ed40ffad0 2024-03-18T22:50:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.008+0000 7f4ed3fff640 1 --2- 172.21.15.40:0/3956802698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed4103460 0x7f4ed419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.009+0000 7f4ed37fe640 1 --2- 172.21.15.40:0/3956802698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ed40ffad0 0x7f4ed4197a80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4ec402c170 tx=0x7f4ec4005e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.009+0000 7f4ed0ff9640 1 -- 172.21.15.40:0/3956802698 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4043020 con 0x7f4ed40ffad0 2024-03-18T22:50:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.009+0000 7f4eda1d3640 1 -- 172.21.15.40:0/3956802698 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed419f930 con 0x7f4ed40ffad0 2024-03-18T22:50:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.009+0000 7f4ed0ff9640 1 -- 172.21.15.40:0/3956802698 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ec40043d0 con 0x7f4ed40ffad0 2024-03-18T22:50:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.009+0000 7f4ed0ff9640 1 -- 172.21.15.40:0/3956802698 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4003bf0 con 0x7f4ed40ffad0 2024-03-18T22:50:15.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.009+0000 7f4eda1d3640 1 -- 172.21.15.40:0/3956802698 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed419fea0 con 0x7f4ed40ffad0 2024-03-18T22:50:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.011+0000 7f4ed0ff9640 1 -- 172.21.15.40:0/3956802698 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4ec4046020 con 0x7f4ed40ffad0 2024-03-18T22:50:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.011+0000 7f4edcdb7740 1 Processor -- start 2024-03-18T22:50:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.011+0000 7f4edcdb7740 1 -- start start 2024-03-18T22:50:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4edcdb7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652080182f0 0x565208018710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4edcdb7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565208018ce0 0x56520801d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4edcdb7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56520801de50 0x5652080202e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4edcdb7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x565207f7c4b0 con 0x5652080182f0 2024-03-18T22:50:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4edcdb7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x565207f24d70 con 0x56520801de50 2024-03-18T22:50:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4edcdb7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x565207f78550 con 0x565208018ce0 2024-03-18T22:50:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652080182f0 0x565208018710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed37fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565208018ce0 0x56520801d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed0ff9640 1 --2- 172.21.15.40:0/3956802698 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4ebc0a2800 0x7f4ebc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652080182f0 0x565208018710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45806/0 (socket says 172.21.15.40:45806) 2024-03-18T22:50:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed37fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565208018ce0 0x56520801d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53924/0 (socket says 172.21.15.40:53924) 2024-03-18T22:50:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed3fff640 1 -- 172.21.15.40:0/2297852554 learned_addr learned my addr 172.21.15.40:0/2297852554 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed2ffd640 1 --2- 172.21.15.40:0/2297852554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56520801de50 0x5652080202e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed37fe640 1 -- 172.21.15.40:0/2297852554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56520801de50 msgr2=0x5652080202e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed37fe640 1 --2- 172.21.15.40:0/2297852554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56520801de50 0x5652080202e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed37fe640 1 -- 172.21.15.40:0/2297852554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652080182f0 msgr2=0x565208018710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed37fe640 1 --2- 172.21.15.40:0/2297852554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652080182f0 0x565208018710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed0ff9640 1 -- 172.21.15.40:0/3956802698 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4ec40f0fa0 con 0x7f4ed40ffad0 2024-03-18T22:50:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed37fe640 1 -- 172.21.15.40:0/2297852554 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565207f47210 con 0x565208018ce0 2024-03-18T22:50:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed3fff640 1 --2- 172.21.15.40:0/2297852554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652080182f0 0x565208018710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed3fff640 1 --2- 172.21.15.40:0/3956802698 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4ebc0a2800 0x7f4ebc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.012+0000 7f4ed37fe640 1 --2- 172.21.15.40:0/2297852554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565208018ce0 0x56520801d7e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4ec40ee700 tx=0x7f4ec402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb0fa9d0966fd67c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.013+0000 7f4eb9ffb640 1 -- 172.21.15.40:0/2297852554 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec40bd090 con 0x565208018ce0 2024-03-18T22:50:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.013+0000 7f4eb9ffb640 1 -- 172.21.15.40:0/2297852554 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ec40c16a0 con 0x565208018ce0 2024-03-18T22:50:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.013+0000 7f4eb9ffb640 1 -- 172.21.15.40:0/2297852554 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec40bceb0 con 0x565208018ce0 2024-03-18T22:50:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.013+0000 7f4edcdb7740 1 -- 172.21.15.40:0/2297852554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565208018ce0 msgr2=0x56520801d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.013+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/2297852554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565208018ce0 0x56520801d7e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4ec40ee700 tx=0x7f4ec402bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.013+0000 7f4ed3fff640 1 --2- 172.21.15.40:0/3956802698 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4ebc0a2800 0x7f4ebc0a4cc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x565208019540 tx=0x7f4ecc004a40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.013+0000 7f4edcdb7740 1 -- 172.21.15.40:0/2297852554 shutdown_connections 2024-03-18T22:50:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.013+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/2297852554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56520801de50 0x5652080202e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.013+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/2297852554 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x565208018ce0 0x56520801d7e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.013+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/2297852554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652080182f0 0x565208018710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.013+0000 7f4edcdb7740 1 -- 172.21.15.40:0/2297852554 >> 172.21.15.40:0/2297852554 conn(0x565208013860 msgr2=0x565208013c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.013+0000 7f4edcdb7740 1 -- 172.21.15.40:0/2297852554 shutdown_connections 2024-03-18T22:50:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.013+0000 7f4edcdb7740 1 -- 172.21.15.40:0/2297852554 wait complete. 2024-03-18T22:50:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.014+0000 7f4edcdb7740 1 -- start start 2024-03-18T22:50:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.014+0000 7f4edcdb7740 1 Processor -- start 2024-03-18T22:50:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.014+0000 7f4edcdb7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652080182f0 0x5652080268a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.014+0000 7f4edcdb7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565208018ce0 0x565208026de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.014+0000 7f4edcdb7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56520801de50 0x56520802de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.014+0000 7f4edcdb7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x565208024fd0 con 0x5652080182f0 2024-03-18T22:50:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.014+0000 7f4edcdb7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x565207f38310 con 0x565208018ce0 2024-03-18T22:50:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.014+0000 7f4edcdb7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x565208025150 con 0x56520801de50 2024-03-18T22:50:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.014+0000 7f4edcdb7740 10 client.0 _ll_register_callbacks cb 0x565207f2c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.014+0000 7f4ed37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652080182f0 0x5652080268a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.014+0000 7f4ed37fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652080182f0 0x5652080268a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45808/0 (socket says 172.21.15.40:45808) 2024-03-18T22:50:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.014+0000 7f4ed2ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565208018ce0 0x565208026de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.014+0000 7f4ed37fe640 1 -- 172.21.15.40:0/2382725277 learned_addr learned my addr 172.21.15.40:0/2382725277 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.014+0000 7f4ed3fff640 1 --2- 172.21.15.40:0/2382725277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56520801de50 0x56520802de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.015+0000 7f4ed2ffd640 1 -- 172.21.15.40:0/2382725277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56520801de50 msgr2=0x56520802de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.015+0000 7f4ed2ffd640 1 --2- 172.21.15.40:0/2382725277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56520801de50 0x56520802de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.015+0000 7f4ed2ffd640 1 -- 172.21.15.40:0/2382725277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652080182f0 msgr2=0x5652080268a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.015+0000 7f4ed2ffd640 1 --2- 172.21.15.40:0/2382725277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652080182f0 0x5652080268a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.015+0000 7f4ed2ffd640 1 -- 172.21.15.40:0/2382725277 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565207c903f0 con 0x565208018ce0 2024-03-18T22:50:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.015+0000 7f4ed3fff640 1 --2- 172.21.15.40:0/2382725277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56520801de50 0x56520802de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.015+0000 7f4ed2ffd640 1 --2- 172.21.15.40:0/2382725277 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565208018ce0 0x565208026de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4198f90 tx=0x7f4ec8012aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0418c593dc782d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.015+0000 7f4eb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:50:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.015+0000 7f4eb2ffd640 1 -- 172.21.15.40:0/2382725277 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec800f560 con 0x565208018ce0 2024-03-18T22:50:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.015+0000 7f4eb2ffd640 1 -- 172.21.15.40:0/2382725277 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4ec800f700 con 0x565208018ce0 2024-03-18T22:50:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.015+0000 7f4edcdb7740 1 -- 172.21.15.40:0/2382725277 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x565207fda3e0 con 0x565208018ce0 2024-03-18T22:50:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.015+0000 7f4eb2ffd640 1 -- 172.21.15.40:0/2382725277 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4ec800f900 con 0x565208018ce0 2024-03-18T22:50:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.015+0000 7f4edcdb7740 10 client.27323 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.015+0000 7f4edcdb7740 1 -- 172.21.15.40:0/2382725277 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x565207d5a7d0 con 0x565208018ce0 2024-03-18T22:50:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.016+0000 7f4eb2ffd640 1 -- 172.21.15.40:0/2382725277 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x565207fda3e0 con 0x565208018ce0 2024-03-18T22:50:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.016+0000 7f4eb2ffd640 1 -- 172.21.15.40:0/2382725277 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f4ec8010b50 con 0x565208018ce0 2024-03-18T22:50:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.016+0000 7f4edcdb7740 10 client.27323 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.016+0000 7f4edcdb7740 10 client.27323 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:50:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.016+0000 7f4edcdb7740 20 client.27323 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.016+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/2382725277 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x565208032790 0x565208034bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.016+0000 7f4edcdb7740 4 client.27323 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.016+0000 7f4edcdb7740 1 -- 172.21.15.40:0/2382725277 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565207f899f0 con 0x565208032790 2024-03-18T22:50:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.016+0000 7f4ed37fe640 1 --2- 172.21.15.40:0/2382725277 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x565208032790 0x565208034bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.016+0000 7f4eb2ffd640 10 client.27323 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:50:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.016+0000 7f4ed37fe640 1 --2- 172.21.15.40:0/2382725277 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x565208032790 0x565208034bc0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eacf81fe8f84920f server_cookie=a41b08b5c5481a7c in_seq=0 out_seq=0 2024-03-18T22:50:15.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.017+0000 7f4eb2ffd640 1 -- 172.21.15.40:0/2382725277 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x565207f899f0 con 0x565208032790 2024-03-18T22:50:15.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.017+0000 7f4eb2ffd640 10 client.27323 handle_command_reply: tid=0 2024-03-18T22:50:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.017+0000 7f4edcdb7740 1 client.27323 shutdown 2024-03-18T22:50:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.017+0000 7f4edcdb7740 1 -- 172.21.15.40:0/2382725277 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565208018ce0 msgr2=0x565208026de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.017+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/2382725277 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565208018ce0 0x565208026de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4ed4198f90 tx=0x7f4ec8012aa0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.017+0000 7f4edcdb7740 1 -- 172.21.15.40:0/2382725277 shutdown_connections 2024-03-18T22:50:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.017+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/2382725277 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x565208032790 0x565208034bc0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.017+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/2382725277 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56520801de50 0x56520802de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.017+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/2382725277 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565208018ce0 0x565208026de0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.017+0000 7f4eb2ffd640 0 client.27323 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:50:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.017+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/2382725277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5652080182f0 0x5652080268a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.017+0000 7f4edcdb7740 1 -- 172.21.15.40:0/2382725277 >> 172.21.15.40:0/2382725277 conn(0x565208013860 msgr2=0x565208012ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.017+0000 7f4edcdb7740 1 -- 172.21.15.40:0/2382725277 shutdown_connections 2024-03-18T22:50:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.017+0000 7f4edcdb7740 1 -- 172.21.15.40:0/2382725277 wait complete. 2024-03-18T22:50:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.018+0000 7f4edcdb7740 20 client.27323 trim_cache size 0 max 16384 2024-03-18T22:50:15.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4edcdb7740 1 Processor -- start 2024-03-18T22:50:15.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4edcdb7740 1 -- start start 2024-03-18T22:50:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4edcdb7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652080182f0 0x5652080268a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4edcdb7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565208018ce0 0x565208026de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4edcdb7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56520801de50 0x56520802dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4edcdb7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed41a8390 con 0x565208018ce0 2024-03-18T22:50:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4edcdb7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed41a9f90 con 0x5652080182f0 2024-03-18T22:50:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4edcdb7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed41a8a60 con 0x56520801de50 2024-03-18T22:50:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4ed2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565208018ce0 0x565208026de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4ed37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652080182f0 0x5652080268a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4ed2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565208018ce0 0x565208026de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45810/0 (socket says 172.21.15.40:45810) 2024-03-18T22:50:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4ed37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652080182f0 0x5652080268a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43696/0 (socket says 172.21.15.40:43696) 2024-03-18T22:50:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4ed2ffd640 1 -- 172.21.15.40:0/254743436 learned_addr learned my addr 172.21.15.40:0/254743436 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4ed3fff640 1 --2- 172.21.15.40:0/254743436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56520801de50 0x56520802dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4ed2ffd640 1 -- 172.21.15.40:0/254743436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56520801de50 msgr2=0x56520802dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4ed2ffd640 1 --2- 172.21.15.40:0/254743436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56520801de50 0x56520802dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4ed2ffd640 1 -- 172.21.15.40:0/254743436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652080182f0 msgr2=0x5652080268a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4ed2ffd640 1 --2- 172.21.15.40:0/254743436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652080182f0 0x5652080268a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4ed2ffd640 1 -- 172.21.15.40:0/254743436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56520802f2a0 con 0x565208018ce0 2024-03-18T22:50:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4ed3fff640 1 --2- 172.21.15.40:0/254743436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56520801de50 0x56520802dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.032+0000 7f4ed37fe640 1 --2- 172.21.15.40:0/254743436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652080182f0 0x5652080268a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.033+0000 7f4ed2ffd640 1 --2- 172.21.15.40:0/254743436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565208018ce0 0x565208026de0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800f770 tx=0x7f4ec800f7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606f43184f805819 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.033+0000 7f4eb97fa640 1 -- 172.21.15.40:0/254743436 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec800faa0 con 0x565208018ce0 2024-03-18T22:50:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.033+0000 7f4eb97fa640 1 -- 172.21.15.40:0/254743436 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ec8005560 con 0x565208018ce0 2024-03-18T22:50:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.033+0000 7f4eb97fa640 1 -- 172.21.15.40:0/254743436 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec801cc50 con 0x565208018ce0 2024-03-18T22:50:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.033+0000 7f4edcdb7740 1 -- 172.21.15.40:0/254743436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565208018ce0 msgr2=0x565208026de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.033+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/254743436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565208018ce0 0x565208026de0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800f770 tx=0x7f4ec800f7a0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.033+0000 7f4edcdb7740 1 -- 172.21.15.40:0/254743436 shutdown_connections 2024-03-18T22:50:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.033+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/254743436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56520801de50 0x56520802dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.033+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/254743436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x565208018ce0 0x565208026de0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.034+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/254743436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652080182f0 0x5652080268a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.034+0000 7f4edcdb7740 1 -- 172.21.15.40:0/254743436 >> 172.21.15.40:0/254743436 conn(0x565208013860 msgr2=0x565208013c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.034+0000 7f4edcdb7740 1 -- 172.21.15.40:0/254743436 shutdown_connections 2024-03-18T22:50:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.034+0000 7f4edcdb7740 1 -- 172.21.15.40:0/254743436 wait complete. 2024-03-18T22:50:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.034+0000 7f4edcdb7740 1 -- start start 2024-03-18T22:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.034+0000 7f4edcdb7740 1 Processor -- start 2024-03-18T22:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.034+0000 7f4edcdb7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5652080182f0 0x56520802b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.034+0000 7f4edcdb7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565208018ce0 0x56520802b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.034+0000 7f4edcdb7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56520801de50 0x56520802c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.034+0000 7f4edcdb7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed41a98c0 con 0x56520801de50 2024-03-18T22:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.034+0000 7f4edcdb7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed41a9130 con 0x565208018ce0 2024-03-18T22:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.034+0000 7f4edcdb7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed41a6370 con 0x5652080182f0 2024-03-18T22:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.034+0000 7f4edcdb7740 10 client.0 _ll_register_callbacks cb 0x7f4ec8009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.034+0000 7f4ed3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56520801de50 0x56520802c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4ed3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56520801de50 0x56520802c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45812/0 (socket says 172.21.15.40:45812) 2024-03-18T22:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4ed2ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565208018ce0 0x56520802b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4ed3fff640 1 -- 172.21.15.40:0/3129689673 learned_addr learned my addr 172.21.15.40:0/3129689673 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4ed37fe640 1 --2- 172.21.15.40:0/3129689673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5652080182f0 0x56520802b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4ed2ffd640 1 -- 172.21.15.40:0/3129689673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5652080182f0 msgr2=0x56520802b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4ed2ffd640 1 --2- 172.21.15.40:0/3129689673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5652080182f0 0x56520802b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4ed2ffd640 1 -- 172.21.15.40:0/3129689673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56520801de50 msgr2=0x56520802c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4ed2ffd640 1 --2- 172.21.15.40:0/3129689673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56520801de50 0x56520802c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4ed2ffd640 1 -- 172.21.15.40:0/3129689673 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56520802cd20 con 0x565208018ce0 2024-03-18T22:50:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4ed3fff640 1 --2- 172.21.15.40:0/3129689673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56520801de50 0x56520802c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4ed37fe640 1 --2- 172.21.15.40:0/3129689673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5652080182f0 0x56520802b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4ed2ffd640 1 --2- 172.21.15.40:0/3129689673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565208018ce0 0x56520802b720 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x5652080195e0 tx=0x7f4ec800fd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63cd7e6a01187742 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4eb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:50:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4eb2ffd640 1 -- 172.21.15.40:0/3129689673 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec801cb60 con 0x565208018ce0 2024-03-18T22:50:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4edcdb7740 1 -- 172.21.15.40:0/3129689673 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ec8005560 con 0x565208018ce0 2024-03-18T22:50:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4eb2ffd640 1 -- 172.21.15.40:0/3129689673 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4ec801cd00 con 0x565208018ce0 2024-03-18T22:50:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4eb2ffd640 1 -- 172.21.15.40:0/3129689673 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4ec803b050 con 0x565208018ce0 2024-03-18T22:50:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4edcdb7740 10 client.27329 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.035+0000 7f4edcdb7740 1 -- 172.21.15.40:0/3129689673 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56520802e020 con 0x565208018ce0 2024-03-18T22:50:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.036+0000 7f4eb2ffd640 1 -- 172.21.15.40:0/3129689673 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ec8005560 con 0x565208018ce0 2024-03-18T22:50:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.036+0000 7f4eb2ffd640 1 -- 172.21.15.40:0/3129689673 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f4ec8029030 con 0x565208018ce0 2024-03-18T22:50:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.036+0000 7f4edcdb7740 10 client.27329 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.036+0000 7f4edcdb7740 10 client.27329 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:50:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.036+0000 7f4edcdb7740 20 client.27329 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.036+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/3129689673 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x565208032790 0x56520809ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.036+0000 7f4edcdb7740 4 client.27329 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:50:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.036+0000 7f4edcdb7740 1 -- 172.21.15.40:0/3129689673 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x565207f83d40 con 0x565208032790 2024-03-18T22:50:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.036+0000 7f4ed37fe640 1 --2- 172.21.15.40:0/3129689673 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x565208032790 0x56520809ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.036+0000 7f4ed37fe640 1 --2- 172.21.15.40:0/3129689673 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x565208032790 0x56520809ae40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2e7b4b0a870ac383 server_cookie=4d614a082702d448 in_seq=0 out_seq=0 2024-03-18T22:50:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.036+0000 7f4eb2ffd640 10 client.27329 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:50:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.037+0000 7f4eb2ffd640 1 -- 172.21.15.40:0/3129689673 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x565207f83d40 con 0x565208032790 2024-03-18T22:50:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.037+0000 7f4eb2ffd640 10 client.27329 handle_command_reply: tid=0 2024-03-18T22:50:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.037+0000 7f4edcdb7740 1 client.27329 shutdown 2024-03-18T22:50:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.037+0000 7f4edcdb7740 1 -- 172.21.15.40:0/3129689673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565208018ce0 msgr2=0x56520802b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.037+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/3129689673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565208018ce0 0x56520802b720 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x5652080195e0 tx=0x7f4ec800fd50 comp rx=0 tx=0).stop 2024-03-18T22:50:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.037+0000 7f4edcdb7740 1 -- 172.21.15.40:0/3129689673 shutdown_connections 2024-03-18T22:50:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.037+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/3129689673 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x565208032790 0x56520809ae40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.037+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/3129689673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56520801de50 0x56520802c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.037+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/3129689673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565208018ce0 0x56520802b720 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.037+0000 7f4eb2ffd640 0 client.27329 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:50:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.037+0000 7f4edcdb7740 1 --2- 172.21.15.40:0/3129689673 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5652080182f0 0x56520802b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.037+0000 7f4edcdb7740 1 -- 172.21.15.40:0/3129689673 >> 172.21.15.40:0/3129689673 conn(0x565208013860 msgr2=0x565208026840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.037+0000 7f4edcdb7740 1 -- 172.21.15.40:0/3129689673 shutdown_connections 2024-03-18T22:50:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.037+0000 7f4edcdb7740 1 -- 172.21.15.40:0/3129689673 wait complete. 2024-03-18T22:50:15.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.037+0000 7f4edcdb7740 20 client.27329 trim_cache size 0 max 16384 2024-03-18T22:50:15.043 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:50:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.038+0000 7f4eda1d3640 1 -- 172.21.15.40:0/3956802698 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4ebc0a2800 msgr2=0x7f4ebc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.038+0000 7f4eda1d3640 1 --2- 172.21.15.40:0/3956802698 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4ebc0a2800 0x7f4ebc0a4cc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x565208019540 tx=0x7f4ecc004a40 comp rx=0 tx=0).stop 2024-03-18T22:50:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.038+0000 7f4eda1d3640 1 -- 172.21.15.40:0/3956802698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ed40ffad0 msgr2=0x7f4ed4197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.038+0000 7f4eda1d3640 1 --2- 172.21.15.40:0/3956802698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ed40ffad0 0x7f4ed4197a80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4ec402c170 tx=0x7f4ec4005e40 comp rx=0 tx=0).stop 2024-03-18T22:50:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.038+0000 7f4eda1d3640 1 -- 172.21.15.40:0/3956802698 shutdown_connections 2024-03-18T22:50:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.038+0000 7f4eda1d3640 1 --2- 172.21.15.40:0/3956802698 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4ebc0a2800 0x7f4ebc0a4cc0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.038+0000 7f4eda1d3640 1 --2- 172.21.15.40:0/3956802698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ed4103460 0x7f4ed419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.038+0000 7f4eda1d3640 1 --2- 172.21.15.40:0/3956802698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4ed4102aa0 0x7f4ed4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.038+0000 7f4eda1d3640 1 --2- 172.21.15.40:0/3956802698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4ed40ffad0 0x7f4ed4197a80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.038+0000 7f4eda1d3640 1 -- 172.21.15.40:0/3956802698 >> 172.21.15.40:0/3956802698 conn(0x7f4ed40fb400 msgr2=0x7f4ed4105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.038+0000 7f4eda1d3640 1 -- 172.21.15.40:0/3956802698 shutdown_connections 2024-03-18T22:50:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.038+0000 7f4eda1d3640 1 -- 172.21.15.40:0/3956802698 wait complete. 2024-03-18T22:50:15.051 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:50:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.171+0000 7fc7590ac640 1 -- 172.21.15.40:0/1533467512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc754100760 msgr2=0x7fc754100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.171+0000 7fc7590ac640 1 --2- 172.21.15.40:0/1533467512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc754100760 0x7fc754100b80 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fc744009ca0 tx=0x7fc74402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.171+0000 7fc7590ac640 1 -- 172.21.15.40:0/1533467512 shutdown_connections 2024-03-18T22:50:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.171+0000 7fc7590ac640 1 --2- 172.21.15.40:0/1533467512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc754102310 0x7fc754108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.171+0000 7fc7590ac640 1 --2- 172.21.15.40:0/1533467512 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc754101950 0x7fc754101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.171+0000 7fc7590ac640 1 --2- 172.21.15.40:0/1533467512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc754100760 0x7fc754100b80 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.171+0000 7fc7590ac640 1 -- 172.21.15.40:0/1533467512 >> 172.21.15.40:0/1533467512 conn(0x7fc7540fbcf0 msgr2=0x7fc7540fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.171+0000 7fc7590ac640 1 -- 172.21.15.40:0/1533467512 shutdown_connections 2024-03-18T22:50:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.171+0000 7fc7590ac640 1 -- 172.21.15.40:0/1533467512 wait complete. 2024-03-18T22:50:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.172+0000 7fc7590ac640 1 Processor -- start 2024-03-18T22:50:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.172+0000 7fc7590ac640 1 -- start start 2024-03-18T22:50:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.172+0000 7fc7590ac640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc754100760 0x7fc754197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.172+0000 7fc7590ac640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc754101950 0x7fc754198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.172+0000 7fc7590ac640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc754102310 0x7fc75419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.172+0000 7fc7590ac640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc75410c260 con 0x7fc754102310 2024-03-18T22:50:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.172+0000 7fc7590ac640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc75410c0e0 con 0x7fc754100760 2024-03-18T22:50:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.172+0000 7fc7590ac640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc75410c3e0 con 0x7fc754101950 2024-03-18T22:50:15.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.172+0000 7fc75359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc754102310 0x7fc75419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.172+0000 7fc752d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc754100760 0x7fc754197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.172+0000 7fc75359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc754102310 0x7fc75419f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45828/0 (socket says 172.21.15.40:45828) 2024-03-18T22:50:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.172+0000 7fc752d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc754100760 0x7fc754197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43728/0 (socket says 172.21.15.40:43728) 2024-03-18T22:50:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.173+0000 7fc75359e640 1 -- 172.21.15.40:0/2081266728 learned_addr learned my addr 172.21.15.40:0/2081266728 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.173+0000 7fc74a59c640 1 --2- 172.21.15.40:0/2081266728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc754101950 0x7fc754198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.173+0000 7fc752d9d640 1 -- 172.21.15.40:0/2081266728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc754101950 msgr2=0x7fc754198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.173+0000 7fc752d9d640 1 --2- 172.21.15.40:0/2081266728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc754101950 0x7fc754198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.173+0000 7fc752d9d640 1 -- 172.21.15.40:0/2081266728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc754102310 msgr2=0x7fc75419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.173+0000 7fc752d9d640 1 --2- 172.21.15.40:0/2081266728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc754102310 0x7fc75419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.173+0000 7fc752d9d640 1 -- 172.21.15.40:0/2081266728 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75419f6b0 con 0x7fc754100760 2024-03-18T22:50:15.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.173+0000 7fc75359e640 1 --2- 172.21.15.40:0/2081266728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc754102310 0x7fc75419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.173+0000 7fc74a59c640 1 --2- 172.21.15.40:0/2081266728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc754101950 0x7fc754198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.173+0000 7fc752d9d640 1 --2- 172.21.15.40:0/2081266728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc754100760 0x7fc754197ae0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc74402c630 tx=0x7fc7440328a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.173+0000 7fc750d99640 1 -- 172.21.15.40:0/2081266728 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc744041020 con 0x7fc754100760 2024-03-18T22:50:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.173+0000 7fc750d99640 1 -- 172.21.15.40:0/2081266728 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc744003e20 con 0x7fc754100760 2024-03-18T22:50:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.173+0000 7fc750d99640 1 -- 172.21.15.40:0/2081266728 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc744039040 con 0x7fc754100760 2024-03-18T22:50:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.173+0000 7fc7590ac640 1 -- 172.21.15.40:0/2081266728 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc75419f940 con 0x7fc754100760 2024-03-18T22:50:15.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.173+0000 7fc7590ac640 1 -- 172.21.15.40:0/2081266728 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc75419fe00 con 0x7fc754100760 2024-03-18T22:50:15.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.176+0000 7fc750d99640 1 -- 172.21.15.40:0/2081266728 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc744044020 con 0x7fc754100760 2024-03-18T22:50:15.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.176+0000 7fc75bd75740 1 Processor -- start 2024-03-18T22:50:15.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.176+0000 7fc75bd75740 1 -- start start 2024-03-18T22:50:15.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.176+0000 7fc750d99640 1 --2- 172.21.15.40:0/2081266728 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc7400a2800 0x7fc7400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.176+0000 7fc750d99640 1 -- 172.21.15.40:0/2081266728 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc7440f0ed0 con 0x7fc754100760 2024-03-18T22:50:15.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.176+0000 7fc75359e640 1 --2- 172.21.15.40:0/2081266728 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc7400a2800 0x7fc7400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc75bd75740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed42b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc75bd75740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed430f0 0x55cf2ed47bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc75bd75740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed48260 0x55cf2ed4a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc75bd75740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cf2eca24b0 con 0x55cf2ed42700 2024-03-18T22:50:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc75bd75740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cf2ec4ad70 con 0x55cf2ed430f0 2024-03-18T22:50:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc75bd75740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55cf2ec9e550 con 0x55cf2ed48260 2024-03-18T22:50:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc752d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed42b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc752d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed42b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45836/0 (socket says 172.21.15.40:45836) 2024-03-18T22:50:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc752d9d640 1 -- 172.21.15.40:0/1222554919 learned_addr learned my addr 172.21.15.40:0/1222554919 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc74a59c640 1 --2- 172.21.15.40:0/1222554919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed430f0 0x55cf2ed47bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc75359e640 1 --2- 172.21.15.40:0/1222554919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed48260 0x55cf2ed4a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc74a59c640 1 -- 172.21.15.40:0/1222554919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed48260 msgr2=0x55cf2ed4a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc75359e640 1 --2- 172.21.15.40:0/2081266728 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc7400a2800 0x7fc7400a4cc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc74c011550 tx=0x7fc74c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc74a59c640 1 --2- 172.21.15.40:0/1222554919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed48260 0x55cf2ed4a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc74a59c640 1 -- 172.21.15.40:0/1222554919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 msgr2=0x55cf2ed42b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc74a59c640 1 --2- 172.21.15.40:0/1222554919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed42b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc74a59c640 1 -- 172.21.15.40:0/1222554919 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf2ec6d210 con 0x55cf2ed430f0 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc74a59c640 1 --2- 172.21.15.40:0/1222554919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed430f0 0x55cf2ed47bf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc73c009fa0 tx=0x7fc73c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67db9b17e5d9cfb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.177+0000 7fc74959a640 1 -- 172.21.15.40:0/1222554919 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c004030 con 0x55cf2ed430f0 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.178+0000 7fc74959a640 1 -- 172.21.15.40:0/1222554919 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc73c0054d0 con 0x55cf2ed430f0 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.178+0000 7fc74959a640 1 -- 172.21.15.40:0/1222554919 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c004ce0 con 0x55cf2ed430f0 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.178+0000 7fc75bd75740 1 -- 172.21.15.40:0/1222554919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed430f0 msgr2=0x55cf2ed47bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.178+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1222554919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed430f0 0x55cf2ed47bf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc73c009fa0 tx=0x7fc73c0118d0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.178+0000 7fc75bd75740 1 -- 172.21.15.40:0/1222554919 shutdown_connections 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.178+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1222554919 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed48260 0x55cf2ed4a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.178+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1222554919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed430f0 0x55cf2ed47bf0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.178+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1222554919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed42b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.179+0000 7fc75bd75740 1 -- 172.21.15.40:0/1222554919 >> 172.21.15.40:0/1222554919 conn(0x55cf2ed39860 msgr2=0x55cf2ed39c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.179+0000 7fc75bd75740 1 -- 172.21.15.40:0/1222554919 shutdown_connections 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.179+0000 7fc75bd75740 1 -- 172.21.15.40:0/1222554919 wait complete. 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.179+0000 7fc75bd75740 1 -- start start 2024-03-18T22:50:15.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.179+0000 7fc75bd75740 1 Processor -- start 2024-03-18T22:50:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.179+0000 7fc75bd75740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed4cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.179+0000 7fc75bd75740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed430f0 0x55cf2ed4d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc75bd75740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed48260 0x55cf2ed541c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc75bd75740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cf2ed4b210 con 0x55cf2ed42700 2024-03-18T22:50:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc75bd75740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cf2ec5e310 con 0x55cf2ed48260 2024-03-18T22:50:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc752d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed4cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc75bd75740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55cf2ed4b390 con 0x55cf2ed430f0 2024-03-18T22:50:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc752d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed4cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45840/0 (socket says 172.21.15.40:45840) 2024-03-18T22:50:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc752d9d640 1 -- 172.21.15.40:0/1967233598 learned_addr learned my addr 172.21.15.40:0/1967233598 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc75359e640 1 --2- 172.21.15.40:0/1967233598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed48260 0x55cf2ed541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc74a59c640 1 --2- 172.21.15.40:0/1967233598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed430f0 0x55cf2ed4d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc75bd75740 10 client.0 _ll_register_callbacks cb 0x55cf2ec522d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc752d9d640 1 -- 172.21.15.40:0/1967233598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed430f0 msgr2=0x55cf2ed4d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc752d9d640 1 --2- 172.21.15.40:0/1967233598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed430f0 0x55cf2ed4d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc752d9d640 1 -- 172.21.15.40:0/1967233598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed48260 msgr2=0x55cf2ed541c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc752d9d640 1 --2- 172.21.15.40:0/1967233598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed48260 0x55cf2ed541c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc752d9d640 1 -- 172.21.15.40:0/1967233598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf2e9b63f0 con 0x55cf2ed42700 2024-03-18T22:50:15.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc75bd75740 1 -- 172.21.15.40:0/1967233598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc73c0054d0 con 0x55cf2ed42700 2024-03-18T22:50:15.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc752d9d640 1 --2- 172.21.15.40:0/1967233598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed4cc40 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fc7440ee630 tx=0x7fc7440bd380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39f4b9a1fbbc4d93 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc73a7fc640 10 client.17526 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:15.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc73a7fc640 1 -- 172.21.15.40:0/1967233598 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7440bd480 con 0x55cf2ed42700 2024-03-18T22:50:15.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.180+0000 7fc73a7fc640 1 -- 172.21.15.40:0/1967233598 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc7440d6220 con 0x55cf2ed42700 2024-03-18T22:50:15.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.181+0000 7fc73a7fc640 1 -- 172.21.15.40:0/1967233598 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc7440c15d0 con 0x55cf2ed42700 2024-03-18T22:50:15.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.181+0000 7fc75bd75740 10 client.17526 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:15.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.181+0000 7fc75bd75740 1 -- 172.21.15.40:0/1967233598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({fsmap=34}) -- 0x55cf2ea807d0 con 0x55cf2ed42700 2024-03-18T22:50:15.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.181+0000 7fc73a7fc640 1 -- 172.21.15.40:0/1967233598 <== mon.0 v2:172.21.15.40:3300/0 4 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fc74409d020 con 0x55cf2ed42700 2024-03-18T22:50:15.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.181+0000 7fc75bd75740 10 client.17526 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:15.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.181+0000 7fc75bd75740 10 client.17526 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:50:15.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.181+0000 7fc75bd75740 20 client.17526 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:15.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.181+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1967233598 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55cf2ed58860 0x55cf2ed5ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.181+0000 7fc75bd75740 4 client.17526 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:15.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.181+0000 7fc75bd75740 1 -- 172.21.15.40:0/1967233598 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf2ed003e0 con 0x55cf2ed58860 2024-03-18T22:50:15.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.181+0000 7fc74a59c640 1 --2- 172.21.15.40:0/1967233598 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55cf2ed58860 0x55cf2ed5ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.182+0000 7fc74a59c640 1 --2- 172.21.15.40:0/1967233598 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55cf2ed58860 0x55cf2ed5ad20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c502576b256c6080 server_cookie=251b3178242d5e0c in_seq=0 out_seq=0 2024-03-18T22:50:15.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.182+0000 7fc73a7fc640 10 client.17526 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:50:15.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.183+0000 7fc73a7fc640 1 -- 172.21.15.40:0/1967233598 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cf2ed003e0 con 0x55cf2ed58860 2024-03-18T22:50:15.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.183+0000 7fc73a7fc640 10 client.17526 handle_command_reply: tid=0 2024-03-18T22:50:15.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.183+0000 7fc75bd75740 1 client.17526 shutdown 2024-03-18T22:50:15.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.183+0000 7fc75bd75740 1 -- 172.21.15.40:0/1967233598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 msgr2=0x55cf2ed4cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.183+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1967233598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed4cc40 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fc7440ee630 tx=0x7fc7440bd380 comp rx=0 tx=0).stop 2024-03-18T22:50:15.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.183+0000 7fc75bd75740 1 -- 172.21.15.40:0/1967233598 shutdown_connections 2024-03-18T22:50:15.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.183+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1967233598 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55cf2ed58860 0x55cf2ed5ad20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.183+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1967233598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed48260 0x55cf2ed541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.183+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1967233598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed430f0 0x55cf2ed4d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.183+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1967233598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed4cc40 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.183+0000 7fc75bd75740 1 -- 172.21.15.40:0/1967233598 >> 172.21.15.40:0/1967233598 conn(0x55cf2ed39860 msgr2=0x55cf2ed38ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.183+0000 7fc73a7fc640 0 client.17526 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:50:15.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.183+0000 7fc75bd75740 1 -- 172.21.15.40:0/1967233598 shutdown_connections 2024-03-18T22:50:15.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.183+0000 7fc75bd75740 1 -- 172.21.15.40:0/1967233598 wait complete. 2024-03-18T22:50:15.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.183+0000 7fc75bd75740 20 client.17526 trim_cache size 0 max 16384 2024-03-18T22:50:15.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc75bd75740 1 Processor -- start 2024-03-18T22:50:15.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc75bd75740 1 -- start start 2024-03-18T22:50:15.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc75bd75740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed42700 0x55cf2ed4d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc75bd75740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed430f0 0x55cf2ed54180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc75bd75740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed48260 0x55cf2ed3d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc75bd75740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc7541a8310 con 0x55cf2ed48260 2024-03-18T22:50:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc75bd75740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc7541a9f10 con 0x55cf2ed430f0 2024-03-18T22:50:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc75bd75740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc7541a89e0 con 0x55cf2ed42700 2024-03-18T22:50:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc75359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed48260 0x55cf2ed3d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc75359e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed48260 0x55cf2ed3d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45850/0 (socket says 172.21.15.40:45850) 2024-03-18T22:50:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc74a59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed430f0 0x55cf2ed54180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc75359e640 1 -- 172.21.15.40:0/249220942 learned_addr learned my addr 172.21.15.40:0/249220942 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc752d9d640 1 --2- 172.21.15.40:0/249220942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed42700 0x55cf2ed4d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc74a59c640 1 -- 172.21.15.40:0/249220942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed42700 msgr2=0x55cf2ed4d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc74a59c640 1 --2- 172.21.15.40:0/249220942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed42700 0x55cf2ed4d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc74a59c640 1 -- 172.21.15.40:0/249220942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed48260 msgr2=0x55cf2ed3d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc74a59c640 1 --2- 172.21.15.40:0/249220942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed48260 0x55cf2ed3d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.199+0000 7fc74a59c640 1 -- 172.21.15.40:0/249220942 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf2ed0f880 con 0x55cf2ed430f0 2024-03-18T22:50:15.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.200+0000 7fc74a59c640 1 --2- 172.21.15.40:0/249220942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed430f0 0x55cf2ed54180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x55cf2ed3a7d0 tx=0x7fc73c0096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a00cbf73c9aa233f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.200+0000 7fc748d99640 1 -- 172.21.15.40:0/249220942 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c01d040 con 0x55cf2ed430f0 2024-03-18T22:50:15.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.200+0000 7fc748d99640 1 -- 172.21.15.40:0/249220942 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc73c02f020 con 0x55cf2ed430f0 2024-03-18T22:50:15.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.200+0000 7fc748d99640 1 -- 172.21.15.40:0/249220942 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c0035f0 con 0x55cf2ed430f0 2024-03-18T22:50:15.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.201+0000 7fc75bd75740 1 -- 172.21.15.40:0/249220942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed430f0 msgr2=0x55cf2ed54180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.201+0000 7fc75bd75740 1 --2- 172.21.15.40:0/249220942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed430f0 0x55cf2ed54180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x55cf2ed3a7d0 tx=0x7fc73c0096c0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.201+0000 7fc75bd75740 1 -- 172.21.15.40:0/249220942 shutdown_connections 2024-03-18T22:50:15.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.201+0000 7fc75bd75740 1 --2- 172.21.15.40:0/249220942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed48260 0x55cf2ed3d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.201+0000 7fc75bd75740 1 --2- 172.21.15.40:0/249220942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed430f0 0x55cf2ed54180 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.201+0000 7fc75bd75740 1 --2- 172.21.15.40:0/249220942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed42700 0x55cf2ed4d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.201+0000 7fc75bd75740 1 -- 172.21.15.40:0/249220942 >> 172.21.15.40:0/249220942 conn(0x55cf2ed39860 msgr2=0x55cf2ed3cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.201+0000 7fc75bd75740 1 -- 172.21.15.40:0/249220942 shutdown_connections 2024-03-18T22:50:15.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.201+0000 7fc75bd75740 1 -- 172.21.15.40:0/249220942 wait complete. 2024-03-18T22:50:15.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.201+0000 7fc75bd75740 1 -- start start 2024-03-18T22:50:15.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.201+0000 7fc75bd75740 1 Processor -- start 2024-03-18T22:50:15.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc75bd75740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed51d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc75bd75740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed430f0 0x55cf2ed52270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc75bd75740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed48260 0x55cf2ed53210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc75bd75740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc7541a9840 con 0x55cf2ed42700 2024-03-18T22:50:15.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc75bd75740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc7541a90b0 con 0x55cf2ed48260 2024-03-18T22:50:15.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc75bd75740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc7541a62f0 con 0x55cf2ed430f0 2024-03-18T22:50:15.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc75bd75740 10 client.0 _ll_register_callbacks cb 0x7fc7440bd250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:15.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc752d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed51d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc752d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed51d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45862/0 (socket says 172.21.15.40:45862) 2024-03-18T22:50:15.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc752d9d640 1 -- 172.21.15.40:0/1334889724 learned_addr learned my addr 172.21.15.40:0/1334889724 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc74a59c640 1 --2- 172.21.15.40:0/1334889724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed430f0 0x55cf2ed52270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc75359e640 1 --2- 172.21.15.40:0/1334889724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed48260 0x55cf2ed53210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc74a59c640 1 -- 172.21.15.40:0/1334889724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed48260 msgr2=0x55cf2ed53210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc74a59c640 1 --2- 172.21.15.40:0/1334889724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed48260 0x55cf2ed53210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc74a59c640 1 -- 172.21.15.40:0/1334889724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 msgr2=0x55cf2ed51d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc74a59c640 1 --2- 172.21.15.40:0/1334889724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed51d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc74a59c640 1 -- 172.21.15.40:0/1334889724 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf2ed53800 con 0x55cf2ed430f0 2024-03-18T22:50:15.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc752d9d640 1 --2- 172.21.15.40:0/1334889724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed51d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc75359e640 1 --2- 172.21.15.40:0/1334889724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed48260 0x55cf2ed53210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc74a59c640 1 --2- 172.21.15.40:0/1334889724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed430f0 0x55cf2ed52270 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55cf2ed424e0 tx=0x7fc73c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96d9b462823ef240 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc73a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:50:15.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc73a7fc640 1 -- 172.21.15.40:0/1334889724 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c01d040 con 0x55cf2ed430f0 2024-03-18T22:50:15.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc75bd75740 1 -- 172.21.15.40:0/1334889724 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc73c02f020 con 0x55cf2ed430f0 2024-03-18T22:50:15.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.202+0000 7fc73a7fc640 1 -- 172.21.15.40:0/1334889724 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc73c030070 con 0x55cf2ed430f0 2024-03-18T22:50:15.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.203+0000 7fc73a7fc640 1 -- 172.21.15.40:0/1334889724 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc73c03d050 con 0x55cf2ed430f0 2024-03-18T22:50:15.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.203+0000 7fc75bd75740 10 client.27340 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:15.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.203+0000 7fc75bd75740 1 -- 172.21.15.40:0/1334889724 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cf2edbf0a0 con 0x55cf2ed430f0 2024-03-18T22:50:15.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.203+0000 7fc73a7fc640 1 -- 172.21.15.40:0/1334889724 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc73c02f020 con 0x55cf2ed430f0 2024-03-18T22:50:15.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.203+0000 7fc73a7fc640 1 -- 172.21.15.40:0/1334889724 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fc73c00f4e0 con 0x55cf2ed430f0 2024-03-18T22:50:15.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.203+0000 7fc75bd75740 10 client.27340 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:15.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.203+0000 7fc75bd75740 10 client.27340 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:50:15.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.203+0000 7fc75bd75740 20 client.27340 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:15.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.203+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1334889724 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55cf2edbf810 0x55cf2edc1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.203+0000 7fc75bd75740 4 client.27340 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:50:15.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.203+0000 7fc75bd75740 1 -- 172.21.15.40:0/1334889724 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cf2ecaf9f0 con 0x55cf2edbf810 2024-03-18T22:50:15.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.203+0000 7fc752d9d640 1 --2- 172.21.15.40:0/1334889724 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55cf2edbf810 0x55cf2edc1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc752d9d640 1 --2- 172.21.15.40:0/1334889724 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55cf2edbf810 0x55cf2edc1c00 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=78390b5df15026bc server_cookie=12d2dfa75967b1e1 in_seq=0 out_seq=0 2024-03-18T22:50:15.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc73a7fc640 10 client.27340 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:50:15.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc73a7fc640 1 -- 172.21.15.40:0/1334889724 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cf2ecaf9f0 con 0x55cf2edbf810 2024-03-18T22:50:15.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc73a7fc640 10 client.27340 handle_command_reply: tid=0 2024-03-18T22:50:15.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc75bd75740 1 client.27340 shutdown 2024-03-18T22:50:15.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc75bd75740 1 -- 172.21.15.40:0/1334889724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed430f0 msgr2=0x55cf2ed52270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1334889724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed430f0 0x55cf2ed52270 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55cf2ed424e0 tx=0x7fc73c009b50 comp rx=0 tx=0).stop 2024-03-18T22:50:15.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc75bd75740 1 -- 172.21.15.40:0/1334889724 shutdown_connections 2024-03-18T22:50:15.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1334889724 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55cf2edbf810 0x55cf2edc1c00 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1334889724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf2ed48260 0x55cf2ed53210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1334889724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cf2ed430f0 0x55cf2ed52270 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc73a7fc640 0 client.27340 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:50:15.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc75bd75740 1 --2- 172.21.15.40:0/1334889724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cf2ed42700 0x55cf2ed51d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc75bd75740 1 -- 172.21.15.40:0/1334889724 >> 172.21.15.40:0/1334889724 conn(0x55cf2ed39860 msgr2=0x55cf2ed3d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc75bd75740 1 -- 172.21.15.40:0/1334889724 shutdown_connections 2024-03-18T22:50:15.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc75bd75740 1 -- 172.21.15.40:0/1334889724 wait complete. 2024-03-18T22:50:15.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.204+0000 7fc75bd75740 20 client.27340 trim_cache size 0 max 16384 2024-03-18T22:50:15.213 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:50:15.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.205+0000 7fc7590ac640 1 -- 172.21.15.40:0/2081266728 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc7400a2800 msgr2=0x7fc7400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.205+0000 7fc7590ac640 1 --2- 172.21.15.40:0/2081266728 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc7400a2800 0x7fc7400a4cc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc74c011550 tx=0x7fc74c005f90 comp rx=0 tx=0).stop 2024-03-18T22:50:15.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.205+0000 7fc7590ac640 1 -- 172.21.15.40:0/2081266728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc754100760 msgr2=0x7fc754197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.205+0000 7fc7590ac640 1 --2- 172.21.15.40:0/2081266728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc754100760 0x7fc754197ae0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc74402c630 tx=0x7fc7440328a0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.205+0000 7fc7590ac640 1 -- 172.21.15.40:0/2081266728 shutdown_connections 2024-03-18T22:50:15.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.205+0000 7fc7590ac640 1 --2- 172.21.15.40:0/2081266728 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc7400a2800 0x7fc7400a4cc0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.205+0000 7fc7590ac640 1 --2- 172.21.15.40:0/2081266728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc754102310 0x7fc75419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.205+0000 7fc7590ac640 1 --2- 172.21.15.40:0/2081266728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc754101950 0x7fc754198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.205+0000 7fc7590ac640 1 --2- 172.21.15.40:0/2081266728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc754100760 0x7fc754197ae0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.205+0000 7fc7590ac640 1 -- 172.21.15.40:0/2081266728 >> 172.21.15.40:0/2081266728 conn(0x7fc7540fbcf0 msgr2=0x7fc7540fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.205+0000 7fc7590ac640 1 -- 172.21.15.40:0/2081266728 shutdown_connections 2024-03-18T22:50:15.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.205+0000 7fc7590ac640 1 -- 172.21.15.40:0/2081266728 wait complete. 2024-03-18T22:50:15.217 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:50:15.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:15 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/690563411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:50:15.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.336+0000 7fa9c44e8640 1 -- 172.21.15.40:0/1659846939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc1022d0 msgr2=0x7fa9bc108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.336+0000 7fa9c44e8640 1 --2- 172.21.15.40:0/1659846939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc1022d0 0x7fa9bc108b60 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4009ca0 tx=0x7fa9b402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.336+0000 7fa9c44e8640 1 -- 172.21.15.40:0/1659846939 shutdown_connections 2024-03-18T22:50:15.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.336+0000 7fa9c44e8640 1 --2- 172.21.15.40:0/1659846939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc1022d0 0x7fa9bc108b60 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.336+0000 7fa9c44e8640 1 --2- 172.21.15.40:0/1659846939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9bc101910 0x7fa9bc101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.336+0000 7fa9c44e8640 1 --2- 172.21.15.40:0/1659846939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9bc100720 0x7fa9bc100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.336+0000 7fa9c44e8640 1 -- 172.21.15.40:0/1659846939 >> 172.21.15.40:0/1659846939 conn(0x7fa9bc0fbc90 msgr2=0x7fa9bc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.336+0000 7fa9c44e8640 1 -- 172.21.15.40:0/1659846939 shutdown_connections 2024-03-18T22:50:15.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.336+0000 7fa9c44e8640 1 -- 172.21.15.40:0/1659846939 wait complete. 2024-03-18T22:50:15.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.337+0000 7fa9c44e8640 1 Processor -- start 2024-03-18T22:50:15.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.337+0000 7fa9c44e8640 1 -- start start 2024-03-18T22:50:15.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.337+0000 7fa9c44e8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9bc100720 0x7fa9bc199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.337+0000 7fa9c44e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc101910 0x7fa9bc19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c44e8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9bc1022d0 0x7fa9bc1a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c44e8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc10c230 con 0x7fa9bc101910 2024-03-18T22:50:15.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c44e8640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc10c0b0 con 0x7fa9bc1022d0 2024-03-18T22:50:15.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c44e8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc10c3b0 con 0x7fa9bc100720 2024-03-18T22:50:15.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c1a83640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc101910 0x7fa9bc19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c1a83640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc101910 0x7fa9bc19a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45876/0 (socket says 172.21.15.40:45876) 2024-03-18T22:50:15.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c1a83640 1 -- 172.21.15.40:0/2823222615 learned_addr learned my addr 172.21.15.40:0/2823222615 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c2284640 1 --2- 172.21.15.40:0/2823222615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9bc100720 0x7fa9bc199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c2a85640 1 --2- 172.21.15.40:0/2823222615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9bc1022d0 0x7fa9bc1a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c1a83640 1 -- 172.21.15.40:0/2823222615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9bc100720 msgr2=0x7fa9bc199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c1a83640 1 --2- 172.21.15.40:0/2823222615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9bc100720 0x7fa9bc199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c1a83640 1 -- 172.21.15.40:0/2823222615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9bc1022d0 msgr2=0x7fa9bc1a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c1a83640 1 --2- 172.21.15.40:0/2823222615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9bc1022d0 0x7fa9bc1a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c1a83640 1 -- 172.21.15.40:0/2823222615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9bc1a1850 con 0x7fa9bc101910 2024-03-18T22:50:15.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c1a83640 1 --2- 172.21.15.40:0/2823222615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc101910 0x7fa9bc19a180 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac00d950 tx=0x7fa9ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9b37fe640 1 -- 172.21.15.40:0/2823222615 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac017030 con 0x7fa9bc101910 2024-03-18T22:50:15.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9b37fe640 1 -- 172.21.15.40:0/2823222615 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9ac004020 con 0x7fa9bc101910 2024-03-18T22:50:15.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9c44e8640 1 -- 172.21.15.40:0/2823222615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9bc1a1b40 con 0x7fa9bc101910 2024-03-18T22:50:15.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.338+0000 7fa9b37fe640 1 -- 172.21.15.40:0/2823222615 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac0047a0 con 0x7fa9bc101910 2024-03-18T22:50:15.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.339+0000 7fa9c44e8640 1 -- 172.21.15.40:0/2823222615 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9bc1a2090 con 0x7fa9bc101910 2024-03-18T22:50:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.341+0000 7fa9b37fe640 1 -- 172.21.15.40:0/2823222615 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa9ac0041d0 con 0x7fa9bc101910 2024-03-18T22:50:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9c71ae740 1 Processor -- start 2024-03-18T22:50:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9c71ae740 1 -- start start 2024-03-18T22:50:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9b37fe640 1 --2- 172.21.15.40:0/2823222615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa9a80a2800 0x7fa9a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9c71ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8e3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9c71ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8e87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9c71ae740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e8e50 0x55ba4a8eb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9c71ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ba4a8474b0 con 0x55ba4a8e32f0 2024-03-18T22:50:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9c71ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ba4a7efd70 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9c71ae740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55ba4a843550 con 0x55ba4a8e8e50 2024-03-18T22:50:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9c2a85640 1 --2- 172.21.15.40:0/2823222615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa9a80a2800 0x7fa9a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9b37fe640 1 -- 172.21.15.40:0/2823222615 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa9ac0ce870 con 0x7fa9bc101910 2024-03-18T22:50:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9c2a85640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e8e50 0x55ba4a8eb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9c2284640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8e3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9c2284640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8e3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45890/0 (socket says 172.21.15.40:45890) 2024-03-18T22:50:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9c2284640 1 -- 172.21.15.40:0/2028036302 learned_addr learned my addr 172.21.15.40:0/2028036302 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.342+0000 7fa9c2a85640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e8e50 0x55ba4a8eb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:54032/0 (socket says 172.21.15.40:54032) 2024-03-18T22:50:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.343+0000 7fa9c1a83640 1 --2- 172.21.15.40:0/2028036302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8e87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.343+0000 7fa9c1a83640 1 -- 172.21.15.40:0/2028036302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e8e50 msgr2=0x55ba4a8eb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.343+0000 7fa9c1a83640 1 --2- 172.21.15.40:0/2028036302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e8e50 0x55ba4a8eb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.343+0000 7fa9c1a83640 1 -- 172.21.15.40:0/2028036302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e32f0 msgr2=0x55ba4a8e3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.343+0000 7fa9c1a83640 1 --2- 172.21.15.40:0/2028036302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8e3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.343+0000 7fa9c1a83640 1 -- 172.21.15.40:0/2028036302 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba4a812210 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.343+0000 7fa9c2284640 1 --2- 172.21.15.40:0/2028036302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8e3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.343+0000 7fa9c2a85640 1 --2- 172.21.15.40:0/2823222615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa9a80a2800 0x7fa9a80a4cc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4006010 tx=0x7fa9b4009930 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.343+0000 7fa9c1a83640 1 --2- 172.21.15.40:0/2028036302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8e87e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0cbf80 tx=0x7fa9ac097160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b78f15b5477f4c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.343+0000 7fa9b0bfe640 1 -- 172.21.15.40:0/2028036302 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac09ae20 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.343+0000 7fa9b0bfe640 1 -- 172.21.15.40:0/2028036302 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9ac09ef70 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.343+0000 7fa9b0bfe640 1 -- 172.21.15.40:0/2028036302 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac096ae0 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.344+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2028036302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e3ce0 msgr2=0x55ba4a8e87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.344+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/2028036302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8e87e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0cbf80 tx=0x7fa9ac097160 comp rx=0 tx=0).stop 2024-03-18T22:50:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.344+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2028036302 shutdown_connections 2024-03-18T22:50:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.344+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/2028036302 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e8e50 0x55ba4a8eb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.344+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/2028036302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8e87e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.344+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/2028036302 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8e3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.344+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2028036302 >> 172.21.15.40:0/2028036302 conn(0x55ba4a8de860 msgr2=0x55ba4a8dec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.344+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2028036302 shutdown_connections 2024-03-18T22:50:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.344+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2028036302 wait complete. 2024-03-18T22:50:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.344+0000 7fa9c71ae740 1 -- start start 2024-03-18T22:50:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c71ae740 1 Processor -- start 2024-03-18T22:50:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c71ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8ed890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c71ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8eddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c71ae740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e8e50 0x55ba4a8f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c71ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ba4a8ebe00 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c71ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ba4a803310 con 0x55ba4a8e32f0 2024-03-18T22:50:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c71ae740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55ba4a8ebf80 con 0x55ba4a8e8e50 2024-03-18T22:50:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c71ae740 10 client.0 _ll_register_callbacks cb 0x55ba4a7f72d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c1a83640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8eddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c2a85640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e8e50 0x55ba4a8f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c2284640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8ed890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c1a83640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8eddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45896/0 (socket says 172.21.15.40:45896) 2024-03-18T22:50:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c1a83640 1 -- 172.21.15.40:0/2376612539 learned_addr learned my addr 172.21.15.40:0/2376612539 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c1a83640 1 -- 172.21.15.40:0/2376612539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e8e50 msgr2=0x55ba4a8f4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c1a83640 1 --2- 172.21.15.40:0/2376612539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e8e50 0x55ba4a8f4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c1a83640 1 -- 172.21.15.40:0/2376612539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e32f0 msgr2=0x55ba4a8ed890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c1a83640 1 --2- 172.21.15.40:0/2376612539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8ed890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c1a83640 1 -- 172.21.15.40:0/2376612539 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba4a55b3f0 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c2a85640 1 --2- 172.21.15.40:0/2376612539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e8e50 0x55ba4a8f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9c1a83640 1 --2- 172.21.15.40:0/2376612539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8eddd0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0971f0 tx=0x7fa9ac0045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7230d548ff43bf7a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.345+0000 7fa9a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.346+0000 7fa9a5ffb640 1 -- 172.21.15.40:0/2376612539 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac059020 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.346+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2376612539 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9ac09ef70 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.346+0000 7fa9a5ffb640 1 -- 172.21.15.40:0/2376612539 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa9ac09a480 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.346+0000 7fa9a5ffb640 1 -- 172.21.15.40:0/2376612539 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa9ac097400 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.346+0000 7fa9c71ae740 10 client.17544 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.346+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2376612539 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ba4a6257d0 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.346+0000 7fa9a5ffb640 1 -- 172.21.15.40:0/2376612539 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9ac09ef70 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.346+0000 7fa9a5ffb640 1 -- 172.21.15.40:0/2376612539 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fa9ac055ad0 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.346+0000 7fa9c71ae740 10 client.17544 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:15.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.346+0000 7fa9c71ae740 10 client.17544 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:50:15.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.346+0000 7fa9c71ae740 20 client.17544 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:15.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.346+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/2376612539 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ba4a8f9630 0x55ba4a8fba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.346+0000 7fa9c71ae740 4 client.17544 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:15.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.346+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2376612539 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba4a8a53e0 con 0x55ba4a8f9630 2024-03-18T22:50:15.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.346+0000 7fa9c2284640 1 --2- 172.21.15.40:0/2376612539 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ba4a8f9630 0x55ba4a8fba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.347+0000 7fa9a5ffb640 10 client.17544 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:50:15.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.347+0000 7fa9c2284640 1 --2- 172.21.15.40:0/2376612539 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ba4a8f9630 0x55ba4a8fba50 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9f1d43a678163062 server_cookie=70df515a67308faf in_seq=0 out_seq=0 2024-03-18T22:50:15.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.347+0000 7fa9a5ffb640 1 -- 172.21.15.40:0/2376612539 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ba4a8a53e0 con 0x55ba4a8f9630 2024-03-18T22:50:15.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.347+0000 7fa9a5ffb640 10 client.17544 handle_command_reply: tid=0 2024-03-18T22:50:15.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.348+0000 7fa9c71ae740 1 client.17544 shutdown 2024-03-18T22:50:15.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.348+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2376612539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e3ce0 msgr2=0x55ba4a8eddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.348+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/2376612539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8eddd0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0971f0 tx=0x7fa9ac0045b0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.348+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2376612539 shutdown_connections 2024-03-18T22:50:15.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.348+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/2376612539 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ba4a8f9630 0x55ba4a8fba50 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.348+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/2376612539 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e8e50 0x55ba4a8f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.348+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/2376612539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8eddd0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.348+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/2376612539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8ed890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.348+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2376612539 >> 172.21.15.40:0/2376612539 conn(0x55ba4a8de860 msgr2=0x55ba4a8ddba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.348+0000 7fa9a5ffb640 0 client.17544 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:50:15.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.348+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2376612539 shutdown_connections 2024-03-18T22:50:15.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.348+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2376612539 wait complete. 2024-03-18T22:50:15.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.348+0000 7fa9c71ae740 20 client.17544 trim_cache size 0 max 16384 2024-03-18T22:50:15.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:15 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/690563411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:50:15.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.363+0000 7fa9c71ae740 1 Processor -- start 2024-03-18T22:50:15.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.363+0000 7fa9c71ae740 1 -- start start 2024-03-18T22:50:15.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.363+0000 7fa9c71ae740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8de860 0x55ba4a8ec380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.363+0000 7fa9c71ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8ec8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.363+0000 7fa9c71ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.363+0000 7fa9c71ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1ac7f0 con 0x55ba4a8e32f0 2024-03-18T22:50:15.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.363+0000 7fa9c71ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1ae3f0 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.363+0000 7fa9c71ae740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1acec0 con 0x55ba4a8de860 2024-03-18T22:50:15.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9c1a83640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8ec8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9c2a85640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9c2284640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8de860 0x55ba4a8ec380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9c2a85640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8f4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43802/0 (socket says 172.21.15.40:43802) 2024-03-18T22:50:15.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9c2284640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8de860 0x55ba4a8ec380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:54046/0 (socket says 172.21.15.40:54046) 2024-03-18T22:50:15.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9c2a85640 1 -- 172.21.15.40:0/2141116383 learned_addr learned my addr 172.21.15.40:0/2141116383 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9c2a85640 1 -- 172.21.15.40:0/2141116383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8de860 msgr2=0x55ba4a8ec380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9c2a85640 1 --2- 172.21.15.40:0/2141116383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8de860 0x55ba4a8ec380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9c2a85640 1 -- 172.21.15.40:0/2141116383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e32f0 msgr2=0x55ba4a8ec8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9c2a85640 1 --2- 172.21.15.40:0/2141116383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8ec8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9c2a85640 1 -- 172.21.15.40:0/2141116383 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba4a8b4880 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9c2284640 1 --2- 172.21.15.40:0/2141116383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8de860 0x55ba4a8ec380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9c2a85640 1 --2- 172.21.15.40:0/2141116383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8f4d60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4009ca0 tx=0x7fa9b4047000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f93a5eaee95fb71 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9a7fff640 1 -- 172.21.15.40:0/2141116383 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4039040 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9a7fff640 1 -- 172.21.15.40:0/2141116383 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9b4052070 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.364+0000 7fa9a7fff640 1 -- 172.21.15.40:0/2141116383 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4004860 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.365+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2141116383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e3ce0 msgr2=0x55ba4a8f4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.365+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/2141116383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8f4d60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4009ca0 tx=0x7fa9b4047000 comp rx=0 tx=0).stop 2024-03-18T22:50:15.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.365+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2141116383 shutdown_connections 2024-03-18T22:50:15.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.365+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/2141116383 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a8f4d60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.365+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/2141116383 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8ec8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.365+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/2141116383 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8de860 0x55ba4a8ec380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.365+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2141116383 >> 172.21.15.40:0/2141116383 conn(0x55ba4a8e4ac0 msgr2=0x55ba4a8ed890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.365+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2141116383 shutdown_connections 2024-03-18T22:50:15.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.365+0000 7fa9c71ae740 1 -- 172.21.15.40:0/2141116383 wait complete. 2024-03-18T22:50:15.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.365+0000 7fa9c71ae740 1 -- start start 2024-03-18T22:50:15.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.365+0000 7fa9c71ae740 1 Processor -- start 2024-03-18T22:50:15.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c71ae740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8de860 0x55ba4a8f1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c71ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8f2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c71ae740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a95d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c71ae740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1add20 con 0x55ba4a8de860 2024-03-18T22:50:15.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c71ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1ad590 con 0x55ba4a8e32f0 2024-03-18T22:50:15.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c71ae740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1aa7d0 con 0x55ba4a8e3ce0 2024-03-18T22:50:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c71ae740 10 client.0 _ll_register_callbacks cb 0x7fa9b8004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c2284640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8de860 0x55ba4a8f1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c2284640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8de860 0x55ba4a8f1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45916/0 (socket says 172.21.15.40:45916) 2024-03-18T22:50:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c2284640 1 -- 172.21.15.40:0/214102684 learned_addr learned my addr 172.21.15.40:0/214102684 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c1a83640 1 --2- 172.21.15.40:0/214102684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c2a85640 1 --2- 172.21.15.40:0/214102684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a95d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c2284640 1 -- 172.21.15.40:0/214102684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e3ce0 msgr2=0x55ba4a95d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c2284640 1 --2- 172.21.15.40:0/214102684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a95d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c2284640 1 -- 172.21.15.40:0/214102684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e32f0 msgr2=0x55ba4a8f2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c2284640 1 --2- 172.21.15.40:0/214102684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8f2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c2284640 1 -- 172.21.15.40:0/214102684 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba4a8f2690 con 0x55ba4a8de860 2024-03-18T22:50:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c2284640 1 --2- 172.21.15.40:0/214102684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8de860 0x55ba4a8f1b10 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x55ba4a8df7d0 tx=0x7fa9b800bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=312e678d76680d26 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9a5ffb640 1 -- 172.21.15.40:0/214102684 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b80200b0 con 0x55ba4a8de860 2024-03-18T22:50:15.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9a5ffb640 1 -- 172.21.15.40:0/214102684 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa9b8020250 con 0x55ba4a8de860 2024-03-18T22:50:15.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.366+0000 7fa9c71ae740 1 -- 172.21.15.40:0/214102684 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9b4052070 con 0x55ba4a8de860 2024-03-18T22:50:15.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.367+0000 7fa9a5ffb640 1 -- 172.21.15.40:0/214102684 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa9b802e050 con 0x55ba4a8de860 2024-03-18T22:50:15.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.367+0000 7fa9c71ae740 10 client.17550 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:15.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.367+0000 7fa9c71ae740 1 -- 172.21.15.40:0/214102684 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ba4a95efa0 con 0x55ba4a8de860 2024-03-18T22:50:15.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.367+0000 7fa9a5ffb640 1 -- 172.21.15.40:0/214102684 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9b4052070 con 0x55ba4a8de860 2024-03-18T22:50:15.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.367+0000 7fa9a5ffb640 1 -- 172.21.15.40:0/214102684 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fa9b8011c40 con 0x55ba4a8de860 2024-03-18T22:50:15.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.367+0000 7fa9c71ae740 10 client.17550 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:15.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.367+0000 7fa9c71ae740 10 client.17550 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:50:15.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.367+0000 7fa9c71ae740 20 client.17550 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:15.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.367+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/214102684 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ba4a8e8e50 0x55ba4a961ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.367+0000 7fa9c71ae740 4 client.17550 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:50:15.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.367+0000 7fa9c71ae740 1 -- 172.21.15.40:0/214102684 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ba4a8549f0 con 0x55ba4a8e8e50 2024-03-18T22:50:15.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.367+0000 7fa9c1a83640 1 --2- 172.21.15.40:0/214102684 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ba4a8e8e50 0x55ba4a961ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9c1a83640 1 --2- 172.21.15.40:0/214102684 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ba4a8e8e50 0x55ba4a961ad0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3f6b52ffc07fc25a server_cookie=378046c12f4a34aa in_seq=0 out_seq=0 2024-03-18T22:50:15.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9a5ffb640 10 client.17550 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:50:15.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9a5ffb640 1 -- 172.21.15.40:0/214102684 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ba4a8549f0 con 0x55ba4a8e8e50 2024-03-18T22:50:15.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9a5ffb640 10 client.17550 handle_command_reply: tid=0 2024-03-18T22:50:15.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9c71ae740 1 client.17550 shutdown 2024-03-18T22:50:15.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9c71ae740 1 -- 172.21.15.40:0/214102684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8de860 msgr2=0x55ba4a8f1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/214102684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8de860 0x55ba4a8f1b10 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x55ba4a8df7d0 tx=0x7fa9b800bed0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9c71ae740 1 -- 172.21.15.40:0/214102684 shutdown_connections 2024-03-18T22:50:15.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/214102684 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ba4a8e8e50 0x55ba4a961ad0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/214102684 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ba4a8e3ce0 0x55ba4a95d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/214102684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ba4a8e32f0 0x55ba4a8f2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9a5ffb640 0 client.17550 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:50:15.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9c71ae740 1 --2- 172.21.15.40:0/214102684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ba4a8de860 0x55ba4a8f1b10 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9c71ae740 1 -- 172.21.15.40:0/214102684 >> 172.21.15.40:0/214102684 conn(0x55ba4a8e4ac0 msgr2=0x55ba4a8e1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9c71ae740 1 -- 172.21.15.40:0/214102684 shutdown_connections 2024-03-18T22:50:15.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9c71ae740 1 -- 172.21.15.40:0/214102684 wait complete. 2024-03-18T22:50:15.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.368+0000 7fa9c71ae740 20 client.17550 trim_cache size 0 max 16384 2024-03-18T22:50:15.376 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:50:15.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.369+0000 7fa9c44e8640 1 -- 172.21.15.40:0/2823222615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa9a80a2800 msgr2=0x7fa9a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.369+0000 7fa9c44e8640 1 --2- 172.21.15.40:0/2823222615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa9a80a2800 0x7fa9a80a4cc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4006010 tx=0x7fa9b4009930 comp rx=0 tx=0).stop 2024-03-18T22:50:15.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.369+0000 7fa9c44e8640 1 -- 172.21.15.40:0/2823222615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc101910 msgr2=0x7fa9bc19a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.369+0000 7fa9c44e8640 1 --2- 172.21.15.40:0/2823222615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc101910 0x7fa9bc19a180 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac00d950 tx=0x7fa9ac00de10 comp rx=0 tx=0).stop 2024-03-18T22:50:15.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.369+0000 7fa9c44e8640 1 -- 172.21.15.40:0/2823222615 shutdown_connections 2024-03-18T22:50:15.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.369+0000 7fa9c44e8640 1 --2- 172.21.15.40:0/2823222615 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fa9a80a2800 0x7fa9a80a4cc0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.369+0000 7fa9c44e8640 1 --2- 172.21.15.40:0/2823222615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9bc1022d0 0x7fa9bc1a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.369+0000 7fa9c44e8640 1 --2- 172.21.15.40:0/2823222615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc101910 0x7fa9bc19a180 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.369+0000 7fa9c44e8640 1 --2- 172.21.15.40:0/2823222615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fa9bc100720 0x7fa9bc199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.369+0000 7fa9c44e8640 1 -- 172.21.15.40:0/2823222615 >> 172.21.15.40:0/2823222615 conn(0x7fa9bc0fbc90 msgr2=0x7fa9bc0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.369+0000 7fa9c44e8640 1 -- 172.21.15.40:0/2823222615 shutdown_connections 2024-03-18T22:50:15.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.369+0000 7fa9c44e8640 1 -- 172.21.15.40:0/2823222615 wait complete. 2024-03-18T22:50:15.380 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:15 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/690563411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:50:15.381 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:50:15.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.496+0000 7f9591110640 1 -- 172.21.15.40:0/2383481850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958c102aa0 msgr2=0x7f958c102f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.496+0000 7f9591110640 1 --2- 172.21.15.40:0/2383481850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958c102aa0 0x7f958c102f20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f9574009900 tx=0x7f957402bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:15.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.497+0000 7f9591110640 1 -- 172.21.15.40:0/2383481850 shutdown_connections 2024-03-18T22:50:15.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.497+0000 7f9591110640 1 --2- 172.21.15.40:0/2383481850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f958c103460 0x7f958c109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.497+0000 7f9591110640 1 --2- 172.21.15.40:0/2383481850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958c102aa0 0x7f958c102f20 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.497+0000 7f9591110640 1 --2- 172.21.15.40:0/2383481850 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f958c1018b0 0x7f958c101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.497+0000 7f9591110640 1 -- 172.21.15.40:0/2383481850 >> 172.21.15.40:0/2383481850 conn(0x7f958c0fce40 msgr2=0x7f958c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.497+0000 7f9591110640 1 -- 172.21.15.40:0/2383481850 shutdown_connections 2024-03-18T22:50:15.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.497+0000 7f9591110640 1 -- 172.21.15.40:0/2383481850 wait complete. 2024-03-18T22:50:15.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.497+0000 7f9591110640 1 Processor -- start 2024-03-18T22:50:15.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.497+0000 7f9591110640 1 -- start start 2024-03-18T22:50:15.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.497+0000 7f9591110640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f958c1018b0 0x7f958c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.497+0000 7f9591110640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958c102aa0 0x7f958c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.497+0000 7f9591110640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f958c103460 0x7f958c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f9591110640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f958c10d370 con 0x7f958c102aa0 2024-03-18T22:50:15.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f9591110640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f958c10d1f0 con 0x7f958c103460 2024-03-18T22:50:15.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f9591110640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f958c10d4f0 con 0x7f958c1018b0 2024-03-18T22:50:15.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f9583fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958c102aa0 0x7f958c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f958ad9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f958c1018b0 0x7f958c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f9583fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958c102aa0 0x7f958c197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45936/0 (socket says 172.21.15.40:45936) 2024-03-18T22:50:15.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f9583fff640 1 -- 172.21.15.40:0/3611277069 learned_addr learned my addr 172.21.15.40:0/3611277069 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f958b59e640 1 --2- 172.21.15.40:0/3611277069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f958c103460 0x7f958c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f958ad9d640 1 -- 172.21.15.40:0/3611277069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f958c103460 msgr2=0x7f958c19f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f958ad9d640 1 --2- 172.21.15.40:0/3611277069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f958c103460 0x7f958c19f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f958ad9d640 1 -- 172.21.15.40:0/3611277069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958c102aa0 msgr2=0x7f958c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f958ad9d640 1 --2- 172.21.15.40:0/3611277069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958c102aa0 0x7f958c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f958ad9d640 1 -- 172.21.15.40:0/3611277069 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c19f6b0 con 0x7f958c1018b0 2024-03-18T22:50:15.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f958ad9d640 1 --2- 172.21.15.40:0/3611277069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f958c1018b0 0x7f958c197a90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f957c00e9c0 tx=0x7f957c00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f9588d99640 1 -- 172.21.15.40:0/3611277069 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c00ccc0 con 0x7f958c1018b0 2024-03-18T22:50:15.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f9588d99640 1 -- 172.21.15.40:0/3611277069 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f957c0035f0 con 0x7f958c1018b0 2024-03-18T22:50:15.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f9591110640 1 -- 172.21.15.40:0/3611277069 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f958c19f9a0 con 0x7f958c1018b0 2024-03-18T22:50:15.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f9588d99640 1 -- 172.21.15.40:0/3611277069 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c00ce60 con 0x7f958c1018b0 2024-03-18T22:50:15.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.498+0000 7f9591110640 1 -- 172.21.15.40:0/3611277069 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f958c19fea0 con 0x7f958c1018b0 2024-03-18T22:50:15.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.501+0000 7f9588d99640 1 -- 172.21.15.40:0/3611277069 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f957c0037a0 con 0x7f958c1018b0 2024-03-18T22:50:15.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.501+0000 7f9588d99640 1 --2- 172.21.15.40:0/3611277069 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f95780a2800 0x7f95780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.501+0000 7f9593dde740 1 Processor -- start 2024-03-18T22:50:15.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.501+0000 7f9593dde740 1 -- start start 2024-03-18T22:50:15.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.501+0000 7f9583fff640 1 --2- 172.21.15.40:0/3611277069 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f95780a2800 0x7f95780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.501+0000 7f9588d99640 1 -- 172.21.15.40:0/3611277069 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f957c0cf150 con 0x7f958c1018b0 2024-03-18T22:50:15.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f9593dde740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1ab710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f9593dde740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 0x56360d1b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f9593dde740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 0x56360d1b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f9593dde740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56360d10f4b0 con 0x56360d1b0e50 2024-03-18T22:50:15.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f9593dde740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56360d0b7d70 con 0x56360d1abce0 2024-03-18T22:50:15.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f9593dde740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56360d10b550 con 0x56360d1ab2f0 2024-03-18T22:50:15.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f958b59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 0x56360d1b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f958ad9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1ab710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f958ad9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1ab710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:54090/0 (socket says 172.21.15.40:54090) 2024-03-18T22:50:15.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f958ad9d640 1 -- 172.21.15.40:0/2386835680 learned_addr learned my addr 172.21.15.40:0/2386835680 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f958b59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 0x56360d1b32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45942/0 (socket says 172.21.15.40:45942) 2024-03-18T22:50:15.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f9583fff640 1 --2- 172.21.15.40:0/2386835680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 0x56360d1b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f9583fff640 1 --2- 172.21.15.40:0/3611277069 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f95780a2800 0x7f95780a4cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f958c198fa0 tx=0x7f957402c7e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f958ad9d640 1 -- 172.21.15.40:0/2386835680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 msgr2=0x56360d1b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f958ad9d640 1 --2- 172.21.15.40:0/2386835680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 0x56360d1b07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f958ad9d640 1 -- 172.21.15.40:0/2386835680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 msgr2=0x56360d1b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f958ad9d640 1 --2- 172.21.15.40:0/2386835680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 0x56360d1b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f958ad9d640 1 -- 172.21.15.40:0/2386835680 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56360d0da210 con 0x56360d1ab2f0 2024-03-18T22:50:15.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f958ad9d640 1 --2- 172.21.15.40:0/2386835680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1ab710 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f957c09b690 tx=0x7f957c09be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60be3cbd5013f11c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f95817fa640 1 -- 172.21.15.40:0/2386835680 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c0b4070 con 0x56360d1ab2f0 2024-03-18T22:50:15.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.502+0000 7f95817fa640 1 -- 172.21.15.40:0/2386835680 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f957c0a3ea0 con 0x56360d1ab2f0 2024-03-18T22:50:15.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.503+0000 7f95817fa640 1 -- 172.21.15.40:0/2386835680 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c0a3a10 con 0x56360d1ab2f0 2024-03-18T22:50:15.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.503+0000 7f9593dde740 1 -- 172.21.15.40:0/2386835680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 msgr2=0x56360d1ab710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.503+0000 7f9593dde740 1 --2- 172.21.15.40:0/2386835680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1ab710 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f957c09b690 tx=0x7f957c09be00 comp rx=0 tx=0).stop 2024-03-18T22:50:15.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.503+0000 7f9593dde740 1 -- 172.21.15.40:0/2386835680 shutdown_connections 2024-03-18T22:50:15.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.503+0000 7f9593dde740 1 --2- 172.21.15.40:0/2386835680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 0x56360d1b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.503+0000 7f9593dde740 1 --2- 172.21.15.40:0/2386835680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 0x56360d1b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.503+0000 7f9593dde740 1 --2- 172.21.15.40:0/2386835680 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1ab710 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.503+0000 7f9593dde740 1 -- 172.21.15.40:0/2386835680 >> 172.21.15.40:0/2386835680 conn(0x56360d1a6860 msgr2=0x56360d1a6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.503+0000 7f9593dde740 1 -- 172.21.15.40:0/2386835680 shutdown_connections 2024-03-18T22:50:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.504+0000 7f9593dde740 1 -- 172.21.15.40:0/2386835680 wait complete. 2024-03-18T22:50:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.504+0000 7f9593dde740 1 -- start start 2024-03-18T22:50:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.504+0000 7f9593dde740 1 Processor -- start 2024-03-18T22:50:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.504+0000 7f9593dde740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1b5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.504+0000 7f9593dde740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 0x56360d1b5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.504+0000 7f9593dde740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 0x56360d1bcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.504+0000 7f9593dde740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56360d1b3e00 con 0x56360d1b0e50 2024-03-18T22:50:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.504+0000 7f9593dde740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56360d0cb310 con 0x56360d1abce0 2024-03-18T22:50:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.504+0000 7f9593dde740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56360d1b3f80 con 0x56360d1ab2f0 2024-03-18T22:50:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.504+0000 7f9593dde740 10 client.0 _ll_register_callbacks cb 0x56360d0bf2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.504+0000 7f958ad9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.504+0000 7f958b59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 0x56360d1bcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.504+0000 7f958ad9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1b5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:54100/0 (socket says 172.21.15.40:54100) 2024-03-18T22:50:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.504+0000 7f9583fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 0x56360d1b5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.504+0000 7f958ad9d640 1 -- 172.21.15.40:0/3176921370 learned_addr learned my addr 172.21.15.40:0/3176921370 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.505+0000 7f958ad9d640 1 -- 172.21.15.40:0/3176921370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 msgr2=0x56360d1b5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.505+0000 7f958ad9d640 1 --2- 172.21.15.40:0/3176921370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 0x56360d1b5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.505+0000 7f958ad9d640 1 -- 172.21.15.40:0/3176921370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 msgr2=0x56360d1bcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.505+0000 7f958ad9d640 1 --2- 172.21.15.40:0/3176921370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 0x56360d1bcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.505+0000 7f958ad9d640 1 -- 172.21.15.40:0/3176921370 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56360ce233f0 con 0x56360d1ab2f0 2024-03-18T22:50:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.505+0000 7f958ad9d640 1 --2- 172.21.15.40:0/3176921370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1b5830 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f957c0ccbd0 tx=0x7f957c09f910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5add6e3593689fc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.505+0000 7f95727fc640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:50:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.505+0000 7f95727fc640 1 -- 172.21.15.40:0/3176921370 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c0b8020 con 0x56360d1ab2f0 2024-03-18T22:50:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.505+0000 7f95727fc640 1 -- 172.21.15.40:0/3176921370 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f957c0a3c10 con 0x56360d1ab2f0 2024-03-18T22:50:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.505+0000 7f9593dde740 1 -- 172.21.15.40:0/3176921370 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f957c0a3ea0 con 0x56360d1ab2f0 2024-03-18T22:50:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.505+0000 7f95727fc640 1 -- 172.21.15.40:0/3176921370 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f957c0b4070 con 0x56360d1ab2f0 2024-03-18T22:50:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.505+0000 7f9593dde740 10 client.27370 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.505+0000 7f9593dde740 1 -- 172.21.15.40:0/3176921370 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56360ceed7d0 con 0x56360d1ab2f0 2024-03-18T22:50:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.505+0000 7f95727fc640 1 -- 172.21.15.40:0/3176921370 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f957c0a3ea0 con 0x56360d1ab2f0 2024-03-18T22:50:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.506+0000 7f95727fc640 1 -- 172.21.15.40:0/3176921370 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f957c097570 con 0x56360d1ab2f0 2024-03-18T22:50:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.506+0000 7f9593dde740 10 client.27370 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.506+0000 7f9593dde740 10 client.27370 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.506+0000 7f9593dde740 20 client.27370 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.506+0000 7f9593dde740 1 --2- 172.21.15.40:0/3176921370 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56360d1c12a0 0x56360d1c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.506+0000 7f9593dde740 4 client.27370 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.506+0000 7f9593dde740 1 -- 172.21.15.40:0/3176921370 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56360d16d3e0 con 0x56360d1c12a0 2024-03-18T22:50:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.506+0000 7f9583fff640 1 --2- 172.21.15.40:0/3176921370 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56360d1c12a0 0x56360d1c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.506+0000 7f9583fff640 1 --2- 172.21.15.40:0/3176921370 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56360d1c12a0 0x56360d1c3690 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9f0be0ad0d90a0a server_cookie=a4c20d2eecdc235 in_seq=0 out_seq=0 2024-03-18T22:50:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.506+0000 7f95727fc640 10 client.27370 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.507+0000 7f95727fc640 1 -- 172.21.15.40:0/3176921370 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56360d16d3e0 con 0x56360d1c12a0 2024-03-18T22:50:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.507+0000 7f95727fc640 10 client.27370 handle_command_reply: tid=0 2024-03-18T22:50:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.507+0000 7f9593dde740 1 client.27370 shutdown 2024-03-18T22:50:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.507+0000 7f9593dde740 1 -- 172.21.15.40:0/3176921370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 msgr2=0x56360d1b5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.507+0000 7f9593dde740 1 --2- 172.21.15.40:0/3176921370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1b5830 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f957c0ccbd0 tx=0x7f957c09f910 comp rx=0 tx=0).stop 2024-03-18T22:50:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.507+0000 7f9593dde740 1 -- 172.21.15.40:0/3176921370 shutdown_connections 2024-03-18T22:50:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.507+0000 7f9593dde740 1 --2- 172.21.15.40:0/3176921370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 0x56360d1bcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.507+0000 7f9593dde740 1 --2- 172.21.15.40:0/3176921370 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56360d1c12a0 0x56360d1c3690 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.507+0000 7f9593dde740 1 --2- 172.21.15.40:0/3176921370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 0x56360d1b5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.507+0000 7f9593dde740 1 --2- 172.21.15.40:0/3176921370 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1b5830 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.507+0000 7f95727fc640 0 client.27370 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.507+0000 7f9593dde740 1 -- 172.21.15.40:0/3176921370 >> 172.21.15.40:0/3176921370 conn(0x56360d1a6860 msgr2=0x56360d1a5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.508+0000 7f9593dde740 1 -- 172.21.15.40:0/3176921370 shutdown_connections 2024-03-18T22:50:15.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.508+0000 7f9593dde740 1 -- 172.21.15.40:0/3176921370 wait complete. 2024-03-18T22:50:15.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.508+0000 7f9593dde740 20 client.27370 trim_cache size 0 max 16384 2024-03-18T22:50:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.525+0000 7f9593dde740 1 Processor -- start 2024-03-18T22:50:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.525+0000 7f9593dde740 1 -- start start 2024-03-18T22:50:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.525+0000 7f9593dde740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1b5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.525+0000 7f9593dde740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 0x56360d1bcd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.525+0000 7f9593dde740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 0x56360d1a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.525+0000 7f9593dde740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1aa510 con 0x56360d1b0e50 2024-03-18T22:50:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.525+0000 7f9593dde740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1ac110 con 0x56360d1abce0 2024-03-18T22:50:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.525+0000 7f9593dde740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1aabe0 con 0x56360d1ab2f0 2024-03-18T22:50:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.525+0000 7f958b59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 0x56360d1a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.525+0000 7f958ad9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1b5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.525+0000 7f9583fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 0x56360d1bcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.525+0000 7f958b59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 0x56360d1a5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45956/0 (socket says 172.21.15.40:45956) 2024-03-18T22:50:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.525+0000 7f958b59e640 1 -- 172.21.15.40:0/167287078 learned_addr learned my addr 172.21.15.40:0/167287078 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.525+0000 7f9583fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 0x56360d1bcd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43850/0 (socket says 172.21.15.40:43850) 2024-03-18T22:50:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.525+0000 7f958ad9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1b5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:54104/0 (socket says 172.21.15.40:54104) 2024-03-18T22:50:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.526+0000 7f958b59e640 1 -- 172.21.15.40:0/167287078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 msgr2=0x56360d1b5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.526+0000 7f958b59e640 1 --2- 172.21.15.40:0/167287078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1b5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.526+0000 7f958b59e640 1 -- 172.21.15.40:0/167287078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 msgr2=0x56360d1bcd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.526+0000 7f958b59e640 1 --2- 172.21.15.40:0/167287078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 0x56360d1bcd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.526+0000 7f958b59e640 1 -- 172.21.15.40:0/167287078 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56360d17c880 con 0x56360d1b0e50 2024-03-18T22:50:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.526+0000 7f9583fff640 1 --2- 172.21.15.40:0/167287078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 0x56360d1bcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.526+0000 7f958ad9d640 1 --2- 172.21.15.40:0/167287078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1b5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.526+0000 7f958b59e640 1 --2- 172.21.15.40:0/167287078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 0x56360d1a5a20 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9584012610 tx=0x7f9584009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685201b89e979c15 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.526+0000 7f9580ff9640 1 -- 172.21.15.40:0/167287078 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9584022070 con 0x56360d1b0e50 2024-03-18T22:50:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.526+0000 7f9580ff9640 1 -- 172.21.15.40:0/167287078 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9584003e20 con 0x56360d1b0e50 2024-03-18T22:50:15.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.526+0000 7f9580ff9640 1 -- 172.21.15.40:0/167287078 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958400f840 con 0x56360d1b0e50 2024-03-18T22:50:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 -- 172.21.15.40:0/167287078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 msgr2=0x56360d1a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 --2- 172.21.15.40:0/167287078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 0x56360d1a5a20 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9584012610 tx=0x7f9584009500 comp rx=0 tx=0).stop 2024-03-18T22:50:15.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 -- 172.21.15.40:0/167287078 shutdown_connections 2024-03-18T22:50:15.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 --2- 172.21.15.40:0/167287078 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1b0e50 0x56360d1a5a20 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 --2- 172.21.15.40:0/167287078 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1abce0 0x56360d1bcd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 --2- 172.21.15.40:0/167287078 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1ab2f0 0x56360d1b5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 -- 172.21.15.40:0/167287078 >> 172.21.15.40:0/167287078 conn(0x56360d1a6860 msgr2=0x56360d1a5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 -- 172.21.15.40:0/167287078 shutdown_connections 2024-03-18T22:50:15.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 -- 172.21.15.40:0/167287078 wait complete. 2024-03-18T22:50:15.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 -- start start 2024-03-18T22:50:15.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 Processor -- start 2024-03-18T22:50:15.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1ab2f0 0x56360d1ba6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1abce0 0x56360d1bac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1b0e50 0x56360d1bbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1aba40 con 0x56360d1abce0 2024-03-18T22:50:15.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1ab2b0 con 0x56360d1ab2f0 2024-03-18T22:50:15.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.527+0000 7f9593dde740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1a84d0 con 0x56360d1b0e50 2024-03-18T22:50:15.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f9593dde740 10 client.0 _ll_register_callbacks cb 0x7f957c0b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:15.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f9583fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1abce0 0x56360d1bac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f9583fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1abce0 0x56360d1bac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45962/0 (socket says 172.21.15.40:45962) 2024-03-18T22:50:15.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f9583fff640 1 -- 172.21.15.40:0/1440548686 learned_addr learned my addr 172.21.15.40:0/1440548686 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:15.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f958ad9d640 1 --2- 172.21.15.40:0/1440548686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1ab2f0 0x56360d1ba6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f958b59e640 1 --2- 172.21.15.40:0/1440548686 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1b0e50 0x56360d1bbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f9583fff640 1 -- 172.21.15.40:0/1440548686 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1b0e50 msgr2=0x56360d1bbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f9583fff640 1 --2- 172.21.15.40:0/1440548686 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1b0e50 0x56360d1bbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f9583fff640 1 -- 172.21.15.40:0/1440548686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1ab2f0 msgr2=0x56360d1ba6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:15.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f9583fff640 1 --2- 172.21.15.40:0/1440548686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1ab2f0 0x56360d1ba6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f9583fff640 1 -- 172.21.15.40:0/1440548686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56360d1bc210 con 0x56360d1abce0 2024-03-18T22:50:15.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f9583fff640 1 --2- 172.21.15.40:0/1440548686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1abce0 0x56360d1bac10 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9574036040 tx=0x7f957402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eefc3bc8a784a8bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:15.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f95727fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:15.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f95727fc640 1 -- 172.21.15.40:0/1440548686 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9574055e50 con 0x56360d1abce0 2024-03-18T22:50:15.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f95727fc640 1 -- 172.21.15.40:0/1440548686 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9574004350 con 0x56360d1abce0 2024-03-18T22:50:15.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f9593dde740 1 -- 172.21.15.40:0/1440548686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9584003e20 con 0x56360d1abce0 2024-03-18T22:50:15.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f95727fc640 1 -- 172.21.15.40:0/1440548686 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f9574065050 con 0x56360d1abce0 2024-03-18T22:50:15.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f9593dde740 10 client.17568 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:15.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.528+0000 7f9593dde740 1 -- 172.21.15.40:0/1440548686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56360d227690 con 0x56360d1abce0 2024-03-18T22:50:15.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.529+0000 7f95727fc640 1 -- 172.21.15.40:0/1440548686 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9584003e20 con 0x56360d1abce0 2024-03-18T22:50:15.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.529+0000 7f95727fc640 1 -- 172.21.15.40:0/1440548686 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f95740046c0 con 0x56360d1abce0 2024-03-18T22:50:15.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.529+0000 7f9593dde740 10 client.17568 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:15.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.529+0000 7f9593dde740 10 client.17568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:15.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.529+0000 7f9593dde740 20 client.17568 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:15.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.529+0000 7f9593dde740 1 --2- 172.21.15.40:0/1440548686 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56360d228070 0x56360d22a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:15.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.529+0000 7f9593dde740 4 client.17568 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:50:15.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.529+0000 7f9593dde740 1 -- 172.21.15.40:0/1440548686 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56360d1b0bc0 con 0x56360d228070 2024-03-18T22:50:15.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.529+0000 7f958ad9d640 1 --2- 172.21.15.40:0/1440548686 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56360d228070 0x56360d22a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:15.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.529+0000 7f95727fc640 10 client.17568 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:15.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f958ad9d640 1 --2- 172.21.15.40:0/1440548686 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56360d228070 0x56360d22a490 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3831ea63c997914 server_cookie=af8e4c9f905a85df in_seq=0 out_seq=0 2024-03-18T22:50:15.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f95727fc640 1 -- 172.21.15.40:0/1440548686 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56360d1b0bc0 con 0x56360d228070 2024-03-18T22:50:15.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f95727fc640 10 client.17568 handle_command_reply: tid=0 2024-03-18T22:50:15.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f9593dde740 1 client.17568 shutdown 2024-03-18T22:50:15.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f9593dde740 1 -- 172.21.15.40:0/1440548686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1abce0 msgr2=0x56360d1bac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f9593dde740 1 --2- 172.21.15.40:0/1440548686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1abce0 0x56360d1bac10 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9574036040 tx=0x7f957402bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:15.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f9593dde740 1 -- 172.21.15.40:0/1440548686 shutdown_connections 2024-03-18T22:50:15.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f9593dde740 1 --2- 172.21.15.40:0/1440548686 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56360d1b0e50 0x56360d1bbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f9593dde740 1 --2- 172.21.15.40:0/1440548686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56360d1abce0 0x56360d1bac10 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f9593dde740 1 --2- 172.21.15.40:0/1440548686 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56360d228070 0x56360d22a490 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f9593dde740 1 --2- 172.21.15.40:0/1440548686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56360d1ab2f0 0x56360d1ba6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f9593dde740 1 -- 172.21.15.40:0/1440548686 >> 172.21.15.40:0/1440548686 conn(0x56360d1a6860 msgr2=0x56360d1aa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f95727fc640 0 client.17568 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:15.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f9593dde740 1 -- 172.21.15.40:0/1440548686 shutdown_connections 2024-03-18T22:50:15.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f9593dde740 1 -- 172.21.15.40:0/1440548686 wait complete. 2024-03-18T22:50:15.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.530+0000 7f9593dde740 20 client.17568 trim_cache size 0 max 16384 2024-03-18T22:50:15.537 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:50:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.531+0000 7f9591110640 1 -- 172.21.15.40:0/3611277069 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f95780a2800 msgr2=0x7f95780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.531+0000 7f9591110640 1 --2- 172.21.15.40:0/3611277069 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f95780a2800 0x7f95780a4cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f958c198fa0 tx=0x7f957402c7e0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.531+0000 7f9591110640 1 -- 172.21.15.40:0/3611277069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f958c1018b0 msgr2=0x7f958c197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.531+0000 7f9591110640 1 --2- 172.21.15.40:0/3611277069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f958c1018b0 0x7f958c197a90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f957c00e9c0 tx=0x7f957c00edc0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.531+0000 7f9591110640 1 -- 172.21.15.40:0/3611277069 shutdown_connections 2024-03-18T22:50:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.531+0000 7f9591110640 1 --2- 172.21.15.40:0/3611277069 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f95780a2800 0x7f95780a4cc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.531+0000 7f9591110640 1 --2- 172.21.15.40:0/3611277069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f958c103460 0x7f958c19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.531+0000 7f9591110640 1 --2- 172.21.15.40:0/3611277069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958c102aa0 0x7f958c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.531+0000 7f9591110640 1 --2- 172.21.15.40:0/3611277069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f958c1018b0 0x7f958c197a90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:15.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.531+0000 7f9591110640 1 -- 172.21.15.40:0/3611277069 >> 172.21.15.40:0/3611277069 conn(0x7f958c0fce40 msgr2=0x7f958c105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:15.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.531+0000 7f9591110640 1 -- 172.21.15.40:0/3611277069 shutdown_connections 2024-03-18T22:50:15.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:15.531+0000 7f9591110640 1 -- 172.21.15.40:0/3611277069 wait complete. 2024-03-18T22:50:16.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:16 smithi113 ceph-mon[66775]: pgmap v579: 129 pgs: 129 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:50:16.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:16 smithi103 ceph-mon[66377]: pgmap v579: 129 pgs: 129 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:50:16.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:16 smithi040 ceph-mon[62643]: pgmap v579: 129 pgs: 129 active+clean; 9.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:50:16.544 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:50:16.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:50:16.545 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:50:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.667+0000 7fd1a6bdf640 1 -- 172.21.15.40:0/3759389115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a01022d0 msgr2=0x7fd1a0108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.667+0000 7fd1a6bdf640 1 --2- 172.21.15.40:0/3759389115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a01022d0 0x7fd1a0108b60 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fd198009ca0 tx=0x7fd19802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.667+0000 7fd1a6bdf640 1 -- 172.21.15.40:0/3759389115 shutdown_connections 2024-03-18T22:50:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.667+0000 7fd1a6bdf640 1 --2- 172.21.15.40:0/3759389115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a01022d0 0x7fd1a0108b60 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.667+0000 7fd1a6bdf640 1 --2- 172.21.15.40:0/3759389115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1a0101910 0x7fd1a0101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.667+0000 7fd1a6bdf640 1 --2- 172.21.15.40:0/3759389115 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a0100720 0x7fd1a0100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.667+0000 7fd1a6bdf640 1 -- 172.21.15.40:0/3759389115 >> 172.21.15.40:0/3759389115 conn(0x7fd1a00fbcb0 msgr2=0x7fd1a00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.667+0000 7fd1a6bdf640 1 -- 172.21.15.40:0/3759389115 shutdown_connections 2024-03-18T22:50:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.667+0000 7fd1a6bdf640 1 -- 172.21.15.40:0/3759389115 wait complete. 2024-03-18T22:50:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a6bdf640 1 Processor -- start 2024-03-18T22:50:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a6bdf640 1 -- start start 2024-03-18T22:50:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a6bdf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0100720 0x7fd1a00694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a6bdf640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a0101910 0x7fd1a019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a6bdf640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1a01022d0 0x7fd1a019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a6bdf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a010c230 con 0x7fd1a0100720 2024-03-18T22:50:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a6bdf640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a010c0b0 con 0x7fd1a01022d0 2024-03-18T22:50:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a6bdf640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a010c3b0 con 0x7fd1a0101910 2024-03-18T22:50:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a497b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0100720 0x7fd1a00694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd19ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a0101910 0x7fd1a019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a497b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0100720 0x7fd1a00694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45970/0 (socket says 172.21.15.40:45970) 2024-03-18T22:50:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a517c640 1 --2- 172.21.15.40:0/492331794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1a01022d0 0x7fd1a019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a497b640 1 -- 172.21.15.40:0/492331794 learned_addr learned my addr 172.21.15.40:0/492331794 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a497b640 1 -- 172.21.15.40:0/492331794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a0101910 msgr2=0x7fd1a019c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a497b640 1 --2- 172.21.15.40:0/492331794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a0101910 0x7fd1a019c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a497b640 1 -- 172.21.15.40:0/492331794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1a01022d0 msgr2=0x7fd1a019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a497b640 1 --2- 172.21.15.40:0/492331794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1a01022d0 0x7fd1a019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a497b640 1 -- 172.21.15.40:0/492331794 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a019f6b0 con 0x7fd1a0100720 2024-03-18T22:50:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.668+0000 7fd1a517c640 1 --2- 172.21.15.40:0/492331794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1a01022d0 0x7fd1a019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.669+0000 7fd1a497b640 1 --2- 172.21.15.40:0/492331794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0100720 0x7fd1a00694f0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd19400e9c0 tx=0x7fd19400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.669+0000 7fd19dffb640 1 -- 172.21.15.40:0/492331794 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19400cd80 con 0x7fd1a0100720 2024-03-18T22:50:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.669+0000 7fd1a6bdf640 1 -- 172.21.15.40:0/492331794 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1a019f940 con 0x7fd1a0100720 2024-03-18T22:50:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.669+0000 7fd19dffb640 1 -- 172.21.15.40:0/492331794 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1940035f0 con 0x7fd1a0100720 2024-03-18T22:50:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.669+0000 7fd19dffb640 1 -- 172.21.15.40:0/492331794 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194010690 con 0x7fd1a0100720 2024-03-18T22:50:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.669+0000 7fd1a6bdf640 1 -- 172.21.15.40:0/492331794 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1a019fee0 con 0x7fd1a0100720 2024-03-18T22:50:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.671+0000 7fd19dffb640 1 -- 172.21.15.40:0/492331794 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd1940037a0 con 0x7fd1a0100720 2024-03-18T22:50:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd19dffb640 1 --2- 172.21.15.40:0/492331794 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1900a2800 0x7fd1900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd1a97ca740 1 Processor -- start 2024-03-18T22:50:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd19dffb640 1 -- 172.21.15.40:0/492331794 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd1940cf1e0 con 0x7fd1a0100720 2024-03-18T22:50:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd19ffff640 1 --2- 172.21.15.40:0/492331794 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1900a2800 0x7fd1900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd1a97ca740 1 -- start start 2024-03-18T22:50:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd1a97ca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aae67b0 0x559e2aae6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd1a97ca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 0x559e2aaebca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd1a97ca740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aaec310 0x559e2aaee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd1a97ca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559e2aa47160 con 0x559e2aae71a0 2024-03-18T22:50:16.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd1a97ca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559e2a9efe10 con 0x559e2aae67b0 2024-03-18T22:50:16.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd1a97ca740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559e2aa43200 con 0x559e2aaec310 2024-03-18T22:50:16.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd1a497b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aae67b0 0x559e2aae6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd1a497b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aae67b0 0x559e2aae6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43886/0 (socket says 172.21.15.40:43886) 2024-03-18T22:50:16.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd1a497b640 1 -- 172.21.15.40:0/256311440 learned_addr learned my addr 172.21.15.40:0/256311440 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:16.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd1a517c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aaec310 0x559e2aaee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.672+0000 7fd19ffff640 1 --2- 172.21.15.40:0/256311440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 0x559e2aaebca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.673+0000 7fd19ffff640 1 --2- 172.21.15.40:0/492331794 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1900a2800 0x7fd1900a4cc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0069260 tx=0x7fd18c011440 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:16.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.673+0000 7fd1a497b640 1 -- 172.21.15.40:0/256311440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aaec310 msgr2=0x559e2aaee7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:16.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.673+0000 7fd1a497b640 1 --2- 172.21.15.40:0/256311440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aaec310 0x559e2aaee7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.673+0000 7fd1a497b640 1 -- 172.21.15.40:0/256311440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 msgr2=0x559e2aaebca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:16.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.673+0000 7fd1a497b640 1 --2- 172.21.15.40:0/256311440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 0x559e2aaebca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.673+0000 7fd1a497b640 1 -- 172.21.15.40:0/256311440 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e2a8237e0 con 0x559e2aae67b0 2024-03-18T22:50:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.673+0000 7fd1a517c640 1 --2- 172.21.15.40:0/256311440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aaec310 0x559e2aaee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.673+0000 7fd1a497b640 1 --2- 172.21.15.40:0/256311440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aae67b0 0x559e2aae6bd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd1940a3a00 tx=0x7fd19409b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dcb59bd8bb64644 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.673+0000 7fd18affd640 1 -- 172.21.15.40:0/256311440 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1940b4070 con 0x559e2aae67b0 2024-03-18T22:50:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.673+0000 7fd18affd640 1 -- 172.21.15.40:0/256311440 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd19409b790 con 0x559e2aae67b0 2024-03-18T22:50:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.673+0000 7fd18affd640 1 -- 172.21.15.40:0/256311440 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194097e00 con 0x559e2aae67b0 2024-03-18T22:50:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.674+0000 7fd1a97ca740 1 -- 172.21.15.40:0/256311440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aae67b0 msgr2=0x559e2aae6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.674+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/256311440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aae67b0 0x559e2aae6bd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd1940a3a00 tx=0x7fd19409b390 comp rx=0 tx=0).stop 2024-03-18T22:50:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.674+0000 7fd1a97ca740 1 -- 172.21.15.40:0/256311440 shutdown_connections 2024-03-18T22:50:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.674+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/256311440 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aaec310 0x559e2aaee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.674+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/256311440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 0x559e2aaebca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.674+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/256311440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aae67b0 0x559e2aae6bd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.674+0000 7fd1a97ca740 1 -- 172.21.15.40:0/256311440 >> 172.21.15.40:0/256311440 conn(0x559e2aadd910 msgr2=0x559e2aaddd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.674+0000 7fd1a97ca740 1 -- 172.21.15.40:0/256311440 shutdown_connections 2024-03-18T22:50:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.674+0000 7fd1a97ca740 1 -- 172.21.15.40:0/256311440 wait complete. 2024-03-18T22:50:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.674+0000 7fd1a97ca740 1 -- start start 2024-03-18T22:50:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.674+0000 7fd1a97ca740 1 Processor -- start 2024-03-18T22:50:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd1a97ca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aae67b0 0x559e2aaf0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd1a97ca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 0x559e2aaf11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd1a97ca740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aaec310 0x559e2aaf81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd1a97ca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559e2aaef2c0 con 0x559e2aae71a0 2024-03-18T22:50:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd1a97ca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559e2aa031a0 con 0x559e2aae67b0 2024-03-18T22:50:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd1a97ca740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559e2aaef440 con 0x559e2aaec310 2024-03-18T22:50:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd1a97ca740 10 client.0 _ll_register_callbacks cb 0x559e2a75ac60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd1a497b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aae67b0 0x559e2aaf0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd19ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 0x559e2aaf11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd1a497b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aae67b0 0x559e2aaf0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:43888/0 (socket says 172.21.15.40:43888) 2024-03-18T22:50:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd1a497b640 1 -- 172.21.15.40:0/1342173881 learned_addr learned my addr 172.21.15.40:0/1342173881 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd1a517c640 1 --2- 172.21.15.40:0/1342173881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aaec310 0x559e2aaf81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd19ffff640 1 -- 172.21.15.40:0/1342173881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aaec310 msgr2=0x559e2aaf81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd19ffff640 1 --2- 172.21.15.40:0/1342173881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aaec310 0x559e2aaf81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd19ffff640 1 -- 172.21.15.40:0/1342173881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aae67b0 msgr2=0x559e2aaf0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd19ffff640 1 --2- 172.21.15.40:0/1342173881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aae67b0 0x559e2aaf0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd19ffff640 1 -- 172.21.15.40:0/1342173881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e2a883f70 con 0x559e2aae71a0 2024-03-18T22:50:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd19ffff640 1 --2- 172.21.15.40:0/1342173881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 0x559e2aaf11b0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009a90 tx=0x7fd18c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32df42b0f3aa91a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.675+0000 7fd17ffff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.676+0000 7fd17ffff640 1 -- 172.21.15.40:0/1342173881 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18c017040 con 0x559e2aae71a0 2024-03-18T22:50:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.676+0000 7fd17ffff640 1 -- 172.21.15.40:0/1342173881 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd18c005020 con 0x559e2aae71a0 2024-03-18T22:50:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.676+0000 7fd1a97ca740 1 -- 172.21.15.40:0/1342173881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd19409b790 con 0x559e2aae71a0 2024-03-18T22:50:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.676+0000 7fd17ffff640 1 -- 172.21.15.40:0/1342173881 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fd18c005220 con 0x559e2aae71a0 2024-03-18T22:50:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.676+0000 7fd1a97ca740 10 client.17580 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.676+0000 7fd1a97ca740 1 -- 172.21.15.40:0/1342173881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559e2a71b170 con 0x559e2aae71a0 2024-03-18T22:50:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.676+0000 7fd17ffff640 1 -- 172.21.15.40:0/1342173881 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd19409b790 con 0x559e2aae71a0 2024-03-18T22:50:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.676+0000 7fd17ffff640 1 -- 172.21.15.40:0/1342173881 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fd18c025070 con 0x559e2aae71a0 2024-03-18T22:50:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.676+0000 7fd1a97ca740 10 client.17580 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.676+0000 7fd1a97ca740 10 client.17580 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.676+0000 7fd1a97ca740 20 client.17580 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.676+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/1342173881 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559e2aafcb20 0x559e2aafefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.676+0000 7fd1a97ca740 4 client.17580 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.676+0000 7fd1a97ca740 1 -- 172.21.15.40:0/1342173881 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e2aa546a0 con 0x559e2aafcb20 2024-03-18T22:50:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.677+0000 7fd1a497b640 1 --2- 172.21.15.40:0/1342173881 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559e2aafcb20 0x559e2aafefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.677+0000 7fd17ffff640 10 client.17580 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.677+0000 7fd1a497b640 1 --2- 172.21.15.40:0/1342173881 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559e2aafcb20 0x559e2aafefa0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ebdc1c95d067eb5 server_cookie=88a3f490e58d3c9e in_seq=0 out_seq=0 2024-03-18T22:50:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.678+0000 7fd17ffff640 1 -- 172.21.15.40:0/1342173881 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559e2aa546a0 con 0x559e2aafcb20 2024-03-18T22:50:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.678+0000 7fd17ffff640 10 client.17580 handle_command_reply: tid=0 2024-03-18T22:50:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.678+0000 7fd1a97ca740 1 client.17580 shutdown 2024-03-18T22:50:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.678+0000 7fd1a97ca740 1 -- 172.21.15.40:0/1342173881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 msgr2=0x559e2aaf11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.678+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/1342173881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 0x559e2aaf11b0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009a90 tx=0x7fd18c004730 comp rx=0 tx=0).stop 2024-03-18T22:50:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.678+0000 7fd1a97ca740 1 -- 172.21.15.40:0/1342173881 shutdown_connections 2024-03-18T22:50:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.678+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/1342173881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aaec310 0x559e2aaf81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.678+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/1342173881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 0x559e2aaf11b0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.678+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/1342173881 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559e2aafcb20 0x559e2aafefa0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.678+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/1342173881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aae67b0 0x559e2aaf0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.678+0000 7fd17ffff640 0 client.17580 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.678+0000 7fd1a97ca740 1 -- 172.21.15.40:0/1342173881 >> 172.21.15.40:0/1342173881 conn(0x559e2aadd910 msgr2=0x559e2aadcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.678+0000 7fd1a97ca740 1 -- 172.21.15.40:0/1342173881 shutdown_connections 2024-03-18T22:50:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.678+0000 7fd1a97ca740 1 -- 172.21.15.40:0/1342173881 wait complete. 2024-03-18T22:50:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.678+0000 7fd1a97ca740 20 client.17580 trim_cache size 0 max 16384 2024-03-18T22:50:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.698+0000 7fd1a97ca740 1 Processor -- start 2024-03-18T22:50:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.698+0000 7fd1a97ca740 1 -- start start 2024-03-18T22:50:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.698+0000 7fd1a97ca740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aae67b0 0x559e2aae1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.698+0000 7fd1a97ca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 0x559e2aaf81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.698+0000 7fd1a97ca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aaec310 0x559e2ab03860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.698+0000 7fd1a97ca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01a8390 con 0x559e2aae71a0 2024-03-18T22:50:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.698+0000 7fd1a97ca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01a9f90 con 0x559e2aaec310 2024-03-18T22:50:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.698+0000 7fd1a97ca740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01a8a60 con 0x559e2aae67b0 2024-03-18T22:50:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.698+0000 7fd19ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 0x559e2aaf81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.698+0000 7fd1a517c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aaec310 0x559e2ab03860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.698+0000 7fd19ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 0x559e2aaf81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46010/0 (socket says 172.21.15.40:46010) 2024-03-18T22:50:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.698+0000 7fd19ffff640 1 -- 172.21.15.40:0/2756408107 learned_addr learned my addr 172.21.15.40:0/2756408107 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.698+0000 7fd1a497b640 1 --2- 172.21.15.40:0/2756408107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aae67b0 0x559e2aae1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.699+0000 7fd1a517c640 1 -- 172.21.15.40:0/2756408107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aae67b0 msgr2=0x559e2aae1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.699+0000 7fd1a517c640 1 --2- 172.21.15.40:0/2756408107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aae67b0 0x559e2aae1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.699+0000 7fd1a517c640 1 -- 172.21.15.40:0/2756408107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 msgr2=0x559e2aaf81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.699+0000 7fd1a517c640 1 --2- 172.21.15.40:0/2756408107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 0x559e2aaf81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.699+0000 7fd1a517c640 1 -- 172.21.15.40:0/2756408107 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e2aae7890 con 0x559e2aaec310 2024-03-18T22:50:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.699+0000 7fd1a517c640 1 --2- 172.21.15.40:0/2756408107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aaec310 0x559e2ab03860 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd19802c630 tx=0x7fd1980324f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=115244819ad6655a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.699+0000 7fd18a7fc640 1 -- 172.21.15.40:0/2756408107 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd198045070 con 0x559e2aaec310 2024-03-18T22:50:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.699+0000 7fd18a7fc640 1 -- 172.21.15.40:0/2756408107 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd198003e20 con 0x559e2aaec310 2024-03-18T22:50:16.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.699+0000 7fd18a7fc640 1 -- 172.21.15.40:0/2756408107 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd198039040 con 0x559e2aaec310 2024-03-18T22:50:16.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.700+0000 7fd1a97ca740 1 -- 172.21.15.40:0/2756408107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aaec310 msgr2=0x559e2ab03860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.700+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/2756408107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aaec310 0x559e2ab03860 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd19802c630 tx=0x7fd1980324f0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.700+0000 7fd1a97ca740 1 -- 172.21.15.40:0/2756408107 shutdown_connections 2024-03-18T22:50:16.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.700+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/2756408107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aaec310 0x559e2ab03860 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.700+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/2756408107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae71a0 0x559e2aaf81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.700+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/2756408107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aae67b0 0x559e2aae1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.700+0000 7fd1a97ca740 1 -- 172.21.15.40:0/2756408107 >> 172.21.15.40:0/2756408107 conn(0x559e2aadd910 msgr2=0x559e2aae0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.700+0000 7fd1a97ca740 1 -- 172.21.15.40:0/2756408107 shutdown_connections 2024-03-18T22:50:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.700+0000 7fd1a97ca740 1 -- 172.21.15.40:0/2756408107 wait complete. 2024-03-18T22:50:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.700+0000 7fd1a97ca740 1 -- start start 2024-03-18T22:50:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.701+0000 7fd1a97ca740 1 Processor -- start 2024-03-18T22:50:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.701+0000 7fd1a97ca740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae67b0 0x559e2aaf81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.701+0000 7fd1a97ca740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aae71a0 0x559e2aaf67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.701+0000 7fd1a97ca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aaec310 0x559e2aaf6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.701+0000 7fd1a97ca740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01a98c0 con 0x559e2aae67b0 2024-03-18T22:50:16.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.701+0000 7fd1a97ca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01a9130 con 0x559e2aaec310 2024-03-18T22:50:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.701+0000 7fd1a97ca740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a01a6370 con 0x559e2aae71a0 2024-03-18T22:50:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.701+0000 7fd1a97ca740 10 client.0 _ll_register_callbacks cb 0x7fd18c00f4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.701+0000 7fd1a497b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae67b0 0x559e2aaf81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.701+0000 7fd1a517c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aaec310 0x559e2aaf6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.701+0000 7fd1a497b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae67b0 0x559e2aaf81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46020/0 (socket says 172.21.15.40:46020) 2024-03-18T22:50:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.701+0000 7fd1a497b640 1 -- 172.21.15.40:0/980663001 learned_addr learned my addr 172.21.15.40:0/980663001 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.702+0000 7fd1a497b640 1 -- 172.21.15.40:0/980663001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aae71a0 msgr2=0x559e2aaf67e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:50:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.702+0000 7fd1a497b640 1 --2- 172.21.15.40:0/980663001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aae71a0 0x559e2aaf67e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.702+0000 7fd1a497b640 1 -- 172.21.15.40:0/980663001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aaec310 msgr2=0x559e2aaf6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.702+0000 7fd1a497b640 1 --2- 172.21.15.40:0/980663001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aaec310 0x559e2aaf6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.702+0000 7fd1a497b640 1 -- 172.21.15.40:0/980663001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e2ab62ea0 con 0x559e2aae67b0 2024-03-18T22:50:16.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.702+0000 7fd1a517c640 1 --2- 172.21.15.40:0/980663001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aaec310 0x559e2aaf6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.702+0000 7fd1a497b640 1 --2- 172.21.15.40:0/980663001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae67b0 0x559e2aaf81d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd19409b490 tx=0x7fd1940cf170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8718a0aca7590d7b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.702+0000 7fd17ffff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.702+0000 7fd17ffff640 1 -- 172.21.15.40:0/980663001 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19407f060 con 0x559e2aae67b0 2024-03-18T22:50:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.702+0000 7fd17ffff640 1 -- 172.21.15.40:0/980663001 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd194099830 con 0x559e2aae67b0 2024-03-18T22:50:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.702+0000 7fd1a97ca740 1 -- 172.21.15.40:0/980663001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd198003e20 con 0x559e2aae67b0 2024-03-18T22:50:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.702+0000 7fd17ffff640 1 -- 172.21.15.40:0/980663001 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fd1940b4040 con 0x559e2aae67b0 2024-03-18T22:50:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.702+0000 7fd1a97ca740 10 client.17586 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.702+0000 7fd1a97ca740 1 -- 172.21.15.40:0/980663001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559e2ab64040 con 0x559e2aae67b0 2024-03-18T22:50:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.703+0000 7fd17ffff640 1 -- 172.21.15.40:0/980663001 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd198003e20 con 0x559e2aae67b0 2024-03-18T22:50:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.703+0000 7fd17ffff640 1 -- 172.21.15.40:0/980663001 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fd1940a8020 con 0x559e2aae67b0 2024-03-18T22:50:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.703+0000 7fd1a97ca740 10 client.17586 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.703+0000 7fd1a97ca740 10 client.17586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.703+0000 7fd1a97ca740 20 client.17586 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:16.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.703+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/980663001 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559e2ab645b0 0x559e2ab669a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.703+0000 7fd1a97ca740 4 client.17586 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:50:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.703+0000 7fd1a97ca740 1 -- 172.21.15.40:0/980663001 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559e2aa4e9f0 con 0x559e2ab645b0 2024-03-18T22:50:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.703+0000 7fd19ffff640 1 --2- 172.21.15.40:0/980663001 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559e2ab645b0 0x559e2ab669a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.704+0000 7fd17ffff640 10 client.17586 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:16.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.704+0000 7fd19ffff640 1 --2- 172.21.15.40:0/980663001 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559e2ab645b0 0x559e2ab669a0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70873f0ebc5e615c server_cookie=68b4a239333bc31f in_seq=0 out_seq=0 2024-03-18T22:50:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.705+0000 7fd17ffff640 1 -- 172.21.15.40:0/980663001 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559e2aa4e9f0 con 0x559e2ab645b0 2024-03-18T22:50:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.705+0000 7fd17ffff640 10 client.17586 handle_command_reply: tid=0 2024-03-18T22:50:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.705+0000 7fd1a97ca740 1 client.17586 shutdown 2024-03-18T22:50:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.706+0000 7fd1a97ca740 1 -- 172.21.15.40:0/980663001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae67b0 msgr2=0x559e2aaf81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.706+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/980663001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae67b0 0x559e2aaf81d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd19409b490 tx=0x7fd1940cf170 comp rx=0 tx=0).stop 2024-03-18T22:50:16.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.706+0000 7fd1a97ca740 1 -- 172.21.15.40:0/980663001 shutdown_connections 2024-03-18T22:50:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.706+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/980663001 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559e2ab645b0 0x559e2ab669a0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.706+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/980663001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559e2aaec310 0x559e2aaf6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.706+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/980663001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559e2aae71a0 0x559e2aaf67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.706+0000 7fd1a97ca740 1 --2- 172.21.15.40:0/980663001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559e2aae67b0 0x559e2aaf81d0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.706+0000 7fd17ffff640 0 client.17586 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.706+0000 7fd1a97ca740 1 -- 172.21.15.40:0/980663001 >> 172.21.15.40:0/980663001 conn(0x559e2aadd910 msgr2=0x559e2aae1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.706+0000 7fd1a97ca740 1 -- 172.21.15.40:0/980663001 shutdown_connections 2024-03-18T22:50:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.706+0000 7fd1a97ca740 1 -- 172.21.15.40:0/980663001 wait complete. 2024-03-18T22:50:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.706+0000 7fd1a97ca740 20 client.17586 trim_cache size 0 max 16384 2024-03-18T22:50:16.711 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:50:16.712 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:50:16.712 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "ef7a13e3-ab20-4fa8-a202-d042c612cfe4", 2024-03-18T22:50:16.712 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:50:16.712 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:50:16.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.707+0000 7fd1a6bdf640 1 -- 172.21.15.40:0/492331794 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1900a2800 msgr2=0x7fd1900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.707+0000 7fd1a6bdf640 1 --2- 172.21.15.40:0/492331794 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1900a2800 0x7fd1900a4cc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0069260 tx=0x7fd18c011440 comp rx=0 tx=0).stop 2024-03-18T22:50:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.707+0000 7fd1a6bdf640 1 -- 172.21.15.40:0/492331794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0100720 msgr2=0x7fd1a00694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.707+0000 7fd1a6bdf640 1 --2- 172.21.15.40:0/492331794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0100720 0x7fd1a00694f0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd19400e9c0 tx=0x7fd19400ee80 comp rx=0 tx=0).stop 2024-03-18T22:50:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.707+0000 7fd1a6bdf640 1 -- 172.21.15.40:0/492331794 shutdown_connections 2024-03-18T22:50:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.707+0000 7fd1a6bdf640 1 --2- 172.21.15.40:0/492331794 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1900a2800 0x7fd1900a4cc0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.707+0000 7fd1a6bdf640 1 --2- 172.21.15.40:0/492331794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1a01022d0 0x7fd1a019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.707+0000 7fd1a6bdf640 1 --2- 172.21.15.40:0/492331794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1a0101910 0x7fd1a019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.707+0000 7fd1a6bdf640 1 --2- 172.21.15.40:0/492331794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1a0100720 0x7fd1a00694f0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.707+0000 7fd1a6bdf640 1 -- 172.21.15.40:0/492331794 >> 172.21.15.40:0/492331794 conn(0x7fd1a00fbcb0 msgr2=0x7fd1a0104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.707+0000 7fd1a6bdf640 1 -- 172.21.15.40:0/492331794 shutdown_connections 2024-03-18T22:50:16.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.707+0000 7fd1a6bdf640 1 -- 172.21.15.40:0/492331794 wait complete. 2024-03-18T22:50:16.721 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:50:16.721 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:50:16.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.833+0000 7f245aabf640 1 -- 172.21.15.40:0/942428728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2454101910 msgr2=0x7f2454101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.833+0000 7f245aabf640 1 --2- 172.21.15.40:0/942428728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2454101910 0x7f2454101d90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2440009900 tx=0x7f244002bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:16.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.833+0000 7f245aabf640 1 -- 172.21.15.40:0/942428728 shutdown_connections 2024-03-18T22:50:16.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.833+0000 7f245aabf640 1 --2- 172.21.15.40:0/942428728 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24541022d0 0x7f2454108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.833+0000 7f245aabf640 1 --2- 172.21.15.40:0/942428728 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2454101910 0x7f2454101d90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.833+0000 7f245aabf640 1 --2- 172.21.15.40:0/942428728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2454100720 0x7f2454100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.833+0000 7f245aabf640 1 -- 172.21.15.40:0/942428728 >> 172.21.15.40:0/942428728 conn(0x7f24540fbcb0 msgr2=0x7f24540fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:16.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.833+0000 7f245aabf640 1 -- 172.21.15.40:0/942428728 shutdown_connections 2024-03-18T22:50:16.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.834+0000 7f245aabf640 1 -- 172.21.15.40:0/942428728 wait complete. 2024-03-18T22:50:16.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.834+0000 7f245aabf640 1 Processor -- start 2024-03-18T22:50:16.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.834+0000 7f245aabf640 1 -- start start 2024-03-18T22:50:16.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.834+0000 7f245aabf640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2454100720 0x7f24541046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.834+0000 7f245aabf640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2454101910 0x7f2454102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.834+0000 7f245aabf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24541022d0 0x7f2454103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.834+0000 7f245aabf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f245410c230 con 0x7f24541022d0 2024-03-18T22:50:16.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.834+0000 7f245aabf640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f245410c0b0 con 0x7f2454100720 2024-03-18T22:50:16.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.834+0000 7f245aabf640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f245410c3b0 con 0x7f2454101910 2024-03-18T22:50:16.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.834+0000 7f245905c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24541022d0 0x7f2454103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.834+0000 7f245905c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24541022d0 0x7f2454103230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46040/0 (socket says 172.21.15.40:46040) 2024-03-18T22:50:16.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.834+0000 7f245905c640 1 -- 172.21.15.40:0/809726059 learned_addr learned my addr 172.21.15.40:0/809726059 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:16.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.834+0000 7f245885b640 1 --2- 172.21.15.40:0/809726059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2454100720 0x7f24541046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.834+0000 7f2453fff640 1 --2- 172.21.15.40:0/809726059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2454101910 0x7f2454102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.835+0000 7f245905c640 1 -- 172.21.15.40:0/809726059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2454101910 msgr2=0x7f2454102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.835+0000 7f245905c640 1 --2- 172.21.15.40:0/809726059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2454101910 0x7f2454102cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.835+0000 7f245905c640 1 -- 172.21.15.40:0/809726059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2454100720 msgr2=0x7f24541046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.835+0000 7f245905c640 1 --2- 172.21.15.40:0/809726059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2454100720 0x7f24541046b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.835+0000 7f245905c640 1 -- 172.21.15.40:0/809726059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2454103ac0 con 0x7f24541022d0 2024-03-18T22:50:16.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.835+0000 7f245885b640 1 --2- 172.21.15.40:0/809726059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2454100720 0x7f24541046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:16.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.835+0000 7f245905c640 1 --2- 172.21.15.40:0/809726059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24541022d0 0x7f2454103230 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f244c00efd0 tx=0x7f244c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:16.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.835+0000 7f2451ffb640 1 -- 172.21.15.40:0/809726059 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c003b70 con 0x7f24541022d0 2024-03-18T22:50:16.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.835+0000 7f245aabf640 1 -- 172.21.15.40:0/809726059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24541a1c30 con 0x7f24541022d0 2024-03-18T22:50:16.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.835+0000 7f2451ffb640 1 -- 172.21.15.40:0/809726059 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f244c003d10 con 0x7f24541022d0 2024-03-18T22:50:16.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.835+0000 7f2451ffb640 1 -- 172.21.15.40:0/809726059 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c010980 con 0x7f24541022d0 2024-03-18T22:50:16.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.835+0000 7f245aabf640 1 -- 172.21.15.40:0/809726059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24541a2100 con 0x7f24541022d0 2024-03-18T22:50:16.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f2451ffb640 1 -- 172.21.15.40:0/809726059 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f244c004020 con 0x7f24541022d0 2024-03-18T22:50:16.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f245d782740 1 Processor -- start 2024-03-18T22:50:16.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f245d782740 1 -- start start 2024-03-18T22:50:16.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f245d782740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 0x560d734d6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f245d782740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734d7440 0x560d734dbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f245d782740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734dc5b0 0x560d734dea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f2451ffb640 1 --2- 172.21.15.40:0/809726059 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f24440a2800 0x7f24440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f245d782740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560d7343b500 con 0x560d734dc5b0 2024-03-18T22:50:16.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f245d782740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560d733e3be0 con 0x560d734d7440 2024-03-18T22:50:16.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f245d782740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560d734375a0 con 0x560d734d6a50 2024-03-18T22:50:16.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f2453fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 0x560d734d6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f245905c640 1 --2- 172.21.15.40:0/809726059 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f24440a2800 0x7f24440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f2453fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 0x560d734d6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:54204/0 (socket says 172.21.15.40:54204) 2024-03-18T22:50:16.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f245885b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734d7440 0x560d734dbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f2453fff640 1 -- 172.21.15.40:0/3363015073 learned_addr learned my addr 172.21.15.40:0/3363015073 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:16.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.838+0000 7f2453fff640 1 --2- 172.21.15.40:0/3363015073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734dc5b0 0x560d734dea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.839+0000 7f2451ffb640 1 -- 172.21.15.40:0/809726059 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f244c0cf2b0 con 0x7f24541022d0 2024-03-18T22:50:16.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.839+0000 7f245885b640 1 -- 172.21.15.40:0/3363015073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 msgr2=0x560d734d6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:16.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.839+0000 7f245885b640 1 --2- 172.21.15.40:0/3363015073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 0x560d734d6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.839+0000 7f245885b640 1 -- 172.21.15.40:0/3363015073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734dc5b0 msgr2=0x560d734dea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:16.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.839+0000 7f245885b640 1 --2- 172.21.15.40:0/3363015073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734dc5b0 0x560d734dea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.839+0000 7f245885b640 1 -- 172.21.15.40:0/3363015073 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d73405000 con 0x560d734d7440 2024-03-18T22:50:16.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.839+0000 7f245905c640 1 --2- 172.21.15.40:0/809726059 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f24440a2800 0x7f24440a4cc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f244c0ccd30 tx=0x7f244c0c62b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:16.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.839+0000 7f245885b640 1 --2- 172.21.15.40:0/3363015073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734d7440 0x560d734dbf40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2448009fa0 tx=0x7f24480118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=803d89340f01941e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:16.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.839+0000 7f243effd640 1 -- 172.21.15.40:0/3363015073 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244800f540 con 0x560d734d7440 2024-03-18T22:50:16.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.839+0000 7f243effd640 1 -- 172.21.15.40:0/3363015073 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f244800fb90 con 0x560d734d7440 2024-03-18T22:50:16.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.839+0000 7f243effd640 1 -- 172.21.15.40:0/3363015073 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2448019b90 con 0x560d734d7440 2024-03-18T22:50:16.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.840+0000 7f245d782740 1 -- 172.21.15.40:0/3363015073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734d7440 msgr2=0x560d734dbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.840+0000 7f245d782740 1 --2- 172.21.15.40:0/3363015073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734d7440 0x560d734dbf40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2448009fa0 tx=0x7f24480118d0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.840+0000 7f245d782740 1 -- 172.21.15.40:0/3363015073 shutdown_connections 2024-03-18T22:50:16.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.840+0000 7f245d782740 1 --2- 172.21.15.40:0/3363015073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734dc5b0 0x560d734dea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.840+0000 7f245d782740 1 --2- 172.21.15.40:0/3363015073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734d7440 0x560d734dbf40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.840+0000 7f245d782740 1 --2- 172.21.15.40:0/3363015073 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 0x560d734d6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.840+0000 7f245d782740 1 -- 172.21.15.40:0/3363015073 >> 172.21.15.40:0/3363015073 conn(0x560d734d1fc0 msgr2=0x560d734d23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:16.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.840+0000 7f245d782740 1 -- 172.21.15.40:0/3363015073 shutdown_connections 2024-03-18T22:50:16.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.840+0000 7f245d782740 1 -- 172.21.15.40:0/3363015073 wait complete. 2024-03-18T22:50:16.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.840+0000 7f245d782740 1 -- start start 2024-03-18T22:50:16.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f245d782740 1 Processor -- start 2024-03-18T22:50:16.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f245d782740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 0x560d734e0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f245d782740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d7440 0x560d734e14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f245d782740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734dc5b0 0x560d734e84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f245d782740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560d734df560 con 0x560d734d7440 2024-03-18T22:50:16.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f245d782740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560d733f7160 con 0x560d734dc5b0 2024-03-18T22:50:16.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f245d782740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560d734df6e0 con 0x560d734d6a50 2024-03-18T22:50:16.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f245d782740 10 client.0 _ll_register_callbacks cb 0x560d73433a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:16.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f245885b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d7440 0x560d734e14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f2453fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 0x560d734e0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f2453fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 0x560d734e0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:54210/0 (socket says 172.21.15.40:54210) 2024-03-18T22:50:16.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f2453fff640 1 -- 172.21.15.40:0/2678526848 learned_addr learned my addr 172.21.15.40:0/2678526848 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:16.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f2453fff640 1 --2- 172.21.15.40:0/2678526848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734dc5b0 0x560d734e84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f2453fff640 1 -- 172.21.15.40:0/2678526848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734dc5b0 msgr2=0x560d734e84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:16.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f2453fff640 1 --2- 172.21.15.40:0/2678526848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734dc5b0 0x560d734e84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f2453fff640 1 -- 172.21.15.40:0/2678526848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d7440 msgr2=0x560d734e14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:16.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f2453fff640 1 --2- 172.21.15.40:0/2678526848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d7440 0x560d734e14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f2453fff640 1 -- 172.21.15.40:0/2678526848 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d732197d0 con 0x560d734d6a50 2024-03-18T22:50:16.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.841+0000 7f245885b640 1 --2- 172.21.15.40:0/2678526848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d7440 0x560d734e14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:16.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f2453fff640 1 --2- 172.21.15.40:0/2678526848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 0x560d734e0fa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2440000c00 tx=0x7f2440005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e097dc11a1864a12 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:16.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f2433fff640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:50:16.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f2433fff640 1 -- 172.21.15.40:0/2678526848 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24400046c0 con 0x560d734d6a50 2024-03-18T22:50:16.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f245d782740 1 -- 172.21.15.40:0/2678526848 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f244800fb90 con 0x560d734d6a50 2024-03-18T22:50:16.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f2433fff640 1 -- 172.21.15.40:0/2678526848 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2440004860 con 0x560d734d6a50 2024-03-18T22:50:16.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f2433fff640 1 -- 172.21.15.40:0/2678526848 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f2440057050 con 0x560d734d6a50 2024-03-18T22:50:16.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f245d782740 10 client.27403 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:16.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f245d782740 1 -- 172.21.15.40:0/2678526848 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560d7314f3f0 con 0x560d734d6a50 2024-03-18T22:50:16.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f2433fff640 1 -- 172.21.15.40:0/2678526848 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f244800fb90 con 0x560d734d6a50 2024-03-18T22:50:16.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f2433fff640 1 -- 172.21.15.40:0/2678526848 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f244004e070 con 0x560d734d6a50 2024-03-18T22:50:16.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f245d782740 10 client.27403 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:16.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f245d782740 10 client.27403 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:16.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f245d782740 20 client.27403 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f245d782740 1 --2- 172.21.15.40:0/2678526848 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d734ecd00 0x560d734ef170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f245d782740 4 client.27403 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.842+0000 7f245d782740 1 -- 172.21.15.40:0/2678526848 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d73448a40 con 0x560d734ecd00 2024-03-18T22:50:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.843+0000 7f245885b640 1 --2- 172.21.15.40:0/2678526848 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d734ecd00 0x560d734ef170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.843+0000 7f2433fff640 10 client.27403 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.843+0000 7f245885b640 1 --2- 172.21.15.40:0/2678526848 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d734ecd00 0x560d734ef170 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc4dca6e3d15ddc9 server_cookie=b5c3b7bb319ad656 in_seq=0 out_seq=0 2024-03-18T22:50:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.844+0000 7f2433fff640 1 -- 172.21.15.40:0/2678526848 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560d73448a40 con 0x560d734ecd00 2024-03-18T22:50:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.844+0000 7f2433fff640 10 client.27403 handle_command_reply: tid=0 2024-03-18T22:50:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.844+0000 7f245d782740 1 client.27403 shutdown 2024-03-18T22:50:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.844+0000 7f245d782740 1 -- 172.21.15.40:0/2678526848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 msgr2=0x560d734e0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.844+0000 7f245d782740 1 --2- 172.21.15.40:0/2678526848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 0x560d734e0fa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2440000c00 tx=0x7f2440005d50 comp rx=0 tx=0).stop 2024-03-18T22:50:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.844+0000 7f245d782740 1 -- 172.21.15.40:0/2678526848 shutdown_connections 2024-03-18T22:50:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.844+0000 7f245d782740 1 --2- 172.21.15.40:0/2678526848 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d734ecd00 0x560d734ef170 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.844+0000 7f245d782740 1 --2- 172.21.15.40:0/2678526848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734dc5b0 0x560d734e84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.844+0000 7f245d782740 1 --2- 172.21.15.40:0/2678526848 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d7440 0x560d734e14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.844+0000 7f2433fff640 0 client.27403 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:16.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.844+0000 7f245d782740 1 --2- 172.21.15.40:0/2678526848 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 0x560d734e0fa0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.844+0000 7f245d782740 1 -- 172.21.15.40:0/2678526848 >> 172.21.15.40:0/2678526848 conn(0x560d734d1fc0 msgr2=0x560d734d11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:16.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.844+0000 7f245d782740 1 -- 172.21.15.40:0/2678526848 shutdown_connections 2024-03-18T22:50:16.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.844+0000 7f245d782740 1 -- 172.21.15.40:0/2678526848 wait complete. 2024-03-18T22:50:16.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.844+0000 7f245d782740 20 client.27403 trim_cache size 0 max 16384 2024-03-18T22:50:16.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.864+0000 7f245d782740 1 Processor -- start 2024-03-18T22:50:16.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.864+0000 7f245d782740 1 -- start start 2024-03-18T22:50:16.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.864+0000 7f245d782740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d6a50 0x560d734e1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.864+0000 7f245d782740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d7440 0x560d734e8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.864+0000 7f245d782740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734dc5b0 0x560d734d5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.864+0000 7f245d782740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f24541ac6c0 con 0x560d734d6a50 2024-03-18T22:50:16.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.864+0000 7f245d782740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24541ae2c0 con 0x560d734dc5b0 2024-03-18T22:50:16.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.864+0000 7f245d782740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f24541acd90 con 0x560d734d7440 2024-03-18T22:50:16.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.864+0000 7f2453fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d6a50 0x560d734e1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.864+0000 7f2453fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d6a50 0x560d734e1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46074/0 (socket says 172.21.15.40:46074) 2024-03-18T22:50:16.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.864+0000 7f245885b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d7440 0x560d734e8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.864+0000 7f2453fff640 1 -- 172.21.15.40:0/1459061055 learned_addr learned my addr 172.21.15.40:0/1459061055 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:16.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.864+0000 7f2453fff640 1 --2- 172.21.15.40:0/1459061055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734dc5b0 0x560d734d5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.865+0000 7f2453fff640 1 -- 172.21.15.40:0/1459061055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d7440 msgr2=0x560d734e8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:16.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.865+0000 7f2453fff640 1 --2- 172.21.15.40:0/1459061055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d7440 0x560d734e8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.865+0000 7f2453fff640 1 -- 172.21.15.40:0/1459061055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734dc5b0 msgr2=0x560d734d5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:16.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.865+0000 7f2453fff640 1 --2- 172.21.15.40:0/1459061055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734dc5b0 0x560d734d5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.865+0000 7f2453fff640 1 -- 172.21.15.40:0/1459061055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d7310f0f0 con 0x560d734d6a50 2024-03-18T22:50:16.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.865+0000 7f2453fff640 1 --2- 172.21.15.40:0/1459061055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d6a50 0x560d734e1440 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2440003bc0 tx=0x7f2440005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58c8e19c75ad6df server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:16.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.865+0000 7f243e7fc640 1 -- 172.21.15.40:0/1459061055 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2440003fc0 con 0x560d734d6a50 2024-03-18T22:50:16.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.865+0000 7f243e7fc640 1 -- 172.21.15.40:0/1459061055 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24400035f0 con 0x560d734d6a50 2024-03-18T22:50:16.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.865+0000 7f243e7fc640 1 -- 172.21.15.40:0/1459061055 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24400506d0 con 0x560d734d6a50 2024-03-18T22:50:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.866+0000 7f245d782740 1 -- 172.21.15.40:0/1459061055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d6a50 msgr2=0x560d734e1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.866+0000 7f245d782740 1 --2- 172.21.15.40:0/1459061055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d6a50 0x560d734e1440 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2440003bc0 tx=0x7f2440005fd0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.866+0000 7f245d782740 1 -- 172.21.15.40:0/1459061055 shutdown_connections 2024-03-18T22:50:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.866+0000 7f245d782740 1 --2- 172.21.15.40:0/1459061055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734dc5b0 0x560d734d5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.866+0000 7f245d782740 1 --2- 172.21.15.40:0/1459061055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d7440 0x560d734e8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.866+0000 7f245d782740 1 --2- 172.21.15.40:0/1459061055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d6a50 0x560d734e1440 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.866+0000 7f245d782740 1 -- 172.21.15.40:0/1459061055 >> 172.21.15.40:0/1459061055 conn(0x560d734d1fc0 msgr2=0x560d734d5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.866+0000 7f245d782740 1 -- 172.21.15.40:0/1459061055 shutdown_connections 2024-03-18T22:50:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.866+0000 7f245d782740 1 -- 172.21.15.40:0/1459061055 wait complete. 2024-03-18T22:50:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.866+0000 7f245d782740 1 -- start start 2024-03-18T22:50:16.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.866+0000 7f245d782740 1 Processor -- start 2024-03-18T22:50:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245d782740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 0x560d734e5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245d782740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d7440 0x560d734e6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245d782740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734dc5b0 0x560d734e7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245d782740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f24541adbf0 con 0x560d734d7440 2024-03-18T22:50:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245d782740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24541ad460 con 0x560d734dc5b0 2024-03-18T22:50:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245d782740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f24541aa6a0 con 0x560d734d6a50 2024-03-18T22:50:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245d782740 10 client.0 _ll_register_callbacks cb 0x7f2440005040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245885b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d7440 0x560d734e6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245885b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d7440 0x560d734e6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46082/0 (socket says 172.21.15.40:46082) 2024-03-18T22:50:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245885b640 1 -- 172.21.15.40:0/3917437326 learned_addr learned my addr 172.21.15.40:0/3917437326 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f2453fff640 1 --2- 172.21.15.40:0/3917437326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 0x560d734e5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f2453fff640 1 --2- 172.21.15.40:0/3917437326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734dc5b0 0x560d734e7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245885b640 1 -- 172.21.15.40:0/3917437326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 msgr2=0x560d734e5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245885b640 1 --2- 172.21.15.40:0/3917437326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 0x560d734e5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245885b640 1 -- 172.21.15.40:0/3917437326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734dc5b0 msgr2=0x560d734e7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:16.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245885b640 1 --2- 172.21.15.40:0/3917437326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734dc5b0 0x560d734e7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245885b640 1 -- 172.21.15.40:0/3917437326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d734e7a50 con 0x560d734d7440 2024-03-18T22:50:16.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f245885b640 1 --2- 172.21.15.40:0/3917437326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d7440 0x560d734e6450 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f24480097e0 tx=0x7f2448009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22086ada2547a8e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:16.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.867+0000 7f2433fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:16.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.868+0000 7f2433fff640 1 -- 172.21.15.40:0/3917437326 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24480055d0 con 0x560d734d7440 2024-03-18T22:50:16.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.868+0000 7f2433fff640 1 -- 172.21.15.40:0/3917437326 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2448016040 con 0x560d734d7440 2024-03-18T22:50:16.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.868+0000 7f245d782740 1 -- 172.21.15.40:0/3917437326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24400035f0 con 0x560d734d7440 2024-03-18T22:50:16.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.868+0000 7f2433fff640 1 -- 172.21.15.40:0/3917437326 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f244800f540 con 0x560d734d7440 2024-03-18T22:50:16.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.868+0000 7f245d782740 10 client.17604 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:16.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.868+0000 7f245d782740 1 -- 172.21.15.40:0/3917437326 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560d735533f0 con 0x560d734d7440 2024-03-18T22:50:16.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.868+0000 7f2433fff640 1 -- 172.21.15.40:0/3917437326 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24400035f0 con 0x560d734d7440 2024-03-18T22:50:16.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.868+0000 7f2433fff640 1 -- 172.21.15.40:0/3917437326 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f244802d070 con 0x560d734d7440 2024-03-18T22:50:16.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.868+0000 7f245d782740 10 client.17604 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:16.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.868+0000 7f245d782740 10 client.17604 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:16.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.868+0000 7f245d782740 20 client.17604 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.868+0000 7f245d782740 1 --2- 172.21.15.40:0/3917437326 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d73553c60 0x560d73556080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.868+0000 7f245d782740 4 client.17604 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:50:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.868+0000 7f245d782740 1 -- 172.21.15.40:0/3917437326 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560d73442d90 con 0x560d73553c60 2024-03-18T22:50:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.869+0000 7f2453fff640 1 --2- 172.21.15.40:0/3917437326 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d73553c60 0x560d73556080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.869+0000 7f2433fff640 10 client.17604 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.869+0000 7f2453fff640 1 --2- 172.21.15.40:0/3917437326 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d73553c60 0x560d73556080 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e15611a42debbeae server_cookie=85978863a89aabd6 in_seq=0 out_seq=0 2024-03-18T22:50:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.869+0000 7f2433fff640 1 -- 172.21.15.40:0/3917437326 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560d73442d90 con 0x560d73553c60 2024-03-18T22:50:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.869+0000 7f2433fff640 10 client.17604 handle_command_reply: tid=0 2024-03-18T22:50:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.869+0000 7f245d782740 1 client.17604 shutdown 2024-03-18T22:50:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.869+0000 7f245d782740 1 -- 172.21.15.40:0/3917437326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d7440 msgr2=0x560d734e6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.870+0000 7f245d782740 1 --2- 172.21.15.40:0/3917437326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d7440 0x560d734e6450 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f24480097e0 tx=0x7f2448009310 comp rx=0 tx=0).stop 2024-03-18T22:50:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.870+0000 7f245d782740 1 -- 172.21.15.40:0/3917437326 shutdown_connections 2024-03-18T22:50:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.870+0000 7f245d782740 1 --2- 172.21.15.40:0/3917437326 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560d73553c60 0x560d73556080 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.870+0000 7f2433fff640 0 client.17604 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:16.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.870+0000 7f245d782740 1 --2- 172.21.15.40:0/3917437326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560d734dc5b0 0x560d734e7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.870+0000 7f245d782740 1 --2- 172.21.15.40:0/3917437326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560d734d7440 0x560d734e6450 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.870+0000 7f245d782740 1 --2- 172.21.15.40:0/3917437326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560d734d6a50 0x560d734e5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.870+0000 7f245d782740 1 -- 172.21.15.40:0/3917437326 >> 172.21.15.40:0/3917437326 conn(0x560d734d1fc0 msgr2=0x560d734d5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:16.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.870+0000 7f245d782740 1 -- 172.21.15.40:0/3917437326 shutdown_connections 2024-03-18T22:50:16.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.870+0000 7f245d782740 1 -- 172.21.15.40:0/3917437326 wait complete. 2024-03-18T22:50:16.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.870+0000 7f245d782740 20 client.17604 trim_cache size 0 max 16384 2024-03-18T22:50:16.876 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:50:16.876 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:50:16.876 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:50:16.877 INFO:teuthology.orchestra.run.smithi040.stdout: "ef7a13e3-ab20-4fa8-a202-d042c612cfe4": { 2024-03-18T22:50:16.877 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:50:16.877 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "ef7a13e3-ab20-4fa8-a202-d042c612cfe4", 2024-03-18T22:50:16.877 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:50:16.877 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:50:16.877 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:50:16.877 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:50:16.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.871+0000 7f245aabf640 1 -- 172.21.15.40:0/809726059 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f24440a2800 msgr2=0x7f24440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.871+0000 7f245aabf640 1 --2- 172.21.15.40:0/809726059 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f24440a2800 0x7f24440a4cc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f244c0ccd30 tx=0x7f244c0c62b0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.871+0000 7f245aabf640 1 -- 172.21.15.40:0/809726059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24541022d0 msgr2=0x7f2454103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:16.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.871+0000 7f245aabf640 1 --2- 172.21.15.40:0/809726059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24541022d0 0x7f2454103230 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f244c00efd0 tx=0x7f244c00c6a0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.871+0000 7f245aabf640 1 -- 172.21.15.40:0/809726059 shutdown_connections 2024-03-18T22:50:16.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.871+0000 7f245aabf640 1 --2- 172.21.15.40:0/809726059 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f24440a2800 0x7f24440a4cc0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.871+0000 7f245aabf640 1 --2- 172.21.15.40:0/809726059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24541022d0 0x7f2454103230 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.871+0000 7f245aabf640 1 --2- 172.21.15.40:0/809726059 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2454101910 0x7f2454102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.871+0000 7f245aabf640 1 --2- 172.21.15.40:0/809726059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2454100720 0x7f24541046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:16.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.871+0000 7f245aabf640 1 -- 172.21.15.40:0/809726059 >> 172.21.15.40:0/809726059 conn(0x7f24540fbcb0 msgr2=0x7f24540689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:16.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.871+0000 7f245aabf640 1 -- 172.21.15.40:0/809726059 shutdown_connections 2024-03-18T22:50:16.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:16.871+0000 7f245aabf640 1 -- 172.21.15.40:0/809726059 wait complete. 2024-03-18T22:50:16.885 INFO:tasks.cephfs.filesystem:scrub status for tag:ef7a13e3-ab20-4fa8-a202-d042c612cfe4 - {'path': '/', 'tag': 'ef7a13e3-ab20-4fa8-a202-d042c612cfe4', 'options': 'recursive,force'} 2024-03-18T22:50:17.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:17 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:50:17.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:17 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:50:17.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:17 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:50:17.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:17 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:50:17.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:17 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:50:17.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:17 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:50:17.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:17 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:50:17.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:17 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:50:17.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:17 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:50:17.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:17 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:50:17.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:17 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:50:17.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:17 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:50:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:18 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:50:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:18 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:50:18.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:18 smithi113 ceph-mon[66775]: pgmap v580: 129 pgs: 129 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 20 op/s 2024-03-18T22:50:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:18 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:50:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:18 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:50:18.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:18 smithi103 ceph-mon[66377]: pgmap v580: 129 pgs: 129 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 20 op/s 2024-03-18T22:50:18.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:18 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:50:18.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:18 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:50:18.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:18 smithi040 ceph-mon[62643]: pgmap v580: 129 pgs: 129 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 30 MiB/s wr, 20 op/s 2024-03-18T22:50:20.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:20 smithi113 ceph-mon[66775]: pgmap v581: 129 pgs: 129 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 21 op/s 2024-03-18T22:50:20.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:20 smithi103 ceph-mon[66377]: pgmap v581: 129 pgs: 129 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 21 op/s 2024-03-18T22:50:20.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:20 smithi040 ceph-mon[62643]: pgmap v581: 129 pgs: 129 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 21 op/s 2024-03-18T22:50:22.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:22 smithi113 ceph-mon[66775]: pgmap v582: 129 pgs: 129 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 MiB/s wr, 24 op/s 2024-03-18T22:50:22.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:22 smithi103 ceph-mon[66377]: pgmap v582: 129 pgs: 129 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 MiB/s wr, 24 op/s 2024-03-18T22:50:22.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:22 smithi040 ceph-mon[62643]: pgmap v582: 129 pgs: 129 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 MiB/s wr, 24 op/s 2024-03-18T22:50:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:23 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:50:23.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:23 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:50:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:23 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:50:23.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:23 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:50:23.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:23 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:50:23.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:23 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:50:24.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:50:24.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: pgmap v583: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 32 MiB/s wr, 25 op/s 2024-03-18T22:50:24.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: Adjusting osd_memory_target on smithi113 to 176.2M 2024-03-18T22:50:24.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-03-18T22:50:24.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: Unable to set osd_memory_target on smithi113 to 184772403: error parsing value: Value '184772403' is below minimum 939524096 2024-03-18T22:50:24.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: Unable to set osd_memory_target on smithi103 to 184789606: error parsing value: Value '184789606' is below minimum 939524096 2024-03-18T22:50:24.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:50:24.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:50:24.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:50:24.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:50:24.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: pgmap v583: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 32 MiB/s wr, 25 op/s 2024-03-18T22:50:24.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: Adjusting osd_memory_target on smithi113 to 176.2M 2024-03-18T22:50:24.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-03-18T22:50:24.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: Unable to set osd_memory_target on smithi113 to 184772403: error parsing value: Value '184772403' is below minimum 939524096 2024-03-18T22:50:24.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: Unable to set osd_memory_target on smithi103 to 184789606: error parsing value: Value '184789606' is below minimum 939524096 2024-03-18T22:50:24.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:50:24.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:50:24.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:50:24.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:50:24.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: pgmap v583: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 32 MiB/s wr, 25 op/s 2024-03-18T22:50:24.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T22:50:24.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: Adjusting osd_memory_target on smithi113 to 176.2M 2024-03-18T22:50:24.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-03-18T22:50:24.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: Unable to set osd_memory_target on smithi113 to 184772403: error parsing value: Value '184772403' is below minimum 939524096 2024-03-18T22:50:24.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: Unable to set osd_memory_target on smithi103 to 184789606: error parsing value: Value '184789606' is below minimum 939524096 2024-03-18T22:50:24.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:50:24.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:50:24.395 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:50:26.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:26 smithi113 ceph-mon[66775]: pgmap v584: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 MiB/s wr, 25 op/s 2024-03-18T22:50:26.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:26 smithi103 ceph-mon[66377]: pgmap v584: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 MiB/s wr, 25 op/s 2024-03-18T22:50:26.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:26 smithi040 ceph-mon[62643]: pgmap v584: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 MiB/s wr, 25 op/s 2024-03-18T22:50:28.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:28 smithi113 ceph-mon[66775]: pgmap v585: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 23 op/s 2024-03-18T22:50:28.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:28 smithi103 ceph-mon[66377]: pgmap v585: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 23 op/s 2024-03-18T22:50:28.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:28 smithi040 ceph-mon[62643]: pgmap v585: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 23 op/s 2024-03-18T22:50:30.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:30 smithi113 ceph-mon[66775]: pgmap v586: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:50:30.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:30 smithi103 ceph-mon[66377]: pgmap v586: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:50:30.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:30 smithi040 ceph-mon[62643]: pgmap v586: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:50:32.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:32 smithi113 ceph-mon[66775]: pgmap v587: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 25 MiB/s wr, 24 op/s 2024-03-18T22:50:32.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:32 smithi103 ceph-mon[66377]: pgmap v587: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 25 MiB/s wr, 24 op/s 2024-03-18T22:50:32.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:32 smithi040 ceph-mon[62643]: pgmap v587: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 25 MiB/s wr, 24 op/s 2024-03-18T22:50:34.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:34 smithi103 ceph-mon[66377]: pgmap v588: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 23 op/s 2024-03-18T22:50:34.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:34 smithi040 ceph-mon[62643]: pgmap v588: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 23 op/s 2024-03-18T22:50:34.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:34 smithi113 ceph-mon[66775]: pgmap v588: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 23 op/s 2024-03-18T22:50:36.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:36 smithi103 ceph-mon[66377]: pgmap v589: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:50:36.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:36 smithi040 ceph-mon[62643]: pgmap v589: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:50:36.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:36 smithi113 ceph-mon[66775]: pgmap v589: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:50:38.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:38 smithi103 ceph-mon[66377]: pgmap v590: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:50:38.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:38 smithi040 ceph-mon[62643]: pgmap v590: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:50:38.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:38 smithi113 ceph-mon[66775]: pgmap v590: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:50:40.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:40 smithi103 ceph-mon[66377]: pgmap v591: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:50:40.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:40 smithi040 ceph-mon[62643]: pgmap v591: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:50:40.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:40 smithi113 ceph-mon[66775]: pgmap v591: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:50:42.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:42 smithi103 ceph-mon[66377]: pgmap v592: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:50:42.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:42 smithi040 ceph-mon[62643]: pgmap v592: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:50:42.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:42 smithi113 ceph-mon[66775]: pgmap v592: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 27 MiB/s wr, 22 op/s 2024-03-18T22:50:44.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:44 smithi103 ceph-mon[66377]: pgmap v593: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:50:44.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:44 smithi040 ceph-mon[62643]: pgmap v593: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:50:44.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:44 smithi113 ceph-mon[66775]: pgmap v593: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:50:46.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:46 smithi103 ceph-mon[66377]: pgmap v594: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:50:46.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:46 smithi040 ceph-mon[62643]: pgmap v594: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:50:46.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:46 smithi113 ceph-mon[66775]: pgmap v594: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:50:46.886 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:50:46.887 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:50:48.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:48 smithi103 ceph-mon[66377]: pgmap v595: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 27 MiB/s wr, 19 op/s 2024-03-18T22:50:48.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:48 smithi040 ceph-mon[62643]: pgmap v595: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 27 MiB/s wr, 19 op/s 2024-03-18T22:50:48.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:48 smithi113 ceph-mon[66775]: pgmap v595: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 27 MiB/s wr, 19 op/s 2024-03-18T22:50:50.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:50 smithi040 ceph-mon[62643]: pgmap v596: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-03-18T22:50:50.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:50 smithi103 ceph-mon[66377]: pgmap v596: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-03-18T22:50:50.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:50 smithi113 ceph-mon[66775]: pgmap v596: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 24 MiB/s wr, 19 op/s 2024-03-18T22:50:52.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:52 smithi103 ceph-mon[66377]: pgmap v597: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:50:52.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:52 smithi040 ceph-mon[62643]: pgmap v597: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:50:52.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:52 smithi113 ceph-mon[66775]: pgmap v597: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:50:54.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:54 smithi103 ceph-mon[66377]: pgmap v598: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:50:54.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:54 smithi040 ceph-mon[62643]: pgmap v598: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:50:54.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:54 smithi113 ceph-mon[66775]: pgmap v598: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:50:56.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:56 smithi103 ceph-mon[66377]: pgmap v599: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:50:56.382 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:56 smithi113 ceph-mon[66775]: pgmap v599: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:50:56.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:56 smithi040 ceph-mon[62643]: pgmap v599: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:50:56.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.582+0000 7fd294c61640 1 -- 172.21.15.40:0/2907659591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2901022d0 msgr2=0x7fd290108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:56.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.582+0000 7fd294c61640 1 --2- 172.21.15.40:0/2907659591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2901022d0 0x7fd290108b60 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fd288009ca0 tx=0x7fd28802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.582+0000 7fd294c61640 1 -- 172.21.15.40:0/2907659591 shutdown_connections 2024-03-18T22:50:56.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.582+0000 7fd294c61640 1 --2- 172.21.15.40:0/2907659591 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2901022d0 0x7fd290108b60 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.582+0000 7fd294c61640 1 --2- 172.21.15.40:0/2907659591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd290101910 0x7fd290101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.582+0000 7fd294c61640 1 --2- 172.21.15.40:0/2907659591 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd290100720 0x7fd290100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.582+0000 7fd294c61640 1 -- 172.21.15.40:0/2907659591 >> 172.21.15.40:0/2907659591 conn(0x7fd2900fbcb0 msgr2=0x7fd2900fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:56.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.582+0000 7fd294c61640 1 -- 172.21.15.40:0/2907659591 shutdown_connections 2024-03-18T22:50:56.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.582+0000 7fd294c61640 1 -- 172.21.15.40:0/2907659591 wait complete. 2024-03-18T22:50:56.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd294c61640 1 Processor -- start 2024-03-18T22:50:56.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd294c61640 1 -- start start 2024-03-18T22:50:56.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd294c61640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd290100720 0x7fd290104850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd294c61640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd290101910 0x7fd290102ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd294c61640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2901022d0 0x7fd290103420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd294c61640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd29010c250 con 0x7fd290101910 2024-03-18T22:50:56.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd294c61640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd29010c0d0 con 0x7fd290100720 2024-03-18T22:50:56.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd294c61640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd29010c3d0 con 0x7fd2901022d0 2024-03-18T22:50:56.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd287fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd290101910 0x7fd290102ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd287fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd290101910 0x7fd290102ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49636/0 (socket says 172.21.15.40:49636) 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd287fff640 1 -- 172.21.15.40:0/2246677984 learned_addr learned my addr 172.21.15.40:0/2246677984 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd28e59c640 1 --2- 172.21.15.40:0/2246677984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd290100720 0x7fd290104850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd28ed9d640 1 --2- 172.21.15.40:0/2246677984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2901022d0 0x7fd290103420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd287fff640 1 -- 172.21.15.40:0/2246677984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2901022d0 msgr2=0x7fd290103420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd287fff640 1 --2- 172.21.15.40:0/2246677984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2901022d0 0x7fd290103420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd287fff640 1 -- 172.21.15.40:0/2246677984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd290100720 msgr2=0x7fd290104850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd287fff640 1 --2- 172.21.15.40:0/2246677984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd290100720 0x7fd290104850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd287fff640 1 -- 172.21.15.40:0/2246677984 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd290103ce0 con 0x7fd290101910 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.583+0000 7fd28e59c640 1 --2- 172.21.15.40:0/2246677984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd290100720 0x7fd290104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.584+0000 7fd287fff640 1 --2- 172.21.15.40:0/2246677984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd290101910 0x7fd290102ee0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fd27800e9f0 tx=0x7fd27800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.584+0000 7fd2877fe640 1 -- 172.21.15.40:0/2246677984 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27800cd80 con 0x7fd290101910 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.584+0000 7fd2877fe640 1 -- 172.21.15.40:0/2246677984 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2780108e0 con 0x7fd290101910 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.584+0000 7fd294c61640 1 -- 172.21.15.40:0/2246677984 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2901a1be0 con 0x7fd290101910 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.584+0000 7fd2877fe640 1 -- 172.21.15.40:0/2246677984 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278018920 con 0x7fd290101910 2024-03-18T22:50:56.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.584+0000 7fd294c61640 1 -- 172.21.15.40:0/2246677984 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2901a2100 con 0x7fd290101910 2024-03-18T22:50:56.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.599+0000 7fd2877fe640 1 -- 172.21.15.40:0/2246677984 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd278010a90 con 0x7fd290101910 2024-03-18T22:50:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.599+0000 7fd2877fe640 1 --2- 172.21.15.40:0/2246677984 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd27c0a2800 0x7fd27c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.600+0000 7fd28e59c640 1 --2- 172.21.15.40:0/2246677984 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd27c0a2800 0x7fd27c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.600+0000 7fd2877fe640 1 -- 172.21.15.40:0/2246677984 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd2780cf160 con 0x7fd290101910 2024-03-18T22:50:56.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.600+0000 7fd28e59c640 1 --2- 172.21.15.40:0/2246677984 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd27c0a2800 0x7fd27c0a4cc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd290101770 tx=0x7fd280005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:56.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd297851740 1 Processor -- start 2024-03-18T22:50:56.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd297851740 1 -- start start 2024-03-18T22:50:56.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd297851740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515686a50 0x564515686e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd297851740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564515687440 0x56451568bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd297851740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56451568c5b0 0x56451568ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd297851740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5645155eb500 con 0x56451568c5b0 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd297851740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564515593be0 con 0x564515686a50 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd297851740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5645155e75a0 con 0x564515687440 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd28ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56451568c5b0 0x56451568ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd287fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564515687440 0x56451568bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd28ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56451568c5b0 0x56451568ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49638/0 (socket says 172.21.15.40:49638) 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd28ed9d640 1 -- 172.21.15.40:0/2039123491 learned_addr learned my addr 172.21.15.40:0/2039123491 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd28e59c640 1 --2- 172.21.15.40:0/2039123491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515686a50 0x564515686e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd287fff640 1 -- 172.21.15.40:0/2039123491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515686a50 msgr2=0x564515686e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd287fff640 1 --2- 172.21.15.40:0/2039123491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515686a50 0x564515686e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd287fff640 1 -- 172.21.15.40:0/2039123491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56451568c5b0 msgr2=0x56451568ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd287fff640 1 --2- 172.21.15.40:0/2039123491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56451568c5b0 0x56451568ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.641+0000 7fd287fff640 1 -- 172.21.15.40:0/2039123491 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645155b5000 con 0x564515687440 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.642+0000 7fd287fff640 1 --2- 172.21.15.40:0/2039123491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564515687440 0x56451568bf40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd2780ccbe0 tx=0x7fd278010ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d71c0fcb8a352076 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.642+0000 7fd284ff9640 1 -- 172.21.15.40:0/2039123491 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2780a3eb0 con 0x564515687440 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.642+0000 7fd284ff9640 1 -- 172.21.15.40:0/2039123491 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd278099ac0 con 0x564515687440 2024-03-18T22:50:56.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.642+0000 7fd284ff9640 1 -- 172.21.15.40:0/2039123491 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278097510 con 0x564515687440 2024-03-18T22:50:56.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.643+0000 7fd297851740 1 -- 172.21.15.40:0/2039123491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564515687440 msgr2=0x56451568bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:56.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.643+0000 7fd297851740 1 --2- 172.21.15.40:0/2039123491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564515687440 0x56451568bf40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd2780ccbe0 tx=0x7fd278010ea0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.643+0000 7fd297851740 1 -- 172.21.15.40:0/2039123491 shutdown_connections 2024-03-18T22:50:56.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.643+0000 7fd297851740 1 --2- 172.21.15.40:0/2039123491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56451568c5b0 0x56451568ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.643+0000 7fd297851740 1 --2- 172.21.15.40:0/2039123491 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564515687440 0x56451568bf40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.643+0000 7fd297851740 1 --2- 172.21.15.40:0/2039123491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515686a50 0x564515686e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.643+0000 7fd297851740 1 -- 172.21.15.40:0/2039123491 >> 172.21.15.40:0/2039123491 conn(0x564515681fc0 msgr2=0x5645156823c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:56.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.643+0000 7fd297851740 1 -- 172.21.15.40:0/2039123491 shutdown_connections 2024-03-18T22:50:56.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.643+0000 7fd297851740 1 -- 172.21.15.40:0/2039123491 wait complete. 2024-03-18T22:50:56.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.645+0000 7fd297851740 1 -- start start 2024-03-18T22:50:56.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.646+0000 7fd297851740 1 Processor -- start 2024-03-18T22:50:56.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.646+0000 7fd297851740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515686a50 0x564515690dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.646+0000 7fd297851740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564515687440 0x564515691300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.646+0000 7fd297851740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56451568c5b0 0x564515698340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.646+0000 7fd297851740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56451568f4a0 con 0x564515686a50 2024-03-18T22:50:56.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.646+0000 7fd297851740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5645155a7160 con 0x56451568c5b0 2024-03-18T22:50:56.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.646+0000 7fd297851740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56451568f620 con 0x564515687440 2024-03-18T22:50:56.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.646+0000 7fd28e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515686a50 0x564515690dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.646+0000 7fd28e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515686a50 0x564515690dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49640/0 (socket says 172.21.15.40:49640) 2024-03-18T22:50:56.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.646+0000 7fd28e59c640 1 -- 172.21.15.40:0/2460371961 learned_addr learned my addr 172.21.15.40:0/2460371961 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:56.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.646+0000 7fd287fff640 1 --2- 172.21.15.40:0/2460371961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564515687440 0x564515691300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.646+0000 7fd28ed9d640 1 --2- 172.21.15.40:0/2460371961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56451568c5b0 0x564515698340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.646+0000 7fd297851740 10 client.0 _ll_register_callbacks cb 0x5645155e3a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:56.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.646+0000 7fd287fff640 1 -- 172.21.15.40:0/2460371961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56451568c5b0 msgr2=0x564515698340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:56.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.646+0000 7fd287fff640 1 --2- 172.21.15.40:0/2460371961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56451568c5b0 0x564515698340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.647+0000 7fd287fff640 1 -- 172.21.15.40:0/2460371961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515686a50 msgr2=0x564515690dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:56.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.647+0000 7fd287fff640 1 --2- 172.21.15.40:0/2460371961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515686a50 0x564515690dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.647+0000 7fd287fff640 1 -- 172.21.15.40:0/2460371961 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645153c97d0 con 0x564515687440 2024-03-18T22:50:56.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.647+0000 7fd28e59c640 1 --2- 172.21.15.40:0/2460371961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515686a50 0x564515690dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:56.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.647+0000 7fd28ed9d640 1 --2- 172.21.15.40:0/2460371961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56451568c5b0 0x564515698340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:56.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.647+0000 7fd287fff640 1 --2- 172.21.15.40:0/2460371961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564515687440 0x564515691300 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd2780cc870 tx=0x7fd27809b020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cf048fdf2cfc4d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:56.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.647+0000 7fd275ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:50:56.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.647+0000 7fd275ffb640 1 -- 172.21.15.40:0/2460371961 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2780b3070 con 0x564515687440 2024-03-18T22:50:56.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.647+0000 7fd275ffb640 1 -- 172.21.15.40:0/2460371961 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd278010430 con 0x564515687440 2024-03-18T22:50:56.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.649+0000 7fd297851740 1 -- 172.21.15.40:0/2460371961 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd278099ac0 con 0x564515687440 2024-03-18T22:50:56.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.649+0000 7fd275ffb640 1 -- 172.21.15.40:0/2460371961 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fd278010670 con 0x564515687440 2024-03-18T22:50:56.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.649+0000 7fd297851740 10 client.27421 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:56.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.649+0000 7fd297851740 1 -- 172.21.15.40:0/2460371961 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({fsmap=34}) -- 0x5645152ff3f0 con 0x564515687440 2024-03-18T22:50:56.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.649+0000 7fd275ffb640 1 -- 172.21.15.40:0/2460371961 <== mon.1 v2:172.21.15.113:3300/0 4 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fd2780b4260 con 0x564515687440 2024-03-18T22:50:56.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.650+0000 7fd297851740 10 client.27421 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:56.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.650+0000 7fd297851740 10 client.27421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:56.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.650+0000 7fd297851740 20 client.27421 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:56.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.650+0000 7fd297851740 1 --2- 172.21.15.40:0/2460371961 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56451569c730 0x56451569eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.650+0000 7fd297851740 4 client.27421 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:56.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.650+0000 7fd297851740 1 -- 172.21.15.40:0/2460371961 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645155f8a40 con 0x56451569c730 2024-03-18T22:50:56.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.650+0000 7fd28e59c640 1 --2- 172.21.15.40:0/2460371961 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56451569c730 0x56451569eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.650+0000 7fd275ffb640 10 client.27421 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:56.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.650+0000 7fd28e59c640 1 --2- 172.21.15.40:0/2460371961 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56451569c730 0x56451569eb20 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aae7fd0a9d8e4108 server_cookie=93aa2ca3440af4c6 in_seq=0 out_seq=0 2024-03-18T22:50:56.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.651+0000 7fd275ffb640 1 -- 172.21.15.40:0/2460371961 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5645155f8a40 con 0x56451569c730 2024-03-18T22:50:56.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.651+0000 7fd275ffb640 10 client.27421 handle_command_reply: tid=0 2024-03-18T22:50:56.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.651+0000 7fd297851740 1 client.27421 shutdown 2024-03-18T22:50:56.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.651+0000 7fd297851740 1 -- 172.21.15.40:0/2460371961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564515687440 msgr2=0x564515691300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:56.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.651+0000 7fd297851740 1 --2- 172.21.15.40:0/2460371961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564515687440 0x564515691300 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd2780cc870 tx=0x7fd27809b020 comp rx=0 tx=0).stop 2024-03-18T22:50:56.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.651+0000 7fd297851740 1 -- 172.21.15.40:0/2460371961 shutdown_connections 2024-03-18T22:50:56.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.651+0000 7fd297851740 1 --2- 172.21.15.40:0/2460371961 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56451569c730 0x56451569eb20 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.651+0000 7fd297851740 1 --2- 172.21.15.40:0/2460371961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56451568c5b0 0x564515698340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.651+0000 7fd275ffb640 0 client.27421 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:56.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.651+0000 7fd297851740 1 --2- 172.21.15.40:0/2460371961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x564515687440 0x564515691300 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.651+0000 7fd297851740 1 --2- 172.21.15.40:0/2460371961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515686a50 0x564515690dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.651+0000 7fd297851740 1 -- 172.21.15.40:0/2460371961 >> 172.21.15.40:0/2460371961 conn(0x564515681fc0 msgr2=0x5645156811e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:56.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.651+0000 7fd297851740 1 -- 172.21.15.40:0/2460371961 shutdown_connections 2024-03-18T22:50:56.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.651+0000 7fd297851740 1 -- 172.21.15.40:0/2460371961 wait complete. 2024-03-18T22:50:56.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.651+0000 7fd297851740 20 client.27421 trim_cache size 0 max 16384 2024-03-18T22:50:56.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd297851740 1 Processor -- start 2024-03-18T22:50:56.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd297851740 1 -- start start 2024-03-18T22:50:56.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd297851740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515686a50 0x564515685930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd297851740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515687440 0x564515698300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd297851740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56451568c5b0 0x5645156a32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd297851740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901ac700 con 0x564515686a50 2024-03-18T22:50:56.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd297851740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901ae300 con 0x564515687440 2024-03-18T22:50:56.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd297851740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901acdd0 con 0x56451568c5b0 2024-03-18T22:50:56.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd28e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515686a50 0x564515685930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd28ed9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56451568c5b0 0x5645156a32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd28e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515686a50 0x564515685930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49654/0 (socket says 172.21.15.40:49654) 2024-03-18T22:50:56.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd28e59c640 1 -- 172.21.15.40:0/731559584 learned_addr learned my addr 172.21.15.40:0/731559584 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:56.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd287fff640 1 --2- 172.21.15.40:0/731559584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515687440 0x564515698300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd28ed9d640 1 -- 172.21.15.40:0/731559584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515687440 msgr2=0x564515698300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:56.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd28ed9d640 1 --2- 172.21.15.40:0/731559584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515687440 0x564515698300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd28ed9d640 1 -- 172.21.15.40:0/731559584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515686a50 msgr2=0x564515685930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:56.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd28ed9d640 1 --2- 172.21.15.40:0/731559584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515686a50 0x564515685930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd28ed9d640 1 -- 172.21.15.40:0/731559584 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645152bf0f0 con 0x56451568c5b0 2024-03-18T22:50:56.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.673+0000 7fd28e59c640 1 --2- 172.21.15.40:0/731559584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515686a50 0x564515685930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:56.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.674+0000 7fd28ed9d640 1 --2- 172.21.15.40:0/731559584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56451568c5b0 0x5645156a32a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd288032380 tx=0x7fd2880324f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=525e17c8f5601630 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:56.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.674+0000 7fd277fff640 1 -- 172.21.15.40:0/731559584 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd288039040 con 0x56451568c5b0 2024-03-18T22:50:56.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.674+0000 7fd277fff640 1 -- 172.21.15.40:0/731559584 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd288003e20 con 0x56451568c5b0 2024-03-18T22:50:56.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.674+0000 7fd277fff640 1 -- 172.21.15.40:0/731559584 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd288004740 con 0x56451568c5b0 2024-03-18T22:50:56.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.675+0000 7fd297851740 1 -- 172.21.15.40:0/731559584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56451568c5b0 msgr2=0x5645156a32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:56.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.675+0000 7fd297851740 1 --2- 172.21.15.40:0/731559584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56451568c5b0 0x5645156a32a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd288032380 tx=0x7fd2880324f0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.675+0000 7fd297851740 1 -- 172.21.15.40:0/731559584 shutdown_connections 2024-03-18T22:50:56.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.675+0000 7fd297851740 1 --2- 172.21.15.40:0/731559584 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56451568c5b0 0x5645156a32a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.675+0000 7fd297851740 1 --2- 172.21.15.40:0/731559584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515687440 0x564515698300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.675+0000 7fd297851740 1 --2- 172.21.15.40:0/731559584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515686a50 0x564515685930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.675+0000 7fd297851740 1 -- 172.21.15.40:0/731559584 >> 172.21.15.40:0/731559584 conn(0x564515681fc0 msgr2=0x564515685220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:56.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.675+0000 7fd297851740 1 -- 172.21.15.40:0/731559584 shutdown_connections 2024-03-18T22:50:56.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.675+0000 7fd297851740 1 -- 172.21.15.40:0/731559584 wait complete. 2024-03-18T22:50:56.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.675+0000 7fd297851740 1 -- start start 2024-03-18T22:50:56.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.675+0000 7fd297851740 1 Processor -- start 2024-03-18T22:50:56.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd297851740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515686a50 0x564515696000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd297851740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515687440 0x564515696540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd297851740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56451568c5b0 0x5645156974e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd297851740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901adc30 con 0x564515687440 2024-03-18T22:50:56.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd297851740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901ad4a0 con 0x564515686a50 2024-03-18T22:50:56.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd297851740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd2901aa6e0 con 0x56451568c5b0 2024-03-18T22:50:56.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd297851740 10 client.0 _ll_register_callbacks cb 0x7fd278005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:56.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd28e59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515686a50 0x564515696000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd28ed9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56451568c5b0 0x5645156974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd287fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515687440 0x564515696540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd28e59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515686a50 0x564515696000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46922/0 (socket says 172.21.15.40:46922) 2024-03-18T22:50:56.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd28ed9d640 1 --2- 172.21.15.40:0/2086868873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56451568c5b0 0x5645156974e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:34082/0 (socket says 172.21.15.40:34082) 2024-03-18T22:50:56.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd28e59c640 1 -- 172.21.15.40:0/2086868873 learned_addr learned my addr 172.21.15.40:0/2086868873 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:56.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd28e59c640 1 -- 172.21.15.40:0/2086868873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56451568c5b0 msgr2=0x5645156974e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:56.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd28e59c640 1 --2- 172.21.15.40:0/2086868873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56451568c5b0 0x5645156974e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd28e59c640 1 -- 172.21.15.40:0/2086868873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515687440 msgr2=0x564515696540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:56.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd28e59c640 1 --2- 172.21.15.40:0/2086868873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515687440 0x564515696540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd28e59c640 1 -- 172.21.15.40:0/2086868873 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564515697ad0 con 0x564515686a50 2024-03-18T22:50:56.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd287fff640 1 --2- 172.21.15.40:0/2086868873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515687440 0x564515696540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:56.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd28e59c640 1 --2- 172.21.15.40:0/2086868873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515686a50 0x564515696000 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x56451569f220 tx=0x7fd280013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f48d71811207acc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:56.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd275ffb640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:50:56.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd275ffb640 1 -- 172.21.15.40:0/2086868873 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28002c170 con 0x564515686a50 2024-03-18T22:50:56.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd297851740 1 -- 172.21.15.40:0/2086868873 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd288003e20 con 0x564515686a50 2024-03-18T22:50:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.676+0000 7fd275ffb640 1 -- 172.21.15.40:0/2086868873 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd28002c310 con 0x564515686a50 2024-03-18T22:50:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.677+0000 7fd275ffb640 1 -- 172.21.15.40:0/2086868873 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fd280037050 con 0x564515686a50 2024-03-18T22:50:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.677+0000 7fd297851740 10 client.27437 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.677+0000 7fd297851740 1 -- 172.21.15.40:0/2086868873 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x564515702e00 con 0x564515686a50 2024-03-18T22:50:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.677+0000 7fd275ffb640 1 -- 172.21.15.40:0/2086868873 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd288003e20 con 0x564515686a50 2024-03-18T22:50:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.677+0000 7fd275ffb640 1 -- 172.21.15.40:0/2086868873 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fd280029c70 con 0x564515686a50 2024-03-18T22:50:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.677+0000 7fd297851740 10 client.27437 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.677+0000 7fd297851740 10 client.27437 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.677+0000 7fd297851740 20 client.27437 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.677+0000 7fd297851740 1 --2- 172.21.15.40:0/2086868873 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x564515703750 0x564515705b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.677+0000 7fd297851740 4 client.27437 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:50:56.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.677+0000 7fd297851740 1 -- 172.21.15.40:0/2086868873 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5645155f2d90 con 0x564515703750 2024-03-18T22:50:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.677+0000 7fd287fff640 1 --2- 172.21.15.40:0/2086868873 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x564515703750 0x564515705b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd287fff640 1 --2- 172.21.15.40:0/2086868873 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x564515703750 0x564515705b70 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f88482f7a55694be server_cookie=fc3493b84ecdf1ab in_seq=0 out_seq=0 2024-03-18T22:50:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd275ffb640 10 client.27437 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd275ffb640 1 -- 172.21.15.40:0/2086868873 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5645155f2d90 con 0x564515703750 2024-03-18T22:50:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd275ffb640 10 client.27437 handle_command_reply: tid=0 2024-03-18T22:50:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd297851740 1 client.27437 shutdown 2024-03-18T22:50:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd297851740 1 -- 172.21.15.40:0/2086868873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515686a50 msgr2=0x564515696000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd297851740 1 --2- 172.21.15.40:0/2086868873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515686a50 0x564515696000 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x56451569f220 tx=0x7fd280013040 comp rx=0 tx=0).stop 2024-03-18T22:50:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd297851740 1 -- 172.21.15.40:0/2086868873 shutdown_connections 2024-03-18T22:50:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd297851740 1 --2- 172.21.15.40:0/2086868873 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56451568c5b0 0x5645156974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd297851740 1 --2- 172.21.15.40:0/2086868873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564515687440 0x564515696540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd297851740 1 --2- 172.21.15.40:0/2086868873 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x564515703750 0x564515705b70 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd297851740 1 --2- 172.21.15.40:0/2086868873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564515686a50 0x564515696000 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd275ffb640 0 client.27437 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd297851740 1 -- 172.21.15.40:0/2086868873 >> 172.21.15.40:0/2086868873 conn(0x564515681fc0 msgr2=0x564515685ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd297851740 1 -- 172.21.15.40:0/2086868873 shutdown_connections 2024-03-18T22:50:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd297851740 1 -- 172.21.15.40:0/2086868873 wait complete. 2024-03-18T22:50:56.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.678+0000 7fd297851740 20 client.27437 trim_cache size 0 max 16384 2024-03-18T22:50:56.688 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:50:56.688 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:50:56.688 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:50:56.688 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:50:56.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.679+0000 7fd294c61640 1 -- 172.21.15.40:0/2246677984 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd27c0a2800 msgr2=0x7fd27c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:56.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.679+0000 7fd294c61640 1 --2- 172.21.15.40:0/2246677984 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd27c0a2800 0x7fd27c0a4cc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd290101770 tx=0x7fd280005f90 comp rx=0 tx=0).stop 2024-03-18T22:50:56.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.679+0000 7fd294c61640 1 -- 172.21.15.40:0/2246677984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd290101910 msgr2=0x7fd290102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:56.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.679+0000 7fd294c61640 1 --2- 172.21.15.40:0/2246677984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd290101910 0x7fd290102ee0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fd27800e9f0 tx=0x7fd27800eeb0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.679+0000 7fd294c61640 1 -- 172.21.15.40:0/2246677984 shutdown_connections 2024-03-18T22:50:56.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.679+0000 7fd294c61640 1 --2- 172.21.15.40:0/2246677984 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd27c0a2800 0x7fd27c0a4cc0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.679+0000 7fd294c61640 1 --2- 172.21.15.40:0/2246677984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd2901022d0 0x7fd290103420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.679+0000 7fd294c61640 1 --2- 172.21.15.40:0/2246677984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd290101910 0x7fd290102ee0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.679+0000 7fd294c61640 1 --2- 172.21.15.40:0/2246677984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd290100720 0x7fd290104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.679+0000 7fd294c61640 1 -- 172.21.15.40:0/2246677984 >> 172.21.15.40:0/2246677984 conn(0x7fd2900fbcb0 msgr2=0x7fd290068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:56.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.679+0000 7fd294c61640 1 -- 172.21.15.40:0/2246677984 shutdown_connections 2024-03-18T22:50:56.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.679+0000 7fd294c61640 1 -- 172.21.15.40:0/2246677984 wait complete. 2024-03-18T22:50:56.733 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:50:56.734 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:50:56.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.883+0000 7fb9377f4640 1 -- 172.21.15.40:0/3830415659 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9301022f0 msgr2=0x7fb930108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.883+0000 7fb9377f4640 1 --2- 172.21.15.40:0/3830415659 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9301022f0 0x7fb930108b80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb928009be0 tx=0x7fb92802bef0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.883+0000 7fb9377f4640 1 -- 172.21.15.40:0/3830415659 shutdown_connections 2024-03-18T22:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.883+0000 7fb9377f4640 1 --2- 172.21.15.40:0/3830415659 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9301022f0 0x7fb930108b80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.883+0000 7fb9377f4640 1 --2- 172.21.15.40:0/3830415659 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb930101930 0x7fb930101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.883+0000 7fb9377f4640 1 --2- 172.21.15.40:0/3830415659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb930100740 0x7fb930100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.883+0000 7fb9377f4640 1 -- 172.21.15.40:0/3830415659 >> 172.21.15.40:0/3830415659 conn(0x7fb9300fbcb0 msgr2=0x7fb9300fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.883+0000 7fb9377f4640 1 -- 172.21.15.40:0/3830415659 shutdown_connections 2024-03-18T22:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.883+0000 7fb9377f4640 1 -- 172.21.15.40:0/3830415659 wait complete. 2024-03-18T22:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb9377f4640 1 Processor -- start 2024-03-18T22:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb9377f4640 1 -- start start 2024-03-18T22:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb9377f4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb930100740 0x7fb930197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb9377f4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb930101930 0x7fb930197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb9377f4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9301022f0 0x7fb93019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb9377f4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb93010c270 con 0x7fb930100740 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb9377f4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb93010c0f0 con 0x7fb930101930 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb9377f4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb93010c3f0 con 0x7fb9301022f0 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb935590640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb930100740 0x7fb930197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb935590640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb930100740 0x7fb930197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49676/0 (socket says 172.21.15.40:49676) 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb935590640 1 -- 172.21.15.40:0/121060007 learned_addr learned my addr 172.21.15.40:0/121060007 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb935d91640 1 --2- 172.21.15.40:0/121060007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9301022f0 0x7fb93019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb934d8f640 1 --2- 172.21.15.40:0/121060007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb930101930 0x7fb930197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb935590640 1 -- 172.21.15.40:0/121060007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9301022f0 msgr2=0x7fb93019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb935590640 1 --2- 172.21.15.40:0/121060007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9301022f0 0x7fb93019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb935590640 1 -- 172.21.15.40:0/121060007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb930101930 msgr2=0x7fb930197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb935590640 1 --2- 172.21.15.40:0/121060007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb930101930 0x7fb930197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.884+0000 7fb935590640 1 -- 172.21.15.40:0/121060007 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb93019f650 con 0x7fb930100740 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.885+0000 7fb935590640 1 --2- 172.21.15.40:0/121060007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb930100740 0x7fb930197a30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00e9a0 tx=0x7fb92c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.885+0000 7fb9267fc640 1 -- 172.21.15.40:0/121060007 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb92c00cd80 con 0x7fb930100740 2024-03-18T22:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.885+0000 7fb9377f4640 1 -- 172.21.15.40:0/121060007 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb93019f940 con 0x7fb930100740 2024-03-18T22:50:56.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.885+0000 7fb9267fc640 1 -- 172.21.15.40:0/121060007 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb92c0108e0 con 0x7fb930100740 2024-03-18T22:50:56.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.885+0000 7fb9267fc640 1 -- 172.21.15.40:0/121060007 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb92c0188a0 con 0x7fb930100740 2024-03-18T22:50:56.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.885+0000 7fb9377f4640 1 -- 172.21.15.40:0/121060007 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb93019fee0 con 0x7fb930100740 2024-03-18T22:50:56.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.886+0000 7fb9377f4640 1 -- 172.21.15.40:0/121060007 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9301a0170 con 0x7fb930100740 2024-03-18T22:50:56.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.887+0000 7fb9267fc640 1 -- 172.21.15.40:0/121060007 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb92c010a90 con 0x7fb930100740 2024-03-18T22:50:56.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.888+0000 7fb9267fc640 1 --2- 172.21.15.40:0/121060007 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb91c0a2800 0x7fb91c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:56.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.888+0000 7fb934d8f640 1 --2- 172.21.15.40:0/121060007 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb91c0a2800 0x7fb91c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:56.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.888+0000 7fb9267fc640 1 -- 172.21.15.40:0/121060007 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb92c0cf1d0 con 0x7fb930100740 2024-03-18T22:50:56.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.888+0000 7fb934d8f640 1 --2- 172.21.15.40:0/121060007 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb91c0a2800 0x7fb91c0a4cc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb930198f40 tx=0x7fb920005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:56.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:56.894+0000 7fb9267fc640 1 -- 172.21.15.40:0/121060007 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb92c053740 con 0x7fb930100740 2024-03-18T22:50:57.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.197+0000 7fb9377f4640 1 -- 172.21.15.40:0/121060007 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9301a0340 con 0x7fb930100740 2024-03-18T22:50:57.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.198+0000 7fb9267fc640 1 -- 172.21.15.40:0/121060007 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11954 (secure 0 0 0) 0x7fb92c09be80 con 0x7fb930100740 2024-03-18T22:50:57.200 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:50:57.200 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:49:24.214483+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24401,24388,24473,24509,24481]},"id":1}]} 2024-03-18T22:50:57.200 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 34 2024-03-18T22:50:57.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.200+0000 7fb9377f4640 1 -- 172.21.15.40:0/121060007 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb91c0a2800 msgr2=0x7fb91c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.200+0000 7fb9377f4640 1 --2- 172.21.15.40:0/121060007 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb91c0a2800 0x7fb91c0a4cc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb930198f40 tx=0x7fb920005f90 comp rx=0 tx=0).stop 2024-03-18T22:50:57.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.200+0000 7fb9377f4640 1 -- 172.21.15.40:0/121060007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb930100740 msgr2=0x7fb930197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.200+0000 7fb9377f4640 1 --2- 172.21.15.40:0/121060007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb930100740 0x7fb930197a30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00e9a0 tx=0x7fb92c00ee60 comp rx=0 tx=0).stop 2024-03-18T22:50:57.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.201+0000 7fb9377f4640 1 -- 172.21.15.40:0/121060007 shutdown_connections 2024-03-18T22:50:57.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.201+0000 7fb9377f4640 1 --2- 172.21.15.40:0/121060007 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb91c0a2800 0x7fb91c0a4cc0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.201+0000 7fb9377f4640 1 --2- 172.21.15.40:0/121060007 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9301022f0 0x7fb93019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.201+0000 7fb9377f4640 1 --2- 172.21.15.40:0/121060007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb930101930 0x7fb930197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.201+0000 7fb9377f4640 1 --2- 172.21.15.40:0/121060007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb930100740 0x7fb930197a30 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.201+0000 7fb9377f4640 1 -- 172.21.15.40:0/121060007 >> 172.21.15.40:0/121060007 conn(0x7fb9300fbcb0 msgr2=0x7fb9300fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.201+0000 7fb9377f4640 1 -- 172.21.15.40:0/121060007 shutdown_connections 2024-03-18T22:50:57.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.201+0000 7fb9377f4640 1 -- 172.21.15.40:0/121060007 wait complete. 2024-03-18T22:50:57.226 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:50:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.340+0000 7f4e61a7e640 1 -- 172.21.15.40:0/474228347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e5c102af0 msgr2=0x7f4e5c102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.340+0000 7f4e61a7e640 1 --2- 172.21.15.40:0/474228347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e5c102af0 0x7f4e5c102f70 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009930 tx=0x7f4e5002bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.340+0000 7f4e61a7e640 1 -- 172.21.15.40:0/474228347 shutdown_connections 2024-03-18T22:50:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.340+0000 7f4e61a7e640 1 --2- 172.21.15.40:0/474228347 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e5c1034b0 0x7f4e5c109d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.340+0000 7f4e61a7e640 1 --2- 172.21.15.40:0/474228347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e5c102af0 0x7f4e5c102f70 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.340+0000 7f4e61a7e640 1 --2- 172.21.15.40:0/474228347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e5c0ffb20 0x7f4e5c0fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.340+0000 7f4e61a7e640 1 -- 172.21.15.40:0/474228347 >> 172.21.15.40:0/474228347 conn(0x7f4e5c0fb430 msgr2=0x7f4e5c0fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.340+0000 7f4e61a7e640 1 -- 172.21.15.40:0/474228347 shutdown_connections 2024-03-18T22:50:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.340+0000 7f4e61a7e640 1 -- 172.21.15.40:0/474228347 wait complete. 2024-03-18T22:50:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.341+0000 7f4e61a7e640 1 Processor -- start 2024-03-18T22:50:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.341+0000 7f4e61a7e640 1 -- start start 2024-03-18T22:50:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.341+0000 7f4e61a7e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e5c0ffb20 0x7f4e5c197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.341+0000 7f4e61a7e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e5c102af0 0x7f4e5c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.341+0000 7f4e61a7e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e5c1034b0 0x7f4e5c19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.341+0000 7f4e61a7e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5c10d390 con 0x7f4e5c1034b0 2024-03-18T22:50:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.341+0000 7f4e61a7e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5c10d210 con 0x7f4e5c102af0 2024-03-18T22:50:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.341+0000 7f4e61a7e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5c10d510 con 0x7f4e5c0ffb20 2024-03-18T22:50:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.341+0000 7f4e5bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e5c1034b0 0x7f4e5c19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.341+0000 7f4e5bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e5c1034b0 0x7f4e5c19f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49692/0 (socket says 172.21.15.40:49692) 2024-03-18T22:50:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.341+0000 7f4e5bfff640 1 -- 172.21.15.40:0/2519744997 learned_addr learned my addr 172.21.15.40:0/2519744997 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.341+0000 7f4e5affd640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e5c102af0 0x7f4e5c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.341+0000 7f4e5b7fe640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e5c0ffb20 0x7f4e5c197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.342+0000 7f4e5b7fe640 1 -- 172.21.15.40:0/2519744997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e5c102af0 msgr2=0x7f4e5c197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.342+0000 7f4e5b7fe640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e5c102af0 0x7f4e5c197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.342+0000 7f4e5b7fe640 1 -- 172.21.15.40:0/2519744997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e5c1034b0 msgr2=0x7f4e5c19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.342+0000 7f4e5b7fe640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e5c1034b0 0x7f4e5c19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.342+0000 7f4e5b7fe640 1 -- 172.21.15.40:0/2519744997 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e5c19f690 con 0x7f4e5c0ffb20 2024-03-18T22:50:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.342+0000 7f4e5bfff640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e5c1034b0 0x7f4e5c19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.342+0000 7f4e5affd640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e5c102af0 0x7f4e5c197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.342+0000 7f4e5b7fe640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e5c0ffb20 0x7f4e5c197a70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c00e9c0 tx=0x7f4e4c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.342+0000 7f4e58ff9640 1 -- 172.21.15.40:0/2519744997 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c00ccc0 con 0x7f4e5c0ffb20 2024-03-18T22:50:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.342+0000 7f4e61a7e640 1 -- 172.21.15.40:0/2519744997 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e5c19f980 con 0x7f4e5c0ffb20 2024-03-18T22:50:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.342+0000 7f4e58ff9640 1 -- 172.21.15.40:0/2519744997 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e4c0035f0 con 0x7f4e5c0ffb20 2024-03-18T22:50:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.342+0000 7f4e58ff9640 1 -- 172.21.15.40:0/2519744997 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c00ce60 con 0x7f4e5c0ffb20 2024-03-18T22:50:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.342+0000 7f4e61a7e640 1 -- 172.21.15.40:0/2519744997 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e5c19fef0 con 0x7f4e5c0ffb20 2024-03-18T22:50:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.344+0000 7f4e58ff9640 1 -- 172.21.15.40:0/2519744997 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4e4c0037a0 con 0x7f4e5c0ffb20 2024-03-18T22:50:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.345+0000 7f4e64666740 1 Processor -- start 2024-03-18T22:50:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.345+0000 7f4e64666740 1 -- start start 2024-03-18T22:50:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.345+0000 7f4e58ff9640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4e440a2800 0x7f4e440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.345+0000 7f4e64666740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5589225862f0 0x558922586710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.345+0000 7f4e64666740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558922586ce0 0x55892258b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.345+0000 7f4e5bfff640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4e440a2800 0x7f4e440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.345+0000 7f4e64666740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55892258be50 0x55892258e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.345+0000 7f4e64666740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5589224ea4b0 con 0x5589225862f0 2024-03-18T22:50:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.345+0000 7f4e64666740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558922492d70 con 0x558922586ce0 2024-03-18T22:50:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.345+0000 7f4e64666740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5589224e6550 con 0x55892258be50 2024-03-18T22:50:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.345+0000 7f4e58ff9640 1 -- 172.21.15.40:0/2519744997 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4e4c0cf2f0 con 0x7f4e5c0ffb20 2024-03-18T22:50:57.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e5affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558922586ce0 0x55892258b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e5b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5589225862f0 0x558922586710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e5affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558922586ce0 0x55892258b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46958/0 (socket says 172.21.15.40:46958) 2024-03-18T22:50:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e5affd640 1 -- 172.21.15.40:0/3034741072 learned_addr learned my addr 172.21.15.40:0/3034741072 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e5bfff640 1 --2- 172.21.15.40:0/3034741072 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55892258be50 0x55892258e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e5affd640 1 -- 172.21.15.40:0/3034741072 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55892258be50 msgr2=0x55892258e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e5affd640 1 --2- 172.21.15.40:0/3034741072 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55892258be50 0x55892258e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e5affd640 1 -- 172.21.15.40:0/3034741072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5589225862f0 msgr2=0x558922586710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e5affd640 1 --2- 172.21.15.40:0/3034741072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5589225862f0 0x558922586710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e5affd640 1 -- 172.21.15.40:0/3034741072 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589224b5210 con 0x558922586ce0 2024-03-18T22:50:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e5bfff640 1 --2- 172.21.15.40:0/3034741072 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55892258be50 0x55892258e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:50:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e5bfff640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4e440a2800 0x7f4e440a4cc0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4e540049d0 tx=0x7f4e5400ff80 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e5affd640 1 --2- 172.21.15.40:0/3034741072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558922586ce0 0x55892258b7e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4e50036040 tx=0x7f4e5002cd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0cda2e535cffdec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e41ffb640 1 -- 172.21.15.40:0/3034741072 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e50045020 con 0x558922586ce0 2024-03-18T22:50:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e41ffb640 1 -- 172.21.15.40:0/3034741072 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e50005040 con 0x558922586ce0 2024-03-18T22:50:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.346+0000 7f4e41ffb640 1 -- 172.21.15.40:0/3034741072 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e50004920 con 0x558922586ce0 2024-03-18T22:50:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.347+0000 7f4e64666740 1 -- 172.21.15.40:0/3034741072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558922586ce0 msgr2=0x55892258b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.347+0000 7f4e64666740 1 --2- 172.21.15.40:0/3034741072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558922586ce0 0x55892258b7e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4e50036040 tx=0x7f4e5002cd00 comp rx=0 tx=0).stop 2024-03-18T22:50:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.347+0000 7f4e64666740 1 -- 172.21.15.40:0/3034741072 shutdown_connections 2024-03-18T22:50:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.347+0000 7f4e64666740 1 --2- 172.21.15.40:0/3034741072 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55892258be50 0x55892258e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.347+0000 7f4e64666740 1 --2- 172.21.15.40:0/3034741072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558922586ce0 0x55892258b7e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.347+0000 7f4e64666740 1 --2- 172.21.15.40:0/3034741072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5589225862f0 0x558922586710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.347+0000 7f4e64666740 1 -- 172.21.15.40:0/3034741072 >> 172.21.15.40:0/3034741072 conn(0x558922581860 msgr2=0x558922581c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.347+0000 7f4e64666740 1 -- 172.21.15.40:0/3034741072 shutdown_connections 2024-03-18T22:50:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.347+0000 7f4e64666740 1 -- 172.21.15.40:0/3034741072 wait complete. 2024-03-18T22:50:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.347+0000 7f4e64666740 1 -- start start 2024-03-18T22:50:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e64666740 1 Processor -- start 2024-03-18T22:50:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e64666740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5589225862f0 0x558922590750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e64666740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 0x558922590c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e64666740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55892258be50 0x558922597cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e64666740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55892258eda0 con 0x55892258be50 2024-03-18T22:50:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e64666740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5589224a6310 con 0x5589225862f0 2024-03-18T22:50:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e64666740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55892258ef20 con 0x558922586ce0 2024-03-18T22:50:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e64666740 10 client.0 _ll_register_callbacks cb 0x55892249a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e5bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55892258be50 0x558922597cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e5affd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 0x558922590c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e5b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5589225862f0 0x558922590750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e5bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55892258be50 0x558922597cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49720/0 (socket says 172.21.15.40:49720) 2024-03-18T22:50:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e5affd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 0x558922590c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:34134/0 (socket says 172.21.15.40:34134) 2024-03-18T22:50:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e5b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5589225862f0 0x558922590750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46968/0 (socket says 172.21.15.40:46968) 2024-03-18T22:50:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e5bfff640 1 -- 172.21.15.40:0/2620455254 learned_addr learned my addr 172.21.15.40:0/2620455254 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e5b7fe640 1 -- 172.21.15.40:0/2620455254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 msgr2=0x558922590c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e5b7fe640 1 --2- 172.21.15.40:0/2620455254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 0x558922590c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e5b7fe640 1 -- 172.21.15.40:0/2620455254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55892258be50 msgr2=0x558922597cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e5b7fe640 1 --2- 172.21.15.40:0/2620455254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55892258be50 0x558922597cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e5b7fe640 1 -- 172.21.15.40:0/2620455254 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589221fe3f0 con 0x5589225862f0 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.348+0000 7f4e5bfff640 1 --2- 172.21.15.40:0/2620455254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55892258be50 0x558922597cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e5b7fe640 1 --2- 172.21.15.40:0/2620455254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5589225862f0 0x558922590750 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c0cc9a0 tx=0x7f4e4c09c080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17bd15c71ecd7a86 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e3affd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e3affd640 1 -- 172.21.15.40:0/2620455254 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c0b3020 con 0x5589225862f0 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e3affd640 1 -- 172.21.15.40:0/2620455254 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4e4c09cb70 con 0x5589225862f0 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e64666740 1 -- 172.21.15.40:0/2620455254 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e50005040 con 0x5589225862f0 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e3affd640 1 -- 172.21.15.40:0/2620455254 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4e4c0cce60 con 0x5589225862f0 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e64666740 10 client.27455 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e64666740 1 -- 172.21.15.40:0/2620455254 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5589222c87d0 con 0x5589225862f0 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e3affd640 1 -- 172.21.15.40:0/2620455254 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e50005040 con 0x5589225862f0 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e3affd640 1 -- 172.21.15.40:0/2620455254 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f4e4c0976b0 con 0x5589225862f0 2024-03-18T22:50:57.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e64666740 10 client.27455 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:57.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e64666740 10 client.27455 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:50:57.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e64666740 20 client.27455 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:57.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e64666740 1 --2- 172.21.15.40:0/2620455254 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55892259c5d0 0x55892259ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e64666740 4 client.27455 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:57.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.349+0000 7f4e64666740 1 -- 172.21.15.40:0/2620455254 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589225483e0 con 0x55892259c5d0 2024-03-18T22:50:57.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.350+0000 7f4e5affd640 1 --2- 172.21.15.40:0/2620455254 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55892259c5d0 0x55892259ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.350+0000 7f4e5affd640 1 --2- 172.21.15.40:0/2620455254 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55892259c5d0 0x55892259ea60 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=661afdb5d3a2e7f1 server_cookie=faaf99e0d2847151 in_seq=0 out_seq=0 2024-03-18T22:50:57.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.350+0000 7f4e3affd640 10 client.27455 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:50:57.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.351+0000 7f4e3affd640 1 -- 172.21.15.40:0/2620455254 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5589225483e0 con 0x55892259c5d0 2024-03-18T22:50:57.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.351+0000 7f4e3affd640 10 client.27455 handle_command_reply: tid=0 2024-03-18T22:50:57.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.351+0000 7f4e64666740 1 client.27455 shutdown 2024-03-18T22:50:57.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.351+0000 7f4e64666740 1 -- 172.21.15.40:0/2620455254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5589225862f0 msgr2=0x558922590750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.351+0000 7f4e64666740 1 --2- 172.21.15.40:0/2620455254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5589225862f0 0x558922590750 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c0cc9a0 tx=0x7f4e4c09c080 comp rx=0 tx=0).stop 2024-03-18T22:50:57.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.351+0000 7f4e64666740 1 -- 172.21.15.40:0/2620455254 shutdown_connections 2024-03-18T22:50:57.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.351+0000 7f4e64666740 1 --2- 172.21.15.40:0/2620455254 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55892259c5d0 0x55892259ea60 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.351+0000 7f4e64666740 1 --2- 172.21.15.40:0/2620455254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55892258be50 0x558922597cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.351+0000 7f4e64666740 1 --2- 172.21.15.40:0/2620455254 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 0x558922590c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.351+0000 7f4e3affd640 0 client.27455 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:50:57.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.351+0000 7f4e64666740 1 --2- 172.21.15.40:0/2620455254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5589225862f0 0x558922590750 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.351+0000 7f4e64666740 1 -- 172.21.15.40:0/2620455254 >> 172.21.15.40:0/2620455254 conn(0x558922581860 msgr2=0x558922580ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.351+0000 7f4e64666740 1 -- 172.21.15.40:0/2620455254 shutdown_connections 2024-03-18T22:50:57.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.351+0000 7f4e64666740 1 -- 172.21.15.40:0/2620455254 wait complete. 2024-03-18T22:50:57.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.351+0000 7f4e64666740 20 client.27455 trim_cache size 0 max 16384 2024-03-18T22:50:57.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.367+0000 7f4e64666740 1 Processor -- start 2024-03-18T22:50:57.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.367+0000 7f4e64666740 1 -- start start 2024-03-18T22:50:57.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.367+0000 7f4e64666740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5589225862f0 0x558922590a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.367+0000 7f4e64666740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 0x558922597c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.367+0000 7f4e64666740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892258be50 0x558922580310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.367+0000 7f4e64666740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5c1aa4c0 con 0x5589225862f0 2024-03-18T22:50:57.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.367+0000 7f4e64666740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5c1ac0c0 con 0x55892258be50 2024-03-18T22:50:57.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.367+0000 7f4e64666740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5c1aab90 con 0x558922586ce0 2024-03-18T22:50:57.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.367+0000 7f4e5b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5589225862f0 0x558922590a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.367+0000 7f4e5affd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 0x558922597c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.367+0000 7f4e5b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5589225862f0 0x558922590a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49722/0 (socket says 172.21.15.40:49722) 2024-03-18T22:50:57.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.367+0000 7f4e5bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892258be50 0x558922580310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.367+0000 7f4e5b7fe640 1 -- 172.21.15.40:0/2544078976 learned_addr learned my addr 172.21.15.40:0/2544078976 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.368+0000 7f4e5bfff640 1 -- 172.21.15.40:0/2544078976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 msgr2=0x558922597c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.368+0000 7f4e5bfff640 1 --2- 172.21.15.40:0/2544078976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 0x558922597c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.368+0000 7f4e5bfff640 1 -- 172.21.15.40:0/2544078976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5589225862f0 msgr2=0x558922590a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.368+0000 7f4e5bfff640 1 --2- 172.21.15.40:0/2544078976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5589225862f0 0x558922590a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.368+0000 7f4e5bfff640 1 -- 172.21.15.40:0/2544078976 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558922557880 con 0x55892258be50 2024-03-18T22:50:57.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.368+0000 7f4e5b7fe640 1 --2- 172.21.15.40:0/2544078976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5589225862f0 0x558922590a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.368+0000 7f4e5bfff640 1 --2- 172.21.15.40:0/2544078976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892258be50 0x558922580310 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4e540122e0 tx=0x7f4e5401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5fd3fa412c90f74 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.368+0000 7f4e417fa640 1 -- 172.21.15.40:0/2544078976 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e540107b0 con 0x55892258be50 2024-03-18T22:50:57.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.368+0000 7f4e417fa640 1 -- 172.21.15.40:0/2544078976 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e54010e00 con 0x55892258be50 2024-03-18T22:50:57.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.368+0000 7f4e417fa640 1 -- 172.21.15.40:0/2544078976 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e54020400 con 0x55892258be50 2024-03-18T22:50:57.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.369+0000 7f4e64666740 1 -- 172.21.15.40:0/2544078976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892258be50 msgr2=0x558922580310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.369+0000 7f4e64666740 1 --2- 172.21.15.40:0/2544078976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892258be50 0x558922580310 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4e540122e0 tx=0x7f4e5401f000 comp rx=0 tx=0).stop 2024-03-18T22:50:57.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.369+0000 7f4e64666740 1 -- 172.21.15.40:0/2544078976 shutdown_connections 2024-03-18T22:50:57.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.369+0000 7f4e64666740 1 --2- 172.21.15.40:0/2544078976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892258be50 0x558922580310 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.369+0000 7f4e64666740 1 --2- 172.21.15.40:0/2544078976 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 0x558922597c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.369+0000 7f4e64666740 1 --2- 172.21.15.40:0/2544078976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5589225862f0 0x558922590a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.369+0000 7f4e64666740 1 -- 172.21.15.40:0/2544078976 >> 172.21.15.40:0/2544078976 conn(0x558922581860 msgr2=0x558922590360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.369+0000 7f4e64666740 1 -- 172.21.15.40:0/2544078976 shutdown_connections 2024-03-18T22:50:57.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.369+0000 7f4e64666740 1 -- 172.21.15.40:0/2544078976 wait complete. 2024-03-18T22:50:57.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.369+0000 7f4e64666740 1 -- start start 2024-03-18T22:50:57.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e64666740 1 Processor -- start 2024-03-18T22:50:57.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e64666740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5589225862f0 0x558922595870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e64666740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 0x558922595db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e64666740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55892258be50 0x558922596dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e64666740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5c1ab9f0 con 0x55892258be50 2024-03-18T22:50:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e64666740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5c1ab260 con 0x5589225862f0 2024-03-18T22:50:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e64666740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5c1a84a0 con 0x558922586ce0 2024-03-18T22:50:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e64666740 10 client.0 _ll_register_callbacks cb 0x7f4e4c0046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e5affd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 0x558922595db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e5bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55892258be50 0x558922596dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e5b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5589225862f0 0x558922595870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e5bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55892258be50 0x558922596dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49732/0 (socket says 172.21.15.40:49732) 2024-03-18T22:50:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e5affd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 0x558922595db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:34148/0 (socket says 172.21.15.40:34148) 2024-03-18T22:50:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e5b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5589225862f0 0x558922595870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46984/0 (socket says 172.21.15.40:46984) 2024-03-18T22:50:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e5bfff640 1 -- 172.21.15.40:0/3900814445 learned_addr learned my addr 172.21.15.40:0/3900814445 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e5bfff640 1 -- 172.21.15.40:0/3900814445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 msgr2=0x558922595db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e5bfff640 1 --2- 172.21.15.40:0/3900814445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 0x558922595db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e5bfff640 1 -- 172.21.15.40:0/3900814445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5589225862f0 msgr2=0x558922595870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e5bfff640 1 --2- 172.21.15.40:0/3900814445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5589225862f0 0x558922595870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e5bfff640 1 -- 172.21.15.40:0/3900814445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55892258fb70 con 0x55892258be50 2024-03-18T22:50:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e5b7fe640 1 --2- 172.21.15.40:0/3900814445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5589225862f0 0x558922595870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e5bfff640 1 --2- 172.21.15.40:0/3900814445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55892258be50 0x558922596dc0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4e5402b8d0 tx=0x7f4e5402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1064ca5250ab05d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.370+0000 7f4e3affd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.371+0000 7f4e3affd640 1 -- 172.21.15.40:0/3900814445 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e540109c0 con 0x55892258be50 2024-03-18T22:50:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.371+0000 7f4e64666740 1 -- 172.21.15.40:0/3900814445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e54010e00 con 0x55892258be50 2024-03-18T22:50:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.371+0000 7f4e3affd640 1 -- 172.21.15.40:0/3900814445 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4e54010b60 con 0x55892258be50 2024-03-18T22:50:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.371+0000 7f4e3affd640 1 -- 172.21.15.40:0/3900814445 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4e54034440 con 0x55892258be50 2024-03-18T22:50:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.371+0000 7f4e64666740 10 client.17646 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.371+0000 7f4e64666740 1 -- 172.21.15.40:0/3900814445 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x558922602c70 con 0x55892258be50 2024-03-18T22:50:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.371+0000 7f4e3affd640 1 -- 172.21.15.40:0/3900814445 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e54010e00 con 0x55892258be50 2024-03-18T22:50:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.371+0000 7f4e3affd640 1 -- 172.21.15.40:0/3900814445 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f4e54032070 con 0x55892258be50 2024-03-18T22:50:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.371+0000 7f4e64666740 10 client.17646 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.371+0000 7f4e64666740 10 client.17646 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:50:57.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.371+0000 7f4e64666740 20 client.17646 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:57.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.371+0000 7f4e64666740 1 --2- 172.21.15.40:0/3900814445 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x558922603600 0x558922605a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.371+0000 7f4e64666740 4 client.17646 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:50:57.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.371+0000 7f4e64666740 1 -- 172.21.15.40:0/3900814445 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5589224f79f0 con 0x558922603600 2024-03-18T22:50:57.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e5b7fe640 1 --2- 172.21.15.40:0/3900814445 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x558922603600 0x558922605a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e3affd640 10 client.17646 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:50:57.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e5b7fe640 1 --2- 172.21.15.40:0/3900814445 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x558922603600 0x558922605a20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2efaf812817de056 server_cookie=48dd14ecb19e6333 in_seq=0 out_seq=0 2024-03-18T22:50:57.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e3affd640 1 -- 172.21.15.40:0/3900814445 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5589224f79f0 con 0x558922603600 2024-03-18T22:50:57.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e3affd640 10 client.17646 handle_command_reply: tid=0 2024-03-18T22:50:57.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e64666740 1 client.17646 shutdown 2024-03-18T22:50:57.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e64666740 1 -- 172.21.15.40:0/3900814445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55892258be50 msgr2=0x558922596dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e64666740 1 --2- 172.21.15.40:0/3900814445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55892258be50 0x558922596dc0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4e5402b8d0 tx=0x7f4e5402b900 comp rx=0 tx=0).stop 2024-03-18T22:50:57.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e64666740 1 -- 172.21.15.40:0/3900814445 shutdown_connections 2024-03-18T22:50:57.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e64666740 1 --2- 172.21.15.40:0/3900814445 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x558922603600 0x558922605a20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e64666740 1 --2- 172.21.15.40:0/3900814445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55892258be50 0x558922596dc0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e64666740 1 --2- 172.21.15.40:0/3900814445 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558922586ce0 0x558922595db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e64666740 1 --2- 172.21.15.40:0/3900814445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5589225862f0 0x558922595870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e64666740 1 -- 172.21.15.40:0/3900814445 >> 172.21.15.40:0/3900814445 conn(0x558922581860 msgr2=0x558922584ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e3affd640 0 client.17646 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:50:57.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e64666740 1 -- 172.21.15.40:0/3900814445 shutdown_connections 2024-03-18T22:50:57.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.372+0000 7f4e64666740 1 -- 172.21.15.40:0/3900814445 wait complete. 2024-03-18T22:50:57.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.373+0000 7f4e64666740 20 client.17646 trim_cache size 0 max 16384 2024-03-18T22:50:57.380 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:50:57.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.373+0000 7f4e61a7e640 1 -- 172.21.15.40:0/2519744997 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4e440a2800 msgr2=0x7f4e440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.373+0000 7f4e61a7e640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4e440a2800 0x7f4e440a4cc0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4e540049d0 tx=0x7f4e5400ff80 comp rx=0 tx=0).stop 2024-03-18T22:50:57.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.373+0000 7f4e61a7e640 1 -- 172.21.15.40:0/2519744997 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e5c0ffb20 msgr2=0x7f4e5c197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.373+0000 7f4e61a7e640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e5c0ffb20 0x7f4e5c197a70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c00e9c0 tx=0x7f4e4c00ee80 comp rx=0 tx=0).stop 2024-03-18T22:50:57.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.373+0000 7f4e61a7e640 1 -- 172.21.15.40:0/2519744997 shutdown_connections 2024-03-18T22:50:57.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.373+0000 7f4e61a7e640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4e440a2800 0x7f4e440a4cc0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.373+0000 7f4e61a7e640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e5c1034b0 0x7f4e5c19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.373+0000 7f4e61a7e640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e5c102af0 0x7f4e5c197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.373+0000 7f4e61a7e640 1 --2- 172.21.15.40:0/2519744997 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4e5c0ffb20 0x7f4e5c197a70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.373+0000 7f4e61a7e640 1 -- 172.21.15.40:0/2519744997 >> 172.21.15.40:0/2519744997 conn(0x7f4e5c0fb430 msgr2=0x7f4e5c105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.373+0000 7f4e61a7e640 1 -- 172.21.15.40:0/2519744997 shutdown_connections 2024-03-18T22:50:57.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.373+0000 7f4e61a7e640 1 -- 172.21.15.40:0/2519744997 wait complete. 2024-03-18T22:50:57.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:50:57.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.497+0000 7fde124d1640 1 -- 172.21.15.40:0/2277785487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c100720 msgr2=0x7fde0c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.497+0000 7fde124d1640 1 --2- 172.21.15.40:0/2277785487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c100720 0x7fde0c100b40 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009900 tx=0x7fddfc02bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:57.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.497+0000 7fde124d1640 1 -- 172.21.15.40:0/2277785487 shutdown_connections 2024-03-18T22:50:57.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.497+0000 7fde124d1640 1 --2- 172.21.15.40:0/2277785487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde0c1022d0 0x7fde0c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.497+0000 7fde124d1640 1 --2- 172.21.15.40:0/2277785487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde0c101910 0x7fde0c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.497+0000 7fde124d1640 1 --2- 172.21.15.40:0/2277785487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c100720 0x7fde0c100b40 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.497+0000 7fde124d1640 1 -- 172.21.15.40:0/2277785487 >> 172.21.15.40:0/2277785487 conn(0x7fde0c0fbc90 msgr2=0x7fde0c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.497+0000 7fde124d1640 1 -- 172.21.15.40:0/2277785487 shutdown_connections 2024-03-18T22:50:57.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.497+0000 7fde124d1640 1 -- 172.21.15.40:0/2277785487 wait complete. 2024-03-18T22:50:57.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.498+0000 7fde124d1640 1 Processor -- start 2024-03-18T22:50:57.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.498+0000 7fde124d1640 1 -- start start 2024-03-18T22:50:57.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.498+0000 7fde124d1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde0c100720 0x7fde0c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.498+0000 7fde124d1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c101910 0x7fde0c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.498+0000 7fde124d1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde0c1022d0 0x7fde0c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.498+0000 7fde124d1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c10c270 con 0x7fde0c101910 2024-03-18T22:50:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.498+0000 7fde124d1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c10c0f0 con 0x7fde0c100720 2024-03-18T22:50:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.498+0000 7fde124d1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c10c3f0 con 0x7fde0c1022d0 2024-03-18T22:50:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.498+0000 7fde0b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c101910 0x7fde0c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.498+0000 7fde0b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c101910 0x7fde0c197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49756/0 (socket says 172.21.15.40:49756) 2024-03-18T22:50:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.498+0000 7fde0b7fe640 1 -- 172.21.15.40:0/2567216939 learned_addr learned my addr 172.21.15.40:0/2567216939 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.498+0000 7fde0bfff640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde0c100720 0x7fde0c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.498+0000 7fde10a6e640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde0c1022d0 0x7fde0c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.499+0000 7fde0bfff640 1 -- 172.21.15.40:0/2567216939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde0c1022d0 msgr2=0x7fde0c19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.499+0000 7fde0bfff640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde0c1022d0 0x7fde0c19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.499+0000 7fde0bfff640 1 -- 172.21.15.40:0/2567216939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c101910 msgr2=0x7fde0c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.499+0000 7fde0bfff640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c101910 0x7fde0c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.499+0000 7fde0bfff640 1 -- 172.21.15.40:0/2567216939 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0c19f6a0 con 0x7fde0c100720 2024-03-18T22:50:57.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.499+0000 7fde0b7fe640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c101910 0x7fde0c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.499+0000 7fde10a6e640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde0c1022d0 0x7fde0c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.499+0000 7fde0bfff640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde0c100720 0x7fde0c197a80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fddfc036040 tx=0x7fddfc0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.499+0000 7fde097fa640 1 -- 172.21.15.40:0/2567216939 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc043020 con 0x7fde0c100720 2024-03-18T22:50:57.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.499+0000 7fde097fa640 1 -- 172.21.15.40:0/2567216939 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fddfc004a50 con 0x7fde0c100720 2024-03-18T22:50:57.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.499+0000 7fde124d1640 1 -- 172.21.15.40:0/2567216939 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde0c19f930 con 0x7fde0c100720 2024-03-18T22:50:57.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.499+0000 7fde097fa640 1 -- 172.21.15.40:0/2567216939 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc003b50 con 0x7fde0c100720 2024-03-18T22:50:57.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.499+0000 7fde124d1640 1 -- 172.21.15.40:0/2567216939 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde0c19fe50 con 0x7fde0c100720 2024-03-18T22:50:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.502+0000 7fde097fa640 1 -- 172.21.15.40:0/2567216939 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fddfc046020 con 0x7fde0c100720 2024-03-18T22:50:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.502+0000 7fde15196740 1 Processor -- start 2024-03-18T22:50:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.502+0000 7fde15196740 1 -- start start 2024-03-18T22:50:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.502+0000 7fde15196740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d6262f0 0x55836d626710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.502+0000 7fde15196740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d626ce0 0x55836d62b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.502+0000 7fde15196740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d62be50 0x55836d62e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.502+0000 7fde15196740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55836d58a4b0 con 0x55836d6262f0 2024-03-18T22:50:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.502+0000 7fde15196740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55836d532d70 con 0x55836d62be50 2024-03-18T22:50:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.502+0000 7fde15196740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55836d586550 con 0x55836d626ce0 2024-03-18T22:50:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.502+0000 7fde10a6e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d6262f0 0x55836d626710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.502+0000 7fde10a6e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d6262f0 0x55836d626710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49760/0 (socket says 172.21.15.40:49760) 2024-03-18T22:50:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.502+0000 7fde10a6e640 1 -- 172.21.15.40:0/987559902 learned_addr learned my addr 172.21.15.40:0/987559902 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.502+0000 7fde0bfff640 1 --2- 172.21.15.40:0/987559902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d626ce0 0x55836d62b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.502+0000 7fde0b7fe640 1 --2- 172.21.15.40:0/987559902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d62be50 0x55836d62e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.502+0000 7fde097fa640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fddf40a2800 0x7fddf40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.503+0000 7fde0bfff640 1 -- 172.21.15.40:0/987559902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d62be50 msgr2=0x55836d62e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.503+0000 7fde0bfff640 1 --2- 172.21.15.40:0/987559902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d62be50 0x55836d62e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.503+0000 7fde0bfff640 1 -- 172.21.15.40:0/987559902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d6262f0 msgr2=0x55836d626710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.503+0000 7fde0bfff640 1 --2- 172.21.15.40:0/987559902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d6262f0 0x55836d626710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.503+0000 7fde0bfff640 1 -- 172.21.15.40:0/987559902 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55836d555210 con 0x55836d626ce0 2024-03-18T22:50:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.503+0000 7fde0b7fe640 1 --2- 172.21.15.40:0/987559902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d62be50 0x55836d62e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.503+0000 7fde10a6e640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fddf40a2800 0x7fddf40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.503+0000 7fde097fa640 1 -- 172.21.15.40:0/2567216939 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fddfc0f0f30 con 0x7fde0c100720 2024-03-18T22:50:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.503+0000 7fde0bfff640 1 --2- 172.21.15.40:0/987559902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d626ce0 0x55836d62b7e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0ee610 tx=0x7fddfc0883c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38bc95f2e73f6ebd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.503+0000 7fddf27fc640 1 -- 172.21.15.40:0/987559902 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0bd020 con 0x55836d626ce0 2024-03-18T22:50:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.503+0000 7fddf27fc640 1 -- 172.21.15.40:0/987559902 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fddfc0bdbe0 con 0x55836d626ce0 2024-03-18T22:50:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.503+0000 7fde10a6e640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fddf40a2800 0x7fddf40a4cc0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x55836d627540 tx=0x7fde04004ab0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.503+0000 7fddf27fc640 1 -- 172.21.15.40:0/987559902 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0bce40 con 0x55836d626ce0 2024-03-18T22:50:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.504+0000 7fde15196740 1 -- 172.21.15.40:0/987559902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d626ce0 msgr2=0x55836d62b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.504+0000 7fde15196740 1 --2- 172.21.15.40:0/987559902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d626ce0 0x55836d62b7e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0ee610 tx=0x7fddfc0883c0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.504+0000 7fde15196740 1 -- 172.21.15.40:0/987559902 shutdown_connections 2024-03-18T22:50:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.504+0000 7fde15196740 1 --2- 172.21.15.40:0/987559902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d62be50 0x55836d62e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.504+0000 7fde15196740 1 --2- 172.21.15.40:0/987559902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d626ce0 0x55836d62b7e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.504+0000 7fde15196740 1 --2- 172.21.15.40:0/987559902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d6262f0 0x55836d626710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.504+0000 7fde15196740 1 -- 172.21.15.40:0/987559902 >> 172.21.15.40:0/987559902 conn(0x55836d621860 msgr2=0x55836d621c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.504+0000 7fde15196740 1 -- 172.21.15.40:0/987559902 shutdown_connections 2024-03-18T22:50:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.504+0000 7fde15196740 1 -- 172.21.15.40:0/987559902 wait complete. 2024-03-18T22:50:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.504+0000 7fde15196740 1 -- start start 2024-03-18T22:50:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.504+0000 7fde15196740 1 Processor -- start 2024-03-18T22:50:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde15196740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d6262f0 0x55836d6303b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde15196740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d626ce0 0x55836d6308f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde15196740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d62be50 0x55836d637930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde15196740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55836d586550 con 0x55836d62be50 2024-03-18T22:50:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde15196740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55836d546310 con 0x55836d626ce0 2024-03-18T22:50:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde15196740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55836d62ebc0 con 0x55836d6262f0 2024-03-18T22:50:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde15196740 10 client.0 _ll_register_callbacks cb 0x55836d53a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde0bfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d6262f0 0x55836d6303b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde10a6e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d62be50 0x55836d637930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde0bfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d6262f0 0x55836d6303b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:34176/0 (socket says 172.21.15.40:34176) 2024-03-18T22:50:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde0bfff640 1 -- 172.21.15.40:0/2538325933 learned_addr learned my addr 172.21.15.40:0/2538325933 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde0b7fe640 1 --2- 172.21.15.40:0/2538325933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d626ce0 0x55836d6308f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde0bfff640 1 -- 172.21.15.40:0/2538325933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d626ce0 msgr2=0x55836d6308f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde0bfff640 1 --2- 172.21.15.40:0/2538325933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d626ce0 0x55836d6308f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde0bfff640 1 -- 172.21.15.40:0/2538325933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d62be50 msgr2=0x55836d637930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde0bfff640 1 --2- 172.21.15.40:0/2538325933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d62be50 0x55836d637930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde0bfff640 1 -- 172.21.15.40:0/2538325933 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55836d29e3f0 con 0x55836d6262f0 2024-03-18T22:50:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fde0bfff640 1 --2- 172.21.15.40:0/2538325933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d6262f0 0x55836d6303b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0ee610 tx=0x7fddfc02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b2b393d18c97236 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.505+0000 7fddeb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:50:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.506+0000 7fddeb7fe640 1 -- 172.21.15.40:0/2538325933 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0bce40 con 0x55836d6262f0 2024-03-18T22:50:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.506+0000 7fddeb7fe640 1 -- 172.21.15.40:0/2538325933 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fddfc0d1580 con 0x55836d6262f0 2024-03-18T22:50:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.506+0000 7fde15196740 1 -- 172.21.15.40:0/2538325933 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddfc0bdbe0 con 0x55836d6262f0 2024-03-18T22:50:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.506+0000 7fddeb7fe640 1 -- 172.21.15.40:0/2538325933 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fddfc0bd2b0 con 0x55836d6262f0 2024-03-18T22:50:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.506+0000 7fde15196740 10 client.27463 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:57.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.506+0000 7fde15196740 1 -- 172.21.15.40:0/2538325933 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55836d3687d0 con 0x55836d6262f0 2024-03-18T22:50:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.506+0000 7fddeb7fe640 1 -- 172.21.15.40:0/2538325933 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fddfc0bdbe0 con 0x55836d6262f0 2024-03-18T22:50:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.506+0000 7fddeb7fe640 1 -- 172.21.15.40:0/2538325933 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fddfc0d6040 con 0x55836d6262f0 2024-03-18T22:50:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.506+0000 7fde15196740 10 client.27463 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.506+0000 7fde15196740 10 client.27463 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:50:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.506+0000 7fde15196740 20 client.27463 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.506+0000 7fde15196740 1 --2- 172.21.15.40:0/2538325933 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55836d63c080 0x55836d63e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.506+0000 7fde15196740 4 client.27463 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.506+0000 7fde15196740 1 -- 172.21.15.40:0/2538325933 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55836d5e83e0 con 0x55836d63c080 2024-03-18T22:50:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.506+0000 7fde0b7fe640 1 --2- 172.21.15.40:0/2538325933 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55836d63c080 0x55836d63e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.507+0000 7fddeb7fe640 10 client.27463 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:50:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.507+0000 7fde0b7fe640 1 --2- 172.21.15.40:0/2538325933 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55836d63c080 0x55836d63e4e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=286ec72f5ff38f01 server_cookie=dcd47b9090aa5dec in_seq=0 out_seq=0 2024-03-18T22:50:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.507+0000 7fddeb7fe640 1 -- 172.21.15.40:0/2538325933 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55836d5e83e0 con 0x55836d63c080 2024-03-18T22:50:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.507+0000 7fddeb7fe640 10 client.27463 handle_command_reply: tid=0 2024-03-18T22:50:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.508+0000 7fde15196740 1 client.27463 shutdown 2024-03-18T22:50:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.508+0000 7fde15196740 1 -- 172.21.15.40:0/2538325933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d6262f0 msgr2=0x55836d6303b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.508+0000 7fde15196740 1 --2- 172.21.15.40:0/2538325933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d6262f0 0x55836d6303b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0ee610 tx=0x7fddfc02bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.508+0000 7fde15196740 1 -- 172.21.15.40:0/2538325933 shutdown_connections 2024-03-18T22:50:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.508+0000 7fde15196740 1 --2- 172.21.15.40:0/2538325933 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55836d63c080 0x55836d63e4e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.508+0000 7fde15196740 1 --2- 172.21.15.40:0/2538325933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d62be50 0x55836d637930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.508+0000 7fde15196740 1 --2- 172.21.15.40:0/2538325933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d626ce0 0x55836d6308f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.508+0000 7fde15196740 1 --2- 172.21.15.40:0/2538325933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d6262f0 0x55836d6303b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.508+0000 7fde15196740 1 -- 172.21.15.40:0/2538325933 >> 172.21.15.40:0/2538325933 conn(0x55836d621860 msgr2=0x55836d620ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.508+0000 7fddeb7fe640 0 client.27463 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:50:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.508+0000 7fde15196740 1 -- 172.21.15.40:0/2538325933 shutdown_connections 2024-03-18T22:50:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.508+0000 7fde15196740 1 -- 172.21.15.40:0/2538325933 wait complete. 2024-03-18T22:50:57.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.508+0000 7fde15196740 20 client.27463 trim_cache size 0 max 16384 2024-03-18T22:50:57.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.522+0000 7fde15196740 1 Processor -- start 2024-03-18T22:50:57.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.522+0000 7fde15196740 1 -- start start 2024-03-18T22:50:57.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.522+0000 7fde15196740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d6262f0 0x55836d630850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.522+0000 7fde15196740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d626ce0 0x55836d61ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.522+0000 7fde15196740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d62be50 0x55836d6251d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.522+0000 7fde15196740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c1aa500 con 0x55836d626ce0 2024-03-18T22:50:57.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.522+0000 7fde15196740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c1ac100 con 0x55836d6262f0 2024-03-18T22:50:57.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.522+0000 7fde15196740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c1aabd0 con 0x55836d62be50 2024-03-18T22:50:57.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.522+0000 7fde0b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d626ce0 0x55836d61ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.522+0000 7fde0b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d626ce0 0x55836d61ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49778/0 (socket says 172.21.15.40:49778) 2024-03-18T22:50:57.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.522+0000 7fde0b7fe640 1 -- 172.21.15.40:0/1251704009 learned_addr learned my addr 172.21.15.40:0/1251704009 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.522+0000 7fde10a6e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d62be50 0x55836d6251d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.523+0000 7fde0bfff640 1 --2- 172.21.15.40:0/1251704009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d6262f0 0x55836d630850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.523+0000 7fde0b7fe640 1 -- 172.21.15.40:0/1251704009 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d62be50 msgr2=0x55836d6251d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.523+0000 7fde0b7fe640 1 --2- 172.21.15.40:0/1251704009 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d62be50 0x55836d6251d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.523+0000 7fde0b7fe640 1 -- 172.21.15.40:0/1251704009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d6262f0 msgr2=0x55836d630850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.523+0000 7fde0b7fe640 1 --2- 172.21.15.40:0/1251704009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d6262f0 0x55836d630850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.523+0000 7fde0b7fe640 1 -- 172.21.15.40:0/1251704009 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55836d5f7880 con 0x55836d626ce0 2024-03-18T22:50:57.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.523+0000 7fde10a6e640 1 --2- 172.21.15.40:0/1251704009 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d62be50 0x55836d6251d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.523+0000 7fde0bfff640 1 --2- 172.21.15.40:0/1251704009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d6262f0 0x55836d630850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.523+0000 7fde0b7fe640 1 --2- 172.21.15.40:0/1251704009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d626ce0 0x55836d61ca30 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x55836d6227d0 tx=0x7fde00005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0cb780bb941585c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.523+0000 7fddf1ffb640 1 -- 172.21.15.40:0/1251704009 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde00024020 con 0x55836d626ce0 2024-03-18T22:50:57.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.523+0000 7fddf1ffb640 1 -- 172.21.15.40:0/1251704009 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde00010d10 con 0x55836d626ce0 2024-03-18T22:50:57.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.523+0000 7fddf1ffb640 1 -- 172.21.15.40:0/1251704009 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde00005630 con 0x55836d626ce0 2024-03-18T22:50:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.524+0000 7fde15196740 1 -- 172.21.15.40:0/1251704009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d626ce0 msgr2=0x55836d61ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.524+0000 7fde15196740 1 --2- 172.21.15.40:0/1251704009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d626ce0 0x55836d61ca30 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x55836d6227d0 tx=0x7fde00005f90 comp rx=0 tx=0).stop 2024-03-18T22:50:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.524+0000 7fde15196740 1 -- 172.21.15.40:0/1251704009 shutdown_connections 2024-03-18T22:50:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.524+0000 7fde15196740 1 --2- 172.21.15.40:0/1251704009 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d62be50 0x55836d6251d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.524+0000 7fde15196740 1 --2- 172.21.15.40:0/1251704009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d626ce0 0x55836d61ca30 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.524+0000 7fde15196740 1 --2- 172.21.15.40:0/1251704009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d6262f0 0x55836d630850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.524+0000 7fde15196740 1 -- 172.21.15.40:0/1251704009 >> 172.21.15.40:0/1251704009 conn(0x55836d621860 msgr2=0x55836d624ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.524+0000 7fde15196740 1 -- 172.21.15.40:0/1251704009 shutdown_connections 2024-03-18T22:50:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.524+0000 7fde15196740 1 -- 172.21.15.40:0/1251704009 wait complete. 2024-03-18T22:50:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.524+0000 7fde15196740 1 -- start start 2024-03-18T22:50:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde15196740 1 Processor -- start 2024-03-18T22:50:57.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde15196740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d6262f0 0x55836d69e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde15196740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d626ce0 0x55836d69e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde15196740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d62be50 0x55836d6a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde15196740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c1aba30 con 0x55836d6262f0 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde15196740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c1ab2a0 con 0x55836d626ce0 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde15196740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c1a84e0 con 0x55836d62be50 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde15196740 10 client.0 _ll_register_callbacks cb 0x7fde04009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde0bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d6262f0 0x55836d69e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde0bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d6262f0 0x55836d69e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49786/0 (socket says 172.21.15.40:49786) 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde0bfff640 1 -- 172.21.15.40:0/3606011470 learned_addr learned my addr 172.21.15.40:0/3606011470 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde10a6e640 1 --2- 172.21.15.40:0/3606011470 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d62be50 0x55836d6a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde0b7fe640 1 --2- 172.21.15.40:0/3606011470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d626ce0 0x55836d69e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde0bfff640 1 -- 172.21.15.40:0/3606011470 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d62be50 msgr2=0x55836d6a57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde0bfff640 1 --2- 172.21.15.40:0/3606011470 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d62be50 0x55836d6a57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde0bfff640 1 -- 172.21.15.40:0/3606011470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d626ce0 msgr2=0x55836d69e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde0bfff640 1 --2- 172.21.15.40:0/3606011470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d626ce0 0x55836d69e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde0bfff640 1 -- 172.21.15.40:0/3606011470 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55836d6a5db0 con 0x55836d6262f0 2024-03-18T22:50:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.525+0000 7fde10a6e640 1 --2- 172.21.15.40:0/3606011470 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d62be50 0x55836d6a57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fde0bfff640 1 --2- 172.21.15.40:0/3606011470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d6262f0 0x55836d69e320 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0bd1e0 tx=0x7fddfc009280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6588f79af04d9433 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fddeb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fddeb7fe640 1 -- 172.21.15.40:0/3606011470 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0bd020 con 0x55836d6262f0 2024-03-18T22:50:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fddeb7fe640 1 -- 172.21.15.40:0/3606011470 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fddfc0d1500 con 0x55836d6262f0 2024-03-18T22:50:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fde15196740 1 -- 172.21.15.40:0/3606011470 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde00010d10 con 0x55836d6262f0 2024-03-18T22:50:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fddeb7fe640 1 -- 172.21.15.40:0/3606011470 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fddfc0bce40 con 0x55836d6262f0 2024-03-18T22:50:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fde15196740 10 client.17664 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fde15196740 1 -- 172.21.15.40:0/3606011470 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55836d6a5db0 con 0x55836d6262f0 2024-03-18T22:50:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fddeb7fe640 1 -- 172.21.15.40:0/3606011470 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fde00010d10 con 0x55836d6262f0 2024-03-18T22:50:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fddeb7fe640 1 -- 172.21.15.40:0/3606011470 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fddfc0bd740 con 0x55836d6262f0 2024-03-18T22:50:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fde15196740 10 client.17664 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fde15196740 10 client.17664 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:50:57.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fde15196740 20 client.17664 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fde15196740 1 --2- 172.21.15.40:0/3606011470 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55836d6a75f0 0x55836d6a9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fde15196740 4 client.17664 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:50:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.526+0000 7fde15196740 1 -- 172.21.15.40:0/3606011470 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55836d5979f0 con 0x55836d6a75f0 2024-03-18T22:50:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fde0b7fe640 1 --2- 172.21.15.40:0/3606011470 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55836d6a75f0 0x55836d6a9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fde0b7fe640 1 --2- 172.21.15.40:0/3606011470 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55836d6a75f0 0x55836d6a9a10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=23fc395cf843d418 server_cookie=723bc8d11d0cbcd5 in_seq=0 out_seq=0 2024-03-18T22:50:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fddeb7fe640 10 client.17664 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:50:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fddeb7fe640 1 -- 172.21.15.40:0/3606011470 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55836d5979f0 con 0x55836d6a75f0 2024-03-18T22:50:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fddeb7fe640 10 client.17664 handle_command_reply: tid=0 2024-03-18T22:50:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fde15196740 1 client.17664 shutdown 2024-03-18T22:50:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fde15196740 1 -- 172.21.15.40:0/3606011470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d6262f0 msgr2=0x55836d69e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fde15196740 1 --2- 172.21.15.40:0/3606011470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d6262f0 0x55836d69e320 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fddfc0bd1e0 tx=0x7fddfc009280 comp rx=0 tx=0).stop 2024-03-18T22:50:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fde15196740 1 -- 172.21.15.40:0/3606011470 shutdown_connections 2024-03-18T22:50:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fde15196740 1 --2- 172.21.15.40:0/3606011470 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55836d6a75f0 0x55836d6a9a10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fde15196740 1 --2- 172.21.15.40:0/3606011470 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55836d62be50 0x55836d6a57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fde15196740 1 --2- 172.21.15.40:0/3606011470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55836d626ce0 0x55836d69e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fddeb7fe640 0 client.17664 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:50:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fde15196740 1 --2- 172.21.15.40:0/3606011470 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55836d6262f0 0x55836d69e320 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fde15196740 1 -- 172.21.15.40:0/3606011470 >> 172.21.15.40:0/3606011470 conn(0x55836d621860 msgr2=0x55836d625350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fde15196740 1 -- 172.21.15.40:0/3606011470 shutdown_connections 2024-03-18T22:50:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.527+0000 7fde15196740 1 -- 172.21.15.40:0/3606011470 wait complete. 2024-03-18T22:50:57.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.528+0000 7fde15196740 20 client.17664 trim_cache size 0 max 16384 2024-03-18T22:50:57.535 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:50:57.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.528+0000 7fde124d1640 1 -- 172.21.15.40:0/2567216939 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fddf40a2800 msgr2=0x7fddf40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.528+0000 7fde124d1640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fddf40a2800 0x7fddf40a4cc0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x55836d627540 tx=0x7fde04004ab0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.528+0000 7fde124d1640 1 -- 172.21.15.40:0/2567216939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde0c100720 msgr2=0x7fde0c197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.528+0000 7fde124d1640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde0c100720 0x7fde0c197a80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fddfc036040 tx=0x7fddfc0042f0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.529+0000 7fde124d1640 1 -- 172.21.15.40:0/2567216939 shutdown_connections 2024-03-18T22:50:57.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.529+0000 7fde124d1640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fddf40a2800 0x7fddf40a4cc0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.529+0000 7fde124d1640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fde0c1022d0 0x7fde0c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.529+0000 7fde124d1640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fde0c101910 0x7fde0c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.529+0000 7fde124d1640 1 --2- 172.21.15.40:0/2567216939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde0c100720 0x7fde0c197a80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.529+0000 7fde124d1640 1 -- 172.21.15.40:0/2567216939 >> 172.21.15.40:0/2567216939 conn(0x7fde0c0fbc90 msgr2=0x7fde0c068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.529+0000 7fde124d1640 1 -- 172.21.15.40:0/2567216939 shutdown_connections 2024-03-18T22:50:57.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.529+0000 7fde124d1640 1 -- 172.21.15.40:0/2567216939 wait complete. 2024-03-18T22:50:57.542 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:50:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.655+0000 7f6c772bf640 1 -- 172.21.15.40:0/286186191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c70100720 msgr2=0x7f6c70100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.655+0000 7f6c772bf640 1 --2- 172.21.15.40:0/286186191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c70100720 0x7f6c70100b40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c009900 tx=0x7f6c6c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.655+0000 7f6c772bf640 1 -- 172.21.15.40:0/286186191 shutdown_connections 2024-03-18T22:50:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.655+0000 7f6c772bf640 1 --2- 172.21.15.40:0/286186191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c701022d0 0x7f6c70108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.655+0000 7f6c772bf640 1 --2- 172.21.15.40:0/286186191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c70101910 0x7f6c70101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.655+0000 7f6c772bf640 1 --2- 172.21.15.40:0/286186191 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c70100720 0x7f6c70100b40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.655+0000 7f6c772bf640 1 -- 172.21.15.40:0/286186191 >> 172.21.15.40:0/286186191 conn(0x7f6c700fbcb0 msgr2=0x7f6c700fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.655+0000 7f6c772bf640 1 -- 172.21.15.40:0/286186191 shutdown_connections 2024-03-18T22:50:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.655+0000 7f6c772bf640 1 -- 172.21.15.40:0/286186191 wait complete. 2024-03-18T22:50:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c772bf640 1 Processor -- start 2024-03-18T22:50:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c772bf640 1 -- start start 2024-03-18T22:50:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c772bf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c70100720 0x7f6c70196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c772bf640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c70101910 0x7f6c70196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c772bf640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c701022d0 0x7f6c7019df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c772bf640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7010c220 con 0x7f6c70100720 2024-03-18T22:50:57.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c772bf640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7010c0a0 con 0x7f6c701022d0 2024-03-18T22:50:57.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c772bf640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7010c3a0 con 0x7f6c70101910 2024-03-18T22:50:57.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c7505b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c70100720 0x7f6c70196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c7505b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c70100720 0x7f6c70196950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49810/0 (socket says 172.21.15.40:49810) 2024-03-18T22:50:57.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c7505b640 1 -- 172.21.15.40:0/1078618968 learned_addr learned my addr 172.21.15.40:0/1078618968 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c7585c640 1 --2- 172.21.15.40:0/1078618968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c701022d0 0x7f6c7019df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c7485a640 1 --2- 172.21.15.40:0/1078618968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c70101910 0x7f6c70196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c7505b640 1 -- 172.21.15.40:0/1078618968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c70101910 msgr2=0x7f6c70196e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c7505b640 1 --2- 172.21.15.40:0/1078618968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c70101910 0x7f6c70196e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c7505b640 1 -- 172.21.15.40:0/1078618968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c701022d0 msgr2=0x7f6c7019df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c7505b640 1 --2- 172.21.15.40:0/1078618968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c701022d0 0x7f6c7019df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.656+0000 7f6c7505b640 1 -- 172.21.15.40:0/1078618968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c7019e570 con 0x7f6c70100720 2024-03-18T22:50:57.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.657+0000 7f6c7505b640 1 --2- 172.21.15.40:0/1078618968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c70100720 0x7f6c70196950 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c000c00 tx=0x7f6c6c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.657+0000 7f6c667fc640 1 -- 172.21.15.40:0/1078618968 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c043020 con 0x7f6c70100720 2024-03-18T22:50:57.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.657+0000 7f6c772bf640 1 -- 172.21.15.40:0/1078618968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c7019e800 con 0x7f6c70100720 2024-03-18T22:50:57.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.657+0000 7f6c667fc640 1 -- 172.21.15.40:0/1078618968 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c6c004350 con 0x7f6c70100720 2024-03-18T22:50:57.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.657+0000 7f6c667fc640 1 -- 172.21.15.40:0/1078618968 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c003b80 con 0x7f6c70100720 2024-03-18T22:50:57.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.657+0000 7f6c772bf640 1 -- 172.21.15.40:0/1078618968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c7019eda0 con 0x7f6c70100720 2024-03-18T22:50:57.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c79f89740 1 Processor -- start 2024-03-18T22:50:57.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c667fc640 1 -- 172.21.15.40:0/1078618968 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6c6c046020 con 0x7f6c70100720 2024-03-18T22:50:57.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c79f89740 1 -- start start 2024-03-18T22:50:57.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c79f89740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1500 0x560a47ec1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c79f89740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec1ef0 0x560a47ec69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c79f89740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec7060 0x560a47ec94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c79f89740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560a47e234b0 con 0x560a47ec1500 2024-03-18T22:50:57.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c79f89740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560a47dcbd70 con 0x560a47ec7060 2024-03-18T22:50:57.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c79f89740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560a47e1f550 con 0x560a47ec1ef0 2024-03-18T22:50:57.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c7585c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1500 0x560a47ec1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c7485a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec7060 0x560a47ec94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c7585c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1500 0x560a47ec1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49816/0 (socket says 172.21.15.40:49816) 2024-03-18T22:50:57.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c7485a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec7060 0x560a47ec94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47054/0 (socket says 172.21.15.40:47054) 2024-03-18T22:50:57.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c7585c640 1 -- 172.21.15.40:0/3641585214 learned_addr learned my addr 172.21.15.40:0/3641585214 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c7505b640 1 --2- 172.21.15.40:0/3641585214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec1ef0 0x560a47ec69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c7485a640 1 -- 172.21.15.40:0/3641585214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec1ef0 msgr2=0x560a47ec69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c7485a640 1 --2- 172.21.15.40:0/3641585214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec1ef0 0x560a47ec69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c7485a640 1 -- 172.21.15.40:0/3641585214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1500 msgr2=0x560a47ec1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c7485a640 1 --2- 172.21.15.40:0/3641585214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1500 0x560a47ec1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c7485a640 1 -- 172.21.15.40:0/3641585214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a47dee210 con 0x560a47ec7060 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c7585c640 1 --2- 172.21.15.40:0/3641585214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1500 0x560a47ec1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.660+0000 7f6c667fc640 1 --2- 172.21.15.40:0/1078618968 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6c5c0a2800 0x7f6c5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.661+0000 7f6c7485a640 1 --2- 172.21.15.40:0/3641585214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec7060 0x560a47ec94f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6c60009bf0 tx=0x7f6c60011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaf451ac706ea481 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.661+0000 7f6c7585c640 1 --2- 172.21.15.40:0/1078618968 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6c5c0a2800 0x7f6c5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.661+0000 7f6c667fc640 1 -- 172.21.15.40:0/1078618968 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6c6c0f0fa0 con 0x7f6c70100720 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.661+0000 7f6c5b7fe640 1 -- 172.21.15.40:0/3641585214 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c60004050 con 0x560a47ec7060 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.661+0000 7f6c5b7fe640 1 -- 172.21.15.40:0/3641585214 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c600054d0 con 0x560a47ec7060 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.661+0000 7f6c5b7fe640 1 -- 172.21.15.40:0/3641585214 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c60004cc0 con 0x560a47ec7060 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.662+0000 7f6c79f89740 1 -- 172.21.15.40:0/3641585214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec7060 msgr2=0x560a47ec94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.662+0000 7f6c79f89740 1 --2- 172.21.15.40:0/3641585214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec7060 0x560a47ec94f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6c60009bf0 tx=0x7f6c60011810 comp rx=0 tx=0).stop 2024-03-18T22:50:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.662+0000 7f6c7585c640 1 --2- 172.21.15.40:0/1078618968 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6c5c0a2800 0x7f6c5c0a4cc0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x560a47ec2750 tx=0x7f6c68009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.662+0000 7f6c79f89740 1 -- 172.21.15.40:0/3641585214 shutdown_connections 2024-03-18T22:50:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.662+0000 7f6c79f89740 1 --2- 172.21.15.40:0/3641585214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec7060 0x560a47ec94f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.662+0000 7f6c79f89740 1 --2- 172.21.15.40:0/3641585214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec1ef0 0x560a47ec69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.662+0000 7f6c79f89740 1 --2- 172.21.15.40:0/3641585214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1500 0x560a47ec1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.662+0000 7f6c79f89740 1 -- 172.21.15.40:0/3641585214 >> 172.21.15.40:0/3641585214 conn(0x560a47eba860 msgr2=0x560a47ebac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.662+0000 7f6c79f89740 1 -- 172.21.15.40:0/3641585214 shutdown_connections 2024-03-18T22:50:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.662+0000 7f6c79f89740 1 -- 172.21.15.40:0/3641585214 wait complete. 2024-03-18T22:50:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.662+0000 7f6c79f89740 1 -- start start 2024-03-18T22:50:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.662+0000 7f6c79f89740 1 Processor -- start 2024-03-18T22:50:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c79f89740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec1500 0x560a47ecb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c79f89740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1ef0 0x560a47ecbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c79f89740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec7060 0x560a47ed2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c79f89740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560a47eca010 con 0x560a47ec7060 2024-03-18T22:50:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c79f89740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560a47ddf310 con 0x560a47ec1ef0 2024-03-18T22:50:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c79f89740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560a47eca190 con 0x560a47ec1500 2024-03-18T22:50:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c7585c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec7060 0x560a47ed2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c7485a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1ef0 0x560a47ecbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c7585c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec7060 0x560a47ed2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49820/0 (socket says 172.21.15.40:49820) 2024-03-18T22:50:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c7585c640 1 -- 172.21.15.40:0/4157766039 learned_addr learned my addr 172.21.15.40:0/4157766039 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c7485a640 1 --2- 172.21.15.40:0/4157766039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1ef0 0x560a47ecbed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47060/0 (socket says 172.21.15.40:47060) 2024-03-18T22:50:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c79f89740 10 client.0 _ll_register_callbacks cb 0x560a47dd32d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c7505b640 1 --2- 172.21.15.40:0/4157766039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec1500 0x560a47ecb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c7585c640 1 -- 172.21.15.40:0/4157766039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec1500 msgr2=0x560a47ecb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c7585c640 1 --2- 172.21.15.40:0/4157766039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec1500 0x560a47ecb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c7585c640 1 -- 172.21.15.40:0/4157766039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1ef0 msgr2=0x560a47ecbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c7585c640 1 --2- 172.21.15.40:0/4157766039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1ef0 0x560a47ecbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c7585c640 1 -- 172.21.15.40:0/4157766039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a47b373f0 con 0x560a47ec7060 2024-03-18T22:50:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c7505b640 1 --2- 172.21.15.40:0/4157766039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec1500 0x560a47ecb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c7585c640 1 --2- 172.21.15.40:0/4157766039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec7060 0x560a47ed2f10 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f6c68003bc0 tx=0x7f6c68004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75053d9602094fa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.663+0000 7f6c58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.664+0000 7f6c58ff9640 1 -- 172.21.15.40:0/4157766039 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68004310 con 0x560a47ec7060 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.664+0000 7f6c58ff9640 1 -- 172.21.15.40:0/4157766039 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6c680044b0 con 0x560a47ec7060 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.664+0000 7f6c79f89740 1 -- 172.21.15.40:0/4157766039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c600054d0 con 0x560a47ec7060 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.664+0000 7f6c58ff9640 1 -- 172.21.15.40:0/4157766039 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f6c680046b0 con 0x560a47ec7060 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.664+0000 7f6c79f89740 10 client.17676 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.664+0000 7f6c79f89740 1 -- 172.21.15.40:0/4157766039 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560a47c017d0 con 0x560a47ec7060 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.664+0000 7f6c58ff9640 1 -- 172.21.15.40:0/4157766039 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c600054d0 con 0x560a47ec7060 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.664+0000 7f6c58ff9640 1 -- 172.21.15.40:0/4157766039 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f6c68026070 con 0x560a47ec7060 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.664+0000 7f6c79f89740 10 client.17676 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.664+0000 7f6c79f89740 10 client.17676 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.664+0000 7f6c79f89740 20 client.17676 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.664+0000 7f6c79f89740 1 --2- 172.21.15.40:0/4157766039 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x560a47ed7ac0 0x560a47ed9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.664+0000 7f6c79f89740 4 client.17676 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.664+0000 7f6c79f89740 1 -- 172.21.15.40:0/4157766039 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a47e813e0 con 0x560a47ed7ac0 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.664+0000 7f6c7505b640 1 --2- 172.21.15.40:0/4157766039 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x560a47ed7ac0 0x560a47ed9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.665+0000 7f6c58ff9640 10 client.17676 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.665+0000 7f6c7505b640 1 --2- 172.21.15.40:0/4157766039 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x560a47ed7ac0 0x560a47ed9f80 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d2a159a073847270 server_cookie=e926997325b1f588 in_seq=0 out_seq=0 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.665+0000 7f6c58ff9640 1 -- 172.21.15.40:0/4157766039 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560a47e813e0 con 0x560a47ed7ac0 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.666+0000 7f6c58ff9640 10 client.17676 handle_command_reply: tid=0 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.666+0000 7f6c79f89740 1 client.17676 shutdown 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.666+0000 7f6c79f89740 1 -- 172.21.15.40:0/4157766039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec7060 msgr2=0x560a47ed2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.666+0000 7f6c79f89740 1 --2- 172.21.15.40:0/4157766039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec7060 0x560a47ed2f10 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f6c68003bc0 tx=0x7f6c68004ab0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.666+0000 7f6c79f89740 1 -- 172.21.15.40:0/4157766039 shutdown_connections 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.666+0000 7f6c79f89740 1 --2- 172.21.15.40:0/4157766039 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x560a47ed7ac0 0x560a47ed9f80 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.666+0000 7f6c79f89740 1 --2- 172.21.15.40:0/4157766039 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec7060 0x560a47ed2f10 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.666+0000 7f6c79f89740 1 --2- 172.21.15.40:0/4157766039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1ef0 0x560a47ecbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.666+0000 7f6c79f89740 1 --2- 172.21.15.40:0/4157766039 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec1500 0x560a47ecb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.666+0000 7f6c79f89740 1 -- 172.21.15.40:0/4157766039 >> 172.21.15.40:0/4157766039 conn(0x560a47eba860 msgr2=0x560a47eb9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.666+0000 7f6c58ff9640 0 client.17676 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.666+0000 7f6c79f89740 1 -- 172.21.15.40:0/4157766039 shutdown_connections 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.666+0000 7f6c79f89740 1 -- 172.21.15.40:0/4157766039 wait complete. 2024-03-18T22:50:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.666+0000 7f6c79f89740 20 client.17676 trim_cache size 0 max 16384 2024-03-18T22:50:57.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.680+0000 7f6c79f89740 1 Processor -- start 2024-03-18T22:50:57.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.680+0000 7f6c79f89740 1 -- start start 2024-03-18T22:50:57.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.680+0000 7f6c79f89740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1500 0x560a47ed3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.680+0000 7f6c79f89740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1ef0 0x560a47ed2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.680+0000 7f6c79f89740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec7060 0x560a47ed3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.680+0000 7f6c79f89740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6c701a8350 con 0x560a47ec1500 2024-03-18T22:50:57.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.680+0000 7f6c79f89740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6c701a9f50 con 0x560a47ec1ef0 2024-03-18T22:50:57.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.680+0000 7f6c79f89740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6c701a8a20 con 0x560a47ec7060 2024-03-18T22:50:57.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.681+0000 7f6c7505b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1500 0x560a47ed3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.681+0000 7f6c7585c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec7060 0x560a47ed3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.681+0000 7f6c7485a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1ef0 0x560a47ed2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.681+0000 7f6c7505b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1500 0x560a47ed3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49836/0 (socket says 172.21.15.40:49836) 2024-03-18T22:50:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.681+0000 7f6c7505b640 1 -- 172.21.15.40:0/3146749694 learned_addr learned my addr 172.21.15.40:0/3146749694 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.681+0000 7f6c7585c640 1 -- 172.21.15.40:0/3146749694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1ef0 msgr2=0x560a47ed2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.681+0000 7f6c7585c640 1 --2- 172.21.15.40:0/3146749694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1ef0 0x560a47ed2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.681+0000 7f6c7585c640 1 -- 172.21.15.40:0/3146749694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1500 msgr2=0x560a47ed3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.681+0000 7f6c7585c640 1 --2- 172.21.15.40:0/3146749694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1500 0x560a47ed3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.681+0000 7f6c7585c640 1 -- 172.21.15.40:0/3146749694 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a47e90880 con 0x560a47ec7060 2024-03-18T22:50:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.681+0000 7f6c7585c640 1 --2- 172.21.15.40:0/3146749694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec7060 0x560a47ed3ea0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6c68018040 tx=0x7f6c680095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee38e3ad78f75f25 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.681+0000 7f6c5affd640 1 -- 172.21.15.40:0/3146749694 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6803b070 con 0x560a47ec7060 2024-03-18T22:50:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.681+0000 7f6c5affd640 1 -- 172.21.15.40:0/3146749694 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c68004850 con 0x560a47ec7060 2024-03-18T22:50:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.681+0000 7f6c5affd640 1 -- 172.21.15.40:0/3146749694 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68036800 con 0x560a47ec7060 2024-03-18T22:50:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.682+0000 7f6c79f89740 1 -- 172.21.15.40:0/3146749694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec7060 msgr2=0x560a47ed3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.682+0000 7f6c79f89740 1 --2- 172.21.15.40:0/3146749694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec7060 0x560a47ed3ea0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6c68018040 tx=0x7f6c680095d0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.682+0000 7f6c79f89740 1 -- 172.21.15.40:0/3146749694 shutdown_connections 2024-03-18T22:50:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.682+0000 7f6c79f89740 1 --2- 172.21.15.40:0/3146749694 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec7060 0x560a47ed3ea0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.682+0000 7f6c79f89740 1 --2- 172.21.15.40:0/3146749694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1ef0 0x560a47ed2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.682+0000 7f6c79f89740 1 --2- 172.21.15.40:0/3146749694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1500 0x560a47ed3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.682+0000 7f6c79f89740 1 -- 172.21.15.40:0/3146749694 >> 172.21.15.40:0/3146749694 conn(0x560a47eba860 msgr2=0x560a47eb5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.682+0000 7f6c79f89740 1 -- 172.21.15.40:0/3146749694 shutdown_connections 2024-03-18T22:50:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.682+0000 7f6c79f89740 1 -- 172.21.15.40:0/3146749694 wait complete. 2024-03-18T22:50:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.682+0000 7f6c79f89740 1 -- start start 2024-03-18T22:50:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c79f89740 1 Processor -- start 2024-03-18T22:50:57.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c79f89740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1500 0x560a47ed04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c79f89740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1ef0 0x560a47ed0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c79f89740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec7060 0x560a47ed1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c79f89740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6c701a9880 con 0x560a47ec1ef0 2024-03-18T22:50:57.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c79f89740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6c701a90f0 con 0x560a47ec1500 2024-03-18T22:50:57.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c79f89740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6c701a6330 con 0x560a47ec7060 2024-03-18T22:50:57.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c79f89740 10 client.0 _ll_register_callbacks cb 0x7f6c68011e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:57.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c7485a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1ef0 0x560a47ed0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c7505b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1500 0x560a47ed04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c7485a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1ef0 0x560a47ed0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49842/0 (socket says 172.21.15.40:49842) 2024-03-18T22:50:57.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c7505b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1500 0x560a47ed04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47088/0 (socket says 172.21.15.40:47088) 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c7485a640 1 -- 172.21.15.40:0/193990089 learned_addr learned my addr 172.21.15.40:0/193990089 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c7585c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec7060 0x560a47ed1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c7505b640 1 -- 172.21.15.40:0/193990089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec7060 msgr2=0x560a47ed1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c7505b640 1 --2- 172.21.15.40:0/193990089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec7060 0x560a47ed1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c7505b640 1 -- 172.21.15.40:0/193990089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1ef0 msgr2=0x560a47ed0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c7505b640 1 --2- 172.21.15.40:0/193990089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1ef0 0x560a47ed0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c7505b640 1 -- 172.21.15.40:0/193990089 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a47ed2020 con 0x560a47ec1500 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c7585c640 1 --2- 172.21.15.40:0/193990089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec7060 0x560a47ed1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.683+0000 7f6c7485a640 1 --2- 172.21.15.40:0/193990089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1ef0 0x560a47ed0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c7505b640 1 --2- 172.21.15.40:0/193990089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1500 0x560a47ed04e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c0bd250 tx=0x7f6c6c0a8040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f87716a4c3099de server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c58ff9640 1 -- 172.21.15.40:0/193990089 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c0b97f0 con 0x560a47ec1500 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c79f89740 1 -- 172.21.15.40:0/193990089 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c68004850 con 0x560a47ec1500 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c58ff9640 1 -- 172.21.15.40:0/193990089 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6c6c0054d0 con 0x560a47ec1500 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c58ff9640 1 -- 172.21.15.40:0/193990089 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f6c6c0bceb0 con 0x560a47ec1500 2024-03-18T22:50:57.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c79f89740 10 client.27497 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:57.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c79f89740 1 -- 172.21.15.40:0/193990089 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560a47f3dec0 con 0x560a47ec1500 2024-03-18T22:50:57.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c58ff9640 1 -- 172.21.15.40:0/193990089 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c68004850 con 0x560a47ec1500 2024-03-18T22:50:57.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c58ff9640 1 -- 172.21.15.40:0/193990089 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f6c6c0f30b0 con 0x560a47ec1500 2024-03-18T22:50:57.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c79f89740 10 client.27497 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:57.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c79f89740 10 client.27497 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:50:57.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c79f89740 20 client.27497 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:57.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c79f89740 1 --2- 172.21.15.40:0/193990089 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x560a47f3e720 0x560a47f40b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c79f89740 4 client.27497 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:50:57.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.684+0000 7f6c79f89740 1 -- 172.21.15.40:0/193990089 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560a47e309f0 con 0x560a47f3e720 2024-03-18T22:50:57.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.685+0000 7f6c7485a640 1 --2- 172.21.15.40:0/193990089 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x560a47f3e720 0x560a47f40b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.685+0000 7f6c58ff9640 10 client.27497 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:50:57.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.685+0000 7f6c7485a640 1 --2- 172.21.15.40:0/193990089 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x560a47f3e720 0x560a47f40b40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=74cad9700fe8ea6b server_cookie=e624103f7ac0ef0f in_seq=0 out_seq=0 2024-03-18T22:50:57.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.685+0000 7f6c58ff9640 1 -- 172.21.15.40:0/193990089 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560a47e309f0 con 0x560a47f3e720 2024-03-18T22:50:57.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.685+0000 7f6c58ff9640 10 client.27497 handle_command_reply: tid=0 2024-03-18T22:50:57.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.685+0000 7f6c79f89740 1 client.27497 shutdown 2024-03-18T22:50:57.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.685+0000 7f6c79f89740 1 -- 172.21.15.40:0/193990089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1500 msgr2=0x560a47ed04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.685+0000 7f6c79f89740 1 --2- 172.21.15.40:0/193990089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1500 0x560a47ed04e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c0bd250 tx=0x7f6c6c0a8040 comp rx=0 tx=0).stop 2024-03-18T22:50:57.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.685+0000 7f6c79f89740 1 -- 172.21.15.40:0/193990089 shutdown_connections 2024-03-18T22:50:57.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.685+0000 7f6c79f89740 1 --2- 172.21.15.40:0/193990089 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x560a47f3e720 0x560a47f40b40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.685+0000 7f6c79f89740 1 --2- 172.21.15.40:0/193990089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560a47ec7060 0x560a47ed1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.685+0000 7f6c79f89740 1 --2- 172.21.15.40:0/193990089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560a47ec1ef0 0x560a47ed0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.685+0000 7f6c79f89740 1 --2- 172.21.15.40:0/193990089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560a47ec1500 0x560a47ed04e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.685+0000 7f6c58ff9640 0 client.27497 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:50:57.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.685+0000 7f6c79f89740 1 -- 172.21.15.40:0/193990089 >> 172.21.15.40:0/193990089 conn(0x560a47eba860 msgr2=0x560a47ecb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.686+0000 7f6c79f89740 1 -- 172.21.15.40:0/193990089 shutdown_connections 2024-03-18T22:50:57.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.686+0000 7f6c79f89740 1 -- 172.21.15.40:0/193990089 wait complete. 2024-03-18T22:50:57.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.686+0000 7f6c79f89740 20 client.27497 trim_cache size 0 max 16384 2024-03-18T22:50:57.693 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:50:57.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.686+0000 7f6c772bf640 1 -- 172.21.15.40:0/1078618968 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6c5c0a2800 msgr2=0x7f6c5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.686+0000 7f6c772bf640 1 --2- 172.21.15.40:0/1078618968 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6c5c0a2800 0x7f6c5c0a4cc0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x560a47ec2750 tx=0x7f6c68009500 comp rx=0 tx=0).stop 2024-03-18T22:50:57.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.686+0000 7f6c772bf640 1 -- 172.21.15.40:0/1078618968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c70100720 msgr2=0x7f6c70196950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.686+0000 7f6c772bf640 1 --2- 172.21.15.40:0/1078618968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c70100720 0x7f6c70196950 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c000c00 tx=0x7f6c6c005dc0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.686+0000 7f6c772bf640 1 -- 172.21.15.40:0/1078618968 shutdown_connections 2024-03-18T22:50:57.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.686+0000 7f6c772bf640 1 --2- 172.21.15.40:0/1078618968 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6c5c0a2800 0x7f6c5c0a4cc0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.686+0000 7f6c772bf640 1 --2- 172.21.15.40:0/1078618968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c701022d0 0x7f6c7019df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.686+0000 7f6c772bf640 1 --2- 172.21.15.40:0/1078618968 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6c70101910 0x7f6c70196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.686+0000 7f6c772bf640 1 --2- 172.21.15.40:0/1078618968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c70100720 0x7f6c70196950 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.686+0000 7f6c772bf640 1 -- 172.21.15.40:0/1078618968 >> 172.21.15.40:0/1078618968 conn(0x7f6c700fbcb0 msgr2=0x7f6c70104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.686+0000 7f6c772bf640 1 -- 172.21.15.40:0/1078618968 shutdown_connections 2024-03-18T22:50:57.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.686+0000 7f6c772bf640 1 -- 172.21.15.40:0/1078618968 wait complete. 2024-03-18T22:50:57.699 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:50:57.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.817+0000 7f0b3d5bc640 1 -- 172.21.15.40:0/2770136913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b38100720 msgr2=0x7f0b38100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.817+0000 7f0b3d5bc640 1 --2- 172.21.15.40:0/2770136913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b38100720 0x7f0b38100b40 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f0b28009900 tx=0x7f0b2802bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:57.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.818+0000 7f0b3d5bc640 1 -- 172.21.15.40:0/2770136913 shutdown_connections 2024-03-18T22:50:57.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.818+0000 7f0b3d5bc640 1 --2- 172.21.15.40:0/2770136913 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b381022d0 0x7f0b38108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.818+0000 7f0b3d5bc640 1 --2- 172.21.15.40:0/2770136913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b38101910 0x7f0b38101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.818+0000 7f0b3d5bc640 1 --2- 172.21.15.40:0/2770136913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b38100720 0x7f0b38100b40 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.818+0000 7f0b3d5bc640 1 -- 172.21.15.40:0/2770136913 >> 172.21.15.40:0/2770136913 conn(0x7f0b380fbc90 msgr2=0x7f0b380fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.818+0000 7f0b3d5bc640 1 -- 172.21.15.40:0/2770136913 shutdown_connections 2024-03-18T22:50:57.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.818+0000 7f0b3d5bc640 1 -- 172.21.15.40:0/2770136913 wait complete. 2024-03-18T22:50:57.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.818+0000 7f0b3d5bc640 1 Processor -- start 2024-03-18T22:50:57.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.818+0000 7f0b3d5bc640 1 -- start start 2024-03-18T22:50:57.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.818+0000 7f0b3d5bc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b38100720 0x7f0b381046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b3d5bc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b38101910 0x7f0b38102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b3d5bc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b381022d0 0x7f0b38103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b3d5bc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3810c250 con 0x7f0b381022d0 2024-03-18T22:50:57.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b3d5bc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3810c0d0 con 0x7f0b38101910 2024-03-18T22:50:57.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b3d5bc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3810c3d0 con 0x7f0b38100720 2024-03-18T22:50:57.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b36ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b38100720 0x7f0b381046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b377fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b381022d0 0x7f0b38103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b367fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b38101910 0x7f0b38102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b36ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b38100720 0x7f0b381046b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:34268/0 (socket says 172.21.15.40:34268) 2024-03-18T22:50:57.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b377fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b381022d0 0x7f0b38103230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49870/0 (socket says 172.21.15.40:49870) 2024-03-18T22:50:57.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b36ffd640 1 -- 172.21.15.40:0/2347726229 learned_addr learned my addr 172.21.15.40:0/2347726229 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b367fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b38101910 0x7f0b38102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47110/0 (socket says 172.21.15.40:47110) 2024-03-18T22:50:57.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b377fe640 1 -- 172.21.15.40:0/2347726229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b38100720 msgr2=0x7f0b381046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b377fe640 1 --2- 172.21.15.40:0/2347726229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b38100720 0x7f0b381046b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b377fe640 1 -- 172.21.15.40:0/2347726229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b38101910 msgr2=0x7f0b38102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b377fe640 1 --2- 172.21.15.40:0/2347726229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b38101910 0x7f0b38102cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b377fe640 1 -- 172.21.15.40:0/2347726229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b38103ac0 con 0x7f0b381022d0 2024-03-18T22:50:57.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b36ffd640 1 --2- 172.21.15.40:0/2347726229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b38100720 0x7f0b381046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b367fc640 1 --2- 172.21.15.40:0/2347726229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b38101910 0x7f0b38102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b377fe640 1 --2- 172.21.15.40:0/2347726229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b381022d0 0x7f0b38103230 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009fd0 tx=0x7f0b3000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b1ffff640 1 -- 172.21.15.40:0/2347726229 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3000ce20 con 0x7f0b381022d0 2024-03-18T22:50:57.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b3d5bc640 1 -- 172.21.15.40:0/2347726229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b381a1b90 con 0x7f0b381022d0 2024-03-18T22:50:57.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b1ffff640 1 -- 172.21.15.40:0/2347726229 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b30005020 con 0x7f0b381022d0 2024-03-18T22:50:57.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.819+0000 7f0b1ffff640 1 -- 172.21.15.40:0/2347726229 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b30010690 con 0x7f0b381022d0 2024-03-18T22:50:57.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.820+0000 7f0b3d5bc640 1 -- 172.21.15.40:0/2347726229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b381a20e0 con 0x7f0b381022d0 2024-03-18T22:50:57.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.822+0000 7f0b1ffff640 1 -- 172.21.15.40:0/2347726229 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0b300051d0 con 0x7f0b381022d0 2024-03-18T22:50:57.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b4019b740 1 Processor -- start 2024-03-18T22:50:57.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b4019b740 1 -- start start 2024-03-18T22:50:57.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b4019b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224d82f0 0x5647224d8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b4019b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224dd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b4019b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224dde50 0x5647224e02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b4019b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56472243c4b0 con 0x5647224d8ce0 2024-03-18T22:50:57.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b4019b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5647223e4d70 con 0x5647224dde50 2024-03-18T22:50:57.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b4019b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x564722438550 con 0x5647224d82f0 2024-03-18T22:50:57.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b36ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224dd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b36ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224dd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49872/0 (socket says 172.21.15.40:49872) 2024-03-18T22:50:57.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b36ffd640 1 -- 172.21.15.40:0/1404165603 learned_addr learned my addr 172.21.15.40:0/1404165603 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b367fc640 1 --2- 172.21.15.40:0/1404165603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224d82f0 0x5647224d8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b367fc640 1 --2- 172.21.15.40:0/1404165603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224dde50 0x5647224e02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b1ffff640 1 --2- 172.21.15.40:0/2347726229 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0b200a2800 0x7f0b200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b36ffd640 1 -- 172.21.15.40:0/1404165603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224d82f0 msgr2=0x5647224d8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b36ffd640 1 --2- 172.21.15.40:0/1404165603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224d82f0 0x5647224d8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b36ffd640 1 -- 172.21.15.40:0/1404165603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224dde50 msgr2=0x5647224e02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b36ffd640 1 --2- 172.21.15.40:0/1404165603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224dde50 0x5647224e02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b36ffd640 1 -- 172.21.15.40:0/1404165603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564722407210 con 0x5647224d8ce0 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b377fe640 1 --2- 172.21.15.40:0/2347726229 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0b200a2800 0x7f0b200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.823+0000 7f0b1ffff640 1 -- 172.21.15.40:0/2347726229 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0b300cf540 con 0x7f0b381022d0 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.824+0000 7f0b36ffd640 1 --2- 172.21.15.40:0/1404165603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224dd7e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0b28036040 tx=0x7f0b28009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca3306788f12e83a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.824+0000 7f0b1d7fa640 1 -- 172.21.15.40:0/1404165603 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b28045020 con 0x5647224d8ce0 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.824+0000 7f0b1d7fa640 1 -- 172.21.15.40:0/1404165603 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b2802cc90 con 0x5647224d8ce0 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.824+0000 7f0b1d7fa640 1 -- 172.21.15.40:0/1404165603 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b280048c0 con 0x5647224d8ce0 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.825+0000 7f0b4019b740 1 -- 172.21.15.40:0/1404165603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 msgr2=0x5647224dd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.825+0000 7f0b4019b740 1 --2- 172.21.15.40:0/1404165603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224dd7e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0b28036040 tx=0x7f0b28009660 comp rx=0 tx=0).stop 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.825+0000 7f0b377fe640 1 --2- 172.21.15.40:0/2347726229 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0b200a2800 0x7f0b200a4cc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0b3009ba80 tx=0x7f0b30013700 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.825+0000 7f0b4019b740 1 -- 172.21.15.40:0/1404165603 shutdown_connections 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.825+0000 7f0b4019b740 1 --2- 172.21.15.40:0/1404165603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224dde50 0x5647224e02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.825+0000 7f0b4019b740 1 --2- 172.21.15.40:0/1404165603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224dd7e0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.825+0000 7f0b4019b740 1 --2- 172.21.15.40:0/1404165603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224d82f0 0x5647224d8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.825+0000 7f0b4019b740 1 -- 172.21.15.40:0/1404165603 >> 172.21.15.40:0/1404165603 conn(0x5647224d3860 msgr2=0x5647224d3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.825+0000 7f0b4019b740 1 -- 172.21.15.40:0/1404165603 shutdown_connections 2024-03-18T22:50:57.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.825+0000 7f0b4019b740 1 -- 172.21.15.40:0/1404165603 wait complete. 2024-03-18T22:50:57.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.825+0000 7f0b4019b740 1 -- start start 2024-03-18T22:50:57.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.825+0000 7f0b4019b740 1 Processor -- start 2024-03-18T22:50:57.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b4019b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 0x5647224e27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b4019b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b4019b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224dde50 0x5647224e9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b4019b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5647224e0da0 con 0x5647224d8ce0 2024-03-18T22:50:57.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b4019b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5647223f8310 con 0x5647224d82f0 2024-03-18T22:50:57.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b4019b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5647224e0f20 con 0x5647224dde50 2024-03-18T22:50:57.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b4019b740 10 client.0 _ll_register_callbacks cb 0x5647223ec2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:57.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b36ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b367fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 0x5647224e27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b36ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49884/0 (socket says 172.21.15.40:49884) 2024-03-18T22:50:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b36ffd640 1 -- 172.21.15.40:0/518662015 learned_addr learned my addr 172.21.15.40:0/518662015 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b367fc640 1 --2- 172.21.15.40:0/518662015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224dde50 0x5647224e9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b36ffd640 1 -- 172.21.15.40:0/518662015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224dde50 msgr2=0x5647224e9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b36ffd640 1 --2- 172.21.15.40:0/518662015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224dde50 0x5647224e9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b36ffd640 1 -- 172.21.15.40:0/518662015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 msgr2=0x5647224e27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b36ffd640 1 --2- 172.21.15.40:0/518662015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 0x5647224e27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b36ffd640 1 -- 172.21.15.40:0/518662015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647221503f0 con 0x5647224d8ce0 2024-03-18T22:50:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b367fc640 1 --2- 172.21.15.40:0/518662015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 0x5647224e27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b36ffd640 1 --2- 172.21.15.40:0/518662015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e2ce0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0b28005ab0 tx=0x7f0b28005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89aee6f24cfcbe0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b1aff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.826+0000 7f0b1aff5640 1 -- 172.21.15.40:0/518662015 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b28050030 con 0x5647224d8ce0 2024-03-18T22:50:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.827+0000 7f0b4019b740 1 -- 172.21.15.40:0/518662015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b2802cc90 con 0x5647224d8ce0 2024-03-18T22:50:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.827+0000 7f0b1aff5640 1 -- 172.21.15.40:0/518662015 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0b28003800 con 0x5647224d8ce0 2024-03-18T22:50:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.827+0000 7f0b1aff5640 1 -- 172.21.15.40:0/518662015 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0b2805a050 con 0x5647224d8ce0 2024-03-18T22:50:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.827+0000 7f0b4019b740 10 client.17694 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.827+0000 7f0b4019b740 1 -- 172.21.15.40:0/518662015 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56472221a7d0 con 0x5647224d8ce0 2024-03-18T22:50:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.827+0000 7f0b1aff5640 1 -- 172.21.15.40:0/518662015 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b2802cc90 con 0x5647224d8ce0 2024-03-18T22:50:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.827+0000 7f0b1aff5640 1 -- 172.21.15.40:0/518662015 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f0b28041b20 con 0x5647224d8ce0 2024-03-18T22:50:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.827+0000 7f0b4019b740 10 client.17694 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.827+0000 7f0b4019b740 10 client.17694 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:50:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.827+0000 7f0b4019b740 20 client.17694 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.827+0000 7f0b4019b740 1 --2- 172.21.15.40:0/518662015 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5647224ee260 0x5647224f0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.827+0000 7f0b4019b740 4 client.17694 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.827+0000 7f0b4019b740 1 -- 172.21.15.40:0/518662015 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56472249a3e0 con 0x5647224ee260 2024-03-18T22:50:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.827+0000 7f0b367fc640 1 --2- 172.21.15.40:0/518662015 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5647224ee260 0x5647224f0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.828+0000 7f0b1aff5640 10 client.17694 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:50:57.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.828+0000 7f0b367fc640 1 --2- 172.21.15.40:0/518662015 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5647224ee260 0x5647224f0690 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=62cd78933419c115 server_cookie=aa400242f7abac31 in_seq=0 out_seq=0 2024-03-18T22:50:57.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.828+0000 7f0b1aff5640 1 -- 172.21.15.40:0/518662015 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56472249a3e0 con 0x5647224ee260 2024-03-18T22:50:57.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.828+0000 7f0b1aff5640 10 client.17694 handle_command_reply: tid=0 2024-03-18T22:50:57.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.828+0000 7f0b4019b740 1 client.17694 shutdown 2024-03-18T22:50:57.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.829+0000 7f0b4019b740 1 -- 172.21.15.40:0/518662015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 msgr2=0x5647224e2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.829+0000 7f0b4019b740 1 --2- 172.21.15.40:0/518662015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e2ce0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0b28005ab0 tx=0x7f0b28005b50 comp rx=0 tx=0).stop 2024-03-18T22:50:57.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.829+0000 7f0b4019b740 1 -- 172.21.15.40:0/518662015 shutdown_connections 2024-03-18T22:50:57.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.829+0000 7f0b4019b740 1 --2- 172.21.15.40:0/518662015 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5647224ee260 0x5647224f0690 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.829+0000 7f0b4019b740 1 --2- 172.21.15.40:0/518662015 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224dde50 0x5647224e9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.829+0000 7f0b4019b740 1 --2- 172.21.15.40:0/518662015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e2ce0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.829+0000 7f0b4019b740 1 --2- 172.21.15.40:0/518662015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 0x5647224e27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.829+0000 7f0b4019b740 1 -- 172.21.15.40:0/518662015 >> 172.21.15.40:0/518662015 conn(0x5647224d3860 msgr2=0x5647224d2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.829+0000 7f0b1aff5640 0 client.17694 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:50:57.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.829+0000 7f0b4019b740 1 -- 172.21.15.40:0/518662015 shutdown_connections 2024-03-18T22:50:57.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.829+0000 7f0b4019b740 1 -- 172.21.15.40:0/518662015 wait complete. 2024-03-18T22:50:57.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.829+0000 7f0b4019b740 20 client.17694 trim_cache size 0 max 16384 2024-03-18T22:50:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.845+0000 7f0b4019b740 1 Processor -- start 2024-03-18T22:50:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.845+0000 7f0b4019b740 1 -- start start 2024-03-18T22:50:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.845+0000 7f0b4019b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 0x5647224d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.845+0000 7f0b4019b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b4019b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224dde50 0x5647224d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b4019b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0b381ac670 con 0x5647224d8ce0 2024-03-18T22:50:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b4019b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0b381ae270 con 0x5647224d82f0 2024-03-18T22:50:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b4019b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0b381acd40 con 0x5647224dde50 2024-03-18T22:50:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b36ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b36ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49898/0 (socket says 172.21.15.40:49898) 2024-03-18T22:50:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b36ffd640 1 -- 172.21.15.40:0/2526242958 learned_addr learned my addr 172.21.15.40:0/2526242958 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b367fc640 1 --2- 172.21.15.40:0/2526242958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 0x5647224d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b367fc640 1 --2- 172.21.15.40:0/2526242958 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224dde50 0x5647224d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b367fc640 1 -- 172.21.15.40:0/2526242958 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224dde50 msgr2=0x5647224d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b367fc640 1 --2- 172.21.15.40:0/2526242958 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224dde50 0x5647224d2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b367fc640 1 -- 172.21.15.40:0/2526242958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 msgr2=0x5647224e9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b367fc640 1 --2- 172.21.15.40:0/2526242958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b367fc640 1 -- 172.21.15.40:0/2526242958 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647224d47d0 con 0x5647224d82f0 2024-03-18T22:50:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b36ffd640 1 --2- 172.21.15.40:0/2526242958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b367fc640 1 --2- 172.21.15.40:0/2526242958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 0x5647224d2310 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x5647224f0ea0 tx=0x7f0b2c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86f0452ef4dd628b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b1cff9640 1 -- 172.21.15.40:0/2526242958 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c004310 con 0x5647224d82f0 2024-03-18T22:50:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.846+0000 7f0b1cff9640 1 -- 172.21.15.40:0/2526242958 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b2c0044b0 con 0x5647224d82f0 2024-03-18T22:50:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.847+0000 7f0b1cff9640 1 -- 172.21.15.40:0/2526242958 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c0047a0 con 0x5647224d82f0 2024-03-18T22:50:57.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.847+0000 7f0b4019b740 1 -- 172.21.15.40:0/2526242958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 msgr2=0x5647224d2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.847+0000 7f0b4019b740 1 --2- 172.21.15.40:0/2526242958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 0x5647224d2310 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x5647224f0ea0 tx=0x7f0b2c0046c0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.847+0000 7f0b4019b740 1 -- 172.21.15.40:0/2526242958 shutdown_connections 2024-03-18T22:50:57.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.847+0000 7f0b4019b740 1 --2- 172.21.15.40:0/2526242958 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224dde50 0x5647224d2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.847+0000 7f0b4019b740 1 --2- 172.21.15.40:0/2526242958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.847+0000 7f0b4019b740 1 --2- 172.21.15.40:0/2526242958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 0x5647224d2310 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.847+0000 7f0b4019b740 1 -- 172.21.15.40:0/2526242958 >> 172.21.15.40:0/2526242958 conn(0x5647224d3860 msgr2=0x5647224e27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.847+0000 7f0b4019b740 1 -- 172.21.15.40:0/2526242958 shutdown_connections 2024-03-18T22:50:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.847+0000 7f0b4019b740 1 -- 172.21.15.40:0/2526242958 wait complete. 2024-03-18T22:50:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b4019b740 1 -- start start 2024-03-18T22:50:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b4019b740 1 Processor -- start 2024-03-18T22:50:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b4019b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 0x5647224e7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b4019b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b4019b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224dde50 0x5647224e8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b4019b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0b381adba0 con 0x5647224d8ce0 2024-03-18T22:50:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b4019b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0b381ad410 con 0x5647224d82f0 2024-03-18T22:50:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b4019b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0b381aa650 con 0x5647224dde50 2024-03-18T22:50:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b4019b740 10 client.0 _ll_register_callbacks cb 0x7f0b2c011610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b36ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b36ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49912/0 (socket says 172.21.15.40:49912) 2024-03-18T22:50:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b36ffd640 1 -- 172.21.15.40:0/2451117435 learned_addr learned my addr 172.21.15.40:0/2451117435 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b367fc640 1 --2- 172.21.15.40:0/2451117435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 0x5647224e7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b367fc640 1 --2- 172.21.15.40:0/2451117435 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224dde50 0x5647224e8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b36ffd640 1 -- 172.21.15.40:0/2451117435 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224dde50 msgr2=0x5647224e8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b36ffd640 1 --2- 172.21.15.40:0/2451117435 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224dde50 0x5647224e8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b36ffd640 1 -- 172.21.15.40:0/2451117435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 msgr2=0x5647224e7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b36ffd640 1 --2- 172.21.15.40:0/2451117435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 0x5647224e7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.848+0000 7f0b36ffd640 1 -- 172.21.15.40:0/2451117435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647224e8e50 con 0x5647224d8ce0 2024-03-18T22:50:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b36ffd640 1 --2- 172.21.15.40:0/2451117435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e7850 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f0b28005ab0 tx=0x7f0b2804c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf83fc09e628f22 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b1aff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b1aff5640 1 -- 172.21.15.40:0/2451117435 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b28050070 con 0x5647224d8ce0 2024-03-18T22:50:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b1aff5640 1 -- 172.21.15.40:0/2451117435 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0b2804ce00 con 0x5647224d8ce0 2024-03-18T22:50:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b4019b740 1 -- 172.21.15.40:0/2451117435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b2c0044b0 con 0x5647224d8ce0 2024-03-18T22:50:57.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b1aff5640 1 -- 172.21.15.40:0/2451117435 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0b28065050 con 0x5647224d8ce0 2024-03-18T22:50:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b4019b740 10 client.17700 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b4019b740 1 -- 172.21.15.40:0/2451117435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x564722554590 con 0x5647224d8ce0 2024-03-18T22:50:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b1aff5640 1 -- 172.21.15.40:0/2451117435 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b2c0044b0 con 0x5647224d8ce0 2024-03-18T22:50:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b1aff5640 1 -- 172.21.15.40:0/2451117435 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f0b280051d0 con 0x5647224d8ce0 2024-03-18T22:50:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b4019b740 10 client.17700 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b4019b740 10 client.17700 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:50:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b4019b740 20 client.17700 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b4019b740 1 --2- 172.21.15.40:0/2451117435 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x564722554f30 0x564722557350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b4019b740 4 client.17700 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:50:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b4019b740 1 -- 172.21.15.40:0/2451117435 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5647224499f0 con 0x564722554f30 2024-03-18T22:50:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.849+0000 7f0b367fc640 1 --2- 172.21.15.40:0/2451117435 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x564722554f30 0x564722557350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b367fc640 1 --2- 172.21.15.40:0/2451117435 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x564722554f30 0x564722557350 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9855de9e79bfdd1e server_cookie=dc6d0ad304128c95 in_seq=0 out_seq=0 2024-03-18T22:50:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b1aff5640 10 client.17700 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:50:57.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b1aff5640 1 -- 172.21.15.40:0/2451117435 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5647224499f0 con 0x564722554f30 2024-03-18T22:50:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b1aff5640 10 client.17700 handle_command_reply: tid=0 2024-03-18T22:50:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b4019b740 1 client.17700 shutdown 2024-03-18T22:50:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b4019b740 1 -- 172.21.15.40:0/2451117435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 msgr2=0x5647224e7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b4019b740 1 --2- 172.21.15.40:0/2451117435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e7850 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f0b28005ab0 tx=0x7f0b2804c850 comp rx=0 tx=0).stop 2024-03-18T22:50:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b4019b740 1 -- 172.21.15.40:0/2451117435 shutdown_connections 2024-03-18T22:50:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b4019b740 1 --2- 172.21.15.40:0/2451117435 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x564722554f30 0x564722557350 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b4019b740 1 --2- 172.21.15.40:0/2451117435 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5647224dde50 0x5647224e8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b4019b740 1 --2- 172.21.15.40:0/2451117435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5647224d8ce0 0x5647224e7850 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b1aff5640 0 client.17700 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:50:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b4019b740 1 --2- 172.21.15.40:0/2451117435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5647224d82f0 0x5647224e7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b4019b740 1 -- 172.21.15.40:0/2451117435 >> 172.21.15.40:0/2451117435 conn(0x5647224d3860 msgr2=0x5647224d6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b4019b740 1 -- 172.21.15.40:0/2451117435 shutdown_connections 2024-03-18T22:50:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b4019b740 1 -- 172.21.15.40:0/2451117435 wait complete. 2024-03-18T22:50:57.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.850+0000 7f0b4019b740 20 client.17700 trim_cache size 0 max 16384 2024-03-18T22:50:57.859 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:50:57.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.851+0000 7f0b3d5bc640 1 -- 172.21.15.40:0/2347726229 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0b200a2800 msgr2=0x7f0b200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.851+0000 7f0b3d5bc640 1 --2- 172.21.15.40:0/2347726229 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0b200a2800 0x7f0b200a4cc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0b3009ba80 tx=0x7f0b30013700 comp rx=0 tx=0).stop 2024-03-18T22:50:57.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.851+0000 7f0b3d5bc640 1 -- 172.21.15.40:0/2347726229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b381022d0 msgr2=0x7f0b38103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.851+0000 7f0b3d5bc640 1 --2- 172.21.15.40:0/2347726229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b381022d0 0x7f0b38103230 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009fd0 tx=0x7f0b3000ef10 comp rx=0 tx=0).stop 2024-03-18T22:50:57.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.851+0000 7f0b3d5bc640 1 -- 172.21.15.40:0/2347726229 shutdown_connections 2024-03-18T22:50:57.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.851+0000 7f0b3d5bc640 1 --2- 172.21.15.40:0/2347726229 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0b200a2800 0x7f0b200a4cc0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.851+0000 7f0b3d5bc640 1 --2- 172.21.15.40:0/2347726229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b381022d0 0x7f0b38103230 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.851+0000 7f0b3d5bc640 1 --2- 172.21.15.40:0/2347726229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0b38101910 0x7f0b38102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.851+0000 7f0b3d5bc640 1 --2- 172.21.15.40:0/2347726229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0b38100720 0x7f0b381046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.851+0000 7f0b3d5bc640 1 -- 172.21.15.40:0/2347726229 >> 172.21.15.40:0/2347726229 conn(0x7f0b380fbc90 msgr2=0x7f0b380fc330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.851+0000 7f0b3d5bc640 1 -- 172.21.15.40:0/2347726229 shutdown_connections 2024-03-18T22:50:57.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.851+0000 7f0b3d5bc640 1 -- 172.21.15.40:0/2347726229 wait complete. 2024-03-18T22:50:57.866 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:50:57.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.982+0000 7fade2675640 1 -- 172.21.15.40:0/3699570261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc1022d0 msgr2=0x7faddc108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.982+0000 7fade2675640 1 --2- 172.21.15.40:0/3699570261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc1022d0 0x7faddc108b60 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009ca0 tx=0x7fadd402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.982+0000 7fade2675640 1 -- 172.21.15.40:0/3699570261 shutdown_connections 2024-03-18T22:50:57.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.982+0000 7fade2675640 1 --2- 172.21.15.40:0/3699570261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc1022d0 0x7faddc108b60 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.982+0000 7fade2675640 1 --2- 172.21.15.40:0/3699570261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc101910 0x7faddc101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.982+0000 7fade2675640 1 --2- 172.21.15.40:0/3699570261 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faddc100720 0x7faddc100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.982+0000 7fade2675640 1 -- 172.21.15.40:0/3699570261 >> 172.21.15.40:0/3699570261 conn(0x7faddc0fbc90 msgr2=0x7faddc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.982+0000 7fade2675640 1 -- 172.21.15.40:0/3699570261 shutdown_connections 2024-03-18T22:50:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.982+0000 7fade2675640 1 -- 172.21.15.40:0/3699570261 wait complete. 2024-03-18T22:50:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.983+0000 7fade2675640 1 Processor -- start 2024-03-18T22:50:57.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.983+0000 7fade2675640 1 -- start start 2024-03-18T22:50:57.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.983+0000 7fade2675640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc100720 0x7faddc1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.983+0000 7fade2675640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faddc101910 0x7faddc197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.983+0000 7fade2675640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc1022d0 0x7faddc19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.983+0000 7fade2675640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faddc10c230 con 0x7faddc100720 2024-03-18T22:50:57.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.983+0000 7fade2675640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faddc10c0b0 con 0x7faddc1022d0 2024-03-18T22:50:57.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.983+0000 7fade2675640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7faddc10c3b0 con 0x7faddc101910 2024-03-18T22:50:57.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.983+0000 7faddbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc100720 0x7faddc1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.983+0000 7faddbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc100720 0x7faddc1979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49922/0 (socket says 172.21.15.40:49922) 2024-03-18T22:50:57.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.983+0000 7faddbfff640 1 -- 172.21.15.40:0/3247932087 learned_addr learned my addr 172.21.15.40:0/3247932087 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.983+0000 7faddb7fe640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faddc101910 0x7faddc197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.983+0000 7fade0c12640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc1022d0 0x7faddc19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.984+0000 7faddbfff640 1 -- 172.21.15.40:0/3247932087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faddc101910 msgr2=0x7faddc197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.984+0000 7faddbfff640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faddc101910 0x7faddc197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.984+0000 7faddbfff640 1 -- 172.21.15.40:0/3247932087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc1022d0 msgr2=0x7faddc19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.984+0000 7faddbfff640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc1022d0 0x7faddc19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.984+0000 7faddbfff640 1 -- 172.21.15.40:0/3247932087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faddc19f610 con 0x7faddc100720 2024-03-18T22:50:57.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.984+0000 7fade0c12640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc1022d0 0x7faddc19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.984+0000 7faddb7fe640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faddc101910 0x7faddc197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.984+0000 7faddbfff640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc100720 0x7faddc1979f0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00e9c0 tx=0x7fadcc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.984+0000 7fadd97fa640 1 -- 172.21.15.40:0/3247932087 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc00cd80 con 0x7faddc100720 2024-03-18T22:50:57.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.984+0000 7fadd97fa640 1 -- 172.21.15.40:0/3247932087 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadcc0035f0 con 0x7faddc100720 2024-03-18T22:50:57.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.984+0000 7fade2675640 1 -- 172.21.15.40:0/3247932087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faddc19f900 con 0x7faddc100720 2024-03-18T22:50:57.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.984+0000 7fadd97fa640 1 -- 172.21.15.40:0/3247932087 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc010690 con 0x7faddc100720 2024-03-18T22:50:57.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.984+0000 7fade2675640 1 -- 172.21.15.40:0/3247932087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faddc19fe50 con 0x7faddc100720 2024-03-18T22:50:57.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.986+0000 7fadd97fa640 1 -- 172.21.15.40:0/3247932087 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fadcc0037a0 con 0x7faddc100720 2024-03-18T22:50:57.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.987+0000 7fadd97fa640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadc40a2800 0x7fadc40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.987+0000 7faddb7fe640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadc40a2800 0x7fadc40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.987+0000 7fadd97fa640 1 -- 172.21.15.40:0/3247932087 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fadcc0cf1d0 con 0x7faddc100720 2024-03-18T22:50:57.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.987+0000 7fade525a740 1 Processor -- start 2024-03-18T22:50:57.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.987+0000 7fade525a740 1 -- start start 2024-03-18T22:50:57.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.987+0000 7fade525a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a00f0 0x55a16a8a0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.987+0000 7fade525a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 0x55a16a8a5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.987+0000 7fade525a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a56e0 0x55a16a8a7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.987+0000 7fade525a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a16a7fb4b0 con 0x55a16a8a56e0 2024-03-18T22:50:57.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.987+0000 7fade525a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a16a7a3d70 con 0x55a16a8a0ae0 2024-03-18T22:50:57.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.987+0000 7fade525a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55a16a7f7550 con 0x55a16a8a00f0 2024-03-18T22:50:57.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.987+0000 7faddb7fe640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadc40a2800 0x7fadc40a4cc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7faddc198f00 tx=0x7fadd0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7faddb7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 0x55a16a8a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7faddbfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a00f0 0x55a16a8a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7fade0c12640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a56e0 0x55a16a8a7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7faddbfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a00f0 0x55a16a8a0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:34328/0 (socket says 172.21.15.40:34328) 2024-03-18T22:50:57.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7faddbfff640 1 -- 172.21.15.40:0/1987218731 learned_addr learned my addr 172.21.15.40:0/1987218731 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7faddb7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 0x55a16a8a5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47168/0 (socket says 172.21.15.40:47168) 2024-03-18T22:50:57.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7faddbfff640 1 -- 172.21.15.40:0/1987218731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 msgr2=0x55a16a8a5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7faddbfff640 1 --2- 172.21.15.40:0/1987218731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 0x55a16a8a5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7faddbfff640 1 -- 172.21.15.40:0/1987218731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a56e0 msgr2=0x55a16a8a7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7faddbfff640 1 --2- 172.21.15.40:0/1987218731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a56e0 0x55a16a8a7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7faddbfff640 1 -- 172.21.15.40:0/1987218731 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a16a7c6210 con 0x55a16a8a00f0 2024-03-18T22:50:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7fade0c12640 1 --2- 172.21.15.40:0/1987218731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a56e0 0x55a16a8a7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7faddb7fe640 1 --2- 172.21.15.40:0/1987218731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 0x55a16a8a5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7faddbfff640 1 --2- 172.21.15.40:0/1987218731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a00f0 0x55a16a8a0510 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0a39f0 tx=0x7fadcc09b380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d9458fbd66d099d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7fadc27fc640 1 -- 172.21.15.40:0/1987218731 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc0b4070 con 0x55a16a8a00f0 2024-03-18T22:50:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7fadc27fc640 1 -- 172.21.15.40:0/1987218731 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadcc09b780 con 0x55a16a8a00f0 2024-03-18T22:50:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.988+0000 7fadc27fc640 1 -- 172.21.15.40:0/1987218731 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc097df0 con 0x55a16a8a00f0 2024-03-18T22:50:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.989+0000 7fade525a740 1 -- 172.21.15.40:0/1987218731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a00f0 msgr2=0x55a16a8a0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.989+0000 7fade525a740 1 --2- 172.21.15.40:0/1987218731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a00f0 0x55a16a8a0510 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0a39f0 tx=0x7fadcc09b380 comp rx=0 tx=0).stop 2024-03-18T22:50:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.989+0000 7fade525a740 1 -- 172.21.15.40:0/1987218731 shutdown_connections 2024-03-18T22:50:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.989+0000 7fade525a740 1 --2- 172.21.15.40:0/1987218731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a56e0 0x55a16a8a7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.989+0000 7fade525a740 1 --2- 172.21.15.40:0/1987218731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 0x55a16a8a5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.989+0000 7fade525a740 1 --2- 172.21.15.40:0/1987218731 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a00f0 0x55a16a8a0510 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.989+0000 7fade525a740 1 -- 172.21.15.40:0/1987218731 >> 172.21.15.40:0/1987218731 conn(0x55a16a892800 msgr2=0x55a16a892c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:57.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.989+0000 7fade525a740 1 -- 172.21.15.40:0/1987218731 shutdown_connections 2024-03-18T22:50:57.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.989+0000 7fade525a740 1 -- 172.21.15.40:0/1987218731 wait complete. 2024-03-18T22:50:57.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.989+0000 7fade525a740 1 -- start start 2024-03-18T22:50:57.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7fade525a740 1 Processor -- start 2024-03-18T22:50:57.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7fade525a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a8aa090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7fade525a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 0x55a16a8aa5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7fade525a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a56e0 0x55a16a8b1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:57.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7fade525a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a16a8a8670 con 0x55a16a8a00f0 2024-03-18T22:50:57.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7fade525a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a16a7b7310 con 0x55a16a8a0ae0 2024-03-18T22:50:57.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7fade525a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55a16a8a87f0 con 0x55a16a8a56e0 2024-03-18T22:50:57.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7fade525a740 10 client.0 _ll_register_callbacks cb 0x55a16a7ab2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:57.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7faddbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a8aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7fade0c12640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a56e0 0x55a16a8b1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7faddbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a8aa090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49944/0 (socket says 172.21.15.40:49944) 2024-03-18T22:50:57.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7faddbfff640 1 -- 172.21.15.40:0/4194479764 learned_addr learned my addr 172.21.15.40:0/4194479764 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:57.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7faddb7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 0x55a16a8aa5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:57.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7fade0c12640 1 -- 172.21.15.40:0/4194479764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 msgr2=0x55a16a8aa5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7fade0c12640 1 --2- 172.21.15.40:0/4194479764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 0x55a16a8aa5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7fade0c12640 1 -- 172.21.15.40:0/4194479764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 msgr2=0x55a16a8aa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:57.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7fade0c12640 1 --2- 172.21.15.40:0/4194479764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a8aa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:57.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7fade0c12640 1 -- 172.21.15.40:0/4194479764 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a16a50f3f0 con 0x55a16a8a56e0 2024-03-18T22:50:57.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.990+0000 7faddbfff640 1 --2- 172.21.15.40:0/4194479764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a8aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:57.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fade0c12640 1 --2- 172.21.15.40:0/4194479764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a56e0 0x55a16a8b1610 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fadd4037040 tx=0x7fadd4009960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53cbf68998214bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:57.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fadbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:50:57.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fadbb7fe640 1 -- 172.21.15.40:0/4194479764 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd4044070 con 0x55a16a8a56e0 2024-03-18T22:50:57.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fade525a740 1 -- 172.21.15.40:0/4194479764 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadcc09b780 con 0x55a16a8a56e0 2024-03-18T22:50:57.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fadbb7fe640 1 -- 172.21.15.40:0/4194479764 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fadd4003b70 con 0x55a16a8a56e0 2024-03-18T22:50:57.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fadbb7fe640 1 -- 172.21.15.40:0/4194479764 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fadd404f050 con 0x55a16a8a56e0 2024-03-18T22:50:57.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fade525a740 10 client.27517 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:58.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fade525a740 1 -- 172.21.15.40:0/4194479764 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a16a5d97d0 con 0x55a16a8a56e0 2024-03-18T22:50:58.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fadbb7fe640 1 -- 172.21.15.40:0/4194479764 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadcc09b780 con 0x55a16a8a56e0 2024-03-18T22:50:58.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fadbb7fe640 1 -- 172.21.15.40:0/4194479764 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fadd4040460 con 0x55a16a8a56e0 2024-03-18T22:50:58.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fade525a740 10 client.27517 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:58.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fade525a740 10 client.27517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:58.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fade525a740 20 client.27517 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:58.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fade525a740 1 --2- 172.21.15.40:0/4194479764 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a16a8b5e90 0x55a16a8b82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:58.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fade525a740 4 client.27517 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:58.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.991+0000 7fade525a740 1 -- 172.21.15.40:0/4194479764 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a16a8593e0 con 0x55a16a8b5e90 2024-03-18T22:50:58.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.992+0000 7faddbfff640 1 --2- 172.21.15.40:0/4194479764 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a16a8b5e90 0x55a16a8b82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:58.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.992+0000 7fadbb7fe640 10 client.27517 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:58.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.992+0000 7faddbfff640 1 --2- 172.21.15.40:0/4194479764 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a16a8b5e90 0x55a16a8b82c0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96ac2301a3ae4907 server_cookie=77383a97bc3e7a79 in_seq=0 out_seq=0 2024-03-18T22:50:58.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.993+0000 7fadbb7fe640 1 -- 172.21.15.40:0/4194479764 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a16a8593e0 con 0x55a16a8b5e90 2024-03-18T22:50:58.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.993+0000 7fadbb7fe640 10 client.27517 handle_command_reply: tid=0 2024-03-18T22:50:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.993+0000 7fade525a740 1 client.27517 shutdown 2024-03-18T22:50:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.993+0000 7fade525a740 1 -- 172.21.15.40:0/4194479764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a56e0 msgr2=0x55a16a8b1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.993+0000 7fade525a740 1 --2- 172.21.15.40:0/4194479764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a56e0 0x55a16a8b1610 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fadd4037040 tx=0x7fadd4009960 comp rx=0 tx=0).stop 2024-03-18T22:50:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.993+0000 7fade525a740 1 -- 172.21.15.40:0/4194479764 shutdown_connections 2024-03-18T22:50:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.993+0000 7fade525a740 1 --2- 172.21.15.40:0/4194479764 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a56e0 0x55a16a8b1610 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.993+0000 7fade525a740 1 --2- 172.21.15.40:0/4194479764 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a16a8b5e90 0x55a16a8b82c0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.993+0000 7fade525a740 1 --2- 172.21.15.40:0/4194479764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 0x55a16a8aa5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.993+0000 7fade525a740 1 --2- 172.21.15.40:0/4194479764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a8aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.993+0000 7fadbb7fe640 0 client.27517 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.993+0000 7fade525a740 1 -- 172.21.15.40:0/4194479764 >> 172.21.15.40:0/4194479764 conn(0x55a16a892800 msgr2=0x55a16a891ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.993+0000 7fade525a740 1 -- 172.21.15.40:0/4194479764 shutdown_connections 2024-03-18T22:50:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.993+0000 7fade525a740 1 -- 172.21.15.40:0/4194479764 wait complete. 2024-03-18T22:50:58.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:57.993+0000 7fade525a740 20 client.27517 trim_cache size 0 max 16384 2024-03-18T22:50:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.011+0000 7fade525a740 1 Processor -- start 2024-03-18T22:50:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.011+0000 7fade525a740 1 -- start start 2024-03-18T22:50:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.011+0000 7fade525a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a88da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.011+0000 7fade525a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a0ae0 0x55a16a8b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.011+0000 7fade525a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a56e0 0x55a16a89eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.011+0000 7fade525a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faddc1a8330 con 0x55a16a8a00f0 2024-03-18T22:50:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.011+0000 7fade525a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faddc1a9f30 con 0x55a16a8a56e0 2024-03-18T22:50:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.011+0000 7fade525a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7faddc1a8a00 con 0x55a16a8a0ae0 2024-03-18T22:50:58.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.011+0000 7faddbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a88da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7faddb7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a0ae0 0x55a16a8b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7fade0c12640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a56e0 0x55a16a89eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7faddb7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a0ae0 0x55a16a8b15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:34340/0 (socket says 172.21.15.40:34340) 2024-03-18T22:50:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7fade0c12640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a56e0 0x55a16a89eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47180/0 (socket says 172.21.15.40:47180) 2024-03-18T22:50:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7faddb7fe640 1 -- 172.21.15.40:0/2590098043 learned_addr learned my addr 172.21.15.40:0/2590098043 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7faddbfff640 1 -- 172.21.15.40:0/2590098043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a0ae0 msgr2=0x55a16a8b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7faddbfff640 1 --2- 172.21.15.40:0/2590098043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a0ae0 0x55a16a8b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7faddbfff640 1 -- 172.21.15.40:0/2590098043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a56e0 msgr2=0x55a16a89eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7faddbfff640 1 --2- 172.21.15.40:0/2590098043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a56e0 0x55a16a89eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7faddbfff640 1 -- 172.21.15.40:0/2590098043 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a16a868880 con 0x55a16a8a00f0 2024-03-18T22:50:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7faddb7fe640 1 --2- 172.21.15.40:0/2590098043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a0ae0 0x55a16a8b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7fade0c12640 1 --2- 172.21.15.40:0/2590098043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a56e0 0x55a16a89eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7faddbfff640 1 --2- 172.21.15.40:0/2590098043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a88da30 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0ccc50 tx=0x7fadcc09b3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb695725340e80d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7fadc1ffb640 1 -- 172.21.15.40:0/2590098043 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc0b4040 con 0x55a16a8a00f0 2024-03-18T22:50:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7fadc1ffb640 1 -- 172.21.15.40:0/2590098043 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadcc09b780 con 0x55a16a8a00f0 2024-03-18T22:50:58.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.012+0000 7fadc1ffb640 1 -- 172.21.15.40:0/2590098043 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc0a3a90 con 0x55a16a8a00f0 2024-03-18T22:50:58.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.013+0000 7fade525a740 1 -- 172.21.15.40:0/2590098043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 msgr2=0x55a16a88da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:58.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.013+0000 7fade525a740 1 --2- 172.21.15.40:0/2590098043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a88da30 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0ccc50 tx=0x7fadcc09b3f0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.013+0000 7fade525a740 1 -- 172.21.15.40:0/2590098043 shutdown_connections 2024-03-18T22:50:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.013+0000 7fade525a740 1 --2- 172.21.15.40:0/2590098043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a56e0 0x55a16a89eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.013+0000 7fade525a740 1 --2- 172.21.15.40:0/2590098043 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a0ae0 0x55a16a8b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.013+0000 7fade525a740 1 --2- 172.21.15.40:0/2590098043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a88da30 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.013+0000 7fade525a740 1 -- 172.21.15.40:0/2590098043 >> 172.21.15.40:0/2590098043 conn(0x55a16a892800 msgr2=0x55a16a8a8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.013+0000 7fade525a740 1 -- 172.21.15.40:0/2590098043 shutdown_connections 2024-03-18T22:50:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.013+0000 7fade525a740 1 -- 172.21.15.40:0/2590098043 wait complete. 2024-03-18T22:50:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.013+0000 7fade525a740 1 -- start start 2024-03-18T22:50:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7fade525a740 1 Processor -- start 2024-03-18T22:50:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7fade525a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a8aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7fade525a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 0x55a16a8af0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7fade525a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a56e0 0x55a16a8b0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7fade525a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7faddc1a9860 con 0x55a16a8a00f0 2024-03-18T22:50:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7fade525a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faddc1a90d0 con 0x55a16a8a0ae0 2024-03-18T22:50:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7fade525a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7faddc1a6310 con 0x55a16a8a56e0 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7fade525a740 10 client.0 _ll_register_callbacks cb 0x7fadcc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7faddbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a8aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7faddbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a8aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49956/0 (socket says 172.21.15.40:49956) 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7faddbfff640 1 -- 172.21.15.40:0/1689058987 learned_addr learned my addr 172.21.15.40:0/1689058987 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7fade0c12640 1 --2- 172.21.15.40:0/1689058987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a56e0 0x55a16a8b0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7faddb7fe640 1 --2- 172.21.15.40:0/1689058987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 0x55a16a8af0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7faddbfff640 1 -- 172.21.15.40:0/1689058987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a56e0 msgr2=0x55a16a8b0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7faddbfff640 1 --2- 172.21.15.40:0/1689058987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a56e0 0x55a16a8b0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7faddbfff640 1 -- 172.21.15.40:0/1689058987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 msgr2=0x55a16a8af0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7faddbfff640 1 --2- 172.21.15.40:0/1689058987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 0x55a16a8af0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7faddbfff640 1 -- 172.21.15.40:0/1689058987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a16a8a88f0 con 0x55a16a8a00f0 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7faddbfff640 1 --2- 172.21.15.40:0/1689058987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a8aeba0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fadcc098100 tx=0x7fadcc003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253db6935844a8fd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.014+0000 7fadbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.015+0000 7fadbb7fe640 1 -- 172.21.15.40:0/1689058987 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc0b8070 con 0x55a16a8a00f0 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.015+0000 7fadbb7fe640 1 -- 172.21.15.40:0/1689058987 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fadcc0a3a90 con 0x55a16a8a00f0 2024-03-18T22:50:58.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.015+0000 7fade525a740 1 -- 172.21.15.40:0/1689058987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadcc09b780 con 0x55a16a8a00f0 2024-03-18T22:50:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.015+0000 7fadbb7fe640 1 -- 172.21.15.40:0/1689058987 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fadcc0a5050 con 0x55a16a8a00f0 2024-03-18T22:50:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.015+0000 7fade525a740 10 client.17718 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.015+0000 7fade525a740 1 -- 172.21.15.40:0/1689058987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a16a8b1770 con 0x55a16a8a00f0 2024-03-18T22:50:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.015+0000 7fadbb7fe640 1 -- 172.21.15.40:0/1689058987 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fadcc09b780 con 0x55a16a8a00f0 2024-03-18T22:50:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.015+0000 7fadbb7fe640 1 -- 172.21.15.40:0/1689058987 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fadcc097040 con 0x55a16a8a00f0 2024-03-18T22:50:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.015+0000 7fade525a740 10 client.17718 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.015+0000 7fade525a740 10 client.17718 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.015+0000 7fade525a740 20 client.17718 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.015+0000 7fade525a740 1 --2- 172.21.15.40:0/1689058987 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a16a91c340 0x55a16a91e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.015+0000 7fade525a740 4 client.17718 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:50:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.015+0000 7fade525a740 1 -- 172.21.15.40:0/1689058987 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a16a8089f0 con 0x55a16a91c340 2024-03-18T22:50:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.015+0000 7faddb7fe640 1 --2- 172.21.15.40:0/1689058987 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a16a91c340 0x55a16a91e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7faddb7fe640 1 --2- 172.21.15.40:0/1689058987 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a16a91c340 0x55a16a91e730 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8cbf055e6f2d951 server_cookie=c4903a461f860459 in_seq=0 out_seq=0 2024-03-18T22:50:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fadbb7fe640 10 client.17718 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fadbb7fe640 1 -- 172.21.15.40:0/1689058987 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a16a8089f0 con 0x55a16a91c340 2024-03-18T22:50:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fadbb7fe640 10 client.17718 handle_command_reply: tid=0 2024-03-18T22:50:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fade525a740 1 client.17718 shutdown 2024-03-18T22:50:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fade525a740 1 -- 172.21.15.40:0/1689058987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 msgr2=0x55a16a8aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fade525a740 1 --2- 172.21.15.40:0/1689058987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a8aeba0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fadcc098100 tx=0x7fadcc003d60 comp rx=0 tx=0).stop 2024-03-18T22:50:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fade525a740 1 -- 172.21.15.40:0/1689058987 shutdown_connections 2024-03-18T22:50:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fade525a740 1 --2- 172.21.15.40:0/1689058987 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a16a8a56e0 0x55a16a8b0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fade525a740 1 --2- 172.21.15.40:0/1689058987 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a16a91c340 0x55a16a91e730 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fade525a740 1 --2- 172.21.15.40:0/1689058987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a16a8a0ae0 0x55a16a8af0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fade525a740 1 --2- 172.21.15.40:0/1689058987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a16a8a00f0 0x55a16a8aeba0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fadbb7fe640 0 client.17718 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fade525a740 1 -- 172.21.15.40:0/1689058987 >> 172.21.15.40:0/1689058987 conn(0x55a16a892800 msgr2=0x55a16a88da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fade525a740 1 -- 172.21.15.40:0/1689058987 shutdown_connections 2024-03-18T22:50:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fade525a740 1 -- 172.21.15.40:0/1689058987 wait complete. 2024-03-18T22:50:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.016+0000 7fade525a740 20 client.17718 trim_cache size 0 max 16384 2024-03-18T22:50:58.024 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:50:58.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.017+0000 7fade2675640 1 -- 172.21.15.40:0/3247932087 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadc40a2800 msgr2=0x7fadc40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:58.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.017+0000 7fade2675640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadc40a2800 0x7fadc40a4cc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7faddc198f00 tx=0x7fadd0005f90 comp rx=0 tx=0).stop 2024-03-18T22:50:58.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.017+0000 7fade2675640 1 -- 172.21.15.40:0/3247932087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc100720 msgr2=0x7faddc1979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:58.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.017+0000 7fade2675640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc100720 0x7faddc1979f0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00e9c0 tx=0x7fadcc00ee80 comp rx=0 tx=0).stop 2024-03-18T22:50:58.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.017+0000 7fade2675640 1 -- 172.21.15.40:0/3247932087 shutdown_connections 2024-03-18T22:50:58.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.017+0000 7fade2675640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadc40a2800 0x7fadc40a4cc0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.017+0000 7fade2675640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faddc1022d0 0x7faddc19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.017+0000 7fade2675640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7faddc101910 0x7faddc197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.017+0000 7fade2675640 1 --2- 172.21.15.40:0/3247932087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faddc100720 0x7faddc1979f0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:58.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.017+0000 7fade2675640 1 -- 172.21.15.40:0/3247932087 >> 172.21.15.40:0/3247932087 conn(0x7faddc0fbc90 msgr2=0x7faddc0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:58.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.017+0000 7fade2675640 1 -- 172.21.15.40:0/3247932087 shutdown_connections 2024-03-18T22:50:58.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:58.017+0000 7fade2675640 1 -- 172.21.15.40:0/3247932087 wait complete. 2024-03-18T22:50:58.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:58 smithi040 ceph-mon[62643]: pgmap v600: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:50:58.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:50:58 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/121060007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:50:58.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:58 smithi103 ceph-mon[66377]: pgmap v600: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:50:58.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:50:58 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/121060007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:50:58.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:58 smithi113 ceph-mon[66775]: pgmap v600: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:50:58.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:50:58 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/121060007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:50:59.030 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:50:59.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:50:59.030 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:50:59.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.158+0000 7fbf20612640 1 -- 172.21.15.40:0/3444008120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf18100720 msgr2=0x7fbf18100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.158+0000 7fbf20612640 1 --2- 172.21.15.40:0/3444008120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf18100720 0x7fbf18100b40 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fbf14009900 tx=0x7fbf1402bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:59.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.158+0000 7fbf20612640 1 -- 172.21.15.40:0/3444008120 shutdown_connections 2024-03-18T22:50:59.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.158+0000 7fbf20612640 1 --2- 172.21.15.40:0/3444008120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf181022d0 0x7fbf18108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.158+0000 7fbf20612640 1 --2- 172.21.15.40:0/3444008120 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf18101910 0x7fbf18101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.158+0000 7fbf20612640 1 --2- 172.21.15.40:0/3444008120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf18100720 0x7fbf18100b40 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.158+0000 7fbf20612640 1 -- 172.21.15.40:0/3444008120 >> 172.21.15.40:0/3444008120 conn(0x7fbf180fbcb0 msgr2=0x7fbf180fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:59.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.158+0000 7fbf20612640 1 -- 172.21.15.40:0/3444008120 shutdown_connections 2024-03-18T22:50:59.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.158+0000 7fbf20612640 1 -- 172.21.15.40:0/3444008120 wait complete. 2024-03-18T22:50:59.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf20612640 1 Processor -- start 2024-03-18T22:50:59.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf20612640 1 -- start start 2024-03-18T22:50:59.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf20612640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf18100720 0x7fbf18197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf20612640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf18101910 0x7fbf181980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf20612640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf181022d0 0x7fbf1819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf20612640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1810c240 con 0x7fbf18101910 2024-03-18T22:50:59.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf20612640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1810c0c0 con 0x7fbf18100720 2024-03-18T22:50:59.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf20612640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1810c3c0 con 0x7fbf181022d0 2024-03-18T22:50:59.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf1dbad640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf18101910 0x7fbf181980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf1dbad640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf18101910 0x7fbf181980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47796/0 (socket says 172.21.15.40:47796) 2024-03-18T22:50:59.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf1dbad640 1 -- 172.21.15.40:0/1907168834 learned_addr learned my addr 172.21.15.40:0/1907168834 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:59.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf1e3ae640 1 --2- 172.21.15.40:0/1907168834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf18100720 0x7fbf18197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf1ebaf640 1 --2- 172.21.15.40:0/1907168834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf181022d0 0x7fbf1819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf1dbad640 1 -- 172.21.15.40:0/1907168834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf181022d0 msgr2=0x7fbf1819f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf1dbad640 1 --2- 172.21.15.40:0/1907168834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf181022d0 0x7fbf1819f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf1dbad640 1 -- 172.21.15.40:0/1907168834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf18100720 msgr2=0x7fbf18197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf1dbad640 1 --2- 172.21.15.40:0/1907168834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf18100720 0x7fbf18197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf1dbad640 1 -- 172.21.15.40:0/1907168834 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf1819f710 con 0x7fbf18101910 2024-03-18T22:50:59.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.159+0000 7fbf1e3ae640 1 --2- 172.21.15.40:0/1907168834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf18100720 0x7fbf18197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:59.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.160+0000 7fbf1dbad640 1 --2- 172.21.15.40:0/1907168834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf18101910 0x7fbf181980d0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fbf0800e9f0 tx=0x7fbf0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:59.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.160+0000 7fbf0f7fe640 1 -- 172.21.15.40:0/1907168834 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0800cd80 con 0x7fbf18101910 2024-03-18T22:50:59.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.160+0000 7fbf0f7fe640 1 -- 172.21.15.40:0/1907168834 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf080054d0 con 0x7fbf18101910 2024-03-18T22:50:59.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.160+0000 7fbf0f7fe640 1 -- 172.21.15.40:0/1907168834 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf08010490 con 0x7fbf18101910 2024-03-18T22:50:59.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.160+0000 7fbf20612640 1 -- 172.21.15.40:0/1907168834 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf1819fa00 con 0x7fbf18101910 2024-03-18T22:50:59.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.160+0000 7fbf20612640 1 -- 172.21.15.40:0/1907168834 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf1819ff50 con 0x7fbf18101910 2024-03-18T22:50:59.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.162+0000 7fbf0f7fe640 1 -- 172.21.15.40:0/1907168834 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbf08005020 con 0x7fbf18101910 2024-03-18T22:50:59.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.163+0000 7fbf23204740 1 Processor -- start 2024-03-18T22:50:59.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.163+0000 7fbf0f7fe640 1 --2- 172.21.15.40:0/1907168834 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbf040a2880 0x7fbf040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.163+0000 7fbf0f7fe640 1 -- 172.21.15.40:0/1907168834 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fbf080cf150 con 0x7fbf18101910 2024-03-18T22:50:59.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.163+0000 7fbf23204740 1 -- start start 2024-03-18T22:50:59.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.163+0000 7fbf1ebaf640 1 --2- 172.21.15.40:0/1907168834 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbf040a2880 0x7fbf040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.163+0000 7fbf23204740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818977c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.163+0000 7fbf23204740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x55978189c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.163+0000 7fbf23204740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55978189cf00 0x55978189f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.163+0000 7fbf23204740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5597817fc160 con 0x5597818973a0 2024-03-18T22:50:59.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.163+0000 7fbf23204740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5597817a4e10 con 0x55978189cf00 2024-03-18T22:50:59.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.163+0000 7fbf23204740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5597817f8200 con 0x559781897d90 2024-03-18T22:50:59.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf1e3ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf1dbad640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x55978189c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf1dbad640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x55978189c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:59762/0 (socket says 172.21.15.40:59762) 2024-03-18T22:50:59.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf1dbad640 1 -- 172.21.15.40:0/4217900259 learned_addr learned my addr 172.21.15.40:0/4217900259 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:59.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf1ebaf640 1 --2- 172.21.15.40:0/4217900259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55978189cf00 0x55978189f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf1ebaf640 1 --2- 172.21.15.40:0/1907168834 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbf040a2880 0x7fbf040a4d40 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fbf10005fd0 tx=0x7fbf10009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:59.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf1dbad640 1 -- 172.21.15.40:0/4217900259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55978189cf00 msgr2=0x55978189f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf1dbad640 1 --2- 172.21.15.40:0/4217900259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55978189cf00 0x55978189f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf1dbad640 1 -- 172.21.15.40:0/4217900259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 msgr2=0x5597818977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf1dbad640 1 --2- 172.21.15.40:0/4217900259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf1dbad640 1 -- 172.21.15.40:0/4217900259 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597815d87e0 con 0x559781897d90 2024-03-18T22:50:59.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf1ebaf640 1 --2- 172.21.15.40:0/4217900259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55978189cf00 0x55978189f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:50:59.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf1dbad640 1 --2- 172.21.15.40:0/4217900259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x55978189c890 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbf0809b690 tx=0x7fbf0809b120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6bd4faf3283a676 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:59.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf0cbfe640 1 -- 172.21.15.40:0/4217900259 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf080b4070 con 0x559781897d90 2024-03-18T22:50:59.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf0cbfe640 1 -- 172.21.15.40:0/4217900259 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf0809be00 con 0x559781897d90 2024-03-18T22:50:59.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.164+0000 7fbf0cbfe640 1 -- 172.21.15.40:0/4217900259 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf080046c0 con 0x559781897d90 2024-03-18T22:50:59.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.165+0000 7fbf23204740 1 -- 172.21.15.40:0/4217900259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 msgr2=0x55978189c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.165+0000 7fbf23204740 1 --2- 172.21.15.40:0/4217900259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x55978189c890 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbf0809b690 tx=0x7fbf0809b120 comp rx=0 tx=0).stop 2024-03-18T22:50:59.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.165+0000 7fbf23204740 1 -- 172.21.15.40:0/4217900259 shutdown_connections 2024-03-18T22:50:59.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.165+0000 7fbf23204740 1 --2- 172.21.15.40:0/4217900259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55978189cf00 0x55978189f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.165+0000 7fbf23204740 1 --2- 172.21.15.40:0/4217900259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x55978189c890 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.165+0000 7fbf23204740 1 --2- 172.21.15.40:0/4217900259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.165+0000 7fbf23204740 1 -- 172.21.15.40:0/4217900259 >> 172.21.15.40:0/4217900259 conn(0x559781892910 msgr2=0x559781892d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:59.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.165+0000 7fbf23204740 1 -- 172.21.15.40:0/4217900259 shutdown_connections 2024-03-18T22:50:59.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.165+0000 7fbf23204740 1 -- 172.21.15.40:0/4217900259 wait complete. 2024-03-18T22:50:59.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf23204740 1 -- start start 2024-03-18T22:50:59.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf23204740 1 Processor -- start 2024-03-18T22:50:59.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf23204740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818a1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf23204740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559781897d90 0x5597818a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf23204740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55978189cf00 0x5597818a8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf23204740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55978189feb0 con 0x5597818973a0 2024-03-18T22:50:59.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf23204740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5597817b81a0 con 0x559781897d90 2024-03-18T22:50:59.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf23204740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5597818a0030 con 0x55978189cf00 2024-03-18T22:50:59.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf23204740 10 client.0 _ll_register_callbacks cb 0x55978150fc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:59.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf1e3ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf1e3ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818a1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47820/0 (socket says 172.21.15.40:47820) 2024-03-18T22:50:59.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf1e3ae640 1 -- 172.21.15.40:0/1834228639 learned_addr learned my addr 172.21.15.40:0/1834228639 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:59.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf1ebaf640 1 --2- 172.21.15.40:0/1834228639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55978189cf00 0x5597818a8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf1dbad640 1 --2- 172.21.15.40:0/1834228639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559781897d90 0x5597818a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf1ebaf640 1 -- 172.21.15.40:0/1834228639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559781897d90 msgr2=0x5597818a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf1ebaf640 1 --2- 172.21.15.40:0/1834228639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559781897d90 0x5597818a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf1ebaf640 1 -- 172.21.15.40:0/1834228639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 msgr2=0x5597818a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf1ebaf640 1 --2- 172.21.15.40:0/1834228639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf1ebaf640 1 -- 172.21.15.40:0/1834228639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559781638f70 con 0x55978189cf00 2024-03-18T22:50:59.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.166+0000 7fbf1e3ae640 1 --2- 172.21.15.40:0/1834228639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:59.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf1ebaf640 1 --2- 172.21.15.40:0/1834228639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55978189cf00 0x5597818a8da0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbf10009d60 tx=0x7fbf10004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33d0ebb16300ae42 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:59.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:50:59.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf01ffb640 1 -- 172.21.15.40:0/1834228639 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1002a060 con 0x55978189cf00 2024-03-18T22:50:59.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf23204740 1 -- 172.21.15.40:0/1834228639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf0809be00 con 0x55978189cf00 2024-03-18T22:50:59.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf01ffb640 1 -- 172.21.15.40:0/1834228639 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fbf1001c710 con 0x55978189cf00 2024-03-18T22:50:59.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf01ffb640 1 -- 172.21.15.40:0/1834228639 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fbf10031050 con 0x55978189cf00 2024-03-18T22:50:59.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf23204740 10 client.27535 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:59.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf23204740 1 -- 172.21.15.40:0/1834228639 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5597814d0170 con 0x55978189cf00 2024-03-18T22:50:59.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf01ffb640 1 -- 172.21.15.40:0/1834228639 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf0809be00 con 0x55978189cf00 2024-03-18T22:50:59.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf01ffb640 1 -- 172.21.15.40:0/1834228639 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fbf1001bb70 con 0x55978189cf00 2024-03-18T22:50:59.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf23204740 10 client.27535 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:59.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf23204740 10 client.27535 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:59.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf23204740 20 client.27535 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:59.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf23204740 1 --2- 172.21.15.40:0/1834228639 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5597818ad620 0x5597818afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf23204740 4 client.27535 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:59.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.167+0000 7fbf23204740 1 -- 172.21.15.40:0/1834228639 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597818096a0 con 0x5597818ad620 2024-03-18T22:50:59.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.168+0000 7fbf1e3ae640 1 --2- 172.21.15.40:0/1834228639 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5597818ad620 0x5597818afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.168+0000 7fbf01ffb640 10 client.27535 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:59.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.168+0000 7fbf1e3ae640 1 --2- 172.21.15.40:0/1834228639 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5597818ad620 0x5597818afaa0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ca4da66636a56a2 server_cookie=67fd2f0f9e463be4 in_seq=0 out_seq=0 2024-03-18T22:50:59.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.169+0000 7fbf01ffb640 1 -- 172.21.15.40:0/1834228639 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5597818096a0 con 0x5597818ad620 2024-03-18T22:50:59.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.169+0000 7fbf01ffb640 10 client.27535 handle_command_reply: tid=0 2024-03-18T22:50:59.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.169+0000 7fbf23204740 1 client.27535 shutdown 2024-03-18T22:50:59.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.169+0000 7fbf23204740 1 -- 172.21.15.40:0/1834228639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55978189cf00 msgr2=0x5597818a8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.169+0000 7fbf23204740 1 --2- 172.21.15.40:0/1834228639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55978189cf00 0x5597818a8da0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbf10009d60 tx=0x7fbf10004ab0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.169+0000 7fbf23204740 1 -- 172.21.15.40:0/1834228639 shutdown_connections 2024-03-18T22:50:59.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.169+0000 7fbf23204740 1 --2- 172.21.15.40:0/1834228639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55978189cf00 0x5597818a8da0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.169+0000 7fbf23204740 1 --2- 172.21.15.40:0/1834228639 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5597818ad620 0x5597818afaa0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.169+0000 7fbf23204740 1 --2- 172.21.15.40:0/1834228639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559781897d90 0x5597818a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.169+0000 7fbf23204740 1 --2- 172.21.15.40:0/1834228639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.169+0000 7fbf01ffb640 0 client.27535 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:59.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.169+0000 7fbf23204740 1 -- 172.21.15.40:0/1834228639 >> 172.21.15.40:0/1834228639 conn(0x559781892910 msgr2=0x559781891b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:59.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.169+0000 7fbf23204740 1 -- 172.21.15.40:0/1834228639 shutdown_connections 2024-03-18T22:50:59.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.169+0000 7fbf23204740 1 -- 172.21.15.40:0/1834228639 wait complete. 2024-03-18T22:50:59.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.169+0000 7fbf23204740 20 client.27535 trim_cache size 0 max 16384 2024-03-18T22:50:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.188+0000 7fbf23204740 1 Processor -- start 2024-03-18T22:50:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.188+0000 7fbf23204740 1 -- start start 2024-03-18T22:50:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.188+0000 7fbf23204740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5597818973a0 0x5597818912a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.188+0000 7fbf23204740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x5597818a8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.188+0000 7fbf23204740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55978189cf00 0x5597818917e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.188+0000 7fbf23204740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbf181aa570 con 0x55978189cf00 2024-03-18T22:50:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.188+0000 7fbf23204740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbf181ac170 con 0x5597818973a0 2024-03-18T22:50:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.188+0000 7fbf23204740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbf181aac40 con 0x559781897d90 2024-03-18T22:50:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.188+0000 7fbf1dbad640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x5597818a8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.188+0000 7fbf1ebaf640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55978189cf00 0x5597818917e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.188+0000 7fbf1dbad640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x5597818a8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:59786/0 (socket says 172.21.15.40:59786) 2024-03-18T22:50:59.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.188+0000 7fbf1dbad640 1 -- 172.21.15.40:0/943376148 learned_addr learned my addr 172.21.15.40:0/943376148 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:59.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.188+0000 7fbf1e3ae640 1 --2- 172.21.15.40:0/943376148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5597818973a0 0x5597818912a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.188+0000 7fbf1dbad640 1 -- 172.21.15.40:0/943376148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5597818973a0 msgr2=0x5597818912a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.189+0000 7fbf1dbad640 1 --2- 172.21.15.40:0/943376148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5597818973a0 0x5597818912a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.189+0000 7fbf1dbad640 1 -- 172.21.15.40:0/943376148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55978189cf00 msgr2=0x5597818917e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.189+0000 7fbf1dbad640 1 --2- 172.21.15.40:0/943376148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55978189cf00 0x5597818917e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.189+0000 7fbf1dbad640 1 -- 172.21.15.40:0/943376148 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55978188e160 con 0x559781897d90 2024-03-18T22:50:59.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.189+0000 7fbf1e3ae640 1 --2- 172.21.15.40:0/943376148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5597818973a0 0x5597818912a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:59.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.189+0000 7fbf1dbad640 1 --2- 172.21.15.40:0/943376148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x5597818a8d60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbf080cf470 tx=0x7fbf080979a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e8d2ab0f1aad1e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:59.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.189+0000 7fbf03fff640 1 -- 172.21.15.40:0/943376148 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf080b4040 con 0x559781897d90 2024-03-18T22:50:59.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.189+0000 7fbf03fff640 1 -- 172.21.15.40:0/943376148 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf0809be00 con 0x559781897d90 2024-03-18T22:50:59.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.189+0000 7fbf03fff640 1 -- 172.21.15.40:0/943376148 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf08086050 con 0x559781897d90 2024-03-18T22:50:59.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.190+0000 7fbf23204740 1 -- 172.21.15.40:0/943376148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 msgr2=0x5597818a8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.190+0000 7fbf23204740 1 --2- 172.21.15.40:0/943376148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x5597818a8d60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbf080cf470 tx=0x7fbf080979a0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.190+0000 7fbf23204740 1 -- 172.21.15.40:0/943376148 shutdown_connections 2024-03-18T22:50:59.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.190+0000 7fbf23204740 1 --2- 172.21.15.40:0/943376148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55978189cf00 0x5597818917e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.190+0000 7fbf23204740 1 --2- 172.21.15.40:0/943376148 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x5597818a8d60 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.190+0000 7fbf23204740 1 --2- 172.21.15.40:0/943376148 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5597818973a0 0x5597818912a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.190+0000 7fbf23204740 1 -- 172.21.15.40:0/943376148 >> 172.21.15.40:0/943376148 conn(0x559781892910 msgr2=0x5597818a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:59.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.190+0000 7fbf23204740 1 -- 172.21.15.40:0/943376148 shutdown_connections 2024-03-18T22:50:59.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.190+0000 7fbf23204740 1 -- 172.21.15.40:0/943376148 wait complete. 2024-03-18T22:50:59.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.190+0000 7fbf23204740 1 -- start start 2024-03-18T22:50:59.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.190+0000 7fbf23204740 1 Processor -- start 2024-03-18T22:50:59.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.190+0000 7fbf23204740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818a6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf23204740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x5597818a6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf23204740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55978189cf00 0x5597818a7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf23204740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbf181abaa0 con 0x5597818973a0 2024-03-18T22:50:59.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf23204740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbf181ab310 con 0x55978189cf00 2024-03-18T22:50:59.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf23204740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbf181a8550 con 0x559781897d90 2024-03-18T22:50:59.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf23204740 10 client.0 _ll_register_callbacks cb 0x7fbf10009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:59.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf1e3ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf1e3ae640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818a6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47846/0 (socket says 172.21.15.40:47846) 2024-03-18T22:50:59.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf1dbad640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x5597818a6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf1e3ae640 1 -- 172.21.15.40:0/470266984 learned_addr learned my addr 172.21.15.40:0/470266984 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:59.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf1ebaf640 1 --2- 172.21.15.40:0/470266984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55978189cf00 0x5597818a7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf1e3ae640 1 -- 172.21.15.40:0/470266984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 msgr2=0x5597818a6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf1e3ae640 1 --2- 172.21.15.40:0/470266984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x5597818a6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf1e3ae640 1 -- 172.21.15.40:0/470266984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55978189cf00 msgr2=0x5597818a7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf1e3ae640 1 --2- 172.21.15.40:0/470266984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55978189cf00 0x5597818a7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf1e3ae640 1 -- 172.21.15.40:0/470266984 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597818a8090 con 0x5597818973a0 2024-03-18T22:50:59.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf1e3ae640 1 --2- 172.21.15.40:0/470266984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818a6550 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fbf14036040 tx=0x7fbf14005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21aea3ae1a3e0025 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:59.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:59.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf01ffb640 1 -- 172.21.15.40:0/470266984 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1404bd80 con 0x5597818973a0 2024-03-18T22:50:59.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf01ffb640 1 -- 172.21.15.40:0/470266984 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fbf14005140 con 0x5597818973a0 2024-03-18T22:50:59.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.191+0000 7fbf23204740 1 -- 172.21.15.40:0/470266984 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf0809be00 con 0x5597818973a0 2024-03-18T22:50:59.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.192+0000 7fbf01ffb640 1 -- 172.21.15.40:0/470266984 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fbf14059050 con 0x5597818973a0 2024-03-18T22:50:59.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.192+0000 7fbf23204740 10 client.17736 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:59.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.192+0000 7fbf23204740 1 -- 172.21.15.40:0/470266984 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559781913ad0 con 0x5597818973a0 2024-03-18T22:50:59.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.192+0000 7fbf01ffb640 1 -- 172.21.15.40:0/470266984 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf0809be00 con 0x5597818973a0 2024-03-18T22:50:59.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.192+0000 7fbf01ffb640 1 -- 172.21.15.40:0/470266984 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7fbf14004880 con 0x5597818973a0 2024-03-18T22:50:59.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.192+0000 7fbf23204740 10 client.17736 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:59.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.192+0000 7fbf23204740 10 client.17736 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:59.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.192+0000 7fbf23204740 20 client.17736 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:59.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.192+0000 7fbf23204740 1 --2- 172.21.15.40:0/470266984 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5597819144d0 0x5597819168f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.192+0000 7fbf23204740 4 client.17736 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:50:59.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.192+0000 7fbf23204740 1 -- 172.21.15.40:0/470266984 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5597818039f0 con 0x5597819144d0 2024-03-18T22:50:59.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.192+0000 7fbf1dbad640 1 --2- 172.21.15.40:0/470266984 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5597819144d0 0x5597819168f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.193+0000 7fbf1dbad640 1 --2- 172.21.15.40:0/470266984 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5597819144d0 0x5597819168f0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c62d25d2637bad5 server_cookie=35ae19f49b9c5889 in_seq=0 out_seq=0 2024-03-18T22:50:59.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.193+0000 7fbf01ffb640 10 client.17736 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:59.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.194+0000 7fbf01ffb640 1 -- 172.21.15.40:0/470266984 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5597818039f0 con 0x5597819144d0 2024-03-18T22:50:59.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.194+0000 7fbf01ffb640 10 client.17736 handle_command_reply: tid=0 2024-03-18T22:50:59.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.194+0000 7fbf23204740 1 client.17736 shutdown 2024-03-18T22:50:59.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.194+0000 7fbf23204740 1 -- 172.21.15.40:0/470266984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 msgr2=0x5597818a6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.194+0000 7fbf23204740 1 --2- 172.21.15.40:0/470266984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818a6550 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fbf14036040 tx=0x7fbf14005be0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.194+0000 7fbf23204740 1 -- 172.21.15.40:0/470266984 shutdown_connections 2024-03-18T22:50:59.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.194+0000 7fbf23204740 1 --2- 172.21.15.40:0/470266984 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5597819144d0 0x5597819168f0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.194+0000 7fbf23204740 1 --2- 172.21.15.40:0/470266984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55978189cf00 0x5597818a7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.194+0000 7fbf23204740 1 --2- 172.21.15.40:0/470266984 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559781897d90 0x5597818a6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.194+0000 7fbf23204740 1 --2- 172.21.15.40:0/470266984 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5597818973a0 0x5597818a6550 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.194+0000 7fbf23204740 1 -- 172.21.15.40:0/470266984 >> 172.21.15.40:0/470266984 conn(0x559781892910 msgr2=0x559781895b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:59.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.194+0000 7fbf01ffb640 0 client.17736 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:59.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.194+0000 7fbf23204740 1 -- 172.21.15.40:0/470266984 shutdown_connections 2024-03-18T22:50:59.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.194+0000 7fbf23204740 1 -- 172.21.15.40:0/470266984 wait complete. 2024-03-18T22:50:59.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.194+0000 7fbf23204740 20 client.17736 trim_cache size 0 max 16384 2024-03-18T22:50:59.202 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:50:59.202 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:50:59.202 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "87f07e11-cede-4227-a3f4-9b5a1b9fff49", 2024-03-18T22:50:59.202 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:50:59.202 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:50:59.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.195+0000 7fbf20612640 1 -- 172.21.15.40:0/1907168834 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbf040a2880 msgr2=0x7fbf040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.195+0000 7fbf20612640 1 --2- 172.21.15.40:0/1907168834 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbf040a2880 0x7fbf040a4d40 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fbf10005fd0 tx=0x7fbf10009500 comp rx=0 tx=0).stop 2024-03-18T22:50:59.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.195+0000 7fbf20612640 1 -- 172.21.15.40:0/1907168834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf18101910 msgr2=0x7fbf181980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.195+0000 7fbf20612640 1 --2- 172.21.15.40:0/1907168834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf18101910 0x7fbf181980d0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fbf0800e9f0 tx=0x7fbf0800edf0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.195+0000 7fbf20612640 1 -- 172.21.15.40:0/1907168834 shutdown_connections 2024-03-18T22:50:59.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.195+0000 7fbf20612640 1 --2- 172.21.15.40:0/1907168834 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbf040a2880 0x7fbf040a4d40 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.195+0000 7fbf20612640 1 --2- 172.21.15.40:0/1907168834 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbf181022d0 0x7fbf1819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.195+0000 7fbf20612640 1 --2- 172.21.15.40:0/1907168834 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf18101910 0x7fbf181980d0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.195+0000 7fbf20612640 1 --2- 172.21.15.40:0/1907168834 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf18100720 0x7fbf18197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.195+0000 7fbf20612640 1 -- 172.21.15.40:0/1907168834 >> 172.21.15.40:0/1907168834 conn(0x7fbf180fbcb0 msgr2=0x7fbf18068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:59.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.195+0000 7fbf20612640 1 -- 172.21.15.40:0/1907168834 shutdown_connections 2024-03-18T22:50:59.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.195+0000 7fbf20612640 1 -- 172.21.15.40:0/1907168834 wait complete. 2024-03-18T22:50:59.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:50:59.210 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:50:59.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.327+0000 7f0ae1bcd640 1 -- 172.21.15.40:0/719149797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0adc101910 msgr2=0x7f0adc101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.327+0000 7f0ae1bcd640 1 --2- 172.21.15.40:0/719149797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0adc101910 0x7f0adc101d90 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0009900 tx=0x7f0ad002bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.327+0000 7f0ae1bcd640 1 -- 172.21.15.40:0/719149797 shutdown_connections 2024-03-18T22:50:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.327+0000 7f0ae1bcd640 1 --2- 172.21.15.40:0/719149797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0adc1022d0 0x7f0adc108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.327+0000 7f0ae1bcd640 1 --2- 172.21.15.40:0/719149797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0adc101910 0x7f0adc101d90 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.327+0000 7f0ae1bcd640 1 --2- 172.21.15.40:0/719149797 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0adc100720 0x7f0adc100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.327+0000 7f0ae1bcd640 1 -- 172.21.15.40:0/719149797 >> 172.21.15.40:0/719149797 conn(0x7f0adc0fbcb0 msgr2=0x7f0adc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.327+0000 7f0ae1bcd640 1 -- 172.21.15.40:0/719149797 shutdown_connections 2024-03-18T22:50:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.327+0000 7f0ae1bcd640 1 -- 172.21.15.40:0/719149797 wait complete. 2024-03-18T22:50:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.328+0000 7f0ae1bcd640 1 Processor -- start 2024-03-18T22:50:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.328+0000 7f0ae1bcd640 1 -- start start 2024-03-18T22:50:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.328+0000 7f0ae1bcd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0adc100720 0x7f0adc0694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.328+0000 7f0ae1bcd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0adc101910 0x7f0adc19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.328+0000 7f0ae1bcd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0adc1022d0 0x7f0adc19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.328+0000 7f0ae1bcd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc10c230 con 0x7f0adc101910 2024-03-18T22:50:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.328+0000 7f0ae1bcd640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc10c0b0 con 0x7f0adc1022d0 2024-03-18T22:50:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.328+0000 7f0ae1bcd640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc10c3b0 con 0x7f0adc100720 2024-03-18T22:50:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.328+0000 7f0adaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0adc101910 0x7f0adc19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.328+0000 7f0adaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0adc101910 0x7f0adc19c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47874/0 (socket says 172.21.15.40:47874) 2024-03-18T22:50:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.328+0000 7f0adaffd640 1 -- 172.21.15.40:0/1149002123 learned_addr learned my addr 172.21.15.40:0/1149002123 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.328+0000 7f0adbfff640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0adc1022d0 0x7f0adc19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.328+0000 7f0adb7fe640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0adc100720 0x7f0adc0694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.328+0000 7f0adbfff640 1 -- 172.21.15.40:0/1149002123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0adc100720 msgr2=0x7f0adc0694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.329+0000 7f0adbfff640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0adc100720 0x7f0adc0694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.329+0000 7f0adbfff640 1 -- 172.21.15.40:0/1149002123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0adc101910 msgr2=0x7f0adc19c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.329+0000 7f0adbfff640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0adc101910 0x7f0adc19c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.329+0000 7f0adbfff640 1 -- 172.21.15.40:0/1149002123 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0adc19f720 con 0x7f0adc1022d0 2024-03-18T22:50:59.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.329+0000 7f0adaffd640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0adc101910 0x7f0adc19c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:59.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.329+0000 7f0adb7fe640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0adc100720 0x7f0adc0694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:59.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.329+0000 7f0adbfff640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0adc1022d0 0x7f0adc19ee10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4009fd0 tx=0x7f0ad400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:59.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.329+0000 7f0ad8ff9640 1 -- 172.21.15.40:0/1149002123 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad400ce20 con 0x7f0adc1022d0 2024-03-18T22:50:59.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.329+0000 7f0ae1bcd640 1 -- 172.21.15.40:0/1149002123 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0adc19f9b0 con 0x7f0adc1022d0 2024-03-18T22:50:59.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.329+0000 7f0ad8ff9640 1 -- 172.21.15.40:0/1149002123 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ad4005020 con 0x7f0adc1022d0 2024-03-18T22:50:59.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.329+0000 7f0ad8ff9640 1 -- 172.21.15.40:0/1149002123 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4010690 con 0x7f0adc1022d0 2024-03-18T22:50:59.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.329+0000 7f0ae1bcd640 1 -- 172.21.15.40:0/1149002123 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0adc19ff00 con 0x7f0adc1022d0 2024-03-18T22:50:59.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.331+0000 7f0ad8ff9640 1 -- 172.21.15.40:0/1149002123 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0ad40051d0 con 0x7f0adc1022d0 2024-03-18T22:50:59.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0ae47ba740 1 Processor -- start 2024-03-18T22:50:59.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0ad8ff9640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0ac40a2800 0x7f0ac40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0ae47ba740 1 -- start start 2024-03-18T22:50:59.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0adaffd640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0ac40a2800 0x7f0ac40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0ad8ff9640 1 -- 172.21.15.40:0/1149002123 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0ad40cf4f0 con 0x7f0adc1022d0 2024-03-18T22:50:59.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0ae47ba740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973aa50 0x55e90973ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0ae47ba740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973b440 0x55e90973ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0ae47ba740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 0x55e909742a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0ae47ba740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e90969f500 con 0x55e90973aa50 2024-03-18T22:50:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0ae47ba740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e909647be0 con 0x55e9097405b0 2024-03-18T22:50:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0ae47ba740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e90969b5a0 con 0x55e90973b440 2024-03-18T22:50:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0adb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973aa50 0x55e90973ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0adb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973aa50 0x55e90973ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47882/0 (socket says 172.21.15.40:47882) 2024-03-18T22:50:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0adb7fe640 1 -- 172.21.15.40:0/2563876860 learned_addr learned my addr 172.21.15.40:0/2563876860 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0adaffd640 1 --2- 172.21.15.40:0/2563876860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973b440 0x55e90973ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.332+0000 7f0adbfff640 1 --2- 172.21.15.40:0/2563876860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 0x55e909742a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.333+0000 7f0adb7fe640 1 -- 172.21.15.40:0/2563876860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973b440 msgr2=0x55e90973ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.333+0000 7f0adb7fe640 1 --2- 172.21.15.40:0/2563876860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973b440 0x55e90973ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.333+0000 7f0adb7fe640 1 -- 172.21.15.40:0/2563876860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 msgr2=0x55e909742a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.333+0000 7f0adb7fe640 1 --2- 172.21.15.40:0/2563876860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 0x55e909742a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.333+0000 7f0adb7fe640 1 -- 172.21.15.40:0/2563876860 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e909669000 con 0x55e90973aa50 2024-03-18T22:50:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.333+0000 7f0adbfff640 1 --2- 172.21.15.40:0/2563876860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 0x55e909742a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:50:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.333+0000 7f0adaffd640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0ac40a2800 0x7f0ac40a4cc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0adc069260 tx=0x7f0ad002cc90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.333+0000 7f0adb7fe640 1 --2- 172.21.15.40:0/2563876860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973aa50 0x55e90973ae70 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0114c0 tx=0x7f0acc011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc2d5fdcdd43c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.333+0000 7f0ac1ffb640 1 -- 172.21.15.40:0/2563876860 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc004020 con 0x55e90973aa50 2024-03-18T22:50:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.333+0000 7f0ac1ffb640 1 -- 172.21.15.40:0/2563876860 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0acc0054d0 con 0x55e90973aa50 2024-03-18T22:50:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.333+0000 7f0ac1ffb640 1 -- 172.21.15.40:0/2563876860 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc004cb0 con 0x55e90973aa50 2024-03-18T22:50:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.334+0000 7f0ae47ba740 1 -- 172.21.15.40:0/2563876860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973aa50 msgr2=0x55e90973ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.334+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/2563876860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973aa50 0x55e90973ae70 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0114c0 tx=0x7f0acc011980 comp rx=0 tx=0).stop 2024-03-18T22:50:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.334+0000 7f0ae47ba740 1 -- 172.21.15.40:0/2563876860 shutdown_connections 2024-03-18T22:50:59.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.334+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/2563876860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 0x55e909742a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.334+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/2563876860 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973b440 0x55e90973ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.334+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/2563876860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973aa50 0x55e90973ae70 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.334+0000 7f0ae47ba740 1 -- 172.21.15.40:0/2563876860 >> 172.21.15.40:0/2563876860 conn(0x55e909735fc0 msgr2=0x55e9097363c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:59.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.334+0000 7f0ae47ba740 1 -- 172.21.15.40:0/2563876860 shutdown_connections 2024-03-18T22:50:59.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.334+0000 7f0ae47ba740 1 -- 172.21.15.40:0/2563876860 wait complete. 2024-03-18T22:50:59.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.334+0000 7f0ae47ba740 1 -- start start 2024-03-18T22:50:59.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.334+0000 7f0ae47ba740 1 Processor -- start 2024-03-18T22:50:59.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0ae47ba740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973aa50 0x55e909744e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0ae47ba740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973b440 0x55e909745380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0ae47ba740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 0x55e90974c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0ae47ba740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e909743560 con 0x55e90973b440 2024-03-18T22:50:59.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0ae47ba740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e90965b160 con 0x55e9097405b0 2024-03-18T22:50:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0ae47ba740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e9097436e0 con 0x55e90973aa50 2024-03-18T22:50:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0ae47ba740 10 client.0 _ll_register_callbacks cb 0x55e909697a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0adaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973b440 0x55e909745380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0adb7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973aa50 0x55e909744e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0adaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973b440 0x55e909745380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47898/0 (socket says 172.21.15.40:47898) 2024-03-18T22:50:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0adaffd640 1 -- 172.21.15.40:0/417111196 learned_addr learned my addr 172.21.15.40:0/417111196 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0adbfff640 1 --2- 172.21.15.40:0/417111196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 0x55e90974c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0adaffd640 1 -- 172.21.15.40:0/417111196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973aa50 msgr2=0x55e909744e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0adaffd640 1 --2- 172.21.15.40:0/417111196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973aa50 0x55e909744e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0adaffd640 1 -- 172.21.15.40:0/417111196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 msgr2=0x55e90974c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0adaffd640 1 --2- 172.21.15.40:0/417111196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 0x55e90974c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0adaffd640 1 -- 172.21.15.40:0/417111196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e90947d7d0 con 0x55e90973b440 2024-03-18T22:50:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0adaffd640 1 --2- 172.21.15.40:0/417111196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973b440 0x55e909745380 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0009430 tx=0x7f0ad002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcb8f8ddf4d4fd11 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0abaffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:50:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0abaffd640 1 -- 172.21.15.40:0/417111196 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0047040 con 0x55e90973b440 2024-03-18T22:50:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0abaffd640 1 -- 172.21.15.40:0/417111196 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0ad00048d0 con 0x55e90973b440 2024-03-18T22:50:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.335+0000 7f0ae47ba740 1 -- 172.21.15.40:0/417111196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0acc0054d0 con 0x55e90973b440 2024-03-18T22:50:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.336+0000 7f0abaffd640 1 -- 172.21.15.40:0/417111196 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0ad0005600 con 0x55e90973b440 2024-03-18T22:50:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.336+0000 7f0ae47ba740 10 client.17748 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.336+0000 7f0ae47ba740 1 -- 172.21.15.40:0/417111196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55e9093b33f0 con 0x55e90973b440 2024-03-18T22:50:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.336+0000 7f0abaffd640 1 -- 172.21.15.40:0/417111196 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0acc0054d0 con 0x55e90973b440 2024-03-18T22:50:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.336+0000 7f0abaffd640 1 -- 172.21.15.40:0/417111196 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f0ad0003d00 con 0x55e90973b440 2024-03-18T22:50:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.336+0000 7f0ae47ba740 10 client.17748 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.336+0000 7f0ae47ba740 10 client.17748 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.336+0000 7f0ae47ba740 20 client.17748 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.336+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/417111196 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e909750d20 0x55e909753180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.336+0000 7f0ae47ba740 4 client.17748 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:50:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.336+0000 7f0ae47ba740 1 -- 172.21.15.40:0/417111196 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9096aca40 con 0x55e909750d20 2024-03-18T22:50:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.336+0000 7f0adb7fe640 1 --2- 172.21.15.40:0/417111196 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e909750d20 0x55e909753180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.337+0000 7f0abaffd640 10 client.17748 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.337+0000 7f0adb7fe640 1 --2- 172.21.15.40:0/417111196 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e909750d20 0x55e909753180 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=642fad405b718382 server_cookie=41130df58fa7f2d0 in_seq=0 out_seq=0 2024-03-18T22:50:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.337+0000 7f0abaffd640 1 -- 172.21.15.40:0/417111196 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e9096aca40 con 0x55e909750d20 2024-03-18T22:50:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.337+0000 7f0abaffd640 10 client.17748 handle_command_reply: tid=0 2024-03-18T22:50:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.338+0000 7f0ae47ba740 1 client.17748 shutdown 2024-03-18T22:50:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.338+0000 7f0ae47ba740 1 -- 172.21.15.40:0/417111196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973b440 msgr2=0x55e909745380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.338+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/417111196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973b440 0x55e909745380 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0009430 tx=0x7f0ad002bc30 comp rx=0 tx=0).stop 2024-03-18T22:50:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.338+0000 7f0ae47ba740 1 -- 172.21.15.40:0/417111196 shutdown_connections 2024-03-18T22:50:59.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.338+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/417111196 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e909750d20 0x55e909753180 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.338+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/417111196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 0x55e90974c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.338+0000 7f0abaffd640 0 client.17748 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:59.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.338+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/417111196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973b440 0x55e909745380 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.338+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/417111196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973aa50 0x55e909744e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.338+0000 7f0ae47ba740 1 -- 172.21.15.40:0/417111196 >> 172.21.15.40:0/417111196 conn(0x55e909735fc0 msgr2=0x55e9097351e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:59.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.338+0000 7f0ae47ba740 1 -- 172.21.15.40:0/417111196 shutdown_connections 2024-03-18T22:50:59.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.338+0000 7f0ae47ba740 1 -- 172.21.15.40:0/417111196 wait complete. 2024-03-18T22:50:59.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.338+0000 7f0ae47ba740 20 client.17748 trim_cache size 0 max 16384 2024-03-18T22:50:59.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.358+0000 7f0ae47ba740 1 Processor -- start 2024-03-18T22:50:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.358+0000 7f0ae47ba740 1 -- start start 2024-03-18T22:50:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.358+0000 7f0ae47ba740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e90973aa50 0x55e909739930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.358+0000 7f0ae47ba740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973b440 0x55e90974c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.358+0000 7f0ae47ba740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9097405b0 0x55e909757a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.358+0000 7f0ae47ba740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc1a8290 con 0x55e9097405b0 2024-03-18T22:50:59.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.358+0000 7f0ae47ba740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc1a9e90 con 0x55e90973aa50 2024-03-18T22:50:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.358+0000 7f0ae47ba740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc1a8960 con 0x55e90973b440 2024-03-18T22:50:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.358+0000 7f0adbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9097405b0 0x55e909757a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.358+0000 7f0adbfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9097405b0 0x55e909757a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47910/0 (socket says 172.21.15.40:47910) 2024-03-18T22:50:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.358+0000 7f0adbfff640 1 -- 172.21.15.40:0/190444079 learned_addr learned my addr 172.21.15.40:0/190444079 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.358+0000 7f0adb7fe640 1 --2- 172.21.15.40:0/190444079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e90973aa50 0x55e909739930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.358+0000 7f0adaffd640 1 --2- 172.21.15.40:0/190444079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973b440 0x55e90974c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.359+0000 7f0adbfff640 1 -- 172.21.15.40:0/190444079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973b440 msgr2=0x55e90974c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.359+0000 7f0adbfff640 1 --2- 172.21.15.40:0/190444079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973b440 0x55e90974c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.359+0000 7f0adbfff640 1 -- 172.21.15.40:0/190444079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e90973aa50 msgr2=0x55e909739930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.359+0000 7f0adbfff640 1 --2- 172.21.15.40:0/190444079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e90973aa50 0x55e909739930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.359+0000 7f0adbfff640 1 -- 172.21.15.40:0/190444079 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9093730f0 con 0x55e9097405b0 2024-03-18T22:50:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.359+0000 7f0adb7fe640 1 --2- 172.21.15.40:0/190444079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e90973aa50 0x55e909739930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.359+0000 7f0adaffd640 1 --2- 172.21.15.40:0/190444079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973b440 0x55e90974c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.359+0000 7f0adbfff640 1 --2- 172.21.15.40:0/190444079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9097405b0 0x55e909757a40 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0ad40ccc50 tx=0x7f0ad409bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf48315bdc3eec7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.359+0000 7f0ac17fa640 1 -- 172.21.15.40:0/190444079 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad40b9070 con 0x55e9097405b0 2024-03-18T22:50:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.359+0000 7f0ac17fa640 1 -- 172.21.15.40:0/190444079 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ad4099da0 con 0x55e9097405b0 2024-03-18T22:50:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.359+0000 7f0ac17fa640 1 -- 172.21.15.40:0/190444079 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad40a3db0 con 0x55e9097405b0 2024-03-18T22:50:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.360+0000 7f0ae47ba740 1 -- 172.21.15.40:0/190444079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9097405b0 msgr2=0x55e909757a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.360+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/190444079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9097405b0 0x55e909757a40 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0ad40ccc50 tx=0x7f0ad409bb10 comp rx=0 tx=0).stop 2024-03-18T22:50:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.360+0000 7f0ae47ba740 1 -- 172.21.15.40:0/190444079 shutdown_connections 2024-03-18T22:50:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.360+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/190444079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e9097405b0 0x55e909757a40 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.360+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/190444079 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973b440 0x55e90974c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.360+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/190444079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e90973aa50 0x55e909739930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.360+0000 7f0ae47ba740 1 -- 172.21.15.40:0/190444079 >> 172.21.15.40:0/190444079 conn(0x55e909735fc0 msgr2=0x55e909739220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:59.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.360+0000 7f0ae47ba740 1 -- 172.21.15.40:0/190444079 shutdown_connections 2024-03-18T22:50:59.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.360+0000 7f0ae47ba740 1 -- 172.21.15.40:0/190444079 wait complete. 2024-03-18T22:50:59.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.360+0000 7f0ae47ba740 1 -- start start 2024-03-18T22:50:59.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0ae47ba740 1 Processor -- start 2024-03-18T22:50:59.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0ae47ba740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973aa50 0x55e909749ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0ae47ba740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973b440 0x55e90974a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0ae47ba740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 0x55e90974b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0ae47ba740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc1a97c0 con 0x55e90973b440 2024-03-18T22:50:59.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0ae47ba740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc1a9030 con 0x55e9097405b0 2024-03-18T22:50:59.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0ae47ba740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0adc1a6270 con 0x55e90973aa50 2024-03-18T22:50:59.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0ae47ba740 10 client.0 _ll_register_callbacks cb 0x7f0acc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:50:59.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0adaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973b440 0x55e90974a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0adaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973b440 0x55e90974a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47912/0 (socket says 172.21.15.40:47912) 2024-03-18T22:50:59.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0adaffd640 1 -- 172.21.15.40:0/2945602810 learned_addr learned my addr 172.21.15.40:0/2945602810 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:50:59.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0adb7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973aa50 0x55e909749ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0adbfff640 1 --2- 172.21.15.40:0/2945602810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 0x55e90974b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0adb7fe640 1 -- 172.21.15.40:0/2945602810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 msgr2=0x55e90974b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0adb7fe640 1 --2- 172.21.15.40:0/2945602810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 0x55e90974b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0adb7fe640 1 -- 172.21.15.40:0/2945602810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973b440 msgr2=0x55e90974a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:50:59.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0adb7fe640 1 --2- 172.21.15.40:0/2945602810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973b440 0x55e90974a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0adb7fe640 1 -- 172.21.15.40:0/2945602810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e90974b9b0 con 0x55e90973aa50 2024-03-18T22:50:59.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0adaffd640 1 --2- 172.21.15.40:0/2945602810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973b440 0x55e90974a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:50:59.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0adb7fe640 1 --2- 172.21.15.40:0/2945602810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973aa50 0x55e909749ec0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0acc01b040 tx=0x7f0acc005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfd14b6aac1ab1f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:50:59.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.361+0000 7f0abaffd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:50:59.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.362+0000 7f0abaffd640 1 -- 172.21.15.40:0/2945602810 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc005480 con 0x55e90973aa50 2024-03-18T22:50:59.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.362+0000 7f0ae47ba740 1 -- 172.21.15.40:0/2945602810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ad4099da0 con 0x55e90973aa50 2024-03-18T22:50:59.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.362+0000 7f0abaffd640 1 -- 172.21.15.40:0/2945602810 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0acc030160 con 0x55e90973aa50 2024-03-18T22:50:59.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.362+0000 7f0abaffd640 1 -- 172.21.15.40:0/2945602810 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0acc040050 con 0x55e90973aa50 2024-03-18T22:50:59.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.362+0000 7f0ae47ba740 10 client.27559 fetch_fsmap learned FSMap version 34 2024-03-18T22:50:59.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.362+0000 7f0ae47ba740 1 -- 172.21.15.40:0/2945602810 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55e9097b74f0 con 0x55e90973aa50 2024-03-18T22:50:59.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.362+0000 7f0abaffd640 1 -- 172.21.15.40:0/2945602810 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ad4099da0 con 0x55e90973aa50 2024-03-18T22:50:59.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.362+0000 7f0abaffd640 1 -- 172.21.15.40:0/2945602810 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 34) ==== 8378+0+0 (secure 0 0 0) 0x7f0acc004b90 con 0x55e90973aa50 2024-03-18T22:50:59.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.362+0000 7f0ae47ba740 10 client.27559 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T22:50:59.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.362+0000 7f0ae47ba740 10 client.27559 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:50:59.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.362+0000 7f0ae47ba740 20 client.27559 populate_metadata read hostname 'smithi040' 2024-03-18T22:50:59.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.362+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/2945602810 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e909750d20 0x55e9097b9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:50:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.362+0000 7f0ae47ba740 4 client.27559 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:50:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.362+0000 7f0ae47ba740 1 -- 172.21.15.40:0/2945602810 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e9096a6d90 con 0x55e909750d20 2024-03-18T22:50:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0adaffd640 1 --2- 172.21.15.40:0/2945602810 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e909750d20 0x55e9097b9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:50:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0adaffd640 1 --2- 172.21.15.40:0/2945602810 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e909750d20 0x55e9097b9d10 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc254fba1a991f0 server_cookie=33a14c61f6b6f296 in_seq=0 out_seq=0 2024-03-18T22:50:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0abaffd640 10 client.27559 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0abaffd640 1 -- 172.21.15.40:0/2945602810 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e9096a6d90 con 0x55e909750d20 2024-03-18T22:50:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0abaffd640 10 client.27559 handle_command_reply: tid=0 2024-03-18T22:50:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0ae47ba740 1 client.27559 shutdown 2024-03-18T22:50:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0ae47ba740 1 -- 172.21.15.40:0/2945602810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973aa50 msgr2=0x55e909749ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/2945602810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973aa50 0x55e909749ec0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0acc01b040 tx=0x7f0acc005040 comp rx=0 tx=0).stop 2024-03-18T22:50:59.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0ae47ba740 1 -- 172.21.15.40:0/2945602810 shutdown_connections 2024-03-18T22:50:59.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/2945602810 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e909750d20 0x55e9097b9d10 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/2945602810 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9097405b0 0x55e90974b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/2945602810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e90973b440 0x55e90974a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0abaffd640 0 client.27559 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:50:59.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0ae47ba740 1 --2- 172.21.15.40:0/2945602810 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e90973aa50 0x55e909749ec0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0ae47ba740 1 -- 172.21.15.40:0/2945602810 >> 172.21.15.40:0/2945602810 conn(0x55e909735fc0 msgr2=0x55e909739ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:59.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0ae47ba740 1 -- 172.21.15.40:0/2945602810 shutdown_connections 2024-03-18T22:50:59.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0ae47ba740 1 -- 172.21.15.40:0/2945602810 wait complete. 2024-03-18T22:50:59.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.363+0000 7f0ae47ba740 20 client.27559 trim_cache size 0 max 16384 2024-03-18T22:50:59.376 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:50:59.376 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:50:59.376 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:50:59.376 INFO:teuthology.orchestra.run.smithi040.stdout: "87f07e11-cede-4227-a3f4-9b5a1b9fff49": { 2024-03-18T22:50:59.376 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:50:59.376 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "87f07e11-cede-4227-a3f4-9b5a1b9fff49", 2024-03-18T22:50:59.376 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:50:59.376 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:50:59.376 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:50:59.376 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:50:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.364+0000 7f0ae1bcd640 1 -- 172.21.15.40:0/1149002123 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0ac40a2800 msgr2=0x7f0ac40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.364+0000 7f0ae1bcd640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0ac40a2800 0x7f0ac40a4cc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0adc069260 tx=0x7f0ad002cc90 comp rx=0 tx=0).stop 2024-03-18T22:50:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.364+0000 7f0ae1bcd640 1 -- 172.21.15.40:0/1149002123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0adc1022d0 msgr2=0x7f0adc19ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:50:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.364+0000 7f0ae1bcd640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0adc1022d0 0x7f0adc19ee10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4009fd0 tx=0x7f0ad400ef10 comp rx=0 tx=0).stop 2024-03-18T22:50:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.364+0000 7f0ae1bcd640 1 -- 172.21.15.40:0/1149002123 shutdown_connections 2024-03-18T22:50:59.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.364+0000 7f0ae1bcd640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0ac40a2800 0x7f0ac40a4cc0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.364+0000 7f0ae1bcd640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0adc1022d0 0x7f0adc19ee10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.364+0000 7f0ae1bcd640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0adc101910 0x7f0adc19c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.364+0000 7f0ae1bcd640 1 --2- 172.21.15.40:0/1149002123 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0adc100720 0x7f0adc0694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:50:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.364+0000 7f0ae1bcd640 1 -- 172.21.15.40:0/1149002123 >> 172.21.15.40:0/1149002123 conn(0x7f0adc0fbcb0 msgr2=0x7f0adc104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:50:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.364+0000 7f0ae1bcd640 1 -- 172.21.15.40:0/1149002123 shutdown_connections 2024-03-18T22:50:59.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:50:59.364+0000 7f0ae1bcd640 1 -- 172.21.15.40:0/1149002123 wait complete. 2024-03-18T22:50:59.379 INFO:tasks.cephfs.filesystem:scrub status for tag:87f07e11-cede-4227-a3f4-9b5a1b9fff49 - {'path': '/', 'tag': '87f07e11-cede-4227-a3f4-9b5a1b9fff49', 'options': 'recursive,force'} 2024-03-18T22:51:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:00 smithi103 ceph-mon[66377]: pgmap v601: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:51:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:00 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:51:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:00 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:00 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:51:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:00 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:51:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:00 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:00.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:00 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:00.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:00 smithi040 ceph-mon[62643]: pgmap v601: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:51:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:00 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:51:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:00 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:00 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:51:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:00 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:51:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:00 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:00.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:00 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:00.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:00 smithi113 ceph-mon[66775]: pgmap v601: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 21 op/s 2024-03-18T22:51:00.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:00 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:51:00.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:00 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:00.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:00 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:51:00.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:00 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:51:00.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:00 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:00.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:00 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:02.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:02 smithi040 ceph-mon[62643]: pgmap v602: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 29 MiB/s wr, 24 op/s 2024-03-18T22:51:02.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:02 smithi113 ceph-mon[66775]: pgmap v602: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 29 MiB/s wr, 24 op/s 2024-03-18T22:51:02.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:02 smithi103 ceph-mon[66377]: pgmap v602: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 29 MiB/s wr, 24 op/s 2024-03-18T22:51:04.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:04 smithi040 ceph-mon[62643]: pgmap v603: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:51:04.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:04 smithi113 ceph-mon[66775]: pgmap v603: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:51:04.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:04 smithi103 ceph-mon[66377]: pgmap v603: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:51:06.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:06 smithi040 ceph-mon[62643]: pgmap v604: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:51:06.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:06 smithi113 ceph-mon[66775]: pgmap v604: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:51:06.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:06 smithi103 ceph-mon[66377]: pgmap v604: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:51:08.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:08 smithi040 ceph-mon[62643]: pgmap v605: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:51:08.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:08 smithi113 ceph-mon[66775]: pgmap v605: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:51:08.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:08 smithi103 ceph-mon[66377]: pgmap v605: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:51:10.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:10 smithi040 ceph-mon[62643]: pgmap v606: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:51:10.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:10 smithi113 ceph-mon[66775]: pgmap v606: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:51:10.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:10 smithi103 ceph-mon[66377]: pgmap v606: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:51:12.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:12 smithi040 ceph-mon[62643]: pgmap v607: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:51:12.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:12 smithi113 ceph-mon[66775]: pgmap v607: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:51:12.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:12 smithi103 ceph-mon[66377]: pgmap v607: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 22 op/s 2024-03-18T22:51:13.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:13 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:51:13.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:13 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:51:13.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:13 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:51:14.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:14 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:51:14.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:14 smithi040 ceph-mon[62643]: pgmap v608: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 27 MiB/s wr, 21 op/s 2024-03-18T22:51:14.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:14 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:51:14.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:14 smithi113 ceph-mon[66775]: pgmap v608: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 27 MiB/s wr, 21 op/s 2024-03-18T22:51:14.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:14 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:51:14.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:14 smithi103 ceph-mon[66377]: pgmap v608: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 27 MiB/s wr, 21 op/s 2024-03-18T22:51:16.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:16 smithi040 ceph-mon[62643]: pgmap v609: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:51:16.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:16 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:51:16.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:16 smithi113 ceph-mon[66775]: pgmap v609: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:51:16.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:16 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:51:16.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:16 smithi103 ceph-mon[66377]: pgmap v609: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 MiB/s wr, 22 op/s 2024-03-18T22:51:16.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:16 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:51:18.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:18 smithi040 ceph-mon[62643]: pgmap v610: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:51:18.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:18 smithi113 ceph-mon[66775]: pgmap v610: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:51:18.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:18 smithi103 ceph-mon[66377]: pgmap v610: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 21 op/s 2024-03-18T22:51:20.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:20 smithi040 ceph-mon[62643]: pgmap v611: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 21 op/s 2024-03-18T22:51:20.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:20 smithi113 ceph-mon[66775]: pgmap v611: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 21 op/s 2024-03-18T22:51:20.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:20 smithi103 ceph-mon[66377]: pgmap v611: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 21 op/s 2024-03-18T22:51:22.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:22 smithi040 ceph-mon[62643]: pgmap v612: 129 pgs: 129 active+clean; 12 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:51:22.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:22 smithi113 ceph-mon[66775]: pgmap v612: 129 pgs: 129 active+clean; 12 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:51:22.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:22 smithi103 ceph-mon[66377]: pgmap v612: 129 pgs: 129 active+clean; 12 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:51:24.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:24 smithi040 ceph-mon[62643]: pgmap v613: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 30 MiB/s wr, 23 op/s 2024-03-18T22:51:24.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:24 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:51:24.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:24 smithi113 ceph-mon[66775]: pgmap v613: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 30 MiB/s wr, 23 op/s 2024-03-18T22:51:24.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:24 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:51:24.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:24 smithi103 ceph-mon[66377]: pgmap v613: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 30 MiB/s wr, 23 op/s 2024-03-18T22:51:24.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:24 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:51:26.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:26 smithi040 ceph-mon[62643]: pgmap v614: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 MiB/s wr, 23 op/s 2024-03-18T22:51:26.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:51:26.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:51:26.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:51:26.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:26 smithi113 ceph-mon[66775]: pgmap v614: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 MiB/s wr, 23 op/s 2024-03-18T22:51:26.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:51:26.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:51:26.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:51:26.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:26 smithi103 ceph-mon[66377]: pgmap v614: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 MiB/s wr, 23 op/s 2024-03-18T22:51:26.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:51:26.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:51:26.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:51:28.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:28 smithi040 ceph-mon[62643]: pgmap v615: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:51:28.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:28 smithi113 ceph-mon[66775]: pgmap v615: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:51:28.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:28 smithi103 ceph-mon[66377]: pgmap v615: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 30 MiB/s wr, 21 op/s 2024-03-18T22:51:29.381 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:51:29.381 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:51:29.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.826+0000 7fd8ff952640 1 -- 172.21.15.40:0/3086708961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f8102340 msgr2=0x7fd8f8104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:29.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.826+0000 7fd8ff952640 1 --2- 172.21.15.40:0/3086708961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f8102340 0x7fd8f8104730 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4009900 tx=0x7fd8f402bc30 comp rx=0 tx=0).stop 2024-03-18T22:51:29.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.826+0000 7fd8ff952640 1 -- 172.21.15.40:0/3086708961 shutdown_connections 2024-03-18T22:51:29.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.826+0000 7fd8ff952640 1 --2- 172.21.15.40:0/3086708961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8f81075a0 0x7fd8f8109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.826+0000 7fd8ff952640 1 --2- 172.21.15.40:0/3086708961 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8f8104c70 0x7fd8f8107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.826+0000 7fd8ff952640 1 --2- 172.21.15.40:0/3086708961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f8102340 0x7fd8f8104730 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.826+0000 7fd8ff952640 1 -- 172.21.15.40:0/3086708961 >> 172.21.15.40:0/3086708961 conn(0x7fd8f80fbcb0 msgr2=0x7fd8f80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:29.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.826+0000 7fd8ff952640 1 -- 172.21.15.40:0/3086708961 shutdown_connections 2024-03-18T22:51:29.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.826+0000 7fd8ff952640 1 -- 172.21.15.40:0/3086708961 wait complete. 2024-03-18T22:51:29.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8ff952640 1 Processor -- start 2024-03-18T22:51:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8ff952640 1 -- start start 2024-03-18T22:51:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8ff952640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f8102340 0x7fd8f8197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8ff952640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8f8104c70 0x7fd8f8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8ff952640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8f81075a0 0x7fd8f819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8ff952640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f810d340 con 0x7fd8f8102340 2024-03-18T22:51:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8ff952640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f810d1c0 con 0x7fd8f81075a0 2024-03-18T22:51:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8ff952640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f810d4c0 con 0x7fd8f8104c70 2024-03-18T22:51:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8fd6ee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f8102340 0x7fd8f8197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8fd6ee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f8102340 0x7fd8f8197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49346/0 (socket says 172.21.15.40:49346) 2024-03-18T22:51:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8fd6ee640 1 -- 172.21.15.40:0/3574747292 learned_addr learned my addr 172.21.15.40:0/3574747292 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8fceed640 1 --2- 172.21.15.40:0/3574747292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8f8104c70 0x7fd8f8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8fdeef640 1 --2- 172.21.15.40:0/3574747292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8f81075a0 0x7fd8f819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8fd6ee640 1 -- 172.21.15.40:0/3574747292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8f8104c70 msgr2=0x7fd8f8198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8fd6ee640 1 --2- 172.21.15.40:0/3574747292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8f8104c70 0x7fd8f8198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8fd6ee640 1 -- 172.21.15.40:0/3574747292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8f81075a0 msgr2=0x7fd8f819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:29.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8fd6ee640 1 --2- 172.21.15.40:0/3574747292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8f81075a0 0x7fd8f819f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.827+0000 7fd8fd6ee640 1 -- 172.21.15.40:0/3574747292 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f819f6f0 con 0x7fd8f8102340 2024-03-18T22:51:29.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.828+0000 7fd8fd6ee640 1 --2- 172.21.15.40:0/3574747292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f8102340 0x7fd8f8197ad0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4006010 tx=0x7fd8f4005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:29.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.828+0000 7fd8ee7fc640 1 -- 172.21.15.40:0/3574747292 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4043020 con 0x7fd8f8102340 2024-03-18T22:51:29.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.828+0000 7fd8ff952640 1 -- 172.21.15.40:0/3574747292 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8f819f980 con 0x7fd8f8102340 2024-03-18T22:51:29.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.828+0000 7fd8ee7fc640 1 -- 172.21.15.40:0/3574747292 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8f40043d0 con 0x7fd8f8102340 2024-03-18T22:51:29.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.828+0000 7fd8ee7fc640 1 -- 172.21.15.40:0/3574747292 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4003b80 con 0x7fd8f8102340 2024-03-18T22:51:29.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.828+0000 7fd8ff952640 1 -- 172.21.15.40:0/3574747292 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8f819ff20 con 0x7fd8f8102340 2024-03-18T22:51:29.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.830+0000 7fd8ee7fc640 1 -- 172.21.15.40:0/3574747292 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd8f4046020 con 0x7fd8f8102340 2024-03-18T22:51:29.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.831+0000 7fd90260e740 1 Processor -- start 2024-03-18T22:51:29.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.831+0000 7fd90260e740 1 -- start start 2024-03-18T22:51:29.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.831+0000 7fd90260e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9efa50 0x56248d9efe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.831+0000 7fd90260e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f0440 0x56248d9f4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.831+0000 7fd90260e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9f55b0 0x56248d9f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.831+0000 7fd90260e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56248d954500 con 0x56248d9efa50 2024-03-18T22:51:29.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.831+0000 7fd90260e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56248d8fcbe0 con 0x56248d9f0440 2024-03-18T22:51:29.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.831+0000 7fd90260e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56248d9505a0 con 0x56248d9f55b0 2024-03-18T22:51:29.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.831+0000 7fd8ee7fc640 1 --2- 172.21.15.40:0/3574747292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd8e40a2800 0x7fd8e40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.831+0000 7fd8fdeef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9efa50 0x56248d9efe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.831+0000 7fd8fdeef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9efa50 0x56248d9efe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49350/0 (socket says 172.21.15.40:49350) 2024-03-18T22:51:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.831+0000 7fd8fd6ee640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f0440 0x56248d9f4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.831+0000 7fd8fdeef640 1 -- 172.21.15.40:0/2137937131 learned_addr learned my addr 172.21.15.40:0/2137937131 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.831+0000 7fd8fceed640 1 --2- 172.21.15.40:0/2137937131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9f55b0 0x56248d9f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.832+0000 7fd8fceed640 1 -- 172.21.15.40:0/2137937131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f0440 msgr2=0x56248d9f4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.832+0000 7fd8fceed640 1 --2- 172.21.15.40:0/2137937131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f0440 0x56248d9f4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.832+0000 7fd8fceed640 1 -- 172.21.15.40:0/2137937131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9efa50 msgr2=0x56248d9efe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.832+0000 7fd8fceed640 1 --2- 172.21.15.40:0/2137937131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9efa50 0x56248d9efe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.832+0000 7fd8fceed640 1 -- 172.21.15.40:0/2137937131 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56248d91e000 con 0x56248d9f55b0 2024-03-18T22:51:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.832+0000 7fd8ee7fc640 1 -- 172.21.15.40:0/3574747292 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd8f40f1060 con 0x7fd8f8102340 2024-03-18T22:51:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.832+0000 7fd8fdeef640 1 --2- 172.21.15.40:0/3574747292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd8e40a2800 0x7fd8e40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.832+0000 7fd8fceed640 1 --2- 172.21.15.40:0/2137937131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9f55b0 0x56248d9f7a40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8069440 tx=0x7fd8e8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f88fd53173d11c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.832+0000 7fd8e37fe640 1 -- 172.21.15.40:0/2137937131 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8004050 con 0x56248d9f55b0 2024-03-18T22:51:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.832+0000 7fd8e37fe640 1 -- 172.21.15.40:0/2137937131 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8e80054d0 con 0x56248d9f55b0 2024-03-18T22:51:29.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.832+0000 7fd8e37fe640 1 -- 172.21.15.40:0/2137937131 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8004c80 con 0x56248d9f55b0 2024-03-18T22:51:29.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.833+0000 7fd90260e740 1 -- 172.21.15.40:0/2137937131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9f55b0 msgr2=0x56248d9f7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:29.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.833+0000 7fd8fdeef640 1 --2- 172.21.15.40:0/3574747292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd8e40a2800 0x7fd8e40a4cc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0009fd0 tx=0x7fd8f0004a40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:29.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.833+0000 7fd90260e740 1 --2- 172.21.15.40:0/2137937131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9f55b0 0x56248d9f7a40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8069440 tx=0x7fd8e8011810 comp rx=0 tx=0).stop 2024-03-18T22:51:29.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.833+0000 7fd90260e740 1 -- 172.21.15.40:0/2137937131 shutdown_connections 2024-03-18T22:51:29.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.833+0000 7fd90260e740 1 --2- 172.21.15.40:0/2137937131 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9f55b0 0x56248d9f7a40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.833+0000 7fd90260e740 1 --2- 172.21.15.40:0/2137937131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f0440 0x56248d9f4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.833+0000 7fd90260e740 1 --2- 172.21.15.40:0/2137937131 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9efa50 0x56248d9efe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.833+0000 7fd90260e740 1 -- 172.21.15.40:0/2137937131 >> 172.21.15.40:0/2137937131 conn(0x56248d9eafc0 msgr2=0x56248d9eb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:29.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.833+0000 7fd90260e740 1 -- 172.21.15.40:0/2137937131 shutdown_connections 2024-03-18T22:51:29.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.833+0000 7fd90260e740 1 -- 172.21.15.40:0/2137937131 wait complete. 2024-03-18T22:51:29.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.833+0000 7fd90260e740 1 -- start start 2024-03-18T22:51:29.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.833+0000 7fd90260e740 1 Processor -- start 2024-03-18T22:51:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd90260e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9efa50 0x56248d9fe2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd90260e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f0440 0x56248d9fe7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd90260e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f55b0 0x56248da05830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd90260e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56248d9fc970 con 0x56248d9f55b0 2024-03-18T22:51:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd90260e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56248d910160 con 0x56248d9f0440 2024-03-18T22:51:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd90260e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56248d9fcaf0 con 0x56248d9efa50 2024-03-18T22:51:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd90260e740 10 client.0 _ll_register_callbacks cb 0x56248d94ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd8fceed640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f0440 0x56248d9fe7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd8fd6ee640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9efa50 0x56248d9fe2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd8fceed640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f0440 0x56248d9fe7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:35268/0 (socket says 172.21.15.40:35268) 2024-03-18T22:51:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd8fceed640 1 -- 172.21.15.40:0/790059517 learned_addr learned my addr 172.21.15.40:0/790059517 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd8fdeef640 1 --2- 172.21.15.40:0/790059517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f55b0 0x56248da05830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd8fceed640 1 -- 172.21.15.40:0/790059517 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9efa50 msgr2=0x56248d9fe2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd8fceed640 1 --2- 172.21.15.40:0/790059517 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9efa50 0x56248d9fe2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd8fceed640 1 -- 172.21.15.40:0/790059517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f55b0 msgr2=0x56248da05830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd8fceed640 1 --2- 172.21.15.40:0/790059517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f55b0 0x56248da05830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.834+0000 7fd8fceed640 1 -- 172.21.15.40:0/790059517 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56248d7327d0 con 0x56248d9f0440 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.835+0000 7fd8fceed640 1 --2- 172.21.15.40:0/790059517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f0440 0x56248d9fe7f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8000c00 tx=0x7fd8e8009fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50363bbdf040d7cc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.835+0000 7fd8e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.835+0000 7fd8e0ff9640 1 -- 172.21.15.40:0/790059517 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8029070 con 0x56248d9f0440 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.835+0000 7fd8e0ff9640 1 -- 172.21.15.40:0/790059517 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd8e8003a00 con 0x56248d9f0440 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.835+0000 7fd90260e740 1 -- 172.21.15.40:0/790059517 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8e80054d0 con 0x56248d9f0440 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.835+0000 7fd8e0ff9640 1 -- 172.21.15.40:0/790059517 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd8e8035050 con 0x56248d9f0440 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.835+0000 7fd90260e740 10 client.27581 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.835+0000 7fd90260e740 1 -- 172.21.15.40:0/790059517 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56248d6683f0 con 0x56248d9f0440 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.835+0000 7fd8e0ff9640 1 -- 172.21.15.40:0/790059517 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8e80054d0 con 0x56248d9f0440 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.836+0000 7fd8e0ff9640 1 -- 172.21.15.40:0/790059517 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7fd8e8005170 con 0x56248d9f0440 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.836+0000 7fd90260e740 10 client.27581 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.836+0000 7fd90260e740 10 client.27581 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:51:29.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.836+0000 7fd90260e740 20 client.27581 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:29.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.836+0000 7fd90260e740 1 --2- 172.21.15.40:0/790059517 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56248da09ea0 0x56248da0c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.836+0000 7fd90260e740 4 client.27581 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:51:29.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.836+0000 7fd90260e740 1 -- 172.21.15.40:0/790059517 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56248d961a40 con 0x56248da09ea0 2024-03-18T22:51:29.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.836+0000 7fd8fd6ee640 1 --2- 172.21.15.40:0/790059517 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56248da09ea0 0x56248da0c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.836+0000 7fd8e0ff9640 10 client.27581 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:29.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.836+0000 7fd8fd6ee640 1 --2- 172.21.15.40:0/790059517 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56248da09ea0 0x56248da0c300 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f62c4bc1d395faba server_cookie=4bf84d9c857ed943 in_seq=0 out_seq=0 2024-03-18T22:51:29.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.837+0000 7fd8e0ff9640 1 -- 172.21.15.40:0/790059517 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56248d961a40 con 0x56248da09ea0 2024-03-18T22:51:29.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.837+0000 7fd8e0ff9640 10 client.27581 handle_command_reply: tid=0 2024-03-18T22:51:29.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.837+0000 7fd90260e740 1 client.27581 shutdown 2024-03-18T22:51:29.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.837+0000 7fd90260e740 1 -- 172.21.15.40:0/790059517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f0440 msgr2=0x56248d9fe7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:29.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.837+0000 7fd90260e740 1 --2- 172.21.15.40:0/790059517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f0440 0x56248d9fe7f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8000c00 tx=0x7fd8e8009fa0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.837+0000 7fd90260e740 1 -- 172.21.15.40:0/790059517 shutdown_connections 2024-03-18T22:51:29.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.837+0000 7fd90260e740 1 --2- 172.21.15.40:0/790059517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f55b0 0x56248da05830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.837+0000 7fd90260e740 1 --2- 172.21.15.40:0/790059517 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56248da09ea0 0x56248da0c300 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.837+0000 7fd90260e740 1 --2- 172.21.15.40:0/790059517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f0440 0x56248d9fe7f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.837+0000 7fd8e0ff9640 0 client.27581 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.837+0000 7fd90260e740 1 --2- 172.21.15.40:0/790059517 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9efa50 0x56248d9fe2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.837+0000 7fd90260e740 1 -- 172.21.15.40:0/790059517 >> 172.21.15.40:0/790059517 conn(0x56248d9eafc0 msgr2=0x56248d9ea1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.838+0000 7fd90260e740 1 -- 172.21.15.40:0/790059517 shutdown_connections 2024-03-18T22:51:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.838+0000 7fd90260e740 1 -- 172.21.15.40:0/790059517 wait complete. 2024-03-18T22:51:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.838+0000 7fd90260e740 20 client.27581 trim_cache size 0 max 16384 2024-03-18T22:51:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.862+0000 7fd90260e740 1 Processor -- start 2024-03-18T22:51:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.862+0000 7fd90260e740 1 -- start start 2024-03-18T22:51:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.862+0000 7fd90260e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9efa50 0x56248d9e9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.862+0000 7fd90260e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9f0440 0x56248da057f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.862+0000 7fd90260e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f55b0 0x56248d9e9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.862+0000 7fd90260e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81a8390 con 0x56248d9f55b0 2024-03-18T22:51:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.862+0000 7fd90260e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81a9f30 con 0x56248d9efa50 2024-03-18T22:51:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.862+0000 7fd90260e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81a8a60 con 0x56248d9f0440 2024-03-18T22:51:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.862+0000 7fd8fdeef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f55b0 0x56248d9e9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.862+0000 7fd8fceed640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9f0440 0x56248da057f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.862+0000 7fd8fceed640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9f0440 0x56248da057f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53804/0 (socket says 172.21.15.40:53804) 2024-03-18T22:51:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.862+0000 7fd8fdeef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f55b0 0x56248d9e9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49366/0 (socket says 172.21.15.40:49366) 2024-03-18T22:51:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.862+0000 7fd8fd6ee640 1 --2- 172.21.15.40:0/730110326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9efa50 0x56248d9e9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.862+0000 7fd8fceed640 1 -- 172.21.15.40:0/730110326 learned_addr learned my addr 172.21.15.40:0/730110326 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.863+0000 7fd8fceed640 1 -- 172.21.15.40:0/730110326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9efa50 msgr2=0x56248d9e9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:29.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.863+0000 7fd8fceed640 1 --2- 172.21.15.40:0/730110326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9efa50 0x56248d9e9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.863+0000 7fd8fceed640 1 -- 172.21.15.40:0/730110326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f55b0 msgr2=0x56248d9e9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:29.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.863+0000 7fd8fceed640 1 --2- 172.21.15.40:0/730110326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f55b0 0x56248d9e9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.863+0000 7fd8fceed640 1 -- 172.21.15.40:0/730110326 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56248d6280f0 con 0x56248d9f0440 2024-03-18T22:51:29.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.863+0000 7fd8fceed640 1 --2- 172.21.15.40:0/730110326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9f0440 0x56248da057f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8011850 tx=0x7fd8e8009a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=915056b21d98860d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:29.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.863+0000 7fd8e2ffd640 1 -- 172.21.15.40:0/730110326 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8029070 con 0x56248d9f0440 2024-03-18T22:51:29.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.863+0000 7fd8e2ffd640 1 -- 172.21.15.40:0/730110326 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8e8005020 con 0x56248d9f0440 2024-03-18T22:51:29.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.863+0000 7fd8e2ffd640 1 -- 172.21.15.40:0/730110326 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8003ad0 con 0x56248d9f0440 2024-03-18T22:51:29.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.864+0000 7fd90260e740 1 -- 172.21.15.40:0/730110326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9f0440 msgr2=0x56248da057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:29.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.864+0000 7fd90260e740 1 --2- 172.21.15.40:0/730110326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9f0440 0x56248da057f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8011850 tx=0x7fd8e8009a90 comp rx=0 tx=0).stop 2024-03-18T22:51:29.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.864+0000 7fd90260e740 1 -- 172.21.15.40:0/730110326 shutdown_connections 2024-03-18T22:51:29.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.864+0000 7fd90260e740 1 --2- 172.21.15.40:0/730110326 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f55b0 0x56248d9e9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.864+0000 7fd90260e740 1 --2- 172.21.15.40:0/730110326 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9f0440 0x56248da057f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.864+0000 7fd90260e740 1 --2- 172.21.15.40:0/730110326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9efa50 0x56248d9e9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.864+0000 7fd90260e740 1 -- 172.21.15.40:0/730110326 >> 172.21.15.40:0/730110326 conn(0x56248d9eafc0 msgr2=0x56248d9f1860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.864+0000 7fd90260e740 1 -- 172.21.15.40:0/730110326 shutdown_connections 2024-03-18T22:51:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.864+0000 7fd90260e740 1 -- 172.21.15.40:0/730110326 wait complete. 2024-03-18T22:51:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.864+0000 7fd90260e740 1 -- start start 2024-03-18T22:51:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.864+0000 7fd90260e740 1 Processor -- start 2024-03-18T22:51:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd90260e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9efa50 0x56248da027e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd90260e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f0440 0x56248da02d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd90260e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f55b0 0x56248da6e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd90260e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81a9860 con 0x56248d9f0440 2024-03-18T22:51:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd90260e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81a9130 con 0x56248d9f55b0 2024-03-18T22:51:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd90260e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81a6370 con 0x56248d9efa50 2024-03-18T22:51:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd90260e740 10 client.0 _ll_register_callbacks cb 0x7fd8f0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd8fd6ee640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9efa50 0x56248da027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd8fceed640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f0440 0x56248da02d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd8fd6ee640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9efa50 0x56248da027e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53808/0 (socket says 172.21.15.40:53808) 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd8fd6ee640 1 -- 172.21.15.40:0/3872970016 learned_addr learned my addr 172.21.15.40:0/3872970016 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd8fdeef640 1 --2- 172.21.15.40:0/3872970016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f55b0 0x56248da6e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd8fd6ee640 1 -- 172.21.15.40:0/3872970016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f55b0 msgr2=0x56248da6e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd8fd6ee640 1 --2- 172.21.15.40:0/3872970016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f55b0 0x56248da6e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd8fd6ee640 1 -- 172.21.15.40:0/3872970016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f0440 msgr2=0x56248da02d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd8fd6ee640 1 --2- 172.21.15.40:0/3872970016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f0440 0x56248da02d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd8fd6ee640 1 -- 172.21.15.40:0/3872970016 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56248da6e7b0 con 0x56248d9efa50 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd8fceed640 1 --2- 172.21.15.40:0/3872970016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f0440 0x56248da02d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd8fd6ee640 1 --2- 172.21.15.40:0/3872970016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9efa50 0x56248da027e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd8f40bd310 tx=0x7fd8f40cb040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c96a6b5fadc03a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd8e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd8e0ff9640 1 -- 172.21.15.40:0/3872970016 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f40bd610 con 0x56248d9efa50 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd8e0ff9640 1 -- 172.21.15.40:0/3872970016 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd8f4003d20 con 0x56248d9efa50 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.865+0000 7fd90260e740 1 -- 172.21.15.40:0/3872970016 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8e8005020 con 0x56248d9efa50 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.866+0000 7fd8e0ff9640 1 -- 172.21.15.40:0/3872970016 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd8f40c1760 con 0x56248d9efa50 2024-03-18T22:51:29.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.866+0000 7fd90260e740 10 client.27577 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.866+0000 7fd90260e740 1 -- 172.21.15.40:0/3872970016 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56248da6fb30 con 0x56248d9efa50 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.866+0000 7fd8e0ff9640 1 -- 172.21.15.40:0/3872970016 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8e8005020 con 0x56248d9efa50 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.866+0000 7fd8e0ff9640 1 -- 172.21.15.40:0/3872970016 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7fd8f40bd870 con 0x56248d9efa50 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.866+0000 7fd90260e740 10 client.27577 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.866+0000 7fd90260e740 10 client.27577 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.866+0000 7fd90260e740 20 client.27577 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.866+0000 7fd90260e740 1 --2- 172.21.15.40:0/3872970016 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56248da09ea0 0x56248da72570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.866+0000 7fd90260e740 4 client.27577 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.866+0000 7fd90260e740 1 -- 172.21.15.40:0/3872970016 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56248d95bd90 con 0x56248da09ea0 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.866+0000 7fd8fceed640 1 --2- 172.21.15.40:0/3872970016 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56248da09ea0 0x56248da72570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd8fceed640 1 --2- 172.21.15.40:0/3872970016 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56248da09ea0 0x56248da72570 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72aca7e47fd4336c server_cookie=9f550722113d2674 in_seq=0 out_seq=0 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd8e0ff9640 10 client.27577 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd8e0ff9640 1 -- 172.21.15.40:0/3872970016 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56248d95bd90 con 0x56248da09ea0 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd8e0ff9640 10 client.27577 handle_command_reply: tid=0 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd90260e740 1 client.27577 shutdown 2024-03-18T22:51:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd90260e740 1 -- 172.21.15.40:0/3872970016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9efa50 msgr2=0x56248da027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:29.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd90260e740 1 --2- 172.21.15.40:0/3872970016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9efa50 0x56248da027e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd8f40bd310 tx=0x7fd8f40cb040 comp rx=0 tx=0).stop 2024-03-18T22:51:29.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd90260e740 1 -- 172.21.15.40:0/3872970016 shutdown_connections 2024-03-18T22:51:29.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd90260e740 1 --2- 172.21.15.40:0/3872970016 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56248da09ea0 0x56248da72570 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd90260e740 1 --2- 172.21.15.40:0/3872970016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56248d9f55b0 0x56248da6e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd90260e740 1 --2- 172.21.15.40:0/3872970016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56248d9f0440 0x56248da02d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd8e0ff9640 0 client.27577 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:29.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd90260e740 1 --2- 172.21.15.40:0/3872970016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56248d9efa50 0x56248da027e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd90260e740 1 -- 172.21.15.40:0/3872970016 >> 172.21.15.40:0/3872970016 conn(0x56248d9eafc0 msgr2=0x56248d9ee220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:29.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd90260e740 1 -- 172.21.15.40:0/3872970016 shutdown_connections 2024-03-18T22:51:29.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd90260e740 1 -- 172.21.15.40:0/3872970016 wait complete. 2024-03-18T22:51:29.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.867+0000 7fd90260e740 20 client.27577 trim_cache size 0 max 16384 2024-03-18T22:51:29.874 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:51:29.874 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:51:29.874 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:51:29.874 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:51:29.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.868+0000 7fd8ff952640 1 -- 172.21.15.40:0/3574747292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd8e40a2800 msgr2=0x7fd8e40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:29.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.868+0000 7fd8ff952640 1 --2- 172.21.15.40:0/3574747292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd8e40a2800 0x7fd8e40a4cc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0009fd0 tx=0x7fd8f0004a40 comp rx=0 tx=0).stop 2024-03-18T22:51:29.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.868+0000 7fd8ff952640 1 -- 172.21.15.40:0/3574747292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f8102340 msgr2=0x7fd8f8197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:29.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.868+0000 7fd8ff952640 1 --2- 172.21.15.40:0/3574747292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f8102340 0x7fd8f8197ad0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4006010 tx=0x7fd8f4005e40 comp rx=0 tx=0).stop 2024-03-18T22:51:29.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.868+0000 7fd8ff952640 1 -- 172.21.15.40:0/3574747292 shutdown_connections 2024-03-18T22:51:29.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.868+0000 7fd8ff952640 1 --2- 172.21.15.40:0/3574747292 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd8e40a2800 0x7fd8e40a4cc0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.868+0000 7fd8ff952640 1 --2- 172.21.15.40:0/3574747292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8f81075a0 0x7fd8f819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.868+0000 7fd8ff952640 1 --2- 172.21.15.40:0/3574747292 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd8f8104c70 0x7fd8f8198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.868+0000 7fd8ff952640 1 --2- 172.21.15.40:0/3574747292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd8f8102340 0x7fd8f8197ad0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:29.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.868+0000 7fd8ff952640 1 -- 172.21.15.40:0/3574747292 >> 172.21.15.40:0/3574747292 conn(0x7fd8f80fbcb0 msgr2=0x7fd8f81041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:29.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.868+0000 7fd8ff952640 1 -- 172.21.15.40:0/3574747292 shutdown_connections 2024-03-18T22:51:29.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:29.868+0000 7fd8ff952640 1 -- 172.21.15.40:0/3574747292 wait complete. 2024-03-18T22:51:29.881 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:51:29.881 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:51:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.007+0000 7ff52cb09640 1 -- 172.21.15.40:0/3418396328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff528101930 msgr2=0x7ff528101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.007+0000 7ff52cb09640 1 --2- 172.21.15.40:0/3418396328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff528101930 0x7ff528101db0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7ff510009900 tx=0x7ff51002bc30 comp rx=0 tx=0).stop 2024-03-18T22:51:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.007+0000 7ff52cb09640 1 -- 172.21.15.40:0/3418396328 shutdown_connections 2024-03-18T22:51:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.007+0000 7ff52cb09640 1 --2- 172.21.15.40:0/3418396328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5281022f0 0x7ff528108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.007+0000 7ff52cb09640 1 --2- 172.21.15.40:0/3418396328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff528101930 0x7ff528101db0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.007+0000 7ff52cb09640 1 --2- 172.21.15.40:0/3418396328 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff528100740 0x7ff528100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.007+0000 7ff52cb09640 1 -- 172.21.15.40:0/3418396328 >> 172.21.15.40:0/3418396328 conn(0x7ff5280fbcb0 msgr2=0x7ff5280fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.007+0000 7ff52cb09640 1 -- 172.21.15.40:0/3418396328 shutdown_connections 2024-03-18T22:51:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.007+0000 7ff52cb09640 1 -- 172.21.15.40:0/3418396328 wait complete. 2024-03-18T22:51:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.007+0000 7ff52cb09640 1 Processor -- start 2024-03-18T22:51:30.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.007+0000 7ff52cb09640 1 -- start start 2024-03-18T22:51:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff52cb09640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff528100740 0x7ff528197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff52cb09640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff528101930 0x7ff528198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff52cb09640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5281022f0 0x7ff52819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff52cb09640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff52810c250 con 0x7ff528100740 2024-03-18T22:51:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff52cb09640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff52810c0d0 con 0x7ff5281022f0 2024-03-18T22:51:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff52cb09640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff52810c3d0 con 0x7ff528101930 2024-03-18T22:51:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff52659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff528100740 0x7ff528197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff51ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff528101930 0x7ff528198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff52659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff528100740 0x7ff528197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49386/0 (socket says 172.21.15.40:49386) 2024-03-18T22:51:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff52659c640 1 -- 172.21.15.40:0/3739831223 learned_addr learned my addr 172.21.15.40:0/3739831223 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff526d9d640 1 --2- 172.21.15.40:0/3739831223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5281022f0 0x7ff52819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff51ffff640 1 -- 172.21.15.40:0/3739831223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5281022f0 msgr2=0x7ff52819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff51ffff640 1 --2- 172.21.15.40:0/3739831223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5281022f0 0x7ff52819f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff51ffff640 1 -- 172.21.15.40:0/3739831223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff528100740 msgr2=0x7ff528197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff51ffff640 1 --2- 172.21.15.40:0/3739831223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff528100740 0x7ff528197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff51ffff640 1 -- 172.21.15.40:0/3739831223 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52819f6f0 con 0x7ff528101930 2024-03-18T22:51:30.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff52659c640 1 --2- 172.21.15.40:0/3739831223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff528100740 0x7ff528197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.008+0000 7ff51ffff640 1 --2- 172.21.15.40:0/3739831223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff528101930 0x7ff528198010 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff510000c00 tx=0x7ff510005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.009+0000 7ff51f7fe640 1 -- 172.21.15.40:0/3739831223 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff510043020 con 0x7ff528101930 2024-03-18T22:51:30.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.009+0000 7ff51f7fe640 1 -- 172.21.15.40:0/3739831223 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff510004350 con 0x7ff528101930 2024-03-18T22:51:30.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.009+0000 7ff51f7fe640 1 -- 172.21.15.40:0/3739831223 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff510003b50 con 0x7ff528101930 2024-03-18T22:51:30.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.009+0000 7ff52cb09640 1 -- 172.21.15.40:0/3739831223 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff52819f980 con 0x7ff528101930 2024-03-18T22:51:30.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.009+0000 7ff52cb09640 1 -- 172.21.15.40:0/3739831223 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff52819ff20 con 0x7ff528101930 2024-03-18T22:51:30.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.010+0000 7ff52cb09640 1 -- 172.21.15.40:0/3739831223 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5281a00f0 con 0x7ff528101930 2024-03-18T22:51:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.011+0000 7ff51f7fe640 1 -- 172.21.15.40:0/3739831223 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff510046020 con 0x7ff528101930 2024-03-18T22:51:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.011+0000 7ff51f7fe640 1 --2- 172.21.15.40:0/3739831223 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff5140a2880 0x7ff5140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.012+0000 7ff52659c640 1 --2- 172.21.15.40:0/3739831223 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff5140a2880 0x7ff5140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.012+0000 7ff51f7fe640 1 -- 172.21.15.40:0/3739831223 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff5100f1420 con 0x7ff528101930 2024-03-18T22:51:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.012+0000 7ff52659c640 1 --2- 172.21.15.40:0/3739831223 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff5140a2880 0x7ff5140a4d40 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff5180058c0 tx=0x7ff518005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.015+0000 7ff51f7fe640 1 -- 172.21.15.40:0/3739831223 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff51004d020 con 0x7ff528101930 2024-03-18T22:51:30.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.277+0000 7ff52cb09640 1 -- 172.21.15.40:0/3739831223 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5281a02c0 con 0x7ff528101930 2024-03-18T22:51:30.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.278+0000 7ff51f7fe640 1 -- 172.21.15.40:0/3739831223 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11950 (secure 0 0 0) 0x7ff5100be0d0 con 0x7ff528101930 2024-03-18T22:51:30.279 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:51:30.279 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:51:14.215890+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24481,24509,24473,24401,24388]},"id":1}]} 2024-03-18T22:51:30.279 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 35 2024-03-18T22:51:30.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.280+0000 7ff52cb09640 1 -- 172.21.15.40:0/3739831223 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff5140a2880 msgr2=0x7ff5140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.280+0000 7ff52cb09640 1 --2- 172.21.15.40:0/3739831223 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff5140a2880 0x7ff5140a4d40 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff5180058c0 tx=0x7ff518005f90 comp rx=0 tx=0).stop 2024-03-18T22:51:30.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.280+0000 7ff52cb09640 1 -- 172.21.15.40:0/3739831223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff528101930 msgr2=0x7ff528198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.280+0000 7ff52cb09640 1 --2- 172.21.15.40:0/3739831223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff528101930 0x7ff528198010 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff510000c00 tx=0x7ff510005dc0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.280+0000 7ff52cb09640 1 -- 172.21.15.40:0/3739831223 shutdown_connections 2024-03-18T22:51:30.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.280+0000 7ff52cb09640 1 --2- 172.21.15.40:0/3739831223 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff5140a2880 0x7ff5140a4d40 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.280+0000 7ff52cb09640 1 --2- 172.21.15.40:0/3739831223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5281022f0 0x7ff52819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.280+0000 7ff52cb09640 1 --2- 172.21.15.40:0/3739831223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff528101930 0x7ff528198010 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.280+0000 7ff52cb09640 1 --2- 172.21.15.40:0/3739831223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff528100740 0x7ff528197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.280+0000 7ff52cb09640 1 -- 172.21.15.40:0/3739831223 >> 172.21.15.40:0/3739831223 conn(0x7ff5280fbcb0 msgr2=0x7ff528068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.280+0000 7ff52cb09640 1 -- 172.21.15.40:0/3739831223 shutdown_connections 2024-03-18T22:51:30.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.280+0000 7ff52cb09640 1 -- 172.21.15.40:0/3739831223 wait complete. 2024-03-18T22:51:30.293 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:51:30.409 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:30 smithi103 ceph-mon[66377]: pgmap v616: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 21 op/s 2024-03-18T22:51:30.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.411+0000 7f0025d9f640 1 -- 172.21.15.40:0/235495223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00201072f0 msgr2=0x7f0020109740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.411+0000 7f0025d9f640 1 --2- 172.21.15.40:0/235495223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00201072f0 0x7f0020109740 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f0018009ca0 tx=0x7f001802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.411+0000 7f0025d9f640 1 -- 172.21.15.40:0/235495223 shutdown_connections 2024-03-18T22:51:30.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.411+0000 7f0025d9f640 1 --2- 172.21.15.40:0/235495223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00201072f0 0x7f0020109740 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.411+0000 7f0025d9f640 1 --2- 172.21.15.40:0/235495223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00201049c0 0x7f0020106db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.411+0000 7f0025d9f640 1 --2- 172.21.15.40:0/235495223 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0020102090 0x7f0020104480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.411+0000 7f0025d9f640 1 -- 172.21.15.40:0/235495223 >> 172.21.15.40:0/235495223 conn(0x7f00200fbac0 msgr2=0x7f00200fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.411+0000 7f0025d9f640 1 -- 172.21.15.40:0/235495223 shutdown_connections 2024-03-18T22:51:30.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.411+0000 7f0025d9f640 1 -- 172.21.15.40:0/235495223 wait complete. 2024-03-18T22:51:30.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.412+0000 7f0025d9f640 1 Processor -- start 2024-03-18T22:51:30.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.412+0000 7f0025d9f640 1 -- start start 2024-03-18T22:51:30.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.412+0000 7f0025d9f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0020102090 0x7f0020197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.412+0000 7f0025d9f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00201049c0 0x7f0020197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.412+0000 7f0025d9f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00201072f0 0x7f002019ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.412+0000 7f0025d9f640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f002010d140 con 0x7f00201049c0 2024-03-18T22:51:30.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.412+0000 7f0025d9f640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f002010cfc0 con 0x7f0020102090 2024-03-18T22:51:30.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.412+0000 7f0025d9f640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f002010d2c0 con 0x7f00201072f0 2024-03-18T22:51:30.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.412+0000 7f001ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00201049c0 0x7f0020197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.412+0000 7f0024d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0020102090 0x7f0020197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.412+0000 7f001ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00201049c0 0x7f0020197cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49404/0 (socket says 172.21.15.40:49404) 2024-03-18T22:51:30.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.412+0000 7f001ffff640 1 -- 172.21.15.40:0/728202172 learned_addr learned my addr 172.21.15.40:0/728202172 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.412+0000 7f002559e640 1 --2- 172.21.15.40:0/728202172 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00201072f0 0x7f002019ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.413+0000 7f002559e640 1 -- 172.21.15.40:0/728202172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0020102090 msgr2=0x7f0020197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.413+0000 7f002559e640 1 --2- 172.21.15.40:0/728202172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0020102090 0x7f0020197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.413+0000 7f002559e640 1 -- 172.21.15.40:0/728202172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00201049c0 msgr2=0x7f0020197cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.413+0000 7f002559e640 1 --2- 172.21.15.40:0/728202172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00201049c0 0x7f0020197cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.413+0000 7f002559e640 1 -- 172.21.15.40:0/728202172 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f002019f3a0 con 0x7f00201072f0 2024-03-18T22:51:30.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.413+0000 7f0024d9d640 1 --2- 172.21.15.40:0/728202172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0020102090 0x7f0020197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.413+0000 7f001ffff640 1 --2- 172.21.15.40:0/728202172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00201049c0 0x7f0020197cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.413+0000 7f002559e640 1 --2- 172.21.15.40:0/728202172 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00201072f0 0x7f002019ed30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f001802c630 tx=0x7f00180328a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.413+0000 7f001dffb640 1 -- 172.21.15.40:0/728202172 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0018041020 con 0x7f00201072f0 2024-03-18T22:51:30.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.413+0000 7f0025d9f640 1 -- 172.21.15.40:0/728202172 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00200ffe80 con 0x7f00201072f0 2024-03-18T22:51:30.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.413+0000 7f001dffb640 1 -- 172.21.15.40:0/728202172 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0018003e20 con 0x7f00201072f0 2024-03-18T22:51:30.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.413+0000 7f001dffb640 1 -- 172.21.15.40:0/728202172 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0018039040 con 0x7f00201072f0 2024-03-18T22:51:30.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.413+0000 7f0025d9f640 1 -- 172.21.15.40:0/728202172 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00201003c0 con 0x7f00201072f0 2024-03-18T22:51:30.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.416+0000 7f001dffb640 1 -- 172.21.15.40:0/728202172 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0018044020 con 0x7f00201072f0 2024-03-18T22:51:30.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.416+0000 7f0029be0740 1 Processor -- start 2024-03-18T22:51:30.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.416+0000 7f0029be0740 1 -- start start 2024-03-18T22:51:30.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.416+0000 7f0029be0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74d4d0 0x55c25e74d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.416+0000 7f0029be0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 0x55c25e7529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.416+0000 7f0029be0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e753030 0x55c25e7554c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.416+0000 7f0029be0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c25e6b14b0 con 0x55c25e753030 2024-03-18T22:51:30.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.416+0000 7f0029be0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c25e659d70 con 0x55c25e74dec0 2024-03-18T22:51:30.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.416+0000 7f0029be0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c25e6ad550 con 0x55c25e74d4d0 2024-03-18T22:51:30.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.416+0000 7f001ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e753030 0x55c25e7554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.416+0000 7f0024d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 0x55c25e7529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.416+0000 7f001ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e753030 0x55c25e7554c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49410/0 (socket says 172.21.15.40:49410) 2024-03-18T22:51:30.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.416+0000 7f001ffff640 1 -- 172.21.15.40:0/1892274446 learned_addr learned my addr 172.21.15.40:0/1892274446 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.417+0000 7f001ffff640 1 --2- 172.21.15.40:0/1892274446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74d4d0 0x55c25e74d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.417+0000 7f001dffb640 1 --2- 172.21.15.40:0/728202172 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00100a2800 0x7f00100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.417+0000 7f0024d9d640 1 -- 172.21.15.40:0/1892274446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74d4d0 msgr2=0x55c25e74d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.417+0000 7f0024d9d640 1 --2- 172.21.15.40:0/1892274446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74d4d0 0x55c25e74d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.417+0000 7f0024d9d640 1 -- 172.21.15.40:0/1892274446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e753030 msgr2=0x55c25e7554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.417+0000 7f0024d9d640 1 --2- 172.21.15.40:0/1892274446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e753030 0x55c25e7554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.417+0000 7f0024d9d640 1 -- 172.21.15.40:0/1892274446 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c25e67c210 con 0x55c25e74dec0 2024-03-18T22:51:30.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.417+0000 7f001ffff640 1 --2- 172.21.15.40:0/1892274446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e753030 0x55c25e7554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.417+0000 7f002559e640 1 --2- 172.21.15.40:0/728202172 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00100a2800 0x7f00100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.417+0000 7f001dffb640 1 -- 172.21.15.40:0/728202172 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f00180f0fe0 con 0x7f00201072f0 2024-03-18T22:51:30.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.417+0000 7f0024d9d640 1 --2- 172.21.15.40:0/1892274446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 0x55c25e7529c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0014010f70 tx=0x7f0014005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=774d8a3c2a28ad9c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.417+0000 7f000affd640 1 -- 172.21.15.40:0/1892274446 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001401e070 con 0x55c25e74dec0 2024-03-18T22:51:30.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.417+0000 7f000affd640 1 -- 172.21.15.40:0/1892274446 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00140054d0 con 0x55c25e74dec0 2024-03-18T22:51:30.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.417+0000 7f000affd640 1 -- 172.21.15.40:0/1892274446 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0014004d10 con 0x55c25e74dec0 2024-03-18T22:51:30.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.418+0000 7f0029be0740 1 -- 172.21.15.40:0/1892274446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 msgr2=0x55c25e7529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.418+0000 7f002559e640 1 --2- 172.21.15.40:0/728202172 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00100a2800 0x7f00100a4cc0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f00180b99f0 tx=0x7f0018088350 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.418+0000 7f0029be0740 1 --2- 172.21.15.40:0/1892274446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 0x55c25e7529c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0014010f70 tx=0x7f0014005f90 comp rx=0 tx=0).stop 2024-03-18T22:51:30.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.418+0000 7f0029be0740 1 -- 172.21.15.40:0/1892274446 shutdown_connections 2024-03-18T22:51:30.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.418+0000 7f0029be0740 1 --2- 172.21.15.40:0/1892274446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e753030 0x55c25e7554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.418+0000 7f0029be0740 1 --2- 172.21.15.40:0/1892274446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 0x55c25e7529c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.418+0000 7f0029be0740 1 --2- 172.21.15.40:0/1892274446 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74d4d0 0x55c25e74d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.418+0000 7f0029be0740 1 -- 172.21.15.40:0/1892274446 >> 172.21.15.40:0/1892274446 conn(0x55c25e748a40 msgr2=0x55c25e748e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.418+0000 7f0029be0740 1 -- 172.21.15.40:0/1892274446 shutdown_connections 2024-03-18T22:51:30.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.418+0000 7f0029be0740 1 -- 172.21.15.40:0/1892274446 wait complete. 2024-03-18T22:51:30.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.418+0000 7f0029be0740 1 -- start start 2024-03-18T22:51:30.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.418+0000 7f0029be0740 1 Processor -- start 2024-03-18T22:51:30.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f0029be0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e74d4d0 0x55c25e7579b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f0029be0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74dec0 0x55c25e757ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f0029be0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e753030 0x55c25e75ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f0029be0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c25e755fe0 con 0x55c25e74d4d0 2024-03-18T22:51:30.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f0029be0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c25e66d310 con 0x55c25e753030 2024-03-18T22:51:30.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f0029be0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c25e756160 con 0x55c25e74dec0 2024-03-18T22:51:30.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f0029be0740 10 client.0 _ll_register_callbacks cb 0x55c25e6612d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:30.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f001ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e74d4d0 0x55c25e7579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f0024d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74dec0 0x55c25e757ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f0024d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74dec0 0x55c25e757ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53850/0 (socket says 172.21.15.40:53850) 2024-03-18T22:51:30.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f0024d9d640 1 -- 172.21.15.40:0/2399121855 learned_addr learned my addr 172.21.15.40:0/2399121855 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f001ffff640 1 --2- 172.21.15.40:0/2399121855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e753030 0x55c25e75ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f0024d9d640 1 -- 172.21.15.40:0/2399121855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e753030 msgr2=0x55c25e75ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f0024d9d640 1 --2- 172.21.15.40:0/2399121855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e753030 0x55c25e75ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f0024d9d640 1 -- 172.21.15.40:0/2399121855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e74d4d0 msgr2=0x55c25e7579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f0024d9d640 1 --2- 172.21.15.40:0/2399121855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e74d4d0 0x55c25e7579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f0024d9d640 1 -- 172.21.15.40:0/2399121855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c25e3c53f0 con 0x55c25e74dec0 2024-03-18T22:51:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.419+0000 7f001ffff640 1 --2- 172.21.15.40:0/2399121855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e74d4d0 0x55c25e7579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7f0024d9d640 1 --2- 172.21.15.40:0/2399121855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74dec0 0x55c25e757ef0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0014010f40 tx=0x7f001400bd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be23f468dc793194 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7efffffff640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:51:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7efffffff640 1 -- 172.21.15.40:0/2399121855 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0014029070 con 0x55c25e74dec0 2024-03-18T22:51:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7efffffff640 1 -- 172.21.15.40:0/2399121855 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0014004e60 con 0x55c25e74dec0 2024-03-18T22:51:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7f0029be0740 1 -- 172.21.15.40:0/2399121855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00140054d0 con 0x55c25e74dec0 2024-03-18T22:51:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7efffffff640 1 -- 172.21.15.40:0/2399121855 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0014035050 con 0x55c25e74dec0 2024-03-18T22:51:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7f0029be0740 10 client.27595 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7f0029be0740 1 -- 172.21.15.40:0/2399121855 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c25e48f7d0 con 0x55c25e74dec0 2024-03-18T22:51:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7efffffff640 1 -- 172.21.15.40:0/2399121855 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00140054d0 con 0x55c25e74dec0 2024-03-18T22:51:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7efffffff640 1 -- 172.21.15.40:0/2399121855 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f00140258e0 con 0x55c25e74dec0 2024-03-18T22:51:30.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7f0029be0740 10 client.27595 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7f0029be0740 10 client.27595 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:51:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7f0029be0740 20 client.27595 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7f0029be0740 1 --2- 172.21.15.40:0/2399121855 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55c25e763420 0x55c25e765830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7f0029be0740 4 client.27595 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:51:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.420+0000 7f0029be0740 1 -- 172.21.15.40:0/2399121855 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c25e70f3e0 con 0x55c25e763420 2024-03-18T22:51:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.421+0000 7f001ffff640 1 --2- 172.21.15.40:0/2399121855 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55c25e763420 0x55c25e765830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.421+0000 7efffffff640 10 client.27595 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:51:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.421+0000 7f001ffff640 1 --2- 172.21.15.40:0/2399121855 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55c25e763420 0x55c25e765830 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8fb7b8da8eb59d9f server_cookie=ab02da50e48d4b05 in_seq=0 out_seq=0 2024-03-18T22:51:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.422+0000 7efffffff640 1 -- 172.21.15.40:0/2399121855 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c25e70f3e0 con 0x55c25e763420 2024-03-18T22:51:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.422+0000 7efffffff640 10 client.27595 handle_command_reply: tid=0 2024-03-18T22:51:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.422+0000 7f0029be0740 1 client.27595 shutdown 2024-03-18T22:51:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.422+0000 7f0029be0740 1 -- 172.21.15.40:0/2399121855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74dec0 msgr2=0x55c25e757ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.422+0000 7f0029be0740 1 --2- 172.21.15.40:0/2399121855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74dec0 0x55c25e757ef0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0014010f40 tx=0x7f001400bd50 comp rx=0 tx=0).stop 2024-03-18T22:51:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.422+0000 7f0029be0740 1 -- 172.21.15.40:0/2399121855 shutdown_connections 2024-03-18T22:51:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.422+0000 7f0029be0740 1 --2- 172.21.15.40:0/2399121855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e753030 0x55c25e75ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.422+0000 7f0029be0740 1 --2- 172.21.15.40:0/2399121855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74dec0 0x55c25e757ef0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.422+0000 7f0029be0740 1 --2- 172.21.15.40:0/2399121855 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55c25e763420 0x55c25e765830 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.422+0000 7f0029be0740 1 --2- 172.21.15.40:0/2399121855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e74d4d0 0x55c25e7579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.422+0000 7efffffff640 0 client.27595 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:51:30.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.422+0000 7f0029be0740 1 -- 172.21.15.40:0/2399121855 >> 172.21.15.40:0/2399121855 conn(0x55c25e748a40 msgr2=0x55c25e747d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.422+0000 7f0029be0740 1 -- 172.21.15.40:0/2399121855 shutdown_connections 2024-03-18T22:51:30.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.422+0000 7f0029be0740 1 -- 172.21.15.40:0/2399121855 wait complete. 2024-03-18T22:51:30.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.422+0000 7f0029be0740 20 client.27595 trim_cache size 0 max 16384 2024-03-18T22:51:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0029be0740 1 Processor -- start 2024-03-18T22:51:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0029be0740 1 -- start start 2024-03-18T22:51:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0029be0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e748a40 0x55c25e756fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0029be0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e74d4d0 0x55c25e75eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0029be0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 0x55c25e747470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0029be0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f00201a8090 con 0x55c25e74d4d0 2024-03-18T22:51:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0029be0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f00201a9c30 con 0x55c25e74dec0 2024-03-18T22:51:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0029be0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f00201a8760 con 0x55c25e748a40 2024-03-18T22:51:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0024d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e74d4d0 0x55c25e75eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0024d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e74d4d0 0x55c25e75eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49428/0 (socket says 172.21.15.40:49428) 2024-03-18T22:51:30.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0024d9d640 1 -- 172.21.15.40:0/1799536205 learned_addr learned my addr 172.21.15.40:0/1799536205 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f001ffff640 1 --2- 172.21.15.40:0/1799536205 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e748a40 0x55c25e756fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f001ffff640 1 --2- 172.21.15.40:0/1799536205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 0x55c25e747470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0024d9d640 1 -- 172.21.15.40:0/1799536205 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e748a40 msgr2=0x55c25e756fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0024d9d640 1 --2- 172.21.15.40:0/1799536205 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e748a40 0x55c25e756fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0024d9d640 1 -- 172.21.15.40:0/1799536205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 msgr2=0x55c25e747470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0024d9d640 1 --2- 172.21.15.40:0/1799536205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 0x55c25e747470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0024d9d640 1 -- 172.21.15.40:0/1799536205 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c25e74fc50 con 0x55c25e74d4d0 2024-03-18T22:51:30.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f001ffff640 1 --2- 172.21.15.40:0/1799536205 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e748a40 0x55c25e756fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f0024d9d640 1 --2- 172.21.15.40:0/1799536205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e74d4d0 0x55c25e75eef0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f0014025d80 tx=0x7f0014010e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba6bd348904bf8de server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.438+0000 7f000a7fc640 1 -- 172.21.15.40:0/1799536205 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0014029070 con 0x55c25e74d4d0 2024-03-18T22:51:30.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.439+0000 7f000a7fc640 1 -- 172.21.15.40:0/1799536205 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00140258e0 con 0x55c25e74d4d0 2024-03-18T22:51:30.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.439+0000 7f000a7fc640 1 -- 172.21.15.40:0/1799536205 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001401e040 con 0x55c25e74d4d0 2024-03-18T22:51:30.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.439+0000 7f0029be0740 1 -- 172.21.15.40:0/1799536205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e74d4d0 msgr2=0x55c25e75eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.439+0000 7f0029be0740 1 --2- 172.21.15.40:0/1799536205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e74d4d0 0x55c25e75eef0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f0014025d80 tx=0x7f0014010e40 comp rx=0 tx=0).stop 2024-03-18T22:51:30.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.439+0000 7f0029be0740 1 -- 172.21.15.40:0/1799536205 shutdown_connections 2024-03-18T22:51:30.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.439+0000 7f0029be0740 1 --2- 172.21.15.40:0/1799536205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 0x55c25e747470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.439+0000 7f0029be0740 1 --2- 172.21.15.40:0/1799536205 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e74d4d0 0x55c25e75eef0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.439+0000 7f0029be0740 1 --2- 172.21.15.40:0/1799536205 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e748a40 0x55c25e756fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.439+0000 7f0029be0740 1 -- 172.21.15.40:0/1799536205 >> 172.21.15.40:0/1799536205 conn(0x55c25e74eca0 msgr2=0x55c25e7579b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.439+0000 7f0029be0740 1 -- 172.21.15.40:0/1799536205 shutdown_connections 2024-03-18T22:51:30.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.439+0000 7f0029be0740 1 -- 172.21.15.40:0/1799536205 wait complete. 2024-03-18T22:51:30.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.440+0000 7f0029be0740 1 -- start start 2024-03-18T22:51:30.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.440+0000 7f0029be0740 1 Processor -- start 2024-03-18T22:51:30.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.440+0000 7f0029be0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e748a40 0x55c25e75cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.440+0000 7f0029be0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74d4d0 0x55c25e75d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.440+0000 7f0029be0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 0x55c25e75e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.440+0000 7f0029be0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f00201a9560 con 0x55c25e748a40 2024-03-18T22:51:30.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.440+0000 7f0029be0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f00201a8e30 con 0x55c25e74dec0 2024-03-18T22:51:30.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.440+0000 7f0029be0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f00201a6070 con 0x55c25e74d4d0 2024-03-18T22:51:30.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.440+0000 7f0029be0740 10 client.0 _ll_register_callbacks cb 0x7f000c009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:30.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.440+0000 7f001ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e748a40 0x55c25e75cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.440+0000 7f001ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e748a40 0x55c25e75cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49438/0 (socket says 172.21.15.40:49438) 2024-03-18T22:51:30.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.440+0000 7f001ffff640 1 -- 172.21.15.40:0/3155602069 learned_addr learned my addr 172.21.15.40:0/3155602069 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.440+0000 7f0024d9d640 1 --2- 172.21.15.40:0/3155602069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74d4d0 0x55c25e75d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.440+0000 7f001ffff640 1 --2- 172.21.15.40:0/3155602069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 0x55c25e75e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.441+0000 7f001ffff640 1 -- 172.21.15.40:0/3155602069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74d4d0 msgr2=0x55c25e75d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.441+0000 7f001ffff640 1 --2- 172.21.15.40:0/3155602069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74d4d0 0x55c25e75d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.441+0000 7f001ffff640 1 -- 172.21.15.40:0/3155602069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 msgr2=0x55c25e75e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.441+0000 7f001ffff640 1 --2- 172.21.15.40:0/3155602069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 0x55c25e75e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.441+0000 7f001ffff640 1 -- 172.21.15.40:0/3155602069 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c25e75e6c0 con 0x55c25e748a40 2024-03-18T22:51:30.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.441+0000 7f001ffff640 1 --2- 172.21.15.40:0/3155602069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e748a40 0x55c25e75cbd0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f000c016010 tx=0x7f000c028040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84928ca150555d11 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.441+0000 7efffffff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:51:30.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.441+0000 7efffffff640 1 -- 172.21.15.40:0/3155602069 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000c014e60 con 0x55c25e748a40 2024-03-18T22:51:30.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.441+0000 7efffffff640 1 -- 172.21.15.40:0/3155602069 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f000c005220 con 0x55c25e748a40 2024-03-18T22:51:30.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.441+0000 7f0029be0740 1 -- 172.21.15.40:0/3155602069 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00140258e0 con 0x55c25e748a40 2024-03-18T22:51:30.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.441+0000 7efffffff640 1 -- 172.21.15.40:0/3155602069 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f000c01a040 con 0x55c25e748a40 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.441+0000 7f0029be0740 10 client.17796 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.441+0000 7f0029be0740 1 -- 172.21.15.40:0/3155602069 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c25e7c9d30 con 0x55c25e748a40 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.441+0000 7efffffff640 1 -- 172.21.15.40:0/3155602069 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00140258e0 con 0x55c25e748a40 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.442+0000 7efffffff640 1 -- 172.21.15.40:0/3155602069 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f000c031030 con 0x55c25e748a40 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.442+0000 7f0029be0740 10 client.17796 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.442+0000 7f0029be0740 10 client.17796 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.442+0000 7f0029be0740 20 client.17796 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.442+0000 7f0029be0740 1 --2- 172.21.15.40:0/3155602069 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55c25e753030 0x55c25e7cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.442+0000 7f0029be0740 4 client.17796 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.442+0000 7f0029be0740 1 -- 172.21.15.40:0/3155602069 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c25e7577a0 con 0x55c25e753030 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.442+0000 7f0024d9d640 1 --2- 172.21.15.40:0/3155602069 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55c25e753030 0x55c25e7cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.442+0000 7efffffff640 10 client.17796 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.442+0000 7f0024d9d640 1 --2- 172.21.15.40:0/3155602069 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55c25e753030 0x55c25e7cc6f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b0c8caa859c190bc server_cookie=1416b6ceee1d1b18 in_seq=0 out_seq=0 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.442+0000 7efffffff640 1 -- 172.21.15.40:0/3155602069 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c25e7577a0 con 0x55c25e753030 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.443+0000 7efffffff640 10 client.17796 handle_command_reply: tid=0 2024-03-18T22:51:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.443+0000 7f0029be0740 1 client.17796 shutdown 2024-03-18T22:51:30.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.443+0000 7f0029be0740 1 -- 172.21.15.40:0/3155602069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e748a40 msgr2=0x55c25e75cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.443+0000 7f0029be0740 1 --2- 172.21.15.40:0/3155602069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e748a40 0x55c25e75cbd0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f000c016010 tx=0x7f000c028040 comp rx=0 tx=0).stop 2024-03-18T22:51:30.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.443+0000 7f0029be0740 1 -- 172.21.15.40:0/3155602069 shutdown_connections 2024-03-18T22:51:30.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.443+0000 7f0029be0740 1 --2- 172.21.15.40:0/3155602069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c25e74dec0 0x55c25e75e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.443+0000 7f0029be0740 1 --2- 172.21.15.40:0/3155602069 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c25e74d4d0 0x55c25e75d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.443+0000 7f0029be0740 1 --2- 172.21.15.40:0/3155602069 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55c25e753030 0x55c25e7cc6f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.443+0000 7f0029be0740 1 --2- 172.21.15.40:0/3155602069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c25e748a40 0x55c25e75cbd0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.443+0000 7f0029be0740 1 -- 172.21.15.40:0/3155602069 >> 172.21.15.40:0/3155602069 conn(0x55c25e74eca0 msgr2=0x55c25e7579b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.443+0000 7efffffff640 0 client.17796 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:51:30.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.443+0000 7f0029be0740 1 -- 172.21.15.40:0/3155602069 shutdown_connections 2024-03-18T22:51:30.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.443+0000 7f0029be0740 1 -- 172.21.15.40:0/3155602069 wait complete. 2024-03-18T22:51:30.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.443+0000 7f0029be0740 20 client.17796 trim_cache size 0 max 16384 2024-03-18T22:51:30.450 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:51:30.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.444+0000 7f0025d9f640 1 -- 172.21.15.40:0/728202172 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00100a2800 msgr2=0x7f00100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.444+0000 7f0025d9f640 1 --2- 172.21.15.40:0/728202172 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00100a2800 0x7f00100a4cc0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f00180b99f0 tx=0x7f0018088350 comp rx=0 tx=0).stop 2024-03-18T22:51:30.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.444+0000 7f0025d9f640 1 -- 172.21.15.40:0/728202172 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00201072f0 msgr2=0x7f002019ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.444+0000 7f0025d9f640 1 --2- 172.21.15.40:0/728202172 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00201072f0 0x7f002019ed30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f001802c630 tx=0x7f00180328a0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.444+0000 7f0025d9f640 1 -- 172.21.15.40:0/728202172 shutdown_connections 2024-03-18T22:51:30.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.444+0000 7f0025d9f640 1 --2- 172.21.15.40:0/728202172 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f00100a2800 0x7f00100a4cc0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.444+0000 7f0025d9f640 1 --2- 172.21.15.40:0/728202172 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f00201072f0 0x7f002019ed30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.444+0000 7f0025d9f640 1 --2- 172.21.15.40:0/728202172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00201049c0 0x7f0020197cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.444+0000 7f0025d9f640 1 --2- 172.21.15.40:0/728202172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0020102090 0x7f0020197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.444+0000 7f0025d9f640 1 -- 172.21.15.40:0/728202172 >> 172.21.15.40:0/728202172 conn(0x7f00200fbac0 msgr2=0x7f00201029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.444+0000 7f0025d9f640 1 -- 172.21.15.40:0/728202172 shutdown_connections 2024-03-18T22:51:30.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.444+0000 7f0025d9f640 1 -- 172.21.15.40:0/728202172 wait complete. 2024-03-18T22:51:30.454 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:30 smithi040 ceph-mon[62643]: pgmap v616: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 21 op/s 2024-03-18T22:51:30.455 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:51:30.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.566+0000 7fbffd4c6640 1 -- 172.21.15.40:0/2116599892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff8101860 msgr2=0x7fbff8101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.566+0000 7fbffd4c6640 1 --2- 172.21.15.40:0/2116599892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff8101860 0x7fbff8101c80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8009900 tx=0x7fbfe802bc30 comp rx=0 tx=0).stop 2024-03-18T22:51:30.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.566+0000 7fbffd4c6640 1 -- 172.21.15.40:0/2116599892 shutdown_connections 2024-03-18T22:51:30.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.566+0000 7fbffd4c6640 1 --2- 172.21.15.40:0/2116599892 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbff8103410 0x7fbff8109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.566+0000 7fbffd4c6640 1 --2- 172.21.15.40:0/2116599892 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff8102a50 0x7fbff8102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.566+0000 7fbffd4c6640 1 --2- 172.21.15.40:0/2116599892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff8101860 0x7fbff8101c80 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.566+0000 7fbffd4c6640 1 -- 172.21.15.40:0/2116599892 >> 172.21.15.40:0/2116599892 conn(0x7fbff80fcdb0 msgr2=0x7fbff80ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.566+0000 7fbffd4c6640 1 -- 172.21.15.40:0/2116599892 shutdown_connections 2024-03-18T22:51:30.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.566+0000 7fbffd4c6640 1 -- 172.21.15.40:0/2116599892 wait complete. 2024-03-18T22:51:30.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.567+0000 7fbffd4c6640 1 Processor -- start 2024-03-18T22:51:30.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.567+0000 7fbffd4c6640 1 -- start start 2024-03-18T22:51:30.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.567+0000 7fbffd4c6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff8101860 0x7fbff80755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.567+0000 7fbffd4c6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbff8102a50 0x7fbff8073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.567+0000 7fbffd4c6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff8103410 0x7fbff8074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.567+0000 7fbffd4c6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbff810d390 con 0x7fbff8103410 2024-03-18T22:51:30.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.567+0000 7fbffd4c6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbff810d210 con 0x7fbff8101860 2024-03-18T22:51:30.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbffd4c6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbff810d510 con 0x7fbff8102a50 2024-03-18T22:51:30.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbff77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff8103410 0x7fbff8074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbff6ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff8101860 0x7fbff80755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbff6ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff8101860 0x7fbff80755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:35346/0 (socket says 172.21.15.40:35346) 2024-03-18T22:51:30.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbff77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff8103410 0x7fbff8074150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49456/0 (socket says 172.21.15.40:49456) 2024-03-18T22:51:30.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbff6ffd640 1 -- 172.21.15.40:0/2366959186 learned_addr learned my addr 172.21.15.40:0/2366959186 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbff67fc640 1 --2- 172.21.15.40:0/2366959186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbff8102a50 0x7fbff8073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbff77fe640 1 -- 172.21.15.40:0/2366959186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbff8102a50 msgr2=0x7fbff8073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbff77fe640 1 --2- 172.21.15.40:0/2366959186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbff8102a50 0x7fbff8073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbff77fe640 1 -- 172.21.15.40:0/2366959186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff8101860 msgr2=0x7fbff80755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbff77fe640 1 --2- 172.21.15.40:0/2366959186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff8101860 0x7fbff80755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbff77fe640 1 -- 172.21.15.40:0/2366959186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff80749e0 con 0x7fbff8103410 2024-03-18T22:51:30.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbff67fc640 1 --2- 172.21.15.40:0/2366959186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbff8102a50 0x7fbff8073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbff6ffd640 1 --2- 172.21.15.40:0/2366959186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff8101860 0x7fbff80755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbff77fe640 1 --2- 172.21.15.40:0/2366959186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff8103410 0x7fbff8074150 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009fd0 tx=0x7fbff000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbfdffff640 1 -- 172.21.15.40:0/2366959186 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff000ce20 con 0x7fbff8103410 2024-03-18T22:51:30.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbfdffff640 1 -- 172.21.15.40:0/2366959186 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbff0005020 con 0x7fbff8103410 2024-03-18T22:51:30.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbffd4c6640 1 -- 172.21.15.40:0/2366959186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff81a1bb0 con 0x7fbff8103410 2024-03-18T22:51:30.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbfdffff640 1 -- 172.21.15.40:0/2366959186 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff0010690 con 0x7fbff8103410 2024-03-18T22:51:30.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.568+0000 7fbffd4c6640 1 -- 172.21.15.40:0/2366959186 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff81a2100 con 0x7fbff8103410 2024-03-18T22:51:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.571+0000 7fc00018c740 1 Processor -- start 2024-03-18T22:51:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.571+0000 7fbfdffff640 1 -- 172.21.15.40:0/2366959186 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbff00051d0 con 0x7fbff8103410 2024-03-18T22:51:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.571+0000 7fc00018c740 1 -- start start 2024-03-18T22:51:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.571+0000 7fc00018c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab2012f0 0x559dab201710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.571+0000 7fc00018c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab201ce0 0x559dab2067e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.571+0000 7fc00018c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab206e50 0x559dab2092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.571+0000 7fc00018c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559dab1654b0 con 0x559dab206e50 2024-03-18T22:51:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.571+0000 7fc00018c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559dab10dd70 con 0x559dab2012f0 2024-03-18T22:51:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.571+0000 7fc00018c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559dab161550 con 0x559dab201ce0 2024-03-18T22:51:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.571+0000 7fbff67fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab206e50 0x559dab2092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.571+0000 7fbff67fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab206e50 0x559dab2092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49468/0 (socket says 172.21.15.40:49468) 2024-03-18T22:51:30.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.571+0000 7fbff6ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab201ce0 0x559dab2067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.571+0000 7fbff67fc640 1 -- 172.21.15.40:0/1486668114 learned_addr learned my addr 172.21.15.40:0/1486668114 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.572+0000 7fbff67fc640 1 --2- 172.21.15.40:0/1486668114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab2012f0 0x559dab201710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.572+0000 7fbff6ffd640 1 -- 172.21.15.40:0/1486668114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab2012f0 msgr2=0x559dab201710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.572+0000 7fbff6ffd640 1 --2- 172.21.15.40:0/1486668114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab2012f0 0x559dab201710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.572+0000 7fbff6ffd640 1 -- 172.21.15.40:0/1486668114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab206e50 msgr2=0x559dab2092e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.572+0000 7fbff6ffd640 1 --2- 172.21.15.40:0/1486668114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab206e50 0x559dab2092e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.572+0000 7fbff6ffd640 1 -- 172.21.15.40:0/1486668114 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dab130210 con 0x559dab201ce0 2024-03-18T22:51:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.572+0000 7fbff67fc640 1 --2- 172.21.15.40:0/1486668114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab206e50 0x559dab2092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.572+0000 7fbfdffff640 1 --2- 172.21.15.40:0/2366959186 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbfe00a2800 0x7fbfe00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.572+0000 7fbff6ffd640 1 --2- 172.21.15.40:0/1486668114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab201ce0 0x559dab2067e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8006010 tx=0x7fbfe8009660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4a66a8bc0a3eb7f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.572+0000 7fbff67fc640 1 --2- 172.21.15.40:0/2366959186 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbfe00a2800 0x7fbfe00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.572+0000 7fbfdffff640 1 -- 172.21.15.40:0/2366959186 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fbff00cf540 con 0x7fbff8103410 2024-03-18T22:51:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.572+0000 7fbfdd7fa640 1 -- 172.21.15.40:0/1486668114 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8045020 con 0x559dab201ce0 2024-03-18T22:51:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.572+0000 7fbfdd7fa640 1 -- 172.21.15.40:0/1486668114 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfe802cc90 con 0x559dab201ce0 2024-03-18T22:51:30.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.572+0000 7fbfdd7fa640 1 -- 172.21.15.40:0/1486668114 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8004920 con 0x559dab201ce0 2024-03-18T22:51:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.573+0000 7fc00018c740 1 -- 172.21.15.40:0/1486668114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab201ce0 msgr2=0x559dab2067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.573+0000 7fbff67fc640 1 --2- 172.21.15.40:0/2366959186 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbfe00a2800 0x7fbfe00a4cc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x559dab202540 tx=0x7fbfec009ba0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.573+0000 7fc00018c740 1 --2- 172.21.15.40:0/1486668114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab201ce0 0x559dab2067e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8006010 tx=0x7fbfe8009660 comp rx=0 tx=0).stop 2024-03-18T22:51:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.573+0000 7fc00018c740 1 -- 172.21.15.40:0/1486668114 shutdown_connections 2024-03-18T22:51:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.573+0000 7fc00018c740 1 --2- 172.21.15.40:0/1486668114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab206e50 0x559dab2092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.573+0000 7fc00018c740 1 --2- 172.21.15.40:0/1486668114 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab201ce0 0x559dab2067e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.573+0000 7fc00018c740 1 --2- 172.21.15.40:0/1486668114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab2012f0 0x559dab201710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.573+0000 7fc00018c740 1 -- 172.21.15.40:0/1486668114 >> 172.21.15.40:0/1486668114 conn(0x559dab1fc860 msgr2=0x559dab1fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.573+0000 7fc00018c740 1 -- 172.21.15.40:0/1486668114 shutdown_connections 2024-03-18T22:51:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.573+0000 7fc00018c740 1 -- 172.21.15.40:0/1486668114 wait complete. 2024-03-18T22:51:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.573+0000 7fc00018c740 1 -- start start 2024-03-18T22:51:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.573+0000 7fc00018c740 1 Processor -- start 2024-03-18T22:51:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fc00018c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab2012f0 0x559dab20b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fc00018c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab201ce0 0x559dab20bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fc00018c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab206e50 0x559dab212d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fc00018c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559dab209da0 con 0x559dab2012f0 2024-03-18T22:51:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fc00018c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559dab121310 con 0x559dab206e50 2024-03-18T22:51:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fc00018c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559dab209f20 con 0x559dab201ce0 2024-03-18T22:51:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fbff6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab2012f0 0x559dab20b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fbff6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab2012f0 0x559dab20b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49478/0 (socket says 172.21.15.40:49478) 2024-03-18T22:51:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fbff6ffd640 1 -- 172.21.15.40:0/1498037097 learned_addr learned my addr 172.21.15.40:0/1498037097 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fc00018c740 10 client.0 _ll_register_callbacks cb 0x559dab1152d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fbff67fc640 1 --2- 172.21.15.40:0/1498037097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab201ce0 0x559dab20bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fbff77fe640 1 --2- 172.21.15.40:0/1498037097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab206e50 0x559dab212d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fbff6ffd640 1 -- 172.21.15.40:0/1498037097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab201ce0 msgr2=0x559dab20bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fbff6ffd640 1 --2- 172.21.15.40:0/1498037097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab201ce0 0x559dab20bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fbff6ffd640 1 -- 172.21.15.40:0/1498037097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab206e50 msgr2=0x559dab212d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fbff6ffd640 1 --2- 172.21.15.40:0/1498037097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab206e50 0x559dab212d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fbff6ffd640 1 -- 172.21.15.40:0/1498037097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559daae793f0 con 0x559dab2012f0 2024-03-18T22:51:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fbff6ffd640 1 --2- 172.21.15.40:0/1498037097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab2012f0 0x559dab20b7b0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8004630 tx=0x7fbfe802c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bcac80bccfc55a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fbfdaff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:51:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.574+0000 7fbfdaff5640 1 -- 172.21.15.40:0/1498037097 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8045040 con 0x559dab2012f0 2024-03-18T22:51:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.575+0000 7fbfdaff5640 1 -- 172.21.15.40:0/1498037097 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fbfe8003b40 con 0x559dab2012f0 2024-03-18T22:51:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.575+0000 7fc00018c740 1 -- 172.21.15.40:0/1498037097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfe802cc90 con 0x559dab2012f0 2024-03-18T22:51:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.575+0000 7fbfdaff5640 1 -- 172.21.15.40:0/1498037097 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbfe8058050 con 0x559dab2012f0 2024-03-18T22:51:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.575+0000 7fc00018c740 10 client.17808 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.575+0000 7fc00018c740 1 -- 172.21.15.40:0/1498037097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559daaf437d0 con 0x559dab2012f0 2024-03-18T22:51:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.575+0000 7fbfdaff5640 1 -- 172.21.15.40:0/1498037097 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfe802cc90 con 0x559dab2012f0 2024-03-18T22:51:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.575+0000 7fbfdaff5640 1 -- 172.21.15.40:0/1498037097 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7fbfe8053440 con 0x559dab2012f0 2024-03-18T22:51:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.575+0000 7fc00018c740 10 client.17808 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.575+0000 7fc00018c740 10 client.17808 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:51:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.575+0000 7fc00018c740 20 client.17808 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.575+0000 7fc00018c740 1 --2- 172.21.15.40:0/1498037097 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559dab217200 0x559dab2195f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.575+0000 7fc00018c740 4 client.17808 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:51:30.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.575+0000 7fc00018c740 1 -- 172.21.15.40:0/1498037097 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559dab1c33e0 con 0x559dab217200 2024-03-18T22:51:30.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.575+0000 7fbff67fc640 1 --2- 172.21.15.40:0/1498037097 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559dab217200 0x559dab2195f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.576+0000 7fbff67fc640 1 --2- 172.21.15.40:0/1498037097 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559dab217200 0x559dab2195f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e013636b317e35e9 server_cookie=da42cd3946572f7e in_seq=0 out_seq=0 2024-03-18T22:51:30.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.576+0000 7fbfdaff5640 10 client.17808 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:51:30.582 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:30 smithi113 ceph-mon[66775]: pgmap v616: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 21 op/s 2024-03-18T22:51:30.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.577+0000 7fbfdaff5640 1 -- 172.21.15.40:0/1498037097 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559dab1c33e0 con 0x559dab217200 2024-03-18T22:51:30.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.577+0000 7fbfdaff5640 10 client.17808 handle_command_reply: tid=0 2024-03-18T22:51:30.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.577+0000 7fc00018c740 1 client.17808 shutdown 2024-03-18T22:51:30.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.577+0000 7fc00018c740 1 -- 172.21.15.40:0/1498037097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab2012f0 msgr2=0x559dab20b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.577+0000 7fc00018c740 1 --2- 172.21.15.40:0/1498037097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab2012f0 0x559dab20b7b0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8004630 tx=0x7fbfe802c3d0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.577+0000 7fc00018c740 1 -- 172.21.15.40:0/1498037097 shutdown_connections 2024-03-18T22:51:30.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.577+0000 7fc00018c740 1 --2- 172.21.15.40:0/1498037097 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559dab217200 0x559dab2195f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.577+0000 7fc00018c740 1 --2- 172.21.15.40:0/1498037097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab206e50 0x559dab212d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.577+0000 7fc00018c740 1 --2- 172.21.15.40:0/1498037097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab201ce0 0x559dab20bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.577+0000 7fc00018c740 1 --2- 172.21.15.40:0/1498037097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab2012f0 0x559dab20b7b0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.577+0000 7fc00018c740 1 -- 172.21.15.40:0/1498037097 >> 172.21.15.40:0/1498037097 conn(0x559dab1fc860 msgr2=0x559dab1fbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.577+0000 7fbfdaff5640 0 client.17808 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:51:30.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.577+0000 7fc00018c740 1 -- 172.21.15.40:0/1498037097 shutdown_connections 2024-03-18T22:51:30.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.577+0000 7fc00018c740 1 -- 172.21.15.40:0/1498037097 wait complete. 2024-03-18T22:51:30.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.577+0000 7fc00018c740 20 client.17808 trim_cache size 0 max 16384 2024-03-18T22:51:30.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.591+0000 7fc00018c740 1 Processor -- start 2024-03-18T22:51:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.591+0000 7fc00018c740 1 -- start start 2024-03-18T22:51:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.591+0000 7fc00018c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab2012f0 0x559dab20bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.591+0000 7fc00018c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab201ce0 0x559dab212cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.591+0000 7fc00018c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab206e50 0x559dab2001d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.591+0000 7fc00018c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81ac650 con 0x559dab206e50 2024-03-18T22:51:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.591+0000 7fc00018c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81ae1f0 con 0x559dab201ce0 2024-03-18T22:51:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.591+0000 7fc00018c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81acd20 con 0x559dab2012f0 2024-03-18T22:51:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.591+0000 7fbff77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab206e50 0x559dab2001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.591+0000 7fbff6ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab2012f0 0x559dab20bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.591+0000 7fbff6ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab2012f0 0x559dab20bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53912/0 (socket says 172.21.15.40:53912) 2024-03-18T22:51:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.591+0000 7fbff77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab206e50 0x559dab2001d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49494/0 (socket says 172.21.15.40:49494) 2024-03-18T22:51:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.591+0000 7fbff67fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab201ce0 0x559dab212cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.591+0000 7fbff6ffd640 1 -- 172.21.15.40:0/1107947076 learned_addr learned my addr 172.21.15.40:0/1107947076 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.592+0000 7fbff6ffd640 1 -- 172.21.15.40:0/1107947076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab201ce0 msgr2=0x559dab212cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.592+0000 7fbff6ffd640 1 --2- 172.21.15.40:0/1107947076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab201ce0 0x559dab212cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.592+0000 7fbff6ffd640 1 -- 172.21.15.40:0/1107947076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab206e50 msgr2=0x559dab2001d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.592+0000 7fbff6ffd640 1 --2- 172.21.15.40:0/1107947076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab206e50 0x559dab2001d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.592+0000 7fbff6ffd640 1 -- 172.21.15.40:0/1107947076 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dab2097a0 con 0x559dab2012f0 2024-03-18T22:51:30.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.592+0000 7fbff6ffd640 1 --2- 172.21.15.40:0/1107947076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab2012f0 0x559dab20bab0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbfe80538e0 tx=0x7fbfe80095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad47663f9ff8cbc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.592+0000 7fbfdcff9640 1 -- 172.21.15.40:0/1107947076 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8045040 con 0x559dab2012f0 2024-03-18T22:51:30.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.592+0000 7fbfdcff9640 1 -- 172.21.15.40:0/1107947076 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfe802cc90 con 0x559dab2012f0 2024-03-18T22:51:30.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.592+0000 7fbfdcff9640 1 -- 172.21.15.40:0/1107947076 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8005320 con 0x559dab2012f0 2024-03-18T22:51:30.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.593+0000 7fc00018c740 1 -- 172.21.15.40:0/1107947076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab2012f0 msgr2=0x559dab20bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.593+0000 7fc00018c740 1 --2- 172.21.15.40:0/1107947076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab2012f0 0x559dab20bab0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbfe80538e0 tx=0x7fbfe80095f0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.593+0000 7fc00018c740 1 -- 172.21.15.40:0/1107947076 shutdown_connections 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.593+0000 7fc00018c740 1 --2- 172.21.15.40:0/1107947076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab206e50 0x559dab2001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.593+0000 7fc00018c740 1 --2- 172.21.15.40:0/1107947076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab201ce0 0x559dab212cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.593+0000 7fc00018c740 1 --2- 172.21.15.40:0/1107947076 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab2012f0 0x559dab20bab0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.593+0000 7fc00018c740 1 -- 172.21.15.40:0/1107947076 >> 172.21.15.40:0/1107947076 conn(0x559dab1fc860 msgr2=0x559dab1ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.593+0000 7fc00018c740 1 -- 172.21.15.40:0/1107947076 shutdown_connections 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.593+0000 7fc00018c740 1 -- 172.21.15.40:0/1107947076 wait complete. 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.593+0000 7fc00018c740 1 -- start start 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.593+0000 7fc00018c740 1 Processor -- start 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fc00018c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab2012f0 0x559dab210890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fc00018c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab201ce0 0x559dab210dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fc00018c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab206e50 0x559dab211d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fc00018c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81adb20 con 0x559dab201ce0 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fc00018c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81ad3f0 con 0x559dab206e50 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fc00018c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbff81aa630 con 0x559dab2012f0 2024-03-18T22:51:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fc00018c740 10 client.0 _ll_register_callbacks cb 0x7fbfec009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fbff67fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab201ce0 0x559dab210dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fbff6ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab2012f0 0x559dab210890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fbff6ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab2012f0 0x559dab210890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53928/0 (socket says 172.21.15.40:53928) 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fbff67fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab201ce0 0x559dab210dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49502/0 (socket says 172.21.15.40:49502) 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fbff6ffd640 1 -- 172.21.15.40:0/2151660799 learned_addr learned my addr 172.21.15.40:0/2151660799 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fbff77fe640 1 --2- 172.21.15.40:0/2151660799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab206e50 0x559dab211d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fbff6ffd640 1 -- 172.21.15.40:0/2151660799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab206e50 msgr2=0x559dab211d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fbff6ffd640 1 --2- 172.21.15.40:0/2151660799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab206e50 0x559dab211d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fbff6ffd640 1 -- 172.21.15.40:0/2151660799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab201ce0 msgr2=0x559dab210dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fbff6ffd640 1 --2- 172.21.15.40:0/2151660799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab201ce0 0x559dab210dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fbff6ffd640 1 -- 172.21.15.40:0/2151660799 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dab212380 con 0x559dab2012f0 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fbff67fc640 1 --2- 172.21.15.40:0/2151660799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab201ce0 0x559dab210dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fbff6ffd640 1 --2- 172.21.15.40:0/2151660799 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab2012f0 0x559dab210890 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8004650 tx=0x7fbfe802bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a63459b6d531b5b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fbfdaff5640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.594+0000 7fbfdaff5640 1 -- 172.21.15.40:0/2151660799 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8045070 con 0x559dab2012f0 2024-03-18T22:51:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.595+0000 7fc00018c740 1 -- 172.21.15.40:0/2151660799 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfe802cc90 con 0x559dab2012f0 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.595+0000 7fbfdaff5640 1 -- 172.21.15.40:0/2151660799 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fbfe8003d50 con 0x559dab2012f0 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.595+0000 7fbfdaff5640 1 -- 172.21.15.40:0/2151660799 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbfe806e050 con 0x559dab2012f0 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.595+0000 7fc00018c740 10 client.27619 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.595+0000 7fc00018c740 1 -- 172.21.15.40:0/2151660799 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559dab27d860 con 0x559dab2012f0 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.595+0000 7fbfdaff5640 1 -- 172.21.15.40:0/2151660799 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfe802cc90 con 0x559dab2012f0 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.595+0000 7fbfdaff5640 1 -- 172.21.15.40:0/2151660799 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7fbfe8004860 con 0x559dab2012f0 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.595+0000 7fc00018c740 10 client.27619 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.595+0000 7fc00018c740 10 client.27619 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.595+0000 7fc00018c740 20 client.27619 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.595+0000 7fc00018c740 1 --2- 172.21.15.40:0/2151660799 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559dab27e180 0x559dab2805a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.595+0000 7fc00018c740 4 client.27619 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.595+0000 7fc00018c740 1 -- 172.21.15.40:0/2151660799 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559dab1729f0 con 0x559dab27e180 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.595+0000 7fbff67fc640 1 --2- 172.21.15.40:0/2151660799 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559dab27e180 0x559dab2805a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fbfdaff5640 10 client.27619 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fbff67fc640 1 --2- 172.21.15.40:0/2151660799 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559dab27e180 0x559dab2805a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7b54ec499b77f225 server_cookie=f11ec42071451c5c in_seq=0 out_seq=0 2024-03-18T22:51:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fbfdaff5640 1 -- 172.21.15.40:0/2151660799 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559dab1729f0 con 0x559dab27e180 2024-03-18T22:51:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fbfdaff5640 10 client.27619 handle_command_reply: tid=0 2024-03-18T22:51:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fc00018c740 1 client.27619 shutdown 2024-03-18T22:51:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fc00018c740 1 -- 172.21.15.40:0/2151660799 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab2012f0 msgr2=0x559dab210890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fc00018c740 1 --2- 172.21.15.40:0/2151660799 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab2012f0 0x559dab210890 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8004650 tx=0x7fbfe802bc30 comp rx=0 tx=0).stop 2024-03-18T22:51:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fc00018c740 1 -- 172.21.15.40:0/2151660799 shutdown_connections 2024-03-18T22:51:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fc00018c740 1 --2- 172.21.15.40:0/2151660799 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x559dab27e180 0x559dab2805a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fc00018c740 1 --2- 172.21.15.40:0/2151660799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559dab206e50 0x559dab211d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fc00018c740 1 --2- 172.21.15.40:0/2151660799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559dab201ce0 0x559dab210dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fc00018c740 1 --2- 172.21.15.40:0/2151660799 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559dab2012f0 0x559dab210890 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fbfdaff5640 0 client.27619 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:51:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fc00018c740 1 -- 172.21.15.40:0/2151660799 >> 172.21.15.40:0/2151660799 conn(0x559dab1fc860 msgr2=0x559dab200350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fc00018c740 1 -- 172.21.15.40:0/2151660799 shutdown_connections 2024-03-18T22:51:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fc00018c740 1 -- 172.21.15.40:0/2151660799 wait complete. 2024-03-18T22:51:30.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.596+0000 7fc00018c740 20 client.27619 trim_cache size 0 max 16384 2024-03-18T22:51:30.603 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:51:30.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.597+0000 7fbffd4c6640 1 -- 172.21.15.40:0/2366959186 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbfe00a2800 msgr2=0x7fbfe00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.597+0000 7fbffd4c6640 1 --2- 172.21.15.40:0/2366959186 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbfe00a2800 0x7fbfe00a4cc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x559dab202540 tx=0x7fbfec009ba0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.597+0000 7fbffd4c6640 1 -- 172.21.15.40:0/2366959186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff8103410 msgr2=0x7fbff8074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.597+0000 7fbffd4c6640 1 --2- 172.21.15.40:0/2366959186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff8103410 0x7fbff8074150 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009fd0 tx=0x7fbff000ef10 comp rx=0 tx=0).stop 2024-03-18T22:51:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.597+0000 7fbffd4c6640 1 -- 172.21.15.40:0/2366959186 shutdown_connections 2024-03-18T22:51:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.597+0000 7fbffd4c6640 1 --2- 172.21.15.40:0/2366959186 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbfe00a2800 0x7fbfe00a4cc0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.597+0000 7fbffd4c6640 1 --2- 172.21.15.40:0/2366959186 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbff8103410 0x7fbff8074150 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.597+0000 7fbffd4c6640 1 --2- 172.21.15.40:0/2366959186 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbff8102a50 0x7fbff8073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.597+0000 7fbffd4c6640 1 --2- 172.21.15.40:0/2366959186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbff8101860 0x7fbff80755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.597+0000 7fbffd4c6640 1 -- 172.21.15.40:0/2366959186 >> 172.21.15.40:0/2366959186 conn(0x7fbff80fcdb0 msgr2=0x7fbff8105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.597+0000 7fbffd4c6640 1 -- 172.21.15.40:0/2366959186 shutdown_connections 2024-03-18T22:51:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.597+0000 7fbffd4c6640 1 -- 172.21.15.40:0/2366959186 wait complete. 2024-03-18T22:51:30.609 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:51:30.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.726+0000 7f8e8bfd4640 1 -- 172.21.15.40:0/3285490843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e841018b0 msgr2=0x7f8e84101cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.726+0000 7f8e8bfd4640 1 --2- 172.21.15.40:0/3285490843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e841018b0 0x7f8e84101cd0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f8e80009900 tx=0x7f8e8002bc30 comp rx=0 tx=0).stop 2024-03-18T22:51:30.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.726+0000 7f8e8bfd4640 1 -- 172.21.15.40:0/3285490843 shutdown_connections 2024-03-18T22:51:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.726+0000 7f8e8bfd4640 1 --2- 172.21.15.40:0/3285490843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e84103460 0x7f8e84109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.726+0000 7f8e8bfd4640 1 --2- 172.21.15.40:0/3285490843 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e84102aa0 0x7f8e84102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.726+0000 7f8e8bfd4640 1 --2- 172.21.15.40:0/3285490843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e841018b0 0x7f8e84101cd0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.727+0000 7f8e8bfd4640 1 -- 172.21.15.40:0/3285490843 >> 172.21.15.40:0/3285490843 conn(0x7f8e840fce40 msgr2=0x7f8e840ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.727+0000 7f8e8bfd4640 1 -- 172.21.15.40:0/3285490843 shutdown_connections 2024-03-18T22:51:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.727+0000 7f8e8bfd4640 1 -- 172.21.15.40:0/3285490843 wait complete. 2024-03-18T22:51:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.727+0000 7f8e8bfd4640 1 Processor -- start 2024-03-18T22:51:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.727+0000 7f8e8bfd4640 1 -- start start 2024-03-18T22:51:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.727+0000 7f8e8bfd4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e841018b0 0x7f8e84197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.727+0000 7f8e8bfd4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e84102aa0 0x7f8e84197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.727+0000 7f8e8bfd4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e84103460 0x7f8e8419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.727+0000 7f8e8bfd4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8410d340 con 0x7f8e84102aa0 2024-03-18T22:51:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.727+0000 7f8e8bfd4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8410d1c0 con 0x7f8e841018b0 2024-03-18T22:51:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.727+0000 7f8e8bfd4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8410d4c0 con 0x7f8e84103460 2024-03-18T22:51:30.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e8956f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e84102aa0 0x7f8e84197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e8956f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e84102aa0 0x7f8e84197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49522/0 (socket says 172.21.15.40:49522) 2024-03-18T22:51:30.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e8956f640 1 -- 172.21.15.40:0/1516018436 learned_addr learned my addr 172.21.15.40:0/1516018436 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e89d70640 1 --2- 172.21.15.40:0/1516018436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e841018b0 0x7f8e84197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e8a571640 1 --2- 172.21.15.40:0/1516018436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e84103460 0x7f8e8419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e8956f640 1 -- 172.21.15.40:0/1516018436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e84103460 msgr2=0x7f8e8419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e8956f640 1 --2- 172.21.15.40:0/1516018436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e84103460 0x7f8e8419eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e8956f640 1 -- 172.21.15.40:0/1516018436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e841018b0 msgr2=0x7f8e84197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e8956f640 1 --2- 172.21.15.40:0/1516018436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e841018b0 0x7f8e84197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e8956f640 1 -- 172.21.15.40:0/1516018436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e8419f660 con 0x7f8e84102aa0 2024-03-18T22:51:30.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e8956f640 1 --2- 172.21.15.40:0/1516018436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e84102aa0 0x7f8e84197f80 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400e9f0 tx=0x7f8e7400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e7affd640 1 -- 172.21.15.40:0/1516018436 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7400cd80 con 0x7f8e84102aa0 2024-03-18T22:51:30.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e7affd640 1 -- 172.21.15.40:0/1516018436 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e740108e0 con 0x7f8e84102aa0 2024-03-18T22:51:30.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e8bfd4640 1 -- 172.21.15.40:0/1516018436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e8419f950 con 0x7f8e84102aa0 2024-03-18T22:51:30.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e7affd640 1 -- 172.21.15.40:0/1516018436 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e74018920 con 0x7f8e84102aa0 2024-03-18T22:51:30.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.728+0000 7f8e8bfd4640 1 -- 172.21.15.40:0/1516018436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e8419fef0 con 0x7f8e84102aa0 2024-03-18T22:51:30.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.731+0000 7f8e7affd640 1 -- 172.21.15.40:0/1516018436 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8e74010a90 con 0x7f8e84102aa0 2024-03-18T22:51:30.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.731+0000 7f8e8ebc2740 1 Processor -- start 2024-03-18T22:51:30.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.731+0000 7f8e8ebc2740 1 -- start start 2024-03-18T22:51:30.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8ebc2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b92f0 0x556bc74b9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8ebc2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b9ce0 0x556bc74be7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8ebc2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 0x556bc74c12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8ebc2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556bc741d4b0 con 0x556bc74b9ce0 2024-03-18T22:51:30.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8ebc2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556bc73c5d70 con 0x556bc74bee50 2024-03-18T22:51:30.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8ebc2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556bc7419550 con 0x556bc74b92f0 2024-03-18T22:51:30.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e7affd640 1 --2- 172.21.15.40:0/1516018436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e700a2880 0x7f8e700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e89d70640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b9ce0 0x556bc74be7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8a571640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b92f0 0x556bc74b9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e89d70640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b9ce0 0x556bc74be7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49526/0 (socket says 172.21.15.40:49526) 2024-03-18T22:51:30.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8a571640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b92f0 0x556bc74b9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53954/0 (socket says 172.21.15.40:53954) 2024-03-18T22:51:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e89d70640 1 -- 172.21.15.40:0/1491581362 learned_addr learned my addr 172.21.15.40:0/1491581362 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8956f640 1 --2- 172.21.15.40:0/1491581362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 0x556bc74c12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8a571640 1 --2- 172.21.15.40:0/1516018436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e700a2880 0x7f8e700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e7affd640 1 -- 172.21.15.40:0/1516018436 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8e740cf0e0 con 0x7f8e84102aa0 2024-03-18T22:51:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8a571640 1 -- 172.21.15.40:0/1491581362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 msgr2=0x556bc74c12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8a571640 1 --2- 172.21.15.40:0/1491581362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 0x556bc74c12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8a571640 1 -- 172.21.15.40:0/1491581362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b9ce0 msgr2=0x556bc74be7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8a571640 1 --2- 172.21.15.40:0/1491581362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b9ce0 0x556bc74be7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8a571640 1 -- 172.21.15.40:0/1491581362 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bc73e8210 con 0x556bc74b92f0 2024-03-18T22:51:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e89d70640 1 --2- 172.21.15.40:0/1491581362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b9ce0 0x556bc74be7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.732+0000 7f8e8956f640 1 --2- 172.21.15.40:0/1491581362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 0x556bc74c12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.733+0000 7f8e8a571640 1 --2- 172.21.15.40:0/1491581362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b92f0 0x556bc74b9710 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c011d30 tx=0x7f8e7c00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84917d2a451cffaa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.733+0000 7f8e8a571640 1 --2- 172.21.15.40:0/1516018436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e700a2880 0x7f8e700a4d40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c00fc90 tx=0x7f8e7c018000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.733+0000 7f8e6ffff640 1 -- 172.21.15.40:0/1491581362 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c011570 con 0x556bc74b92f0 2024-03-18T22:51:30.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.733+0000 7f8e6ffff640 1 -- 172.21.15.40:0/1491581362 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e7c009d30 con 0x556bc74b92f0 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.733+0000 7f8e6ffff640 1 -- 172.21.15.40:0/1491581362 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c005870 con 0x556bc74b92f0 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.734+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/1491581362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b92f0 msgr2=0x556bc74b9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.734+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/1491581362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b92f0 0x556bc74b9710 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c011d30 tx=0x7f8e7c00f790 comp rx=0 tx=0).stop 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.734+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/1491581362 shutdown_connections 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.734+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/1491581362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 0x556bc74c12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.734+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/1491581362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b9ce0 0x556bc74be7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.734+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/1491581362 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b92f0 0x556bc74b9710 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.734+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/1491581362 >> 172.21.15.40:0/1491581362 conn(0x556bc74b4860 msgr2=0x556bc74b4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.734+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/1491581362 shutdown_connections 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.734+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/1491581362 wait complete. 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.734+0000 7f8e8ebc2740 1 -- start start 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.734+0000 7f8e8ebc2740 1 Processor -- start 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e8ebc2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b92f0 0x556bc74c3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e8ebc2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b9ce0 0x556bc74c3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e8ebc2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 0x556bc74cad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e8ebc2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556bc74c1d40 con 0x556bc74b92f0 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e8ebc2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556bc73d9310 con 0x556bc74bee50 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e8ebc2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556bc74c1ec0 con 0x556bc74b9ce0 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e8ebc2740 10 client.0 _ll_register_callbacks cb 0x556bc73cd2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e89d70640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b92f0 0x556bc74c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e8956f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b9ce0 0x556bc74c3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e89d70640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b92f0 0x556bc74c3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49540/0 (socket says 172.21.15.40:49540) 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e89d70640 1 -- 172.21.15.40:0/2168784564 learned_addr learned my addr 172.21.15.40:0/2168784564 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e8956f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b9ce0 0x556bc74c3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53956/0 (socket says 172.21.15.40:53956) 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e8a571640 1 --2- 172.21.15.40:0/2168784564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 0x556bc74cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e89d70640 1 -- 172.21.15.40:0/2168784564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b9ce0 msgr2=0x556bc74c3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e89d70640 1 --2- 172.21.15.40:0/2168784564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b9ce0 0x556bc74c3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e89d70640 1 -- 172.21.15.40:0/2168784564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 msgr2=0x556bc74cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e89d70640 1 --2- 172.21.15.40:0/2168784564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 0x556bc74cad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e89d70640 1 -- 172.21.15.40:0/2168784564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bc71313f0 con 0x556bc74b92f0 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e8956f640 1 --2- 172.21.15.40:0/2168784564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b9ce0 0x556bc74c3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e89d70640 1 --2- 172.21.15.40:0/2168784564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b92f0 0x556bc74c3780 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f8e80036040 tx=0x7f8e80004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06b0ec2a7827d76 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:51:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e6d7fa640 1 -- 172.21.15.40:0/2168784564 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80005030 con 0x556bc74b92f0 2024-03-18T22:51:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e6d7fa640 1 -- 172.21.15.40:0/2168784564 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8e800051d0 con 0x556bc74b92f0 2024-03-18T22:51:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.735+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/2168784564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e7c009d30 con 0x556bc74b92f0 2024-03-18T22:51:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.736+0000 7f8e6d7fa640 1 -- 172.21.15.40:0/2168784564 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8e80051050 con 0x556bc74b92f0 2024-03-18T22:51:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.736+0000 7f8e8ebc2740 10 client.17826 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.736+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/2168784564 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556bc71fb7d0 con 0x556bc74b92f0 2024-03-18T22:51:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.736+0000 7f8e6d7fa640 1 -- 172.21.15.40:0/2168784564 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e7c009d30 con 0x556bc74b92f0 2024-03-18T22:51:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.736+0000 7f8e6d7fa640 1 -- 172.21.15.40:0/2168784564 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f8e800035f0 con 0x556bc74b92f0 2024-03-18T22:51:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.736+0000 7f8e8ebc2740 10 client.17826 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.736+0000 7f8e8ebc2740 10 client.17826 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:51:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.736+0000 7f8e8ebc2740 20 client.17826 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.736+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/2168784564 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556bc74cf630 0x556bc74d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.736+0000 7f8e8ebc2740 4 client.17826 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:51:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.736+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/2168784564 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556bc747b3e0 con 0x556bc74cf630 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.736+0000 7f8e8956f640 1 --2- 172.21.15.40:0/2168784564 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556bc74cf630 0x556bc74d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.737+0000 7f8e8956f640 1 --2- 172.21.15.40:0/2168784564 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556bc74cf630 0x556bc74d1ad0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c031f3971e1a0448 server_cookie=beb871028327c30b in_seq=0 out_seq=0 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.737+0000 7f8e6d7fa640 10 client.17826 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.737+0000 7f8e6d7fa640 1 -- 172.21.15.40:0/2168784564 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556bc747b3e0 con 0x556bc74cf630 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.737+0000 7f8e6d7fa640 10 client.17826 handle_command_reply: tid=0 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.737+0000 7f8e8ebc2740 1 client.17826 shutdown 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.738+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/2168784564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b92f0 msgr2=0x556bc74c3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.738+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/2168784564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b92f0 0x556bc74c3780 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f8e80036040 tx=0x7f8e80004270 comp rx=0 tx=0).stop 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.738+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/2168784564 shutdown_connections 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.738+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/2168784564 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556bc74cf630 0x556bc74d1ad0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.738+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/2168784564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 0x556bc74cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.738+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/2168784564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b9ce0 0x556bc74c3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.738+0000 7f8e6d7fa640 0 client.17826 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.738+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/2168784564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b92f0 0x556bc74c3780 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.738+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/2168784564 >> 172.21.15.40:0/2168784564 conn(0x556bc74b4860 msgr2=0x556bc74b3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.738+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/2168784564 shutdown_connections 2024-03-18T22:51:30.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.738+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/2168784564 wait complete. 2024-03-18T22:51:30.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.738+0000 7f8e8ebc2740 20 client.17826 trim_cache size 0 max 16384 2024-03-18T22:51:30.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.751+0000 7f8e8ebc2740 1 Processor -- start 2024-03-18T22:51:30.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.751+0000 7f8e8ebc2740 1 -- start start 2024-03-18T22:51:30.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e8ebc2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b92f0 0x556bc74ba700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e8ebc2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b9ce0 0x556bc74c2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e8ebc2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 0x556bc74c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e8ebc2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8e841aa4f0 con 0x556bc74b9ce0 2024-03-18T22:51:30.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e8ebc2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8e841ac090 con 0x556bc74bee50 2024-03-18T22:51:30.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e8ebc2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8e841aabc0 con 0x556bc74b92f0 2024-03-18T22:51:30.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e89d70640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b92f0 0x556bc74ba700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e8956f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b9ce0 0x556bc74c2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e89d70640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b92f0 0x556bc74ba700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53958/0 (socket says 172.21.15.40:53958) 2024-03-18T22:51:30.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e89d70640 1 -- 172.21.15.40:0/4216793324 learned_addr learned my addr 172.21.15.40:0/4216793324 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e8956f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b9ce0 0x556bc74c2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49556/0 (socket says 172.21.15.40:49556) 2024-03-18T22:51:30.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e8a571640 1 --2- 172.21.15.40:0/4216793324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 0x556bc74c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e89d70640 1 -- 172.21.15.40:0/4216793324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 msgr2=0x556bc74c32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e89d70640 1 --2- 172.21.15.40:0/4216793324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 0x556bc74c32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e89d70640 1 -- 172.21.15.40:0/4216793324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b9ce0 msgr2=0x556bc74c2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e89d70640 1 --2- 172.21.15.40:0/4216793324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b9ce0 0x556bc74c2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e89d70640 1 -- 172.21.15.40:0/4216793324 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bc74b8f30 con 0x556bc74b92f0 2024-03-18T22:51:30.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e89d70640 1 --2- 172.21.15.40:0/4216793324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b92f0 0x556bc74ba700 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8e800096c0 tx=0x7f8e8002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afff968029e7c2b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.752+0000 7f8e6f7fe640 1 -- 172.21.15.40:0/4216793324 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80003c30 con 0x556bc74b92f0 2024-03-18T22:51:30.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.753+0000 7f8e6f7fe640 1 -- 172.21.15.40:0/4216793324 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e80003dd0 con 0x556bc74b92f0 2024-03-18T22:51:30.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.753+0000 7f8e6f7fe640 1 -- 172.21.15.40:0/4216793324 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e800050e0 con 0x556bc74b92f0 2024-03-18T22:51:30.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.753+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/4216793324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b92f0 msgr2=0x556bc74ba700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.753+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/4216793324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b92f0 0x556bc74ba700 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8e800096c0 tx=0x7f8e8002bc30 comp rx=0 tx=0).stop 2024-03-18T22:51:30.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.753+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/4216793324 shutdown_connections 2024-03-18T22:51:30.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.753+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/4216793324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74bee50 0x556bc74c32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.753+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/4216793324 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74b9ce0 0x556bc74c2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.753+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/4216793324 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b92f0 0x556bc74ba700 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.753+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/4216793324 >> 172.21.15.40:0/4216793324 conn(0x556bc74b4860 msgr2=0x556bc74b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.753+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/4216793324 shutdown_connections 2024-03-18T22:51:30.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.753+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/4216793324 wait complete. 2024-03-18T22:51:30.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e8ebc2740 1 -- start start 2024-03-18T22:51:30.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e8ebc2740 1 Processor -- start 2024-03-18T22:51:30.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e8ebc2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74b92f0 0x556bc7530f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e8ebc2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b9ce0 0x556bc75314a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e8ebc2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74bee50 0x556bc75384e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e8ebc2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8e841ab9c0 con 0x556bc74bee50 2024-03-18T22:51:30.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e8ebc2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8e841ab290 con 0x556bc74b92f0 2024-03-18T22:51:30.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e8ebc2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8e841a84d0 con 0x556bc74b9ce0 2024-03-18T22:51:30.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e8ebc2740 10 client.0 _ll_register_callbacks cb 0x7f8e80009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:30.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e8956f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b9ce0 0x556bc75314a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e8a571640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74bee50 0x556bc75384e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e89d70640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74b92f0 0x556bc7530f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e8a571640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74bee50 0x556bc75384e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49562/0 (socket says 172.21.15.40:49562) 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e89d70640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74b92f0 0x556bc7530f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:35434/0 (socket says 172.21.15.40:35434) 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e8a571640 1 -- 172.21.15.40:0/4126707407 learned_addr learned my addr 172.21.15.40:0/4126707407 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.754+0000 7f8e8956f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b9ce0 0x556bc75314a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53964/0 (socket says 172.21.15.40:53964) 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.755+0000 7f8e8956f640 1 -- 172.21.15.40:0/4126707407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74b92f0 msgr2=0x556bc7530f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.755+0000 7f8e8956f640 1 --2- 172.21.15.40:0/4126707407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74b92f0 0x556bc7530f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.755+0000 7f8e8956f640 1 -- 172.21.15.40:0/4126707407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74bee50 msgr2=0x556bc75384e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.755+0000 7f8e8956f640 1 --2- 172.21.15.40:0/4126707407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74bee50 0x556bc75384e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.755+0000 7f8e8956f640 1 -- 172.21.15.40:0/4126707407 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bc7538ad0 con 0x556bc74b9ce0 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.755+0000 7f8e89d70640 1 --2- 172.21.15.40:0/4126707407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74b92f0 0x556bc7530f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.755+0000 7f8e8956f640 1 --2- 172.21.15.40:0/4126707407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b9ce0 0x556bc75314a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8e740cc840 tx=0x7f8e7406b320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b6a5192d47c3874 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.755+0000 7f8e6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.755+0000 7f8e6d7fa640 1 -- 172.21.15.40:0/4126707407 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e740a3e30 con 0x556bc74b9ce0 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.755+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/4126707407 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e80003dd0 con 0x556bc74b9ce0 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.755+0000 7f8e6d7fa640 1 -- 172.21.15.40:0/4126707407 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8e740860b0 con 0x556bc74b9ce0 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.755+0000 7f8e6d7fa640 1 -- 172.21.15.40:0/4126707407 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8e7406d040 con 0x556bc74b9ce0 2024-03-18T22:51:30.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.755+0000 7f8e8ebc2740 10 client.27637 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:30.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.755+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/4126707407 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556bc7539e40 con 0x556bc74b9ce0 2024-03-18T22:51:30.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.755+0000 7f8e6d7fa640 1 -- 172.21.15.40:0/4126707407 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e80003dd0 con 0x556bc74b9ce0 2024-03-18T22:51:30.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.756+0000 7f8e6d7fa640 1 -- 172.21.15.40:0/4126707407 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f8e7409ab90 con 0x556bc74b9ce0 2024-03-18T22:51:30.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.756+0000 7f8e8ebc2740 10 client.27637 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:30.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.756+0000 7f8e8ebc2740 10 client.27637 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:51:30.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.756+0000 7f8e8ebc2740 20 client.27637 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:30.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.756+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/4126707407 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556bc753a7a0 0x556bc753cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.756+0000 7f8e8ebc2740 4 client.27637 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:51:30.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.756+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/4126707407 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556bc742a9f0 con 0x556bc753a7a0 2024-03-18T22:51:30.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.756+0000 7f8e89d70640 1 --2- 172.21.15.40:0/4126707407 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556bc753a7a0 0x556bc753cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.756+0000 7f8e89d70640 1 --2- 172.21.15.40:0/4126707407 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556bc753a7a0 0x556bc753cbc0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=437b54d81e125a41 server_cookie=6209ae824b1aa462 in_seq=0 out_seq=0 2024-03-18T22:51:30.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.756+0000 7f8e6d7fa640 10 client.27637 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:51:30.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.756+0000 7f8e6d7fa640 1 -- 172.21.15.40:0/4126707407 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556bc742a9f0 con 0x556bc753a7a0 2024-03-18T22:51:30.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.756+0000 7f8e6d7fa640 10 client.27637 handle_command_reply: tid=0 2024-03-18T22:51:30.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.756+0000 7f8e8ebc2740 1 client.27637 shutdown 2024-03-18T22:51:30.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/4126707407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b9ce0 msgr2=0x556bc75314a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/4126707407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b9ce0 0x556bc75314a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8e740cc840 tx=0x7f8e7406b320 comp rx=0 tx=0).stop 2024-03-18T22:51:30.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/4126707407 shutdown_connections 2024-03-18T22:51:30.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/4126707407 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556bc753a7a0 0x556bc753cbc0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/4126707407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556bc74bee50 0x556bc75384e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e6d7fa640 0 client.27637 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:51:30.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/4126707407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556bc74b9ce0 0x556bc75314a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8ebc2740 1 --2- 172.21.15.40:0/4126707407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bc74b92f0 0x556bc7530f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/4126707407 >> 172.21.15.40:0/4126707407 conn(0x556bc74b4860 msgr2=0x556bc74b3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/4126707407 shutdown_connections 2024-03-18T22:51:30.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8ebc2740 1 -- 172.21.15.40:0/4126707407 wait complete. 2024-03-18T22:51:30.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8ebc2740 20 client.27637 trim_cache size 0 max 16384 2024-03-18T22:51:30.764 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:51:30.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8bfd4640 1 -- 172.21.15.40:0/1516018436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e700a2880 msgr2=0x7f8e700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8bfd4640 1 --2- 172.21.15.40:0/1516018436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e700a2880 0x7f8e700a4d40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c00fc90 tx=0x7f8e7c018000 comp rx=0 tx=0).stop 2024-03-18T22:51:30.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8bfd4640 1 -- 172.21.15.40:0/1516018436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e84102aa0 msgr2=0x7f8e84197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8bfd4640 1 --2- 172.21.15.40:0/1516018436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e84102aa0 0x7f8e84197f80 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400e9f0 tx=0x7f8e7400eeb0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8bfd4640 1 -- 172.21.15.40:0/1516018436 shutdown_connections 2024-03-18T22:51:30.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8bfd4640 1 --2- 172.21.15.40:0/1516018436 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8e700a2880 0x7f8e700a4d40 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8bfd4640 1 --2- 172.21.15.40:0/1516018436 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8e84103460 0x7f8e8419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8bfd4640 1 --2- 172.21.15.40:0/1516018436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e84102aa0 0x7f8e84197f80 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8bfd4640 1 --2- 172.21.15.40:0/1516018436 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e841018b0 0x7f8e84197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.757+0000 7f8e8bfd4640 1 -- 172.21.15.40:0/1516018436 >> 172.21.15.40:0/1516018436 conn(0x7f8e840fce40 msgr2=0x7f8e84105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.758+0000 7f8e8bfd4640 1 -- 172.21.15.40:0/1516018436 shutdown_connections 2024-03-18T22:51:30.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.758+0000 7f8e8bfd4640 1 -- 172.21.15.40:0/1516018436 wait complete. 2024-03-18T22:51:30.770 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:51:30.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.898+0000 7f0ffc8dc640 1 -- 172.21.15.40:0/2169524552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ff8101840 msgr2=0x7f0ff8101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.898+0000 7f0ffc8dc640 1 --2- 172.21.15.40:0/2169524552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ff8101840 0x7f0ff8101c60 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009900 tx=0x7f0fe802bc30 comp rx=0 tx=0).stop 2024-03-18T22:51:30.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.898+0000 7f0ffc8dc640 1 -- 172.21.15.40:0/2169524552 shutdown_connections 2024-03-18T22:51:30.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.898+0000 7f0ffc8dc640 1 --2- 172.21.15.40:0/2169524552 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ff81033f0 0x7f0ff8109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.898+0000 7f0ffc8dc640 1 --2- 172.21.15.40:0/2169524552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ff8102a30 0x7f0ff8102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.898+0000 7f0ffc8dc640 1 --2- 172.21.15.40:0/2169524552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ff8101840 0x7f0ff8101c60 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.898+0000 7f0ffc8dc640 1 -- 172.21.15.40:0/2169524552 >> 172.21.15.40:0/2169524552 conn(0x7f0ff80fcdd0 msgr2=0x7f0ff80ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.898+0000 7f0ffc8dc640 1 -- 172.21.15.40:0/2169524552 shutdown_connections 2024-03-18T22:51:30.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.898+0000 7f0ffc8dc640 1 -- 172.21.15.40:0/2169524552 wait complete. 2024-03-18T22:51:30.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ffc8dc640 1 Processor -- start 2024-03-18T22:51:30.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ffc8dc640 1 -- start start 2024-03-18T22:51:30.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ffc8dc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ff8101840 0x7f0ff8197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ffc8dc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ff8102a30 0x7f0ff8197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ffc8dc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ff81033f0 0x7f0ff819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ffc8dc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff810d340 con 0x7f0ff8101840 2024-03-18T22:51:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ffc8dc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff810d1c0 con 0x7f0ff8102a30 2024-03-18T22:51:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ffc8dc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff810d4c0 con 0x7f0ff81033f0 2024-03-18T22:51:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ff659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ff8101840 0x7f0ff8197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ff659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ff8101840 0x7f0ff8197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49578/0 (socket says 172.21.15.40:49578) 2024-03-18T22:51:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ff659c640 1 -- 172.21.15.40:0/1306041028 learned_addr learned my addr 172.21.15.40:0/1306041028 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ff6d9d640 1 --2- 172.21.15.40:0/1306041028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ff81033f0 0x7f0ff819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0feffff640 1 --2- 172.21.15.40:0/1306041028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ff8102a30 0x7f0ff8197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ff6d9d640 1 -- 172.21.15.40:0/1306041028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ff8102a30 msgr2=0x7f0ff8197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ff6d9d640 1 --2- 172.21.15.40:0/1306041028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ff8102a30 0x7f0ff8197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ff6d9d640 1 -- 172.21.15.40:0/1306041028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ff8101840 msgr2=0x7f0ff8197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ff6d9d640 1 --2- 172.21.15.40:0/1306041028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ff8101840 0x7f0ff8197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ff6d9d640 1 -- 172.21.15.40:0/1306041028 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff819f660 con 0x7f0ff81033f0 2024-03-18T22:51:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.899+0000 7f0ff659c640 1 --2- 172.21.15.40:0/1306041028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ff8101840 0x7f0ff8197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.900+0000 7f0ff6d9d640 1 --2- 172.21.15.40:0/1306041028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ff81033f0 0x7f0ff819eff0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009fd0 tx=0x7f0ff000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.900+0000 7f0fef7fe640 1 -- 172.21.15.40:0/1306041028 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff000ce20 con 0x7f0ff81033f0 2024-03-18T22:51:30.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.900+0000 7f0ffc8dc640 1 -- 172.21.15.40:0/1306041028 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ff819f950 con 0x7f0ff81033f0 2024-03-18T22:51:30.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.900+0000 7f0fef7fe640 1 -- 172.21.15.40:0/1306041028 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ff0005020 con 0x7f0ff81033f0 2024-03-18T22:51:30.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.900+0000 7f0fef7fe640 1 -- 172.21.15.40:0/1306041028 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0010690 con 0x7f0ff81033f0 2024-03-18T22:51:30.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.900+0000 7f0ffc8dc640 1 -- 172.21.15.40:0/1306041028 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ff819fea0 con 0x7f0ff81033f0 2024-03-18T22:51:30.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.902+0000 7f0fef7fe640 1 -- 172.21.15.40:0/1306041028 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0ff00051d0 con 0x7f0ff81033f0 2024-03-18T22:51:30.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.902+0000 7f0fff5a5740 1 Processor -- start 2024-03-18T22:51:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0fff5a5740 1 -- start start 2024-03-18T22:51:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0fff5a5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece2f2f0 0x555cece2f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0fff5a5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece347e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0fff5a5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece34e50 0x555cece372e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0fff5a5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555cecd934b0 con 0x555cece2fce0 2024-03-18T22:51:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0fff5a5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555cecd3bd70 con 0x555cece2f2f0 2024-03-18T22:51:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0fff5a5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x555cecd8f550 con 0x555cece34e50 2024-03-18T22:51:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0ff659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece347e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0ff659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece347e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49580/0 (socket says 172.21.15.40:49580) 2024-03-18T22:51:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0feffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece34e50 0x555cece372e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0ff659c640 1 -- 172.21.15.40:0/2459317704 learned_addr learned my addr 172.21.15.40:0/2459317704 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0fef7fe640 1 --2- 172.21.15.40:0/1306041028 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0fe40a2800 0x7f0fe40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0feffff640 1 --2- 172.21.15.40:0/2459317704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece2f2f0 0x555cece2f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0ff6d9d640 1 --2- 172.21.15.40:0/1306041028 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0fe40a2800 0x7f0fe40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0fef7fe640 1 -- 172.21.15.40:0/1306041028 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0ff00cf4f0 con 0x7f0ff81033f0 2024-03-18T22:51:30.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0feffff640 1 -- 172.21.15.40:0/2459317704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece2f2f0 msgr2=0x555cece2f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.903+0000 7f0feffff640 1 --2- 172.21.15.40:0/2459317704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece2f2f0 0x555cece2f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.904+0000 7f0feffff640 1 -- 172.21.15.40:0/2459317704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 msgr2=0x555cece347e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.904+0000 7f0feffff640 1 --2- 172.21.15.40:0/2459317704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece347e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.904+0000 7f0feffff640 1 -- 172.21.15.40:0/2459317704 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cecd5e210 con 0x555cece34e50 2024-03-18T22:51:30.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.904+0000 7f0ff659c640 1 --2- 172.21.15.40:0/2459317704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece347e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.904+0000 7f0feffff640 1 --2- 172.21.15.40:0/2459317704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece34e50 0x555cece372e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0009fd0 tx=0x7f0fe00096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abcf9c1ae6e7ae3b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.904+0000 7f0ff6d9d640 1 --2- 172.21.15.40:0/1306041028 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0fe40a2800 0x7f0fe40a4cc0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0097f00 tx=0x7f0ff009e040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.904+0000 7f0fecff9640 1 -- 172.21.15.40:0/2459317704 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0004050 con 0x555cece34e50 2024-03-18T22:51:30.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.904+0000 7f0fecff9640 1 -- 172.21.15.40:0/2459317704 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0fe00054d0 con 0x555cece34e50 2024-03-18T22:51:30.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.904+0000 7f0fecff9640 1 -- 172.21.15.40:0/2459317704 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0004c30 con 0x555cece34e50 2024-03-18T22:51:30.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.905+0000 7f0fff5a5740 1 -- 172.21.15.40:0/2459317704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece34e50 msgr2=0x555cece372e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.905+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/2459317704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece34e50 0x555cece372e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0009fd0 tx=0x7f0fe00096c0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.905+0000 7f0fff5a5740 1 -- 172.21.15.40:0/2459317704 shutdown_connections 2024-03-18T22:51:30.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.905+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/2459317704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece34e50 0x555cece372e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.905+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/2459317704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece347e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.905+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/2459317704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece2f2f0 0x555cece2f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.905+0000 7f0fff5a5740 1 -- 172.21.15.40:0/2459317704 >> 172.21.15.40:0/2459317704 conn(0x555cece2a860 msgr2=0x555cece2ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.905+0000 7f0fff5a5740 1 -- 172.21.15.40:0/2459317704 shutdown_connections 2024-03-18T22:51:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.905+0000 7f0fff5a5740 1 -- 172.21.15.40:0/2459317704 wait complete. 2024-03-18T22:51:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.905+0000 7f0fff5a5740 1 -- start start 2024-03-18T22:51:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.905+0000 7f0fff5a5740 1 Processor -- start 2024-03-18T22:51:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0fff5a5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece2f2f0 0x555cece39830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0fff5a5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece39d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0fff5a5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece34e50 0x555cece40db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0fff5a5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555cece37e00 con 0x555cece2fce0 2024-03-18T22:51:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0fff5a5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555cecd4f310 con 0x555cece2f2f0 2024-03-18T22:51:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0fff5a5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x555cece37f80 con 0x555cece34e50 2024-03-18T22:51:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0fff5a5740 10 client.0 _ll_register_callbacks cb 0x555cecd432d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0feffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece2f2f0 0x555cece39830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0ff659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece39d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0feffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece2f2f0 0x555cece39830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:35476/0 (socket says 172.21.15.40:35476) 2024-03-18T22:51:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0feffff640 1 -- 172.21.15.40:0/1812590862 learned_addr learned my addr 172.21.15.40:0/1812590862 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0feffff640 1 --2- 172.21.15.40:0/1812590862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece34e50 0x555cece40db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0feffff640 1 -- 172.21.15.40:0/1812590862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece34e50 msgr2=0x555cece40db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0feffff640 1 --2- 172.21.15.40:0/1812590862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece34e50 0x555cece40db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0feffff640 1 -- 172.21.15.40:0/1812590862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 msgr2=0x555cece39d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0feffff640 1 --2- 172.21.15.40:0/1812590862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece39d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0feffff640 1 -- 172.21.15.40:0/1812590862 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cecaa73f0 con 0x555cece2f2f0 2024-03-18T22:51:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0ff659c640 1 --2- 172.21.15.40:0/1812590862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece39d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0feffff640 1 --2- 172.21.15.40:0/1812590862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece2f2f0 0x555cece39830 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0004620 tx=0x7f0fe0011550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=252275945e21622b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.906+0000 7f0fddffb640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:51:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.907+0000 7f0fddffb640 1 -- 172.21.15.40:0/1812590862 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe00050b0 con 0x555cece2f2f0 2024-03-18T22:51:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.907+0000 7f0fddffb640 1 -- 172.21.15.40:0/1812590862 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0fe0005250 con 0x555cece2f2f0 2024-03-18T22:51:30.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.907+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1812590862 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fe00054d0 con 0x555cece2f2f0 2024-03-18T22:51:30.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.907+0000 7f0fddffb640 1 -- 172.21.15.40:0/1812590862 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0fe0032050 con 0x555cece2f2f0 2024-03-18T22:51:30.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.907+0000 7f0fff5a5740 10 client.27659 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:30.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.907+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1812590862 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x555cecb717d0 con 0x555cece2f2f0 2024-03-18T22:51:30.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.907+0000 7f0fddffb640 1 -- 172.21.15.40:0/1812590862 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0fe00054d0 con 0x555cece2f2f0 2024-03-18T22:51:30.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.907+0000 7f0fddffb640 1 -- 172.21.15.40:0/1812590862 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f0fe0010430 con 0x555cece2f2f0 2024-03-18T22:51:30.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.907+0000 7f0fff5a5740 10 client.27659 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:30.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.907+0000 7f0fff5a5740 10 client.27659 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:51:30.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.907+0000 7f0fff5a5740 20 client.27659 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:30.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.907+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1812590862 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x555cece45420 0x555cece47830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.907+0000 7f0fff5a5740 4 client.27659 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:51:30.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.907+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1812590862 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555cecdf13e0 con 0x555cece45420 2024-03-18T22:51:30.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.907+0000 7f0ff659c640 1 --2- 172.21.15.40:0/1812590862 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x555cece45420 0x555cece47830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.908+0000 7f0fddffb640 10 client.27659 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:51:30.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.908+0000 7f0ff659c640 1 --2- 172.21.15.40:0/1812590862 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x555cece45420 0x555cece47830 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4444e8587ed1bfb5 server_cookie=82ad66c481b893e3 in_seq=0 out_seq=0 2024-03-18T22:51:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.908+0000 7f0fddffb640 1 -- 172.21.15.40:0/1812590862 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555cecdf13e0 con 0x555cece45420 2024-03-18T22:51:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.908+0000 7f0fddffb640 10 client.27659 handle_command_reply: tid=0 2024-03-18T22:51:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.908+0000 7f0fff5a5740 1 client.27659 shutdown 2024-03-18T22:51:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.909+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1812590862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece2f2f0 msgr2=0x555cece39830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.909+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1812590862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece2f2f0 0x555cece39830 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0004620 tx=0x7f0fe0011550 comp rx=0 tx=0).stop 2024-03-18T22:51:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.909+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1812590862 shutdown_connections 2024-03-18T22:51:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.909+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1812590862 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x555cece45420 0x555cece47830 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.909+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1812590862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece34e50 0x555cece40db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.909+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1812590862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece39d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.909+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1812590862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece2f2f0 0x555cece39830 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.909+0000 7f0fddffb640 0 client.27659 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:51:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.909+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1812590862 >> 172.21.15.40:0/1812590862 conn(0x555cece2a860 msgr2=0x555cece29ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.909+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1812590862 shutdown_connections 2024-03-18T22:51:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.909+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1812590862 wait complete. 2024-03-18T22:51:30.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.909+0000 7f0fff5a5740 20 client.27659 trim_cache size 0 max 16384 2024-03-18T22:51:30.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0fff5a5740 1 Processor -- start 2024-03-18T22:51:30.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0fff5a5740 1 -- start start 2024-03-18T22:51:30.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0fff5a5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece2f2f0 0x555cece39440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0fff5a5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece40d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0fff5a5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece34e50 0x555cece39980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0fff5a5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff81aa4d0 con 0x555cece2fce0 2024-03-18T22:51:30.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0fff5a5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff81ac070 con 0x555cece34e50 2024-03-18T22:51:30.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0fff5a5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff81aaba0 con 0x555cece2f2f0 2024-03-18T22:51:30.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0ff659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece40d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0ff659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece40d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49598/0 (socket says 172.21.15.40:49598) 2024-03-18T22:51:30.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0ff659c640 1 -- 172.21.15.40:0/1672052423 learned_addr learned my addr 172.21.15.40:0/1672052423 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0feffff640 1 --2- 172.21.15.40:0/1672052423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece2f2f0 0x555cece39440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0feffff640 1 --2- 172.21.15.40:0/1672052423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece34e50 0x555cece39980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0ff659c640 1 -- 172.21.15.40:0/1672052423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece2f2f0 msgr2=0x555cece39440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0ff659c640 1 --2- 172.21.15.40:0/1672052423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece2f2f0 0x555cece39440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0ff659c640 1 -- 172.21.15.40:0/1672052423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece34e50 msgr2=0x555cece39980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0ff659c640 1 --2- 172.21.15.40:0/1672052423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece34e50 0x555cece39980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0ff659c640 1 -- 172.21.15.40:0/1672052423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cece00880 con 0x555cece2fce0 2024-03-18T22:51:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.924+0000 7f0feffff640 1 --2- 172.21.15.40:0/1672052423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece34e50 0x555cece39980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:51:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.925+0000 7f0ff659c640 1 --2- 172.21.15.40:0/1672052423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece40d70 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8005ab0 tx=0x7f0fe8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac4526fd2f0987d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.925+0000 7f0fdffff640 1 -- 172.21.15.40:0/1672052423 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8005040 con 0x555cece2fce0 2024-03-18T22:51:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.925+0000 7f0fdffff640 1 -- 172.21.15.40:0/1672052423 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0fe80046c0 con 0x555cece2fce0 2024-03-18T22:51:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.925+0000 7f0fdffff640 1 -- 172.21.15.40:0/1672052423 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe80035f0 con 0x555cece2fce0 2024-03-18T22:51:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.925+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1672052423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 msgr2=0x555cece40d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.925+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1672052423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece40d70 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8005ab0 tx=0x7f0fe8005fd0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.925+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1672052423 shutdown_connections 2024-03-18T22:51:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.925+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1672052423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece34e50 0x555cece39980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1672052423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2fce0 0x555cece40d70 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1672052423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece2f2f0 0x555cece39440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1672052423 >> 172.21.15.40:0/1672052423 conn(0x555cece2a860 msgr2=0x555cece25a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1672052423 shutdown_connections 2024-03-18T22:51:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1672052423 wait complete. 2024-03-18T22:51:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0fff5a5740 1 -- start start 2024-03-18T22:51:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0fff5a5740 1 Processor -- start 2024-03-18T22:51:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0fff5a5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2f2f0 0x555cece3db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0fff5a5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece2fce0 0x555cece3e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0fff5a5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece34e50 0x555cecea9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0fff5a5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff81ab9a0 con 0x555cece2f2f0 2024-03-18T22:51:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0fff5a5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff81ab270 con 0x555cece34e50 2024-03-18T22:51:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0fff5a5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff81a84b0 con 0x555cece2fce0 2024-03-18T22:51:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0fff5a5740 10 client.0 _ll_register_callbacks cb 0x7f0ff00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:30.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0feffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2f2f0 0x555cece3db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0feffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2f2f0 0x555cece3db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49602/0 (socket says 172.21.15.40:49602) 2024-03-18T22:51:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0feffff640 1 -- 172.21.15.40:0/1161971245 learned_addr learned my addr 172.21.15.40:0/1161971245 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0ff659c640 1 --2- 172.21.15.40:0/1161971245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece2fce0 0x555cece3e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.926+0000 7f0feffff640 1 --2- 172.21.15.40:0/1161971245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece34e50 0x555cecea9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.927+0000 7f0feffff640 1 -- 172.21.15.40:0/1161971245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece2fce0 msgr2=0x555cece3e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.927+0000 7f0feffff640 1 --2- 172.21.15.40:0/1161971245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece2fce0 0x555cece3e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.927+0000 7f0feffff640 1 -- 172.21.15.40:0/1161971245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece34e50 msgr2=0x555cecea9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.927+0000 7f0feffff640 1 --2- 172.21.15.40:0/1161971245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece34e50 0x555cecea9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.927+0000 7f0feffff640 1 -- 172.21.15.40:0/1161971245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cece3e740 con 0x555cece2f2f0 2024-03-18T22:51:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.927+0000 7f0feffff640 1 --2- 172.21.15.40:0/1161971245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2f2f0 0x555cece3db60 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0010850 tx=0x7f0fe0011680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d975c5f5a05b8aa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.927+0000 7f0fddffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:51:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.927+0000 7f0fddffb640 1 -- 172.21.15.40:0/1161971245 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0010a40 con 0x555cece2f2f0 2024-03-18T22:51:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.927+0000 7f0fddffb640 1 -- 172.21.15.40:0/1161971245 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0fe0010be0 con 0x555cece2f2f0 2024-03-18T22:51:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.927+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1161971245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fe80046c0 con 0x555cece2f2f0 2024-03-18T22:51:30.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.927+0000 7f0fddffb640 1 -- 172.21.15.40:0/1161971245 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0fe000f790 con 0x555cece2f2f0 2024-03-18T22:51:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.927+0000 7f0fff5a5740 10 client.17850 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.927+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1161971245 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x555ceceaad40 con 0x555cece2f2f0 2024-03-18T22:51:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.927+0000 7f0fddffb640 1 -- 172.21.15.40:0/1161971245 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0fe80046c0 con 0x555cece2f2f0 2024-03-18T22:51:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.928+0000 7f0fddffb640 1 -- 172.21.15.40:0/1161971245 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f0fe0004cc0 con 0x555cece2f2f0 2024-03-18T22:51:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.928+0000 7f0fff5a5740 10 client.17850 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.928+0000 7f0fff5a5740 10 client.17850 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:51:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.928+0000 7f0fff5a5740 20 client.17850 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.928+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1161971245 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x555ceceab780 0x555ceceadba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.928+0000 7f0fff5a5740 4 client.17850 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:51:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.928+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1161971245 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555cecda09f0 con 0x555ceceab780 2024-03-18T22:51:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.928+0000 7f0ff659c640 1 --2- 172.21.15.40:0/1161971245 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x555ceceab780 0x555ceceadba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.928+0000 7f0ff659c640 1 --2- 172.21.15.40:0/1161971245 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x555ceceab780 0x555ceceadba0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=55a226c73ab8aa9b server_cookie=21c010fd0a37060d in_seq=0 out_seq=0 2024-03-18T22:51:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.928+0000 7f0fddffb640 10 client.17850 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:51:30.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.928+0000 7f0fddffb640 1 -- 172.21.15.40:0/1161971245 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555cecda09f0 con 0x555ceceab780 2024-03-18T22:51:30.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.928+0000 7f0fddffb640 10 client.17850 handle_command_reply: tid=0 2024-03-18T22:51:30.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.928+0000 7f0fff5a5740 1 client.17850 shutdown 2024-03-18T22:51:30.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.928+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1161971245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2f2f0 msgr2=0x555cece3db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.928+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1161971245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2f2f0 0x555cece3db60 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0010850 tx=0x7f0fe0011680 comp rx=0 tx=0).stop 2024-03-18T22:51:30.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1161971245 shutdown_connections 2024-03-18T22:51:30.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1161971245 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x555ceceab780 0x555ceceadba0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1161971245 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555cece34e50 0x555cecea9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1161971245 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555cece2fce0 0x555cece3e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0fff5a5740 1 --2- 172.21.15.40:0/1161971245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555cece2f2f0 0x555cece3db60 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1161971245 >> 172.21.15.40:0/1161971245 conn(0x555cece2a860 msgr2=0x555cece39440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0fddffb640 0 client.17850 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:51:30.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1161971245 shutdown_connections 2024-03-18T22:51:30.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0fff5a5740 1 -- 172.21.15.40:0/1161971245 wait complete. 2024-03-18T22:51:30.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0fff5a5740 20 client.17850 trim_cache size 0 max 16384 2024-03-18T22:51:30.936 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:51:30.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0ffc8dc640 1 -- 172.21.15.40:0/1306041028 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0fe40a2800 msgr2=0x7f0fe40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0ffc8dc640 1 --2- 172.21.15.40:0/1306041028 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0fe40a2800 0x7f0fe40a4cc0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0097f00 tx=0x7f0ff009e040 comp rx=0 tx=0).stop 2024-03-18T22:51:30.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0ffc8dc640 1 -- 172.21.15.40:0/1306041028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ff81033f0 msgr2=0x7f0ff819eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:30.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0ffc8dc640 1 --2- 172.21.15.40:0/1306041028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ff81033f0 0x7f0ff819eff0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009fd0 tx=0x7f0ff000ef10 comp rx=0 tx=0).stop 2024-03-18T22:51:30.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0ffc8dc640 1 -- 172.21.15.40:0/1306041028 shutdown_connections 2024-03-18T22:51:30.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0ffc8dc640 1 --2- 172.21.15.40:0/1306041028 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0fe40a2800 0x7f0fe40a4cc0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0ffc8dc640 1 --2- 172.21.15.40:0/1306041028 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ff81033f0 0x7f0ff819eff0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0ffc8dc640 1 --2- 172.21.15.40:0/1306041028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ff8102a30 0x7f0ff8197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0ffc8dc640 1 --2- 172.21.15.40:0/1306041028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ff8101840 0x7f0ff8197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:30.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0ffc8dc640 1 -- 172.21.15.40:0/1306041028 >> 172.21.15.40:0/1306041028 conn(0x7f0ff80fcdd0 msgr2=0x7f0ff8105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:30.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.929+0000 7f0ffc8dc640 1 -- 172.21.15.40:0/1306041028 shutdown_connections 2024-03-18T22:51:30.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:30.930+0000 7f0ffc8dc640 1 -- 172.21.15.40:0/1306041028 wait complete. 2024-03-18T22:51:30.944 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:51:31.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.056+0000 7f2d74177640 1 -- 172.21.15.40:0/1185418143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d6c107560 msgr2=0x7f2d6c109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.056+0000 7f2d74177640 1 --2- 172.21.15.40:0/1185418143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d6c107560 0x7f2d6c109950 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f2d64009ca0 tx=0x7f2d6402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.056+0000 7f2d74177640 1 -- 172.21.15.40:0/1185418143 shutdown_connections 2024-03-18T22:51:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.056+0000 7f2d74177640 1 --2- 172.21.15.40:0/1185418143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d6c107560 0x7f2d6c109950 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.056+0000 7f2d74177640 1 --2- 172.21.15.40:0/1185418143 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d6c104c30 0x7f2d6c107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.056+0000 7f2d74177640 1 --2- 172.21.15.40:0/1185418143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d6c102300 0x7f2d6c1046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.056+0000 7f2d74177640 1 -- 172.21.15.40:0/1185418143 >> 172.21.15.40:0/1185418143 conn(0x7f2d6c0fbc90 msgr2=0x7f2d6c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.056+0000 7f2d74177640 1 -- 172.21.15.40:0/1185418143 shutdown_connections 2024-03-18T22:51:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.056+0000 7f2d74177640 1 -- 172.21.15.40:0/1185418143 wait complete. 2024-03-18T22:51:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d74177640 1 Processor -- start 2024-03-18T22:51:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d74177640 1 -- start start 2024-03-18T22:51:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d74177640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d6c102300 0x7f2d6c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d74177640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d6c104c30 0x7f2d6c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d74177640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d6c107560 0x7f2d6c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d74177640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c10d370 con 0x7f2d6c102300 2024-03-18T22:51:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d74177640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c10d1f0 con 0x7f2d6c107560 2024-03-18T22:51:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d74177640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c10d4f0 con 0x7f2d6c104c30 2024-03-18T22:51:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d71f13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d6c102300 0x7f2d6c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d71f13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d6c102300 0x7f2d6c197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49626/0 (socket says 172.21.15.40:49626) 2024-03-18T22:51:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d71f13640 1 -- 172.21.15.40:0/1063011080 learned_addr learned my addr 172.21.15.40:0/1063011080 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d71712640 1 --2- 172.21.15.40:0/1063011080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d6c104c30 0x7f2d6c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d72714640 1 --2- 172.21.15.40:0/1063011080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d6c107560 0x7f2d6c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d71f13640 1 -- 172.21.15.40:0/1063011080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d6c104c30 msgr2=0x7f2d6c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d71f13640 1 --2- 172.21.15.40:0/1063011080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d6c104c30 0x7f2d6c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d71f13640 1 -- 172.21.15.40:0/1063011080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d6c107560 msgr2=0x7f2d6c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d71f13640 1 --2- 172.21.15.40:0/1063011080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d6c107560 0x7f2d6c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.057+0000 7f2d71f13640 1 -- 172.21.15.40:0/1063011080 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d6c19f650 con 0x7f2d6c102300 2024-03-18T22:51:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.058+0000 7f2d71f13640 1 --2- 172.21.15.40:0/1063011080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d6c102300 0x7f2d6c197a80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800ea60 tx=0x7f2d6800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.058+0000 7f2d62ffd640 1 -- 172.21.15.40:0/1063011080 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6800cde0 con 0x7f2d6c102300 2024-03-18T22:51:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.058+0000 7f2d74177640 1 -- 172.21.15.40:0/1063011080 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d6c19f940 con 0x7f2d6c102300 2024-03-18T22:51:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.058+0000 7f2d62ffd640 1 -- 172.21.15.40:0/1063011080 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d680035f0 con 0x7f2d6c102300 2024-03-18T22:51:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.058+0000 7f2d62ffd640 1 -- 172.21.15.40:0/1063011080 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d68010690 con 0x7f2d6c102300 2024-03-18T22:51:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.058+0000 7f2d74177640 1 -- 172.21.15.40:0/1063011080 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d6c19fe90 con 0x7f2d6c102300 2024-03-18T22:51:31.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.060+0000 7f2d62ffd640 1 -- 172.21.15.40:0/1063011080 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2d680037a0 con 0x7f2d6c102300 2024-03-18T22:51:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.061+0000 7f2d62ffd640 1 --2- 172.21.15.40:0/1063011080 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d580a2800 0x7f2d580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.061+0000 7f2d769fd740 1 Processor -- start 2024-03-18T22:51:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.061+0000 7f2d62ffd640 1 -- 172.21.15.40:0/1063011080 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2d680cf150 con 0x7f2d6c102300 2024-03-18T22:51:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.061+0000 7f2d71712640 1 --2- 172.21.15.40:0/1063011080 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d580a2800 0x7f2d580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.061+0000 7f2d769fd740 1 -- start start 2024-03-18T22:51:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.061+0000 7f2d769fd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c03710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.061+0000 7f2d769fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 0x55ed27c087e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.061+0000 7f2d769fd740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 0x55ed27c0b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.061+0000 7f2d769fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ed27b674b0 con 0x55ed27c03ce0 2024-03-18T22:51:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.061+0000 7f2d769fd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ed27b0fd70 con 0x55ed27c032f0 2024-03-18T22:51:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.061+0000 7f2d769fd740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55ed27b63550 con 0x55ed27c08e50 2024-03-18T22:51:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.061+0000 7f2d71f13640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c03710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.061+0000 7f2d72714640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 0x55ed27c0b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.061+0000 7f2d71f13640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c03710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:35502/0 (socket says 172.21.15.40:35502) 2024-03-18T22:51:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.061+0000 7f2d71f13640 1 -- 172.21.15.40:0/1474077766 learned_addr learned my addr 172.21.15.40:0/1474077766 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.062+0000 7f2d71712640 1 --2- 172.21.15.40:0/1474077766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 0x55ed27c087e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.062+0000 7f2d71712640 1 --2- 172.21.15.40:0/1063011080 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d580a2800 0x7f2d580a4cc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c198f40 tx=0x7f2d5c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.062+0000 7f2d71f13640 1 -- 172.21.15.40:0/1474077766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 msgr2=0x55ed27c0b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.062+0000 7f2d71f13640 1 --2- 172.21.15.40:0/1474077766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 0x55ed27c0b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.062+0000 7f2d71f13640 1 -- 172.21.15.40:0/1474077766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 msgr2=0x55ed27c087e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.062+0000 7f2d71f13640 1 --2- 172.21.15.40:0/1474077766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 0x55ed27c087e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.062+0000 7f2d71f13640 1 -- 172.21.15.40:0/1474077766 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed27b32210 con 0x55ed27c032f0 2024-03-18T22:51:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.062+0000 7f2d71712640 1 --2- 172.21.15.40:0/1474077766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 0x55ed27c087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:51:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.062+0000 7f2d72714640 1 --2- 172.21.15.40:0/1474077766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 0x55ed27c0b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.062+0000 7f2d71f13640 1 --2- 172.21.15.40:0/1474077766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c03710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2d6809b690 tx=0x7f2d6809b240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69f08fa4b90ceb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.062+0000 7f2d57fff640 1 -- 172.21.15.40:0/1474077766 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d680b4070 con 0x55ed27c032f0 2024-03-18T22:51:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.062+0000 7f2d57fff640 1 -- 172.21.15.40:0/1474077766 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d6809be00 con 0x55ed27c032f0 2024-03-18T22:51:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.062+0000 7f2d57fff640 1 -- 172.21.15.40:0/1474077766 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d68097d70 con 0x55ed27c032f0 2024-03-18T22:51:31.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.063+0000 7f2d769fd740 1 -- 172.21.15.40:0/1474077766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 msgr2=0x55ed27c03710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.063+0000 7f2d769fd740 1 --2- 172.21.15.40:0/1474077766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c03710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2d6809b690 tx=0x7f2d6809b240 comp rx=0 tx=0).stop 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.063+0000 7f2d769fd740 1 -- 172.21.15.40:0/1474077766 shutdown_connections 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.063+0000 7f2d769fd740 1 --2- 172.21.15.40:0/1474077766 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 0x55ed27c0b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.063+0000 7f2d769fd740 1 --2- 172.21.15.40:0/1474077766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 0x55ed27c087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.063+0000 7f2d769fd740 1 --2- 172.21.15.40:0/1474077766 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c03710 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.063+0000 7f2d769fd740 1 -- 172.21.15.40:0/1474077766 >> 172.21.15.40:0/1474077766 conn(0x55ed27bfe860 msgr2=0x55ed27bfec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.063+0000 7f2d769fd740 1 -- 172.21.15.40:0/1474077766 shutdown_connections 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.063+0000 7f2d769fd740 1 -- 172.21.15.40:0/1474077766 wait complete. 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.063+0000 7f2d769fd740 1 -- start start 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.063+0000 7f2d769fd740 1 Processor -- start 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d769fd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c0d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d769fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 0x55ed27c0dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d769fd740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 0x55ed27c14d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d769fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ed27c0be00 con 0x55ed27c03ce0 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d769fd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ed27b23310 con 0x55ed27c032f0 2024-03-18T22:51:31.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d769fd740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55ed27c0bf80 con 0x55ed27c08e50 2024-03-18T22:51:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d769fd740 10 client.0 _ll_register_callbacks cb 0x55ed27b172d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d71712640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 0x55ed27c0dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d71f13640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c0d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d71712640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 0x55ed27c0dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49642/0 (socket says 172.21.15.40:49642) 2024-03-18T22:51:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d71712640 1 -- 172.21.15.40:0/1590190785 learned_addr learned my addr 172.21.15.40:0/1590190785 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d72714640 1 --2- 172.21.15.40:0/1590190785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 0x55ed27c14d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d71f13640 1 -- 172.21.15.40:0/1590190785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 msgr2=0x55ed27c14d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d71f13640 1 --2- 172.21.15.40:0/1590190785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 0x55ed27c14d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d71f13640 1 -- 172.21.15.40:0/1590190785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 msgr2=0x55ed27c0dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d71f13640 1 --2- 172.21.15.40:0/1590190785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 0x55ed27c0dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.064+0000 7f2d71f13640 1 -- 172.21.15.40:0/1590190785 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed2787b3f0 con 0x55ed27c032f0 2024-03-18T22:51:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d71f13640 1 --2- 172.21.15.40:0/1590190785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c0d7f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x55ed27c0ba50 tx=0x7f2d6809f970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7736796cd5e3a309 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d557fa640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:51:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d557fa640 1 -- 172.21.15.40:0/1590190785 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d680b4070 con 0x55ed27c032f0 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d557fa640 1 -- 172.21.15.40:0/1590190785 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2d6809ab50 con 0x55ed27c032f0 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d769fd740 1 -- 172.21.15.40:0/1590190785 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d6809be00 con 0x55ed27c032f0 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d557fa640 1 -- 172.21.15.40:0/1590190785 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2d6809ad50 con 0x55ed27c032f0 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d769fd740 10 client.27677 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d769fd740 1 -- 172.21.15.40:0/1590190785 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ed279457d0 con 0x55ed27c032f0 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d557fa640 1 -- 172.21.15.40:0/1590190785 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d6809be00 con 0x55ed27c032f0 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d557fa640 1 -- 172.21.15.40:0/1590190785 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f2d68097450 con 0x55ed27c032f0 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d769fd740 10 client.27677 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d769fd740 10 client.27677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d769fd740 20 client.27677 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d769fd740 1 --2- 172.21.15.40:0/1590190785 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ed27c19260 0x55ed27c1b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d769fd740 4 client.27677 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.065+0000 7f2d769fd740 1 -- 172.21.15.40:0/1590190785 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed27bc53e0 con 0x55ed27c19260 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.066+0000 7f2d71712640 1 --2- 172.21.15.40:0/1590190785 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ed27c19260 0x55ed27c1b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.066+0000 7f2d557fa640 10 client.27677 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.066+0000 7f2d71712640 1 --2- 172.21.15.40:0/1590190785 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ed27c19260 0x55ed27c1b650 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7ca9ed9afdef744 server_cookie=c46eff4594b9b492 in_seq=0 out_seq=0 2024-03-18T22:51:31.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.067+0000 7f2d557fa640 1 -- 172.21.15.40:0/1590190785 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ed27bc53e0 con 0x55ed27c19260 2024-03-18T22:51:31.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.067+0000 7f2d557fa640 10 client.27677 handle_command_reply: tid=0 2024-03-18T22:51:31.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.067+0000 7f2d769fd740 1 client.27677 shutdown 2024-03-18T22:51:31.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.067+0000 7f2d769fd740 1 -- 172.21.15.40:0/1590190785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 msgr2=0x55ed27c0d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:31.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.067+0000 7f2d769fd740 1 --2- 172.21.15.40:0/1590190785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c0d7f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x55ed27c0ba50 tx=0x7f2d6809f970 comp rx=0 tx=0).stop 2024-03-18T22:51:31.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.067+0000 7f2d769fd740 1 -- 172.21.15.40:0/1590190785 shutdown_connections 2024-03-18T22:51:31.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.067+0000 7f2d769fd740 1 --2- 172.21.15.40:0/1590190785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 0x55ed27c14d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.067+0000 7f2d769fd740 1 --2- 172.21.15.40:0/1590190785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 0x55ed27c0dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.067+0000 7f2d769fd740 1 --2- 172.21.15.40:0/1590190785 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ed27c19260 0x55ed27c1b650 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.067+0000 7f2d769fd740 1 --2- 172.21.15.40:0/1590190785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c0d7f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.067+0000 7f2d557fa640 0 client.27677 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:31.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.067+0000 7f2d769fd740 1 -- 172.21.15.40:0/1590190785 >> 172.21.15.40:0/1590190785 conn(0x55ed27bfe860 msgr2=0x55ed27bfdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:31.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.067+0000 7f2d769fd740 1 -- 172.21.15.40:0/1590190785 shutdown_connections 2024-03-18T22:51:31.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.067+0000 7f2d769fd740 1 -- 172.21.15.40:0/1590190785 wait complete. 2024-03-18T22:51:31.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.067+0000 7f2d769fd740 20 client.27677 trim_cache size 0 max 16384 2024-03-18T22:51:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.081+0000 7f2d769fd740 1 Processor -- start 2024-03-18T22:51:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.081+0000 7f2d769fd740 1 -- start start 2024-03-18T22:51:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.081+0000 7f2d769fd740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c032f0 0x55ed27c0db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.081+0000 7f2d769fd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c03ce0 0x55ed27c14d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.081+0000 7f2d769fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c08e50 0x55ed27bfd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.081+0000 7f2d769fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c1aa640 con 0x55ed27c08e50 2024-03-18T22:51:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.081+0000 7f2d769fd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c1ac1e0 con 0x55ed27c03ce0 2024-03-18T22:51:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.081+0000 7f2d769fd740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c1aad10 con 0x55ed27c032f0 2024-03-18T22:51:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.081+0000 7f2d71f13640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c032f0 0x55ed27c0db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.081+0000 7f2d72714640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c08e50 0x55ed27bfd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.081+0000 7f2d71712640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c03ce0 0x55ed27c14d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.081+0000 7f2d71f13640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c032f0 0x55ed27c0db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:54062/0 (socket says 172.21.15.40:54062) 2024-03-18T22:51:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.081+0000 7f2d71f13640 1 -- 172.21.15.40:0/348451902 learned_addr learned my addr 172.21.15.40:0/348451902 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.081+0000 7f2d72714640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c08e50 0x55ed27bfd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49654/0 (socket says 172.21.15.40:49654) 2024-03-18T22:51:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.082+0000 7f2d71f13640 1 -- 172.21.15.40:0/348451902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c03ce0 msgr2=0x55ed27c14d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.082+0000 7f2d71f13640 1 --2- 172.21.15.40:0/348451902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c03ce0 0x55ed27c14d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.082+0000 7f2d71f13640 1 -- 172.21.15.40:0/348451902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c08e50 msgr2=0x55ed27bfd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.082+0000 7f2d71f13640 1 --2- 172.21.15.40:0/348451902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c08e50 0x55ed27bfd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.082+0000 7f2d71f13640 1 -- 172.21.15.40:0/348451902 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed27bd4880 con 0x55ed27c032f0 2024-03-18T22:51:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.082+0000 7f2d72714640 1 --2- 172.21.15.40:0/348451902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c08e50 0x55ed27bfd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.082+0000 7f2d71f13640 1 --2- 172.21.15.40:0/348451902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c032f0 0x55ed27c0db10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2d680980d0 tx=0x7f2d6809b160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6aaf1ef9f37a97f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.082+0000 7f2d577fe640 1 -- 172.21.15.40:0/348451902 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d680b4040 con 0x55ed27c032f0 2024-03-18T22:51:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.082+0000 7f2d577fe640 1 -- 172.21.15.40:0/348451902 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d6809be00 con 0x55ed27c032f0 2024-03-18T22:51:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.082+0000 7f2d577fe640 1 -- 172.21.15.40:0/348451902 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d68097d70 con 0x55ed27c032f0 2024-03-18T22:51:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 -- 172.21.15.40:0/348451902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c032f0 msgr2=0x55ed27c0db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 --2- 172.21.15.40:0/348451902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c032f0 0x55ed27c0db10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2d680980d0 tx=0x7f2d6809b160 comp rx=0 tx=0).stop 2024-03-18T22:51:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 -- 172.21.15.40:0/348451902 shutdown_connections 2024-03-18T22:51:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 --2- 172.21.15.40:0/348451902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c08e50 0x55ed27bfd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 --2- 172.21.15.40:0/348451902 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c03ce0 0x55ed27c14d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 --2- 172.21.15.40:0/348451902 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c032f0 0x55ed27c0db10 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 -- 172.21.15.40:0/348451902 >> 172.21.15.40:0/348451902 conn(0x55ed27bfe860 msgr2=0x55ed27c0d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 -- 172.21.15.40:0/348451902 shutdown_connections 2024-03-18T22:51:31.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 -- 172.21.15.40:0/348451902 wait complete. 2024-03-18T22:51:31.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 -- start start 2024-03-18T22:51:31.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 Processor -- start 2024-03-18T22:51:31.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c121a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 0x55ed27c126e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 0x55ed27c136f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c1abb10 con 0x55ed27c03ce0 2024-03-18T22:51:31.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c1ab3e0 con 0x55ed27c032f0 2024-03-18T22:51:31.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6c1a8620 con 0x55ed27c08e50 2024-03-18T22:51:31.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d769fd740 10 client.0 _ll_register_callbacks cb 0x7f2d680b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:31.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d71f13640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c121a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d71712640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 0x55ed27c126e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d72714640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 0x55ed27c136f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.083+0000 7f2d71f13640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c121a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:35526/0 (socket says 172.21.15.40:35526) 2024-03-18T22:51:31.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.084+0000 7f2d71f13640 1 -- 172.21.15.40:0/3988549213 learned_addr learned my addr 172.21.15.40:0/3988549213 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:31.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.084+0000 7f2d71712640 1 -- 172.21.15.40:0/3988549213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 msgr2=0x55ed27c136f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:31.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.084+0000 7f2d71712640 1 --2- 172.21.15.40:0/3988549213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 0x55ed27c136f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.084+0000 7f2d71712640 1 -- 172.21.15.40:0/3988549213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 msgr2=0x55ed27c121a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:31.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.084+0000 7f2d71712640 1 --2- 172.21.15.40:0/3988549213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c121a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.084+0000 7f2d71712640 1 -- 172.21.15.40:0/3988549213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed27c13ce0 con 0x55ed27c03ce0 2024-03-18T22:51:31.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.084+0000 7f2d71712640 1 --2- 172.21.15.40:0/3988549213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 0x55ed27c126e0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x55ed27c0e830 tx=0x7f2d5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb1385a45140d8d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:31.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.084+0000 7f2d557fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:51:31.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.084+0000 7f2d557fa640 1 -- 172.21.15.40:0/3988549213 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c029da0 con 0x55ed27c03ce0 2024-03-18T22:51:31.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.084+0000 7f2d557fa640 1 -- 172.21.15.40:0/3988549213 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2d5c029380 con 0x55ed27c03ce0 2024-03-18T22:51:31.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.084+0000 7f2d769fd740 1 -- 172.21.15.40:0/3988549213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d6809be00 con 0x55ed27c03ce0 2024-03-18T22:51:31.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.084+0000 7f2d557fa640 1 -- 172.21.15.40:0/3988549213 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2d5c029580 con 0x55ed27c03ce0 2024-03-18T22:51:31.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.084+0000 7f2d769fd740 10 client.17868 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:31.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.084+0000 7f2d769fd740 1 -- 172.21.15.40:0/3988549213 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ed27c7f340 con 0x55ed27c03ce0 2024-03-18T22:51:31.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.085+0000 7f2d557fa640 1 -- 172.21.15.40:0/3988549213 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d6809be00 con 0x55ed27c03ce0 2024-03-18T22:51:31.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.085+0000 7f2d557fa640 1 -- 172.21.15.40:0/3988549213 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f2d5c0048a0 con 0x55ed27c03ce0 2024-03-18T22:51:31.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.085+0000 7f2d769fd740 10 client.17868 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:31.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.085+0000 7f2d769fd740 10 client.17868 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:51:31.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.085+0000 7f2d769fd740 20 client.17868 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:31.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.085+0000 7f2d769fd740 1 --2- 172.21.15.40:0/3988549213 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ed27c7fb30 0x55ed27c81f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:31.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.085+0000 7f2d769fd740 4 client.17868 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:51:31.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.085+0000 7f2d769fd740 1 -- 172.21.15.40:0/3988549213 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ed27b749f0 con 0x55ed27c7fb30 2024-03-18T22:51:31.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.085+0000 7f2d71f13640 1 --2- 172.21.15.40:0/3988549213 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ed27c7fb30 0x55ed27c81f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:31.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.085+0000 7f2d71f13640 1 --2- 172.21.15.40:0/3988549213 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ed27c7fb30 0x55ed27c81f50 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=687a3f96c1e2186a server_cookie=e8c5a909eea6bbcb in_seq=0 out_seq=0 2024-03-18T22:51:31.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.085+0000 7f2d557fa640 10 client.17868 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:31.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.085+0000 7f2d557fa640 1 -- 172.21.15.40:0/3988549213 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ed27b749f0 con 0x55ed27c7fb30 2024-03-18T22:51:31.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.085+0000 7f2d557fa640 10 client.17868 handle_command_reply: tid=0 2024-03-18T22:51:31.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.085+0000 7f2d769fd740 1 client.17868 shutdown 2024-03-18T22:51:31.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d769fd740 1 -- 172.21.15.40:0/3988549213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 msgr2=0x55ed27c126e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:31.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d769fd740 1 --2- 172.21.15.40:0/3988549213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 0x55ed27c126e0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x55ed27c0e830 tx=0x7f2d5c004730 comp rx=0 tx=0).stop 2024-03-18T22:51:31.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d769fd740 1 -- 172.21.15.40:0/3988549213 shutdown_connections 2024-03-18T22:51:31.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d769fd740 1 --2- 172.21.15.40:0/3988549213 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ed27c08e50 0x55ed27c136f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d769fd740 1 --2- 172.21.15.40:0/3988549213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed27c03ce0 0x55ed27c126e0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d769fd740 1 --2- 172.21.15.40:0/3988549213 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ed27c7fb30 0x55ed27c81f50 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d769fd740 1 --2- 172.21.15.40:0/3988549213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ed27c032f0 0x55ed27c121a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d557fa640 0 client.17868 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:31.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d769fd740 1 -- 172.21.15.40:0/3988549213 >> 172.21.15.40:0/3988549213 conn(0x55ed27bfe860 msgr2=0x55ed27c01ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:31.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d769fd740 1 -- 172.21.15.40:0/3988549213 shutdown_connections 2024-03-18T22:51:31.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d769fd740 1 -- 172.21.15.40:0/3988549213 wait complete. 2024-03-18T22:51:31.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d769fd740 20 client.17868 trim_cache size 0 max 16384 2024-03-18T22:51:31.093 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:51:31.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d74177640 1 -- 172.21.15.40:0/1063011080 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d580a2800 msgr2=0x7f2d580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:31.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d74177640 1 --2- 172.21.15.40:0/1063011080 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d580a2800 0x7f2d580a4cc0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c198f40 tx=0x7f2d5c005f90 comp rx=0 tx=0).stop 2024-03-18T22:51:31.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d74177640 1 -- 172.21.15.40:0/1063011080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d6c102300 msgr2=0x7f2d6c197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:31.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d74177640 1 --2- 172.21.15.40:0/1063011080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d6c102300 0x7f2d6c197a80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800ea60 tx=0x7f2d6800ef20 comp rx=0 tx=0).stop 2024-03-18T22:51:31.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d74177640 1 -- 172.21.15.40:0/1063011080 shutdown_connections 2024-03-18T22:51:31.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d74177640 1 --2- 172.21.15.40:0/1063011080 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f2d580a2800 0x7f2d580a4cc0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d74177640 1 --2- 172.21.15.40:0/1063011080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d6c107560 0x7f2d6c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d74177640 1 --2- 172.21.15.40:0/1063011080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2d6c104c30 0x7f2d6c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d74177640 1 --2- 172.21.15.40:0/1063011080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2d6c102300 0x7f2d6c197a80 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:31.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d74177640 1 -- 172.21.15.40:0/1063011080 >> 172.21.15.40:0/1063011080 conn(0x7f2d6c0fbc90 msgr2=0x7f2d6c102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:31.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d74177640 1 -- 172.21.15.40:0/1063011080 shutdown_connections 2024-03-18T22:51:31.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:31.086+0000 7f2d74177640 1 -- 172.21.15.40:0/1063011080 wait complete. 2024-03-18T22:51:31.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:31 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3739831223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:51:31.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:31 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3739831223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:51:31.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:31 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3739831223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:51:32.100 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:51:32.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:51:32.100 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:51:32.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.222+0000 7fdc214dc640 1 -- 172.21.15.40:0/2170851974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc1c100720 msgr2=0x7fdc1c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.222+0000 7fdc214dc640 1 --2- 172.21.15.40:0/2170851974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc1c100720 0x7fdc1c100b40 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009900 tx=0x7fdc0c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:51:32.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.222+0000 7fdc214dc640 1 -- 172.21.15.40:0/2170851974 shutdown_connections 2024-03-18T22:51:32.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.222+0000 7fdc214dc640 1 --2- 172.21.15.40:0/2170851974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc1c1022d0 0x7fdc1c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.222+0000 7fdc214dc640 1 --2- 172.21.15.40:0/2170851974 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc1c101910 0x7fdc1c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.222+0000 7fdc214dc640 1 --2- 172.21.15.40:0/2170851974 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc1c100720 0x7fdc1c100b40 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.222+0000 7fdc214dc640 1 -- 172.21.15.40:0/2170851974 >> 172.21.15.40:0/2170851974 conn(0x7fdc1c0fbcb0 msgr2=0x7fdc1c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:32.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.223+0000 7fdc214dc640 1 -- 172.21.15.40:0/2170851974 shutdown_connections 2024-03-18T22:51:32.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.223+0000 7fdc214dc640 1 -- 172.21.15.40:0/2170851974 wait complete. 2024-03-18T22:51:32.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.223+0000 7fdc214dc640 1 Processor -- start 2024-03-18T22:51:32.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.223+0000 7fdc214dc640 1 -- start start 2024-03-18T22:51:32.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.223+0000 7fdc214dc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc1c100720 0x7fdc1c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.223+0000 7fdc214dc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc1c101910 0x7fdc1c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.223+0000 7fdc214dc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc1c1022d0 0x7fdc1c19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.223+0000 7fdc214dc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c10c240 con 0x7fdc1c1022d0 2024-03-18T22:51:32.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.223+0000 7fdc214dc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c10c0c0 con 0x7fdc1c101910 2024-03-18T22:51:32.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.223+0000 7fdc214dc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c10c3c0 con 0x7fdc1c100720 2024-03-18T22:51:32.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.223+0000 7fdc1b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc1c1022d0 0x7fdc1c19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.223+0000 7fdc1affd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc1c100720 0x7fdc1c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc1affd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc1c100720 0x7fdc1c197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:54098/0 (socket says 172.21.15.40:54098) 2024-03-18T22:51:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc1affd640 1 -- 172.21.15.40:0/58466637 learned_addr learned my addr 172.21.15.40:0/58466637 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc1b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc1c1022d0 0x7fdc1c19f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49682/0 (socket says 172.21.15.40:49682) 2024-03-18T22:51:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc13fff640 1 --2- 172.21.15.40:0/58466637 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc1c101910 0x7fdc1c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc1affd640 1 -- 172.21.15.40:0/58466637 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc1c101910 msgr2=0x7fdc1c1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc1affd640 1 --2- 172.21.15.40:0/58466637 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc1c101910 0x7fdc1c1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc1affd640 1 -- 172.21.15.40:0/58466637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc1c1022d0 msgr2=0x7fdc1c19f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc1affd640 1 --2- 172.21.15.40:0/58466637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc1c1022d0 0x7fdc1c19f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc1affd640 1 -- 172.21.15.40:0/58466637 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c19f740 con 0x7fdc1c100720 2024-03-18T22:51:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc13fff640 1 --2- 172.21.15.40:0/58466637 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc1c101910 0x7fdc1c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc1b7fe640 1 --2- 172.21.15.40:0/58466637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc1c1022d0 0x7fdc1c19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc1affd640 1 --2- 172.21.15.40:0/58466637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc1c100720 0x7fdc1c197b70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c000c00 tx=0x7fdc0c004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc18ff9640 1 -- 172.21.15.40:0/58466637 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c043020 con 0x7fdc1c100720 2024-03-18T22:51:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc18ff9640 1 -- 172.21.15.40:0/58466637 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc0c004440 con 0x7fdc1c100720 2024-03-18T22:51:32.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc214dc640 1 -- 172.21.15.40:0/58466637 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc1c19f9d0 con 0x7fdc1c100720 2024-03-18T22:51:32.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc18ff9640 1 -- 172.21.15.40:0/58466637 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c003c90 con 0x7fdc1c100720 2024-03-18T22:51:32.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.224+0000 7fdc214dc640 1 -- 172.21.15.40:0/58466637 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc1c19ff70 con 0x7fdc1c100720 2024-03-18T22:51:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.226+0000 7fdc18ff9640 1 -- 172.21.15.40:0/58466637 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdc0c046020 con 0x7fdc1c100720 2024-03-18T22:51:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.227+0000 7fdc18ff9640 1 --2- 172.21.15.40:0/58466637 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc080a2880 0x7fdc080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.227+0000 7fdc13fff640 1 --2- 172.21.15.40:0/58466637 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc080a2880 0x7fdc080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.227+0000 7fdc18ff9640 1 -- 172.21.15.40:0/58466637 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdc0c0f0fa0 con 0x7fdc1c100720 2024-03-18T22:51:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.227+0000 7fdc241a2740 1 Processor -- start 2024-03-18T22:51:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.227+0000 7fdc241a2740 1 -- start start 2024-03-18T22:51:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.227+0000 7fdc13fff640 1 --2- 172.21.15.40:0/58466637 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc080a2880 0x7fdc080a4d40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c199030 tx=0x7fdc04005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc241a2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a1883a0 0x557e0a1887c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc241a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a188d90 0x557e0a18d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc241a2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a18df00 0x557e0a190390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc241a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557e0a0ed160 con 0x557e0a188d90 2024-03-18T22:51:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc241a2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557e0a095e10 con 0x557e0a1883a0 2024-03-18T22:51:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc241a2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557e0a0e9200 con 0x557e0a18df00 2024-03-18T22:51:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc13fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a188d90 0x557e0a18d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc13fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a188d90 0x557e0a18d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49692/0 (socket says 172.21.15.40:49692) 2024-03-18T22:51:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc13fff640 1 -- 172.21.15.40:0/2447880322 learned_addr learned my addr 172.21.15.40:0/2447880322 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc1affd640 1 --2- 172.21.15.40:0/2447880322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a1883a0 0x557e0a1887c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc1b7fe640 1 --2- 172.21.15.40:0/2447880322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a18df00 0x557e0a190390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc13fff640 1 -- 172.21.15.40:0/2447880322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a18df00 msgr2=0x557e0a190390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc13fff640 1 --2- 172.21.15.40:0/2447880322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a18df00 0x557e0a190390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc13fff640 1 -- 172.21.15.40:0/2447880322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a1883a0 msgr2=0x557e0a1887c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc13fff640 1 --2- 172.21.15.40:0/2447880322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a1883a0 0x557e0a1887c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc13fff640 1 -- 172.21.15.40:0/2447880322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e09ec97e0 con 0x557e0a188d90 2024-03-18T22:51:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc1affd640 1 --2- 172.21.15.40:0/2447880322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a1883a0 0x557e0a1887c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.228+0000 7fdc13fff640 1 --2- 172.21.15.40:0/2447880322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a188d90 0x557e0a18d890 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fdc040042b0 tx=0x7fdc04014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2e8b5480c145fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.229+0000 7fdc117fa640 1 -- 172.21.15.40:0/2447880322 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04026480 con 0x557e0a188d90 2024-03-18T22:51:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.229+0000 7fdc117fa640 1 -- 172.21.15.40:0/2447880322 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc0400f930 con 0x557e0a188d90 2024-03-18T22:51:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.229+0000 7fdc117fa640 1 -- 172.21.15.40:0/2447880322 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0401c8b0 con 0x557e0a188d90 2024-03-18T22:51:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.229+0000 7fdc241a2740 1 -- 172.21.15.40:0/2447880322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a188d90 msgr2=0x557e0a18d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.229+0000 7fdc241a2740 1 --2- 172.21.15.40:0/2447880322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a188d90 0x557e0a18d890 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fdc040042b0 tx=0x7fdc04014000 comp rx=0 tx=0).stop 2024-03-18T22:51:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 1 -- 172.21.15.40:0/2447880322 shutdown_connections 2024-03-18T22:51:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 1 --2- 172.21.15.40:0/2447880322 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a18df00 0x557e0a190390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 1 --2- 172.21.15.40:0/2447880322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a188d90 0x557e0a18d890 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 1 --2- 172.21.15.40:0/2447880322 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a1883a0 0x557e0a1887c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 1 -- 172.21.15.40:0/2447880322 >> 172.21.15.40:0/2447880322 conn(0x557e0a183910 msgr2=0x557e0a183d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:32.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 1 -- 172.21.15.40:0/2447880322 shutdown_connections 2024-03-18T22:51:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 1 -- 172.21.15.40:0/2447880322 wait complete. 2024-03-18T22:51:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 1 -- start start 2024-03-18T22:51:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 1 Processor -- start 2024-03-18T22:51:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a1883a0 0x557e0a192830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a188d90 0x557e0a192d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a18df00 0x557e0a199db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557e0a190eb0 con 0x557e0a188d90 2024-03-18T22:51:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557e0a0a91a0 con 0x557e0a18df00 2024-03-18T22:51:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557e0a191030 con 0x557e0a1883a0 2024-03-18T22:51:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc241a2740 10 client.0 _ll_register_callbacks cb 0x557e09e00c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc1affd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a1883a0 0x557e0a192830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.230+0000 7fdc13fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a188d90 0x557e0a192d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc13fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a188d90 0x557e0a192d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49700/0 (socket says 172.21.15.40:49700) 2024-03-18T22:51:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc13fff640 1 -- 172.21.15.40:0/2336533387 learned_addr learned my addr 172.21.15.40:0/2336533387 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:32.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc1affd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a1883a0 0x557e0a192830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:54112/0 (socket says 172.21.15.40:54112) 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc1b7fe640 1 --2- 172.21.15.40:0/2336533387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a18df00 0x557e0a199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc1affd640 1 -- 172.21.15.40:0/2336533387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a18df00 msgr2=0x557e0a199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc1affd640 1 --2- 172.21.15.40:0/2336533387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a18df00 0x557e0a199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc1affd640 1 -- 172.21.15.40:0/2336533387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a188d90 msgr2=0x557e0a192d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc1affd640 1 --2- 172.21.15.40:0/2336533387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a188d90 0x557e0a192d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc1affd640 1 -- 172.21.15.40:0/2336533387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e09f29f70 con 0x557e0a1883a0 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc1affd640 1 --2- 172.21.15.40:0/2336533387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a1883a0 0x557e0a192830 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0eea20 tx=0x7fdc0c0bd110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c54f190b82b17ab4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc027fc640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc027fc640 1 -- 172.21.15.40:0/2336533387 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c0bdc50 con 0x557e0a1883a0 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc027fc640 1 -- 172.21.15.40:0/2336533387 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdc0c005590 con 0x557e0a1883a0 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc241a2740 1 -- 172.21.15.40:0/2336533387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc0400f930 con 0x557e0a1883a0 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc027fc640 1 -- 172.21.15.40:0/2336533387 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fdc0c0bd550 con 0x557e0a1883a0 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc241a2740 10 client.27685 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.231+0000 7fdc241a2740 1 -- 172.21.15.40:0/2336533387 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557e09dc1170 con 0x557e0a1883a0 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.232+0000 7fdc027fc640 1 -- 172.21.15.40:0/2336533387 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc0400f930 con 0x557e0a1883a0 2024-03-18T22:51:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.232+0000 7fdc027fc640 1 -- 172.21.15.40:0/2336533387 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7fdc0c0c5860 con 0x557e0a1883a0 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.232+0000 7fdc241a2740 10 client.27685 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.232+0000 7fdc241a2740 10 client.27685 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.232+0000 7fdc241a2740 20 client.27685 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.232+0000 7fdc241a2740 1 --2- 172.21.15.40:0/2336533387 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557e0a19e630 0x557e0a1a0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.232+0000 7fdc241a2740 4 client.27685 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.232+0000 7fdc241a2740 1 -- 172.21.15.40:0/2336533387 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e0a0fa6a0 con 0x557e0a19e630 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.232+0000 7fdc13fff640 1 --2- 172.21.15.40:0/2336533387 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557e0a19e630 0x557e0a1a0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.232+0000 7fdc027fc640 10 client.27685 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.232+0000 7fdc13fff640 1 --2- 172.21.15.40:0/2336533387 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557e0a19e630 0x557e0a1a0aa0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a14daf246116bf0d server_cookie=77197bd8f7bf9a23 in_seq=0 out_seq=0 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.233+0000 7fdc027fc640 1 -- 172.21.15.40:0/2336533387 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557e0a0fa6a0 con 0x557e0a19e630 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.233+0000 7fdc027fc640 10 client.27685 handle_command_reply: tid=0 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.233+0000 7fdc241a2740 1 client.27685 shutdown 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.233+0000 7fdc241a2740 1 -- 172.21.15.40:0/2336533387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a1883a0 msgr2=0x557e0a192830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.233+0000 7fdc241a2740 1 --2- 172.21.15.40:0/2336533387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a1883a0 0x557e0a192830 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0eea20 tx=0x7fdc0c0bd110 comp rx=0 tx=0).stop 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.233+0000 7fdc241a2740 1 -- 172.21.15.40:0/2336533387 shutdown_connections 2024-03-18T22:51:32.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.233+0000 7fdc241a2740 1 --2- 172.21.15.40:0/2336533387 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557e0a19e630 0x557e0a1a0aa0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.233+0000 7fdc241a2740 1 --2- 172.21.15.40:0/2336533387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a18df00 0x557e0a199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.233+0000 7fdc241a2740 1 --2- 172.21.15.40:0/2336533387 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a188d90 0x557e0a192d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.233+0000 7fdc027fc640 0 client.27685 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.233+0000 7fdc241a2740 1 --2- 172.21.15.40:0/2336533387 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a1883a0 0x557e0a192830 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.233+0000 7fdc241a2740 1 -- 172.21.15.40:0/2336533387 >> 172.21.15.40:0/2336533387 conn(0x557e0a183910 msgr2=0x557e0a182b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.233+0000 7fdc241a2740 1 -- 172.21.15.40:0/2336533387 shutdown_connections 2024-03-18T22:51:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.233+0000 7fdc241a2740 1 -- 172.21.15.40:0/2336533387 wait complete. 2024-03-18T22:51:32.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.233+0000 7fdc241a2740 20 client.27685 trim_cache size 0 max 16384 2024-03-18T22:51:32.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.256+0000 7fdc241a2740 1 Processor -- start 2024-03-18T22:51:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.256+0000 7fdc241a2740 1 -- start start 2024-03-18T22:51:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.256+0000 7fdc241a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a1883a0 0x557e0a192cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.256+0000 7fdc241a2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a188d90 0x557e0a199d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.256+0000 7fdc241a2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a18df00 0x557e0a187280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.256+0000 7fdc241a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1aa590 con 0x557e0a1883a0 2024-03-18T22:51:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.256+0000 7fdc241a2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1ac130 con 0x557e0a188d90 2024-03-18T22:51:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.256+0000 7fdc241a2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1aac60 con 0x557e0a18df00 2024-03-18T22:51:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.256+0000 7fdc1affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a1883a0 0x557e0a192cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.256+0000 7fdc1affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a1883a0 0x557e0a192cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49710/0 (socket says 172.21.15.40:49710) 2024-03-18T22:51:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.256+0000 7fdc1affd640 1 -- 172.21.15.40:0/546636683 learned_addr learned my addr 172.21.15.40:0/546636683 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.256+0000 7fdc13fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a188d90 0x557e0a199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.256+0000 7fdc1b7fe640 1 --2- 172.21.15.40:0/546636683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a18df00 0x557e0a187280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.257+0000 7fdc1affd640 1 -- 172.21.15.40:0/546636683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a18df00 msgr2=0x557e0a187280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.257+0000 7fdc1affd640 1 --2- 172.21.15.40:0/546636683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a18df00 0x557e0a187280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.257+0000 7fdc1affd640 1 -- 172.21.15.40:0/546636683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a188d90 msgr2=0x557e0a199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.257+0000 7fdc1affd640 1 --2- 172.21.15.40:0/546636683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a188d90 0x557e0a199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.257+0000 7fdc1affd640 1 -- 172.21.15.40:0/546636683 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e0a1834b0 con 0x557e0a1883a0 2024-03-18T22:51:32.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.257+0000 7fdc13fff640 1 --2- 172.21.15.40:0/546636683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a188d90 0x557e0a199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:32.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.257+0000 7fdc1affd640 1 --2- 172.21.15.40:0/546636683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a1883a0 0x557e0a192cd0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0b99b0 tx=0x7fdc0c0bba60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ba805145942577 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:32.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.257+0000 7fdc10ff9640 1 -- 172.21.15.40:0/546636683 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c0bdc50 con 0x557e0a1883a0 2024-03-18T22:51:32.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.257+0000 7fdc10ff9640 1 -- 172.21.15.40:0/546636683 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc0c0c16a0 con 0x557e0a1883a0 2024-03-18T22:51:32.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.257+0000 7fdc10ff9640 1 -- 172.21.15.40:0/546636683 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c005600 con 0x557e0a1883a0 2024-03-18T22:51:32.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.258+0000 7fdc241a2740 1 -- 172.21.15.40:0/546636683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a1883a0 msgr2=0x557e0a192cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.258+0000 7fdc241a2740 1 --2- 172.21.15.40:0/546636683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a1883a0 0x557e0a192cd0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0b99b0 tx=0x7fdc0c0bba60 comp rx=0 tx=0).stop 2024-03-18T22:51:32.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.258+0000 7fdc241a2740 1 -- 172.21.15.40:0/546636683 shutdown_connections 2024-03-18T22:51:32.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.258+0000 7fdc241a2740 1 --2- 172.21.15.40:0/546636683 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a18df00 0x557e0a187280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.258+0000 7fdc241a2740 1 --2- 172.21.15.40:0/546636683 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a188d90 0x557e0a199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.258+0000 7fdc241a2740 1 --2- 172.21.15.40:0/546636683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a1883a0 0x557e0a192cd0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.258+0000 7fdc241a2740 1 -- 172.21.15.40:0/546636683 >> 172.21.15.40:0/546636683 conn(0x557e0a183910 msgr2=0x557e0a186b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:32.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.258+0000 7fdc241a2740 1 -- 172.21.15.40:0/546636683 shutdown_connections 2024-03-18T22:51:32.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.258+0000 7fdc241a2740 1 -- 172.21.15.40:0/546636683 wait complete. 2024-03-18T22:51:32.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.258+0000 7fdc241a2740 1 -- start start 2024-03-18T22:51:32.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.258+0000 7fdc241a2740 1 Processor -- start 2024-03-18T22:51:32.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.258+0000 7fdc241a2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a1883a0 0x557e0a1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.258+0000 7fdc241a2740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a188d90 0x557e0a197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.258+0000 7fdc241a2740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a18df00 0x557e0a198530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.258+0000 7fdc241a2740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1aba60 con 0x557e0a18df00 2024-03-18T22:51:32.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc241a2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1ab330 con 0x557e0a1883a0 2024-03-18T22:51:32.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc241a2740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1a8570 con 0x557e0a188d90 2024-03-18T22:51:32.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc241a2740 10 client.0 _ll_register_callbacks cb 0x7fdc0c0b97f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:32.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc1b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a18df00 0x557e0a198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc1affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a1883a0 0x557e0a1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc1b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a18df00 0x557e0a198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49712/0 (socket says 172.21.15.40:49712) 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc1b7fe640 1 -- 172.21.15.40:0/1535500798 learned_addr learned my addr 172.21.15.40:0/1535500798 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc13fff640 1 --2- 172.21.15.40:0/1535500798 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a188d90 0x557e0a197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc1b7fe640 1 -- 172.21.15.40:0/1535500798 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a188d90 msgr2=0x557e0a197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc1b7fe640 1 --2- 172.21.15.40:0/1535500798 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a188d90 0x557e0a197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc1b7fe640 1 -- 172.21.15.40:0/1535500798 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a1883a0 msgr2=0x557e0a1999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc1b7fe640 1 --2- 172.21.15.40:0/1535500798 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a1883a0 0x557e0a1999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc1b7fe640 1 -- 172.21.15.40:0/1535500798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e0a184380 con 0x557e0a18df00 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc1b7fe640 1 --2- 172.21.15.40:0/1535500798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a18df00 0x557e0a198530 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fdc14010d30 tx=0x7fdc14012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da9e6a78425c997 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc027fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc027fc640 1 -- 172.21.15.40:0/1535500798 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc14003d80 con 0x557e0a18df00 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc027fc640 1 -- 172.21.15.40:0/1535500798 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdc14004050 con 0x557e0a18df00 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc241a2740 1 -- 172.21.15.40:0/1535500798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc0c0c16a0 con 0x557e0a18df00 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.259+0000 7fdc027fc640 1 -- 172.21.15.40:0/1535500798 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fdc1402a050 con 0x557e0a18df00 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.260+0000 7fdc241a2740 10 client.17886 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.260+0000 7fdc241a2740 1 -- 172.21.15.40:0/1535500798 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557e0a205590 con 0x557e0a18df00 2024-03-18T22:51:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.260+0000 7fdc027fc640 1 -- 172.21.15.40:0/1535500798 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc0c0c16a0 con 0x557e0a18df00 2024-03-18T22:51:32.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.260+0000 7fdc027fc640 1 -- 172.21.15.40:0/1535500798 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7fdc14025030 con 0x557e0a18df00 2024-03-18T22:51:32.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.260+0000 7fdc241a2740 10 client.17886 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:32.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.260+0000 7fdc241a2740 10 client.17886 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:51:32.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.260+0000 7fdc241a2740 20 client.17886 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:32.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.260+0000 7fdc241a2740 1 --2- 172.21.15.40:0/1535500798 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557e0a205fc0 0x557e0a2083e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.260+0000 7fdc241a2740 4 client.17886 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:51:32.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.260+0000 7fdc241a2740 1 -- 172.21.15.40:0/1535500798 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557e0a0f49f0 con 0x557e0a205fc0 2024-03-18T22:51:32.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.260+0000 7fdc1affd640 1 --2- 172.21.15.40:0/1535500798 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557e0a205fc0 0x557e0a2083e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.260+0000 7fdc1affd640 1 --2- 172.21.15.40:0/1535500798 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557e0a205fc0 0x557e0a2083e0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=360d84870aba1ec4 server_cookie=36181fe8fb4e2f23 in_seq=0 out_seq=0 2024-03-18T22:51:32.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.260+0000 7fdc027fc640 10 client.17886 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:32.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.261+0000 7fdc027fc640 1 -- 172.21.15.40:0/1535500798 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557e0a0f49f0 con 0x557e0a205fc0 2024-03-18T22:51:32.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.261+0000 7fdc027fc640 10 client.17886 handle_command_reply: tid=0 2024-03-18T22:51:32.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.262+0000 7fdc241a2740 1 client.17886 shutdown 2024-03-18T22:51:32.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.262+0000 7fdc241a2740 1 -- 172.21.15.40:0/1535500798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a18df00 msgr2=0x557e0a198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.262+0000 7fdc241a2740 1 --2- 172.21.15.40:0/1535500798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a18df00 0x557e0a198530 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fdc14010d30 tx=0x7fdc14012820 comp rx=0 tx=0).stop 2024-03-18T22:51:32.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.262+0000 7fdc241a2740 1 -- 172.21.15.40:0/1535500798 shutdown_connections 2024-03-18T22:51:32.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.262+0000 7fdc241a2740 1 --2- 172.21.15.40:0/1535500798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557e0a18df00 0x557e0a198530 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.262+0000 7fdc241a2740 1 --2- 172.21.15.40:0/1535500798 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557e0a188d90 0x557e0a197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.262+0000 7fdc241a2740 1 --2- 172.21.15.40:0/1535500798 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557e0a205fc0 0x557e0a2083e0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.262+0000 7fdc241a2740 1 --2- 172.21.15.40:0/1535500798 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557e0a1883a0 0x557e0a1999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.262+0000 7fdc241a2740 1 -- 172.21.15.40:0/1535500798 >> 172.21.15.40:0/1535500798 conn(0x557e0a183910 msgr2=0x557e0a186b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:32.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.262+0000 7fdc027fc640 0 client.17886 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:32.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.262+0000 7fdc241a2740 1 -- 172.21.15.40:0/1535500798 shutdown_connections 2024-03-18T22:51:32.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.262+0000 7fdc241a2740 1 -- 172.21.15.40:0/1535500798 wait complete. 2024-03-18T22:51:32.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.262+0000 7fdc241a2740 20 client.17886 trim_cache size 0 max 16384 2024-03-18T22:51:32.267 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:51:32.268 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:51:32.268 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "d5bc0dd5-180e-4f08-8a83-eb56ba2c8496", 2024-03-18T22:51:32.268 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:51:32.268 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:51:32.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.263+0000 7fdc214dc640 1 -- 172.21.15.40:0/58466637 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc080a2880 msgr2=0x7fdc080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.263+0000 7fdc214dc640 1 --2- 172.21.15.40:0/58466637 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc080a2880 0x7fdc080a4d40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c199030 tx=0x7fdc04005f90 comp rx=0 tx=0).stop 2024-03-18T22:51:32.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.263+0000 7fdc214dc640 1 -- 172.21.15.40:0/58466637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc1c100720 msgr2=0x7fdc1c197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.263+0000 7fdc214dc640 1 --2- 172.21.15.40:0/58466637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc1c100720 0x7fdc1c197b70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c000c00 tx=0x7fdc0c004040 comp rx=0 tx=0).stop 2024-03-18T22:51:32.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.263+0000 7fdc214dc640 1 -- 172.21.15.40:0/58466637 shutdown_connections 2024-03-18T22:51:32.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.263+0000 7fdc214dc640 1 --2- 172.21.15.40:0/58466637 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc080a2880 0x7fdc080a4d40 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.263+0000 7fdc214dc640 1 --2- 172.21.15.40:0/58466637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc1c1022d0 0x7fdc1c19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.263+0000 7fdc214dc640 1 --2- 172.21.15.40:0/58466637 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc1c101910 0x7fdc1c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.263+0000 7fdc214dc640 1 --2- 172.21.15.40:0/58466637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc1c100720 0x7fdc1c197b70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.263+0000 7fdc214dc640 1 -- 172.21.15.40:0/58466637 >> 172.21.15.40:0/58466637 conn(0x7fdc1c0fbcb0 msgr2=0x7fdc1c068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:32.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.263+0000 7fdc214dc640 1 -- 172.21.15.40:0/58466637 shutdown_connections 2024-03-18T22:51:32.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.263+0000 7fdc214dc640 1 -- 172.21.15.40:0/58466637 wait complete. 2024-03-18T22:51:32.276 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:51:32.277 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:51:32.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:32 smithi040 ceph-mon[62643]: pgmap v617: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 28 MiB/s wr, 23 op/s 2024-03-18T22:51:32.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.410+0000 7f3525d3b640 1 -- 172.21.15.40:0/2863914542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35201022d0 msgr2=0x7f3520108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.410+0000 7f3525d3b640 1 --2- 172.21.15.40:0/2863914542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35201022d0 0x7f3520108b60 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f3518009ca0 tx=0x7f351802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.410+0000 7f3525d3b640 1 -- 172.21.15.40:0/2863914542 shutdown_connections 2024-03-18T22:51:32.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.410+0000 7f3525d3b640 1 --2- 172.21.15.40:0/2863914542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35201022d0 0x7f3520108b60 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.410+0000 7f3525d3b640 1 --2- 172.21.15.40:0/2863914542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3520101910 0x7f3520101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.410+0000 7f3525d3b640 1 --2- 172.21.15.40:0/2863914542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3520100720 0x7f3520100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.410+0000 7f3525d3b640 1 -- 172.21.15.40:0/2863914542 >> 172.21.15.40:0/2863914542 conn(0x7f35200fbcb0 msgr2=0x7f35200fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:32.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.410+0000 7f3525d3b640 1 -- 172.21.15.40:0/2863914542 shutdown_connections 2024-03-18T22:51:32.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.410+0000 7f3525d3b640 1 -- 172.21.15.40:0/2863914542 wait complete. 2024-03-18T22:51:32.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.410+0000 7f3525d3b640 1 Processor -- start 2024-03-18T22:51:32.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f3525d3b640 1 -- start start 2024-03-18T22:51:32.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f3525d3b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3520100720 0x7f35201046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f3525d3b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3520101910 0x7f3520102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f3525d3b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35201022d0 0x7f3520103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f3525d3b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f352010c250 con 0x7f3520101910 2024-03-18T22:51:32.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f3525d3b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f352010c0d0 con 0x7f3520100720 2024-03-18T22:51:32.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f3525d3b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f352010c3d0 con 0x7f35201022d0 2024-03-18T22:51:32.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f351effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3520101910 0x7f3520102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f351f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3520100720 0x7f35201046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f351effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3520101910 0x7f3520102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49734/0 (socket says 172.21.15.40:49734) 2024-03-18T22:51:32.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f351ffff640 1 --2- 172.21.15.40:0/2533358884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35201022d0 0x7f3520103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f351effd640 1 -- 172.21.15.40:0/2533358884 learned_addr learned my addr 172.21.15.40:0/2533358884 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:32.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f351f7fe640 1 -- 172.21.15.40:0/2533358884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35201022d0 msgr2=0x7f3520103230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f351f7fe640 1 --2- 172.21.15.40:0/2533358884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35201022d0 0x7f3520103230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f351f7fe640 1 -- 172.21.15.40:0/2533358884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3520101910 msgr2=0x7f3520102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f351f7fe640 1 --2- 172.21.15.40:0/2533358884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3520101910 0x7f3520102cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f351f7fe640 1 -- 172.21.15.40:0/2533358884 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3520103ac0 con 0x7f3520100720 2024-03-18T22:51:32.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f351effd640 1 --2- 172.21.15.40:0/2533358884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3520101910 0x7f3520102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:32.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.411+0000 7f351f7fe640 1 --2- 172.21.15.40:0/2533358884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3520100720 0x7f35201046b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f351000e9c0 tx=0x7f351000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:32.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.412+0000 7f351cff9640 1 -- 172.21.15.40:0/2533358884 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351000ccc0 con 0x7f3520100720 2024-03-18T22:51:32.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.412+0000 7f3525d3b640 1 -- 172.21.15.40:0/2533358884 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35201a1b90 con 0x7f3520100720 2024-03-18T22:51:32.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.412+0000 7f351cff9640 1 -- 172.21.15.40:0/2533358884 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35100035f0 con 0x7f3520100720 2024-03-18T22:51:32.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.412+0000 7f351cff9640 1 -- 172.21.15.40:0/2533358884 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351000ce60 con 0x7f3520100720 2024-03-18T22:51:32.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.412+0000 7f3525d3b640 1 -- 172.21.15.40:0/2533358884 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35201a2090 con 0x7f3520100720 2024-03-18T22:51:32.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.414+0000 7f351cff9640 1 -- 172.21.15.40:0/2533358884 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f35100037a0 con 0x7f3520100720 2024-03-18T22:51:32.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f3528a07740 1 Processor -- start 2024-03-18T22:51:32.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f3528a07740 1 -- start start 2024-03-18T22:51:32.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f351cff9640 1 --2- 172.21.15.40:0/2533358884 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f35080a2800 0x7f35080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f3528a07740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f06fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f3528a07740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f070440 0x55636f074f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f3528a07740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f0755b0 0x55636f077a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f3528a07740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55636efd4500 con 0x55636f070440 2024-03-18T22:51:32.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f3528a07740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55636ef7cbe0 con 0x55636f0755b0 2024-03-18T22:51:32.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f3528a07740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55636efd05a0 con 0x55636f06fa50 2024-03-18T22:51:32.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f351ffff640 1 --2- 172.21.15.40:0/2533358884 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f35080a2800 0x7f35080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f351f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f06fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f351effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f070440 0x55636f074f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f351effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f070440 0x55636f074f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49742/0 (socket says 172.21.15.40:49742) 2024-03-18T22:51:32.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f351f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f06fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:54144/0 (socket says 172.21.15.40:54144) 2024-03-18T22:51:32.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f351effd640 1 -- 172.21.15.40:0/1949287405 learned_addr learned my addr 172.21.15.40:0/1949287405 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:32.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f351ffff640 1 --2- 172.21.15.40:0/1949287405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f0755b0 0x55636f077a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.415+0000 7f351cff9640 1 -- 172.21.15.40:0/2533358884 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f35100cf220 con 0x7f3520100720 2024-03-18T22:51:32.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.416+0000 7f351f7fe640 1 -- 172.21.15.40:0/1949287405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f0755b0 msgr2=0x55636f077a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.416+0000 7f351f7fe640 1 --2- 172.21.15.40:0/1949287405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f0755b0 0x55636f077a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.416+0000 7f351f7fe640 1 -- 172.21.15.40:0/1949287405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f070440 msgr2=0x55636f074f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.416+0000 7f351f7fe640 1 --2- 172.21.15.40:0/1949287405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f070440 0x55636f074f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.416+0000 7f351f7fe640 1 -- 172.21.15.40:0/1949287405 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55636ef9e000 con 0x55636f06fa50 2024-03-18T22:51:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.416+0000 7f351ffff640 1 --2- 172.21.15.40:0/2533358884 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f35080a2800 0x7f35080a4cc0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3518006010 tx=0x7f3518009930 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.416+0000 7f351f7fe640 1 --2- 172.21.15.40:0/1949287405 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f06fe70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f35100cc980 tx=0x7f351009b660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdd8c168cba479df server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.416+0000 7f3505ffb640 1 -- 172.21.15.40:0/1949287405 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35100bba50 con 0x55636f06fa50 2024-03-18T22:51:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.416+0000 7f3505ffb640 1 -- 172.21.15.40:0/1949287405 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3510003b10 con 0x55636f06fa50 2024-03-18T22:51:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.416+0000 7f3505ffb640 1 -- 172.21.15.40:0/1949287405 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3510099ad0 con 0x55636f06fa50 2024-03-18T22:51:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.417+0000 7f3528a07740 1 -- 172.21.15.40:0/1949287405 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 msgr2=0x55636f06fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.417+0000 7f3528a07740 1 --2- 172.21.15.40:0/1949287405 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f06fe70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f35100cc980 tx=0x7f351009b660 comp rx=0 tx=0).stop 2024-03-18T22:51:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.417+0000 7f3528a07740 1 -- 172.21.15.40:0/1949287405 shutdown_connections 2024-03-18T22:51:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.417+0000 7f3528a07740 1 --2- 172.21.15.40:0/1949287405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f0755b0 0x55636f077a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.417+0000 7f3528a07740 1 --2- 172.21.15.40:0/1949287405 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f070440 0x55636f074f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.417+0000 7f3528a07740 1 --2- 172.21.15.40:0/1949287405 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f06fe70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.417+0000 7f3528a07740 1 -- 172.21.15.40:0/1949287405 >> 172.21.15.40:0/1949287405 conn(0x55636f06afc0 msgr2=0x55636f06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.417+0000 7f3528a07740 1 -- 172.21.15.40:0/1949287405 shutdown_connections 2024-03-18T22:51:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.417+0000 7f3528a07740 1 -- 172.21.15.40:0/1949287405 wait complete. 2024-03-18T22:51:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.417+0000 7f3528a07740 1 -- start start 2024-03-18T22:51:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.417+0000 7f3528a07740 1 Processor -- start 2024-03-18T22:51:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f3528a07740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f079df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f3528a07740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f070440 0x55636f07a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f3528a07740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f0755b0 0x55636f081370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f3528a07740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55636f078560 con 0x55636f070440 2024-03-18T22:51:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f3528a07740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55636ef90160 con 0x55636f0755b0 2024-03-18T22:51:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f3528a07740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55636f0786e0 con 0x55636f06fa50 2024-03-18T22:51:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f3528a07740 10 client.0 _ll_register_callbacks cb 0x55636efcca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f351effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f070440 0x55636f07a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f351f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f079df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f351f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f079df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:54154/0 (socket says 172.21.15.40:54154) 2024-03-18T22:51:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f351f7fe640 1 -- 172.21.15.40:0/3041927701 learned_addr learned my addr 172.21.15.40:0/3041927701 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f351effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f070440 0x55636f07a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49746/0 (socket says 172.21.15.40:49746) 2024-03-18T22:51:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f351ffff640 1 --2- 172.21.15.40:0/3041927701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f0755b0 0x55636f081370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f351f7fe640 1 -- 172.21.15.40:0/3041927701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f0755b0 msgr2=0x55636f081370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f351f7fe640 1 --2- 172.21.15.40:0/3041927701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f0755b0 0x55636f081370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f351f7fe640 1 -- 172.21.15.40:0/3041927701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f070440 msgr2=0x55636f07a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f351f7fe640 1 --2- 172.21.15.40:0/3041927701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f070440 0x55636f07a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f351f7fe640 1 -- 172.21.15.40:0/3041927701 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55636edb27d0 con 0x55636f06fa50 2024-03-18T22:51:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f351f7fe640 1 --2- 172.21.15.40:0/3041927701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f079df0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f351009b760 tx=0x7f35100bbc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c3b39adb7a37761 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f34feffd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:51:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f34feffd640 1 -- 172.21.15.40:0/3041927701 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351009b7d0 con 0x55636f06fa50 2024-03-18T22:51:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f34feffd640 1 -- 172.21.15.40:0/3041927701 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f3510086110 con 0x55636f06fa50 2024-03-18T22:51:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.418+0000 7f3528a07740 1 -- 172.21.15.40:0/3041927701 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55636efe1a40 con 0x55636f06fa50 2024-03-18T22:51:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.419+0000 7f34feffd640 1 -- 172.21.15.40:0/3041927701 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f351009bed0 con 0x55636f06fa50 2024-03-18T22:51:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.419+0000 7f3528a07740 10 client.27703 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:32.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.419+0000 7f3528a07740 1 -- 172.21.15.40:0/3041927701 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55636ece83f0 con 0x55636f06fa50 2024-03-18T22:51:32.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.419+0000 7f34feffd640 1 -- 172.21.15.40:0/3041927701 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55636efe1a40 con 0x55636f06fa50 2024-03-18T22:51:32.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.419+0000 7f34feffd640 1 -- 172.21.15.40:0/3041927701 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f351009cb00 con 0x55636f06fa50 2024-03-18T22:51:32.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.419+0000 7f3528a07740 10 client.27703 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:32.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.419+0000 7f3528a07740 10 client.27703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:51:32.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.419+0000 7f3528a07740 20 client.27703 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:32.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.419+0000 7f3528a07740 1 --2- 172.21.15.40:0/3041927701 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55636f085860 0x55636f087c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.419+0000 7f3528a07740 4 client.27703 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:51:32.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.419+0000 7f3528a07740 1 -- 172.21.15.40:0/3041927701 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55636efdbd90 con 0x55636f085860 2024-03-18T22:51:32.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.419+0000 7f351effd640 1 --2- 172.21.15.40:0/3041927701 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55636f085860 0x55636f087c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.420+0000 7f34feffd640 10 client.27703 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:32.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.420+0000 7f351effd640 1 --2- 172.21.15.40:0/3041927701 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55636f085860 0x55636f087c50 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e70a87552c522ee server_cookie=d7ac4e8b8403848a in_seq=0 out_seq=0 2024-03-18T22:51:32.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.421+0000 7f34feffd640 1 -- 172.21.15.40:0/3041927701 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55636efdbd90 con 0x55636f085860 2024-03-18T22:51:32.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.421+0000 7f34feffd640 10 client.27703 handle_command_reply: tid=0 2024-03-18T22:51:32.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.421+0000 7f3528a07740 1 client.27703 shutdown 2024-03-18T22:51:32.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.421+0000 7f3528a07740 1 -- 172.21.15.40:0/3041927701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 msgr2=0x55636f079df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.421+0000 7f3528a07740 1 --2- 172.21.15.40:0/3041927701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f079df0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f351009b760 tx=0x7f35100bbc40 comp rx=0 tx=0).stop 2024-03-18T22:51:32.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.421+0000 7f3528a07740 1 -- 172.21.15.40:0/3041927701 shutdown_connections 2024-03-18T22:51:32.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.421+0000 7f3528a07740 1 --2- 172.21.15.40:0/3041927701 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55636f085860 0x55636f087c50 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.421+0000 7f3528a07740 1 --2- 172.21.15.40:0/3041927701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f0755b0 0x55636f081370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.421+0000 7f3528a07740 1 --2- 172.21.15.40:0/3041927701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f070440 0x55636f07a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.421+0000 7f34feffd640 0 client.27703 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:32.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.421+0000 7f3528a07740 1 --2- 172.21.15.40:0/3041927701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f079df0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.421+0000 7f3528a07740 1 -- 172.21.15.40:0/3041927701 >> 172.21.15.40:0/3041927701 conn(0x55636f06afc0 msgr2=0x55636f06a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:32.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.421+0000 7f3528a07740 1 -- 172.21.15.40:0/3041927701 shutdown_connections 2024-03-18T22:51:32.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.421+0000 7f3528a07740 1 -- 172.21.15.40:0/3041927701 wait complete. 2024-03-18T22:51:32.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.421+0000 7f3528a07740 20 client.27703 trim_cache size 0 max 16384 2024-03-18T22:51:32.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.439+0000 7f3528a07740 1 Processor -- start 2024-03-18T22:51:32.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.439+0000 7f3528a07740 1 -- start start 2024-03-18T22:51:32.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f3528a07740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f08c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f3528a07740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f070440 0x55636f081330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f3528a07740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f0755b0 0x55636f08c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f3528a07740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f35201ac700 con 0x55636f0755b0 2024-03-18T22:51:32.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f3528a07740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f35201ae2a0 con 0x55636f070440 2024-03-18T22:51:32.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f3528a07740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f35201acdd0 con 0x55636f06fa50 2024-03-18T22:51:32.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f351ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f0755b0 0x55636f08c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f351effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f070440 0x55636f081330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f351f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f08c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f351effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f070440 0x55636f081330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:35614/0 (socket says 172.21.15.40:35614) 2024-03-18T22:51:32.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f351f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f08c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:54156/0 (socket says 172.21.15.40:54156) 2024-03-18T22:51:32.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f351effd640 1 -- 172.21.15.40:0/3405222751 learned_addr learned my addr 172.21.15.40:0/3405222751 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:32.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f351effd640 1 -- 172.21.15.40:0/3405222751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 msgr2=0x55636f08c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f351effd640 1 --2- 172.21.15.40:0/3405222751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f08c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f351effd640 1 -- 172.21.15.40:0/3405222751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f0755b0 msgr2=0x55636f08c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f351effd640 1 --2- 172.21.15.40:0/3405222751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f0755b0 0x55636f08c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f351effd640 1 -- 172.21.15.40:0/3405222751 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55636eca80f0 con 0x55636f070440 2024-03-18T22:51:32.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f351ffff640 1 --2- 172.21.15.40:0/3405222751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f0755b0 0x55636f08c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:32.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.440+0000 7f351effd640 1 --2- 172.21.15.40:0/3405222751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f070440 0x55636f081330 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x55636f06bf30 tx=0x7f3514012c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d9bc0ee62c08286 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:32.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.441+0000 7f35057fa640 1 -- 172.21.15.40:0/3405222751 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514004020 con 0x55636f070440 2024-03-18T22:51:32.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.441+0000 7f35057fa640 1 -- 172.21.15.40:0/3405222751 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f351400fda0 con 0x55636f070440 2024-03-18T22:51:32.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.441+0000 7f35057fa640 1 -- 172.21.15.40:0/3405222751 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35140054a0 con 0x55636f070440 2024-03-18T22:51:32.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.441+0000 7f3528a07740 1 -- 172.21.15.40:0/3405222751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f070440 msgr2=0x55636f081330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.441+0000 7f3528a07740 1 --2- 172.21.15.40:0/3405222751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f070440 0x55636f081330 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x55636f06bf30 tx=0x7f3514012c40 comp rx=0 tx=0).stop 2024-03-18T22:51:32.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.441+0000 7f3528a07740 1 -- 172.21.15.40:0/3405222751 shutdown_connections 2024-03-18T22:51:32.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.441+0000 7f3528a07740 1 --2- 172.21.15.40:0/3405222751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f0755b0 0x55636f08c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.441+0000 7f3528a07740 1 --2- 172.21.15.40:0/3405222751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f070440 0x55636f081330 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.441+0000 7f3528a07740 1 --2- 172.21.15.40:0/3405222751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f08c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.441+0000 7f3528a07740 1 -- 172.21.15.40:0/3405222751 >> 172.21.15.40:0/3405222751 conn(0x55636f06afc0 msgr2=0x55636f079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:32.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.441+0000 7f3528a07740 1 -- 172.21.15.40:0/3405222751 shutdown_connections 2024-03-18T22:51:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.441+0000 7f3528a07740 1 -- 172.21.15.40:0/3405222751 wait complete. 2024-03-18T22:51:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.442+0000 7f3528a07740 1 -- start start 2024-03-18T22:51:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.442+0000 7f3528a07740 1 Processor -- start 2024-03-18T22:51:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.442+0000 7f3528a07740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f07f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.442+0000 7f3528a07740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f070440 0x55636f07f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.442+0000 7f3528a07740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f0755b0 0x55636f080530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.442+0000 7f3528a07740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f35201adbd0 con 0x55636f0755b0 2024-03-18T22:51:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.442+0000 7f3528a07740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f35201ad4a0 con 0x55636f070440 2024-03-18T22:51:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.442+0000 7f3528a07740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f35201aa6e0 con 0x55636f06fa50 2024-03-18T22:51:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.442+0000 7f3528a07740 10 client.0 _ll_register_callbacks cb 0x7f3510097a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:51:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.442+0000 7f351ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f0755b0 0x55636f080530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.442+0000 7f351effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f070440 0x55636f07f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.442+0000 7f351f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f07f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.442+0000 7f351ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f0755b0 0x55636f080530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49768/0 (socket says 172.21.15.40:49768) 2024-03-18T22:51:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.442+0000 7f351ffff640 1 -- 172.21.15.40:0/867587176 learned_addr learned my addr 172.21.15.40:0/867587176 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.443+0000 7f351f7fe640 1 -- 172.21.15.40:0/867587176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f070440 msgr2=0x55636f07f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.443+0000 7f351f7fe640 1 --2- 172.21.15.40:0/867587176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f070440 0x55636f07f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.443+0000 7f351f7fe640 1 -- 172.21.15.40:0/867587176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f0755b0 msgr2=0x55636f080530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.443+0000 7f351f7fe640 1 --2- 172.21.15.40:0/867587176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f0755b0 0x55636f080530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.443+0000 7f351f7fe640 1 -- 172.21.15.40:0/867587176 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55636f080b20 con 0x55636f06fa50 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.443+0000 7f351ffff640 1 --2- 172.21.15.40:0/867587176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f0755b0 0x55636f080530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.443+0000 7f351f7fe640 1 --2- 172.21.15.40:0/867587176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f07f030 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f351000c880 tx=0x7f3510003e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=971e7336c3c4d974 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.443+0000 7f34feffd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.443+0000 7f34feffd640 1 -- 172.21.15.40:0/867587176 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351009f940 con 0x55636f06fa50 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.443+0000 7f34feffd640 1 -- 172.21.15.40:0/867587176 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f35100afbb0 con 0x55636f06fa50 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.443+0000 7f3528a07740 1 -- 172.21.15.40:0/867587176 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f351400fda0 con 0x55636f06fa50 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.443+0000 7f34feffd640 1 -- 172.21.15.40:0/867587176 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3510004000 con 0x55636f06fa50 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.443+0000 7f3528a07740 10 client.27709 fetch_fsmap learned FSMap version 35 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.443+0000 7f3528a07740 1 -- 172.21.15.40:0/867587176 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55636f080b20 con 0x55636f06fa50 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.443+0000 7f34feffd640 1 -- 172.21.15.40:0/867587176 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f351400fda0 con 0x55636f06fa50 2024-03-18T22:51:32.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.444+0000 7f34feffd640 1 -- 172.21.15.40:0/867587176 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f3510081020 con 0x55636f06fa50 2024-03-18T22:51:32.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.444+0000 7f3528a07740 10 client.27709 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:51:32.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.444+0000 7f3528a07740 10 client.27709 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:51:32.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.444+0000 7f3528a07740 20 client.27709 populate_metadata read hostname 'smithi040' 2024-03-18T22:51:32.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.444+0000 7f3528a07740 1 --2- 172.21.15.40:0/867587176 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55636f0ec530 0x55636f0ee950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:51:32.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.444+0000 7f3528a07740 4 client.27709 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:51:32.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.444+0000 7f3528a07740 1 -- 172.21.15.40:0/867587176 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55636ef59540 con 0x55636f0ec530 2024-03-18T22:51:32.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.444+0000 7f351effd640 1 --2- 172.21.15.40:0/867587176 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55636f0ec530 0x55636f0ee950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:51:32.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.444+0000 7f351effd640 1 --2- 172.21.15.40:0/867587176 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55636f0ec530 0x55636f0ee950 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24859af7ecec1acd server_cookie=9219e4fafbab02f6 in_seq=0 out_seq=0 2024-03-18T22:51:32.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.444+0000 7f34feffd640 10 client.27709 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:32.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.444+0000 7f34feffd640 1 -- 172.21.15.40:0/867587176 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55636ef59540 con 0x55636f0ec530 2024-03-18T22:51:32.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.444+0000 7f34feffd640 10 client.27709 handle_command_reply: tid=0 2024-03-18T22:51:32.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.444+0000 7f3528a07740 1 client.27709 shutdown 2024-03-18T22:51:32.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3528a07740 1 -- 172.21.15.40:0/867587176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 msgr2=0x55636f07f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3528a07740 1 --2- 172.21.15.40:0/867587176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f07f030 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f351000c880 tx=0x7f3510003e10 comp rx=0 tx=0).stop 2024-03-18T22:51:32.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3528a07740 1 -- 172.21.15.40:0/867587176 shutdown_connections 2024-03-18T22:51:32.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3528a07740 1 --2- 172.21.15.40:0/867587176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55636f0755b0 0x55636f080530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3528a07740 1 --2- 172.21.15.40:0/867587176 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55636f0ec530 0x55636f0ee950 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3528a07740 1 --2- 172.21.15.40:0/867587176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55636f070440 0x55636f07f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3528a07740 1 --2- 172.21.15.40:0/867587176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55636f06fa50 0x55636f07f030 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f34feffd640 0 client.27709 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:51:32.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3528a07740 1 -- 172.21.15.40:0/867587176 >> 172.21.15.40:0/867587176 conn(0x55636f06afc0 msgr2=0x55636f069950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:32.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3528a07740 1 -- 172.21.15.40:0/867587176 shutdown_connections 2024-03-18T22:51:32.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3528a07740 1 -- 172.21.15.40:0/867587176 wait complete. 2024-03-18T22:51:32.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3528a07740 20 client.27709 trim_cache size 0 max 16384 2024-03-18T22:51:32.451 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:51:32.452 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:51:32.452 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:51:32.452 INFO:teuthology.orchestra.run.smithi040.stdout: "d5bc0dd5-180e-4f08-8a83-eb56ba2c8496": { 2024-03-18T22:51:32.452 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:51:32.452 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "d5bc0dd5-180e-4f08-8a83-eb56ba2c8496", 2024-03-18T22:51:32.452 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:51:32.452 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:51:32.452 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:51:32.452 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:51:32.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3525d3b640 1 -- 172.21.15.40:0/2533358884 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f35080a2800 msgr2=0x7f35080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3525d3b640 1 --2- 172.21.15.40:0/2533358884 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f35080a2800 0x7f35080a4cc0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3518006010 tx=0x7f3518009930 comp rx=0 tx=0).stop 2024-03-18T22:51:32.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3525d3b640 1 -- 172.21.15.40:0/2533358884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3520100720 msgr2=0x7f35201046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:51:32.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3525d3b640 1 --2- 172.21.15.40:0/2533358884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3520100720 0x7f35201046b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f351000e9c0 tx=0x7f351000ee80 comp rx=0 tx=0).stop 2024-03-18T22:51:32.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3525d3b640 1 -- 172.21.15.40:0/2533358884 shutdown_connections 2024-03-18T22:51:32.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3525d3b640 1 --2- 172.21.15.40:0/2533358884 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f35080a2800 0x7f35080a4cc0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3525d3b640 1 --2- 172.21.15.40:0/2533358884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35201022d0 0x7f3520103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3525d3b640 1 --2- 172.21.15.40:0/2533358884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3520101910 0x7f3520102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3525d3b640 1 --2- 172.21.15.40:0/2533358884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3520100720 0x7f35201046b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:51:32.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3525d3b640 1 -- 172.21.15.40:0/2533358884 >> 172.21.15.40:0/2533358884 conn(0x7f35200fbcb0 msgr2=0x7f35200fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:51:32.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.445+0000 7f3525d3b640 1 -- 172.21.15.40:0/2533358884 shutdown_connections 2024-03-18T22:51:32.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:51:32.446+0000 7f3525d3b640 1 -- 172.21.15.40:0/2533358884 wait complete. 2024-03-18T22:51:32.458 INFO:tasks.cephfs.filesystem:scrub status for tag:d5bc0dd5-180e-4f08-8a83-eb56ba2c8496 - {'path': '/', 'tag': 'd5bc0dd5-180e-4f08-8a83-eb56ba2c8496', 'options': 'recursive,force'} 2024-03-18T22:51:32.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:32 smithi113 ceph-mon[66775]: pgmap v617: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 28 MiB/s wr, 23 op/s 2024-03-18T22:51:32.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:32 smithi103 ceph-mon[66377]: pgmap v617: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 28 MiB/s wr, 23 op/s 2024-03-18T22:51:33.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:33 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:51:33.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:33 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:33.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:33 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:51:33.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:33 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:51:33.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:33 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:51:33.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:33 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:33.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:33 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:33.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:33 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:51:33.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:33 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:33.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:33 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:51:33.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:33 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:51:33.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:33 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:51:33.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:33 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:33.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:33 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:33.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:33 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:51:33.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:33 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:33.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:33 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:51:33.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:33 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:51:33.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:33 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:51:33.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:33 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:33.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:33 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:34.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:34 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:34.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:34 smithi113 ceph-mon[66775]: pgmap v618: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 29 MiB/s wr, 22 op/s 2024-03-18T22:51:34.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:34 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:34.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:34 smithi103 ceph-mon[66377]: pgmap v618: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 29 MiB/s wr, 22 op/s 2024-03-18T22:51:34.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:34 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:51:34.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:34 smithi040 ceph-mon[62643]: pgmap v618: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 29 MiB/s wr, 22 op/s 2024-03-18T22:51:36.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:36 smithi040 ceph-mon[62643]: pgmap v619: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:51:36.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:36 smithi113 ceph-mon[66775]: pgmap v619: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:51:36.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:36 smithi103 ceph-mon[66377]: pgmap v619: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:51:38.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:38 smithi113 ceph-mon[66775]: pgmap v620: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:51:38.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:38 smithi103 ceph-mon[66377]: pgmap v620: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:51:38.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:38 smithi040 ceph-mon[62643]: pgmap v620: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 30 MiB/s wr, 22 op/s 2024-03-18T22:51:40.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:40 smithi113 ceph-mon[66775]: pgmap v621: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 MiB/s wr, 21 op/s 2024-03-18T22:51:40.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:40 smithi103 ceph-mon[66377]: pgmap v621: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 MiB/s wr, 21 op/s 2024-03-18T22:51:40.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:40 smithi040 ceph-mon[62643]: pgmap v621: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 MiB/s wr, 21 op/s 2024-03-18T22:51:42.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:42 smithi113 ceph-mon[66775]: pgmap v622: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:51:42.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:42 smithi103 ceph-mon[66377]: pgmap v622: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:51:42.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:42 smithi040 ceph-mon[62643]: pgmap v622: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 MiB/s wr, 22 op/s 2024-03-18T22:51:43.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:43 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:51:43.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:43 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:51:43.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:43 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:51:44.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:44 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:51:44.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:44 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:51:44.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:44 smithi113 ceph-mon[66775]: pgmap v623: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:51:44.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:44 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:51:44.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:44 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:51:44.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:44 smithi103 ceph-mon[66377]: pgmap v623: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:51:44.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:44 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:51:44.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:44 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:51:44.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:44 smithi040 ceph-mon[62643]: pgmap v623: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:51:46.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:46 smithi113 ceph-mon[66775]: pgmap v624: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:51:46.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:46 smithi103 ceph-mon[66377]: pgmap v624: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:51:46.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:46 smithi040 ceph-mon[62643]: pgmap v624: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:51:48.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:48 smithi113 ceph-mon[66775]: pgmap v625: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:51:48.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:48 smithi103 ceph-mon[66377]: pgmap v625: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:51:48.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:48 smithi040 ceph-mon[62643]: pgmap v625: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 28 MiB/s wr, 20 op/s 2024-03-18T22:51:50.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:50 smithi113 ceph-mon[66775]: pgmap v626: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 21 op/s 2024-03-18T22:51:50.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:50 smithi103 ceph-mon[66377]: pgmap v626: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 21 op/s 2024-03-18T22:51:50.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:50 smithi040 ceph-mon[62643]: pgmap v626: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 27 MiB/s wr, 21 op/s 2024-03-18T22:51:52.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:52 smithi113 ceph-mon[66775]: pgmap v627: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:51:52.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:52 smithi103 ceph-mon[66377]: pgmap v627: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:51:52.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:52 smithi040 ceph-mon[62643]: pgmap v627: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 MiB/s wr, 23 op/s 2024-03-18T22:51:54.528 INFO:tasks.workunit.client.0.smithi040.stdout:fs setup took 466 secs 2024-03-18T22:51:54.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:54 smithi113 ceph-mon[66775]: pgmap v628: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:51:54.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:54 smithi103 ceph-mon[66377]: pgmap v628: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:51:54.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:54 smithi040 ceph-mon[62643]: pgmap v628: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 28 MiB/s wr, 22 op/s 2024-03-18T22:51:56.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:56 smithi113 ceph-mon[66775]: pgmap v629: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:51:56.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:56 smithi103 ceph-mon[66377]: pgmap v629: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:51:56.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:56 smithi040 ceph-mon[62643]: pgmap v629: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 MiB/s wr, 23 op/s 2024-03-18T22:51:58.575 INFO:tasks.workunit.client.0.smithi040.stdout:Syncing()...4 sec 2024-03-18T22:51:58.578 INFO:tasks.workunit.client.0.smithi040.stdout:Starting Actual Benchmark At: Mon Mar 18 22:51:58 2024 2024-03-18T22:51:58.578 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:51:58.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:51:58 smithi113 ceph-mon[66775]: pgmap v630: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:51:58.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:51:58 smithi103 ceph-mon[66377]: pgmap v630: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:51:58.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:51:58 smithi040 ceph-mon[62643]: pgmap v630: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 26 MiB/s wr, 20 op/s 2024-03-18T22:52:00.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:00 smithi113 ceph-mon[66775]: pgmap v631: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 MiB/s wr, 19 op/s 2024-03-18T22:52:00.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:00 smithi103 ceph-mon[66377]: pgmap v631: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 MiB/s wr, 19 op/s 2024-03-18T22:52:00.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:00 smithi040 ceph-mon[62643]: pgmap v631: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 23 MiB/s wr, 19 op/s 2024-03-18T22:52:02.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:52:02.460 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:52:02.538 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:02 smithi040 ceph-mon[62643]: pgmap v632: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 19 MiB/s wr, 230 op/s 2024-03-18T22:52:02.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:02 smithi113 ceph-mon[66775]: pgmap v632: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 19 MiB/s wr, 230 op/s 2024-03-18T22:52:02.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:02 smithi103 ceph-mon[66377]: pgmap v632: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 19 MiB/s wr, 230 op/s 2024-03-18T22:52:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.726+0000 7f1dc929a640 1 -- 172.21.15.40:0/2585650350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dc4107e60 msgr2=0x7f1dc4108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.726+0000 7f1dc929a640 1 --2- 172.21.15.40:0/2585650350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dc4107e60 0x7f1dc4108280 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009860 tx=0x7f1db402bb90 comp rx=0 tx=0).stop 2024-03-18T22:52:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.727+0000 7f1dc929a640 1 -- 172.21.15.40:0/2585650350 shutdown_connections 2024-03-18T22:52:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.727+0000 7f1dc929a640 1 --2- 172.21.15.40:0/2585650350 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1dc4109970 0x7f1dc4110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.727+0000 7f1dc929a640 1 --2- 172.21.15.40:0/2585650350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dc4108fb0 0x7f1dc4109430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.727+0000 7f1dc929a640 1 --2- 172.21.15.40:0/2585650350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dc4107e60 0x7f1dc4108280 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.727+0000 7f1dc929a640 1 -- 172.21.15.40:0/2585650350 >> 172.21.15.40:0/2585650350 conn(0x7f1dc4076df0 msgr2=0x7f1dc4079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.728+0000 7f1dc929a640 1 -- 172.21.15.40:0/2585650350 shutdown_connections 2024-03-18T22:52:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.728+0000 7f1dc929a640 1 -- 172.21.15.40:0/2585650350 wait complete. 2024-03-18T22:52:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.729+0000 7f1dc929a640 1 Processor -- start 2024-03-18T22:52:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.730+0000 7f1dc929a640 1 -- start start 2024-03-18T22:52:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.730+0000 7f1dc929a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dc4107e60 0x7f1dc407f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.730+0000 7f1dc929a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1dc4108fb0 0x7f1dc407f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.730+0000 7f1dc929a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dc4109970 0x7f1dc41ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.730+0000 7f1dc929a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc41139e0 con 0x7f1dc4109970 2024-03-18T22:52:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.730+0000 7f1dc929a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc4113860 con 0x7f1dc4107e60 2024-03-18T22:52:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.730+0000 7f1dc929a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc4113b60 con 0x7f1dc4108fb0 2024-03-18T22:52:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.730+0000 7f1dc27fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1dc4108fb0 0x7f1dc407f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.730+0000 7f1dc27fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1dc4108fb0 0x7f1dc407f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:52112/0 (socket says 172.21.15.40:52112) 2024-03-18T22:52:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.730+0000 7f1dc27fc640 1 -- 172.21.15.40:0/514101609 learned_addr learned my addr 172.21.15.40:0/514101609 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.730+0000 7f1dc2ffd640 1 --2- 172.21.15.40:0/514101609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dc4107e60 0x7f1dc407f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.730+0000 7f1dc37fe640 1 --2- 172.21.15.40:0/514101609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dc4109970 0x7f1dc41ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.731+0000 7f1dc37fe640 1 -- 172.21.15.40:0/514101609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1dc4108fb0 msgr2=0x7f1dc407f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:02.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.731+0000 7f1dc37fe640 1 --2- 172.21.15.40:0/514101609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1dc4108fb0 0x7f1dc407f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.731+0000 7f1dc37fe640 1 -- 172.21.15.40:0/514101609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dc4107e60 msgr2=0x7f1dc407f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:02.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.731+0000 7f1dc37fe640 1 --2- 172.21.15.40:0/514101609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dc4107e60 0x7f1dc407f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.731+0000 7f1dc37fe640 1 -- 172.21.15.40:0/514101609 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc406b240 con 0x7f1dc4109970 2024-03-18T22:52:02.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.731+0000 7f1dc37fe640 1 --2- 172.21.15.40:0/514101609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dc4109970 0x7f1dc41ae0f0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00ef00 tx=0x7f1dbc00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:02.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.733+0000 7f1dabfff640 1 -- 172.21.15.40:0/514101609 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc004b20 con 0x7f1dc4109970 2024-03-18T22:52:02.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.733+0000 7f1dc929a640 1 -- 172.21.15.40:0/514101609 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dc406b530 con 0x7f1dc4109970 2024-03-18T22:52:02.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.733+0000 7f1dc929a640 1 -- 172.21.15.40:0/514101609 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dc406ba80 con 0x7f1dc4109970 2024-03-18T22:52:02.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.733+0000 7f1dabfff640 1 -- 172.21.15.40:0/514101609 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1dbc003e20 con 0x7f1dc4109970 2024-03-18T22:52:02.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.733+0000 7f1dabfff640 1 -- 172.21.15.40:0/514101609 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc00b570 con 0x7f1dc4109970 2024-03-18T22:52:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.736+0000 7f1dcbf67740 1 Processor -- start 2024-03-18T22:52:02.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.737+0000 7f1dabfff640 1 -- 172.21.15.40:0/514101609 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1dbc01d020 con 0x7f1dc4109970 2024-03-18T22:52:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.738+0000 7f1dabfff640 1 --2- 172.21.15.40:0/514101609 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1dac0a2a10 0x7f1dac0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.738+0000 7f1dabfff640 1 -- 172.21.15.40:0/514101609 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1dbc0ced10 con 0x7f1dc4109970 2024-03-18T22:52:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.738+0000 7f1dcbf67740 1 -- start start 2024-03-18T22:52:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.739+0000 7f1dcbf67740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697cfa50 0x5583697cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.739+0000 7f1dcbf67740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d0440 0x5583697d4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.739+0000 7f1dcbf67740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697d55b0 0x5583697d7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.739+0000 7f1dcbf67740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558369734500 con 0x5583697d0440 2024-03-18T22:52:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.739+0000 7f1dcbf67740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5583696dcbe0 con 0x5583697d55b0 2024-03-18T22:52:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.739+0000 7f1dcbf67740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5583697305a0 con 0x5583697cfa50 2024-03-18T22:52:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.739+0000 7f1dc27fc640 1 --2- 172.21.15.40:0/514101609 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1dac0a2a10 0x7f1dac0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.739+0000 7f1dc27fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d0440 0x5583697d4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.739+0000 7f1dc37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697d55b0 0x5583697d7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.739+0000 7f1dc37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697d55b0 0x5583697d7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48778/0 (socket says 172.21.15.40:48778) 2024-03-18T22:52:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.739+0000 7f1dc37fe640 1 -- 172.21.15.40:0/814877309 learned_addr learned my addr 172.21.15.40:0/814877309 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.739+0000 7f1dc27fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d0440 0x5583697d4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59040/0 (socket says 172.21.15.40:59040) 2024-03-18T22:52:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.739+0000 7f1dc2ffd640 1 --2- 172.21.15.40:0/814877309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697cfa50 0x5583697cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.740+0000 7f1dc27fc640 1 --2- 172.21.15.40:0/514101609 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1dac0a2a10 0x7f1dac0a4ed0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4080530 tx=0x7f1db80138e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.740+0000 7f1dc37fe640 1 -- 172.21.15.40:0/814877309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697cfa50 msgr2=0x5583697cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.740+0000 7f1dc37fe640 1 --2- 172.21.15.40:0/814877309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697cfa50 0x5583697cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.740+0000 7f1dc37fe640 1 -- 172.21.15.40:0/814877309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d0440 msgr2=0x5583697d4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.740+0000 7f1dc37fe640 1 --2- 172.21.15.40:0/814877309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d0440 0x5583697d4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.740+0000 7f1dc37fe640 1 -- 172.21.15.40:0/814877309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583696fe000 con 0x5583697d55b0 2024-03-18T22:52:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.740+0000 7f1dc27fc640 1 --2- 172.21.15.40:0/814877309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d0440 0x5583697d4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:52:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.740+0000 7f1dc37fe640 1 --2- 172.21.15.40:0/814877309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697d55b0 0x5583697d7a40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc09b1f0 tx=0x7f1dbc09baa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d19bd68155e9fce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.740+0000 7f1da97fa640 1 -- 172.21.15.40:0/814877309 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc01a040 con 0x5583697d55b0 2024-03-18T22:52:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.741+0000 7f1da97fa640 1 -- 172.21.15.40:0/814877309 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1dbc09f170 con 0x5583697d55b0 2024-03-18T22:52:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.741+0000 7f1da97fa640 1 -- 172.21.15.40:0/814877309 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc01d310 con 0x5583697d55b0 2024-03-18T22:52:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.742+0000 7f1dcbf67740 1 -- 172.21.15.40:0/814877309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697d55b0 msgr2=0x5583697d7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.742+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/814877309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697d55b0 0x5583697d7a40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc09b1f0 tx=0x7f1dbc09baa0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.742+0000 7f1dcbf67740 1 -- 172.21.15.40:0/814877309 shutdown_connections 2024-03-18T22:52:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.742+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/814877309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697d55b0 0x5583697d7a40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.742+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/814877309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d0440 0x5583697d4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.742+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/814877309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697cfa50 0x5583697cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.742+0000 7f1dcbf67740 1 -- 172.21.15.40:0/814877309 >> 172.21.15.40:0/814877309 conn(0x5583697cafc0 msgr2=0x5583697cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.742+0000 7f1dcbf67740 1 -- 172.21.15.40:0/814877309 shutdown_connections 2024-03-18T22:52:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.742+0000 7f1dcbf67740 1 -- 172.21.15.40:0/814877309 wait complete. 2024-03-18T22:52:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.743+0000 7f1dcbf67740 1 -- start start 2024-03-18T22:52:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.743+0000 7f1dcbf67740 1 Processor -- start 2024-03-18T22:52:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.743+0000 7f1dcbf67740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697cfa50 0x5583697fbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.743+0000 7f1dcbf67740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697d0440 0x5583697fc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.743+0000 7f1dcbf67740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d55b0 0x5583698033f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.744+0000 7f1dcbf67740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5583697d85b0 con 0x5583697d55b0 2024-03-18T22:52:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.744+0000 7f1dcbf67740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5583696f0160 con 0x5583697cfa50 2024-03-18T22:52:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.744+0000 7f1dcbf67740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5583697d8730 con 0x5583697d0440 2024-03-18T22:52:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.744+0000 7f1dcbf67740 10 client.0 _ll_register_callbacks cb 0x55836972ca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.744+0000 7f1dc2ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697cfa50 0x5583697fbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.744+0000 7f1dc27fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697d0440 0x5583697fc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.744+0000 7f1dc27fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697d0440 0x5583697fc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:52126/0 (socket says 172.21.15.40:52126) 2024-03-18T22:52:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.744+0000 7f1dc27fc640 1 -- 172.21.15.40:0/3608326804 learned_addr learned my addr 172.21.15.40:0/3608326804 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.744+0000 7f1dc37fe640 1 --2- 172.21.15.40:0/3608326804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d55b0 0x5583698033f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.745+0000 7f1dc2ffd640 1 -- 172.21.15.40:0/3608326804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697d0440 msgr2=0x5583697fc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.745+0000 7f1dc2ffd640 1 --2- 172.21.15.40:0/3608326804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697d0440 0x5583697fc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.745+0000 7f1dc2ffd640 1 -- 172.21.15.40:0/3608326804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d55b0 msgr2=0x5583698033f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.745+0000 7f1dc2ffd640 1 --2- 172.21.15.40:0/3608326804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d55b0 0x5583698033f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.745+0000 7f1dc2ffd640 1 -- 172.21.15.40:0/3608326804 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583695127d0 con 0x5583697cfa50 2024-03-18T22:52:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.747+0000 7f1dc2ffd640 1 --2- 172.21.15.40:0/3608326804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697cfa50 0x5583697fbe70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009780 tx=0x7f1db40095b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba7fd31f51fef219 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.747+0000 7f1da6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:52:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.749+0000 7f1da6ff5640 1 -- 172.21.15.40:0/3608326804 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db4039040 con 0x5583697cfa50 2024-03-18T22:52:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.749+0000 7f1dcbf67740 1 -- 172.21.15.40:0/3608326804 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dbc09f170 con 0x5583697cfa50 2024-03-18T22:52:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.749+0000 7f1da6ff5640 1 -- 172.21.15.40:0/3608326804 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1db4004230 con 0x5583697cfa50 2024-03-18T22:52:02.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.751+0000 7f1da6ff5640 1 -- 172.21.15.40:0/3608326804 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1db404d050 con 0x5583697cfa50 2024-03-18T22:52:02.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.751+0000 7f1dcbf67740 10 client.27731 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:02.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.751+0000 7f1dcbf67740 1 -- 172.21.15.40:0/3608326804 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5583694483f0 con 0x5583697cfa50 2024-03-18T22:52:02.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.753+0000 7f1da6ff5640 1 -- 172.21.15.40:0/3608326804 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1dbc09f170 con 0x5583697cfa50 2024-03-18T22:52:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.754+0000 7f1da6ff5640 1 -- 172.21.15.40:0/3608326804 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f1db402c6c0 con 0x5583697cfa50 2024-03-18T22:52:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.754+0000 7f1dcbf67740 10 client.27731 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.754+0000 7f1dcbf67740 10 client.27731 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.754+0000 7f1dcbf67740 20 client.27731 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.754+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/3608326804 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558369808180 0x55836980a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.754+0000 7f1dcbf67740 4 client.27731 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.754+0000 7f1dcbf67740 1 -- 172.21.15.40:0/3608326804 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558369741a40 con 0x558369808180 2024-03-18T22:52:02.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.755+0000 7f1dc27fc640 1 --2- 172.21.15.40:0/3608326804 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558369808180 0x55836980a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.758+0000 7f1dc27fc640 1 --2- 172.21.15.40:0/3608326804 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558369808180 0x55836980a620 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd1e6d233ca88c89 server_cookie=4e20abc74d5a14e8 in_seq=0 out_seq=0 2024-03-18T22:52:02.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.758+0000 7f1da6ff5640 10 client.27731 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:02.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.759+0000 7f1da6ff5640 1 -- 172.21.15.40:0/3608326804 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558369741a40 con 0x558369808180 2024-03-18T22:52:02.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.759+0000 7f1da6ff5640 10 client.27731 handle_command_reply: tid=0 2024-03-18T22:52:02.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.759+0000 7f1dcbf67740 1 client.27731 shutdown 2024-03-18T22:52:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.763+0000 7f1dcbf67740 1 -- 172.21.15.40:0/3608326804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697cfa50 msgr2=0x5583697fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:02.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.763+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/3608326804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697cfa50 0x5583697fbe70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009780 tx=0x7f1db40095b0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.767+0000 7f1dcbf67740 1 -- 172.21.15.40:0/3608326804 shutdown_connections 2024-03-18T22:52:02.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.767+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/3608326804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d55b0 0x5583698033f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.767+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/3608326804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697d0440 0x5583697fc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.767+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/3608326804 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x558369808180 0x55836980a620 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.767+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/3608326804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697cfa50 0x5583697fbe70 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.767+0000 7f1dcbf67740 1 -- 172.21.15.40:0/3608326804 >> 172.21.15.40:0/3608326804 conn(0x5583697cafc0 msgr2=0x5583697ca1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:02.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.767+0000 7f1da6ff5640 0 client.27731 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:02.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.767+0000 7f1dcbf67740 1 -- 172.21.15.40:0/3608326804 shutdown_connections 2024-03-18T22:52:02.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.768+0000 7f1dcbf67740 1 -- 172.21.15.40:0/3608326804 wait complete. 2024-03-18T22:52:02.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.768+0000 7f1dcbf67740 20 client.27731 trim_cache size 0 max 16384 2024-03-18T22:52:02.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.817+0000 7f1dcbf67740 1 Processor -- start 2024-03-18T22:52:02.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.819+0000 7f1dcbf67740 1 -- start start 2024-03-18T22:52:02.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.819+0000 7f1dcbf67740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697cfa50 0x5583697fc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.819+0000 7f1dcbf67740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d0440 0x558369803310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.819+0000 7f1dcbf67740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697d55b0 0x5583697c9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.819+0000 7f1dcbf67740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1db000fe10 con 0x5583697d0440 2024-03-18T22:52:02.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.819+0000 7f1dcbf67740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1db00119b0 con 0x5583697cfa50 2024-03-18T22:52:02.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.819+0000 7f1dcbf67740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1db00104e0 con 0x5583697d55b0 2024-03-18T22:52:02.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.820+0000 7f1dc37fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697d55b0 0x5583697c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.820+0000 7f1dc27fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d0440 0x558369803310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.820+0000 7f1dc27fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d0440 0x558369803310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59068/0 (socket says 172.21.15.40:59068) 2024-03-18T22:52:02.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.820+0000 7f1dc27fc640 1 -- 172.21.15.40:0/642040111 learned_addr learned my addr 172.21.15.40:0/642040111 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:02.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.822+0000 7f1dc2ffd640 1 --2- 172.21.15.40:0/642040111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697cfa50 0x5583697fc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.822+0000 7f1dc27fc640 1 -- 172.21.15.40:0/642040111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697d55b0 msgr2=0x5583697c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:02.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.822+0000 7f1dc27fc640 1 --2- 172.21.15.40:0/642040111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697d55b0 0x5583697c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.822+0000 7f1dc27fc640 1 -- 172.21.15.40:0/642040111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697cfa50 msgr2=0x5583697fc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:02.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.822+0000 7f1dc27fc640 1 --2- 172.21.15.40:0/642040111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697cfa50 0x5583697fc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.822+0000 7f1dc27fc640 1 -- 172.21.15.40:0/642040111 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583694080f0 con 0x5583697d0440 2024-03-18T22:52:02.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.823+0000 7f1dc27fc640 1 --2- 172.21.15.40:0/642040111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d0440 0x558369803310 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f1db8015010 tx=0x7f1db801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b1c67f67f6ae6de server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:02.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.824+0000 7f1da8ff9640 1 -- 172.21.15.40:0/642040111 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db8011d80 con 0x5583697d0440 2024-03-18T22:52:02.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.824+0000 7f1da8ff9640 1 -- 172.21.15.40:0/642040111 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1db802f070 con 0x5583697d0440 2024-03-18T22:52:02.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.824+0000 7f1da8ff9640 1 -- 172.21.15.40:0/642040111 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db801f6a0 con 0x5583697d0440 2024-03-18T22:52:02.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.825+0000 7f1dcbf67740 1 -- 172.21.15.40:0/642040111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d0440 msgr2=0x558369803310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:02.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.825+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/642040111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d0440 0x558369803310 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f1db8015010 tx=0x7f1db801d000 comp rx=0 tx=0).stop 2024-03-18T22:52:02.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.826+0000 7f1dcbf67740 1 -- 172.21.15.40:0/642040111 shutdown_connections 2024-03-18T22:52:02.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.826+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/642040111 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5583697d55b0 0x5583697c9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.826+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/642040111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d0440 0x558369803310 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.826+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/642040111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697cfa50 0x5583697fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.826+0000 7f1dcbf67740 1 -- 172.21.15.40:0/642040111 >> 172.21.15.40:0/642040111 conn(0x5583697cafc0 msgr2=0x5583697fbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:02.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.837+0000 7f1dcbf67740 1 -- 172.21.15.40:0/642040111 shutdown_connections 2024-03-18T22:52:02.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.843+0000 7f1dcbf67740 1 -- 172.21.15.40:0/642040111 wait complete. 2024-03-18T22:52:02.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.843+0000 7f1dcbf67740 1 -- start start 2024-03-18T22:52:02.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.844+0000 7f1dcbf67740 1 Processor -- start 2024-03-18T22:52:02.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.844+0000 7f1dcbf67740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697d0440 0x558369800c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.844+0000 7f1dcbf67740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d55b0 0x5583698011d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.844+0000 7f1dcbf67740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558369802140 0x5583698025c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.844+0000 7f1dcbf67740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1db00112e0 con 0x5583697d55b0 2024-03-18T22:52:02.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.844+0000 7f1dcbf67740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1db0010bb0 con 0x5583697d0440 2024-03-18T22:52:02.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.844+0000 7f1dcbf67740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1db000ddf0 con 0x558369802140 2024-03-18T22:52:02.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.844+0000 7f1dcbf67740 10 client.0 _ll_register_callbacks cb 0x7f1db4037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:02.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.844+0000 7f1dc27fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d55b0 0x5583698011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.844+0000 7f1dc27fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d55b0 0x5583698011d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59070/0 (socket says 172.21.15.40:59070) 2024-03-18T22:52:02.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.844+0000 7f1dc27fc640 1 -- 172.21.15.40:0/2434907158 learned_addr learned my addr 172.21.15.40:0/2434907158 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:02.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.845+0000 7f1dc37fe640 1 --2- 172.21.15.40:0/2434907158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558369802140 0x5583698025c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.845+0000 7f1dc2ffd640 1 --2- 172.21.15.40:0/2434907158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697d0440 0x558369800c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.847+0000 7f1dc27fc640 1 -- 172.21.15.40:0/2434907158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558369802140 msgr2=0x5583698025c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:02.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.847+0000 7f1dc27fc640 1 --2- 172.21.15.40:0/2434907158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558369802140 0x5583698025c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.847+0000 7f1dc27fc640 1 -- 172.21.15.40:0/2434907158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697d0440 msgr2=0x558369800c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:02.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.847+0000 7f1dc27fc640 1 --2- 172.21.15.40:0/2434907158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697d0440 0x558369800c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.847+0000 7f1dc27fc640 1 -- 172.21.15.40:0/2434907158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558369802b00 con 0x5583697d55b0 2024-03-18T22:52:02.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.849+0000 7f1dc27fc640 1 --2- 172.21.15.40:0/2434907158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d55b0 0x5583698011d0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f1db800bed0 tx=0x7f1db8020e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=364aa916e61c60a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:02.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.849+0000 7f1da6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:02.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.851+0000 7f1da6ff5640 1 -- 172.21.15.40:0/2434907158 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db8043060 con 0x5583697d55b0 2024-03-18T22:52:02.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.851+0000 7f1dcbf67740 1 -- 172.21.15.40:0/2434907158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1db802f070 con 0x5583697d55b0 2024-03-18T22:52:02.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.851+0000 7f1da6ff5640 1 -- 172.21.15.40:0/2434907158 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1db8020920 con 0x5583697d55b0 2024-03-18T22:52:02.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.857+0000 7f1da6ff5640 1 -- 172.21.15.40:0/2434907158 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1db8020b60 con 0x5583697d55b0 2024-03-18T22:52:02.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.857+0000 7f1dcbf67740 10 client.17922 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:02.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.858+0000 7f1dcbf67740 1 -- 172.21.15.40:0/2434907158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55836986eb10 con 0x5583697d55b0 2024-03-18T22:52:02.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.859+0000 7f1da6ff5640 1 -- 172.21.15.40:0/2434907158 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1db802f070 con 0x5583697d55b0 2024-03-18T22:52:02.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.859+0000 7f1da6ff5640 1 -- 172.21.15.40:0/2434907158 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f1db80046c0 con 0x5583697d55b0 2024-03-18T22:52:02.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.859+0000 7f1dcbf67740 10 client.17922 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:02.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.859+0000 7f1dcbf67740 10 client.17922 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:02.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.859+0000 7f1dcbf67740 20 client.17922 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:02.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.859+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/2434907158 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55836986f3e0 0x558369871800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:02.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.859+0000 7f1dcbf67740 4 client.17922 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:52:02.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.859+0000 7f1dcbf67740 1 -- 172.21.15.40:0/2434907158 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55836973bd90 con 0x55836986f3e0 2024-03-18T22:52:02.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.860+0000 7f1dc2ffd640 1 --2- 172.21.15.40:0/2434907158 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55836986f3e0 0x558369871800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:02.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.861+0000 7f1dc2ffd640 1 --2- 172.21.15.40:0/2434907158 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55836986f3e0 0x558369871800 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b5421ac538e553b server_cookie=c5301cd49eac6dae in_seq=0 out_seq=0 2024-03-18T22:52:02.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.861+0000 7f1da6ff5640 10 client.17922 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:02.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.862+0000 7f1da6ff5640 1 -- 172.21.15.40:0/2434907158 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55836973bd90 con 0x55836986f3e0 2024-03-18T22:52:02.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.862+0000 7f1da6ff5640 10 client.17922 handle_command_reply: tid=0 2024-03-18T22:52:02.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.862+0000 7f1dcbf67740 1 client.17922 shutdown 2024-03-18T22:52:02.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.873+0000 7f1dcbf67740 1 -- 172.21.15.40:0/2434907158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d55b0 msgr2=0x5583698011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:02.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.873+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/2434907158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d55b0 0x5583698011d0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f1db800bed0 tx=0x7f1db8020e10 comp rx=0 tx=0).stop 2024-03-18T22:52:02.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.874+0000 7f1dcbf67740 1 -- 172.21.15.40:0/2434907158 shutdown_connections 2024-03-18T22:52:02.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.874+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/2434907158 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558369802140 0x5583698025c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.874+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/2434907158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5583697d55b0 0x5583698011d0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.874+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/2434907158 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55836986f3e0 0x558369871800 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.874+0000 7f1dcbf67740 1 --2- 172.21.15.40:0/2434907158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583697d0440 0x558369800c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.874+0000 7f1dcbf67740 1 -- 172.21.15.40:0/2434907158 >> 172.21.15.40:0/2434907158 conn(0x5583697cafc0 msgr2=0x5583697ce220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:02.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.874+0000 7f1da6ff5640 0 client.17922 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:02.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.877+0000 7f1dcbf67740 1 -- 172.21.15.40:0/2434907158 shutdown_connections 2024-03-18T22:52:02.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.877+0000 7f1dcbf67740 1 -- 172.21.15.40:0/2434907158 wait complete. 2024-03-18T22:52:02.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.878+0000 7f1dcbf67740 20 client.17922 trim_cache size 0 max 16384 2024-03-18T22:52:02.880 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:52:02.880 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:52:02.880 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:52:02.880 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:52:02.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.882+0000 7f1dc929a640 1 -- 172.21.15.40:0/514101609 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1dac0a2a10 msgr2=0x7f1dac0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:02.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.882+0000 7f1dc929a640 1 --2- 172.21.15.40:0/514101609 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1dac0a2a10 0x7f1dac0a4ed0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4080530 tx=0x7f1db80138e0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.882+0000 7f1dc929a640 1 -- 172.21.15.40:0/514101609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dc4109970 msgr2=0x7f1dc41ae0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:02.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.882+0000 7f1dc929a640 1 --2- 172.21.15.40:0/514101609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dc4109970 0x7f1dc41ae0f0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00ef00 tx=0x7f1dbc00c740 comp rx=0 tx=0).stop 2024-03-18T22:52:02.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.882+0000 7f1dc929a640 1 -- 172.21.15.40:0/514101609 shutdown_connections 2024-03-18T22:52:02.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.882+0000 7f1dc929a640 1 --2- 172.21.15.40:0/514101609 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1dac0a2a10 0x7f1dac0a4ed0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.882+0000 7f1dc929a640 1 --2- 172.21.15.40:0/514101609 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1dc4109970 0x7f1dc41ae0f0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.882+0000 7f1dc929a640 1 --2- 172.21.15.40:0/514101609 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1dc4108fb0 0x7f1dc407f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.882+0000 7f1dc929a640 1 --2- 172.21.15.40:0/514101609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1dc4107e60 0x7f1dc407f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:02.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.882+0000 7f1dc929a640 1 -- 172.21.15.40:0/514101609 >> 172.21.15.40:0/514101609 conn(0x7f1dc4076df0 msgr2=0x7f1dc410c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:02.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.886+0000 7f1dc929a640 1 -- 172.21.15.40:0/514101609 shutdown_connections 2024-03-18T22:52:02.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:02.886+0000 7f1dc929a640 1 -- 172.21.15.40:0/514101609 wait complete. 2024-03-18T22:52:02.918 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:52:02.919 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:52:03.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.175+0000 7fee10a87640 1 -- 172.21.15.40:0/103538153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee0c10ae00 msgr2=0x7fee0c10d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:03.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.175+0000 7fee10a87640 1 --2- 172.21.15.40:0/103538153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee0c10ae00 0x7fee0c10d270 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fedf4009900 tx=0x7fedf402bc80 comp rx=0 tx=0).stop 2024-03-18T22:52:03.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.178+0000 7fee10a87640 1 -- 172.21.15.40:0/103538153 shutdown_connections 2024-03-18T22:52:03.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.178+0000 7fee10a87640 1 --2- 172.21.15.40:0/103538153 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee0c10d8e0 0x7fee0c118760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:03.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.178+0000 7fee10a87640 1 --2- 172.21.15.40:0/103538153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee0c10ae00 0x7fee0c10d270 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:03.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.178+0000 7fee10a87640 1 --2- 172.21.15.40:0/103538153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee0c073620 0x7fee0c10a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:03.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.178+0000 7fee10a87640 1 -- 172.21.15.40:0/103538153 >> 172.21.15.40:0/103538153 conn(0x7fee0c06e550 msgr2=0x7fee0c070990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:03.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.184+0000 7fee10a87640 1 -- 172.21.15.40:0/103538153 shutdown_connections 2024-03-18T22:52:03.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.184+0000 7fee10a87640 1 -- 172.21.15.40:0/103538153 wait complete. 2024-03-18T22:52:03.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.185+0000 7fee10a87640 1 Processor -- start 2024-03-18T22:52:03.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.185+0000 7fee10a87640 1 -- start start 2024-03-18T22:52:03.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.185+0000 7fee10a87640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee0c073620 0x7fee0c1150b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:03.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.185+0000 7fee10a87640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee0c10ae00 0x7fee0c1155f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:03.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.185+0000 7fee10a87640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee0c10d8e0 0x7fee0c1b66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:03.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.185+0000 7fee10a87640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c11bfe0 con 0x7fee0c073620 2024-03-18T22:52:03.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.185+0000 7fee10a87640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c11be60 con 0x7fee0c10ae00 2024-03-18T22:52:03.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.185+0000 7fee10a87640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c11c160 con 0x7fee0c10d8e0 2024-03-18T22:52:03.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.187+0000 7fee0bfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee0c10d8e0 0x7fee0c1b66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:03.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.187+0000 7fee0bfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee0c10d8e0 0x7fee0c1b66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:52170/0 (socket says 172.21.15.40:52170) 2024-03-18T22:52:03.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.187+0000 7fee0bfff640 1 -- 172.21.15.40:0/3557512286 learned_addr learned my addr 172.21.15.40:0/3557512286 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:03.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.187+0000 7fee0affd640 1 --2- 172.21.15.40:0/3557512286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee0c10ae00 0x7fee0c1155f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:03.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.187+0000 7fee0b7fe640 1 --2- 172.21.15.40:0/3557512286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee0c073620 0x7fee0c1150b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:03.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.188+0000 7fee0affd640 1 -- 172.21.15.40:0/3557512286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee0c10d8e0 msgr2=0x7fee0c1b66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:03.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.188+0000 7fee0affd640 1 --2- 172.21.15.40:0/3557512286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee0c10d8e0 0x7fee0c1b66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:03.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.188+0000 7fee0affd640 1 -- 172.21.15.40:0/3557512286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee0c073620 msgr2=0x7fee0c1150b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:03.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.188+0000 7fee0affd640 1 --2- 172.21.15.40:0/3557512286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee0c073620 0x7fee0c1150b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:03.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.188+0000 7fee0affd640 1 -- 172.21.15.40:0/3557512286 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0c1b6cc0 con 0x7fee0c10ae00 2024-03-18T22:52:03.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.189+0000 7fee0affd640 1 --2- 172.21.15.40:0/3557512286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee0c10ae00 0x7fee0c1155f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fedf402c290 tx=0x7fedf4005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:03.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.191+0000 7fee08ff9640 1 -- 172.21.15.40:0/3557512286 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4042020 con 0x7fee0c10ae00 2024-03-18T22:52:03.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.192+0000 7fee10a87640 1 -- 172.21.15.40:0/3557512286 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee0c1b6f50 con 0x7fee0c10ae00 2024-03-18T22:52:03.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.192+0000 7fee10a87640 1 -- 172.21.15.40:0/3557512286 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee0c1b73b0 con 0x7fee0c10ae00 2024-03-18T22:52:03.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.192+0000 7fee08ff9640 1 -- 172.21.15.40:0/3557512286 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fedf4004d70 con 0x7fee0c10ae00 2024-03-18T22:52:03.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.192+0000 7fee08ff9640 1 -- 172.21.15.40:0/3557512286 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf403e460 con 0x7fee0c10ae00 2024-03-18T22:52:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.195+0000 7fee10a87640 1 -- 172.21.15.40:0/3557512286 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedec0052c0 con 0x7fee0c10ae00 2024-03-18T22:52:03.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.196+0000 7fee08ff9640 1 -- 172.21.15.40:0/3557512286 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fedf4045020 con 0x7fee0c10ae00 2024-03-18T22:52:03.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.197+0000 7fee08ff9640 1 --2- 172.21.15.40:0/3557512286 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fedf80a2a10 0x7fedf80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:03.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.197+0000 7fee08ff9640 1 -- 172.21.15.40:0/3557512286 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fedf40f11f0 con 0x7fee0c10ae00 2024-03-18T22:52:03.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.198+0000 7fee0b7fe640 1 --2- 172.21.15.40:0/3557512286 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fedf80a2a10 0x7fedf80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:03.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.199+0000 7fee0b7fe640 1 --2- 172.21.15.40:0/3557512286 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fedf80a2a10 0x7fedf80a4ed0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0058c0 tx=0x7fedfc00a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:03.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.206+0000 7fee08ff9640 1 -- 172.21.15.40:0/3557512286 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fedf404c020 con 0x7fee0c10ae00 2024-03-18T22:52:03.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.826+0000 7fee10a87640 1 -- 172.21.15.40:0/3557512286 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedec0055b0 con 0x7fee0c10ae00 2024-03-18T22:52:03.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.829+0000 7fee08ff9640 1 -- 172.21.15.40:0/3557512286 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11950 (secure 0 0 0) 0x7fedf402cb80 con 0x7fee0c10ae00 2024-03-18T22:52:03.830 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:52:03.830 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:51:14.215890+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24481,24509,24473,24401,24388]},"id":1}]} 2024-03-18T22:52:03.830 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 35 2024-03-18T22:52:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.832+0000 7fee10a87640 1 -- 172.21.15.40:0/3557512286 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fedf80a2a10 msgr2=0x7fedf80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.832+0000 7fee10a87640 1 --2- 172.21.15.40:0/3557512286 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fedf80a2a10 0x7fedf80a4ed0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0058c0 tx=0x7fedfc00a2d0 comp rx=0 tx=0).stop 2024-03-18T22:52:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.832+0000 7fee10a87640 1 -- 172.21.15.40:0/3557512286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee0c10ae00 msgr2=0x7fee0c1155f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:03.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.833+0000 7fee10a87640 1 --2- 172.21.15.40:0/3557512286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee0c10ae00 0x7fee0c1155f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fedf402c290 tx=0x7fedf4005250 comp rx=0 tx=0).stop 2024-03-18T22:52:03.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.835+0000 7fee10a87640 1 -- 172.21.15.40:0/3557512286 shutdown_connections 2024-03-18T22:52:03.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.836+0000 7fee10a87640 1 --2- 172.21.15.40:0/3557512286 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fedf80a2a10 0x7fedf80a4ed0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.836+0000 7fee10a87640 1 --2- 172.21.15.40:0/3557512286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee0c10d8e0 0x7fee0c1b66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.836+0000 7fee10a87640 1 --2- 172.21.15.40:0/3557512286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee0c10ae00 0x7fee0c1155f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.836+0000 7fee10a87640 1 --2- 172.21.15.40:0/3557512286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee0c073620 0x7fee0c1150b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.836+0000 7fee10a87640 1 -- 172.21.15.40:0/3557512286 >> 172.21.15.40:0/3557512286 conn(0x7fee0c06e550 msgr2=0x7fee0c109880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.837+0000 7fee10a87640 1 -- 172.21.15.40:0/3557512286 shutdown_connections 2024-03-18T22:52:03.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:03.837+0000 7fee10a87640 1 -- 172.21.15.40:0/3557512286 wait complete. 2024-03-18T22:52:03.866 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:52:04.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.131+0000 7f337e661640 1 -- 172.21.15.40:0/3567999625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33781022d0 msgr2=0x7f3378108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.131+0000 7f337e661640 1 --2- 172.21.15.40:0/3567999625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33781022d0 0x7f3378108b60 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f337001acb0 tx=0x7f337003f340 comp rx=0 tx=0).stop 2024-03-18T22:52:04.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.133+0000 7f337e661640 1 -- 172.21.15.40:0/3567999625 shutdown_connections 2024-03-18T22:52:04.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.133+0000 7f337e661640 1 --2- 172.21.15.40:0/3567999625 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33781022d0 0x7f3378108b60 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.133+0000 7f337e661640 1 --2- 172.21.15.40:0/3567999625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3378101910 0x7f3378101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.133+0000 7f337e661640 1 --2- 172.21.15.40:0/3567999625 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3378100720 0x7f3378100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.133+0000 7f337e661640 1 -- 172.21.15.40:0/3567999625 >> 172.21.15.40:0/3567999625 conn(0x7f33780fbc90 msgr2=0x7f33780fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:04.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.138+0000 7f337e661640 1 -- 172.21.15.40:0/3567999625 shutdown_connections 2024-03-18T22:52:04.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.139+0000 7f337e661640 1 -- 172.21.15.40:0/3567999625 wait complete. 2024-03-18T22:52:04.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.140+0000 7f337e661640 1 Processor -- start 2024-03-18T22:52:04.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.142+0000 7f337e661640 1 -- start start 2024-03-18T22:52:04.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.143+0000 7f337e661640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3378100720 0x7f3378107c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.143+0000 7f337e661640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3378101910 0x7f3378108170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.143+0000 7f337e661640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33781022d0 0x7f3378194620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.143+0000 7f337e661640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f337806cc20 con 0x7f3378101910 2024-03-18T22:52:04.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.143+0000 7f337e661640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f337806caa0 con 0x7f33781022d0 2024-03-18T22:52:04.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.144+0000 7f337e661640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f337806cda0 con 0x7f3378100720 2024-03-18T22:52:04.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.144+0000 7f3377fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3378100720 0x7f3378107c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.144+0000 7f3377fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3378100720 0x7f3378107c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:52178/0 (socket says 172.21.15.40:52178) 2024-03-18T22:52:04.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.144+0000 7f3377fff640 1 -- 172.21.15.40:0/4188979276 learned_addr learned my addr 172.21.15.40:0/4188979276 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:04.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.145+0000 7f3377fff640 1 -- 172.21.15.40:0/4188979276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33781022d0 msgr2=0x7f3378194620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T22:52:04.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.145+0000 7f3377fff640 1 --2- 172.21.15.40:0/4188979276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33781022d0 0x7f3378194620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.145+0000 7f3377fff640 1 -- 172.21.15.40:0/4188979276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3378101910 msgr2=0x7f3378108170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.145+0000 7f3377fff640 1 --2- 172.21.15.40:0/4188979276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3378101910 0x7f3378108170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.145+0000 7f3377fff640 1 -- 172.21.15.40:0/4188979276 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3378194c90 con 0x7f3378100720 2024-03-18T22:52:04.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.146+0000 7f3377fff640 1 --2- 172.21.15.40:0/4188979276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3378100720 0x7f3378107c30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f336800e9c0 tx=0x7f336800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:04.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.147+0000 7f33757fa640 1 -- 172.21.15.40:0/4188979276 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336800cd80 con 0x7f3378100720 2024-03-18T22:52:04.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.147+0000 7f33757fa640 1 -- 172.21.15.40:0/4188979276 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33680035f0 con 0x7f3378100720 2024-03-18T22:52:04.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.147+0000 7f33757fa640 1 -- 172.21.15.40:0/4188979276 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3368010690 con 0x7f3378100720 2024-03-18T22:52:04.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.147+0000 7f337e661640 1 -- 172.21.15.40:0/4188979276 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3378194f80 con 0x7f3378100720 2024-03-18T22:52:04.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.147+0000 7f337e661640 1 -- 172.21.15.40:0/4188979276 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3378195500 con 0x7f3378100720 2024-03-18T22:52:04.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.151+0000 7f3381252740 1 Processor -- start 2024-03-18T22:52:04.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.157+0000 7f3381252740 1 -- start start 2024-03-18T22:52:04.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.157+0000 7f33757fa640 1 -- 172.21.15.40:0/4188979276 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3368010830 con 0x7f3378100720 2024-03-18T22:52:04.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.158+0000 7f33757fa640 1 --2- 172.21.15.40:0/4188979276 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f33600a29d0 0x7f33600a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.158+0000 7f33757fa640 1 -- 172.21.15.40:0/4188979276 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f33680d01f0 con 0x7f3378100720 2024-03-18T22:52:04.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.159+0000 7f3381252740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d62f0 0x56352f2d6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.159+0000 7f3381252740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2d6ce0 0x56352f2db7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.159+0000 7f3381252740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2dbe50 0x56352f2de2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.159+0000 7f3381252740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56352f23a4b0 con 0x56352f2d6ce0 2024-03-18T22:52:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.159+0000 7f3381252740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56352f1e2d70 con 0x56352f2d62f0 2024-03-18T22:52:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.159+0000 7f3381252740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56352f236550 con 0x56352f2dbe50 2024-03-18T22:52:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.159+0000 7f337cbfe640 1 --2- 172.21.15.40:0/4188979276 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f33600a29d0 0x7f33600a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.159+0000 7f3377fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d62f0 0x56352f2d6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.159+0000 7f3377fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d62f0 0x56352f2d6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48856/0 (socket says 172.21.15.40:48856) 2024-03-18T22:52:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.159+0000 7f3377fff640 1 -- 172.21.15.40:0/1233018817 learned_addr learned my addr 172.21.15.40:0/1233018817 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.160+0000 7f33777fe640 1 --2- 172.21.15.40:0/1233018817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2d6ce0 0x56352f2db7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.160+0000 7f3377fff640 1 -- 172.21.15.40:0/1233018817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2dbe50 msgr2=0x56352f2de2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:52:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.160+0000 7f3377fff640 1 --2- 172.21.15.40:0/1233018817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2dbe50 0x56352f2de2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.160+0000 7f3377fff640 1 -- 172.21.15.40:0/1233018817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2d6ce0 msgr2=0x56352f2db7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.160+0000 7f3377fff640 1 --2- 172.21.15.40:0/1233018817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2d6ce0 0x56352f2db7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.160+0000 7f3377fff640 1 -- 172.21.15.40:0/1233018817 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56352f205210 con 0x56352f2d62f0 2024-03-18T22:52:04.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.160+0000 7f33777fe640 1 --2- 172.21.15.40:0/1233018817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2d6ce0 0x56352f2db7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:52:04.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.162+0000 7f337cbfe640 1 --2- 172.21.15.40:0/4188979276 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f33600a29d0 0x7f33600a4e90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3370005e30 tx=0x7f3370045340 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:04.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.162+0000 7f3377fff640 1 --2- 172.21.15.40:0/1233018817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d62f0 0x56352f2d6710 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f336800c6a0 tx=0x7f336809cf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91915faff7ff19bc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:04.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.163+0000 7f335e7fc640 1 -- 172.21.15.40:0/1233018817 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33680b4070 con 0x56352f2d62f0 2024-03-18T22:52:04.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.163+0000 7f335e7fc640 1 -- 172.21.15.40:0/1233018817 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33680a4ee0 con 0x56352f2d62f0 2024-03-18T22:52:04.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.164+0000 7f3381252740 1 -- 172.21.15.40:0/1233018817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d62f0 msgr2=0x56352f2d6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.164+0000 7f3381252740 1 --2- 172.21.15.40:0/1233018817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d62f0 0x56352f2d6710 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f336800c6a0 tx=0x7f336809cf00 comp rx=0 tx=0).stop 2024-03-18T22:52:04.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.166+0000 7f3381252740 1 -- 172.21.15.40:0/1233018817 shutdown_connections 2024-03-18T22:52:04.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.166+0000 7f3381252740 1 --2- 172.21.15.40:0/1233018817 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2dbe50 0x56352f2de2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.166+0000 7f3381252740 1 --2- 172.21.15.40:0/1233018817 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2d6ce0 0x56352f2db7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.166+0000 7f3381252740 1 --2- 172.21.15.40:0/1233018817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d62f0 0x56352f2d6710 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.166+0000 7f3381252740 1 -- 172.21.15.40:0/1233018817 >> 172.21.15.40:0/1233018817 conn(0x56352f2d1860 msgr2=0x56352f2d1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:04.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.167+0000 7f3381252740 1 -- 172.21.15.40:0/1233018817 shutdown_connections 2024-03-18T22:52:04.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.167+0000 7f3381252740 1 -- 172.21.15.40:0/1233018817 wait complete. 2024-03-18T22:52:04.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.167+0000 7f3381252740 1 -- start start 2024-03-18T22:52:04.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.167+0000 7f3381252740 1 Processor -- start 2024-03-18T22:52:04.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.168+0000 7f3381252740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d62f0 0x56352f2e0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.168+0000 7f3381252740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2d6ce0 0x56352f2e0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.168+0000 7f3381252740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2dbe50 0x56352f2e7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.168+0000 7f3381252740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56352f2dee50 con 0x56352f2dbe50 2024-03-18T22:52:04.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.168+0000 7f3381252740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56352f1f6310 con 0x56352f2d62f0 2024-03-18T22:52:04.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.168+0000 7f3381252740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56352f2defd0 con 0x56352f2d6ce0 2024-03-18T22:52:04.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.168+0000 7f3381252740 10 client.0 _ll_register_callbacks cb 0x56352f1ea2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:04.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.169+0000 7f33777fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2d6ce0 0x56352f2e0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.169+0000 7f33777fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2d6ce0 0x56352f2e0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:52188/0 (socket says 172.21.15.40:52188) 2024-03-18T22:52:04.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.169+0000 7f33777fe640 1 -- 172.21.15.40:0/1961402486 learned_addr learned my addr 172.21.15.40:0/1961402486 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:04.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.169+0000 7f337cbfe640 1 --2- 172.21.15.40:0/1961402486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2dbe50 0x56352f2e7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.170+0000 7f337cbfe640 1 -- 172.21.15.40:0/1961402486 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2d6ce0 msgr2=0x56352f2e0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:04.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.170+0000 7f337cbfe640 1 --2- 172.21.15.40:0/1961402486 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2d6ce0 0x56352f2e0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.170+0000 7f337cbfe640 1 -- 172.21.15.40:0/1961402486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d62f0 msgr2=0x56352f2e0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:04.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.170+0000 7f337cbfe640 1 --2- 172.21.15.40:0/1961402486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d62f0 0x56352f2e0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.170+0000 7f337cbfe640 1 -- 172.21.15.40:0/1961402486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56352ef4e3f0 con 0x56352f2dbe50 2024-03-18T22:52:04.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.170+0000 7f337cbfe640 1 --2- 172.21.15.40:0/1961402486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2dbe50 0x56352f2e7ff0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f3370006010 tx=0x7f337003f340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b297a9ff3bfac8a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:04.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.170+0000 7f33577fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:04.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.171+0000 7f33577fe640 1 -- 172.21.15.40:0/1961402486 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3370060020 con 0x56352f2dbe50 2024-03-18T22:52:04.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.171+0000 7f3381252740 1 -- 172.21.15.40:0/1961402486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56352f2983e0 con 0x56352f2dbe50 2024-03-18T22:52:04.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.171+0000 7f33577fe640 1 -- 172.21.15.40:0/1961402486 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f3370004590 con 0x56352f2dbe50 2024-03-18T22:52:04.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.172+0000 7f33577fe640 1 -- 172.21.15.40:0/1961402486 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f337006b050 con 0x56352f2dbe50 2024-03-18T22:52:04.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.172+0000 7f3381252740 10 client.17934 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:04.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.172+0000 7f3381252740 1 -- 172.21.15.40:0/1961402486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56352f0187d0 con 0x56352f2dbe50 2024-03-18T22:52:04.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.173+0000 7f33577fe640 1 -- 172.21.15.40:0/1961402486 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56352f2983e0 con 0x56352f2dbe50 2024-03-18T22:52:04.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.173+0000 7f33577fe640 1 -- 172.21.15.40:0/1961402486 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f337005b030 con 0x56352f2dbe50 2024-03-18T22:52:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.174+0000 7f3381252740 10 client.17934 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.174+0000 7f3381252740 10 client.17934 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:52:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.174+0000 7f3381252740 20 client.17934 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.174+0000 7f3381252740 1 --2- 172.21.15.40:0/1961402486 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x56352f2ece10 0x56352f2ef270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.174+0000 7f3381252740 4 client.17934 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.174+0000 7f3381252740 1 -- 172.21.15.40:0/1961402486 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56352f2479f0 con 0x56352f2ece10 2024-03-18T22:52:04.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.174+0000 7f3377fff640 1 --2- 172.21.15.40:0/1961402486 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x56352f2ece10 0x56352f2ef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.184+0000 7f3377fff640 1 --2- 172.21.15.40:0/1961402486 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x56352f2ece10 0x56352f2ef270 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=37d5235634b23713 server_cookie=9796bc947414e776 in_seq=0 out_seq=0 2024-03-18T22:52:04.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.184+0000 7f33577fe640 10 client.17934 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:52:04.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.185+0000 7f33577fe640 1 -- 172.21.15.40:0/1961402486 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56352f2479f0 con 0x56352f2ece10 2024-03-18T22:52:04.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.185+0000 7f33577fe640 10 client.17934 handle_command_reply: tid=0 2024-03-18T22:52:04.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.188+0000 7f3381252740 1 client.17934 shutdown 2024-03-18T22:52:04.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.194+0000 7f3381252740 1 -- 172.21.15.40:0/1961402486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2dbe50 msgr2=0x56352f2e7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.194+0000 7f3381252740 1 --2- 172.21.15.40:0/1961402486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2dbe50 0x56352f2e7ff0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f3370006010 tx=0x7f337003f340 comp rx=0 tx=0).stop 2024-03-18T22:52:04.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.199+0000 7f3381252740 1 -- 172.21.15.40:0/1961402486 shutdown_connections 2024-03-18T22:52:04.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.199+0000 7f3381252740 1 --2- 172.21.15.40:0/1961402486 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x56352f2ece10 0x56352f2ef270 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.199+0000 7f3381252740 1 --2- 172.21.15.40:0/1961402486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2dbe50 0x56352f2e7ff0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.199+0000 7f3381252740 1 --2- 172.21.15.40:0/1961402486 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2d6ce0 0x56352f2e0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.199+0000 7f3381252740 1 --2- 172.21.15.40:0/1961402486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d62f0 0x56352f2e0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.199+0000 7f3381252740 1 -- 172.21.15.40:0/1961402486 >> 172.21.15.40:0/1961402486 conn(0x56352f2d1860 msgr2=0x56352f2d0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:04.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.199+0000 7f33577fe640 0 client.17934 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:52:04.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.207+0000 7f3381252740 1 -- 172.21.15.40:0/1961402486 shutdown_connections 2024-03-18T22:52:04.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.213+0000 7f3381252740 1 -- 172.21.15.40:0/1961402486 wait complete. 2024-03-18T22:52:04.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.213+0000 7f3381252740 20 client.17934 trim_cache size 0 max 16384 2024-03-18T22:52:04.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.261+0000 7f3381252740 1 Processor -- start 2024-03-18T22:52:04.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.261+0000 7f3381252740 1 -- start start 2024-03-18T22:52:04.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.261+0000 7f3381252740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2d62f0 0x56352f2e0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.261+0000 7f3381252740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d6ce0 0x56352f2e0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.261+0000 7f3381252740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2dbe50 0x56352f2e7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.261+0000 7f3381252740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f336400fe80 con 0x56352f2d62f0 2024-03-18T22:52:04.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.261+0000 7f3381252740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3364011a20 con 0x56352f2d6ce0 2024-03-18T22:52:04.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.261+0000 7f3381252740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f3364010550 con 0x56352f2dbe50 2024-03-18T22:52:04.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.264+0000 7f3377fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2d62f0 0x56352f2e0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.264+0000 7f3377fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2d62f0 0x56352f2e0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59152/0 (socket says 172.21.15.40:59152) 2024-03-18T22:52:04.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.264+0000 7f3377fff640 1 -- 172.21.15.40:0/3934438998 learned_addr learned my addr 172.21.15.40:0/3934438998 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:04.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.264+0000 7f3377fff640 1 -- 172.21.15.40:0/3934438998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2dbe50 msgr2=0x56352f2e7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:04.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.264+0000 7f3377fff640 1 --2- 172.21.15.40:0/3934438998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2dbe50 0x56352f2e7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.264+0000 7f3377fff640 1 -- 172.21.15.40:0/3934438998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d6ce0 msgr2=0x56352f2e0fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:52:04.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.264+0000 7f3377fff640 1 --2- 172.21.15.40:0/3934438998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d6ce0 0x56352f2e0fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.264+0000 7f3377fff640 1 -- 172.21.15.40:0/3934438998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56352f2a7880 con 0x56352f2d62f0 2024-03-18T22:52:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.265+0000 7f3377fff640 1 --2- 172.21.15.40:0/3934438998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2d62f0 0x56352f2e0a70 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f3368003f80 tx=0x7f3368003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e2bc8068019836 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.266+0000 7f335dffb640 1 -- 172.21.15.40:0/3934438998 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33680b4040 con 0x56352f2d62f0 2024-03-18T22:52:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.266+0000 7f335dffb640 1 -- 172.21.15.40:0/3934438998 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f336809aa40 con 0x56352f2d62f0 2024-03-18T22:52:04.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.266+0000 7f335dffb640 1 -- 172.21.15.40:0/3934438998 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33680a4cb0 con 0x56352f2d62f0 2024-03-18T22:52:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.267+0000 7f3381252740 1 -- 172.21.15.40:0/3934438998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2d62f0 msgr2=0x56352f2e0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.267+0000 7f3381252740 1 --2- 172.21.15.40:0/3934438998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2d62f0 0x56352f2e0a70 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f3368003f80 tx=0x7f3368003d70 comp rx=0 tx=0).stop 2024-03-18T22:52:04.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.270+0000 7f3381252740 1 -- 172.21.15.40:0/3934438998 shutdown_connections 2024-03-18T22:52:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.270+0000 7f3381252740 1 --2- 172.21.15.40:0/3934438998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2dbe50 0x56352f2e7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.270+0000 7f3381252740 1 --2- 172.21.15.40:0/3934438998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d6ce0 0x56352f2e0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.270+0000 7f3381252740 1 --2- 172.21.15.40:0/3934438998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2d62f0 0x56352f2e0a70 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.270+0000 7f3381252740 1 -- 172.21.15.40:0/3934438998 >> 172.21.15.40:0/3934438998 conn(0x56352f2d1860 msgr2=0x56352f2eee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.271+0000 7f3381252740 1 -- 172.21.15.40:0/3934438998 shutdown_connections 2024-03-18T22:52:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.271+0000 7f3381252740 1 -- 172.21.15.40:0/3934438998 wait complete. 2024-03-18T22:52:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.272+0000 7f3381252740 1 -- start start 2024-03-18T22:52:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.272+0000 7f3381252740 1 Processor -- start 2024-03-18T22:52:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.273+0000 7f3381252740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2d62f0 0x56352f2e79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.273+0000 7f3381252740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d6ce0 0x56352f2e7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.273+0000 7f3381252740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2dbe50 0x56352f2e4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.273+0000 7f3377fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2d62f0 0x56352f2e79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.273+0000 7f3377fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2d62f0 0x56352f2e79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:52198/0 (socket says 172.21.15.40:52198) 2024-03-18T22:52:04.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.273+0000 7f3381252740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3364011350 con 0x56352f2dbe50 2024-03-18T22:52:04.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.273+0000 7f3377fff640 1 -- 172.21.15.40:0/444232244 learned_addr learned my addr 172.21.15.40:0/444232244 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:04.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.273+0000 7f3381252740 1 -- 172.21.15.40:0/444232244 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3364010c20 con 0x56352f2d6ce0 2024-03-18T22:52:04.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.273+0000 7f3381252740 1 -- 172.21.15.40:0/444232244 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f336400de60 con 0x56352f2d62f0 2024-03-18T22:52:04.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.273+0000 7f3381252740 10 client.0 _ll_register_callbacks cb 0x7f337004a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:04.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.273+0000 7f33777fe640 1 --2- 172.21.15.40:0/444232244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d6ce0 0x56352f2e7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.276+0000 7f337cbfe640 1 --2- 172.21.15.40:0/444232244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2dbe50 0x56352f2e4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.277+0000 7f33777fe640 1 -- 172.21.15.40:0/444232244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2d62f0 msgr2=0x56352f2e79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:04.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.277+0000 7f33777fe640 1 --2- 172.21.15.40:0/444232244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2d62f0 0x56352f2e79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.277+0000 7f33777fe640 1 -- 172.21.15.40:0/444232244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2dbe50 msgr2=0x56352f2e4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:04.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.277+0000 7f33777fe640 1 --2- 172.21.15.40:0/444232244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2dbe50 0x56352f2e4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.277+0000 7f33777fe640 1 -- 172.21.15.40:0/444232244 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56352f2e4e80 con 0x56352f2d6ce0 2024-03-18T22:52:04.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.279+0000 7f33777fe640 1 --2- 172.21.15.40:0/444232244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d6ce0 0x56352f2e7f30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x56352f2d74f0 tx=0x7f336c00f950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5026424e3efe6d5b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:04.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.279+0000 7f33577fe640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:52:04.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.280+0000 7f33577fe640 1 -- 172.21.15.40:0/444232244 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c013850 con 0x56352f2d6ce0 2024-03-18T22:52:04.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.280+0000 7f3381252740 1 -- 172.21.15.40:0/444232244 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f336809aa40 con 0x56352f2d6ce0 2024-03-18T22:52:04.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.280+0000 7f33577fe640 1 -- 172.21.15.40:0/444232244 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f336c0139f0 con 0x56352f2d6ce0 2024-03-18T22:52:04.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.282+0000 7f33577fe640 1 -- 172.21.15.40:0/444232244 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f336c024050 con 0x56352f2d6ce0 2024-03-18T22:52:04.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.283+0000 7f3381252740 10 client.27746 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:04.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.283+0000 7f3381252740 1 -- 172.21.15.40:0/444232244 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56352f352060 con 0x56352f2d6ce0 2024-03-18T22:52:04.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.287+0000 7f33577fe640 1 -- 172.21.15.40:0/444232244 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f336c013bf0 con 0x56352f2d6ce0 2024-03-18T22:52:04.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.288+0000 7f33577fe640 1 -- 172.21.15.40:0/444232244 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f336c0123c0 con 0x56352f2d6ce0 2024-03-18T22:52:04.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.288+0000 7f3381252740 10 client.27746 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:04.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.288+0000 7f3381252740 10 client.27746 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:52:04.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.288+0000 7f3381252740 20 client.27746 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.288+0000 7f3381252740 1 --2- 172.21.15.40:0/444232244 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x56352f352a20 0x56352f354e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.288+0000 7f3381252740 4 client.27746 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:52:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.288+0000 7f3381252740 1 -- 172.21.15.40:0/444232244 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f336809aa40 con 0x56352f352a20 2024-03-18T22:52:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.288+0000 7f3377fff640 1 --2- 172.21.15.40:0/444232244 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x56352f352a20 0x56352f354e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.293+0000 7f3377fff640 1 --2- 172.21.15.40:0/444232244 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x56352f352a20 0x56352f354e40 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1541176373c904b7 server_cookie=48ae907dd6674ded in_seq=0 out_seq=0 2024-03-18T22:52:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.293+0000 7f33577fe640 10 client.27746 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:52:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.294+0000 7f33577fe640 1 -- 172.21.15.40:0/444232244 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f336809aa40 con 0x56352f352a20 2024-03-18T22:52:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.294+0000 7f33577fe640 10 client.27746 handle_command_reply: tid=0 2024-03-18T22:52:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.294+0000 7f3381252740 1 client.27746 shutdown 2024-03-18T22:52:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.296+0000 7f3381252740 1 -- 172.21.15.40:0/444232244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d6ce0 msgr2=0x56352f2e7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.296+0000 7f3381252740 1 --2- 172.21.15.40:0/444232244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d6ce0 0x56352f2e7f30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x56352f2d74f0 tx=0x7f336c00f950 comp rx=0 tx=0).stop 2024-03-18T22:52:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.297+0000 7f3381252740 1 -- 172.21.15.40:0/444232244 shutdown_connections 2024-03-18T22:52:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.297+0000 7f3381252740 1 --2- 172.21.15.40:0/444232244 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x56352f352a20 0x56352f354e40 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.297+0000 7f3381252740 1 --2- 172.21.15.40:0/444232244 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56352f2dbe50 0x56352f2e4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.297+0000 7f3381252740 1 --2- 172.21.15.40:0/444232244 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56352f2d6ce0 0x56352f2e7f30 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.297+0000 7f3381252740 1 --2- 172.21.15.40:0/444232244 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56352f2d62f0 0x56352f2e79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.297+0000 7f3381252740 1 -- 172.21.15.40:0/444232244 >> 172.21.15.40:0/444232244 conn(0x56352f2d1860 msgr2=0x56352f2d83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.298+0000 7f33577fe640 0 client.27746 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:52:04.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.298+0000 7f3381252740 1 -- 172.21.15.40:0/444232244 shutdown_connections 2024-03-18T22:52:04.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.298+0000 7f3381252740 1 -- 172.21.15.40:0/444232244 wait complete. 2024-03-18T22:52:04.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.298+0000 7f3381252740 20 client.27746 trim_cache size 0 max 16384 2024-03-18T22:52:04.301 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:52:04.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.301+0000 7f337e661640 1 -- 172.21.15.40:0/4188979276 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f33600a29d0 msgr2=0x7f33600a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.301+0000 7f337e661640 1 --2- 172.21.15.40:0/4188979276 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f33600a29d0 0x7f33600a4e90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3370005e30 tx=0x7f3370045340 comp rx=0 tx=0).stop 2024-03-18T22:52:04.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.301+0000 7f337e661640 1 -- 172.21.15.40:0/4188979276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3378100720 msgr2=0x7f3378107c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.301+0000 7f337e661640 1 --2- 172.21.15.40:0/4188979276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3378100720 0x7f3378107c30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f336800e9c0 tx=0x7f336800ee80 comp rx=0 tx=0).stop 2024-03-18T22:52:04.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.302+0000 7f337e661640 1 -- 172.21.15.40:0/4188979276 shutdown_connections 2024-03-18T22:52:04.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.302+0000 7f337e661640 1 --2- 172.21.15.40:0/4188979276 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f33600a29d0 0x7f33600a4e90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.302+0000 7f337e661640 1 --2- 172.21.15.40:0/4188979276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33781022d0 0x7f3378194620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.302+0000 7f337e661640 1 --2- 172.21.15.40:0/4188979276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3378101910 0x7f3378108170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.302+0000 7f337e661640 1 --2- 172.21.15.40:0/4188979276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3378100720 0x7f3378107c30 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.302+0000 7f337e661640 1 -- 172.21.15.40:0/4188979276 >> 172.21.15.40:0/4188979276 conn(0x7f33780fbc90 msgr2=0x7f337806f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:04.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.305+0000 7f337e661640 1 -- 172.21.15.40:0/4188979276 shutdown_connections 2024-03-18T22:52:04.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.306+0000 7f337e661640 1 -- 172.21.15.40:0/4188979276 wait complete. 2024-03-18T22:52:04.332 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:52:04.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:04 smithi113 ceph-mon[66775]: pgmap v633: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 20 MiB/s wr, 980 op/s 2024-03-18T22:52:04.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:04 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3557512286' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:52:04.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:04 smithi103 ceph-mon[66377]: pgmap v633: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 20 MiB/s wr, 980 op/s 2024-03-18T22:52:04.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:04 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3557512286' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:52:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.615+0000 7f905157e640 1 -- 172.21.15.40:0/2065700623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f904c108a90 msgr2=0x7f904c10ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.615+0000 7f905157e640 1 --2- 172.21.15.40:0/2065700623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f904c108a90 0x7f904c10ae80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f903c00d910 tx=0x7f903c034200 comp rx=0 tx=0).stop 2024-03-18T22:52:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.615+0000 7f905157e640 1 -- 172.21.15.40:0/2065700623 shutdown_connections 2024-03-18T22:52:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.615+0000 7f905157e640 1 --2- 172.21.15.40:0/2065700623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f904c10ded0 0x7f904c110390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.615+0000 7f905157e640 1 --2- 172.21.15.40:0/2065700623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f904c10b450 0x7f904c10d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.615+0000 7f905157e640 1 --2- 172.21.15.40:0/2065700623 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f904c108a90 0x7f904c10ae80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.615+0000 7f905157e640 1 -- 172.21.15.40:0/2065700623 >> 172.21.15.40:0/2065700623 conn(0x7f904c06ea00 msgr2=0x7f904c070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.618+0000 7f905157e640 1 -- 172.21.15.40:0/2065700623 shutdown_connections 2024-03-18T22:52:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.620+0000 7f905157e640 1 -- 172.21.15.40:0/2065700623 wait complete. 2024-03-18T22:52:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.624+0000 7f905157e640 1 Processor -- start 2024-03-18T22:52:04.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.626+0000 7f905157e640 1 -- start start 2024-03-18T22:52:04.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.626+0000 7f905157e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f904c108a90 0x7f904c19de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.626+0000 7f905157e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f904c10b450 0x7f904c19e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.626+0000 7f905157e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f904c10ded0 0x7f904c1a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.626+0000 7f905157e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f904c1139b0 con 0x7f904c108a90 2024-03-18T22:52:04.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.626+0000 7f905157e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f904c113830 con 0x7f904c10ded0 2024-03-18T22:52:04.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.626+0000 7f905157e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f904c113b30 con 0x7f904c10b450 2024-03-18T22:52:04.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.627+0000 7f904affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f904c108a90 0x7f904c19de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.627+0000 7f904affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f904c108a90 0x7f904c19de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59174/0 (socket says 172.21.15.40:59174) 2024-03-18T22:52:04.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.627+0000 7f904affd640 1 -- 172.21.15.40:0/1049555414 learned_addr learned my addr 172.21.15.40:0/1049555414 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:04.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.627+0000 7f904a7fc640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f904c10b450 0x7f904c19e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.627+0000 7f904b7fe640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f904c10ded0 0x7f904c1a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.627+0000 7f904affd640 1 -- 172.21.15.40:0/1049555414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f904c10b450 msgr2=0x7f904c19e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.627+0000 7f904affd640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f904c10b450 0x7f904c19e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.627+0000 7f904affd640 1 -- 172.21.15.40:0/1049555414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f904c10ded0 msgr2=0x7f904c1a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.627+0000 7f904affd640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f904c10ded0 0x7f904c1a5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.627+0000 7f904affd640 1 -- 172.21.15.40:0/1049555414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904c1b6a80 con 0x7f904c108a90 2024-03-18T22:52:04.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.628+0000 7f904a7fc640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f904c10b450 0x7f904c19e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:52:04.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.628+0000 7f904b7fe640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f904c10ded0 0x7f904c1a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:52:04.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.628+0000 7f904affd640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f904c108a90 0x7f904c19de50 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f903c00d860 tx=0x7f903c00ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:04.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.628+0000 7f9033fff640 1 -- 172.21.15.40:0/1049555414 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c062030 con 0x7f904c108a90 2024-03-18T22:52:04.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.629+0000 7f905157e640 1 -- 172.21.15.40:0/1049555414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f904c1a5940 con 0x7f904c108a90 2024-03-18T22:52:04.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.629+0000 7f905157e640 1 -- 172.21.15.40:0/1049555414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f904c1a5e80 con 0x7f904c108a90 2024-03-18T22:52:04.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.630+0000 7f9033fff640 1 -- 172.21.15.40:0/1049555414 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f903c034db0 con 0x7f904c108a90 2024-03-18T22:52:04.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.630+0000 7f9033fff640 1 -- 172.21.15.40:0/1049555414 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c00cd80 con 0x7f904c108a90 2024-03-18T22:52:04.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.633+0000 7f9033fff640 1 -- 172.21.15.40:0/1049555414 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f903c065070 con 0x7f904c108a90 2024-03-18T22:52:04.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.638+0000 7f9033fff640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90340a28d0 0x7f90340a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.638+0000 7f9033fff640 1 -- 172.21.15.40:0/1049555414 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f903c0dc4f0 con 0x7f904c108a90 2024-03-18T22:52:04.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.639+0000 7f905416d740 1 Processor -- start 2024-03-18T22:52:04.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.639+0000 7f904a7fc640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90340a28d0 0x7f90340a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.639+0000 7f905416d740 1 -- start start 2024-03-18T22:52:04.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.639+0000 7f905416d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f22f0 0x5620aa8f2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.639+0000 7f905416d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f2ce0 0x5620aa8f77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.639+0000 7f905416d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f7e50 0x5620aa8fa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.639+0000 7f905416d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5620aa8564b0 con 0x5620aa8f7e50 2024-03-18T22:52:04.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.639+0000 7f905416d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5620aa7fed70 con 0x5620aa8f2ce0 2024-03-18T22:52:04.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.639+0000 7f905416d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5620aa852550 con 0x5620aa8f22f0 2024-03-18T22:52:04.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.640+0000 7f904b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f7e50 0x5620aa8fa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.640+0000 7f904b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f7e50 0x5620aa8fa2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59188/0 (socket says 172.21.15.40:59188) 2024-03-18T22:52:04.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.640+0000 7f904b7fe640 1 -- 172.21.15.40:0/1591884901 learned_addr learned my addr 172.21.15.40:0/1591884901 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:04.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.640+0000 7f904a7fc640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90340a28d0 0x7f90340a4d90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f904c19f360 tx=0x7f904000f040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:04.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.640+0000 7f904a7fc640 1 --2- 172.21.15.40:0/1591884901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f2ce0 0x5620aa8f77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.640+0000 7f904affd640 1 --2- 172.21.15.40:0/1591884901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f22f0 0x5620aa8f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.641+0000 7f904a7fc640 1 -- 172.21.15.40:0/1591884901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f22f0 msgr2=0x5620aa8f2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:04.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.641+0000 7f904a7fc640 1 --2- 172.21.15.40:0/1591884901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f22f0 0x5620aa8f2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.641+0000 7f904a7fc640 1 -- 172.21.15.40:0/1591884901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f7e50 msgr2=0x5620aa8fa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:04.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.641+0000 7f904a7fc640 1 --2- 172.21.15.40:0/1591884901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f7e50 0x5620aa8fa2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.641+0000 7f904a7fc640 1 -- 172.21.15.40:0/1591884901 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620aa821210 con 0x5620aa8f2ce0 2024-03-18T22:52:04.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:04 smithi040 ceph-mon[62643]: pgmap v633: 129 pgs: 129 active+clean; 13 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 20 MiB/s wr, 980 op/s 2024-03-18T22:52:04.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:04 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3557512286' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:52:04.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.642+0000 7f904a7fc640 1 --2- 172.21.15.40:0/1591884901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f2ce0 0x5620aa8f77e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f904000de30 tx=0x7f9040011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77e4e6c24a41fef3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:04.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.643+0000 7f90317fa640 1 -- 172.21.15.40:0/1591884901 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904001abb0 con 0x5620aa8f2ce0 2024-03-18T22:52:04.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.644+0000 7f90317fa640 1 -- 172.21.15.40:0/1591884901 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f904001ad50 con 0x5620aa8f2ce0 2024-03-18T22:52:04.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.645+0000 7f905416d740 1 -- 172.21.15.40:0/1591884901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f2ce0 msgr2=0x5620aa8f77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.645+0000 7f905416d740 1 --2- 172.21.15.40:0/1591884901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f2ce0 0x5620aa8f77e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f904000de30 tx=0x7f9040011000 comp rx=0 tx=0).stop 2024-03-18T22:52:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.648+0000 7f905416d740 1 -- 172.21.15.40:0/1591884901 shutdown_connections 2024-03-18T22:52:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.648+0000 7f905416d740 1 --2- 172.21.15.40:0/1591884901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f7e50 0x5620aa8fa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.648+0000 7f905416d740 1 --2- 172.21.15.40:0/1591884901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f2ce0 0x5620aa8f77e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.648+0000 7f905416d740 1 --2- 172.21.15.40:0/1591884901 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f22f0 0x5620aa8f2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.648+0000 7f905416d740 1 -- 172.21.15.40:0/1591884901 >> 172.21.15.40:0/1591884901 conn(0x5620aa8ed860 msgr2=0x5620aa8edc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.650+0000 7f905416d740 1 -- 172.21.15.40:0/1591884901 shutdown_connections 2024-03-18T22:52:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.653+0000 7f905416d740 1 -- 172.21.15.40:0/1591884901 wait complete. 2024-03-18T22:52:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.654+0000 7f905416d740 1 -- start start 2024-03-18T22:52:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.654+0000 7f905416d740 1 Processor -- start 2024-03-18T22:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.658+0000 7f905416d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f22f0 0x5620aa8fc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.658+0000 7f905416d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f2ce0 0x5620aa8fcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.658+0000 7f905416d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f7e50 0x5620aa903d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.658+0000 7f905416d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5620aa8faea0 con 0x5620aa8f22f0 2024-03-18T22:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.658+0000 7f905416d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5620aa812310 con 0x5620aa8f2ce0 2024-03-18T22:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.658+0000 7f905416d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5620aa8fb020 con 0x5620aa8f7e50 2024-03-18T22:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.658+0000 7f905416d740 10 client.0 _ll_register_callbacks cb 0x5620aa8062d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.658+0000 7f904affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f22f0 0x5620aa8fc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.658+0000 7f904affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f22f0 0x5620aa8fc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59202/0 (socket says 172.21.15.40:59202) 2024-03-18T22:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.658+0000 7f904affd640 1 -- 172.21.15.40:0/13535443 learned_addr learned my addr 172.21.15.40:0/13535443 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.658+0000 7f904a7fc640 1 --2- 172.21.15.40:0/13535443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f2ce0 0x5620aa8fcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.659+0000 7f904a7fc640 1 -- 172.21.15.40:0/13535443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f7e50 msgr2=0x5620aa903d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.659+0000 7f904a7fc640 1 --2- 172.21.15.40:0/13535443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f7e50 0x5620aa903d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.659+0000 7f904a7fc640 1 -- 172.21.15.40:0/13535443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f22f0 msgr2=0x5620aa8fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.659+0000 7f904a7fc640 1 --2- 172.21.15.40:0/13535443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f22f0 0x5620aa8fc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.659+0000 7f904a7fc640 1 -- 172.21.15.40:0/13535443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620aa56a3f0 con 0x5620aa8f2ce0 2024-03-18T22:52:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.659+0000 7f904a7fc640 1 --2- 172.21.15.40:0/13535443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f2ce0 0x5620aa8fcd10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x5620aa8f30e0 tx=0x7f9040005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26368a09f467af63 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.659+0000 7f902eff5640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:52:04.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.661+0000 7f902eff5640 1 -- 172.21.15.40:0/13535443 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9040030030 con 0x5620aa8f2ce0 2024-03-18T22:52:04.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.661+0000 7f902eff5640 1 -- 172.21.15.40:0/13535443 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f90400053e0 con 0x5620aa8f2ce0 2024-03-18T22:52:04.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.662+0000 7f905416d740 1 -- 172.21.15.40:0/13535443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f904001ad50 con 0x5620aa8f2ce0 2024-03-18T22:52:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.663+0000 7f902eff5640 1 -- 172.21.15.40:0/13535443 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9040028040 con 0x5620aa8f2ce0 2024-03-18T22:52:04.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.665+0000 7f905416d740 10 client.27755 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:04.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.665+0000 7f905416d740 1 -- 172.21.15.40:0/13535443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5620aa6347d0 con 0x5620aa8f2ce0 2024-03-18T22:52:04.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.668+0000 7f902eff5640 1 -- 172.21.15.40:0/13535443 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f904001ad50 con 0x5620aa8f2ce0 2024-03-18T22:52:04.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.668+0000 7f902eff5640 1 -- 172.21.15.40:0/13535443 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f904002b030 con 0x5620aa8f2ce0 2024-03-18T22:52:04.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.672+0000 7f905416d740 10 client.27755 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:04.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.672+0000 7f905416d740 10 client.27755 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:52:04.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.672+0000 7f905416d740 20 client.27755 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:04.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.672+0000 7f905416d740 1 --2- 172.21.15.40:0/13535443 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x5620aa908430 0x5620aa90a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.672+0000 7f905416d740 4 client.27755 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:04.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.672+0000 7f905416d740 1 -- 172.21.15.40:0/13535443 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620aa8b43e0 con 0x5620aa908430 2024-03-18T22:52:04.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.673+0000 7f904affd640 1 --2- 172.21.15.40:0/13535443 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x5620aa908430 0x5620aa90a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.678+0000 7f904affd640 1 --2- 172.21.15.40:0/13535443 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x5620aa908430 0x5620aa90a850 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a977218ef29e9b5b server_cookie=fc38dee302826b78 in_seq=0 out_seq=0 2024-03-18T22:52:04.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.678+0000 7f902eff5640 10 client.27755 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:52:04.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.686+0000 7f902eff5640 1 -- 172.21.15.40:0/13535443 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5620aa8b43e0 con 0x5620aa908430 2024-03-18T22:52:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.686+0000 7f902eff5640 10 client.27755 handle_command_reply: tid=0 2024-03-18T22:52:04.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.687+0000 7f905416d740 1 client.27755 shutdown 2024-03-18T22:52:04.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.690+0000 7f905416d740 1 -- 172.21.15.40:0/13535443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f2ce0 msgr2=0x5620aa8fcd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.690+0000 7f905416d740 1 --2- 172.21.15.40:0/13535443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f2ce0 0x5620aa8fcd10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x5620aa8f30e0 tx=0x7f9040005270 comp rx=0 tx=0).stop 2024-03-18T22:52:04.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.692+0000 7f905416d740 1 -- 172.21.15.40:0/13535443 shutdown_connections 2024-03-18T22:52:04.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.692+0000 7f905416d740 1 --2- 172.21.15.40:0/13535443 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x5620aa908430 0x5620aa90a850 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.692+0000 7f905416d740 1 --2- 172.21.15.40:0/13535443 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f7e50 0x5620aa903d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.692+0000 7f905416d740 1 --2- 172.21.15.40:0/13535443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f2ce0 0x5620aa8fcd10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.692+0000 7f905416d740 1 --2- 172.21.15.40:0/13535443 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f22f0 0x5620aa8fc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.692+0000 7f905416d740 1 -- 172.21.15.40:0/13535443 >> 172.21.15.40:0/13535443 conn(0x5620aa8ed860 msgr2=0x5620aa8ecba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:04.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.695+0000 7f902eff5640 0 client.27755 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:52:04.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.697+0000 7f905416d740 1 -- 172.21.15.40:0/13535443 shutdown_connections 2024-03-18T22:52:04.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.697+0000 7f905416d740 1 -- 172.21.15.40:0/13535443 wait complete. 2024-03-18T22:52:04.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.697+0000 7f905416d740 20 client.27755 trim_cache size 0 max 16384 2024-03-18T22:52:04.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.740+0000 7f905416d740 1 Processor -- start 2024-03-18T22:52:04.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.740+0000 7f905416d740 1 -- start start 2024-03-18T22:52:04.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.740+0000 7f905416d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f22f0 0x5620aa8fc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.741+0000 7f905416d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f2ce0 0x5620aa903d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.741+0000 7f905416d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f7e50 0x5620aa8fcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.741+0000 7f905416d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f903800feb0 con 0x5620aa8f7e50 2024-03-18T22:52:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.741+0000 7f905416d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9038011a50 con 0x5620aa8f22f0 2024-03-18T22:52:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.741+0000 7f905416d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9038010580 con 0x5620aa8f2ce0 2024-03-18T22:52:04.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.742+0000 7f904affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f22f0 0x5620aa8fc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.742+0000 7f904affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f22f0 0x5620aa8fc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48894/0 (socket says 172.21.15.40:48894) 2024-03-18T22:52:04.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.742+0000 7f904affd640 1 -- 172.21.15.40:0/457444704 learned_addr learned my addr 172.21.15.40:0/457444704 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.742+0000 7f904a7fc640 1 --2- 172.21.15.40:0/457444704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f2ce0 0x5620aa903d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.744+0000 7f904b7fe640 1 --2- 172.21.15.40:0/457444704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f7e50 0x5620aa8fcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.744+0000 7f904affd640 1 -- 172.21.15.40:0/457444704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f2ce0 msgr2=0x5620aa903d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.744+0000 7f904affd640 1 --2- 172.21.15.40:0/457444704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f2ce0 0x5620aa903d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.744+0000 7f904affd640 1 -- 172.21.15.40:0/457444704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f7e50 msgr2=0x5620aa8fcd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.744+0000 7f904affd640 1 --2- 172.21.15.40:0/457444704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f7e50 0x5620aa8fcd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.744+0000 7f904affd640 1 -- 172.21.15.40:0/457444704 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620aa8fa7a0 con 0x5620aa8f22f0 2024-03-18T22:52:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.744+0000 7f904affd640 1 --2- 172.21.15.40:0/457444704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f22f0 0x5620aa8fc7d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f903c0da8b0 tx=0x7f903c10f480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a368b6cfd1d710c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:04.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.744+0000 7f9030ff9640 1 -- 172.21.15.40:0/457444704 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c0f6070 con 0x5620aa8f22f0 2024-03-18T22:52:04.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.745+0000 7f9030ff9640 1 -- 172.21.15.40:0/457444704 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f903c038020 con 0x5620aa8f22f0 2024-03-18T22:52:04.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.745+0000 7f9030ff9640 1 -- 172.21.15.40:0/457444704 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c10f6c0 con 0x5620aa8f22f0 2024-03-18T22:52:04.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.751+0000 7f905416d740 1 -- 172.21.15.40:0/457444704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f22f0 msgr2=0x5620aa8fc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.751+0000 7f905416d740 1 --2- 172.21.15.40:0/457444704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f22f0 0x5620aa8fc7d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f903c0da8b0 tx=0x7f903c10f480 comp rx=0 tx=0).stop 2024-03-18T22:52:04.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.752+0000 7f905416d740 1 -- 172.21.15.40:0/457444704 shutdown_connections 2024-03-18T22:52:04.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.752+0000 7f905416d740 1 --2- 172.21.15.40:0/457444704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f7e50 0x5620aa8fcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.753+0000 7f905416d740 1 --2- 172.21.15.40:0/457444704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f2ce0 0x5620aa903d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.753+0000 7f905416d740 1 --2- 172.21.15.40:0/457444704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f22f0 0x5620aa8fc7d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.753+0000 7f905416d740 1 -- 172.21.15.40:0/457444704 >> 172.21.15.40:0/457444704 conn(0x5620aa8ed860 msgr2=0x5620aa8f40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.753+0000 7f905416d740 1 -- 172.21.15.40:0/457444704 shutdown_connections 2024-03-18T22:52:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.754+0000 7f905416d740 1 -- 172.21.15.40:0/457444704 wait complete. 2024-03-18T22:52:04.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.755+0000 7f905416d740 1 -- start start 2024-03-18T22:52:04.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.755+0000 7f905416d740 1 Processor -- start 2024-03-18T22:52:04.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.756+0000 7f905416d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f22f0 0x5620aa900bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.756+0000 7f905416d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f2ce0 0x5620aa901130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.756+0000 7f905416d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f7e50 0x5620aa96c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.756+0000 7f905416d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9038011380 con 0x5620aa8f2ce0 2024-03-18T22:52:04.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.756+0000 7f905416d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9038010c50 con 0x5620aa8f22f0 2024-03-18T22:52:04.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.756+0000 7f905416d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f903800de90 con 0x5620aa8f7e50 2024-03-18T22:52:04.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.756+0000 7f905416d740 10 client.0 _ll_register_callbacks cb 0x7f9044003b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:04.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.757+0000 7f904a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f2ce0 0x5620aa901130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.757+0000 7f904a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f2ce0 0x5620aa901130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59230/0 (socket says 172.21.15.40:59230) 2024-03-18T22:52:04.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.757+0000 7f904a7fc640 1 -- 172.21.15.40:0/851001361 learned_addr learned my addr 172.21.15.40:0/851001361 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:04.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.757+0000 7f904affd640 1 --2- 172.21.15.40:0/851001361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f22f0 0x5620aa900bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.757+0000 7f904a7fc640 1 -- 172.21.15.40:0/851001361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f7e50 msgr2=0x5620aa96c720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:52:04.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.757+0000 7f904a7fc640 1 --2- 172.21.15.40:0/851001361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f7e50 0x5620aa96c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.757+0000 7f904a7fc640 1 -- 172.21.15.40:0/851001361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f22f0 msgr2=0x5620aa900bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:04.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.757+0000 7f904a7fc640 1 --2- 172.21.15.40:0/851001361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f22f0 0x5620aa900bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.757+0000 7f904a7fc640 1 -- 172.21.15.40:0/851001361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620aa96cc60 con 0x5620aa8f2ce0 2024-03-18T22:52:04.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.758+0000 7f904a7fc640 1 --2- 172.21.15.40:0/851001361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f2ce0 0x5620aa901130 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x5620aa8fdcb0 tx=0x7f9040008750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44c930a7560ed26d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:04.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.758+0000 7f902eff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:04.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.760+0000 7f902eff5640 1 -- 172.21.15.40:0/851001361 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9040030040 con 0x5620aa8f2ce0 2024-03-18T22:52:04.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.760+0000 7f905416d740 1 -- 172.21.15.40:0/851001361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f903c038020 con 0x5620aa8f2ce0 2024-03-18T22:52:04.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.760+0000 7f902eff5640 1 -- 172.21.15.40:0/851001361 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f904000da50 con 0x5620aa8f2ce0 2024-03-18T22:52:04.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.761+0000 7f902eff5640 1 -- 172.21.15.40:0/851001361 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9040044050 con 0x5620aa8f2ce0 2024-03-18T22:52:04.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.761+0000 7f905416d740 10 client.17958 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:04.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.761+0000 7f905416d740 1 -- 172.21.15.40:0/851001361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5620aa96de90 con 0x5620aa8f2ce0 2024-03-18T22:52:04.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.763+0000 7f902eff5640 1 -- 172.21.15.40:0/851001361 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f903c038020 con 0x5620aa8f2ce0 2024-03-18T22:52:04.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.763+0000 7f902eff5640 1 -- 172.21.15.40:0/851001361 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f904003d060 con 0x5620aa8f2ce0 2024-03-18T22:52:04.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.763+0000 7f905416d740 10 client.17958 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:04.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.763+0000 7f905416d740 10 client.17958 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:52:04.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.763+0000 7f905416d740 20 client.17958 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:04.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.763+0000 7f905416d740 1 --2- 172.21.15.40:0/851001361 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x5620aa96e870 0x5620aa970c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:04.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.763+0000 7f905416d740 4 client.17958 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:52:04.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.763+0000 7f905416d740 1 -- 172.21.15.40:0/851001361 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5620aa8639f0 con 0x5620aa96e870 2024-03-18T22:52:04.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.763+0000 7f904affd640 1 --2- 172.21.15.40:0/851001361 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x5620aa96e870 0x5620aa970c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:04.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.764+0000 7f904affd640 1 --2- 172.21.15.40:0/851001361 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x5620aa96e870 0x5620aa970c90 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b1a682630edd1d08 server_cookie=48c13e286aac026f in_seq=0 out_seq=0 2024-03-18T22:52:04.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.764+0000 7f902eff5640 10 client.17958 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:52:04.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.765+0000 7f902eff5640 1 -- 172.21.15.40:0/851001361 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5620aa8639f0 con 0x5620aa96e870 2024-03-18T22:52:04.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.765+0000 7f902eff5640 10 client.17958 handle_command_reply: tid=0 2024-03-18T22:52:04.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.765+0000 7f905416d740 1 client.17958 shutdown 2024-03-18T22:52:04.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.766+0000 7f905416d740 1 -- 172.21.15.40:0/851001361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f2ce0 msgr2=0x5620aa901130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.766+0000 7f905416d740 1 --2- 172.21.15.40:0/851001361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f2ce0 0x5620aa901130 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x5620aa8fdcb0 tx=0x7f9040008750 comp rx=0 tx=0).stop 2024-03-18T22:52:04.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.767+0000 7f905416d740 1 -- 172.21.15.40:0/851001361 shutdown_connections 2024-03-18T22:52:04.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.767+0000 7f905416d740 1 --2- 172.21.15.40:0/851001361 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x5620aa96e870 0x5620aa970c90 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.767+0000 7f905416d740 1 --2- 172.21.15.40:0/851001361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5620aa8f7e50 0x5620aa96c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.767+0000 7f905416d740 1 --2- 172.21.15.40:0/851001361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5620aa8f2ce0 0x5620aa901130 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.767+0000 7f905416d740 1 --2- 172.21.15.40:0/851001361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5620aa8f22f0 0x5620aa900bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.767+0000 7f905416d740 1 -- 172.21.15.40:0/851001361 >> 172.21.15.40:0/851001361 conn(0x5620aa8ed860 msgr2=0x5620aa8f40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:04.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.767+0000 7f902eff5640 0 client.17958 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:52:04.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.767+0000 7f905416d740 1 -- 172.21.15.40:0/851001361 shutdown_connections 2024-03-18T22:52:04.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.768+0000 7f905416d740 1 -- 172.21.15.40:0/851001361 wait complete. 2024-03-18T22:52:04.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.768+0000 7f905416d740 20 client.17958 trim_cache size 0 max 16384 2024-03-18T22:52:04.769 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:52:04.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.769+0000 7f905157e640 1 -- 172.21.15.40:0/1049555414 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90340a28d0 msgr2=0x7f90340a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.769+0000 7f905157e640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90340a28d0 0x7f90340a4d90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f904c19f360 tx=0x7f904000f040 comp rx=0 tx=0).stop 2024-03-18T22:52:04.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.769+0000 7f905157e640 1 -- 172.21.15.40:0/1049555414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f904c108a90 msgr2=0x7f904c19de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:04.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.769+0000 7f905157e640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f904c108a90 0x7f904c19de50 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f903c00d860 tx=0x7f903c00ca00 comp rx=0 tx=0).stop 2024-03-18T22:52:04.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.769+0000 7f905157e640 1 -- 172.21.15.40:0/1049555414 shutdown_connections 2024-03-18T22:52:04.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.769+0000 7f905157e640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90340a28d0 0x7f90340a4d90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.769+0000 7f905157e640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f904c10ded0 0x7f904c1a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.769+0000 7f905157e640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f904c10b450 0x7f904c19e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.769+0000 7f905157e640 1 --2- 172.21.15.40:0/1049555414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f904c108a90 0x7f904c19de50 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:04.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.769+0000 7f905157e640 1 -- 172.21.15.40:0/1049555414 >> 172.21.15.40:0/1049555414 conn(0x7f904c06ea00 msgr2=0x7f904c10bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:04.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.771+0000 7f905157e640 1 -- 172.21.15.40:0/1049555414 shutdown_connections 2024-03-18T22:52:04.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:04.771+0000 7f905157e640 1 -- 172.21.15.40:0/1049555414 wait complete. 2024-03-18T22:52:04.804 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:52:05.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.068+0000 7f108d59a640 1 -- 172.21.15.40:0/3481096894 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080004c40 con 0x7f1090101190 2024-03-18T22:52:05.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.069+0000 7f1094f59640 1 -- 172.21.15.40:0/3481096894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1090101190 msgr2=0x7f1090105a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.069+0000 7f1094f59640 1 --2- 172.21.15.40:0/3481096894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1090101190 0x7f1090105a50 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f108001a910 tx=0x7f108003cc90 comp rx=0 tx=0).stop 2024-03-18T22:52:05.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.069+0000 7f1094f59640 1 -- 172.21.15.40:0/3481096894 shutdown_connections 2024-03-18T22:52:05.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.069+0000 7f1094f59640 1 --2- 172.21.15.40:0/3481096894 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1090101190 0x7f1090105a50 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.069+0000 7f1094f59640 1 --2- 172.21.15.40:0/3481096894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10901007f0 0x7f1090100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.069+0000 7f1094f59640 1 --2- 172.21.15.40:0/3481096894 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10900ff6c0 0x7f10900ffae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.069+0000 7f1094f59640 1 -- 172.21.15.40:0/3481096894 >> 172.21.15.40:0/3481096894 conn(0x7f10900fb000 msgr2=0x7f10900fd420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.073+0000 7f1094f59640 1 -- 172.21.15.40:0/3481096894 shutdown_connections 2024-03-18T22:52:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.076+0000 7f1094f59640 1 -- 172.21.15.40:0/3481096894 wait complete. 2024-03-18T22:52:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.077+0000 7f1094f59640 1 Processor -- start 2024-03-18T22:52:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.077+0000 7f1094f59640 1 -- start start 2024-03-18T22:52:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.077+0000 7f1094f59640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10900ff6c0 0x7f1090195780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.077+0000 7f1094f59640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10901007f0 0x7f1090195cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.077+0000 7f1094f59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1090101190 0x7f10901bed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.077+0000 7f1094f59640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f10901092a0 con 0x7f1090101190 2024-03-18T22:52:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.077+0000 7f1094f59640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1090109120 con 0x7f10900ff6c0 2024-03-18T22:52:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.077+0000 7f1094f59640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1090109420 con 0x7f10901007f0 2024-03-18T22:52:05.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.078+0000 7f108dd9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10901007f0 0x7f1090195cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.078+0000 7f108dd9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10901007f0 0x7f1090195cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:52258/0 (socket says 172.21.15.40:52258) 2024-03-18T22:52:05.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.078+0000 7f108dd9b640 1 -- 172.21.15.40:0/2347874531 learned_addr learned my addr 172.21.15.40:0/2347874531 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:05.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.078+0000 7f108e59c640 1 --2- 172.21.15.40:0/2347874531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10900ff6c0 0x7f1090195780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.079+0000 7f108dd9b640 1 -- 172.21.15.40:0/2347874531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10900ff6c0 msgr2=0x7f1090195780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.079+0000 7f108dd9b640 1 --2- 172.21.15.40:0/2347874531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10900ff6c0 0x7f1090195780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.079+0000 7f108dd9b640 1 -- 172.21.15.40:0/2347874531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1090101190 msgr2=0x7f10901bed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.079+0000 7f108dd9b640 1 --2- 172.21.15.40:0/2347874531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1090101190 0x7f10901bed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.079+0000 7f108dd9b640 1 -- 172.21.15.40:0/2347874531 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10901bf420 con 0x7f10901007f0 2024-03-18T22:52:05.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.081+0000 7f108dd9b640 1 --2- 172.21.15.40:0/2347874531 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10901007f0 0x7f1090195cc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f108800ebc0 tx=0x7f108800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.081+0000 7f10877fe640 1 -- 172.21.15.40:0/2347874531 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1088010630 con 0x7f10901007f0 2024-03-18T22:52:05.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.082+0000 7f10877fe640 1 -- 172.21.15.40:0/2347874531 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f108800ce50 con 0x7f10901007f0 2024-03-18T22:52:05.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.082+0000 7f10877fe640 1 -- 172.21.15.40:0/2347874531 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1088018ad0 con 0x7f10901007f0 2024-03-18T22:52:05.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.082+0000 7f1094f59640 1 -- 172.21.15.40:0/2347874531 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10901bf710 con 0x7f10901007f0 2024-03-18T22:52:05.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.082+0000 7f1094f59640 1 -- 172.21.15.40:0/2347874531 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10901bfcb0 con 0x7f10901007f0 2024-03-18T22:52:05.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.086+0000 7f10877fe640 1 -- 172.21.15.40:0/2347874531 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1088018c70 con 0x7f10901007f0 2024-03-18T22:52:05.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.086+0000 7f1097c17740 1 Processor -- start 2024-03-18T22:52:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.087+0000 7f1097c17740 1 -- start start 2024-03-18T22:52:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.087+0000 7f1097c17740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0f92f0 0x5593fe0f9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.087+0000 7f1097c17740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0f9ce0 0x5593fe0fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.087+0000 7f1097c17740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0fee50 0x5593fe1012e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.087+0000 7f1097c17740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5593fe05d4b0 con 0x5593fe0fee50 2024-03-18T22:52:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.087+0000 7f1097c17740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5593fe005d70 con 0x5593fe0f92f0 2024-03-18T22:52:05.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.087+0000 7f1097c17740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5593fe059550 con 0x5593fe0f9ce0 2024-03-18T22:52:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.088+0000 7f108ed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0f92f0 0x5593fe0f9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.088+0000 7f108ed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0f92f0 0x5593fe0f9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48920/0 (socket says 172.21.15.40:48920) 2024-03-18T22:52:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.088+0000 7f108ed9d640 1 -- 172.21.15.40:0/3009480382 learned_addr learned my addr 172.21.15.40:0/3009480382 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.088+0000 7f108e59c640 1 --2- 172.21.15.40:0/3009480382 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0f9ce0 0x5593fe0fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.088+0000 7f10877fe640 1 --2- 172.21.15.40:0/2347874531 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f107c0a2950 0x7f107c0a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.088+0000 7f108ed9d640 1 -- 172.21.15.40:0/3009480382 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0f9ce0 msgr2=0x5593fe0fe7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.088+0000 7f108ed9d640 1 --2- 172.21.15.40:0/3009480382 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0f9ce0 0x5593fe0fe7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.089+0000 7f108ed9d640 1 -- 172.21.15.40:0/3009480382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0fee50 msgr2=0x5593fe1012e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.089+0000 7f108ed9d640 1 --2- 172.21.15.40:0/3009480382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0fee50 0x5593fe1012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.089+0000 7f108ed9d640 1 -- 172.21.15.40:0/3009480382 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593fe028210 con 0x5593fe0f92f0 2024-03-18T22:52:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.089+0000 7f10877fe640 1 -- 172.21.15.40:0/2347874531 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f10880d0380 con 0x7f10901007f0 2024-03-18T22:52:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.089+0000 7f108ed9d640 1 --2- 172.21.15.40:0/3009480382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0f92f0 0x5593fe0f9710 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f10800048a0 tx=0x7f1080005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb1c7eeeab33cb35 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.089+0000 7f108ed9d640 1 --2- 172.21.15.40:0/2347874531 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f107c0a2950 0x7f107c0a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.089+0000 7f1084ff9640 1 -- 172.21.15.40:0/3009480382 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080004a80 con 0x5593fe0f92f0 2024-03-18T22:52:05.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.090+0000 7f1084ff9640 1 -- 172.21.15.40:0/3009480382 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1080004c20 con 0x5593fe0f92f0 2024-03-18T22:52:05.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.091+0000 7f1097c17740 1 -- 172.21.15.40:0/3009480382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0f92f0 msgr2=0x5593fe0f9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.091+0000 7f1097c17740 1 --2- 172.21.15.40:0/3009480382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0f92f0 0x5593fe0f9710 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f10800048a0 tx=0x7f1080005be0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.091+0000 7f1097c17740 1 -- 172.21.15.40:0/3009480382 shutdown_connections 2024-03-18T22:52:05.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.091+0000 7f1097c17740 1 --2- 172.21.15.40:0/3009480382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0fee50 0x5593fe1012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.091+0000 7f1097c17740 1 --2- 172.21.15.40:0/3009480382 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0f9ce0 0x5593fe0fe7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.091+0000 7f1097c17740 1 --2- 172.21.15.40:0/3009480382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0f92f0 0x5593fe0f9710 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.091+0000 7f1097c17740 1 -- 172.21.15.40:0/3009480382 >> 172.21.15.40:0/3009480382 conn(0x5593fe0f4860 msgr2=0x5593fe0f4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.091+0000 7f1097c17740 1 -- 172.21.15.40:0/3009480382 shutdown_connections 2024-03-18T22:52:05.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.091+0000 7f108ed9d640 1 --2- 172.21.15.40:0/2347874531 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f107c0a2950 0x7f107c0a4e10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f108003df60 tx=0x7f1080005510 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.091+0000 7f1097c17740 1 -- 172.21.15.40:0/3009480382 wait complete. 2024-03-18T22:52:05.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.092+0000 7f1097c17740 1 -- start start 2024-03-18T22:52:05.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.092+0000 7f1097c17740 1 Processor -- start 2024-03-18T22:52:05.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.092+0000 7f1097c17740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f9ce0 0x5593fe114690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.092+0000 7f1097c17740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0fee50 0x5593fe114bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.092+0000 7f1097c17740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe119b90 0x5593fe11bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.092+0000 7f1097c17740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5593fe112e50 con 0x5593fe0f9ce0 2024-03-18T22:52:05.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.092+0000 7f1097c17740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5593fe019310 con 0x5593fe119b90 2024-03-18T22:52:05.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.092+0000 7f1097c17740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5593fe112fd0 con 0x5593fe0fee50 2024-03-18T22:52:05.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.092+0000 7f1097c17740 10 client.0 _ll_register_callbacks cb 0x5593fe00d2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:05.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.093+0000 7f108e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f9ce0 0x5593fe114690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.093+0000 7f108e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f9ce0 0x5593fe114690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59264/0 (socket says 172.21.15.40:59264) 2024-03-18T22:52:05.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.093+0000 7f108e59c640 1 -- 172.21.15.40:0/3036382340 learned_addr learned my addr 172.21.15.40:0/3036382340 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:05.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.093+0000 7f108ed9d640 1 --2- 172.21.15.40:0/3036382340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe119b90 0x5593fe11bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.094+0000 7f108e59c640 1 -- 172.21.15.40:0/3036382340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0fee50 msgr2=0x5593fe114bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.094+0000 7f108dd9b640 1 --2- 172.21.15.40:0/3036382340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0fee50 0x5593fe114bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.094+0000 7f108e59c640 1 --2- 172.21.15.40:0/3036382340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0fee50 0x5593fe114bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.094+0000 7f108e59c640 1 -- 172.21.15.40:0/3036382340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe119b90 msgr2=0x5593fe11bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.094+0000 7f108e59c640 1 --2- 172.21.15.40:0/3036382340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe119b90 0x5593fe11bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.094+0000 7f108e59c640 1 -- 172.21.15.40:0/3036382340 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593fdd713f0 con 0x5593fe0f9ce0 2024-03-18T22:52:05.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.094+0000 7f108dd9b640 1 --2- 172.21.15.40:0/3036382340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0fee50 0x5593fe114bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:52:05.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.095+0000 7f108e59c640 1 --2- 172.21.15.40:0/3036382340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f9ce0 0x5593fe114690 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1078012740 tx=0x7f1078012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec99a72d0524f22 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.095+0000 7f1075ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:05.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.095+0000 7f1075ffb640 1 -- 172.21.15.40:0/3036382340 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107801f060 con 0x5593fe0f9ce0 2024-03-18T22:52:05.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.095+0000 7f1097c17740 1 -- 172.21.15.40:0/3036382340 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5593fe0bb3e0 con 0x5593fe0f9ce0 2024-03-18T22:52:05.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.096+0000 7f1075ffb640 1 -- 172.21.15.40:0/3036382340 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1078010b50 con 0x5593fe0f9ce0 2024-03-18T22:52:05.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.096+0000 7f1075ffb640 1 -- 172.21.15.40:0/3036382340 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1078026050 con 0x5593fe0f9ce0 2024-03-18T22:52:05.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.096+0000 7f1097c17740 10 client.17964 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:05.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.096+0000 7f1097c17740 1 -- 172.21.15.40:0/3036382340 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5593fde3b7d0 con 0x5593fe0f9ce0 2024-03-18T22:52:05.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.097+0000 7f1075ffb640 1 -- 172.21.15.40:0/3036382340 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5593fe0bb3e0 con 0x5593fe0f9ce0 2024-03-18T22:52:05.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.097+0000 7f1075ffb640 1 -- 172.21.15.40:0/3036382340 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f107800d340 con 0x5593fe0f9ce0 2024-03-18T22:52:05.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.097+0000 7f1097c17740 10 client.17964 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:05.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.097+0000 7f1097c17740 10 client.17964 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:52:05.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.097+0000 7f1097c17740 20 client.17964 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:05.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.097+0000 7f1097c17740 1 --2- 172.21.15.40:0/3036382340 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5593fe120db0 0x5593fe123250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.097+0000 7f1097c17740 4 client.17964 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:05.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.097+0000 7f1097c17740 1 -- 172.21.15.40:0/3036382340 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593fe06a9f0 con 0x5593fe120db0 2024-03-18T22:52:05.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.100+0000 7f108dd9b640 1 --2- 172.21.15.40:0/3036382340 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5593fe120db0 0x5593fe123250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.106+0000 7f108dd9b640 1 --2- 172.21.15.40:0/3036382340 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5593fe120db0 0x5593fe123250 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b24441ae755a0bfd server_cookie=82430fbd7872d670 in_seq=0 out_seq=0 2024-03-18T22:52:05.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.106+0000 7f1075ffb640 10 client.17964 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:52:05.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.109+0000 7f1075ffb640 1 -- 172.21.15.40:0/3036382340 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5593fe06a9f0 con 0x5593fe120db0 2024-03-18T22:52:05.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.109+0000 7f1075ffb640 10 client.17964 handle_command_reply: tid=0 2024-03-18T22:52:05.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.110+0000 7f1097c17740 1 client.17964 shutdown 2024-03-18T22:52:05.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.111+0000 7f1097c17740 1 -- 172.21.15.40:0/3036382340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f9ce0 msgr2=0x5593fe114690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.111+0000 7f1097c17740 1 --2- 172.21.15.40:0/3036382340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f9ce0 0x5593fe114690 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1078012740 tx=0x7f1078012c00 comp rx=0 tx=0).stop 2024-03-18T22:52:05.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.118+0000 7f1097c17740 1 -- 172.21.15.40:0/3036382340 shutdown_connections 2024-03-18T22:52:05.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.118+0000 7f1097c17740 1 --2- 172.21.15.40:0/3036382340 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5593fe120db0 0x5593fe123250 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.118+0000 7f1097c17740 1 --2- 172.21.15.40:0/3036382340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe119b90 0x5593fe11bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.118+0000 7f1097c17740 1 --2- 172.21.15.40:0/3036382340 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0fee50 0x5593fe114bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.118+0000 7f1097c17740 1 --2- 172.21.15.40:0/3036382340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f9ce0 0x5593fe114690 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.118+0000 7f1097c17740 1 -- 172.21.15.40:0/3036382340 >> 172.21.15.40:0/3036382340 conn(0x5593fe0f4860 msgr2=0x5593fe0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.118+0000 7f1075ffb640 0 client.17964 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:52:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.119+0000 7f1097c17740 1 -- 172.21.15.40:0/3036382340 shutdown_connections 2024-03-18T22:52:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.119+0000 7f1097c17740 1 -- 172.21.15.40:0/3036382340 wait complete. 2024-03-18T22:52:05.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.120+0000 7f1097c17740 20 client.17964 trim_cache size 0 max 16384 2024-03-18T22:52:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.156+0000 7f1097c17740 1 Processor -- start 2024-03-18T22:52:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.156+0000 7f1097c17740 1 -- start start 2024-03-18T22:52:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.157+0000 7f1097c17740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0fe990 0x5593fe0f7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.157+0000 7f1097c17740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f4860 0x5593fe0f8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.157+0000 7f1097c17740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0f9ce0 0x5593fe113960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.157+0000 7f1097c17740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f10901c1d30 con 0x5593fe0f4860 2024-03-18T22:52:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.157+0000 7f108dd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f4860 0x5593fe0f8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.157+0000 7f108dd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f4860 0x5593fe0f8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59270/0 (socket says 172.21.15.40:59270) 2024-03-18T22:52:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.157+0000 7f108dd9b640 1 -- 172.21.15.40:0/2268157116 learned_addr learned my addr 172.21.15.40:0/2268157116 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.157+0000 7f1097c17740 1 -- 172.21.15.40:0/2268157116 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f10901c38d0 con 0x5593fe0f9ce0 2024-03-18T22:52:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.157+0000 7f1097c17740 1 -- 172.21.15.40:0/2268157116 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f10901c2400 con 0x5593fe0fe990 2024-03-18T22:52:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.158+0000 7f108ed9d640 1 --2- 172.21.15.40:0/2268157116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0f9ce0 0x5593fe113960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.158+0000 7f108e59c640 1 --2- 172.21.15.40:0/2268157116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0fe990 0x5593fe0f7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.158+0000 7f108ed9d640 1 -- 172.21.15.40:0/2268157116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0fe990 msgr2=0x5593fe0f7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.158+0000 7f108ed9d640 1 --2- 172.21.15.40:0/2268157116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0fe990 0x5593fe0f7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.158+0000 7f108ed9d640 1 -- 172.21.15.40:0/2268157116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f4860 msgr2=0x5593fe0f8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.158+0000 7f108ed9d640 1 --2- 172.21.15.40:0/2268157116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f4860 0x5593fe0f8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.158+0000 7f108ed9d640 1 -- 172.21.15.40:0/2268157116 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593fe0ca880 con 0x5593fe0f9ce0 2024-03-18T22:52:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.158+0000 7f108dd9b640 1 --2- 172.21.15.40:0/2268157116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f4860 0x5593fe0f8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:52:05.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.159+0000 7f108ed9d640 1 --2- 172.21.15.40:0/2268157116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0f9ce0 0x5593fe113960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f108003d2a0 tx=0x7f1080050000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9085cbf23e346c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.159+0000 7f1077fff640 1 -- 172.21.15.40:0/2268157116 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080005850 con 0x5593fe0f9ce0 2024-03-18T22:52:05.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.160+0000 7f1077fff640 1 -- 172.21.15.40:0/2268157116 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1080004040 con 0x5593fe0f9ce0 2024-03-18T22:52:05.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.160+0000 7f1077fff640 1 -- 172.21.15.40:0/2268157116 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108003daa0 con 0x5593fe0f9ce0 2024-03-18T22:52:05.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.161+0000 7f1097c17740 1 -- 172.21.15.40:0/2268157116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0f9ce0 msgr2=0x5593fe113960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.161+0000 7f1097c17740 1 --2- 172.21.15.40:0/2268157116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0f9ce0 0x5593fe113960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f108003d2a0 tx=0x7f1080050000 comp rx=0 tx=0).stop 2024-03-18T22:52:05.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.163+0000 7f1097c17740 1 -- 172.21.15.40:0/2268157116 shutdown_connections 2024-03-18T22:52:05.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.163+0000 7f1097c17740 1 --2- 172.21.15.40:0/2268157116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0f9ce0 0x5593fe113960 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.163+0000 7f1097c17740 1 --2- 172.21.15.40:0/2268157116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f4860 0x5593fe0f8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.163+0000 7f1097c17740 1 --2- 172.21.15.40:0/2268157116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0fe990 0x5593fe0f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.163+0000 7f1097c17740 1 -- 172.21.15.40:0/2268157116 >> 172.21.15.40:0/2268157116 conn(0x5593fe0fa8e0 msgr2=0x5593fe0efa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.164+0000 7f1097c17740 1 -- 172.21.15.40:0/2268157116 shutdown_connections 2024-03-18T22:52:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.164+0000 7f1097c17740 1 -- 172.21.15.40:0/2268157116 wait complete. 2024-03-18T22:52:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.164+0000 7f1097c17740 1 -- start start 2024-03-18T22:52:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.164+0000 7f1097c17740 1 Processor -- start 2024-03-18T22:52:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.165+0000 7f1097c17740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0f4860 0x5593fe131830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.165+0000 7f1097c17740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f9ce0 0x5593fe131d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.165+0000 7f1097c17740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0fe990 0x5593fe132d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.165+0000 7f1097c17740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f10901c3200 con 0x5593fe0f9ce0 2024-03-18T22:52:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.165+0000 7f1097c17740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f10901c2ad0 con 0x5593fe0fe990 2024-03-18T22:52:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.165+0000 7f1097c17740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f10900737c0 con 0x5593fe0f4860 2024-03-18T22:52:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.165+0000 7f1097c17740 10 client.0 _ll_register_callbacks cb 0x7f107800a5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.165+0000 7f108dd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f9ce0 0x5593fe131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.165+0000 7f108dd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f9ce0 0x5593fe131d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59278/0 (socket says 172.21.15.40:59278) 2024-03-18T22:52:05.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.165+0000 7f108dd9b640 1 -- 172.21.15.40:0/3245856300 learned_addr learned my addr 172.21.15.40:0/3245856300 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:05.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.165+0000 7f108ed9d640 1 --2- 172.21.15.40:0/3245856300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0fe990 0x5593fe132d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.166+0000 7f108e59c640 1 --2- 172.21.15.40:0/3245856300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0f4860 0x5593fe131830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.166+0000 7f108dd9b640 1 -- 172.21.15.40:0/3245856300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0f4860 msgr2=0x5593fe131830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.166+0000 7f108dd9b640 1 --2- 172.21.15.40:0/3245856300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0f4860 0x5593fe131830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.166+0000 7f108dd9b640 1 -- 172.21.15.40:0/3245856300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0fe990 msgr2=0x5593fe132d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.166+0000 7f108dd9b640 1 --2- 172.21.15.40:0/3245856300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0fe990 0x5593fe132d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.166+0000 7f108dd9b640 1 -- 172.21.15.40:0/3245856300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593fe133370 con 0x5593fe0f9ce0 2024-03-18T22:52:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.166+0000 7f108e59c640 1 --2- 172.21.15.40:0/3245856300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0f4860 0x5593fe131830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:52:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.166+0000 7f108dd9b640 1 --2- 172.21.15.40:0/3245856300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f9ce0 0x5593fe131d70 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f10880cda00 tx=0x7f1088000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab9cdc68e02ce6a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.166+0000 7f1075ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.166+0000 7f1075ffb640 1 -- 172.21.15.40:0/3245856300 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10880b5070 con 0x5593fe0f9ce0 2024-03-18T22:52:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.167+0000 7f1075ffb640 1 -- 172.21.15.40:0/3245856300 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f108809abd0 con 0x5593fe0f9ce0 2024-03-18T22:52:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.167+0000 7f1097c17740 1 -- 172.21.15.40:0/3245856300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1080004040 con 0x5593fe0f9ce0 2024-03-18T22:52:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.167+0000 7f1075ffb640 1 -- 172.21.15.40:0/3245856300 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1088085e60 con 0x5593fe0f9ce0 2024-03-18T22:52:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.167+0000 7f1097c17740 10 client.17967 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.167+0000 7f1097c17740 1 -- 172.21.15.40:0/3245856300 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5593fe134650 con 0x5593fe0f9ce0 2024-03-18T22:52:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.167+0000 7f1075ffb640 1 -- 172.21.15.40:0/3245856300 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1080004040 con 0x5593fe0f9ce0 2024-03-18T22:52:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.168+0000 7f1075ffb640 1 -- 172.21.15.40:0/3245856300 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f1088098f40 con 0x5593fe0f9ce0 2024-03-18T22:52:05.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.168+0000 7f1097c17740 10 client.17967 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:05.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.168+0000 7f1097c17740 10 client.17967 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:52:05.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.168+0000 7f1097c17740 20 client.17967 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:05.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.168+0000 7f1097c17740 1 --2- 172.21.15.40:0/3245856300 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5593fe0fee50 0x5593fe189310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.168+0000 7f1097c17740 4 client.17967 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:52:05.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.168+0000 7f1097c17740 1 -- 172.21.15.40:0/3245856300 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5593fe064d40 con 0x5593fe0fee50 2024-03-18T22:52:05.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.168+0000 7f108e59c640 1 --2- 172.21.15.40:0/3245856300 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5593fe0fee50 0x5593fe189310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.169+0000 7f108e59c640 1 --2- 172.21.15.40:0/3245856300 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5593fe0fee50 0x5593fe189310 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=479a544cabc0d3a8 server_cookie=bed23e2c3e998f4a in_seq=0 out_seq=0 2024-03-18T22:52:05.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.169+0000 7f1075ffb640 10 client.17967 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:52:05.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.169+0000 7f1075ffb640 1 -- 172.21.15.40:0/3245856300 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5593fe064d40 con 0x5593fe0fee50 2024-03-18T22:52:05.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.169+0000 7f1075ffb640 10 client.17967 handle_command_reply: tid=0 2024-03-18T22:52:05.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.169+0000 7f1097c17740 1 client.17967 shutdown 2024-03-18T22:52:05.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.170+0000 7f1097c17740 1 -- 172.21.15.40:0/3245856300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f9ce0 msgr2=0x5593fe131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.170+0000 7f1097c17740 1 --2- 172.21.15.40:0/3245856300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f9ce0 0x5593fe131d70 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f10880cda00 tx=0x7f1088000f40 comp rx=0 tx=0).stop 2024-03-18T22:52:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.170+0000 7f1097c17740 1 -- 172.21.15.40:0/3245856300 shutdown_connections 2024-03-18T22:52:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.170+0000 7f1097c17740 1 --2- 172.21.15.40:0/3245856300 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x5593fe0fee50 0x5593fe189310 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.170+0000 7f1097c17740 1 --2- 172.21.15.40:0/3245856300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5593fe0fe990 0x5593fe132d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.170+0000 7f1097c17740 1 --2- 172.21.15.40:0/3245856300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5593fe0f9ce0 0x5593fe131d70 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.170+0000 7f1097c17740 1 --2- 172.21.15.40:0/3245856300 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5593fe0f4860 0x5593fe131830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.170+0000 7f1097c17740 1 -- 172.21.15.40:0/3245856300 >> 172.21.15.40:0/3245856300 conn(0x5593fe0fa8e0 msgr2=0x5593fe0efa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.170+0000 7f1075ffb640 0 client.17967 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:52:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.170+0000 7f1097c17740 1 -- 172.21.15.40:0/3245856300 shutdown_connections 2024-03-18T22:52:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.171+0000 7f1097c17740 1 -- 172.21.15.40:0/3245856300 wait complete. 2024-03-18T22:52:05.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.171+0000 7f1097c17740 20 client.17967 trim_cache size 0 max 16384 2024-03-18T22:52:05.173 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:52:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.172+0000 7f1094f59640 1 -- 172.21.15.40:0/2347874531 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f107c0a2950 msgr2=0x7f107c0a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.172+0000 7f1094f59640 1 --2- 172.21.15.40:0/2347874531 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f107c0a2950 0x7f107c0a4e10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f108003df60 tx=0x7f1080005510 comp rx=0 tx=0).stop 2024-03-18T22:52:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.172+0000 7f1094f59640 1 -- 172.21.15.40:0/2347874531 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10901007f0 msgr2=0x7f1090195cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.172+0000 7f1094f59640 1 --2- 172.21.15.40:0/2347874531 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10901007f0 0x7f1090195cc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f108800ebc0 tx=0x7f108800c6a0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.173+0000 7f1094f59640 1 -- 172.21.15.40:0/2347874531 shutdown_connections 2024-03-18T22:52:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.173+0000 7f1094f59640 1 --2- 172.21.15.40:0/2347874531 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f107c0a2950 0x7f107c0a4e10 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.174+0000 7f1094f59640 1 --2- 172.21.15.40:0/2347874531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1090101190 0x7f10901bed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.174+0000 7f1094f59640 1 --2- 172.21.15.40:0/2347874531 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f10901007f0 0x7f1090195cc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.174+0000 7f1094f59640 1 --2- 172.21.15.40:0/2347874531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10900ff6c0 0x7f1090195780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.174+0000 7f1094f59640 1 -- 172.21.15.40:0/2347874531 >> 172.21.15.40:0/2347874531 conn(0x7f10900fb000 msgr2=0x7f1090103a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.174+0000 7f1094f59640 1 -- 172.21.15.40:0/2347874531 shutdown_connections 2024-03-18T22:52:05.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.174+0000 7f1094f59640 1 -- 172.21.15.40:0/2347874531 wait complete. 2024-03-18T22:52:05.200 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:52:05.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.431+0000 7efd77fff640 1 -- 172.21.15.40:0/2446764228 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70034db0 con 0x7efd8010db40 2024-03-18T22:52:05.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.431+0000 7efd878d3640 1 -- 172.21.15.40:0/2446764228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd8010db40 msgr2=0x7efd8010dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.431+0000 7efd878d3640 1 --2- 172.21.15.40:0/2446764228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd8010db40 0x7efd8010dfc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efd70009860 tx=0x7efd7002bb90 comp rx=0 tx=0).stop 2024-03-18T22:52:05.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.434+0000 7efd878d3640 1 -- 172.21.15.40:0/2446764228 shutdown_connections 2024-03-18T22:52:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.434+0000 7efd878d3640 1 --2- 172.21.15.40:0/2446764228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd8010db40 0x7efd8010dfc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.434+0000 7efd878d3640 1 --2- 172.21.15.40:0/2446764228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd8010b190 0x7efd8010d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.434+0000 7efd878d3640 1 --2- 172.21.15.40:0/2446764228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd80108820 0x7efd8010ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.434+0000 7efd878d3640 1 -- 172.21.15.40:0/2446764228 >> 172.21.15.40:0/2446764228 conn(0x7efd8006e8a0 msgr2=0x7efd80070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.436+0000 7efd878d3640 1 -- 172.21.15.40:0/2446764228 shutdown_connections 2024-03-18T22:52:05.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.437+0000 7efd878d3640 1 -- 172.21.15.40:0/2446764228 wait complete. 2024-03-18T22:52:05.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.438+0000 7efd878d3640 1 Processor -- start 2024-03-18T22:52:05.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.438+0000 7efd878d3640 1 -- start start 2024-03-18T22:52:05.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.438+0000 7efd878d3640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd80108820 0x7efd801a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.438+0000 7efd878d3640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd8010b190 0x7efd801a6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.438+0000 7efd878d3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd8010db40 0x7efd801ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.438+0000 7efd878d3640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efd8011c200 con 0x7efd8010b190 2024-03-18T22:52:05.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.438+0000 7efd878d3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efd8011c080 con 0x7efd8010db40 2024-03-18T22:52:05.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.438+0000 7efd878d3640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7efd8011c380 con 0x7efd80108820 2024-03-18T22:52:05.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.439+0000 7efd85e70640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd8010db40 0x7efd801ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.439+0000 7efd85e70640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd8010db40 0x7efd801ade30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48964/0 (socket says 172.21.15.40:48964) 2024-03-18T22:52:05.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.439+0000 7efd85e70640 1 -- 172.21.15.40:0/691513563 learned_addr learned my addr 172.21.15.40:0/691513563 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.440+0000 7efd8566f640 1 --2- 172.21.15.40:0/691513563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd80108820 0x7efd801a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.440+0000 7efd85e70640 1 -- 172.21.15.40:0/691513563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd80108820 msgr2=0x7efd801a68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.440+0000 7efd85e70640 1 --2- 172.21.15.40:0/691513563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd80108820 0x7efd801a68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.440+0000 7efd85e70640 1 -- 172.21.15.40:0/691513563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd8010b190 msgr2=0x7efd801a6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.440+0000 7efd85e70640 1 --2- 172.21.15.40:0/691513563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd8010b190 0x7efd801a6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.440+0000 7efd85e70640 1 -- 172.21.15.40:0/691513563 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd80110080 con 0x7efd8010db40 2024-03-18T22:52:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.441+0000 7efd85e70640 1 --2- 172.21.15.40:0/691513563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd8010db40 0x7efd801ade30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efd7002c1a0 tx=0x7efd70004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.445+0000 7efd767fc640 1 -- 172.21.15.40:0/691513563 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70041020 con 0x7efd8010db40 2024-03-18T22:52:05.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.446+0000 7efd767fc640 1 -- 172.21.15.40:0/691513563 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd700044a0 con 0x7efd8010db40 2024-03-18T22:52:05.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.446+0000 7efd878d3640 1 -- 172.21.15.40:0/691513563 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd80110310 con 0x7efd8010db40 2024-03-18T22:52:05.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.446+0000 7efd878d3640 1 -- 172.21.15.40:0/691513563 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd80110880 con 0x7efd8010db40 2024-03-18T22:52:05.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.447+0000 7efd767fc640 1 -- 172.21.15.40:0/691513563 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70039040 con 0x7efd8010db40 2024-03-18T22:52:05.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.450+0000 7efd8a595740 1 Processor -- start 2024-03-18T22:52:05.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.451+0000 7efd8a595740 1 -- start start 2024-03-18T22:52:05.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.451+0000 7efd8a595740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823a62f0 0x55e7823a6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.451+0000 7efd8a595740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 0x55e7823ab7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.451+0000 7efd8a595740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823abe50 0x55e7823ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.451+0000 7efd8a595740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e78230a4b0 con 0x55e7823a62f0 2024-03-18T22:52:05.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.451+0000 7efd8a595740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e7822b2d70 con 0x55e7823abe50 2024-03-18T22:52:05.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.451+0000 7efd8a595740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e782306550 con 0x55e7823a6ce0 2024-03-18T22:52:05.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.453+0000 7efd8566f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 0x55e7823ab7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.453+0000 7efd84e6e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823a62f0 0x55e7823a6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.453+0000 7efd84e6e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823a62f0 0x55e7823a6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59294/0 (socket says 172.21.15.40:59294) 2024-03-18T22:52:05.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.453+0000 7efd84e6e640 1 -- 172.21.15.40:0/4118800127 learned_addr learned my addr 172.21.15.40:0/4118800127 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:05.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.454+0000 7efd84e6e640 1 -- 172.21.15.40:0/4118800127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 msgr2=0x55e7823ab7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.454+0000 7efd84e6e640 1 --2- 172.21.15.40:0/4118800127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 0x55e7823ab7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.454+0000 7efd84e6e640 1 -- 172.21.15.40:0/4118800127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823abe50 msgr2=0x55e7823ae2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.454+0000 7efd84e6e640 1 --2- 172.21.15.40:0/4118800127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823abe50 0x55e7823ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.454+0000 7efd84e6e640 1 -- 172.21.15.40:0/4118800127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7822d5210 con 0x55e7823a62f0 2024-03-18T22:52:05.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.454+0000 7efd84e6e640 1 --2- 172.21.15.40:0/4118800127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823a62f0 0x55e7823a6710 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7efd7c00bd50 tx=0x7efd7c00d420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58874fa0d4ddc8f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.455+0000 7efd767fc640 1 -- 172.21.15.40:0/691513563 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7efd70044020 con 0x7efd8010db40 2024-03-18T22:52:05.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.455+0000 7efd6b7fe640 1 -- 172.21.15.40:0/4118800127 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c019070 con 0x55e7823a62f0 2024-03-18T22:52:05.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.455+0000 7efd6b7fe640 1 -- 172.21.15.40:0/4118800127 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd7c006590 con 0x55e7823a62f0 2024-03-18T22:52:05.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.456+0000 7efd8a595740 1 -- 172.21.15.40:0/4118800127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823a62f0 msgr2=0x55e7823a6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.456+0000 7efd8a595740 1 --2- 172.21.15.40:0/4118800127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823a62f0 0x55e7823a6710 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7efd7c00bd50 tx=0x7efd7c00d420 comp rx=0 tx=0).stop 2024-03-18T22:52:05.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.457+0000 7efd767fc640 1 --2- 172.21.15.40:0/691513563 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efd6c0a29d0 0x7efd6c0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.458+0000 7efd767fc640 1 -- 172.21.15.40:0/691513563 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7efd700f07b0 con 0x7efd8010db40 2024-03-18T22:52:05.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.461+0000 7efd8a595740 1 -- 172.21.15.40:0/4118800127 shutdown_connections 2024-03-18T22:52:05.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.461+0000 7efd8a595740 1 --2- 172.21.15.40:0/4118800127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823abe50 0x55e7823ae2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.461+0000 7efd8a595740 1 --2- 172.21.15.40:0/4118800127 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 0x55e7823ab7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.461+0000 7efd8a595740 1 --2- 172.21.15.40:0/4118800127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823a62f0 0x55e7823a6710 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.461+0000 7efd8a595740 1 -- 172.21.15.40:0/4118800127 >> 172.21.15.40:0/4118800127 conn(0x55e7823a1860 msgr2=0x55e7823a1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.461+0000 7efd84e6e640 1 --2- 172.21.15.40:0/691513563 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efd6c0a29d0 0x7efd6c0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.461+0000 7efd8a595740 1 -- 172.21.15.40:0/4118800127 shutdown_connections 2024-03-18T22:52:05.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.465+0000 7efd8a595740 1 -- 172.21.15.40:0/4118800127 wait complete. 2024-03-18T22:52:05.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.465+0000 7efd8a595740 1 -- start start 2024-03-18T22:52:05.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.465+0000 7efd8a595740 1 Processor -- start 2024-03-18T22:52:05.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.469+0000 7efd84e6e640 1 --2- 172.21.15.40:0/691513563 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efd6c0a29d0 0x7efd6c0a4e90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efd7c006880 tx=0x7efd7c006040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.470+0000 7efd8a595740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 0x55e7823b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.470+0000 7efd8a595740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823a6ce0 0x55e7823b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.470+0000 7efd8a595740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823abe50 0x55e7823b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.470+0000 7efd8a595740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e7823aee50 con 0x55e7823a6ce0 2024-03-18T22:52:05.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.470+0000 7efd8a595740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e7822c6310 con 0x55e7823a62f0 2024-03-18T22:52:05.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.470+0000 7efd8a595740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e7823aefd0 con 0x55e7823abe50 2024-03-18T22:52:05.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.470+0000 7efd8a595740 10 client.0 _ll_register_callbacks cb 0x55e7822ba2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:05.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.471+0000 7efd85e70640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823abe50 0x55e7823b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.471+0000 7efd85e70640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823abe50 0x55e7823b7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:52344/0 (socket says 172.21.15.40:52344) 2024-03-18T22:52:05.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.471+0000 7efd85e70640 1 -- 172.21.15.40:0/1995213361 learned_addr learned my addr 172.21.15.40:0/1995213361 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:05.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.471+0000 7efd8566f640 1 --2- 172.21.15.40:0/1995213361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 0x55e7823b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.471+0000 7efd8566f640 1 -- 172.21.15.40:0/1995213361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823abe50 msgr2=0x55e7823b7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.471+0000 7efd8566f640 1 --2- 172.21.15.40:0/1995213361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823abe50 0x55e7823b7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.472+0000 7efd8566f640 1 -- 172.21.15.40:0/1995213361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823a6ce0 msgr2=0x55e7823b0b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:52:05.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.472+0000 7efd8566f640 1 --2- 172.21.15.40:0/1995213361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823a6ce0 0x55e7823b0b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.472+0000 7efd8566f640 1 -- 172.21.15.40:0/1995213361 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e78201e3f0 con 0x55e7823a62f0 2024-03-18T22:52:05.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.472+0000 7efd8566f640 1 --2- 172.21.15.40:0/1995213361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 0x55e7823b05e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55e7823a70e0 tx=0x7efd7800b360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a723b05913913d5c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.474+0000 7efd68ff9640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:52:05.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.476+0000 7efd68ff9640 1 -- 172.21.15.40:0/1995213361 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7801d020 con 0x55e7823a62f0 2024-03-18T22:52:05.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.476+0000 7efd8a595740 1 -- 172.21.15.40:0/1995213361 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd7c006590 con 0x55e7823a62f0 2024-03-18T22:52:05.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.476+0000 7efd68ff9640 1 -- 172.21.15.40:0/1995213361 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7efd78019400 con 0x55e7823a62f0 2024-03-18T22:52:05.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.482+0000 7efd68ff9640 1 -- 172.21.15.40:0/1995213361 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efd78019600 con 0x55e7823a62f0 2024-03-18T22:52:05.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.482+0000 7efd8a595740 10 client.27785 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:05.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.482+0000 7efd8a595740 1 -- 172.21.15.40:0/1995213361 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e7820e87d0 con 0x55e7823a62f0 2024-03-18T22:52:05.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.483+0000 7efd68ff9640 1 -- 172.21.15.40:0/1995213361 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd7c006590 con 0x55e7823a62f0 2024-03-18T22:52:05.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.483+0000 7efd68ff9640 1 -- 172.21.15.40:0/1995213361 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7efd7800b040 con 0x55e7823a62f0 2024-03-18T22:52:05.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.483+0000 7efd8a595740 10 client.27785 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:05.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.483+0000 7efd8a595740 10 client.27785 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:52:05.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.483+0000 7efd8a595740 20 client.27785 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:05.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.483+0000 7efd8a595740 1 --2- 172.21.15.40:0/1995213361 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55e7823bc900 0x55e7823bed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.483+0000 7efd8a595740 4 client.27785 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:05.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.483+0000 7efd8a595740 1 -- 172.21.15.40:0/1995213361 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7823683e0 con 0x55e7823bc900 2024-03-18T22:52:05.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.484+0000 7efd84e6e640 1 --2- 172.21.15.40:0/1995213361 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55e7823bc900 0x55e7823bed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.486+0000 7efd84e6e640 1 --2- 172.21.15.40:0/1995213361 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55e7823bc900 0x55e7823bed60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=449d89e2f31a5199 server_cookie=a64b8a333990c382 in_seq=0 out_seq=0 2024-03-18T22:52:05.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.486+0000 7efd68ff9640 10 client.27785 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:52:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.488+0000 7efd68ff9640 1 -- 172.21.15.40:0/1995213361 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e7823683e0 con 0x55e7823bc900 2024-03-18T22:52:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.488+0000 7efd68ff9640 10 client.27785 handle_command_reply: tid=0 2024-03-18T22:52:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.490+0000 7efd8a595740 1 client.27785 shutdown 2024-03-18T22:52:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.490+0000 7efd8a595740 1 -- 172.21.15.40:0/1995213361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 msgr2=0x55e7823b05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.490+0000 7efd8a595740 1 --2- 172.21.15.40:0/1995213361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 0x55e7823b05e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55e7823a70e0 tx=0x7efd7800b360 comp rx=0 tx=0).stop 2024-03-18T22:52:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.490+0000 7efd8a595740 1 -- 172.21.15.40:0/1995213361 shutdown_connections 2024-03-18T22:52:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.490+0000 7efd8a595740 1 --2- 172.21.15.40:0/1995213361 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55e7823bc900 0x55e7823bed60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.490+0000 7efd8a595740 1 --2- 172.21.15.40:0/1995213361 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823abe50 0x55e7823b7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.490+0000 7efd8a595740 1 --2- 172.21.15.40:0/1995213361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823a6ce0 0x55e7823b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.490+0000 7efd8a595740 1 --2- 172.21.15.40:0/1995213361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 0x55e7823b05e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.490+0000 7efd8a595740 1 -- 172.21.15.40:0/1995213361 >> 172.21.15.40:0/1995213361 conn(0x55e7823a1860 msgr2=0x55e7823a0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.490+0000 7efd68ff9640 0 client.27785 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:52:05.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.493+0000 7efd8a595740 1 -- 172.21.15.40:0/1995213361 shutdown_connections 2024-03-18T22:52:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.494+0000 7efd8a595740 1 -- 172.21.15.40:0/1995213361 wait complete. 2024-03-18T22:52:05.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.494+0000 7efd8a595740 20 client.27785 trim_cache size 0 max 16384 2024-03-18T22:52:05.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.524+0000 7efd8a595740 1 Processor -- start 2024-03-18T22:52:05.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.524+0000 7efd8a595740 1 -- start start 2024-03-18T22:52:05.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.525+0000 7efd8a595740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 0x55e7823a1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.525+0000 7efd8a595740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 0x55e7823b79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.525+0000 7efd8a595740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823abe50 0x55e7823a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.525+0000 7efd8a595740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efd6400feb0 con 0x55e7823abe50 2024-03-18T22:52:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.525+0000 7efd8a595740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efd64011a50 con 0x55e7823a62f0 2024-03-18T22:52:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.525+0000 7efd8a595740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7efd64010580 con 0x55e7823a6ce0 2024-03-18T22:52:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.525+0000 7efd84e6e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 0x55e7823b79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.525+0000 7efd85e70640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823abe50 0x55e7823a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.525+0000 7efd85e70640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823abe50 0x55e7823a4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59306/0 (socket says 172.21.15.40:59306) 2024-03-18T22:52:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.525+0000 7efd84e6e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 0x55e7823b79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:52350/0 (socket says 172.21.15.40:52350) 2024-03-18T22:52:05.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.525+0000 7efd85e70640 1 -- 172.21.15.40:0/2045710563 learned_addr learned my addr 172.21.15.40:0/2045710563 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:05.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.526+0000 7efd8566f640 1 --2- 172.21.15.40:0/2045710563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 0x55e7823a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.526+0000 7efd85e70640 1 -- 172.21.15.40:0/2045710563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 msgr2=0x55e7823b79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.526+0000 7efd85e70640 1 --2- 172.21.15.40:0/2045710563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 0x55e7823b79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.526+0000 7efd85e70640 1 -- 172.21.15.40:0/2045710563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 msgr2=0x55e7823a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.526+0000 7efd85e70640 1 --2- 172.21.15.40:0/2045710563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 0x55e7823a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.526+0000 7efd85e70640 1 -- 172.21.15.40:0/2045710563 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e782377880 con 0x55e7823abe50 2024-03-18T22:52:05.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.526+0000 7efd84e6e640 1 --2- 172.21.15.40:0/2045710563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 0x55e7823b79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:52:05.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.527+0000 7efd85e70640 1 --2- 172.21.15.40:0/2045710563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823abe50 0x55e7823a4ac0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7efd70004650 tx=0x7efd700349d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c724dd3a6de03f2c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.527+0000 7efd6affd640 1 -- 172.21.15.40:0/2045710563 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd700da070 con 0x55e7823abe50 2024-03-18T22:52:05.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.527+0000 7efd6affd640 1 -- 172.21.15.40:0/2045710563 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd700d3070 con 0x55e7823abe50 2024-03-18T22:52:05.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.527+0000 7efd6affd640 1 -- 172.21.15.40:0/2045710563 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd700f03a0 con 0x55e7823abe50 2024-03-18T22:52:05.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.528+0000 7efd8a595740 1 -- 172.21.15.40:0/2045710563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823abe50 msgr2=0x55e7823a4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.528+0000 7efd8a595740 1 --2- 172.21.15.40:0/2045710563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823abe50 0x55e7823a4ac0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7efd70004650 tx=0x7efd700349d0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.529+0000 7efd8a595740 1 -- 172.21.15.40:0/2045710563 shutdown_connections 2024-03-18T22:52:05.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.529+0000 7efd8a595740 1 --2- 172.21.15.40:0/2045710563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823abe50 0x55e7823a4ac0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.529+0000 7efd8a595740 1 --2- 172.21.15.40:0/2045710563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 0x55e7823b79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.529+0000 7efd8a595740 1 --2- 172.21.15.40:0/2045710563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 0x55e7823a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.529+0000 7efd8a595740 1 -- 172.21.15.40:0/2045710563 >> 172.21.15.40:0/2045710563 conn(0x55e7823a1860 msgr2=0x55e7823a1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.529+0000 7efd8a595740 1 -- 172.21.15.40:0/2045710563 shutdown_connections 2024-03-18T22:52:05.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.529+0000 7efd8a595740 1 -- 172.21.15.40:0/2045710563 wait complete. 2024-03-18T22:52:05.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.530+0000 7efd8a595740 1 -- start start 2024-03-18T22:52:05.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.531+0000 7efd8a595740 1 Processor -- start 2024-03-18T22:52:05.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.531+0000 7efd8a595740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 0x55e7823b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.532+0000 7efd8a595740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 0x55e7823b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.532+0000 7efd8566f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 0x55e7823b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.532+0000 7efd8566f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 0x55e7823b49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48996/0 (socket says 172.21.15.40:48996) 2024-03-18T22:52:05.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.532+0000 7efd8566f640 1 -- 172.21.15.40:0/3783306286 learned_addr learned my addr 172.21.15.40:0/3783306286 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:05.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.532+0000 7efd84e6e640 1 --2- 172.21.15.40:0/3783306286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 0x55e7823b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.532+0000 7efd8a595740 1 --2- 172.21.15.40:0/3783306286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823abe50 0x55e782420c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.532+0000 7efd8a595740 1 -- 172.21.15.40:0/3783306286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efd64011380 con 0x55e7823abe50 2024-03-18T22:52:05.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.532+0000 7efd8a595740 1 -- 172.21.15.40:0/3783306286 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efd64010c50 con 0x55e7823a62f0 2024-03-18T22:52:05.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.532+0000 7efd8a595740 1 -- 172.21.15.40:0/3783306286 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7efd6400de90 con 0x55e7823a6ce0 2024-03-18T22:52:05.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.532+0000 7efd85e70640 1 --2- 172.21.15.40:0/3783306286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823abe50 0x55e782420c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.533+0000 7efd85e70640 1 -- 172.21.15.40:0/3783306286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 msgr2=0x55e7823b4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.533+0000 7efd85e70640 1 --2- 172.21.15.40:0/3783306286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 0x55e7823b4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.533+0000 7efd85e70640 1 -- 172.21.15.40:0/3783306286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 msgr2=0x55e7823b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.533+0000 7efd85e70640 1 --2- 172.21.15.40:0/3783306286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 0x55e7823b49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.533+0000 7efd85e70640 1 -- 172.21.15.40:0/3783306286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7824211d0 con 0x55e7823abe50 2024-03-18T22:52:05.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.534+0000 7efd8a595740 10 client.0 _ll_register_callbacks cb 0x7efd7800f750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:05.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.534+0000 7efd8a595740 1 -- 172.21.15.40:0/3783306286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd700d3070 con 0x55e7823abe50 2024-03-18T22:52:05.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.534+0000 7efd85e70640 1 --2- 172.21.15.40:0/3783306286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823abe50 0x55e782420c90 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7efd70004650 tx=0x7efd700bd190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=864ca71a0bacf0d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.534+0000 7efd68ff9640 10 client.17979 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:05.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.535+0000 7efd68ff9640 1 -- 172.21.15.40:0/3783306286 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd700da070 con 0x55e7823abe50 2024-03-18T22:52:05.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.535+0000 7efd68ff9640 1 -- 172.21.15.40:0/3783306286 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7efd700c5220 con 0x55e7823abe50 2024-03-18T22:52:05.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.535+0000 7efd68ff9640 1 -- 172.21.15.40:0/3783306286 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efd700c0e50 con 0x55e7823abe50 2024-03-18T22:52:05.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.535+0000 7efd8a595740 10 client.17979 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:05.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.535+0000 7efd8a595740 1 -- 172.21.15.40:0/3783306286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({fsmap=35}) -- 0x55e782422180 con 0x55e7823abe50 2024-03-18T22:52:05.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.536+0000 7efd68ff9640 1 -- 172.21.15.40:0/3783306286 <== mon.0 v2:172.21.15.40:3300/0 4 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7efd7003c030 con 0x55e7823abe50 2024-03-18T22:52:05.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.536+0000 7efd8a595740 10 client.17979 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:05.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.536+0000 7efd8a595740 10 client.17979 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:52:05.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.536+0000 7efd8a595740 20 client.17979 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:05.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.536+0000 7efd8a595740 1 --2- 172.21.15.40:0/3783306286 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55e7824228e0 0x55e782424d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.536+0000 7efd8a595740 4 client.17979 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:52:05.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.536+0000 7efd8a595740 1 -- 172.21.15.40:0/3783306286 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e7823179f0 con 0x55e7824228e0 2024-03-18T22:52:05.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.536+0000 7efd8566f640 1 --2- 172.21.15.40:0/3783306286 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55e7824228e0 0x55e782424d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.537+0000 7efd8566f640 1 --2- 172.21.15.40:0/3783306286 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55e7824228e0 0x55e782424d00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=188495d46eff67ba server_cookie=5b4939acdbaf973e in_seq=0 out_seq=0 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.537+0000 7efd68ff9640 10 client.17979 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.537+0000 7efd68ff9640 1 -- 172.21.15.40:0/3783306286 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e7823179f0 con 0x55e7824228e0 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.537+0000 7efd68ff9640 10 client.17979 handle_command_reply: tid=0 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.537+0000 7efd8a595740 1 client.17979 shutdown 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.538+0000 7efd8a595740 1 -- 172.21.15.40:0/3783306286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823abe50 msgr2=0x55e782420c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.538+0000 7efd8a595740 1 --2- 172.21.15.40:0/3783306286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823abe50 0x55e782420c90 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7efd70004650 tx=0x7efd700bd190 comp rx=0 tx=0).stop 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.538+0000 7efd8a595740 1 -- 172.21.15.40:0/3783306286 shutdown_connections 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.538+0000 7efd8a595740 1 --2- 172.21.15.40:0/3783306286 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55e7824228e0 0x55e782424d00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.538+0000 7efd8a595740 1 --2- 172.21.15.40:0/3783306286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7823abe50 0x55e782420c90 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.538+0000 7efd8a595740 1 --2- 172.21.15.40:0/3783306286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7823a6ce0 0x55e7823b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.538+0000 7efd8a595740 1 --2- 172.21.15.40:0/3783306286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7823a62f0 0x55e7823b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.538+0000 7efd8a595740 1 -- 172.21.15.40:0/3783306286 >> 172.21.15.40:0/3783306286 conn(0x55e7823a1860 msgr2=0x55e7823a4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.538+0000 7efd68ff9640 0 client.17979 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.538+0000 7efd8a595740 1 -- 172.21.15.40:0/3783306286 shutdown_connections 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.538+0000 7efd8a595740 1 -- 172.21.15.40:0/3783306286 wait complete. 2024-03-18T22:52:05.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.538+0000 7efd8a595740 20 client.17979 trim_cache size 0 max 16384 2024-03-18T22:52:05.540 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:52:05.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.541+0000 7efd878d3640 1 -- 172.21.15.40:0/691513563 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efd6c0a29d0 msgr2=0x7efd6c0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.541+0000 7efd878d3640 1 --2- 172.21.15.40:0/691513563 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efd6c0a29d0 0x7efd6c0a4e90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efd7c006880 tx=0x7efd7c006040 comp rx=0 tx=0).stop 2024-03-18T22:52:05.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.542+0000 7efd878d3640 1 -- 172.21.15.40:0/691513563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd8010db40 msgr2=0x7efd801ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.542+0000 7efd878d3640 1 --2- 172.21.15.40:0/691513563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd8010db40 0x7efd801ade30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efd7002c1a0 tx=0x7efd70004270 comp rx=0 tx=0).stop 2024-03-18T22:52:05.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.543+0000 7efd878d3640 1 -- 172.21.15.40:0/691513563 shutdown_connections 2024-03-18T22:52:05.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.543+0000 7efd878d3640 1 --2- 172.21.15.40:0/691513563 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efd6c0a29d0 0x7efd6c0a4e90 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.544+0000 7efd878d3640 1 --2- 172.21.15.40:0/691513563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd8010db40 0x7efd801ade30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.544+0000 7efd878d3640 1 --2- 172.21.15.40:0/691513563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd8010b190 0x7efd801a6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.544+0000 7efd878d3640 1 --2- 172.21.15.40:0/691513563 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efd80108820 0x7efd801a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.544+0000 7efd878d3640 1 -- 172.21.15.40:0/691513563 >> 172.21.15.40:0/691513563 conn(0x7efd8006e8a0 msgr2=0x7efd8010d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.544+0000 7efd878d3640 1 -- 172.21.15.40:0/691513563 shutdown_connections 2024-03-18T22:52:05.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.544+0000 7efd878d3640 1 -- 172.21.15.40:0/691513563 wait complete. 2024-03-18T22:52:05.566 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:52:05.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.822+0000 7f69963c4640 1 -- 172.21.15.40:0/809592229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6990108820 msgr2=0x7f699010ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.822+0000 7f69963c4640 1 --2- 172.21.15.40:0/809592229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6990108820 0x7f699010ac50 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f698401a910 tx=0x7f698403cc60 comp rx=0 tx=0).stop 2024-03-18T22:52:05.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.826+0000 7f69963c4640 1 -- 172.21.15.40:0/809592229 shutdown_connections 2024-03-18T22:52:05.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.826+0000 7f69963c4640 1 --2- 172.21.15.40:0/809592229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f699010db40 0x7f699010dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.826+0000 7f69963c4640 1 --2- 172.21.15.40:0/809592229 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f699010b190 0x7f699010d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.826+0000 7f69963c4640 1 --2- 172.21.15.40:0/809592229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6990108820 0x7f699010ac50 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.826+0000 7f69963c4640 1 -- 172.21.15.40:0/809592229 >> 172.21.15.40:0/809592229 conn(0x7f699006e8a0 msgr2=0x7f6990070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.829+0000 7f69963c4640 1 -- 172.21.15.40:0/809592229 shutdown_connections 2024-03-18T22:52:05.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.832+0000 7f69963c4640 1 -- 172.21.15.40:0/809592229 wait complete. 2024-03-18T22:52:05.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.832+0000 7f69963c4640 1 Processor -- start 2024-03-18T22:52:05.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.833+0000 7f69963c4640 1 -- start start 2024-03-18T22:52:05.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.833+0000 7f69963c4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6990108820 0x7f699019e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.834+0000 7f69963c4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f699010b190 0x7f699019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.834+0000 7f69963c4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699010db40 0x7f69901a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.834+0000 7f69963c4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f69901119c0 con 0x7f699010db40 2024-03-18T22:52:05.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.834+0000 7f69963c4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6990111840 con 0x7f699010b190 2024-03-18T22:52:05.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.834+0000 7f69963c4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6990111b40 con 0x7f6990108820 2024-03-18T22:52:05.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.834+0000 7f698f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f699010b190 0x7f699019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.834+0000 7f698f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f699010b190 0x7f699019e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49020/0 (socket says 172.21.15.40:49020) 2024-03-18T22:52:05.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.834+0000 7f698f7fe640 1 -- 172.21.15.40:0/1039568709 learned_addr learned my addr 172.21.15.40:0/1039568709 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:05.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.835+0000 7f6994961640 1 --2- 172.21.15.40:0/1039568709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699010db40 0x7f69901a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.835+0000 7f698f7fe640 1 -- 172.21.15.40:0/1039568709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6990108820 msgr2=0x7f699019e0e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T22:52:05.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.835+0000 7f698f7fe640 1 --2- 172.21.15.40:0/1039568709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6990108820 0x7f699019e0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.835+0000 7f698f7fe640 1 -- 172.21.15.40:0/1039568709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699010db40 msgr2=0x7f69901a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.835+0000 7f698f7fe640 1 --2- 172.21.15.40:0/1039568709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699010db40 0x7f69901a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.835+0000 7f698f7fe640 1 -- 172.21.15.40:0/1039568709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69901a5d00 con 0x7f699010b190 2024-03-18T22:52:05.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.839+0000 7f698f7fe640 1 --2- 172.21.15.40:0/1039568709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f699010b190 0x7f699019e620 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f698800ba50 tx=0x7f698800bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.841+0000 7f698d7fa640 1 -- 172.21.15.40:0/1039568709 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988019070 con 0x7f699010b190 2024-03-18T22:52:05.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.841+0000 7f69963c4640 1 -- 172.21.15.40:0/1039568709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69901a5ff0 con 0x7f699010b190 2024-03-18T22:52:05.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.841+0000 7f69963c4640 1 -- 172.21.15.40:0/1039568709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69901a64b0 con 0x7f699010b190 2024-03-18T22:52:05.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.841+0000 7f698d7fa640 1 -- 172.21.15.40:0/1039568709 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f698800cbe0 con 0x7f699010b190 2024-03-18T22:52:05.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.841+0000 7f698d7fa640 1 -- 172.21.15.40:0/1039568709 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69880075d0 con 0x7f699010b190 2024-03-18T22:52:05.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.847+0000 7f6998fab740 1 Processor -- start 2024-03-18T22:52:05.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.848+0000 7f6998fab740 1 -- start start 2024-03-18T22:52:05.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.848+0000 7f698d7fa640 1 -- 172.21.15.40:0/1039568709 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f698801c070 con 0x7f699010b190 2024-03-18T22:52:05.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.849+0000 7f6998fab740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69ad2f0 0x5633c69ad710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.849+0000 7f6998fab740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.849+0000 7f6998fab740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69b2e50 0x5633c69b52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.849+0000 7f6998fab740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5633c69114b0 con 0x5633c69adce0 2024-03-18T22:52:05.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.849+0000 7f6998fab740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5633c68b9d70 con 0x5633c69ad2f0 2024-03-18T22:52:05.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.849+0000 7f6998fab740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5633c690d550 con 0x5633c69b2e50 2024-03-18T22:52:05.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.849+0000 7f698ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.849+0000 7f698ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69b27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59342/0 (socket says 172.21.15.40:59342) 2024-03-18T22:52:05.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.849+0000 7f698f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69b2e50 0x5633c69b52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.849+0000 7f698ffff640 1 -- 172.21.15.40:0/2017832117 learned_addr learned my addr 172.21.15.40:0/2017832117 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:05.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.849+0000 7f6994961640 1 --2- 172.21.15.40:0/2017832117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69ad2f0 0x5633c69ad710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.850+0000 7f698d7fa640 1 --2- 172.21.15.40:0/1039568709 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f69780a2a00 0x7f69780a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.850+0000 7f698ffff640 1 -- 172.21.15.40:0/2017832117 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69b2e50 msgr2=0x5633c69b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.850+0000 7f698ffff640 1 --2- 172.21.15.40:0/2017832117 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69b2e50 0x5633c69b52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.850+0000 7f698ffff640 1 -- 172.21.15.40:0/2017832117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69ad2f0 msgr2=0x5633c69ad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.850+0000 7f698ffff640 1 --2- 172.21.15.40:0/2017832117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69ad2f0 0x5633c69ad710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.850+0000 7f698ffff640 1 -- 172.21.15.40:0/2017832117 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633c68dc210 con 0x5633c69adce0 2024-03-18T22:52:05.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.850+0000 7f698d7fa640 1 -- 172.21.15.40:0/1039568709 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f69880ce2f0 con 0x7f699010b190 2024-03-18T22:52:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.851+0000 7f698ffff640 1 --2- 172.21.15.40:0/2017832117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69b27e0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f698403d270 tx=0x7f698401a5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a56a18f51fe36 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.852+0000 7f6994961640 1 --2- 172.21.15.40:0/1039568709 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f69780a2a00 0x7f69780a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.853+0000 7f6994961640 1 --2- 172.21.15.40:0/1039568709 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f69780a2a00 0x7f69780a4ec0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x5633c69ae540 tx=0x7f69800046c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.856+0000 7f69767fc640 1 -- 172.21.15.40:0/2017832117 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984052020 con 0x5633c69adce0 2024-03-18T22:52:05.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.856+0000 7f69767fc640 1 -- 172.21.15.40:0/2017832117 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6984004040 con 0x5633c69adce0 2024-03-18T22:52:05.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.856+0000 7f69767fc640 1 -- 172.21.15.40:0/2017832117 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984005480 con 0x5633c69adce0 2024-03-18T22:52:05.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.858+0000 7f6998fab740 1 -- 172.21.15.40:0/2017832117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 msgr2=0x5633c69b27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.858+0000 7f6998fab740 1 --2- 172.21.15.40:0/2017832117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69b27e0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f698403d270 tx=0x7f698401a5e0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.858+0000 7f6998fab740 1 -- 172.21.15.40:0/2017832117 shutdown_connections 2024-03-18T22:52:05.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.858+0000 7f6998fab740 1 --2- 172.21.15.40:0/2017832117 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69b2e50 0x5633c69b52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.858+0000 7f6998fab740 1 --2- 172.21.15.40:0/2017832117 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69b27e0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.858+0000 7f6998fab740 1 --2- 172.21.15.40:0/2017832117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69ad2f0 0x5633c69ad710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.858+0000 7f6998fab740 1 -- 172.21.15.40:0/2017832117 >> 172.21.15.40:0/2017832117 conn(0x5633c69a8860 msgr2=0x5633c69a8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.860+0000 7f6998fab740 1 -- 172.21.15.40:0/2017832117 shutdown_connections 2024-03-18T22:52:05.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.860+0000 7f6998fab740 1 -- 172.21.15.40:0/2017832117 wait complete. 2024-03-18T22:52:05.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.861+0000 7f6998fab740 1 -- start start 2024-03-18T22:52:05.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.861+0000 7f6998fab740 1 Processor -- start 2024-03-18T22:52:05.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.862+0000 7f6998fab740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69ad2f0 0x5633c69b77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.862+0000 7f6998fab740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69b7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.862+0000 7f6998fab740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69b2e50 0x5633c69bed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.862+0000 7f6998fab740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5633c69b5da0 con 0x5633c69adce0 2024-03-18T22:52:05.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.862+0000 7f6998fab740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5633c68cd310 con 0x5633c69b2e50 2024-03-18T22:52:05.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.862+0000 7f6998fab740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5633c69b5f20 con 0x5633c69ad2f0 2024-03-18T22:52:05.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.862+0000 7f698f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69b7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.862+0000 7f698f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69b7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59346/0 (socket says 172.21.15.40:59346) 2024-03-18T22:52:05.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.862+0000 7f698f7fe640 1 -- 172.21.15.40:0/1009541826 learned_addr learned my addr 172.21.15.40:0/1009541826 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:05.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.862+0000 7f6998fab740 10 client.0 _ll_register_callbacks cb 0x5633c68c12d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:05.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.863+0000 7f6994961640 1 --2- 172.21.15.40:0/1009541826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69b2e50 0x5633c69bed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.863+0000 7f698f7fe640 1 -- 172.21.15.40:0/1009541826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69ad2f0 msgr2=0x5633c69b77f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:52:05.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.863+0000 7f698f7fe640 1 --2- 172.21.15.40:0/1009541826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69ad2f0 0x5633c69b77f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.863+0000 7f698f7fe640 1 -- 172.21.15.40:0/1009541826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69b2e50 msgr2=0x5633c69bed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.863+0000 7f698f7fe640 1 --2- 172.21.15.40:0/1009541826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69b2e50 0x5633c69bed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.863+0000 7f698f7fe640 1 -- 172.21.15.40:0/1009541826 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633c66253f0 con 0x5633c69adce0 2024-03-18T22:52:05.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.863+0000 7f698f7fe640 1 --2- 172.21.15.40:0/1009541826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69b7d30 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f698809a7d0 tx=0x7f698809a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c17f498e1535f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.864+0000 7f696f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:05.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.864+0000 7f696f7fe640 1 -- 172.21.15.40:0/1009541826 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69880cbe60 con 0x5633c69adce0 2024-03-18T22:52:05.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.864+0000 7f696f7fe640 1 -- 172.21.15.40:0/1009541826 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f69880a2b50 con 0x5633c69adce0 2024-03-18T22:52:05.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.864+0000 7f6998fab740 1 -- 172.21.15.40:0/1009541826 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6984004040 con 0x5633c69adce0 2024-03-18T22:52:05.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.865+0000 7f696f7fe640 1 -- 172.21.15.40:0/1009541826 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f69880ce0d0 con 0x5633c69adce0 2024-03-18T22:52:05.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.865+0000 7f6998fab740 10 client.17991 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:05.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.865+0000 7f6998fab740 1 -- 172.21.15.40:0/1009541826 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5633c66ef7d0 con 0x5633c69adce0 2024-03-18T22:52:05.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.866+0000 7f696f7fe640 1 -- 172.21.15.40:0/1009541826 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69880d1e40 con 0x5633c69adce0 2024-03-18T22:52:05.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.866+0000 7f696f7fe640 1 -- 172.21.15.40:0/1009541826 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f6988096590 con 0x5633c69adce0 2024-03-18T22:52:05.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.866+0000 7f6998fab740 10 client.17991 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:05.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.866+0000 7f6998fab740 10 client.17991 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:05.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.866+0000 7f6998fab740 20 client.17991 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:05.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.866+0000 7f6998fab740 1 --2- 172.21.15.40:0/1009541826 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5633c69c3610 0x5633c69c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.866+0000 7f6998fab740 4 client.17991 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:05.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.866+0000 7f6998fab740 1 -- 172.21.15.40:0/1009541826 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6984004040 con 0x5633c69c3610 2024-03-18T22:52:05.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.868+0000 7f698ffff640 1 --2- 172.21.15.40:0/1009541826 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5633c69c3610 0x5633c69c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.884+0000 7f698ffff640 1 --2- 172.21.15.40:0/1009541826 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5633c69c3610 0x5633c69c5ab0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7a0bea7d60ae4b4 server_cookie=7da4210e4e5dc2d0 in_seq=0 out_seq=0 2024-03-18T22:52:05.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.884+0000 7f696f7fe640 10 client.17991 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:05.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.887+0000 7f696f7fe640 1 -- 172.21.15.40:0/1009541826 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f6984004040 con 0x5633c69c3610 2024-03-18T22:52:05.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.887+0000 7f696f7fe640 10 client.17991 handle_command_reply: tid=0 2024-03-18T22:52:05.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.888+0000 7f6998fab740 1 client.17991 shutdown 2024-03-18T22:52:05.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.892+0000 7f6998fab740 1 -- 172.21.15.40:0/1009541826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 msgr2=0x5633c69b7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.892+0000 7f6998fab740 1 --2- 172.21.15.40:0/1009541826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69b7d30 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f698809a7d0 tx=0x7f698809a610 comp rx=0 tx=0).stop 2024-03-18T22:52:05.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.892+0000 7f6998fab740 1 -- 172.21.15.40:0/1009541826 shutdown_connections 2024-03-18T22:52:05.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.892+0000 7f6998fab740 1 --2- 172.21.15.40:0/1009541826 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5633c69c3610 0x5633c69c5ab0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.892+0000 7f6998fab740 1 --2- 172.21.15.40:0/1009541826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69b2e50 0x5633c69bed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.892+0000 7f6998fab740 1 --2- 172.21.15.40:0/1009541826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69b7d30 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.892+0000 7f6998fab740 1 --2- 172.21.15.40:0/1009541826 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69ad2f0 0x5633c69b77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.892+0000 7f6998fab740 1 -- 172.21.15.40:0/1009541826 >> 172.21.15.40:0/1009541826 conn(0x5633c69a8860 msgr2=0x5633c69a7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.893+0000 7f696f7fe640 0 client.17991 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:05.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.893+0000 7f6998fab740 1 -- 172.21.15.40:0/1009541826 shutdown_connections 2024-03-18T22:52:05.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.894+0000 7f6998fab740 1 -- 172.21.15.40:0/1009541826 wait complete. 2024-03-18T22:52:05.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.894+0000 7f6998fab740 20 client.17991 trim_cache size 0 max 16384 2024-03-18T22:52:05.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.938+0000 7f6998fab740 1 Processor -- start 2024-03-18T22:52:05.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.939+0000 7f6998fab740 1 -- start start 2024-03-18T22:52:05.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.939+0000 7f6998fab740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69ad2f0 0x5633c69b7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.939+0000 7f6998fab740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69bed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.940+0000 7f698ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69ad2f0 0x5633c69b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.940+0000 7f698ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69ad2f0 0x5633c69b7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49054/0 (socket says 172.21.15.40:49054) 2024-03-18T22:52:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.940+0000 7f698f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69bed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.940+0000 7f698f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69bed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59350/0 (socket says 172.21.15.40:59350) 2024-03-18T22:52:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.940+0000 7f698ffff640 1 -- 172.21.15.40:0/197042616 learned_addr learned my addr 172.21.15.40:0/197042616 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.940+0000 7f6998fab740 1 --2- 172.21.15.40:0/197042616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69b2e50 0x5633c69a7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.940+0000 7f6998fab740 1 -- 172.21.15.40:0/197042616 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f697c00fe10 con 0x5633c69adce0 2024-03-18T22:52:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.940+0000 7f6998fab740 1 -- 172.21.15.40:0/197042616 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f697c0119b0 con 0x5633c69ad2f0 2024-03-18T22:52:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.940+0000 7f6994961640 1 --2- 172.21.15.40:0/197042616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69b2e50 0x5633c69a7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.940+0000 7f6998fab740 1 -- 172.21.15.40:0/197042616 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f697c0104e0 con 0x5633c69b2e50 2024-03-18T22:52:05.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.941+0000 7f698ffff640 1 -- 172.21.15.40:0/197042616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69b2e50 msgr2=0x5633c69a7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.941+0000 7f698ffff640 1 --2- 172.21.15.40:0/197042616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69b2e50 0x5633c69a7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.941+0000 7f698ffff640 1 -- 172.21.15.40:0/197042616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 msgr2=0x5633c69bed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.941+0000 7f698ffff640 1 --2- 172.21.15.40:0/197042616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69bed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.941+0000 7f698ffff640 1 -- 172.21.15.40:0/197042616 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633c69b2490 con 0x5633c69ad2f0 2024-03-18T22:52:05.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.942+0000 7f698ffff640 1 --2- 172.21.15.40:0/197042616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69ad2f0 0x5633c69b7210 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6984005ab0 tx=0x7f698403d180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e230bdde545b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.943+0000 7f6975ffb640 1 -- 172.21.15.40:0/197042616 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984063070 con 0x5633c69ad2f0 2024-03-18T22:52:05.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.943+0000 7f6975ffb640 1 -- 172.21.15.40:0/197042616 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6984005850 con 0x5633c69ad2f0 2024-03-18T22:52:05.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.943+0000 7f6975ffb640 1 -- 172.21.15.40:0/197042616 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984052040 con 0x5633c69ad2f0 2024-03-18T22:52:05.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.944+0000 7f6998fab740 1 -- 172.21.15.40:0/197042616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69ad2f0 msgr2=0x5633c69b7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.944+0000 7f6998fab740 1 --2- 172.21.15.40:0/197042616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69ad2f0 0x5633c69b7210 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6984005ab0 tx=0x7f698403d180 comp rx=0 tx=0).stop 2024-03-18T22:52:05.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.944+0000 7f6998fab740 1 -- 172.21.15.40:0/197042616 shutdown_connections 2024-03-18T22:52:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.944+0000 7f6998fab740 1 --2- 172.21.15.40:0/197042616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69b2e50 0x5633c69a7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.945+0000 7f6998fab740 1 --2- 172.21.15.40:0/197042616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69adce0 0x5633c69bed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.945+0000 7f6998fab740 1 --2- 172.21.15.40:0/197042616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69ad2f0 0x5633c69b7210 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.945+0000 7f6998fab740 1 -- 172.21.15.40:0/197042616 >> 172.21.15.40:0/197042616 conn(0x5633c69a8860 msgr2=0x5633c69b6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.945+0000 7f6998fab740 1 -- 172.21.15.40:0/197042616 shutdown_connections 2024-03-18T22:52:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.945+0000 7f6998fab740 1 -- 172.21.15.40:0/197042616 wait complete. 2024-03-18T22:52:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.946+0000 7f6998fab740 1 -- start start 2024-03-18T22:52:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.946+0000 7f6998fab740 1 Processor -- start 2024-03-18T22:52:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.946+0000 7f6998fab740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69adce0 0x5633c69bbd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.947+0000 7f6998fab740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69b2e50 0x5633c69bc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.947+0000 7f698f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69b2e50 0x5633c69bc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.947+0000 7f698ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69adce0 0x5633c69bbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.947+0000 7f698ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69adce0 0x5633c69bbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:52406/0 (socket says 172.21.15.40:52406) 2024-03-18T22:52:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.947+0000 7f698ffff640 1 -- 172.21.15.40:0/1798778616 learned_addr learned my addr 172.21.15.40:0/1798778616 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.948+0000 7f6998fab740 1 --2- 172.21.15.40:0/1798778616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69bd2c0 0x5633c69bd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.948+0000 7f6998fab740 1 -- 172.21.15.40:0/1798778616 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f697c0112e0 con 0x5633c69b2e50 2024-03-18T22:52:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.948+0000 7f6998fab740 1 -- 172.21.15.40:0/1798778616 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f697c010bb0 con 0x5633c69bd2c0 2024-03-18T22:52:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.948+0000 7f6994961640 1 --2- 172.21.15.40:0/1798778616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69bd2c0 0x5633c69bd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.948+0000 7f6998fab740 1 -- 172.21.15.40:0/1798778616 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f697c00ddf0 con 0x5633c69adce0 2024-03-18T22:52:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.948+0000 7f6998fab740 10 client.0 _ll_register_callbacks cb 0x7f6988009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.949+0000 7f698ffff640 1 -- 172.21.15.40:0/1798778616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69bd2c0 msgr2=0x5633c69bd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.949+0000 7f698ffff640 1 --2- 172.21.15.40:0/1798778616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69bd2c0 0x5633c69bd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.949+0000 7f698ffff640 1 -- 172.21.15.40:0/1798778616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69b2e50 msgr2=0x5633c69bc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.949+0000 7f698ffff640 1 --2- 172.21.15.40:0/1798778616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69b2e50 0x5633c69bc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.949+0000 7f698ffff640 1 -- 172.21.15.40:0/1798778616 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633c69bdc80 con 0x5633c69adce0 2024-03-18T22:52:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.949+0000 7f698ffff640 1 --2- 172.21.15.40:0/1798778616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69adce0 0x5633c69bbd70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6984006010 tx=0x7f698401a810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcb93456d3557b84 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.949+0000 7f696f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:52:05.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.950+0000 7f696f7fe640 1 -- 172.21.15.40:0/1798778616 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984063070 con 0x5633c69adce0 2024-03-18T22:52:05.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.950+0000 7f6994961640 1 --2- 172.21.15.40:0/1798778616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69bd2c0 0x5633c69bd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:52:05.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.951+0000 7f6998fab740 1 -- 172.21.15.40:0/1798778616 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6984005850 con 0x5633c69adce0 2024-03-18T22:52:05.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.951+0000 7f696f7fe640 1 -- 172.21.15.40:0/1798778616 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6984004940 con 0x5633c69adce0 2024-03-18T22:52:05.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.951+0000 7f696f7fe640 1 -- 172.21.15.40:0/1798778616 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f698405e070 con 0x5633c69adce0 2024-03-18T22:52:05.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.956+0000 7f6998fab740 10 client.27787 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:05.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.956+0000 7f6998fab740 1 -- 172.21.15.40:0/1798778616 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5633c69bef60 con 0x5633c69adce0 2024-03-18T22:52:05.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.959+0000 7f696f7fe640 1 -- 172.21.15.40:0/1798778616 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f698403dcc0 con 0x5633c69adce0 2024-03-18T22:52:05.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.959+0000 7f696f7fe640 1 -- 172.21.15.40:0/1798778616 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7f698404e500 con 0x5633c69adce0 2024-03-18T22:52:05.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.959+0000 7f6998fab740 10 client.27787 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:05.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.960+0000 7f6998fab740 10 client.27787 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.960+0000 7f6998fab740 20 client.27787 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.960+0000 7f6998fab740 1 --2- 172.21.15.40:0/1798778616 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5633c6a29b20 0x5633c6a2bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.960+0000 7f6998fab740 4 client.27787 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:52:05.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.960+0000 7f6998fab740 1 -- 172.21.15.40:0/1798778616 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f6984005850 con 0x5633c6a29b20 2024-03-18T22:52:05.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.961+0000 7f698f7fe640 1 --2- 172.21.15.40:0/1798778616 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5633c6a29b20 0x5633c6a2bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:05.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.965+0000 7f698f7fe640 1 --2- 172.21.15.40:0/1798778616 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5633c6a29b20 0x5633c6a2bf10 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75f6d845ea08177b server_cookie=3c2cf026e3e2214c in_seq=0 out_seq=0 2024-03-18T22:52:05.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.967+0000 7f696f7fe640 10 client.27787 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:05.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.967+0000 7f696f7fe640 1 -- 172.21.15.40:0/1798778616 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f6984005850 con 0x5633c6a29b20 2024-03-18T22:52:05.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.967+0000 7f696f7fe640 10 client.27787 handle_command_reply: tid=0 2024-03-18T22:52:05.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.967+0000 7f6998fab740 1 client.27787 shutdown 2024-03-18T22:52:05.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.970+0000 7f6998fab740 1 -- 172.21.15.40:0/1798778616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69adce0 msgr2=0x5633c69bbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.971+0000 7f6998fab740 1 --2- 172.21.15.40:0/1798778616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69adce0 0x5633c69bbd70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6984006010 tx=0x7f698401a810 comp rx=0 tx=0).stop 2024-03-18T22:52:05.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.974+0000 7f6998fab740 1 -- 172.21.15.40:0/1798778616 shutdown_connections 2024-03-18T22:52:05.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.974+0000 7f6998fab740 1 --2- 172.21.15.40:0/1798778616 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5633c6a29b20 0x5633c6a2bf10 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.974+0000 7f6998fab740 1 --2- 172.21.15.40:0/1798778616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5633c69bd2c0 0x5633c69bd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.974+0000 7f6998fab740 1 --2- 172.21.15.40:0/1798778616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5633c69b2e50 0x5633c69bc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.974+0000 7f696f7fe640 0 client.27787 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:05.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.974+0000 7f6998fab740 1 --2- 172.21.15.40:0/1798778616 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5633c69adce0 0x5633c69bbd70 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.974+0000 7f6998fab740 1 -- 172.21.15.40:0/1798778616 >> 172.21.15.40:0/1798778616 conn(0x5633c69a8860 msgr2=0x5633c69abac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.975+0000 7f6998fab740 1 -- 172.21.15.40:0/1798778616 shutdown_connections 2024-03-18T22:52:05.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.975+0000 7f6998fab740 1 -- 172.21.15.40:0/1798778616 wait complete. 2024-03-18T22:52:05.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.977+0000 7f6998fab740 20 client.27787 trim_cache size 0 max 16384 2024-03-18T22:52:05.983 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:52:05.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.994+0000 7f69963c4640 1 -- 172.21.15.40:0/1039568709 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f69780a2a00 msgr2=0x7f69780a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.994+0000 7f69963c4640 1 --2- 172.21.15.40:0/1039568709 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f69780a2a00 0x7f69780a4ec0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x5633c69ae540 tx=0x7f69800046c0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.994+0000 7f69963c4640 1 -- 172.21.15.40:0/1039568709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f699010b190 msgr2=0x7f699019e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:05.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.994+0000 7f69963c4640 1 --2- 172.21.15.40:0/1039568709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f699010b190 0x7f699019e620 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f698800ba50 tx=0x7f698800bf10 comp rx=0 tx=0).stop 2024-03-18T22:52:05.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.995+0000 7f69963c4640 1 -- 172.21.15.40:0/1039568709 shutdown_connections 2024-03-18T22:52:05.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.996+0000 7f69963c4640 1 --2- 172.21.15.40:0/1039568709 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f69780a2a00 0x7f69780a4ec0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.996+0000 7f69963c4640 1 --2- 172.21.15.40:0/1039568709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f699010db40 0x7f69901a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.996+0000 7f69963c4640 1 --2- 172.21.15.40:0/1039568709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f699010b190 0x7f699019e620 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.996+0000 7f69963c4640 1 --2- 172.21.15.40:0/1039568709 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6990108820 0x7f699019e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:05.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.996+0000 7f69963c4640 1 -- 172.21.15.40:0/1039568709 >> 172.21.15.40:0/1039568709 conn(0x7f699006e8a0 msgr2=0x7f6990108fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:05.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.996+0000 7f69963c4640 1 -- 172.21.15.40:0/1039568709 shutdown_connections 2024-03-18T22:52:05.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:05.997+0000 7f69963c4640 1 -- 172.21.15.40:0/1039568709 wait complete. 2024-03-18T22:52:06.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:06 smithi040 ceph-mon[62643]: pgmap v634: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 1.41k op/s 2024-03-18T22:52:06.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:06 smithi113 ceph-mon[66775]: pgmap v634: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 1.41k op/s 2024-03-18T22:52:06.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:06 smithi103 ceph-mon[66377]: pgmap v634: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 1.41k op/s 2024-03-18T22:52:07.029 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:52:07.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:52:07.029 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:52:07.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.258+0000 7ff78d362640 1 -- 172.21.15.40:0/3935783747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff78810ddb0 msgr2=0x7ff788110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.258+0000 7ff78d362640 1 --2- 172.21.15.40:0/3935783747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff78810ddb0 0x7ff788110270 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7ff7700074e0 tx=0x7ff77002c8f0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.258+0000 7ff78d362640 1 -- 172.21.15.40:0/3935783747 shutdown_connections 2024-03-18T22:52:07.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.258+0000 7ff78d362640 1 --2- 172.21.15.40:0/3935783747 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff78810ddb0 0x7ff788110270 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.258+0000 7ff78d362640 1 --2- 172.21.15.40:0/3935783747 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff78810b330 0x7ff78810d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.258+0000 7ff78d362640 1 --2- 172.21.15.40:0/3935783747 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff788108a00 0x7ff78810adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.258+0000 7ff78d362640 1 -- 172.21.15.40:0/3935783747 >> 172.21.15.40:0/3935783747 conn(0x7ff78806e8f0 msgr2=0x7ff788070d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:07.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.259+0000 7ff78d362640 1 -- 172.21.15.40:0/3935783747 shutdown_connections 2024-03-18T22:52:07.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.259+0000 7ff78d362640 1 -- 172.21.15.40:0/3935783747 wait complete. 2024-03-18T22:52:07.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.260+0000 7ff78d362640 1 Processor -- start 2024-03-18T22:52:07.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.260+0000 7ff78d362640 1 -- start start 2024-03-18T22:52:07.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.260+0000 7ff78d362640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff788108a00 0x7ff78819f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.260+0000 7ff78d362640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff78810b330 0x7ff78819db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.260+0000 7ff78d362640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7881a3c10 0x7ff78819e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.260+0000 7ff78d362640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff788113910 con 0x7ff78810b330 2024-03-18T22:52:07.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.261+0000 7ff78d362640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff788113790 con 0x7ff7881a3c10 2024-03-18T22:52:07.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.261+0000 7ff78d362640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff788113a90 con 0x7ff788108a00 2024-03-18T22:52:07.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.261+0000 7ff787fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff788108a00 0x7ff78819f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.261+0000 7ff787fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff788108a00 0x7ff78819f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:52422/0 (socket says 172.21.15.40:52422) 2024-03-18T22:52:07.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.261+0000 7ff787fff640 1 -- 172.21.15.40:0/2618832372 learned_addr learned my addr 172.21.15.40:0/2618832372 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:07.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.261+0000 7ff78cb61640 1 --2- 172.21.15.40:0/2618832372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7881a3c10 0x7ff78819e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.262+0000 7ff787fff640 1 -- 172.21.15.40:0/2618832372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7881a3c10 msgr2=0x7ff78819e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.262+0000 7ff787fff640 1 --2- 172.21.15.40:0/2618832372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7881a3c10 0x7ff78819e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.262+0000 7ff787fff640 1 -- 172.21.15.40:0/2618832372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff78810b330 msgr2=0x7ff78819db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.262+0000 7ff787fff640 1 --2- 172.21.15.40:0/2618832372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff78810b330 0x7ff78819db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.262+0000 7ff787fff640 1 -- 172.21.15.40:0/2618832372 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7881a4000 con 0x7ff788108a00 2024-03-18T22:52:07.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.263+0000 7ff787fff640 1 --2- 172.21.15.40:0/2618832372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff788108a00 0x7ff78819f500 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff77800b520 tx=0x7ff77800b9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:07.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.264+0000 7ff7857fa640 1 -- 172.21.15.40:0/2618832372 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff778014070 con 0x7ff788108a00 2024-03-18T22:52:07.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.264+0000 7ff78d362640 1 -- 172.21.15.40:0/2618832372 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7881a42f0 con 0x7ff788108a00 2024-03-18T22:52:07.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.264+0000 7ff78d362640 1 -- 172.21.15.40:0/2618832372 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7881a4840 con 0x7ff788108a00 2024-03-18T22:52:07.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.265+0000 7ff7857fa640 1 -- 172.21.15.40:0/2618832372 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff778010d40 con 0x7ff788108a00 2024-03-18T22:52:07.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.266+0000 7ff7857fa640 1 -- 172.21.15.40:0/2618832372 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77800d380 con 0x7ff788108a00 2024-03-18T22:52:07.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.268+0000 7ff7911aa740 1 Processor -- start 2024-03-18T22:52:07.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.271+0000 7ff7911aa740 1 -- start start 2024-03-18T22:52:07.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.271+0000 7ff7911aa740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 0x563dd77af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.271+0000 7ff7911aa740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77b4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.271+0000 7ff7911aa740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77b50e0 0x563dd77b7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.271+0000 7ff7911aa740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563dd7714160 con 0x563dd77b50e0 2024-03-18T22:52:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.271+0000 7ff7911aa740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563dd76bce10 con 0x563dd77af580 2024-03-18T22:52:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.271+0000 7ff7911aa740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563dd7710200 con 0x563dd77aff70 2024-03-18T22:52:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.272+0000 7ff78cb61640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 0x563dd77af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.272+0000 7ff78cb61640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 0x563dd77af9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49088/0 (socket says 172.21.15.40:49088) 2024-03-18T22:52:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.272+0000 7ff78cb61640 1 -- 172.21.15.40:0/1135582692 learned_addr learned my addr 172.21.15.40:0/1135582692 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:07.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.272+0000 7ff7877fe640 1 --2- 172.21.15.40:0/1135582692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77b50e0 0x563dd77b7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.273+0000 7ff787fff640 1 --2- 172.21.15.40:0/1135582692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77b4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.274+0000 7ff78cb61640 1 -- 172.21.15.40:0/1135582692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 msgr2=0x563dd77b4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.274+0000 7ff78cb61640 1 --2- 172.21.15.40:0/1135582692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77b4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.274+0000 7ff78cb61640 1 -- 172.21.15.40:0/1135582692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77b50e0 msgr2=0x563dd77b7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.274+0000 7ff78cb61640 1 --2- 172.21.15.40:0/1135582692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77b50e0 0x563dd77b7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.274+0000 7ff78cb61640 1 -- 172.21.15.40:0/1135582692 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dd74f07e0 con 0x563dd77af580 2024-03-18T22:52:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.274+0000 7ff7877fe640 1 --2- 172.21.15.40:0/1135582692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77b50e0 0x563dd77b7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:52:07.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.274+0000 7ff7857fa640 1 -- 172.21.15.40:0/2618832372 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff778017070 con 0x7ff788108a00 2024-03-18T22:52:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.275+0000 7ff7857fa640 1 --2- 172.21.15.40:0/2618832372 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff7740a2a10 0x7ff7740a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.275+0000 7ff7857fa640 1 -- 172.21.15.40:0/2618832372 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff7780ce5e0 con 0x7ff788108a00 2024-03-18T22:52:07.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.276+0000 7ff78cb61640 1 --2- 172.21.15.40:0/1135582692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 0x563dd77af9a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff770006010 tx=0x7ff770004820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc7d05b4f615eca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:07.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.280+0000 7ff77e7fc640 1 -- 172.21.15.40:0/1135582692 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77003f030 con 0x563dd77af580 2024-03-18T22:52:07.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.280+0000 7ff77e7fc640 1 -- 172.21.15.40:0/1135582692 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff770004040 con 0x563dd77af580 2024-03-18T22:52:07.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.280+0000 7ff77e7fc640 1 -- 172.21.15.40:0/1135582692 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff770036070 con 0x563dd77af580 2024-03-18T22:52:07.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.281+0000 7ff7911aa740 1 -- 172.21.15.40:0/1135582692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 msgr2=0x563dd77af9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.281+0000 7ff7911aa740 1 --2- 172.21.15.40:0/1135582692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 0x563dd77af9a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff770006010 tx=0x7ff770004820 comp rx=0 tx=0).stop 2024-03-18T22:52:07.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.281+0000 7ff787fff640 1 --2- 172.21.15.40:0/2618832372 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff7740a2a10 0x7ff7740a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.281+0000 7ff7911aa740 1 -- 172.21.15.40:0/1135582692 shutdown_connections 2024-03-18T22:52:07.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.281+0000 7ff7911aa740 1 --2- 172.21.15.40:0/1135582692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77b50e0 0x563dd77b7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.281+0000 7ff7911aa740 1 --2- 172.21.15.40:0/1135582692 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77b4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.281+0000 7ff7911aa740 1 --2- 172.21.15.40:0/1135582692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 0x563dd77af9a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.281+0000 7ff7911aa740 1 -- 172.21.15.40:0/1135582692 >> 172.21.15.40:0/1135582692 conn(0x563dd77aaaa0 msgr2=0x563dd77aaea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:07.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.281+0000 7ff7911aa740 1 -- 172.21.15.40:0/1135582692 shutdown_connections 2024-03-18T22:52:07.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.281+0000 7ff7911aa740 1 -- 172.21.15.40:0/1135582692 wait complete. 2024-03-18T22:52:07.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.281+0000 7ff7911aa740 1 -- start start 2024-03-18T22:52:07.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.282+0000 7ff7911aa740 1 Processor -- start 2024-03-18T22:52:07.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.282+0000 7ff7911aa740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77af580 0x563dd77b9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.282+0000 7ff7911aa740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77ba030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.282+0000 7ff7911aa740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77b50e0 0x563dd77c1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.282+0000 7ff7911aa740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563dd77b8030 con 0x563dd77af580 2024-03-18T22:52:07.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.282+0000 7ff7911aa740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563dd76d01a0 con 0x563dd77b50e0 2024-03-18T22:52:07.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.282+0000 7ff7911aa740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563dd77b81b0 con 0x563dd77aff70 2024-03-18T22:52:07.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.282+0000 7ff7911aa740 10 client.0 _ll_register_callbacks cb 0x563dd7427c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:07.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.283+0000 7ff7877fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77ba030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.283+0000 7ff7877fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77ba030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:52432/0 (socket says 172.21.15.40:52432) 2024-03-18T22:52:07.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.283+0000 7ff7877fe640 1 -- 172.21.15.40:0/2160536167 learned_addr learned my addr 172.21.15.40:0/2160536167 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:07.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.283+0000 7ff78cb61640 1 --2- 172.21.15.40:0/2160536167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77b50e0 0x563dd77c1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.283+0000 7ff7877fe640 1 -- 172.21.15.40:0/2160536167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77b50e0 msgr2=0x563dd77c1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:07.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.283+0000 7ff78cb61640 1 --2- 172.21.15.40:0/2160536167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77af580 0x563dd77b9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.283+0000 7ff7877fe640 1 --2- 172.21.15.40:0/2160536167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77b50e0 0x563dd77c1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.283+0000 7ff7877fe640 1 -- 172.21.15.40:0/2160536167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77af580 msgr2=0x563dd77b9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:07.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.283+0000 7ff7877fe640 1 --2- 172.21.15.40:0/2160536167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77af580 0x563dd77b9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.283+0000 7ff7877fe640 1 -- 172.21.15.40:0/2160536167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dd7550f70 con 0x563dd77aff70 2024-03-18T22:52:07.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.283+0000 7ff78cb61640 1 --2- 172.21.15.40:0/2160536167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77af580 0x563dd77b9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:52:07.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.285+0000 7ff7877fe640 1 --2- 172.21.15.40:0/2160536167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77ba030 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff78000e9c0 tx=0x7ff78000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9beaaab1a730d8ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:07.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.285+0000 7ff787fff640 1 --2- 172.21.15.40:0/2618832372 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff7740a2a10 0x7ff7740a4ed0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff778097020 tx=0x7ff7780047d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.292+0000 7ff7677fe640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:52:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.292+0000 7ff7677fe640 1 -- 172.21.15.40:0/2160536167 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff78000cc60 con 0x563dd77aff70 2024-03-18T22:52:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.292+0000 7ff7911aa740 1 -- 172.21.15.40:0/2160536167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff770004040 con 0x563dd77aff70 2024-03-18T22:52:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.293+0000 7ff7677fe640 1 -- 172.21.15.40:0/2160536167 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff78000ce00 con 0x563dd77aff70 2024-03-18T22:52:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.293+0000 7ff7677fe640 1 -- 172.21.15.40:0/2160536167 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff780010430 con 0x563dd77aff70 2024-03-18T22:52:07.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.293+0000 7ff7911aa740 10 client.27793 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.293+0000 7ff7911aa740 1 -- 172.21.15.40:0/2160536167 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563dd73e8170 con 0x563dd77aff70 2024-03-18T22:52:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.293+0000 7ff7677fe640 1 -- 172.21.15.40:0/2160536167 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff770004040 con 0x563dd77aff70 2024-03-18T22:52:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.294+0000 7ff7677fe640 1 -- 172.21.15.40:0/2160536167 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7ff78001a7b0 con 0x563dd77aff70 2024-03-18T22:52:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.295+0000 7ff7911aa740 10 client.27793 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.295+0000 7ff7911aa740 10 client.27793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.295+0000 7ff7911aa740 20 client.27793 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.295+0000 7ff7911aa740 1 --2- 172.21.15.40:0/2160536167 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563dd77c5db0 0x563dd77c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.295+0000 7ff7911aa740 4 client.27793 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.295+0000 7ff7911aa740 1 -- 172.21.15.40:0/2160536167 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563dd77216a0 con 0x563dd77c5db0 2024-03-18T22:52:07.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.298+0000 7ff78cb61640 1 --2- 172.21.15.40:0/2160536167 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563dd77c5db0 0x563dd77c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.300+0000 7ff78cb61640 1 --2- 172.21.15.40:0/2160536167 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563dd77c5db0 0x563dd77c8230 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e0def395172fa32 server_cookie=8d827f83aefb8335 in_seq=0 out_seq=0 2024-03-18T22:52:07.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.300+0000 7ff7677fe640 10 client.27793 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:07.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.301+0000 7ff7677fe640 1 -- 172.21.15.40:0/2160536167 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563dd77216a0 con 0x563dd77c5db0 2024-03-18T22:52:07.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.301+0000 7ff7677fe640 10 client.27793 handle_command_reply: tid=0 2024-03-18T22:52:07.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.302+0000 7ff7911aa740 1 client.27793 shutdown 2024-03-18T22:52:07.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.304+0000 7ff7911aa740 1 -- 172.21.15.40:0/2160536167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 msgr2=0x563dd77ba030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.304+0000 7ff7911aa740 1 --2- 172.21.15.40:0/2160536167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77ba030 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff78000e9c0 tx=0x7ff78000ee80 comp rx=0 tx=0).stop 2024-03-18T22:52:07.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.304+0000 7ff7911aa740 1 -- 172.21.15.40:0/2160536167 shutdown_connections 2024-03-18T22:52:07.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.304+0000 7ff7911aa740 1 --2- 172.21.15.40:0/2160536167 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563dd77c5db0 0x563dd77c8230 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.304+0000 7ff7911aa740 1 --2- 172.21.15.40:0/2160536167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77b50e0 0x563dd77c1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.304+0000 7ff7911aa740 1 --2- 172.21.15.40:0/2160536167 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77ba030 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.304+0000 7ff7911aa740 1 --2- 172.21.15.40:0/2160536167 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77af580 0x563dd77b9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.304+0000 7ff7911aa740 1 -- 172.21.15.40:0/2160536167 >> 172.21.15.40:0/2160536167 conn(0x563dd77aaaa0 msgr2=0x563dd77a9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:07.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.304+0000 7ff7677fe640 0 client.27793 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:07.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.304+0000 7ff7911aa740 1 -- 172.21.15.40:0/2160536167 shutdown_connections 2024-03-18T22:52:07.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.304+0000 7ff7911aa740 1 -- 172.21.15.40:0/2160536167 wait complete. 2024-03-18T22:52:07.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.304+0000 7ff7911aa740 20 client.27793 trim_cache size 0 max 16384 2024-03-18T22:52:07.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.353+0000 7ff7911aa740 1 Processor -- start 2024-03-18T22:52:07.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.353+0000 7ff7911aa740 1 -- start start 2024-03-18T22:52:07.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.353+0000 7ff7911aa740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 0x563dd77ae460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.353+0000 7ff7911aa740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77c0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.353+0000 7ff7911aa740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77b50e0 0x563dd77ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.353+0000 7ff7911aa740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff76800fe20 con 0x563dd77b50e0 2024-03-18T22:52:07.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.353+0000 7ff7911aa740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff7680119c0 con 0x563dd77af580 2024-03-18T22:52:07.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.353+0000 7ff7911aa740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff7680104f0 con 0x563dd77aff70 2024-03-18T22:52:07.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.354+0000 7ff78cb61640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77b50e0 0x563dd77ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.354+0000 7ff78cb61640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 0x563dd77ae460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.354+0000 7ff78cb61640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 0x563dd77ae460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:49106/0 (socket says 172.21.15.40:49106) 2024-03-18T22:52:07.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.354+0000 7ff78cb61640 1 -- 172.21.15.40:0/624530906 learned_addr learned my addr 172.21.15.40:0/624530906 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:07.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.354+0000 7ff7877fe640 1 --2- 172.21.15.40:0/624530906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77c0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.354+0000 7ff78cb61640 1 -- 172.21.15.40:0/624530906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 msgr2=0x563dd77c0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:07.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.354+0000 7ff78cb61640 1 --2- 172.21.15.40:0/624530906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77c0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.354+0000 7ff78cb61640 1 -- 172.21.15.40:0/624530906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77b50e0 msgr2=0x563dd77ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:07.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.354+0000 7ff78cb61640 1 --2- 172.21.15.40:0/624530906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77b50e0 0x563dd77ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.354+0000 7ff78cb61640 1 -- 172.21.15.40:0/624530906 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dd77ab860 con 0x563dd77af580 2024-03-18T22:52:07.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.355+0000 7ff78cb61640 1 --2- 172.21.15.40:0/624530906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 0x563dd77ae460 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff770000c00 tx=0x7ff77003d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc4f9d84cca1b7dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:07.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.356+0000 7ff77dffb640 1 -- 172.21.15.40:0/624530906 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77003f040 con 0x563dd77af580 2024-03-18T22:52:07.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.356+0000 7ff77dffb640 1 -- 172.21.15.40:0/624530906 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff770008ce0 con 0x563dd77af580 2024-03-18T22:52:07.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.356+0000 7ff77dffb640 1 -- 172.21.15.40:0/624530906 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff770036040 con 0x563dd77af580 2024-03-18T22:52:07.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.357+0000 7ff7911aa740 1 -- 172.21.15.40:0/624530906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 msgr2=0x563dd77ae460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.357+0000 7ff7911aa740 1 --2- 172.21.15.40:0/624530906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 0x563dd77ae460 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff770000c00 tx=0x7ff77003d040 comp rx=0 tx=0).stop 2024-03-18T22:52:07.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.358+0000 7ff7911aa740 1 -- 172.21.15.40:0/624530906 shutdown_connections 2024-03-18T22:52:07.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.358+0000 7ff7911aa740 1 --2- 172.21.15.40:0/624530906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77b50e0 0x563dd77ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.358+0000 7ff7911aa740 1 --2- 172.21.15.40:0/624530906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77c0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.358+0000 7ff7911aa740 1 --2- 172.21.15.40:0/624530906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 0x563dd77ae460 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.358+0000 7ff7911aa740 1 -- 172.21.15.40:0/624530906 >> 172.21.15.40:0/624530906 conn(0x563dd77aaaa0 msgr2=0x563dd77add50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:07.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.360+0000 7ff7911aa740 1 -- 172.21.15.40:0/624530906 shutdown_connections 2024-03-18T22:52:07.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.360+0000 7ff7911aa740 1 -- 172.21.15.40:0/624530906 wait complete. 2024-03-18T22:52:07.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.362+0000 7ff7911aa740 1 -- start start 2024-03-18T22:52:07.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.362+0000 7ff7911aa740 1 Processor -- start 2024-03-18T22:52:07.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.363+0000 7ff7911aa740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 0x563dd77c0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.363+0000 7ff7911aa740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77bf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.363+0000 7ff7911aa740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77b50e0 0x563dd77bf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.363+0000 7ff7911aa740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff7680112f0 con 0x563dd77b50e0 2024-03-18T22:52:07.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.363+0000 7ff7877fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77bf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.363+0000 7ff7877fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77bf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:52438/0 (socket says 172.21.15.40:52438) 2024-03-18T22:52:07.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.363+0000 7ff7877fe640 1 -- 172.21.15.40:0/1164290599 learned_addr learned my addr 172.21.15.40:0/1164290599 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:07.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.363+0000 7ff7911aa740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff768010bc0 con 0x563dd77af580 2024-03-18T22:52:07.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.363+0000 7ff7911aa740 1 -- 172.21.15.40:0/1164290599 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff76800de00 con 0x563dd77aff70 2024-03-18T22:52:07.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.364+0000 7ff7911aa740 10 client.0 _ll_register_callbacks cb 0x7ff780007520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:07.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.364+0000 7ff7877fe640 1 -- 172.21.15.40:0/1164290599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 msgr2=0x563dd77c0d60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:52:07.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.364+0000 7ff7877fe640 1 --2- 172.21.15.40:0/1164290599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 0x563dd77c0d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.364+0000 7ff7877fe640 1 -- 172.21.15.40:0/1164290599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77b50e0 msgr2=0x563dd77bf8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:52:07.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.364+0000 7ff7877fe640 1 --2- 172.21.15.40:0/1164290599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77b50e0 0x563dd77bf8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.364+0000 7ff7877fe640 1 -- 172.21.15.40:0/1164290599 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dd77bfef0 con 0x563dd77aff70 2024-03-18T22:52:07.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.364+0000 7ff7877fe640 1 --2- 172.21.15.40:0/1164290599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77bf370 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff78000f010 tx=0x7ff780007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1189b9e28c1881b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:07.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.364+0000 7ff7677fe640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:52:07.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.366+0000 7ff7677fe640 1 -- 172.21.15.40:0/1164290599 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff78001a7f0 con 0x563dd77aff70 2024-03-18T22:52:07.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.366+0000 7ff7911aa740 1 -- 172.21.15.40:0/1164290599 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff770008ce0 con 0x563dd77aff70 2024-03-18T22:52:07.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.367+0000 7ff7677fe640 1 -- 172.21.15.40:0/1164290599 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff780003970 con 0x563dd77aff70 2024-03-18T22:52:07.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.367+0000 7ff7677fe640 1 -- 172.21.15.40:0/1164290599 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff78001a990 con 0x563dd77aff70 2024-03-18T22:52:07.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.367+0000 7ff7911aa740 10 client.27796 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:07.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.367+0000 7ff7911aa740 1 -- 172.21.15.40:0/1164290599 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563dd782d050 con 0x563dd77aff70 2024-03-18T22:52:07.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.368+0000 7ff7677fe640 1 -- 172.21.15.40:0/1164290599 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff770008ce0 con 0x563dd77aff70 2024-03-18T22:52:07.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.368+0000 7ff7677fe640 1 -- 172.21.15.40:0/1164290599 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7ff7800044d0 con 0x563dd77aff70 2024-03-18T22:52:07.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.368+0000 7ff7911aa740 10 client.27796 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:07.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.368+0000 7ff7911aa740 10 client.27796 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:07.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.368+0000 7ff7911aa740 20 client.27796 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:07.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.369+0000 7ff7911aa740 1 --2- 172.21.15.40:0/1164290599 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563dd782d9e0 0x563dd782fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.369+0000 7ff7911aa740 4 client.27796 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:52:07.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.369+0000 7ff7911aa740 1 -- 172.21.15.40:0/1164290599 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563dd771b9f0 con 0x563dd782d9e0 2024-03-18T22:52:07.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.369+0000 7ff78cb61640 1 --2- 172.21.15.40:0/1164290599 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563dd782d9e0 0x563dd782fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.372+0000 7ff78cb61640 1 --2- 172.21.15.40:0/1164290599 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563dd782d9e0 0x563dd782fe00 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35004f6cfabc762c server_cookie=95babcb9d69020e5 in_seq=0 out_seq=0 2024-03-18T22:52:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.372+0000 7ff7677fe640 10 client.27796 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:07.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.374+0000 7ff7677fe640 1 -- 172.21.15.40:0/1164290599 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563dd771b9f0 con 0x563dd782d9e0 2024-03-18T22:52:07.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.374+0000 7ff7677fe640 10 client.27796 handle_command_reply: tid=0 2024-03-18T22:52:07.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.374+0000 7ff7911aa740 1 client.27796 shutdown 2024-03-18T22:52:07.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.378+0000 7ff7911aa740 1 -- 172.21.15.40:0/1164290599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 msgr2=0x563dd77bf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.378+0000 7ff7911aa740 1 --2- 172.21.15.40:0/1164290599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77bf370 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff78000f010 tx=0x7ff780007f60 comp rx=0 tx=0).stop 2024-03-18T22:52:07.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.378+0000 7ff7911aa740 1 -- 172.21.15.40:0/1164290599 shutdown_connections 2024-03-18T22:52:07.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.378+0000 7ff7911aa740 1 --2- 172.21.15.40:0/1164290599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563dd77b50e0 0x563dd77bf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.378+0000 7ff7911aa740 1 --2- 172.21.15.40:0/1164290599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563dd77aff70 0x563dd77bf370 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.378+0000 7ff7911aa740 1 --2- 172.21.15.40:0/1164290599 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563dd782d9e0 0x563dd782fe00 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.378+0000 7ff7911aa740 1 --2- 172.21.15.40:0/1164290599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563dd77af580 0x563dd77c0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.378+0000 7ff7911aa740 1 -- 172.21.15.40:0/1164290599 >> 172.21.15.40:0/1164290599 conn(0x563dd77aaaa0 msgr2=0x563dd77ae5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:07.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.378+0000 7ff7677fe640 0 client.27796 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:07.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.378+0000 7ff7911aa740 1 -- 172.21.15.40:0/1164290599 shutdown_connections 2024-03-18T22:52:07.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.379+0000 7ff7911aa740 1 -- 172.21.15.40:0/1164290599 wait complete. 2024-03-18T22:52:07.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.379+0000 7ff7911aa740 20 client.27796 trim_cache size 0 max 16384 2024-03-18T22:52:07.383 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:52:07.384 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:52:07.384 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "2a217dbe-47f6-4fd5-94c0-c49c0f03b1a9", 2024-03-18T22:52:07.384 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:52:07.384 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:52:07.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.385+0000 7ff78d362640 1 -- 172.21.15.40:0/2618832372 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff7740a2a10 msgr2=0x7ff7740a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.385+0000 7ff78d362640 1 --2- 172.21.15.40:0/2618832372 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff7740a2a10 0x7ff7740a4ed0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff778097020 tx=0x7ff7780047d0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.385+0000 7ff78d362640 1 -- 172.21.15.40:0/2618832372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff788108a00 msgr2=0x7ff78819f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.385+0000 7ff78d362640 1 --2- 172.21.15.40:0/2618832372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff788108a00 0x7ff78819f500 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff77800b520 tx=0x7ff77800b9e0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.387+0000 7ff78d362640 1 -- 172.21.15.40:0/2618832372 shutdown_connections 2024-03-18T22:52:07.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.387+0000 7ff78d362640 1 --2- 172.21.15.40:0/2618832372 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff7740a2a10 0x7ff7740a4ed0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.387+0000 7ff78d362640 1 --2- 172.21.15.40:0/2618832372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7881a3c10 0x7ff78819e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.387+0000 7ff78d362640 1 --2- 172.21.15.40:0/2618832372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff78810b330 0x7ff78819db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.387+0000 7ff78d362640 1 --2- 172.21.15.40:0/2618832372 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff788108a00 0x7ff78819f500 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.387+0000 7ff78d362640 1 -- 172.21.15.40:0/2618832372 >> 172.21.15.40:0/2618832372 conn(0x7ff78806e8f0 msgr2=0x7ff788070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:07.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.387+0000 7ff78d362640 1 -- 172.21.15.40:0/2618832372 shutdown_connections 2024-03-18T22:52:07.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.387+0000 7ff78d362640 1 -- 172.21.15.40:0/2618832372 wait complete. 2024-03-18T22:52:07.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:52:07.419 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:52:07.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.578+0000 7fcff1e0c640 1 -- 172.21.15.40:0/474651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfec100720 msgr2=0x7fcfec100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.578+0000 7fcff1e0c640 1 --2- 172.21.15.40:0/474651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfec100720 0x7fcfec100b40 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009900 tx=0x7fcfdc02bc30 comp rx=0 tx=0).stop 2024-03-18T22:52:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.578+0000 7fcff1e0c640 1 -- 172.21.15.40:0/474651 shutdown_connections 2024-03-18T22:52:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.578+0000 7fcff1e0c640 1 --2- 172.21.15.40:0/474651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfec1022d0 0x7fcfec108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.578+0000 7fcff1e0c640 1 --2- 172.21.15.40:0/474651 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfec101910 0x7fcfec101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.578+0000 7fcff1e0c640 1 --2- 172.21.15.40:0/474651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfec100720 0x7fcfec100b40 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.578+0000 7fcff1e0c640 1 -- 172.21.15.40:0/474651 >> 172.21.15.40:0/474651 conn(0x7fcfec0fbcb0 msgr2=0x7fcfec0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.578+0000 7fcff1e0c640 1 -- 172.21.15.40:0/474651 shutdown_connections 2024-03-18T22:52:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.578+0000 7fcff1e0c640 1 -- 172.21.15.40:0/474651 wait complete. 2024-03-18T22:52:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.578+0000 7fcff1e0c640 1 Processor -- start 2024-03-18T22:52:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcff1e0c640 1 -- start start 2024-03-18T22:52:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcff1e0c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfec100720 0x7fcfec1969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcff1e0c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfec101910 0x7fcfec196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcff1e0c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfec1022d0 0x7fcfec19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcff1e0c640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec10c290 con 0x7fcfec101910 2024-03-18T22:52:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcff1e0c640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec10c110 con 0x7fcfec100720 2024-03-18T22:52:07.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcff1e0c640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec10c410 con 0x7fcfec1022d0 2024-03-18T22:52:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcfeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfec101910 0x7fcfec196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcfeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfec101910 0x7fcfec196f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59436/0 (socket says 172.21.15.40:59436) 2024-03-18T22:52:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfec100720 0x7fcfec1969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcfeaffd640 1 -- 172.21.15.40:0/1799953670 learned_addr learned my addr 172.21.15.40:0/1799953670 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcfebfff640 1 --2- 172.21.15.40:0/1799953670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfec1022d0 0x7fcfec19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcfeaffd640 1 -- 172.21.15.40:0/1799953670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfec1022d0 msgr2=0x7fcfec19df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcfeaffd640 1 --2- 172.21.15.40:0/1799953670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfec1022d0 0x7fcfec19df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcfeaffd640 1 -- 172.21.15.40:0/1799953670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfec100720 msgr2=0x7fcfec1969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcfeaffd640 1 --2- 172.21.15.40:0/1799953670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfec100720 0x7fcfec1969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcfeaffd640 1 -- 172.21.15.40:0/1799953670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfec19e600 con 0x7fcfec101910 2024-03-18T22:52:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcfeaffd640 1 --2- 172.21.15.40:0/1799953670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfec101910 0x7fcfec196f20 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fcfe000eb00 tx=0x7fcfe000ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcfe8ff9640 1 -- 172.21.15.40:0/1799953670 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe000bc30 con 0x7fcfec101910 2024-03-18T22:52:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcfe8ff9640 1 -- 172.21.15.40:0/1799953670 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfe000bdd0 con 0x7fcfec101910 2024-03-18T22:52:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcff1e0c640 1 -- 172.21.15.40:0/1799953670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfec0689e0 con 0x7fcfec101910 2024-03-18T22:52:07.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcfe8ff9640 1 -- 172.21.15.40:0/1799953670 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe0017410 con 0x7fcfec101910 2024-03-18T22:52:07.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.579+0000 7fcff1e0c640 1 -- 172.21.15.40:0/1799953670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfec068f30 con 0x7fcfec101910 2024-03-18T22:52:07.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.581+0000 7fcfe8ff9640 1 -- 172.21.15.40:0/1799953670 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcfe0004f90 con 0x7fcfec101910 2024-03-18T22:52:07.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcff49f0740 1 Processor -- start 2024-03-18T22:52:07.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcff49f0740 1 -- start start 2024-03-18T22:52:07.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcff49f0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68aba50 0x561fb68abe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcff49f0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68ac440 0x561fb68b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcff49f0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68b15b0 0x561fb68b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcff49f0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561fb6810500 con 0x561fb68b15b0 2024-03-18T22:52:07.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcff49f0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561fb67b8be0 con 0x561fb68aba50 2024-03-18T22:52:07.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcff49f0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x561fb680c5a0 con 0x561fb68ac440 2024-03-18T22:52:07.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcfe8ff9640 1 --2- 172.21.15.40:0/1799953670 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcfd40a2880 0x7fcfd40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68ac440 0x561fb68b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcfeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68b15b0 0x561fb68b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68ac440 0x561fb68b0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:52472/0 (socket says 172.21.15.40:52472) 2024-03-18T22:52:07.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcfeb7fe640 1 -- 172.21.15.40:0/1525100519 learned_addr learned my addr 172.21.15.40:0/1525100519 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:07.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcfebfff640 1 --2- 172.21.15.40:0/1525100519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68aba50 0x561fb68abe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcfebfff640 1 --2- 172.21.15.40:0/1799953670 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcfd40a2880 0x7fcfd40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcfe8ff9640 1 -- 172.21.15.40:0/1799953670 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fcfe00d0670 con 0x7fcfec101910 2024-03-18T22:52:07.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcfeb7fe640 1 -- 172.21.15.40:0/1525100519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68aba50 msgr2=0x561fb68abe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:07.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcfeb7fe640 1 --2- 172.21.15.40:0/1525100519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68aba50 0x561fb68abe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcfeb7fe640 1 -- 172.21.15.40:0/1525100519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68b15b0 msgr2=0x561fb68b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:07.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcfeb7fe640 1 --2- 172.21.15.40:0/1525100519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68b15b0 0x561fb68b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.582+0000 7fcfeb7fe640 1 -- 172.21.15.40:0/1525100519 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fb67da000 con 0x561fb68ac440 2024-03-18T22:52:07.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.583+0000 7fcfebfff640 1 --2- 172.21.15.40:0/1799953670 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcfd40a2880 0x7fcfd40a4d40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4005fd0 tx=0x7fcfe4009df0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:07.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.583+0000 7fcfeb7fe640 1 --2- 172.21.15.40:0/1525100519 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68ac440 0x561fb68b0f40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc000c00 tx=0x7fcfdc005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d711b85c3c7a0d56 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:07.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.583+0000 7fcfd1ffb640 1 -- 172.21.15.40:0/1525100519 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc045020 con 0x561fb68ac440 2024-03-18T22:52:07.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.583+0000 7fcfd1ffb640 1 -- 172.21.15.40:0/1525100519 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfdc02ca20 con 0x561fb68ac440 2024-03-18T22:52:07.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.583+0000 7fcfd1ffb640 1 -- 172.21.15.40:0/1525100519 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc0055f0 con 0x561fb68ac440 2024-03-18T22:52:07.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.583+0000 7fcff49f0740 1 -- 172.21.15.40:0/1525100519 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68ac440 msgr2=0x561fb68b0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.583+0000 7fcff49f0740 1 --2- 172.21.15.40:0/1525100519 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68ac440 0x561fb68b0f40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc000c00 tx=0x7fcfdc005d50 comp rx=0 tx=0).stop 2024-03-18T22:52:07.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 1 -- 172.21.15.40:0/1525100519 shutdown_connections 2024-03-18T22:52:07.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 1 --2- 172.21.15.40:0/1525100519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68b15b0 0x561fb68b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 1 --2- 172.21.15.40:0/1525100519 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68ac440 0x561fb68b0f40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 1 --2- 172.21.15.40:0/1525100519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68aba50 0x561fb68abe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 1 -- 172.21.15.40:0/1525100519 >> 172.21.15.40:0/1525100519 conn(0x561fb68a6fc0 msgr2=0x561fb68a73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:07.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 1 -- 172.21.15.40:0/1525100519 shutdown_connections 2024-03-18T22:52:07.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 1 -- 172.21.15.40:0/1525100519 wait complete. 2024-03-18T22:52:07.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 1 -- start start 2024-03-18T22:52:07.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 1 Processor -- start 2024-03-18T22:52:07.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68aba50 0x561fb68b5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68ac440 0x561fb68b6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68b15b0 0x561fb68bd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561fb68b4500 con 0x561fb68b15b0 2024-03-18T22:52:07.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561fb67cc160 con 0x561fb68ac440 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x561fb68b4680 con 0x561fb68aba50 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcff49f0740 10 client.0 _ll_register_callbacks cb 0x561fb6808a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcfebfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68b15b0 0x561fb68bd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68aba50 0x561fb68b5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcfebfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68b15b0 0x561fb68bd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59446/0 (socket says 172.21.15.40:59446) 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcfebfff640 1 -- 172.21.15.40:0/2334082954 learned_addr learned my addr 172.21.15.40:0/2334082954 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.584+0000 7fcfeaffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68ac440 0x561fb68b6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcfebfff640 1 -- 172.21.15.40:0/2334082954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68aba50 msgr2=0x561fb68b5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcfebfff640 1 --2- 172.21.15.40:0/2334082954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68aba50 0x561fb68b5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcfebfff640 1 -- 172.21.15.40:0/2334082954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68ac440 msgr2=0x561fb68b6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcfebfff640 1 --2- 172.21.15.40:0/2334082954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68ac440 0x561fb68b6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcfebfff640 1 -- 172.21.15.40:0/2334082954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fb65ee7d0 con 0x561fb68b15b0 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcfeb7fe640 1 --2- 172.21.15.40:0/2334082954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68aba50 0x561fb68b5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcfebfff640 1 --2- 172.21.15.40:0/2334082954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68b15b0 0x561fb68bd3a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400fe20 tx=0x7fcfe401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ad2064dda11c22 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcfcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcfcaffd640 1 -- 172.21.15.40:0/2334082954 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4003e00 con 0x561fb68b15b0 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcfcaffd640 1 -- 172.21.15.40:0/2334082954 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fcfe4004530 con 0x561fb68b15b0 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcff49f0740 1 -- 172.21.15.40:0/2334082954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfdc02ca20 con 0x561fb68b15b0 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcfcaffd640 1 -- 172.21.15.40:0/2334082954 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcfe4032050 con 0x561fb68b15b0 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcff49f0740 10 client.18015 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcff49f0740 1 -- 172.21.15.40:0/2334082954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561fb65243f0 con 0x561fb68b15b0 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.585+0000 7fcfcaffd640 1 -- 172.21.15.40:0/2334082954 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfdc02ca20 con 0x561fb68b15b0 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.586+0000 7fcfcaffd640 1 -- 172.21.15.40:0/2334082954 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7fcfe401e040 con 0x561fb68b15b0 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.586+0000 7fcff49f0740 10 client.18015 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.586+0000 7fcff49f0740 10 client.18015 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.586+0000 7fcff49f0740 20 client.18015 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.586+0000 7fcff49f0740 1 --2- 172.21.15.40:0/2334082954 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561fb68c1d20 0x561fb68c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.586+0000 7fcff49f0740 4 client.18015 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.586+0000 7fcff49f0740 1 -- 172.21.15.40:0/2334082954 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fb681da40 con 0x561fb68c1d20 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.586+0000 7fcfeb7fe640 1 --2- 172.21.15.40:0/2334082954 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561fb68c1d20 0x561fb68c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.586+0000 7fcfeb7fe640 1 --2- 172.21.15.40:0/2334082954 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561fb68c1d20 0x561fb68c41e0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83b77f04f90c578 server_cookie=11ff1e1f402927b8 in_seq=0 out_seq=0 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.586+0000 7fcfcaffd640 10 client.18015 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.587+0000 7fcfcaffd640 1 -- 172.21.15.40:0/2334082954 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561fb681da40 con 0x561fb68c1d20 2024-03-18T22:52:07.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.587+0000 7fcfcaffd640 10 client.18015 handle_command_reply: tid=0 2024-03-18T22:52:07.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.587+0000 7fcff49f0740 1 client.18015 shutdown 2024-03-18T22:52:07.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.587+0000 7fcff49f0740 1 -- 172.21.15.40:0/2334082954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68b15b0 msgr2=0x561fb68bd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.587+0000 7fcff49f0740 1 --2- 172.21.15.40:0/2334082954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68b15b0 0x561fb68bd3a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400fe20 tx=0x7fcfe401c000 comp rx=0 tx=0).stop 2024-03-18T22:52:07.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.587+0000 7fcff49f0740 1 -- 172.21.15.40:0/2334082954 shutdown_connections 2024-03-18T22:52:07.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.587+0000 7fcff49f0740 1 --2- 172.21.15.40:0/2334082954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68b15b0 0x561fb68bd3a0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.587+0000 7fcff49f0740 1 --2- 172.21.15.40:0/2334082954 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561fb68c1d20 0x561fb68c41e0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.587+0000 7fcff49f0740 1 --2- 172.21.15.40:0/2334082954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68ac440 0x561fb68b6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.587+0000 7fcff49f0740 1 --2- 172.21.15.40:0/2334082954 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68aba50 0x561fb68b5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.587+0000 7fcfcaffd640 0 client.18015 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:07.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.587+0000 7fcff49f0740 1 -- 172.21.15.40:0/2334082954 >> 172.21.15.40:0/2334082954 conn(0x561fb68a6fc0 msgr2=0x561fb68a61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:07.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.587+0000 7fcff49f0740 1 -- 172.21.15.40:0/2334082954 shutdown_connections 2024-03-18T22:52:07.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.587+0000 7fcff49f0740 1 -- 172.21.15.40:0/2334082954 wait complete. 2024-03-18T22:52:07.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.587+0000 7fcff49f0740 20 client.18015 trim_cache size 0 max 16384 2024-03-18T22:52:07.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcff49f0740 1 Processor -- start 2024-03-18T22:52:07.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcff49f0740 1 -- start start 2024-03-18T22:52:07.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcff49f0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68aba50 0x561fb68a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcff49f0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68ac440 0x561fb68bd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcff49f0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68b15b0 0x561fb68aa930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcff49f0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec1a8270 con 0x561fb68aba50 2024-03-18T22:52:07.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcff49f0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec1a9e10 con 0x561fb68ac440 2024-03-18T22:52:07.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcff49f0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec1a8940 con 0x561fb68b15b0 2024-03-18T22:52:07.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68aba50 0x561fb68a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68aba50 0x561fb68a1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59456/0 (socket says 172.21.15.40:59456) 2024-03-18T22:52:07.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcfeb7fe640 1 -- 172.21.15.40:0/1076826434 learned_addr learned my addr 172.21.15.40:0/1076826434 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:07.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcfebfff640 1 --2- 172.21.15.40:0/1076826434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68b15b0 0x561fb68aa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcfeaffd640 1 --2- 172.21.15.40:0/1076826434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68ac440 0x561fb68bd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcfeb7fe640 1 -- 172.21.15.40:0/1076826434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68b15b0 msgr2=0x561fb68aa930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:07.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcfeb7fe640 1 --2- 172.21.15.40:0/1076826434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68b15b0 0x561fb68aa930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcfeb7fe640 1 -- 172.21.15.40:0/1076826434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68ac440 msgr2=0x561fb68bd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcfeb7fe640 1 --2- 172.21.15.40:0/1076826434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68ac440 0x561fb68bd360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.605+0000 7fcfeb7fe640 1 -- 172.21.15.40:0/1076826434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fb68adf20 con 0x561fb68aba50 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.606+0000 7fcfeb7fe640 1 --2- 172.21.15.40:0/1076826434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68aba50 0x561fb68a1f80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc005bc0 tx=0x7fcfdc02c2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3208f3fb0b70ae2d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.606+0000 7fcfd17fa640 1 -- 172.21.15.40:0/1076826434 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc052070 con 0x561fb68aba50 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.606+0000 7fcfd17fa640 1 -- 172.21.15.40:0/1076826434 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfdc02ca20 con 0x561fb68aba50 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.606+0000 7fcfd17fa640 1 -- 172.21.15.40:0/1076826434 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc045040 con 0x561fb68aba50 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.606+0000 7fcff49f0740 1 -- 172.21.15.40:0/1076826434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68aba50 msgr2=0x561fb68a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.606+0000 7fcff49f0740 1 --2- 172.21.15.40:0/1076826434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68aba50 0x561fb68a1f80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc005bc0 tx=0x7fcfdc02c2b0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.606+0000 7fcff49f0740 1 -- 172.21.15.40:0/1076826434 shutdown_connections 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.606+0000 7fcff49f0740 1 --2- 172.21.15.40:0/1076826434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68b15b0 0x561fb68aa930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.606+0000 7fcff49f0740 1 --2- 172.21.15.40:0/1076826434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68ac440 0x561fb68bd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.606+0000 7fcff49f0740 1 --2- 172.21.15.40:0/1076826434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68aba50 0x561fb68a1f80 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcff49f0740 1 -- 172.21.15.40:0/1076826434 >> 172.21.15.40:0/1076826434 conn(0x561fb68a6fc0 msgr2=0x561fb68aa220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcff49f0740 1 -- 172.21.15.40:0/1076826434 shutdown_connections 2024-03-18T22:52:07.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcff49f0740 1 -- 172.21.15.40:0/1076826434 wait complete. 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcff49f0740 1 -- start start 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcff49f0740 1 Processor -- start 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcff49f0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68aba50 0x561fb68baa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcff49f0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68ac440 0x561fb68baf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcff49f0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68b15b0 0x561fb68bbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcff49f0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec1a9740 con 0x561fb68aba50 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcff49f0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec1a9010 con 0x561fb68ac440 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcff49f0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec1a6250 con 0x561fb68b15b0 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcff49f0740 10 client.0 _ll_register_callbacks cb 0x7fcfe400faf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68aba50 0x561fb68baa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68aba50 0x561fb68baa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59466/0 (socket says 172.21.15.40:59466) 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcfeb7fe640 1 -- 172.21.15.40:0/2052642711 learned_addr learned my addr 172.21.15.40:0/2052642711 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcfebfff640 1 --2- 172.21.15.40:0/2052642711 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68b15b0 0x561fb68bbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.607+0000 7fcfeaffd640 1 --2- 172.21.15.40:0/2052642711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68ac440 0x561fb68baf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.608+0000 7fcfeb7fe640 1 -- 172.21.15.40:0/2052642711 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68b15b0 msgr2=0x561fb68bbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.608+0000 7fcfeb7fe640 1 --2- 172.21.15.40:0/2052642711 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68b15b0 0x561fb68bbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.608+0000 7fcfeb7fe640 1 -- 172.21.15.40:0/2052642711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68ac440 msgr2=0x561fb68baf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.608+0000 7fcfeb7fe640 1 --2- 172.21.15.40:0/2052642711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68ac440 0x561fb68baf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.608+0000 7fcfeb7fe640 1 -- 172.21.15.40:0/2052642711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fb64e40f0 con 0x561fb68aba50 2024-03-18T22:52:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.608+0000 7fcfeb7fe640 1 --2- 172.21.15.40:0/2052642711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68aba50 0x561fb68baa10 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc02c170 tx=0x7fcfdc02ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=499fd82d93d517c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.608+0000 7fcfcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.608+0000 7fcfcaffd640 1 -- 172.21.15.40:0/2052642711 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc045040 con 0x561fb68aba50 2024-03-18T22:52:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.608+0000 7fcff49f0740 1 -- 172.21.15.40:0/2052642711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfdc02ca20 con 0x561fb68aba50 2024-03-18T22:52:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.608+0000 7fcfcaffd640 1 -- 172.21.15.40:0/2052642711 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fcfdc0035f0 con 0x561fb68aba50 2024-03-18T22:52:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.608+0000 7fcfcaffd640 1 -- 172.21.15.40:0/2052642711 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcfdc052040 con 0x561fb68aba50 2024-03-18T22:52:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.608+0000 7fcff49f0740 10 client.18021 fetch_fsmap learned FSMap version 35 2024-03-18T22:52:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.608+0000 7fcff49f0740 1 -- 172.21.15.40:0/2052642711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561fb68bd690 con 0x561fb68aba50 2024-03-18T22:52:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.608+0000 7fcfcaffd640 1 -- 172.21.15.40:0/2052642711 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcfdc02ca20 con 0x561fb68aba50 2024-03-18T22:52:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.608+0000 7fcfcaffd640 1 -- 172.21.15.40:0/2052642711 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) ==== 8370+0+0 (secure 0 0 0) 0x7fcfdc041750 con 0x561fb68aba50 2024-03-18T22:52:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 10 client.18021 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T22:52:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 10 client.18021 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 20 client.18021 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 1 --2- 172.21.15.40:0/2052642711 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561fb6928420 0x561fb692a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 4 client.18021 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 1 -- 172.21.15.40:0/2052642711 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561fb6817d90 con 0x561fb6928420 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcfeaffd640 1 --2- 172.21.15.40:0/2052642711 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561fb6928420 0x561fb692a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcfeaffd640 1 --2- 172.21.15.40:0/2052642711 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561fb6928420 0x561fb692a810 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e5525a9d6e54f18 server_cookie=fff732bafc6e33bf in_seq=0 out_seq=0 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcfcaffd640 10 client.18021 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcfcaffd640 1 -- 172.21.15.40:0/2052642711 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x561fb6817d90 con 0x561fb6928420 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcfcaffd640 10 client.18021 handle_command_reply: tid=0 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 1 client.18021 shutdown 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 1 -- 172.21.15.40:0/2052642711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68aba50 msgr2=0x561fb68baa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 1 --2- 172.21.15.40:0/2052642711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68aba50 0x561fb68baa10 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc02c170 tx=0x7fcfdc02ccf0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 1 -- 172.21.15.40:0/2052642711 shutdown_connections 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 1 --2- 172.21.15.40:0/2052642711 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561fb68b15b0 0x561fb68bbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 1 --2- 172.21.15.40:0/2052642711 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561fb6928420 0x561fb692a810 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 1 --2- 172.21.15.40:0/2052642711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561fb68ac440 0x561fb68baf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 1 --2- 172.21.15.40:0/2052642711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561fb68aba50 0x561fb68baa10 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcfcaffd640 0 client.18021 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:07.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 1 -- 172.21.15.40:0/2052642711 >> 172.21.15.40:0/2052642711 conn(0x561fb68a6fc0 msgr2=0x561fb68aaab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:07.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 1 -- 172.21.15.40:0/2052642711 shutdown_connections 2024-03-18T22:52:07.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.609+0000 7fcff49f0740 1 -- 172.21.15.40:0/2052642711 wait complete. 2024-03-18T22:52:07.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.610+0000 7fcff49f0740 20 client.18021 trim_cache size 0 max 16384 2024-03-18T22:52:07.618 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:52:07.618 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "scrub active (66 inodes in the stack)", 2024-03-18T22:52:07.618 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:52:07.618 INFO:teuthology.orchestra.run.smithi040.stdout: "2a217dbe-47f6-4fd5-94c0-c49c0f03b1a9": { 2024-03-18T22:52:07.618 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:52:07.618 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "2a217dbe-47f6-4fd5-94c0-c49c0f03b1a9", 2024-03-18T22:52:07.618 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:52:07.618 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:52:07.618 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:52:07.618 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:52:07.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.610+0000 7fcff1e0c640 1 -- 172.21.15.40:0/1799953670 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcfd40a2880 msgr2=0x7fcfd40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.610+0000 7fcff1e0c640 1 --2- 172.21.15.40:0/1799953670 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcfd40a2880 0x7fcfd40a4d40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4005fd0 tx=0x7fcfe4009df0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.610+0000 7fcff1e0c640 1 -- 172.21.15.40:0/1799953670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfec101910 msgr2=0x7fcfec196f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:07.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.610+0000 7fcff1e0c640 1 --2- 172.21.15.40:0/1799953670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfec101910 0x7fcfec196f20 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fcfe000eb00 tx=0x7fcfe000ef00 comp rx=0 tx=0).stop 2024-03-18T22:52:07.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.610+0000 7fcff1e0c640 1 -- 172.21.15.40:0/1799953670 shutdown_connections 2024-03-18T22:52:07.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.610+0000 7fcff1e0c640 1 --2- 172.21.15.40:0/1799953670 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcfd40a2880 0x7fcfd40a4d40 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.610+0000 7fcff1e0c640 1 --2- 172.21.15.40:0/1799953670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcfec1022d0 0x7fcfec19df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.610+0000 7fcff1e0c640 1 --2- 172.21.15.40:0/1799953670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcfec101910 0x7fcfec196f20 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.610+0000 7fcff1e0c640 1 --2- 172.21.15.40:0/1799953670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfec100720 0x7fcfec1969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:07.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.610+0000 7fcff1e0c640 1 -- 172.21.15.40:0/1799953670 >> 172.21.15.40:0/1799953670 conn(0x7fcfec0fbcb0 msgr2=0x7fcfec104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:07.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.610+0000 7fcff1e0c640 1 -- 172.21.15.40:0/1799953670 shutdown_connections 2024-03-18T22:52:07.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:07.610+0000 7fcff1e0c640 1 -- 172.21.15.40:0/1799953670 wait complete. 2024-03-18T22:52:07.621 INFO:tasks.cephfs.filesystem:scrub status for tag:2a217dbe-47f6-4fd5-94c0-c49c0f03b1a9 - {'path': '/', 'tag': '2a217dbe-47f6-4fd5-94c0-c49c0f03b1a9', 'options': 'recursive,force'} 2024-03-18T22:52:08.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:08 smithi113 ceph-mon[66775]: pgmap v635: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 16 MiB/s wr, 2.49k op/s 2024-03-18T22:52:08.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:08 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:52:08.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:08 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:08.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:08 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:52:08.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:08 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:52:08.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:08 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:08.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:08 smithi103 ceph-mon[66377]: pgmap v635: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 16 MiB/s wr, 2.49k op/s 2024-03-18T22:52:08.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:08 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:52:08.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:08 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:08.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:08 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:52:08.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:08 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:52:08.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:08 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:08.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:08 smithi040 ceph-mon[62643]: pgmap v635: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 16 MiB/s wr, 2.49k op/s 2024-03-18T22:52:08.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:08 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:52:08.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:08 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:08.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:08 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:52:08.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:08 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:52:08.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:08 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:09.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:09 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:52:09.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:09 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:09.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:09 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:09.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:09 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:52:09.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:09 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:09.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:09 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:09.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:09 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:52:09.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:09 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:09.647 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:09 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:10.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:10 smithi113 ceph-mon[66775]: pgmap v636: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 3.08k op/s 2024-03-18T22:52:10.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:10 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:52:10.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:10 smithi103 ceph-mon[66377]: pgmap v636: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 3.08k op/s 2024-03-18T22:52:10.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:10 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:52:10.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:10 smithi040 ceph-mon[62643]: pgmap v636: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 3.08k op/s 2024-03-18T22:52:10.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:10 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:52:12.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:12 smithi113 ceph-mon[66775]: pgmap v637: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 MiB/s wr, 3.81k op/s 2024-03-18T22:52:12.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:12 smithi103 ceph-mon[66377]: pgmap v637: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 MiB/s wr, 3.81k op/s 2024-03-18T22:52:12.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:12 smithi040 ceph-mon[62643]: pgmap v637: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 MiB/s wr, 3.81k op/s 2024-03-18T22:52:14.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:14 smithi113 ceph-mon[66775]: pgmap v638: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 MiB/s wr, 4.65k op/s 2024-03-18T22:52:14.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:14 smithi103 ceph-mon[66377]: pgmap v638: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 MiB/s wr, 4.65k op/s 2024-03-18T22:52:14.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:14 smithi040 ceph-mon[62643]: pgmap v638: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 MiB/s wr, 4.65k op/s 2024-03-18T22:52:16.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:16 smithi113 ceph-mon[66775]: pgmap v639: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 MiB/s wr, 4.36k op/s 2024-03-18T22:52:16.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:16 smithi103 ceph-mon[66377]: pgmap v639: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 MiB/s wr, 4.36k op/s 2024-03-18T22:52:16.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:16 smithi040 ceph-mon[62643]: pgmap v639: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 MiB/s wr, 4.36k op/s 2024-03-18T22:52:18.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:18 smithi113 ceph-mon[66775]: pgmap v640: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 MiB/s wr, 4.94k op/s 2024-03-18T22:52:18.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:18 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:52:18.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:18 smithi103 ceph-mon[66377]: pgmap v640: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 MiB/s wr, 4.94k op/s 2024-03-18T22:52:18.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:18 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:52:18.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:18 smithi040 ceph-mon[62643]: pgmap v640: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 MiB/s wr, 4.94k op/s 2024-03-18T22:52:18.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:18 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:52:19.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:19 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:52:19.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:19 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:52:19.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:19 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:52:19.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:19 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:52:19.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:19 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:52:19.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:19 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:52:20.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:20 smithi113 ceph-mon[66775]: pgmap v641: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 17 MiB/s wr, 4.43k op/s 2024-03-18T22:52:20.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:20 smithi103 ceph-mon[66377]: pgmap v641: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 17 MiB/s wr, 4.43k op/s 2024-03-18T22:52:20.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:20 smithi040 ceph-mon[62643]: pgmap v641: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 17 MiB/s wr, 4.43k op/s 2024-03-18T22:52:22.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:22 smithi113 ceph-mon[66775]: pgmap v642: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 18 MiB/s wr, 4.50k op/s 2024-03-18T22:52:22.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:22 smithi103 ceph-mon[66377]: pgmap v642: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 18 MiB/s wr, 4.50k op/s 2024-03-18T22:52:22.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:22 smithi040 ceph-mon[62643]: pgmap v642: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 18 MiB/s wr, 4.50k op/s 2024-03-18T22:52:24.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:24 smithi113 ceph-mon[66775]: pgmap v643: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 MiB/s wr, 4.86k op/s 2024-03-18T22:52:24.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:24 smithi103 ceph-mon[66377]: pgmap v643: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 MiB/s wr, 4.86k op/s 2024-03-18T22:52:24.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:24 smithi040 ceph-mon[62643]: pgmap v643: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 MiB/s wr, 4.86k op/s 2024-03-18T22:52:26.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:26 smithi113 ceph-mon[66775]: pgmap v644: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 MiB/s wr, 4.22k op/s 2024-03-18T22:52:26.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:26 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:52:26.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:26 smithi103 ceph-mon[66377]: pgmap v644: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 MiB/s wr, 4.22k op/s 2024-03-18T22:52:26.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:26 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:52:26.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:26 smithi040 ceph-mon[62643]: pgmap v644: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 MiB/s wr, 4.22k op/s 2024-03-18T22:52:26.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:26 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:52:28.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:28 smithi113 ceph-mon[66775]: pgmap v645: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 19 MiB/s wr, 4.79k op/s 2024-03-18T22:52:28.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:28 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:52:28.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:28 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:52:28.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:28 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:52:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:28 smithi103 ceph-mon[66377]: pgmap v645: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 19 MiB/s wr, 4.79k op/s 2024-03-18T22:52:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:28 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:52:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:28 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:52:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:28 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:52:28.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:28 smithi040 ceph-mon[62643]: pgmap v645: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 19 MiB/s wr, 4.79k op/s 2024-03-18T22:52:28.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:28 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:52:28.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:28 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:52:28.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:28 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:52:30.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:30 smithi113 ceph-mon[66775]: pgmap v646: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 MiB/s wr, 4.32k op/s 2024-03-18T22:52:30.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:30 smithi103 ceph-mon[66377]: pgmap v646: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 MiB/s wr, 4.32k op/s 2024-03-18T22:52:30.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:30 smithi040 ceph-mon[62643]: pgmap v646: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 MiB/s wr, 4.32k op/s 2024-03-18T22:52:32.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:32 smithi103 ceph-mon[66377]: pgmap v647: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 MiB/s wr, 4.38k op/s 2024-03-18T22:52:32.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:32 smithi040 ceph-mon[62643]: pgmap v647: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 MiB/s wr, 4.38k op/s 2024-03-18T22:52:32.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:32 smithi113 ceph-mon[66775]: pgmap v647: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 MiB/s wr, 4.38k op/s 2024-03-18T22:52:34.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:34 smithi103 ceph-mon[66377]: pgmap v648: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 19 MiB/s wr, 4.84k op/s 2024-03-18T22:52:34.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:34 smithi040 ceph-mon[62643]: pgmap v648: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 19 MiB/s wr, 4.84k op/s 2024-03-18T22:52:34.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:34 smithi113 ceph-mon[66775]: pgmap v648: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 19 MiB/s wr, 4.84k op/s 2024-03-18T22:52:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:36 smithi103 ceph-mon[66377]: pgmap v649: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 MiB/s wr, 4.22k op/s 2024-03-18T22:52:36.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:36 smithi040 ceph-mon[62643]: pgmap v649: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 MiB/s wr, 4.22k op/s 2024-03-18T22:52:36.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:36 smithi113 ceph-mon[66775]: pgmap v649: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 MiB/s wr, 4.22k op/s 2024-03-18T22:52:37.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:52:37.622 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:52:37.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.955+0000 7ff096152640 1 -- 172.21.15.40:0/3253299868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff090072c00 msgr2=0x7ff090073020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:37.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.955+0000 7ff096152640 1 --2- 172.21.15.40:0/3253299868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff090072c00 0x7ff090073020 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff084009860 tx=0x7ff08402bc10 comp rx=0 tx=0).stop 2024-03-18T22:52:37.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.959+0000 7ff096152640 1 -- 172.21.15.40:0/3253299868 shutdown_connections 2024-03-18T22:52:37.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.959+0000 7ff096152640 1 --2- 172.21.15.40:0/3253299868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0901098d0 0x7ff09010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:37.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.959+0000 7ff096152640 1 --2- 172.21.15.40:0/3253299868 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff090108f30 0x7ff090109390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:37.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.959+0000 7ff096152640 1 --2- 172.21.15.40:0/3253299868 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff090072c00 0x7ff090073020 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:37.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.959+0000 7ff096152640 1 -- 172.21.15.40:0/3253299868 >> 172.21.15.40:0/3253299868 conn(0x7ff09006e700 msgr2=0x7ff090070b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:37.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.970+0000 7ff096152640 1 -- 172.21.15.40:0/3253299868 shutdown_connections 2024-03-18T22:52:37.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.970+0000 7ff096152640 1 -- 172.21.15.40:0/3253299868 wait complete. 2024-03-18T22:52:37.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.971+0000 7ff096152640 1 Processor -- start 2024-03-18T22:52:37.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.974+0000 7ff096152640 1 -- start start 2024-03-18T22:52:37.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.974+0000 7ff096152640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff090072c00 0x7ff0901a6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:37.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.974+0000 7ff096152640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff090108f30 0x7ff0901a6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:37.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.974+0000 7ff096152640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0901098d0 0x7ff0901adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:37.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.974+0000 7ff096152640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff09011c130 con 0x7ff090072c00 2024-03-18T22:52:37.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.974+0000 7ff096152640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff09011bfb0 con 0x7ff0901098d0 2024-03-18T22:52:37.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.974+0000 7ff096152640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff09011c2b0 con 0x7ff090108f30 2024-03-18T22:52:37.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.976+0000 7ff08ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0901098d0 0x7ff0901adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:37.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.976+0000 7ff08ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0901098d0 0x7ff0901adde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:55636/0 (socket says 172.21.15.40:55636) 2024-03-18T22:52:37.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.976+0000 7ff08ffff640 1 -- 172.21.15.40:0/3327382198 learned_addr learned my addr 172.21.15.40:0/3327382198 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:37.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.978+0000 7ff08ffff640 1 -- 172.21.15.40:0/3327382198 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff090108f30 msgr2=0x7ff0901a6d70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T22:52:37.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.978+0000 7ff08ffff640 1 --2- 172.21.15.40:0/3327382198 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff090108f30 0x7ff0901a6d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:37.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.978+0000 7ff08ffff640 1 -- 172.21.15.40:0/3327382198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff090072c00 msgr2=0x7ff0901a6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:37.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.978+0000 7ff08ffff640 1 --2- 172.21.15.40:0/3327382198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff090072c00 0x7ff0901a6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:37.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.978+0000 7ff08ffff640 1 -- 172.21.15.40:0/3327382198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff090110080 con 0x7ff0901098d0 2024-03-18T22:52:37.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.979+0000 7ff08ffff640 1 --2- 172.21.15.40:0/3327382198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0901098d0 0x7ff0901adde0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff08000b3e0 tx=0x7ff080009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:37.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.980+0000 7ff08cff9640 1 -- 172.21.15.40:0/3327382198 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff080015020 con 0x7ff0901098d0 2024-03-18T22:52:37.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.980+0000 7ff096152640 1 -- 172.21.15.40:0/3327382198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff090110370 con 0x7ff0901098d0 2024-03-18T22:52:37.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.980+0000 7ff096152640 1 -- 172.21.15.40:0/3327382198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0901108c0 con 0x7ff0901098d0 2024-03-18T22:52:37.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.980+0000 7ff08cff9640 1 -- 172.21.15.40:0/3327382198 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff080011ce0 con 0x7ff0901098d0 2024-03-18T22:52:37.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.980+0000 7ff08cff9640 1 -- 172.21.15.40:0/3327382198 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff080010de0 con 0x7ff0901098d0 2024-03-18T22:52:37.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.986+0000 7ff098e10740 1 Processor -- start 2024-03-18T22:52:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.988+0000 7ff08cff9640 1 -- 172.21.15.40:0/3327382198 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff080018020 con 0x7ff0901098d0 2024-03-18T22:52:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.988+0000 7ff08cff9640 1 --2- 172.21.15.40:0/3327382198 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff0780a2a00 0x7ff0780a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.989+0000 7ff098e10740 1 -- start start 2024-03-18T22:52:37.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.989+0000 7ff08cff9640 1 -- 172.21.15.40:0/3327382198 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff0800ce590 con 0x7ff0901098d0 2024-03-18T22:52:37.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.989+0000 7ff098e10740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1e9c10 0x55568f1ea030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:37.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.989+0000 7ff098e10740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1ea600 0x55568f1ef100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:37.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.989+0000 7ff098e10740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ef770 0x55568f1f1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:37.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.989+0000 7ff098e10740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55568f14e500 con 0x55568f1ea600 2024-03-18T22:52:37.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.989+0000 7ff098e10740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55568f0f6be0 con 0x55568f1ef770 2024-03-18T22:52:37.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.989+0000 7ff098e10740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55568f14a5a0 con 0x55568f1e9c10 2024-03-18T22:52:37.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.989+0000 7ff08f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1e9c10 0x55568f1ea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:37.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.989+0000 7ff08f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1e9c10 0x55568f1ea030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:42288/0 (socket says 172.21.15.40:42288) 2024-03-18T22:52:37.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.989+0000 7ff08f7fe640 1 -- 172.21.15.40:0/2612440885 learned_addr learned my addr 172.21.15.40:0/2612440885 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:37.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.990+0000 7ff08effd640 1 --2- 172.21.15.40:0/2612440885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1ea600 0x55568f1ef100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:37.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.990+0000 7ff08ffff640 1 --2- 172.21.15.40:0/2612440885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ef770 0x55568f1f1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:37.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.992+0000 7ff08effd640 1 --2- 172.21.15.40:0/3327382198 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff0780a2a00 0x7ff0780a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:37.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.992+0000 7ff08effd640 1 -- 172.21.15.40:0/2612440885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1e9c10 msgr2=0x55568f1ea030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:37.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.992+0000 7ff08effd640 1 --2- 172.21.15.40:0/2612440885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1e9c10 0x55568f1ea030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:37.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.992+0000 7ff08effd640 1 -- 172.21.15.40:0/2612440885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ef770 msgr2=0x55568f1f1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:37.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.992+0000 7ff08effd640 1 --2- 172.21.15.40:0/2612440885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ef770 0x55568f1f1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:37.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.992+0000 7ff08effd640 1 -- 172.21.15.40:0/2612440885 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55568f118000 con 0x55568f1ea600 2024-03-18T22:52:37.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.997+0000 7ff08effd640 1 --2- 172.21.15.40:0/2612440885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1ea600 0x55568f1ef100 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff088005fd0 tx=0x7ff08800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b421220a0601bad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:38.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.997+0000 7ff08effd640 1 --2- 172.21.15.40:0/3327382198 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff0780a2a00 0x7ff0780a4ec0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff088005c40 tx=0x7ff08801b000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:38.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.998+0000 7ff075ffb640 1 -- 172.21.15.40:0/2612440885 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088021070 con 0x55568f1ea600 2024-03-18T22:52:38.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.998+0000 7ff075ffb640 1 -- 172.21.15.40:0/2612440885 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff088003990 con 0x55568f1ea600 2024-03-18T22:52:38.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.999+0000 7ff098e10740 1 -- 172.21.15.40:0/2612440885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1ea600 msgr2=0x55568f1ef100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:38.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:37.999+0000 7ff098e10740 1 --2- 172.21.15.40:0/2612440885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1ea600 0x55568f1ef100 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff088005fd0 tx=0x7ff08800ec10 comp rx=0 tx=0).stop 2024-03-18T22:52:38.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.000+0000 7ff098e10740 1 -- 172.21.15.40:0/2612440885 shutdown_connections 2024-03-18T22:52:38.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.000+0000 7ff098e10740 1 --2- 172.21.15.40:0/2612440885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ef770 0x55568f1f1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.000+0000 7ff098e10740 1 --2- 172.21.15.40:0/2612440885 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1ea600 0x55568f1ef100 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.000+0000 7ff098e10740 1 --2- 172.21.15.40:0/2612440885 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1e9c10 0x55568f1ea030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.000+0000 7ff098e10740 1 -- 172.21.15.40:0/2612440885 >> 172.21.15.40:0/2612440885 conn(0x55568f1e5180 msgr2=0x55568f1e5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:38.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.005+0000 7ff098e10740 1 -- 172.21.15.40:0/2612440885 shutdown_connections 2024-03-18T22:52:38.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.005+0000 7ff098e10740 1 -- 172.21.15.40:0/2612440885 wait complete. 2024-03-18T22:52:38.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.005+0000 7ff098e10740 1 -- start start 2024-03-18T22:52:38.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.006+0000 7ff098e10740 1 Processor -- start 2024-03-18T22:52:38.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.006+0000 7ff098e10740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1e9c10 0x55568f1f4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:38.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.006+0000 7ff098e10740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ea600 0x55568f1f4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:38.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.006+0000 7ff098e10740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ef770 0x55568f1fb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:38.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.006+0000 7ff098e10740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55568f1f2710 con 0x55568f1e9c10 2024-03-18T22:52:38.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.006+0000 7ff098e10740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55568f10a160 con 0x55568f1ef770 2024-03-18T22:52:38.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.006+0000 7ff098e10740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55568f1f2890 con 0x55568f1ea600 2024-03-18T22:52:38.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.006+0000 7ff098e10740 10 client.0 _ll_register_callbacks cb 0x55568f146a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:38.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.007+0000 7ff08f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1e9c10 0x55568f1f4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:38.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.007+0000 7ff08f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1e9c10 0x55568f1f4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38360/0 (socket says 172.21.15.40:38360) 2024-03-18T22:52:38.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.007+0000 7ff08f7fe640 1 -- 172.21.15.40:0/2870467298 learned_addr learned my addr 172.21.15.40:0/2870467298 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:38.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.008+0000 7ff08f7fe640 1 -- 172.21.15.40:0/2870467298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ea600 msgr2=0x55568f1f4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:38.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.008+0000 7ff08f7fe640 1 --2- 172.21.15.40:0/2870467298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ea600 0x55568f1f4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.008+0000 7ff08f7fe640 1 -- 172.21.15.40:0/2870467298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ef770 msgr2=0x55568f1fb660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:52:38.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.008+0000 7ff08f7fe640 1 --2- 172.21.15.40:0/2870467298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ef770 0x55568f1fb660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.008+0000 7ff08f7fe640 1 -- 172.21.15.40:0/2870467298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55568ef2c7d0 con 0x55568f1e9c10 2024-03-18T22:52:38.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.009+0000 7ff08f7fe640 1 --2- 172.21.15.40:0/2870467298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1e9c10 0x55568f1f4150 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff084009860 tx=0x7ff084009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36f07e8672442208 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:38.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.009+0000 7ff06effd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:38.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.009+0000 7ff06effd640 1 -- 172.21.15.40:0/2870467298 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff084039040 con 0x55568f1e9c10 2024-03-18T22:52:38.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.009+0000 7ff06effd640 1 -- 172.21.15.40:0/2870467298 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff084034d30 con 0x55568f1e9c10 2024-03-18T22:52:38.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.009+0000 7ff098e10740 1 -- 172.21.15.40:0/2870467298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff088003990 con 0x55568f1e9c10 2024-03-18T22:52:38.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.010+0000 7ff06effd640 1 -- 172.21.15.40:0/2870467298 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff084036ea0 con 0x55568f1e9c10 2024-03-18T22:52:38.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.012+0000 7ff098e10740 10 client.18027 fetch_fsmap learned FSMap version 36 2024-03-18T22:52:38.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.013+0000 7ff098e10740 1 -- 172.21.15.40:0/2870467298 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55568ee623f0 con 0x55568f1e9c10 2024-03-18T22:52:38.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.013+0000 7ff06effd640 1 -- 172.21.15.40:0/2870467298 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff088003990 con 0x55568f1e9c10 2024-03-18T22:52:38.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.013+0000 7ff06effd640 1 -- 172.21.15.40:0/2870467298 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 36) ==== 8378+0+0 (secure 0 0 0) 0x7ff084047030 con 0x55568f1e9c10 2024-03-18T22:52:38.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.014+0000 7ff098e10740 10 client.18027 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T22:52:38.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.014+0000 7ff098e10740 10 client.18027 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:38.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.014+0000 7ff098e10740 20 client.18027 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:38.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.014+0000 7ff098e10740 1 --2- 172.21.15.40:0/2870467298 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55568f2004d0 0x55568f202970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:38.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.014+0000 7ff098e10740 4 client.18027 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:38.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.014+0000 7ff098e10740 1 -- 172.21.15.40:0/2870467298 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55568f15ba40 con 0x55568f2004d0 2024-03-18T22:52:38.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.016+0000 7ff08effd640 1 --2- 172.21.15.40:0/2870467298 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55568f2004d0 0x55568f202970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:38.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.026+0000 7ff08effd640 1 --2- 172.21.15.40:0/2870467298 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55568f2004d0 0x55568f202970 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13149a058ec5ab50 server_cookie=47a78e4c744197b6 in_seq=0 out_seq=0 2024-03-18T22:52:38.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.027+0000 7ff06effd640 10 client.18027 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:38.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.031+0000 7ff06effd640 1 -- 172.21.15.40:0/2870467298 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55568f15ba40 con 0x55568f2004d0 2024-03-18T22:52:38.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.031+0000 7ff06effd640 10 client.18027 handle_command_reply: tid=0 2024-03-18T22:52:38.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.031+0000 7ff098e10740 1 client.18027 shutdown 2024-03-18T22:52:38.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.039+0000 7ff098e10740 1 -- 172.21.15.40:0/2870467298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1e9c10 msgr2=0x55568f1f4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:38.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.039+0000 7ff098e10740 1 --2- 172.21.15.40:0/2870467298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1e9c10 0x55568f1f4150 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff084009860 tx=0x7ff084009520 comp rx=0 tx=0).stop 2024-03-18T22:52:38.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.039+0000 7ff098e10740 1 -- 172.21.15.40:0/2870467298 shutdown_connections 2024-03-18T22:52:38.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.039+0000 7ff098e10740 1 --2- 172.21.15.40:0/2870467298 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55568f2004d0 0x55568f202970 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.039+0000 7ff098e10740 1 --2- 172.21.15.40:0/2870467298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ef770 0x55568f1fb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.039+0000 7ff098e10740 1 --2- 172.21.15.40:0/2870467298 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ea600 0x55568f1f4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.039+0000 7ff098e10740 1 --2- 172.21.15.40:0/2870467298 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1e9c10 0x55568f1f4150 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.039+0000 7ff098e10740 1 -- 172.21.15.40:0/2870467298 >> 172.21.15.40:0/2870467298 conn(0x55568f1e5180 msgr2=0x55568f1e44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:38.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.040+0000 7ff06effd640 0 client.18027 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:38.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.044+0000 7ff098e10740 1 -- 172.21.15.40:0/2870467298 shutdown_connections 2024-03-18T22:52:38.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.044+0000 7ff098e10740 1 -- 172.21.15.40:0/2870467298 wait complete. 2024-03-18T22:52:38.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.045+0000 7ff098e10740 20 client.18027 trim_cache size 0 max 16384 2024-03-18T22:52:38.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.097+0000 7ff098e10740 1 Processor -- start 2024-03-18T22:52:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.098+0000 7ff098e10740 1 -- start start 2024-03-18T22:52:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.098+0000 7ff098e10740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1e9c10 0x55568f1e3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.099+0000 7ff098e10740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1ea600 0x55568f1fb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.099+0000 7ff098e10740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ef770 0x55568f1e4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.099+0000 7ff098e10740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c00fde0 con 0x55568f1ea600 2024-03-18T22:52:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.099+0000 7ff098e10740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c0119e0 con 0x55568f1e9c10 2024-03-18T22:52:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.099+0000 7ff08ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ef770 0x55568f1e4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.099+0000 7ff08ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ef770 0x55568f1e4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:42296/0 (socket says 172.21.15.40:42296) 2024-03-18T22:52:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.099+0000 7ff08ffff640 1 -- 172.21.15.40:0/2098353406 learned_addr learned my addr 172.21.15.40:0/2098353406 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.099+0000 7ff098e10740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c0104b0 con 0x55568f1ef770 2024-03-18T22:52:38.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.100+0000 7ff08ffff640 1 -- 172.21.15.40:0/2098353406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1e9c10 msgr2=0x55568f1e3c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:52:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.100+0000 7ff08ffff640 1 --2- 172.21.15.40:0/2098353406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1e9c10 0x55568f1e3c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.100+0000 7ff08ffff640 1 -- 172.21.15.40:0/2098353406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1ea600 msgr2=0x55568f1fb620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:52:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.100+0000 7ff08ffff640 1 --2- 172.21.15.40:0/2098353406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1ea600 0x55568f1fb620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.100+0000 7ff08ffff640 1 -- 172.21.15.40:0/2098353406 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55568ee220f0 con 0x55568f1ef770 2024-03-18T22:52:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.100+0000 7ff08ffff640 1 --2- 172.21.15.40:0/2098353406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ef770 0x55568f1e4150 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff08000bd30 tx=0x7ff08009aba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c8a45dde1bb27e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.101+0000 7ff0757fa640 1 -- 172.21.15.40:0/2098353406 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0800b2070 con 0x55568f1ef770 2024-03-18T22:52:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.101+0000 7ff0757fa640 1 -- 172.21.15.40:0/2098353406 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0800cbd50 con 0x55568f1ef770 2024-03-18T22:52:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.102+0000 7ff098e10740 1 -- 172.21.15.40:0/2098353406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ef770 msgr2=0x55568f1e4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.102+0000 7ff098e10740 1 --2- 172.21.15.40:0/2098353406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ef770 0x55568f1e4150 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff08000bd30 tx=0x7ff08009aba0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.102+0000 7ff098e10740 1 -- 172.21.15.40:0/2098353406 shutdown_connections 2024-03-18T22:52:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.102+0000 7ff098e10740 1 --2- 172.21.15.40:0/2098353406 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ef770 0x55568f1e4150 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.102+0000 7ff098e10740 1 --2- 172.21.15.40:0/2098353406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1ea600 0x55568f1fb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.102+0000 7ff098e10740 1 --2- 172.21.15.40:0/2098353406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1e9c10 0x55568f1e3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.102+0000 7ff098e10740 1 -- 172.21.15.40:0/2098353406 >> 172.21.15.40:0/2098353406 conn(0x55568f1e5180 msgr2=0x55568f1ebb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.103+0000 7ff098e10740 1 -- 172.21.15.40:0/2098353406 shutdown_connections 2024-03-18T22:52:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.103+0000 7ff098e10740 1 -- 172.21.15.40:0/2098353406 wait complete. 2024-03-18T22:52:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.103+0000 7ff098e10740 1 -- start start 2024-03-18T22:52:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.103+0000 7ff098e10740 1 Processor -- start 2024-03-18T22:52:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.104+0000 7ff098e10740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1e9c10 0x55568f1f8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.104+0000 7ff098e10740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ea600 0x55568f1f8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.104+0000 7ff098e10740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ef770 0x55568f1f9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.104+0000 7ff098e10740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c011310 con 0x55568f1e9c10 2024-03-18T22:52:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.104+0000 7ff098e10740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c010b80 con 0x55568f1ea600 2024-03-18T22:52:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.104+0000 7ff098e10740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c00ddc0 con 0x55568f1ef770 2024-03-18T22:52:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.104+0000 7ff098e10740 10 client.0 _ll_register_callbacks cb 0x7ff088009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.104+0000 7ff08effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ea600 0x55568f1f8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.104+0000 7ff08effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ea600 0x55568f1f8bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:55678/0 (socket says 172.21.15.40:55678) 2024-03-18T22:52:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.104+0000 7ff08effd640 1 -- 172.21.15.40:0/4050247438 learned_addr learned my addr 172.21.15.40:0/4050247438 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.105+0000 7ff08ffff640 1 --2- 172.21.15.40:0/4050247438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ef770 0x55568f1f9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.105+0000 7ff08effd640 1 -- 172.21.15.40:0/4050247438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ef770 msgr2=0x55568f1f9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.105+0000 7ff08effd640 1 --2- 172.21.15.40:0/4050247438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ef770 0x55568f1f9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.105+0000 7ff08effd640 1 -- 172.21.15.40:0/4050247438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1e9c10 msgr2=0x55568f1f8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.105+0000 7ff08effd640 1 --2- 172.21.15.40:0/4050247438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1e9c10 0x55568f1f8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.105+0000 7ff08effd640 1 -- 172.21.15.40:0/4050247438 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55568f1fa1b0 con 0x55568f1ea600 2024-03-18T22:52:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.107+0000 7ff08effd640 1 --2- 172.21.15.40:0/4050247438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ea600 0x55568f1f8bb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff088005fd0 tx=0x7ff088009d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b92f7f3ed5e48f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.107+0000 7ff06effd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:52:38.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.108+0000 7ff06effd640 1 -- 172.21.15.40:0/4050247438 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088021040 con 0x55568f1ea600 2024-03-18T22:52:38.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.108+0000 7ff098e10740 1 -- 172.21.15.40:0/4050247438 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0800cbd50 con 0x55568f1ea600 2024-03-18T22:52:38.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.109+0000 7ff06effd640 1 -- 172.21.15.40:0/4050247438 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff08801ace0 con 0x55568f1ea600 2024-03-18T22:52:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.109+0000 7ff06effd640 1 -- 172.21.15.40:0/4050247438 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff088003970 con 0x55568f1ea600 2024-03-18T22:52:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.109+0000 7ff098e10740 10 client.27842 fetch_fsmap learned FSMap version 36 2024-03-18T22:52:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.109+0000 7ff098e10740 1 -- 172.21.15.40:0/4050247438 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55568f1fb560 con 0x55568f1ea600 2024-03-18T22:52:38.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.113+0000 7ff06effd640 1 -- 172.21.15.40:0/4050247438 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0800cbd50 con 0x55568f1ea600 2024-03-18T22:52:38.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.113+0000 7ff06effd640 1 -- 172.21.15.40:0/4050247438 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 36) ==== 8378+0+0 (secure 0 0 0) 0x7ff08800b050 con 0x55568f1ea600 2024-03-18T22:52:38.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.115+0000 7ff098e10740 10 client.27842 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T22:52:38.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.115+0000 7ff098e10740 10 client.27842 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:38.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.115+0000 7ff098e10740 20 client.27842 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:38.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.115+0000 7ff098e10740 1 --2- 172.21.15.40:0/4050247438 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55568f266a70 0x55568f268e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:38.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.115+0000 7ff098e10740 4 client.27842 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:52:38.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.115+0000 7ff098e10740 1 -- 172.21.15.40:0/4050247438 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55568f155d90 con 0x55568f266a70 2024-03-18T22:52:38.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.115+0000 7ff08f7fe640 1 --2- 172.21.15.40:0/4050247438 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55568f266a70 0x55568f268e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.119+0000 7ff08f7fe640 1 --2- 172.21.15.40:0/4050247438 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55568f266a70 0x55568f268e60 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23c7b75a0a8eb604 server_cookie=70e65b9283742be6 in_seq=0 out_seq=0 2024-03-18T22:52:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.119+0000 7ff06effd640 10 client.27842 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.119+0000 7ff06effd640 1 -- 172.21.15.40:0/4050247438 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55568f155d90 con 0x55568f266a70 2024-03-18T22:52:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.119+0000 7ff06effd640 10 client.27842 handle_command_reply: tid=0 2024-03-18T22:52:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.120+0000 7ff098e10740 1 client.27842 shutdown 2024-03-18T22:52:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.120+0000 7ff098e10740 1 -- 172.21.15.40:0/4050247438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ea600 msgr2=0x55568f1f8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:38.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.120+0000 7ff098e10740 1 --2- 172.21.15.40:0/4050247438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ea600 0x55568f1f8bb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff088005fd0 tx=0x7ff088009d70 comp rx=0 tx=0).stop 2024-03-18T22:52:38.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.120+0000 7ff098e10740 1 -- 172.21.15.40:0/4050247438 shutdown_connections 2024-03-18T22:52:38.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.120+0000 7ff098e10740 1 --2- 172.21.15.40:0/4050247438 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55568f1ef770 0x55568f1f9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.120+0000 7ff098e10740 1 --2- 172.21.15.40:0/4050247438 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55568f266a70 0x55568f268e60 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.120+0000 7ff098e10740 1 --2- 172.21.15.40:0/4050247438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55568f1ea600 0x55568f1f8bb0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.120+0000 7ff098e10740 1 --2- 172.21.15.40:0/4050247438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55568f1e9c10 0x55568f1f8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.120+0000 7ff098e10740 1 -- 172.21.15.40:0/4050247438 >> 172.21.15.40:0/4050247438 conn(0x55568f1e5180 msgr2=0x55568f1e83e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:38.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.120+0000 7ff06effd640 0 client.27842 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:38.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.120+0000 7ff098e10740 1 -- 172.21.15.40:0/4050247438 shutdown_connections 2024-03-18T22:52:38.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.120+0000 7ff098e10740 1 -- 172.21.15.40:0/4050247438 wait complete. 2024-03-18T22:52:38.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.121+0000 7ff098e10740 20 client.27842 trim_cache size 0 max 16384 2024-03-18T22:52:38.124 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:52:38.124 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:52:38.124 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:52:38.124 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:52:38.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.122+0000 7ff096152640 1 -- 172.21.15.40:0/3327382198 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff0780a2a00 msgr2=0x7ff0780a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:38.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.122+0000 7ff096152640 1 --2- 172.21.15.40:0/3327382198 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff0780a2a00 0x7ff0780a4ec0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff088005c40 tx=0x7ff08801b000 comp rx=0 tx=0).stop 2024-03-18T22:52:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.122+0000 7ff096152640 1 -- 172.21.15.40:0/3327382198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0901098d0 msgr2=0x7ff0901adde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.122+0000 7ff096152640 1 --2- 172.21.15.40:0/3327382198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0901098d0 0x7ff0901adde0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff08000b3e0 tx=0x7ff080009f90 comp rx=0 tx=0).stop 2024-03-18T22:52:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.122+0000 7ff096152640 1 -- 172.21.15.40:0/3327382198 shutdown_connections 2024-03-18T22:52:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.122+0000 7ff096152640 1 --2- 172.21.15.40:0/3327382198 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff0780a2a00 0x7ff0780a4ec0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.122+0000 7ff096152640 1 --2- 172.21.15.40:0/3327382198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff0901098d0 0x7ff0901adde0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.123+0000 7ff096152640 1 --2- 172.21.15.40:0/3327382198 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff090108f30 0x7ff0901a6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.123+0000 7ff096152640 1 --2- 172.21.15.40:0/3327382198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff090072c00 0x7ff0901a6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.123+0000 7ff096152640 1 -- 172.21.15.40:0/3327382198 >> 172.21.15.40:0/3327382198 conn(0x7ff09006e700 msgr2=0x7ff090070400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.123+0000 7ff096152640 1 -- 172.21.15.40:0/3327382198 shutdown_connections 2024-03-18T22:52:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.123+0000 7ff096152640 1 -- 172.21.15.40:0/3327382198 wait complete. 2024-03-18T22:52:38.160 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:52:38.161 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:52:38.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.310+0000 7f576b368640 1 -- 172.21.15.40:0/2342994820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5764101860 msgr2=0x7f5764101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:38.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.310+0000 7f576b368640 1 --2- 172.21.15.40:0/2342994820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5764101860 0x7f5764101c80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f575c009900 tx=0x7f575c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:52:38.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f576b368640 1 -- 172.21.15.40:0/2342994820 shutdown_connections 2024-03-18T22:52:38.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f576b368640 1 --2- 172.21.15.40:0/2342994820 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5764103410 0x7f5764109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f576b368640 1 --2- 172.21.15.40:0/2342994820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5764102a50 0x7f5764102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f576b368640 1 --2- 172.21.15.40:0/2342994820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5764101860 0x7f5764101c80 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f576b368640 1 -- 172.21.15.40:0/2342994820 >> 172.21.15.40:0/2342994820 conn(0x7f57640fcdd0 msgr2=0x7f57640ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f576b368640 1 -- 172.21.15.40:0/2342994820 shutdown_connections 2024-03-18T22:52:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f576b368640 1 -- 172.21.15.40:0/2342994820 wait complete. 2024-03-18T22:52:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f576b368640 1 Processor -- start 2024-03-18T22:52:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f576b368640 1 -- start start 2024-03-18T22:52:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f576b368640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5764101860 0x7f5764197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f576b368640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5764102a50 0x7f5764198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f576b368640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5764103410 0x7f576419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f576b368640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f576410d360 con 0x7f5764102a50 2024-03-18T22:52:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f576b368640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f576410d1e0 con 0x7f5764103410 2024-03-18T22:52:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f576b368640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f576410d4e0 con 0x7f5764101860 2024-03-18T22:52:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f5768903640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5764102a50 0x7f5764198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f5768903640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5764102a50 0x7f5764198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38390/0 (socket says 172.21.15.40:38390) 2024-03-18T22:52:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f5768903640 1 -- 172.21.15.40:0/3437214904 learned_addr learned my addr 172.21.15.40:0/3437214904 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.311+0000 7f5769104640 1 --2- 172.21.15.40:0/3437214904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5764101860 0x7f5764197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.312+0000 7f5769905640 1 --2- 172.21.15.40:0/3437214904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5764103410 0x7f576419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.312+0000 7f5768903640 1 -- 172.21.15.40:0/3437214904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5764101860 msgr2=0x7f5764197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.312+0000 7f5768903640 1 --2- 172.21.15.40:0/3437214904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5764101860 0x7f5764197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.312+0000 7f5768903640 1 -- 172.21.15.40:0/3437214904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5764103410 msgr2=0x7f576419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.312+0000 7f5768903640 1 --2- 172.21.15.40:0/3437214904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5764103410 0x7f576419f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.312+0000 7f5768903640 1 -- 172.21.15.40:0/3437214904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f576419f700 con 0x7f5764102a50 2024-03-18T22:52:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.312+0000 7f5768903640 1 --2- 172.21.15.40:0/3437214904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5764102a50 0x7f5764198020 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f575800e9f0 tx=0x7f575800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.312+0000 7f57627fc640 1 -- 172.21.15.40:0/3437214904 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575800cd80 con 0x7f5764102a50 2024-03-18T22:52:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.312+0000 7f57627fc640 1 -- 172.21.15.40:0/3437214904 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57580108e0 con 0x7f5764102a50 2024-03-18T22:52:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.312+0000 7f576b368640 1 -- 172.21.15.40:0/3437214904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f576419f9f0 con 0x7f5764102a50 2024-03-18T22:52:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.312+0000 7f57627fc640 1 -- 172.21.15.40:0/3437214904 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758018920 con 0x7f5764102a50 2024-03-18T22:52:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.312+0000 7f576b368640 1 -- 172.21.15.40:0/3437214904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f576419ff40 con 0x7f5764102a50 2024-03-18T22:52:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.313+0000 7f576b368640 1 -- 172.21.15.40:0/3437214904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5764108c50 con 0x7f5764102a50 2024-03-18T22:52:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.314+0000 7f57627fc640 1 -- 172.21.15.40:0/3437214904 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5758010a90 con 0x7f5764102a50 2024-03-18T22:52:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.315+0000 7f57627fc640 1 --2- 172.21.15.40:0/3437214904 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f574c0a2800 0x7f574c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.315+0000 7f5769104640 1 --2- 172.21.15.40:0/3437214904 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f574c0a2800 0x7f574c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.315+0000 7f57627fc640 1 -- 172.21.15.40:0/3437214904 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f57580cf140 con 0x7f5764102a50 2024-03-18T22:52:38.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.315+0000 7f5769104640 1 --2- 172.21.15.40:0/3437214904 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f574c0a2800 0x7f574c0a4cc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f575c005ab0 tx=0x7f575c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:38.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.319+0000 7f57627fc640 1 -- 172.21.15.40:0/3437214904 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f575809b7a0 con 0x7f5764102a50 2024-03-18T22:52:38.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:38 smithi103 ceph-mon[66377]: pgmap v650: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 19 MiB/s wr, 4.81k op/s 2024-03-18T22:52:38.647 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:38 smithi040 ceph-mon[62643]: pgmap v650: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 19 MiB/s wr, 4.81k op/s 2024-03-18T22:52:38.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.762+0000 7f576b368640 1 -- 172.21.15.40:0/3437214904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f576402d1f0 con 0x7f5764102a50 2024-03-18T22:52:38.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.763+0000 7f57627fc640 1 -- 172.21.15.40:0/3437214904 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11954 (secure 0 0 0) 0x7f575809f840 con 0x7f5764102a50 2024-03-18T22:52:38.765 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:52:38.766 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:52:09.217946+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24388,24401,24473,24481,24509]},"id":1}]} 2024-03-18T22:52:38.766 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 36 2024-03-18T22:52:38.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.769+0000 7f576b368640 1 -- 172.21.15.40:0/3437214904 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f574c0a2800 msgr2=0x7f574c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:38.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.769+0000 7f576b368640 1 --2- 172.21.15.40:0/3437214904 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f574c0a2800 0x7f574c0a4cc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f575c005ab0 tx=0x7f575c005fd0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.769+0000 7f576b368640 1 -- 172.21.15.40:0/3437214904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5764102a50 msgr2=0x7f5764198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:38.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.769+0000 7f576b368640 1 --2- 172.21.15.40:0/3437214904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5764102a50 0x7f5764198020 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f575800e9f0 tx=0x7f575800eeb0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.774+0000 7f576b368640 1 -- 172.21.15.40:0/3437214904 shutdown_connections 2024-03-18T22:52:38.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.774+0000 7f576b368640 1 --2- 172.21.15.40:0/3437214904 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f574c0a2800 0x7f574c0a4cc0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.774+0000 7f576b368640 1 --2- 172.21.15.40:0/3437214904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5764103410 0x7f576419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.774+0000 7f576b368640 1 --2- 172.21.15.40:0/3437214904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5764102a50 0x7f5764198020 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.774+0000 7f576b368640 1 --2- 172.21.15.40:0/3437214904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5764101860 0x7f5764197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:38.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.774+0000 7f576b368640 1 -- 172.21.15.40:0/3437214904 >> 172.21.15.40:0/3437214904 conn(0x7f57640fcdd0 msgr2=0x7f5764105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:38.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.778+0000 7f576b368640 1 -- 172.21.15.40:0/3437214904 shutdown_connections 2024-03-18T22:52:38.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:38.778+0000 7f576b368640 1 -- 172.21.15.40:0/3437214904 wait complete. 2024-03-18T22:52:38.819 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:52:38.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:38 smithi113 ceph-mon[66775]: pgmap v650: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 19 MiB/s wr, 4.81k op/s 2024-03-18T22:52:39.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.051+0000 7fcb5d027640 1 -- 172.21.15.40:0/53742373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb58100720 msgr2=0x7fcb58100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.051+0000 7fcb5d027640 1 --2- 172.21.15.40:0/53742373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb58100720 0x7fcb58100b40 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009900 tx=0x7fcb4c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:52:39.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.051+0000 7fcb5d027640 1 -- 172.21.15.40:0/53742373 shutdown_connections 2024-03-18T22:52:39.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.051+0000 7fcb5d027640 1 --2- 172.21.15.40:0/53742373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb581022d0 0x7fcb58108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.051+0000 7fcb5d027640 1 --2- 172.21.15.40:0/53742373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb58101910 0x7fcb58101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.051+0000 7fcb5d027640 1 --2- 172.21.15.40:0/53742373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb58100720 0x7fcb58100b40 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.051+0000 7fcb5d027640 1 -- 172.21.15.40:0/53742373 >> 172.21.15.40:0/53742373 conn(0x7fcb580fbc90 msgr2=0x7fcb580fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.051+0000 7fcb5d027640 1 -- 172.21.15.40:0/53742373 shutdown_connections 2024-03-18T22:52:39.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.051+0000 7fcb5d027640 1 -- 172.21.15.40:0/53742373 wait complete. 2024-03-18T22:52:39.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.052+0000 7fcb5d027640 1 Processor -- start 2024-03-18T22:52:39.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.052+0000 7fcb5d027640 1 -- start start 2024-03-18T22:52:39.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb5d027640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb58100720 0x7fcb58196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb5d027640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb58101910 0x7fcb58196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb5d027640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb581022d0 0x7fcb5819df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb5d027640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5810c240 con 0x7fcb58100720 2024-03-18T22:52:39.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb5d027640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5810c0c0 con 0x7fcb581022d0 2024-03-18T22:52:39.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb56d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb58100720 0x7fcb58196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb5d027640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5810c3c0 con 0x7fcb58101910 2024-03-18T22:52:39.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb56d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb58100720 0x7fcb58196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32768/0 (socket says 172.21.15.40:32768) 2024-03-18T22:52:39.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb56d9d640 1 -- 172.21.15.40:0/2670012523 learned_addr learned my addr 172.21.15.40:0/2670012523 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:39.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb5659c640 1 --2- 172.21.15.40:0/2670012523 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb58101910 0x7fcb58196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb5759e640 1 --2- 172.21.15.40:0/2670012523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb581022d0 0x7fcb5819df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb56d9d640 1 -- 172.21.15.40:0/2670012523 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb58101910 msgr2=0x7fcb58196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb56d9d640 1 --2- 172.21.15.40:0/2670012523 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb58101910 0x7fcb58196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb56d9d640 1 -- 172.21.15.40:0/2670012523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb581022d0 msgr2=0x7fcb5819df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb56d9d640 1 --2- 172.21.15.40:0/2670012523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb581022d0 0x7fcb5819df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb56d9d640 1 -- 172.21.15.40:0/2670012523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5819e5b0 con 0x7fcb58100720 2024-03-18T22:52:39.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb56d9d640 1 --2- 172.21.15.40:0/2670012523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb58100720 0x7fcb58196990 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c000c00 tx=0x7fcb4c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb3ffff640 1 -- 172.21.15.40:0/2670012523 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c043020 con 0x7fcb58100720 2024-03-18T22:52:39.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb3ffff640 1 -- 172.21.15.40:0/2670012523 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb4c004440 con 0x7fcb58100720 2024-03-18T22:52:39.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb5d027640 1 -- 172.21.15.40:0/2670012523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb5819e840 con 0x7fcb58100720 2024-03-18T22:52:39.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb3ffff640 1 -- 172.21.15.40:0/2670012523 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c003c90 con 0x7fcb58100720 2024-03-18T22:52:39.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.053+0000 7fcb5d027640 1 -- 172.21.15.40:0/2670012523 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb5819ed90 con 0x7fcb58100720 2024-03-18T22:52:39.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb3ffff640 1 -- 172.21.15.40:0/2670012523 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcb4c046020 con 0x7fcb58100720 2024-03-18T22:52:39.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb5fc15740 1 Processor -- start 2024-03-18T22:52:39.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb5fc15740 1 -- start start 2024-03-18T22:52:39.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb5fc15740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 0x55f45a781920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb5fc15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a781ef0 0x55f45a7869f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb5fc15740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a787060 0x55f45a7894f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb5fc15740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f45a6e34b0 con 0x55f45a781ef0 2024-03-18T22:52:39.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb5fc15740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f45a68bd70 con 0x55f45a787060 2024-03-18T22:52:39.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb5fc15740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f45a6df550 con 0x55f45a781500 2024-03-18T22:52:39.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb3ffff640 1 --2- 172.21.15.40:0/2670012523 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcb400a2800 0x7fcb400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb56d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a781ef0 0x55f45a7869f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb5759e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 0x55f45a781920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb56d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a781ef0 0x55f45a7869f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32782/0 (socket says 172.21.15.40:32782) 2024-03-18T22:52:39.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb56d9d640 1 -- 172.21.15.40:0/2745476268 learned_addr learned my addr 172.21.15.40:0/2745476268 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:39.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb5659c640 1 --2- 172.21.15.40:0/2745476268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a787060 0x55f45a7894f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb3ffff640 1 -- 172.21.15.40:0/2670012523 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fcb4c0f1060 con 0x7fcb58100720 2024-03-18T22:52:39.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.056+0000 7fcb5759e640 1 --2- 172.21.15.40:0/2670012523 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcb400a2800 0x7fcb400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.057+0000 7fcb5659c640 1 -- 172.21.15.40:0/2745476268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 msgr2=0x55f45a781920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.057+0000 7fcb5659c640 1 --2- 172.21.15.40:0/2745476268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 0x55f45a781920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.057+0000 7fcb5659c640 1 -- 172.21.15.40:0/2745476268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a781ef0 msgr2=0x55f45a7869f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.057+0000 7fcb5659c640 1 --2- 172.21.15.40:0/2745476268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a781ef0 0x55f45a7869f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.057+0000 7fcb5659c640 1 -- 172.21.15.40:0/2745476268 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f45a6ae210 con 0x55f45a787060 2024-03-18T22:52:39.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.057+0000 7fcb5659c640 1 --2- 172.21.15.40:0/2745476268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a787060 0x55f45a7894f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009bd0 tx=0x7fcb48011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a76610eb9e2c1f79 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.057+0000 7fcb5759e640 1 --2- 172.21.15.40:0/2670012523 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcb400a2800 0x7fcb400a4cc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x55f45a782750 tx=0x7fcb50004ab0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.057+0000 7fcb3d7fa640 1 -- 172.21.15.40:0/2745476268 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb48004310 con 0x55f45a787060 2024-03-18T22:52:39.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.057+0000 7fcb3d7fa640 1 -- 172.21.15.40:0/2745476268 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb480044b0 con 0x55f45a787060 2024-03-18T22:52:39.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.057+0000 7fcb3d7fa640 1 -- 172.21.15.40:0/2745476268 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb480035f0 con 0x55f45a787060 2024-03-18T22:52:39.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 -- 172.21.15.40:0/2745476268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a787060 msgr2=0x55f45a7894f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/2745476268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a787060 0x55f45a7894f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcb48009bd0 tx=0x7fcb48011440 comp rx=0 tx=0).stop 2024-03-18T22:52:39.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 -- 172.21.15.40:0/2745476268 shutdown_connections 2024-03-18T22:52:39.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/2745476268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a787060 0x55f45a7894f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/2745476268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a781ef0 0x55f45a7869f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/2745476268 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 0x55f45a781920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 -- 172.21.15.40:0/2745476268 >> 172.21.15.40:0/2745476268 conn(0x55f45a77a860 msgr2=0x55f45a77ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 -- 172.21.15.40:0/2745476268 shutdown_connections 2024-03-18T22:52:39.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 -- 172.21.15.40:0/2745476268 wait complete. 2024-03-18T22:52:39.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 -- start start 2024-03-18T22:52:39.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 Processor -- start 2024-03-18T22:52:39.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a781500 0x55f45a78b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a781ef0 0x55f45a78bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a787060 0x55f45a792f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f45a78a010 con 0x55f45a781500 2024-03-18T22:52:39.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f45a69f310 con 0x55f45a781ef0 2024-03-18T22:52:39.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f45a78a190 con 0x55f45a787060 2024-03-18T22:52:39.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5fc15740 10 client.0 _ll_register_callbacks cb 0x55f45a6932d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:39.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb56d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a781500 0x55f45a78b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb56d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a781500 0x55f45a78b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32788/0 (socket says 172.21.15.40:32788) 2024-03-18T22:52:39.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb56d9d640 1 -- 172.21.15.40:0/4233274906 learned_addr learned my addr 172.21.15.40:0/4233274906 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:39.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.058+0000 7fcb5759e640 1 --2- 172.21.15.40:0/4233274906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a787060 0x55f45a792f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.059+0000 7fcb5659c640 1 --2- 172.21.15.40:0/4233274906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a781ef0 0x55f45a78bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.059+0000 7fcb56d9d640 1 -- 172.21.15.40:0/4233274906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a787060 msgr2=0x55f45a792f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.059+0000 7fcb56d9d640 1 --2- 172.21.15.40:0/4233274906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a787060 0x55f45a792f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.059+0000 7fcb56d9d640 1 -- 172.21.15.40:0/4233274906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a781ef0 msgr2=0x55f45a78bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.059+0000 7fcb56d9d640 1 --2- 172.21.15.40:0/4233274906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a781ef0 0x55f45a78bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.059+0000 7fcb56d9d640 1 -- 172.21.15.40:0/4233274906 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f45a3f73f0 con 0x55f45a781500 2024-03-18T22:52:39.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.059+0000 7fcb56d9d640 1 --2- 172.21.15.40:0/4233274906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a781500 0x55f45a78b9f0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c0b9a20 tx=0x7fcb4c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36417094ab878506 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.059+0000 7fcb3aff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:39.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.059+0000 7fcb3aff5640 1 -- 172.21.15.40:0/4233274906 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c0c5b20 con 0x55f45a781500 2024-03-18T22:52:39.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.059+0000 7fcb3aff5640 1 -- 172.21.15.40:0/4233274906 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fcb4c0f30d0 con 0x55f45a781500 2024-03-18T22:52:39.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.059+0000 7fcb5fc15740 1 -- 172.21.15.40:0/4233274906 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb480044b0 con 0x55f45a781500 2024-03-18T22:52:39.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.059+0000 7fcb3aff5640 1 -- 172.21.15.40:0/4233274906 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcb4c0035f0 con 0x55f45a781500 2024-03-18T22:52:39.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.059+0000 7fcb5fc15740 10 client.18045 fetch_fsmap learned FSMap version 36 2024-03-18T22:52:39.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.059+0000 7fcb5fc15740 1 -- 172.21.15.40:0/4233274906 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55f45a4c17d0 con 0x55f45a781500 2024-03-18T22:52:39.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.059+0000 7fcb3aff5640 1 -- 172.21.15.40:0/4233274906 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb480044b0 con 0x55f45a781500 2024-03-18T22:52:39.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.060+0000 7fcb3aff5640 1 -- 172.21.15.40:0/4233274906 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 36) ==== 8378+0+0 (secure 0 0 0) 0x7fcb4c0a6ba0 con 0x55f45a781500 2024-03-18T22:52:39.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.060+0000 7fcb5fc15740 10 client.18045 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T22:52:39.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.060+0000 7fcb5fc15740 10 client.18045 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:52:39.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.060+0000 7fcb5fc15740 20 client.18045 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:39.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.060+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/4233274906 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f45a7978d0 0x55f45a799d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.060+0000 7fcb5fc15740 4 client.18045 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:39.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.060+0000 7fcb5fc15740 1 -- 172.21.15.40:0/4233274906 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f45a7413e0 con 0x55f45a7978d0 2024-03-18T22:52:39.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.060+0000 7fcb5659c640 1 --2- 172.21.15.40:0/4233274906 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f45a7978d0 0x55f45a799d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.060+0000 7fcb5659c640 1 --2- 172.21.15.40:0/4233274906 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f45a7978d0 0x55f45a799d60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ee6d78b1706b7e83 server_cookie=371dde2a3752b39f in_seq=0 out_seq=0 2024-03-18T22:52:39.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.060+0000 7fcb3aff5640 10 client.18045 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:52:39.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.061+0000 7fcb3aff5640 1 -- 172.21.15.40:0/4233274906 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f45a7413e0 con 0x55f45a7978d0 2024-03-18T22:52:39.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.061+0000 7fcb3aff5640 10 client.18045 handle_command_reply: tid=0 2024-03-18T22:52:39.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.061+0000 7fcb5fc15740 1 client.18045 shutdown 2024-03-18T22:52:39.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.061+0000 7fcb5fc15740 1 -- 172.21.15.40:0/4233274906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a781500 msgr2=0x55f45a78b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.061+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/4233274906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a781500 0x55f45a78b9f0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c0b9a20 tx=0x7fcb4c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:52:39.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.061+0000 7fcb5fc15740 1 -- 172.21.15.40:0/4233274906 shutdown_connections 2024-03-18T22:52:39.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.061+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/4233274906 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a787060 0x55f45a792f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.061+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/4233274906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a781ef0 0x55f45a78bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.061+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/4233274906 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f45a7978d0 0x55f45a799d60 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.061+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/4233274906 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a781500 0x55f45a78b9f0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.061+0000 7fcb5fc15740 1 -- 172.21.15.40:0/4233274906 >> 172.21.15.40:0/4233274906 conn(0x55f45a77a860 msgr2=0x55f45a779ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.061+0000 7fcb3aff5640 0 client.18045 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:52:39.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.061+0000 7fcb5fc15740 1 -- 172.21.15.40:0/4233274906 shutdown_connections 2024-03-18T22:52:39.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.061+0000 7fcb5fc15740 1 -- 172.21.15.40:0/4233274906 wait complete. 2024-03-18T22:52:39.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.061+0000 7fcb5fc15740 20 client.18045 trim_cache size 0 max 16384 2024-03-18T22:52:39.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.074+0000 7fcb5fc15740 1 Processor -- start 2024-03-18T22:52:39.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.074+0000 7fcb5fc15740 1 -- start start 2024-03-18T22:52:39.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.074+0000 7fcb5fc15740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 0x55f45a78be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.074+0000 7fcb5fc15740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a781ef0 0x55f45a792f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.074+0000 7fcb5fc15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a787060 0x55f45a782860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.074+0000 7fcb5fc15740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcb581a82c0 con 0x55f45a787060 2024-03-18T22:52:39.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.074+0000 7fcb5fc15740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcb581a9ec0 con 0x55f45a781ef0 2024-03-18T22:52:39.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.074+0000 7fcb5fc15740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fcb581a8990 con 0x55f45a781500 2024-03-18T22:52:39.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.074+0000 7fcb5759e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a787060 0x55f45a782860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.075+0000 7fcb56d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 0x55f45a78be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.075+0000 7fcb5759e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a787060 0x55f45a782860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32796/0 (socket says 172.21.15.40:32796) 2024-03-18T22:52:39.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.075+0000 7fcb5759e640 1 -- 172.21.15.40:0/474626391 learned_addr learned my addr 172.21.15.40:0/474626391 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.075+0000 7fcb5659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a781ef0 0x55f45a792f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.075+0000 7fcb5759e640 1 -- 172.21.15.40:0/474626391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 msgr2=0x55f45a78be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.075+0000 7fcb5759e640 1 --2- 172.21.15.40:0/474626391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 0x55f45a78be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.075+0000 7fcb5759e640 1 -- 172.21.15.40:0/474626391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a781ef0 msgr2=0x55f45a792f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.075+0000 7fcb5759e640 1 --2- 172.21.15.40:0/474626391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a781ef0 0x55f45a792f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.075+0000 7fcb5759e640 1 -- 172.21.15.40:0/474626391 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f45a750880 con 0x55f45a787060 2024-03-18T22:52:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.075+0000 7fcb5759e640 1 --2- 172.21.15.40:0/474626391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a787060 0x55f45a782860 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fcb500126f0 tx=0x7fcb50009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d713756ba0b960ad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.075+0000 7fcb3cff9640 1 -- 172.21.15.40:0/474626391 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb50027070 con 0x55f45a787060 2024-03-18T22:52:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.075+0000 7fcb3cff9640 1 -- 172.21.15.40:0/474626391 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb50011ba0 con 0x55f45a787060 2024-03-18T22:52:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.075+0000 7fcb3cff9640 1 -- 172.21.15.40:0/474626391 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb50023c50 con 0x55f45a787060 2024-03-18T22:52:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 -- 172.21.15.40:0/474626391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a787060 msgr2=0x55f45a782860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/474626391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a787060 0x55f45a782860 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fcb500126f0 tx=0x7fcb50009920 comp rx=0 tx=0).stop 2024-03-18T22:52:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 -- 172.21.15.40:0/474626391 shutdown_connections 2024-03-18T22:52:39.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/474626391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a787060 0x55f45a782860 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/474626391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a781ef0 0x55f45a792f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/474626391 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 0x55f45a78be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 -- 172.21.15.40:0/474626391 >> 172.21.15.40:0/474626391 conn(0x55f45a77a860 msgr2=0x55f45a775a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 -- 172.21.15.40:0/474626391 shutdown_connections 2024-03-18T22:52:39.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 -- 172.21.15.40:0/474626391 wait complete. 2024-03-18T22:52:39.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 -- start start 2024-03-18T22:52:39.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 Processor -- start 2024-03-18T22:52:39.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 0x55f45a7904f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a781ef0 0x55f45a790a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a787060 0x55f45a7919d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fcb581a97f0 con 0x55f45a787060 2024-03-18T22:52:39.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcb581a9060 con 0x55f45a781ef0 2024-03-18T22:52:39.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fcb581a62a0 con 0x55f45a781500 2024-03-18T22:52:39.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5fc15740 10 client.0 _ll_register_callbacks cb 0x7fcb4c0bd3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:39.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5759e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a787060 0x55f45a7919d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb56d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 0x55f45a7904f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb56d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 0x55f45a7904f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53584/0 (socket says 172.21.15.40:53584) 2024-03-18T22:52:39.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5759e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a787060 0x55f45a7919d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32806/0 (socket says 172.21.15.40:32806) 2024-03-18T22:52:39.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb56d9d640 1 -- 172.21.15.40:0/2777497207 learned_addr learned my addr 172.21.15.40:0/2777497207 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:39.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.076+0000 7fcb5659c640 1 --2- 172.21.15.40:0/2777497207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a781ef0 0x55f45a790a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.077+0000 7fcb56d9d640 1 -- 172.21.15.40:0/2777497207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a781ef0 msgr2=0x55f45a790a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.077+0000 7fcb56d9d640 1 --2- 172.21.15.40:0/2777497207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a781ef0 0x55f45a790a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.077+0000 7fcb56d9d640 1 -- 172.21.15.40:0/2777497207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a787060 msgr2=0x55f45a7919d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.077+0000 7fcb56d9d640 1 --2- 172.21.15.40:0/2777497207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a787060 0x55f45a7919d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.077+0000 7fcb56d9d640 1 -- 172.21.15.40:0/2777497207 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f45a791fc0 con 0x55f45a781500 2024-03-18T22:52:39.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.077+0000 7fcb5759e640 1 --2- 172.21.15.40:0/2777497207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a787060 0x55f45a7919d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:52:39.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.077+0000 7fcb56d9d640 1 --2- 172.21.15.40:0/2777497207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 0x55f45a7904f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c0bb8c0 tx=0x7fcb4c0057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e090d26505aee47 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.077+0000 7fcb3aff5640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:52:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.077+0000 7fcb3aff5640 1 -- 172.21.15.40:0/2777497207 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c0bd150 con 0x55f45a781500 2024-03-18T22:52:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.077+0000 7fcb5fc15740 1 -- 172.21.15.40:0/2777497207 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb50011ba0 con 0x55f45a781500 2024-03-18T22:52:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.077+0000 7fcb3aff5640 1 -- 172.21.15.40:0/2777497207 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fcb4c0bca60 con 0x55f45a781500 2024-03-18T22:52:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.077+0000 7fcb3aff5640 1 -- 172.21.15.40:0/2777497207 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcb4c0bd640 con 0x55f45a781500 2024-03-18T22:52:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.077+0000 7fcb5fc15740 10 client.27850 fetch_fsmap learned FSMap version 36 2024-03-18T22:52:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.077+0000 7fcb5fc15740 1 -- 172.21.15.40:0/2777497207 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55f45a7fdca0 con 0x55f45a781500 2024-03-18T22:52:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.077+0000 7fcb3aff5640 1 -- 172.21.15.40:0/2777497207 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb50011ba0 con 0x55f45a781500 2024-03-18T22:52:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb3aff5640 1 -- 172.21.15.40:0/2777497207 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 36) ==== 8378+0+0 (secure 0 0 0) 0x7fcb4c0bc4d0 con 0x55f45a781500 2024-03-18T22:52:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 10 client.27850 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T22:52:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 10 client.27850 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:52:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 20 client.27850 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/2777497207 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f45a7fe2b0 0x55f45a8006a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 4 client.27850 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:52:39.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 1 -- 172.21.15.40:0/2777497207 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f45a6f09f0 con 0x55f45a7fe2b0 2024-03-18T22:52:39.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5659c640 1 --2- 172.21.15.40:0/2777497207 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f45a7fe2b0 0x55f45a8006a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb3aff5640 10 client.27850 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:52:39.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5659c640 1 --2- 172.21.15.40:0/2777497207 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f45a7fe2b0 0x55f45a8006a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d493da579b7b4e55 server_cookie=94548a9d2360cc13 in_seq=0 out_seq=0 2024-03-18T22:52:39.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb3aff5640 1 -- 172.21.15.40:0/2777497207 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f45a6f09f0 con 0x55f45a7fe2b0 2024-03-18T22:52:39.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb3aff5640 10 client.27850 handle_command_reply: tid=0 2024-03-18T22:52:39.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 1 client.27850 shutdown 2024-03-18T22:52:39.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 1 -- 172.21.15.40:0/2777497207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 msgr2=0x55f45a7904f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/2777497207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 0x55f45a7904f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c0bb8c0 tx=0x7fcb4c0057c0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 1 -- 172.21.15.40:0/2777497207 shutdown_connections 2024-03-18T22:52:39.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/2777497207 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55f45a7fe2b0 0x55f45a8006a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/2777497207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f45a787060 0x55f45a7919d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/2777497207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f45a781ef0 0x55f45a790a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 1 --2- 172.21.15.40:0/2777497207 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f45a781500 0x55f45a7904f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb3aff5640 0 client.27850 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:52:39.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 1 -- 172.21.15.40:0/2777497207 >> 172.21.15.40:0/2777497207 conn(0x55f45a77a860 msgr2=0x55f45a782840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 1 -- 172.21.15.40:0/2777497207 shutdown_connections 2024-03-18T22:52:39.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.078+0000 7fcb5fc15740 1 -- 172.21.15.40:0/2777497207 wait complete. 2024-03-18T22:52:39.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.079+0000 7fcb5fc15740 20 client.27850 trim_cache size 0 max 16384 2024-03-18T22:52:39.088 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:52:39.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.079+0000 7fcb5d027640 1 -- 172.21.15.40:0/2670012523 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcb400a2800 msgr2=0x7fcb400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.079+0000 7fcb5d027640 1 --2- 172.21.15.40:0/2670012523 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcb400a2800 0x7fcb400a4cc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x55f45a782750 tx=0x7fcb50004ab0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.079+0000 7fcb5d027640 1 -- 172.21.15.40:0/2670012523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb58100720 msgr2=0x7fcb58196990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.079+0000 7fcb5d027640 1 --2- 172.21.15.40:0/2670012523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb58100720 0x7fcb58196990 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c000c00 tx=0x7fcb4c004040 comp rx=0 tx=0).stop 2024-03-18T22:52:39.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.079+0000 7fcb5d027640 1 -- 172.21.15.40:0/2670012523 shutdown_connections 2024-03-18T22:52:39.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.079+0000 7fcb5d027640 1 --2- 172.21.15.40:0/2670012523 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fcb400a2800 0x7fcb400a4cc0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.079+0000 7fcb5d027640 1 --2- 172.21.15.40:0/2670012523 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb581022d0 0x7fcb5819df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.079+0000 7fcb5d027640 1 --2- 172.21.15.40:0/2670012523 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fcb58101910 0x7fcb58196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.079+0000 7fcb5d027640 1 --2- 172.21.15.40:0/2670012523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcb58100720 0x7fcb58196990 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.079+0000 7fcb5d027640 1 -- 172.21.15.40:0/2670012523 >> 172.21.15.40:0/2670012523 conn(0x7fcb580fbc90 msgr2=0x7fcb58104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.079+0000 7fcb5d027640 1 -- 172.21.15.40:0/2670012523 shutdown_connections 2024-03-18T22:52:39.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.079+0000 7fcb5d027640 1 -- 172.21.15.40:0/2670012523 wait complete. 2024-03-18T22:52:39.091 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:52:39.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.194+0000 7f46c6659640 1 -- 172.21.15.40:0/594006993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c0102a10 msgr2=0x7f46c0102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.194+0000 7f46c6659640 1 --2- 172.21.15.40:0/594006993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c0102a10 0x7f46c0102e90 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f46b4009900 tx=0x7f46b402bc30 comp rx=0 tx=0).stop 2024-03-18T22:52:39.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.194+0000 7f46c6659640 1 -- 172.21.15.40:0/594006993 shutdown_connections 2024-03-18T22:52:39.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.194+0000 7f46c6659640 1 --2- 172.21.15.40:0/594006993 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46c01033d0 0x7f46c0109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.194+0000 7f46c6659640 1 --2- 172.21.15.40:0/594006993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c0102a10 0x7f46c0102e90 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.194+0000 7f46c6659640 1 --2- 172.21.15.40:0/594006993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46c0101820 0x7f46c0101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.194+0000 7f46c6659640 1 -- 172.21.15.40:0/594006993 >> 172.21.15.40:0/594006993 conn(0x7f46c00fcd90 msgr2=0x7f46c00ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.194+0000 7f46c6659640 1 -- 172.21.15.40:0/594006993 shutdown_connections 2024-03-18T22:52:39.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.194+0000 7f46c6659640 1 -- 172.21.15.40:0/594006993 wait complete. 2024-03-18T22:52:39.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.194+0000 7f46c6659640 1 Processor -- start 2024-03-18T22:52:39.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46c6659640 1 -- start start 2024-03-18T22:52:39.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46c6659640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46c0101820 0x7f46c0197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46c6659640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46c0102a10 0x7f46c0197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46c6659640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c01033d0 0x7f46c019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46c6659640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f46c010d350 con 0x7f46c01033d0 2024-03-18T22:52:39.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46c6659640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f46c010d1d0 con 0x7f46c0102a10 2024-03-18T22:52:39.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46c6659640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f46c010d4d0 con 0x7f46c0101820 2024-03-18T22:52:39.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46c4bf6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c01033d0 0x7f46c019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46bffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46c0101820 0x7f46c0197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46bffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46c0101820 0x7f46c0197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53592/0 (socket says 172.21.15.40:53592) 2024-03-18T22:52:39.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46c4bf6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c01033d0 0x7f46c019f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32830/0 (socket says 172.21.15.40:32830) 2024-03-18T22:52:39.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46bf7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46c0102a10 0x7f46c0197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46bffff640 1 -- 172.21.15.40:0/43399152 learned_addr learned my addr 172.21.15.40:0/43399152 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:39.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46bffff640 1 -- 172.21.15.40:0/43399152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46c0102a10 msgr2=0x7f46c0197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46bffff640 1 --2- 172.21.15.40:0/43399152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46c0102a10 0x7f46c0197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46bffff640 1 -- 172.21.15.40:0/43399152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c01033d0 msgr2=0x7f46c019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46bffff640 1 --2- 172.21.15.40:0/43399152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c01033d0 0x7f46c019f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46bffff640 1 -- 172.21.15.40:0/43399152 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c019f680 con 0x7f46c0101820 2024-03-18T22:52:39.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.195+0000 7f46bffff640 1 --2- 172.21.15.40:0/43399152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46c0101820 0x7f46c0197a60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f46b000e9c0 tx=0x7f46b000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.196+0000 7f46bd7fa640 1 -- 172.21.15.40:0/43399152 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b000cd80 con 0x7f46c0101820 2024-03-18T22:52:39.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.196+0000 7f46c6659640 1 -- 172.21.15.40:0/43399152 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46c019f970 con 0x7f46c0101820 2024-03-18T22:52:39.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.196+0000 7f46bd7fa640 1 -- 172.21.15.40:0/43399152 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46b00035f0 con 0x7f46c0101820 2024-03-18T22:52:39.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.196+0000 7f46bd7fa640 1 -- 172.21.15.40:0/43399152 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b0010690 con 0x7f46c0101820 2024-03-18T22:52:39.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.196+0000 7f46c6659640 1 -- 172.21.15.40:0/43399152 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46c019fe90 con 0x7f46c0101820 2024-03-18T22:52:39.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c9242740 1 Processor -- start 2024-03-18T22:52:39.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46bd7fa640 1 -- 172.21.15.40:0/43399152 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f46b00037a0 con 0x7f46c0101820 2024-03-18T22:52:39.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c9242740 1 -- start start 2024-03-18T22:52:39.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c9242740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d682f0 0x55e073d68710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c9242740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d68ce0 0x55e073d6d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c9242740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d6de50 0x55e073d702e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c9242740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e073ccc4b0 con 0x55e073d6de50 2024-03-18T22:52:39.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c9242740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e073c74d70 con 0x55e073d682f0 2024-03-18T22:52:39.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c9242740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e073cc8550 con 0x55e073d68ce0 2024-03-18T22:52:39.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c4bf6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d682f0 0x55e073d68710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46bf7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d6de50 0x55e073d702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c4bf6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d682f0 0x55e073d68710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:35962/0 (socket says 172.21.15.40:35962) 2024-03-18T22:52:39.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c4bf6640 1 -- 172.21.15.40:0/1045284859 learned_addr learned my addr 172.21.15.40:0/1045284859 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:39.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46bffff640 1 --2- 172.21.15.40:0/1045284859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d68ce0 0x55e073d6d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c4bf6640 1 -- 172.21.15.40:0/1045284859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d68ce0 msgr2=0x55e073d6d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c4bf6640 1 --2- 172.21.15.40:0/1045284859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d68ce0 0x55e073d6d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c4bf6640 1 -- 172.21.15.40:0/1045284859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d6de50 msgr2=0x55e073d702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c4bf6640 1 --2- 172.21.15.40:0/1045284859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d6de50 0x55e073d702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.199+0000 7f46c4bf6640 1 -- 172.21.15.40:0/1045284859 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e073c97210 con 0x55e073d682f0 2024-03-18T22:52:39.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.200+0000 7f46c4bf6640 1 --2- 172.21.15.40:0/1045284859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d682f0 0x55e073d68710 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f46b8009fd0 tx=0x7f46b80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fa31d03dd0033bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.200+0000 7f46bd7fa640 1 --2- 172.21.15.40:0/43399152 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f46a80a2800 0x7f46a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.200+0000 7f46a67fc640 1 -- 172.21.15.40:0/1045284859 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b8004d20 con 0x55e073d682f0 2024-03-18T22:52:39.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.200+0000 7f46a67fc640 1 -- 172.21.15.40:0/1045284859 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46b8004050 con 0x55e073d682f0 2024-03-18T22:52:39.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.200+0000 7f46a67fc640 1 -- 172.21.15.40:0/1045284859 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b8005280 con 0x55e073d682f0 2024-03-18T22:52:39.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.201+0000 7f46bd7fa640 1 -- 172.21.15.40:0/43399152 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f46b00cf1e0 con 0x7f46c0101820 2024-03-18T22:52:39.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.201+0000 7f46bffff640 1 --2- 172.21.15.40:0/43399152 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f46a80a2800 0x7f46a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.203+0000 7f46c9242740 1 -- 172.21.15.40:0/1045284859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d682f0 msgr2=0x55e073d68710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.203+0000 7f46bffff640 1 --2- 172.21.15.40:0/43399152 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f46a80a2800 0x7f46a80a4cc0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f46b00ccc60 tx=0x7f46b00d2cc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.203+0000 7f46c9242740 1 --2- 172.21.15.40:0/1045284859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d682f0 0x55e073d68710 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f46b8009fd0 tx=0x7f46b80118d0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.203+0000 7f46c9242740 1 -- 172.21.15.40:0/1045284859 shutdown_connections 2024-03-18T22:52:39.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.203+0000 7f46c9242740 1 --2- 172.21.15.40:0/1045284859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d6de50 0x55e073d702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.203+0000 7f46c9242740 1 --2- 172.21.15.40:0/1045284859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d68ce0 0x55e073d6d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.203+0000 7f46c9242740 1 --2- 172.21.15.40:0/1045284859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d682f0 0x55e073d68710 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.203+0000 7f46c9242740 1 -- 172.21.15.40:0/1045284859 >> 172.21.15.40:0/1045284859 conn(0x55e073d63860 msgr2=0x55e073d63c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.203+0000 7f46c9242740 1 -- 172.21.15.40:0/1045284859 shutdown_connections 2024-03-18T22:52:39.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.203+0000 7f46c9242740 1 -- 172.21.15.40:0/1045284859 wait complete. 2024-03-18T22:52:39.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.203+0000 7f46c9242740 1 -- start start 2024-03-18T22:52:39.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.204+0000 7f46c9242740 1 Processor -- start 2024-03-18T22:52:39.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.204+0000 7f46c9242740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d682f0 0x55e073d727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.204+0000 7f46c9242740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d68ce0 0x55e073d72cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.205+0000 7f46c9242740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d6de50 0x55e073d79d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.205+0000 7f46c4bf6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d682f0 0x55e073d727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.205+0000 7f46c9242740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e073d70e00 con 0x55e073d6de50 2024-03-18T22:52:39.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.205+0000 7f46c9242740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e073c88310 con 0x55e073d68ce0 2024-03-18T22:52:39.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.205+0000 7f46c9242740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e073d70f80 con 0x55e073d682f0 2024-03-18T22:52:39.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.205+0000 7f46c9242740 10 client.0 _ll_register_callbacks cb 0x55e073c7c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:39.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.205+0000 7f46c4bf6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d682f0 0x55e073d727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53610/0 (socket says 172.21.15.40:53610) 2024-03-18T22:52:39.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.205+0000 7f46c4bf6640 1 -- 172.21.15.40:0/1819811335 learned_addr learned my addr 172.21.15.40:0/1819811335 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:39.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.205+0000 7f46bf7fe640 1 --2- 172.21.15.40:0/1819811335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d68ce0 0x55e073d72cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.208+0000 7f46bf7fe640 1 -- 172.21.15.40:0/1819811335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d682f0 msgr2=0x55e073d727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.208+0000 7f46bf7fe640 1 --2- 172.21.15.40:0/1819811335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d682f0 0x55e073d727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.208+0000 7f46bf7fe640 1 -- 172.21.15.40:0/1819811335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d6de50 msgr2=0x55e073d79d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:52:39.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.209+0000 7f46c4bf6640 1 --2- 172.21.15.40:0/1819811335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d6de50 0x55e073d79d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.209+0000 7f46bf7fe640 1 --2- 172.21.15.40:0/1819811335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d6de50 0x55e073d79d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.209+0000 7f46bf7fe640 1 -- 172.21.15.40:0/1819811335 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0739e03f0 con 0x55e073d68ce0 2024-03-18T22:52:39.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.210+0000 7f46c4bf6640 1 --2- 172.21.15.40:0/1819811335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d6de50 0x55e073d79d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:52:39.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.214+0000 7f46bf7fe640 1 --2- 172.21.15.40:0/1819811335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d68ce0 0x55e073d72cf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f46b4000c00 tx=0x7f46b4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b11b6b5439dbc523 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.214+0000 7f469f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:52:39.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.215+0000 7f469f7fe640 1 -- 172.21.15.40:0/1819811335 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b4005030 con 0x55e073d68ce0 2024-03-18T22:52:39.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.215+0000 7f469f7fe640 1 -- 172.21.15.40:0/1819811335 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f46b40051d0 con 0x55e073d68ce0 2024-03-18T22:52:39.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.215+0000 7f46c9242740 1 -- 172.21.15.40:0/1819811335 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46b8004050 con 0x55e073d68ce0 2024-03-18T22:52:39.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.216+0000 7f469f7fe640 1 -- 172.21.15.40:0/1819811335 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f46b40053f0 con 0x55e073d68ce0 2024-03-18T22:52:39.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.218+0000 7f46c9242740 10 client.27878 fetch_fsmap learned FSMap version 36 2024-03-18T22:52:39.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.218+0000 7f46c9242740 1 -- 172.21.15.40:0/1819811335 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e073aaa7d0 con 0x55e073d68ce0 2024-03-18T22:52:39.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.219+0000 7f469f7fe640 1 -- 172.21.15.40:0/1819811335 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46b8004050 con 0x55e073d68ce0 2024-03-18T22:52:39.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.219+0000 7f469f7fe640 1 -- 172.21.15.40:0/1819811335 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 36) ==== 8378+0+0 (secure 0 0 0) 0x7f46b40036c0 con 0x55e073d68ce0 2024-03-18T22:52:39.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.222+0000 7f46c9242740 10 client.27878 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T22:52:39.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.222+0000 7f46c9242740 10 client.27878 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:52:39.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.222+0000 7f46c9242740 20 client.27878 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:39.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.222+0000 7f46c9242740 1 --2- 172.21.15.40:0/1819811335 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e073d7c860 0x55e073d7ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.222+0000 7f46c9242740 4 client.27878 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:39.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.222+0000 7f46c9242740 1 -- 172.21.15.40:0/1819811335 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e073d2a3e0 con 0x55e073d7c860 2024-03-18T22:52:39.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.224+0000 7f46c4bf6640 1 --2- 172.21.15.40:0/1819811335 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e073d7c860 0x55e073d7ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.227+0000 7f46c4bf6640 1 --2- 172.21.15.40:0/1819811335 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e073d7c860 0x55e073d7ecc0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ae6fae9c92a54edb server_cookie=d8ed1526065dfdfc in_seq=0 out_seq=0 2024-03-18T22:52:39.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.227+0000 7f469f7fe640 10 client.27878 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:52:39.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.230+0000 7f469f7fe640 1 -- 172.21.15.40:0/1819811335 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e073d2a3e0 con 0x55e073d7c860 2024-03-18T22:52:39.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.230+0000 7f469f7fe640 10 client.27878 handle_command_reply: tid=0 2024-03-18T22:52:39.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.234+0000 7f46c9242740 1 client.27878 shutdown 2024-03-18T22:52:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.243+0000 7f46c9242740 1 -- 172.21.15.40:0/1819811335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d68ce0 msgr2=0x55e073d72cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.243+0000 7f46c9242740 1 --2- 172.21.15.40:0/1819811335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d68ce0 0x55e073d72cf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f46b4000c00 tx=0x7f46b4004270 comp rx=0 tx=0).stop 2024-03-18T22:52:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.248+0000 7f46c9242740 1 -- 172.21.15.40:0/1819811335 shutdown_connections 2024-03-18T22:52:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.248+0000 7f46c9242740 1 --2- 172.21.15.40:0/1819811335 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e073d7c860 0x55e073d7ecc0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.248+0000 7f46c9242740 1 --2- 172.21.15.40:0/1819811335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d6de50 0x55e073d79d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.248+0000 7f46c9242740 1 --2- 172.21.15.40:0/1819811335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d68ce0 0x55e073d72cf0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.248+0000 7f46c9242740 1 --2- 172.21.15.40:0/1819811335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d682f0 0x55e073d727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.248+0000 7f46c9242740 1 -- 172.21.15.40:0/1819811335 >> 172.21.15.40:0/1819811335 conn(0x55e073d63860 msgr2=0x55e073d62ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.248+0000 7f469f7fe640 0 client.27878 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:52:39.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.252+0000 7f46c9242740 1 -- 172.21.15.40:0/1819811335 shutdown_connections 2024-03-18T22:52:39.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.252+0000 7f46c9242740 1 -- 172.21.15.40:0/1819811335 wait complete. 2024-03-18T22:52:39.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.253+0000 7f46c9242740 20 client.27878 trim_cache size 0 max 16384 2024-03-18T22:52:39.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.307+0000 7f46c9242740 1 Processor -- start 2024-03-18T22:52:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.309+0000 7f46c9242740 1 -- start start 2024-03-18T22:52:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.309+0000 7f46c9242740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d682f0 0x55e073d727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.309+0000 7f46c9242740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d68ce0 0x55e073d72cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.309+0000 7f46c9242740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d6de50 0x55e073d6a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.309+0000 7f46c9242740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f46c01a84d0 con 0x55e073d682f0 2024-03-18T22:52:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.309+0000 7f46c9242740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f46c01aa0d0 con 0x55e073d6de50 2024-03-18T22:52:39.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.309+0000 7f46c9242740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f46c01a8ba0 con 0x55e073d68ce0 2024-03-18T22:52:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.310+0000 7f46c4bf6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d682f0 0x55e073d727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.310+0000 7f46c4bf6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d682f0 0x55e073d727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32840/0 (socket says 172.21.15.40:32840) 2024-03-18T22:52:39.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.310+0000 7f46c4bf6640 1 -- 172.21.15.40:0/2923247572 learned_addr learned my addr 172.21.15.40:0/2923247572 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:39.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.312+0000 7f46bf7fe640 1 --2- 172.21.15.40:0/2923247572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d68ce0 0x55e073d72cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.314+0000 7f46bf7fe640 1 -- 172.21.15.40:0/2923247572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d6de50 msgr2=0x55e073d6a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.314+0000 7f46bf7fe640 1 --2- 172.21.15.40:0/2923247572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d6de50 0x55e073d6a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.314+0000 7f46bf7fe640 1 -- 172.21.15.40:0/2923247572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d682f0 msgr2=0x55e073d727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.315+0000 7f46bf7fe640 1 --2- 172.21.15.40:0/2923247572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d682f0 0x55e073d727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.315+0000 7f46bf7fe640 1 -- 172.21.15.40:0/2923247572 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e073d6d460 con 0x55e073d68ce0 2024-03-18T22:52:39.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.315+0000 7f46c4bf6640 1 --2- 172.21.15.40:0/2923247572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d682f0 0x55e073d727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:52:39.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.316+0000 7f46bf7fe640 1 --2- 172.21.15.40:0/2923247572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d68ce0 0x55e073d72cf0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f46b4003ab0 tx=0x7f46b402c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41ac4839a4c39715 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.320+0000 7f46a5ffb640 1 -- 172.21.15.40:0/2923247572 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b4058070 con 0x55e073d68ce0 2024-03-18T22:52:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.320+0000 7f46a5ffb640 1 -- 172.21.15.40:0/2923247572 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46b4005790 con 0x55e073d68ce0 2024-03-18T22:52:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.320+0000 7f46a5ffb640 1 -- 172.21.15.40:0/2923247572 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b40349e0 con 0x55e073d68ce0 2024-03-18T22:52:39.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.321+0000 7f46c9242740 1 -- 172.21.15.40:0/2923247572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d68ce0 msgr2=0x55e073d72cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.321+0000 7f46c9242740 1 --2- 172.21.15.40:0/2923247572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d68ce0 0x55e073d72cf0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f46b4003ab0 tx=0x7f46b402c800 comp rx=0 tx=0).stop 2024-03-18T22:52:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.323+0000 7f46c9242740 1 -- 172.21.15.40:0/2923247572 shutdown_connections 2024-03-18T22:52:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.323+0000 7f46c9242740 1 --2- 172.21.15.40:0/2923247572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073d6de50 0x55e073d6a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.323+0000 7f46c9242740 1 --2- 172.21.15.40:0/2923247572 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d68ce0 0x55e073d72cf0 secure :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f46b4003ab0 tx=0x7f46b402c800 comp rx=0 tx=0).stop 2024-03-18T22:52:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.323+0000 7f46c9242740 1 --2- 172.21.15.40:0/2923247572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d682f0 0x55e073d727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.323+0000 7f46c9242740 1 -- 172.21.15.40:0/2923247572 >> 172.21.15.40:0/2923247572 conn(0x55e073d63860 msgr2=0x55e073d63c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.325+0000 7f46c9242740 1 -- 172.21.15.40:0/2923247572 shutdown_connections 2024-03-18T22:52:39.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.327+0000 7f46c9242740 1 -- 172.21.15.40:0/2923247572 wait complete. 2024-03-18T22:52:39.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.327+0000 7f46c9242740 1 -- start start 2024-03-18T22:52:39.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.328+0000 7f46c9242740 1 Processor -- start 2024-03-18T22:52:39.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.329+0000 7f46c9242740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d682f0 0x55e073ddeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.329+0000 7f46c9242740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d6de50 0x55e073de1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.329+0000 7f46c9242740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073de1570 0x55e073de3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.329+0000 7f46c9242740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f46c01a9a00 con 0x55e073d682f0 2024-03-18T22:52:39.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.329+0000 7f46c9242740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f46c01a9270 con 0x55e073de1570 2024-03-18T22:52:39.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.329+0000 7f46c9242740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f46c01a64b0 con 0x55e073d6de50 2024-03-18T22:52:39.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.329+0000 7f46c9242740 10 client.0 _ll_register_callbacks cb 0x7f46b4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:39.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.334+0000 7f46c4bf6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d682f0 0x55e073ddeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.334+0000 7f46c4bf6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d682f0 0x55e073ddeae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32850/0 (socket says 172.21.15.40:32850) 2024-03-18T22:52:39.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.335+0000 7f46c4bf6640 1 -- 172.21.15.40:0/2068620886 learned_addr learned my addr 172.21.15.40:0/2068620886 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:39.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.335+0000 7f46c4bf6640 1 --2- 172.21.15.40:0/2068620886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073de1570 0x55e073de3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.335+0000 7f46bf7fe640 1 --2- 172.21.15.40:0/2068620886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d6de50 0x55e073de1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.336+0000 7f46c4bf6640 1 -- 172.21.15.40:0/2068620886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d6de50 msgr2=0x55e073de1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.336+0000 7f46c4bf6640 1 --2- 172.21.15.40:0/2068620886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d6de50 0x55e073de1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.336+0000 7f46c4bf6640 1 -- 172.21.15.40:0/2068620886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d682f0 msgr2=0x55e073ddeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.336+0000 7f46c4bf6640 1 --2- 172.21.15.40:0/2068620886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d682f0 0x55e073ddeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.336+0000 7f46c4bf6640 1 -- 172.21.15.40:0/2068620886 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e073df5010 con 0x55e073de1570 2024-03-18T22:52:39.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.337+0000 7f46c4bf6640 1 --2- 172.21.15.40:0/2068620886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073de1570 0x55e073de3960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f46b801b040 tx=0x7f46b402cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c221d4176d64e20 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.337+0000 7f469f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:52:39.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.337+0000 7f469f7fe640 1 -- 172.21.15.40:0/2068620886 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b80347f0 con 0x55e073de1570 2024-03-18T22:52:39.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.337+0000 7f469f7fe640 1 -- 172.21.15.40:0/2068620886 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f46b8005040 con 0x55e073de1570 2024-03-18T22:52:39.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.337+0000 7f46c9242740 1 -- 172.21.15.40:0/2068620886 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46b4005790 con 0x55e073de1570 2024-03-18T22:52:39.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.346+0000 7f469f7fe640 1 -- 172.21.15.40:0/2068620886 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f46b8034b90 con 0x55e073de1570 2024-03-18T22:52:39.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.347+0000 7f46c9242740 10 client.27881 fetch_fsmap learned FSMap version 36 2024-03-18T22:52:39.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.347+0000 7f46c9242740 1 -- 172.21.15.40:0/2068620886 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e073df6380 con 0x55e073de1570 2024-03-18T22:52:39.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.349+0000 7f469f7fe640 1 -- 172.21.15.40:0/2068620886 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46b4005790 con 0x55e073de1570 2024-03-18T22:52:39.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.350+0000 7f469f7fe640 1 -- 172.21.15.40:0/2068620886 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 36) ==== 8378+0+0 (secure 0 0 0) 0x7f46b800f710 con 0x55e073de1570 2024-03-18T22:52:39.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.350+0000 7f46c9242740 10 client.27881 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T22:52:39.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.350+0000 7f46c9242740 10 client.27881 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:52:39.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.350+0000 7f46c9242740 20 client.27881 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:39.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.350+0000 7f46c9242740 1 --2- 172.21.15.40:0/2068620886 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e073df6b80 0x55e073df8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.350+0000 7f46c9242740 4 client.27881 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:52:39.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.350+0000 7f46c9242740 1 -- 172.21.15.40:0/2068620886 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e073cd99f0 con 0x55e073df6b80 2024-03-18T22:52:39.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.352+0000 7f46bf7fe640 1 --2- 172.21.15.40:0/2068620886 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e073df6b80 0x55e073df8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.369+0000 7f46bf7fe640 1 --2- 172.21.15.40:0/2068620886 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e073df6b80 0x55e073df8fa0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=57fbf30f5c39e75 server_cookie=d0d5ed305639c839 in_seq=0 out_seq=0 2024-03-18T22:52:39.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.369+0000 7f469f7fe640 10 client.27881 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:52:39.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.369+0000 7f469f7fe640 1 -- 172.21.15.40:0/2068620886 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e073cd99f0 con 0x55e073df6b80 2024-03-18T22:52:39.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.369+0000 7f469f7fe640 10 client.27881 handle_command_reply: tid=0 2024-03-18T22:52:39.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.369+0000 7f46c9242740 1 client.27881 shutdown 2024-03-18T22:52:39.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.371+0000 7f46c9242740 1 -- 172.21.15.40:0/2068620886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073de1570 msgr2=0x55e073de3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.371+0000 7f46c9242740 1 --2- 172.21.15.40:0/2068620886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073de1570 0x55e073de3960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f46b801b040 tx=0x7f46b402cd30 comp rx=0 tx=0).stop 2024-03-18T22:52:39.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.373+0000 7f46c9242740 1 -- 172.21.15.40:0/2068620886 shutdown_connections 2024-03-18T22:52:39.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.373+0000 7f46c9242740 1 --2- 172.21.15.40:0/2068620886 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55e073df6b80 0x55e073df8fa0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.373+0000 7f46c9242740 1 --2- 172.21.15.40:0/2068620886 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e073de1570 0x55e073de3960 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.373+0000 7f46c9242740 1 --2- 172.21.15.40:0/2068620886 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e073d6de50 0x55e073de1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.373+0000 7f46c9242740 1 --2- 172.21.15.40:0/2068620886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e073d682f0 0x55e073ddeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.373+0000 7f46c9242740 1 -- 172.21.15.40:0/2068620886 >> 172.21.15.40:0/2068620886 conn(0x55e073d63860 msgr2=0x55e073d72750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.374+0000 7f469f7fe640 0 client.27881 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:52:39.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.377+0000 7f46c9242740 1 -- 172.21.15.40:0/2068620886 shutdown_connections 2024-03-18T22:52:39.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.377+0000 7f46c9242740 1 -- 172.21.15.40:0/2068620886 wait complete. 2024-03-18T22:52:39.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.381+0000 7f46c9242740 20 client.27881 trim_cache size 0 max 16384 2024-03-18T22:52:39.383 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:52:39.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.386+0000 7f46c6659640 1 -- 172.21.15.40:0/43399152 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f46a80a2800 msgr2=0x7f46a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.387+0000 7f46c6659640 1 --2- 172.21.15.40:0/43399152 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f46a80a2800 0x7f46a80a4cc0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f46b00ccc60 tx=0x7f46b00d2cc0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.387+0000 7f46c6659640 1 -- 172.21.15.40:0/43399152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46c0101820 msgr2=0x7f46c0197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.387+0000 7f46c6659640 1 --2- 172.21.15.40:0/43399152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46c0101820 0x7f46c0197a60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f46b000e9c0 tx=0x7f46b000ee80 comp rx=0 tx=0).stop 2024-03-18T22:52:39.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.388+0000 7f46c6659640 1 -- 172.21.15.40:0/43399152 shutdown_connections 2024-03-18T22:52:39.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.388+0000 7f46c6659640 1 --2- 172.21.15.40:0/43399152 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f46a80a2800 0x7f46a80a4cc0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.388+0000 7f46c6659640 1 --2- 172.21.15.40:0/43399152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46c01033d0 0x7f46c019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.388+0000 7f46c6659640 1 --2- 172.21.15.40:0/43399152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46c0102a10 0x7f46c0197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.388+0000 7f46c6659640 1 --2- 172.21.15.40:0/43399152 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f46c0101820 0x7f46c0197a60 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.388+0000 7f46c6659640 1 -- 172.21.15.40:0/43399152 >> 172.21.15.40:0/43399152 conn(0x7f46c00fcd90 msgr2=0x7f46c0105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.389+0000 7f46c6659640 1 -- 172.21.15.40:0/43399152 shutdown_connections 2024-03-18T22:52:39.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.391+0000 7f46c6659640 1 -- 172.21.15.40:0/43399152 wait complete. 2024-03-18T22:52:39.432 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:52:39.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:39 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3437214904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:52:39.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:39 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3437214904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:52:39.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.705+0000 7f0abf7fe640 1 -- 172.21.15.40:0/1504646639 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab80328b0 con 0x7f0ac0108e60 2024-03-18T22:52:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.706+0000 7f0ac6c43640 1 -- 172.21.15.40:0/1504646639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ac0108e60 msgr2=0x7f0ac01092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.706+0000 7f0ac6c43640 1 --2- 172.21.15.40:0/1504646639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ac0108e60 0x7f0ac01092e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009ca0 tx=0x7f0ab802c140 comp rx=0 tx=0).stop 2024-03-18T22:52:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.706+0000 7f0ac6c43640 1 -- 172.21.15.40:0/1504646639 shutdown_connections 2024-03-18T22:52:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.706+0000 7f0ac6c43640 1 --2- 172.21.15.40:0/1504646639 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ac0109820 0x7f0ac010e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.706+0000 7f0ac6c43640 1 --2- 172.21.15.40:0/1504646639 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ac0108e60 0x7f0ac01092e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.706+0000 7f0ac6c43640 1 --2- 172.21.15.40:0/1504646639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ac0072b90 0x7f0ac0072fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.706+0000 7f0ac6c43640 1 -- 172.21.15.40:0/1504646639 >> 172.21.15.40:0/1504646639 conn(0x7f0ac006e640 msgr2=0x7f0ac0070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.706+0000 7f0ac6c43640 1 -- 172.21.15.40:0/1504646639 shutdown_connections 2024-03-18T22:52:39.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.706+0000 7f0ac6c43640 1 -- 172.21.15.40:0/1504646639 wait complete. 2024-03-18T22:52:39.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.707+0000 7f0ac6c43640 1 Processor -- start 2024-03-18T22:52:39.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.707+0000 7f0ac6c43640 1 -- start start 2024-03-18T22:52:39.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.707+0000 7f0ac6c43640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ac0072b90 0x7f0ac01a6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.707+0000 7f0ac6c43640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ac0108e60 0x7f0ac01a6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.707+0000 7f0ac6c43640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ac0109820 0x7f0ac01addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.707+0000 7f0ac6c43640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac011c100 con 0x7f0ac0072b90 2024-03-18T22:52:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.707+0000 7f0ac6c43640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac011bf80 con 0x7f0ac0109820 2024-03-18T22:52:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.707+0000 7f0ac6c43640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac011c280 con 0x7f0ac0108e60 2024-03-18T22:52:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.708+0000 7f0ac51e0640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ac0109820 0x7f0ac01addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.708+0000 7f0ac51e0640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ac0109820 0x7f0ac01addf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36000/0 (socket says 172.21.15.40:36000) 2024-03-18T22:52:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.708+0000 7f0ac51e0640 1 -- 172.21.15.40:0/2018759629 learned_addr learned my addr 172.21.15.40:0/2018759629 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.708+0000 7f0ac49df640 1 --2- 172.21.15.40:0/2018759629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ac0072b90 0x7f0ac01a6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.708+0000 7f0abffff640 1 --2- 172.21.15.40:0/2018759629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ac0108e60 0x7f0ac01a6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.708+0000 7f0ac49df640 1 -- 172.21.15.40:0/2018759629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ac0108e60 msgr2=0x7f0ac01a6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.708+0000 7f0ac49df640 1 --2- 172.21.15.40:0/2018759629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ac0108e60 0x7f0ac01a6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.709+0000 7f0ac49df640 1 -- 172.21.15.40:0/2018759629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ac0109820 msgr2=0x7f0ac01addf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.709+0000 7f0ac49df640 1 --2- 172.21.15.40:0/2018759629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ac0109820 0x7f0ac01addf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.709+0000 7f0ac49df640 1 -- 172.21.15.40:0/2018759629 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac01ae460 con 0x7f0ac0072b90 2024-03-18T22:52:39.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.709+0000 7f0abffff640 1 --2- 172.21.15.40:0/2018759629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ac0108e60 0x7f0ac01a6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:52:39.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.709+0000 7f0ac49df640 1 --2- 172.21.15.40:0/2018759629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ac0072b90 0x7f0ac01a6840 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000b3f0 tx=0x7f0ab000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.710+0000 7f0abdffb640 1 -- 172.21.15.40:0/2018759629 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0015020 con 0x7f0ac0072b90 2024-03-18T22:52:39.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.710+0000 7f0ac6c43640 1 -- 172.21.15.40:0/2018759629 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac01ae750 con 0x7f0ac0072b90 2024-03-18T22:52:39.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.710+0000 7f0abdffb640 1 -- 172.21.15.40:0/2018759629 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ab0011ce0 con 0x7f0ac0072b90 2024-03-18T22:52:39.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.711+0000 7f0abdffb640 1 -- 172.21.15.40:0/2018759629 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0010d80 con 0x7f0ac0072b90 2024-03-18T22:52:39.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.711+0000 7f0ac6c43640 1 -- 172.21.15.40:0/2018759629 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac01aed20 con 0x7f0ac0072b90 2024-03-18T22:52:39.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.715+0000 7f0ac982a740 1 Processor -- start 2024-03-18T22:52:39.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.715+0000 7f0abdffb640 1 -- 172.21.15.40:0/2018759629 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0ab0018020 con 0x7f0ac0072b90 2024-03-18T22:52:39.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.715+0000 7f0ac982a740 1 -- start start 2024-03-18T22:52:39.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.717+0000 7f0abdffb640 1 --2- 172.21.15.40:0/2018759629 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0aa80a2830 0x7f0aa80a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.717+0000 7f0abdffb640 1 -- 172.21.15.40:0/2018759629 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0ab00ce480 con 0x7f0ac0072b90 2024-03-18T22:52:39.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.717+0000 7f0ac982a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 0x555d261c08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.717+0000 7f0ac982a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c0e80 0x555d261c5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.717+0000 7f0ac982a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c5ff0 0x555d261c8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.717+0000 7f0ac982a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555d261244b0 con 0x555d261c5ff0 2024-03-18T22:52:39.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.717+0000 7f0ac982a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555d260ccd70 con 0x555d261c0490 2024-03-18T22:52:39.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.717+0000 7f0ac982a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x555d26120550 con 0x555d261c0e80 2024-03-18T22:52:39.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.718+0000 7f0ac51e0640 1 --2- 172.21.15.40:0/2018759629 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0aa80a2830 0x7f0aa80a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.721+0000 7f0ac51e0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c5ff0 0x555d261c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.721+0000 7f0ac51e0640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c5ff0 0x555d261c8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32872/0 (socket says 172.21.15.40:32872) 2024-03-18T22:52:39.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.721+0000 7f0ac51e0640 1 -- 172.21.15.40:0/3565101026 learned_addr learned my addr 172.21.15.40:0/3565101026 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:39.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.721+0000 7f0abffff640 1 --2- 172.21.15.40:0/3565101026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c0e80 0x555d261c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.721+0000 7f0ac49df640 1 --2- 172.21.15.40:0/3565101026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 0x555d261c08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.722+0000 7f0ac51e0640 1 -- 172.21.15.40:0/3565101026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c0e80 msgr2=0x555d261c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.722+0000 7f0ac51e0640 1 --2- 172.21.15.40:0/3565101026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c0e80 0x555d261c5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.722+0000 7f0ac51e0640 1 -- 172.21.15.40:0/3565101026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 msgr2=0x555d261c08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.722+0000 7f0ac51e0640 1 --2- 172.21.15.40:0/3565101026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 0x555d261c08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.722+0000 7f0ac51e0640 1 -- 172.21.15.40:0/3565101026 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d260ef210 con 0x555d261c5ff0 2024-03-18T22:52:39.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.723+0000 7f0ac51e0640 1 --2- 172.21.15.40:0/3565101026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c5ff0 0x555d261c8480 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4017650 tx=0x7f0ab4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39829070efa851f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.724+0000 7f0aa6ffd640 1 -- 172.21.15.40:0/3565101026 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab401d070 con 0x555d261c5ff0 2024-03-18T22:52:39.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.724+0000 7f0aa6ffd640 1 -- 172.21.15.40:0/3565101026 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ab4010040 con 0x555d261c5ff0 2024-03-18T22:52:39.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.725+0000 7f0ac982a740 1 -- 172.21.15.40:0/3565101026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c5ff0 msgr2=0x555d261c8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.725+0000 7f0ac982a740 1 --2- 172.21.15.40:0/3565101026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c5ff0 0x555d261c8480 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4017650 tx=0x7f0ab4017000 comp rx=0 tx=0).stop 2024-03-18T22:52:39.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.727+0000 7f0ac51e0640 1 --2- 172.21.15.40:0/2018759629 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0aa80a2830 0x7f0aa80a4cf0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4009fc0 tx=0x7f0ab4009900 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.728+0000 7f0ac982a740 1 -- 172.21.15.40:0/3565101026 shutdown_connections 2024-03-18T22:52:39.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.728+0000 7f0ac982a740 1 --2- 172.21.15.40:0/3565101026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c5ff0 0x555d261c8480 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.728+0000 7f0ac982a740 1 --2- 172.21.15.40:0/3565101026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c0e80 0x555d261c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.728+0000 7f0ac982a740 1 --2- 172.21.15.40:0/3565101026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 0x555d261c08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.728+0000 7f0ac982a740 1 -- 172.21.15.40:0/3565101026 >> 172.21.15.40:0/3565101026 conn(0x555d261bba00 msgr2=0x555d261bbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.728+0000 7f0ac982a740 1 -- 172.21.15.40:0/3565101026 shutdown_connections 2024-03-18T22:52:39.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.728+0000 7f0ac982a740 1 -- 172.21.15.40:0/3565101026 wait complete. 2024-03-18T22:52:39.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.728+0000 7f0ac982a740 1 -- start start 2024-03-18T22:52:39.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.728+0000 7f0ac982a740 1 Processor -- start 2024-03-18T22:52:39.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.729+0000 7f0ac982a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 0x555d261ca900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.729+0000 7f0ac982a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c0e80 0x555d261cae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.729+0000 7f0ac982a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c5ff0 0x555d261d1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.729+0000 7f0ac982a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x555d261c8f90 con 0x555d261c0e80 2024-03-18T22:52:39.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.729+0000 7f0ac982a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555d260e0310 con 0x555d261c0490 2024-03-18T22:52:39.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.729+0000 7f0ac982a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x555d261c9110 con 0x555d261c5ff0 2024-03-18T22:52:39.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.729+0000 7f0ac982a740 10 client.0 _ll_register_callbacks cb 0x555d260d42d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:39.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.730+0000 7f0ac49df640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 0x555d261ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.730+0000 7f0ac49df640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 0x555d261ca900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36024/0 (socket says 172.21.15.40:36024) 2024-03-18T22:52:39.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.730+0000 7f0ac49df640 1 -- 172.21.15.40:0/2832291221 learned_addr learned my addr 172.21.15.40:0/2832291221 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:39.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.730+0000 7f0abffff640 1 --2- 172.21.15.40:0/2832291221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c0e80 0x555d261cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.730+0000 7f0ac51e0640 1 --2- 172.21.15.40:0/2832291221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c5ff0 0x555d261d1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.731+0000 7f0abffff640 1 -- 172.21.15.40:0/2832291221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c5ff0 msgr2=0x555d261d1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.731+0000 7f0abffff640 1 --2- 172.21.15.40:0/2832291221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c5ff0 0x555d261d1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.731+0000 7f0abffff640 1 -- 172.21.15.40:0/2832291221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 msgr2=0x555d261ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.731+0000 7f0abffff640 1 --2- 172.21.15.40:0/2832291221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 0x555d261ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.731+0000 7f0abffff640 1 -- 172.21.15.40:0/2832291221 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d25e383f0 con 0x555d261c0e80 2024-03-18T22:52:39.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.731+0000 7f0abffff640 1 --2- 172.21.15.40:0/2832291221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c0e80 0x555d261cae40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0ac01a7d50 tx=0x7f0ab8009980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78dafcd2feb01a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.731+0000 7f0a9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:39.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.732+0000 7f0a9ffff640 1 -- 172.21.15.40:0/2832291221 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8039050 con 0x555d261c0e80 2024-03-18T22:52:39.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.732+0000 7f0ac982a740 1 -- 172.21.15.40:0/2832291221 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ab4010040 con 0x555d261c0e80 2024-03-18T22:52:39.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.732+0000 7f0a9ffff640 1 -- 172.21.15.40:0/2832291221 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0ab8032850 con 0x555d261c0e80 2024-03-18T22:52:39.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.732+0000 7f0a9ffff640 1 -- 172.21.15.40:0/2832291221 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0ab804e050 con 0x555d261c0e80 2024-03-18T22:52:39.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.733+0000 7f0ac982a740 10 client.18075 fetch_fsmap learned FSMap version 36 2024-03-18T22:52:39.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.733+0000 7f0ac982a740 1 -- 172.21.15.40:0/2832291221 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555d25f027d0 con 0x555d261c0e80 2024-03-18T22:52:39.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.733+0000 7f0a9ffff640 1 -- 172.21.15.40:0/2832291221 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ab4010040 con 0x555d261c0e80 2024-03-18T22:52:39.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.733+0000 7f0a9ffff640 1 -- 172.21.15.40:0/2832291221 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 36) ==== 8378+0+0 (secure 0 0 0) 0x7f0ab80346f0 con 0x555d261c0e80 2024-03-18T22:52:39.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.734+0000 7f0ac982a740 10 client.18075 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T22:52:39.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.734+0000 7f0ac982a740 10 client.18075 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:52:39.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.734+0000 7f0ac982a740 20 client.18075 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:39.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.734+0000 7f0ac982a740 1 --2- 172.21.15.40:0/2832291221 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d261e7bb0 0x555d261ea020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.735+0000 7f0ac982a740 4 client.18075 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:39.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.735+0000 7f0ac982a740 1 -- 172.21.15.40:0/2832291221 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d261823e0 con 0x555d261e7bb0 2024-03-18T22:52:39.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.737+0000 7f0ac49df640 1 --2- 172.21.15.40:0/2832291221 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d261e7bb0 0x555d261ea020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.745+0000 7f0ac49df640 1 --2- 172.21.15.40:0/2832291221 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d261e7bb0 0x555d261ea020 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ca3921b030c42ccb server_cookie=b6da4dff9107131b in_seq=0 out_seq=0 2024-03-18T22:52:39.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.745+0000 7f0a9ffff640 10 client.18075 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:52:39.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.749+0000 7f0a9ffff640 1 -- 172.21.15.40:0/2832291221 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555d261823e0 con 0x555d261e7bb0 2024-03-18T22:52:39.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.749+0000 7f0a9ffff640 10 client.18075 handle_command_reply: tid=0 2024-03-18T22:52:39.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.749+0000 7f0ac982a740 1 client.18075 shutdown 2024-03-18T22:52:39.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.751+0000 7f0ac982a740 1 -- 172.21.15.40:0/2832291221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c0e80 msgr2=0x555d261cae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.751+0000 7f0ac982a740 1 --2- 172.21.15.40:0/2832291221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c0e80 0x555d261cae40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0ac01a7d50 tx=0x7f0ab8009980 comp rx=0 tx=0).stop 2024-03-18T22:52:39.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.751+0000 7f0ac982a740 1 -- 172.21.15.40:0/2832291221 shutdown_connections 2024-03-18T22:52:39.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.751+0000 7f0ac982a740 1 --2- 172.21.15.40:0/2832291221 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d261e7bb0 0x555d261ea020 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.751+0000 7f0ac982a740 1 --2- 172.21.15.40:0/2832291221 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c5ff0 0x555d261d1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.751+0000 7f0ac982a740 1 --2- 172.21.15.40:0/2832291221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c0e80 0x555d261cae40 secure :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f0ac01a7d50 tx=0x7f0ab8009980 comp rx=0 tx=0).stop 2024-03-18T22:52:39.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.751+0000 7f0ac982a740 1 --2- 172.21.15.40:0/2832291221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 0x555d261ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.751+0000 7f0ac982a740 1 -- 172.21.15.40:0/2832291221 >> 172.21.15.40:0/2832291221 conn(0x555d261bba00 msgr2=0x555d261bad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.752+0000 7f0a9ffff640 0 client.18075 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:52:39.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.758+0000 7f0ac982a740 1 -- 172.21.15.40:0/2832291221 shutdown_connections 2024-03-18T22:52:39.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.760+0000 7f0ac982a740 1 -- 172.21.15.40:0/2832291221 wait complete. 2024-03-18T22:52:39.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.760+0000 7f0ac982a740 20 client.18075 trim_cache size 0 max 16384 2024-03-18T22:52:39.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.802+0000 7f0ac982a740 1 Processor -- start 2024-03-18T22:52:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.802+0000 7f0ac982a740 1 -- start start 2024-03-18T22:52:39.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.802+0000 7f0ac982a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 0x555d261c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.802+0000 7f0ac982a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c5ff0 0x555d261d1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.802+0000 7f0ac982a740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261b6b90 0x555d261c22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.802+0000 7f0ac982a740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac01b0960 con 0x555d261c5ff0 2024-03-18T22:52:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.802+0000 7f0ac982a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac01b2560 con 0x555d261c0490 2024-03-18T22:52:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.802+0000 7f0ac982a740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac01b1030 con 0x555d261b6b90 2024-03-18T22:52:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.803+0000 7f0ac51e0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261b6b90 0x555d261c22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.803+0000 7f0ac51e0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261b6b90 0x555d261c22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53668/0 (socket says 172.21.15.40:53668) 2024-03-18T22:52:39.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.803+0000 7f0ac51e0640 1 -- 172.21.15.40:0/1633393102 learned_addr learned my addr 172.21.15.40:0/1633393102 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:39.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.804+0000 7f0abffff640 1 --2- 172.21.15.40:0/1633393102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c5ff0 0x555d261d1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.805+0000 7f0ac49df640 1 --2- 172.21.15.40:0/1633393102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 0x555d261c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.807+0000 7f0ac49df640 1 -- 172.21.15.40:0/1633393102 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261b6b90 msgr2=0x555d261c22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.807+0000 7f0ac49df640 1 --2- 172.21.15.40:0/1633393102 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261b6b90 0x555d261c22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.807+0000 7f0ac49df640 1 -- 172.21.15.40:0/1633393102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c5ff0 msgr2=0x555d261d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.807+0000 7f0ac49df640 1 --2- 172.21.15.40:0/1633393102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c5ff0 0x555d261d1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.807+0000 7f0ac49df640 1 -- 172.21.15.40:0/1633393102 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d261c1260 con 0x555d261c0490 2024-03-18T22:52:39.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.807+0000 7f0ac49df640 1 --2- 172.21.15.40:0/1633393102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 0x555d261c58b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000bdb0 tx=0x7f0ab00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25100b3aec63195f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.807+0000 7f0aa67fc640 1 -- 172.21.15.40:0/1633393102 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab00b8070 con 0x555d261c0490 2024-03-18T22:52:39.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.808+0000 7f0aa67fc640 1 -- 172.21.15.40:0/1633393102 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ab00cbc40 con 0x555d261c0490 2024-03-18T22:52:39.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.808+0000 7f0aa67fc640 1 -- 172.21.15.40:0/1633393102 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab00ae070 con 0x555d261c0490 2024-03-18T22:52:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.809+0000 7f0ac982a740 1 -- 172.21.15.40:0/1633393102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 msgr2=0x555d261c58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.809+0000 7f0ac982a740 1 --2- 172.21.15.40:0/1633393102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 0x555d261c58b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000bdb0 tx=0x7f0ab00046c0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.810+0000 7f0ac982a740 1 -- 172.21.15.40:0/1633393102 shutdown_connections 2024-03-18T22:52:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.810+0000 7f0ac982a740 1 --2- 172.21.15.40:0/1633393102 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261b6b90 0x555d261c22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.810+0000 7f0ac982a740 1 --2- 172.21.15.40:0/1633393102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c5ff0 0x555d261d1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.810+0000 7f0ac982a740 1 --2- 172.21.15.40:0/1633393102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261c0490 0x555d261c58b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.810+0000 7f0ac982a740 1 -- 172.21.15.40:0/1633393102 >> 172.21.15.40:0/1633393102 conn(0x555d261bba00 msgr2=0x555d261bbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.812+0000 7f0ac982a740 1 -- 172.21.15.40:0/1633393102 shutdown_connections 2024-03-18T22:52:39.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.813+0000 7f0ac982a740 1 -- 172.21.15.40:0/1633393102 wait complete. 2024-03-18T22:52:39.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.813+0000 7f0ac982a740 1 -- start start 2024-03-18T22:52:39.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.813+0000 7f0ac982a740 1 Processor -- start 2024-03-18T22:52:39.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.814+0000 7f0ac982a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261b6b90 0x555d261cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.814+0000 7f0ac982a740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c0490 0x555d261cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.815+0000 7f0ac49df640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261b6b90 0x555d261cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.815+0000 7f0ac49df640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261b6b90 0x555d261cf360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36042/0 (socket says 172.21.15.40:36042) 2024-03-18T22:52:39.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.815+0000 7f0abffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c0490 0x555d261cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.815+0000 7f0abffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c0490 0x555d261cf8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32912/0 (socket says 172.21.15.40:32912) 2024-03-18T22:52:39.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.815+0000 7f0ac49df640 1 -- 172.21.15.40:0/3651138653 learned_addr learned my addr 172.21.15.40:0/3651138653 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:39.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.815+0000 7f0ac982a740 1 --2- 172.21.15.40:0/3651138653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c5ff0 0x555d261d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.815+0000 7f0ac982a740 1 -- 172.21.15.40:0/3651138653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac01b1e90 con 0x555d261c0490 2024-03-18T22:52:39.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.815+0000 7f0ac982a740 1 -- 172.21.15.40:0/3651138653 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac01b1700 con 0x555d261b6b90 2024-03-18T22:52:39.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.815+0000 7f0ac51e0640 1 --2- 172.21.15.40:0/3651138653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c5ff0 0x555d261d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.815+0000 7f0ac982a740 1 -- 172.21.15.40:0/3651138653 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac01180e0 con 0x555d261c5ff0 2024-03-18T22:52:39.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.816+0000 7f0abffff640 1 -- 172.21.15.40:0/3651138653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c5ff0 msgr2=0x555d261d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.816+0000 7f0abffff640 1 --2- 172.21.15.40:0/3651138653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c5ff0 0x555d261d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.816+0000 7f0abffff640 1 -- 172.21.15.40:0/3651138653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261b6b90 msgr2=0x555d261cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:39.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.816+0000 7f0abffff640 1 --2- 172.21.15.40:0/3651138653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261b6b90 0x555d261cf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.816+0000 7f0abffff640 1 -- 172.21.15.40:0/3651138653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d261d0ea0 con 0x555d261c0490 2024-03-18T22:52:39.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.816+0000 7f0ac51e0640 1 --2- 172.21.15.40:0/3651138653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c5ff0 0x555d261d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:52:39.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.816+0000 7f0ac49df640 1 --2- 172.21.15.40:0/3651138653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261b6b90 0x555d261cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:52:39.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.817+0000 7f0ac982a740 10 client.0 _ll_register_callbacks cb 0x7f0ab8037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:39.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.817+0000 7f0ac982a740 1 -- 172.21.15.40:0/3651138653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ab00cbc40 con 0x555d261c0490 2024-03-18T22:52:39.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.817+0000 7f0abffff640 1 --2- 172.21.15.40:0/3651138653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c0490 0x555d261cf8a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x555d261cb720 tx=0x7f0ab8032880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8debd894b629f02 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:39.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.817+0000 7f0a9ffff640 10 client.18081 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:39.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.818+0000 7f0a9ffff640 1 -- 172.21.15.40:0/3651138653 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8057070 con 0x555d261c0490 2024-03-18T22:52:39.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.818+0000 7f0a9ffff640 1 -- 172.21.15.40:0/3651138653 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0ab8034850 con 0x555d261c0490 2024-03-18T22:52:39.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.818+0000 7f0a9ffff640 1 -- 172.21.15.40:0/3651138653 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0ab8039040 con 0x555d261c0490 2024-03-18T22:52:39.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.822+0000 7f0ac982a740 10 client.18081 fetch_fsmap learned FSMap version 36 2024-03-18T22:52:39.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.823+0000 7f0ac982a740 1 -- 172.21.15.40:0/3651138653 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({fsmap=36}) -- 0x555d261d2050 con 0x555d261c0490 2024-03-18T22:52:39.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.823+0000 7f0a9ffff640 1 -- 172.21.15.40:0/3651138653 <== mon.0 v2:172.21.15.40:3300/0 4 ==== fsmap(e 36) ==== 8378+0+0 (secure 0 0 0) 0x7f0ab805a030 con 0x555d261c0490 2024-03-18T22:52:39.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.825+0000 7f0ac982a740 10 client.18081 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T22:52:39.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.826+0000 7f0ac982a740 10 client.18081 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:52:39.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.826+0000 7f0ac982a740 20 client.18081 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:39.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.826+0000 7f0ac982a740 1 --2- 172.21.15.40:0/3651138653 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d2624e3e0 0x555d26250800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:39.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.826+0000 7f0ac982a740 4 client.18081 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:52:39.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.827+0000 7f0ac982a740 1 -- 172.21.15.40:0/3651138653 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555d261319f0 con 0x555d2624e3e0 2024-03-18T22:52:39.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.827+0000 7f0ac49df640 1 --2- 172.21.15.40:0/3651138653 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d2624e3e0 0x555d26250800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:39.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:39 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3437214904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:52:39.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.834+0000 7f0ac49df640 1 --2- 172.21.15.40:0/3651138653 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d2624e3e0 0x555d26250800 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=64255aaa1032e21c server_cookie=8a21187d5d1ba602 in_seq=0 out_seq=0 2024-03-18T22:52:39.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.834+0000 7f0a9ffff640 10 client.18081 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:52:39.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.835+0000 7f0a9ffff640 1 -- 172.21.15.40:0/3651138653 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555d261319f0 con 0x555d2624e3e0 2024-03-18T22:52:39.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.835+0000 7f0a9ffff640 10 client.18081 handle_command_reply: tid=0 2024-03-18T22:52:39.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.835+0000 7f0ac982a740 1 client.18081 shutdown 2024-03-18T22:52:39.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.840+0000 7f0ac982a740 1 -- 172.21.15.40:0/3651138653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c0490 msgr2=0x555d261cf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.840+0000 7f0ac982a740 1 --2- 172.21.15.40:0/3651138653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c0490 0x555d261cf8a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x555d261cb720 tx=0x7f0ab8032880 comp rx=0 tx=0).stop 2024-03-18T22:52:39.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.847+0000 7f0ac982a740 1 -- 172.21.15.40:0/3651138653 shutdown_connections 2024-03-18T22:52:39.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.847+0000 7f0ac982a740 1 --2- 172.21.15.40:0/3651138653 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x555d2624e3e0 0x555d26250800 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.847+0000 7f0ac982a740 1 --2- 172.21.15.40:0/3651138653 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x555d261c5ff0 0x555d261d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.847+0000 7f0ac982a740 1 --2- 172.21.15.40:0/3651138653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x555d261c0490 0x555d261cf8a0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.847+0000 7f0ac982a740 1 --2- 172.21.15.40:0/3651138653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d261b6b90 0x555d261cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.847+0000 7f0ac982a740 1 -- 172.21.15.40:0/3651138653 >> 172.21.15.40:0/3651138653 conn(0x555d261bba00 msgr2=0x555d261c22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.853+0000 7f0a9ffff640 0 client.18081 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:52:39.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.858+0000 7f0ac982a740 1 -- 172.21.15.40:0/3651138653 shutdown_connections 2024-03-18T22:52:39.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.858+0000 7f0ac982a740 1 -- 172.21.15.40:0/3651138653 wait complete. 2024-03-18T22:52:39.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.861+0000 7f0ac982a740 20 client.18081 trim_cache size 0 max 16384 2024-03-18T22:52:39.864 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:52:39.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.866+0000 7f0ac6c43640 1 -- 172.21.15.40:0/2018759629 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0aa80a2830 msgr2=0x7f0aa80a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.866+0000 7f0ac6c43640 1 --2- 172.21.15.40:0/2018759629 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0aa80a2830 0x7f0aa80a4cf0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4009fc0 tx=0x7f0ab4009900 comp rx=0 tx=0).stop 2024-03-18T22:52:39.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.866+0000 7f0ac6c43640 1 -- 172.21.15.40:0/2018759629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ac0072b90 msgr2=0x7f0ac01a6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:39.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.866+0000 7f0ac6c43640 1 --2- 172.21.15.40:0/2018759629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ac0072b90 0x7f0ac01a6840 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000b3f0 tx=0x7f0ab000b8b0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.869+0000 7f0ac6c43640 1 -- 172.21.15.40:0/2018759629 shutdown_connections 2024-03-18T22:52:39.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.869+0000 7f0ac6c43640 1 --2- 172.21.15.40:0/2018759629 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0aa80a2830 0x7f0aa80a4cf0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.870+0000 7f0ac6c43640 1 --2- 172.21.15.40:0/2018759629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ac0109820 0x7f0ac01addf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.870+0000 7f0ac6c43640 1 --2- 172.21.15.40:0/2018759629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0ac0108e60 0x7f0ac01a6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.870+0000 7f0ac6c43640 1 --2- 172.21.15.40:0/2018759629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0ac0072b90 0x7f0ac01a6840 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:39.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.870+0000 7f0ac6c43640 1 -- 172.21.15.40:0/2018759629 >> 172.21.15.40:0/2018759629 conn(0x7f0ac006e640 msgr2=0x7f0ac00703c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:39.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.870+0000 7f0ac6c43640 1 -- 172.21.15.40:0/2018759629 shutdown_connections 2024-03-18T22:52:39.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:39.870+0000 7f0ac6c43640 1 -- 172.21.15.40:0/2018759629 wait complete. 2024-03-18T22:52:39.901 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:52:40.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.157+0000 7f906f59e640 1 -- 172.21.15.40:0/2122634227 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90701098b0 msgr2=0x7f907010e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.157+0000 7f906f59e640 1 --2- 172.21.15.40:0/2122634227 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90701098b0 0x7f907010e150 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9068009ca0 tx=0x7f906802c120 comp rx=0 tx=0).stop 2024-03-18T22:52:40.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.157+0000 7f906f59e640 1 -- 172.21.15.40:0/2122634227 shutdown_connections 2024-03-18T22:52:40.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.157+0000 7f906f59e640 1 --2- 172.21.15.40:0/2122634227 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90701098b0 0x7f907010e150 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.157+0000 7f906f59e640 1 --2- 172.21.15.40:0/2122634227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9070108ef0 0x7f9070109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.157+0000 7f906f59e640 1 --2- 172.21.15.40:0/2122634227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9070072c30 0x7f9070073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.157+0000 7f906f59e640 1 -- 172.21.15.40:0/2122634227 >> 172.21.15.40:0/2122634227 conn(0x7f907006e700 msgr2=0x7f9070070b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:40.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.162+0000 7f906f59e640 1 -- 172.21.15.40:0/2122634227 shutdown_connections 2024-03-18T22:52:40.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.162+0000 7f906f59e640 1 -- 172.21.15.40:0/2122634227 wait complete. 2024-03-18T22:52:40.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.163+0000 7f906f59e640 1 Processor -- start 2024-03-18T22:52:40.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.163+0000 7f906f59e640 1 -- start start 2024-03-18T22:52:40.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.164+0000 7f906f59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9070072c30 0x7f9070114f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.164+0000 7f906f59e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9070108ef0 0x7f9070115480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.164+0000 7f906f59e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90701098b0 0x7f90701b6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.164+0000 7f906f59e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f907011c170 con 0x7f9070072c30 2024-03-18T22:52:40.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.164+0000 7f906f59e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f907011bff0 con 0x7f9070108ef0 2024-03-18T22:52:40.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.164+0000 7f906f59e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f907011c2f0 con 0x7f90701098b0 2024-03-18T22:52:40.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.164+0000 7f906dd9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9070108ef0 0x7f9070115480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.164+0000 7f906dd9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9070108ef0 0x7f9070115480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36068/0 (socket says 172.21.15.40:36068) 2024-03-18T22:52:40.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.164+0000 7f906dd9b640 1 -- 172.21.15.40:0/3565811710 learned_addr learned my addr 172.21.15.40:0/3565811710 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:40.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.164+0000 7f906ed9d640 1 --2- 172.21.15.40:0/3565811710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90701098b0 0x7f90701b6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.165+0000 7f906e59c640 1 --2- 172.21.15.40:0/3565811710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9070072c30 0x7f9070114f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.165+0000 7f906ed9d640 1 -- 172.21.15.40:0/3565811710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9070108ef0 msgr2=0x7f9070115480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.165+0000 7f906ed9d640 1 --2- 172.21.15.40:0/3565811710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9070108ef0 0x7f9070115480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.165+0000 7f906ed9d640 1 -- 172.21.15.40:0/3565811710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9070072c30 msgr2=0x7f9070114f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.165+0000 7f906ed9d640 1 --2- 172.21.15.40:0/3565811710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9070072c30 0x7f9070114f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.165+0000 7f906ed9d640 1 -- 172.21.15.40:0/3565811710 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90701b6dc0 con 0x7f90701098b0 2024-03-18T22:52:40.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.166+0000 7f906ed9d640 1 --2- 172.21.15.40:0/3565811710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90701098b0 0x7f90701b6880 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9068009bf0 tx=0x7f9068003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:40.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.167+0000 7f905f7fe640 1 -- 172.21.15.40:0/3565811710 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9068041020 con 0x7f90701098b0 2024-03-18T22:52:40.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.167+0000 7f905f7fe640 1 -- 172.21.15.40:0/3565811710 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90680046f0 con 0x7f90701098b0 2024-03-18T22:52:40.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.168+0000 7f905f7fe640 1 -- 172.21.15.40:0/3565811710 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9068039040 con 0x7f90701098b0 2024-03-18T22:52:40.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.169+0000 7f906f59e640 1 -- 172.21.15.40:0/3565811710 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90701b6ff0 con 0x7f90701098b0 2024-03-18T22:52:40.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.169+0000 7f906f59e640 1 -- 172.21.15.40:0/3565811710 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90701b7480 con 0x7f90701098b0 2024-03-18T22:52:40.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.172+0000 7f9077652740 1 Processor -- start 2024-03-18T22:52:40.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.175+0000 7f905f7fe640 1 -- 172.21.15.40:0/3565811710 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9068044020 con 0x7f90701098b0 2024-03-18T22:52:40.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.175+0000 7f905f7fe640 1 --2- 172.21.15.40:0/3565811710 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90580a2a10 0x7f90580a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.176+0000 7f905f7fe640 1 -- 172.21.15.40:0/3565811710 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f90680f0780 con 0x7f90701098b0 2024-03-18T22:52:40.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.177+0000 7f9077652740 1 -- start start 2024-03-18T22:52:40.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.177+0000 7f9077652740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280ee490 0x55da280ee8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.177+0000 7f9077652740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 0x55da280f3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.177+0000 7f9077652740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da280f6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.177+0000 7f9077652740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55da280524b0 con 0x55da280eee80 2024-03-18T22:52:40.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.177+0000 7f9077652740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55da27ffad70 con 0x55da280ee490 2024-03-18T22:52:40.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.177+0000 7f9077652740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55da2804e550 con 0x55da280f3ff0 2024-03-18T22:52:40.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.177+0000 7f906dd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 0x55da280f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.177+0000 7f906dd9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 0x55da280f3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32940/0 (socket says 172.21.15.40:32940) 2024-03-18T22:52:40.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.177+0000 7f906dd9b640 1 -- 172.21.15.40:0/379871240 learned_addr learned my addr 172.21.15.40:0/379871240 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:40.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.177+0000 7f906dd9b640 1 --2- 172.21.15.40:0/3565811710 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90580a2a10 0x7f90580a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.181+0000 7f906ed9d640 1 --2- 172.21.15.40:0/379871240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da280f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.182+0000 7f906e59c640 1 --2- 172.21.15.40:0/379871240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280ee490 0x55da280ee8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.183+0000 7f906e59c640 1 -- 172.21.15.40:0/379871240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 msgr2=0x55da280f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:40.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.183+0000 7f906e59c640 1 --2- 172.21.15.40:0/379871240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da280f6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.183+0000 7f906e59c640 1 -- 172.21.15.40:0/379871240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 msgr2=0x55da280f3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:40.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.183+0000 7f906e59c640 1 --2- 172.21.15.40:0/379871240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 0x55da280f3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.183+0000 7f906e59c640 1 -- 172.21.15.40:0/379871240 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da2801d210 con 0x55da280ee490 2024-03-18T22:52:40.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.183+0000 7f906e59c640 1 --2- 172.21.15.40:0/379871240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280ee490 0x55da280ee8b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f906000b730 tx=0x7f906000bbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94f6f0dceca51b3a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:40.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.185+0000 7f905cff9640 1 -- 172.21.15.40:0/379871240 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906000be40 con 0x55da280ee490 2024-03-18T22:52:40.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.185+0000 7f905cff9640 1 -- 172.21.15.40:0/379871240 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9060019070 con 0x55da280ee490 2024-03-18T22:52:40.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.186+0000 7f9077652740 1 -- 172.21.15.40:0/379871240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280ee490 msgr2=0x55da280ee8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.186+0000 7f906dd9b640 1 --2- 172.21.15.40:0/3565811710 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90580a2a10 0x7f90580a4ed0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x55da280ef280 tx=0x7f9064009240 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:40.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.186+0000 7f9077652740 1 --2- 172.21.15.40:0/379871240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280ee490 0x55da280ee8b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f906000b730 tx=0x7f906000bbf0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.186+0000 7f9077652740 1 -- 172.21.15.40:0/379871240 shutdown_connections 2024-03-18T22:52:40.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.186+0000 7f9077652740 1 --2- 172.21.15.40:0/379871240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da280f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.186+0000 7f9077652740 1 --2- 172.21.15.40:0/379871240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 0x55da280f3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.186+0000 7f9077652740 1 --2- 172.21.15.40:0/379871240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280ee490 0x55da280ee8b0 secure :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f906000b730 tx=0x7f906000bbf0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.186+0000 7f9077652740 1 -- 172.21.15.40:0/379871240 >> 172.21.15.40:0/379871240 conn(0x55da280e9a00 msgr2=0x55da280e9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:40.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.193+0000 7f9077652740 1 -- 172.21.15.40:0/379871240 shutdown_connections 2024-03-18T22:52:40.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.194+0000 7f9077652740 1 -- 172.21.15.40:0/379871240 wait complete. 2024-03-18T22:52:40.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.194+0000 7f9077652740 1 -- start start 2024-03-18T22:52:40.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.194+0000 7f9077652740 1 Processor -- start 2024-03-18T22:52:40.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.196+0000 7f9077652740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 0x55da280f8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.196+0000 7f9077652740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da280f9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.196+0000 7f9077652740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 0x55da28100460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.196+0000 7f9077652740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55da280f6ff0 con 0x55da280eee80 2024-03-18T22:52:40.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.196+0000 7f9077652740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55da2800e310 con 0x55da280fe070 2024-03-18T22:52:40.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.196+0000 7f9077652740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55da280f7170 con 0x55da280f3ff0 2024-03-18T22:52:40.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.196+0000 7f9077652740 10 client.0 _ll_register_callbacks cb 0x55da280022d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:40.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.199+0000 7f906dd9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da280f9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.201+0000 7f906dd9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da280f9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53722/0 (socket says 172.21.15.40:53722) 2024-03-18T22:52:40.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.201+0000 7f906dd9b640 1 -- 172.21.15.40:0/2012667794 learned_addr learned my addr 172.21.15.40:0/2012667794 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:40.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.202+0000 7f906ed9d640 1 --2- 172.21.15.40:0/2012667794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 0x55da28100460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.206+0000 7f906e59c640 1 --2- 172.21.15.40:0/2012667794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 0x55da280f8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.206+0000 7f906dd9b640 1 -- 172.21.15.40:0/2012667794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 msgr2=0x55da28100460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:40.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.206+0000 7f906dd9b640 1 --2- 172.21.15.40:0/2012667794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 0x55da28100460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.206+0000 7f906dd9b640 1 -- 172.21.15.40:0/2012667794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 msgr2=0x55da280f8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:40.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.206+0000 7f906dd9b640 1 --2- 172.21.15.40:0/2012667794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 0x55da280f8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.206+0000 7f906dd9b640 1 -- 172.21.15.40:0/2012667794 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da27d663f0 con 0x55da280f3ff0 2024-03-18T22:52:40.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.207+0000 7f906dd9b640 1 --2- 172.21.15.40:0/2012667794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da280f9040 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9064006f00 tx=0x7f90640047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94a652941457fecd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:40.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.207+0000 7f9055ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:52:40.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.207+0000 7f9055ffb640 1 -- 172.21.15.40:0/2012667794 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90640168a0 con 0x55da280f3ff0 2024-03-18T22:52:40.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.207+0000 7f9077652740 1 -- 172.21.15.40:0/2012667794 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9060019070 con 0x55da280f3ff0 2024-03-18T22:52:40.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.207+0000 7f9055ffb640 1 -- 172.21.15.40:0/2012667794 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9064016a40 con 0x55da280f3ff0 2024-03-18T22:52:40.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.207+0000 7f9055ffb640 1 -- 172.21.15.40:0/2012667794 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f9064016c40 con 0x55da280f3ff0 2024-03-18T22:52:40.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.207+0000 7f9077652740 10 client.27889 fetch_fsmap learned FSMap version 36 2024-03-18T22:52:40.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.207+0000 7f9077652740 1 -- 172.21.15.40:0/2012667794 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55da27e307d0 con 0x55da280f3ff0 2024-03-18T22:52:40.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.208+0000 7f9055ffb640 1 -- 172.21.15.40:0/2012667794 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9060019070 con 0x55da280f3ff0 2024-03-18T22:52:40.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.210+0000 7f9055ffb640 1 -- 172.21.15.40:0/2012667794 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 36) ==== 8378+0+0 (secure 0 0 0) 0x7f90640213f0 con 0x55da280f3ff0 2024-03-18T22:52:40.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.210+0000 7f9077652740 10 client.27889 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T22:52:40.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.210+0000 7f9077652740 10 client.27889 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:52:40.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.210+0000 7f9077652740 20 client.27889 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:40.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.214+0000 7f9077652740 1 --2- 172.21.15.40:0/2012667794 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55da28105240 0x55da28107680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.214+0000 7f9077652740 4 client.27889 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:40.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.214+0000 7f9077652740 1 -- 172.21.15.40:0/2012667794 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da280b03e0 con 0x55da28105240 2024-03-18T22:52:40.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.216+0000 7f906e59c640 1 --2- 172.21.15.40:0/2012667794 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55da28105240 0x55da28107680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.219+0000 7f906e59c640 1 --2- 172.21.15.40:0/2012667794 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55da28105240 0x55da28107680 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=80337b2666e85a07 server_cookie=2a8cbf69d2a2fff0 in_seq=0 out_seq=0 2024-03-18T22:52:40.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.219+0000 7f9055ffb640 10 client.27889 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:52:40.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.226+0000 7f9055ffb640 1 -- 172.21.15.40:0/2012667794 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55da280b03e0 con 0x55da28105240 2024-03-18T22:52:40.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.226+0000 7f9055ffb640 10 client.27889 handle_command_reply: tid=0 2024-03-18T22:52:40.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.226+0000 7f9077652740 1 client.27889 shutdown 2024-03-18T22:52:40.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.240+0000 7f9077652740 1 -- 172.21.15.40:0/2012667794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 msgr2=0x55da280f9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.240+0000 7f9077652740 1 --2- 172.21.15.40:0/2012667794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da280f9040 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9064006f00 tx=0x7f90640047d0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.240+0000 7f9077652740 1 -- 172.21.15.40:0/2012667794 shutdown_connections 2024-03-18T22:52:40.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.240+0000 7f9077652740 1 --2- 172.21.15.40:0/2012667794 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55da28105240 0x55da28107680 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.240+0000 7f9077652740 1 --2- 172.21.15.40:0/2012667794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 0x55da28100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.240+0000 7f9077652740 1 --2- 172.21.15.40:0/2012667794 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da280f9040 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.240+0000 7f9077652740 1 --2- 172.21.15.40:0/2012667794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 0x55da280f8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.240+0000 7f9077652740 1 -- 172.21.15.40:0/2012667794 >> 172.21.15.40:0/2012667794 conn(0x55da280e9a00 msgr2=0x55da280e8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:40.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.241+0000 7f9055ffb640 0 client.27889 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:52:40.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.241+0000 7f9077652740 1 -- 172.21.15.40:0/2012667794 shutdown_connections 2024-03-18T22:52:40.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.241+0000 7f9077652740 1 -- 172.21.15.40:0/2012667794 wait complete. 2024-03-18T22:52:40.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.248+0000 7f9077652740 20 client.27889 trim_cache size 0 max 16384 2024-03-18T22:52:40.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.282+0000 7f9077652740 1 Processor -- start 2024-03-18T22:52:40.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.284+0000 7f9077652740 1 -- start start 2024-03-18T22:52:40.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.284+0000 7f9077652740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 0x55da280e4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.284+0000 7f9077652740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da280f8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.284+0000 7f9077652740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 0x55da280f9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.284+0000 7f9077652740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f90701c1a20 con 0x55da280eee80 2024-03-18T22:52:40.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.284+0000 7f9077652740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f90701c3620 con 0x55da280fe070 2024-03-18T22:52:40.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.284+0000 7f9077652740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f90701c20f0 con 0x55da280f3ff0 2024-03-18T22:52:40.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.284+0000 7f906ed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 0x55da280f9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.284+0000 7f906ed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 0x55da280f9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36096/0 (socket says 172.21.15.40:36096) 2024-03-18T22:52:40.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.284+0000 7f906ed9d640 1 -- 172.21.15.40:0/860890854 learned_addr learned my addr 172.21.15.40:0/860890854 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:40.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.284+0000 7f906dd9b640 1 --2- 172.21.15.40:0/860890854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da280f8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.286+0000 7f906dd9b640 1 -- 172.21.15.40:0/860890854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 msgr2=0x55da280f9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:40.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.287+0000 7f906dd9b640 1 --2- 172.21.15.40:0/860890854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 0x55da280f9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.287+0000 7f906dd9b640 1 -- 172.21.15.40:0/860890854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 msgr2=0x55da280e4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:40.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.287+0000 7f906dd9b640 1 --2- 172.21.15.40:0/860890854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 0x55da280e4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.287+0000 7f906dd9b640 1 -- 172.21.15.40:0/860890854 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da280efef0 con 0x55da280f3ff0 2024-03-18T22:52:40.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.287+0000 7f906dd9b640 1 --2- 172.21.15.40:0/860890854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da280f8b00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f90640217b0 tx=0x7f9064009e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=815c8275bb8226b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:40.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.293+0000 7f9057fff640 1 -- 172.21.15.40:0/860890854 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90640167a0 con 0x55da280f3ff0 2024-03-18T22:52:40.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.293+0000 7f9057fff640 1 -- 172.21.15.40:0/860890854 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9064026070 con 0x55da280f3ff0 2024-03-18T22:52:40.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.293+0000 7f9057fff640 1 -- 172.21.15.40:0/860890854 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9064021d30 con 0x55da280f3ff0 2024-03-18T22:52:40.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.296+0000 7f9077652740 1 -- 172.21.15.40:0/860890854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 msgr2=0x55da280f8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.296+0000 7f9077652740 1 --2- 172.21.15.40:0/860890854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da280f8b00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f90640217b0 tx=0x7f9064009e10 comp rx=0 tx=0).stop 2024-03-18T22:52:40.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.298+0000 7f9077652740 1 -- 172.21.15.40:0/860890854 shutdown_connections 2024-03-18T22:52:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.298+0000 7f9077652740 1 --2- 172.21.15.40:0/860890854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 0x55da280f9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.298+0000 7f9077652740 1 --2- 172.21.15.40:0/860890854 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da280f8b00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.298+0000 7f9077652740 1 --2- 172.21.15.40:0/860890854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 0x55da280e4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.298+0000 7f9077652740 1 -- 172.21.15.40:0/860890854 >> 172.21.15.40:0/860890854 conn(0x55da280e9a00 msgr2=0x55da280e9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:40.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.306+0000 7f9077652740 1 -- 172.21.15.40:0/860890854 shutdown_connections 2024-03-18T22:52:40.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.306+0000 7f9077652740 1 -- 172.21.15.40:0/860890854 wait complete. 2024-03-18T22:52:40.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.307+0000 7f9077652740 1 -- start start 2024-03-18T22:52:40.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.307+0000 7f9077652740 1 Processor -- start 2024-03-18T22:52:40.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.307+0000 7f9077652740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 0x55da281156a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.308+0000 7f9077652740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da28115be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.308+0000 7f9077652740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 0x55da28168f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.308+0000 7f9077652740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f90701c2f50 con 0x55da280eee80 2024-03-18T22:52:40.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.308+0000 7f9077652740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f90701c27c0 con 0x55da280fe070 2024-03-18T22:52:40.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.308+0000 7f9077652740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f90701bfa00 con 0x55da280f3ff0 2024-03-18T22:52:40.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.308+0000 7f9077652740 10 client.0 _ll_register_callbacks cb 0x7f9064015f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:40.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.310+0000 7f906ed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 0x55da28168f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.310+0000 7f906ed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 0x55da28168f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36102/0 (socket says 172.21.15.40:36102) 2024-03-18T22:52:40.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.310+0000 7f906ed9d640 1 -- 172.21.15.40:0/2863456202 learned_addr learned my addr 172.21.15.40:0/2863456202 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:40.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.310+0000 7f906dd9b640 1 --2- 172.21.15.40:0/2863456202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da28115be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.310+0000 7f906ed9d640 1 -- 172.21.15.40:0/2863456202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 msgr2=0x55da28115be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:40.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.310+0000 7f906ed9d640 1 --2- 172.21.15.40:0/2863456202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da28115be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.310+0000 7f906ed9d640 1 -- 172.21.15.40:0/2863456202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 msgr2=0x55da281156a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.310+0000 7f906ed9d640 1 --2- 172.21.15.40:0/2863456202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 0x55da281156a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.310+0000 7f906ed9d640 1 -- 172.21.15.40:0/2863456202 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da281694b0 con 0x55da280fe070 2024-03-18T22:52:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.311+0000 7f906ed9d640 1 --2- 172.21.15.40:0/2863456202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 0x55da28168f70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9068037040 tx=0x7f90680329d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91f1ccb66ba8f180 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:40.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.311+0000 7f906dd9b640 1 --2- 172.21.15.40:0/2863456202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da28115be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:52:40.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.317+0000 7f9055ffb640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:52:40.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.317+0000 7f9055ffb640 1 -- 172.21.15.40:0/2863456202 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906804b050 con 0x55da280fe070 2024-03-18T22:52:40.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.317+0000 7f9055ffb640 1 -- 172.21.15.40:0/2863456202 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f90680345d0 con 0x55da280fe070 2024-03-18T22:52:40.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.318+0000 7f9077652740 1 -- 172.21.15.40:0/2863456202 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9064026070 con 0x55da280fe070 2024-03-18T22:52:40.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.320+0000 7f9055ffb640 1 -- 172.21.15.40:0/2863456202 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f90680c0e20 con 0x55da280fe070 2024-03-18T22:52:40.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.321+0000 7f9077652740 10 client.27908 fetch_fsmap learned FSMap version 36 2024-03-18T22:52:40.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.321+0000 7f9077652740 1 -- 172.21.15.40:0/2863456202 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55da2816a800 con 0x55da280fe070 2024-03-18T22:52:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.323+0000 7f9055ffb640 1 -- 172.21.15.40:0/2863456202 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9064026070 con 0x55da280fe070 2024-03-18T22:52:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.323+0000 7f9055ffb640 1 -- 172.21.15.40:0/2863456202 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 36) ==== 8378+0+0 (secure 0 0 0) 0x7f90680ef040 con 0x55da280fe070 2024-03-18T22:52:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.324+0000 7f9077652740 10 client.27908 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T22:52:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.324+0000 7f9077652740 10 client.27908 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:52:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.324+0000 7f9077652740 20 client.27908 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:40.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.324+0000 7f9077652740 1 --2- 172.21.15.40:0/2863456202 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55da2816b0f0 0x55da2816d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.324+0000 7f9077652740 4 client.27908 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:52:40.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.324+0000 7f9077652740 1 -- 172.21.15.40:0/2863456202 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55da2805f9f0 con 0x55da2816b0f0 2024-03-18T22:52:40.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.327+0000 7f906e59c640 1 --2- 172.21.15.40:0/2863456202 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55da2816b0f0 0x55da2816d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.333+0000 7f906e59c640 1 --2- 172.21.15.40:0/2863456202 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55da2816b0f0 0x55da2816d510 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c46dda7bfe91f5d2 server_cookie=8731ed8120fa0082 in_seq=0 out_seq=0 2024-03-18T22:52:40.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.333+0000 7f9055ffb640 10 client.27908 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:52:40.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.335+0000 7f9055ffb640 1 -- 172.21.15.40:0/2863456202 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55da2805f9f0 con 0x55da2816b0f0 2024-03-18T22:52:40.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.335+0000 7f9055ffb640 10 client.27908 handle_command_reply: tid=0 2024-03-18T22:52:40.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.335+0000 7f9077652740 1 client.27908 shutdown 2024-03-18T22:52:40.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.337+0000 7f9077652740 1 -- 172.21.15.40:0/2863456202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 msgr2=0x55da28168f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.337+0000 7f9077652740 1 --2- 172.21.15.40:0/2863456202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 0x55da28168f70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9068037040 tx=0x7f90680329d0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.339+0000 7f9077652740 1 -- 172.21.15.40:0/2863456202 shutdown_connections 2024-03-18T22:52:40.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.339+0000 7f9077652740 1 --2- 172.21.15.40:0/2863456202 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55da2816b0f0 0x55da2816d510 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.339+0000 7f9077652740 1 --2- 172.21.15.40:0/2863456202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da280fe070 0x55da28168f70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.339+0000 7f9077652740 1 --2- 172.21.15.40:0/2863456202 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da280f3ff0 0x55da28115be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.339+0000 7f9077652740 1 --2- 172.21.15.40:0/2863456202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da280eee80 0x55da281156a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.339+0000 7f9077652740 1 -- 172.21.15.40:0/2863456202 >> 172.21.15.40:0/2863456202 conn(0x55da280e9a00 msgr2=0x55da280e4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:40.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.341+0000 7f9055ffb640 0 client.27908 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:52:40.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.343+0000 7f9077652740 1 -- 172.21.15.40:0/2863456202 shutdown_connections 2024-03-18T22:52:40.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.343+0000 7f9077652740 1 -- 172.21.15.40:0/2863456202 wait complete. 2024-03-18T22:52:40.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.346+0000 7f9077652740 20 client.27908 trim_cache size 0 max 16384 2024-03-18T22:52:40.354 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:52:40.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.354+0000 7f906f59e640 1 -- 172.21.15.40:0/3565811710 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90580a2a10 msgr2=0x7f90580a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.355+0000 7f906f59e640 1 --2- 172.21.15.40:0/3565811710 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90580a2a10 0x7f90580a4ed0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x55da280ef280 tx=0x7f9064009240 comp rx=0 tx=0).stop 2024-03-18T22:52:40.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.355+0000 7f906f59e640 1 -- 172.21.15.40:0/3565811710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90701098b0 msgr2=0x7f90701b6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.355+0000 7f906f59e640 1 --2- 172.21.15.40:0/3565811710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90701098b0 0x7f90701b6880 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9068009bf0 tx=0x7f9068003f10 comp rx=0 tx=0).stop 2024-03-18T22:52:40.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.355+0000 7f906f59e640 1 -- 172.21.15.40:0/3565811710 shutdown_connections 2024-03-18T22:52:40.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.355+0000 7f906f59e640 1 --2- 172.21.15.40:0/3565811710 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f90580a2a10 0x7f90580a4ed0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.355+0000 7f906f59e640 1 --2- 172.21.15.40:0/3565811710 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f90701098b0 0x7f90701b6880 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.355+0000 7f906f59e640 1 --2- 172.21.15.40:0/3565811710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9070108ef0 0x7f9070115480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.355+0000 7f906f59e640 1 --2- 172.21.15.40:0/3565811710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9070072c30 0x7f9070114f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.355+0000 7f906f59e640 1 -- 172.21.15.40:0/3565811710 >> 172.21.15.40:0/3565811710 conn(0x7f907006e700 msgr2=0x7f907006f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:40.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.358+0000 7f906f59e640 1 -- 172.21.15.40:0/3565811710 shutdown_connections 2024-03-18T22:52:40.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.359+0000 7f906f59e640 1 -- 172.21.15.40:0/3565811710 wait complete. 2024-03-18T22:52:40.366 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:40 smithi040 ceph-mon[62643]: pgmap v651: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 17 MiB/s wr, 4.23k op/s 2024-03-18T22:52:40.408 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:52:40.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.685+0000 7f5a12030640 1 -- 172.21.15.40:0/4140321593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a0c1098b0 msgr2=0x7f5a0c10e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.685+0000 7f5a12030640 1 --2- 172.21.15.40:0/4140321593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a0c1098b0 0x7f5a0c10e150 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009be0 tx=0x7f5a0402bef0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.685+0000 7f5a12030640 1 -- 172.21.15.40:0/4140321593 shutdown_connections 2024-03-18T22:52:40.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.685+0000 7f5a12030640 1 --2- 172.21.15.40:0/4140321593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a0c1098b0 0x7f5a0c10e150 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.685+0000 7f5a12030640 1 --2- 172.21.15.40:0/4140321593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a0c108ef0 0x7f5a0c109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.685+0000 7f5a12030640 1 --2- 172.21.15.40:0/4140321593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a0c072c30 0x7f5a0c073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.685+0000 7f5a12030640 1 -- 172.21.15.40:0/4140321593 >> 172.21.15.40:0/4140321593 conn(0x7f5a0c06e700 msgr2=0x7f5a0c070b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.686+0000 7f5a12030640 1 -- 172.21.15.40:0/4140321593 shutdown_connections 2024-03-18T22:52:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.686+0000 7f5a12030640 1 -- 172.21.15.40:0/4140321593 wait complete. 2024-03-18T22:52:40.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.687+0000 7f5a12030640 1 Processor -- start 2024-03-18T22:52:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.687+0000 7f5a12030640 1 -- start start 2024-03-18T22:52:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.687+0000 7f5a12030640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a0c072c30 0x7f5a0c114ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.687+0000 7f5a12030640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a0c108ef0 0x7f5a0c115430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.687+0000 7f5a12030640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a0c1098b0 0x7f5a0c1a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.687+0000 7f5a12030640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c11c0b0 con 0x7f5a0c072c30 2024-03-18T22:52:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.687+0000 7f5a12030640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c11bf30 con 0x7f5a0c1098b0 2024-03-18T22:52:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.687+0000 7f5a12030640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c11c230 con 0x7f5a0c108ef0 2024-03-18T22:52:40.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.688+0000 7f5a1082d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a0c108ef0 0x7f5a0c115430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.688+0000 7f5a1082d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a0c108ef0 0x7f5a0c115430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53762/0 (socket says 172.21.15.40:53762) 2024-03-18T22:52:40.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.688+0000 7f5a1082d640 1 -- 172.21.15.40:0/4253265262 learned_addr learned my addr 172.21.15.40:0/4253265262 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:40.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.688+0000 7f5a1182f640 1 --2- 172.21.15.40:0/4253265262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a0c1098b0 0x7f5a0c1a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.688+0000 7f5a1082d640 1 -- 172.21.15.40:0/4253265262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a0c1098b0 msgr2=0x7f5a0c1a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.688+0000 7f5a1082d640 1 --2- 172.21.15.40:0/4253265262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a0c1098b0 0x7f5a0c1a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.688+0000 7f5a1082d640 1 -- 172.21.15.40:0/4253265262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a0c072c30 msgr2=0x7f5a0c114ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.688+0000 7f5a1082d640 1 --2- 172.21.15.40:0/4253265262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a0c072c30 0x7f5a0c114ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.688+0000 7f5a1082d640 1 -- 172.21.15.40:0/4253265262 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0c1a5c90 con 0x7f5a0c108ef0 2024-03-18T22:52:40.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.689+0000 7f5a1082d640 1 --2- 172.21.15.40:0/4253265262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a0c108ef0 0x7f5a0c115430 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00e9f0 tx=0x7f59fc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:40.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.696+0000 7f5a027fc640 1 -- 172.21.15.40:0/4253265262 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc00cd80 con 0x7f5a0c108ef0 2024-03-18T22:52:40.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.696+0000 7f5a027fc640 1 -- 172.21.15.40:0/4253265262 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59fc0108e0 con 0x7f5a0c108ef0 2024-03-18T22:52:40.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.696+0000 7f5a027fc640 1 -- 172.21.15.40:0/4253265262 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc0188e0 con 0x7f5a0c108ef0 2024-03-18T22:52:40.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.696+0000 7f5a12030640 1 -- 172.21.15.40:0/4253265262 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a0c1a5f80 con 0x7f5a0c108ef0 2024-03-18T22:52:40.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.696+0000 7f5a12030640 1 -- 172.21.15.40:0/4253265262 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a0c1a6480 con 0x7f5a0c108ef0 2024-03-18T22:52:40.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.702+0000 7f5a15e7d740 1 Processor -- start 2024-03-18T22:52:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.702+0000 7f5a15e7d740 1 -- start start 2024-03-18T22:52:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.702+0000 7f5a15e7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47490 0x5634f8b478b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.702+0000 7f5a15e7d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47e80 0x5634f8b4c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.703+0000 7f5a15e7d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b4cff0 0x5634f8b4f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.703+0000 7f5a15e7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5634f8aab4b0 con 0x5634f8b47490 2024-03-18T22:52:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.703+0000 7f5a15e7d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5634f8a53d70 con 0x5634f8b4cff0 2024-03-18T22:52:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.703+0000 7f5a15e7d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5634f8aa7550 con 0x5634f8b47e80 2024-03-18T22:52:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.704+0000 7f5a1082d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b4cff0 0x5634f8b4f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.704+0000 7f5a1102e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47e80 0x5634f8b4c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.704+0000 7f5a1102e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47e80 0x5634f8b4c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53766/0 (socket says 172.21.15.40:53766) 2024-03-18T22:52:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.704+0000 7f5a1102e640 1 -- 172.21.15.40:0/383426495 learned_addr learned my addr 172.21.15.40:0/383426495 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.704+0000 7f5a1082d640 1 -- 172.21.15.40:0/383426495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47e80 msgr2=0x5634f8b4c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.704+0000 7f5a1082d640 1 --2- 172.21.15.40:0/383426495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47e80 0x5634f8b4c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.704+0000 7f5a1082d640 1 -- 172.21.15.40:0/383426495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47490 msgr2=0x5634f8b478b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:40.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.704+0000 7f5a1082d640 1 --2- 172.21.15.40:0/383426495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47490 0x5634f8b478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.704+0000 7f5a1082d640 1 -- 172.21.15.40:0/383426495 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634f8a76210 con 0x5634f8b4cff0 2024-03-18T22:52:40.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.704+0000 7f5a1082d640 1 --2- 172.21.15.40:0/383426495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b4cff0 0x5634f8b4f480 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f59fc0cc910 tx=0x7f59fc0d7910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9f7100d0ea50ce2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:40.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.706+0000 7f59f77fe640 1 -- 172.21.15.40:0/383426495 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc0e2070 con 0x5634f8b4cff0 2024-03-18T22:52:40.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.706+0000 7f59f77fe640 1 -- 172.21.15.40:0/383426495 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59fc0d14e0 con 0x5634f8b4cff0 2024-03-18T22:52:40.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.706+0000 7f59f77fe640 1 -- 172.21.15.40:0/383426495 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc0dda10 con 0x5634f8b4cff0 2024-03-18T22:52:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.706+0000 7f5a027fc640 1 -- 172.21.15.40:0/4253265262 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f59fc018a80 con 0x7f5a0c108ef0 2024-03-18T22:52:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.707+0000 7f5a15e7d740 1 -- 172.21.15.40:0/383426495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b4cff0 msgr2=0x5634f8b4f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.707+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/383426495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b4cff0 0x5634f8b4f480 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f59fc0cc910 tx=0x7f59fc0d7910 comp rx=0 tx=0).stop 2024-03-18T22:52:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.707+0000 7f5a027fc640 1 --2- 172.21.15.40:0/4253265262 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f59f80a29d0 0x7f59f80a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.708+0000 7f5a1182f640 1 --2- 172.21.15.40:0/4253265262 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f59f80a29d0 0x7f59f80a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.708+0000 7f5a027fc640 1 -- 172.21.15.40:0/4253265262 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f59fc0cf2c0 con 0x7f5a0c108ef0 2024-03-18T22:52:40.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.712+0000 7f5a15e7d740 1 -- 172.21.15.40:0/383426495 shutdown_connections 2024-03-18T22:52:40.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.712+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/383426495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b4cff0 0x5634f8b4f480 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.712+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/383426495 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47e80 0x5634f8b4c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.712+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/383426495 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47490 0x5634f8b478b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.712+0000 7f5a15e7d740 1 -- 172.21.15.40:0/383426495 >> 172.21.15.40:0/383426495 conn(0x5634f8b42a00 msgr2=0x5634f8b42e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:40.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.715+0000 7f5a1182f640 1 --2- 172.21.15.40:0/4253265262 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f59f80a29d0 0x7f59f80a4e90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5a04006010 tx=0x7f5a04009840 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:40.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.715+0000 7f5a15e7d740 1 -- 172.21.15.40:0/383426495 shutdown_connections 2024-03-18T22:52:40.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.717+0000 7f5a15e7d740 1 -- 172.21.15.40:0/383426495 wait complete. 2024-03-18T22:52:40.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.718+0000 7f5a15e7d740 1 -- start start 2024-03-18T22:52:40.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.718+0000 7f5a15e7d740 1 Processor -- start 2024-03-18T22:52:40.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.719+0000 7f5a15e7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47490 0x5634f8b51a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.719+0000 7f5a15e7d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b47e80 0x5634f8b51f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.719+0000 7f5a15e7d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b4cff0 0x5634f8b58f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.719+0000 7f5a15e7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5634f8b4ffa0 con 0x5634f8b47490 2024-03-18T22:52:40.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.719+0000 7f5a15e7d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5634f8a67310 con 0x5634f8b47e80 2024-03-18T22:52:40.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.719+0000 7f5a15e7d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5634f8b50120 con 0x5634f8b4cff0 2024-03-18T22:52:40.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.719+0000 7f5a15e7d740 10 client.0 _ll_register_callbacks cb 0x5634f8a5b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:40.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.721+0000 7f5a1082d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b47e80 0x5634f8b51f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.721+0000 7f5a1082d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b47e80 0x5634f8b51f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36130/0 (socket says 172.21.15.40:36130) 2024-03-18T22:52:40.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.721+0000 7f5a1082d640 1 -- 172.21.15.40:0/1677993589 learned_addr learned my addr 172.21.15.40:0/1677993589 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:40.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.721+0000 7f5a1082d640 1 -- 172.21.15.40:0/1677993589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b4cff0 msgr2=0x5634f8b58f80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:52:40.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.721+0000 7f5a1082d640 1 --2- 172.21.15.40:0/1677993589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b4cff0 0x5634f8b58f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.721+0000 7f5a1082d640 1 -- 172.21.15.40:0/1677993589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47490 msgr2=0x5634f8b51a00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:52:40.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.721+0000 7f5a1082d640 1 --2- 172.21.15.40:0/1677993589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47490 0x5634f8b51a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.721+0000 7f5a1082d640 1 -- 172.21.15.40:0/1677993589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634f87bf3f0 con 0x5634f8b47e80 2024-03-18T22:52:40.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.722+0000 7f5a1082d640 1 --2- 172.21.15.40:0/1677993589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b47e80 0x5634f8b51f40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f59fc09b7a0 tx=0x7f59fc0d7f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=865de26f62049674 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:40.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.722+0000 7f59f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:52:40.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.725+0000 7f59f4ff9640 1 -- 172.21.15.40:0/1677993589 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc097ab0 con 0x5634f8b47e80 2024-03-18T22:52:40.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.725+0000 7f5a15e7d740 1 -- 172.21.15.40:0/1677993589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59fc0d14e0 con 0x5634f8b47e80 2024-03-18T22:52:40.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.725+0000 7f59f4ff9640 1 -- 172.21.15.40:0/1677993589 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f59fc0cca80 con 0x5634f8b47e80 2024-03-18T22:52:40.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.725+0000 7f59f4ff9640 1 -- 172.21.15.40:0/1677993589 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f59fc09b170 con 0x5634f8b47e80 2024-03-18T22:52:40.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.729+0000 7f5a15e7d740 10 client.27920 fetch_fsmap learned FSMap version 36 2024-03-18T22:52:40.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.729+0000 7f5a15e7d740 1 -- 172.21.15.40:0/1677993589 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5634f88897d0 con 0x5634f8b47e80 2024-03-18T22:52:40.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.731+0000 7f59f4ff9640 1 -- 172.21.15.40:0/1677993589 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59fc0d14e0 con 0x5634f8b47e80 2024-03-18T22:52:40.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.731+0000 7f59f4ff9640 1 -- 172.21.15.40:0/1677993589 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 36) ==== 8378+0+0 (secure 0 0 0) 0x7f59fc09ba70 con 0x5634f8b47e80 2024-03-18T22:52:40.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.731+0000 7f5a15e7d740 10 client.27920 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T22:52:40.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.731+0000 7f5a15e7d740 10 client.27920 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:40.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.731+0000 7f5a15e7d740 20 client.27920 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:40.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.731+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/1677993589 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5634f8b5d470 0x5634f8b5f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.731+0000 7f5a15e7d740 4 client.27920 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:40.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.731+0000 7f5a15e7d740 1 -- 172.21.15.40:0/1677993589 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634f8b093e0 con 0x5634f8b5d470 2024-03-18T22:52:40.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.733+0000 7f5a1102e640 1 --2- 172.21.15.40:0/1677993589 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5634f8b5d470 0x5634f8b5f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.734+0000 7f5a1102e640 1 --2- 172.21.15.40:0/1677993589 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5634f8b5d470 0x5634f8b5f860 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dd7acfb9ddd6585 server_cookie=e59d83c1f9f819be in_seq=0 out_seq=0 2024-03-18T22:52:40.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.735+0000 7f59f4ff9640 10 client.27920 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:40.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.735+0000 7f59f4ff9640 1 -- 172.21.15.40:0/1677993589 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5634f8b093e0 con 0x5634f8b5d470 2024-03-18T22:52:40.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.735+0000 7f59f4ff9640 10 client.27920 handle_command_reply: tid=0 2024-03-18T22:52:40.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.735+0000 7f5a15e7d740 1 client.27920 shutdown 2024-03-18T22:52:40.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.740+0000 7f5a15e7d740 1 -- 172.21.15.40:0/1677993589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b47e80 msgr2=0x5634f8b51f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.740+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/1677993589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b47e80 0x5634f8b51f40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f59fc09b7a0 tx=0x7f59fc0d7f90 comp rx=0 tx=0).stop 2024-03-18T22:52:40.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.740+0000 7f5a15e7d740 1 -- 172.21.15.40:0/1677993589 shutdown_connections 2024-03-18T22:52:40.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.740+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/1677993589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b4cff0 0x5634f8b58f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.740+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/1677993589 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5634f8b5d470 0x5634f8b5f860 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.740+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/1677993589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b47e80 0x5634f8b51f40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.740+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/1677993589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47490 0x5634f8b51a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.740+0000 7f5a15e7d740 1 -- 172.21.15.40:0/1677993589 >> 172.21.15.40:0/1677993589 conn(0x5634f8b42a00 msgr2=0x5634f8b41d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:40.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.741+0000 7f59f4ff9640 0 client.27920 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:40.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.741+0000 7f5a15e7d740 1 -- 172.21.15.40:0/1677993589 shutdown_connections 2024-03-18T22:52:40.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.741+0000 7f5a15e7d740 1 -- 172.21.15.40:0/1677993589 wait complete. 2024-03-18T22:52:40.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.742+0000 7f5a15e7d740 20 client.27920 trim_cache size 0 max 16384 2024-03-18T22:52:40.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.788+0000 7f5a15e7d740 1 Processor -- start 2024-03-18T22:52:40.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.788+0000 7f5a15e7d740 1 -- start start 2024-03-18T22:52:40.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.789+0000 7f5a15e7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47490 0x5634f8b49380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.789+0000 7f5a15e7d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47e80 0x5634f8b498c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.789+0000 7f5a15e7d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b4cff0 0x5634f8b58f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.789+0000 7f5a15e7d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f59f000fe70 con 0x5634f8b47490 2024-03-18T22:52:40.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.789+0000 7f5a1082d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47e80 0x5634f8b498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.789+0000 7f5a1082d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47e80 0x5634f8b498c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53788/0 (socket says 172.21.15.40:53788) 2024-03-18T22:52:40.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.789+0000 7f5a1082d640 1 -- 172.21.15.40:0/1984088558 learned_addr learned my addr 172.21.15.40:0/1984088558 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:40.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.789+0000 7f5a15e7d740 1 -- 172.21.15.40:0/1984088558 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f59f0011a70 con 0x5634f8b4cff0 2024-03-18T22:52:40.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.789+0000 7f5a1182f640 1 --2- 172.21.15.40:0/1984088558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b4cff0 0x5634f8b58f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.789+0000 7f5a15e7d740 1 -- 172.21.15.40:0/1984088558 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f59f0010540 con 0x5634f8b47e80 2024-03-18T22:52:40.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.790+0000 7f5a1182f640 1 -- 172.21.15.40:0/1984088558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47e80 msgr2=0x5634f8b498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:40.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.790+0000 7f5a1182f640 1 --2- 172.21.15.40:0/1984088558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47e80 0x5634f8b498c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.790+0000 7f5a1182f640 1 -- 172.21.15.40:0/1984088558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47490 msgr2=0x5634f8b49380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:52:40.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.790+0000 7f5a1182f640 1 --2- 172.21.15.40:0/1984088558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47490 0x5634f8b49380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.790+0000 7f5a1182f640 1 -- 172.21.15.40:0/1984088558 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634f8b5fd10 con 0x5634f8b4cff0 2024-03-18T22:52:40.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.790+0000 7f5a1182f640 1 --2- 172.21.15.40:0/1984088558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b4cff0 0x5634f8b58f40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5a04036040 tx=0x7f5a0403e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca14f8fda57d118d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:40.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.793+0000 7f59f6ffd640 1 -- 172.21.15.40:0/1984088558 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04051070 con 0x5634f8b4cff0 2024-03-18T22:52:40.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.793+0000 7f59f6ffd640 1 -- 172.21.15.40:0/1984088558 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a04048040 con 0x5634f8b4cff0 2024-03-18T22:52:40.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.794+0000 7f5a15e7d740 1 -- 172.21.15.40:0/1984088558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b4cff0 msgr2=0x5634f8b58f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.794+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/1984088558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b4cff0 0x5634f8b58f40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5a04036040 tx=0x7f5a0403e000 comp rx=0 tx=0).stop 2024-03-18T22:52:40.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.795+0000 7f5a15e7d740 1 -- 172.21.15.40:0/1984088558 shutdown_connections 2024-03-18T22:52:40.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.795+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/1984088558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b4cff0 0x5634f8b58f40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.795+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/1984088558 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47e80 0x5634f8b498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.795+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/1984088558 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47490 0x5634f8b49380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.795+0000 7f5a15e7d740 1 -- 172.21.15.40:0/1984088558 >> 172.21.15.40:0/1984088558 conn(0x5634f8b42a00 msgr2=0x5634f8b51a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:40.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.795+0000 7f5a15e7d740 1 -- 172.21.15.40:0/1984088558 shutdown_connections 2024-03-18T22:52:40.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.795+0000 7f5a15e7d740 1 -- 172.21.15.40:0/1984088558 wait complete. 2024-03-18T22:52:40.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.796+0000 7f5a15e7d740 1 -- start start 2024-03-18T22:52:40.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.796+0000 7f5a15e7d740 1 Processor -- start 2024-03-18T22:52:40.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.797+0000 7f5a15e7d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47490 0x5634f8b56490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.797+0000 7f5a15e7d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47e80 0x5634f8b569d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.797+0000 7f5a15e7d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b579e0 0x5634f8b57e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.797+0000 7f5a1082d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47e80 0x5634f8b569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.800+0000 7f5a1102e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47490 0x5634f8b56490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.803+0000 7f5a1082d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47e80 0x5634f8b569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32994/0 (socket says 172.21.15.40:32994) 2024-03-18T22:52:40.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.803+0000 7f5a1102e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47490 0x5634f8b56490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53792/0 (socket says 172.21.15.40:53792) 2024-03-18T22:52:40.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.803+0000 7f5a1082d640 1 -- 172.21.15.40:0/3430469712 learned_addr learned my addr 172.21.15.40:0/3430469712 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:40.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.803+0000 7f5a15e7d740 1 -- 172.21.15.40:0/3430469712 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f59f00113a0 con 0x5634f8b47e80 2024-03-18T22:52:40.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.803+0000 7f5a15e7d740 1 -- 172.21.15.40:0/3430469712 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f59f0010c10 con 0x5634f8b579e0 2024-03-18T22:52:40.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.803+0000 7f5a1182f640 1 --2- 172.21.15.40:0/3430469712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b579e0 0x5634f8b57e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.803+0000 7f5a15e7d740 1 -- 172.21.15.40:0/3430469712 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f59f000de50 con 0x5634f8b47490 2024-03-18T22:52:40.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.804+0000 7f5a15e7d740 10 client.0 _ll_register_callbacks cb 0x7f59fc0e0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:40.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.804+0000 7f5a1182f640 1 -- 172.21.15.40:0/3430469712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47490 msgr2=0x5634f8b56490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:40.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.804+0000 7f5a1182f640 1 --2- 172.21.15.40:0/3430469712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47490 0x5634f8b56490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.804+0000 7f5a1182f640 1 -- 172.21.15.40:0/3430469712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47e80 msgr2=0x5634f8b569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:40.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.804+0000 7f5a1182f640 1 --2- 172.21.15.40:0/3430469712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47e80 0x5634f8b569d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.804+0000 7f5a1182f640 1 -- 172.21.15.40:0/3430469712 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634f8b583a0 con 0x5634f8b579e0 2024-03-18T22:52:40.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.804+0000 7f5a1182f640 1 --2- 172.21.15.40:0/3430469712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b579e0 0x5634f8b57e60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5a0402c430 tx=0x7f5a0402c5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84bd13cb9fcb2f06 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:40.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.804+0000 7f59f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:52:40.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.807+0000 7f59f4ff9640 1 -- 172.21.15.40:0/3430469712 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04051070 con 0x5634f8b579e0 2024-03-18T22:52:40.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.807+0000 7f5a15e7d740 1 -- 172.21.15.40:0/3430469712 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5634f8ab89f0 con 0x5634f8b579e0 2024-03-18T22:52:40.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.808+0000 7f59f4ff9640 1 -- 172.21.15.40:0/3430469712 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5a0402ca90 con 0x5634f8b579e0 2024-03-18T22:52:40.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.810+0000 7f59f4ff9640 1 -- 172.21.15.40:0/3430469712 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f5a0405d050 con 0x5634f8b579e0 2024-03-18T22:52:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.811+0000 7f5a15e7d740 10 client.27926 fetch_fsmap learned FSMap version 36 2024-03-18T22:52:40.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.811+0000 7f5a15e7d740 1 -- 172.21.15.40:0/3430469712 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5634f8bc3750 con 0x5634f8b579e0 2024-03-18T22:52:40.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.814+0000 7f59f4ff9640 1 -- 172.21.15.40:0/3430469712 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a04048070 con 0x5634f8b579e0 2024-03-18T22:52:40.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.815+0000 7f59f4ff9640 1 -- 172.21.15.40:0/3430469712 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 36) ==== 8378+0+0 (secure 0 0 0) 0x7f5a04041480 con 0x5634f8b579e0 2024-03-18T22:52:40.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.816+0000 7f5a15e7d740 10 client.27926 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T22:52:40.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.816+0000 7f5a15e7d740 10 client.27926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:40.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.816+0000 7f5a15e7d740 20 client.27926 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:40.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.816+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/3430469712 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5634f8bc3f90 0x5634f8bc63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:40.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.816+0000 7f5a15e7d740 4 client.27926 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:52:40.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.816+0000 7f5a15e7d740 1 -- 172.21.15.40:0/3430469712 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5634f8ab89f0 con 0x5634f8bc3f90 2024-03-18T22:52:40.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.823+0000 7f5a1102e640 1 --2- 172.21.15.40:0/3430469712 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5634f8bc3f90 0x5634f8bc63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:40.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:40 smithi113 ceph-mon[66775]: pgmap v651: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 17 MiB/s wr, 4.23k op/s 2024-03-18T22:52:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.830+0000 7f5a1102e640 1 --2- 172.21.15.40:0/3430469712 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5634f8bc3f90 0x5634f8bc63b0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=810ff2ac77cc79a5 server_cookie=682ade328c19c531 in_seq=0 out_seq=0 2024-03-18T22:52:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.830+0000 7f59f4ff9640 10 client.27926 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.830+0000 7f59f4ff9640 1 -- 172.21.15.40:0/3430469712 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5634f8ab89f0 con 0x5634f8bc3f90 2024-03-18T22:52:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.831+0000 7f59f4ff9640 10 client.27926 handle_command_reply: tid=0 2024-03-18T22:52:40.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.831+0000 7f5a15e7d740 1 client.27926 shutdown 2024-03-18T22:52:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.835+0000 7f5a15e7d740 1 -- 172.21.15.40:0/3430469712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b579e0 msgr2=0x5634f8b57e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.835+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/3430469712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b579e0 0x5634f8b57e60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5a0402c430 tx=0x7f5a0402c5e0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.836+0000 7f5a15e7d740 1 -- 172.21.15.40:0/3430469712 shutdown_connections 2024-03-18T22:52:40.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.836+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/3430469712 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5634f8bc3f90 0x5634f8bc63b0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.836+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/3430469712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5634f8b579e0 0x5634f8b57e60 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.836+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/3430469712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5634f8b47e80 0x5634f8b569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.837+0000 7f5a15e7d740 1 --2- 172.21.15.40:0/3430469712 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5634f8b47490 0x5634f8b56490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.837+0000 7f5a15e7d740 1 -- 172.21.15.40:0/3430469712 >> 172.21.15.40:0/3430469712 conn(0x5634f8b42a00 msgr2=0x5634f8b51a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.837+0000 7f59f4ff9640 0 client.27926 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.837+0000 7f5a15e7d740 1 -- 172.21.15.40:0/3430469712 shutdown_connections 2024-03-18T22:52:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.837+0000 7f5a15e7d740 1 -- 172.21.15.40:0/3430469712 wait complete. 2024-03-18T22:52:40.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.837+0000 7f5a15e7d740 20 client.27926 trim_cache size 0 max 16384 2024-03-18T22:52:40.842 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:52:40.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.841+0000 7f5a12030640 1 -- 172.21.15.40:0/4253265262 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f59f80a29d0 msgr2=0x7f59f80a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.841+0000 7f5a12030640 1 --2- 172.21.15.40:0/4253265262 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f59f80a29d0 0x7f59f80a4e90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5a04006010 tx=0x7f5a04009840 comp rx=0 tx=0).stop 2024-03-18T22:52:40.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.841+0000 7f5a12030640 1 -- 172.21.15.40:0/4253265262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a0c108ef0 msgr2=0x7f5a0c115430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:40.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.841+0000 7f5a12030640 1 --2- 172.21.15.40:0/4253265262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a0c108ef0 0x7f5a0c115430 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00e9f0 tx=0x7f59fc00eeb0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.841+0000 7f5a12030640 1 -- 172.21.15.40:0/4253265262 shutdown_connections 2024-03-18T22:52:40.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.841+0000 7f5a12030640 1 --2- 172.21.15.40:0/4253265262 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f59f80a29d0 0x7f59f80a4e90 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.841+0000 7f5a12030640 1 --2- 172.21.15.40:0/4253265262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a0c1098b0 0x7f5a0c1a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.841+0000 7f5a12030640 1 --2- 172.21.15.40:0/4253265262 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a0c108ef0 0x7f5a0c115430 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.841+0000 7f5a12030640 1 --2- 172.21.15.40:0/4253265262 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a0c072c30 0x7f5a0c114ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:40.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.841+0000 7f5a12030640 1 -- 172.21.15.40:0/4253265262 >> 172.21.15.40:0/4253265262 conn(0x7f5a0c06e700 msgr2=0x7f5a0c06f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:40.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.841+0000 7f5a12030640 1 -- 172.21.15.40:0/4253265262 shutdown_connections 2024-03-18T22:52:40.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:40.842+0000 7f5a12030640 1 -- 172.21.15.40:0/4253265262 wait complete. 2024-03-18T22:52:40.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:40 smithi103 ceph-mon[66377]: pgmap v651: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 17 MiB/s wr, 4.23k op/s 2024-03-18T22:52:41.881 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:52:41.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:52:41.882 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:52:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.145+0000 7fbe52edc640 1 -- 172.21.15.40:0/3804801072 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe4c10b190 msgr2=0x7fbe4c10d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.145+0000 7fbe52edc640 1 --2- 172.21.15.40:0/3804801072 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe4c10b190 0x7fbe4c10d600 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009900 tx=0x7fbe4002bc80 comp rx=0 tx=0).stop 2024-03-18T22:52:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.145+0000 7fbe52edc640 1 -- 172.21.15.40:0/3804801072 shutdown_connections 2024-03-18T22:52:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.145+0000 7fbe52edc640 1 --2- 172.21.15.40:0/3804801072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe4c10db40 0x7fbe4c10dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.145+0000 7fbe52edc640 1 --2- 172.21.15.40:0/3804801072 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe4c10b190 0x7fbe4c10d600 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.145+0000 7fbe52edc640 1 --2- 172.21.15.40:0/3804801072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe4c108820 0x7fbe4c10ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.145+0000 7fbe52edc640 1 -- 172.21.15.40:0/3804801072 >> 172.21.15.40:0/3804801072 conn(0x7fbe4c06e680 msgr2=0x7fbe4c070ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:42.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.148+0000 7fbe52edc640 1 -- 172.21.15.40:0/3804801072 shutdown_connections 2024-03-18T22:52:42.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.148+0000 7fbe52edc640 1 -- 172.21.15.40:0/3804801072 wait complete. 2024-03-18T22:52:42.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.149+0000 7fbe52edc640 1 Processor -- start 2024-03-18T22:52:42.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.149+0000 7fbe52edc640 1 -- start start 2024-03-18T22:52:42.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.149+0000 7fbe52edc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe4c108820 0x7fbe4c114e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.149+0000 7fbe52edc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe4c10b190 0x7fbe4c115370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.149+0000 7fbe52edc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe4c10db40 0x7fbe4c1a58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.149+0000 7fbe52edc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4c11c1a0 con 0x7fbe4c108820 2024-03-18T22:52:42.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.149+0000 7fbe52edc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4c11c020 con 0x7fbe4c10db40 2024-03-18T22:52:42.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.149+0000 7fbe52edc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4c11c320 con 0x7fbe4c10b190 2024-03-18T22:52:42.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.149+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe4c10b190 0x7fbe4c115370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.149+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe4c10b190 0x7fbe4c115370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53796/0 (socket says 172.21.15.40:53796) 2024-03-18T22:52:42.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.149+0000 7fbe4bfff640 1 -- 172.21.15.40:0/1207933760 learned_addr learned my addr 172.21.15.40:0/1207933760 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:42.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.149+0000 7fbe51479640 1 --2- 172.21.15.40:0/1207933760 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe4c10db40 0x7fbe4c1a58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.150+0000 7fbe50c78640 1 --2- 172.21.15.40:0/1207933760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe4c108820 0x7fbe4c114e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.151+0000 7fbe4bfff640 1 -- 172.21.15.40:0/1207933760 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe4c10db40 msgr2=0x7fbe4c1a58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.151+0000 7fbe4bfff640 1 --2- 172.21.15.40:0/1207933760 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe4c10db40 0x7fbe4c1a58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.151+0000 7fbe4bfff640 1 -- 172.21.15.40:0/1207933760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe4c108820 msgr2=0x7fbe4c114e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.151+0000 7fbe4bfff640 1 --2- 172.21.15.40:0/1207933760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe4c108820 0x7fbe4c114e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.151+0000 7fbe4bfff640 1 -- 172.21.15.40:0/1207933760 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe4c1184f0 con 0x7fbe4c10b190 2024-03-18T22:52:42.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.153+0000 7fbe4bfff640 1 --2- 172.21.15.40:0/1207933760 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe4c10b190 0x7fbe4c115370 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009530 tx=0x7fbe4002ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:42.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.154+0000 7fbe49ffb640 1 -- 172.21.15.40:0/1207933760 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe40041020 con 0x7fbe4c10b190 2024-03-18T22:52:42.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.154+0000 7fbe52edc640 1 -- 172.21.15.40:0/1207933760 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe4c1a5e40 con 0x7fbe4c10b190 2024-03-18T22:52:42.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.155+0000 7fbe52edc640 1 -- 172.21.15.40:0/1207933760 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe4c1a63b0 con 0x7fbe4c10b190 2024-03-18T22:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.158+0000 7fbe55ba5740 1 Processor -- start 2024-03-18T22:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.158+0000 7fbe55ba5740 1 -- start start 2024-03-18T22:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.158+0000 7fbe55ba5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 0x55ad1582b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.158+0000 7fbe55ba5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad1582bd90 0x55ad15830890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.158+0000 7fbe55ba5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad15830f00 0x55ad15833390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.158+0000 7fbe55ba5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ad15790160 con 0x55ad1582bd90 2024-03-18T22:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.158+0000 7fbe55ba5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ad15738e10 con 0x55ad1582b3a0 2024-03-18T22:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.158+0000 7fbe55ba5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55ad1578c200 con 0x55ad15830f00 2024-03-18T22:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.159+0000 7fbe49ffb640 1 -- 172.21.15.40:0/1207933760 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe4002e2e0 con 0x7fbe4c10b190 2024-03-18T22:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.159+0000 7fbe49ffb640 1 -- 172.21.15.40:0/1207933760 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe40005040 con 0x7fbe4c10b190 2024-03-18T22:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.160+0000 7fbe50c78640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad1582bd90 0x55ad15830890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.160+0000 7fbe50c78640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad1582bd90 0x55ad15830890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33018/0 (socket says 172.21.15.40:33018) 2024-03-18T22:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.160+0000 7fbe50c78640 1 -- 172.21.15.40:0/1432693196 learned_addr learned my addr 172.21.15.40:0/1432693196 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.160+0000 7fbe50c78640 1 -- 172.21.15.40:0/1432693196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad15830f00 msgr2=0x55ad15833390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.161+0000 7fbe50c78640 1 --2- 172.21.15.40:0/1432693196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad15830f00 0x55ad15833390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.161+0000 7fbe50c78640 1 -- 172.21.15.40:0/1432693196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 msgr2=0x55ad1582b7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.161+0000 7fbe51479640 1 --2- 172.21.15.40:0/1432693196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 0x55ad1582b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.161+0000 7fbe50c78640 1 --2- 172.21.15.40:0/1432693196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 0x55ad1582b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.161+0000 7fbe50c78640 1 -- 172.21.15.40:0/1432693196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad1556c7e0 con 0x55ad1582bd90 2024-03-18T22:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.162+0000 7fbe51479640 1 --2- 172.21.15.40:0/1432693196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 0x55ad1582b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.163+0000 7fbe49ffb640 1 -- 172.21.15.40:0/1207933760 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbe40044020 con 0x7fbe4c10b190 2024-03-18T22:52:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.164+0000 7fbe50c78640 1 --2- 172.21.15.40:0/1432693196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad1582bd90 0x55ad15830890 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00a8e0 tx=0x7fbe3c00ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3247e506d2fb7fe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:42.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.164+0000 7fbe49ffb640 1 --2- 172.21.15.40:0/1207933760 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbe340a29d0 0x7fbe340a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.165+0000 7fbe49ffb640 1 -- 172.21.15.40:0/1207933760 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fbe400f0890 con 0x7fbe4c10b190 2024-03-18T22:52:42.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.165+0000 7fbe51479640 1 --2- 172.21.15.40:0/1207933760 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbe340a29d0 0x7fbe340a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.166+0000 7fbe51479640 1 --2- 172.21.15.40:0/1207933760 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbe340a29d0 0x7fbe340a4e90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x55ad1582c5f0 tx=0x7fbe44009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:42.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.167+0000 7fbe32ffd640 1 -- 172.21.15.40:0/1432693196 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c015070 con 0x55ad1582bd90 2024-03-18T22:52:42.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.167+0000 7fbe32ffd640 1 -- 172.21.15.40:0/1432693196 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe3c0113f0 con 0x55ad1582bd90 2024-03-18T22:52:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.168+0000 7fbe55ba5740 1 -- 172.21.15.40:0/1432693196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad1582bd90 msgr2=0x55ad15830890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.168+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/1432693196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad1582bd90 0x55ad15830890 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00a8e0 tx=0x7fbe3c00ada0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.168+0000 7fbe55ba5740 1 -- 172.21.15.40:0/1432693196 shutdown_connections 2024-03-18T22:52:42.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.168+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/1432693196 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad15830f00 0x55ad15833390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.168+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/1432693196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad1582bd90 0x55ad15830890 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.168+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/1432693196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 0x55ad1582b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.168+0000 7fbe55ba5740 1 -- 172.21.15.40:0/1432693196 >> 172.21.15.40:0/1432693196 conn(0x55ad15826910 msgr2=0x55ad15826d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:42.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.171+0000 7fbe55ba5740 1 -- 172.21.15.40:0/1432693196 shutdown_connections 2024-03-18T22:52:42.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.172+0000 7fbe55ba5740 1 -- 172.21.15.40:0/1432693196 wait complete. 2024-03-18T22:52:42.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.172+0000 7fbe55ba5740 1 -- start start 2024-03-18T22:52:42.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.172+0000 7fbe55ba5740 1 Processor -- start 2024-03-18T22:52:42.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.173+0000 7fbe55ba5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 0x55ad15835870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.173+0000 7fbe55ba5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad15830f00 0x55ad15835db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.173+0000 7fbe55ba5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad1583ade0 0x55ad1583d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.173+0000 7fbe55ba5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ad15833f00 con 0x55ad15830f00 2024-03-18T22:52:42.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.173+0000 7fbe55ba5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ad1574c1a0 con 0x55ad1582b3a0 2024-03-18T22:52:42.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.173+0000 7fbe55ba5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55ad15834080 con 0x55ad1583ade0 2024-03-18T22:52:42.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.173+0000 7fbe55ba5740 10 client.0 _ll_register_callbacks cb 0x55ad154a3c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:42.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.174+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad15830f00 0x55ad15835db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.174+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad15830f00 0x55ad15835db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33032/0 (socket says 172.21.15.40:33032) 2024-03-18T22:52:42.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.174+0000 7fbe4bfff640 1 -- 172.21.15.40:0/2463899691 learned_addr learned my addr 172.21.15.40:0/2463899691 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:42.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.174+0000 7fbe51479640 1 --2- 172.21.15.40:0/2463899691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad1583ade0 0x55ad1583d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.174+0000 7fbe4bfff640 1 -- 172.21.15.40:0/2463899691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad1583ade0 msgr2=0x55ad1583d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:42.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.174+0000 7fbe4bfff640 1 --2- 172.21.15.40:0/2463899691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad1583ade0 0x55ad1583d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.174+0000 7fbe4bfff640 1 -- 172.21.15.40:0/2463899691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 msgr2=0x55ad15835870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:52:42.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.174+0000 7fbe4bfff640 1 --2- 172.21.15.40:0/2463899691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 0x55ad15835870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.174+0000 7fbe4bfff640 1 -- 172.21.15.40:0/2463899691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad155ccf70 con 0x55ad15830f00 2024-03-18T22:52:42.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.175+0000 7fbe4bfff640 1 --2- 172.21.15.40:0/2463899691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad15830f00 0x55ad15835db0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fbe400bcdf0 tx=0x7fbe400bd620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b64c55b8c1f13aa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:42.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.175+0000 7fbe2bfff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:42.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.176+0000 7fbe2bfff640 1 -- 172.21.15.40:0/2463899691 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe400da060 con 0x55ad15830f00 2024-03-18T22:52:42.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.176+0000 7fbe2bfff640 1 -- 172.21.15.40:0/2463899691 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fbe400bc4b0 con 0x55ad15830f00 2024-03-18T22:52:42.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.176+0000 7fbe55ba5740 1 -- 172.21.15.40:0/2463899691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ad1579d6a0 con 0x55ad15830f00 2024-03-18T22:52:42.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.177+0000 7fbe2bfff640 1 -- 172.21.15.40:0/2463899691 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbe400c5600 con 0x55ad15830f00 2024-03-18T22:52:42.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.177+0000 7fbe55ba5740 10 client.18102 fetch_fsmap learned FSMap version 36 2024-03-18T22:52:42.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.177+0000 7fbe55ba5740 1 -- 172.21.15.40:0/2463899691 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ad15464170 con 0x55ad15830f00 2024-03-18T22:52:42.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.178+0000 7fbe2bfff640 1 -- 172.21.15.40:0/2463899691 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ad1579d6a0 con 0x55ad15830f00 2024-03-18T22:52:42.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.178+0000 7fbe2bfff640 1 -- 172.21.15.40:0/2463899691 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 36) ==== 8378+0+0 (secure 0 0 0) 0x7fbe400c1fe0 con 0x55ad15830f00 2024-03-18T22:52:42.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.178+0000 7fbe55ba5740 10 client.18102 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T22:52:42.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.179+0000 7fbe55ba5740 10 client.18102 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:42.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.179+0000 7fbe55ba5740 20 client.18102 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:42.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.179+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/2463899691 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ad15841ab0 0x55ad15843f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.179+0000 7fbe55ba5740 4 client.18102 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:42.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.179+0000 7fbe55ba5740 1 -- 172.21.15.40:0/2463899691 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad157979f0 con 0x55ad15841ab0 2024-03-18T22:52:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.184+0000 7fbe50c78640 1 --2- 172.21.15.40:0/2463899691 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ad15841ab0 0x55ad15843f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.184+0000 7fbe50c78640 1 --2- 172.21.15.40:0/2463899691 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ad15841ab0 0x55ad15843f30 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b8896393a5cf225 server_cookie=a03a97844c7b09b3 in_seq=0 out_seq=0 2024-03-18T22:52:42.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.184+0000 7fbe2bfff640 10 client.18102 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.187+0000 7fbe2bfff640 1 -- 172.21.15.40:0/2463899691 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ad157979f0 con 0x55ad15841ab0 2024-03-18T22:52:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.187+0000 7fbe2bfff640 10 client.18102 handle_command_reply: tid=0 2024-03-18T22:52:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.187+0000 7fbe55ba5740 1 client.18102 shutdown 2024-03-18T22:52:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.187+0000 7fbe55ba5740 1 -- 172.21.15.40:0/2463899691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad15830f00 msgr2=0x55ad15835db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.187+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/2463899691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad15830f00 0x55ad15835db0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fbe400bcdf0 tx=0x7fbe400bd620 comp rx=0 tx=0).stop 2024-03-18T22:52:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.188+0000 7fbe55ba5740 1 -- 172.21.15.40:0/2463899691 shutdown_connections 2024-03-18T22:52:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.188+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/2463899691 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad1583ade0 0x55ad1583d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.188+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/2463899691 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad15830f00 0x55ad15835db0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.188+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/2463899691 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ad15841ab0 0x55ad15843f30 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.188+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/2463899691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 0x55ad15835870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.188+0000 7fbe55ba5740 1 -- 172.21.15.40:0/2463899691 >> 172.21.15.40:0/2463899691 conn(0x55ad15826910 msgr2=0x55ad15825b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.188+0000 7fbe2bfff640 0 client.18102 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:42.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.188+0000 7fbe55ba5740 1 -- 172.21.15.40:0/2463899691 shutdown_connections 2024-03-18T22:52:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.188+0000 7fbe55ba5740 1 -- 172.21.15.40:0/2463899691 wait complete. 2024-03-18T22:52:42.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.188+0000 7fbe55ba5740 20 client.18102 trim_cache size 0 max 16384 2024-03-18T22:52:42.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.227+0000 7fbe55ba5740 1 Processor -- start 2024-03-18T22:52:42.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.228+0000 7fbe55ba5740 1 -- start start 2024-03-18T22:52:42.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.228+0000 7fbe55ba5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad1582b3a0 0x55ad15835270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.228+0000 7fbe55ba5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad15830f00 0x55ad15848830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.228+0000 7fbe55ba5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1583ade0 0x55ad15848d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.228+0000 7fbe55ba5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3800fe60 con 0x55ad1582b3a0 2024-03-18T22:52:42.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.228+0000 7fbe55ba5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbe38011a60 con 0x55ad1583ade0 2024-03-18T22:52:42.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.228+0000 7fbe55ba5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbe38010530 con 0x55ad15830f00 2024-03-18T22:52:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.228+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad15830f00 0x55ad15848830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.228+0000 7fbe4bfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad15830f00 0x55ad15848830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:53828/0 (socket says 172.21.15.40:53828) 2024-03-18T22:52:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.228+0000 7fbe4bfff640 1 -- 172.21.15.40:0/1304557332 learned_addr learned my addr 172.21.15.40:0/1304557332 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.229+0000 7fbe51479640 1 --2- 172.21.15.40:0/1304557332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1583ade0 0x55ad15848d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.229+0000 7fbe4bfff640 1 -- 172.21.15.40:0/1304557332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1583ade0 msgr2=0x55ad15848d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.229+0000 7fbe4bfff640 1 --2- 172.21.15.40:0/1304557332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1583ade0 0x55ad15848d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.229+0000 7fbe4bfff640 1 -- 172.21.15.40:0/1304557332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad1582b3a0 msgr2=0x55ad15835270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.229+0000 7fbe4bfff640 1 --2- 172.21.15.40:0/1304557332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad1582b3a0 0x55ad15835270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.229+0000 7fbe4bfff640 1 -- 172.21.15.40:0/1304557332 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad158369f0 con 0x55ad15830f00 2024-03-18T22:52:42.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.229+0000 7fbe4bfff640 1 --2- 172.21.15.40:0/1304557332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad15830f00 0x55ad15848830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbe400b92c0 tx=0x7fbe400c1130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a9332ecc19cbee4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:42.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.233+0000 7fbe327fc640 1 -- 172.21.15.40:0/1304557332 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe400da040 con 0x55ad15830f00 2024-03-18T22:52:42.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.233+0000 7fbe327fc640 1 -- 172.21.15.40:0/1304557332 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe400c1ed0 con 0x55ad15830f00 2024-03-18T22:52:42.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.233+0000 7fbe327fc640 1 -- 172.21.15.40:0/1304557332 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe400bbd20 con 0x55ad15830f00 2024-03-18T22:52:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.234+0000 7fbe55ba5740 1 -- 172.21.15.40:0/1304557332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad15830f00 msgr2=0x55ad15848830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.234+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/1304557332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad15830f00 0x55ad15848830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbe400b92c0 tx=0x7fbe400c1130 comp rx=0 tx=0).stop 2024-03-18T22:52:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.234+0000 7fbe55ba5740 1 -- 172.21.15.40:0/1304557332 shutdown_connections 2024-03-18T22:52:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.234+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/1304557332 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1583ade0 0x55ad15848d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.234+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/1304557332 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad15830f00 0x55ad15848830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.234+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/1304557332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad1582b3a0 0x55ad15835270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.234+0000 7fbe55ba5740 1 -- 172.21.15.40:0/1304557332 >> 172.21.15.40:0/1304557332 conn(0x55ad15826910 msgr2=0x55ad15834b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:42.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.234+0000 7fbe55ba5740 1 -- 172.21.15.40:0/1304557332 shutdown_connections 2024-03-18T22:52:42.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.235+0000 7fbe55ba5740 1 -- 172.21.15.40:0/1304557332 wait complete. 2024-03-18T22:52:42.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.235+0000 7fbe55ba5740 1 -- start start 2024-03-18T22:52:42.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.235+0000 7fbe55ba5740 1 Processor -- start 2024-03-18T22:52:42.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.236+0000 7fbe55ba5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 0x55ad15853710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.236+0000 7fbe55ba5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad15830f00 0x55ad15853c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.236+0000 7fbe50c78640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 0x55ad15853710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.236+0000 7fbe50c78640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 0x55ad15853710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36206/0 (socket says 172.21.15.40:36206) 2024-03-18T22:52:42.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.237+0000 7fbe50c78640 1 -- 172.21.15.40:0/3617267949 learned_addr learned my addr 172.21.15.40:0/3617267949 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:42.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.237+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/3617267949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad1583ade0 0x55ad15854190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.237+0000 7fbe55ba5740 1 -- 172.21.15.40:0/3617267949 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fbe38011390 con 0x55ad15830f00 2024-03-18T22:52:42.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.237+0000 7fbe51479640 1 --2- 172.21.15.40:0/3617267949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad1583ade0 0x55ad15854190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.237+0000 7fbe55ba5740 1 -- 172.21.15.40:0/3617267949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbe38010c00 con 0x55ad1582b3a0 2024-03-18T22:52:42.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.237+0000 7fbe55ba5740 1 -- 172.21.15.40:0/3617267949 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3800de40 con 0x55ad1583ade0 2024-03-18T22:52:42.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.238+0000 7fbe50c78640 1 -- 172.21.15.40:0/3617267949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad1583ade0 msgr2=0x55ad15854190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:42.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.238+0000 7fbe50c78640 1 --2- 172.21.15.40:0/3617267949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad1583ade0 0x55ad15854190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.238+0000 7fbe50c78640 1 -- 172.21.15.40:0/3617267949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad15830f00 msgr2=0x55ad15853c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:42.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.238+0000 7fbe50c78640 1 --2- 172.21.15.40:0/3617267949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad15830f00 0x55ad15853c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.238+0000 7fbe50c78640 1 -- 172.21.15.40:0/3617267949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad158a7830 con 0x55ad1582b3a0 2024-03-18T22:52:42.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.238+0000 7fbe55ba5740 10 client.0 _ll_register_callbacks cb 0x7fbe4002e490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:42.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.238+0000 7fbe50c78640 1 --2- 172.21.15.40:0/3617267949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 0x55ad15853710 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c000c00 tx=0x7fbe3c00a310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb1be98ade96704c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:42.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.238+0000 7fbe55ba5740 1 -- 172.21.15.40:0/3617267949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe400c1ed0 con 0x55ad1582b3a0 2024-03-18T22:52:42.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.241+0000 7fbe2bfff640 10 client.27938 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:52:42.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.241+0000 7fbe2bfff640 1 -- 172.21.15.40:0/3617267949 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c02a060 con 0x55ad1582b3a0 2024-03-18T22:52:42.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.241+0000 7fbe2bfff640 1 -- 172.21.15.40:0/3617267949 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fbe3c00ca80 con 0x55ad1582b3a0 2024-03-18T22:52:42.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.242+0000 7fbe2bfff640 1 -- 172.21.15.40:0/3617267949 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbe3c015040 con 0x55ad1582b3a0 2024-03-18T22:52:42.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.242+0000 7fbe55ba5740 10 client.27938 fetch_fsmap learned FSMap version 36 2024-03-18T22:52:42.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.242+0000 7fbe55ba5740 1 -- 172.21.15.40:0/3617267949 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({fsmap=36}) -- 0x55ad158a8a70 con 0x55ad1582b3a0 2024-03-18T22:52:42.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.245+0000 7fbe2bfff640 1 -- 172.21.15.40:0/3617267949 <== mon.2 v2:172.21.15.103:3300/0 4 ==== fsmap(e 36) ==== 8378+0+0 (secure 0 0 0) 0x7fbe3c00ccc0 con 0x55ad1582b3a0 2024-03-18T22:52:42.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.245+0000 7fbe55ba5740 10 client.27938 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T22:52:42.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.245+0000 7fbe55ba5740 10 client.27938 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:42.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.245+0000 7fbe55ba5740 20 client.27938 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:42.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.245+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/3617267949 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ad158a9430 0x55ad158ab870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.245+0000 7fbe55ba5740 4 client.27938 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:52:42.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.245+0000 7fbe55ba5740 1 -- 172.21.15.40:0/3617267949 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fbe400c1ed0 con 0x55ad158a9430 2024-03-18T22:52:42.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.248+0000 7fbe4bfff640 1 --2- 172.21.15.40:0/3617267949 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ad158a9430 0x55ad158ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.252+0000 7fbe4bfff640 1 --2- 172.21.15.40:0/3617267949 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ad158a9430 0x55ad158ab870 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eeaf23d89dfcf3b server_cookie=bac3b00c1faa15e1 in_seq=0 out_seq=0 2024-03-18T22:52:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.255+0000 7fbe2bfff640 10 client.27938 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.255+0000 7fbe2bfff640 1 -- 172.21.15.40:0/3617267949 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fbe400c1ed0 con 0x55ad158a9430 2024-03-18T22:52:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.255+0000 7fbe2bfff640 10 client.27938 handle_command_reply: tid=0 2024-03-18T22:52:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.255+0000 7fbe55ba5740 1 client.27938 shutdown 2024-03-18T22:52:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.255+0000 7fbe55ba5740 1 -- 172.21.15.40:0/3617267949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 msgr2=0x55ad15853710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.255+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/3617267949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 0x55ad15853710 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c000c00 tx=0x7fbe3c00a310 comp rx=0 tx=0).stop 2024-03-18T22:52:42.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.255+0000 7fbe55ba5740 1 -- 172.21.15.40:0/3617267949 shutdown_connections 2024-03-18T22:52:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.255+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/3617267949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ad1583ade0 0x55ad15854190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.255+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/3617267949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ad15830f00 0x55ad15853c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.255+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/3617267949 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55ad158a9430 0x55ad158ab870 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.255+0000 7fbe55ba5740 1 --2- 172.21.15.40:0/3617267949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ad1582b3a0 0x55ad15853710 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.256+0000 7fbe55ba5740 1 -- 172.21.15.40:0/3617267949 >> 172.21.15.40:0/3617267949 conn(0x55ad15826910 msgr2=0x55ad158252a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:42.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.256+0000 7fbe2bfff640 0 client.27938 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:42.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.266+0000 7fbe55ba5740 1 -- 172.21.15.40:0/3617267949 shutdown_connections 2024-03-18T22:52:42.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.266+0000 7fbe55ba5740 1 -- 172.21.15.40:0/3617267949 wait complete. 2024-03-18T22:52:42.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.266+0000 7fbe55ba5740 20 client.27938 trim_cache size 0 max 16384 2024-03-18T22:52:42.270 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:52:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:52:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "d35d5141-ffc5-4c30-92df-07e85e1417bb", 2024-03-18T22:52:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:52:42.271 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:52:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.281+0000 7fbe52edc640 1 -- 172.21.15.40:0/1207933760 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbe340a29d0 msgr2=0x7fbe340a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.282+0000 7fbe52edc640 1 --2- 172.21.15.40:0/1207933760 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbe340a29d0 0x7fbe340a4e90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x55ad1582c5f0 tx=0x7fbe44009500 comp rx=0 tx=0).stop 2024-03-18T22:52:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.282+0000 7fbe52edc640 1 -- 172.21.15.40:0/1207933760 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe4c10b190 msgr2=0x7fbe4c115370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.282+0000 7fbe52edc640 1 --2- 172.21.15.40:0/1207933760 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe4c10b190 0x7fbe4c115370 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009530 tx=0x7fbe4002ecd0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.284+0000 7fbe52edc640 1 -- 172.21.15.40:0/1207933760 shutdown_connections 2024-03-18T22:52:42.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.284+0000 7fbe52edc640 1 --2- 172.21.15.40:0/1207933760 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fbe340a29d0 0x7fbe340a4e90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.284+0000 7fbe52edc640 1 --2- 172.21.15.40:0/1207933760 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe4c10db40 0x7fbe4c1a58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.284+0000 7fbe52edc640 1 --2- 172.21.15.40:0/1207933760 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fbe4c10b190 0x7fbe4c115370 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.284+0000 7fbe52edc640 1 --2- 172.21.15.40:0/1207933760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe4c108820 0x7fbe4c114e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.284+0000 7fbe52edc640 1 -- 172.21.15.40:0/1207933760 >> 172.21.15.40:0/1207933760 conn(0x7fbe4c06e680 msgr2=0x7fbe4c10a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:42.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.287+0000 7fbe52edc640 1 -- 172.21.15.40:0/1207933760 shutdown_connections 2024-03-18T22:52:42.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.287+0000 7fbe52edc640 1 -- 172.21.15.40:0/1207933760 wait complete. 2024-03-18T22:52:42.319 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:52:42.319 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:52:42.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.494+0000 7feef759e640 1 -- 172.21.15.40:0/425305350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feef8100600 msgr2=0x7feef8100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.494+0000 7feef759e640 1 --2- 172.21.15.40:0/425305350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feef8100600 0x7feef8100a20 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7feee8009900 tx=0x7feee802bc30 comp rx=0 tx=0).stop 2024-03-18T22:52:42.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.494+0000 7feef759e640 1 -- 172.21.15.40:0/425305350 shutdown_connections 2024-03-18T22:52:42.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.494+0000 7feef759e640 1 --2- 172.21.15.40:0/425305350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef81021b0 0x7feef8108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.494+0000 7feef759e640 1 --2- 172.21.15.40:0/425305350 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef81017f0 0x7feef8101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.494+0000 7feef759e640 1 --2- 172.21.15.40:0/425305350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feef8100600 0x7feef8100a20 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.494+0000 7feef759e640 1 -- 172.21.15.40:0/425305350 >> 172.21.15.40:0/425305350 conn(0x7feef80fbb90 msgr2=0x7feef80fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:42.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.494+0000 7feef759e640 1 -- 172.21.15.40:0/425305350 shutdown_connections 2024-03-18T22:52:42.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.494+0000 7feef759e640 1 -- 172.21.15.40:0/425305350 wait complete. 2024-03-18T22:52:42.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feef759e640 1 Processor -- start 2024-03-18T22:52:42.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feef759e640 1 -- start start 2024-03-18T22:52:42.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feef759e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef8100600 0x7feef8196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feef759e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feef81017f0 0x7feef8196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feef759e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef81021b0 0x7feef819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feef759e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feef810c190 con 0x7feef81017f0 2024-03-18T22:52:42.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feef759e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feef810c010 con 0x7feef8100600 2024-03-18T22:52:42.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feef759e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7feef810c310 con 0x7feef81021b0 2024-03-18T22:52:42.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feeeffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feef81017f0 0x7feef8196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feeeffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feef81017f0 0x7feef8196dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33076/0 (socket says 172.21.15.40:33076) 2024-03-18T22:52:42.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feeeffff640 1 -- 172.21.15.40:0/666968168 learned_addr learned my addr 172.21.15.40:0/666968168 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:42.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feef6d9d640 1 --2- 172.21.15.40:0/666968168 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef81021b0 0x7feef819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feef659c640 1 --2- 172.21.15.40:0/666968168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef8100600 0x7feef8196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feeeffff640 1 -- 172.21.15.40:0/666968168 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef81021b0 msgr2=0x7feef819de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feeeffff640 1 --2- 172.21.15.40:0/666968168 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef81021b0 0x7feef819de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feeeffff640 1 -- 172.21.15.40:0/666968168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef8100600 msgr2=0x7feef8196890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feeeffff640 1 --2- 172.21.15.40:0/666968168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef8100600 0x7feef8196890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feeeffff640 1 -- 172.21.15.40:0/666968168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef819e4b0 con 0x7feef81017f0 2024-03-18T22:52:42.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.495+0000 7feeeffff640 1 --2- 172.21.15.40:0/666968168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feef81017f0 0x7feef8196dd0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7feee000d950 tx=0x7feee000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:42.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.496+0000 7feeef7fe640 1 -- 172.21.15.40:0/666968168 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee0017030 con 0x7feef81017f0 2024-03-18T22:52:42.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.496+0000 7feef759e640 1 -- 172.21.15.40:0/666968168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef819e7a0 con 0x7feef81017f0 2024-03-18T22:52:42.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.496+0000 7feeef7fe640 1 -- 172.21.15.40:0/666968168 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feee0004020 con 0x7feef81017f0 2024-03-18T22:52:42.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.496+0000 7feeef7fe640 1 -- 172.21.15.40:0/666968168 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee00047a0 con 0x7feef81017f0 2024-03-18T22:52:42.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.496+0000 7feef759e640 1 -- 172.21.15.40:0/666968168 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef819ecf0 con 0x7feef81017f0 2024-03-18T22:52:42.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.498+0000 7feeef7fe640 1 -- 172.21.15.40:0/666968168 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7feee00041d0 con 0x7feef81017f0 2024-03-18T22:52:42.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.498+0000 7feeff7e0740 1 Processor -- start 2024-03-18T22:52:42.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.498+0000 7feeff7e0740 1 -- start start 2024-03-18T22:52:42.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.498+0000 7feeff7e0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bee20 0x563ba35bf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.498+0000 7feeff7e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bf810 0x563ba35c4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.498+0000 7feeff7e0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35c4980 0x563ba35c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.498+0000 7feeff7e0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563ba3521500 con 0x563ba35bf810 2024-03-18T22:52:42.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.498+0000 7feeff7e0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563ba34c9be0 con 0x563ba35bee20 2024-03-18T22:52:42.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.498+0000 7feeff7e0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563ba351d5a0 con 0x563ba35c4980 2024-03-18T22:52:42.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feeef7fe640 1 --2- 172.21.15.40:0/666968168 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feee40a2800 0x7feee40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feef659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bf810 0x563ba35c4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feef659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bf810 0x563ba35c4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33078/0 (socket says 172.21.15.40:33078) 2024-03-18T22:52:42.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feef659c640 1 -- 172.21.15.40:0/132690077 learned_addr learned my addr 172.21.15.40:0/132690077 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:42.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feeeffff640 1 --2- 172.21.15.40:0/132690077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35c4980 0x563ba35c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feef6d9d640 1 --2- 172.21.15.40:0/132690077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bee20 0x563ba35bf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feef6d9d640 1 --2- 172.21.15.40:0/666968168 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feee40a2800 0x7feee40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feeef7fe640 1 -- 172.21.15.40:0/666968168 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7feee00ce9c0 con 0x7feef81017f0 2024-03-18T22:52:42.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feef659c640 1 -- 172.21.15.40:0/132690077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35c4980 msgr2=0x563ba35c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:42.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feef659c640 1 --2- 172.21.15.40:0/132690077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35c4980 0x563ba35c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feef659c640 1 -- 172.21.15.40:0/132690077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bee20 msgr2=0x563ba35bf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feef659c640 1 --2- 172.21.15.40:0/132690077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bee20 0x563ba35bf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feef659c640 1 -- 172.21.15.40:0/132690077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ba34eb000 con 0x563ba35bf810 2024-03-18T22:52:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feef6d9d640 1 --2- 172.21.15.40:0/666968168 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feee40a2800 0x7feee40a4cc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7feef0009900 tx=0x7feef0009650 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feef659c640 1 --2- 172.21.15.40:0/132690077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bf810 0x563ba35c4310 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7feee8009900 tx=0x7feee8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c74d8d044ba1f15 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feeecff9640 1 -- 172.21.15.40:0/132690077 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8044020 con 0x563ba35bf810 2024-03-18T22:52:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feeecff9640 1 -- 172.21.15.40:0/132690077 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feee8005040 con 0x563ba35bf810 2024-03-18T22:52:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.499+0000 7feeecff9640 1 -- 172.21.15.40:0/132690077 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8004900 con 0x563ba35bf810 2024-03-18T22:52:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.500+0000 7feeff7e0740 1 -- 172.21.15.40:0/132690077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bf810 msgr2=0x563ba35c4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.500+0000 7feeff7e0740 1 --2- 172.21.15.40:0/132690077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bf810 0x563ba35c4310 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7feee8009900 tx=0x7feee8005e80 comp rx=0 tx=0).stop 2024-03-18T22:52:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.500+0000 7feeff7e0740 1 -- 172.21.15.40:0/132690077 shutdown_connections 2024-03-18T22:52:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.500+0000 7feeff7e0740 1 --2- 172.21.15.40:0/132690077 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35c4980 0x563ba35c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.500+0000 7feeff7e0740 1 --2- 172.21.15.40:0/132690077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bf810 0x563ba35c4310 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.500+0000 7feeff7e0740 1 --2- 172.21.15.40:0/132690077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bee20 0x563ba35bf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.500+0000 7feeff7e0740 1 -- 172.21.15.40:0/132690077 >> 172.21.15.40:0/132690077 conn(0x563ba35b8180 msgr2=0x563ba35b8580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:42.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.500+0000 7feeff7e0740 1 -- 172.21.15.40:0/132690077 shutdown_connections 2024-03-18T22:52:42.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.500+0000 7feeff7e0740 1 -- 172.21.15.40:0/132690077 wait complete. 2024-03-18T22:52:42.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.500+0000 7feeff7e0740 1 -- start start 2024-03-18T22:52:42.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feeff7e0740 1 Processor -- start 2024-03-18T22:52:42.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feeff7e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bee20 0x563ba35c9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feeff7e0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bf810 0x563ba35c9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feeff7e0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35c4980 0x563ba35d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feeff7e0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x563ba35c78d0 con 0x563ba35bee20 2024-03-18T22:52:42.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feeff7e0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563ba34dd160 con 0x563ba35bf810 2024-03-18T22:52:42.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feeff7e0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x563ba35c7a50 con 0x563ba35c4980 2024-03-18T22:52:42.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feeff7e0740 10 client.0 _ll_register_callbacks cb 0x563ba3519a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:42.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feef659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bee20 0x563ba35c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feef659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bee20 0x563ba35c9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33082/0 (socket says 172.21.15.40:33082) 2024-03-18T22:52:42.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feef659c640 1 -- 172.21.15.40:0/3347271154 learned_addr learned my addr 172.21.15.40:0/3347271154 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:42.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feeeffff640 1 --2- 172.21.15.40:0/3347271154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bf810 0x563ba35c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feef6d9d640 1 --2- 172.21.15.40:0/3347271154 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35c4980 0x563ba35d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feef659c640 1 -- 172.21.15.40:0/3347271154 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35c4980 msgr2=0x563ba35d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:42.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feef659c640 1 --2- 172.21.15.40:0/3347271154 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35c4980 0x563ba35d07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feef659c640 1 -- 172.21.15.40:0/3347271154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bf810 msgr2=0x563ba35c9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:42.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feef659c640 1 --2- 172.21.15.40:0/3347271154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bf810 0x563ba35c9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feef659c640 1 -- 172.21.15.40:0/3347271154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ba32ff7d0 con 0x563ba35bee20 2024-03-18T22:52:42.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feeeffff640 1 --2- 172.21.15.40:0/3347271154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bf810 0x563ba35c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:52:42.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feef659c640 1 --2- 172.21.15.40:0/3347271154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bee20 0x563ba35c9240 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7feee8000c00 tx=0x7feee80048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81196e9554f2447 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:42.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feeddffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:42.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.501+0000 7feeddffb640 1 -- 172.21.15.40:0/3347271154 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee804d070 con 0x563ba35bee20 2024-03-18T22:52:42.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.502+0000 7feeddffb640 1 -- 172.21.15.40:0/3347271154 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7feee8004cf0 con 0x563ba35bee20 2024-03-18T22:52:42.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.502+0000 7feeff7e0740 1 -- 172.21.15.40:0/3347271154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feee8005040 con 0x563ba35bee20 2024-03-18T22:52:42.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.502+0000 7feeddffb640 1 -- 172.21.15.40:0/3347271154 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7feee805a050 con 0x563ba35bee20 2024-03-18T22:52:42.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.502+0000 7feeff7e0740 10 client.18114 fetch_fsmap learned FSMap version 37 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.502+0000 7feeff7e0740 1 -- 172.21.15.40:0/3347271154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563ba32353f0 con 0x563ba35bee20 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.502+0000 7feeddffb640 1 -- 172.21.15.40:0/3347271154 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feee8005040 con 0x563ba35bee20 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.502+0000 7feeddffb640 1 -- 172.21.15.40:0/3347271154 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7feee8034440 con 0x563ba35bee20 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.502+0000 7feeff7e0740 10 client.18114 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.502+0000 7feeff7e0740 10 client.18114 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.502+0000 7feeff7e0740 20 client.18114 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.502+0000 7feeff7e0740 1 --2- 172.21.15.40:0/3347271154 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba35d4cb0 0x563ba35d70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.502+0000 7feeff7e0740 4 client.18114 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.502+0000 7feeff7e0740 1 -- 172.21.15.40:0/3347271154 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ba352ea40 con 0x563ba35d4cb0 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.502+0000 7feeeffff640 1 --2- 172.21.15.40:0/3347271154 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba35d4cb0 0x563ba35d70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.503+0000 7feeddffb640 10 client.18114 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.503+0000 7feeeffff640 1 --2- 172.21.15.40:0/3347271154 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba35d4cb0 0x563ba35d70a0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad5b46d1d1a2adb7 server_cookie=b2e4340ffc9e0150 in_seq=0 out_seq=0 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.504+0000 7feeddffb640 1 -- 172.21.15.40:0/3347271154 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563ba352ea40 con 0x563ba35d4cb0 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.504+0000 7feeddffb640 10 client.18114 handle_command_reply: tid=0 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.504+0000 7feeff7e0740 1 client.18114 shutdown 2024-03-18T22:52:42.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.504+0000 7feeff7e0740 1 -- 172.21.15.40:0/3347271154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bee20 msgr2=0x563ba35c9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.504+0000 7feeff7e0740 1 --2- 172.21.15.40:0/3347271154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bee20 0x563ba35c9240 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7feee8000c00 tx=0x7feee80048f0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.504+0000 7feeff7e0740 1 -- 172.21.15.40:0/3347271154 shutdown_connections 2024-03-18T22:52:42.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.504+0000 7feeff7e0740 1 --2- 172.21.15.40:0/3347271154 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35c4980 0x563ba35d07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.504+0000 7feeff7e0740 1 --2- 172.21.15.40:0/3347271154 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba35d4cb0 0x563ba35d70a0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.504+0000 7feeddffb640 0 client.18114 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:42.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.504+0000 7feeff7e0740 1 --2- 172.21.15.40:0/3347271154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bf810 0x563ba35c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.504+0000 7feeff7e0740 1 --2- 172.21.15.40:0/3347271154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bee20 0x563ba35c9240 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.504+0000 7feeff7e0740 1 -- 172.21.15.40:0/3347271154 >> 172.21.15.40:0/3347271154 conn(0x563ba35b8180 msgr2=0x563ba35b74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:42.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.504+0000 7feeff7e0740 1 -- 172.21.15.40:0/3347271154 shutdown_connections 2024-03-18T22:52:42.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.504+0000 7feeff7e0740 1 -- 172.21.15.40:0/3347271154 wait complete. 2024-03-18T22:52:42.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.504+0000 7feeff7e0740 20 client.18114 trim_cache size 0 max 16384 2024-03-18T22:52:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.535+0000 7feeff7e0740 1 Processor -- start 2024-03-18T22:52:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.535+0000 7feeff7e0740 1 -- start start 2024-03-18T22:52:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feeff7e0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35b8180 0x563ba35b8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feeff7e0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bee20 0x563ba35d0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feeff7e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bf810 0x563ba35c8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feeff7e0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feef81a8150 con 0x563ba35bf810 2024-03-18T22:52:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feeff7e0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feef81a9d80 con 0x563ba35bee20 2024-03-18T22:52:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feeff7e0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7feef81a8820 con 0x563ba35b8180 2024-03-18T22:52:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feef6d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bf810 0x563ba35c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feef6d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bf810 0x563ba35c8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33092/0 (socket says 172.21.15.40:33092) 2024-03-18T22:52:42.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feef6d9d640 1 -- 172.21.15.40:0/3179676259 learned_addr learned my addr 172.21.15.40:0/3179676259 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:42.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feeeffff640 1 --2- 172.21.15.40:0/3179676259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bee20 0x563ba35d0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feef659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35b8180 0x563ba35b8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feeeffff640 1 -- 172.21.15.40:0/3179676259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35b8180 msgr2=0x563ba35b8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:42.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feeeffff640 1 --2- 172.21.15.40:0/3179676259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35b8180 0x563ba35b8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feeeffff640 1 -- 172.21.15.40:0/3179676259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bf810 msgr2=0x563ba35c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:42.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feeeffff640 1 --2- 172.21.15.40:0/3179676259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bf810 0x563ba35c8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.536+0000 7feeeffff640 1 -- 172.21.15.40:0/3179676259 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ba31f50f0 con 0x563ba35bee20 2024-03-18T22:52:42.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.537+0000 7feeeffff640 1 --2- 172.21.15.40:0/3179676259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bee20 0x563ba35d0780 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7feee009ac10 tx=0x7feee00ce040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=209ad343a6dd40c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:42.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.537+0000 7feedffff640 1 -- 172.21.15.40:0/3179676259 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee009f060 con 0x563ba35bee20 2024-03-18T22:52:42.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.537+0000 7feedffff640 1 -- 172.21.15.40:0/3179676259 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feee00a36b0 con 0x563ba35bee20 2024-03-18T22:52:42.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.537+0000 7feedffff640 1 -- 172.21.15.40:0/3179676259 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee0009a30 con 0x563ba35bee20 2024-03-18T22:52:42.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.538+0000 7feeff7e0740 1 -- 172.21.15.40:0/3179676259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bee20 msgr2=0x563ba35d0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.538+0000 7feeff7e0740 1 --2- 172.21.15.40:0/3179676259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bee20 0x563ba35d0780 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7feee009ac10 tx=0x7feee00ce040 comp rx=0 tx=0).stop 2024-03-18T22:52:42.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.539+0000 7feeff7e0740 1 -- 172.21.15.40:0/3179676259 shutdown_connections 2024-03-18T22:52:42.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.539+0000 7feeff7e0740 1 --2- 172.21.15.40:0/3179676259 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35bf810 0x563ba35c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.539+0000 7feeff7e0740 1 --2- 172.21.15.40:0/3179676259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bee20 0x563ba35d0780 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.539+0000 7feeff7e0740 1 --2- 172.21.15.40:0/3179676259 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35b8180 0x563ba35b8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.539+0000 7feeff7e0740 1 -- 172.21.15.40:0/3179676259 >> 172.21.15.40:0/3179676259 conn(0x563ba35d7410 msgr2=0x563ba35bbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:42.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.539+0000 7feeff7e0740 1 -- 172.21.15.40:0/3179676259 shutdown_connections 2024-03-18T22:52:42.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.539+0000 7feeff7e0740 1 -- 172.21.15.40:0/3179676259 wait complete. 2024-03-18T22:52:42.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.539+0000 7feeff7e0740 1 -- start start 2024-03-18T22:52:42.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.539+0000 7feeff7e0740 1 Processor -- start 2024-03-18T22:52:42.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.540+0000 7feeff7e0740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35b8180 0x563ba35cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.540+0000 7feeff7e0740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35bee20 0x563ba35ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.540+0000 7feeff7e0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bf810 0x563ba35cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.540+0000 7feeff7e0740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7feef81a96b0 con 0x563ba35b8180 2024-03-18T22:52:42.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.540+0000 7feeff7e0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feef81a8f20 con 0x563ba35bf810 2024-03-18T22:52:42.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.540+0000 7feeff7e0740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7feef81a6130 con 0x563ba35bee20 2024-03-18T22:52:42.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.540+0000 7feef659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35b8180 0x563ba35cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.540+0000 7feeff7e0740 10 client.0 _ll_register_callbacks cb 0x7feef000b410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:52:42.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.540+0000 7feef659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35b8180 0x563ba35cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33108/0 (socket says 172.21.15.40:33108) 2024-03-18T22:52:42.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.540+0000 7feef659c640 1 -- 172.21.15.40:0/1912928947 learned_addr learned my addr 172.21.15.40:0/1912928947 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:52:42.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.540+0000 7feef6d9d640 1 --2- 172.21.15.40:0/1912928947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bf810 0x563ba35cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.541+0000 7feef659c640 1 -- 172.21.15.40:0/1912928947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35bee20 msgr2=0x563ba35ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:42.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.541+0000 7feef659c640 1 --2- 172.21.15.40:0/1912928947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35bee20 0x563ba35ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.541+0000 7feef659c640 1 -- 172.21.15.40:0/1912928947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bf810 msgr2=0x563ba35cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:52:42.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.541+0000 7feef659c640 1 --2- 172.21.15.40:0/1912928947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bf810 0x563ba35cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.541+0000 7feef659c640 1 -- 172.21.15.40:0/1912928947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ba35cfa80 con 0x563ba35b8180 2024-03-18T22:52:42.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.541+0000 7feef659c640 1 --2- 172.21.15.40:0/1912928947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35b8180 0x563ba35cdf40 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7feee8034810 tx=0x7feee8009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d7e050a0302f963 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:52:42.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.541+0000 7feeddffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:52:42.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.542+0000 7feeddffb640 1 -- 172.21.15.40:0/1912928947 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8044040 con 0x563ba35b8180 2024-03-18T22:52:42.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.542+0000 7feeff7e0740 1 -- 172.21.15.40:0/1912928947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feee00a36b0 con 0x563ba35b8180 2024-03-18T22:52:42.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.542+0000 7feeddffb640 1 -- 172.21.15.40:0/1912928947 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7feee8034440 con 0x563ba35b8180 2024-03-18T22:52:42.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.542+0000 7feeddffb640 1 -- 172.21.15.40:0/1912928947 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7feee804d040 con 0x563ba35b8180 2024-03-18T22:52:42.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.542+0000 7feeff7e0740 10 client.18120 fetch_fsmap learned FSMap version 37 2024-03-18T22:52:42.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.543+0000 7feeff7e0740 1 -- 172.21.15.40:0/1912928947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563ba35bc9d0 con 0x563ba35b8180 2024-03-18T22:52:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.544+0000 7feeddffb640 1 -- 172.21.15.40:0/1912928947 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feee00a36b0 con 0x563ba35b8180 2024-03-18T22:52:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.544+0000 7feeddffb640 1 -- 172.21.15.40:0/1912928947 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7feee8076bb0 con 0x563ba35b8180 2024-03-18T22:52:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.544+0000 7feeff7e0740 10 client.18120 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:52:42.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.544+0000 7feeff7e0740 10 client.18120 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:52:42.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.544+0000 7feeff7e0740 20 client.18120 populate_metadata read hostname 'smithi040' 2024-03-18T22:52:42.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.545+0000 7feeff7e0740 1 --2- 172.21.15.40:0/1912928947 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba35c4980 0x563ba35bd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:52:42.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.545+0000 7feeff7e0740 4 client.18120 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:52:42.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.545+0000 7feeff7e0740 1 -- 172.21.15.40:0/1912928947 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563ba3528d90 con 0x563ba35c4980 2024-03-18T22:52:42.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.545+0000 7feeeffff640 1 --2- 172.21.15.40:0/1912928947 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba35c4980 0x563ba35bd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:52:42.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.547+0000 7feeeffff640 1 --2- 172.21.15.40:0/1912928947 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba35c4980 0x563ba35bd430 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7c982ddbe3b3e7a server_cookie=8a3d7fd96e359ab7 in_seq=0 out_seq=0 2024-03-18T22:52:42.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.547+0000 7feeddffb640 10 client.18120 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:42.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.547+0000 7feeddffb640 1 -- 172.21.15.40:0/1912928947 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x563ba3528d90 con 0x563ba35c4980 2024-03-18T22:52:42.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.547+0000 7feeddffb640 10 client.18120 handle_command_reply: tid=0 2024-03-18T22:52:42.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.547+0000 7feeff7e0740 1 client.18120 shutdown 2024-03-18T22:52:42.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.547+0000 7feeff7e0740 1 -- 172.21.15.40:0/1912928947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35b8180 msgr2=0x563ba35cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.547+0000 7feeff7e0740 1 --2- 172.21.15.40:0/1912928947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35b8180 0x563ba35cdf40 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7feee8034810 tx=0x7feee8009520 comp rx=0 tx=0).stop 2024-03-18T22:52:42.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.548+0000 7feeff7e0740 1 -- 172.21.15.40:0/1912928947 shutdown_connections 2024-03-18T22:52:42.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.548+0000 7feeff7e0740 1 --2- 172.21.15.40:0/1912928947 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x563ba35c4980 0x563ba35bd430 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.548+0000 7feeff7e0740 1 --2- 172.21.15.40:0/1912928947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563ba35bf810 0x563ba35cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.548+0000 7feeff7e0740 1 --2- 172.21.15.40:0/1912928947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x563ba35bee20 0x563ba35ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.548+0000 7feeff7e0740 1 --2- 172.21.15.40:0/1912928947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x563ba35b8180 0x563ba35cdf40 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.548+0000 7feeff7e0740 1 -- 172.21.15.40:0/1912928947 >> 172.21.15.40:0/1912928947 conn(0x563ba35d7410 msgr2=0x563ba35b6c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:42.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.548+0000 7feeddffb640 0 client.18120 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:52:42.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.548+0000 7feeff7e0740 1 -- 172.21.15.40:0/1912928947 shutdown_connections 2024-03-18T22:52:42.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.548+0000 7feeff7e0740 1 -- 172.21.15.40:0/1912928947 wait complete. 2024-03-18T22:52:42.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.549+0000 7feeff7e0740 20 client.18120 trim_cache size 0 max 16384 2024-03-18T22:52:42.551 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:52:42.551 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "scrub active (97 inodes in the stack)", 2024-03-18T22:52:42.551 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:52:42.551 INFO:teuthology.orchestra.run.smithi040.stdout: "d35d5141-ffc5-4c30-92df-07e85e1417bb": { 2024-03-18T22:52:42.552 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:52:42.552 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "d35d5141-ffc5-4c30-92df-07e85e1417bb", 2024-03-18T22:52:42.552 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:52:42.552 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:52:42.552 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:52:42.552 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:52:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.550+0000 7feef759e640 1 -- 172.21.15.40:0/666968168 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feee40a2800 msgr2=0x7feee40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.550+0000 7feef759e640 1 --2- 172.21.15.40:0/666968168 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feee40a2800 0x7feee40a4cc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7feef0009900 tx=0x7feef0009650 comp rx=0 tx=0).stop 2024-03-18T22:52:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.550+0000 7feef759e640 1 -- 172.21.15.40:0/666968168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feef81017f0 msgr2=0x7feef8196dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:52:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.550+0000 7feef759e640 1 --2- 172.21.15.40:0/666968168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feef81017f0 0x7feef8196dd0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7feee000d950 tx=0x7feee000de10 comp rx=0 tx=0).stop 2024-03-18T22:52:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.550+0000 7feef759e640 1 -- 172.21.15.40:0/666968168 shutdown_connections 2024-03-18T22:52:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.550+0000 7feef759e640 1 --2- 172.21.15.40:0/666968168 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7feee40a2800 0x7feee40a4cc0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.550+0000 7feef759e640 1 --2- 172.21.15.40:0/666968168 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7feef81021b0 0x7feef819de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.550+0000 7feef759e640 1 --2- 172.21.15.40:0/666968168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feef81017f0 0x7feef8196dd0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.550+0000 7feef759e640 1 --2- 172.21.15.40:0/666968168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feef8100600 0x7feef8196890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:52:42.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.550+0000 7feef759e640 1 -- 172.21.15.40:0/666968168 >> 172.21.15.40:0/666968168 conn(0x7feef80fbb90 msgr2=0x7feef8104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:52:42.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.551+0000 7feef759e640 1 -- 172.21.15.40:0/666968168 shutdown_connections 2024-03-18T22:52:42.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:52:42.551+0000 7feef759e640 1 -- 172.21.15.40:0/666968168 wait complete. 2024-03-18T22:52:42.575 INFO:tasks.cephfs.filesystem:scrub status for tag:d35d5141-ffc5-4c30-92df-07e85e1417bb - {'path': '/', 'tag': 'd35d5141-ffc5-4c30-92df-07e85e1417bb', 'options': 'recursive,force'} 2024-03-18T22:52:42.575 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:42 smithi040 ceph-mon[62643]: pgmap v652: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 MiB/s wr, 4.44k op/s 2024-03-18T22:52:42.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:42 smithi113 ceph-mon[66775]: pgmap v652: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 MiB/s wr, 4.44k op/s 2024-03-18T22:52:42.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:42 smithi103 ceph-mon[66377]: pgmap v652: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 MiB/s wr, 4.44k op/s 2024-03-18T22:52:43.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:43 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:52:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:43 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:43 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:52:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:43 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:52:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:43 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:52:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:43 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:52:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:43 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:52:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:43 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:52:43.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:43 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:43.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:43 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:52:43.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:43 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:43.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:43 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:52:43.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:43 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:52:43.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:43 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:52:43.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:43 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:52:43.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:43 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:52:43.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:43 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:52:43.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:43 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:43.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:43 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:52:43.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:43 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:43.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:43 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:52:43.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:43 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:52:43.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:43 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:52:43.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:43 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:52:43.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:43 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:52:43.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:43 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:52:43.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:43 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:44.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:44 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:44.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:44 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:44.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:44 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:44.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:44 smithi113 ceph-mon[66775]: pgmap v653: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 MiB/s wr, 4.89k op/s 2024-03-18T22:52:44.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:44 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:44.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:44 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:44.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:44 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:44.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:44 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:44.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:44 smithi103 ceph-mon[66377]: pgmap v653: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 MiB/s wr, 4.89k op/s 2024-03-18T22:52:44.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:44 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:44.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:44 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:44.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:44 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:44.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:44 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:44.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:44 smithi040 ceph-mon[62643]: pgmap v653: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 19 MiB/s wr, 4.89k op/s 2024-03-18T22:52:44.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:44 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:52:46.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:46 smithi113 ceph-mon[66775]: pgmap v654: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 4.16k op/s 2024-03-18T22:52:46.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:46 smithi103 ceph-mon[66377]: pgmap v654: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 4.16k op/s 2024-03-18T22:52:46.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:46 smithi040 ceph-mon[62643]: pgmap v654: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 MiB/s wr, 4.16k op/s 2024-03-18T22:52:48.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:48 smithi113 ceph-mon[66775]: pgmap v655: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 MiB/s wr, 4.67k op/s 2024-03-18T22:52:48.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:48 smithi103 ceph-mon[66377]: pgmap v655: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 MiB/s wr, 4.67k op/s 2024-03-18T22:52:48.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:48 smithi040 ceph-mon[62643]: pgmap v655: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 MiB/s wr, 4.67k op/s 2024-03-18T22:52:50.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:50 smithi113 ceph-mon[66775]: pgmap v656: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 4.11k op/s 2024-03-18T22:52:50.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:50 smithi103 ceph-mon[66377]: pgmap v656: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 4.11k op/s 2024-03-18T22:52:50.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:50 smithi040 ceph-mon[62643]: pgmap v656: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 4.11k op/s 2024-03-18T22:52:52.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:52 smithi113 ceph-mon[66775]: pgmap v657: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 MiB/s wr, 4.37k op/s 2024-03-18T22:52:52.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:52 smithi103 ceph-mon[66377]: pgmap v657: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 MiB/s wr, 4.37k op/s 2024-03-18T22:52:52.896 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:52 smithi040 ceph-mon[62643]: pgmap v657: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 MiB/s wr, 4.37k op/s 2024-03-18T22:52:53.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:53 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:52:53.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:53 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:52:53.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:53 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:52:53.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:53 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:52:53.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:53 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:52:53.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:53 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:52:54.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:54 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:52:54.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:54 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:52:54.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:54 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:52:54.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:54 smithi113 ceph-mon[66775]: pgmap v658: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 MiB/s wr, 4.65k op/s 2024-03-18T22:52:54.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:54 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:52:54.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:54 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:52:54.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:54 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:52:54.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:54 smithi103 ceph-mon[66377]: pgmap v658: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 MiB/s wr, 4.65k op/s 2024-03-18T22:52:54.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:54 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:52:54.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:54 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:52:54.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:54 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:52:54.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:54 smithi040 ceph-mon[62643]: pgmap v658: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 18 MiB/s wr, 4.65k op/s 2024-03-18T22:52:56.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:56 smithi113 ceph-mon[66775]: pgmap v659: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 MiB/s wr, 3.99k op/s 2024-03-18T22:52:56.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:56 smithi103 ceph-mon[66377]: pgmap v659: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 MiB/s wr, 3.99k op/s 2024-03-18T22:52:56.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:56 smithi040 ceph-mon[62643]: pgmap v659: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 MiB/s wr, 3.99k op/s 2024-03-18T22:52:58.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:52:58 smithi113 ceph-mon[66775]: pgmap v660: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 18 MiB/s wr, 4.56k op/s 2024-03-18T22:52:58.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:52:58 smithi103 ceph-mon[66377]: pgmap v660: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 18 MiB/s wr, 4.56k op/s 2024-03-18T22:52:58.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:52:58 smithi040 ceph-mon[62643]: pgmap v660: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 18 MiB/s wr, 4.56k op/s 2024-03-18T22:53:00.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:00 smithi113 ceph-mon[66775]: pgmap v661: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 MiB/s wr, 3.98k op/s 2024-03-18T22:53:00.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:00 smithi103 ceph-mon[66377]: pgmap v661: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 MiB/s wr, 3.98k op/s 2024-03-18T22:53:00.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:00 smithi040 ceph-mon[62643]: pgmap v661: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 MiB/s wr, 3.98k op/s 2024-03-18T22:53:02.831 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:02 smithi113 ceph-mon[66775]: pgmap v662: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 MiB/s wr, 4.28k op/s 2024-03-18T22:53:02.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:02 smithi103 ceph-mon[66377]: pgmap v662: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 MiB/s wr, 4.28k op/s 2024-03-18T22:53:02.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:02 smithi040 ceph-mon[62643]: pgmap v662: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 MiB/s wr, 4.28k op/s 2024-03-18T22:53:04.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:04 smithi113 ceph-mon[66775]: pgmap v663: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 MiB/s wr, 4.55k op/s 2024-03-18T22:53:04.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:04 smithi103 ceph-mon[66377]: pgmap v663: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 MiB/s wr, 4.55k op/s 2024-03-18T22:53:04.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:04 smithi040 ceph-mon[62643]: pgmap v663: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 MiB/s wr, 4.55k op/s 2024-03-18T22:53:06.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:06 smithi113 ceph-mon[66775]: pgmap v664: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 MiB/s wr, 3.82k op/s 2024-03-18T22:53:06.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:06 smithi103 ceph-mon[66377]: pgmap v664: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 MiB/s wr, 3.82k op/s 2024-03-18T22:53:06.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:06 smithi040 ceph-mon[62643]: pgmap v664: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 MiB/s wr, 3.82k op/s 2024-03-18T22:53:08.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:08 smithi113 ceph-mon[66775]: pgmap v665: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 MiB/s wr, 4.32k op/s 2024-03-18T22:53:08.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:08 smithi103 ceph-mon[66377]: pgmap v665: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 MiB/s wr, 4.32k op/s 2024-03-18T22:53:08.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:08 smithi040 ceph-mon[62643]: pgmap v665: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 MiB/s wr, 4.32k op/s 2024-03-18T22:53:10.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:10 smithi113 ceph-mon[66775]: pgmap v666: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 MiB/s wr, 3.88k op/s 2024-03-18T22:53:10.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:10 smithi103 ceph-mon[66377]: pgmap v666: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 MiB/s wr, 3.88k op/s 2024-03-18T22:53:10.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:10 smithi040 ceph-mon[62643]: pgmap v666: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 MiB/s wr, 3.88k op/s 2024-03-18T22:53:12.575 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:53:12.576 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:53:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:12 smithi113 ceph-mon[66775]: pgmap v667: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 MiB/s wr, 4.20k op/s 2024-03-18T22:53:12.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:12 smithi103 ceph-mon[66377]: pgmap v667: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 MiB/s wr, 4.20k op/s 2024-03-18T22:53:12.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.879+0000 7f57ae073640 1 -- 172.21.15.40:0/468807658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a81098d0 msgr2=0x7f57a810e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:12.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.879+0000 7f57ae073640 1 --2- 172.21.15.40:0/468807658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a81098d0 0x7f57a810e190 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f579c009860 tx=0x7f579c02bc50 comp rx=0 tx=0).stop 2024-03-18T22:53:12.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.882+0000 7f57ae073640 1 -- 172.21.15.40:0/468807658 shutdown_connections 2024-03-18T22:53:12.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.882+0000 7f57ae073640 1 --2- 172.21.15.40:0/468807658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a81098d0 0x7f57a810e190 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.882+0000 7f57ae073640 1 --2- 172.21.15.40:0/468807658 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a8108f30 0x7f57a8109390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.882+0000 7f57ae073640 1 --2- 172.21.15.40:0/468807658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57a8072c00 0x7f57a8073020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.882+0000 7f57ae073640 1 -- 172.21.15.40:0/468807658 >> 172.21.15.40:0/468807658 conn(0x7f57a806e700 msgr2=0x7f57a8070b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:12.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.891+0000 7f57ae073640 1 -- 172.21.15.40:0/468807658 shutdown_connections 2024-03-18T22:53:12.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:12 smithi040 ceph-mon[62643]: pgmap v667: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 MiB/s wr, 4.20k op/s 2024-03-18T22:53:12.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.894+0000 7f57ae073640 1 -- 172.21.15.40:0/468807658 wait complete. 2024-03-18T22:53:12.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.895+0000 7f57ae073640 1 Processor -- start 2024-03-18T22:53:12.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.895+0000 7f57ae073640 1 -- start start 2024-03-18T22:53:12.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.895+0000 7f57ae073640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57a8072c00 0x7f57a81a6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:12.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.895+0000 7f57ae073640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a8108f30 0x7f57a81a6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:12.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.895+0000 7f57ae073640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a81098d0 0x7f57a81ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:12.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.895+0000 7f57ae073640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f57a811c0f0 con 0x7f57a81098d0 2024-03-18T22:53:12.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.895+0000 7f57ae073640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57a811bf70 con 0x7f57a8072c00 2024-03-18T22:53:12.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.895+0000 7f57ae073640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f57a811c270 con 0x7f57a8108f30 2024-03-18T22:53:12.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.899+0000 7f57a77fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57a8072c00 0x7f57a81a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:12.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.899+0000 7f57a7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a81098d0 0x7f57a81ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:12.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.899+0000 7f57a7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a81098d0 0x7f57a81ade20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39248/0 (socket says 172.21.15.40:39248) 2024-03-18T22:53:12.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.899+0000 7f57a7fff640 1 -- 172.21.15.40:0/4063284813 learned_addr learned my addr 172.21.15.40:0/4063284813 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:12.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.899+0000 7f57a77fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57a8072c00 0x7f57a81a6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:51994/0 (socket says 172.21.15.40:51994) 2024-03-18T22:53:12.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.902+0000 7f57a6ffd640 1 --2- 172.21.15.40:0/4063284813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a8108f30 0x7f57a81a6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:12.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.905+0000 7f57a6ffd640 1 -- 172.21.15.40:0/4063284813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57a8072c00 msgr2=0x7f57a81a6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:12.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.905+0000 7f57a6ffd640 1 --2- 172.21.15.40:0/4063284813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57a8072c00 0x7f57a81a6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.905+0000 7f57a6ffd640 1 -- 172.21.15.40:0/4063284813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a81098d0 msgr2=0x7f57a81ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:12.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.905+0000 7f57a6ffd640 1 --2- 172.21.15.40:0/4063284813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a81098d0 0x7f57a81ade20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.905+0000 7f57a6ffd640 1 -- 172.21.15.40:0/4063284813 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57a8110080 con 0x7f57a8108f30 2024-03-18T22:53:12.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.906+0000 7f57a6ffd640 1 --2- 172.21.15.40:0/4063284813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a8108f30 0x7f57a81a6db0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f579800e790 tx=0x7f579800ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:12.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.909+0000 7f57a4ff9640 1 -- 172.21.15.40:0/4063284813 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579800eea0 con 0x7f57a8108f30 2024-03-18T22:53:12.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.909+0000 7f57a4ff9640 1 -- 172.21.15.40:0/4063284813 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f579800ccc0 con 0x7f57a8108f30 2024-03-18T22:53:12.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.909+0000 7f57a4ff9640 1 -- 172.21.15.40:0/4063284813 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5798018710 con 0x7f57a8108f30 2024-03-18T22:53:12.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.909+0000 7f57ae073640 1 -- 172.21.15.40:0/4063284813 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57a8110370 con 0x7f57a8108f30 2024-03-18T22:53:12.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.910+0000 7f57ae073640 1 -- 172.21.15.40:0/4063284813 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57a8110870 con 0x7f57a8108f30 2024-03-18T22:53:12.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.915+0000 7f57a4ff9640 1 -- 172.21.15.40:0/4063284813 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f57980188b0 con 0x7f57a8108f30 2024-03-18T22:53:12.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.918+0000 7f57a4ff9640 1 --2- 172.21.15.40:0/4063284813 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57900a2990 0x7f57900a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:12.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.918+0000 7f57b0c5d740 1 Processor -- start 2024-03-18T22:53:12.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.918+0000 7f57a4ff9640 1 -- 172.21.15.40:0/4063284813 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f579809a560 con 0x7f57a8108f30 2024-03-18T22:53:12.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.918+0000 7f57b0c5d740 1 -- start start 2024-03-18T22:53:12.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.919+0000 7f57b0c5d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14ec50 0x55c52b14f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:12.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.919+0000 7f57b0c5d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b14f640 0x55c52b154140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:12.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.919+0000 7f57b0c5d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b1547b0 0x55c52b156c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:12.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.919+0000 7f57b0c5d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c52b0b3500 con 0x55c52b1547b0 2024-03-18T22:53:12.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.919+0000 7f57b0c5d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c52b05bbe0 con 0x55c52b14f640 2024-03-18T22:53:12.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.919+0000 7f57b0c5d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c52b0af5a0 con 0x55c52b14ec50 2024-03-18T22:53:12.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.919+0000 7f57a77fe640 1 --2- 172.21.15.40:0/4063284813 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57900a2990 0x7f57900a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:12.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.919+0000 7f57a7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b1547b0 0x55c52b156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:12.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.920+0000 7f57a7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b1547b0 0x55c52b156c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39260/0 (socket says 172.21.15.40:39260) 2024-03-18T22:53:12.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.920+0000 7f57a7fff640 1 -- 172.21.15.40:0/2496514900 learned_addr learned my addr 172.21.15.40:0/2496514900 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:12.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.920+0000 7f57a77fe640 1 --2- 172.21.15.40:0/2496514900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14ec50 0x55c52b14f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:12.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.921+0000 7f57a7fff640 1 -- 172.21.15.40:0/2496514900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14ec50 msgr2=0x55c52b14f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:12.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.921+0000 7f57a7fff640 1 --2- 172.21.15.40:0/2496514900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14ec50 0x55c52b14f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.921+0000 7f57a7fff640 1 -- 172.21.15.40:0/2496514900 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b14f640 msgr2=0x55c52b154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:12.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.921+0000 7f57a7fff640 1 --2- 172.21.15.40:0/2496514900 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b14f640 0x55c52b154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.921+0000 7f57a7fff640 1 -- 172.21.15.40:0/2496514900 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c52b07d000 con 0x55c52b1547b0 2024-03-18T22:53:12.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.921+0000 7f57a77fe640 1 --2- 172.21.15.40:0/4063284813 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57900a2990 0x7f57900a4e50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f57a0009fd0 tx=0x7f57a0013c00 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:12.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.925+0000 7f57a7fff640 1 --2- 172.21.15.40:0/2496514900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b1547b0 0x55c52b156c40 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f579c005b60 tx=0x7f579c009420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3454cd550596928c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:12.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.927+0000 7f578dffb640 1 -- 172.21.15.40:0/2496514900 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c039040 con 0x55c52b1547b0 2024-03-18T22:53:12.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.927+0000 7f578dffb640 1 -- 172.21.15.40:0/2496514900 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f579c0054f0 con 0x55c52b1547b0 2024-03-18T22:53:12.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.927+0000 7f578dffb640 1 -- 172.21.15.40:0/2496514900 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c004d00 con 0x55c52b1547b0 2024-03-18T22:53:12.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.932+0000 7f57b0c5d740 1 -- 172.21.15.40:0/2496514900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b1547b0 msgr2=0x55c52b156c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:12.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.932+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/2496514900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b1547b0 0x55c52b156c40 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f579c005b60 tx=0x7f579c009420 comp rx=0 tx=0).stop 2024-03-18T22:53:12.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.932+0000 7f57b0c5d740 1 -- 172.21.15.40:0/2496514900 shutdown_connections 2024-03-18T22:53:12.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.932+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/2496514900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b1547b0 0x55c52b156c40 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.932+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/2496514900 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b14f640 0x55c52b154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.932+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/2496514900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14ec50 0x55c52b14f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.932+0000 7f57b0c5d740 1 -- 172.21.15.40:0/2496514900 >> 172.21.15.40:0/2496514900 conn(0x55c52b14a1c0 msgr2=0x55c52b14a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:12.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.938+0000 7f57b0c5d740 1 -- 172.21.15.40:0/2496514900 shutdown_connections 2024-03-18T22:53:12.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.938+0000 7f57b0c5d740 1 -- 172.21.15.40:0/2496514900 wait complete. 2024-03-18T22:53:12.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.939+0000 7f57b0c5d740 1 -- start start 2024-03-18T22:53:12.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.939+0000 7f57b0c5d740 1 Processor -- start 2024-03-18T22:53:12.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.939+0000 7f57b0c5d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14ec50 0x55c52b159100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:12.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.939+0000 7f57b0c5d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14f640 0x55c52b159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:12.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.939+0000 7f57b0c5d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b1547b0 0x55c52b160680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:12.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.939+0000 7f57b0c5d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c52b157700 con 0x55c52b14f640 2024-03-18T22:53:12.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.940+0000 7f57b0c5d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c52b06f160 con 0x55c52b1547b0 2024-03-18T22:53:12.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.940+0000 7f57b0c5d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c52b157880 con 0x55c52b14ec50 2024-03-18T22:53:12.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.940+0000 7f57b0c5d740 10 client.0 _ll_register_callbacks cb 0x55c52b0aba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:12.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.940+0000 7f57a77fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14ec50 0x55c52b159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:12.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.940+0000 7f57a77fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14ec50 0x55c52b159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41672/0 (socket says 172.21.15.40:41672) 2024-03-18T22:53:12.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.940+0000 7f57a77fe640 1 -- 172.21.15.40:0/3963329732 learned_addr learned my addr 172.21.15.40:0/3963329732 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:12.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.940+0000 7f57a7fff640 1 --2- 172.21.15.40:0/3963329732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b1547b0 0x55c52b160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:12.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.940+0000 7f57a6ffd640 1 --2- 172.21.15.40:0/3963329732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14f640 0x55c52b159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:12.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.941+0000 7f57a77fe640 1 -- 172.21.15.40:0/3963329732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b1547b0 msgr2=0x55c52b160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:12.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.941+0000 7f57a77fe640 1 --2- 172.21.15.40:0/3963329732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b1547b0 0x55c52b160680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.941+0000 7f57a77fe640 1 -- 172.21.15.40:0/3963329732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14f640 msgr2=0x55c52b159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:12.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.941+0000 7f57a77fe640 1 --2- 172.21.15.40:0/3963329732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14f640 0x55c52b159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.941+0000 7f57a77fe640 1 -- 172.21.15.40:0/3963329732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c52ae917d0 con 0x55c52b14ec50 2024-03-18T22:53:12.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.946+0000 7f57a77fe640 1 --2- 172.21.15.40:0/3963329732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14ec50 0x55c52b159100 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f57a000fd60 tx=0x7f57a0004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=636229c7f5c5ec3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:12.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.946+0000 7f5786ffd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:53:12.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.949+0000 7f5786ffd640 1 -- 172.21.15.40:0/3963329732 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0004740 con 0x55c52b14ec50 2024-03-18T22:53:12.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.949+0000 7f5786ffd640 1 -- 172.21.15.40:0/3963329732 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f57a0010430 con 0x55c52b14ec50 2024-03-18T22:53:12.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.949+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3963329732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f579c0054f0 con 0x55c52b14ec50 2024-03-18T22:53:12.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.950+0000 7f5786ffd640 1 -- 172.21.15.40:0/3963329732 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f57a0010630 con 0x55c52b14ec50 2024-03-18T22:53:12.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.950+0000 7f57b0c5d740 10 client.27949 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:12.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.950+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3963329732 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c52adc73f0 con 0x55c52b14ec50 2024-03-18T22:53:12.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.950+0000 7f5786ffd640 1 -- 172.21.15.40:0/3963329732 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f579c0054f0 con 0x55c52b14ec50 2024-03-18T22:53:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.951+0000 7f5786ffd640 1 -- 172.21.15.40:0/3963329732 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f57a001f070 con 0x55c52b14ec50 2024-03-18T22:53:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.952+0000 7f57b0c5d740 10 client.27949 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.952+0000 7f57b0c5d740 10 client.27949 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:53:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.952+0000 7f57b0c5d740 20 client.27949 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.952+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3963329732 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c52b164f40 0x55c52b1673e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.952+0000 7f57b0c5d740 4 client.27949 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:53:12.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.952+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3963329732 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c52b0c0a40 con 0x55c52b164f40 2024-03-18T22:53:12.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.957+0000 7f57a6ffd640 1 --2- 172.21.15.40:0/3963329732 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c52b164f40 0x55c52b1673e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:12.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.958+0000 7f57a6ffd640 1 --2- 172.21.15.40:0/3963329732 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c52b164f40 0x55c52b1673e0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48c7eb71d746748f server_cookie=6f795cd1d396054b in_seq=0 out_seq=0 2024-03-18T22:53:12.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.960+0000 7f5786ffd640 10 client.27949 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:12.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.961+0000 7f5786ffd640 1 -- 172.21.15.40:0/3963329732 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c52b0c0a40 con 0x55c52b164f40 2024-03-18T22:53:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.961+0000 7f5786ffd640 10 client.27949 handle_command_reply: tid=0 2024-03-18T22:53:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.961+0000 7f57b0c5d740 1 client.27949 shutdown 2024-03-18T22:53:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.961+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3963329732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14ec50 msgr2=0x55c52b159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.961+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3963329732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14ec50 0x55c52b159100 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f57a000fd60 tx=0x7f57a0004ab0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.961+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3963329732 shutdown_connections 2024-03-18T22:53:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.961+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3963329732 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c52b164f40 0x55c52b1673e0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.961+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3963329732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b1547b0 0x55c52b160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.961+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3963329732 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14f640 0x55c52b159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.961+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3963329732 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14ec50 0x55c52b159100 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.961+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3963329732 >> 172.21.15.40:0/3963329732 conn(0x55c52b14a1c0 msgr2=0x55c52b1494a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.961+0000 7f5786ffd640 0 client.27949 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.961+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3963329732 shutdown_connections 2024-03-18T22:53:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.961+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3963329732 wait complete. 2024-03-18T22:53:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:12.962+0000 7f57b0c5d740 20 client.27949 trim_cache size 0 max 16384 2024-03-18T22:53:13.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.035+0000 7f57b0c5d740 1 Processor -- start 2024-03-18T22:53:13.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.035+0000 7f57b0c5d740 1 -- start start 2024-03-18T22:53:13.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.036+0000 7f57b0c5d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14a1c0 0x55c52b159420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:13.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.036+0000 7f57b0c5d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b14ec50 0x55c52b160640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:13.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.036+0000 7f57b0c5d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14f640 0x55c52b148c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:13.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.036+0000 7f57b0c5d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f57a81c1a90 con 0x55c52b14a1c0 2024-03-18T22:53:13.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.036+0000 7f57b0c5d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57a81c36c0 con 0x55c52b14ec50 2024-03-18T22:53:13.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.036+0000 7f57b0c5d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f57a81c2160 con 0x55c52b14f640 2024-03-18T22:53:13.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.036+0000 7f57a77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14a1c0 0x55c52b159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:13.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.036+0000 7f57a7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14f640 0x55c52b148c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:13.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.036+0000 7f57a77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14a1c0 0x55c52b159420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39282/0 (socket says 172.21.15.40:39282) 2024-03-18T22:53:13.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.036+0000 7f57a77fe640 1 -- 172.21.15.40:0/3964681588 learned_addr learned my addr 172.21.15.40:0/3964681588 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:13.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.036+0000 7f57a6ffd640 1 --2- 172.21.15.40:0/3964681588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b14ec50 0x55c52b160640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:13.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.037+0000 7f57a7fff640 1 -- 172.21.15.40:0/3964681588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b14ec50 msgr2=0x55c52b160640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:13.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.037+0000 7f57a7fff640 1 --2- 172.21.15.40:0/3964681588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b14ec50 0x55c52b160640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.037+0000 7f57a7fff640 1 -- 172.21.15.40:0/3964681588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14a1c0 msgr2=0x55c52b159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:13.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.037+0000 7f57a7fff640 1 --2- 172.21.15.40:0/3964681588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14a1c0 0x55c52b159420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.037+0000 7f57a7fff640 1 -- 172.21.15.40:0/3964681588 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c52ad870f0 con 0x55c52b14f640 2024-03-18T22:53:13.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.037+0000 7f57a77fe640 1 --2- 172.21.15.40:0/3964681588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14a1c0 0x55c52b159420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:53:13.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.038+0000 7f57a7fff640 1 --2- 172.21.15.40:0/3964681588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14f640 0x55c52b148c10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f579c009490 tx=0x7f579c02e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b412edc2bd264945 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:13.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.041+0000 7f578d7fa640 1 -- 172.21.15.40:0/3964681588 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c039070 con 0x55c52b14f640 2024-03-18T22:53:13.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.041+0000 7f578d7fa640 1 -- 172.21.15.40:0/3964681588 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f579c004240 con 0x55c52b14f640 2024-03-18T22:53:13.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.042+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3964681588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14f640 msgr2=0x55c52b148c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:13.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.042+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3964681588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14f640 0x55c52b148c10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f579c009490 tx=0x7f579c02e040 comp rx=0 tx=0).stop 2024-03-18T22:53:13.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.045+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3964681588 shutdown_connections 2024-03-18T22:53:13.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.045+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3964681588 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14f640 0x55c52b148c10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.045+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3964681588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b14ec50 0x55c52b160640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.045+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3964681588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14a1c0 0x55c52b159420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.045+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3964681588 >> 172.21.15.40:0/3964681588 conn(0x55c52b150420 msgr2=0x55c52b158d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:13.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.050+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3964681588 shutdown_connections 2024-03-18T22:53:13.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.050+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3964681588 wait complete. 2024-03-18T22:53:13.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.050+0000 7f57b0c5d740 1 -- start start 2024-03-18T22:53:13.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.051+0000 7f57b0c5d740 1 Processor -- start 2024-03-18T22:53:13.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.051+0000 7f57b0c5d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b14a1c0 0x55c52b15ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:13.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.051+0000 7f57b0c5d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14ec50 0x55c52b15e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:13.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.051+0000 7f57b0c5d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14f640 0x55c52b15f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:13.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.051+0000 7f57b0c5d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f57a81c2ff0 con 0x55c52b14ec50 2024-03-18T22:53:13.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.051+0000 7f57b0c5d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57a81c2860 con 0x55c52b14a1c0 2024-03-18T22:53:13.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.051+0000 7f57b0c5d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f57a81184b0 con 0x55c52b14f640 2024-03-18T22:53:13.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.051+0000 7f57b0c5d740 10 client.0 _ll_register_callbacks cb 0x7f57a0013e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.052+0000 7f57a6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14ec50 0x55c52b15e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.052+0000 7f57a6ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14ec50 0x55c52b15e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39284/0 (socket says 172.21.15.40:39284) 2024-03-18T22:53:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.052+0000 7f57a6ffd640 1 -- 172.21.15.40:0/3461251498 learned_addr learned my addr 172.21.15.40:0/3461251498 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.052+0000 7f57a77fe640 1 --2- 172.21.15.40:0/3461251498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b14a1c0 0x55c52b15ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:13.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.052+0000 7f57a7fff640 1 --2- 172.21.15.40:0/3461251498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14f640 0x55c52b15f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:13.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.056+0000 7f57a6ffd640 1 -- 172.21.15.40:0/3461251498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14f640 msgr2=0x55c52b15f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:13.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.056+0000 7f57a6ffd640 1 --2- 172.21.15.40:0/3461251498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14f640 0x55c52b15f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.056+0000 7f57a6ffd640 1 -- 172.21.15.40:0/3461251498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b14a1c0 msgr2=0x55c52b15ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:13.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.056+0000 7f57a6ffd640 1 --2- 172.21.15.40:0/3461251498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b14a1c0 0x55c52b15ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.056+0000 7f57a6ffd640 1 -- 172.21.15.40:0/3461251498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c52b15f930 con 0x55c52b14ec50 2024-03-18T22:53:13.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.063+0000 7f57a6ffd640 1 --2- 172.21.15.40:0/3461251498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14ec50 0x55c52b15e330 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f57980993d0 tx=0x7f57980a4760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd9fec4f2629cd2e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:13.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.070+0000 7f5786ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:53:13.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.070+0000 7f5786ffd640 1 -- 172.21.15.40:0/3461251498 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5798098ef0 con 0x55c52b14ec50 2024-03-18T22:53:13.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.070+0000 7f5786ffd640 1 -- 172.21.15.40:0/3461251498 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5798098210 con 0x55c52b14ec50 2024-03-18T22:53:13.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.072+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3461251498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c52b0bad90 con 0x55c52b14ec50 2024-03-18T22:53:13.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.075+0000 7f5786ffd640 1 -- 172.21.15.40:0/3461251498 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5798099090 con 0x55c52b14ec50 2024-03-18T22:53:13.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.075+0000 7f57b0c5d740 10 client.18138 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:13.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.075+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3461251498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c52b1cb430 con 0x55c52b14ec50 2024-03-18T22:53:13.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.076+0000 7f5786ffd640 1 -- 172.21.15.40:0/3461251498 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55c52b0bad90 con 0x55c52b14ec50 2024-03-18T22:53:13.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.076+0000 7f5786ffd640 1 -- 172.21.15.40:0/3461251498 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f5798098590 con 0x55c52b14ec50 2024-03-18T22:53:13.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.076+0000 7f57b0c5d740 10 client.18138 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:13.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.076+0000 7f57b0c5d740 10 client.18138 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:53:13.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.076+0000 7f57b0c5d740 20 client.18138 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:13.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.076+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3461251498 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c52b1547b0 0x55c52b1cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:13.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.076+0000 7f57b0c5d740 4 client.18138 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:53:13.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.076+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3461251498 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c52b038540 con 0x55c52b1547b0 2024-03-18T22:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.076+0000 7f57a77fe640 1 --2- 172.21.15.40:0/3461251498 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c52b1547b0 0x55c52b1cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.077+0000 7f5786ffd640 10 client.18138 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.077+0000 7f57a77fe640 1 --2- 172.21.15.40:0/3461251498 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c52b1547b0 0x55c52b1cddd0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0cee95a7d3c8cce server_cookie=a860f1c647c0ef23 in_seq=0 out_seq=0 2024-03-18T22:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.077+0000 7f5786ffd640 1 -- 172.21.15.40:0/3461251498 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c52b038540 con 0x55c52b1547b0 2024-03-18T22:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.077+0000 7f5786ffd640 10 client.18138 handle_command_reply: tid=0 2024-03-18T22:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.077+0000 7f57b0c5d740 1 client.18138 shutdown 2024-03-18T22:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.078+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3461251498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14ec50 msgr2=0x55c52b15e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.078+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3461251498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14ec50 0x55c52b15e330 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f57980993d0 tx=0x7f57980a4760 comp rx=0 tx=0).stop 2024-03-18T22:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.078+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3461251498 shutdown_connections 2024-03-18T22:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.078+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3461251498 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c52b14f640 0x55c52b15f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.078+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3461251498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c52b14ec50 0x55c52b15e330 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.078+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3461251498 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c52b1547b0 0x55c52b1cddd0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.078+0000 7f57b0c5d740 1 --2- 172.21.15.40:0/3461251498 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c52b14a1c0 0x55c52b15ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.078+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3461251498 >> 172.21.15.40:0/3461251498 conn(0x55c52b150420 msgr2=0x55c52b14d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.078+0000 7f5786ffd640 0 client.18138 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.078+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3461251498 shutdown_connections 2024-03-18T22:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.078+0000 7f57b0c5d740 1 -- 172.21.15.40:0/3461251498 wait complete. 2024-03-18T22:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.078+0000 7f57b0c5d740 20 client.18138 trim_cache size 0 max 16384 2024-03-18T22:53:13.080 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:53:13.081 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:53:13.081 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:53:13.081 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:53:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.079+0000 7f57ae073640 1 -- 172.21.15.40:0/4063284813 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57900a2990 msgr2=0x7f57900a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.079+0000 7f57ae073640 1 --2- 172.21.15.40:0/4063284813 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57900a2990 0x7f57900a4e50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f57a0009fd0 tx=0x7f57a0013c00 comp rx=0 tx=0).stop 2024-03-18T22:53:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.079+0000 7f57ae073640 1 -- 172.21.15.40:0/4063284813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a8108f30 msgr2=0x7f57a81a6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.079+0000 7f57ae073640 1 --2- 172.21.15.40:0/4063284813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a8108f30 0x7f57a81a6db0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f579800e790 tx=0x7f579800ec50 comp rx=0 tx=0).stop 2024-03-18T22:53:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.079+0000 7f57ae073640 1 -- 172.21.15.40:0/4063284813 shutdown_connections 2024-03-18T22:53:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.079+0000 7f57ae073640 1 --2- 172.21.15.40:0/4063284813 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57900a2990 0x7f57900a4e50 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.079+0000 7f57ae073640 1 --2- 172.21.15.40:0/4063284813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57a81098d0 0x7f57a81ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.079+0000 7f57ae073640 1 --2- 172.21.15.40:0/4063284813 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57a8108f30 0x7f57a81a6db0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.079+0000 7f57ae073640 1 --2- 172.21.15.40:0/4063284813 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57a8072c00 0x7f57a81a6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.079+0000 7f57ae073640 1 -- 172.21.15.40:0/4063284813 >> 172.21.15.40:0/4063284813 conn(0x7f57a806e700 msgr2=0x7f57a80704d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:13.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.079+0000 7f57ae073640 1 -- 172.21.15.40:0/4063284813 shutdown_connections 2024-03-18T22:53:13.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.079+0000 7f57ae073640 1 -- 172.21.15.40:0/4063284813 wait complete. 2024-03-18T22:53:13.091 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:53:13.091 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:53:13.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.201+0000 7f36e9061640 1 -- 172.21.15.40:0/4109515459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e4101940 msgr2=0x7f36e4101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:13.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.201+0000 7f36e9061640 1 --2- 172.21.15.40:0/4109515459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e4101940 0x7f36e4101dc0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009900 tx=0x7f36d802bc30 comp rx=0 tx=0).stop 2024-03-18T22:53:13.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.203+0000 7f36e9061640 1 -- 172.21.15.40:0/4109515459 shutdown_connections 2024-03-18T22:53:13.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.203+0000 7f36e9061640 1 --2- 172.21.15.40:0/4109515459 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36e4102300 0x7f36e4108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.203+0000 7f36e9061640 1 --2- 172.21.15.40:0/4109515459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e4101940 0x7f36e4101dc0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.203+0000 7f36e9061640 1 --2- 172.21.15.40:0/4109515459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36e4100750 0x7f36e4100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.203+0000 7f36e9061640 1 -- 172.21.15.40:0/4109515459 >> 172.21.15.40:0/4109515459 conn(0x7f36e40fbd00 msgr2=0x7f36e40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:13.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.206+0000 7f36e9061640 1 -- 172.21.15.40:0/4109515459 shutdown_connections 2024-03-18T22:53:13.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.206+0000 7f36e9061640 1 -- 172.21.15.40:0/4109515459 wait complete. 2024-03-18T22:53:13.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.207+0000 7f36e9061640 1 Processor -- start 2024-03-18T22:53:13.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.207+0000 7f36e9061640 1 -- start start 2024-03-18T22:53:13.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.208+0000 7f36e9061640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36e4100750 0x7f36e4195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:13.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.208+0000 7f36e9061640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36e4101940 0x7f36e41964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:13.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.208+0000 7f36e9061640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e4102300 0x7f36e419ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:13.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.208+0000 7f36e2d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36e4100750 0x7f36e4195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:13.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.208+0000 7f36e2d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36e4100750 0x7f36e4195f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41702/0 (socket says 172.21.15.40:41702) 2024-03-18T22:53:13.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.208+0000 7f36e259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36e4101940 0x7f36e41964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:13.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.208+0000 7f36e2d9d640 1 -- 172.21.15.40:0/3426805926 learned_addr learned my addr 172.21.15.40:0/3426805926 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:13.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.208+0000 7f36e9061640 1 -- 172.21.15.40:0/3426805926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f36e410c280 con 0x7f36e4102300 2024-03-18T22:53:13.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.208+0000 7f36e9061640 1 -- 172.21.15.40:0/3426805926 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f36e410c100 con 0x7f36e4101940 2024-03-18T22:53:13.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.208+0000 7f36e9061640 1 -- 172.21.15.40:0/3426805926 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f36e410c400 con 0x7f36e4100750 2024-03-18T22:53:13.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.209+0000 7f36e2d9d640 1 -- 172.21.15.40:0/3426805926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36e4101940 msgr2=0x7f36e41964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:13.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.209+0000 7f36e2d9d640 1 --2- 172.21.15.40:0/3426805926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36e4101940 0x7f36e41964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.209+0000 7f36e2d9d640 1 -- 172.21.15.40:0/3426805926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e4102300 msgr2=0x7f36e419ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:13.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.209+0000 7f36e2d9d640 1 --2- 172.21.15.40:0/3426805926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e4102300 0x7f36e419ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.209+0000 7f36e2d9d640 1 -- 172.21.15.40:0/3426805926 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e419d4e0 con 0x7f36e4100750 2024-03-18T22:53:13.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.209+0000 7f36e2d9d640 1 --2- 172.21.15.40:0/3426805926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36e4100750 0x7f36e4195f90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f36d400e9c0 tx=0x7f36d400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:13.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.210+0000 7f36cbfff640 1 -- 172.21.15.40:0/3426805926 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d400cde0 con 0x7f36e4100750 2024-03-18T22:53:13.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.210+0000 7f36e9061640 1 -- 172.21.15.40:0/3426805926 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36e419d7d0 con 0x7f36e4100750 2024-03-18T22:53:13.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.210+0000 7f36e9061640 1 -- 172.21.15.40:0/3426805926 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36e419dd20 con 0x7f36e4100750 2024-03-18T22:53:13.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.210+0000 7f36cbfff640 1 -- 172.21.15.40:0/3426805926 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36d40108e0 con 0x7f36e4100750 2024-03-18T22:53:13.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.210+0000 7f36cbfff640 1 -- 172.21.15.40:0/3426805926 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d4018990 con 0x7f36e4100750 2024-03-18T22:53:13.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.219+0000 7f36cbfff640 1 -- 172.21.15.40:0/3426805926 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f36d401f020 con 0x7f36e4100750 2024-03-18T22:53:13.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.220+0000 7f36e9061640 1 -- 172.21.15.40:0/3426805926 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36d00052c0 con 0x7f36e4100750 2024-03-18T22:53:13.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.227+0000 7f36cbfff640 1 --2- 172.21.15.40:0/3426805926 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f36cc0a29d0 0x7f36cc0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:13.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.227+0000 7f36cbfff640 1 -- 172.21.15.40:0/3426805926 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f36d40cf090 con 0x7f36e4100750 2024-03-18T22:53:13.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.230+0000 7f36e259c640 1 --2- 172.21.15.40:0/3426805926 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f36cc0a29d0 0x7f36cc0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:13.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.233+0000 7f36cbfff640 1 -- 172.21.15.40:0/3426805926 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f36d409bce0 con 0x7f36e4100750 2024-03-18T22:53:13.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.240+0000 7f36e259c640 1 --2- 172.21.15.40:0/3426805926 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f36cc0a29d0 0x7f36cc0a4e90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009230 tx=0x7f36d8005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:13.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.751+0000 7f36e9061640 1 -- 172.21.15.40:0/3426805926 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36d00055b0 con 0x7f36e4100750 2024-03-18T22:53:13.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.754+0000 7f36cbfff640 1 -- 172.21.15.40:0/3426805926 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11955 (secure 0 0 0) 0x7f36d409b5e0 con 0x7f36e4100750 2024-03-18T22:53:13.758 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:53:13.758 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:52:41.445175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24509,24481,24473,24388,24401]},"id":1}]} 2024-03-18T22:53:13.758 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 37 2024-03-18T22:53:13.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.769+0000 7f36c9ffb640 1 -- 172.21.15.40:0/3426805926 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f36cc0a29d0 msgr2=0x7f36cc0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:13.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.769+0000 7f36c9ffb640 1 --2- 172.21.15.40:0/3426805926 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f36cc0a29d0 0x7f36cc0a4e90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009230 tx=0x7f36d8005fd0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.769+0000 7f36c9ffb640 1 -- 172.21.15.40:0/3426805926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36e4100750 msgr2=0x7f36e4195f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:13.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.769+0000 7f36c9ffb640 1 --2- 172.21.15.40:0/3426805926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36e4100750 0x7f36e4195f90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f36d400e9c0 tx=0x7f36d400ee80 comp rx=0 tx=0).stop 2024-03-18T22:53:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.781+0000 7f36c9ffb640 1 -- 172.21.15.40:0/3426805926 shutdown_connections 2024-03-18T22:53:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.781+0000 7f36c9ffb640 1 --2- 172.21.15.40:0/3426805926 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f36cc0a29d0 0x7f36cc0a4e90 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.781+0000 7f36c9ffb640 1 --2- 172.21.15.40:0/3426805926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36e4102300 0x7f36e419ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.781+0000 7f36c9ffb640 1 --2- 172.21.15.40:0/3426805926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f36e4101940 0x7f36e41964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.781+0000 7f36c9ffb640 1 --2- 172.21.15.40:0/3426805926 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f36e4100750 0x7f36e4195f90 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:13.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.781+0000 7f36c9ffb640 1 -- 172.21.15.40:0/3426805926 >> 172.21.15.40:0/3426805926 conn(0x7f36e40fbd00 msgr2=0x7f36e40fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.782+0000 7f36c9ffb640 1 -- 172.21.15.40:0/3426805926 shutdown_connections 2024-03-18T22:53:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:13.782+0000 7f36c9ffb640 1 -- 172.21.15.40:0/3426805926 wait complete. 2024-03-18T22:53:13.819 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:53:14.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.099+0000 7fe8bf3e6640 1 -- 172.21.15.40:0/667733564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b8072d30 msgr2=0x7fe8b80731b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.099+0000 7fe8bf3e6640 1 --2- 172.21.15.40:0/667733564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b8072d30 0x7fe8b80731b0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8009900 tx=0x7fe8a802bc80 comp rx=0 tx=0).stop 2024-03-18T22:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.099+0000 7fe8bf3e6640 1 -- 172.21.15.40:0/667733564 shutdown_connections 2024-03-18T22:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.099+0000 7fe8bf3e6640 1 --2- 172.21.15.40:0/667733564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b810fb60 0x7fe8b8111f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.099+0000 7fe8bf3e6640 1 --2- 172.21.15.40:0/667733564 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b8072d30 0x7fe8b80731b0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.099+0000 7fe8bf3e6640 1 --2- 172.21.15.40:0/667733564 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8b8109430 0x7fe8b80727f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.099+0000 7fe8bf3e6640 1 -- 172.21.15.40:0/667733564 >> 172.21.15.40:0/667733564 conn(0x7fe8b806e640 msgr2=0x7fe8b8070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.101+0000 7fe8bf3e6640 1 -- 172.21.15.40:0/667733564 shutdown_connections 2024-03-18T22:53:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.101+0000 7fe8bf3e6640 1 -- 172.21.15.40:0/667733564 wait complete. 2024-03-18T22:53:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.102+0000 7fe8bf3e6640 1 Processor -- start 2024-03-18T22:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.102+0000 7fe8bf3e6640 1 -- start start 2024-03-18T22:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.102+0000 7fe8bf3e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b8072d30 0x7fe8b8119020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.103+0000 7fe8bf3e6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b8109430 0x7fe8b8119560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.103+0000 7fe8bf3e6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8b810fb60 0x7fe8b811a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.103+0000 7fe8bf3e6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b81137d0 con 0x7fe8b8072d30 2024-03-18T22:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.103+0000 7fe8bf3e6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b8113650 con 0x7fe8b8109430 2024-03-18T22:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.103+0000 7fe8bf3e6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b8113950 con 0x7fe8b810fb60 2024-03-18T22:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.103+0000 7fe8bc981640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b8109430 0x7fe8b8119560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.103+0000 7fe8bd182640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b8072d30 0x7fe8b8119020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.103+0000 7fe8bc981640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b8109430 0x7fe8b8119560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52080/0 (socket says 172.21.15.40:52080) 2024-03-18T22:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.103+0000 7fe8bc981640 1 -- 172.21.15.40:0/2074920048 learned_addr learned my addr 172.21.15.40:0/2074920048 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.103+0000 7fe8bd983640 1 --2- 172.21.15.40:0/2074920048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8b810fb60 0x7fe8b811a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.104+0000 7fe8bc981640 1 -- 172.21.15.40:0/2074920048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8b810fb60 msgr2=0x7fe8b811a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.104+0000 7fe8bc981640 1 --2- 172.21.15.40:0/2074920048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8b810fb60 0x7fe8b811a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.104+0000 7fe8bc981640 1 -- 172.21.15.40:0/2074920048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b8072d30 msgr2=0x7fe8b8119020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.104+0000 7fe8bc981640 1 --2- 172.21.15.40:0/2074920048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b8072d30 0x7fe8b8119020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.104+0000 7fe8bc981640 1 -- 172.21.15.40:0/2074920048 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8b811ac10 con 0x7fe8b8109430 2024-03-18T22:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.104+0000 7fe8bc981640 1 --2- 172.21.15.40:0/2074920048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b8109430 0x7fe8b8119560 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe8a802c290 tx=0x7fe8a80043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.105+0000 7fe8ae7fc640 1 -- 172.21.15.40:0/2074920048 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a8043020 con 0x7fe8b8109430 2024-03-18T22:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.105+0000 7fe8bf3e6640 1 -- 172.21.15.40:0/2074920048 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8b811aea0 con 0x7fe8b8109430 2024-03-18T22:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.105+0000 7fe8bf3e6640 1 -- 172.21.15.40:0/2074920048 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8b811b4c0 con 0x7fe8b8109430 2024-03-18T22:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.105+0000 7fe8ae7fc640 1 -- 172.21.15.40:0/2074920048 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8a80054c0 con 0x7fe8b8109430 2024-03-18T22:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.105+0000 7fe8ae7fc640 1 -- 172.21.15.40:0/2074920048 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a8003d60 con 0x7fe8b8109430 2024-03-18T22:53:14.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.111+0000 7fe8ae7fc640 1 -- 172.21.15.40:0/2074920048 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe8a8046020 con 0x7fe8b8109430 2024-03-18T22:53:14.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.111+0000 7fe8ae7fc640 1 --2- 172.21.15.40:0/2074920048 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe8a40a2940 0x7fe8a40a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.112+0000 7fe8bd182640 1 --2- 172.21.15.40:0/2074920048 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe8a40a2940 0x7fe8a40a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.112+0000 7fe8ae7fc640 1 -- 172.21.15.40:0/2074920048 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe8a80f11a0 con 0x7fe8b8109430 2024-03-18T22:53:14.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.114+0000 7fe8bd182640 1 --2- 172.21.15.40:0/2074920048 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe8a40a2940 0x7fe8a40a4e00 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4006fd0 tx=0x7fe8b40090e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.114+0000 7fe8c1fce740 1 Processor -- start 2024-03-18T22:53:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.115+0000 7fe8c1fce740 1 -- start start 2024-03-18T22:53:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.115+0000 7fe8c1fce740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686490 0x55cda26868b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.115+0000 7fe8c1fce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda2686e80 0x55cda268b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.115+0000 7fe8c1fce740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda268bff0 0x55cda268e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.115+0000 7fe8c1fce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cda25ea4b0 con 0x55cda2686e80 2024-03-18T22:53:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.115+0000 7fe8c1fce740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cda2592d70 con 0x55cda268bff0 2024-03-18T22:53:14.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.115+0000 7fe8c1fce740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55cda25e6550 con 0x55cda2686490 2024-03-18T22:53:14.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.118+0000 7fe8bd983640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda268bff0 0x55cda268e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.118+0000 7fe8bd983640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda268bff0 0x55cda268e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52094/0 (socket says 172.21.15.40:52094) 2024-03-18T22:53:14.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.118+0000 7fe8bd983640 1 -- 172.21.15.40:0/4253882610 learned_addr learned my addr 172.21.15.40:0/4253882610 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:14.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.118+0000 7fe8bd182640 1 --2- 172.21.15.40:0/4253882610 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686490 0x55cda26868b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.118+0000 7fe8bd983640 1 -- 172.21.15.40:0/4253882610 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686490 msgr2=0x55cda26868b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.118+0000 7fe8bd983640 1 --2- 172.21.15.40:0/4253882610 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686490 0x55cda26868b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.119+0000 7fe8bd983640 1 -- 172.21.15.40:0/4253882610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda2686e80 msgr2=0x55cda268b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.119+0000 7fe8bd983640 1 --2- 172.21.15.40:0/4253882610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda2686e80 0x55cda268b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.119+0000 7fe8bd983640 1 -- 172.21.15.40:0/4253882610 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cda25b5210 con 0x55cda268bff0 2024-03-18T22:53:14.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.120+0000 7fe8bd983640 1 --2- 172.21.15.40:0/4253882610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda268bff0 0x55cda268e480 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000a4d0 tx=0x7fe8b0009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5070859ad51e305 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.122+0000 7fe8a37fe640 1 -- 172.21.15.40:0/4253882610 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0018020 con 0x55cda268bff0 2024-03-18T22:53:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.122+0000 7fe8a37fe640 1 -- 172.21.15.40:0/4253882610 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8b0014cc0 con 0x55cda268bff0 2024-03-18T22:53:14.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.123+0000 7fe8c1fce740 1 -- 172.21.15.40:0/4253882610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda268bff0 msgr2=0x55cda268e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.123+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/4253882610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda268bff0 0x55cda268e480 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000a4d0 tx=0x7fe8b0009650 comp rx=0 tx=0).stop 2024-03-18T22:53:14.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.124+0000 7fe8c1fce740 1 -- 172.21.15.40:0/4253882610 shutdown_connections 2024-03-18T22:53:14.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.124+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/4253882610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda268bff0 0x55cda268e480 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.124+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/4253882610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda2686e80 0x55cda268b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.124+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/4253882610 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686490 0x55cda26868b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.124+0000 7fe8c1fce740 1 -- 172.21.15.40:0/4253882610 >> 172.21.15.40:0/4253882610 conn(0x55cda2681a00 msgr2=0x55cda2681e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:14.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.124+0000 7fe8c1fce740 1 -- 172.21.15.40:0/4253882610 shutdown_connections 2024-03-18T22:53:14.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.124+0000 7fe8c1fce740 1 -- 172.21.15.40:0/4253882610 wait complete. 2024-03-18T22:53:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.125+0000 7fe8c1fce740 1 -- start start 2024-03-18T22:53:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.125+0000 7fe8c1fce740 1 Processor -- start 2024-03-18T22:53:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.126+0000 7fe8c1fce740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686490 0x55cda2690860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.126+0000 7fe8c1fce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda2686e80 0x55cda2690da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.126+0000 7fe8c1fce740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda268bff0 0x55cda2697de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.126+0000 7fe8c1fce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cda268eff0 con 0x55cda2686e80 2024-03-18T22:53:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.126+0000 7fe8c1fce740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cda25a6310 con 0x55cda268bff0 2024-03-18T22:53:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.126+0000 7fe8c1fce740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55cda268f170 con 0x55cda2686490 2024-03-18T22:53:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.126+0000 7fe8bd182640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686490 0x55cda2690860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.126+0000 7fe8c1fce740 10 client.0 _ll_register_callbacks cb 0x55cda259a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.126+0000 7fe8bd182640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686490 0x55cda2690860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41736/0 (socket says 172.21.15.40:41736) 2024-03-18T22:53:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.126+0000 7fe8bd182640 1 -- 172.21.15.40:0/3907693574 learned_addr learned my addr 172.21.15.40:0/3907693574 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:14.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.126+0000 7fe8bd983640 1 --2- 172.21.15.40:0/3907693574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda268bff0 0x55cda2697de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.127+0000 7fe8bd182640 1 -- 172.21.15.40:0/3907693574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda268bff0 msgr2=0x55cda2697de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.127+0000 7fe8bd182640 1 --2- 172.21.15.40:0/3907693574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda268bff0 0x55cda2697de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.127+0000 7fe8bd182640 1 -- 172.21.15.40:0/3907693574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda2686e80 msgr2=0x55cda2690da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.127+0000 7fe8bd182640 1 --2- 172.21.15.40:0/3907693574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda2686e80 0x55cda2690da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.127+0000 7fe8bd182640 1 -- 172.21.15.40:0/3907693574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cda22fe3f0 con 0x55cda2686490 2024-03-18T22:53:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.127+0000 7fe8bd182640 1 --2- 172.21.15.40:0/3907693574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686490 0x55cda2690860 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4015730 tx=0x7fe8b4004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76b0bc2fbf64b7de server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.127+0000 7fe8a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:53:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.128+0000 7fe8a0ff9640 1 -- 172.21.15.40:0/3907693574 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4025070 con 0x55cda2686490 2024-03-18T22:53:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.128+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3907693574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8b0014cc0 con 0x55cda2686490 2024-03-18T22:53:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.128+0000 7fe8a0ff9640 1 -- 172.21.15.40:0/3907693574 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe8b40137d0 con 0x55cda2686490 2024-03-18T22:53:14.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.128+0000 7fe8a0ff9640 1 -- 172.21.15.40:0/3907693574 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe8b4031050 con 0x55cda2686490 2024-03-18T22:53:14.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.132+0000 7fe8c1fce740 10 client.27970 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:14.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.132+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3907693574 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cda23c87d0 con 0x55cda2686490 2024-03-18T22:53:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.133+0000 7fe8a0ff9640 1 -- 172.21.15.40:0/3907693574 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8b0014cc0 con 0x55cda2686490 2024-03-18T22:53:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.134+0000 7fe8a0ff9640 1 -- 172.21.15.40:0/3907693574 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7fe8b4021c10 con 0x55cda2686490 2024-03-18T22:53:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.134+0000 7fe8c1fce740 10 client.27970 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.134+0000 7fe8c1fce740 10 client.27970 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:53:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.134+0000 7fe8c1fce740 20 client.27970 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.134+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/3907693574 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55cda269cc30 0x55cda269f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.134+0000 7fe8c1fce740 4 client.27970 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:53:14.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.134+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3907693574 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cda26483e0 con 0x55cda269cc30 2024-03-18T22:53:14.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.140+0000 7fe8bc981640 1 --2- 172.21.15.40:0/3907693574 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55cda269cc30 0x55cda269f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.150+0000 7fe8bc981640 1 --2- 172.21.15.40:0/3907693574 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55cda269cc30 0x55cda269f0f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8b67b9af6002d5c5 server_cookie=77738a5d27359a4d in_seq=0 out_seq=0 2024-03-18T22:53:14.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.150+0000 7fe8a0ff9640 10 client.27970 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:53:14.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.157+0000 7fe8a0ff9640 1 -- 172.21.15.40:0/3907693574 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cda26483e0 con 0x55cda269cc30 2024-03-18T22:53:14.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.157+0000 7fe8a0ff9640 10 client.27970 handle_command_reply: tid=0 2024-03-18T22:53:14.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.157+0000 7fe8c1fce740 1 client.27970 shutdown 2024-03-18T22:53:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.164+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3907693574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686490 msgr2=0x55cda2690860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.164+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/3907693574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686490 0x55cda2690860 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4015730 tx=0x7fe8b4004750 comp rx=0 tx=0).stop 2024-03-18T22:53:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.164+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3907693574 shutdown_connections 2024-03-18T22:53:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.164+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/3907693574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda268bff0 0x55cda2697de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.164+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/3907693574 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55cda269cc30 0x55cda269f0f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.164+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/3907693574 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda2686e80 0x55cda2690da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.164+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/3907693574 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686490 0x55cda2690860 secure :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4015730 tx=0x7fe8b4004750 comp rx=0 tx=0).stop 2024-03-18T22:53:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.164+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3907693574 >> 172.21.15.40:0/3907693574 conn(0x55cda2681a00 msgr2=0x55cda2680d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.165+0000 7fe8a0ff9640 0 client.27970 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:53:14.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.169+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3907693574 shutdown_connections 2024-03-18T22:53:14.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.169+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3907693574 wait complete. 2024-03-18T22:53:14.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.169+0000 7fe8c1fce740 20 client.27970 trim_cache size 0 max 16384 2024-03-18T22:53:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.208+0000 7fe8c1fce740 1 Processor -- start 2024-03-18T22:53:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.208+0000 7fe8c1fce740 1 -- start start 2024-03-18T22:53:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.209+0000 7fe8c1fce740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda2686e80 0x55cda2681de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.209+0000 7fe8c1fce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda268bff0 0x55cda2697d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.209+0000 7fe8bd182640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda2686e80 0x55cda2681de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.209+0000 7fe8bd182640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda2686e80 0x55cda2681de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52112/0 (socket says 172.21.15.40:52112) 2024-03-18T22:53:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.209+0000 7fe8bc981640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda268bff0 0x55cda2697d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.209+0000 7fe8bc981640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda268bff0 0x55cda2697d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39348/0 (socket says 172.21.15.40:39348) 2024-03-18T22:53:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.209+0000 7fe8bd182640 1 -- 172.21.15.40:0/4030102309 learned_addr learned my addr 172.21.15.40:0/4030102309 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.209+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/4030102309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda26815a0 0x55cda267cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.209+0000 7fe8c1fce740 1 -- 172.21.15.40:0/4030102309 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c00feb0 con 0x55cda268bff0 2024-03-18T22:53:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.210+0000 7fe8c1fce740 1 -- 172.21.15.40:0/4030102309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c011ae0 con 0x55cda2686e80 2024-03-18T22:53:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.210+0000 7fe8bd983640 1 --2- 172.21.15.40:0/4030102309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda26815a0 0x55cda267cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.210+0000 7fe8c1fce740 1 -- 172.21.15.40:0/4030102309 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c010580 con 0x55cda26815a0 2024-03-18T22:53:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.210+0000 7fe8bd182640 1 -- 172.21.15.40:0/4030102309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda26815a0 msgr2=0x55cda267cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.210+0000 7fe8bd182640 1 --2- 172.21.15.40:0/4030102309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda26815a0 0x55cda267cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.210+0000 7fe8bd182640 1 -- 172.21.15.40:0/4030102309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda268bff0 msgr2=0x55cda2697d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.210+0000 7fe8bd182640 1 --2- 172.21.15.40:0/4030102309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda268bff0 0x55cda2697d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.210+0000 7fe8bd182640 1 -- 172.21.15.40:0/4030102309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cda268b3d0 con 0x55cda2686e80 2024-03-18T22:53:14.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.214+0000 7fe8bd182640 1 --2- 172.21.15.40:0/4030102309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda2686e80 0x55cda2681de0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe8b40205a0 tx=0x7fe8b4009150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3820cc61021487c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.217+0000 7fe8a2ffd640 1 -- 172.21.15.40:0/4030102309 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b403a070 con 0x55cda2686e80 2024-03-18T22:53:14.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.217+0000 7fe8a2ffd640 1 -- 172.21.15.40:0/4030102309 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8b402e070 con 0x55cda2686e80 2024-03-18T22:53:14.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.217+0000 7fe8a2ffd640 1 -- 172.21.15.40:0/4030102309 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4025040 con 0x55cda2686e80 2024-03-18T22:53:14.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.218+0000 7fe8c1fce740 1 -- 172.21.15.40:0/4030102309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda2686e80 msgr2=0x55cda2681de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.218+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/4030102309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda2686e80 0x55cda2681de0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe8b40205a0 tx=0x7fe8b4009150 comp rx=0 tx=0).stop 2024-03-18T22:53:14.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.218+0000 7fe8c1fce740 1 -- 172.21.15.40:0/4030102309 shutdown_connections 2024-03-18T22:53:14.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.218+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/4030102309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda26815a0 0x55cda267cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.218+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/4030102309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda268bff0 0x55cda2697d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.218+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/4030102309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda2686e80 0x55cda2681de0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.218+0000 7fe8c1fce740 1 -- 172.21.15.40:0/4030102309 >> 172.21.15.40:0/4030102309 conn(0x55cda2681a00 msgr2=0x55cda268b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:14.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.218+0000 7fe8c1fce740 1 -- 172.21.15.40:0/4030102309 shutdown_connections 2024-03-18T22:53:14.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.218+0000 7fe8c1fce740 1 -- 172.21.15.40:0/4030102309 wait complete. 2024-03-18T22:53:14.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.218+0000 7fe8c1fce740 1 -- start start 2024-03-18T22:53:14.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.218+0000 7fe8c1fce740 1 Processor -- start 2024-03-18T22:53:14.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.219+0000 7fe8c1fce740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda26815a0 0x55cda2694b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.219+0000 7fe8c1fce740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686e80 0x55cda26950c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.219+0000 7fe8c1fce740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda268bff0 0x55cda2700fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.219+0000 7fe8c1fce740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c011410 con 0x55cda268bff0 2024-03-18T22:53:14.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.219+0000 7fe8c1fce740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c010c80 con 0x55cda26815a0 2024-03-18T22:53:14.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.219+0000 7fe8c1fce740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c00de90 con 0x55cda2686e80 2024-03-18T22:53:14.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.219+0000 7fe8c1fce740 10 client.0 _ll_register_callbacks cb 0x7fe8b400e850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:14.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.219+0000 7fe8bc981640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686e80 0x55cda26950c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.219+0000 7fe8bd182640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda26815a0 0x55cda2694b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.219+0000 7fe8bd983640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda268bff0 0x55cda2700fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.219+0000 7fe8bd983640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda268bff0 0x55cda2700fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39354/0 (socket says 172.21.15.40:39354) 2024-03-18T22:53:14.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.219+0000 7fe8bd983640 1 -- 172.21.15.40:0/3540064670 learned_addr learned my addr 172.21.15.40:0/3540064670 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:14.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.220+0000 7fe8bd182640 1 -- 172.21.15.40:0/3540064670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686e80 msgr2=0x55cda26950c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.220+0000 7fe8bd182640 1 --2- 172.21.15.40:0/3540064670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686e80 0x55cda26950c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.220+0000 7fe8bd182640 1 -- 172.21.15.40:0/3540064670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda268bff0 msgr2=0x55cda2700fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.220+0000 7fe8bd182640 1 --2- 172.21.15.40:0/3540064670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda268bff0 0x55cda2700fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.220+0000 7fe8bd182640 1 -- 172.21.15.40:0/3540064670 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cda2701520 con 0x55cda26815a0 2024-03-18T22:53:14.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.220+0000 7fe8bd983640 1 --2- 172.21.15.40:0/3540064670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda268bff0 0x55cda2700fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:53:14.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.221+0000 7fe8bd182640 1 --2- 172.21.15.40:0/3540064670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda26815a0 0x55cda2694b80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe8b400e6c0 tx=0x7fe8b400ff80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffe6a6ef801ca85b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.221+0000 7fe8a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:53:14.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.221+0000 7fe8a0ff9640 1 -- 172.21.15.40:0/3540064670 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b403a050 con 0x55cda26815a0 2024-03-18T22:53:14.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.221+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3540064670 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8b402e070 con 0x55cda26815a0 2024-03-18T22:53:14.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.222+0000 7fe8a0ff9640 1 -- 172.21.15.40:0/3540064670 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe8b4013a90 con 0x55cda26815a0 2024-03-18T22:53:14.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.222+0000 7fe8a0ff9640 1 -- 172.21.15.40:0/3540064670 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe8b4025040 con 0x55cda26815a0 2024-03-18T22:53:14.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.222+0000 7fe8c1fce740 10 client.27989 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:14.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.222+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3540064670 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cda27026a0 con 0x55cda26815a0 2024-03-18T22:53:14.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.223+0000 7fe8a0ff9640 1 -- 172.21.15.40:0/3540064670 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8b402e070 con 0x55cda26815a0 2024-03-18T22:53:14.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.223+0000 7fe8a0ff9640 1 -- 172.21.15.40:0/3540064670 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7fe8b4003690 con 0x55cda26815a0 2024-03-18T22:53:14.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.223+0000 7fe8c1fce740 10 client.27989 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:14.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.223+0000 7fe8c1fce740 10 client.27989 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:53:14.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.224+0000 7fe8c1fce740 20 client.27989 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:14.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.224+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/3540064670 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55cda2702f70 0x55cda2705390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.224+0000 7fe8c1fce740 4 client.27989 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:53:14.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.224+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3540064670 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cda25f79f0 con 0x55cda2702f70 2024-03-18T22:53:14.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.224+0000 7fe8bc981640 1 --2- 172.21.15.40:0/3540064670 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55cda2702f70 0x55cda2705390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.226+0000 7fe8bc981640 1 --2- 172.21.15.40:0/3540064670 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55cda2702f70 0x55cda2705390 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1433add248e34cb9 server_cookie=a6cadba1050997d1 in_seq=0 out_seq=0 2024-03-18T22:53:14.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.226+0000 7fe8a0ff9640 10 client.27989 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:53:14.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.226+0000 7fe8a0ff9640 1 -- 172.21.15.40:0/3540064670 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cda25f79f0 con 0x55cda2702f70 2024-03-18T22:53:14.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.226+0000 7fe8a0ff9640 10 client.27989 handle_command_reply: tid=0 2024-03-18T22:53:14.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.226+0000 7fe8c1fce740 1 client.27989 shutdown 2024-03-18T22:53:14.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.226+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3540064670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda26815a0 msgr2=0x55cda2694b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.226+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/3540064670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda26815a0 0x55cda2694b80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe8b400e6c0 tx=0x7fe8b400ff80 comp rx=0 tx=0).stop 2024-03-18T22:53:14.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.226+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3540064670 shutdown_connections 2024-03-18T22:53:14.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.226+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/3540064670 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55cda2702f70 0x55cda2705390 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.226+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/3540064670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cda268bff0 0x55cda2700fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.226+0000 7fe8a0ff9640 0 client.27989 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:53:14.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.226+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/3540064670 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cda2686e80 0x55cda26950c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.226+0000 7fe8c1fce740 1 --2- 172.21.15.40:0/3540064670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cda26815a0 0x55cda2694b80 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.226+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3540064670 >> 172.21.15.40:0/3540064670 conn(0x55cda2681a00 msgr2=0x55cda267cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:14.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.227+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3540064670 shutdown_connections 2024-03-18T22:53:14.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.227+0000 7fe8c1fce740 1 -- 172.21.15.40:0/3540064670 wait complete. 2024-03-18T22:53:14.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.227+0000 7fe8c1fce740 20 client.27989 trim_cache size 0 max 16384 2024-03-18T22:53:14.228 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:53:14.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.228+0000 7fe8bf3e6640 1 -- 172.21.15.40:0/2074920048 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe8a40a2940 msgr2=0x7fe8a40a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.228+0000 7fe8bf3e6640 1 --2- 172.21.15.40:0/2074920048 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe8a40a2940 0x7fe8a40a4e00 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4006fd0 tx=0x7fe8b40090e0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.228+0000 7fe8bf3e6640 1 -- 172.21.15.40:0/2074920048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b8109430 msgr2=0x7fe8b8119560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.228+0000 7fe8bf3e6640 1 --2- 172.21.15.40:0/2074920048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b8109430 0x7fe8b8119560 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe8a802c290 tx=0x7fe8a80043b0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.228+0000 7fe8bf3e6640 1 -- 172.21.15.40:0/2074920048 shutdown_connections 2024-03-18T22:53:14.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.228+0000 7fe8bf3e6640 1 --2- 172.21.15.40:0/2074920048 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe8a40a2940 0x7fe8a40a4e00 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.228+0000 7fe8bf3e6640 1 --2- 172.21.15.40:0/2074920048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe8b810fb60 0x7fe8b811a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.228+0000 7fe8bf3e6640 1 --2- 172.21.15.40:0/2074920048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe8b8109430 0x7fe8b8119560 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.228+0000 7fe8bf3e6640 1 --2- 172.21.15.40:0/2074920048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b8072d30 0x7fe8b8119020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.228+0000 7fe8bf3e6640 1 -- 172.21.15.40:0/2074920048 >> 172.21.15.40:0/2074920048 conn(0x7fe8b806e640 msgr2=0x7fe8b810bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:14.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.228+0000 7fe8bf3e6640 1 -- 172.21.15.40:0/2074920048 shutdown_connections 2024-03-18T22:53:14.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.228+0000 7fe8bf3e6640 1 -- 172.21.15.40:0/2074920048 wait complete. 2024-03-18T22:53:14.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:53:14.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.469+0000 7f931f1cd640 1 -- 172.21.15.40:0/525006369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f931810db40 msgr2=0x7f931810dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.469+0000 7f931f1cd640 1 --2- 172.21.15.40:0/525006369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f931810db40 0x7f931810dfc0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f9300009900 tx=0x7f930002bc40 comp rx=0 tx=0).stop 2024-03-18T22:53:14.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.470+0000 7f931f1cd640 1 -- 172.21.15.40:0/525006369 shutdown_connections 2024-03-18T22:53:14.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.470+0000 7f931f1cd640 1 --2- 172.21.15.40:0/525006369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f931810db40 0x7f931810dfc0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.470+0000 7f931f1cd640 1 --2- 172.21.15.40:0/525006369 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f931810b190 0x7f931810d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.470+0000 7f931f1cd640 1 --2- 172.21.15.40:0/525006369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9318108820 0x7f931810ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.470+0000 7f931f1cd640 1 -- 172.21.15.40:0/525006369 >> 172.21.15.40:0/525006369 conn(0x7f931806e8a0 msgr2=0x7f9318070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:14.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.472+0000 7f931f1cd640 1 -- 172.21.15.40:0/525006369 shutdown_connections 2024-03-18T22:53:14.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.474+0000 7f931f1cd640 1 -- 172.21.15.40:0/525006369 wait complete. 2024-03-18T22:53:14.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.475+0000 7f931f1cd640 1 Processor -- start 2024-03-18T22:53:14.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.475+0000 7f931f1cd640 1 -- start start 2024-03-18T22:53:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.475+0000 7f931f1cd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9318108820 0x7f9318114fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.475+0000 7f931f1cd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f931810b190 0x7f9318115510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.475+0000 7f931f1cd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f931810db40 0x7f93181b6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.475+0000 7f931f1cd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f931811c210 con 0x7f9318108820 2024-03-18T22:53:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.475+0000 7f931f1cd640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f931811c090 con 0x7f931810b190 2024-03-18T22:53:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.475+0000 7f931f1cd640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f931811c390 con 0x7f931810db40 2024-03-18T22:53:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.476+0000 7f9317fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f931810b190 0x7f9318115510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.476+0000 7f9317fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f931810b190 0x7f9318115510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52144/0 (socket says 172.21.15.40:52144) 2024-03-18T22:53:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.476+0000 7f9317fff640 1 -- 172.21.15.40:0/3918433740 learned_addr learned my addr 172.21.15.40:0/3918433740 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.476+0000 7f931cf69640 1 --2- 172.21.15.40:0/3918433740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9318108820 0x7f9318114fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.476+0000 7f931d76a640 1 --2- 172.21.15.40:0/3918433740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f931810db40 0x7f93181b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.479+0000 7f931d76a640 1 -- 172.21.15.40:0/3918433740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f931810b190 msgr2=0x7f9318115510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.479+0000 7f931d76a640 1 --2- 172.21.15.40:0/3918433740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f931810b190 0x7f9318115510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.479+0000 7f931d76a640 1 -- 172.21.15.40:0/3918433740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9318108820 msgr2=0x7f9318114fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.479+0000 7f931d76a640 1 --2- 172.21.15.40:0/3918433740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9318108820 0x7f9318114fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.479+0000 7f931d76a640 1 -- 172.21.15.40:0/3918433740 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93181b6e60 con 0x7f931810db40 2024-03-18T22:53:14.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.483+0000 7f931d76a640 1 --2- 172.21.15.40:0/3918433740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f931810db40 0x7f93181b6920 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f930002c180 tx=0x7f930002dcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.484+0000 7f9315ffb640 1 -- 172.21.15.40:0/3918433740 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9300043020 con 0x7f931810db40 2024-03-18T22:53:14.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.484+0000 7f9315ffb640 1 -- 172.21.15.40:0/3918433740 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9300004250 con 0x7f931810db40 2024-03-18T22:53:14.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.484+0000 7f9315ffb640 1 -- 172.21.15.40:0/3918433740 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9300005220 con 0x7f931810db40 2024-03-18T22:53:14.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.487+0000 7f931f1cd640 1 -- 172.21.15.40:0/3918433740 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93181b7030 con 0x7f931810db40 2024-03-18T22:53:14.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.487+0000 7f931f1cd640 1 -- 172.21.15.40:0/3918433740 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93181b7580 con 0x7f931810db40 2024-03-18T22:53:14.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.492+0000 7f9321dbb740 1 Processor -- start 2024-03-18T22:53:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.492+0000 7f9321dbb740 1 -- start start 2024-03-18T22:53:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.492+0000 7f9321dbb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf8d2f0 0x55de2cf8d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.492+0000 7f9321dbb740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cf927e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.492+0000 7f9321dbb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf92e50 0x55de2cf952e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.492+0000 7f9321dbb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55de2cef14b0 con 0x55de2cf8d2f0 2024-03-18T22:53:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.492+0000 7f9321dbb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55de2ce99d70 con 0x55de2cf92e50 2024-03-18T22:53:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.492+0000 7f9321dbb740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55de2ceed550 con 0x55de2cf8dce0 2024-03-18T22:53:14.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.493+0000 7f9317fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf8d2f0 0x55de2cf8d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.493+0000 7f9317fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf8d2f0 0x55de2cf8d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39376/0 (socket says 172.21.15.40:39376) 2024-03-18T22:53:14.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.493+0000 7f9317fff640 1 -- 172.21.15.40:0/3382762793 learned_addr learned my addr 172.21.15.40:0/3382762793 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:14.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.494+0000 7f9317fff640 1 --2- 172.21.15.40:0/3382762793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf92e50 0x55de2cf952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.496+0000 7f931cf69640 1 --2- 172.21.15.40:0/3382762793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cf927e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.503+0000 7f9317fff640 1 -- 172.21.15.40:0/3382762793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 msgr2=0x55de2cf927e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.503+0000 7f9317fff640 1 --2- 172.21.15.40:0/3382762793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cf927e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.503+0000 7f9317fff640 1 -- 172.21.15.40:0/3382762793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf92e50 msgr2=0x55de2cf952e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.503+0000 7f9317fff640 1 --2- 172.21.15.40:0/3382762793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf92e50 0x55de2cf952e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.503+0000 7f9317fff640 1 -- 172.21.15.40:0/3382762793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de2cebc210 con 0x55de2cf8d2f0 2024-03-18T22:53:14.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.503+0000 7f9317fff640 1 --2- 172.21.15.40:0/3382762793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf8d2f0 0x55de2cf8d710 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f9308038770 tx=0x7f930800a370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678197d33c288e68 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.506+0000 7f9315ffb640 1 -- 172.21.15.40:0/3918433740 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9300046020 con 0x7f931810db40 2024-03-18T22:53:14.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.507+0000 7f9315ffb640 1 --2- 172.21.15.40:0/3918433740 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f93040a29d0 0x7f93040a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.508+0000 7f9315ffb640 1 -- 172.21.15.40:0/3918433740 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f93000f0d20 con 0x7f931810db40 2024-03-18T22:53:14.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.509+0000 7f931cf69640 1 --2- 172.21.15.40:0/3918433740 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f93040a29d0 0x7f93040a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.512+0000 7f931cf69640 1 --2- 172.21.15.40:0/3918433740 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f93040a29d0 0x7f93040a4e90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x55de2cf8e0e0 tx=0x7f931000e040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.512+0000 7f930effd640 1 -- 172.21.15.40:0/3382762793 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930803e070 con 0x55de2cf8d2f0 2024-03-18T22:53:14.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.513+0000 7f930effd640 1 -- 172.21.15.40:0/3382762793 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9308038cf0 con 0x55de2cf8d2f0 2024-03-18T22:53:14.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.514+0000 7f9321dbb740 1 -- 172.21.15.40:0/3382762793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf8d2f0 msgr2=0x55de2cf8d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.514+0000 7f9321dbb740 1 --2- 172.21.15.40:0/3382762793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf8d2f0 0x55de2cf8d710 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f9308038770 tx=0x7f930800a370 comp rx=0 tx=0).stop 2024-03-18T22:53:14.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.514+0000 7f9321dbb740 1 -- 172.21.15.40:0/3382762793 shutdown_connections 2024-03-18T22:53:14.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.514+0000 7f9321dbb740 1 --2- 172.21.15.40:0/3382762793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf92e50 0x55de2cf952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.514+0000 7f9321dbb740 1 --2- 172.21.15.40:0/3382762793 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cf927e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.514+0000 7f9321dbb740 1 --2- 172.21.15.40:0/3382762793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf8d2f0 0x55de2cf8d710 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.514+0000 7f9321dbb740 1 -- 172.21.15.40:0/3382762793 >> 172.21.15.40:0/3382762793 conn(0x55de2cf88860 msgr2=0x55de2cf88c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:14.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.515+0000 7f9321dbb740 1 -- 172.21.15.40:0/3382762793 shutdown_connections 2024-03-18T22:53:14.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.515+0000 7f9321dbb740 1 -- 172.21.15.40:0/3382762793 wait complete. 2024-03-18T22:53:14.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.515+0000 7f9321dbb740 1 -- start start 2024-03-18T22:53:14.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.516+0000 7f9321dbb740 1 Processor -- start 2024-03-18T22:53:14.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.516+0000 7f9321dbb740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cf97790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.516+0000 7f9321dbb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf92e50 0x55de2cf97cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.516+0000 7f9321dbb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf9cd00 0x55de2cf9f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.516+0000 7f9321dbb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55de2cf95e50 con 0x55de2cf92e50 2024-03-18T22:53:14.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.516+0000 7f9321dbb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55de2cead310 con 0x55de2cf9cd00 2024-03-18T22:53:14.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.516+0000 7f9321dbb740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55de2cf95fd0 con 0x55de2cf8dce0 2024-03-18T22:53:14.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.516+0000 7f9321dbb740 10 client.0 _ll_register_callbacks cb 0x55de2cea12d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:14.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.517+0000 7f931cf69640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cf97790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.517+0000 7f931cf69640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cf97790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41784/0 (socket says 172.21.15.40:41784) 2024-03-18T22:53:14.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.517+0000 7f931cf69640 1 -- 172.21.15.40:0/148805407 learned_addr learned my addr 172.21.15.40:0/148805407 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:14.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.517+0000 7f931d76a640 1 --2- 172.21.15.40:0/148805407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf9cd00 0x55de2cf9f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.517+0000 7f931cf69640 1 -- 172.21.15.40:0/148805407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf9cd00 msgr2=0x55de2cf9f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.517+0000 7f931cf69640 1 --2- 172.21.15.40:0/148805407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf9cd00 0x55de2cf9f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.517+0000 7f931cf69640 1 -- 172.21.15.40:0/148805407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf92e50 msgr2=0x55de2cf97cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.517+0000 7f931cf69640 1 --2- 172.21.15.40:0/148805407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf92e50 0x55de2cf97cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.517+0000 7f931cf69640 1 -- 172.21.15.40:0/148805407 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de2cc053f0 con 0x55de2cf8dce0 2024-03-18T22:53:14.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.517+0000 7f931d76a640 1 --2- 172.21.15.40:0/148805407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf9cd00 0x55de2cf9f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:53:14.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.520+0000 7f931cf69640 1 --2- 172.21.15.40:0/148805407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cf97790 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9310019ee0 tx=0x7f9310004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6de50faffb691e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.521+0000 7f92f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:53:14.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.521+0000 7f92f7fff640 1 -- 172.21.15.40:0/148805407 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9310027060 con 0x55de2cf8dce0 2024-03-18T22:53:14.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.521+0000 7f9321dbb740 1 -- 172.21.15.40:0/148805407 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55de2cf4f3e0 con 0x55de2cf8dce0 2024-03-18T22:53:14.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.522+0000 7f92f7fff640 1 -- 172.21.15.40:0/148805407 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f93100192e0 con 0x55de2cf8dce0 2024-03-18T22:53:14.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.522+0000 7f92f7fff640 1 -- 172.21.15.40:0/148805407 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9310016e20 con 0x55de2cf8dce0 2024-03-18T22:53:14.526 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:14 smithi040 ceph-mon[62643]: pgmap v668: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 4.16k op/s 2024-03-18T22:53:14.526 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:14 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3426805926' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:53:14.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.523+0000 7f9321dbb740 10 client.27985 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:14.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.523+0000 7f9321dbb740 1 -- 172.21.15.40:0/148805407 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55de2cccf7d0 con 0x55de2cf8dce0 2024-03-18T22:53:14.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.526+0000 7f92f7fff640 1 -- 172.21.15.40:0/148805407 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55de2cf4f3e0 con 0x55de2cf8dce0 2024-03-18T22:53:14.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.526+0000 7f92f7fff640 1 -- 172.21.15.40:0/148805407 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f9310016070 con 0x55de2cf8dce0 2024-03-18T22:53:14.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.526+0000 7f9321dbb740 10 client.27985 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:14.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.526+0000 7f9321dbb740 10 client.27985 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:53:14.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.526+0000 7f9321dbb740 20 client.27985 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:14.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.526+0000 7f9321dbb740 1 --2- 172.21.15.40:0/148805407 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55de2cfa3f40 0x55de2cfa63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.526+0000 7f9321dbb740 4 client.27985 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:53:14.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.526+0000 7f9321dbb740 1 -- 172.21.15.40:0/148805407 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de2cefe9f0 con 0x55de2cfa3f40 2024-03-18T22:53:14.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.534+0000 7f9317fff640 1 --2- 172.21.15.40:0/148805407 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55de2cfa3f40 0x55de2cfa63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.543+0000 7f9317fff640 1 --2- 172.21.15.40:0/148805407 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55de2cfa3f40 0x55de2cfa63e0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b6b15ee192d59a45 server_cookie=7fc0a21a00a3f4cd in_seq=0 out_seq=0 2024-03-18T22:53:14.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.543+0000 7f92f7fff640 10 client.27985 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:53:14.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.545+0000 7f92f7fff640 1 -- 172.21.15.40:0/148805407 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55de2cefe9f0 con 0x55de2cfa3f40 2024-03-18T22:53:14.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.545+0000 7f92f7fff640 10 client.27985 handle_command_reply: tid=0 2024-03-18T22:53:14.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.547+0000 7f9321dbb740 1 client.27985 shutdown 2024-03-18T22:53:14.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.548+0000 7f9321dbb740 1 -- 172.21.15.40:0/148805407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 msgr2=0x55de2cf97790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.548+0000 7f9321dbb740 1 --2- 172.21.15.40:0/148805407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cf97790 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9310019ee0 tx=0x7f9310004a40 comp rx=0 tx=0).stop 2024-03-18T22:53:14.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.553+0000 7f9321dbb740 1 -- 172.21.15.40:0/148805407 shutdown_connections 2024-03-18T22:53:14.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.553+0000 7f9321dbb740 1 --2- 172.21.15.40:0/148805407 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55de2cfa3f40 0x55de2cfa63e0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.553+0000 7f9321dbb740 1 --2- 172.21.15.40:0/148805407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf9cd00 0x55de2cf9f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.553+0000 7f9321dbb740 1 --2- 172.21.15.40:0/148805407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf92e50 0x55de2cf97cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.553+0000 7f9321dbb740 1 --2- 172.21.15.40:0/148805407 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cf97790 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.553+0000 7f9321dbb740 1 -- 172.21.15.40:0/148805407 >> 172.21.15.40:0/148805407 conn(0x55de2cf88860 msgr2=0x55de2cf8d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:14.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.553+0000 7f92f7fff640 0 client.27985 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:53:14.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.553+0000 7f9321dbb740 1 -- 172.21.15.40:0/148805407 shutdown_connections 2024-03-18T22:53:14.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.553+0000 7f9321dbb740 1 -- 172.21.15.40:0/148805407 wait complete. 2024-03-18T22:53:14.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.553+0000 7f9321dbb740 20 client.27985 trim_cache size 0 max 16384 2024-03-18T22:53:14.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.585+0000 7f9321dbb740 1 Processor -- start 2024-03-18T22:53:14.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.585+0000 7f9321dbb740 1 -- start start 2024-03-18T22:53:14.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.586+0000 7f9321dbb740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cf8bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.586+0000 7f9321dbb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf92e50 0x55de2cf8c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.586+0000 7f9321dbb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf9cd00 0x55de2cf973a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.586+0000 7f9321dbb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f93181abcc0 con 0x55de2cf9cd00 2024-03-18T22:53:14.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.586+0000 7f9321dbb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f93181ad8f0 con 0x55de2cf92e50 2024-03-18T22:53:14.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.586+0000 7f9321dbb740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f93181ac390 con 0x55de2cf8dce0 2024-03-18T22:53:14.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.589+0000 7f9317fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf92e50 0x55de2cf8c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.589+0000 7f9317fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf92e50 0x55de2cf8c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52176/0 (socket says 172.21.15.40:52176) 2024-03-18T22:53:14.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.589+0000 7f9317fff640 1 -- 172.21.15.40:0/2730978784 learned_addr learned my addr 172.21.15.40:0/2730978784 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:14.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.590+0000 7f931cf69640 1 --2- 172.21.15.40:0/2730978784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cf8bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.592+0000 7f931d76a640 1 --2- 172.21.15.40:0/2730978784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf9cd00 0x55de2cf973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.592+0000 7f9317fff640 1 -- 172.21.15.40:0/2730978784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 msgr2=0x55de2cf8bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.592+0000 7f9317fff640 1 --2- 172.21.15.40:0/2730978784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cf8bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.592+0000 7f9317fff640 1 -- 172.21.15.40:0/2730978784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf9cd00 msgr2=0x55de2cf973a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.592+0000 7f9317fff640 1 --2- 172.21.15.40:0/2730978784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf9cd00 0x55de2cf973a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.592+0000 7f9317fff640 1 -- 172.21.15.40:0/2730978784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de2cf5e880 con 0x55de2cf92e50 2024-03-18T22:53:14.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.593+0000 7f9317fff640 1 --2- 172.21.15.40:0/2730978784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf92e50 0x55de2cf8c000 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9308038770 tx=0x7f930800ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40c327a015b19d47 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.594+0000 7f930e7fc640 1 -- 172.21.15.40:0/2730978784 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930803e040 con 0x55de2cf92e50 2024-03-18T22:53:14.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.595+0000 7f930e7fc640 1 -- 172.21.15.40:0/2730978784 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f930804b030 con 0x55de2cf92e50 2024-03-18T22:53:14.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.596+0000 7f9321dbb740 1 -- 172.21.15.40:0/2730978784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf92e50 msgr2=0x55de2cf8c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.596+0000 7f9321dbb740 1 --2- 172.21.15.40:0/2730978784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf92e50 0x55de2cf8c000 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9308038770 tx=0x7f930800ac20 comp rx=0 tx=0).stop 2024-03-18T22:53:14.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.597+0000 7f9321dbb740 1 -- 172.21.15.40:0/2730978784 shutdown_connections 2024-03-18T22:53:14.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.597+0000 7f9321dbb740 1 --2- 172.21.15.40:0/2730978784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf9cd00 0x55de2cf973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.597+0000 7f9321dbb740 1 --2- 172.21.15.40:0/2730978784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf92e50 0x55de2cf8c000 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.597+0000 7f9321dbb740 1 --2- 172.21.15.40:0/2730978784 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cf8bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.598+0000 7f9321dbb740 1 -- 172.21.15.40:0/2730978784 >> 172.21.15.40:0/2730978784 conn(0x55de2cf88860 msgr2=0x55de2cf83a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:14.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.603+0000 7f9321dbb740 1 -- 172.21.15.40:0/2730978784 shutdown_connections 2024-03-18T22:53:14.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.604+0000 7f9321dbb740 1 -- 172.21.15.40:0/2730978784 wait complete. 2024-03-18T22:53:14.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.604+0000 7f9321dbb740 1 -- start start 2024-03-18T22:53:14.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.604+0000 7f9321dbb740 1 Processor -- start 2024-03-18T22:53:14.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.605+0000 7f9321dbb740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cfb4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.605+0000 7f9321dbb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf92e50 0x55de2cfb52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.605+0000 7f9321dbb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf9cd00 0x55de2cfb6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.605+0000 7f9321dbb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f93181ad220 con 0x55de2cf92e50 2024-03-18T22:53:14.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.605+0000 7f9321dbb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f93181aca90 con 0x55de2cf9cd00 2024-03-18T22:53:14.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.605+0000 7f9321dbb740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f93181a9ca0 con 0x55de2cf8dce0 2024-03-18T22:53:14.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.605+0000 7f9321dbb740 10 client.0 _ll_register_callbacks cb 0x7f9310009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:14.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.606+0000 7f931cf69640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cfb4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.607+0000 7f931d76a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf9cd00 0x55de2cfb6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.607+0000 7f9317fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf92e50 0x55de2cfb52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.607+0000 7f9317fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf92e50 0x55de2cfb52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39416/0 (socket says 172.21.15.40:39416) 2024-03-18T22:53:14.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.607+0000 7f9317fff640 1 -- 172.21.15.40:0/3121484626 learned_addr learned my addr 172.21.15.40:0/3121484626 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:14.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.607+0000 7f9317fff640 1 -- 172.21.15.40:0/3121484626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 msgr2=0x55de2cfb4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.607+0000 7f9317fff640 1 --2- 172.21.15.40:0/3121484626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cfb4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.607+0000 7f9317fff640 1 -- 172.21.15.40:0/3121484626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf9cd00 msgr2=0x55de2cfb6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.608+0000 7f9317fff640 1 --2- 172.21.15.40:0/3121484626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf9cd00 0x55de2cfb6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.608+0000 7f9317fff640 1 -- 172.21.15.40:0/3121484626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de2cfb6840 con 0x55de2cf92e50 2024-03-18T22:53:14.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.608+0000 7f9317fff640 1 --2- 172.21.15.40:0/3121484626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf92e50 0x55de2cfb52b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f9308038740 tx=0x7f930800a820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56319ccbcd9515fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.608+0000 7f92f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:53:14.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.608+0000 7f92f7fff640 1 -- 172.21.15.40:0/3121484626 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9308056070 con 0x55de2cf92e50 2024-03-18T22:53:14.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.609+0000 7f92f7fff640 1 -- 172.21.15.40:0/3121484626 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9308038cf0 con 0x55de2cf92e50 2024-03-18T22:53:14.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.609+0000 7f9321dbb740 1 -- 172.21.15.40:0/3121484626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f930804b030 con 0x55de2cf92e50 2024-03-18T22:53:14.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.609+0000 7f931cf69640 1 --2- 172.21.15.40:0/3121484626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cfb4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:53:14.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.609+0000 7f92f7fff640 1 -- 172.21.15.40:0/3121484626 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9308060050 con 0x55de2cf92e50 2024-03-18T22:53:14.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.609+0000 7f9321dbb740 10 client.18168 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:14.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.610+0000 7f9321dbb740 1 -- 172.21.15.40:0/3121484626 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55de2d009dd0 con 0x55de2cf92e50 2024-03-18T22:53:14.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.610+0000 7f92f7fff640 1 -- 172.21.15.40:0/3121484626 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f930804b030 con 0x55de2cf92e50 2024-03-18T22:53:14.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.610+0000 7f92f7fff640 1 -- 172.21.15.40:0/3121484626 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f9308047270 con 0x55de2cf92e50 2024-03-18T22:53:14.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.611+0000 7f9321dbb740 10 client.18168 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:14.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.611+0000 7f9321dbb740 10 client.18168 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:53:14.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.611+0000 7f9321dbb740 20 client.18168 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:14.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.611+0000 7f9321dbb740 1 --2- 172.21.15.40:0/3121484626 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55de2d00a430 0x55de2d00c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.611+0000 7f9321dbb740 4 client.18168 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:53:14.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.611+0000 7f9321dbb740 1 -- 172.21.15.40:0/3121484626 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55de2cef8d40 con 0x55de2d00a430 2024-03-18T22:53:14.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.615+0000 7f931cf69640 1 --2- 172.21.15.40:0/3121484626 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55de2d00a430 0x55de2d00c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.634+0000 7f931cf69640 1 --2- 172.21.15.40:0/3121484626 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55de2d00a430 0x55de2d00c850 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=52fd4072153052ad server_cookie=2a1bdba86d7566b3 in_seq=0 out_seq=0 2024-03-18T22:53:14.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.636+0000 7f92f7fff640 10 client.18168 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:53:14.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.642+0000 7f92f7fff640 1 -- 172.21.15.40:0/3121484626 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55de2cef8d40 con 0x55de2d00a430 2024-03-18T22:53:14.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.642+0000 7f92f7fff640 10 client.18168 handle_command_reply: tid=0 2024-03-18T22:53:14.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.642+0000 7f9321dbb740 1 client.18168 shutdown 2024-03-18T22:53:14.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.642+0000 7f9321dbb740 1 -- 172.21.15.40:0/3121484626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf92e50 msgr2=0x55de2cfb52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.642+0000 7f9321dbb740 1 --2- 172.21.15.40:0/3121484626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf92e50 0x55de2cfb52b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f9308038740 tx=0x7f930800a820 comp rx=0 tx=0).stop 2024-03-18T22:53:14.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.642+0000 7f9321dbb740 1 -- 172.21.15.40:0/3121484626 shutdown_connections 2024-03-18T22:53:14.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.642+0000 7f9321dbb740 1 --2- 172.21.15.40:0/3121484626 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55de2d00a430 0x55de2d00c850 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.642+0000 7f9321dbb740 1 --2- 172.21.15.40:0/3121484626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55de2cf9cd00 0x55de2cfb6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.642+0000 7f9321dbb740 1 --2- 172.21.15.40:0/3121484626 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55de2cf92e50 0x55de2cfb52b0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.642+0000 7f9321dbb740 1 --2- 172.21.15.40:0/3121484626 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55de2cf8dce0 0x55de2cfb4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.642+0000 7f9321dbb740 1 -- 172.21.15.40:0/3121484626 >> 172.21.15.40:0/3121484626 conn(0x55de2cf88860 msgr2=0x55de2cf8bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:14.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.642+0000 7f92f7fff640 0 client.18168 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:53:14.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.642+0000 7f9321dbb740 1 -- 172.21.15.40:0/3121484626 shutdown_connections 2024-03-18T22:53:14.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.642+0000 7f9321dbb740 1 -- 172.21.15.40:0/3121484626 wait complete. 2024-03-18T22:53:14.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.642+0000 7f9321dbb740 20 client.18168 trim_cache size 0 max 16384 2024-03-18T22:53:14.644 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:53:14.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.646+0000 7f931f1cd640 1 -- 172.21.15.40:0/3918433740 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f93040a29d0 msgr2=0x7f93040a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.646+0000 7f931f1cd640 1 --2- 172.21.15.40:0/3918433740 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f93040a29d0 0x7f93040a4e90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x55de2cf8e0e0 tx=0x7f931000e040 comp rx=0 tx=0).stop 2024-03-18T22:53:14.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.646+0000 7f931f1cd640 1 -- 172.21.15.40:0/3918433740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f931810db40 msgr2=0x7f93181b6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.646+0000 7f931f1cd640 1 --2- 172.21.15.40:0/3918433740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f931810db40 0x7f93181b6920 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f930002c180 tx=0x7f930002dcf0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.648+0000 7f931f1cd640 1 -- 172.21.15.40:0/3918433740 shutdown_connections 2024-03-18T22:53:14.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.648+0000 7f931f1cd640 1 --2- 172.21.15.40:0/3918433740 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f93040a29d0 0x7f93040a4e90 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.648+0000 7f931f1cd640 1 --2- 172.21.15.40:0/3918433740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f931810db40 0x7f93181b6920 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.648+0000 7f931f1cd640 1 --2- 172.21.15.40:0/3918433740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f931810b190 0x7f9318115510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.648+0000 7f931f1cd640 1 --2- 172.21.15.40:0/3918433740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9318108820 0x7f9318114fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.648+0000 7f931f1cd640 1 -- 172.21.15.40:0/3918433740 >> 172.21.15.40:0/3918433740 conn(0x7f931806e8a0 msgr2=0x7f931810c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:14.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.648+0000 7f931f1cd640 1 -- 172.21.15.40:0/3918433740 shutdown_connections 2024-03-18T22:53:14.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.649+0000 7f931f1cd640 1 -- 172.21.15.40:0/3918433740 wait complete. 2024-03-18T22:53:14.679 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:53:14.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:14 smithi113 ceph-mon[66775]: pgmap v668: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 4.16k op/s 2024-03-18T22:53:14.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:14 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3426805926' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:53:14.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:14 smithi103 ceph-mon[66377]: pgmap v668: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 4.16k op/s 2024-03-18T22:53:14.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:14 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3426805926' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:53:14.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.935+0000 7f6ee311e640 1 -- 172.21.15.40:0/3790532940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6edc108e20 msgr2=0x7f6edc109280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.935+0000 7f6ee311e640 1 --2- 172.21.15.40:0/3790532940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6edc108e20 0x7f6edc109280 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009be0 tx=0x7f6ed402bf30 comp rx=0 tx=0).stop 2024-03-18T22:53:14.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.935+0000 7f6ee311e640 1 -- 172.21.15.40:0/3790532940 shutdown_connections 2024-03-18T22:53:14.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.935+0000 7f6ee311e640 1 --2- 172.21.15.40:0/3790532940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6edc1097c0 0x7f6edc10e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.935+0000 7f6ee311e640 1 --2- 172.21.15.40:0/3790532940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6edc108e20 0x7f6edc109280 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.935+0000 7f6ee311e640 1 --2- 172.21.15.40:0/3790532940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6edc072b70 0x7f6edc072f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.935+0000 7f6ee311e640 1 -- 172.21.15.40:0/3790532940 >> 172.21.15.40:0/3790532940 conn(0x7f6edc06e640 msgr2=0x7f6edc070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:14.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.944+0000 7f6ee311e640 1 -- 172.21.15.40:0/3790532940 shutdown_connections 2024-03-18T22:53:14.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.944+0000 7f6ee311e640 1 -- 172.21.15.40:0/3790532940 wait complete. 2024-03-18T22:53:14.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.945+0000 7f6ee311e640 1 Processor -- start 2024-03-18T22:53:14.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.945+0000 7f6ee311e640 1 -- start start 2024-03-18T22:53:14.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.945+0000 7f6ee311e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6edc072b70 0x7f6edc114f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.945+0000 7f6ee311e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6edc108e20 0x7f6edc1154d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.945+0000 7f6ee311e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6edc1097c0 0x7f6edc1b66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.945+0000 7f6ee311e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc11bfd0 con 0x7f6edc072b70 2024-03-18T22:53:14.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.945+0000 7f6ee311e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc11be50 con 0x7f6edc1097c0 2024-03-18T22:53:14.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.945+0000 7f6ee311e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc11c150 con 0x7f6edc108e20 2024-03-18T22:53:14.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.946+0000 7f6ee291d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6edc1097c0 0x7f6edc1b66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.946+0000 7f6ee291d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6edc1097c0 0x7f6edc1b66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52196/0 (socket says 172.21.15.40:52196) 2024-03-18T22:53:14.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.946+0000 7f6ee291d640 1 -- 172.21.15.40:0/183624291 learned_addr learned my addr 172.21.15.40:0/183624291 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:14.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.946+0000 7f6ee191b640 1 --2- 172.21.15.40:0/183624291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6edc108e20 0x7f6edc1154d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.947+0000 7f6ee211c640 1 --2- 172.21.15.40:0/183624291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6edc072b70 0x7f6edc114f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.949+0000 7f6ee291d640 1 -- 172.21.15.40:0/183624291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6edc108e20 msgr2=0x7f6edc1154d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.949+0000 7f6ee291d640 1 --2- 172.21.15.40:0/183624291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6edc108e20 0x7f6edc1154d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.949+0000 7f6ee291d640 1 -- 172.21.15.40:0/183624291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6edc072b70 msgr2=0x7f6edc114f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.949+0000 7f6ee291d640 1 --2- 172.21.15.40:0/183624291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6edc072b70 0x7f6edc114f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.949+0000 7f6ee291d640 1 -- 172.21.15.40:0/183624291 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6edc1b6c20 con 0x7f6edc1097c0 2024-03-18T22:53:14.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.949+0000 7f6ee291d640 1 --2- 172.21.15.40:0/183624291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6edc1097c0 0x7f6edc1b66e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6ed800d750 tx=0x7f6ed800dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.950+0000 7f6ed37fe640 1 -- 172.21.15.40:0/183624291 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed8017070 con 0x7f6edc1097c0 2024-03-18T22:53:14.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.950+0000 7f6ee311e640 1 -- 172.21.15.40:0/183624291 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6edc1b6e50 con 0x7f6edc1097c0 2024-03-18T22:53:14.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.950+0000 7f6ee311e640 1 -- 172.21.15.40:0/183624291 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6edc1b7340 con 0x7f6edc1097c0 2024-03-18T22:53:14.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.953+0000 7f6ee6f68740 1 Processor -- start 2024-03-18T22:53:14.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.954+0000 7f6ed37fe640 1 -- 172.21.15.40:0/183624291 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ed80054d0 con 0x7f6edc1097c0 2024-03-18T22:53:14.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.954+0000 7f6ed37fe640 1 -- 172.21.15.40:0/183624291 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed8004cb0 con 0x7f6edc1097c0 2024-03-18T22:53:14.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.956+0000 7f6ee6f68740 1 -- start start 2024-03-18T22:53:14.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.956+0000 7f6ee6f68740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7477564d0 0x55e7477568f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.956+0000 7f6ee6f68740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e747756ec0 0x55e74775b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.956+0000 7f6ee6f68740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e74775c030 0x55e74775e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.956+0000 7f6ee6f68740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e7476ba4b0 con 0x55e74775c030 2024-03-18T22:53:14.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.956+0000 7f6ee6f68740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e747662d70 con 0x55e7477564d0 2024-03-18T22:53:14.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.956+0000 7f6ee6f68740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e7476b6550 con 0x55e747756ec0 2024-03-18T22:53:14.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.958+0000 7f6ee211c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e747756ec0 0x55e74775b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.958+0000 7f6ee191b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7477564d0 0x55e7477568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.958+0000 7f6ee191b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7477564d0 0x55e7477568f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52208/0 (socket says 172.21.15.40:52208) 2024-03-18T22:53:14.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.958+0000 7f6ee191b640 1 -- 172.21.15.40:0/1747943621 learned_addr learned my addr 172.21.15.40:0/1747943621 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:14.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.958+0000 7f6ee191b640 1 --2- 172.21.15.40:0/1747943621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e74775c030 0x55e74775e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.959+0000 7f6ed37fe640 1 -- 172.21.15.40:0/183624291 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6ed8009bd0 con 0x7f6edc1097c0 2024-03-18T22:53:14.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.959+0000 7f6ee191b640 1 -- 172.21.15.40:0/1747943621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e747756ec0 msgr2=0x55e74775b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.959+0000 7f6ee191b640 1 --2- 172.21.15.40:0/1747943621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e747756ec0 0x55e74775b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.959+0000 7f6ee191b640 1 -- 172.21.15.40:0/1747943621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e74775c030 msgr2=0x55e74775e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.959+0000 7f6ee191b640 1 --2- 172.21.15.40:0/1747943621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e74775c030 0x55e74775e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.959+0000 7f6ee191b640 1 -- 172.21.15.40:0/1747943621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e747685210 con 0x55e7477564d0 2024-03-18T22:53:14.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.959+0000 7f6ee191b640 1 --2- 172.21.15.40:0/1747943621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7477564d0 0x55e7477568f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009810 tx=0x7f6ed40333e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98438e28954838ef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.960+0000 7f6ed0bfe640 1 -- 172.21.15.40:0/1747943621 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4048070 con 0x55e7477564d0 2024-03-18T22:53:14.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.960+0000 7f6ed0bfe640 1 -- 172.21.15.40:0/1747943621 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ed4003e20 con 0x55e7477564d0 2024-03-18T22:53:14.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.961+0000 7f6ee6f68740 1 -- 172.21.15.40:0/1747943621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7477564d0 msgr2=0x55e7477568f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.961+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/1747943621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7477564d0 0x55e7477568f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009810 tx=0x7f6ed40333e0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.961+0000 7f6ed37fe640 1 --2- 172.21.15.40:0/183624291 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ec80a29d0 0x7f6ec80a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.962+0000 7f6ed37fe640 1 -- 172.21.15.40:0/183624291 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6ed80ce660 con 0x7f6edc1097c0 2024-03-18T22:53:14.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.963+0000 7f6ee191b640 1 --2- 172.21.15.40:0/183624291 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ec80a29d0 0x7f6ec80a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.965+0000 7f6ee191b640 1 --2- 172.21.15.40:0/183624291 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ec80a29d0 0x7f6ec80a4e90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4004fd0 tx=0x7f6ed4004b80 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.974+0000 7f6ee6f68740 1 -- 172.21.15.40:0/1747943621 shutdown_connections 2024-03-18T22:53:14.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.974+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/1747943621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e74775c030 0x55e74775e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.974+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/1747943621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e747756ec0 0x55e74775b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.974+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/1747943621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7477564d0 0x55e7477568f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.974+0000 7f6ee6f68740 1 -- 172.21.15.40:0/1747943621 >> 172.21.15.40:0/1747943621 conn(0x55e747751a40 msgr2=0x55e747751e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:14.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.980+0000 7f6ee6f68740 1 -- 172.21.15.40:0/1747943621 shutdown_connections 2024-03-18T22:53:14.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.980+0000 7f6ee6f68740 1 -- 172.21.15.40:0/1747943621 wait complete. 2024-03-18T22:53:14.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.981+0000 7f6ee6f68740 1 -- start start 2024-03-18T22:53:14.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.981+0000 7f6ee6f68740 1 Processor -- start 2024-03-18T22:53:14.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.981+0000 7f6ee6f68740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7477564d0 0x55e747760a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.981+0000 7f6ee6f68740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e747756ec0 0x55e747760f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.981+0000 7f6ee6f68740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e74775c030 0x55e747767fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.981+0000 7f6ee6f68740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e74775efd0 con 0x55e7477564d0 2024-03-18T22:53:14.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.981+0000 7f6ee6f68740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e747676310 con 0x55e747756ec0 2024-03-18T22:53:14.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.981+0000 7f6ee6f68740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e74775f150 con 0x55e74775c030 2024-03-18T22:53:14.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.981+0000 7f6ee6f68740 10 client.0 _ll_register_callbacks cb 0x55e74766a2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:14.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.981+0000 7f6ee191b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e747756ec0 0x55e747760f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.982+0000 7f6ee191b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e747756ec0 0x55e747760f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52212/0 (socket says 172.21.15.40:52212) 2024-03-18T22:53:14.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.982+0000 7f6ee191b640 1 -- 172.21.15.40:0/1412275621 learned_addr learned my addr 172.21.15.40:0/1412275621 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:14.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.982+0000 7f6ee291d640 1 --2- 172.21.15.40:0/1412275621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e74775c030 0x55e747767fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.982+0000 7f6ee211c640 1 --2- 172.21.15.40:0/1412275621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7477564d0 0x55e747760a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.982+0000 7f6ee191b640 1 -- 172.21.15.40:0/1412275621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e74775c030 msgr2=0x55e747767fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.982+0000 7f6ee191b640 1 --2- 172.21.15.40:0/1412275621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e74775c030 0x55e747767fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.982+0000 7f6ee191b640 1 -- 172.21.15.40:0/1412275621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7477564d0 msgr2=0x55e747760a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:14.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.982+0000 7f6ee191b640 1 --2- 172.21.15.40:0/1412275621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7477564d0 0x55e747760a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.982+0000 7f6ee191b640 1 -- 172.21.15.40:0/1412275621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7473ce3f0 con 0x55e747756ec0 2024-03-18T22:53:14.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.983+0000 7f6ee191b640 1 --2- 172.21.15.40:0/1412275621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e747756ec0 0x55e747760f80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4036040 tx=0x7f6ed40333e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=423778e306e66523 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:14.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.983+0000 7f6ec5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:53:14.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.983+0000 7f6ec5ffb640 1 -- 172.21.15.40:0/1412275621 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4056040 con 0x55e747756ec0 2024-03-18T22:53:14.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.984+0000 7f6ee6f68740 1 -- 172.21.15.40:0/1412275621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ed4003e20 con 0x55e747756ec0 2024-03-18T22:53:14.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.984+0000 7f6ec5ffb640 1 -- 172.21.15.40:0/1412275621 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6ed4034630 con 0x55e747756ec0 2024-03-18T22:53:14.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.984+0000 7f6ec5ffb640 1 -- 172.21.15.40:0/1412275621 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f6ed4068050 con 0x55e747756ec0 2024-03-18T22:53:14.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.987+0000 7f6ee6f68740 10 client.28013 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:14.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.987+0000 7f6ee6f68740 1 -- 172.21.15.40:0/1412275621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e7474987d0 con 0x55e747756ec0 2024-03-18T22:53:14.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.987+0000 7f6ec5ffb640 1 -- 172.21.15.40:0/1412275621 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ed4003e20 con 0x55e747756ec0 2024-03-18T22:53:14.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.988+0000 7f6ec5ffb640 1 -- 172.21.15.40:0/1412275621 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f6ed4005040 con 0x55e747756ec0 2024-03-18T22:53:14.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.989+0000 7f6ee6f68740 10 client.28013 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:14.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.989+0000 7f6ee6f68740 10 client.28013 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:53:14.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.989+0000 7f6ee6f68740 20 client.28013 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:14.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.989+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/1412275621 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55e74776c9a0 0x55e74776ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:14.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.989+0000 7f6ee6f68740 4 client.28013 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:53:14.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.989+0000 7f6ee6f68740 1 -- 172.21.15.40:0/1412275621 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7477183e0 con 0x55e74776c9a0 2024-03-18T22:53:14.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.989+0000 7f6ee211c640 1 --2- 172.21.15.40:0/1412275621 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55e74776c9a0 0x55e74776ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:14.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.990+0000 7f6ee211c640 1 --2- 172.21.15.40:0/1412275621 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55e74776c9a0 0x55e74776ee20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=572a733e6b3fab93 server_cookie=4b632857084f8213 in_seq=0 out_seq=0 2024-03-18T22:53:14.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.990+0000 7f6ec5ffb640 10 client.28013 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:53:14.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.991+0000 7f6ec5ffb640 1 -- 172.21.15.40:0/1412275621 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e7477183e0 con 0x55e74776c9a0 2024-03-18T22:53:14.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.991+0000 7f6ec5ffb640 10 client.28013 handle_command_reply: tid=0 2024-03-18T22:53:14.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.991+0000 7f6ee6f68740 1 client.28013 shutdown 2024-03-18T22:53:14.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.992+0000 7f6ee6f68740 1 -- 172.21.15.40:0/1412275621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e747756ec0 msgr2=0x55e747760f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:14.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.992+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/1412275621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e747756ec0 0x55e747760f80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4036040 tx=0x7f6ed40333e0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.992+0000 7f6ee6f68740 1 -- 172.21.15.40:0/1412275621 shutdown_connections 2024-03-18T22:53:14.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.992+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/1412275621 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55e74776c9a0 0x55e74776ee20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.992+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/1412275621 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e74775c030 0x55e747767fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.992+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/1412275621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e747756ec0 0x55e747760f80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.992+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/1412275621 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e7477564d0 0x55e747760a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:14.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.992+0000 7f6ee6f68740 1 -- 172.21.15.40:0/1412275621 >> 172.21.15.40:0/1412275621 conn(0x55e747751a40 msgr2=0x55e747750d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:14.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.992+0000 7f6ec5ffb640 0 client.28013 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:53:14.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.992+0000 7f6ee6f68740 1 -- 172.21.15.40:0/1412275621 shutdown_connections 2024-03-18T22:53:14.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.992+0000 7f6ee6f68740 1 -- 172.21.15.40:0/1412275621 wait complete. 2024-03-18T22:53:14.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:14.993+0000 7f6ee6f68740 20 client.28013 trim_cache size 0 max 16384 2024-03-18T22:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.044+0000 7f6ee6f68740 1 Processor -- start 2024-03-18T22:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.044+0000 7f6ee6f68740 1 -- start start 2024-03-18T22:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.044+0000 7f6ee6f68740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7477564d0 0x55e747760a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.044+0000 7f6ee6f68740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e747756ec0 0x55e747767f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.044+0000 7f6ee6f68740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e74775c030 0x55e747760f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.044+0000 7f6ee6f68740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec000fd90 con 0x55e747756ec0 2024-03-18T22:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.044+0000 7f6ee6f68740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec00119c0 con 0x55e7477564d0 2024-03-18T22:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.044+0000 7f6ee6f68740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec0010460 con 0x55e74775c030 2024-03-18T22:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.044+0000 7f6ee191b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e747756ec0 0x55e747767f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.045+0000 7f6ee191b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e747756ec0 0x55e747767f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39458/0 (socket says 172.21.15.40:39458) 2024-03-18T22:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.045+0000 7f6ee191b640 1 -- 172.21.15.40:0/740148247 learned_addr learned my addr 172.21.15.40:0/740148247 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.045+0000 7f6ee191b640 1 -- 172.21.15.40:0/740148247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e74775c030 msgr2=0x55e747760f80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:53:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.045+0000 7f6ee191b640 1 --2- 172.21.15.40:0/740148247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e74775c030 0x55e747760f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.045+0000 7f6ee191b640 1 -- 172.21.15.40:0/740148247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7477564d0 msgr2=0x55e747760a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:53:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.045+0000 7f6ee191b640 1 --2- 172.21.15.40:0/740148247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7477564d0 0x55e747760a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.045+0000 7f6ee191b640 1 -- 172.21.15.40:0/740148247 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7477527b0 con 0x55e747756ec0 2024-03-18T22:53:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.045+0000 7f6ee191b640 1 --2- 172.21.15.40:0/740148247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e747756ec0 0x55e747767f80 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4000f80 tx=0x7f6ed4009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a827231ca24a7569 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.046+0000 7f6ec7fff640 1 -- 172.21.15.40:0/740148247 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4056040 con 0x55e747756ec0 2024-03-18T22:53:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.046+0000 7f6ec7fff640 1 -- 172.21.15.40:0/740148247 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ed405f070 con 0x55e747756ec0 2024-03-18T22:53:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.047+0000 7f6ee6f68740 1 -- 172.21.15.40:0/740148247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e747756ec0 msgr2=0x55e747767f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.047+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/740148247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e747756ec0 0x55e747767f80 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4000f80 tx=0x7f6ed4009810 comp rx=0 tx=0).stop 2024-03-18T22:53:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.052+0000 7f6ee6f68740 1 -- 172.21.15.40:0/740148247 shutdown_connections 2024-03-18T22:53:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.052+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/740148247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e74775c030 0x55e747760f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.052+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/740148247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e747756ec0 0x55e747767f80 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.052+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/740148247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7477564d0 0x55e747760a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.052+0000 7f6ee6f68740 1 -- 172.21.15.40:0/740148247 >> 172.21.15.40:0/740148247 conn(0x55e747751a40 msgr2=0x55e7477583c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.052+0000 7f6ee6f68740 1 -- 172.21.15.40:0/740148247 shutdown_connections 2024-03-18T22:53:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.052+0000 7f6ee6f68740 1 -- 172.21.15.40:0/740148247 wait complete. 2024-03-18T22:53:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.053+0000 7f6ee6f68740 1 -- start start 2024-03-18T22:53:15.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.053+0000 7f6ee6f68740 1 Processor -- start 2024-03-18T22:53:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.053+0000 7f6ee6f68740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7477564d0 0x55e747765010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.053+0000 7f6ee6f68740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e747756ec0 0x55e747765550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.053+0000 7f6ee6f68740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e74775c030 0x55e7477d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.053+0000 7f6ee6f68740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec00112f0 con 0x55e747756ec0 2024-03-18T22:53:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.053+0000 7f6ee6f68740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec0010b60 con 0x55e74775c030 2024-03-18T22:53:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.053+0000 7f6ee6f68740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec000dd70 con 0x55e7477564d0 2024-03-18T22:53:15.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.053+0000 7f6ee6f68740 10 client.0 _ll_register_callbacks cb 0x7f6ed4053040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.054+0000 7f6ee291d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e74775c030 0x55e7477d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.054+0000 7f6ee211c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7477564d0 0x55e747765010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.054+0000 7f6ee291d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e74775c030 0x55e7477d0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52220/0 (socket says 172.21.15.40:52220) 2024-03-18T22:53:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.054+0000 7f6ee291d640 1 -- 172.21.15.40:0/2818452465 learned_addr learned my addr 172.21.15.40:0/2818452465 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.055+0000 7f6ee191b640 1 --2- 172.21.15.40:0/2818452465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e747756ec0 0x55e747765550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.055+0000 7f6ee211c640 1 -- 172.21.15.40:0/2818452465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e74775c030 msgr2=0x55e7477d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.055+0000 7f6ee211c640 1 --2- 172.21.15.40:0/2818452465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e74775c030 0x55e7477d0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.055+0000 7f6ee211c640 1 -- 172.21.15.40:0/2818452465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e747756ec0 msgr2=0x55e747765550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.055+0000 7f6ee211c640 1 --2- 172.21.15.40:0/2818452465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e747756ec0 0x55e747765550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.055+0000 7f6ee211c640 1 -- 172.21.15.40:0/2818452465 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7477d12d0 con 0x55e7477564d0 2024-03-18T22:53:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.055+0000 7f6ee211c640 1 --2- 172.21.15.40:0/2818452465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7477564d0 0x55e747765010 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x55e74776f660 tx=0x7f6ecc009f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b90a9ca96b3ed07e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.055+0000 7f6ec5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:53:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.055+0000 7f6ec5ffb640 1 -- 172.21.15.40:0/2818452465 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc01d520 con 0x55e7477564d0 2024-03-18T22:53:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.056+0000 7f6ee6f68740 1 -- 172.21.15.40:0/2818452465 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ed405f070 con 0x55e7477564d0 2024-03-18T22:53:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.056+0000 7f6ec5ffb640 1 -- 172.21.15.40:0/2818452465 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6ecc01d6c0 con 0x55e7477564d0 2024-03-18T22:53:15.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.056+0000 7f6ec5ffb640 1 -- 172.21.15.40:0/2818452465 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f6ecc02c050 con 0x55e7477564d0 2024-03-18T22:53:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.056+0000 7f6ee6f68740 10 client.28003 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.056+0000 7f6ee6f68740 1 -- 172.21.15.40:0/2818452465 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e7477d26a0 con 0x55e7477564d0 2024-03-18T22:53:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.057+0000 7f6ec5ffb640 1 -- 172.21.15.40:0/2818452465 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ed405f070 con 0x55e7477564d0 2024-03-18T22:53:15.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.058+0000 7f6ec5ffb640 1 -- 172.21.15.40:0/2818452465 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f6ecc0264b0 con 0x55e7477564d0 2024-03-18T22:53:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.060+0000 7f6ee6f68740 10 client.28003 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.060+0000 7f6ee6f68740 10 client.28003 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:53:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.060+0000 7f6ee6f68740 20 client.28003 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.060+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/2818452465 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55e7477d30d0 0x55e7477d54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.060+0000 7f6ee6f68740 4 client.28003 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:53:15.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.060+0000 7f6ee6f68740 1 -- 172.21.15.40:0/2818452465 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e7476c79f0 con 0x55e7477d30d0 2024-03-18T22:53:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.060+0000 7f6ee191b640 1 --2- 172.21.15.40:0/2818452465 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55e7477d30d0 0x55e7477d54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.067+0000 7f6ee191b640 1 --2- 172.21.15.40:0/2818452465 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55e7477d30d0 0x55e7477d54f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a010c1e6d813b181 server_cookie=36be8b4354588777 in_seq=0 out_seq=0 2024-03-18T22:53:15.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.067+0000 7f6ec5ffb640 10 client.28003 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:53:15.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.069+0000 7f6ec5ffb640 1 -- 172.21.15.40:0/2818452465 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e7476c79f0 con 0x55e7477d30d0 2024-03-18T22:53:15.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.069+0000 7f6ec5ffb640 10 client.28003 handle_command_reply: tid=0 2024-03-18T22:53:15.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.069+0000 7f6ee6f68740 1 client.28003 shutdown 2024-03-18T22:53:15.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.069+0000 7f6ee6f68740 1 -- 172.21.15.40:0/2818452465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7477564d0 msgr2=0x55e747765010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.069+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/2818452465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7477564d0 0x55e747765010 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x55e74776f660 tx=0x7f6ecc009f40 comp rx=0 tx=0).stop 2024-03-18T22:53:15.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.069+0000 7f6ee6f68740 1 -- 172.21.15.40:0/2818452465 shutdown_connections 2024-03-18T22:53:15.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.069+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/2818452465 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55e7477d30d0 0x55e7477d54f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.069+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/2818452465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e74775c030 0x55e7477d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.069+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/2818452465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e747756ec0 0x55e747765550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.069+0000 7f6ee6f68740 1 --2- 172.21.15.40:0/2818452465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e7477564d0 0x55e747765010 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.069+0000 7f6ee6f68740 1 -- 172.21.15.40:0/2818452465 >> 172.21.15.40:0/2818452465 conn(0x55e747751a40 msgr2=0x55e7477583c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:15.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.069+0000 7f6ec5ffb640 0 client.28003 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:53:15.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.069+0000 7f6ee6f68740 1 -- 172.21.15.40:0/2818452465 shutdown_connections 2024-03-18T22:53:15.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.069+0000 7f6ee6f68740 1 -- 172.21.15.40:0/2818452465 wait complete. 2024-03-18T22:53:15.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.070+0000 7f6ee6f68740 20 client.28003 trim_cache size 0 max 16384 2024-03-18T22:53:15.072 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:53:15.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.075+0000 7f6ee311e640 1 -- 172.21.15.40:0/183624291 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ec80a29d0 msgr2=0x7f6ec80a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.075+0000 7f6ee311e640 1 --2- 172.21.15.40:0/183624291 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ec80a29d0 0x7f6ec80a4e90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4004fd0 tx=0x7f6ed4004b80 comp rx=0 tx=0).stop 2024-03-18T22:53:15.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.075+0000 7f6ee311e640 1 -- 172.21.15.40:0/183624291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6edc1097c0 msgr2=0x7f6edc1b66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.075+0000 7f6ee311e640 1 --2- 172.21.15.40:0/183624291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6edc1097c0 0x7f6edc1b66e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6ed800d750 tx=0x7f6ed800dc10 comp rx=0 tx=0).stop 2024-03-18T22:53:15.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.075+0000 7f6ee311e640 1 -- 172.21.15.40:0/183624291 shutdown_connections 2024-03-18T22:53:15.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.076+0000 7f6ee311e640 1 --2- 172.21.15.40:0/183624291 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6ec80a29d0 0x7f6ec80a4e90 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.076+0000 7f6ee311e640 1 --2- 172.21.15.40:0/183624291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6edc1097c0 0x7f6edc1b66e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.076+0000 7f6ee311e640 1 --2- 172.21.15.40:0/183624291 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6edc108e20 0x7f6edc1154d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.076+0000 7f6ee311e640 1 --2- 172.21.15.40:0/183624291 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6edc072b70 0x7f6edc114f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.076+0000 7f6ee311e640 1 -- 172.21.15.40:0/183624291 >> 172.21.15.40:0/183624291 conn(0x7f6edc06e640 msgr2=0x7f6edc070350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:15.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.076+0000 7f6ee311e640 1 -- 172.21.15.40:0/183624291 shutdown_connections 2024-03-18T22:53:15.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.076+0000 7f6ee311e640 1 -- 172.21.15.40:0/183624291 wait complete. 2024-03-18T22:53:15.119 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:53:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.314+0000 7f6bb2787640 1 -- 172.21.15.40:0/3934098052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bac1022d0 msgr2=0x7f6bac108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.314+0000 7f6bb2787640 1 --2- 172.21.15.40:0/3934098052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bac1022d0 0x7f6bac108b60 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009ca0 tx=0x7f6ba402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.314+0000 7f6bb2787640 1 -- 172.21.15.40:0/3934098052 shutdown_connections 2024-03-18T22:53:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.314+0000 7f6bb2787640 1 --2- 172.21.15.40:0/3934098052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bac1022d0 0x7f6bac108b60 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.314+0000 7f6bb2787640 1 --2- 172.21.15.40:0/3934098052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6bac101910 0x7f6bac101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.314+0000 7f6bb2787640 1 --2- 172.21.15.40:0/3934098052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac100720 0x7f6bac100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.314+0000 7f6bb2787640 1 -- 172.21.15.40:0/3934098052 >> 172.21.15.40:0/3934098052 conn(0x7f6bac0fbc90 msgr2=0x7f6bac0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.314+0000 7f6bb2787640 1 -- 172.21.15.40:0/3934098052 shutdown_connections 2024-03-18T22:53:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.314+0000 7f6bb2787640 1 -- 172.21.15.40:0/3934098052 wait complete. 2024-03-18T22:53:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.315+0000 7f6bb2787640 1 Processor -- start 2024-03-18T22:53:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.315+0000 7f6bb2787640 1 -- start start 2024-03-18T22:53:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.315+0000 7f6bb2787640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bac100720 0x7f6bac1046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.315+0000 7f6bb2787640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac101910 0x7f6bac102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.315+0000 7f6bb2787640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6bac1022d0 0x7f6bac103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.315+0000 7f6bb2787640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac10c250 con 0x7f6bac100720 2024-03-18T22:53:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.315+0000 7f6bb2787640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac10c0d0 con 0x7f6bac1022d0 2024-03-18T22:53:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.315+0000 7f6bb2787640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac10c3d0 con 0x7f6bac101910 2024-03-18T22:53:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.315+0000 7f6babfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bac100720 0x7f6bac1046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.315+0000 7f6babfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bac100720 0x7f6bac1046b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39482/0 (socket says 172.21.15.40:39482) 2024-03-18T22:53:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.315+0000 7f6babfff640 1 -- 172.21.15.40:0/1850049907 learned_addr learned my addr 172.21.15.40:0/1850049907 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.315+0000 7f6bb0d24640 1 --2- 172.21.15.40:0/1850049907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6bac1022d0 0x7f6bac103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.315+0000 7f6bab7fe640 1 --2- 172.21.15.40:0/1850049907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac101910 0x7f6bac102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.316+0000 7f6babfff640 1 -- 172.21.15.40:0/1850049907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac101910 msgr2=0x7f6bac102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.316+0000 7f6babfff640 1 --2- 172.21.15.40:0/1850049907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac101910 0x7f6bac102cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.316+0000 7f6babfff640 1 -- 172.21.15.40:0/1850049907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6bac1022d0 msgr2=0x7f6bac103230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.316+0000 7f6babfff640 1 --2- 172.21.15.40:0/1850049907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6bac1022d0 0x7f6bac103230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.316+0000 7f6babfff640 1 -- 172.21.15.40:0/1850049907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bac103ac0 con 0x7f6bac100720 2024-03-18T22:53:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.316+0000 7f6bb0d24640 1 --2- 172.21.15.40:0/1850049907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6bac1022d0 0x7f6bac103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:53:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.316+0000 7f6babfff640 1 --2- 172.21.15.40:0/1850049907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bac100720 0x7f6bac1046b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00e9c0 tx=0x7f6b9c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.316+0000 7f6ba97fa640 1 -- 172.21.15.40:0/1850049907 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c00cd80 con 0x7f6bac100720 2024-03-18T22:53:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.316+0000 7f6ba97fa640 1 -- 172.21.15.40:0/1850049907 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b9c0035f0 con 0x7f6bac100720 2024-03-18T22:53:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.316+0000 7f6ba97fa640 1 -- 172.21.15.40:0/1850049907 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c010690 con 0x7f6bac100720 2024-03-18T22:53:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.316+0000 7f6bb2787640 1 -- 172.21.15.40:0/1850049907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bac1a1b50 con 0x7f6bac100720 2024-03-18T22:53:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.316+0000 7f6bb2787640 1 -- 172.21.15.40:0/1850049907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bac1a2050 con 0x7f6bac100720 2024-03-18T22:53:15.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.318+0000 7f6ba97fa640 1 -- 172.21.15.40:0/1850049907 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6b9c0037a0 con 0x7f6bac100720 2024-03-18T22:53:15.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb5372740 1 Processor -- start 2024-03-18T22:53:15.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb5372740 1 -- start start 2024-03-18T22:53:15.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb5372740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20a92f0 0x5604f20a9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb5372740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a9ce0 0x5604f20ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb5372740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20aee50 0x5604f20b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb5372740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5604f200d4b0 con 0x5604f20a92f0 2024-03-18T22:53:15.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb5372740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5604f1fb5d70 con 0x5604f20a9ce0 2024-03-18T22:53:15.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb5372740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5604f2009550 con 0x5604f20aee50 2024-03-18T22:53:15.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb0d24640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20a92f0 0x5604f20a9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb0d24640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20a92f0 0x5604f20a9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39496/0 (socket says 172.21.15.40:39496) 2024-03-18T22:53:15.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb0d24640 1 -- 172.21.15.40:0/4134777947 learned_addr learned my addr 172.21.15.40:0/4134777947 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:15.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6ba97fa640 1 --2- 172.21.15.40:0/1850049907 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6b940a2880 0x7f6b940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bab7fe640 1 --2- 172.21.15.40:0/4134777947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20aee50 0x5604f20b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6babfff640 1 --2- 172.21.15.40:0/4134777947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a9ce0 0x5604f20ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb0d24640 1 --2- 172.21.15.40:0/1850049907 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6b940a2880 0x7f6b940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb0d24640 1 -- 172.21.15.40:0/4134777947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20aee50 msgr2=0x5604f20b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb0d24640 1 --2- 172.21.15.40:0/4134777947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20aee50 0x5604f20b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb0d24640 1 -- 172.21.15.40:0/4134777947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a9ce0 msgr2=0x5604f20ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb0d24640 1 --2- 172.21.15.40:0/4134777947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a9ce0 0x5604f20ae7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb0d24640 1 -- 172.21.15.40:0/4134777947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604f1fd8210 con 0x5604f20a92f0 2024-03-18T22:53:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bab7fe640 1 --2- 172.21.15.40:0/4134777947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20aee50 0x5604f20b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:53:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6ba97fa640 1 -- 172.21.15.40:0/1850049907 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6b9c0cf400 con 0x7f6bac100720 2024-03-18T22:53:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.319+0000 7f6bb0d24640 1 --2- 172.21.15.40:0/4134777947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20a92f0 0x5604f20a9710 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4032380 tx=0x7f6ba40328e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad78bdac6df8d87e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.320+0000 7f6b927fc640 1 -- 172.21.15.40:0/4134777947 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4050070 con 0x5604f20a92f0 2024-03-18T22:53:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.320+0000 7f6b927fc640 1 -- 172.21.15.40:0/4134777947 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ba40047b0 con 0x5604f20a92f0 2024-03-18T22:53:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.320+0000 7f6b927fc640 1 -- 172.21.15.40:0/4134777947 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4039040 con 0x5604f20a92f0 2024-03-18T22:53:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.320+0000 7f6bb0d24640 1 --2- 172.21.15.40:0/1850049907 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6b940a2880 0x7f6b940a4d40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4005fe0 tx=0x7f6ba4048000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.320+0000 7f6bb5372740 1 -- 172.21.15.40:0/4134777947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20a92f0 msgr2=0x5604f20a9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.320+0000 7f6bb5372740 1 --2- 172.21.15.40:0/4134777947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20a92f0 0x5604f20a9710 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4032380 tx=0x7f6ba40328e0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.320+0000 7f6bb5372740 1 -- 172.21.15.40:0/4134777947 shutdown_connections 2024-03-18T22:53:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.320+0000 7f6bb5372740 1 --2- 172.21.15.40:0/4134777947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20aee50 0x5604f20b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.320+0000 7f6bb5372740 1 --2- 172.21.15.40:0/4134777947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a9ce0 0x5604f20ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.320+0000 7f6bb5372740 1 --2- 172.21.15.40:0/4134777947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20a92f0 0x5604f20a9710 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.320+0000 7f6bb5372740 1 -- 172.21.15.40:0/4134777947 >> 172.21.15.40:0/4134777947 conn(0x5604f20a4860 msgr2=0x5604f20a4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.320+0000 7f6bb5372740 1 -- 172.21.15.40:0/4134777947 shutdown_connections 2024-03-18T22:53:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.320+0000 7f6bb5372740 1 -- 172.21.15.40:0/4134777947 wait complete. 2024-03-18T22:53:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.321+0000 7f6bb5372740 1 -- start start 2024-03-18T22:53:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.321+0000 7f6bb5372740 1 Processor -- start 2024-03-18T22:53:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.321+0000 7f6bb5372740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a92f0 0x5604f20b3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.321+0000 7f6bb5372740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20a9ce0 0x5604f20b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.321+0000 7f6bb5372740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20aee50 0x5604f20bad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.321+0000 7f6bb5372740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5604f20b1e00 con 0x5604f20aee50 2024-03-18T22:53:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.321+0000 7f6bb5372740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5604f1fc9310 con 0x5604f20a92f0 2024-03-18T22:53:15.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.321+0000 7f6bb5372740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5604f20b1f80 con 0x5604f20a9ce0 2024-03-18T22:53:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.321+0000 7f6bb5372740 10 client.0 _ll_register_callbacks cb 0x5604f1fbd2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.321+0000 7f6bb0d24640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20aee50 0x5604f20bad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.321+0000 7f6bb0d24640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20aee50 0x5604f20bad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39504/0 (socket says 172.21.15.40:39504) 2024-03-18T22:53:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.321+0000 7f6bb0d24640 1 -- 172.21.15.40:0/1676302454 learned_addr learned my addr 172.21.15.40:0/1676302454 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.321+0000 7f6babfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a92f0 0x5604f20b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.321+0000 7f6bab7fe640 1 --2- 172.21.15.40:0/1676302454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20a9ce0 0x5604f20b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6bb0d24640 1 -- 172.21.15.40:0/1676302454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20a9ce0 msgr2=0x5604f20b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6bb0d24640 1 --2- 172.21.15.40:0/1676302454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20a9ce0 0x5604f20b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6bb0d24640 1 -- 172.21.15.40:0/1676302454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a92f0 msgr2=0x5604f20b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6bb0d24640 1 --2- 172.21.15.40:0/1676302454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a92f0 0x5604f20b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6bb0d24640 1 -- 172.21.15.40:0/1676302454 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604f1d213f0 con 0x5604f20aee50 2024-03-18T22:53:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6babfff640 1 --2- 172.21.15.40:0/1676302454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a92f0 0x5604f20b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:53:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6bb0d24640 1 --2- 172.21.15.40:0/1676302454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20aee50 0x5604f20bad20 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4037040 tx=0x7f6ba402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82fe58f4850d2616 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:15.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6b8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:53:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6b8b7fe640 1 -- 172.21.15.40:0/1676302454 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4058020 con 0x5604f20aee50 2024-03-18T22:53:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6b8b7fe640 1 -- 172.21.15.40:0/1676302454 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6ba4054460 con 0x5604f20aee50 2024-03-18T22:53:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6bb5372740 1 -- 172.21.15.40:0/1676302454 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ba40047b0 con 0x5604f20aee50 2024-03-18T22:53:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6b8b7fe640 1 -- 172.21.15.40:0/1676302454 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f6ba4063050 con 0x5604f20aee50 2024-03-18T22:53:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6bb5372740 10 client.18195 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6bb5372740 1 -- 172.21.15.40:0/1676302454 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5604f1deb7d0 con 0x5604f20aee50 2024-03-18T22:53:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6b8b7fe640 1 -- 172.21.15.40:0/1676302454 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ba40047b0 con 0x5604f20aee50 2024-03-18T22:53:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6b8b7fe640 1 -- 172.21.15.40:0/1676302454 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f6ba403f070 con 0x5604f20aee50 2024-03-18T22:53:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6bb5372740 10 client.18195 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6bb5372740 10 client.18195 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:53:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.322+0000 7f6bb5372740 20 client.18195 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.323+0000 7f6bb5372740 1 --2- 172.21.15.40:0/1676302454 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5604f20bf330 0x5604f20c1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.323+0000 7f6bb5372740 4 client.18195 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:53:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.323+0000 7f6bb5372740 1 -- 172.21.15.40:0/1676302454 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604f206b3e0 con 0x5604f20bf330 2024-03-18T22:53:15.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.323+0000 7f6babfff640 1 --2- 172.21.15.40:0/1676302454 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5604f20bf330 0x5604f20c1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.323+0000 7f6babfff640 1 --2- 172.21.15.40:0/1676302454 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5604f20bf330 0x5604f20c1790 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2b17a07b0bf4040e server_cookie=48d29cbed3181bdc in_seq=0 out_seq=0 2024-03-18T22:53:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.323+0000 7f6b8b7fe640 10 client.18195 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:53:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.324+0000 7f6b8b7fe640 1 -- 172.21.15.40:0/1676302454 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5604f206b3e0 con 0x5604f20bf330 2024-03-18T22:53:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.324+0000 7f6b8b7fe640 10 client.18195 handle_command_reply: tid=0 2024-03-18T22:53:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.324+0000 7f6bb5372740 1 client.18195 shutdown 2024-03-18T22:53:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.324+0000 7f6bb5372740 1 -- 172.21.15.40:0/1676302454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20aee50 msgr2=0x5604f20bad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.324+0000 7f6bb5372740 1 --2- 172.21.15.40:0/1676302454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20aee50 0x5604f20bad20 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4037040 tx=0x7f6ba402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.324+0000 7f6bb5372740 1 -- 172.21.15.40:0/1676302454 shutdown_connections 2024-03-18T22:53:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.324+0000 7f6bb5372740 1 --2- 172.21.15.40:0/1676302454 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5604f20bf330 0x5604f20c1790 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.324+0000 7f6bb5372740 1 --2- 172.21.15.40:0/1676302454 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20aee50 0x5604f20bad20 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.324+0000 7f6bb5372740 1 --2- 172.21.15.40:0/1676302454 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20a9ce0 0x5604f20b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.324+0000 7f6bb5372740 1 --2- 172.21.15.40:0/1676302454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a92f0 0x5604f20b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.324+0000 7f6b8b7fe640 0 client.18195 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:53:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.324+0000 7f6bb5372740 1 -- 172.21.15.40:0/1676302454 >> 172.21.15.40:0/1676302454 conn(0x5604f20a4860 msgr2=0x5604f20a3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:15.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.324+0000 7f6bb5372740 1 -- 172.21.15.40:0/1676302454 shutdown_connections 2024-03-18T22:53:15.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.324+0000 7f6bb5372740 1 -- 172.21.15.40:0/1676302454 wait complete. 2024-03-18T22:53:15.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.325+0000 7f6bb5372740 20 client.18195 trim_cache size 0 max 16384 2024-03-18T22:53:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6bb5372740 1 Processor -- start 2024-03-18T22:53:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6bb5372740 1 -- start start 2024-03-18T22:53:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6bb5372740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20a92f0 0x5604f209fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6bb5372740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a9ce0 0x5604f20bace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6bb5372740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20aee50 0x5604f20a3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6bb5372740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac1ac6e0 con 0x5604f20a92f0 2024-03-18T22:53:15.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6bb5372740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac1ae310 con 0x5604f20a9ce0 2024-03-18T22:53:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6bb5372740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac1acdb0 con 0x5604f20aee50 2024-03-18T22:53:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6babfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20a92f0 0x5604f209fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6babfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20a92f0 0x5604f209fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39512/0 (socket says 172.21.15.40:39512) 2024-03-18T22:53:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6babfff640 1 -- 172.21.15.40:0/633382369 learned_addr learned my addr 172.21.15.40:0/633382369 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6bb0d24640 1 --2- 172.21.15.40:0/633382369 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20aee50 0x5604f20a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6bab7fe640 1 --2- 172.21.15.40:0/633382369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a9ce0 0x5604f20bace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6babfff640 1 -- 172.21.15.40:0/633382369 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20aee50 msgr2=0x5604f20a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6babfff640 1 --2- 172.21.15.40:0/633382369 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20aee50 0x5604f20a3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6babfff640 1 -- 172.21.15.40:0/633382369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a9ce0 msgr2=0x5604f20bace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6babfff640 1 --2- 172.21.15.40:0/633382369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a9ce0 0x5604f20bace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6babfff640 1 -- 172.21.15.40:0/633382369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604f207a880 con 0x5604f20a92f0 2024-03-18T22:53:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.338+0000 7f6babfff640 1 --2- 172.21.15.40:0/633382369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20a92f0 0x5604f209fa30 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c09b8e0 tx=0x7f6b9c086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=257672bcf0f48b5c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.339+0000 7f6b91ffb640 1 -- 172.21.15.40:0/633382369 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c09b2b0 con 0x5604f20a92f0 2024-03-18T22:53:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.339+0000 7f6b91ffb640 1 -- 172.21.15.40:0/633382369 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b9c003b10 con 0x5604f20a92f0 2024-03-18T22:53:15.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.339+0000 7f6b91ffb640 1 -- 172.21.15.40:0/633382369 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c0cf150 con 0x5604f20a92f0 2024-03-18T22:53:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.339+0000 7f6bb5372740 1 -- 172.21.15.40:0/633382369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20a92f0 msgr2=0x5604f209fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.339+0000 7f6bb5372740 1 --2- 172.21.15.40:0/633382369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20a92f0 0x5604f209fa30 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c09b8e0 tx=0x7f6b9c086040 comp rx=0 tx=0).stop 2024-03-18T22:53:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.339+0000 7f6bb5372740 1 -- 172.21.15.40:0/633382369 shutdown_connections 2024-03-18T22:53:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.339+0000 7f6bb5372740 1 --2- 172.21.15.40:0/633382369 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20aee50 0x5604f20a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.339+0000 7f6bb5372740 1 --2- 172.21.15.40:0/633382369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a9ce0 0x5604f20bace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.339+0000 7f6bb5372740 1 --2- 172.21.15.40:0/633382369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20a92f0 0x5604f209fa30 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.339+0000 7f6bb5372740 1 -- 172.21.15.40:0/633382369 >> 172.21.15.40:0/633382369 conn(0x5604f20a4860 msgr2=0x5604f20a4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.339+0000 7f6bb5372740 1 -- 172.21.15.40:0/633382369 shutdown_connections 2024-03-18T22:53:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.339+0000 7f6bb5372740 1 -- 172.21.15.40:0/633382369 wait complete. 2024-03-18T22:53:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.339+0000 7f6bb5372740 1 -- start start 2024-03-18T22:53:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb5372740 1 Processor -- start 2024-03-18T22:53:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb5372740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a92f0 0x5604f20b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb5372740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20a9ce0 0x5604f20b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb5372740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20aee50 0x5604f20b9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb5372740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac1adc40 con 0x5604f20aee50 2024-03-18T22:53:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb5372740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac1ad4b0 con 0x5604f20a92f0 2024-03-18T22:53:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb5372740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac1aa6c0 con 0x5604f20a9ce0 2024-03-18T22:53:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb5372740 10 client.0 _ll_register_callbacks cb 0x7f6ba404e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb0d24640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20aee50 0x5604f20b9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bab7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20a9ce0 0x5604f20b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6babfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a92f0 0x5604f20b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb0d24640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20aee50 0x5604f20b9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39524/0 (socket says 172.21.15.40:39524) 2024-03-18T22:53:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb0d24640 1 -- 172.21.15.40:0/382314884 learned_addr learned my addr 172.21.15.40:0/382314884 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb0d24640 1 -- 172.21.15.40:0/382314884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20a9ce0 msgr2=0x5604f20b8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb0d24640 1 --2- 172.21.15.40:0/382314884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20a9ce0 0x5604f20b8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb0d24640 1 -- 172.21.15.40:0/382314884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a92f0 msgr2=0x5604f20b87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb0d24640 1 --2- 172.21.15.40:0/382314884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a92f0 0x5604f20b87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb0d24640 1 -- 172.21.15.40:0/382314884 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604f20ba310 con 0x5604f20aee50 2024-03-18T22:53:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6bb0d24640 1 --2- 172.21.15.40:0/382314884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20aee50 0x5604f20b9d20 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4005f50 tx=0x7f6ba4043700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74a93e3653f9796 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.340+0000 7f6b8b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:53:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.341+0000 7f6b8b7fe640 1 -- 172.21.15.40:0/382314884 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba406c060 con 0x5604f20aee50 2024-03-18T22:53:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.341+0000 7f6b8b7fe640 1 -- 172.21.15.40:0/382314884 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6ba4004410 con 0x5604f20aee50 2024-03-18T22:53:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.341+0000 7f6bb5372740 1 -- 172.21.15.40:0/382314884 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5604f201a9f0 con 0x5604f20aee50 2024-03-18T22:53:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.341+0000 7f6b8b7fe640 1 -- 172.21.15.40:0/382314884 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f6ba4073050 con 0x5604f20aee50 2024-03-18T22:53:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.341+0000 7f6bb5372740 10 client.18201 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.341+0000 7f6bb5372740 1 -- 172.21.15.40:0/382314884 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5604f2125af0 con 0x5604f20aee50 2024-03-18T22:53:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.341+0000 7f6b8b7fe640 1 -- 172.21.15.40:0/382314884 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5604f201a9f0 con 0x5604f20aee50 2024-03-18T22:53:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.341+0000 7f6b8b7fe640 1 -- 172.21.15.40:0/382314884 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f6ba403f040 con 0x5604f20aee50 2024-03-18T22:53:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.341+0000 7f6bb5372740 10 client.18201 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.341+0000 7f6bb5372740 10 client.18201 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:53:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.341+0000 7f6bb5372740 20 client.18201 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:15.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.341+0000 7f6bb5372740 1 --2- 172.21.15.40:0/382314884 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5604f2126580 0x5604f21289a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.341+0000 7f6bb5372740 4 client.18201 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.341+0000 7f6bb5372740 1 -- 172.21.15.40:0/382314884 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5604f2014d40 con 0x5604f2126580 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.341+0000 7f6babfff640 1 --2- 172.21.15.40:0/382314884 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5604f2126580 0x5604f21289a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6b8b7fe640 10 client.18201 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6babfff640 1 --2- 172.21.15.40:0/382314884 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5604f2126580 0x5604f21289a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=26f32470adeec545 server_cookie=b07a2a2a86e6b1ff in_seq=0 out_seq=0 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6b8b7fe640 1 -- 172.21.15.40:0/382314884 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5604f2014d40 con 0x5604f2126580 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6b8b7fe640 10 client.18201 handle_command_reply: tid=0 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6bb5372740 1 client.18201 shutdown 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6bb5372740 1 -- 172.21.15.40:0/382314884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20aee50 msgr2=0x5604f20b9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6bb5372740 1 --2- 172.21.15.40:0/382314884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20aee50 0x5604f20b9d20 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4005f50 tx=0x7f6ba4043700 comp rx=0 tx=0).stop 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6bb5372740 1 -- 172.21.15.40:0/382314884 shutdown_connections 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6bb5372740 1 --2- 172.21.15.40:0/382314884 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5604f2126580 0x5604f21289a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6bb5372740 1 --2- 172.21.15.40:0/382314884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5604f20aee50 0x5604f20b9d20 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6bb5372740 1 --2- 172.21.15.40:0/382314884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5604f20a9ce0 0x5604f20b8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6bb5372740 1 --2- 172.21.15.40:0/382314884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5604f20a92f0 0x5604f20b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6b8b7fe640 0 client.18201 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:53:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6bb5372740 1 -- 172.21.15.40:0/382314884 >> 172.21.15.40:0/382314884 conn(0x5604f20a4860 msgr2=0x5604f20a7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6bb5372740 1 -- 172.21.15.40:0/382314884 shutdown_connections 2024-03-18T22:53:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6bb5372740 1 -- 172.21.15.40:0/382314884 wait complete. 2024-03-18T22:53:15.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.342+0000 7f6bb5372740 20 client.18201 trim_cache size 0 max 16384 2024-03-18T22:53:15.350 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:53:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.343+0000 7f6bb2787640 1 -- 172.21.15.40:0/1850049907 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6b940a2880 msgr2=0x7f6b940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.343+0000 7f6bb2787640 1 --2- 172.21.15.40:0/1850049907 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6b940a2880 0x7f6b940a4d40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4005fe0 tx=0x7f6ba4048000 comp rx=0 tx=0).stop 2024-03-18T22:53:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.343+0000 7f6bb2787640 1 -- 172.21.15.40:0/1850049907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bac100720 msgr2=0x7f6bac1046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.343+0000 7f6bb2787640 1 --2- 172.21.15.40:0/1850049907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bac100720 0x7f6bac1046b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00e9c0 tx=0x7f6b9c00ee80 comp rx=0 tx=0).stop 2024-03-18T22:53:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.343+0000 7f6bb2787640 1 -- 172.21.15.40:0/1850049907 shutdown_connections 2024-03-18T22:53:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.343+0000 7f6bb2787640 1 --2- 172.21.15.40:0/1850049907 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6b940a2880 0x7f6b940a4d40 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.343+0000 7f6bb2787640 1 --2- 172.21.15.40:0/1850049907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6bac1022d0 0x7f6bac103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.343+0000 7f6bb2787640 1 --2- 172.21.15.40:0/1850049907 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6bac101910 0x7f6bac102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.343+0000 7f6bb2787640 1 --2- 172.21.15.40:0/1850049907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bac100720 0x7f6bac1046b0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.343+0000 7f6bb2787640 1 -- 172.21.15.40:0/1850049907 >> 172.21.15.40:0/1850049907 conn(0x7f6bac0fbc90 msgr2=0x7f6bac0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.343+0000 7f6bb2787640 1 -- 172.21.15.40:0/1850049907 shutdown_connections 2024-03-18T22:53:15.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.343+0000 7f6bb2787640 1 -- 172.21.15.40:0/1850049907 wait complete. 2024-03-18T22:53:15.354 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:53:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.510+0000 7f8d93fff640 1 -- 172.21.15.40:0/2891437210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d94108e20 msgr2=0x7f8d94109280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.510+0000 7f8d93fff640 1 --2- 172.21.15.40:0/2891437210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d94108e20 0x7f8d94109280 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009860 tx=0x7f8d8802bb70 comp rx=0 tx=0).stop 2024-03-18T22:53:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.510+0000 7f8d93fff640 1 -- 172.21.15.40:0/2891437210 shutdown_connections 2024-03-18T22:53:15.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.510+0000 7f8d93fff640 1 --2- 172.21.15.40:0/2891437210 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d941097c0 0x7f8d9410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.510+0000 7f8d93fff640 1 --2- 172.21.15.40:0/2891437210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d94108e20 0x7f8d94109280 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.510+0000 7f8d93fff640 1 --2- 172.21.15.40:0/2891437210 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d94072b70 0x7f8d94072f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.510+0000 7f8d93fff640 1 -- 172.21.15.40:0/2891437210 >> 172.21.15.40:0/2891437210 conn(0x7f8d9406e640 msgr2=0x7f8d94070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.510+0000 7f8d93fff640 1 -- 172.21.15.40:0/2891437210 shutdown_connections 2024-03-18T22:53:15.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.510+0000 7f8d93fff640 1 -- 172.21.15.40:0/2891437210 wait complete. 2024-03-18T22:53:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.511+0000 7f8d93fff640 1 Processor -- start 2024-03-18T22:53:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.511+0000 7f8d93fff640 1 -- start start 2024-03-18T22:53:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.511+0000 7f8d93fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d94072b70 0x7f8d94110350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.511+0000 7f8d93fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d94108e20 0x7f8d94110890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.511+0000 7f8d93fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d941097c0 0x7f8d94117900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.511+0000 7f8d93fff640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9411bfd0 con 0x7f8d941097c0 2024-03-18T22:53:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.511+0000 7f8d93fff640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9411be50 con 0x7f8d94108e20 2024-03-18T22:53:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.511+0000 7f8d93fff640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9411c150 con 0x7f8d94072b70 2024-03-18T22:53:15.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.512+0000 7f8d927fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d94108e20 0x7f8d94110890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.512+0000 7f8d92ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d94072b70 0x7f8d94110350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.512+0000 7f8d927fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d94108e20 0x7f8d94110890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52294/0 (socket says 172.21.15.40:52294) 2024-03-18T22:53:15.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.512+0000 7f8d927fc640 1 -- 172.21.15.40:0/1291698722 learned_addr learned my addr 172.21.15.40:0/1291698722 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:15.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.512+0000 7f8d937fe640 1 --2- 172.21.15.40:0/1291698722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d941097c0 0x7f8d94117900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.514+0000 7f8d927fc640 1 -- 172.21.15.40:0/1291698722 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d94072b70 msgr2=0x7f8d94110350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.514+0000 7f8d927fc640 1 --2- 172.21.15.40:0/1291698722 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d94072b70 0x7f8d94110350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.514+0000 7f8d927fc640 1 -- 172.21.15.40:0/1291698722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d941097c0 msgr2=0x7f8d94117900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.514+0000 7f8d927fc640 1 --2- 172.21.15.40:0/1291698722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d941097c0 0x7f8d94117900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.514+0000 7f8d927fc640 1 -- 172.21.15.40:0/1291698722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d94117f70 con 0x7f8d94108e20 2024-03-18T22:53:15.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.518+0000 7f8d927fc640 1 --2- 172.21.15.40:0/1291698722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d94108e20 0x7f8d94110890 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009980 tx=0x7f8d88004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:15.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.520+0000 7f8d7bfff640 1 -- 172.21.15.40:0/1291698722 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d88041020 con 0x7f8d94108e20 2024-03-18T22:53:15.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.520+0000 7f8d93fff640 1 -- 172.21.15.40:0/1291698722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d94118200 con 0x7f8d94108e20 2024-03-18T22:53:15.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.520+0000 7f8d93fff640 1 -- 172.21.15.40:0/1291698722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d941b7280 con 0x7f8d94108e20 2024-03-18T22:53:15.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.520+0000 7f8d7bfff640 1 -- 172.21.15.40:0/1291698722 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d8802c9c0 con 0x7f8d94108e20 2024-03-18T22:53:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.520+0000 7f8d7bfff640 1 -- 172.21.15.40:0/1291698722 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d88039040 con 0x7f8d94108e20 2024-03-18T22:53:15.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.524+0000 7f8d9bfd1740 1 Processor -- start 2024-03-18T22:53:15.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.525+0000 7f8d9bfd1740 1 -- start start 2024-03-18T22:53:15.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.525+0000 7f8d9bfd1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a214d0 0x55e433a218f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.525+0000 7f8d9bfd1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a21ec0 0x55e433a269c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.525+0000 7f8d9bfd1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a27030 0x55e433a294c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.525+0000 7f8d9bfd1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e4339854b0 con 0x55e433a21ec0 2024-03-18T22:53:15.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.525+0000 7f8d9bfd1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e43392dd70 con 0x55e433a27030 2024-03-18T22:53:15.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.525+0000 7f8d9bfd1740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e433981550 con 0x55e433a214d0 2024-03-18T22:53:15.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.529+0000 7f8d92ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a21ec0 0x55e433a269c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.529+0000 7f8d927fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a27030 0x55e433a294c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.529+0000 7f8d92ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a21ec0 0x55e433a269c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39544/0 (socket says 172.21.15.40:39544) 2024-03-18T22:53:15.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.529+0000 7f8d92ffd640 1 -- 172.21.15.40:0/335509645 learned_addr learned my addr 172.21.15.40:0/335509645 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:15.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.532+0000 7f8d7bfff640 1 -- 172.21.15.40:0/1291698722 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8d88044020 con 0x7f8d94108e20 2024-03-18T22:53:15.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.533+0000 7f8d927fc640 1 -- 172.21.15.40:0/335509645 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a214d0 msgr2=0x55e433a218f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:53:15.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.533+0000 7f8d927fc640 1 --2- 172.21.15.40:0/335509645 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a214d0 0x55e433a218f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.533+0000 7f8d927fc640 1 -- 172.21.15.40:0/335509645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a21ec0 msgr2=0x55e433a269c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.533+0000 7f8d927fc640 1 --2- 172.21.15.40:0/335509645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a21ec0 0x55e433a269c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.533+0000 7f8d927fc640 1 -- 172.21.15.40:0/335509645 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e433950210 con 0x55e433a27030 2024-03-18T22:53:15.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.541+0000 7f8d7bfff640 1 --2- 172.21.15.40:0/1291698722 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8d7c0a2a00 0x7f8d7c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.541+0000 7f8d927fc640 1 --2- 172.21.15.40:0/335509645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a27030 0x55e433a294c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8d880f29d0 tx=0x7f8d880f7da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b8bdaa84753352e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:15.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.541+0000 7f8d7bfff640 1 -- 172.21.15.40:0/1291698722 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8d880f5380 con 0x7f8d94108e20 2024-03-18T22:53:15.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.543+0000 7f8d937fe640 1 --2- 172.21.15.40:0/1291698722 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8d7c0a2a00 0x7f8d7c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.546+0000 7f8d797fa640 1 -- 172.21.15.40:0/335509645 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d88106070 con 0x55e433a27030 2024-03-18T22:53:15.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.546+0000 7f8d797fa640 1 -- 172.21.15.40:0/335509645 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d880c1fd0 con 0x55e433a27030 2024-03-18T22:53:15.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.547+0000 7f8d797fa640 1 -- 172.21.15.40:0/335509645 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d880c1410 con 0x55e433a27030 2024-03-18T22:53:15.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.549+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/335509645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a27030 msgr2=0x55e433a294c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.549+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/335509645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a27030 0x55e433a294c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8d880f29d0 tx=0x7f8d880f7da0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.552+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/335509645 shutdown_connections 2024-03-18T22:53:15.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.552+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/335509645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a27030 0x55e433a294c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.552+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/335509645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a21ec0 0x55e433a269c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.552+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/335509645 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a214d0 0x55e433a218f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.552+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/335509645 >> 172.21.15.40:0/335509645 conn(0x55e433a1ca40 msgr2=0x55e433a1ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:15.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.556+0000 7f8d937fe640 1 --2- 172.21.15.40:0/1291698722 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8d7c0a2a00 0x7f8d7c0a4ec0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x55e433a22720 tx=0x7f8d8c009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:15.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.558+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/335509645 shutdown_connections 2024-03-18T22:53:15.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.558+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/335509645 wait complete. 2024-03-18T22:53:15.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.559+0000 7f8d9bfd1740 1 -- start start 2024-03-18T22:53:15.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.559+0000 7f8d9bfd1740 1 Processor -- start 2024-03-18T22:53:15.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.559+0000 7f8d9bfd1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 0x55e433a2b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.559+0000 7f8d9bfd1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a21ec0 0x55e433a2bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.559+0000 7f8d9bfd1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a27030 0x55e433a32cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.559+0000 7f8d9bfd1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e433a29da0 con 0x55e433a27030 2024-03-18T22:53:15.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.559+0000 7f8d9bfd1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e433941310 con 0x55e433a214d0 2024-03-18T22:53:15.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.559+0000 7f8d9bfd1740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e433a29f20 con 0x55e433a21ec0 2024-03-18T22:53:15.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.559+0000 7f8d9bfd1740 10 client.0 _ll_register_callbacks cb 0x55e4339352d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:15.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.560+0000 7f8d937fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a27030 0x55e433a32cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.560+0000 7f8d927fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a21ec0 0x55e433a2bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.560+0000 7f8d937fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a27030 0x55e433a32cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39546/0 (socket says 172.21.15.40:39546) 2024-03-18T22:53:15.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.560+0000 7f8d937fe640 1 -- 172.21.15.40:0/3455984779 learned_addr learned my addr 172.21.15.40:0/3455984779 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:15.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.560+0000 7f8d927fc640 1 -- 172.21.15.40:0/3455984779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 msgr2=0x55e433a2b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.560+0000 7f8d927fc640 1 --2- 172.21.15.40:0/3455984779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 0x55e433a2b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.560+0000 7f8d927fc640 1 -- 172.21.15.40:0/3455984779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a27030 msgr2=0x55e433a32cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.560+0000 7f8d927fc640 1 --2- 172.21.15.40:0/3455984779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a27030 0x55e433a32cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.560+0000 7f8d927fc640 1 -- 172.21.15.40:0/3455984779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4336993f0 con 0x55e433a21ec0 2024-03-18T22:53:15.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.560+0000 7f8d927fc640 1 --2- 172.21.15.40:0/3455984779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a21ec0 0x55e433a2bcb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8d880bdca0 tx=0x7f8d88005ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a049016ab0d699e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:15.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.561+0000 7f8d76ff5640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:53:15.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.574+0000 7f8d76ff5640 1 -- 172.21.15.40:0/3455984779 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d880bfa10 con 0x55e433a21ec0 2024-03-18T22:53:15.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.574+0000 7f8d76ff5640 1 -- 172.21.15.40:0/3455984779 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8d880c1b30 con 0x55e433a21ec0 2024-03-18T22:53:15.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.575+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3455984779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d880c1fd0 con 0x55e433a21ec0 2024-03-18T22:53:15.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.582+0000 7f8d76ff5640 1 -- 172.21.15.40:0/3455984779 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8d880c1230 con 0x55e433a21ec0 2024-03-18T22:53:15.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.583+0000 7f8d9bfd1740 10 client.28030 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:15.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.583+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3455984779 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e4337637d0 con 0x55e433a21ec0 2024-03-18T22:53:15.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.584+0000 7f8d76ff5640 1 -- 172.21.15.40:0/3455984779 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d880c1fd0 con 0x55e433a21ec0 2024-03-18T22:53:15.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.584+0000 7f8d76ff5640 1 -- 172.21.15.40:0/3455984779 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f8d8803c030 con 0x55e433a21ec0 2024-03-18T22:53:15.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.585+0000 7f8d9bfd1740 10 client.28030 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:15.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.585+0000 7f8d9bfd1740 10 client.28030 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:53:15.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.585+0000 7f8d9bfd1740 20 client.28030 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:15.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.585+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/3455984779 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e433a37380 0x55e433a397e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.585+0000 7f8d9bfd1740 4 client.28030 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:53:15.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.585+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3455984779 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4339e33e0 con 0x55e433a37380 2024-03-18T22:53:15.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.585+0000 7f8d92ffd640 1 --2- 172.21.15.40:0/3455984779 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e433a37380 0x55e433a397e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.586+0000 7f8d76ff5640 10 client.28030 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:15.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.586+0000 7f8d92ffd640 1 --2- 172.21.15.40:0/3455984779 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e433a37380 0x55e433a397e0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=100f99d4a03c110 server_cookie=7340ff88938e756c in_seq=0 out_seq=0 2024-03-18T22:53:15.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.589+0000 7f8d76ff5640 1 -- 172.21.15.40:0/3455984779 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e4339e33e0 con 0x55e433a37380 2024-03-18T22:53:15.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.590+0000 7f8d76ff5640 10 client.28030 handle_command_reply: tid=0 2024-03-18T22:53:15.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.590+0000 7f8d9bfd1740 1 client.28030 shutdown 2024-03-18T22:53:15.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.590+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3455984779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a21ec0 msgr2=0x55e433a2bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.590+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/3455984779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a21ec0 0x55e433a2bcb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8d880bdca0 tx=0x7f8d88005ad0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.593+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3455984779 shutdown_connections 2024-03-18T22:53:15.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.593+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/3455984779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a27030 0x55e433a32cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.593+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/3455984779 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a21ec0 0x55e433a2bcb0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.593+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/3455984779 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e433a37380 0x55e433a397e0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.593+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/3455984779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 0x55e433a2b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.593+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3455984779 >> 172.21.15.40:0/3455984779 conn(0x55e433a1ca40 msgr2=0x55e433a1bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:15.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.594+0000 7f8d76ff5640 0 client.28030 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:15.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.594+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3455984779 shutdown_connections 2024-03-18T22:53:15.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.594+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3455984779 wait complete. 2024-03-18T22:53:15.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.594+0000 7f8d9bfd1740 20 client.28030 trim_cache size 0 max 16384 2024-03-18T22:53:15.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.630+0000 7f8d9bfd1740 1 Processor -- start 2024-03-18T22:53:15.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.633+0000 7f8d9bfd1740 1 -- start start 2024-03-18T22:53:15.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.633+0000 7f8d9bfd1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 0x55e433a203b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.634+0000 7f8d92ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 0x55e433a203b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.634+0000 7f8d9bfd1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a21ec0 0x55e433a32cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.634+0000 7f8d92ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 0x55e433a203b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52306/0 (socket says 172.21.15.40:52306) 2024-03-18T22:53:15.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.634+0000 7f8d92ffd640 1 -- 172.21.15.40:0/4088365995 learned_addr learned my addr 172.21.15.40:0/4088365995 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:15.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.634+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/4088365995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a27030 0x55e433a2a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.634+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/4088365995 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8000fdd0 con 0x55e433a27030 2024-03-18T22:53:15.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.634+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/4088365995 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80011a00 con 0x55e433a214d0 2024-03-18T22:53:15.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.634+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/4088365995 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8d800104a0 con 0x55e433a21ec0 2024-03-18T22:53:15.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.634+0000 7f8d927fc640 1 --2- 172.21.15.40:0/4088365995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a21ec0 0x55e433a32cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.634+0000 7f8d937fe640 1 --2- 172.21.15.40:0/4088365995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a27030 0x55e433a2a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.635+0000 7f8d92ffd640 1 -- 172.21.15.40:0/4088365995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a21ec0 msgr2=0x55e433a32cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.640+0000 7f8d92ffd640 1 --2- 172.21.15.40:0/4088365995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a21ec0 0x55e433a32cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.640+0000 7f8d92ffd640 1 -- 172.21.15.40:0/4088365995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a27030 msgr2=0x55e433a2a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.640+0000 7f8d92ffd640 1 --2- 172.21.15.40:0/4088365995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a27030 0x55e433a2a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.640+0000 7f8d92ffd640 1 -- 172.21.15.40:0/4088365995 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e433a2c4a0 con 0x55e433a214d0 2024-03-18T22:53:15.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.640+0000 7f8d927fc640 1 --2- 172.21.15.40:0/4088365995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a21ec0 0x55e433a32cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:53:15.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.641+0000 7f8d937fe640 1 --2- 172.21.15.40:0/4088365995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a27030 0x55e433a2a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:53:15.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.643+0000 7f8d92ffd640 1 --2- 172.21.15.40:0/4088365995 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 0x55e433a203b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8d840092a0 tx=0x7f8d84009880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=713c666623dd87e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:15.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.645+0000 7f8d78ff9640 1 -- 172.21.15.40:0/4088365995 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d840140d0 con 0x55e433a214d0 2024-03-18T22:53:15.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.645+0000 7f8d78ff9640 1 -- 172.21.15.40:0/4088365995 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d8401b7e0 con 0x55e433a214d0 2024-03-18T22:53:15.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.650+0000 7f8d78ff9640 1 -- 172.21.15.40:0/4088365995 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84012740 con 0x55e433a214d0 2024-03-18T22:53:15.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.654+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/4088365995 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 msgr2=0x55e433a203b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.654+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/4088365995 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 0x55e433a203b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8d840092a0 tx=0x7f8d84009880 comp rx=0 tx=0).stop 2024-03-18T22:53:15.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.656+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/4088365995 shutdown_connections 2024-03-18T22:53:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.656+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/4088365995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a27030 0x55e433a2a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.656+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/4088365995 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a21ec0 0x55e433a32cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.656+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/4088365995 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 0x55e433a203b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.656+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/4088365995 >> 172.21.15.40:0/4088365995 conn(0x55e433a1ca40 msgr2=0x55e433a1fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.656+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/4088365995 shutdown_connections 2024-03-18T22:53:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.656+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/4088365995 wait complete. 2024-03-18T22:53:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.657+0000 7f8d9bfd1740 1 -- start start 2024-03-18T22:53:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.657+0000 7f8d9bfd1740 1 Processor -- start 2024-03-18T22:53:15.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.658+0000 7f8d9bfd1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 0x55e433a304b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.658+0000 7f8d9bfd1740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a21ec0 0x55e433a309f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.658+0000 7f8d9bfd1740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a27030 0x55e433a31a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.658+0000 7f8d9bfd1740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80011330 con 0x55e433a21ec0 2024-03-18T22:53:15.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.658+0000 7f8d9bfd1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80010ba0 con 0x55e433a214d0 2024-03-18T22:53:15.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.658+0000 7f8d9bfd1740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8000ddb0 con 0x55e433a27030 2024-03-18T22:53:15.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.658+0000 7f8d9bfd1740 10 client.0 _ll_register_callbacks cb 0x7f8d88104070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:15.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.658+0000 7f8d937fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a27030 0x55e433a31a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.658+0000 7f8d92ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 0x55e433a304b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.658+0000 7f8d92ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 0x55e433a304b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52318/0 (socket says 172.21.15.40:52318) 2024-03-18T22:53:15.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.658+0000 7f8d92ffd640 1 -- 172.21.15.40:0/3990776303 learned_addr learned my addr 172.21.15.40:0/3990776303 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:15.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.658+0000 7f8d937fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a27030 0x55e433a31a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:41970/0 (socket says 172.21.15.40:41970) 2024-03-18T22:53:15.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.658+0000 7f8d927fc640 1 --2- 172.21.15.40:0/3990776303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a21ec0 0x55e433a309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.660+0000 7f8d927fc640 1 -- 172.21.15.40:0/3990776303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a27030 msgr2=0x55e433a31a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.660+0000 7f8d927fc640 1 --2- 172.21.15.40:0/3990776303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a27030 0x55e433a31a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.660+0000 7f8d927fc640 1 -- 172.21.15.40:0/3990776303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 msgr2=0x55e433a304b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:15.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.660+0000 7f8d927fc640 1 --2- 172.21.15.40:0/3990776303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 0x55e433a304b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.660+0000 7f8d927fc640 1 -- 172.21.15.40:0/3990776303 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e433a21110 con 0x55e433a21ec0 2024-03-18T22:53:15.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.660+0000 7f8d92ffd640 1 --2- 172.21.15.40:0/3990776303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 0x55e433a304b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:53:15.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.660+0000 7f8d927fc640 1 --2- 172.21.15.40:0/3990776303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a21ec0 0x55e433a309f0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8d88000c00 tx=0x7f8d880c9be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f0c5062c6baff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:15.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.660+0000 7f8d76ff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:53:15.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.661+0000 7f8d76ff5640 1 -- 172.21.15.40:0/3990776303 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d880fb020 con 0x55e433a21ec0 2024-03-18T22:53:15.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.661+0000 7f8d76ff5640 1 -- 172.21.15.40:0/3990776303 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8d8802c510 con 0x55e433a21ec0 2024-03-18T22:53:15.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.661+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3990776303 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d8401b7e0 con 0x55e433a21ec0 2024-03-18T22:53:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.662+0000 7f8d76ff5640 1 -- 172.21.15.40:0/3990776303 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8d880bfa10 con 0x55e433a21ec0 2024-03-18T22:53:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.663+0000 7f8d9bfd1740 10 client.18219 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.663+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3990776303 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e433a21110 con 0x55e433a21ec0 2024-03-18T22:53:15.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.664+0000 7f8d76ff5640 1 -- 172.21.15.40:0/3990776303 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d880f5160 con 0x55e433a21ec0 2024-03-18T22:53:15.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.664+0000 7f8d76ff5640 1 -- 172.21.15.40:0/3990776303 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f8d880bd590 con 0x55e433a21ec0 2024-03-18T22:53:15.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.665+0000 7f8d9bfd1740 10 client.18219 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:15.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.665+0000 7f8d9bfd1740 10 client.18219 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:53:15.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.665+0000 7f8d9bfd1740 20 client.18219 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:15.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.665+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/3990776303 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e433a9dc70 0x55e433aa0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:15.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.665+0000 7f8d9bfd1740 4 client.18219 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:53:15.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.665+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3990776303 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f8d8401b7e0 con 0x55e433a9dc70 2024-03-18T22:53:15.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.666+0000 7f8d92ffd640 1 --2- 172.21.15.40:0/3990776303 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e433a9dc70 0x55e433aa0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:15.670 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:53:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d92ffd640 1 --2- 172.21.15.40:0/3990776303 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e433a9dc70 0x55e433aa0090 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ce0fbe239ee78b8 server_cookie=413d2309a49b46c3 in_seq=0 out_seq=0 2024-03-18T22:53:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d76ff5640 10 client.18219 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d76ff5640 1 -- 172.21.15.40:0/3990776303 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f8d8401b7e0 con 0x55e433a9dc70 2024-03-18T22:53:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d76ff5640 10 client.18219 handle_command_reply: tid=0 2024-03-18T22:53:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d9bfd1740 1 client.18219 shutdown 2024-03-18T22:53:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3990776303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a21ec0 msgr2=0x55e433a309f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/3990776303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a21ec0 0x55e433a309f0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8d88000c00 tx=0x7f8d880c9be0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3990776303 shutdown_connections 2024-03-18T22:53:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/3990776303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e433a27030 0x55e433a31a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/3990776303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e433a21ec0 0x55e433a309f0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/3990776303 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e433a9dc70 0x55e433aa0090 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d9bfd1740 1 --2- 172.21.15.40:0/3990776303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e433a214d0 0x55e433a304b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3990776303 >> 172.21.15.40:0/3990776303 conn(0x55e433a1ca40 msgr2=0x55e433a20530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d76ff5640 0 client.18219 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3990776303 shutdown_connections 2024-03-18T22:53:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.669+0000 7f8d9bfd1740 1 -- 172.21.15.40:0/3990776303 wait complete. 2024-03-18T22:53:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.670+0000 7f8d9bfd1740 20 client.18219 trim_cache size 0 max 16384 2024-03-18T22:53:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.670+0000 7f8d93fff640 1 -- 172.21.15.40:0/1291698722 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8d7c0a2a00 msgr2=0x7f8d7c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.670+0000 7f8d93fff640 1 --2- 172.21.15.40:0/1291698722 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8d7c0a2a00 0x7f8d7c0a4ec0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x55e433a22720 tx=0x7f8d8c009500 comp rx=0 tx=0).stop 2024-03-18T22:53:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.671+0000 7f8d93fff640 1 -- 172.21.15.40:0/1291698722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d94108e20 msgr2=0x7f8d94110890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.671+0000 7f8d93fff640 1 --2- 172.21.15.40:0/1291698722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d94108e20 0x7f8d94110890 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009980 tx=0x7f8d88004270 comp rx=0 tx=0).stop 2024-03-18T22:53:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.671+0000 7f8d93fff640 1 -- 172.21.15.40:0/1291698722 shutdown_connections 2024-03-18T22:53:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.671+0000 7f8d93fff640 1 --2- 172.21.15.40:0/1291698722 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8d7c0a2a00 0x7f8d7c0a4ec0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.671+0000 7f8d93fff640 1 --2- 172.21.15.40:0/1291698722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d941097c0 0x7f8d94117900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.671+0000 7f8d93fff640 1 --2- 172.21.15.40:0/1291698722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d94108e20 0x7f8d94110890 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.671+0000 7f8d93fff640 1 --2- 172.21.15.40:0/1291698722 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d94072b70 0x7f8d94110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.671+0000 7f8d93fff640 1 -- 172.21.15.40:0/1291698722 >> 172.21.15.40:0/1291698722 conn(0x7f8d9406e640 msgr2=0x7f8d94070360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.671+0000 7f8d93fff640 1 -- 172.21.15.40:0/1291698722 shutdown_connections 2024-03-18T22:53:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:15.671+0000 7f8d93fff640 1 -- 172.21.15.40:0/1291698722 wait complete. 2024-03-18T22:53:16.720 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:53:16.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:53:16.720 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:53:16.793 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:16 smithi040 ceph-mon[62643]: pgmap v669: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 MiB/s wr, 3.83k op/s 2024-03-18T22:53:16.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:16 smithi113 ceph-mon[66775]: pgmap v669: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 MiB/s wr, 3.83k op/s 2024-03-18T22:53:16.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:16 smithi103 ceph-mon[66377]: pgmap v669: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 MiB/s wr, 3.83k op/s 2024-03-18T22:53:16.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.976+0000 7f7ea27fc640 1 -- 172.21.15.40:0/2774955785 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9402cc10 con 0x7f7ea4105290 2024-03-18T22:53:16.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.977+0000 7f7ea8d49640 1 -- 172.21.15.40:0/2774955785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ea4105290 msgr2=0x7f7ea4105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:16.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.977+0000 7f7ea8d49640 1 --2- 172.21.15.40:0/2774955785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ea4105290 0x7f7ea4105710 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7e94009900 tx=0x7f7e9402bc60 comp rx=0 tx=0).stop 2024-03-18T22:53:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.977+0000 7f7ea8d49640 1 -- 172.21.15.40:0/2774955785 shutdown_connections 2024-03-18T22:53:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.977+0000 7f7ea8d49640 1 --2- 172.21.15.40:0/2774955785 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ea4105290 0x7f7ea4105710 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.977+0000 7f7ea8d49640 1 --2- 172.21.15.40:0/2774955785 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ea41028e0 0x7f7ea4104d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.977+0000 7f7ea8d49640 1 --2- 172.21.15.40:0/2774955785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea40fff70 0x7f7ea41023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.977+0000 7f7ea8d49640 1 -- 172.21.15.40:0/2774955785 >> 172.21.15.40:0/2774955785 conn(0x7f7ea40fbb90 msgr2=0x7f7ea40fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.977+0000 7f7ea8d49640 1 -- 172.21.15.40:0/2774955785 shutdown_connections 2024-03-18T22:53:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.977+0000 7f7ea8d49640 1 -- 172.21.15.40:0/2774955785 wait complete. 2024-03-18T22:53:16.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.978+0000 7f7ea8d49640 1 Processor -- start 2024-03-18T22:53:16.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.978+0000 7f7ea8d49640 1 -- start start 2024-03-18T22:53:16.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.979+0000 7f7ea8d49640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ea40fff70 0x7f7ea419df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:16.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.979+0000 7f7ea8d49640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ea41028e0 0x7f7ea419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:16.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.979+0000 7f7ea8d49640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea4105290 0x7f7ea41a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:16.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.979+0000 7f7ea8d49640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea41138a0 con 0x7f7ea41028e0 2024-03-18T22:53:16.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.980+0000 7f7ea8d49640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea4113720 con 0x7f7ea4105290 2024-03-18T22:53:16.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.980+0000 7f7ea8d49640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea4113a20 con 0x7f7ea40fff70 2024-03-18T22:53:16.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.980+0000 7f7ea2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ea41028e0 0x7f7ea419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:16.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.980+0000 7f7ea2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ea41028e0 0x7f7ea419e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39578/0 (socket says 172.21.15.40:39578) 2024-03-18T22:53:16.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.980+0000 7f7ea2ffd640 1 -- 172.21.15.40:0/3629758063 learned_addr learned my addr 172.21.15.40:0/3629758063 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:16.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.980+0000 7f7ea3fff640 1 --2- 172.21.15.40:0/3629758063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea4105290 0x7f7ea41a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:16.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.981+0000 7f7ea37fe640 1 --2- 172.21.15.40:0/3629758063 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ea40fff70 0x7f7ea419df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:16.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.981+0000 7f7ea2ffd640 1 -- 172.21.15.40:0/3629758063 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ea40fff70 msgr2=0x7f7ea419df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:16.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.981+0000 7f7ea2ffd640 1 --2- 172.21.15.40:0/3629758063 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ea40fff70 0x7f7ea419df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:16.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.981+0000 7f7ea2ffd640 1 -- 172.21.15.40:0/3629758063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea4105290 msgr2=0x7f7ea41a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:16.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.981+0000 7f7ea2ffd640 1 --2- 172.21.15.40:0/3629758063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea4105290 0x7f7ea41a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:16.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.981+0000 7f7ea2ffd640 1 -- 172.21.15.40:0/3629758063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ea41a5b50 con 0x7f7ea41028e0 2024-03-18T22:53:16.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.981+0000 7f7ea2ffd640 1 --2- 172.21.15.40:0/3629758063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ea41028e0 0x7f7ea419e470 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800e9f0 tx=0x7f7e9800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:16.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.989+0000 7f7ea0ff9640 1 -- 172.21.15.40:0/3629758063 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9800cd80 con 0x7f7ea41028e0 2024-03-18T22:53:16.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.989+0000 7f7ea0ff9640 1 -- 172.21.15.40:0/3629758063 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e980108e0 con 0x7f7ea41028e0 2024-03-18T22:53:16.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.989+0000 7f7ea8d49640 1 -- 172.21.15.40:0/3629758063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ea41a5e40 con 0x7f7ea41028e0 2024-03-18T22:53:16.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.989+0000 7f7ea0ff9640 1 -- 172.21.15.40:0/3629758063 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e98018850 con 0x7f7ea41028e0 2024-03-18T22:53:16.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.989+0000 7f7ea8d49640 1 -- 172.21.15.40:0/3629758063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ea41a6380 con 0x7f7ea41028e0 2024-03-18T22:53:16.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.994+0000 7f7eacb8b740 1 Processor -- start 2024-03-18T22:53:16.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.995+0000 7f7ea0ff9640 1 -- 172.21.15.40:0/3629758063 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7e98010a90 con 0x7f7ea41028e0 2024-03-18T22:53:16.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.996+0000 7f7ea0ff9640 1 --2- 172.21.15.40:0/3629758063 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7e8c0a29d0 0x7f7e8c0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:16.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.996+0000 7f7eacb8b740 1 -- start start 2024-03-18T22:53:16.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.996+0000 7f7ea0ff9640 1 -- 172.21.15.40:0/3629758063 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7e980cf180 con 0x7f7ea41028e0 2024-03-18T22:53:16.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.997+0000 7f7eacb8b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7f580 0x55a436e7f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:16.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.997+0000 7f7eacb8b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7ff70 0x55a436e84a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:16.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.997+0000 7f7eacb8b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e850e0 0x55a436e87570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:16.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.997+0000 7f7eacb8b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a436de4160 con 0x55a436e850e0 2024-03-18T22:53:16.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.997+0000 7f7eacb8b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a436d8ce10 con 0x55a436e7f580 2024-03-18T22:53:16.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.997+0000 7f7eacb8b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55a436de0200 con 0x55a436e7ff70 2024-03-18T22:53:16.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.997+0000 7f7ea37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7f580 0x55a436e7f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:16.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.997+0000 7f7ea2ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7ff70 0x55a436e84a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:16.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.997+0000 7f7ea2ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7ff70 0x55a436e84a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:42002/0 (socket says 172.21.15.40:42002) 2024-03-18T22:53:16.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.997+0000 7f7ea2ffd640 1 -- 172.21.15.40:0/929921012 learned_addr learned my addr 172.21.15.40:0/929921012 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:16.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.998+0000 7f7ea3fff640 1 --2- 172.21.15.40:0/3629758063 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7e8c0a29d0 0x7f7e8c0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:16.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.998+0000 7f7ea2ffd640 1 -- 172.21.15.40:0/929921012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7f580 msgr2=0x55a436e7f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:16.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.998+0000 7f7ea2ffd640 1 --2- 172.21.15.40:0/929921012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7f580 0x55a436e7f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:16.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.998+0000 7f7ea2ffd640 1 -- 172.21.15.40:0/929921012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e850e0 msgr2=0x55a436e87570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:16.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.998+0000 7f7ea2ffd640 1 --2- 172.21.15.40:0/929921012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e850e0 0x55a436e87570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:16.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.998+0000 7f7ea2ffd640 1 -- 172.21.15.40:0/929921012 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a436bc07e0 con 0x55a436e7ff70 2024-03-18T22:53:17.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.999+0000 7f7ea3fff640 1 --2- 172.21.15.40:0/3629758063 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7e8c0a29d0 0x7f7e8c0a4e90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7e94006010 tx=0x7f7e9402c600 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:17.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.999+0000 7f7ea2ffd640 1 --2- 172.21.15.40:0/929921012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7ff70 0x55a436e84a70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7e9809b660 tx=0x7f7e98010de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2e707d12fcb9e3c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:17.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:16.999+0000 7f7e89ffb640 1 -- 172.21.15.40:0/929921012 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e980b4070 con 0x55a436e7ff70 2024-03-18T22:53:17.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.000+0000 7f7e89ffb640 1 -- 172.21.15.40:0/929921012 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e9809b6d0 con 0x55a436e7ff70 2024-03-18T22:53:17.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.000+0000 7f7e89ffb640 1 -- 172.21.15.40:0/929921012 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9809ab20 con 0x55a436e7ff70 2024-03-18T22:53:17.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.001+0000 7f7eacb8b740 1 -- 172.21.15.40:0/929921012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7ff70 msgr2=0x55a436e84a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:17.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.001+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/929921012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7ff70 0x55a436e84a70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7e9809b660 tx=0x7f7e98010de0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.003+0000 7f7eacb8b740 1 -- 172.21.15.40:0/929921012 shutdown_connections 2024-03-18T22:53:17.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.003+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/929921012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e850e0 0x55a436e87570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.003+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/929921012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7ff70 0x55a436e84a70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.003+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/929921012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7f580 0x55a436e7f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.003+0000 7f7eacb8b740 1 -- 172.21.15.40:0/929921012 >> 172.21.15.40:0/929921012 conn(0x55a436e7aaa0 msgr2=0x55a436e7aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:17.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.004+0000 7f7eacb8b740 1 -- 172.21.15.40:0/929921012 shutdown_connections 2024-03-18T22:53:17.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.004+0000 7f7eacb8b740 1 -- 172.21.15.40:0/929921012 wait complete. 2024-03-18T22:53:17.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.004+0000 7f7eacb8b740 1 -- start start 2024-03-18T22:53:17.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.004+0000 7f7eacb8b740 1 Processor -- start 2024-03-18T22:53:17.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.005+0000 7f7eacb8b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7f580 0x55a436e9aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.005+0000 7f7eacb8b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7ff70 0x55a436e9aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.005+0000 7f7eacb8b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e850e0 0x55a436ea1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.005+0000 7f7eacb8b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a436e990a0 con 0x55a436e7ff70 2024-03-18T22:53:17.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.005+0000 7f7eacb8b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a436da01a0 con 0x55a436e850e0 2024-03-18T22:53:17.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.005+0000 7f7eacb8b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55a436e99220 con 0x55a436e7f580 2024-03-18T22:53:17.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.005+0000 7f7eacb8b740 10 client.0 _ll_register_callbacks cb 0x55a436af7c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:17.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.005+0000 7f7ea2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7ff70 0x55a436e9aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.005+0000 7f7ea37fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7f580 0x55a436e9aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.005+0000 7f7ea2ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7ff70 0x55a436e9aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39606/0 (socket says 172.21.15.40:39606) 2024-03-18T22:53:17.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.005+0000 7f7ea2ffd640 1 -- 172.21.15.40:0/809559923 learned_addr learned my addr 172.21.15.40:0/809559923 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:17.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.005+0000 7f7ea37fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7f580 0x55a436e9aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:42006/0 (socket says 172.21.15.40:42006) 2024-03-18T22:53:17.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.005+0000 7f7ea3fff640 1 --2- 172.21.15.40:0/809559923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e850e0 0x55a436ea1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.006+0000 7f7ea2ffd640 1 -- 172.21.15.40:0/809559923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7f580 msgr2=0x55a436e9aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:17.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.006+0000 7f7ea2ffd640 1 --2- 172.21.15.40:0/809559923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7f580 0x55a436e9aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.006+0000 7f7ea2ffd640 1 -- 172.21.15.40:0/809559923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e850e0 msgr2=0x55a436ea1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:17.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.006+0000 7f7ea2ffd640 1 --2- 172.21.15.40:0/809559923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e850e0 0x55a436ea1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.006+0000 7f7ea2ffd640 1 -- 172.21.15.40:0/809559923 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a436c20f70 con 0x55a436e7ff70 2024-03-18T22:53:17.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.006+0000 7f7ea2ffd640 1 --2- 172.21.15.40:0/809559923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7ff70 0x55a436e9aff0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7e980a3940 tx=0x7f7e980a3ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c30360351d9c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:17.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.006+0000 7f7e82ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:53:17.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.006+0000 7f7e82ffd640 1 -- 172.21.15.40:0/809559923 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e980b8020 con 0x55a436e7ff70 2024-03-18T22:53:17.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.006+0000 7f7e82ffd640 1 -- 172.21.15.40:0/809559923 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7e98010630 con 0x55a436e7ff70 2024-03-18T22:53:17.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.006+0000 7f7eacb8b740 1 -- 172.21.15.40:0/809559923 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e9809b6d0 con 0x55a436e7ff70 2024-03-18T22:53:17.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.007+0000 7f7e82ffd640 1 -- 172.21.15.40:0/809559923 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7e980ae050 con 0x55a436e7ff70 2024-03-18T22:53:17.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.007+0000 7f7eacb8b740 10 client.18228 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:17.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.007+0000 7f7eacb8b740 1 -- 172.21.15.40:0/809559923 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a436ab8170 con 0x55a436e7ff70 2024-03-18T22:53:17.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.008+0000 7f7e82ffd640 1 -- 172.21.15.40:0/809559923 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e9809b6d0 con 0x55a436e7ff70 2024-03-18T22:53:17.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.008+0000 7f7e82ffd640 1 -- 172.21.15.40:0/809559923 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f7e980afab0 con 0x55a436e7ff70 2024-03-18T22:53:17.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.008+0000 7f7eacb8b740 10 client.18228 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:17.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.008+0000 7f7eacb8b740 10 client.18228 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:53:17.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.008+0000 7f7eacb8b740 20 client.18228 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:17.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.008+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/809559923 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a436ea64b0 0x55a436ea88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.008+0000 7f7eacb8b740 4 client.18228 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:53:17.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.008+0000 7f7eacb8b740 1 -- 172.21.15.40:0/809559923 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a436df16a0 con 0x55a436ea64b0 2024-03-18T22:53:17.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.008+0000 7f7ea37fe640 1 --2- 172.21.15.40:0/809559923 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a436ea64b0 0x55a436ea88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.009+0000 7f7ea37fe640 1 --2- 172.21.15.40:0/809559923 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a436ea64b0 0x55a436ea88a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d216db9a6d15c4e server_cookie=6ee26685e7e9eeb8 in_seq=0 out_seq=0 2024-03-18T22:53:17.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.009+0000 7f7e82ffd640 10 client.18228 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:17.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.010+0000 7f7e82ffd640 1 -- 172.21.15.40:0/809559923 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a436df16a0 con 0x55a436ea64b0 2024-03-18T22:53:17.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.010+0000 7f7e82ffd640 10 client.18228 handle_command_reply: tid=0 2024-03-18T22:53:17.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.010+0000 7f7eacb8b740 1 client.18228 shutdown 2024-03-18T22:53:17.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.010+0000 7f7eacb8b740 1 -- 172.21.15.40:0/809559923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7ff70 msgr2=0x55a436e9aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:17.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.010+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/809559923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7ff70 0x55a436e9aff0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7e980a3940 tx=0x7f7e980a3ef0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.010+0000 7f7eacb8b740 1 -- 172.21.15.40:0/809559923 shutdown_connections 2024-03-18T22:53:17.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.010+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/809559923 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a436ea64b0 0x55a436ea88a0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.010+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/809559923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e850e0 0x55a436ea1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.010+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/809559923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7ff70 0x55a436e9aff0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.010+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/809559923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7f580 0x55a436e9aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.010+0000 7f7eacb8b740 1 -- 172.21.15.40:0/809559923 >> 172.21.15.40:0/809559923 conn(0x55a436e7aaa0 msgr2=0x55a436e79d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:17.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.011+0000 7f7e82ffd640 0 client.18228 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:17.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.011+0000 7f7eacb8b740 1 -- 172.21.15.40:0/809559923 shutdown_connections 2024-03-18T22:53:17.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.011+0000 7f7eacb8b740 1 -- 172.21.15.40:0/809559923 wait complete. 2024-03-18T22:53:17.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.011+0000 7f7eacb8b740 20 client.18228 trim_cache size 0 max 16384 2024-03-18T22:53:17.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.054+0000 7f7eacb8b740 1 Processor -- start 2024-03-18T22:53:17.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.054+0000 7f7eacb8b740 1 -- start start 2024-03-18T22:53:17.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.054+0000 7f7eacb8b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7aaa0 0x55a436e7e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.054+0000 7f7eacb8b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7f580 0x55a436ea1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.054+0000 7f7eacb8b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7ff70 0x55a436ead060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.054+0000 7f7eacb8b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea41a81c0 con 0x55a436e7aaa0 2024-03-18T22:53:17.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.054+0000 7f7eacb8b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea41a9df0 con 0x55a436e7ff70 2024-03-18T22:53:17.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.054+0000 7f7eacb8b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea41a8890 con 0x55a436e7f580 2024-03-18T22:53:17.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.055+0000 7f7ea3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7ff70 0x55a436ead060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.055+0000 7f7ea3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7ff70 0x55a436ead060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52354/0 (socket says 172.21.15.40:52354) 2024-03-18T22:53:17.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.055+0000 7f7ea3fff640 1 -- 172.21.15.40:0/1501082414 learned_addr learned my addr 172.21.15.40:0/1501082414 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:17.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.055+0000 7f7ea2ffd640 1 --2- 172.21.15.40:0/1501082414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7f580 0x55a436ea1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.057+0000 7f7ea3fff640 1 -- 172.21.15.40:0/1501082414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7f580 msgr2=0x55a436ea1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:17.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.057+0000 7f7ea3fff640 1 --2- 172.21.15.40:0/1501082414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7f580 0x55a436ea1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.057+0000 7f7ea3fff640 1 -- 172.21.15.40:0/1501082414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7aaa0 msgr2=0x55a436e7e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:17.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.057+0000 7f7ea3fff640 1 --2- 172.21.15.40:0/1501082414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7aaa0 0x55a436e7e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.057+0000 7f7ea3fff640 1 -- 172.21.15.40:0/1501082414 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a436ea8c10 con 0x55a436e7ff70 2024-03-18T22:53:17.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.059+0000 7f7ea3fff640 1 --2- 172.21.15.40:0/1501082414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7ff70 0x55a436ead060 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7e94009900 tx=0x7f7e9403f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f04a644186de13e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:17.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.060+0000 7f7e897fa640 1 -- 172.21.15.40:0/1501082414 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e94050070 con 0x55a436e7ff70 2024-03-18T22:53:17.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.060+0000 7f7e897fa640 1 -- 172.21.15.40:0/1501082414 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e940054f0 con 0x55a436e7ff70 2024-03-18T22:53:17.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.060+0000 7f7e897fa640 1 -- 172.21.15.40:0/1501082414 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e94004ae0 con 0x55a436e7ff70 2024-03-18T22:53:17.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.061+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1501082414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7ff70 msgr2=0x55a436ead060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:17.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.061+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/1501082414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7ff70 0x55a436ead060 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7e94009900 tx=0x7f7e9403f000 comp rx=0 tx=0).stop 2024-03-18T22:53:17.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.063+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1501082414 shutdown_connections 2024-03-18T22:53:17.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.063+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/1501082414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7ff70 0x55a436ead060 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.064+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/1501082414 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7f580 0x55a436ea1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.064+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/1501082414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7aaa0 0x55a436e7e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.064+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1501082414 >> 172.21.15.40:0/1501082414 conn(0x55a436e80d50 msgr2=0x55a436e7dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:17.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.065+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1501082414 shutdown_connections 2024-03-18T22:53:17.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.066+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1501082414 wait complete. 2024-03-18T22:53:17.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.066+0000 7f7eacb8b740 1 -- start start 2024-03-18T22:53:17.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.066+0000 7f7eacb8b740 1 Processor -- start 2024-03-18T22:53:17.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.068+0000 7f7eacb8b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7aaa0 0x55a436ea1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.069+0000 7f7ea37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7aaa0 0x55a436ea1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.069+0000 7f7ea37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7aaa0 0x55a436ea1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52366/0 (socket says 172.21.15.40:52366) 2024-03-18T22:53:17.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.069+0000 7f7ea37fe640 1 -- 172.21.15.40:0/1728621891 learned_addr learned my addr 172.21.15.40:0/1728621891 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:17.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.069+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/1728621891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7f580 0x55a436ea0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.069+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/1728621891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7ff70 0x55a436ea07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.069+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1728621891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea41a9720 con 0x55a436e7f580 2024-03-18T22:53:17.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.069+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1728621891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea41a8f90 con 0x55a436e7aaa0 2024-03-18T22:53:17.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.069+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1728621891 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea40735f0 con 0x55a436e7ff70 2024-03-18T22:53:17.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.070+0000 7f7eacb8b740 10 client.0 _ll_register_callbacks cb 0x7f7e980b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:17.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.070+0000 7f7ea2ffd640 1 --2- 172.21.15.40:0/1728621891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7f580 0x55a436ea0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.071+0000 7f7ea37fe640 1 -- 172.21.15.40:0/1728621891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7ff70 msgr2=0x55a436ea07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:17.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.071+0000 7f7ea37fe640 1 --2- 172.21.15.40:0/1728621891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7ff70 0x55a436ea07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.071+0000 7f7ea37fe640 1 -- 172.21.15.40:0/1728621891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7f580 msgr2=0x55a436ea0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:17.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.071+0000 7f7ea37fe640 1 --2- 172.21.15.40:0/1728621891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7f580 0x55a436ea0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.071+0000 7f7ea37fe640 1 -- 172.21.15.40:0/1728621891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a436ea0ce0 con 0x55a436e7aaa0 2024-03-18T22:53:17.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.074+0000 7f7ea37fe640 1 --2- 172.21.15.40:0/1728621891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7aaa0 0x55a436ea1be0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55a436ea9010 tx=0x7f7e9c00bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf48388f5fd52d17 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:17.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.074+0000 7f7e82ffd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:53:17.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.075+0000 7f7e82ffd640 1 -- 172.21.15.40:0/1728621891 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c01ad50 con 0x55a436e7aaa0 2024-03-18T22:53:17.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.075+0000 7f7e82ffd640 1 -- 172.21.15.40:0/1728621891 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7e9c01d150 con 0x55a436e7aaa0 2024-03-18T22:53:17.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.076+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1728621891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e940054f0 con 0x55a436e7aaa0 2024-03-18T22:53:17.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.077+0000 7f7e82ffd640 1 -- 172.21.15.40:0/1728621891 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7e9c01d350 con 0x55a436e7aaa0 2024-03-18T22:53:17.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.077+0000 7f7eacb8b740 10 client.28064 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:17.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.077+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1728621891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a436f0d500 con 0x55a436e7aaa0 2024-03-18T22:53:17.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.078+0000 7f7e82ffd640 1 -- 172.21.15.40:0/1728621891 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e940054f0 con 0x55a436e7aaa0 2024-03-18T22:53:17.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.078+0000 7f7e82ffd640 1 -- 172.21.15.40:0/1728621891 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f7e9c026c70 con 0x55a436e7aaa0 2024-03-18T22:53:17.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.078+0000 7f7eacb8b740 10 client.28064 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:17.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.079+0000 7f7eacb8b740 10 client.28064 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:53:17.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.079+0000 7f7eacb8b740 20 client.28064 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:17.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.079+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/1728621891 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a436e850e0 0x55a436f0ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.079+0000 7f7eacb8b740 4 client.28064 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:53:17.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.079+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1728621891 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a436deb9f0 con 0x55a436e850e0 2024-03-18T22:53:17.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.079+0000 7f7ea2ffd640 1 --2- 172.21.15.40:0/1728621891 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a436e850e0 0x55a436f0ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.084+0000 7f7ea2ffd640 1 --2- 172.21.15.40:0/1728621891 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a436e850e0 0x55a436f0ff00 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51bf954eaac9aea4 server_cookie=c3777ce72577a8c3 in_seq=0 out_seq=0 2024-03-18T22:53:17.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.084+0000 7f7e82ffd640 10 client.28064 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:17.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.094+0000 7f7e82ffd640 1 -- 172.21.15.40:0/1728621891 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a436deb9f0 con 0x55a436e850e0 2024-03-18T22:53:17.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.094+0000 7f7e82ffd640 10 client.28064 handle_command_reply: tid=0 2024-03-18T22:53:17.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.094+0000 7f7eacb8b740 1 client.28064 shutdown 2024-03-18T22:53:17.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.095+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1728621891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7aaa0 msgr2=0x55a436ea1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:17.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.095+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/1728621891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7aaa0 0x55a436ea1be0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55a436ea9010 tx=0x7f7e9c00bd30 comp rx=0 tx=0).stop 2024-03-18T22:53:17.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.098+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1728621891 shutdown_connections 2024-03-18T22:53:17.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.098+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/1728621891 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a436e7ff70 0x55a436ea07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.098+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/1728621891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a436e7f580 0x55a436ea0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.098+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/1728621891 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55a436e850e0 0x55a436f0ff00 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.098+0000 7f7eacb8b740 1 --2- 172.21.15.40:0/1728621891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a436e7aaa0 0x55a436ea1be0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.098+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1728621891 >> 172.21.15.40:0/1728621891 conn(0x55a436e80d50 msgr2=0x55a436e7e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:17.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.098+0000 7f7e82ffd640 0 client.28064 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:17.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.098+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1728621891 shutdown_connections 2024-03-18T22:53:17.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.098+0000 7f7eacb8b740 1 -- 172.21.15.40:0/1728621891 wait complete. 2024-03-18T22:53:17.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.098+0000 7f7eacb8b740 20 client.28064 trim_cache size 0 max 16384 2024-03-18T22:53:17.100 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:53:17.100 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:53:17.100 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "6a7b3721-c0a7-4062-9ca5-671585e8609b", 2024-03-18T22:53:17.100 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:53:17.100 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:53:17.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.104+0000 7f7ea8d49640 1 -- 172.21.15.40:0/3629758063 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7e8c0a29d0 msgr2=0x7f7e8c0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:17.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.104+0000 7f7ea8d49640 1 --2- 172.21.15.40:0/3629758063 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7e8c0a29d0 0x7f7e8c0a4e90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7e94006010 tx=0x7f7e9402c600 comp rx=0 tx=0).stop 2024-03-18T22:53:17.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.104+0000 7f7ea8d49640 1 -- 172.21.15.40:0/3629758063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ea41028e0 msgr2=0x7f7ea419e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:17.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.104+0000 7f7ea8d49640 1 --2- 172.21.15.40:0/3629758063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ea41028e0 0x7f7ea419e470 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800e9f0 tx=0x7f7e9800eeb0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.105+0000 7f7ea8d49640 1 -- 172.21.15.40:0/3629758063 shutdown_connections 2024-03-18T22:53:17.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.105+0000 7f7ea8d49640 1 --2- 172.21.15.40:0/3629758063 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7e8c0a29d0 0x7f7e8c0a4e90 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.105+0000 7f7ea8d49640 1 --2- 172.21.15.40:0/3629758063 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7ea4105290 0x7f7ea41a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.105+0000 7f7ea8d49640 1 --2- 172.21.15.40:0/3629758063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ea41028e0 0x7f7ea419e470 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.105+0000 7f7ea8d49640 1 --2- 172.21.15.40:0/3629758063 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7ea40fff70 0x7f7ea419df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.105+0000 7f7ea8d49640 1 -- 172.21.15.40:0/3629758063 >> 172.21.15.40:0/3629758063 conn(0x7f7ea40fbb90 msgr2=0x7f7ea41008f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.106+0000 7f7ea8d49640 1 -- 172.21.15.40:0/3629758063 shutdown_connections 2024-03-18T22:53:17.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.107+0000 7f7ea8d49640 1 -- 172.21.15.40:0/3629758063 wait complete. 2024-03-18T22:53:17.132 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:53:17.132 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:53:17.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.422+0000 7f8fd48f7640 1 -- 172.21.15.40:0/1634360811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fd0108ef0 msgr2=0x7f8fd0109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:17.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.422+0000 7f8fd48f7640 1 --2- 172.21.15.40:0/1634360811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fd0108ef0 0x7f8fd0109370 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009900 tx=0x7f8fc402bc60 comp rx=0 tx=0).stop 2024-03-18T22:53:17.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.422+0000 7f8fd48f7640 1 -- 172.21.15.40:0/1634360811 shutdown_connections 2024-03-18T22:53:17.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.422+0000 7f8fd48f7640 1 --2- 172.21.15.40:0/1634360811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fd01098b0 0x7f8fd010e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.422+0000 7f8fd48f7640 1 --2- 172.21.15.40:0/1634360811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fd0108ef0 0x7f8fd0109370 secure :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009900 tx=0x7f8fc402bc60 comp rx=0 tx=0).stop 2024-03-18T22:53:17.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.422+0000 7f8fd48f7640 1 --2- 172.21.15.40:0/1634360811 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fd0072c30 0x7f8fd0073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.422+0000 7f8fd48f7640 1 -- 172.21.15.40:0/1634360811 >> 172.21.15.40:0/1634360811 conn(0x7f8fd006e700 msgr2=0x7f8fd0070b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:17.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.424+0000 7f8fd48f7640 1 -- 172.21.15.40:0/1634360811 shutdown_connections 2024-03-18T22:53:17.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.427+0000 7f8fd48f7640 1 -- 172.21.15.40:0/1634360811 wait complete. 2024-03-18T22:53:17.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.428+0000 7f8fd48f7640 1 Processor -- start 2024-03-18T22:53:17.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.428+0000 7f8fd48f7640 1 -- start start 2024-03-18T22:53:17.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.428+0000 7f8fd48f7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fd0072c30 0x7f8fd01a69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.428+0000 7f8fd48f7640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fd01098b0 0x7f8fd01a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.428+0000 7f8fd48f7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fd01abf70 0x7f8fd01ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.428+0000 7f8fd48f7640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd011c0f0 con 0x7f8fd01abf70 2024-03-18T22:53:17.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.428+0000 7f8fd48f7640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd011bf70 con 0x7f8fd0072c30 2024-03-18T22:53:17.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.428+0000 7f8fd48f7640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd011c270 con 0x7f8fd01098b0 2024-03-18T22:53:17.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.428+0000 7f8fced9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fd0072c30 0x7f8fd01a69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.428+0000 7f8fced9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fd0072c30 0x7f8fd01a69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:52382/0 (socket says 172.21.15.40:52382) 2024-03-18T22:53:17.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.429+0000 7f8fced9d640 1 -- 172.21.15.40:0/4009206505 learned_addr learned my addr 172.21.15.40:0/4009206505 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:17.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.429+0000 7f8fcf59e640 1 --2- 172.21.15.40:0/4009206505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fd01abf70 0x7f8fd01ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.430+0000 7f8fcf59e640 1 -- 172.21.15.40:0/4009206505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fd01098b0 msgr2=0x7f8fd01a6f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T22:53:17.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.430+0000 7f8fcf59e640 1 --2- 172.21.15.40:0/4009206505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fd01098b0 0x7f8fd01a6f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.430+0000 7f8fcf59e640 1 -- 172.21.15.40:0/4009206505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fd0072c30 msgr2=0x7f8fd01a69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:17.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.430+0000 7f8fcf59e640 1 --2- 172.21.15.40:0/4009206505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fd0072c30 0x7f8fd01a69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.430+0000 7f8fcf59e640 1 -- 172.21.15.40:0/4009206505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd01ae9d0 con 0x7f8fd01abf70 2024-03-18T22:53:17.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.431+0000 7f8fcf59e640 1 --2- 172.21.15.40:0/4009206505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fd01abf70 0x7f8fd01ae360 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009fd0 tx=0x7f8fc800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:17.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.431+0000 7f8fb7fff640 1 -- 172.21.15.40:0/4009206505 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc800ce20 con 0x7f8fd01abf70 2024-03-18T22:53:17.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.431+0000 7f8fd48f7640 1 -- 172.21.15.40:0/4009206505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd01aecc0 con 0x7f8fd01abf70 2024-03-18T22:53:17.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.431+0000 7f8fd48f7640 1 -- 172.21.15.40:0/4009206505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd01af240 con 0x7f8fd01abf70 2024-03-18T22:53:17.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.432+0000 7f8fb7fff640 1 -- 172.21.15.40:0/4009206505 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fc8005020 con 0x7f8fd01abf70 2024-03-18T22:53:17.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.432+0000 7f8fb7fff640 1 -- 172.21.15.40:0/4009206505 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8010690 con 0x7f8fd01abf70 2024-03-18T22:53:17.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.436+0000 7f8fd7dbf740 1 Processor -- start 2024-03-18T22:53:17.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.437+0000 7f8fd7dbf740 1 -- start start 2024-03-18T22:53:17.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.438+0000 7f8fd7dbf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fdec10 0x556ed1fdf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.438+0000 7f8fd7dbf740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdf600 0x556ed1fe4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.438+0000 7f8fd7dbf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fe4770 0x556ed1fe6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.438+0000 7f8fd7dbf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556ed1f43500 con 0x556ed1fe4770 2024-03-18T22:53:17.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.438+0000 7f8fd7dbf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556ed1eebbe0 con 0x556ed1fdec10 2024-03-18T22:53:17.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.438+0000 7f8fd7dbf740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556ed1f3f5a0 con 0x556ed1fdf600 2024-03-18T22:53:17.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.438+0000 7f8fb7fff640 1 -- 172.21.15.40:0/4009206505 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8fc8004a40 con 0x7f8fd01abf70 2024-03-18T22:53:17.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.438+0000 7f8fced9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdf600 0x556ed1fe4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.438+0000 7f8fced9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdf600 0x556ed1fe4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:42022/0 (socket says 172.21.15.40:42022) 2024-03-18T22:53:17.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.438+0000 7f8fced9d640 1 -- 172.21.15.40:0/801410309 learned_addr learned my addr 172.21.15.40:0/801410309 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:17.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.438+0000 7f8fce59c640 1 --2- 172.21.15.40:0/801410309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fdec10 0x556ed1fdf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.438+0000 7f8fce59c640 1 --2- 172.21.15.40:0/801410309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fe4770 0x556ed1fe6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.439+0000 7f8fced9d640 1 -- 172.21.15.40:0/801410309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fdec10 msgr2=0x556ed1fdf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:17.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.439+0000 7f8fced9d640 1 --2- 172.21.15.40:0/801410309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fdec10 0x556ed1fdf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.439+0000 7f8fced9d640 1 -- 172.21.15.40:0/801410309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fe4770 msgr2=0x556ed1fe6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:17.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.439+0000 7f8fced9d640 1 --2- 172.21.15.40:0/801410309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fe4770 0x556ed1fe6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.439+0000 7f8fced9d640 1 -- 172.21.15.40:0/801410309 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ed1f0d000 con 0x556ed1fdf600 2024-03-18T22:53:17.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.439+0000 7f8fce59c640 1 --2- 172.21.15.40:0/801410309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fe4770 0x556ed1fe6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:53:17.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.440+0000 7f8fb7fff640 1 --2- 172.21.15.40:0/4009206505 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8fb80a2a00 0x7f8fb80a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.440+0000 7f8fb7fff640 1 -- 172.21.15.40:0/4009206505 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8fc80cf4f0 con 0x7f8fd01abf70 2024-03-18T22:53:17.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.440+0000 7f8fce59c640 1 --2- 172.21.15.40:0/4009206505 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8fb80a2a00 0x7f8fb80a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.442+0000 7f8fced9d640 1 --2- 172.21.15.40:0/801410309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdf600 0x556ed1fe4100 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0007850 tx=0x7f8fc0007d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c54cc1df4f8b629 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:17.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.444+0000 7f8fb57fa640 1 -- 172.21.15.40:0/801410309 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc000d950 con 0x556ed1fdf600 2024-03-18T22:53:17.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.444+0000 7f8fce59c640 1 --2- 172.21.15.40:0/4009206505 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8fb80a2a00 0x7f8fb80a4ec0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8fc402c1f0 tx=0x7f8fc4000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:17.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.444+0000 7f8fb57fa640 1 -- 172.21.15.40:0/801410309 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fc0018070 con 0x556ed1fdf600 2024-03-18T22:53:17.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.445+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/801410309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdf600 msgr2=0x556ed1fe4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:17.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.445+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/801410309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdf600 0x556ed1fe4100 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0007850 tx=0x7f8fc0007d10 comp rx=0 tx=0).stop 2024-03-18T22:53:17.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.449+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/801410309 shutdown_connections 2024-03-18T22:53:17.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.449+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/801410309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fe4770 0x556ed1fe6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.449+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/801410309 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdf600 0x556ed1fe4100 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.449+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/801410309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fdec10 0x556ed1fdf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.449+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/801410309 >> 172.21.15.40:0/801410309 conn(0x556ed1fda180 msgr2=0x556ed1fda580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:17.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.452+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/801410309 shutdown_connections 2024-03-18T22:53:17.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.452+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/801410309 wait complete. 2024-03-18T22:53:17.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.452+0000 7f8fd7dbf740 1 -- start start 2024-03-18T22:53:17.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.453+0000 7f8fd7dbf740 1 Processor -- start 2024-03-18T22:53:17.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.453+0000 7f8fd7dbf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fdec10 0x556ed1ffa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.453+0000 7f8fd7dbf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fdf600 0x556ed1ffa820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.453+0000 7f8fd7dbf740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fe4770 0x556ed2001860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.453+0000 7f8fd7dbf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556ed1fe77c0 con 0x556ed1fdec10 2024-03-18T22:53:17.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.453+0000 7f8fd7dbf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556ed1eff160 con 0x556ed1fdf600 2024-03-18T22:53:17.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.453+0000 7f8fd7dbf740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556ed1fe7940 con 0x556ed1fe4770 2024-03-18T22:53:17.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.453+0000 7f8fd7dbf740 10 client.0 _ll_register_callbacks cb 0x556ed1f3ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:17.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.453+0000 7f8fced9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fdec10 0x556ed1ffa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.453+0000 7f8fced9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fdec10 0x556ed1ffa2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39656/0 (socket says 172.21.15.40:39656) 2024-03-18T22:53:17.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.453+0000 7f8fced9d640 1 -- 172.21.15.40:0/1021236228 learned_addr learned my addr 172.21.15.40:0/1021236228 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:17.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.453+0000 7f8fcf59e640 1 --2- 172.21.15.40:0/1021236228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fe4770 0x556ed2001860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.454+0000 7f8fce59c640 1 --2- 172.21.15.40:0/1021236228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fdf600 0x556ed1ffa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.454+0000 7f8fced9d640 1 -- 172.21.15.40:0/1021236228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fe4770 msgr2=0x556ed2001860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:17.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.454+0000 7f8fced9d640 1 --2- 172.21.15.40:0/1021236228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fe4770 0x556ed2001860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.454+0000 7f8fced9d640 1 -- 172.21.15.40:0/1021236228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fdf600 msgr2=0x556ed1ffa820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:17.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.454+0000 7f8fced9d640 1 --2- 172.21.15.40:0/1021236228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fdf600 0x556ed1ffa820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.454+0000 7f8fced9d640 1 -- 172.21.15.40:0/1021236228 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ed1d217d0 con 0x556ed1fdec10 2024-03-18T22:53:17.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.454+0000 7f8fced9d640 1 --2- 172.21.15.40:0/1021236228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fdec10 0x556ed1ffa2e0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000eb90 tx=0x7f8fc000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a99711a3d3b47ee8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:17.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.454+0000 7f8fb2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:53:17.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.457+0000 7f8fb2ff5640 1 -- 172.21.15.40:0/1021236228 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0022070 con 0x556ed1fdec10 2024-03-18T22:53:17.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.457+0000 7f8fb2ff5640 1 -- 172.21.15.40:0/1021236228 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8fc0004240 con 0x556ed1fdec10 2024-03-18T22:53:17.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.457+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/1021236228 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556ed1f50a40 con 0x556ed1fdec10 2024-03-18T22:53:17.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.458+0000 7f8fb2ff5640 1 -- 172.21.15.40:0/1021236228 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8fc002c050 con 0x556ed1fdec10 2024-03-18T22:53:17.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.458+0000 7f8fd7dbf740 10 client.18240 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:17.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.458+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/1021236228 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556ed1c573f0 con 0x556ed1fdec10 2024-03-18T22:53:17.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.458+0000 7f8fb2ff5640 1 -- 172.21.15.40:0/1021236228 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x556ed1f50a40 con 0x556ed1fdec10 2024-03-18T22:53:17.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.458+0000 7f8fb2ff5640 1 -- 172.21.15.40:0/1021236228 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f8fc0003820 con 0x556ed1fdec10 2024-03-18T22:53:17.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.459+0000 7f8fce59c640 1 --2- 172.21.15.40:0/1021236228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fdf600 0x556ed1ffa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:53:17.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.459+0000 7f8fd7dbf740 10 client.18240 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:17.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.459+0000 7f8fd7dbf740 10 client.18240 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:53:17.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.459+0000 7f8fd7dbf740 20 client.18240 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:17.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.459+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/1021236228 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed1febb80 0x556ed1fee000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.459+0000 7f8fd7dbf740 4 client.18240 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:53:17.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.459+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/1021236228 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ed1f4ad90 con 0x556ed1febb80 2024-03-18T22:53:17.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.463+0000 7f8fce59c640 1 --2- 172.21.15.40:0/1021236228 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed1febb80 0x556ed1fee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.470+0000 7f8fce59c640 1 --2- 172.21.15.40:0/1021236228 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed1febb80 0x556ed1fee000 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=accfcc7ac9b6895c server_cookie=989d8e81bd7a1fab in_seq=0 out_seq=0 2024-03-18T22:53:17.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.470+0000 7f8fb2ff5640 10 client.18240 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:17.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.471+0000 7f8fb2ff5640 1 -- 172.21.15.40:0/1021236228 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556ed1f4ad90 con 0x556ed1febb80 2024-03-18T22:53:17.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.471+0000 7f8fb2ff5640 10 client.18240 handle_command_reply: tid=0 2024-03-18T22:53:17.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.473+0000 7f8fd7dbf740 1 client.18240 shutdown 2024-03-18T22:53:17.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.473+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/1021236228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fdec10 msgr2=0x556ed1ffa2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:17.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.474+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/1021236228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fdec10 0x556ed1ffa2e0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000eb90 tx=0x7f8fc000ebc0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.474+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/1021236228 shutdown_connections 2024-03-18T22:53:17.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.474+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/1021236228 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fe4770 0x556ed2001860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.474+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/1021236228 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed1febb80 0x556ed1fee000 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.474+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/1021236228 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fdf600 0x556ed1ffa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.474+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/1021236228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fdec10 0x556ed1ffa2e0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.474+0000 7f8fb2ff5640 0 client.18240 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:17.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.474+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/1021236228 >> 172.21.15.40:0/1021236228 conn(0x556ed1fda180 msgr2=0x556ed1fd94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:17.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.476+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/1021236228 shutdown_connections 2024-03-18T22:53:17.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.477+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/1021236228 wait complete. 2024-03-18T22:53:17.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.480+0000 7f8fd7dbf740 20 client.18240 trim_cache size 0 max 16384 2024-03-18T22:53:17.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.521+0000 7f8fd7dbf740 1 Processor -- start 2024-03-18T22:53:17.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.521+0000 7f8fd7dbf740 1 -- start start 2024-03-18T22:53:17.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.522+0000 7f8fd7dbf740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdec10 0x556ed1fe7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.522+0000 7f8fd7dbf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fdf600 0x556ed2001780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.522+0000 7f8fd7dbf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fe4770 0x556ed1fddaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.522+0000 7f8fd7dbf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc00feb0 con 0x556ed1fdf600 2024-03-18T22:53:17.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.522+0000 7f8fd7dbf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc011ae0 con 0x556ed1fe4770 2024-03-18T22:53:17.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.522+0000 7f8fd7dbf740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc010580 con 0x556ed1fdec10 2024-03-18T22:53:17.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.523+0000 7f8fced9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdec10 0x556ed1fe7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.523+0000 7f8fced9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdec10 0x556ed1fe7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:42032/0 (socket says 172.21.15.40:42032) 2024-03-18T22:53:17.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.523+0000 7f8fced9d640 1 -- 172.21.15.40:0/1026589217 learned_addr learned my addr 172.21.15.40:0/1026589217 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:17.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.523+0000 7f8fce59c640 1 --2- 172.21.15.40:0/1026589217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fdf600 0x556ed2001780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.524+0000 7f8fce59c640 1 -- 172.21.15.40:0/1026589217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdec10 msgr2=0x556ed1fe7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:17.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.524+0000 7f8fce59c640 1 --2- 172.21.15.40:0/1026589217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdec10 0x556ed1fe7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.524+0000 7f8fce59c640 1 -- 172.21.15.40:0/1026589217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fe4770 msgr2=0x556ed1fddaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:17.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.524+0000 7f8fce59c640 1 --2- 172.21.15.40:0/1026589217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fe4770 0x556ed1fddaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.524+0000 7f8fce59c640 1 -- 172.21.15.40:0/1026589217 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ed1c170f0 con 0x556ed1fdf600 2024-03-18T22:53:17.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.524+0000 7f8fced9d640 1 --2- 172.21.15.40:0/1026589217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdec10 0x556ed1fe7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:53:17.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.525+0000 7f8fce59c640 1 --2- 172.21.15.40:0/1026589217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fdf600 0x556ed2001780 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4008000 tx=0x7f8fc403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d6aea74b6fa1fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:17.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.531+0000 7f8fb4ff9640 1 -- 172.21.15.40:0/1026589217 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4052170 con 0x556ed1fdf600 2024-03-18T22:53:17.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.532+0000 7f8fb4ff9640 1 -- 172.21.15.40:0/1026589217 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fc4058070 con 0x556ed1fdf600 2024-03-18T22:53:17.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.532+0000 7f8fb4ff9640 1 -- 172.21.15.40:0/1026589217 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4003ae0 con 0x556ed1fdf600 2024-03-18T22:53:17.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.533+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/1026589217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fdf600 msgr2=0x556ed2001780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:17.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.533+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/1026589217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fdf600 0x556ed2001780 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4008000 tx=0x7f8fc403f000 comp rx=0 tx=0).stop 2024-03-18T22:53:17.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.536+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/1026589217 shutdown_connections 2024-03-18T22:53:17.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.536+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/1026589217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fe4770 0x556ed1fddaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.536+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/1026589217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fdf600 0x556ed2001780 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.536+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/1026589217 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdec10 0x556ed1fe7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.536+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/1026589217 >> 172.21.15.40:0/1026589217 conn(0x556ed1fda180 msgr2=0x556ed1fdd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:17.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.542+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/1026589217 shutdown_connections 2024-03-18T22:53:17.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.543+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/1026589217 wait complete. 2024-03-18T22:53:17.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.543+0000 7f8fd7dbf740 1 -- start start 2024-03-18T22:53:17.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.543+0000 7f8fd7dbf740 1 Processor -- start 2024-03-18T22:53:17.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.544+0000 7f8fd7dbf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fdec10 0x556ed1ffea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.544+0000 7f8fd7dbf740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdf600 0x556ed1ffefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.544+0000 7f8fd7dbf740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fe4770 0x556ed1ffffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.544+0000 7f8fd7dbf740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc011410 con 0x556ed1fe4770 2024-03-18T22:53:17.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.544+0000 7f8fd7dbf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc010c80 con 0x556ed1fdec10 2024-03-18T22:53:17.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.545+0000 7f8fd7dbf740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc00de90 con 0x556ed1fdf600 2024-03-18T22:53:17.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.545+0000 7f8fcf59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fe4770 0x556ed1ffffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.545+0000 7f8fd7dbf740 10 client.0 _ll_register_callbacks cb 0x7f8fc00076d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:17.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.545+0000 7f8fce59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdf600 0x556ed1ffefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.545+0000 7f8fce59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdf600 0x556ed1ffefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:42048/0 (socket says 172.21.15.40:42048) 2024-03-18T22:53:17.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.545+0000 7f8fce59c640 1 -- 172.21.15.40:0/585799151 learned_addr learned my addr 172.21.15.40:0/585799151 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:17.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.545+0000 7f8fcf59e640 1 -- 172.21.15.40:0/585799151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdf600 msgr2=0x556ed1ffefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:17.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.545+0000 7f8fcf59e640 1 --2- 172.21.15.40:0/585799151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdf600 0x556ed1ffefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.545+0000 7f8fcf59e640 1 -- 172.21.15.40:0/585799151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fdec10 msgr2=0x556ed1ffea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:17.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.545+0000 7f8fcf59e640 1 --2- 172.21.15.40:0/585799151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fdec10 0x556ed1ffea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.545+0000 7f8fcf59e640 1 -- 172.21.15.40:0/585799151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ed20005a0 con 0x556ed1fe4770 2024-03-18T22:53:17.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.546+0000 7f8fcf59e640 1 --2- 172.21.15.40:0/585799151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fe4770 0x556ed1ffffb0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80ccba0 tx=0x7f8fc809fc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9bfa9aaa5736a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:17.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.546+0000 7f8fb2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:53:17.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.546+0000 7f8fb2ff5640 1 -- 172.21.15.40:0/585799151 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80b9060 con 0x556ed1fe4770 2024-03-18T22:53:17.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.547+0000 7f8fb2ff5640 1 -- 172.21.15.40:0/585799151 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8fc809bca0 con 0x556ed1fe4770 2024-03-18T22:53:17.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.549+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/585799151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fc4058070 con 0x556ed1fe4770 2024-03-18T22:53:17.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.551+0000 7f8fb2ff5640 1 -- 172.21.15.40:0/585799151 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8fc80ccd70 con 0x556ed1fe4770 2024-03-18T22:53:17.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.551+0000 7f8fd7dbf740 10 client.18246 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:17.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.551+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/585799151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556ed2001980 con 0x556ed1fe4770 2024-03-18T22:53:17.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.553+0000 7f8fb2ff5640 1 -- 172.21.15.40:0/585799151 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fc809bea0 con 0x556ed1fe4770 2024-03-18T22:53:17.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.554+0000 7f8fb2ff5640 1 -- 172.21.15.40:0/585799151 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f8fc80b4b00 con 0x556ed1fe4770 2024-03-18T22:53:17.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.554+0000 7f8fd7dbf740 10 client.18246 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:17.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.554+0000 7f8fd7dbf740 10 client.18246 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:53:17.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.554+0000 7f8fd7dbf740 20 client.18246 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:17.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.554+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/585799151 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed207da30 0x556ed207fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:17.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.554+0000 7f8fd7dbf740 4 client.18246 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:53:17.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.554+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/585799151 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f8fc4058070 con 0x556ed207da30 2024-03-18T22:53:17.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.555+0000 7f8fced9d640 1 --2- 172.21.15.40:0/585799151 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed207da30 0x556ed207fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:17.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.556+0000 7f8fced9d640 1 --2- 172.21.15.40:0/585799151 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed207da30 0x556ed207fe50 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=441d8ea9d08d45f6 server_cookie=de0f2b5ff3981e04 in_seq=0 out_seq=0 2024-03-18T22:53:17.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.556+0000 7f8fb2ff5640 10 client.18246 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:17.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.556+0000 7f8fb2ff5640 1 -- 172.21.15.40:0/585799151 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x7f8fc4058070 con 0x556ed207da30 2024-03-18T22:53:17.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.556+0000 7f8fb2ff5640 10 client.18246 handle_command_reply: tid=0 2024-03-18T22:53:17.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.556+0000 7f8fd7dbf740 1 client.18246 shutdown 2024-03-18T22:53:17.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.558+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/585799151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fe4770 msgr2=0x556ed1ffffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:17.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.558+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/585799151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fe4770 0x556ed1ffffb0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80ccba0 tx=0x7f8fc809fc50 comp rx=0 tx=0).stop 2024-03-18T22:53:17.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.558+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/585799151 shutdown_connections 2024-03-18T22:53:17.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.558+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/585799151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556ed1fe4770 0x556ed1ffffb0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.558+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/585799151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556ed1fdf600 0x556ed1ffefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.558+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/585799151 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x556ed207da30 0x556ed207fe50 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.558+0000 7f8fd7dbf740 1 --2- 172.21.15.40:0/585799151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ed1fdec10 0x556ed1ffea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.558+0000 7f8fb2ff5640 0 client.18246 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:17.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.559+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/585799151 >> 172.21.15.40:0/585799151 conn(0x556ed1fda180 msgr2=0x556ed1fddc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:17.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.559+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/585799151 shutdown_connections 2024-03-18T22:53:17.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.559+0000 7f8fd7dbf740 1 -- 172.21.15.40:0/585799151 wait complete. 2024-03-18T22:53:17.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.559+0000 7f8fd7dbf740 20 client.18246 trim_cache size 0 max 16384 2024-03-18T22:53:17.561 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:53:17.561 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "scrub active (100 inodes in the stack)", 2024-03-18T22:53:17.561 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:53:17.561 INFO:teuthology.orchestra.run.smithi040.stdout: "6a7b3721-c0a7-4062-9ca5-671585e8609b": { 2024-03-18T22:53:17.561 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:53:17.561 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "6a7b3721-c0a7-4062-9ca5-671585e8609b", 2024-03-18T22:53:17.561 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:53:17.561 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:53:17.561 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:53:17.561 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:53:17.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.560+0000 7f8fd48f7640 1 -- 172.21.15.40:0/4009206505 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8fb80a2a00 msgr2=0x7f8fb80a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:17.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.560+0000 7f8fd48f7640 1 --2- 172.21.15.40:0/4009206505 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8fb80a2a00 0x7f8fb80a4ec0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8fc402c1f0 tx=0x7f8fc4000bc0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.560+0000 7f8fd48f7640 1 -- 172.21.15.40:0/4009206505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fd01abf70 msgr2=0x7f8fd01ae360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:17.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.560+0000 7f8fd48f7640 1 --2- 172.21.15.40:0/4009206505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fd01abf70 0x7f8fd01ae360 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009fd0 tx=0x7f8fc800ef10 comp rx=0 tx=0).stop 2024-03-18T22:53:17.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.561+0000 7f8fd48f7640 1 -- 172.21.15.40:0/4009206505 shutdown_connections 2024-03-18T22:53:17.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.561+0000 7f8fd48f7640 1 --2- 172.21.15.40:0/4009206505 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8fb80a2a00 0x7f8fb80a4ec0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.561+0000 7f8fd48f7640 1 --2- 172.21.15.40:0/4009206505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8fd01abf70 0x7f8fd01ae360 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.561+0000 7f8fd48f7640 1 --2- 172.21.15.40:0/4009206505 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8fd01098b0 0x7f8fd01a6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.561+0000 7f8fd48f7640 1 --2- 172.21.15.40:0/4009206505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fd0072c30 0x7f8fd01a69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:17.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.561+0000 7f8fd48f7640 1 -- 172.21.15.40:0/4009206505 >> 172.21.15.40:0/4009206505 conn(0x7f8fd006e700 msgr2=0x7f8fd00703d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:17.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.561+0000 7f8fd48f7640 1 -- 172.21.15.40:0/4009206505 shutdown_connections 2024-03-18T22:53:17.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:17.561+0000 7f8fd48f7640 1 -- 172.21.15.40:0/4009206505 wait complete. 2024-03-18T22:53:17.605 INFO:tasks.cephfs.filesystem:scrub status for tag:6a7b3721-c0a7-4062-9ca5-671585e8609b - {'path': '/', 'tag': '6a7b3721-c0a7-4062-9ca5-671585e8609b', 'options': 'recursive,force'} 2024-03-18T22:53:17.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:17 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:53:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:17 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:53:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:17 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:53:17.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:17 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:53:17.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:17 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:53:17.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:17 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:53:17.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:17 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:53:17.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:17 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:53:17.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:17 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:53:18.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:18 smithi113 ceph-mon[66775]: pgmap v670: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 MiB/s wr, 4.36k op/s 2024-03-18T22:53:18.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:18 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:53:18.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:18 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:53:18.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:18 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:53:18.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:18 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:53:18.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:18 smithi103 ceph-mon[66377]: pgmap v670: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 MiB/s wr, 4.36k op/s 2024-03-18T22:53:18.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:18 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:53:18.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:18 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:53:18.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:18 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:53:18.860 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:18 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:53:18.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:18 smithi040 ceph-mon[62643]: pgmap v670: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 MiB/s wr, 4.36k op/s 2024-03-18T22:53:18.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:18 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:53:18.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:18 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:53:18.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:18 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:53:18.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:18 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:53:20.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:20 smithi113 ceph-mon[66775]: pgmap v671: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 3.99k op/s 2024-03-18T22:53:20.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:20 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:53:20.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:20 smithi103 ceph-mon[66377]: pgmap v671: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 3.99k op/s 2024-03-18T22:53:20.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:20 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:53:20.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:20 smithi040 ceph-mon[62643]: pgmap v671: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 3.99k op/s 2024-03-18T22:53:20.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:20 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:53:22.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:22 smithi113 ceph-mon[66775]: pgmap v672: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 MiB/s wr, 3.83k op/s 2024-03-18T22:53:22.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:22 smithi103 ceph-mon[66377]: pgmap v672: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 MiB/s wr, 3.83k op/s 2024-03-18T22:53:22.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:22 smithi040 ceph-mon[62643]: pgmap v672: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 MiB/s wr, 3.83k op/s 2024-03-18T22:53:23.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:23 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:53:23.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:23 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:53:23.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:23 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:53:24.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:24 smithi113 ceph-mon[66775]: pgmap v673: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 4.14k op/s 2024-03-18T22:53:24.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:24 smithi103 ceph-mon[66377]: pgmap v673: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 4.14k op/s 2024-03-18T22:53:24.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:24 smithi040 ceph-mon[62643]: pgmap v673: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 4.14k op/s 2024-03-18T22:53:26.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:26 smithi113 ceph-mon[66775]: pgmap v674: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 MiB/s wr, 3.97k op/s 2024-03-18T22:53:26.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:26 smithi103 ceph-mon[66377]: pgmap v674: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 MiB/s wr, 3.97k op/s 2024-03-18T22:53:26.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:26 smithi040 ceph-mon[62643]: pgmap v674: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 MiB/s wr, 3.97k op/s 2024-03-18T22:53:28.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:28 smithi113 ceph-mon[66775]: pgmap v675: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 MiB/s wr, 3.90k op/s 2024-03-18T22:53:28.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:28 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:53:28.861 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:28 smithi103 ceph-mon[66377]: pgmap v675: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 MiB/s wr, 3.90k op/s 2024-03-18T22:53:28.861 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:28 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:53:28.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:28 smithi040 ceph-mon[62643]: pgmap v675: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 MiB/s wr, 3.90k op/s 2024-03-18T22:53:28.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:28 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:53:30.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:30 smithi113 ceph-mon[66775]: pgmap v676: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 3.74k op/s 2024-03-18T22:53:30.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:30 smithi103 ceph-mon[66377]: pgmap v676: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 3.74k op/s 2024-03-18T22:53:30.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:30 smithi040 ceph-mon[62643]: pgmap v676: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 3.74k op/s 2024-03-18T22:53:32.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:32 smithi103 ceph-mon[66377]: pgmap v677: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 MiB/s wr, 3.44k op/s 2024-03-18T22:53:32.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:32 smithi040 ceph-mon[62643]: pgmap v677: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 MiB/s wr, 3.44k op/s 2024-03-18T22:53:33.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:32 smithi113 ceph-mon[66775]: pgmap v677: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 MiB/s wr, 3.44k op/s 2024-03-18T22:53:34.852 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:34 smithi103 ceph-mon[66377]: pgmap v678: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 MiB/s wr, 3.77k op/s 2024-03-18T22:53:34.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:34 smithi040 ceph-mon[62643]: pgmap v678: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 MiB/s wr, 3.77k op/s 2024-03-18T22:53:35.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:34 smithi113 ceph-mon[66775]: pgmap v678: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 MiB/s wr, 3.77k op/s 2024-03-18T22:53:35.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:35 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:53:35.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:35 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:53:36.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:35 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:53:36.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:36 smithi103 ceph-mon[66377]: pgmap v679: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 MiB/s wr, 3.13k op/s 2024-03-18T22:53:36.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:36 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:53:36.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:36 smithi040 ceph-mon[62643]: pgmap v679: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 MiB/s wr, 3.13k op/s 2024-03-18T22:53:36.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:36 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:53:37.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:36 smithi113 ceph-mon[66775]: pgmap v679: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 MiB/s wr, 3.13k op/s 2024-03-18T22:53:37.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:36 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:53:38.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:38 smithi103 ceph-mon[66377]: pgmap v680: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.81k op/s 2024-03-18T22:53:38.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:38.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:38.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:38.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:38 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:38.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:38 smithi040 ceph-mon[62643]: pgmap v680: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.81k op/s 2024-03-18T22:53:38.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:38.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:38.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:38.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:38 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:39.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:38 smithi113 ceph-mon[66775]: pgmap v680: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.81k op/s 2024-03-18T22:53:39.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:39.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:39.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:39.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:38 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:40.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:40 smithi040 ceph-mon[62643]: pgmap v681: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-03-18T22:53:41.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:40 smithi113 ceph-mon[66775]: pgmap v681: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-03-18T22:53:41.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:40 smithi103 ceph-mon[66377]: pgmap v681: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-03-18T22:53:42.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:42 smithi040 ceph-mon[62643]: pgmap v682: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 2.20k op/s 2024-03-18T22:53:43.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:42 smithi113 ceph-mon[66775]: pgmap v682: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 2.20k op/s 2024-03-18T22:53:43.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:42 smithi103 ceph-mon[66377]: pgmap v682: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 2.20k op/s 2024-03-18T22:53:45.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:44 smithi040 ceph-mon[62643]: pgmap v683: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 2.27k op/s 2024-03-18T22:53:45.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:45.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:45.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:53:45.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:53:45.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:44 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:44 smithi113 ceph-mon[66775]: pgmap v683: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 2.27k op/s 2024-03-18T22:53:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:45.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:53:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:53:45.332 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:44 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:44 smithi103 ceph-mon[66377]: pgmap v683: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 2.27k op/s 2024-03-18T22:53:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:53:45.360 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:53:45.361 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:44 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:53:47.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:46 smithi040 ceph-mon[62643]: pgmap v684: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.8 MiB/s wr, 1.98k op/s 2024-03-18T22:53:47.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:46 smithi113 ceph-mon[66775]: pgmap v684: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.8 MiB/s wr, 1.98k op/s 2024-03-18T22:53:47.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:46 smithi103 ceph-mon[66377]: pgmap v684: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.8 MiB/s wr, 1.98k op/s 2024-03-18T22:53:47.607 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:53:47.607 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:53:47.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.742+0000 7f35e8a81640 1 -- 172.21.15.40:0/162840105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35e41022d0 msgr2=0x7f35e4108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:47.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.742+0000 7f35e8a81640 1 --2- 172.21.15.40:0/162840105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35e41022d0 0x7f35e4108b60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f35d4009900 tx=0x7f35d402bc30 comp rx=0 tx=0).stop 2024-03-18T22:53:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.742+0000 7f35e8a81640 1 -- 172.21.15.40:0/162840105 shutdown_connections 2024-03-18T22:53:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.742+0000 7f35e8a81640 1 --2- 172.21.15.40:0/162840105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35e41022d0 0x7f35e4108b60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.742+0000 7f35e8a81640 1 --2- 172.21.15.40:0/162840105 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35e4101910 0x7f35e4101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.742+0000 7f35e8a81640 1 --2- 172.21.15.40:0/162840105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e4100720 0x7f35e4100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.742+0000 7f35e8a81640 1 -- 172.21.15.40:0/162840105 >> 172.21.15.40:0/162840105 conn(0x7f35e40fbcb0 msgr2=0x7f35e40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.742+0000 7f35e8a81640 1 -- 172.21.15.40:0/162840105 shutdown_connections 2024-03-18T22:53:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.742+0000 7f35e8a81640 1 -- 172.21.15.40:0/162840105 wait complete. 2024-03-18T22:53:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.743+0000 7f35e8a81640 1 Processor -- start 2024-03-18T22:53:47.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.743+0000 7f35e8a81640 1 -- start start 2024-03-18T22:53:47.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.743+0000 7f35e8a81640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e4100720 0x7f35e40694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.743+0000 7f35e8a81640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35e4101910 0x7f35e419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.743+0000 7f35e8a81640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35e41022d0 0x7f35e419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.743+0000 7f35e8a81640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f35e410c2b0 con 0x7f35e4100720 2024-03-18T22:53:47.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.743+0000 7f35e8a81640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f35e410c130 con 0x7f35e41022d0 2024-03-18T22:53:47.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.743+0000 7f35e8a81640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f35e410c430 con 0x7f35e4101910 2024-03-18T22:53:47.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.744+0000 7f35e259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e4100720 0x7f35e40694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.744+0000 7f35e259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e4100720 0x7f35e40694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:32986/0 (socket says 172.21.15.40:32986) 2024-03-18T22:53:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.744+0000 7f35e259c640 1 -- 172.21.15.40:0/915522812 learned_addr learned my addr 172.21.15.40:0/915522812 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.744+0000 7f35e2d9d640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35e41022d0 0x7f35e419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.744+0000 7f35e1d9b640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35e4101910 0x7f35e419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.744+0000 7f35e259c640 1 -- 172.21.15.40:0/915522812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35e4101910 msgr2=0x7f35e419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.744+0000 7f35e259c640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35e4101910 0x7f35e419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.744+0000 7f35e259c640 1 -- 172.21.15.40:0/915522812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35e41022d0 msgr2=0x7f35e419ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.744+0000 7f35e259c640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35e41022d0 0x7f35e419ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.744+0000 7f35e259c640 1 -- 172.21.15.40:0/915522812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e4196720 con 0x7f35e4100720 2024-03-18T22:53:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.744+0000 7f35e1d9b640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35e4101910 0x7f35e419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:53:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.744+0000 7f35e2d9d640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35e41022d0 0x7f35e419ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:53:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.744+0000 7f35e259c640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e4100720 0x7f35e40694f0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f35d800e9c0 tx=0x7f35d800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.744+0000 7f35d37fe640 1 -- 172.21.15.40:0/915522812 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d800cd80 con 0x7f35e4100720 2024-03-18T22:53:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.745+0000 7f35d37fe640 1 -- 172.21.15.40:0/915522812 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35d80035f0 con 0x7f35e4100720 2024-03-18T22:53:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.745+0000 7f35e8a81640 1 -- 172.21.15.40:0/915522812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35e41969b0 con 0x7f35e4100720 2024-03-18T22:53:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.745+0000 7f35d37fe640 1 -- 172.21.15.40:0/915522812 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d8010690 con 0x7f35e4100720 2024-03-18T22:53:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.745+0000 7f35e8a81640 1 -- 172.21.15.40:0/915522812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e4196fd0 con 0x7f35e4100720 2024-03-18T22:53:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.749+0000 7f35eb66d740 1 Processor -- start 2024-03-18T22:53:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.749+0000 7f35eb66d740 1 -- start start 2024-03-18T22:53:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.749+0000 7f35eb66d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0db260 0x55d2ed0db680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35d37fe640 1 -- 172.21.15.40:0/915522812 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f35d80108f0 con 0x7f35e4100720 2024-03-18T22:53:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35eb66d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0e0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35eb66d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0e3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35eb66d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d2ed037500 con 0x55d2ed0e0dc0 2024-03-18T22:53:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35eb66d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d2ecfdfbe0 con 0x55d2ed0dbc50 2024-03-18T22:53:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35eb66d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55d2ed0335a0 con 0x55d2ed0db260 2024-03-18T22:53:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35e2d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0db260 0x55d2ed0db680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35e259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0e0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35e2d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0db260 0x55d2ed0db680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:46510/0 (socket says 172.21.15.40:46510) 2024-03-18T22:53:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35e2d9d640 1 -- 172.21.15.40:0/4195036310 learned_addr learned my addr 172.21.15.40:0/4195036310 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35e1d9b640 1 --2- 172.21.15.40:0/4195036310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0e3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35e2d9d640 1 -- 172.21.15.40:0/4195036310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0dbc50 msgr2=0x55d2ed0e0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35e2d9d640 1 --2- 172.21.15.40:0/4195036310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0e0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35e2d9d640 1 -- 172.21.15.40:0/4195036310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0e0dc0 msgr2=0x55d2ed0e3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35e2d9d640 1 --2- 172.21.15.40:0/4195036310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0e3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35e2d9d640 1 -- 172.21.15.40:0/4195036310 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2ed001000 con 0x55d2ed0db260 2024-03-18T22:53:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.750+0000 7f35e259c640 1 --2- 172.21.15.40:0/4195036310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0e0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:53:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.751+0000 7f35e2d9d640 1 --2- 172.21.15.40:0/4195036310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0db260 0x55d2ed0db680 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f35d4009900 tx=0x7f35d4005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8134c44159ea841 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.751+0000 7f35d0ff9640 1 -- 172.21.15.40:0/4195036310 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d4045020 con 0x55d2ed0db260 2024-03-18T22:53:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.751+0000 7f35d0ff9640 1 -- 172.21.15.40:0/4195036310 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35d4005040 con 0x55d2ed0db260 2024-03-18T22:53:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.751+0000 7f35d0ff9640 1 -- 172.21.15.40:0/4195036310 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d4004a90 con 0x55d2ed0db260 2024-03-18T22:53:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.751+0000 7f35d37fe640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f35cc0a2900 0x7f35cc0a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.751+0000 7f35e259c640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f35cc0a2900 0x7f35cc0a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.752+0000 7f35d37fe640 1 -- 172.21.15.40:0/915522812 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f35d80cf2b0 con 0x7f35e4100720 2024-03-18T22:53:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.752+0000 7f35eb66d740 1 -- 172.21.15.40:0/4195036310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0db260 msgr2=0x55d2ed0db680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.752+0000 7f35eb66d740 1 --2- 172.21.15.40:0/4195036310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0db260 0x55d2ed0db680 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f35d4009900 tx=0x7f35d4005e80 comp rx=0 tx=0).stop 2024-03-18T22:53:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.752+0000 7f35eb66d740 1 -- 172.21.15.40:0/4195036310 shutdown_connections 2024-03-18T22:53:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.752+0000 7f35eb66d740 1 --2- 172.21.15.40:0/4195036310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0e3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.752+0000 7f35eb66d740 1 --2- 172.21.15.40:0/4195036310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0e0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.752+0000 7f35eb66d740 1 --2- 172.21.15.40:0/4195036310 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0db260 0x55d2ed0db680 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.752+0000 7f35eb66d740 1 -- 172.21.15.40:0/4195036310 >> 172.21.15.40:0/4195036310 conn(0x55d2ed0cdfc0 msgr2=0x55d2ed0ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.752+0000 7f35e259c640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f35cc0a2900 0x7f35cc0a4dc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f35d809c400 tx=0x7f35d80d2c70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.752+0000 7f35eb66d740 1 -- 172.21.15.40:0/4195036310 shutdown_connections 2024-03-18T22:53:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.752+0000 7f35eb66d740 1 -- 172.21.15.40:0/4195036310 wait complete. 2024-03-18T22:53:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.753+0000 7f35eb66d740 1 -- start start 2024-03-18T22:53:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.753+0000 7f35eb66d740 1 Processor -- start 2024-03-18T22:53:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.753+0000 7f35eb66d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0db260 0x55d2ed0e56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.753+0000 7f35eb66d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0e5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.753+0000 7f35eb66d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0ecc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.753+0000 7f35e2d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0db260 0x55d2ed0e56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.753+0000 7f35e2d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0db260 0x55d2ed0e56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:46518/0 (socket says 172.21.15.40:46518) 2024-03-18T22:53:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.753+0000 7f35e2d9d640 1 -- 172.21.15.40:0/2847036646 learned_addr learned my addr 172.21.15.40:0/2847036646 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.753+0000 7f35eb66d740 1 -- 172.21.15.40:0/2847036646 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55d2ed0e3d10 con 0x55d2ed0e0dc0 2024-03-18T22:53:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.753+0000 7f35eb66d740 1 -- 172.21.15.40:0/2847036646 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d2ecff3160 con 0x55d2ed0dbc50 2024-03-18T22:53:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.753+0000 7f35eb66d740 1 -- 172.21.15.40:0/2847036646 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55d2ed0e3e90 con 0x55d2ed0db260 2024-03-18T22:53:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.754+0000 7f35e2d9d640 1 --2- 172.21.15.40:0/2847036646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0ecc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.754+0000 7f35e1d9b640 1 --2- 172.21.15.40:0/2847036646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0e5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.754+0000 7f35eb66d740 10 client.0 _ll_register_callbacks cb 0x55d2ed02fa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.754+0000 7f35e2d9d640 1 -- 172.21.15.40:0/2847036646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0dbc50 msgr2=0x55d2ed0e5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.754+0000 7f35e2d9d640 1 --2- 172.21.15.40:0/2847036646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0e5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.754+0000 7f35e2d9d640 1 -- 172.21.15.40:0/2847036646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0e0dc0 msgr2=0x55d2ed0ecc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.754+0000 7f35e2d9d640 1 --2- 172.21.15.40:0/2847036646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0ecc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.754+0000 7f35e2d9d640 1 -- 172.21.15.40:0/2847036646 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2ece157d0 con 0x55d2ed0db260 2024-03-18T22:53:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.754+0000 7f35e2d9d640 1 --2- 172.21.15.40:0/2847036646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0db260 0x55d2ed0e56d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f35d4036040 tx=0x7f35d402cd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb573d334b8cdf56 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.754+0000 7f35c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:53:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.754+0000 7f35c9ffb640 1 -- 172.21.15.40:0/2847036646 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d4045030 con 0x55d2ed0db260 2024-03-18T22:53:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.754+0000 7f35eb66d740 1 -- 172.21.15.40:0/2847036646 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35d4005040 con 0x55d2ed0db260 2024-03-18T22:53:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.754+0000 7f35c9ffb640 1 -- 172.21.15.40:0/2847036646 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f35d4041890 con 0x55d2ed0db260 2024-03-18T22:53:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.755+0000 7f35c9ffb640 1 -- 172.21.15.40:0/2847036646 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f35d4041a90 con 0x55d2ed0db260 2024-03-18T22:53:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.755+0000 7f35eb66d740 10 client.28069 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.755+0000 7f35eb66d740 1 -- 172.21.15.40:0/2847036646 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d2ecd4b3f0 con 0x55d2ed0db260 2024-03-18T22:53:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.755+0000 7f35c9ffb640 1 -- 172.21.15.40:0/2847036646 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35d4005040 con 0x55d2ed0db260 2024-03-18T22:53:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.755+0000 7f35c9ffb640 1 -- 172.21.15.40:0/2847036646 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f35d4034980 con 0x55d2ed0db260 2024-03-18T22:53:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.755+0000 7f35eb66d740 10 client.28069 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.755+0000 7f35eb66d740 10 client.28069 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:53:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.755+0000 7f35eb66d740 20 client.28069 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.755+0000 7f35eb66d740 1 --2- 172.21.15.40:0/2847036646 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2ed0f1100 0x55d2ed0f34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.755+0000 7f35eb66d740 4 client.28069 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:53:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.755+0000 7f35eb66d740 1 -- 172.21.15.40:0/2847036646 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2ed044a40 con 0x55d2ed0f1100 2024-03-18T22:53:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.755+0000 7f35e1d9b640 1 --2- 172.21.15.40:0/2847036646 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2ed0f1100 0x55d2ed0f34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.756+0000 7f35e1d9b640 1 --2- 172.21.15.40:0/2847036646 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2ed0f1100 0x55d2ed0f34f0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=935de49f2523d5aa server_cookie=f900aaaf518cf6c8 in_seq=0 out_seq=0 2024-03-18T22:53:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.756+0000 7f35c9ffb640 10 client.28069 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.757+0000 7f35c9ffb640 1 -- 172.21.15.40:0/2847036646 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d2ed044a40 con 0x55d2ed0f1100 2024-03-18T22:53:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.757+0000 7f35c9ffb640 10 client.28069 handle_command_reply: tid=0 2024-03-18T22:53:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.757+0000 7f35eb66d740 1 client.28069 shutdown 2024-03-18T22:53:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.757+0000 7f35eb66d740 1 -- 172.21.15.40:0/2847036646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0db260 msgr2=0x55d2ed0e56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.757+0000 7f35eb66d740 1 --2- 172.21.15.40:0/2847036646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0db260 0x55d2ed0e56d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f35d4036040 tx=0x7f35d402cd90 comp rx=0 tx=0).stop 2024-03-18T22:53:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.757+0000 7f35eb66d740 1 -- 172.21.15.40:0/2847036646 shutdown_connections 2024-03-18T22:53:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.757+0000 7f35eb66d740 1 --2- 172.21.15.40:0/2847036646 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0ecc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.757+0000 7f35eb66d740 1 --2- 172.21.15.40:0/2847036646 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2ed0f1100 0x55d2ed0f34f0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.757+0000 7f35eb66d740 1 --2- 172.21.15.40:0/2847036646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0e5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.757+0000 7f35eb66d740 1 --2- 172.21.15.40:0/2847036646 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0db260 0x55d2ed0e56d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.757+0000 7f35eb66d740 1 -- 172.21.15.40:0/2847036646 >> 172.21.15.40:0/2847036646 conn(0x55d2ed0cdfc0 msgr2=0x55d2ed0cd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.757+0000 7f35c9ffb640 0 client.28069 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.757+0000 7f35eb66d740 1 -- 172.21.15.40:0/2847036646 shutdown_connections 2024-03-18T22:53:47.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.757+0000 7f35eb66d740 1 -- 172.21.15.40:0/2847036646 wait complete. 2024-03-18T22:53:47.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.758+0000 7f35eb66d740 20 client.28069 trim_cache size 0 max 16384 2024-03-18T22:53:47.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.787+0000 7f35eb66d740 1 Processor -- start 2024-03-18T22:53:47.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.787+0000 7f35eb66d740 1 -- start start 2024-03-18T22:53:47.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.787+0000 7f35eb66d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0db260 0x55d2ed0d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.787+0000 7f35eb66d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0f7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.787+0000 7f35eb66d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0f8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.787+0000 7f35eb66d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f35e41a83d0 con 0x55d2ed0dbc50 2024-03-18T22:53:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.787+0000 7f35eb66d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f35e41aa000 con 0x55d2ed0db260 2024-03-18T22:53:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.787+0000 7f35eb66d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f35e41a8aa0 con 0x55d2ed0e0dc0 2024-03-18T22:53:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.788+0000 7f35e1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0f7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.788+0000 7f35e1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0f7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33018/0 (socket says 172.21.15.40:33018) 2024-03-18T22:53:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.788+0000 7f35e1d9b640 1 -- 172.21.15.40:0/800916064 learned_addr learned my addr 172.21.15.40:0/800916064 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.788+0000 7f35e2d9d640 1 --2- 172.21.15.40:0/800916064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0db260 0x55d2ed0d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.788+0000 7f35e2d9d640 1 --2- 172.21.15.40:0/800916064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0f8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.788+0000 7f35e1d9b640 1 -- 172.21.15.40:0/800916064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0e0dc0 msgr2=0x55d2ed0f8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.788+0000 7f35e1d9b640 1 --2- 172.21.15.40:0/800916064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0f8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.788+0000 7f35e1d9b640 1 -- 172.21.15.40:0/800916064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0db260 msgr2=0x55d2ed0d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.788+0000 7f35e1d9b640 1 --2- 172.21.15.40:0/800916064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0db260 0x55d2ed0d1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.788+0000 7f35e1d9b640 1 -- 172.21.15.40:0/800916064 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2ecd0b0f0 con 0x55d2ed0dbc50 2024-03-18T22:53:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.788+0000 7f35e1d9b640 1 --2- 172.21.15.40:0/800916064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0f7d50 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f35e4069260 tx=0x7f35dc0077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8206e3a11caf8f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.788+0000 7f35cbfff640 1 -- 172.21.15.40:0/800916064 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc019530 con 0x55d2ed0dbc50 2024-03-18T22:53:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.788+0000 7f35cbfff640 1 -- 172.21.15.40:0/800916064 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35dc0196d0 con 0x55d2ed0dbc50 2024-03-18T22:53:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.788+0000 7f35cbfff640 1 -- 172.21.15.40:0/800916064 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc010800 con 0x55d2ed0dbc50 2024-03-18T22:53:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.789+0000 7f35eb66d740 1 -- 172.21.15.40:0/800916064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0dbc50 msgr2=0x55d2ed0f7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.789+0000 7f35eb66d740 1 --2- 172.21.15.40:0/800916064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0f7d50 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f35e4069260 tx=0x7f35dc0077c0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.789+0000 7f35eb66d740 1 -- 172.21.15.40:0/800916064 shutdown_connections 2024-03-18T22:53:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.789+0000 7f35eb66d740 1 --2- 172.21.15.40:0/800916064 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0f8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.789+0000 7f35eb66d740 1 --2- 172.21.15.40:0/800916064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0f7d50 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.789+0000 7f35eb66d740 1 --2- 172.21.15.40:0/800916064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0db260 0x55d2ed0d1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.789+0000 7f35eb66d740 1 -- 172.21.15.40:0/800916064 >> 172.21.15.40:0/800916064 conn(0x55d2ed0cdfc0 msgr2=0x55d2ed0d1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.789+0000 7f35eb66d740 1 -- 172.21.15.40:0/800916064 shutdown_connections 2024-03-18T22:53:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.790+0000 7f35eb66d740 1 -- 172.21.15.40:0/800916064 wait complete. 2024-03-18T22:53:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.790+0000 7f35eb66d740 1 -- start start 2024-03-18T22:53:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.790+0000 7f35eb66d740 1 Processor -- start 2024-03-18T22:53:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.790+0000 7f35eb66d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0db260 0x55d2ed0ea390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.790+0000 7f35eb66d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0ea8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.790+0000 7f35eb66d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0eb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.790+0000 7f35eb66d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f35e41a9930 con 0x55d2ed0e0dc0 2024-03-18T22:53:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.790+0000 7f35eb66d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f35e41a91a0 con 0x55d2ed0db260 2024-03-18T22:53:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.790+0000 7f35eb66d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f35e41a63b0 con 0x55d2ed0dbc50 2024-03-18T22:53:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.791+0000 7f35eb66d740 10 client.0 _ll_register_callbacks cb 0x7f35dc007660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:53:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.791+0000 7f35e1d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0ea8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.791+0000 7f35e1d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0ea8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:46534/0 (socket says 172.21.15.40:46534) 2024-03-18T22:53:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.791+0000 7f35e1d9b640 1 -- 172.21.15.40:0/3890963877 learned_addr learned my addr 172.21.15.40:0/3890963877 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:53:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.791+0000 7f35e2d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0db260 0x55d2ed0ea390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.791+0000 7f35e2d9d640 1 --2- 172.21.15.40:0/3890963877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0eb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.791+0000 7f35e1d9b640 1 -- 172.21.15.40:0/3890963877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0db260 msgr2=0x55d2ed0ea390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.791+0000 7f35e1d9b640 1 --2- 172.21.15.40:0/3890963877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0db260 0x55d2ed0ea390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.791+0000 7f35e1d9b640 1 -- 172.21.15.40:0/3890963877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0e0dc0 msgr2=0x55d2ed0eb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:53:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.791+0000 7f35e1d9b640 1 --2- 172.21.15.40:0/3890963877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0eb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.791+0000 7f35e1d9b640 1 -- 172.21.15.40:0/3890963877 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2ed0ebed0 con 0x55d2ed0dbc50 2024-03-18T22:53:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.791+0000 7f35e2d9d640 1 --2- 172.21.15.40:0/3890963877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0db260 0x55d2ed0ea390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:53:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.791+0000 7f35e1d9b640 1 --2- 172.21.15.40:0/3890963877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0ea8d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f35dc004a80 tx=0x7f35dc007e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f211e78922b2bf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:53:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.791+0000 7f35c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:53:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.792+0000 7f35c9ffb640 1 -- 172.21.15.40:0/3890963877 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc011830 con 0x55d2ed0dbc50 2024-03-18T22:53:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.792+0000 7f35eb66d740 1 -- 172.21.15.40:0/3890963877 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35dc0196d0 con 0x55d2ed0dbc50 2024-03-18T22:53:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.792+0000 7f35c9ffb640 1 -- 172.21.15.40:0/3890963877 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f35dc0119d0 con 0x55d2ed0dbc50 2024-03-18T22:53:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.792+0000 7f35c9ffb640 1 -- 172.21.15.40:0/3890963877 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f35dc0105a0 con 0x55d2ed0dbc50 2024-03-18T22:53:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.792+0000 7f35eb66d740 10 client.28075 fetch_fsmap learned FSMap version 37 2024-03-18T22:53:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.792+0000 7f35eb66d740 1 -- 172.21.15.40:0/3890963877 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55d2ed157750 con 0x55d2ed0dbc50 2024-03-18T22:53:47.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.792+0000 7f35c9ffb640 1 -- 172.21.15.40:0/3890963877 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35dc035420 con 0x55d2ed0dbc50 2024-03-18T22:53:47.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.793+0000 7f35c9ffb640 1 -- 172.21.15.40:0/3890963877 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f35dc02a030 con 0x55d2ed0dbc50 2024-03-18T22:53:47.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.793+0000 7f35eb66d740 10 client.28075 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:53:47.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.793+0000 7f35eb66d740 10 client.28075 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:53:47.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.793+0000 7f35eb66d740 20 client.28075 populate_metadata read hostname 'smithi040' 2024-03-18T22:53:47.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.793+0000 7f35eb66d740 1 --2- 172.21.15.40:0/3890963877 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2ed1580d0 0x55d2ed15a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:53:47.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.793+0000 7f35eb66d740 4 client.28075 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:53:47.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.793+0000 7f35eb66d740 1 -- 172.21.15.40:0/3890963877 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f35dc0196d0 con 0x55d2ed1580d0 2024-03-18T22:53:47.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.793+0000 7f35e2d9d640 1 --2- 172.21.15.40:0/3890963877 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2ed1580d0 0x55d2ed15a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:53:47.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.793+0000 7f35c9ffb640 10 client.28075 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:47.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.793+0000 7f35e2d9d640 1 --2- 172.21.15.40:0/3890963877 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2ed1580d0 0x55d2ed15a4f0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0e2e1cffb261ce1 server_cookie=31ac24d6f29d6166 in_seq=0 out_seq=0 2024-03-18T22:53:47.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.793+0000 7f35c9ffb640 1 -- 172.21.15.40:0/3890963877 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f35dc0196d0 con 0x55d2ed1580d0 2024-03-18T22:53:47.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.793+0000 7f35c9ffb640 10 client.28075 handle_command_reply: tid=0 2024-03-18T22:53:47.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.793+0000 7f35eb66d740 1 client.28075 shutdown 2024-03-18T22:53:47.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35eb66d740 1 -- 172.21.15.40:0/3890963877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0dbc50 msgr2=0x55d2ed0ea8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:47.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35eb66d740 1 --2- 172.21.15.40:0/3890963877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0ea8d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f35dc004a80 tx=0x7f35dc007e90 comp rx=0 tx=0).stop 2024-03-18T22:53:47.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35eb66d740 1 -- 172.21.15.40:0/3890963877 shutdown_connections 2024-03-18T22:53:47.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35eb66d740 1 --2- 172.21.15.40:0/3890963877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d2ed0e0dc0 0x55d2ed0eb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35eb66d740 1 --2- 172.21.15.40:0/3890963877 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55d2ed0dbc50 0x55d2ed0ea8d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35eb66d740 1 --2- 172.21.15.40:0/3890963877 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55d2ed1580d0 0x55d2ed15a4f0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35c9ffb640 0 client.28075 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:53:47.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35eb66d740 1 --2- 172.21.15.40:0/3890963877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2ed0db260 0x55d2ed0ea390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35eb66d740 1 -- 172.21.15.40:0/3890963877 >> 172.21.15.40:0/3890963877 conn(0x55d2ed0cdfc0 msgr2=0x55d2ed0d1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:47.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35eb66d740 1 -- 172.21.15.40:0/3890963877 shutdown_connections 2024-03-18T22:53:47.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35eb66d740 1 -- 172.21.15.40:0/3890963877 wait complete. 2024-03-18T22:53:47.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35eb66d740 20 client.28075 trim_cache size 0 max 16384 2024-03-18T22:53:47.800 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:53:47.800 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:53:47.801 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:53:47.801 INFO:teuthology.orchestra.run.smithi040.stdout: "6a7b3721-c0a7-4062-9ca5-671585e8609b": { 2024-03-18T22:53:47.801 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:53:47.801 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "6a7b3721-c0a7-4062-9ca5-671585e8609b", 2024-03-18T22:53:47.801 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:53:47.801 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:53:47.801 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:53:47.801 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:53:47.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35e8a81640 1 -- 172.21.15.40:0/915522812 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f35cc0a2900 msgr2=0x7f35cc0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:47.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35e8a81640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f35cc0a2900 0x7f35cc0a4dc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f35d809c400 tx=0x7f35d80d2c70 comp rx=0 tx=0).stop 2024-03-18T22:53:47.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35e8a81640 1 -- 172.21.15.40:0/915522812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e4100720 msgr2=0x7f35e40694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:53:47.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.794+0000 7f35e8a81640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e4100720 0x7f35e40694f0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f35d800e9c0 tx=0x7f35d800ee80 comp rx=0 tx=0).stop 2024-03-18T22:53:47.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.795+0000 7f35e8a81640 1 -- 172.21.15.40:0/915522812 shutdown_connections 2024-03-18T22:53:47.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.795+0000 7f35e8a81640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f35cc0a2900 0x7f35cc0a4dc0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.795+0000 7f35e8a81640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35e41022d0 0x7f35e419ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.795+0000 7f35e8a81640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f35e4101910 0x7f35e419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.795+0000 7f35e8a81640 1 --2- 172.21.15.40:0/915522812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35e4100720 0x7f35e40694f0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:53:47.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.795+0000 7f35e8a81640 1 -- 172.21.15.40:0/915522812 >> 172.21.15.40:0/915522812 conn(0x7f35e40fbcb0 msgr2=0x7f35e4104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:53:47.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.795+0000 7f35e8a81640 1 -- 172.21.15.40:0/915522812 shutdown_connections 2024-03-18T22:53:47.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:53:47.795+0000 7f35e8a81640 1 -- 172.21.15.40:0/915522812 wait complete. 2024-03-18T22:53:47.808 INFO:tasks.cephfs.filesystem:scrub status for tag:6a7b3721-c0a7-4062-9ca5-671585e8609b - {'path': '/', 'tag': '6a7b3721-c0a7-4062-9ca5-671585e8609b', 'options': 'recursive,force'} 2024-03-18T22:53:49.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:48 smithi113 ceph-mon[66775]: pgmap v685: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.3 MiB/s wr, 1.86k op/s 2024-03-18T22:53:49.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:48 smithi103 ceph-mon[66377]: pgmap v685: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.3 MiB/s wr, 1.86k op/s 2024-03-18T22:53:49.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:48 smithi040 ceph-mon[62643]: pgmap v685: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.3 MiB/s wr, 1.86k op/s 2024-03-18T22:53:51.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:50 smithi113 ceph-mon[66775]: pgmap v686: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.7 MiB/s wr, 1.97k op/s 2024-03-18T22:53:51.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:50 smithi103 ceph-mon[66377]: pgmap v686: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.7 MiB/s wr, 1.97k op/s 2024-03-18T22:53:51.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:50 smithi040 ceph-mon[62643]: pgmap v686: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 7.7 MiB/s wr, 1.97k op/s 2024-03-18T22:53:53.329 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:52 smithi040 ceph-mon[62643]: pgmap v687: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 1.64k op/s 2024-03-18T22:53:53.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:52 smithi113 ceph-mon[66775]: pgmap v687: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 1.64k op/s 2024-03-18T22:53:53.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:52 smithi103 ceph-mon[66377]: pgmap v687: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 1.64k op/s 2024-03-18T22:53:55.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:54 smithi113 ceph-mon[66775]: pgmap v688: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 6.7 MiB/s wr, 1.72k op/s 2024-03-18T22:53:55.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:54 smithi103 ceph-mon[66377]: pgmap v688: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 6.7 MiB/s wr, 1.72k op/s 2024-03-18T22:53:55.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:54 smithi040 ceph-mon[62643]: pgmap v688: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 6.7 MiB/s wr, 1.72k op/s 2024-03-18T22:53:57.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:56 smithi113 ceph-mon[66775]: pgmap v689: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 1.64k op/s 2024-03-18T22:53:57.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:56 smithi103 ceph-mon[66377]: pgmap v689: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 1.64k op/s 2024-03-18T22:53:57.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:56 smithi040 ceph-mon[62643]: pgmap v689: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 1.64k op/s 2024-03-18T22:53:59.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:53:58 smithi113 ceph-mon[66775]: pgmap v690: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.6 MiB/s wr, 1.44k op/s 2024-03-18T22:53:59.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:53:58 smithi103 ceph-mon[66377]: pgmap v690: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.6 MiB/s wr, 1.44k op/s 2024-03-18T22:53:59.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:53:58 smithi040 ceph-mon[62643]: pgmap v690: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.6 MiB/s wr, 1.44k op/s 2024-03-18T22:54:01.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:00 smithi113 ceph-mon[66775]: pgmap v691: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.9 MiB/s wr, 1.51k op/s 2024-03-18T22:54:01.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:00 smithi103 ceph-mon[66377]: pgmap v691: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.9 MiB/s wr, 1.51k op/s 2024-03-18T22:54:01.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:00 smithi040 ceph-mon[62643]: pgmap v691: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.9 MiB/s wr, 1.51k op/s 2024-03-18T22:54:03.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:02 smithi113 ceph-mon[66775]: pgmap v692: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 MiB/s wr, 1.34k op/s 2024-03-18T22:54:03.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:02 smithi103 ceph-mon[66377]: pgmap v692: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 MiB/s wr, 1.34k op/s 2024-03-18T22:54:03.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:02 smithi040 ceph-mon[62643]: pgmap v692: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.2 MiB/s wr, 1.34k op/s 2024-03-18T22:54:05.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:04 smithi113 ceph-mon[66775]: pgmap v693: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.1 MiB/s wr, 1.57k op/s 2024-03-18T22:54:05.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:04 smithi103 ceph-mon[66377]: pgmap v693: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.1 MiB/s wr, 1.57k op/s 2024-03-18T22:54:05.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:04 smithi040 ceph-mon[62643]: pgmap v693: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.1 MiB/s wr, 1.57k op/s 2024-03-18T22:54:07.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:06 smithi113 ceph-mon[66775]: pgmap v694: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.3 MiB/s wr, 1.61k op/s 2024-03-18T22:54:07.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:06 smithi103 ceph-mon[66377]: pgmap v694: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.3 MiB/s wr, 1.61k op/s 2024-03-18T22:54:07.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:06 smithi040 ceph-mon[62643]: pgmap v694: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.3 MiB/s wr, 1.61k op/s 2024-03-18T22:54:09.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:08 smithi113 ceph-mon[66775]: pgmap v695: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.2 MiB/s wr, 1.59k op/s 2024-03-18T22:54:09.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:08 smithi103 ceph-mon[66377]: pgmap v695: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.2 MiB/s wr, 1.59k op/s 2024-03-18T22:54:09.396 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:08 smithi040 ceph-mon[62643]: pgmap v695: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.2 MiB/s wr, 1.59k op/s 2024-03-18T22:54:11.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:10 smithi113 ceph-mon[66775]: pgmap v696: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.8 MiB/s wr, 1.74k op/s 2024-03-18T22:54:11.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:10 smithi103 ceph-mon[66377]: pgmap v696: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.8 MiB/s wr, 1.74k op/s 2024-03-18T22:54:11.398 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:10 smithi040 ceph-mon[62643]: pgmap v696: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.8 MiB/s wr, 1.74k op/s 2024-03-18T22:54:13.252 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:12 smithi103 ceph-mon[66377]: pgmap v697: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 1.67k op/s 2024-03-18T22:54:13.331 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:12 smithi113 ceph-mon[66775]: pgmap v697: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 1.67k op/s 2024-03-18T22:54:13.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:12 smithi040 ceph-mon[62643]: pgmap v697: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 1.67k op/s 2024-03-18T22:54:14.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:13 smithi113 ceph-mon[66775]: pgmap v698: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.0 MiB/s wr, 2.06k op/s 2024-03-18T22:54:14.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:13 smithi103 ceph-mon[66377]: pgmap v698: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.0 MiB/s wr, 2.06k op/s 2024-03-18T22:54:14.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:13 smithi040 ceph-mon[62643]: pgmap v698: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.0 MiB/s wr, 2.06k op/s 2024-03-18T22:54:16.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:16 smithi103 ceph-mon[66377]: pgmap v699: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 2.12k op/s 2024-03-18T22:54:16.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:16 smithi040 ceph-mon[62643]: pgmap v699: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 2.12k op/s 2024-03-18T22:54:16.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:16 smithi113 ceph-mon[66775]: pgmap v699: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 2.12k op/s 2024-03-18T22:54:17.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:54:17.810 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:54:18.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.007+0000 7f17f2700640 1 -- 172.21.15.40:0/1805265703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17ec072cb0 msgr2=0x7f17ec0730d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:18.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.007+0000 7f17f2700640 1 --2- 172.21.15.40:0/1805265703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17ec072cb0 0x7f17ec0730d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f17dc009860 tx=0x7f17dc02bbb0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.010+0000 7f17f2700640 1 -- 172.21.15.40:0/1805265703 shutdown_connections 2024-03-18T22:54:18.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.010+0000 7f17f2700640 1 --2- 172.21.15.40:0/1805265703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17ec1098b0 0x7f17ec10e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.010+0000 7f17f2700640 1 --2- 172.21.15.40:0/1805265703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17ec108ef0 0x7f17ec109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.010+0000 7f17f2700640 1 --2- 172.21.15.40:0/1805265703 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17ec072cb0 0x7f17ec0730d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.010+0000 7f17f2700640 1 -- 172.21.15.40:0/1805265703 >> 172.21.15.40:0/1805265703 conn(0x7f17ec06e700 msgr2=0x7f17ec070b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:54:18.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.010+0000 7f17f2700640 1 -- 172.21.15.40:0/1805265703 shutdown_connections 2024-03-18T22:54:18.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.010+0000 7f17f2700640 1 -- 172.21.15.40:0/1805265703 wait complete. 2024-03-18T22:54:18.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.011+0000 7f17f2700640 1 Processor -- start 2024-03-18T22:54:18.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.011+0000 7f17f2700640 1 -- start start 2024-03-18T22:54:18.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.012+0000 7f17f2700640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17ec072cb0 0x7f17ec1a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.012+0000 7f17f2700640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17ec108ef0 0x7f17ec1a6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.012+0000 7f17f2700640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17ec1098b0 0x7f17ec1add00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.012+0000 7f17f2700640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec11c080 con 0x7f17ec1098b0 2024-03-18T22:54:18.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.012+0000 7f17f2700640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec11bf00 con 0x7f17ec108ef0 2024-03-18T22:54:18.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.012+0000 7f17f2700640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec11c200 con 0x7f17ec072cb0 2024-03-18T22:54:18.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.013+0000 7f17f16fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17ec072cb0 0x7f17ec1a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.013+0000 7f17f16fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17ec072cb0 0x7f17ec1a6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57372/0 (socket says 172.21.15.40:57372) 2024-03-18T22:54:18.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.013+0000 7f17f16fe640 1 -- 172.21.15.40:0/4023785013 learned_addr learned my addr 172.21.15.40:0/4023785013 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:54:18.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.013+0000 7f17f0efd640 1 --2- 172.21.15.40:0/4023785013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17ec108ef0 0x7f17ec1a6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.014+0000 7f17f1eff640 1 --2- 172.21.15.40:0/4023785013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17ec1098b0 0x7f17ec1add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.015+0000 7f17f0efd640 1 -- 172.21.15.40:0/4023785013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17ec072cb0 msgr2=0x7f17ec1a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:18.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.015+0000 7f17f0efd640 1 --2- 172.21.15.40:0/4023785013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17ec072cb0 0x7f17ec1a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.015+0000 7f17f0efd640 1 -- 172.21.15.40:0/4023785013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17ec1098b0 msgr2=0x7f17ec1add00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:18.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.015+0000 7f17f0efd640 1 --2- 172.21.15.40:0/4023785013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17ec1098b0 0x7f17ec1add00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.015+0000 7f17f0efd640 1 -- 172.21.15.40:0/4023785013 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ec1ae370 con 0x7f17ec108ef0 2024-03-18T22:54:18.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.015+0000 7f17f1eff640 1 --2- 172.21.15.40:0/4023785013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17ec1098b0 0x7f17ec1add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:54:18.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.017+0000 7f17f0efd640 1 --2- 172.21.15.40:0/4023785013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17ec108ef0 0x7f17ec1a6c90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f17ec0729d0 tx=0x7f17e800dbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:54:18.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.020+0000 7f17e27fc640 1 -- 172.21.15.40:0/4023785013 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8017070 con 0x7f17ec108ef0 2024-03-18T22:54:18.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.021+0000 7f17f2700640 1 -- 172.21.15.40:0/4023785013 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17ec1ae660 con 0x7f17ec108ef0 2024-03-18T22:54:18.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.021+0000 7f17f2700640 1 -- 172.21.15.40:0/4023785013 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17ec1aec30 con 0x7f17ec108ef0 2024-03-18T22:54:18.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.021+0000 7f17e27fc640 1 -- 172.21.15.40:0/4023785013 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17e8010460 con 0x7f17ec108ef0 2024-03-18T22:54:18.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.021+0000 7f17e27fc640 1 -- 172.21.15.40:0/4023785013 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e800f600 con 0x7f17ec108ef0 2024-03-18T22:54:18.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.025+0000 7f17e27fc640 1 -- 172.21.15.40:0/4023785013 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f17e801a070 con 0x7f17ec108ef0 2024-03-18T22:54:18.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.026+0000 7f17e27fc640 1 --2- 172.21.15.40:0/4023785013 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17d80a29d0 0x7f17d80a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.027+0000 7f17e27fc640 1 -- 172.21.15.40:0/4023785013 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f17e80ce6c0 con 0x7f17ec108ef0 2024-03-18T22:54:18.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.028+0000 7f17f16fe640 1 --2- 172.21.15.40:0/4023785013 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17d80a29d0 0x7f17d80a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.030+0000 7f17f662e740 1 Processor -- start 2024-03-18T22:54:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.030+0000 7f17f662e740 1 -- start start 2024-03-18T22:54:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.030+0000 7f17f662e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f333c10 0x562c8f334030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.030+0000 7f17f662e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f334600 0x562c8f339100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.030+0000 7f17f662e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f339770 0x562c8f33bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.030+0000 7f17f662e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562c8f298500 con 0x562c8f333c10 2024-03-18T22:54:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.030+0000 7f17f662e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562c8f240be0 con 0x562c8f339770 2024-03-18T22:54:18.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.031+0000 7f17f662e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x562c8f2945a0 con 0x562c8f334600 2024-03-18T22:54:18.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.031+0000 7f17f1eff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f339770 0x562c8f33bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.031+0000 7f17f1eff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f339770 0x562c8f33bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:37914/0 (socket says 172.21.15.40:37914) 2024-03-18T22:54:18.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.031+0000 7f17f1eff640 1 -- 172.21.15.40:0/519489080 learned_addr learned my addr 172.21.15.40:0/519489080 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:54:18.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.031+0000 7f17f0efd640 1 --2- 172.21.15.40:0/519489080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f334600 0x562c8f339100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.031+0000 7f17f16fe640 1 --2- 172.21.15.40:0/4023785013 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17d80a29d0 0x7f17d80a4e90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f17dc005e00 tx=0x7f17dc035df0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:54:18.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.031+0000 7f17f16fe640 1 --2- 172.21.15.40:0/519489080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f333c10 0x562c8f334030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.035+0000 7f17f1eff640 1 -- 172.21.15.40:0/519489080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f334600 msgr2=0x562c8f339100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:54:18.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.035+0000 7f17f1eff640 1 --2- 172.21.15.40:0/519489080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f334600 0x562c8f339100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.035+0000 7f17f1eff640 1 -- 172.21.15.40:0/519489080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f333c10 msgr2=0x562c8f334030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:54:18.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.035+0000 7f17f1eff640 1 --2- 172.21.15.40:0/519489080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f333c10 0x562c8f334030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.035+0000 7f17f1eff640 1 -- 172.21.15.40:0/519489080 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c8f262000 con 0x562c8f339770 2024-03-18T22:54:18.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.039+0000 7f17f1eff640 1 --2- 172.21.15.40:0/519489080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f339770 0x562c8f33bc00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f17e400c450 tx=0x7f17e400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65da58362c42d254 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:54:18.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.045+0000 7f17d77fe640 1 -- 172.21.15.40:0/519489080 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4007430 con 0x562c8f339770 2024-03-18T22:54:18.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.045+0000 7f17d77fe640 1 -- 172.21.15.40:0/519489080 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17e4007a80 con 0x562c8f339770 2024-03-18T22:54:18.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.045+0000 7f17d77fe640 1 -- 172.21.15.40:0/519489080 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e400aa60 con 0x562c8f339770 2024-03-18T22:54:18.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.046+0000 7f17f662e740 1 -- 172.21.15.40:0/519489080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f339770 msgr2=0x562c8f33bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:18.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.046+0000 7f17f662e740 1 --2- 172.21.15.40:0/519489080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f339770 0x562c8f33bc00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f17e400c450 tx=0x7f17e400c910 comp rx=0 tx=0).stop 2024-03-18T22:54:18.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.046+0000 7f17f662e740 1 -- 172.21.15.40:0/519489080 shutdown_connections 2024-03-18T22:54:18.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.046+0000 7f17f662e740 1 --2- 172.21.15.40:0/519489080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f339770 0x562c8f33bc00 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.046+0000 7f17f662e740 1 --2- 172.21.15.40:0/519489080 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f334600 0x562c8f339100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.046+0000 7f17f662e740 1 --2- 172.21.15.40:0/519489080 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f333c10 0x562c8f334030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.046+0000 7f17f662e740 1 -- 172.21.15.40:0/519489080 >> 172.21.15.40:0/519489080 conn(0x562c8f32f180 msgr2=0x562c8f32f580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:54:18.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.052+0000 7f17f662e740 1 -- 172.21.15.40:0/519489080 shutdown_connections 2024-03-18T22:54:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.054+0000 7f17f662e740 1 -- 172.21.15.40:0/519489080 wait complete. 2024-03-18T22:54:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.055+0000 7f17f662e740 1 -- start start 2024-03-18T22:54:18.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.055+0000 7f17f662e740 1 Processor -- start 2024-03-18T22:54:18.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.060+0000 7f17f662e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f333c10 0x562c8f34f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.060+0000 7f17f662e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f334600 0x562c8f34f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.060+0000 7f17f662e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f339770 0x562c8f356610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.060+0000 7f17f662e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562c8f34d730 con 0x562c8f334600 2024-03-18T22:54:18.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.060+0000 7f17f662e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562c8f254160 con 0x562c8f333c10 2024-03-18T22:54:18.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.060+0000 7f17f662e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x562c8f34d8b0 con 0x562c8f339770 2024-03-18T22:54:18.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.060+0000 7f17f662e740 10 client.0 _ll_register_callbacks cb 0x562c8f290a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:54:18.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.061+0000 7f17f1eff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f339770 0x562c8f356610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.061+0000 7f17f1eff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f339770 0x562c8f356610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57398/0 (socket says 172.21.15.40:57398) 2024-03-18T22:54:18.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.062+0000 7f17f1eff640 1 -- 172.21.15.40:0/4111563762 learned_addr learned my addr 172.21.15.40:0/4111563762 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:54:18.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.062+0000 7f17f16fe640 1 --2- 172.21.15.40:0/4111563762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f333c10 0x562c8f34f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.063+0000 7f17f16fe640 1 -- 172.21.15.40:0/4111563762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f339770 msgr2=0x562c8f356610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:54:18.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.063+0000 7f17f16fe640 1 --2- 172.21.15.40:0/4111563762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f339770 0x562c8f356610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.064+0000 7f17f0efd640 1 --2- 172.21.15.40:0/4111563762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f334600 0x562c8f34f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.067+0000 7f17f16fe640 1 -- 172.21.15.40:0/4111563762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f334600 msgr2=0x562c8f34f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:54:18.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.067+0000 7f17f16fe640 1 --2- 172.21.15.40:0/4111563762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f334600 0x562c8f34f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.067+0000 7f17f16fe640 1 -- 172.21.15.40:0/4111563762 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c8f0767d0 con 0x562c8f333c10 2024-03-18T22:54:18.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.067+0000 7f17f0efd640 1 --2- 172.21.15.40:0/4111563762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f334600 0x562c8f34f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:54:18.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.069+0000 7f17f16fe640 1 --2- 172.21.15.40:0/4111563762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f333c10 0x562c8f34f090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0095f0 tx=0x7f17dc02bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82f7dfaeb31f45d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:54:18.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.070+0000 7f17d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:54:18.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.070+0000 7f17d4ff9640 1 -- 172.21.15.40:0/4111563762 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc039070 con 0x562c8f333c10 2024-03-18T22:54:18.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.070+0000 7f17d4ff9640 1 -- 172.21.15.40:0/4111563762 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f17dc005250 con 0x562c8f333c10 2024-03-18T22:54:18.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.072+0000 7f17f662e740 1 -- 172.21.15.40:0/4111563762 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17e4007a80 con 0x562c8f333c10 2024-03-18T22:54:18.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.077+0000 7f17d4ff9640 1 -- 172.21.15.40:0/4111563762 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f17dc005490 con 0x562c8f333c10 2024-03-18T22:54:18.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.082+0000 7f17f662e740 10 client.28103 fetch_fsmap learned FSMap version 37 2024-03-18T22:54:18.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.082+0000 7f17f662e740 1 -- 172.21.15.40:0/4111563762 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562c8efac3f0 con 0x562c8f333c10 2024-03-18T22:54:18.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.084+0000 7f17d4ff9640 1 -- 172.21.15.40:0/4111563762 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17e4007a80 con 0x562c8f333c10 2024-03-18T22:54:18.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.084+0000 7f17d4ff9640 1 -- 172.21.15.40:0/4111563762 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f17dc003800 con 0x562c8f333c10 2024-03-18T22:54:18.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.086+0000 7f17f662e740 10 client.28103 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:54:18.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.086+0000 7f17f662e740 10 client.28103 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:54:18.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.086+0000 7f17f662e740 20 client.28103 populate_metadata read hostname 'smithi040' 2024-03-18T22:54:18.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.086+0000 7f17f662e740 1 --2- 172.21.15.40:0/4111563762 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c8f35ae30 0x562c8f35d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.086+0000 7f17f662e740 4 client.28103 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:54:18.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.086+0000 7f17f662e740 1 -- 172.21.15.40:0/4111563762 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c8f2a5a40 con 0x562c8f35ae30 2024-03-18T22:54:18.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.093+0000 7f17f0efd640 1 --2- 172.21.15.40:0/4111563762 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c8f35ae30 0x562c8f35d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.096+0000 7f17f0efd640 1 --2- 172.21.15.40:0/4111563762 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c8f35ae30 0x562c8f35d2f0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87d607ae49ce992c server_cookie=7f9cbde6987de4eb in_seq=0 out_seq=0 2024-03-18T22:54:18.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.096+0000 7f17d4ff9640 10 client.28103 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:54:18.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.098+0000 7f17d4ff9640 1 -- 172.21.15.40:0/4111563762 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562c8f2a5a40 con 0x562c8f35ae30 2024-03-18T22:54:18.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.098+0000 7f17d4ff9640 10 client.28103 handle_command_reply: tid=0 2024-03-18T22:54:18.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.098+0000 7f17f662e740 1 client.28103 shutdown 2024-03-18T22:54:18.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.100+0000 7f17f662e740 1 -- 172.21.15.40:0/4111563762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f333c10 msgr2=0x562c8f34f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:18.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.100+0000 7f17f662e740 1 --2- 172.21.15.40:0/4111563762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f333c10 0x562c8f34f090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0095f0 tx=0x7f17dc02bbb0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.112+0000 7f17f662e740 1 -- 172.21.15.40:0/4111563762 shutdown_connections 2024-03-18T22:54:18.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.112+0000 7f17f662e740 1 --2- 172.21.15.40:0/4111563762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f339770 0x562c8f356610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.112+0000 7f17f662e740 1 --2- 172.21.15.40:0/4111563762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f334600 0x562c8f34f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.112+0000 7f17f662e740 1 --2- 172.21.15.40:0/4111563762 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c8f35ae30 0x562c8f35d2f0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.112+0000 7f17f662e740 1 --2- 172.21.15.40:0/4111563762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f333c10 0x562c8f34f090 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.112+0000 7f17f662e740 1 -- 172.21.15.40:0/4111563762 >> 172.21.15.40:0/4111563762 conn(0x562c8f32f180 msgr2=0x562c8f32e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:54:18.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.112+0000 7f17d4ff9640 0 client.28103 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:54:18.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.113+0000 7f17f662e740 1 -- 172.21.15.40:0/4111563762 shutdown_connections 2024-03-18T22:54:18.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.113+0000 7f17f662e740 1 -- 172.21.15.40:0/4111563762 wait complete. 2024-03-18T22:54:18.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.114+0000 7f17f662e740 20 client.28103 trim_cache size 0 max 16384 2024-03-18T22:54:18.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.172+0000 7f17f662e740 1 Processor -- start 2024-03-18T22:54:18.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.172+0000 7f17f662e740 1 -- start start 2024-03-18T22:54:18.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.172+0000 7f17f662e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f333c10 0x562c8f34eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.172+0000 7f17f662e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f334600 0x562c8f3565d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.172+0000 7f17f662e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f339770 0x562c8f34f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.172+0000 7f17f662e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f17d000fe70 con 0x562c8f334600 2024-03-18T22:54:18.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.172+0000 7f17f662e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f17d0011aa0 con 0x562c8f339770 2024-03-18T22:54:18.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.172+0000 7f17f662e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f17d0010540 con 0x562c8f333c10 2024-03-18T22:54:18.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.173+0000 7f17f1eff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f339770 0x562c8f34f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.173+0000 7f17f1eff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f339770 0x562c8f34f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:37932/0 (socket says 172.21.15.40:37932) 2024-03-18T22:54:18.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.173+0000 7f17f1eff640 1 -- 172.21.15.40:0/94852320 learned_addr learned my addr 172.21.15.40:0/94852320 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:54:18.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.174+0000 7f17f0efd640 1 --2- 172.21.15.40:0/94852320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f334600 0x562c8f3565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.174+0000 7f17f0efd640 1 -- 172.21.15.40:0/94852320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f333c10 msgr2=0x562c8f34eca0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:54:18.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.174+0000 7f17f0efd640 1 --2- 172.21.15.40:0/94852320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f333c10 0x562c8f34eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.174+0000 7f17f0efd640 1 -- 172.21.15.40:0/94852320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f339770 msgr2=0x562c8f34f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:54:18.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.174+0000 7f17f0efd640 1 --2- 172.21.15.40:0/94852320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f339770 0x562c8f34f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.174+0000 7f17f0efd640 1 -- 172.21.15.40:0/94852320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c8ef6c0f0 con 0x562c8f334600 2024-03-18T22:54:18.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.175+0000 7f17f0efd640 1 --2- 172.21.15.40:0/94852320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f334600 0x562c8f3565d0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f17e80cbd70 tx=0x7f17e809bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ac20509a4d3cd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:54:18.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.175+0000 7f17d6ffd640 1 -- 172.21.15.40:0/94852320 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e80b8070 con 0x562c8f334600 2024-03-18T22:54:18.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.175+0000 7f17d6ffd640 1 -- 172.21.15.40:0/94852320 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17e80cbe80 con 0x562c8f334600 2024-03-18T22:54:18.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.175+0000 7f17d6ffd640 1 -- 172.21.15.40:0/94852320 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8021040 con 0x562c8f334600 2024-03-18T22:54:18.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.176+0000 7f17f662e740 1 -- 172.21.15.40:0/94852320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f334600 msgr2=0x562c8f3565d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:18.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.176+0000 7f17f662e740 1 --2- 172.21.15.40:0/94852320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f334600 0x562c8f3565d0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f17e80cbd70 tx=0x7f17e809bac0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.179+0000 7f17f662e740 1 -- 172.21.15.40:0/94852320 shutdown_connections 2024-03-18T22:54:18.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.179+0000 7f17f662e740 1 --2- 172.21.15.40:0/94852320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f339770 0x562c8f34f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.179+0000 7f17f662e740 1 --2- 172.21.15.40:0/94852320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f334600 0x562c8f3565d0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.179+0000 7f17f662e740 1 --2- 172.21.15.40:0/94852320 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f333c10 0x562c8f34eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.179+0000 7f17f662e740 1 -- 172.21.15.40:0/94852320 >> 172.21.15.40:0/94852320 conn(0x562c8f32f180 msgr2=0x562c8f335b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:54:18.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.185+0000 7f17f662e740 1 -- 172.21.15.40:0/94852320 shutdown_connections 2024-03-18T22:54:18.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.185+0000 7f17f662e740 1 -- 172.21.15.40:0/94852320 wait complete. 2024-03-18T22:54:18.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.185+0000 7f17f662e740 1 -- start start 2024-03-18T22:54:18.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.185+0000 7f17f662e740 1 Processor -- start 2024-03-18T22:54:18.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.186+0000 7f17f662e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f333c10 0x562c8f353920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.186+0000 7f17f662e740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f334600 0x562c8f353e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.186+0000 7f17f662e740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f339770 0x562c8f354e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.186+0000 7f17f662e740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f17d00113d0 con 0x562c8f339770 2024-03-18T22:54:18.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.186+0000 7f17f662e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f17d0010c40 con 0x562c8f333c10 2024-03-18T22:54:18.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.186+0000 7f17f662e740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f17d000de50 con 0x562c8f334600 2024-03-18T22:54:18.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.186+0000 7f17f662e740 10 client.0 _ll_register_callbacks cb 0x7f17dc0096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:54:18.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.186+0000 7f17f0efd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f334600 0x562c8f353e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.187+0000 7f17f0efd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f334600 0x562c8f353e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57418/0 (socket says 172.21.15.40:57418) 2024-03-18T22:54:18.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.187+0000 7f17f0efd640 1 -- 172.21.15.40:0/2862228487 learned_addr learned my addr 172.21.15.40:0/2862228487 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:54:18.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.187+0000 7f17f0efd640 1 -- 172.21.15.40:0/2862228487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f333c10 msgr2=0x562c8f353920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:54:18.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.187+0000 7f17f16fe640 1 --2- 172.21.15.40:0/2862228487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f333c10 0x562c8f353920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.187+0000 7f17f0efd640 1 --2- 172.21.15.40:0/2862228487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f333c10 0x562c8f353920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.187+0000 7f17f0efd640 1 -- 172.21.15.40:0/2862228487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f339770 msgr2=0x562c8f354e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:54:18.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.187+0000 7f17f0efd640 1 --2- 172.21.15.40:0/2862228487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f339770 0x562c8f354e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.187+0000 7f17f0efd640 1 -- 172.21.15.40:0/2862228487 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c8f355460 con 0x562c8f334600 2024-03-18T22:54:18.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.188+0000 7f17f0efd640 1 --2- 172.21.15.40:0/2862228487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f334600 0x562c8f353e60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f17e8005fd0 tx=0x7f17e80cc030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d9a942f08853f61 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:54:18.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.188+0000 7f17d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:54:18.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.188+0000 7f17f16fe640 1 --2- 172.21.15.40:0/2862228487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f333c10 0x562c8f353920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:54:18.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.190+0000 7f17d4ff9640 1 -- 172.21.15.40:0/2862228487 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e80a9070 con 0x562c8f334600 2024-03-18T22:54:18.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.191+0000 7f17f662e740 1 -- 172.21.15.40:0/2862228487 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17e80cbe80 con 0x562c8f334600 2024-03-18T22:54:18.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.191+0000 7f17d4ff9640 1 -- 172.21.15.40:0/2862228487 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f17e80b9490 con 0x562c8f334600 2024-03-18T22:54:18.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.193+0000 7f17d4ff9640 1 -- 172.21.15.40:0/2862228487 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f17e8081050 con 0x562c8f334600 2024-03-18T22:54:18.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.194+0000 7f17f662e740 10 client.28090 fetch_fsmap learned FSMap version 37 2024-03-18T22:54:18.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.194+0000 7f17f662e740 1 -- 172.21.15.40:0/2862228487 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562c8f356810 con 0x562c8f334600 2024-03-18T22:54:18.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.196+0000 7f17d4ff9640 1 -- 172.21.15.40:0/2862228487 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17e80cbe80 con 0x562c8f334600 2024-03-18T22:54:18.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.196+0000 7f17d4ff9640 1 -- 172.21.15.40:0/2862228487 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 37) ==== 8382+0+0 (secure 0 0 0) 0x7f17e809aae0 con 0x562c8f334600 2024-03-18T22:54:18.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.196+0000 7f17f662e740 10 client.28090 fetch_fsmap finished waiting for FSMap version 37 2024-03-18T22:54:18.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.196+0000 7f17f662e740 10 client.28090 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:54:18.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.196+0000 7f17f662e740 20 client.28090 populate_metadata read hostname 'smithi040' 2024-03-18T22:54:18.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.196+0000 7f17f662e740 1 --2- 172.21.15.40:0/2862228487 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c8f3c1840 0x562c8f3c3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:18.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.196+0000 7f17f662e740 4 client.28090 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:54:18.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.196+0000 7f17f662e740 1 -- 172.21.15.40:0/2862228487 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562c8f29fd90 con 0x562c8f3c1840 2024-03-18T22:54:18.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.201+0000 7f17f16fe640 1 --2- 172.21.15.40:0/2862228487 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c8f3c1840 0x562c8f3c3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:18.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.204+0000 7f17f16fe640 1 --2- 172.21.15.40:0/2862228487 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c8f3c1840 0x562c8f3c3c60 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=999c0ce041b50381 server_cookie=6dcd09c4a946192c in_seq=0 out_seq=0 2024-03-18T22:54:18.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.204+0000 7f17d4ff9640 10 client.28090 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:54:18.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.206+0000 7f17d4ff9640 1 -- 172.21.15.40:0/2862228487 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562c8f29fd90 con 0x562c8f3c1840 2024-03-18T22:54:18.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.206+0000 7f17d4ff9640 10 client.28090 handle_command_reply: tid=0 2024-03-18T22:54:18.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.207+0000 7f17f662e740 1 client.28090 shutdown 2024-03-18T22:54:18.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.210+0000 7f17f662e740 1 -- 172.21.15.40:0/2862228487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f334600 msgr2=0x562c8f353e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:18.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.210+0000 7f17f662e740 1 --2- 172.21.15.40:0/2862228487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f334600 0x562c8f353e60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f17e8005fd0 tx=0x7f17e80cc030 comp rx=0 tx=0).stop 2024-03-18T22:54:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.210+0000 7f17f662e740 1 -- 172.21.15.40:0/2862228487 shutdown_connections 2024-03-18T22:54:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.210+0000 7f17f662e740 1 --2- 172.21.15.40:0/2862228487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562c8f339770 0x562c8f354e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.210+0000 7f17f662e740 1 --2- 172.21.15.40:0/2862228487 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562c8f334600 0x562c8f353e60 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.210+0000 7f17f662e740 1 --2- 172.21.15.40:0/2862228487 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562c8f3c1840 0x562c8f3c3c60 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.210+0000 7f17f662e740 1 --2- 172.21.15.40:0/2862228487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c8f333c10 0x562c8f353920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.210+0000 7f17f662e740 1 -- 172.21.15.40:0/2862228487 >> 172.21.15.40:0/2862228487 conn(0x562c8f32f180 msgr2=0x562c8f34ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:54:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.210+0000 7f17d4ff9640 0 client.28090 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:54:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.210+0000 7f17f662e740 1 -- 172.21.15.40:0/2862228487 shutdown_connections 2024-03-18T22:54:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.210+0000 7f17f662e740 1 -- 172.21.15.40:0/2862228487 wait complete. 2024-03-18T22:54:18.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.210+0000 7f17f662e740 20 client.28090 trim_cache size 0 max 16384 2024-03-18T22:54:18.212 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:54:18.212 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:54:18.212 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:54:18.212 INFO:teuthology.orchestra.run.smithi040.stdout: "6a7b3721-c0a7-4062-9ca5-671585e8609b": { 2024-03-18T22:54:18.212 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:54:18.212 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "6a7b3721-c0a7-4062-9ca5-671585e8609b", 2024-03-18T22:54:18.212 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:54:18.212 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:54:18.212 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:54:18.212 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:54:18.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.213+0000 7f17f2700640 1 -- 172.21.15.40:0/4023785013 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17d80a29d0 msgr2=0x7f17d80a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:18.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.213+0000 7f17f2700640 1 --2- 172.21.15.40:0/4023785013 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17d80a29d0 0x7f17d80a4e90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f17dc005e00 tx=0x7f17dc035df0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.213+0000 7f17f2700640 1 -- 172.21.15.40:0/4023785013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17ec108ef0 msgr2=0x7f17ec1a6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:18.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.214+0000 7f17f2700640 1 --2- 172.21.15.40:0/4023785013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17ec108ef0 0x7f17ec1a6c90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f17ec0729d0 tx=0x7f17e800dbb0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.215+0000 7f17f2700640 1 -- 172.21.15.40:0/4023785013 shutdown_connections 2024-03-18T22:54:18.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.216+0000 7f17f2700640 1 --2- 172.21.15.40:0/4023785013 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17d80a29d0 0x7f17d80a4e90 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.216+0000 7f17f2700640 1 --2- 172.21.15.40:0/4023785013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17ec1098b0 0x7f17ec1add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.216+0000 7f17f2700640 1 --2- 172.21.15.40:0/4023785013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17ec108ef0 0x7f17ec1a6c90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.216+0000 7f17f2700640 1 --2- 172.21.15.40:0/4023785013 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17ec072cb0 0x7f17ec1a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:18.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.216+0000 7f17f2700640 1 -- 172.21.15.40:0/4023785013 >> 172.21.15.40:0/4023785013 conn(0x7f17ec06e700 msgr2=0x7f17ec070400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:54:18.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.217+0000 7f17f2700640 1 -- 172.21.15.40:0/4023785013 shutdown_connections 2024-03-18T22:54:18.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:18.217+0000 7f17f2700640 1 -- 172.21.15.40:0/4023785013 wait complete. 2024-03-18T22:54:18.253 INFO:tasks.cephfs.filesystem:scrub status for tag:6a7b3721-c0a7-4062-9ca5-671585e8609b - {'path': '/', 'tag': '6a7b3721-c0a7-4062-9ca5-671585e8609b', 'options': 'recursive,force'} 2024-03-18T22:54:18.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:18 smithi040 ceph-mon[62643]: pgmap v700: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.0 MiB/s wr, 2.29k op/s 2024-03-18T22:54:18.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:18 smithi113 ceph-mon[66775]: pgmap v700: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.0 MiB/s wr, 2.29k op/s 2024-03-18T22:54:18.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:18 smithi103 ceph-mon[66377]: pgmap v700: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 9.0 MiB/s wr, 2.29k op/s 2024-03-18T22:54:20.397 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:20 smithi040 ceph-mon[62643]: pgmap v701: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 9.9 MiB/s wr, 2.54k op/s 2024-03-18T22:54:20.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:20 smithi113 ceph-mon[66775]: pgmap v701: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 9.9 MiB/s wr, 2.54k op/s 2024-03-18T22:54:20.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:20 smithi103 ceph-mon[66377]: pgmap v701: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 9.9 MiB/s wr, 2.54k op/s 2024-03-18T22:54:21.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:21 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:54:21.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:21 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:54:21.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:21 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:54:22.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:22 smithi113 ceph-mon[66775]: pgmap v702: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 2.47k op/s 2024-03-18T22:54:22.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:22 smithi103 ceph-mon[66377]: pgmap v702: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 2.47k op/s 2024-03-18T22:54:22.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:22 smithi040 ceph-mon[62643]: pgmap v702: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.7 MiB/s wr, 2.47k op/s 2024-03-18T22:54:24.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:24 smithi113 ceph-mon[66775]: pgmap v703: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 11 MiB/s wr, 2.85k op/s 2024-03-18T22:54:24.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:24 smithi103 ceph-mon[66377]: pgmap v703: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 11 MiB/s wr, 2.85k op/s 2024-03-18T22:54:24.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:24 smithi040 ceph-mon[62643]: pgmap v703: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 11 MiB/s wr, 2.85k op/s 2024-03-18T22:54:26.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:26 smithi113 ceph-mon[66775]: pgmap v704: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.81k op/s 2024-03-18T22:54:26.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:26 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:54:26.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:26 smithi103 ceph-mon[66377]: pgmap v704: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.81k op/s 2024-03-18T22:54:26.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:26 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:54:26.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:26 smithi040 ceph-mon[62643]: pgmap v704: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.81k op/s 2024-03-18T22:54:26.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:26 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:54:28.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:28 smithi113 ceph-mon[66775]: pgmap v705: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 11 MiB/s wr, 2.88k op/s 2024-03-18T22:54:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:28 smithi103 ceph-mon[66377]: pgmap v705: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 11 MiB/s wr, 2.88k op/s 2024-03-18T22:54:28.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:28 smithi040 ceph-mon[62643]: pgmap v705: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 11 MiB/s wr, 2.88k op/s 2024-03-18T22:54:30.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:30 smithi113 ceph-mon[66775]: pgmap v706: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 11 MiB/s wr, 2.92k op/s 2024-03-18T22:54:30.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:30 smithi103 ceph-mon[66377]: pgmap v706: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 11 MiB/s wr, 2.92k op/s 2024-03-18T22:54:30.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:30 smithi040 ceph-mon[62643]: pgmap v706: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 11 MiB/s wr, 2.92k op/s 2024-03-18T22:54:32.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:32 smithi113 ceph-mon[66775]: pgmap v707: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 MiB/s wr, 2.67k op/s 2024-03-18T22:54:32.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:32 smithi103 ceph-mon[66377]: pgmap v707: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 MiB/s wr, 2.67k op/s 2024-03-18T22:54:32.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:32 smithi040 ceph-mon[62643]: pgmap v707: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 MiB/s wr, 2.67k op/s 2024-03-18T22:54:33.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:33 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:54:33.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:33 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:54:33.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:33 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:54:34.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:34 smithi113 ceph-mon[66775]: pgmap v708: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 12 MiB/s wr, 3.03k op/s 2024-03-18T22:54:34.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:34 smithi103 ceph-mon[66377]: pgmap v708: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 12 MiB/s wr, 3.03k op/s 2024-03-18T22:54:34.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:34 smithi040 ceph-mon[62643]: pgmap v708: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 12 MiB/s wr, 3.03k op/s 2024-03-18T22:54:36.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:36 smithi113 ceph-mon[66775]: pgmap v709: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.87k op/s 2024-03-18T22:54:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:36 smithi103 ceph-mon[66377]: pgmap v709: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.87k op/s 2024-03-18T22:54:36.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:36 smithi040 ceph-mon[62643]: pgmap v709: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.87k op/s 2024-03-18T22:54:38.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:38 smithi113 ceph-mon[66775]: pgmap v710: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-03-18T22:54:38.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:38 smithi103 ceph-mon[66377]: pgmap v710: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-03-18T22:54:38.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:38 smithi040 ceph-mon[62643]: pgmap v710: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-03-18T22:54:40.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:40 smithi113 ceph-mon[66775]: pgmap v711: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 12 MiB/s wr, 2.96k op/s 2024-03-18T22:54:40.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:40 smithi103 ceph-mon[66377]: pgmap v711: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 12 MiB/s wr, 2.96k op/s 2024-03-18T22:54:40.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:40 smithi040 ceph-mon[62643]: pgmap v711: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 12 MiB/s wr, 2.96k op/s 2024-03-18T22:54:42.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:42 smithi113 ceph-mon[66775]: pgmap v712: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-03-18T22:54:42.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:42 smithi103 ceph-mon[66377]: pgmap v712: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-03-18T22:54:42.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:42 smithi040 ceph-mon[62643]: pgmap v712: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-03-18T22:54:44.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:44 smithi113 ceph-mon[66775]: pgmap v713: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 3.09k op/s 2024-03-18T22:54:44.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:44 smithi103 ceph-mon[66377]: pgmap v713: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 3.09k op/s 2024-03-18T22:54:44.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:44 smithi040 ceph-mon[62643]: pgmap v713: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 3.09k op/s 2024-03-18T22:54:45.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:45 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:54:45.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:45 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:54:45.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:45 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:54:47.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:46 smithi113 ceph-mon[66775]: pgmap v714: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.89k op/s 2024-03-18T22:54:47.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:46 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:46 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:46 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:46 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:46 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:46 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:46 smithi103 ceph-mon[66377]: pgmap v714: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.89k op/s 2024-03-18T22:54:47.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:46 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:46 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:46 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:46 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:46 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:46 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:46 smithi040 ceph-mon[62643]: pgmap v714: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.89k op/s 2024-03-18T22:54:47.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:46 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:46 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:46 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:46 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:46 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:47.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:46 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:48.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:47 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:54:48.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:47 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:54:48.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:47 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:48.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:47 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:54:48.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:47 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:54:48.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:47 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:48.255 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:54:48.255 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:54:48.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:47 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:54:48.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:47 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:54:48.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:47 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:54:48.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.430+0000 7fd1b6753640 1 -- 172.21.15.40:0/3098367862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b0102a40 msgr2=0x7fd1b0102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:48.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.430+0000 7fd1b6753640 1 --2- 172.21.15.40:0/3098367862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b0102a40 0x7fd1b0102ec0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009900 tx=0x7fd1a402bc30 comp rx=0 tx=0).stop 2024-03-18T22:54:48.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.430+0000 7fd1b6753640 1 -- 172.21.15.40:0/3098367862 shutdown_connections 2024-03-18T22:54:48.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.430+0000 7fd1b6753640 1 --2- 172.21.15.40:0/3098367862 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1b0103400 0x7fd1b0109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.430+0000 7fd1b6753640 1 --2- 172.21.15.40:0/3098367862 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b0102a40 0x7fd1b0102ec0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.430+0000 7fd1b6753640 1 --2- 172.21.15.40:0/3098367862 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b0101850 0x7fd1b0101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.430+0000 7fd1b6753640 1 -- 172.21.15.40:0/3098367862 >> 172.21.15.40:0/3098367862 conn(0x7fd1b00fcdc0 msgr2=0x7fd1b00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:54:48.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.430+0000 7fd1b6753640 1 -- 172.21.15.40:0/3098367862 shutdown_connections 2024-03-18T22:54:48.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.430+0000 7fd1b6753640 1 -- 172.21.15.40:0/3098367862 wait complete. 2024-03-18T22:54:48.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.431+0000 7fd1b6753640 1 Processor -- start 2024-03-18T22:54:48.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.431+0000 7fd1b6753640 1 -- start start 2024-03-18T22:54:48.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.431+0000 7fd1b6753640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b0101850 0x7fd1b019e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.431+0000 7fd1b6753640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b0102a40 0x7fd1b019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.431+0000 7fd1b6753640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1b0103400 0x7fd1b01a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.431+0000 7fd1b6753640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b010d3d0 con 0x7fd1b0102a40 2024-03-18T22:54:48.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.431+0000 7fd1b6753640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b010d250 con 0x7fd1b0101850 2024-03-18T22:54:48.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.431+0000 7fd1b6753640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b010d550 con 0x7fd1b0103400 2024-03-18T22:54:48.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.432+0000 7fd1b4cf0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1b0103400 0x7fd1b01a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.432+0000 7fd1b4cf0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1b0103400 0x7fd1b01a5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:51932/0 (socket says 172.21.15.40:51932) 2024-03-18T22:54:48.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.432+0000 7fd1b4cf0640 1 -- 172.21.15.40:0/716374151 learned_addr learned my addr 172.21.15.40:0/716374151 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:54:48.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.432+0000 7fd1af7fe640 1 --2- 172.21.15.40:0/716374151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b0102a40 0x7fd1b019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.432+0000 7fd1b4cf0640 1 -- 172.21.15.40:0/716374151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b0101850 msgr2=0x7fd1b019e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:48.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.432+0000 7fd1b4cf0640 1 --2- 172.21.15.40:0/716374151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b0101850 0x7fd1b019e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.432+0000 7fd1b4cf0640 1 -- 172.21.15.40:0/716374151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b0102a40 msgr2=0x7fd1b019e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:48.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.432+0000 7fd1b4cf0640 1 --2- 172.21.15.40:0/716374151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b0102a40 0x7fd1b019e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.432+0000 7fd1b4cf0640 1 -- 172.21.15.40:0/716374151 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b01a5ce0 con 0x7fd1b0103400 2024-03-18T22:54:48.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.432+0000 7fd1af7fe640 1 --2- 172.21.15.40:0/716374151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b0102a40 0x7fd1b019e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:54:48.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.433+0000 7fd1b4cf0640 1 --2- 172.21.15.40:0/716374151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1b0103400 0x7fd1b01a5670 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800eb10 tx=0x7fd1a800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:54:48.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.433+0000 7fd1ad7fa640 1 -- 172.21.15.40:0/716374151 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8004b20 con 0x7fd1b0103400 2024-03-18T22:54:48.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.433+0000 7fd1b6753640 1 -- 172.21.15.40:0/716374151 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b01a5fd0 con 0x7fd1b0103400 2024-03-18T22:54:48.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.433+0000 7fd1b6753640 1 -- 172.21.15.40:0/716374151 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b01a64e0 con 0x7fd1b0103400 2024-03-18T22:54:48.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.435+0000 7fd1ad7fa640 1 -- 172.21.15.40:0/716374151 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1a8004cc0 con 0x7fd1b0103400 2024-03-18T22:54:48.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.435+0000 7fd1ad7fa640 1 -- 172.21.15.40:0/716374151 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8010690 con 0x7fd1b0103400 2024-03-18T22:54:48.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.436+0000 7fd1b941d740 1 Processor -- start 2024-03-18T22:54:48.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.439+0000 7fd1ad7fa640 1 -- 172.21.15.40:0/716374151 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd1a8005020 con 0x7fd1b0103400 2024-03-18T22:54:48.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.439+0000 7fd1b941d740 1 -- start start 2024-03-18T22:54:48.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.439+0000 7fd1b941d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6267a50 0x55e8a6267e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.439+0000 7fd1b941d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6268440 0x55e8a626cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.439+0000 7fd1b941d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a626d5b0 0x55e8a626fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.439+0000 7fd1b941d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e8a61cc500 con 0x55e8a6267a50 2024-03-18T22:54:48.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.439+0000 7fd1b941d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e8a6174be0 con 0x55e8a6268440 2024-03-18T22:54:48.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.439+0000 7fd1b941d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e8a61c85a0 con 0x55e8a626d5b0 2024-03-18T22:54:48.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.440+0000 7fd1ad7fa640 1 --2- 172.21.15.40:0/716374151 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1980a28d0 0x7fd1980a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.440+0000 7fd1affff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6268440 0x55e8a626cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.440+0000 7fd1af7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6267a50 0x55e8a6267e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.440+0000 7fd1affff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6268440 0x55e8a626cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:57492/0 (socket says 172.21.15.40:57492) 2024-03-18T22:54:48.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.440+0000 7fd1affff640 1 -- 172.21.15.40:0/1645771643 learned_addr learned my addr 172.21.15.40:0/1645771643 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:54:48.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.440+0000 7fd1ad7fa640 1 -- 172.21.15.40:0/716374151 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd1a80cf570 con 0x7fd1b0103400 2024-03-18T22:54:48.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.440+0000 7fd1b4cf0640 1 --2- 172.21.15.40:0/716374151 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1980a28d0 0x7fd1980a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.440+0000 7fd1af7fe640 1 --2- 172.21.15.40:0/1645771643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a626d5b0 0x55e8a626fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.440+0000 7fd1affff640 1 -- 172.21.15.40:0/1645771643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a626d5b0 msgr2=0x55e8a626fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:54:48.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.440+0000 7fd1affff640 1 --2- 172.21.15.40:0/1645771643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a626d5b0 0x55e8a626fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.440+0000 7fd1affff640 1 -- 172.21.15.40:0/1645771643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6267a50 msgr2=0x55e8a6267e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:54:48.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.440+0000 7fd1affff640 1 --2- 172.21.15.40:0/1645771643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6267a50 0x55e8a6267e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.440+0000 7fd1affff640 1 -- 172.21.15.40:0/1645771643 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8a6196000 con 0x55e8a6268440 2024-03-18T22:54:48.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.440+0000 7fd1b4cf0640 1 --2- 172.21.15.40:0/716374151 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1980a28d0 0x7fd1980a4d90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd1a80ccc80 tx=0x7fd1a8005b40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:54:48.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.440+0000 7fd1affff640 1 --2- 172.21.15.40:0/1645771643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6268440 0x55e8a626cf40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0009bf0 tx=0x7fd1a000e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4da75971152603ea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:54:48.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.441+0000 7fd1967fc640 1 -- 172.21.15.40:0/1645771643 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0005020 con 0x55e8a6268440 2024-03-18T22:54:48.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.441+0000 7fd1967fc640 1 -- 172.21.15.40:0/1645771643 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1a00047a0 con 0x55e8a6268440 2024-03-18T22:54:48.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.441+0000 7fd1967fc640 1 -- 172.21.15.40:0/1645771643 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0010690 con 0x55e8a6268440 2024-03-18T22:54:48.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.442+0000 7fd1b941d740 1 -- 172.21.15.40:0/1645771643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6268440 msgr2=0x55e8a626cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:48.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.442+0000 7fd1b941d740 1 --2- 172.21.15.40:0/1645771643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6268440 0x55e8a626cf40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0009bf0 tx=0x7fd1a000e720 comp rx=0 tx=0).stop 2024-03-18T22:54:48.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.442+0000 7fd1b941d740 1 -- 172.21.15.40:0/1645771643 shutdown_connections 2024-03-18T22:54:48.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.442+0000 7fd1b941d740 1 --2- 172.21.15.40:0/1645771643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a626d5b0 0x55e8a626fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.442+0000 7fd1b941d740 1 --2- 172.21.15.40:0/1645771643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6268440 0x55e8a626cf40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.442+0000 7fd1b941d740 1 --2- 172.21.15.40:0/1645771643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6267a50 0x55e8a6267e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.442+0000 7fd1b941d740 1 -- 172.21.15.40:0/1645771643 >> 172.21.15.40:0/1645771643 conn(0x55e8a6262fc0 msgr2=0x55e8a62633c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:54:48.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.442+0000 7fd1b941d740 1 -- 172.21.15.40:0/1645771643 shutdown_connections 2024-03-18T22:54:48.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.442+0000 7fd1b941d740 1 -- 172.21.15.40:0/1645771643 wait complete. 2024-03-18T22:54:48.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.442+0000 7fd1b941d740 1 -- start start 2024-03-18T22:54:48.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.443+0000 7fd1b941d740 1 Processor -- start 2024-03-18T22:54:48.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.443+0000 7fd1b941d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6267a50 0x55e8a6271ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.443+0000 7fd1b941d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a6268440 0x55e8a6272400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.443+0000 7fd1b941d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a626d5b0 0x55e8a62793d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.443+0000 7fd1b941d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e8a62705b0 con 0x55e8a6267a50 2024-03-18T22:54:48.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.443+0000 7fd1b941d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e8a6188160 con 0x55e8a626d5b0 2024-03-18T22:54:48.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.443+0000 7fd1b941d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e8a6270730 con 0x55e8a6268440 2024-03-18T22:54:48.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.443+0000 7fd1b941d740 10 client.0 _ll_register_callbacks cb 0x55e8a61c4a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:54:48.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.443+0000 7fd1af7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6267a50 0x55e8a6271ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.443+0000 7fd1af7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6267a50 0x55e8a6271ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39808/0 (socket says 172.21.15.40:39808) 2024-03-18T22:54:48.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.443+0000 7fd1af7fe640 1 -- 172.21.15.40:0/2174136031 learned_addr learned my addr 172.21.15.40:0/2174136031 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:54:48.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.443+0000 7fd1af7fe640 1 --2- 172.21.15.40:0/2174136031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a626d5b0 0x55e8a62793d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.444+0000 7fd1affff640 1 --2- 172.21.15.40:0/2174136031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a6268440 0x55e8a6272400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.444+0000 7fd1af7fe640 1 -- 172.21.15.40:0/2174136031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a6268440 msgr2=0x55e8a6272400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:54:48.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.444+0000 7fd1af7fe640 1 --2- 172.21.15.40:0/2174136031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a6268440 0x55e8a6272400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.444+0000 7fd1af7fe640 1 -- 172.21.15.40:0/2174136031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a626d5b0 msgr2=0x55e8a62793d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:54:48.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.444+0000 7fd1af7fe640 1 --2- 172.21.15.40:0/2174136031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a626d5b0 0x55e8a62793d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.444+0000 7fd1af7fe640 1 -- 172.21.15.40:0/2174136031 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8a5faa7d0 con 0x55e8a6267a50 2024-03-18T22:54:48.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.444+0000 7fd1af7fe640 1 --2- 172.21.15.40:0/2174136031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6267a50 0x55e8a6271ec0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009230 tx=0x7fd1a402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974d015623f40d3f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:54:48.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.444+0000 7fd18f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:54:48.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.444+0000 7fd18f7fe640 1 -- 172.21.15.40:0/2174136031 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4005520 con 0x55e8a6267a50 2024-03-18T22:54:48.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.444+0000 7fd18f7fe640 1 -- 172.21.15.40:0/2174136031 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd1a40046c0 con 0x55e8a6267a50 2024-03-18T22:54:48.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.444+0000 7fd1b941d740 1 -- 172.21.15.40:0/2174136031 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1a00047a0 con 0x55e8a6267a50 2024-03-18T22:54:48.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.444+0000 7fd18f7fe640 1 -- 172.21.15.40:0/2174136031 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fd1a4055050 con 0x55e8a6267a50 2024-03-18T22:54:48.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.445+0000 7fd1b941d740 10 client.18285 fetch_fsmap learned FSMap version 40 2024-03-18T22:54:48.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.445+0000 7fd1b941d740 1 -- 172.21.15.40:0/2174136031 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e8a5ee03f0 con 0x55e8a6267a50 2024-03-18T22:54:48.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.445+0000 7fd18f7fe640 1 -- 172.21.15.40:0/2174136031 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1a00047a0 con 0x55e8a6267a50 2024-03-18T22:54:48.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.445+0000 7fd18f7fe640 1 -- 172.21.15.40:0/2174136031 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fd1a4040040 con 0x55e8a6267a50 2024-03-18T22:54:48.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.446+0000 7fd1b941d740 10 client.18285 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:54:48.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.446+0000 7fd1b941d740 10 client.18285 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:54:48.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.446+0000 7fd1b941d740 20 client.18285 populate_metadata read hostname 'smithi040' 2024-03-18T22:54:48.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.446+0000 7fd1b941d740 1 --2- 172.21.15.40:0/2174136031 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e8a627dc50 0x55e8a6280110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.446+0000 7fd1b941d740 4 client.18285 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:54:48.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.446+0000 7fd1b941d740 1 -- 172.21.15.40:0/2174136031 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8a61d9a40 con 0x55e8a627dc50 2024-03-18T22:54:48.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.446+0000 7fd1affff640 1 --2- 172.21.15.40:0/2174136031 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e8a627dc50 0x55e8a6280110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.446+0000 7fd18f7fe640 10 client.18285 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:54:48.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.446+0000 7fd1affff640 1 --2- 172.21.15.40:0/2174136031 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e8a627dc50 0x55e8a6280110 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa9c93f670fca4f3 server_cookie=49b2cb264ba47d2d in_seq=0 out_seq=0 2024-03-18T22:54:48.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.447+0000 7fd18f7fe640 1 -- 172.21.15.40:0/2174136031 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e8a61d9a40 con 0x55e8a627dc50 2024-03-18T22:54:48.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.447+0000 7fd18f7fe640 10 client.18285 handle_command_reply: tid=0 2024-03-18T22:54:48.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.447+0000 7fd1b941d740 1 client.18285 shutdown 2024-03-18T22:54:48.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.448+0000 7fd1b941d740 1 -- 172.21.15.40:0/2174136031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6267a50 msgr2=0x55e8a6271ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:48.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.448+0000 7fd1b941d740 1 --2- 172.21.15.40:0/2174136031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6267a50 0x55e8a6271ec0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009230 tx=0x7fd1a402bc30 comp rx=0 tx=0).stop 2024-03-18T22:54:48.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.448+0000 7fd1b941d740 1 -- 172.21.15.40:0/2174136031 shutdown_connections 2024-03-18T22:54:48.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.448+0000 7fd1b941d740 1 --2- 172.21.15.40:0/2174136031 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e8a627dc50 0x55e8a6280110 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.448+0000 7fd1b941d740 1 --2- 172.21.15.40:0/2174136031 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a626d5b0 0x55e8a62793d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.448+0000 7fd1b941d740 1 --2- 172.21.15.40:0/2174136031 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a6268440 0x55e8a6272400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.448+0000 7fd1b941d740 1 --2- 172.21.15.40:0/2174136031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6267a50 0x55e8a6271ec0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.448+0000 7fd1b941d740 1 -- 172.21.15.40:0/2174136031 >> 172.21.15.40:0/2174136031 conn(0x55e8a6262fc0 msgr2=0x55e8a62621e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:54:48.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.448+0000 7fd18f7fe640 0 client.18285 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:54:48.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.448+0000 7fd1b941d740 1 -- 172.21.15.40:0/2174136031 shutdown_connections 2024-03-18T22:54:48.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.448+0000 7fd1b941d740 1 -- 172.21.15.40:0/2174136031 wait complete. 2024-03-18T22:54:48.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.448+0000 7fd1b941d740 20 client.18285 trim_cache size 0 max 16384 2024-03-18T22:54:48.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.478+0000 7fd1b941d740 1 Processor -- start 2024-03-18T22:54:48.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.478+0000 7fd1b941d740 1 -- start start 2024-03-18T22:54:48.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.479+0000 7fd1b941d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a6267a50 0x55e8a6271ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.479+0000 7fd1b941d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6268440 0x55e8a6279360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.479+0000 7fd1b941d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a626d5b0 0x55e8a6272010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.479+0000 7fd1b941d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c00fd50 con 0x55e8a626d5b0 2024-03-18T22:54:48.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.479+0000 7fd1b941d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c011890 con 0x55e8a6268440 2024-03-18T22:54:48.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.479+0000 7fd1b941d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c010420 con 0x55e8a6267a50 2024-03-18T22:54:48.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.479+0000 7fd1af7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a626d5b0 0x55e8a6272010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.479+0000 7fd1af7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a626d5b0 0x55e8a6272010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39814/0 (socket says 172.21.15.40:39814) 2024-03-18T22:54:48.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.479+0000 7fd1af7fe640 1 -- 172.21.15.40:0/1431203214 learned_addr learned my addr 172.21.15.40:0/1431203214 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:54:48.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.479+0000 7fd1affff640 1 --2- 172.21.15.40:0/1431203214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6268440 0x55e8a6279360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.479+0000 7fd1af7fe640 1 --2- 172.21.15.40:0/1431203214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a6267a50 0x55e8a6271ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.480+0000 7fd1affff640 1 -- 172.21.15.40:0/1431203214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a6267a50 msgr2=0x55e8a6271ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:54:48.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.480+0000 7fd1affff640 1 --2- 172.21.15.40:0/1431203214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a6267a50 0x55e8a6271ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.480+0000 7fd1affff640 1 -- 172.21.15.40:0/1431203214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a626d5b0 msgr2=0x55e8a6272010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:54:48.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.480+0000 7fd1affff640 1 --2- 172.21.15.40:0/1431203214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a626d5b0 0x55e8a6272010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.480+0000 7fd1affff640 1 -- 172.21.15.40:0/1431203214 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8a5ea00f0 con 0x55e8a6268440 2024-03-18T22:54:48.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.480+0000 7fd1affff640 1 --2- 172.21.15.40:0/1431203214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6268440 0x55e8a6279360 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55e8a6263f30 tx=0x7fd1a0009770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2391500623f4db84 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:54:48.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.480+0000 7fd195ffb640 1 -- 172.21.15.40:0/1431203214 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0028070 con 0x55e8a6268440 2024-03-18T22:54:48.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.480+0000 7fd195ffb640 1 -- 172.21.15.40:0/1431203214 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1a0029710 con 0x55e8a6268440 2024-03-18T22:54:48.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.480+0000 7fd195ffb640 1 -- 172.21.15.40:0/1431203214 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a001d040 con 0x55e8a6268440 2024-03-18T22:54:48.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.481+0000 7fd1b941d740 1 -- 172.21.15.40:0/1431203214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6268440 msgr2=0x55e8a6279360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:48.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.481+0000 7fd1b941d740 1 --2- 172.21.15.40:0/1431203214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6268440 0x55e8a6279360 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55e8a6263f30 tx=0x7fd1a0009770 comp rx=0 tx=0).stop 2024-03-18T22:54:48.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.481+0000 7fd1b941d740 1 -- 172.21.15.40:0/1431203214 shutdown_connections 2024-03-18T22:54:48.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.481+0000 7fd1b941d740 1 --2- 172.21.15.40:0/1431203214 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a626d5b0 0x55e8a6272010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.481+0000 7fd1b941d740 1 --2- 172.21.15.40:0/1431203214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6268440 0x55e8a6279360 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.481+0000 7fd1b941d740 1 --2- 172.21.15.40:0/1431203214 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a6267a50 0x55e8a6271ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.481+0000 7fd1b941d740 1 -- 172.21.15.40:0/1431203214 >> 172.21.15.40:0/1431203214 conn(0x55e8a6262fc0 msgr2=0x55e8a62633a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:54:48.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.481+0000 7fd1b941d740 1 -- 172.21.15.40:0/1431203214 shutdown_connections 2024-03-18T22:54:48.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.481+0000 7fd1b941d740 1 -- 172.21.15.40:0/1431203214 wait complete. 2024-03-18T22:54:48.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.482+0000 7fd1b941d740 1 -- start start 2024-03-18T22:54:48.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.482+0000 7fd1b941d740 1 Processor -- start 2024-03-18T22:54:48.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.482+0000 7fd1b941d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6267a50 0x55e8a6276450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.482+0000 7fd1b941d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6268440 0x55e8a6276990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.482+0000 7fd1b941d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a626d5b0 0x55e8a62779a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.482+0000 7fd1b941d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c0111c0 con 0x55e8a6268440 2024-03-18T22:54:48.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.482+0000 7fd1b941d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c010af0 con 0x55e8a6267a50 2024-03-18T22:54:48.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.482+0000 7fd1b941d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c00dd30 con 0x55e8a626d5b0 2024-03-18T22:54:48.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.482+0000 7fd1b941d740 10 client.0 _ll_register_callbacks cb 0x7fd1a80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:54:48.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.483+0000 7fd1af7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a626d5b0 0x55e8a62779a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.483+0000 7fd1af7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a626d5b0 0x55e8a62779a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:51982/0 (socket says 172.21.15.40:51982) 2024-03-18T22:54:48.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.483+0000 7fd1af7fe640 1 -- 172.21.15.40:0/1369574928 learned_addr learned my addr 172.21.15.40:0/1369574928 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:54:48.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.483+0000 7fd1affff640 1 --2- 172.21.15.40:0/1369574928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6268440 0x55e8a6276990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.483+0000 7fd1affff640 1 -- 172.21.15.40:0/1369574928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a626d5b0 msgr2=0x55e8a62779a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:54:48.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.483+0000 7fd1affff640 1 --2- 172.21.15.40:0/1369574928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a626d5b0 0x55e8a62779a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.483+0000 7fd1affff640 1 -- 172.21.15.40:0/1369574928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6267a50 msgr2=0x55e8a6276450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:54:48.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.483+0000 7fd1affff640 1 --2- 172.21.15.40:0/1369574928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6267a50 0x55e8a6276450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.483+0000 7fd1affff640 1 -- 172.21.15.40:0/1369574928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8a6277f90 con 0x55e8a6268440 2024-03-18T22:54:48.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.484+0000 7fd1affff640 1 --2- 172.21.15.40:0/1369574928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6268440 0x55e8a6276990 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000e760 tx=0x7fd1a0009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a02c8ce31cd3cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:54:48.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.484+0000 7fd18f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:54:48.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.484+0000 7fd18f7fe640 1 -- 172.21.15.40:0/1369574928 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0028040 con 0x55e8a6268440 2024-03-18T22:54:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.484+0000 7fd18f7fe640 1 -- 172.21.15.40:0/1369574928 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd1a0009ce0 con 0x55e8a6268440 2024-03-18T22:54:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.484+0000 7fd1b941d740 1 -- 172.21.15.40:0/1369574928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1a0029710 con 0x55e8a6268440 2024-03-18T22:54:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.485+0000 7fd18f7fe640 1 -- 172.21.15.40:0/1369574928 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fd1a0005470 con 0x55e8a6268440 2024-03-18T22:54:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.485+0000 7fd1b941d740 10 client.18291 fetch_fsmap learned FSMap version 40 2024-03-18T22:54:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.485+0000 7fd1b941d740 1 -- 172.21.15.40:0/1369574928 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e8a62792a0 con 0x55e8a6268440 2024-03-18T22:54:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.485+0000 7fd18f7fe640 1 -- 172.21.15.40:0/1369574928 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1a0029710 con 0x55e8a6268440 2024-03-18T22:54:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.485+0000 7fd18f7fe640 1 -- 172.21.15.40:0/1369574928 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fd1a0024460 con 0x55e8a6268440 2024-03-18T22:54:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.485+0000 7fd1b941d740 10 client.18291 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:54:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.485+0000 7fd1b941d740 10 client.18291 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:54:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.485+0000 7fd1b941d740 20 client.18291 populate_metadata read hostname 'smithi040' 2024-03-18T22:54:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.485+0000 7fd1b941d740 1 --2- 172.21.15.40:0/1369574928 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e8a62e41e0 0x55e8a62e65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:54:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.485+0000 7fd1b941d740 4 client.18291 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:54:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.485+0000 7fd1b941d740 1 -- 172.21.15.40:0/1369574928 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e8a61d3d90 con 0x55e8a62e41e0 2024-03-18T22:54:48.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.486+0000 7fd1af7fe640 1 --2- 172.21.15.40:0/1369574928 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e8a62e41e0 0x55e8a62e65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:54:48.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.486+0000 7fd1af7fe640 1 --2- 172.21.15.40:0/1369574928 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e8a62e41e0 0x55e8a62e65d0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7a0da2fc4291ff5 server_cookie=a54ee69728d333f in_seq=0 out_seq=0 2024-03-18T22:54:48.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.486+0000 7fd18f7fe640 10 client.18291 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:54:48.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.487+0000 7fd18f7fe640 1 -- 172.21.15.40:0/1369574928 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e8a61d3d90 con 0x55e8a62e41e0 2024-03-18T22:54:48.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.487+0000 7fd18f7fe640 10 client.18291 handle_command_reply: tid=0 2024-03-18T22:54:48.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.487+0000 7fd1b941d740 1 client.18291 shutdown 2024-03-18T22:54:48.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.487+0000 7fd1b941d740 1 -- 172.21.15.40:0/1369574928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6268440 msgr2=0x55e8a6276990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:48.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.487+0000 7fd1b941d740 1 --2- 172.21.15.40:0/1369574928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6268440 0x55e8a6276990 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000e760 tx=0x7fd1a0009630 comp rx=0 tx=0).stop 2024-03-18T22:54:48.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.487+0000 7fd1b941d740 1 -- 172.21.15.40:0/1369574928 shutdown_connections 2024-03-18T22:54:48.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.487+0000 7fd1b941d740 1 --2- 172.21.15.40:0/1369574928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e8a626d5b0 0x55e8a62779a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.487+0000 7fd1b941d740 1 --2- 172.21.15.40:0/1369574928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e8a6268440 0x55e8a6276990 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.487+0000 7fd1b941d740 1 --2- 172.21.15.40:0/1369574928 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e8a62e41e0 0x55e8a62e65d0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.487+0000 7fd1b941d740 1 --2- 172.21.15.40:0/1369574928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e8a6267a50 0x55e8a6276450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.487+0000 7fd1b941d740 1 -- 172.21.15.40:0/1369574928 >> 172.21.15.40:0/1369574928 conn(0x55e8a6262fc0 msgr2=0x55e8a6271ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:54:48.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.487+0000 7fd18f7fe640 0 client.18291 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:54:48.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.487+0000 7fd1b941d740 1 -- 172.21.15.40:0/1369574928 shutdown_connections 2024-03-18T22:54:48.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.487+0000 7fd1b941d740 1 -- 172.21.15.40:0/1369574928 wait complete. 2024-03-18T22:54:48.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.487+0000 7fd1b941d740 20 client.18291 trim_cache size 0 max 16384 2024-03-18T22:54:48.491 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:54:48.491 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:54:48.491 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:54:48.491 INFO:teuthology.orchestra.run.smithi040.stdout: "6a7b3721-c0a7-4062-9ca5-671585e8609b": { 2024-03-18T22:54:48.491 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:54:48.491 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "6a7b3721-c0a7-4062-9ca5-671585e8609b", 2024-03-18T22:54:48.492 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:54:48.492 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:54:48.492 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:54:48.492 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:54:48.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.488+0000 7fd1b6753640 1 -- 172.21.15.40:0/716374151 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1980a28d0 msgr2=0x7fd1980a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:48.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.488+0000 7fd1b6753640 1 --2- 172.21.15.40:0/716374151 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1980a28d0 0x7fd1980a4d90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd1a80ccc80 tx=0x7fd1a8005b40 comp rx=0 tx=0).stop 2024-03-18T22:54:48.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.488+0000 7fd1b6753640 1 -- 172.21.15.40:0/716374151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1b0103400 msgr2=0x7fd1b01a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:54:48.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.488+0000 7fd1b6753640 1 --2- 172.21.15.40:0/716374151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1b0103400 0x7fd1b01a5670 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800eb10 tx=0x7fd1a800c2d0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.488+0000 7fd1b6753640 1 -- 172.21.15.40:0/716374151 shutdown_connections 2024-03-18T22:54:48.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.488+0000 7fd1b6753640 1 --2- 172.21.15.40:0/716374151 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd1980a28d0 0x7fd1980a4d90 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.488+0000 7fd1b6753640 1 --2- 172.21.15.40:0/716374151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd1b0103400 0x7fd1b01a5670 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.488+0000 7fd1b6753640 1 --2- 172.21.15.40:0/716374151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1b0102a40 0x7fd1b019e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.488+0000 7fd1b6753640 1 --2- 172.21.15.40:0/716374151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b0101850 0x7fd1b019e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:54:48.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.489+0000 7fd1b6753640 1 -- 172.21.15.40:0/716374151 >> 172.21.15.40:0/716374151 conn(0x7fd1b00fcdc0 msgr2=0x7fd1b0103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:54:48.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.489+0000 7fd1b6753640 1 -- 172.21.15.40:0/716374151 shutdown_connections 2024-03-18T22:54:48.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:54:48.489+0000 7fd1b6753640 1 -- 172.21.15.40:0/716374151 wait complete. 2024-03-18T22:54:48.507 INFO:tasks.cephfs.filesystem:scrub status for tag:6a7b3721-c0a7-4062-9ca5-671585e8609b - {'path': '/', 'tag': '6a7b3721-c0a7-4062-9ca5-671585e8609b', 'options': 'recursive,force'} 2024-03-18T22:54:49.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:48 smithi103 ceph-mon[66377]: pgmap v715: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 2.90k op/s 2024-03-18T22:54:49.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:48 smithi040 ceph-mon[62643]: pgmap v715: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 2.90k op/s 2024-03-18T22:54:49.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:48 smithi113 ceph-mon[66775]: pgmap v715: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 2.90k op/s 2024-03-18T22:54:51.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:50 smithi103 ceph-mon[66377]: pgmap v716: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-03-18T22:54:51.145 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:50 smithi040 ceph-mon[62643]: pgmap v716: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-03-18T22:54:51.330 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:50 smithi113 ceph-mon[66775]: pgmap v716: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-03-18T22:54:53.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:52 smithi103 ceph-mon[66377]: pgmap v717: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 2.56k op/s 2024-03-18T22:54:53.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:52 smithi040 ceph-mon[62643]: pgmap v717: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 2.56k op/s 2024-03-18T22:54:53.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:52 smithi113 ceph-mon[66775]: pgmap v717: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 2.56k op/s 2024-03-18T22:54:54.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:53 smithi113 ceph-mon[66775]: pgmap v718: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 3.15k op/s 2024-03-18T22:54:54.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:53 smithi103 ceph-mon[66377]: pgmap v718: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 3.15k op/s 2024-03-18T22:54:54.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:53 smithi040 ceph-mon[62643]: pgmap v718: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 3.15k op/s 2024-03-18T22:54:56.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:56 smithi040 ceph-mon[62643]: pgmap v719: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-03-18T22:54:56.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:56 smithi113 ceph-mon[66775]: pgmap v719: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-03-18T22:54:56.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:56 smithi103 ceph-mon[66377]: pgmap v719: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-03-18T22:54:58.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:54:58 smithi040 ceph-mon[62643]: pgmap v720: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 2.94k op/s 2024-03-18T22:54:58.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:54:58 smithi113 ceph-mon[66775]: pgmap v720: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 2.94k op/s 2024-03-18T22:54:58.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:54:58 smithi103 ceph-mon[66377]: pgmap v720: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 2.94k op/s 2024-03-18T22:55:00.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:00 smithi040 ceph-mon[62643]: pgmap v721: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 3.00k op/s 2024-03-18T22:55:00.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:00 smithi113 ceph-mon[66775]: pgmap v721: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 3.00k op/s 2024-03-18T22:55:00.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:00 smithi103 ceph-mon[66377]: pgmap v721: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 3.00k op/s 2024-03-18T22:55:02.409 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:02 smithi103 ceph-mon[66377]: pgmap v722: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 2.63k op/s 2024-03-18T22:55:02.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:02 smithi113 ceph-mon[66775]: pgmap v722: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 2.63k op/s 2024-03-18T22:55:02.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:02 smithi040 ceph-mon[62643]: pgmap v722: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 2.63k op/s 2024-03-18T22:55:04.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:04 smithi113 ceph-mon[66775]: pgmap v723: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 3.10k op/s 2024-03-18T22:55:04.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:04 smithi103 ceph-mon[66377]: pgmap v723: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 3.10k op/s 2024-03-18T22:55:04.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:04 smithi040 ceph-mon[62643]: pgmap v723: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 3.10k op/s 2024-03-18T22:55:06.445 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:06 smithi040 ceph-mon[62643]: pgmap v724: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-03-18T22:55:06.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:06 smithi113 ceph-mon[66775]: pgmap v724: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-03-18T22:55:06.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:06 smithi103 ceph-mon[66377]: pgmap v724: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-03-18T22:55:08.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:08 smithi113 ceph-mon[66775]: pgmap v725: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 10 MiB/s wr, 2.65k op/s 2024-03-18T22:55:08.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:08 smithi103 ceph-mon[66377]: pgmap v725: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 10 MiB/s wr, 2.65k op/s 2024-03-18T22:55:08.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:08 smithi040 ceph-mon[62643]: pgmap v725: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 10 MiB/s wr, 2.65k op/s 2024-03-18T22:55:09.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:09 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:09.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:09 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:09.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:09 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:10.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:10 smithi113 ceph-mon[66775]: pgmap v726: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 11 MiB/s wr, 2.81k op/s 2024-03-18T22:55:10.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:10 smithi103 ceph-mon[66377]: pgmap v726: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 11 MiB/s wr, 2.81k op/s 2024-03-18T22:55:10.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:10 smithi040 ceph-mon[62643]: pgmap v726: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 11 MiB/s wr, 2.81k op/s 2024-03-18T22:55:12.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:12 smithi113 ceph-mon[66775]: pgmap v727: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.6 MiB/s wr, 2.45k op/s 2024-03-18T22:55:12.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:12 smithi103 ceph-mon[66377]: pgmap v727: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.6 MiB/s wr, 2.45k op/s 2024-03-18T22:55:12.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:12 smithi040 ceph-mon[62643]: pgmap v727: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.6 MiB/s wr, 2.45k op/s 2024-03-18T22:55:13.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:13 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:55:13.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:13 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:55:13.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:13 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:55:14.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:14 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:55:14.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:14 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:55:14.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:14 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:55:14.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:14 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:55:14.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:14 smithi113 ceph-mon[66775]: pgmap v728: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 11 MiB/s wr, 2.87k op/s 2024-03-18T22:55:14.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:14 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:55:14.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:14 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:55:14.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:14 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:55:14.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:14 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:55:14.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:14 smithi103 ceph-mon[66377]: pgmap v728: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 11 MiB/s wr, 2.87k op/s 2024-03-18T22:55:14.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:14 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:55:14.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:14 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:55:14.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:14 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:55:14.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:14 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:55:14.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:14 smithi040 ceph-mon[62643]: pgmap v728: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 11 MiB/s wr, 2.87k op/s 2024-03-18T22:55:16.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:16 smithi113 ceph-mon[66775]: pgmap v729: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.8 MiB/s wr, 2.50k op/s 2024-03-18T22:55:16.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:16 smithi103 ceph-mon[66377]: pgmap v729: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.8 MiB/s wr, 2.50k op/s 2024-03-18T22:55:16.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:16 smithi040 ceph-mon[62643]: pgmap v729: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.8 MiB/s wr, 2.50k op/s 2024-03-18T22:55:18.508 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:55:18.509 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:55:18.551 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:18 smithi040 ceph-mon[62643]: pgmap v730: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 9.2 MiB/s wr, 2.34k op/s 2024-03-18T22:55:18.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:18 smithi113 ceph-mon[66775]: pgmap v730: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 9.2 MiB/s wr, 2.34k op/s 2024-03-18T22:55:18.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:18 smithi103 ceph-mon[66377]: pgmap v730: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 9.2 MiB/s wr, 2.34k op/s 2024-03-18T22:55:18.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.666+0000 7fb0b4e85640 1 -- 172.21.15.40:0/2649609395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0b0100730 msgr2=0x7fb0b0100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:18.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.666+0000 7fb0b4e85640 1 --2- 172.21.15.40:0/2649609395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0b0100730 0x7fb0b0100b50 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009ca0 tx=0x7fb0a802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.666+0000 7fb0b4e85640 1 -- 172.21.15.40:0/2649609395 shutdown_connections 2024-03-18T22:55:18.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.666+0000 7fb0b4e85640 1 --2- 172.21.15.40:0/2649609395 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0b01022e0 0x7fb0b0108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.666+0000 7fb0b4e85640 1 --2- 172.21.15.40:0/2649609395 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0b0101920 0x7fb0b0101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.666+0000 7fb0b4e85640 1 --2- 172.21.15.40:0/2649609395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0b0100730 0x7fb0b0100b50 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.666+0000 7fb0b4e85640 1 -- 172.21.15.40:0/2649609395 >> 172.21.15.40:0/2649609395 conn(0x7fb0b00fbcc0 msgr2=0x7fb0b00fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:18.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.666+0000 7fb0b4e85640 1 -- 172.21.15.40:0/2649609395 shutdown_connections 2024-03-18T22:55:18.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.666+0000 7fb0b4e85640 1 -- 172.21.15.40:0/2649609395 wait complete. 2024-03-18T22:55:18.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.667+0000 7fb0b4e85640 1 Processor -- start 2024-03-18T22:55:18.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.667+0000 7fb0b4e85640 1 -- start start 2024-03-18T22:55:18.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.667+0000 7fb0b4e85640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0b0100730 0x7fb0b006a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.667+0000 7fb0b4e85640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0b0101920 0x7fb0b019ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.667+0000 7fb0b4e85640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0b01022e0 0x7fb0b01a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.667+0000 7fb0b4e85640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b010c230 con 0x7fb0b0100730 2024-03-18T22:55:18.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.667+0000 7fb0b4e85640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b010c0b0 con 0x7fb0b01022e0 2024-03-18T22:55:18.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.667+0000 7fb0b4e85640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b010c3b0 con 0x7fb0b0101920 2024-03-18T22:55:18.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.667+0000 7fb0ae59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0b0100730 0x7fb0b006a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.667+0000 7fb0ae59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0b0100730 0x7fb0b006a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35172/0 (socket says 172.21.15.40:35172) 2024-03-18T22:55:18.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.667+0000 7fb0ae59c640 1 -- 172.21.15.40:0/2814760097 learned_addr learned my addr 172.21.15.40:0/2814760097 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:18.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.668+0000 7fb0add9b640 1 --2- 172.21.15.40:0/2814760097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0b0101920 0x7fb0b019ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.668+0000 7fb0aed9d640 1 --2- 172.21.15.40:0/2814760097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0b01022e0 0x7fb0b01a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.668+0000 7fb0ae59c640 1 -- 172.21.15.40:0/2814760097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0b0101920 msgr2=0x7fb0b019ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:18.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.668+0000 7fb0ae59c640 1 --2- 172.21.15.40:0/2814760097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0b0101920 0x7fb0b019ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.668+0000 7fb0ae59c640 1 -- 172.21.15.40:0/2814760097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0b01022e0 msgr2=0x7fb0b01a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:18.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.668+0000 7fb0ae59c640 1 --2- 172.21.15.40:0/2814760097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0b01022e0 0x7fb0b01a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.668+0000 7fb0ae59c640 1 -- 172.21.15.40:0/2814760097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b01a1970 con 0x7fb0b0100730 2024-03-18T22:55:18.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.668+0000 7fb0ae59c640 1 --2- 172.21.15.40:0/2814760097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0b0100730 0x7fb0b006a960 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb0a802c700 tx=0x7fb0a80328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:18.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.668+0000 7fb09f7fe640 1 -- 172.21.15.40:0/2814760097 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8041020 con 0x7fb0b0100730 2024-03-18T22:55:18.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.668+0000 7fb09f7fe640 1 -- 172.21.15.40:0/2814760097 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0a8003e20 con 0x7fb0b0100730 2024-03-18T22:55:18.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.668+0000 7fb0b4e85640 1 -- 172.21.15.40:0/2814760097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b01a1ba0 con 0x7fb0b0100730 2024-03-18T22:55:18.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.668+0000 7fb09f7fe640 1 -- 172.21.15.40:0/2814760097 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8039040 con 0x7fb0b0100730 2024-03-18T22:55:18.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.668+0000 7fb0b4e85640 1 -- 172.21.15.40:0/2814760097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b01a2090 con 0x7fb0b0100730 2024-03-18T22:55:18.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.670+0000 7fb09f7fe640 1 -- 172.21.15.40:0/2814760097 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb0a8044020 con 0x7fb0b0100730 2024-03-18T22:55:18.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.671+0000 7fb0b7a75740 1 Processor -- start 2024-03-18T22:55:18.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.671+0000 7fb0b7a75740 1 -- start start 2024-03-18T22:55:18.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.671+0000 7fb0b7a75740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67308a50 0x562a67308e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.671+0000 7fb0b7a75740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67309440 0x562a6730df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.671+0000 7fb09f7fe640 1 --2- 172.21.15.40:0/2814760097 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0980a2830 0x7fb0980a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.671+0000 7fb0b7a75740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6730e5b0 0x562a67310a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.671+0000 7fb0b7a75740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562a6726d500 con 0x562a67308a50 2024-03-18T22:55:18.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.671+0000 7fb0b7a75740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562a67215be0 con 0x562a6730e5b0 2024-03-18T22:55:18.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.671+0000 7fb0b7a75740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x562a672695a0 con 0x562a67309440 2024-03-18T22:55:18.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb0aed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67308a50 0x562a67308e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb0ae59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67309440 0x562a6730df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb0aed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67308a50 0x562a67308e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35184/0 (socket says 172.21.15.40:35184) 2024-03-18T22:55:18.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb0aed9d640 1 -- 172.21.15.40:0/2850008235 learned_addr learned my addr 172.21.15.40:0/2850008235 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:18.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb0add9b640 1 --2- 172.21.15.40:0/2814760097 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0980a2830 0x7fb0980a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb09f7fe640 1 -- 172.21.15.40:0/2814760097 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb0a80f1110 con 0x7fb0b0100730 2024-03-18T22:55:18.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb0aed9d640 1 --2- 172.21.15.40:0/2850008235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6730e5b0 0x562a67310a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb0ae59c640 1 -- 172.21.15.40:0/2850008235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6730e5b0 msgr2=0x562a67310a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:18.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb0ae59c640 1 --2- 172.21.15.40:0/2850008235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6730e5b0 0x562a67310a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb0ae59c640 1 -- 172.21.15.40:0/2850008235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67308a50 msgr2=0x562a67308e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:18.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb0ae59c640 1 --2- 172.21.15.40:0/2850008235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67308a50 0x562a67308e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb0ae59c640 1 -- 172.21.15.40:0/2850008235 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a67237000 con 0x562a67309440 2024-03-18T22:55:18.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb0add9b640 1 --2- 172.21.15.40:0/2814760097 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0980a2830 0x7fb0980a4cf0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fb0b006a6d0 tx=0x7fb0a400b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:18.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb0ae59c640 1 --2- 172.21.15.40:0/2850008235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67309440 0x562a6730df40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80bd360 tx=0x7fb0a803af40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ba568180216b9b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:18.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb09cff9640 1 -- 172.21.15.40:0/2850008235 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80bd430 con 0x562a67309440 2024-03-18T22:55:18.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb09cff9640 1 -- 172.21.15.40:0/2850008235 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0a8004040 con 0x562a67309440 2024-03-18T22:55:18.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.672+0000 7fb09cff9640 1 -- 172.21.15.40:0/2850008235 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80d4d90 con 0x562a67309440 2024-03-18T22:55:18.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.673+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2850008235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67309440 msgr2=0x562a6730df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:18.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.673+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/2850008235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67309440 0x562a6730df40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80bd360 tx=0x7fb0a803af40 comp rx=0 tx=0).stop 2024-03-18T22:55:18.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.673+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2850008235 shutdown_connections 2024-03-18T22:55:18.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.673+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/2850008235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6730e5b0 0x562a67310a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.673+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/2850008235 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67309440 0x562a6730df40 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.673+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/2850008235 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67308a50 0x562a67308e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.673+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2850008235 >> 172.21.15.40:0/2850008235 conn(0x562a67303fc0 msgr2=0x562a673043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:18.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.673+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2850008235 shutdown_connections 2024-03-18T22:55:18.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.673+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2850008235 wait complete. 2024-03-18T22:55:18.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0b7a75740 1 -- start start 2024-03-18T22:55:18.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0b7a75740 1 Processor -- start 2024-03-18T22:55:18.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0b7a75740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67308a50 0x562a67312bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0b7a75740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67309440 0x562a67313130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0b7a75740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6730e5b0 0x562a6731a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0b7a75740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x562a67311320 con 0x562a67309440 2024-03-18T22:55:18.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0b7a75740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562a67229160 con 0x562a6730e5b0 2024-03-18T22:55:18.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0b7a75740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x562a673114a0 con 0x562a67308a50 2024-03-18T22:55:18.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0b7a75740 10 client.0 _ll_register_callbacks cb 0x562a67265a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:18.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0add9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67309440 0x562a67313130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0ae59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67308a50 0x562a67312bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0add9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67309440 0x562a67313130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35188/0 (socket says 172.21.15.40:35188) 2024-03-18T22:55:18.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0add9b640 1 -- 172.21.15.40:0/4233493024 learned_addr learned my addr 172.21.15.40:0/4233493024 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:18.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0aed9d640 1 --2- 172.21.15.40:0/4233493024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6730e5b0 0x562a6731a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0ae59c640 1 -- 172.21.15.40:0/4233493024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6730e5b0 msgr2=0x562a6731a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:18.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0ae59c640 1 --2- 172.21.15.40:0/4233493024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6730e5b0 0x562a6731a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0ae59c640 1 -- 172.21.15.40:0/4233493024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67309440 msgr2=0x562a67313130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:18.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0ae59c640 1 --2- 172.21.15.40:0/4233493024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67309440 0x562a67313130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0ae59c640 1 -- 172.21.15.40:0/4233493024 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a6704b7d0 con 0x562a67308a50 2024-03-18T22:55:18.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.674+0000 7fb0add9b640 1 --2- 172.21.15.40:0/4233493024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67309440 0x562a67313130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:55:18.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb0ae59c640 1 --2- 172.21.15.40:0/4233493024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67308a50 0x562a67312bf0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x562a67311230 tx=0x7fb0a80d5040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c51969ac7a6f2c07 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:18.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb095ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:55:18.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb095ffb640 1 -- 172.21.15.40:0/4233493024 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80bcfc0 con 0x562a67308a50 2024-03-18T22:55:18.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb095ffb640 1 -- 172.21.15.40:0/4233493024 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb0a80ee840 con 0x562a67308a50 2024-03-18T22:55:18.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb0b7a75740 1 -- 172.21.15.40:0/4233493024 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562a6727aa40 con 0x562a67308a50 2024-03-18T22:55:18.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb095ffb640 1 -- 172.21.15.40:0/4233493024 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fb0a80bd1a0 con 0x562a67308a50 2024-03-18T22:55:18.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb0b7a75740 10 client.28120 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:18.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb0b7a75740 1 -- 172.21.15.40:0/4233493024 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x562a66f813f0 con 0x562a67308a50 2024-03-18T22:55:18.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb095ffb640 1 -- 172.21.15.40:0/4233493024 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x562a6727aa40 con 0x562a67308a50 2024-03-18T22:55:18.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb095ffb640 1 -- 172.21.15.40:0/4233493024 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fb0a803c030 con 0x562a67308a50 2024-03-18T22:55:18.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb0b7a75740 10 client.28120 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:18.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb0b7a75740 10 client.28120 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:55:18.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb0b7a75740 20 client.28120 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:18.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/4233493024 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a6731e7c0 0x562a67320c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb0b7a75740 4 client.28120 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:55:18.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.675+0000 7fb0b7a75740 1 -- 172.21.15.40:0/4233493024 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a67274d90 con 0x562a6731e7c0 2024-03-18T22:55:18.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.676+0000 7fb0add9b640 1 --2- 172.21.15.40:0/4233493024 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a6731e7c0 0x562a67320c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.676+0000 7fb0add9b640 1 --2- 172.21.15.40:0/4233493024 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a6731e7c0 0x562a67320c60 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b48acb993668000 server_cookie=87c1d6056e8f04e3 in_seq=0 out_seq=0 2024-03-18T22:55:18.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.676+0000 7fb095ffb640 10 client.28120 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:18.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.677+0000 7fb095ffb640 1 -- 172.21.15.40:0/4233493024 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562a67274d90 con 0x562a6731e7c0 2024-03-18T22:55:18.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.677+0000 7fb095ffb640 10 client.28120 handle_command_reply: tid=0 2024-03-18T22:55:18.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.677+0000 7fb0b7a75740 1 client.28120 shutdown 2024-03-18T22:55:18.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.677+0000 7fb0b7a75740 1 -- 172.21.15.40:0/4233493024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67308a50 msgr2=0x562a67312bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:18.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.677+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/4233493024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67308a50 0x562a67312bf0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x562a67311230 tx=0x7fb0a80d5040 comp rx=0 tx=0).stop 2024-03-18T22:55:18.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.677+0000 7fb0b7a75740 1 -- 172.21.15.40:0/4233493024 shutdown_connections 2024-03-18T22:55:18.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.677+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/4233493024 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a6731e7c0 0x562a67320c60 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.677+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/4233493024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6730e5b0 0x562a6731a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.677+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/4233493024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67309440 0x562a67313130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.677+0000 7fb095ffb640 0 client.28120 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:18.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.677+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/4233493024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67308a50 0x562a67312bf0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.677+0000 7fb0b7a75740 1 -- 172.21.15.40:0/4233493024 >> 172.21.15.40:0/4233493024 conn(0x562a67303fc0 msgr2=0x562a673031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:18.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.677+0000 7fb0b7a75740 1 -- 172.21.15.40:0/4233493024 shutdown_connections 2024-03-18T22:55:18.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.677+0000 7fb0b7a75740 1 -- 172.21.15.40:0/4233493024 wait complete. 2024-03-18T22:55:18.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.677+0000 7fb0b7a75740 20 client.28120 trim_cache size 0 max 16384 2024-03-18T22:55:18.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0b7a75740 1 Processor -- start 2024-03-18T22:55:18.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0b7a75740 1 -- start start 2024-03-18T22:55:18.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0b7a75740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67308a50 0x562a67313090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0b7a75740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a67309440 0x562a673043a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0b7a75740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a6730e5b0 0x562a67302950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0b7a75740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b01ac790 con 0x562a67308a50 2024-03-18T22:55:18.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0b7a75740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b01ae2d0 con 0x562a67309440 2024-03-18T22:55:18.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0b7a75740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b01ace60 con 0x562a6730e5b0 2024-03-18T22:55:18.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0add9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a67309440 0x562a673043a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0ae59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67308a50 0x562a67313090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0aed9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a6730e5b0 0x562a67302950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0aed9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a6730e5b0 0x562a67302950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:37654/0 (socket says 172.21.15.40:37654) 2024-03-18T22:55:18.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0ae59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67308a50 0x562a67313090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35202/0 (socket says 172.21.15.40:35202) 2024-03-18T22:55:18.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0add9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a67309440 0x562a673043a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46836/0 (socket says 172.21.15.40:46836) 2024-03-18T22:55:18.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0aed9d640 1 -- 172.21.15.40:0/2883386940 learned_addr learned my addr 172.21.15.40:0/2883386940 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:18.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0aed9d640 1 -- 172.21.15.40:0/2883386940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a67309440 msgr2=0x562a673043a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:18.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0aed9d640 1 --2- 172.21.15.40:0/2883386940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a67309440 0x562a673043a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0aed9d640 1 -- 172.21.15.40:0/2883386940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67308a50 msgr2=0x562a67313090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:18.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0aed9d640 1 --2- 172.21.15.40:0/2883386940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67308a50 0x562a67313090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0aed9d640 1 -- 172.21.15.40:0/2883386940 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a66f410f0 con 0x562a6730e5b0 2024-03-18T22:55:18.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.695+0000 7fb0aed9d640 1 --2- 172.21.15.40:0/2883386940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a6730e5b0 0x562a67302950 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000bb10 tx=0x7fb0a0005cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3df85003fae2748 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:18.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.696+0000 7fb097fff640 1 -- 172.21.15.40:0/2883386940 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0003ab0 con 0x562a6730e5b0 2024-03-18T22:55:18.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.696+0000 7fb097fff640 1 -- 172.21.15.40:0/2883386940 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0a00054d0 con 0x562a6730e5b0 2024-03-18T22:55:18.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.696+0000 7fb097fff640 1 -- 172.21.15.40:0/2883386940 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0004ad0 con 0x562a6730e5b0 2024-03-18T22:55:18.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.696+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2883386940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a6730e5b0 msgr2=0x562a67302950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:18.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.696+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/2883386940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a6730e5b0 0x562a67302950 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000bb10 tx=0x7fb0a0005cc0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.696+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2883386940 shutdown_connections 2024-03-18T22:55:18.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.696+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/2883386940 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a6730e5b0 0x562a67302950 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.696+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/2883386940 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a67309440 0x562a673043a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.696+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/2883386940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a67308a50 0x562a67313090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.696+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2883386940 >> 172.21.15.40:0/2883386940 conn(0x562a67303fc0 msgr2=0x562a672ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:18.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.696+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2883386940 shutdown_connections 2024-03-18T22:55:18.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.696+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2883386940 wait complete. 2024-03-18T22:55:18.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0b7a75740 1 -- start start 2024-03-18T22:55:18.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0b7a75740 1 Processor -- start 2024-03-18T22:55:18.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0b7a75740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a67308a50 0x562a67317440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0b7a75740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67309440 0x562a67317980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0b7a75740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6730e5b0 0x562a67318990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0b7a75740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b01adc00 con 0x562a6730e5b0 2024-03-18T22:55:18.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0b7a75740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b01ad530 con 0x562a67308a50 2024-03-18T22:55:18.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0b7a75740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b01aa770 con 0x562a67309440 2024-03-18T22:55:18.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0b7a75740 10 client.0 _ll_register_callbacks cb 0x7fb0a0004410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:18.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0aed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6730e5b0 0x562a67318990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0aed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6730e5b0 0x562a67318990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35208/0 (socket says 172.21.15.40:35208) 2024-03-18T22:55:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0aed9d640 1 -- 172.21.15.40:0/2648226231 learned_addr learned my addr 172.21.15.40:0/2648226231 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0add9b640 1 --2- 172.21.15.40:0/2648226231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67309440 0x562a67317980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0ae59c640 1 --2- 172.21.15.40:0/2648226231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a67308a50 0x562a67317440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0aed9d640 1 -- 172.21.15.40:0/2648226231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67309440 msgr2=0x562a67317980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0aed9d640 1 --2- 172.21.15.40:0/2648226231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67309440 0x562a67317980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0aed9d640 1 -- 172.21.15.40:0/2648226231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a67308a50 msgr2=0x562a67317440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0aed9d640 1 --2- 172.21.15.40:0/2648226231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a67308a50 0x562a67317440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0aed9d640 1 -- 172.21.15.40:0/2648226231 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a67318f80 con 0x562a6730e5b0 2024-03-18T22:55:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.697+0000 7fb0add9b640 1 --2- 172.21.15.40:0/2648226231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67309440 0x562a67317980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:55:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb0aed9d640 1 --2- 172.21.15.40:0/2648226231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6730e5b0 0x562a67318990 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000b7b0 tx=0x7fb0a0005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fdefec7b2d81b49 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb095ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:55:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb095ffb640 1 -- 172.21.15.40:0/2648226231 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a00046c0 con 0x562a6730e5b0 2024-03-18T22:55:18.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2648226231 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0a00054d0 con 0x562a6730e5b0 2024-03-18T22:55:18.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb095ffb640 1 -- 172.21.15.40:0/2648226231 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb0a0003cc0 con 0x562a6730e5b0 2024-03-18T22:55:18.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb095ffb640 1 -- 172.21.15.40:0/2648226231 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fb0a0004a50 con 0x562a6730e5b0 2024-03-18T22:55:18.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb0b7a75740 10 client.18309 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:18.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2648226231 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x562a67384b60 con 0x562a6730e5b0 2024-03-18T22:55:18.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb095ffb640 1 -- 172.21.15.40:0/2648226231 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0a00054d0 con 0x562a6730e5b0 2024-03-18T22:55:18.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb095ffb640 1 -- 172.21.15.40:0/2648226231 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fb0a0027a20 con 0x562a6730e5b0 2024-03-18T22:55:18.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb0b7a75740 10 client.18309 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:18.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb0b7a75740 10 client.18309 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:55:18.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb0b7a75740 20 client.18309 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:18.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/2648226231 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a67385480 0x562a673878a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb0b7a75740 4 client.18309 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:55:18.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2648226231 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562a671f2540 con 0x562a67385480 2024-03-18T22:55:18.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.698+0000 7fb0ae59c640 1 --2- 172.21.15.40:0/2648226231 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a67385480 0x562a673878a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb095ffb640 10 client.18309 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:18.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb0ae59c640 1 --2- 172.21.15.40:0/2648226231 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a67385480 0x562a673878a0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa6e49f653be57c5 server_cookie=161fa89a986e2481 in_seq=0 out_seq=0 2024-03-18T22:55:18.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb095ffb640 1 -- 172.21.15.40:0/2648226231 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562a671f2540 con 0x562a67385480 2024-03-18T22:55:18.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb095ffb640 10 client.18309 handle_command_reply: tid=0 2024-03-18T22:55:18.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb0b7a75740 1 client.18309 shutdown 2024-03-18T22:55:18.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2648226231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6730e5b0 msgr2=0x562a67318990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:18.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/2648226231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6730e5b0 0x562a67318990 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000b7b0 tx=0x7fb0a0005230 comp rx=0 tx=0).stop 2024-03-18T22:55:18.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2648226231 shutdown_connections 2024-03-18T22:55:18.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/2648226231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562a6730e5b0 0x562a67318990 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/2648226231 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x562a67309440 0x562a67317980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/2648226231 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x562a67385480 0x562a673878a0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb0b7a75740 1 --2- 172.21.15.40:0/2648226231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a67308a50 0x562a67317440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb095ffb640 0 client.18309 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:18.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2648226231 >> 172.21.15.40:0/2648226231 conn(0x562a67303fc0 msgr2=0x562a673043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:18.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2648226231 shutdown_connections 2024-03-18T22:55:18.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb0b7a75740 1 -- 172.21.15.40:0/2648226231 wait complete. 2024-03-18T22:55:18.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.699+0000 7fb0b7a75740 20 client.18309 trim_cache size 0 max 16384 2024-03-18T22:55:18.708 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:55:18.708 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:55:18.708 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:55:18.708 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:55:18.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.700+0000 7fb0b4e85640 1 -- 172.21.15.40:0/2814760097 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0980a2830 msgr2=0x7fb0980a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:18.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.700+0000 7fb0b4e85640 1 --2- 172.21.15.40:0/2814760097 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0980a2830 0x7fb0980a4cf0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fb0b006a6d0 tx=0x7fb0a400b410 comp rx=0 tx=0).stop 2024-03-18T22:55:18.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.700+0000 7fb0b4e85640 1 -- 172.21.15.40:0/2814760097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0b0100730 msgr2=0x7fb0b006a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:18.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.700+0000 7fb0b4e85640 1 --2- 172.21.15.40:0/2814760097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0b0100730 0x7fb0b006a960 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb0a802c700 tx=0x7fb0a80328a0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.700+0000 7fb0b4e85640 1 -- 172.21.15.40:0/2814760097 shutdown_connections 2024-03-18T22:55:18.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.700+0000 7fb0b4e85640 1 --2- 172.21.15.40:0/2814760097 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb0980a2830 0x7fb0980a4cf0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.700+0000 7fb0b4e85640 1 --2- 172.21.15.40:0/2814760097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0b01022e0 0x7fb0b01a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.700+0000 7fb0b4e85640 1 --2- 172.21.15.40:0/2814760097 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb0b0101920 0x7fb0b019ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.700+0000 7fb0b4e85640 1 --2- 172.21.15.40:0/2814760097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0b0100730 0x7fb0b006a960 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.700+0000 7fb0b4e85640 1 -- 172.21.15.40:0/2814760097 >> 172.21.15.40:0/2814760097 conn(0x7fb0b00fbcc0 msgr2=0x7fb0b0104de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:18.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.700+0000 7fb0b4e85640 1 -- 172.21.15.40:0/2814760097 shutdown_connections 2024-03-18T22:55:18.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.700+0000 7fb0b4e85640 1 -- 172.21.15.40:0/2814760097 wait complete. 2024-03-18T22:55:18.712 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:55:18.712 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:55:18.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.932+0000 7f05fb7fe640 1 -- 172.21.15.40:0/174602949 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f002cc00 con 0x7f05f4094c90 2024-03-18T22:55:18.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.932+0000 7f0601b2e640 1 -- 172.21.15.40:0/174602949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05f4094c90 msgr2=0x7f05f40950b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:18.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.932+0000 7f0601b2e640 1 --2- 172.21.15.40:0/174602949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05f4094c90 0x7f05f40950b0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f05f0009900 tx=0x7f05f002bc50 comp rx=0 tx=0).stop 2024-03-18T22:55:18.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.932+0000 7f0601b2e640 1 -- 172.21.15.40:0/174602949 shutdown_connections 2024-03-18T22:55:18.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.932+0000 7f0601b2e640 1 --2- 172.21.15.40:0/174602949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05f4096840 0x7f05f409d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.932+0000 7f0601b2e640 1 --2- 172.21.15.40:0/174602949 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05f4095e80 0x7f05f4096300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.932+0000 7f0601b2e640 1 --2- 172.21.15.40:0/174602949 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05f4094c90 0x7f05f40950b0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.932+0000 7f0601b2e640 1 -- 172.21.15.40:0/174602949 >> 172.21.15.40:0/174602949 conn(0x7f05f40901e0 msgr2=0x7f05f4092640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:18.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.933+0000 7f0601b2e640 1 -- 172.21.15.40:0/174602949 shutdown_connections 2024-03-18T22:55:18.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.933+0000 7f0601b2e640 1 -- 172.21.15.40:0/174602949 wait complete. 2024-03-18T22:55:18.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.933+0000 7f0601b2e640 1 Processor -- start 2024-03-18T22:55:18.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.933+0000 7f0601b2e640 1 -- start start 2024-03-18T22:55:18.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.934+0000 7f0601b2e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05f4094c90 0x7f05f4129e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.934+0000 7f0601b2e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05f4095e80 0x7f05f412a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.934+0000 7f0601b2e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05f4096840 0x7f05f4131470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.934+0000 7f0601b2e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f05f409f7c0 con 0x7f05f4094c90 2024-03-18T22:55:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.934+0000 7f0601b2e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f05f409f640 con 0x7f05f4096840 2024-03-18T22:55:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.934+0000 7f0601b2e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f05f409f940 con 0x7f05f4095e80 2024-03-18T22:55:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.934+0000 7f05fbfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05f4095e80 0x7f05f412a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.934+0000 7f05fbfff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05f4095e80 0x7f05f412a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:37676/0 (socket says 172.21.15.40:37676) 2024-03-18T22:55:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.934+0000 7f05fbfff640 1 -- 172.21.15.40:0/4202353723 learned_addr learned my addr 172.21.15.40:0/4202353723 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.934+0000 7f060132d640 1 --2- 172.21.15.40:0/4202353723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05f4096840 0x7f05f4131470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.934+0000 7f05fbfff640 1 -- 172.21.15.40:0/4202353723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05f4096840 msgr2=0x7f05f4131470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.934+0000 7f05fbfff640 1 --2- 172.21.15.40:0/4202353723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05f4096840 0x7f05f4131470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.934+0000 7f05fbfff640 1 -- 172.21.15.40:0/4202353723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05f4094c90 msgr2=0x7f05f4129e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T22:55:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.934+0000 7f05fbfff640 1 --2- 172.21.15.40:0/4202353723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05f4094c90 0x7f05f4129e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.934+0000 7f05fbfff640 1 -- 172.21.15.40:0/4202353723 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05f4131b10 con 0x7f05f4095e80 2024-03-18T22:55:18.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.935+0000 7f05fbfff640 1 --2- 172.21.15.40:0/4202353723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05f4095e80 0x7f05f412a3c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f05e800b450 tx=0x7f05e800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:18.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.935+0000 7f05f9ffb640 1 -- 172.21.15.40:0/4202353723 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e8015020 con 0x7f05f4095e80 2024-03-18T22:55:18.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.935+0000 7f0601b2e640 1 -- 172.21.15.40:0/4202353723 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05f4131e00 con 0x7f05f4095e80 2024-03-18T22:55:18.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.935+0000 7f05f9ffb640 1 -- 172.21.15.40:0/4202353723 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05e8011d40 con 0x7f05f4095e80 2024-03-18T22:55:18.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.935+0000 7f05f9ffb640 1 -- 172.21.15.40:0/4202353723 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e801ade0 con 0x7f05f4095e80 2024-03-18T22:55:18.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.935+0000 7f0601b2e640 1 -- 172.21.15.40:0/4202353723 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05f4132420 con 0x7f05f4095e80 2024-03-18T22:55:18.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.937+0000 7f0601b2e640 1 -- 172.21.15.40:0/4202353723 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05f41326e0 con 0x7f05f4095e80 2024-03-18T22:55:18.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.940+0000 7f05f9ffb640 1 -- 172.21.15.40:0/4202353723 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f05e8018020 con 0x7f05f4095e80 2024-03-18T22:55:18.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.941+0000 7f05f9ffb640 1 --2- 172.21.15.40:0/4202353723 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f05ec0a2870 0x7f05ec0a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:18.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.941+0000 7f05f9ffb640 1 -- 172.21.15.40:0/4202353723 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f05e80cef00 con 0x7f05f4095e80 2024-03-18T22:55:18.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.942+0000 7f0600b2c640 1 --2- 172.21.15.40:0/4202353723 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f05ec0a2870 0x7f05ec0a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:18.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.943+0000 7f0600b2c640 1 --2- 172.21.15.40:0/4202353723 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f05ec0a2870 0x7f05ec0a4d30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f05f0036040 tx=0x7f05f0005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:18.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:18.947+0000 7f05f9ffb640 1 -- 172.21.15.40:0/4202353723 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f05e801f020 con 0x7f05f4095e80 2024-03-18T22:55:19.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.339+0000 7f0601b2e640 1 -- 172.21.15.40:0/4202353723 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05f41328e0 con 0x7f05f4095e80 2024-03-18T22:55:19.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.342+0000 7f05f9ffb640 1 -- 172.21.15.40:0/4202353723 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11947 (secure 0 0 0) 0x7f05e800d040 con 0x7f05f4095e80 2024-03-18T22:55:19.344 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:55:19.345 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:54:31.450044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24388,24401,24473,24481,24509]},"id":1}]} 2024-03-18T22:55:19.345 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 40 2024-03-18T22:55:19.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.346+0000 7f0601b2e640 1 -- 172.21.15.40:0/4202353723 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f05ec0a2870 msgr2=0x7f05ec0a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.346+0000 7f0601b2e640 1 --2- 172.21.15.40:0/4202353723 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f05ec0a2870 0x7f05ec0a4d30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f05f0036040 tx=0x7f05f0005fd0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.347+0000 7f0601b2e640 1 -- 172.21.15.40:0/4202353723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05f4095e80 msgr2=0x7f05f412a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.347+0000 7f0601b2e640 1 --2- 172.21.15.40:0/4202353723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05f4095e80 0x7f05f412a3c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f05e800b450 tx=0x7f05e800b910 comp rx=0 tx=0).stop 2024-03-18T22:55:19.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.347+0000 7f0601b2e640 1 -- 172.21.15.40:0/4202353723 shutdown_connections 2024-03-18T22:55:19.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.347+0000 7f0601b2e640 1 --2- 172.21.15.40:0/4202353723 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f05ec0a2870 0x7f05ec0a4d30 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.347+0000 7f0601b2e640 1 --2- 172.21.15.40:0/4202353723 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05f4096840 0x7f05f4131470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.347+0000 7f0601b2e640 1 --2- 172.21.15.40:0/4202353723 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f05f4095e80 0x7f05f412a3c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.347+0000 7f0601b2e640 1 --2- 172.21.15.40:0/4202353723 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05f4094c90 0x7f05f4129e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.347+0000 7f0601b2e640 1 -- 172.21.15.40:0/4202353723 >> 172.21.15.40:0/4202353723 conn(0x7f05f40901e0 msgr2=0x7f05f4099250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:19.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.348+0000 7f0601b2e640 1 -- 172.21.15.40:0/4202353723 shutdown_connections 2024-03-18T22:55:19.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.348+0000 7f0601b2e640 1 -- 172.21.15.40:0/4202353723 wait complete. 2024-03-18T22:55:19.364 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:55:19.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.520+0000 7efdccc0b640 1 -- 172.21.15.40:0/3157721164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc8072b90 msgr2=0x7efdc8072fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.520+0000 7efdccc0b640 1 --2- 172.21.15.40:0/3157721164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc8072b90 0x7efdc8072fb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7efdc0009be0 tx=0x7efdc002bf30 comp rx=0 tx=0).stop 2024-03-18T22:55:19.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.527+0000 7efdccc0b640 1 -- 172.21.15.40:0/3157721164 shutdown_connections 2024-03-18T22:55:19.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.527+0000 7efdccc0b640 1 --2- 172.21.15.40:0/3157721164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdc8109820 0x7efdc810e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.527+0000 7efdccc0b640 1 --2- 172.21.15.40:0/3157721164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc8108e60 0x7efdc81092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.527+0000 7efdccc0b640 1 --2- 172.21.15.40:0/3157721164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc8072b90 0x7efdc8072fb0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.527+0000 7efdccc0b640 1 -- 172.21.15.40:0/3157721164 >> 172.21.15.40:0/3157721164 conn(0x7efdc806e640 msgr2=0x7efdc8070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:19.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.531+0000 7efdccc0b640 1 -- 172.21.15.40:0/3157721164 shutdown_connections 2024-03-18T22:55:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.534+0000 7efdccc0b640 1 -- 172.21.15.40:0/3157721164 wait complete. 2024-03-18T22:55:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.535+0000 7efdccc0b640 1 Processor -- start 2024-03-18T22:55:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.537+0000 7efdccc0b640 1 -- start start 2024-03-18T22:55:19.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.537+0000 7efdccc0b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc8072b90 0x7efdc81a66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.537+0000 7efdccc0b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdc8108e60 0x7efdc81a6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.537+0000 7efdccc0b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc8109820 0x7efdc8079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.537+0000 7efdccc0b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efdc811c060 con 0x7efdc8072b90 2024-03-18T22:55:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.537+0000 7efdccc0b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efdc811bee0 con 0x7efdc8109820 2024-03-18T22:55:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.537+0000 7efdccc0b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7efdc811c1e0 con 0x7efdc8108e60 2024-03-18T22:55:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.539+0000 7efdc5d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdc8108e60 0x7efdc81a6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.539+0000 7efdc5d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdc8108e60 0x7efdc81a6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:37698/0 (socket says 172.21.15.40:37698) 2024-03-18T22:55:19.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.539+0000 7efdc5d9b640 1 -- 172.21.15.40:0/3181395633 learned_addr learned my addr 172.21.15.40:0/3181395633 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:19.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.539+0000 7efdc6d9d640 1 --2- 172.21.15.40:0/3181395633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc8109820 0x7efdc8079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.541+0000 7efdc659c640 1 --2- 172.21.15.40:0/3181395633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc8072b90 0x7efdc81a66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.542+0000 7efdc659c640 1 -- 172.21.15.40:0/3181395633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdc8108e60 msgr2=0x7efdc81a6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.542+0000 7efdc659c640 1 --2- 172.21.15.40:0/3181395633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdc8108e60 0x7efdc81a6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.542+0000 7efdc659c640 1 -- 172.21.15.40:0/3181395633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc8109820 msgr2=0x7efdc8079a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.564+0000 7efdc659c640 1 --2- 172.21.15.40:0/3181395633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc8109820 0x7efdc8079a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.564+0000 7efdc659c640 1 -- 172.21.15.40:0/3181395633 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc807a120 con 0x7efdc8072b90 2024-03-18T22:55:19.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.564+0000 7efdc659c640 1 --2- 172.21.15.40:0/3181395633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc8072b90 0x7efdc81a66a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7efdc002c470 tx=0x7efdc0033d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:19.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.565+0000 7efdc6d9d640 1 --2- 172.21.15.40:0/3181395633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc8109820 0x7efdc8079a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:55:19.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.568+0000 7efdb77fe640 1 -- 172.21.15.40:0/3181395633 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0043020 con 0x7efdc8072b90 2024-03-18T22:55:19.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.568+0000 7efdccc0b640 1 -- 172.21.15.40:0/3181395633 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdc807a3b0 con 0x7efdc8072b90 2024-03-18T22:55:19.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.568+0000 7efdccc0b640 1 -- 172.21.15.40:0/3181395633 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdc807a950 con 0x7efdc8072b90 2024-03-18T22:55:19.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.568+0000 7efdb77fe640 1 -- 172.21.15.40:0/3181395633 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efdc0034af0 con 0x7efdc8072b90 2024-03-18T22:55:19.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.568+0000 7efdb77fe640 1 -- 172.21.15.40:0/3181395633 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc003ed70 con 0x7efdc8072b90 2024-03-18T22:55:19.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.572+0000 7efdcf7f6740 1 Processor -- start 2024-03-18T22:55:19.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.573+0000 7efdcf7f6740 1 -- start start 2024-03-18T22:55:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.573+0000 7efdcf7f6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 0x557866f5b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.573+0000 7efdcf7f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f5be90 0x557866f60990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.573+0000 7efdcf7f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f61000 0x557866f63490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.573+0000 7efdcf7f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557866eae4b0 con 0x557866f61000 2024-03-18T22:55:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.573+0000 7efdcf7f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557866e56d70 con 0x557866f5be90 2024-03-18T22:55:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.573+0000 7efdcf7f6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557866eaa550 con 0x557866f5b4a0 2024-03-18T22:55:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.574+0000 7efdc6d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 0x557866f5b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.575+0000 7efdc5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f61000 0x557866f63490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.575+0000 7efdc5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f61000 0x557866f63490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35248/0 (socket says 172.21.15.40:35248) 2024-03-18T22:55:19.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.575+0000 7efdc5d9b640 1 -- 172.21.15.40:0/2127737089 learned_addr learned my addr 172.21.15.40:0/2127737089 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:19.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.577+0000 7efdb77fe640 1 -- 172.21.15.40:0/3181395633 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7efdc0046020 con 0x7efdc8072b90 2024-03-18T22:55:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.578+0000 7efdc6d9d640 1 -- 172.21.15.40:0/2127737089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f5be90 msgr2=0x557866f60990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:55:19.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.578+0000 7efdc6d9d640 1 --2- 172.21.15.40:0/2127737089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f5be90 0x557866f60990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.579+0000 7efdc6d9d640 1 -- 172.21.15.40:0/2127737089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f61000 msgr2=0x557866f63490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.579+0000 7efdc6d9d640 1 --2- 172.21.15.40:0/2127737089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f61000 0x557866f63490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.579+0000 7efdc6d9d640 1 -- 172.21.15.40:0/2127737089 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557866e79210 con 0x557866f5b4a0 2024-03-18T22:55:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.579+0000 7efdc5d9b640 1 --2- 172.21.15.40:0/2127737089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f61000 0x557866f63490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:55:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.579+0000 7efdb77fe640 1 --2- 172.21.15.40:0/3181395633 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efdb00a29d0 0x7efdb00a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.579+0000 7efdc6d9d640 1 --2- 172.21.15.40:0/2127737089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 0x557866f5b8c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efdb800a8c0 tx=0x7efdb800ad80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eeb3e8ee9037276d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:19.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.580+0000 7efdc659c640 1 --2- 172.21.15.40:0/3181395633 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efdb00a29d0 0x7efdb00a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.580+0000 7efdb77fe640 1 -- 172.21.15.40:0/3181395633 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7efdc00f1820 con 0x7efdc8072b90 2024-03-18T22:55:19.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.580+0000 7efdb4ff9640 1 -- 172.21.15.40:0/2127737089 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb8015070 con 0x557866f5b4a0 2024-03-18T22:55:19.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.580+0000 7efdb4ff9640 1 -- 172.21.15.40:0/2127737089 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efdb80113f0 con 0x557866f5b4a0 2024-03-18T22:55:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.582+0000 7efdcf7f6740 1 -- 172.21.15.40:0/2127737089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 msgr2=0x557866f5b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.582+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/2127737089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 0x557866f5b8c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efdb800a8c0 tx=0x7efdb800ad80 comp rx=0 tx=0).stop 2024-03-18T22:55:19.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.582+0000 7efdc659c640 1 --2- 172.21.15.40:0/3181395633 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efdb00a29d0 0x7efdb00a4e90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7efdc00bda70 tx=0x7efdc0090b20 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:19.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.583+0000 7efdcf7f6740 1 -- 172.21.15.40:0/2127737089 shutdown_connections 2024-03-18T22:55:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.583+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/2127737089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f61000 0x557866f63490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.583+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/2127737089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f5be90 0x557866f60990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.583+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/2127737089 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 0x557866f5b8c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.583+0000 7efdcf7f6740 1 -- 172.21.15.40:0/2127737089 >> 172.21.15.40:0/2127737089 conn(0x557866f56a10 msgr2=0x557866f56e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.584+0000 7efdcf7f6740 1 -- 172.21.15.40:0/2127737089 shutdown_connections 2024-03-18T22:55:19.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.584+0000 7efdcf7f6740 1 -- 172.21.15.40:0/2127737089 wait complete. 2024-03-18T22:55:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.584+0000 7efdcf7f6740 1 -- start start 2024-03-18T22:55:19.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.584+0000 7efdcf7f6740 1 Processor -- start 2024-03-18T22:55:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.585+0000 7efdcf7f6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 0x557866f65a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.585+0000 7efdcf7f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f5be90 0x557866f65fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.585+0000 7efdcf7f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f61000 0x557866f6cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.585+0000 7efdcf7f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557866f64000 con 0x557866f5be90 2024-03-18T22:55:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.585+0000 7efdcf7f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557866e6a310 con 0x557866f61000 2024-03-18T22:55:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.585+0000 7efdcf7f6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557866f64180 con 0x557866f5b4a0 2024-03-18T22:55:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.585+0000 7efdcf7f6740 10 client.0 _ll_register_callbacks cb 0x557866e5e2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.585+0000 7efdc5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f5be90 0x557866f65fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.585+0000 7efdc5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f5be90 0x557866f65fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35254/0 (socket says 172.21.15.40:35254) 2024-03-18T22:55:19.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.585+0000 7efdc5d9b640 1 -- 172.21.15.40:0/218383159 learned_addr learned my addr 172.21.15.40:0/218383159 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:19.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.587+0000 7efdc5d9b640 1 -- 172.21.15.40:0/218383159 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 msgr2=0x557866f65a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:19.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.587+0000 7efdc5d9b640 1 --2- 172.21.15.40:0/218383159 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 0x557866f65a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.587+0000 7efdc5d9b640 1 -- 172.21.15.40:0/218383159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f61000 msgr2=0x557866f6cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:19.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.587+0000 7efdc5d9b640 1 --2- 172.21.15.40:0/218383159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f61000 0x557866f6cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.587+0000 7efdc5d9b640 1 -- 172.21.15.40:0/218383159 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557866bc23f0 con 0x557866f5be90 2024-03-18T22:55:19.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.587+0000 7efdc5d9b640 1 --2- 172.21.15.40:0/218383159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f5be90 0x557866f65fa0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7efdbc011410 tx=0x7efdbc0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7599c3412d5eb64f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:19.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.588+0000 7efdadffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:55:19.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.588+0000 7efdadffb640 1 -- 172.21.15.40:0/218383159 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc004440 con 0x557866f5be90 2024-03-18T22:55:19.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.588+0000 7efdadffb640 1 -- 172.21.15.40:0/218383159 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7efdbc005020 con 0x557866f5be90 2024-03-18T22:55:19.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.588+0000 7efdcf7f6740 1 -- 172.21.15.40:0/218383159 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdb80113f0 con 0x557866f5be90 2024-03-18T22:55:19.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.594+0000 7efdadffb640 1 -- 172.21.15.40:0/218383159 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7efdbc005220 con 0x557866f5be90 2024-03-18T22:55:19.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.595+0000 7efdcf7f6740 10 client.18324 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:19.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.595+0000 7efdcf7f6740 1 -- 172.21.15.40:0/218383159 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x557866c8c7d0 con 0x557866f5be90 2024-03-18T22:55:19.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.595+0000 7efdadffb640 1 -- 172.21.15.40:0/218383159 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efdb80113f0 con 0x557866f5be90 2024-03-18T22:55:19.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.595+0000 7efdadffb640 1 -- 172.21.15.40:0/218383159 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7efdbc003770 con 0x557866f5be90 2024-03-18T22:55:19.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.595+0000 7efdcf7f6740 10 client.18324 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:19.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.595+0000 7efdcf7f6740 10 client.18324 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:55:19.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.596+0000 7efdcf7f6740 20 client.18324 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:19.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.596+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/218383159 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557866f71e40 0x557866f742a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.596+0000 7efdcf7f6740 4 client.18324 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:55:19.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.596+0000 7efdcf7f6740 1 -- 172.21.15.40:0/218383159 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557866f0c3e0 con 0x557866f71e40 2024-03-18T22:55:19.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.597+0000 7efdc6d9d640 1 --2- 172.21.15.40:0/218383159 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557866f71e40 0x557866f742a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.600+0000 7efdc6d9d640 1 --2- 172.21.15.40:0/218383159 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557866f71e40 0x557866f742a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=71cddd163da628fc server_cookie=ad780a3c91f2aa1d in_seq=0 out_seq=0 2024-03-18T22:55:19.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.600+0000 7efdadffb640 10 client.18324 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:55:19.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.602+0000 7efdadffb640 1 -- 172.21.15.40:0/218383159 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557866f0c3e0 con 0x557866f71e40 2024-03-18T22:55:19.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.602+0000 7efdadffb640 10 client.18324 handle_command_reply: tid=0 2024-03-18T22:55:19.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.602+0000 7efdcf7f6740 1 client.18324 shutdown 2024-03-18T22:55:19.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.606+0000 7efdcf7f6740 1 -- 172.21.15.40:0/218383159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f5be90 msgr2=0x557866f65fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.606+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/218383159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f5be90 0x557866f65fa0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7efdbc011410 tx=0x7efdbc0118d0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.606+0000 7efdcf7f6740 1 -- 172.21.15.40:0/218383159 shutdown_connections 2024-03-18T22:55:19.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.606+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/218383159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f61000 0x557866f6cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.606+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/218383159 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557866f71e40 0x557866f742a0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.606+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/218383159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f5be90 0x557866f65fa0 secure :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7efdbc011410 tx=0x7efdbc0118d0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.606+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/218383159 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 0x557866f65a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.606+0000 7efdcf7f6740 1 -- 172.21.15.40:0/218383159 >> 172.21.15.40:0/218383159 conn(0x557866f56a10 msgr2=0x557866f44d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:19.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.606+0000 7efdadffb640 0 client.18324 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:55:19.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.606+0000 7efdcf7f6740 1 -- 172.21.15.40:0/218383159 shutdown_connections 2024-03-18T22:55:19.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.606+0000 7efdcf7f6740 1 -- 172.21.15.40:0/218383159 wait complete. 2024-03-18T22:55:19.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.607+0000 7efdcf7f6740 20 client.18324 trim_cache size 0 max 16384 2024-03-18T22:55:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.633+0000 7efdcf7f6740 1 Processor -- start 2024-03-18T22:55:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.633+0000 7efdcf7f6740 1 -- start start 2024-03-18T22:55:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.633+0000 7efdcf7f6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 0x557866f65460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.633+0000 7efdcf7f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f61000 0x557866f6cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.633+0000 7efdcf7f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f57250 0x557866f44470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.633+0000 7efdcf7f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efda800feb0 con 0x557866f57250 2024-03-18T22:55:19.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.633+0000 7efdcf7f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efda80119f0 con 0x557866f61000 2024-03-18T22:55:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.633+0000 7efdcf7f6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7efda8010580 con 0x557866f5b4a0 2024-03-18T22:55:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.633+0000 7efdc6d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 0x557866f65460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.633+0000 7efdc6d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 0x557866f65460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:37718/0 (socket says 172.21.15.40:37718) 2024-03-18T22:55:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.633+0000 7efdc5d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f61000 0x557866f6cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.633+0000 7efdc6d9d640 1 -- 172.21.15.40:0/1421872444 learned_addr learned my addr 172.21.15.40:0/1421872444 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.633+0000 7efdc6d9d640 1 --2- 172.21.15.40:0/1421872444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f57250 0x557866f44470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.634+0000 7efdc6d9d640 1 -- 172.21.15.40:0/1421872444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f61000 msgr2=0x557866f6cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.634+0000 7efdc6d9d640 1 --2- 172.21.15.40:0/1421872444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f61000 0x557866f6cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.634+0000 7efdc6d9d640 1 -- 172.21.15.40:0/1421872444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f57250 msgr2=0x557866f44470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.634+0000 7efdc6d9d640 1 --2- 172.21.15.40:0/1421872444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f57250 0x557866f44470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.634+0000 7efdc6d9d640 1 -- 172.21.15.40:0/1421872444 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557866f605e0 con 0x557866f5b4a0 2024-03-18T22:55:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.634+0000 7efdc5d9b640 1 --2- 172.21.15.40:0/1421872444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f61000 0x557866f6cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:55:19.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.634+0000 7efdc6d9d640 1 --2- 172.21.15.40:0/1421872444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 0x557866f65460 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efdb8000c00 tx=0x7efdbc00baf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=824d04851395aea5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:19.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.634+0000 7efdaffff640 1 -- 172.21.15.40:0/1421872444 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb8015040 con 0x557866f5b4a0 2024-03-18T22:55:19.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.634+0000 7efdaffff640 1 -- 172.21.15.40:0/1421872444 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efdb8011ab0 con 0x557866f5b4a0 2024-03-18T22:55:19.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.634+0000 7efdaffff640 1 -- 172.21.15.40:0/1421872444 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb8023230 con 0x557866f5b4a0 2024-03-18T22:55:19.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.635+0000 7efdcf7f6740 1 -- 172.21.15.40:0/1421872444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 msgr2=0x557866f65460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.635+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/1421872444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 0x557866f65460 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efdb8000c00 tx=0x7efdbc00baf0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.635+0000 7efdcf7f6740 1 -- 172.21.15.40:0/1421872444 shutdown_connections 2024-03-18T22:55:19.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.635+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/1421872444 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f57250 0x557866f44470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.635+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/1421872444 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f61000 0x557866f6cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.635+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/1421872444 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f5b4a0 0x557866f65460 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.635+0000 7efdcf7f6740 1 -- 172.21.15.40:0/1421872444 >> 172.21.15.40:0/1421872444 conn(0x557866f56a10 msgr2=0x557866f64d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:19.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.635+0000 7efdcf7f6740 1 -- 172.21.15.40:0/1421872444 shutdown_connections 2024-03-18T22:55:19.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.635+0000 7efdcf7f6740 1 -- 172.21.15.40:0/1421872444 wait complete. 2024-03-18T22:55:19.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.635+0000 7efdcf7f6740 1 -- start start 2024-03-18T22:55:19.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.635+0000 7efdcf7f6740 1 Processor -- start 2024-03-18T22:55:19.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdcf7f6740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f57250 0x557866f6a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdcf7f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f5b4a0 0x557866f6aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdcf7f6740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f61000 0x557866f6bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdcf7f6740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efda8011320 con 0x557866f57250 2024-03-18T22:55:19.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdcf7f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efda8010c50 con 0x557866f5b4a0 2024-03-18T22:55:19.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdcf7f6740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7efda800de90 con 0x557866f61000 2024-03-18T22:55:19.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdcf7f6740 10 client.0 _ll_register_callbacks cb 0x7efdbc00bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:19.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdc6d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f57250 0x557866f6a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdc6d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f57250 0x557866f6a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35264/0 (socket says 172.21.15.40:35264) 2024-03-18T22:55:19.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdc6d9d640 1 -- 172.21.15.40:0/1320051571 learned_addr learned my addr 172.21.15.40:0/1320051571 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:19.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdc6d9d640 1 --2- 172.21.15.40:0/1320051571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f61000 0x557866f6bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdc5d9b640 1 --2- 172.21.15.40:0/1320051571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f5b4a0 0x557866f6aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdc6d9d640 1 -- 172.21.15.40:0/1320051571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f61000 msgr2=0x557866f6bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:19.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdc6d9d640 1 --2- 172.21.15.40:0/1320051571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f61000 0x557866f6bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdc6d9d640 1 -- 172.21.15.40:0/1320051571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f5b4a0 msgr2=0x557866f6aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:19.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdc6d9d640 1 --2- 172.21.15.40:0/1320051571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f5b4a0 0x557866f6aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdc6d9d640 1 -- 172.21.15.40:0/1320051571 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557866f6c0a0 con 0x557866f57250 2024-03-18T22:55:19.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdc6d9d640 1 --2- 172.21.15.40:0/1320051571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f57250 0x557866f6a560 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7efdb8004700 tx=0x7efdb800a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60955779d6f020ef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:19.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.636+0000 7efdadffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:55:19.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.637+0000 7efdadffb640 1 -- 172.21.15.40:0/1320051571 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb8015040 con 0x557866f57250 2024-03-18T22:55:19.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.637+0000 7efdcf7f6740 1 -- 172.21.15.40:0/1320051571 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdb8011ab0 con 0x557866f57250 2024-03-18T22:55:19.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.637+0000 7efdadffb640 1 -- 172.21.15.40:0/1320051571 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7efdb8011800 con 0x557866f57250 2024-03-18T22:55:19.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.637+0000 7efdadffb640 1 -- 172.21.15.40:0/1320051571 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7efdb803e050 con 0x557866f57250 2024-03-18T22:55:19.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.637+0000 7efdcf7f6740 10 client.18330 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:19.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.637+0000 7efdcf7f6740 1 -- 172.21.15.40:0/1320051571 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x557866fd81a0 con 0x557866f57250 2024-03-18T22:55:19.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.637+0000 7efdadffb640 1 -- 172.21.15.40:0/1320051571 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efdb8011ab0 con 0x557866f57250 2024-03-18T22:55:19.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.637+0000 7efdadffb640 1 -- 172.21.15.40:0/1320051571 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7efdb801a450 con 0x557866f57250 2024-03-18T22:55:19.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.637+0000 7efdcf7f6740 10 client.18330 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:19.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.637+0000 7efdcf7f6740 10 client.18330 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:55:19.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.637+0000 7efdcf7f6740 20 client.18330 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:19.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.637+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/1320051571 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557866fd8810 0x557866fdac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.637+0000 7efdcf7f6740 4 client.18330 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:55:19.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.637+0000 7efdcf7f6740 1 -- 172.21.15.40:0/1320051571 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557866ebb9f0 con 0x557866fd8810 2024-03-18T22:55:19.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.638+0000 7efdc5d9b640 1 --2- 172.21.15.40:0/1320051571 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557866fd8810 0x557866fdac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.638+0000 7efdadffb640 10 client.18330 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:55:19.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.638+0000 7efdc5d9b640 1 --2- 172.21.15.40:0/1320051571 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557866fd8810 0x557866fdac30 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c386534cc17792a9 server_cookie=f8d27a342164020b in_seq=0 out_seq=0 2024-03-18T22:55:19.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.638+0000 7efdadffb640 1 -- 172.21.15.40:0/1320051571 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557866ebb9f0 con 0x557866fd8810 2024-03-18T22:55:19.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.638+0000 7efdadffb640 10 client.18330 handle_command_reply: tid=0 2024-03-18T22:55:19.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.638+0000 7efdcf7f6740 1 client.18330 shutdown 2024-03-18T22:55:19.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.640+0000 7efdcf7f6740 1 -- 172.21.15.40:0/1320051571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f57250 msgr2=0x557866f6a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.640+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/1320051571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f57250 0x557866f6a560 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7efdb8004700 tx=0x7efdb800a7c0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.640+0000 7efdcf7f6740 1 -- 172.21.15.40:0/1320051571 shutdown_connections 2024-03-18T22:55:19.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.640+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/1320051571 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557866f61000 0x557866f6bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.640+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/1320051571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557866f5b4a0 0x557866f6aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.640+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/1320051571 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557866fd8810 0x557866fdac30 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.640+0000 7efdcf7f6740 1 --2- 172.21.15.40:0/1320051571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557866f57250 0x557866f6a560 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.640+0000 7efdcf7f6740 1 -- 172.21.15.40:0/1320051571 >> 172.21.15.40:0/1320051571 conn(0x557866f56a10 msgr2=0x557866f44470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:19.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.640+0000 7efdadffb640 0 client.18330 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:55:19.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.641+0000 7efdcf7f6740 1 -- 172.21.15.40:0/1320051571 shutdown_connections 2024-03-18T22:55:19.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.642+0000 7efdcf7f6740 1 -- 172.21.15.40:0/1320051571 wait complete. 2024-03-18T22:55:19.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.642+0000 7efdcf7f6740 20 client.18330 trim_cache size 0 max 16384 2024-03-18T22:55:19.645 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:55:19.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.642+0000 7efdccc0b640 1 -- 172.21.15.40:0/3181395633 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efdb00a29d0 msgr2=0x7efdb00a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.642+0000 7efdccc0b640 1 --2- 172.21.15.40:0/3181395633 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efdb00a29d0 0x7efdb00a4e90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7efdc00bda70 tx=0x7efdc0090b20 comp rx=0 tx=0).stop 2024-03-18T22:55:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.642+0000 7efdccc0b640 1 -- 172.21.15.40:0/3181395633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc8072b90 msgr2=0x7efdc81a66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.642+0000 7efdccc0b640 1 --2- 172.21.15.40:0/3181395633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc8072b90 0x7efdc81a66a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7efdc002c470 tx=0x7efdc0033d70 comp rx=0 tx=0).stop 2024-03-18T22:55:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.643+0000 7efdccc0b640 1 -- 172.21.15.40:0/3181395633 shutdown_connections 2024-03-18T22:55:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.643+0000 7efdccc0b640 1 --2- 172.21.15.40:0/3181395633 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efdb00a29d0 0x7efdb00a4e90 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.643+0000 7efdccc0b640 1 --2- 172.21.15.40:0/3181395633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efdc8109820 0x7efdc8079a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.643+0000 7efdccc0b640 1 --2- 172.21.15.40:0/3181395633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efdc8108e60 0x7efdc81a6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.643+0000 7efdccc0b640 1 --2- 172.21.15.40:0/3181395633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efdc8072b90 0x7efdc81a66a0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.643+0000 7efdccc0b640 1 -- 172.21.15.40:0/3181395633 >> 172.21.15.40:0/3181395633 conn(0x7efdc806e640 msgr2=0x7efdc8070280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:19.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.646+0000 7efdccc0b640 1 -- 172.21.15.40:0/3181395633 shutdown_connections 2024-03-18T22:55:19.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.647+0000 7efdccc0b640 1 -- 172.21.15.40:0/3181395633 wait complete. 2024-03-18T22:55:19.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:55:19.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.834+0000 7f678e844640 1 -- 172.21.15.40:0/2191715512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6788102a00 msgr2=0x7f6788104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.834+0000 7f678e844640 1 --2- 172.21.15.40:0/2191715512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6788102a00 0x7f6788104e70 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f6778009900 tx=0x7f677802bc80 comp rx=0 tx=0).stop 2024-03-18T22:55:19.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.835+0000 7f678e844640 1 -- 172.21.15.40:0/2191715512 shutdown_connections 2024-03-18T22:55:19.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.835+0000 7f678e844640 1 --2- 172.21.15.40:0/2191715512 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f67881053b0 0x7f6788105830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.835+0000 7f678e844640 1 --2- 172.21.15.40:0/2191715512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6788102a00 0x7f6788104e70 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.835+0000 7f678e844640 1 --2- 172.21.15.40:0/2191715512 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6788100090 0x7f67881024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.835+0000 7f678e844640 1 -- 172.21.15.40:0/2191715512 >> 172.21.15.40:0/2191715512 conn(0x7f67880fbc90 msgr2=0x7f67880fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:19.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.838+0000 7f678e844640 1 -- 172.21.15.40:0/2191715512 shutdown_connections 2024-03-18T22:55:19.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.839+0000 7f678e844640 1 -- 172.21.15.40:0/2191715512 wait complete. 2024-03-18T22:55:19.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.839+0000 7f678e844640 1 Processor -- start 2024-03-18T22:55:19.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.839+0000 7f678e844640 1 -- start start 2024-03-18T22:55:19.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.840+0000 7f678e844640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6788100090 0x7f678819e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.840+0000 7f678e844640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6788102a00 0x7f678819e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.840+0000 7f678e844640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67881053b0 0x7f67881a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.840+0000 7f678e844640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6788113aa0 con 0x7f67881053b0 2024-03-18T22:55:19.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.840+0000 7f678e844640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6788113920 con 0x7f6788102a00 2024-03-18T22:55:19.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.840+0000 7f678e844640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6788113c20 con 0x7f6788100090 2024-03-18T22:55:19.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.840+0000 7f67877fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6788102a00 0x7f678819e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.840+0000 7f67877fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6788102a00 0x7f678819e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46906/0 (socket says 172.21.15.40:46906) 2024-03-18T22:55:19.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.840+0000 7f67877fe640 1 -- 172.21.15.40:0/3848864855 learned_addr learned my addr 172.21.15.40:0/3848864855 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:19.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.840+0000 7f6787fff640 1 --2- 172.21.15.40:0/3848864855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6788100090 0x7f678819e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.841+0000 7f67877fe640 1 -- 172.21.15.40:0/3848864855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6788100090 msgr2=0x7f678819e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.841+0000 7f67877fe640 1 --2- 172.21.15.40:0/3848864855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6788100090 0x7f678819e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.841+0000 7f67877fe640 1 -- 172.21.15.40:0/3848864855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67881053b0 msgr2=0x7f67881a57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.841+0000 7f67877fe640 1 --2- 172.21.15.40:0/3848864855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67881053b0 0x7f67881a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.841+0000 7f67877fe640 1 -- 172.21.15.40:0/3848864855 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67881a5e20 con 0x7f6788102a00 2024-03-18T22:55:19.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.841+0000 7f6787fff640 1 --2- 172.21.15.40:0/3848864855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6788100090 0x7f678819e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:55:19.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.841+0000 7f67877fe640 1 --2- 172.21.15.40:0/3848864855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6788102a00 0x7f678819e740 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6778000c00 tx=0x7f6778004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:19.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.845+0000 7f67857fa640 1 -- 172.21.15.40:0/3848864855 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6778043020 con 0x7f6788102a00 2024-03-18T22:55:19.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.845+0000 7f678e844640 1 -- 172.21.15.40:0/3848864855 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67881a60b0 con 0x7f6788102a00 2024-03-18T22:55:19.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.845+0000 7f678e844640 1 -- 172.21.15.40:0/3848864855 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67881a6680 con 0x7f6788102a00 2024-03-18T22:55:19.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.845+0000 7f67857fa640 1 -- 172.21.15.40:0/3848864855 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67780052b0 con 0x7f6788102a00 2024-03-18T22:55:19.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.845+0000 7f67857fa640 1 -- 172.21.15.40:0/3848864855 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6778003c70 con 0x7f6788102a00 2024-03-18T22:55:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.850+0000 7f67857fa640 1 -- 172.21.15.40:0/3848864855 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6778046020 con 0x7f6788102a00 2024-03-18T22:55:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.851+0000 7f679142d740 1 Processor -- start 2024-03-18T22:55:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.851+0000 7f679142d740 1 -- start start 2024-03-18T22:55:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.851+0000 7f679142d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc96300 0x558ffbc96720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.851+0000 7f679142d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96cf0 0x558ffbc9b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.851+0000 7f679142d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc9be60 0x558ffbc9e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.851+0000 7f679142d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558ffbbe94b0 con 0x558ffbc96300 2024-03-18T22:55:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.851+0000 7f679142d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558ffbb91d70 con 0x558ffbc96cf0 2024-03-18T22:55:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.851+0000 7f679142d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x558ffbbe5550 con 0x558ffbc9be60 2024-03-18T22:55:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.851+0000 7f6787fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96cf0 0x558ffbc9b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.851+0000 7f6787fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96cf0 0x558ffbc9b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46916/0 (socket says 172.21.15.40:46916) 2024-03-18T22:55:19.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.851+0000 7f6787fff640 1 -- 172.21.15.40:0/2894693662 learned_addr learned my addr 172.21.15.40:0/2894693662 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.852+0000 7f6787fff640 1 -- 172.21.15.40:0/2894693662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc9be60 msgr2=0x558ffbc9e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:55:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.852+0000 7f67877fe640 1 --2- 172.21.15.40:0/2894693662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc9be60 0x558ffbc9e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.852+0000 7f67857fa640 1 --2- 172.21.15.40:0/3848864855 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f67700a2a00 0x7f67700a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.852+0000 7f678cde1640 1 --2- 172.21.15.40:0/2894693662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc96300 0x558ffbc96720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.852+0000 7f6787fff640 1 --2- 172.21.15.40:0/2894693662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc9be60 0x558ffbc9e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.852+0000 7f6787fff640 1 -- 172.21.15.40:0/2894693662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc96300 msgr2=0x558ffbc96720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.852+0000 7f6787fff640 1 --2- 172.21.15.40:0/2894693662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc96300 0x558ffbc96720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.852+0000 7f6787fff640 1 -- 172.21.15.40:0/2894693662 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ffbbb4210 con 0x558ffbc96cf0 2024-03-18T22:55:19.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.852+0000 7f67857fa640 1 -- 172.21.15.40:0/3848864855 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f67780f11d0 con 0x7f6788102a00 2024-03-18T22:55:19.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.853+0000 7f67877fe640 1 --2- 172.21.15.40:0/2894693662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc9be60 0x558ffbc9e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:55:19.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.853+0000 7f678cde1640 1 --2- 172.21.15.40:0/2894693662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc96300 0x558ffbc96720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:55:19.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.853+0000 7f6787fff640 1 --2- 172.21.15.40:0/2894693662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96cf0 0x558ffbc9b7f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f677c00aa90 tx=0x7f677c00af50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3518368bbcea223b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.855+0000 7f676e7fc640 1 -- 172.21.15.40:0/2894693662 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c00cd20 con 0x558ffbc96cf0 2024-03-18T22:55:19.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.855+0000 7f676e7fc640 1 -- 172.21.15.40:0/2894693662 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f677c0118a0 con 0x558ffbc96cf0 2024-03-18T22:55:19.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.855+0000 7f676e7fc640 1 -- 172.21.15.40:0/2894693662 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c010910 con 0x558ffbc96cf0 2024-03-18T22:55:19.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.857+0000 7f678cde1640 1 --2- 172.21.15.40:0/3848864855 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f67700a2a00 0x7f67700a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.859+0000 7f679142d740 1 -- 172.21.15.40:0/2894693662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96cf0 msgr2=0x558ffbc9b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.859+0000 7f679142d740 1 --2- 172.21.15.40:0/2894693662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96cf0 0x558ffbc9b7f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f677c00aa90 tx=0x7f677c00af50 comp rx=0 tx=0).stop 2024-03-18T22:55:19.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.860+0000 7f679142d740 1 -- 172.21.15.40:0/2894693662 shutdown_connections 2024-03-18T22:55:19.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.860+0000 7f679142d740 1 --2- 172.21.15.40:0/2894693662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc9be60 0x558ffbc9e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.861+0000 7f679142d740 1 --2- 172.21.15.40:0/2894693662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96cf0 0x558ffbc9b7f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.861+0000 7f679142d740 1 --2- 172.21.15.40:0/2894693662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc96300 0x558ffbc96720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.861+0000 7f679142d740 1 -- 172.21.15.40:0/2894693662 >> 172.21.15.40:0/2894693662 conn(0x558ffbc80860 msgr2=0x558ffbc80c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.861+0000 7f678cde1640 1 --2- 172.21.15.40:0/3848864855 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f67700a2a00 0x7f67700a4ec0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x558ffbc97550 tx=0x7f6780004ad0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:19.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.868+0000 7f679142d740 1 -- 172.21.15.40:0/2894693662 shutdown_connections 2024-03-18T22:55:19.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.868+0000 7f679142d740 1 -- 172.21.15.40:0/2894693662 wait complete. 2024-03-18T22:55:19.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.868+0000 7f679142d740 1 -- start start 2024-03-18T22:55:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.869+0000 7f679142d740 1 Processor -- start 2024-03-18T22:55:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.869+0000 7f679142d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96300 0x558ffbca0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.869+0000 7f679142d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96cf0 0x558ffbca0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.870+0000 7f679142d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc9be60 0x558ffbca7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.870+0000 7f67877fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96cf0 0x558ffbca0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.870+0000 7f67877fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96cf0 0x558ffbca0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46932/0 (socket says 172.21.15.40:46932) 2024-03-18T22:55:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.870+0000 7f679142d740 1 -- 172.21.15.40:0/3149815128 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x558ffbc9ee10 con 0x558ffbc9be60 2024-03-18T22:55:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.870+0000 7f679142d740 1 -- 172.21.15.40:0/3149815128 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558ffbba5310 con 0x558ffbc96cf0 2024-03-18T22:55:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.870+0000 7f679142d740 1 -- 172.21.15.40:0/3149815128 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x558ffbc9ef90 con 0x558ffbc96300 2024-03-18T22:55:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.870+0000 7f679142d740 10 client.0 _ll_register_callbacks cb 0x558ffbb992d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.870+0000 7f67877fe640 1 -- 172.21.15.40:0/3149815128 learned_addr learned my addr 172.21.15.40:0/3149815128 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.870+0000 7f6787fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96300 0x558ffbca0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.871+0000 7f678cde1640 1 --2- 172.21.15.40:0/3149815128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc9be60 0x558ffbca7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.873+0000 7f6787fff640 1 -- 172.21.15.40:0/3149815128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96cf0 msgr2=0x558ffbca0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.874+0000 7f6787fff640 1 --2- 172.21.15.40:0/3149815128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96cf0 0x558ffbca0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.874+0000 7f6787fff640 1 -- 172.21.15.40:0/3149815128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc9be60 msgr2=0x558ffbca7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.874+0000 7f6787fff640 1 --2- 172.21.15.40:0/3149815128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc9be60 0x558ffbca7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.874+0000 7f6787fff640 1 -- 172.21.15.40:0/3149815128 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ffb8fd3f0 con 0x558ffbc96300 2024-03-18T22:55:19.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.878+0000 7f6787fff640 1 --2- 172.21.15.40:0/3149815128 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96300 0x558ffbca0810 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x558ffbc817d0 tx=0x7f677c00cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=184cd3f8d6ae0b8e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:19.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.884+0000 7f67677fe640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:55:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.884+0000 7f67677fe640 1 -- 172.21.15.40:0/3149815128 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c00cea0 con 0x558ffbc96300 2024-03-18T22:55:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.884+0000 7f67677fe640 1 -- 172.21.15.40:0/3149815128 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f677c011c60 con 0x558ffbc96300 2024-03-18T22:55:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.886+0000 7f679142d740 1 -- 172.21.15.40:0/3149815128 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f677c0118a0 con 0x558ffbc96300 2024-03-18T22:55:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.887+0000 7f67677fe640 1 -- 172.21.15.40:0/3149815128 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f677c00a660 con 0x558ffbc96300 2024-03-18T22:55:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.890+0000 7f679142d740 10 client.28144 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.890+0000 7f679142d740 1 -- 172.21.15.40:0/3149815128 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x558ffb9c77d0 con 0x558ffbc96300 2024-03-18T22:55:19.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.893+0000 7f67677fe640 1 -- 172.21.15.40:0/3149815128 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f677c0118a0 con 0x558ffbc96300 2024-03-18T22:55:19.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.893+0000 7f67677fe640 1 -- 172.21.15.40:0/3149815128 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f677c02b030 con 0x558ffbc96300 2024-03-18T22:55:19.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.895+0000 7f679142d740 10 client.28144 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:19.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.895+0000 7f679142d740 10 client.28144 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:55:19.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.895+0000 7f679142d740 20 client.28144 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:19.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.895+0000 7f679142d740 1 --2- 172.21.15.40:0/3149815128 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x558ffbcac2a0 0x558ffbcae690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.895+0000 7f679142d740 4 client.28144 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:55:19.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.895+0000 7f679142d740 1 -- 172.21.15.40:0/3149815128 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ffbc473e0 con 0x558ffbcac2a0 2024-03-18T22:55:19.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.901+0000 7f67877fe640 1 --2- 172.21.15.40:0/3149815128 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x558ffbcac2a0 0x558ffbcae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.915+0000 7f67877fe640 1 --2- 172.21.15.40:0/3149815128 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x558ffbcac2a0 0x558ffbcae690 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3c81ae39d9a4cc21 server_cookie=df2ee616834fbd12 in_seq=0 out_seq=0 2024-03-18T22:55:19.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.916+0000 7f67677fe640 10 client.28144 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:55:19.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.918+0000 7f67677fe640 1 -- 172.21.15.40:0/3149815128 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558ffbc473e0 con 0x558ffbcac2a0 2024-03-18T22:55:19.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.918+0000 7f67677fe640 10 client.28144 handle_command_reply: tid=0 2024-03-18T22:55:19.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.919+0000 7f679142d740 1 client.28144 shutdown 2024-03-18T22:55:19.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.923+0000 7f679142d740 1 -- 172.21.15.40:0/3149815128 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96300 msgr2=0x558ffbca0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.923+0000 7f679142d740 1 --2- 172.21.15.40:0/3149815128 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96300 0x558ffbca0810 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x558ffbc817d0 tx=0x7f677c00cd30 comp rx=0 tx=0).stop 2024-03-18T22:55:19.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.926+0000 7f679142d740 1 -- 172.21.15.40:0/3149815128 shutdown_connections 2024-03-18T22:55:19.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.926+0000 7f679142d740 1 --2- 172.21.15.40:0/3149815128 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x558ffbcac2a0 0x558ffbcae690 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.926+0000 7f679142d740 1 --2- 172.21.15.40:0/3149815128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc9be60 0x558ffbca7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.926+0000 7f679142d740 1 --2- 172.21.15.40:0/3149815128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96cf0 0x558ffbca0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.926+0000 7f679142d740 1 --2- 172.21.15.40:0/3149815128 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96300 0x558ffbca0810 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.926+0000 7f679142d740 1 -- 172.21.15.40:0/3149815128 >> 172.21.15.40:0/3149815128 conn(0x558ffbc80860 msgr2=0x558ffbc7fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:19.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.926+0000 7f67677fe640 0 client.28144 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:55:19.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.926+0000 7f679142d740 1 -- 172.21.15.40:0/3149815128 shutdown_connections 2024-03-18T22:55:19.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.927+0000 7f679142d740 1 -- 172.21.15.40:0/3149815128 wait complete. 2024-03-18T22:55:19.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.927+0000 7f679142d740 20 client.28144 trim_cache size 0 max 16384 2024-03-18T22:55:19.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.959+0000 7f679142d740 1 Processor -- start 2024-03-18T22:55:19.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.961+0000 7f679142d740 1 -- start start 2024-03-18T22:55:19.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.961+0000 7f679142d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96300 0x558ffbc7fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.961+0000 7f679142d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96cf0 0x558ffbca7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.961+0000 7f679142d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc9be60 0x558ffbcb2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.961+0000 7f679142d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f677400fe20 con 0x558ffbc9be60 2024-03-18T22:55:19.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.961+0000 7f679142d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6774011960 con 0x558ffbc96300 2024-03-18T22:55:19.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.961+0000 7f679142d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f67740104f0 con 0x558ffbc96cf0 2024-03-18T22:55:19.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.962+0000 7f6787fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96300 0x558ffbc7fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.962+0000 7f6787fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96300 0x558ffbc7fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46938/0 (socket says 172.21.15.40:46938) 2024-03-18T22:55:19.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.962+0000 7f6787fff640 1 -- 172.21.15.40:0/3813814587 learned_addr learned my addr 172.21.15.40:0/3813814587 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:19.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.962+0000 7f67877fe640 1 --2- 172.21.15.40:0/3813814587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96cf0 0x558ffbca7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.962+0000 7f678cde1640 1 --2- 172.21.15.40:0/3813814587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc9be60 0x558ffbcb2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.963+0000 7f6787fff640 1 -- 172.21.15.40:0/3813814587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96cf0 msgr2=0x558ffbca7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:19.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.963+0000 7f6787fff640 1 --2- 172.21.15.40:0/3813814587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96cf0 0x558ffbca7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.963+0000 7f6787fff640 1 -- 172.21.15.40:0/3813814587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc9be60 msgr2=0x558ffbcb2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:19.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.963+0000 7f6787fff640 1 --2- 172.21.15.40:0/3813814587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc9be60 0x558ffbcb2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.963+0000 7f6787fff640 1 -- 172.21.15.40:0/3813814587 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ffbc97bf0 con 0x558ffbc96300 2024-03-18T22:55:19.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.964+0000 7f6787fff640 1 --2- 172.21.15.40:0/3813814587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96300 0x558ffbc7fa20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f677c00a5a0 tx=0x7f677c00a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efb32d65fa0a197e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:19.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.967+0000 7f676dffb640 1 -- 172.21.15.40:0/3813814587 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c034070 con 0x558ffbc96300 2024-03-18T22:55:19.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.967+0000 7f676dffb640 1 -- 172.21.15.40:0/3813814587 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f677c00a370 con 0x558ffbc96300 2024-03-18T22:55:19.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.967+0000 7f676dffb640 1 -- 172.21.15.40:0/3813814587 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c02cba0 con 0x558ffbc96300 2024-03-18T22:55:19.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.969+0000 7f679142d740 1 -- 172.21.15.40:0/3813814587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96300 msgr2=0x558ffbc7fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:19.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.969+0000 7f679142d740 1 --2- 172.21.15.40:0/3813814587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96300 0x558ffbc7fa20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f677c00a5a0 tx=0x7f677c00a5d0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.971+0000 7f679142d740 1 -- 172.21.15.40:0/3813814587 shutdown_connections 2024-03-18T22:55:19.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.972+0000 7f679142d740 1 --2- 172.21.15.40:0/3813814587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc9be60 0x558ffbcb2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.972+0000 7f679142d740 1 --2- 172.21.15.40:0/3813814587 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96cf0 0x558ffbca7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.972+0000 7f679142d740 1 --2- 172.21.15.40:0/3813814587 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc96300 0x558ffbc7fa20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.972+0000 7f679142d740 1 -- 172.21.15.40:0/3813814587 >> 172.21.15.40:0/3813814587 conn(0x558ffbc80860 msgr2=0x558ffbc7f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:19.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.973+0000 7f679142d740 1 -- 172.21.15.40:0/3813814587 shutdown_connections 2024-03-18T22:55:19.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.973+0000 7f679142d740 1 -- 172.21.15.40:0/3813814587 wait complete. 2024-03-18T22:55:19.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.974+0000 7f679142d740 1 -- start start 2024-03-18T22:55:19.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.974+0000 7f679142d740 1 Processor -- start 2024-03-18T22:55:19.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.975+0000 7f679142d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96300 0x558ffbca4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.975+0000 7f679142d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc96cf0 0x558ffbca4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.975+0000 7f679142d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc9be60 0x558ffbd10500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.975+0000 7f679142d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6774011290 con 0x558ffbc96cf0 2024-03-18T22:55:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.975+0000 7f679142d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6774010bc0 con 0x558ffbc9be60 2024-03-18T22:55:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.975+0000 7f679142d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f677400de00 con 0x558ffbc96300 2024-03-18T22:55:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.975+0000 7f679142d740 10 client.0 _ll_register_callbacks cb 0x7f677c00a8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:19.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.976+0000 7f67877fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc96cf0 0x558ffbca4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.976+0000 7f67877fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc96cf0 0x558ffbca4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35310/0 (socket says 172.21.15.40:35310) 2024-03-18T22:55:19.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.976+0000 7f67877fe640 1 -- 172.21.15.40:0/2724342098 learned_addr learned my addr 172.21.15.40:0/2724342098 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:19.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.979+0000 7f678cde1640 1 --2- 172.21.15.40:0/2724342098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc9be60 0x558ffbd10500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.979+0000 7f6787fff640 1 --2- 172.21.15.40:0/2724342098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96300 0x558ffbca4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.980+0000 7f67877fe640 1 -- 172.21.15.40:0/2724342098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96300 msgr2=0x558ffbca4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:19.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.980+0000 7f67877fe640 1 --2- 172.21.15.40:0/2724342098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96300 0x558ffbca4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.980+0000 7f67877fe640 1 -- 172.21.15.40:0/2724342098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc9be60 msgr2=0x558ffbd10500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:19.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.980+0000 7f67877fe640 1 --2- 172.21.15.40:0/2724342098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc9be60 0x558ffbd10500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:19.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.980+0000 7f67877fe640 1 -- 172.21.15.40:0/2724342098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ffbca7ea0 con 0x558ffbc96cf0 2024-03-18T22:55:19.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.981+0000 7f67877fe640 1 --2- 172.21.15.40:0/2724342098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc96cf0 0x558ffbca4dd0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f67780c5990 tx=0x7f677802bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da557534d7baac2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:19.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.981+0000 7f67677fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:55:19.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.982+0000 7f67677fe640 1 -- 172.21.15.40:0/2724342098 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67780bde20 con 0x558ffbc96cf0 2024-03-18T22:55:19.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.982+0000 7f67677fe640 1 -- 172.21.15.40:0/2724342098 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f67780b9d00 con 0x558ffbc96cf0 2024-03-18T22:55:19.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.982+0000 7f679142d740 1 -- 172.21.15.40:0/2724342098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f677c00a370 con 0x558ffbc96cf0 2024-03-18T22:55:19.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.982+0000 7f67677fe640 1 -- 172.21.15.40:0/2724342098 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f67780bd720 con 0x558ffbc96cf0 2024-03-18T22:55:19.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.982+0000 7f679142d740 10 client.18342 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:19.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.982+0000 7f679142d740 1 -- 172.21.15.40:0/2724342098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x558ffbd11a50 con 0x558ffbc96cf0 2024-03-18T22:55:19.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.983+0000 7f67677fe640 1 -- 172.21.15.40:0/2724342098 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67780c1870 con 0x558ffbc96cf0 2024-03-18T22:55:19.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.983+0000 7f67677fe640 1 -- 172.21.15.40:0/2724342098 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f67780bd980 con 0x558ffbc96cf0 2024-03-18T22:55:19.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.983+0000 7f679142d740 10 client.18342 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:19.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.983+0000 7f679142d740 10 client.18342 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:55:19.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.983+0000 7f679142d740 20 client.18342 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:19.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.983+0000 7f679142d740 1 --2- 172.21.15.40:0/2724342098 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x558ffbd123e0 0x558ffbd14800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:19.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.983+0000 7f679142d740 4 client.18342 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:55:19.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.983+0000 7f679142d740 1 -- 172.21.15.40:0/2724342098 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f677c00a370 con 0x558ffbd123e0 2024-03-18T22:55:19.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.984+0000 7f6787fff640 1 --2- 172.21.15.40:0/2724342098 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x558ffbd123e0 0x558ffbd14800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.995+0000 7f6787fff640 1 --2- 172.21.15.40:0/2724342098 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x558ffbd123e0 0x558ffbd14800 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4e5c9209057e2894 server_cookie=e92786329f1c637 in_seq=0 out_seq=0 2024-03-18T22:55:19.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.995+0000 7f67677fe640 10 client.18342 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:55:19.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.998+0000 7f67677fe640 1 -- 172.21.15.40:0/2724342098 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f677c00a370 con 0x558ffbd123e0 2024-03-18T22:55:19.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:19.998+0000 7f67677fe640 10 client.18342 handle_command_reply: tid=0 2024-03-18T22:55:20.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.001+0000 7f679142d740 1 client.18342 shutdown 2024-03-18T22:55:20.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.005+0000 7f679142d740 1 -- 172.21.15.40:0/2724342098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc96cf0 msgr2=0x558ffbca4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.005+0000 7f679142d740 1 --2- 172.21.15.40:0/2724342098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc96cf0 0x558ffbca4dd0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f67780c5990 tx=0x7f677802bc80 comp rx=0 tx=0).stop 2024-03-18T22:55:20.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.006+0000 7f679142d740 1 -- 172.21.15.40:0/2724342098 shutdown_connections 2024-03-18T22:55:20.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.006+0000 7f679142d740 1 --2- 172.21.15.40:0/2724342098 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x558ffbd123e0 0x558ffbd14800 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.006+0000 7f679142d740 1 --2- 172.21.15.40:0/2724342098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558ffbc9be60 0x558ffbd10500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.006+0000 7f679142d740 1 --2- 172.21.15.40:0/2724342098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558ffbc96cf0 0x558ffbca4dd0 secure :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f67780c5990 tx=0x7f677802bc80 comp rx=0 tx=0).stop 2024-03-18T22:55:20.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.006+0000 7f679142d740 1 --2- 172.21.15.40:0/2724342098 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x558ffbc96300 0x558ffbca4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.006+0000 7f679142d740 1 -- 172.21.15.40:0/2724342098 >> 172.21.15.40:0/2724342098 conn(0x558ffbc80860 msgr2=0x558ffbc84350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.006+0000 7f67677fe640 0 client.18342 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:55:20.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.008+0000 7f679142d740 1 -- 172.21.15.40:0/2724342098 shutdown_connections 2024-03-18T22:55:20.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.011+0000 7f679142d740 1 -- 172.21.15.40:0/2724342098 wait complete. 2024-03-18T22:55:20.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.014+0000 7f679142d740 20 client.18342 trim_cache size 0 max 16384 2024-03-18T22:55:20.015 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:55:20.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.020+0000 7f678e844640 1 -- 172.21.15.40:0/3848864855 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f67700a2a00 msgr2=0x7f67700a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.020+0000 7f678e844640 1 --2- 172.21.15.40:0/3848864855 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f67700a2a00 0x7f67700a4ec0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x558ffbc97550 tx=0x7f6780004ad0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.020+0000 7f678e844640 1 -- 172.21.15.40:0/3848864855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6788102a00 msgr2=0x7f678819e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.020+0000 7f678e844640 1 --2- 172.21.15.40:0/3848864855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6788102a00 0x7f678819e740 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6778000c00 tx=0x7f6778004270 comp rx=0 tx=0).stop 2024-03-18T22:55:20.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.022+0000 7f678e844640 1 -- 172.21.15.40:0/3848864855 shutdown_connections 2024-03-18T22:55:20.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.022+0000 7f678e844640 1 --2- 172.21.15.40:0/3848864855 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f67700a2a00 0x7f67700a4ec0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.022+0000 7f678e844640 1 --2- 172.21.15.40:0/3848864855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67881053b0 0x7f67881a57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.022+0000 7f678e844640 1 --2- 172.21.15.40:0/3848864855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6788102a00 0x7f678819e740 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.022+0000 7f678e844640 1 --2- 172.21.15.40:0/3848864855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6788100090 0x7f678819e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.022+0000 7f678e844640 1 -- 172.21.15.40:0/3848864855 >> 172.21.15.40:0/3848864855 conn(0x7f67880fbc90 msgr2=0x7f6788100ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.023+0000 7f678e844640 1 -- 172.21.15.40:0/3848864855 shutdown_connections 2024-03-18T22:55:20.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.023+0000 7f678e844640 1 -- 172.21.15.40:0/3848864855 wait complete. 2024-03-18T22:55:20.055 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:55:20.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.279+0000 7ff1f3374640 1 -- 172.21.15.40:0/1673109005 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1ec100c40 msgr2=0x7ff1ec1010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.279+0000 7ff1f3374640 1 --2- 172.21.15.40:0/1673109005 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1ec100c40 0x7ff1ec1010c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc009860 tx=0x7ff1dc02bb70 comp rx=0 tx=0).stop 2024-03-18T22:55:20.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.279+0000 7ff1f3374640 1 -- 172.21.15.40:0/1673109005 shutdown_connections 2024-03-18T22:55:20.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.279+0000 7ff1f3374640 1 --2- 172.21.15.40:0/1673109005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ec101600 0x7ff1ec107e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.279+0000 7ff1f3374640 1 --2- 172.21.15.40:0/1673109005 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1ec100c40 0x7ff1ec1010c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.279+0000 7ff1f3374640 1 --2- 172.21.15.40:0/1673109005 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1ec0ffa50 0x7ff1ec0ffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.279+0000 7ff1f3374640 1 -- 172.21.15.40:0/1673109005 >> 172.21.15.40:0/1673109005 conn(0x7ff1ec0fb000 msgr2=0x7ff1ec0fd420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.279+0000 7ff1f3374640 1 -- 172.21.15.40:0/1673109005 shutdown_connections 2024-03-18T22:55:20.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.280+0000 7ff1f3374640 1 -- 172.21.15.40:0/1673109005 wait complete. 2024-03-18T22:55:20.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.280+0000 7ff1f3374640 1 Processor -- start 2024-03-18T22:55:20.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.280+0000 7ff1f3374640 1 -- start start 2024-03-18T22:55:20.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.281+0000 7ff1f3374640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1ec0ffa50 0x7ff1ec199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.281+0000 7ff1f3374640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1ec100c40 0x7ff1ec19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.281+0000 7ff1f3374640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ec101600 0x7ff1ec1a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.281+0000 7ff1f3374640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec10b150 con 0x7ff1ec101600 2024-03-18T22:55:20.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.281+0000 7ff1f3374640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec10afd0 con 0x7ff1ec0ffa50 2024-03-18T22:55:20.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.281+0000 7ff1f3374640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec10b2d0 con 0x7ff1ec100c40 2024-03-18T22:55:20.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.281+0000 7ff1f1110640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1ec0ffa50 0x7ff1ec199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.283+0000 7ff1f090f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1ec100c40 0x7ff1ec19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.284+0000 7ff1f090f640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1ec100c40 0x7ff1ec19a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:37788/0 (socket says 172.21.15.40:37788) 2024-03-18T22:55:20.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.284+0000 7ff1f090f640 1 -- 172.21.15.40:0/2174133599 learned_addr learned my addr 172.21.15.40:0/2174133599 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:20.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.284+0000 7ff1f090f640 1 -- 172.21.15.40:0/2174133599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1ec0ffa50 msgr2=0x7ff1ec199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.284+0000 7ff1f090f640 1 --2- 172.21.15.40:0/2174133599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1ec0ffa50 0x7ff1ec199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.284+0000 7ff1f090f640 1 -- 172.21.15.40:0/2174133599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ec101600 msgr2=0x7ff1ec1a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.284+0000 7ff1f090f640 1 --2- 172.21.15.40:0/2174133599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ec101600 0x7ff1ec1a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.284+0000 7ff1f090f640 1 -- 172.21.15.40:0/2174133599 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ec190830 con 0x7ff1ec100c40 2024-03-18T22:55:20.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.285+0000 7ff1f090f640 1 --2- 172.21.15.40:0/2174133599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1ec100c40 0x7ff1ec19a1b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0057c0 tx=0x7ff1dc0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.286+0000 7ff1e27fc640 1 -- 172.21.15.40:0/2174133599 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc041030 con 0x7ff1ec100c40 2024-03-18T22:55:20.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.287+0000 7ff1e27fc640 1 -- 172.21.15.40:0/2174133599 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1dc02c9c0 con 0x7ff1ec100c40 2024-03-18T22:55:20.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.287+0000 7ff1f3374640 1 -- 172.21.15.40:0/2174133599 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ec190ac0 con 0x7ff1ec100c40 2024-03-18T22:55:20.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.287+0000 7ff1e27fc640 1 -- 172.21.15.40:0/2174133599 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc039040 con 0x7ff1ec100c40 2024-03-18T22:55:20.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.287+0000 7ff1f3374640 1 -- 172.21.15.40:0/2174133599 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1ec190fb0 con 0x7ff1ec100c40 2024-03-18T22:55:20.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.291+0000 7ff1f6036740 1 Processor -- start 2024-03-18T22:55:20.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.291+0000 7ff1f6036740 1 -- start start 2024-03-18T22:55:20.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.292+0000 7ff1f6036740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514e02f0 0x556b514e0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.292+0000 7ff1f6036740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b514e0ce0 0x556b514e57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.292+0000 7ff1f6036740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e5e50 0x556b514e82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.292+0000 7ff1f6036740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556b514444b0 con 0x556b514e5e50 2024-03-18T22:55:20.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.292+0000 7ff1f6036740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556b513ecd70 con 0x556b514e02f0 2024-03-18T22:55:20.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.292+0000 7ff1f6036740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556b51440550 con 0x556b514e0ce0 2024-03-18T22:55:20.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.292+0000 7ff1f1911640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514e02f0 0x556b514e0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.292+0000 7ff1f1110640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b514e0ce0 0x556b514e57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.292+0000 7ff1f1911640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514e02f0 0x556b514e0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46974/0 (socket says 172.21.15.40:46974) 2024-03-18T22:55:20.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.292+0000 7ff1f1911640 1 -- 172.21.15.40:0/281898016 learned_addr learned my addr 172.21.15.40:0/281898016 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:20.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.292+0000 7ff1f1110640 1 -- 172.21.15.40:0/281898016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514e02f0 msgr2=0x556b514e0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.292+0000 7ff1f1110640 1 --2- 172.21.15.40:0/281898016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514e02f0 0x556b514e0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.292+0000 7ff1f1110640 1 -- 172.21.15.40:0/281898016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e5e50 msgr2=0x556b514e82e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:55:20.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.292+0000 7ff1f1110640 1 --2- 172.21.15.40:0/281898016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e5e50 0x556b514e82e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.292+0000 7ff1f1110640 1 -- 172.21.15.40:0/281898016 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b5140f210 con 0x556b514e0ce0 2024-03-18T22:55:20.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.293+0000 7ff1e27fc640 1 -- 172.21.15.40:0/2174133599 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff1dc044070 con 0x7ff1ec100c40 2024-03-18T22:55:20.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.294+0000 7ff1e27fc640 1 --2- 172.21.15.40:0/2174133599 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1d80a29d0 0x7ff1d80a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.295+0000 7ff1f1110640 1 --2- 172.21.15.40:0/281898016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b514e0ce0 0x556b514e57e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec100aa0 tx=0x7ff1e8010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7e48a907afab8ad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.295+0000 7ff1e27fc640 1 -- 172.21.15.40:0/2174133599 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff1dc0f0320 con 0x7ff1ec100c40 2024-03-18T22:55:20.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.296+0000 7ff1f1911640 1 --2- 172.21.15.40:0/2174133599 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1d80a29d0 0x7ff1d80a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.296+0000 7ff1d77fe640 1 -- 172.21.15.40:0/281898016 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8009e40 con 0x556b514e0ce0 2024-03-18T22:55:20.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.296+0000 7ff1d77fe640 1 -- 172.21.15.40:0/281898016 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1e801b930 con 0x556b514e0ce0 2024-03-18T22:55:20.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.296+0000 7ff1d77fe640 1 -- 172.21.15.40:0/281898016 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e80129e0 con 0x556b514e0ce0 2024-03-18T22:55:20.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.298+0000 7ff1f6036740 1 -- 172.21.15.40:0/281898016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b514e0ce0 msgr2=0x556b514e57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.298+0000 7ff1f6036740 1 --2- 172.21.15.40:0/281898016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b514e0ce0 0x556b514e57e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec100aa0 tx=0x7ff1e8010740 comp rx=0 tx=0).stop 2024-03-18T22:55:20.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.298+0000 7ff1f6036740 1 -- 172.21.15.40:0/281898016 shutdown_connections 2024-03-18T22:55:20.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.298+0000 7ff1f6036740 1 --2- 172.21.15.40:0/281898016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e5e50 0x556b514e82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.298+0000 7ff1f6036740 1 --2- 172.21.15.40:0/281898016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b514e0ce0 0x556b514e57e0 secure :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec100aa0 tx=0x7ff1e8010740 comp rx=0 tx=0).stop 2024-03-18T22:55:20.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.298+0000 7ff1f6036740 1 --2- 172.21.15.40:0/281898016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514e02f0 0x556b514e0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.298+0000 7ff1f6036740 1 -- 172.21.15.40:0/281898016 >> 172.21.15.40:0/281898016 conn(0x556b514db860 msgr2=0x556b514dbc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.298+0000 7ff1f1911640 1 --2- 172.21.15.40:0/2174133599 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1d80a29d0 0x7ff1d80a4e90 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x556b514e1540 tx=0x7ff1e400c040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.300+0000 7ff1f6036740 1 -- 172.21.15.40:0/281898016 shutdown_connections 2024-03-18T22:55:20.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.300+0000 7ff1f6036740 1 -- 172.21.15.40:0/281898016 wait complete. 2024-03-18T22:55:20.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.301+0000 7ff1f6036740 1 -- start start 2024-03-18T22:55:20.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.301+0000 7ff1f6036740 1 Processor -- start 2024-03-18T22:55:20.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.302+0000 7ff1f6036740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514e02f0 0x556b514fba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.302+0000 7ff1f6036740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e5e50 0x556b514fbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.302+0000 7ff1f6036740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b51500ff0 0x556b515033e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.302+0000 7ff1f1110640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514e02f0 0x556b514fba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.302+0000 7ff1f1110640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514e02f0 0x556b514fba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46978/0 (socket says 172.21.15.40:46978) 2024-03-18T22:55:20.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.303+0000 7ff1f1110640 1 -- 172.21.15.40:0/3542848542 learned_addr learned my addr 172.21.15.40:0/3542848542 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:20.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.303+0000 7ff1f6036740 1 -- 172.21.15.40:0/3542848542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x556b514f9e10 con 0x556b514e5e50 2024-03-18T22:55:20.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.303+0000 7ff1f6036740 1 -- 172.21.15.40:0/3542848542 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556b51400310 con 0x556b514e02f0 2024-03-18T22:55:20.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.303+0000 7ff1f6036740 1 -- 172.21.15.40:0/3542848542 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x556b514f9f90 con 0x556b51500ff0 2024-03-18T22:55:20.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.303+0000 7ff1f6036740 10 client.0 _ll_register_callbacks cb 0x556b513f42d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:20.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.304+0000 7ff1f090f640 1 --2- 172.21.15.40:0/3542848542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e5e50 0x556b514fbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.305+0000 7ff1f090f640 1 -- 172.21.15.40:0/3542848542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b51500ff0 msgr2=0x556b515033e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.305+0000 7ff1f090f640 1 --2- 172.21.15.40:0/3542848542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b51500ff0 0x556b515033e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.305+0000 7ff1f090f640 1 -- 172.21.15.40:0/3542848542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514e02f0 msgr2=0x556b514fba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.305+0000 7ff1f090f640 1 --2- 172.21.15.40:0/3542848542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514e02f0 0x556b514fba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.305+0000 7ff1f090f640 1 -- 172.21.15.40:0/3542848542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b511583f0 con 0x556b514e5e50 2024-03-18T22:55:20.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.307+0000 7ff1f090f640 1 --2- 172.21.15.40:0/3542848542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e5e50 0x556b514fbfc0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0ed9d0 tx=0x7ff1dc0b8bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89a13c794add73 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.307+0000 7ff1d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:55:20.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.307+0000 7ff1d4ff9640 1 -- 172.21.15.40:0/3542848542 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc0d5060 con 0x556b514e5e50 2024-03-18T22:55:20.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.307+0000 7ff1f6036740 1 -- 172.21.15.40:0/3542848542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1e801b930 con 0x556b514e5e50 2024-03-18T22:55:20.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.307+0000 7ff1d4ff9640 1 -- 172.21.15.40:0/3542848542 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff1dc0c4b80 con 0x556b514e5e50 2024-03-18T22:55:20.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.308+0000 7ff1d4ff9640 1 -- 172.21.15.40:0/3542848542 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7ff1dc004460 con 0x556b514e5e50 2024-03-18T22:55:20.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.308+0000 7ff1f6036740 10 client.18348 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:20.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.308+0000 7ff1f6036740 1 -- 172.21.15.40:0/3542848542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x556b512227d0 con 0x556b514e5e50 2024-03-18T22:55:20.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.308+0000 7ff1d4ff9640 1 -- 172.21.15.40:0/3542848542 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1e801b930 con 0x556b514e5e50 2024-03-18T22:55:20.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.309+0000 7ff1d4ff9640 1 -- 172.21.15.40:0/3542848542 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7ff1dc04e060 con 0x556b514e5e50 2024-03-18T22:55:20.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.309+0000 7ff1f6036740 10 client.18348 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:20.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.309+0000 7ff1f6036740 10 client.18348 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:55:20.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.309+0000 7ff1f6036740 20 client.18348 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:20.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.309+0000 7ff1f6036740 1 --2- 172.21.15.40:0/3542848542 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556b51507c40 0x556b5150a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.309+0000 7ff1f6036740 4 client.18348 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:55:20.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.309+0000 7ff1f6036740 1 -- 172.21.15.40:0/3542848542 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b514a23e0 con 0x556b51507c40 2024-03-18T22:55:20.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.309+0000 7ff1f1110640 1 --2- 172.21.15.40:0/3542848542 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556b51507c40 0x556b5150a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.310+0000 7ff1f1110640 1 --2- 172.21.15.40:0/3542848542 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556b51507c40 0x556b5150a0b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1cd91145ea962f7a server_cookie=d46b09d5acfc8297 in_seq=0 out_seq=0 2024-03-18T22:55:20.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.310+0000 7ff1d4ff9640 10 client.18348 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:55:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.311+0000 7ff1d4ff9640 1 -- 172.21.15.40:0/3542848542 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556b514a23e0 con 0x556b51507c40 2024-03-18T22:55:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.312+0000 7ff1d4ff9640 10 client.18348 handle_command_reply: tid=0 2024-03-18T22:55:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.312+0000 7ff1f6036740 1 client.18348 shutdown 2024-03-18T22:55:20.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.312+0000 7ff1f6036740 1 -- 172.21.15.40:0/3542848542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e5e50 msgr2=0x556b514fbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.312+0000 7ff1f6036740 1 --2- 172.21.15.40:0/3542848542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e5e50 0x556b514fbfc0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0ed9d0 tx=0x7ff1dc0b8bd0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.312+0000 7ff1f6036740 1 -- 172.21.15.40:0/3542848542 shutdown_connections 2024-03-18T22:55:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.312+0000 7ff1f6036740 1 --2- 172.21.15.40:0/3542848542 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556b51507c40 0x556b5150a0b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.312+0000 7ff1f6036740 1 --2- 172.21.15.40:0/3542848542 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b51500ff0 0x556b515033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.312+0000 7ff1f6036740 1 --2- 172.21.15.40:0/3542848542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e5e50 0x556b514fbfc0 secure :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0ed9d0 tx=0x7ff1dc0b8bd0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.312+0000 7ff1f6036740 1 --2- 172.21.15.40:0/3542848542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514e02f0 0x556b514fba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.312+0000 7ff1f6036740 1 -- 172.21.15.40:0/3542848542 >> 172.21.15.40:0/3542848542 conn(0x556b514db860 msgr2=0x556b514daba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.312+0000 7ff1d4ff9640 0 client.18348 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:55:20.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.314+0000 7ff1f6036740 1 -- 172.21.15.40:0/3542848542 shutdown_connections 2024-03-18T22:55:20.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.315+0000 7ff1f6036740 1 -- 172.21.15.40:0/3542848542 wait complete. 2024-03-18T22:55:20.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.315+0000 7ff1f6036740 20 client.18348 trim_cache size 0 max 16384 2024-03-18T22:55:20.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.352+0000 7ff1f6036740 1 Processor -- start 2024-03-18T22:55:20.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.352+0000 7ff1f6036740 1 -- start start 2024-03-18T22:55:20.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.353+0000 7ff1f6036740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b514e02f0 0x556b514e2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.353+0000 7ff1f6036740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b51500ff0 0x556b514e28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.353+0000 7ff1f6036740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514da310 0x556b514da770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.353+0000 7ff1f6036740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d000fdd0 con 0x556b51500ff0 2024-03-18T22:55:20.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.353+0000 7ff1f6036740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d0011910 con 0x556b514da310 2024-03-18T22:55:20.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.353+0000 7ff1f6036740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d00104a0 con 0x556b514e02f0 2024-03-18T22:55:20.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.353+0000 7ff1f1911640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514da310 0x556b514da770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.353+0000 7ff1f1911640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514da310 0x556b514da770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46990/0 (socket says 172.21.15.40:46990) 2024-03-18T22:55:20.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.353+0000 7ff1f1911640 1 -- 172.21.15.40:0/2193163055 learned_addr learned my addr 172.21.15.40:0/2193163055 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:20.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.354+0000 7ff1f1110640 1 --2- 172.21.15.40:0/2193163055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b514e02f0 0x556b514e2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.354+0000 7ff1f090f640 1 --2- 172.21.15.40:0/2193163055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b51500ff0 0x556b514e28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.355+0000 7ff1f1110640 1 -- 172.21.15.40:0/2193163055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514da310 msgr2=0x556b514da770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.355+0000 7ff1f1110640 1 --2- 172.21.15.40:0/2193163055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514da310 0x556b514da770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.356+0000 7ff1f1110640 1 -- 172.21.15.40:0/2193163055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b51500ff0 msgr2=0x556b514e28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.356+0000 7ff1f1110640 1 --2- 172.21.15.40:0/2193163055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b51500ff0 0x556b514e28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.356+0000 7ff1f1110640 1 -- 172.21.15.40:0/2193163055 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b514e10c0 con 0x556b514e02f0 2024-03-18T22:55:20.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.358+0000 7ff1f1110640 1 --2- 172.21.15.40:0/2193163055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b514e02f0 0x556b514e2390 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x556b5150a900 tx=0x7ff1e8009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6438f50e1e0faa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.360+0000 7ff1d6ffd640 1 -- 172.21.15.40:0/2193163055 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8005240 con 0x556b514e02f0 2024-03-18T22:55:20.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.360+0000 7ff1d6ffd640 1 -- 172.21.15.40:0/2193163055 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1e8029660 con 0x556b514e02f0 2024-03-18T22:55:20.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.360+0000 7ff1d6ffd640 1 -- 172.21.15.40:0/2193163055 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8004660 con 0x556b514e02f0 2024-03-18T22:55:20.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.361+0000 7ff1f6036740 1 -- 172.21.15.40:0/2193163055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b514e02f0 msgr2=0x556b514e2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.361+0000 7ff1f6036740 1 --2- 172.21.15.40:0/2193163055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b514e02f0 0x556b514e2390 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x556b5150a900 tx=0x7ff1e8009f70 comp rx=0 tx=0).stop 2024-03-18T22:55:20.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.362+0000 7ff1f6036740 1 -- 172.21.15.40:0/2193163055 shutdown_connections 2024-03-18T22:55:20.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.363+0000 7ff1f6036740 1 --2- 172.21.15.40:0/2193163055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514da310 0x556b514da770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.363+0000 7ff1f6036740 1 --2- 172.21.15.40:0/2193163055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b51500ff0 0x556b514e28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.363+0000 7ff1f6036740 1 --2- 172.21.15.40:0/2193163055 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b514e02f0 0x556b514e2390 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.363+0000 7ff1f6036740 1 -- 172.21.15.40:0/2193163055 >> 172.21.15.40:0/2193163055 conn(0x556b514db860 msgr2=0x556b514e17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.363+0000 7ff1f6036740 1 -- 172.21.15.40:0/2193163055 shutdown_connections 2024-03-18T22:55:20.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.363+0000 7ff1f6036740 1 -- 172.21.15.40:0/2193163055 wait complete. 2024-03-18T22:55:20.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.364+0000 7ff1f6036740 1 -- start start 2024-03-18T22:55:20.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.364+0000 7ff1f6036740 1 Processor -- start 2024-03-18T22:55:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f6036740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514da310 0x556b51518c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f6036740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e02f0 0x556b51519190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f6036740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b51500ff0 0x556b5151a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f6036740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d0011240 con 0x556b514e02f0 2024-03-18T22:55:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f6036740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d0010b70 con 0x556b514da310 2024-03-18T22:55:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f6036740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d000ddb0 con 0x556b51500ff0 2024-03-18T22:55:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f6036740 10 client.0 _ll_register_callbacks cb 0x7ff1dc009490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f090f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e02f0 0x556b51519190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f090f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e02f0 0x556b51519190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35358/0 (socket says 172.21.15.40:35358) 2024-03-18T22:55:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f090f640 1 -- 172.21.15.40:0/619136153 learned_addr learned my addr 172.21.15.40:0/619136153 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f1110640 1 --2- 172.21.15.40:0/619136153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514da310 0x556b51518c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f1911640 1 --2- 172.21.15.40:0/619136153 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b51500ff0 0x556b5151a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f090f640 1 -- 172.21.15.40:0/619136153 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b51500ff0 msgr2=0x556b5151a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f090f640 1 --2- 172.21.15.40:0/619136153 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b51500ff0 0x556b5151a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f090f640 1 -- 172.21.15.40:0/619136153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514da310 msgr2=0x556b51518c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f090f640 1 --2- 172.21.15.40:0/619136153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514da310 0x556b51518c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.365+0000 7ff1f090f640 1 -- 172.21.15.40:0/619136153 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b5151a790 con 0x556b514e02f0 2024-03-18T22:55:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.366+0000 7ff1f1110640 1 --2- 172.21.15.40:0/619136153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514da310 0x556b51518c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:55:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.366+0000 7ff1f1911640 1 --2- 172.21.15.40:0/619136153 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b51500ff0 0x556b5151a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:55:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.366+0000 7ff1f090f640 1 --2- 172.21.15.40:0/619136153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e02f0 0x556b51519190 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0c4ae0 tx=0x7ff1dc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc80289e120cbe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.366+0000 7ff1d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:55:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.366+0000 7ff1d4ff9640 1 -- 172.21.15.40:0/619136153 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc0bc870 con 0x556b514e02f0 2024-03-18T22:55:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.366+0000 7ff1d4ff9640 1 -- 172.21.15.40:0/619136153 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff1dc0bdba0 con 0x556b514e02f0 2024-03-18T22:55:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.366+0000 7ff1f6036740 1 -- 172.21.15.40:0/619136153 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1e8029660 con 0x556b514e02f0 2024-03-18T22:55:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.366+0000 7ff1d4ff9640 1 -- 172.21.15.40:0/619136153 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7ff1dc0bcf70 con 0x556b514e02f0 2024-03-18T22:55:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.366+0000 7ff1f6036740 10 client.18354 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:20.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.366+0000 7ff1f6036740 1 -- 172.21.15.40:0/619136153 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x556b5156dd30 con 0x556b514e02f0 2024-03-18T22:55:20.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.367+0000 7ff1d4ff9640 1 -- 172.21.15.40:0/619136153 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1e8029660 con 0x556b514e02f0 2024-03-18T22:55:20.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.368+0000 7ff1d4ff9640 1 -- 172.21.15.40:0/619136153 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7ff1dc03c030 con 0x556b514e02f0 2024-03-18T22:55:20.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.368+0000 7ff1f6036740 10 client.18354 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:20.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.368+0000 7ff1f6036740 10 client.18354 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:55:20.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.368+0000 7ff1f6036740 20 client.18354 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:20.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.368+0000 7ff1f6036740 1 --2- 172.21.15.40:0/619136153 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556b5156e320 0x556b51570740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.368+0000 7ff1f6036740 4 client.18354 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:55:20.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.368+0000 7ff1f6036740 1 -- 172.21.15.40:0/619136153 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556b514519f0 con 0x556b5156e320 2024-03-18T22:55:20.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.370+0000 7ff1f1110640 1 --2- 172.21.15.40:0/619136153 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556b5156e320 0x556b51570740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.376+0000 7ff1f1110640 1 --2- 172.21.15.40:0/619136153 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556b5156e320 0x556b51570740 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8c34e2b797525b7 server_cookie=f40a41064a40ebd7 in_seq=0 out_seq=0 2024-03-18T22:55:20.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.380+0000 7ff1d4ff9640 10 client.18354 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:55:20.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.380+0000 7ff1d4ff9640 1 -- 172.21.15.40:0/619136153 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556b514519f0 con 0x556b5156e320 2024-03-18T22:55:20.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.380+0000 7ff1d4ff9640 10 client.18354 handle_command_reply: tid=0 2024-03-18T22:55:20.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.380+0000 7ff1f6036740 1 client.18354 shutdown 2024-03-18T22:55:20.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.386+0000 7ff1f6036740 1 -- 172.21.15.40:0/619136153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e02f0 msgr2=0x556b51519190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.386+0000 7ff1f6036740 1 --2- 172.21.15.40:0/619136153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e02f0 0x556b51519190 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0c4ae0 tx=0x7ff1dc005f90 comp rx=0 tx=0).stop 2024-03-18T22:55:20.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.387+0000 7ff1f6036740 1 -- 172.21.15.40:0/619136153 shutdown_connections 2024-03-18T22:55:20.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.387+0000 7ff1f6036740 1 --2- 172.21.15.40:0/619136153 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x556b5156e320 0x556b51570740 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.387+0000 7ff1f6036740 1 --2- 172.21.15.40:0/619136153 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x556b51500ff0 0x556b5151a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.387+0000 7ff1f6036740 1 --2- 172.21.15.40:0/619136153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556b514e02f0 0x556b51519190 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.387+0000 7ff1f6036740 1 --2- 172.21.15.40:0/619136153 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b514da310 0x556b51518c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.387+0000 7ff1d4ff9640 0 client.18354 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:55:20.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.387+0000 7ff1f6036740 1 -- 172.21.15.40:0/619136153 >> 172.21.15.40:0/619136153 conn(0x556b514db860 msgr2=0x556b514e2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.387+0000 7ff1f6036740 1 -- 172.21.15.40:0/619136153 shutdown_connections 2024-03-18T22:55:20.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.387+0000 7ff1f6036740 1 -- 172.21.15.40:0/619136153 wait complete. 2024-03-18T22:55:20.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.388+0000 7ff1f6036740 20 client.18354 trim_cache size 0 max 16384 2024-03-18T22:55:20.390 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:55:20.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:20 smithi040 ceph-mon[62643]: pgmap v731: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.1 MiB/s wr, 2.33k op/s 2024-03-18T22:55:20.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:20 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/4202353723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:55:20.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.393+0000 7ff1f3374640 1 -- 172.21.15.40:0/2174133599 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1d80a29d0 msgr2=0x7ff1d80a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.393+0000 7ff1f3374640 1 --2- 172.21.15.40:0/2174133599 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1d80a29d0 0x7ff1d80a4e90 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x556b514e1540 tx=0x7ff1e400c040 comp rx=0 tx=0).stop 2024-03-18T22:55:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.393+0000 7ff1f3374640 1 -- 172.21.15.40:0/2174133599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1ec100c40 msgr2=0x7ff1ec19a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.393+0000 7ff1f3374640 1 --2- 172.21.15.40:0/2174133599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1ec100c40 0x7ff1ec19a1b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0057c0 tx=0x7ff1dc0042f0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.393+0000 7ff1f3374640 1 -- 172.21.15.40:0/2174133599 shutdown_connections 2024-03-18T22:55:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.393+0000 7ff1f3374640 1 --2- 172.21.15.40:0/2174133599 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1d80a29d0 0x7ff1d80a4e90 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.393+0000 7ff1f3374640 1 --2- 172.21.15.40:0/2174133599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ec101600 0x7ff1ec1a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.393+0000 7ff1f3374640 1 --2- 172.21.15.40:0/2174133599 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1ec100c40 0x7ff1ec19a1b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.393+0000 7ff1f3374640 1 --2- 172.21.15.40:0/2174133599 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1ec0ffa50 0x7ff1ec199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.393+0000 7ff1f3374640 1 -- 172.21.15.40:0/2174133599 >> 172.21.15.40:0/2174133599 conn(0x7ff1ec0fb000 msgr2=0x7ff1ec104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.395+0000 7ff1f3374640 1 -- 172.21.15.40:0/2174133599 shutdown_connections 2024-03-18T22:55:20.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.395+0000 7ff1f3374640 1 -- 172.21.15.40:0/2174133599 wait complete. 2024-03-18T22:55:20.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:55:20.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.573+0000 7f8767b52640 1 -- 172.21.15.40:0/2653634014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8760102a90 msgr2=0x7f8760102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.573+0000 7f8767b52640 1 --2- 172.21.15.40:0/2653634014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8760102a90 0x7f8760102f10 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f8750009900 tx=0x7f875002bc30 comp rx=0 tx=0).stop 2024-03-18T22:55:20.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.573+0000 7f8767b52640 1 -- 172.21.15.40:0/2653634014 shutdown_connections 2024-03-18T22:55:20.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.573+0000 7f8767b52640 1 --2- 172.21.15.40:0/2653634014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8760103450 0x7f8760109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.573+0000 7f8767b52640 1 --2- 172.21.15.40:0/2653634014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8760102a90 0x7f8760102f10 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.573+0000 7f8767b52640 1 --2- 172.21.15.40:0/2653634014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f87601018a0 0x7f8760101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.573+0000 7f8767b52640 1 -- 172.21.15.40:0/2653634014 >> 172.21.15.40:0/2653634014 conn(0x7f87600fcdf0 msgr2=0x7f87600ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.574+0000 7f8767b52640 1 -- 172.21.15.40:0/2653634014 shutdown_connections 2024-03-18T22:55:20.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.574+0000 7f8767b52640 1 -- 172.21.15.40:0/2653634014 wait complete. 2024-03-18T22:55:20.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.575+0000 7f8767b52640 1 Processor -- start 2024-03-18T22:55:20.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.575+0000 7f8767b52640 1 -- start start 2024-03-18T22:55:20.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.575+0000 7f8767b52640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87601018a0 0x7f8760197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.575+0000 7f8767b52640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8760102a90 0x7f8760197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f87658ee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87601018a0 0x7f8760197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f87658ee640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87601018a0 0x7f8760197820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35374/0 (socket says 172.21.15.40:35374) 2024-03-18T22:55:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f87650ed640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8760102a90 0x7f8760197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f87650ed640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8760102a90 0x7f8760197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47010/0 (socket says 172.21.15.40:47010) 2024-03-18T22:55:20.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f8767b52640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8760103450 0x7f876019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f87658ee640 1 -- 172.21.15.40:0/99823975 learned_addr learned my addr 172.21.15.40:0/99823975 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:20.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f8767b52640 1 -- 172.21.15.40:0/99823975 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f876010d390 con 0x7f87601018a0 2024-03-18T22:55:20.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f8767b52640 1 -- 172.21.15.40:0/99823975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f876010d210 con 0x7f8760102a90 2024-03-18T22:55:20.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f8767b52640 1 -- 172.21.15.40:0/99823975 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f876010d510 con 0x7f8760103450 2024-03-18T22:55:20.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f87660ef640 1 --2- 172.21.15.40:0/99823975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8760103450 0x7f876019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f87650ed640 1 -- 172.21.15.40:0/99823975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8760103450 msgr2=0x7f876019edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f87650ed640 1 --2- 172.21.15.40:0/99823975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8760103450 0x7f876019edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f87650ed640 1 -- 172.21.15.40:0/99823975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87601018a0 msgr2=0x7f8760197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f87650ed640 1 --2- 172.21.15.40:0/99823975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87601018a0 0x7f8760197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f87650ed640 1 -- 172.21.15.40:0/99823975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876019f440 con 0x7f8760102a90 2024-03-18T22:55:20.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.576+0000 7f87658ee640 1 --2- 172.21.15.40:0/99823975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87601018a0 0x7f8760197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:55:20.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.577+0000 7f87650ed640 1 --2- 172.21.15.40:0/99823975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8760102a90 0x7f8760197d60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8750036040 tx=0x7f87500042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.577+0000 7f8756ffd640 1 -- 172.21.15.40:0/99823975 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8750043020 con 0x7f8760102a90 2024-03-18T22:55:20.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.577+0000 7f8756ffd640 1 -- 172.21.15.40:0/99823975 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87500054c0 con 0x7f8760102a90 2024-03-18T22:55:20.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.577+0000 7f8756ffd640 1 -- 172.21.15.40:0/99823975 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8750003d50 con 0x7f8760102a90 2024-03-18T22:55:20.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.577+0000 7f8767b52640 1 -- 172.21.15.40:0/99823975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8760073a50 con 0x7f8760102a90 2024-03-18T22:55:20.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.577+0000 7f8767b52640 1 -- 172.21.15.40:0/99823975 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8760073f20 con 0x7f8760102a90 2024-03-18T22:55:20.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:20 smithi113 ceph-mon[66775]: pgmap v731: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.1 MiB/s wr, 2.33k op/s 2024-03-18T22:55:20.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:20 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/4202353723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:55:20.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.579+0000 7f8756ffd640 1 -- 172.21.15.40:0/99823975 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8750046020 con 0x7f8760102a90 2024-03-18T22:55:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.580+0000 7f876a810740 1 Processor -- start 2024-03-18T22:55:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.580+0000 7f8756ffd640 1 --2- 172.21.15.40:0/99823975 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f874c0a2880 0x7f874c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.580+0000 7f876a810740 1 -- start start 2024-03-18T22:55:20.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.580+0000 7f8756ffd640 1 -- 172.21.15.40:0/99823975 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f87500f10d0 con 0x7f8760102a90 2024-03-18T22:55:20.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.580+0000 7f87660ef640 1 --2- 172.21.15.40:0/99823975 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f874c0a2880 0x7f874c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.580+0000 7f876a810740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.580+0000 7f876a810740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.580+0000 7f876a810740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397d4e50 0x55ee397d72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f876a810740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ee397334b0 con 0x55ee397cfce0 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f876a810740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ee396dbd70 con 0x55ee397d4e50 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f876a810740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55ee3972f550 con 0x55ee397cf2f0 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f87658ee640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f87658ee640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397cf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:37846/0 (socket says 172.21.15.40:37846) 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f87658ee640 1 -- 172.21.15.40:0/3323703413 learned_addr learned my addr 172.21.15.40:0/3323703413 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f87660ef640 1 --2- 172.21.15.40:0/3323703413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397d4e50 0x55ee397d72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f87650ed640 1 --2- 172.21.15.40:0/3323703413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f87658ee640 1 -- 172.21.15.40:0/3323703413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397d4e50 msgr2=0x55ee397d72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f87658ee640 1 --2- 172.21.15.40:0/3323703413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397d4e50 0x55ee397d72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f87658ee640 1 -- 172.21.15.40:0/3323703413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 msgr2=0x55ee397d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f87658ee640 1 --2- 172.21.15.40:0/3323703413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397d47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f87658ee640 1 -- 172.21.15.40:0/3323703413 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee396fe210 con 0x55ee397cf2f0 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f87650ed640 1 --2- 172.21.15.40:0/3323703413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f87660ef640 1 --2- 172.21.15.40:0/99823975 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f874c0a2880 0x7f874c0a4d40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8758005fd0 tx=0x7f8758010d80 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f87658ee640 1 --2- 172.21.15.40:0/3323703413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397cf710 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f875c0115c0 tx=0x7f875c011a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37c8326691a9c1a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f874bfff640 1 -- 172.21.15.40:0/3323703413 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c005020 con 0x55ee397cf2f0 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f874bfff640 1 -- 172.21.15.40:0/3323703413 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f875c0047a0 con 0x55ee397cf2f0 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.581+0000 7f874bfff640 1 -- 172.21.15.40:0/3323703413 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c00f8c0 con 0x55ee397cf2f0 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.582+0000 7f876a810740 1 -- 172.21.15.40:0/3323703413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 msgr2=0x55ee397cf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.582+0000 7f876a810740 1 --2- 172.21.15.40:0/3323703413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397cf710 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f875c0115c0 tx=0x7f875c011a80 comp rx=0 tx=0).stop 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.582+0000 7f876a810740 1 -- 172.21.15.40:0/3323703413 shutdown_connections 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.582+0000 7f876a810740 1 --2- 172.21.15.40:0/3323703413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397d4e50 0x55ee397d72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.582+0000 7f876a810740 1 --2- 172.21.15.40:0/3323703413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.582+0000 7f876a810740 1 --2- 172.21.15.40:0/3323703413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397cf710 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.582+0000 7f876a810740 1 -- 172.21.15.40:0/3323703413 >> 172.21.15.40:0/3323703413 conn(0x55ee397ca860 msgr2=0x55ee397cac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.582+0000 7f876a810740 1 -- 172.21.15.40:0/3323703413 shutdown_connections 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.582+0000 7f876a810740 1 -- 172.21.15.40:0/3323703413 wait complete. 2024-03-18T22:55:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.582+0000 7f876a810740 1 -- start start 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f876a810740 1 Processor -- start 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f876a810740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397d97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f876a810740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397cfce0 0x55ee397d9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f876a810740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397d4e50 0x55ee397e0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f876a810740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55ee397d7e00 con 0x55ee397d4e50 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f876a810740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ee396ef310 con 0x55ee397cfce0 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f876a810740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55ee397d7f80 con 0x55ee397cf2f0 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f876a810740 10 client.0 _ll_register_callbacks cb 0x55ee396e32d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f87658ee640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397d97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f87660ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397d4e50 0x55ee397e0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f87650ed640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397cfce0 0x55ee397d9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f87658ee640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397d97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:37850/0 (socket says 172.21.15.40:37850) 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f87658ee640 1 -- 172.21.15.40:0/4185312595 learned_addr learned my addr 172.21.15.40:0/4185312595 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f87658ee640 1 -- 172.21.15.40:0/4185312595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397cfce0 msgr2=0x55ee397d9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f87658ee640 1 --2- 172.21.15.40:0/4185312595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397cfce0 0x55ee397d9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f87658ee640 1 -- 172.21.15.40:0/4185312595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397d4e50 msgr2=0x55ee397e0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f87658ee640 1 --2- 172.21.15.40:0/4185312595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397d4e50 0x55ee397e0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.583+0000 7f87658ee640 1 -- 172.21.15.40:0/4185312595 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee394473f0 con 0x55ee397cf2f0 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f87658ee640 1 --2- 172.21.15.40:0/4185312595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397d97e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f875c0058c0 tx=0x7f875c005230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbf74b96212d4f20 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f87497fa640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f87497fa640 1 -- 172.21.15.40:0/4185312595 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c01d050 con 0x55ee397cf2f0 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f876a810740 1 -- 172.21.15.40:0/4185312595 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f875c0047a0 con 0x55ee397cf2f0 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f87497fa640 1 -- 172.21.15.40:0/4185312595 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f875c00f660 con 0x55ee397cf2f0 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f87497fa640 1 -- 172.21.15.40:0/4185312595 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f875c032050 con 0x55ee397cf2f0 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f876a810740 10 client.28177 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f876a810740 1 -- 172.21.15.40:0/4185312595 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ee395117d0 con 0x55ee397cf2f0 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f87497fa640 1 -- 172.21.15.40:0/4185312595 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f875c0047a0 con 0x55ee397cf2f0 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f87497fa640 1 -- 172.21.15.40:0/4185312595 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f875c0247b0 con 0x55ee397cf2f0 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f876a810740 10 client.28177 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f876a810740 10 client.28177 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f876a810740 20 client.28177 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f876a810740 1 --2- 172.21.15.40:0/4185312595 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ee397e53c0 0x55ee397e7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f876a810740 4 client.28177 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:55:20.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.584+0000 7f876a810740 1 -- 172.21.15.40:0/4185312595 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee397913e0 con 0x55ee397e53c0 2024-03-18T22:55:20.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.585+0000 7f87650ed640 1 --2- 172.21.15.40:0/4185312595 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ee397e53c0 0x55ee397e7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.585+0000 7f87497fa640 10 client.28177 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:55:20.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.585+0000 7f87650ed640 1 --2- 172.21.15.40:0/4185312595 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ee397e53c0 0x55ee397e7840 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ecdc9600d889a463 server_cookie=487b5c0393bfb475 in_seq=0 out_seq=0 2024-03-18T22:55:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.586+0000 7f87497fa640 1 -- 172.21.15.40:0/4185312595 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ee397913e0 con 0x55ee397e53c0 2024-03-18T22:55:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.586+0000 7f87497fa640 10 client.28177 handle_command_reply: tid=0 2024-03-18T22:55:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.586+0000 7f876a810740 1 client.28177 shutdown 2024-03-18T22:55:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.586+0000 7f876a810740 1 -- 172.21.15.40:0/4185312595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 msgr2=0x55ee397d97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.586+0000 7f876a810740 1 --2- 172.21.15.40:0/4185312595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397d97e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f875c0058c0 tx=0x7f875c005230 comp rx=0 tx=0).stop 2024-03-18T22:55:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.586+0000 7f876a810740 1 -- 172.21.15.40:0/4185312595 shutdown_connections 2024-03-18T22:55:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.586+0000 7f876a810740 1 --2- 172.21.15.40:0/4185312595 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ee397e53c0 0x55ee397e7840 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.586+0000 7f876a810740 1 --2- 172.21.15.40:0/4185312595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397d4e50 0x55ee397e0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.586+0000 7f87497fa640 0 client.28177 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:55:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.586+0000 7f876a810740 1 --2- 172.21.15.40:0/4185312595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397cfce0 0x55ee397d9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.586+0000 7f876a810740 1 --2- 172.21.15.40:0/4185312595 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397d97e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.586+0000 7f876a810740 1 -- 172.21.15.40:0/4185312595 >> 172.21.15.40:0/4185312595 conn(0x55ee397ca860 msgr2=0x55ee397c9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.586+0000 7f876a810740 1 -- 172.21.15.40:0/4185312595 shutdown_connections 2024-03-18T22:55:20.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.586+0000 7f876a810740 1 -- 172.21.15.40:0/4185312595 wait complete. 2024-03-18T22:55:20.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.586+0000 7f876a810740 20 client.28177 trim_cache size 0 max 16384 2024-03-18T22:55:20.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:20 smithi103 ceph-mon[66377]: pgmap v731: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.1 MiB/s wr, 2.33k op/s 2024-03-18T22:55:20.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:20 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/4202353723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:55:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.608+0000 7f876a810740 1 Processor -- start 2024-03-18T22:55:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.608+0000 7f876a810740 1 -- start start 2024-03-18T22:55:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f876a810740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397cf2f0 0x55ee397d8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f876a810740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397e0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f876a810740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397d4e50 0x55ee397d9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f876a810740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f87601a8110 con 0x55ee397cfce0 2024-03-18T22:55:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f876a810740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f87601a9c50 con 0x55ee397cf2f0 2024-03-18T22:55:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f876a810740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f87601a87e0 con 0x55ee397d4e50 2024-03-18T22:55:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f87650ed640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397e0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f87658ee640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397cf2f0 0x55ee397d8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f87658ee640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397cf2f0 0x55ee397d8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47032/0 (socket says 172.21.15.40:47032) 2024-03-18T22:55:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f87658ee640 1 -- 172.21.15.40:0/3216556373 learned_addr learned my addr 172.21.15.40:0/3216556373 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f87660ef640 1 --2- 172.21.15.40:0/3216556373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397d4e50 0x55ee397d9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f87650ed640 1 -- 172.21.15.40:0/3216556373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397d4e50 msgr2=0x55ee397d9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f87650ed640 1 --2- 172.21.15.40:0/3216556373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397d4e50 0x55ee397d9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f87650ed640 1 -- 172.21.15.40:0/3216556373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397cf2f0 msgr2=0x55ee397d8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f87650ed640 1 --2- 172.21.15.40:0/3216556373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397cf2f0 0x55ee397d8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f87650ed640 1 -- 172.21.15.40:0/3216556373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee397a0880 con 0x55ee397cfce0 2024-03-18T22:55:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f87658ee640 1 --2- 172.21.15.40:0/3216556373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397cf2f0 0x55ee397d8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:55:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.609+0000 7f87650ed640 1 --2- 172.21.15.40:0/3216556373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397e0d20 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f87500bd380 tx=0x7f87500ee6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d21aae0fe2cbe8a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.610+0000 7f874b7fe640 1 -- 172.21.15.40:0/3216556373 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87500bdd80 con 0x55ee397cfce0 2024-03-18T22:55:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.610+0000 7f874b7fe640 1 -- 172.21.15.40:0/3216556373 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87500dd040 con 0x55ee397cfce0 2024-03-18T22:55:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.610+0000 7f874b7fe640 1 -- 172.21.15.40:0/3216556373 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87500c5e20 con 0x55ee397cfce0 2024-03-18T22:55:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.610+0000 7f876a810740 1 -- 172.21.15.40:0/3216556373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 msgr2=0x55ee397e0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.610+0000 7f876a810740 1 --2- 172.21.15.40:0/3216556373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397e0d20 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f87500bd380 tx=0x7f87500ee6b0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.610+0000 7f876a810740 1 -- 172.21.15.40:0/3216556373 shutdown_connections 2024-03-18T22:55:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.610+0000 7f876a810740 1 --2- 172.21.15.40:0/3216556373 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397d4e50 0x55ee397d9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.610+0000 7f876a810740 1 --2- 172.21.15.40:0/3216556373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397e0d20 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.610+0000 7f876a810740 1 --2- 172.21.15.40:0/3216556373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397cf2f0 0x55ee397d8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.610+0000 7f876a810740 1 -- 172.21.15.40:0/3216556373 >> 172.21.15.40:0/3216556373 conn(0x55ee397ca860 msgr2=0x55ee397ce350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.610+0000 7f876a810740 1 -- 172.21.15.40:0/3216556373 shutdown_connections 2024-03-18T22:55:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.610+0000 7f876a810740 1 -- 172.21.15.40:0/3216556373 wait complete. 2024-03-18T22:55:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f876a810740 1 -- start start 2024-03-18T22:55:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f876a810740 1 Processor -- start 2024-03-18T22:55:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f876a810740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397de170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f876a810740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397de6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f876a810740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397d4e50 0x55ee397df670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f876a810740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f87601a9580 con 0x55ee397cfce0 2024-03-18T22:55:20.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f876a810740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f87601a8eb0 con 0x55ee397d4e50 2024-03-18T22:55:20.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f876a810740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f87601a60f0 con 0x55ee397cf2f0 2024-03-18T22:55:20.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f876a810740 10 client.0 _ll_register_callbacks cb 0x7f875800b940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:20.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f87650ed640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397de6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f87650ed640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397de6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35398/0 (socket says 172.21.15.40:35398) 2024-03-18T22:55:20.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f87650ed640 1 -- 172.21.15.40:0/97754622 learned_addr learned my addr 172.21.15.40:0/97754622 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:20.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f87658ee640 1 --2- 172.21.15.40:0/97754622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397de170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f87660ef640 1 --2- 172.21.15.40:0/97754622 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397d4e50 0x55ee397df670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f87650ed640 1 -- 172.21.15.40:0/97754622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 msgr2=0x55ee397de170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f87650ed640 1 --2- 172.21.15.40:0/97754622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397de170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f87650ed640 1 -- 172.21.15.40:0/97754622 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397d4e50 msgr2=0x55ee397df670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f87650ed640 1 --2- 172.21.15.40:0/97754622 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397d4e50 0x55ee397df670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.611+0000 7f87650ed640 1 -- 172.21.15.40:0/97754622 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee397dfc60 con 0x55ee397cfce0 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.612+0000 7f87650ed640 1 --2- 172.21.15.40:0/97754622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397de6b0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f875002ce10 tx=0x7f87500b92f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab94f8d55467eb68 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.612+0000 7f87497fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.612+0000 7f87497fa640 1 -- 172.21.15.40:0/97754622 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87500a1060 con 0x55ee397cfce0 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.612+0000 7f87497fa640 1 -- 172.21.15.40:0/97754622 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f87500035f0 con 0x55ee397cfce0 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.612+0000 7f876a810740 1 -- 172.21.15.40:0/97754622 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87500dd040 con 0x55ee397cfce0 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.612+0000 7f87497fa640 1 -- 172.21.15.40:0/97754622 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f87500da070 con 0x55ee397cfce0 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.612+0000 7f876a810740 10 client.18372 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.612+0000 7f876a810740 1 -- 172.21.15.40:0/97754622 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55ee397e0f60 con 0x55ee397cfce0 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.612+0000 7f87497fa640 1 -- 172.21.15.40:0/97754622 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87500dd040 con 0x55ee397cfce0 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.612+0000 7f87497fa640 1 -- 172.21.15.40:0/97754622 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f87500c5990 con 0x55ee397cfce0 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.612+0000 7f876a810740 10 client.18372 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.612+0000 7f876a810740 10 client.18372 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:55:20.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.612+0000 7f876a810740 20 client.18372 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.612+0000 7f876a810740 1 --2- 172.21.15.40:0/97754622 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ee3984bd40 0x55ee3984e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f876a810740 4 client.18372 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:55:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f876a810740 1 -- 172.21.15.40:0/97754622 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ee397409f0 con 0x55ee3984bd40 2024-03-18T22:55:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f87658ee640 1 --2- 172.21.15.40:0/97754622 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ee3984bd40 0x55ee3984e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f87497fa640 10 client.18372 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:55:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f87658ee640 1 --2- 172.21.15.40:0/97754622 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ee3984bd40 0x55ee3984e160 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6f1b801b3083db5 server_cookie=b717d94c21197cf7 in_seq=0 out_seq=0 2024-03-18T22:55:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f87497fa640 1 -- 172.21.15.40:0/97754622 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ee397409f0 con 0x55ee3984bd40 2024-03-18T22:55:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f87497fa640 10 client.18372 handle_command_reply: tid=0 2024-03-18T22:55:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f876a810740 1 client.18372 shutdown 2024-03-18T22:55:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f876a810740 1 -- 172.21.15.40:0/97754622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 msgr2=0x55ee397de6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f876a810740 1 --2- 172.21.15.40:0/97754622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397de6b0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f875002ce10 tx=0x7f87500b92f0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f876a810740 1 -- 172.21.15.40:0/97754622 shutdown_connections 2024-03-18T22:55:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f876a810740 1 --2- 172.21.15.40:0/97754622 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55ee3984bd40 0x55ee3984e160 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f876a810740 1 --2- 172.21.15.40:0/97754622 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee397d4e50 0x55ee397df670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f876a810740 1 --2- 172.21.15.40:0/97754622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ee397cfce0 0x55ee397de6b0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f876a810740 1 --2- 172.21.15.40:0/97754622 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55ee397cf2f0 0x55ee397de170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f87497fa640 0 client.18372 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:55:20.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.613+0000 7f876a810740 1 -- 172.21.15.40:0/97754622 >> 172.21.15.40:0/97754622 conn(0x55ee397ca860 msgr2=0x55ee397c9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.614+0000 7f876a810740 1 -- 172.21.15.40:0/97754622 shutdown_connections 2024-03-18T22:55:20.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.614+0000 7f876a810740 1 -- 172.21.15.40:0/97754622 wait complete. 2024-03-18T22:55:20.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.614+0000 7f876a810740 20 client.18372 trim_cache size 0 max 16384 2024-03-18T22:55:20.621 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:55:20.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.614+0000 7f8767b52640 1 -- 172.21.15.40:0/99823975 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f874c0a2880 msgr2=0x7f874c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.614+0000 7f8767b52640 1 --2- 172.21.15.40:0/99823975 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f874c0a2880 0x7f874c0a4d40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8758005fd0 tx=0x7f8758010d80 comp rx=0 tx=0).stop 2024-03-18T22:55:20.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.614+0000 7f8767b52640 1 -- 172.21.15.40:0/99823975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8760102a90 msgr2=0x7f8760197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.614+0000 7f8767b52640 1 --2- 172.21.15.40:0/99823975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8760102a90 0x7f8760197d60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8750036040 tx=0x7f87500042f0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.614+0000 7f8767b52640 1 -- 172.21.15.40:0/99823975 shutdown_connections 2024-03-18T22:55:20.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.614+0000 7f8767b52640 1 --2- 172.21.15.40:0/99823975 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f874c0a2880 0x7f874c0a4d40 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.614+0000 7f8767b52640 1 --2- 172.21.15.40:0/99823975 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8760103450 0x7f876019edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.614+0000 7f8767b52640 1 --2- 172.21.15.40:0/99823975 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8760102a90 0x7f8760197d60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.614+0000 7f8767b52640 1 --2- 172.21.15.40:0/99823975 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87601018a0 0x7f8760197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.614+0000 7f8767b52640 1 -- 172.21.15.40:0/99823975 >> 172.21.15.40:0/99823975 conn(0x7f87600fcdf0 msgr2=0x7f8760105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.614+0000 7f8767b52640 1 -- 172.21.15.40:0/99823975 shutdown_connections 2024-03-18T22:55:20.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.614+0000 7f8767b52640 1 -- 172.21.15.40:0/99823975 wait complete. 2024-03-18T22:55:20.628 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:55:20.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.761+0000 7efc3d87e640 1 -- 172.21.15.40:0/3996884092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc38100720 msgr2=0x7efc38100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.761+0000 7efc3d87e640 1 --2- 172.21.15.40:0/3996884092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc38100720 0x7efc38100b40 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7efc28009900 tx=0x7efc2802bc30 comp rx=0 tx=0).stop 2024-03-18T22:55:20.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.761+0000 7efc3d87e640 1 -- 172.21.15.40:0/3996884092 shutdown_connections 2024-03-18T22:55:20.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.761+0000 7efc3d87e640 1 --2- 172.21.15.40:0/3996884092 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc381022d0 0x7efc38108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.761+0000 7efc3d87e640 1 --2- 172.21.15.40:0/3996884092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101910 0x7efc38101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.761+0000 7efc3d87e640 1 --2- 172.21.15.40:0/3996884092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc38100720 0x7efc38100b40 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.761+0000 7efc3d87e640 1 -- 172.21.15.40:0/3996884092 >> 172.21.15.40:0/3996884092 conn(0x7efc380fbcb0 msgr2=0x7efc380fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.761+0000 7efc3d87e640 1 -- 172.21.15.40:0/3996884092 shutdown_connections 2024-03-18T22:55:20.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.761+0000 7efc3d87e640 1 -- 172.21.15.40:0/3996884092 wait complete. 2024-03-18T22:55:20.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.762+0000 7efc3d87e640 1 Processor -- start 2024-03-18T22:55:20.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.762+0000 7efc3d87e640 1 -- start start 2024-03-18T22:55:20.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.762+0000 7efc3d87e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc38100720 0x7efc38197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.762+0000 7efc3d87e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101910 0x7efc38197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.762+0000 7efc3d87e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc381022d0 0x7efc3819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.762+0000 7efc3d87e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efc3810d370 con 0x7efc381022d0 2024-03-18T22:55:20.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.762+0000 7efc3d87e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efc3810d1f0 con 0x7efc38101910 2024-03-18T22:55:20.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.762+0000 7efc3d87e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7efc3810d4f0 con 0x7efc38100720 2024-03-18T22:55:20.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.762+0000 7efc377fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc381022d0 0x7efc3819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.762+0000 7efc36ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc38100720 0x7efc38197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.762+0000 7efc2e7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101910 0x7efc38197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.762+0000 7efc377fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc381022d0 0x7efc3819efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35406/0 (socket says 172.21.15.40:35406) 2024-03-18T22:55:20.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.762+0000 7efc377fe640 1 -- 172.21.15.40:0/3914058617 learned_addr learned my addr 172.21.15.40:0/3914058617 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:20.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.762+0000 7efc36ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc38100720 0x7efc38197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:37886/0 (socket says 172.21.15.40:37886) 2024-03-18T22:55:20.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.763+0000 7efc377fe640 1 -- 172.21.15.40:0/3914058617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc38100720 msgr2=0x7efc38197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.763+0000 7efc377fe640 1 --2- 172.21.15.40:0/3914058617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc38100720 0x7efc38197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.763+0000 7efc377fe640 1 -- 172.21.15.40:0/3914058617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101910 msgr2=0x7efc38197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.763+0000 7efc377fe640 1 --2- 172.21.15.40:0/3914058617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101910 0x7efc38197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.763+0000 7efc377fe640 1 -- 172.21.15.40:0/3914058617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc3819f650 con 0x7efc381022d0 2024-03-18T22:55:20.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.763+0000 7efc36ffd640 1 --2- 172.21.15.40:0/3914058617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc38100720 0x7efc38197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:55:20.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.763+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/3914058617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101910 0x7efc38197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:55:20.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.763+0000 7efc377fe640 1 --2- 172.21.15.40:0/3914058617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc381022d0 0x7efc3819efe0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7efc3000eb10 tx=0x7efc3000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.763+0000 7efc3c87c640 1 -- 172.21.15.40:0/3914058617 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc30004b20 con 0x7efc381022d0 2024-03-18T22:55:20.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.763+0000 7efc3c87c640 1 -- 172.21.15.40:0/3914058617 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc30004cc0 con 0x7efc381022d0 2024-03-18T22:55:20.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.763+0000 7efc3d87e640 1 -- 172.21.15.40:0/3914058617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc3819f940 con 0x7efc381022d0 2024-03-18T22:55:20.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.763+0000 7efc3c87c640 1 -- 172.21.15.40:0/3914058617 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc30010690 con 0x7efc381022d0 2024-03-18T22:55:20.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.763+0000 7efc3d87e640 1 -- 172.21.15.40:0/3914058617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc3819fee0 con 0x7efc381022d0 2024-03-18T22:55:20.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.765+0000 7efc3c87c640 1 -- 172.21.15.40:0/3914058617 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7efc30005020 con 0x7efc381022d0 2024-03-18T22:55:20.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.765+0000 7efc40466740 1 Processor -- start 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.765+0000 7efc40466740 1 -- start start 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc40466740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf6d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc40466740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf727e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc40466740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf72e50 0x55cbdcf752e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc40466740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cbdced14b0 con 0x55cbdcf6dce0 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc40466740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cbdce79d70 con 0x55cbdcf6d2f0 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc40466740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55cbdcecd550 con 0x55cbdcf72e50 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc3c87c640 1 --2- 172.21.15.40:0/3914058617 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efc240a2880 0x7efc240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc36ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc2e7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf6d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc2e7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf6d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47060/0 (socket says 172.21.15.40:47060) 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc2e7fc640 1 -- 172.21.15.40:0/602574160 learned_addr learned my addr 172.21.15.40:0/602574160 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/602574160 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf72e50 0x55cbdcf752e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc36ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf727e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35412/0 (socket says 172.21.15.40:35412) 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc377fe640 1 --2- 172.21.15.40:0/3914058617 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efc240a2880 0x7efc240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc3c87c640 1 -- 172.21.15.40:0/3914058617 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7efc300cf3e0 con 0x7efc381022d0 2024-03-18T22:55:20.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc2e7fc640 1 -- 172.21.15.40:0/602574160 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf72e50 msgr2=0x55cbdcf752e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/602574160 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf72e50 0x55cbdcf752e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc2e7fc640 1 -- 172.21.15.40:0/602574160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf6dce0 msgr2=0x55cbdcf727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/602574160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf727e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc2e7fc640 1 -- 172.21.15.40:0/602574160 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbdce9c210 con 0x55cbdcf6d2f0 2024-03-18T22:55:20.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/602574160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf6d710 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efc20011be0 tx=0x7efc20009ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17a528d0d9a11952 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc377fe640 1 --2- 172.21.15.40:0/3914058617 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efc240a2880 0x7efc240a4d40 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7efc300049d0 tx=0x7efc30090f10 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.766+0000 7efc2dffb640 1 -- 172.21.15.40:0/602574160 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2000fc50 con 0x55cbdcf6d2f0 2024-03-18T22:55:20.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.767+0000 7efc2dffb640 1 -- 172.21.15.40:0/602574160 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc2000fdf0 con 0x55cbdcf6d2f0 2024-03-18T22:55:20.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.767+0000 7efc2dffb640 1 -- 172.21.15.40:0/602574160 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2000e610 con 0x55cbdcf6d2f0 2024-03-18T22:55:20.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.767+0000 7efc40466740 1 -- 172.21.15.40:0/602574160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 msgr2=0x55cbdcf6d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.767+0000 7efc40466740 1 --2- 172.21.15.40:0/602574160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf6d710 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efc20011be0 tx=0x7efc20009ba0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.767+0000 7efc40466740 1 -- 172.21.15.40:0/602574160 shutdown_connections 2024-03-18T22:55:20.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.767+0000 7efc40466740 1 --2- 172.21.15.40:0/602574160 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf72e50 0x55cbdcf752e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.767+0000 7efc40466740 1 --2- 172.21.15.40:0/602574160 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf727e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.767+0000 7efc40466740 1 --2- 172.21.15.40:0/602574160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf6d710 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.767+0000 7efc40466740 1 -- 172.21.15.40:0/602574160 >> 172.21.15.40:0/602574160 conn(0x55cbdcf68860 msgr2=0x55cbdcf68c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.767+0000 7efc40466740 1 -- 172.21.15.40:0/602574160 shutdown_connections 2024-03-18T22:55:20.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.767+0000 7efc40466740 1 -- 172.21.15.40:0/602574160 wait complete. 2024-03-18T22:55:20.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.767+0000 7efc40466740 1 -- start start 2024-03-18T22:55:20.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc40466740 1 Processor -- start 2024-03-18T22:55:20.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc40466740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf777d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc40466740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf77d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc40466740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf72e50 0x55cbdcf7ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc40466740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55cbdcf75e00 con 0x55cbdcf72e50 2024-03-18T22:55:20.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc40466740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cbdce8d310 con 0x55cbdcf6d2f0 2024-03-18T22:55:20.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc40466740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55cbdcf75f80 con 0x55cbdcf6dce0 2024-03-18T22:55:20.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc40466740 10 client.0 _ll_register_callbacks cb 0x55cbdce812d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:20.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc36ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf77d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc2e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf72e50 0x55cbdcf7ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc36ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf77d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:37900/0 (socket says 172.21.15.40:37900) 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc2e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf72e50 0x55cbdcf7ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35424/0 (socket says 172.21.15.40:35424) 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc36ffd640 1 -- 172.21.15.40:0/1044506726 learned_addr learned my addr 172.21.15.40:0/1044506726 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/1044506726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf777d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc36ffd640 1 -- 172.21.15.40:0/1044506726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 msgr2=0x55cbdcf777d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc36ffd640 1 --2- 172.21.15.40:0/1044506726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf777d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc36ffd640 1 -- 172.21.15.40:0/1044506726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf72e50 msgr2=0x55cbdcf7ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc36ffd640 1 --2- 172.21.15.40:0/1044506726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf72e50 0x55cbdcf7ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.768+0000 7efc36ffd640 1 -- 172.21.15.40:0/1044506726 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbdcbe53f0 con 0x55cbdcf6dce0 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc36ffd640 1 --2- 172.21.15.40:0/1044506726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf77d10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efc2802c840 tx=0x7efc28005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28c810bc2a65b779 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc16ffd640 1 -- 172.21.15.40:0/1044506726 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc28005d50 con 0x55cbdcf6dce0 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc16ffd640 1 -- 172.21.15.40:0/1044506726 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7efc280046c0 con 0x55cbdcf6dce0 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc40466740 1 -- 172.21.15.40:0/1044506726 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc2000fdf0 con 0x55cbdcf6dce0 2024-03-18T22:55:20.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc16ffd640 1 -- 172.21.15.40:0/1044506726 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7efc28053050 con 0x55cbdcf6dce0 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc40466740 10 client.28195 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc40466740 1 -- 172.21.15.40:0/1044506726 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55cbdccaf7d0 con 0x55cbdcf6dce0 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc16ffd640 1 -- 172.21.15.40:0/1044506726 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc2000fdf0 con 0x55cbdcf6dce0 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc16ffd640 1 -- 172.21.15.40:0/1044506726 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7efc28003820 con 0x55cbdcf6dce0 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc40466740 10 client.28195 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc40466740 10 client.28195 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc40466740 20 client.28195 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc40466740 1 --2- 172.21.15.40:0/1044506726 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55cbdcf836d0 0x55cbdcf85b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc40466740 4 client.28195 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.769+0000 7efc40466740 1 -- 172.21.15.40:0/1044506726 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbdcf2f3e0 con 0x55cbdcf836d0 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.770+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/1044506726 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55cbdcf836d0 0x55cbdcf85b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.770+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/1044506726 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55cbdcf836d0 0x55cbdcf85b10 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7edf0693ccd76d2a server_cookie=bda28cf4ff20ed87 in_seq=0 out_seq=0 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.770+0000 7efc16ffd640 10 client.28195 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.771+0000 7efc16ffd640 1 -- 172.21.15.40:0/1044506726 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cbdcf2f3e0 con 0x55cbdcf836d0 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.771+0000 7efc16ffd640 10 client.28195 handle_command_reply: tid=0 2024-03-18T22:55:20.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.771+0000 7efc40466740 1 client.28195 shutdown 2024-03-18T22:55:20.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.771+0000 7efc40466740 1 -- 172.21.15.40:0/1044506726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 msgr2=0x55cbdcf77d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.771+0000 7efc40466740 1 --2- 172.21.15.40:0/1044506726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf77d10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efc2802c840 tx=0x7efc28005270 comp rx=0 tx=0).stop 2024-03-18T22:55:20.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.771+0000 7efc40466740 1 -- 172.21.15.40:0/1044506726 shutdown_connections 2024-03-18T22:55:20.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.771+0000 7efc40466740 1 --2- 172.21.15.40:0/1044506726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf72e50 0x55cbdcf7ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.771+0000 7efc40466740 1 --2- 172.21.15.40:0/1044506726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf77d10 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.771+0000 7efc40466740 1 --2- 172.21.15.40:0/1044506726 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55cbdcf836d0 0x55cbdcf85b10 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.771+0000 7efc40466740 1 --2- 172.21.15.40:0/1044506726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf777d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.771+0000 7efc16ffd640 0 client.28195 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:20.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.771+0000 7efc40466740 1 -- 172.21.15.40:0/1044506726 >> 172.21.15.40:0/1044506726 conn(0x55cbdcf68860 msgr2=0x55cbdcf67ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.771+0000 7efc40466740 1 -- 172.21.15.40:0/1044506726 shutdown_connections 2024-03-18T22:55:20.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.771+0000 7efc40466740 1 -- 172.21.15.40:0/1044506726 wait complete. 2024-03-18T22:55:20.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.771+0000 7efc40466740 20 client.28195 trim_cache size 0 max 16384 2024-03-18T22:55:20.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.784+0000 7efc40466740 1 Processor -- start 2024-03-18T22:55:20.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.784+0000 7efc40466740 1 -- start start 2024-03-18T22:55:20.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.784+0000 7efc40466740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf6f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.784+0000 7efc40466740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf6f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.784+0000 7efc40466740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf72e50 0x55cbdcf7ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.784+0000 7efc40466740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efc381aa4c0 con 0x55cbdcf6d2f0 2024-03-18T22:55:20.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.784+0000 7efc40466740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efc381ac000 con 0x55cbdcf72e50 2024-03-18T22:55:20.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.784+0000 7efc40466740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7efc381aab90 con 0x55cbdcf6dce0 2024-03-18T22:55:20.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.784+0000 7efc2e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf6f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.784+0000 7efc2e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf6f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35434/0 (socket says 172.21.15.40:35434) 2024-03-18T22:55:20.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.784+0000 7efc2e7fc640 1 -- 172.21.15.40:0/1027579459 learned_addr learned my addr 172.21.15.40:0/1027579459 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:20.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.785+0000 7efc36ffd640 1 --2- 172.21.15.40:0/1027579459 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf6f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.785+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/1027579459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf72e50 0x55cbdcf7ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.785+0000 7efc2e7fc640 1 -- 172.21.15.40:0/1027579459 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 msgr2=0x55cbdcf6f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.785+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/1027579459 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf6f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.785+0000 7efc2e7fc640 1 -- 172.21.15.40:0/1027579459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf72e50 msgr2=0x55cbdcf7ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.785+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/1027579459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf72e50 0x55cbdcf7ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.785+0000 7efc2e7fc640 1 -- 172.21.15.40:0/1027579459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbdcf86190 con 0x55cbdcf6d2f0 2024-03-18T22:55:20.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.785+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/1027579459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf6f1e0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7efc2000fc20 tx=0x7efc20009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8327e34f52db73e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.785+0000 7efc2d7fa640 1 -- 172.21.15.40:0/1027579459 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2000fd90 con 0x55cbdcf6d2f0 2024-03-18T22:55:20.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.785+0000 7efc2d7fa640 1 -- 172.21.15.40:0/1027579459 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc20011040 con 0x55cbdcf6d2f0 2024-03-18T22:55:20.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.785+0000 7efc2d7fa640 1 -- 172.21.15.40:0/1027579459 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc20004640 con 0x55cbdcf6d2f0 2024-03-18T22:55:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 -- 172.21.15.40:0/1027579459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf6d2f0 msgr2=0x55cbdcf6f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 --2- 172.21.15.40:0/1027579459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf6f1e0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7efc2000fc20 tx=0x7efc20009f40 comp rx=0 tx=0).stop 2024-03-18T22:55:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 -- 172.21.15.40:0/1027579459 shutdown_connections 2024-03-18T22:55:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 --2- 172.21.15.40:0/1027579459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf72e50 0x55cbdcf7ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 --2- 172.21.15.40:0/1027579459 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf6f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 --2- 172.21.15.40:0/1027579459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf6f1e0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 -- 172.21.15.40:0/1027579459 >> 172.21.15.40:0/1027579459 conn(0x55cbdcf68860 msgr2=0x55cbdcf777d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 -- 172.21.15.40:0/1027579459 shutdown_connections 2024-03-18T22:55:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 -- 172.21.15.40:0/1027579459 wait complete. 2024-03-18T22:55:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 -- start start 2024-03-18T22:55:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 Processor -- start 2024-03-18T22:55:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf7c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf7c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf72e50 0x55cbdcf7d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7efc381ab930 con 0x55cbdcf72e50 2024-03-18T22:55:20.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efc381ab260 con 0x55cbdcf6d2f0 2024-03-18T22:55:20.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7efc381a84a0 con 0x55cbdcf6dce0 2024-03-18T22:55:20.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.786+0000 7efc40466740 10 client.0 _ll_register_callbacks cb 0x7efc28009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:20.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc36ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf7c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc2e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf72e50 0x55cbdcf7d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc36ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf7c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:37912/0 (socket says 172.21.15.40:37912) 2024-03-18T22:55:20.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc36ffd640 1 -- 172.21.15.40:0/1331908935 learned_addr learned my addr 172.21.15.40:0/1331908935 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:20.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc2e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf72e50 0x55cbdcf7d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35440/0 (socket says 172.21.15.40:35440) 2024-03-18T22:55:20.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/1331908935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf7c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc2e7fc640 1 -- 172.21.15.40:0/1331908935 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 msgr2=0x55cbdcf7c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/1331908935 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf7c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc2e7fc640 1 -- 172.21.15.40:0/1331908935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 msgr2=0x55cbdcf7c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:20.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/1331908935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf7c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc2e7fc640 1 -- 172.21.15.40:0/1331908935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbdcf3e880 con 0x55cbdcf72e50 2024-03-18T22:55:20.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc36ffd640 1 --2- 172.21.15.40:0/1331908935 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf7c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:55:20.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc2e7fc640 1 --2- 172.21.15.40:0/1331908935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf72e50 0x55cbdcf7d630 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7efc20032010 tx=0x7efc200119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd941ef298ab217d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:20.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:55:20.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc16ffd640 1 -- 172.21.15.40:0/1331908935 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc20030450 con 0x55cbdcf72e50 2024-03-18T22:55:20.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc16ffd640 1 -- 172.21.15.40:0/1331908935 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7efc20005300 con 0x55cbdcf72e50 2024-03-18T22:55:20.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc40466740 1 -- 172.21.15.40:0/1331908935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc20011040 con 0x55cbdcf72e50 2024-03-18T22:55:20.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc16ffd640 1 -- 172.21.15.40:0/1331908935 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7efc2004f050 con 0x55cbdcf72e50 2024-03-18T22:55:20.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc40466740 10 client.18390 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:20.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc40466740 1 -- 172.21.15.40:0/1331908935 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55cbdcf7ed50 con 0x55cbdcf72e50 2024-03-18T22:55:20.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.787+0000 7efc16ffd640 1 -- 172.21.15.40:0/1331908935 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc20011040 con 0x55cbdcf72e50 2024-03-18T22:55:20.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc16ffd640 1 -- 172.21.15.40:0/1331908935 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7efc2001d020 con 0x55cbdcf72e50 2024-03-18T22:55:20.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc40466740 10 client.18390 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:20.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc40466740 10 client.18390 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:55:20.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc40466740 20 client.18390 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:20.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc40466740 1 --2- 172.21.15.40:0/1331908935 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55cbdcfe9d40 0x55cbdcfec130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:20.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc40466740 4 client.18390 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:55:20.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc40466740 1 -- 172.21.15.40:0/1331908935 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cbdcf6ecf0 con 0x55cbdcfe9d40 2024-03-18T22:55:20.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc36ffd640 1 --2- 172.21.15.40:0/1331908935 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55cbdcfe9d40 0x55cbdcfec130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:20.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc16ffd640 10 client.18390 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:20.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc36ffd640 1 --2- 172.21.15.40:0/1331908935 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55cbdcfe9d40 0x55cbdcfec130 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad9bf19bfbb96a5d server_cookie=aac9d3412574102a in_seq=0 out_seq=0 2024-03-18T22:55:20.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc16ffd640 1 -- 172.21.15.40:0/1331908935 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cbdcf6ecf0 con 0x55cbdcfe9d40 2024-03-18T22:55:20.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc16ffd640 10 client.18390 handle_command_reply: tid=0 2024-03-18T22:55:20.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc40466740 1 client.18390 shutdown 2024-03-18T22:55:20.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc40466740 1 -- 172.21.15.40:0/1331908935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf72e50 msgr2=0x55cbdcf7d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc40466740 1 --2- 172.21.15.40:0/1331908935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf72e50 0x55cbdcf7d630 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7efc20032010 tx=0x7efc200119b0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc40466740 1 -- 172.21.15.40:0/1331908935 shutdown_connections 2024-03-18T22:55:20.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc40466740 1 --2- 172.21.15.40:0/1331908935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55cbdcf72e50 0x55cbdcf7d630 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc40466740 1 --2- 172.21.15.40:0/1331908935 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55cbdcf6dce0 0x55cbdcf7c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.788+0000 7efc40466740 1 --2- 172.21.15.40:0/1331908935 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55cbdcfe9d40 0x55cbdcfec130 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc40466740 1 --2- 172.21.15.40:0/1331908935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbdcf6d2f0 0x55cbdcf7c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc16ffd640 0 client.18390 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:20.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc40466740 1 -- 172.21.15.40:0/1331908935 >> 172.21.15.40:0/1331908935 conn(0x55cbdcf68860 msgr2=0x55cbdcf776a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc40466740 1 -- 172.21.15.40:0/1331908935 shutdown_connections 2024-03-18T22:55:20.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc40466740 1 -- 172.21.15.40:0/1331908935 wait complete. 2024-03-18T22:55:20.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc40466740 20 client.18390 trim_cache size 0 max 16384 2024-03-18T22:55:20.799 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:55:20.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc3d87e640 1 -- 172.21.15.40:0/3914058617 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efc240a2880 msgr2=0x7efc240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc3d87e640 1 --2- 172.21.15.40:0/3914058617 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efc240a2880 0x7efc240a4d40 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7efc300049d0 tx=0x7efc30090f10 comp rx=0 tx=0).stop 2024-03-18T22:55:20.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc3d87e640 1 -- 172.21.15.40:0/3914058617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc381022d0 msgr2=0x7efc3819efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:20.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc3d87e640 1 --2- 172.21.15.40:0/3914058617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc381022d0 0x7efc3819efe0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7efc3000eb10 tx=0x7efc3000c2d0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc3d87e640 1 -- 172.21.15.40:0/3914058617 shutdown_connections 2024-03-18T22:55:20.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc3d87e640 1 --2- 172.21.15.40:0/3914058617 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7efc240a2880 0x7efc240a4d40 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc3d87e640 1 --2- 172.21.15.40:0/3914058617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc381022d0 0x7efc3819efe0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc3d87e640 1 --2- 172.21.15.40:0/3914058617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc38101910 0x7efc38197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc3d87e640 1 --2- 172.21.15.40:0/3914058617 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7efc38100720 0x7efc38197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:20.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc3d87e640 1 -- 172.21.15.40:0/3914058617 >> 172.21.15.40:0/3914058617 conn(0x7efc380fbcb0 msgr2=0x7efc38104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:20.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc3d87e640 1 -- 172.21.15.40:0/3914058617 shutdown_connections 2024-03-18T22:55:20.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:20.789+0000 7efc3d87e640 1 -- 172.21.15.40:0/3914058617 wait complete. 2024-03-18T22:55:21.805 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:55:21.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:55:21.806 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:55:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.035+0000 7f634bfc6640 1 -- 172.21.15.40:0/512553933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6344108f90 msgr2=0x7f6344109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.035+0000 7f634bfc6640 1 --2- 172.21.15.40:0/512553933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6344108f90 0x7f6344109410 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6334009900 tx=0x7f633402bc80 comp rx=0 tx=0).stop 2024-03-18T22:55:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.035+0000 7f634bfc6640 1 -- 172.21.15.40:0/512553933 shutdown_connections 2024-03-18T22:55:22.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.035+0000 7f634bfc6640 1 --2- 172.21.15.40:0/512553933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6344109950 0x7f6344110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.035+0000 7f634bfc6640 1 --2- 172.21.15.40:0/512553933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6344108f90 0x7f6344109410 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.035+0000 7f634bfc6640 1 --2- 172.21.15.40:0/512553933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6344107e40 0x7f6344108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.035+0000 7f634bfc6640 1 -- 172.21.15.40:0/512553933 >> 172.21.15.40:0/512553933 conn(0x7f6344076de0 msgr2=0x7f6344079220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:22.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.036+0000 7f634bfc6640 1 -- 172.21.15.40:0/512553933 shutdown_connections 2024-03-18T22:55:22.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.037+0000 7f634bfc6640 1 -- 172.21.15.40:0/512553933 wait complete. 2024-03-18T22:55:22.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.037+0000 7f634bfc6640 1 Processor -- start 2024-03-18T22:55:22.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.038+0000 7f634bfc6640 1 -- start start 2024-03-18T22:55:22.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.038+0000 7f634bfc6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6344107e40 0x7f6344070260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.038+0000 7f634bfc6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6344108f90 0x7f63440707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.038+0000 7f634bfc6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6344109950 0x7f63441ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.038+0000 7f634bfc6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63441139c0 con 0x7f6344108f90 2024-03-18T22:55:22.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.038+0000 7f634bfc6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6344113840 con 0x7f6344109950 2024-03-18T22:55:22.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.038+0000 7f634bfc6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6344113b40 con 0x7f6344107e40 2024-03-18T22:55:22.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.038+0000 7f634a563640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6344109950 0x7f63441ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.038+0000 7f634a563640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6344109950 0x7f63441ae0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47098/0 (socket says 172.21.15.40:47098) 2024-03-18T22:55:22.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.038+0000 7f634a563640 1 -- 172.21.15.40:0/853751933 learned_addr learned my addr 172.21.15.40:0/853751933 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:22.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.039+0000 7f6349d62640 1 --2- 172.21.15.40:0/853751933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6344107e40 0x7f6344070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.039+0000 7f634a563640 1 -- 172.21.15.40:0/853751933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6344107e40 msgr2=0x7f6344070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.039+0000 7f634a563640 1 --2- 172.21.15.40:0/853751933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6344107e40 0x7f6344070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.039+0000 7f634a563640 1 -- 172.21.15.40:0/853751933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6344108f90 msgr2=0x7f63440707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.039+0000 7f634a563640 1 --2- 172.21.15.40:0/853751933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6344108f90 0x7f63440707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.039+0000 7f634a563640 1 -- 172.21.15.40:0/853751933 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63441ae610 con 0x7f6344109950 2024-03-18T22:55:22.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.040+0000 7f634a563640 1 --2- 172.21.15.40:0/853751933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6344109950 0x7f63441ae0d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f633c00eb10 tx=0x7f633c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.040+0000 7f633affd640 1 -- 172.21.15.40:0/853751933 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c004b20 con 0x7f6344109950 2024-03-18T22:55:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.040+0000 7f633affd640 1 -- 172.21.15.40:0/853751933 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f633c004cc0 con 0x7f6344109950 2024-03-18T22:55:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.040+0000 7f633affd640 1 -- 172.21.15.40:0/853751933 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c010690 con 0x7f6344109950 2024-03-18T22:55:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.042+0000 7f634bfc6640 1 -- 172.21.15.40:0/853751933 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63441ae7e0 con 0x7f6344109950 2024-03-18T22:55:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.042+0000 7f634bfc6640 1 -- 172.21.15.40:0/853751933 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63441aec50 con 0x7f6344109950 2024-03-18T22:55:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.045+0000 7f634ebb5740 1 Processor -- start 2024-03-18T22:55:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.046+0000 7f633affd640 1 -- 172.21.15.40:0/853751933 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f633c005000 con 0x7f6344109950 2024-03-18T22:55:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.046+0000 7f634ebb5740 1 -- start start 2024-03-18T22:55:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.047+0000 7f634ebb5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a82f3a0 0x55590a82f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.047+0000 7f634ebb5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82fd90 0x55590a834890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.047+0000 7f634ebb5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a834f00 0x55590a837390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.047+0000 7f634ebb5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55590a794160 con 0x55590a834f00 2024-03-18T22:55:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.047+0000 7f634ebb5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55590a73ce10 con 0x55590a82fd90 2024-03-18T22:55:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.047+0000 7f634ebb5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55590a790200 con 0x55590a82f3a0 2024-03-18T22:55:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.047+0000 7f6349561640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a82f3a0 0x55590a82f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.047+0000 7f6349561640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a82f3a0 0x55590a82f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:37946/0 (socket says 172.21.15.40:37946) 2024-03-18T22:55:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.047+0000 7f6349561640 1 -- 172.21.15.40:0/1306214615 learned_addr learned my addr 172.21.15.40:0/1306214615 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:22.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.047+0000 7f6349d62640 1 --2- 172.21.15.40:0/1306214615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82fd90 0x55590a834890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.050+0000 7f6349561640 1 --2- 172.21.15.40:0/1306214615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a834f00 0x55590a837390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.050+0000 7f6349d62640 1 -- 172.21.15.40:0/1306214615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a82f3a0 msgr2=0x55590a82f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.050+0000 7f6349d62640 1 --2- 172.21.15.40:0/1306214615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a82f3a0 0x55590a82f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.050+0000 7f6349d62640 1 -- 172.21.15.40:0/1306214615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a834f00 msgr2=0x55590a837390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.050+0000 7f633affd640 1 --2- 172.21.15.40:0/853751933 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63300a2980 0x7f63300a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.050+0000 7f633affd640 1 -- 172.21.15.40:0/853751933 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f633c0cf420 con 0x7f6344109950 2024-03-18T22:55:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.050+0000 7f6349d62640 1 --2- 172.21.15.40:0/1306214615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a834f00 0x55590a837390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.050+0000 7f6349d62640 1 -- 172.21.15.40:0/1306214615 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55590a5707e0 con 0x55590a82fd90 2024-03-18T22:55:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.050+0000 7f6349561640 1 --2- 172.21.15.40:0/1306214615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a834f00 0x55590a837390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:55:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.051+0000 7f6349561640 1 --2- 172.21.15.40:0/853751933 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63300a2980 0x7f63300a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.051+0000 7f6349d62640 1 --2- 172.21.15.40:0/1306214615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82fd90 0x55590a834890 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f634000bbf0 tx=0x7f63400107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adde6bd6137c6d65 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.051+0000 7f6349561640 1 --2- 172.21.15.40:0/853751933 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63300a2980 0x7f63300a4e40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6334004c40 tx=0x7f6334009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.051+0000 7f632ffff640 1 -- 172.21.15.40:0/1306214615 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340004030 con 0x55590a82fd90 2024-03-18T22:55:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.052+0000 7f632ffff640 1 -- 172.21.15.40:0/1306214615 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63400035f0 con 0x55590a82fd90 2024-03-18T22:55:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.053+0000 7f634ebb5740 1 -- 172.21.15.40:0/1306214615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82fd90 msgr2=0x55590a834890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.053+0000 7f634ebb5740 1 --2- 172.21.15.40:0/1306214615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82fd90 0x55590a834890 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f634000bbf0 tx=0x7f63400107a0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.053+0000 7f634ebb5740 1 -- 172.21.15.40:0/1306214615 shutdown_connections 2024-03-18T22:55:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.053+0000 7f634ebb5740 1 --2- 172.21.15.40:0/1306214615 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a834f00 0x55590a837390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.053+0000 7f634ebb5740 1 --2- 172.21.15.40:0/1306214615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82fd90 0x55590a834890 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.053+0000 7f634ebb5740 1 --2- 172.21.15.40:0/1306214615 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a82f3a0 0x55590a82f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.053+0000 7f634ebb5740 1 -- 172.21.15.40:0/1306214615 >> 172.21.15.40:0/1306214615 conn(0x55590a82a910 msgr2=0x55590a82ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:22.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.054+0000 7f634ebb5740 1 -- 172.21.15.40:0/1306214615 shutdown_connections 2024-03-18T22:55:22.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.054+0000 7f634ebb5740 1 -- 172.21.15.40:0/1306214615 wait complete. 2024-03-18T22:55:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.054+0000 7f634ebb5740 1 -- start start 2024-03-18T22:55:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.054+0000 7f634ebb5740 1 Processor -- start 2024-03-18T22:55:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.055+0000 7f634ebb5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82f3a0 0x55590a84a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.055+0000 7f634ebb5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82fd90 0x55590a84ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.055+0000 7f634ebb5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a834f00 0x55590a851dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.055+0000 7f634ebb5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55590a837f00 con 0x55590a82fd90 2024-03-18T22:55:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.055+0000 7f634ebb5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55590a7501a0 con 0x55590a82f3a0 2024-03-18T22:55:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.055+0000 7f634ebb5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55590a838080 con 0x55590a834f00 2024-03-18T22:55:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.055+0000 7f634ebb5740 10 client.0 _ll_register_callbacks cb 0x55590a4a7c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.055+0000 7f6349d62640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82f3a0 0x55590a84a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.055+0000 7f6349d62640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82f3a0 0x55590a84a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47110/0 (socket says 172.21.15.40:47110) 2024-03-18T22:55:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.055+0000 7f6349d62640 1 -- 172.21.15.40:0/1050773101 learned_addr learned my addr 172.21.15.40:0/1050773101 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.055+0000 7f634a563640 1 --2- 172.21.15.40:0/1050773101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a834f00 0x55590a851dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.056+0000 7f6349561640 1 --2- 172.21.15.40:0/1050773101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82fd90 0x55590a84ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.057+0000 7f6349561640 1 -- 172.21.15.40:0/1050773101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a834f00 msgr2=0x55590a851dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.057+0000 7f6349561640 1 --2- 172.21.15.40:0/1050773101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a834f00 0x55590a851dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.057+0000 7f6349561640 1 -- 172.21.15.40:0/1050773101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82f3a0 msgr2=0x55590a84a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.057+0000 7f6349561640 1 --2- 172.21.15.40:0/1050773101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82f3a0 0x55590a84a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.057+0000 7f6349561640 1 -- 172.21.15.40:0/1050773101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55590a5d0f70 con 0x55590a82fd90 2024-03-18T22:55:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.057+0000 7f6349561640 1 --2- 172.21.15.40:0/1050773101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82fd90 0x55590a84ad80 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f633402c210 tx=0x7f6334000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8674f0ad34f86e78 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.057+0000 7f632d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:55:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.057+0000 7f632d7fa640 1 -- 172.21.15.40:0/1050773101 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633404e020 con 0x55590a82fd90 2024-03-18T22:55:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.058+0000 7f634ebb5740 1 -- 172.21.15.40:0/1050773101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63400035f0 con 0x55590a82fd90 2024-03-18T22:55:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.058+0000 7f632d7fa640 1 -- 172.21.15.40:0/1050773101 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6334005430 con 0x55590a82fd90 2024-03-18T22:55:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.058+0000 7f632d7fa640 1 -- 172.21.15.40:0/1050773101 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f6334059050 con 0x55590a82fd90 2024-03-18T22:55:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.058+0000 7f634ebb5740 10 client.18396 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.058+0000 7f634ebb5740 1 -- 172.21.15.40:0/1050773101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55590a468170 con 0x55590a82fd90 2024-03-18T22:55:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.058+0000 7f632d7fa640 1 -- 172.21.15.40:0/1050773101 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63400035f0 con 0x55590a82fd90 2024-03-18T22:55:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.058+0000 7f632d7fa640 1 -- 172.21.15.40:0/1050773101 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f63340554c0 con 0x55590a82fd90 2024-03-18T22:55:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.058+0000 7f634ebb5740 10 client.18396 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.059+0000 7f634ebb5740 10 client.18396 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:55:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.059+0000 7f634ebb5740 20 client.18396 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.059+0000 7f634ebb5740 1 --2- 172.21.15.40:0/1050773101 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55590a83c770 0x55590a83ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.059+0000 7f634ebb5740 4 client.18396 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:55:22.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.059+0000 7f634ebb5740 1 -- 172.21.15.40:0/1050773101 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55590a7a16a0 con 0x55590a83c770 2024-03-18T22:55:22.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.059+0000 7f6349d62640 1 --2- 172.21.15.40:0/1050773101 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55590a83c770 0x55590a83ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.064+0000 7f6349d62640 1 --2- 172.21.15.40:0/1050773101 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55590a83c770 0x55590a83ec30 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40843a40d00920eb server_cookie=665e31dfcac48e3 in_seq=0 out_seq=0 2024-03-18T22:55:22.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.064+0000 7f632d7fa640 10 client.18396 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:22.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.065+0000 7f632d7fa640 1 -- 172.21.15.40:0/1050773101 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55590a7a16a0 con 0x55590a83c770 2024-03-18T22:55:22.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.065+0000 7f632d7fa640 10 client.18396 handle_command_reply: tid=0 2024-03-18T22:55:22.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.065+0000 7f634ebb5740 1 client.18396 shutdown 2024-03-18T22:55:22.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.065+0000 7f634ebb5740 1 -- 172.21.15.40:0/1050773101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82fd90 msgr2=0x55590a84ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.065+0000 7f634ebb5740 1 --2- 172.21.15.40:0/1050773101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82fd90 0x55590a84ad80 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f633402c210 tx=0x7f6334000bc0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.065+0000 7f634ebb5740 1 -- 172.21.15.40:0/1050773101 shutdown_connections 2024-03-18T22:55:22.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.065+0000 7f634ebb5740 1 --2- 172.21.15.40:0/1050773101 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a834f00 0x55590a851dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.065+0000 7f634ebb5740 1 --2- 172.21.15.40:0/1050773101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82fd90 0x55590a84ad80 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.065+0000 7f634ebb5740 1 --2- 172.21.15.40:0/1050773101 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55590a83c770 0x55590a83ec30 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.065+0000 7f634ebb5740 1 --2- 172.21.15.40:0/1050773101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82f3a0 0x55590a84a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.065+0000 7f634ebb5740 1 -- 172.21.15.40:0/1050773101 >> 172.21.15.40:0/1050773101 conn(0x55590a82a910 msgr2=0x55590a829b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:22.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.065+0000 7f632d7fa640 0 client.18396 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:22.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.065+0000 7f634ebb5740 1 -- 172.21.15.40:0/1050773101 shutdown_connections 2024-03-18T22:55:22.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.065+0000 7f634ebb5740 1 -- 172.21.15.40:0/1050773101 wait complete. 2024-03-18T22:55:22.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.065+0000 7f634ebb5740 20 client.18396 trim_cache size 0 max 16384 2024-03-18T22:55:22.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.101+0000 7f634ebb5740 1 Processor -- start 2024-03-18T22:55:22.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.101+0000 7f634ebb5740 1 -- start start 2024-03-18T22:55:22.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.101+0000 7f634ebb5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82f3a0 0x55590a8292a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.101+0000 7f634ebb5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82fd90 0x55590a851d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.101+0000 7f634ebb5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a834f00 0x55590a8297e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.102+0000 7f634ebb5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63441b0b80 con 0x55590a82f3a0 2024-03-18T22:55:22.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.102+0000 7f6349d62640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82f3a0 0x55590a8292a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.102+0000 7f6349d62640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82f3a0 0x55590a8292a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35492/0 (socket says 172.21.15.40:35492) 2024-03-18T22:55:22.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.102+0000 7f6349d62640 1 -- 172.21.15.40:0/2054078174 learned_addr learned my addr 172.21.15.40:0/2054078174 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:22.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.102+0000 7f634ebb5740 1 -- 172.21.15.40:0/2054078174 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63441b26c0 con 0x55590a82fd90 2024-03-18T22:55:22.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.102+0000 7f634ebb5740 1 -- 172.21.15.40:0/2054078174 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f63441b1250 con 0x55590a834f00 2024-03-18T22:55:22.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.102+0000 7f6349561640 1 --2- 172.21.15.40:0/2054078174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82fd90 0x55590a851d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.102+0000 7f634a563640 1 --2- 172.21.15.40:0/2054078174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a834f00 0x55590a8297e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.102+0000 7f6349d62640 1 -- 172.21.15.40:0/2054078174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a834f00 msgr2=0x55590a8297e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:22.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.102+0000 7f6349d62640 1 --2- 172.21.15.40:0/2054078174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a834f00 0x55590a8297e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.102+0000 7f6349d62640 1 -- 172.21.15.40:0/2054078174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82fd90 msgr2=0x55590a851d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:22.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.102+0000 7f6349d62640 1 --2- 172.21.15.40:0/2054078174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82fd90 0x55590a851d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.102+0000 7f6349d62640 1 -- 172.21.15.40:0/2054078174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55590a83acc0 con 0x55590a82f3a0 2024-03-18T22:55:22.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.102+0000 7f6349561640 1 --2- 172.21.15.40:0/2054078174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82fd90 0x55590a851d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:55:22.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.103+0000 7f6349d62640 1 --2- 172.21.15.40:0/2054078174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82f3a0 0x55590a8292a0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f6340011010 tx=0x7f6340005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a847ba9b593bfb9b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:22.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.103+0000 7f632f7fe640 1 -- 172.21.15.40:0/2054078174 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634002f070 con 0x55590a82f3a0 2024-03-18T22:55:22.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.103+0000 7f632f7fe640 1 -- 172.21.15.40:0/2054078174 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6340011840 con 0x55590a82f3a0 2024-03-18T22:55:22.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.103+0000 7f632f7fe640 1 -- 172.21.15.40:0/2054078174 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340003850 con 0x55590a82f3a0 2024-03-18T22:55:22.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.104+0000 7f634ebb5740 1 -- 172.21.15.40:0/2054078174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82f3a0 msgr2=0x55590a8292a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.104+0000 7f634ebb5740 1 --2- 172.21.15.40:0/2054078174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82f3a0 0x55590a8292a0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f6340011010 tx=0x7f6340005040 comp rx=0 tx=0).stop 2024-03-18T22:55:22.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.104+0000 7f634ebb5740 1 -- 172.21.15.40:0/2054078174 shutdown_connections 2024-03-18T22:55:22.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.104+0000 7f634ebb5740 1 --2- 172.21.15.40:0/2054078174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a834f00 0x55590a8297e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.104+0000 7f634ebb5740 1 --2- 172.21.15.40:0/2054078174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a82fd90 0x55590a851d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.104+0000 7f634ebb5740 1 --2- 172.21.15.40:0/2054078174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82f3a0 0x55590a8292a0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.104+0000 7f634ebb5740 1 -- 172.21.15.40:0/2054078174 >> 172.21.15.40:0/2054078174 conn(0x55590a82a910 msgr2=0x55590a8310b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:22.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.105+0000 7f634ebb5740 1 -- 172.21.15.40:0/2054078174 shutdown_connections 2024-03-18T22:55:22.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.105+0000 7f634ebb5740 1 -- 172.21.15.40:0/2054078174 wait complete. 2024-03-18T22:55:22.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.105+0000 7f634ebb5740 1 -- start start 2024-03-18T22:55:22.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.106+0000 7f634ebb5740 1 Processor -- start 2024-03-18T22:55:22.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.106+0000 7f634ebb5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a82f3a0 0x55590a851b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.106+0000 7f634ebb5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82fd90 0x55590a850130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.107+0000 7f6349d62640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a82f3a0 0x55590a851b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.107+0000 7f634ebb5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a834f00 0x55590a850670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.107+0000 7f634ebb5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f63441b1ff0 con 0x55590a82fd90 2024-03-18T22:55:22.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.107+0000 7f634ebb5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63441b1920 con 0x55590a834f00 2024-03-18T22:55:22.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.107+0000 7f634ebb5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6344082470 con 0x55590a82f3a0 2024-03-18T22:55:22.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.107+0000 7f6349d62640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a82f3a0 0x55590a851b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:37984/0 (socket says 172.21.15.40:37984) 2024-03-18T22:55:22.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.107+0000 7f6349d62640 1 -- 172.21.15.40:0/2458816585 learned_addr learned my addr 172.21.15.40:0/2458816585 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:22.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.107+0000 7f634ebb5740 10 client.0 _ll_register_callbacks cb 0x7f633c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:22.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.107+0000 7f6349561640 1 --2- 172.21.15.40:0/2458816585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82fd90 0x55590a850130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.107+0000 7f634a563640 1 --2- 172.21.15.40:0/2458816585 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a834f00 0x55590a850670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.107+0000 7f6349d62640 1 -- 172.21.15.40:0/2458816585 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a834f00 msgr2=0x55590a850670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:22.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.107+0000 7f6349d62640 1 --2- 172.21.15.40:0/2458816585 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a834f00 0x55590a850670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.107+0000 7f6349d62640 1 -- 172.21.15.40:0/2458816585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82fd90 msgr2=0x55590a850130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:22.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.107+0000 7f6349d62640 1 --2- 172.21.15.40:0/2458816585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82fd90 0x55590a850130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.107+0000 7f6349d62640 1 -- 172.21.15.40:0/2458816585 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55590a850c60 con 0x55590a82f3a0 2024-03-18T22:55:22.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.108+0000 7f6349d62640 1 --2- 172.21.15.40:0/2458816585 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a82f3a0 0x55590a851b20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6340011c20 tx=0x7f6340010dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3587e60783ff0ee0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:22.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.109+0000 7f632d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:55:22.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.109+0000 7f632d7fa640 1 -- 172.21.15.40:0/2458816585 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634003a060 con 0x55590a82f3a0 2024-03-18T22:55:22.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.110+0000 7f632d7fa640 1 -- 172.21.15.40:0/2458816585 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f63400052b0 con 0x55590a82f3a0 2024-03-18T22:55:22.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.110+0000 7f634ebb5740 1 -- 172.21.15.40:0/2458816585 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6340011840 con 0x55590a82f3a0 2024-03-18T22:55:22.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.110+0000 7f632d7fa640 1 -- 172.21.15.40:0/2458816585 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f6340041050 con 0x55590a82f3a0 2024-03-18T22:55:22.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.111+0000 7f634ebb5740 10 client.28219 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:22.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.111+0000 7f634ebb5740 1 -- 172.21.15.40:0/2458816585 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55590a8ce240 con 0x55590a82f3a0 2024-03-18T22:55:22.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.112+0000 7f632d7fa640 1 -- 172.21.15.40:0/2458816585 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6340011e00 con 0x55590a82f3a0 2024-03-18T22:55:22.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.112+0000 7f632d7fa640 1 -- 172.21.15.40:0/2458816585 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f63400035f0 con 0x55590a82f3a0 2024-03-18T22:55:22.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.112+0000 7f634ebb5740 10 client.28219 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:22.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.112+0000 7f634ebb5740 10 client.28219 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:55:22.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.112+0000 7f634ebb5740 20 client.28219 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:22.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.112+0000 7f634ebb5740 1 --2- 172.21.15.40:0/2458816585 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55590a8ce880 0x55590a8d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.112+0000 7f634ebb5740 4 client.28219 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:55:22.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.112+0000 7f634ebb5740 1 -- 172.21.15.40:0/2458816585 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f6340011840 con 0x55590a8ce880 2024-03-18T22:55:22.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.113+0000 7f6349561640 1 --2- 172.21.15.40:0/2458816585 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55590a8ce880 0x55590a8d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.114+0000 7f632d7fa640 10 client.28219 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:22.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.114+0000 7f6349561640 1 --2- 172.21.15.40:0/2458816585 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55590a8ce880 0x55590a8d0ca0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5b92543c59a5fd8 server_cookie=b0991b7a6554633f in_seq=0 out_seq=0 2024-03-18T22:55:22.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.115+0000 7f632d7fa640 1 -- 172.21.15.40:0/2458816585 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f6340011840 con 0x55590a8ce880 2024-03-18T22:55:22.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.115+0000 7f632d7fa640 10 client.28219 handle_command_reply: tid=0 2024-03-18T22:55:22.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.115+0000 7f634ebb5740 1 client.28219 shutdown 2024-03-18T22:55:22.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.115+0000 7f634ebb5740 1 -- 172.21.15.40:0/2458816585 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a82f3a0 msgr2=0x55590a851b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.115+0000 7f634ebb5740 1 --2- 172.21.15.40:0/2458816585 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a82f3a0 0x55590a851b20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f6340011c20 tx=0x7f6340010dd0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.116+0000 7f634ebb5740 1 -- 172.21.15.40:0/2458816585 shutdown_connections 2024-03-18T22:55:22.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.116+0000 7f634ebb5740 1 --2- 172.21.15.40:0/2458816585 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55590a8ce880 0x55590a8d0ca0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.116+0000 7f634ebb5740 1 --2- 172.21.15.40:0/2458816585 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55590a834f00 0x55590a850670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.116+0000 7f632d7fa640 0 client.28219 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:22.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.116+0000 7f634ebb5740 1 --2- 172.21.15.40:0/2458816585 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55590a82fd90 0x55590a850130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.116+0000 7f634ebb5740 1 --2- 172.21.15.40:0/2458816585 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55590a82f3a0 0x55590a851b20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.116+0000 7f634ebb5740 1 -- 172.21.15.40:0/2458816585 >> 172.21.15.40:0/2458816585 conn(0x55590a82a910 msgr2=0x55590a82e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:22.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.118+0000 7f634ebb5740 1 -- 172.21.15.40:0/2458816585 shutdown_connections 2024-03-18T22:55:22.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.118+0000 7f634ebb5740 1 -- 172.21.15.40:0/2458816585 wait complete. 2024-03-18T22:55:22.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.119+0000 7f634ebb5740 20 client.28219 trim_cache size 0 max 16384 2024-03-18T22:55:22.121 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:55:22.121 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:55:22.121 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "fb8e8f52-e4b3-4c44-babd-6e7ce44455f5", 2024-03-18T22:55:22.121 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:55:22.121 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:55:22.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.121+0000 7f634bfc6640 1 -- 172.21.15.40:0/853751933 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63300a2980 msgr2=0x7f63300a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.121+0000 7f634bfc6640 1 --2- 172.21.15.40:0/853751933 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63300a2980 0x7f63300a4e40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6334004c40 tx=0x7f6334009260 comp rx=0 tx=0).stop 2024-03-18T22:55:22.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.121+0000 7f634bfc6640 1 -- 172.21.15.40:0/853751933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6344109950 msgr2=0x7f63441ae0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.121+0000 7f634bfc6640 1 --2- 172.21.15.40:0/853751933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6344109950 0x7f63441ae0d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f633c00eb10 tx=0x7f633c00c2d0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.123+0000 7f634bfc6640 1 -- 172.21.15.40:0/853751933 shutdown_connections 2024-03-18T22:55:22.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.123+0000 7f634bfc6640 1 --2- 172.21.15.40:0/853751933 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f63300a2980 0x7f63300a4e40 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.123+0000 7f634bfc6640 1 --2- 172.21.15.40:0/853751933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6344109950 0x7f63441ae0d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.123+0000 7f634bfc6640 1 --2- 172.21.15.40:0/853751933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6344108f90 0x7f63440707a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.123+0000 7f634bfc6640 1 --2- 172.21.15.40:0/853751933 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6344107e40 0x7f6344070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.123+0000 7f634bfc6640 1 -- 172.21.15.40:0/853751933 >> 172.21.15.40:0/853751933 conn(0x7f6344076de0 msgr2=0x7f634410c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:22.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.123+0000 7f634bfc6640 1 -- 172.21.15.40:0/853751933 shutdown_connections 2024-03-18T22:55:22.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.123+0000 7f634bfc6640 1 -- 172.21.15.40:0/853751933 wait complete. 2024-03-18T22:55:22.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:55:22.148 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:55:22.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.338+0000 7f79472dc640 1 -- 172.21.15.40:0/2994203855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940102310 msgr2=0x7f7940108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.338+0000 7f79472dc640 1 --2- 172.21.15.40:0/2994203855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940102310 0x7f7940108ba0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f7938009ca0 tx=0x7f793802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.338+0000 7f79472dc640 1 -- 172.21.15.40:0/2994203855 shutdown_connections 2024-03-18T22:55:22.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.338+0000 7f79472dc640 1 --2- 172.21.15.40:0/2994203855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940102310 0x7f7940108ba0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.338+0000 7f79472dc640 1 --2- 172.21.15.40:0/2994203855 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7940101950 0x7f7940101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.338+0000 7f79472dc640 1 --2- 172.21.15.40:0/2994203855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7940100760 0x7f7940100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.338+0000 7f79472dc640 1 -- 172.21.15.40:0/2994203855 >> 172.21.15.40:0/2994203855 conn(0x7f79400fbcd0 msgr2=0x7f79400fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:22.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.338+0000 7f79472dc640 1 -- 172.21.15.40:0/2994203855 shutdown_connections 2024-03-18T22:55:22.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.341+0000 7f79472dc640 1 -- 172.21.15.40:0/2994203855 wait complete. 2024-03-18T22:55:22.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.342+0000 7f79472dc640 1 Processor -- start 2024-03-18T22:55:22.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.342+0000 7f79472dc640 1 -- start start 2024-03-18T22:55:22.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.343+0000 7f79472dc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940100760 0x7f7940197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.343+0000 7f79472dc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7940101950 0x7f7940197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.343+0000 7f79472dc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7940102310 0x7f794019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.343+0000 7f79472dc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f794010d360 con 0x7f7940100760 2024-03-18T22:55:22.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.343+0000 7f79472dc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f794010d1e0 con 0x7f7940102310 2024-03-18T22:55:22.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.343+0000 7f79472dc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f794010d4e0 con 0x7f7940101950 2024-03-18T22:55:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.344+0000 7f7945879640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7940102310 0x7f794019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.344+0000 7f7945879640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7940102310 0x7f794019f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47154/0 (socket says 172.21.15.40:47154) 2024-03-18T22:55:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.344+0000 7f7945879640 1 -- 172.21.15.40:0/465526559 learned_addr learned my addr 172.21.15.40:0/465526559 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.344+0000 7f7945078640 1 --2- 172.21.15.40:0/465526559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940100760 0x7f7940197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.344+0000 7f7944877640 1 --2- 172.21.15.40:0/465526559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7940101950 0x7f7940197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.345+0000 7f7945078640 1 -- 172.21.15.40:0/465526559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7940101950 msgr2=0x7f7940197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.345+0000 7f7945078640 1 --2- 172.21.15.40:0/465526559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7940101950 0x7f7940197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.345+0000 7f7945078640 1 -- 172.21.15.40:0/465526559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7940102310 msgr2=0x7f794019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.345+0000 7f7945078640 1 --2- 172.21.15.40:0/465526559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7940102310 0x7f794019f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.345+0000 7f7945078640 1 -- 172.21.15.40:0/465526559 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794019f680 con 0x7f7940100760 2024-03-18T22:55:22.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.345+0000 7f7945078640 1 --2- 172.21.15.40:0/465526559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940100760 0x7f7940197a60 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f793c00d6a0 tx=0x7f793c00db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:22.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.345+0000 7f79367fc640 1 -- 172.21.15.40:0/465526559 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c004220 con 0x7f7940100760 2024-03-18T22:55:22.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.345+0000 7f79367fc640 1 -- 172.21.15.40:0/465526559 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f793c028070 con 0x7f7940100760 2024-03-18T22:55:22.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.345+0000 7f79367fc640 1 -- 172.21.15.40:0/465526559 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c003c50 con 0x7f7940100760 2024-03-18T22:55:22.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.346+0000 7f79472dc640 1 -- 172.21.15.40:0/465526559 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f794006b2a0 con 0x7f7940100760 2024-03-18T22:55:22.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.346+0000 7f79472dc640 1 -- 172.21.15.40:0/465526559 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f794006b7a0 con 0x7f7940100760 2024-03-18T22:55:22.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.349+0000 7f79367fc640 1 -- 172.21.15.40:0/465526559 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f793c009820 con 0x7f7940100760 2024-03-18T22:55:22.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.349+0000 7f79367fc640 1 --2- 172.21.15.40:0/465526559 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f792c0a29d0 0x7f792c0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.349+0000 7f79367fc640 1 -- 172.21.15.40:0/465526559 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f793c0df780 con 0x7f7940100760 2024-03-18T22:55:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.349+0000 7f7944877640 1 --2- 172.21.15.40:0/465526559 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f792c0a29d0 0x7f792c0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.350+0000 7f7944877640 1 --2- 172.21.15.40:0/465526559 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f792c0a29d0 0x7f792c0a4e90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7940198f70 tx=0x7f7930005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:22.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.351+0000 7f7949fa3740 1 Processor -- start 2024-03-18T22:55:22.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.351+0000 7f7949fa3740 1 -- start start 2024-03-18T22:55:22.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.351+0000 7f7949fa3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f7582df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.351+0000 7f7949fa3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f75833c0 0x5624f7585eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.351+0000 7f7949fa3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f7586520 0x5624f7593390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.351+0000 7f7949fa3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5624f74e7500 con 0x5624f75833c0 2024-03-18T22:55:22.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.351+0000 7f7949fa3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5624f748fbe0 con 0x5624f75829d0 2024-03-18T22:55:22.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.351+0000 7f7949fa3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5624f74e35a0 con 0x5624f7586520 2024-03-18T22:55:22.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.352+0000 7f7945078640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f7582df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.352+0000 7f7945078640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f7582df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47162/0 (socket says 172.21.15.40:47162) 2024-03-18T22:55:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.352+0000 7f7945078640 1 -- 172.21.15.40:0/2986726698 learned_addr learned my addr 172.21.15.40:0/2986726698 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.352+0000 7f7945879640 1 --2- 172.21.15.40:0/2986726698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f7586520 0x5624f7593390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.352+0000 7f7945078640 1 -- 172.21.15.40:0/2986726698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f7586520 msgr2=0x5624f7593390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.352+0000 7f7945078640 1 --2- 172.21.15.40:0/2986726698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f7586520 0x5624f7593390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.352+0000 7f7945078640 1 -- 172.21.15.40:0/2986726698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f75833c0 msgr2=0x5624f7585eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.352+0000 7f7945078640 1 --2- 172.21.15.40:0/2986726698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f75833c0 0x5624f7585eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.352+0000 7f7945078640 1 -- 172.21.15.40:0/2986726698 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624f74b1000 con 0x5624f75829d0 2024-03-18T22:55:22.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.353+0000 7f7945078640 1 --2- 172.21.15.40:0/2986726698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f7582df0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f793c0ab9d0 tx=0x7f793c009d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6807cd7e106a9360 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:22.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.354+0000 7f792b7fe640 1 -- 172.21.15.40:0/2986726698 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c031050 con 0x5624f75829d0 2024-03-18T22:55:22.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.354+0000 7f792b7fe640 1 -- 172.21.15.40:0/2986726698 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f793c0afe20 con 0x5624f75829d0 2024-03-18T22:55:22.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.356+0000 7f7949fa3740 1 -- 172.21.15.40:0/2986726698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 msgr2=0x5624f7582df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.356+0000 7f7949fa3740 1 --2- 172.21.15.40:0/2986726698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f7582df0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f793c0ab9d0 tx=0x7f793c009d60 comp rx=0 tx=0).stop 2024-03-18T22:55:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.356+0000 7f7949fa3740 1 -- 172.21.15.40:0/2986726698 shutdown_connections 2024-03-18T22:55:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.356+0000 7f7949fa3740 1 --2- 172.21.15.40:0/2986726698 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f7586520 0x5624f7593390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.356+0000 7f7949fa3740 1 --2- 172.21.15.40:0/2986726698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f75833c0 0x5624f7585eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.356+0000 7f7949fa3740 1 --2- 172.21.15.40:0/2986726698 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f7582df0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.356+0000 7f7949fa3740 1 -- 172.21.15.40:0/2986726698 >> 172.21.15.40:0/2986726698 conn(0x5624f757dfc0 msgr2=0x5624f757e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.357+0000 7f7949fa3740 1 -- 172.21.15.40:0/2986726698 shutdown_connections 2024-03-18T22:55:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.357+0000 7f7949fa3740 1 -- 172.21.15.40:0/2986726698 wait complete. 2024-03-18T22:55:22.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.357+0000 7f7949fa3740 1 -- start start 2024-03-18T22:55:22.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.357+0000 7f7949fa3740 1 Processor -- start 2024-03-18T22:55:22.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.358+0000 7f7949fa3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f75829d0 0x5624f758ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.358+0000 7f7949fa3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 0x5624f758d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.358+0000 7f7949fa3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f7586520 0x5624f7592370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.358+0000 7f7949fa3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5624f7586f50 con 0x5624f75829d0 2024-03-18T22:55:22.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.358+0000 7f7949fa3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5624f74a3160 con 0x5624f7586520 2024-03-18T22:55:22.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.359+0000 7f7949fa3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5624f75870d0 con 0x5624f75833c0 2024-03-18T22:55:22.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.359+0000 7f7949fa3740 10 client.0 _ll_register_callbacks cb 0x5624f74dfa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:22.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.359+0000 7f7945879640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f7586520 0x5624f7592370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.359+0000 7f7945879640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f7586520 0x5624f7592370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47178/0 (socket says 172.21.15.40:47178) 2024-03-18T22:55:22.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.359+0000 7f7945879640 1 -- 172.21.15.40:0/3996708773 learned_addr learned my addr 172.21.15.40:0/3996708773 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:22.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.359+0000 7f7945879640 1 -- 172.21.15.40:0/3996708773 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 msgr2=0x5624f758d340 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:55:22.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.359+0000 7f7945879640 1 --2- 172.21.15.40:0/3996708773 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 0x5624f758d340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.359+0000 7f7945879640 1 -- 172.21.15.40:0/3996708773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f75829d0 msgr2=0x5624f758ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:22.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.359+0000 7f7945879640 1 --2- 172.21.15.40:0/3996708773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f75829d0 0x5624f758ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.359+0000 7f7945879640 1 -- 172.21.15.40:0/3996708773 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624f72c57d0 con 0x5624f7586520 2024-03-18T22:55:22.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.362+0000 7f7945879640 1 --2- 172.21.15.40:0/3996708773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f7586520 0x5624f7592370 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7938005e30 tx=0x7f7938005f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e689f5bbbca3b94f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:22.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.364+0000 7f7928ff9640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:55:22.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.364+0000 7f7928ff9640 1 -- 172.21.15.40:0/3996708773 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7938039070 con 0x5624f7586520 2024-03-18T22:55:22.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.364+0000 7f7928ff9640 1 -- 172.21.15.40:0/3996708773 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7938003970 con 0x5624f7586520 2024-03-18T22:55:22.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.365+0000 7f7949fa3740 1 -- 172.21.15.40:0/3996708773 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f793c0afe20 con 0x5624f7586520 2024-03-18T22:55:22.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.365+0000 7f7928ff9640 1 -- 172.21.15.40:0/3996708773 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7938003b70 con 0x5624f7586520 2024-03-18T22:55:22.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.365+0000 7f7949fa3740 10 client.28244 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:22.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.365+0000 7f7949fa3740 1 -- 172.21.15.40:0/3996708773 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5624f71fb3f0 con 0x5624f7586520 2024-03-18T22:55:22.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.367+0000 7f7928ff9640 1 -- 172.21.15.40:0/3996708773 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f793c0afe20 con 0x5624f7586520 2024-03-18T22:55:22.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.367+0000 7f7928ff9640 1 -- 172.21.15.40:0/3996708773 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f7938040480 con 0x5624f7586520 2024-03-18T22:55:22.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.367+0000 7f7949fa3740 10 client.28244 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:22.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.367+0000 7f7949fa3740 10 client.28244 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:55:22.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.367+0000 7f7949fa3740 20 client.28244 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:22.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.368+0000 7f7949fa3740 1 --2- 172.21.15.40:0/3996708773 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5624f75a9d90 0x5624f75ac230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.368+0000 7f7949fa3740 4 client.28244 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:55:22.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.368+0000 7f7949fa3740 1 -- 172.21.15.40:0/3996708773 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624f74f4a40 con 0x5624f75a9d90 2024-03-18T22:55:22.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.376+0000 7f7945078640 1 --2- 172.21.15.40:0/3996708773 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5624f75a9d90 0x5624f75ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.380+0000 7f7945078640 1 --2- 172.21.15.40:0/3996708773 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5624f75a9d90 0x5624f75ac230 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=522629ad465d2f84 server_cookie=ed1f0a316d340c2c in_seq=0 out_seq=0 2024-03-18T22:55:22.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.381+0000 7f7928ff9640 10 client.28244 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:22.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.381+0000 7f7928ff9640 1 -- 172.21.15.40:0/3996708773 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5624f74f4a40 con 0x5624f75a9d90 2024-03-18T22:55:22.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.381+0000 7f7928ff9640 10 client.28244 handle_command_reply: tid=0 2024-03-18T22:55:22.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.382+0000 7f7949fa3740 1 client.28244 shutdown 2024-03-18T22:55:22.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.382+0000 7f7949fa3740 1 -- 172.21.15.40:0/3996708773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f7586520 msgr2=0x5624f7592370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.382+0000 7f7949fa3740 1 --2- 172.21.15.40:0/3996708773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f7586520 0x5624f7592370 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7938005e30 tx=0x7f7938005f60 comp rx=0 tx=0).stop 2024-03-18T22:55:22.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.384+0000 7f7949fa3740 1 -- 172.21.15.40:0/3996708773 shutdown_connections 2024-03-18T22:55:22.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.384+0000 7f7949fa3740 1 --2- 172.21.15.40:0/3996708773 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5624f75a9d90 0x5624f75ac230 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.384+0000 7f7949fa3740 1 --2- 172.21.15.40:0/3996708773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f7586520 0x5624f7592370 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.384+0000 7f7928ff9640 0 client.28244 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:22.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.384+0000 7f7949fa3740 1 --2- 172.21.15.40:0/3996708773 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 0x5624f758d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.384+0000 7f7949fa3740 1 --2- 172.21.15.40:0/3996708773 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f75829d0 0x5624f758ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.384+0000 7f7949fa3740 1 -- 172.21.15.40:0/3996708773 >> 172.21.15.40:0/3996708773 conn(0x5624f757dfc0 msgr2=0x5624f757d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:22.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.385+0000 7f7949fa3740 1 -- 172.21.15.40:0/3996708773 shutdown_connections 2024-03-18T22:55:22.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.385+0000 7f7949fa3740 1 -- 172.21.15.40:0/3996708773 wait complete. 2024-03-18T22:55:22.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.386+0000 7f7949fa3740 20 client.28244 trim_cache size 0 max 16384 2024-03-18T22:55:22.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.445+0000 7f7949fa3740 1 Processor -- start 2024-03-18T22:55:22.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.445+0000 7f7949fa3740 1 -- start start 2024-03-18T22:55:22.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.445+0000 7f7949fa3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f758bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.445+0000 7f7949fa3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 0x5624f757c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.445+0000 7f7949fa3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f7586520 0x5624f757ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.445+0000 7f7949fa3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f794019fba0 con 0x5624f7586520 2024-03-18T22:55:22.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.445+0000 7f7949fa3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79401a16e0 con 0x5624f75829d0 2024-03-18T22:55:22.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.445+0000 7f7949fa3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f79401a0270 con 0x5624f75833c0 2024-03-18T22:55:22.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.446+0000 7f7944877640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 0x5624f757c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.446+0000 7f7944877640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 0x5624f757c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38022/0 (socket says 172.21.15.40:38022) 2024-03-18T22:55:22.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.446+0000 7f7944877640 1 -- 172.21.15.40:0/161312637 learned_addr learned my addr 172.21.15.40:0/161312637 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:22.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.446+0000 7f7945078640 1 --2- 172.21.15.40:0/161312637 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f758bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.446+0000 7f7945879640 1 --2- 172.21.15.40:0/161312637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f7586520 0x5624f757ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.446+0000 7f7945078640 1 -- 172.21.15.40:0/161312637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 msgr2=0x5624f757c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:22.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.446+0000 7f7945078640 1 --2- 172.21.15.40:0/161312637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 0x5624f757c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.446+0000 7f7945078640 1 -- 172.21.15.40:0/161312637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f7586520 msgr2=0x5624f757ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:22.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.446+0000 7f7945078640 1 --2- 172.21.15.40:0/161312637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f7586520 0x5624f757ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.446+0000 7f7945078640 1 -- 172.21.15.40:0/161312637 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624f71bb0f0 con 0x5624f75829d0 2024-03-18T22:55:22.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.449+0000 7f7945078640 1 --2- 172.21.15.40:0/161312637 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f758bb70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f793c0df6f0 tx=0x7f793c0a9fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b69c36cab93b2b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.450+0000 7f792affd640 1 -- 172.21.15.40:0/161312637 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c031070 con 0x5624f75829d0 2024-03-18T22:55:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.450+0000 7f792affd640 1 -- 172.21.15.40:0/161312637 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f793c0c9070 con 0x5624f75829d0 2024-03-18T22:55:22.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.450+0000 7f792affd640 1 -- 172.21.15.40:0/161312637 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c0ca270 con 0x5624f75829d0 2024-03-18T22:55:22.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.452+0000 7f7949fa3740 1 -- 172.21.15.40:0/161312637 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 msgr2=0x5624f758bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.452+0000 7f7949fa3740 1 --2- 172.21.15.40:0/161312637 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f758bb70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f793c0df6f0 tx=0x7f793c0a9fd0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.453+0000 7f7949fa3740 1 -- 172.21.15.40:0/161312637 shutdown_connections 2024-03-18T22:55:22.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.453+0000 7f7949fa3740 1 --2- 172.21.15.40:0/161312637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f7586520 0x5624f757ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.453+0000 7f7949fa3740 1 --2- 172.21.15.40:0/161312637 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 0x5624f757c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.453+0000 7f7949fa3740 1 --2- 172.21.15.40:0/161312637 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f758bb70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.453+0000 7f7949fa3740 1 -- 172.21.15.40:0/161312637 >> 172.21.15.40:0/161312637 conn(0x5624f757dfc0 msgr2=0x5624f758b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:22.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.461+0000 7f7949fa3740 1 -- 172.21.15.40:0/161312637 shutdown_connections 2024-03-18T22:55:22.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.461+0000 7f7949fa3740 1 -- 172.21.15.40:0/161312637 wait complete. 2024-03-18T22:55:22.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.461+0000 7f7949fa3740 1 -- start start 2024-03-18T22:55:22.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.461+0000 7f7949fa3740 1 Processor -- start 2024-03-18T22:55:22.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.462+0000 7f7949fa3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f75bb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.462+0000 7f7949fa3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 0x5624f75bb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.462+0000 7f7949fa3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f7586520 0x5624f75bc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.462+0000 7f7949fa3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f79401a1010 con 0x5624f7586520 2024-03-18T22:55:22.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.462+0000 7f7949fa3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f79401a0940 con 0x5624f75829d0 2024-03-18T22:55:22.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.462+0000 7f7949fa3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7940071cc0 con 0x5624f75833c0 2024-03-18T22:55:22.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.462+0000 7f7949fa3740 10 client.0 _ll_register_callbacks cb 0x7f7938037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:22.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.464+0000 7f7944877640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 0x5624f75bb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.464+0000 7f7945078640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f75bb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.464+0000 7f7944877640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 0x5624f75bb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38024/0 (socket says 172.21.15.40:38024) 2024-03-18T22:55:22.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.464+0000 7f7945078640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f75bb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:47186/0 (socket says 172.21.15.40:47186) 2024-03-18T22:55:22.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.464+0000 7f7944877640 1 -- 172.21.15.40:0/2789388236 learned_addr learned my addr 172.21.15.40:0/2789388236 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:22.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.464+0000 7f7945879640 1 --2- 172.21.15.40:0/2789388236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f7586520 0x5624f75bc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.466+0000 7f7944877640 1 -- 172.21.15.40:0/2789388236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 msgr2=0x5624f75bb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:22.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.466+0000 7f7944877640 1 --2- 172.21.15.40:0/2789388236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f75bb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.466+0000 7f7944877640 1 -- 172.21.15.40:0/2789388236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f7586520 msgr2=0x5624f75bc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:22.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.466+0000 7f7944877640 1 --2- 172.21.15.40:0/2789388236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f7586520 0x5624f75bc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.467+0000 7f7944877640 1 -- 172.21.15.40:0/2789388236 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624f75bcc70 con 0x5624f75833c0 2024-03-18T22:55:22.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.467+0000 7f7945879640 1 --2- 172.21.15.40:0/2789388236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f7586520 0x5624f75bc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:55:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.468+0000 7f7944877640 1 --2- 172.21.15.40:0/2789388236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 0x5624f75bb670 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f79300058c0 tx=0x7f79300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d31bb7a7350135f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.468+0000 7f7928ff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:55:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.468+0000 7f7928ff9640 1 -- 172.21.15.40:0/2789388236 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793002b060 con 0x5624f75833c0 2024-03-18T22:55:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.468+0000 7f7928ff9640 1 -- 172.21.15.40:0/2789388236 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f793000ecc0 con 0x5624f75833c0 2024-03-18T22:55:22.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.469+0000 7f7949fa3740 1 -- 172.21.15.40:0/2789388236 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f793c0c9070 con 0x5624f75833c0 2024-03-18T22:55:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.470+0000 7f7928ff9640 1 -- 172.21.15.40:0/2789388236 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f79300263f0 con 0x5624f75833c0 2024-03-18T22:55:22.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.471+0000 7f7949fa3740 10 client.28231 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:22.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.471+0000 7f7949fa3740 1 -- 172.21.15.40:0/2789388236 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5624f7610320 con 0x5624f75833c0 2024-03-18T22:55:22.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.475+0000 7f7928ff9640 1 -- 172.21.15.40:0/2789388236 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f793c0c9070 con 0x5624f75833c0 2024-03-18T22:55:22.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.475+0000 7f7928ff9640 1 -- 172.21.15.40:0/2789388236 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f7930025030 con 0x5624f75833c0 2024-03-18T22:55:22.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.477+0000 7f7949fa3740 10 client.28231 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:22.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.477+0000 7f7949fa3740 10 client.28231 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:55:22.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.477+0000 7f7949fa3740 20 client.28231 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:22.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.477+0000 7f7949fa3740 1 --2- 172.21.15.40:0/2789388236 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5624f75a9d90 0x5624f7612cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:22.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.477+0000 7f7949fa3740 4 client.28231 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:55:22.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.477+0000 7f7949fa3740 1 -- 172.21.15.40:0/2789388236 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5624f74eed90 con 0x5624f75a9d90 2024-03-18T22:55:22.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.478+0000 7f7945078640 1 --2- 172.21.15.40:0/2789388236 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5624f75a9d90 0x5624f7612cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:22.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.479+0000 7f7945078640 1 --2- 172.21.15.40:0/2789388236 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5624f75a9d90 0x5624f7612cc0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c23afe07d50e9c63 server_cookie=baa926d883c8d0bd in_seq=0 out_seq=0 2024-03-18T22:55:22.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.481+0000 7f7928ff9640 10 client.28231 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:22.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.481+0000 7f7928ff9640 1 -- 172.21.15.40:0/2789388236 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5624f74eed90 con 0x5624f75a9d90 2024-03-18T22:55:22.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.481+0000 7f7928ff9640 10 client.28231 handle_command_reply: tid=0 2024-03-18T22:55:22.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.481+0000 7f7949fa3740 1 client.28231 shutdown 2024-03-18T22:55:22.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.484+0000 7f7949fa3740 1 -- 172.21.15.40:0/2789388236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 msgr2=0x5624f75bb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.484+0000 7f7949fa3740 1 --2- 172.21.15.40:0/2789388236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 0x5624f75bb670 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f79300058c0 tx=0x7f79300046c0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.484+0000 7f7949fa3740 1 -- 172.21.15.40:0/2789388236 shutdown_connections 2024-03-18T22:55:22.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.484+0000 7f7949fa3740 1 --2- 172.21.15.40:0/2789388236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5624f7586520 0x5624f75bc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.484+0000 7f7949fa3740 1 --2- 172.21.15.40:0/2789388236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5624f75833c0 0x5624f75bb670 secure :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f79300058c0 tx=0x7f79300046c0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.484+0000 7f7949fa3740 1 --2- 172.21.15.40:0/2789388236 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5624f75a9d90 0x5624f7612cc0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.484+0000 7f7949fa3740 1 --2- 172.21.15.40:0/2789388236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5624f75829d0 0x5624f75bb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.484+0000 7f7949fa3740 1 -- 172.21.15.40:0/2789388236 >> 172.21.15.40:0/2789388236 conn(0x5624f757dfc0 msgr2=0x5624f758bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:22.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.485+0000 7f7928ff9640 0 client.28231 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:22.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.485+0000 7f7949fa3740 1 -- 172.21.15.40:0/2789388236 shutdown_connections 2024-03-18T22:55:22.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.485+0000 7f7949fa3740 1 -- 172.21.15.40:0/2789388236 wait complete. 2024-03-18T22:55:22.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.486+0000 7f7949fa3740 20 client.28231 trim_cache size 0 max 16384 2024-03-18T22:55:22.491 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:55:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "scrub active (100 inodes in the stack)", 2024-03-18T22:55:22.491 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:55:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "fb8e8f52-e4b3-4c44-babd-6e7ce44455f5": { 2024-03-18T22:55:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:55:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "fb8e8f52-e4b3-4c44-babd-6e7ce44455f5", 2024-03-18T22:55:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:55:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:55:22.492 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:55:22.492 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:55:22.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.490+0000 7f79472dc640 1 -- 172.21.15.40:0/465526559 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f792c0a29d0 msgr2=0x7f792c0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.490+0000 7f79472dc640 1 --2- 172.21.15.40:0/465526559 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f792c0a29d0 0x7f792c0a4e90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7940198f70 tx=0x7f7930005f90 comp rx=0 tx=0).stop 2024-03-18T22:55:22.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.490+0000 7f79472dc640 1 -- 172.21.15.40:0/465526559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940100760 msgr2=0x7f7940197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:22.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.490+0000 7f79472dc640 1 --2- 172.21.15.40:0/465526559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940100760 0x7f7940197a60 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f793c00d6a0 tx=0x7f793c00db60 comp rx=0 tx=0).stop 2024-03-18T22:55:22.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.490+0000 7f79472dc640 1 -- 172.21.15.40:0/465526559 shutdown_connections 2024-03-18T22:55:22.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.490+0000 7f79472dc640 1 --2- 172.21.15.40:0/465526559 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f792c0a29d0 0x7f792c0a4e90 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.490+0000 7f79472dc640 1 --2- 172.21.15.40:0/465526559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7940102310 0x7f794019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.490+0000 7f79472dc640 1 --2- 172.21.15.40:0/465526559 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7940101950 0x7f7940197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.490+0000 7f79472dc640 1 --2- 172.21.15.40:0/465526559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7940100760 0x7f7940197a60 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:22.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.491+0000 7f79472dc640 1 -- 172.21.15.40:0/465526559 >> 172.21.15.40:0/465526559 conn(0x7f79400fbcd0 msgr2=0x7f7940104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:22.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.493+0000 7f79472dc640 1 -- 172.21.15.40:0/465526559 shutdown_connections 2024-03-18T22:55:22.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:22.495+0000 7f79472dc640 1 -- 172.21.15.40:0/465526559 wait complete. 2024-03-18T22:55:22.522 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:22 smithi040 ceph-mon[62643]: pgmap v732: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 1.89k op/s 2024-03-18T22:55:22.523 INFO:tasks.cephfs.filesystem:scrub status for tag:fb8e8f52-e4b3-4c44-babd-6e7ce44455f5 - {'path': '/', 'tag': 'fb8e8f52-e4b3-4c44-babd-6e7ce44455f5', 'options': 'recursive,force'} 2024-03-18T22:55:22.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:22 smithi113 ceph-mon[66775]: pgmap v732: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 1.89k op/s 2024-03-18T22:55:22.611 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:22 smithi103 ceph-mon[66377]: pgmap v732: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 1.89k op/s 2024-03-18T22:55:23.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:23 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:55:23.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:23 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:23.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:23 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:55:23.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:23 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:55:23.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:23 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:55:23.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:23 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:55:23.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:23 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:55:23.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:23 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:55:23.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:23 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:23.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:23 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:55:23.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:23 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:55:23.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:23 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:55:23.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:23 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:55:23.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:23 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:55:23.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:23 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:55:23.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:23 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:23.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:23 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:55:23.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:23 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:55:23.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:23 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:55:23.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:23 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:55:23.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:23 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:55:24.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:24 smithi113 ceph-mon[66775]: pgmap v733: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 2.22k op/s 2024-03-18T22:55:24.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:24 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:24.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:24 smithi103 ceph-mon[66377]: pgmap v733: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 2.22k op/s 2024-03-18T22:55:24.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:24 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:24.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:24 smithi040 ceph-mon[62643]: pgmap v733: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 2.22k op/s 2024-03-18T22:55:24.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:24 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:25.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:25 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:25.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:25 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:25.647 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:25 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:26.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:26 smithi113 ceph-mon[66775]: pgmap v734: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 1.97k op/s 2024-03-18T22:55:26.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:26 smithi103 ceph-mon[66377]: pgmap v734: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 1.97k op/s 2024-03-18T22:55:26.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:26 smithi040 ceph-mon[62643]: pgmap v734: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 1.97k op/s 2024-03-18T22:55:28.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:28 smithi113 ceph-mon[66775]: pgmap v735: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.6 MiB/s wr, 1.94k op/s 2024-03-18T22:55:28.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:28 smithi103 ceph-mon[66377]: pgmap v735: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.6 MiB/s wr, 1.94k op/s 2024-03-18T22:55:28.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:28 smithi040 ceph-mon[62643]: pgmap v735: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.6 MiB/s wr, 1.94k op/s 2024-03-18T22:55:30.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:30 smithi113 ceph-mon[66775]: pgmap v736: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.2 MiB/s wr, 2.10k op/s 2024-03-18T22:55:30.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:30 smithi103 ceph-mon[66377]: pgmap v736: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.2 MiB/s wr, 2.10k op/s 2024-03-18T22:55:30.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:30 smithi040 ceph-mon[62643]: pgmap v736: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.2 MiB/s wr, 2.10k op/s 2024-03-18T22:55:32.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:32 smithi113 ceph-mon[66775]: pgmap v737: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 1.83k op/s 2024-03-18T22:55:32.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:32 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:32.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:32 smithi103 ceph-mon[66377]: pgmap v737: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 1.83k op/s 2024-03-18T22:55:32.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:32 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:32.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:32 smithi040 ceph-mon[62643]: pgmap v737: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 1.83k op/s 2024-03-18T22:55:32.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:32 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:34.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:34 smithi113 ceph-mon[66775]: pgmap v738: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.4 MiB/s wr, 2.14k op/s 2024-03-18T22:55:34.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:34 smithi103 ceph-mon[66377]: pgmap v738: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.4 MiB/s wr, 2.14k op/s 2024-03-18T22:55:34.648 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:34 smithi040 ceph-mon[62643]: pgmap v738: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.4 MiB/s wr, 2.14k op/s 2024-03-18T22:55:36.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:36 smithi113 ceph-mon[66775]: pgmap v739: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 8.0 MiB/s wr, 2.05k op/s 2024-03-18T22:55:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:36 smithi103 ceph-mon[66377]: pgmap v739: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 8.0 MiB/s wr, 2.05k op/s 2024-03-18T22:55:36.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:36 smithi040 ceph-mon[62643]: pgmap v739: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 8.0 MiB/s wr, 2.05k op/s 2024-03-18T22:55:38.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:38 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:38.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:38 smithi113 ceph-mon[66775]: pgmap v740: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 8.1 MiB/s wr, 2.06k op/s 2024-03-18T22:55:38.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:38 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:38.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:38 smithi103 ceph-mon[66377]: pgmap v740: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 8.1 MiB/s wr, 2.06k op/s 2024-03-18T22:55:38.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:38 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:55:38.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:38 smithi040 ceph-mon[62643]: pgmap v740: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 8.1 MiB/s wr, 2.06k op/s 2024-03-18T22:55:40.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:40 smithi113 ceph-mon[66775]: pgmap v741: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.9 MiB/s wr, 2.53k op/s 2024-03-18T22:55:40.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:40 smithi103 ceph-mon[66377]: pgmap v741: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.9 MiB/s wr, 2.53k op/s 2024-03-18T22:55:40.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:40 smithi040 ceph-mon[62643]: pgmap v741: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.9 MiB/s wr, 2.53k op/s 2024-03-18T22:55:42.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:42 smithi113 ceph-mon[66775]: pgmap v742: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.9 MiB/s wr, 2.27k op/s 2024-03-18T22:55:42.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:42 smithi103 ceph-mon[66377]: pgmap v742: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.9 MiB/s wr, 2.27k op/s 2024-03-18T22:55:42.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:42 smithi040 ceph-mon[62643]: pgmap v742: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.9 MiB/s wr, 2.27k op/s 2024-03-18T22:55:44.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:44 smithi113 ceph-mon[66775]: pgmap v743: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 MiB/s wr, 2.68k op/s 2024-03-18T22:55:44.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:44 smithi103 ceph-mon[66377]: pgmap v743: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 MiB/s wr, 2.68k op/s 2024-03-18T22:55:44.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:44 smithi040 ceph-mon[62643]: pgmap v743: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 10 MiB/s wr, 2.68k op/s 2024-03-18T22:55:46.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:46 smithi113 ceph-mon[66775]: pgmap v744: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.70k op/s 2024-03-18T22:55:46.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:46 smithi103 ceph-mon[66377]: pgmap v744: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.70k op/s 2024-03-18T22:55:46.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:46 smithi040 ceph-mon[62643]: pgmap v744: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.70k op/s 2024-03-18T22:55:48.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:48 smithi103 ceph-mon[66377]: pgmap v745: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 MiB/s wr, 2.56k op/s 2024-03-18T22:55:48.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:48 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:55:48.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:48 smithi040 ceph-mon[62643]: pgmap v745: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 MiB/s wr, 2.56k op/s 2024-03-18T22:55:48.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:48 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:55:48.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:48 smithi113 ceph-mon[66775]: pgmap v745: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 MiB/s wr, 2.56k op/s 2024-03-18T22:55:48.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:48 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:55:50.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:50 smithi103 ceph-mon[66377]: pgmap v746: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 11 MiB/s wr, 2.77k op/s 2024-03-18T22:55:50.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:50 smithi040 ceph-mon[62643]: pgmap v746: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 11 MiB/s wr, 2.77k op/s 2024-03-18T22:55:50.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:50 smithi113 ceph-mon[66775]: pgmap v746: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 11 MiB/s wr, 2.77k op/s 2024-03-18T22:55:52.524 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:55:52.524 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:55:52.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:52 smithi103 ceph-mon[66377]: pgmap v747: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 MiB/s wr, 2.25k op/s 2024-03-18T22:55:52.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:52 smithi040 ceph-mon[62643]: pgmap v747: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 MiB/s wr, 2.25k op/s 2024-03-18T22:55:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.731+0000 7f4c316d1640 1 -- 172.21.15.40:0/4092150983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c2c104c90 msgr2=0x7f4c2c107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.731+0000 7f4c316d1640 1 --2- 172.21.15.40:0/4092150983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c2c104c90 0x7f4c2c107080 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f4c14009900 tx=0x7f4c1402bc30 comp rx=0 tx=0).stop 2024-03-18T22:55:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.731+0000 7f4c316d1640 1 -- 172.21.15.40:0/4092150983 shutdown_connections 2024-03-18T22:55:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.731+0000 7f4c316d1640 1 --2- 172.21.15.40:0/4092150983 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c2c1075c0 0x7f4c2c1099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.731+0000 7f4c316d1640 1 --2- 172.21.15.40:0/4092150983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c2c104c90 0x7f4c2c107080 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.731+0000 7f4c316d1640 1 --2- 172.21.15.40:0/4092150983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c2c102360 0x7f4c2c104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.731+0000 7f4c316d1640 1 -- 172.21.15.40:0/4092150983 >> 172.21.15.40:0/4092150983 conn(0x7f4c2c0fbcb0 msgr2=0x7f4c2c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:52.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.732+0000 7f4c316d1640 1 -- 172.21.15.40:0/4092150983 shutdown_connections 2024-03-18T22:55:52.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.733+0000 7f4c316d1640 1 -- 172.21.15.40:0/4092150983 wait complete. 2024-03-18T22:55:52.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.733+0000 7f4c316d1640 1 Processor -- start 2024-03-18T22:55:52.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.734+0000 7f4c316d1640 1 -- start start 2024-03-18T22:55:52.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.734+0000 7f4c316d1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c2c102360 0x7f4c2c195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.734+0000 7f4c316d1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c2c104c90 0x7f4c2c195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.734+0000 7f4c316d1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c2c1075c0 0x7f4c2c1a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.734+0000 7f4c316d1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c10d3b0 con 0x7f4c2c1075c0 2024-03-18T22:55:52.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.734+0000 7f4c316d1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c10d230 con 0x7f4c2c104c90 2024-03-18T22:55:52.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.734+0000 7f4c316d1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c10d530 con 0x7f4c2c102360 2024-03-18T22:55:52.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.734+0000 7f4c2a7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c2c104c90 0x7f4c2c195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:52.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.734+0000 7f4c2a7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c2c104c90 0x7f4c2c195e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36638/0 (socket says 172.21.15.40:36638) 2024-03-18T22:55:52.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.734+0000 7f4c2a7fc640 1 -- 172.21.15.40:0/4156657240 learned_addr learned my addr 172.21.15.40:0/4156657240 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:52.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.735+0000 7f4c2affd640 1 --2- 172.21.15.40:0/4156657240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c2c102360 0x7f4c2c195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:52.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.735+0000 7f4c2b7fe640 1 --2- 172.21.15.40:0/4156657240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c2c1075c0 0x7f4c2c1a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:52.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.735+0000 7f4c2a7fc640 1 -- 172.21.15.40:0/4156657240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c2c102360 msgr2=0x7f4c2c195900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:52.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.735+0000 7f4c2a7fc640 1 --2- 172.21.15.40:0/4156657240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c2c102360 0x7f4c2c195900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.735+0000 7f4c2a7fc640 1 -- 172.21.15.40:0/4156657240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c2c1075c0 msgr2=0x7f4c2c1a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:52.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.735+0000 7f4c2a7fc640 1 --2- 172.21.15.40:0/4156657240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c2c1075c0 0x7f4c2c1a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.735+0000 7f4c2a7fc640 1 -- 172.21.15.40:0/4156657240 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2c1a5d50 con 0x7f4c2c104c90 2024-03-18T22:55:52.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.735+0000 7f4c2a7fc640 1 --2- 172.21.15.40:0/4156657240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c2c104c90 0x7f4c2c195e40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4c14000c00 tx=0x7f4c140040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:52.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.735+0000 7f4c23fff640 1 -- 172.21.15.40:0/4156657240 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14043020 con 0x7f4c2c104c90 2024-03-18T22:55:52.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.736+0000 7f4c316d1640 1 -- 172.21.15.40:0/4156657240 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c2c1a5fe0 con 0x7f4c2c104c90 2024-03-18T22:55:52.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.736+0000 7f4c23fff640 1 -- 172.21.15.40:0/4156657240 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c140044c0 con 0x7f4c2c104c90 2024-03-18T22:55:52.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.736+0000 7f4c23fff640 1 -- 172.21.15.40:0/4156657240 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14003c20 con 0x7f4c2c104c90 2024-03-18T22:55:52.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.736+0000 7f4c316d1640 1 -- 172.21.15.40:0/4156657240 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c2c1a64f0 con 0x7f4c2c104c90 2024-03-18T22:55:52.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.739+0000 7f4c23fff640 1 -- 172.21.15.40:0/4156657240 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4c14046020 con 0x7f4c2c104c90 2024-03-18T22:55:52.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.739+0000 7f4c34396740 1 Processor -- start 2024-03-18T22:55:52.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.739+0000 7f4c34396740 1 -- start start 2024-03-18T22:55:52.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.740+0000 7f4c34396740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 0x5587c8776e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.740+0000 7f4c34396740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c8777440 0x5587c877bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.740+0000 7f4c34396740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c877c5b0 0x5587c877ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.740+0000 7f4c34396740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5587c86db500 con 0x5587c877c5b0 2024-03-18T22:55:52.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.740+0000 7f4c34396740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5587c8683be0 con 0x5587c8776a50 2024-03-18T22:55:52.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.740+0000 7f4c34396740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5587c86d75a0 con 0x5587c8777440 2024-03-18T22:55:52.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.740+0000 7f4c2b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 0x5587c8776e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:52.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.740+0000 7f4c2a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c877c5b0 0x5587c877ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:52.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.740+0000 7f4c2b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 0x5587c8776e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36646/0 (socket says 172.21.15.40:36646) 2024-03-18T22:55:52.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.740+0000 7f4c2b7fe640 1 -- 172.21.15.40:0/1550002345 learned_addr learned my addr 172.21.15.40:0/1550002345 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:52.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.740+0000 7f4c2affd640 1 --2- 172.21.15.40:0/1550002345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c8777440 0x5587c877bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:52.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.740+0000 7f4c23fff640 1 --2- 172.21.15.40:0/4156657240 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4c180a2800 0x7f4c180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.741+0000 7f4c2b7fe640 1 -- 172.21.15.40:0/1550002345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c8777440 msgr2=0x5587c877bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:52.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.741+0000 7f4c2b7fe640 1 --2- 172.21.15.40:0/1550002345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c8777440 0x5587c877bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.741+0000 7f4c2b7fe640 1 -- 172.21.15.40:0/1550002345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c877c5b0 msgr2=0x5587c877ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:52.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.741+0000 7f4c2b7fe640 1 --2- 172.21.15.40:0/1550002345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c877c5b0 0x5587c877ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.741+0000 7f4c2b7fe640 1 -- 172.21.15.40:0/1550002345 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587c86a5000 con 0x5587c8776a50 2024-03-18T22:55:52.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.741+0000 7f4c2a7fc640 1 --2- 172.21.15.40:0/1550002345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c877c5b0 0x5587c877ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:55:52.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.741+0000 7f4c23fff640 1 -- 172.21.15.40:0/4156657240 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4c140f10c0 con 0x7f4c2c104c90 2024-03-18T22:55:52.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.741+0000 7f4c2b7fe640 1 --2- 172.21.15.40:0/1550002345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 0x5587c8776e70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009f50 tx=0x7f4c1c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9555ed9823a26ca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:52.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.741+0000 7f4c2b7fe640 1 --2- 172.21.15.40:0/4156657240 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4c180a2800 0x7f4c180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:52.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.741+0000 7f4c217fa640 1 -- 172.21.15.40:0/1550002345 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c01e070 con 0x5587c8776a50 2024-03-18T22:55:52.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.741+0000 7f4c217fa640 1 -- 172.21.15.40:0/1550002345 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c1c003df0 con 0x5587c8776a50 2024-03-18T22:55:52.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.742+0000 7f4c217fa640 1 -- 172.21.15.40:0/1550002345 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c019430 con 0x5587c8776a50 2024-03-18T22:55:52.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.742+0000 7f4c34396740 1 -- 172.21.15.40:0/1550002345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 msgr2=0x5587c8776e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:52.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.742+0000 7f4c34396740 1 --2- 172.21.15.40:0/1550002345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 0x5587c8776e70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009f50 tx=0x7f4c1c0118d0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.742+0000 7f4c2b7fe640 1 --2- 172.21.15.40:0/4156657240 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4c180a2800 0x7f4c180a4cc0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c01b000 tx=0x7f4c1c00ffd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:52.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.742+0000 7f4c34396740 1 -- 172.21.15.40:0/1550002345 shutdown_connections 2024-03-18T22:55:52.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.742+0000 7f4c34396740 1 --2- 172.21.15.40:0/1550002345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c877c5b0 0x5587c877ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.742+0000 7f4c34396740 1 --2- 172.21.15.40:0/1550002345 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c8777440 0x5587c877bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.742+0000 7f4c34396740 1 --2- 172.21.15.40:0/1550002345 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 0x5587c8776e70 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.742+0000 7f4c34396740 1 -- 172.21.15.40:0/1550002345 >> 172.21.15.40:0/1550002345 conn(0x5587c8771fc0 msgr2=0x5587c87723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:52.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.742+0000 7f4c34396740 1 -- 172.21.15.40:0/1550002345 shutdown_connections 2024-03-18T22:55:52.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.742+0000 7f4c34396740 1 -- 172.21.15.40:0/1550002345 wait complete. 2024-03-18T22:55:52.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.743+0000 7f4c34396740 1 -- start start 2024-03-18T22:55:52.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.743+0000 7f4c34396740 1 Processor -- start 2024-03-18T22:55:52.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.743+0000 7f4c34396740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c8776a50 0x5587c8780ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.743+0000 7f4c34396740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 0x5587c8781420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.743+0000 7f4c34396740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c877c5b0 0x5587c8788460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.743+0000 7f4c34396740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5587c877f5a0 con 0x5587c8777440 2024-03-18T22:55:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.743+0000 7f4c2a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 0x5587c8781420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.743+0000 7f4c2a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 0x5587c8781420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42432/0 (socket says 172.21.15.40:42432) 2024-03-18T22:55:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.743+0000 7f4c2a7fc640 1 -- 172.21.15.40:0/3834485400 learned_addr learned my addr 172.21.15.40:0/3834485400 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.743+0000 7f4c2affd640 1 --2- 172.21.15.40:0/3834485400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c8776a50 0x5587c8780ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.743+0000 7f4c34396740 1 -- 172.21.15.40:0/3834485400 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5587c8697160 con 0x5587c877c5b0 2024-03-18T22:55:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.743+0000 7f4c34396740 1 -- 172.21.15.40:0/3834485400 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5587c877f720 con 0x5587c8776a50 2024-03-18T22:55:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.743+0000 7f4c34396740 10 client.0 _ll_register_callbacks cb 0x5587c86d3a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.743+0000 7f4c2b7fe640 1 --2- 172.21.15.40:0/3834485400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c877c5b0 0x5587c8788460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.744+0000 7f4c2a7fc640 1 -- 172.21.15.40:0/3834485400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c8776a50 msgr2=0x5587c8780ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.744+0000 7f4c2a7fc640 1 --2- 172.21.15.40:0/3834485400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c8776a50 0x5587c8780ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.744+0000 7f4c2a7fc640 1 -- 172.21.15.40:0/3834485400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c877c5b0 msgr2=0x5587c8788460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.744+0000 7f4c2a7fc640 1 --2- 172.21.15.40:0/3834485400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c877c5b0 0x5587c8788460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.744+0000 7f4c2a7fc640 1 -- 172.21.15.40:0/3834485400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587c84b97d0 con 0x5587c8777440 2024-03-18T22:55:52.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.744+0000 7f4c2affd640 1 --2- 172.21.15.40:0/3834485400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c8776a50 0x5587c8780ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:55:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.744+0000 7f4c2a7fc640 1 --2- 172.21.15.40:0/3834485400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 0x5587c8781420 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f4c1402ce10 tx=0x7f4c140035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46fa5995a036b51e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.744+0000 7f4c127fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:55:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.744+0000 7f4c127fc640 1 -- 172.21.15.40:0/3834485400 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c140bd3e0 con 0x5587c8777440 2024-03-18T22:55:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.744+0000 7f4c127fc640 1 -- 172.21.15.40:0/3834485400 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4c140c1490 con 0x5587c8777440 2024-03-18T22:55:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.744+0000 7f4c34396740 1 -- 172.21.15.40:0/3834485400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c1c003df0 con 0x5587c8777440 2024-03-18T22:55:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.744+0000 7f4c127fc640 1 -- 172.21.15.40:0/3834485400 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f4c140bcf70 con 0x5587c8777440 2024-03-18T22:55:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.744+0000 7f4c34396740 10 client.18426 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.744+0000 7f4c34396740 1 -- 172.21.15.40:0/3834485400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5587c83ef3f0 con 0x5587c8777440 2024-03-18T22:55:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.744+0000 7f4c127fc640 1 -- 172.21.15.40:0/3834485400 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c1c003df0 con 0x5587c8777440 2024-03-18T22:55:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.745+0000 7f4c127fc640 1 -- 172.21.15.40:0/3834485400 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f4c140c1120 con 0x5587c8777440 2024-03-18T22:55:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.745+0000 7f4c34396740 10 client.18426 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.745+0000 7f4c34396740 10 client.18426 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:55:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.745+0000 7f4c34396740 20 client.18426 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:52.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.745+0000 7f4c34396740 1 --2- 172.21.15.40:0/3834485400 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5587c878cc20 0x5587c878f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.745+0000 7f4c34396740 4 client.18426 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:55:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.745+0000 7f4c34396740 1 -- 172.21.15.40:0/3834485400 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587c86e8a40 con 0x5587c878cc20 2024-03-18T22:55:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.745+0000 7f4c2affd640 1 --2- 172.21.15.40:0/3834485400 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5587c878cc20 0x5587c878f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.745+0000 7f4c127fc640 10 client.18426 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.745+0000 7f4c2affd640 1 --2- 172.21.15.40:0/3834485400 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5587c878cc20 0x5587c878f0e0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91839d68699c67c server_cookie=e4284ded138a89ad in_seq=0 out_seq=0 2024-03-18T22:55:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.746+0000 7f4c127fc640 1 -- 172.21.15.40:0/3834485400 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5587c86e8a40 con 0x5587c878cc20 2024-03-18T22:55:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.746+0000 7f4c127fc640 10 client.18426 handle_command_reply: tid=0 2024-03-18T22:55:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.746+0000 7f4c34396740 1 client.18426 shutdown 2024-03-18T22:55:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.746+0000 7f4c34396740 1 -- 172.21.15.40:0/3834485400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 msgr2=0x5587c8781420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.746+0000 7f4c34396740 1 --2- 172.21.15.40:0/3834485400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 0x5587c8781420 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f4c1402ce10 tx=0x7f4c140035f0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.746+0000 7f4c34396740 1 -- 172.21.15.40:0/3834485400 shutdown_connections 2024-03-18T22:55:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.746+0000 7f4c34396740 1 --2- 172.21.15.40:0/3834485400 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5587c878cc20 0x5587c878f0e0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.746+0000 7f4c34396740 1 --2- 172.21.15.40:0/3834485400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c877c5b0 0x5587c8788460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.746+0000 7f4c127fc640 0 client.18426 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.746+0000 7f4c34396740 1 --2- 172.21.15.40:0/3834485400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 0x5587c8781420 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.746+0000 7f4c34396740 1 --2- 172.21.15.40:0/3834485400 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c8776a50 0x5587c8780ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.746+0000 7f4c34396740 1 -- 172.21.15.40:0/3834485400 >> 172.21.15.40:0/3834485400 conn(0x5587c8771fc0 msgr2=0x5587c87711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.746+0000 7f4c34396740 1 -- 172.21.15.40:0/3834485400 shutdown_connections 2024-03-18T22:55:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.747+0000 7f4c34396740 1 -- 172.21.15.40:0/3834485400 wait complete. 2024-03-18T22:55:52.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.747+0000 7f4c34396740 20 client.18426 trim_cache size 0 max 16384 2024-03-18T22:55:52.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.778+0000 7f4c34396740 1 Processor -- start 2024-03-18T22:55:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.786+0000 7f4c34396740 1 -- start start 2024-03-18T22:55:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.786+0000 7f4c34396740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 0x5587c876cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.786+0000 7f4c34396740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 0x5587c8788420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.786+0000 7f4c34396740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c877c5b0 0x5587c8775930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.786+0000 7f4c34396740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c1b0be0 con 0x5587c8777440 2024-03-18T22:55:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.786+0000 7f4c34396740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c1b2720 con 0x5587c8776a50 2024-03-18T22:55:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.786+0000 7f4c34396740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c1b12b0 con 0x5587c877c5b0 2024-03-18T22:55:52.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.787+0000 7f4c2affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 0x5587c876cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:52.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.787+0000 7f4c2affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 0x5587c876cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:36654/0 (socket says 172.21.15.40:36654) 2024-03-18T22:55:52.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.787+0000 7f4c2affd640 1 -- 172.21.15.40:0/3929709557 learned_addr learned my addr 172.21.15.40:0/3929709557 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:52.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.787+0000 7f4c2affd640 1 -- 172.21.15.40:0/3929709557 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c877c5b0 msgr2=0x5587c8775930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:55:52.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.787+0000 7f4c2affd640 1 --2- 172.21.15.40:0/3929709557 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c877c5b0 0x5587c8775930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.787+0000 7f4c2affd640 1 -- 172.21.15.40:0/3929709557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 msgr2=0x5587c8788420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:55:52.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.787+0000 7f4c2affd640 1 --2- 172.21.15.40:0/3929709557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 0x5587c8788420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.787+0000 7f4c2affd640 1 -- 172.21.15.40:0/3929709557 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587c8778ea0 con 0x5587c8776a50 2024-03-18T22:55:52.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.788+0000 7f4c2affd640 1 --2- 172.21.15.40:0/3929709557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 0x5587c876cf80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4c24005f80 tx=0x7f4c2400b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d360f481b74f143 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:52.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.788+0000 7f4c20ff9640 1 -- 172.21.15.40:0/3929709557 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c24021070 con 0x5587c8776a50 2024-03-18T22:55:52.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.788+0000 7f4c20ff9640 1 -- 172.21.15.40:0/3929709557 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c2401de50 con 0x5587c8776a50 2024-03-18T22:55:52.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.788+0000 7f4c20ff9640 1 -- 172.21.15.40:0/3929709557 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c24005410 con 0x5587c8776a50 2024-03-18T22:55:52.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.790+0000 7f4c34396740 1 -- 172.21.15.40:0/3929709557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 msgr2=0x5587c876cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:52.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.790+0000 7f4c34396740 1 --2- 172.21.15.40:0/3929709557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 0x5587c876cf80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4c24005f80 tx=0x7f4c2400b410 comp rx=0 tx=0).stop 2024-03-18T22:55:52.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.792+0000 7f4c34396740 1 -- 172.21.15.40:0/3929709557 shutdown_connections 2024-03-18T22:55:52.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.792+0000 7f4c34396740 1 --2- 172.21.15.40:0/3929709557 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c877c5b0 0x5587c8775930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.792+0000 7f4c34396740 1 --2- 172.21.15.40:0/3929709557 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 0x5587c8788420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.792+0000 7f4c34396740 1 --2- 172.21.15.40:0/3929709557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 0x5587c876cf80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.792+0000 7f4c34396740 1 -- 172.21.15.40:0/3929709557 >> 172.21.15.40:0/3929709557 conn(0x5587c8771fc0 msgr2=0x5587c8775220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:52.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.793+0000 7f4c34396740 1 -- 172.21.15.40:0/3929709557 shutdown_connections 2024-03-18T22:55:52.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.793+0000 7f4c34396740 1 -- 172.21.15.40:0/3929709557 wait complete. 2024-03-18T22:55:52.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.794+0000 7f4c34396740 1 -- start start 2024-03-18T22:55:52.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.794+0000 7f4c34396740 1 Processor -- start 2024-03-18T22:55:52.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.794+0000 7f4c34396740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 0x5587c87840a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.794+0000 7f4c34396740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 0x5587c87845e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.794+0000 7f4c34396740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c877c5b0 0x5587c8787600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.794+0000 7f4c34396740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c1b2050 con 0x5587c8777440 2024-03-18T22:55:52.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.794+0000 7f4c34396740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c1b1980 con 0x5587c8776a50 2024-03-18T22:55:52.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.794+0000 7f4c34396740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2c1aebc0 con 0x5587c877c5b0 2024-03-18T22:55:52.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.794+0000 7f4c34396740 10 client.0 _ll_register_callbacks cb 0x7f4c14005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:55:52.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.795+0000 7f4c2a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 0x5587c87845e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:52.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.795+0000 7f4c2a7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 0x5587c87845e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42444/0 (socket says 172.21.15.40:42444) 2024-03-18T22:55:52.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.795+0000 7f4c2a7fc640 1 -- 172.21.15.40:0/1370583655 learned_addr learned my addr 172.21.15.40:0/1370583655 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:55:52.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.795+0000 7f4c2a7fc640 1 -- 172.21.15.40:0/1370583655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c877c5b0 msgr2=0x5587c8787600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:55:52.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.795+0000 7f4c2a7fc640 1 --2- 172.21.15.40:0/1370583655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c877c5b0 0x5587c8787600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.795+0000 7f4c2a7fc640 1 -- 172.21.15.40:0/1370583655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 msgr2=0x5587c87840a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:55:52.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.795+0000 7f4c2a7fc640 1 --2- 172.21.15.40:0/1370583655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 0x5587c87840a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.795+0000 7f4c2a7fc640 1 -- 172.21.15.40:0/1370583655 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587c83af0f0 con 0x5587c8777440 2024-03-18T22:55:52.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.795+0000 7f4c2a7fc640 1 --2- 172.21.15.40:0/1370583655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 0x5587c87845e0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f4c140bd310 tx=0x7f4c140bd610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef89201a6919c00 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:55:52.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.796+0000 7f4c127fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:55:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.798+0000 7f4c127fc640 1 -- 172.21.15.40:0/1370583655 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c140bd150 con 0x5587c8777440 2024-03-18T22:55:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.798+0000 7f4c34396740 1 -- 172.21.15.40:0/1370583655 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c2401de50 con 0x5587c8777440 2024-03-18T22:55:52.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.798+0000 7f4c127fc640 1 -- 172.21.15.40:0/1370583655 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4c140f4340 con 0x5587c8777440 2024-03-18T22:55:52.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.805+0000 7f4c127fc640 1 -- 172.21.15.40:0/1370583655 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f4c140bd3e0 con 0x5587c8777440 2024-03-18T22:55:52.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.805+0000 7f4c34396740 10 client.18429 fetch_fsmap learned FSMap version 40 2024-03-18T22:55:52.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.805+0000 7f4c34396740 1 -- 172.21.15.40:0/1370583655 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5587c8804470 con 0x5587c8777440 2024-03-18T22:55:52.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.809+0000 7f4c127fc640 1 -- 172.21.15.40:0/1370583655 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c2401de50 con 0x5587c8777440 2024-03-18T22:55:52.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.809+0000 7f4c127fc640 1 -- 172.21.15.40:0/1370583655 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f4c140c5920 con 0x5587c8777440 2024-03-18T22:55:52.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.809+0000 7f4c34396740 10 client.18429 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:55:52.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.809+0000 7f4c34396740 10 client.18429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:55:52.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.809+0000 7f4c34396740 20 client.18429 populate_metadata read hostname 'smithi040' 2024-03-18T22:55:52.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.809+0000 7f4c34396740 1 --2- 172.21.15.40:0/1370583655 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5587c8804bb0 0x5587c8806fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:55:52.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.809+0000 7f4c34396740 4 client.18429 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:55:52.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.809+0000 7f4c34396740 1 -- 172.21.15.40:0/1370583655 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5587c86e2d90 con 0x5587c8804bb0 2024-03-18T22:55:52.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.810+0000 7f4c2affd640 1 --2- 172.21.15.40:0/1370583655 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5587c8804bb0 0x5587c8806fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:55:52.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.820+0000 7f4c2affd640 1 --2- 172.21.15.40:0/1370583655 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5587c8804bb0 0x5587c8806fd0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb382a755bf933b2 server_cookie=8f4820b9539ac0a3 in_seq=0 out_seq=0 2024-03-18T22:55:52.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.822+0000 7f4c127fc640 10 client.18429 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:52.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.824+0000 7f4c127fc640 1 -- 172.21.15.40:0/1370583655 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5587c86e2d90 con 0x5587c8804bb0 2024-03-18T22:55:52.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.824+0000 7f4c127fc640 10 client.18429 handle_command_reply: tid=0 2024-03-18T22:55:52.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.824+0000 7f4c34396740 1 client.18429 shutdown 2024-03-18T22:55:52.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.825+0000 7f4c34396740 1 -- 172.21.15.40:0/1370583655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 msgr2=0x5587c87845e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:52.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.825+0000 7f4c34396740 1 --2- 172.21.15.40:0/1370583655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 0x5587c87845e0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f4c140bd310 tx=0x7f4c140bd610 comp rx=0 tx=0).stop 2024-03-18T22:55:52.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:52 smithi113 ceph-mon[66775]: pgmap v747: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.8 MiB/s wr, 2.25k op/s 2024-03-18T22:55:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.831+0000 7f4c34396740 1 -- 172.21.15.40:0/1370583655 shutdown_connections 2024-03-18T22:55:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.831+0000 7f4c34396740 1 --2- 172.21.15.40:0/1370583655 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5587c877c5b0 0x5587c8787600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.831+0000 7f4c34396740 1 --2- 172.21.15.40:0/1370583655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5587c8777440 0x5587c87845e0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.831+0000 7f4c34396740 1 --2- 172.21.15.40:0/1370583655 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5587c8804bb0 0x5587c8806fd0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.831+0000 7f4c34396740 1 --2- 172.21.15.40:0/1370583655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587c8776a50 0x5587c87840a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.831+0000 7f4c34396740 1 -- 172.21.15.40:0/1370583655 >> 172.21.15.40:0/1370583655 conn(0x5587c8771fc0 msgr2=0x5587c8775ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:52.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.833+0000 7f4c127fc640 0 client.18429 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:55:52.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.833+0000 7f4c34396740 1 -- 172.21.15.40:0/1370583655 shutdown_connections 2024-03-18T22:55:52.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.833+0000 7f4c34396740 1 -- 172.21.15.40:0/1370583655 wait complete. 2024-03-18T22:55:52.835 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:55:52.835 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:55:52.835 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:55:52.835 INFO:teuthology.orchestra.run.smithi040.stdout: "fb8e8f52-e4b3-4c44-babd-6e7ce44455f5": { 2024-03-18T22:55:52.835 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:55:52.835 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "fb8e8f52-e4b3-4c44-babd-6e7ce44455f5", 2024-03-18T22:55:52.835 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:55:52.835 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:55:52.835 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:55:52.835 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:55:52.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.833+0000 7f4c34396740 20 client.18429 trim_cache size 0 max 16384 2024-03-18T22:55:52.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.841+0000 7f4c316d1640 1 -- 172.21.15.40:0/4156657240 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4c180a2800 msgr2=0x7f4c180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:52.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.841+0000 7f4c316d1640 1 --2- 172.21.15.40:0/4156657240 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4c180a2800 0x7f4c180a4cc0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c01b000 tx=0x7f4c1c00ffd0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.841+0000 7f4c316d1640 1 -- 172.21.15.40:0/4156657240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c2c104c90 msgr2=0x7f4c2c195e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:55:52.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.841+0000 7f4c316d1640 1 --2- 172.21.15.40:0/4156657240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c2c104c90 0x7f4c2c195e40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4c14000c00 tx=0x7f4c140040c0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.841+0000 7f4c316d1640 1 -- 172.21.15.40:0/4156657240 shutdown_connections 2024-03-18T22:55:52.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.841+0000 7f4c316d1640 1 --2- 172.21.15.40:0/4156657240 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f4c180a2800 0x7f4c180a4cc0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.841+0000 7f4c316d1640 1 --2- 172.21.15.40:0/4156657240 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c2c1075c0 0x7f4c2c1a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.841+0000 7f4c316d1640 1 --2- 172.21.15.40:0/4156657240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c2c104c90 0x7f4c2c195e40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.841+0000 7f4c316d1640 1 --2- 172.21.15.40:0/4156657240 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4c2c102360 0x7f4c2c195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:55:52.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.841+0000 7f4c316d1640 1 -- 172.21.15.40:0/4156657240 >> 172.21.15.40:0/4156657240 conn(0x7f4c2c0fbcb0 msgr2=0x7f4c2c101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:55:52.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.845+0000 7f4c316d1640 1 -- 172.21.15.40:0/4156657240 shutdown_connections 2024-03-18T22:55:52.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:55:52.845+0000 7f4c316d1640 1 -- 172.21.15.40:0/4156657240 wait complete. 2024-03-18T22:55:52.870 INFO:tasks.cephfs.filesystem:scrub status for tag:fb8e8f52-e4b3-4c44-babd-6e7ce44455f5 - {'path': '/', 'tag': 'fb8e8f52-e4b3-4c44-babd-6e7ce44455f5', 'options': 'recursive,force'} 2024-03-18T22:55:54.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:54 smithi103 ceph-mon[66377]: pgmap v748: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 10 MiB/s wr, 2.63k op/s 2024-03-18T22:55:54.647 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:54 smithi040 ceph-mon[62643]: pgmap v748: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 10 MiB/s wr, 2.63k op/s 2024-03-18T22:55:54.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:54 smithi113 ceph-mon[66775]: pgmap v748: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 10 MiB/s wr, 2.63k op/s 2024-03-18T22:55:56.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:56 smithi103 ceph-mon[66377]: pgmap v749: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 2.62k op/s 2024-03-18T22:55:56.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:56 smithi040 ceph-mon[62643]: pgmap v749: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 2.62k op/s 2024-03-18T22:55:56.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:56 smithi113 ceph-mon[66775]: pgmap v749: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 2.62k op/s 2024-03-18T22:55:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:55:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:55:57.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:55:57.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:55:57.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:55:57.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:55:57.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:55:57.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:55:57.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:55:58.731 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:55:58 smithi113 ceph-mon[66775]: pgmap v750: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 9.0 MiB/s wr, 2.31k op/s 2024-03-18T22:55:58.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:55:58 smithi103 ceph-mon[66377]: pgmap v750: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 9.0 MiB/s wr, 2.31k op/s 2024-03-18T22:55:58.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:55:58 smithi040 ceph-mon[62643]: pgmap v750: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 9.0 MiB/s wr, 2.31k op/s 2024-03-18T22:56:00.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:00 smithi113 ceph-mon[66775]: pgmap v751: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 11 MiB/s wr, 2.74k op/s 2024-03-18T22:56:00.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:00 smithi103 ceph-mon[66377]: pgmap v751: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 11 MiB/s wr, 2.74k op/s 2024-03-18T22:56:00.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:00 smithi040 ceph-mon[62643]: pgmap v751: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 11 MiB/s wr, 2.74k op/s 2024-03-18T22:56:02.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:02 smithi113 ceph-mon[66775]: pgmap v752: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.5 MiB/s wr, 2.43k op/s 2024-03-18T22:56:02.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:02 smithi103 ceph-mon[66377]: pgmap v752: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.5 MiB/s wr, 2.43k op/s 2024-03-18T22:56:02.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:02 smithi040 ceph-mon[62643]: pgmap v752: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.5 MiB/s wr, 2.43k op/s 2024-03-18T22:56:04.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:04 smithi113 ceph-mon[66775]: pgmap v753: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 11 MiB/s wr, 2.78k op/s 2024-03-18T22:56:04.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:04 smithi103 ceph-mon[66377]: pgmap v753: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 11 MiB/s wr, 2.78k op/s 2024-03-18T22:56:04.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:04 smithi040 ceph-mon[62643]: pgmap v753: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 11 MiB/s wr, 2.78k op/s 2024-03-18T22:56:06.696 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:06 smithi040 ceph-mon[62643]: pgmap v754: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.74k op/s 2024-03-18T22:56:06.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:06 smithi113 ceph-mon[66775]: pgmap v754: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.74k op/s 2024-03-18T22:56:06.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:06 smithi103 ceph-mon[66377]: pgmap v754: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.74k op/s 2024-03-18T22:56:08.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:08 smithi113 ceph-mon[66775]: pgmap v755: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 9.2 MiB/s wr, 2.36k op/s 2024-03-18T22:56:08.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:08 smithi103 ceph-mon[66377]: pgmap v755: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 9.2 MiB/s wr, 2.36k op/s 2024-03-18T22:56:08.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:08 smithi040 ceph-mon[62643]: pgmap v755: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 9.2 MiB/s wr, 2.36k op/s 2024-03-18T22:56:10.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:10 smithi113 ceph-mon[66775]: pgmap v756: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-03-18T22:56:10.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:10 smithi103 ceph-mon[66377]: pgmap v756: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-03-18T22:56:10.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:10 smithi040 ceph-mon[62643]: pgmap v756: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-03-18T22:56:12.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:12 smithi113 ceph-mon[66775]: pgmap v757: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.9 MiB/s wr, 2.52k op/s 2024-03-18T22:56:12.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:12 smithi103 ceph-mon[66377]: pgmap v757: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.9 MiB/s wr, 2.52k op/s 2024-03-18T22:56:12.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:12 smithi040 ceph-mon[62643]: pgmap v757: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.9 MiB/s wr, 2.52k op/s 2024-03-18T22:56:14.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:14 smithi113 ceph-mon[66775]: pgmap v758: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-03-18T22:56:14.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:14 smithi103 ceph-mon[66377]: pgmap v758: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-03-18T22:56:14.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:14 smithi040 ceph-mon[62643]: pgmap v758: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-03-18T22:56:16.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:16 smithi113 ceph-mon[66775]: pgmap v759: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 2.68k op/s 2024-03-18T22:56:16.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:16 smithi103 ceph-mon[66377]: pgmap v759: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 2.68k op/s 2024-03-18T22:56:16.897 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:16 smithi040 ceph-mon[62643]: pgmap v759: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 MiB/s wr, 2.68k op/s 2024-03-18T22:56:18.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:18 smithi113 ceph-mon[66775]: pgmap v760: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 9.0 MiB/s wr, 2.31k op/s 2024-03-18T22:56:18.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:18 smithi103 ceph-mon[66377]: pgmap v760: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 9.0 MiB/s wr, 2.31k op/s 2024-03-18T22:56:18.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:18 smithi040 ceph-mon[62643]: pgmap v760: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 9.0 MiB/s wr, 2.31k op/s 2024-03-18T22:56:20.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:20 smithi113 ceph-mon[66775]: pgmap v761: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 11 MiB/s wr, 2.68k op/s 2024-03-18T22:56:20.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:20 smithi103 ceph-mon[66377]: pgmap v761: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 11 MiB/s wr, 2.68k op/s 2024-03-18T22:56:20.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:20 smithi040 ceph-mon[62643]: pgmap v761: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 11 MiB/s wr, 2.68k op/s 2024-03-18T22:56:22.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:22 smithi113 ceph-mon[66775]: pgmap v762: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.74k op/s 2024-03-18T22:56:22.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:22 smithi103 ceph-mon[66377]: pgmap v762: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.74k op/s 2024-03-18T22:56:22.872 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:56:22.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:56:22.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:22 smithi040 ceph-mon[62643]: pgmap v762: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.74k op/s 2024-03-18T22:56:23.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.532+0000 7fdba96f1640 1 -- 172.21.15.40:0/927787108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdba41098d0 msgr2=0x7fdba4110160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:56:23.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.532+0000 7fdba96f1640 1 --2- 172.21.15.40:0/927787108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdba41098d0 0x7fdba4110160 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009ca0 tx=0x7fdb9c02c120 comp rx=0 tx=0).stop 2024-03-18T22:56:23.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.541+0000 7fdba96f1640 1 -- 172.21.15.40:0/927787108 shutdown_connections 2024-03-18T22:56:23.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.541+0000 7fdba96f1640 1 --2- 172.21.15.40:0/927787108 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdba41098d0 0x7fdba4110160 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.541+0000 7fdba96f1640 1 --2- 172.21.15.40:0/927787108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdba4108f10 0x7fdba4109390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.541+0000 7fdba96f1640 1 --2- 172.21.15.40:0/927787108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdba4107d20 0x7fdba4108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.541+0000 7fdba96f1640 1 -- 172.21.15.40:0/927787108 >> 172.21.15.40:0/927787108 conn(0x7fdba4076cc0 msgr2=0x7fdba4079100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:56:23.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.547+0000 7fdba96f1640 1 -- 172.21.15.40:0/927787108 shutdown_connections 2024-03-18T22:56:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.549+0000 7fdba96f1640 1 -- 172.21.15.40:0/927787108 wait complete. 2024-03-18T22:56:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.550+0000 7fdba96f1640 1 Processor -- start 2024-03-18T22:56:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.550+0000 7fdba96f1640 1 -- start start 2024-03-18T22:56:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.551+0000 7fdba96f1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdba4107d20 0x7fdba407fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.551+0000 7fdba96f1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdba4108f10 0x7fdba4080340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.551+0000 7fdba96f1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdba41098d0 0x7fdba40813c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.551+0000 7fdba96f1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdba407a540 con 0x7fdba4107d20 2024-03-18T22:56:23.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.551+0000 7fdba96f1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdba407a3c0 con 0x7fdba4108f10 2024-03-18T22:56:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.551+0000 7fdba96f1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdba407a6c0 con 0x7fdba41098d0 2024-03-18T22:56:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.551+0000 7fdba37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdba4108f10 0x7fdba4080340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.551+0000 7fdba37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdba4108f10 0x7fdba4080340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46918/0 (socket says 172.21.15.40:46918) 2024-03-18T22:56:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.551+0000 7fdba37fe640 1 -- 172.21.15.40:0/4012122643 learned_addr learned my addr 172.21.15.40:0/4012122643 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:56:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.551+0000 7fdba37fe640 1 -- 172.21.15.40:0/4012122643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdba41098d0 msgr2=0x7fdba40813c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T22:56:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.551+0000 7fdba37fe640 1 --2- 172.21.15.40:0/4012122643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdba41098d0 0x7fdba40813c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.551+0000 7fdba37fe640 1 -- 172.21.15.40:0/4012122643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdba4107d20 msgr2=0x7fdba407fe00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T22:56:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.551+0000 7fdba37fe640 1 --2- 172.21.15.40:0/4012122643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdba4107d20 0x7fdba407fe00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.551+0000 7fdba37fe640 1 -- 172.21.15.40:0/4012122643 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba4081a60 con 0x7fdba4108f10 2024-03-18T22:56:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.552+0000 7fdba37fe640 1 --2- 172.21.15.40:0/4012122643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdba4108f10 0x7fdba4080340 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fdb9800b450 tx=0x7fdb9800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:56:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.553+0000 7fdba17fa640 1 -- 172.21.15.40:0/4012122643 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb98015020 con 0x7fdba4108f10 2024-03-18T22:56:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.553+0000 7fdba96f1640 1 -- 172.21.15.40:0/4012122643 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdba4081d50 con 0x7fdba4108f10 2024-03-18T22:56:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.553+0000 7fdba96f1640 1 -- 172.21.15.40:0/4012122643 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdba40822a0 con 0x7fdba4108f10 2024-03-18T22:56:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.553+0000 7fdba17fa640 1 -- 172.21.15.40:0/4012122643 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb98011d40 con 0x7fdba4108f10 2024-03-18T22:56:23.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.554+0000 7fdba17fa640 1 -- 172.21.15.40:0/4012122643 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb98010e40 con 0x7fdba4108f10 2024-03-18T22:56:23.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.566+0000 7fdbad61c740 1 Processor -- start 2024-03-18T22:56:23.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.566+0000 7fdbad61c740 1 -- start start 2024-03-18T22:56:23.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.566+0000 7fdbad61c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec7ac10 0x55c45ec7b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.567+0000 7fdbad61c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec80100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.567+0000 7fdbad61c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec80770 0x55c45ec82c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.567+0000 7fdbad61c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c45ebdf500 con 0x55c45ec80770 2024-03-18T22:56:23.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.567+0000 7fdbad61c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c45eb87be0 con 0x55c45ec7b600 2024-03-18T22:56:23.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.567+0000 7fdbad61c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c45ebdb5a0 con 0x55c45ec7ac10 2024-03-18T22:56:23.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.567+0000 7fdba8ef0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec7ac10 0x55c45ec7b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:23.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.567+0000 7fdba8ef0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec7ac10 0x55c45ec7b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:39174/0 (socket says 172.21.15.40:39174) 2024-03-18T22:56:23.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.567+0000 7fdba8ef0640 1 -- 172.21.15.40:0/3013127579 learned_addr learned my addr 172.21.15.40:0/3013127579 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:56:23.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.567+0000 7fdba3fff640 1 --2- 172.21.15.40:0/3013127579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec80100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:23.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.567+0000 7fdba17fa640 1 -- 172.21.15.40:0/4012122643 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdb98018020 con 0x7fdba4108f10 2024-03-18T22:56:23.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.568+0000 7fdba37fe640 1 --2- 172.21.15.40:0/3013127579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec80770 0x55c45ec82c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:23.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.568+0000 7fdba17fa640 1 --2- 172.21.15.40:0/4012122643 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb8c0a28e0 0x7fdb8c0a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.569+0000 7fdba17fa640 1 -- 172.21.15.40:0/4012122643 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdb980ce870 con 0x7fdba4108f10 2024-03-18T22:56:23.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.570+0000 7fdba8ef0640 1 -- 172.21.15.40:0/3013127579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 msgr2=0x55c45ec80100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:56:23.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.570+0000 7fdba8ef0640 1 --2- 172.21.15.40:0/3013127579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec80100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.570+0000 7fdba8ef0640 1 -- 172.21.15.40:0/3013127579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec80770 msgr2=0x55c45ec82c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:56:23.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.570+0000 7fdba8ef0640 1 --2- 172.21.15.40:0/3013127579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec80770 0x55c45ec82c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.570+0000 7fdba8ef0640 1 -- 172.21.15.40:0/3013127579 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c45eba9000 con 0x55c45ec7ac10 2024-03-18T22:56:23.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.570+0000 7fdba37fe640 1 --2- 172.21.15.40:0/3013127579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec80770 0x55c45ec82c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T22:56:23.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.570+0000 7fdba3fff640 1 --2- 172.21.15.40:0/3013127579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec80100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:56:23.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.572+0000 7fdba8ef0640 1 --2- 172.21.15.40:0/4012122643 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb8c0a28e0 0x7fdb8c0a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:23.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.572+0000 7fdba8ef0640 1 --2- 172.21.15.40:0/3013127579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec7ac10 0x55c45ec7b030 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c02c6b0 tx=0x7fdb9c005fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fc0e9b9e55e1ffd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:56:23.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.578+0000 7fdb8a7fc640 1 -- 172.21.15.40:0/3013127579 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c065030 con 0x55c45ec7ac10 2024-03-18T22:56:23.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.578+0000 7fdb8a7fc640 1 -- 172.21.15.40:0/3013127579 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb9c05b040 con 0x55c45ec7ac10 2024-03-18T22:56:23.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.578+0000 7fdb8a7fc640 1 -- 172.21.15.40:0/3013127579 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c004040 con 0x55c45ec7ac10 2024-03-18T22:56:23.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.579+0000 7fdbad61c740 1 -- 172.21.15.40:0/3013127579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec7ac10 msgr2=0x55c45ec7b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:56:23.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.579+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3013127579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec7ac10 0x55c45ec7b030 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c02c6b0 tx=0x7fdb9c005fc0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.586+0000 7fdba8ef0640 1 --2- 172.21.15.40:0/4012122643 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb8c0a28e0 0x7fdb8c0a4da0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c069000 tx=0x7fdb9c06a000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:56:23.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.587+0000 7fdbad61c740 1 -- 172.21.15.40:0/3013127579 shutdown_connections 2024-03-18T22:56:23.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.587+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3013127579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec80770 0x55c45ec82c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.587+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3013127579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec80100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.587+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3013127579 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec7ac10 0x55c45ec7b030 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.587+0000 7fdbad61c740 1 -- 172.21.15.40:0/3013127579 >> 172.21.15.40:0/3013127579 conn(0x55c45ec76180 msgr2=0x55c45ec76580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:56:23.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.591+0000 7fdbad61c740 1 -- 172.21.15.40:0/3013127579 shutdown_connections 2024-03-18T22:56:23.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.595+0000 7fdbad61c740 1 -- 172.21.15.40:0/3013127579 wait complete. 2024-03-18T22:56:23.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.595+0000 7fdbad61c740 1 -- start start 2024-03-18T22:56:23.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.595+0000 7fdbad61c740 1 Processor -- start 2024-03-18T22:56:23.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.595+0000 7fdbad61c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec7ac10 0x55c45ec850a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.595+0000 7fdbad61c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec855e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.596+0000 7fdbad61c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec80770 0x55c45ec8c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.596+0000 7fdbad61c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c45ec83720 con 0x55c45ec7ac10 2024-03-18T22:56:23.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.596+0000 7fdbad61c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c45eb9b160 con 0x55c45ec7b600 2024-03-18T22:56:23.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.596+0000 7fdbad61c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c45ec838a0 con 0x55c45ec80770 2024-03-18T22:56:23.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.596+0000 7fdbad61c740 10 client.0 _ll_register_callbacks cb 0x55c45ebd7a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:56:23.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.598+0000 7fdba8ef0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec80770 0x55c45ec8c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:23.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.598+0000 7fdba37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec855e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:23.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.599+0000 7fdba8ef0640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec80770 0x55c45ec8c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:39176/0 (socket says 172.21.15.40:39176) 2024-03-18T22:56:23.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.599+0000 7fdba8ef0640 1 -- 172.21.15.40:0/3460462682 learned_addr learned my addr 172.21.15.40:0/3460462682 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:56:23.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.600+0000 7fdba3fff640 1 --2- 172.21.15.40:0/3460462682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec7ac10 0x55c45ec850a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:23.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.602+0000 7fdba37fe640 1 -- 172.21.15.40:0/3460462682 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec80770 msgr2=0x55c45ec8c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:56:23.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.602+0000 7fdba37fe640 1 --2- 172.21.15.40:0/3460462682 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec80770 0x55c45ec8c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.602+0000 7fdba37fe640 1 -- 172.21.15.40:0/3460462682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec7ac10 msgr2=0x55c45ec850a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:56:23.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.602+0000 7fdba37fe640 1 --2- 172.21.15.40:0/3460462682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec7ac10 0x55c45ec850a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.602+0000 7fdba37fe640 1 -- 172.21.15.40:0/3460462682 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c45e9bd7d0 con 0x55c45ec7b600 2024-03-18T22:56:23.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.605+0000 7fdba37fe640 1 --2- 172.21.15.40:0/3460462682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec855e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdb9809aac0 tx=0x7fdb9809b4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6393e512f6fa9ee1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:56:23.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.605+0000 7fdb837fe640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:56:23.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.607+0000 7fdb837fe640 1 -- 172.21.15.40:0/3460462682 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9809b220 con 0x55c45ec7b600 2024-03-18T22:56:23.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.607+0000 7fdbad61c740 1 -- 172.21.15.40:0/3460462682 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb9c05b040 con 0x55c45ec7b600 2024-03-18T22:56:23.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.607+0000 7fdb837fe640 1 -- 172.21.15.40:0/3460462682 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdb980a30d0 con 0x55c45ec7b600 2024-03-18T22:56:23.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.608+0000 7fdb837fe640 1 -- 172.21.15.40:0/3460462682 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fdb980a3560 con 0x55c45ec7b600 2024-03-18T22:56:23.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.609+0000 7fdbad61c740 10 client.28274 fetch_fsmap learned FSMap version 40 2024-03-18T22:56:23.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.609+0000 7fdbad61c740 1 -- 172.21.15.40:0/3460462682 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c45e8f33f0 con 0x55c45ec7b600 2024-03-18T22:56:23.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.610+0000 7fdb837fe640 1 -- 172.21.15.40:0/3460462682 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb9c05b040 con 0x55c45ec7b600 2024-03-18T22:56:23.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.611+0000 7fdb837fe640 1 -- 172.21.15.40:0/3460462682 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fdb98096b10 con 0x55c45ec7b600 2024-03-18T22:56:23.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.611+0000 7fdbad61c740 10 client.28274 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:56:23.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.611+0000 7fdbad61c740 10 client.28274 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:56:23.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.611+0000 7fdbad61c740 20 client.28274 populate_metadata read hostname 'smithi040' 2024-03-18T22:56:23.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.611+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3460462682 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c45ec91350 0x55c45ec937f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.611+0000 7fdbad61c740 4 client.28274 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:56:23.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.611+0000 7fdbad61c740 1 -- 172.21.15.40:0/3460462682 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c45ebeca40 con 0x55c45ec91350 2024-03-18T22:56:23.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.615+0000 7fdba3fff640 1 --2- 172.21.15.40:0/3460462682 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c45ec91350 0x55c45ec937f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:23.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.617+0000 7fdba3fff640 1 --2- 172.21.15.40:0/3460462682 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c45ec91350 0x55c45ec937f0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e532d1adc20021e3 server_cookie=968ab3617de524a5 in_seq=0 out_seq=0 2024-03-18T22:56:23.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.620+0000 7fdb837fe640 10 client.28274 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:56:23.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.620+0000 7fdb837fe640 1 -- 172.21.15.40:0/3460462682 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c45ebeca40 con 0x55c45ec91350 2024-03-18T22:56:23.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.620+0000 7fdb837fe640 10 client.28274 handle_command_reply: tid=0 2024-03-18T22:56:23.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.623+0000 7fdbad61c740 1 client.28274 shutdown 2024-03-18T22:56:23.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.634+0000 7fdbad61c740 1 -- 172.21.15.40:0/3460462682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 msgr2=0x55c45ec855e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:56:23.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.634+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3460462682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec855e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdb9809aac0 tx=0x7fdb9809b4c0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.634+0000 7fdbad61c740 1 -- 172.21.15.40:0/3460462682 shutdown_connections 2024-03-18T22:56:23.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.634+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3460462682 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec80770 0x55c45ec8c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.634+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3460462682 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c45ec91350 0x55c45ec937f0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.634+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3460462682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec855e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.634+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3460462682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec7ac10 0x55c45ec850a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.634+0000 7fdbad61c740 1 -- 172.21.15.40:0/3460462682 >> 172.21.15.40:0/3460462682 conn(0x55c45ec76180 msgr2=0x55c45ec754a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:56:23.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.636+0000 7fdb837fe640 0 client.28274 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:56:23.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.636+0000 7fdbad61c740 1 -- 172.21.15.40:0/3460462682 shutdown_connections 2024-03-18T22:56:23.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.637+0000 7fdbad61c740 1 -- 172.21.15.40:0/3460462682 wait complete. 2024-03-18T22:56:23.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.637+0000 7fdbad61c740 20 client.28274 trim_cache size 0 max 16384 2024-03-18T22:56:23.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.677+0000 7fdbad61c740 1 Processor -- start 2024-03-18T22:56:23.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.679+0000 7fdbad61c740 1 -- start start 2024-03-18T22:56:23.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.679+0000 7fdbad61c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec7ac10 0x55c45ec7ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.679+0000 7fdbad61c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec7cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.679+0000 7fdbad61c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec80770 0x55c45ec8c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.679+0000 7fdbad61c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9000fe20 con 0x55c45ec80770 2024-03-18T22:56:23.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.679+0000 7fdbad61c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90011960 con 0x55c45ec7b600 2024-03-18T22:56:23.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.679+0000 7fdbad61c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdb900104f0 con 0x55c45ec7ac10 2024-03-18T22:56:23.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.679+0000 7fdba3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec7ac10 0x55c45ec7ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:23.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.679+0000 7fdba3fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec7ac10 0x55c45ec7ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:39192/0 (socket says 172.21.15.40:39192) 2024-03-18T22:56:23.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.679+0000 7fdba3fff640 1 -- 172.21.15.40:0/3622404746 learned_addr learned my addr 172.21.15.40:0/3622404746 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:56:23.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.679+0000 7fdba37fe640 1 --2- 172.21.15.40:0/3622404746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec7cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:23.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.680+0000 7fdba37fe640 1 -- 172.21.15.40:0/3622404746 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec7ac10 msgr2=0x55c45ec7ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:56:23.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.680+0000 7fdba37fe640 1 --2- 172.21.15.40:0/3622404746 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec7ac10 0x55c45ec7ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.680+0000 7fdba37fe640 1 -- 172.21.15.40:0/3622404746 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec80770 msgr2=0x55c45ec8c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:56:23.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.680+0000 7fdba37fe640 1 --2- 172.21.15.40:0/3622404746 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec80770 0x55c45ec8c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.680+0000 7fdba37fe640 1 -- 172.21.15.40:0/3622404746 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c45e8b30f0 con 0x55c45ec7b600 2024-03-18T22:56:23.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.687+0000 7fdba37fe640 1 --2- 172.21.15.40:0/3622404746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec7cfa0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdb98097790 tx=0x7fdb980992d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aedf7086e34156a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:56:23.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.688+0000 7fdb89ffb640 1 -- 172.21.15.40:0/3622404746 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb980a30d0 con 0x55c45ec7b600 2024-03-18T22:56:23.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.688+0000 7fdb89ffb640 1 -- 172.21.15.40:0/3622404746 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb980a3270 con 0x55c45ec7b600 2024-03-18T22:56:23.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.689+0000 7fdbad61c740 1 -- 172.21.15.40:0/3622404746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 msgr2=0x55c45ec7cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:56:23.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.689+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3622404746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec7cfa0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdb98097790 tx=0x7fdb980992d0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.691+0000 7fdbad61c740 1 -- 172.21.15.40:0/3622404746 shutdown_connections 2024-03-18T22:56:23.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.691+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3622404746 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec80770 0x55c45ec8c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.691+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3622404746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec7cfa0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.691+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3622404746 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec7ac10 0x55c45ec7ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.691+0000 7fdbad61c740 1 -- 172.21.15.40:0/3622404746 >> 172.21.15.40:0/3622404746 conn(0x55c45ec76180 msgr2=0x55c45ec850a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:56:23.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.693+0000 7fdbad61c740 1 -- 172.21.15.40:0/3622404746 shutdown_connections 2024-03-18T22:56:23.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.693+0000 7fdbad61c740 1 -- 172.21.15.40:0/3622404746 wait complete. 2024-03-18T22:56:23.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.694+0000 7fdbad61c740 1 -- start start 2024-03-18T22:56:23.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.694+0000 7fdbad61c740 1 Processor -- start 2024-03-18T22:56:23.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.694+0000 7fdbad61c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec7ac10 0x55c45ec89c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.694+0000 7fdbad61c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec8a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.694+0000 7fdbad61c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec80770 0x55c45ec8b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.694+0000 7fdbad61c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90011290 con 0x55c45ec7ac10 2024-03-18T22:56:23.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.694+0000 7fdbad61c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb90010bc0 con 0x55c45ec7b600 2024-03-18T22:56:23.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.694+0000 7fdbad61c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9000de00 con 0x55c45ec80770 2024-03-18T22:56:23.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.694+0000 7fdbad61c740 10 client.0 _ll_register_callbacks cb 0x7fdb98009200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:56:23.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.695+0000 7fdba3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec7ac10 0x55c45ec89c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:23.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.695+0000 7fdba3fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec7ac10 0x55c45ec89c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:53080/0 (socket says 172.21.15.40:53080) 2024-03-18T22:56:23.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.695+0000 7fdba3fff640 1 -- 172.21.15.40:0/3577910230 learned_addr learned my addr 172.21.15.40:0/3577910230 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:56:23.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.697+0000 7fdba8ef0640 1 --2- 172.21.15.40:0/3577910230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec80770 0x55c45ec8b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:23.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.697+0000 7fdba37fe640 1 --2- 172.21.15.40:0/3577910230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec8a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:23.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.698+0000 7fdba3fff640 1 -- 172.21.15.40:0/3577910230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec80770 msgr2=0x55c45ec8b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:56:23.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.698+0000 7fdba3fff640 1 --2- 172.21.15.40:0/3577910230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec80770 0x55c45ec8b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.698+0000 7fdba3fff640 1 -- 172.21.15.40:0/3577910230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 msgr2=0x55c45ec8a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:56:23.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.698+0000 7fdba3fff640 1 --2- 172.21.15.40:0/3577910230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec8a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.698+0000 7fdba3fff640 1 -- 172.21.15.40:0/3577910230 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c45ec8b740 con 0x55c45ec7ac10 2024-03-18T22:56:23.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.702+0000 7fdba3fff640 1 --2- 172.21.15.40:0/3577910230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec7ac10 0x55c45ec89c00 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fdb9400b9c0 tx=0x7fdb9401bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a7a5d5b65485e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:56:23.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.702+0000 7fdb837fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:56:23.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.702+0000 7fdb837fe640 1 -- 172.21.15.40:0/3577910230 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9401b330 con 0x55c45ec7ac10 2024-03-18T22:56:23.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.702+0000 7fdb837fe640 1 -- 172.21.15.40:0/3577910230 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdb9401b4d0 con 0x55c45ec7ac10 2024-03-18T22:56:23.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.703+0000 7fdbad61c740 1 -- 172.21.15.40:0/3577910230 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb980a3270 con 0x55c45ec7ac10 2024-03-18T22:56:23.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.703+0000 7fdb837fe640 1 -- 172.21.15.40:0/3577910230 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fdb94029050 con 0x55c45ec7ac10 2024-03-18T22:56:23.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.704+0000 7fdbad61c740 10 client.18438 fetch_fsmap learned FSMap version 40 2024-03-18T22:56:23.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.704+0000 7fdbad61c740 1 -- 172.21.15.40:0/3577910230 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c45ecf7730 con 0x55c45ec7ac10 2024-03-18T22:56:23.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.709+0000 7fdb837fe640 1 -- 172.21.15.40:0/3577910230 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb980a3270 con 0x55c45ec7ac10 2024-03-18T22:56:23.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.711+0000 7fdb837fe640 1 -- 172.21.15.40:0/3577910230 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fdb94011440 con 0x55c45ec7ac10 2024-03-18T22:56:23.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.711+0000 7fdbad61c740 10 client.18438 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:56:23.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.711+0000 7fdbad61c740 10 client.18438 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:56:23.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.711+0000 7fdbad61c740 20 client.18438 populate_metadata read hostname 'smithi040' 2024-03-18T22:56:23.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.711+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3577910230 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c45ecf80b0 0x55c45ecfa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:23.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.711+0000 7fdbad61c740 4 client.18438 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:56:23.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.711+0000 7fdbad61c740 1 -- 172.21.15.40:0/3577910230 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c45ebe6d90 con 0x55c45ecf80b0 2024-03-18T22:56:23.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.712+0000 7fdba37fe640 1 --2- 172.21.15.40:0/3577910230 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c45ecf80b0 0x55c45ecfa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:23.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.721+0000 7fdba37fe640 1 --2- 172.21.15.40:0/3577910230 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c45ecf80b0 0x55c45ecfa4d0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6d0f48da58ed84c server_cookie=d335747d68134dea in_seq=0 out_seq=0 2024-03-18T22:56:23.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.721+0000 7fdb837fe640 10 client.18438 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:56:23.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.723+0000 7fdb837fe640 1 -- 172.21.15.40:0/3577910230 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c45ebe6d90 con 0x55c45ecf80b0 2024-03-18T22:56:23.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.723+0000 7fdb837fe640 10 client.18438 handle_command_reply: tid=0 2024-03-18T22:56:23.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.723+0000 7fdbad61c740 1 client.18438 shutdown 2024-03-18T22:56:23.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.725+0000 7fdbad61c740 1 -- 172.21.15.40:0/3577910230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec7ac10 msgr2=0x55c45ec89c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:56:23.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.725+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3577910230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec7ac10 0x55c45ec89c00 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fdb9400b9c0 tx=0x7fdb9401bee0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.725+0000 7fdbad61c740 1 -- 172.21.15.40:0/3577910230 shutdown_connections 2024-03-18T22:56:23.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.725+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3577910230 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c45ec80770 0x55c45ec8b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.725+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3577910230 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c45ecf80b0 0x55c45ecfa4d0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.725+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3577910230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c45ec7b600 0x55c45ec8a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.725+0000 7fdbad61c740 1 --2- 172.21.15.40:0/3577910230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c45ec7ac10 0x55c45ec89c00 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.725+0000 7fdbad61c740 1 -- 172.21.15.40:0/3577910230 >> 172.21.15.40:0/3577910230 conn(0x55c45ec76180 msgr2=0x55c45ec85040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:56:23.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.725+0000 7fdb837fe640 0 client.18438 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:56:23.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.734+0000 7fdbad61c740 1 -- 172.21.15.40:0/3577910230 shutdown_connections 2024-03-18T22:56:23.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.735+0000 7fdbad61c740 1 -- 172.21.15.40:0/3577910230 wait complete. 2024-03-18T22:56:23.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.735+0000 7fdbad61c740 20 client.18438 trim_cache size 0 max 16384 2024-03-18T22:56:23.741 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:56:23.741 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:56:23.741 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:56:23.741 INFO:teuthology.orchestra.run.smithi040.stdout: "fb8e8f52-e4b3-4c44-babd-6e7ce44455f5": { 2024-03-18T22:56:23.741 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:56:23.741 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "fb8e8f52-e4b3-4c44-babd-6e7ce44455f5", 2024-03-18T22:56:23.741 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:56:23.741 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:56:23.741 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:56:23.741 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:56:23.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.743+0000 7fdba96f1640 1 -- 172.21.15.40:0/4012122643 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb8c0a28e0 msgr2=0x7fdb8c0a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:56:23.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.743+0000 7fdba96f1640 1 --2- 172.21.15.40:0/4012122643 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb8c0a28e0 0x7fdb8c0a4da0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c069000 tx=0x7fdb9c06a000 comp rx=0 tx=0).stop 2024-03-18T22:56:23.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.743+0000 7fdba96f1640 1 -- 172.21.15.40:0/4012122643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdba4108f10 msgr2=0x7fdba4080340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:56:23.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.743+0000 7fdba96f1640 1 --2- 172.21.15.40:0/4012122643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdba4108f10 0x7fdba4080340 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fdb9800b450 tx=0x7fdb9800b910 comp rx=0 tx=0).stop 2024-03-18T22:56:23.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.749+0000 7fdba96f1640 1 -- 172.21.15.40:0/4012122643 shutdown_connections 2024-03-18T22:56:23.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.750+0000 7fdba96f1640 1 --2- 172.21.15.40:0/4012122643 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb8c0a28e0 0x7fdb8c0a4da0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.750+0000 7fdba96f1640 1 --2- 172.21.15.40:0/4012122643 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdba41098d0 0x7fdba40813c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.750+0000 7fdba96f1640 1 --2- 172.21.15.40:0/4012122643 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdba4108f10 0x7fdba4080340 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.750+0000 7fdba96f1640 1 --2- 172.21.15.40:0/4012122643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdba4107d20 0x7fdba407fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:23.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.750+0000 7fdba96f1640 1 -- 172.21.15.40:0/4012122643 >> 172.21.15.40:0/4012122643 conn(0x7fdba4076cc0 msgr2=0x7fdba410c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:56:23.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.754+0000 7fdba96f1640 1 -- 172.21.15.40:0/4012122643 shutdown_connections 2024-03-18T22:56:23.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:23.754+0000 7fdba96f1640 1 -- 172.21.15.40:0/4012122643 wait complete. 2024-03-18T22:56:23.797 INFO:tasks.cephfs.filesystem:scrub status for tag:fb8e8f52-e4b3-4c44-babd-6e7ce44455f5 - {'path': '/', 'tag': 'fb8e8f52-e4b3-4c44-babd-6e7ce44455f5', 'options': 'recursive,force'} 2024-03-18T22:56:24.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:24 smithi113 ceph-mon[66775]: pgmap v763: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 11 MiB/s wr, 2.84k op/s 2024-03-18T22:56:24.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:24 smithi103 ceph-mon[66377]: pgmap v763: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 11 MiB/s wr, 2.84k op/s 2024-03-18T22:56:24.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:24 smithi040 ceph-mon[62643]: pgmap v763: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 11 MiB/s wr, 2.84k op/s 2024-03-18T22:56:26.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:26 smithi113 ceph-mon[66775]: pgmap v764: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 2.71k op/s 2024-03-18T22:56:26.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:26 smithi103 ceph-mon[66377]: pgmap v764: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 2.71k op/s 2024-03-18T22:56:26.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:26 smithi040 ceph-mon[62643]: pgmap v764: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 2.71k op/s 2024-03-18T22:56:28.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:28 smithi113 ceph-mon[66775]: pgmap v765: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 9.6 MiB/s wr, 2.45k op/s 2024-03-18T22:56:28.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:28 smithi103 ceph-mon[66377]: pgmap v765: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 9.6 MiB/s wr, 2.45k op/s 2024-03-18T22:56:28.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:28 smithi040 ceph-mon[62643]: pgmap v765: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 9.6 MiB/s wr, 2.45k op/s 2024-03-18T22:56:30.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:30 smithi113 ceph-mon[66775]: pgmap v766: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 2.89k op/s 2024-03-18T22:56:30.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:30 smithi103 ceph-mon[66377]: pgmap v766: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 2.89k op/s 2024-03-18T22:56:30.896 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:30 smithi040 ceph-mon[62643]: pgmap v766: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 2.89k op/s 2024-03-18T22:56:32.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:32 smithi113 ceph-mon[66775]: pgmap v767: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.89k op/s 2024-03-18T22:56:32.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:32 smithi103 ceph-mon[66377]: pgmap v767: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.89k op/s 2024-03-18T22:56:32.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:32 smithi040 ceph-mon[62643]: pgmap v767: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.89k op/s 2024-03-18T22:56:34.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:34 smithi113 ceph-mon[66775]: pgmap v768: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 2.91k op/s 2024-03-18T22:56:34.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:34 smithi103 ceph-mon[66377]: pgmap v768: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 2.91k op/s 2024-03-18T22:56:34.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:34 smithi040 ceph-mon[62643]: pgmap v768: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 2.91k op/s 2024-03-18T22:56:36.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:36 smithi113 ceph-mon[66775]: pgmap v769: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-03-18T22:56:36.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:36 smithi103 ceph-mon[66377]: pgmap v769: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-03-18T22:56:36.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:36 smithi040 ceph-mon[62643]: pgmap v769: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-03-18T22:56:38.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:38 smithi113 ceph-mon[66775]: pgmap v770: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 9.7 MiB/s wr, 2.47k op/s 2024-03-18T22:56:38.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:38 smithi103 ceph-mon[66377]: pgmap v770: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 9.7 MiB/s wr, 2.47k op/s 2024-03-18T22:56:38.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:38 smithi040 ceph-mon[62643]: pgmap v770: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 9.7 MiB/s wr, 2.47k op/s 2024-03-18T22:56:40.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:40 smithi103 ceph-mon[66377]: pgmap v771: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 12 MiB/s wr, 2.99k op/s 2024-03-18T22:56:40.897 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:40 smithi040 ceph-mon[62643]: pgmap v771: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 12 MiB/s wr, 2.99k op/s 2024-03-18T22:56:41.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:40 smithi113 ceph-mon[66775]: pgmap v771: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 12 MiB/s wr, 2.99k op/s 2024-03-18T22:56:42.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:42 smithi103 ceph-mon[66377]: pgmap v772: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 11 MiB/s wr, 2.84k op/s 2024-03-18T22:56:42.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:42 smithi040 ceph-mon[62643]: pgmap v772: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 11 MiB/s wr, 2.84k op/s 2024-03-18T22:56:43.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:42 smithi113 ceph-mon[66775]: pgmap v772: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 11 MiB/s wr, 2.84k op/s 2024-03-18T22:56:44.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:44 smithi040 ceph-mon[62643]: pgmap v773: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-03-18T22:56:45.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:44 smithi113 ceph-mon[66775]: pgmap v773: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-03-18T22:56:45.108 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:44 smithi103 ceph-mon[66377]: pgmap v773: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-03-18T22:56:46.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:46 smithi040 ceph-mon[62643]: pgmap v774: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-03-18T22:56:47.081 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:46 smithi113 ceph-mon[66775]: pgmap v774: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-03-18T22:56:47.108 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:46 smithi103 ceph-mon[66377]: pgmap v774: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 2.69k op/s 2024-03-18T22:56:48.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:48 smithi040 ceph-mon[62643]: pgmap v775: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 10 MiB/s wr, 2.67k op/s 2024-03-18T22:56:49.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:48 smithi113 ceph-mon[66775]: pgmap v775: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 10 MiB/s wr, 2.67k op/s 2024-03-18T22:56:49.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:48 smithi103 ceph-mon[66377]: pgmap v775: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 10 MiB/s wr, 2.67k op/s 2024-03-18T22:56:50.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:50 smithi040 ceph-mon[62643]: pgmap v776: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 12 MiB/s wr, 3.08k op/s 2024-03-18T22:56:51.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:50 smithi113 ceph-mon[66775]: pgmap v776: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 12 MiB/s wr, 3.08k op/s 2024-03-18T22:56:51.108 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:50 smithi103 ceph-mon[66377]: pgmap v776: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 12 MiB/s wr, 3.08k op/s 2024-03-18T22:56:52.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:52 smithi040 ceph-mon[62643]: pgmap v777: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.78k op/s 2024-03-18T22:56:53.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:52 smithi113 ceph-mon[66775]: pgmap v777: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.78k op/s 2024-03-18T22:56:53.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:52 smithi103 ceph-mon[66377]: pgmap v777: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.78k op/s 2024-03-18T22:56:53.798 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:56:53.824 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:56:54.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.177+0000 7fe14344b640 1 -- 172.21.15.40:0/537473889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe13c108820 msgr2=0x7fe13c10ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:56:54.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.177+0000 7fe14344b640 1 --2- 172.21.15.40:0/537473889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe13c108820 0x7fe13c10ac50 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009900 tx=0x7fe12c02bc60 comp rx=0 tx=0).stop 2024-03-18T22:56:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.181+0000 7fe14344b640 1 -- 172.21.15.40:0/537473889 shutdown_connections 2024-03-18T22:56:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.181+0000 7fe14344b640 1 --2- 172.21.15.40:0/537473889 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe13c10db40 0x7fe13c10dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.181+0000 7fe14344b640 1 --2- 172.21.15.40:0/537473889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe13c10b190 0x7fe13c10d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.181+0000 7fe14344b640 1 --2- 172.21.15.40:0/537473889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe13c108820 0x7fe13c10ac50 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.181+0000 7fe14344b640 1 -- 172.21.15.40:0/537473889 >> 172.21.15.40:0/537473889 conn(0x7fe13c06e680 msgr2=0x7fe13c070ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:56:54.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.189+0000 7fe14344b640 1 -- 172.21.15.40:0/537473889 shutdown_connections 2024-03-18T22:56:54.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.193+0000 7fe14344b640 1 -- 172.21.15.40:0/537473889 wait complete. 2024-03-18T22:56:54.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.193+0000 7fe14344b640 1 Processor -- start 2024-03-18T22:56:54.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.194+0000 7fe14344b640 1 -- start start 2024-03-18T22:56:54.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.194+0000 7fe14344b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe13c108820 0x7fe13c1a6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.194+0000 7fe14344b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe13c10b190 0x7fe13c1a6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.194+0000 7fe14344b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe13c10db40 0x7fe13c079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.194+0000 7fe14344b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c11c290 con 0x7fe13c10b190 2024-03-18T22:56:54.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.194+0000 7fe14344b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c11c110 con 0x7fe13c10db40 2024-03-18T22:56:54.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.194+0000 7fe14344b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe13c11c410 con 0x7fe13c108820 2024-03-18T22:56:54.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.194+0000 7fe1419e8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe13c10db40 0x7fe13c079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.194+0000 7fe1419e8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe13c10db40 0x7fe13c079a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46248/0 (socket says 172.21.15.40:46248) 2024-03-18T22:56:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.194+0000 7fe1419e8640 1 -- 172.21.15.40:0/3249592248 learned_addr learned my addr 172.21.15.40:0/3249592248 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:56:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.195+0000 7fe1419e8640 1 -- 172.21.15.40:0/3249592248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe13c108820 msgr2=0x7fe13c1a6a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T22:56:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.195+0000 7fe1419e8640 1 --2- 172.21.15.40:0/3249592248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe13c108820 0x7fe13c1a6a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.195+0000 7fe1419e8640 1 -- 172.21.15.40:0/3249592248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe13c10b190 msgr2=0x7fe13c1a6f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T22:56:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.195+0000 7fe1419e8640 1 --2- 172.21.15.40:0/3249592248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe13c10b190 0x7fe13c1a6f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.195+0000 7fe1419e8640 1 -- 172.21.15.40:0/3249592248 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13c07a1b0 con 0x7fe13c10db40 2024-03-18T22:56:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.195+0000 7fe1419e8640 1 --2- 172.21.15.40:0/3249592248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe13c10db40 0x7fe13c079a80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe13400b9e0 tx=0x7fe13400bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:56:54.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.196+0000 7fe1327fc640 1 -- 172.21.15.40:0/3249592248 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe134019070 con 0x7fe13c10db40 2024-03-18T22:56:54.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.196+0000 7fe14344b640 1 -- 172.21.15.40:0/3249592248 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe13c07a4a0 con 0x7fe13c10db40 2024-03-18T22:56:54.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.196+0000 7fe14344b640 1 -- 172.21.15.40:0/3249592248 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe13c07aa70 con 0x7fe13c10db40 2024-03-18T22:56:54.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.200+0000 7fe14603c740 1 Processor -- start 2024-03-18T22:56:54.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.200+0000 7fe1327fc640 1 -- 172.21.15.40:0/3249592248 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe134014440 con 0x7fe13c10db40 2024-03-18T22:56:54.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.200+0000 7fe1327fc640 1 -- 172.21.15.40:0/3249592248 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1340125a0 con 0x7fe13c10db40 2024-03-18T22:56:54.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.203+0000 7fe14603c740 1 -- start start 2024-03-18T22:56:54.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.203+0000 7fe14603c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd0a50 0x55c820fd0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.203+0000 7fe14603c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd1440 0x55c820fd5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.203+0000 7fe14603c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd65b0 0x55c820fd8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.203+0000 7fe14603c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c820f35500 con 0x55c820fd0a50 2024-03-18T22:56:54.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.203+0000 7fe14603c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c820eddbe0 con 0x55c820fd65b0 2024-03-18T22:56:54.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.203+0000 7fe14603c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c820f315a0 con 0x55c820fd1440 2024-03-18T22:56:54.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.204+0000 7fe1409e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd0a50 0x55c820fd0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:54.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.204+0000 7fe1409e6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd0a50 0x55c820fd0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37254/0 (socket says 172.21.15.40:37254) 2024-03-18T22:56:54.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.204+0000 7fe1409e6640 1 -- 172.21.15.40:0/1446542273 learned_addr learned my addr 172.21.15.40:0/1446542273 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:56:54.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.204+0000 7fe1409e6640 1 --2- 172.21.15.40:0/1446542273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd65b0 0x55c820fd8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:54.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.205+0000 7fe1327fc640 1 -- 172.21.15.40:0/3249592248 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe13401c070 con 0x7fe13c10db40 2024-03-18T22:56:54.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.205+0000 7fe1409e6640 1 -- 172.21.15.40:0/1446542273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd1440 msgr2=0x55c820fd5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:56:54.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.206+0000 7fe1411e7640 1 --2- 172.21.15.40:0/1446542273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd1440 0x55c820fd5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:54.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.206+0000 7fe1327fc640 1 --2- 172.21.15.40:0/3249592248 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe1280a2a10 0x7fe1280a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.206+0000 7fe1409e6640 1 --2- 172.21.15.40:0/1446542273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd1440 0x55c820fd5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.206+0000 7fe1409e6640 1 -- 172.21.15.40:0/1446542273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd65b0 msgr2=0x55c820fd8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:56:54.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.206+0000 7fe1409e6640 1 --2- 172.21.15.40:0/1446542273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd65b0 0x55c820fd8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.206+0000 7fe1409e6640 1 -- 172.21.15.40:0/1446542273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c820eff000 con 0x55c820fd0a50 2024-03-18T22:56:54.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.207+0000 7fe1327fc640 1 -- 172.21.15.40:0/3249592248 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe1340ce2f0 con 0x7fe13c10db40 2024-03-18T22:56:54.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.208+0000 7fe1409e6640 1 --2- 172.21.15.40:0/1446542273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd0a50 0x55c820fd0e70 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fe138014390 tx=0x7fe138009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b09250d069a8d014 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:56:54.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.209+0000 7fe1277fe640 1 -- 172.21.15.40:0/1446542273 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13800c070 con 0x55c820fd0a50 2024-03-18T22:56:54.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.209+0000 7fe1277fe640 1 -- 172.21.15.40:0/1446542273 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1380044d0 con 0x55c820fd0a50 2024-03-18T22:56:54.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.209+0000 7fe1277fe640 1 -- 172.21.15.40:0/1446542273 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13800f040 con 0x55c820fd0a50 2024-03-18T22:56:54.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.210+0000 7fe14603c740 1 -- 172.21.15.40:0/1446542273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd0a50 msgr2=0x55c820fd0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:56:54.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.210+0000 7fe14603c740 1 --2- 172.21.15.40:0/1446542273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd0a50 0x55c820fd0e70 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fe138014390 tx=0x7fe138009060 comp rx=0 tx=0).stop 2024-03-18T22:56:54.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.212+0000 7fe1419e8640 1 --2- 172.21.15.40:0/3249592248 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe1280a2a10 0x7fe1280a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:54.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.212+0000 7fe14603c740 1 -- 172.21.15.40:0/1446542273 shutdown_connections 2024-03-18T22:56:54.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.212+0000 7fe14603c740 1 --2- 172.21.15.40:0/1446542273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd65b0 0x55c820fd8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.212+0000 7fe14603c740 1 --2- 172.21.15.40:0/1446542273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd1440 0x55c820fd5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.212+0000 7fe14603c740 1 --2- 172.21.15.40:0/1446542273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd0a50 0x55c820fd0e70 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.212+0000 7fe14603c740 1 -- 172.21.15.40:0/1446542273 >> 172.21.15.40:0/1446542273 conn(0x55c820fcbfc0 msgr2=0x55c820fcc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:56:54.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.212+0000 7fe14603c740 1 -- 172.21.15.40:0/1446542273 shutdown_connections 2024-03-18T22:56:54.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.212+0000 7fe14603c740 1 -- 172.21.15.40:0/1446542273 wait complete. 2024-03-18T22:56:54.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.213+0000 7fe14603c740 1 -- start start 2024-03-18T22:56:54.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.213+0000 7fe14603c740 1 Processor -- start 2024-03-18T22:56:54.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.213+0000 7fe14603c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd0a50 0x55c820fdad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.213+0000 7fe14603c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd1440 0x55c820fdb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.213+0000 7fe14603c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd65b0 0x55c820fe2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.214+0000 7fe1409e6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd0a50 0x55c820fdad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:54.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.214+0000 7fe14603c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c820fd9500 con 0x55c820fd65b0 2024-03-18T22:56:54.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.214+0000 7fe14603c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c820ef1160 con 0x55c820fd1440 2024-03-18T22:56:54.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.214+0000 7fe1409e6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd0a50 0x55c820fdad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:52002/0 (socket says 172.21.15.40:52002) 2024-03-18T22:56:54.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.214+0000 7fe1409e6640 1 -- 172.21.15.40:0/1509353509 learned_addr learned my addr 172.21.15.40:0/1509353509 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:56:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.214+0000 7fe1411e7640 1 --2- 172.21.15.40:0/1509353509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd1440 0x55c820fdb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.214+0000 7fe14603c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c820fd9680 con 0x55c820fd0a50 2024-03-18T22:56:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.214+0000 7fe14603c740 10 client.0 _ll_register_callbacks cb 0x55c820f2da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:56:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.216+0000 7fe1409e6640 1 -- 172.21.15.40:0/1509353509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd1440 msgr2=0x55c820fdb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:56:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.216+0000 7fe1409e6640 1 --2- 172.21.15.40:0/1509353509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd1440 0x55c820fdb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.216+0000 7fe1409e6640 1 -- 172.21.15.40:0/1509353509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd65b0 msgr2=0x55c820fe2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:56:54.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.216+0000 7fe1409e6640 1 --2- 172.21.15.40:0/1509353509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd65b0 0x55c820fe2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.216+0000 7fe1409e6640 1 -- 172.21.15.40:0/1509353509 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c820d137d0 con 0x55c820fd0a50 2024-03-18T22:56:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.216+0000 7fe1419e8640 1 --2- 172.21.15.40:0/3249592248 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe1280a2a10 0x7fe1280a4ed0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe134016000 tx=0x7fe134096ae0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:56:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.218+0000 7fe1409e6640 1 --2- 172.21.15.40:0/1509353509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd0a50 0x55c820fdad00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe138006c60 tx=0x7fe138003f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14c8915b245bec46 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:56:54.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.218+0000 7fe124ff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:56:54.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.220+0000 7fe124ff9640 1 -- 172.21.15.40:0/1509353509 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe138025070 con 0x55c820fd0a50 2024-03-18T22:56:54.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.220+0000 7fe124ff9640 1 -- 172.21.15.40:0/1509353509 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe1380041d0 con 0x55c820fd0a50 2024-03-18T22:56:54.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.220+0000 7fe14603c740 1 -- 172.21.15.40:0/1509353509 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1380044d0 con 0x55c820fd0a50 2024-03-18T22:56:54.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.221+0000 7fe124ff9640 1 -- 172.21.15.40:0/1509353509 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe13802f050 con 0x55c820fd0a50 2024-03-18T22:56:54.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.222+0000 7fe14603c740 10 client.28261 fetch_fsmap learned FSMap version 40 2024-03-18T22:56:54.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.222+0000 7fe14603c740 1 -- 172.21.15.40:0/1509353509 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c820c493f0 con 0x55c820fd0a50 2024-03-18T22:56:54.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.223+0000 7fe124ff9640 1 -- 172.21.15.40:0/1509353509 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe138021910 con 0x55c820fd0a50 2024-03-18T22:56:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.224+0000 7fe124ff9640 1 -- 172.21.15.40:0/1509353509 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fe1380174f0 con 0x55c820fd0a50 2024-03-18T22:56:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.224+0000 7fe14603c740 10 client.28261 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:56:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.224+0000 7fe14603c740 10 client.28261 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:56:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.224+0000 7fe14603c740 20 client.28261 populate_metadata read hostname 'smithi040' 2024-03-18T22:56:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.224+0000 7fe14603c740 1 --2- 172.21.15.40:0/1509353509 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c820ff7870 0x55c820ff9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.224+0000 7fe14603c740 4 client.28261 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:56:54.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.224+0000 7fe14603c740 1 -- 172.21.15.40:0/1509353509 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe1380044d0 con 0x55c820ff7870 2024-03-18T22:56:54.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.225+0000 7fe1411e7640 1 --2- 172.21.15.40:0/1509353509 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c820ff7870 0x55c820ff9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:54.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.235+0000 7fe1411e7640 1 --2- 172.21.15.40:0/1509353509 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c820ff7870 0x55c820ff9c60 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0ea3359dbfc569e server_cookie=8b781b9fc6dcf535 in_seq=0 out_seq=0 2024-03-18T22:56:54.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.236+0000 7fe124ff9640 10 client.28261 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:56:54.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.238+0000 7fe124ff9640 1 -- 172.21.15.40:0/1509353509 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fe1380044d0 con 0x55c820ff7870 2024-03-18T22:56:54.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.238+0000 7fe124ff9640 10 client.28261 handle_command_reply: tid=0 2024-03-18T22:56:54.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.239+0000 7fe14603c740 1 client.28261 shutdown 2024-03-18T22:56:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.243+0000 7fe14603c740 1 -- 172.21.15.40:0/1509353509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd0a50 msgr2=0x55c820fdad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:56:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.243+0000 7fe14603c740 1 --2- 172.21.15.40:0/1509353509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd0a50 0x55c820fdad00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe138006c60 tx=0x7fe138003f60 comp rx=0 tx=0).stop 2024-03-18T22:56:54.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.246+0000 7fe14603c740 1 -- 172.21.15.40:0/1509353509 shutdown_connections 2024-03-18T22:56:54.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.246+0000 7fe14603c740 1 --2- 172.21.15.40:0/1509353509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd65b0 0x55c820fe2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.246+0000 7fe14603c740 1 --2- 172.21.15.40:0/1509353509 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c820ff7870 0x55c820ff9c60 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.246+0000 7fe14603c740 1 --2- 172.21.15.40:0/1509353509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd1440 0x55c820fdb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.246+0000 7fe14603c740 1 --2- 172.21.15.40:0/1509353509 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd0a50 0x55c820fdad00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.246+0000 7fe14603c740 1 -- 172.21.15.40:0/1509353509 >> 172.21.15.40:0/1509353509 conn(0x55c820fcbfc0 msgr2=0x55c820fcb1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:56:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.246+0000 7fe124ff9640 0 client.28261 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:56:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.246+0000 7fe14603c740 1 -- 172.21.15.40:0/1509353509 shutdown_connections 2024-03-18T22:56:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.246+0000 7fe14603c740 1 -- 172.21.15.40:0/1509353509 wait complete. 2024-03-18T22:56:54.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.250+0000 7fe14603c740 20 client.28261 trim_cache size 0 max 16384 2024-03-18T22:56:54.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.301+0000 7fe14603c740 1 Processor -- start 2024-03-18T22:56:54.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.301+0000 7fe14603c740 1 -- start start 2024-03-18T22:56:54.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.301+0000 7fe14603c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd0a50 0x55c820fda910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.301+0000 7fe14603c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd1440 0x55c820fe2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.301+0000 7fe14603c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd65b0 0x55c820fdae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.301+0000 7fe14603c740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe12000fce0 con 0x55c820fd0a50 2024-03-18T22:56:54.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.301+0000 7fe14603c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe120011820 con 0x55c820fd1440 2024-03-18T22:56:54.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.301+0000 7fe14603c740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe1200103b0 con 0x55c820fd65b0 2024-03-18T22:56:54.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.302+0000 7fe1411e7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd1440 0x55c820fe2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:54.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.302+0000 7fe1411e7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd1440 0x55c820fe2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:46268/0 (socket says 172.21.15.40:46268) 2024-03-18T22:56:54.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.302+0000 7fe1411e7640 1 -- 172.21.15.40:0/1941901402 learned_addr learned my addr 172.21.15.40:0/1941901402 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:56:54.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.302+0000 7fe1409e6640 1 --2- 172.21.15.40:0/1941901402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd0a50 0x55c820fda910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:54.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.303+0000 7fe1411e7640 1 -- 172.21.15.40:0/1941901402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd65b0 msgr2=0x55c820fdae50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:56:54.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.303+0000 7fe1411e7640 1 --2- 172.21.15.40:0/1941901402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd65b0 0x55c820fdae50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.303+0000 7fe1411e7640 1 -- 172.21.15.40:0/1941901402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd0a50 msgr2=0x55c820fda910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:56:54.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.303+0000 7fe1411e7640 1 --2- 172.21.15.40:0/1941901402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd0a50 0x55c820fda910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.303+0000 7fe1411e7640 1 -- 172.21.15.40:0/1941901402 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c820c090f0 con 0x55c820fd1440 2024-03-18T22:56:54.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.304+0000 7fe1411e7640 1 --2- 172.21.15.40:0/1941901402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd1440 0x55c820fe2240 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009850 tx=0x7fe12c02c180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8229947ee72f0e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:56:54.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.305+0000 7fe126ffd640 1 -- 172.21.15.40:0/1941901402 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c004490 con 0x55c820fd1440 2024-03-18T22:56:54.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.305+0000 7fe126ffd640 1 -- 172.21.15.40:0/1941901402 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe12c0054f0 con 0x55c820fd1440 2024-03-18T22:56:54.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.305+0000 7fe126ffd640 1 -- 172.21.15.40:0/1941901402 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c004940 con 0x55c820fd1440 2024-03-18T22:56:54.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.313+0000 7fe14603c740 1 -- 172.21.15.40:0/1941901402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd1440 msgr2=0x55c820fe2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:56:54.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.313+0000 7fe14603c740 1 --2- 172.21.15.40:0/1941901402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd1440 0x55c820fe2240 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009850 tx=0x7fe12c02c180 comp rx=0 tx=0).stop 2024-03-18T22:56:54.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.314+0000 7fe14603c740 1 -- 172.21.15.40:0/1941901402 shutdown_connections 2024-03-18T22:56:54.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.314+0000 7fe14603c740 1 --2- 172.21.15.40:0/1941901402 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd65b0 0x55c820fdae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.314+0000 7fe14603c740 1 --2- 172.21.15.40:0/1941901402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd1440 0x55c820fe2240 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.314+0000 7fe14603c740 1 --2- 172.21.15.40:0/1941901402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd0a50 0x55c820fda910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.314+0000 7fe14603c740 1 -- 172.21.15.40:0/1941901402 >> 172.21.15.40:0/1941901402 conn(0x55c820fcbfc0 msgr2=0x55c820fd26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:56:54.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.314+0000 7fe14603c740 1 -- 172.21.15.40:0/1941901402 shutdown_connections 2024-03-18T22:56:54.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.316+0000 7fe14603c740 1 -- 172.21.15.40:0/1941901402 wait complete. 2024-03-18T22:56:54.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.316+0000 7fe14603c740 1 -- start start 2024-03-18T22:56:54.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.317+0000 7fe14603c740 1 Processor -- start 2024-03-18T22:56:54.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.318+0000 7fe14603c740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd0a50 0x55c820fdf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.318+0000 7fe14603c740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd1440 0x55c820fdf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.318+0000 7fe14603c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd65b0 0x55c820fe0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.318+0000 7fe1411e7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd1440 0x55c820fdf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:54.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.318+0000 7fe1411e7640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd1440 0x55c820fdf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37286/0 (socket says 172.21.15.40:37286) 2024-03-18T22:56:54.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.318+0000 7fe1411e7640 1 -- 172.21.15.40:0/2662792722 learned_addr learned my addr 172.21.15.40:0/2662792722 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:56:54.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.318+0000 7fe14603c740 1 -- 172.21.15.40:0/2662792722 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe120011150 con 0x55c820fd1440 2024-03-18T22:56:54.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.318+0000 7fe14603c740 1 -- 172.21.15.40:0/2662792722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe120010a80 con 0x55c820fd65b0 2024-03-18T22:56:54.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.318+0000 7fe14603c740 1 -- 172.21.15.40:0/2662792722 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe12000dd70 con 0x55c820fd0a50 2024-03-18T22:56:54.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.319+0000 7fe14603c740 10 client.0 _ll_register_callbacks cb 0x7fe138009a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:56:54.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.319+0000 7fe1409e6640 1 --2- 172.21.15.40:0/2662792722 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd0a50 0x55c820fdf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.320+0000 7fe1409e6640 1 --2- 172.21.15.40:0/2662792722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd65b0 0x55c820fe0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.320+0000 7fe1411e7640 1 -- 172.21.15.40:0/2662792722 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd0a50 msgr2=0x55c820fdf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:56:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.320+0000 7fe1411e7640 1 --2- 172.21.15.40:0/2662792722 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd0a50 0x55c820fdf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.320+0000 7fe1411e7640 1 -- 172.21.15.40:0/2662792722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd65b0 msgr2=0x55c820fe0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:56:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.320+0000 7fe1411e7640 1 --2- 172.21.15.40:0/2662792722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd65b0 0x55c820fe0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.320+0000 7fe1411e7640 1 -- 172.21.15.40:0/2662792722 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c820fe0f10 con 0x55c820fd1440 2024-03-18T22:56:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.321+0000 7fe1411e7640 1 --2- 172.21.15.40:0/2662792722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd1440 0x55c820fdf910 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fe12c02f020 tx=0x7fe12c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1abe93bb40e9751 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:56:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.321+0000 7fe124ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:56:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.321+0000 7fe124ff9640 1 -- 172.21.15.40:0/2662792722 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c052070 con 0x55c820fd1440 2024-03-18T22:56:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.322+0000 7fe124ff9640 1 -- 172.21.15.40:0/2662792722 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe12c02c810 con 0x55c820fd1440 2024-03-18T22:56:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.322+0000 7fe14603c740 1 -- 172.21.15.40:0/2662792722 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe12c0054f0 con 0x55c820fd1440 2024-03-18T22:56:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.322+0000 7fe124ff9640 1 -- 172.21.15.40:0/2662792722 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe12c02cbe0 con 0x55c820fd1440 2024-03-18T22:56:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.325+0000 7fe14603c740 10 client.18450 fetch_fsmap learned FSMap version 40 2024-03-18T22:56:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.325+0000 7fe14603c740 1 -- 172.21.15.40:0/2662792722 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c820fe2230 con 0x55c820fd1440 2024-03-18T22:56:54.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:54 smithi113 ceph-mon[66775]: pgmap v778: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 12 MiB/s wr, 2.93k op/s 2024-03-18T22:56:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.326+0000 7fe124ff9640 1 -- 172.21.15.40:0/2662792722 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe12c0048e0 con 0x55c820fd1440 2024-03-18T22:56:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.326+0000 7fe124ff9640 1 -- 172.21.15.40:0/2662792722 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fe12c0039b0 con 0x55c820fd1440 2024-03-18T22:56:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.326+0000 7fe14603c740 10 client.18450 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:56:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.326+0000 7fe14603c740 10 client.18450 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:56:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.326+0000 7fe14603c740 20 client.18450 populate_metadata read hostname 'smithi040' 2024-03-18T22:56:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.326+0000 7fe14603c740 1 --2- 172.21.15.40:0/2662792722 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c82105dd40 0x55c821060160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:56:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.326+0000 7fe14603c740 4 client.18450 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:56:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.326+0000 7fe14603c740 1 -- 172.21.15.40:0/2662792722 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fe12c0054f0 con 0x55c82105dd40 2024-03-18T22:56:54.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.337+0000 7fe1409e6640 1 --2- 172.21.15.40:0/2662792722 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c82105dd40 0x55c821060160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:56:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.338+0000 7fe1409e6640 1 --2- 172.21.15.40:0/2662792722 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c82105dd40 0x55c821060160 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e11e2641805a185 server_cookie=b64827ee1d4d1cf6 in_seq=0 out_seq=0 2024-03-18T22:56:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.338+0000 7fe124ff9640 10 client.18450 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:56:54.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.339+0000 7fe124ff9640 1 -- 172.21.15.40:0/2662792722 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7fe12c0054f0 con 0x55c82105dd40 2024-03-18T22:56:54.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.339+0000 7fe124ff9640 10 client.18450 handle_command_reply: tid=0 2024-03-18T22:56:54.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.339+0000 7fe14603c740 1 client.18450 shutdown 2024-03-18T22:56:54.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:54 smithi103 ceph-mon[66377]: pgmap v778: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 12 MiB/s wr, 2.93k op/s 2024-03-18T22:56:54.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.367+0000 7fe14603c740 1 -- 172.21.15.40:0/2662792722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd1440 msgr2=0x55c820fdf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:56:54.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.367+0000 7fe14603c740 1 --2- 172.21.15.40:0/2662792722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd1440 0x55c820fdf910 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fe12c02f020 tx=0x7fe12c0044c0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.370+0000 7fe14603c740 1 -- 172.21.15.40:0/2662792722 shutdown_connections 2024-03-18T22:56:54.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.370+0000 7fe14603c740 1 --2- 172.21.15.40:0/2662792722 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c82105dd40 0x55c821060160 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.370+0000 7fe124ff9640 0 client.18450 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:56:54.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.370+0000 7fe14603c740 1 --2- 172.21.15.40:0/2662792722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c820fd65b0 0x55c820fe0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.370+0000 7fe14603c740 1 --2- 172.21.15.40:0/2662792722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c820fd1440 0x55c820fdf910 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.370+0000 7fe14603c740 1 --2- 172.21.15.40:0/2662792722 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c820fd0a50 0x55c820fdf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.370+0000 7fe14603c740 1 -- 172.21.15.40:0/2662792722 >> 172.21.15.40:0/2662792722 conn(0x55c820fcbfc0 msgr2=0x55c820fda910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:56:54.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.379+0000 7fe14603c740 1 -- 172.21.15.40:0/2662792722 shutdown_connections 2024-03-18T22:56:54.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.379+0000 7fe14603c740 1 -- 172.21.15.40:0/2662792722 wait complete. 2024-03-18T22:56:54.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.379+0000 7fe14603c740 20 client.18450 trim_cache size 0 max 16384 2024-03-18T22:56:54.387 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:56:54.388 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:56:54.388 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:56:54.388 INFO:teuthology.orchestra.run.smithi040.stdout: "fb8e8f52-e4b3-4c44-babd-6e7ce44455f5": { 2024-03-18T22:56:54.388 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:56:54.388 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "fb8e8f52-e4b3-4c44-babd-6e7ce44455f5", 2024-03-18T22:56:54.388 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:56:54.388 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:56:54.388 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:56:54.388 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:56:54.394 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:54 smithi040 ceph-mon[62643]: pgmap v778: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 12 MiB/s wr, 2.93k op/s 2024-03-18T22:56:54.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.401+0000 7fe14344b640 1 -- 172.21.15.40:0/3249592248 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe1280a2a10 msgr2=0x7fe1280a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:56:54.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.401+0000 7fe14344b640 1 --2- 172.21.15.40:0/3249592248 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe1280a2a10 0x7fe1280a4ed0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe134016000 tx=0x7fe134096ae0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.401+0000 7fe14344b640 1 -- 172.21.15.40:0/3249592248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe13c10db40 msgr2=0x7fe13c079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:56:54.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.401+0000 7fe14344b640 1 --2- 172.21.15.40:0/3249592248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe13c10db40 0x7fe13c079a80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe13400b9e0 tx=0x7fe13400bea0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.404+0000 7fe14344b640 1 -- 172.21.15.40:0/3249592248 shutdown_connections 2024-03-18T22:56:54.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.404+0000 7fe14344b640 1 --2- 172.21.15.40:0/3249592248 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe1280a2a10 0x7fe1280a4ed0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.404+0000 7fe14344b640 1 --2- 172.21.15.40:0/3249592248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe13c10db40 0x7fe13c079a80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.404+0000 7fe14344b640 1 --2- 172.21.15.40:0/3249592248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe13c10b190 0x7fe13c1a6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.404+0000 7fe14344b640 1 --2- 172.21.15.40:0/3249592248 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe13c108820 0x7fe13c1a6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:56:54.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.404+0000 7fe14344b640 1 -- 172.21.15.40:0/3249592248 >> 172.21.15.40:0/3249592248 conn(0x7fe13c06e680 msgr2=0x7fe13c10c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:56:54.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.409+0000 7fe14344b640 1 -- 172.21.15.40:0/3249592248 shutdown_connections 2024-03-18T22:56:54.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:56:54.409+0000 7fe14344b640 1 -- 172.21.15.40:0/3249592248 wait complete. 2024-03-18T22:56:54.452 INFO:tasks.cephfs.filesystem:scrub status for tag:fb8e8f52-e4b3-4c44-babd-6e7ce44455f5 - {'path': '/', 'tag': 'fb8e8f52-e4b3-4c44-babd-6e7ce44455f5', 'options': 'recursive,force'} 2024-03-18T22:56:56.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:56 smithi113 ceph-mon[66775]: pgmap v779: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 10 MiB/s wr, 2.67k op/s 2024-03-18T22:56:56.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:56 smithi103 ceph-mon[66377]: pgmap v779: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 10 MiB/s wr, 2.67k op/s 2024-03-18T22:56:56.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:56 smithi040 ceph-mon[62643]: pgmap v779: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 10 MiB/s wr, 2.67k op/s 2024-03-18T22:56:57.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:57 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:56:57.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:57 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:56:57.647 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:57 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:56:58.480 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:56:58 smithi113 ceph-mon[66775]: pgmap v780: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 11 MiB/s wr, 2.82k op/s 2024-03-18T22:56:58.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:56:58 smithi103 ceph-mon[66377]: pgmap v780: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 11 MiB/s wr, 2.82k op/s 2024-03-18T22:56:58.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:56:58 smithi040 ceph-mon[62643]: pgmap v780: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 11 MiB/s wr, 2.82k op/s 2024-03-18T22:57:00.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:00 smithi113 ceph-mon[66775]: pgmap v781: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 12 MiB/s wr, 3.06k op/s 2024-03-18T22:57:00.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:00 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:57:00.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:00 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:57:00.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:00 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:57:00.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:00 smithi103 ceph-mon[66377]: pgmap v781: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 12 MiB/s wr, 3.06k op/s 2024-03-18T22:57:00.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:00 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:57:00.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:00 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:57:00.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:00 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:57:00.647 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:00 smithi040 ceph-mon[62643]: pgmap v781: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 12 MiB/s wr, 3.06k op/s 2024-03-18T22:57:00.647 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:00 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:57:00.647 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:00 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:57:00.647 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:00 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:57:02.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:02 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:02.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:02 smithi113 ceph-mon[66775]: pgmap v782: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.89k op/s 2024-03-18T22:57:02.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:02 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:02.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:02 smithi103 ceph-mon[66377]: pgmap v782: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.89k op/s 2024-03-18T22:57:02.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:02 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:02.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:02 smithi040 ceph-mon[62643]: pgmap v782: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 2.89k op/s 2024-03-18T22:57:04.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:04 smithi113 ceph-mon[66775]: pgmap v783: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-03-18T22:57:04.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:04 smithi103 ceph-mon[66377]: pgmap v783: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-03-18T22:57:04.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:04 smithi040 ceph-mon[62643]: pgmap v783: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 2.98k op/s 2024-03-18T22:57:06.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:06 smithi113 ceph-mon[66775]: pgmap v784: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-03-18T22:57:06.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:06 smithi103 ceph-mon[66377]: pgmap v784: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-03-18T22:57:06.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:06 smithi040 ceph-mon[62643]: pgmap v784: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 2.83k op/s 2024-03-18T22:57:08.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:08 smithi113 ceph-mon[66775]: pgmap v785: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-03-18T22:57:08.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:08 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:57:08.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:08 smithi103 ceph-mon[66377]: pgmap v785: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-03-18T22:57:08.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:08 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:57:08.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:08 smithi040 ceph-mon[62643]: pgmap v785: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 2.72k op/s 2024-03-18T22:57:08.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:08 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:57:09.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:09 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:57:09.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:09 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:57:09.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:09 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:57:09.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:09 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:57:09.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:09 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:57:09.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:09 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:57:09.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:09 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:57:09.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:09 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:57:09.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:09 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:57:09.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:09 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:57:09.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:09 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:57:09.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:09 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:57:10.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:10 smithi113 ceph-mon[66775]: pgmap v786: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 12 MiB/s wr, 2.96k op/s 2024-03-18T22:57:10.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:10 smithi103 ceph-mon[66377]: pgmap v786: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 12 MiB/s wr, 2.96k op/s 2024-03-18T22:57:10.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:10 smithi040 ceph-mon[62643]: pgmap v786: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 12 MiB/s wr, 2.96k op/s 2024-03-18T22:57:12.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:12 smithi113 ceph-mon[66775]: pgmap v787: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 2.42k op/s 2024-03-18T22:57:12.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:12 smithi103 ceph-mon[66377]: pgmap v787: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 2.42k op/s 2024-03-18T22:57:12.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:12 smithi040 ceph-mon[62643]: pgmap v787: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 2.42k op/s 2024-03-18T22:57:14.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:14 smithi113 ceph-mon[66775]: pgmap v788: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 9.2 MiB/s wr, 2.32k op/s 2024-03-18T22:57:14.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:14 smithi103 ceph-mon[66377]: pgmap v788: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 9.2 MiB/s wr, 2.32k op/s 2024-03-18T22:57:14.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:14 smithi040 ceph-mon[62643]: pgmap v788: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 9.2 MiB/s wr, 2.32k op/s 2024-03-18T22:57:16.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:16 smithi113 ceph-mon[66775]: pgmap v789: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 2.16k op/s 2024-03-18T22:57:16.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:16 smithi103 ceph-mon[66377]: pgmap v789: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 2.16k op/s 2024-03-18T22:57:16.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:16 smithi040 ceph-mon[62643]: pgmap v789: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 2.16k op/s 2024-03-18T22:57:18.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:18 smithi113 ceph-mon[66775]: pgmap v790: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.8 MiB/s wr, 1.72k op/s 2024-03-18T22:57:18.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:18 smithi103 ceph-mon[66377]: pgmap v790: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.8 MiB/s wr, 1.72k op/s 2024-03-18T22:57:18.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:18 smithi040 ceph-mon[62643]: pgmap v790: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.8 MiB/s wr, 1.72k op/s 2024-03-18T22:57:20.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:20 smithi113 ceph-mon[66775]: pgmap v791: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 7.4 MiB/s wr, 1.86k op/s 2024-03-18T22:57:20.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:20 smithi103 ceph-mon[66377]: pgmap v791: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 7.4 MiB/s wr, 1.86k op/s 2024-03-18T22:57:20.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:20 smithi040 ceph-mon[62643]: pgmap v791: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 7.4 MiB/s wr, 1.86k op/s 2024-03-18T22:57:22.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:22 smithi113 ceph-mon[66775]: pgmap v792: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.8 MiB/s wr, 1.21k op/s 2024-03-18T22:57:22.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:22 smithi103 ceph-mon[66377]: pgmap v792: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.8 MiB/s wr, 1.21k op/s 2024-03-18T22:57:22.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:22 smithi040 ceph-mon[62643]: pgmap v792: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.8 MiB/s wr, 1.21k op/s 2024-03-18T22:57:24.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:57:24.453 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:57:24.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.578+0000 7f30cf681640 1 -- 172.21.15.40:0/425917535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8100730 msgr2=0x7f30c8100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:24.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.578+0000 7f30cf681640 1 --2- 172.21.15.40:0/425917535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8100730 0x7f30c8100b50 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f30c4009900 tx=0x7f30c402bc30 comp rx=0 tx=0).stop 2024-03-18T22:57:24.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.578+0000 7f30cf681640 1 -- 172.21.15.40:0/425917535 shutdown_connections 2024-03-18T22:57:24.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.578+0000 7f30cf681640 1 --2- 172.21.15.40:0/425917535 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c81022e0 0x7f30c8108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.578+0000 7f30cf681640 1 --2- 172.21.15.40:0/425917535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c8101920 0x7f30c8101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.578+0000 7f30cf681640 1 --2- 172.21.15.40:0/425917535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c8100730 0x7f30c8100b50 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.578+0000 7f30cf681640 1 -- 172.21.15.40:0/425917535 >> 172.21.15.40:0/425917535 conn(0x7f30c80fbcc0 msgr2=0x7f30c80fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:24.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.578+0000 7f30cf681640 1 -- 172.21.15.40:0/425917535 shutdown_connections 2024-03-18T22:57:24.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.578+0000 7f30cf681640 1 -- 172.21.15.40:0/425917535 wait complete. 2024-03-18T22:57:24.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.579+0000 7f30cf681640 1 Processor -- start 2024-03-18T22:57:24.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.579+0000 7f30cf681640 1 -- start start 2024-03-18T22:57:24.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.579+0000 7f30cf681640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c8100730 0x7f30c806a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.579+0000 7f30cf681640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c8101920 0x7f30c819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.579+0000 7f30cf681640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c81022e0 0x7f30c819ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.579+0000 7f30cf681640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f30c810c260 con 0x7f30c81022e0 2024-03-18T22:57:24.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.579+0000 7f30cf681640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f30c810c0e0 con 0x7f30c8101920 2024-03-18T22:57:24.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.579+0000 7f30cf681640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f30c810c3e0 con 0x7f30c8100730 2024-03-18T22:57:24.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.579+0000 7f30cdc1e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c81022e0 0x7f30c819ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.579+0000 7f30cd41d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c8100730 0x7f30c806a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.579+0000 7f30cdc1e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c81022e0 0x7f30c819ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49202/0 (socket says 172.21.15.40:49202) 2024-03-18T22:57:24.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.579+0000 7f30cdc1e640 1 -- 172.21.15.40:0/4285031704 learned_addr learned my addr 172.21.15.40:0/4285031704 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:24.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.579+0000 7f30ccc1c640 1 --2- 172.21.15.40:0/4285031704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c8101920 0x7f30c819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.580+0000 7f30cdc1e640 1 -- 172.21.15.40:0/4285031704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c8100730 msgr2=0x7f30c806a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:24.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.580+0000 7f30cdc1e640 1 --2- 172.21.15.40:0/4285031704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c8100730 0x7f30c806a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.580+0000 7f30cdc1e640 1 -- 172.21.15.40:0/4285031704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c8101920 msgr2=0x7f30c819c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:24.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.580+0000 7f30cdc1e640 1 --2- 172.21.15.40:0/4285031704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c8101920 0x7f30c819c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.580+0000 7f30cdc1e640 1 -- 172.21.15.40:0/4285031704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30c819f710 con 0x7f30c81022e0 2024-03-18T22:57:24.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.580+0000 7f30cd41d640 1 --2- 172.21.15.40:0/4285031704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c8100730 0x7f30c806a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:24.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.580+0000 7f30ccc1c640 1 --2- 172.21.15.40:0/4285031704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c8101920 0x7f30c819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:24.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.580+0000 7f30cdc1e640 1 --2- 172.21.15.40:0/4285031704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c81022e0 0x7f30c819ee00 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f30c000eb10 tx=0x7f30c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:24.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.580+0000 7f30be7fc640 1 -- 172.21.15.40:0/4285031704 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c0004b20 con 0x7f30c81022e0 2024-03-18T22:57:24.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.580+0000 7f30cf681640 1 -- 172.21.15.40:0/4285031704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30c819f9a0 con 0x7f30c81022e0 2024-03-18T22:57:24.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.580+0000 7f30be7fc640 1 -- 172.21.15.40:0/4285031704 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30c0004cc0 con 0x7f30c81022e0 2024-03-18T22:57:24.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.580+0000 7f30be7fc640 1 -- 172.21.15.40:0/4285031704 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c0010690 con 0x7f30c81022e0 2024-03-18T22:57:24.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.580+0000 7f30cf681640 1 -- 172.21.15.40:0/4285031704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30c819fef0 con 0x7f30c81022e0 2024-03-18T22:57:24.589 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:24 smithi113 ceph-mon[66775]: pgmap v793: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.2 MiB/s wr, 1.59k op/s 2024-03-18T22:57:24.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.583+0000 7f30be7fc640 1 -- 172.21.15.40:0/4285031704 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f30c0005020 con 0x7f30c81022e0 2024-03-18T22:57:24.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.583+0000 7f30be7fc640 1 --2- 172.21.15.40:0/4285031704 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f30b40a2800 0x7f30b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30d226b740 1 Processor -- start 2024-03-18T22:57:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30d226b740 1 -- start start 2024-03-18T22:57:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30cd41d640 1 --2- 172.21.15.40:0/4285031704 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f30b40a2800 0x7f30b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30be7fc640 1 -- 172.21.15.40:0/4285031704 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f30c00cf510 con 0x7f30c81022e0 2024-03-18T22:57:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30d226b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7f260 0x55599fa7f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30d226b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa7fc50 0x55599fa84750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30d226b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa84dc0 0x55599fa87250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30d226b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55599f9db500 con 0x55599fa7fc50 2024-03-18T22:57:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30d226b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55599f983be0 con 0x55599fa84dc0 2024-03-18T22:57:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30d226b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55599f9d75a0 con 0x55599fa7f260 2024-03-18T22:57:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30ccc1c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa7fc50 0x55599fa84750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30ccc1c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa7fc50 0x55599fa84750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49212/0 (socket says 172.21.15.40:49212) 2024-03-18T22:57:24.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30ccc1c640 1 -- 172.21.15.40:0/3061006745 learned_addr learned my addr 172.21.15.40:0/3061006745 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30cd41d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7f260 0x55599fa7f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30cdc1e640 1 --2- 172.21.15.40:0/3061006745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa84dc0 0x55599fa87250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30cd41d640 1 --2- 172.21.15.40:0/4285031704 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f30b40a2800 0x7f30b40a4cc0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f30c4005ab0 tx=0x7f30c402cc90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30cd41d640 1 -- 172.21.15.40:0/3061006745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa84dc0 msgr2=0x55599fa87250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30cd41d640 1 --2- 172.21.15.40:0/3061006745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa84dc0 0x55599fa87250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30cd41d640 1 -- 172.21.15.40:0/3061006745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa7fc50 msgr2=0x55599fa84750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30cd41d640 1 --2- 172.21.15.40:0/3061006745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa7fc50 0x55599fa84750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30cd41d640 1 -- 172.21.15.40:0/3061006745 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55599f9a5000 con 0x55599fa7f260 2024-03-18T22:57:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.584+0000 7f30ccc1c640 1 --2- 172.21.15.40:0/3061006745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa7fc50 0x55599fa84750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.585+0000 7f30cd41d640 1 --2- 172.21.15.40:0/3061006745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7f260 0x55599fa7f680 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f30c4036040 tx=0x7f30c4048000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c002698d3cb8ff33 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.585+0000 7f30b37fe640 1 -- 172.21.15.40:0/3061006745 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4005180 con 0x55599fa7f260 2024-03-18T22:57:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.585+0000 7f30b37fe640 1 -- 172.21.15.40:0/3061006745 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30c4051070 con 0x55599fa7f260 2024-03-18T22:57:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.585+0000 7f30b37fe640 1 -- 172.21.15.40:0/3061006745 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4003850 con 0x55599fa7f260 2024-03-18T22:57:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.586+0000 7f30d226b740 1 -- 172.21.15.40:0/3061006745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7f260 msgr2=0x55599fa7f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:24.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.586+0000 7f30d226b740 1 --2- 172.21.15.40:0/3061006745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7f260 0x55599fa7f680 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f30c4036040 tx=0x7f30c4048000 comp rx=0 tx=0).stop 2024-03-18T22:57:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.586+0000 7f30d226b740 1 -- 172.21.15.40:0/3061006745 shutdown_connections 2024-03-18T22:57:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.586+0000 7f30d226b740 1 --2- 172.21.15.40:0/3061006745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa84dc0 0x55599fa87250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.586+0000 7f30d226b740 1 --2- 172.21.15.40:0/3061006745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa7fc50 0x55599fa84750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.586+0000 7f30d226b740 1 --2- 172.21.15.40:0/3061006745 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7f260 0x55599fa7f680 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.586+0000 7f30d226b740 1 -- 172.21.15.40:0/3061006745 >> 172.21.15.40:0/3061006745 conn(0x55599fa71fc0 msgr2=0x55599fa723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.586+0000 7f30d226b740 1 -- 172.21.15.40:0/3061006745 shutdown_connections 2024-03-18T22:57:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.586+0000 7f30d226b740 1 -- 172.21.15.40:0/3061006745 wait complete. 2024-03-18T22:57:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.586+0000 7f30d226b740 1 -- start start 2024-03-18T22:57:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.586+0000 7f30d226b740 1 Processor -- start 2024-03-18T22:57:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.586+0000 7f30d226b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa7f260 0x55599fa896e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.586+0000 7f30d226b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7fc50 0x55599fa89c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30d226b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa84dc0 0x55599fa90c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30d226b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55599fa87d10 con 0x55599fa7f260 2024-03-18T22:57:24.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30d226b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55599f997160 con 0x55599fa84dc0 2024-03-18T22:57:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30d226b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55599fa87e90 con 0x55599fa7fc50 2024-03-18T22:57:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30cd41d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa7f260 0x55599fa896e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30ccc1c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7fc50 0x55599fa89c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30ccc1c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7fc50 0x55599fa89c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:47676/0 (socket says 172.21.15.40:47676) 2024-03-18T22:57:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30cd41d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa7f260 0x55599fa896e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49216/0 (socket says 172.21.15.40:49216) 2024-03-18T22:57:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30ccc1c640 1 -- 172.21.15.40:0/2616537897 learned_addr learned my addr 172.21.15.40:0/2616537897 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30d226b740 10 client.0 _ll_register_callbacks cb 0x55599f9d3a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30cdc1e640 1 --2- 172.21.15.40:0/2616537897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa84dc0 0x55599fa90c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30ccc1c640 1 -- 172.21.15.40:0/2616537897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa84dc0 msgr2=0x55599fa90c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30ccc1c640 1 --2- 172.21.15.40:0/2616537897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa84dc0 0x55599fa90c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30ccc1c640 1 -- 172.21.15.40:0/2616537897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa7f260 msgr2=0x55599fa896e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30ccc1c640 1 --2- 172.21.15.40:0/2616537897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa7f260 0x55599fa896e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30ccc1c640 1 -- 172.21.15.40:0/2616537897 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55599f7b97d0 con 0x55599fa7fc50 2024-03-18T22:57:24.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30cdc1e640 1 --2- 172.21.15.40:0/2616537897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa84dc0 0x55599fa90c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:57:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30cd41d640 1 --2- 172.21.15.40:0/2616537897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa7f260 0x55599fa896e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30ccc1c640 1 --2- 172.21.15.40:0/2616537897 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7fc50 0x55599fa89c20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f30b8012d20 tx=0x7f30b800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbca96467379584a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:57:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30b0ff9640 1 -- 172.21.15.40:0/2616537897 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b80120d0 con 0x55599fa7fc50 2024-03-18T22:57:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30d226b740 1 -- 172.21.15.40:0/2616537897 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30c4051070 con 0x55599fa7fc50 2024-03-18T22:57:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.587+0000 7f30b0ff9640 1 -- 172.21.15.40:0/2616537897 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f30b80047a0 con 0x55599fa7fc50 2024-03-18T22:57:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.588+0000 7f30b0ff9640 1 -- 172.21.15.40:0/2616537897 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f30b8012270 con 0x55599fa7fc50 2024-03-18T22:57:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.588+0000 7f30d226b740 10 client.28276 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.588+0000 7f30d226b740 1 -- 172.21.15.40:0/2616537897 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55599f6ef3f0 con 0x55599fa7fc50 2024-03-18T22:57:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.588+0000 7f30b0ff9640 1 -- 172.21.15.40:0/2616537897 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30c4051070 con 0x55599fa7fc50 2024-03-18T22:57:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.588+0000 7f30b0ff9640 1 -- 172.21.15.40:0/2616537897 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f30b8010550 con 0x55599fa7fc50 2024-03-18T22:57:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.588+0000 7f30d226b740 10 client.28276 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.588+0000 7f30d226b740 10 client.28276 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:57:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.588+0000 7f30d226b740 20 client.28276 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:24.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.588+0000 7f30d226b740 1 --2- 172.21.15.40:0/2616537897 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55599fa95a10 0x55599fa97ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.588+0000 7f30d226b740 4 client.28276 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:57:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.588+0000 7f30d226b740 1 -- 172.21.15.40:0/2616537897 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55599f9e8a40 con 0x55599fa95a10 2024-03-18T22:57:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.588+0000 7f30cd41d640 1 --2- 172.21.15.40:0/2616537897 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55599fa95a10 0x55599fa97ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.589+0000 7f30b0ff9640 10 client.28276 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.589+0000 7f30cd41d640 1 --2- 172.21.15.40:0/2616537897 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55599fa95a10 0x55599fa97ed0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab3f9d700df68f39 server_cookie=55ba236507ec48ba in_seq=0 out_seq=0 2024-03-18T22:57:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.589+0000 7f30b0ff9640 1 -- 172.21.15.40:0/2616537897 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55599f9e8a40 con 0x55599fa95a10 2024-03-18T22:57:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.589+0000 7f30b0ff9640 10 client.28276 handle_command_reply: tid=0 2024-03-18T22:57:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.589+0000 7f30d226b740 1 client.28276 shutdown 2024-03-18T22:57:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.590+0000 7f30d226b740 1 -- 172.21.15.40:0/2616537897 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7fc50 msgr2=0x55599fa89c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.590+0000 7f30d226b740 1 --2- 172.21.15.40:0/2616537897 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7fc50 0x55599fa89c20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f30b8012d20 tx=0x7f30b800f790 comp rx=0 tx=0).stop 2024-03-18T22:57:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.590+0000 7f30d226b740 1 -- 172.21.15.40:0/2616537897 shutdown_connections 2024-03-18T22:57:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.590+0000 7f30d226b740 1 --2- 172.21.15.40:0/2616537897 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55599fa95a10 0x55599fa97ed0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.590+0000 7f30d226b740 1 --2- 172.21.15.40:0/2616537897 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa84dc0 0x55599fa90c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.590+0000 7f30d226b740 1 --2- 172.21.15.40:0/2616537897 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7fc50 0x55599fa89c20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.590+0000 7f30d226b740 1 --2- 172.21.15.40:0/2616537897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa7f260 0x55599fa896e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.590+0000 7f30d226b740 1 -- 172.21.15.40:0/2616537897 >> 172.21.15.40:0/2616537897 conn(0x55599fa71fc0 msgr2=0x55599fa711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.590+0000 7f30b0ff9640 0 client.28276 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.590+0000 7f30d226b740 1 -- 172.21.15.40:0/2616537897 shutdown_connections 2024-03-18T22:57:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.590+0000 7f30d226b740 1 -- 172.21.15.40:0/2616537897 wait complete. 2024-03-18T22:57:24.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.590+0000 7f30d226b740 20 client.28276 trim_cache size 0 max 16384 2024-03-18T22:57:24.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:24 smithi103 ceph-mon[66377]: pgmap v793: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.2 MiB/s wr, 1.59k op/s 2024-03-18T22:57:24.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.611+0000 7f30d226b740 1 Processor -- start 2024-03-18T22:57:24.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.611+0000 7f30d226b740 1 -- start start 2024-03-18T22:57:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.611+0000 7f30d226b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa7f260 0x55599fa723a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.611+0000 7f30d226b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7fc50 0x55599fa90c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.611+0000 7f30d226b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa84dc0 0x55599fa810f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.611+0000 7f30d226b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f30c81a8310 con 0x55599fa84dc0 2024-03-18T22:57:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.611+0000 7f30d226b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f30c81a9e50 con 0x55599fa7f260 2024-03-18T22:57:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.611+0000 7f30d226b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f30c81a89e0 con 0x55599fa7fc50 2024-03-18T22:57:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.611+0000 7f30cdc1e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa84dc0 0x55599fa810f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.611+0000 7f30cdc1e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa84dc0 0x55599fa810f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49228/0 (socket says 172.21.15.40:49228) 2024-03-18T22:57:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.611+0000 7f30cdc1e640 1 -- 172.21.15.40:0/1658573676 learned_addr learned my addr 172.21.15.40:0/1658573676 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.611+0000 7f30ccc1c640 1 --2- 172.21.15.40:0/1658573676 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7fc50 0x55599fa90c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.611+0000 7f30cd41d640 1 --2- 172.21.15.40:0/1658573676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa7f260 0x55599fa723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.612+0000 7f30cdc1e640 1 -- 172.21.15.40:0/1658573676 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7fc50 msgr2=0x55599fa90c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:24.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.612+0000 7f30cdc1e640 1 --2- 172.21.15.40:0/1658573676 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7fc50 0x55599fa90c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.612+0000 7f30cdc1e640 1 -- 172.21.15.40:0/1658573676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa7f260 msgr2=0x55599fa723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.612+0000 7f30cdc1e640 1 --2- 172.21.15.40:0/1658573676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa7f260 0x55599fa723a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.612+0000 7f30cdc1e640 1 -- 172.21.15.40:0/1658573676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55599f6af0f0 con 0x55599fa84dc0 2024-03-18T22:57:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.612+0000 7f30cdc1e640 1 --2- 172.21.15.40:0/1658573676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa84dc0 0x55599fa810f0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f30c009b7c0 tx=0x7f30c0009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7884f88de1a246ea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.612+0000 7f30b2ffd640 1 -- 172.21.15.40:0/1658573676 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c00b3070 con 0x55599fa84dc0 2024-03-18T22:57:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.612+0000 7f30b2ffd640 1 -- 172.21.15.40:0/1658573676 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30c009bac0 con 0x55599fa84dc0 2024-03-18T22:57:24.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.612+0000 7f30b2ffd640 1 -- 172.21.15.40:0/1658573676 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c00cf160 con 0x55599fa84dc0 2024-03-18T22:57:24.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.613+0000 7f30d226b740 1 -- 172.21.15.40:0/1658573676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa84dc0 msgr2=0x55599fa810f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.613+0000 7f30d226b740 1 --2- 172.21.15.40:0/1658573676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa84dc0 0x55599fa810f0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f30c009b7c0 tx=0x7f30c0009a70 comp rx=0 tx=0).stop 2024-03-18T22:57:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.613+0000 7f30d226b740 1 -- 172.21.15.40:0/1658573676 shutdown_connections 2024-03-18T22:57:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.613+0000 7f30d226b740 1 --2- 172.21.15.40:0/1658573676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa84dc0 0x55599fa810f0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.613+0000 7f30d226b740 1 --2- 172.21.15.40:0/1658573676 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7fc50 0x55599fa90c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.613+0000 7f30d226b740 1 --2- 172.21.15.40:0/1658573676 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa7f260 0x55599fa723a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.613+0000 7f30d226b740 1 -- 172.21.15.40:0/1658573676 >> 172.21.15.40:0/1658573676 conn(0x55599fa71fc0 msgr2=0x55599fa896e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.613+0000 7f30d226b740 1 -- 172.21.15.40:0/1658573676 shutdown_connections 2024-03-18T22:57:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.613+0000 7f30d226b740 1 -- 172.21.15.40:0/1658573676 wait complete. 2024-03-18T22:57:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.613+0000 7f30d226b740 1 -- start start 2024-03-18T22:57:24.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.613+0000 7f30d226b740 1 Processor -- start 2024-03-18T22:57:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30d226b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7f260 0x55599fa8de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30d226b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa7fc50 0x55599fa8e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30d226b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa84dc0 0x55599fa8f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30d226b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f30c81a9780 con 0x55599fa84dc0 2024-03-18T22:57:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30d226b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f30c81a90b0 con 0x55599fa7fc50 2024-03-18T22:57:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30d226b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f30c81a62f0 con 0x55599fa7f260 2024-03-18T22:57:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30d226b740 10 client.0 _ll_register_callbacks cb 0x7f30b8009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30cdc1e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa84dc0 0x55599fa8f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30ccc1c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa7fc50 0x55599fa8e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30cdc1e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa84dc0 0x55599fa8f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49230/0 (socket says 172.21.15.40:49230) 2024-03-18T22:57:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30ccc1c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa7fc50 0x55599fa8e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:38048/0 (socket says 172.21.15.40:38048) 2024-03-18T22:57:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30cdc1e640 1 -- 172.21.15.40:0/1549532176 learned_addr learned my addr 172.21.15.40:0/1549532176 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:24.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30cd41d640 1 --2- 172.21.15.40:0/1549532176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7f260 0x55599fa8de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30cdc1e640 1 -- 172.21.15.40:0/1549532176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7f260 msgr2=0x55599fa8de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:24.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30cdc1e640 1 --2- 172.21.15.40:0/1549532176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7f260 0x55599fa8de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30cdc1e640 1 -- 172.21.15.40:0/1549532176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa7fc50 msgr2=0x55599fa8e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:24.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30cdc1e640 1 --2- 172.21.15.40:0/1549532176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa7fc50 0x55599fa8e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30cdc1e640 1 -- 172.21.15.40:0/1549532176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55599fa8f900 con 0x55599fa84dc0 2024-03-18T22:57:24.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30cdc1e640 1 --2- 172.21.15.40:0/1549532176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa84dc0 0x55599fa8f310 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f30c0098490 tx=0x7f30c0004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c9912f7b8f4d316 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:24.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.614+0000 7f30b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:57:24.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.615+0000 7f30b0ff9640 1 -- 172.21.15.40:0/1549532176 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c00b3070 con 0x55599fa84dc0 2024-03-18T22:57:24.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.615+0000 7f30b0ff9640 1 -- 172.21.15.40:0/1549532176 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f30c00cf160 con 0x55599fa84dc0 2024-03-18T22:57:24.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.615+0000 7f30d226b740 1 -- 172.21.15.40:0/1549532176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30c009bac0 con 0x55599fa84dc0 2024-03-18T22:57:24.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.615+0000 7f30b0ff9640 1 -- 172.21.15.40:0/1549532176 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f30c00cf360 con 0x55599fa84dc0 2024-03-18T22:57:24.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.615+0000 7f30d226b740 10 client.18468 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:24.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.615+0000 7f30d226b740 1 -- 172.21.15.40:0/1549532176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55599fa90c10 con 0x55599fa84dc0 2024-03-18T22:57:24.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.615+0000 7f30b0ff9640 1 -- 172.21.15.40:0/1549532176 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30c009bac0 con 0x55599fa84dc0 2024-03-18T22:57:24.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.615+0000 7f30b0ff9640 1 -- 172.21.15.40:0/1549532176 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f30c0097810 con 0x55599fa84dc0 2024-03-18T22:57:24.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.615+0000 7f30d226b740 10 client.18468 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:24.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.615+0000 7f30d226b740 10 client.18468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:57:24.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.615+0000 7f30d226b740 20 client.18468 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:24.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.615+0000 7f30d226b740 1 --2- 172.21.15.40:0/1549532176 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55599fafc270 0x55599fafe690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.615+0000 7f30d226b740 4 client.18468 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:57:24.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.615+0000 7f30d226b740 1 -- 172.21.15.40:0/1549532176 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55599f9e2d90 con 0x55599fafc270 2024-03-18T22:57:24.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.615+0000 7f30cd41d640 1 --2- 172.21.15.40:0/1549532176 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55599fafc270 0x55599fafe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30b0ff9640 10 client.18468 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:24.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30cd41d640 1 --2- 172.21.15.40:0/1549532176 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55599fafc270 0x55599fafe690 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0e6f23550b72d35 server_cookie=6837ca42aca231de in_seq=0 out_seq=0 2024-03-18T22:57:24.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30b0ff9640 1 -- 172.21.15.40:0/1549532176 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55599f9e2d90 con 0x55599fafc270 2024-03-18T22:57:24.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30b0ff9640 10 client.18468 handle_command_reply: tid=0 2024-03-18T22:57:24.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30d226b740 1 client.18468 shutdown 2024-03-18T22:57:24.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30d226b740 1 -- 172.21.15.40:0/1549532176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa84dc0 msgr2=0x55599fa8f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:24.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30d226b740 1 --2- 172.21.15.40:0/1549532176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa84dc0 0x55599fa8f310 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f30c0098490 tx=0x7f30c0004f20 comp rx=0 tx=0).stop 2024-03-18T22:57:24.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30d226b740 1 -- 172.21.15.40:0/1549532176 shutdown_connections 2024-03-18T22:57:24.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30d226b740 1 --2- 172.21.15.40:0/1549532176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55599fa84dc0 0x55599fa8f310 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30d226b740 1 --2- 172.21.15.40:0/1549532176 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55599fafc270 0x55599fafe690 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30d226b740 1 --2- 172.21.15.40:0/1549532176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55599fa7fc50 0x55599fa8e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30d226b740 1 --2- 172.21.15.40:0/1549532176 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55599fa7f260 0x55599fa8de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30b0ff9640 0 client.18468 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:24.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30d226b740 1 -- 172.21.15.40:0/1549532176 >> 172.21.15.40:0/1549532176 conn(0x55599fa71fc0 msgr2=0x55599fa810f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:24.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30d226b740 1 -- 172.21.15.40:0/1549532176 shutdown_connections 2024-03-18T22:57:24.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30d226b740 1 -- 172.21.15.40:0/1549532176 wait complete. 2024-03-18T22:57:24.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.616+0000 7f30d226b740 20 client.18468 trim_cache size 0 max 16384 2024-03-18T22:57:24.624 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:57:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:57:24.624 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:57:24.624 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:57:24.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.617+0000 7f30cf681640 1 -- 172.21.15.40:0/4285031704 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f30b40a2800 msgr2=0x7f30b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:24.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.617+0000 7f30cf681640 1 --2- 172.21.15.40:0/4285031704 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f30b40a2800 0x7f30b40a4cc0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f30c4005ab0 tx=0x7f30c402cc90 comp rx=0 tx=0).stop 2024-03-18T22:57:24.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.617+0000 7f30cf681640 1 -- 172.21.15.40:0/4285031704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c81022e0 msgr2=0x7f30c819ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:24.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.617+0000 7f30cf681640 1 --2- 172.21.15.40:0/4285031704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c81022e0 0x7f30c819ee00 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f30c000eb10 tx=0x7f30c000c2d0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.617+0000 7f30cf681640 1 -- 172.21.15.40:0/4285031704 shutdown_connections 2024-03-18T22:57:24.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.617+0000 7f30cf681640 1 --2- 172.21.15.40:0/4285031704 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f30b40a2800 0x7f30b40a4cc0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.617+0000 7f30cf681640 1 --2- 172.21.15.40:0/4285031704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f30c81022e0 0x7f30c819ee00 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.617+0000 7f30cf681640 1 --2- 172.21.15.40:0/4285031704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30c8101920 0x7f30c819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.617+0000 7f30cf681640 1 --2- 172.21.15.40:0/4285031704 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f30c8100730 0x7f30c806a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.617+0000 7f30cf681640 1 -- 172.21.15.40:0/4285031704 >> 172.21.15.40:0/4285031704 conn(0x7f30c80fbcc0 msgr2=0x7f30c8104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:24.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.617+0000 7f30cf681640 1 -- 172.21.15.40:0/4285031704 shutdown_connections 2024-03-18T22:57:24.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.617+0000 7f30cf681640 1 -- 172.21.15.40:0/4285031704 wait complete. 2024-03-18T22:57:24.627 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:24 smithi040 ceph-mon[62643]: pgmap v793: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.2 MiB/s wr, 1.59k op/s 2024-03-18T22:57:24.628 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:57:24.628 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:57:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.741+0000 7fb908710640 1 -- 172.21.15.40:0/85351161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9001075a0 msgr2=0x7fb900109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.741+0000 7fb908710640 1 --2- 172.21.15.40:0/85351161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9001075a0 0x7fb900109990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8009ca0 tx=0x7fb8f802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.741+0000 7fb908710640 1 -- 172.21.15.40:0/85351161 shutdown_connections 2024-03-18T22:57:24.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.741+0000 7fb908710640 1 --2- 172.21.15.40:0/85351161 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9001075a0 0x7fb900109990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.741+0000 7fb908710640 1 --2- 172.21.15.40:0/85351161 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb900104c70 0x7fb900107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.741+0000 7fb908710640 1 --2- 172.21.15.40:0/85351161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb900102340 0x7fb900104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.741+0000 7fb908710640 1 -- 172.21.15.40:0/85351161 >> 172.21.15.40:0/85351161 conn(0x7fb9000fbcb0 msgr2=0x7fb9000fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:24.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.741+0000 7fb908710640 1 -- 172.21.15.40:0/85351161 shutdown_connections 2024-03-18T22:57:24.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.741+0000 7fb908710640 1 -- 172.21.15.40:0/85351161 wait complete. 2024-03-18T22:57:24.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb908710640 1 Processor -- start 2024-03-18T22:57:24.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb908710640 1 -- start start 2024-03-18T22:57:24.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb908710640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb900102340 0x7fb900197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb908710640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb900104c70 0x7fb900197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb908710640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9001075a0 0x7fb90019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb908710640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb90010d370 con 0x7fb900102340 2024-03-18T22:57:24.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb908710640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb90010d1f0 con 0x7fb9001075a0 2024-03-18T22:57:24.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb908710640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb90010d4f0 con 0x7fb900104c70 2024-03-18T22:57:24.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb9064ac640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb900102340 0x7fb900197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb905cab640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb900104c70 0x7fb900197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb905cab640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb900104c70 0x7fb900197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:47700/0 (socket says 172.21.15.40:47700) 2024-03-18T22:57:24.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb9064ac640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb900102340 0x7fb900197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49246/0 (socket says 172.21.15.40:49246) 2024-03-18T22:57:24.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb905cab640 1 -- 172.21.15.40:0/3616912151 learned_addr learned my addr 172.21.15.40:0/3616912151 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:24.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb906cad640 1 --2- 172.21.15.40:0/3616912151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9001075a0 0x7fb90019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb905cab640 1 -- 172.21.15.40:0/3616912151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9001075a0 msgr2=0x7fb90019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:24.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb905cab640 1 --2- 172.21.15.40:0/3616912151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9001075a0 0x7fb90019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb905cab640 1 -- 172.21.15.40:0/3616912151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb900102340 msgr2=0x7fb900197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:24.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb905cab640 1 --2- 172.21.15.40:0/3616912151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb900102340 0x7fb900197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:24.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb905cab640 1 -- 172.21.15.40:0/3616912151 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90019f660 con 0x7fb900104c70 2024-03-18T22:57:24.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.742+0000 7fb9064ac640 1 --2- 172.21.15.40:0/3616912151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb900102340 0x7fb900197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:24.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.743+0000 7fb905cab640 1 --2- 172.21.15.40:0/3616912151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb900104c70 0x7fb900197f80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb8f000e9c0 tx=0x7fb8f000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:24.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.743+0000 7fb8f77fe640 1 -- 172.21.15.40:0/3616912151 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f000cd80 con 0x7fb900104c70 2024-03-18T22:57:24.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.743+0000 7fb8f77fe640 1 -- 172.21.15.40:0/3616912151 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8f00054d0 con 0x7fb900104c70 2024-03-18T22:57:24.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.743+0000 7fb908710640 1 -- 172.21.15.40:0/3616912151 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb90019f950 con 0x7fb900104c70 2024-03-18T22:57:24.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.743+0000 7fb8f77fe640 1 -- 172.21.15.40:0/3616912151 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f0010430 con 0x7fb900104c70 2024-03-18T22:57:24.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.743+0000 7fb908710640 1 -- 172.21.15.40:0/3616912151 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb90019fea0 con 0x7fb900104c70 2024-03-18T22:57:24.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.744+0000 7fb908710640 1 -- 172.21.15.40:0/3616912151 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9000fd7a0 con 0x7fb900104c70 2024-03-18T22:57:24.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.745+0000 7fb8f77fe640 1 -- 172.21.15.40:0/3616912151 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb8f0004fd0 con 0x7fb900104c70 2024-03-18T22:57:24.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.746+0000 7fb8f77fe640 1 --2- 172.21.15.40:0/3616912151 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb8ec0a2800 0x7fb8ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:24.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.746+0000 7fb8f77fe640 1 -- 172.21.15.40:0/3616912151 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb8f00cf220 con 0x7fb900104c70 2024-03-18T22:57:24.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.746+0000 7fb9064ac640 1 --2- 172.21.15.40:0/3616912151 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb8ec0a2800 0x7fb8ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:24.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.746+0000 7fb9064ac640 1 --2- 172.21.15.40:0/3616912151 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb8ec0a2800 0x7fb8ec0a4cc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009fc0 tx=0x7fb8fc005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:24.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:24.750+0000 7fb8f77fe640 1 -- 172.21.15.40:0/3616912151 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb8f009bed0 con 0x7fb900104c70 2024-03-18T22:57:25.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.074+0000 7fb908710640 1 -- 172.21.15.40:0/3616912151 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb900062420 con 0x7fb900104c70 2024-03-18T22:57:25.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.075+0000 7fb8f77fe640 1 -- 172.21.15.40:0/3616912151 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11947 (secure 0 0 0) 0x7fb8f009b7d0 con 0x7fb900104c70 2024-03-18T22:57:25.077 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:57:25.077 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:54:31.450044+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24388,24401,24473,24481,24509]},"id":1}]} 2024-03-18T22:57:25.077 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 40 2024-03-18T22:57:25.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.078+0000 7fb908710640 1 -- 172.21.15.40:0/3616912151 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb8ec0a2800 msgr2=0x7fb8ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.078+0000 7fb908710640 1 --2- 172.21.15.40:0/3616912151 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb8ec0a2800 0x7fb8ec0a4cc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009fc0 tx=0x7fb8fc005f90 comp rx=0 tx=0).stop 2024-03-18T22:57:25.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.078+0000 7fb908710640 1 -- 172.21.15.40:0/3616912151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb900104c70 msgr2=0x7fb900197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.078+0000 7fb908710640 1 --2- 172.21.15.40:0/3616912151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb900104c70 0x7fb900197f80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb8f000e9c0 tx=0x7fb8f000ee80 comp rx=0 tx=0).stop 2024-03-18T22:57:25.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.078+0000 7fb908710640 1 -- 172.21.15.40:0/3616912151 shutdown_connections 2024-03-18T22:57:25.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.078+0000 7fb908710640 1 --2- 172.21.15.40:0/3616912151 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb8ec0a2800 0x7fb8ec0a4cc0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.078+0000 7fb908710640 1 --2- 172.21.15.40:0/3616912151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb9001075a0 0x7fb90019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.078+0000 7fb908710640 1 --2- 172.21.15.40:0/3616912151 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb900104c70 0x7fb900197f80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.078+0000 7fb908710640 1 --2- 172.21.15.40:0/3616912151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb900102340 0x7fb900197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.078+0000 7fb908710640 1 -- 172.21.15.40:0/3616912151 >> 172.21.15.40:0/3616912151 conn(0x7fb9000fbcb0 msgr2=0x7fb900102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.078+0000 7fb908710640 1 -- 172.21.15.40:0/3616912151 shutdown_connections 2024-03-18T22:57:25.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.078+0000 7fb908710640 1 -- 172.21.15.40:0/3616912151 wait complete. 2024-03-18T22:57:25.090 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:57:25.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.198+0000 7f778ca31640 1 -- 172.21.15.40:0/3332144116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7788101d00 msgr2=0x7f7788102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.198+0000 7f778ca31640 1 --2- 172.21.15.40:0/3332144116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7788101d00 0x7f7788102120 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f7780009ca0 tx=0x7f778002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.198+0000 7f778ca31640 1 -- 172.21.15.40:0/3332144116 shutdown_connections 2024-03-18T22:57:25.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.198+0000 7f778ca31640 1 --2- 172.21.15.40:0/3332144116 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f77881013f0 0x7f7788108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.198+0000 7f778ca31640 1 --2- 172.21.15.40:0/3332144116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7788100a30 0x7f7788100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.198+0000 7f778ca31640 1 --2- 172.21.15.40:0/3332144116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7788101d00 0x7f7788102120 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.198+0000 7f778ca31640 1 -- 172.21.15.40:0/3332144116 >> 172.21.15.40:0/3332144116 conn(0x7f77880fb460 msgr2=0x7f77880fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.198+0000 7f778ca31640 1 -- 172.21.15.40:0/3332144116 shutdown_connections 2024-03-18T22:57:25.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.198+0000 7f778ca31640 1 -- 172.21.15.40:0/3332144116 wait complete. 2024-03-18T22:57:25.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778ca31640 1 Processor -- start 2024-03-18T22:57:25.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778ca31640 1 -- start start 2024-03-18T22:57:25.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778ca31640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7788100a30 0x7f7788100490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778ca31640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77881013f0 0x7f778819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778ca31640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7788101d00 0x7f778819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778ca31640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f778810c280 con 0x7f7788101d00 2024-03-18T22:57:25.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778ca31640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f778810c100 con 0x7f77881013f0 2024-03-18T22:57:25.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778ca31640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f778810c400 con 0x7f7788100a30 2024-03-18T22:57:25.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7788100a30 0x7f7788100490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7788100a30 0x7f7788100490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:47726/0 (socket says 172.21.15.40:47726) 2024-03-18T22:57:25.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f7786d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7788101d00 0x7f778819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778659c640 1 -- 172.21.15.40:0/4041120589 learned_addr learned my addr 172.21.15.40:0/4041120589 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f7785d9b640 1 --2- 172.21.15.40:0/4041120589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77881013f0 0x7f778819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778659c640 1 -- 172.21.15.40:0/4041120589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77881013f0 msgr2=0x7f778819c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778659c640 1 --2- 172.21.15.40:0/4041120589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77881013f0 0x7f778819c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778659c640 1 -- 172.21.15.40:0/4041120589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7788101d00 msgr2=0x7f778819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778659c640 1 --2- 172.21.15.40:0/4041120589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7788101d00 0x7f778819edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778659c640 1 -- 172.21.15.40:0/4041120589 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f778819f6e0 con 0x7f7788100a30 2024-03-18T22:57:25.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f7786d9d640 1 --2- 172.21.15.40:0/4041120589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7788101d00 0x7f778819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:25.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.199+0000 7f778659c640 1 --2- 172.21.15.40:0/4041120589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7788100a30 0x7f7788100490 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7780032380 tx=0x7f7780032820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.200+0000 7f777f7fe640 1 -- 172.21.15.40:0/4041120589 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7780041020 con 0x7f7788100a30 2024-03-18T22:57:25.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.200+0000 7f778ca31640 1 -- 172.21.15.40:0/4041120589 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f778819f910 con 0x7f7788100a30 2024-03-18T22:57:25.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.200+0000 7f777f7fe640 1 -- 172.21.15.40:0/4041120589 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7780003970 con 0x7f7788100a30 2024-03-18T22:57:25.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.200+0000 7f777f7fe640 1 -- 172.21.15.40:0/4041120589 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7780039040 con 0x7f7788100a30 2024-03-18T22:57:25.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.200+0000 7f778ca31640 1 -- 172.21.15.40:0/4041120589 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f778819fe00 con 0x7f7788100a30 2024-03-18T22:57:25.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.202+0000 7f777f7fe640 1 -- 172.21.15.40:0/4041120589 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7780044020 con 0x7f7788100a30 2024-03-18T22:57:25.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.202+0000 7f778f618740 1 Processor -- start 2024-03-18T22:57:25.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.202+0000 7f778f618740 1 -- start start 2024-03-18T22:57:25.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.202+0000 7f778f618740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 0x55fcd092d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.202+0000 7f778f618740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092dce0 0x55fcd09327e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.202+0000 7f778f618740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 0x55fcd09352e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.202+0000 7f778f618740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fcd08914b0 con 0x55fcd092dce0 2024-03-18T22:57:25.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.202+0000 7f778f618740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fcd0839d70 con 0x55fcd092d2f0 2024-03-18T22:57:25.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.202+0000 7f778f618740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55fcd088d550 con 0x55fcd0932e50 2024-03-18T22:57:25.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.202+0000 7f777f7fe640 1 --2- 172.21.15.40:0/4041120589 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77740a2800 0x7f77740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f778659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092dce0 0x55fcd09327e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f7785d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 0x55fcd09352e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f7786d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 0x55fcd092d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f7785d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 0x55fcd09352e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:47734/0 (socket says 172.21.15.40:47734) 2024-03-18T22:57:25.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f7785d9b640 1 -- 172.21.15.40:0/156893780 learned_addr learned my addr 172.21.15.40:0/156893780 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f7786d9d640 1 --2- 172.21.15.40:0/4041120589 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77740a2800 0x7f77740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f777f7fe640 1 -- 172.21.15.40:0/4041120589 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f77800f0eb0 con 0x7f7788100a30 2024-03-18T22:57:25.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f7785d9b640 1 -- 172.21.15.40:0/156893780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 msgr2=0x55fcd092d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f7785d9b640 1 --2- 172.21.15.40:0/156893780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 0x55fcd092d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f7785d9b640 1 -- 172.21.15.40:0/156893780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092dce0 msgr2=0x55fcd09327e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f7785d9b640 1 --2- 172.21.15.40:0/156893780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092dce0 0x55fcd09327e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f7785d9b640 1 -- 172.21.15.40:0/156893780 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcd085c210 con 0x55fcd0932e50 2024-03-18T22:57:25.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f7786d9d640 1 --2- 172.21.15.40:0/156893780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 0x55fcd092d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:57:25.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f7785d9b640 1 --2- 172.21.15.40:0/156893780 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 0x55fcd09352e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7770009c90 tx=0x7f7770011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67dbc61b7be9093a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f7786d9d640 1 --2- 172.21.15.40:0/4041120589 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77740a2800 0x7f77740a4cc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x55fcd092e540 tx=0x7f7778005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f777cff9640 1 -- 172.21.15.40:0/156893780 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777001e070 con 0x55fcd0932e50 2024-03-18T22:57:25.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f777cff9640 1 -- 172.21.15.40:0/156893780 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7770009e40 con 0x55fcd0932e50 2024-03-18T22:57:25.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.203+0000 7f777cff9640 1 -- 172.21.15.40:0/156893780 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777000e960 con 0x55fcd0932e50 2024-03-18T22:57:25.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.204+0000 7f778f618740 1 -- 172.21.15.40:0/156893780 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 msgr2=0x55fcd09352e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.204+0000 7f778f618740 1 --2- 172.21.15.40:0/156893780 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 0x55fcd09352e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7770009c90 tx=0x7f7770011810 comp rx=0 tx=0).stop 2024-03-18T22:57:25.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.204+0000 7f778f618740 1 -- 172.21.15.40:0/156893780 shutdown_connections 2024-03-18T22:57:25.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.204+0000 7f778f618740 1 --2- 172.21.15.40:0/156893780 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 0x55fcd09352e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.204+0000 7f778f618740 1 --2- 172.21.15.40:0/156893780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092dce0 0x55fcd09327e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.204+0000 7f778f618740 1 --2- 172.21.15.40:0/156893780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 0x55fcd092d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.204+0000 7f778f618740 1 -- 172.21.15.40:0/156893780 >> 172.21.15.40:0/156893780 conn(0x55fcd0928860 msgr2=0x55fcd0928c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.204+0000 7f778f618740 1 -- 172.21.15.40:0/156893780 shutdown_connections 2024-03-18T22:57:25.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.204+0000 7f778f618740 1 -- 172.21.15.40:0/156893780 wait complete. 2024-03-18T22:57:25.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.204+0000 7f778f618740 1 -- start start 2024-03-18T22:57:25.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.204+0000 7f778f618740 1 Processor -- start 2024-03-18T22:57:25.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f778f618740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 0x55fcd0937860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f778f618740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd092dce0 0x55fcd0937da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f778f618740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd0932e50 0x55fcd093ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f778f618740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55fcd0935e00 con 0x55fcd0932e50 2024-03-18T22:57:25.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f778f618740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fcd084d310 con 0x55fcd092d2f0 2024-03-18T22:57:25.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f778f618740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55fcd0935f80 con 0x55fcd092dce0 2024-03-18T22:57:25.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f778f618740 10 client.0 _ll_register_callbacks cb 0x55fcd08412d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:25.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f7786d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd0932e50 0x55fcd093ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f7785d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd092dce0 0x55fcd0937da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f7786d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd0932e50 0x55fcd093ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49282/0 (socket says 172.21.15.40:49282) 2024-03-18T22:57:25.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f7786d9d640 1 -- 172.21.15.40:0/2978235048 learned_addr learned my addr 172.21.15.40:0/2978235048 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f778659c640 1 --2- 172.21.15.40:0/2978235048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 0x55fcd0937860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f7785d9b640 1 -- 172.21.15.40:0/2978235048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 msgr2=0x55fcd0937860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f7785d9b640 1 --2- 172.21.15.40:0/2978235048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 0x55fcd0937860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f7785d9b640 1 -- 172.21.15.40:0/2978235048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd0932e50 msgr2=0x55fcd093ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f7785d9b640 1 --2- 172.21.15.40:0/2978235048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd0932e50 0x55fcd093ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f7785d9b640 1 -- 172.21.15.40:0/2978235048 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcd05a53f0 con 0x55fcd092dce0 2024-03-18T22:57:25.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f7786d9d640 1 --2- 172.21.15.40:0/2978235048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd0932e50 0x55fcd093ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:25.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f7785d9b640 1 --2- 172.21.15.40:0/2978235048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd092dce0 0x55fcd0937da0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7770011d10 tx=0x7f7770009850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=995d2027fc622c6b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f776dffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:57:25.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f776dffb640 1 -- 172.21.15.40:0/2978235048 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777001e050 con 0x55fcd092dce0 2024-03-18T22:57:25.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f778f618740 1 -- 172.21.15.40:0/2978235048 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fcd08ef3e0 con 0x55fcd092dce0 2024-03-18T22:57:25.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.205+0000 7f776dffb640 1 -- 172.21.15.40:0/2978235048 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7770004230 con 0x55fcd092dce0 2024-03-18T22:57:25.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.206+0000 7f776dffb640 1 -- 172.21.15.40:0/2978235048 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7770032050 con 0x55fcd092dce0 2024-03-18T22:57:25.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.206+0000 7f778f618740 10 client.28300 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:25.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.206+0000 7f778f618740 1 -- 172.21.15.40:0/2978235048 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55fcd066f7d0 con 0x55fcd092dce0 2024-03-18T22:57:25.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.206+0000 7f776dffb640 1 -- 172.21.15.40:0/2978235048 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55fcd08ef3e0 con 0x55fcd092dce0 2024-03-18T22:57:25.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.206+0000 7f776dffb640 1 -- 172.21.15.40:0/2978235048 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f777002d030 con 0x55fcd092dce0 2024-03-18T22:57:25.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.206+0000 7f778f618740 10 client.28300 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:25.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.206+0000 7f778f618740 10 client.28300 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:57:25.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.206+0000 7f778f618740 20 client.28300 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:25.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.206+0000 7f778f618740 1 --2- 172.21.15.40:0/2978235048 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55fcd09434e0 0x55fcd0945960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.206+0000 7f778f618740 4 client.28300 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:57:25.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.206+0000 7f778f618740 1 -- 172.21.15.40:0/2978235048 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcd089e9f0 con 0x55fcd09434e0 2024-03-18T22:57:25.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.206+0000 7f778659c640 1 --2- 172.21.15.40:0/2978235048 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55fcd09434e0 0x55fcd0945960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.207+0000 7f776dffb640 10 client.28300 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:57:25.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.207+0000 7f778659c640 1 --2- 172.21.15.40:0/2978235048 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55fcd09434e0 0x55fcd0945960 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7e7aaebb56e23eae server_cookie=b0017bbd22c23d5c in_seq=0 out_seq=0 2024-03-18T22:57:25.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.208+0000 7f776dffb640 1 -- 172.21.15.40:0/2978235048 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fcd089e9f0 con 0x55fcd09434e0 2024-03-18T22:57:25.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.208+0000 7f776dffb640 10 client.28300 handle_command_reply: tid=0 2024-03-18T22:57:25.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.208+0000 7f778f618740 1 client.28300 shutdown 2024-03-18T22:57:25.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.208+0000 7f778f618740 1 -- 172.21.15.40:0/2978235048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd092dce0 msgr2=0x55fcd0937da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.208+0000 7f778f618740 1 --2- 172.21.15.40:0/2978235048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd092dce0 0x55fcd0937da0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7770011d10 tx=0x7f7770009850 comp rx=0 tx=0).stop 2024-03-18T22:57:25.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.208+0000 7f778f618740 1 -- 172.21.15.40:0/2978235048 shutdown_connections 2024-03-18T22:57:25.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.208+0000 7f778f618740 1 --2- 172.21.15.40:0/2978235048 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55fcd09434e0 0x55fcd0945960 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.208+0000 7f778f618740 1 --2- 172.21.15.40:0/2978235048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd0932e50 0x55fcd093ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.208+0000 7f778f618740 1 --2- 172.21.15.40:0/2978235048 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd092dce0 0x55fcd0937da0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.208+0000 7f778f618740 1 --2- 172.21.15.40:0/2978235048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 0x55fcd0937860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.208+0000 7f776dffb640 0 client.28300 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:57:25.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.208+0000 7f778f618740 1 -- 172.21.15.40:0/2978235048 >> 172.21.15.40:0/2978235048 conn(0x55fcd0928860 msgr2=0x55fcd0927ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.208+0000 7f778f618740 1 -- 172.21.15.40:0/2978235048 shutdown_connections 2024-03-18T22:57:25.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.208+0000 7f778f618740 1 -- 172.21.15.40:0/2978235048 wait complete. 2024-03-18T22:57:25.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.208+0000 7f778f618740 20 client.28300 trim_cache size 0 max 16384 2024-03-18T22:57:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.221+0000 7f778f618740 1 Processor -- start 2024-03-18T22:57:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.221+0000 7f778f618740 1 -- start start 2024-03-18T22:57:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.221+0000 7f778f618740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092d2f0 0x55fcd0927310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.221+0000 7f778f618740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092dce0 0x55fcd093eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.221+0000 7f778f618740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 0x55fcd0927850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.221+0000 7f778f618740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f77881aa570 con 0x55fcd092d2f0 2024-03-18T22:57:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.221+0000 7f778f618740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f77881ac0b0 con 0x55fcd092dce0 2024-03-18T22:57:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.221+0000 7f778f618740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f77881aac40 con 0x55fcd0932e50 2024-03-18T22:57:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.221+0000 7f778659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092d2f0 0x55fcd0927310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.221+0000 7f778659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092d2f0 0x55fcd0927310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49294/0 (socket says 172.21.15.40:49294) 2024-03-18T22:57:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.221+0000 7f778659c640 1 -- 172.21.15.40:0/919805633 learned_addr learned my addr 172.21.15.40:0/919805633 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.221+0000 7f7785d9b640 1 --2- 172.21.15.40:0/919805633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092dce0 0x55fcd093eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.221+0000 7f7786d9d640 1 --2- 172.21.15.40:0/919805633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 0x55fcd0927850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f778659c640 1 -- 172.21.15.40:0/919805633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 msgr2=0x55fcd0927850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f778659c640 1 --2- 172.21.15.40:0/919805633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 0x55fcd0927850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f778659c640 1 -- 172.21.15.40:0/919805633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092dce0 msgr2=0x55fcd093eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f778659c640 1 --2- 172.21.15.40:0/919805633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092dce0 0x55fcd093eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f778659c640 1 -- 172.21.15.40:0/919805633 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcd09297d0 con 0x55fcd092d2f0 2024-03-18T22:57:25.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f778659c640 1 --2- 172.21.15.40:0/919805633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092d2f0 0x55fcd0927310 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f77800ee560 tx=0x7f77800b9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84cbec943d04728a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f776ffff640 1 -- 172.21.15.40:0/919805633 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77800bcd60 con 0x55fcd092d2f0 2024-03-18T22:57:25.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f776ffff640 1 -- 172.21.15.40:0/919805633 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77800bdb00 con 0x55fcd092d2f0 2024-03-18T22:57:25.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f776ffff640 1 -- 172.21.15.40:0/919805633 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77800c2470 con 0x55fcd092d2f0 2024-03-18T22:57:25.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f778f618740 1 -- 172.21.15.40:0/919805633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092d2f0 msgr2=0x55fcd0927310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f778f618740 1 --2- 172.21.15.40:0/919805633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092d2f0 0x55fcd0927310 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f77800ee560 tx=0x7f77800b9040 comp rx=0 tx=0).stop 2024-03-18T22:57:25.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f778f618740 1 -- 172.21.15.40:0/919805633 shutdown_connections 2024-03-18T22:57:25.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f778f618740 1 --2- 172.21.15.40:0/919805633 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 0x55fcd0927850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f778f618740 1 --2- 172.21.15.40:0/919805633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092dce0 0x55fcd093eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f778f618740 1 --2- 172.21.15.40:0/919805633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092d2f0 0x55fcd0927310 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f778f618740 1 -- 172.21.15.40:0/919805633 >> 172.21.15.40:0/919805633 conn(0x55fcd0928860 msgr2=0x55fcd0936320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.222+0000 7f778f618740 1 -- 172.21.15.40:0/919805633 shutdown_connections 2024-03-18T22:57:25.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f778f618740 1 -- 172.21.15.40:0/919805633 wait complete. 2024-03-18T22:57:25.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f778f618740 1 -- start start 2024-03-18T22:57:25.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f778f618740 1 Processor -- start 2024-03-18T22:57:25.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f778f618740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 0x55fcd093bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f778f618740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092dce0 0x55fcd093c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f778f618740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 0x55fcd09a7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f778f618740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f77881ab9e0 con 0x55fcd092dce0 2024-03-18T22:57:25.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f778f618740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f77881ab310 con 0x55fcd092d2f0 2024-03-18T22:57:25.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f778f618740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f77881a8550 con 0x55fcd0932e50 2024-03-18T22:57:25.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f778f618740 10 client.0 _ll_register_callbacks cb 0x7f777001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:25.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f7785d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092dce0 0x55fcd093c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f7785d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092dce0 0x55fcd093c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49310/0 (socket says 172.21.15.40:49310) 2024-03-18T22:57:25.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f7785d9b640 1 -- 172.21.15.40:0/265682957 learned_addr learned my addr 172.21.15.40:0/265682957 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f7786d9d640 1 --2- 172.21.15.40:0/265682957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 0x55fcd09a7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f778659c640 1 --2- 172.21.15.40:0/265682957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 0x55fcd093bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f7785d9b640 1 -- 172.21.15.40:0/265682957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 msgr2=0x55fcd09a7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f7785d9b640 1 --2- 172.21.15.40:0/265682957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 0x55fcd09a7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f7785d9b640 1 -- 172.21.15.40:0/265682957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 msgr2=0x55fcd093bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f7785d9b640 1 --2- 172.21.15.40:0/265682957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 0x55fcd093bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f7785d9b640 1 -- 172.21.15.40:0/265682957 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcd09a7d90 con 0x55fcd092dce0 2024-03-18T22:57:25.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f7786d9d640 1 --2- 172.21.15.40:0/265682957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 0x55fcd09a7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:57:25.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.223+0000 7f778659c640 1 --2- 172.21.15.40:0/265682957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 0x55fcd093bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:57:25.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f7785d9b640 1 --2- 172.21.15.40:0/265682957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092dce0 0x55fcd093c1d0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f777000eaa0 tx=0x7f777000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d6ec94c330ac2a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f776dffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:57:25.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f776dffb640 1 -- 172.21.15.40:0/265682957 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777001e040 con 0x55fcd092dce0 2024-03-18T22:57:25.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f776dffb640 1 -- 172.21.15.40:0/265682957 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f777000e830 con 0x55fcd092dce0 2024-03-18T22:57:25.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f778f618740 1 -- 172.21.15.40:0/265682957 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77800bdb00 con 0x55fcd092dce0 2024-03-18T22:57:25.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f776dffb640 1 -- 172.21.15.40:0/265682957 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7770043050 con 0x55fcd092dce0 2024-03-18T22:57:25.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f778f618740 10 client.18492 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:25.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f778f618740 1 -- 172.21.15.40:0/265682957 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55fcd09a8fc0 con 0x55fcd092dce0 2024-03-18T22:57:25.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f776dffb640 1 -- 172.21.15.40:0/265682957 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77800bdb00 con 0x55fcd092dce0 2024-03-18T22:57:25.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f776dffb640 1 -- 172.21.15.40:0/265682957 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f777003e030 con 0x55fcd092dce0 2024-03-18T22:57:25.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f778f618740 10 client.18492 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:25.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f778f618740 10 client.18492 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:57:25.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f778f618740 20 client.18492 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:25.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f778f618740 1 --2- 172.21.15.40:0/265682957 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55fcd09a99f0 0x55fcd09abe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f778f618740 4 client.18492 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:57:25.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f778f618740 1 -- 172.21.15.40:0/265682957 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fcd0898d40 con 0x55fcd09a99f0 2024-03-18T22:57:25.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.224+0000 7f778659c640 1 --2- 172.21.15.40:0/265682957 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55fcd09a99f0 0x55fcd09abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f776dffb640 10 client.18492 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:57:25.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f778659c640 1 --2- 172.21.15.40:0/265682957 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55fcd09a99f0 0x55fcd09abe10 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=287556bc3d4df324 server_cookie=8c5c3f31c702f453 in_seq=0 out_seq=0 2024-03-18T22:57:25.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f776dffb640 1 -- 172.21.15.40:0/265682957 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fcd0898d40 con 0x55fcd09a99f0 2024-03-18T22:57:25.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f776dffb640 10 client.18492 handle_command_reply: tid=0 2024-03-18T22:57:25.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f778f618740 1 client.18492 shutdown 2024-03-18T22:57:25.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f778f618740 1 -- 172.21.15.40:0/265682957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092dce0 msgr2=0x55fcd093c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f778f618740 1 --2- 172.21.15.40:0/265682957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092dce0 0x55fcd093c1d0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f777000eaa0 tx=0x7f777000ef40 comp rx=0 tx=0).stop 2024-03-18T22:57:25.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f778f618740 1 -- 172.21.15.40:0/265682957 shutdown_connections 2024-03-18T22:57:25.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f778f618740 1 --2- 172.21.15.40:0/265682957 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55fcd0932e50 0x55fcd09a7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f778f618740 1 --2- 172.21.15.40:0/265682957 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55fcd09a99f0 0x55fcd09abe10 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f778f618740 1 --2- 172.21.15.40:0/265682957 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcd092dce0 0x55fcd093c1d0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f778f618740 1 --2- 172.21.15.40:0/265682957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd092d2f0 0x55fcd093bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f776dffb640 0 client.18492 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:57:25.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f778f618740 1 -- 172.21.15.40:0/265682957 >> 172.21.15.40:0/265682957 conn(0x55fcd0928860 msgr2=0x55fcd092bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f778f618740 1 -- 172.21.15.40:0/265682957 shutdown_connections 2024-03-18T22:57:25.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f778f618740 1 -- 172.21.15.40:0/265682957 wait complete. 2024-03-18T22:57:25.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.225+0000 7f778f618740 20 client.18492 trim_cache size 0 max 16384 2024-03-18T22:57:25.233 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:57:25.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.226+0000 7f778ca31640 1 -- 172.21.15.40:0/4041120589 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77740a2800 msgr2=0x7f77740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.226+0000 7f778ca31640 1 --2- 172.21.15.40:0/4041120589 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77740a2800 0x7f77740a4cc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x55fcd092e540 tx=0x7f7778005f90 comp rx=0 tx=0).stop 2024-03-18T22:57:25.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.226+0000 7f778ca31640 1 -- 172.21.15.40:0/4041120589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7788100a30 msgr2=0x7f7788100490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.226+0000 7f778ca31640 1 --2- 172.21.15.40:0/4041120589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7788100a30 0x7f7788100490 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7780032380 tx=0x7f7780032820 comp rx=0 tx=0).stop 2024-03-18T22:57:25.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.226+0000 7f778ca31640 1 -- 172.21.15.40:0/4041120589 shutdown_connections 2024-03-18T22:57:25.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.226+0000 7f778ca31640 1 --2- 172.21.15.40:0/4041120589 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f77740a2800 0x7f77740a4cc0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.226+0000 7f778ca31640 1 --2- 172.21.15.40:0/4041120589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7788101d00 0x7f778819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.226+0000 7f778ca31640 1 --2- 172.21.15.40:0/4041120589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f77881013f0 0x7f778819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.226+0000 7f778ca31640 1 --2- 172.21.15.40:0/4041120589 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7788100a30 0x7f7788100490 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.226+0000 7f778ca31640 1 -- 172.21.15.40:0/4041120589 >> 172.21.15.40:0/4041120589 conn(0x7f77880fb460 msgr2=0x7f7788104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.226+0000 7f778ca31640 1 -- 172.21.15.40:0/4041120589 shutdown_connections 2024-03-18T22:57:25.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.226+0000 7f778ca31640 1 -- 172.21.15.40:0/4041120589 wait complete. 2024-03-18T22:57:25.237 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:57:25.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.346+0000 7f4759e36640 1 -- 172.21.15.40:0/1083816928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47541022d0 msgr2=0x7f4754108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.346+0000 7f4759e36640 1 --2- 172.21.15.40:0/1083816928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47541022d0 0x7f4754108b60 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f474c009ca0 tx=0x7f474c02c0f0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.346+0000 7f4759e36640 1 -- 172.21.15.40:0/1083816928 shutdown_connections 2024-03-18T22:57:25.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.346+0000 7f4759e36640 1 --2- 172.21.15.40:0/1083816928 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47541022d0 0x7f4754108b60 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.346+0000 7f4759e36640 1 --2- 172.21.15.40:0/1083816928 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f4754101910 0x7f4754101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.346+0000 7f4759e36640 1 --2- 172.21.15.40:0/1083816928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4754100720 0x7f4754100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.346+0000 7f4759e36640 1 -- 172.21.15.40:0/1083816928 >> 172.21.15.40:0/1083816928 conn(0x7f47540fbc90 msgr2=0x7f47540fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.346+0000 7f4759e36640 1 -- 172.21.15.40:0/1083816928 shutdown_connections 2024-03-18T22:57:25.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.346+0000 7f4759e36640 1 -- 172.21.15.40:0/1083816928 wait complete. 2024-03-18T22:57:25.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.346+0000 7f4759e36640 1 Processor -- start 2024-03-18T22:57:25.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.346+0000 7f4759e36640 1 -- start start 2024-03-18T22:57:25.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4759e36640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4754100720 0x7f4754197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4759e36640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4754101910 0x7f4754197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4759e36640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47541022d0 0x7f475419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4759e36640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f475410c230 con 0x7f4754101910 2024-03-18T22:57:25.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4759e36640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f475410c0b0 con 0x7f4754100720 2024-03-18T22:57:25.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4759e36640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f475410c3b0 con 0x7f47541022d0 2024-03-18T22:57:25.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4752ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4754101910 0x7f4754197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4752ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4754101910 0x7f4754197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49330/0 (socket says 172.21.15.40:49330) 2024-03-18T22:57:25.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4752ffd640 1 -- 172.21.15.40:0/1177334719 learned_addr learned my addr 172.21.15.40:0/1177334719 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f47537fe640 1 --2- 172.21.15.40:0/1177334719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4754100720 0x7f4754197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4753fff640 1 --2- 172.21.15.40:0/1177334719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47541022d0 0x7f475419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4752ffd640 1 -- 172.21.15.40:0/1177334719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47541022d0 msgr2=0x7f475419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4752ffd640 1 --2- 172.21.15.40:0/1177334719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47541022d0 0x7f475419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4752ffd640 1 -- 172.21.15.40:0/1177334719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4754100720 msgr2=0x7f4754197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4752ffd640 1 --2- 172.21.15.40:0/1177334719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4754100720 0x7f4754197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4752ffd640 1 -- 172.21.15.40:0/1177334719 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f475419f620 con 0x7f4754101910 2024-03-18T22:57:25.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4752ffd640 1 --2- 172.21.15.40:0/1177334719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4754101910 0x7f4754197f40 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f474800ea60 tx=0x7f474800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4750ff9640 1 -- 172.21.15.40:0/1177334719 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474800cde0 con 0x7f4754101910 2024-03-18T22:57:25.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4759e36640 1 -- 172.21.15.40:0/1177334719 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f475419f910 con 0x7f4754101910 2024-03-18T22:57:25.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4750ff9640 1 -- 172.21.15.40:0/1177334719 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47480035f0 con 0x7f4754101910 2024-03-18T22:57:25.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4750ff9640 1 -- 172.21.15.40:0/1177334719 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4748010690 con 0x7f4754101910 2024-03-18T22:57:25.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.347+0000 7f4759e36640 1 -- 172.21.15.40:0/1177334719 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f475419feb0 con 0x7f4754101910 2024-03-18T22:57:25.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.350+0000 7f4750ff9640 1 -- 172.21.15.40:0/1177334719 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f47480037a0 con 0x7f4754101910 2024-03-18T22:57:25.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.350+0000 7f475ca29740 1 Processor -- start 2024-03-18T22:57:25.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.350+0000 7f475ca29740 1 -- start start 2024-03-18T22:57:25.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.350+0000 7f4750ff9640 1 --2- 172.21.15.40:0/1177334719 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f473c0a2880 0x7f473c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.350+0000 7f475ca29740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a2b2f0 0x560792a2b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.350+0000 7f475ca29740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2bce0 0x560792a307e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.350+0000 7f475ca29740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a30e50 0x560792a332e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.350+0000 7f475ca29740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56079298f4b0 con 0x560792a2bce0 2024-03-18T22:57:25.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.350+0000 7f475ca29740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560792937d70 con 0x560792a30e50 2024-03-18T22:57:25.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.350+0000 7f475ca29740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56079298b550 con 0x560792a2b2f0 2024-03-18T22:57:25.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.350+0000 7f4753fff640 1 --2- 172.21.15.40:0/1177334719 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f473c0a2880 0x7f473c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.350+0000 7f4750ff9640 1 -- 172.21.15.40:0/1177334719 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f47480cf0d0 con 0x7f4754101910 2024-03-18T22:57:25.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.350+0000 7f4752ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2bce0 0x560792a307e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.350+0000 7f4753fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a30e50 0x560792a332e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.350+0000 7f47537fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a2b2f0 0x560792a2b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.351+0000 7f47537fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a2b2f0 0x560792a2b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:47778/0 (socket says 172.21.15.40:47778) 2024-03-18T22:57:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.351+0000 7f4752ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2bce0 0x560792a307e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49346/0 (socket says 172.21.15.40:49346) 2024-03-18T22:57:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.351+0000 7f47537fe640 1 -- 172.21.15.40:0/4250342715 learned_addr learned my addr 172.21.15.40:0/4250342715 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.351+0000 7f47537fe640 1 -- 172.21.15.40:0/4250342715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a30e50 msgr2=0x560792a332e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.351+0000 7f47537fe640 1 --2- 172.21.15.40:0/4250342715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a30e50 0x560792a332e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.351+0000 7f47537fe640 1 -- 172.21.15.40:0/4250342715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2bce0 msgr2=0x560792a307e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.351+0000 7f47537fe640 1 --2- 172.21.15.40:0/4250342715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2bce0 0x560792a307e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.351+0000 7f47537fe640 1 -- 172.21.15.40:0/4250342715 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56079295a210 con 0x560792a2b2f0 2024-03-18T22:57:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.351+0000 7f4753fff640 1 --2- 172.21.15.40:0/1177334719 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f473c0a2880 0x7f473c0a4d40 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f474c006010 tx=0x7f474c0099b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.351+0000 7f47537fe640 1 --2- 172.21.15.40:0/4250342715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a2b2f0 0x560792a2b710 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4744009ca0 tx=0x7f4744011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f4118b1b876564b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.351+0000 7f4739ffb640 1 -- 172.21.15.40:0/4250342715 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4744004020 con 0x560792a2b2f0 2024-03-18T22:57:25.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.351+0000 7f4739ffb640 1 -- 172.21.15.40:0/4250342715 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47440041c0 con 0x560792a2b2f0 2024-03-18T22:57:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.351+0000 7f4739ffb640 1 -- 172.21.15.40:0/4250342715 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4744019670 con 0x560792a2b2f0 2024-03-18T22:57:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.352+0000 7f475ca29740 1 -- 172.21.15.40:0/4250342715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a2b2f0 msgr2=0x560792a2b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.352+0000 7f475ca29740 1 --2- 172.21.15.40:0/4250342715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a2b2f0 0x560792a2b710 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4744009ca0 tx=0x7f4744011810 comp rx=0 tx=0).stop 2024-03-18T22:57:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.352+0000 7f475ca29740 1 -- 172.21.15.40:0/4250342715 shutdown_connections 2024-03-18T22:57:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.352+0000 7f475ca29740 1 --2- 172.21.15.40:0/4250342715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a30e50 0x560792a332e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.352+0000 7f475ca29740 1 --2- 172.21.15.40:0/4250342715 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2bce0 0x560792a307e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.352+0000 7f475ca29740 1 --2- 172.21.15.40:0/4250342715 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a2b2f0 0x560792a2b710 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.352+0000 7f475ca29740 1 -- 172.21.15.40:0/4250342715 >> 172.21.15.40:0/4250342715 conn(0x560792a26860 msgr2=0x560792a26c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.352+0000 7f475ca29740 1 -- 172.21.15.40:0/4250342715 shutdown_connections 2024-03-18T22:57:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.352+0000 7f475ca29740 1 -- 172.21.15.40:0/4250342715 wait complete. 2024-03-18T22:57:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.352+0000 7f475ca29740 1 -- start start 2024-03-18T22:57:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.352+0000 7f475ca29740 1 Processor -- start 2024-03-18T22:57:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f475ca29740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2b2f0 0x560792a357f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f475ca29740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a2bce0 0x560792a35d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f475ca29740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a30e50 0x560792a3cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f475ca29740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560792a33e00 con 0x560792a2b2f0 2024-03-18T22:57:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f475ca29740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56079294b310 con 0x560792a2bce0 2024-03-18T22:57:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f475ca29740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560792a33f80 con 0x560792a30e50 2024-03-18T22:57:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f475ca29740 10 client.0 _ll_register_callbacks cb 0x56079293f2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f47537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2b2f0 0x560792a357f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f47537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2b2f0 0x560792a357f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49350/0 (socket says 172.21.15.40:49350) 2024-03-18T22:57:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f4753fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a30e50 0x560792a3cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f47537fe640 1 -- 172.21.15.40:0/1110683134 learned_addr learned my addr 172.21.15.40:0/1110683134 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f4752ffd640 1 --2- 172.21.15.40:0/1110683134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a2bce0 0x560792a35d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f47537fe640 1 -- 172.21.15.40:0/1110683134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a30e50 msgr2=0x560792a3cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f47537fe640 1 --2- 172.21.15.40:0/1110683134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a30e50 0x560792a3cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f47537fe640 1 -- 172.21.15.40:0/1110683134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a2bce0 msgr2=0x560792a35d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f47537fe640 1 --2- 172.21.15.40:0/1110683134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a2bce0 0x560792a35d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f47537fe640 1 -- 172.21.15.40:0/1110683134 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607926a33f0 con 0x560792a2b2f0 2024-03-18T22:57:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f47537fe640 1 --2- 172.21.15.40:0/1110683134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2b2f0 0x560792a357f0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f474401b040 tx=0x7f4744003a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998158b30fc9a5be server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f4732ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:57:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f4732ffd640 1 -- 172.21.15.40:0/1110683134 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4744028070 con 0x560792a2b2f0 2024-03-18T22:57:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.353+0000 7f4732ffd640 1 -- 172.21.15.40:0/1110683134 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4744009d90 con 0x560792a2b2f0 2024-03-18T22:57:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.354+0000 7f475ca29740 1 -- 172.21.15.40:0/1110683134 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47440041c0 con 0x560792a2b2f0 2024-03-18T22:57:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.354+0000 7f4732ffd640 1 -- 172.21.15.40:0/1110683134 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f4744034050 con 0x560792a2b2f0 2024-03-18T22:57:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.354+0000 7f475ca29740 10 client.18504 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.354+0000 7f475ca29740 1 -- 172.21.15.40:0/1110683134 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x56079276d7d0 con 0x560792a2b2f0 2024-03-18T22:57:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.354+0000 7f4732ffd640 1 -- 172.21.15.40:0/1110683134 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47440041c0 con 0x560792a2b2f0 2024-03-18T22:57:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.354+0000 7f4732ffd640 1 -- 172.21.15.40:0/1110683134 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f47440049d0 con 0x560792a2b2f0 2024-03-18T22:57:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.354+0000 7f475ca29740 10 client.18504 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.354+0000 7f475ca29740 10 client.18504 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:57:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.354+0000 7f475ca29740 20 client.18504 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:25.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.354+0000 7f475ca29740 1 --2- 172.21.15.40:0/1110683134 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x560792a41380 0x560792a43820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.354+0000 7f475ca29740 4 client.18504 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:57:25.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.354+0000 7f475ca29740 1 -- 172.21.15.40:0/1110683134 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607929ed3e0 con 0x560792a41380 2024-03-18T22:57:25.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.354+0000 7f4752ffd640 1 --2- 172.21.15.40:0/1110683134 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x560792a41380 0x560792a43820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.355+0000 7f4732ffd640 10 client.18504 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:57:25.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.355+0000 7f4752ffd640 1 --2- 172.21.15.40:0/1110683134 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x560792a41380 0x560792a43820 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b9c5f73280bb17d8 server_cookie=2d2e29b6aeb123e5 in_seq=0 out_seq=0 2024-03-18T22:57:25.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.356+0000 7f4732ffd640 1 -- 172.21.15.40:0/1110683134 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5607929ed3e0 con 0x560792a41380 2024-03-18T22:57:25.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.356+0000 7f4732ffd640 10 client.18504 handle_command_reply: tid=0 2024-03-18T22:57:25.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.356+0000 7f475ca29740 1 client.18504 shutdown 2024-03-18T22:57:25.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.356+0000 7f475ca29740 1 -- 172.21.15.40:0/1110683134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2b2f0 msgr2=0x560792a357f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.356+0000 7f475ca29740 1 --2- 172.21.15.40:0/1110683134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2b2f0 0x560792a357f0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f474401b040 tx=0x7f4744003a40 comp rx=0 tx=0).stop 2024-03-18T22:57:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.356+0000 7f475ca29740 1 -- 172.21.15.40:0/1110683134 shutdown_connections 2024-03-18T22:57:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.356+0000 7f475ca29740 1 --2- 172.21.15.40:0/1110683134 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x560792a41380 0x560792a43820 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.356+0000 7f475ca29740 1 --2- 172.21.15.40:0/1110683134 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a30e50 0x560792a3cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.356+0000 7f4732ffd640 0 client.18504 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:57:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.356+0000 7f475ca29740 1 --2- 172.21.15.40:0/1110683134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a2bce0 0x560792a35d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.356+0000 7f475ca29740 1 --2- 172.21.15.40:0/1110683134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2b2f0 0x560792a357f0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.356+0000 7f475ca29740 1 -- 172.21.15.40:0/1110683134 >> 172.21.15.40:0/1110683134 conn(0x560792a26860 msgr2=0x560792a25ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.356+0000 7f475ca29740 1 -- 172.21.15.40:0/1110683134 shutdown_connections 2024-03-18T22:57:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.356+0000 7f475ca29740 1 -- 172.21.15.40:0/1110683134 wait complete. 2024-03-18T22:57:25.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.356+0000 7f475ca29740 20 client.18504 trim_cache size 0 max 16384 2024-03-18T22:57:25.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f475ca29740 1 Processor -- start 2024-03-18T22:57:25.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f475ca29740 1 -- start start 2024-03-18T22:57:25.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f475ca29740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2b2f0 0x560792a34e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f475ca29740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a2bce0 0x560792a3ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f475ca29740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a30e50 0x560792a35350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f475ca29740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f47541aa520 con 0x560792a2b2f0 2024-03-18T22:57:25.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f475ca29740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f47541ac060 con 0x560792a2bce0 2024-03-18T22:57:25.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f475ca29740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f47541aabf0 con 0x560792a30e50 2024-03-18T22:57:25.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f47537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2b2f0 0x560792a34e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f4753fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a30e50 0x560792a35350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f4752ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a2bce0 0x560792a3ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f47537fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2b2f0 0x560792a34e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49356/0 (socket says 172.21.15.40:49356) 2024-03-18T22:57:25.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f47537fe640 1 -- 172.21.15.40:0/2122958165 learned_addr learned my addr 172.21.15.40:0/2122958165 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f47537fe640 1 -- 172.21.15.40:0/2122958165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a30e50 msgr2=0x560792a35350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f47537fe640 1 --2- 172.21.15.40:0/2122958165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a30e50 0x560792a35350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f47537fe640 1 -- 172.21.15.40:0/2122958165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a2bce0 msgr2=0x560792a3ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f47537fe640 1 --2- 172.21.15.40:0/2122958165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a2bce0 0x560792a3ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f47537fe640 1 -- 172.21.15.40:0/2122958165 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560792a2c0c0 con 0x560792a2b2f0 2024-03-18T22:57:25.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f4753fff640 1 --2- 172.21.15.40:0/2122958165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a30e50 0x560792a35350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:25.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f47537fe640 1 --2- 172.21.15.40:0/2122958165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2b2f0 0x560792a34e10 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4744009380 tx=0x7f4744009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cea848f7da177f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.370+0000 7f47397fa640 1 -- 172.21.15.40:0/2122958165 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4744028070 con 0x560792a2b2f0 2024-03-18T22:57:25.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.371+0000 7f47397fa640 1 -- 172.21.15.40:0/2122958165 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47440041c0 con 0x560792a2b2f0 2024-03-18T22:57:25.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.371+0000 7f47397fa640 1 -- 172.21.15.40:0/2122958165 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47440047e0 con 0x560792a2b2f0 2024-03-18T22:57:25.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.371+0000 7f475ca29740 1 -- 172.21.15.40:0/2122958165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2b2f0 msgr2=0x560792a34e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.371+0000 7f475ca29740 1 --2- 172.21.15.40:0/2122958165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2b2f0 0x560792a34e10 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4744009380 tx=0x7f4744009310 comp rx=0 tx=0).stop 2024-03-18T22:57:25.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.371+0000 7f475ca29740 1 -- 172.21.15.40:0/2122958165 shutdown_connections 2024-03-18T22:57:25.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.371+0000 7f475ca29740 1 --2- 172.21.15.40:0/2122958165 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a30e50 0x560792a35350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.371+0000 7f475ca29740 1 --2- 172.21.15.40:0/2122958165 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a2bce0 0x560792a3ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.371+0000 7f475ca29740 1 --2- 172.21.15.40:0/2122958165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2b2f0 0x560792a34e10 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.371+0000 7f475ca29740 1 -- 172.21.15.40:0/2122958165 >> 172.21.15.40:0/2122958165 conn(0x560792a26860 msgr2=0x560792a2a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.371+0000 7f475ca29740 1 -- 172.21.15.40:0/2122958165 shutdown_connections 2024-03-18T22:57:25.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.371+0000 7f475ca29740 1 -- 172.21.15.40:0/2122958165 wait complete. 2024-03-18T22:57:25.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f475ca29740 1 -- start start 2024-03-18T22:57:25.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f475ca29740 1 Processor -- start 2024-03-18T22:57:25.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f475ca29740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a2b2f0 0x560792a39f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f475ca29740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2bce0 0x560792a3a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f475ca29740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a30e50 0x560792a3b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f475ca29740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f47541ab990 con 0x560792a2bce0 2024-03-18T22:57:25.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f475ca29740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f47541ab2c0 con 0x560792a30e50 2024-03-18T22:57:25.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f475ca29740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f47541a8500 con 0x560792a2b2f0 2024-03-18T22:57:25.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f475ca29740 10 client.0 _ll_register_callbacks cb 0x7f4748004560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f4752ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2bce0 0x560792a3a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f47537fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a2b2f0 0x560792a39f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f4752ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2bce0 0x560792a3a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49358/0 (socket says 172.21.15.40:49358) 2024-03-18T22:57:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f4752ffd640 1 -- 172.21.15.40:0/2157331042 learned_addr learned my addr 172.21.15.40:0/2157331042 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f4753fff640 1 --2- 172.21.15.40:0/2157331042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a30e50 0x560792a3b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f4752ffd640 1 -- 172.21.15.40:0/2157331042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a2b2f0 msgr2=0x560792a39f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f4752ffd640 1 --2- 172.21.15.40:0/2157331042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a2b2f0 0x560792a39f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f4752ffd640 1 -- 172.21.15.40:0/2157331042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a30e50 msgr2=0x560792a3b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f4752ffd640 1 --2- 172.21.15.40:0/2157331042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a30e50 0x560792a3b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f4752ffd640 1 -- 172.21.15.40:0/2157331042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560792a3ba50 con 0x560792a2bce0 2024-03-18T22:57:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f47537fe640 1 --2- 172.21.15.40:0/2157331042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a2b2f0 0x560792a39f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f4752ffd640 1 --2- 172.21.15.40:0/2157331042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2bce0 0x560792a3a450 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f4748009920 tx=0x7f47480cc6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc0b878ce1d99e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f4732ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:57:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f4732ffd640 1 -- 172.21.15.40:0/2157331042 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47480b8070 con 0x560792a2bce0 2024-03-18T22:57:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f4732ffd640 1 -- 172.21.15.40:0/2157331042 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4748097cf0 con 0x560792a2bce0 2024-03-18T22:57:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.372+0000 7f475ca29740 1 -- 172.21.15.40:0/2157331042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47440041c0 con 0x560792a2bce0 2024-03-18T22:57:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.373+0000 7f4732ffd640 1 -- 172.21.15.40:0/2157331042 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f474809bd80 con 0x560792a2bce0 2024-03-18T22:57:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.373+0000 7f475ca29740 10 client.18510 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.373+0000 7f475ca29740 1 -- 172.21.15.40:0/2157331042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x560792a3cdc0 con 0x560792a2bce0 2024-03-18T22:57:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.373+0000 7f4732ffd640 1 -- 172.21.15.40:0/2157331042 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f47440041c0 con 0x560792a2bce0 2024-03-18T22:57:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.373+0000 7f4732ffd640 1 -- 172.21.15.40:0/2157331042 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f47480a3990 con 0x560792a2bce0 2024-03-18T22:57:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.373+0000 7f475ca29740 10 client.18510 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.373+0000 7f475ca29740 10 client.18510 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:57:25.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.373+0000 7f475ca29740 20 client.18510 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.373+0000 7f475ca29740 1 --2- 172.21.15.40:0/2157331042 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x560792a41380 0x560792aa9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.373+0000 7f475ca29740 4 client.18510 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:57:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.373+0000 7f475ca29740 1 -- 172.21.15.40:0/2157331042 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56079299c9f0 con 0x560792a41380 2024-03-18T22:57:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.373+0000 7f47537fe640 1 --2- 172.21.15.40:0/2157331042 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x560792a41380 0x560792aa9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f47537fe640 1 --2- 172.21.15.40:0/2157331042 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x560792a41380 0x560792aa9d80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=776108342940ff60 server_cookie=94789ad322191dd4 in_seq=0 out_seq=0 2024-03-18T22:57:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f4732ffd640 10 client.18510 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:57:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f4732ffd640 1 -- 172.21.15.40:0/2157331042 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56079299c9f0 con 0x560792a41380 2024-03-18T22:57:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f4732ffd640 10 client.18510 handle_command_reply: tid=0 2024-03-18T22:57:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f475ca29740 1 client.18510 shutdown 2024-03-18T22:57:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f475ca29740 1 -- 172.21.15.40:0/2157331042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2bce0 msgr2=0x560792a3a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f475ca29740 1 --2- 172.21.15.40:0/2157331042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2bce0 0x560792a3a450 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f4748009920 tx=0x7f47480cc6b0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f475ca29740 1 -- 172.21.15.40:0/2157331042 shutdown_connections 2024-03-18T22:57:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f475ca29740 1 --2- 172.21.15.40:0/2157331042 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x560792a41380 0x560792aa9d80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f475ca29740 1 --2- 172.21.15.40:0/2157331042 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560792a30e50 0x560792a3b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f475ca29740 1 --2- 172.21.15.40:0/2157331042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560792a2bce0 0x560792a3a450 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f4732ffd640 0 client.18510 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:57:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f475ca29740 1 --2- 172.21.15.40:0/2157331042 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560792a2b2f0 0x560792a39f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f475ca29740 1 -- 172.21.15.40:0/2157331042 >> 172.21.15.40:0/2157331042 conn(0x560792a26860 msgr2=0x560792a25310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f475ca29740 1 -- 172.21.15.40:0/2157331042 shutdown_connections 2024-03-18T22:57:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f475ca29740 1 -- 172.21.15.40:0/2157331042 wait complete. 2024-03-18T22:57:25.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f475ca29740 20 client.18510 trim_cache size 0 max 16384 2024-03-18T22:57:25.382 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:57:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f4759e36640 1 -- 172.21.15.40:0/1177334719 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f473c0a2880 msgr2=0x7f473c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f4759e36640 1 --2- 172.21.15.40:0/1177334719 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f473c0a2880 0x7f473c0a4d40 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f474c006010 tx=0x7f474c0099b0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f4759e36640 1 -- 172.21.15.40:0/1177334719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4754101910 msgr2=0x7f4754197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.374+0000 7f4759e36640 1 --2- 172.21.15.40:0/1177334719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4754101910 0x7f4754197f40 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f474800ea60 tx=0x7f474800ef20 comp rx=0 tx=0).stop 2024-03-18T22:57:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.375+0000 7f4759e36640 1 -- 172.21.15.40:0/1177334719 shutdown_connections 2024-03-18T22:57:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.375+0000 7f4759e36640 1 --2- 172.21.15.40:0/1177334719 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f473c0a2880 0x7f473c0a4d40 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.375+0000 7f4759e36640 1 --2- 172.21.15.40:0/1177334719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f47541022d0 0x7f475419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.375+0000 7f4759e36640 1 --2- 172.21.15.40:0/1177334719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4754101910 0x7f4754197f40 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.375+0000 7f4759e36640 1 --2- 172.21.15.40:0/1177334719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4754100720 0x7f4754197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.375+0000 7f4759e36640 1 -- 172.21.15.40:0/1177334719 >> 172.21.15.40:0/1177334719 conn(0x7f47540fbc90 msgr2=0x7f47540fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.375+0000 7f4759e36640 1 -- 172.21.15.40:0/1177334719 shutdown_connections 2024-03-18T22:57:25.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.375+0000 7f4759e36640 1 -- 172.21.15.40:0/1177334719 wait complete. 2024-03-18T22:57:25.385 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:25 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3616912151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:57:25.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:57:25.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.493+0000 7fadf217e640 1 -- 172.21.15.40:0/1646502956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadec100740 msgr2=0x7fadec100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.493+0000 7fadf217e640 1 --2- 172.21.15.40:0/1646502956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadec100740 0x7fadec100b60 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7faddc009900 tx=0x7faddc02bc30 comp rx=0 tx=0).stop 2024-03-18T22:57:25.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.493+0000 7fadf217e640 1 -- 172.21.15.40:0/1646502956 shutdown_connections 2024-03-18T22:57:25.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.493+0000 7fadf217e640 1 --2- 172.21.15.40:0/1646502956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fadec1022f0 0x7fadec108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.493+0000 7fadf217e640 1 --2- 172.21.15.40:0/1646502956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec101930 0x7fadec101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.493+0000 7fadf217e640 1 --2- 172.21.15.40:0/1646502956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadec100740 0x7fadec100b60 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.493+0000 7fadf217e640 1 -- 172.21.15.40:0/1646502956 >> 172.21.15.40:0/1646502956 conn(0x7fadec0fbcf0 msgr2=0x7fadec0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.493+0000 7fadf217e640 1 -- 172.21.15.40:0/1646502956 shutdown_connections 2024-03-18T22:57:25.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.493+0000 7fadf217e640 1 -- 172.21.15.40:0/1646502956 wait complete. 2024-03-18T22:57:25.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadf217e640 1 Processor -- start 2024-03-18T22:57:25.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadf217e640 1 -- start start 2024-03-18T22:57:25.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadf217e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec100740 0x7fadec06a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadf217e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fadec101930 0x7fadec19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadf217e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadec1022f0 0x7fadec19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadf217e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fadec10c240 con 0x7fadec1022f0 2024-03-18T22:57:25.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadf217e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fadec10c0c0 con 0x7fadec100740 2024-03-18T22:57:25.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadf217e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fadec10c3c0 con 0x7fadec101930 2024-03-18T22:57:25.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadebfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadec1022f0 0x7fadec19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadebfff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadec1022f0 0x7fadec19edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49374/0 (socket says 172.21.15.40:49374) 2024-03-18T22:57:25.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadebfff640 1 -- 172.21.15.40:0/2640062247 learned_addr learned my addr 172.21.15.40:0/2640062247 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadeb7fe640 1 --2- 172.21.15.40:0/2640062247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec100740 0x7fadec06a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadeaffd640 1 --2- 172.21.15.40:0/2640062247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fadec101930 0x7fadec19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadebfff640 1 -- 172.21.15.40:0/2640062247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fadec101930 msgr2=0x7fadec19c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadebfff640 1 --2- 172.21.15.40:0/2640062247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fadec101930 0x7fadec19c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadebfff640 1 -- 172.21.15.40:0/2640062247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec100740 msgr2=0x7fadec06a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadebfff640 1 --2- 172.21.15.40:0/2640062247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec100740 0x7fadec06a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.494+0000 7fadebfff640 1 -- 172.21.15.40:0/2640062247 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadec19f6c0 con 0x7fadec1022f0 2024-03-18T22:57:25.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.495+0000 7fadebfff640 1 --2- 172.21.15.40:0/2640062247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadec1022f0 0x7fadec19edb0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fade400efd0 tx=0x7fade400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.495+0000 7fade8ff9640 1 -- 172.21.15.40:0/2640062247 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade4003b70 con 0x7fadec1022f0 2024-03-18T22:57:25.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.495+0000 7fade8ff9640 1 -- 172.21.15.40:0/2640062247 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fade4003d10 con 0x7fadec1022f0 2024-03-18T22:57:25.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.495+0000 7fadf217e640 1 -- 172.21.15.40:0/2640062247 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadec19f950 con 0x7fadec1022f0 2024-03-18T22:57:25.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.495+0000 7fade8ff9640 1 -- 172.21.15.40:0/2640062247 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade4010980 con 0x7fadec1022f0 2024-03-18T22:57:25.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.495+0000 7fadf217e640 1 -- 172.21.15.40:0/2640062247 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadec19fea0 con 0x7fadec1022f0 2024-03-18T22:57:25.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.497+0000 7fade8ff9640 1 -- 172.21.15.40:0/2640062247 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fade4004020 con 0x7fadec1022f0 2024-03-18T22:57:25.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.497+0000 7fadf4d6b740 1 Processor -- start 2024-03-18T22:57:25.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.497+0000 7fadf4d6b740 1 -- start start 2024-03-18T22:57:25.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.497+0000 7fadf4d6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c189944710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.497+0000 7fadf4d6b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 0x55c1899497e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.497+0000 7fadf4d6b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189949e50 0x55c18994c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.497+0000 7fadf4d6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c1898a84b0 con 0x55c1899442f0 2024-03-18T22:57:25.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.497+0000 7fadf4d6b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c189850d70 con 0x55c189944ce0 2024-03-18T22:57:25.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.497+0000 7fadf4d6b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c1898a4550 con 0x55c189949e50 2024-03-18T22:57:25.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fadeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c189944710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fadeb7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 0x55c1899497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fadeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c189944710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49384/0 (socket says 172.21.15.40:49384) 2024-03-18T22:57:25.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fadeaffd640 1 -- 172.21.15.40:0/1480836413 learned_addr learned my addr 172.21.15.40:0/1480836413 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fadeaffd640 1 --2- 172.21.15.40:0/1480836413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189949e50 0x55c18994c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fade8ff9640 1 --2- 172.21.15.40:0/2640062247 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadd40a2880 0x7fadd40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fadeb7fe640 1 -- 172.21.15.40:0/1480836413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189949e50 msgr2=0x55c18994c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fadeb7fe640 1 --2- 172.21.15.40:0/1480836413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189949e50 0x55c18994c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fadeb7fe640 1 -- 172.21.15.40:0/1480836413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 msgr2=0x55c189944710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fadeb7fe640 1 --2- 172.21.15.40:0/1480836413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c189944710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fadeb7fe640 1 -- 172.21.15.40:0/1480836413 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c189873210 con 0x55c189944ce0 2024-03-18T22:57:25.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fadebfff640 1 --2- 172.21.15.40:0/2640062247 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadd40a2880 0x7fadd40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fadeb7fe640 1 --2- 172.21.15.40:0/1480836413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 0x55c1899497e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7faddc036040 tx=0x7faddc009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=824c3a71abf34a81 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fade8ff9640 1 -- 172.21.15.40:0/2640062247 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fade40cf290 con 0x7fadec1022f0 2024-03-18T22:57:25.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fadd1ffb640 1 -- 172.21.15.40:0/1480836413 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faddc045020 con 0x55c189944ce0 2024-03-18T22:57:25.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fadd1ffb640 1 -- 172.21.15.40:0/1480836413 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faddc004040 con 0x55c189944ce0 2024-03-18T22:57:25.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.498+0000 7fadd1ffb640 1 -- 172.21.15.40:0/1480836413 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faddc005530 con 0x55c189944ce0 2024-03-18T22:57:25.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.499+0000 7fadf4d6b740 1 -- 172.21.15.40:0/1480836413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 msgr2=0x55c1899497e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.499+0000 7fadebfff640 1 --2- 172.21.15.40:0/2640062247 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadd40a2880 0x7fadd40a4d40 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fade400cba0 tx=0x7fade4086040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.499+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/1480836413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 0x55c1899497e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7faddc036040 tx=0x7faddc009560 comp rx=0 tx=0).stop 2024-03-18T22:57:25.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.499+0000 7fadf4d6b740 1 -- 172.21.15.40:0/1480836413 shutdown_connections 2024-03-18T22:57:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.499+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/1480836413 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189949e50 0x55c18994c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.499+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/1480836413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 0x55c1899497e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.499+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/1480836413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c189944710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.499+0000 7fadf4d6b740 1 -- 172.21.15.40:0/1480836413 >> 172.21.15.40:0/1480836413 conn(0x55c18993f860 msgr2=0x55c18993fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.499+0000 7fadf4d6b740 1 -- 172.21.15.40:0/1480836413 shutdown_connections 2024-03-18T22:57:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.499+0000 7fadf4d6b740 1 -- 172.21.15.40:0/1480836413 wait complete. 2024-03-18T22:57:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.499+0000 7fadf4d6b740 1 -- start start 2024-03-18T22:57:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadf4d6b740 1 Processor -- start 2024-03-18T22:57:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadf4d6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c18994e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadf4d6b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 0x55c18994ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadf4d6b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189949e50 0x55c189955cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadf4d6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c18994cda0 con 0x55c1899442f0 2024-03-18T22:57:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadf4d6b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c189864310 con 0x55c189944ce0 2024-03-18T22:57:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadf4d6b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c18994cf20 con 0x55c189949e50 2024-03-18T22:57:25.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadf4d6b740 10 client.0 _ll_register_callbacks cb 0x55c1898582d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c18994e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c18994e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49394/0 (socket says 172.21.15.40:49394) 2024-03-18T22:57:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadeaffd640 1 -- 172.21.15.40:0/3069549482 learned_addr learned my addr 172.21.15.40:0/3069549482 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadeb7fe640 1 --2- 172.21.15.40:0/3069549482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 0x55c18994ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadeaffd640 1 --2- 172.21.15.40:0/3069549482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189949e50 0x55c189955cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadeaffd640 1 -- 172.21.15.40:0/3069549482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189949e50 msgr2=0x55c189955cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadeaffd640 1 --2- 172.21.15.40:0/3069549482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189949e50 0x55c189955cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadeaffd640 1 -- 172.21.15.40:0/3069549482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 msgr2=0x55c18994ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadeaffd640 1 --2- 172.21.15.40:0/3069549482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 0x55c18994ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadeaffd640 1 -- 172.21.15.40:0/3069549482 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1895bc3f0 con 0x55c1899442f0 2024-03-18T22:57:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadeb7fe640 1 --2- 172.21.15.40:0/3069549482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 0x55c18994ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadeaffd640 1 --2- 172.21.15.40:0/3069549482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c18994e750 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fade0012f10 tx=0x7fade0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbbb068e386e4292 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:57:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.500+0000 7fadcaffd640 1 -- 172.21.15.40:0/3069549482 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade00120d0 con 0x55c1899442f0 2024-03-18T22:57:25.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.501+0000 7fadcaffd640 1 -- 172.21.15.40:0/3069549482 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fade0004260 con 0x55c1899442f0 2024-03-18T22:57:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.501+0000 7fadf4d6b740 1 -- 172.21.15.40:0/3069549482 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faddc004040 con 0x55c1899442f0 2024-03-18T22:57:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.501+0000 7fadcaffd640 1 -- 172.21.15.40:0/3069549482 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fade0029050 con 0x55c1899442f0 2024-03-18T22:57:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.501+0000 7fadf4d6b740 10 client.18522 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.501+0000 7fadf4d6b740 1 -- 172.21.15.40:0/3069549482 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c1896867d0 con 0x55c1899442f0 2024-03-18T22:57:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.501+0000 7fadcaffd640 1 -- 172.21.15.40:0/3069549482 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faddc004040 con 0x55c1899442f0 2024-03-18T22:57:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.501+0000 7fadcaffd640 1 -- 172.21.15.40:0/3069549482 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fade0010430 con 0x55c1899442f0 2024-03-18T22:57:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.501+0000 7fadf4d6b740 10 client.18522 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.501+0000 7fadf4d6b740 10 client.18522 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:57:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.501+0000 7fadf4d6b740 20 client.18522 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.501+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/3069549482 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c18995a6a0 0x55c18995cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.501+0000 7fadf4d6b740 4 client.18522 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:57:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.501+0000 7fadf4d6b740 1 -- 172.21.15.40:0/3069549482 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1899063e0 con 0x55c18995a6a0 2024-03-18T22:57:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.501+0000 7fadeb7fe640 1 --2- 172.21.15.40:0/3069549482 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c18995a6a0 0x55c18995cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.502+0000 7fadeb7fe640 1 --2- 172.21.15.40:0/3069549482 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c18995a6a0 0x55c18995cb30 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fc9af4724fe515d8 server_cookie=785f142772cdcda2 in_seq=0 out_seq=0 2024-03-18T22:57:25.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.502+0000 7fadcaffd640 10 client.18522 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:57:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.503+0000 7fadcaffd640 1 -- 172.21.15.40:0/3069549482 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c1899063e0 con 0x55c18995a6a0 2024-03-18T22:57:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.503+0000 7fadcaffd640 10 client.18522 handle_command_reply: tid=0 2024-03-18T22:57:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.503+0000 7fadf4d6b740 1 client.18522 shutdown 2024-03-18T22:57:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.503+0000 7fadf4d6b740 1 -- 172.21.15.40:0/3069549482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 msgr2=0x55c18994e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.503+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/3069549482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c18994e750 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fade0012f10 tx=0x7fade0005f90 comp rx=0 tx=0).stop 2024-03-18T22:57:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.503+0000 7fadf4d6b740 1 -- 172.21.15.40:0/3069549482 shutdown_connections 2024-03-18T22:57:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.503+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/3069549482 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c18995a6a0 0x55c18995cb30 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.503+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/3069549482 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189949e50 0x55c189955cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.503+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/3069549482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 0x55c18994ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.503+0000 7fadcaffd640 0 client.18522 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:57:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.503+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/3069549482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c18994e750 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.503+0000 7fadf4d6b740 1 -- 172.21.15.40:0/3069549482 >> 172.21.15.40:0/3069549482 conn(0x55c18993f860 msgr2=0x55c18993eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.503+0000 7fadf4d6b740 1 -- 172.21.15.40:0/3069549482 shutdown_connections 2024-03-18T22:57:25.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.503+0000 7fadf4d6b740 1 -- 172.21.15.40:0/3069549482 wait complete. 2024-03-18T22:57:25.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.503+0000 7fadf4d6b740 20 client.18522 trim_cache size 0 max 16384 2024-03-18T22:57:25.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.516+0000 7fadf4d6b740 1 Processor -- start 2024-03-18T22:57:25.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.516+0000 7fadf4d6b740 1 -- start start 2024-03-18T22:57:25.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.516+0000 7fadf4d6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c18994e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.516+0000 7fadf4d6b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 0x55c189955c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.516+0000 7fadf4d6b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189949e50 0x55c18994e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.516+0000 7fadf4d6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fadec1aa3b0 con 0x55c1899442f0 2024-03-18T22:57:25.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.516+0000 7fadf4d6b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fadec1abef0 con 0x55c189944ce0 2024-03-18T22:57:25.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.516+0000 7fadf4d6b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fadec1aaa80 con 0x55c189949e50 2024-03-18T22:57:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.516+0000 7fadeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c18994e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.516+0000 7fadeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c18994e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49398/0 (socket says 172.21.15.40:49398) 2024-03-18T22:57:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.517+0000 7fadeb7fe640 1 --2- 172.21.15.40:0/1635808819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 0x55c189955c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.517+0000 7fadeaffd640 1 -- 172.21.15.40:0/1635808819 learned_addr learned my addr 172.21.15.40:0/1635808819 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.517+0000 7fadeaffd640 1 --2- 172.21.15.40:0/1635808819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189949e50 0x55c18994e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.517+0000 7fadeaffd640 1 -- 172.21.15.40:0/1635808819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189949e50 msgr2=0x55c18994e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.517+0000 7fadeaffd640 1 --2- 172.21.15.40:0/1635808819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189949e50 0x55c18994e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.517+0000 7fadeaffd640 1 -- 172.21.15.40:0/1635808819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 msgr2=0x55c189955c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.517+0000 7fadeaffd640 1 --2- 172.21.15.40:0/1635808819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 0x55c189955c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.517+0000 7fadeaffd640 1 -- 172.21.15.40:0/1635808819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c189915880 con 0x55c1899442f0 2024-03-18T22:57:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.517+0000 7fadeaffd640 1 --2- 172.21.15.40:0/1635808819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c18994e360 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fade0001eb0 tx=0x7fade000ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7486bfb6a34a893 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.517+0000 7fadd17fa640 1 -- 172.21.15.40:0/1635808819 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade00120d0 con 0x55c1899442f0 2024-03-18T22:57:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.517+0000 7fadd17fa640 1 -- 172.21.15.40:0/1635808819 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fade0012270 con 0x55c1899442f0 2024-03-18T22:57:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.517+0000 7fadd17fa640 1 -- 172.21.15.40:0/1635808819 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade00035f0 con 0x55c1899442f0 2024-03-18T22:57:25.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 -- 172.21.15.40:0/1635808819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 msgr2=0x55c18994e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/1635808819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c18994e360 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fade0001eb0 tx=0x7fade000ff90 comp rx=0 tx=0).stop 2024-03-18T22:57:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 -- 172.21.15.40:0/1635808819 shutdown_connections 2024-03-18T22:57:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/1635808819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189949e50 0x55c18994e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/1635808819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189944ce0 0x55c189955c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/1635808819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c18994e360 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 -- 172.21.15.40:0/1635808819 >> 172.21.15.40:0/1635808819 conn(0x55c18993f860 msgr2=0x55c18993fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 -- 172.21.15.40:0/1635808819 shutdown_connections 2024-03-18T22:57:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 -- 172.21.15.40:0/1635808819 wait complete. 2024-03-18T22:57:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 -- start start 2024-03-18T22:57:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 Processor -- start 2024-03-18T22:57:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c1899532d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189944ce0 0x55c189953810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189949e50 0x55c189954820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fadec1ab820 con 0x55c1899442f0 2024-03-18T22:57:25.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fadec1ab150 con 0x55c189949e50 2024-03-18T22:57:25.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fadec1a8390 con 0x55c189944ce0 2024-03-18T22:57:25.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadf4d6b740 10 client.0 _ll_register_callbacks cb 0x7fade0009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:25.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c1899532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadeaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c1899532d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49402/0 (socket says 172.21.15.40:49402) 2024-03-18T22:57:25.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadeaffd640 1 -- 172.21.15.40:0/4095471660 learned_addr learned my addr 172.21.15.40:0/4095471660 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadeaffd640 1 --2- 172.21.15.40:0/4095471660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189949e50 0x55c189954820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.518+0000 7fadeb7fe640 1 --2- 172.21.15.40:0/4095471660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189944ce0 0x55c189953810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadeaffd640 1 -- 172.21.15.40:0/4095471660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189944ce0 msgr2=0x55c189953810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadeaffd640 1 --2- 172.21.15.40:0/4095471660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189944ce0 0x55c189953810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadeaffd640 1 -- 172.21.15.40:0/4095471660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189949e50 msgr2=0x55c189954820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadeaffd640 1 --2- 172.21.15.40:0/4095471660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189949e50 0x55c189954820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadeaffd640 1 -- 172.21.15.40:0/4095471660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c189954e10 con 0x55c1899442f0 2024-03-18T22:57:25.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadeb7fe640 1 --2- 172.21.15.40:0/4095471660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189944ce0 0x55c189953810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:57:25.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadeaffd640 1 --2- 172.21.15.40:0/4095471660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c1899532d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fade00095f0 tx=0x7fade0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bd784b535364fb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:57:25.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadcaffd640 1 -- 172.21.15.40:0/4095471660 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade0016070 con 0x55c1899442f0 2024-03-18T22:57:25.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadcaffd640 1 -- 172.21.15.40:0/4095471660 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fade00054d0 con 0x55c1899442f0 2024-03-18T22:57:25.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadf4d6b740 1 -- 172.21.15.40:0/4095471660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fade0012270 con 0x55c1899442f0 2024-03-18T22:57:25.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadcaffd640 1 -- 172.21.15.40:0/4095471660 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fade00035f0 con 0x55c1899442f0 2024-03-18T22:57:25.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadf4d6b740 10 client.18528 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:25.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadf4d6b740 1 -- 172.21.15.40:0/4095471660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55c1899c0a30 con 0x55c1899442f0 2024-03-18T22:57:25.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadcaffd640 1 -- 172.21.15.40:0/4095471660 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fade0012270 con 0x55c1899442f0 2024-03-18T22:57:25.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadcaffd640 1 -- 172.21.15.40:0/4095471660 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fade002d550 con 0x55c1899442f0 2024-03-18T22:57:25.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadf4d6b740 10 client.18528 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:25.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadf4d6b740 10 client.18528 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:57:25.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadf4d6b740 20 client.18528 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:25.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/4095471660 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c1899c1170 0x55c1899c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadf4d6b740 4 client.18528 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:57:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.519+0000 7fadf4d6b740 1 -- 172.21.15.40:0/4095471660 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c1898b59f0 con 0x55c1899c1170 2024-03-18T22:57:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadeb7fe640 1 --2- 172.21.15.40:0/4095471660 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c1899c1170 0x55c1899c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadeb7fe640 1 --2- 172.21.15.40:0/4095471660 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c1899c1170 0x55c1899c3560 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=77692663d30c07a3 server_cookie=ef1e592da66bd710 in_seq=0 out_seq=0 2024-03-18T22:57:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadcaffd640 10 client.18528 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:57:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadcaffd640 1 -- 172.21.15.40:0/4095471660 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c1898b59f0 con 0x55c1899c1170 2024-03-18T22:57:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadcaffd640 10 client.18528 handle_command_reply: tid=0 2024-03-18T22:57:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadf4d6b740 1 client.18528 shutdown 2024-03-18T22:57:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadf4d6b740 1 -- 172.21.15.40:0/4095471660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 msgr2=0x55c1899532d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/4095471660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c1899532d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fade00095f0 tx=0x7fade0005dc0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadf4d6b740 1 -- 172.21.15.40:0/4095471660 shutdown_connections 2024-03-18T22:57:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/4095471660 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55c1899c1170 0x55c1899c3560 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/4095471660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c189949e50 0x55c189954820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadcaffd640 0 client.18528 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:57:25.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/4095471660 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c189944ce0 0x55c189953810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadf4d6b740 1 --2- 172.21.15.40:0/4095471660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c1899442f0 0x55c1899532d0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadf4d6b740 1 -- 172.21.15.40:0/4095471660 >> 172.21.15.40:0/4095471660 conn(0x55c18993f860 msgr2=0x55c18994e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadf4d6b740 1 -- 172.21.15.40:0/4095471660 shutdown_connections 2024-03-18T22:57:25.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadf4d6b740 1 -- 172.21.15.40:0/4095471660 wait complete. 2024-03-18T22:57:25.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.520+0000 7fadf4d6b740 20 client.18528 trim_cache size 0 max 16384 2024-03-18T22:57:25.529 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:57:25.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.521+0000 7fadf217e640 1 -- 172.21.15.40:0/2640062247 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadd40a2880 msgr2=0x7fadd40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.521+0000 7fadf217e640 1 --2- 172.21.15.40:0/2640062247 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadd40a2880 0x7fadd40a4d40 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fade400cba0 tx=0x7fade4086040 comp rx=0 tx=0).stop 2024-03-18T22:57:25.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.521+0000 7fadf217e640 1 -- 172.21.15.40:0/2640062247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadec1022f0 msgr2=0x7fadec19edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.521+0000 7fadf217e640 1 --2- 172.21.15.40:0/2640062247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadec1022f0 0x7fadec19edb0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fade400efd0 tx=0x7fade400c6a0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.521+0000 7fadf217e640 1 -- 172.21.15.40:0/2640062247 shutdown_connections 2024-03-18T22:57:25.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.521+0000 7fadf217e640 1 --2- 172.21.15.40:0/2640062247 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fadd40a2880 0x7fadd40a4d40 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.521+0000 7fadf217e640 1 --2- 172.21.15.40:0/2640062247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fadec1022f0 0x7fadec19edb0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.521+0000 7fadf217e640 1 --2- 172.21.15.40:0/2640062247 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fadec101930 0x7fadec19c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.521+0000 7fadf217e640 1 --2- 172.21.15.40:0/2640062247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadec100740 0x7fadec06a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.521+0000 7fadf217e640 1 -- 172.21.15.40:0/2640062247 >> 172.21.15.40:0/2640062247 conn(0x7fadec0fbcf0 msgr2=0x7fadec104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.521+0000 7fadf217e640 1 -- 172.21.15.40:0/2640062247 shutdown_connections 2024-03-18T22:57:25.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.521+0000 7fadf217e640 1 -- 172.21.15.40:0/2640062247 wait complete. 2024-03-18T22:57:25.532 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:57:25.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:25 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3616912151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:57:25.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:25 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3616912151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:57:25.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.636+0000 7fee73777640 1 -- 172.21.15.40:0/3864556603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6c100760 msgr2=0x7fee6c100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.636+0000 7fee73777640 1 --2- 172.21.15.40:0/3864556603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6c100760 0x7fee6c100b80 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fee68009900 tx=0x7fee6802bc30 comp rx=0 tx=0).stop 2024-03-18T22:57:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.636+0000 7fee73777640 1 -- 172.21.15.40:0/3864556603 shutdown_connections 2024-03-18T22:57:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.636+0000 7fee73777640 1 --2- 172.21.15.40:0/3864556603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee6c102310 0x7fee6c108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.636+0000 7fee73777640 1 --2- 172.21.15.40:0/3864556603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee6c101950 0x7fee6c101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.636+0000 7fee73777640 1 --2- 172.21.15.40:0/3864556603 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6c100760 0x7fee6c100b80 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.636+0000 7fee73777640 1 -- 172.21.15.40:0/3864556603 >> 172.21.15.40:0/3864556603 conn(0x7fee6c0fbcd0 msgr2=0x7fee6c0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.636+0000 7fee73777640 1 -- 172.21.15.40:0/3864556603 shutdown_connections 2024-03-18T22:57:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.636+0000 7fee73777640 1 -- 172.21.15.40:0/3864556603 wait complete. 2024-03-18T22:57:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee73777640 1 Processor -- start 2024-03-18T22:57:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee73777640 1 -- start start 2024-03-18T22:57:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee73777640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee6c100760 0x7fee6c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee73777640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee6c101950 0x7fee6c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee73777640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6c102310 0x7fee6c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee73777640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c10c290 con 0x7fee6c101950 2024-03-18T22:57:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee73777640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c10c110 con 0x7fee6c100760 2024-03-18T22:57:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee73777640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c10c410 con 0x7fee6c102310 2024-03-18T22:57:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee70d12640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee6c101950 0x7fee6c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee70d12640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee6c101950 0x7fee6c197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49418/0 (socket says 172.21.15.40:49418) 2024-03-18T22:57:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee70d12640 1 -- 172.21.15.40:0/1277119029 learned_addr learned my addr 172.21.15.40:0/1277119029 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee71513640 1 --2- 172.21.15.40:0/1277119029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee6c100760 0x7fee6c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee71d14640 1 --2- 172.21.15.40:0/1277119029 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6c102310 0x7fee6c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee70d12640 1 -- 172.21.15.40:0/1277119029 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6c102310 msgr2=0x7fee6c19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee70d12640 1 --2- 172.21.15.40:0/1277119029 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6c102310 0x7fee6c19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee70d12640 1 -- 172.21.15.40:0/1277119029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee6c100760 msgr2=0x7fee6c197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee70d12640 1 --2- 172.21.15.40:0/1277119029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee6c100760 0x7fee6c197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee70d12640 1 -- 172.21.15.40:0/1277119029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6c19f680 con 0x7fee6c101950 2024-03-18T22:57:25.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee71513640 1 --2- 172.21.15.40:0/1277119029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee6c100760 0x7fee6c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:25.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.637+0000 7fee70d12640 1 --2- 172.21.15.40:0/1277119029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee6c101950 0x7fee6c197fa0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fee5c00e9f0 tx=0x7fee5c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.638+0000 7fee627fc640 1 -- 172.21.15.40:0/1277119029 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c00cd80 con 0x7fee6c101950 2024-03-18T22:57:25.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.638+0000 7fee73777640 1 -- 172.21.15.40:0/1277119029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee6c19f970 con 0x7fee6c101950 2024-03-18T22:57:25.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.638+0000 7fee627fc640 1 -- 172.21.15.40:0/1277119029 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee5c0108e0 con 0x7fee6c101950 2024-03-18T22:57:25.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.638+0000 7fee627fc640 1 -- 172.21.15.40:0/1277119029 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c018920 con 0x7fee6c101950 2024-03-18T22:57:25.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.638+0000 7fee73777640 1 -- 172.21.15.40:0/1277119029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee6c19ff10 con 0x7fee6c101950 2024-03-18T22:57:25.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.640+0000 7fee627fc640 1 -- 172.21.15.40:0/1277119029 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fee5c010a90 con 0x7fee6c101950 2024-03-18T22:57:25.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.640+0000 7fee75ffd740 1 Processor -- start 2024-03-18T22:57:25.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.640+0000 7fee75ffd740 1 -- start start 2024-03-18T22:57:25.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.640+0000 7fee75ffd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d2700 0x5561790d2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.640+0000 7fee75ffd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790d7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.640+0000 7fee75ffd740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d8260 0x5561790da6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.640+0000 7fee75ffd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5561790324b0 con 0x5561790d2700 2024-03-18T22:57:25.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.640+0000 7fee75ffd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556178fdad70 con 0x5561790d30f0 2024-03-18T22:57:25.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.640+0000 7fee75ffd740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55617902e550 con 0x5561790d8260 2024-03-18T22:57:25.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee71d14640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d2700 0x5561790d2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee71513640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790d7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee71d14640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d2700 0x5561790d2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49426/0 (socket says 172.21.15.40:49426) 2024-03-18T22:57:25.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee71d14640 1 -- 172.21.15.40:0/3517152249 learned_addr learned my addr 172.21.15.40:0/3517152249 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee71513640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790d7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:38250/0 (socket says 172.21.15.40:38250) 2024-03-18T22:57:25.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee70d12640 1 --2- 172.21.15.40:0/3517152249 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d8260 0x5561790da6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee627fc640 1 --2- 172.21.15.40:0/1277119029 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fee580a2800 0x7fee580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee71513640 1 -- 172.21.15.40:0/3517152249 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d8260 msgr2=0x5561790da6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee71513640 1 --2- 172.21.15.40:0/3517152249 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d8260 0x5561790da6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee71513640 1 -- 172.21.15.40:0/3517152249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d2700 msgr2=0x5561790d2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee71513640 1 --2- 172.21.15.40:0/3517152249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d2700 0x5561790d2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee71513640 1 -- 172.21.15.40:0/3517152249 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556178ffd210 con 0x5561790d30f0 2024-03-18T22:57:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee71d14640 1 --2- 172.21.15.40:0/3517152249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d2700 0x5561790d2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee71d14640 1 --2- 172.21.15.40:0/1277119029 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fee580a2800 0x7fee580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee627fc640 1 -- 172.21.15.40:0/1277119029 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fee5c0cf160 con 0x7fee6c101950 2024-03-18T22:57:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee71513640 1 --2- 172.21.15.40:0/3517152249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790d7bf0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fee68036040 tx=0x7fee68009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91291776fce50dd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee577fe640 1 -- 172.21.15.40:0/3517152249 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee68045020 con 0x5561790d30f0 2024-03-18T22:57:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee71d14640 1 --2- 172.21.15.40:0/1277119029 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fee580a2800 0x7fee580a4cc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x5561790d3950 tx=0x7fee64004a40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee577fe640 1 -- 172.21.15.40:0/3517152249 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee6802cc90 con 0x5561790d30f0 2024-03-18T22:57:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.641+0000 7fee577fe640 1 -- 172.21.15.40:0/3517152249 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee68004920 con 0x5561790d30f0 2024-03-18T22:57:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.642+0000 7fee75ffd740 1 -- 172.21.15.40:0/3517152249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 msgr2=0x5561790d7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.642+0000 7fee75ffd740 1 --2- 172.21.15.40:0/3517152249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790d7bf0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fee68036040 tx=0x7fee68009580 comp rx=0 tx=0).stop 2024-03-18T22:57:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.642+0000 7fee75ffd740 1 -- 172.21.15.40:0/3517152249 shutdown_connections 2024-03-18T22:57:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.642+0000 7fee75ffd740 1 --2- 172.21.15.40:0/3517152249 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d8260 0x5561790da6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.642+0000 7fee75ffd740 1 --2- 172.21.15.40:0/3517152249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790d7bf0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.642+0000 7fee75ffd740 1 --2- 172.21.15.40:0/3517152249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d2700 0x5561790d2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.642+0000 7fee75ffd740 1 -- 172.21.15.40:0/3517152249 >> 172.21.15.40:0/3517152249 conn(0x5561790cdc70 msgr2=0x5561790ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.642+0000 7fee75ffd740 1 -- 172.21.15.40:0/3517152249 shutdown_connections 2024-03-18T22:57:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.642+0000 7fee75ffd740 1 -- 172.21.15.40:0/3517152249 wait complete. 2024-03-18T22:57:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.642+0000 7fee75ffd740 1 -- start start 2024-03-18T22:57:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.642+0000 7fee75ffd740 1 Processor -- start 2024-03-18T22:57:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee75ffd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d2700 0x5561790dcb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee75ffd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790dd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee75ffd740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d8260 0x5561790e40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee75ffd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5561790db1b0 con 0x5561790d2700 2024-03-18T22:57:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee75ffd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556178fee310 con 0x5561790d30f0 2024-03-18T22:57:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee75ffd740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5561790db330 con 0x5561790d8260 2024-03-18T22:57:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee75ffd740 10 client.0 _ll_register_callbacks cb 0x556178fe22d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee71513640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d2700 0x5561790dcb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee71513640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d2700 0x5561790dcb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49440/0 (socket says 172.21.15.40:49440) 2024-03-18T22:57:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee71d14640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d8260 0x5561790e40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee71513640 1 -- 172.21.15.40:0/3728367104 learned_addr learned my addr 172.21.15.40:0/3728367104 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee70d12640 1 --2- 172.21.15.40:0/3728367104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790dd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee71d14640 1 -- 172.21.15.40:0/3728367104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 msgr2=0x5561790dd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee71d14640 1 --2- 172.21.15.40:0/3728367104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790dd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee71d14640 1 -- 172.21.15.40:0/3728367104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d2700 msgr2=0x5561790dcb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee71d14640 1 --2- 172.21.15.40:0/3728367104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d2700 0x5561790dcb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee71d14640 1 -- 172.21.15.40:0/3728367104 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556178d463f0 con 0x5561790d8260 2024-03-18T22:57:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee71513640 1 --2- 172.21.15.40:0/3728367104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d2700 0x5561790dcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee71d14640 1 --2- 172.21.15.40:0/3728367104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d8260 0x5561790e40e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fee6400f790 tx=0x7fee64004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9270ed741e60007 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.643+0000 7fee54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:57:25.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.644+0000 7fee54ff9640 1 -- 172.21.15.40:0/3728367104 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64004d20 con 0x5561790d8260 2024-03-18T22:57:25.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.644+0000 7fee54ff9640 1 -- 172.21.15.40:0/3728367104 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fee6401c610 con 0x5561790d8260 2024-03-18T22:57:25.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.644+0000 7fee75ffd740 1 -- 172.21.15.40:0/3728367104 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee6802cc90 con 0x5561790d8260 2024-03-18T22:57:25.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.644+0000 7fee54ff9640 1 -- 172.21.15.40:0/3728367104 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fee64009cb0 con 0x5561790d8260 2024-03-18T22:57:25.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.644+0000 7fee75ffd740 10 client.28354 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:25.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.644+0000 7fee75ffd740 1 -- 172.21.15.40:0/3728367104 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x556178e107d0 con 0x5561790d8260 2024-03-18T22:57:25.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.644+0000 7fee54ff9640 1 -- 172.21.15.40:0/3728367104 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee6802cc90 con 0x5561790d8260 2024-03-18T22:57:25.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.644+0000 7fee54ff9640 1 -- 172.21.15.40:0/3728367104 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fee64026070 con 0x5561790d8260 2024-03-18T22:57:25.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.644+0000 7fee75ffd740 10 client.28354 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:25.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.644+0000 7fee75ffd740 10 client.28354 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:57:25.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.644+0000 7fee75ffd740 20 client.28354 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:25.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.644+0000 7fee75ffd740 1 --2- 172.21.15.40:0/3728367104 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5561790e8b60 0x5561790eb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.644+0000 7fee75ffd740 4 client.28354 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:57:25.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.644+0000 7fee75ffd740 1 -- 172.21.15.40:0/3728367104 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561790903e0 con 0x5561790e8b60 2024-03-18T22:57:25.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.644+0000 7fee71513640 1 --2- 172.21.15.40:0/3728367104 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5561790e8b60 0x5561790eb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.645+0000 7fee71513640 1 --2- 172.21.15.40:0/3728367104 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5561790e8b60 0x5561790eb020 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f8c06bd95ed34aaa server_cookie=1bc19323144010e3 in_seq=0 out_seq=0 2024-03-18T22:57:25.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.645+0000 7fee54ff9640 10 client.28354 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:57:25.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.646+0000 7fee54ff9640 1 -- 172.21.15.40:0/3728367104 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5561790903e0 con 0x5561790e8b60 2024-03-18T22:57:25.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.646+0000 7fee54ff9640 10 client.28354 handle_command_reply: tid=0 2024-03-18T22:57:25.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.646+0000 7fee75ffd740 1 client.28354 shutdown 2024-03-18T22:57:25.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.646+0000 7fee75ffd740 1 -- 172.21.15.40:0/3728367104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d8260 msgr2=0x5561790e40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.646+0000 7fee75ffd740 1 --2- 172.21.15.40:0/3728367104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d8260 0x5561790e40e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fee6400f790 tx=0x7fee64004ab0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.646+0000 7fee75ffd740 1 -- 172.21.15.40:0/3728367104 shutdown_connections 2024-03-18T22:57:25.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.646+0000 7fee75ffd740 1 --2- 172.21.15.40:0/3728367104 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x5561790e8b60 0x5561790eb020 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.646+0000 7fee75ffd740 1 --2- 172.21.15.40:0/3728367104 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d8260 0x5561790e40e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.646+0000 7fee75ffd740 1 --2- 172.21.15.40:0/3728367104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790dd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.646+0000 7fee75ffd740 1 --2- 172.21.15.40:0/3728367104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d2700 0x5561790dcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.646+0000 7fee54ff9640 0 client.28354 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:57:25.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.646+0000 7fee75ffd740 1 -- 172.21.15.40:0/3728367104 >> 172.21.15.40:0/3728367104 conn(0x5561790cdc70 msgr2=0x5561790c8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.646+0000 7fee75ffd740 1 -- 172.21.15.40:0/3728367104 shutdown_connections 2024-03-18T22:57:25.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.646+0000 7fee75ffd740 1 -- 172.21.15.40:0/3728367104 wait complete. 2024-03-18T22:57:25.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.646+0000 7fee75ffd740 20 client.28354 trim_cache size 0 max 16384 2024-03-18T22:57:25.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.659+0000 7fee75ffd740 1 Processor -- start 2024-03-18T22:57:25.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.659+0000 7fee75ffd740 1 -- start start 2024-03-18T22:57:25.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.659+0000 7fee75ffd740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d2700 0x5561790c8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.659+0000 7fee75ffd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790e40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.659+0000 7fee75ffd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d8260 0x5561790c8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.659+0000 7fee75ffd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c1a82e0 con 0x5561790d8260 2024-03-18T22:57:25.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.659+0000 7fee75ffd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c1a9e20 con 0x5561790d30f0 2024-03-18T22:57:25.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.659+0000 7fee75ffd740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c1a89b0 con 0x5561790d2700 2024-03-18T22:57:25.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.659+0000 7fee71d14640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d8260 0x5561790c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.660+0000 7fee71d14640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d8260 0x5561790c8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49450/0 (socket says 172.21.15.40:49450) 2024-03-18T22:57:25.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.660+0000 7fee71d14640 1 -- 172.21.15.40:0/3366447792 learned_addr learned my addr 172.21.15.40:0/3366447792 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.660+0000 7fee71513640 1 --2- 172.21.15.40:0/3366447792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d2700 0x5561790c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.660+0000 7fee70d12640 1 --2- 172.21.15.40:0/3366447792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790e40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.660+0000 7fee71d14640 1 -- 172.21.15.40:0/3366447792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d2700 msgr2=0x5561790c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.660+0000 7fee71d14640 1 --2- 172.21.15.40:0/3366447792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d2700 0x5561790c8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.660+0000 7fee71d14640 1 -- 172.21.15.40:0/3366447792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 msgr2=0x5561790e40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.660+0000 7fee71d14640 1 --2- 172.21.15.40:0/3366447792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790e40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.660+0000 7fee71d14640 1 -- 172.21.15.40:0/3366447792 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561790eb6a0 con 0x5561790d8260 2024-03-18T22:57:25.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.660+0000 7fee71d14640 1 --2- 172.21.15.40:0/3366447792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d8260 0x5561790c8850 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fee64000f80 tx=0x7fee64009320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe28d8085e7012f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.660+0000 7fee56ffd640 1 -- 172.21.15.40:0/3366447792 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6403a070 con 0x5561790d8260 2024-03-18T22:57:25.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.660+0000 7fee56ffd640 1 -- 172.21.15.40:0/3366447792 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee64009500 con 0x5561790d8260 2024-03-18T22:57:25.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.660+0000 7fee56ffd640 1 -- 172.21.15.40:0/3366447792 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6401ca30 con 0x5561790d8260 2024-03-18T22:57:25.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 -- 172.21.15.40:0/3366447792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d8260 msgr2=0x5561790c8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 --2- 172.21.15.40:0/3366447792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d8260 0x5561790c8850 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fee64000f80 tx=0x7fee64009320 comp rx=0 tx=0).stop 2024-03-18T22:57:25.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 -- 172.21.15.40:0/3366447792 shutdown_connections 2024-03-18T22:57:25.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 --2- 172.21.15.40:0/3366447792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d8260 0x5561790c8850 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 --2- 172.21.15.40:0/3366447792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790e40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 --2- 172.21.15.40:0/3366447792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d2700 0x5561790c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 -- 172.21.15.40:0/3366447792 >> 172.21.15.40:0/3366447792 conn(0x5561790cdc70 msgr2=0x5561790d4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 -- 172.21.15.40:0/3366447792 shutdown_connections 2024-03-18T22:57:25.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 -- 172.21.15.40:0/3366447792 wait complete. 2024-03-18T22:57:25.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 -- start start 2024-03-18T22:57:25.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 Processor -- start 2024-03-18T22:57:25.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d2700 0x5561790e2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790e2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d8260 0x5561790e3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c1a9750 con 0x5561790d8260 2024-03-18T22:57:25.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c1a9080 con 0x5561790d30f0 2024-03-18T22:57:25.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c1a62c0 con 0x5561790d2700 2024-03-18T22:57:25.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee75ffd740 10 client.0 _ll_register_callbacks cb 0x7fee64010550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:25.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee71d14640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d8260 0x5561790e3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee70d12640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790e2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee71513640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d2700 0x5561790e2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee71d14640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d8260 0x5561790e3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49462/0 (socket says 172.21.15.40:49462) 2024-03-18T22:57:25.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee70d12640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790e2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:38272/0 (socket says 172.21.15.40:38272) 2024-03-18T22:57:25.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee71d14640 1 -- 172.21.15.40:0/33731730 learned_addr learned my addr 172.21.15.40:0/33731730 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.661+0000 7fee71513640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d2700 0x5561790e2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:47930/0 (socket says 172.21.15.40:47930) 2024-03-18T22:57:25.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee71513640 1 -- 172.21.15.40:0/33731730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 msgr2=0x5561790e2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee71513640 1 --2- 172.21.15.40:0/33731730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790e2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee71513640 1 -- 172.21.15.40:0/33731730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d8260 msgr2=0x5561790e3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee71513640 1 --2- 172.21.15.40:0/33731730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d8260 0x5561790e3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee71513640 1 -- 172.21.15.40:0/33731730 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561790cd810 con 0x5561790d2700 2024-03-18T22:57:25.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee71d14640 1 --2- 172.21.15.40:0/33731730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d8260 0x5561790e3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:25.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee71513640 1 --2- 172.21.15.40:0/33731730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d2700 0x5561790e2020 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x5561790eb8a0 tx=0x7fee68009670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5d10df8e896ea9b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee54ff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:57:25.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee54ff9640 1 -- 172.21.15.40:0/33731730 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee68053070 con 0x5561790d2700 2024-03-18T22:57:25.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee54ff9640 1 -- 172.21.15.40:0/33731730 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fee68005450 con 0x5561790d2700 2024-03-18T22:57:25.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee75ffd740 1 -- 172.21.15.40:0/33731730 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55617903f9f0 con 0x5561790d2700 2024-03-18T22:57:25.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee54ff9640 1 -- 172.21.15.40:0/33731730 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fee6805f050 con 0x5561790d2700 2024-03-18T22:57:25.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee75ffd740 10 client.28360 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:25.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee75ffd740 1 -- 172.21.15.40:0/33731730 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55617909f880 con 0x5561790d2700 2024-03-18T22:57:25.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee54ff9640 1 -- 172.21.15.40:0/33731730 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55617903f9f0 con 0x5561790d2700 2024-03-18T22:57:25.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee54ff9640 1 -- 172.21.15.40:0/33731730 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fee680047e0 con 0x5561790d2700 2024-03-18T22:57:25.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.662+0000 7fee75ffd740 10 client.28360 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:25.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 10 client.28360 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:57:25.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 20 client.28360 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:25.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 1 --2- 172.21.15.40:0/33731730 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55617914f9f0 0x556179151de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 4 client.28360 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:57:25.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 1 -- 172.21.15.40:0/33731730 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556179039d40 con 0x55617914f9f0 2024-03-18T22:57:25.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee70d12640 1 --2- 172.21.15.40:0/33731730 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55617914f9f0 0x556179151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee54ff9640 10 client.28360 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:57:25.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee70d12640 1 --2- 172.21.15.40:0/33731730 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55617914f9f0 0x556179151de0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fcd8e47413a0908 server_cookie=19a6ec19d511e97b in_seq=0 out_seq=0 2024-03-18T22:57:25.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee54ff9640 1 -- 172.21.15.40:0/33731730 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556179039d40 con 0x55617914f9f0 2024-03-18T22:57:25.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee54ff9640 10 client.28360 handle_command_reply: tid=0 2024-03-18T22:57:25.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 1 client.28360 shutdown 2024-03-18T22:57:25.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 1 -- 172.21.15.40:0/33731730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d2700 msgr2=0x5561790e2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 1 --2- 172.21.15.40:0/33731730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d2700 0x5561790e2020 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x5561790eb8a0 tx=0x7fee68009670 comp rx=0 tx=0).stop 2024-03-18T22:57:25.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 1 -- 172.21.15.40:0/33731730 shutdown_connections 2024-03-18T22:57:25.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 1 --2- 172.21.15.40:0/33731730 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55617914f9f0 0x556179151de0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 1 --2- 172.21.15.40:0/33731730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5561790d8260 0x5561790e3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 1 --2- 172.21.15.40:0/33731730 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561790d30f0 0x5561790e2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee54ff9640 0 client.28360 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:57:25.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 1 --2- 172.21.15.40:0/33731730 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5561790d2700 0x5561790e2020 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 1 -- 172.21.15.40:0/33731730 >> 172.21.15.40:0/33731730 conn(0x5561790cdc70 msgr2=0x5561790c8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 1 -- 172.21.15.40:0/33731730 shutdown_connections 2024-03-18T22:57:25.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.663+0000 7fee75ffd740 1 -- 172.21.15.40:0/33731730 wait complete. 2024-03-18T22:57:25.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.664+0000 7fee75ffd740 20 client.28360 trim_cache size 0 max 16384 2024-03-18T22:57:25.671 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:57:25.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.664+0000 7fee73777640 1 -- 172.21.15.40:0/1277119029 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fee580a2800 msgr2=0x7fee580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.664+0000 7fee73777640 1 --2- 172.21.15.40:0/1277119029 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fee580a2800 0x7fee580a4cc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x5561790d3950 tx=0x7fee64004a40 comp rx=0 tx=0).stop 2024-03-18T22:57:25.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.664+0000 7fee73777640 1 -- 172.21.15.40:0/1277119029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee6c101950 msgr2=0x7fee6c197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.664+0000 7fee73777640 1 --2- 172.21.15.40:0/1277119029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee6c101950 0x7fee6c197fa0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fee5c00e9f0 tx=0x7fee5c00eeb0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.664+0000 7fee73777640 1 -- 172.21.15.40:0/1277119029 shutdown_connections 2024-03-18T22:57:25.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.664+0000 7fee73777640 1 --2- 172.21.15.40:0/1277119029 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fee580a2800 0x7fee580a4cc0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.664+0000 7fee73777640 1 --2- 172.21.15.40:0/1277119029 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fee6c102310 0x7fee6c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.664+0000 7fee73777640 1 --2- 172.21.15.40:0/1277119029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee6c101950 0x7fee6c197fa0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.664+0000 7fee73777640 1 --2- 172.21.15.40:0/1277119029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fee6c100760 0x7fee6c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.664+0000 7fee73777640 1 -- 172.21.15.40:0/1277119029 >> 172.21.15.40:0/1277119029 conn(0x7fee6c0fbcd0 msgr2=0x7fee6c0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.664+0000 7fee73777640 1 -- 172.21.15.40:0/1277119029 shutdown_connections 2024-03-18T22:57:25.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.664+0000 7fee73777640 1 -- 172.21.15.40:0/1277119029 wait complete. 2024-03-18T22:57:25.675 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:57:25.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.780+0000 7fe5eca59640 1 -- 172.21.15.40:0/4030969951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5e8100750 msgr2=0x7fe5e8100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.780+0000 7fe5eca59640 1 --2- 172.21.15.40:0/4030969951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5e8100750 0x7fe5e8100b70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8009ca0 tx=0x7fe5d802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.780+0000 7fe5eca59640 1 -- 172.21.15.40:0/4030969951 shutdown_connections 2024-03-18T22:57:25.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.780+0000 7fe5eca59640 1 --2- 172.21.15.40:0/4030969951 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5e8102300 0x7fe5e8108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.780+0000 7fe5eca59640 1 --2- 172.21.15.40:0/4030969951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5e8101940 0x7fe5e8101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.780+0000 7fe5eca59640 1 --2- 172.21.15.40:0/4030969951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5e8100750 0x7fe5e8100b70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.780+0000 7fe5eca59640 1 -- 172.21.15.40:0/4030969951 >> 172.21.15.40:0/4030969951 conn(0x7fe5e80fbcc0 msgr2=0x7fe5e80fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.780+0000 7fe5eca59640 1 -- 172.21.15.40:0/4030969951 shutdown_connections 2024-03-18T22:57:25.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.780+0000 7fe5eca59640 1 -- 172.21.15.40:0/4030969951 wait complete. 2024-03-18T22:57:25.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5eca59640 1 Processor -- start 2024-03-18T22:57:25.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5eca59640 1 -- start start 2024-03-18T22:57:25.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5eca59640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5e8100750 0x7fe5e81046e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5eca59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5e8101940 0x7fe5e8102d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5eca59640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5e8102300 0x7fe5e8103260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5eca59640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e810c250 con 0x7fe5e8101940 2024-03-18T22:57:25.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5eca59640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e810c0d0 con 0x7fe5e8100750 2024-03-18T22:57:25.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5eca59640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e810c3d0 con 0x7fe5e8102300 2024-03-18T22:57:25.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5e5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5e8101940 0x7fe5e8102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5e5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5e8101940 0x7fe5e8102d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49490/0 (socket says 172.21.15.40:49490) 2024-03-18T22:57:25.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5e659c640 1 --2- 172.21.15.40:0/680830343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5e8100750 0x7fe5e81046e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5e5d9b640 1 -- 172.21.15.40:0/680830343 learned_addr learned my addr 172.21.15.40:0/680830343 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5e6d9d640 1 --2- 172.21.15.40:0/680830343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5e8102300 0x7fe5e8103260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5e5d9b640 1 -- 172.21.15.40:0/680830343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5e8102300 msgr2=0x7fe5e8103260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5e5d9b640 1 --2- 172.21.15.40:0/680830343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5e8102300 0x7fe5e8103260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5e5d9b640 1 -- 172.21.15.40:0/680830343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5e8100750 msgr2=0x7fe5e81046e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5e5d9b640 1 --2- 172.21.15.40:0/680830343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5e8100750 0x7fe5e81046e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.781+0000 7fe5e5d9b640 1 -- 172.21.15.40:0/680830343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5e8103af0 con 0x7fe5e8101940 2024-03-18T22:57:25.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.782+0000 7fe5e5d9b640 1 --2- 172.21.15.40:0/680830343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5e8101940 0x7fe5e8102d20 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc00e9f0 tx=0x7fe5dc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.782+0000 7fe5d77fe640 1 -- 172.21.15.40:0/680830343 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc00ccc0 con 0x7fe5e8101940 2024-03-18T22:57:25.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.782+0000 7fe5d77fe640 1 -- 172.21.15.40:0/680830343 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5dc0108e0 con 0x7fe5e8101940 2024-03-18T22:57:25.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.782+0000 7fe5eca59640 1 -- 172.21.15.40:0/680830343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5e81a1bf0 con 0x7fe5e8101940 2024-03-18T22:57:25.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.782+0000 7fe5d77fe640 1 -- 172.21.15.40:0/680830343 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc0187c0 con 0x7fe5e8101940 2024-03-18T22:57:25.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.782+0000 7fe5eca59640 1 -- 172.21.15.40:0/680830343 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5e81a2140 con 0x7fe5e8101940 2024-03-18T22:57:25.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.784+0000 7fe5d77fe640 1 -- 172.21.15.40:0/680830343 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe5dc010a90 con 0x7fe5e8101940 2024-03-18T22:57:25.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.784+0000 7fe5ef640740 1 Processor -- start 2024-03-18T22:57:25.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5ef640740 1 -- start start 2024-03-18T22:57:25.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5d77fe640 1 --2- 172.21.15.40:0/680830343 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe5d00a2800 0x7fe5d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5ef640740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f02f0 0x55e6862f0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5ef640740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f0ce0 0x55e6862f57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5ef640740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f5e50 0x55e6862f82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5ef640740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e6862544b0 con 0x55e6862f0ce0 2024-03-18T22:57:25.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5ef640740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e6861fcd70 con 0x55e6862f5e50 2024-03-18T22:57:25.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5ef640740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e686250550 con 0x55e6862f02f0 2024-03-18T22:57:25.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5d77fe640 1 -- 172.21.15.40:0/680830343 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe5dc0cf2a0 con 0x7fe5e8101940 2024-03-18T22:57:25.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5e6d9d640 1 --2- 172.21.15.40:0/680830343 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe5d00a2800 0x7fe5d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5e5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f0ce0 0x55e6862f57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5e5d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f0ce0 0x55e6862f57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49498/0 (socket says 172.21.15.40:49498) 2024-03-18T22:57:25.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5e659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f02f0 0x55e6862f0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5e5d9b640 1 -- 172.21.15.40:0/2967490629 learned_addr learned my addr 172.21.15.40:0/2967490629 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5e6d9d640 1 --2- 172.21.15.40:0/2967490629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f5e50 0x55e6862f82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5e659c640 1 -- 172.21.15.40:0/2967490629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f5e50 msgr2=0x55e6862f82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5e659c640 1 --2- 172.21.15.40:0/2967490629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f5e50 0x55e6862f82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5e659c640 1 -- 172.21.15.40:0/2967490629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f0ce0 msgr2=0x55e6862f57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5e659c640 1 --2- 172.21.15.40:0/2967490629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f0ce0 0x55e6862f57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5e659c640 1 -- 172.21.15.40:0/2967490629 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e68621f210 con 0x55e6862f02f0 2024-03-18T22:57:25.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5e5d9b640 1 --2- 172.21.15.40:0/2967490629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f0ce0 0x55e6862f57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:25.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.785+0000 7fe5e6d9d640 1 --2- 172.21.15.40:0/680830343 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe5d00a2800 0x7fe5d00a4cc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0011490 tx=0x7fe5e0011890 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.786+0000 7fe5e659c640 1 --2- 172.21.15.40:0/2967490629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f02f0 0x55e6862f0710 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe5d802c630 tx=0x7fe5d80093f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=561808992f5a67c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.786+0000 7fe5d4ff9640 1 -- 172.21.15.40:0/2967490629 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8043070 con 0x55e6862f02f0 2024-03-18T22:57:25.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.786+0000 7fe5d4ff9640 1 -- 172.21.15.40:0/2967490629 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5d8032db0 con 0x55e6862f02f0 2024-03-18T22:57:25.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.786+0000 7fe5d4ff9640 1 -- 172.21.15.40:0/2967490629 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8039040 con 0x55e6862f02f0 2024-03-18T22:57:25.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.786+0000 7fe5ef640740 1 -- 172.21.15.40:0/2967490629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f02f0 msgr2=0x55e6862f0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.786+0000 7fe5ef640740 1 --2- 172.21.15.40:0/2967490629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f02f0 0x55e6862f0710 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe5d802c630 tx=0x7fe5d80093f0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.786+0000 7fe5ef640740 1 -- 172.21.15.40:0/2967490629 shutdown_connections 2024-03-18T22:57:25.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.786+0000 7fe5ef640740 1 --2- 172.21.15.40:0/2967490629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f5e50 0x55e6862f82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.786+0000 7fe5ef640740 1 --2- 172.21.15.40:0/2967490629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f0ce0 0x55e6862f57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.786+0000 7fe5ef640740 1 --2- 172.21.15.40:0/2967490629 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f02f0 0x55e6862f0710 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.786+0000 7fe5ef640740 1 -- 172.21.15.40:0/2967490629 >> 172.21.15.40:0/2967490629 conn(0x55e6862eb860 msgr2=0x55e6862ebc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5ef640740 1 -- 172.21.15.40:0/2967490629 shutdown_connections 2024-03-18T22:57:25.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5ef640740 1 -- 172.21.15.40:0/2967490629 wait complete. 2024-03-18T22:57:25.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5ef640740 1 -- start start 2024-03-18T22:57:25.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5ef640740 1 Processor -- start 2024-03-18T22:57:25.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5ef640740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f02f0 0x55e6862fa6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5ef640740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f0ce0 0x55e6862fac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5ef640740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f5e50 0x55e686301c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5ef640740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e6862f8da0 con 0x55e6862f02f0 2024-03-18T22:57:25.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5ef640740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e686210310 con 0x55e6862f0ce0 2024-03-18T22:57:25.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5ef640740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e6862f8f20 con 0x55e6862f5e50 2024-03-18T22:57:25.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5ef640740 10 client.0 _ll_register_callbacks cb 0x55e6862042d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:25.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5e659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f02f0 0x55e6862fa6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5e5d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f0ce0 0x55e6862fac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5e6d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f5e50 0x55e686301c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5e6d9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f5e50 0x55e686301c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:47978/0 (socket says 172.21.15.40:47978) 2024-03-18T22:57:25.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5e659c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f02f0 0x55e6862fa6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49506/0 (socket says 172.21.15.40:49506) 2024-03-18T22:57:25.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5e6d9d640 1 -- 172.21.15.40:0/649946416 learned_addr learned my addr 172.21.15.40:0/649946416 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5e5d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f0ce0 0x55e6862fac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:38312/0 (socket says 172.21.15.40:38312) 2024-03-18T22:57:25.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5e6d9d640 1 -- 172.21.15.40:0/649946416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f0ce0 msgr2=0x55e6862fac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5e6d9d640 1 --2- 172.21.15.40:0/649946416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f0ce0 0x55e6862fac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5e6d9d640 1 -- 172.21.15.40:0/649946416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f02f0 msgr2=0x55e6862fa6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5e6d9d640 1 --2- 172.21.15.40:0/649946416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f02f0 0x55e6862fa6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.787+0000 7fe5e6d9d640 1 -- 172.21.15.40:0/649946416 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e685f683f0 con 0x55e6862f5e50 2024-03-18T22:57:25.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5e6d9d640 1 --2- 172.21.15.40:0/649946416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f5e50 0x55e686301c60 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000fa10 tx=0x7fe5e0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ae0c226e853ca26 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:57:25.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5cdffb640 1 -- 172.21.15.40:0/649946416 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0024070 con 0x55e6862f5e50 2024-03-18T22:57:25.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5cdffb640 1 -- 172.21.15.40:0/649946416 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe5e0004260 con 0x55e6862f5e50 2024-03-18T22:57:25.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5ef640740 1 -- 172.21.15.40:0/649946416 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5d8032db0 con 0x55e6862f5e50 2024-03-18T22:57:25.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5cdffb640 1 -- 172.21.15.40:0/649946416 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe5e0030050 con 0x55e6862f5e50 2024-03-18T22:57:25.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5ef640740 10 client.28372 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:25.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5ef640740 1 -- 172.21.15.40:0/649946416 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e6860327d0 con 0x55e6862f5e50 2024-03-18T22:57:25.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5cdffb640 1 -- 172.21.15.40:0/649946416 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5d8032db0 con 0x55e6862f5e50 2024-03-18T22:57:25.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5cdffb640 1 -- 172.21.15.40:0/649946416 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fe5e00038e0 con 0x55e6862f5e50 2024-03-18T22:57:25.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5ef640740 10 client.28372 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:25.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5ef640740 10 client.28372 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:57:25.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5ef640740 20 client.28372 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:25.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5ef640740 1 --2- 172.21.15.40:0/649946416 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e6863065c0 0x55e686308a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5ef640740 4 client.28372 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:57:25.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.788+0000 7fe5ef640740 1 -- 172.21.15.40:0/649946416 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6862b23e0 con 0x55e6863065c0 2024-03-18T22:57:25.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.789+0000 7fe5e659c640 1 --2- 172.21.15.40:0/649946416 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e6863065c0 0x55e686308a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.789+0000 7fe5cdffb640 10 client.28372 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:25.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.789+0000 7fe5e659c640 1 --2- 172.21.15.40:0/649946416 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e6863065c0 0x55e686308a80 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61f3ae7302b6f2f6 server_cookie=10a7284359ec80bf in_seq=0 out_seq=0 2024-03-18T22:57:25.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.790+0000 7fe5cdffb640 1 -- 172.21.15.40:0/649946416 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e6862b23e0 con 0x55e6863065c0 2024-03-18T22:57:25.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.790+0000 7fe5cdffb640 10 client.28372 handle_command_reply: tid=0 2024-03-18T22:57:25.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.790+0000 7fe5ef640740 1 client.28372 shutdown 2024-03-18T22:57:25.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.790+0000 7fe5ef640740 1 -- 172.21.15.40:0/649946416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f5e50 msgr2=0x55e686301c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.790+0000 7fe5ef640740 1 --2- 172.21.15.40:0/649946416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f5e50 0x55e686301c60 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000fa10 tx=0x7fe5e0004730 comp rx=0 tx=0).stop 2024-03-18T22:57:25.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.790+0000 7fe5ef640740 1 -- 172.21.15.40:0/649946416 shutdown_connections 2024-03-18T22:57:25.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.790+0000 7fe5ef640740 1 --2- 172.21.15.40:0/649946416 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f5e50 0x55e686301c60 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.790+0000 7fe5ef640740 1 --2- 172.21.15.40:0/649946416 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e6863065c0 0x55e686308a80 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.790+0000 7fe5ef640740 1 --2- 172.21.15.40:0/649946416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f0ce0 0x55e6862fac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.790+0000 7fe5ef640740 1 --2- 172.21.15.40:0/649946416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f02f0 0x55e6862fa6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.790+0000 7fe5ef640740 1 -- 172.21.15.40:0/649946416 >> 172.21.15.40:0/649946416 conn(0x55e6862eb860 msgr2=0x55e6862eaba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.790+0000 7fe5cdffb640 0 client.28372 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:25.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.790+0000 7fe5ef640740 1 -- 172.21.15.40:0/649946416 shutdown_connections 2024-03-18T22:57:25.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.790+0000 7fe5ef640740 1 -- 172.21.15.40:0/649946416 wait complete. 2024-03-18T22:57:25.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.790+0000 7fe5ef640740 20 client.28372 trim_cache size 0 max 16384 2024-03-18T22:57:25.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5ef640740 1 Processor -- start 2024-03-18T22:57:25.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5ef640740 1 -- start start 2024-03-18T22:57:25.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5ef640740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f02f0 0x55e6862ea310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5ef640740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f0ce0 0x55e686301c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5ef640740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f5e50 0x55e6862ea850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5ef640740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e81ac760 con 0x55e6862f5e50 2024-03-18T22:57:25.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5ef640740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e81ae2a0 con 0x55e6862f02f0 2024-03-18T22:57:25.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5ef640740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e81ace30 con 0x55e6862f0ce0 2024-03-18T22:57:25.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5e6d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f5e50 0x55e6862ea850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5e5d9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f0ce0 0x55e686301c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5e6d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f5e50 0x55e6862ea850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49514/0 (socket says 172.21.15.40:49514) 2024-03-18T22:57:25.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5e6d9d640 1 -- 172.21.15.40:0/2680742306 learned_addr learned my addr 172.21.15.40:0/2680742306 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5e659c640 1 --2- 172.21.15.40:0/2680742306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f02f0 0x55e6862ea310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5e6d9d640 1 -- 172.21.15.40:0/2680742306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f0ce0 msgr2=0x55e686301c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5e6d9d640 1 --2- 172.21.15.40:0/2680742306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f0ce0 0x55e686301c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5e6d9d640 1 -- 172.21.15.40:0/2680742306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f02f0 msgr2=0x55e6862ea310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5e6d9d640 1 --2- 172.21.15.40:0/2680742306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f02f0 0x55e6862ea310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.804+0000 7fe5e6d9d640 1 -- 172.21.15.40:0/2680742306 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6862eb400 con 0x55e6862f5e50 2024-03-18T22:57:25.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.805+0000 7fe5e6d9d640 1 --2- 172.21.15.40:0/2680742306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f5e50 0x55e6862ea850 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fe5e00091c0 tx=0x7fe5e00091f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a81db8f9ce691a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.805+0000 7fe5cffff640 1 -- 172.21.15.40:0/2680742306 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e0024040 con 0x55e6862f5e50 2024-03-18T22:57:25.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.805+0000 7fe5cffff640 1 -- 172.21.15.40:0/2680742306 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5e0003aa0 con 0x55e6862f5e50 2024-03-18T22:57:25.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.805+0000 7fe5cffff640 1 -- 172.21.15.40:0/2680742306 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e001a040 con 0x55e6862f5e50 2024-03-18T22:57:25.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.805+0000 7fe5ef640740 1 -- 172.21.15.40:0/2680742306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f5e50 msgr2=0x55e6862ea850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.805+0000 7fe5ef640740 1 --2- 172.21.15.40:0/2680742306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f5e50 0x55e6862ea850 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fe5e00091c0 tx=0x7fe5e00091f0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.805+0000 7fe5ef640740 1 -- 172.21.15.40:0/2680742306 shutdown_connections 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.805+0000 7fe5ef640740 1 --2- 172.21.15.40:0/2680742306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f5e50 0x55e6862ea850 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.805+0000 7fe5ef640740 1 --2- 172.21.15.40:0/2680742306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f0ce0 0x55e686301c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.805+0000 7fe5ef640740 1 --2- 172.21.15.40:0/2680742306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f02f0 0x55e6862ea310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.805+0000 7fe5ef640740 1 -- 172.21.15.40:0/2680742306 >> 172.21.15.40:0/2680742306 conn(0x55e6862eb860 msgr2=0x55e6862f2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.805+0000 7fe5ef640740 1 -- 172.21.15.40:0/2680742306 shutdown_connections 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.805+0000 7fe5ef640740 1 -- 172.21.15.40:0/2680742306 wait complete. 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5ef640740 1 -- start start 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5ef640740 1 Processor -- start 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5ef640740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f02f0 0x55e6862ff6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5ef640740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f0ce0 0x55e6862ffc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5ef640740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f5e50 0x55e686300c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5ef640740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e81adbd0 con 0x55e6862f5e50 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5ef640740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e81ad500 con 0x55e6862f0ce0 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5ef640740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe5e81aa740 con 0x55e6862f02f0 2024-03-18T22:57:25.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5ef640740 10 client.0 _ll_register_callbacks cb 0x7fe5e000f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5e659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f02f0 0x55e6862ff6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5e659c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f02f0 0x55e6862ff6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:47986/0 (socket says 172.21.15.40:47986) 2024-03-18T22:57:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5e6d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f5e50 0x55e686300c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5e659c640 1 -- 172.21.15.40:0/3345240724 learned_addr learned my addr 172.21.15.40:0/3345240724 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5e5d9b640 1 --2- 172.21.15.40:0/3345240724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f0ce0 0x55e6862ffc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5e659c640 1 -- 172.21.15.40:0/3345240724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f0ce0 msgr2=0x55e6862ffc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5e659c640 1 --2- 172.21.15.40:0/3345240724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f0ce0 0x55e6862ffc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5e659c640 1 -- 172.21.15.40:0/3345240724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f5e50 msgr2=0x55e686300c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5e659c640 1 --2- 172.21.15.40:0/3345240724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f5e50 0x55e686300c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5e659c640 1 -- 172.21.15.40:0/3345240724 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6862c1880 con 0x55e6862f02f0 2024-03-18T22:57:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.806+0000 7fe5e6d9d640 1 --2- 172.21.15.40:0/3345240724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f5e50 0x55e686300c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5e659c640 1 --2- 172.21.15.40:0/3345240724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f02f0 0x55e6862ff6d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8009a60 tx=0x7fe5d802c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee765d3dfff174a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:57:25.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5cdffb640 1 -- 172.21.15.40:0/3345240724 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8039040 con 0x55e6862f02f0 2024-03-18T22:57:25.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5ef640740 1 -- 172.21.15.40:0/3345240724 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5e0003aa0 con 0x55e6862f02f0 2024-03-18T22:57:25.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5cdffb640 1 -- 172.21.15.40:0/3345240724 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe5d8032d70 con 0x55e6862f02f0 2024-03-18T22:57:25.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5cdffb640 1 -- 172.21.15.40:0/3345240724 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fe5d8061050 con 0x55e6862f02f0 2024-03-18T22:57:25.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5ef640740 10 client.28378 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:25.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5ef640740 1 -- 172.21.15.40:0/3345240724 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e68636cb90 con 0x55e6862f02f0 2024-03-18T22:57:25.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5cdffb640 1 -- 172.21.15.40:0/3345240724 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5e0003aa0 con 0x55e6862f02f0 2024-03-18T22:57:25.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5cdffb640 1 -- 172.21.15.40:0/3345240724 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fe5d8051630 con 0x55e6862f02f0 2024-03-18T22:57:25.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5ef640740 10 client.28378 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:25.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5ef640740 10 client.28378 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:57:25.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5ef640740 20 client.28378 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:25.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5ef640740 1 --2- 172.21.15.40:0/3345240724 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e68636d1c0 0x55e68636f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:25.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5ef640740 4 client.28378 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:57:25.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5ef640740 1 -- 172.21.15.40:0/3345240724 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e6862619f0 con 0x55e68636d1c0 2024-03-18T22:57:25.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.807+0000 7fe5e5d9b640 1 --2- 172.21.15.40:0/3345240724 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e68636d1c0 0x55e68636f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:25.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5e5d9b640 1 --2- 172.21.15.40:0/3345240724 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e68636d1c0 0x55e68636f5b0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3641a9424898afad server_cookie=277c405efedbe443 in_seq=0 out_seq=0 2024-03-18T22:57:25.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5cdffb640 10 client.28378 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:25.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5cdffb640 1 -- 172.21.15.40:0/3345240724 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e6862619f0 con 0x55e68636d1c0 2024-03-18T22:57:25.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5cdffb640 10 client.28378 handle_command_reply: tid=0 2024-03-18T22:57:25.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5ef640740 1 client.28378 shutdown 2024-03-18T22:57:25.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5ef640740 1 -- 172.21.15.40:0/3345240724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f02f0 msgr2=0x55e6862ff6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5ef640740 1 --2- 172.21.15.40:0/3345240724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f02f0 0x55e6862ff6d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8009a60 tx=0x7fe5d802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5ef640740 1 -- 172.21.15.40:0/3345240724 shutdown_connections 2024-03-18T22:57:25.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5ef640740 1 --2- 172.21.15.40:0/3345240724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e6862f5e50 0x55e686300c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5ef640740 1 --2- 172.21.15.40:0/3345240724 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e68636d1c0 0x55e68636f5b0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5ef640740 1 --2- 172.21.15.40:0/3345240724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e6862f0ce0 0x55e6862ffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5ef640740 1 --2- 172.21.15.40:0/3345240724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e6862f02f0 0x55e6862ff6d0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5cdffb640 0 client.28378 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:25.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5ef640740 1 -- 172.21.15.40:0/3345240724 >> 172.21.15.40:0/3345240724 conn(0x55e6862eb860 msgr2=0x55e6862eeac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5ef640740 1 -- 172.21.15.40:0/3345240724 shutdown_connections 2024-03-18T22:57:25.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5ef640740 1 -- 172.21.15.40:0/3345240724 wait complete. 2024-03-18T22:57:25.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.808+0000 7fe5ef640740 20 client.28378 trim_cache size 0 max 16384 2024-03-18T22:57:25.816 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:57:25.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.809+0000 7fe5eca59640 1 -- 172.21.15.40:0/680830343 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe5d00a2800 msgr2=0x7fe5d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.809+0000 7fe5eca59640 1 --2- 172.21.15.40:0/680830343 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe5d00a2800 0x7fe5d00a4cc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0011490 tx=0x7fe5e0011890 comp rx=0 tx=0).stop 2024-03-18T22:57:25.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.809+0000 7fe5eca59640 1 -- 172.21.15.40:0/680830343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5e8101940 msgr2=0x7fe5e8102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:25.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.809+0000 7fe5eca59640 1 --2- 172.21.15.40:0/680830343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5e8101940 0x7fe5e8102d20 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc00e9f0 tx=0x7fe5dc00edf0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.809+0000 7fe5eca59640 1 -- 172.21.15.40:0/680830343 shutdown_connections 2024-03-18T22:57:25.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.809+0000 7fe5eca59640 1 --2- 172.21.15.40:0/680830343 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe5d00a2800 0x7fe5d00a4cc0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.809+0000 7fe5eca59640 1 --2- 172.21.15.40:0/680830343 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe5e8102300 0x7fe5e8103260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.809+0000 7fe5eca59640 1 --2- 172.21.15.40:0/680830343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5e8101940 0x7fe5e8102d20 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.809+0000 7fe5eca59640 1 --2- 172.21.15.40:0/680830343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe5e8100750 0x7fe5e81046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:25.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.809+0000 7fe5eca59640 1 -- 172.21.15.40:0/680830343 >> 172.21.15.40:0/680830343 conn(0x7fe5e80fbcc0 msgr2=0x7fe5e8069f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:25.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.809+0000 7fe5eca59640 1 -- 172.21.15.40:0/680830343 shutdown_connections 2024-03-18T22:57:25.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:25.809+0000 7fe5eca59640 1 -- 172.21.15.40:0/680830343 wait complete. 2024-03-18T22:57:26.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:26 smithi113 ceph-mon[66775]: pgmap v794: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.0 MiB/s wr, 1.28k op/s 2024-03-18T22:57:26.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:26 smithi103 ceph-mon[66377]: pgmap v794: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.0 MiB/s wr, 1.28k op/s 2024-03-18T22:57:26.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:26 smithi040 ceph-mon[62643]: pgmap v794: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.0 MiB/s wr, 1.28k op/s 2024-03-18T22:57:26.821 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:57:26.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:57:26.821 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:57:26.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.940+0000 7f5d0d6dc640 1 -- 172.21.15.40:0/2536196598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08100730 msgr2=0x7f5d08100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:26.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.940+0000 7f5d0d6dc640 1 --2- 172.21.15.40:0/2536196598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08100730 0x7f5d08100b50 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8009900 tx=0x7f5cf802bc30 comp rx=0 tx=0).stop 2024-03-18T22:57:26.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.940+0000 7f5d0d6dc640 1 -- 172.21.15.40:0/2536196598 shutdown_connections 2024-03-18T22:57:26.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.940+0000 7f5d0d6dc640 1 --2- 172.21.15.40:0/2536196598 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d081022e0 0x7f5d08108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.940+0000 7f5d0d6dc640 1 --2- 172.21.15.40:0/2536196598 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08101920 0x7f5d08101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.940+0000 7f5d0d6dc640 1 --2- 172.21.15.40:0/2536196598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d08100730 0x7f5d08100b50 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.940+0000 7f5d0d6dc640 1 -- 172.21.15.40:0/2536196598 >> 172.21.15.40:0/2536196598 conn(0x7f5d080fbcc0 msgr2=0x7f5d080fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:26.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.940+0000 7f5d0d6dc640 1 -- 172.21.15.40:0/2536196598 shutdown_connections 2024-03-18T22:57:26.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.940+0000 7f5d0d6dc640 1 -- 172.21.15.40:0/2536196598 wait complete. 2024-03-18T22:57:26.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.941+0000 7f5d0d6dc640 1 Processor -- start 2024-03-18T22:57:26.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.941+0000 7f5d0d6dc640 1 -- start start 2024-03-18T22:57:26.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.941+0000 7f5d0d6dc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d08100730 0x7f5d081046c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.941+0000 7f5d0d6dc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08101920 0x7f5d08102d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.941+0000 7f5d0d6dc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d081022e0 0x7f5d08103240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.941+0000 7f5d0d6dc640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0810c230 con 0x7f5d081022e0 2024-03-18T22:57:26.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.941+0000 7f5d0d6dc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0810c0b0 con 0x7f5d08101920 2024-03-18T22:57:26.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.941+0000 7f5d0d6dc640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0810c3b0 con 0x7f5d08100730 2024-03-18T22:57:26.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.941+0000 7f5d077fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d081022e0 0x7f5d08103240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.941+0000 7f5d067fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08101920 0x7f5d08102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.941+0000 7f5d06ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d08100730 0x7f5d081046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.941+0000 7f5d077fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d081022e0 0x7f5d08103240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49548/0 (socket says 172.21.15.40:49548) 2024-03-18T22:57:26.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.941+0000 7f5d077fe640 1 -- 172.21.15.40:0/103305208 learned_addr learned my addr 172.21.15.40:0/103305208 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:26.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.941+0000 7f5d077fe640 1 -- 172.21.15.40:0/103305208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d08100730 msgr2=0x7f5d081046c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:26.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.942+0000 7f5d077fe640 1 --2- 172.21.15.40:0/103305208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d08100730 0x7f5d081046c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.942+0000 7f5d077fe640 1 -- 172.21.15.40:0/103305208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08101920 msgr2=0x7f5d08102d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:26.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.942+0000 7f5d077fe640 1 --2- 172.21.15.40:0/103305208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08101920 0x7f5d08102d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.942+0000 7f5d077fe640 1 -- 172.21.15.40:0/103305208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d08103ad0 con 0x7f5d081022e0 2024-03-18T22:57:26.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.942+0000 7f5d067fc640 1 --2- 172.21.15.40:0/103305208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08101920 0x7f5d08102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:26.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.942+0000 7f5d06ffd640 1 --2- 172.21.15.40:0/103305208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d08100730 0x7f5d081046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:26.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.942+0000 7f5d077fe640 1 --2- 172.21.15.40:0/103305208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d081022e0 0x7f5d08103240 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000efd0 tx=0x7f5d0000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:26.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.942+0000 7f5ceffff640 1 -- 172.21.15.40:0/103305208 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d00003b70 con 0x7f5d081022e0 2024-03-18T22:57:26.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.942+0000 7f5d0d6dc640 1 -- 172.21.15.40:0/103305208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d081a1be0 con 0x7f5d081022e0 2024-03-18T22:57:26.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.942+0000 7f5ceffff640 1 -- 172.21.15.40:0/103305208 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5d00003d10 con 0x7f5d081022e0 2024-03-18T22:57:26.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.942+0000 7f5ceffff640 1 -- 172.21.15.40:0/103305208 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d00010980 con 0x7f5d081022e0 2024-03-18T22:57:26.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.942+0000 7f5d0d6dc640 1 -- 172.21.15.40:0/103305208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d081a20e0 con 0x7f5d081022e0 2024-03-18T22:57:26.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.944+0000 7f5ceffff640 1 -- 172.21.15.40:0/103305208 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5d00004020 con 0x7f5d081022e0 2024-03-18T22:57:26.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.944+0000 7f5d103a7740 1 Processor -- start 2024-03-18T22:57:26.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d103a7740 1 -- start start 2024-03-18T22:57:26.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d103a7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fcf3a0 0x559469fcf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d103a7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcfd90 0x559469fd4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d103a7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fd4f00 0x559469fd7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d103a7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559469f34160 con 0x559469fcf3a0 2024-03-18T22:57:26.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d103a7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559469edce10 con 0x559469fcfd90 2024-03-18T22:57:26.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d103a7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559469f30200 con 0x559469fd4f00 2024-03-18T22:57:26.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d067fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fcf3a0 0x559469fcf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d067fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fcf3a0 0x559469fcf7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49554/0 (socket says 172.21.15.40:49554) 2024-03-18T22:57:26.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d067fc640 1 -- 172.21.15.40:0/3179960296 learned_addr learned my addr 172.21.15.40:0/3179960296 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:26.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5ceffff640 1 --2- 172.21.15.40:0/103305208 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5cf00a2800 0x7f5cf00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d067fc640 1 --2- 172.21.15.40:0/3179960296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fd4f00 0x559469fd7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d06ffd640 1 --2- 172.21.15.40:0/3179960296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcfd90 0x559469fd4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d077fe640 1 --2- 172.21.15.40:0/103305208 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5cf00a2800 0x7f5cf00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d067fc640 1 -- 172.21.15.40:0/3179960296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fd4f00 msgr2=0x559469fd7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:26.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d067fc640 1 --2- 172.21.15.40:0/3179960296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fd4f00 0x559469fd7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5ceffff640 1 -- 172.21.15.40:0/103305208 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5d000cf2b0 con 0x7f5d081022e0 2024-03-18T22:57:26.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d067fc640 1 -- 172.21.15.40:0/3179960296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcfd90 msgr2=0x559469fd4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d067fc640 1 --2- 172.21.15.40:0/3179960296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcfd90 0x559469fd4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d067fc640 1 -- 172.21.15.40:0/3179960296 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559469d107e0 con 0x559469fcf3a0 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.945+0000 7f5d06ffd640 1 --2- 172.21.15.40:0/3179960296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcfd90 0x559469fd4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.946+0000 7f5d067fc640 1 --2- 172.21.15.40:0/3179960296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fcf3a0 0x559469fcf7c0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc013c20 tx=0x7f5cfc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95ec608b4bef700 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.946+0000 7f5d077fe640 1 --2- 172.21.15.40:0/103305208 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5cf00a2800 0x7f5cf00a4cc0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5d000919a0 tx=0x7f5d000c62b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.946+0000 7f5ced7fa640 1 -- 172.21.15.40:0/3179960296 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc013d10 con 0x559469fcf3a0 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.946+0000 7f5ced7fa640 1 -- 172.21.15.40:0/3179960296 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cfc013170 con 0x559469fcf3a0 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.946+0000 7f5ced7fa640 1 -- 172.21.15.40:0/3179960296 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc004a00 con 0x559469fcf3a0 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.946+0000 7f5d103a7740 1 -- 172.21.15.40:0/3179960296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fcf3a0 msgr2=0x559469fcf7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.946+0000 7f5d103a7740 1 --2- 172.21.15.40:0/3179960296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fcf3a0 0x559469fcf7c0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc013c20 tx=0x7f5cfc009b50 comp rx=0 tx=0).stop 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.946+0000 7f5d103a7740 1 -- 172.21.15.40:0/3179960296 shutdown_connections 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.946+0000 7f5d103a7740 1 --2- 172.21.15.40:0/3179960296 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fd4f00 0x559469fd7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.946+0000 7f5d103a7740 1 --2- 172.21.15.40:0/3179960296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcfd90 0x559469fd4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.946+0000 7f5d103a7740 1 --2- 172.21.15.40:0/3179960296 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fcf3a0 0x559469fcf7c0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.946+0000 7f5d103a7740 1 -- 172.21.15.40:0/3179960296 >> 172.21.15.40:0/3179960296 conn(0x559469fca910 msgr2=0x559469fcad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:26.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.946+0000 7f5d103a7740 1 -- 172.21.15.40:0/3179960296 shutdown_connections 2024-03-18T22:57:26.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d103a7740 1 -- 172.21.15.40:0/3179960296 wait complete. 2024-03-18T22:57:26.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d103a7740 1 -- start start 2024-03-18T22:57:26.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d103a7740 1 Processor -- start 2024-03-18T22:57:26.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d103a7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcf3a0 0x559469fd9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d103a7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fcfd90 0x559469fd9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d103a7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fd4f00 0x559469fe0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d103a7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x559469fd7e50 con 0x559469fcfd90 2024-03-18T22:57:26.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d103a7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559469ef01a0 con 0x559469fd4f00 2024-03-18T22:57:26.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d103a7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x559469fd7fd0 con 0x559469fcf3a0 2024-03-18T22:57:26.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d103a7740 10 client.0 _ll_register_callbacks cb 0x559469c47c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:26.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d06ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fcfd90 0x559469fd9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d067fc640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcf3a0 0x559469fd9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d06ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fcfd90 0x559469fd9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49558/0 (socket says 172.21.15.40:49558) 2024-03-18T22:57:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d06ffd640 1 -- 172.21.15.40:0/3634430110 learned_addr learned my addr 172.21.15.40:0/3634430110 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d067fc640 1 --2- 172.21.15.40:0/3634430110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fd4f00 0x559469fe0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d06ffd640 1 -- 172.21.15.40:0/3634430110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcf3a0 msgr2=0x559469fd9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.947+0000 7f5d06ffd640 1 --2- 172.21.15.40:0/3634430110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcf3a0 0x559469fd9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5d06ffd640 1 -- 172.21.15.40:0/3634430110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fd4f00 msgr2=0x559469fe0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5d06ffd640 1 --2- 172.21.15.40:0/3634430110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fd4f00 0x559469fe0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5d06ffd640 1 -- 172.21.15.40:0/3634430110 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559469d70f70 con 0x559469fcfd90 2024-03-18T22:57:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5d06ffd640 1 --2- 172.21.15.40:0/3634430110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fcfd90 0x559469fd9c40 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8009900 tx=0x7f5cf8005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28a0f4581bc630a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5ceaff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:57:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5ceaff5640 1 -- 172.21.15.40:0/3634430110 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8046070 con 0x559469fcfd90 2024-03-18T22:57:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5ceaff5640 1 -- 172.21.15.40:0/3634430110 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5cf80046c0 con 0x559469fcfd90 2024-03-18T22:57:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5d103a7740 1 -- 172.21.15.40:0/3634430110 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cfc013170 con 0x559469fcfd90 2024-03-18T22:57:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5ceaff5640 1 -- 172.21.15.40:0/3634430110 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f5cf8052050 con 0x559469fcfd90 2024-03-18T22:57:26.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5d103a7740 10 client.18576 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5d103a7740 1 -- 172.21.15.40:0/3634430110 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x559469c08170 con 0x559469fcfd90 2024-03-18T22:57:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5ceaff5640 1 -- 172.21.15.40:0/3634430110 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cfc013170 con 0x559469fcfd90 2024-03-18T22:57:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5ceaff5640 1 -- 172.21.15.40:0/3634430110 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f5cf8003710 con 0x559469fcfd90 2024-03-18T22:57:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5d103a7740 10 client.18576 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5d103a7740 10 client.18576 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:57:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5d103a7740 20 client.18576 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.948+0000 7f5d103a7740 1 --2- 172.21.15.40:0/3634430110 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559469fe55e0 0x559469fe7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.949+0000 7f5d103a7740 4 client.18576 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:57:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.949+0000 7f5d103a7740 1 -- 172.21.15.40:0/3634430110 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559469f416a0 con 0x559469fe55e0 2024-03-18T22:57:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.949+0000 7f5d067fc640 1 --2- 172.21.15.40:0/3634430110 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559469fe55e0 0x559469fe7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.949+0000 7f5d067fc640 1 --2- 172.21.15.40:0/3634430110 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559469fe55e0 0x559469fe7a20 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aac5115bb4db934e server_cookie=d608ea976ea2c254 in_seq=0 out_seq=0 2024-03-18T22:57:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.949+0000 7f5ceaff5640 10 client.18576 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:26.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.950+0000 7f5ceaff5640 1 -- 172.21.15.40:0/3634430110 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559469f416a0 con 0x559469fe55e0 2024-03-18T22:57:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.950+0000 7f5ceaff5640 10 client.18576 handle_command_reply: tid=0 2024-03-18T22:57:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.950+0000 7f5d103a7740 1 client.18576 shutdown 2024-03-18T22:57:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.950+0000 7f5d103a7740 1 -- 172.21.15.40:0/3634430110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fcfd90 msgr2=0x559469fd9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.950+0000 7f5d103a7740 1 --2- 172.21.15.40:0/3634430110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fcfd90 0x559469fd9c40 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8009900 tx=0x7f5cf8005250 comp rx=0 tx=0).stop 2024-03-18T22:57:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.950+0000 7f5d103a7740 1 -- 172.21.15.40:0/3634430110 shutdown_connections 2024-03-18T22:57:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.950+0000 7f5d103a7740 1 --2- 172.21.15.40:0/3634430110 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x559469fe55e0 0x559469fe7a20 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.950+0000 7f5d103a7740 1 --2- 172.21.15.40:0/3634430110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fd4f00 0x559469fe0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.950+0000 7f5d103a7740 1 --2- 172.21.15.40:0/3634430110 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fcfd90 0x559469fd9c40 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.950+0000 7f5ceaff5640 0 client.18576 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.950+0000 7f5d103a7740 1 --2- 172.21.15.40:0/3634430110 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcf3a0 0x559469fd9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.950+0000 7f5d103a7740 1 -- 172.21.15.40:0/3634430110 >> 172.21.15.40:0/3634430110 conn(0x559469fca910 msgr2=0x559469fc9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.950+0000 7f5d103a7740 1 -- 172.21.15.40:0/3634430110 shutdown_connections 2024-03-18T22:57:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.950+0000 7f5d103a7740 1 -- 172.21.15.40:0/3634430110 wait complete. 2024-03-18T22:57:26.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.950+0000 7f5d103a7740 20 client.18576 trim_cache size 0 max 16384 2024-03-18T22:57:26.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.968+0000 7f5d103a7740 1 Processor -- start 2024-03-18T22:57:26.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.968+0000 7f5d103a7740 1 -- start start 2024-03-18T22:57:26.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.968+0000 7f5d103a7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcf3a0 0x559469fce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.968+0000 7f5d103a7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcfd90 0x559469fe0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.968+0000 7f5d103a7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fd4f00 0x559469fd83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.968+0000 7f5d103a7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5d081ac6a0 con 0x559469fd4f00 2024-03-18T22:57:26.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.968+0000 7f5d103a7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5d081ae1e0 con 0x559469fcf3a0 2024-03-18T22:57:26.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.968+0000 7f5d103a7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5d081acd70 con 0x559469fcfd90 2024-03-18T22:57:26.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.968+0000 7f5d067fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fd4f00 0x559469fd83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.968+0000 7f5d06ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcfd90 0x559469fe0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.968+0000 7f5d067fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fd4f00 0x559469fd83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49570/0 (socket says 172.21.15.40:49570) 2024-03-18T22:57:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.968+0000 7f5d067fc640 1 -- 172.21.15.40:0/3894601130 learned_addr learned my addr 172.21.15.40:0/3894601130 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.968+0000 7f5d067fc640 1 --2- 172.21.15.40:0/3894601130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcf3a0 0x559469fce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.969+0000 7f5d06ffd640 1 -- 172.21.15.40:0/3894601130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcf3a0 msgr2=0x559469fce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.969+0000 7f5d06ffd640 1 --2- 172.21.15.40:0/3894601130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcf3a0 0x559469fce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.969+0000 7f5d06ffd640 1 -- 172.21.15.40:0/3894601130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fd4f00 msgr2=0x559469fd83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.969+0000 7f5d06ffd640 1 --2- 172.21.15.40:0/3894601130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fd4f00 0x559469fd83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.969+0000 7f5d06ffd640 1 -- 172.21.15.40:0/3894601130 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559469fcb150 con 0x559469fcfd90 2024-03-18T22:57:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.969+0000 7f5d067fc640 1 --2- 172.21.15.40:0/3894601130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fd4f00 0x559469fd83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.969+0000 7f5d06ffd640 1 --2- 172.21.15.40:0/3894601130 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcfd90 0x559469fe0c40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5cf802c170 tx=0x7f5cf802bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cb9dcdfa9c13a57 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.969+0000 7f5cecff9640 1 -- 172.21.15.40:0/3894601130 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf804f030 con 0x559469fcfd90 2024-03-18T22:57:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.969+0000 7f5cecff9640 1 -- 172.21.15.40:0/3894601130 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cf802cc90 con 0x559469fcfd90 2024-03-18T22:57:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.969+0000 7f5cecff9640 1 -- 172.21.15.40:0/3894601130 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8046070 con 0x559469fcfd90 2024-03-18T22:57:26.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 -- 172.21.15.40:0/3894601130 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcfd90 msgr2=0x559469fe0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 --2- 172.21.15.40:0/3894601130 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcfd90 0x559469fe0c40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5cf802c170 tx=0x7f5cf802bc30 comp rx=0 tx=0).stop 2024-03-18T22:57:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 -- 172.21.15.40:0/3894601130 shutdown_connections 2024-03-18T22:57:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 --2- 172.21.15.40:0/3894601130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fd4f00 0x559469fd83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 --2- 172.21.15.40:0/3894601130 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcfd90 0x559469fe0c40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 --2- 172.21.15.40:0/3894601130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcf3a0 0x559469fce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 -- 172.21.15.40:0/3894601130 >> 172.21.15.40:0/3894601130 conn(0x559469fca910 msgr2=0x559469fcdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 -- 172.21.15.40:0/3894601130 shutdown_connections 2024-03-18T22:57:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 -- 172.21.15.40:0/3894601130 wait complete. 2024-03-18T22:57:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 -- start start 2024-03-18T22:57:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 Processor -- start 2024-03-18T22:57:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcf3a0 0x559469fde750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcfd90 0x559469fdec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fd4f00 0x559469fdfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5d081adb10 con 0x559469fd4f00 2024-03-18T22:57:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5d081ad440 con 0x559469fcf3a0 2024-03-18T22:57:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5d081aa680 con 0x559469fcfd90 2024-03-18T22:57:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d103a7740 10 client.0 _ll_register_callbacks cb 0x7f5cf8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d067fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fd4f00 0x559469fdfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d067fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcf3a0 0x559469fde750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d06ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcfd90 0x559469fdec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d06ffd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcfd90 0x559469fdec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:48044/0 (socket says 172.21.15.40:48044) 2024-03-18T22:57:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d067fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcf3a0 0x559469fde750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:38410/0 (socket says 172.21.15.40:38410) 2024-03-18T22:57:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.970+0000 7f5d06ffd640 1 -- 172.21.15.40:0/2359520316 learned_addr learned my addr 172.21.15.40:0/2359520316 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.971+0000 7f5d06ffd640 1 -- 172.21.15.40:0/2359520316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcf3a0 msgr2=0x559469fde750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.971+0000 7f5d06ffd640 1 --2- 172.21.15.40:0/2359520316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcf3a0 0x559469fde750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.971+0000 7f5d06ffd640 1 -- 172.21.15.40:0/2359520316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fd4f00 msgr2=0x559469fdfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.971+0000 7f5d06ffd640 1 --2- 172.21.15.40:0/2359520316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fd4f00 0x559469fdfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.971+0000 7f5d06ffd640 1 -- 172.21.15.40:0/2359520316 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559469fe0240 con 0x559469fcfd90 2024-03-18T22:57:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.971+0000 7f5d06ffd640 1 --2- 172.21.15.40:0/2359520316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcfd90 0x559469fdec90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5cf802c840 tx=0x7f5cf802c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dde1296a5b6c011 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.971+0000 7f5ceaff5640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:57:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.971+0000 7f5ceaff5640 1 -- 172.21.15.40:0/2359520316 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf805a070 con 0x559469fcfd90 2024-03-18T22:57:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.971+0000 7f5d103a7740 1 -- 172.21.15.40:0/2359520316 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cf802cc90 con 0x559469fcfd90 2024-03-18T22:57:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.971+0000 7f5ceaff5640 1 -- 172.21.15.40:0/2359520316 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5cf80046c0 con 0x559469fcfd90 2024-03-18T22:57:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.971+0000 7f5ceaff5640 1 -- 172.21.15.40:0/2359520316 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f5cf8066050 con 0x559469fcfd90 2024-03-18T22:57:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.971+0000 7f5d103a7740 10 client.28396 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.971+0000 7f5d103a7740 1 -- 172.21.15.40:0/2359520316 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55946a04bc80 con 0x559469fcfd90 2024-03-18T22:57:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.971+0000 7f5ceaff5640 1 -- 172.21.15.40:0/2359520316 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cf802cc90 con 0x559469fcfd90 2024-03-18T22:57:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.972+0000 7f5ceaff5640 1 -- 172.21.15.40:0/2359520316 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f5cf8040680 con 0x559469fcfd90 2024-03-18T22:57:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.972+0000 7f5d103a7740 10 client.28396 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.972+0000 7f5d103a7740 10 client.28396 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:57:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.972+0000 7f5d103a7740 20 client.28396 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:26.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.972+0000 7f5d103a7740 1 --2- 172.21.15.40:0/2359520316 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55946a04c5a0 0x55946a04e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.972+0000 7f5d103a7740 4 client.28396 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:57:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.972+0000 7f5d103a7740 1 -- 172.21.15.40:0/2359520316 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559469f3b9f0 con 0x55946a04c5a0 2024-03-18T22:57:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.972+0000 7f5d067fc640 1 --2- 172.21.15.40:0/2359520316 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55946a04c5a0 0x55946a04e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.972+0000 7f5ceaff5640 10 client.28396 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.972+0000 7f5d067fc640 1 --2- 172.21.15.40:0/2359520316 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55946a04c5a0 0x55946a04e9c0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9e09cc20b275185 server_cookie=32c95663b568dc4d in_seq=0 out_seq=0 2024-03-18T22:57:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.973+0000 7f5ceaff5640 1 -- 172.21.15.40:0/2359520316 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559469f3b9f0 con 0x55946a04c5a0 2024-03-18T22:57:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.973+0000 7f5ceaff5640 10 client.28396 handle_command_reply: tid=0 2024-03-18T22:57:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.973+0000 7f5d103a7740 1 client.28396 shutdown 2024-03-18T22:57:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.973+0000 7f5d103a7740 1 -- 172.21.15.40:0/2359520316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcfd90 msgr2=0x559469fdec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.973+0000 7f5d103a7740 1 --2- 172.21.15.40:0/2359520316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcfd90 0x559469fdec90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5cf802c840 tx=0x7f5cf802c330 comp rx=0 tx=0).stop 2024-03-18T22:57:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.973+0000 7f5d103a7740 1 -- 172.21.15.40:0/2359520316 shutdown_connections 2024-03-18T22:57:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.973+0000 7f5d103a7740 1 --2- 172.21.15.40:0/2359520316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559469fd4f00 0x559469fdfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.973+0000 7f5d103a7740 1 --2- 172.21.15.40:0/2359520316 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x559469fcfd90 0x559469fdec90 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.973+0000 7f5d103a7740 1 --2- 172.21.15.40:0/2359520316 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55946a04c5a0 0x55946a04e9c0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.973+0000 7f5d103a7740 1 --2- 172.21.15.40:0/2359520316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559469fcf3a0 0x559469fde750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.973+0000 7f5d103a7740 1 -- 172.21.15.40:0/2359520316 >> 172.21.15.40:0/2359520316 conn(0x559469fca910 msgr2=0x559469fce400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.973+0000 7f5ceaff5640 0 client.28396 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.973+0000 7f5d103a7740 1 -- 172.21.15.40:0/2359520316 shutdown_connections 2024-03-18T22:57:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.973+0000 7f5d103a7740 1 -- 172.21.15.40:0/2359520316 wait complete. 2024-03-18T22:57:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.974+0000 7f5d103a7740 20 client.28396 trim_cache size 0 max 16384 2024-03-18T22:57:26.980 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:57:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:57:26.980 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "4b316b6b-7f0f-4c78-91b4-3b82fdb3e273", 2024-03-18T22:57:26.981 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:57:26.981 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:57:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.974+0000 7f5d0d6dc640 1 -- 172.21.15.40:0/103305208 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5cf00a2800 msgr2=0x7f5cf00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.974+0000 7f5d0d6dc640 1 --2- 172.21.15.40:0/103305208 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5cf00a2800 0x7f5cf00a4cc0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5d000919a0 tx=0x7f5d000c62b0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.974+0000 7f5d0d6dc640 1 -- 172.21.15.40:0/103305208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d081022e0 msgr2=0x7f5d08103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.974+0000 7f5d0d6dc640 1 --2- 172.21.15.40:0/103305208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d081022e0 0x7f5d08103240 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000efd0 tx=0x7f5d0000c6a0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.974+0000 7f5d0d6dc640 1 -- 172.21.15.40:0/103305208 shutdown_connections 2024-03-18T22:57:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.974+0000 7f5d0d6dc640 1 --2- 172.21.15.40:0/103305208 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5cf00a2800 0x7f5cf00a4cc0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.974+0000 7f5d0d6dc640 1 --2- 172.21.15.40:0/103305208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d081022e0 0x7f5d08103240 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.974+0000 7f5d0d6dc640 1 --2- 172.21.15.40:0/103305208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d08101920 0x7f5d08102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.974+0000 7f5d0d6dc640 1 --2- 172.21.15.40:0/103305208 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5d08100730 0x7f5d081046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.974+0000 7f5d0d6dc640 1 -- 172.21.15.40:0/103305208 >> 172.21.15.40:0/103305208 conn(0x7f5d080fbcc0 msgr2=0x7f5d08069e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.974+0000 7f5d0d6dc640 1 -- 172.21.15.40:0/103305208 shutdown_connections 2024-03-18T22:57:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:26.974+0000 7f5d0d6dc640 1 -- 172.21.15.40:0/103305208 wait complete. 2024-03-18T22:57:26.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:57:26.987 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:57:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.093+0000 7fd9f5132640 1 -- 172.21.15.40:0/3034972899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f0101910 msgr2=0x7fd9f0101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.093+0000 7fd9f5132640 1 --2- 172.21.15.40:0/3034972899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f0101910 0x7fd9f0101d90 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009900 tx=0x7fd9d802bc30 comp rx=0 tx=0).stop 2024-03-18T22:57:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.093+0000 7fd9f5132640 1 -- 172.21.15.40:0/3034972899 shutdown_connections 2024-03-18T22:57:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.093+0000 7fd9f5132640 1 --2- 172.21.15.40:0/3034972899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9f01022d0 0x7fd9f0108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.093+0000 7fd9f5132640 1 --2- 172.21.15.40:0/3034972899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f0101910 0x7fd9f0101d90 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.093+0000 7fd9f5132640 1 --2- 172.21.15.40:0/3034972899 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9f0100720 0x7fd9f0100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.093+0000 7fd9f5132640 1 -- 172.21.15.40:0/3034972899 >> 172.21.15.40:0/3034972899 conn(0x7fd9f00fbcb0 msgr2=0x7fd9f00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.093+0000 7fd9f5132640 1 -- 172.21.15.40:0/3034972899 shutdown_connections 2024-03-18T22:57:27.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.093+0000 7fd9f5132640 1 -- 172.21.15.40:0/3034972899 wait complete. 2024-03-18T22:57:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.094+0000 7fd9f5132640 1 Processor -- start 2024-03-18T22:57:27.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.094+0000 7fd9f5132640 1 -- start start 2024-03-18T22:57:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.094+0000 7fd9f5132640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f0100720 0x7fd9f01046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.094+0000 7fd9f5132640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9f0101910 0x7fd9f0102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.094+0000 7fd9f5132640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9f01022d0 0x7fd9f0103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.094+0000 7fd9f5132640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f010c250 con 0x7fd9f0100720 2024-03-18T22:57:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.094+0000 7fd9f5132640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f010c0d0 con 0x7fd9f01022d0 2024-03-18T22:57:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.094+0000 7fd9f5132640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f010c3d0 con 0x7fd9f0101910 2024-03-18T22:57:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9eed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f0100720 0x7fd9f01046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9ef59e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9f01022d0 0x7fd9f0103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9ee59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9f0101910 0x7fd9f0102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9ef59e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9f01022d0 0x7fd9f0103230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:38436/0 (socket says 172.21.15.40:38436) 2024-03-18T22:57:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9eed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f0100720 0x7fd9f01046b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49606/0 (socket says 172.21.15.40:49606) 2024-03-18T22:57:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9ef59e640 1 -- 172.21.15.40:0/4107817936 learned_addr learned my addr 172.21.15.40:0/4107817936 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:27.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9ef59e640 1 -- 172.21.15.40:0/4107817936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9f0101910 msgr2=0x7fd9f0102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:27.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9ef59e640 1 --2- 172.21.15.40:0/4107817936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9f0101910 0x7fd9f0102cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9ef59e640 1 -- 172.21.15.40:0/4107817936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f0100720 msgr2=0x7fd9f01046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:27.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9ef59e640 1 --2- 172.21.15.40:0/4107817936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f0100720 0x7fd9f01046b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9ef59e640 1 -- 172.21.15.40:0/4107817936 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f0103ac0 con 0x7fd9f01022d0 2024-03-18T22:57:27.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9ee59c640 1 --2- 172.21.15.40:0/4107817936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9f0101910 0x7fd9f0102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:27.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9eed9d640 1 --2- 172.21.15.40:0/4107817936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f0100720 0x7fd9f01046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:27.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9ef59e640 1 --2- 172.21.15.40:0/4107817936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9f01022d0 0x7fd9f0103230 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800e9b0 tx=0x7fd9e800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:27.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9e7fff640 1 -- 172.21.15.40:0/4107817936 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e800cd80 con 0x7fd9f01022d0 2024-03-18T22:57:27.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9f5132640 1 -- 172.21.15.40:0/4107817936 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f01a1bf0 con 0x7fd9f01022d0 2024-03-18T22:57:27.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9e7fff640 1 -- 172.21.15.40:0/4107817936 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9e80035f0 con 0x7fd9f01022d0 2024-03-18T22:57:27.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9e7fff640 1 -- 172.21.15.40:0/4107817936 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8010690 con 0x7fd9f01022d0 2024-03-18T22:57:27.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.095+0000 7fd9f5132640 1 -- 172.21.15.40:0/4107817936 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f01a2110 con 0x7fd9f01022d0 2024-03-18T22:57:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.098+0000 7fd9e7fff640 1 -- 172.21.15.40:0/4107817936 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd9e80037a0 con 0x7fd9f01022d0 2024-03-18T22:57:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.098+0000 7fd9f7df5740 1 Processor -- start 2024-03-18T22:57:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.098+0000 7fd9f7df5740 1 -- start start 2024-03-18T22:57:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.098+0000 7fd9f7df5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e79e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.098+0000 7fd9f7df5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7a440 0x557cd5e7ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.098+0000 7fd9f7df5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e7f5b0 0x557cd5e81a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.098+0000 7fd9f7df5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557cd5dde500 con 0x557cd5e79a50 2024-03-18T22:57:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.098+0000 7fd9f7df5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557cd5d86be0 con 0x557cd5e7f5b0 2024-03-18T22:57:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.098+0000 7fd9f7df5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557cd5dda5a0 con 0x557cd5e7a440 2024-03-18T22:57:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.098+0000 7fd9ee59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e79e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.098+0000 7fd9ee59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e79e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49614/0 (socket says 172.21.15.40:49614) 2024-03-18T22:57:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.098+0000 7fd9ee59c640 1 -- 172.21.15.40:0/2845025112 learned_addr learned my addr 172.21.15.40:0/2845025112 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.098+0000 7fd9eed9d640 1 --2- 172.21.15.40:0/2845025112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7a440 0x557cd5e7ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.098+0000 7fd9ee59c640 1 --2- 172.21.15.40:0/2845025112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e7f5b0 0x557cd5e81a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.098+0000 7fd9e7fff640 1 --2- 172.21.15.40:0/4107817936 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd9dc0a2800 0x7fd9dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.099+0000 7fd9ee59c640 1 -- 172.21.15.40:0/2845025112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7a440 msgr2=0x557cd5e7ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.099+0000 7fd9ee59c640 1 --2- 172.21.15.40:0/2845025112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7a440 0x557cd5e7ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.099+0000 7fd9ee59c640 1 -- 172.21.15.40:0/2845025112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e7f5b0 msgr2=0x557cd5e81a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.099+0000 7fd9ee59c640 1 --2- 172.21.15.40:0/2845025112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e7f5b0 0x557cd5e81a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.099+0000 7fd9ee59c640 1 -- 172.21.15.40:0/2845025112 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cd5da8000 con 0x557cd5e79a50 2024-03-18T22:57:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.099+0000 7fd9ef59e640 1 --2- 172.21.15.40:0/4107817936 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd9dc0a2800 0x7fd9dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.099+0000 7fd9eed9d640 1 --2- 172.21.15.40:0/2845025112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7a440 0x557cd5e7ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.099+0000 7fd9e7fff640 1 -- 172.21.15.40:0/4107817936 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd9e80cf1b0 con 0x7fd9f01022d0 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.099+0000 7fd9ee59c640 1 --2- 172.21.15.40:0/2845025112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e79e70 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fd9d802c170 tx=0x7fd9d8005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85b19f0489e0941 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.099+0000 7fd9e57fa640 1 -- 172.21.15.40:0/2845025112 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8047020 con 0x557cd5e79a50 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.099+0000 7fd9e57fa640 1 -- 172.21.15.40:0/2845025112 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9d8005040 con 0x557cd5e79a50 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.099+0000 7fd9e57fa640 1 -- 172.21.15.40:0/2845025112 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8004ae0 con 0x557cd5e79a50 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.099+0000 7fd9ef59e640 1 --2- 172.21.15.40:0/4107817936 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd9dc0a2800 0x7fd9dc0a4cc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd9e809b460 tx=0x7fd9e8066520 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.100+0000 7fd9f7df5740 1 -- 172.21.15.40:0/2845025112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 msgr2=0x557cd5e79e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.100+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/2845025112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e79e70 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fd9d802c170 tx=0x7fd9d8005b50 comp rx=0 tx=0).stop 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.100+0000 7fd9f7df5740 1 -- 172.21.15.40:0/2845025112 shutdown_connections 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.100+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/2845025112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e7f5b0 0x557cd5e81a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.100+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/2845025112 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7a440 0x557cd5e7ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.100+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/2845025112 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e79e70 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.100+0000 7fd9f7df5740 1 -- 172.21.15.40:0/2845025112 >> 172.21.15.40:0/2845025112 conn(0x557cd5e74fc0 msgr2=0x557cd5e753c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.100+0000 7fd9f7df5740 1 -- 172.21.15.40:0/2845025112 shutdown_connections 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.100+0000 7fd9f7df5740 1 -- 172.21.15.40:0/2845025112 wait complete. 2024-03-18T22:57:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.100+0000 7fd9f7df5740 1 -- start start 2024-03-18T22:57:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.100+0000 7fd9f7df5740 1 Processor -- start 2024-03-18T22:57:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.100+0000 7fd9f7df5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e83ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.100+0000 7fd9f7df5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e7a440 0x557cd5e843e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.100+0000 7fd9f7df5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7f5b0 0x557cd5e8b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9f7df5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557cd5e82500 con 0x557cd5e79a50 2024-03-18T22:57:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9f7df5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557cd5d9a160 con 0x557cd5e7a440 2024-03-18T22:57:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9f7df5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557cd5e82680 con 0x557cd5e7f5b0 2024-03-18T22:57:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9f7df5740 10 client.0 _ll_register_callbacks cb 0x557cd5dd6a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9ee59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e83ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9ee59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e83ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49628/0 (socket says 172.21.15.40:49628) 2024-03-18T22:57:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9ee59c640 1 -- 172.21.15.40:0/250145853 learned_addr learned my addr 172.21.15.40:0/250145853 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9eed9d640 1 --2- 172.21.15.40:0/250145853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e7a440 0x557cd5e843e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9ee59c640 1 --2- 172.21.15.40:0/250145853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7f5b0 0x557cd5e8b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9ee59c640 1 -- 172.21.15.40:0/250145853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7f5b0 msgr2=0x557cd5e8b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9ee59c640 1 --2- 172.21.15.40:0/250145853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7f5b0 0x557cd5e8b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9ee59c640 1 -- 172.21.15.40:0/250145853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e7a440 msgr2=0x557cd5e843e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9ee59c640 1 --2- 172.21.15.40:0/250145853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e7a440 0x557cd5e843e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9ee59c640 1 -- 172.21.15.40:0/250145853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cd5bbc7d0 con 0x557cd5e79a50 2024-03-18T22:57:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9ee59c640 1 --2- 172.21.15.40:0/250145853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e83ea0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8005ab0 tx=0x7fd9d80041c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fdbc9a6adba2833 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:57:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9d67fc640 1 -- 172.21.15.40:0/250145853 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8055020 con 0x557cd5e79a50 2024-03-18T22:57:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9d67fc640 1 -- 172.21.15.40:0/250145853 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd9d8004a60 con 0x557cd5e79a50 2024-03-18T22:57:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9f7df5740 1 -- 172.21.15.40:0/250145853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9d8005040 con 0x557cd5e79a50 2024-03-18T22:57:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9d67fc640 1 -- 172.21.15.40:0/250145853 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fd9d8060050 con 0x557cd5e79a50 2024-03-18T22:57:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9f7df5740 10 client.18594 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.101+0000 7fd9f7df5740 1 -- 172.21.15.40:0/250145853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x557cd5af23f0 con 0x557cd5e79a50 2024-03-18T22:57:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.102+0000 7fd9d67fc640 1 -- 172.21.15.40:0/250145853 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9d8005040 con 0x557cd5e79a50 2024-03-18T22:57:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.102+0000 7fd9d67fc640 1 -- 172.21.15.40:0/250145853 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fd9d8040040 con 0x557cd5e79a50 2024-03-18T22:57:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.102+0000 7fd9f7df5740 10 client.18594 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.102+0000 7fd9f7df5740 10 client.18594 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:57:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.102+0000 7fd9f7df5740 20 client.18594 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.102+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/250145853 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557cd5e8f910 0x557cd5e91d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.102+0000 7fd9f7df5740 4 client.18594 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:57:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.102+0000 7fd9f7df5740 1 -- 172.21.15.40:0/250145853 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557cd5deba40 con 0x557cd5e8f910 2024-03-18T22:57:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.102+0000 7fd9eed9d640 1 --2- 172.21.15.40:0/250145853 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557cd5e8f910 0x557cd5e91d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.103+0000 7fd9d67fc640 10 client.18594 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.103+0000 7fd9eed9d640 1 --2- 172.21.15.40:0/250145853 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557cd5e8f910 0x557cd5e91d00 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=794a4ba69431d649 server_cookie=6f63c20f3f257fd6 in_seq=0 out_seq=0 2024-03-18T22:57:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.103+0000 7fd9d67fc640 1 -- 172.21.15.40:0/250145853 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557cd5deba40 con 0x557cd5e8f910 2024-03-18T22:57:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.103+0000 7fd9d67fc640 10 client.18594 handle_command_reply: tid=0 2024-03-18T22:57:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.103+0000 7fd9f7df5740 1 client.18594 shutdown 2024-03-18T22:57:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.104+0000 7fd9f7df5740 1 -- 172.21.15.40:0/250145853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 msgr2=0x557cd5e83ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.104+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/250145853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e83ea0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8005ab0 tx=0x7fd9d80041c0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.104+0000 7fd9f7df5740 1 -- 172.21.15.40:0/250145853 shutdown_connections 2024-03-18T22:57:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.104+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/250145853 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7f5b0 0x557cd5e8b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.104+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/250145853 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557cd5e8f910 0x557cd5e91d00 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.104+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/250145853 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e7a440 0x557cd5e843e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.104+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/250145853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e83ea0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.104+0000 7fd9d67fc640 0 client.18594 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.104+0000 7fd9f7df5740 1 -- 172.21.15.40:0/250145853 >> 172.21.15.40:0/250145853 conn(0x557cd5e74fc0 msgr2=0x557cd5e741e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.104+0000 7fd9f7df5740 1 -- 172.21.15.40:0/250145853 shutdown_connections 2024-03-18T22:57:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.104+0000 7fd9f7df5740 1 -- 172.21.15.40:0/250145853 wait complete. 2024-03-18T22:57:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.104+0000 7fd9f7df5740 20 client.18594 trim_cache size 0 max 16384 2024-03-18T22:57:27.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.121+0000 7fd9f7df5740 1 Processor -- start 2024-03-18T22:57:27.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.121+0000 7fd9f7df5740 1 -- start start 2024-03-18T22:57:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.121+0000 7fd9f7df5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e79a50 0x557cd5e84340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.121+0000 7fd9f7df5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e7a440 0x557cd5e8b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.121+0000 7fd9f7df5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7f5b0 0x557cd5e6ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.121+0000 7fd9f7df5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f01ac5d0 con 0x557cd5e7a440 2024-03-18T22:57:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.121+0000 7fd9f7df5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f01ae110 con 0x557cd5e79a50 2024-03-18T22:57:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.121+0000 7fd9f7df5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f01acca0 con 0x557cd5e7f5b0 2024-03-18T22:57:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.122+0000 7fd9eed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e7a440 0x557cd5e8b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.122+0000 7fd9eed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e7a440 0x557cd5e8b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49632/0 (socket says 172.21.15.40:49632) 2024-03-18T22:57:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.122+0000 7fd9ee59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e79a50 0x557cd5e84340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.122+0000 7fd9eed9d640 1 -- 172.21.15.40:0/158462420 learned_addr learned my addr 172.21.15.40:0/158462420 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.122+0000 7fd9ee59c640 1 --2- 172.21.15.40:0/158462420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7f5b0 0x557cd5e6ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.122+0000 7fd9eed9d640 1 -- 172.21.15.40:0/158462420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7f5b0 msgr2=0x557cd5e6ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.122+0000 7fd9eed9d640 1 --2- 172.21.15.40:0/158462420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7f5b0 0x557cd5e6ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.122+0000 7fd9eed9d640 1 -- 172.21.15.40:0/158462420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e79a50 msgr2=0x557cd5e84340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.122+0000 7fd9eed9d640 1 --2- 172.21.15.40:0/158462420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e79a50 0x557cd5e84340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.122+0000 7fd9eed9d640 1 -- 172.21.15.40:0/158462420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557cd5ab20f0 con 0x557cd5e7a440 2024-03-18T22:57:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.122+0000 7fd9ee59c640 1 --2- 172.21.15.40:0/158462420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e79a50 0x557cd5e84340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.122+0000 7fd9eed9d640 1 --2- 172.21.15.40:0/158462420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e7a440 0x557cd5e8b3b0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0005fd0 tx=0x7fd9e0009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62fdc55d795b9350 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.122+0000 7fd9e4ff9640 1 -- 172.21.15.40:0/158462420 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e0004b20 con 0x557cd5e7a440 2024-03-18T22:57:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.122+0000 7fd9e4ff9640 1 -- 172.21.15.40:0/158462420 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9e0003e20 con 0x557cd5e7a440 2024-03-18T22:57:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.122+0000 7fd9e4ff9640 1 -- 172.21.15.40:0/158462420 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e0004030 con 0x557cd5e7a440 2024-03-18T22:57:27.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.123+0000 7fd9f7df5740 1 -- 172.21.15.40:0/158462420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e7a440 msgr2=0x557cd5e8b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:27.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.123+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/158462420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e7a440 0x557cd5e8b3b0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0005fd0 tx=0x7fd9e0009500 comp rx=0 tx=0).stop 2024-03-18T22:57:27.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.123+0000 7fd9f7df5740 1 -- 172.21.15.40:0/158462420 shutdown_connections 2024-03-18T22:57:27.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.123+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/158462420 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7f5b0 0x557cd5e6ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.123+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/158462420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e7a440 0x557cd5e8b3b0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.123+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/158462420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e79a50 0x557cd5e84340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.123+0000 7fd9f7df5740 1 -- 172.21.15.40:0/158462420 >> 172.21.15.40:0/158462420 conn(0x557cd5e74fc0 msgr2=0x557cd5e73950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:27.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.123+0000 7fd9f7df5740 1 -- 172.21.15.40:0/158462420 shutdown_connections 2024-03-18T22:57:27.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.123+0000 7fd9f7df5740 1 -- 172.21.15.40:0/158462420 wait complete. 2024-03-18T22:57:27.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.123+0000 7fd9f7df5740 1 -- start start 2024-03-18T22:57:27.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.123+0000 7fd9f7df5740 1 Processor -- start 2024-03-18T22:57:27.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9f7df5740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e88bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9f7df5740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7a440 0x557cd5e89130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9f7df5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e7f5b0 0x557cd5e8a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9f7df5740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f01ada40 con 0x557cd5e79a50 2024-03-18T22:57:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9f7df5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f01ad370 con 0x557cd5e7f5b0 2024-03-18T22:57:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9f7df5740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f01aa5b0 con 0x557cd5e7a440 2024-03-18T22:57:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9f7df5740 10 client.0 _ll_register_callbacks cb 0x7fd9e0009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9ee59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e88bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9ee59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e88bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49644/0 (socket says 172.21.15.40:49644) 2024-03-18T22:57:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9ee59c640 1 -- 172.21.15.40:0/733215701 learned_addr learned my addr 172.21.15.40:0/733215701 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9eed9d640 1 --2- 172.21.15.40:0/733215701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7a440 0x557cd5e89130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9ee59c640 1 --2- 172.21.15.40:0/733215701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e7f5b0 0x557cd5e8a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9ee59c640 1 -- 172.21.15.40:0/733215701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7a440 msgr2=0x557cd5e89130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9ee59c640 1 --2- 172.21.15.40:0/733215701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7a440 0x557cd5e89130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9ee59c640 1 -- 172.21.15.40:0/733215701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e7f5b0 msgr2=0x557cd5e8a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9ee59c640 1 --2- 172.21.15.40:0/733215701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e7f5b0 0x557cd5e8a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9ee59c640 1 -- 172.21.15.40:0/733215701 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cd5e8a6e0 con 0x557cd5e79a50 2024-03-18T22:57:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9ee59c640 1 --2- 172.21.15.40:0/733215701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e88bf0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009900 tx=0x7fd9d8004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca67679eebedb111 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:57:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9d67fc640 1 -- 172.21.15.40:0/733215701 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8055040 con 0x557cd5e79a50 2024-03-18T22:57:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9d67fc640 1 -- 172.21.15.40:0/733215701 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd9d804cd00 con 0x557cd5e79a50 2024-03-18T22:57:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.124+0000 7fd9f7df5740 1 -- 172.21.15.40:0/733215701 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9e0003e20 con 0x557cd5e79a50 2024-03-18T22:57:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.125+0000 7fd9d67fc640 1 -- 172.21.15.40:0/733215701 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fd9d8047030 con 0x557cd5e79a50 2024-03-18T22:57:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.125+0000 7fd9f7df5740 10 client.18600 fetch_fsmap learned FSMap version 40 2024-03-18T22:57:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.125+0000 7fd9f7df5740 1 -- 172.21.15.40:0/733215701 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x557cd5ef5c50 con 0x557cd5e79a50 2024-03-18T22:57:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.125+0000 7fd9d67fc640 1 -- 172.21.15.40:0/733215701 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9e0003e20 con 0x557cd5e79a50 2024-03-18T22:57:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.125+0000 7fd9d67fc640 1 -- 172.21.15.40:0/733215701 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fd9d80046c0 con 0x557cd5e79a50 2024-03-18T22:57:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.125+0000 7fd9f7df5740 10 client.18600 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T22:57:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.125+0000 7fd9f7df5740 10 client.18600 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:57:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.125+0000 7fd9f7df5740 20 client.18600 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.125+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/733215701 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557cd5ef6510 0x557cd5ef8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.125+0000 7fd9f7df5740 4 client.18600 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:57:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.125+0000 7fd9f7df5740 1 -- 172.21.15.40:0/733215701 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557cd5de5d90 con 0x557cd5ef6510 2024-03-18T22:57:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.125+0000 7fd9eed9d640 1 --2- 172.21.15.40:0/733215701 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557cd5ef6510 0x557cd5ef8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9eed9d640 1 --2- 172.21.15.40:0/733215701 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557cd5ef6510 0x557cd5ef8930 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5cdd792e2d457c3 server_cookie=50a06b6ae558273b in_seq=0 out_seq=0 2024-03-18T22:57:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9d67fc640 10 client.18600 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9d67fc640 1 -- 172.21.15.40:0/733215701 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557cd5de5d90 con 0x557cd5ef6510 2024-03-18T22:57:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9d67fc640 10 client.18600 handle_command_reply: tid=0 2024-03-18T22:57:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f7df5740 1 client.18600 shutdown 2024-03-18T22:57:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f7df5740 1 -- 172.21.15.40:0/733215701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 msgr2=0x557cd5e88bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/733215701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e88bf0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009900 tx=0x7fd9d8004370 comp rx=0 tx=0).stop 2024-03-18T22:57:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f7df5740 1 -- 172.21.15.40:0/733215701 shutdown_connections 2024-03-18T22:57:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/733215701 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x557cd5ef6510 0x557cd5ef8930 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/733215701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557cd5e7f5b0 0x557cd5e8a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9d67fc640 0 client.18600 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/733215701 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557cd5e7a440 0x557cd5e89130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f7df5740 1 --2- 172.21.15.40:0/733215701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cd5e79a50 0x557cd5e88bf0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f7df5740 1 -- 172.21.15.40:0/733215701 >> 172.21.15.40:0/733215701 conn(0x557cd5e74fc0 msgr2=0x557cd5e741e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f7df5740 1 -- 172.21.15.40:0/733215701 shutdown_connections 2024-03-18T22:57:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f7df5740 1 -- 172.21.15.40:0/733215701 wait complete. 2024-03-18T22:57:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f7df5740 20 client.18600 trim_cache size 0 max 16384 2024-03-18T22:57:27.133 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:57:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T22:57:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:57:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "4b316b6b-7f0f-4c78-91b4-3b82fdb3e273": { 2024-03-18T22:57:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:57:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "4b316b6b-7f0f-4c78-91b4-3b82fdb3e273", 2024-03-18T22:57:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:57:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:57:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:57:27.134 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:57:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f5132640 1 -- 172.21.15.40:0/4107817936 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd9dc0a2800 msgr2=0x7fd9dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f5132640 1 --2- 172.21.15.40:0/4107817936 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd9dc0a2800 0x7fd9dc0a4cc0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd9e809b460 tx=0x7fd9e8066520 comp rx=0 tx=0).stop 2024-03-18T22:57:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f5132640 1 -- 172.21.15.40:0/4107817936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9f01022d0 msgr2=0x7fd9f0103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.126+0000 7fd9f5132640 1 --2- 172.21.15.40:0/4107817936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9f01022d0 0x7fd9f0103230 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd9e800e9b0 tx=0x7fd9e800ee70 comp rx=0 tx=0).stop 2024-03-18T22:57:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.127+0000 7fd9f5132640 1 -- 172.21.15.40:0/4107817936 shutdown_connections 2024-03-18T22:57:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.127+0000 7fd9f5132640 1 --2- 172.21.15.40:0/4107817936 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd9dc0a2800 0x7fd9dc0a4cc0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.127+0000 7fd9f5132640 1 --2- 172.21.15.40:0/4107817936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd9f01022d0 0x7fd9f0103230 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.127+0000 7fd9f5132640 1 --2- 172.21.15.40:0/4107817936 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd9f0101910 0x7fd9f0102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.127+0000 7fd9f5132640 1 --2- 172.21.15.40:0/4107817936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9f0100720 0x7fd9f01046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.127+0000 7fd9f5132640 1 -- 172.21.15.40:0/4107817936 >> 172.21.15.40:0/4107817936 conn(0x7fd9f00fbcb0 msgr2=0x7fd9f00689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.127+0000 7fd9f5132640 1 -- 172.21.15.40:0/4107817936 shutdown_connections 2024-03-18T22:57:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:27.127+0000 7fd9f5132640 1 -- 172.21.15.40:0/4107817936 wait complete. 2024-03-18T22:57:27.138 INFO:tasks.cephfs.filesystem:scrub status for tag:4b316b6b-7f0f-4c78-91b4-3b82fdb3e273 - {'path': '/', 'tag': '4b316b6b-7f0f-4c78-91b4-3b82fdb3e273', 'options': 'recursive,force'} 2024-03-18T22:57:28.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:28 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:57:28.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:28 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:28 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:57:28.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:28 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:57:28.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:28 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:57:28.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:28 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:57:28.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:28 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:57:28.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:28 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:28 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:28 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:28 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:28 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.581 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:28 smithi113 ceph-mon[66775]: pgmap v795: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.7 MiB/s wr, 950 op/s 2024-03-18T22:57:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:28 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:57:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:28 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:28 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:57:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:28 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:57:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:28 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:57:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:28 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:57:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:28 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:57:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:28 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:28 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:28 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:28 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:28 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:28 smithi103 ceph-mon[66377]: pgmap v795: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.7 MiB/s wr, 950 op/s 2024-03-18T22:57:28.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:28 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:57:28.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:28 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:28 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:57:28.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:28 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:57:28.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:28 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:57:28.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:28 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:57:28.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:28 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:57:28.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:28 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:28 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:28 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:28 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:28 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:57:28.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:28 smithi040 ceph-mon[62643]: pgmap v795: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.7 MiB/s wr, 950 op/s 2024-03-18T22:57:30.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:30 smithi113 ceph-mon[66775]: pgmap v796: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.4 MiB/s wr, 1.11k op/s 2024-03-18T22:57:30.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:30 smithi103 ceph-mon[66377]: pgmap v796: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.4 MiB/s wr, 1.11k op/s 2024-03-18T22:57:30.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:30 smithi040 ceph-mon[62643]: pgmap v796: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 4.4 MiB/s wr, 1.11k op/s 2024-03-18T22:57:32.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:32 smithi113 ceph-mon[66775]: pgmap v797: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.6 MiB/s wr, 673 op/s 2024-03-18T22:57:32.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:32 smithi103 ceph-mon[66377]: pgmap v797: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.6 MiB/s wr, 673 op/s 2024-03-18T22:57:32.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:32 smithi040 ceph-mon[62643]: pgmap v797: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.6 MiB/s wr, 673 op/s 2024-03-18T22:57:34.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:34 smithi113 ceph-mon[66775]: pgmap v798: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.6 MiB/s wr, 671 op/s 2024-03-18T22:57:34.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:34 smithi103 ceph-mon[66377]: pgmap v798: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.6 MiB/s wr, 671 op/s 2024-03-18T22:57:34.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:34 smithi040 ceph-mon[62643]: pgmap v798: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.6 MiB/s wr, 671 op/s 2024-03-18T22:57:36.582 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:36 smithi113 ceph-mon[66775]: pgmap v799: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 666 KiB/s wr, 174 op/s 2024-03-18T22:57:36.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:36 smithi103 ceph-mon[66377]: pgmap v799: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 666 KiB/s wr, 174 op/s 2024-03-18T22:57:36.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:36 smithi040 ceph-mon[62643]: pgmap v799: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 666 KiB/s wr, 174 op/s 2024-03-18T22:57:38.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:38 smithi113 ceph-mon[66775]: pgmap v800: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 665 KiB/s wr, 172 op/s 2024-03-18T22:57:38.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:38 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:57:38.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:38 smithi103 ceph-mon[66377]: pgmap v800: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 665 KiB/s wr, 172 op/s 2024-03-18T22:57:38.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:38 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:57:38.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:38 smithi040 ceph-mon[62643]: pgmap v800: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 665 KiB/s wr, 172 op/s 2024-03-18T22:57:38.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:38 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:57:39.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:39 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:57:39.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:39 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:57:39.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:39 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:57:39.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:39 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:57:39.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:39 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:57:39.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:39 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:57:39.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:39 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:57:39.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:39 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:57:39.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:39 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:57:39.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:39 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:57:39.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:39 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:57:39.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:39 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:57:40.580 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:40 smithi113 ceph-mon[66775]: pgmap v801: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 665 KiB/s wr, 173 op/s 2024-03-18T22:57:40.611 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:40 smithi103 ceph-mon[66377]: pgmap v801: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 665 KiB/s wr, 173 op/s 2024-03-18T22:57:40.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:40 smithi040 ceph-mon[62643]: pgmap v801: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 665 KiB/s wr, 173 op/s 2024-03-18T22:57:42.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:42 smithi113 ceph-mon[66775]: pgmap v802: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-03-18T22:57:42.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:42 smithi103 ceph-mon[66377]: pgmap v802: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-03-18T22:57:42.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:42 smithi040 ceph-mon[62643]: pgmap v802: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-03-18T22:57:44.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:44 smithi113 ceph-mon[66775]: pgmap v803: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:57:44.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:44 smithi103 ceph-mon[66377]: pgmap v803: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:57:44.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:44 smithi040 ceph-mon[62643]: pgmap v803: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:57:44.777 INFO:tasks.workunit.client.0.smithi040.stdout:Syncing()...38 sec 2024-03-18T22:57:44.777 INFO:tasks.workunit.client.0.smithi040.stdout:FFSB benchmark finished at: Mon Mar 18 22:57:44 2024 2024-03-18T22:57:44.777 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:57:44.777 INFO:tasks.workunit.client.0.smithi040.stdout:Results: 2024-03-18T22:57:44.777 INFO:tasks.workunit.client.0.smithi040.stdout:Benchmark took 346.20 sec 2024-03-18T22:57:44.778 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:57:44.778 INFO:tasks.workunit.client.0.smithi040.stdout:Total Results 2024-03-18T22:57:44.778 INFO:tasks.workunit.client.0.smithi040.stdout:=============== 2024-03-18T22:57:44.778 INFO:tasks.workunit.client.0.smithi040.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-03-18T22:57:44.778 INFO:tasks.workunit.client.0.smithi040.stdout: ======= ============ ========= ======= =========== ========== 2024-03-18T22:57:44.778 INFO:tasks.workunit.client.0.smithi040.stdout: write : 962560 2780.34 100.000% 100.000% 10.9MB/sec 2024-03-18T22:57:44.778 INFO:tasks.workunit.client.0.smithi040.stdout:- 2024-03-18T22:57:44.778 INFO:tasks.workunit.client.0.smithi040.stdout:2780.34 Transactions per Second 2024-03-18T22:57:44.778 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:57:44.778 INFO:tasks.workunit.client.0.smithi040.stdout:Throughput Results 2024-03-18T22:57:44.778 INFO:tasks.workunit.client.0.smithi040.stdout:=================== 2024-03-18T22:57:44.778 INFO:tasks.workunit.client.0.smithi040.stdout:Write Throughput: 10.9MB/sec 2024-03-18T22:57:44.782 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:57:44.782 INFO:tasks.workunit.client.0.smithi040.stdout:System Call Latency statistics in millisecs 2024-03-18T22:57:44.782 INFO:tasks.workunit.client.0.smithi040.stdout:===== 2024-03-18T22:57:44.782 INFO:tasks.workunit.client.0.smithi040.stdout: Min Avg Max Total Calls 2024-03-18T22:57:44.782 INFO:tasks.workunit.client.0.smithi040.stdout: ======== ======== ======== ============ 2024-03-18T22:57:44.782 INFO:tasks.workunit.client.0.smithi040.stdout:[ open] 0.649000 29.970739 354.619995 752 2024-03-18T22:57:44.782 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-18T22:57:44.782 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-18T22:57:44.782 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-18T22:57:44.783 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-18T22:57:44.783 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[4] 200.000000 - 500.000000 : 3 2024-03-18T22:57:44.783 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[5] 100.000000 - 200.000000 : 14 2024-03-18T22:57:44.783 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[6] 50.000000 - 100.000000 : 115 2024-03-18T22:57:44.783 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[7] 20.000000 - 50.000000 : 257 2024-03-18T22:57:44.783 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[8] 10.000000 - 20.000000 : 184 2024-03-18T22:57:44.783 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[9] 5.000000 - 10.000000 : 100 2024-03-18T22:57:44.783 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[10] 2.000000 - 5.000000 : 58 2024-03-18T22:57:44.783 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[11] 1.000000 - 2.000000 : 18 2024-03-18T22:57:44.783 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[12] 0.500000 - 1.000000 : 3 2024-03-18T22:57:44.783 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[13] 0.200000 - 0.500000 : 0 2024-03-18T22:57:44.783 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-03-18T22:57:44.783 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-03-18T22:57:44.783 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-03-18T22:57:44.783 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-03-18T22:57:44.784 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-03-18T22:57:44.784 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:57:44.784 INFO:tasks.workunit.client.0.smithi040.stdout:[ write] 0.093000 10.091821 830.841980 962560 2024-03-18T22:57:44.784 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-18T22:57:44.784 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-18T22:57:44.784 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-18T22:57:44.784 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[3] 500.000000 - 1000.000000 : 63 2024-03-18T22:57:44.784 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[4] 200.000000 - 500.000000 : 1380 2024-03-18T22:57:44.784 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[5] 100.000000 - 200.000000 : 6632 2024-03-18T22:57:44.784 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[6] 50.000000 - 100.000000 : 26258 2024-03-18T22:57:44.784 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[7] 20.000000 - 50.000000 : 106397 2024-03-18T22:57:44.784 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[8] 10.000000 - 20.000000 : 135625 2024-03-18T22:57:44.784 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[9] 5.000000 - 10.000000 : 129762 2024-03-18T22:57:44.784 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[10] 2.000000 - 5.000000 : 108454 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[11] 1.000000 - 2.000000 : 35152 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[12] 0.500000 - 1.000000 : 38893 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[13] 0.200000 - 0.500000 : 201673 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[14] 0.100000 - 0.200000 : 172236 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[15] 0.050000 - 0.100000 : 35 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout:[ lseek] 0.000000 0.000889 2.206000 962560 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-18T22:57:44.785 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-03-18T22:57:44.786 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-03-18T22:57:44.786 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-03-18T22:57:44.786 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-03-18T22:57:44.786 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2024-03-18T22:57:44.786 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[10] 2.000000 - 5.000000 : 1 2024-03-18T22:57:44.786 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[11] 1.000000 - 2.000000 : 0 2024-03-18T22:57:44.786 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[12] 0.500000 - 1.000000 : 2 2024-03-18T22:57:44.786 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[13] 0.200000 - 0.500000 : 6 2024-03-18T22:57:44.786 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[14] 0.100000 - 0.200000 : 10 2024-03-18T22:57:44.786 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[15] 0.050000 - 0.100000 : 10 2024-03-18T22:57:44.786 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[16] 0.020000 - 0.050000 : 138 2024-03-18T22:57:44.786 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[17] 0.010000 - 0.020000 : 708 2024-03-18T22:57:44.786 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[18] 0.000000 - 0.010000 : 961685 2024-03-18T22:57:44.786 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:57:44.787 INFO:tasks.workunit.client.0.smithi040.stdout:[ close] 0.022000 5.364082 793.708984 752 2024-03-18T22:57:44.787 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-18T22:57:44.787 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-18T22:57:44.787 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-18T22:57:44.787 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[3] 500.000000 - 1000.000000 : 1 2024-03-18T22:57:44.787 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-18T22:57:44.787 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[5] 100.000000 - 200.000000 : 3 2024-03-18T22:57:44.787 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[6] 50.000000 - 100.000000 : 10 2024-03-18T22:57:44.787 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[7] 20.000000 - 50.000000 : 37 2024-03-18T22:57:44.787 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[8] 10.000000 - 20.000000 : 22 2024-03-18T22:57:44.787 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[9] 5.000000 - 10.000000 : 37 2024-03-18T22:57:44.787 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[10] 2.000000 - 5.000000 : 67 2024-03-18T22:57:44.787 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[11] 1.000000 - 2.000000 : 46 2024-03-18T22:57:44.788 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[12] 0.500000 - 1.000000 : 26 2024-03-18T22:57:44.788 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[13] 0.200000 - 0.500000 : 56 2024-03-18T22:57:44.788 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[14] 0.100000 - 0.200000 : 133 2024-03-18T22:57:44.788 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[15] 0.050000 - 0.100000 : 72 2024-03-18T22:57:44.788 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[16] 0.020000 - 0.050000 : 242 2024-03-18T22:57:44.788 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-03-18T22:57:44.788 INFO:tasks.workunit.client.0.smithi040.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-03-18T22:57:44.788 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:57:44.788 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:57:44.788 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:57:44.788 INFO:tasks.workunit.client.0.smithi040.stdout: 2024-03-18T22:57:44.788 INFO:tasks.workunit.client.0.smithi040.stdout:0.4% User Time 2024-03-18T22:57:44.788 INFO:tasks.workunit.client.0.smithi040.stdout:31.4% System Time 2024-03-18T22:57:44.788 INFO:tasks.workunit.client.0.smithi040.stdout:31.8% CPU Utilization 2024-03-18T22:57:44.802 INFO:tasks.workunit.client.0.smithi040.stderr:+ cd .. 2024-03-18T22:57:44.810 INFO:tasks.workunit.client.0.smithi040.stderr:+ rm -r tmp ffsb 2024-03-18T22:57:46.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:46 smithi113 ceph-mon[66775]: pgmap v804: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:57:46.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:46 smithi103 ceph-mon[66377]: pgmap v804: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:57:46.646 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:46 smithi040 ceph-mon[62643]: pgmap v804: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:57:48.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:48 smithi113 ceph-mon[66775]: pgmap v805: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-18T22:57:48.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:48 smithi103 ceph-mon[66377]: pgmap v805: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-18T22:57:48.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:48 smithi040 ceph-mon[62643]: pgmap v805: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-18T22:57:50.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:50 smithi113 ceph-mon[66775]: pgmap v806: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-18T22:57:50.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:50 smithi103 ceph-mon[66377]: pgmap v806: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-18T22:57:50.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:50 smithi040 ceph-mon[62643]: pgmap v806: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-18T22:57:52.579 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:52 smithi113 ceph-mon[66775]: pgmap v807: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 KiB/s wr, 27 op/s 2024-03-18T22:57:52.608 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:52 smithi103 ceph-mon[66377]: pgmap v807: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 KiB/s wr, 27 op/s 2024-03-18T22:57:52.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:52 smithi040 ceph-mon[62643]: pgmap v807: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 KiB/s wr, 27 op/s 2024-03-18T22:57:53.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:53 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:53.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:53 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:53.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:53 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:54.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:54 smithi103 ceph-mon[66377]: pgmap v808: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 63 op/s 2024-03-18T22:57:54.609 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:54 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:54.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:54 smithi040 ceph-mon[62643]: pgmap v808: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 63 op/s 2024-03-18T22:57:54.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:54 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:54.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:54 smithi113 ceph-mon[66775]: pgmap v808: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 63 op/s 2024-03-18T22:57:54.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:54 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:55.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:55 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:55.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:55 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:55.942 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:55 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:56.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:56 smithi040 ceph-mon[62643]: pgmap v809: 129 pgs: 129 active+clean; 9.0 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 60 KiB/s wr, 95 op/s 2024-03-18T22:57:56.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:56 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:56.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:56 smithi113 ceph-mon[66775]: pgmap v809: 129 pgs: 129 active+clean; 9.0 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 60 KiB/s wr, 95 op/s 2024-03-18T22:57:56.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:56 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:56.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:56 smithi103 ceph-mon[66377]: pgmap v809: 129 pgs: 129 active+clean; 9.0 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 60 KiB/s wr, 95 op/s 2024-03-18T22:57:56.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:56 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:57.138 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:57:57.139 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:57:57.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.283+0000 7fdb3169a640 1 -- 172.21.15.40:0/3566267282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c101830 msgr2=0x7fdb2c101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:57.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.283+0000 7fdb3169a640 1 --2- 172.21.15.40:0/3566267282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c101830 0x7fdb2c101c50 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009ca0 tx=0x7fdb1c02c0f0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.283+0000 7fdb3169a640 1 -- 172.21.15.40:0/3566267282 shutdown_connections 2024-03-18T22:57:57.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.283+0000 7fdb3169a640 1 --2- 172.21.15.40:0/3566267282 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c1033e0 0x7fdb2c109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.283+0000 7fdb3169a640 1 --2- 172.21.15.40:0/3566267282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb2c102a20 0x7fdb2c102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.283+0000 7fdb3169a640 1 --2- 172.21.15.40:0/3566267282 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c101830 0x7fdb2c101c50 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.283+0000 7fdb3169a640 1 -- 172.21.15.40:0/3566267282 >> 172.21.15.40:0/3566267282 conn(0x7fdb2c0fcd80 msgr2=0x7fdb2c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:57.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.283+0000 7fdb3169a640 1 -- 172.21.15.40:0/3566267282 shutdown_connections 2024-03-18T22:57:57.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.283+0000 7fdb3169a640 1 -- 172.21.15.40:0/3566267282 wait complete. 2024-03-18T22:57:57.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.284+0000 7fdb3169a640 1 Processor -- start 2024-03-18T22:57:57.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.284+0000 7fdb3169a640 1 -- start start 2024-03-18T22:57:57.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.284+0000 7fdb3169a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c101830 0x7fdb2c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.284+0000 7fdb3169a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb2c102a20 0x7fdb2c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.284+0000 7fdb3169a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c1033e0 0x7fdb2c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.284+0000 7fdb3169a640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c10d3b0 con 0x7fdb2c101830 2024-03-18T22:57:57.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.284+0000 7fdb3169a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c10d230 con 0x7fdb2c102a20 2024-03-18T22:57:57.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.284+0000 7fdb3169a640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c10d530 con 0x7fdb2c1033e0 2024-03-18T22:57:57.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.284+0000 7fdb2affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c101830 0x7fdb2c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.284+0000 7fdb2affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c101830 0x7fdb2c197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45118/0 (socket says 172.21.15.40:45118) 2024-03-18T22:57:57.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.284+0000 7fdb2affd640 1 -- 172.21.15.40:0/1297880812 learned_addr learned my addr 172.21.15.40:0/1297880812 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:57.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.284+0000 7fdb2b7fe640 1 --2- 172.21.15.40:0/1297880812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c1033e0 0x7fdb2c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.284+0000 7fdb2a7fc640 1 --2- 172.21.15.40:0/1297880812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb2c102a20 0x7fdb2c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.285+0000 7fdb2affd640 1 -- 172.21.15.40:0/1297880812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c1033e0 msgr2=0x7fdb2c19f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:57.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.285+0000 7fdb2affd640 1 --2- 172.21.15.40:0/1297880812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c1033e0 0x7fdb2c19f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.285+0000 7fdb2affd640 1 -- 172.21.15.40:0/1297880812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb2c102a20 msgr2=0x7fdb2c198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:57.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.285+0000 7fdb2affd640 1 --2- 172.21.15.40:0/1297880812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb2c102a20 0x7fdb2c198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.285+0000 7fdb2affd640 1 -- 172.21.15.40:0/1297880812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb2c19f720 con 0x7fdb2c101830 2024-03-18T22:57:57.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.285+0000 7fdb2affd640 1 --2- 172.21.15.40:0/1297880812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c101830 0x7fdb2c197b50 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c032380 tx=0x7fdb1c032820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:57.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.285+0000 7fdb13fff640 1 -- 172.21.15.40:0/1297880812 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c041020 con 0x7fdb2c101830 2024-03-18T22:57:57.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.285+0000 7fdb3169a640 1 -- 172.21.15.40:0/1297880812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb2c19f9b0 con 0x7fdb2c101830 2024-03-18T22:57:57.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.285+0000 7fdb13fff640 1 -- 172.21.15.40:0/1297880812 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb1c003e20 con 0x7fdb2c101830 2024-03-18T22:57:57.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.285+0000 7fdb3169a640 1 -- 172.21.15.40:0/1297880812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb2c19fef0 con 0x7fdb2c101830 2024-03-18T22:57:57.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.285+0000 7fdb13fff640 1 -- 172.21.15.40:0/1297880812 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c039040 con 0x7fdb2c101830 2024-03-18T22:57:57.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.288+0000 7fdb13fff640 1 -- 172.21.15.40:0/1297880812 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdb1c044020 con 0x7fdb2c101830 2024-03-18T22:57:57.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.288+0000 7fdb34364740 1 Processor -- start 2024-03-18T22:57:57.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.288+0000 7fdb13fff640 1 --2- 172.21.15.40:0/1297880812 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb140a2850 0x7fdb140a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.288+0000 7fdb34364740 1 -- start start 2024-03-18T22:57:57.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb34364740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63e9b0 0x561b2c63edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb34364740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c63f3a0 0x561b2c641e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb34364740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c642500 0x561b2c64f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb34364740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561b2c5a3500 con 0x561b2c642500 2024-03-18T22:57:57.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb34364740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561b2c54bbe0 con 0x561b2c63e9b0 2024-03-18T22:57:57.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb13fff640 1 -- 172.21.15.40:0/1297880812 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdb1c0f0e10 con 0x7fdb2c101830 2024-03-18T22:57:57.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb34364740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x561b2c59f5a0 con 0x561b2c63f3a0 2024-03-18T22:57:57.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb2b7fe640 1 --2- 172.21.15.40:0/1297880812 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb140a2850 0x7fdb140a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb2b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c642500 0x561b2c64f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb2b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c642500 0x561b2c64f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45134/0 (socket says 172.21.15.40:45134) 2024-03-18T22:57:57.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb2b7fe640 1 -- 172.21.15.40:0/2366122751 learned_addr learned my addr 172.21.15.40:0/2366122751 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:57.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb2a7fc640 1 --2- 172.21.15.40:0/2366122751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c63f3a0 0x561b2c641e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb2affd640 1 --2- 172.21.15.40:0/2366122751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63e9b0 0x561b2c63edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb2b7fe640 1 -- 172.21.15.40:0/2366122751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c63f3a0 msgr2=0x561b2c641e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:57.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb2b7fe640 1 --2- 172.21.15.40:0/2366122751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c63f3a0 0x561b2c641e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb2b7fe640 1 -- 172.21.15.40:0/2366122751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63e9b0 msgr2=0x561b2c63edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:57.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb2b7fe640 1 --2- 172.21.15.40:0/2366122751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63e9b0 0x561b2c63edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb2b7fe640 1 -- 172.21.15.40:0/2366122751 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b2c56d000 con 0x561b2c642500 2024-03-18T22:57:57.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb2b7fe640 1 --2- 172.21.15.40:0/1297880812 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb140a2850 0x7fdb140a4d10 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdb24011970 tx=0x7fdb24009c30 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:57.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.289+0000 7fdb2b7fe640 1 --2- 172.21.15.40:0/2366122751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c642500 0x561b2c64f350 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fdb24005a70 tx=0x7fdb2401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=598bd4f870681bc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:57.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.290+0000 7fdb117fa640 1 -- 172.21.15.40:0/2366122751 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb24024070 con 0x561b2c642500 2024-03-18T22:57:57.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.290+0000 7fdb117fa640 1 -- 172.21.15.40:0/2366122751 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb2400fe50 con 0x561b2c642500 2024-03-18T22:57:57.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.290+0000 7fdb117fa640 1 -- 172.21.15.40:0/2366122751 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2401f6d0 con 0x561b2c642500 2024-03-18T22:57:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.291+0000 7fdb34364740 1 -- 172.21.15.40:0/2366122751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c642500 msgr2=0x561b2c64f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.291+0000 7fdb34364740 1 --2- 172.21.15.40:0/2366122751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c642500 0x561b2c64f350 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fdb24005a70 tx=0x7fdb2401d000 comp rx=0 tx=0).stop 2024-03-18T22:57:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.293+0000 7fdb34364740 1 -- 172.21.15.40:0/2366122751 shutdown_connections 2024-03-18T22:57:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.293+0000 7fdb34364740 1 --2- 172.21.15.40:0/2366122751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c642500 0x561b2c64f350 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.293+0000 7fdb34364740 1 --2- 172.21.15.40:0/2366122751 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c63f3a0 0x561b2c641e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.293+0000 7fdb34364740 1 --2- 172.21.15.40:0/2366122751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63e9b0 0x561b2c63edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.293+0000 7fdb34364740 1 -- 172.21.15.40:0/2366122751 >> 172.21.15.40:0/2366122751 conn(0x561b2c639fc0 msgr2=0x561b2c63a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.294+0000 7fdb34364740 1 -- 172.21.15.40:0/2366122751 shutdown_connections 2024-03-18T22:57:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.294+0000 7fdb34364740 1 -- 172.21.15.40:0/2366122751 wait complete. 2024-03-18T22:57:57.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.294+0000 7fdb34364740 1 -- start start 2024-03-18T22:57:57.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.294+0000 7fdb34364740 1 Processor -- start 2024-03-18T22:57:57.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb34364740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c651680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb34364740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63f3a0 0x561b2c651bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb34364740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c642500 0x561b2c658c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb2affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c651680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb34364740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x561b2c64fd20 con 0x561b2c63e9b0 2024-03-18T22:57:57.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb2affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c651680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45140/0 (socket says 172.21.15.40:45140) 2024-03-18T22:57:57.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb34364740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561b2c55f160 con 0x561b2c63f3a0 2024-03-18T22:57:57.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb2affd640 1 -- 172.21.15.40:0/3367066136 learned_addr learned my addr 172.21.15.40:0/3367066136 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:57.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb34364740 1 -- 172.21.15.40:0/3367066136 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x561b2c64fea0 con 0x561b2c642500 2024-03-18T22:57:57.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb34364740 10 client.0 _ll_register_callbacks cb 0x561b2c59ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:57.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb2b7fe640 1 --2- 172.21.15.40:0/3367066136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c642500 0x561b2c658c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb2affd640 1 -- 172.21.15.40:0/3367066136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c642500 msgr2=0x561b2c658c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:57.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb2affd640 1 --2- 172.21.15.40:0/3367066136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c642500 0x561b2c658c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb2affd640 1 -- 172.21.15.40:0/3367066136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63f3a0 msgr2=0x561b2c651bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:57.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb2affd640 1 --2- 172.21.15.40:0/3367066136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63f3a0 0x561b2c651bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.295+0000 7fdb2affd640 1 -- 172.21.15.40:0/3367066136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b2c3817d0 con 0x561b2c63e9b0 2024-03-18T22:57:57.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb2affd640 1 --2- 172.21.15.40:0/3367066136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c651680 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c0ee520 tx=0x7fdb1c0bcd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc17f842a2fada7f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:57.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb0eff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:57:57.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb0eff5640 1 -- 172.21.15.40:0/3367066136 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c0d6060 con 0x561b2c63e9b0 2024-03-18T22:57:57.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb0eff5640 1 -- 172.21.15.40:0/3367066136 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdb1c0bb6c0 con 0x561b2c63e9b0 2024-03-18T22:57:57.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb34364740 1 -- 172.21.15.40:0/3367066136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb2400fe50 con 0x561b2c63e9b0 2024-03-18T22:57:57.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb0eff5640 1 -- 172.21.15.40:0/3367066136 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fdb1c0c1510 con 0x561b2c63e9b0 2024-03-18T22:57:57.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb34364740 10 client.18612 fetch_fsmap learned FSMap version 44 2024-03-18T22:57:57.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb34364740 1 -- 172.21.15.40:0/3367066136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x561b2c2b73f0 con 0x561b2c63e9b0 2024-03-18T22:57:57.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb0eff5640 1 -- 172.21.15.40:0/3367066136 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb1c0c5b60 con 0x561b2c63e9b0 2024-03-18T22:57:57.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb0eff5640 1 -- 172.21.15.40:0/3367066136 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 44) ==== 8378+0+0 (secure 0 0 0) 0x7fdb1c03c030 con 0x561b2c63e9b0 2024-03-18T22:57:57.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb34364740 10 client.18612 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T22:57:57.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb34364740 10 client.18612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:57:57.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb34364740 20 client.18612 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:57.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb34364740 1 --2- 172.21.15.40:0/3367066136 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561b2c65d480 0x561b2c65f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb34364740 4 client.18612 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:57:57.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.296+0000 7fdb34364740 1 -- 172.21.15.40:0/3367066136 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fdb2400fe50 con 0x561b2c65d480 2024-03-18T22:57:57.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.297+0000 7fdb2a7fc640 1 --2- 172.21.15.40:0/3367066136 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561b2c65d480 0x561b2c65f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.297+0000 7fdb2a7fc640 1 --2- 172.21.15.40:0/3367066136 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561b2c65d480 0x561b2c65f900 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98e0cfcfbb55ea7c server_cookie=c7ff4e2a74cfb56c in_seq=0 out_seq=0 2024-03-18T22:57:57.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.297+0000 7fdb0eff5640 10 client.18612 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:57.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.301+0000 7fdb0eff5640 1 -- 172.21.15.40:0/3367066136 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fdb2400fe50 con 0x561b2c65d480 2024-03-18T22:57:57.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.301+0000 7fdb0eff5640 10 client.18612 handle_command_reply: tid=0 2024-03-18T22:57:57.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.301+0000 7fdb34364740 1 client.18612 shutdown 2024-03-18T22:57:57.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.301+0000 7fdb34364740 1 -- 172.21.15.40:0/3367066136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 msgr2=0x561b2c651680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:57.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.301+0000 7fdb34364740 1 --2- 172.21.15.40:0/3367066136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c651680 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c0ee520 tx=0x7fdb1c0bcd20 comp rx=0 tx=0).stop 2024-03-18T22:57:57.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.301+0000 7fdb34364740 1 -- 172.21.15.40:0/3367066136 shutdown_connections 2024-03-18T22:57:57.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.301+0000 7fdb34364740 1 --2- 172.21.15.40:0/3367066136 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c642500 0x561b2c658c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.301+0000 7fdb34364740 1 --2- 172.21.15.40:0/3367066136 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561b2c65d480 0x561b2c65f900 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.301+0000 7fdb34364740 1 --2- 172.21.15.40:0/3367066136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63f3a0 0x561b2c651bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.301+0000 7fdb34364740 1 --2- 172.21.15.40:0/3367066136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c651680 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.301+0000 7fdb34364740 1 -- 172.21.15.40:0/3367066136 >> 172.21.15.40:0/3367066136 conn(0x561b2c639fc0 msgr2=0x561b2c6391e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:57.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.301+0000 7fdb0eff5640 0 client.18612 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:57.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.301+0000 7fdb34364740 1 -- 172.21.15.40:0/3367066136 shutdown_connections 2024-03-18T22:57:57.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.301+0000 7fdb34364740 1 -- 172.21.15.40:0/3367066136 wait complete. 2024-03-18T22:57:57.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.301+0000 7fdb34364740 20 client.18612 trim_cache size 0 max 16384 2024-03-18T22:57:57.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.327+0000 7fdb34364740 1 Processor -- start 2024-03-18T22:57:57.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.327+0000 7fdb34364740 1 -- start start 2024-03-18T22:57:57.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.327+0000 7fdb34364740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c650290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.327+0000 7fdb34364740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63f3a0 0x561b2c658b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.327+0000 7fdb34364740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c642500 0x561b2c6519a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.327+0000 7fdb34364740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c1a8400 con 0x561b2c63e9b0 2024-03-18T22:57:57.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.327+0000 7fdb34364740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c1a9fd0 con 0x561b2c63f3a0 2024-03-18T22:57:57.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.327+0000 7fdb34364740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c1a8ad0 con 0x561b2c642500 2024-03-18T22:57:57.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.328+0000 7fdb2a7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63f3a0 0x561b2c658b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.328+0000 7fdb2affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c650290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.328+0000 7fdb2a7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63f3a0 0x561b2c658b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:45038/0 (socket says 172.21.15.40:45038) 2024-03-18T22:57:57.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.328+0000 7fdb2a7fc640 1 -- 172.21.15.40:0/4221504154 learned_addr learned my addr 172.21.15.40:0/4221504154 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.328+0000 7fdb2b7fe640 1 --2- 172.21.15.40:0/4221504154 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c642500 0x561b2c6519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.328+0000 7fdb2affd640 1 -- 172.21.15.40:0/4221504154 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c642500 msgr2=0x561b2c6519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.328+0000 7fdb2affd640 1 --2- 172.21.15.40:0/4221504154 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c642500 0x561b2c6519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.328+0000 7fdb2affd640 1 -- 172.21.15.40:0/4221504154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63f3a0 msgr2=0x561b2c658b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.328+0000 7fdb2affd640 1 --2- 172.21.15.40:0/4221504154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63f3a0 0x561b2c658b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.328+0000 7fdb2affd640 1 -- 172.21.15.40:0/4221504154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b2c2770f0 con 0x561b2c63e9b0 2024-03-18T22:57:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.328+0000 7fdb2affd640 1 --2- 172.21.15.40:0/4221504154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c650290 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c004af0 tx=0x7fdb1c0b9060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daffbe5f2c5b573c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:57.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.329+0000 7fdb10ff9640 1 -- 172.21.15.40:0/4221504154 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c0bdac0 con 0x561b2c63e9b0 2024-03-18T22:57:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.329+0000 7fdb10ff9640 1 -- 172.21.15.40:0/4221504154 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb1c0c5b60 con 0x561b2c63e9b0 2024-03-18T22:57:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.329+0000 7fdb10ff9640 1 -- 172.21.15.40:0/4221504154 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c0d6060 con 0x561b2c63e9b0 2024-03-18T22:57:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.330+0000 7fdb34364740 1 -- 172.21.15.40:0/4221504154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 msgr2=0x561b2c650290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:57.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.330+0000 7fdb34364740 1 --2- 172.21.15.40:0/4221504154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c650290 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c004af0 tx=0x7fdb1c0b9060 comp rx=0 tx=0).stop 2024-03-18T22:57:57.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.330+0000 7fdb34364740 1 -- 172.21.15.40:0/4221504154 shutdown_connections 2024-03-18T22:57:57.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.330+0000 7fdb34364740 1 --2- 172.21.15.40:0/4221504154 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c642500 0x561b2c6519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.330+0000 7fdb34364740 1 --2- 172.21.15.40:0/4221504154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63f3a0 0x561b2c658b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.330+0000 7fdb34364740 1 --2- 172.21.15.40:0/4221504154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c650290 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.330+0000 7fdb34364740 1 -- 172.21.15.40:0/4221504154 >> 172.21.15.40:0/4221504154 conn(0x561b2c639fc0 msgr2=0x561b2c651290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:57.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.335+0000 7fdb34364740 1 -- 172.21.15.40:0/4221504154 shutdown_connections 2024-03-18T22:57:57.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.336+0000 7fdb34364740 1 -- 172.21.15.40:0/4221504154 wait complete. 2024-03-18T22:57:57.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.336+0000 7fdb34364740 1 -- start start 2024-03-18T22:57:57.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.336+0000 7fdb34364740 1 Processor -- start 2024-03-18T22:57:57.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.336+0000 7fdb34364740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c655de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb34364740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63f3a0 0x561b2c656320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb34364740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c642500 0x561b2c6572e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb34364740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c1a98d0 con 0x561b2c63e9b0 2024-03-18T22:57:57.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb34364740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c1a91d0 con 0x561b2c63f3a0 2024-03-18T22:57:57.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb34364740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c1a63b0 con 0x561b2c642500 2024-03-18T22:57:57.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb34364740 10 client.0 _ll_register_callbacks cb 0x7fdb1c0b8fb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:57.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb2affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c655de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb2affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c655de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45156/0 (socket says 172.21.15.40:45156) 2024-03-18T22:57:57.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb2affd640 1 -- 172.21.15.40:0/3411146083 learned_addr learned my addr 172.21.15.40:0/3411146083 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:57.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb2a7fc640 1 --2- 172.21.15.40:0/3411146083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63f3a0 0x561b2c656320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb2b7fe640 1 --2- 172.21.15.40:0/3411146083 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c642500 0x561b2c6572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb2affd640 1 -- 172.21.15.40:0/3411146083 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c642500 msgr2=0x561b2c6572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:57.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb2affd640 1 --2- 172.21.15.40:0/3411146083 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c642500 0x561b2c6572e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb2affd640 1 -- 172.21.15.40:0/3411146083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63f3a0 msgr2=0x561b2c656320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:57.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb2affd640 1 --2- 172.21.15.40:0/3411146083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63f3a0 0x561b2c656320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.337+0000 7fdb2affd640 1 -- 172.21.15.40:0/3411146083 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b2c6578d0 con 0x561b2c63e9b0 2024-03-18T22:57:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.338+0000 7fdb2affd640 1 --2- 172.21.15.40:0/3411146083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c655de0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c0b9d90 tx=0x7fdb1c0bd540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=843cff74705cc85d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.338+0000 7fdb0eff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:57:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.338+0000 7fdb0eff5640 1 -- 172.21.15.40:0/3411146083 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c0bdac0 con 0x561b2c63e9b0 2024-03-18T22:57:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.338+0000 7fdb0eff5640 1 -- 172.21.15.40:0/3411146083 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdb1c0dd0d0 con 0x561b2c63e9b0 2024-03-18T22:57:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.338+0000 7fdb34364740 1 -- 172.21.15.40:0/3411146083 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb1c0c5b60 con 0x561b2c63e9b0 2024-03-18T22:57:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.338+0000 7fdb0eff5640 1 -- 172.21.15.40:0/3411146083 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fdb1c0dd2d0 con 0x561b2c63e9b0 2024-03-18T22:57:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.338+0000 7fdb34364740 10 client.18618 fetch_fsmap learned FSMap version 44 2024-03-18T22:57:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.338+0000 7fdb34364740 1 -- 172.21.15.40:0/3411146083 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x561b2c6578d0 con 0x561b2c63e9b0 2024-03-18T22:57:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.338+0000 7fdb0eff5640 1 -- 172.21.15.40:0/3411146083 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdb1c004b40 con 0x561b2c63e9b0 2024-03-18T22:57:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.339+0000 7fdb0eff5640 1 -- 172.21.15.40:0/3411146083 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 44) ==== 8378+0+0 (secure 0 0 0) 0x7fdb1c0bc810 con 0x561b2c63e9b0 2024-03-18T22:57:57.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.339+0000 7fdb34364740 10 client.18618 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T22:57:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.339+0000 7fdb34364740 10 client.18618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:57:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.339+0000 7fdb34364740 20 client.18618 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.339+0000 7fdb34364740 1 --2- 172.21.15.40:0/3411146083 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561b2c6c3900 0x561b2c6c5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.339+0000 7fdb34364740 4 client.18618 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:57:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.339+0000 7fdb34364740 1 -- 172.21.15.40:0/3411146083 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fdb1c0c5b60 con 0x561b2c6c3900 2024-03-18T22:57:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.339+0000 7fdb2a7fc640 1 --2- 172.21.15.40:0/3411146083 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561b2c6c3900 0x561b2c6c5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.339+0000 7fdb2a7fc640 1 --2- 172.21.15.40:0/3411146083 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561b2c6c3900 0x561b2c6c5cf0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=830ad585ec675b0c server_cookie=5b0427f96fa99e94 in_seq=0 out_seq=0 2024-03-18T22:57:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.339+0000 7fdb0eff5640 10 client.18618 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.341+0000 7fdb0eff5640 1 -- 172.21.15.40:0/3411146083 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fdb1c0c5b60 con 0x561b2c6c3900 2024-03-18T22:57:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.341+0000 7fdb0eff5640 10 client.18618 handle_command_reply: tid=0 2024-03-18T22:57:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.341+0000 7fdb34364740 1 client.18618 shutdown 2024-03-18T22:57:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.342+0000 7fdb34364740 1 -- 172.21.15.40:0/3411146083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 msgr2=0x561b2c655de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.342+0000 7fdb34364740 1 --2- 172.21.15.40:0/3411146083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c655de0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c0b9d90 tx=0x7fdb1c0bd540 comp rx=0 tx=0).stop 2024-03-18T22:57:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.342+0000 7fdb34364740 1 -- 172.21.15.40:0/3411146083 shutdown_connections 2024-03-18T22:57:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.342+0000 7fdb34364740 1 --2- 172.21.15.40:0/3411146083 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x561b2c642500 0x561b2c6572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.342+0000 7fdb34364740 1 --2- 172.21.15.40:0/3411146083 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x561b2c6c3900 0x561b2c6c5cf0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.342+0000 7fdb34364740 1 --2- 172.21.15.40:0/3411146083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561b2c63f3a0 0x561b2c656320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.342+0000 7fdb34364740 1 --2- 172.21.15.40:0/3411146083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b2c63e9b0 0x561b2c655de0 secure :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c0b9d90 tx=0x7fdb1c0bd540 comp rx=0 tx=0).stop 2024-03-18T22:57:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.342+0000 7fdb0eff5640 0 client.18618 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:57.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.342+0000 7fdb34364740 1 -- 172.21.15.40:0/3411146083 >> 172.21.15.40:0/3411146083 conn(0x561b2c639fc0 msgr2=0x561b2c63d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:57.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.342+0000 7fdb34364740 1 -- 172.21.15.40:0/3411146083 shutdown_connections 2024-03-18T22:57:57.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.342+0000 7fdb34364740 1 -- 172.21.15.40:0/3411146083 wait complete. 2024-03-18T22:57:57.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.343+0000 7fdb34364740 20 client.18618 trim_cache size 0 max 16384 2024-03-18T22:57:57.344 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:57:57.344 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:57:57.344 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:57:57.345 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:57:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.345+0000 7fdb3169a640 1 -- 172.21.15.40:0/1297880812 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb140a2850 msgr2=0x7fdb140a4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.345+0000 7fdb3169a640 1 --2- 172.21.15.40:0/1297880812 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb140a2850 0x7fdb140a4d10 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdb24011970 tx=0x7fdb24009c30 comp rx=0 tx=0).stop 2024-03-18T22:57:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.345+0000 7fdb3169a640 1 -- 172.21.15.40:0/1297880812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c101830 msgr2=0x7fdb2c197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.345+0000 7fdb3169a640 1 --2- 172.21.15.40:0/1297880812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c101830 0x7fdb2c197b50 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c032380 tx=0x7fdb1c032820 comp rx=0 tx=0).stop 2024-03-18T22:57:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.345+0000 7fdb3169a640 1 -- 172.21.15.40:0/1297880812 shutdown_connections 2024-03-18T22:57:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.345+0000 7fdb3169a640 1 --2- 172.21.15.40:0/1297880812 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdb140a2850 0x7fdb140a4d10 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.345+0000 7fdb3169a640 1 --2- 172.21.15.40:0/1297880812 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdb2c1033e0 0x7fdb2c19f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.345+0000 7fdb3169a640 1 --2- 172.21.15.40:0/1297880812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb2c102a20 0x7fdb2c198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.345+0000 7fdb3169a640 1 --2- 172.21.15.40:0/1297880812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb2c101830 0x7fdb2c197b50 secure :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c032380 tx=0x7fdb1c032820 comp rx=0 tx=0).stop 2024-03-18T22:57:57.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.345+0000 7fdb3169a640 1 -- 172.21.15.40:0/1297880812 >> 172.21.15.40:0/1297880812 conn(0x7fdb2c0fcd80 msgr2=0x7fdb2c105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:57.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.346+0000 7fdb3169a640 1 -- 172.21.15.40:0/1297880812 shutdown_connections 2024-03-18T22:57:57.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.347+0000 7fdb3169a640 1 -- 172.21.15.40:0/1297880812 wait complete. 2024-03-18T22:57:57.361 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:57:57.361 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:57:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.542+0000 7f932680b640 1 -- 172.21.15.40:0/508238465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f932010b190 msgr2=0x7f932010d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.542+0000 7f932680b640 1 --2- 172.21.15.40:0/508238465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f932010b190 0x7f932010d620 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9310009900 tx=0x7f931002bc60 comp rx=0 tx=0).stop 2024-03-18T22:57:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.543+0000 7f932680b640 1 -- 172.21.15.40:0/508238465 shutdown_connections 2024-03-18T22:57:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.543+0000 7f932680b640 1 --2- 172.21.15.40:0/508238465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f932010db60 0x7f932010e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.543+0000 7f932680b640 1 --2- 172.21.15.40:0/508238465 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f932010b190 0x7f932010d620 secure :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9310009900 tx=0x7f931002bc60 comp rx=0 tx=0).stop 2024-03-18T22:57:57.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.543+0000 7f932680b640 1 --2- 172.21.15.40:0/508238465 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9320108820 0x7f932010ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.543+0000 7f932680b640 1 -- 172.21.15.40:0/508238465 >> 172.21.15.40:0/508238465 conn(0x7f932006e690 msgr2=0x7f9320070ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:57.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.545+0000 7f932680b640 1 -- 172.21.15.40:0/508238465 shutdown_connections 2024-03-18T22:57:57.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.545+0000 7f932680b640 1 -- 172.21.15.40:0/508238465 wait complete. 2024-03-18T22:57:57.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.546+0000 7f932680b640 1 Processor -- start 2024-03-18T22:57:57.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.546+0000 7f932680b640 1 -- start start 2024-03-18T22:57:57.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.546+0000 7f932680b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9320108820 0x7f9320119ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.546+0000 7f932680b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f932010db60 0x7f932011a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.546+0000 7f932680b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9320118330 0x7f93201187e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.546+0000 7f932680b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9320110850 con 0x7f9320108820 2024-03-18T22:57:57.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.546+0000 7f932680b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f93201106d0 con 0x7f932010db60 2024-03-18T22:57:57.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.546+0000 7f932680b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f93201109d0 con 0x7f9320118330 2024-03-18T22:57:57.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.547+0000 7f931ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9320108820 0x7f9320119ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.547+0000 7f931ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9320108820 0x7f9320119ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45172/0 (socket says 172.21.15.40:45172) 2024-03-18T22:57:57.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.547+0000 7f931ffff640 1 -- 172.21.15.40:0/3932940534 learned_addr learned my addr 172.21.15.40:0/3932940534 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:57.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.547+0000 7f9324da8640 1 --2- 172.21.15.40:0/3932940534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9320118330 0x7f93201187e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.547+0000 7f931ffff640 1 -- 172.21.15.40:0/3932940534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9320118330 msgr2=0x7f93201187e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:57.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.547+0000 7f931ffff640 1 --2- 172.21.15.40:0/3932940534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9320118330 0x7f93201187e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.547+0000 7f931ffff640 1 -- 172.21.15.40:0/3932940534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f932010db60 msgr2=0x7f932011a1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T22:57:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.547+0000 7f931ffff640 1 --2- 172.21.15.40:0/3932940534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f932010db60 0x7f932011a1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.547+0000 7f931ffff640 1 -- 172.21.15.40:0/3932940534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9320119050 con 0x7f9320108820 2024-03-18T22:57:57.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.549+0000 7f931ffff640 1 --2- 172.21.15.40:0/3932940534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9320108820 0x7f9320119ca0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f931800b9c0 tx=0x7f931800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:57.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.550+0000 7f931d7fa640 1 -- 172.21.15.40:0/3932940534 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9318019070 con 0x7f9320108820 2024-03-18T22:57:57.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.551+0000 7f931d7fa640 1 -- 172.21.15.40:0/3932940534 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f931800cba0 con 0x7f9320108820 2024-03-18T22:57:57.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.551+0000 7f931d7fa640 1 -- 172.21.15.40:0/3932940534 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9318012580 con 0x7f9320108820 2024-03-18T22:57:57.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.551+0000 7f932680b640 1 -- 172.21.15.40:0/3932940534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93201b7740 con 0x7f9320108820 2024-03-18T22:57:57.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.551+0000 7f932680b640 1 -- 172.21.15.40:0/3932940534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93201b7cc0 con 0x7f9320108820 2024-03-18T22:57:57.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.555+0000 7f932680b640 1 -- 172.21.15.40:0/3932940534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9320112190 con 0x7f9320108820 2024-03-18T22:57:57.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.556+0000 7f931d7fa640 1 -- 172.21.15.40:0/3932940534 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9318014440 con 0x7f9320108820 2024-03-18T22:57:57.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.557+0000 7f931d7fa640 1 --2- 172.21.15.40:0/3932940534 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f93080a28d0 0x7f93080a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.557+0000 7f931d7fa640 1 -- 172.21.15.40:0/3932940534 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9318014730 con 0x7f9320108820 2024-03-18T22:57:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.558+0000 7f931f7fe640 1 --2- 172.21.15.40:0/3932940534 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f93080a28d0 0x7f93080a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:57.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.560+0000 7f931f7fe640 1 --2- 172.21.15.40:0/3932940534 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f93080a28d0 0x7f93080a4d90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9310009850 tx=0x7f9310005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:57.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.567+0000 7f931d7fa640 1 -- 172.21.15.40:0/3932940534 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f931804f740 con 0x7f9320108820 2024-03-18T22:57:57.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.954+0000 7f932680b640 1 -- 172.21.15.40:0/3932940534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93201b7fe0 con 0x7f9320108820 2024-03-18T22:57:57.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.956+0000 7f931d7fa640 1 -- 172.21.15.40:0/3932940534 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11958 (secure 0 0 0) 0x7f931809b110 con 0x7f9320108820 2024-03-18T22:57:57.956 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:57:57.956 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":45,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:57:56.473164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24509,24481,24473,24388,24401]},"id":1}]} 2024-03-18T22:57:57.957 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 45 2024-03-18T22:57:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.958+0000 7f932680b640 1 -- 172.21.15.40:0/3932940534 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f93080a28d0 msgr2=0x7f93080a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.958+0000 7f932680b640 1 --2- 172.21.15.40:0/3932940534 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f93080a28d0 0x7f93080a4d90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9310009850 tx=0x7f9310005fd0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.958+0000 7f932680b640 1 -- 172.21.15.40:0/3932940534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9320108820 msgr2=0x7f9320119ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.958+0000 7f932680b640 1 --2- 172.21.15.40:0/3932940534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9320108820 0x7f9320119ca0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f931800b9c0 tx=0x7f931800be80 comp rx=0 tx=0).stop 2024-03-18T22:57:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.958+0000 7f932680b640 1 -- 172.21.15.40:0/3932940534 shutdown_connections 2024-03-18T22:57:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.958+0000 7f932680b640 1 --2- 172.21.15.40:0/3932940534 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f93080a28d0 0x7f93080a4d90 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.958+0000 7f932680b640 1 --2- 172.21.15.40:0/3932940534 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9320118330 0x7f93201187e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.958+0000 7f932680b640 1 --2- 172.21.15.40:0/3932940534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f932010db60 0x7f932011a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.958+0000 7f932680b640 1 --2- 172.21.15.40:0/3932940534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9320108820 0x7f9320119ca0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.958+0000 7f932680b640 1 -- 172.21.15.40:0/3932940534 >> 172.21.15.40:0/3932940534 conn(0x7f932006e690 msgr2=0x7f932010bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.959+0000 7f932680b640 1 -- 172.21.15.40:0/3932940534 shutdown_connections 2024-03-18T22:57:57.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:57.959+0000 7f932680b640 1 -- 172.21.15.40:0/3932940534 wait complete. 2024-03-18T22:57:57.976 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:57:58.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.156+0000 7f1244ff4640 1 -- 172.21.15.40:0/4097907181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f124010db40 msgr2=0x7f124010dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.156+0000 7f1244ff4640 1 --2- 172.21.15.40:0/4097907181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f124010db40 0x7f124010dfc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f123801acb0 tx=0x7f123803d150 comp rx=0 tx=0).stop 2024-03-18T22:57:58.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.163+0000 7f1244ff4640 1 -- 172.21.15.40:0/4097907181 shutdown_connections 2024-03-18T22:57:58.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.163+0000 7f1244ff4640 1 --2- 172.21.15.40:0/4097907181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f124010db40 0x7f124010dfc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.163+0000 7f1244ff4640 1 --2- 172.21.15.40:0/4097907181 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f124010b190 0x7f124010d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.163+0000 7f1244ff4640 1 --2- 172.21.15.40:0/4097907181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1240108820 0x7f124010ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.163+0000 7f1244ff4640 1 -- 172.21.15.40:0/4097907181 >> 172.21.15.40:0/4097907181 conn(0x7f124006e8a0 msgr2=0x7f1240070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.170+0000 7f1244ff4640 1 -- 172.21.15.40:0/4097907181 shutdown_connections 2024-03-18T22:57:58.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.172+0000 7f1244ff4640 1 -- 172.21.15.40:0/4097907181 wait complete. 2024-03-18T22:57:58.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.173+0000 7f1244ff4640 1 Processor -- start 2024-03-18T22:57:58.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.178+0000 7f1244ff4640 1 -- start start 2024-03-18T22:57:58.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.178+0000 7f1244ff4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1240108820 0x7f124019e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.178+0000 7f1244ff4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f124010b190 0x7f124019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.178+0000 7f1244ff4640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f124010db40 0x7f12401a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.178+0000 7f1244ff4640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f12401119b0 con 0x7f1240108820 2024-03-18T22:57:58.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.178+0000 7f1244ff4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1240111830 con 0x7f124010b190 2024-03-18T22:57:58.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.178+0000 7f1244ff4640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1240111b30 con 0x7f124010db40 2024-03-18T22:57:58.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.178+0000 7f123e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1240108820 0x7f124019e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.179+0000 7f123e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1240108820 0x7f124019e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45186/0 (socket says 172.21.15.40:45186) 2024-03-18T22:57:58.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.179+0000 7f123e59c640 1 -- 172.21.15.40:0/1723192547 learned_addr learned my addr 172.21.15.40:0/1723192547 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.179+0000 7f123e59c640 1 -- 172.21.15.40:0/1723192547 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f124010db40 msgr2=0x7f12401a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.179+0000 7f123ed9d640 1 --2- 172.21.15.40:0/1723192547 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f124010db40 0x7f12401a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.179+0000 7f123e59c640 1 --2- 172.21.15.40:0/1723192547 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f124010db40 0x7f12401a5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.179+0000 7f123e59c640 1 -- 172.21.15.40:0/1723192547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f124010b190 msgr2=0x7f124019e620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T22:57:58.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.179+0000 7f123e59c640 1 --2- 172.21.15.40:0/1723192547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f124010b190 0x7f124019e620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.179+0000 7f123e59c640 1 -- 172.21.15.40:0/1723192547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12401a5d00 con 0x7f1240108820 2024-03-18T22:57:58.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.179+0000 7f123e59c640 1 --2- 172.21.15.40:0/1723192547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1240108820 0x7f124019e0e0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f123400b450 tx=0x7f123400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.181+0000 7f122f7fe640 1 -- 172.21.15.40:0/1723192547 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1234015020 con 0x7f1240108820 2024-03-18T22:57:58.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.181+0000 7f122f7fe640 1 -- 172.21.15.40:0/1723192547 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1234011d40 con 0x7f1240108820 2024-03-18T22:57:58.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.181+0000 7f122f7fe640 1 -- 172.21.15.40:0/1723192547 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f123400f340 con 0x7f1240108820 2024-03-18T22:57:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.182+0000 7f1244ff4640 1 -- 172.21.15.40:0/1723192547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12401a5ff0 con 0x7f1240108820 2024-03-18T22:57:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.182+0000 7f1244ff4640 1 -- 172.21.15.40:0/1723192547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12401a6400 con 0x7f1240108820 2024-03-18T22:57:58.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.186+0000 7f122f7fe640 1 -- 172.21.15.40:0/1723192547 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1234018020 con 0x7f1240108820 2024-03-18T22:57:58.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.186+0000 7f122f7fe640 1 --2- 172.21.15.40:0/1723192547 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f12280a2a00 0x7f12280a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.187+0000 7f122f7fe640 1 -- 172.21.15.40:0/1723192547 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f12340cf910 con 0x7f1240108820 2024-03-18T22:57:58.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.187+0000 7f1247be7740 1 Processor -- start 2024-03-18T22:57:58.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.187+0000 7f123dd9b640 1 --2- 172.21.15.40:0/1723192547 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f12280a2a00 0x7f12280a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.187+0000 7f1247be7740 1 -- start start 2024-03-18T22:57:58.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.187+0000 7f1247be7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd942f0 0x55da2cd94710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.187+0000 7f1247be7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd94ce0 0x55da2cd997e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.187+0000 7f1247be7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd99e50 0x55da2cd9c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.187+0000 7f1247be7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55da2ccf84b0 con 0x55da2cd942f0 2024-03-18T22:57:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.187+0000 7f1247be7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55da2cca0d70 con 0x55da2cd99e50 2024-03-18T22:57:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.187+0000 7f1247be7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55da2ccf4550 con 0x55da2cd94ce0 2024-03-18T22:57:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.188+0000 7f123e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd942f0 0x55da2cd94710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.188+0000 7f123e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd942f0 0x55da2cd94710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45194/0 (socket says 172.21.15.40:45194) 2024-03-18T22:57:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.188+0000 7f123e59c640 1 -- 172.21.15.40:0/2162848243 learned_addr learned my addr 172.21.15.40:0/2162848243 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.188+0000 7f123e59c640 1 -- 172.21.15.40:0/2162848243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd94ce0 msgr2=0x55da2cd997e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:57:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.188+0000 7f123e59c640 1 --2- 172.21.15.40:0/2162848243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd94ce0 0x55da2cd997e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.188+0000 7f123e59c640 1 -- 172.21.15.40:0/2162848243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd99e50 msgr2=0x55da2cd9c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.188+0000 7f123e59c640 1 --2- 172.21.15.40:0/2162848243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd99e50 0x55da2cd9c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.188+0000 7f123e59c640 1 -- 172.21.15.40:0/2162848243 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da2ccc3210 con 0x55da2cd942f0 2024-03-18T22:57:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.189+0000 7f123e59c640 1 --2- 172.21.15.40:0/2162848243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd942f0 0x55da2cd94710 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f12340982c0 tx=0x7f123409c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=122202db254b388e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.190+0000 7f122cff9640 1 -- 172.21.15.40:0/2162848243 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12340a4600 con 0x55da2cd942f0 2024-03-18T22:57:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.190+0000 7f122cff9640 1 -- 172.21.15.40:0/2162848243 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f123409a160 con 0x55da2cd942f0 2024-03-18T22:57:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.191+0000 7f1247be7740 1 -- 172.21.15.40:0/2162848243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd942f0 msgr2=0x55da2cd94710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.191+0000 7f1247be7740 1 --2- 172.21.15.40:0/2162848243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd942f0 0x55da2cd94710 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f12340982c0 tx=0x7f123409c6a0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.191+0000 7f123dd9b640 1 --2- 172.21.15.40:0/1723192547 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f12280a2a00 0x7f12280a4ec0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f124019f5f0 tx=0x7f123000b370 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.192+0000 7f1247be7740 1 -- 172.21.15.40:0/2162848243 shutdown_connections 2024-03-18T22:57:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.192+0000 7f1247be7740 1 --2- 172.21.15.40:0/2162848243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd99e50 0x55da2cd9c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.192+0000 7f1247be7740 1 --2- 172.21.15.40:0/2162848243 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd94ce0 0x55da2cd997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.192+0000 7f1247be7740 1 --2- 172.21.15.40:0/2162848243 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd942f0 0x55da2cd94710 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.192+0000 7f1247be7740 1 -- 172.21.15.40:0/2162848243 >> 172.21.15.40:0/2162848243 conn(0x55da2cd8f860 msgr2=0x55da2cd8fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.192+0000 7f1247be7740 1 -- 172.21.15.40:0/2162848243 shutdown_connections 2024-03-18T22:57:58.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.192+0000 7f1247be7740 1 -- 172.21.15.40:0/2162848243 wait complete. 2024-03-18T22:57:58.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.193+0000 7f1247be7740 1 -- start start 2024-03-18T22:57:58.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.193+0000 7f1247be7740 1 Processor -- start 2024-03-18T22:57:58.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.193+0000 7f1247be7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd942f0 0x55da2cd9ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.193+0000 7f1247be7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd94ce0 0x55da2cd9efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.193+0000 7f1247be7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd99e50 0x55da2cda5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.193+0000 7f1247be7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55da2cd9ce50 con 0x55da2cd942f0 2024-03-18T22:57:58.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.193+0000 7f1247be7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55da2ccb4310 con 0x55da2cd94ce0 2024-03-18T22:57:58.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.193+0000 7f1247be7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55da2cd9cfd0 con 0x55da2cd99e50 2024-03-18T22:57:58.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.193+0000 7f1247be7740 10 client.0 _ll_register_callbacks cb 0x55da2cca82d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:58.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.193+0000 7f123ed9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd99e50 0x55da2cda5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.194+0000 7f123ed9d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd99e50 0x55da2cda5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57100/0 (socket says 172.21.15.40:57100) 2024-03-18T22:57:58.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.194+0000 7f123ed9d640 1 -- 172.21.15.40:0/2146980881 learned_addr learned my addr 172.21.15.40:0/2146980881 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.194+0000 7f123e59c640 1 --2- 172.21.15.40:0/2146980881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd942f0 0x55da2cd9ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.194+0000 7f123e59c640 1 -- 172.21.15.40:0/2146980881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd99e50 msgr2=0x55da2cda5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.194+0000 7f123e59c640 1 --2- 172.21.15.40:0/2146980881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd99e50 0x55da2cda5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.194+0000 7f123e59c640 1 -- 172.21.15.40:0/2146980881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd94ce0 msgr2=0x55da2cd9efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.194+0000 7f123e59c640 1 --2- 172.21.15.40:0/2146980881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd94ce0 0x55da2cd9efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.194+0000 7f123e59c640 1 -- 172.21.15.40:0/2146980881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da2ca0c3f0 con 0x55da2cd942f0 2024-03-18T22:57:58.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.196+0000 7f123e59c640 1 --2- 172.21.15.40:0/2146980881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd942f0 0x55da2cd9ea70 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f123409bdf0 tx=0x7f1234010960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f81e8faadc5dd913 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.196+0000 7f1225ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:57:58.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.197+0000 7f1225ffb640 1 -- 172.21.15.40:0/2146980881 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12340a9060 con 0x55da2cd942f0 2024-03-18T22:57:58.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.197+0000 7f1247be7740 1 -- 172.21.15.40:0/2146980881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55da2cd563e0 con 0x55da2cd942f0 2024-03-18T22:57:58.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.197+0000 7f1225ffb640 1 -- 172.21.15.40:0/2146980881 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f12340a4170 con 0x55da2cd942f0 2024-03-18T22:57:58.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.199+0000 7f1225ffb640 1 -- 172.21.15.40:0/2146980881 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f12340a4370 con 0x55da2cd942f0 2024-03-18T22:57:58.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.199+0000 7f1247be7740 10 client.18630 fetch_fsmap learned FSMap version 45 2024-03-18T22:57:58.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.199+0000 7f1247be7740 1 -- 172.21.15.40:0/2146980881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55da2cad67d0 con 0x55da2cd942f0 2024-03-18T22:57:58.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.203+0000 7f1225ffb640 1 -- 172.21.15.40:0/2146980881 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55da2cd563e0 con 0x55da2cd942f0 2024-03-18T22:57:58.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.203+0000 7f1225ffb640 1 -- 172.21.15.40:0/2146980881 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 45) ==== 8394+0+0 (secure 0 0 0) 0x7f123409aec0 con 0x55da2cd942f0 2024-03-18T22:57:58.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.203+0000 7f1247be7740 10 client.18630 fetch_fsmap finished waiting for FSMap version 45 2024-03-18T22:57:58.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.203+0000 7f1247be7740 10 client.18630 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:57:58.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.203+0000 7f1247be7740 20 client.18630 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:58.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.203+0000 7f1247be7740 1 --2- 172.21.15.40:0/2146980881 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55da2cdaaae0 0x55da2cdacf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.203+0000 7f1247be7740 4 client.18630 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:57:58.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.203+0000 7f1247be7740 1 -- 172.21.15.40:0/2146980881 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da2cd059f0 con 0x55da2cdaaae0 2024-03-18T22:57:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.206+0000 7f123dd9b640 1 --2- 172.21.15.40:0/2146980881 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55da2cdaaae0 0x55da2cdacf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.208+0000 7f123dd9b640 1 --2- 172.21.15.40:0/2146980881 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55da2cdaaae0 0x55da2cdacf00 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=72bdfa506205db4 server_cookie=19d1930db4d5cd3 in_seq=0 out_seq=0 2024-03-18T22:57:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.209+0000 7f1225ffb640 10 client.18630 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:57:58.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.211+0000 7f1225ffb640 1 -- 172.21.15.40:0/2146980881 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55da2cd059f0 con 0x55da2cdaaae0 2024-03-18T22:57:58.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.211+0000 7f1225ffb640 10 client.18630 handle_command_reply: tid=0 2024-03-18T22:57:58.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.212+0000 7f1247be7740 1 client.18630 shutdown 2024-03-18T22:57:58.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.213+0000 7f1247be7740 1 -- 172.21.15.40:0/2146980881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd942f0 msgr2=0x55da2cd9ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.213+0000 7f1247be7740 1 --2- 172.21.15.40:0/2146980881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd942f0 0x55da2cd9ea70 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f123409bdf0 tx=0x7f1234010960 comp rx=0 tx=0).stop 2024-03-18T22:57:58.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.213+0000 7f1247be7740 1 -- 172.21.15.40:0/2146980881 shutdown_connections 2024-03-18T22:57:58.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.213+0000 7f1247be7740 1 --2- 172.21.15.40:0/2146980881 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd99e50 0x55da2cda5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.213+0000 7f1247be7740 1 --2- 172.21.15.40:0/2146980881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd94ce0 0x55da2cd9efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.213+0000 7f1247be7740 1 --2- 172.21.15.40:0/2146980881 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55da2cdaaae0 0x55da2cdacf00 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.213+0000 7f1247be7740 1 --2- 172.21.15.40:0/2146980881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd942f0 0x55da2cd9ea70 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.213+0000 7f1247be7740 1 -- 172.21.15.40:0/2146980881 >> 172.21.15.40:0/2146980881 conn(0x55da2cd8f860 msgr2=0x55da2cd8eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.214+0000 7f1225ffb640 0 client.18630 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:57:58.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.215+0000 7f1247be7740 1 -- 172.21.15.40:0/2146980881 shutdown_connections 2024-03-18T22:57:58.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.215+0000 7f1247be7740 1 -- 172.21.15.40:0/2146980881 wait complete. 2024-03-18T22:57:58.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.216+0000 7f1247be7740 20 client.18630 trim_cache size 0 max 16384 2024-03-18T22:57:58.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.246+0000 7f1247be7740 1 Processor -- start 2024-03-18T22:57:58.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.248+0000 7f1247be7740 1 -- start start 2024-03-18T22:57:58.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.248+0000 7f1247be7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd942f0 0x55da2cd9ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.248+0000 7f1247be7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd94ce0 0x55da2cda5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.249+0000 7f123e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd942f0 0x55da2cd9ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.249+0000 7f123e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd942f0 0x55da2cd9ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57114/0 (socket says 172.21.15.40:57114) 2024-03-18T22:57:58.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.249+0000 7f1247be7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd99e50 0x55da2cd8e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.249+0000 7f123dd9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd94ce0 0x55da2cda5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.249+0000 7f123dd9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd94ce0 0x55da2cda5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:45086/0 (socket says 172.21.15.40:45086) 2024-03-18T22:57:58.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.249+0000 7f123e59c640 1 -- 172.21.15.40:0/1622652979 learned_addr learned my addr 172.21.15.40:0/1622652979 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.249+0000 7f1247be7740 1 -- 172.21.15.40:0/1622652979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f12401b0c80 con 0x55da2cd99e50 2024-03-18T22:57:58.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.249+0000 7f1247be7740 1 -- 172.21.15.40:0/1622652979 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f12401b28e0 con 0x55da2cd94ce0 2024-03-18T22:57:58.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.249+0000 7f1247be7740 1 -- 172.21.15.40:0/1622652979 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f12401b1350 con 0x55da2cd942f0 2024-03-18T22:57:58.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.249+0000 7f123ed9d640 1 --2- 172.21.15.40:0/1622652979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd99e50 0x55da2cd8e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.252+0000 7f123ed9d640 1 -- 172.21.15.40:0/1622652979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd942f0 msgr2=0x55da2cd9ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.252+0000 7f123ed9d640 1 --2- 172.21.15.40:0/1622652979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd942f0 0x55da2cd9ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.252+0000 7f123ed9d640 1 -- 172.21.15.40:0/1622652979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd94ce0 msgr2=0x55da2cda5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.252+0000 7f123ed9d640 1 --2- 172.21.15.40:0/1622652979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd94ce0 0x55da2cda5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.252+0000 7f123ed9d640 1 -- 172.21.15.40:0/1622652979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da2cd905d0 con 0x55da2cd99e50 2024-03-18T22:57:58.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.255+0000 7f123ed9d640 1 --2- 172.21.15.40:0/1622652979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd99e50 0x55da2cd8e310 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x55da2cd907a0 tx=0x7f1238005f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632982d6e7f54376 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.256+0000 7f1227fff640 1 -- 172.21.15.40:0/1622652979 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1238054070 con 0x55da2cd99e50 2024-03-18T22:57:58.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.256+0000 7f1227fff640 1 -- 172.21.15.40:0/1622652979 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f123804b040 con 0x55da2cd99e50 2024-03-18T22:57:58.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.256+0000 7f1227fff640 1 -- 172.21.15.40:0/1622652979 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f123804a040 con 0x55da2cd99e50 2024-03-18T22:57:58.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.257+0000 7f1247be7740 1 -- 172.21.15.40:0/1622652979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd99e50 msgr2=0x55da2cd8e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.257+0000 7f1247be7740 1 --2- 172.21.15.40:0/1622652979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd99e50 0x55da2cd8e310 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x55da2cd907a0 tx=0x7f1238005f60 comp rx=0 tx=0).stop 2024-03-18T22:57:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.260+0000 7f1247be7740 1 -- 172.21.15.40:0/1622652979 shutdown_connections 2024-03-18T22:57:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.260+0000 7f1247be7740 1 --2- 172.21.15.40:0/1622652979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd99e50 0x55da2cd8e310 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.260+0000 7f1247be7740 1 --2- 172.21.15.40:0/1622652979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd94ce0 0x55da2cda5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.260+0000 7f1247be7740 1 --2- 172.21.15.40:0/1622652979 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd942f0 0x55da2cd9ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.260+0000 7f1247be7740 1 -- 172.21.15.40:0/1622652979 >> 172.21.15.40:0/1622652979 conn(0x55da2cd8f860 msgr2=0x55da2cd9e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.263+0000 7f1247be7740 1 -- 172.21.15.40:0/1622652979 shutdown_connections 2024-03-18T22:57:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.266+0000 7f1247be7740 1 -- 172.21.15.40:0/1622652979 wait complete. 2024-03-18T22:57:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.266+0000 7f1247be7740 1 -- start start 2024-03-18T22:57:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.266+0000 7f1247be7740 1 Processor -- start 2024-03-18T22:57:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.267+0000 7f1247be7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd942f0 0x55da2cda5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.267+0000 7f1247be7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd94ce0 0x55da2cda28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.267+0000 7f1247be7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd99e50 0x55da2cda2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.267+0000 7f1247be7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f12401b21e0 con 0x55da2cd99e50 2024-03-18T22:57:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.267+0000 7f1247be7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f12401b1ae0 con 0x55da2cd942f0 2024-03-18T22:57:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.267+0000 7f1247be7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f12401aec30 con 0x55da2cd94ce0 2024-03-18T22:57:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.267+0000 7f123dd9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd94ce0 0x55da2cda28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.267+0000 7f123dd9b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd94ce0 0x55da2cda28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57120/0 (socket says 172.21.15.40:57120) 2024-03-18T22:57:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.267+0000 7f123dd9b640 1 -- 172.21.15.40:0/1543585980 learned_addr learned my addr 172.21.15.40:0/1543585980 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.267+0000 7f123ed9d640 1 --2- 172.21.15.40:0/1543585980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd99e50 0x55da2cda2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.267+0000 7f1247be7740 10 client.0 _ll_register_callbacks cb 0x7f123409b850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:58.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.269+0000 7f123dd9b640 1 -- 172.21.15.40:0/1543585980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd942f0 msgr2=0x55da2cda5e40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:57:58.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.269+0000 7f123dd9b640 1 --2- 172.21.15.40:0/1543585980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd942f0 0x55da2cda5e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.269+0000 7f123dd9b640 1 -- 172.21.15.40:0/1543585980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd99e50 msgr2=0x55da2cda2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.269+0000 7f123dd9b640 1 --2- 172.21.15.40:0/1543585980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd99e50 0x55da2cda2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.269+0000 7f123dd9b640 1 -- 172.21.15.40:0/1543585980 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da2ce0ee10 con 0x55da2cd94ce0 2024-03-18T22:57:58.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.270+0000 7f123dd9b640 1 --2- 172.21.15.40:0/1543585980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd94ce0 0x55da2cda28e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1230012040 tx=0x7f1230004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77501df4fc7eca73 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.271+0000 7f1225ffb640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:57:58.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.271+0000 7f1225ffb640 1 -- 172.21.15.40:0/1543585980 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1230052060 con 0x55da2cd94ce0 2024-03-18T22:57:58.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.271+0000 7f1247be7740 1 -- 172.21.15.40:0/1543585980 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f123804b040 con 0x55da2cd94ce0 2024-03-18T22:57:58.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.271+0000 7f1225ffb640 1 -- 172.21.15.40:0/1543585980 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1230010870 con 0x55da2cd94ce0 2024-03-18T22:57:58.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.271+0000 7f1225ffb640 1 -- 172.21.15.40:0/1543585980 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f1230059050 con 0x55da2cd94ce0 2024-03-18T22:57:58.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.273+0000 7f1247be7740 10 client.28450 fetch_fsmap learned FSMap version 45 2024-03-18T22:57:58.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.273+0000 7f1247be7740 1 -- 172.21.15.40:0/1543585980 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55da2ce100e0 con 0x55da2cd94ce0 2024-03-18T22:57:58.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.275+0000 7f1225ffb640 1 -- 172.21.15.40:0/1543585980 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1230016050 con 0x55da2cd94ce0 2024-03-18T22:57:58.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.275+0000 7f1225ffb640 1 -- 172.21.15.40:0/1543585980 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 45) ==== 8394+0+0 (secure 0 0 0) 0x7f123001ec80 con 0x55da2cd94ce0 2024-03-18T22:57:58.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.275+0000 7f1247be7740 10 client.28450 fetch_fsmap finished waiting for FSMap version 45 2024-03-18T22:57:58.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.275+0000 7f1247be7740 10 client.28450 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:57:58.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.275+0000 7f1247be7740 20 client.28450 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:58.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.275+0000 7f1247be7740 1 --2- 172.21.15.40:0/1543585980 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55da2ce10aa0 0x55da2ce12ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.275+0000 7f1247be7740 4 client.28450 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:57:58.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.275+0000 7f1247be7740 1 -- 172.21.15.40:0/1543585980 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f123804b040 con 0x55da2ce10aa0 2024-03-18T22:57:58.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.283+0000 7f123e59c640 1 --2- 172.21.15.40:0/1543585980 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55da2ce10aa0 0x55da2ce12ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.304+0000 7f123e59c640 1 --2- 172.21.15.40:0/1543585980 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55da2ce10aa0 0x55da2ce12ec0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3db2c834d221c0e8 server_cookie=3605e7b6a34818f7 in_seq=0 out_seq=0 2024-03-18T22:57:58.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.304+0000 7f1225ffb640 10 client.28450 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:57:58.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.308+0000 7f1225ffb640 1 -- 172.21.15.40:0/1543585980 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f123804b040 con 0x55da2ce10aa0 2024-03-18T22:57:58.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.308+0000 7f1225ffb640 10 client.28450 handle_command_reply: tid=0 2024-03-18T22:57:58.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.309+0000 7f1247be7740 1 client.28450 shutdown 2024-03-18T22:57:58.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.311+0000 7f1247be7740 1 -- 172.21.15.40:0/1543585980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd94ce0 msgr2=0x55da2cda28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.311+0000 7f1247be7740 1 --2- 172.21.15.40:0/1543585980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd94ce0 0x55da2cda28e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1230012040 tx=0x7f1230004730 comp rx=0 tx=0).stop 2024-03-18T22:57:58.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.311+0000 7f1247be7740 1 -- 172.21.15.40:0/1543585980 shutdown_connections 2024-03-18T22:57:58.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.311+0000 7f1247be7740 1 --2- 172.21.15.40:0/1543585980 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x55da2ce10aa0 0x55da2ce12ec0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.311+0000 7f1247be7740 1 --2- 172.21.15.40:0/1543585980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55da2cd99e50 0x55da2cda2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.311+0000 7f1247be7740 1 --2- 172.21.15.40:0/1543585980 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55da2cd94ce0 0x55da2cda28e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.311+0000 7f1247be7740 1 --2- 172.21.15.40:0/1543585980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55da2cd942f0 0x55da2cda5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.311+0000 7f1247be7740 1 -- 172.21.15.40:0/1543585980 >> 172.21.15.40:0/1543585980 conn(0x55da2cd8f860 msgr2=0x55da2cd92ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.311+0000 7f1225ffb640 0 client.28450 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:57:58.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.311+0000 7f1247be7740 1 -- 172.21.15.40:0/1543585980 shutdown_connections 2024-03-18T22:57:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.313+0000 7f1247be7740 1 -- 172.21.15.40:0/1543585980 wait complete. 2024-03-18T22:57:58.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.314+0000 7f1247be7740 20 client.28450 trim_cache size 0 max 16384 2024-03-18T22:57:58.317 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:57:58.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.318+0000 7f1244ff4640 1 -- 172.21.15.40:0/1723192547 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f12280a2a00 msgr2=0x7f12280a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.318+0000 7f1244ff4640 1 --2- 172.21.15.40:0/1723192547 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f12280a2a00 0x7f12280a4ec0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f124019f5f0 tx=0x7f123000b370 comp rx=0 tx=0).stop 2024-03-18T22:57:58.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.318+0000 7f1244ff4640 1 -- 172.21.15.40:0/1723192547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1240108820 msgr2=0x7f124019e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.318+0000 7f1244ff4640 1 --2- 172.21.15.40:0/1723192547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1240108820 0x7f124019e0e0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f123400b450 tx=0x7f123400b910 comp rx=0 tx=0).stop 2024-03-18T22:57:58.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.318+0000 7f1244ff4640 1 -- 172.21.15.40:0/1723192547 shutdown_connections 2024-03-18T22:57:58.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.318+0000 7f1244ff4640 1 --2- 172.21.15.40:0/1723192547 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f12280a2a00 0x7f12280a4ec0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.318+0000 7f1244ff4640 1 --2- 172.21.15.40:0/1723192547 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f124010db40 0x7f12401a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.318+0000 7f1244ff4640 1 --2- 172.21.15.40:0/1723192547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f124010b190 0x7f124019e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.318+0000 7f1244ff4640 1 --2- 172.21.15.40:0/1723192547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1240108820 0x7f124019e0e0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.319+0000 7f1244ff4640 1 -- 172.21.15.40:0/1723192547 >> 172.21.15.40:0/1723192547 conn(0x7f124006e8a0 msgr2=0x7f124010b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.319+0000 7f1244ff4640 1 -- 172.21.15.40:0/1723192547 shutdown_connections 2024-03-18T22:57:58.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.319+0000 7f1244ff4640 1 -- 172.21.15.40:0/1723192547 wait complete. 2024-03-18T22:57:58.338 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:57:58.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.493+0000 7f9e35b87640 1 -- 172.21.15.40:0/195442647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30107540 msgr2=0x7f9e30109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.493+0000 7f9e35b87640 1 --2- 172.21.15.40:0/195442647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30107540 0x7f9e30109990 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009900 tx=0x7f9e24030040 comp rx=0 tx=0).stop 2024-03-18T22:57:58.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.493+0000 7f9e35b87640 1 -- 172.21.15.40:0/195442647 shutdown_connections 2024-03-18T22:57:58.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.493+0000 7f9e35b87640 1 --2- 172.21.15.40:0/195442647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30107540 0x7f9e30109990 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.493+0000 7f9e35b87640 1 --2- 172.21.15.40:0/195442647 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e30104c10 0x7f9e30107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.493+0000 7f9e35b87640 1 --2- 172.21.15.40:0/195442647 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e301022e0 0x7f9e301046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.493+0000 7f9e35b87640 1 -- 172.21.15.40:0/195442647 >> 172.21.15.40:0/195442647 conn(0x7f9e300fbcf0 msgr2=0x7f9e300fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.493+0000 7f9e35b87640 1 -- 172.21.15.40:0/195442647 shutdown_connections 2024-03-18T22:57:58.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.493+0000 7f9e35b87640 1 -- 172.21.15.40:0/195442647 wait complete. 2024-03-18T22:57:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.494+0000 7f9e35b87640 1 Processor -- start 2024-03-18T22:57:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.494+0000 7f9e35b87640 1 -- start start 2024-03-18T22:57:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.494+0000 7f9e35b87640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e301022e0 0x7f9e3006b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.494+0000 7f9e35b87640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30104c10 0x7f9e3006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.494+0000 7f9e35b87640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e30107540 0x7f9e30072bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.494+0000 7f9e35b87640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3010b1f0 con 0x7f9e30104c10 2024-03-18T22:57:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.494+0000 7f9e35b87640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3010b070 con 0x7f9e301022e0 2024-03-18T22:57:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.494+0000 7f9e35b87640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3010b370 con 0x7f9e30107540 2024-03-18T22:57:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.495+0000 7f9e2effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30104c10 0x7f9e3006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.495+0000 7f9e2effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30104c10 0x7f9e3006bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45252/0 (socket says 172.21.15.40:45252) 2024-03-18T22:57:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.495+0000 7f9e2effd640 1 -- 172.21.15.40:0/1457305249 learned_addr learned my addr 172.21.15.40:0/1457305249 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.495+0000 7f9e2ffff640 1 --2- 172.21.15.40:0/1457305249 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e30107540 0x7f9e30072bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.495+0000 7f9e2effd640 1 -- 172.21.15.40:0/1457305249 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e30107540 msgr2=0x7f9e30072bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.495+0000 7f9e2effd640 1 --2- 172.21.15.40:0/1457305249 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e30107540 0x7f9e30072bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.495+0000 7f9e2effd640 1 -- 172.21.15.40:0/1457305249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e301022e0 msgr2=0x7f9e3006b620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T22:57:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.495+0000 7f9e2effd640 1 --2- 172.21.15.40:0/1457305249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e301022e0 0x7f9e3006b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.495+0000 7f9e2effd640 1 -- 172.21.15.40:0/1457305249 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e30073240 con 0x7f9e30104c10 2024-03-18T22:57:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.495+0000 7f9e2effd640 1 --2- 172.21.15.40:0/1457305249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30104c10 0x7f9e3006bb60 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f9e2002d560 tx=0x7f9e2002da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.496+0000 7f9e2cff9640 1 -- 172.21.15.40:0/1457305249 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20031860 con 0x7f9e30104c10 2024-03-18T22:57:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.496+0000 7f9e2cff9640 1 -- 172.21.15.40:0/1457305249 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e20039400 con 0x7f9e30104c10 2024-03-18T22:57:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.496+0000 7f9e2cff9640 1 -- 172.21.15.40:0/1457305249 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2002f650 con 0x7f9e30104c10 2024-03-18T22:57:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.496+0000 7f9e35b87640 1 -- 172.21.15.40:0/1457305249 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e30073530 con 0x7f9e30104c10 2024-03-18T22:57:58.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.496+0000 7f9e35b87640 1 -- 172.21.15.40:0/1457305249 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e30195530 con 0x7f9e30104c10 2024-03-18T22:57:58.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.500+0000 7f9e38772740 1 Processor -- start 2024-03-18T22:57:58.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.500+0000 7f9e38772740 1 -- start start 2024-03-18T22:57:58.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.500+0000 7f9e38772740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717aab00 0x55f1717aaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.500+0000 7f9e38772740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717ab4f0 0x55f1717afff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.500+0000 7f9e38772740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717b0660 0x55f1717b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.500+0000 7f9e38772740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f1717064b0 con 0x55f1717ab4f0 2024-03-18T22:57:58.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.500+0000 7f9e38772740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f1716aed70 con 0x55f1717aab00 2024-03-18T22:57:58.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.500+0000 7f9e38772740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f171702550 con 0x55f1717b0660 2024-03-18T22:57:58.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.500+0000 7f9e2cff9640 1 -- 172.21.15.40:0/1457305249 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9e2002f040 con 0x7f9e30104c10 2024-03-18T22:57:58.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.500+0000 7f9e2f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717ab4f0 0x55f1717afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.500+0000 7f9e2f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717ab4f0 0x55f1717afff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45258/0 (socket says 172.21.15.40:45258) 2024-03-18T22:57:58.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.500+0000 7f9e2f7fe640 1 -- 172.21.15.40:0/1880237719 learned_addr learned my addr 172.21.15.40:0/1880237719 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.500+0000 7f9e2effd640 1 --2- 172.21.15.40:0/1880237719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717b0660 0x55f1717b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.500+0000 7f9e2ffff640 1 --2- 172.21.15.40:0/1880237719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717aab00 0x55f1717aaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.501+0000 7f9e2effd640 1 -- 172.21.15.40:0/1880237719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717aab00 msgr2=0x55f1717aaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.501+0000 7f9e2effd640 1 --2- 172.21.15.40:0/1880237719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717aab00 0x55f1717aaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.501+0000 7f9e2effd640 1 -- 172.21.15.40:0/1880237719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717ab4f0 msgr2=0x55f1717afff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.501+0000 7f9e2effd640 1 --2- 172.21.15.40:0/1880237719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717ab4f0 0x55f1717afff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.501+0000 7f9e2effd640 1 -- 172.21.15.40:0/1880237719 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1716d1210 con 0x55f1717b0660 2024-03-18T22:57:58.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.502+0000 7f9e2cff9640 1 --2- 172.21.15.40:0/1457305249 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9e180a29d0 0x7f9e180a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.502+0000 7f9e2cff9640 1 -- 172.21.15.40:0/1457305249 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9e200f0430 con 0x7f9e30104c10 2024-03-18T22:57:58.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.502+0000 7f9e2effd640 1 --2- 172.21.15.40:0/1880237719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717b0660 0x55f1717b2af0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9e200eda80 tx=0x7f9e200f89b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ea2e57d768758eb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.502+0000 7f9e2ffff640 1 --2- 172.21.15.40:0/1457305249 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9e180a29d0 0x7f9e180a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.503+0000 7f9e15ffb640 1 -- 172.21.15.40:0/1880237719 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e200b8c20 con 0x55f1717b0660 2024-03-18T22:57:58.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.503+0000 7f9e15ffb640 1 -- 172.21.15.40:0/1880237719 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e200bd080 con 0x55f1717b0660 2024-03-18T22:57:58.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.503+0000 7f9e15ffb640 1 -- 172.21.15.40:0/1880237719 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e200bc2e0 con 0x55f1717b0660 2024-03-18T22:57:58.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.504+0000 7f9e38772740 1 -- 172.21.15.40:0/1880237719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717b0660 msgr2=0x55f1717b2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.504+0000 7f9e38772740 1 --2- 172.21.15.40:0/1880237719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717b0660 0x55f1717b2af0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9e200eda80 tx=0x7f9e200f89b0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.504+0000 7f9e2ffff640 1 --2- 172.21.15.40:0/1457305249 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9e180a29d0 0x7f9e180a4e90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9e24006010 tx=0x7f9e24004c00 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.504+0000 7f9e38772740 1 -- 172.21.15.40:0/1880237719 shutdown_connections 2024-03-18T22:57:58.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.504+0000 7f9e38772740 1 --2- 172.21.15.40:0/1880237719 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717b0660 0x55f1717b2af0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.504+0000 7f9e38772740 1 --2- 172.21.15.40:0/1880237719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717ab4f0 0x55f1717afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.504+0000 7f9e38772740 1 --2- 172.21.15.40:0/1880237719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717aab00 0x55f1717aaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.504+0000 7f9e38772740 1 -- 172.21.15.40:0/1880237719 >> 172.21.15.40:0/1880237719 conn(0x55f1717a6070 msgr2=0x55f1717a6470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.504+0000 7f9e38772740 1 -- 172.21.15.40:0/1880237719 shutdown_connections 2024-03-18T22:57:58.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.504+0000 7f9e38772740 1 -- 172.21.15.40:0/1880237719 wait complete. 2024-03-18T22:57:58.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.504+0000 7f9e38772740 1 -- start start 2024-03-18T22:57:58.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.504+0000 7f9e38772740 1 Processor -- start 2024-03-18T22:57:58.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e38772740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e38772740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717ab4f0 0x55f1717b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e38772740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717b0660 0x55f1717bc670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e38772740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55f1717b3660 con 0x55f1717ab4f0 2024-03-18T22:57:58.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e2f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e2effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717ab4f0 0x55f1717b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e38772740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f1716c2310 con 0x55f1717b0660 2024-03-18T22:57:58.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e38772740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55f1717b37e0 con 0x55f1717aab00 2024-03-18T22:57:58.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e2f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717b50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57150/0 (socket says 172.21.15.40:57150) 2024-03-18T22:57:58.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e2f7fe640 1 -- 172.21.15.40:0/71264460 learned_addr learned my addr 172.21.15.40:0/71264460 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e2ffff640 1 --2- 172.21.15.40:0/71264460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717b0660 0x55f1717bc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e38772740 10 client.0 _ll_register_callbacks cb 0x55f1716b62d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:58.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e2ffff640 1 -- 172.21.15.40:0/71264460 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 msgr2=0x55f1717b50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e2ffff640 1 --2- 172.21.15.40:0/71264460 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717b50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e2ffff640 1 -- 172.21.15.40:0/71264460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717ab4f0 msgr2=0x55f1717b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e2ffff640 1 --2- 172.21.15.40:0/71264460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717ab4f0 0x55f1717b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e2ffff640 1 -- 172.21.15.40:0/71264460 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f17141a3f0 con 0x55f1717b0660 2024-03-18T22:57:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.505+0000 7f9e2f7fe640 1 --2- 172.21.15.40:0/71264460 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717b50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.506+0000 7f9e38772740 1 -- 172.21.15.40:0/71264460 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e200bd080 con 0x55f1717b0660 2024-03-18T22:57:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.506+0000 7f9e2ffff640 1 --2- 172.21.15.40:0/71264460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717b0660 0x55f1717bc670 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009230 tx=0x7f9e24030040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5341b475305764d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.506+0000 7f9e0effd640 10 client.28472 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:57:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.506+0000 7f9e0effd640 1 -- 172.21.15.40:0/71264460 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24049040 con 0x55f1717b0660 2024-03-18T22:57:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.506+0000 7f9e0effd640 1 -- 172.21.15.40:0/71264460 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9e24038790 con 0x55f1717b0660 2024-03-18T22:57:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.506+0000 7f9e0effd640 1 -- 172.21.15.40:0/71264460 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f9e24038990 con 0x55f1717b0660 2024-03-18T22:57:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.506+0000 7f9e38772740 10 client.28472 fetch_fsmap learned FSMap version 46 2024-03-18T22:57:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.507+0000 7f9e38772740 1 -- 172.21.15.40:0/71264460 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({fsmap=46}) -- 0x55f1714e47d0 con 0x55f1717b0660 2024-03-18T22:57:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.507+0000 7f9e0effd640 1 -- 172.21.15.40:0/71264460 <== mon.2 v2:172.21.15.103:3300/0 4 ==== fsmap(e 46) ==== 8406+0+0 (secure 0 0 0) 0x7f9e24038b30 con 0x55f1717b0660 2024-03-18T22:57:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.507+0000 7f9e38772740 10 client.28472 fetch_fsmap finished waiting for FSMap version 46 2024-03-18T22:57:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.507+0000 7f9e38772740 10 client.28472 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:57:58.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.507+0000 7f9e38772740 20 client.28472 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:58.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.507+0000 7f9e38772740 1 --2- 172.21.15.40:0/71264460 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55f1717c1200 0x55f1717c36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.507+0000 7f9e38772740 4 client.28472 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:57:58.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.508+0000 7f9e38772740 1 -- 172.21.15.40:0/71264460 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1717643e0 con 0x55f1717c1200 2024-03-18T22:57:58.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.509+0000 7f9e2f7fe640 1 --2- 172.21.15.40:0/71264460 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55f1717c1200 0x55f1717c36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.509+0000 7f9e0effd640 10 client.28472 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:57:58.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.509+0000 7f9e2f7fe640 1 --2- 172.21.15.40:0/71264460 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55f1717c1200 0x55f1717c36a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2cd31408f84516e1 server_cookie=e1da029f585fd8c3 in_seq=0 out_seq=0 2024-03-18T22:57:58.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.510+0000 7f9e0effd640 1 -- 172.21.15.40:0/71264460 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f1717643e0 con 0x55f1717c1200 2024-03-18T22:57:58.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.510+0000 7f9e0effd640 10 client.28472 handle_command_reply: tid=0 2024-03-18T22:57:58.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.510+0000 7f9e38772740 1 client.28472 shutdown 2024-03-18T22:57:58.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.510+0000 7f9e38772740 1 -- 172.21.15.40:0/71264460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717b0660 msgr2=0x55f1717bc670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.510+0000 7f9e38772740 1 --2- 172.21.15.40:0/71264460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717b0660 0x55f1717bc670 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009230 tx=0x7f9e24030040 comp rx=0 tx=0).stop 2024-03-18T22:57:58.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.511+0000 7f9e38772740 1 -- 172.21.15.40:0/71264460 shutdown_connections 2024-03-18T22:57:58.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.511+0000 7f9e38772740 1 --2- 172.21.15.40:0/71264460 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55f1717c1200 0x55f1717c36a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.511+0000 7f9e38772740 1 --2- 172.21.15.40:0/71264460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717b0660 0x55f1717bc670 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.511+0000 7f9e38772740 1 --2- 172.21.15.40:0/71264460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717ab4f0 0x55f1717b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.511+0000 7f9e38772740 1 --2- 172.21.15.40:0/71264460 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717b50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.511+0000 7f9e38772740 1 -- 172.21.15.40:0/71264460 >> 172.21.15.40:0/71264460 conn(0x55f1717a6070 msgr2=0x55f17179cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.511+0000 7f9e0effd640 0 client.28472 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:57:58.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.511+0000 7f9e38772740 1 -- 172.21.15.40:0/71264460 shutdown_connections 2024-03-18T22:57:58.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.511+0000 7f9e38772740 1 -- 172.21.15.40:0/71264460 wait complete. 2024-03-18T22:57:58.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.511+0000 7f9e38772740 20 client.28472 trim_cache size 0 max 16384 2024-03-18T22:57:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.530+0000 7f9e38772740 1 Processor -- start 2024-03-18T22:57:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.530+0000 7f9e38772740 1 -- start start 2024-03-18T22:57:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.530+0000 7f9e38772740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.530+0000 7f9e38772740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717ab4f0 0x55f1717b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.530+0000 7f9e38772740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717b0660 0x55f1717bc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.530+0000 7f9e38772740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3019fca0 con 0x55f1717b0660 2024-03-18T22:57:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.530+0000 7f9e38772740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e301a1990 con 0x55f1717ab4f0 2024-03-18T22:57:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.530+0000 7f9e38772740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9e301a0370 con 0x55f1717aab00 2024-03-18T22:57:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.531+0000 7f9e2ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717b0660 0x55f1717bc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.531+0000 7f9e2f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.531+0000 7f9e2f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717b50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57162/0 (socket says 172.21.15.40:57162) 2024-03-18T22:57:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.531+0000 7f9e2f7fe640 1 -- 172.21.15.40:0/3609663493 learned_addr learned my addr 172.21.15.40:0/3609663493 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.531+0000 7f9e2effd640 1 --2- 172.21.15.40:0/3609663493 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717ab4f0 0x55f1717b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.531+0000 7f9e2ffff640 1 -- 172.21.15.40:0/3609663493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 msgr2=0x55f1717b50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.531+0000 7f9e2ffff640 1 --2- 172.21.15.40:0/3609663493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717b50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.531+0000 7f9e2ffff640 1 -- 172.21.15.40:0/3609663493 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717ab4f0 msgr2=0x55f1717b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.531+0000 7f9e2ffff640 1 --2- 172.21.15.40:0/3609663493 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717ab4f0 0x55f1717b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.531+0000 7f9e2ffff640 1 -- 172.21.15.40:0/3609663493 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f171773880 con 0x55f1717b0660 2024-03-18T22:57:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.531+0000 7f9e2f7fe640 1 --2- 172.21.15.40:0/3609663493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717b50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:57:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.531+0000 7f9e2ffff640 1 --2- 172.21.15.40:0/3609663493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717b0660 0x55f1717bc590 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f9e24030580 tx=0x7f9e24045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500727d1cb735164 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.532+0000 7f9e157fa640 1 -- 172.21.15.40:0/3609663493 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24049040 con 0x55f1717b0660 2024-03-18T22:57:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.532+0000 7f9e157fa640 1 -- 172.21.15.40:0/3609663493 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e200bd080 con 0x55f1717b0660 2024-03-18T22:57:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.532+0000 7f9e157fa640 1 -- 172.21.15.40:0/3609663493 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24003b60 con 0x55f1717b0660 2024-03-18T22:57:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.533+0000 7f9e38772740 1 -- 172.21.15.40:0/3609663493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717b0660 msgr2=0x55f1717bc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.533+0000 7f9e38772740 1 --2- 172.21.15.40:0/3609663493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717b0660 0x55f1717bc590 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f9e24030580 tx=0x7f9e24045000 comp rx=0 tx=0).stop 2024-03-18T22:57:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.533+0000 7f9e38772740 1 -- 172.21.15.40:0/3609663493 shutdown_connections 2024-03-18T22:57:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.533+0000 7f9e38772740 1 --2- 172.21.15.40:0/3609663493 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717b0660 0x55f1717bc590 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.533+0000 7f9e38772740 1 --2- 172.21.15.40:0/3609663493 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717ab4f0 0x55f1717b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.533+0000 7f9e38772740 1 --2- 172.21.15.40:0/3609663493 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717b50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.533+0000 7f9e38772740 1 -- 172.21.15.40:0/3609663493 >> 172.21.15.40:0/3609663493 conn(0x55f1717a6070 msgr2=0x55f1717abce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.533+0000 7f9e38772740 1 -- 172.21.15.40:0/3609663493 shutdown_connections 2024-03-18T22:57:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.533+0000 7f9e38772740 1 -- 172.21.15.40:0/3609663493 wait complete. 2024-03-18T22:57:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.533+0000 7f9e38772740 1 -- start start 2024-03-18T22:57:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.534+0000 7f9e38772740 1 Processor -- start 2024-03-18T22:57:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.534+0000 7f9e38772740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717bc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.534+0000 7f9e38772740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717ab4f0 0x55f1717b8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.535+0000 7f9e2f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717bc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.535+0000 7f9e2f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717bc2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:57176/0 (socket says 172.21.15.40:57176) 2024-03-18T22:57:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.535+0000 7f9e2effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717ab4f0 0x55f1717b8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.535+0000 7f9e2effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717ab4f0 0x55f1717b8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:45138/0 (socket says 172.21.15.40:45138) 2024-03-18T22:57:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.535+0000 7f9e2f7fe640 1 -- 172.21.15.40:0/2384312726 learned_addr learned my addr 172.21.15.40:0/2384312726 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.535+0000 7f9e38772740 1 --2- 172.21.15.40:0/2384312726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717b0660 0x55f1717b92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.535+0000 7f9e38772740 1 -- 172.21.15.40:0/2384312726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9e301a1200 con 0x55f1717b0660 2024-03-18T22:57:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.535+0000 7f9e2ffff640 1 --2- 172.21.15.40:0/2384312726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717b0660 0x55f1717b92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.535+0000 7f9e38772740 1 -- 172.21.15.40:0/2384312726 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e301a0b00 con 0x55f1717ab4f0 2024-03-18T22:57:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.535+0000 7f9e38772740 1 -- 172.21.15.40:0/2384312726 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3019dc50 con 0x55f1717aab00 2024-03-18T22:57:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.536+0000 7f9e38772740 10 client.0 _ll_register_callbacks cb 0x7f9e24046040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.536+0000 7f9e2f7fe640 1 -- 172.21.15.40:0/2384312726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717ab4f0 msgr2=0x55f1717b8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.536+0000 7f9e2f7fe640 1 --2- 172.21.15.40:0/2384312726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717ab4f0 0x55f1717b8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.536+0000 7f9e2f7fe640 1 -- 172.21.15.40:0/2384312726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717b0660 msgr2=0x55f1717b92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.536+0000 7f9e2f7fe640 1 --2- 172.21.15.40:0/2384312726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717b0660 0x55f1717b92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.536+0000 7f9e2f7fe640 1 -- 172.21.15.40:0/2384312726 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1717bc910 con 0x55f1717aab00 2024-03-18T22:57:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.536+0000 7f9e2f7fe640 1 --2- 172.21.15.40:0/2384312726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717bc2e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9e28010570 tx=0x7f9e28012db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5468d4c60b67948 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.536+0000 7f9e38772740 1 -- 172.21.15.40:0/2384312726 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e200bd080 con 0x55f1717aab00 2024-03-18T22:57:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.537+0000 7f9e0effd640 10 client.28465 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:57:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.537+0000 7f9e0effd640 1 -- 172.21.15.40:0/2384312726 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2801c3f0 con 0x55f1717aab00 2024-03-18T22:57:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.537+0000 7f9e0effd640 1 -- 172.21.15.40:0/2384312726 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9e2801c590 con 0x55f1717aab00 2024-03-18T22:57:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.537+0000 7f9e0effd640 1 -- 172.21.15.40:0/2384312726 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f9e2802d050 con 0x55f1717aab00 2024-03-18T22:57:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.537+0000 7f9e38772740 10 client.28465 fetch_fsmap learned FSMap version 46 2024-03-18T22:57:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.537+0000 7f9e38772740 1 -- 172.21.15.40:0/2384312726 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({fsmap=46}) -- 0x55f171826490 con 0x55f1717aab00 2024-03-18T22:57:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.537+0000 7f9e0effd640 1 -- 172.21.15.40:0/2384312726 <== mon.1 v2:172.21.15.113:3300/0 4 ==== fsmap(e 46) ==== 8406+0+0 (secure 0 0 0) 0x7f9e28003970 con 0x55f1717aab00 2024-03-18T22:57:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.537+0000 7f9e38772740 10 client.28465 fetch_fsmap finished waiting for FSMap version 46 2024-03-18T22:57:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.537+0000 7f9e38772740 10 client.28465 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:57:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.537+0000 7f9e38772740 20 client.28465 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.537+0000 7f9e38772740 1 --2- 172.21.15.40:0/2384312726 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55f171826c20 0x55f171829040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.538+0000 7f9e38772740 4 client.28465 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:57:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.538+0000 7f9e38772740 1 -- 172.21.15.40:0/2384312726 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f9e200bd080 con 0x55f171826c20 2024-03-18T22:57:58.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.541+0000 7f9e2effd640 1 --2- 172.21.15.40:0/2384312726 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55f171826c20 0x55f171829040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.541+0000 7f9e0effd640 10 client.28465 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:57:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.541+0000 7f9e2effd640 1 --2- 172.21.15.40:0/2384312726 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55f171826c20 0x55f171829040 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e36cb6435eb4b3c8 server_cookie=a9d1dc137a4a5385 in_seq=0 out_seq=0 2024-03-18T22:57:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.542+0000 7f9e0effd640 1 -- 172.21.15.40:0/2384312726 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f9e200bd080 con 0x55f171826c20 2024-03-18T22:57:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.542+0000 7f9e0effd640 10 client.28465 handle_command_reply: tid=0 2024-03-18T22:57:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.542+0000 7f9e38772740 1 client.28465 shutdown 2024-03-18T22:57:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.542+0000 7f9e38772740 1 -- 172.21.15.40:0/2384312726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 msgr2=0x55f1717bc2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.542+0000 7f9e38772740 1 --2- 172.21.15.40:0/2384312726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717bc2e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9e28010570 tx=0x7f9e28012db0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.542+0000 7f9e38772740 1 -- 172.21.15.40:0/2384312726 shutdown_connections 2024-03-18T22:57:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.542+0000 7f9e38772740 1 --2- 172.21.15.40:0/2384312726 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55f171826c20 0x55f171829040 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.542+0000 7f9e38772740 1 --2- 172.21.15.40:0/2384312726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55f1717b0660 0x55f1717b92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.542+0000 7f9e0effd640 0 client.28465 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:57:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.542+0000 7f9e38772740 1 --2- 172.21.15.40:0/2384312726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f1717ab4f0 0x55f1717b8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.542+0000 7f9e38772740 1 --2- 172.21.15.40:0/2384312726 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55f1717aab00 0x55f1717bc2e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.542+0000 7f9e38772740 1 -- 172.21.15.40:0/2384312726 >> 172.21.15.40:0/2384312726 conn(0x55f1717a6070 msgr2=0x55f1717b50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.542+0000 7f9e38772740 1 -- 172.21.15.40:0/2384312726 shutdown_connections 2024-03-18T22:57:58.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.542+0000 7f9e38772740 1 -- 172.21.15.40:0/2384312726 wait complete. 2024-03-18T22:57:58.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.542+0000 7f9e38772740 20 client.28465 trim_cache size 0 max 16384 2024-03-18T22:57:58.545 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:57:58.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.543+0000 7f9e35b87640 1 -- 172.21.15.40:0/1457305249 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9e180a29d0 msgr2=0x7f9e180a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.543+0000 7f9e35b87640 1 --2- 172.21.15.40:0/1457305249 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9e180a29d0 0x7f9e180a4e90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9e24006010 tx=0x7f9e24004c00 comp rx=0 tx=0).stop 2024-03-18T22:57:58.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.543+0000 7f9e35b87640 1 -- 172.21.15.40:0/1457305249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30104c10 msgr2=0x7f9e3006bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.543+0000 7f9e35b87640 1 --2- 172.21.15.40:0/1457305249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30104c10 0x7f9e3006bb60 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f9e2002d560 tx=0x7f9e2002da20 comp rx=0 tx=0).stop 2024-03-18T22:57:58.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.544+0000 7f9e35b87640 1 -- 172.21.15.40:0/1457305249 shutdown_connections 2024-03-18T22:57:58.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.544+0000 7f9e35b87640 1 --2- 172.21.15.40:0/1457305249 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9e180a29d0 0x7f9e180a4e90 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.544+0000 7f9e35b87640 1 --2- 172.21.15.40:0/1457305249 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9e30107540 0x7f9e30072bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.544+0000 7f9e35b87640 1 --2- 172.21.15.40:0/1457305249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e30104c10 0x7f9e3006bb60 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.544+0000 7f9e35b87640 1 --2- 172.21.15.40:0/1457305249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e301022e0 0x7f9e3006b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.544+0000 7f9e35b87640 1 -- 172.21.15.40:0/1457305249 >> 172.21.15.40:0/1457305249 conn(0x7f9e300fbcf0 msgr2=0x7f9e30102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.544+0000 7f9e35b87640 1 -- 172.21.15.40:0/1457305249 shutdown_connections 2024-03-18T22:57:58.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.544+0000 7f9e35b87640 1 -- 172.21.15.40:0/1457305249 wait complete. 2024-03-18T22:57:58.560 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:57:58.596 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:58 smithi040 ceph-mon[62643]: pgmap v810: 129 pgs: 129 active+clean; 7.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 66 KiB/s wr, 119 op/s 2024-03-18T22:57:58.596 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:58 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:58.596 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:58 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3932940534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:57:58.632 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:58 smithi113 ceph-mon[66775]: pgmap v810: 129 pgs: 129 active+clean; 7.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 66 KiB/s wr, 119 op/s 2024-03-18T22:57:58.632 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:58 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:58.632 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:58 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3932940534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:57:58.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.696+0000 7f17e8fb9640 1 -- 172.21.15.40:0/1161122348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4101950 msgr2=0x7f17e4101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.696+0000 7f17e8fb9640 1 --2- 172.21.15.40:0/1161122348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4101950 0x7f17e4101dd0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009900 tx=0x7f17d802bc30 comp rx=0 tx=0).stop 2024-03-18T22:57:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.696+0000 7f17e8fb9640 1 -- 172.21.15.40:0/1161122348 shutdown_connections 2024-03-18T22:57:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.696+0000 7f17e8fb9640 1 --2- 172.21.15.40:0/1161122348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17e4102310 0x7f17e4108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.696+0000 7f17e8fb9640 1 --2- 172.21.15.40:0/1161122348 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4101950 0x7f17e4101dd0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.696+0000 7f17e8fb9640 1 --2- 172.21.15.40:0/1161122348 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17e4100760 0x7f17e4100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.696+0000 7f17e8fb9640 1 -- 172.21.15.40:0/1161122348 >> 172.21.15.40:0/1161122348 conn(0x7f17e40fbcd0 msgr2=0x7f17e40fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.696+0000 7f17e8fb9640 1 -- 172.21.15.40:0/1161122348 shutdown_connections 2024-03-18T22:57:58.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.696+0000 7f17e8fb9640 1 -- 172.21.15.40:0/1161122348 wait complete. 2024-03-18T22:57:58.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e8fb9640 1 Processor -- start 2024-03-18T22:57:58.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e8fb9640 1 -- start start 2024-03-18T22:57:58.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e8fb9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17e4100760 0x7f17e406aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e8fb9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4101950 0x7f17e419c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e8fb9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17e4102310 0x7f17e419edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e8fb9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f17e410c270 con 0x7f17e4101950 2024-03-18T22:57:58.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e8fb9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f17e410c0f0 con 0x7f17e4102310 2024-03-18T22:57:58.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e8fb9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f17e410c3f0 con 0x7f17e4100760 2024-03-18T22:57:58.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4101950 0x7f17e419c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4101950 0x7f17e419c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57186/0 (socket says 172.21.15.40:57186) 2024-03-18T22:57:58.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e1d9b640 1 -- 172.21.15.40:0/767044344 learned_addr learned my addr 172.21.15.40:0/767044344 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e259c640 1 --2- 172.21.15.40:0/767044344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17e4100760 0x7f17e406aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e2d9d640 1 --2- 172.21.15.40:0/767044344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17e4102310 0x7f17e419edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e1d9b640 1 -- 172.21.15.40:0/767044344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17e4100760 msgr2=0x7f17e406aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e1d9b640 1 --2- 172.21.15.40:0/767044344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17e4100760 0x7f17e406aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e1d9b640 1 -- 172.21.15.40:0/767044344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17e4102310 msgr2=0x7f17e419edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e1d9b640 1 --2- 172.21.15.40:0/767044344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17e4102310 0x7f17e419edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e1d9b640 1 -- 172.21.15.40:0/767044344 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e419f690 con 0x7f17e4101950 2024-03-18T22:57:58.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.697+0000 7f17e1d9b640 1 --2- 172.21.15.40:0/767044344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4101950 0x7f17e419c790 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f17d8006010 tx=0x7f17d8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.698+0000 7f17d37fe640 1 -- 172.21.15.40:0/767044344 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8043020 con 0x7f17e4101950 2024-03-18T22:57:58.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.698+0000 7f17d37fe640 1 -- 172.21.15.40:0/767044344 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17d8004440 con 0x7f17e4101950 2024-03-18T22:57:58.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.698+0000 7f17e8fb9640 1 -- 172.21.15.40:0/767044344 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17e419f920 con 0x7f17e4101950 2024-03-18T22:57:58.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.698+0000 7f17d37fe640 1 -- 172.21.15.40:0/767044344 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8003bf0 con 0x7f17e4101950 2024-03-18T22:57:58.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.698+0000 7f17e8fb9640 1 -- 172.21.15.40:0/767044344 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17e419fec0 con 0x7f17e4101950 2024-03-18T22:57:58.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.701+0000 7f17ebba3740 1 Processor -- start 2024-03-18T22:57:58.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.701+0000 7f17d37fe640 1 -- 172.21.15.40:0/767044344 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f17d8046020 con 0x7f17e4101950 2024-03-18T22:57:58.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.701+0000 7f17ebba3740 1 -- start start 2024-03-18T22:57:58.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.701+0000 7f17ebba3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b0700 0x55977f5b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.701+0000 7f17ebba3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.701+0000 7f17ebba3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b6260 0x55977f5b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.701+0000 7f17ebba3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55977f5104b0 con 0x55977f5b10f0 2024-03-18T22:57:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.701+0000 7f17ebba3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55977f4b8d70 con 0x55977f5b0700 2024-03-18T22:57:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.701+0000 7f17ebba3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55977f50c550 con 0x55977f5b6260 2024-03-18T22:57:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.701+0000 7f17e259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.701+0000 7f17e259c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5b5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57192/0 (socket says 172.21.15.40:57192) 2024-03-18T22:57:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.701+0000 7f17e259c640 1 -- 172.21.15.40:0/2894033052 learned_addr learned my addr 172.21.15.40:0/2894033052 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.701+0000 7f17e2d9d640 1 --2- 172.21.15.40:0/2894033052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b0700 0x55977f5b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.702+0000 7f17e1d9b640 1 --2- 172.21.15.40:0/2894033052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b6260 0x55977f5b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.702+0000 7f17d37fe640 1 --2- 172.21.15.40:0/767044344 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17cc0a2880 0x7f17cc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.702+0000 7f17e259c640 1 -- 172.21.15.40:0/2894033052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b6260 msgr2=0x55977f5b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.702+0000 7f17e259c640 1 --2- 172.21.15.40:0/2894033052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b6260 0x55977f5b86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.702+0000 7f17e259c640 1 -- 172.21.15.40:0/2894033052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b0700 msgr2=0x55977f5b0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.702+0000 7f17e259c640 1 --2- 172.21.15.40:0/2894033052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b0700 0x55977f5b0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.702+0000 7f17e259c640 1 -- 172.21.15.40:0/2894033052 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55977f4db210 con 0x55977f5b10f0 2024-03-18T22:57:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.702+0000 7f17e2d9d640 1 --2- 172.21.15.40:0/2894033052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b0700 0x55977f5b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:57:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.702+0000 7f17e2d9d640 1 --2- 172.21.15.40:0/767044344 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17cc0a2880 0x7f17cc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.702+0000 7f17d37fe640 1 -- 172.21.15.40:0/767044344 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f17d80f10d0 con 0x7f17e4101950 2024-03-18T22:57:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.702+0000 7f17e259c640 1 --2- 172.21.15.40:0/2894033052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5b5bf0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f17d400b7d0 tx=0x7f17d4004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46a5c1e93e50291 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.702+0000 7f17d0ff9640 1 -- 172.21.15.40:0/2894033052 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d4003dd0 con 0x55977f5b10f0 2024-03-18T22:57:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.702+0000 7f17d0ff9640 1 -- 172.21.15.40:0/2894033052 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17d4015070 con 0x55977f5b10f0 2024-03-18T22:57:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.702+0000 7f17d0ff9640 1 -- 172.21.15.40:0/2894033052 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d40116d0 con 0x55977f5b10f0 2024-03-18T22:57:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.703+0000 7f17ebba3740 1 -- 172.21.15.40:0/2894033052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 msgr2=0x55977f5b5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.703+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2894033052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5b5bf0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f17d400b7d0 tx=0x7f17d4004610 comp rx=0 tx=0).stop 2024-03-18T22:57:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.703+0000 7f17e2d9d640 1 --2- 172.21.15.40:0/767044344 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17cc0a2880 0x7f17cc0a4d40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x55977f5b1950 tx=0x7f17dc014040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.703+0000 7f17ebba3740 1 -- 172.21.15.40:0/2894033052 shutdown_connections 2024-03-18T22:57:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.703+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2894033052 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b6260 0x55977f5b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.703+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2894033052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5b5bf0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.703+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2894033052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b0700 0x55977f5b0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.703+0000 7f17ebba3740 1 -- 172.21.15.40:0/2894033052 >> 172.21.15.40:0/2894033052 conn(0x55977f5a7860 msgr2=0x55977f5a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.703+0000 7f17ebba3740 1 -- 172.21.15.40:0/2894033052 shutdown_connections 2024-03-18T22:57:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.703+0000 7f17ebba3740 1 -- 172.21.15.40:0/2894033052 wait complete. 2024-03-18T22:57:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.703+0000 7f17ebba3740 1 -- start start 2024-03-18T22:57:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.703+0000 7f17ebba3740 1 Processor -- start 2024-03-18T22:57:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17ebba3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b0700 0x55977f5bac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17ebba3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b10f0 0x55977f5bb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17ebba3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b6260 0x55977f5c2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17ebba3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55977f5b9210 con 0x55977f5b6260 2024-03-18T22:57:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17ebba3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55977f4cc310 con 0x55977f5b0700 2024-03-18T22:57:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17ebba3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55977f5b9390 con 0x55977f5b10f0 2024-03-18T22:57:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17ebba3740 10 client.0 _ll_register_callbacks cb 0x55977f4c02d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17e2d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b6260 0x55977f5c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17e2d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b6260 0x55977f5c2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57194/0 (socket says 172.21.15.40:57194) 2024-03-18T22:57:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17e2d9d640 1 -- 172.21.15.40:0/2929549969 learned_addr learned my addr 172.21.15.40:0/2929549969 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17e259c640 1 --2- 172.21.15.40:0/2929549969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b0700 0x55977f5bac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17e1d9b640 1 --2- 172.21.15.40:0/2929549969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b10f0 0x55977f5bb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17e2d9d640 1 -- 172.21.15.40:0/2929549969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b10f0 msgr2=0x55977f5bb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17e2d9d640 1 --2- 172.21.15.40:0/2929549969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b10f0 0x55977f5bb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17e2d9d640 1 -- 172.21.15.40:0/2929549969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b0700 msgr2=0x55977f5bac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17e2d9d640 1 --2- 172.21.15.40:0/2929549969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b0700 0x55977f5bac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17e2d9d640 1 -- 172.21.15.40:0/2929549969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55977f2243f0 con 0x55977f5b6260 2024-03-18T22:57:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17e2d9d640 1 --2- 172.21.15.40:0/2929549969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b6260 0x55977f5c2190 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0094d0 tx=0x7f17dc009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a273f1504a04723 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:57:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17c9ffb640 1 -- 172.21.15.40:0/2929549969 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17dc01c950 con 0x55977f5b6260 2024-03-18T22:57:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17c9ffb640 1 -- 172.21.15.40:0/2929549969 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f17dc01caf0 con 0x55977f5b6260 2024-03-18T22:57:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17ebba3740 1 -- 172.21.15.40:0/2929549969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17d4015070 con 0x55977f5b6260 2024-03-18T22:57:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17c9ffb640 1 -- 172.21.15.40:0/2929549969 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f17dc01ccf0 con 0x55977f5b6260 2024-03-18T22:57:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.704+0000 7f17ebba3740 10 client.18666 fetch_fsmap learned FSMap version 46 2024-03-18T22:57:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.705+0000 7f17ebba3740 1 -- 172.21.15.40:0/2929549969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55977f2ee7d0 con 0x55977f5b6260 2024-03-18T22:57:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.705+0000 7f17c9ffb640 1 -- 172.21.15.40:0/2929549969 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17d4015070 con 0x55977f5b6260 2024-03-18T22:57:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.705+0000 7f17c9ffb640 1 -- 172.21.15.40:0/2929549969 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 46) ==== 8406+0+0 (secure 0 0 0) 0x7f17dc025020 con 0x55977f5b6260 2024-03-18T22:57:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.705+0000 7f17ebba3740 10 client.18666 fetch_fsmap finished waiting for FSMap version 46 2024-03-18T22:57:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.705+0000 7f17ebba3740 10 client.18666 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:57:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.705+0000 7f17ebba3740 20 client.18666 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.705+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2929549969 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55977f5c6b30 0x55977f5c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.705+0000 7f17ebba3740 4 client.18666 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:57:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.705+0000 7f17ebba3740 1 -- 172.21.15.40:0/2929549969 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55977f56e3e0 con 0x55977f5c6b30 2024-03-18T22:57:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.705+0000 7f17e259c640 1 --2- 172.21.15.40:0/2929549969 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55977f5c6b30 0x55977f5c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.706+0000 7f17c9ffb640 10 client.18666 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:57:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.706+0000 7f17e259c640 1 --2- 172.21.15.40:0/2929549969 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55977f5c6b30 0x55977f5c8f90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bb1b6afb9daeadf6 server_cookie=accac367349cef34 in_seq=0 out_seq=0 2024-03-18T22:57:58.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.707+0000 7f17c9ffb640 1 -- 172.21.15.40:0/2929549969 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55977f56e3e0 con 0x55977f5c6b30 2024-03-18T22:57:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.707+0000 7f17c9ffb640 10 client.18666 handle_command_reply: tid=0 2024-03-18T22:57:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.707+0000 7f17ebba3740 1 client.18666 shutdown 2024-03-18T22:57:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.707+0000 7f17ebba3740 1 -- 172.21.15.40:0/2929549969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b6260 msgr2=0x55977f5c2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.707+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2929549969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b6260 0x55977f5c2190 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0094d0 tx=0x7f17dc009500 comp rx=0 tx=0).stop 2024-03-18T22:57:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.707+0000 7f17ebba3740 1 -- 172.21.15.40:0/2929549969 shutdown_connections 2024-03-18T22:57:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.707+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2929549969 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55977f5c6b30 0x55977f5c8f90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.707+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2929549969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b6260 0x55977f5c2190 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.707+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2929549969 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b10f0 0x55977f5bb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.707+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2929549969 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b0700 0x55977f5bac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.707+0000 7f17c9ffb640 0 client.18666 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:57:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.707+0000 7f17ebba3740 1 -- 172.21.15.40:0/2929549969 >> 172.21.15.40:0/2929549969 conn(0x55977f5a7860 msgr2=0x55977f5a6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.707+0000 7f17ebba3740 1 -- 172.21.15.40:0/2929549969 shutdown_connections 2024-03-18T22:57:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.707+0000 7f17ebba3740 1 -- 172.21.15.40:0/2929549969 wait complete. 2024-03-18T22:57:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.707+0000 7f17ebba3740 20 client.18666 trim_cache size 0 max 16384 2024-03-18T22:57:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.720+0000 7f17ebba3740 1 Processor -- start 2024-03-18T22:57:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.720+0000 7f17ebba3740 1 -- start start 2024-03-18T22:57:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.720+0000 7f17ebba3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b0700 0x55977f5b9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.720+0000 7f17ebba3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.720+0000 7f17ebba3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b6260 0x55977f5b9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.720+0000 7f17ebba3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f17e41a8320 con 0x55977f5b10f0 2024-03-18T22:57:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.720+0000 7f17ebba3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f17e41aa010 con 0x55977f5b6260 2024-03-18T22:57:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.720+0000 7f17ebba3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f17e41a89f0 con 0x55977f5b0700 2024-03-18T22:57:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17e1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17e1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5c2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57196/0 (socket says 172.21.15.40:57196) 2024-03-18T22:57:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17e1d9b640 1 -- 172.21.15.40:0/1506425335 learned_addr learned my addr 172.21.15.40:0/1506425335 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17e2d9d640 1 --2- 172.21.15.40:0/1506425335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b6260 0x55977f5b9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17e259c640 1 --2- 172.21.15.40:0/1506425335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b0700 0x55977f5b9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17e1d9b640 1 -- 172.21.15.40:0/1506425335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b0700 msgr2=0x55977f5b9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17e1d9b640 1 --2- 172.21.15.40:0/1506425335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b0700 0x55977f5b9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17e1d9b640 1 -- 172.21.15.40:0/1506425335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b6260 msgr2=0x55977f5b9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17e1d9b640 1 --2- 172.21.15.40:0/1506425335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b6260 0x55977f5b9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17e1d9b640 1 -- 172.21.15.40:0/1506425335 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55977f57d880 con 0x55977f5b10f0 2024-03-18T22:57:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17e259c640 1 --2- 172.21.15.40:0/1506425335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b0700 0x55977f5b9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:57:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17e2d9d640 1 --2- 172.21.15.40:0/1506425335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b6260 0x55977f5b9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:57:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17e1d9b640 1 --2- 172.21.15.40:0/1506425335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5c2150 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f17d80ba050 tx=0x7f17d80f48e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a2afc4f6fc6b04 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17cbfff640 1 -- 172.21.15.40:0/1506425335 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d80bd450 con 0x55977f5b10f0 2024-03-18T22:57:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17cbfff640 1 -- 172.21.15.40:0/1506425335 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17d80dd040 con 0x55977f5b10f0 2024-03-18T22:57:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.721+0000 7f17cbfff640 1 -- 172.21.15.40:0/1506425335 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d80bcfe0 con 0x55977f5b10f0 2024-03-18T22:57:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 -- 172.21.15.40:0/1506425335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 msgr2=0x55977f5c2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 --2- 172.21.15.40:0/1506425335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5c2150 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f17d80ba050 tx=0x7f17d80f48e0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 -- 172.21.15.40:0/1506425335 shutdown_connections 2024-03-18T22:57:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 --2- 172.21.15.40:0/1506425335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b6260 0x55977f5b9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 --2- 172.21.15.40:0/1506425335 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5c2150 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 --2- 172.21.15.40:0/1506425335 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b0700 0x55977f5b9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 -- 172.21.15.40:0/1506425335 >> 172.21.15.40:0/1506425335 conn(0x55977f5a7860 msgr2=0x55977f5ab350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 -- 172.21.15.40:0/1506425335 shutdown_connections 2024-03-18T22:57:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 -- 172.21.15.40:0/1506425335 wait complete. 2024-03-18T22:57:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 -- start start 2024-03-18T22:57:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 Processor -- start 2024-03-18T22:57:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b0700 0x55977f5c0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5c0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b6260 0x55977f5c16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f17e41a9880 con 0x55977f5b10f0 2024-03-18T22:57:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f17e41a9180 con 0x55977f5b6260 2024-03-18T22:57:58.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f17e41a62d0 con 0x55977f5b0700 2024-03-18T22:57:58.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17ebba3740 10 client.0 _ll_register_callbacks cb 0x7f17dc009db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:58.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17e1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17e1d9b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57212/0 (socket says 172.21.15.40:57212) 2024-03-18T22:57:58.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17e1d9b640 1 -- 172.21.15.40:0/2323156256 learned_addr learned my addr 172.21.15.40:0/2323156256 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17e259c640 1 --2- 172.21.15.40:0/2323156256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b0700 0x55977f5c0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.722+0000 7f17e2d9d640 1 --2- 172.21.15.40:0/2323156256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b6260 0x55977f5c16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17e1d9b640 1 -- 172.21.15.40:0/2323156256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b0700 msgr2=0x55977f5c0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17e1d9b640 1 --2- 172.21.15.40:0/2323156256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b0700 0x55977f5c0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17e1d9b640 1 -- 172.21.15.40:0/2323156256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b6260 msgr2=0x55977f5c16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17e1d9b640 1 --2- 172.21.15.40:0/2323156256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b6260 0x55977f5c16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17e1d9b640 1 -- 172.21.15.40:0/2323156256 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55977f5c1c90 con 0x55977f5b10f0 2024-03-18T22:57:58.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17e1d9b640 1 --2- 172.21.15.40:0/2323156256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5c0690 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f17d80bd380 tx=0x7f17d80bd110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed015aae33f32e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:57:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17c9ffb640 1 -- 172.21.15.40:0/2323156256 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d80bd1c0 con 0x55977f5b10f0 2024-03-18T22:57:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17c9ffb640 1 -- 172.21.15.40:0/2323156256 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f17d80d62b0 con 0x55977f5b10f0 2024-03-18T22:57:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17ebba3740 1 -- 172.21.15.40:0/2323156256 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17d80dd040 con 0x55977f5b10f0 2024-03-18T22:57:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17c9ffb640 1 -- 172.21.15.40:0/2323156256 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f17d80a80b0 con 0x55977f5b10f0 2024-03-18T22:57:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17ebba3740 10 client.18672 fetch_fsmap learned FSMap version 46 2024-03-18T22:57:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17ebba3740 1 -- 172.21.15.40:0/2323156256 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55977f62d760 con 0x55977f5b10f0 2024-03-18T22:57:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17c9ffb640 1 -- 172.21.15.40:0/2323156256 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17d80dd040 con 0x55977f5b10f0 2024-03-18T22:57:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17c9ffb640 1 -- 172.21.15.40:0/2323156256 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 46) ==== 8406+0+0 (secure 0 0 0) 0x7f17d80bcad0 con 0x55977f5b10f0 2024-03-18T22:57:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17ebba3740 10 client.18672 fetch_fsmap finished waiting for FSMap version 46 2024-03-18T22:57:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17ebba3740 10 client.18672 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:57:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17ebba3740 20 client.18672 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2323156256 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55977f62e1d0 0x55977f6305f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17ebba3740 4 client.18672 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:57:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.723+0000 7f17ebba3740 1 -- 172.21.15.40:0/2323156256 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55977f51d9f0 con 0x55977f62e1d0 2024-03-18T22:57:58.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17e259c640 1 --2- 172.21.15.40:0/2323156256 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55977f62e1d0 0x55977f6305f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17c9ffb640 10 client.18672 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:57:58.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17e259c640 1 --2- 172.21.15.40:0/2323156256 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55977f62e1d0 0x55977f6305f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6fce6de273e865e7 server_cookie=60328e7cad889068 in_seq=0 out_seq=0 2024-03-18T22:57:58.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17c9ffb640 1 -- 172.21.15.40:0/2323156256 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55977f51d9f0 con 0x55977f62e1d0 2024-03-18T22:57:58.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17c9ffb640 10 client.18672 handle_command_reply: tid=0 2024-03-18T22:57:58.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17ebba3740 1 client.18672 shutdown 2024-03-18T22:57:58.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17ebba3740 1 -- 172.21.15.40:0/2323156256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 msgr2=0x55977f5c0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2323156256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5c0690 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f17d80bd380 tx=0x7f17d80bd110 comp rx=0 tx=0).stop 2024-03-18T22:57:58.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17ebba3740 1 -- 172.21.15.40:0/2323156256 shutdown_connections 2024-03-18T22:57:58.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2323156256 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55977f62e1d0 0x55977f6305f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2323156256 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55977f5b6260 0x55977f5c16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2323156256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55977f5b10f0 0x55977f5c0690 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17ebba3740 1 --2- 172.21.15.40:0/2323156256 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55977f5b0700 0x55977f5c0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17c9ffb640 0 client.18672 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:57:58.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17ebba3740 1 -- 172.21.15.40:0/2323156256 >> 172.21.15.40:0/2323156256 conn(0x55977f5a7860 msgr2=0x55977f5a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17ebba3740 1 -- 172.21.15.40:0/2323156256 shutdown_connections 2024-03-18T22:57:58.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17ebba3740 1 -- 172.21.15.40:0/2323156256 wait complete. 2024-03-18T22:57:58.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.724+0000 7f17ebba3740 20 client.18672 trim_cache size 0 max 16384 2024-03-18T22:57:58.733 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:57:58.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.725+0000 7f17e8fb9640 1 -- 172.21.15.40:0/767044344 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17cc0a2880 msgr2=0x7f17cc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.725+0000 7f17e8fb9640 1 --2- 172.21.15.40:0/767044344 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17cc0a2880 0x7f17cc0a4d40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x55977f5b1950 tx=0x7f17dc014040 comp rx=0 tx=0).stop 2024-03-18T22:57:58.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.725+0000 7f17e8fb9640 1 -- 172.21.15.40:0/767044344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4101950 msgr2=0x7f17e419c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.725+0000 7f17e8fb9640 1 --2- 172.21.15.40:0/767044344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4101950 0x7f17e419c790 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f17d8006010 tx=0x7f17d8004040 comp rx=0 tx=0).stop 2024-03-18T22:57:58.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.725+0000 7f17e8fb9640 1 -- 172.21.15.40:0/767044344 shutdown_connections 2024-03-18T22:57:58.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.725+0000 7f17e8fb9640 1 --2- 172.21.15.40:0/767044344 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f17cc0a2880 0x7f17cc0a4d40 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.725+0000 7f17e8fb9640 1 --2- 172.21.15.40:0/767044344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17e4102310 0x7f17e419edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.725+0000 7f17e8fb9640 1 --2- 172.21.15.40:0/767044344 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17e4101950 0x7f17e419c790 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.725+0000 7f17e8fb9640 1 --2- 172.21.15.40:0/767044344 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f17e4100760 0x7f17e406aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.725+0000 7f17e8fb9640 1 -- 172.21.15.40:0/767044344 >> 172.21.15.40:0/767044344 conn(0x7f17e40fbcd0 msgr2=0x7f17e4104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.725+0000 7f17e8fb9640 1 -- 172.21.15.40:0/767044344 shutdown_connections 2024-03-18T22:57:58.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.725+0000 7f17e8fb9640 1 -- 172.21.15.40:0/767044344 wait complete. 2024-03-18T22:57:58.737 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:57:58.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:58 smithi103 ceph-mon[66377]: pgmap v810: 129 pgs: 129 active+clean; 7.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 66 KiB/s wr, 119 op/s 2024-03-18T22:57:58.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:58 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:58.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:58 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3932940534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:57:58.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.955+0000 7f97251e8640 1 -- 172.21.15.40:0/3882615394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9720100830 msgr2=0x7f9720100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.955+0000 7f97251e8640 1 --2- 172.21.15.40:0/3882615394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9720100830 0x7f9720100cb0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f971801acb0 tx=0x7f971803d150 comp rx=0 tx=0).stop 2024-03-18T22:57:58.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.959+0000 7f97251e8640 1 -- 172.21.15.40:0/3882615394 shutdown_connections 2024-03-18T22:57:58.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.959+0000 7f97251e8640 1 --2- 172.21.15.40:0/3882615394 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f97201011f0 0x7f9720105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.959+0000 7f97251e8640 1 --2- 172.21.15.40:0/3882615394 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9720100830 0x7f9720100cb0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.959+0000 7f97251e8640 1 --2- 172.21.15.40:0/3882615394 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97200ffad0 0x7f97200ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.959+0000 7f97251e8640 1 -- 172.21.15.40:0/3882615394 >> 172.21.15.40:0/3882615394 conn(0x7f97200fb400 msgr2=0x7f97200fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.960+0000 7f97251e8640 1 -- 172.21.15.40:0/3882615394 shutdown_connections 2024-03-18T22:57:58.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.960+0000 7f97251e8640 1 -- 172.21.15.40:0/3882615394 wait complete. 2024-03-18T22:57:58.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.961+0000 7f97251e8640 1 Processor -- start 2024-03-18T22:57:58.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.965+0000 7f97251e8640 1 -- start start 2024-03-18T22:57:58.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.965+0000 7f97251e8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97200ffad0 0x7f9720071f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.965+0000 7f97251e8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9720100830 0x7f97200724c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.965+0000 7f97251e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97201011f0 0x7f9720072a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.965+0000 7f97251e8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9720109280 con 0x7f97201011f0 2024-03-18T22:57:58.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.965+0000 7f97251e8640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9720109100 con 0x7f9720100830 2024-03-18T22:57:58.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.965+0000 7f97251e8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9720109400 con 0x7f97200ffad0 2024-03-18T22:57:58.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.966+0000 7f971f59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97201011f0 0x7f9720072a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.966+0000 7f971f59e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97201011f0 0x7f9720072a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57226/0 (socket says 172.21.15.40:57226) 2024-03-18T22:57:58.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.966+0000 7f971f59e640 1 -- 172.21.15.40:0/1498778884 learned_addr learned my addr 172.21.15.40:0/1498778884 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.967+0000 7f971f59e640 1 -- 172.21.15.40:0/1498778884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97200ffad0 msgr2=0x7f9720071f80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T22:57:58.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.967+0000 7f971f59e640 1 --2- 172.21.15.40:0/1498778884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97200ffad0 0x7f9720071f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.967+0000 7f971f59e640 1 -- 172.21.15.40:0/1498778884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9720100830 msgr2=0x7f97200724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.967+0000 7f971f59e640 1 --2- 172.21.15.40:0/1498778884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9720100830 0x7f97200724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.967+0000 7f971f59e640 1 -- 172.21.15.40:0/1498778884 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97201a5d00 con 0x7f97201011f0 2024-03-18T22:57:58.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.967+0000 7f971f59e640 1 --2- 172.21.15.40:0/1498778884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97201011f0 0x7f9720072a20 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f971000b3f0 tx=0x7f971000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.968+0000 7f9707fff640 1 -- 172.21.15.40:0/1498778884 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9710015020 con 0x7f97201011f0 2024-03-18T22:57:58.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.968+0000 7f97251e8640 1 -- 172.21.15.40:0/1498778884 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97201a5ff0 con 0x7f97201011f0 2024-03-18T22:57:58.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.968+0000 7f97251e8640 1 -- 172.21.15.40:0/1498778884 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97201a64b0 con 0x7f97201011f0 2024-03-18T22:57:58.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.968+0000 7f9707fff640 1 -- 172.21.15.40:0/1498778884 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9710011ce0 con 0x7f97201011f0 2024-03-18T22:57:58.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.968+0000 7f9707fff640 1 -- 172.21.15.40:0/1498778884 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9710010da0 con 0x7f97201011f0 2024-03-18T22:57:58.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.972+0000 7f9727dcb740 1 Processor -- start 2024-03-18T22:57:58.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.973+0000 7f9727dcb740 1 -- start start 2024-03-18T22:57:58.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.974+0000 7f9707fff640 1 -- 172.21.15.40:0/1498778884 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9710018020 con 0x7f97201011f0 2024-03-18T22:57:58.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.974+0000 7f9727dcb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63b2f0 0x56008f63b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.974+0000 7f9727dcb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f63bce0 0x56008f6407e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.974+0000 7f9727dcb740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f640e50 0x56008f6432e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.974+0000 7f9727dcb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56008f59f4b0 con 0x56008f63b2f0 2024-03-18T22:57:58.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.974+0000 7f9727dcb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56008f547d70 con 0x56008f63bce0 2024-03-18T22:57:58.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.974+0000 7f9727dcb740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56008f59b550 con 0x56008f640e50 2024-03-18T22:57:58.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.974+0000 7f971ed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f63bce0 0x56008f6407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.974+0000 7f971ed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f63bce0 0x56008f6407e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60400/0 (socket says 172.21.15.40:60400) 2024-03-18T22:57:58.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.974+0000 7f971ed9d640 1 -- 172.21.15.40:0/1537363439 learned_addr learned my addr 172.21.15.40:0/1537363439 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.974+0000 7f971e59c640 1 --2- 172.21.15.40:0/1537363439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63b2f0 0x56008f63b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.974+0000 7f971e59c640 1 --2- 172.21.15.40:0/1537363439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f640e50 0x56008f6432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.974+0000 7f9707fff640 1 --2- 172.21.15.40:0/1498778884 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f97080a2940 0x7f97080a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.975+0000 7f9707fff640 1 -- 172.21.15.40:0/1498778884 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f97100ce6b0 con 0x7f97201011f0 2024-03-18T22:57:58.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.975+0000 7f971e59c640 1 -- 172.21.15.40:0/1537363439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f640e50 msgr2=0x56008f6432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.975+0000 7f971e59c640 1 --2- 172.21.15.40:0/1537363439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f640e50 0x56008f6432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.975+0000 7f971e59c640 1 -- 172.21.15.40:0/1537363439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f63bce0 msgr2=0x56008f6407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.975+0000 7f971e59c640 1 --2- 172.21.15.40:0/1537363439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f63bce0 0x56008f6407e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.975+0000 7f971e59c640 1 -- 172.21.15.40:0/1537363439 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56008f56a210 con 0x56008f63b2f0 2024-03-18T22:57:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.975+0000 7f971e59c640 1 --2- 172.21.15.40:0/1537363439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63b2f0 0x56008f63b710 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f971801ac00 tx=0x7f971803d670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4836ee82fbef4b8a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.975+0000 7f971f59e640 1 --2- 172.21.15.40:0/1498778884 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f97080a2940 0x7f97080a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.975+0000 7f97057fa640 1 -- 172.21.15.40:0/1537363439 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971804a040 con 0x56008f63b2f0 2024-03-18T22:57:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.975+0000 7f97057fa640 1 -- 172.21.15.40:0/1537363439 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9718043700 con 0x56008f63b2f0 2024-03-18T22:57:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.975+0000 7f97057fa640 1 -- 172.21.15.40:0/1537363439 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9718004310 con 0x56008f63b2f0 2024-03-18T22:57:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.976+0000 7f9727dcb740 1 -- 172.21.15.40:0/1537363439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63b2f0 msgr2=0x56008f63b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:58.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.976+0000 7f9727dcb740 1 --2- 172.21.15.40:0/1537363439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63b2f0 0x56008f63b710 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f971801ac00 tx=0x7f971803d670 comp rx=0 tx=0).stop 2024-03-18T22:57:58.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.976+0000 7f971f59e640 1 --2- 172.21.15.40:0/1498778884 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f97080a2940 0x7f97080a4e00 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f97100a2e70 tx=0x7f9710009240 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.978+0000 7f9727dcb740 1 -- 172.21.15.40:0/1537363439 shutdown_connections 2024-03-18T22:57:58.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.978+0000 7f9727dcb740 1 --2- 172.21.15.40:0/1537363439 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f640e50 0x56008f6432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.978+0000 7f9727dcb740 1 --2- 172.21.15.40:0/1537363439 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f63bce0 0x56008f6407e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.978+0000 7f9727dcb740 1 --2- 172.21.15.40:0/1537363439 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63b2f0 0x56008f63b710 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.978+0000 7f9727dcb740 1 -- 172.21.15.40:0/1537363439 >> 172.21.15.40:0/1537363439 conn(0x56008f636860 msgr2=0x56008f636c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:58.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.982+0000 7f9727dcb740 1 -- 172.21.15.40:0/1537363439 shutdown_connections 2024-03-18T22:57:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.983+0000 7f9727dcb740 1 -- 172.21.15.40:0/1537363439 wait complete. 2024-03-18T22:57:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.983+0000 7f9727dcb740 1 -- start start 2024-03-18T22:57:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.984+0000 7f9727dcb740 1 Processor -- start 2024-03-18T22:57:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.984+0000 7f9727dcb740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f63b2f0 0x56008f645860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.984+0000 7f9727dcb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63bce0 0x56008f645da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.984+0000 7f9727dcb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f640e50 0x56008f64cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.984+0000 7f9727dcb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56008f643d40 con 0x56008f63bce0 2024-03-18T22:57:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.984+0000 7f9727dcb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56008f55b310 con 0x56008f640e50 2024-03-18T22:57:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.984+0000 7f9727dcb740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56008f643ec0 con 0x56008f63b2f0 2024-03-18T22:57:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.984+0000 7f9727dcb740 10 client.0 _ll_register_callbacks cb 0x56008f54f2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.985+0000 7f971e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f63b2f0 0x56008f645860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.985+0000 7f971e59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f640e50 0x56008f64cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.985+0000 7f971e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f63b2f0 0x56008f645860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55298/0 (socket says 172.21.15.40:55298) 2024-03-18T22:57:58.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.985+0000 7f971e59c640 1 -- 172.21.15.40:0/2314954740 learned_addr learned my addr 172.21.15.40:0/2314954740 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.985+0000 7f971e59c640 1 -- 172.21.15.40:0/2314954740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f640e50 msgr2=0x56008f64cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.985+0000 7f971e59c640 1 --2- 172.21.15.40:0/2314954740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f640e50 0x56008f64cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.986+0000 7f971e59c640 1 -- 172.21.15.40:0/2314954740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63bce0 msgr2=0x56008f645da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.986+0000 7f971e59c640 1 --2- 172.21.15.40:0/2314954740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63bce0 0x56008f645da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.986+0000 7f971e59c640 1 -- 172.21.15.40:0/2314954740 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56008f2b33f0 con 0x56008f63b2f0 2024-03-18T22:57:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.986+0000 7f971e59c640 1 --2- 172.21.15.40:0/2314954740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f63b2f0 0x56008f645860 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f971801a630 tx=0x7f9718000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4377f30162070d17 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.986+0000 7f9702ff5640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:57:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.986+0000 7f9702ff5640 1 -- 172.21.15.40:0/2314954740 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971804a040 con 0x56008f63b2f0 2024-03-18T22:57:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.986+0000 7f9702ff5640 1 -- 172.21.15.40:0/2314954740 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9718004d50 con 0x56008f63b2f0 2024-03-18T22:57:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.987+0000 7f9727dcb740 1 -- 172.21.15.40:0/2314954740 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56008f5fd3e0 con 0x56008f63b2f0 2024-03-18T22:57:58.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.994+0000 7f9702ff5640 1 -- 172.21.15.40:0/2314954740 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f9718004460 con 0x56008f63b2f0 2024-03-18T22:57:58.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.994+0000 7f9727dcb740 10 client.28489 fetch_fsmap learned FSMap version 46 2024-03-18T22:57:58.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:58.994+0000 7f9727dcb740 1 -- 172.21.15.40:0/2314954740 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x56008f37d7d0 con 0x56008f63b2f0 2024-03-18T22:57:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.000+0000 7f9702ff5640 1 -- 172.21.15.40:0/2314954740 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56008f5fd3e0 con 0x56008f63b2f0 2024-03-18T22:57:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.000+0000 7f9702ff5640 1 -- 172.21.15.40:0/2314954740 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 46) ==== 8406+0+0 (secure 0 0 0) 0x7f9718050050 con 0x56008f63b2f0 2024-03-18T22:57:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.001+0000 7f9727dcb740 10 client.28489 fetch_fsmap finished waiting for FSMap version 46 2024-03-18T22:57:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.001+0000 7f9727dcb740 10 client.28489 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:57:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.001+0000 7f9727dcb740 20 client.28489 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.001+0000 7f9727dcb740 1 --2- 172.21.15.40:0/2314954740 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x56008f651260 0x56008f653650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.001+0000 7f9727dcb740 4 client.28489 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:57:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.001+0000 7f9727dcb740 1 -- 172.21.15.40:0/2314954740 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56008f5ac9f0 con 0x56008f651260 2024-03-18T22:57:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.002+0000 7f971ed9d640 1 --2- 172.21.15.40:0/2314954740 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x56008f651260 0x56008f653650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.006+0000 7f971ed9d640 1 --2- 172.21.15.40:0/2314954740 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x56008f651260 0x56008f653650 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=468aef837cbad578 server_cookie=2f7370d24a1b0852 in_seq=0 out_seq=0 2024-03-18T22:57:59.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.006+0000 7f9702ff5640 10 client.28489 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:57:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.008+0000 7f9702ff5640 1 -- 172.21.15.40:0/2314954740 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56008f5ac9f0 con 0x56008f651260 2024-03-18T22:57:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.008+0000 7f9702ff5640 10 client.28489 handle_command_reply: tid=0 2024-03-18T22:57:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.010+0000 7f9727dcb740 1 client.28489 shutdown 2024-03-18T22:57:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.010+0000 7f9727dcb740 1 -- 172.21.15.40:0/2314954740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f63b2f0 msgr2=0x56008f645860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:59.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.010+0000 7f9727dcb740 1 --2- 172.21.15.40:0/2314954740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f63b2f0 0x56008f645860 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f971801a630 tx=0x7f9718000f40 comp rx=0 tx=0).stop 2024-03-18T22:57:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.011+0000 7f9727dcb740 1 -- 172.21.15.40:0/2314954740 shutdown_connections 2024-03-18T22:57:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.011+0000 7f9727dcb740 1 --2- 172.21.15.40:0/2314954740 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x56008f651260 0x56008f653650 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.011+0000 7f9727dcb740 1 --2- 172.21.15.40:0/2314954740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f640e50 0x56008f64cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.011+0000 7f9727dcb740 1 --2- 172.21.15.40:0/2314954740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63bce0 0x56008f645da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.011+0000 7f9727dcb740 1 --2- 172.21.15.40:0/2314954740 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f63b2f0 0x56008f645860 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.011+0000 7f9727dcb740 1 -- 172.21.15.40:0/2314954740 >> 172.21.15.40:0/2314954740 conn(0x56008f636860 msgr2=0x56008f635ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:59.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.012+0000 7f9702ff5640 0 client.28489 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:57:59.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.012+0000 7f9727dcb740 1 -- 172.21.15.40:0/2314954740 shutdown_connections 2024-03-18T22:57:59.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.013+0000 7f9727dcb740 1 -- 172.21.15.40:0/2314954740 wait complete. 2024-03-18T22:57:59.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.015+0000 7f9727dcb740 20 client.28489 trim_cache size 0 max 16384 2024-03-18T22:57:59.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.041+0000 7f9727dcb740 1 Processor -- start 2024-03-18T22:57:59.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.041+0000 7f9727dcb740 1 -- start start 2024-03-18T22:57:59.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.042+0000 7f9727dcb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63b2f0 0x56008f645b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.042+0000 7f9727dcb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f63bce0 0x56008f64cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.042+0000 7f9727dcb740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f640e50 0x56008f635310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.042+0000 7f9727dcb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f970c00fe90 con 0x56008f63b2f0 2024-03-18T22:57:59.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.042+0000 7f9727dcb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f970c011b80 con 0x56008f63bce0 2024-03-18T22:57:59.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.042+0000 7f9727dcb740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f970c010560 con 0x56008f640e50 2024-03-18T22:57:59.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.042+0000 7f971e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63b2f0 0x56008f645b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.042+0000 7f971e59c640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63b2f0 0x56008f645b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57250/0 (socket says 172.21.15.40:57250) 2024-03-18T22:57:59.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.042+0000 7f971e59c640 1 -- 172.21.15.40:0/1349835543 learned_addr learned my addr 172.21.15.40:0/1349835543 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.042+0000 7f971e59c640 1 --2- 172.21.15.40:0/1349835543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f640e50 0x56008f635310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.042+0000 7f971e59c640 1 -- 172.21.15.40:0/1349835543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f640e50 msgr2=0x56008f635310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.042+0000 7f971e59c640 1 --2- 172.21.15.40:0/1349835543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f640e50 0x56008f635310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.043+0000 7f971e59c640 1 -- 172.21.15.40:0/1349835543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f63bce0 msgr2=0x56008f64cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.043+0000 7f971e59c640 1 --2- 172.21.15.40:0/1349835543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f63bce0 0x56008f64cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.043+0000 7f971e59c640 1 -- 172.21.15.40:0/1349835543 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56008f6372b0 con 0x56008f63b2f0 2024-03-18T22:57:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.043+0000 7f971e59c640 1 --2- 172.21.15.40:0/1349835543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63b2f0 0x56008f645b80 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f9718045e20 tx=0x7f9718043340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a0a891bab35b4ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:59.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.044+0000 7f9704ff9640 1 -- 172.21.15.40:0/1349835543 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971806e070 con 0x56008f63b2f0 2024-03-18T22:57:59.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.044+0000 7f9704ff9640 1 -- 172.21.15.40:0/1349835543 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f971806f040 con 0x56008f63b2f0 2024-03-18T22:57:59.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.044+0000 7f9704ff9640 1 -- 172.21.15.40:0/1349835543 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971804a070 con 0x56008f63b2f0 2024-03-18T22:57:59.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.045+0000 7f9727dcb740 1 -- 172.21.15.40:0/1349835543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63b2f0 msgr2=0x56008f645b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:59.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.045+0000 7f9727dcb740 1 --2- 172.21.15.40:0/1349835543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63b2f0 0x56008f645b80 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f9718045e20 tx=0x7f9718043340 comp rx=0 tx=0).stop 2024-03-18T22:57:59.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.045+0000 7f9727dcb740 1 -- 172.21.15.40:0/1349835543 shutdown_connections 2024-03-18T22:57:59.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.045+0000 7f9727dcb740 1 --2- 172.21.15.40:0/1349835543 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f640e50 0x56008f635310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.045+0000 7f9727dcb740 1 --2- 172.21.15.40:0/1349835543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f63bce0 0x56008f64cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.046+0000 7f9727dcb740 1 --2- 172.21.15.40:0/1349835543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63b2f0 0x56008f645b80 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.046+0000 7f9727dcb740 1 -- 172.21.15.40:0/1349835543 >> 172.21.15.40:0/1349835543 conn(0x56008f636860 msgr2=0x56008f645470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:59.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.047+0000 7f9727dcb740 1 -- 172.21.15.40:0/1349835543 shutdown_connections 2024-03-18T22:57:59.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.048+0000 7f9727dcb740 1 -- 172.21.15.40:0/1349835543 wait complete. 2024-03-18T22:57:59.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.048+0000 7f9727dcb740 1 -- start start 2024-03-18T22:57:59.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.048+0000 7f9727dcb740 1 Processor -- start 2024-03-18T22:57:59.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.048+0000 7f9727dcb740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f63b2f0 0x56008f649860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.048+0000 7f9727dcb740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63bce0 0x56008f649da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.048+0000 7f9727dcb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f640e50 0x56008f6b5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.049+0000 7f9727dcb740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f970c0113f0 con 0x56008f63bce0 2024-03-18T22:57:59.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.049+0000 7f9727dcb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f970c010cf0 con 0x56008f640e50 2024-03-18T22:57:59.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.049+0000 7f9727dcb740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f970c00de40 con 0x56008f63b2f0 2024-03-18T22:57:59.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.049+0000 7f9727dcb740 10 client.0 _ll_register_callbacks cb 0x7f9714009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:59.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.050+0000 7f971ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63bce0 0x56008f649da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.050+0000 7f971e59c640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f63b2f0 0x56008f649860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.050+0000 7f971ed9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63bce0 0x56008f649da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57260/0 (socket says 172.21.15.40:57260) 2024-03-18T22:57:59.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.050+0000 7f971ed9d640 1 -- 172.21.15.40:0/2291717434 learned_addr learned my addr 172.21.15.40:0/2291717434 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:59.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.050+0000 7f971ed9d640 1 -- 172.21.15.40:0/2291717434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f63b2f0 msgr2=0x56008f649860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:59.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.050+0000 7f971ed9d640 1 --2- 172.21.15.40:0/2291717434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f63b2f0 0x56008f649860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.050+0000 7f971ed9d640 1 -- 172.21.15.40:0/2291717434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f640e50 msgr2=0x56008f6b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:59.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.050+0000 7f971e59c640 1 --2- 172.21.15.40:0/2291717434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f640e50 0x56008f6b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.050+0000 7f971ed9d640 1 --2- 172.21.15.40:0/2291717434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f640e50 0x56008f6b5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.050+0000 7f971ed9d640 1 -- 172.21.15.40:0/2291717434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56008f64ce70 con 0x56008f63bce0 2024-03-18T22:57:59.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.051+0000 7f971ed9d640 1 --2- 172.21.15.40:0/2291717434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63bce0 0x56008f649da0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f97140093b0 tx=0x7f9714015780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb946aee0ba27694 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:59.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.051+0000 7f9702ff5640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:57:59.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.051+0000 7f9702ff5640 1 -- 172.21.15.40:0/2291717434 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9714015d00 con 0x56008f63bce0 2024-03-18T22:57:59.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.051+0000 7f9702ff5640 1 -- 172.21.15.40:0/2291717434 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9714019160 con 0x56008f63bce0 2024-03-18T22:57:59.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.051+0000 7f9727dcb740 1 -- 172.21.15.40:0/2291717434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f971806f040 con 0x56008f63bce0 2024-03-18T22:57:59.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.051+0000 7f9702ff5640 1 -- 172.21.15.40:0/2291717434 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f971402b050 con 0x56008f63bce0 2024-03-18T22:57:59.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.051+0000 7f9727dcb740 10 client.18687 fetch_fsmap learned FSMap version 46 2024-03-18T22:57:59.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.051+0000 7f9727dcb740 1 -- 172.21.15.40:0/2291717434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x56008f6b6920 con 0x56008f63bce0 2024-03-18T22:57:59.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.052+0000 7f9702ff5640 1 -- 172.21.15.40:0/2291717434 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f971401d070 con 0x56008f63bce0 2024-03-18T22:57:59.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.052+0000 7f9702ff5640 1 -- 172.21.15.40:0/2291717434 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 46) ==== 8406+0+0 (secure 0 0 0) 0x7f9714022ae0 con 0x56008f63bce0 2024-03-18T22:57:59.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.052+0000 7f9727dcb740 10 client.18687 fetch_fsmap finished waiting for FSMap version 46 2024-03-18T22:57:59.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.052+0000 7f9727dcb740 10 client.18687 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:57:59.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.052+0000 7f9727dcb740 20 client.18687 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:59.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.052+0000 7f9727dcb740 1 --2- 172.21.15.40:0/2291717434 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x56008f6b7380 0x56008f6b97a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.052+0000 7f9727dcb740 4 client.18687 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:57:59.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.052+0000 7f9727dcb740 1 -- 172.21.15.40:0/2291717434 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f971806f040 con 0x56008f6b7380 2024-03-18T22:57:59.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.053+0000 7f971e59c640 1 --2- 172.21.15.40:0/2291717434 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x56008f6b7380 0x56008f6b97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.054+0000 7f971e59c640 1 --2- 172.21.15.40:0/2291717434 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x56008f6b7380 0x56008f6b97a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=92d67fcb3568757d server_cookie=a3b3e900623f0f1b in_seq=0 out_seq=0 2024-03-18T22:57:59.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.055+0000 7f9702ff5640 10 client.18687 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:57:59.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.059+0000 7f9702ff5640 1 -- 172.21.15.40:0/2291717434 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f971806f040 con 0x56008f6b7380 2024-03-18T22:57:59.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.059+0000 7f9702ff5640 10 client.18687 handle_command_reply: tid=0 2024-03-18T22:57:59.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.059+0000 7f9727dcb740 1 client.18687 shutdown 2024-03-18T22:57:59.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.061+0000 7f9727dcb740 1 -- 172.21.15.40:0/2291717434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63bce0 msgr2=0x56008f649da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:59.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.061+0000 7f9727dcb740 1 --2- 172.21.15.40:0/2291717434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63bce0 0x56008f649da0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f97140093b0 tx=0x7f9714015780 comp rx=0 tx=0).stop 2024-03-18T22:57:59.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.061+0000 7f9727dcb740 1 -- 172.21.15.40:0/2291717434 shutdown_connections 2024-03-18T22:57:59.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.061+0000 7f9727dcb740 1 --2- 172.21.15.40:0/2291717434 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x56008f6b7380 0x56008f6b97a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.062+0000 7f9727dcb740 1 --2- 172.21.15.40:0/2291717434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56008f640e50 0x56008f6b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.062+0000 7f9727dcb740 1 --2- 172.21.15.40:0/2291717434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56008f63bce0 0x56008f649da0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.062+0000 7f9727dcb740 1 --2- 172.21.15.40:0/2291717434 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56008f63b2f0 0x56008f649860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.062+0000 7f9727dcb740 1 -- 172.21.15.40:0/2291717434 >> 172.21.15.40:0/2291717434 conn(0x56008f636860 msgr2=0x56008f63a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:59.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.063+0000 7f9702ff5640 0 client.18687 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:57:59.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.063+0000 7f9727dcb740 1 -- 172.21.15.40:0/2291717434 shutdown_connections 2024-03-18T22:57:59.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.063+0000 7f9727dcb740 1 -- 172.21.15.40:0/2291717434 wait complete. 2024-03-18T22:57:59.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.064+0000 7f9727dcb740 20 client.18687 trim_cache size 0 max 16384 2024-03-18T22:57:59.065 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:57:59.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.065+0000 7f97251e8640 1 -- 172.21.15.40:0/1498778884 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f97080a2940 msgr2=0x7f97080a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:59.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.065+0000 7f97251e8640 1 --2- 172.21.15.40:0/1498778884 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f97080a2940 0x7f97080a4e00 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f97100a2e70 tx=0x7f9710009240 comp rx=0 tx=0).stop 2024-03-18T22:57:59.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.065+0000 7f97251e8640 1 -- 172.21.15.40:0/1498778884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97201011f0 msgr2=0x7f9720072a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.065+0000 7f97251e8640 1 --2- 172.21.15.40:0/1498778884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97201011f0 0x7f9720072a20 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f971000b3f0 tx=0x7f971000b8b0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.065+0000 7f97251e8640 1 -- 172.21.15.40:0/1498778884 shutdown_connections 2024-03-18T22:57:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.065+0000 7f97251e8640 1 --2- 172.21.15.40:0/1498778884 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f97080a2940 0x7f97080a4e00 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.065+0000 7f97251e8640 1 --2- 172.21.15.40:0/1498778884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97201011f0 0x7f9720072a20 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.065+0000 7f97251e8640 1 --2- 172.21.15.40:0/1498778884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9720100830 0x7f97200724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.065+0000 7f97251e8640 1 --2- 172.21.15.40:0/1498778884 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f97200ffad0 0x7f9720071f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.065+0000 7f97251e8640 1 -- 172.21.15.40:0/1498778884 >> 172.21.15.40:0/1498778884 conn(0x7f97200fb400 msgr2=0x7f9720103b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.067+0000 7f97251e8640 1 -- 172.21.15.40:0/1498778884 shutdown_connections 2024-03-18T22:57:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.069+0000 7f97251e8640 1 -- 172.21.15.40:0/1498778884 wait complete. 2024-03-18T22:57:59.089 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:57:59.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.258+0000 7fea333b9640 1 -- 172.21.15.40:0/1263045211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea2c0ffe90 msgr2=0x7fea2c1002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:59.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.258+0000 7fea333b9640 1 --2- 172.21.15.40:0/1263045211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea2c0ffe90 0x7fea2c1002b0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fea2800aa10 tx=0x7fea2802cf70 comp rx=0 tx=0).stop 2024-03-18T22:57:59.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.258+0000 7fea333b9640 1 -- 172.21.15.40:0/1263045211 shutdown_connections 2024-03-18T22:57:59.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.258+0000 7fea333b9640 1 --2- 172.21.15.40:0/1263045211 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea2c1019a0 0x7fea2c1082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.258+0000 7fea333b9640 1 --2- 172.21.15.40:0/1263045211 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea2c100fe0 0x7fea2c101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.258+0000 7fea333b9640 1 --2- 172.21.15.40:0/1263045211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea2c0ffe90 0x7fea2c1002b0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.258+0000 7fea333b9640 1 -- 172.21.15.40:0/1263045211 >> 172.21.15.40:0/1263045211 conn(0x7fea2c0fb400 msgr2=0x7fea2c0fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:59.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.259+0000 7fea333b9640 1 -- 172.21.15.40:0/1263045211 shutdown_connections 2024-03-18T22:57:59.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.259+0000 7fea333b9640 1 -- 172.21.15.40:0/1263045211 wait complete. 2024-03-18T22:57:59.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.260+0000 7fea333b9640 1 Processor -- start 2024-03-18T22:57:59.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.260+0000 7fea333b9640 1 -- start start 2024-03-18T22:57:59.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.260+0000 7fea333b9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea2c0ffe90 0x7fea2c195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.260+0000 7fea333b9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea2c100fe0 0x7fea2c195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.260+0000 7fea333b9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea2c1019a0 0x7fea2c19cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.261+0000 7fea30954640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea2c100fe0 0x7fea2c195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.261+0000 7fea30954640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea2c100fe0 0x7fea2c195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55358/0 (socket says 172.21.15.40:55358) 2024-03-18T22:57:59.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.261+0000 7fea31956640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea2c1019a0 0x7fea2c19cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.261+0000 7fea31956640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea2c1019a0 0x7fea2c19cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57280/0 (socket says 172.21.15.40:57280) 2024-03-18T22:57:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.261+0000 7fea30954640 1 -- 172.21.15.40:0/1726793376 learned_addr learned my addr 172.21.15.40:0/1726793376 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.261+0000 7fea333b9640 1 -- 172.21.15.40:0/1726793376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c10b140 con 0x7fea2c1019a0 2024-03-18T22:57:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.261+0000 7fea333b9640 1 -- 172.21.15.40:0/1726793376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c10afc0 con 0x7fea2c0ffe90 2024-03-18T22:57:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.261+0000 7fea333b9640 1 -- 172.21.15.40:0/1726793376 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c10b2c0 con 0x7fea2c100fe0 2024-03-18T22:57:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.261+0000 7fea31155640 1 --2- 172.21.15.40:0/1726793376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea2c0ffe90 0x7fea2c195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.262+0000 7fea31956640 1 -- 172.21.15.40:0/1726793376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea2c100fe0 msgr2=0x7fea2c195d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.262+0000 7fea31956640 1 --2- 172.21.15.40:0/1726793376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea2c100fe0 0x7fea2c195d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.262+0000 7fea31956640 1 -- 172.21.15.40:0/1726793376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea2c0ffe90 msgr2=0x7fea2c195820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.262+0000 7fea31956640 1 --2- 172.21.15.40:0/1726793376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea2c0ffe90 0x7fea2c195820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.262+0000 7fea31956640 1 -- 172.21.15.40:0/1726793376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2c19d440 con 0x7fea2c1019a0 2024-03-18T22:57:59.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.262+0000 7fea31956640 1 --2- 172.21.15.40:0/1726793376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea2c1019a0 0x7fea2c19cdd0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fea24009fd0 tx=0x7fea2400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:59.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.262+0000 7fea227fc640 1 -- 172.21.15.40:0/1726793376 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2400ce20 con 0x7fea2c1019a0 2024-03-18T22:57:59.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.263+0000 7fea227fc640 1 -- 172.21.15.40:0/1726793376 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea24005020 con 0x7fea2c1019a0 2024-03-18T22:57:59.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.263+0000 7fea227fc640 1 -- 172.21.15.40:0/1726793376 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea24010690 con 0x7fea2c1019a0 2024-03-18T22:57:59.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.263+0000 7fea333b9640 1 -- 172.21.15.40:0/1726793376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea2c19d730 con 0x7fea2c1019a0 2024-03-18T22:57:59.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.263+0000 7fea333b9640 1 -- 172.21.15.40:0/1726793376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea2c19dc30 con 0x7fea2c1019a0 2024-03-18T22:57:59.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.267+0000 7fea35fa4740 1 Processor -- start 2024-03-18T22:57:59.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.268+0000 7fea35fa4740 1 -- start start 2024-03-18T22:57:59.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.268+0000 7fea35fa4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 0x560bbe2f7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.268+0000 7fea35fa4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe3050a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.268+0000 7fea35fa4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe305670 0x560bbe307ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.268+0000 7fea35fa4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560bbe25b4b0 con 0x560bbe2f7670 2024-03-18T22:57:59.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.268+0000 7fea35fa4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560bbe203d70 con 0x560bbe2f7fd0 2024-03-18T22:57:59.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.268+0000 7fea35fa4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560bbe257550 con 0x560bbe305670 2024-03-18T22:57:59.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.268+0000 7fea227fc640 1 -- 172.21.15.40:0/1726793376 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fea240051d0 con 0x7fea2c1019a0 2024-03-18T22:57:59.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.269+0000 7fea227fc640 1 --2- 172.21.15.40:0/1726793376 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fea180a2900 0x7fea180a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.270+0000 7fea227fc640 1 -- 172.21.15.40:0/1726793376 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fea240cf960 con 0x7fea2c1019a0 2024-03-18T22:57:59.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.270+0000 7fea31155640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe3050a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.270+0000 7fea31155640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe3050a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60464/0 (socket says 172.21.15.40:60464) 2024-03-18T22:57:59.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.270+0000 7fea31155640 1 -- 172.21.15.40:0/1472290988 learned_addr learned my addr 172.21.15.40:0/1472290988 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:59.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.270+0000 7fea31956640 1 --2- 172.21.15.40:0/1726793376 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fea180a2900 0x7fea180a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.270+0000 7fea30954640 1 --2- 172.21.15.40:0/1472290988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe305670 0x560bbe307ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.271+0000 7fea30954640 1 --2- 172.21.15.40:0/1472290988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 0x560bbe2f7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.271+0000 7fea31155640 1 -- 172.21.15.40:0/1472290988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe305670 msgr2=0x560bbe307ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:59.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.271+0000 7fea31155640 1 --2- 172.21.15.40:0/1472290988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe305670 0x560bbe307ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.271+0000 7fea31155640 1 -- 172.21.15.40:0/1472290988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 msgr2=0x560bbe2f7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:59.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.271+0000 7fea31155640 1 --2- 172.21.15.40:0/1472290988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 0x560bbe2f7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.271+0000 7fea31155640 1 -- 172.21.15.40:0/1472290988 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bbe226210 con 0x560bbe2f7fd0 2024-03-18T22:57:59.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.272+0000 7fea31155640 1 --2- 172.21.15.40:0/1472290988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe3050a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fea28006010 tx=0x7fea2800a780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cd527bee8f6a6a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:59.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.272+0000 7fea31956640 1 --2- 172.21.15.40:0/1726793376 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fea180a2900 0x7fea180a4dc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fea240a4120 tx=0x7fea2400c890 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:59.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.273+0000 7fea177fe640 1 -- 172.21.15.40:0/1472290988 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28035440 con 0x560bbe2f7fd0 2024-03-18T22:57:59.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.273+0000 7fea177fe640 1 -- 172.21.15.40:0/1472290988 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea280355e0 con 0x560bbe2f7fd0 2024-03-18T22:57:59.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.273+0000 7fea177fe640 1 -- 172.21.15.40:0/1472290988 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28004030 con 0x560bbe2f7fd0 2024-03-18T22:57:59.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.274+0000 7fea35fa4740 1 -- 172.21.15.40:0/1472290988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 msgr2=0x560bbe3050a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:59.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.274+0000 7fea35fa4740 1 --2- 172.21.15.40:0/1472290988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe3050a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fea28006010 tx=0x7fea2800a780 comp rx=0 tx=0).stop 2024-03-18T22:57:59.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.276+0000 7fea35fa4740 1 -- 172.21.15.40:0/1472290988 shutdown_connections 2024-03-18T22:57:59.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.276+0000 7fea35fa4740 1 --2- 172.21.15.40:0/1472290988 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe305670 0x560bbe307ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.276+0000 7fea35fa4740 1 --2- 172.21.15.40:0/1472290988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe3050a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.276+0000 7fea35fa4740 1 --2- 172.21.15.40:0/1472290988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 0x560bbe2f7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.276+0000 7fea35fa4740 1 -- 172.21.15.40:0/1472290988 >> 172.21.15.40:0/1472290988 conn(0x560bbe2f3e20 msgr2=0x560bbe2f2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:59.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.276+0000 7fea35fa4740 1 -- 172.21.15.40:0/1472290988 shutdown_connections 2024-03-18T22:57:59.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.276+0000 7fea35fa4740 1 -- 172.21.15.40:0/1472290988 wait complete. 2024-03-18T22:57:59.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.277+0000 7fea35fa4740 1 -- start start 2024-03-18T22:57:59.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.277+0000 7fea35fa4740 1 Processor -- start 2024-03-18T22:57:59.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.278+0000 7fea35fa4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 0x560bbe31aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.278+0000 7fea35fa4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe31b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.278+0000 7fea35fa4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe305670 0x560bbe322570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.278+0000 7fea35fa4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x560bbe319500 con 0x560bbe2f7670 2024-03-18T22:57:59.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.278+0000 7fea35fa4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560bbe217310 con 0x560bbe2f7fd0 2024-03-18T22:57:59.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.278+0000 7fea35fa4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x560bbe319680 con 0x560bbe305670 2024-03-18T22:57:59.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.278+0000 7fea35fa4740 10 client.0 _ll_register_callbacks cb 0x560bbe20b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.280+0000 7fea31155640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe31b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.281+0000 7fea31155640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe31b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60468/0 (socket says 172.21.15.40:60468) 2024-03-18T22:57:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.281+0000 7fea31155640 1 -- 172.21.15.40:0/87886325 learned_addr learned my addr 172.21.15.40:0/87886325 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.281+0000 7fea30954640 1 --2- 172.21.15.40:0/87886325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe305670 0x560bbe322570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.281+0000 7fea30954640 1 --2- 172.21.15.40:0/87886325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 0x560bbe31aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.281+0000 7fea31155640 1 -- 172.21.15.40:0/87886325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe305670 msgr2=0x560bbe322570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.281+0000 7fea31155640 1 --2- 172.21.15.40:0/87886325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe305670 0x560bbe322570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.281+0000 7fea31155640 1 -- 172.21.15.40:0/87886325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 msgr2=0x560bbe31aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:59.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.281+0000 7fea31155640 1 --2- 172.21.15.40:0/87886325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 0x560bbe31aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.281+0000 7fea31155640 1 -- 172.21.15.40:0/87886325 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bbdf6f3f0 con 0x560bbe2f7fd0 2024-03-18T22:57:59.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.281+0000 7fea31155640 1 --2- 172.21.15.40:0/87886325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe31b530 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fea2800aa10 tx=0x7fea2802dd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a148b7afd50699db server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:59.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.281+0000 7fea14ff9640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:57:59.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.282+0000 7fea14ff9640 1 -- 172.21.15.40:0/87886325 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2804a020 con 0x560bbe2f7fd0 2024-03-18T22:57:59.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.282+0000 7fea35fa4740 1 -- 172.21.15.40:0/87886325 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea280355e0 con 0x560bbe2f7fd0 2024-03-18T22:57:59.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.282+0000 7fea14ff9640 1 -- 172.21.15.40:0/87886325 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fea28004030 con 0x560bbe2f7fd0 2024-03-18T22:57:59.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.289+0000 7fea14ff9640 1 -- 172.21.15.40:0/87886325 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fea28055050 con 0x560bbe2f7fd0 2024-03-18T22:57:59.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.290+0000 7fea35fa4740 10 client.28511 fetch_fsmap learned FSMap version 46 2024-03-18T22:57:59.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.290+0000 7fea35fa4740 1 -- 172.21.15.40:0/87886325 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x560bbe0397d0 con 0x560bbe2f7fd0 2024-03-18T22:57:59.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.290+0000 7fea14ff9640 1 -- 172.21.15.40:0/87886325 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea280355e0 con 0x560bbe2f7fd0 2024-03-18T22:57:59.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.290+0000 7fea14ff9640 1 -- 172.21.15.40:0/87886325 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 46) ==== 8406+0+0 (secure 0 0 0) 0x7fea2800b040 con 0x560bbe2f7fd0 2024-03-18T22:57:59.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.290+0000 7fea35fa4740 10 client.28511 fetch_fsmap finished waiting for FSMap version 46 2024-03-18T22:57:59.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.290+0000 7fea35fa4740 10 client.28511 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:57:59.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.290+0000 7fea35fa4740 20 client.28511 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:59.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.290+0000 7fea35fa4740 1 --2- 172.21.15.40:0/87886325 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560bbe326a60 0x560bbe328ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.290+0000 7fea35fa4740 4 client.28511 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:57:59.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.290+0000 7fea35fa4740 1 -- 172.21.15.40:0/87886325 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bbe2b93e0 con 0x560bbe326a60 2024-03-18T22:57:59.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.291+0000 7fea30954640 1 --2- 172.21.15.40:0/87886325 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560bbe326a60 0x560bbe328ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.301+0000 7fea30954640 1 --2- 172.21.15.40:0/87886325 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560bbe326a60 0x560bbe328ea0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e3b95e9faf1dd5e server_cookie=5af11702a87893c0 in_seq=0 out_seq=0 2024-03-18T22:57:59.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.301+0000 7fea14ff9640 10 client.28511 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:59.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.303+0000 7fea14ff9640 1 -- 172.21.15.40:0/87886325 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560bbe2b93e0 con 0x560bbe326a60 2024-03-18T22:57:59.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.303+0000 7fea14ff9640 10 client.28511 handle_command_reply: tid=0 2024-03-18T22:57:59.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.303+0000 7fea35fa4740 1 client.28511 shutdown 2024-03-18T22:57:59.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.303+0000 7fea35fa4740 1 -- 172.21.15.40:0/87886325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 msgr2=0x560bbe31b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:59.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.303+0000 7fea35fa4740 1 --2- 172.21.15.40:0/87886325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe31b530 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fea2800aa10 tx=0x7fea2802dd50 comp rx=0 tx=0).stop 2024-03-18T22:57:59.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.303+0000 7fea35fa4740 1 -- 172.21.15.40:0/87886325 shutdown_connections 2024-03-18T22:57:59.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.303+0000 7fea35fa4740 1 --2- 172.21.15.40:0/87886325 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe305670 0x560bbe322570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.303+0000 7fea35fa4740 1 --2- 172.21.15.40:0/87886325 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560bbe326a60 0x560bbe328ea0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.303+0000 7fea35fa4740 1 --2- 172.21.15.40:0/87886325 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe31b530 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.303+0000 7fea35fa4740 1 --2- 172.21.15.40:0/87886325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 0x560bbe31aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.303+0000 7fea35fa4740 1 -- 172.21.15.40:0/87886325 >> 172.21.15.40:0/87886325 conn(0x560bbe2f3e20 msgr2=0x560bbe2f1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:59.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.303+0000 7fea14ff9640 0 client.28511 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:59.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.304+0000 7fea35fa4740 1 -- 172.21.15.40:0/87886325 shutdown_connections 2024-03-18T22:57:59.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.305+0000 7fea35fa4740 1 -- 172.21.15.40:0/87886325 wait complete. 2024-03-18T22:57:59.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.305+0000 7fea35fa4740 20 client.28511 trim_cache size 0 max 16384 2024-03-18T22:57:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.331+0000 7fea35fa4740 1 Processor -- start 2024-03-18T22:57:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.331+0000 7fea35fa4740 1 -- start start 2024-03-18T22:57:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.331+0000 7fea35fa4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 0x560bbe2f1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.331+0000 7fea35fa4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe2f7fd0 0x560bbe322530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.331+0000 7fea35fa4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe305670 0x560bbe2f1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.331+0000 7fea35fa4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c19fb70 con 0x560bbe2f7670 2024-03-18T22:57:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.331+0000 7fea35fa4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c1a1860 con 0x560bbe305670 2024-03-18T22:57:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.331+0000 7fea35fa4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c1a0240 con 0x560bbe2f7fd0 2024-03-18T22:57:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.332+0000 7fea30954640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 0x560bbe2f1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.332+0000 7fea30954640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 0x560bbe2f1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57300/0 (socket says 172.21.15.40:57300) 2024-03-18T22:57:59.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.332+0000 7fea30954640 1 -- 172.21.15.40:0/777623561 learned_addr learned my addr 172.21.15.40:0/777623561 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:59.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.332+0000 7fea30954640 1 --2- 172.21.15.40:0/777623561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe305670 0x560bbe2f1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.332+0000 7fea31155640 1 --2- 172.21.15.40:0/777623561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe2f7fd0 0x560bbe322530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.332+0000 7fea30954640 1 -- 172.21.15.40:0/777623561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe2f7fd0 msgr2=0x560bbe322530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:59.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.332+0000 7fea30954640 1 --2- 172.21.15.40:0/777623561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe2f7fd0 0x560bbe322530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.332+0000 7fea30954640 1 -- 172.21.15.40:0/777623561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe305670 msgr2=0x560bbe2f1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:59.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.332+0000 7fea30954640 1 --2- 172.21.15.40:0/777623561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe305670 0x560bbe2f1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.332+0000 7fea30954640 1 -- 172.21.15.40:0/777623561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bbe2f3710 con 0x560bbe2f7670 2024-03-18T22:57:59.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.333+0000 7fea30954640 1 --2- 172.21.15.40:0/777623561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 0x560bbe2f1310 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00b690 tx=0x7fea1c00b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba188884590c5f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:59.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.333+0000 7fea16ffd640 1 -- 172.21.15.40:0/777623561 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c020380 con 0x560bbe2f7670 2024-03-18T22:57:59.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.333+0000 7fea16ffd640 1 -- 172.21.15.40:0/777623561 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea1c020520 con 0x560bbe2f7670 2024-03-18T22:57:59.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.333+0000 7fea16ffd640 1 -- 172.21.15.40:0/777623561 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c012de0 con 0x560bbe2f7670 2024-03-18T22:57:59.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.334+0000 7fea35fa4740 1 -- 172.21.15.40:0/777623561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 msgr2=0x560bbe2f1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:59.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.334+0000 7fea35fa4740 1 --2- 172.21.15.40:0/777623561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 0x560bbe2f1310 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00b690 tx=0x7fea1c00b410 comp rx=0 tx=0).stop 2024-03-18T22:57:59.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.334+0000 7fea35fa4740 1 -- 172.21.15.40:0/777623561 shutdown_connections 2024-03-18T22:57:59.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.334+0000 7fea35fa4740 1 --2- 172.21.15.40:0/777623561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe305670 0x560bbe2f1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.334+0000 7fea35fa4740 1 --2- 172.21.15.40:0/777623561 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe2f7fd0 0x560bbe322530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.334+0000 7fea35fa4740 1 --2- 172.21.15.40:0/777623561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe2f7670 0x560bbe2f1310 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.334+0000 7fea35fa4740 1 -- 172.21.15.40:0/777623561 >> 172.21.15.40:0/777623561 conn(0x560bbe2f3e20 msgr2=0x560bbe319a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:59.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.334+0000 7fea35fa4740 1 -- 172.21.15.40:0/777623561 shutdown_connections 2024-03-18T22:57:59.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.334+0000 7fea35fa4740 1 -- 172.21.15.40:0/777623561 wait complete. 2024-03-18T22:57:59.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.334+0000 7fea35fa4740 1 -- start start 2024-03-18T22:57:59.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.334+0000 7fea35fa4740 1 Processor -- start 2024-03-18T22:57:59.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.335+0000 7fea35fa4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe2f7670 0x560bbe31f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.335+0000 7fea35fa4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe31fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.335+0000 7fea35fa4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe305670 0x560bbe38ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.335+0000 7fea35fa4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c1a10d0 con 0x560bbe305670 2024-03-18T22:57:59.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.335+0000 7fea35fa4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c1a09d0 con 0x560bbe2f7fd0 2024-03-18T22:57:59.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.335+0000 7fea35fa4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c073610 con 0x560bbe2f7670 2024-03-18T22:57:59.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.335+0000 7fea35fa4740 10 client.0 _ll_register_callbacks cb 0x7fea1c00bad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:57:59.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.335+0000 7fea31155640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe31fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.335+0000 7fea30954640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe2f7670 0x560bbe31f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.335+0000 7fea31155640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe31fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:60478/0 (socket says 172.21.15.40:60478) 2024-03-18T22:57:59.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.335+0000 7fea31155640 1 -- 172.21.15.40:0/2194243485 learned_addr learned my addr 172.21.15.40:0/2194243485 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:57:59.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.335+0000 7fea30954640 1 --2- 172.21.15.40:0/2194243485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe305670 0x560bbe38ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.336+0000 7fea30954640 1 -- 172.21.15.40:0/2194243485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 msgr2=0x560bbe31fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:59.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.336+0000 7fea30954640 1 --2- 172.21.15.40:0/2194243485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe31fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.336+0000 7fea30954640 1 -- 172.21.15.40:0/2194243485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe305670 msgr2=0x560bbe38ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:57:59.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.336+0000 7fea30954640 1 --2- 172.21.15.40:0/2194243485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe305670 0x560bbe38ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.336+0000 7fea30954640 1 -- 172.21.15.40:0/2194243485 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bbe38b300 con 0x560bbe2f7670 2024-03-18T22:57:59.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.336+0000 7fea30954640 1 --2- 172.21.15.40:0/2194243485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe2f7670 0x560bbe31f5d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0124d0 tx=0x7fea1c00b480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feecaddc4d65eedd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:57:59.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.336+0000 7fea14ff9640 10 client.0 ms_handle_connect on v2:172.21.15.113:3300/0 2024-03-18T22:57:59.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.336+0000 7fea14ff9640 1 -- 172.21.15.40:0/2194243485 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c012250 con 0x560bbe2f7670 2024-03-18T22:57:59.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.336+0000 7fea35fa4740 1 -- 172.21.15.40:0/2194243485 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea1c020520 con 0x560bbe2f7670 2024-03-18T22:57:59.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.336+0000 7fea14ff9640 1 -- 172.21.15.40:0/2194243485 <== mon.1 v2:172.21.15.113:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fea1c035030 con 0x560bbe2f7670 2024-03-18T22:57:59.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.336+0000 7fea14ff9640 1 -- 172.21.15.40:0/2194243485 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fea1c012560 con 0x560bbe2f7670 2024-03-18T22:57:59.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.336+0000 7fea35fa4740 10 client.28510 fetch_fsmap learned FSMap version 46 2024-03-18T22:57:59.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.336+0000 7fea35fa4740 1 -- 172.21.15.40:0/2194243485 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x560bbe38c5e0 con 0x560bbe2f7670 2024-03-18T22:57:59.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.337+0000 7fea14ff9640 1 -- 172.21.15.40:0/2194243485 <== mon.1 v2:172.21.15.113:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea1c011440 con 0x560bbe2f7670 2024-03-18T22:57:59.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.337+0000 7fea14ff9640 1 -- 172.21.15.40:0/2194243485 <== mon.1 v2:172.21.15.113:3300/0 5 ==== fsmap(e 46) ==== 8406+0+0 (secure 0 0 0) 0x7fea1c02b3c0 con 0x560bbe2f7670 2024-03-18T22:57:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.337+0000 7fea35fa4740 10 client.28510 fetch_fsmap finished waiting for FSMap version 46 2024-03-18T22:57:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.337+0000 7fea35fa4740 10 client.28510 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:57:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.337+0000 7fea35fa4740 20 client.28510 populate_metadata read hostname 'smithi040' 2024-03-18T22:57:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.337+0000 7fea35fa4740 1 --2- 172.21.15.40:0/2194243485 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560bbe38cef0 0x560bbe38f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:57:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.337+0000 7fea35fa4740 4 client.28510 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:57:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.337+0000 7fea35fa4740 1 -- 172.21.15.40:0/2194243485 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fea1c020520 con 0x560bbe38cef0 2024-03-18T22:57:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.338+0000 7fea31155640 1 --2- 172.21.15.40:0/2194243485 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560bbe38cef0 0x560bbe38f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:57:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.338+0000 7fea31155640 1 --2- 172.21.15.40:0/2194243485 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560bbe38cef0 0x560bbe38f310 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40c3ea7df9a149ce server_cookie=6b1e2ffae94905ad in_seq=0 out_seq=0 2024-03-18T22:57:59.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.338+0000 7fea14ff9640 10 client.28510 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.342+0000 7fea14ff9640 1 -- 172.21.15.40:0/2194243485 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fea1c020520 con 0x560bbe38cef0 2024-03-18T22:57:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.342+0000 7fea14ff9640 10 client.28510 handle_command_reply: tid=0 2024-03-18T22:57:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.342+0000 7fea35fa4740 1 client.28510 shutdown 2024-03-18T22:57:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.342+0000 7fea35fa4740 1 -- 172.21.15.40:0/2194243485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe2f7670 msgr2=0x560bbe31f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:59.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.342+0000 7fea35fa4740 1 --2- 172.21.15.40:0/2194243485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe2f7670 0x560bbe31f5d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0124d0 tx=0x7fea1c00b480 comp rx=0 tx=0).stop 2024-03-18T22:57:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.342+0000 7fea35fa4740 1 -- 172.21.15.40:0/2194243485 shutdown_connections 2024-03-18T22:57:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.342+0000 7fea35fa4740 1 --2- 172.21.15.40:0/2194243485 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560bbe305670 0x560bbe38ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.342+0000 7fea35fa4740 1 --2- 172.21.15.40:0/2194243485 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x560bbe38cef0 0x560bbe38f310 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.342+0000 7fea35fa4740 1 --2- 172.21.15.40:0/2194243485 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560bbe2f7fd0 0x560bbe31fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.342+0000 7fea14ff9640 0 client.28510 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:57:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.342+0000 7fea35fa4740 1 --2- 172.21.15.40:0/2194243485 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x560bbe2f7670 0x560bbe31f5d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.342+0000 7fea35fa4740 1 -- 172.21.15.40:0/2194243485 >> 172.21.15.40:0/2194243485 conn(0x560bbe2f3e20 msgr2=0x560bbe2f1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.342+0000 7fea35fa4740 1 -- 172.21.15.40:0/2194243485 shutdown_connections 2024-03-18T22:57:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.342+0000 7fea35fa4740 1 -- 172.21.15.40:0/2194243485 wait complete. 2024-03-18T22:57:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.342+0000 7fea35fa4740 20 client.28510 trim_cache size 0 max 16384 2024-03-18T22:57:59.347 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:57:59.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.343+0000 7fea333b9640 1 -- 172.21.15.40:0/1726793376 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fea180a2900 msgr2=0x7fea180a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:59.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.343+0000 7fea333b9640 1 --2- 172.21.15.40:0/1726793376 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fea180a2900 0x7fea180a4dc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fea240a4120 tx=0x7fea2400c890 comp rx=0 tx=0).stop 2024-03-18T22:57:59.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.343+0000 7fea333b9640 1 -- 172.21.15.40:0/1726793376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea2c1019a0 msgr2=0x7fea2c19cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:57:59.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.343+0000 7fea333b9640 1 --2- 172.21.15.40:0/1726793376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea2c1019a0 0x7fea2c19cdd0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fea24009fd0 tx=0x7fea2400ef10 comp rx=0 tx=0).stop 2024-03-18T22:57:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.343+0000 7fea333b9640 1 -- 172.21.15.40:0/1726793376 shutdown_connections 2024-03-18T22:57:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.343+0000 7fea333b9640 1 --2- 172.21.15.40:0/1726793376 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fea180a2900 0x7fea180a4dc0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.343+0000 7fea333b9640 1 --2- 172.21.15.40:0/1726793376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fea2c1019a0 0x7fea2c19cdd0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.343+0000 7fea333b9640 1 --2- 172.21.15.40:0/1726793376 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fea2c100fe0 0x7fea2c195d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.343+0000 7fea333b9640 1 --2- 172.21.15.40:0/1726793376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea2c0ffe90 0x7fea2c195820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:57:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.343+0000 7fea333b9640 1 -- 172.21.15.40:0/1726793376 >> 172.21.15.40:0/1726793376 conn(0x7fea2c0fb400 msgr2=0x7fea2c1043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:57:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.343+0000 7fea333b9640 1 -- 172.21.15.40:0/1726793376 shutdown_connections 2024-03-18T22:57:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:57:59.343+0000 7fea333b9640 1 -- 172.21.15.40:0/1726793376 wait complete. 2024-03-18T22:57:59.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:57:59 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:59.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:57:59 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:57:59.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:57:59 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:58:00.356 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T22:58:00.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:58:00.357 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T22:58:00.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.519+0000 7f219287b640 1 -- 172.21.15.40:0/1737760648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f218c108ef0 msgr2=0x7f218c109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.519+0000 7f219287b640 1 --2- 172.21.15.40:0/1737760648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f218c108ef0 0x7f218c109370 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f217c009900 tx=0x7f217c02bc60 comp rx=0 tx=0).stop 2024-03-18T22:58:00.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.523+0000 7f219287b640 1 -- 172.21.15.40:0/1737760648 shutdown_connections 2024-03-18T22:58:00.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.523+0000 7f219287b640 1 --2- 172.21.15.40:0/1737760648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f218c1098b0 0x7f218c10e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.523+0000 7f219287b640 1 --2- 172.21.15.40:0/1737760648 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f218c108ef0 0x7f218c109370 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.523+0000 7f219287b640 1 --2- 172.21.15.40:0/1737760648 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f218c072c30 0x7f218c073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.523+0000 7f219287b640 1 -- 172.21.15.40:0/1737760648 >> 172.21.15.40:0/1737760648 conn(0x7f218c06e700 msgr2=0x7f218c070b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:00.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.527+0000 7f219287b640 1 -- 172.21.15.40:0/1737760648 shutdown_connections 2024-03-18T22:58:00.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.528+0000 7f219287b640 1 -- 172.21.15.40:0/1737760648 wait complete. 2024-03-18T22:58:00.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.529+0000 7f219287b640 1 Processor -- start 2024-03-18T22:58:00.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.529+0000 7f219287b640 1 -- start start 2024-03-18T22:58:00.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.530+0000 7f219287b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f218c072c30 0x7f218c1a6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.530+0000 7f219287b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f218c108ef0 0x7f218c1a6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.530+0000 7f219287b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f218c1098b0 0x7f218c1adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.530+0000 7f219287b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f218c11c0f0 con 0x7f218c072c30 2024-03-18T22:58:00.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.530+0000 7f219287b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f218c11bf70 con 0x7f218c1098b0 2024-03-18T22:58:00.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.530+0000 7f219287b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f218c11c270 con 0x7f218c108ef0 2024-03-18T22:58:00.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.532+0000 7f2191879640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f218c072c30 0x7f218c1a6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.532+0000 7f2191078640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f218c108ef0 0x7f218c1a6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.532+0000 7f2191879640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f218c072c30 0x7f218c1a6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57316/0 (socket says 172.21.15.40:57316) 2024-03-18T22:58:00.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.532+0000 7f2191879640 1 -- 172.21.15.40:0/2705399275 learned_addr learned my addr 172.21.15.40:0/2705399275 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:00.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.536+0000 7f2191078640 1 -- 172.21.15.40:0/2705399275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f218c1098b0 msgr2=0x7f218c1adcc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T22:58:00.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.536+0000 7f2191078640 1 --2- 172.21.15.40:0/2705399275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f218c1098b0 0x7f218c1adcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.536+0000 7f2191078640 1 -- 172.21.15.40:0/2705399275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f218c072c30 msgr2=0x7f218c1a6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.536+0000 7f2191078640 1 --2- 172.21.15.40:0/2705399275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f218c072c30 0x7f218c1a6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.536+0000 7f2191078640 1 -- 172.21.15.40:0/2705399275 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218c110080 con 0x7f218c108ef0 2024-03-18T22:58:00.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.537+0000 7f2191078640 1 --2- 172.21.15.40:0/2705399275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f218c108ef0 0x7f218c1a6c50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f217c036040 tx=0x7f217c0045c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:00.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.539+0000 7f2182ffd640 1 -- 172.21.15.40:0/2705399275 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c042020 con 0x7f218c108ef0 2024-03-18T22:58:00.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.539+0000 7f219287b640 1 -- 172.21.15.40:0/2705399275 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f218c110310 con 0x7f218c108ef0 2024-03-18T22:58:00.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.539+0000 7f219287b640 1 -- 172.21.15.40:0/2705399275 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f218c110800 con 0x7f218c108ef0 2024-03-18T22:58:00.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.539+0000 7f2182ffd640 1 -- 172.21.15.40:0/2705399275 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f217c005350 con 0x7f218c108ef0 2024-03-18T22:58:00.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.540+0000 7f2182ffd640 1 -- 172.21.15.40:0/2705399275 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c03e770 con 0x7f218c108ef0 2024-03-18T22:58:00.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.543+0000 7f219679d740 1 Processor -- start 2024-03-18T22:58:00.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.543+0000 7f219679d740 1 -- start start 2024-03-18T22:58:00.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.544+0000 7f219679d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55580 0x5567a2d559a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.544+0000 7f219679d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d55f70 0x5567a2d5aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.544+0000 7f219679d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d5b0e0 0x5567a2d5d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.544+0000 7f219679d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5567a2cba160 con 0x5567a2d5b0e0 2024-03-18T22:58:00.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.544+0000 7f219679d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5567a2c62e10 con 0x5567a2d55f70 2024-03-18T22:58:00.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.544+0000 7f219679d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5567a2cb6200 con 0x5567a2d55580 2024-03-18T22:58:00.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.544+0000 7f219207a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55580 0x5567a2d559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.544+0000 7f219207a640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55580 0x5567a2d559a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55410/0 (socket says 172.21.15.40:55410) 2024-03-18T22:58:00.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.544+0000 7f219207a640 1 -- 172.21.15.40:0/3547985678 learned_addr learned my addr 172.21.15.40:0/3547985678 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.545+0000 7f219207a640 1 -- 172.21.15.40:0/3547985678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d55f70 msgr2=0x5567a2d5aa70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T22:58:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.545+0000 7f219207a640 1 --2- 172.21.15.40:0/3547985678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d55f70 0x5567a2d5aa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.545+0000 7f219207a640 1 -- 172.21.15.40:0/3547985678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d5b0e0 msgr2=0x5567a2d5d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.545+0000 7f219207a640 1 --2- 172.21.15.40:0/3547985678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d5b0e0 0x5567a2d5d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.545+0000 7f219207a640 1 -- 172.21.15.40:0/3547985678 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567a2a967e0 con 0x5567a2d55580 2024-03-18T22:58:00.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.546+0000 7f2182ffd640 1 -- 172.21.15.40:0/2705399275 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f217c045020 con 0x7f218c108ef0 2024-03-18T22:58:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.547+0000 7f2182ffd640 1 --2- 172.21.15.40:0/2705399275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21780a2a10 0x7f21780a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.547+0000 7f219207a640 1 --2- 172.21.15.40:0/3547985678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55580 0x5567a2d559a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f21880096f0 tx=0x7f2188009bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c3379e4fe43d6f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:00.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.547+0000 7f2182ffd640 1 -- 172.21.15.40:0/2705399275 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f217c0f3210 con 0x7f218c108ef0 2024-03-18T22:58:00.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.552+0000 7f2191879640 1 --2- 172.21.15.40:0/2705399275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21780a2a10 0x7f21780a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.552+0000 7f2177fff640 1 -- 172.21.15.40:0/3547985678 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2188015020 con 0x5567a2d55580 2024-03-18T22:58:00.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.552+0000 7f2177fff640 1 -- 172.21.15.40:0/3547985678 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f218800bd80 con 0x5567a2d55580 2024-03-18T22:58:00.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.552+0000 7f2177fff640 1 -- 172.21.15.40:0/3547985678 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2188010380 con 0x5567a2d55580 2024-03-18T22:58:00.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.553+0000 7f219679d740 1 -- 172.21.15.40:0/3547985678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55580 msgr2=0x5567a2d559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.553+0000 7f219679d740 1 --2- 172.21.15.40:0/3547985678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55580 0x5567a2d559a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f21880096f0 tx=0x7f2188009bb0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.553+0000 7f219679d740 1 -- 172.21.15.40:0/3547985678 shutdown_connections 2024-03-18T22:58:00.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.553+0000 7f219679d740 1 --2- 172.21.15.40:0/3547985678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d5b0e0 0x5567a2d5d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.553+0000 7f219679d740 1 --2- 172.21.15.40:0/3547985678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d55f70 0x5567a2d5aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.554+0000 7f219679d740 1 --2- 172.21.15.40:0/3547985678 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55580 0x5567a2d559a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.554+0000 7f219679d740 1 -- 172.21.15.40:0/3547985678 >> 172.21.15.40:0/3547985678 conn(0x5567a2d50aa0 msgr2=0x5567a2d50ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:00.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.554+0000 7f219679d740 1 -- 172.21.15.40:0/3547985678 shutdown_connections 2024-03-18T22:58:00.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.554+0000 7f219679d740 1 -- 172.21.15.40:0/3547985678 wait complete. 2024-03-18T22:58:00.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.554+0000 7f219679d740 1 -- start start 2024-03-18T22:58:00.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.554+0000 7f219679d740 1 Processor -- start 2024-03-18T22:58:00.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.555+0000 7f2191879640 1 --2- 172.21.15.40:0/2705399275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21780a2a10 0x7f21780a4ed0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x5567a2d51a60 tx=0x7f2184007320 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:00.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.555+0000 7f219679d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d55580 0x5567a2d5fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.555+0000 7f219679d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d55f70 0x5567a2d60010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.555+0000 7f219679d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d5b0e0 0x5567a2d67050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.555+0000 7f219679d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x5567a2d5e090 con 0x5567a2d55580 2024-03-18T22:58:00.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.555+0000 7f219679d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5567a2c761a0 con 0x5567a2d55f70 2024-03-18T22:58:00.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.555+0000 7f219679d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x5567a2d5e210 con 0x5567a2d5b0e0 2024-03-18T22:58:00.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.555+0000 7f219679d740 10 client.0 _ll_register_callbacks cb 0x5567a29cdc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:00.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.556+0000 7f2191879640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d55580 0x5567a2d5fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.556+0000 7f2191879640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d55580 0x5567a2d5fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57338/0 (socket says 172.21.15.40:57338) 2024-03-18T22:58:00.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.556+0000 7f2191879640 1 -- 172.21.15.40:0/1401867299 learned_addr learned my addr 172.21.15.40:0/1401867299 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:00.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.556+0000 7f219207a640 1 --2- 172.21.15.40:0/1401867299 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d5b0e0 0x5567a2d67050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.556+0000 7f2191879640 1 -- 172.21.15.40:0/1401867299 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d5b0e0 msgr2=0x5567a2d67050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:00.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.556+0000 7f2191879640 1 --2- 172.21.15.40:0/1401867299 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d5b0e0 0x5567a2d67050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.556+0000 7f2191879640 1 -- 172.21.15.40:0/1401867299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d55f70 msgr2=0x5567a2d60010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T22:58:00.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.556+0000 7f2191879640 1 --2- 172.21.15.40:0/1401867299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d55f70 0x5567a2d60010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.556+0000 7f2191879640 1 -- 172.21.15.40:0/1401867299 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567a2af6f70 con 0x5567a2d55580 2024-03-18T22:58:00.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.557+0000 7f2191879640 1 --2- 172.21.15.40:0/1401867299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d55580 0x5567a2d5fad0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f2184010e20 tx=0x7f2184004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ee97ad5a990a88 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:00.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.557+0000 7f21757fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:58:00.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.557+0000 7f21757fa640 1 -- 172.21.15.40:0/1401867299 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2184021070 con 0x5567a2d55580 2024-03-18T22:58:00.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.558+0000 7f219679d740 1 -- 172.21.15.40:0/1401867299 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f218800bd80 con 0x5567a2d55580 2024-03-18T22:58:00.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.558+0000 7f21757fa640 1 -- 172.21.15.40:0/1401867299 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2184004b20 con 0x5567a2d55580 2024-03-18T22:58:00.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.558+0000 7f21757fa640 1 -- 172.21.15.40:0/1401867299 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f218402c050 con 0x5567a2d55580 2024-03-18T22:58:00.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.558+0000 7f219679d740 10 client.18711 fetch_fsmap learned FSMap version 48 2024-03-18T22:58:00.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.558+0000 7f219679d740 1 -- 172.21.15.40:0/1401867299 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5567a298e170 con 0x5567a2d55580 2024-03-18T22:58:00.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.559+0000 7f21757fa640 1 -- 172.21.15.40:0/1401867299 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f218800bd80 con 0x5567a2d55580 2024-03-18T22:58:00.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.560+0000 7f21757fa640 1 -- 172.21.15.40:0/1401867299 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 48) ==== 8414+0+0 (secure 0 0 0) 0x7f218401c550 con 0x5567a2d55580 2024-03-18T22:58:00.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.560+0000 7f219679d740 10 client.18711 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T22:58:00.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.560+0000 7f219679d740 10 client.18711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:58:00.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.560+0000 7f219679d740 20 client.18711 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:00.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.560+0000 7f219679d740 1 --2- 172.21.15.40:0/1401867299 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5567a2d6bef0 0x5567a2d6e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.560+0000 7f219679d740 4 client.18711 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:58:00.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.560+0000 7f219679d740 1 -- 172.21.15.40:0/1401867299 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567a2cc76a0 con 0x5567a2d6bef0 2024-03-18T22:58:00.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.562+0000 7f2191078640 1 --2- 172.21.15.40:0/1401867299 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5567a2d6bef0 0x5567a2d6e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.563+0000 7f2191078640 1 --2- 172.21.15.40:0/1401867299 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5567a2d6bef0 0x5567a2d6e3b0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7d4d446ec297176 server_cookie=c3fbd9a887317dba in_seq=0 out_seq=0 2024-03-18T22:58:00.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.563+0000 7f21757fa640 10 client.18711 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.564+0000 7f21757fa640 1 -- 172.21.15.40:0/1401867299 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5567a2cc76a0 con 0x5567a2d6bef0 2024-03-18T22:58:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.564+0000 7f21757fa640 10 client.18711 handle_command_reply: tid=0 2024-03-18T22:58:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.564+0000 7f219679d740 1 client.18711 shutdown 2024-03-18T22:58:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.565+0000 7f219679d740 1 -- 172.21.15.40:0/1401867299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d55580 msgr2=0x5567a2d5fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.565+0000 7f219679d740 1 --2- 172.21.15.40:0/1401867299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d55580 0x5567a2d5fad0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f2184010e20 tx=0x7f2184004a40 comp rx=0 tx=0).stop 2024-03-18T22:58:00.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.565+0000 7f219679d740 1 -- 172.21.15.40:0/1401867299 shutdown_connections 2024-03-18T22:58:00.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.565+0000 7f219679d740 1 --2- 172.21.15.40:0/1401867299 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d5b0e0 0x5567a2d67050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.565+0000 7f219679d740 1 --2- 172.21.15.40:0/1401867299 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5567a2d6bef0 0x5567a2d6e3b0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.565+0000 7f219679d740 1 --2- 172.21.15.40:0/1401867299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d55f70 0x5567a2d60010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.565+0000 7f219679d740 1 --2- 172.21.15.40:0/1401867299 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d55580 0x5567a2d5fad0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.565+0000 7f219679d740 1 -- 172.21.15.40:0/1401867299 >> 172.21.15.40:0/1401867299 conn(0x5567a2d50aa0 msgr2=0x5567a2d4fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:00.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.565+0000 7f21757fa640 0 client.18711 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:00.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.565+0000 7f219679d740 1 -- 172.21.15.40:0/1401867299 shutdown_connections 2024-03-18T22:58:00.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.565+0000 7f219679d740 1 -- 172.21.15.40:0/1401867299 wait complete. 2024-03-18T22:58:00.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.565+0000 7f219679d740 20 client.18711 trim_cache size 0 max 16384 2024-03-18T22:58:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.599+0000 7f219679d740 1 Processor -- start 2024-03-18T22:58:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.599+0000 7f219679d740 1 -- start start 2024-03-18T22:58:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.599+0000 7f219679d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d55580 0x5567a2d5fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.599+0000 7f219679d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55f70 0x5567a2d67010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.599+0000 7f219679d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d5b0e0 0x5567a2d4f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.599+0000 7f219679d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f217000fd40 con 0x5567a2d5b0e0 2024-03-18T22:58:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.599+0000 7f219679d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2170011a60 con 0x5567a2d55580 2024-03-18T22:58:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.599+0000 7f219679d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f2170010410 con 0x5567a2d55f70 2024-03-18T22:58:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.599+0000 7f219207a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d5b0e0 0x5567a2d4f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.599+0000 7f219207a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d5b0e0 0x5567a2d4f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57348/0 (socket says 172.21.15.40:57348) 2024-03-18T22:58:00.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.599+0000 7f219207a640 1 -- 172.21.15.40:0/3440148662 learned_addr learned my addr 172.21.15.40:0/3440148662 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:00.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.600+0000 7f2191879640 1 --2- 172.21.15.40:0/3440148662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d55580 0x5567a2d5fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.600+0000 7f219207a640 1 -- 172.21.15.40:0/3440148662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55f70 msgr2=0x5567a2d67010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:00.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.600+0000 7f219207a640 1 --2- 172.21.15.40:0/3440148662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55f70 0x5567a2d67010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.600+0000 7f219207a640 1 -- 172.21.15.40:0/3440148662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d55580 msgr2=0x5567a2d5fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:00.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.600+0000 7f219207a640 1 --2- 172.21.15.40:0/3440148662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d55580 0x5567a2d5fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.600+0000 7f219207a640 1 -- 172.21.15.40:0/3440148662 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567a2d56350 con 0x5567a2d5b0e0 2024-03-18T22:58:00.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.600+0000 7f219207a640 1 --2- 172.21.15.40:0/3440148662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d5b0e0 0x5567a2d4f470 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f2188009eb0 tx=0x7f2188009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d54f76eb542dae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:00.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.603+0000 7f21777fe640 1 -- 172.21.15.40:0/3440148662 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2188015050 con 0x5567a2d5b0e0 2024-03-18T22:58:00.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.604+0000 7f21777fe640 1 -- 172.21.15.40:0/3440148662 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2188011df0 con 0x5567a2d5b0e0 2024-03-18T22:58:00.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.604+0000 7f21777fe640 1 -- 172.21.15.40:0/3440148662 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21880042a0 con 0x5567a2d5b0e0 2024-03-18T22:58:00.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.605+0000 7f219679d740 1 -- 172.21.15.40:0/3440148662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d5b0e0 msgr2=0x5567a2d4f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.605+0000 7f219679d740 1 --2- 172.21.15.40:0/3440148662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d5b0e0 0x5567a2d4f470 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f2188009eb0 tx=0x7f2188009ee0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.606+0000 7f219679d740 1 -- 172.21.15.40:0/3440148662 shutdown_connections 2024-03-18T22:58:00.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.606+0000 7f219679d740 1 --2- 172.21.15.40:0/3440148662 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d5b0e0 0x5567a2d4f470 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.606+0000 7f219679d740 1 --2- 172.21.15.40:0/3440148662 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55f70 0x5567a2d67010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.606+0000 7f219679d740 1 --2- 172.21.15.40:0/3440148662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d55580 0x5567a2d5fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.606+0000 7f219679d740 1 -- 172.21.15.40:0/3440148662 >> 172.21.15.40:0/3440148662 conn(0x5567a2d50aa0 msgr2=0x5567a2d5f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:00.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.612+0000 7f219679d740 1 -- 172.21.15.40:0/3440148662 shutdown_connections 2024-03-18T22:58:00.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.612+0000 7f219679d740 1 -- 172.21.15.40:0/3440148662 wait complete. 2024-03-18T22:58:00.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.612+0000 7f219679d740 1 -- start start 2024-03-18T22:58:00.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.612+0000 7f219679d740 1 Processor -- start 2024-03-18T22:58:00.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.613+0000 7f219679d740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55580 0x5567a2d64870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.613+0000 7f219679d740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d55f70 0x5567a2d64db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.613+0000 7f219679d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d5b0e0 0x5567a2d65d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.613+0000 7f219679d740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f21700112d0 con 0x5567a2d55f70 2024-03-18T22:58:00.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.613+0000 7f219679d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2170010ba0 con 0x5567a2d5b0e0 2024-03-18T22:58:00.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.613+0000 7f219679d740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f217000dd70 con 0x5567a2d55580 2024-03-18T22:58:00.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.613+0000 7f219679d740 10 client.0 _ll_register_callbacks cb 0x7f2184014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:00.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.613+0000 7f2191879640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55580 0x5567a2d64870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.613+0000 7f2191879640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55580 0x5567a2d64870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:55448/0 (socket says 172.21.15.40:55448) 2024-03-18T22:58:00.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.613+0000 7f2191879640 1 -- 172.21.15.40:0/3898203859 learned_addr learned my addr 172.21.15.40:0/3898203859 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:00.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.613+0000 7f219207a640 1 --2- 172.21.15.40:0/3898203859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d5b0e0 0x5567a2d65d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.614+0000 7f2191078640 1 --2- 172.21.15.40:0/3898203859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d55f70 0x5567a2d64db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.614+0000 7f2191078640 1 -- 172.21.15.40:0/3898203859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55580 msgr2=0x5567a2d64870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:00.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.614+0000 7f2191078640 1 --2- 172.21.15.40:0/3898203859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55580 0x5567a2d64870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.614+0000 7f2191078640 1 -- 172.21.15.40:0/3898203859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d5b0e0 msgr2=0x5567a2d65d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:00.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.614+0000 7f2191078640 1 --2- 172.21.15.40:0/3898203859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d5b0e0 0x5567a2d65d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.614+0000 7f2191078640 1 -- 172.21.15.40:0/3898203859 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567a2d662f0 con 0x5567a2d55f70 2024-03-18T22:58:00.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.616+0000 7f2191078640 1 --2- 172.21.15.40:0/3898203859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d55f70 0x5567a2d64db0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x5567a2d60fb0 tx=0x7f217c02efe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659cf96cb3044f56 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:00.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.616+0000 7f21757fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:58:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.622+0000 7f21757fa640 1 -- 172.21.15.40:0/3898203859 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217c02cc10 con 0x5567a2d55f70 2024-03-18T22:58:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.622+0000 7f21757fa640 1 -- 172.21.15.40:0/3898203859 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f217c0f09d0 con 0x5567a2d55f70 2024-03-18T22:58:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.623+0000 7f219679d740 1 -- 172.21.15.40:0/3898203859 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2188011df0 con 0x5567a2d55f70 2024-03-18T22:58:00.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.624+0000 7f21757fa640 1 -- 172.21.15.40:0/3898203859 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f217c0bfe60 con 0x5567a2d55f70 2024-03-18T22:58:00.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.629+0000 7f219679d740 10 client.18717 fetch_fsmap learned FSMap version 48 2024-03-18T22:58:00.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.629+0000 7f219679d740 1 -- 172.21.15.40:0/3898203859 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5567a2dd2450 con 0x5567a2d55f70 2024-03-18T22:58:00.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.631+0000 7f21757fa640 1 -- 172.21.15.40:0/3898203859 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2188011df0 con 0x5567a2d55f70 2024-03-18T22:58:00.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.631+0000 7f21757fa640 1 -- 172.21.15.40:0/3898203859 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 48) ==== 8414+0+0 (secure 0 0 0) 0x7f217c0bbdd0 con 0x5567a2d55f70 2024-03-18T22:58:00.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.631+0000 7f219679d740 10 client.18717 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T22:58:00.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.631+0000 7f219679d740 10 client.18717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:58:00.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.631+0000 7f219679d740 20 client.18717 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:00.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.631+0000 7f219679d740 1 --2- 172.21.15.40:0/3898203859 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5567a2dd2f90 0x5567a2dd53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.631+0000 7f219679d740 4 client.18717 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T22:58:00.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.631+0000 7f219679d740 1 -- 172.21.15.40:0/3898203859 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5567a2cc19f0 con 0x5567a2dd2f90 2024-03-18T22:58:00.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.631+0000 7f2191879640 1 --2- 172.21.15.40:0/3898203859 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5567a2dd2f90 0x5567a2dd53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.633+0000 7f2191879640 1 --2- 172.21.15.40:0/3898203859 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5567a2dd2f90 0x5567a2dd53b0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ab1ec54a35fe7bc server_cookie=371efba0c9b8db61 in_seq=0 out_seq=0 2024-03-18T22:58:00.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.633+0000 7f21757fa640 10 client.18717 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.639+0000 7f21757fa640 1 -- 172.21.15.40:0/3898203859 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5567a2cc19f0 con 0x5567a2dd2f90 2024-03-18T22:58:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.639+0000 7f21757fa640 10 client.18717 handle_command_reply: tid=0 2024-03-18T22:58:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.639+0000 7f219679d740 1 client.18717 shutdown 2024-03-18T22:58:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.639+0000 7f219679d740 1 -- 172.21.15.40:0/3898203859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d55f70 msgr2=0x5567a2d64db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.639+0000 7f219679d740 1 --2- 172.21.15.40:0/3898203859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d55f70 0x5567a2d64db0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x5567a2d60fb0 tx=0x7f217c02efe0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.639+0000 7f219679d740 1 -- 172.21.15.40:0/3898203859 shutdown_connections 2024-03-18T22:58:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.639+0000 7f219679d740 1 --2- 172.21.15.40:0/3898203859 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x5567a2dd2f90 0x5567a2dd53b0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.639+0000 7f219679d740 1 --2- 172.21.15.40:0/3898203859 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5567a2d5b0e0 0x5567a2d65d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.639+0000 7f219679d740 1 --2- 172.21.15.40:0/3898203859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5567a2d55f70 0x5567a2d64db0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.639+0000 7f219679d740 1 --2- 172.21.15.40:0/3898203859 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x5567a2d55580 0x5567a2d64870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.639+0000 7f21757fa640 0 client.18717 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.639+0000 7f219679d740 1 -- 172.21.15.40:0/3898203859 >> 172.21.15.40:0/3898203859 conn(0x5567a2d50aa0 msgr2=0x5567a2d53d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.640+0000 7f219679d740 1 -- 172.21.15.40:0/3898203859 shutdown_connections 2024-03-18T22:58:00.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.640+0000 7f219679d740 1 -- 172.21.15.40:0/3898203859 wait complete. 2024-03-18T22:58:00.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.640+0000 7f219679d740 20 client.18717 trim_cache size 0 max 16384 2024-03-18T22:58:00.642 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:58:00.642 INFO:teuthology.orchestra.run.smithi040.stdout: "return_code": 0, 2024-03-18T22:58:00.642 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_tag": "bb665ba0-d486-4abd-b75e-c3b21fdcd107", 2024-03-18T22:58:00.643 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": "asynchronous" 2024-03-18T22:58:00.643 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:58:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.641+0000 7f219287b640 1 -- 172.21.15.40:0/2705399275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21780a2a10 msgr2=0x7f21780a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.641+0000 7f219287b640 1 --2- 172.21.15.40:0/2705399275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21780a2a10 0x7f21780a4ed0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x5567a2d51a60 tx=0x7f2184007320 comp rx=0 tx=0).stop 2024-03-18T22:58:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.641+0000 7f219287b640 1 -- 172.21.15.40:0/2705399275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f218c108ef0 msgr2=0x7f218c1a6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.641+0000 7f219287b640 1 --2- 172.21.15.40:0/2705399275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f218c108ef0 0x7f218c1a6c50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f217c036040 tx=0x7f217c0045c0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.641+0000 7f219287b640 1 -- 172.21.15.40:0/2705399275 shutdown_connections 2024-03-18T22:58:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.641+0000 7f219287b640 1 --2- 172.21.15.40:0/2705399275 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f21780a2a10 0x7f21780a4ed0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.641+0000 7f219287b640 1 --2- 172.21.15.40:0/2705399275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f218c1098b0 0x7f218c1adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.642+0000 7f219287b640 1 --2- 172.21.15.40:0/2705399275 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f218c108ef0 0x7f218c1a6c50 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.642+0000 7f219287b640 1 --2- 172.21.15.40:0/2705399275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f218c072c30 0x7f218c1a6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.642+0000 7f219287b640 1 -- 172.21.15.40:0/2705399275 >> 172.21.15.40:0/2705399275 conn(0x7f218c06e700 msgr2=0x7f218c070340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:00.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:00 smithi040 ceph-mon[62643]: pgmap v811: 129 pgs: 129 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 70 KiB/s wr, 171 op/s 2024-03-18T22:58:00.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:00 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:58:00.645 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:00 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:58:00.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.642+0000 7f219287b640 1 -- 172.21.15.40:0/2705399275 shutdown_connections 2024-03-18T22:58:00.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.642+0000 7f219287b640 1 -- 172.21.15.40:0/2705399275 wait complete. 2024-03-18T22:58:00.659 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:58:00.659 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:58:00.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:00 smithi113 ceph-mon[66775]: pgmap v811: 129 pgs: 129 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 70 KiB/s wr, 171 op/s 2024-03-18T22:58:00.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:00 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:58:00.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:00 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:58:00.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.836+0000 7f9d8a159640 1 -- 172.21.15.40:0/1835092137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84100720 msgr2=0x7f9d84100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.836+0000 7f9d8a159640 1 --2- 172.21.15.40:0/1835092137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84100720 0x7f9d84100b40 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009900 tx=0x7f9d7402bc30 comp rx=0 tx=0).stop 2024-03-18T22:58:00.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.836+0000 7f9d8a159640 1 -- 172.21.15.40:0/1835092137 shutdown_connections 2024-03-18T22:58:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.836+0000 7f9d8a159640 1 --2- 172.21.15.40:0/1835092137 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d841022d0 0x7f9d84108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.836+0000 7f9d8a159640 1 --2- 172.21.15.40:0/1835092137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d84101910 0x7f9d84101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.836+0000 7f9d8a159640 1 --2- 172.21.15.40:0/1835092137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84100720 0x7f9d84100b40 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.836+0000 7f9d8a159640 1 -- 172.21.15.40:0/1835092137 >> 172.21.15.40:0/1835092137 conn(0x7f9d840fbcb0 msgr2=0x7f9d840fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.836+0000 7f9d8a159640 1 -- 172.21.15.40:0/1835092137 shutdown_connections 2024-03-18T22:58:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.836+0000 7f9d8a159640 1 -- 172.21.15.40:0/1835092137 wait complete. 2024-03-18T22:58:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.836+0000 7f9d8a159640 1 Processor -- start 2024-03-18T22:58:00.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d8a159640 1 -- start start 2024-03-18T22:58:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d8a159640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84100720 0x7f9d84069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d8a159640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d84101910 0x7f9d8419a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d8a159640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d841022d0 0x7f9d8419ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d8a159640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8410c250 con 0x7f9d84100720 2024-03-18T22:58:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d8a159640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8410c0d0 con 0x7f9d841022d0 2024-03-18T22:58:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d8a159640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8410c3d0 con 0x7f9d84101910 2024-03-18T22:58:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d837fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84100720 0x7f9d84069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d837fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84100720 0x7f9d84069510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57376/0 (socket says 172.21.15.40:57376) 2024-03-18T22:58:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d837fe640 1 -- 172.21.15.40:0/362153990 learned_addr learned my addr 172.21.15.40:0/362153990 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d83fff640 1 --2- 172.21.15.40:0/362153990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d841022d0 0x7f9d8419ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d82ffd640 1 --2- 172.21.15.40:0/362153990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d84101910 0x7f9d8419a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d837fe640 1 -- 172.21.15.40:0/362153990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d84101910 msgr2=0x7f9d8419a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d837fe640 1 --2- 172.21.15.40:0/362153990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d84101910 0x7f9d8419a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d837fe640 1 -- 172.21.15.40:0/362153990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d841022d0 msgr2=0x7f9d8419ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d837fe640 1 --2- 172.21.15.40:0/362153990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d841022d0 0x7f9d8419ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d837fe640 1 -- 172.21.15.40:0/362153990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8419b4f0 con 0x7f9d84100720 2024-03-18T22:58:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d837fe640 1 --2- 172.21.15.40:0/362153990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84100720 0x7f9d84069510 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f9d74000c00 tx=0x7f9d74004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.837+0000 7f9d80ff9640 1 -- 172.21.15.40:0/362153990 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d74043020 con 0x7f9d84100720 2024-03-18T22:58:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.838+0000 7f9d80ff9640 1 -- 172.21.15.40:0/362153990 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d74004440 con 0x7f9d84100720 2024-03-18T22:58:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.838+0000 7f9d80ff9640 1 -- 172.21.15.40:0/362153990 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d74003c90 con 0x7f9d84100720 2024-03-18T22:58:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.838+0000 7f9d8a159640 1 -- 172.21.15.40:0/362153990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d8419b780 con 0x7f9d84100720 2024-03-18T22:58:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.838+0000 7f9d8a159640 1 -- 172.21.15.40:0/362153990 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d8419bc80 con 0x7f9d84100720 2024-03-18T22:58:00.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.840+0000 7f9d80ff9640 1 -- 172.21.15.40:0/362153990 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9d74046020 con 0x7f9d84100720 2024-03-18T22:58:00.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.840+0000 7f9d80ff9640 1 --2- 172.21.15.40:0/362153990 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9d6c0a2880 0x7f9d6c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.840+0000 7f9d80ff9640 1 -- 172.21.15.40:0/362153990 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9d740f1020 con 0x7f9d84100720 2024-03-18T22:58:00.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.840+0000 7f9d82ffd640 1 --2- 172.21.15.40:0/362153990 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9d6c0a2880 0x7f9d6c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.841+0000 7f9d82ffd640 1 --2- 172.21.15.40:0/362153990 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9d6c0a2880 0x7f9d6c0a4d40 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9d78005d00 tx=0x7f9d78005c50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:00.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.841+0000 7f9d8ce16740 1 Processor -- start 2024-03-18T22:58:00.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.841+0000 7f9d8ce16740 1 -- start start 2024-03-18T22:58:00.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.841+0000 7f9d8ce16740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad31a10 0x55e97ad31e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.841+0000 7f9d8ce16740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad32400 0x55e97ad36f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d8ce16740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad37570 0x55e97ad39a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d8ce16740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e97ac96500 con 0x55e97ad32400 2024-03-18T22:58:00.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d8ce16740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e97ac3ebe0 con 0x55e97ad31a10 2024-03-18T22:58:00.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d8ce16740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e97ac925a0 con 0x55e97ad37570 2024-03-18T22:58:00.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d82ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad32400 0x55e97ad36f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d82ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad32400 0x55e97ad36f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57384/0 (socket says 172.21.15.40:57384) 2024-03-18T22:58:00.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d82ffd640 1 -- 172.21.15.40:0/2616378129 learned_addr learned my addr 172.21.15.40:0/2616378129 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:00.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d837fe640 1 --2- 172.21.15.40:0/2616378129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad31a10 0x55e97ad31e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d83fff640 1 --2- 172.21.15.40:0/2616378129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad37570 0x55e97ad39a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d82ffd640 1 -- 172.21.15.40:0/2616378129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad37570 msgr2=0x55e97ad39a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:00.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d82ffd640 1 --2- 172.21.15.40:0/2616378129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad37570 0x55e97ad39a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d82ffd640 1 -- 172.21.15.40:0/2616378129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad31a10 msgr2=0x55e97ad31e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:00.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d82ffd640 1 --2- 172.21.15.40:0/2616378129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad31a10 0x55e97ad31e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d82ffd640 1 -- 172.21.15.40:0/2616378129 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e97ac60000 con 0x55e97ad32400 2024-03-18T22:58:00.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d82ffd640 1 --2- 172.21.15.40:0/2616378129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad32400 0x55e97ad36f00 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f9d7800bf80 tx=0x7f9d78012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2916d0078d0ca61d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:00.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d69ffb640 1 -- 172.21.15.40:0/2616378129 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d78022070 con 0x55e97ad32400 2024-03-18T22:58:00.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d69ffb640 1 -- 172.21.15.40:0/2616378129 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d78011e50 con 0x55e97ad32400 2024-03-18T22:58:00.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.842+0000 7f9d69ffb640 1 -- 172.21.15.40:0/2616378129 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d78010a70 con 0x55e97ad32400 2024-03-18T22:58:00.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.843+0000 7f9d8ce16740 1 -- 172.21.15.40:0/2616378129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad32400 msgr2=0x55e97ad36f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.843+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/2616378129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad32400 0x55e97ad36f00 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f9d7800bf80 tx=0x7f9d78012000 comp rx=0 tx=0).stop 2024-03-18T22:58:00.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.843+0000 7f9d8ce16740 1 -- 172.21.15.40:0/2616378129 shutdown_connections 2024-03-18T22:58:00.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.843+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/2616378129 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad37570 0x55e97ad39a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.843+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/2616378129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad32400 0x55e97ad36f00 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.843+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/2616378129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad31a10 0x55e97ad31e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.843+0000 7f9d8ce16740 1 -- 172.21.15.40:0/2616378129 >> 172.21.15.40:0/2616378129 conn(0x55e97ad2cf30 msgr2=0x55e97ad2d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:00.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.843+0000 7f9d8ce16740 1 -- 172.21.15.40:0/2616378129 shutdown_connections 2024-03-18T22:58:00.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.843+0000 7f9d8ce16740 1 -- 172.21.15.40:0/2616378129 wait complete. 2024-03-18T22:58:00.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.843+0000 7f9d8ce16740 1 -- start start 2024-03-18T22:58:00.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.844+0000 7f9d8ce16740 1 Processor -- start 2024-03-18T22:58:00.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.844+0000 7f9d8ce16740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad31a10 0x55e97ad3be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.844+0000 7f9d8ce16740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad32400 0x55e97ad3c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.844+0000 7f9d8ce16740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad37570 0x55e97ad433e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.844+0000 7f9d8ce16740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55e97ad3a520 con 0x55e97ad31a10 2024-03-18T22:58:00.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.844+0000 7f9d8ce16740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e97ac52160 con 0x55e97ad37570 2024-03-18T22:58:00.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.844+0000 7f9d8ce16740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55e97ad3a6a0 con 0x55e97ad32400 2024-03-18T22:58:00.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.844+0000 7f9d837fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad31a10 0x55e97ad3be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.844+0000 7f9d837fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad31a10 0x55e97ad3be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57394/0 (socket says 172.21.15.40:57394) 2024-03-18T22:58:00.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.844+0000 7f9d837fe640 1 -- 172.21.15.40:0/3464860236 learned_addr learned my addr 172.21.15.40:0/3464860236 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:00.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.844+0000 7f9d83fff640 1 --2- 172.21.15.40:0/3464860236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad37570 0x55e97ad433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.844+0000 7f9d82ffd640 1 --2- 172.21.15.40:0/3464860236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad32400 0x55e97ad3c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.844+0000 7f9d8ce16740 10 client.0 _ll_register_callbacks cb 0x55e97ac8ea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.844+0000 7f9d837fe640 1 -- 172.21.15.40:0/3464860236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad32400 msgr2=0x55e97ad3c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d837fe640 1 --2- 172.21.15.40:0/3464860236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad32400 0x55e97ad3c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d837fe640 1 -- 172.21.15.40:0/3464860236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad37570 msgr2=0x55e97ad433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d837fe640 1 --2- 172.21.15.40:0/3464860236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad37570 0x55e97ad433e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d837fe640 1 -- 172.21.15.40:0/3464860236 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e97aa747d0 con 0x55e97ad31a10 2024-03-18T22:58:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d837fe640 1 --2- 172.21.15.40:0/3464860236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad31a10 0x55e97ad3be60 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f9d74006010 tx=0x7f9d740bd1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236fb6b1e8c84b74 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:58:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d62ffd640 1 -- 172.21.15.40:0/3464860236 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d740d5070 con 0x55e97ad31a10 2024-03-18T22:58:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d62ffd640 1 -- 172.21.15.40:0/3464860236 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9d740c58e0 con 0x55e97ad31a10 2024-03-18T22:58:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d8ce16740 1 -- 172.21.15.40:0/3464860236 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d78011e50 con 0x55e97ad31a10 2024-03-18T22:58:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d62ffd640 1 -- 172.21.15.40:0/3464860236 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f9d740d6460 con 0x55e97ad31a10 2024-03-18T22:58:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d8ce16740 10 client.18729 fetch_fsmap learned FSMap version 48 2024-03-18T22:58:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d8ce16740 1 -- 172.21.15.40:0/3464860236 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55e97a9aa3f0 con 0x55e97ad31a10 2024-03-18T22:58:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d62ffd640 1 -- 172.21.15.40:0/3464860236 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d78011e50 con 0x55e97ad31a10 2024-03-18T22:58:00.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d62ffd640 1 -- 172.21.15.40:0/3464860236 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 48) ==== 8414+0+0 (secure 0 0 0) 0x7f9d740a6b60 con 0x55e97ad31a10 2024-03-18T22:58:00.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d8ce16740 10 client.18729 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T22:58:00.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d8ce16740 10 client.18729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:58:00.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d8ce16740 20 client.18729 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:00.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/3464860236 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e97ad47cf0 0x55e97ad4a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d8ce16740 4 client.18729 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:58:00.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.845+0000 7f9d8ce16740 1 -- 172.21.15.40:0/3464860236 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e97aca3a40 con 0x55e97ad47cf0 2024-03-18T22:58:00.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.849+0000 7f9d82ffd640 1 --2- 172.21.15.40:0/3464860236 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e97ad47cf0 0x55e97ad4a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.849+0000 7f9d82ffd640 1 --2- 172.21.15.40:0/3464860236 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e97ad47cf0 0x55e97ad4a190 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e2bf46fb6cb5f92 server_cookie=d4c879c01843dc40 in_seq=0 out_seq=0 2024-03-18T22:58:00.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.849+0000 7f9d62ffd640 10 client.18729 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.852+0000 7f9d62ffd640 1 -- 172.21.15.40:0/3464860236 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e97aca3a40 con 0x55e97ad47cf0 2024-03-18T22:58:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.852+0000 7f9d62ffd640 10 client.18729 handle_command_reply: tid=0 2024-03-18T22:58:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.852+0000 7f9d8ce16740 1 client.18729 shutdown 2024-03-18T22:58:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.852+0000 7f9d8ce16740 1 -- 172.21.15.40:0/3464860236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad31a10 msgr2=0x55e97ad3be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.852+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/3464860236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad31a10 0x55e97ad3be60 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f9d74006010 tx=0x7f9d740bd1d0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.852+0000 7f9d8ce16740 1 -- 172.21.15.40:0/3464860236 shutdown_connections 2024-03-18T22:58:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.852+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/3464860236 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e97ad47cf0 0x55e97ad4a190 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.852+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/3464860236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad37570 0x55e97ad433e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.852+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/3464860236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad32400 0x55e97ad3c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.852+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/3464860236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad31a10 0x55e97ad3be60 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.852+0000 7f9d62ffd640 0 client.18729 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:00.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.852+0000 7f9d8ce16740 1 -- 172.21.15.40:0/3464860236 >> 172.21.15.40:0/3464860236 conn(0x55e97ad2cf30 msgr2=0x55e97ad2c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:00.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.852+0000 7f9d8ce16740 1 -- 172.21.15.40:0/3464860236 shutdown_connections 2024-03-18T22:58:00.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.852+0000 7f9d8ce16740 1 -- 172.21.15.40:0/3464860236 wait complete. 2024-03-18T22:58:00.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.853+0000 7f9d8ce16740 20 client.18729 trim_cache size 0 max 16384 2024-03-18T22:58:00.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:00 smithi103 ceph-mon[66377]: pgmap v811: 129 pgs: 129 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 70 KiB/s wr, 171 op/s 2024-03-18T22:58:00.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:00 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:58:00.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:00 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:58:00.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.878+0000 7f9d8ce16740 1 Processor -- start 2024-03-18T22:58:00.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.878+0000 7f9d8ce16740 1 -- start start 2024-03-18T22:58:00.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.878+0000 7f9d8ce16740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad31a10 0x55e97ad3c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.878+0000 7f9d8ce16740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad32400 0x55e97ad433a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.878+0000 7f9d8ce16740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad37570 0x55e97ad2c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.878+0000 7f9d8ce16740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9d841b0bb0 con 0x55e97ad37570 2024-03-18T22:58:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.878+0000 7f9d8ce16740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9d841b28d0 con 0x55e97ad31a10 2024-03-18T22:58:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.878+0000 7f9d8ce16740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9d841b1280 con 0x55e97ad32400 2024-03-18T22:58:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.878+0000 7f9d83fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad37570 0x55e97ad2c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.878+0000 7f9d83fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad37570 0x55e97ad2c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57400/0 (socket says 172.21.15.40:57400) 2024-03-18T22:58:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.878+0000 7f9d83fff640 1 -- 172.21.15.40:0/2015834942 learned_addr learned my addr 172.21.15.40:0/2015834942 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.878+0000 7f9d837fe640 1 --2- 172.21.15.40:0/2015834942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad31a10 0x55e97ad3c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.878+0000 7f9d82ffd640 1 --2- 172.21.15.40:0/2015834942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad32400 0x55e97ad433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.879+0000 7f9d83fff640 1 -- 172.21.15.40:0/2015834942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad32400 msgr2=0x55e97ad433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.879+0000 7f9d83fff640 1 --2- 172.21.15.40:0/2015834942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad32400 0x55e97ad433a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.879+0000 7f9d83fff640 1 -- 172.21.15.40:0/2015834942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad31a10 msgr2=0x55e97ad3c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.879+0000 7f9d83fff640 1 --2- 172.21.15.40:0/2015834942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad31a10 0x55e97ad3c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.879+0000 7f9d83fff640 1 -- 172.21.15.40:0/2015834942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e97a96a0f0 con 0x55e97ad37570 2024-03-18T22:58:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.879+0000 7f9d82ffd640 1 --2- 172.21.15.40:0/2015834942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad32400 0x55e97ad433a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:58:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.879+0000 7f9d83fff640 1 --2- 172.21.15.40:0/2015834942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad37570 0x55e97ad2c010 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c003cc0 tx=0x7f9d7c004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8daeb07158ccfc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.879+0000 7f9d697fa640 1 -- 172.21.15.40:0/2015834942 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c010430 con 0x55e97ad37570 2024-03-18T22:58:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.879+0000 7f9d697fa640 1 -- 172.21.15.40:0/2015834942 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d7c010a80 con 0x55e97ad37570 2024-03-18T22:58:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.879+0000 7f9d697fa640 1 -- 172.21.15.40:0/2015834942 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c01ba70 con 0x55e97ad37570 2024-03-18T22:58:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 -- 172.21.15.40:0/2015834942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad37570 msgr2=0x55e97ad2c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/2015834942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad37570 0x55e97ad2c010 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c003cc0 tx=0x7f9d7c004410 comp rx=0 tx=0).stop 2024-03-18T22:58:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 -- 172.21.15.40:0/2015834942 shutdown_connections 2024-03-18T22:58:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/2015834942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad37570 0x55e97ad2c010 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/2015834942 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad32400 0x55e97ad433a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/2015834942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad31a10 0x55e97ad3c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 -- 172.21.15.40:0/2015834942 >> 172.21.15.40:0/2015834942 conn(0x55e97ad2cf30 msgr2=0x55e97ad2b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 -- 172.21.15.40:0/2015834942 shutdown_connections 2024-03-18T22:58:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 -- 172.21.15.40:0/2015834942 wait complete. 2024-03-18T22:58:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 -- start start 2024-03-18T22:58:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 Processor -- start 2024-03-18T22:58:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad31a10 0x55e97ad413c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad32400 0x55e97ad41900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad37570 0x55e97ad42830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f9d841b2140 con 0x55e97ad31a10 2024-03-18T22:58:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9d841b1a10 con 0x55e97ad32400 2024-03-18T22:58:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f9d841aeb30 con 0x55e97ad37570 2024-03-18T22:58:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.880+0000 7f9d8ce16740 10 client.0 _ll_register_callbacks cb 0x7f9d740a80b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d837fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad31a10 0x55e97ad413c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d837fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad31a10 0x55e97ad413c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57412/0 (socket says 172.21.15.40:57412) 2024-03-18T22:58:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d837fe640 1 -- 172.21.15.40:0/565736547 learned_addr learned my addr 172.21.15.40:0/565736547 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d82ffd640 1 --2- 172.21.15.40:0/565736547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad32400 0x55e97ad41900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d83fff640 1 --2- 172.21.15.40:0/565736547 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad37570 0x55e97ad42830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d837fe640 1 -- 172.21.15.40:0/565736547 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad37570 msgr2=0x55e97ad42830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d837fe640 1 --2- 172.21.15.40:0/565736547 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad37570 0x55e97ad42830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d837fe640 1 -- 172.21.15.40:0/565736547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad32400 msgr2=0x55e97ad41900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d837fe640 1 --2- 172.21.15.40:0/565736547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad32400 0x55e97ad41900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d837fe640 1 -- 172.21.15.40:0/565736547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e97ad42e20 con 0x55e97ad31a10 2024-03-18T22:58:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d83fff640 1 --2- 172.21.15.40:0/565736547 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad37570 0x55e97ad42830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:58:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d837fe640 1 --2- 172.21.15.40:0/565736547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad31a10 0x55e97ad413c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f9d740c1720 tx=0x7f9d740c5c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4349781f8bb62339 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:58:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d62ffd640 1 -- 172.21.15.40:0/565736547 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d740d5040 con 0x55e97ad31a10 2024-03-18T22:58:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d62ffd640 1 -- 172.21.15.40:0/565736547 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9d740bca20 con 0x55e97ad31a10 2024-03-18T22:58:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d8ce16740 1 -- 172.21.15.40:0/565736547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d7c010a80 con 0x55e97ad31a10 2024-03-18T22:58:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d62ffd640 1 -- 172.21.15.40:0/565736547 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f9d740cd050 con 0x55e97ad31a10 2024-03-18T22:58:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d8ce16740 10 client.18735 fetch_fsmap learned FSMap version 48 2024-03-18T22:58:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.881+0000 7f9d8ce16740 1 -- 172.21.15.40:0/565736547 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55e97adae8c0 con 0x55e97ad31a10 2024-03-18T22:58:00.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.882+0000 7f9d62ffd640 1 -- 172.21.15.40:0/565736547 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d7c010a80 con 0x55e97ad31a10 2024-03-18T22:58:00.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.882+0000 7f9d62ffd640 1 -- 172.21.15.40:0/565736547 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 48) ==== 8414+0+0 (secure 0 0 0) 0x7f9d740d13e0 con 0x55e97ad31a10 2024-03-18T22:58:00.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.882+0000 7f9d8ce16740 10 client.18735 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T22:58:00.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.882+0000 7f9d8ce16740 10 client.18735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:58:00.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.882+0000 7f9d8ce16740 20 client.18735 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:00.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.882+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/565736547 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e97adaf170 0x55e97adb1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:00.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.882+0000 7f9d8ce16740 4 client.18735 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:58:00.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.882+0000 7f9d8ce16740 1 -- 172.21.15.40:0/565736547 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e97ac9dd90 con 0x55e97adaf170 2024-03-18T22:58:00.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.882+0000 7f9d82ffd640 1 --2- 172.21.15.40:0/565736547 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e97adaf170 0x55e97adb1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:00.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.885+0000 7f9d82ffd640 1 --2- 172.21.15.40:0/565736547 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e97adaf170 0x55e97adb1590 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cab58dfb806eb1 server_cookie=dc5a453f638af55f in_seq=0 out_seq=0 2024-03-18T22:58:00.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.888+0000 7f9d62ffd640 10 client.18735 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:00.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.889+0000 7f9d62ffd640 1 -- 172.21.15.40:0/565736547 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x55e97ac9dd90 con 0x55e97adaf170 2024-03-18T22:58:00.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.889+0000 7f9d62ffd640 10 client.18735 handle_command_reply: tid=0 2024-03-18T22:58:00.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.891+0000 7f9d8ce16740 1 client.18735 shutdown 2024-03-18T22:58:00.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.891+0000 7f9d8ce16740 1 -- 172.21.15.40:0/565736547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad31a10 msgr2=0x55e97ad413c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.891+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/565736547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad31a10 0x55e97ad413c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f9d740c1720 tx=0x7f9d740c5c20 comp rx=0 tx=0).stop 2024-03-18T22:58:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.893+0000 7f9d8ce16740 1 -- 172.21.15.40:0/565736547 shutdown_connections 2024-03-18T22:58:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.893+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/565736547 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55e97ad37570 0x55e97ad42830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.893+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/565736547 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55e97adaf170 0x55e97adb1590 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.893+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/565736547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e97ad32400 0x55e97ad41900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.893+0000 7f9d8ce16740 1 --2- 172.21.15.40:0/565736547 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e97ad31a10 0x55e97ad413c0 secure :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f9d740c1720 tx=0x7f9d740c5c20 comp rx=0 tx=0).stop 2024-03-18T22:58:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.893+0000 7f9d8ce16740 1 -- 172.21.15.40:0/565736547 >> 172.21.15.40:0/565736547 conn(0x55e97ad2cf30 msgr2=0x55e97ad2c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.893+0000 7f9d62ffd640 0 client.18735 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.893+0000 7f9d8ce16740 1 -- 172.21.15.40:0/565736547 shutdown_connections 2024-03-18T22:58:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.893+0000 7f9d8ce16740 1 -- 172.21.15.40:0/565736547 wait complete. 2024-03-18T22:58:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.893+0000 7f9d8ce16740 20 client.18735 trim_cache size 0 max 16384 2024-03-18T22:58:00.896 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:58:00.896 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "scrub active (10 inodes in the stack)", 2024-03-18T22:58:00.896 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": { 2024-03-18T22:58:00.896 INFO:teuthology.orchestra.run.smithi040.stdout: "bb665ba0-d486-4abd-b75e-c3b21fdcd107": { 2024-03-18T22:58:00.896 INFO:teuthology.orchestra.run.smithi040.stdout: "path": "/", 2024-03-18T22:58:00.896 INFO:teuthology.orchestra.run.smithi040.stdout: "tag": "bb665ba0-d486-4abd-b75e-c3b21fdcd107", 2024-03-18T22:58:00.896 INFO:teuthology.orchestra.run.smithi040.stdout: "options": "recursive,force" 2024-03-18T22:58:00.896 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:00.896 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:00.896 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:58:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.898+0000 7f9d8a159640 1 -- 172.21.15.40:0/362153990 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9d6c0a2880 msgr2=0x7f9d6c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.898+0000 7f9d8a159640 1 --2- 172.21.15.40:0/362153990 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9d6c0a2880 0x7f9d6c0a4d40 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9d78005d00 tx=0x7f9d78005c50 comp rx=0 tx=0).stop 2024-03-18T22:58:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.898+0000 7f9d8a159640 1 -- 172.21.15.40:0/362153990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84100720 msgr2=0x7f9d84069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.898+0000 7f9d8a159640 1 --2- 172.21.15.40:0/362153990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84100720 0x7f9d84069510 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f9d74000c00 tx=0x7f9d74004040 comp rx=0 tx=0).stop 2024-03-18T22:58:00.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.901+0000 7f9d8a159640 1 -- 172.21.15.40:0/362153990 shutdown_connections 2024-03-18T22:58:00.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.901+0000 7f9d8a159640 1 --2- 172.21.15.40:0/362153990 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f9d6c0a2880 0x7f9d6c0a4d40 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.901+0000 7f9d8a159640 1 --2- 172.21.15.40:0/362153990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d841022d0 0x7f9d8419ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.901+0000 7f9d8a159640 1 --2- 172.21.15.40:0/362153990 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f9d84101910 0x7f9d8419a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.901+0000 7f9d8a159640 1 --2- 172.21.15.40:0/362153990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d84100720 0x7f9d84069510 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.901+0000 7f9d8a159640 1 -- 172.21.15.40:0/362153990 >> 172.21.15.40:0/362153990 conn(0x7f9d840fbcb0 msgr2=0x7f9d84104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:00.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.904+0000 7f9d8a159640 1 -- 172.21.15.40:0/362153990 shutdown_connections 2024-03-18T22:58:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:00.905+0000 7f9d8a159640 1 -- 172.21.15.40:0/362153990 wait complete. 2024-03-18T22:58:00.928 INFO:tasks.cephfs.filesystem:scrub status for tag:bb665ba0-d486-4abd-b75e-c3b21fdcd107 - {'path': '/', 'tag': 'bb665ba0-d486-4abd-b75e-c3b21fdcd107', 'options': 'recursive,force'} 2024-03-18T22:58:01.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:01 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:58:01.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:01 smithi113 ceph-mon[66775]: scrub queued for path: / 2024-03-18T22:58:01.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:01 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:58:01.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:01 smithi113 ceph-mon[66775]: scrub summary: active paths [/] 2024-03-18T22:58:01.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:01 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:58:01.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:01 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:58:01.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:01 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:58:01.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:01 smithi103 ceph-mon[66377]: scrub queued for path: / 2024-03-18T22:58:01.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:01 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:58:01.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:01 smithi103 ceph-mon[66377]: scrub summary: active paths [/] 2024-03-18T22:58:01.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:01 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:58:01.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:01 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:58:01.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:01 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:58:01.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:01 smithi040 ceph-mon[62643]: scrub queued for path: / 2024-03-18T22:58:01.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:01 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:58:01.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:01 smithi040 ceph-mon[62643]: scrub summary: active paths [/] 2024-03-18T22:58:01.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:01 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:58:01.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:01 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:58:02.130 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-18T22:58:02.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-18T22:58:02.227 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-03-18T22:58:02.227 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-18T22:58:02.759 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:02 smithi103 ceph-mon[66377]: scrub summary: idle+waiting paths [/] 2024-03-18T22:58:02.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:02 smithi103 ceph-mon[66377]: pgmap v812: 129 pgs: 129 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 115 KiB/s wr, 212 op/s 2024-03-18T22:58:02.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:02 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:58:02.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:02 smithi113 ceph-mon[66775]: scrub summary: idle+waiting paths [/] 2024-03-18T22:58:02.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:02 smithi113 ceph-mon[66775]: pgmap v812: 129 pgs: 129 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 115 KiB/s wr, 212 op/s 2024-03-18T22:58:02.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:02 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:58:02.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:02 smithi040 ceph-mon[62643]: scrub summary: idle+waiting paths [/] 2024-03-18T22:58:02.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:02 smithi040 ceph-mon[62643]: pgmap v812: 129 pgs: 129 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 115 KiB/s wr, 212 op/s 2024-03-18T22:58:02.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:02 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:58:03.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:03 smithi113 ceph-mon[66775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:58:03.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:03 smithi113 ceph-mon[66775]: scrub summary: idle 2024-03-18T22:58:03.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:03 smithi103 ceph-mon[66377]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:58:03.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:03 smithi103 ceph-mon[66377]: scrub summary: idle 2024-03-18T22:58:03.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:03 smithi040 ceph-mon[62643]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T22:58:03.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:03 smithi040 ceph-mon[62643]: scrub summary: idle 2024-03-18T22:58:04.343 DEBUG:teuthology.parallel:result is None 2024-03-18T22:58:04.343 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-18T22:58:04.382 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-18T22:58:04.382 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-18T22:58:04.392 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-18T22:58:04.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:04 smithi040 ceph-mon[62643]: pgmap v813: 129 pgs: 129 active+clean; 2.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 104 KiB/s wr, 243 op/s 2024-03-18T22:58:04.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:04 smithi113 ceph-mon[66775]: pgmap v813: 129 pgs: 129 active+clean; 2.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 104 KiB/s wr, 243 op/s 2024-03-18T22:58:04.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:04 smithi103 ceph-mon[66377]: pgmap v813: 129 pgs: 129 active+clean; 2.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 104 KiB/s wr, 243 op/s 2024-03-18T22:58:06.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:06 smithi113 ceph-mon[66775]: pgmap v814: 129 pgs: 129 active+clean; 1.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 123 KiB/s wr, 234 op/s 2024-03-18T22:58:06.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:06 smithi103 ceph-mon[66377]: pgmap v814: 129 pgs: 129 active+clean; 1.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 123 KiB/s wr, 234 op/s 2024-03-18T22:58:06.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:06 smithi040 ceph-mon[62643]: pgmap v814: 129 pgs: 129 active+clean; 1.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 123 KiB/s wr, 234 op/s 2024-03-18T22:58:08.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:08 smithi113 ceph-mon[66775]: pgmap v815: 129 pgs: 129 active+clean; 495 MiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 222 op/s 2024-03-18T22:58:08.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:08 smithi103 ceph-mon[66377]: pgmap v815: 129 pgs: 129 active+clean; 495 MiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 222 op/s 2024-03-18T22:58:08.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:08 smithi040 ceph-mon[62643]: pgmap v815: 129 pgs: 129 active+clean; 495 MiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 222 op/s 2024-03-18T22:58:10.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:10 smithi113 ceph-mon[66775]: pgmap v816: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 79 KiB/s wr, 210 op/s 2024-03-18T22:58:10.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:10 smithi103 ceph-mon[66377]: pgmap v816: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 79 KiB/s wr, 210 op/s 2024-03-18T22:58:10.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:10 smithi040 ceph-mon[62643]: pgmap v816: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 79 KiB/s wr, 210 op/s 2024-03-18T22:58:12.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:12 smithi113 ceph-mon[66775]: pgmap v817: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 79 KiB/s wr, 160 op/s 2024-03-18T22:58:12.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:12 smithi103 ceph-mon[66377]: pgmap v817: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 79 KiB/s wr, 160 op/s 2024-03-18T22:58:12.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:12 smithi040 ceph-mon[62643]: pgmap v817: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 79 KiB/s wr, 160 op/s 2024-03-18T22:58:14.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:14 smithi113 ceph-mon[66775]: pgmap v818: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 34 KiB/s wr, 118 op/s 2024-03-18T22:58:14.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:14 smithi103 ceph-mon[66377]: pgmap v818: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 34 KiB/s wr, 118 op/s 2024-03-18T22:58:14.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:14 smithi040 ceph-mon[62643]: pgmap v818: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 34 KiB/s wr, 118 op/s 2024-03-18T22:58:16.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:16 smithi113 ceph-mon[66775]: pgmap v819: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 32 KiB/s wr, 73 op/s 2024-03-18T22:58:16.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:16 smithi103 ceph-mon[66377]: pgmap v819: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 32 KiB/s wr, 73 op/s 2024-03-18T22:58:16.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:16 smithi040 ceph-mon[62643]: pgmap v819: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 32 KiB/s wr, 73 op/s 2024-03-18T22:58:18.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:18 smithi113 ceph-mon[66775]: pgmap v820: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.8 KiB/s wr, 42 op/s 2024-03-18T22:58:18.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:18 smithi103 ceph-mon[66377]: pgmap v820: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.8 KiB/s wr, 42 op/s 2024-03-18T22:58:18.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:18 smithi040 ceph-mon[62643]: pgmap v820: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.8 KiB/s wr, 42 op/s 2024-03-18T22:58:20.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:20 smithi113 ceph-mon[66775]: pgmap v821: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.5 KiB/s wr, 23 op/s 2024-03-18T22:58:20.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:20 smithi103 ceph-mon[66377]: pgmap v821: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.5 KiB/s wr, 23 op/s 2024-03-18T22:58:20.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:20 smithi040 ceph-mon[62643]: pgmap v821: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.5 KiB/s wr, 23 op/s 2024-03-18T22:58:22.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:22 smithi113 ceph-mon[66775]: pgmap v822: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 4.2 KiB/s wr, 12 op/s 2024-03-18T22:58:22.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:22 smithi103 ceph-mon[66377]: pgmap v822: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 4.2 KiB/s wr, 12 op/s 2024-03-18T22:58:22.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:22 smithi040 ceph-mon[62643]: pgmap v822: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 4.2 KiB/s wr, 12 op/s 2024-03-18T22:58:24.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:24 smithi113 ceph-mon[66775]: pgmap v823: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-18T22:58:24.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:24 smithi103 ceph-mon[66377]: pgmap v823: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-18T22:58:24.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:24 smithi040 ceph-mon[62643]: pgmap v823: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-18T22:58:26.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:26 smithi113 ceph-mon[66775]: pgmap v824: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T22:58:26.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:26 smithi103 ceph-mon[66377]: pgmap v824: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T22:58:26.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:26 smithi040 ceph-mon[62643]: pgmap v824: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T22:58:28.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:28 smithi113 ceph-mon[66775]: pgmap v825: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T22:58:28.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:28 smithi103 ceph-mon[66377]: pgmap v825: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T22:58:28.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:28 smithi040 ceph-mon[62643]: pgmap v825: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T22:58:30.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:30 smithi113 ceph-mon[66775]: pgmap v826: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:58:30.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:30 smithi103 ceph-mon[66377]: pgmap v826: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:58:30.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:30 smithi040 ceph-mon[62643]: pgmap v826: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:58:30.929 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T22:58:30.929 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T22:58:31.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.046+0000 7f1b7eb59640 1 -- 172.21.15.40:0/1877386236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b78102a30 msgr2=0x7f1b78102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.046+0000 7f1b7eb59640 1 --2- 172.21.15.40:0/1877386236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b78102a30 0x7f1b78102eb0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009900 tx=0x7f1b6802bc30 comp rx=0 tx=0).stop 2024-03-18T22:58:31.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.046+0000 7f1b7eb59640 1 -- 172.21.15.40:0/1877386236 shutdown_connections 2024-03-18T22:58:31.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.046+0000 7f1b7eb59640 1 --2- 172.21.15.40:0/1877386236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b781033f0 0x7f1b78109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.046+0000 7f1b7eb59640 1 --2- 172.21.15.40:0/1877386236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b78102a30 0x7f1b78102eb0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.046+0000 7f1b7eb59640 1 --2- 172.21.15.40:0/1877386236 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b78101840 0x7f1b78101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.046+0000 7f1b7eb59640 1 -- 172.21.15.40:0/1877386236 >> 172.21.15.40:0/1877386236 conn(0x7f1b780fcdb0 msgr2=0x7f1b780ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.046+0000 7f1b7eb59640 1 -- 172.21.15.40:0/1877386236 shutdown_connections 2024-03-18T22:58:31.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.046+0000 7f1b7eb59640 1 -- 172.21.15.40:0/1877386236 wait complete. 2024-03-18T22:58:31.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7eb59640 1 Processor -- start 2024-03-18T22:58:31.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7eb59640 1 -- start start 2024-03-18T22:58:31.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7eb59640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b78101840 0x7f1b78197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7eb59640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b78102a30 0x7f1b78197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7eb59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b781033f0 0x7f1b7819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7eb59640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7810d370 con 0x7f1b781033f0 2024-03-18T22:58:31.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7eb59640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7810d1f0 con 0x7f1b78101840 2024-03-18T22:58:31.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7eb59640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7810d4f0 con 0x7f1b78102a30 2024-03-18T22:58:31.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7d0f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b781033f0 0x7f1b7819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7d0f6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b781033f0 0x7f1b7819f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33536/0 (socket says 172.21.15.40:33536) 2024-03-18T22:58:31.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7d0f6640 1 -- 172.21.15.40:0/2488560276 learned_addr learned my addr 172.21.15.40:0/2488560276 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7c8f5640 1 --2- 172.21.15.40:0/2488560276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b78101840 0x7f1b78197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b6ffff640 1 --2- 172.21.15.40:0/2488560276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b78102a30 0x7f1b78197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7d0f6640 1 -- 172.21.15.40:0/2488560276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b78102a30 msgr2=0x7f1b78197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7d0f6640 1 --2- 172.21.15.40:0/2488560276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b78102a30 0x7f1b78197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7d0f6640 1 -- 172.21.15.40:0/2488560276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b78101840 msgr2=0x7f1b78197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.047+0000 7f1b7d0f6640 1 --2- 172.21.15.40:0/2488560276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b78101840 0x7f1b78197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.048+0000 7f1b7d0f6640 1 -- 172.21.15.40:0/2488560276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7819f6c0 con 0x7f1b781033f0 2024-03-18T22:58:31.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.048+0000 7f1b7d0f6640 1 --2- 172.21.15.40:0/2488560276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b781033f0 0x7f1b7819f050 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000efd0 tx=0x7f1b7000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.048+0000 7f1b6dffb640 1 -- 172.21.15.40:0/2488560276 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b70003b70 con 0x7f1b781033f0 2024-03-18T22:58:31.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.048+0000 7f1b7eb59640 1 -- 172.21.15.40:0/2488560276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b7819f9b0 con 0x7f1b781033f0 2024-03-18T22:58:31.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.048+0000 7f1b6dffb640 1 -- 172.21.15.40:0/2488560276 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b70003d10 con 0x7f1b781033f0 2024-03-18T22:58:31.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.048+0000 7f1b6dffb640 1 -- 172.21.15.40:0/2488560276 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b70010980 con 0x7f1b781033f0 2024-03-18T22:58:31.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.048+0000 7f1b7eb59640 1 -- 172.21.15.40:0/2488560276 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b7819fed0 con 0x7f1b781033f0 2024-03-18T22:58:31.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.050+0000 7f1b6dffb640 1 -- 172.21.15.40:0/2488560276 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1b70004020 con 0x7f1b781033f0 2024-03-18T22:58:31.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b81813740 1 Processor -- start 2024-03-18T22:58:31.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b6dffb640 1 --2- 172.21.15.40:0/2488560276 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1b600a2800 0x7f1b600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b81813740 1 -- start start 2024-03-18T22:58:31.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b6dffb640 1 -- 172.21.15.40:0/2488560276 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1b700cf310 con 0x7f1b781033f0 2024-03-18T22:58:31.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b6ffff640 1 --2- 172.21.15.40:0/2488560276 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1b600a2800 0x7f1b600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b81813740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d739280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b81813740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 0x55c89d73e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b81813740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d740e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b81813740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c89d699500 con 0x55c89d738e60 2024-03-18T22:58:31.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b81813740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c89d641be0 con 0x55c89d739850 2024-03-18T22:58:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b81813740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c89d6955a0 con 0x55c89d73e9c0 2024-03-18T22:58:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b7c8f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d739280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b7c8f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d739280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33550/0 (socket says 172.21.15.40:33550) 2024-03-18T22:58:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b7c8f5640 1 -- 172.21.15.40:0/3283578632 learned_addr learned my addr 172.21.15.40:0/3283578632 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b6ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 0x55c89d73e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b7d0f6640 1 --2- 172.21.15.40:0/3283578632 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d740e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b7c8f5640 1 -- 172.21.15.40:0/3283578632 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 msgr2=0x55c89d740e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b7c8f5640 1 --2- 172.21.15.40:0/3283578632 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d740e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b7c8f5640 1 -- 172.21.15.40:0/3283578632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 msgr2=0x55c89d73e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b7c8f5640 1 --2- 172.21.15.40:0/3283578632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 0x55c89d73e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b7c8f5640 1 -- 172.21.15.40:0/3283578632 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c89d663000 con 0x55c89d738e60 2024-03-18T22:58:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b7d0f6640 1 --2- 172.21.15.40:0/3283578632 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d740e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.051+0000 7f1b6ffff640 1 --2- 172.21.15.40:0/2488560276 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1b600a2800 0x7f1b600a4cc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009230 tx=0x7f1b6802cb90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.052+0000 7f1b7c8f5640 1 --2- 172.21.15.40:0/3283578632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d739280 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009ca0 tx=0x7f1b74011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8cdf16cce515d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.052+0000 7f1b5effd640 1 -- 172.21.15.40:0/3283578632 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b74009e50 con 0x55c89d738e60 2024-03-18T22:58:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.052+0000 7f1b5effd640 1 -- 172.21.15.40:0/3283578632 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b740035f0 con 0x55c89d738e60 2024-03-18T22:58:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.052+0000 7f1b5effd640 1 -- 172.21.15.40:0/3283578632 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b74019670 con 0x55c89d738e60 2024-03-18T22:58:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.052+0000 7f1b81813740 1 -- 172.21.15.40:0/3283578632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 msgr2=0x55c89d739280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.052+0000 7f1b81813740 1 --2- 172.21.15.40:0/3283578632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d739280 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009ca0 tx=0x7f1b74011810 comp rx=0 tx=0).stop 2024-03-18T22:58:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.052+0000 7f1b81813740 1 -- 172.21.15.40:0/3283578632 shutdown_connections 2024-03-18T22:58:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.052+0000 7f1b81813740 1 --2- 172.21.15.40:0/3283578632 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d740e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.052+0000 7f1b81813740 1 --2- 172.21.15.40:0/3283578632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 0x55c89d73e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.052+0000 7f1b81813740 1 --2- 172.21.15.40:0/3283578632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d739280 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.052+0000 7f1b81813740 1 -- 172.21.15.40:0/3283578632 >> 172.21.15.40:0/3283578632 conn(0x55c89d72ffc0 msgr2=0x55c89d7303c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b81813740 1 -- 172.21.15.40:0/3283578632 shutdown_connections 2024-03-18T22:58:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b81813740 1 -- 172.21.15.40:0/3283578632 wait complete. 2024-03-18T22:58:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b81813740 1 -- start start 2024-03-18T22:58:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b81813740 1 Processor -- start 2024-03-18T22:58:31.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b81813740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d743230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b81813740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 0x55c89d743770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b81813740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d74a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b81813740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c89d741970 con 0x55c89d738e60 2024-03-18T22:58:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b81813740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c89d655160 con 0x55c89d739850 2024-03-18T22:58:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b81813740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c89d741af0 con 0x55c89d73e9c0 2024-03-18T22:58:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b81813740 10 client.0 _ll_register_callbacks cb 0x55c89d691a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b7c8f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d743230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b7c8f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d743230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33564/0 (socket says 172.21.15.40:33564) 2024-03-18T22:58:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b7c8f5640 1 -- 172.21.15.40:0/2302089661 learned_addr learned my addr 172.21.15.40:0/2302089661 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b6ffff640 1 --2- 172.21.15.40:0/2302089661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 0x55c89d743770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.053+0000 7f1b7d0f6640 1 --2- 172.21.15.40:0/2302089661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d74a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.054+0000 7f1b7c8f5640 1 -- 172.21.15.40:0/2302089661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 msgr2=0x55c89d74a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.054+0000 7f1b7c8f5640 1 --2- 172.21.15.40:0/2302089661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d74a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.054+0000 7f1b7c8f5640 1 -- 172.21.15.40:0/2302089661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 msgr2=0x55c89d743770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.054+0000 7f1b7c8f5640 1 --2- 172.21.15.40:0/2302089661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 0x55c89d743770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.054+0000 7f1b7c8f5640 1 -- 172.21.15.40:0/2302089661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c89d4777d0 con 0x55c89d738e60 2024-03-18T22:58:31.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.054+0000 7f1b7d0f6640 1 --2- 172.21.15.40:0/2302089661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d74a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.054+0000 7f1b6ffff640 1 --2- 172.21.15.40:0/2302089661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 0x55c89d743770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.054+0000 7f1b7c8f5640 1 --2- 172.21.15.40:0/2302089661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d743230 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f1b74011d10 tx=0x7f1b740041e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37878c89ea19fa26 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.054+0000 7f1b57fff640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:58:31.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.054+0000 7f1b57fff640 1 -- 172.21.15.40:0/2302089661 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b74028020 con 0x55c89d738e60 2024-03-18T22:58:31.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.054+0000 7f1b57fff640 1 -- 172.21.15.40:0/2302089661 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1b74016040 con 0x55c89d738e60 2024-03-18T22:58:31.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.054+0000 7f1b81813740 1 -- 172.21.15.40:0/2302089661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b740035f0 con 0x55c89d738e60 2024-03-18T22:58:31.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.054+0000 7f1b57fff640 1 -- 172.21.15.40:0/2302089661 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f1b74033050 con 0x55c89d738e60 2024-03-18T22:58:31.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.054+0000 7f1b81813740 10 client.18747 fetch_fsmap learned FSMap version 49 2024-03-18T22:58:31.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.054+0000 7f1b81813740 1 -- 172.21.15.40:0/2302089661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55c89d3ad3f0 con 0x55c89d738e60 2024-03-18T22:58:31.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.055+0000 7f1b57fff640 1 -- 172.21.15.40:0/2302089661 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b740035f0 con 0x55c89d738e60 2024-03-18T22:58:31.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.055+0000 7f1b57fff640 1 -- 172.21.15.40:0/2302089661 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8418+0+0 (secure 0 0 0) 0x7f1b74004850 con 0x55c89d738e60 2024-03-18T22:58:31.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.055+0000 7f1b81813740 10 client.18747 fetch_fsmap finished waiting for FSMap version 49 2024-03-18T22:58:31.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.055+0000 7f1b81813740 10 client.18747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:58:31.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.055+0000 7f1b81813740 20 client.18747 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:31.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.055+0000 7f1b81813740 1 --2- 172.21.15.40:0/2302089661 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c89d74ef30 0x55c89d7513b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.055+0000 7f1b81813740 4 client.18747 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:58:31.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.055+0000 7f1b81813740 1 -- 172.21.15.40:0/2302089661 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c89d6a6a40 con 0x55c89d74ef30 2024-03-18T22:58:31.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.055+0000 7f1b6ffff640 1 --2- 172.21.15.40:0/2302089661 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c89d74ef30 0x55c89d7513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.055+0000 7f1b57fff640 10 client.18747 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:31.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.055+0000 7f1b6ffff640 1 --2- 172.21.15.40:0/2302089661 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c89d74ef30 0x55c89d7513b0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d142cbffabbf173 server_cookie=c986239c6502bff1 in_seq=0 out_seq=0 2024-03-18T22:58:31.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.056+0000 7f1b57fff640 1 -- 172.21.15.40:0/2302089661 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c89d6a6a40 con 0x55c89d74ef30 2024-03-18T22:58:31.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.056+0000 7f1b57fff640 10 client.18747 handle_command_reply: tid=0 2024-03-18T22:58:31.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.056+0000 7f1b81813740 1 client.18747 shutdown 2024-03-18T22:58:31.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.057+0000 7f1b81813740 1 -- 172.21.15.40:0/2302089661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 msgr2=0x55c89d743230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.057+0000 7f1b81813740 1 --2- 172.21.15.40:0/2302089661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d743230 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f1b74011d10 tx=0x7f1b740041e0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.057+0000 7f1b81813740 1 -- 172.21.15.40:0/2302089661 shutdown_connections 2024-03-18T22:58:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.057+0000 7f1b81813740 1 --2- 172.21.15.40:0/2302089661 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d74a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.057+0000 7f1b81813740 1 --2- 172.21.15.40:0/2302089661 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c89d74ef30 0x55c89d7513b0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.057+0000 7f1b81813740 1 --2- 172.21.15.40:0/2302089661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 0x55c89d743770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.057+0000 7f1b81813740 1 --2- 172.21.15.40:0/2302089661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d743230 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.057+0000 7f1b81813740 1 -- 172.21.15.40:0/2302089661 >> 172.21.15.40:0/2302089661 conn(0x55c89d72ffc0 msgr2=0x55c89d72f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.057+0000 7f1b57fff640 0 client.18747 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.057+0000 7f1b81813740 1 -- 172.21.15.40:0/2302089661 shutdown_connections 2024-03-18T22:58:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.057+0000 7f1b81813740 1 -- 172.21.15.40:0/2302089661 wait complete. 2024-03-18T22:58:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.057+0000 7f1b81813740 20 client.18747 trim_cache size 0 max 16384 2024-03-18T22:58:31.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b81813740 1 Processor -- start 2024-03-18T22:58:31.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b81813740 1 -- start start 2024-03-18T22:58:31.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b81813740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d738e60 0x55c89d72af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b81813740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d739850 0x55c89d74a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b81813740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d742850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b81813740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1b781a8360 con 0x55c89d739850 2024-03-18T22:58:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b81813740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1b781aa080 con 0x55c89d738e60 2024-03-18T22:58:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b81813740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1b781a8a30 con 0x55c89d73e9c0 2024-03-18T22:58:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b6ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d739850 0x55c89d74a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b6ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d739850 0x55c89d74a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33578/0 (socket says 172.21.15.40:33578) 2024-03-18T22:58:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b7c8f5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d738e60 0x55c89d72af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b6ffff640 1 -- 172.21.15.40:0/3147580948 learned_addr learned my addr 172.21.15.40:0/3147580948 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b7d0f6640 1 --2- 172.21.15.40:0/3147580948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d742850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b6ffff640 1 -- 172.21.15.40:0/3147580948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 msgr2=0x55c89d742850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b6ffff640 1 --2- 172.21.15.40:0/3147580948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d742850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b6ffff640 1 -- 172.21.15.40:0/3147580948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d738e60 msgr2=0x55c89d72af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b6ffff640 1 --2- 172.21.15.40:0/3147580948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d738e60 0x55c89d72af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b6ffff640 1 -- 172.21.15.40:0/3147580948 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c89d36d0f0 con 0x55c89d739850 2024-03-18T22:58:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b7c8f5640 1 --2- 172.21.15.40:0/3147580948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d738e60 0x55c89d72af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b6ffff640 1 --2- 172.21.15.40:0/3147580948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d739850 0x55c89d74a770 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009900 tx=0x7f1b68045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc4ffe555e6d42a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.075+0000 7f1b5e7fc640 1 -- 172.21.15.40:0/3147580948 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b68049070 con 0x55c89d739850 2024-03-18T22:58:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.076+0000 7f1b5e7fc640 1 -- 172.21.15.40:0/3147580948 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b68005590 con 0x55c89d739850 2024-03-18T22:58:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.076+0000 7f1b5e7fc640 1 -- 172.21.15.40:0/3147580948 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b68003820 con 0x55c89d739850 2024-03-18T22:58:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.076+0000 7f1b81813740 1 -- 172.21.15.40:0/3147580948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d739850 msgr2=0x55c89d74a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.076+0000 7f1b81813740 1 --2- 172.21.15.40:0/3147580948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d739850 0x55c89d74a770 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009900 tx=0x7f1b68045000 comp rx=0 tx=0).stop 2024-03-18T22:58:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.076+0000 7f1b81813740 1 -- 172.21.15.40:0/3147580948 shutdown_connections 2024-03-18T22:58:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.076+0000 7f1b81813740 1 --2- 172.21.15.40:0/3147580948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d742850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.076+0000 7f1b81813740 1 --2- 172.21.15.40:0/3147580948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d739850 0x55c89d74a770 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.076+0000 7f1b81813740 1 --2- 172.21.15.40:0/3147580948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d738e60 0x55c89d72af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.076+0000 7f1b81813740 1 -- 172.21.15.40:0/3147580948 >> 172.21.15.40:0/3147580948 conn(0x55c89d72ffc0 msgr2=0x55c89d733ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.076+0000 7f1b81813740 1 -- 172.21.15.40:0/3147580948 shutdown_connections 2024-03-18T22:58:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.076+0000 7f1b81813740 1 -- 172.21.15.40:0/3147580948 wait complete. 2024-03-18T22:58:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.076+0000 7f1b81813740 1 -- start start 2024-03-18T22:58:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.076+0000 7f1b81813740 1 Processor -- start 2024-03-18T22:58:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b81813740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d7476a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b81813740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 0x55c89d747be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b81813740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d7b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b81813740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1b781a98f0 con 0x55c89d738e60 2024-03-18T22:58:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b81813740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1b781a91c0 con 0x55c89d739850 2024-03-18T22:58:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b81813740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1b781a62e0 con 0x55c89d73e9c0 2024-03-18T22:58:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b81813740 10 client.0 _ll_register_callbacks cb 0x7f1b74005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b7c8f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d7476a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b6ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 0x55c89d747be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b7c8f5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d7476a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33590/0 (socket says 172.21.15.40:33590) 2024-03-18T22:58:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b7c8f5640 1 -- 172.21.15.40:0/1798039706 learned_addr learned my addr 172.21.15.40:0/1798039706 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b7d0f6640 1 --2- 172.21.15.40:0/1798039706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d7b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b6ffff640 1 -- 172.21.15.40:0/1798039706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 msgr2=0x55c89d7b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b6ffff640 1 --2- 172.21.15.40:0/1798039706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d7b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b6ffff640 1 -- 172.21.15.40:0/1798039706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 msgr2=0x55c89d7476a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b6ffff640 1 --2- 172.21.15.40:0/1798039706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d7476a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b6ffff640 1 -- 172.21.15.40:0/1798039706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c89d7b3820 con 0x55c89d739850 2024-03-18T22:58:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b7c8f5640 1 --2- 172.21.15.40:0/1798039706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d7476a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b6ffff640 1 --2- 172.21.15.40:0/1798039706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 0x55c89d747be0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009900 tx=0x7f1b68009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9165c7799c81364e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.077+0000 7f1b57fff640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:58:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.078+0000 7f1b57fff640 1 -- 172.21.15.40:0/1798039706 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b68061030 con 0x55c89d739850 2024-03-18T22:58:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.078+0000 7f1b57fff640 1 -- 172.21.15.40:0/1798039706 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1b68054040 con 0x55c89d739850 2024-03-18T22:58:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.078+0000 7f1b81813740 1 -- 172.21.15.40:0/1798039706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b68005590 con 0x55c89d739850 2024-03-18T22:58:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.078+0000 7f1b57fff640 1 -- 172.21.15.40:0/1798039706 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f1b6806e050 con 0x55c89d739850 2024-03-18T22:58:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.078+0000 7f1b81813740 10 client.28562 fetch_fsmap learned FSMap version 49 2024-03-18T22:58:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.078+0000 7f1b81813740 1 -- 172.21.15.40:0/1798039706 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55c89d7b4a80 con 0x55c89d739850 2024-03-18T22:58:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.078+0000 7f1b57fff640 1 -- 172.21.15.40:0/1798039706 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b68005590 con 0x55c89d739850 2024-03-18T22:58:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.078+0000 7f1b57fff640 1 -- 172.21.15.40:0/1798039706 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 49) ==== 8418+0+0 (secure 0 0 0) 0x7f1b68040040 con 0x55c89d739850 2024-03-18T22:58:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.078+0000 7f1b81813740 10 client.28562 fetch_fsmap finished waiting for FSMap version 49 2024-03-18T22:58:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.078+0000 7f1b81813740 10 client.28562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:58:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.078+0000 7f1b81813740 20 client.28562 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.078+0000 7f1b81813740 1 --2- 172.21.15.40:0/1798039706 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c89d7b54c0 0x55c89d7b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.078+0000 7f1b81813740 4 client.28562 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T22:58:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.078+0000 7f1b81813740 1 -- 172.21.15.40:0/1798039706 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c89d6a0d90 con 0x55c89d7b54c0 2024-03-18T22:58:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.078+0000 7f1b7c8f5640 1 --2- 172.21.15.40:0/1798039706 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c89d7b54c0 0x55c89d7b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b57fff640 10 client.28562 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b7c8f5640 1 --2- 172.21.15.40:0/1798039706 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c89d7b54c0 0x55c89d7b78e0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eaf0df9d2c11bd82 server_cookie=535cfe862f2180d7 in_seq=0 out_seq=0 2024-03-18T22:58:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b57fff640 1 -- 172.21.15.40:0/1798039706 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c89d6a0d90 con 0x55c89d7b54c0 2024-03-18T22:58:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b57fff640 10 client.28562 handle_command_reply: tid=0 2024-03-18T22:58:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b81813740 1 client.28562 shutdown 2024-03-18T22:58:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b81813740 1 -- 172.21.15.40:0/1798039706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 msgr2=0x55c89d747be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b81813740 1 --2- 172.21.15.40:0/1798039706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 0x55c89d747be0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009900 tx=0x7f1b68009260 comp rx=0 tx=0).stop 2024-03-18T22:58:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b81813740 1 -- 172.21.15.40:0/1798039706 shutdown_connections 2024-03-18T22:58:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b81813740 1 --2- 172.21.15.40:0/1798039706 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c89d73e9c0 0x55c89d7b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b81813740 1 --2- 172.21.15.40:0/1798039706 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x55c89d7b54c0 0x55c89d7b78e0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b81813740 1 --2- 172.21.15.40:0/1798039706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89d739850 0x55c89d747be0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b81813740 1 --2- 172.21.15.40:0/1798039706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c89d738e60 0x55c89d7476a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b57fff640 0 client.28562 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b81813740 1 -- 172.21.15.40:0/1798039706 >> 172.21.15.40:0/1798039706 conn(0x55c89d72ffc0 msgr2=0x55c89d72e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b81813740 1 -- 172.21.15.40:0/1798039706 shutdown_connections 2024-03-18T22:58:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b81813740 1 -- 172.21.15.40:0/1798039706 wait complete. 2024-03-18T22:58:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.079+0000 7f1b81813740 20 client.28562 trim_cache size 0 max 16384 2024-03-18T22:58:31.086 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:58:31.086 INFO:teuthology.orchestra.run.smithi040.stdout: "status": "no active scrubs running", 2024-03-18T22:58:31.087 INFO:teuthology.orchestra.run.smithi040.stdout: "scrubs": {} 2024-03-18T22:58:31.087 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:58:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.080+0000 7f1b7eb59640 1 -- 172.21.15.40:0/2488560276 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1b600a2800 msgr2=0x7f1b600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.080+0000 7f1b7eb59640 1 --2- 172.21.15.40:0/2488560276 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1b600a2800 0x7f1b600a4cc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009230 tx=0x7f1b6802cb90 comp rx=0 tx=0).stop 2024-03-18T22:58:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.080+0000 7f1b7eb59640 1 -- 172.21.15.40:0/2488560276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b781033f0 msgr2=0x7f1b7819f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.080+0000 7f1b7eb59640 1 --2- 172.21.15.40:0/2488560276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b781033f0 0x7f1b7819f050 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000efd0 tx=0x7f1b7000c6a0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.080+0000 7f1b7eb59640 1 -- 172.21.15.40:0/2488560276 shutdown_connections 2024-03-18T22:58:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.080+0000 7f1b7eb59640 1 --2- 172.21.15.40:0/2488560276 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1b600a2800 0x7f1b600a4cc0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.080+0000 7f1b7eb59640 1 --2- 172.21.15.40:0/2488560276 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b781033f0 0x7f1b7819f050 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.080+0000 7f1b7eb59640 1 --2- 172.21.15.40:0/2488560276 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1b78102a30 0x7f1b78197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.080+0000 7f1b7eb59640 1 --2- 172.21.15.40:0/2488560276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b78101840 0x7f1b78197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.080+0000 7f1b7eb59640 1 -- 172.21.15.40:0/2488560276 >> 172.21.15.40:0/2488560276 conn(0x7f1b780fcdb0 msgr2=0x7f1b78105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.080+0000 7f1b7eb59640 1 -- 172.21.15.40:0/2488560276 shutdown_connections 2024-03-18T22:58:31.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.080+0000 7f1b7eb59640 1 -- 172.21.15.40:0/2488560276 wait complete. 2024-03-18T22:58:31.091 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T22:58:31.091 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:58:31.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.194+0000 7f40a43c6640 1 -- 172.21.15.40:0/702323841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f409c102e30 msgr2=0x7f409c103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.194+0000 7f40a43c6640 1 --2- 172.21.15.40:0/702323841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f409c102e30 0x7f409c103250 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4098009be0 tx=0x7f409802bef0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.194+0000 7f40a43c6640 1 -- 172.21.15.40:0/702323841 shutdown_connections 2024-03-18T22:58:31.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.194+0000 7f40a43c6640 1 --2- 172.21.15.40:0/702323841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f409c100260 0x7f409c109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.194+0000 7f40a43c6640 1 --2- 172.21.15.40:0/702323841 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f409c0ff8a0 0x7f409c0ffd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.194+0000 7f40a43c6640 1 --2- 172.21.15.40:0/702323841 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f409c102e30 0x7f409c103250 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.194+0000 7f40a43c6640 1 -- 172.21.15.40:0/702323841 >> 172.21.15.40:0/702323841 conn(0x7f409c0fb440 msgr2=0x7f409c0fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.194+0000 7f40a43c6640 1 -- 172.21.15.40:0/702323841 shutdown_connections 2024-03-18T22:58:31.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.194+0000 7f40a43c6640 1 -- 172.21.15.40:0/702323841 wait complete. 2024-03-18T22:58:31.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a43c6640 1 Processor -- start 2024-03-18T22:58:31.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a43c6640 1 -- start start 2024-03-18T22:58:31.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a43c6640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f409c0ff8a0 0x7f409c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a43c6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f409c100260 0x7f409c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a43c6640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f409c102e30 0x7f409c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a43c6640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f409c10d380 con 0x7f409c102e30 2024-03-18T22:58:31.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a43c6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f409c10d200 con 0x7f409c100260 2024-03-18T22:58:31.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a43c6640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f409c10d500 con 0x7f409c0ff8a0 2024-03-18T22:58:31.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a2162640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f409c0ff8a0 0x7f409c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a2963640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f409c102e30 0x7f409c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a2162640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f409c0ff8a0 0x7f409c197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38024/0 (socket says 172.21.15.40:38024) 2024-03-18T22:58:31.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a2162640 1 -- 172.21.15.40:0/1174168962 learned_addr learned my addr 172.21.15.40:0/1174168962 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a1961640 1 --2- 172.21.15.40:0/1174168962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f409c100260 0x7f409c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a2162640 1 -- 172.21.15.40:0/1174168962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f409c100260 msgr2=0x7f409c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a2162640 1 --2- 172.21.15.40:0/1174168962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f409c100260 0x7f409c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a2162640 1 -- 172.21.15.40:0/1174168962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f409c102e30 msgr2=0x7f409c19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a2162640 1 --2- 172.21.15.40:0/1174168962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f409c102e30 0x7f409c19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a2162640 1 -- 172.21.15.40:0/1174168962 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409c19f6a0 con 0x7f409c0ff8a0 2024-03-18T22:58:31.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a2963640 1 --2- 172.21.15.40:0/1174168962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f409c102e30 0x7f409c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.195+0000 7f40a2162640 1 --2- 172.21.15.40:0/1174168962 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f409c0ff8a0 0x7f409c197a80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4098000f80 tx=0x7f4098033cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.196+0000 7f40937fe640 1 -- 172.21.15.40:0/1174168962 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4098043020 con 0x7f409c0ff8a0 2024-03-18T22:58:31.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.196+0000 7f40937fe640 1 -- 172.21.15.40:0/1174168962 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f409802ca60 con 0x7f409c0ff8a0 2024-03-18T22:58:31.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.196+0000 7f40937fe640 1 -- 172.21.15.40:0/1174168962 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4098034430 con 0x7f409c0ff8a0 2024-03-18T22:58:31.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.196+0000 7f40a43c6640 1 -- 172.21.15.40:0/1174168962 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f409c19f930 con 0x7f409c0ff8a0 2024-03-18T22:58:31.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.196+0000 7f40a43c6640 1 -- 172.21.15.40:0/1174168962 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f409c19fe80 con 0x7f409c0ff8a0 2024-03-18T22:58:31.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.197+0000 7f40a43c6640 1 -- 172.21.15.40:0/1174168962 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f409c108db0 con 0x7f409c0ff8a0 2024-03-18T22:58:31.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.198+0000 7f40937fe640 1 -- 172.21.15.40:0/1174168962 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4098046020 con 0x7f409c0ff8a0 2024-03-18T22:58:31.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.198+0000 7f40937fe640 1 --2- 172.21.15.40:0/1174168962 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f40880a2800 0x7f40880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.198+0000 7f40a1961640 1 --2- 172.21.15.40:0/1174168962 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f40880a2800 0x7f40880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.198+0000 7f40937fe640 1 -- 172.21.15.40:0/1174168962 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f40980f1c50 con 0x7f409c0ff8a0 2024-03-18T22:58:31.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.199+0000 7f40a1961640 1 --2- 172.21.15.40:0/1174168962 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f40880a2800 0x7f40880a4cc0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f409c198f90 tx=0x7f408c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.202+0000 7f40937fe640 1 -- 172.21.15.40:0/1174168962 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f409804d020 con 0x7f409c0ff8a0 2024-03-18T22:58:31.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.425+0000 7f40a43c6640 1 -- 172.21.15.40:0/1174168962 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f409c02d1f0 con 0x7f409c0ff8a0 2024-03-18T22:58:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.426+0000 7f40937fe640 1 -- 172.21.15.40:0/1174168962 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11970 (secure 0 0 0) 0x7f40980be900 con 0x7f409c0ff8a0 2024-03-18T22:58:31.427 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:58:31.427 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:58:01.473332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24509,24481,24473,24388,24401]},"id":1}]} 2024-03-18T22:58:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 49 2024-03-18T22:58:31.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.428+0000 7f40a43c6640 1 -- 172.21.15.40:0/1174168962 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f40880a2800 msgr2=0x7f40880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.428+0000 7f40a43c6640 1 --2- 172.21.15.40:0/1174168962 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f40880a2800 0x7f40880a4cc0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f409c198f90 tx=0x7f408c009260 comp rx=0 tx=0).stop 2024-03-18T22:58:31.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.428+0000 7f40a43c6640 1 -- 172.21.15.40:0/1174168962 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f409c0ff8a0 msgr2=0x7f409c197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.428+0000 7f40a43c6640 1 --2- 172.21.15.40:0/1174168962 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f409c0ff8a0 0x7f409c197a80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4098000f80 tx=0x7f4098033cd0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.428+0000 7f40a43c6640 1 -- 172.21.15.40:0/1174168962 shutdown_connections 2024-03-18T22:58:31.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.428+0000 7f40a43c6640 1 --2- 172.21.15.40:0/1174168962 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f40880a2800 0x7f40880a4cc0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.428+0000 7f40a43c6640 1 --2- 172.21.15.40:0/1174168962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f409c102e30 0x7f409c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.428+0000 7f40a43c6640 1 --2- 172.21.15.40:0/1174168962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f409c100260 0x7f409c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.428+0000 7f40a43c6640 1 --2- 172.21.15.40:0/1174168962 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f409c0ff8a0 0x7f409c197a80 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.428+0000 7f40a43c6640 1 -- 172.21.15.40:0/1174168962 >> 172.21.15.40:0/1174168962 conn(0x7f409c0fb440 msgr2=0x7f409c105f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.428+0000 7f40a43c6640 1 -- 172.21.15.40:0/1174168962 shutdown_connections 2024-03-18T22:58:31.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.428+0000 7f40a43c6640 1 -- 172.21.15.40:0/1174168962 wait complete. 2024-03-18T22:58:31.439 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T22:58:31.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.545+0000 7f1c8607e640 1 -- 172.21.15.40:0/4228798273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c80103530 msgr2=0x7f1c80109dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.545+0000 7f1c8607e640 1 --2- 172.21.15.40:0/4228798273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c80103530 0x7f1c80109dc0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009ca0 tx=0x7f1c7802c140 comp rx=0 tx=0).stop 2024-03-18T22:58:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.545+0000 7f1c8607e640 1 -- 172.21.15.40:0/4228798273 shutdown_connections 2024-03-18T22:58:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.545+0000 7f1c8607e640 1 --2- 172.21.15.40:0/4228798273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c80103530 0x7f1c80109dc0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.545+0000 7f1c8607e640 1 --2- 172.21.15.40:0/4228798273 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c80102b70 0x7f1c80102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.545+0000 7f1c8607e640 1 --2- 172.21.15.40:0/4228798273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c800ffb00 0x7f1c800fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.545+0000 7f1c8607e640 1 -- 172.21.15.40:0/4228798273 >> 172.21.15.40:0/4228798273 conn(0x7f1c800fb430 msgr2=0x7f1c800fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.545+0000 7f1c8607e640 1 -- 172.21.15.40:0/4228798273 shutdown_connections 2024-03-18T22:58:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.545+0000 7f1c8607e640 1 -- 172.21.15.40:0/4228798273 wait complete. 2024-03-18T22:58:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.546+0000 7f1c8607e640 1 Processor -- start 2024-03-18T22:58:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.546+0000 7f1c8607e640 1 -- start start 2024-03-18T22:58:31.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.546+0000 7f1c8607e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c800ffb00 0x7f1c80197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.546+0000 7f1c8607e640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c80102b70 0x7f1c80197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.546+0000 7f1c8607e640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c80103530 0x7f1c8019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.546+0000 7f1c8607e640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8010d360 con 0x7f1c80103530 2024-03-18T22:58:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.546+0000 7f1c8607e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8010d1e0 con 0x7f1c800ffb00 2024-03-18T22:58:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.546+0000 7f1c8607e640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8010d4e0 con 0x7f1c80102b70 2024-03-18T22:58:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.546+0000 7f1c7ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c80103530 0x7f1c8019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.546+0000 7f1c7ffff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c80103530 0x7f1c8019efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33620/0 (socket says 172.21.15.40:33620) 2024-03-18T22:58:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.546+0000 7f1c7f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c800ffb00 0x7f1c80197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.546+0000 7f1c7ffff640 1 -- 172.21.15.40:0/3477039272 learned_addr learned my addr 172.21.15.40:0/3477039272 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.546+0000 7f1c7effd640 1 --2- 172.21.15.40:0/3477039272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c80102b70 0x7f1c80197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.547+0000 7f1c7ffff640 1 -- 172.21.15.40:0/3477039272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c80102b70 msgr2=0x7f1c80197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.547+0000 7f1c7ffff640 1 --2- 172.21.15.40:0/3477039272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c80102b70 0x7f1c80197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.547+0000 7f1c7ffff640 1 -- 172.21.15.40:0/3477039272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c800ffb00 msgr2=0x7f1c80197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.547+0000 7f1c7ffff640 1 --2- 172.21.15.40:0/3477039272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c800ffb00 0x7f1c80197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.547+0000 7f1c7ffff640 1 -- 172.21.15.40:0/3477039272 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8019f620 con 0x7f1c80103530 2024-03-18T22:58:31.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.547+0000 7f1c7ffff640 1 --2- 172.21.15.40:0/3477039272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c80103530 0x7f1c8019efb0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f1c7802c750 tx=0x7f1c780328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.547+0000 7f1c7cff9640 1 -- 172.21.15.40:0/3477039272 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c78041020 con 0x7f1c80103530 2024-03-18T22:58:31.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.547+0000 7f1c8607e640 1 -- 172.21.15.40:0/3477039272 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c8019f8b0 con 0x7f1c80103530 2024-03-18T22:58:31.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.547+0000 7f1c7cff9640 1 -- 172.21.15.40:0/3477039272 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c78003e20 con 0x7f1c80103530 2024-03-18T22:58:31.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.547+0000 7f1c7cff9640 1 -- 172.21.15.40:0/3477039272 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c78039040 con 0x7f1c80103530 2024-03-18T22:58:31.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.547+0000 7f1c8607e640 1 -- 172.21.15.40:0/3477039272 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c8019fda0 con 0x7f1c80103530 2024-03-18T22:58:31.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c88c62740 1 Processor -- start 2024-03-18T22:58:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c7cff9640 1 -- 172.21.15.40:0/3477039272 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1c78044020 con 0x7f1c80103530 2024-03-18T22:58:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c88c62740 1 -- start start 2024-03-18T22:58:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c88c62740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f152f0 0x557679f15710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c88c62740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f15ce0 0x557679f1a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c88c62740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f1d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c88c62740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557679e794b0 con 0x557679f15ce0 2024-03-18T22:58:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c88c62740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557679e21d70 con 0x557679f1ae50 2024-03-18T22:58:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c88c62740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557679e75550 con 0x557679f152f0 2024-03-18T22:58:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c7f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f15ce0 0x557679f1a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c7effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f1d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c7effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f1d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48584/0 (socket says 172.21.15.40:48584) 2024-03-18T22:58:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c7f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f15ce0 0x557679f1a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33632/0 (socket says 172.21.15.40:33632) 2024-03-18T22:58:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c7effd640 1 -- 172.21.15.40:0/4131542908 learned_addr learned my addr 172.21.15.40:0/4131542908 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c7effd640 1 --2- 172.21.15.40:0/4131542908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f152f0 0x557679f15710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.550+0000 7f1c7cff9640 1 --2- 172.21.15.40:0/3477039272 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1c680a2800 0x7f1c680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.551+0000 7f1c7f7fe640 1 -- 172.21.15.40:0/4131542908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f152f0 msgr2=0x557679f15710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.551+0000 7f1c7f7fe640 1 --2- 172.21.15.40:0/4131542908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f152f0 0x557679f15710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.551+0000 7f1c7f7fe640 1 -- 172.21.15.40:0/4131542908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 msgr2=0x557679f1d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.551+0000 7f1c7f7fe640 1 --2- 172.21.15.40:0/4131542908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f1d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.551+0000 7f1c7f7fe640 1 -- 172.21.15.40:0/4131542908 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557679e44210 con 0x557679f15ce0 2024-03-18T22:58:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.551+0000 7f1c7effd640 1 --2- 172.21.15.40:0/4131542908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f1d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.551+0000 7f1c7ffff640 1 --2- 172.21.15.40:0/3477039272 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1c680a2800 0x7f1c680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.551+0000 7f1c7cff9640 1 -- 172.21.15.40:0/3477039272 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1c780f0fe0 con 0x7f1c80103530 2024-03-18T22:58:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.551+0000 7f1c7f7fe640 1 --2- 172.21.15.40:0/4131542908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f15ce0 0x557679f1a7e0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f1c70009bf0 tx=0x7f1c70011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd943d31282cd476 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.551+0000 7f1c65ffb640 1 -- 172.21.15.40:0/4131542908 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c70009da0 con 0x557679f15ce0 2024-03-18T22:58:31.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.551+0000 7f1c65ffb640 1 -- 172.21.15.40:0/4131542908 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c70005020 con 0x557679f15ce0 2024-03-18T22:58:31.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.551+0000 7f1c65ffb640 1 -- 172.21.15.40:0/4131542908 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c70004ad0 con 0x557679f15ce0 2024-03-18T22:58:31.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 -- 172.21.15.40:0/4131542908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f15ce0 msgr2=0x557679f1a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c7ffff640 1 --2- 172.21.15.40:0/3477039272 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1c680a2800 0x7f1c680a4cc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1c780eea60 tx=0x7f1c780a8040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 --2- 172.21.15.40:0/4131542908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f15ce0 0x557679f1a7e0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f1c70009bf0 tx=0x7f1c70011810 comp rx=0 tx=0).stop 2024-03-18T22:58:31.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 -- 172.21.15.40:0/4131542908 shutdown_connections 2024-03-18T22:58:31.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 --2- 172.21.15.40:0/4131542908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f1d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 --2- 172.21.15.40:0/4131542908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f15ce0 0x557679f1a7e0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 --2- 172.21.15.40:0/4131542908 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f152f0 0x557679f15710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 -- 172.21.15.40:0/4131542908 >> 172.21.15.40:0/4131542908 conn(0x557679f10860 msgr2=0x557679f10c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 -- 172.21.15.40:0/4131542908 shutdown_connections 2024-03-18T22:58:31.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 -- 172.21.15.40:0/4131542908 wait complete. 2024-03-18T22:58:31.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 -- start start 2024-03-18T22:58:31.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 Processor -- start 2024-03-18T22:58:31.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f1f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 0x557679f1fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f26d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x557679f1de00 con 0x557679f152f0 2024-03-18T22:58:31.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557679e35310 con 0x557679f1ae50 2024-03-18T22:58:31.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x557679f1df80 con 0x557679f15ce0 2024-03-18T22:58:31.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.552+0000 7f1c88c62740 10 client.0 _ll_register_callbacks cb 0x557679e292d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:31.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c7effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f1f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c7effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f1f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33642/0 (socket says 172.21.15.40:33642) 2024-03-18T22:58:31.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c7f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 0x557679f1fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c7effd640 1 -- 172.21.15.40:0/3751006058 learned_addr learned my addr 172.21.15.40:0/3751006058 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c7effd640 1 --2- 172.21.15.40:0/3751006058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f26d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c7effd640 1 -- 172.21.15.40:0/3751006058 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 msgr2=0x557679f1fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c7effd640 1 --2- 172.21.15.40:0/3751006058 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 0x557679f1fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c7effd640 1 -- 172.21.15.40:0/3751006058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 msgr2=0x557679f26d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c7effd640 1 --2- 172.21.15.40:0/3751006058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f26d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c7effd640 1 -- 172.21.15.40:0/3751006058 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557679b8d3f0 con 0x557679f152f0 2024-03-18T22:58:31.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c7effd640 1 --2- 172.21.15.40:0/3751006058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f1f810 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f1c74011d60 tx=0x7f1c740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1675bc0abd6ca4e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c5effd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:58:31.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c5effd640 1 -- 172.21.15.40:0/3751006058 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c74004030 con 0x557679f152f0 2024-03-18T22:58:31.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c5effd640 1 -- 172.21.15.40:0/3751006058 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1c740041d0 con 0x557679f152f0 2024-03-18T22:58:31.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c88c62740 1 -- 172.21.15.40:0/3751006058 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c70005020 con 0x557679f152f0 2024-03-18T22:58:31.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c5effd640 1 -- 172.21.15.40:0/3751006058 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f1c740043d0 con 0x557679f152f0 2024-03-18T22:58:31.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c88c62740 10 client.18771 fetch_fsmap learned FSMap version 49 2024-03-18T22:58:31.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c88c62740 1 -- 172.21.15.40:0/3751006058 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x557679c577d0 con 0x557679f152f0 2024-03-18T22:58:31.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.553+0000 7f1c5effd640 1 -- 172.21.15.40:0/3751006058 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c70005020 con 0x557679f152f0 2024-03-18T22:58:31.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.554+0000 7f1c5effd640 1 -- 172.21.15.40:0/3751006058 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8418+0+0 (secure 0 0 0) 0x7f1c74020670 con 0x557679f152f0 2024-03-18T22:58:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.554+0000 7f1c88c62740 10 client.18771 fetch_fsmap finished waiting for FSMap version 49 2024-03-18T22:58:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.554+0000 7f1c88c62740 10 client.18771 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:58:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.554+0000 7f1c88c62740 20 client.18771 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.554+0000 7f1c88c62740 1 --2- 172.21.15.40:0/3751006058 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557679f2b650 0x557679f2daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.554+0000 7f1c88c62740 4 client.18771 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:58:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.554+0000 7f1c88c62740 1 -- 172.21.15.40:0/3751006058 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557679ed73e0 con 0x557679f2b650 2024-03-18T22:58:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.554+0000 7f1c7f7fe640 1 --2- 172.21.15.40:0/3751006058 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557679f2b650 0x557679f2daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.554+0000 7f1c5effd640 10 client.18771 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:58:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.554+0000 7f1c7f7fe640 1 --2- 172.21.15.40:0/3751006058 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557679f2b650 0x557679f2daf0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4ab2f5292b302c12 server_cookie=9faf27b549f5ca4e in_seq=0 out_seq=0 2024-03-18T22:58:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.555+0000 7f1c5effd640 1 -- 172.21.15.40:0/3751006058 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557679ed73e0 con 0x557679f2b650 2024-03-18T22:58:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.555+0000 7f1c5effd640 10 client.18771 handle_command_reply: tid=0 2024-03-18T22:58:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.555+0000 7f1c88c62740 1 client.18771 shutdown 2024-03-18T22:58:31.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.555+0000 7f1c88c62740 1 -- 172.21.15.40:0/3751006058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 msgr2=0x557679f1f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.555+0000 7f1c88c62740 1 --2- 172.21.15.40:0/3751006058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f1f810 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f1c74011d60 tx=0x7f1c740046c0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.555+0000 7f1c88c62740 1 -- 172.21.15.40:0/3751006058 shutdown_connections 2024-03-18T22:58:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.555+0000 7f1c88c62740 1 --2- 172.21.15.40:0/3751006058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f26d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.555+0000 7f1c88c62740 1 --2- 172.21.15.40:0/3751006058 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 0x557679f1fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.555+0000 7f1c88c62740 1 --2- 172.21.15.40:0/3751006058 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557679f2b650 0x557679f2daf0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.555+0000 7f1c88c62740 1 --2- 172.21.15.40:0/3751006058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f1f810 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.555+0000 7f1c5effd640 0 client.18771 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:58:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.555+0000 7f1c88c62740 1 -- 172.21.15.40:0/3751006058 >> 172.21.15.40:0/3751006058 conn(0x557679f10860 msgr2=0x557679f0fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.555+0000 7f1c88c62740 1 -- 172.21.15.40:0/3751006058 shutdown_connections 2024-03-18T22:58:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.555+0000 7f1c88c62740 1 -- 172.21.15.40:0/3751006058 wait complete. 2024-03-18T22:58:31.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.556+0000 7f1c88c62740 20 client.18771 trim_cache size 0 max 16384 2024-03-18T22:58:31.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c88c62740 1 Processor -- start 2024-03-18T22:58:31.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c88c62740 1 -- start start 2024-03-18T22:58:31.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c88c62740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f1fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c88c62740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 0x557679f26d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c88c62740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f0fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c88c62740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1c801aa580 con 0x557679f152f0 2024-03-18T22:58:31.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c88c62740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c801ac2a0 con 0x557679f1ae50 2024-03-18T22:58:31.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c88c62740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1c801aac50 con 0x557679f15ce0 2024-03-18T22:58:31.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c7effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f1fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c7effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f1fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33652/0 (socket says 172.21.15.40:33652) 2024-03-18T22:58:31.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c7effd640 1 -- 172.21.15.40:0/3075490757 learned_addr learned my addr 172.21.15.40:0/3075490757 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c7f7fe640 1 --2- 172.21.15.40:0/3075490757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 0x557679f26d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c7effd640 1 --2- 172.21.15.40:0/3075490757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f0fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c7effd640 1 -- 172.21.15.40:0/3075490757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 msgr2=0x557679f26d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c7effd640 1 --2- 172.21.15.40:0/3075490757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 0x557679f26d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c7effd640 1 -- 172.21.15.40:0/3075490757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 msgr2=0x557679f0fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c7effd640 1 --2- 172.21.15.40:0/3075490757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f0fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c7effd640 1 -- 172.21.15.40:0/3075490757 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557679ee6880 con 0x557679f152f0 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c7f7fe640 1 --2- 172.21.15.40:0/3075490757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 0x557679f26d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c7effd640 1 --2- 172.21.15.40:0/3075490757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f1fcb0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f1c740059f0 tx=0x7f1c74001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85822a3c33667a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c657fa640 1 -- 172.21.15.40:0/3075490757 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c74020470 con 0x557679f152f0 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c657fa640 1 -- 172.21.15.40:0/3075490757 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c740054f0 con 0x557679f152f0 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.569+0000 7f1c657fa640 1 -- 172.21.15.40:0/3075490757 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c74004730 con 0x557679f152f0 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.570+0000 7f1c88c62740 1 -- 172.21.15.40:0/3075490757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 msgr2=0x557679f1fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.570+0000 7f1c88c62740 1 --2- 172.21.15.40:0/3075490757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f1fcb0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f1c740059f0 tx=0x7f1c74001e70 comp rx=0 tx=0).stop 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.570+0000 7f1c88c62740 1 -- 172.21.15.40:0/3075490757 shutdown_connections 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.570+0000 7f1c88c62740 1 --2- 172.21.15.40:0/3075490757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f0fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.570+0000 7f1c88c62740 1 --2- 172.21.15.40:0/3075490757 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 0x557679f26d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.570+0000 7f1c88c62740 1 --2- 172.21.15.40:0/3075490757 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f1fcb0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.570+0000 7f1c88c62740 1 -- 172.21.15.40:0/3075490757 >> 172.21.15.40:0/3075490757 conn(0x557679f10860 msgr2=0x557679f0f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.570+0000 7f1c88c62740 1 -- 172.21.15.40:0/3075490757 shutdown_connections 2024-03-18T22:58:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.570+0000 7f1c88c62740 1 -- 172.21.15.40:0/3075490757 wait complete. 2024-03-18T22:58:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.570+0000 7f1c88c62740 1 -- start start 2024-03-18T22:58:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c88c62740 1 Processor -- start 2024-03-18T22:58:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c88c62740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f24690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c88c62740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 0x557679f24bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c88c62740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f25b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c88c62740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1c801abb10 con 0x557679f152f0 2024-03-18T22:58:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c88c62740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1c801ab3e0 con 0x557679f1ae50 2024-03-18T22:58:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c88c62740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1c801a8500 con 0x557679f15ce0 2024-03-18T22:58:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c88c62740 10 client.0 _ll_register_callbacks cb 0x7f1c74009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c7effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f24690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c7effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f24690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33664/0 (socket says 172.21.15.40:33664) 2024-03-18T22:58:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c7f7fe640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 0x557679f24bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c7effd640 1 -- 172.21.15.40:0/2804590164 learned_addr learned my addr 172.21.15.40:0/2804590164 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c7effd640 1 --2- 172.21.15.40:0/2804590164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f25b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c7effd640 1 -- 172.21.15.40:0/2804590164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 msgr2=0x557679f24bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c7effd640 1 --2- 172.21.15.40:0/2804590164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 0x557679f24bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c7effd640 1 -- 172.21.15.40:0/2804590164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 msgr2=0x557679f25b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c7effd640 1 --2- 172.21.15.40:0/2804590164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f25b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c7effd640 1 -- 172.21.15.40:0/2804590164 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557679f26180 con 0x557679f152f0 2024-03-18T22:58:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c7effd640 1 --2- 172.21.15.40:0/2804590164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f24690 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009800 tx=0x7f1c74011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c837c9d46ddf08 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c5effd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:58:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c5effd640 1 -- 172.21.15.40:0/2804590164 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7403f020 con 0x557679f152f0 2024-03-18T22:58:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.571+0000 7f1c5effd640 1 -- 172.21.15.40:0/2804590164 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1c74020760 con 0x557679f152f0 2024-03-18T22:58:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.572+0000 7f1c88c62740 1 -- 172.21.15.40:0/2804590164 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c740054f0 con 0x557679f152f0 2024-03-18T22:58:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.572+0000 7f1c5effd640 1 -- 172.21.15.40:0/2804590164 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f1c74020470 con 0x557679f152f0 2024-03-18T22:58:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.572+0000 7f1c88c62740 10 client.18777 fetch_fsmap learned FSMap version 49 2024-03-18T22:58:31.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.572+0000 7f1c88c62740 1 -- 172.21.15.40:0/2804590164 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x557679f91b00 con 0x557679f152f0 2024-03-18T22:58:31.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.572+0000 7f1c5effd640 1 -- 172.21.15.40:0/2804590164 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c740054f0 con 0x557679f152f0 2024-03-18T22:58:31.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.572+0000 7f1c5effd640 1 -- 172.21.15.40:0/2804590164 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8418+0+0 (secure 0 0 0) 0x7f1c7400fbb0 con 0x557679f152f0 2024-03-18T22:58:31.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.572+0000 7f1c88c62740 10 client.18777 fetch_fsmap finished waiting for FSMap version 49 2024-03-18T22:58:31.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.572+0000 7f1c88c62740 10 client.18777 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-18T22:58:31.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.572+0000 7f1c88c62740 20 client.18777 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:31.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.572+0000 7f1c88c62740 1 --2- 172.21.15.40:0/2804590164 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557679f92480 0x557679f948a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.572+0000 7f1c88c62740 4 client.18777 mds_command: new command op to 24388 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:58:31.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.572+0000 7f1c88c62740 1 -- 172.21.15.40:0/2804590164 --> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557679e869f0 con 0x557679f92480 2024-03-18T22:58:31.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.572+0000 7f1c7f7fe640 1 --2- 172.21.15.40:0/2804590164 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557679f92480 0x557679f948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c5effd640 10 client.18777 ms_handle_connect on v2:172.21.15.113:6832/2086888629 2024-03-18T22:58:31.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c7f7fe640 1 --2- 172.21.15.40:0/2804590164 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557679f92480 0x557679f948a0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=83f34d4624e17b20 server_cookie=881079a54304efce in_seq=0 out_seq=0 2024-03-18T22:58:31.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c5effd640 1 -- 172.21.15.40:0/2804590164 <== mds.4 v2:172.21.15.113:6832/2086888629 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557679e869f0 con 0x557679f92480 2024-03-18T22:58:31.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c5effd640 10 client.18777 handle_command_reply: tid=0 2024-03-18T22:58:31.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c88c62740 1 client.18777 shutdown 2024-03-18T22:58:31.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c88c62740 1 -- 172.21.15.40:0/2804590164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 msgr2=0x557679f24690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c88c62740 1 --2- 172.21.15.40:0/2804590164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f24690 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009800 tx=0x7f1c74011a60 comp rx=0 tx=0).stop 2024-03-18T22:58:31.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c88c62740 1 -- 172.21.15.40:0/2804590164 shutdown_connections 2024-03-18T22:58:31.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c88c62740 1 --2- 172.21.15.40:0/2804590164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557679f1ae50 0x557679f25b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c88c62740 1 --2- 172.21.15.40:0/2804590164 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x557679f15ce0 0x557679f24bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c88c62740 1 --2- 172.21.15.40:0/2804590164 >> [v2:172.21.15.113:6832/2086888629,v1:172.21.15.113:6833/2086888629] conn(0x557679f92480 0x557679f948a0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c88c62740 1 --2- 172.21.15.40:0/2804590164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557679f152f0 0x557679f24690 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c5effd640 0 client.18777 ms_handle_reset on v2:172.21.15.113:6832/2086888629 2024-03-18T22:58:31.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c88c62740 1 -- 172.21.15.40:0/2804590164 >> 172.21.15.40:0/2804590164 conn(0x557679f10860 msgr2=0x557679f1fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c88c62740 1 -- 172.21.15.40:0/2804590164 shutdown_connections 2024-03-18T22:58:31.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c88c62740 1 -- 172.21.15.40:0/2804590164 wait complete. 2024-03-18T22:58:31.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.573+0000 7f1c88c62740 20 client.18777 trim_cache size 0 max 16384 2024-03-18T22:58:31.581 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:58:31.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.574+0000 7f1c8607e640 1 -- 172.21.15.40:0/3477039272 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1c680a2800 msgr2=0x7f1c680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.574+0000 7f1c8607e640 1 --2- 172.21.15.40:0/3477039272 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1c680a2800 0x7f1c680a4cc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1c780eea60 tx=0x7f1c780a8040 comp rx=0 tx=0).stop 2024-03-18T22:58:31.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.574+0000 7f1c8607e640 1 -- 172.21.15.40:0/3477039272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c80103530 msgr2=0x7f1c8019efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.574+0000 7f1c8607e640 1 --2- 172.21.15.40:0/3477039272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c80103530 0x7f1c8019efb0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f1c7802c750 tx=0x7f1c780328a0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.574+0000 7f1c8607e640 1 -- 172.21.15.40:0/3477039272 shutdown_connections 2024-03-18T22:58:31.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.574+0000 7f1c8607e640 1 --2- 172.21.15.40:0/3477039272 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1c680a2800 0x7f1c680a4cc0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.574+0000 7f1c8607e640 1 --2- 172.21.15.40:0/3477039272 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1c80103530 0x7f1c8019efb0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.574+0000 7f1c8607e640 1 --2- 172.21.15.40:0/3477039272 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1c80102b70 0x7f1c80197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.574+0000 7f1c8607e640 1 --2- 172.21.15.40:0/3477039272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1c800ffb00 0x7f1c80197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.574+0000 7f1c8607e640 1 -- 172.21.15.40:0/3477039272 >> 172.21.15.40:0/3477039272 conn(0x7f1c800fb430 msgr2=0x7f1c80105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.574+0000 7f1c8607e640 1 -- 172.21.15.40:0/3477039272 shutdown_connections 2024-03-18T22:58:31.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.574+0000 7f1c8607e640 1 -- 172.21.15.40:0/3477039272 wait complete. 2024-03-18T22:58:31.585 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T22:58:31.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.689+0000 7f8dce7cd640 1 -- 172.21.15.40:0/1618358286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc8101910 msgr2=0x7f8dc8101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.689+0000 7f8dce7cd640 1 --2- 172.21.15.40:0/1618358286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc8101910 0x7f8dc8101d90 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc009900 tx=0x7f8dbc02bc30 comp rx=0 tx=0).stop 2024-03-18T22:58:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.689+0000 7f8dce7cd640 1 -- 172.21.15.40:0/1618358286 shutdown_connections 2024-03-18T22:58:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.689+0000 7f8dce7cd640 1 --2- 172.21.15.40:0/1618358286 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8dc81022d0 0x7f8dc8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.689+0000 7f8dce7cd640 1 --2- 172.21.15.40:0/1618358286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc8101910 0x7f8dc8101d90 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.689+0000 7f8dce7cd640 1 --2- 172.21.15.40:0/1618358286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8dc8100720 0x7f8dc8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.689+0000 7f8dce7cd640 1 -- 172.21.15.40:0/1618358286 >> 172.21.15.40:0/1618358286 conn(0x7f8dc80fbc90 msgr2=0x7f8dc80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.689+0000 7f8dce7cd640 1 -- 172.21.15.40:0/1618358286 shutdown_connections 2024-03-18T22:58:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.689+0000 7f8dce7cd640 1 -- 172.21.15.40:0/1618358286 wait complete. 2024-03-18T22:58:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dce7cd640 1 Processor -- start 2024-03-18T22:58:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dce7cd640 1 -- start start 2024-03-18T22:58:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dce7cd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8dc8100720 0x7f8dc8197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dce7cd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc8101910 0x7f8dc8197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dce7cd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8dc81022d0 0x7f8dc819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dce7cd640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc810c230 con 0x7f8dc8101910 2024-03-18T22:58:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dce7cd640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc810c0b0 con 0x7f8dc8100720 2024-03-18T22:58:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dce7cd640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc810c3b0 con 0x7f8dc81022d0 2024-03-18T22:58:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dc77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc8101910 0x7f8dc8197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dc77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc8101910 0x7f8dc8197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33672/0 (socket says 172.21.15.40:33672) 2024-03-18T22:58:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dc77fe640 1 -- 172.21.15.40:0/1832266904 learned_addr learned my addr 172.21.15.40:0/1832266904 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dc7fff640 1 --2- 172.21.15.40:0/1832266904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8dc8100720 0x7f8dc8197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dccd6a640 1 --2- 172.21.15.40:0/1832266904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8dc81022d0 0x7f8dc819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dc77fe640 1 -- 172.21.15.40:0/1832266904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8dc81022d0 msgr2=0x7f8dc819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dc77fe640 1 --2- 172.21.15.40:0/1832266904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8dc81022d0 0x7f8dc819f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dc77fe640 1 -- 172.21.15.40:0/1832266904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8dc8100720 msgr2=0x7f8dc8197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dc77fe640 1 --2- 172.21.15.40:0/1832266904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8dc8100720 0x7f8dc8197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.690+0000 7f8dc77fe640 1 -- 172.21.15.40:0/1832266904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc819f670 con 0x7f8dc8101910 2024-03-18T22:58:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.691+0000 7f8dc77fe640 1 --2- 172.21.15.40:0/1832266904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc8101910 0x7f8dc8197f90 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc006010 tx=0x7f8dbc005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.691+0000 7f8dc57fa640 1 -- 172.21.15.40:0/1832266904 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc043020 con 0x7f8dc8101910 2024-03-18T22:58:31.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.691+0000 7f8dc57fa640 1 -- 172.21.15.40:0/1832266904 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8dbc0043d0 con 0x7f8dc8101910 2024-03-18T22:58:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.691+0000 7f8dce7cd640 1 -- 172.21.15.40:0/1832266904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dc819f900 con 0x7f8dc8101910 2024-03-18T22:58:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.691+0000 7f8dc57fa640 1 -- 172.21.15.40:0/1832266904 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc003b80 con 0x7f8dc8101910 2024-03-18T22:58:31.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.691+0000 7f8dce7cd640 1 -- 172.21.15.40:0/1832266904 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dc819fe50 con 0x7f8dc8101910 2024-03-18T22:58:31.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.693+0000 7f8dc57fa640 1 -- 172.21.15.40:0/1832266904 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8dbc046020 con 0x7f8dc8101910 2024-03-18T22:58:31.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.693+0000 7f8dd13b9740 1 Processor -- start 2024-03-18T22:58:31.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.693+0000 7f8dd13b9740 1 -- start start 2024-03-18T22:58:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.693+0000 7f8dc57fa640 1 --2- 172.21.15.40:0/1832266904 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8db00a2800 0x7f8db00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dd13b9740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f32f0 0x55c4fe5f3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dd13b9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe5f87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dd13b9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f8e50 0x55c4fe5fb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dd13b9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c4fe5574b0 con 0x55c4fe5f8e50 2024-03-18T22:58:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dd13b9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c4fe4ffd70 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dd13b9740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c4fe553550 con 0x55c4fe5f32f0 2024-03-18T22:58:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dc57fa640 1 -- 172.21.15.40:0/1832266904 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8dbc0f1040 con 0x7f8dc8101910 2024-03-18T22:58:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dc7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f32f0 0x55c4fe5f3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dccd6a640 1 --2- 172.21.15.40:0/1832266904 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8db00a2800 0x7f8db00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dc7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f32f0 0x55c4fe5f3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38092/0 (socket says 172.21.15.40:38092) 2024-03-18T22:58:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dc77fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe5f87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dc7fff640 1 -- 172.21.15.40:0/2467357610 learned_addr learned my addr 172.21.15.40:0/2467357610 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dccd6a640 1 --2- 172.21.15.40:0/2467357610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f8e50 0x55c4fe5fb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dc7fff640 1 -- 172.21.15.40:0/2467357610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f3ce0 msgr2=0x55c4fe5f87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dc7fff640 1 --2- 172.21.15.40:0/2467357610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe5f87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dc7fff640 1 -- 172.21.15.40:0/2467357610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f8e50 msgr2=0x55c4fe5fb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dc7fff640 1 --2- 172.21.15.40:0/2467357610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f8e50 0x55c4fe5fb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dc7fff640 1 -- 172.21.15.40:0/2467357610 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4fe522210 con 0x55c4fe5f32f0 2024-03-18T22:58:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dc77fe640 1 --2- 172.21.15.40:0/2467357610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe5f87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dccd6a640 1 --2- 172.21.15.40:0/1832266904 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8db00a2800 0x7f8db00a4cc0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0005fd0 tx=0x7f8dc0009f10 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dc7fff640 1 --2- 172.21.15.40:0/2467357610 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f32f0 0x55c4fe5f3710 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009c80 tx=0x7f8db8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2093abf07bac2816 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dae7fc640 1 -- 172.21.15.40:0/2467357610 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db801e070 con 0x55c4fe5f32f0 2024-03-18T22:58:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.694+0000 7f8dae7fc640 1 -- 172.21.15.40:0/2467357610 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8db8009e30 con 0x55c4fe5f32f0 2024-03-18T22:58:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.695+0000 7f8dae7fc640 1 -- 172.21.15.40:0/2467357610 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db800e940 con 0x55c4fe5f32f0 2024-03-18T22:58:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.695+0000 7f8dd13b9740 1 -- 172.21.15.40:0/2467357610 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f32f0 msgr2=0x55c4fe5f3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.695+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/2467357610 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f32f0 0x55c4fe5f3710 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009c80 tx=0x7f8db8011810 comp rx=0 tx=0).stop 2024-03-18T22:58:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.695+0000 7f8dd13b9740 1 -- 172.21.15.40:0/2467357610 shutdown_connections 2024-03-18T22:58:31.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.695+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/2467357610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f8e50 0x55c4fe5fb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.695+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/2467357610 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe5f87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.695+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/2467357610 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f32f0 0x55c4fe5f3710 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.695+0000 7f8dd13b9740 1 -- 172.21.15.40:0/2467357610 >> 172.21.15.40:0/2467357610 conn(0x55c4fe5ee860 msgr2=0x55c4fe5eec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.695+0000 7f8dd13b9740 1 -- 172.21.15.40:0/2467357610 shutdown_connections 2024-03-18T22:58:31.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.695+0000 7f8dd13b9740 1 -- 172.21.15.40:0/2467357610 wait complete. 2024-03-18T22:58:31.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.695+0000 7f8dd13b9740 1 -- start start 2024-03-18T22:58:31.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dd13b9740 1 Processor -- start 2024-03-18T22:58:31.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dd13b9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f32f0 0x55c4fe5fd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dd13b9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe5fdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dd13b9740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f8e50 0x55c4fe604d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dd13b9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55c4fe5fbe00 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dd13b9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c4fe513310 con 0x55c4fe5f32f0 2024-03-18T22:58:31.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dd13b9740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55c4fe5fbf80 con 0x55c4fe5f8e50 2024-03-18T22:58:31.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dd13b9740 10 client.0 _ll_register_callbacks cb 0x55c4fe5072d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dc77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe5fdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dc77fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe5fdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33690/0 (socket says 172.21.15.40:33690) 2024-03-18T22:58:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dc77fe640 1 -- 172.21.15.40:0/1304813592 learned_addr learned my addr 172.21.15.40:0/1304813592 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dc7fff640 1 --2- 172.21.15.40:0/1304813592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f32f0 0x55c4fe5fd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dccd6a640 1 --2- 172.21.15.40:0/1304813592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f8e50 0x55c4fe604d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dc77fe640 1 -- 172.21.15.40:0/1304813592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f8e50 msgr2=0x55c4fe604d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dc77fe640 1 --2- 172.21.15.40:0/1304813592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f8e50 0x55c4fe604d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dc77fe640 1 -- 172.21.15.40:0/1304813592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f32f0 msgr2=0x55c4fe5fd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dc77fe640 1 --2- 172.21.15.40:0/1304813592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f32f0 0x55c4fe5fd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dc77fe640 1 -- 172.21.15.40:0/1304813592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4fe26b3f0 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8dc77fe640 1 --2- 172.21.15.40:0/1304813592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe5fdcf0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc0eeac0 tx=0x7f8dbc0bd1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49320c1a44aa1f31 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.696+0000 7f8da77fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:58:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.697+0000 7f8da77fe640 1 -- 172.21.15.40:0/1304813592 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc0bdcf0 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.697+0000 7f8da77fe640 1 -- 172.21.15.40:0/1304813592 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8dbc003d20 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.697+0000 7f8dd13b9740 1 -- 172.21.15.40:0/1304813592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8db8009e30 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.697+0000 7f8da77fe640 1 -- 172.21.15.40:0/1304813592 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f8dbc0916e0 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.697+0000 7f8dd13b9740 10 client.18789 fetch_fsmap learned FSMap version 49 2024-03-18T22:58:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.697+0000 7f8dd13b9740 1 -- 172.21.15.40:0/1304813592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55c4fe3357d0 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.697+0000 7f8da77fe640 1 -- 172.21.15.40:0/1304813592 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8db8009e30 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.697+0000 7f8da77fe640 1 -- 172.21.15.40:0/1304813592 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8418+0+0 (secure 0 0 0) 0x7f8dbc0c5900 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.697+0000 7f8dd13b9740 10 client.18789 fetch_fsmap finished waiting for FSMap version 49 2024-03-18T22:58:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.697+0000 7f8dd13b9740 10 client.18789 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:58:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.697+0000 7f8dd13b9740 20 client.18789 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.697+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/1304813592 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55c4fe609510 0x55c4fe60b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.697+0000 7f8dd13b9740 4 client.18789 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:58:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.697+0000 7f8dd13b9740 1 -- 172.21.15.40:0/1304813592 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4fe5b53e0 con 0x55c4fe609510 2024-03-18T22:58:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.697+0000 7f8dc7fff640 1 --2- 172.21.15.40:0/1304813592 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55c4fe609510 0x55c4fe60b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.698+0000 7f8da77fe640 10 client.18789 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:58:31.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.698+0000 7f8dc7fff640 1 --2- 172.21.15.40:0/1304813592 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55c4fe609510 0x55c4fe60b9b0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5e919a7c5825c517 server_cookie=f9df83211ab9dea1 in_seq=0 out_seq=0 2024-03-18T22:58:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.698+0000 7f8da77fe640 1 -- 172.21.15.40:0/1304813592 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c4fe5b53e0 con 0x55c4fe609510 2024-03-18T22:58:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.698+0000 7f8da77fe640 10 client.18789 handle_command_reply: tid=0 2024-03-18T22:58:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.698+0000 7f8dd13b9740 1 client.18789 shutdown 2024-03-18T22:58:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.699+0000 7f8dd13b9740 1 -- 172.21.15.40:0/1304813592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f3ce0 msgr2=0x55c4fe5fdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.699+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/1304813592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe5fdcf0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc0eeac0 tx=0x7f8dbc0bd1f0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.699+0000 7f8dd13b9740 1 -- 172.21.15.40:0/1304813592 shutdown_connections 2024-03-18T22:58:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.699+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/1304813592 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55c4fe609510 0x55c4fe60b9b0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.699+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/1304813592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f8e50 0x55c4fe604d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.699+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/1304813592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe5fdcf0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.699+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/1304813592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f32f0 0x55c4fe5fd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.699+0000 7f8da77fe640 0 client.18789 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:58:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.699+0000 7f8dd13b9740 1 -- 172.21.15.40:0/1304813592 >> 172.21.15.40:0/1304813592 conn(0x55c4fe5ee860 msgr2=0x55c4fe5edba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.699+0000 7f8dd13b9740 1 -- 172.21.15.40:0/1304813592 shutdown_connections 2024-03-18T22:58:31.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.699+0000 7f8dd13b9740 1 -- 172.21.15.40:0/1304813592 wait complete. 2024-03-18T22:58:31.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.699+0000 7f8dd13b9740 20 client.18789 trim_cache size 0 max 16384 2024-03-18T22:58:31.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dd13b9740 1 Processor -- start 2024-03-18T22:58:31.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dd13b9740 1 -- start start 2024-03-18T22:58:31.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dd13b9740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5ee860 0x55c4fe5f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dd13b9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f32f0 0x55c4fe604cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dd13b9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe6102b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dd13b9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc81aa4e0 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dd13b9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc81ac200 con 0x55c4fe5f32f0 2024-03-18T22:58:31.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dd13b9740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc81aabb0 con 0x55c4fe5ee860 2024-03-18T22:58:31.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dccd6a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe6102b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dc77fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f32f0 0x55c4fe604cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dc77fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f32f0 0x55c4fe604cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48652/0 (socket says 172.21.15.40:48652) 2024-03-18T22:58:31.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dc7fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5ee860 0x55c4fe5f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dc77fe640 1 -- 172.21.15.40:0/2767994718 learned_addr learned my addr 172.21.15.40:0/2767994718 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dccd6a640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe6102b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33698/0 (socket says 172.21.15.40:33698) 2024-03-18T22:58:31.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dccd6a640 1 -- 172.21.15.40:0/2767994718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5ee860 msgr2=0x55c4fe5f21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dccd6a640 1 --2- 172.21.15.40:0/2767994718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5ee860 0x55c4fe5f21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dccd6a640 1 -- 172.21.15.40:0/2767994718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f32f0 msgr2=0x55c4fe604cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dccd6a640 1 --2- 172.21.15.40:0/2767994718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f32f0 0x55c4fe604cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.712+0000 7f8dccd6a640 1 -- 172.21.15.40:0/2767994718 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4fe5c4880 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.713+0000 7f8dccd6a640 1 --2- 172.21.15.40:0/2767994718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe6102b0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0005bd0 tx=0x7f8dc0021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=733221f573e02fdf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.713+0000 7f8dadffb640 1 -- 172.21.15.40:0/2767994718 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc001ed90 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.713+0000 7f8dadffb640 1 -- 172.21.15.40:0/2767994718 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8dc0003e20 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.713+0000 7f8dadffb640 1 -- 172.21.15.40:0/2767994718 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0004560 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.713+0000 7f8dd13b9740 1 -- 172.21.15.40:0/2767994718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f3ce0 msgr2=0x55c4fe6102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.713+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/2767994718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe6102b0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0005bd0 tx=0x7f8dc0021000 comp rx=0 tx=0).stop 2024-03-18T22:58:31.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.713+0000 7f8dd13b9740 1 -- 172.21.15.40:0/2767994718 shutdown_connections 2024-03-18T22:58:31.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.713+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/2767994718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe6102b0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.713+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/2767994718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f32f0 0x55c4fe604cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.713+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/2767994718 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5ee860 0x55c4fe5f21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.713+0000 7f8dd13b9740 1 -- 172.21.15.40:0/2767994718 >> 172.21.15.40:0/2767994718 conn(0x55c4fe5f4ac0 msgr2=0x55c4fe5f1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.713+0000 7f8dd13b9740 1 -- 172.21.15.40:0/2767994718 shutdown_connections 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.713+0000 7f8dd13b9740 1 -- 172.21.15.40:0/2767994718 wait complete. 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dd13b9740 1 -- start start 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dd13b9740 1 Processor -- start 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dd13b9740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5ee860 0x55c4fe602d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dd13b9740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f32f0 0x55c4fe603290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dd13b9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe6042a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dd13b9740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc81aba70 con 0x55c4fe5ee860 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dd13b9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc81ab340 con 0x55c4fe5f3ce0 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dd13b9740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc81a8460 con 0x55c4fe5f32f0 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dd13b9740 10 client.0 _ll_register_callbacks cb 0x7f8dbc009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dc7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5ee860 0x55c4fe602d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dc7fff640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5ee860 0x55c4fe602d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33702/0 (socket says 172.21.15.40:33702) 2024-03-18T22:58:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dc7fff640 1 -- 172.21.15.40:0/418562190 learned_addr learned my addr 172.21.15.40:0/418562190 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dc77fe640 1 --2- 172.21.15.40:0/418562190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f32f0 0x55c4fe603290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dccd6a640 1 --2- 172.21.15.40:0/418562190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe6042a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dc7fff640 1 -- 172.21.15.40:0/418562190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f32f0 msgr2=0x55c4fe603290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dc7fff640 1 --2- 172.21.15.40:0/418562190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f32f0 0x55c4fe603290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dc7fff640 1 -- 172.21.15.40:0/418562190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f3ce0 msgr2=0x55c4fe6042a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.714+0000 7f8dc7fff640 1 --2- 172.21.15.40:0/418562190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe6042a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8dc7fff640 1 -- 172.21.15.40:0/418562190 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4fe604860 con 0x55c4fe5ee860 2024-03-18T22:58:31.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8dc7fff640 1 --2- 172.21.15.40:0/418562190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5ee860 0x55c4fe602d50 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009ba0 tx=0x7f8db8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bd24600e49c868b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8da77fe640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:58:31.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8da77fe640 1 -- 172.21.15.40:0/418562190 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db8031020 con 0x55c4fe5ee860 2024-03-18T22:58:31.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8da77fe640 1 -- 172.21.15.40:0/418562190 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8db802d350 con 0x55c4fe5ee860 2024-03-18T22:58:31.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8dd13b9740 1 -- 172.21.15.40:0/418562190 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8dc0003e20 con 0x55c4fe5ee860 2024-03-18T22:58:31.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8da77fe640 1 -- 172.21.15.40:0/418562190 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f8db803c050 con 0x55c4fe5ee860 2024-03-18T22:58:31.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8dd13b9740 10 client.18795 fetch_fsmap learned FSMap version 49 2024-03-18T22:58:31.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8dd13b9740 1 -- 172.21.15.40:0/418562190 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55c4fe6701b0 con 0x55c4fe5ee860 2024-03-18T22:58:31.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8da77fe640 1 -- 172.21.15.40:0/418562190 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8dc0003e20 con 0x55c4fe5ee860 2024-03-18T22:58:31.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8da77fe640 1 -- 172.21.15.40:0/418562190 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8418+0+0 (secure 0 0 0) 0x7f8db8024040 con 0x55c4fe5ee860 2024-03-18T22:58:31.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8dd13b9740 10 client.18795 fetch_fsmap finished waiting for FSMap version 49 2024-03-18T22:58:31.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8dd13b9740 10 client.18795 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-18T22:58:31.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8dd13b9740 20 client.18795 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:31.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/418562190 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55c4fe5f8e50 0x55c4fe672a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8dd13b9740 4 client.18795 mds_command: new command op to 24401 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:58:31.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.715+0000 7f8dd13b9740 1 -- 172.21.15.40:0/418562190 --> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c4fe5649f0 con 0x55c4fe5f8e50 2024-03-18T22:58:31.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8dc77fe640 1 --2- 172.21.15.40:0/418562190 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55c4fe5f8e50 0x55c4fe672a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8da77fe640 10 client.18795 ms_handle_connect on v2:172.21.15.103:6832/3761932784 2024-03-18T22:58:31.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8dc77fe640 1 --2- 172.21.15.40:0/418562190 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55c4fe5f8e50 0x55c4fe672a30 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=709eff7d6ab4ce8b server_cookie=5843916496efd5af in_seq=0 out_seq=0 2024-03-18T22:58:31.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8da77fe640 1 -- 172.21.15.40:0/418562190 <== mds.3 v2:172.21.15.103:6832/3761932784 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c4fe5649f0 con 0x55c4fe5f8e50 2024-03-18T22:58:31.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8da77fe640 10 client.18795 handle_command_reply: tid=0 2024-03-18T22:58:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8dd13b9740 1 client.18795 shutdown 2024-03-18T22:58:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8dd13b9740 1 -- 172.21.15.40:0/418562190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5ee860 msgr2=0x55c4fe602d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/418562190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5ee860 0x55c4fe602d50 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009ba0 tx=0x7f8db8009310 comp rx=0 tx=0).stop 2024-03-18T22:58:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8dd13b9740 1 -- 172.21.15.40:0/418562190 shutdown_connections 2024-03-18T22:58:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/418562190 >> [v2:172.21.15.103:6832/3761932784,v1:172.21.15.103:6833/3761932784] conn(0x55c4fe5f8e50 0x55c4fe672a30 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/418562190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c4fe5f3ce0 0x55c4fe6042a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/418562190 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55c4fe5f32f0 0x55c4fe603290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8da77fe640 0 client.18795 ms_handle_reset on v2:172.21.15.103:6832/3761932784 2024-03-18T22:58:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8dd13b9740 1 --2- 172.21.15.40:0/418562190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c4fe5ee860 0x55c4fe602d50 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8dd13b9740 1 -- 172.21.15.40:0/418562190 >> 172.21.15.40:0/418562190 conn(0x55c4fe5f4ac0 msgr2=0x55c4fe5f2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8dd13b9740 1 -- 172.21.15.40:0/418562190 shutdown_connections 2024-03-18T22:58:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.716+0000 7f8dd13b9740 1 -- 172.21.15.40:0/418562190 wait complete. 2024-03-18T22:58:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.717+0000 7f8dd13b9740 20 client.18795 trim_cache size 0 max 16384 2024-03-18T22:58:31.724 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:58:31.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.717+0000 7f8dce7cd640 1 -- 172.21.15.40:0/1832266904 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8db00a2800 msgr2=0x7f8db00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.717+0000 7f8dce7cd640 1 --2- 172.21.15.40:0/1832266904 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8db00a2800 0x7f8db00a4cc0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0005fd0 tx=0x7f8dc0009f10 comp rx=0 tx=0).stop 2024-03-18T22:58:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.717+0000 7f8dce7cd640 1 -- 172.21.15.40:0/1832266904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc8101910 msgr2=0x7f8dc8197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.717+0000 7f8dce7cd640 1 --2- 172.21.15.40:0/1832266904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc8101910 0x7f8dc8197f90 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc006010 tx=0x7f8dbc005e40 comp rx=0 tx=0).stop 2024-03-18T22:58:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.717+0000 7f8dce7cd640 1 -- 172.21.15.40:0/1832266904 shutdown_connections 2024-03-18T22:58:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.717+0000 7f8dce7cd640 1 --2- 172.21.15.40:0/1832266904 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8db00a2800 0x7f8db00a4cc0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.717+0000 7f8dce7cd640 1 --2- 172.21.15.40:0/1832266904 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8dc81022d0 0x7f8dc819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.717+0000 7f8dce7cd640 1 --2- 172.21.15.40:0/1832266904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dc8101910 0x7f8dc8197f90 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.717+0000 7f8dce7cd640 1 --2- 172.21.15.40:0/1832266904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8dc8100720 0x7f8dc8197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.717+0000 7f8dce7cd640 1 -- 172.21.15.40:0/1832266904 >> 172.21.15.40:0/1832266904 conn(0x7f8dc80fbc90 msgr2=0x7f8dc80fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.717+0000 7f8dce7cd640 1 -- 172.21.15.40:0/1832266904 shutdown_connections 2024-03-18T22:58:31.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.717+0000 7f8dce7cd640 1 -- 172.21.15.40:0/1832266904 wait complete. 2024-03-18T22:58:31.727 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:31 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1174168962' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:58:31.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T22:58:31.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:31 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1174168962' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:58:31.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 -- 172.21.15.40:0/319556347 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2581033d0 msgr2=0x7fc258109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 --2- 172.21.15.40:0/319556347 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2581033d0 0x7fc258109c60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc248009be0 tx=0x7fc24802bef0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 -- 172.21.15.40:0/319556347 shutdown_connections 2024-03-18T22:58:31.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 --2- 172.21.15.40:0/319556347 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc2581033d0 0x7fc258109c60 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 --2- 172.21.15.40:0/319556347 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258102a10 0x7fc258102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 --2- 172.21.15.40:0/319556347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc258101820 0x7fc258101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 -- 172.21.15.40:0/319556347 >> 172.21.15.40:0/319556347 conn(0x7fc2580fcd90 msgr2=0x7fc2580ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 -- 172.21.15.40:0/319556347 shutdown_connections 2024-03-18T22:58:31.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 -- 172.21.15.40:0/319556347 wait complete. 2024-03-18T22:58:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 Processor -- start 2024-03-18T22:58:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 -- start start 2024-03-18T22:58:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc258101820 0x7fc2581979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258102a10 0x7fc258197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2581033d0 0x7fc25819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc25810d330 con 0x7fc258102a10 2024-03-18T22:58:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc25810d1b0 con 0x7fc2581033d0 2024-03-18T22:58:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25f2e8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc25810d4b0 con 0x7fc258101820 2024-03-18T22:58:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25c883640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258102a10 0x7fc258197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.833+0000 7fc25d084640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc258101820 0x7fc2581979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc25d084640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc258101820 0x7fc2581979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38142/0 (socket says 172.21.15.40:38142) 2024-03-18T22:58:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc25d084640 1 -- 172.21.15.40:0/500518306 learned_addr learned my addr 172.21.15.40:0/500518306 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc25c883640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258102a10 0x7fc258197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33728/0 (socket says 172.21.15.40:33728) 2024-03-18T22:58:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc25d885640 1 --2- 172.21.15.40:0/500518306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2581033d0 0x7fc25819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc25d084640 1 -- 172.21.15.40:0/500518306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2581033d0 msgr2=0x7fc25819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc25d084640 1 --2- 172.21.15.40:0/500518306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2581033d0 0x7fc25819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc25d084640 1 -- 172.21.15.40:0/500518306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258102a10 msgr2=0x7fc258197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc25d084640 1 --2- 172.21.15.40:0/500518306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258102a10 0x7fc258197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc25d084640 1 -- 172.21.15.40:0/500518306 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc25819f600 con 0x7fc258101820 2024-03-18T22:58:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc25c883640 1 --2- 172.21.15.40:0/500518306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258102a10 0x7fc258197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc25d084640 1 --2- 172.21.15.40:0/500518306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc258101820 0x7fc2581979e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc25000e9c0 tx=0x7fc25000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc2567fc640 1 -- 172.21.15.40:0/500518306 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25000cd80 con 0x7fc258101820 2024-03-18T22:58:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc25f2e8640 1 -- 172.21.15.40:0/500518306 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc25819f8f0 con 0x7fc258101820 2024-03-18T22:58:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc2567fc640 1 -- 172.21.15.40:0/500518306 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc2500035f0 con 0x7fc258101820 2024-03-18T22:58:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc2567fc640 1 -- 172.21.15.40:0/500518306 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc250010690 con 0x7fc258101820 2024-03-18T22:58:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.834+0000 7fc25f2e8640 1 -- 172.21.15.40:0/500518306 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc25819fe40 con 0x7fc258101820 2024-03-18T22:58:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.837+0000 7fc2567fc640 1 -- 172.21.15.40:0/500518306 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc2500037a0 con 0x7fc258101820 2024-03-18T22:58:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.837+0000 7fc261fac740 1 Processor -- start 2024-03-18T22:58:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.837+0000 7fc261fac740 1 -- start start 2024-03-18T22:58:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.837+0000 7fc261fac740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce91042f0 0x55dce9104710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.837+0000 7fc261fac740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce9104ce0 0x55dce91097e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.837+0000 7fc261fac740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9109e50 0x55dce910c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.837+0000 7fc261fac740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dce90684b0 con 0x55dce9109e50 2024-03-18T22:58:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.837+0000 7fc261fac740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dce9010d70 con 0x55dce9104ce0 2024-03-18T22:58:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.837+0000 7fc261fac740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55dce9064550 con 0x55dce91042f0 2024-03-18T22:58:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc25d885640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce91042f0 0x55dce9104710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc25c883640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9109e50 0x55dce910c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc25d084640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce9104ce0 0x55dce91097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc25d885640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce91042f0 0x55dce9104710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38146/0 (socket says 172.21.15.40:38146) 2024-03-18T22:58:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc25d885640 1 -- 172.21.15.40:0/3606898849 learned_addr learned my addr 172.21.15.40:0/3606898849 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc2567fc640 1 --2- 172.21.15.40:0/500518306 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2400a2800 0x7fc2400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc25c883640 1 -- 172.21.15.40:0/3606898849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce91042f0 msgr2=0x55dce9104710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc25c883640 1 --2- 172.21.15.40:0/3606898849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce91042f0 0x55dce9104710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc25c883640 1 -- 172.21.15.40:0/3606898849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce9104ce0 msgr2=0x55dce91097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc25c883640 1 --2- 172.21.15.40:0/3606898849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce9104ce0 0x55dce91097e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc25c883640 1 -- 172.21.15.40:0/3606898849 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dce9033210 con 0x55dce9109e50 2024-03-18T22:58:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc25d885640 1 --2- 172.21.15.40:0/3606898849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce91042f0 0x55dce9104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc25d084640 1 --2- 172.21.15.40:0/3606898849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce9104ce0 0x55dce91097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc25d885640 1 --2- 172.21.15.40:0/500518306 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2400a2800 0x7fc2400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc2567fc640 1 -- 172.21.15.40:0/500518306 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc2500cf240 con 0x7fc258101820 2024-03-18T22:58:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc25c883640 1 --2- 172.21.15.40:0/3606898849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9109e50 0x55dce910c2e0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009530 tx=0x7fc24c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a423d863ca1a2cd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc23f7fe640 1 -- 172.21.15.40:0/3606898849 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c004020 con 0x55dce9109e50 2024-03-18T22:58:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc23f7fe640 1 -- 172.21.15.40:0/3606898849 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc24c0041c0 con 0x55dce9109e50 2024-03-18T22:58:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.838+0000 7fc23f7fe640 1 -- 172.21.15.40:0/3606898849 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c00f850 con 0x55dce9109e50 2024-03-18T22:58:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.839+0000 7fc261fac740 1 -- 172.21.15.40:0/3606898849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9109e50 msgr2=0x55dce910c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.839+0000 7fc25d885640 1 --2- 172.21.15.40:0/500518306 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2400a2800 0x7fc2400a4cc0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc2480099a0 tx=0x7fc248009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.839+0000 7fc261fac740 1 --2- 172.21.15.40:0/3606898849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9109e50 0x55dce910c2e0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009530 tx=0x7fc24c0118d0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.839+0000 7fc261fac740 1 -- 172.21.15.40:0/3606898849 shutdown_connections 2024-03-18T22:58:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.839+0000 7fc261fac740 1 --2- 172.21.15.40:0/3606898849 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9109e50 0x55dce910c2e0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.839+0000 7fc261fac740 1 --2- 172.21.15.40:0/3606898849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce9104ce0 0x55dce91097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.839+0000 7fc261fac740 1 --2- 172.21.15.40:0/3606898849 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce91042f0 0x55dce9104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.839+0000 7fc261fac740 1 -- 172.21.15.40:0/3606898849 >> 172.21.15.40:0/3606898849 conn(0x55dce90ff860 msgr2=0x55dce90ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.839+0000 7fc261fac740 1 -- 172.21.15.40:0/3606898849 shutdown_connections 2024-03-18T22:58:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.839+0000 7fc261fac740 1 -- 172.21.15.40:0/3606898849 wait complete. 2024-03-18T22:58:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.839+0000 7fc261fac740 1 -- start start 2024-03-18T22:58:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.839+0000 7fc261fac740 1 Processor -- start 2024-03-18T22:58:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc261fac740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce91042f0 0x55dce9112c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc261fac740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9104ce0 0x55dce9113170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc261fac740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce9109e50 0x55dce911a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc261fac740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55dce910ce00 con 0x55dce9104ce0 2024-03-18T22:58:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc261fac740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dce9024310 con 0x55dce91042f0 2024-03-18T22:58:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc261fac740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55dce910cf80 con 0x55dce9109e50 2024-03-18T22:58:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc261fac740 10 client.0 _ll_register_callbacks cb 0x55dce90182d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc25c883640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9104ce0 0x55dce9113170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc25d084640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce91042f0 0x55dce9112c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc25d084640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce91042f0 0x55dce9112c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48702/0 (socket says 172.21.15.40:48702) 2024-03-18T22:58:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc25d084640 1 -- 172.21.15.40:0/1265854998 learned_addr learned my addr 172.21.15.40:0/1265854998 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc25d885640 1 --2- 172.21.15.40:0/1265854998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce9109e50 0x55dce911a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc25c883640 1 -- 172.21.15.40:0/1265854998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce9109e50 msgr2=0x55dce911a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc25c883640 1 --2- 172.21.15.40:0/1265854998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce9109e50 0x55dce911a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc25c883640 1 -- 172.21.15.40:0/1265854998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce91042f0 msgr2=0x55dce9112c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc25c883640 1 --2- 172.21.15.40:0/1265854998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce91042f0 0x55dce9112c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc25c883640 1 -- 172.21.15.40:0/1265854998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dce8d7c3f0 con 0x55dce9104ce0 2024-03-18T22:58:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc25d084640 1 --2- 172.21.15.40:0/1265854998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce91042f0 0x55dce9112c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc25c883640 1 --2- 172.21.15.40:0/1265854998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9104ce0 0x55dce9113170 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc24c000c00 tx=0x7fc24c003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=672fa485b4f7912d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.840+0000 7fc23cff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:58:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.841+0000 7fc23cff9640 1 -- 172.21.15.40:0/1265854998 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c01d050 con 0x55dce9104ce0 2024-03-18T22:58:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.841+0000 7fc261fac740 1 -- 172.21.15.40:0/1265854998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc24c0041c0 con 0x55dce9104ce0 2024-03-18T22:58:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.841+0000 7fc23cff9640 1 -- 172.21.15.40:0/1265854998 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc24c016040 con 0x55dce9104ce0 2024-03-18T22:58:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.841+0000 7fc23cff9640 1 -- 172.21.15.40:0/1265854998 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fc24c031050 con 0x55dce9104ce0 2024-03-18T22:58:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.841+0000 7fc261fac740 10 client.18807 fetch_fsmap learned FSMap version 49 2024-03-18T22:58:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.841+0000 7fc261fac740 1 -- 172.21.15.40:0/1265854998 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55dce8e467d0 con 0x55dce9104ce0 2024-03-18T22:58:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.841+0000 7fc23cff9640 1 -- 172.21.15.40:0/1265854998 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc24c0041c0 con 0x55dce9104ce0 2024-03-18T22:58:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.841+0000 7fc23cff9640 1 -- 172.21.15.40:0/1265854998 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8418+0+0 (secure 0 0 0) 0x7fc24c004a40 con 0x55dce9104ce0 2024-03-18T22:58:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.841+0000 7fc261fac740 10 client.18807 fetch_fsmap finished waiting for FSMap version 49 2024-03-18T22:58:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.841+0000 7fc261fac740 10 client.18807 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:58:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.841+0000 7fc261fac740 20 client.18807 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.841+0000 7fc261fac740 1 --2- 172.21.15.40:0/1265854998 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dce911e800 0x55dce9120ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.841+0000 7fc261fac740 4 client.18807 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:58:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.841+0000 7fc261fac740 1 -- 172.21.15.40:0/1265854998 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dce90c63e0 con 0x55dce911e800 2024-03-18T22:58:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.841+0000 7fc25d084640 1 --2- 172.21.15.40:0/1265854998 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dce911e800 0x55dce9120ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.842+0000 7fc23cff9640 10 client.18807 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:58:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.842+0000 7fc25d084640 1 --2- 172.21.15.40:0/1265854998 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dce911e800 0x55dce9120ca0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=371554af56810508 server_cookie=881112b111825147 in_seq=0 out_seq=0 2024-03-18T22:58:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.843+0000 7fc23cff9640 1 -- 172.21.15.40:0/1265854998 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dce90c63e0 con 0x55dce911e800 2024-03-18T22:58:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.843+0000 7fc23cff9640 10 client.18807 handle_command_reply: tid=0 2024-03-18T22:58:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.843+0000 7fc261fac740 1 client.18807 shutdown 2024-03-18T22:58:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.843+0000 7fc261fac740 1 -- 172.21.15.40:0/1265854998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9104ce0 msgr2=0x55dce9113170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.843+0000 7fc261fac740 1 --2- 172.21.15.40:0/1265854998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9104ce0 0x55dce9113170 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc24c000c00 tx=0x7fc24c003cd0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.843+0000 7fc261fac740 1 -- 172.21.15.40:0/1265854998 shutdown_connections 2024-03-18T22:58:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.843+0000 7fc261fac740 1 --2- 172.21.15.40:0/1265854998 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dce911e800 0x55dce9120ca0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.843+0000 7fc261fac740 1 --2- 172.21.15.40:0/1265854998 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce9109e50 0x55dce911a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.843+0000 7fc23cff9640 0 client.18807 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:58:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.843+0000 7fc261fac740 1 --2- 172.21.15.40:0/1265854998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9104ce0 0x55dce9113170 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.843+0000 7fc261fac740 1 --2- 172.21.15.40:0/1265854998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce91042f0 0x55dce9112c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.843+0000 7fc261fac740 1 -- 172.21.15.40:0/1265854998 >> 172.21.15.40:0/1265854998 conn(0x55dce90ff860 msgr2=0x55dce90feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.843+0000 7fc261fac740 1 -- 172.21.15.40:0/1265854998 shutdown_connections 2024-03-18T22:58:31.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.843+0000 7fc261fac740 1 -- 172.21.15.40:0/1265854998 wait complete. 2024-03-18T22:58:31.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.843+0000 7fc261fac740 20 client.18807 trim_cache size 0 max 16384 2024-03-18T22:58:31.861 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:31 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1174168962' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:58:31.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.856+0000 7fc261fac740 1 Processor -- start 2024-03-18T22:58:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.856+0000 7fc261fac740 1 -- start start 2024-03-18T22:58:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.856+0000 7fc261fac740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce90ff860 0x55dce91119a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.856+0000 7fc261fac740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce91042f0 0x55dce911a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.856+0000 7fc261fac740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce9104ce0 0x55dce90fe310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.856+0000 7fc261fac740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc2581a83e0 con 0x55dce90ff860 2024-03-18T22:58:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.856+0000 7fc261fac740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc2581aa100 con 0x55dce9104ce0 2024-03-18T22:58:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.856+0000 7fc261fac740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc2581a8ab0 con 0x55dce91042f0 2024-03-18T22:58:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.857+0000 7fc25d084640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce90ff860 0x55dce91119a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.857+0000 7fc25d084640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce90ff860 0x55dce91119a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33754/0 (socket says 172.21.15.40:33754) 2024-03-18T22:58:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.857+0000 7fc25d084640 1 -- 172.21.15.40:0/1847647303 learned_addr learned my addr 172.21.15.40:0/1847647303 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.857+0000 7fc25d885640 1 --2- 172.21.15.40:0/1847647303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce9104ce0 0x55dce90fe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.857+0000 7fc25c883640 1 --2- 172.21.15.40:0/1847647303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce91042f0 0x55dce911a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.857+0000 7fc25d084640 1 -- 172.21.15.40:0/1847647303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce91042f0 msgr2=0x55dce911a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.857+0000 7fc25d084640 1 --2- 172.21.15.40:0/1847647303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce91042f0 0x55dce911a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.857+0000 7fc25d084640 1 -- 172.21.15.40:0/1847647303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce9104ce0 msgr2=0x55dce90fe310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.857+0000 7fc25d084640 1 --2- 172.21.15.40:0/1847647303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce9104ce0 0x55dce90fe310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.857+0000 7fc25d084640 1 -- 172.21.15.40:0/1847647303 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dce90d5880 con 0x55dce90ff860 2024-03-18T22:58:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.857+0000 7fc25d084640 1 --2- 172.21.15.40:0/1847647303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce90ff860 0x55dce91119a0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fc25009b490 tx=0x7fc2500cc7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8fb148a941a2e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.857+0000 7fc23effd640 1 -- 172.21.15.40:0/1847647303 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25009b0f0 con 0x55dce90ff860 2024-03-18T22:58:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.857+0000 7fc23effd640 1 -- 172.21.15.40:0/1847647303 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc250003b30 con 0x55dce90ff860 2024-03-18T22:58:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.857+0000 7fc23effd640 1 -- 172.21.15.40:0/1847647303 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2500cca00 con 0x55dce90ff860 2024-03-18T22:58:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 -- 172.21.15.40:0/1847647303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce90ff860 msgr2=0x55dce91119a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 --2- 172.21.15.40:0/1847647303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce90ff860 0x55dce91119a0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fc25009b490 tx=0x7fc2500cc7c0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 -- 172.21.15.40:0/1847647303 shutdown_connections 2024-03-18T22:58:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 --2- 172.21.15.40:0/1847647303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce9104ce0 0x55dce90fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 --2- 172.21.15.40:0/1847647303 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce91042f0 0x55dce911a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 --2- 172.21.15.40:0/1847647303 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce90ff860 0x55dce91119a0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 -- 172.21.15.40:0/1847647303 >> 172.21.15.40:0/1847647303 conn(0x55dce90fae90 msgr2=0x55dce9111290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 -- 172.21.15.40:0/1847647303 shutdown_connections 2024-03-18T22:58:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 -- 172.21.15.40:0/1847647303 wait complete. 2024-03-18T22:58:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 -- start start 2024-03-18T22:58:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 Processor -- start 2024-03-18T22:58:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce90ff860 0x55dce9117510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce91042f0 0x55dce9117a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9104ce0 0x55dce9118a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fc2581a9970 con 0x55dce9104ce0 2024-03-18T22:58:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc2581a9240 con 0x55dce91042f0 2024-03-18T22:58:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fc2581a6360 con 0x55dce90ff860 2024-03-18T22:58:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.858+0000 7fc261fac740 10 client.0 _ll_register_callbacks cb 0x7fc250097a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc25d885640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9104ce0 0x55dce9118a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc25d084640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce90ff860 0x55dce9117510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc25d885640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9104ce0 0x55dce9118a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33760/0 (socket says 172.21.15.40:33760) 2024-03-18T22:58:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc25d885640 1 -- 172.21.15.40:0/327420934 learned_addr learned my addr 172.21.15.40:0/327420934 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc25c883640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce91042f0 0x55dce9117a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc25d885640 1 -- 172.21.15.40:0/327420934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce90ff860 msgr2=0x55dce9117510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc25d885640 1 --2- 172.21.15.40:0/327420934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce90ff860 0x55dce9117510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc25d885640 1 -- 172.21.15.40:0/327420934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce91042f0 msgr2=0x55dce9117a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc25d885640 1 --2- 172.21.15.40:0/327420934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce91042f0 0x55dce9117a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc25d885640 1 -- 172.21.15.40:0/327420934 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dce9119000 con 0x55dce9104ce0 2024-03-18T22:58:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc25d084640 1 --2- 172.21.15.40:0/327420934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce90ff860 0x55dce9117510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc25c883640 1 --2- 172.21.15.40:0/327420934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce91042f0 0x55dce9117a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc25d885640 1 --2- 172.21.15.40:0/327420934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9104ce0 0x55dce9118a10 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fc248034ec0 tx=0x7fc248005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13cfa9857dc6a710 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc23cff9640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:58:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc23cff9640 1 -- 172.21.15.40:0/327420934 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc248004250 con 0x55dce9104ce0 2024-03-18T22:58:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc23cff9640 1 -- 172.21.15.40:0/327420934 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc24803a040 con 0x55dce9104ce0 2024-03-18T22:58:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc261fac740 1 -- 172.21.15.40:0/327420934 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dce90759f0 con 0x55dce9104ce0 2024-03-18T22:58:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc23cff9640 1 -- 172.21.15.40:0/327420934 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fc2480043f0 con 0x55dce9104ce0 2024-03-18T22:58:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc261fac740 10 client.18813 fetch_fsmap learned FSMap version 49 2024-03-18T22:58:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc261fac740 1 -- 172.21.15.40:0/327420934 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55dce911a370 con 0x55dce9104ce0 2024-03-18T22:58:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.859+0000 7fc23cff9640 1 -- 172.21.15.40:0/327420934 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55dce90759f0 con 0x55dce9104ce0 2024-03-18T22:58:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.860+0000 7fc23cff9640 1 -- 172.21.15.40:0/327420934 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8418+0+0 (secure 0 0 0) 0x7fc24804d430 con 0x55dce9104ce0 2024-03-18T22:58:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.860+0000 7fc261fac740 10 client.18813 fetch_fsmap finished waiting for FSMap version 49 2024-03-18T22:58:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.860+0000 7fc261fac740 10 client.18813 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-18T22:58:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.860+0000 7fc261fac740 20 client.18813 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.860+0000 7fc261fac740 1 --2- 172.21.15.40:0/327420934 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dce9109e50 0x55dce91873c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.860+0000 7fc261fac740 4 client.18813 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:58:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.860+0000 7fc261fac740 1 -- 172.21.15.40:0/327420934 --> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dce906fd40 con 0x55dce9109e50 2024-03-18T22:58:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.860+0000 7fc25d084640 1 --2- 172.21.15.40:0/327420934 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dce9109e50 0x55dce91873c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.860+0000 7fc23cff9640 10 client.18813 ms_handle_connect on v2:172.21.15.103:6836/1713853423 2024-03-18T22:58:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.860+0000 7fc25d084640 1 --2- 172.21.15.40:0/327420934 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dce9109e50 0x55dce91873c0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2d08f2e518e216b0 server_cookie=c4e1af28e5e1b5a5 in_seq=0 out_seq=0 2024-03-18T22:58:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.860+0000 7fc23cff9640 1 -- 172.21.15.40:0/327420934 <== mds.2 v2:172.21.15.103:6836/1713853423 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dce906fd40 con 0x55dce9109e50 2024-03-18T22:58:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.860+0000 7fc23cff9640 10 client.18813 handle_command_reply: tid=0 2024-03-18T22:58:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.860+0000 7fc261fac740 1 client.18813 shutdown 2024-03-18T22:58:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.860+0000 7fc261fac740 1 -- 172.21.15.40:0/327420934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9104ce0 msgr2=0x55dce9118a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.860+0000 7fc261fac740 1 --2- 172.21.15.40:0/327420934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9104ce0 0x55dce9118a10 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fc248034ec0 tx=0x7fc248005fd0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc261fac740 1 -- 172.21.15.40:0/327420934 shutdown_connections 2024-03-18T22:58:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc261fac740 1 --2- 172.21.15.40:0/327420934 >> [v2:172.21.15.103:6836/1713853423,v1:172.21.15.103:6837/1713853423] conn(0x55dce9109e50 0x55dce91873c0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc261fac740 1 --2- 172.21.15.40:0/327420934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55dce9104ce0 0x55dce9118a10 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc23cff9640 0 client.18813 ms_handle_reset on v2:172.21.15.103:6836/1713853423 2024-03-18T22:58:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc261fac740 1 --2- 172.21.15.40:0/327420934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dce91042f0 0x55dce9117a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc261fac740 1 --2- 172.21.15.40:0/327420934 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55dce90ff860 0x55dce9117510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc261fac740 1 -- 172.21.15.40:0/327420934 >> 172.21.15.40:0/327420934 conn(0x55dce90fae90 msgr2=0x55dce9111b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc261fac740 1 -- 172.21.15.40:0/327420934 shutdown_connections 2024-03-18T22:58:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc261fac740 1 -- 172.21.15.40:0/327420934 wait complete. 2024-03-18T22:58:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc261fac740 20 client.18813 trim_cache size 0 max 16384 2024-03-18T22:58:31.869 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:58:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc25f2e8640 1 -- 172.21.15.40:0/500518306 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2400a2800 msgr2=0x7fc2400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc25f2e8640 1 --2- 172.21.15.40:0/500518306 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2400a2800 0x7fc2400a4cc0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc2480099a0 tx=0x7fc248009500 comp rx=0 tx=0).stop 2024-03-18T22:58:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc25f2e8640 1 -- 172.21.15.40:0/500518306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc258101820 msgr2=0x7fc2581979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc25f2e8640 1 --2- 172.21.15.40:0/500518306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc258101820 0x7fc2581979e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc25000e9c0 tx=0x7fc25000ee80 comp rx=0 tx=0).stop 2024-03-18T22:58:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc25f2e8640 1 -- 172.21.15.40:0/500518306 shutdown_connections 2024-03-18T22:58:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc25f2e8640 1 --2- 172.21.15.40:0/500518306 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fc2400a2800 0x7fc2400a4cc0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc25f2e8640 1 --2- 172.21.15.40:0/500518306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2581033d0 0x7fc25819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc25f2e8640 1 --2- 172.21.15.40:0/500518306 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258102a10 0x7fc258197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc25f2e8640 1 --2- 172.21.15.40:0/500518306 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fc258101820 0x7fc2581979e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc25f2e8640 1 -- 172.21.15.40:0/500518306 >> 172.21.15.40:0/500518306 conn(0x7fc2580fcd90 msgr2=0x7fc258105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc25f2e8640 1 -- 172.21.15.40:0/500518306 shutdown_connections 2024-03-18T22:58:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.861+0000 7fc25f2e8640 1 -- 172.21.15.40:0/500518306 wait complete. 2024-03-18T22:58:31.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T22:58:31.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.976+0000 7f893591b640 1 -- 172.21.15.40:0/845804206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8930102300 msgr2=0x7f8930108b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.976+0000 7f893591b640 1 --2- 172.21.15.40:0/845804206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8930102300 0x7f8930108b90 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8920009be0 tx=0x7f892002bef0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.976+0000 7f893591b640 1 -- 172.21.15.40:0/845804206 shutdown_connections 2024-03-18T22:58:31.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.976+0000 7f893591b640 1 --2- 172.21.15.40:0/845804206 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8930102300 0x7f8930108b90 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.976+0000 7f893591b640 1 --2- 172.21.15.40:0/845804206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8930101940 0x7f8930101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.976+0000 7f893591b640 1 --2- 172.21.15.40:0/845804206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8930100750 0x7f8930100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.976+0000 7f893591b640 1 -- 172.21.15.40:0/845804206 >> 172.21.15.40:0/845804206 conn(0x7f89300fbce0 msgr2=0x7f89300fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.976+0000 7f893591b640 1 -- 172.21.15.40:0/845804206 shutdown_connections 2024-03-18T22:58:31.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.976+0000 7f893591b640 1 -- 172.21.15.40:0/845804206 wait complete. 2024-03-18T22:58:31.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.977+0000 7f893591b640 1 Processor -- start 2024-03-18T22:58:31.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.977+0000 7f893591b640 1 -- start start 2024-03-18T22:58:31.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.977+0000 7f893591b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8930100750 0x7f893010b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.977+0000 7f893591b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8930101940 0x7f8930109710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.977+0000 7f893591b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8930102300 0x7f8930109c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.977+0000 7f893591b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f893010d3a0 con 0x7f8930100750 2024-03-18T22:58:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.977+0000 7f893591b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f893010d220 con 0x7f8930101940 2024-03-18T22:58:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.977+0000 7f893591b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f893010d520 con 0x7f8930102300 2024-03-18T22:58:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.977+0000 7f892effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8930100750 0x7f893010b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.977+0000 7f892effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8930100750 0x7f893010b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33784/0 (socket says 172.21.15.40:33784) 2024-03-18T22:58:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.977+0000 7f892effd640 1 -- 172.21.15.40:0/690906241 learned_addr learned my addr 172.21.15.40:0/690906241 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.977+0000 7f892f7fe640 1 --2- 172.21.15.40:0/690906241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8930102300 0x7f8930109c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.977+0000 7f892e7fc640 1 --2- 172.21.15.40:0/690906241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8930101940 0x7f8930109710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.978+0000 7f892effd640 1 -- 172.21.15.40:0/690906241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8930102300 msgr2=0x7f8930109c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.978+0000 7f892effd640 1 --2- 172.21.15.40:0/690906241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8930102300 0x7f8930109c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.978+0000 7f892effd640 1 -- 172.21.15.40:0/690906241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8930101940 msgr2=0x7f8930109710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.978+0000 7f892effd640 1 --2- 172.21.15.40:0/690906241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8930101940 0x7f8930109710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.978+0000 7f892effd640 1 -- 172.21.15.40:0/690906241 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f893010a4e0 con 0x7f8930100750 2024-03-18T22:58:31.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.978+0000 7f892effd640 1 --2- 172.21.15.40:0/690906241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8930100750 0x7f893010b0d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f892800e9c0 tx=0x7f892800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.978+0000 7f8934919640 1 -- 172.21.15.40:0/690906241 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892800cd80 con 0x7f8930100750 2024-03-18T22:58:31.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.978+0000 7f8934919640 1 -- 172.21.15.40:0/690906241 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f89280035f0 con 0x7f8930100750 2024-03-18T22:58:31.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.978+0000 7f893591b640 1 -- 172.21.15.40:0/690906241 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89301a1ad0 con 0x7f8930100750 2024-03-18T22:58:31.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.978+0000 7f8934919640 1 -- 172.21.15.40:0/690906241 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8928010690 con 0x7f8930100750 2024-03-18T22:58:31.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.978+0000 7f893591b640 1 -- 172.21.15.40:0/690906241 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89301a2020 con 0x7f8930100750 2024-03-18T22:58:31.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.980+0000 7f8934919640 1 -- 172.21.15.40:0/690906241 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f89280037a0 con 0x7f8930100750 2024-03-18T22:58:31.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.980+0000 7f89385e7740 1 Processor -- start 2024-03-18T22:58:31.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f89385e7740 1 -- start start 2024-03-18T22:58:31.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f89385e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b82f0 0x55a6a60b8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f89385e7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 0x55a6a60bd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f89385e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f89385e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a6a601c4b0 con 0x55a6a60bde50 2024-03-18T22:58:31.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f89385e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a6a5fc4d70 con 0x55a6a60b82f0 2024-03-18T22:58:31.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f89385e7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55a6a6018550 con 0x55a6a60b8ce0 2024-03-18T22:58:31.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f8934919640 1 --2- 172.21.15.40:0/690906241 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f891c0a2800 0x7f891c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f892e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f892effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 0x55a6a60bd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f892e7fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33794/0 (socket says 172.21.15.40:33794) 2024-03-18T22:58:31.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f892e7fc640 1 -- 172.21.15.40:0/1183490593 learned_addr learned my addr 172.21.15.40:0/1183490593 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f892f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b82f0 0x55a6a60b8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f8934919640 1 -- 172.21.15.40:0/690906241 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f89280cf2a0 con 0x7f8930100750 2024-03-18T22:58:31.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f892f7fe640 1 --2- 172.21.15.40:0/690906241 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f891c0a2800 0x7f891c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f892effd640 1 -- 172.21.15.40:0/1183490593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b82f0 msgr2=0x55a6a60b8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f892effd640 1 --2- 172.21.15.40:0/1183490593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b82f0 0x55a6a60b8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f892effd640 1 -- 172.21.15.40:0/1183490593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 msgr2=0x55a6a60c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f892effd640 1 --2- 172.21.15.40:0/1183490593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f892effd640 1 -- 172.21.15.40:0/1183490593 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6a5fe7210 con 0x55a6a60b8ce0 2024-03-18T22:58:31.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f892e7fc640 1 --2- 172.21.15.40:0/1183490593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f892f7fe640 1 --2- 172.21.15.40:0/690906241 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f891c0a2800 0x7f891c0a4cc0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8920004af0 tx=0x7f8920005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.981+0000 7f892effd640 1 --2- 172.21.15.40:0/1183490593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 0x55a6a60bd7e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f892809b780 tx=0x7f892806e520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d58941ee766d85f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.982+0000 7f8925ffb640 1 -- 172.21.15.40:0/1183490593 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892809bef0 con 0x55a6a60b8ce0 2024-03-18T22:58:31.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.982+0000 7f8925ffb640 1 -- 172.21.15.40:0/1183490593 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f892809b7f0 con 0x55a6a60b8ce0 2024-03-18T22:58:31.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.982+0000 7f8925ffb640 1 -- 172.21.15.40:0/1183490593 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8928086200 con 0x55a6a60b8ce0 2024-03-18T22:58:31.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.982+0000 7f89385e7740 1 -- 172.21.15.40:0/1183490593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 msgr2=0x55a6a60bd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.982+0000 7f89385e7740 1 --2- 172.21.15.40:0/1183490593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 0x55a6a60bd7e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f892809b780 tx=0x7f892806e520 comp rx=0 tx=0).stop 2024-03-18T22:58:31.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.982+0000 7f89385e7740 1 -- 172.21.15.40:0/1183490593 shutdown_connections 2024-03-18T22:58:31.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.982+0000 7f89385e7740 1 --2- 172.21.15.40:0/1183490593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.982+0000 7f89385e7740 1 --2- 172.21.15.40:0/1183490593 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 0x55a6a60bd7e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.982+0000 7f89385e7740 1 --2- 172.21.15.40:0/1183490593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b82f0 0x55a6a60b8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.982+0000 7f89385e7740 1 -- 172.21.15.40:0/1183490593 >> 172.21.15.40:0/1183490593 conn(0x55a6a60b3860 msgr2=0x55a6a60b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.982+0000 7f89385e7740 1 -- 172.21.15.40:0/1183490593 shutdown_connections 2024-03-18T22:58:31.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.982+0000 7f89385e7740 1 -- 172.21.15.40:0/1183490593 wait complete. 2024-03-18T22:58:31.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.983+0000 7f89385e7740 1 -- start start 2024-03-18T22:58:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.983+0000 7f89385e7740 1 Processor -- start 2024-03-18T22:58:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.983+0000 7f89385e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b82f0 0x55a6a60c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.983+0000 7f89385e7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 0x55a6a60c2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.983+0000 7f89385e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.983+0000 7f89385e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x55a6a60c0da0 con 0x55a6a60bde50 2024-03-18T22:58:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.983+0000 7f89385e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a6a5fd8310 con 0x55a6a60b82f0 2024-03-18T22:58:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.983+0000 7f89385e7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x55a6a60c0f20 con 0x55a6a60b8ce0 2024-03-18T22:58:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.983+0000 7f89385e7740 10 client.0 _ll_register_callbacks cb 0x55a6a5fcc2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.983+0000 7f892f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.983+0000 7f892f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33808/0 (socket says 172.21.15.40:33808) 2024-03-18T22:58:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.983+0000 7f892f7fe640 1 -- 172.21.15.40:0/2802302966 learned_addr learned my addr 172.21.15.40:0/2802302966 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.983+0000 7f892effd640 1 --2- 172.21.15.40:0/2802302966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b82f0 0x55a6a60c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.983+0000 7f892e7fc640 1 --2- 172.21.15.40:0/2802302966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 0x55a6a60c2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f892f7fe640 1 -- 172.21.15.40:0/2802302966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 msgr2=0x55a6a60c2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f892f7fe640 1 --2- 172.21.15.40:0/2802302966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 0x55a6a60c2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f892f7fe640 1 -- 172.21.15.40:0/2802302966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b82f0 msgr2=0x55a6a60c27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:31.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f892f7fe640 1 --2- 172.21.15.40:0/2802302966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b82f0 0x55a6a60c27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f892f7fe640 1 -- 172.21.15.40:0/2802302966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6a5d303f0 con 0x55a6a60bde50 2024-03-18T22:58:31.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f892effd640 1 --2- 172.21.15.40:0/2802302966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b82f0 0x55a6a60c27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:31.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f892f7fe640 1 --2- 172.21.15.40:0/2802302966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c9d30 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f8920005f00 tx=0x7f892002bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfabb700c64b3b16 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:31.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f890effd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:58:31.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f890effd640 1 -- 172.21.15.40:0/2802302966 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8920053020 con 0x55a6a60bde50 2024-03-18T22:58:31.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f890effd640 1 -- 172.21.15.40:0/2802302966 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f892002dbf0 con 0x55a6a60bde50 2024-03-18T22:58:31.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f89385e7740 1 -- 172.21.15.40:0/2802302966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f892809b7f0 con 0x55a6a60bde50 2024-03-18T22:58:31.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f890effd640 1 -- 172.21.15.40:0/2802302966 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f892005e050 con 0x55a6a60bde50 2024-03-18T22:58:31.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f89385e7740 10 client.18825 fetch_fsmap learned FSMap version 49 2024-03-18T22:58:31.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f89385e7740 1 -- 172.21.15.40:0/2802302966 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55a6a5dfa7d0 con 0x55a6a60bde50 2024-03-18T22:58:31.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f890effd640 1 -- 172.21.15.40:0/2802302966 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f892809b7f0 con 0x55a6a60bde50 2024-03-18T22:58:31.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f890effd640 1 -- 172.21.15.40:0/2802302966 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8418+0+0 (secure 0 0 0) 0x7f892004e030 con 0x55a6a60bde50 2024-03-18T22:58:31.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f89385e7740 10 client.18825 fetch_fsmap finished waiting for FSMap version 49 2024-03-18T22:58:31.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f89385e7740 10 client.18825 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:58:31.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f89385e7740 20 client.18825 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:31.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.984+0000 7f89385e7740 1 --2- 172.21.15.40:0/2802302966 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55a6a60ce6d0 0x55a6a60d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:31.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.985+0000 7f89385e7740 4 client.18825 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:58:31.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.985+0000 7f89385e7740 1 -- 172.21.15.40:0/2802302966 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6a607a3e0 con 0x55a6a60ce6d0 2024-03-18T22:58:31.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.985+0000 7f892effd640 1 --2- 172.21.15.40:0/2802302966 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55a6a60ce6d0 0x55a6a60d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:31.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.985+0000 7f892effd640 1 --2- 172.21.15.40:0/2802302966 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55a6a60ce6d0 0x55a6a60d0b30 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=51db571166f67a1f server_cookie=97d45dfe8f7ece59 in_seq=0 out_seq=0 2024-03-18T22:58:31.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.985+0000 7f890effd640 10 client.18825 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:58:31.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.986+0000 7f890effd640 1 -- 172.21.15.40:0/2802302966 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a6a607a3e0 con 0x55a6a60ce6d0 2024-03-18T22:58:31.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.986+0000 7f890effd640 10 client.18825 handle_command_reply: tid=0 2024-03-18T22:58:31.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.986+0000 7f89385e7740 1 client.18825 shutdown 2024-03-18T22:58:31.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.986+0000 7f89385e7740 1 -- 172.21.15.40:0/2802302966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 msgr2=0x55a6a60c9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:31.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.986+0000 7f89385e7740 1 --2- 172.21.15.40:0/2802302966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c9d30 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f8920005f00 tx=0x7f892002bef0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.986+0000 7f89385e7740 1 -- 172.21.15.40:0/2802302966 shutdown_connections 2024-03-18T22:58:31.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.986+0000 7f89385e7740 1 --2- 172.21.15.40:0/2802302966 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55a6a60ce6d0 0x55a6a60d0b30 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.986+0000 7f89385e7740 1 --2- 172.21.15.40:0/2802302966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c9d30 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.986+0000 7f89385e7740 1 --2- 172.21.15.40:0/2802302966 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 0x55a6a60c2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.986+0000 7f890effd640 0 client.18825 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:58:31.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.986+0000 7f89385e7740 1 --2- 172.21.15.40:0/2802302966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b82f0 0x55a6a60c27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:31.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.986+0000 7f89385e7740 1 -- 172.21.15.40:0/2802302966 >> 172.21.15.40:0/2802302966 conn(0x55a6a60b3860 msgr2=0x55a6a60b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:31.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.986+0000 7f89385e7740 1 -- 172.21.15.40:0/2802302966 shutdown_connections 2024-03-18T22:58:31.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.986+0000 7f89385e7740 1 -- 172.21.15.40:0/2802302966 wait complete. 2024-03-18T22:58:31.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.986+0000 7f89385e7740 20 client.18825 trim_cache size 0 max 16384 2024-03-18T22:58:32.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.999+0000 7f89385e7740 1 Processor -- start 2024-03-18T22:58:32.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.999+0000 7f89385e7740 1 -- start start 2024-03-18T22:58:32.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.999+0000 7f89385e7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b82f0 0x55a6a60b9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.999+0000 7f89385e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b8ce0 0x55a6a60c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.999+0000 7f89385e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.999+0000 7f89385e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f89301ac760 con 0x55a6a60bde50 2024-03-18T22:58:32.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.999+0000 7f89385e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89301ae480 con 0x55a6a60b8ce0 2024-03-18T22:58:32.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.999+0000 7f89385e7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f89301ace30 con 0x55a6a60b82f0 2024-03-18T22:58:32.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:31.999+0000 7f892f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.000+0000 7f892effd640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b82f0 0x55a6a60b9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.000+0000 7f892f7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33820/0 (socket says 172.21.15.40:33820) 2024-03-18T22:58:32.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.000+0000 7f892f7fe640 1 -- 172.21.15.40:0/2054012333 learned_addr learned my addr 172.21.15.40:0/2054012333 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.000+0000 7f892e7fc640 1 --2- 172.21.15.40:0/2054012333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b8ce0 0x55a6a60c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.000+0000 7f892f7fe640 1 -- 172.21.15.40:0/2054012333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b82f0 msgr2=0x55a6a60b9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.000+0000 7f892f7fe640 1 --2- 172.21.15.40:0/2054012333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b82f0 0x55a6a60b9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.000+0000 7f892f7fe640 1 -- 172.21.15.40:0/2054012333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b8ce0 msgr2=0x55a6a60c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.000+0000 7f892f7fe640 1 --2- 172.21.15.40:0/2054012333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b8ce0 0x55a6a60c9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.000+0000 7f892f7fe640 1 -- 172.21.15.40:0/2054012333 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6a60b90c0 con 0x55a6a60bde50 2024-03-18T22:58:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.000+0000 7f892f7fe640 1 --2- 172.21.15.40:0/2054012333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c12c0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f8920033ff0 tx=0x7f8920003f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd661b133f6241ba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.000+0000 7f89257fa640 1 -- 172.21.15.40:0/2054012333 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8920053070 con 0x55a6a60bde50 2024-03-18T22:58:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.000+0000 7f89257fa640 1 -- 172.21.15.40:0/2054012333 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8920004840 con 0x55a6a60bde50 2024-03-18T22:58:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.000+0000 7f89257fa640 1 -- 172.21.15.40:0/2054012333 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892004b040 con 0x55a6a60bde50 2024-03-18T22:58:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 -- 172.21.15.40:0/2054012333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 msgr2=0x55a6a60c12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 --2- 172.21.15.40:0/2054012333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c12c0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f8920033ff0 tx=0x7f8920003f00 comp rx=0 tx=0).stop 2024-03-18T22:58:32.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 -- 172.21.15.40:0/2054012333 shutdown_connections 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 --2- 172.21.15.40:0/2054012333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60bde50 0x55a6a60c12c0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 --2- 172.21.15.40:0/2054012333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60b8ce0 0x55a6a60c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 --2- 172.21.15.40:0/2054012333 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b82f0 0x55a6a60b9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 -- 172.21.15.40:0/2054012333 >> 172.21.15.40:0/2054012333 conn(0x55a6a60b3860 msgr2=0x55a6a60b3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 -- 172.21.15.40:0/2054012333 shutdown_connections 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 -- 172.21.15.40:0/2054012333 wait complete. 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 -- start start 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 Processor -- start 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60b82f0 0x55a6a60c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 0x55a6a60c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60bde50 0x55a6a6132a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f89301adcf0 con 0x55a6a60b82f0 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89301ad5c0 con 0x55a6a60bde50 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f89301aa6e0 con 0x55a6a60b8ce0 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f89385e7740 10 client.0 _ll_register_callbacks cb 0x7f8920009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f892effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60b82f0 0x55a6a60c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f892effd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60b82f0 0x55a6a60c6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33836/0 (socket says 172.21.15.40:33836) 2024-03-18T22:58:32.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.001+0000 7f892effd640 1 -- 172.21.15.40:0/2341964869 learned_addr learned my addr 172.21.15.40:0/2341964869 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:32.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f892f7fe640 1 --2- 172.21.15.40:0/2341964869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60bde50 0x55a6a6132a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f892e7fc640 1 --2- 172.21.15.40:0/2341964869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 0x55a6a60c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f892effd640 1 -- 172.21.15.40:0/2341964869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 msgr2=0x55a6a60c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:32.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f892effd640 1 --2- 172.21.15.40:0/2341964869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 0x55a6a60c71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f892effd640 1 -- 172.21.15.40:0/2341964869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60bde50 msgr2=0x55a6a6132a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:32.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f892effd640 1 --2- 172.21.15.40:0/2341964869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60bde50 0x55a6a6132a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f892effd640 1 -- 172.21.15.40:0/2341964869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6a6133020 con 0x55a6a60b82f0 2024-03-18T22:58:32.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f892effd640 1 --2- 172.21.15.40:0/2341964869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60b82f0 0x55a6a60c6cb0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f892809b4f0 tx=0x7f892809b190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d81c1b9eda59447 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:32.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f890effd640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:58:32.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f890effd640 1 -- 172.21.15.40:0/2341964869 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892809f940 con 0x55a6a60b82f0 2024-03-18T22:58:32.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f890effd640 1 -- 172.21.15.40:0/2341964869 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f89280860c0 con 0x55a6a60b82f0 2024-03-18T22:58:32.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f89385e7740 1 -- 172.21.15.40:0/2341964869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a6a60299f0 con 0x55a6a60b82f0 2024-03-18T22:58:32.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f890effd640 1 -- 172.21.15.40:0/2341964869 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f89280a3f90 con 0x55a6a60b82f0 2024-03-18T22:58:32.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f89385e7740 10 client.18831 fetch_fsmap learned FSMap version 49 2024-03-18T22:58:32.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f89385e7740 1 -- 172.21.15.40:0/2341964869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55a6a6134390 con 0x55a6a60b82f0 2024-03-18T22:58:32.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.002+0000 7f890effd640 1 -- 172.21.15.40:0/2341964869 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a6a60299f0 con 0x55a6a60b82f0 2024-03-18T22:58:32.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f890effd640 1 -- 172.21.15.40:0/2341964869 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8418+0+0 (secure 0 0 0) 0x7f89280b6ce0 con 0x55a6a60b82f0 2024-03-18T22:58:32.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f89385e7740 10 client.18831 fetch_fsmap finished waiting for FSMap version 49 2024-03-18T22:58:32.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f89385e7740 10 client.18831 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T22:58:32.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f89385e7740 20 client.18831 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:32.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f89385e7740 1 --2- 172.21.15.40:0/2341964869 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55a6a6134a10 0x55a6a6136e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f89385e7740 4 client.18831 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:58:32.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f89385e7740 1 -- 172.21.15.40:0/2341964869 --> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a6a6023d40 con 0x55a6a6134a10 2024-03-18T22:58:32.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f892e7fc640 1 --2- 172.21.15.40:0/2341964869 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55a6a6134a10 0x55a6a6136e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f892e7fc640 1 --2- 172.21.15.40:0/2341964869 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55a6a6134a10 0x55a6a6136e00 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5b03cf51c23247a9 server_cookie=48542cd6fb484bf0 in_seq=0 out_seq=0 2024-03-18T22:58:32.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f890effd640 10 client.18831 ms_handle_connect on v2:172.21.15.113:6838/2986827418 2024-03-18T22:58:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f890effd640 1 -- 172.21.15.40:0/2341964869 <== mds.1 v2:172.21.15.113:6838/2986827418 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a6a6023d40 con 0x55a6a6134a10 2024-03-18T22:58:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f890effd640 10 client.18831 handle_command_reply: tid=0 2024-03-18T22:58:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f89385e7740 1 client.18831 shutdown 2024-03-18T22:58:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f89385e7740 1 -- 172.21.15.40:0/2341964869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60b82f0 msgr2=0x55a6a60c6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f89385e7740 1 --2- 172.21.15.40:0/2341964869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60b82f0 0x55a6a60c6cb0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f892809b4f0 tx=0x7f892809b190 comp rx=0 tx=0).stop 2024-03-18T22:58:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f89385e7740 1 -- 172.21.15.40:0/2341964869 shutdown_connections 2024-03-18T22:58:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.003+0000 7f89385e7740 1 --2- 172.21.15.40:0/2341964869 >> [v2:172.21.15.113:6838/2986827418,v1:172.21.15.113:6839/2986827418] conn(0x55a6a6134a10 0x55a6a6136e00 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f89385e7740 1 --2- 172.21.15.40:0/2341964869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6a60bde50 0x55a6a6132a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f890effd640 0 client.18831 ms_handle_reset on v2:172.21.15.113:6838/2986827418 2024-03-18T22:58:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f89385e7740 1 --2- 172.21.15.40:0/2341964869 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x55a6a60b8ce0 0x55a6a60c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f89385e7740 1 --2- 172.21.15.40:0/2341964869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a6a60b82f0 0x55a6a60c6cb0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f89385e7740 1 -- 172.21.15.40:0/2341964869 >> 172.21.15.40:0/2341964869 conn(0x55a6a60b3860 msgr2=0x55a6a60c12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f89385e7740 1 -- 172.21.15.40:0/2341964869 shutdown_connections 2024-03-18T22:58:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f89385e7740 1 -- 172.21.15.40:0/2341964869 wait complete. 2024-03-18T22:58:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f89385e7740 20 client.18831 trim_cache size 0 max 16384 2024-03-18T22:58:32.011 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:58:32.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f893591b640 1 -- 172.21.15.40:0/690906241 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f891c0a2800 msgr2=0x7f891c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:32.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f893591b640 1 --2- 172.21.15.40:0/690906241 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f891c0a2800 0x7f891c0a4cc0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8920004af0 tx=0x7f8920005fd0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f893591b640 1 -- 172.21.15.40:0/690906241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8930100750 msgr2=0x7f893010b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:32.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f893591b640 1 --2- 172.21.15.40:0/690906241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8930100750 0x7f893010b0d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f892800e9c0 tx=0x7f892800ee80 comp rx=0 tx=0).stop 2024-03-18T22:58:32.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f893591b640 1 -- 172.21.15.40:0/690906241 shutdown_connections 2024-03-18T22:58:32.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f893591b640 1 --2- 172.21.15.40:0/690906241 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f891c0a2800 0x7f891c0a4cc0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f893591b640 1 --2- 172.21.15.40:0/690906241 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8930102300 0x7f8930109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f893591b640 1 --2- 172.21.15.40:0/690906241 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8930101940 0x7f8930109710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f893591b640 1 --2- 172.21.15.40:0/690906241 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8930100750 0x7f893010b0d0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f893591b640 1 -- 172.21.15.40:0/690906241 >> 172.21.15.40:0/690906241 conn(0x7f89300fbce0 msgr2=0x7f8930104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:32.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f893591b640 1 -- 172.21.15.40:0/690906241 shutdown_connections 2024-03-18T22:58:32.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.004+0000 7f893591b640 1 -- 172.21.15.40:0/690906241 wait complete. 2024-03-18T22:58:32.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T22:58:32.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.120+0000 7fab93ef8640 1 -- 172.21.15.40:0/2393157016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab8c100720 msgr2=0x7fab8c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:32.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.120+0000 7fab93ef8640 1 --2- 172.21.15.40:0/2393157016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab8c100720 0x7fab8c100b40 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fab88009900 tx=0x7fab8802bc30 comp rx=0 tx=0).stop 2024-03-18T22:58:32.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.120+0000 7fab93ef8640 1 -- 172.21.15.40:0/2393157016 shutdown_connections 2024-03-18T22:58:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.120+0000 7fab93ef8640 1 --2- 172.21.15.40:0/2393157016 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab8c1022d0 0x7fab8c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.120+0000 7fab93ef8640 1 --2- 172.21.15.40:0/2393157016 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab8c101910 0x7fab8c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.120+0000 7fab93ef8640 1 --2- 172.21.15.40:0/2393157016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab8c100720 0x7fab8c100b40 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.120+0000 7fab93ef8640 1 -- 172.21.15.40:0/2393157016 >> 172.21.15.40:0/2393157016 conn(0x7fab8c0fbcb0 msgr2=0x7fab8c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.120+0000 7fab93ef8640 1 -- 172.21.15.40:0/2393157016 shutdown_connections 2024-03-18T22:58:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.120+0000 7fab93ef8640 1 -- 172.21.15.40:0/2393157016 wait complete. 2024-03-18T22:58:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.120+0000 7fab93ef8640 1 Processor -- start 2024-03-18T22:58:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab93ef8640 1 -- start start 2024-03-18T22:58:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab93ef8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab8c100720 0x7fab8c196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab93ef8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab8c101910 0x7fab8c196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab93ef8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab8c1022d0 0x7fab8c19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab93ef8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fab8c10c250 con 0x7fab8c101910 2024-03-18T22:58:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab93ef8640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fab8c10c0d0 con 0x7fab8c100720 2024-03-18T22:58:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab93ef8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fab8c10c3d0 con 0x7fab8c1022d0 2024-03-18T22:58:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab91493640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab8c101910 0x7fab8c196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab91c94640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab8c100720 0x7fab8c196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab91493640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab8c101910 0x7fab8c196e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33856/0 (socket says 172.21.15.40:33856) 2024-03-18T22:58:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab91493640 1 -- 172.21.15.40:0/3970154024 learned_addr learned my addr 172.21.15.40:0/3970154024 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab92495640 1 --2- 172.21.15.40:0/3970154024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab8c1022d0 0x7fab8c19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab91493640 1 -- 172.21.15.40:0/3970154024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab8c1022d0 msgr2=0x7fab8c19dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab91493640 1 --2- 172.21.15.40:0/3970154024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab8c1022d0 0x7fab8c19dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab91493640 1 -- 172.21.15.40:0/3970154024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab8c100720 msgr2=0x7fab8c196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab91493640 1 --2- 172.21.15.40:0/3970154024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab8c100720 0x7fab8c196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab91493640 1 -- 172.21.15.40:0/3970154024 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab8c19e550 con 0x7fab8c101910 2024-03-18T22:58:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab91493640 1 --2- 172.21.15.40:0/3970154024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab8c101910 0x7fab8c196e70 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fab7c00e9f0 tx=0x7fab7c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab82ffd640 1 -- 172.21.15.40:0/3970154024 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7c00cd80 con 0x7fab8c101910 2024-03-18T22:58:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab82ffd640 1 -- 172.21.15.40:0/3970154024 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab7c0108e0 con 0x7fab8c101910 2024-03-18T22:58:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab82ffd640 1 -- 172.21.15.40:0/3970154024 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7c018920 con 0x7fab8c101910 2024-03-18T22:58:32.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab93ef8640 1 -- 172.21.15.40:0/3970154024 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab8c19e840 con 0x7fab8c101910 2024-03-18T22:58:32.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.121+0000 7fab93ef8640 1 -- 172.21.15.40:0/3970154024 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab8c19ede0 con 0x7fab8c101910 2024-03-18T22:58:32.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.124+0000 7fab82ffd640 1 -- 172.21.15.40:0/3970154024 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fab7c010a90 con 0x7fab8c101910 2024-03-18T22:58:32.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.124+0000 7fab96bc4740 1 Processor -- start 2024-03-18T22:58:32.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.124+0000 7fab96bc4740 1 -- start start 2024-03-18T22:58:32.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.124+0000 7fab96bc4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0c7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.124+0000 7fab96bc4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0c7ce0 0x56527b0cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.124+0000 7fab96bc4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0cce50 0x56527b0cf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.124+0000 7fab96bc4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56527b02b4b0 con 0x56527b0c72f0 2024-03-18T22:58:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.124+0000 7fab96bc4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56527afd3d70 con 0x56527b0cce50 2024-03-18T22:58:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.124+0000 7fab96bc4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56527b027550 con 0x56527b0c7ce0 2024-03-18T22:58:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.124+0000 7fab92495640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.124+0000 7fab92495640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0c7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33860/0 (socket says 172.21.15.40:33860) 2024-03-18T22:58:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.124+0000 7fab92495640 1 -- 172.21.15.40:0/3337994819 learned_addr learned my addr 172.21.15.40:0/3337994819 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.124+0000 7fab91493640 1 --2- 172.21.15.40:0/3337994819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0cce50 0x56527b0cf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.124+0000 7fab91c94640 1 --2- 172.21.15.40:0/3337994819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0c7ce0 0x56527b0cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.124+0000 7fab82ffd640 1 --2- 172.21.15.40:0/3970154024 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab780a2880 0x7fab780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.125+0000 7fab92495640 1 -- 172.21.15.40:0/3337994819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0c7ce0 msgr2=0x56527b0cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.125+0000 7fab92495640 1 --2- 172.21.15.40:0/3337994819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0c7ce0 0x56527b0cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.125+0000 7fab92495640 1 -- 172.21.15.40:0/3337994819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0cce50 msgr2=0x56527b0cf2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:32.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.125+0000 7fab92495640 1 --2- 172.21.15.40:0/3337994819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0cce50 0x56527b0cf2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.125+0000 7fab92495640 1 -- 172.21.15.40:0/3337994819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56527aff6210 con 0x56527b0c72f0 2024-03-18T22:58:32.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.125+0000 7fab91c94640 1 --2- 172.21.15.40:0/3337994819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0c7ce0 0x56527b0cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:32.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.125+0000 7fab82ffd640 1 -- 172.21.15.40:0/3970154024 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fab7c0cf0e0 con 0x7fab8c101910 2024-03-18T22:58:32.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.125+0000 7fab92495640 1 --2- 172.21.15.40:0/3970154024 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab780a2880 0x7fab780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.125+0000 7fab92495640 1 --2- 172.21.15.40:0/3337994819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0c7710 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fab84011590 tx=0x7fab84011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eea727208561e60 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:32.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.125+0000 7fab77fff640 1 -- 172.21.15.40:0/3337994819 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab84017050 con 0x56527b0c72f0 2024-03-18T22:58:32.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.125+0000 7fab77fff640 1 -- 172.21.15.40:0/3337994819 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab8400fcf0 con 0x56527b0c72f0 2024-03-18T22:58:32.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.125+0000 7fab77fff640 1 -- 172.21.15.40:0/3337994819 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8401b9b0 con 0x56527b0c72f0 2024-03-18T22:58:32.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.126+0000 7fab96bc4740 1 -- 172.21.15.40:0/3337994819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 msgr2=0x56527b0c7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:32.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.126+0000 7fab92495640 1 --2- 172.21.15.40:0/3970154024 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab780a2880 0x7fab780a4d40 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fab84022000 tx=0x7fab84023000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:32.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.126+0000 7fab96bc4740 1 --2- 172.21.15.40:0/3337994819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0c7710 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fab84011590 tx=0x7fab84011a50 comp rx=0 tx=0).stop 2024-03-18T22:58:32.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.126+0000 7fab96bc4740 1 -- 172.21.15.40:0/3337994819 shutdown_connections 2024-03-18T22:58:32.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.126+0000 7fab96bc4740 1 --2- 172.21.15.40:0/3337994819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0cce50 0x56527b0cf2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.126+0000 7fab96bc4740 1 --2- 172.21.15.40:0/3337994819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0c7ce0 0x56527b0cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.126+0000 7fab96bc4740 1 --2- 172.21.15.40:0/3337994819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0c7710 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.126+0000 7fab96bc4740 1 -- 172.21.15.40:0/3337994819 >> 172.21.15.40:0/3337994819 conn(0x56527b0c2860 msgr2=0x56527b0c2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:32.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.126+0000 7fab96bc4740 1 -- 172.21.15.40:0/3337994819 shutdown_connections 2024-03-18T22:58:32.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.126+0000 7fab96bc4740 1 -- 172.21.15.40:0/3337994819 wait complete. 2024-03-18T22:58:32.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.126+0000 7fab96bc4740 1 -- start start 2024-03-18T22:58:32.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.126+0000 7fab96bc4740 1 Processor -- start 2024-03-18T22:58:32.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab96bc4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c72f0 0x56527b0d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab96bc4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0c7ce0 0x56527b0d1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab96bc4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0cce50 0x56527b0d8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab96bc4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x56527b0cfe00 con 0x56527b0cce50 2024-03-18T22:58:32.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab96bc4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56527afe7310 con 0x56527b0c72f0 2024-03-18T22:58:32.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab96bc4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x56527b0cff80 con 0x56527b0c7ce0 2024-03-18T22:58:32.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab96bc4740 10 client.0 _ll_register_callbacks cb 0x56527afdb2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:32.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab92495640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0cce50 0x56527b0d8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab91c94640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c72f0 0x56527b0d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab92495640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0cce50 0x56527b0d8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33862/0 (socket says 172.21.15.40:33862) 2024-03-18T22:58:32.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab91493640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0c7ce0 0x56527b0d1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab92495640 1 -- 172.21.15.40:0/2972521269 learned_addr learned my addr 172.21.15.40:0/2972521269 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:32.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab91c94640 1 -- 172.21.15.40:0/2972521269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0c7ce0 msgr2=0x56527b0d1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:32.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab91c94640 1 --2- 172.21.15.40:0/2972521269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0c7ce0 0x56527b0d1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab91c94640 1 -- 172.21.15.40:0/2972521269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0cce50 msgr2=0x56527b0d8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:32.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab91c94640 1 --2- 172.21.15.40:0/2972521269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0cce50 0x56527b0d8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab91c94640 1 -- 172.21.15.40:0/2972521269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56527ad3f3f0 con 0x56527b0c72f0 2024-03-18T22:58:32.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab92495640 1 --2- 172.21.15.40:0/2972521269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0cce50 0x56527b0d8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:32.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab91493640 1 --2- 172.21.15.40:0/2972521269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0c7ce0 0x56527b0d1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:32.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab91c94640 1 --2- 172.21.15.40:0/2972521269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c72f0 0x56527b0d17d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fab88036040 tx=0x7fab88004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fda9865e12c70926 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:32.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab757fa640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-03-18T22:58:32.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.127+0000 7fab757fa640 1 -- 172.21.15.40:0/2972521269 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88005030 con 0x56527b0c72f0 2024-03-18T22:58:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.128+0000 7fab96bc4740 1 -- 172.21.15.40:0/2972521269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab8400fcf0 con 0x56527b0c72f0 2024-03-18T22:58:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.128+0000 7fab757fa640 1 -- 172.21.15.40:0/2972521269 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fab8803a040 con 0x56527b0c72f0 2024-03-18T22:58:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.128+0000 7fab757fa640 1 -- 172.21.15.40:0/2972521269 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fab880051d0 con 0x56527b0c72f0 2024-03-18T22:58:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.128+0000 7fab96bc4740 10 client.28652 fetch_fsmap learned FSMap version 49 2024-03-18T22:58:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.128+0000 7fab96bc4740 1 -- 172.21.15.40:0/2972521269 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56527ae097d0 con 0x56527b0c72f0 2024-03-18T22:58:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.128+0000 7fab757fa640 1 -- 172.21.15.40:0/2972521269 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab8400fcf0 con 0x56527b0c72f0 2024-03-18T22:58:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.128+0000 7fab757fa640 1 -- 172.21.15.40:0/2972521269 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 49) ==== 8418+0+0 (secure 0 0 0) 0x7fab880035f0 con 0x56527b0c72f0 2024-03-18T22:58:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.128+0000 7fab96bc4740 10 client.28652 fetch_fsmap finished waiting for FSMap version 49 2024-03-18T22:58:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.128+0000 7fab96bc4740 10 client.28652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:58:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.128+0000 7fab96bc4740 20 client.28652 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.128+0000 7fab96bc4740 1 --2- 172.21.15.40:0/2972521269 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56527b0dd5c0 0x56527b0dfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.128+0000 7fab96bc4740 4 client.28652 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T22:58:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.128+0000 7fab96bc4740 1 -- 172.21.15.40:0/2972521269 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56527b0893e0 con 0x56527b0dd5c0 2024-03-18T22:58:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.128+0000 7fab91493640 1 --2- 172.21.15.40:0/2972521269 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56527b0dd5c0 0x56527b0dfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.129+0000 7fab757fa640 10 client.28652 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:32.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.129+0000 7fab91493640 1 --2- 172.21.15.40:0/2972521269 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56527b0dd5c0 0x56527b0dfa40 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8879cb46c63c2d69 server_cookie=970c7fb00ef65f5b in_seq=0 out_seq=0 2024-03-18T22:58:32.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.130+0000 7fab757fa640 1 -- 172.21.15.40:0/2972521269 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56527b0893e0 con 0x56527b0dd5c0 2024-03-18T22:58:32.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.130+0000 7fab757fa640 10 client.28652 handle_command_reply: tid=0 2024-03-18T22:58:32.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.130+0000 7fab96bc4740 1 client.28652 shutdown 2024-03-18T22:58:32.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.130+0000 7fab96bc4740 1 -- 172.21.15.40:0/2972521269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c72f0 msgr2=0x56527b0d17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:32.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.130+0000 7fab96bc4740 1 --2- 172.21.15.40:0/2972521269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c72f0 0x56527b0d17d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fab88036040 tx=0x7fab88004270 comp rx=0 tx=0).stop 2024-03-18T22:58:32.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.130+0000 7fab96bc4740 1 -- 172.21.15.40:0/2972521269 shutdown_connections 2024-03-18T22:58:32.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.130+0000 7fab96bc4740 1 --2- 172.21.15.40:0/2972521269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0cce50 0x56527b0d8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.130+0000 7fab96bc4740 1 --2- 172.21.15.40:0/2972521269 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0c7ce0 0x56527b0d1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.130+0000 7fab96bc4740 1 --2- 172.21.15.40:0/2972521269 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56527b0dd5c0 0x56527b0dfa40 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.130+0000 7fab96bc4740 1 --2- 172.21.15.40:0/2972521269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c72f0 0x56527b0d17d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.130+0000 7fab757fa640 0 client.28652 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:32.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.130+0000 7fab96bc4740 1 -- 172.21.15.40:0/2972521269 >> 172.21.15.40:0/2972521269 conn(0x56527b0c2860 msgr2=0x56527b0c1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:32.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.130+0000 7fab96bc4740 1 -- 172.21.15.40:0/2972521269 shutdown_connections 2024-03-18T22:58:32.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.130+0000 7fab96bc4740 1 -- 172.21.15.40:0/2972521269 wait complete. 2024-03-18T22:58:32.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.130+0000 7fab96bc4740 20 client.28652 trim_cache size 0 max 16384 2024-03-18T22:58:32.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.143+0000 7fab96bc4740 1 Processor -- start 2024-03-18T22:58:32.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab96bc4740 1 -- start start 2024-03-18T22:58:32.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab96bc4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0bda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab96bc4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c7ce0 0x56527b0d8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab96bc4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0cce50 0x56527b0c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab96bc4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fab8c1a82f0 con 0x56527b0c72f0 2024-03-18T22:58:32.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab96bc4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fab8c1aa010 con 0x56527b0c7ce0 2024-03-18T22:58:32.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab96bc4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fab8c1a89c0 con 0x56527b0cce50 2024-03-18T22:58:32.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab91c94640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0bda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab91c94640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0bda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33868/0 (socket says 172.21.15.40:33868) 2024-03-18T22:58:32.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab91c94640 1 -- 172.21.15.40:0/2420364762 learned_addr learned my addr 172.21.15.40:0/2420364762 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:32.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab91493640 1 --2- 172.21.15.40:0/2420364762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c7ce0 0x56527b0d8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab92495640 1 --2- 172.21.15.40:0/2420364762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0cce50 0x56527b0c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab91c94640 1 -- 172.21.15.40:0/2420364762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0cce50 msgr2=0x56527b0c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:32.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab91c94640 1 --2- 172.21.15.40:0/2420364762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0cce50 0x56527b0c61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab91c94640 1 -- 172.21.15.40:0/2420364762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c7ce0 msgr2=0x56527b0d8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:32.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab91c94640 1 --2- 172.21.15.40:0/2420364762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c7ce0 0x56527b0d8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab91c94640 1 -- 172.21.15.40:0/2420364762 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56527b0c6f30 con 0x56527b0c72f0 2024-03-18T22:58:32.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab91c94640 1 --2- 172.21.15.40:0/2420364762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0bda30 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fab88000c00 tx=0x7fab88005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c4d106cbdc60a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:32.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab777fe640 1 -- 172.21.15.40:0/2420364762 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88055030 con 0x56527b0c72f0 2024-03-18T22:58:32.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab777fe640 1 -- 172.21.15.40:0/2420364762 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab88003c30 con 0x56527b0c72f0 2024-03-18T22:58:32.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.144+0000 7fab777fe640 1 -- 172.21.15.40:0/2420364762 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88004a20 con 0x56527b0c72f0 2024-03-18T22:58:32.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.145+0000 7fab96bc4740 1 -- 172.21.15.40:0/2420364762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 msgr2=0x56527b0bda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:32.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.145+0000 7fab96bc4740 1 --2- 172.21.15.40:0/2420364762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0bda30 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fab88000c00 tx=0x7fab88005a20 comp rx=0 tx=0).stop 2024-03-18T22:58:32.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.145+0000 7fab96bc4740 1 -- 172.21.15.40:0/2420364762 shutdown_connections 2024-03-18T22:58:32.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.145+0000 7fab96bc4740 1 --2- 172.21.15.40:0/2420364762 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0cce50 0x56527b0c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.145+0000 7fab96bc4740 1 --2- 172.21.15.40:0/2420364762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c7ce0 0x56527b0d8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.145+0000 7fab96bc4740 1 --2- 172.21.15.40:0/2420364762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0bda30 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.145+0000 7fab96bc4740 1 -- 172.21.15.40:0/2420364762 >> 172.21.15.40:0/2420364762 conn(0x56527b0c2860 msgr2=0x56527b0c5ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:32.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.145+0000 7fab96bc4740 1 -- 172.21.15.40:0/2420364762 shutdown_connections 2024-03-18T22:58:32.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.145+0000 7fab96bc4740 1 -- 172.21.15.40:0/2420364762 wait complete. 2024-03-18T22:58:32.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.145+0000 7fab96bc4740 1 -- start start 2024-03-18T22:58:32.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.145+0000 7fab96bc4740 1 Processor -- start 2024-03-18T22:58:32.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab96bc4740 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0d5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab96bc4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c7ce0 0x56527b0d6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab96bc4740 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0cce50 0x56527b141930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab96bc4740 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fab8c1a9880 con 0x56527b0c72f0 2024-03-18T22:58:32.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab96bc4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fab8c1a9150 con 0x56527b0c7ce0 2024-03-18T22:58:32.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab96bc4740 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fab8c1a6270 con 0x56527b0cce50 2024-03-18T22:58:32.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab96bc4740 10 client.0 _ll_register_callbacks cb 0x7fab88009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T22:58:32.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab91c94640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0d5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab91c94640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0d5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33884/0 (socket says 172.21.15.40:33884) 2024-03-18T22:58:32.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab91c94640 1 -- 172.21.15.40:0/1113179412 learned_addr learned my addr 172.21.15.40:0/1113179412 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:32.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab92495640 1 --2- 172.21.15.40:0/1113179412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0cce50 0x56527b141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab91493640 1 --2- 172.21.15.40:0/1113179412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c7ce0 0x56527b0d6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab91c94640 1 -- 172.21.15.40:0/1113179412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0cce50 msgr2=0x56527b141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:32.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab91c94640 1 --2- 172.21.15.40:0/1113179412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0cce50 0x56527b141930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab91c94640 1 -- 172.21.15.40:0/1113179412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c7ce0 msgr2=0x56527b0d6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T22:58:32.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab91c94640 1 --2- 172.21.15.40:0/1113179412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c7ce0 0x56527b0d6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab91c94640 1 -- 172.21.15.40:0/1113179412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56527b141e70 con 0x56527b0c72f0 2024-03-18T22:58:32.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab91493640 1 --2- 172.21.15.40:0/1113179412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c7ce0 0x56527b0d6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:32.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab91c94640 1 --2- 172.21.15.40:0/1113179412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0d5af0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fab8802c170 tx=0x7fab88004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58faac47fea006ae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:32.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab757fa640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-18T22:58:32.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab757fa640 1 -- 172.21.15.40:0/1113179412 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88055070 con 0x56527b0c72f0 2024-03-18T22:58:32.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab757fa640 1 -- 172.21.15.40:0/1113179412 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fab880047a0 con 0x56527b0c72f0 2024-03-18T22:58:32.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.146+0000 7fab96bc4740 1 -- 172.21.15.40:0/1113179412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab88003c30 con 0x56527b0c72f0 2024-03-18T22:58:32.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.147+0000 7fab757fa640 1 -- 172.21.15.40:0/1113179412 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fab88064050 con 0x56527b0c72f0 2024-03-18T22:58:32.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.147+0000 7fab96bc4740 10 client.18849 fetch_fsmap learned FSMap version 49 2024-03-18T22:58:32.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.147+0000 7fab96bc4740 1 -- 172.21.15.40:0/1113179412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56527b143050 con 0x56527b0c72f0 2024-03-18T22:58:32.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.147+0000 7fab757fa640 1 -- 172.21.15.40:0/1113179412 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab88003c30 con 0x56527b0c72f0 2024-03-18T22:58:32.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.147+0000 7fab757fa640 1 -- 172.21.15.40:0/1113179412 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 49) ==== 8418+0+0 (secure 0 0 0) 0x7fab8803e960 con 0x56527b0c72f0 2024-03-18T22:58:32.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.147+0000 7fab96bc4740 10 client.18849 fetch_fsmap finished waiting for FSMap version 49 2024-03-18T22:58:32.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.147+0000 7fab96bc4740 10 client.18849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T22:58:32.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.147+0000 7fab96bc4740 20 client.18849 populate_metadata read hostname 'smithi040' 2024-03-18T22:58:32.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.147+0000 7fab96bc4740 1 --2- 172.21.15.40:0/1113179412 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56527b143a00 0x56527b145e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:32.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.147+0000 7fab96bc4740 4 client.18849 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T22:58:32.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.147+0000 7fab96bc4740 1 -- 172.21.15.40:0/1113179412 --> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56527b0389f0 con 0x56527b143a00 2024-03-18T22:58:32.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.147+0000 7fab91493640 1 --2- 172.21.15.40:0/1113179412 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56527b143a00 0x56527b145e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:32.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab91493640 1 --2- 172.21.15.40:0/1113179412 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56527b143a00 0x56527b145e20 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fb75b611ec4c7e0 server_cookie=7ad778c2c516779e in_seq=0 out_seq=0 2024-03-18T22:58:32.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab757fa640 10 client.18849 ms_handle_connect on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:32.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab757fa640 1 -- 172.21.15.40:0/1113179412 <== mds.0 v2:172.21.15.103:6838/2609728202 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56527b0389f0 con 0x56527b143a00 2024-03-18T22:58:32.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab757fa640 10 client.18849 handle_command_reply: tid=0 2024-03-18T22:58:32.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab96bc4740 1 client.18849 shutdown 2024-03-18T22:58:32.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab96bc4740 1 -- 172.21.15.40:0/1113179412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 msgr2=0x56527b0d5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:32.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab96bc4740 1 --2- 172.21.15.40:0/1113179412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0d5af0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fab8802c170 tx=0x7fab88004250 comp rx=0 tx=0).stop 2024-03-18T22:58:32.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab96bc4740 1 -- 172.21.15.40:0/1113179412 shutdown_connections 2024-03-18T22:58:32.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab96bc4740 1 --2- 172.21.15.40:0/1113179412 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x56527b0cce50 0x56527b141930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab96bc4740 1 --2- 172.21.15.40:0/1113179412 >> [v2:172.21.15.103:6838/2609728202,v1:172.21.15.103:6839/2609728202] conn(0x56527b143a00 0x56527b145e20 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab96bc4740 1 --2- 172.21.15.40:0/1113179412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56527b0c7ce0 0x56527b0d6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab96bc4740 1 --2- 172.21.15.40:0/1113179412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56527b0c72f0 0x56527b0d5af0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab757fa640 0 client.18849 ms_handle_reset on v2:172.21.15.103:6838/2609728202 2024-03-18T22:58:32.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab96bc4740 1 -- 172.21.15.40:0/1113179412 >> 172.21.15.40:0/1113179412 conn(0x56527b0c2860 msgr2=0x56527b0c6350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:32.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab96bc4740 1 -- 172.21.15.40:0/1113179412 shutdown_connections 2024-03-18T22:58:32.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab96bc4740 1 -- 172.21.15.40:0/1113179412 wait complete. 2024-03-18T22:58:32.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.148+0000 7fab96bc4740 20 client.18849 trim_cache size 0 max 16384 2024-03-18T22:58:32.156 INFO:teuthology.orchestra.run.smithi040.stdout:[] 2024-03-18T22:58:32.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.149+0000 7fab93ef8640 1 -- 172.21.15.40:0/3970154024 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab780a2880 msgr2=0x7fab780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:32.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.149+0000 7fab93ef8640 1 --2- 172.21.15.40:0/3970154024 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab780a2880 0x7fab780a4d40 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fab84022000 tx=0x7fab84023000 comp rx=0 tx=0).stop 2024-03-18T22:58:32.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.149+0000 7fab93ef8640 1 -- 172.21.15.40:0/3970154024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab8c101910 msgr2=0x7fab8c196e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:32.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.149+0000 7fab93ef8640 1 --2- 172.21.15.40:0/3970154024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab8c101910 0x7fab8c196e70 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fab7c00e9f0 tx=0x7fab7c00eeb0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.149+0000 7fab93ef8640 1 -- 172.21.15.40:0/3970154024 shutdown_connections 2024-03-18T22:58:32.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.149+0000 7fab93ef8640 1 --2- 172.21.15.40:0/3970154024 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fab780a2880 0x7fab780a4d40 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.149+0000 7fab93ef8640 1 --2- 172.21.15.40:0/3970154024 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fab8c1022d0 0x7fab8c19dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.149+0000 7fab93ef8640 1 --2- 172.21.15.40:0/3970154024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab8c101910 0x7fab8c196e70 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.149+0000 7fab93ef8640 1 --2- 172.21.15.40:0/3970154024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab8c100720 0x7fab8c196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:32.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.149+0000 7fab93ef8640 1 -- 172.21.15.40:0/3970154024 >> 172.21.15.40:0/3970154024 conn(0x7fab8c0fbcb0 msgr2=0x7fab8c104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:32.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.149+0000 7fab93ef8640 1 -- 172.21.15.40:0/3970154024 shutdown_connections 2024-03-18T22:58:32.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:32.149+0000 7fab93ef8640 1 -- 172.21.15.40:0/3970154024 wait complete. 2024-03-18T22:58:32.760 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:32 smithi103 ceph-mon[66377]: pgmap v827: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-18T22:58:32.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:32 smithi113 ceph-mon[66775]: pgmap v827: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-18T22:58:32.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:32 smithi040 ceph-mon[62643]: pgmap v827: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-18T22:58:33.161 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-18T22:58:33.161 INFO:tasks.fwd_scrub:done joining 2024-03-18T22:58:33.161 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-18T22:58:33.172 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:58:33.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.286+0000 7fac7cb59640 1 -- 172.21.15.40:0/3148066816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781022f0 msgr2=0x7fac78108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:33.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.286+0000 7fac7cb59640 1 --2- 172.21.15.40:0/3148066816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781022f0 0x7fac78108b80 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fac68009ca0 tx=0x7fac6802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.287+0000 7fac7cb59640 1 -- 172.21.15.40:0/3148066816 shutdown_connections 2024-03-18T22:58:33.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.287+0000 7fac7cb59640 1 --2- 172.21.15.40:0/3148066816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781022f0 0x7fac78108b80 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.287+0000 7fac7cb59640 1 --2- 172.21.15.40:0/3148066816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac78101930 0x7fac78101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.287+0000 7fac7cb59640 1 --2- 172.21.15.40:0/3148066816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac78100740 0x7fac78100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.287+0000 7fac7cb59640 1 -- 172.21.15.40:0/3148066816 >> 172.21.15.40:0/3148066816 conn(0x7fac780fbcb0 msgr2=0x7fac780fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:33.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.287+0000 7fac7cb59640 1 -- 172.21.15.40:0/3148066816 shutdown_connections 2024-03-18T22:58:33.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.287+0000 7fac7cb59640 1 -- 172.21.15.40:0/3148066816 wait complete. 2024-03-18T22:58:33.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.287+0000 7fac7cb59640 1 Processor -- start 2024-03-18T22:58:33.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.287+0000 7fac7cb59640 1 -- start start 2024-03-18T22:58:33.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac7cb59640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac78100740 0x7fac78197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:33.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac7cb59640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac78101930 0x7fac78197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:33.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac7cb59640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781022f0 0x7fac7819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:33.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac7cb59640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fac7810c250 con 0x7fac781022f0 2024-03-18T22:58:33.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac7cb59640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fac7810c0d0 con 0x7fac78100740 2024-03-18T22:58:33.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac7cb59640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fac7810c3d0 con 0x7fac78101930 2024-03-18T22:58:33.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac76d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781022f0 0x7fac7819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:33.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac7659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac78100740 0x7fac78197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:33.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac76d9d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781022f0 0x7fac7819f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33910/0 (socket says 172.21.15.40:33910) 2024-03-18T22:58:33.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac76d9d640 1 -- 172.21.15.40:0/3710655293 learned_addr learned my addr 172.21.15.40:0/3710655293 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac75d9b640 1 --2- 172.21.15.40:0/3710655293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac78101930 0x7fac78197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac76d9d640 1 -- 172.21.15.40:0/3710655293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac78101930 msgr2=0x7fac78197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac76d9d640 1 --2- 172.21.15.40:0/3710655293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac78101930 0x7fac78197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac76d9d640 1 -- 172.21.15.40:0/3710655293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac78100740 msgr2=0x7fac78197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac76d9d640 1 --2- 172.21.15.40:0/3710655293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac78100740 0x7fac78197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac76d9d640 1 -- 172.21.15.40:0/3710655293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac7819f670 con 0x7fac781022f0 2024-03-18T22:58:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac7659c640 1 --2- 172.21.15.40:0/3710655293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac78100740 0x7fac78197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac76d9d640 1 --2- 172.21.15.40:0/3710655293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781022f0 0x7fac7819f000 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fac6802c700 tx=0x7fac680328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac6f7fe640 1 -- 172.21.15.40:0/3710655293 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac68041020 con 0x7fac781022f0 2024-03-18T22:58:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac6f7fe640 1 -- 172.21.15.40:0/3710655293 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac68003e20 con 0x7fac781022f0 2024-03-18T22:58:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac7cb59640 1 -- 172.21.15.40:0/3710655293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac7819f900 con 0x7fac781022f0 2024-03-18T22:58:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac6f7fe640 1 -- 172.21.15.40:0/3710655293 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac68039040 con 0x7fac781022f0 2024-03-18T22:58:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.288+0000 7fac7cb59640 1 -- 172.21.15.40:0/3710655293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac7819fdf0 con 0x7fac781022f0 2024-03-18T22:58:33.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.290+0000 7fac7cb59640 1 -- 172.21.15.40:0/3710655293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac781a0050 con 0x7fac781022f0 2024-03-18T22:58:33.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.291+0000 7fac6f7fe640 1 -- 172.21.15.40:0/3710655293 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fac68044020 con 0x7fac781022f0 2024-03-18T22:58:33.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.291+0000 7fac6f7fe640 1 --2- 172.21.15.40:0/3710655293 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fac640a2800 0x7fac640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:33.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.291+0000 7fac7659c640 1 --2- 172.21.15.40:0/3710655293 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fac640a2800 0x7fac640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:33.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.292+0000 7fac6f7fe640 1 -- 172.21.15.40:0/3710655293 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fac680f0fe0 con 0x7fac781022f0 2024-03-18T22:58:33.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.292+0000 7fac7659c640 1 --2- 172.21.15.40:0/3710655293 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fac640a2800 0x7fac640a4cc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fac78101790 tx=0x7fac70005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:33.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.296+0000 7fac6f7fe640 1 -- 172.21.15.40:0/3710655293 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fac6804b020 con 0x7fac781022f0 2024-03-18T22:58:33.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.483+0000 7fac7cb59640 1 -- 172.21.15.40:0/3710655293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac781a03e0 con 0x7fac781022f0 2024-03-18T22:58:33.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.484+0000 7fac6f7fe640 1 -- 172.21.15.40:0/3710655293 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24908 (secure 0 0 0) 0x7fac680bdc90 con 0x7fac781022f0 2024-03-18T22:58:33.485 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:58:33.485 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":93,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:42:44.941873+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:58:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.486+0000 7fac7cb59640 1 -- 172.21.15.40:0/3710655293 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fac640a2800 msgr2=0x7fac640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.486+0000 7fac7cb59640 1 --2- 172.21.15.40:0/3710655293 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fac640a2800 0x7fac640a4cc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fac78101790 tx=0x7fac70005f90 comp rx=0 tx=0).stop 2024-03-18T22:58:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.486+0000 7fac7cb59640 1 -- 172.21.15.40:0/3710655293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781022f0 msgr2=0x7fac7819f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.486+0000 7fac7cb59640 1 --2- 172.21.15.40:0/3710655293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781022f0 0x7fac7819f000 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fac6802c700 tx=0x7fac680328a0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.486+0000 7fac7cb59640 1 -- 172.21.15.40:0/3710655293 shutdown_connections 2024-03-18T22:58:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.486+0000 7fac7cb59640 1 --2- 172.21.15.40:0/3710655293 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fac640a2800 0x7fac640a4cc0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.486+0000 7fac7cb59640 1 --2- 172.21.15.40:0/3710655293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fac781022f0 0x7fac7819f000 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.486+0000 7fac7cb59640 1 --2- 172.21.15.40:0/3710655293 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fac78101930 0x7fac78197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.486+0000 7fac7cb59640 1 --2- 172.21.15.40:0/3710655293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac78100740 0x7fac78197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.486+0000 7fac7cb59640 1 -- 172.21.15.40:0/3710655293 >> 172.21.15.40:0/3710655293 conn(0x7fac780fbcb0 msgr2=0x7fac780fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:33.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.487+0000 7fac7cb59640 1 -- 172.21.15.40:0/3710655293 shutdown_connections 2024-03-18T22:58:33.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.487+0000 7fac7cb59640 1 -- 172.21.15.40:0/3710655293 wait complete. 2024-03-18T22:58:33.499 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T22:36:07.060691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T22:40:39.925147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T22:40:40.773757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T22:40:44.068183+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T22:58:33.499 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T22:58:33.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.604+0000 7f6872987640 1 -- 172.21.15.40:0/3396038804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f686c101910 msgr2=0x7f686c101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:33.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.604+0000 7f6872987640 1 --2- 172.21.15.40:0/3396038804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f686c101910 0x7f686c101d90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6860009860 tx=0x7f686002bb70 comp rx=0 tx=0).stop 2024-03-18T22:58:33.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.604+0000 7f6872987640 1 -- 172.21.15.40:0/3396038804 shutdown_connections 2024-03-18T22:58:33.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.604+0000 7f6872987640 1 --2- 172.21.15.40:0/3396038804 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f686c1022d0 0x7f686c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.604+0000 7f6872987640 1 --2- 172.21.15.40:0/3396038804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f686c101910 0x7f686c101d90 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.604+0000 7f6872987640 1 --2- 172.21.15.40:0/3396038804 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f686c100720 0x7f686c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.604+0000 7f6872987640 1 -- 172.21.15.40:0/3396038804 >> 172.21.15.40:0/3396038804 conn(0x7f686c0fbcb0 msgr2=0x7f686c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:33.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.605+0000 7f6872987640 1 -- 172.21.15.40:0/3396038804 shutdown_connections 2024-03-18T22:58:33.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.605+0000 7f6872987640 1 -- 172.21.15.40:0/3396038804 wait complete. 2024-03-18T22:58:33.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.605+0000 7f6872987640 1 Processor -- start 2024-03-18T22:58:33.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.605+0000 7f6872987640 1 -- start start 2024-03-18T22:58:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.605+0000 7f6872987640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f686c100720 0x7f686c1969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.605+0000 7f6872987640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f686c101910 0x7f686c196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.605+0000 7f6872987640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f686c1022d0 0x7f686c19df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.605+0000 7f6872987640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f686c10c250 con 0x7f686c1022d0 2024-03-18T22:58:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.605+0000 7f6872987640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f686c10c0d0 con 0x7f686c100720 2024-03-18T22:58:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.605+0000 7f6872987640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f686c10c3d0 con 0x7f686c101910 2024-03-18T22:58:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f686bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f686c100720 0x7f686c1969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f6870f24640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f686c1022d0 0x7f686c19df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f686bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f686c100720 0x7f686c1969c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48850/0 (socket says 172.21.15.40:48850) 2024-03-18T22:58:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f686bfff640 1 -- 172.21.15.40:0/1322831528 learned_addr learned my addr 172.21.15.40:0/1322831528 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f686b7fe640 1 --2- 172.21.15.40:0/1322831528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f686c101910 0x7f686c196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f6870f24640 1 -- 172.21.15.40:0/1322831528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f686c101910 msgr2=0x7f686c196f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f6870f24640 1 --2- 172.21.15.40:0/1322831528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f686c101910 0x7f686c196f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f6870f24640 1 -- 172.21.15.40:0/1322831528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f686c100720 msgr2=0x7f686c1969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:33.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f6870f24640 1 --2- 172.21.15.40:0/1322831528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f686c100720 0x7f686c1969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f6870f24640 1 -- 172.21.15.40:0/1322831528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f686c19e5e0 con 0x7f686c1022d0 2024-03-18T22:58:33.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f686bfff640 1 --2- 172.21.15.40:0/1322831528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f686c100720 0x7f686c1969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:33.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f6870f24640 1 --2- 172.21.15.40:0/1322831528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f686c1022d0 0x7f686c19df70 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f686400efc0 tx=0x7f686400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:33.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f68697fa640 1 -- 172.21.15.40:0/1322831528 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6864003970 con 0x7f686c1022d0 2024-03-18T22:58:33.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f6872987640 1 -- 172.21.15.40:0/1322831528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f686c19e8d0 con 0x7f686c1022d0 2024-03-18T22:58:33.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f68697fa640 1 -- 172.21.15.40:0/1322831528 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6864004020 con 0x7f686c1022d0 2024-03-18T22:58:33.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f68697fa640 1 -- 172.21.15.40:0/1322831528 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686400b940 con 0x7f686c1022d0 2024-03-18T22:58:33.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.606+0000 7f6872987640 1 -- 172.21.15.40:0/1322831528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f686c19eda0 con 0x7f686c1022d0 2024-03-18T22:58:33.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.607+0000 7f6872987640 1 -- 172.21.15.40:0/1322831528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f686c107bb0 con 0x7f686c1022d0 2024-03-18T22:58:33.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.608+0000 7f68697fa640 1 -- 172.21.15.40:0/1322831528 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f686401d020 con 0x7f686c1022d0 2024-03-18T22:58:33.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.609+0000 7f68697fa640 1 --2- 172.21.15.40:0/1322831528 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f68540a2800 0x7f68540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:33.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.609+0000 7f68697fa640 1 -- 172.21.15.40:0/1322831528 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f68640cece0 con 0x7f686c1022d0 2024-03-18T22:58:33.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.609+0000 7f686bfff640 1 --2- 172.21.15.40:0/1322831528 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f68540a2800 0x7f68540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:33.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.609+0000 7f686bfff640 1 --2- 172.21.15.40:0/1322831528 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f68540a2800 0x7f68540a4cc0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f685c0058c0 tx=0x7f685c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:33.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.613+0000 7f68697fa640 1 -- 172.21.15.40:0/1322831528 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f686409b990 con 0x7f686c1022d0 2024-03-18T22:58:33.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.795+0000 7f6872987640 1 -- 172.21.15.40:0/1322831528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f686c062420 con 0x7f686c1022d0 2024-03-18T22:58:33.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.795+0000 7f68697fa640 1 -- 172.21.15.40:0/1322831528 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f686409b290 con 0x7f686c1022d0 2024-03-18T22:58:33.796 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-18T22:58:33.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.797+0000 7f6872987640 1 -- 172.21.15.40:0/1322831528 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f68540a2800 msgr2=0x7f68540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:33.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.797+0000 7f6872987640 1 --2- 172.21.15.40:0/1322831528 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f68540a2800 0x7f68540a4cc0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f685c0058c0 tx=0x7f685c005f90 comp rx=0 tx=0).stop 2024-03-18T22:58:33.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.797+0000 7f6872987640 1 -- 172.21.15.40:0/1322831528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f686c1022d0 msgr2=0x7f686c19df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:33.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.797+0000 7f6872987640 1 --2- 172.21.15.40:0/1322831528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f686c1022d0 0x7f686c19df70 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f686400efc0 tx=0x7f686400c800 comp rx=0 tx=0).stop 2024-03-18T22:58:33.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.797+0000 7f6872987640 1 -- 172.21.15.40:0/1322831528 shutdown_connections 2024-03-18T22:58:33.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.797+0000 7f6872987640 1 --2- 172.21.15.40:0/1322831528 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f68540a2800 0x7f68540a4cc0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.797+0000 7f6872987640 1 --2- 172.21.15.40:0/1322831528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f686c1022d0 0x7f686c19df70 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.797+0000 7f6872987640 1 --2- 172.21.15.40:0/1322831528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f686c101910 0x7f686c196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.797+0000 7f6872987640 1 --2- 172.21.15.40:0/1322831528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f686c100720 0x7f686c1969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.797+0000 7f6872987640 1 -- 172.21.15.40:0/1322831528 >> 172.21.15.40:0/1322831528 conn(0x7f686c0fbcb0 msgr2=0x7f686c104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:33.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.797+0000 7f6872987640 1 -- 172.21.15.40:0/1322831528 shutdown_connections 2024-03-18T22:58:33.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.797+0000 7f6872987640 1 -- 172.21.15.40:0/1322831528 wait complete. 2024-03-18T22:58:33.807 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:33 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3710655293' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:58:33.809 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T22:58:33.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:33 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3710655293' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:58:33.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:33 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3710655293' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:58:33.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.914+0000 7f5e619a9640 1 -- 172.21.15.40:0/220593687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e5c1022e0 msgr2=0x7f5e5c108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:33.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.914+0000 7f5e619a9640 1 --2- 172.21.15.40:0/220593687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e5c1022e0 0x7f5e5c108b70 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f5e54009ca0 tx=0x7f5e5402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.914+0000 7f5e619a9640 1 -- 172.21.15.40:0/220593687 shutdown_connections 2024-03-18T22:58:33.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.914+0000 7f5e619a9640 1 --2- 172.21.15.40:0/220593687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e5c1022e0 0x7f5e5c108b70 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.914+0000 7f5e619a9640 1 --2- 172.21.15.40:0/220593687 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e5c101920 0x7f5e5c101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.914+0000 7f5e619a9640 1 --2- 172.21.15.40:0/220593687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e5c100730 0x7f5e5c100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.914+0000 7f5e619a9640 1 -- 172.21.15.40:0/220593687 >> 172.21.15.40:0/220593687 conn(0x7f5e5c0fbcc0 msgr2=0x7f5e5c0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:33.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.914+0000 7f5e619a9640 1 -- 172.21.15.40:0/220593687 shutdown_connections 2024-03-18T22:58:33.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.914+0000 7f5e619a9640 1 -- 172.21.15.40:0/220593687 wait complete. 2024-03-18T22:58:33.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.914+0000 7f5e619a9640 1 Processor -- start 2024-03-18T22:58:33.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e619a9640 1 -- start start 2024-03-18T22:58:33.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e619a9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e5c100730 0x7f5e5c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:33.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e619a9640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e5c101920 0x7f5e5c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:33.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e619a9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e5c1022e0 0x7f5e5c19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:33.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e619a9640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5c10c260 con 0x7f5e5c100730 2024-03-18T22:58:33.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e619a9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5c10c0e0 con 0x7f5e5c1022e0 2024-03-18T22:58:33.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e619a9640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5c10c3e0 con 0x7f5e5c101920 2024-03-18T22:58:33.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e5affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e5c100730 0x7f5e5c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:33.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e5affd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e5c100730 0x7f5e5c197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33946/0 (socket says 172.21.15.40:33946) 2024-03-18T22:58:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e5affd640 1 -- 172.21.15.40:0/2923478592 learned_addr learned my addr 172.21.15.40:0/2923478592 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e5b7fe640 1 --2- 172.21.15.40:0/2923478592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e5c1022e0 0x7f5e5c19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e5a7fc640 1 --2- 172.21.15.40:0/2923478592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e5c101920 0x7f5e5c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e5affd640 1 -- 172.21.15.40:0/2923478592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e5c101920 msgr2=0x7f5e5c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e5affd640 1 --2- 172.21.15.40:0/2923478592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e5c101920 0x7f5e5c197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e5affd640 1 -- 172.21.15.40:0/2923478592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e5c1022e0 msgr2=0x7f5e5c19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e5affd640 1 --2- 172.21.15.40:0/2923478592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e5c1022e0 0x7f5e5c19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e5affd640 1 -- 172.21.15.40:0/2923478592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e5c19f670 con 0x7f5e5c100730 2024-03-18T22:58:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e5affd640 1 --2- 172.21.15.40:0/2923478592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e5c100730 0x7f5e5c197a50 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c00e9c0 tx=0x7f5e4c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.915+0000 7f5e609a7640 1 -- 172.21.15.40:0/2923478592 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c00cde0 con 0x7f5e5c100730 2024-03-18T22:58:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.916+0000 7f5e609a7640 1 -- 172.21.15.40:0/2923478592 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5e4c0108e0 con 0x7f5e5c100730 2024-03-18T22:58:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.916+0000 7f5e619a9640 1 -- 172.21.15.40:0/2923478592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e5c19f960 con 0x7f5e5c100730 2024-03-18T22:58:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.916+0000 7f5e609a7640 1 -- 172.21.15.40:0/2923478592 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c018990 con 0x7f5e5c100730 2024-03-18T22:58:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.916+0000 7f5e619a9640 1 -- 172.21.15.40:0/2923478592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e5c19feb0 con 0x7f5e5c100730 2024-03-18T22:58:33.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.916+0000 7f5e619a9640 1 -- 172.21.15.40:0/2923478592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e5c1a0080 con 0x7f5e5c100730 2024-03-18T22:58:33.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.918+0000 7f5e609a7640 1 -- 172.21.15.40:0/2923478592 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5e4c01f020 con 0x7f5e5c100730 2024-03-18T22:58:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.918+0000 7f5e609a7640 1 --2- 172.21.15.40:0/2923478592 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5e440a2800 0x7f5e440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.918+0000 7f5e5a7fc640 1 --2- 172.21.15.40:0/2923478592 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5e440a2800 0x7f5e440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.918+0000 7f5e609a7640 1 -- 172.21.15.40:0/2923478592 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5e4c0cf610 con 0x7f5e5c100730 2024-03-18T22:58:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.919+0000 7f5e5a7fc640 1 --2- 172.21.15.40:0/2923478592 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5e440a2800 0x7f5e440a4cc0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c198f60 tx=0x7f5e50005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:33.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:33.922+0000 7f5e609a7640 1 -- 172.21.15.40:0/2923478592 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5e4c09c2c0 con 0x7f5e5c100730 2024-03-18T22:58:34.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.104+0000 7f5e619a9640 1 -- 172.21.15.40:0/2923478592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5e5c1a0410 con 0x7f5e5c100730 2024-03-18T22:58:34.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.104+0000 7f5e609a7640 1 -- 172.21.15.40:0/2923478592 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f5e4c09bbc0 con 0x7f5e5c100730 2024-03-18T22:58:34.105 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:58:34.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.106+0000 7f5e619a9640 1 -- 172.21.15.40:0/2923478592 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5e440a2800 msgr2=0x7f5e440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:34.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.106+0000 7f5e619a9640 1 --2- 172.21.15.40:0/2923478592 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5e440a2800 0x7f5e440a4cc0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c198f60 tx=0x7f5e50005f90 comp rx=0 tx=0).stop 2024-03-18T22:58:34.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.106+0000 7f5e619a9640 1 -- 172.21.15.40:0/2923478592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e5c100730 msgr2=0x7f5e5c197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:34.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.106+0000 7f5e619a9640 1 --2- 172.21.15.40:0/2923478592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e5c100730 0x7f5e5c197a50 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c00e9c0 tx=0x7f5e4c00ee80 comp rx=0 tx=0).stop 2024-03-18T22:58:34.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.106+0000 7f5e619a9640 1 -- 172.21.15.40:0/2923478592 shutdown_connections 2024-03-18T22:58:34.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.106+0000 7f5e619a9640 1 --2- 172.21.15.40:0/2923478592 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5e440a2800 0x7f5e440a4cc0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.106+0000 7f5e619a9640 1 --2- 172.21.15.40:0/2923478592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5e5c1022e0 0x7f5e5c19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.106+0000 7f5e619a9640 1 --2- 172.21.15.40:0/2923478592 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5e5c101920 0x7f5e5c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.106+0000 7f5e619a9640 1 --2- 172.21.15.40:0/2923478592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e5c100730 0x7f5e5c197a50 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.106+0000 7f5e619a9640 1 -- 172.21.15.40:0/2923478592 >> 172.21.15.40:0/2923478592 conn(0x7f5e5c0fbcc0 msgr2=0x7f5e5c0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:34.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.106+0000 7f5e619a9640 1 -- 172.21.15.40:0/2923478592 shutdown_connections 2024-03-18T22:58:34.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.106+0000 7f5e619a9640 1 -- 172.21.15.40:0/2923478592 wait complete. 2024-03-18T22:58:34.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T22:58:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.223+0000 7f69368e8640 1 -- 172.21.15.40:0/3704857820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6930100740 msgr2=0x7f6930100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.223+0000 7f69368e8640 1 --2- 172.21.15.40:0/3704857820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6930100740 0x7f6930100b60 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f6920009900 tx=0x7f692002bc30 comp rx=0 tx=0).stop 2024-03-18T22:58:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.223+0000 7f69368e8640 1 -- 172.21.15.40:0/3704857820 shutdown_connections 2024-03-18T22:58:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.223+0000 7f69368e8640 1 --2- 172.21.15.40:0/3704857820 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69301022f0 0x7f6930108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.223+0000 7f69368e8640 1 --2- 172.21.15.40:0/3704857820 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6930101930 0x7f6930101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.223+0000 7f69368e8640 1 --2- 172.21.15.40:0/3704857820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6930100740 0x7f6930100b60 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.223+0000 7f69368e8640 1 -- 172.21.15.40:0/3704857820 >> 172.21.15.40:0/3704857820 conn(0x7f69300fbcd0 msgr2=0x7f69300fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.223+0000 7f69368e8640 1 -- 172.21.15.40:0/3704857820 shutdown_connections 2024-03-18T22:58:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.223+0000 7f69368e8640 1 -- 172.21.15.40:0/3704857820 wait complete. 2024-03-18T22:58:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.223+0000 7f69368e8640 1 Processor -- start 2024-03-18T22:58:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.223+0000 7f69368e8640 1 -- start start 2024-03-18T22:58:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f69368e8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6930100740 0x7f6930197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f69368e8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6930101930 0x7f6930198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f69368e8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69301022f0 0x7f693019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f69368e8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f693010d390 con 0x7f69301022f0 2024-03-18T22:58:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f69368e8640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f693010d210 con 0x7f6930101930 2024-03-18T22:58:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f69368e8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f693010d510 con 0x7f6930100740 2024-03-18T22:58:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f6934e85640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69301022f0 0x7f693019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f692f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6930101930 0x7f6930198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f692ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6930100740 0x7f6930197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f692f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6930101930 0x7f6930198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:48888/0 (socket says 172.21.15.40:48888) 2024-03-18T22:58:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f6934e85640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69301022f0 0x7f693019f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33974/0 (socket says 172.21.15.40:33974) 2024-03-18T22:58:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f692f7fe640 1 -- 172.21.15.40:0/3652063415 learned_addr learned my addr 172.21.15.40:0/3652063415 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f692ffff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6930100740 0x7f6930197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:38380/0 (socket says 172.21.15.40:38380) 2024-03-18T22:58:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f692ffff640 1 -- 172.21.15.40:0/3652063415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6930101930 msgr2=0x7f6930198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f692ffff640 1 --2- 172.21.15.40:0/3652063415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6930101930 0x7f6930198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f692ffff640 1 -- 172.21.15.40:0/3652063415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69301022f0 msgr2=0x7f693019f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f692ffff640 1 --2- 172.21.15.40:0/3652063415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69301022f0 0x7f693019f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f692ffff640 1 -- 172.21.15.40:0/3652063415 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f693019f6f0 con 0x7f6930100740 2024-03-18T22:58:34.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f6934e85640 1 --2- 172.21.15.40:0/3652063415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69301022f0 0x7f693019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:34.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.224+0000 7f692ffff640 1 --2- 172.21.15.40:0/3652063415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6930100740 0x7f6930197ad0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6920036040 tx=0x7f6920005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:34.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.225+0000 7f692d7fa640 1 -- 172.21.15.40:0/3652063415 <== mon.1 v2:172.21.15.113:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6920043020 con 0x7f6930100740 2024-03-18T22:58:34.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.225+0000 7f69368e8640 1 -- 172.21.15.40:0/3652063415 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f693019f980 con 0x7f6930100740 2024-03-18T22:58:34.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.225+0000 7f692d7fa640 1 -- 172.21.15.40:0/3652063415 <== mon.1 v2:172.21.15.113:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6920004350 con 0x7f6930100740 2024-03-18T22:58:34.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.225+0000 7f69368e8640 1 -- 172.21.15.40:0/3652063415 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f693019fef0 con 0x7f6930100740 2024-03-18T22:58:34.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.225+0000 7f692d7fa640 1 -- 172.21.15.40:0/3652063415 <== mon.1 v2:172.21.15.113:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6920003ab0 con 0x7f6930100740 2024-03-18T22:58:34.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.226+0000 7f69368e8640 1 -- 172.21.15.40:0/3652063415 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6930107b80 con 0x7f6930100740 2024-03-18T22:58:34.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.227+0000 7f692d7fa640 1 -- 172.21.15.40:0/3652063415 <== mon.1 v2:172.21.15.113:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6920046020 con 0x7f6930100740 2024-03-18T22:58:34.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.227+0000 7f692d7fa640 1 --2- 172.21.15.40:0/3652063415 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f69180a2800 0x7f69180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:34.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.228+0000 7f692f7fe640 1 --2- 172.21.15.40:0/3652063415 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f69180a2800 0x7f69180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:34.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.228+0000 7f692d7fa640 1 -- 172.21.15.40:0/3652063415 <== mon.1 v2:172.21.15.113:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f69200f1620 con 0x7f6930100740 2024-03-18T22:58:34.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.228+0000 7f692f7fe640 1 --2- 172.21.15.40:0/3652063415 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f69180a2800 0x7f69180a4cc0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6930198fe0 tx=0x7f6924005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.231+0000 7f692d7fa640 1 -- 172.21.15.40:0/3652063415 <== mon.1 v2:172.21.15.113:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f692004d020 con 0x7f6930100740 2024-03-18T22:58:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.414+0000 7f69368e8640 1 -- 172.21.15.40:0/3652063415 --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f693002d3a0 con 0x7f6930100740 2024-03-18T22:58:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.415+0000 7f692d7fa640 1 -- 172.21.15.40:0/3652063415 <== mon.1 v2:172.21.15.113:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f69200be270 con 0x7f6930100740 2024-03-18T22:58:34.415 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-18T22:58:34.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.417+0000 7f69368e8640 1 -- 172.21.15.40:0/3652063415 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f69180a2800 msgr2=0x7f69180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:34.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.417+0000 7f69368e8640 1 --2- 172.21.15.40:0/3652063415 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f69180a2800 0x7f69180a4cc0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6930198fe0 tx=0x7f6924005f90 comp rx=0 tx=0).stop 2024-03-18T22:58:34.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.417+0000 7f69368e8640 1 -- 172.21.15.40:0/3652063415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6930100740 msgr2=0x7f6930197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:34.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.417+0000 7f69368e8640 1 --2- 172.21.15.40:0/3652063415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6930100740 0x7f6930197ad0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6920036040 tx=0x7f6920005dc0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.417+0000 7f69368e8640 1 -- 172.21.15.40:0/3652063415 shutdown_connections 2024-03-18T22:58:34.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.417+0000 7f69368e8640 1 --2- 172.21.15.40:0/3652063415 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f69180a2800 0x7f69180a4cc0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.417+0000 7f69368e8640 1 --2- 172.21.15.40:0/3652063415 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69301022f0 0x7f693019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.417+0000 7f69368e8640 1 --2- 172.21.15.40:0/3652063415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6930101930 0x7f6930198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.417+0000 7f69368e8640 1 --2- 172.21.15.40:0/3652063415 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6930100740 0x7f6930197ad0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.417+0000 7f69368e8640 1 -- 172.21.15.40:0/3652063415 >> 172.21.15.40:0/3652063415 conn(0x7f69300fbcd0 msgr2=0x7f6930104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:34.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.417+0000 7f69368e8640 1 -- 172.21.15.40:0/3652063415 shutdown_connections 2024-03-18T22:58:34.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.417+0000 7f69368e8640 1 -- 172.21.15.40:0/3652063415 wait complete. 2024-03-18T22:58:34.428 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T22:58:34.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.534+0000 7f0f406f8640 1 -- 172.21.15.40:0/1281211264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f381033e0 msgr2=0x7f0f38109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:34.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.534+0000 7f0f406f8640 1 --2- 172.21.15.40:0/1281211264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f381033e0 0x7f0f38109c70 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009ca0 tx=0x7f0f3002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.534+0000 7f0f406f8640 1 -- 172.21.15.40:0/1281211264 shutdown_connections 2024-03-18T22:58:34.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.534+0000 7f0f406f8640 1 --2- 172.21.15.40:0/1281211264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f381033e0 0x7f0f38109c70 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.534+0000 7f0f406f8640 1 --2- 172.21.15.40:0/1281211264 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f38102a20 0x7f0f38102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.534+0000 7f0f406f8640 1 --2- 172.21.15.40:0/1281211264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f38101830 0x7f0f38101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.534+0000 7f0f406f8640 1 -- 172.21.15.40:0/1281211264 >> 172.21.15.40:0/1281211264 conn(0x7f0f380fcdc0 msgr2=0x7f0f380ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:34.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.534+0000 7f0f406f8640 1 -- 172.21.15.40:0/1281211264 shutdown_connections 2024-03-18T22:58:34.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.534+0000 7f0f406f8640 1 -- 172.21.15.40:0/1281211264 wait complete. 2024-03-18T22:58:34.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f406f8640 1 Processor -- start 2024-03-18T22:58:34.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f406f8640 1 -- start start 2024-03-18T22:58:34.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f406f8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f38101830 0x7f0f380755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:34.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f406f8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f38102a20 0x7f0f38073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:34.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f406f8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f381033e0 0x7f0f38074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:34.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f406f8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3810d360 con 0x7f0f38101830 2024-03-18T22:58:34.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f406f8640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3810d1e0 con 0x7f0f381033e0 2024-03-18T22:58:34.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f406f8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3810d4e0 con 0x7f0f38102a20 2024-03-18T22:58:34.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f3e494640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f38101830 0x7f0f380755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:34.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f3e494640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f38101830 0x7f0f380755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33990/0 (socket says 172.21.15.40:33990) 2024-03-18T22:58:34.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f3e494640 1 -- 172.21.15.40:0/1337811956 learned_addr learned my addr 172.21.15.40:0/1337811956 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:34.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f3dc93640 1 --2- 172.21.15.40:0/1337811956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f38102a20 0x7f0f38073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f3ec95640 1 --2- 172.21.15.40:0/1337811956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f381033e0 0x7f0f38074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f3e494640 1 -- 172.21.15.40:0/1337811956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f38102a20 msgr2=0x7f0f38073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f3e494640 1 --2- 172.21.15.40:0/1337811956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f38102a20 0x7f0f38073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f3e494640 1 -- 172.21.15.40:0/1337811956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f381033e0 msgr2=0x7f0f38074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f3e494640 1 --2- 172.21.15.40:0/1337811956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f381033e0 0x7f0f38074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f3e494640 1 -- 172.21.15.40:0/1337811956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f380749e0 con 0x7f0f38101830 2024-03-18T22:58:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.535+0000 7f0f3e494640 1 --2- 172.21.15.40:0/1337811956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f38101830 0x7f0f380755d0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400e9c0 tx=0x7f0f3400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.536+0000 7f0f2f7fe640 1 -- 172.21.15.40:0/1337811956 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3400cd80 con 0x7f0f38101830 2024-03-18T22:58:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.536+0000 7f0f2f7fe640 1 -- 172.21.15.40:0/1337811956 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0f340035f0 con 0x7f0f38101830 2024-03-18T22:58:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.536+0000 7f0f2f7fe640 1 -- 172.21.15.40:0/1337811956 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f34010690 con 0x7f0f38101830 2024-03-18T22:58:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.536+0000 7f0f406f8640 1 -- 172.21.15.40:0/1337811956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f381a1b30 con 0x7f0f38101830 2024-03-18T22:58:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.536+0000 7f0f406f8640 1 -- 172.21.15.40:0/1337811956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f381a20d0 con 0x7f0f38101830 2024-03-18T22:58:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.537+0000 7f0f406f8640 1 -- 172.21.15.40:0/1337811956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f38108b50 con 0x7f0f38101830 2024-03-18T22:58:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.538+0000 7f0f2f7fe640 1 -- 172.21.15.40:0/1337811956 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0f340037a0 con 0x7f0f38101830 2024-03-18T22:58:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.539+0000 7f0f2f7fe640 1 --2- 172.21.15.40:0/1337811956 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0f240a2880 0x7f0f240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.539+0000 7f0f3dc93640 1 --2- 172.21.15.40:0/1337811956 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0f240a2880 0x7f0f240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.539+0000 7f0f2f7fe640 1 -- 172.21.15.40:0/1337811956 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0f340cf2d0 con 0x7f0f38101830 2024-03-18T22:58:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.539+0000 7f0f3dc93640 1 --2- 172.21.15.40:0/1337811956 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0f240a2880 0x7f0f240a4d40 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0f38075340 tx=0x7f0f28005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:34.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.542+0000 7f0f2f7fe640 1 -- 172.21.15.40:0/1337811956 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0f3409bf80 con 0x7f0f38101830 2024-03-18T22:58:34.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.726+0000 7f0f406f8640 1 -- 172.21.15.40:0/1337811956 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0f3802d380 con 0x7f0f38101830 2024-03-18T22:58:34.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.726+0000 7f0f2f7fe640 1 -- 172.21.15.40:0/1337811956 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f0f3409b880 con 0x7f0f38101830 2024-03-18T22:58:34.727 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:58:34.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.728+0000 7f0f406f8640 1 -- 172.21.15.40:0/1337811956 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0f240a2880 msgr2=0x7f0f240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:34.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.728+0000 7f0f406f8640 1 --2- 172.21.15.40:0/1337811956 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0f240a2880 0x7f0f240a4d40 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0f38075340 tx=0x7f0f28005f90 comp rx=0 tx=0).stop 2024-03-18T22:58:34.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.728+0000 7f0f406f8640 1 -- 172.21.15.40:0/1337811956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f38101830 msgr2=0x7f0f380755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:34.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.728+0000 7f0f406f8640 1 --2- 172.21.15.40:0/1337811956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f38101830 0x7f0f380755d0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400e9c0 tx=0x7f0f3400ee80 comp rx=0 tx=0).stop 2024-03-18T22:58:34.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.728+0000 7f0f406f8640 1 -- 172.21.15.40:0/1337811956 shutdown_connections 2024-03-18T22:58:34.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.728+0000 7f0f406f8640 1 --2- 172.21.15.40:0/1337811956 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f0f240a2880 0x7f0f240a4d40 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.728+0000 7f0f406f8640 1 --2- 172.21.15.40:0/1337811956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f381033e0 0x7f0f38074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.728+0000 7f0f406f8640 1 --2- 172.21.15.40:0/1337811956 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0f38102a20 0x7f0f38073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.728+0000 7f0f406f8640 1 --2- 172.21.15.40:0/1337811956 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f38101830 0x7f0f380755d0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.728+0000 7f0f406f8640 1 -- 172.21.15.40:0/1337811956 >> 172.21.15.40:0/1337811956 conn(0x7f0f380fcdc0 msgr2=0x7f0f38105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:34.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.728+0000 7f0f406f8640 1 -- 172.21.15.40:0/1337811956 shutdown_connections 2024-03-18T22:58:34.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.728+0000 7f0f406f8640 1 -- 172.21.15.40:0/1337811956 wait complete. 2024-03-18T22:58:34.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:34 smithi040 ceph-mon[62643]: pgmap v828: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:58:34.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:34 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1322831528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:58:34.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:34 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2923478592' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:58:34.739 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:34 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3652063415' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:58:34.740 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-18T22:58:34.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:34 smithi113 ceph-mon[66775]: pgmap v828: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:58:34.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:34 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1322831528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:58:34.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:34 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2923478592' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:58:34.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:34 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3652063415' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:58:34.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.842+0000 7ff0a1344640 1 -- 172.21.15.40:0/965989020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff09c101950 msgr2=0x7ff09c101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:34.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.842+0000 7ff0a1344640 1 --2- 172.21.15.40:0/965989020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff09c101950 0x7ff09c101dd0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7ff084009900 tx=0x7ff08402bc30 comp rx=0 tx=0).stop 2024-03-18T22:58:34.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.842+0000 7ff0a1344640 1 -- 172.21.15.40:0/965989020 shutdown_connections 2024-03-18T22:58:34.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.842+0000 7ff0a1344640 1 --2- 172.21.15.40:0/965989020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff09c102310 0x7ff09c108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.842+0000 7ff0a1344640 1 --2- 172.21.15.40:0/965989020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff09c101950 0x7ff09c101dd0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.842+0000 7ff0a1344640 1 --2- 172.21.15.40:0/965989020 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff09c100760 0x7ff09c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.842+0000 7ff0a1344640 1 -- 172.21.15.40:0/965989020 >> 172.21.15.40:0/965989020 conn(0x7ff09c0fbcd0 msgr2=0x7ff09c0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:34.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.842+0000 7ff0a1344640 1 -- 172.21.15.40:0/965989020 shutdown_connections 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.842+0000 7ff0a1344640 1 -- 172.21.15.40:0/965989020 wait complete. 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.842+0000 7ff0a1344640 1 Processor -- start 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.842+0000 7ff0a1344640 1 -- start start 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff0a1344640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff09c100760 0x7ff09c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff0a1344640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff09c101950 0x7ff09c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff0a1344640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff09c102310 0x7ff09c19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff0a1344640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff09c10c270 con 0x7ff09c102310 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff0a1344640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff09c10c0f0 con 0x7ff09c101950 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff0a1344640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff09c10c3f0 con 0x7ff09c100760 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff09b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff09c102310 0x7ff09c19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff09b7fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff09c102310 0x7ff09c19f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34010/0 (socket says 172.21.15.40:34010) 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff09b7fe640 1 -- 172.21.15.40:0/2539944351 learned_addr learned my addr 172.21.15.40:0/2539944351 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff09affd640 1 --2- 172.21.15.40:0/2539944351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff09c100760 0x7ff09c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff09a7fc640 1 --2- 172.21.15.40:0/2539944351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff09c101950 0x7ff09c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff09b7fe640 1 -- 172.21.15.40:0/2539944351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff09c100760 msgr2=0x7ff09c197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:34.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff09b7fe640 1 --2- 172.21.15.40:0/2539944351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff09c100760 0x7ff09c197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff09b7fe640 1 -- 172.21.15.40:0/2539944351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff09c101950 msgr2=0x7ff09c198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:34.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff09b7fe640 1 --2- 172.21.15.40:0/2539944351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff09c101950 0x7ff09c198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:34.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff09b7fe640 1 -- 172.21.15.40:0/2539944351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff09c19f6e0 con 0x7ff09c102310 2024-03-18T22:58:34.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff09affd640 1 --2- 172.21.15.40:0/2539944351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff09c100760 0x7ff09c197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:58:34.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff09b7fe640 1 --2- 172.21.15.40:0/2539944351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff09c102310 0x7ff09c19f070 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff08c00eb10 tx=0x7ff08c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:34.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff093fff640 1 -- 172.21.15.40:0/2539944351 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c004b20 con 0x7ff09c102310 2024-03-18T22:58:34.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff0a1344640 1 -- 172.21.15.40:0/2539944351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff09c19f9d0 con 0x7ff09c102310 2024-03-18T22:58:34.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff093fff640 1 -- 172.21.15.40:0/2539944351 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff08c004cc0 con 0x7ff09c102310 2024-03-18T22:58:34.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff093fff640 1 -- 172.21.15.40:0/2539944351 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c010690 con 0x7ff09c102310 2024-03-18T22:58:34.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.843+0000 7ff0a1344640 1 -- 172.21.15.40:0/2539944351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff09c19fef0 con 0x7ff09c102310 2024-03-18T22:58:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.844+0000 7ff0a1344640 1 -- 172.21.15.40:0/2539944351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff09c1a0180 con 0x7ff09c102310 2024-03-18T22:58:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.845+0000 7ff093fff640 1 -- 172.21.15.40:0/2539944351 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff08c005020 con 0x7ff09c102310 2024-03-18T22:58:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.846+0000 7ff093fff640 1 --2- 172.21.15.40:0/2539944351 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff0880a2800 0x7ff0880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:58:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.846+0000 7ff09affd640 1 --2- 172.21.15.40:0/2539944351 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff0880a2800 0x7ff0880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:58:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.846+0000 7ff093fff640 1 -- 172.21.15.40:0/2539944351 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff08c0cf440 con 0x7ff09c102310 2024-03-18T22:58:34.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.846+0000 7ff09affd640 1 --2- 172.21.15.40:0/2539944351 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff0880a2800 0x7ff0880a4cc0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff09c1017b0 tx=0x7ff094005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:58:34.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:34.850+0000 7ff093fff640 1 -- 172.21.15.40:0/2539944351 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff08c09c090 con 0x7ff09c102310 2024-03-18T22:58:34.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:34 smithi103 ceph-mon[66377]: pgmap v828: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:58:34.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:34 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1322831528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:58:34.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:34 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2923478592' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:58:34.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:34 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3652063415' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:58:35.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:35.084+0000 7ff0a1344640 1 -- 172.21.15.40:0/2539944351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7ff09c1a03a0 con 0x7ff09c102310 2024-03-18T22:58:35.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:35.099+0000 7ff093fff640 1 -- 172.21.15.40:0/2539944351 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) ==== 81+0+739828 (secure 0 0 0) 0x7ff08c09b990 con 0x7ff09c102310 2024-03-18T22:58:35.109 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:58:35.110 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:58:35.110 INFO:teuthology.orchestra.run.smithi040.stdout: "epoch": 18, 2024-03-18T22:58:35.110 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.110 INFO:teuthology.orchestra.run.smithi040.stdout: "active_gid": 14148, 2024-03-18T22:58:35.110 INFO:teuthology.orchestra.run.smithi040.stdout: "active_name": "x", 2024-03-18T22:58:35.110 INFO:teuthology.orchestra.run.smithi040.stdout: "active_addrs": { 2024-03-18T22:58:35.110 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-03-18T22:58:35.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.110 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-03-18T22:58:35.110 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6800", 2024-03-18T22:58:35.110 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3109254132 2024-03-18T22:58:35.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.111 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-18T22:58:35.111 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:6801", 2024-03-18T22:58:35.111 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 3109254132 2024-03-18T22:58:35.111 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.111 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-18T22:58:35.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.111 INFO:teuthology.orchestra.run.smithi040.stdout: "active_addr": "172.21.15.40:6801/3109254132", 2024-03-18T22:58:35.111 INFO:teuthology.orchestra.run.smithi040.stdout: "active_change": "2024-03-18T22:33:42.721723+0000", 2024-03-18T22:58:35.111 INFO:teuthology.orchestra.run.smithi040.stdout: "active_mgr_features": 4540701547738038271, 2024-03-18T22:58:35.111 INFO:teuthology.orchestra.run.smithi040.stdout: "available": true, 2024-03-18T22:58:35.111 INFO:teuthology.orchestra.run.smithi040.stdout: "standbys": [ 2024-03-18T22:58:35.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.111 INFO:teuthology.orchestra.run.smithi040.stdout: "gid": 24109, 2024-03-18T22:58:35.111 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "z", 2024-03-18T22:58:35.112 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_features": 4540701547738038271, 2024-03-18T22:58:35.112 INFO:teuthology.orchestra.run.smithi040.stdout: "available_modules": [ 2024-03-18T22:58:35.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.112 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "alerts", 2024-03-18T22:58:35.112 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.112 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.112 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.112 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-03-18T22:58:35.112 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-03-18T22:58:35.112 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.112 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.112 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.112 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-03-18T22:58:35.112 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.113 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.114 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.114 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.114 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.114 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.114 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.114 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.114 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.114 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.114 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.114 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.114 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.114 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.115 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.116 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.116 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.116 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.116 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.116 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.116 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.116 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.116 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.116 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.116 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.116 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.116 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.116 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.116 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.116 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.117 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.117 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.117 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.117 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.117 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.117 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.117 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.117 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.117 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.117 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.117 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.117 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.117 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.117 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.118 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.118 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_destination": { 2024-03-18T22:58:35.118 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_destination", 2024-03-18T22:58:35.118 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.118 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.118 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.118 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.118 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.118 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.118 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.118 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email address to send alerts to", 2024-03-18T22:58:35.118 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.118 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.119 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.119 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_from_name": { 2024-03-18T22:58:35.119 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_from_name", 2024-03-18T22:58:35.119 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.119 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.119 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.119 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "Ceph", 2024-03-18T22:58:35.119 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.119 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.119 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.119 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email From: name", 2024-03-18T22:58:35.119 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.119 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.119 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.120 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_host": { 2024-03-18T22:58:35.120 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_host", 2024-03-18T22:58:35.120 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.120 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.120 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.120 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.120 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.120 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.120 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.120 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP server", 2024-03-18T22:58:35.120 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.120 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.120 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.121 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_password": { 2024-03-18T22:58:35.121 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_password", 2024-03-18T22:58:35.121 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.121 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.121 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.121 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.121 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.121 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.121 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.121 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Password to authenticate with", 2024-03-18T22:58:35.121 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.121 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.121 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.122 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_port": { 2024-03-18T22:58:35.122 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_port", 2024-03-18T22:58:35.122 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.122 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.122 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.122 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "465", 2024-03-18T22:58:35.122 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.122 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.122 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.122 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP port", 2024-03-18T22:58:35.122 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.122 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.122 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.123 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_sender": { 2024-03-18T22:58:35.123 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_sender", 2024-03-18T22:58:35.123 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.123 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.123 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.123 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.123 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.123 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.123 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.123 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP envelope sender", 2024-03-18T22:58:35.123 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.123 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.123 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.124 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_ssl": { 2024-03-18T22:58:35.124 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_ssl", 2024-03-18T22:58:35.124 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.124 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.124 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.124 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.124 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.124 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.124 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.124 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T22:58:35.124 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.124 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.124 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.124 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_user": { 2024-03-18T22:58:35.125 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_user", 2024-03-18T22:58:35.125 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.125 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.125 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.125 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.125 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.125 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.125 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.125 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "User to authenticate as", 2024-03-18T22:58:35.125 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.125 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.125 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.125 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.125 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.126 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "balancer", 2024-03-18T22:58:35.126 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.126 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.126 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.126 INFO:teuthology.orchestra.run.smithi040.stdout: "active": { 2024-03-18T22:58:35.126 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "active", 2024-03-18T22:58:35.126 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.126 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.126 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.126 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.126 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.126 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.127 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.127 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T22:58:35.127 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.127 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.127 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.127 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_time": { 2024-03-18T22:58:35.127 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_time", 2024-03-18T22:58:35.127 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.127 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.127 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.127 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0000", 2024-03-18T22:58:35.127 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.128 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.128 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.128 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T22:58:35.128 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T22:58:35.128 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.128 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.128 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_weekday": { 2024-03-18T22:58:35.128 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_weekday", 2024-03-18T22:58:35.128 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-03-18T22:58:35.128 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.128 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.128 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-03-18T22:58:35.128 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-03-18T22:58:35.129 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-03-18T22:58:35.129 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.129 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T22:58:35.129 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T22:58:35.129 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.129 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.129 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_max_iterations": { 2024-03-18T22:58:35.129 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_max_iterations", 2024-03-18T22:58:35.129 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-03-18T22:58:35.129 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.129 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.129 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "25", 2024-03-18T22:58:35.130 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-03-18T22:58:35.130 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "250", 2024-03-18T22:58:35.130 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.130 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T22:58:35.130 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.130 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.130 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.130 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_metrics": { 2024-03-18T22:58:35.130 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_metrics", 2024-03-18T22:58:35.130 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.130 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.130 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.130 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T22:58:35.131 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.131 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.131 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.131 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T22:58:35.131 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T22:58:35.131 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.131 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.131 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_step": { 2024-03-18T22:58:35.131 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_step", 2024-03-18T22:58:35.131 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.131 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.131 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.131 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.5", 2024-03-18T22:58:35.131 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0.001", 2024-03-18T22:58:35.132 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "0.999", 2024-03-18T22:58:35.132 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.132 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "aggressiveness of optimization", 2024-03-18T22:58:35.132 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T22:58:35.132 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.132 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.132 INFO:teuthology.orchestra.run.smithi040.stdout: "end_time": { 2024-03-18T22:58:35.132 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_time", 2024-03-18T22:58:35.132 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.132 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.132 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.132 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2359", 2024-03-18T22:58:35.132 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.132 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.133 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.133 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T22:58:35.133 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T22:58:35.133 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.133 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.133 INFO:teuthology.orchestra.run.smithi040.stdout: "end_weekday": { 2024-03-18T22:58:35.133 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_weekday", 2024-03-18T22:58:35.133 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-03-18T22:58:35.133 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.133 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.133 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-03-18T22:58:35.133 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-03-18T22:58:35.133 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-03-18T22:58:35.134 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.134 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T22:58:35.134 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T22:58:35.134 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.134 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.134 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.134 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.134 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.134 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.134 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.134 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.134 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.134 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.135 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.135 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.135 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.135 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.135 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.135 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.135 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.135 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.135 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.135 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.135 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.135 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.135 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.135 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.136 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.136 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.136 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.136 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.136 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.136 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.136 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.136 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.136 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.136 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.136 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.136 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.137 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.137 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.137 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.137 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.137 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.137 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.137 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.137 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.137 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.137 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.137 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.137 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.137 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.137 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.138 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.138 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.138 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.138 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.138 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.138 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.138 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.138 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.138 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.138 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.138 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.138 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.138 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.139 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.139 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.139 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.139 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.139 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.139 INFO:teuthology.orchestra.run.smithi040.stdout: "min_score": { 2024-03-18T22:58:35.139 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_score", 2024-03-18T22:58:35.139 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.139 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.139 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.139 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-03-18T22:58:35.139 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.139 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.140 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.140 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T22:58:35.140 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.140 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.140 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.140 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-03-18T22:58:35.140 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-03-18T22:58:35.140 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.140 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.140 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.140 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "upmap", 2024-03-18T22:58:35.140 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.140 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.141 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.141 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-compat", 2024-03-18T22:58:35.141 INFO:teuthology.orchestra.run.smithi040.stdout: "none", 2024-03-18T22:58:35.141 INFO:teuthology.orchestra.run.smithi040.stdout: "read", 2024-03-18T22:58:35.141 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap", 2024-03-18T22:58:35.141 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap-read" 2024-03-18T22:58:35.141 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.141 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Balancer mode", 2024-03-18T22:58:35.141 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.141 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.141 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.141 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_ids": { 2024-03-18T22:58:35.141 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_ids", 2024-03-18T22:58:35.141 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.142 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.142 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.142 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.142 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.142 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.142 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.142 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T22:58:35.142 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.142 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.142 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.142 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-03-18T22:58:35.142 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-03-18T22:58:35.143 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.143 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.143 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.143 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-03-18T22:58:35.143 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.143 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.143 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.143 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T22:58:35.143 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.143 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.143 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.143 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_deviation": { 2024-03-18T22:58:35.143 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_deviation", 2024-03-18T22:58:35.143 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.144 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.144 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.144 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-03-18T22:58:35.144 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-03-18T22:58:35.144 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.144 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.144 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T22:58:35.144 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T22:58:35.144 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.144 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.144 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_optimizations": { 2024-03-18T22:58:35.144 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_optimizations", 2024-03-18T22:58:35.144 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-03-18T22:58:35.145 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.145 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.145 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.145 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.145 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.145 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.145 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T22:58:35.145 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.145 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.145 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.145 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.145 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.146 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm", 2024-03-18T22:58:35.146 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.146 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.146 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.146 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_down_multiplier": { 2024-03-18T22:58:35.146 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_down_multiplier", 2024-03-18T22:58:35.146 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.146 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.146 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.146 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-03-18T22:58:35.146 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.146 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.146 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.146 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T22:58:35.147 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.147 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.147 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.147 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_refresh_rate": { 2024-03-18T22:58:35.147 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_refresh_rate", 2024-03-18T22:58:35.147 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.147 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.147 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.147 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "20", 2024-03-18T22:58:35.147 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.147 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.147 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.147 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T22:58:35.147 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.148 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.148 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.148 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_starting_port": { 2024-03-18T22:58:35.148 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_starting_port", 2024-03-18T22:58:35.148 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.148 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.148 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.148 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4721", 2024-03-18T22:58:35.148 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.148 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.148 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.148 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T22:58:35.148 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.149 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.149 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.149 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_ptrace": { 2024-03-18T22:58:35.149 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_ptrace", 2024-03-18T22:58:35.149 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.149 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.149 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.149 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.149 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.149 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.149 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.149 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T22:58:35.150 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T22:58:35.150 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.150 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.150 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_interval": { 2024-03-18T22:58:35.150 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_interval", 2024-03-18T22:58:35.150 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.150 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.150 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.150 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-03-18T22:58:35.150 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.150 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.150 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.150 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T22:58:35.151 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.151 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.151 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.151 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_memory_target_ratio": { 2024-03-18T22:58:35.151 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T22:58:35.151 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.151 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.151 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.151 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.7", 2024-03-18T22:58:35.151 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.151 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.151 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.151 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T22:58:35.151 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.152 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.152 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.152 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm_log_destination": { 2024-03-18T22:58:35.152 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm_log_destination", 2024-03-18T22:58:35.152 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.152 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.152 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.152 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.152 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.152 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.152 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.152 INFO:teuthology.orchestra.run.smithi040.stdout: "file", 2024-03-18T22:58:35.153 INFO:teuthology.orchestra.run.smithi040.stdout: "file,syslog", 2024-03-18T22:58:35.153 INFO:teuthology.orchestra.run.smithi040.stdout: "syslog" 2024-03-18T22:58:35.153 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.153 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T22:58:35.153 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.153 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.153 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.153 INFO:teuthology.orchestra.run.smithi040.stdout: "cgroups_split": { 2024-03-18T22:58:35.153 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cgroups_split", 2024-03-18T22:58:35.153 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.153 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.153 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.153 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.153 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.154 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.154 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.154 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T22:58:35.154 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.154 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.154 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.154 INFO:teuthology.orchestra.run.smithi040.stdout: "config_checks_enabled": { 2024-03-18T22:58:35.154 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_checks_enabled", 2024-03-18T22:58:35.154 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.154 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.154 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.154 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.154 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: "config_dashboard": { 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_dashboard", 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.155 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.156 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T22:58:35.156 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.156 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.156 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.156 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_alertmanager": { 2024-03-18T22:58:35.156 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_alertmanager", 2024-03-18T22:58:35.156 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.156 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.156 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.156 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T22:58:35.156 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.156 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.156 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.156 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-03-18T22:58:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_base": { 2024-03-18T22:58:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_base", 2024-03-18T22:58:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T22:58:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Container image name, without the tag", 2024-03-18T22:58:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.158 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.158 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.158 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_elasticsearch": { 2024-03-18T22:58:35.158 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_elasticsearch", 2024-03-18T22:58:35.158 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.158 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.158 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.158 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T22:58:35.158 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.158 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.158 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.158 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "elasticsearch container image", 2024-03-18T22:58:35.159 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.159 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.159 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.159 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_grafana": { 2024-03-18T22:58:35.159 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_grafana", 2024-03-18T22:58:35.159 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.159 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.159 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.159 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T22:58:35.159 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.159 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.159 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.159 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-03-18T22:58:35.159 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.160 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.160 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.160 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_haproxy": { 2024-03-18T22:58:35.160 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_haproxy", 2024-03-18T22:58:35.160 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.160 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.160 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.160 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T22:58:35.160 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.160 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.160 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.160 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "HAproxy container image", 2024-03-18T22:58:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_agent": { 2024-03-18T22:58:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_agent", 2024-03-18T22:58:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T22:58:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger agent container image", 2024-03-18T22:58:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_collector": { 2024-03-18T22:58:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_collector", 2024-03-18T22:58:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T22:58:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger collector container image", 2024-03-18T22:58:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_query": { 2024-03-18T22:58:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_query", 2024-03-18T22:58:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T22:58:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger query container image", 2024-03-18T22:58:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_keepalived": { 2024-03-18T22:58:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_keepalived", 2024-03-18T22:58:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T22:58:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Keepalived container image", 2024-03-18T22:58:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_loki": { 2024-03-18T22:58:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_loki", 2024-03-18T22:58:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T22:58:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Loki container image", 2024-03-18T22:58:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_node_exporter": { 2024-03-18T22:58:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_node_exporter", 2024-03-18T22:58:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T22:58:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-03-18T22:58:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_nvmeof": { 2024-03-18T22:58:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_nvmeof", 2024-03-18T22:58:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T22:58:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Nvme-of container image", 2024-03-18T22:58:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_prometheus": { 2024-03-18T22:58:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_prometheus", 2024-03-18T22:58:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T22:58:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-03-18T22:58:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_promtail": { 2024-03-18T22:58:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_promtail", 2024-03-18T22:58:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T22:58:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Promtail container image", 2024-03-18T22:58:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_snmp_gateway": { 2024-03-18T22:58:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_snmp_gateway", 2024-03-18T22:58:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T22:58:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SNMP Gateway container image", 2024-03-18T22:58:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "container_init": { 2024-03-18T22:58:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_init", 2024-03-18T22:58:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T22:58:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "daemon_cache_timeout": { 2024-03-18T22:58:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "daemon_cache_timeout", 2024-03-18T22:58:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-03-18T22:58:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T22:58:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "default_cephadm_command_timeout": { 2024-03-18T22:58:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T22:58:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "900", 2024-03-18T22:58:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T22:58:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "default_registry": { 2024-03-18T22:58:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_registry", 2024-03-18T22:58:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io", 2024-03-18T22:58:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T22:58:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "device_cache_timeout": { 2024-03-18T22:58:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_cache_timeout", 2024-03-18T22:58:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1800", 2024-03-18T22:58:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache device inventory", 2024-03-18T22:58:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "device_enhanced_scan": { 2024-03-18T22:58:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_enhanced_scan", 2024-03-18T22:58:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T22:58:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "facts_cache_timeout": { 2024-03-18T22:58:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "facts_cache_timeout", 2024-03-18T22:58:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-03-18T22:58:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache host facts data", 2024-03-18T22:58:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "host_check_interval": { 2024-03-18T22:58:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "host_check_interval", 2024-03-18T22:58:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-03-18T22:58:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to perform a host check", 2024-03-18T22:58:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "hw_monitoring": { 2024-03-18T22:58:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hw_monitoring", 2024-03-18T22:58:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T22:58:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "inventory_list_all": { 2024-03-18T22:58:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "inventory_list_all", 2024-03-18T22:58:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T22:58:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "log_refresh_metadata": { 2024-03-18T22:58:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_refresh_metadata", 2024-03-18T22:58:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T22:58:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T22:58:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T22:58:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T22:58:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "*", 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T22:58:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "max_count_per_host": { 2024-03-18T22:58:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_count_per_host", 2024-03-18T22:58:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of daemons per service per host", 2024-03-18T22:58:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd_draining_count": { 2024-03-18T22:58:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_osd_draining_count", 2024-03-18T22:58:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T22:58:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "migration_current": { 2024-03-18T22:58:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "migration_current", 2024-03-18T22:58:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "internal - do not modify", 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "root", 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm-package", 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "root" 2024-03-18T22:58:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "oob_default_addr": { 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "oob_default_addr", 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "169.254.1.1", 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T22:58:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "prometheus_alerts_path": { 2024-03-18T22:58:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus_alerts_path", 2024-03-18T22:58:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T22:58:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T22:58:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_insecure": { 2024-03-18T22:58:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_insecure", 2024-03-18T22:58:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T22:58:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_password": { 2024-03-18T22:58:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_password", 2024-03-18T22:58:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T22:58:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_url": { 2024-03-18T22:58:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_url", 2024-03-18T22:58:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T22:58:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_username": { 2024-03-18T22:58:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_username", 2024-03-18T22:58:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T22:58:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "secure_monitoring_stack": { 2024-03-18T22:58:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "secure_monitoring_stack", 2024-03-18T22:58:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T22:58:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "service_discovery_port": { 2024-03-18T22:58:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "service_discovery_port", 2024-03-18T22:58:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8765", 2024-03-18T22:58:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "cephadm service discovery port", 2024-03-18T22:58:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "ssh_config_file": { 2024-03-18T22:58:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssh_config_file", 2024-03-18T22:58:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T22:58:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.201 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.201 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.201 INFO:teuthology.orchestra.run.smithi040.stdout: "use_agent": { 2024-03-18T22:58:35.201 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_agent", 2024-03-18T22:58:35.201 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.201 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.201 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.201 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.201 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.201 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.201 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.201 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T22:58:35.201 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "use_repo_digest": { 2024-03-18T22:58:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_repo_digest", 2024-03-18T22:58:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T22:58:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_failed_host_check": { 2024-03-18T22:58:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_failed_host_check", 2024-03-18T22:58:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T22:58:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_daemons": { 2024-03-18T22:58:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_daemons", 2024-03-18T22:58:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T22:58:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_hosts": { 2024-03-18T22:58:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_hosts", 2024-03-18T22:58:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T22:58:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crash", 2024-03-18T22:58:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "retain_interval": { 2024-03-18T22:58:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retain_interval", 2024-03-18T22:58:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "31536000", 2024-03-18T22:58:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T22:58:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_recent_interval": { 2024-03-18T22:58:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_recent_interval", 2024-03-18T22:58:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1209600", 2024-03-18T22:58:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T22:58:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dashboard", 2024-03-18T22:58:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T22:58:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T22:58:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T22:58:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T22:58:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_ENABLED": { 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T22:58:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T22:58:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T22:58:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T22:58:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T22:58:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T22:58:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T22:58:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T22:58:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T22:58:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T22:58:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T22:58:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T22:58:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T22:58:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T22:58:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T22:58:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T22:58:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T22:58:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T22:58:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T22:58:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T22:58:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-03-18T22:58:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T22:58:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T22:58:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_URL": { 2024-03-18T22:58:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_URL", 2024-03-18T22:58:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T22:58:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T22:58:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-03-18T22:58:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T22:58:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T22:58:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T22:58:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T22:58:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T22:58:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T22:58:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T22:58:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T22:58:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T22:58:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T22:58:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "{}", 2024-03-18T22:58:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T22:58:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T22:58:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T22:58:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T22:58:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T22:58:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T22:58:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T22:58:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T22:58:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T22:58:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T22:58:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T22:58:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T22:58:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T22:58:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T22:58:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T22:58:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T22:58:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T22:58:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T22:58:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T22:58:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T22:58:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T22:58:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8", 2024-03-18T22:58:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T22:58:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T22:58:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "45", 2024-03-18T22:58:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T22:58:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T22:58:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T22:58:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T22:58:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-03-18T22:58:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T22:58:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T22:58:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T22:58:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T22:58:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T22:58:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T22:58:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T22:58:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T22:58:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "cross_origin_url": { 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cross_origin_url", 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "crt_file": { 2024-03-18T22:58:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crt_file", 2024-03-18T22:58:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "debug": { 2024-03-18T22:58:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "debug", 2024-03-18T22:58:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable/disable debug options", 2024-03-18T22:58:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "jwt_token_ttl": { 2024-03-18T22:58:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "jwt_token_ttl", 2024-03-18T22:58:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "28800", 2024-03-18T22:58:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-03-18T22:58:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-03-18T22:58:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.263 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.263 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.263 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.263 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.263 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.263 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.263 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.263 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.263 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.263 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.263 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.263 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.263 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.263 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.264 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.264 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.264 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.264 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.264 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.264 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.264 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.264 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.264 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.264 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.264 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.264 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.264 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.264 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.264 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "motd": { 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "motd", 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.265 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.266 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.266 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.266 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "The message of the day", 2024-03-18T22:58:35.266 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.266 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.266 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.266 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.266 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect_resolve_ip_addr": { 2024-03-18T22:58:35.266 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T22:58:35.266 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.266 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.266 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.266 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.266 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.267 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.268 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.268 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.268 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.268 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.268 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-03-18T22:58:35.268 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-03-18T22:58:35.268 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.268 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.268 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.268 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8080", 2024-03-18T22:58:35.268 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.268 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.268 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.269 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.269 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.269 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.269 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.269 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-03-18T22:58:35.269 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-03-18T22:58:35.269 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.269 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.269 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.269 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.269 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.269 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.269 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.269 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl_server_port": { 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl_server_port", 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8443", 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.270 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.271 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.271 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-03-18T22:58:35.271 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-03-18T22:58:35.271 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.271 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.271 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.271 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "redirect", 2024-03-18T22:58:35.271 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.271 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.271 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.271 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.271 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect" 2024-03-18T22:58:35.271 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.272 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.273 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.273 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.273 INFO:teuthology.orchestra.run.smithi040.stdout: "url_prefix": { 2024-03-18T22:58:35.273 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url_prefix", 2024-03-18T22:58:35.273 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.273 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.273 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.273 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.273 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.273 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.273 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.273 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.273 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.273 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.274 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.274 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.274 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.274 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.274 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.274 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "devicehealth", 2024-03-18T22:58:35.274 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.274 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.274 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.274 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_monitoring": { 2024-03-18T22:58:35.274 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_monitoring", 2024-03-18T22:58:35.274 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.274 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.274 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.275 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.275 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.275 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.275 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.275 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "monitor device health metrics", 2024-03-18T22:58:35.275 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.275 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.275 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.275 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.275 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.275 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.275 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.275 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.275 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.275 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.276 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.276 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.276 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.276 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.276 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.276 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.276 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.276 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.276 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.276 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.276 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.276 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.276 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.276 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.277 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.277 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.277 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.277 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.277 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.277 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.277 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.277 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.277 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.277 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.277 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.277 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.277 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.277 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.278 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.278 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.278 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.278 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.278 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.278 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.278 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.278 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.278 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.278 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.278 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.278 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.278 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.278 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.279 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.280 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.280 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.280 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.280 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.280 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.280 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.280 INFO:teuthology.orchestra.run.smithi040.stdout: "mark_out_threshold": { 2024-03-18T22:58:35.280 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mark_out_threshold", 2024-03-18T22:58:35.280 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.280 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.280 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.280 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2419200", 2024-03-18T22:58:35.280 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.280 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.280 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.281 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T22:58:35.281 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.281 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.281 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.281 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": { 2024-03-18T22:58:35.281 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_name", 2024-03-18T22:58:35.281 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.281 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.281 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.281 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "device_health_metrics", 2024-03-18T22:58:35.281 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.281 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.281 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.281 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T22:58:35.282 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.282 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.282 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.282 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.282 INFO:teuthology.orchestra.run.smithi040.stdout: "retention_period": { 2024-03-18T22:58:35.282 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retention_period", 2024-03-18T22:58:35.282 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.282 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.282 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.282 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15552000", 2024-03-18T22:58:35.282 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.282 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.282 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.282 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain device health metrics", 2024-03-18T22:58:35.282 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.283 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.283 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.283 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.283 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_frequency": { 2024-03-18T22:58:35.283 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_frequency", 2024-03-18T22:58:35.283 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.283 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.283 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.283 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-03-18T22:58:35.283 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.283 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.283 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.283 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T22:58:35.283 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.284 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.284 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.284 INFO:teuthology.orchestra.run.smithi040.stdout: "self_heal": { 2024-03-18T22:58:35.284 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "self_heal", 2024-03-18T22:58:35.284 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.284 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.284 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.284 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.284 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.284 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.284 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.284 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T22:58:35.284 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.285 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.285 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.285 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-03-18T22:58:35.285 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-03-18T22:58:35.285 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.285 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.285 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.285 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-03-18T22:58:35.285 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.285 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.285 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.285 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T22:58:35.285 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.285 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.286 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.286 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_threshold": { 2024-03-18T22:58:35.286 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_threshold", 2024-03-18T22:58:35.286 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.286 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.286 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.286 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7257600", 2024-03-18T22:58:35.286 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.286 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.286 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.286 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T22:58:35.286 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.286 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.287 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.287 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.287 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.287 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.287 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.287 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "diskprediction_local", 2024-03-18T22:58:35.287 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.287 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.287 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.287 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.287 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.287 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.287 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.287 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.288 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.288 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.288 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.288 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.288 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.288 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.288 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.288 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.288 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.288 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.288 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.288 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.288 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.288 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.288 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.289 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.290 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.290 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.290 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.290 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.290 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.290 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.290 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.290 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.290 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.290 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.290 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.290 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.290 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.290 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.291 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.292 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.292 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.292 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.292 INFO:teuthology.orchestra.run.smithi040.stdout: "predict_interval": { 2024-03-18T22:58:35.292 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predict_interval", 2024-03-18T22:58:35.292 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.292 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.292 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.292 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-03-18T22:58:35.292 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.292 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.292 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.292 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.292 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: "predictor_model": { 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predictor_model", 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "prophetstor", 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.293 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.294 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-03-18T22:58:35.294 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-03-18T22:58:35.294 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.294 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.294 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.294 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-03-18T22:58:35.294 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.294 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.294 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.294 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.294 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.294 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.295 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.295 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.295 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.295 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.295 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "influx", 2024-03-18T22:58:35.295 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": false, 2024-03-18T22:58:35.295 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "influxdb python module not found", 2024-03-18T22:58:35.295 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.295 INFO:teuthology.orchestra.run.smithi040.stdout: "batch_size": { 2024-03-18T22:58:35.295 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "batch_size", 2024-03-18T22:58:35.295 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.295 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.295 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5000", 2024-03-18T22:58:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T22:58:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: "database": { 2024-03-18T22:58:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "database", 2024-03-18T22:58:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.296 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "ceph", 2024-03-18T22:58:35.297 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.297 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.297 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.297 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T22:58:35.297 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.297 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.297 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.297 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": { 2024-03-18T22:58:35.297 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hostname", 2024-03-18T22:58:35.297 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.297 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.297 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.297 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.297 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.298 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.298 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.298 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server hostname", 2024-03-18T22:58:35.298 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.298 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.298 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.298 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-03-18T22:58:35.298 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-03-18T22:58:35.298 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.298 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.298 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.298 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "30", 2024-03-18T22:58:35.298 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "5", 2024-03-18T22:58:35.298 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T22:58:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.299 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.300 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.300 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.300 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.300 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.300 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.300 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.300 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.300 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.300 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.300 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.300 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.300 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.300 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.300 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.301 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.301 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.301 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.301 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.301 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.301 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.301 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.301 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.301 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.301 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.301 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.301 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.301 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.301 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.302 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.302 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.302 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.302 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.302 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.302 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.302 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.302 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.302 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.302 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.302 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.302 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.302 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.302 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.303 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.303 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.303 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.303 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.303 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.303 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.303 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.303 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.303 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.303 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.303 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.303 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.303 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.303 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.304 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.304 INFO:teuthology.orchestra.run.smithi040.stdout: "password": { 2024-03-18T22:58:35.304 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "password", 2024-03-18T22:58:35.304 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.304 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.304 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.304 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.304 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.304 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.304 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.304 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "password of InfluxDB server user", 2024-03-18T22:58:35.304 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.304 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.304 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.305 INFO:teuthology.orchestra.run.smithi040.stdout: "port": { 2024-03-18T22:58:35.305 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "port", 2024-03-18T22:58:35.305 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.305 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.305 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.305 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8086", 2024-03-18T22:58:35.305 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.305 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.305 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.305 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server port", 2024-03-18T22:58:35.305 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.305 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.305 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.306 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-03-18T22:58:35.306 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-03-18T22:58:35.306 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.306 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.306 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.306 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "false", 2024-03-18T22:58:35.306 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.306 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.306 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.306 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T22:58:35.306 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.306 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.306 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.307 INFO:teuthology.orchestra.run.smithi040.stdout: "threads": { 2024-03-18T22:58:35.307 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threads", 2024-03-18T22:58:35.307 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.307 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.307 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.307 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-03-18T22:58:35.307 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-03-18T22:58:35.307 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "32", 2024-03-18T22:58:35.307 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.307 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T22:58:35.307 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.307 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.307 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.308 INFO:teuthology.orchestra.run.smithi040.stdout: "username": { 2024-03-18T22:58:35.308 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "username", 2024-03-18T22:58:35.308 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.308 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.308 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.308 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.308 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.308 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.308 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.308 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "username of InfluxDB server user", 2024-03-18T22:58:35.308 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.308 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.308 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.309 INFO:teuthology.orchestra.run.smithi040.stdout: "verify_ssl": { 2024-03-18T22:58:35.309 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "verify_ssl", 2024-03-18T22:58:35.309 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.309 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.309 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.309 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "true", 2024-03-18T22:58:35.309 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.309 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.309 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.309 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T22:58:35.309 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.309 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.309 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.309 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.309 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.310 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "insights", 2024-03-18T22:58:35.310 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.310 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.310 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.310 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.310 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.310 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.310 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.310 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.310 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.310 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.310 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.310 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.311 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.312 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.312 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.312 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.312 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.312 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.312 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.312 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.312 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.312 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.312 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.312 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.312 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.312 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.312 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.313 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.313 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.313 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.313 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.313 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.313 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.313 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.313 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.313 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.313 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.313 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.313 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.313 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.313 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.313 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.314 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.314 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.314 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.314 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.314 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.314 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.314 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.314 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.314 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.314 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.314 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.314 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.315 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.315 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.315 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.315 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "iostat", 2024-03-18T22:58:35.315 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.315 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.315 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.315 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.315 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.315 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.315 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.315 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.315 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.316 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.317 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.317 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.317 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.317 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.317 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.317 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.317 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.317 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.317 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.317 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.317 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.317 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.317 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.318 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.318 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.318 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.318 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.318 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.318 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.318 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.318 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.318 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.318 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.318 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.318 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.318 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.318 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.319 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "k8sevents", 2024-03-18T22:58:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_event_retention_days": { 2024-03-18T22:58:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ceph_event_retention_days", 2024-03-18T22:58:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7", 2024-03-18T22:58:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T22:58:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: "config_check_secs": { 2024-03-18T22:58:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_check_secs", 2024-03-18T22:58:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "10", 2024-03-18T22:58:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T22:58:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.322 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.323 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.324 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.324 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.324 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.324 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.324 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.324 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.324 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.324 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.324 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.324 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.324 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.324 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.324 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.325 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.326 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.326 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.326 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.326 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.326 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.326 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.326 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.326 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.326 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.326 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.326 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.326 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.326 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.327 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.327 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.327 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.327 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.327 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.327 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.327 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "localpool", 2024-03-18T22:58:35.327 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.327 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout: "failure_domain": { 2024-03-18T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "failure_domain", 2024-03-18T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "host", 2024-03-18T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "failure domain for any created local pool", 2024-03-18T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.329 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.330 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.330 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.330 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.330 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.330 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.330 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.330 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.330 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.330 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.330 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.330 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.330 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.330 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.331 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.332 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.332 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.332 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.332 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.332 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.332 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.332 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.332 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.332 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.332 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.332 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.332 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.332 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.333 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.333 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.333 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.333 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.333 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.333 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.333 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.333 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.333 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.333 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.333 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": { 2024-03-18T22:58:35.333 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_size", 2024-03-18T22:58:35.333 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.333 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.334 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.334 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.334 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.334 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.334 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.334 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default min_size for any created local pool", 2024-03-18T22:58:35.334 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T22:58:35.334 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.334 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.334 INFO:teuthology.orchestra.run.smithi040.stdout: "num_rep": { 2024-03-18T22:58:35.334 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "num_rep", 2024-03-18T22:58:35.334 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.334 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.335 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.335 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3", 2024-03-18T22:58:35.335 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.335 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.335 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.335 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default replica count for any created local pool", 2024-03-18T22:58:35.335 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.335 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.335 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.335 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": { 2024-03-18T22:58:35.335 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_num", 2024-03-18T22:58:35.335 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.335 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.336 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.336 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "128", 2024-03-18T22:58:35.336 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.336 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.336 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.336 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T22:58:35.336 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.336 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.336 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.336 INFO:teuthology.orchestra.run.smithi040.stdout: "prefix": { 2024-03-18T22:58:35.336 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prefix", 2024-03-18T22:58:35.336 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.336 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.336 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.337 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.337 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.337 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.337 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.337 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name prefix for any created local pool", 2024-03-18T22:58:35.337 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.337 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.337 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.337 INFO:teuthology.orchestra.run.smithi040.stdout: "subtree": { 2024-03-18T22:58:35.337 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "subtree", 2024-03-18T22:58:35.337 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.337 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.337 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.338 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "rack", 2024-03-18T22:58:35.338 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.338 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.338 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.338 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T22:58:35.338 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T22:58:35.338 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.338 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.338 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.338 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.338 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mds_autoscaler", 2024-03-18T22:58:35.339 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.339 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.339 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.339 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.339 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.339 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.339 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.339 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.339 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.339 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.339 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.339 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.339 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.342 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.342 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.342 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.342 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.342 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.342 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.342 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.342 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.342 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.342 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.342 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.342 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.343 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.343 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.343 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.343 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.343 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.343 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.343 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.343 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.343 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.343 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.343 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.343 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.343 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.343 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.344 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.344 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.344 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.344 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.344 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.344 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.344 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.344 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.344 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.344 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.344 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.344 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.344 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.344 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.345 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.345 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.345 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.345 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.345 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.345 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.345 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.345 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.345 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.345 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.345 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.345 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.345 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.345 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.346 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.346 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.346 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.346 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.346 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirroring", 2024-03-18T22:58:35.346 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.346 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.346 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.346 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.346 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.346 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.346 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.347 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.347 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.347 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.347 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.347 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.347 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.347 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.347 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.347 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.347 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.347 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.347 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.347 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.347 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.347 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.348 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.348 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.348 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.348 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.348 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.348 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.348 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.348 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.348 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.348 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.348 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.348 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.348 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.348 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.349 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.349 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.349 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.349 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.349 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.349 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.349 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.349 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.349 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.349 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.349 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.349 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.349 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.350 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.350 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.350 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.350 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.350 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.350 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.350 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.350 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.350 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.350 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.350 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.350 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.350 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.351 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.351 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.351 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.351 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.351 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.351 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.351 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.351 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.351 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.351 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "nfs", 2024-03-18T22:58:35.351 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.352 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.352 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.352 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.352 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.352 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.352 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.352 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.352 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.352 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.352 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.352 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.352 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.352 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.352 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.353 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.353 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.353 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.353 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.353 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.353 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.353 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.353 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.353 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.353 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.353 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.353 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.353 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.353 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.354 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.354 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.354 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.354 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.354 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.354 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.354 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.354 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.354 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.354 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.354 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.354 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.354 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.355 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.355 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.355 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.355 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.355 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.355 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.355 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.355 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.355 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.355 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.355 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.355 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.355 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.355 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.356 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.356 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.356 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.356 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.356 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.356 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.356 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.356 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.356 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.356 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.356 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.356 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.356 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.356 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.357 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.357 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-03-18T22:58:35.357 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.357 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.357 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.357 INFO:teuthology.orchestra.run.smithi040.stdout: "fail_fs": { 2024-03-18T22:58:35.357 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "fail_fs", 2024-03-18T22:58:35.357 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.357 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.357 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.357 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.357 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.358 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.358 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.358 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T22:58:35.358 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.358 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.358 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.358 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.358 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.358 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.358 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.358 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.358 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.358 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.359 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.359 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.359 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.359 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.359 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.359 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.359 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.359 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.359 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.359 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.359 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.359 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.359 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.360 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.360 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.360 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.360 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.360 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.360 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.360 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.360 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.360 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.360 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.360 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.360 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.360 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.360 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.361 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.361 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.361 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.361 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.361 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.361 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.361 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.361 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.361 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.361 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.361 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.361 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.361 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.361 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.361 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.362 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.362 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.362 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.362 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.362 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.362 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.362 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.362 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.362 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.362 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.362 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.362 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator": { 2024-03-18T22:58:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-03-18T22:58:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.363 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.364 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm", 2024-03-18T22:58:35.364 INFO:teuthology.orchestra.run.smithi040.stdout: "rook", 2024-03-18T22:58:35.364 INFO:teuthology.orchestra.run.smithi040.stdout: "test_orchestrator" 2024-03-18T22:58:35.364 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.364 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Orchestrator backend", 2024-03-18T22:58:35.364 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.364 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.364 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.364 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.364 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.364 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_perf_query", 2024-03-18T22:58:35.365 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.365 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.365 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.365 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.365 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.365 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.365 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.365 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.365 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.365 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.365 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.365 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.365 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.365 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.366 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.366 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.366 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.366 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.366 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.366 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.366 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.366 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.366 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.366 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.366 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.366 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.366 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.367 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.367 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.367 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.367 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.367 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.367 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.367 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.367 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.367 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.367 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.367 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.367 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.367 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.368 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.368 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.368 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.368 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.368 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.368 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.368 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.368 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.368 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.368 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.368 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.368 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.368 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.368 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.369 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.369 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.369 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.369 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.369 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.369 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.369 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.369 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.369 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.369 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.369 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.369 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.369 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.370 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.370 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.370 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.370 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.370 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_support", 2024-03-18T22:58:35.370 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.370 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.370 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.370 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.370 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.370 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.371 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.371 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.371 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.371 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.371 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.371 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.371 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.371 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.371 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.371 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.371 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.371 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.371 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.371 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.371 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.372 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.372 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.372 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.372 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.372 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.372 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.372 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.372 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.372 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.372 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.372 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.372 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.372 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.372 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.373 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.373 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.373 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.373 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.373 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.373 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.373 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.373 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.373 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.373 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.373 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.373 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.373 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.374 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.374 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.374 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.374 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.374 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.374 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.374 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.374 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.374 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.374 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.374 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.374 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.374 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.375 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.375 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.375 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.375 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.375 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.375 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.375 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.375 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.375 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.375 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.375 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_autoscaler", 2024-03-18T22:58:35.375 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.375 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.376 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.376 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.376 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.376 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.376 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.376 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.376 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.376 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.376 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.376 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.376 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.376 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.376 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.376 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.377 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.377 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.377 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.377 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.377 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.377 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.377 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.377 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.377 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.377 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.377 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.377 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.377 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.378 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.378 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.378 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.378 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.378 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.378 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.378 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.378 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.378 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.378 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.378 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.378 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.378 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.378 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.379 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.379 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.379 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.379 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.379 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.379 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.379 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.379 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.379 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.379 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.379 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.379 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.379 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.380 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.380 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.380 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.380 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.380 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.380 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.380 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.380 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.380 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.380 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.380 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.380 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.380 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.381 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-03-18T22:58:35.381 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-03-18T22:58:35.381 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.381 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.381 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.381 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-03-18T22:58:35.381 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.381 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.381 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.381 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.381 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.381 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.381 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.382 INFO:teuthology.orchestra.run.smithi040.stdout: "threshold": { 2024-03-18T22:58:35.382 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threshold", 2024-03-18T22:58:35.382 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.382 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.382 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.382 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-03-18T22:58:35.382 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1.0", 2024-03-18T22:58:35.382 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.382 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.382 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "scaling threshold", 2024-03-18T22:58:35.382 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T22:58:35.382 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.382 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.382 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.383 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.383 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "progress", 2024-03-18T22:58:35.383 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.383 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.383 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.383 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_pg_recovery_event": { 2024-03-18T22:58:35.383 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_pg_recovery_event", 2024-03-18T22:58:35.383 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.383 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.383 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.383 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.383 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.384 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.384 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.384 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T22:58:35.384 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.384 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.384 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.384 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-03-18T22:58:35.384 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-03-18T22:58:35.384 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.384 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.384 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.384 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.384 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.384 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.385 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.385 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.385 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.385 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.385 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.385 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.385 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.385 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.385 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.385 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.385 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.385 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.385 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.385 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.386 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.386 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.386 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.386 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.386 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.386 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.386 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.386 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.386 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.386 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.386 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.386 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.386 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.386 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.387 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.387 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.387 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.387 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.387 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.387 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.387 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.387 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.387 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.387 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.387 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.387 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.387 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.388 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.388 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.388 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.388 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.388 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.388 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.388 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.388 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.388 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.388 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.388 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.388 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.388 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.388 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.388 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.389 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.389 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.389 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.389 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.389 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.389 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.389 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.389 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.389 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.389 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.389 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.389 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.389 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.389 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.390 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.390 INFO:teuthology.orchestra.run.smithi040.stdout: "max_completed_events": { 2024-03-18T22:58:35.390 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_completed_events", 2024-03-18T22:58:35.390 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.390 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.390 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.390 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "50", 2024-03-18T22:58:35.390 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.390 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.390 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.390 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "number of past completed events to remember", 2024-03-18T22:58:35.390 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.390 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.391 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.391 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-03-18T22:58:35.391 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-03-18T22:58:35.391 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.391 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.391 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.391 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-03-18T22:58:35.391 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.391 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.391 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.391 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long the module is going to sleep", 2024-03-18T22:58:35.391 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.391 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.391 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus", 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: "cache": { 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cache", 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.392 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: "exclude_perf_counters": { 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "exclude_perf_counters", 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.393 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T22:58:35.394 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T22:58:35.394 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.394 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.394 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.394 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.394 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.394 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.394 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.394 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.394 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.394 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.394 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.394 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.395 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.395 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.395 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.395 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.395 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.395 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.395 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.395 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.395 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.395 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.395 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.395 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.395 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.396 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.396 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.396 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.396 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.396 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.396 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.396 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.396 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.396 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.396 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.396 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.396 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.396 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.396 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.397 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.397 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.397 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.397 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.397 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.397 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.397 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.397 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.397 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.397 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.397 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.397 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.397 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.397 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.398 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.398 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.398 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.398 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.398 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.398 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.398 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.398 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.398 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.398 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.398 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.398 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.398 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.399 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.399 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.399 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools": { 2024-03-18T22:58:35.399 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools", 2024-03-18T22:58:35.399 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.399 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.399 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.399 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.399 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.399 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.399 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.399 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.399 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.399 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.400 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.400 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T22:58:35.400 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T22:58:35.400 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.400 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.400 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.400 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "300", 2024-03-18T22:58:35.400 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.400 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.400 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.400 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.400 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.400 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.401 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.401 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_interval": { 2024-03-18T22:58:35.401 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_interval", 2024-03-18T22:58:35.401 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.401 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.401 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.401 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15.0", 2024-03-18T22:58:35.401 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.401 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.401 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.401 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.401 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.401 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.401 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.402 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-03-18T22:58:35.402 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-03-18T22:58:35.402 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.402 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.402 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.402 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-03-18T22:58:35.402 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.402 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.402 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.402 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T22:58:35.402 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.402 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.402 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "9283", 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.403 INFO:teuthology.orchestra.run.smithi040.stdout: "stale_cache_strategy": { 2024-03-18T22:58:35.404 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stale_cache_strategy", 2024-03-18T22:58:35.404 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.404 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.404 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.404 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "log", 2024-03-18T22:58:35.404 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.404 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.404 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.404 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.404 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.404 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.404 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.405 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-03-18T22:58:35.405 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-03-18T22:58:35.405 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.405 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.405 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.405 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "default", 2024-03-18T22:58:35.405 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.405 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.405 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.405 INFO:teuthology.orchestra.run.smithi040.stdout: "default", 2024-03-18T22:58:35.405 INFO:teuthology.orchestra.run.smithi040.stdout: "error" 2024-03-18T22:58:35.405 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.405 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.405 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.406 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.406 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.406 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-03-18T22:58:35.406 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-03-18T22:58:35.406 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.406 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.406 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.406 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-03-18T22:58:35.406 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-03-18T22:58:35.406 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-03-18T22:58:35.406 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.406 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.407 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.407 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.407 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.407 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.407 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.407 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_support", 2024-03-18T22:58:35.407 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.407 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.407 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.407 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.407 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.407 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.407 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.408 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.408 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.408 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.408 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.408 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.408 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.408 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.408 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.408 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.408 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.408 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.408 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.408 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.408 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.409 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.409 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.409 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.409 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.409 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.409 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.409 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.409 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.409 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.409 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.409 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.409 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.409 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.409 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.410 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.410 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.410 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.410 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.410 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.410 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.410 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.410 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.410 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.410 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.410 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.410 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.410 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.411 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.411 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.411 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.411 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.411 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.411 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.411 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.411 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.411 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.411 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.411 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.411 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.411 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.412 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.412 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.412 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.412 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.412 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.412 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.412 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.412 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.412 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_snap_create": { 2024-03-18T22:58:35.412 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_snap_create", 2024-03-18T22:58:35.412 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.412 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.412 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.412 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.413 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.413 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.413 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.413 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.413 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.413 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.413 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.413 INFO:teuthology.orchestra.run.smithi040.stdout: "mirror_snapshot_schedule": { 2024-03-18T22:58:35.413 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T22:58:35.413 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.413 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.413 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.413 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.413 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.414 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.414 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.414 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.414 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.414 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.414 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.414 INFO:teuthology.orchestra.run.smithi040.stdout: "trash_purge_schedule": { 2024-03-18T22:58:35.414 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "trash_purge_schedule", 2024-03-18T22:58:35.414 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.414 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.414 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.414 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.415 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.415 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.415 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.415 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.415 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.415 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.415 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.415 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.415 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.415 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "restful", 2024-03-18T22:58:35.415 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.415 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_auth": { 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_auth", 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.416 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-03-18T22:58:35.417 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-03-18T22:58:35.417 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.417 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.417 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.417 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.417 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.417 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.417 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.417 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.417 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.417 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.417 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.417 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.417 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.418 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.418 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.418 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.418 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.418 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.418 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.418 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.418 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.418 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.418 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.419 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.420 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.421 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.422 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.422 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.422 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.422 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.422 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.422 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.422 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.422 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.422 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.422 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-03-18T22:58:35.422 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-03-18T22:58:35.423 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.423 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.423 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.423 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.423 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.423 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.423 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.423 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.423 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.423 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.423 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.423 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-03-18T22:58:35.423 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-03-18T22:58:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rgw", 2024-03-18T22:58:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.425 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.426 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.427 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.428 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.429 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rook", 2024-03-18T22:58:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.430 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.431 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.432 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.433 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.434 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.434 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.434 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.434 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.434 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.434 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.434 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.434 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.434 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.434 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.434 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.434 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: "storage_class": { 2024-03-18T22:58:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "storage_class", 2024-03-18T22:58:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "local", 2024-03-18T22:58:35.435 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T22:58:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "selftest", 2024-03-18T22:58:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.436 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.437 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.438 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.439 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.440 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "roption1": { 2024-03-18T22:58:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption1", 2024-03-18T22:58:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: "roption2": { 2024-03-18T22:58:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption2", 2024-03-18T22:58:35.442 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "xyz", 2024-03-18T22:58:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption1": { 2024-03-18T22:58:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption1", 2024-03-18T22:58:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.443 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.444 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.444 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.444 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.444 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.444 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.444 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.444 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.444 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.444 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.444 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption2": { 2024-03-18T22:58:35.444 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption2", 2024-03-18T22:58:35.444 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.444 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.445 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.445 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.445 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.445 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.445 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.445 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.445 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.445 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.445 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.445 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption3": { 2024-03-18T22:58:35.445 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption3", 2024-03-18T22:58:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption4": { 2024-03-18T22:58:35.446 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption4", 2024-03-18T22:58:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption5": { 2024-03-18T22:58:35.447 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption5", 2024-03-18T22:58:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption6": { 2024-03-18T22:58:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption6", 2024-03-18T22:58:35.448 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption7": { 2024-03-18T22:58:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption7", 2024-03-18T22:58:35.449 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.450 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.450 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.450 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.450 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-03-18T22:58:35.450 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "42", 2024-03-18T22:58:35.450 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.450 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.450 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.450 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.450 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.450 INFO:teuthology.orchestra.run.smithi040.stdout: "testkey": { 2024-03-18T22:58:35.450 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testkey", 2024-03-18T22:58:35.450 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: "testlkey": { 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testlkey", 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.451 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "testnewline": { 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testnewline", 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.452 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snap_schedule", 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_m_granularity": { 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_m_granularity", 2024-03-18T22:58:35.453 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.454 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.454 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.454 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.454 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.454 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.454 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.454 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T22:58:35.454 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.454 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.454 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.454 INFO:teuthology.orchestra.run.smithi040.stdout: "dump_on_update": { 2024-03-18T22:58:35.454 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dump_on_update", 2024-03-18T22:58:35.454 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "dump database to debug log on update", 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.455 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.456 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.456 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.456 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.456 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.456 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.456 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.456 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.456 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.456 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.456 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.456 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.456 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.456 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.456 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.456 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.457 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.457 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.457 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.457 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.457 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.457 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.457 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.457 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.457 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.457 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.457 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.457 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.457 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.458 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.458 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.458 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.458 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.458 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.458 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.458 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.458 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.458 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.458 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.458 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.458 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.458 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.459 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.459 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.459 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.459 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.459 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.459 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.459 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.459 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.459 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.459 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.459 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.459 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.459 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.459 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stats", 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.460 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.461 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.461 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.461 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.461 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.461 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.461 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.461 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.461 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.461 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.461 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.461 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.461 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.461 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.461 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.461 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.462 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.462 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.462 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.462 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.462 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.462 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.462 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.462 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.462 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.462 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.462 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.462 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.463 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.464 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "status", 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.465 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.466 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.466 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.466 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.466 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.466 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.466 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.466 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.466 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.466 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.466 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.466 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.466 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.466 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.467 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.467 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.467 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.467 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.467 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.467 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.467 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.467 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.467 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.467 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.467 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.467 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.467 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.468 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.468 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.468 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.468 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.468 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.468 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.468 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.468 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.468 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.468 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.468 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.468 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.468 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.469 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.469 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.469 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.469 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.469 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.469 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.469 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.469 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.469 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.469 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.469 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.469 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.469 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.469 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.470 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.470 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.470 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.470 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.470 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.470 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.470 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.470 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.470 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.470 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.470 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.470 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.470 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.470 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telegraf", 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: "address": { 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "address", 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.471 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15", 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.472 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.473 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.473 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.473 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.473 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.473 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.473 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.473 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.473 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.473 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.473 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.473 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.473 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.473 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.473 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.474 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.474 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.474 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.474 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.474 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.474 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.474 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.474 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.474 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.474 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.474 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.474 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.475 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.475 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.475 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.475 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.475 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.475 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.475 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.475 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.475 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.475 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.475 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.475 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.475 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.476 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.476 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.476 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.476 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.476 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.476 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.476 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.476 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.476 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.476 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.476 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.476 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.476 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.476 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.476 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.477 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.477 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.477 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.477 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.477 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.477 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.477 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.477 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.477 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.477 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.477 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.477 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.477 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.478 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.478 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.478 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telemetry", 2024-03-18T22:58:35.478 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.478 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.478 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.478 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_basic": { 2024-03-18T22:58:35.478 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_basic", 2024-03-18T22:58:35.478 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.478 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.478 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.478 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.479 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.479 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.479 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.479 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T22:58:35.479 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.479 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.479 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.479 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_crash": { 2024-03-18T22:58:35.479 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_crash", 2024-03-18T22:58:35.479 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.479 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.479 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.480 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.480 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.480 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.480 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.480 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T22:58:35.480 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.480 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.480 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.480 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_device": { 2024-03-18T22:58:35.480 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_device", 2024-03-18T22:58:35.480 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.480 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.480 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.481 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.481 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.481 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.481 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.481 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T22:58:35.481 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.481 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.481 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.481 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_ident": { 2024-03-18T22:58:35.481 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_ident", 2024-03-18T22:58:35.481 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.481 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.481 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.482 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.482 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.482 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.482 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.482 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T22:58:35.482 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.482 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.482 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.482 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_perf": { 2024-03-18T22:58:35.482 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_perf", 2024-03-18T22:58:35.482 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: "contact": { 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "contact", 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.483 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: "description": { 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "description", 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.484 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: "device_url": { 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_url", 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.485 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-03-18T22:58:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-03-18T22:58:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.486 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.487 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.487 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.487 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.487 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-03-18T22:58:35.487 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-03-18T22:58:35.487 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.487 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.487 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.487 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "24", 2024-03-18T22:58:35.487 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "8", 2024-03-18T22:58:35.487 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.487 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.487 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: "last_opt_revision": { 2024-03-18T22:58:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "last_opt_revision", 2024-03-18T22:58:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1", 2024-03-18T22:58:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.488 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard": { 2024-03-18T22:58:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard", 2024-03-18T22:58:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.489 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard_description": { 2024-03-18T22:58:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard_description", 2024-03-18T22:58:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.490 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.491 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.492 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.492 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.492 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.492 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.492 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.492 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.492 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.492 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.492 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.492 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.492 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.492 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.492 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.493 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.493 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.493 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.493 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.493 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.493 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.493 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.493 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.493 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.493 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.493 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.493 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.493 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.493 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.494 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.494 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.494 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.494 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.494 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.494 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.494 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.494 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.494 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.494 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.494 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.494 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.494 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: "organization": { 2024-03-18T22:58:35.495 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "organization", 2024-03-18T22:58:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: "proxy": { 2024-03-18T22:58:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "proxy", 2024-03-18T22:58:35.496 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.497 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.497 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.497 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.497 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.497 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.497 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.497 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.497 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.497 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.497 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.497 INFO:teuthology.orchestra.run.smithi040.stdout: "url": { 2024-03-18T22:58:35.497 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url", 2024-03-18T22:58:35.497 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.498 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.498 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.498 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T22:58:35.498 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.498 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.498 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.498 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.498 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.498 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.498 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.498 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.498 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "test_orchestrator", 2024-03-18T22:58:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.499 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.500 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.500 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.500 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.500 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.500 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.500 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.500 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.500 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.500 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.500 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.500 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.500 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.500 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.501 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.501 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.501 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.501 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.501 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.501 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.501 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.501 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.501 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.501 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.501 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.501 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.501 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.502 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.502 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.502 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.502 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.502 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.502 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.502 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.502 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.502 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.502 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.502 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.502 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.502 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.503 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.503 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.503 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.503 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.503 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.503 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.503 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.503 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.503 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.503 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.503 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.503 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.503 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.503 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.504 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.504 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.504 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.504 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.504 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "volumes", 2024-03-18T22:58:35.504 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.504 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.504 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.504 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.504 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.504 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.504 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.504 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.506 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.506 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.506 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.506 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.506 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.506 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.506 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.506 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.506 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.506 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.506 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.506 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.506 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.507 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.507 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.507 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.507 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.507 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.507 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.507 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.507 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.507 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.507 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.507 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.507 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.507 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.507 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.507 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.508 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.508 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.508 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.508 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.508 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.508 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.508 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.508 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.508 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.508 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.508 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.508 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.508 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_clones": { 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_clones", 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4", 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.509 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T22:58:35.510 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.510 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.510 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.510 INFO:teuthology.orchestra.run.smithi040.stdout: "periodic_async_work": { 2024-03-18T22:58:35.510 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "periodic_async_work", 2024-03-18T22:58:35.510 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.510 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.510 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.510 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.510 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.510 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.510 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.510 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Periodically check for async work", 2024-03-18T22:58:35.510 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.511 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.511 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.511 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_delay": { 2024-03-18T22:58:35.511 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_delay", 2024-03-18T22:58:35.511 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.511 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.511 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.511 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-03-18T22:58:35.511 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.511 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.511 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.511 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T22:58:35.511 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_no_wait": { 2024-03-18T22:58:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T22:58:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T22:58:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.512 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix", 2024-03-18T22:58:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: "discovery_interval": { 2024-03-18T22:58:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "discovery_interval", 2024-03-18T22:58:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-03-18T22:58:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.513 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "100", 2024-03-18T22:58:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: "identifier": { 2024-03-18T22:58:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "identifier", 2024-03-18T22:58:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.514 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.515 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.515 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.515 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.515 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.515 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.515 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.515 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.515 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-03-18T22:58:35.515 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-03-18T22:58:35.515 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.515 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.515 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.515 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-03-18T22:58:35.516 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.516 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.516 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.516 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.516 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.516 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.516 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.516 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.516 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.516 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.516 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.516 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.516 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.517 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.518 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.518 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.518 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.518 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.518 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.518 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.518 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.518 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.518 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.518 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.518 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.518 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.519 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.519 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.519 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.519 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.519 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.519 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.519 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.519 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.519 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.519 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.519 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.519 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.519 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.519 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.520 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.520 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.520 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.520 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.520 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.520 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.520 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.520 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.520 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.520 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.520 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.520 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.520 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.520 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_host": { 2024-03-18T22:58:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_host", 2024-03-18T22:58:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.521 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.522 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.522 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.522 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.522 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.522 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_port": { 2024-03-18T22:58:35.522 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_port", 2024-03-18T22:58:35.522 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.522 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.522 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.522 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10051", 2024-03-18T22:58:35.522 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.522 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.522 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.523 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.523 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.523 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.523 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.523 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_sender": { 2024-03-18T22:58:35.523 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_sender", 2024-03-18T22:58:35.523 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.523 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.523 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.523 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T22:58:35.523 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.523 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.524 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.524 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.524 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.524 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.524 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.524 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.524 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.524 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.524 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-18T22:58:35.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.524 INFO:teuthology.orchestra.run.smithi040.stdout: "gid": 24113, 2024-03-18T22:58:35.524 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "y", 2024-03-18T22:58:35.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_features": 4540701547738038271, 2024-03-18T22:58:35.524 INFO:teuthology.orchestra.run.smithi040.stdout: "available_modules": [ 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "alerts", 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T22:58:35.525 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.526 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.526 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.526 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.526 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.526 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.526 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.526 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.526 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.526 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.526 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.526 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.526 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.526 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.526 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.527 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.527 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.527 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.527 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.527 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.527 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.527 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.527 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.527 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.527 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.527 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.527 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.527 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.527 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.528 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.528 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.528 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.528 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.528 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.528 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.528 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.528 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.528 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.528 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.528 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.528 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.528 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.529 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.529 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.529 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.529 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.529 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.529 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.529 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.529 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.529 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.529 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.529 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.529 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.529 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.529 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.530 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_destination": { 2024-03-18T22:58:35.531 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_destination", 2024-03-18T22:58:35.531 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.531 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.531 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.531 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.531 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.531 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.531 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.531 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email address to send alerts to", 2024-03-18T22:58:35.531 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.531 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.531 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.531 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_from_name": { 2024-03-18T22:58:35.531 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_from_name", 2024-03-18T22:58:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "Ceph", 2024-03-18T22:58:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email From: name", 2024-03-18T22:58:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_host": { 2024-03-18T22:58:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_host", 2024-03-18T22:58:35.532 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.533 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.533 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.533 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.533 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.533 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.533 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.533 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP server", 2024-03-18T22:58:35.533 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.533 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.533 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.533 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_password": { 2024-03-18T22:58:35.533 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_password", 2024-03-18T22:58:35.533 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.533 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.534 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.534 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.534 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.534 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.534 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.534 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Password to authenticate with", 2024-03-18T22:58:35.534 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.534 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.534 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.534 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_port": { 2024-03-18T22:58:35.534 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_port", 2024-03-18T22:58:35.534 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.534 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.534 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.535 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "465", 2024-03-18T22:58:35.535 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.535 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.535 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.535 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP port", 2024-03-18T22:58:35.535 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.535 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.535 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.535 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_sender": { 2024-03-18T22:58:35.535 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_sender", 2024-03-18T22:58:35.535 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.535 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.535 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.535 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP envelope sender", 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_ssl": { 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_ssl", 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.536 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.537 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.537 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T22:58:35.537 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.537 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.537 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.537 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_user": { 2024-03-18T22:58:35.537 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_user", 2024-03-18T22:58:35.537 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.537 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.537 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.537 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.537 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.537 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.538 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.538 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "User to authenticate as", 2024-03-18T22:58:35.538 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.538 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.538 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.538 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.538 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.538 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "balancer", 2024-03-18T22:58:35.538 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.538 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.538 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.538 INFO:teuthology.orchestra.run.smithi040.stdout: "active": { 2024-03-18T22:58:35.539 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "active", 2024-03-18T22:58:35.539 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.539 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.539 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.539 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.539 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.539 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.539 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.539 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T22:58:35.539 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.539 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.539 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.539 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_time": { 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_time", 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0000", 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_weekday": { 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_weekday", 2024-03-18T22:58:35.540 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-03-18T22:58:35.541 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.541 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.541 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-03-18T22:58:35.541 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-03-18T22:58:35.541 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-03-18T22:58:35.541 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.541 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T22:58:35.541 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T22:58:35.541 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.541 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.541 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_max_iterations": { 2024-03-18T22:58:35.541 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_max_iterations", 2024-03-18T22:58:35.541 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-03-18T22:58:35.541 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.542 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.542 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "25", 2024-03-18T22:58:35.542 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-03-18T22:58:35.542 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "250", 2024-03-18T22:58:35.542 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.542 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T22:58:35.542 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.542 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.542 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.542 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_metrics": { 2024-03-18T22:58:35.542 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_metrics", 2024-03-18T22:58:35.542 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.542 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.543 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.543 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T22:58:35.543 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.543 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.543 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.543 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T22:58:35.543 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T22:58:35.543 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.543 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.545 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_step": { 2024-03-18T22:58:35.545 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_step", 2024-03-18T22:58:35.546 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.546 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.546 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.546 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.5", 2024-03-18T22:58:35.546 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0.001", 2024-03-18T22:58:35.546 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "0.999", 2024-03-18T22:58:35.546 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.546 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "aggressiveness of optimization", 2024-03-18T22:58:35.546 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T22:58:35.546 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.546 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.546 INFO:teuthology.orchestra.run.smithi040.stdout: "end_time": { 2024-03-18T22:58:35.546 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_time", 2024-03-18T22:58:35.546 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.547 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.547 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.547 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2359", 2024-03-18T22:58:35.547 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.547 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.547 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.547 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T22:58:35.547 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T22:58:35.547 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.547 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.547 INFO:teuthology.orchestra.run.smithi040.stdout: "end_weekday": { 2024-03-18T22:58:35.547 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_weekday", 2024-03-18T22:58:35.547 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-03-18T22:58:35.548 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.548 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.548 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-03-18T22:58:35.548 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-03-18T22:58:35.548 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-03-18T22:58:35.548 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.548 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T22:58:35.548 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T22:58:35.548 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.548 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.548 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.548 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.548 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.549 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.549 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.549 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.549 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.549 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.549 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.549 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.549 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.549 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.549 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.549 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.549 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.549 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.549 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.550 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.550 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.550 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.550 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.550 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.550 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.550 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.550 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.550 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.550 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.550 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.550 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.550 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.551 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.551 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.551 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.551 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.551 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.551 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.551 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.551 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.551 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.551 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.551 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.551 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.551 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.552 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.552 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.552 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.552 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.552 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.552 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.552 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.552 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.552 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.552 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.552 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.552 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.552 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.553 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.553 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.553 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.553 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.553 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.553 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.553 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.553 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.553 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.553 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.553 INFO:teuthology.orchestra.run.smithi040.stdout: "min_score": { 2024-03-18T22:58:35.553 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_score", 2024-03-18T22:58:35.553 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.553 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.554 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.554 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-03-18T22:58:35.554 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.554 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.554 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.554 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T22:58:35.554 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.554 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.554 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.554 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-03-18T22:58:35.554 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-03-18T22:58:35.554 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.554 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.555 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.555 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "upmap", 2024-03-18T22:58:35.555 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.555 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.555 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.555 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-compat", 2024-03-18T22:58:35.555 INFO:teuthology.orchestra.run.smithi040.stdout: "none", 2024-03-18T22:58:35.555 INFO:teuthology.orchestra.run.smithi040.stdout: "read", 2024-03-18T22:58:35.555 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap", 2024-03-18T22:58:35.555 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap-read" 2024-03-18T22:58:35.555 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.555 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Balancer mode", 2024-03-18T22:58:35.555 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.555 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.555 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.556 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_ids": { 2024-03-18T22:58:35.556 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_ids", 2024-03-18T22:58:35.556 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.556 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.556 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.556 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.556 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.556 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.556 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.556 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T22:58:35.556 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.556 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.556 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.557 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-03-18T22:58:35.557 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-03-18T22:58:35.557 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.557 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.557 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.557 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-03-18T22:58:35.557 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.557 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.557 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.557 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T22:58:35.557 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.557 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.557 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.557 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_deviation": { 2024-03-18T22:58:35.558 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_deviation", 2024-03-18T22:58:35.558 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.558 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.558 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.558 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-03-18T22:58:35.558 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-03-18T22:58:35.558 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.558 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.558 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T22:58:35.558 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T22:58:35.558 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.558 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.559 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_optimizations": { 2024-03-18T22:58:35.559 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_optimizations", 2024-03-18T22:58:35.559 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-03-18T22:58:35.559 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.559 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.559 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.559 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.559 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.559 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.559 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T22:58:35.559 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.559 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.559 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.559 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.560 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.560 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm", 2024-03-18T22:58:35.560 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.560 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.560 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.560 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_down_multiplier": { 2024-03-18T22:58:35.560 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_down_multiplier", 2024-03-18T22:58:35.560 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.560 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.560 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.560 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-03-18T22:58:35.560 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.560 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.561 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.561 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T22:58:35.561 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.561 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.561 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.561 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_refresh_rate": { 2024-03-18T22:58:35.561 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_refresh_rate", 2024-03-18T22:58:35.561 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.561 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.561 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.561 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "20", 2024-03-18T22:58:35.561 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.561 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.561 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.562 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T22:58:35.562 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.562 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.562 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.562 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_starting_port": { 2024-03-18T22:58:35.562 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_starting_port", 2024-03-18T22:58:35.562 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.562 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.562 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.562 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4721", 2024-03-18T22:58:35.562 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.562 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.562 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.562 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T22:58:35.563 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.563 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.563 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.563 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_ptrace": { 2024-03-18T22:58:35.563 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_ptrace", 2024-03-18T22:58:35.563 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.563 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.563 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.563 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.563 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.563 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.563 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.563 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T22:58:35.563 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T22:58:35.564 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.564 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.564 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_interval": { 2024-03-18T22:58:35.564 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_interval", 2024-03-18T22:58:35.564 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.564 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.564 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.564 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-03-18T22:58:35.564 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.564 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.564 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.564 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T22:58:35.564 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.564 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.565 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.565 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_memory_target_ratio": { 2024-03-18T22:58:35.565 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T22:58:35.565 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.565 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.565 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.565 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.7", 2024-03-18T22:58:35.565 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.565 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.565 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.565 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T22:58:35.565 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.565 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.566 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.566 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm_log_destination": { 2024-03-18T22:58:35.566 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm_log_destination", 2024-03-18T22:58:35.566 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.566 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.566 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.566 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.566 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.566 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.566 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.566 INFO:teuthology.orchestra.run.smithi040.stdout: "file", 2024-03-18T22:58:35.566 INFO:teuthology.orchestra.run.smithi040.stdout: "file,syslog", 2024-03-18T22:58:35.566 INFO:teuthology.orchestra.run.smithi040.stdout: "syslog" 2024-03-18T22:58:35.566 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.567 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T22:58:35.567 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.567 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.567 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.567 INFO:teuthology.orchestra.run.smithi040.stdout: "cgroups_split": { 2024-03-18T22:58:35.567 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cgroups_split", 2024-03-18T22:58:35.567 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.567 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.567 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.567 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.567 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.567 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.567 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.568 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T22:58:35.568 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.568 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.568 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.568 INFO:teuthology.orchestra.run.smithi040.stdout: "config_checks_enabled": { 2024-03-18T22:58:35.568 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_checks_enabled", 2024-03-18T22:58:35.568 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.568 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.568 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.568 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.568 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.568 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.568 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.569 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T22:58:35.569 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.569 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.569 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.569 INFO:teuthology.orchestra.run.smithi040.stdout: "config_dashboard": { 2024-03-18T22:58:35.569 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_dashboard", 2024-03-18T22:58:35.569 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.569 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.569 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.569 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.569 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.569 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.569 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.569 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T22:58:35.570 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.570 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.570 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.570 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_alertmanager": { 2024-03-18T22:58:35.570 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_alertmanager", 2024-03-18T22:58:35.570 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.570 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.570 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.570 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T22:58:35.570 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.570 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.570 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.570 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_base": { 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_base", 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Container image name, without the tag", 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.571 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.572 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.572 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_elasticsearch": { 2024-03-18T22:58:35.572 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_elasticsearch", 2024-03-18T22:58:35.572 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.572 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.572 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.572 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T22:58:35.572 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.572 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.572 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.572 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "elasticsearch container image", 2024-03-18T22:58:35.572 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.572 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.572 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.573 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_grafana": { 2024-03-18T22:58:35.573 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_grafana", 2024-03-18T22:58:35.573 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.573 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.573 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.573 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T22:58:35.573 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.573 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.573 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.573 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-03-18T22:58:35.573 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.573 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.573 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.574 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_haproxy": { 2024-03-18T22:58:35.574 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_haproxy", 2024-03-18T22:58:35.574 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.574 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.574 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.574 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T22:58:35.574 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.574 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.574 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.574 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "HAproxy container image", 2024-03-18T22:58:35.574 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.574 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.574 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.574 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_agent": { 2024-03-18T22:58:35.575 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_agent", 2024-03-18T22:58:35.575 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.575 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.575 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.575 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T22:58:35.575 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.575 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.575 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.575 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger agent container image", 2024-03-18T22:58:35.575 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.575 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.575 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.575 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_collector": { 2024-03-18T22:58:35.575 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_collector", 2024-03-18T22:58:35.576 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.576 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.576 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.576 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T22:58:35.576 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.576 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.576 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.576 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger collector container image", 2024-03-18T22:58:35.576 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.576 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.576 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.576 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_query": { 2024-03-18T22:58:35.576 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_query", 2024-03-18T22:58:35.576 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.577 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.577 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.577 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T22:58:35.577 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.577 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.577 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.577 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger query container image", 2024-03-18T22:58:35.577 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.577 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.577 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.577 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_keepalived": { 2024-03-18T22:58:35.577 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_keepalived", 2024-03-18T22:58:35.577 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.578 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.578 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.578 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T22:58:35.578 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.578 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.578 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.578 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Keepalived container image", 2024-03-18T22:58:35.578 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.578 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.578 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.578 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_loki": { 2024-03-18T22:58:35.578 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_loki", 2024-03-18T22:58:35.578 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.579 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.579 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.579 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T22:58:35.579 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.579 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.579 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.579 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Loki container image", 2024-03-18T22:58:35.579 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.579 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.579 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.579 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_node_exporter": { 2024-03-18T22:58:35.579 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_node_exporter", 2024-03-18T22:58:35.579 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.579 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.580 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.580 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T22:58:35.580 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.580 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.580 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.580 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-03-18T22:58:35.580 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.580 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.580 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.580 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_nvmeof": { 2024-03-18T22:58:35.580 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_nvmeof", 2024-03-18T22:58:35.580 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.580 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.580 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.581 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T22:58:35.581 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.581 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.581 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.581 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Nvme-of container image", 2024-03-18T22:58:35.581 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.581 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.581 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.581 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_prometheus": { 2024-03-18T22:58:35.581 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_prometheus", 2024-03-18T22:58:35.581 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.581 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.581 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.581 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T22:58:35.582 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.582 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.582 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.582 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-03-18T22:58:35.582 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.582 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.582 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.582 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_promtail": { 2024-03-18T22:58:35.582 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_promtail", 2024-03-18T22:58:35.582 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.582 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.582 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.583 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T22:58:35.583 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.583 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.583 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.583 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Promtail container image", 2024-03-18T22:58:35.583 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.583 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.583 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.583 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.583 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_snmp_gateway": { 2024-03-18T22:58:35.583 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_snmp_gateway", 2024-03-18T22:58:35.583 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.583 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.583 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.583 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SNMP Gateway container image", 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: "container_init": { 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_init", 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.584 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T22:58:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "daemon_cache_timeout": { 2024-03-18T22:58:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "daemon_cache_timeout", 2024-03-18T22:58:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-03-18T22:58:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.586 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T22:58:35.586 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.586 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.586 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.586 INFO:teuthology.orchestra.run.smithi040.stdout: "default_cephadm_command_timeout": { 2024-03-18T22:58:35.586 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T22:58:35.586 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.586 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.586 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.586 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "900", 2024-03-18T22:58:35.586 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.586 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.586 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.586 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T22:58:35.587 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.587 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.587 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.587 INFO:teuthology.orchestra.run.smithi040.stdout: "default_registry": { 2024-03-18T22:58:35.587 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_registry", 2024-03-18T22:58:35.587 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.587 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.587 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.587 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io", 2024-03-18T22:58:35.587 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.587 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.587 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.587 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T22:58:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: "device_cache_timeout": { 2024-03-18T22:58:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_cache_timeout", 2024-03-18T22:58:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1800", 2024-03-18T22:58:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.588 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache device inventory", 2024-03-18T22:58:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: "device_enhanced_scan": { 2024-03-18T22:58:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_enhanced_scan", 2024-03-18T22:58:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T22:58:35.589 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: "facts_cache_timeout": { 2024-03-18T22:58:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "facts_cache_timeout", 2024-03-18T22:58:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-03-18T22:58:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache host facts data", 2024-03-18T22:58:35.590 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: "host_check_interval": { 2024-03-18T22:58:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "host_check_interval", 2024-03-18T22:58:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-03-18T22:58:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to perform a host check", 2024-03-18T22:58:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.591 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: "hw_monitoring": { 2024-03-18T22:58:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hw_monitoring", 2024-03-18T22:58:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T22:58:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.592 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: "inventory_list_all": { 2024-03-18T22:58:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "inventory_list_all", 2024-03-18T22:58:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T22:58:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.594 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.594 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.594 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.594 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.594 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.594 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.594 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.594 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.594 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.594 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.594 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.594 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.594 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.594 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: "log_refresh_metadata": { 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_refresh_metadata", 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.595 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.596 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T22:58:35.596 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.596 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.596 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.596 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.596 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.596 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.596 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.596 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.596 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.596 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.596 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.596 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.596 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T22:58:35.597 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.597 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.597 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.597 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.597 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.597 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.597 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.597 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.597 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.597 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.597 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.597 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.597 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.597 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.598 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.598 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.598 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.598 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.598 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.598 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.598 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.598 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.598 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.598 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.598 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.598 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.599 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.599 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.599 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.599 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.599 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.599 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.599 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.599 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.599 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.599 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.599 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T22:58:35.599 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T22:58:35.599 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.599 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.600 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "*", 2024-03-18T22:58:35.601 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.601 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.601 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.601 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T22:58:35.601 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.601 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.601 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.601 INFO:teuthology.orchestra.run.smithi040.stdout: "max_count_per_host": { 2024-03-18T22:58:35.601 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_count_per_host", 2024-03-18T22:58:35.601 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.601 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.601 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.601 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.602 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.602 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.602 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.602 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of daemons per service per host", 2024-03-18T22:58:35.602 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.602 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.602 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.602 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd_draining_count": { 2024-03-18T22:58:35.602 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_osd_draining_count", 2024-03-18T22:58:35.602 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.602 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.602 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.603 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.603 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.603 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.603 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.603 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T22:58:35.603 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.603 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.603 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.603 INFO:teuthology.orchestra.run.smithi040.stdout: "migration_current": { 2024-03-18T22:58:35.603 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "migration_current", 2024-03-18T22:58:35.603 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.603 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.603 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.603 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.604 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.604 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.604 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.604 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "internal - do not modify", 2024-03-18T22:58:35.604 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.604 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.604 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.604 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-03-18T22:58:35.604 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-03-18T22:58:35.604 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.604 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.604 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.604 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "root", 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm-package", 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: "root" 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: "oob_default_addr": { 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "oob_default_addr", 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.605 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "169.254.1.1", 2024-03-18T22:58:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T22:58:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: "prometheus_alerts_path": { 2024-03-18T22:58:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus_alerts_path", 2024-03-18T22:58:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.606 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T22:58:35.607 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.607 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.607 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.607 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T22:58:35.607 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.607 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.607 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.607 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_insecure": { 2024-03-18T22:58:35.607 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_insecure", 2024-03-18T22:58:35.607 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.607 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.607 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.607 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.607 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.608 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.608 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.608 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T22:58:35.608 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.608 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.608 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.608 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_password": { 2024-03-18T22:58:35.608 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_password", 2024-03-18T22:58:35.608 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.608 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.608 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.608 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.608 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.609 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.609 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.609 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T22:58:35.609 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.609 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.609 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.609 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_url": { 2024-03-18T22:58:35.609 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_url", 2024-03-18T22:58:35.609 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.609 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.609 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.609 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.609 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.609 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.610 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.610 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T22:58:35.610 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.610 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.610 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.610 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_username": { 2024-03-18T22:58:35.610 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_username", 2024-03-18T22:58:35.610 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.610 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.610 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.610 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.610 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.610 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.610 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.611 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T22:58:35.611 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.611 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.611 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.611 INFO:teuthology.orchestra.run.smithi040.stdout: "secure_monitoring_stack": { 2024-03-18T22:58:35.611 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "secure_monitoring_stack", 2024-03-18T22:58:35.611 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.611 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.611 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.611 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.611 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.611 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.611 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.612 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T22:58:35.612 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.612 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.612 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.612 INFO:teuthology.orchestra.run.smithi040.stdout: "service_discovery_port": { 2024-03-18T22:58:35.612 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "service_discovery_port", 2024-03-18T22:58:35.612 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.612 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.612 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.612 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8765", 2024-03-18T22:58:35.612 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.612 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.612 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.612 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "cephadm service discovery port", 2024-03-18T22:58:35.613 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.613 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.613 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.613 INFO:teuthology.orchestra.run.smithi040.stdout: "ssh_config_file": { 2024-03-18T22:58:35.613 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssh_config_file", 2024-03-18T22:58:35.613 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.613 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.613 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.613 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.613 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.613 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.613 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.613 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T22:58:35.614 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.614 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.614 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.614 INFO:teuthology.orchestra.run.smithi040.stdout: "use_agent": { 2024-03-18T22:58:35.614 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_agent", 2024-03-18T22:58:35.614 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.614 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.614 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.614 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.614 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.614 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.614 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.614 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T22:58:35.615 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.615 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.615 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.615 INFO:teuthology.orchestra.run.smithi040.stdout: "use_repo_digest": { 2024-03-18T22:58:35.615 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_repo_digest", 2024-03-18T22:58:35.615 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.615 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.615 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.615 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.615 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.615 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.615 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.615 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T22:58:35.615 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.616 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.616 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.616 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_failed_host_check": { 2024-03-18T22:58:35.616 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_failed_host_check", 2024-03-18T22:58:35.616 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.616 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.616 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.616 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.616 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.616 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.616 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.616 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T22:58:35.616 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.616 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.617 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.617 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_daemons": { 2024-03-18T22:58:35.617 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_daemons", 2024-03-18T22:58:35.617 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.617 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.617 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.617 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.617 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.617 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.617 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.617 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T22:58:35.617 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.617 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.618 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.618 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_hosts": { 2024-03-18T22:58:35.618 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_hosts", 2024-03-18T22:58:35.618 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.618 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.618 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.618 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.618 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.618 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.618 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.618 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T22:58:35.618 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.618 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.619 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.619 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.619 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.619 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crash", 2024-03-18T22:58:35.619 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.619 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.619 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.619 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.619 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.619 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.620 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.621 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.621 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.621 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.621 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.621 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.621 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.621 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.621 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.621 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.621 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.621 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.621 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.621 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.622 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.622 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.622 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.622 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.622 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.622 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.622 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.622 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.622 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.622 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.622 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.622 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.623 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.623 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.623 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.623 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.623 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.623 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.623 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.623 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.623 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.623 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.623 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.623 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.624 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.624 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.624 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.624 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.624 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.624 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.624 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.624 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.624 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.624 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.624 INFO:teuthology.orchestra.run.smithi040.stdout: "retain_interval": { 2024-03-18T22:58:35.624 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retain_interval", 2024-03-18T22:58:35.624 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.624 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.625 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.625 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "31536000", 2024-03-18T22:58:35.625 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.625 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.625 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.625 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T22:58:35.625 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.625 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.625 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.625 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_recent_interval": { 2024-03-18T22:58:35.625 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_recent_interval", 2024-03-18T22:58:35.625 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.625 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.625 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.626 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1209600", 2024-03-18T22:58:35.626 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.626 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.626 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.626 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T22:58:35.626 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.626 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.626 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.626 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.626 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.626 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dashboard", 2024-03-18T22:58:35.626 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.627 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.627 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.627 INFO:teuthology.orchestra.run.smithi040.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T22:58:35.627 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T22:58:35.627 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.627 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.627 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.627 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.627 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.627 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.627 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.627 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.627 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.627 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.628 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.628 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T22:58:35.628 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T22:58:35.628 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.628 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.628 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.628 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.628 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.628 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.628 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.628 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.628 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.628 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.628 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.629 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T22:58:35.629 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T22:58:35.629 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.629 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.629 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.629 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.629 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.629 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.629 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.629 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.629 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.629 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.629 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.630 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_ENABLED": { 2024-03-18T22:58:35.630 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T22:58:35.630 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.630 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.630 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.630 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.630 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.630 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.630 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.630 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.630 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.630 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.630 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.631 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T22:58:35.631 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T22:58:35.631 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.631 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.631 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.631 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.631 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.631 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.631 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.631 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.631 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.631 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.631 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.631 INFO:teuthology.orchestra.run.smithi040.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T22:58:35.632 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T22:58:35.632 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.632 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.632 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.632 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.632 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.632 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.632 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.632 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.632 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.632 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.632 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.632 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T22:58:35.633 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T22:58:35.633 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.633 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.633 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.633 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.633 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.633 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.633 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.633 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.633 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.633 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.633 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.633 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T22:58:35.633 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T22:58:35.634 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.634 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.634 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.634 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.634 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.634 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.634 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.634 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.634 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.634 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.634 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.634 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T22:58:35.635 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T22:58:35.635 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.635 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.635 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.635 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.635 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.635 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.635 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.635 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.635 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.635 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.635 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.635 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T22:58:35.635 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T22:58:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T22:58:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T22:58:35.636 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T22:58:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T22:58:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.637 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.638 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.638 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.638 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.638 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.638 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.638 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.638 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.638 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.638 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.638 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T22:58:35.638 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T22:58:35.638 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.639 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.639 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.639 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.639 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.639 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.639 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.639 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.639 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.639 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.639 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.639 INFO:teuthology.orchestra.run.smithi040.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T22:58:35.639 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T22:58:35.639 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.639 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.640 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-03-18T22:58:35.641 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.641 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.641 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.641 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.641 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.641 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.641 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.641 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T22:58:35.641 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T22:58:35.641 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.641 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.641 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.641 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.641 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.642 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.642 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.642 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.642 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.642 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.642 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.642 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_URL": { 2024-03-18T22:58:35.642 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_URL", 2024-03-18T22:58:35.642 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.642 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.642 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.642 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.642 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.643 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.643 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.643 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.643 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.643 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.643 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.643 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T22:58:35.643 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T22:58:35.643 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.643 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.643 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.643 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-03-18T22:58:35.643 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.644 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.644 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.644 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.644 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.644 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.644 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.644 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T22:58:35.644 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T22:58:35.644 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.644 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.644 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.644 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.644 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.644 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.645 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.645 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.645 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.645 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.645 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.645 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T22:58:35.645 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T22:58:35.645 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.645 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.645 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.645 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.645 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.645 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.645 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.646 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.646 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.646 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.646 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.646 INFO:teuthology.orchestra.run.smithi040.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T22:58:35.646 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T22:58:35.646 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.646 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.646 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.646 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.646 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.646 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.647 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.647 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.647 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.647 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.647 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.647 INFO:teuthology.orchestra.run.smithi040.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T22:58:35.647 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T22:58:35.647 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.647 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.647 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.647 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.647 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.647 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.647 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.648 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.648 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.648 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.648 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.648 INFO:teuthology.orchestra.run.smithi040.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T22:58:35.648 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T22:58:35.648 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.648 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.648 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.648 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "{}", 2024-03-18T22:58:35.648 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.649 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.649 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.649 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.649 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.649 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.649 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.649 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T22:58:35.649 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T22:58:35.649 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.649 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.649 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.649 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.649 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.649 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.650 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.650 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.650 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.650 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.650 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.650 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T22:58:35.650 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T22:58:35.650 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.650 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.650 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.650 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.650 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.650 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.651 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.652 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.652 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.652 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.652 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T22:58:35.652 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T22:58:35.652 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.652 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.652 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.652 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.652 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.652 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.652 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.652 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.653 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.653 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.653 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.653 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T22:58:35.653 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T22:58:35.653 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.653 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.653 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.653 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.653 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.653 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.653 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.653 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.654 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.654 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.654 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.654 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T22:58:35.654 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T22:58:35.654 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.654 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.654 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.654 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.654 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.654 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.654 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.654 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.655 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.655 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.655 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.655 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T22:58:35.655 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T22:58:35.655 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.655 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.655 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.655 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.655 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.655 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.655 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.655 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.655 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.656 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.656 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.656 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T22:58:35.656 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T22:58:35.656 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.656 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.656 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.656 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.656 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.656 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.656 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.656 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.656 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.657 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.657 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.657 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T22:58:35.657 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T22:58:35.657 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.657 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.657 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.657 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.657 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.657 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.657 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.657 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.657 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.657 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.658 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.658 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T22:58:35.658 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T22:58:35.658 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.658 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.658 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.658 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.658 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.658 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.658 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.658 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.658 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.658 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.659 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.659 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T22:58:35.659 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T22:58:35.659 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.659 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.659 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.659 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T22:58:35.659 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.659 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.659 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.659 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.659 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.659 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.659 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.660 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T22:58:35.660 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T22:58:35.660 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.660 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.660 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.660 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.660 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.660 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.660 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.660 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.660 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.660 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.660 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.661 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T22:58:35.661 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T22:58:35.661 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.661 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.661 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.661 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8", 2024-03-18T22:58:35.661 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.661 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.661 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.661 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.661 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.661 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.661 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.662 INFO:teuthology.orchestra.run.smithi040.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T22:58:35.662 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T22:58:35.662 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.662 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.662 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.662 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "45", 2024-03-18T22:58:35.662 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.662 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.662 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.662 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.662 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.662 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.662 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.663 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T22:58:35.663 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T22:58:35.663 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.663 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.663 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.663 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.663 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.663 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.663 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.663 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.663 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.663 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.663 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.664 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T22:58:35.664 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T22:58:35.664 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.664 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.664 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.664 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-03-18T22:58:35.664 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.664 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.664 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.664 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.664 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.664 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.664 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.665 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T22:58:35.665 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T22:58:35.665 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.665 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.665 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.665 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.665 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.665 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.665 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.665 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.665 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.665 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.665 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.665 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T22:58:35.666 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T22:58:35.666 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.666 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.666 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.666 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.666 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.666 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.666 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.666 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.666 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.666 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.666 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.667 INFO:teuthology.orchestra.run.smithi040.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T22:58:35.667 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T22:58:35.667 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.667 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.667 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.667 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.667 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.667 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.667 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.667 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.667 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.667 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.667 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.667 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T22:58:35.668 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T22:58:35.668 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.668 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.668 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.668 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-03-18T22:58:35.668 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.668 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.668 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.668 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.668 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.668 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.668 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.669 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T22:58:35.669 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T22:58:35.669 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.669 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.669 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.669 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.669 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.669 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.669 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.669 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.669 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.669 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.669 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.669 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T22:58:35.670 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T22:58:35.670 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.670 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.670 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.670 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-03-18T22:58:35.670 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.670 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.670 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.670 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.670 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.670 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.670 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.670 INFO:teuthology.orchestra.run.smithi040.stdout: "cross_origin_url": { 2024-03-18T22:58:35.671 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cross_origin_url", 2024-03-18T22:58:35.671 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.671 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.671 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.671 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.671 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.671 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.671 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.671 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.671 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.671 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.671 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.671 INFO:teuthology.orchestra.run.smithi040.stdout: "crt_file": { 2024-03-18T22:58:35.671 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crt_file", 2024-03-18T22:58:35.672 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.672 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.672 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.672 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.672 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.672 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.672 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.672 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.672 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.672 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.672 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.672 INFO:teuthology.orchestra.run.smithi040.stdout: "debug": { 2024-03-18T22:58:35.672 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "debug", 2024-03-18T22:58:35.672 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.673 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.673 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.673 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.673 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.673 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.673 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.673 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable/disable debug options", 2024-03-18T22:58:35.673 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.673 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.673 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.673 INFO:teuthology.orchestra.run.smithi040.stdout: "jwt_token_ttl": { 2024-03-18T22:58:35.673 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "jwt_token_ttl", 2024-03-18T22:58:35.673 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.673 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.674 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.674 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "28800", 2024-03-18T22:58:35.674 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.674 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.674 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.674 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.674 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.674 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.674 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.674 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-03-18T22:58:35.674 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-03-18T22:58:35.674 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.675 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.675 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.675 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.675 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.675 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.675 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.675 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.675 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.675 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.675 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.675 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.675 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.675 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.675 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.676 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.676 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.676 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.676 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.676 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.676 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.676 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.676 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.676 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.676 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.676 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.676 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.676 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.676 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.676 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.677 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.677 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.677 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.677 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.677 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.677 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.677 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.677 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.677 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.677 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.677 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.677 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.677 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.677 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.678 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.678 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.678 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.678 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.678 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.678 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.678 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.678 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.678 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.678 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.678 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.678 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.679 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.680 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.680 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.680 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.680 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.680 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.680 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.680 INFO:teuthology.orchestra.run.smithi040.stdout: "motd": { 2024-03-18T22:58:35.680 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "motd", 2024-03-18T22:58:35.680 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.680 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.680 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.680 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.680 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.680 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.681 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.681 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "The message of the day", 2024-03-18T22:58:35.681 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.681 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.681 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.681 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect_resolve_ip_addr": { 2024-03-18T22:58:35.681 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T22:58:35.681 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.681 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.681 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.681 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.681 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.681 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.681 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.682 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.682 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.682 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.682 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.682 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-03-18T22:58:35.682 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-03-18T22:58:35.682 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.682 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.682 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.682 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-03-18T22:58:35.682 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.682 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.682 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-03-18T22:58:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-03-18T22:58:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8080", 2024-03-18T22:58:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.683 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-03-18T22:58:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-03-18T22:58:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.684 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl_server_port": { 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl_server_port", 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8443", 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.685 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.686 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.686 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-03-18T22:58:35.686 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-03-18T22:58:35.686 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.686 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.686 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.686 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "redirect", 2024-03-18T22:58:35.686 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.686 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.686 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.686 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.686 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect" 2024-03-18T22:58:35.686 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.687 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.687 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.687 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.687 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.687 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-03-18T22:58:35.687 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-03-18T22:58:35.687 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.687 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.687 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.687 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-03-18T22:58:35.687 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-03-18T22:58:35.687 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-03-18T22:58:35.688 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.688 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.688 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.688 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.688 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.688 INFO:teuthology.orchestra.run.smithi040.stdout: "url_prefix": { 2024-03-18T22:58:35.688 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url_prefix", 2024-03-18T22:58:35.688 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.688 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.688 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.688 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.688 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.688 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.688 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "devicehealth", 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_monitoring": { 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_monitoring", 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.689 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.690 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.690 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.690 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.690 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.690 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.690 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "monitor device health metrics", 2024-03-18T22:58:35.690 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.690 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.690 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.690 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.690 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.690 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.691 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.693 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.693 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.693 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.693 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.693 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.693 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.693 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.693 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.693 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.693 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.693 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.693 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.693 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.693 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.693 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.694 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.694 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.694 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.694 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.694 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.694 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.694 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.694 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.694 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.694 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.694 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.694 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: "mark_out_threshold": { 2024-03-18T22:58:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mark_out_threshold", 2024-03-18T22:58:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2419200", 2024-03-18T22:58:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.695 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": { 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_name", 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "device_health_metrics", 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.696 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T22:58:35.697 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.697 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.697 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.697 INFO:teuthology.orchestra.run.smithi040.stdout: "retention_period": { 2024-03-18T22:58:35.697 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retention_period", 2024-03-18T22:58:35.697 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.697 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.697 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.697 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15552000", 2024-03-18T22:58:35.697 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.697 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.697 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.697 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain device health metrics", 2024-03-18T22:58:35.697 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.698 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.698 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.698 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_frequency": { 2024-03-18T22:58:35.698 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_frequency", 2024-03-18T22:58:35.698 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.698 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.698 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.698 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-03-18T22:58:35.698 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.698 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.698 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.698 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: "self_heal": { 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "self_heal", 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.699 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.700 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.700 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-03-18T22:58:35.700 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-03-18T22:58:35.700 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.700 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.700 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.700 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-03-18T22:58:35.700 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.700 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.700 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.700 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T22:58:35.700 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.700 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_threshold": { 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_threshold", 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7257600", 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.701 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.702 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.702 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "diskprediction_local", 2024-03-18T22:58:35.702 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.702 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.702 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.702 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.702 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.702 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.702 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.702 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.702 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.702 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.703 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.703 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.703 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.703 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.703 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.703 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.703 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.703 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.703 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.703 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.703 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.703 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.703 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.703 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.704 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.704 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.704 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.704 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.704 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.704 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.704 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.704 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.704 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.704 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.704 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.704 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.704 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.704 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.705 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.705 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.705 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.705 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.705 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.705 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.705 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.705 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.705 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.705 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.705 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.705 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.705 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.705 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.705 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.706 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.706 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.706 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.706 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.706 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.706 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.706 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.706 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.706 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.706 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.706 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.706 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.706 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.706 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.707 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.707 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.707 INFO:teuthology.orchestra.run.smithi040.stdout: "predict_interval": { 2024-03-18T22:58:35.707 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predict_interval", 2024-03-18T22:58:35.707 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.707 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.707 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.707 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-03-18T22:58:35.707 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.707 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.707 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.707 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.707 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.708 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.708 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.708 INFO:teuthology.orchestra.run.smithi040.stdout: "predictor_model": { 2024-03-18T22:58:35.708 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predictor_model", 2024-03-18T22:58:35.708 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.708 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.708 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.708 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "prophetstor", 2024-03-18T22:58:35.708 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.708 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.708 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.708 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.708 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.709 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.710 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.710 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.710 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "influx", 2024-03-18T22:58:35.710 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": false, 2024-03-18T22:58:35.710 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "influxdb python module not found", 2024-03-18T22:58:35.710 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.710 INFO:teuthology.orchestra.run.smithi040.stdout: "batch_size": { 2024-03-18T22:58:35.710 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "batch_size", 2024-03-18T22:58:35.710 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.710 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.710 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5000", 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: "database": { 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "database", 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "ceph", 2024-03-18T22:58:35.711 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T22:58:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": { 2024-03-18T22:58:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hostname", 2024-03-18T22:58:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server hostname", 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "30", 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "5", 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.713 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T22:58:35.714 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.714 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.714 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.714 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.714 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.714 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.714 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.714 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.714 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.714 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.714 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.714 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.715 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.715 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.715 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.715 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.715 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.715 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.715 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.715 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.715 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.715 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.715 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.715 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.715 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.715 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.716 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.716 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.716 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.716 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.716 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.716 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.716 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.716 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.716 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.716 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.716 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.716 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.716 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.716 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.717 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.717 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.717 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.717 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.717 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.717 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.717 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.717 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.717 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.717 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.717 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.717 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.717 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.717 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.718 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.718 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.718 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.718 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.718 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.718 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.718 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.718 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.718 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.718 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.718 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.718 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.718 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.719 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.719 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.719 INFO:teuthology.orchestra.run.smithi040.stdout: "password": { 2024-03-18T22:58:35.719 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "password", 2024-03-18T22:58:35.719 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.719 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.719 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.719 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.719 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.719 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.719 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.719 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "password of InfluxDB server user", 2024-03-18T22:58:35.719 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.719 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.720 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.720 INFO:teuthology.orchestra.run.smithi040.stdout: "port": { 2024-03-18T22:58:35.720 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "port", 2024-03-18T22:58:35.720 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.720 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.720 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.720 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8086", 2024-03-18T22:58:35.720 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.720 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.720 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.720 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server port", 2024-03-18T22:58:35.720 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.720 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.720 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "false", 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.721 INFO:teuthology.orchestra.run.smithi040.stdout: "threads": { 2024-03-18T22:58:35.722 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threads", 2024-03-18T22:58:35.722 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.722 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.722 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.722 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-03-18T22:58:35.722 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-03-18T22:58:35.722 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "32", 2024-03-18T22:58:35.722 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.722 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T22:58:35.722 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.722 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.722 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.722 INFO:teuthology.orchestra.run.smithi040.stdout: "username": { 2024-03-18T22:58:35.722 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "username", 2024-03-18T22:58:35.723 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.723 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.723 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.723 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.723 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.723 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.723 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.723 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "username of InfluxDB server user", 2024-03-18T22:58:35.723 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.723 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.723 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.723 INFO:teuthology.orchestra.run.smithi040.stdout: "verify_ssl": { 2024-03-18T22:58:35.723 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "verify_ssl", 2024-03-18T22:58:35.723 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.724 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.724 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.724 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "true", 2024-03-18T22:58:35.724 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.724 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.724 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.724 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T22:58:35.724 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.724 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.724 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.724 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.724 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.724 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "insights", 2024-03-18T22:58:35.725 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.725 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.725 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.725 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.725 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.725 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.725 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.725 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.725 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.725 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.725 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.725 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.725 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.725 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.726 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.727 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.728 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.729 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.729 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.729 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.729 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.729 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.729 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.729 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.729 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.729 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.729 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.729 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.729 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.729 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "iostat", 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.730 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.731 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.732 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.733 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.733 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.733 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.733 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.733 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.733 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.733 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.733 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.733 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.733 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.733 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.733 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.733 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.733 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.733 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.734 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.735 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.735 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "k8sevents", 2024-03-18T22:58:35.735 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.735 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.735 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.735 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_event_retention_days": { 2024-03-18T22:58:35.735 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ceph_event_retention_days", 2024-03-18T22:58:35.735 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.735 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.735 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.735 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7", 2024-03-18T22:58:35.735 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.736 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.736 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.736 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T22:58:35.736 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.736 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.736 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.736 INFO:teuthology.orchestra.run.smithi040.stdout: "config_check_secs": { 2024-03-18T22:58:35.736 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_check_secs", 2024-03-18T22:58:35.736 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.736 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.736 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.736 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.736 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "10", 2024-03-18T22:58:35.736 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.737 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.738 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.738 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.738 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.738 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.738 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.738 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.738 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.738 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.738 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.738 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.738 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.738 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.738 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.739 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.740 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.741 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "localpool", 2024-03-18T22:58:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: "failure_domain": { 2024-03-18T22:58:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "failure_domain", 2024-03-18T22:58:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.742 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "host", 2024-03-18T22:58:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "failure domain for any created local pool", 2024-03-18T22:58:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T22:58:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.743 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.744 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.745 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.746 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.747 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": { 2024-03-18T22:58:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_size", 2024-03-18T22:58:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.748 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default min_size for any created local pool", 2024-03-18T22:58:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T22:58:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: "num_rep": { 2024-03-18T22:58:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "num_rep", 2024-03-18T22:58:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3", 2024-03-18T22:58:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.749 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default replica count for any created local pool", 2024-03-18T22:58:35.750 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.750 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.750 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.750 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": { 2024-03-18T22:58:35.750 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_num", 2024-03-18T22:58:35.750 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.750 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.750 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.754 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "128", 2024-03-18T22:58:35.754 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.754 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.754 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.754 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T22:58:35.754 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.754 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.755 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.755 INFO:teuthology.orchestra.run.smithi040.stdout: "prefix": { 2024-03-18T22:58:35.755 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prefix", 2024-03-18T22:58:35.755 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.755 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.755 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.755 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.755 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.755 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.755 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.755 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name prefix for any created local pool", 2024-03-18T22:58:35.755 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.755 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.756 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.756 INFO:teuthology.orchestra.run.smithi040.stdout: "subtree": { 2024-03-18T22:58:35.756 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "subtree", 2024-03-18T22:58:35.756 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.756 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.756 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.756 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "rack", 2024-03-18T22:58:35.756 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.756 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.756 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.756 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T22:58:35.756 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T22:58:35.756 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mds_autoscaler", 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.757 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.758 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.758 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.758 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.758 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.758 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.758 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.758 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.758 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.758 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.758 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.758 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.758 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.758 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.758 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.759 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.759 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.759 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.759 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.759 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.759 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.759 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.759 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.759 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.759 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.759 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.759 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.759 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.759 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.759 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.760 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.761 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.762 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.762 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.762 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.762 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.762 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.762 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirroring", 2024-03-18T22:58:35.762 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.762 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.762 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.762 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.762 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.762 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.762 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.762 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.764 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.764 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.764 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.764 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.764 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.764 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.764 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.764 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.764 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.764 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.764 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.764 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.764 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.764 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.765 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.766 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.766 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.766 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.766 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.766 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.766 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.766 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.766 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.766 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.766 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.766 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.766 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.767 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.767 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.767 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.767 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.767 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.767 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.767 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.767 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.767 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "nfs", 2024-03-18T22:58:35.767 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.767 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.767 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.767 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.767 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.768 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.768 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.768 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.768 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.768 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.768 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.768 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.768 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.768 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.768 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.768 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.768 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.768 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.768 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.768 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.769 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.770 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "fail_fs": { 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "fail_fs", 2024-03-18T22:58:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator": { 2024-03-18T22:58:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-03-18T22:58:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm", 2024-03-18T22:58:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "rook", 2024-03-18T22:58:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "test_orchestrator" 2024-03-18T22:58:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Orchestrator backend", 2024-03-18T22:58:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_perf_query", 2024-03-18T22:58:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_support", 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_autoscaler", 2024-03-18T22:58:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-03-18T22:58:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "threshold": { 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threshold", 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-03-18T22:58:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1.0", 2024-03-18T22:58:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "scaling threshold", 2024-03-18T22:58:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T22:58:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "progress", 2024-03-18T22:58:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_pg_recovery_event": { 2024-03-18T22:58:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_pg_recovery_event", 2024-03-18T22:58:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T22:58:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-03-18T22:58:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-03-18T22:58:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "max_completed_events": { 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_completed_events", 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "50", 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "number of past completed events to remember", 2024-03-18T22:58:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long the module is going to sleep", 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus", 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "cache": { 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cache", 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "exclude_perf_counters": { 2024-03-18T22:58:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "exclude_perf_counters", 2024-03-18T22:58:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T22:58:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T22:58:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools": { 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools", 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T22:58:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "300", 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_interval": { 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_interval", 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15.0", 2024-03-18T22:58:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-03-18T22:58:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-03-18T22:58:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-03-18T22:58:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T22:58:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-03-18T22:58:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-03-18T22:58:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "9283", 2024-03-18T22:58:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "stale_cache_strategy": { 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stale_cache_strategy", 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "log", 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-03-18T22:58:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-03-18T22:58:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "default", 2024-03-18T22:58:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "default", 2024-03-18T22:58:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "error" 2024-03-18T22:58:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-03-18T22:58:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-03-18T22:58:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-03-18T22:58:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-03-18T22:58:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-03-18T22:58:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_support", 2024-03-18T22:58:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_snap_create": { 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_snap_create", 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "mirror_snapshot_schedule": { 2024-03-18T22:58:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T22:58:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "trash_purge_schedule": { 2024-03-18T22:58:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "trash_purge_schedule", 2024-03-18T22:58:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "restful", 2024-03-18T22:58:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_auth": { 2024-03-18T22:58:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_auth", 2024-03-18T22:58:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-03-18T22:58:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-03-18T22:58:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-03-18T22:58:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-03-18T22:58:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-03-18T22:58:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-03-18T22:58:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rgw", 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rook", 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "storage_class": { 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "storage_class", 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "local", 2024-03-18T22:58:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T22:58:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "selftest", 2024-03-18T22:58:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "roption1": { 2024-03-18T22:58:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption1", 2024-03-18T22:58:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "roption2": { 2024-03-18T22:58:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption2", 2024-03-18T22:58:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "xyz", 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption1": { 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption1", 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption2": { 2024-03-18T22:58:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption2", 2024-03-18T22:58:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption3": { 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption3", 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption4": { 2024-03-18T22:58:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption4", 2024-03-18T22:58:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption5": { 2024-03-18T22:58:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption5", 2024-03-18T22:58:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.858 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.858 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.858 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.858 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.858 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption6": { 2024-03-18T22:58:35.858 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption6", 2024-03-18T22:58:35.858 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.858 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.858 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.858 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.858 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.858 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.859 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.859 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.859 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.859 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.859 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.859 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption7": { 2024-03-18T22:58:35.859 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption7", 2024-03-18T22:58:35.859 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.859 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.859 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.859 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.859 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-03-18T22:58:35.859 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "42", 2024-03-18T22:58:35.859 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.860 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.860 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.860 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.860 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.860 INFO:teuthology.orchestra.run.smithi040.stdout: "testkey": { 2024-03-18T22:58:35.860 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testkey", 2024-03-18T22:58:35.860 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.860 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.860 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.860 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.860 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.860 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.860 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.861 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.861 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.861 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.861 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.861 INFO:teuthology.orchestra.run.smithi040.stdout: "testlkey": { 2024-03-18T22:58:35.861 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testlkey", 2024-03-18T22:58:35.861 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.861 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.861 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.861 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.861 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.861 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.861 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.861 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.862 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.862 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.862 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.862 INFO:teuthology.orchestra.run.smithi040.stdout: "testnewline": { 2024-03-18T22:58:35.862 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testnewline", 2024-03-18T22:58:35.862 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.862 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.862 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.862 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.862 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.862 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.862 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.862 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.863 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.863 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.863 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.863 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.863 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.863 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snap_schedule", 2024-03-18T22:58:35.863 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.863 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.863 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.863 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_m_granularity": { 2024-03-18T22:58:35.863 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_m_granularity", 2024-03-18T22:58:35.863 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.863 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.864 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.864 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.864 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.864 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.864 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.864 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T22:58:35.864 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.864 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.864 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.864 INFO:teuthology.orchestra.run.smithi040.stdout: "dump_on_update": { 2024-03-18T22:58:35.864 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dump_on_update", 2024-03-18T22:58:35.864 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.864 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.864 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.865 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.865 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.865 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.865 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.865 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "dump database to debug log on update", 2024-03-18T22:58:35.865 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.865 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.865 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.865 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.865 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.865 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.865 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.865 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.865 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.866 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.866 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.866 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.866 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.866 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.866 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.866 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.866 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.866 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.866 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.866 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.866 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.866 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.866 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.868 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.868 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.868 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.868 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.868 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.868 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.868 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.868 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.868 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.868 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.868 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.868 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.868 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.868 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.869 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.870 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.870 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.870 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.870 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.870 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.870 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.870 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stats", 2024-03-18T22:58:35.870 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.870 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.870 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.870 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.870 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.870 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.871 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.873 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.873 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.873 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.873 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.873 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.873 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.873 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.873 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.873 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.873 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.873 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.873 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.873 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.873 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.873 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.874 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "status", 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.875 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.876 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.877 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.877 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.877 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.877 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.877 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.877 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.877 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.877 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.877 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.877 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.877 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.877 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.877 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.878 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.878 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.878 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.878 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.878 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.878 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.878 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.878 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.878 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.878 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.878 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.878 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.878 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.879 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.879 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.879 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.879 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.879 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.879 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.879 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.879 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.879 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.879 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.879 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.879 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.879 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.879 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.879 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telegraf", 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.880 INFO:teuthology.orchestra.run.smithi040.stdout: "address": { 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "address", 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-03-18T22:58:35.881 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.882 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.882 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.882 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15", 2024-03-18T22:58:35.882 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.882 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.882 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.882 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.882 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.882 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.882 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.882 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.882 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.882 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.883 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.884 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.884 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.884 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.884 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.884 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.884 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.884 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.884 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.884 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.884 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.884 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.884 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.884 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.884 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.885 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.886 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.886 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.886 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.886 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.886 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.886 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.886 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.886 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.886 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.886 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.886 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.886 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.886 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.886 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.887 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.887 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.887 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.887 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.887 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.887 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.887 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.887 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.887 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telemetry", 2024-03-18T22:58:35.887 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.887 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.887 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.887 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_basic": { 2024-03-18T22:58:35.887 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_basic", 2024-03-18T22:58:35.888 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.888 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.888 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.888 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.888 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.888 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.888 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.888 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T22:58:35.888 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.888 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.888 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.888 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_crash": { 2024-03-18T22:58:35.889 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_crash", 2024-03-18T22:58:35.889 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.889 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.889 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.889 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.889 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.889 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.889 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.889 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T22:58:35.889 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.889 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.889 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.889 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.889 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_device": { 2024-03-18T22:58:35.889 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_device", 2024-03-18T22:58:35.890 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.890 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.890 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.890 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.890 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.890 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.890 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.890 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T22:58:35.890 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.890 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.890 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.890 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.890 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_ident": { 2024-03-18T22:58:35.891 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_ident", 2024-03-18T22:58:35.891 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.891 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.891 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.891 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.891 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.891 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.891 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.891 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T22:58:35.891 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.891 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.891 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.891 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_perf": { 2024-03-18T22:58:35.891 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_perf", 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: "contact": { 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "contact", 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.892 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.893 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.893 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.893 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.893 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.893 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.893 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.893 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.893 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.893 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.893 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.893 INFO:teuthology.orchestra.run.smithi040.stdout: "description": { 2024-03-18T22:58:35.893 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "description", 2024-03-18T22:58:35.893 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.893 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.894 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.894 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.894 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.894 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.894 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.894 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.894 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.894 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.894 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.894 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.894 INFO:teuthology.orchestra.run.smithi040.stdout: "device_url": { 2024-03-18T22:58:35.894 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_url", 2024-03-18T22:58:35.894 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.894 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.894 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.895 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T22:58:35.895 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.895 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.895 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.895 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.895 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.895 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.895 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.895 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.895 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-03-18T22:58:35.895 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-03-18T22:58:35.895 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.895 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.895 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.895 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.896 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.896 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.896 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.896 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.896 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.896 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.896 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.896 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.896 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-03-18T22:58:35.896 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-03-18T22:58:35.896 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.896 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.896 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.896 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "24", 2024-03-18T22:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "8", 2024-03-18T22:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout: "last_opt_revision": { 2024-03-18T22:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "last_opt_revision", 2024-03-18T22:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.897 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1", 2024-03-18T22:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard": { 2024-03-18T22:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard", 2024-03-18T22:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.898 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.899 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.899 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.899 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.899 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.899 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.899 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.899 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.899 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard_description": { 2024-03-18T22:58:35.899 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard_description", 2024-03-18T22:58:35.899 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.899 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.899 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.899 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.899 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.900 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.901 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.902 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.902 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.902 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.902 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.902 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.902 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.902 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.902 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.902 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.902 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.902 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.902 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.902 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.902 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.903 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.904 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.904 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.904 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.904 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.904 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.904 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.904 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.904 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.904 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.904 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.904 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.904 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.904 INFO:teuthology.orchestra.run.smithi040.stdout: "organization": { 2024-03-18T22:58:35.904 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "organization", 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: "proxy": { 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "proxy", 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.905 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: "url": { 2024-03-18T22:58:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url", 2024-03-18T22:58:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.906 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.907 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T22:58:35.907 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.907 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.907 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.907 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.907 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.907 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.907 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.907 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.907 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.907 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.907 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.907 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "test_orchestrator", 2024-03-18T22:58:35.907 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.907 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.908 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.908 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.908 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.908 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.908 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.908 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.908 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.908 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.908 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.908 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.908 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.908 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.908 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.908 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.908 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.909 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.910 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.910 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.910 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.910 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.910 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.910 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.910 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.910 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.910 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.910 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.910 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.910 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.910 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.911 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "volumes", 2024-03-18T22:58:35.912 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.913 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.914 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.914 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.914 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.914 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.914 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.914 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.914 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.914 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.914 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.914 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.914 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.914 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.914 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.914 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.915 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.916 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.917 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.917 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.917 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.917 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.917 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.917 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.917 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.917 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.917 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.917 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_clones": { 2024-03-18T22:58:35.917 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_clones", 2024-03-18T22:58:35.917 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.917 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.917 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.918 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4", 2024-03-18T22:58:35.918 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.918 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.918 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.918 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T22:58:35.918 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.918 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.918 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.918 INFO:teuthology.orchestra.run.smithi040.stdout: "periodic_async_work": { 2024-03-18T22:58:35.918 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "periodic_async_work", 2024-03-18T22:58:35.918 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.918 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.918 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.919 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.919 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.919 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.919 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.919 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Periodically check for async work", 2024-03-18T22:58:35.919 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.919 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.919 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.919 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_delay": { 2024-03-18T22:58:35.919 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_delay", 2024-03-18T22:58:35.919 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.919 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.919 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.919 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-03-18T22:58:35.920 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.920 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.920 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.920 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T22:58:35.920 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.920 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.920 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.920 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.920 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_no_wait": { 2024-03-18T22:58:35.920 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T22:58:35.920 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.920 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.920 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.920 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.921 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.921 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.921 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.921 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T22:58:35.921 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.921 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.921 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.921 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.921 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.921 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.921 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.921 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix", 2024-03-18T22:58:35.921 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.921 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.922 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.922 INFO:teuthology.orchestra.run.smithi040.stdout: "discovery_interval": { 2024-03-18T22:58:35.922 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "discovery_interval", 2024-03-18T22:58:35.922 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-03-18T22:58:35.922 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.922 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.922 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "100", 2024-03-18T22:58:35.922 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.922 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.922 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.922 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.922 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.922 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.922 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: "identifier": { 2024-03-18T22:58:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "identifier", 2024-03-18T22:58:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.923 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.924 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.925 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.926 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.926 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.926 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.926 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.926 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.926 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.926 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.926 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.926 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.926 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.926 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.926 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.926 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.927 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.928 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.928 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.928 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.928 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.928 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.928 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.928 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.928 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.928 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.928 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.928 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.928 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.928 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_host": { 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_host", 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.929 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.930 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.930 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.930 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.930 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_port": { 2024-03-18T22:58:35.930 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_port", 2024-03-18T22:58:35.930 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.930 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.930 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.930 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10051", 2024-03-18T22:58:35.930 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.930 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.930 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.930 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_sender": { 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_sender", 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.931 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.932 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.932 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.932 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.932 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.932 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-18T22:58:35.932 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.932 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.932 INFO:teuthology.orchestra.run.smithi040.stdout: "modules": [ 2024-03-18T22:58:35.932 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm", 2024-03-18T22:58:35.932 INFO:teuthology.orchestra.run.smithi040.stdout: "dashboard", 2024-03-18T22:58:35.932 INFO:teuthology.orchestra.run.smithi040.stdout: "iostat", 2024-03-18T22:58:35.932 INFO:teuthology.orchestra.run.smithi040.stdout: "nfs", 2024-03-18T22:58:35.932 INFO:teuthology.orchestra.run.smithi040.stdout: "restful" 2024-03-18T22:58:35.932 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.932 INFO:teuthology.orchestra.run.smithi040.stdout: "available_modules": [ 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "alerts", 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T22:58:35.933 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.934 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.934 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.934 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.934 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.934 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.934 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.934 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.934 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.934 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.934 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.934 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.934 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.934 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.935 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.936 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.937 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.937 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.937 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.937 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.937 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.937 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.937 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.937 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.937 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.937 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.937 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.937 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.937 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.937 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.938 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.938 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.938 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.938 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.938 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.938 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.938 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.938 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.938 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.938 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.938 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_destination": { 2024-03-18T22:58:35.938 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_destination", 2024-03-18T22:58:35.939 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.939 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.939 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.939 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.939 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.939 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.939 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.939 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email address to send alerts to", 2024-03-18T22:58:35.939 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.939 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.939 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.939 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_from_name": { 2024-03-18T22:58:35.939 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_from_name", 2024-03-18T22:58:35.939 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.940 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.940 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.940 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "Ceph", 2024-03-18T22:58:35.940 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.940 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.940 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.940 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Email From: name", 2024-03-18T22:58:35.940 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.940 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.940 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.940 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_host": { 2024-03-18T22:58:35.940 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_host", 2024-03-18T22:58:35.940 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.940 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.941 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.941 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.941 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.941 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.941 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.941 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP server", 2024-03-18T22:58:35.941 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.941 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.941 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.941 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_password": { 2024-03-18T22:58:35.941 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_password", 2024-03-18T22:58:35.941 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.941 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.941 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.942 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.942 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.942 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.942 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.942 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Password to authenticate with", 2024-03-18T22:58:35.942 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.942 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.942 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.942 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_port": { 2024-03-18T22:58:35.942 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_port", 2024-03-18T22:58:35.942 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.942 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.943 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.943 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "465", 2024-03-18T22:58:35.943 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.943 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.943 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.943 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP port", 2024-03-18T22:58:35.943 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.943 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.943 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.943 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_sender": { 2024-03-18T22:58:35.943 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_sender", 2024-03-18T22:58:35.943 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.943 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.944 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.944 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.944 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.944 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.944 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.944 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SMTP envelope sender", 2024-03-18T22:58:35.944 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.944 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.944 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.944 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_ssl": { 2024-03-18T22:58:35.944 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_ssl", 2024-03-18T22:58:35.944 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.944 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.944 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.945 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.945 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.945 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.945 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.945 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T22:58:35.945 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.945 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.945 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.945 INFO:teuthology.orchestra.run.smithi040.stdout: "smtp_user": { 2024-03-18T22:58:35.945 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "smtp_user", 2024-03-18T22:58:35.945 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.945 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.945 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.945 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.946 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.946 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.946 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.946 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "User to authenticate as", 2024-03-18T22:58:35.946 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.946 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.946 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.946 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.946 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.946 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.946 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "balancer", 2024-03-18T22:58:35.946 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.946 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.946 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: "active": { 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "active", 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_time": { 2024-03-18T22:58:35.947 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_time", 2024-03-18T22:58:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0000", 2024-03-18T22:58:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T22:58:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T22:58:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: "begin_weekday": { 2024-03-18T22:58:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "begin_weekday", 2024-03-18T22:58:35.948 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-03-18T22:58:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-03-18T22:58:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-03-18T22:58:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-03-18T22:58:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T22:58:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T22:58:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_max_iterations": { 2024-03-18T22:58:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_max_iterations", 2024-03-18T22:58:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-03-18T22:58:35.949 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.950 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.950 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "25", 2024-03-18T22:58:35.950 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-03-18T22:58:35.950 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "250", 2024-03-18T22:58:35.950 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.950 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T22:58:35.950 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.950 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.950 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.950 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_metrics": { 2024-03-18T22:58:35.950 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_metrics", 2024-03-18T22:58:35.950 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.950 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_compat_step": { 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crush_compat_step", 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.951 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.5", 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0.001", 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "0.999", 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "aggressiveness of optimization", 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "end_time": { 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_time", 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2359", 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.952 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: "end_weekday": { 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "end_weekday", 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "0", 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "6", 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T22:58:35.953 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T22:58:35.954 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.954 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.954 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:35.954 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:35.954 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.954 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.954 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.954 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.954 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.954 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.954 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.954 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.954 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.954 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.955 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.955 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.955 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.955 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.955 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.955 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.955 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.955 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.955 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.955 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:35.955 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:35.955 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.955 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.958 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.958 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.958 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.958 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.958 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.958 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.958 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.958 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.958 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:35.959 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.960 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.960 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.960 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.960 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.960 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:35.960 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:35.960 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.960 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.960 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.960 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.960 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.960 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.960 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.960 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:35.961 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.961 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.961 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.961 INFO:teuthology.orchestra.run.smithi040.stdout: "min_score": { 2024-03-18T22:58:35.961 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_score", 2024-03-18T22:58:35.961 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.961 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.961 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.961 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-03-18T22:58:35.961 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.961 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.961 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.961 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "upmap", 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: "crush-compat", 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: "none", 2024-03-18T22:58:35.962 INFO:teuthology.orchestra.run.smithi040.stdout: "read", 2024-03-18T22:58:35.963 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap", 2024-03-18T22:58:35.963 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap-read" 2024-03-18T22:58:35.963 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.963 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Balancer mode", 2024-03-18T22:58:35.963 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.963 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.963 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.963 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_ids": { 2024-03-18T22:58:35.963 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_ids", 2024-03-18T22:58:35.963 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.963 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.963 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.963 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.963 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.964 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.965 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T22:58:35.965 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.965 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.965 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.965 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_deviation": { 2024-03-18T22:58:35.965 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_deviation", 2024-03-18T22:58:35.965 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.965 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.965 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.965 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-03-18T22:58:35.965 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-03-18T22:58:35.965 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.965 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.965 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T22:58:35.966 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T22:58:35.966 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.966 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.966 INFO:teuthology.orchestra.run.smithi040.stdout: "upmap_max_optimizations": { 2024-03-18T22:58:35.966 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "upmap_max_optimizations", 2024-03-18T22:58:35.966 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-03-18T22:58:35.966 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.966 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.966 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:35.966 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.966 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.966 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.966 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T22:58:35.967 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.967 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.967 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.967 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.967 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:35.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.967 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:35.967 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm", 2024-03-18T22:58:35.967 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:35.967 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:35.967 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:35.967 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_down_multiplier": { 2024-03-18T22:58:35.967 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_down_multiplier", 2024-03-18T22:58:35.967 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.967 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.968 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.968 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-03-18T22:58:35.968 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.968 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.968 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.968 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T22:58:35.968 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.968 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.968 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.968 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.968 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_refresh_rate": { 2024-03-18T22:58:35.968 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_refresh_rate", 2024-03-18T22:58:35.968 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.968 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.968 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "20", 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: "agent_starting_port": { 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "agent_starting_port", 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4721", 2024-03-18T22:58:35.969 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_ptrace": { 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_ptrace", 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.970 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_interval": { 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_interval", 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T22:58:35.971 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.972 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.972 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.972 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.972 INFO:teuthology.orchestra.run.smithi040.stdout: "autotune_memory_target_ratio": { 2024-03-18T22:58:35.972 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T22:58:35.972 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:35.972 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.972 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.972 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0.7", 2024-03-18T22:58:35.972 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.972 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.972 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.972 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T22:58:35.972 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.972 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.973 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.973 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.973 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm_log_destination": { 2024-03-18T22:58:35.973 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cephadm_log_destination", 2024-03-18T22:58:35.973 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.973 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.973 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.973 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:35.973 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.973 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.973 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:35.973 INFO:teuthology.orchestra.run.smithi040.stdout: "file", 2024-03-18T22:58:35.973 INFO:teuthology.orchestra.run.smithi040.stdout: "file,syslog", 2024-03-18T22:58:35.973 INFO:teuthology.orchestra.run.smithi040.stdout: "syslog" 2024-03-18T22:58:35.973 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:35.974 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T22:58:35.974 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.974 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.974 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.974 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.974 INFO:teuthology.orchestra.run.smithi040.stdout: "cgroups_split": { 2024-03-18T22:58:35.974 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cgroups_split", 2024-03-18T22:58:35.974 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.974 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.974 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.974 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.974 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.974 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.974 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.974 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: "config_checks_enabled": { 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_checks_enabled", 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.975 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.976 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.976 INFO:teuthology.orchestra.run.smithi040.stdout: "config_dashboard": { 2024-03-18T22:58:35.976 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_dashboard", 2024-03-18T22:58:35.976 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.976 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.976 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.976 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.976 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.976 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.976 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.976 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T22:58:35.976 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.976 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.977 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.977 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_alertmanager": { 2024-03-18T22:58:35.977 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_alertmanager", 2024-03-18T22:58:35.977 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.977 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.977 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.977 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T22:58:35.977 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.977 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.977 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.977 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-03-18T22:58:35.977 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.977 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.978 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.978 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_base": { 2024-03-18T22:58:35.978 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_base", 2024-03-18T22:58:35.978 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.978 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.978 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:35.978 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T22:58:35.978 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.978 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.978 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.978 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Container image name, without the tag", 2024-03-18T22:58:35.978 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.979 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.979 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.979 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_elasticsearch": { 2024-03-18T22:58:35.979 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_elasticsearch", 2024-03-18T22:58:35.979 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.979 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.979 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.979 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T22:58:35.979 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.979 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.979 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.979 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "elasticsearch container image", 2024-03-18T22:58:35.979 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.979 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.980 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.980 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_grafana": { 2024-03-18T22:58:35.980 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_grafana", 2024-03-18T22:58:35.980 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.980 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.980 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.980 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T22:58:35.980 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.980 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.980 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.980 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-03-18T22:58:35.980 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.980 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.980 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_haproxy": { 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_haproxy", 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "HAproxy container image", 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.981 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_agent": { 2024-03-18T22:58:35.982 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_agent", 2024-03-18T22:58:35.982 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.982 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.982 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.982 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T22:58:35.982 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.982 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.982 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.982 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger agent container image", 2024-03-18T22:58:35.982 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.982 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.982 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.982 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_collector": { 2024-03-18T22:58:35.982 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_collector", 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger collector container image", 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_jaeger_query": { 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_jaeger_query", 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.983 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.984 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.984 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T22:58:35.984 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.984 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.984 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.984 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Jaeger query container image", 2024-03-18T22:58:35.984 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.984 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.984 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.984 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_keepalived": { 2024-03-18T22:58:35.984 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_keepalived", 2024-03-18T22:58:35.984 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.984 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.984 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.985 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T22:58:35.985 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.985 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.985 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.985 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Keepalived container image", 2024-03-18T22:58:35.985 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.985 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.985 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.985 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_loki": { 2024-03-18T22:58:35.985 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_loki", 2024-03-18T22:58:35.985 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.985 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.985 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.985 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Loki container image", 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_node_exporter": { 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_node_exporter", 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.986 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.987 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.987 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-03-18T22:58:35.987 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.987 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.987 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.987 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.987 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_nvmeof": { 2024-03-18T22:58:35.987 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_nvmeof", 2024-03-18T22:58:35.987 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.987 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.987 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.987 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T22:58:35.987 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.987 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.987 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.988 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Nvme-of container image", 2024-03-18T22:58:35.988 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.988 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.988 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.988 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_prometheus": { 2024-03-18T22:58:35.988 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_prometheus", 2024-03-18T22:58:35.988 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.988 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.988 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.988 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T22:58:35.988 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.988 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.988 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.988 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Prometheus container image", 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_promtail": { 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_promtail", 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Promtail container image", 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.989 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image_snmp_gateway": { 2024-03-18T22:58:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_image_snmp_gateway", 2024-03-18T22:58:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T22:58:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "SNMP Gateway container image", 2024-03-18T22:58:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: "container_init": { 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "container_init", 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.991 INFO:teuthology.orchestra.run.smithi040.stdout: "daemon_cache_timeout": { 2024-03-18T22:58:35.992 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "daemon_cache_timeout", 2024-03-18T22:58:35.992 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.992 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.992 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.992 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-03-18T22:58:35.992 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.992 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.992 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.992 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T22:58:35.992 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.992 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.992 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.992 INFO:teuthology.orchestra.run.smithi040.stdout: "default_cephadm_command_timeout": { 2024-03-18T22:58:35.993 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T22:58:35.993 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:35.993 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.993 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.993 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "900", 2024-03-18T22:58:35.993 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.993 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.993 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.993 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T22:58:35.993 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.993 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.993 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.993 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.993 INFO:teuthology.orchestra.run.smithi040.stdout: "default_registry": { 2024-03-18T22:58:35.993 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "default_registry", 2024-03-18T22:58:35.994 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:35.994 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.994 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.994 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "docker.io", 2024-03-18T22:58:35.994 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.994 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.994 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.994 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T22:58:35.994 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.994 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.994 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.994 INFO:teuthology.orchestra.run.smithi040.stdout: "device_cache_timeout": { 2024-03-18T22:58:35.995 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_cache_timeout", 2024-03-18T22:58:35.995 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.995 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.995 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.995 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1800", 2024-03-18T22:58:35.995 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.995 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.995 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.995 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache device inventory", 2024-03-18T22:58:35.995 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.995 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.995 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.995 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.995 INFO:teuthology.orchestra.run.smithi040.stdout: "device_enhanced_scan": { 2024-03-18T22:58:35.995 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_enhanced_scan", 2024-03-18T22:58:35.996 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.996 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.996 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.996 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.996 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.996 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.996 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.996 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T22:58:35.996 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.996 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.996 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.996 INFO:teuthology.orchestra.run.smithi040.stdout: "facts_cache_timeout": { 2024-03-18T22:58:35.996 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "facts_cache_timeout", 2024-03-18T22:58:35.996 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "seconds to cache host facts data", 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: "host_check_interval": { 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "host_check_interval", 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.997 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.998 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-03-18T22:58:35.998 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.998 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.998 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.998 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to perform a host check", 2024-03-18T22:58:35.998 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.998 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.998 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.998 INFO:teuthology.orchestra.run.smithi040.stdout: "hw_monitoring": { 2024-03-18T22:58:35.998 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hw_monitoring", 2024-03-18T22:58:35.998 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.998 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.998 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.999 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:35.999 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:35.999 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:35.999 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:35.999 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T22:58:35.999 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:35.999 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:35.999 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:35.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:35.999 INFO:teuthology.orchestra.run.smithi040.stdout: "inventory_list_all": { 2024-03-18T22:58:35.999 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "inventory_list_all", 2024-03-18T22:58:35.999 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:35.999 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:35.999 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:35.999 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.000 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.000 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.000 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.000 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T22:58:36.000 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.000 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.000 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.000 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.000 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.000 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.000 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.000 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.000 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.000 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: "log_refresh_metadata": { 2024-03-18T22:58:36.001 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_refresh_metadata", 2024-03-18T22:58:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T22:58:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.002 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.003 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.004 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T22:58:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T22:58:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T22:58:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.006 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T22:58:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T22:58:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "*", 2024-03-18T22:58:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T22:58:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.007 INFO:teuthology.orchestra.run.smithi040.stdout: "max_count_per_host": { 2024-03-18T22:58:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_count_per_host", 2024-03-18T22:58:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of daemons per service per host", 2024-03-18T22:58:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.008 INFO:teuthology.orchestra.run.smithi040.stdout: "max_osd_draining_count": { 2024-03-18T22:58:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_osd_draining_count", 2024-03-18T22:58:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T22:58:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: "migration_current": { 2024-03-18T22:58:36.009 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "migration_current", 2024-03-18T22:58:36.010 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.010 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.010 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.010 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.010 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.010 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.010 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.010 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "internal - do not modify", 2024-03-18T22:58:36.010 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.010 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.010 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.010 INFO:teuthology.orchestra.run.smithi040.stdout: "mode": { 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mode", 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "root", 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm-package", 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: "root" 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: "oob_default_addr": { 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "oob_default_addr", 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "169.254.1.1", 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: "prometheus_alerts_path": { 2024-03-18T22:58:36.012 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus_alerts_path", 2024-03-18T22:58:36.013 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.013 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.013 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.013 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T22:58:36.013 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.013 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.013 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.013 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T22:58:36.013 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.013 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.013 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.013 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_insecure": { 2024-03-18T22:58:36.013 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_insecure", 2024-03-18T22:58:36.013 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.014 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.014 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.014 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.014 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.014 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.014 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.014 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T22:58:36.014 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.014 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.014 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.014 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_password": { 2024-03-18T22:58:36.014 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_password", 2024-03-18T22:58:36.014 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.014 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_url": { 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_url", 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.015 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: "registry_username": { 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "registry_username", 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.017 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.017 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T22:58:36.017 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.017 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.017 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.017 INFO:teuthology.orchestra.run.smithi040.stdout: "secure_monitoring_stack": { 2024-03-18T22:58:36.017 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "secure_monitoring_stack", 2024-03-18T22:58:36.017 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.017 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.017 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.017 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.017 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.017 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.017 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.018 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T22:58:36.018 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.018 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.018 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.018 INFO:teuthology.orchestra.run.smithi040.stdout: "service_discovery_port": { 2024-03-18T22:58:36.018 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "service_discovery_port", 2024-03-18T22:58:36.018 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.018 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.018 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.018 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8765", 2024-03-18T22:58:36.018 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "cephadm service discovery port", 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: "ssh_config_file": { 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssh_config_file", 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.019 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.020 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T22:58:36.020 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.020 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.020 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.020 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.020 INFO:teuthology.orchestra.run.smithi040.stdout: "use_agent": { 2024-03-18T22:58:36.020 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_agent", 2024-03-18T22:58:36.020 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.020 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.020 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.020 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.020 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.020 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.020 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.020 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: "use_repo_digest": { 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "use_repo_digest", 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.021 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.022 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.022 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.022 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_failed_host_check": { 2024-03-18T22:58:36.022 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_failed_host_check", 2024-03-18T22:58:36.022 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.022 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.022 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.022 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.022 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.022 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.022 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.022 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T22:58:36.022 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.022 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.022 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.023 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_daemons": { 2024-03-18T22:58:36.023 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_daemons", 2024-03-18T22:58:36.023 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.023 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.023 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.023 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.023 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.023 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.023 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.023 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T22:58:36.023 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.023 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.023 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.024 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_on_stray_hosts": { 2024-03-18T22:58:36.024 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_on_stray_hosts", 2024-03-18T22:58:36.024 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.024 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.024 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.024 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.024 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.024 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.024 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.024 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T22:58:36.024 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.024 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.024 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.024 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.024 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crash", 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.025 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.026 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.026 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.026 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.026 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.026 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.026 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.026 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.026 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.026 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.026 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.026 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.026 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.027 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.028 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.028 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.028 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.028 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.028 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.028 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.028 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.028 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.028 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.028 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.028 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.028 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.028 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.028 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.029 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.030 INFO:teuthology.orchestra.run.smithi040.stdout: "retain_interval": { 2024-03-18T22:58:36.030 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retain_interval", 2024-03-18T22:58:36.030 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:36.030 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.030 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.030 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "31536000", 2024-03-18T22:58:36.030 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.030 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.030 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.030 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T22:58:36.030 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.030 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_recent_interval": { 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_recent_interval", 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1209600", 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.031 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dashboard", 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.032 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.033 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.033 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.033 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.033 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.033 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T22:58:36.033 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T22:58:36.033 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.033 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.033 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.033 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.033 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.033 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.033 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.033 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.034 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.034 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.034 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.034 INFO:teuthology.orchestra.run.smithi040.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T22:58:36.034 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T22:58:36.034 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.034 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.034 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.034 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.034 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.034 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.034 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.034 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.035 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.035 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.035 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.035 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_ENABLED": { 2024-03-18T22:58:36.035 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T22:58:36.035 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.035 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.035 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.035 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.035 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.035 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.035 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.035 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.035 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.036 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.036 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.036 INFO:teuthology.orchestra.run.smithi040.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T22:58:36.036 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T22:58:36.036 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.036 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.036 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.036 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.036 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.036 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.036 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.036 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.036 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.036 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.037 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.037 INFO:teuthology.orchestra.run.smithi040.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T22:58:36.037 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T22:58:36.037 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.037 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.037 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.037 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.037 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.037 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.037 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.037 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.037 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.037 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.037 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.038 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T22:58:36.038 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T22:58:36.038 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.038 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.038 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.038 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.038 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.038 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.038 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.038 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.038 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.038 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.038 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T22:58:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T22:58:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.040 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T22:58:36.040 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T22:58:36.040 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.040 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.040 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.040 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.040 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.040 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.040 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.040 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.040 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.040 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.040 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.040 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T22:58:36.041 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T22:58:36.041 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.041 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.041 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.041 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.041 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.041 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.041 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.041 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.041 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.041 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.041 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.041 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T22:58:36.041 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T22:58:36.042 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.042 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.042 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.042 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.042 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.042 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.042 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.042 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.042 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.042 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.042 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.042 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T22:58:36.042 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T22:58:36.042 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.043 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.043 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.043 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.043 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.043 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.043 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.043 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.043 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.043 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.043 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.043 INFO:teuthology.orchestra.run.smithi040.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T22:58:36.043 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T22:58:36.043 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.043 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.044 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.044 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.044 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.044 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.044 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.044 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.044 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.044 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.044 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.044 INFO:teuthology.orchestra.run.smithi040.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T22:58:36.044 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T22:58:36.044 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.044 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.044 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-03-18T22:58:36.045 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.046 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.046 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.046 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.046 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.046 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.046 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.046 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.046 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T22:58:36.046 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T22:58:36.046 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.046 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.046 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.046 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.046 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.046 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.047 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.047 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.047 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.047 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.047 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.047 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_URL": { 2024-03-18T22:58:36.047 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_URL", 2024-03-18T22:58:36.047 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.047 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.047 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.047 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.047 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.047 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.048 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.048 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.048 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.048 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.048 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.048 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T22:58:36.048 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T22:58:36.048 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.048 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.048 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.048 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-03-18T22:58:36.048 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.048 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.048 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.049 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.049 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.049 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.049 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.049 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T22:58:36.049 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T22:58:36.049 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.049 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.049 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.049 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.049 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.049 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.049 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.050 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.050 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.050 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.050 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.050 INFO:teuthology.orchestra.run.smithi040.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T22:58:36.050 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T22:58:36.050 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.050 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.050 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.050 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.050 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.050 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.051 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.051 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.051 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.051 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.051 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.051 INFO:teuthology.orchestra.run.smithi040.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T22:58:36.051 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T22:58:36.051 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.051 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.051 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.051 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.051 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.051 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.051 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.052 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.052 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.052 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.052 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.052 INFO:teuthology.orchestra.run.smithi040.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T22:58:36.052 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T22:58:36.052 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.052 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.052 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.052 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.052 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.052 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.052 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.052 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "{}", 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.053 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.054 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.054 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T22:58:36.054 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T22:58:36.054 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.054 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.054 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.054 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.054 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.054 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.054 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.054 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.054 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.054 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.055 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.055 INFO:teuthology.orchestra.run.smithi040.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T22:58:36.055 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T22:58:36.055 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.055 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.055 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.055 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.055 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.055 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.055 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.055 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.055 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.055 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.055 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.056 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T22:58:36.056 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T22:58:36.056 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.056 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.056 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.056 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.056 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.056 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.056 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.056 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.056 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.056 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.056 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.057 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T22:58:36.057 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T22:58:36.057 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.057 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.057 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.057 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.057 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.057 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.057 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.057 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.057 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.057 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.057 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.058 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T22:58:36.058 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T22:58:36.058 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.058 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.058 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.058 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.058 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.058 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.058 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.058 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.058 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.058 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.058 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.059 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T22:58:36.059 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T22:58:36.059 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.059 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.059 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.059 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.059 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.059 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.059 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.059 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.059 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.059 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.059 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.060 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T22:58:36.061 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T22:58:36.061 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.061 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.061 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.061 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.061 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.061 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.061 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.061 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.061 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.061 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.061 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.061 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T22:58:36.061 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T22:58:36.062 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.062 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.062 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.062 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.062 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.062 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.062 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.062 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.062 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.062 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.062 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.062 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T22:58:36.063 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T22:58:36.063 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.063 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.063 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.063 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.063 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.063 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.063 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.063 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.063 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.063 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.063 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.063 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T22:58:36.064 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T22:58:36.064 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.064 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.064 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.064 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T22:58:36.064 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.064 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.064 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.064 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.064 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.064 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.064 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.064 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T22:58:36.064 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T22:58:36.065 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.065 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.065 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.065 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:36.065 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.065 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.065 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.065 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.065 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.065 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.065 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.065 INFO:teuthology.orchestra.run.smithi040.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T22:58:36.065 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T22:58:36.065 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.066 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.066 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.066 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8", 2024-03-18T22:58:36.066 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.066 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.066 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.066 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.066 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.066 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.066 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.066 INFO:teuthology.orchestra.run.smithi040.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T22:58:36.066 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T22:58:36.066 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.067 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.067 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.067 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "45", 2024-03-18T22:58:36.067 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.067 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.067 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.067 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.067 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.067 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.067 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.067 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T22:58:36.067 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T22:58:36.067 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.067 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.068 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.068 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.068 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.068 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.068 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.068 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.068 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.068 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.068 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.068 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T22:58:36.068 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T22:58:36.068 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.068 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.068 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.069 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "admin", 2024-03-18T22:58:36.069 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.069 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.069 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.069 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.069 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.069 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.069 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.069 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T22:58:36.069 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T22:58:36.069 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.069 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.069 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.069 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.070 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.070 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.070 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.070 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.070 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.070 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.070 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.070 INFO:teuthology.orchestra.run.smithi040.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T22:58:36.070 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T22:58:36.070 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.070 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.070 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.070 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.070 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.071 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.071 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.071 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.071 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.071 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.071 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.071 INFO:teuthology.orchestra.run.smithi040.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T22:58:36.071 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T22:58:36.071 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.071 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.071 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.071 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.071 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.072 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.072 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.072 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.072 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.072 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.072 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.072 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T22:58:36.072 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T22:58:36.072 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.072 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.072 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.072 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-03-18T22:58:36.072 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.072 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.073 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.073 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.073 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.073 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.073 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.073 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T22:58:36.073 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T22:58:36.073 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.073 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.073 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.073 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:36.073 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.073 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.073 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.074 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.074 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.074 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.074 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.074 INFO:teuthology.orchestra.run.smithi040.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T22:58:36.074 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T22:58:36.074 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.074 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.074 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.074 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-03-18T22:58:36.074 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.074 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.074 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.075 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.075 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.075 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.075 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.075 INFO:teuthology.orchestra.run.smithi040.stdout: "cross_origin_url": { 2024-03-18T22:58:36.075 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cross_origin_url", 2024-03-18T22:58:36.075 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.075 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.075 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.075 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.075 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.075 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.075 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.075 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "crt_file": { 2024-03-18T22:58:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "crt_file", 2024-03-18T22:58:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.076 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "debug": { 2024-03-18T22:58:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "debug", 2024-03-18T22:58:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Enable/disable debug options", 2024-03-18T22:58:36.077 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "jwt_token_ttl": { 2024-03-18T22:58:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "jwt_token_ttl", 2024-03-18T22:58:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "28800", 2024-03-18T22:58:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.078 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.079 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.080 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.081 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.082 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.084 INFO:teuthology.orchestra.run.smithi040.stdout: "motd": { 2024-03-18T22:58:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "motd", 2024-03-18T22:58:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "The message of the day", 2024-03-18T22:58:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.085 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect_resolve_ip_addr": { 2024-03-18T22:58:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T22:58:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.086 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-03-18T22:58:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-03-18T22:58:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-03-18T22:58:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-03-18T22:58:36.087 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-03-18T22:58:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8080", 2024-03-18T22:58:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-03-18T22:58:36.088 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-03-18T22:58:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl_server_port": { 2024-03-18T22:58:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl_server_port", 2024-03-18T22:58:36.089 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.090 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.090 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.090 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8443", 2024-03-18T22:58:36.090 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.090 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.090 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.090 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.090 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.090 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.090 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.090 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-03-18T22:58:36.090 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-03-18T22:58:36.090 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.091 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.091 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.091 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "redirect", 2024-03-18T22:58:36.091 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.091 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.091 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.091 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.091 INFO:teuthology.orchestra.run.smithi040.stdout: "redirect" 2024-03-18T22:58:36.091 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.091 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.091 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.091 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.091 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.091 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-03-18T22:58:36.092 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-03-18T22:58:36.092 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.092 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.092 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.092 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-03-18T22:58:36.092 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-03-18T22:58:36.092 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-03-18T22:58:36.092 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.092 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.092 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.092 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.092 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.092 INFO:teuthology.orchestra.run.smithi040.stdout: "url_prefix": { 2024-03-18T22:58:36.092 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url_prefix", 2024-03-18T22:58:36.093 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.093 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.093 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.093 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.093 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.093 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.093 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.093 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.093 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.093 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.093 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.093 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.093 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.094 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "devicehealth", 2024-03-18T22:58:36.094 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.094 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.094 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.094 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_monitoring": { 2024-03-18T22:58:36.094 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_monitoring", 2024-03-18T22:58:36.094 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.094 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.094 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.094 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.094 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.094 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.094 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.094 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "monitor device health metrics", 2024-03-18T22:58:36.095 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.095 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.095 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.095 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.095 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.095 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.095 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.095 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.095 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.095 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.095 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.095 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.095 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.095 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.096 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.096 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.096 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.096 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.096 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.096 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.096 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.096 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.096 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.096 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.096 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.096 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.096 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.097 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.097 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.097 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.097 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.097 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.097 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.097 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.097 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.097 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.097 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.097 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.097 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.097 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.098 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.098 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.098 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.098 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.098 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.098 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.098 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.098 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.098 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.098 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.098 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.098 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.098 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.098 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.099 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.099 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.099 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.099 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.099 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.099 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.099 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.099 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.099 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.099 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.099 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.099 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.099 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.100 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.100 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.100 INFO:teuthology.orchestra.run.smithi040.stdout: "mark_out_threshold": { 2024-03-18T22:58:36.100 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mark_out_threshold", 2024-03-18T22:58:36.100 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:36.100 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.100 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.100 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "2419200", 2024-03-18T22:58:36.100 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.100 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.100 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.100 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T22:58:36.100 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.100 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.101 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.101 INFO:teuthology.orchestra.run.smithi040.stdout: "pool_name": { 2024-03-18T22:58:36.101 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pool_name", 2024-03-18T22:58:36.101 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.101 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.101 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.101 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "device_health_metrics", 2024-03-18T22:58:36.101 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.101 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.101 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.101 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T22:58:36.101 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.101 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.101 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.102 INFO:teuthology.orchestra.run.smithi040.stdout: "retention_period": { 2024-03-18T22:58:36.102 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "retention_period", 2024-03-18T22:58:36.102 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:36.102 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.102 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.102 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15552000", 2024-03-18T22:58:36.102 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.102 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.102 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.102 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long to retain device health metrics", 2024-03-18T22:58:36.102 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.102 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.102 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_frequency": { 2024-03-18T22:58:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_frequency", 2024-03-18T22:58:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-03-18T22:58:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T22:58:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: "self_heal": { 2024-03-18T22:58:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "self_heal", 2024-03-18T22:58:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T22:58:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.104 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-03-18T22:58:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-03-18T22:58:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-03-18T22:58:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T22:58:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: "warn_threshold": { 2024-03-18T22:58:36.105 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "warn_threshold", 2024-03-18T22:58:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7257600", 2024-03-18T22:58:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T22:58:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "diskprediction_local", 2024-03-18T22:58:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.107 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.108 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.109 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.110 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.110 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.110 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.110 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.110 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.110 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.110 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.110 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.110 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.110 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.110 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.110 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.110 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.110 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.110 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.111 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.111 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.111 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.111 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.111 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.111 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.111 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.111 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.111 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.111 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.111 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.111 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.111 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.111 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.112 INFO:teuthology.orchestra.run.smithi040.stdout: "predict_interval": { 2024-03-18T22:58:36.112 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predict_interval", 2024-03-18T22:58:36.112 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.112 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.112 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.112 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "86400", 2024-03-18T22:58:36.112 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.112 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.112 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.112 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.112 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.112 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.112 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.113 INFO:teuthology.orchestra.run.smithi040.stdout: "predictor_model": { 2024-03-18T22:58:36.113 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "predictor_model", 2024-03-18T22:58:36.113 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.113 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.113 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.113 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "prophetstor", 2024-03-18T22:58:36.113 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.113 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.113 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.113 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.114 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.114 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.114 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.114 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-03-18T22:58:36.114 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-03-18T22:58:36.114 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.114 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.114 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.114 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "600", 2024-03-18T22:58:36.114 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.114 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.114 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.115 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.115 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.115 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.115 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.115 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.115 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.115 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "influx", 2024-03-18T22:58:36.115 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": false, 2024-03-18T22:58:36.115 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "influxdb python module not found", 2024-03-18T22:58:36.115 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.115 INFO:teuthology.orchestra.run.smithi040.stdout: "batch_size": { 2024-03-18T22:58:36.115 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "batch_size", 2024-03-18T22:58:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5000", 2024-03-18T22:58:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T22:58:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: "database": { 2024-03-18T22:58:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "database", 2024-03-18T22:58:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.117 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.117 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.117 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "ceph", 2024-03-18T22:58:36.117 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.117 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.117 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.117 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T22:58:36.117 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.117 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.117 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.117 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": { 2024-03-18T22:58:36.117 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "hostname", 2024-03-18T22:58:36.117 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.117 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.118 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.118 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.118 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.118 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.118 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.118 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server hostname", 2024-03-18T22:58:36.118 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.118 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.118 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.118 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-03-18T22:58:36.118 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-03-18T22:58:36.118 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:36.119 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.119 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.119 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "30", 2024-03-18T22:58:36.119 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "5", 2024-03-18T22:58:36.119 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.119 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.119 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T22:58:36.119 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.119 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.119 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.119 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.119 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.119 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.119 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.120 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.120 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.120 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.120 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.120 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.120 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.120 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.120 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.120 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.120 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.120 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.120 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.120 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.120 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.121 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.122 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.122 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.122 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.122 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.122 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.122 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.122 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.122 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.122 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.122 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.122 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.122 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.122 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.122 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.123 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.123 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.123 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.123 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.123 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.123 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.123 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.123 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.123 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.123 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.123 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.123 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.123 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.124 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.124 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.124 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.124 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.124 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.124 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.124 INFO:teuthology.orchestra.run.smithi040.stdout: "password": { 2024-03-18T22:58:36.124 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "password", 2024-03-18T22:58:36.124 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.124 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.124 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.124 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.124 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.125 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.125 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.125 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "password of InfluxDB server user", 2024-03-18T22:58:36.125 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.125 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.125 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.125 INFO:teuthology.orchestra.run.smithi040.stdout: "port": { 2024-03-18T22:58:36.125 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "port", 2024-03-18T22:58:36.125 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.125 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.125 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.125 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "8086", 2024-03-18T22:58:36.125 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.125 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.126 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.126 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "InfluxDB server port", 2024-03-18T22:58:36.126 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.126 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.126 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.126 INFO:teuthology.orchestra.run.smithi040.stdout: "ssl": { 2024-03-18T22:58:36.126 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ssl", 2024-03-18T22:58:36.126 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.126 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.126 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.126 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "false", 2024-03-18T22:58:36.126 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.127 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.127 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.127 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T22:58:36.127 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.127 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.127 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.127 INFO:teuthology.orchestra.run.smithi040.stdout: "threads": { 2024-03-18T22:58:36.127 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threads", 2024-03-18T22:58:36.127 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.127 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.127 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.127 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-03-18T22:58:36.127 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-03-18T22:58:36.128 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "32", 2024-03-18T22:58:36.128 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.128 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T22:58:36.128 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.128 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.128 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.128 INFO:teuthology.orchestra.run.smithi040.stdout: "username": { 2024-03-18T22:58:36.128 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "username", 2024-03-18T22:58:36.128 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.128 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.128 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.128 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.128 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.129 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.129 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.129 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "username of InfluxDB server user", 2024-03-18T22:58:36.129 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.129 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.129 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.129 INFO:teuthology.orchestra.run.smithi040.stdout: "verify_ssl": { 2024-03-18T22:58:36.129 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "verify_ssl", 2024-03-18T22:58:36.129 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.129 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.129 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.129 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "true", 2024-03-18T22:58:36.129 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.129 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.130 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.130 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T22:58:36.130 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.130 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.130 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.130 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.130 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.130 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "insights", 2024-03-18T22:58:36.130 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.130 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.130 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.133 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.133 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.133 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.133 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.133 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.133 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.133 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.133 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.133 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.133 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.133 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.133 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.133 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.133 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.134 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.134 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.134 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.134 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.134 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.134 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.134 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.134 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.134 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.134 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.134 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.134 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.135 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.135 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.135 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.135 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.135 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.135 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.135 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.135 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.135 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.135 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.135 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.135 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.135 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.135 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.136 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "iostat", 2024-03-18T22:58:36.136 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.136 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.136 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.136 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.136 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.136 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.136 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.136 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.136 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.136 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.136 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.136 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.137 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.138 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.139 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.140 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "k8sevents", 2024-03-18T22:58:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_event_retention_days": { 2024-03-18T22:58:36.141 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "ceph_event_retention_days", 2024-03-18T22:58:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "7", 2024-03-18T22:58:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T22:58:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "config_check_secs": { 2024-03-18T22:58:36.142 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "config_check_secs", 2024-03-18T22:58:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "10", 2024-03-18T22:58:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T22:58:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.143 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.144 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.145 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.146 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.147 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "localpool", 2024-03-18T22:58:36.148 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "failure_domain": { 2024-03-18T22:58:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "failure_domain", 2024-03-18T22:58:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "host", 2024-03-18T22:58:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "failure domain for any created local pool", 2024-03-18T22:58:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T22:58:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.149 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.150 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.151 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.152 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.153 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "min_size": { 2024-03-18T22:58:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "min_size", 2024-03-18T22:58:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.154 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default min_size for any created local pool", 2024-03-18T22:58:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T22:58:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "num_rep": { 2024-03-18T22:58:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "num_rep", 2024-03-18T22:58:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3", 2024-03-18T22:58:36.155 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default replica count for any created local pool", 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_num": { 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_num", 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "128", 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.156 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T22:58:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "prefix": { 2024-03-18T22:58:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prefix", 2024-03-18T22:58:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.157 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "name prefix for any created local pool", 2024-03-18T22:58:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "subtree": { 2024-03-18T22:58:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "subtree", 2024-03-18T22:58:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "rack", 2024-03-18T22:58:36.158 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.162 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.162 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.162 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T22:58:36.162 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T22:58:36.162 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.162 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mds_autoscaler", 2024-03-18T22:58:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.163 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.165 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.166 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.167 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirroring", 2024-03-18T22:58:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.168 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.169 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.170 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.171 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.172 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.173 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "nfs", 2024-03-18T22:58:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.174 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.175 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.176 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.177 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.178 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-03-18T22:58:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.179 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "fail_fs": { 2024-03-18T22:58:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "fail_fs", 2024-03-18T22:58:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T22:58:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.181 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.182 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.183 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.184 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator": { 2024-03-18T22:58:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "orchestrator", 2024-03-18T22:58:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.185 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm", 2024-03-18T22:58:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "rook", 2024-03-18T22:58:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "test_orchestrator" 2024-03-18T22:58:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Orchestrator backend", 2024-03-18T22:58:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.186 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_perf_query", 2024-03-18T22:58:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.187 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.188 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.189 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.190 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.191 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "osd_support", 2024-03-18T22:58:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.192 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.193 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.194 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.195 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.196 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "pg_autoscaler", 2024-03-18T22:58:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.197 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.198 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.199 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.200 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.201 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-03-18T22:58:36.202 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-03-18T22:58:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-03-18T22:58:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "threshold": { 2024-03-18T22:58:36.203 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "threshold", 2024-03-18T22:58:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "3.0", 2024-03-18T22:58:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1.0", 2024-03-18T22:58:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "scaling threshold", 2024-03-18T22:58:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T22:58:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "progress", 2024-03-18T22:58:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_pg_recovery_event": { 2024-03-18T22:58:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_pg_recovery_event", 2024-03-18T22:58:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.205 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T22:58:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-03-18T22:58:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-03-18T22:58:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.206 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.207 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.208 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.209 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.210 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.211 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "max_completed_events": { 2024-03-18T22:58:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_completed_events", 2024-03-18T22:58:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "50", 2024-03-18T22:58:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "number of past completed events to remember", 2024-03-18T22:58:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.212 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "sleep_interval": { 2024-03-18T22:58:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "sleep_interval", 2024-03-18T22:58:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "5", 2024-03-18T22:58:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "how long the module is going to sleep", 2024-03-18T22:58:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.213 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "prometheus", 2024-03-18T22:58:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "cache": { 2024-03-18T22:58:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "cache", 2024-03-18T22:58:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.214 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: "exclude_perf_counters": { 2024-03-18T22:58:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "exclude_perf_counters", 2024-03-18T22:58:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.215 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.216 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.216 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.216 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T22:58:36.216 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T22:58:36.216 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.216 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.216 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.216 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.216 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.216 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.216 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.216 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.217 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.217 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.217 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.217 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.217 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.217 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.217 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.217 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.217 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.217 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.217 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.217 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.217 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.217 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.218 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.218 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.218 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.218 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.218 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.218 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.218 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.218 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.218 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.218 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.218 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.218 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.218 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.219 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.219 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.219 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.219 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.219 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.219 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.219 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.219 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.219 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.219 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.219 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.219 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.219 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.219 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.219 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.220 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.220 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.220 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.220 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.220 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.220 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.220 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.220 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.220 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.220 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.220 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.220 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.220 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.221 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.221 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.221 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.221 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.221 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.221 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.221 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools": { 2024-03-18T22:58:36.221 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools", 2024-03-18T22:58:36.221 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.221 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.221 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.221 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.221 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.221 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.222 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.222 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.222 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.222 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.222 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.222 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T22:58:36.222 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T22:58:36.222 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.222 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.222 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.222 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "300", 2024-03-18T22:58:36.222 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.222 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.223 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.223 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.223 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.223 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.223 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.223 INFO:teuthology.orchestra.run.smithi040.stdout: "scrape_interval": { 2024-03-18T22:58:36.223 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "scrape_interval", 2024-03-18T22:58:36.223 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:36.223 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.223 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.223 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15.0", 2024-03-18T22:58:36.223 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.223 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.223 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.224 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.224 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.224 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.224 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.224 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-03-18T22:58:36.224 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-03-18T22:58:36.224 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.224 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.224 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.224 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "::", 2024-03-18T22:58:36.224 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.224 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.224 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.224 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T22:58:36.225 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.225 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.225 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.225 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-03-18T22:58:36.225 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-03-18T22:58:36.225 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.225 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.225 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.225 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "9283", 2024-03-18T22:58:36.225 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.225 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.225 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.225 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T22:58:36.225 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.226 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.226 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.226 INFO:teuthology.orchestra.run.smithi040.stdout: "stale_cache_strategy": { 2024-03-18T22:58:36.226 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stale_cache_strategy", 2024-03-18T22:58:36.226 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.226 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.226 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.226 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "log", 2024-03-18T22:58:36.226 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.226 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.226 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.226 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.226 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.226 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.227 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.227 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_behaviour": { 2024-03-18T22:58:36.227 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_behaviour", 2024-03-18T22:58:36.227 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.227 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.227 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.227 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "default", 2024-03-18T22:58:36.227 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.227 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.227 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.227 INFO:teuthology.orchestra.run.smithi040.stdout: "default", 2024-03-18T22:58:36.227 INFO:teuthology.orchestra.run.smithi040.stdout: "error" 2024-03-18T22:58:36.227 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.228 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.228 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.228 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.228 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.228 INFO:teuthology.orchestra.run.smithi040.stdout: "standby_error_status_code": { 2024-03-18T22:58:36.228 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "standby_error_status_code", 2024-03-18T22:58:36.228 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.228 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.228 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.228 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "500", 2024-03-18T22:58:36.228 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "400", 2024-03-18T22:58:36.228 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "599", 2024-03-18T22:58:36.228 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.228 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.229 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.229 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.229 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.229 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.229 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.229 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_support", 2024-03-18T22:58:36.229 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.229 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.229 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.229 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.229 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.229 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.230 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.230 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.230 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.230 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.230 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.230 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.230 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.230 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.230 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.230 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.230 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.230 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.230 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.230 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.231 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.231 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.231 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.231 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.231 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.231 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.231 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.231 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.231 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.231 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.231 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.231 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.231 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.232 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.232 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.232 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.232 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.232 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.232 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.232 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.232 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.232 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.232 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.232 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.232 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.232 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.232 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.233 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.233 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.233 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.233 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.233 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.233 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.233 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.233 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.233 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.233 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.233 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.233 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.233 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.233 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.234 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.234 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.234 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.234 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.234 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.234 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.234 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.234 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.234 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_snap_create": { 2024-03-18T22:58:36.234 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_snap_create", 2024-03-18T22:58:36.234 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.234 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.235 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.235 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10", 2024-03-18T22:58:36.235 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.235 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.235 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.235 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.235 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.235 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.235 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.235 INFO:teuthology.orchestra.run.smithi040.stdout: "mirror_snapshot_schedule": { 2024-03-18T22:58:36.235 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T22:58:36.235 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.235 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.235 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.236 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.236 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.236 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.236 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.236 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.236 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.236 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.236 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.236 INFO:teuthology.orchestra.run.smithi040.stdout: "trash_purge_schedule": { 2024-03-18T22:58:36.236 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "trash_purge_schedule", 2024-03-18T22:58:36.236 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.236 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.236 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.236 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.237 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.237 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.237 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.237 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.237 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.237 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.237 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.237 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.237 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.237 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "restful", 2024-03-18T22:58:36.237 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.237 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.237 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.238 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_auth": { 2024-03-18T22:58:36.238 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enable_auth", 2024-03-18T22:58:36.238 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.238 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.238 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.238 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.238 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.238 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.238 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.238 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.238 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.238 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.238 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.239 INFO:teuthology.orchestra.run.smithi040.stdout: "key_file": { 2024-03-18T22:58:36.239 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "key_file", 2024-03-18T22:58:36.239 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.239 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.239 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.239 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.239 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.239 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.239 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.239 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.239 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.239 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.239 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.240 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.240 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.240 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.240 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.240 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.240 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.240 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.240 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.240 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.240 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.240 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.240 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.240 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.240 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.241 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.241 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.241 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.241 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.241 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.241 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.241 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.241 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.241 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.241 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.241 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.241 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.241 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.242 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.242 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.242 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.242 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.242 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.242 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.242 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.242 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.242 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.242 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.242 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.242 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.242 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.243 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.243 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.243 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.243 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.243 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.243 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.243 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.243 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.243 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.243 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.243 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.243 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.243 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.244 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.244 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.244 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.244 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.244 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.244 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.244 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.244 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.244 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.244 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.244 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.244 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.244 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.244 INFO:teuthology.orchestra.run.smithi040.stdout: "server_addr": { 2024-03-18T22:58:36.245 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_addr", 2024-03-18T22:58:36.245 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.245 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.245 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.245 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.245 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.245 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.245 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.245 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.245 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.245 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.245 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.245 INFO:teuthology.orchestra.run.smithi040.stdout: "server_port": { 2024-03-18T22:58:36.246 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "server_port", 2024-03-18T22:58:36.246 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.246 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.246 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.246 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.246 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.246 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.246 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.246 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.246 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.246 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.246 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.246 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.246 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.247 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.247 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rgw", 2024-03-18T22:58:36.247 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.247 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.247 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.247 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.247 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.247 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.247 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.247 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.247 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.247 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.247 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.247 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.248 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.248 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.248 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.248 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.248 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.248 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.248 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.248 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.248 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.248 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.248 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.248 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.249 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.249 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.249 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.249 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.249 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.249 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.249 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.249 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.249 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.249 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.249 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.249 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.249 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.250 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.250 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.250 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.250 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.250 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.250 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.250 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.250 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.250 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.250 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.250 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.250 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.250 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.250 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.251 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.251 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.251 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.251 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.251 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.251 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.251 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.251 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.251 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.251 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.251 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.251 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.251 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.252 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.252 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.252 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.252 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.252 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.252 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.252 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.252 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rook", 2024-03-18T22:58:36.252 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.252 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.252 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.252 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.252 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.253 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.253 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.253 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.253 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.253 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.253 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.253 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.253 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.253 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.253 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.253 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.253 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.253 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.254 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.254 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.254 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.254 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.254 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.254 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.254 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.254 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.254 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.254 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.254 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.254 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.255 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.255 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.255 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.255 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.255 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.255 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.255 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.255 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.255 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.255 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.255 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.255 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.255 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.255 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.256 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.256 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.256 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.256 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.256 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.256 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.256 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.256 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.256 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.256 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.256 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.256 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.256 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.256 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.257 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.257 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.257 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.257 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.257 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.257 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.257 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.257 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.257 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.257 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.257 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.257 INFO:teuthology.orchestra.run.smithi040.stdout: "storage_class": { 2024-03-18T22:58:36.257 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "storage_class", 2024-03-18T22:58:36.257 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.258 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.258 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.258 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "local", 2024-03-18T22:58:36.258 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.258 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.258 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.258 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T22:58:36.258 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.258 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.258 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.258 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.258 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.258 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.259 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.259 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "selftest", 2024-03-18T22:58:36.259 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.259 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.259 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.259 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.259 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.259 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.259 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.259 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.259 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.259 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.259 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.259 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.259 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.260 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.260 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.260 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.260 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.260 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.260 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.260 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.260 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.260 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.260 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.260 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.260 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.260 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.260 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.261 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.261 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.261 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.261 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.261 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.261 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.261 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.261 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.261 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.261 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.261 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.261 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.261 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.262 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.262 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.262 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.262 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.262 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.262 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.262 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.262 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.262 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.262 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.262 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.262 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.262 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.262 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.263 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.263 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.263 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.263 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.263 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.263 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.263 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.263 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.263 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.263 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.263 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.263 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.263 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.264 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.264 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.264 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.264 INFO:teuthology.orchestra.run.smithi040.stdout: "roption1": { 2024-03-18T22:58:36.264 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption1", 2024-03-18T22:58:36.264 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.264 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.264 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.264 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.264 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.264 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.264 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.264 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.264 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.264 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.265 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.265 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.265 INFO:teuthology.orchestra.run.smithi040.stdout: "roption2": { 2024-03-18T22:58:36.265 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "roption2", 2024-03-18T22:58:36.265 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.265 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.265 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.265 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "xyz", 2024-03-18T22:58:36.265 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.265 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.265 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.265 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.265 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.265 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.266 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.266 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.266 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption1": { 2024-03-18T22:58:36.266 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption1", 2024-03-18T22:58:36.266 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.266 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.266 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.266 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.266 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.266 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.266 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.266 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.266 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.267 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.267 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.267 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption2": { 2024-03-18T22:58:36.267 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption2", 2024-03-18T22:58:36.267 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.267 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.267 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.267 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.267 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.267 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.267 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.267 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.267 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.267 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.268 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.268 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption3": { 2024-03-18T22:58:36.268 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption3", 2024-03-18T22:58:36.268 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "float", 2024-03-18T22:58:36.268 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.268 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.268 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.268 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.268 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.268 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.268 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.268 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.269 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.269 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.269 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption4": { 2024-03-18T22:58:36.269 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption4", 2024-03-18T22:58:36.269 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.269 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.269 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.269 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.269 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.269 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.269 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.269 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.269 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.269 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.270 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.270 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.270 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption5": { 2024-03-18T22:58:36.270 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption5", 2024-03-18T22:58:36.270 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.270 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.270 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.270 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.270 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.270 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.270 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.270 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.270 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.271 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.271 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.271 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption6": { 2024-03-18T22:58:36.271 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption6", 2024-03-18T22:58:36.271 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.271 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.271 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.271 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.271 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.271 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.271 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.271 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.272 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.272 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.272 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.272 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.272 INFO:teuthology.orchestra.run.smithi040.stdout: "rwoption7": { 2024-03-18T22:58:36.272 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rwoption7", 2024-03-18T22:58:36.272 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.272 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.272 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.272 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.272 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "1", 2024-03-18T22:58:36.272 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "42", 2024-03-18T22:58:36.272 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.272 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.273 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.273 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.273 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.273 INFO:teuthology.orchestra.run.smithi040.stdout: "testkey": { 2024-03-18T22:58:36.273 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testkey", 2024-03-18T22:58:36.273 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.273 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.273 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.273 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.273 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.273 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.273 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.273 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.273 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.274 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.274 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.274 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.274 INFO:teuthology.orchestra.run.smithi040.stdout: "testlkey": { 2024-03-18T22:58:36.274 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testlkey", 2024-03-18T22:58:36.274 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.274 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.274 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.274 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.274 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.274 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.274 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.274 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.274 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.275 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.275 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.275 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.275 INFO:teuthology.orchestra.run.smithi040.stdout: "testnewline": { 2024-03-18T22:58:36.275 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "testnewline", 2024-03-18T22:58:36.275 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.275 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.275 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.275 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.275 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.275 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.275 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.275 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.275 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.276 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.276 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.276 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.276 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.276 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.276 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.276 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snap_schedule", 2024-03-18T22:58:36.276 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.276 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.276 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.276 INFO:teuthology.orchestra.run.smithi040.stdout: "allow_m_granularity": { 2024-03-18T22:58:36.276 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "allow_m_granularity", 2024-03-18T22:58:36.276 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.276 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.276 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.277 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.277 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.277 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.277 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.277 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T22:58:36.277 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.277 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.277 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.277 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.277 INFO:teuthology.orchestra.run.smithi040.stdout: "dump_on_update": { 2024-03-18T22:58:36.277 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "dump_on_update", 2024-03-18T22:58:36.277 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.277 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.277 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.278 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.278 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.278 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.278 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.278 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "dump database to debug log on update", 2024-03-18T22:58:36.278 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.278 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.278 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.278 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.278 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.278 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.278 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.278 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.279 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.279 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.279 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.279 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.279 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.279 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.279 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.279 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.279 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.279 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.279 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.279 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.279 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.279 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.280 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.280 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.280 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.280 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.280 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.280 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.280 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.280 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.280 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.280 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.280 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.280 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.280 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.280 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.280 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.281 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.281 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.281 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.281 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.281 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.281 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.281 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.281 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.281 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.281 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.281 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.281 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.281 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.281 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.282 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.282 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.282 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.282 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.282 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.282 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.282 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.282 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.282 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.282 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.282 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.282 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.282 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.282 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.283 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.283 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.283 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.283 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.283 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.283 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.283 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.283 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.283 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.283 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.283 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.283 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "stats", 2024-03-18T22:58:36.283 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.284 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.284 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.284 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.284 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.284 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.284 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.284 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.284 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.284 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.284 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.284 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.284 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.284 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.284 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.284 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.285 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.285 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.285 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.285 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.285 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.285 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.285 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.285 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.285 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.285 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.285 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.285 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.285 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.286 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.286 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.286 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.286 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.286 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.286 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.286 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.286 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.286 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.286 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.286 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.286 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.286 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.287 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.287 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.287 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.287 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.287 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.287 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.287 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.287 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.287 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.287 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.287 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.287 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.287 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.287 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.288 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.288 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.288 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.288 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.288 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.288 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.288 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.288 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.288 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.288 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.288 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.288 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.288 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.289 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.289 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.289 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.289 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.289 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "status", 2024-03-18T22:58:36.289 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.289 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.289 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.289 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.289 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.289 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.289 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.289 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.289 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.289 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.290 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.290 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.290 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.290 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.290 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.290 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.290 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.290 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.290 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.290 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.290 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.290 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.290 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.291 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.291 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.291 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.291 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.291 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.291 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.291 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.291 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.291 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.291 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.291 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.291 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.291 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.292 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.292 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.292 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.292 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.292 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.292 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.292 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.292 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.292 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.292 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.292 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.292 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.292 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.293 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.293 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.293 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.293 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.293 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.293 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.293 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.293 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.293 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.293 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.293 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.293 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.293 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.293 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.294 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.294 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.294 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.294 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.294 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.294 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.294 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.294 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.294 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telegraf", 2024-03-18T22:58:36.294 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.294 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.294 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.295 INFO:teuthology.orchestra.run.smithi040.stdout: "address": { 2024-03-18T22:58:36.295 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "address", 2024-03-18T22:58:36.295 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.295 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.295 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.295 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T22:58:36.295 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.295 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.295 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.295 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.295 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.295 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.295 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.295 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-03-18T22:58:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-03-18T22:58:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "15", 2024-03-18T22:58:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.300 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.300 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.300 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.300 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.300 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.300 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.300 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.300 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.300 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.300 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.300 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.300 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.300 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "telemetry", 2024-03-18T22:58:36.301 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.302 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.302 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.302 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_basic": { 2024-03-18T22:58:36.302 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_basic", 2024-03-18T22:58:36.302 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.302 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.302 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.302 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.302 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.302 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.302 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.302 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T22:58:36.302 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.302 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_crash": { 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_crash", 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.304 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_device": { 2024-03-18T22:58:36.304 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_device", 2024-03-18T22:58:36.304 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.304 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.304 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.304 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.304 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.304 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.304 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.304 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T22:58:36.304 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.304 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.304 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.305 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_ident": { 2024-03-18T22:58:36.305 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_ident", 2024-03-18T22:58:36.305 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.305 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.305 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.305 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.305 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.305 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.305 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.305 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T22:58:36.305 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.305 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.305 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.306 INFO:teuthology.orchestra.run.smithi040.stdout: "channel_perf": { 2024-03-18T22:58:36.306 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "channel_perf", 2024-03-18T22:58:36.306 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.306 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.306 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.306 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.306 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.306 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.306 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.306 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T22:58:36.306 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.306 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.306 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.306 INFO:teuthology.orchestra.run.smithi040.stdout: "contact": { 2024-03-18T22:58:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "contact", 2024-03-18T22:58:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: "description": { 2024-03-18T22:58:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "description", 2024-03-18T22:58:36.308 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.308 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.308 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.308 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.308 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.308 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.308 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.308 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.308 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.308 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.308 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.308 INFO:teuthology.orchestra.run.smithi040.stdout: "device_url": { 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "device_url", 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: "enabled": { 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "enabled", 2024-03-18T22:58:36.309 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-03-18T22:58:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-03-18T22:58:36.310 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "24", 2024-03-18T22:58:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "8", 2024-03-18T22:58:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "last_opt_revision": { 2024-03-18T22:58:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "last_opt_revision", 2024-03-18T22:58:36.311 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "1", 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard": { 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard", 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.312 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "leaderboard_description": { 2024-03-18T22:58:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "leaderboard_description", 2024-03-18T22:58:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.313 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.317 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.318 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "organization": { 2024-03-18T22:58:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "organization", 2024-03-18T22:58:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.319 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "proxy": { 2024-03-18T22:58:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "proxy", 2024-03-18T22:58:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.320 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "url": { 2024-03-18T22:58:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "url", 2024-03-18T22:58:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T22:58:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.321 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "test_orchestrator", 2024-03-18T22:58:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.322 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.324 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.325 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.326 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "volumes", 2024-03-18T22:58:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.328 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.329 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.330 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "max_concurrent_clones": { 2024-03-18T22:58:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "max_concurrent_clones", 2024-03-18T22:58:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "4", 2024-03-18T22:58:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T22:58:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.333 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "periodic_async_work": { 2024-03-18T22:58:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "periodic_async_work", 2024-03-18T22:58:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Periodically check for async work", 2024-03-18T22:58:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.334 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_delay": { 2024-03-18T22:58:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_delay", 2024-03-18T22:58:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "0", 2024-03-18T22:58:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T22:58:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.335 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "snapshot_clone_no_wait": { 2024-03-18T22:58:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T22:58:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "True", 2024-03-18T22:58:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T22:58:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.336 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix", 2024-03-18T22:58:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "can_run": true, 2024-03-18T22:58:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "error_string": "", 2024-03-18T22:58:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "module_options": { 2024-03-18T22:58:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "discovery_interval": { 2024-03-18T22:58:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "discovery_interval", 2024-03-18T22:58:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "uint", 2024-03-18T22:58:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "100", 2024-03-18T22:58:36.337 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "identifier": { 2024-03-18T22:58:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "identifier", 2024-03-18T22:58:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.338 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "interval": { 2024-03-18T22:58:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "interval", 2024-03-18T22:58:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "secs", 2024-03-18T22:58:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "60", 2024-03-18T22:58:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.339 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "log_level": { 2024-03-18T22:58:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_level", 2024-03-18T22:58:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.340 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster": { 2024-03-18T22:58:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster", 2024-03-18T22:58:36.341 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_cluster_level": { 2024-03-18T22:58:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_cluster_level", 2024-03-18T22:58:36.342 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "info", 2024-03-18T22:58:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [ 2024-03-18T22:58:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "", 2024-03-18T22:58:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "critical", 2024-03-18T22:58:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "debug", 2024-03-18T22:58:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "error", 2024-03-18T22:58:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "info", 2024-03-18T22:58:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "warning" 2024-03-18T22:58:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.343 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": { 2024-03-18T22:58:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "log_to_file", 2024-03-18T22:58:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "bool", 2024-03-18T22:58:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 1, 2024-03-18T22:58:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "False", 2024-03-18T22:58:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.344 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_host": { 2024-03-18T22:58:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_host", 2024-03-18T22:58:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "", 2024-03-18T22:58:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.345 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_port": { 2024-03-18T22:58:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_port", 2024-03-18T22:58:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "int", 2024-03-18T22:58:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "10051", 2024-03-18T22:58:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.346 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "zabbix_sender": { 2024-03-18T22:58:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "zabbix_sender", 2024-03-18T22:58:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "str", 2024-03-18T22:58:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "level": "advanced", 2024-03-18T22:58:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "flags": 0, 2024-03-18T22:58:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T22:58:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "min": "", 2024-03-18T22:58:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "max": "", 2024-03-18T22:58:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "enum_allowed": [], 2024-03-18T22:58:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "desc": "", 2024-03-18T22:58:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "long_desc": "", 2024-03-18T22:58:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "tags": [], 2024-03-18T22:58:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: "see_also": [] 2024-03-18T22:58:36.347 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "services": { 2024-03-18T22:58:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "dashboard": "https://172.21.15.40:8443/" 2024-03-18T22:58:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "always_on_modules": { 2024-03-18T22:58:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "octopus": [ 2024-03-18T22:58:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "balancer", 2024-03-18T22:58:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "crash", 2024-03-18T22:58:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "devicehealth", 2024-03-18T22:58:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator", 2024-03-18T22:58:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscaler", 2024-03-18T22:58:36.348 INFO:teuthology.orchestra.run.smithi040.stdout: "progress", 2024-03-18T22:58:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_support", 2024-03-18T22:58:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "status", 2024-03-18T22:58:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "telemetry", 2024-03-18T22:58:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "volumes" 2024-03-18T22:58:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "pacific": [ 2024-03-18T22:58:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "balancer", 2024-03-18T22:58:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "crash", 2024-03-18T22:58:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "devicehealth", 2024-03-18T22:58:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator", 2024-03-18T22:58:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscaler", 2024-03-18T22:58:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "progress", 2024-03-18T22:58:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_support", 2024-03-18T22:58:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "status", 2024-03-18T22:58:36.349 INFO:teuthology.orchestra.run.smithi040.stdout: "telemetry", 2024-03-18T22:58:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "volumes" 2024-03-18T22:58:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "quincy": [ 2024-03-18T22:58:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "balancer", 2024-03-18T22:58:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "crash", 2024-03-18T22:58:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "devicehealth", 2024-03-18T22:58:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator", 2024-03-18T22:58:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscaler", 2024-03-18T22:58:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "progress", 2024-03-18T22:58:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_support", 2024-03-18T22:58:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "status", 2024-03-18T22:58:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "telemetry", 2024-03-18T22:58:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: "volumes" 2024-03-18T22:58:36.350 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "reef": [ 2024-03-18T22:58:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "balancer", 2024-03-18T22:58:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "crash", 2024-03-18T22:58:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "devicehealth", 2024-03-18T22:58:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator", 2024-03-18T22:58:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscaler", 2024-03-18T22:58:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "progress", 2024-03-18T22:58:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_support", 2024-03-18T22:58:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "status", 2024-03-18T22:58:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "telemetry", 2024-03-18T22:58:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "volumes" 2024-03-18T22:58:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-03-18T22:58:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "squid": [ 2024-03-18T22:58:36.351 INFO:teuthology.orchestra.run.smithi040.stdout: "balancer", 2024-03-18T22:58:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "crash", 2024-03-18T22:58:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "devicehealth", 2024-03-18T22:58:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "orchestrator", 2024-03-18T22:58:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "pg_autoscaler", 2024-03-18T22:58:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "progress", 2024-03-18T22:58:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_support", 2024-03-18T22:58:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "status", 2024-03-18T22:58:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "telemetry", 2024-03-18T22:58:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "volumes" 2024-03-18T22:58:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-18T22:58:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "last_failure_osd_epoch": 3, 2024-03-18T22:58:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "active_clients": [ 2024-03-18T22:58:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.352 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "devicehealth", 2024-03-18T22:58:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-03-18T22:58:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-03-18T22:58:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-03-18T22:58:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1880935272 2024-03-18T22:58:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-18T22:58:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "libcephsqlite", 2024-03-18T22:58:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-03-18T22:58:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-03-18T22:58:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-03-18T22:58:36.353 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 841930532 2024-03-18T22:58:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-18T22:58:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "rbd_support", 2024-03-18T22:58:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-03-18T22:58:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-03-18T22:58:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-03-18T22:58:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 89988959 2024-03-18T22:58:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-18T22:58:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:36.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "volumes", 2024-03-18T22:58:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: "addrvec": [ 2024-03-18T22:58:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-18T22:58:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v2", 2024-03-18T22:58:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-03-18T22:58:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2728419226 2024-03-18T22:58:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-18T22:58:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:36.355 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-18T22:58:36.355 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:58:36.356 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:35 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1337811956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:58:36.357 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:35 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2539944351' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T22:58:36.357 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:35 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1337811956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:58:36.357 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:35 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2539944351' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T22:58:36.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:35.104+0000 7ff0a1344640 1 -- 172.21.15.40:0/2539944351 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff0880a2800 msgr2=0x7ff0880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:36.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:35.104+0000 7ff0a1344640 1 --2- 172.21.15.40:0/2539944351 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff0880a2800 0x7ff0880a4cc0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff09c1017b0 tx=0x7ff094005f90 comp rx=0 tx=0).stop 2024-03-18T22:58:36.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:35.104+0000 7ff0a1344640 1 -- 172.21.15.40:0/2539944351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff09c102310 msgr2=0x7ff09c19f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:58:36.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:35.104+0000 7ff0a1344640 1 --2- 172.21.15.40:0/2539944351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff09c102310 0x7ff09c19f070 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7ff08c00eb10 tx=0x7ff08c00c2d0 comp rx=0 tx=0).stop 2024-03-18T22:58:36.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:35.105+0000 7ff0a1344640 1 -- 172.21.15.40:0/2539944351 shutdown_connections 2024-03-18T22:58:36.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:35.105+0000 7ff0a1344640 1 --2- 172.21.15.40:0/2539944351 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff0880a2800 0x7ff0880a4cc0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:36.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:35.105+0000 7ff0a1344640 1 --2- 172.21.15.40:0/2539944351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff09c102310 0x7ff09c19f070 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:36.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:35.105+0000 7ff0a1344640 1 --2- 172.21.15.40:0/2539944351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff09c101950 0x7ff09c198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:36.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:35.105+0000 7ff0a1344640 1 --2- 172.21.15.40:0/2539944351 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff09c100760 0x7ff09c197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:58:36.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:35.105+0000 7ff0a1344640 1 -- 172.21.15.40:0/2539944351 >> 172.21.15.40:0/2539944351 conn(0x7ff09c0fbcd0 msgr2=0x7ff09c069f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:58:36.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:35.105+0000 7ff0a1344640 1 -- 172.21.15.40:0/2539944351 shutdown_connections 2024-03-18T22:58:36.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:58:35.105+0000 7ff0a1344640 1 -- 172.21.15.40:0/2539944351 wait complete. 2024-03-18T22:58:36.367 DEBUG:tasks.check_counter:Getting stats from a 2024-03-18T22:58:36.367 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-18T22:58:36.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:35 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1337811956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:58:36.368 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:35 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2539944351' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T22:58:36.635 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:58:36.764 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:36 smithi040 ceph-mon[62643]: pgmap v829: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T22:58:36.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:36 smithi113 ceph-mon[66775]: pgmap v829: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T22:58:36.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:36 smithi103 ceph-mon[66377]: pgmap v829: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T22:58:38.613 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:58:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T22:58:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2188, 2024-03-18T22:58:38.613 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 2190, 2024-03-18T22:58:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 958676, 2024-03-18T22:58:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 695785, 2024-03-18T22:58:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 6, 2024-03-18T22:58:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 2, 2024-03-18T22:58:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.178418778, 2024-03-18T22:58:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.052337433, 2024-03-18T22:58:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.043343405, 2024-03-18T22:58:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.068653190, 2024-03-18T22:58:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2188, 2024-03-18T22:58:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.097267850, 2024-03-18T22:58:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000044455 2024-03-18T22:58:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.614 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T22:58:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T22:58:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T22:58:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 943, 2024-03-18T22:58:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 1161, 2024-03-18T22:58:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 234380, 2024-03-18T22:58:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 928948, 2024-03-18T22:58:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 8, 2024-03-18T22:58:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 4, 2024-03-18T22:58:38.615 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.091558053, 2024-03-18T22:58:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.032802233, 2024-03-18T22:58:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.018837424, 2024-03-18T22:58:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.032868963, 2024-03-18T22:58:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1159, 2024-03-18T22:58:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.060088232, 2024-03-18T22:58:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000051844 2024-03-18T22:58:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-03-18T22:58:38.616 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 631232 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 325, 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 300, 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 665468, 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 180334, 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 2, 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.038664264, 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.015589292, 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.017547050, 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.005741761, 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 299, 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.013448776, 2024-03-18T22:58:38.617 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000044979 2024-03-18T22:58:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 665200, 2024-03-18T22:58:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 180016 2024-03-18T22:58:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-03-18T22:58:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-03-18T22:58:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-03-18T22:58:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.618 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-03-18T22:58:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-18T22:58:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-18T22:58:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 5237, 2024-03-18T22:58:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.122461807, 2024-03-18T22:58:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000023383 2024-03-18T22:58:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-03-18T22:58:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-18T22:58:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-18T22:58:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:38.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 0, 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 0, 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 0, 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 0, 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 0, 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 0, 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-03-18T22:58:38.620 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 21, 2024-03-18T22:58:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 0, 2024-03-18T22:58:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 21, 2024-03-18T22:58:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 0, 2024-03-18T22:58:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 21, 2024-03-18T22:58:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 0, 2024-03-18T22:58:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 0, 2024-03-18T22:58:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 0, 2024-03-18T22:58:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 1, 2024-03-18T22:58:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 0, 2024-03-18T22:58:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 0, 2024-03-18T22:58:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 0, 2024-03-18T22:58:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 0, 2024-03-18T22:58:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 0, 2024-03-18T22:58:38.621 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-03-18T22:58:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2024-03-18T22:58:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2024-03-18T22:58:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-03-18T22:58:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 0, 2024-03-18T22:58:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 0, 2024-03-18T22:58:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 0, 2024-03-18T22:58:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 0, 2024-03-18T22:58:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-03-18T22:58:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2024-03-18T22:58:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-03-18T22:58:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 0, 2024-03-18T22:58:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 0, 2024-03-18T22:58:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 0, 2024-03-18T22:58:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T22:58:38.622 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-03-18T22:58:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T22:58:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-03-18T22:58:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-03-18T22:58:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T22:58:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T22:58:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2024-03-18T22:58:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2024-03-18T22:58:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T22:58:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 0, 2024-03-18T22:58:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T22:58:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T22:58:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T22:58:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 0, 2024-03-18T22:58:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 0, 2024-03-18T22:58:38.623 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 0, 2024-03-18T22:58:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 0 2024-03-18T22:58:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-03-18T22:58:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-03-18T22:58:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2024-03-18T22:58:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2024-03-18T22:58:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 0, 2024-03-18T22:58:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 0, 2024-03-18T22:58:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2024-03-18T22:58:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-03-18T22:58:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-03-18T22:58:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 0, 2024-03-18T22:58:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 0, 2024-03-18T22:58:38.624 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_path": 0, 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_inode": 0, 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 0, 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 0, 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-03-18T22:58:38.625 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-03-18T22:58:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 0, 2024-03-18T22:58:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 0, 2024-03-18T22:58:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 0, 2024-03-18T22:58:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 251, 2024-03-18T22:58:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-03-18T22:58:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 0, 2024-03-18T22:58:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 0, 2024-03-18T22:58:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 0, 2024-03-18T22:58:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 0, 2024-03-18T22:58:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 2, 2024-03-18T22:58:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 2, 2024-03-18T22:58:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-03-18T22:58:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 0, 2024-03-18T22:58:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 4194304, 2024-03-18T22:58:38.626 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 4503927, 2024-03-18T22:58:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 4502276, 2024-03-18T22:58:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-03-18T22:58:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 251 2024-03-18T22:58:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-03-18T22:58:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 4, 2024-03-18T22:58:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 34, 2024-03-18T22:58:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 30, 2024-03-18T22:58:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 2, 2024-03-18T22:58:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 13, 2024-03-18T22:58:38.627 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 11, 2024-03-18T22:58:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 21, 2024-03-18T22:58:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 51, 2024-03-18T22:58:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 30, 2024-03-18T22:58:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 0, 2024-03-18T22:58:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 0, 2024-03-18T22:58:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 0, 2024-03-18T22:58:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 53736, 2024-03-18T22:58:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 215420 2024-03-18T22:58:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-03-18T22:58:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 0, 2024-03-18T22:58:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 0, 2024-03-18T22:58:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 0, 2024-03-18T22:58:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 0, 2024-03-18T22:58:38.628 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 0, 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.629 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-03-18T22:58:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-03-18T22:58:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-03-18T22:58:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.630 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-03-18T22:58:38.631 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-03-18T22:58:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-03-18T22:58:38.632 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-03-18T22:58:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-03-18T22:58:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.633 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-03-18T22:58:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-03-18T22:58:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-03-18T22:58:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.634 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-03-18T22:58:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-03-18T22:58:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-03-18T22:58:38.635 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-03-18T22:58:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-03-18T22:58:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-03-18T22:58:38.636 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.637 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-03-18T22:58:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-03-18T22:58:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-03-18T22:58:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.638 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-03-18T22:58:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-03-18T22:58:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-03-18T22:58:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-03-18T22:58:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 1, 2024-03-18T22:58:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 1, 2024-03-18T22:58:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 0, 2024-03-18T22:58:38.639 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 1, 2024-03-18T22:58:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-03-18T22:58:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 0, 2024-03-18T22:58:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 0, 2024-03-18T22:58:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 4479, 2024-03-18T22:58:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-03-18T22:58:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-03-18T22:58:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-03-18T22:58:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-03-18T22:58:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T22:58:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-03-18T22:58:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T22:58:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-03-18T22:58:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T22:58:38.640 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T22:58:38.641 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-03-18T22:58:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-03-18T22:58:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-03-18T22:58:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T22:58:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T22:58:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-03-18T22:58:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-03-18T22:58:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-03-18T22:58:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-03-18T22:58:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T22:58:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-03-18T22:58:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T22:58:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-03-18T22:58:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 189367, 2024-03-18T22:58:38.642 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 49, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 9184, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 177, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 26768, 2024-03-18T22:58:38.643 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 112, 2024-03-18T22:58:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-03-18T22:58:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-03-18T22:58:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-03-18T22:58:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-03-18T22:58:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-03-18T22:58:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-03-18T22:58:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-03-18T22:58:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 3125, 2024-03-18T22:58:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 21, 2024-03-18T22:58:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 3, 2024-03-18T22:58:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 3125, 2024-03-18T22:58:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-03-18T22:58:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3125, 2024-03-18T22:58:38.644 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 4.299949880, 2024-03-18T22:58:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001375983 2024-03-18T22:58:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-03-18T22:58:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-03-18T22:58:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3125, 2024-03-18T22:58:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 3126 2024-03-18T22:58:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 3125, 2024-03-18T22:58:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 3125, 2024-03-18T22:58:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 0, 2024-03-18T22:58:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-03-18T22:58:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-03-18T22:58:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 1978, 2024-03-18T22:58:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 0, 2024-03-18T22:58:38.645 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 1145, 2024-03-18T22:58:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 0, 2024-03-18T22:58:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 0, 2024-03-18T22:58:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-03-18T22:58:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-03-18T22:58:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-03-18T22:58:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 0, 2024-03-18T22:58:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 0, 2024-03-18T22:58:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-03-18T22:58:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-03-18T22:58:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-03-18T22:58:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 1, 2024-03-18T22:58:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 0, 2024-03-18T22:58:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-03-18T22:58:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-03-18T22:58:38.646 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-03-18T22:58:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 93, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 4, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 5, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 7, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 2, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 0, 2024-03-18T22:58:38.648 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 2, 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 0 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 0, 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 0 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-03-18T22:58:38.649 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 0, 2024-03-18T22:58:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 0, 2024-03-18T22:58:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 0, 2024-03-18T22:58:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-03-18T22:58:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T22:58:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-18T22:58:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-18T22:58:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-18T22:58:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 3452, 2024-03-18T22:58:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 1528225, 2024-03-18T22:58:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 3452, 2024-03-18T22:58:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-18T22:58:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-18T22:58:38.650 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 3452, 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 1528225, 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 3125, 2024-03-18T22:58:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 309628, 2024-03-18T22:58:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 3125, 2024-03-18T22:58:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 309628, 2024-03-18T22:58:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-18T22:58:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-03-18T22:58:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-18T22:58:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-03-18T22:58:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-18T22:58:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-18T22:58:38.652 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 3125, 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 3125, 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 3125, 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 3125, 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-18T22:58:38.653 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-18T22:58:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-18T22:58:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-18T22:58:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-18T22:58:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-18T22:58:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-03-18T22:58:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-03-18T22:58:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-18T22:58:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:38.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x55cfefd306e0": { 2024-03-18T22:58:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-18T22:58:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-18T22:58:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-18T22:58:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-18T22:58:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-18T22:58:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-18T22:58:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-18T22:58:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-03-18T22:58:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-03-18T22:58:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-18T22:58:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:38.655 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:38.656 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:38.656 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:38.656 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:38.656 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:58:38.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:38 smithi113 ceph-mon[66775]: pgmap v830: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T22:58:38.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:38 smithi103 ceph-mon[66377]: pgmap v830: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T22:58:38.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:38 smithi040 ceph-mon[62643]: pgmap v830: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-18T22:58:39.465 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-18T22:58:39.465 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-18T22:58:39.465 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=0 2024-03-18T22:58:39.465 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=0 2024-03-18T22:58:39.465 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-03-18T22:58:39.465 DEBUG:tasks.check_counter:Getting stats from d 2024-03-18T22:58:39.465 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-18T22:58:39.715 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:58:40.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:40 smithi113 ceph-mon[66775]: pgmap v831: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T22:58:40.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:40 smithi103 ceph-mon[66377]: pgmap v831: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T22:58:40.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:40 smithi040 ceph-mon[62643]: pgmap v831: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T22:58:41.741 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:58:41.741 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T22:58:41.741 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 0, 2024-03-18T22:58:41.742 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 2, 2024-03-18T22:58:41.742 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 0, 2024-03-18T22:58:41.742 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 0, 2024-03-18T22:58:41.742 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-03-18T22:58:41.742 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 0, 2024-03-18T22:58:41.742 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.001192605, 2024-03-18T22:58:41.742 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T22:58:41.742 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.000252882, 2024-03-18T22:58:41.742 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T22:58:41.742 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:41.742 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:41.742 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:41.742 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:41.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:41.743 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:41.743 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:41.743 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:41.743 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:41.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:41.743 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T22:58:41.743 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T22:58:41.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:41.743 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T22:58:41.743 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 303, 2024-03-18T22:58:41.743 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 298, 2024-03-18T22:58:41.744 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 502768, 2024-03-18T22:58:41.744 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 182768, 2024-03-18T22:58:41.744 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 2, 2024-03-18T22:58:41.744 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2024-03-18T22:58:41.744 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.038208967, 2024-03-18T22:58:41.744 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.015574787, 2024-03-18T22:58:41.744 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.016055093, 2024-03-18T22:58:41.744 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.006132710, 2024-03-18T22:58:41.744 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:41.744 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 298, 2024-03-18T22:58:41.744 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.013966399, 2024-03-18T22:58:41.744 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000046867 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 502768, 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 182768 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2, 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 226, 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 320, 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 195520, 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-03-18T22:58:41.745 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 2, 2024-03-18T22:58:41.746 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.016019461, 2024-03-18T22:58:41.746 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.012648932, 2024-03-18T22:58:41.746 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.001927861, 2024-03-18T22:58:41.746 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T22:58:41.746 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:41.746 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 224, 2024-03-18T22:58:41.746 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.010749335, 2024-03-18T22:58:41.746 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000047988 2024-03-18T22:58:41.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:41.746 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:41.746 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:41.746 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:41.746 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:41.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:41.747 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T22:58:41.747 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 195520 2024-03-18T22:58:41.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:41.747 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-03-18T22:58:41.747 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 0, 2024-03-18T22:58:41.747 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-03-18T22:58:41.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:41.747 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-03-18T22:58:41.747 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-03-18T22:58:41.747 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-03-18T22:58:41.747 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T22:58:41.747 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-03-18T22:58:41.747 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T22:58:41.747 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-03-18T22:58:41.747 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T22:58:41.748 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T22:58:41.748 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T22:58:41.748 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T22:58:41.748 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T22:58:41.748 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-03-18T22:58:41.748 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T22:58:41.748 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T22:58:41.748 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-03-18T22:58:41.748 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-03-18T22:58:41.748 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-03-18T22:58:41.748 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-03-18T22:58:41.748 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T22:58:41.748 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T22:58:41.748 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-03-18T22:58:41.749 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 148738, 2024-03-18T22:58:41.750 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 31, 2024-03-18T22:58:41.750 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-03-18T22:58:41.750 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-03-18T22:58:41.750 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-03-18T22:58:41.750 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-03-18T22:58:41.750 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-03-18T22:58:41.750 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-03-18T22:58:41.750 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-03-18T22:58:41.750 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-03-18T22:58:41.750 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 0, 2024-03-18T22:58:41.750 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 0, 2024-03-18T22:58:41.750 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T22:58:41.750 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-03-18T22:58:41.750 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 928, 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 2, 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 301, 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 457532, 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:41.751 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 301, 2024-03-18T22:58:41.752 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-18T22:58:41.752 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-18T22:58:41.752 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 301, 2024-03-18T22:58:41.752 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 457532, 2024-03-18T22:58:41.752 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-18T22:58:41.752 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:41.752 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:41.752 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:41.752 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:41.752 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:41.752 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:58:42.725 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:42 smithi040 ceph-mon[62643]: pgmap v832: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 op/s 2024-03-18T22:58:42.754 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-18T22:58:42.754 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-18T22:58:42.754 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-18T22:58:42.755 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-18T22:58:42.755 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-18T22:58:42.755 DEBUG:tasks.check_counter:Getting stats from g 2024-03-18T22:58:42.755 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-18T22:58:42.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:42 smithi113 ceph-mon[66775]: pgmap v832: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 op/s 2024-03-18T22:58:42.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:42 smithi103 ceph-mon[66377]: pgmap v832: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 11 op/s 2024-03-18T22:58:43.006 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:58:44.798 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:44 smithi040 ceph-mon[62643]: pgmap v833: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T22:58:44.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:44 smithi113 ceph-mon[66775]: pgmap v833: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T22:58:44.835 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:58:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T22:58:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 1024, 2024-03-18T22:58:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 1024, 2024-03-18T22:58:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 345657, 2024-03-18T22:58:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 324809, 2024-03-18T22:58:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 7, 2024-03-18T22:58:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 2, 2024-03-18T22:58:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.117087610, 2024-03-18T22:58:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.041288758, 2024-03-18T22:58:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.026774577, 2024-03-18T22:58:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.040369385, 2024-03-18T22:58:44.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1023, 2024-03-18T22:58:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.060124870, 2024-03-18T22:58:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000058773 2024-03-18T22:58:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-18T22:58:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T22:58:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.837 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T22:58:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2066, 2024-03-18T22:58:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 2287, 2024-03-18T22:58:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 797638, 2024-03-18T22:58:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 1123751, 2024-03-18T22:58:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 5, 2024-03-18T22:58:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 4, 2024-03-18T22:58:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.165638139, 2024-03-18T22:58:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.052794980, 2024-03-18T22:58:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.037417264, 2024-03-18T22:58:44.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.065431691, 2024-03-18T22:58:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2285, 2024-03-18T22:58:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.136944032, 2024-03-18T22:58:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000059931 2024-03-18T22:58:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T22:58:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 467024 2024-03-18T22:58:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T22:58:44.839 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 330, 2024-03-18T22:58:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 305, 2024-03-18T22:58:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 667364, 2024-03-18T22:58:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 182901, 2024-03-18T22:58:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 5, 2024-03-18T22:58:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 3, 2024-03-18T22:58:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.048286812, 2024-03-18T22:58:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.023692472, 2024-03-18T22:58:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.018713417, 2024-03-18T22:58:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.005928228, 2024-03-18T22:58:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 304, 2024-03-18T22:58:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.014314681, 2024-03-18T22:58:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000047087 2024-03-18T22:58:44.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 666016, 2024-03-18T22:58:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 181936 2024-03-18T22:58:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-03-18T22:58:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-03-18T22:58:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-03-18T22:58:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-03-18T22:58:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-18T22:58:44.841 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-18T22:58:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 5178, 2024-03-18T22:58:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.129288160, 2024-03-18T22:58:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000024968 2024-03-18T22:58:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-03-18T22:58:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-18T22:58:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-18T22:58:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.842 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-03-18T22:58:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 0, 2024-03-18T22:58:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 0, 2024-03-18T22:58:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-03-18T22:58:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-03-18T22:58:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 0, 2024-03-18T22:58:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 0, 2024-03-18T22:58:44.843 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2024-03-18T22:58:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 0, 2024-03-18T22:58:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 0, 2024-03-18T22:58:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-03-18T22:58:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 19, 2024-03-18T22:58:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 0, 2024-03-18T22:58:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 19, 2024-03-18T22:58:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 0, 2024-03-18T22:58:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 19, 2024-03-18T22:58:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 0, 2024-03-18T22:58:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 0, 2024-03-18T22:58:44.844 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 0, 2024-03-18T22:58:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 1, 2024-03-18T22:58:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 0, 2024-03-18T22:58:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 0, 2024-03-18T22:58:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 0, 2024-03-18T22:58:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 0, 2024-03-18T22:58:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 0, 2024-03-18T22:58:44.845 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-03-18T22:58:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2024-03-18T22:58:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2024-03-18T22:58:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-03-18T22:58:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 0, 2024-03-18T22:58:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 0, 2024-03-18T22:58:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 0, 2024-03-18T22:58:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 0, 2024-03-18T22:58:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-03-18T22:58:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2024-03-18T22:58:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-03-18T22:58:44.846 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 0, 2024-03-18T22:58:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 0, 2024-03-18T22:58:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 0, 2024-03-18T22:58:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T22:58:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-03-18T22:58:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T22:58:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-03-18T22:58:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-03-18T22:58:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T22:58:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T22:58:44.847 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2024-03-18T22:58:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2024-03-18T22:58:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T22:58:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 0, 2024-03-18T22:58:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T22:58:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T22:58:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T22:58:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 0, 2024-03-18T22:58:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 0, 2024-03-18T22:58:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 0, 2024-03-18T22:58:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 0 2024-03-18T22:58:44.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-03-18T22:58:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-03-18T22:58:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2024-03-18T22:58:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2024-03-18T22:58:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 0, 2024-03-18T22:58:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 0, 2024-03-18T22:58:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2024-03-18T22:58:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-03-18T22:58:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-03-18T22:58:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 0, 2024-03-18T22:58:44.849 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 0, 2024-03-18T22:58:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-03-18T22:58:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-03-18T22:58:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-03-18T22:58:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-03-18T22:58:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-03-18T22:58:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-03-18T22:58:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-03-18T22:58:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_path": 0, 2024-03-18T22:58:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_quiesce_inode": 0, 2024-03-18T22:58:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T22:58:44.850 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 0, 2024-03-18T22:58:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-03-18T22:58:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 0, 2024-03-18T22:58:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-03-18T22:58:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-03-18T22:58:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-03-18T22:58:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-03-18T22:58:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 0, 2024-03-18T22:58:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 0, 2024-03-18T22:58:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 0, 2024-03-18T22:58:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 225, 2024-03-18T22:58:44.851 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-03-18T22:58:44.852 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 0, 2024-03-18T22:58:44.852 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 0, 2024-03-18T22:58:44.852 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 0, 2024-03-18T22:58:44.852 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 0, 2024-03-18T22:58:44.852 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 2, 2024-03-18T22:58:44.852 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 2, 2024-03-18T22:58:44.852 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-03-18T22:58:44.852 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 0, 2024-03-18T22:58:44.852 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 4194304, 2024-03-18T22:58:44.852 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 4467424, 2024-03-18T22:58:44.852 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 4465773, 2024-03-18T22:58:44.853 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-03-18T22:58:44.853 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.853 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.853 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.853 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 225 2024-03-18T22:58:44.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.853 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-03-18T22:58:44.853 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 4, 2024-03-18T22:58:44.853 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 32, 2024-03-18T22:58:44.853 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 28, 2024-03-18T22:58:44.853 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 2, 2024-03-18T22:58:44.854 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 13, 2024-03-18T22:58:44.854 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 11, 2024-03-18T22:58:44.854 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 19, 2024-03-18T22:58:44.854 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 47, 2024-03-18T22:58:44.854 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 28, 2024-03-18T22:58:44.854 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 0, 2024-03-18T22:58:44.854 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 0, 2024-03-18T22:58:44.854 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 0, 2024-03-18T22:58:44.854 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 54036, 2024-03-18T22:58:44.854 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 215420 2024-03-18T22:58:44.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.854 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-03-18T22:58:44.855 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 0, 2024-03-18T22:58:44.855 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 0, 2024-03-18T22:58:44.855 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 0, 2024-03-18T22:58:44.855 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 0, 2024-03-18T22:58:44.855 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 0, 2024-03-18T22:58:44.855 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-03-18T22:58:44.855 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.855 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.855 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.855 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-03-18T22:58:44.855 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.855 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.856 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.856 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-03-18T22:58:44.856 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.856 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.856 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.856 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-03-18T22:58:44.856 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.857 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.857 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.857 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-03-18T22:58:44.857 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.857 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.857 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.857 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-03-18T22:58:44.857 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.858 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.858 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.858 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-03-18T22:58:44.858 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.858 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.858 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.858 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-03-18T22:58:44.859 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.859 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.859 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.859 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-03-18T22:58:44.859 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.859 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.859 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.860 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-03-18T22:58:44.860 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.860 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.860 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.860 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-03-18T22:58:44.860 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.860 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.860 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.861 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-03-18T22:58:44.862 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:44 smithi103 ceph-mon[66377]: pgmap v833: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T22:58:44.862 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.862 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.862 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.863 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-03-18T22:58:44.863 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.863 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.863 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.863 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-03-18T22:58:44.863 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.863 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.863 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.864 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-03-18T22:58:44.864 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.864 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.864 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.864 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-03-18T22:58:44.864 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.864 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.865 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.865 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-03-18T22:58:44.865 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.865 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.865 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.865 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-03-18T22:58:44.865 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.865 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.865 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.865 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-03-18T22:58:44.866 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.866 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.866 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.866 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-03-18T22:58:44.866 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.866 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.866 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.867 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-03-18T22:58:44.867 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.867 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.867 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.867 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-03-18T22:58:44.867 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.867 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.867 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.868 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-03-18T22:58:44.868 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.868 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.868 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.868 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-03-18T22:58:44.868 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.868 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.868 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.869 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-03-18T22:58:44.869 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.869 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.869 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.869 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-03-18T22:58:44.869 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.869 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.869 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.869 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-03-18T22:58:44.869 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.869 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.869 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.870 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-03-18T22:58:44.870 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.870 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.870 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.870 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-03-18T22:58:44.870 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.870 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.870 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.870 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-03-18T22:58:44.870 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-03-18T22:58:44.870 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-03-18T22:58:44.870 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.871 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.871 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.871 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:44.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.871 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-03-18T22:58:44.871 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 1, 2024-03-18T22:58:44.871 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 2, 2024-03-18T22:58:44.871 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 1, 2024-03-18T22:58:44.871 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 1, 2024-03-18T22:58:44.871 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-03-18T22:58:44.872 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 0, 2024-03-18T22:58:44.872 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 0, 2024-03-18T22:58:44.872 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 4486, 2024-03-18T22:58:44.872 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-03-18T22:58:44.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.872 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-03-18T22:58:44.872 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-03-18T22:58:44.872 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-03-18T22:58:44.872 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T22:58:44.872 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-03-18T22:58:44.872 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T22:58:44.872 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-03-18T22:58:44.873 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T22:58:44.873 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T22:58:44.873 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T22:58:44.873 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T22:58:44.873 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T22:58:44.873 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-03-18T22:58:44.873 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T22:58:44.873 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T22:58:44.873 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-03-18T22:58:44.873 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-03-18T22:58:44.874 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-03-18T22:58:44.874 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-03-18T22:58:44.874 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T22:58:44.874 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T22:58:44.874 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T22:58:44.874 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T22:58:44.874 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T22:58:44.874 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-03-18T22:58:44.874 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-03-18T22:58:44.874 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-03-18T22:58:44.874 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T22:58:44.875 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T22:58:44.875 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-03-18T22:58:44.875 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-03-18T22:58:44.875 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-03-18T22:58:44.875 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-03-18T22:58:44.875 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T22:58:44.875 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-03-18T22:58:44.875 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T22:58:44.875 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-03-18T22:58:44.875 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 198247, 2024-03-18T22:58:44.875 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 56, 2024-03-18T22:58:44.875 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-03-18T22:58:44.875 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-03-18T22:58:44.875 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-03-18T22:58:44.875 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-03-18T22:58:44.876 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-03-18T22:58:44.876 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-03-18T22:58:44.876 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-03-18T22:58:44.876 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-03-18T22:58:44.876 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 9184, 2024-03-18T22:58:44.876 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 177, 2024-03-18T22:58:44.876 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T22:58:44.876 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-03-18T22:58:44.876 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-03-18T22:58:44.876 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-03-18T22:58:44.876 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 25360, 2024-03-18T22:58:44.876 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 104, 2024-03-18T22:58:44.876 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-03-18T22:58:44.876 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-03-18T22:58:44.877 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-03-18T22:58:44.877 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-03-18T22:58:44.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.877 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-03-18T22:58:44.877 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-03-18T22:58:44.877 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-03-18T22:58:44.877 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 3088, 2024-03-18T22:58:44.877 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 21, 2024-03-18T22:58:44.877 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 3, 2024-03-18T22:58:44.877 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 3088, 2024-03-18T22:58:44.877 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-03-18T22:58:44.877 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3088, 2024-03-18T22:58:44.877 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 36.727576338, 2024-03-18T22:58:44.877 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.011893645 2024-03-18T22:58:44.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-03-18T22:58:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-03-18T22:58:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3088, 2024-03-18T22:58:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 3089 2024-03-18T22:58:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 3088, 2024-03-18T22:58:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 3088, 2024-03-18T22:58:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 0, 2024-03-18T22:58:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-03-18T22:58:44.878 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-03-18T22:58:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 1975, 2024-03-18T22:58:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 0, 2024-03-18T22:58:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 1111, 2024-03-18T22:58:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 0, 2024-03-18T22:58:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 0, 2024-03-18T22:58:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-03-18T22:58:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-03-18T22:58:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-03-18T22:58:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 0, 2024-03-18T22:58:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 0, 2024-03-18T22:58:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-03-18T22:58:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-03-18T22:58:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-03-18T22:58:44.879 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 1, 2024-03-18T22:58:44.880 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 0, 2024-03-18T22:58:44.880 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-03-18T22:58:44.880 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-03-18T22:58:44.880 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-03-18T22:58:44.880 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-03-18T22:58:44.880 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-03-18T22:58:44.880 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-03-18T22:58:44.880 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T22:58:44.880 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-03-18T22:58:44.880 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-03-18T22:58:44.880 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-03-18T22:58:44.880 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-03-18T22:58:44.880 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-03-18T22:58:44.880 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-03-18T22:58:44.880 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-03-18T22:58:44.881 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-03-18T22:58:44.881 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-03-18T22:58:44.881 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-03-18T22:58:44.881 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-03-18T22:58:44.881 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-03-18T22:58:44.881 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-03-18T22:58:44.881 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-03-18T22:58:44.881 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-03-18T22:58:44.881 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-03-18T22:58:44.881 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-03-18T22:58:44.881 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-03-18T22:58:44.882 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-03-18T22:58:44.882 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 93, 2024-03-18T22:58:44.882 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-03-18T22:58:44.882 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 4, 2024-03-18T22:58:44.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 6, 2024-03-18T22:58:44.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 8, 2024-03-18T22:58:44.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 2, 2024-03-18T22:58:44.882 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-03-18T22:58:44.882 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 0, 2024-03-18T22:58:44.882 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 2, 2024-03-18T22:58:44.882 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 0 2024-03-18T22:58:44.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.882 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-03-18T22:58:44.882 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 0, 2024-03-18T22:58:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-03-18T22:58:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-03-18T22:58:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-03-18T22:58:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-03-18T22:58:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 0 2024-03-18T22:58:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-03-18T22:58:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-03-18T22:58:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T22:58:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-03-18T22:58:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 0, 2024-03-18T22:58:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 0, 2024-03-18T22:58:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 0, 2024-03-18T22:58:44.883 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-03-18T22:58:44.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.884 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T22:58:44.884 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-18T22:58:44.884 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-18T22:58:44.884 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-18T22:58:44.884 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 3416, 2024-03-18T22:58:44.884 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 1483530, 2024-03-18T22:58:44.884 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:44.884 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 3416, 2024-03-18T22:58:44.884 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-18T22:58:44.884 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-18T22:58:44.884 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 3416, 2024-03-18T22:58:44.884 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 1483530, 2024-03-18T22:58:44.884 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-18T22:58:44.885 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.885 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.885 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.885 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:44.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.885 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-03-18T22:58:44.885 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-18T22:58:44.885 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-18T22:58:44.885 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-18T22:58:44.886 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-18T22:58:44.886 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-18T22:58:44.886 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:44.886 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:44.886 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 3088, 2024-03-18T22:58:44.886 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 273125, 2024-03-18T22:58:44.886 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 3088, 2024-03-18T22:58:44.886 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 273125, 2024-03-18T22:58:44.886 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-18T22:58:44.886 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.886 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.886 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.886 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:44.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.887 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-03-18T22:58:44.887 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-18T22:58:44.887 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-03-18T22:58:44.887 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-18T22:58:44.887 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-18T22:58:44.887 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-18T22:58:44.887 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:44.887 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:44.887 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 3088, 2024-03-18T22:58:44.887 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 3088, 2024-03-18T22:58:44.887 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 3088, 2024-03-18T22:58:44.888 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 3088, 2024-03-18T22:58:44.888 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-18T22:58:44.888 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.888 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.888 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.888 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:44.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.888 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-03-18T22:58:44.888 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-18T22:58:44.888 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-18T22:58:44.888 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-18T22:58:44.888 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-18T22:58:44.888 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-18T22:58:44.888 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x55c946f606e0": { 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-18T22:58:44.889 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-18T22:58:44.890 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-18T22:58:44.890 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:44.890 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:44.890 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-18T22:58:44.890 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-18T22:58:44.890 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-03-18T22:58:44.890 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-03-18T22:58:44.890 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-18T22:58:44.890 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:44.890 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:44.890 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:44.890 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:44.890 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:44.890 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:58:45.721 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-18T22:58:45.721 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-18T22:58:45.721 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-03-18T22:58:45.721 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-03-18T22:58:45.721 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-03-18T22:58:45.721 DEBUG:tasks.check_counter:Getting stats from j 2024-03-18T22:58:45.722 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-18T22:58:45.983 INFO:teuthology.orchestra.run.smithi040.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T22:58:46.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:46 smithi040 ceph-mon[62643]: pgmap v834: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T22:58:46.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:46 smithi113 ceph-mon[66775]: pgmap v834: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T22:58:46.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:46 smithi103 ceph-mon[66377]: pgmap v834: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T22:58:47.836 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-18T22:58:47.836 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T22:58:47.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 4, 2024-03-18T22:58:47.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 4, 2024-03-18T22:58:47.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 2992, 2024-03-18T22:58:47.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 448, 2024-03-18T22:58:47.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-03-18T22:58:47.836 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 0, 2024-03-18T22:58:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.001479421, 2024-03-18T22:58:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000035908, 2024-03-18T22:58:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.000622583, 2024-03-18T22:58:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T22:58:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3, 2024-03-18T22:58:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000850073, 2024-03-18T22:58:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000283357 2024-03-18T22:58:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-18T22:58:47.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T22:58:47.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:47.838 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T22:58:47.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2, 2024-03-18T22:58:47.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 228, 2024-03-18T22:58:47.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 320, 2024-03-18T22:58:47.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 195872, 2024-03-18T22:58:47.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 4, 2024-03-18T22:58:47.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 2, 2024-03-18T22:58:47.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.016535549, 2024-03-18T22:58:47.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.013067267, 2024-03-18T22:58:47.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.001518587, 2024-03-18T22:58:47.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T22:58:47.838 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:47.839 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 226, 2024-03-18T22:58:47.839 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.013384609, 2024-03-18T22:58:47.839 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000059223 2024-03-18T22:58:47.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:47.839 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:47.839 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:47.839 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:47.839 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:47.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:47.839 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T22:58:47.839 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 195872 2024-03-18T22:58:47.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:47.839 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T22:58:47.839 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 301, 2024-03-18T22:58:47.839 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 298, 2024-03-18T22:58:47.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 500320, 2024-03-18T22:58:47.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 183600, 2024-03-18T22:58:47.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 2, 2024-03-18T22:58:47.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2024-03-18T22:58:47.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.039660503, 2024-03-18T22:58:47.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.016754556, 2024-03-18T22:58:47.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.016418731, 2024-03-18T22:58:47.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.006180951, 2024-03-18T22:58:47.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:47.840 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 297, 2024-03-18T22:58:47.840 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.019632621, 2024-03-18T22:58:47.840 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000066103 2024-03-18T22:58:47.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:47.840 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:47.840 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:47.841 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:47.841 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:47.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:47.841 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 500320, 2024-03-18T22:58:47.841 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 183600 2024-03-18T22:58:47.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:47.841 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-03-18T22:58:47.841 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 0, 2024-03-18T22:58:47.841 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-03-18T22:58:47.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:47.841 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-03-18T22:58:47.841 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-03-18T22:58:47.841 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-03-18T22:58:47.841 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T22:58:47.842 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-03-18T22:58:47.842 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T22:58:47.842 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-03-18T22:58:47.842 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T22:58:47.842 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T22:58:47.842 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T22:58:47.842 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T22:58:47.842 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T22:58:47.842 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-03-18T22:58:47.842 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T22:58:47.842 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T22:58:47.842 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-03-18T22:58:47.842 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-03-18T22:58:47.842 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-03-18T22:58:47.843 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-03-18T22:58:47.843 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T22:58:47.843 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T22:58:47.843 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T22:58:47.843 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T22:58:47.843 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T22:58:47.843 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-03-18T22:58:47.843 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-03-18T22:58:47.843 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-03-18T22:58:47.843 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T22:58:47.843 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T22:58:47.843 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-03-18T22:58:47.843 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-03-18T22:58:47.843 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-03-18T22:58:47.844 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-03-18T22:58:47.844 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T22:58:47.844 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-03-18T22:58:47.844 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T22:58:47.844 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-03-18T22:58:47.844 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 148704, 2024-03-18T22:58:47.844 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 29, 2024-03-18T22:58:47.844 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-03-18T22:58:47.844 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-03-18T22:58:47.844 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-03-18T22:58:47.844 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-03-18T22:58:47.844 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-03-18T22:58:47.844 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-03-18T22:58:47.844 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-03-18T22:58:47.845 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-03-18T22:58:47.845 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 0, 2024-03-18T22:58:47.845 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 0, 2024-03-18T22:58:47.845 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T22:58:47.845 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-03-18T22:58:47.845 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-03-18T22:58:47.845 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-03-18T22:58:47.845 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 928, 2024-03-18T22:58:47.845 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 2, 2024-03-18T22:58:47.845 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-03-18T22:58:47.845 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-03-18T22:58:47.845 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-03-18T22:58:47.845 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-03-18T22:58:47.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-18T22:58:47.845 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T22:58:47.846 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-18T22:58:47.846 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-18T22:58:47.846 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-18T22:58:47.846 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 303, 2024-03-18T22:58:47.846 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 457792, 2024-03-18T22:58:47.846 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:47.846 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 303, 2024-03-18T22:58:47.846 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-18T22:58:47.846 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-18T22:58:47.846 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 303, 2024-03-18T22:58:47.846 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 457792, 2024-03-18T22:58:47.846 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-18T22:58:47.846 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-18T22:58:47.846 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-18T22:58:47.847 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-18T22:58:47.847 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:47.847 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-18T22:58:47.847 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-18T22:58:48.539 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-18T22:58:48.539 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-18T22:58:48.539 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-18T22:58:48.539 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-18T22:58:48.540 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-18T22:58:48.540 DEBUG:tasks.check_counter:Getting stats from b 2024-03-18T22:58:48.540 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-18T22:58:48.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:48 smithi113 ceph-mon[66775]: pgmap v835: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T22:58:48.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:48 smithi103 ceph-mon[66377]: pgmap v835: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T22:58:48.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:48 smithi040 ceph-mon[62643]: pgmap v835: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T22:58:49.045 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T22:58:50.610 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:50 smithi103 ceph-mon[66377]: pgmap v836: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:58:50.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:50 smithi113 ceph-mon[66775]: pgmap v836: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:58:50.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:50 smithi040 ceph-mon[62643]: pgmap v836: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:58:51.911 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-18T22:58:51.911 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T22:58:51.911 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 6491, 2024-03-18T22:58:51.911 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 6998, 2024-03-18T22:58:51.911 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 2176247, 2024-03-18T22:58:51.911 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 12135249, 2024-03-18T22:58:51.912 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 11, 2024-03-18T22:58:51.912 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 5, 2024-03-18T22:58:51.912 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.658875927, 2024-03-18T22:58:51.912 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.163882253, 2024-03-18T22:58:51.912 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.185691752, 2024-03-18T22:58:51.912 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.279447353, 2024-03-18T22:58:51.912 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:51.912 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 6996, 2024-03-18T22:58:51.912 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 1.142472596, 2024-03-18T22:58:51.912 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000163303 2024-03-18T22:58:51.912 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.912 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:51.913 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 5658, 2024-03-18T22:58:51.913 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.001171972, 2024-03-18T22:58:51.913 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000207 2024-03-18T22:58:51.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.913 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T22:58:51.913 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T22:58:51.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.913 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T22:58:51.913 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 6534, 2024-03-18T22:58:51.913 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 9328, 2024-03-18T22:58:51.913 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 1962823, 2024-03-18T22:58:51.913 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 3859553, 2024-03-18T22:58:51.914 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 11, 2024-03-18T22:58:51.914 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 5, 2024-03-18T22:58:51.914 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.808084173, 2024-03-18T22:58:51.914 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.339302784, 2024-03-18T22:58:51.914 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.233663386, 2024-03-18T22:58:51.914 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.237529916, 2024-03-18T22:58:51.914 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:51.914 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 9329, 2024-03-18T22:58:51.914 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 3.370778106, 2024-03-18T22:58:51.914 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000361322 2024-03-18T22:58:51.914 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.914 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:51.914 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 10684, 2024-03-18T22:58:51.915 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.002414311, 2024-03-18T22:58:51.915 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000225 2024-03-18T22:58:51.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.915 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T22:58:51.915 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 573280 2024-03-18T22:58:51.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.915 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T22:58:51.915 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 13217, 2024-03-18T22:58:51.915 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 16316, 2024-03-18T22:58:51.915 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 4840923, 2024-03-18T22:58:51.915 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 10960723, 2024-03-18T22:58:51.915 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 205, 2024-03-18T22:58:51.916 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 8, 2024-03-18T22:58:51.916 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 1.496486128, 2024-03-18T22:58:51.916 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.532354256, 2024-03-18T22:58:51.916 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.559414208, 2024-03-18T22:58:51.916 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.373940352, 2024-03-18T22:58:51.916 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:51.916 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 16313, 2024-03-18T22:58:51.916 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 2.489643014, 2024-03-18T22:58:51.916 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000152617 2024-03-18T22:58:51.916 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.916 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:51.917 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 22404, 2024-03-18T22:58:51.917 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.004066850, 2024-03-18T22:58:51.917 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000181 2024-03-18T22:58:51.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.917 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 771120, 2024-03-18T22:58:51.917 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 245424 2024-03-18T22:58:51.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.917 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-03-18T22:58:51.917 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 1, 2024-03-18T22:58:51.917 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-03-18T22:58:51.917 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.917 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-MDSRank": { 2024-03-18T22:58:51.917 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-03-18T22:58:51.918 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-03-18T22:58:51.918 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 4208, 2024-03-18T22:58:51.918 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 4.359564715, 2024-03-18T22:58:51.918 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.001036018 2024-03-18T22:58:51.918 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:51.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.918 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-PurgeQueue": { 2024-03-18T22:58:51.918 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-03-18T22:58:51.918 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-03-18T22:58:51.918 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2559, 2024-03-18T22:58:51.918 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.232411266, 2024-03-18T22:58:51.918 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000090821 2024-03-18T22:58:51.918 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:51.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.919 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-18T22:58:51.919 INFO:teuthology.orchestra.run.smithi103.stdout: "request": 3788, 2024-03-18T22:58:51.919 INFO:teuthology.orchestra.run.smithi103.stdout: "reply": 3620, 2024-03-18T22:58:51.919 INFO:teuthology.orchestra.run.smithi103.stdout: "reply_latency": { 2024-03-18T22:58:51.919 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 3620, 2024-03-18T22:58:51.919 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 24.148213743, 2024-03-18T22:58:51.919 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.006670777 2024-03-18T22:58:51.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.919 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_reply": 0, 2024-03-18T22:58:51.919 INFO:teuthology.orchestra.run.smithi103.stdout: "forward": 168, 2024-03-18T22:58:51.919 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_complete": 0, 2024-03-18T22:58:51.919 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_keys": 0, 2024-03-18T22:58:51.919 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_commit": 13, 2024-03-18T22:58:51.920 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_split": 0, 2024-03-18T22:58:51.920 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_merge": 1, 2024-03-18T22:58:51.920 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes": 98, 2024-03-18T22:58:51.920 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_top": 6, 2024-03-18T22:58:51.920 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_bottom": 9, 2024-03-18T22:58:51.920 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pin_tail": 83, 2024-03-18T22:58:51.920 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pinned": 89, 2024-03-18T22:58:51.920 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_expired": 321, 2024-03-18T22:58:51.920 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_with_caps": 2, 2024-03-18T22:58:51.920 INFO:teuthology.orchestra.run.smithi103.stdout: "caps": 2, 2024-03-18T22:58:51.920 INFO:teuthology.orchestra.run.smithi103.stdout: "subtrees": 3, 2024-03-18T22:58:51.920 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse": 5935, 2024-03-18T22:58:51.920 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_hit": 5361, 2024-03-18T22:58:51.920 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_forward": 167, 2024-03-18T22:58:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_discover": 8, 2024-03-18T22:58:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_dir_fetch": 0, 2024-03-18T22:58:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_remote_ino": 0, 2024-03-18T22:58:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_lock": 47, 2024-03-18T22:58:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "load_cent": 1, 2024-03-18T22:58:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "q": 0, 2024-03-18T22:58:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "exported": 6, 2024-03-18T22:58:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "exported_inodes": 303, 2024-03-18T22:58:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "imported": 7, 2024-03-18T22:58:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "imported_inodes": 294, 2024-03-18T22:58:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_dir_fetch": 0, 2024-03-18T22:58:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_backtrace_fetch": 3849, 2024-03-18T22:58:51.921 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_peer_discover": 0, 2024-03-18T22:58:51.922 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rfiles": 4, 2024-03-18T22:58:51.922 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rbytes": 234, 2024-03-18T22:58:51.922 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rsnaps": 0, 2024-03-18T22:58:51.922 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_fetch": 3849, 2024-03-18T22:58:51.922 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_set_tag": 0, 2024-03-18T22:58:51.922 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T22:58:51.922 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_inotable_repaired": 0, 2024-03-18T22:58:51.922 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_inodes": 977, 2024-03-18T22:58:51.923 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_base_inodes": 22, 2024-03-18T22:58:51.923 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dirfrag_rstats": 955, 2024-03-18T22:58:51.923 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_file_inodes": 2872, 2024-03-18T22:58:51.923 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_inode_file_caps": 0, 2024-03-18T22:58:51.923 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_revoke": 1254, 2024-03-18T22:58:51.923 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_grant": 1757, 2024-03-18T22:58:51.923 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_trunc": 6, 2024-03-18T22:58:51.923 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T22:58:51.923 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T22:58:51.923 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps": 3140, 2024-03-18T22:58:51.923 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps_dirty": 976, 2024-03-18T22:58:51.923 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_cap_release": 31, 2024-03-18T22:58:51.923 INFO:teuthology.orchestra.run.smithi103.stdout: "process_request_cap_release": 2545 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache": { 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update": 361, 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update_receipt": 0, 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_try_discover": 0, 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_send_discover": 8, 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_handle_discover": 743, 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays": 5, 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_delayed": 0, 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_enqueuing": 0, 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_created": 828, 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_enqueued": 762, 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_reintegrated": 61, 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_migrated": 0, 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_processing": 0, 2024-03-18T22:58:51.924 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_enqueued": 0, 2024-03-18T22:58:51.925 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_prioritized": 0, 2024-03-18T22:58:51.925 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_started": 0, 2024-03-18T22:58:51.925 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_completed": 0, 2024-03-18T22:58:51.925 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_path": 0, 2024-03-18T22:58:51.925 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_inode": 0, 2024-03-18T22:58:51.925 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_enqueue_scrub": 22, 2024-03-18T22:58:51.925 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_exportdir": 6, 2024-03-18T22:58:51.925 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_flush": 0, 2024-03-18T22:58:51.925 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragmentdir": 1, 2024-03-18T22:58:51.925 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragstats": 0, 2024-03-18T22:58:51.926 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_inodestats": 0 2024-03-18T22:58:51.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.926 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log": { 2024-03-18T22:58:51.926 INFO:teuthology.orchestra.run.smithi103.stdout: "evlrg": 0, 2024-03-18T22:58:51.926 INFO:teuthology.orchestra.run.smithi103.stdout: "evadd": 4551, 2024-03-18T22:58:51.926 INFO:teuthology.orchestra.run.smithi103.stdout: "evex": 0, 2024-03-18T22:58:51.926 INFO:teuthology.orchestra.run.smithi103.stdout: "evtrm": 0, 2024-03-18T22:58:51.926 INFO:teuthology.orchestra.run.smithi103.stdout: "ev": 4551, 2024-03-18T22:58:51.926 INFO:teuthology.orchestra.run.smithi103.stdout: "evexg": 0, 2024-03-18T22:58:51.926 INFO:teuthology.orchestra.run.smithi103.stdout: "evexd": 0, 2024-03-18T22:58:51.926 INFO:teuthology.orchestra.run.smithi103.stdout: "segadd": 6, 2024-03-18T22:58:51.926 INFO:teuthology.orchestra.run.smithi103.stdout: "segex": 0, 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: "segtrm": 0, 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: "seg": 6, 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: "segmjr": 2, 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: "segexg": 0, 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: "segexd": 0, 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: "expos": 4194304, 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: "wrpos": 13863838, 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: "rdpos": 0, 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: "jlat": { 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 836, 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 20.681892467, 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.024739105 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: "replayed": 0 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.927 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_mem": { 2024-03-18T22:58:51.928 INFO:teuthology.orchestra.run.smithi103.stdout: "ino": 33, 2024-03-18T22:58:51.928 INFO:teuthology.orchestra.run.smithi103.stdout: "ino+": 1015, 2024-03-18T22:58:51.928 INFO:teuthology.orchestra.run.smithi103.stdout: "ino-": 982, 2024-03-18T22:58:51.928 INFO:teuthology.orchestra.run.smithi103.stdout: "dir": 25, 2024-03-18T22:58:51.928 INFO:teuthology.orchestra.run.smithi103.stdout: "dir+": 208, 2024-03-18T22:58:51.928 INFO:teuthology.orchestra.run.smithi103.stdout: "dir-": 183, 2024-03-18T22:58:51.928 INFO:teuthology.orchestra.run.smithi103.stdout: "dn": 98, 2024-03-18T22:58:51.928 INFO:teuthology.orchestra.run.smithi103.stdout: "dn+": 1628, 2024-03-18T22:58:51.928 INFO:teuthology.orchestra.run.smithi103.stdout: "dn-": 1530, 2024-03-18T22:58:51.928 INFO:teuthology.orchestra.run.smithi103.stdout: "cap": 2, 2024-03-18T22:58:51.928 INFO:teuthology.orchestra.run.smithi103.stdout: "cap+": 973, 2024-03-18T22:58:51.928 INFO:teuthology.orchestra.run.smithi103.stdout: "cap-": 971, 2024-03-18T22:58:51.928 INFO:teuthology.orchestra.run.smithi103.stdout: "rss": 64524, 2024-03-18T22:58:51.929 INFO:teuthology.orchestra.run.smithi103.stdout: "heap": 231804 2024-03-18T22:58:51.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.929 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_server": { 2024-03-18T22:58:51.929 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_client_request": 5088, 2024-03-18T22:58:51.929 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_server_request": 9, 2024-03-18T22:58:51.929 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_request": 3788, 2024-03-18T22:58:51.929 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_session": 27, 2024-03-18T22:58:51.929 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_peer_request": 72, 2024-03-18T22:58:51.929 INFO:teuthology.orchestra.run.smithi103.stdout: "req_create_latency": { 2024-03-18T22:58:51.929 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 644, 2024-03-18T22:58:51.929 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.930375584, 2024-03-18T22:58:51.930 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.001444682 2024-03-18T22:58:51.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.930 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getattr_latency": { 2024-03-18T22:58:51.930 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1199, 2024-03-18T22:58:51.930 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.835547584, 2024-03-18T22:58:51.930 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000696870 2024-03-18T22:58:51.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.930 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getfilelock_latency": { 2024-03-18T22:58:51.930 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.930 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.930 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.930 INFO:teuthology.orchestra.run.smithi103.stdout: "req_link_latency": { 2024-03-18T22:58:51.931 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 61, 2024-03-18T22:58:51.931 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.207318643, 2024-03-18T22:58:51.931 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.003398666 2024-03-18T22:58:51.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.931 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookup_latency": { 2024-03-18T22:58:51.931 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 294, 2024-03-18T22:58:51.931 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 4.376500197, 2024-03-18T22:58:51.931 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.014886055 2024-03-18T22:58:51.931 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.931 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookuphash_latency": { 2024-03-18T22:58:51.931 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.931 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.932 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.932 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupino_latency": { 2024-03-18T22:58:51.932 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.932 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.932 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.932 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupname_latency": { 2024-03-18T22:58:51.932 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.932 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.932 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.932 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.933 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupparent_latency": { 2024-03-18T22:58:51.933 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.933 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.933 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.933 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupsnap_latency": { 2024-03-18T22:58:51.933 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.933 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.933 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.933 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lssnap_latency": { 2024-03-18T22:58:51.933 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.934 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.934 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.935 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mkdir_latency": { 2024-03-18T22:58:51.935 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 105, 2024-03-18T22:58:51.935 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.153492632, 2024-03-18T22:58:51.935 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.001461834 2024-03-18T22:58:51.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.935 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mknod_latency": { 2024-03-18T22:58:51.935 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.935 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.935 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.936 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mksnap_latency": { 2024-03-18T22:58:51.936 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.936 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.936 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.936 INFO:teuthology.orchestra.run.smithi103.stdout: "req_open_latency": { 2024-03-18T22:58:51.936 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 209, 2024-03-18T22:58:51.936 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.174368541, 2024-03-18T22:58:51.936 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000834299 2024-03-18T22:58:51.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.936 INFO:teuthology.orchestra.run.smithi103.stdout: "req_readdir_latency": { 2024-03-18T22:58:51.936 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 170, 2024-03-18T22:58:51.936 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.937175659, 2024-03-18T22:58:51.937 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.005512797 2024-03-18T22:58:51.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.937 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rename_latency": { 2024-03-18T22:58:51.937 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 113, 2024-03-18T22:58:51.937 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 7.998107808, 2024-03-18T22:58:51.937 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.070779715 2024-03-18T22:58:51.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.937 INFO:teuthology.orchestra.run.smithi103.stdout: "req_renamesnap_latency": { 2024-03-18T22:58:51.937 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.937 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.937 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.938 INFO:teuthology.orchestra.run.smithi103.stdout: "req_snapdiff_latency": { 2024-03-18T22:58:51.938 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.938 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.938 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.938 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmdir_latency": { 2024-03-18T22:58:51.938 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 97, 2024-03-18T22:58:51.938 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 6.180936577, 2024-03-18T22:58:51.938 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.063720995 2024-03-18T22:58:51.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.939 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmsnap_latency": { 2024-03-18T22:58:51.939 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.939 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.939 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.939 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmxattr_latency": { 2024-03-18T22:58:51.939 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.939 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.939 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.939 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setattr_latency": { 2024-03-18T22:58:51.939 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 5, 2024-03-18T22:58:51.940 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.006725503, 2024-03-18T22:58:51.940 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.001345100 2024-03-18T22:58:51.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.940 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setdirlayout_latency": { 2024-03-18T22:58:51.940 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.940 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.940 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.940 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setfilelock_latency": { 2024-03-18T22:58:51.940 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 4, 2024-03-18T22:58:51.941 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.002051055, 2024-03-18T22:58:51.941 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000512763 2024-03-18T22:58:51.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.941 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setlayout_latency": { 2024-03-18T22:58:51.941 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.941 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.941 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.941 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.941 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setxattr_latency": { 2024-03-18T22:58:51.941 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 8, 2024-03-18T22:58:51.941 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.020576794, 2024-03-18T22:58:51.941 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.002572099 2024-03-18T22:58:51.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.942 INFO:teuthology.orchestra.run.smithi103.stdout: "req_symlink_latency": { 2024-03-18T22:58:51.942 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 13, 2024-03-18T22:58:51.942 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.014972092, 2024-03-18T22:58:51.942 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.001151699 2024-03-18T22:58:51.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.942 INFO:teuthology.orchestra.run.smithi103.stdout: "req_unlink_latency": { 2024-03-18T22:58:51.942 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 698, 2024-03-18T22:58:51.942 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 2.310065074, 2024-03-18T22:58:51.942 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.003309548 2024-03-18T22:58:51.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.943 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_revoke_eviction": 0, 2024-03-18T22:58:51.943 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_acquisition_throttle": 0, 2024-03-18T22:58:51.943 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getvxattr_latency": { 2024-03-18T22:58:51.943 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.943 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.943 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.943 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:51.943 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.943 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_sessions": { 2024-03-18T22:58:51.943 INFO:teuthology.orchestra.run.smithi103.stdout: "session_count": 1, 2024-03-18T22:58:51.943 INFO:teuthology.orchestra.run.smithi103.stdout: "session_add": 3, 2024-03-18T22:58:51.944 INFO:teuthology.orchestra.run.smithi103.stdout: "session_remove": 2, 2024-03-18T22:58:51.944 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_open": 1, 2024-03-18T22:58:51.944 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_stale": 0, 2024-03-18T22:58:51.944 INFO:teuthology.orchestra.run.smithi103.stdout: "total_load": 766, 2024-03-18T22:58:51.944 INFO:teuthology.orchestra.run.smithi103.stdout: "average_load": 766, 2024-03-18T22:58:51.944 INFO:teuthology.orchestra.run.smithi103.stdout: "avg_session_uptime": 946, 2024-03-18T22:58:51.944 INFO:teuthology.orchestra.run.smithi103.stdout: "mdthresh_evicted": 0 2024-03-18T22:58:51.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.944 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-03-18T22:58:51.944 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 0, 2024-03-18T22:58:51.944 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 0, 2024-03-18T22:58:51.944 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T22:58:51.945 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-03-18T22:58:51.945 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T22:58:51.945 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-03-18T22:58:51.945 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T22:58:51.945 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T22:58:51.945 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T22:58:51.945 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T22:58:51.945 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T22:58:51.945 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-03-18T22:58:51.945 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T22:58:51.945 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T22:58:51.945 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-03-18T22:58:51.946 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-03-18T22:58:51.946 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-03-18T22:58:51.946 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-03-18T22:58:51.946 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T22:58:51.946 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T22:58:51.946 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T22:58:51.946 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T22:58:51.946 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T22:58:51.947 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-03-18T22:58:51.947 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-03-18T22:58:51.947 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-03-18T22:58:51.947 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T22:58:51.947 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T22:58:51.947 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-03-18T22:58:51.947 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-03-18T22:58:51.947 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-03-18T22:58:51.947 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-03-18T22:58:51.947 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T22:58:51.947 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-03-18T22:58:51.947 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T22:58:51.948 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-03-18T22:58:51.948 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 316221, 2024-03-18T22:58:51.948 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 123, 2024-03-18T22:58:51.948 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 0, 2024-03-18T22:58:51.948 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 0, 2024-03-18T22:58:51.948 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-03-18T22:58:51.948 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-03-18T22:58:51.948 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-03-18T22:58:51.948 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-03-18T22:58:51.948 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-03-18T22:58:51.948 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-03-18T22:58:51.949 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 9184, 2024-03-18T22:58:51.949 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 177, 2024-03-18T22:58:51.949 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T22:58:51.949 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-03-18T22:58:51.949 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-03-18T22:58:51.949 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-03-18T22:58:51.949 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 186127, 2024-03-18T22:58:51.949 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 2104, 2024-03-18T22:58:51.949 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-03-18T22:58:51.949 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-03-18T22:58:51.949 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-03-18T22:58:51.949 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-03-18T22:58:51.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.949 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter": { 2024-03-18T22:58:51.949 INFO:teuthology.orchestra.run.smithi103.stdout: "op_active": 0, 2024-03-18T22:58:51.950 INFO:teuthology.orchestra.run.smithi103.stdout: "op_laggy": 0, 2024-03-18T22:58:51.950 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send": 8239, 2024-03-18T22:58:51.950 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send_bytes": 9847194, 2024-03-18T22:58:51.950 INFO:teuthology.orchestra.run.smithi103.stdout: "op_resend": 31, 2024-03-18T22:58:51.950 INFO:teuthology.orchestra.run.smithi103.stdout: "op_reply": 8239, 2024-03-18T22:58:51.950 INFO:teuthology.orchestra.run.smithi103.stdout: "op_latency": { 2024-03-18T22:58:51.950 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 8239, 2024-03-18T22:58:51.950 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 180.227482216, 2024-03-18T22:58:51.950 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.021874921 2024-03-18T22:58:51.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.950 INFO:teuthology.orchestra.run.smithi103.stdout: "op_inflight": 0, 2024-03-18T22:58:51.950 INFO:teuthology.orchestra.run.smithi103.stdout: "oplen_avg": { 2024-03-18T22:58:51.950 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 8239, 2024-03-18T22:58:51.950 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 8434 2024-03-18T22:58:51.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.951 INFO:teuthology.orchestra.run.smithi103.stdout: "op": 8239, 2024-03-18T22:58:51.951 INFO:teuthology.orchestra.run.smithi103.stdout: "op_r": 3946, 2024-03-18T22:58:51.951 INFO:teuthology.orchestra.run.smithi103.stdout: "op_w": 4293, 2024-03-18T22:58:51.951 INFO:teuthology.orchestra.run.smithi103.stdout: "op_rmw": 0, 2024-03-18T22:58:51.951 INFO:teuthology.orchestra.run.smithi103.stdout: "op_pg": 0, 2024-03-18T22:58:51.951 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_stat": 0, 2024-03-18T22:58:51.951 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_create": 11, 2024-03-18T22:58:51.951 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_read": 75, 2024-03-18T22:58:51.951 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_write": 889, 2024-03-18T22:58:51.951 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writefull": 84, 2024-03-18T22:58:51.951 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writesame": 0, 2024-03-18T22:58:51.951 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_append": 0, 2024-03-18T22:58:51.951 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_zero": 0, 2024-03-18T22:58:51.951 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_truncate": 0, 2024-03-18T22:58:51.952 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_delete": 3159, 2024-03-18T22:58:51.952 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_mapext": 0, 2024-03-18T22:58:51.952 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_sparse_read": 0, 2024-03-18T22:58:51.952 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_clonerange": 0, 2024-03-18T22:58:51.952 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_getxattr": 3871, 2024-03-18T22:58:51.952 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_setxattr": 22, 2024-03-18T22:58:51.952 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_cmpxattr": 0, 2024-03-18T22:58:51.952 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_rmxattr": 0, 2024-03-18T22:58:51.952 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_resetxattrs": 0, 2024-03-18T22:58:51.952 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_call": 0, 2024-03-18T22:58:51.952 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_watch": 0, 2024-03-18T22:58:51.952 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_notify": 0, 2024-03-18T22:58:51.952 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T22:58:51.953 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls": 0, 2024-03-18T22:58:51.953 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls_filter": 0, 2024-03-18T22:58:51.953 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_other": 6, 2024-03-18T22:58:51.953 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_active": 0, 2024-03-18T22:58:51.953 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_send": 0, 2024-03-18T22:58:51.953 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_resend": 0, 2024-03-18T22:58:51.953 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_ping": 0, 2024-03-18T22:58:51.953 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_active": 0, 2024-03-18T22:58:51.953 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_send": 0, 2024-03-18T22:58:51.954 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_resend": 0, 2024-03-18T22:58:51.954 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_active": 0, 2024-03-18T22:58:51.954 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_send": 0, 2024-03-18T22:58:51.954 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_resend": 0, 2024-03-18T22:58:51.954 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_active": 0, 2024-03-18T22:58:51.954 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_send": 0, 2024-03-18T22:58:51.954 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_resend": 0, 2024-03-18T22:58:51.954 INFO:teuthology.orchestra.run.smithi103.stdout: "command_active": 0, 2024-03-18T22:58:51.954 INFO:teuthology.orchestra.run.smithi103.stdout: "command_send": 0, 2024-03-18T22:58:51.954 INFO:teuthology.orchestra.run.smithi103.stdout: "command_resend": 0, 2024-03-18T22:58:51.954 INFO:teuthology.orchestra.run.smithi103.stdout: "map_epoch": 93, 2024-03-18T22:58:51.954 INFO:teuthology.orchestra.run.smithi103.stdout: "map_full": 0, 2024-03-18T22:58:51.954 INFO:teuthology.orchestra.run.smithi103.stdout: "map_inc": 9, 2024-03-18T22:58:51.954 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_sessions": 12, 2024-03-18T22:58:51.955 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_open": 12, 2024-03-18T22:58:51.955 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_close": 0, 2024-03-18T22:58:51.955 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_laggy": 0, 2024-03-18T22:58:51.955 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_wr": 241, 2024-03-18T22:58:51.955 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_rd": 0, 2024-03-18T22:58:51.955 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_del": 76 2024-03-18T22:58:51.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.955 INFO:teuthology.orchestra.run.smithi103.stdout: "oft": { 2024-03-18T22:58:51.955 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_objs": 1, 2024-03-18T22:58:51.955 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_kv_pairs": 0, 2024-03-18T22:58:51.955 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_updates": 0, 2024-03-18T22:58:51.955 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_removes": 0 2024-03-18T22:58:51.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.955 INFO:teuthology.orchestra.run.smithi103.stdout: "osd-slow-ops": { 2024-03-18T22:58:51.955 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_ops_count": 0 2024-03-18T22:58:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: "purge_queue": { 2024-03-18T22:58:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops": 0, 2024-03-18T22:58:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops_high_water": 52, 2024-03-18T22:58:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing": 0, 2024-03-18T22:58:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_high_water": 7, 2024-03-18T22:58:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed_ops": 2643, 2024-03-18T22:58:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed": 762, 2024-03-18T22:58:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_item_in_journal": 0 2024-03-18T22:58:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T22:58:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:58:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-03-18T22:58:51.956 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:58:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 26238, 2024-03-18T22:58:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 6589424, 2024-03-18T22:58:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 26238, 2024-03-18T22:58:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-03-18T22:58:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-03-18T22:58:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 26238, 2024-03-18T22:58:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 6589424, 2024-03-18T22:58:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:58:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.957 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_bytes": { 2024-03-18T22:58:51.958 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:58:51.958 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-03-18T22:58:51.958 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:58:51.958 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-03-18T22:58:51.958 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-03-18T22:58:51.958 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:51.958 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:51.958 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 8239, 2024-03-18T22:58:51.958 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 9929412, 2024-03-18T22:58:51.958 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 8239, 2024-03-18T22:58:51.958 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 9929412, 2024-03-18T22:58:51.958 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:58:51.958 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.958 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.959 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.959 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:51.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.959 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_ops": { 2024-03-18T22:58:51.959 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:58:51.959 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 1024, 2024-03-18T22:58:51.959 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:58:51.959 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-03-18T22:58:51.959 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-03-18T22:58:51.959 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:51.959 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:51.959 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 8239, 2024-03-18T22:58:51.959 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 8239, 2024-03-18T22:58:51.959 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 8239, 2024-03-18T22:58:51.959 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 8239, 2024-03-18T22:58:51.960 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:58:51.960 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.960 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.960 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.960 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:51.960 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.960 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle": { 2024-03-18T22:58:51.960 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:58:51.960 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-03-18T22:58:51.960 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:58:51.960 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 762, 2024-03-18T22:58:51.960 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 82218, 2024-03-18T22:58:51.960 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:51.960 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 762, 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 53, 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 82218, 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle-0x562ede27a500": { 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 4551, 2024-03-18T22:58:51.961 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 9669534, 2024-03-18T22:58:51.962 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:51.962 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 4551, 2024-03-18T22:58:51.962 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-03-18T22:58:51.962 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-03-18T22:58:51.962 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 836, 2024-03-18T22:58:51.962 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 9669534, 2024-03-18T22:58:51.962 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:58:51.962 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:51.962 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:51.962 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:51.962 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:51.962 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:51.962 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-18T22:58:52.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:52 smithi113 ceph-mon[66775]: pgmap v837: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-18T22:58:52.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:52 smithi103 ceph-mon[66377]: pgmap v837: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-18T22:58:52.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:52 smithi040 ceph-mon[62643]: pgmap v837: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-18T22:58:53.065 INFO:tasks.check_counter:Daemon mds.b mds.exported=6 2024-03-18T22:58:53.065 INFO:tasks.check_counter:Daemon mds.b mds.imported=7 2024-03-18T22:58:53.065 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=361 2024-03-18T22:58:53.065 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-03-18T22:58:53.065 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=0 2024-03-18T22:58:53.065 DEBUG:tasks.check_counter:Getting stats from e 2024-03-18T22:58:53.065 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-18T22:58:53.346 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T22:58:54.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:54 smithi113 ceph-mon[66775]: pgmap v838: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:58:54.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:54 smithi103 ceph-mon[66377]: pgmap v838: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:58:54.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:54 smithi040 ceph-mon[62643]: pgmap v838: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:58:55.386 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-18T22:58:55.387 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T22:58:55.387 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 339, 2024-03-18T22:58:55.387 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 309, 2024-03-18T22:58:55.387 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 693906, 2024-03-18T22:58:55.387 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 186100, 2024-03-18T22:58:55.387 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 6, 2024-03-18T22:58:55.387 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 1, 2024-03-18T22:58:55.387 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.042993343, 2024-03-18T22:58:55.387 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.012796206, 2024-03-18T22:58:55.387 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.021720359, 2024-03-18T22:58:55.387 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.006007519, 2024-03-18T22:58:55.387 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:55.387 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 308, 2024-03-18T22:58:55.387 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.013498115, 2024-03-18T22:58:55.387 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000043825 2024-03-18T22:58:55.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.388 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:55.388 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.388 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.388 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.388 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 693072, 2024-03-18T22:58:55.388 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 185776 2024-03-18T22:58:55.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.388 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T22:58:55.388 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 996, 2024-03-18T22:58:55.388 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 1222, 2024-03-18T22:58:55.388 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 244952, 2024-03-18T22:58:55.389 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 789167, 2024-03-18T22:58:55.389 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 5, 2024-03-18T22:58:55.389 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 4, 2024-03-18T22:58:55.389 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.083569668, 2024-03-18T22:58:55.389 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.026986022, 2024-03-18T22:58:55.389 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.017212120, 2024-03-18T22:58:55.389 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.031691823, 2024-03-18T22:58:55.389 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:55.389 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1220, 2024-03-18T22:58:55.389 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.039147708, 2024-03-18T22:58:55.389 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000032088 2024-03-18T22:58:55.389 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.389 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:55.389 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.389 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.390 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.390 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T22:58:55.390 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 472752 2024-03-18T22:58:55.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.390 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T22:58:55.390 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 2258, 2024-03-18T22:58:55.390 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 2258, 2024-03-18T22:58:55.390 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 1297505, 2024-03-18T22:58:55.390 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 717278, 2024-03-18T22:58:55.390 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 4, 2024-03-18T22:58:55.390 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 3, 2024-03-18T22:58:55.390 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.171359056, 2024-03-18T22:58:55.390 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.052922649, 2024-03-18T22:58:55.391 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.045252780, 2024-03-18T22:58:55.391 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.061083441, 2024-03-18T22:58:55.391 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:55.391 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2257, 2024-03-18T22:58:55.391 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.094335669, 2024-03-18T22:58:55.391 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000041796 2024-03-18T22:58:55.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.391 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:55.391 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.391 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.391 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.391 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-18T22:58:55.391 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T22:58:55.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.392 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-03-18T22:58:55.392 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 1, 2024-03-18T22:58:55.392 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-03-18T22:58:55.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.392 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-MDSRank": { 2024-03-18T22:58:55.392 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-03-18T22:58:55.392 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-03-18T22:58:55.392 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 5456, 2024-03-18T22:58:55.392 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.133697129, 2024-03-18T22:58:55.392 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000024504 2024-03-18T22:58:55.392 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:55.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.392 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-PurgeQueue": { 2024-03-18T22:58:55.392 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-03-18T22:58:55.393 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-03-18T22:58:55.393 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.393 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.393 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.393 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:55.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.393 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-18T22:58:55.393 INFO:teuthology.orchestra.run.smithi103.stdout: "request": 0, 2024-03-18T22:58:55.393 INFO:teuthology.orchestra.run.smithi103.stdout: "reply": 0, 2024-03-18T22:58:55.393 INFO:teuthology.orchestra.run.smithi103.stdout: "reply_latency": { 2024-03-18T22:58:55.393 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.393 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.393 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.394 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_reply": 0, 2024-03-18T22:58:55.394 INFO:teuthology.orchestra.run.smithi103.stdout: "forward": 0, 2024-03-18T22:58:55.394 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_complete": 0, 2024-03-18T22:58:55.394 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_keys": 0, 2024-03-18T22:58:55.394 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_commit": 0, 2024-03-18T22:58:55.394 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_split": 0, 2024-03-18T22:58:55.394 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_merge": 0, 2024-03-18T22:58:55.394 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes": 38, 2024-03-18T22:58:55.394 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_top": 0, 2024-03-18T22:58:55.394 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_bottom": 38, 2024-03-18T22:58:55.394 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pin_tail": 0, 2024-03-18T22:58:55.394 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pinned": 38, 2024-03-18T22:58:55.395 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_expired": 0, 2024-03-18T22:58:55.395 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_with_caps": 0, 2024-03-18T22:58:55.395 INFO:teuthology.orchestra.run.smithi103.stdout: "caps": 0, 2024-03-18T22:58:55.395 INFO:teuthology.orchestra.run.smithi103.stdout: "subtrees": 1, 2024-03-18T22:58:55.395 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse": 0, 2024-03-18T22:58:55.395 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_hit": 0, 2024-03-18T22:58:55.395 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_forward": 0, 2024-03-18T22:58:55.395 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_discover": 0, 2024-03-18T22:58:55.395 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_dir_fetch": 0, 2024-03-18T22:58:55.395 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_remote_ino": 0, 2024-03-18T22:58:55.395 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_lock": 0, 2024-03-18T22:58:55.395 INFO:teuthology.orchestra.run.smithi103.stdout: "load_cent": 0, 2024-03-18T22:58:55.395 INFO:teuthology.orchestra.run.smithi103.stdout: "q": 0, 2024-03-18T22:58:55.395 INFO:teuthology.orchestra.run.smithi103.stdout: "exported": 0, 2024-03-18T22:58:55.395 INFO:teuthology.orchestra.run.smithi103.stdout: "exported_inodes": 0, 2024-03-18T22:58:55.396 INFO:teuthology.orchestra.run.smithi103.stdout: "imported": 0, 2024-03-18T22:58:55.396 INFO:teuthology.orchestra.run.smithi103.stdout: "imported_inodes": 0, 2024-03-18T22:58:55.396 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_dir_fetch": 0, 2024-03-18T22:58:55.396 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_backtrace_fetch": 0, 2024-03-18T22:58:55.396 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_peer_discover": 0, 2024-03-18T22:58:55.396 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rfiles": 0, 2024-03-18T22:58:55.396 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rbytes": 0, 2024-03-18T22:58:55.396 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rsnaps": 0, 2024-03-18T22:58:55.396 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T22:58:55.396 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_set_tag": 0, 2024-03-18T22:58:55.396 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T22:58:55.396 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_inotable_repaired": 0, 2024-03-18T22:58:55.396 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_inodes": 0, 2024-03-18T22:58:55.396 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T22:58:55.397 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T22:58:55.397 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_file_inodes": 0, 2024-03-18T22:58:55.397 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_inode_file_caps": 0, 2024-03-18T22:58:55.397 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T22:58:55.397 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_grant": 0, 2024-03-18T22:58:55.397 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T22:58:55.397 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T22:58:55.397 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T22:58:55.397 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps": 0, 2024-03-18T22:58:55.397 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps_dirty": 0, 2024-03-18T22:58:55.397 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_cap_release": 0, 2024-03-18T22:58:55.397 INFO:teuthology.orchestra.run.smithi103.stdout: "process_request_cap_release": 0 2024-03-18T22:58:55.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.397 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache": { 2024-03-18T22:58:55.397 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update": 0, 2024-03-18T22:58:55.398 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update_receipt": 0, 2024-03-18T22:58:55.398 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_try_discover": 0, 2024-03-18T22:58:55.398 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_send_discover": 0, 2024-03-18T22:58:55.398 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_handle_discover": 0, 2024-03-18T22:58:55.398 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays": 0, 2024-03-18T22:58:55.398 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_delayed": 0, 2024-03-18T22:58:55.398 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_enqueuing": 0, 2024-03-18T22:58:55.398 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_created": 0, 2024-03-18T22:58:55.398 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_enqueued": 0, 2024-03-18T22:58:55.398 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_reintegrated": 0, 2024-03-18T22:58:55.398 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_migrated": 0, 2024-03-18T22:58:55.398 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_processing": 0, 2024-03-18T22:58:55.398 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_enqueued": 0, 2024-03-18T22:58:55.398 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_prioritized": 0, 2024-03-18T22:58:55.399 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_started": 0, 2024-03-18T22:58:55.399 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_completed": 0, 2024-03-18T22:58:55.399 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_path": 0, 2024-03-18T22:58:55.399 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_inode": 0, 2024-03-18T22:58:55.399 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T22:58:55.399 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_exportdir": 0, 2024-03-18T22:58:55.399 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_flush": 0, 2024-03-18T22:58:55.399 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragmentdir": 0, 2024-03-18T22:58:55.399 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragstats": 0, 2024-03-18T22:58:55.399 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_inodestats": 0 2024-03-18T22:58:55.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.399 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log": { 2024-03-18T22:58:55.399 INFO:teuthology.orchestra.run.smithi103.stdout: "evlrg": 0, 2024-03-18T22:58:55.399 INFO:teuthology.orchestra.run.smithi103.stdout: "evadd": 0, 2024-03-18T22:58:55.400 INFO:teuthology.orchestra.run.smithi103.stdout: "evex": 0, 2024-03-18T22:58:55.400 INFO:teuthology.orchestra.run.smithi103.stdout: "evtrm": 0, 2024-03-18T22:58:55.400 INFO:teuthology.orchestra.run.smithi103.stdout: "ev": 540, 2024-03-18T22:58:55.400 INFO:teuthology.orchestra.run.smithi103.stdout: "evexg": 0, 2024-03-18T22:58:55.400 INFO:teuthology.orchestra.run.smithi103.stdout: "evexd": 0, 2024-03-18T22:58:55.400 INFO:teuthology.orchestra.run.smithi103.stdout: "segadd": 0, 2024-03-18T22:58:55.400 INFO:teuthology.orchestra.run.smithi103.stdout: "segex": 0, 2024-03-18T22:58:55.400 INFO:teuthology.orchestra.run.smithi103.stdout: "segtrm": 0, 2024-03-18T22:58:55.400 INFO:teuthology.orchestra.run.smithi103.stdout: "seg": 2, 2024-03-18T22:58:55.400 INFO:teuthology.orchestra.run.smithi103.stdout: "segmjr": 2, 2024-03-18T22:58:55.400 INFO:teuthology.orchestra.run.smithi103.stdout: "segexg": 0, 2024-03-18T22:58:55.400 INFO:teuthology.orchestra.run.smithi103.stdout: "segexd": 0, 2024-03-18T22:58:55.400 INFO:teuthology.orchestra.run.smithi103.stdout: "expos": 4194304, 2024-03-18T22:58:55.400 INFO:teuthology.orchestra.run.smithi103.stdout: "wrpos": 4822086, 2024-03-18T22:58:55.401 INFO:teuthology.orchestra.run.smithi103.stdout: "rdpos": 4820948, 2024-03-18T22:58:55.401 INFO:teuthology.orchestra.run.smithi103.stdout: "jlat": { 2024-03-18T22:58:55.401 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.401 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.401 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.401 INFO:teuthology.orchestra.run.smithi103.stdout: "replayed": 540 2024-03-18T22:58:55.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.401 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_mem": { 2024-03-18T22:58:55.401 INFO:teuthology.orchestra.run.smithi103.stdout: "ino": 4, 2024-03-18T22:58:55.401 INFO:teuthology.orchestra.run.smithi103.stdout: "ino+": 51, 2024-03-18T22:58:55.401 INFO:teuthology.orchestra.run.smithi103.stdout: "ino-": 47, 2024-03-18T22:58:55.401 INFO:teuthology.orchestra.run.smithi103.stdout: "dir": 2, 2024-03-18T22:58:55.401 INFO:teuthology.orchestra.run.smithi103.stdout: "dir+": 16, 2024-03-18T22:58:55.401 INFO:teuthology.orchestra.run.smithi103.stdout: "dir-": 14, 2024-03-18T22:58:55.402 INFO:teuthology.orchestra.run.smithi103.stdout: "dn": 38, 2024-03-18T22:58:55.402 INFO:teuthology.orchestra.run.smithi103.stdout: "dn+": 85, 2024-03-18T22:58:55.402 INFO:teuthology.orchestra.run.smithi103.stdout: "dn-": 47, 2024-03-18T22:58:55.402 INFO:teuthology.orchestra.run.smithi103.stdout: "cap": 0, 2024-03-18T22:58:55.402 INFO:teuthology.orchestra.run.smithi103.stdout: "cap+": 0, 2024-03-18T22:58:55.402 INFO:teuthology.orchestra.run.smithi103.stdout: "cap-": 0, 2024-03-18T22:58:55.402 INFO:teuthology.orchestra.run.smithi103.stdout: "rss": 52152, 2024-03-18T22:58:55.402 INFO:teuthology.orchestra.run.smithi103.stdout: "heap": 215420 2024-03-18T22:58:55.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.402 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_server": { 2024-03-18T22:58:55.402 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_client_request": 0, 2024-03-18T22:58:55.402 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_server_request": 0, 2024-03-18T22:58:55.402 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_request": 0, 2024-03-18T22:58:55.403 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_session": 0, 2024-03-18T22:58:55.403 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_peer_request": 0, 2024-03-18T22:58:55.403 INFO:teuthology.orchestra.run.smithi103.stdout: "req_create_latency": { 2024-03-18T22:58:55.403 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.403 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.403 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.403 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getattr_latency": { 2024-03-18T22:58:55.403 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.403 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.403 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.403 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getfilelock_latency": { 2024-03-18T22:58:55.403 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.404 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.404 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.404 INFO:teuthology.orchestra.run.smithi103.stdout: "req_link_latency": { 2024-03-18T22:58:55.404 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.404 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.404 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.404 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookup_latency": { 2024-03-18T22:58:55.404 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.404 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.404 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.404 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookuphash_latency": { 2024-03-18T22:58:55.405 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.405 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.405 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.405 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupino_latency": { 2024-03-18T22:58:55.405 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.405 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.405 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.405 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupname_latency": { 2024-03-18T22:58:55.405 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.405 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.405 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.405 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupparent_latency": { 2024-03-18T22:58:55.406 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.406 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.406 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.406 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupsnap_latency": { 2024-03-18T22:58:55.406 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.406 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.406 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.406 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lssnap_latency": { 2024-03-18T22:58:55.406 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.406 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.406 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.407 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mkdir_latency": { 2024-03-18T22:58:55.407 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.407 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.407 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.408 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mknod_latency": { 2024-03-18T22:58:55.408 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.408 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.408 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.408 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mksnap_latency": { 2024-03-18T22:58:55.408 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.408 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.408 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.408 INFO:teuthology.orchestra.run.smithi103.stdout: "req_open_latency": { 2024-03-18T22:58:55.408 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.408 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.408 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.409 INFO:teuthology.orchestra.run.smithi103.stdout: "req_readdir_latency": { 2024-03-18T22:58:55.409 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.409 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.409 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.409 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rename_latency": { 2024-03-18T22:58:55.409 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.409 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.409 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.409 INFO:teuthology.orchestra.run.smithi103.stdout: "req_renamesnap_latency": { 2024-03-18T22:58:55.409 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.409 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.409 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.410 INFO:teuthology.orchestra.run.smithi103.stdout: "req_snapdiff_latency": { 2024-03-18T22:58:55.410 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.410 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.410 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.410 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmdir_latency": { 2024-03-18T22:58:55.410 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.410 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.410 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.410 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmsnap_latency": { 2024-03-18T22:58:55.410 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.410 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.411 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.411 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmxattr_latency": { 2024-03-18T22:58:55.411 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.411 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.411 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.411 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setattr_latency": { 2024-03-18T22:58:55.411 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.411 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.411 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.411 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setdirlayout_latency": { 2024-03-18T22:58:55.411 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.412 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.412 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.412 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setfilelock_latency": { 2024-03-18T22:58:55.412 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.412 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.412 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.412 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setlayout_latency": { 2024-03-18T22:58:55.412 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.412 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.412 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.412 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setxattr_latency": { 2024-03-18T22:58:55.412 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.413 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.413 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.413 INFO:teuthology.orchestra.run.smithi103.stdout: "req_symlink_latency": { 2024-03-18T22:58:55.413 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.413 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.413 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.413 INFO:teuthology.orchestra.run.smithi103.stdout: "req_unlink_latency": { 2024-03-18T22:58:55.413 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.413 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.413 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.413 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_revoke_eviction": 0, 2024-03-18T22:58:55.413 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_acquisition_throttle": 0, 2024-03-18T22:58:55.414 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getvxattr_latency": { 2024-03-18T22:58:55.414 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.414 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.414 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.414 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:55.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.414 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_sessions": { 2024-03-18T22:58:55.414 INFO:teuthology.orchestra.run.smithi103.stdout: "session_count": 1, 2024-03-18T22:58:55.414 INFO:teuthology.orchestra.run.smithi103.stdout: "session_add": 2, 2024-03-18T22:58:55.414 INFO:teuthology.orchestra.run.smithi103.stdout: "session_remove": 1, 2024-03-18T22:58:55.414 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_open": 1, 2024-03-18T22:58:55.414 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_stale": 0, 2024-03-18T22:58:55.414 INFO:teuthology.orchestra.run.smithi103.stdout: "total_load": 0, 2024-03-18T22:58:55.414 INFO:teuthology.orchestra.run.smithi103.stdout: "average_load": 0, 2024-03-18T22:58:55.415 INFO:teuthology.orchestra.run.smithi103.stdout: "avg_session_uptime": 3355, 2024-03-18T22:58:55.415 INFO:teuthology.orchestra.run.smithi103.stdout: "mdthresh_evicted": 0 2024-03-18T22:58:55.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.415 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-03-18T22:58:55.415 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 0, 2024-03-18T22:58:55.415 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 0, 2024-03-18T22:58:55.415 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T22:58:55.415 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-03-18T22:58:55.415 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T22:58:55.415 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-03-18T22:58:55.415 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T22:58:55.415 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T22:58:55.415 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T22:58:55.415 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T22:58:55.415 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T22:58:55.416 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-03-18T22:58:55.416 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T22:58:55.416 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T22:58:55.416 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-03-18T22:58:55.416 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-03-18T22:58:55.416 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-03-18T22:58:55.416 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-03-18T22:58:55.416 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T22:58:55.416 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T22:58:55.416 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T22:58:55.416 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T22:58:55.416 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T22:58:55.416 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-03-18T22:58:55.416 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-03-18T22:58:55.416 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-03-18T22:58:55.417 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T22:58:55.417 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T22:58:55.417 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-03-18T22:58:55.417 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-03-18T22:58:55.417 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-03-18T22:58:55.417 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-03-18T22:58:55.417 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T22:58:55.417 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-03-18T22:58:55.417 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T22:58:55.417 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-03-18T22:58:55.417 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 190184, 2024-03-18T22:58:55.417 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 53, 2024-03-18T22:58:55.417 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 0, 2024-03-18T22:58:55.417 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 0, 2024-03-18T22:58:55.417 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-03-18T22:58:55.418 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-03-18T22:58:55.418 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-03-18T22:58:55.418 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-03-18T22:58:55.418 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-03-18T22:58:55.418 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-03-18T22:58:55.418 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 9184, 2024-03-18T22:58:55.418 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 177, 2024-03-18T22:58:55.418 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T22:58:55.418 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-03-18T22:58:55.418 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-03-18T22:58:55.418 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-03-18T22:58:55.419 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 38744, 2024-03-18T22:58:55.419 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 181, 2024-03-18T22:58:55.419 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-03-18T22:58:55.419 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-03-18T22:58:55.419 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-03-18T22:58:55.419 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-03-18T22:58:55.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.419 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter": { 2024-03-18T22:58:55.419 INFO:teuthology.orchestra.run.smithi103.stdout: "op_active": 0, 2024-03-18T22:58:55.419 INFO:teuthology.orchestra.run.smithi103.stdout: "op_laggy": 0, 2024-03-18T22:58:55.419 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send": 3250, 2024-03-18T22:58:55.419 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send_bytes": 21, 2024-03-18T22:58:55.420 INFO:teuthology.orchestra.run.smithi103.stdout: "op_resend": 3, 2024-03-18T22:58:55.420 INFO:teuthology.orchestra.run.smithi103.stdout: "op_reply": 3250, 2024-03-18T22:58:55.420 INFO:teuthology.orchestra.run.smithi103.stdout: "op_latency": { 2024-03-18T22:58:55.420 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 3250, 2024-03-18T22:58:55.420 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 10.851789586, 2024-03-18T22:58:55.420 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.003339012 2024-03-18T22:58:55.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.420 INFO:teuthology.orchestra.run.smithi103.stdout: "op_inflight": 0, 2024-03-18T22:58:55.420 INFO:teuthology.orchestra.run.smithi103.stdout: "oplen_avg": { 2024-03-18T22:58:55.420 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 3250, 2024-03-18T22:58:55.420 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 3251 2024-03-18T22:58:55.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.420 INFO:teuthology.orchestra.run.smithi103.stdout: "op": 3250, 2024-03-18T22:58:55.420 INFO:teuthology.orchestra.run.smithi103.stdout: "op_r": 3250, 2024-03-18T22:58:55.421 INFO:teuthology.orchestra.run.smithi103.stdout: "op_w": 0, 2024-03-18T22:58:55.421 INFO:teuthology.orchestra.run.smithi103.stdout: "op_rmw": 0, 2024-03-18T22:58:55.421 INFO:teuthology.orchestra.run.smithi103.stdout: "op_pg": 0, 2024-03-18T22:58:55.421 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_stat": 2051, 2024-03-18T22:58:55.421 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_create": 0, 2024-03-18T22:58:55.421 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_read": 1197, 2024-03-18T22:58:55.421 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_write": 0, 2024-03-18T22:58:55.421 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writefull": 0, 2024-03-18T22:58:55.421 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writesame": 0, 2024-03-18T22:58:55.421 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_append": 0, 2024-03-18T22:58:55.421 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_zero": 0, 2024-03-18T22:58:55.421 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_truncate": 0, 2024-03-18T22:58:55.421 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_delete": 0, 2024-03-18T22:58:55.421 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_mapext": 0, 2024-03-18T22:58:55.421 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_sparse_read": 0, 2024-03-18T22:58:55.422 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_clonerange": 0, 2024-03-18T22:58:55.422 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_getxattr": 1, 2024-03-18T22:58:55.422 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_setxattr": 0, 2024-03-18T22:58:55.422 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_cmpxattr": 0, 2024-03-18T22:58:55.422 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_rmxattr": 0, 2024-03-18T22:58:55.422 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_resetxattrs": 0, 2024-03-18T22:58:55.422 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_call": 0, 2024-03-18T22:58:55.422 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_watch": 0, 2024-03-18T22:58:55.422 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_notify": 0, 2024-03-18T22:58:55.422 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T22:58:55.422 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls": 0, 2024-03-18T22:58:55.422 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls_filter": 0, 2024-03-18T22:58:55.422 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_other": 0, 2024-03-18T22:58:55.423 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_active": 0, 2024-03-18T22:58:55.423 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_send": 0, 2024-03-18T22:58:55.423 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_resend": 0, 2024-03-18T22:58:55.423 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_ping": 0, 2024-03-18T22:58:55.423 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_active": 0, 2024-03-18T22:58:55.423 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_send": 0, 2024-03-18T22:58:55.423 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_resend": 0, 2024-03-18T22:58:55.423 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_active": 0, 2024-03-18T22:58:55.423 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_send": 0, 2024-03-18T22:58:55.423 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_resend": 0, 2024-03-18T22:58:55.423 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_active": 0, 2024-03-18T22:58:55.423 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_send": 0, 2024-03-18T22:58:55.423 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_resend": 0, 2024-03-18T22:58:55.423 INFO:teuthology.orchestra.run.smithi103.stdout: "command_active": 0, 2024-03-18T22:58:55.424 INFO:teuthology.orchestra.run.smithi103.stdout: "command_send": 0, 2024-03-18T22:58:55.424 INFO:teuthology.orchestra.run.smithi103.stdout: "command_resend": 0, 2024-03-18T22:58:55.424 INFO:teuthology.orchestra.run.smithi103.stdout: "map_epoch": 93, 2024-03-18T22:58:55.424 INFO:teuthology.orchestra.run.smithi103.stdout: "map_full": 0, 2024-03-18T22:58:55.424 INFO:teuthology.orchestra.run.smithi103.stdout: "map_inc": 4, 2024-03-18T22:58:55.424 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_sessions": 5, 2024-03-18T22:58:55.424 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_open": 6, 2024-03-18T22:58:55.424 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_close": 1, 2024-03-18T22:58:55.424 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_laggy": 0, 2024-03-18T22:58:55.424 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_wr": 0, 2024-03-18T22:58:55.424 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_rd": 2, 2024-03-18T22:58:55.424 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_del": 0 2024-03-18T22:58:55.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.424 INFO:teuthology.orchestra.run.smithi103.stdout: "oft": { 2024-03-18T22:58:55.424 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_objs": 0, 2024-03-18T22:58:55.425 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_kv_pairs": 0, 2024-03-18T22:58:55.425 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_updates": 0, 2024-03-18T22:58:55.425 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_removes": 0 2024-03-18T22:58:55.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.425 INFO:teuthology.orchestra.run.smithi103.stdout: "osd-slow-ops": { 2024-03-18T22:58:55.425 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_ops_count": 0 2024-03-18T22:58:55.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.425 INFO:teuthology.orchestra.run.smithi103.stdout: "purge_queue": { 2024-03-18T22:58:55.425 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops": 0, 2024-03-18T22:58:55.425 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T22:58:55.425 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing": 0, 2024-03-18T22:58:55.425 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_high_water": 0, 2024-03-18T22:58:55.425 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed_ops": 0, 2024-03-18T22:58:55.425 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed": 0, 2024-03-18T22:58:55.426 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_item_in_journal": 0 2024-03-18T22:58:55.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.426 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T22:58:55.426 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:58:55.426 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-03-18T22:58:55.426 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:58:55.426 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 3589, 2024-03-18T22:58:55.426 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 1893125, 2024-03-18T22:58:55.426 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:55.426 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 3589, 2024-03-18T22:58:55.426 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-03-18T22:58:55.426 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-03-18T22:58:55.426 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 3589, 2024-03-18T22:58:55.427 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 1893125, 2024-03-18T22:58:55.427 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:58:55.427 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.427 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.427 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.427 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:55.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.427 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_bytes": { 2024-03-18T22:58:55.427 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:58:55.427 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-03-18T22:58:55.427 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:58:55.427 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-03-18T22:58:55.427 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-03-18T22:58:55.427 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:55.427 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:55.428 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 3250, 2024-03-18T22:58:55.428 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 627787, 2024-03-18T22:58:55.428 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 3250, 2024-03-18T22:58:55.428 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 627787, 2024-03-18T22:58:55.428 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:58:55.428 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.428 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.428 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.428 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:55.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.428 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_ops": { 2024-03-18T22:58:55.428 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:58:55.428 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 1024, 2024-03-18T22:58:55.428 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:58:55.429 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-03-18T22:58:55.429 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-03-18T22:58:55.429 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:55.429 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:55.429 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 3250, 2024-03-18T22:58:55.429 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 3250, 2024-03-18T22:58:55.429 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 3250, 2024-03-18T22:58:55.429 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 3250, 2024-03-18T22:58:55.429 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:58:55.429 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.429 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.429 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.429 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:55.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.429 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle": { 2024-03-18T22:58:55.430 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:58:55.430 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-03-18T22:58:55.430 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:58:55.430 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-03-18T22:58:55.430 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-03-18T22:58:55.430 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:55.430 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:55.430 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-03-18T22:58:55.430 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-03-18T22:58:55.430 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 0, 2024-03-18T22:58:55.430 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 0, 2024-03-18T22:58:55.430 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:58:55.430 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.431 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.431 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.431 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:55.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:55.431 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle-0x55c6ebe946e0": { 2024-03-18T22:58:55.431 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:58:55.431 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-03-18T22:58:55.431 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:58:55.431 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-03-18T22:58:55.431 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-03-18T22:58:55.431 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:55.431 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:55.431 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-03-18T22:58:55.431 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-03-18T22:58:55.431 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 0, 2024-03-18T22:58:55.432 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 0, 2024-03-18T22:58:55.432 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:58:55.432 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:55.432 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:55.432 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:55.432 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:55.432 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:55.432 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-18T22:58:56.625 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-18T22:58:56.625 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-18T22:58:56.625 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-18T22:58:56.626 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-18T22:58:56.626 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-03-18T22:58:56.626 DEBUG:tasks.check_counter:Getting stats from h 2024-03-18T22:58:56.626 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-18T22:58:56.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:56 smithi113 ceph-mon[66775]: pgmap v839: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:58:56.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:56 smithi103 ceph-mon[66377]: pgmap v839: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:58:56.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:56 smithi040 ceph-mon[62643]: pgmap v839: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:58:56.955 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T22:58:58.739 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:58:58 smithi103 ceph-mon[66377]: pgmap v840: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T22:58:58.739 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-18T22:58:58.740 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T22:58:58.740 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 4766, 2024-03-18T22:58:58.740 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 3517, 2024-03-18T22:58:58.740 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 1658351, 2024-03-18T22:58:58.740 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 1822636, 2024-03-18T22:58:58.740 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 11, 2024-03-18T22:58:58.740 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 4, 2024-03-18T22:58:58.740 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.324215326, 2024-03-18T22:58:58.740 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.124246343, 2024-03-18T22:58:58.740 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.217540826, 2024-03-18T22:58:58.740 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.043779050, 2024-03-18T22:58:58.740 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:58.740 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 3519, 2024-03-18T22:58:58.741 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.294056647, 2024-03-18T22:58:58.741 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000083562 2024-03-18T22:58:58.741 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.741 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:58.741 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 5904, 2024-03-18T22:58:58.741 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.001172885, 2024-03-18T22:58:58.741 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000198 2024-03-18T22:58:58.741 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.741 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T22:58:58.741 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T22:58:58.741 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.741 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T22:58:58.741 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 1507, 2024-03-18T22:58:58.741 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 1642, 2024-03-18T22:58:58.741 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 613666, 2024-03-18T22:58:58.742 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 1521629, 2024-03-18T22:58:58.742 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 45, 2024-03-18T22:58:58.742 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 7, 2024-03-18T22:58:58.742 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.133115689, 2024-03-18T22:58:58.742 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.038149744, 2024-03-18T22:58:58.742 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.074340861, 2024-03-18T22:58:58.742 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.033232032, 2024-03-18T22:58:58.742 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:58.742 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1641, 2024-03-18T22:58:58.742 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.349832496, 2024-03-18T22:58:58.742 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000213182 2024-03-18T22:58:58.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.742 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:58.742 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1018, 2024-03-18T22:58:58.743 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000191986, 2024-03-18T22:58:58.743 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000188 2024-03-18T22:58:58.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.743 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-18T22:58:58.743 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 562208 2024-03-18T22:58:58.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.743 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T22:58:58.743 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 1598, 2024-03-18T22:58:58.743 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 1760, 2024-03-18T22:58:58.743 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 1151648, 2024-03-18T22:58:58.743 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 1047849, 2024-03-18T22:58:58.743 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 24, 2024-03-18T22:58:58.743 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 7, 2024-03-18T22:58:58.743 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.146859678, 2024-03-18T22:58:58.744 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.049837048, 2024-03-18T22:58:58.744 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.089443169, 2024-03-18T22:58:58.744 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.018129069, 2024-03-18T22:58:58.744 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:58:58.744 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1755, 2024-03-18T22:58:58.744 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.284395564, 2024-03-18T22:58:58.744 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000162048 2024-03-18T22:58:58.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.744 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:58:58.744 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1621, 2024-03-18T22:58:58.744 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000631051, 2024-03-18T22:58:58.744 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000389 2024-03-18T22:58:58.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.744 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 704192, 2024-03-18T22:58:58.745 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 197536 2024-03-18T22:58:58.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.745 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-03-18T22:58:58.745 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 1, 2024-03-18T22:58:58.745 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-03-18T22:58:58.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.745 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-MDSRank": { 2024-03-18T22:58:58.745 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-03-18T22:58:58.745 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-03-18T22:58:58.745 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 672, 2024-03-18T22:58:58.745 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.166257751, 2024-03-18T22:58:58.745 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000247407 2024-03-18T22:58:58.745 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:58.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.746 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-PurgeQueue": { 2024-03-18T22:58:58.746 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-03-18T22:58:58.746 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-03-18T22:58:58.746 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1770, 2024-03-18T22:58:58.746 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.060265954, 2024-03-18T22:58:58.746 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000034048 2024-03-18T22:58:58.746 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:58.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.746 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-18T22:58:58.746 INFO:teuthology.orchestra.run.smithi103.stdout: "request": 335, 2024-03-18T22:58:58.746 INFO:teuthology.orchestra.run.smithi103.stdout: "reply": 303, 2024-03-18T22:58:58.746 INFO:teuthology.orchestra.run.smithi103.stdout: "reply_latency": { 2024-03-18T22:58:58.746 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 303, 2024-03-18T22:58:58.746 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.842160811, 2024-03-18T22:58:58.747 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.002779408 2024-03-18T22:58:58.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.747 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_reply": 0, 2024-03-18T22:58:58.747 INFO:teuthology.orchestra.run.smithi103.stdout: "forward": 32, 2024-03-18T22:58:58.747 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_complete": 0, 2024-03-18T22:58:58.747 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_keys": 0, 2024-03-18T22:58:58.747 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_commit": 12, 2024-03-18T22:58:58.747 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_split": 0, 2024-03-18T22:58:58.747 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_merge": 1, 2024-03-18T22:58:58.747 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes": 21, 2024-03-18T22:58:58.747 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_top": 4, 2024-03-18T22:58:58.747 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_bottom": 17, 2024-03-18T22:58:58.747 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pin_tail": 0, 2024-03-18T22:58:58.747 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pinned": 13, 2024-03-18T22:58:58.748 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_expired": 171, 2024-03-18T22:58:58.748 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_with_caps": 1, 2024-03-18T22:58:58.748 INFO:teuthology.orchestra.run.smithi103.stdout: "caps": 1, 2024-03-18T22:58:58.748 INFO:teuthology.orchestra.run.smithi103.stdout: "subtrees": 4, 2024-03-18T22:58:58.748 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse": 678, 2024-03-18T22:58:58.748 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_hit": 503, 2024-03-18T22:58:58.748 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_forward": 31, 2024-03-18T22:58:58.748 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_discover": 87, 2024-03-18T22:58:58.748 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_dir_fetch": 0, 2024-03-18T22:58:58.748 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_remote_ino": 0, 2024-03-18T22:58:58.748 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_lock": 0, 2024-03-18T22:58:58.748 INFO:teuthology.orchestra.run.smithi103.stdout: "load_cent": 0, 2024-03-18T22:58:58.748 INFO:teuthology.orchestra.run.smithi103.stdout: "q": 0, 2024-03-18T22:58:58.748 INFO:teuthology.orchestra.run.smithi103.stdout: "exported": 2, 2024-03-18T22:58:58.749 INFO:teuthology.orchestra.run.smithi103.stdout: "exported_inodes": 0, 2024-03-18T22:58:58.749 INFO:teuthology.orchestra.run.smithi103.stdout: "imported": 3, 2024-03-18T22:58:58.749 INFO:teuthology.orchestra.run.smithi103.stdout: "imported_inodes": 35, 2024-03-18T22:58:58.749 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_dir_fetch": 0, 2024-03-18T22:58:58.749 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_backtrace_fetch": 196, 2024-03-18T22:58:58.749 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_peer_discover": 0, 2024-03-18T22:58:58.749 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rfiles": 49, 2024-03-18T22:58:58.749 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rbytes": 729704, 2024-03-18T22:58:58.749 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rsnaps": 0, 2024-03-18T22:58:58.749 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_fetch": 196, 2024-03-18T22:58:58.749 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_set_tag": 0, 2024-03-18T22:58:58.749 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T22:58:58.749 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_inotable_repaired": 0, 2024-03-18T22:58:58.749 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_inodes": 0, 2024-03-18T22:58:58.749 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T22:58:58.750 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T22:58:58.750 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_file_inodes": 196, 2024-03-18T22:58:58.750 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_inode_file_caps": 7, 2024-03-18T22:58:58.750 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_revoke": 108, 2024-03-18T22:58:58.750 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_grant": 160, 2024-03-18T22:58:58.750 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T22:58:58.750 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T22:58:58.750 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T22:58:58.750 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps": 382, 2024-03-18T22:58:58.750 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps_dirty": 222, 2024-03-18T22:58:58.750 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_cap_release": 2, 2024-03-18T22:58:58.750 INFO:teuthology.orchestra.run.smithi103.stdout: "process_request_cap_release": 77 2024-03-18T22:58:58.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.750 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache": { 2024-03-18T22:58:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update": 14, 2024-03-18T22:58:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update_receipt": 276, 2024-03-18T22:58:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_try_discover": 179, 2024-03-18T22:58:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_send_discover": 197, 2024-03-18T22:58:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_handle_discover": 14, 2024-03-18T22:58:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays": 0, 2024-03-18T22:58:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_delayed": 0, 2024-03-18T22:58:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_enqueuing": 0, 2024-03-18T22:58:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_created": 37, 2024-03-18T22:58:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_enqueued": 37, 2024-03-18T22:58:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_reintegrated": 0, 2024-03-18T22:58:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_migrated": 0, 2024-03-18T22:58:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_processing": 0, 2024-03-18T22:58:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_enqueued": 0, 2024-03-18T22:58:58.751 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_prioritized": 0, 2024-03-18T22:58:58.752 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_started": 0, 2024-03-18T22:58:58.752 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_completed": 0, 2024-03-18T22:58:58.752 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_path": 0, 2024-03-18T22:58:58.752 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_inode": 0, 2024-03-18T22:58:58.752 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T22:58:58.752 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_exportdir": 4, 2024-03-18T22:58:58.752 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_flush": 0, 2024-03-18T22:58:58.752 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragmentdir": 1, 2024-03-18T22:58:58.752 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragstats": 0, 2024-03-18T22:58:58.752 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_inodestats": 0 2024-03-18T22:58:58.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.752 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log": { 2024-03-18T22:58:58.752 INFO:teuthology.orchestra.run.smithi103.stdout: "evlrg": 0, 2024-03-18T22:58:58.752 INFO:teuthology.orchestra.run.smithi103.stdout: "evadd": 540, 2024-03-18T22:58:58.753 INFO:teuthology.orchestra.run.smithi103.stdout: "evex": 0, 2024-03-18T22:58:58.753 INFO:teuthology.orchestra.run.smithi103.stdout: "evtrm": 0, 2024-03-18T22:58:58.753 INFO:teuthology.orchestra.run.smithi103.stdout: "ev": 540, 2024-03-18T22:58:58.753 INFO:teuthology.orchestra.run.smithi103.stdout: "evexg": 0, 2024-03-18T22:58:58.753 INFO:teuthology.orchestra.run.smithi103.stdout: "evexd": 0, 2024-03-18T22:58:58.753 INFO:teuthology.orchestra.run.smithi103.stdout: "segadd": 2, 2024-03-18T22:58:58.753 INFO:teuthology.orchestra.run.smithi103.stdout: "segex": 0, 2024-03-18T22:58:58.753 INFO:teuthology.orchestra.run.smithi103.stdout: "segtrm": 0, 2024-03-18T22:58:58.753 INFO:teuthology.orchestra.run.smithi103.stdout: "seg": 2, 2024-03-18T22:58:58.753 INFO:teuthology.orchestra.run.smithi103.stdout: "segmjr": 2, 2024-03-18T22:58:58.753 INFO:teuthology.orchestra.run.smithi103.stdout: "segexg": 0, 2024-03-18T22:58:58.753 INFO:teuthology.orchestra.run.smithi103.stdout: "segexd": 0, 2024-03-18T22:58:58.753 INFO:teuthology.orchestra.run.smithi103.stdout: "expos": 4194304, 2024-03-18T22:58:58.753 INFO:teuthology.orchestra.run.smithi103.stdout: "wrpos": 4822086, 2024-03-18T22:58:58.753 INFO:teuthology.orchestra.run.smithi103.stdout: "rdpos": 0, 2024-03-18T22:58:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: "jlat": { 2024-03-18T22:58:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 180, 2024-03-18T22:58:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 8.110666578, 2024-03-18T22:58:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.045059258 2024-03-18T22:58:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: "replayed": 0 2024-03-18T22:58:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_mem": { 2024-03-18T22:58:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: "ino": 26, 2024-03-18T22:58:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: "ino+": 149, 2024-03-18T22:58:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: "ino-": 123, 2024-03-18T22:58:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: "dir": 25, 2024-03-18T22:58:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: "dir+": 131, 2024-03-18T22:58:58.754 INFO:teuthology.orchestra.run.smithi103.stdout: "dir-": 106, 2024-03-18T22:58:58.755 INFO:teuthology.orchestra.run.smithi103.stdout: "dn": 21, 2024-03-18T22:58:58.755 INFO:teuthology.orchestra.run.smithi103.stdout: "dn+": 266, 2024-03-18T22:58:58.755 INFO:teuthology.orchestra.run.smithi103.stdout: "dn-": 245, 2024-03-18T22:58:58.755 INFO:teuthology.orchestra.run.smithi103.stdout: "cap": 1, 2024-03-18T22:58:58.755 INFO:teuthology.orchestra.run.smithi103.stdout: "cap+": 41, 2024-03-18T22:58:58.755 INFO:teuthology.orchestra.run.smithi103.stdout: "cap-": 40, 2024-03-18T22:58:58.755 INFO:teuthology.orchestra.run.smithi103.stdout: "rss": 55944, 2024-03-18T22:58:58.755 INFO:teuthology.orchestra.run.smithi103.stdout: "heap": 223612 2024-03-18T22:58:58.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.755 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_server": { 2024-03-18T22:58:58.755 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_client_request": 411, 2024-03-18T22:58:58.755 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_server_request": 2, 2024-03-18T22:58:58.755 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_request": 335, 2024-03-18T22:58:58.755 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_session": 16, 2024-03-18T22:58:58.755 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_peer_request": 55, 2024-03-18T22:58:58.756 INFO:teuthology.orchestra.run.smithi103.stdout: "req_create_latency": { 2024-03-18T22:58:58.756 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2, 2024-03-18T22:58:58.756 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.007037876, 2024-03-18T22:58:58.756 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.003518938 2024-03-18T22:58:58.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.756 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getattr_latency": { 2024-03-18T22:58:58.756 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2, 2024-03-18T22:58:58.756 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000627914, 2024-03-18T22:58:58.756 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000313957 2024-03-18T22:58:58.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.756 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getfilelock_latency": { 2024-03-18T22:58:58.756 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.756 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.756 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.757 INFO:teuthology.orchestra.run.smithi103.stdout: "req_link_latency": { 2024-03-18T22:58:58.757 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.757 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.757 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.757 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookup_latency": { 2024-03-18T22:58:58.757 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 112, 2024-03-18T22:58:58.757 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.573134290, 2024-03-18T22:58:58.757 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.005117270 2024-03-18T22:58:58.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.757 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookuphash_latency": { 2024-03-18T22:58:58.757 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.757 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.757 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.758 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupino_latency": { 2024-03-18T22:58:58.758 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.758 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.758 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.758 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupname_latency": { 2024-03-18T22:58:58.758 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.758 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.758 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.758 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupparent_latency": { 2024-03-18T22:58:58.758 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.758 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.758 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.759 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.759 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupsnap_latency": { 2024-03-18T22:58:58.759 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.759 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.759 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.759 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.759 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lssnap_latency": { 2024-03-18T22:58:58.759 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.759 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.760 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.760 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.760 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mkdir_latency": { 2024-03-18T22:58:58.760 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.760 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.760 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.760 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.760 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mknod_latency": { 2024-03-18T22:58:58.760 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.760 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.761 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.761 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.761 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mksnap_latency": { 2024-03-18T22:58:58.761 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.761 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.761 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.761 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.761 INFO:teuthology.orchestra.run.smithi103.stdout: "req_open_latency": { 2024-03-18T22:58:58.761 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 145, 2024-03-18T22:58:58.761 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.112621445, 2024-03-18T22:58:58.761 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000776699 2024-03-18T22:58:58.761 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.761 INFO:teuthology.orchestra.run.smithi103.stdout: "req_readdir_latency": { 2024-03-18T22:58:58.761 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 5, 2024-03-18T22:58:58.761 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.003457362, 2024-03-18T22:58:58.762 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000691472 2024-03-18T22:58:58.762 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.762 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rename_latency": { 2024-03-18T22:58:58.762 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.762 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.762 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.762 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.762 INFO:teuthology.orchestra.run.smithi103.stdout: "req_renamesnap_latency": { 2024-03-18T22:58:58.762 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.762 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.762 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.762 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.762 INFO:teuthology.orchestra.run.smithi103.stdout: "req_snapdiff_latency": { 2024-03-18T22:58:58.762 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.762 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.763 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.763 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmdir_latency": { 2024-03-18T22:58:58.763 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.763 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.763 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.763 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmsnap_latency": { 2024-03-18T22:58:58.763 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.763 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.763 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.763 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmxattr_latency": { 2024-03-18T22:58:58.763 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.763 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.764 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.764 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.764 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setattr_latency": { 2024-03-18T22:58:58.764 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.764 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.764 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.764 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.764 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setdirlayout_latency": { 2024-03-18T22:58:58.764 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.764 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.764 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.764 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.764 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setfilelock_latency": { 2024-03-18T22:58:58.764 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.764 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.765 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.765 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.765 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setlayout_latency": { 2024-03-18T22:58:58.765 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.765 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.765 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.765 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.765 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setxattr_latency": { 2024-03-18T22:58:58.765 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.765 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.765 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.765 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.765 INFO:teuthology.orchestra.run.smithi103.stdout: "req_symlink_latency": { 2024-03-18T22:58:58.765 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.766 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.766 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.766 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.766 INFO:teuthology.orchestra.run.smithi103.stdout: "req_unlink_latency": { 2024-03-18T22:58:58.766 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 37, 2024-03-18T22:58:58.766 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.145281924, 2024-03-18T22:58:58.766 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.003926538 2024-03-18T22:58:58.766 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.766 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_revoke_eviction": 0, 2024-03-18T22:58:58.766 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_acquisition_throttle": 0, 2024-03-18T22:58:58.766 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getvxattr_latency": { 2024-03-18T22:58:58.766 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.766 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.766 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.766 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:58.767 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.767 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_sessions": { 2024-03-18T22:58:58.767 INFO:teuthology.orchestra.run.smithi103.stdout: "session_count": 1, 2024-03-18T22:58:58.767 INFO:teuthology.orchestra.run.smithi103.stdout: "session_add": 2, 2024-03-18T22:58:58.767 INFO:teuthology.orchestra.run.smithi103.stdout: "session_remove": 1, 2024-03-18T22:58:58.767 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_open": 1, 2024-03-18T22:58:58.767 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_stale": 0, 2024-03-18T22:58:58.767 INFO:teuthology.orchestra.run.smithi103.stdout: "total_load": 98, 2024-03-18T22:58:58.767 INFO:teuthology.orchestra.run.smithi103.stdout: "average_load": 98, 2024-03-18T22:58:58.767 INFO:teuthology.orchestra.run.smithi103.stdout: "avg_session_uptime": 955, 2024-03-18T22:58:58.767 INFO:teuthology.orchestra.run.smithi103.stdout: "mdthresh_evicted": 0 2024-03-18T22:58:58.767 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.767 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-03-18T22:58:58.767 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 0, 2024-03-18T22:58:58.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 0, 2024-03-18T22:58:58.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T22:58:58.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-03-18T22:58:58.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T22:58:58.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-03-18T22:58:58.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T22:58:58.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T22:58:58.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T22:58:58.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T22:58:58.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T22:58:58.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-03-18T22:58:58.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T22:58:58.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T22:58:58.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-03-18T22:58:58.768 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-03-18T22:58:58.769 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-03-18T22:58:58.769 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-03-18T22:58:58.769 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T22:58:58.769 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T22:58:58.769 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T22:58:58.769 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T22:58:58.769 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T22:58:58.769 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-03-18T22:58:58.769 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-03-18T22:58:58.769 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-03-18T22:58:58.769 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T22:58:58.769 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T22:58:58.769 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-03-18T22:58:58.769 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-03-18T22:58:58.769 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-03-18T22:58:58.770 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-03-18T22:58:58.770 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T22:58:58.770 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-03-18T22:58:58.770 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T22:58:58.770 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-03-18T22:58:58.770 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 293737, 2024-03-18T22:58:58.770 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 120, 2024-03-18T22:58:58.770 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 0, 2024-03-18T22:58:58.770 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 0, 2024-03-18T22:58:58.770 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-03-18T22:58:58.770 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-03-18T22:58:58.770 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-03-18T22:58:58.770 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-03-18T22:58:58.770 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-03-18T22:58:58.770 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-03-18T22:58:58.771 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 9184, 2024-03-18T22:58:58.771 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 177, 2024-03-18T22:58:58.771 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T22:58:58.771 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-03-18T22:58:58.771 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-03-18T22:58:58.771 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-03-18T22:58:58.771 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 105501, 2024-03-18T22:58:58.771 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 416, 2024-03-18T22:58:58.771 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-03-18T22:58:58.771 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-03-18T22:58:58.771 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-03-18T22:58:58.771 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-03-18T22:58:58.771 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.771 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter": { 2024-03-18T22:58:58.771 INFO:teuthology.orchestra.run.smithi103.stdout: "op_active": 0, 2024-03-18T22:58:58.772 INFO:teuthology.orchestra.run.smithi103.stdout: "op_laggy": 0, 2024-03-18T22:58:58.772 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send": 2361, 2024-03-18T22:58:58.772 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send_bytes": 660446, 2024-03-18T22:58:58.772 INFO:teuthology.orchestra.run.smithi103.stdout: "op_resend": 33, 2024-03-18T22:58:58.772 INFO:teuthology.orchestra.run.smithi103.stdout: "op_reply": 2361, 2024-03-18T22:58:58.772 INFO:teuthology.orchestra.run.smithi103.stdout: "op_latency": { 2024-03-18T22:58:58.772 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2361, 2024-03-18T22:58:58.772 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 141.493251595, 2024-03-18T22:58:58.772 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.059929373 2024-03-18T22:58:58.772 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.772 INFO:teuthology.orchestra.run.smithi103.stdout: "op_inflight": 0, 2024-03-18T22:58:58.772 INFO:teuthology.orchestra.run.smithi103.stdout: "oplen_avg": { 2024-03-18T22:58:58.772 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2361, 2024-03-18T22:58:58.772 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 2483 2024-03-18T22:58:58.772 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.773 INFO:teuthology.orchestra.run.smithi103.stdout: "op": 2361, 2024-03-18T22:58:58.773 INFO:teuthology.orchestra.run.smithi103.stdout: "op_r": 199, 2024-03-18T22:58:58.773 INFO:teuthology.orchestra.run.smithi103.stdout: "op_w": 2162, 2024-03-18T22:58:58.773 INFO:teuthology.orchestra.run.smithi103.stdout: "op_rmw": 0, 2024-03-18T22:58:58.773 INFO:teuthology.orchestra.run.smithi103.stdout: "op_pg": 0, 2024-03-18T22:58:58.773 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_stat": 0, 2024-03-18T22:58:58.773 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_create": 10, 2024-03-18T22:58:58.773 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_read": 3, 2024-03-18T22:58:58.773 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_write": 183, 2024-03-18T22:58:58.773 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writefull": 35, 2024-03-18T22:58:58.773 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writesame": 0, 2024-03-18T22:58:58.773 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_append": 0, 2024-03-18T22:58:58.773 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_zero": 0, 2024-03-18T22:58:58.773 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_truncate": 0, 2024-03-18T22:58:58.773 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_delete": 1834, 2024-03-18T22:58:58.774 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_mapext": 0, 2024-03-18T22:58:58.774 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_sparse_read": 0, 2024-03-18T22:58:58.774 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_clonerange": 0, 2024-03-18T22:58:58.774 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_getxattr": 196, 2024-03-18T22:58:58.774 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_setxattr": 20, 2024-03-18T22:58:58.774 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_cmpxattr": 0, 2024-03-18T22:58:58.774 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_rmxattr": 0, 2024-03-18T22:58:58.774 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_resetxattrs": 0, 2024-03-18T22:58:58.774 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_call": 0, 2024-03-18T22:58:58.774 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_watch": 0, 2024-03-18T22:58:58.774 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_notify": 0, 2024-03-18T22:58:58.774 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T22:58:58.774 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls": 0, 2024-03-18T22:58:58.774 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls_filter": 0, 2024-03-18T22:58:58.774 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_other": 0, 2024-03-18T22:58:58.775 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_active": 0, 2024-03-18T22:58:58.775 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_send": 0, 2024-03-18T22:58:58.775 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_resend": 0, 2024-03-18T22:58:58.775 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_ping": 0, 2024-03-18T22:58:58.775 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_active": 0, 2024-03-18T22:58:58.775 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_send": 0, 2024-03-18T22:58:58.775 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_resend": 0, 2024-03-18T22:58:58.775 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_active": 0, 2024-03-18T22:58:58.775 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_send": 0, 2024-03-18T22:58:58.775 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_resend": 0, 2024-03-18T22:58:58.775 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_active": 0, 2024-03-18T22:58:58.775 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_send": 0, 2024-03-18T22:58:58.775 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_resend": 0, 2024-03-18T22:58:58.775 INFO:teuthology.orchestra.run.smithi103.stdout: "command_active": 0, 2024-03-18T22:58:58.775 INFO:teuthology.orchestra.run.smithi103.stdout: "command_send": 0, 2024-03-18T22:58:58.776 INFO:teuthology.orchestra.run.smithi103.stdout: "command_resend": 0, 2024-03-18T22:58:58.776 INFO:teuthology.orchestra.run.smithi103.stdout: "map_epoch": 93, 2024-03-18T22:58:58.776 INFO:teuthology.orchestra.run.smithi103.stdout: "map_full": 0, 2024-03-18T22:58:58.776 INFO:teuthology.orchestra.run.smithi103.stdout: "map_inc": 4, 2024-03-18T22:58:58.776 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_sessions": 12, 2024-03-18T22:58:58.776 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_open": 12, 2024-03-18T22:58:58.776 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_close": 0, 2024-03-18T22:58:58.776 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_laggy": 0, 2024-03-18T22:58:58.776 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_wr": 166, 2024-03-18T22:58:58.776 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_rd": 0, 2024-03-18T22:58:58.776 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_del": 36 2024-03-18T22:58:58.776 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.776 INFO:teuthology.orchestra.run.smithi103.stdout: "oft": { 2024-03-18T22:58:58.776 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_objs": 1, 2024-03-18T22:58:58.776 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_kv_pairs": 0, 2024-03-18T22:58:58.777 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_updates": 0, 2024-03-18T22:58:58.777 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_removes": 0 2024-03-18T22:58:58.777 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.777 INFO:teuthology.orchestra.run.smithi103.stdout: "osd-slow-ops": { 2024-03-18T22:58:58.777 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_ops_count": 0 2024-03-18T22:58:58.777 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.777 INFO:teuthology.orchestra.run.smithi103.stdout: "purge_queue": { 2024-03-18T22:58:58.777 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops": 0, 2024-03-18T22:58:58.777 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops_high_water": 52, 2024-03-18T22:58:58.777 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing": 0, 2024-03-18T22:58:58.777 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_high_water": 1, 2024-03-18T22:58:58.777 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed_ops": 1820, 2024-03-18T22:58:58.777 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed": 37, 2024-03-18T22:58:58.777 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_item_in_journal": 0 2024-03-18T22:58:58.777 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.778 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T22:58:58.778 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:58:58.778 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-03-18T22:58:58.778 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:58:58.778 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 7867, 2024-03-18T22:58:58.778 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 2693093, 2024-03-18T22:58:58.778 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:58.778 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 7867, 2024-03-18T22:58:58.778 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-03-18T22:58:58.778 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-03-18T22:58:58.778 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 7867, 2024-03-18T22:58:58.778 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 2693093, 2024-03-18T22:58:58.778 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:58:58.778 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.779 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.779 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.779 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:58.779 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.779 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_bytes": { 2024-03-18T22:58:58.779 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:58:58.779 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-03-18T22:58:58.779 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:58:58.779 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-03-18T22:58:58.779 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-03-18T22:58:58.779 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:58.779 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:58.779 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 2361, 2024-03-18T22:58:58.779 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 664479, 2024-03-18T22:58:58.780 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 2361, 2024-03-18T22:58:58.780 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 664479, 2024-03-18T22:58:58.780 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:58:58.780 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.780 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.780 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.780 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:58.780 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.780 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_ops": { 2024-03-18T22:58:58.780 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:58:58.780 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 1024, 2024-03-18T22:58:58.780 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:58:58.780 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-03-18T22:58:58.780 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-03-18T22:58:58.780 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:58.781 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-03-18T22:58:58.781 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 2361, 2024-03-18T22:58:58.781 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 2361, 2024-03-18T22:58:58.781 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 2361, 2024-03-18T22:58:58.781 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 2361, 2024-03-18T22:58:58.781 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:58:58.781 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.781 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.781 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.781 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:58.781 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.781 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle": { 2024-03-18T22:58:58.781 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:58:58.781 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-03-18T22:58:58.781 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:58:58.782 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 37, 2024-03-18T22:58:58.782 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 4033, 2024-03-18T22:58:58.782 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:58.782 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 37, 2024-03-18T22:58:58.782 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-03-18T22:58:58.782 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-03-18T22:58:58.782 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 3, 2024-03-18T22:58:58.782 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 4033, 2024-03-18T22:58:58.782 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:58:58.782 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.782 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.782 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.782 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:58.782 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle-0x56291d85e500": { 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 540, 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 627782, 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 540, 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 180, 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 627782, 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:58:58.783 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:58:58.784 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:58.784 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:58:58.784 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-18T22:58:58.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:58:58 smithi113 ceph-mon[66775]: pgmap v840: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T22:58:58.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:58:58 smithi040 ceph-mon[62643]: pgmap v840: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T22:58:59.542 INFO:tasks.check_counter:Daemon mds.h mds.exported=2 2024-03-18T22:58:59.542 INFO:tasks.check_counter:Daemon mds.h mds.imported=3 2024-03-18T22:58:59.542 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=14 2024-03-18T22:58:59.542 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=276 2024-03-18T22:58:59.542 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-03-18T22:58:59.542 DEBUG:tasks.check_counter:Getting stats from k 2024-03-18T22:58:59.542 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-18T22:58:59.852 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T22:59:00.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:00 smithi113 ceph-mon[66775]: pgmap v841: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:59:00.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:00 smithi103 ceph-mon[66377]: pgmap v841: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:59:00.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:00 smithi040 ceph-mon[62643]: pgmap v841: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:59:01.705 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-03-18T22:59:01.705 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T22:59:01.705 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 4565, 2024-03-18T22:59:01.705 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 3177, 2024-03-18T22:59:01.705 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 1819870, 2024-03-18T22:59:01.705 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 1470580, 2024-03-18T22:59:01.705 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 10, 2024-03-18T22:59:01.705 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 2, 2024-03-18T22:59:01.705 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.295121536, 2024-03-18T22:59:01.705 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.115079526, 2024-03-18T22:59:01.706 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.194848797, 2024-03-18T22:59:01.706 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.030044271, 2024-03-18T22:59:01.706 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:59:01.706 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 3183, 2024-03-18T22:59:01.706 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.337095240, 2024-03-18T22:59:01.706 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000105904 2024-03-18T22:59:01.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.706 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:59:01.706 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 6598, 2024-03-18T22:59:01.706 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.001214129, 2024-03-18T22:59:01.706 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000184 2024-03-18T22:59:01.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.706 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T22:59:01.707 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T22:59:01.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.707 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T22:59:01.707 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 995, 2024-03-18T22:59:01.707 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 970, 2024-03-18T22:59:01.707 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 849885, 2024-03-18T22:59:01.707 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 848465, 2024-03-18T22:59:01.707 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 50, 2024-03-18T22:59:01.707 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 9, 2024-03-18T22:59:01.707 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.103191997, 2024-03-18T22:59:01.707 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.028277760, 2024-03-18T22:59:01.707 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.048114505, 2024-03-18T22:59:01.707 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.026894006, 2024-03-18T22:59:01.708 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:59:01.708 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 966, 2024-03-18T22:59:01.708 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.215517036, 2024-03-18T22:59:01.708 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000223102 2024-03-18T22:59:01.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.708 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:59:01.708 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 40, 2024-03-18T22:59:01.708 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000004413, 2024-03-18T22:59:01.708 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000110 2024-03-18T22:59:01.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.708 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 699360, 2024-03-18T22:59:01.708 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 197744 2024-03-18T22:59:01.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.708 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T22:59:01.709 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 811, 2024-03-18T22:59:01.709 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 1135, 2024-03-18T22:59:01.709 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 326974, 2024-03-18T22:59:01.709 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 1015909, 2024-03-18T22:59:01.709 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 19, 2024-03-18T22:59:01.709 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 11, 2024-03-18T22:59:01.709 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.082384960, 2024-03-18T22:59:01.709 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.030336730, 2024-03-18T22:59:01.709 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.047160474, 2024-03-18T22:59:01.709 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.009636390, 2024-03-18T22:59:01.709 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:59:01.709 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1129, 2024-03-18T22:59:01.709 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.293115678, 2024-03-18T22:59:01.709 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000259624 2024-03-18T22:59:01.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.710 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:59:01.710 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 986, 2024-03-18T22:59:01.710 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000288697, 2024-03-18T22:59:01.710 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000292 2024-03-18T22:59:01.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.710 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T22:59:01.710 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 476432 2024-03-18T22:59:01.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.710 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-03-18T22:59:01.710 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 1, 2024-03-18T22:59:01.710 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-03-18T22:59:01.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.710 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-MDSRank": { 2024-03-18T22:59:01.711 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-03-18T22:59:01.711 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-03-18T22:59:01.711 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 404, 2024-03-18T22:59:01.711 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.101653980, 2024-03-18T22:59:01.711 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000251618 2024-03-18T22:59:01.711 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:59:01.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.711 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-PurgeQueue": { 2024-03-18T22:59:01.711 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-03-18T22:59:01.711 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-03-18T22:59:01.711 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 960, 2024-03-18T22:59:01.711 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.033820703, 2024-03-18T22:59:01.711 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000035229 2024-03-18T22:59:01.711 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:59:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-03-18T22:59:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: "request": 205, 2024-03-18T22:59:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: "reply": 157, 2024-03-18T22:59:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: "reply_latency": { 2024-03-18T22:59:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 157, 2024-03-18T22:59:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.285353996, 2024-03-18T22:59:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.001817541 2024-03-18T22:59:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_reply": 0, 2024-03-18T22:59:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: "forward": 48, 2024-03-18T22:59:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_complete": 0, 2024-03-18T22:59:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_keys": 0, 2024-03-18T22:59:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_commit": 11, 2024-03-18T22:59:01.712 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_split": 0, 2024-03-18T22:59:01.713 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_merge": 0, 2024-03-18T22:59:01.713 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes": 21, 2024-03-18T22:59:01.713 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_top": 4, 2024-03-18T22:59:01.713 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_bottom": 17, 2024-03-18T22:59:01.713 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pin_tail": 0, 2024-03-18T22:59:01.713 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pinned": 13, 2024-03-18T22:59:01.713 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_expired": 171, 2024-03-18T22:59:01.713 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_with_caps": 0, 2024-03-18T22:59:01.713 INFO:teuthology.orchestra.run.smithi103.stdout: "caps": 0, 2024-03-18T22:59:01.713 INFO:teuthology.orchestra.run.smithi103.stdout: "subtrees": 4, 2024-03-18T22:59:01.713 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse": 481, 2024-03-18T22:59:01.713 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_hit": 324, 2024-03-18T22:59:01.713 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_forward": 48, 2024-03-18T22:59:01.713 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_discover": 83, 2024-03-18T22:59:01.713 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_dir_fetch": 0, 2024-03-18T22:59:01.714 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_remote_ino": 0, 2024-03-18T22:59:01.714 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_lock": 0, 2024-03-18T22:59:01.714 INFO:teuthology.orchestra.run.smithi103.stdout: "load_cent": 0, 2024-03-18T22:59:01.714 INFO:teuthology.orchestra.run.smithi103.stdout: "q": 0, 2024-03-18T22:59:01.714 INFO:teuthology.orchestra.run.smithi103.stdout: "exported": 1, 2024-03-18T22:59:01.714 INFO:teuthology.orchestra.run.smithi103.stdout: "exported_inodes": 0, 2024-03-18T22:59:01.714 INFO:teuthology.orchestra.run.smithi103.stdout: "imported": 1, 2024-03-18T22:59:01.714 INFO:teuthology.orchestra.run.smithi103.stdout: "imported_inodes": 20, 2024-03-18T22:59:01.714 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_dir_fetch": 0, 2024-03-18T22:59:01.714 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_backtrace_fetch": 80, 2024-03-18T22:59:01.714 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_peer_discover": 0, 2024-03-18T22:59:01.714 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rfiles": 49, 2024-03-18T22:59:01.714 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rbytes": 729704, 2024-03-18T22:59:01.714 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rsnaps": 0, 2024-03-18T22:59:01.715 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_fetch": 80, 2024-03-18T22:59:01.715 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_set_tag": 0, 2024-03-18T22:59:01.715 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T22:59:01.715 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_inotable_repaired": 0, 2024-03-18T22:59:01.715 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_inodes": 0, 2024-03-18T22:59:01.715 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T22:59:01.715 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T22:59:01.715 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_file_inodes": 80, 2024-03-18T22:59:01.715 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_inode_file_caps": 8, 2024-03-18T22:59:01.715 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_revoke": 65, 2024-03-18T22:59:01.715 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_grant": 73, 2024-03-18T22:59:01.715 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T22:59:01.715 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T22:59:01.715 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T22:59:01.716 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps": 196, 2024-03-18T22:59:01.716 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps_dirty": 114, 2024-03-18T22:59:01.716 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_cap_release": 2, 2024-03-18T22:59:01.716 INFO:teuthology.orchestra.run.smithi103.stdout: "process_request_cap_release": 40 2024-03-18T22:59:01.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.716 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache": { 2024-03-18T22:59:01.716 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update": 1, 2024-03-18T22:59:01.716 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update_receipt": 311, 2024-03-18T22:59:01.716 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_try_discover": 197, 2024-03-18T22:59:01.716 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_send_discover": 224, 2024-03-18T22:59:01.716 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_handle_discover": 1, 2024-03-18T22:59:01.716 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays": 0, 2024-03-18T22:59:01.716 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_delayed": 0, 2024-03-18T22:59:01.716 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_enqueuing": 0, 2024-03-18T22:59:01.716 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_created": 20, 2024-03-18T22:59:01.717 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_enqueued": 20, 2024-03-18T22:59:01.717 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_reintegrated": 0, 2024-03-18T22:59:01.717 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_migrated": 0, 2024-03-18T22:59:01.717 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_processing": 0, 2024-03-18T22:59:01.717 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_enqueued": 0, 2024-03-18T22:59:01.717 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_prioritized": 0, 2024-03-18T22:59:01.717 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_started": 0, 2024-03-18T22:59:01.717 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_completed": 0, 2024-03-18T22:59:01.717 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_path": 0, 2024-03-18T22:59:01.717 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_inode": 0, 2024-03-18T22:59:01.717 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T22:59:01.717 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_exportdir": 1, 2024-03-18T22:59:01.717 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_flush": 0, 2024-03-18T22:59:01.717 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragmentdir": 0, 2024-03-18T22:59:01.717 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragstats": 0, 2024-03-18T22:59:01.718 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_inodestats": 0 2024-03-18T22:59:01.718 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.718 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log": { 2024-03-18T22:59:01.718 INFO:teuthology.orchestra.run.smithi103.stdout: "evlrg": 0, 2024-03-18T22:59:01.718 INFO:teuthology.orchestra.run.smithi103.stdout: "evadd": 251, 2024-03-18T22:59:01.718 INFO:teuthology.orchestra.run.smithi103.stdout: "evex": 0, 2024-03-18T22:59:01.718 INFO:teuthology.orchestra.run.smithi103.stdout: "evtrm": 0, 2024-03-18T22:59:01.718 INFO:teuthology.orchestra.run.smithi103.stdout: "ev": 251, 2024-03-18T22:59:01.718 INFO:teuthology.orchestra.run.smithi103.stdout: "evexg": 0, 2024-03-18T22:59:01.718 INFO:teuthology.orchestra.run.smithi103.stdout: "evexd": 0, 2024-03-18T22:59:01.718 INFO:teuthology.orchestra.run.smithi103.stdout: "segadd": 2, 2024-03-18T22:59:01.719 INFO:teuthology.orchestra.run.smithi103.stdout: "segex": 0, 2024-03-18T22:59:01.719 INFO:teuthology.orchestra.run.smithi103.stdout: "segtrm": 0, 2024-03-18T22:59:01.719 INFO:teuthology.orchestra.run.smithi103.stdout: "seg": 2, 2024-03-18T22:59:01.719 INFO:teuthology.orchestra.run.smithi103.stdout: "segmjr": 2, 2024-03-18T22:59:01.719 INFO:teuthology.orchestra.run.smithi103.stdout: "segexg": 0, 2024-03-18T22:59:01.719 INFO:teuthology.orchestra.run.smithi103.stdout: "segexd": 0, 2024-03-18T22:59:01.719 INFO:teuthology.orchestra.run.smithi103.stdout: "expos": 4194304, 2024-03-18T22:59:01.719 INFO:teuthology.orchestra.run.smithi103.stdout: "wrpos": 4503927, 2024-03-18T22:59:01.719 INFO:teuthology.orchestra.run.smithi103.stdout: "rdpos": 0, 2024-03-18T22:59:01.719 INFO:teuthology.orchestra.run.smithi103.stdout: "jlat": { 2024-03-18T22:59:01.719 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 112, 2024-03-18T22:59:01.719 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 2.360360535, 2024-03-18T22:59:01.719 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.021074647 2024-03-18T22:59:01.719 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.719 INFO:teuthology.orchestra.run.smithi103.stdout: "replayed": 0 2024-03-18T22:59:01.720 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.720 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_mem": { 2024-03-18T22:59:01.720 INFO:teuthology.orchestra.run.smithi103.stdout: "ino": 26, 2024-03-18T22:59:01.720 INFO:teuthology.orchestra.run.smithi103.stdout: "ino+": 132, 2024-03-18T22:59:01.720 INFO:teuthology.orchestra.run.smithi103.stdout: "ino-": 106, 2024-03-18T22:59:01.720 INFO:teuthology.orchestra.run.smithi103.stdout: "dir": 25, 2024-03-18T22:59:01.720 INFO:teuthology.orchestra.run.smithi103.stdout: "dir+": 144, 2024-03-18T22:59:01.720 INFO:teuthology.orchestra.run.smithi103.stdout: "dir-": 119, 2024-03-18T22:59:01.720 INFO:teuthology.orchestra.run.smithi103.stdout: "dn": 21, 2024-03-18T22:59:01.720 INFO:teuthology.orchestra.run.smithi103.stdout: "dn+": 232, 2024-03-18T22:59:01.720 INFO:teuthology.orchestra.run.smithi103.stdout: "dn-": 211, 2024-03-18T22:59:01.720 INFO:teuthology.orchestra.run.smithi103.stdout: "cap": 0, 2024-03-18T22:59:01.720 INFO:teuthology.orchestra.run.smithi103.stdout: "cap+": 21, 2024-03-18T22:59:01.720 INFO:teuthology.orchestra.run.smithi103.stdout: "cap-": 21, 2024-03-18T22:59:01.720 INFO:teuthology.orchestra.run.smithi103.stdout: "rss": 56340, 2024-03-18T22:59:01.721 INFO:teuthology.orchestra.run.smithi103.stdout: "heap": 223612 2024-03-18T22:59:01.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.721 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_server": { 2024-03-18T22:59:01.721 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_client_request": 235, 2024-03-18T22:59:01.721 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_server_request": 3, 2024-03-18T22:59:01.721 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_request": 205, 2024-03-18T22:59:01.721 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_session": 12, 2024-03-18T22:59:01.721 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_peer_request": 27, 2024-03-18T22:59:01.721 INFO:teuthology.orchestra.run.smithi103.stdout: "req_create_latency": { 2024-03-18T22:59:01.721 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.721 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.721 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.721 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getattr_latency": { 2024-03-18T22:59:01.721 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.722 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.722 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.722 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getfilelock_latency": { 2024-03-18T22:59:01.722 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.722 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.722 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.722 INFO:teuthology.orchestra.run.smithi103.stdout: "req_link_latency": { 2024-03-18T22:59:01.722 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.722 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.722 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.722 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.722 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookup_latency": { 2024-03-18T22:59:01.722 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 51, 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.049483403, 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000970262 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookuphash_latency": { 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupino_latency": { 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupname_latency": { 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.723 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.724 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.724 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupparent_latency": { 2024-03-18T22:59:01.724 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.724 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.724 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.724 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupsnap_latency": { 2024-03-18T22:59:01.724 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.724 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.724 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.724 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.724 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lssnap_latency": { 2024-03-18T22:59:01.724 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.724 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.725 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.725 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.725 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mkdir_latency": { 2024-03-18T22:59:01.725 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.725 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.725 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.726 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mknod_latency": { 2024-03-18T22:59:01.726 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.726 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.726 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.726 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mksnap_latency": { 2024-03-18T22:59:01.726 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.726 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.726 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.726 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.726 INFO:teuthology.orchestra.run.smithi103.stdout: "req_open_latency": { 2024-03-18T22:59:01.726 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 84, 2024-03-18T22:59:01.726 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.058648081, 2024-03-18T22:59:01.726 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000698191 2024-03-18T22:59:01.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.727 INFO:teuthology.orchestra.run.smithi103.stdout: "req_readdir_latency": { 2024-03-18T22:59:01.727 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2, 2024-03-18T22:59:01.727 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.001066798, 2024-03-18T22:59:01.727 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000533399 2024-03-18T22:59:01.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.727 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rename_latency": { 2024-03-18T22:59:01.727 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.727 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.727 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.727 INFO:teuthology.orchestra.run.smithi103.stdout: "req_renamesnap_latency": { 2024-03-18T22:59:01.727 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.727 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.728 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.728 INFO:teuthology.orchestra.run.smithi103.stdout: "req_snapdiff_latency": { 2024-03-18T22:59:01.728 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.728 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.728 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.728 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmdir_latency": { 2024-03-18T22:59:01.728 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.728 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.728 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.728 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmsnap_latency": { 2024-03-18T22:59:01.728 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.728 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.729 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.729 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmxattr_latency": { 2024-03-18T22:59:01.729 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.729 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.729 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.729 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setattr_latency": { 2024-03-18T22:59:01.729 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.729 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.729 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.729 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setdirlayout_latency": { 2024-03-18T22:59:01.729 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.729 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.730 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.730 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setfilelock_latency": { 2024-03-18T22:59:01.730 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.730 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.730 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.730 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setlayout_latency": { 2024-03-18T22:59:01.730 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.730 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.730 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.730 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setxattr_latency": { 2024-03-18T22:59:01.731 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.731 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.731 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.731 INFO:teuthology.orchestra.run.smithi103.stdout: "req_symlink_latency": { 2024-03-18T22:59:01.731 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.731 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.731 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.731 INFO:teuthology.orchestra.run.smithi103.stdout: "req_unlink_latency": { 2024-03-18T22:59:01.731 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 20, 2024-03-18T22:59:01.731 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.176155714, 2024-03-18T22:59:01.731 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.008807785 2024-03-18T22:59:01.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.731 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_revoke_eviction": 0, 2024-03-18T22:59:01.732 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_acquisition_throttle": 0, 2024-03-18T22:59:01.732 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getvxattr_latency": { 2024-03-18T22:59:01.732 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.732 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.732 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.732 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:59:01.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.732 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_sessions": { 2024-03-18T22:59:01.732 INFO:teuthology.orchestra.run.smithi103.stdout: "session_count": 1, 2024-03-18T22:59:01.732 INFO:teuthology.orchestra.run.smithi103.stdout: "session_add": 1, 2024-03-18T22:59:01.732 INFO:teuthology.orchestra.run.smithi103.stdout: "session_remove": 0, 2024-03-18T22:59:01.732 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_open": 1, 2024-03-18T22:59:01.732 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_stale": 0, 2024-03-18T22:59:01.732 INFO:teuthology.orchestra.run.smithi103.stdout: "total_load": 52, 2024-03-18T22:59:01.733 INFO:teuthology.orchestra.run.smithi103.stdout: "average_load": 52, 2024-03-18T22:59:01.733 INFO:teuthology.orchestra.run.smithi103.stdout: "avg_session_uptime": 958, 2024-03-18T22:59:01.733 INFO:teuthology.orchestra.run.smithi103.stdout: "mdthresh_evicted": 0 2024-03-18T22:59:01.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.733 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-03-18T22:59:01.733 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 0, 2024-03-18T22:59:01.733 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 0, 2024-03-18T22:59:01.733 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T22:59:01.733 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-03-18T22:59:01.733 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T22:59:01.733 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-03-18T22:59:01.733 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T22:59:01.733 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T22:59:01.733 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T22:59:01.734 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T22:59:01.734 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T22:59:01.734 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-03-18T22:59:01.734 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T22:59:01.734 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T22:59:01.734 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-03-18T22:59:01.734 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-03-18T22:59:01.734 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-03-18T22:59:01.734 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-03-18T22:59:01.734 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T22:59:01.734 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T22:59:01.734 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T22:59:01.734 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T22:59:01.734 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 285932, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 122, 2024-03-18T22:59:01.735 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 0, 2024-03-18T22:59:01.736 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 0, 2024-03-18T22:59:01.736 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-03-18T22:59:01.736 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-03-18T22:59:01.736 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-03-18T22:59:01.736 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-03-18T22:59:01.736 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-03-18T22:59:01.736 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-03-18T22:59:01.736 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 9184, 2024-03-18T22:59:01.736 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 177, 2024-03-18T22:59:01.736 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T22:59:01.736 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-03-18T22:59:01.736 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-03-18T22:59:01.736 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-03-18T22:59:01.736 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 105117, 2024-03-18T22:59:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 408, 2024-03-18T22:59:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-03-18T22:59:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-03-18T22:59:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-03-18T22:59:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-03-18T22:59:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter": { 2024-03-18T22:59:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: "op_active": 0, 2024-03-18T22:59:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: "op_laggy": 0, 2024-03-18T22:59:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send": 1302, 2024-03-18T22:59:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send_bytes": 333176, 2024-03-18T22:59:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: "op_resend": 33, 2024-03-18T22:59:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: "op_reply": 1302, 2024-03-18T22:59:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: "op_latency": { 2024-03-18T22:59:01.737 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1302, 2024-03-18T22:59:01.738 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 221.609693654, 2024-03-18T22:59:01.738 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.170207137 2024-03-18T22:59:01.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.738 INFO:teuthology.orchestra.run.smithi103.stdout: "op_inflight": 0, 2024-03-18T22:59:01.738 INFO:teuthology.orchestra.run.smithi103.stdout: "oplen_avg": { 2024-03-18T22:59:01.738 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1302, 2024-03-18T22:59:01.738 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 1386 2024-03-18T22:59:01.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.738 INFO:teuthology.orchestra.run.smithi103.stdout: "op": 1302, 2024-03-18T22:59:01.738 INFO:teuthology.orchestra.run.smithi103.stdout: "op_r": 82, 2024-03-18T22:59:01.738 INFO:teuthology.orchestra.run.smithi103.stdout: "op_w": 1220, 2024-03-18T22:59:01.738 INFO:teuthology.orchestra.run.smithi103.stdout: "op_rmw": 0, 2024-03-18T22:59:01.738 INFO:teuthology.orchestra.run.smithi103.stdout: "op_pg": 0, 2024-03-18T22:59:01.739 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_stat": 0, 2024-03-18T22:59:01.739 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_create": 10, 2024-03-18T22:59:01.739 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_read": 2, 2024-03-18T22:59:01.739 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_write": 114, 2024-03-18T22:59:01.739 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writefull": 29, 2024-03-18T22:59:01.739 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writesame": 0, 2024-03-18T22:59:01.739 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_append": 0, 2024-03-18T22:59:01.739 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_zero": 0, 2024-03-18T22:59:01.739 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_truncate": 0, 2024-03-18T22:59:01.739 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_delete": 1000, 2024-03-18T22:59:01.739 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_mapext": 0, 2024-03-18T22:59:01.739 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_sparse_read": 0, 2024-03-18T22:59:01.739 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_clonerange": 0, 2024-03-18T22:59:01.739 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_getxattr": 80, 2024-03-18T22:59:01.739 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_setxattr": 20, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_cmpxattr": 0, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_rmxattr": 0, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_resetxattrs": 0, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_call": 0, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_watch": 0, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_notify": 0, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls": 0, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls_filter": 0, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_other": 0, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_active": 0, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_send": 0, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_resend": 0, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_ping": 0, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_active": 0, 2024-03-18T22:59:01.740 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_send": 0, 2024-03-18T22:59:01.741 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_resend": 0, 2024-03-18T22:59:01.741 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_active": 0, 2024-03-18T22:59:01.741 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_send": 0, 2024-03-18T22:59:01.741 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_resend": 0, 2024-03-18T22:59:01.741 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_active": 0, 2024-03-18T22:59:01.741 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_send": 0, 2024-03-18T22:59:01.741 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_resend": 0, 2024-03-18T22:59:01.741 INFO:teuthology.orchestra.run.smithi103.stdout: "command_active": 0, 2024-03-18T22:59:01.741 INFO:teuthology.orchestra.run.smithi103.stdout: "command_send": 0, 2024-03-18T22:59:01.741 INFO:teuthology.orchestra.run.smithi103.stdout: "command_resend": 0, 2024-03-18T22:59:01.741 INFO:teuthology.orchestra.run.smithi103.stdout: "map_epoch": 93, 2024-03-18T22:59:01.741 INFO:teuthology.orchestra.run.smithi103.stdout: "map_full": 0, 2024-03-18T22:59:01.741 INFO:teuthology.orchestra.run.smithi103.stdout: "map_inc": 4, 2024-03-18T22:59:01.741 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_sessions": 12, 2024-03-18T22:59:01.741 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_open": 13, 2024-03-18T22:59:01.742 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_close": 1, 2024-03-18T22:59:01.742 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_laggy": 0, 2024-03-18T22:59:01.742 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_wr": 105, 2024-03-18T22:59:01.742 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_rd": 0, 2024-03-18T22:59:01.742 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_del": 26 2024-03-18T22:59:01.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.742 INFO:teuthology.orchestra.run.smithi103.stdout: "oft": { 2024-03-18T22:59:01.742 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_objs": 1, 2024-03-18T22:59:01.742 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_kv_pairs": 0, 2024-03-18T22:59:01.742 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_updates": 0, 2024-03-18T22:59:01.742 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_removes": 0 2024-03-18T22:59:01.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.742 INFO:teuthology.orchestra.run.smithi103.stdout: "osd-slow-ops": { 2024-03-18T22:59:01.742 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_ops_count": 0 2024-03-18T22:59:01.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.743 INFO:teuthology.orchestra.run.smithi103.stdout: "purge_queue": { 2024-03-18T22:59:01.743 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops": 0, 2024-03-18T22:59:01.743 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops_high_water": 52, 2024-03-18T22:59:01.743 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing": 0, 2024-03-18T22:59:01.743 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_high_water": 1, 2024-03-18T22:59:01.743 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed_ops": 988, 2024-03-18T22:59:01.743 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed": 20, 2024-03-18T22:59:01.743 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_item_in_journal": 0 2024-03-18T22:59:01.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.743 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T22:59:01.743 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:59:01.743 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-03-18T22:59:01.743 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:59:01.743 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 6367, 2024-03-18T22:59:01.744 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 2401466, 2024-03-18T22:59:01.744 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:01.744 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 6367, 2024-03-18T22:59:01.744 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-03-18T22:59:01.744 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-03-18T22:59:01.744 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 6367, 2024-03-18T22:59:01.744 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 2401466, 2024-03-18T22:59:01.744 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:59:01.744 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.744 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.744 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.744 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:59:01.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.744 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_bytes": { 2024-03-18T22:59:01.744 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:59:01.745 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-03-18T22:59:01.745 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:59:01.745 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-03-18T22:59:01.745 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-03-18T22:59:01.745 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:01.745 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-03-18T22:59:01.745 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 1302, 2024-03-18T22:59:01.745 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 335356, 2024-03-18T22:59:01.745 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 1302, 2024-03-18T22:59:01.745 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 335356, 2024-03-18T22:59:01.745 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:59:01.745 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.745 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.745 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.746 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:59:01.746 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.746 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_ops": { 2024-03-18T22:59:01.746 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:59:01.746 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 1024, 2024-03-18T22:59:01.746 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:59:01.746 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-03-18T22:59:01.746 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-03-18T22:59:01.746 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:01.746 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-03-18T22:59:01.746 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 1302, 2024-03-18T22:59:01.746 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 1302, 2024-03-18T22:59:01.746 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 1302, 2024-03-18T22:59:01.746 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 1302, 2024-03-18T22:59:01.747 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:59:01.747 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.747 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.747 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.747 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:59:01.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.747 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle": { 2024-03-18T22:59:01.747 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:59:01.747 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-03-18T22:59:01.747 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:59:01.747 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 20, 2024-03-18T22:59:01.747 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 2180, 2024-03-18T22:59:01.747 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:01.747 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 20, 2024-03-18T22:59:01.747 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-03-18T22:59:01.748 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-03-18T22:59:01.748 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 2, 2024-03-18T22:59:01.748 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 2180, 2024-03-18T22:59:01.748 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:59:01.748 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.748 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.748 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.748 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:59:01.748 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-03-18T22:59:01.748 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle-0x55a2f58be500": { 2024-03-18T22:59:01.748 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-03-18T22:59:01.748 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-03-18T22:59:01.748 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-03-18T22:59:01.748 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 251, 2024-03-18T22:59:01.748 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 309623, 2024-03-18T22:59:01.749 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:01.749 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 251, 2024-03-18T22:59:01.749 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-03-18T22:59:01.749 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-03-18T22:59:01.749 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 112, 2024-03-18T22:59:01.749 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 309623, 2024-03-18T22:59:01.749 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-03-18T22:59:01.749 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-03-18T22:59:01.749 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-03-18T22:59:01.749 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-03-18T22:59:01.749 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:59:01.749 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-03-18T22:59:01.749 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-03-18T22:59:02.474 INFO:tasks.check_counter:Daemon mds.k mds.exported=1 2024-03-18T22:59:02.474 INFO:tasks.check_counter:Daemon mds.k mds.imported=1 2024-03-18T22:59:02.474 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=1 2024-03-18T22:59:02.474 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=311 2024-03-18T22:59:02.474 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-03-18T22:59:02.474 DEBUG:tasks.check_counter:Getting stats from c 2024-03-18T22:59:02.475 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-18T22:59:02.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:02 smithi113 ceph-mon[66775]: pgmap v842: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-18T22:59:02.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:02 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:59:02.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:02 smithi103 ceph-mon[66377]: pgmap v842: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-18T22:59:02.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:02 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:59:02.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:02 smithi040 ceph-mon[62643]: pgmap v842: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-18T22:59:02.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:02 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T22:59:02.971 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:59:03.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:03 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:59:03.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:03 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:59:03.743 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:03 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:59:03.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:03 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:59:03.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:03 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:59:03.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:03 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:59:03.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:03 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T22:59:03.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:03 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T22:59:03.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:03 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' 2024-03-18T22:59:04.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:04 smithi113 ceph-mon[66775]: pgmap v843: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:59:04.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:04 smithi103 ceph-mon[66377]: pgmap v843: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:59:04.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:04 smithi040 ceph-mon[62643]: pgmap v843: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:59:05.698 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-18T22:59:05.698 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T22:59:05.698 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 1268, 2024-03-18T22:59:05.698 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 1270, 2024-03-18T22:59:05.699 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 4605403, 2024-03-18T22:59:05.699 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 403234, 2024-03-18T22:59:05.699 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 6, 2024-03-18T22:59:05.699 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 2, 2024-03-18T22:59:05.699 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.098094988, 2024-03-18T22:59:05.699 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.028163112, 2024-03-18T22:59:05.700 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.027418965, 2024-03-18T22:59:05.700 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.033834234, 2024-03-18T22:59:05.700 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:59:05.700 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 1268, 2024-03-18T22:59:05.700 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.045185014, 2024-03-18T22:59:05.701 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000035634 2024-03-18T22:59:05.701 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.701 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:59:05.701 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.701 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.701 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.701 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.702 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T22:59:05.702 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T22:59:05.702 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.702 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T22:59:05.702 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 468, 2024-03-18T22:59:05.702 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 436, 2024-03-18T22:59:05.703 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 727672, 2024-03-18T22:59:05.703 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 226522, 2024-03-18T22:59:05.703 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 6, 2024-03-18T22:59:05.703 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 3, 2024-03-18T22:59:05.703 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.045946492, 2024-03-18T22:59:05.703 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.015720235, 2024-03-18T22:59:05.704 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.020297560, 2024-03-18T22:59:05.704 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.009456505, 2024-03-18T22:59:05.704 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:59:05.704 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 436, 2024-03-18T22:59:05.704 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.018775688, 2024-03-18T22:59:05.705 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000043063 2024-03-18T22:59:05.705 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.705 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:59:05.705 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.705 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.705 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.706 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.706 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 696496, 2024-03-18T22:59:05.706 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 187024 2024-03-18T22:59:05.706 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.706 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T22:59:05.706 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 2086, 2024-03-18T22:59:05.706 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 2312, 2024-03-18T22:59:05.707 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 6002929, 2024-03-18T22:59:05.707 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 1226059, 2024-03-18T22:59:05.707 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 6, 2024-03-18T22:59:05.707 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 4, 2024-03-18T22:59:05.707 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.135149313, 2024-03-18T22:59:05.707 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.042901166, 2024-03-18T22:59:05.707 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.032641948, 2024-03-18T22:59:05.707 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.050792794, 2024-03-18T22:59:05.707 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:59:05.707 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 2310, 2024-03-18T22:59:05.707 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.093912246, 2024-03-18T22:59:05.708 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000040654 2024-03-18T22:59:05.708 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.708 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:59:05.708 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.708 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.708 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.708 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.708 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T22:59:05.709 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 563664 2024-03-18T22:59:05.709 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.709 INFO:teuthology.orchestra.run.smithi113.stdout: "cct": { 2024-03-18T22:59:05.709 INFO:teuthology.orchestra.run.smithi113.stdout: "total_workers": 1, 2024-03-18T22:59:05.709 INFO:teuthology.orchestra.run.smithi113.stdout: "unhealthy_workers": 0 2024-03-18T22:59:05.709 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.709 INFO:teuthology.orchestra.run.smithi113.stdout: "finisher-MDSRank": { 2024-03-18T22:59:05.709 INFO:teuthology.orchestra.run.smithi113.stdout: "queue_len": 0, 2024-03-18T22:59:05.709 INFO:teuthology.orchestra.run.smithi113.stdout: "complete_latency": { 2024-03-18T22:59:05.709 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 5772, 2024-03-18T22:59:05.710 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.120700156, 2024-03-18T22:59:05.710 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000020911 2024-03-18T22:59:05.710 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:05.710 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.710 INFO:teuthology.orchestra.run.smithi113.stdout: "finisher-PurgeQueue": { 2024-03-18T22:59:05.710 INFO:teuthology.orchestra.run.smithi113.stdout: "queue_len": 0, 2024-03-18T22:59:05.710 INFO:teuthology.orchestra.run.smithi113.stdout: "complete_latency": { 2024-03-18T22:59:05.710 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.710 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.711 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.711 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:05.711 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.711 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-03-18T22:59:05.711 INFO:teuthology.orchestra.run.smithi113.stdout: "request": 0, 2024-03-18T22:59:05.711 INFO:teuthology.orchestra.run.smithi113.stdout: "reply": 0, 2024-03-18T22:59:05.711 INFO:teuthology.orchestra.run.smithi113.stdout: "reply_latency": { 2024-03-18T22:59:05.711 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.711 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.712 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.712 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.712 INFO:teuthology.orchestra.run.smithi113.stdout: "slow_reply": 0, 2024-03-18T22:59:05.712 INFO:teuthology.orchestra.run.smithi113.stdout: "forward": 0, 2024-03-18T22:59:05.712 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_fetch_complete": 0, 2024-03-18T22:59:05.712 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_fetch_keys": 0, 2024-03-18T22:59:05.712 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_commit": 0, 2024-03-18T22:59:05.712 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_split": 0, 2024-03-18T22:59:05.712 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_merge": 0, 2024-03-18T22:59:05.712 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes": 1044, 2024-03-18T22:59:05.713 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_top": 0, 2024-03-18T22:59:05.713 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_bottom": 1044, 2024-03-18T22:59:05.713 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_pin_tail": 0, 2024-03-18T22:59:05.713 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_pinned": 1044, 2024-03-18T22:59:05.713 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_expired": 0, 2024-03-18T22:59:05.713 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_with_caps": 0, 2024-03-18T22:59:05.713 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": 0, 2024-03-18T22:59:05.713 INFO:teuthology.orchestra.run.smithi113.stdout: "subtrees": 2, 2024-03-18T22:59:05.713 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse": 0, 2024-03-18T22:59:05.713 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_hit": 0, 2024-03-18T22:59:05.714 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_forward": 0, 2024-03-18T22:59:05.714 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_discover": 0, 2024-03-18T22:59:05.714 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_dir_fetch": 0, 2024-03-18T22:59:05.714 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_remote_ino": 0, 2024-03-18T22:59:05.714 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_lock": 0, 2024-03-18T22:59:05.714 INFO:teuthology.orchestra.run.smithi113.stdout: "load_cent": 0, 2024-03-18T22:59:05.714 INFO:teuthology.orchestra.run.smithi113.stdout: "q": 0, 2024-03-18T22:59:05.714 INFO:teuthology.orchestra.run.smithi113.stdout: "exported": 0, 2024-03-18T22:59:05.715 INFO:teuthology.orchestra.run.smithi113.stdout: "exported_inodes": 0, 2024-03-18T22:59:05.715 INFO:teuthology.orchestra.run.smithi113.stdout: "imported": 0, 2024-03-18T22:59:05.715 INFO:teuthology.orchestra.run.smithi113.stdout: "imported_inodes": 0, 2024-03-18T22:59:05.715 INFO:teuthology.orchestra.run.smithi113.stdout: "openino_dir_fetch": 0, 2024-03-18T22:59:05.715 INFO:teuthology.orchestra.run.smithi113.stdout: "openino_backtrace_fetch": 0, 2024-03-18T22:59:05.715 INFO:teuthology.orchestra.run.smithi113.stdout: "openino_peer_discover": 0, 2024-03-18T22:59:05.715 INFO:teuthology.orchestra.run.smithi113.stdout: "root_rfiles": 4, 2024-03-18T22:59:05.715 INFO:teuthology.orchestra.run.smithi113.stdout: "root_rbytes": 234, 2024-03-18T22:59:05.715 INFO:teuthology.orchestra.run.smithi113.stdout: "root_rsnaps": 0, 2024-03-18T22:59:05.715 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T22:59:05.716 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_set_tag": 0, 2024-03-18T22:59:05.716 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T22:59:05.716 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_inotable_repaired": 0, 2024-03-18T22:59:05.716 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_dir_inodes": 0, 2024-03-18T22:59:05.716 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T22:59:05.716 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T22:59:05.716 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_file_inodes": 0, 2024-03-18T22:59:05.716 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_inode_file_caps": 0, 2024-03-18T22:59:05.716 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T22:59:05.717 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_grant": 0, 2024-03-18T22:59:05.717 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T22:59:05.717 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T22:59:05.717 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T22:59:05.717 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_caps": 0, 2024-03-18T22:59:05.717 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_caps_dirty": 0, 2024-03-18T22:59:05.717 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_cap_release": 0, 2024-03-18T22:59:05.717 INFO:teuthology.orchestra.run.smithi113.stdout: "process_request_cap_release": 0 2024-03-18T22:59:05.717 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.717 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_cache": { 2024-03-18T22:59:05.718 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_update": 0, 2024-03-18T22:59:05.718 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_update_receipt": 0, 2024-03-18T22:59:05.718 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_try_discover": 0, 2024-03-18T22:59:05.718 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_send_discover": 0, 2024-03-18T22:59:05.718 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_handle_discover": 0, 2024-03-18T22:59:05.718 INFO:teuthology.orchestra.run.smithi113.stdout: "num_strays": 0, 2024-03-18T22:59:05.719 INFO:teuthology.orchestra.run.smithi113.stdout: "num_strays_delayed": 0, 2024-03-18T22:59:05.719 INFO:teuthology.orchestra.run.smithi113.stdout: "num_strays_enqueuing": 0, 2024-03-18T22:59:05.719 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_created": 0, 2024-03-18T22:59:05.719 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_enqueued": 0, 2024-03-18T22:59:05.719 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_reintegrated": 0, 2024-03-18T22:59:05.720 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_migrated": 0, 2024-03-18T22:59:05.720 INFO:teuthology.orchestra.run.smithi113.stdout: "num_recovering_processing": 0, 2024-03-18T22:59:05.720 INFO:teuthology.orchestra.run.smithi113.stdout: "num_recovering_enqueued": 0, 2024-03-18T22:59:05.720 INFO:teuthology.orchestra.run.smithi113.stdout: "num_recovering_prioritized": 0, 2024-03-18T22:59:05.720 INFO:teuthology.orchestra.run.smithi113.stdout: "recovery_started": 0, 2024-03-18T22:59:05.720 INFO:teuthology.orchestra.run.smithi113.stdout: "recovery_completed": 0, 2024-03-18T22:59:05.721 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_quiesce_path": 0, 2024-03-18T22:59:05.721 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_quiesce_inode": 0, 2024-03-18T22:59:05.721 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T22:59:05.721 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_exportdir": 0, 2024-03-18T22:59:05.721 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_flush": 0, 2024-03-18T22:59:05.721 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_fragmentdir": 0, 2024-03-18T22:59:05.722 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_fragstats": 0, 2024-03-18T22:59:05.722 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_inodestats": 0 2024-03-18T22:59:05.722 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.722 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_log": { 2024-03-18T22:59:05.722 INFO:teuthology.orchestra.run.smithi113.stdout: "evlrg": 0, 2024-03-18T22:59:05.722 INFO:teuthology.orchestra.run.smithi113.stdout: "evadd": 0, 2024-03-18T22:59:05.722 INFO:teuthology.orchestra.run.smithi113.stdout: "evex": 0, 2024-03-18T22:59:05.723 INFO:teuthology.orchestra.run.smithi113.stdout: "evtrm": 0, 2024-03-18T22:59:05.723 INFO:teuthology.orchestra.run.smithi113.stdout: "ev": 4551, 2024-03-18T22:59:05.723 INFO:teuthology.orchestra.run.smithi113.stdout: "evexg": 0, 2024-03-18T22:59:05.723 INFO:teuthology.orchestra.run.smithi113.stdout: "evexd": 0, 2024-03-18T22:59:05.723 INFO:teuthology.orchestra.run.smithi113.stdout: "segadd": 0, 2024-03-18T22:59:05.723 INFO:teuthology.orchestra.run.smithi113.stdout: "segex": 0, 2024-03-18T22:59:05.723 INFO:teuthology.orchestra.run.smithi113.stdout: "segtrm": 0, 2024-03-18T22:59:05.724 INFO:teuthology.orchestra.run.smithi113.stdout: "seg": 6, 2024-03-18T22:59:05.724 INFO:teuthology.orchestra.run.smithi113.stdout: "segmjr": 2, 2024-03-18T22:59:05.724 INFO:teuthology.orchestra.run.smithi113.stdout: "segexg": 0, 2024-03-18T22:59:05.724 INFO:teuthology.orchestra.run.smithi113.stdout: "segexd": 0, 2024-03-18T22:59:05.724 INFO:teuthology.orchestra.run.smithi113.stdout: "expos": 4194304, 2024-03-18T22:59:05.724 INFO:teuthology.orchestra.run.smithi113.stdout: "wrpos": 13863838, 2024-03-18T22:59:05.724 INFO:teuthology.orchestra.run.smithi113.stdout: "rdpos": 13863239, 2024-03-18T22:59:05.725 INFO:teuthology.orchestra.run.smithi113.stdout: "jlat": { 2024-03-18T22:59:05.725 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.725 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.725 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.725 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.725 INFO:teuthology.orchestra.run.smithi113.stdout: "replayed": 4551 2024-03-18T22:59:05.726 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.726 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_mem": { 2024-03-18T22:59:05.726 INFO:teuthology.orchestra.run.smithi113.stdout: "ino": 23, 2024-03-18T22:59:05.726 INFO:teuthology.orchestra.run.smithi113.stdout: "ino+": 976, 2024-03-18T22:59:05.726 INFO:teuthology.orchestra.run.smithi113.stdout: "ino-": 953, 2024-03-18T22:59:05.726 INFO:teuthology.orchestra.run.smithi113.stdout: "dir": 18, 2024-03-18T22:59:05.726 INFO:teuthology.orchestra.run.smithi113.stdout: "dir+": 190, 2024-03-18T22:59:05.727 INFO:teuthology.orchestra.run.smithi113.stdout: "dir-": 172, 2024-03-18T22:59:05.727 INFO:teuthology.orchestra.run.smithi113.stdout: "dn": 1044, 2024-03-18T22:59:05.727 INFO:teuthology.orchestra.run.smithi113.stdout: "dn+": 1608, 2024-03-18T22:59:05.727 INFO:teuthology.orchestra.run.smithi113.stdout: "dn-": 564, 2024-03-18T22:59:05.727 INFO:teuthology.orchestra.run.smithi113.stdout: "cap": 0, 2024-03-18T22:59:05.727 INFO:teuthology.orchestra.run.smithi113.stdout: "cap+": 0, 2024-03-18T22:59:05.728 INFO:teuthology.orchestra.run.smithi113.stdout: "cap-": 0, 2024-03-18T22:59:05.728 INFO:teuthology.orchestra.run.smithi113.stdout: "rss": 54392, 2024-03-18T22:59:05.728 INFO:teuthology.orchestra.run.smithi113.stdout: "heap": 215420 2024-03-18T22:59:05.728 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.728 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_server": { 2024-03-18T22:59:05.728 INFO:teuthology.orchestra.run.smithi113.stdout: "dispatch_client_request": 0, 2024-03-18T22:59:05.729 INFO:teuthology.orchestra.run.smithi113.stdout: "dispatch_server_request": 0, 2024-03-18T22:59:05.729 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_request": 0, 2024-03-18T22:59:05.729 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_session": 0, 2024-03-18T22:59:05.729 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_peer_request": 0, 2024-03-18T22:59:05.729 INFO:teuthology.orchestra.run.smithi113.stdout: "req_create_latency": { 2024-03-18T22:59:05.729 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.730 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.730 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.730 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.730 INFO:teuthology.orchestra.run.smithi113.stdout: "req_getattr_latency": { 2024-03-18T22:59:05.730 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.730 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.731 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.731 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.731 INFO:teuthology.orchestra.run.smithi113.stdout: "req_getfilelock_latency": { 2024-03-18T22:59:05.731 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.731 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.731 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.731 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.732 INFO:teuthology.orchestra.run.smithi113.stdout: "req_link_latency": { 2024-03-18T22:59:05.732 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.732 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.732 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.732 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.732 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookup_latency": { 2024-03-18T22:59:05.733 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.733 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.733 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.733 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.733 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookuphash_latency": { 2024-03-18T22:59:05.733 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.734 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.734 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.734 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.734 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupino_latency": { 2024-03-18T22:59:05.734 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.735 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.735 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.735 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.735 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupname_latency": { 2024-03-18T22:59:05.735 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.735 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.735 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.736 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.736 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupparent_latency": { 2024-03-18T22:59:05.736 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.736 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.737 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.737 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.737 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupsnap_latency": { 2024-03-18T22:59:05.737 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.737 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.737 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.738 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.738 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lssnap_latency": { 2024-03-18T22:59:05.738 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.738 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.738 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.739 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.739 INFO:teuthology.orchestra.run.smithi113.stdout: "req_mkdir_latency": { 2024-03-18T22:59:05.740 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.740 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.740 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.741 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.741 INFO:teuthology.orchestra.run.smithi113.stdout: "req_mknod_latency": { 2024-03-18T22:59:05.741 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.741 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.741 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.741 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.742 INFO:teuthology.orchestra.run.smithi113.stdout: "req_mksnap_latency": { 2024-03-18T22:59:05.742 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.742 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.742 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.742 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.743 INFO:teuthology.orchestra.run.smithi113.stdout: "req_open_latency": { 2024-03-18T22:59:05.743 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.743 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.743 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.743 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.743 INFO:teuthology.orchestra.run.smithi113.stdout: "req_readdir_latency": { 2024-03-18T22:59:05.744 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.744 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.744 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.744 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.744 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rename_latency": { 2024-03-18T22:59:05.744 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.745 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.745 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.745 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.745 INFO:teuthology.orchestra.run.smithi113.stdout: "req_renamesnap_latency": { 2024-03-18T22:59:05.745 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.745 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.746 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.746 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.746 INFO:teuthology.orchestra.run.smithi113.stdout: "req_snapdiff_latency": { 2024-03-18T22:59:05.746 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.746 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.746 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.747 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.747 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rmdir_latency": { 2024-03-18T22:59:05.747 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.747 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.747 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.747 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.747 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rmsnap_latency": { 2024-03-18T22:59:05.748 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.748 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.748 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.748 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.748 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rmxattr_latency": { 2024-03-18T22:59:05.748 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.748 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.748 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.748 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.749 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setattr_latency": { 2024-03-18T22:59:05.749 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.749 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.749 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.749 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.749 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setdirlayout_latency": { 2024-03-18T22:59:05.749 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.749 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.749 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.750 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.750 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setfilelock_latency": { 2024-03-18T22:59:05.750 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.750 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.750 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.750 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.750 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setlayout_latency": { 2024-03-18T22:59:05.751 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.751 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.751 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.751 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.751 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setxattr_latency": { 2024-03-18T22:59:05.751 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.751 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.752 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.752 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.752 INFO:teuthology.orchestra.run.smithi113.stdout: "req_symlink_latency": { 2024-03-18T22:59:05.752 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.752 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.752 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.752 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.752 INFO:teuthology.orchestra.run.smithi113.stdout: "req_unlink_latency": { 2024-03-18T22:59:05.752 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.752 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.752 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.753 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.753 INFO:teuthology.orchestra.run.smithi113.stdout: "cap_revoke_eviction": 0, 2024-03-18T22:59:05.753 INFO:teuthology.orchestra.run.smithi113.stdout: "cap_acquisition_throttle": 0, 2024-03-18T22:59:05.753 INFO:teuthology.orchestra.run.smithi113.stdout: "req_getvxattr_latency": { 2024-03-18T22:59:05.753 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.753 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.753 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.753 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:05.753 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.753 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_sessions": { 2024-03-18T22:59:05.753 INFO:teuthology.orchestra.run.smithi113.stdout: "session_count": 1, 2024-03-18T22:59:05.753 INFO:teuthology.orchestra.run.smithi113.stdout: "session_add": 3, 2024-03-18T22:59:05.753 INFO:teuthology.orchestra.run.smithi113.stdout: "session_remove": 2, 2024-03-18T22:59:05.753 INFO:teuthology.orchestra.run.smithi113.stdout: "sessions_open": 1, 2024-03-18T22:59:05.754 INFO:teuthology.orchestra.run.smithi113.stdout: "sessions_stale": 0, 2024-03-18T22:59:05.754 INFO:teuthology.orchestra.run.smithi113.stdout: "total_load": 0, 2024-03-18T22:59:05.754 INFO:teuthology.orchestra.run.smithi113.stdout: "average_load": 0, 2024-03-18T22:59:05.754 INFO:teuthology.orchestra.run.smithi113.stdout: "avg_session_uptime": 4523, 2024-03-18T22:59:05.754 INFO:teuthology.orchestra.run.smithi113.stdout: "mdthresh_evicted": 0 2024-03-18T22:59:05.754 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.754 INFO:teuthology.orchestra.run.smithi113.stdout: "mempool": { 2024-03-18T22:59:05.754 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_bytes": 0, 2024-03-18T22:59:05.754 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_items": 0, 2024-03-18T22:59:05.754 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T22:59:05.754 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_items": 0, 2024-03-18T22:59:05.754 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T22:59:05.754 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_items": 0, 2024-03-18T22:59:05.754 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T22:59:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T22:59:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T22:59:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T22:59:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T22:59:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_items": 0, 2024-03-18T22:59:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T22:59:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T22:59:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_bytes": 0, 2024-03-18T22:59:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_items": 0, 2024-03-18T22:59:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_bytes": 0, 2024-03-18T22:59:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_items": 0, 2024-03-18T22:59:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T22:59:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T22:59:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T22:59:05.755 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_items": 0, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_bytes": 0, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_items": 0, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_bytes": 0, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_items": 0, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_bytes": 0, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_items": 0, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_items": 0, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_items": 0, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_bytes": 198247, 2024-03-18T22:59:05.756 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_items": 56, 2024-03-18T22:59:05.757 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_bytes": 0, 2024-03-18T22:59:05.757 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_items": 0, 2024-03-18T22:59:05.757 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_bytes": 0, 2024-03-18T22:59:05.757 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_items": 0, 2024-03-18T22:59:05.757 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_bytes": 0, 2024-03-18T22:59:05.757 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_items": 0, 2024-03-18T22:59:05.757 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_bytes": 0, 2024-03-18T22:59:05.757 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_items": 0, 2024-03-18T22:59:05.757 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_bytes": 9184, 2024-03-18T22:59:05.757 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_items": 177, 2024-03-18T22:59:05.757 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T22:59:05.757 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_items": 0, 2024-03-18T22:59:05.757 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_bytes": 0, 2024-03-18T22:59:05.757 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_items": 0, 2024-03-18T22:59:05.757 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_bytes": 809684, 2024-03-18T22:59:05.758 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_items": 5005, 2024-03-18T22:59:05.758 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_bytes": 0, 2024-03-18T22:59:05.758 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_items": 0, 2024-03-18T22:59:05.758 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_bytes": 0, 2024-03-18T22:59:05.758 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_items": 0 2024-03-18T22:59:05.758 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.758 INFO:teuthology.orchestra.run.smithi113.stdout: "objecter": { 2024-03-18T22:59:05.758 INFO:teuthology.orchestra.run.smithi113.stdout: "op_active": 0, 2024-03-18T22:59:05.758 INFO:teuthology.orchestra.run.smithi113.stdout: "op_laggy": 0, 2024-03-18T22:59:05.758 INFO:teuthology.orchestra.run.smithi113.stdout: "op_send": 3475, 2024-03-18T22:59:05.759 INFO:teuthology.orchestra.run.smithi113.stdout: "op_send_bytes": 26, 2024-03-18T22:59:05.759 INFO:teuthology.orchestra.run.smithi113.stdout: "op_resend": 4, 2024-03-18T22:59:05.759 INFO:teuthology.orchestra.run.smithi113.stdout: "op_reply": 3475, 2024-03-18T22:59:05.759 INFO:teuthology.orchestra.run.smithi113.stdout: "op_latency": { 2024-03-18T22:59:05.759 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 3475, 2024-03-18T22:59:05.759 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 21.780581752, 2024-03-18T22:59:05.759 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.006267793 2024-03-18T22:59:05.759 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.759 INFO:teuthology.orchestra.run.smithi113.stdout: "op_inflight": 0, 2024-03-18T22:59:05.759 INFO:teuthology.orchestra.run.smithi113.stdout: "oplen_avg": { 2024-03-18T22:59:05.759 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 3475, 2024-03-18T22:59:05.759 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 3476 2024-03-18T22:59:05.759 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.759 INFO:teuthology.orchestra.run.smithi113.stdout: "op": 3475, 2024-03-18T22:59:05.760 INFO:teuthology.orchestra.run.smithi113.stdout: "op_r": 3475, 2024-03-18T22:59:05.760 INFO:teuthology.orchestra.run.smithi113.stdout: "op_w": 0, 2024-03-18T22:59:05.760 INFO:teuthology.orchestra.run.smithi113.stdout: "op_rmw": 0, 2024-03-18T22:59:05.760 INFO:teuthology.orchestra.run.smithi113.stdout: "op_pg": 0, 2024-03-18T22:59:05.760 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_stat": 2126, 2024-03-18T22:59:05.760 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_create": 0, 2024-03-18T22:59:05.760 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_read": 1346, 2024-03-18T22:59:05.760 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_write": 0, 2024-03-18T22:59:05.760 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_writefull": 0, 2024-03-18T22:59:05.760 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_writesame": 0, 2024-03-18T22:59:05.760 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_append": 0, 2024-03-18T22:59:05.760 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_zero": 0, 2024-03-18T22:59:05.760 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_truncate": 0, 2024-03-18T22:59:05.760 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_delete": 0, 2024-03-18T22:59:05.760 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_mapext": 0, 2024-03-18T22:59:05.761 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_sparse_read": 0, 2024-03-18T22:59:05.761 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_clonerange": 0, 2024-03-18T22:59:05.761 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_getxattr": 2, 2024-03-18T22:59:05.761 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_setxattr": 0, 2024-03-18T22:59:05.761 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_cmpxattr": 0, 2024-03-18T22:59:05.761 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_rmxattr": 0, 2024-03-18T22:59:05.761 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_resetxattrs": 0, 2024-03-18T22:59:05.761 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_call": 0, 2024-03-18T22:59:05.761 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_watch": 0, 2024-03-18T22:59:05.761 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_notify": 0, 2024-03-18T22:59:05.761 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T22:59:05.761 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_pgls": 0, 2024-03-18T22:59:05.761 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_pgls_filter": 0, 2024-03-18T22:59:05.761 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_other": 0, 2024-03-18T22:59:05.761 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_active": 0, 2024-03-18T22:59:05.762 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_send": 0, 2024-03-18T22:59:05.762 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_resend": 0, 2024-03-18T22:59:05.762 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_ping": 0, 2024-03-18T22:59:05.762 INFO:teuthology.orchestra.run.smithi113.stdout: "poolop_active": 0, 2024-03-18T22:59:05.762 INFO:teuthology.orchestra.run.smithi113.stdout: "poolop_send": 0, 2024-03-18T22:59:05.762 INFO:teuthology.orchestra.run.smithi113.stdout: "poolop_resend": 0, 2024-03-18T22:59:05.762 INFO:teuthology.orchestra.run.smithi113.stdout: "poolstat_active": 0, 2024-03-18T22:59:05.762 INFO:teuthology.orchestra.run.smithi113.stdout: "poolstat_send": 0, 2024-03-18T22:59:05.762 INFO:teuthology.orchestra.run.smithi113.stdout: "poolstat_resend": 0, 2024-03-18T22:59:05.762 INFO:teuthology.orchestra.run.smithi113.stdout: "statfs_active": 0, 2024-03-18T22:59:05.762 INFO:teuthology.orchestra.run.smithi113.stdout: "statfs_send": 0, 2024-03-18T22:59:05.762 INFO:teuthology.orchestra.run.smithi113.stdout: "statfs_resend": 0, 2024-03-18T22:59:05.762 INFO:teuthology.orchestra.run.smithi113.stdout: "command_active": 0, 2024-03-18T22:59:05.763 INFO:teuthology.orchestra.run.smithi113.stdout: "command_send": 0, 2024-03-18T22:59:05.763 INFO:teuthology.orchestra.run.smithi113.stdout: "command_resend": 0, 2024-03-18T22:59:05.763 INFO:teuthology.orchestra.run.smithi113.stdout: "map_epoch": 93, 2024-03-18T22:59:05.763 INFO:teuthology.orchestra.run.smithi113.stdout: "map_full": 0, 2024-03-18T22:59:05.763 INFO:teuthology.orchestra.run.smithi113.stdout: "map_inc": 4, 2024-03-18T22:59:05.763 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_sessions": 6, 2024-03-18T22:59:05.763 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_session_open": 8, 2024-03-18T22:59:05.763 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_session_close": 2, 2024-03-18T22:59:05.763 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_laggy": 0, 2024-03-18T22:59:05.763 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_wr": 0, 2024-03-18T22:59:05.763 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_rd": 2, 2024-03-18T22:59:05.763 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_del": 0 2024-03-18T22:59:05.763 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.763 INFO:teuthology.orchestra.run.smithi113.stdout: "oft": { 2024-03-18T22:59:05.763 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_objs": 0, 2024-03-18T22:59:05.764 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_kv_pairs": 0, 2024-03-18T22:59:05.764 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_updates": 0, 2024-03-18T22:59:05.764 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_removes": 0 2024-03-18T22:59:05.764 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.764 INFO:teuthology.orchestra.run.smithi113.stdout: "osd-slow-ops": { 2024-03-18T22:59:05.764 INFO:teuthology.orchestra.run.smithi113.stdout: "slow_ops_count": 0 2024-03-18T22:59:05.764 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.764 INFO:teuthology.orchestra.run.smithi113.stdout: "purge_queue": { 2024-03-18T22:59:05.764 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing_ops": 0, 2024-03-18T22:59:05.764 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T22:59:05.764 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing": 0, 2024-03-18T22:59:05.764 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing_high_water": 0, 2024-03-18T22:59:05.764 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executed_ops": 0, 2024-03-18T22:59:05.764 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executed": 0, 2024-03-18T22:59:05.765 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_item_in_journal": 0 2024-03-18T22:59:05.765 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.765 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T22:59:05.765 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:05.765 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 104857600, 2024-03-18T22:59:05.765 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:05.765 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 3818, 2024-03-18T22:59:05.765 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 10972015, 2024-03-18T22:59:05.765 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:05.765 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 3818, 2024-03-18T22:59:05.765 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-18T22:59:05.765 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-18T22:59:05.765 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 3818, 2024-03-18T22:59:05.765 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 10972015, 2024-03-18T22:59:05.765 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:05.766 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.766 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.766 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.766 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:05.766 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.766 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-objecter_bytes": { 2024-03-18T22:59:05.766 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:05.766 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 104857600, 2024-03-18T22:59:05.766 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:05.766 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 0, 2024-03-18T22:59:05.766 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 0, 2024-03-18T22:59:05.766 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:05.766 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 0, 2024-03-18T22:59:05.767 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 3475, 2024-03-18T22:59:05.767 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 9669544, 2024-03-18T22:59:05.767 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 3475, 2024-03-18T22:59:05.767 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 9669544, 2024-03-18T22:59:05.767 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:05.767 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.767 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.767 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.767 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:05.767 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.767 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-objecter_ops": { 2024-03-18T22:59:05.767 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:05.767 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 1024, 2024-03-18T22:59:05.767 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:05.768 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 0, 2024-03-18T22:59:05.768 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 0, 2024-03-18T22:59:05.768 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:05.768 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 0, 2024-03-18T22:59:05.768 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 3475, 2024-03-18T22:59:05.768 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 3475, 2024-03-18T22:59:05.768 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 3475, 2024-03-18T22:59:05.768 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 3475, 2024-03-18T22:59:05.768 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:05.768 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.768 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.768 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.768 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:05.769 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.769 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-write_buf_throttle": { 2024-03-18T22:59:05.769 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:05.769 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 3758096384, 2024-03-18T22:59:05.769 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:05.769 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 0, 2024-03-18T22:59:05.769 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 0, 2024-03-18T22:59:05.769 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:05.769 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 0, 2024-03-18T22:59:05.769 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-18T22:59:05.769 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-18T22:59:05.769 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 0, 2024-03-18T22:59:05.769 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 0, 2024-03-18T22:59:05.769 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:05.769 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.770 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.770 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.770 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:05.770 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:05.770 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-write_buf_throttle-0x55e4fcc486e0": { 2024-03-18T22:59:05.770 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:05.770 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 3758096384, 2024-03-18T22:59:05.770 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:05.770 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 0, 2024-03-18T22:59:05.770 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 0, 2024-03-18T22:59:05.770 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:05.770 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 0, 2024-03-18T22:59:05.771 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-18T22:59:05.771 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-18T22:59:05.771 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 0, 2024-03-18T22:59:05.771 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 0, 2024-03-18T22:59:05.771 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:05.771 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:05.771 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:05.772 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:05.772 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:05.772 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:05.772 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-18T22:59:06.661 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-18T22:59:06.661 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-18T22:59:06.661 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-18T22:59:06.661 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-03-18T22:59:06.661 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-03-18T22:59:06.662 DEBUG:tasks.check_counter:Getting stats from f 2024-03-18T22:59:06.662 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-18T22:59:06.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:06 smithi113 ceph-mon[66775]: pgmap v844: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:59:06.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:06 smithi103 ceph-mon[66377]: pgmap v844: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:59:06.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:06 smithi040 ceph-mon[62643]: pgmap v844: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:59:06.926 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:59:08.736 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:08 smithi113 ceph-mon[66775]: pgmap v845: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T22:59:08.764 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-18T22:59:08.764 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T22:59:08.764 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 1208, 2024-03-18T22:59:08.765 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 1210, 2024-03-18T22:59:08.765 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 397123, 2024-03-18T22:59:08.765 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 384151, 2024-03-18T22:59:08.765 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 6, 2024-03-18T22:59:08.765 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 2, 2024-03-18T22:59:08.765 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.093674009, 2024-03-18T22:59:08.765 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.028184475, 2024-03-18T22:59:08.765 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.024389424, 2024-03-18T22:59:08.765 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.032664908, 2024-03-18T22:59:08.765 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:59:08.765 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 1208, 2024-03-18T22:59:08.765 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.040363692, 2024-03-18T22:59:08.766 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000033413 2024-03-18T22:59:08.766 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.766 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:59:08.766 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.766 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.766 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.766 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.766 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T22:59:08.766 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T22:59:08.766 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.766 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T22:59:08.767 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 2873, 2024-03-18T22:59:08.767 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 2840, 2024-03-18T22:59:08.767 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 4823572, 2024-03-18T22:59:08.767 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 991792, 2024-03-18T22:59:08.767 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 4, 2024-03-18T22:59:08.767 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 3, 2024-03-18T22:59:08.767 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.193955311, 2024-03-18T22:59:08.767 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.054909391, 2024-03-18T22:59:08.767 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.056598194, 2024-03-18T22:59:08.767 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.075554494, 2024-03-18T22:59:08.768 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:59:08.768 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 2840, 2024-03-18T22:59:08.768 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.087668347, 2024-03-18T22:59:08.768 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000030869 2024-03-18T22:59:08.768 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.768 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:59:08.768 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.768 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.768 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.768 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.768 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 696880, 2024-03-18T22:59:08.768 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 188144 2024-03-18T22:59:08.769 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.769 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T22:59:08.769 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 3, 2024-03-18T22:59:08.769 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 231, 2024-03-18T22:59:08.769 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 588, 2024-03-18T22:59:08.769 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 564158, 2024-03-18T22:59:08.769 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 6, 2024-03-18T22:59:08.769 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 3, 2024-03-18T22:59:08.769 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.011731142, 2024-03-18T22:59:08.769 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.007794820, 2024-03-18T22:59:08.770 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.002079433, 2024-03-18T22:59:08.770 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.000022301, 2024-03-18T22:59:08.770 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:59:08.770 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 229, 2024-03-18T22:59:08.770 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.014126974, 2024-03-18T22:59:08.770 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000061689 2024-03-18T22:59:08.770 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.770 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:59:08.770 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.770 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.770 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.770 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.770 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T22:59:08.771 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 563840 2024-03-18T22:59:08.771 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.771 INFO:teuthology.orchestra.run.smithi113.stdout: "cct": { 2024-03-18T22:59:08.771 INFO:teuthology.orchestra.run.smithi113.stdout: "total_workers": 1, 2024-03-18T22:59:08.771 INFO:teuthology.orchestra.run.smithi113.stdout: "unhealthy_workers": 0 2024-03-18T22:59:08.771 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.771 INFO:teuthology.orchestra.run.smithi113.stdout: "finisher-MDSRank": { 2024-03-18T22:59:08.771 INFO:teuthology.orchestra.run.smithi113.stdout: "queue_len": 0, 2024-03-18T22:59:08.771 INFO:teuthology.orchestra.run.smithi113.stdout: "complete_latency": { 2024-03-18T22:59:08.771 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 6420, 2024-03-18T22:59:08.771 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.124017292, 2024-03-18T22:59:08.771 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000019317 2024-03-18T22:59:08.771 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:08.771 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.771 INFO:teuthology.orchestra.run.smithi113.stdout: "finisher-PurgeQueue": { 2024-03-18T22:59:08.772 INFO:teuthology.orchestra.run.smithi113.stdout: "queue_len": 0, 2024-03-18T22:59:08.772 INFO:teuthology.orchestra.run.smithi113.stdout: "complete_latency": { 2024-03-18T22:59:08.772 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.772 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.772 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.772 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:08.772 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.772 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-03-18T22:59:08.772 INFO:teuthology.orchestra.run.smithi113.stdout: "request": 0, 2024-03-18T22:59:08.772 INFO:teuthology.orchestra.run.smithi113.stdout: "reply": 0, 2024-03-18T22:59:08.772 INFO:teuthology.orchestra.run.smithi113.stdout: "reply_latency": { 2024-03-18T22:59:08.772 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.772 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.772 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.773 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.773 INFO:teuthology.orchestra.run.smithi113.stdout: "slow_reply": 0, 2024-03-18T22:59:08.773 INFO:teuthology.orchestra.run.smithi113.stdout: "forward": 0, 2024-03-18T22:59:08.773 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_fetch_complete": 0, 2024-03-18T22:59:08.773 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_fetch_keys": 0, 2024-03-18T22:59:08.773 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_commit": 0, 2024-03-18T22:59:08.773 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_split": 0, 2024-03-18T22:59:08.773 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_merge": 0, 2024-03-18T22:59:08.773 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes": 84, 2024-03-18T22:59:08.773 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_top": 17, 2024-03-18T22:59:08.773 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_bottom": 67, 2024-03-18T22:59:08.774 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_pin_tail": 0, 2024-03-18T22:59:08.774 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_pinned": 59, 2024-03-18T22:59:08.774 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_expired": 0, 2024-03-18T22:59:08.774 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_with_caps": 0, 2024-03-18T22:59:08.774 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": 0, 2024-03-18T22:59:08.774 INFO:teuthology.orchestra.run.smithi113.stdout: "subtrees": 1, 2024-03-18T22:59:08.774 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse": 0, 2024-03-18T22:59:08.774 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_hit": 0, 2024-03-18T22:59:08.774 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_forward": 0, 2024-03-18T22:59:08.774 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_discover": 0, 2024-03-18T22:59:08.774 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_dir_fetch": 0, 2024-03-18T22:59:08.775 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_remote_ino": 0, 2024-03-18T22:59:08.775 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_lock": 0, 2024-03-18T22:59:08.775 INFO:teuthology.orchestra.run.smithi113.stdout: "load_cent": 0, 2024-03-18T22:59:08.775 INFO:teuthology.orchestra.run.smithi113.stdout: "q": 0, 2024-03-18T22:59:08.775 INFO:teuthology.orchestra.run.smithi113.stdout: "exported": 0, 2024-03-18T22:59:08.775 INFO:teuthology.orchestra.run.smithi113.stdout: "exported_inodes": 0, 2024-03-18T22:59:08.775 INFO:teuthology.orchestra.run.smithi113.stdout: "imported": 0, 2024-03-18T22:59:08.775 INFO:teuthology.orchestra.run.smithi113.stdout: "imported_inodes": 0, 2024-03-18T22:59:08.775 INFO:teuthology.orchestra.run.smithi113.stdout: "openino_dir_fetch": 0, 2024-03-18T22:59:08.775 INFO:teuthology.orchestra.run.smithi113.stdout: "openino_backtrace_fetch": 0, 2024-03-18T22:59:08.776 INFO:teuthology.orchestra.run.smithi113.stdout: "openino_peer_discover": 0, 2024-03-18T22:59:08.776 INFO:teuthology.orchestra.run.smithi113.stdout: "root_rfiles": 0, 2024-03-18T22:59:08.776 INFO:teuthology.orchestra.run.smithi113.stdout: "root_rbytes": 0, 2024-03-18T22:59:08.776 INFO:teuthology.orchestra.run.smithi113.stdout: "root_rsnaps": 0, 2024-03-18T22:59:08.776 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T22:59:08.776 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_set_tag": 0, 2024-03-18T22:59:08.776 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T22:59:08.776 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_inotable_repaired": 0, 2024-03-18T22:59:08.776 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_dir_inodes": 0, 2024-03-18T22:59:08.776 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T22:59:08.777 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T22:59:08.777 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_file_inodes": 0, 2024-03-18T22:59:08.777 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_inode_file_caps": 0, 2024-03-18T22:59:08.777 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T22:59:08.777 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_grant": 0, 2024-03-18T22:59:08.777 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T22:59:08.777 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T22:59:08.777 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T22:59:08.777 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_caps": 0, 2024-03-18T22:59:08.777 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_caps_dirty": 0, 2024-03-18T22:59:08.778 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_cap_release": 0, 2024-03-18T22:59:08.778 INFO:teuthology.orchestra.run.smithi113.stdout: "process_request_cap_release": 0 2024-03-18T22:59:08.778 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.778 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_cache": { 2024-03-18T22:59:08.778 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_update": 0, 2024-03-18T22:59:08.778 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_update_receipt": 0, 2024-03-18T22:59:08.778 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_try_discover": 0, 2024-03-18T22:59:08.778 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_send_discover": 0, 2024-03-18T22:59:08.778 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_handle_discover": 0, 2024-03-18T22:59:08.778 INFO:teuthology.orchestra.run.smithi113.stdout: "num_strays": 0, 2024-03-18T22:59:08.778 INFO:teuthology.orchestra.run.smithi113.stdout: "num_strays_delayed": 0, 2024-03-18T22:59:08.779 INFO:teuthology.orchestra.run.smithi113.stdout: "num_strays_enqueuing": 0, 2024-03-18T22:59:08.779 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_created": 0, 2024-03-18T22:59:08.779 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_enqueued": 0, 2024-03-18T22:59:08.779 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_reintegrated": 0, 2024-03-18T22:59:08.779 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_migrated": 0, 2024-03-18T22:59:08.779 INFO:teuthology.orchestra.run.smithi113.stdout: "num_recovering_processing": 0, 2024-03-18T22:59:08.779 INFO:teuthology.orchestra.run.smithi113.stdout: "num_recovering_enqueued": 0, 2024-03-18T22:59:08.779 INFO:teuthology.orchestra.run.smithi113.stdout: "num_recovering_prioritized": 0, 2024-03-18T22:59:08.779 INFO:teuthology.orchestra.run.smithi113.stdout: "recovery_started": 0, 2024-03-18T22:59:08.779 INFO:teuthology.orchestra.run.smithi113.stdout: "recovery_completed": 0, 2024-03-18T22:59:08.780 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_quiesce_path": 0, 2024-03-18T22:59:08.780 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_quiesce_inode": 0, 2024-03-18T22:59:08.780 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T22:59:08.780 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_exportdir": 0, 2024-03-18T22:59:08.780 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_flush": 0, 2024-03-18T22:59:08.780 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_fragmentdir": 0, 2024-03-18T22:59:08.780 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_fragstats": 0, 2024-03-18T22:59:08.780 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_inodestats": 0 2024-03-18T22:59:08.780 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.780 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_log": { 2024-03-18T22:59:08.780 INFO:teuthology.orchestra.run.smithi113.stdout: "evlrg": 0, 2024-03-18T22:59:08.781 INFO:teuthology.orchestra.run.smithi113.stdout: "evadd": 0, 2024-03-18T22:59:08.781 INFO:teuthology.orchestra.run.smithi113.stdout: "evex": 0, 2024-03-18T22:59:08.781 INFO:teuthology.orchestra.run.smithi113.stdout: "evtrm": 0, 2024-03-18T22:59:08.781 INFO:teuthology.orchestra.run.smithi113.stdout: "ev": 1727, 2024-03-18T22:59:08.781 INFO:teuthology.orchestra.run.smithi113.stdout: "evexg": 0, 2024-03-18T22:59:08.781 INFO:teuthology.orchestra.run.smithi113.stdout: "evexd": 0, 2024-03-18T22:59:08.781 INFO:teuthology.orchestra.run.smithi113.stdout: "segadd": 0, 2024-03-18T22:59:08.781 INFO:teuthology.orchestra.run.smithi113.stdout: "segex": 0, 2024-03-18T22:59:08.781 INFO:teuthology.orchestra.run.smithi113.stdout: "segtrm": 0, 2024-03-18T22:59:08.781 INFO:teuthology.orchestra.run.smithi113.stdout: "seg": 3, 2024-03-18T22:59:08.781 INFO:teuthology.orchestra.run.smithi113.stdout: "segmjr": 2, 2024-03-18T22:59:08.781 INFO:teuthology.orchestra.run.smithi113.stdout: "segexg": 0, 2024-03-18T22:59:08.781 INFO:teuthology.orchestra.run.smithi113.stdout: "segexd": 0, 2024-03-18T22:59:08.781 INFO:teuthology.orchestra.run.smithi113.stdout: "expos": 4194304, 2024-03-18T22:59:08.782 INFO:teuthology.orchestra.run.smithi113.stdout: "wrpos": 7684368, 2024-03-18T22:59:08.782 INFO:teuthology.orchestra.run.smithi113.stdout: "rdpos": 7684307, 2024-03-18T22:59:08.782 INFO:teuthology.orchestra.run.smithi113.stdout: "jlat": { 2024-03-18T22:59:08.782 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.782 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.782 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.782 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.782 INFO:teuthology.orchestra.run.smithi113.stdout: "replayed": 1727 2024-03-18T22:59:08.782 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.782 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_mem": { 2024-03-18T22:59:08.782 INFO:teuthology.orchestra.run.smithi113.stdout: "ino": 5, 2024-03-18T22:59:08.782 INFO:teuthology.orchestra.run.smithi113.stdout: "ino+": 324, 2024-03-18T22:59:08.782 INFO:teuthology.orchestra.run.smithi113.stdout: "ino-": 319, 2024-03-18T22:59:08.782 INFO:teuthology.orchestra.run.smithi113.stdout: "dir": 8, 2024-03-18T22:59:08.783 INFO:teuthology.orchestra.run.smithi113.stdout: "dir+": 102, 2024-03-18T22:59:08.783 INFO:teuthology.orchestra.run.smithi113.stdout: "dir-": 94, 2024-03-18T22:59:08.783 INFO:teuthology.orchestra.run.smithi113.stdout: "dn": 84, 2024-03-18T22:59:08.783 INFO:teuthology.orchestra.run.smithi113.stdout: "dn+": 466, 2024-03-18T22:59:08.783 INFO:teuthology.orchestra.run.smithi113.stdout: "dn-": 382, 2024-03-18T22:59:08.783 INFO:teuthology.orchestra.run.smithi113.stdout: "cap": 0, 2024-03-18T22:59:08.783 INFO:teuthology.orchestra.run.smithi113.stdout: "cap+": 0, 2024-03-18T22:59:08.783 INFO:teuthology.orchestra.run.smithi113.stdout: "cap-": 0, 2024-03-18T22:59:08.783 INFO:teuthology.orchestra.run.smithi113.stdout: "rss": 55768, 2024-03-18T22:59:08.783 INFO:teuthology.orchestra.run.smithi113.stdout: "heap": 215420 2024-03-18T22:59:08.783 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.783 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_server": { 2024-03-18T22:59:08.783 INFO:teuthology.orchestra.run.smithi113.stdout: "dispatch_client_request": 0, 2024-03-18T22:59:08.783 INFO:teuthology.orchestra.run.smithi113.stdout: "dispatch_server_request": 0, 2024-03-18T22:59:08.784 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_request": 0, 2024-03-18T22:59:08.784 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_session": 0, 2024-03-18T22:59:08.784 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_peer_request": 0, 2024-03-18T22:59:08.784 INFO:teuthology.orchestra.run.smithi113.stdout: "req_create_latency": { 2024-03-18T22:59:08.784 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.784 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.784 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.784 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.784 INFO:teuthology.orchestra.run.smithi113.stdout: "req_getattr_latency": { 2024-03-18T22:59:08.784 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.784 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.784 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.784 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.784 INFO:teuthology.orchestra.run.smithi113.stdout: "req_getfilelock_latency": { 2024-03-18T22:59:08.784 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.785 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.785 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.785 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.785 INFO:teuthology.orchestra.run.smithi113.stdout: "req_link_latency": { 2024-03-18T22:59:08.785 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.785 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.785 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.785 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.785 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookup_latency": { 2024-03-18T22:59:08.785 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.785 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.785 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.786 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.786 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookuphash_latency": { 2024-03-18T22:59:08.786 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.786 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.786 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.786 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.786 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupino_latency": { 2024-03-18T22:59:08.786 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.786 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.786 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.786 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.786 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupname_latency": { 2024-03-18T22:59:08.786 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.787 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.787 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.787 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.787 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupparent_latency": { 2024-03-18T22:59:08.787 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.787 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.787 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.787 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.787 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupsnap_latency": { 2024-03-18T22:59:08.787 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.787 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.787 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.787 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.787 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lssnap_latency": { 2024-03-18T22:59:08.787 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.788 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.788 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.788 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.788 INFO:teuthology.orchestra.run.smithi113.stdout: "req_mkdir_latency": { 2024-03-18T22:59:08.788 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.789 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.789 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.789 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.789 INFO:teuthology.orchestra.run.smithi113.stdout: "req_mknod_latency": { 2024-03-18T22:59:08.789 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.789 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.789 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.789 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.789 INFO:teuthology.orchestra.run.smithi113.stdout: "req_mksnap_latency": { 2024-03-18T22:59:08.789 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.789 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.789 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.789 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.790 INFO:teuthology.orchestra.run.smithi113.stdout: "req_open_latency": { 2024-03-18T22:59:08.790 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.790 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.790 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.790 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.790 INFO:teuthology.orchestra.run.smithi113.stdout: "req_readdir_latency": { 2024-03-18T22:59:08.790 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.790 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.790 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.790 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.790 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rename_latency": { 2024-03-18T22:59:08.790 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.790 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.791 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.791 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.791 INFO:teuthology.orchestra.run.smithi113.stdout: "req_renamesnap_latency": { 2024-03-18T22:59:08.791 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.791 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.791 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.791 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.791 INFO:teuthology.orchestra.run.smithi113.stdout: "req_snapdiff_latency": { 2024-03-18T22:59:08.791 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.791 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.792 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.792 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.792 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rmdir_latency": { 2024-03-18T22:59:08.792 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.792 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.792 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.792 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.792 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rmsnap_latency": { 2024-03-18T22:59:08.792 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.792 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.792 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.792 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.792 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rmxattr_latency": { 2024-03-18T22:59:08.792 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setattr_latency": { 2024-03-18T22:59:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setdirlayout_latency": { 2024-03-18T22:59:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setfilelock_latency": { 2024-03-18T22:59:08.793 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setlayout_latency": { 2024-03-18T22:59:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setxattr_latency": { 2024-03-18T22:59:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.794 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.795 INFO:teuthology.orchestra.run.smithi113.stdout: "req_symlink_latency": { 2024-03-18T22:59:08.795 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.795 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.795 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.795 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.795 INFO:teuthology.orchestra.run.smithi113.stdout: "req_unlink_latency": { 2024-03-18T22:59:08.795 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.795 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.795 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.795 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.795 INFO:teuthology.orchestra.run.smithi113.stdout: "cap_revoke_eviction": 0, 2024-03-18T22:59:08.795 INFO:teuthology.orchestra.run.smithi113.stdout: "cap_acquisition_throttle": 0, 2024-03-18T22:59:08.795 INFO:teuthology.orchestra.run.smithi113.stdout: "req_getvxattr_latency": { 2024-03-18T22:59:08.795 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.795 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.796 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.796 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:08.796 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.796 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_sessions": { 2024-03-18T22:59:08.796 INFO:teuthology.orchestra.run.smithi113.stdout: "session_count": 1, 2024-03-18T22:59:08.796 INFO:teuthology.orchestra.run.smithi113.stdout: "session_add": 2, 2024-03-18T22:59:08.796 INFO:teuthology.orchestra.run.smithi113.stdout: "session_remove": 1, 2024-03-18T22:59:08.796 INFO:teuthology.orchestra.run.smithi113.stdout: "sessions_open": 1, 2024-03-18T22:59:08.796 INFO:teuthology.orchestra.run.smithi113.stdout: "sessions_stale": 0, 2024-03-18T22:59:08.796 INFO:teuthology.orchestra.run.smithi113.stdout: "total_load": 0, 2024-03-18T22:59:08.796 INFO:teuthology.orchestra.run.smithi113.stdout: "average_load": 0, 2024-03-18T22:59:08.796 INFO:teuthology.orchestra.run.smithi113.stdout: "avg_session_uptime": 4525, 2024-03-18T22:59:08.796 INFO:teuthology.orchestra.run.smithi113.stdout: "mdthresh_evicted": 0 2024-03-18T22:59:08.796 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.797 INFO:teuthology.orchestra.run.smithi113.stdout: "mempool": { 2024-03-18T22:59:08.797 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_bytes": 0, 2024-03-18T22:59:08.797 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_items": 0, 2024-03-18T22:59:08.797 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T22:59:08.797 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_items": 0, 2024-03-18T22:59:08.797 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T22:59:08.797 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_items": 0, 2024-03-18T22:59:08.797 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T22:59:08.797 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T22:59:08.797 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T22:59:08.797 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T22:59:08.797 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T22:59:08.797 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_items": 0, 2024-03-18T22:59:08.797 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T22:59:08.797 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T22:59:08.798 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_bytes": 0, 2024-03-18T22:59:08.798 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_items": 0, 2024-03-18T22:59:08.798 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_bytes": 0, 2024-03-18T22:59:08.798 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_items": 0, 2024-03-18T22:59:08.798 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T22:59:08.798 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T22:59:08.798 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T22:59:08.798 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T22:59:08.798 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T22:59:08.798 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_items": 0, 2024-03-18T22:59:08.798 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_bytes": 0, 2024-03-18T22:59:08.798 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_items": 0, 2024-03-18T22:59:08.798 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T22:59:08.799 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T22:59:08.799 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_bytes": 0, 2024-03-18T22:59:08.799 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_items": 0, 2024-03-18T22:59:08.799 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_bytes": 0, 2024-03-18T22:59:08.799 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_items": 0, 2024-03-18T22:59:08.799 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T22:59:08.799 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_items": 0, 2024-03-18T22:59:08.799 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T22:59:08.799 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_items": 0, 2024-03-18T22:59:08.799 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_bytes": 190207, 2024-03-18T22:59:08.799 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_items": 53, 2024-03-18T22:59:08.799 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_bytes": 0, 2024-03-18T22:59:08.799 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_items": 0, 2024-03-18T22:59:08.799 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_bytes": 0, 2024-03-18T22:59:08.799 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_items": 0, 2024-03-18T22:59:08.800 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_bytes": 0, 2024-03-18T22:59:08.800 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_items": 0, 2024-03-18T22:59:08.800 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_bytes": 0, 2024-03-18T22:59:08.800 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_items": 0, 2024-03-18T22:59:08.800 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_bytes": 9184, 2024-03-18T22:59:08.800 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_items": 177, 2024-03-18T22:59:08.800 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T22:59:08.800 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_items": 0, 2024-03-18T22:59:08.800 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_bytes": 0, 2024-03-18T22:59:08.800 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_items": 0, 2024-03-18T22:59:08.800 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_bytes": 81388, 2024-03-18T22:59:08.800 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_items": 429, 2024-03-18T22:59:08.800 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_bytes": 0, 2024-03-18T22:59:08.800 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_items": 0, 2024-03-18T22:59:08.800 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_bytes": 0, 2024-03-18T22:59:08.801 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_items": 0 2024-03-18T22:59:08.801 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.801 INFO:teuthology.orchestra.run.smithi113.stdout: "objecter": { 2024-03-18T22:59:08.801 INFO:teuthology.orchestra.run.smithi113.stdout: "op_active": 0, 2024-03-18T22:59:08.801 INFO:teuthology.orchestra.run.smithi113.stdout: "op_laggy": 0, 2024-03-18T22:59:08.801 INFO:teuthology.orchestra.run.smithi113.stdout: "op_send": 3736, 2024-03-18T22:59:08.801 INFO:teuthology.orchestra.run.smithi113.stdout: "op_send_bytes": 21, 2024-03-18T22:59:08.801 INFO:teuthology.orchestra.run.smithi113.stdout: "op_resend": 3, 2024-03-18T22:59:08.801 INFO:teuthology.orchestra.run.smithi113.stdout: "op_reply": 3736, 2024-03-18T22:59:08.801 INFO:teuthology.orchestra.run.smithi113.stdout: "op_latency": { 2024-03-18T22:59:08.801 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 3736, 2024-03-18T22:59:08.802 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 9.667813645, 2024-03-18T22:59:08.802 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.002587744 2024-03-18T22:59:08.802 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.802 INFO:teuthology.orchestra.run.smithi113.stdout: "op_inflight": 0, 2024-03-18T22:59:08.802 INFO:teuthology.orchestra.run.smithi113.stdout: "oplen_avg": { 2024-03-18T22:59:08.802 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 3736, 2024-03-18T22:59:08.802 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 3737 2024-03-18T22:59:08.802 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.802 INFO:teuthology.orchestra.run.smithi113.stdout: "op": 3736, 2024-03-18T22:59:08.802 INFO:teuthology.orchestra.run.smithi113.stdout: "op_r": 3736, 2024-03-18T22:59:08.802 INFO:teuthology.orchestra.run.smithi113.stdout: "op_w": 0, 2024-03-18T22:59:08.803 INFO:teuthology.orchestra.run.smithi113.stdout: "op_rmw": 0, 2024-03-18T22:59:08.803 INFO:teuthology.orchestra.run.smithi113.stdout: "op_pg": 0, 2024-03-18T22:59:08.803 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_stat": 2083, 2024-03-18T22:59:08.803 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_create": 0, 2024-03-18T22:59:08.803 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_read": 1651, 2024-03-18T22:59:08.803 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_write": 0, 2024-03-18T22:59:08.803 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_writefull": 0, 2024-03-18T22:59:08.803 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_writesame": 0, 2024-03-18T22:59:08.803 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_append": 0, 2024-03-18T22:59:08.803 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_zero": 0, 2024-03-18T22:59:08.803 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_truncate": 0, 2024-03-18T22:59:08.803 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_delete": 0, 2024-03-18T22:59:08.803 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_mapext": 0, 2024-03-18T22:59:08.803 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_sparse_read": 0, 2024-03-18T22:59:08.804 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_clonerange": 0, 2024-03-18T22:59:08.804 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_getxattr": 1, 2024-03-18T22:59:08.804 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_setxattr": 0, 2024-03-18T22:59:08.804 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_cmpxattr": 0, 2024-03-18T22:59:08.804 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_rmxattr": 0, 2024-03-18T22:59:08.804 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_resetxattrs": 0, 2024-03-18T22:59:08.804 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_call": 0, 2024-03-18T22:59:08.804 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_watch": 0, 2024-03-18T22:59:08.804 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_notify": 0, 2024-03-18T22:59:08.804 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T22:59:08.805 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_pgls": 0, 2024-03-18T22:59:08.805 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_pgls_filter": 0, 2024-03-18T22:59:08.805 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_other": 0, 2024-03-18T22:59:08.805 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_active": 0, 2024-03-18T22:59:08.805 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_send": 0, 2024-03-18T22:59:08.805 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_resend": 0, 2024-03-18T22:59:08.805 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_ping": 0, 2024-03-18T22:59:08.805 INFO:teuthology.orchestra.run.smithi113.stdout: "poolop_active": 0, 2024-03-18T22:59:08.805 INFO:teuthology.orchestra.run.smithi113.stdout: "poolop_send": 0, 2024-03-18T22:59:08.805 INFO:teuthology.orchestra.run.smithi113.stdout: "poolop_resend": 0, 2024-03-18T22:59:08.805 INFO:teuthology.orchestra.run.smithi113.stdout: "poolstat_active": 0, 2024-03-18T22:59:08.805 INFO:teuthology.orchestra.run.smithi113.stdout: "poolstat_send": 0, 2024-03-18T22:59:08.806 INFO:teuthology.orchestra.run.smithi113.stdout: "poolstat_resend": 0, 2024-03-18T22:59:08.806 INFO:teuthology.orchestra.run.smithi113.stdout: "statfs_active": 0, 2024-03-18T22:59:08.806 INFO:teuthology.orchestra.run.smithi113.stdout: "statfs_send": 0, 2024-03-18T22:59:08.806 INFO:teuthology.orchestra.run.smithi113.stdout: "statfs_resend": 0, 2024-03-18T22:59:08.806 INFO:teuthology.orchestra.run.smithi113.stdout: "command_active": 0, 2024-03-18T22:59:08.806 INFO:teuthology.orchestra.run.smithi113.stdout: "command_send": 0, 2024-03-18T22:59:08.806 INFO:teuthology.orchestra.run.smithi113.stdout: "command_resend": 0, 2024-03-18T22:59:08.806 INFO:teuthology.orchestra.run.smithi113.stdout: "map_epoch": 93, 2024-03-18T22:59:08.806 INFO:teuthology.orchestra.run.smithi113.stdout: "map_full": 0, 2024-03-18T22:59:08.806 INFO:teuthology.orchestra.run.smithi113.stdout: "map_inc": 4, 2024-03-18T22:59:08.807 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_sessions": 5, 2024-03-18T22:59:08.807 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_session_open": 6, 2024-03-18T22:59:08.807 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_session_close": 1, 2024-03-18T22:59:08.807 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_laggy": 0, 2024-03-18T22:59:08.807 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_wr": 0, 2024-03-18T22:59:08.807 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_rd": 2, 2024-03-18T22:59:08.807 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_del": 0 2024-03-18T22:59:08.807 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.807 INFO:teuthology.orchestra.run.smithi113.stdout: "oft": { 2024-03-18T22:59:08.807 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_objs": 0, 2024-03-18T22:59:08.807 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_kv_pairs": 0, 2024-03-18T22:59:08.808 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_updates": 0, 2024-03-18T22:59:08.808 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_removes": 0 2024-03-18T22:59:08.808 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.808 INFO:teuthology.orchestra.run.smithi113.stdout: "osd-slow-ops": { 2024-03-18T22:59:08.808 INFO:teuthology.orchestra.run.smithi113.stdout: "slow_ops_count": 0 2024-03-18T22:59:08.808 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.808 INFO:teuthology.orchestra.run.smithi113.stdout: "purge_queue": { 2024-03-18T22:59:08.808 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing_ops": 0, 2024-03-18T22:59:08.808 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T22:59:08.808 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing": 0, 2024-03-18T22:59:08.808 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing_high_water": 0, 2024-03-18T22:59:08.809 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executed_ops": 0, 2024-03-18T22:59:08.809 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executed": 0, 2024-03-18T22:59:08.809 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_item_in_journal": 0 2024-03-18T22:59:08.809 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.809 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T22:59:08.809 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:08.809 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 104857600, 2024-03-18T22:59:08.809 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:08.809 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 4080, 2024-03-18T22:59:08.809 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 4833656, 2024-03-18T22:59:08.809 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:08.810 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 4080, 2024-03-18T22:59:08.810 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-18T22:59:08.810 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-18T22:59:08.810 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 4080, 2024-03-18T22:59:08.810 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 4833656, 2024-03-18T22:59:08.810 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:08.810 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.810 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.810 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.811 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:08.811 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.811 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-objecter_bytes": { 2024-03-18T22:59:08.811 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:08.811 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 104857600, 2024-03-18T22:59:08.811 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:08.811 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 0, 2024-03-18T22:59:08.811 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 0, 2024-03-18T22:59:08.811 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:08.811 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 0, 2024-03-18T22:59:08.811 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 3736, 2024-03-18T22:59:08.812 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 3490069, 2024-03-18T22:59:08.812 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 3736, 2024-03-18T22:59:08.812 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 3490069, 2024-03-18T22:59:08.812 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:08.812 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.812 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.812 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.812 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:08.812 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.813 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-objecter_ops": { 2024-03-18T22:59:08.813 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:08.813 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 1024, 2024-03-18T22:59:08.813 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:08.813 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 0, 2024-03-18T22:59:08.813 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 0, 2024-03-18T22:59:08.813 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:08.813 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 0, 2024-03-18T22:59:08.813 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 3736, 2024-03-18T22:59:08.813 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 3736, 2024-03-18T22:59:08.813 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 3736, 2024-03-18T22:59:08.814 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 3736, 2024-03-18T22:59:08.814 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:08.814 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.814 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.814 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.814 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:08.814 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.814 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-write_buf_throttle": { 2024-03-18T22:59:08.814 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:08.814 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 3758096384, 2024-03-18T22:59:08.814 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:08.815 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 0, 2024-03-18T22:59:08.815 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 0, 2024-03-18T22:59:08.815 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:08.815 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 0, 2024-03-18T22:59:08.815 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-18T22:59:08.815 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-18T22:59:08.815 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 0, 2024-03-18T22:59:08.815 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 0, 2024-03-18T22:59:08.815 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:08.816 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.816 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.816 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.816 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:08.816 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:08.816 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-write_buf_throttle-0x55ebda8f46e0": { 2024-03-18T22:59:08.816 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:08.816 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 3758096384, 2024-03-18T22:59:08.816 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:08.816 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 0, 2024-03-18T22:59:08.816 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 0, 2024-03-18T22:59:08.817 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:08.817 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 0, 2024-03-18T22:59:08.817 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-18T22:59:08.817 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-18T22:59:08.817 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 0, 2024-03-18T22:59:08.817 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 0, 2024-03-18T22:59:08.817 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:08.817 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:08.817 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:08.817 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:08.817 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:08.817 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:08.817 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-18T22:59:08.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:08 smithi103 ceph-mon[66377]: pgmap v845: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T22:59:08.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:08 smithi040 ceph-mon[62643]: pgmap v845: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T22:59:09.585 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-18T22:59:09.586 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-18T22:59:09.586 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-18T22:59:09.586 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-18T22:59:09.586 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-18T22:59:09.586 DEBUG:tasks.check_counter:Getting stats from i 2024-03-18T22:59:09.586 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-18T22:59:09.922 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:59:10.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:10 smithi113 ceph-mon[66775]: pgmap v846: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:59:10.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:10 smithi103 ceph-mon[66377]: pgmap v846: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:59:10.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:10 smithi040 ceph-mon[62643]: pgmap v846: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:59:11.898 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-18T22:59:11.898 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T22:59:11.898 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 3688, 2024-03-18T22:59:11.899 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 5531, 2024-03-18T22:59:11.899 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 1223289, 2024-03-18T22:59:11.899 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 2379391, 2024-03-18T22:59:11.899 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 21, 2024-03-18T22:59:11.899 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 3, 2024-03-18T22:59:11.899 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.290707522, 2024-03-18T22:59:11.899 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.105245471, 2024-03-18T22:59:11.899 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.170776815, 2024-03-18T22:59:11.899 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.032629202, 2024-03-18T22:59:11.899 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:59:11.899 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 5529, 2024-03-18T22:59:11.899 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.333512584, 2024-03-18T22:59:11.900 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000060320 2024-03-18T22:59:11.900 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.900 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:59:11.900 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 6360, 2024-03-18T22:59:11.900 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.001237641, 2024-03-18T22:59:11.900 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000194 2024-03-18T22:59:11.900 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.900 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 928, 2024-03-18T22:59:11.900 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 736 2024-03-18T22:59:11.900 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.900 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T22:59:11.900 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 4577, 2024-03-18T22:59:11.900 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 4443, 2024-03-18T22:59:11.900 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 1776118, 2024-03-18T22:59:11.901 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 2346806, 2024-03-18T22:59:11.901 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 10, 2024-03-18T22:59:11.901 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 6, 2024-03-18T22:59:11.901 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.266202493, 2024-03-18T22:59:11.901 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.099982142, 2024-03-18T22:59:11.901 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.161981651, 2024-03-18T22:59:11.901 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.029895315, 2024-03-18T22:59:11.901 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:59:11.901 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 4441, 2024-03-18T22:59:11.901 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.243067873, 2024-03-18T22:59:11.901 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000054732 2024-03-18T22:59:11.901 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.901 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:59:11.901 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 6462, 2024-03-18T22:59:11.901 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.001063757, 2024-03-18T22:59:11.902 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000164 2024-03-18T22:59:11.902 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.902 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T22:59:11.902 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 479952 2024-03-18T22:59:11.902 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.902 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T22:59:11.902 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 4487, 2024-03-18T22:59:11.902 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 4565, 2024-03-18T22:59:11.902 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 1864427, 2024-03-18T22:59:11.902 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 5749194, 2024-03-18T22:59:11.902 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 48, 2024-03-18T22:59:11.902 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 5, 2024-03-18T22:59:11.902 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.339448293, 2024-03-18T22:59:11.903 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.113538985, 2024-03-18T22:59:11.903 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.114166273, 2024-03-18T22:59:11.903 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.101206176, 2024-03-18T22:59:11.903 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:59:11.903 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 4565, 2024-03-18T22:59:11.903 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.263579764, 2024-03-18T22:59:11.903 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000057739 2024-03-18T22:59:11.903 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.903 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:59:11.903 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 860, 2024-03-18T22:59:11.903 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000171714, 2024-03-18T22:59:11.903 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000199 2024-03-18T22:59:11.903 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.903 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 714560, 2024-03-18T22:59:11.903 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 214560 2024-03-18T22:59:11.904 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.904 INFO:teuthology.orchestra.run.smithi113.stdout: "cct": { 2024-03-18T22:59:11.904 INFO:teuthology.orchestra.run.smithi113.stdout: "total_workers": 1, 2024-03-18T22:59:11.904 INFO:teuthology.orchestra.run.smithi113.stdout: "unhealthy_workers": 0 2024-03-18T22:59:11.904 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.904 INFO:teuthology.orchestra.run.smithi113.stdout: "finisher-MDSRank": { 2024-03-18T22:59:11.904 INFO:teuthology.orchestra.run.smithi113.stdout: "queue_len": 0, 2024-03-18T22:59:11.904 INFO:teuthology.orchestra.run.smithi113.stdout: "complete_latency": { 2024-03-18T22:59:11.904 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 4378, 2024-03-18T22:59:11.904 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 1.113878174, 2024-03-18T22:59:11.904 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000254426 2024-03-18T22:59:11.904 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:11.904 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.904 INFO:teuthology.orchestra.run.smithi113.stdout: "finisher-PurgeQueue": { 2024-03-18T22:59:11.904 INFO:teuthology.orchestra.run.smithi113.stdout: "queue_len": 0, 2024-03-18T22:59:11.905 INFO:teuthology.orchestra.run.smithi113.stdout: "complete_latency": { 2024-03-18T22:59:11.905 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 1389, 2024-03-18T22:59:11.905 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.049906609, 2024-03-18T22:59:11.905 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000035929 2024-03-18T22:59:11.905 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:11.905 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.905 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-03-18T22:59:11.905 INFO:teuthology.orchestra.run.smithi113.stdout: "request": 533, 2024-03-18T22:59:11.905 INFO:teuthology.orchestra.run.smithi113.stdout: "reply": 454, 2024-03-18T22:59:11.905 INFO:teuthology.orchestra.run.smithi113.stdout: "reply_latency": { 2024-03-18T22:59:11.905 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 454, 2024-03-18T22:59:11.905 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 2.947374446, 2024-03-18T22:59:11.905 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.006492014 2024-03-18T22:59:11.905 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.905 INFO:teuthology.orchestra.run.smithi113.stdout: "slow_reply": 0, 2024-03-18T22:59:11.906 INFO:teuthology.orchestra.run.smithi113.stdout: "forward": 79, 2024-03-18T22:59:11.906 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_fetch_complete": 0, 2024-03-18T22:59:11.906 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_fetch_keys": 0, 2024-03-18T22:59:11.906 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_commit": 20, 2024-03-18T22:59:11.906 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_split": 1, 2024-03-18T22:59:11.906 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_merge": 0, 2024-03-18T22:59:11.906 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes": 42, 2024-03-18T22:59:11.906 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_top": 0, 2024-03-18T22:59:11.906 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_bottom": 31, 2024-03-18T22:59:11.906 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_pin_tail": 11, 2024-03-18T22:59:11.906 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_pinned": 14, 2024-03-18T22:59:11.906 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_expired": 432, 2024-03-18T22:59:11.906 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_with_caps": 2, 2024-03-18T22:59:11.906 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": 2, 2024-03-18T22:59:11.907 INFO:teuthology.orchestra.run.smithi113.stdout: "subtrees": 4, 2024-03-18T22:59:11.907 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse": 910, 2024-03-18T22:59:11.907 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_hit": 655, 2024-03-18T22:59:11.907 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_forward": 66, 2024-03-18T22:59:11.907 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_discover": 98, 2024-03-18T22:59:11.907 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_dir_fetch": 0, 2024-03-18T22:59:11.907 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_remote_ino": 0, 2024-03-18T22:59:11.907 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_lock": 0, 2024-03-18T22:59:11.907 INFO:teuthology.orchestra.run.smithi113.stdout: "load_cent": 0, 2024-03-18T22:59:11.907 INFO:teuthology.orchestra.run.smithi113.stdout: "q": 0, 2024-03-18T22:59:11.907 INFO:teuthology.orchestra.run.smithi113.stdout: "exported": 12, 2024-03-18T22:59:11.907 INFO:teuthology.orchestra.run.smithi113.stdout: "exported_inodes": 367, 2024-03-18T22:59:11.907 INFO:teuthology.orchestra.run.smithi113.stdout: "imported": 10, 2024-03-18T22:59:11.907 INFO:teuthology.orchestra.run.smithi113.stdout: "imported_inodes": 303, 2024-03-18T22:59:11.907 INFO:teuthology.orchestra.run.smithi113.stdout: "openino_dir_fetch": 0, 2024-03-18T22:59:11.908 INFO:teuthology.orchestra.run.smithi113.stdout: "openino_backtrace_fetch": 2828, 2024-03-18T22:59:11.908 INFO:teuthology.orchestra.run.smithi113.stdout: "openino_peer_discover": 0, 2024-03-18T22:59:11.908 INFO:teuthology.orchestra.run.smithi113.stdout: "root_rfiles": 49, 2024-03-18T22:59:11.908 INFO:teuthology.orchestra.run.smithi113.stdout: "root_rbytes": 729704, 2024-03-18T22:59:11.908 INFO:teuthology.orchestra.run.smithi113.stdout: "root_rsnaps": 0, 2024-03-18T22:59:11.908 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_backtrace_fetch": 2828, 2024-03-18T22:59:11.908 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_set_tag": 0, 2024-03-18T22:59:11.908 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T22:59:11.908 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_inotable_repaired": 0, 2024-03-18T22:59:11.908 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_dir_inodes": 766, 2024-03-18T22:59:11.908 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T22:59:11.908 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_dirfrag_rstats": 766, 2024-03-18T22:59:11.908 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_file_inodes": 2062, 2024-03-18T22:59:11.908 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_inode_file_caps": 6, 2024-03-18T22:59:11.908 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_revoke": 94, 2024-03-18T22:59:11.909 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_grant": 1195, 2024-03-18T22:59:11.909 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T22:59:11.909 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T22:59:11.909 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T22:59:11.909 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_caps": 1400, 2024-03-18T22:59:11.909 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_caps_dirty": 1253, 2024-03-18T22:59:11.909 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_cap_release": 3, 2024-03-18T22:59:11.909 INFO:teuthology.orchestra.run.smithi113.stdout: "process_request_cap_release": 181 2024-03-18T22:59:11.909 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.909 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_cache": { 2024-03-18T22:59:11.909 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_update": 44, 2024-03-18T22:59:11.909 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_update_receipt": 289, 2024-03-18T22:59:11.909 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_try_discover": 192, 2024-03-18T22:59:11.909 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_send_discover": 194, 2024-03-18T22:59:11.909 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_handle_discover": 89, 2024-03-18T22:59:11.910 INFO:teuthology.orchestra.run.smithi113.stdout: "num_strays": 1, 2024-03-18T22:59:11.910 INFO:teuthology.orchestra.run.smithi113.stdout: "num_strays_delayed": 0, 2024-03-18T22:59:11.910 INFO:teuthology.orchestra.run.smithi113.stdout: "num_strays_enqueuing": 0, 2024-03-18T22:59:11.910 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_created": 30, 2024-03-18T22:59:11.910 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_enqueued": 29, 2024-03-18T22:59:11.910 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_reintegrated": 0, 2024-03-18T22:59:11.910 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_migrated": 0, 2024-03-18T22:59:11.910 INFO:teuthology.orchestra.run.smithi113.stdout: "num_recovering_processing": 0, 2024-03-18T22:59:11.910 INFO:teuthology.orchestra.run.smithi113.stdout: "num_recovering_enqueued": 0, 2024-03-18T22:59:11.910 INFO:teuthology.orchestra.run.smithi113.stdout: "num_recovering_prioritized": 0, 2024-03-18T22:59:11.910 INFO:teuthology.orchestra.run.smithi113.stdout: "recovery_started": 0, 2024-03-18T22:59:11.910 INFO:teuthology.orchestra.run.smithi113.stdout: "recovery_completed": 0, 2024-03-18T22:59:11.910 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_quiesce_path": 0, 2024-03-18T22:59:11.910 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_quiesce_inode": 0, 2024-03-18T22:59:11.910 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T22:59:11.911 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_exportdir": 12, 2024-03-18T22:59:11.911 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_flush": 0, 2024-03-18T22:59:11.911 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_fragmentdir": 1, 2024-03-18T22:59:11.911 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_fragstats": 0, 2024-03-18T22:59:11.911 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_inodestats": 0 2024-03-18T22:59:11.911 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.911 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_log": { 2024-03-18T22:59:11.911 INFO:teuthology.orchestra.run.smithi113.stdout: "evlrg": 0, 2024-03-18T22:59:11.911 INFO:teuthology.orchestra.run.smithi113.stdout: "evadd": 1727, 2024-03-18T22:59:11.911 INFO:teuthology.orchestra.run.smithi113.stdout: "evex": 0, 2024-03-18T22:59:11.911 INFO:teuthology.orchestra.run.smithi113.stdout: "evtrm": 0, 2024-03-18T22:59:11.911 INFO:teuthology.orchestra.run.smithi113.stdout: "ev": 1727, 2024-03-18T22:59:11.911 INFO:teuthology.orchestra.run.smithi113.stdout: "evexg": 0, 2024-03-18T22:59:11.911 INFO:teuthology.orchestra.run.smithi113.stdout: "evexd": 0, 2024-03-18T22:59:11.911 INFO:teuthology.orchestra.run.smithi113.stdout: "segadd": 3, 2024-03-18T22:59:11.912 INFO:teuthology.orchestra.run.smithi113.stdout: "segex": 0, 2024-03-18T22:59:11.912 INFO:teuthology.orchestra.run.smithi113.stdout: "segtrm": 0, 2024-03-18T22:59:11.912 INFO:teuthology.orchestra.run.smithi113.stdout: "seg": 3, 2024-03-18T22:59:11.912 INFO:teuthology.orchestra.run.smithi113.stdout: "segmjr": 2, 2024-03-18T22:59:11.912 INFO:teuthology.orchestra.run.smithi113.stdout: "segexg": 0, 2024-03-18T22:59:11.912 INFO:teuthology.orchestra.run.smithi113.stdout: "segexd": 0, 2024-03-18T22:59:11.912 INFO:teuthology.orchestra.run.smithi113.stdout: "expos": 4194304, 2024-03-18T22:59:11.912 INFO:teuthology.orchestra.run.smithi113.stdout: "wrpos": 7684368, 2024-03-18T22:59:11.912 INFO:teuthology.orchestra.run.smithi113.stdout: "rdpos": 0, 2024-03-18T22:59:11.912 INFO:teuthology.orchestra.run.smithi113.stdout: "jlat": { 2024-03-18T22:59:11.912 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 1223, 2024-03-18T22:59:11.912 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 7.694606608, 2024-03-18T22:59:11.912 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.006291583 2024-03-18T22:59:11.912 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.912 INFO:teuthology.orchestra.run.smithi113.stdout: "replayed": 0 2024-03-18T22:59:11.913 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.913 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_mem": { 2024-03-18T22:59:11.913 INFO:teuthology.orchestra.run.smithi113.stdout: "ino": 23, 2024-03-18T22:59:11.913 INFO:teuthology.orchestra.run.smithi113.stdout: "ino+": 365, 2024-03-18T22:59:11.913 INFO:teuthology.orchestra.run.smithi113.stdout: "ino-": 342, 2024-03-18T22:59:11.913 INFO:teuthology.orchestra.run.smithi113.stdout: "dir": 26, 2024-03-18T22:59:11.913 INFO:teuthology.orchestra.run.smithi113.stdout: "dir+": 146, 2024-03-18T22:59:11.913 INFO:teuthology.orchestra.run.smithi113.stdout: "dir-": 120, 2024-03-18T22:59:11.913 INFO:teuthology.orchestra.run.smithi113.stdout: "dn": 42, 2024-03-18T22:59:11.913 INFO:teuthology.orchestra.run.smithi113.stdout: "dn+": 532, 2024-03-18T22:59:11.913 INFO:teuthology.orchestra.run.smithi113.stdout: "dn-": 490, 2024-03-18T22:59:11.913 INFO:teuthology.orchestra.run.smithi113.stdout: "cap": 2, 2024-03-18T22:59:11.913 INFO:teuthology.orchestra.run.smithi113.stdout: "cap+": 310, 2024-03-18T22:59:11.913 INFO:teuthology.orchestra.run.smithi113.stdout: "cap-": 308, 2024-03-18T22:59:11.913 INFO:teuthology.orchestra.run.smithi113.stdout: "rss": 58968, 2024-03-18T22:59:11.914 INFO:teuthology.orchestra.run.smithi113.stdout: "heap": 223612 2024-03-18T22:59:11.914 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.914 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_server": { 2024-03-18T22:59:11.914 INFO:teuthology.orchestra.run.smithi113.stdout: "dispatch_client_request": 586, 2024-03-18T22:59:11.914 INFO:teuthology.orchestra.run.smithi113.stdout: "dispatch_server_request": 139, 2024-03-18T22:59:11.914 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_request": 533, 2024-03-18T22:59:11.914 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_session": 27, 2024-03-18T22:59:11.914 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_peer_request": 295, 2024-03-18T22:59:11.914 INFO:teuthology.orchestra.run.smithi113.stdout: "req_create_latency": { 2024-03-18T22:59:11.914 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 118, 2024-03-18T22:59:11.914 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.158239833, 2024-03-18T22:59:11.914 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.001341015 2024-03-18T22:59:11.914 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.914 INFO:teuthology.orchestra.run.smithi113.stdout: "req_getattr_latency": { 2024-03-18T22:59:11.915 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 7, 2024-03-18T22:59:11.915 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.002411095, 2024-03-18T22:59:11.915 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000344442 2024-03-18T22:59:11.915 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.915 INFO:teuthology.orchestra.run.smithi113.stdout: "req_getfilelock_latency": { 2024-03-18T22:59:11.915 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.915 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.915 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.915 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.915 INFO:teuthology.orchestra.run.smithi113.stdout: "req_link_latency": { 2024-03-18T22:59:11.915 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.915 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.915 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.915 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.915 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookup_latency": { 2024-03-18T22:59:11.916 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 146, 2024-03-18T22:59:11.916 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 2.100965941, 2024-03-18T22:59:11.916 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.014390177 2024-03-18T22:59:11.916 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.916 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookuphash_latency": { 2024-03-18T22:59:11.916 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.916 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.916 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.916 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.916 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupino_latency": { 2024-03-18T22:59:11.916 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.917 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.917 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.917 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.917 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupname_latency": { 2024-03-18T22:59:11.917 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.917 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.917 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.917 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.917 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupparent_latency": { 2024-03-18T22:59:11.917 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.917 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.917 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.917 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.917 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupsnap_latency": { 2024-03-18T22:59:11.918 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.918 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.918 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.918 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.918 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lssnap_latency": { 2024-03-18T22:59:11.918 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.919 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.919 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.919 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.919 INFO:teuthology.orchestra.run.smithi113.stdout: "req_mkdir_latency": { 2024-03-18T22:59:11.919 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.919 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.919 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.919 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.919 INFO:teuthology.orchestra.run.smithi113.stdout: "req_mknod_latency": { 2024-03-18T22:59:11.919 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.919 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.919 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.919 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.919 INFO:teuthology.orchestra.run.smithi113.stdout: "req_mksnap_latency": { 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: "req_open_latency": { 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 150, 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.286049986, 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.001906999 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: "req_readdir_latency": { 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 3, 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.001684582, 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000561527 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rename_latency": { 2024-03-18T22:59:11.920 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: "req_renamesnap_latency": { 2024-03-18T22:59:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: "req_snapdiff_latency": { 2024-03-18T22:59:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rmdir_latency": { 2024-03-18T22:59:11.921 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 2, 2024-03-18T22:59:11.922 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.320196802, 2024-03-18T22:59:11.922 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.160098401 2024-03-18T22:59:11.922 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.922 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rmsnap_latency": { 2024-03-18T22:59:11.922 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.922 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.922 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.922 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.922 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rmxattr_latency": { 2024-03-18T22:59:11.922 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.922 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.922 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.922 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.922 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setattr_latency": { 2024-03-18T22:59:11.922 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.923 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.923 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.923 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.923 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setdirlayout_latency": { 2024-03-18T22:59:11.923 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.923 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.923 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.923 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.923 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setfilelock_latency": { 2024-03-18T22:59:11.923 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.923 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.923 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.923 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.923 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setlayout_latency": { 2024-03-18T22:59:11.924 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.924 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.924 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.924 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.924 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setxattr_latency": { 2024-03-18T22:59:11.924 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.924 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.924 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.924 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.924 INFO:teuthology.orchestra.run.smithi113.stdout: "req_symlink_latency": { 2024-03-18T22:59:11.924 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.924 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.924 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.924 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.924 INFO:teuthology.orchestra.run.smithi113.stdout: "req_unlink_latency": { 2024-03-18T22:59:11.925 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 28, 2024-03-18T22:59:11.925 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.077826207, 2024-03-18T22:59:11.925 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.002779507 2024-03-18T22:59:11.925 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.925 INFO:teuthology.orchestra.run.smithi113.stdout: "cap_revoke_eviction": 0, 2024-03-18T22:59:11.925 INFO:teuthology.orchestra.run.smithi113.stdout: "cap_acquisition_throttle": 0, 2024-03-18T22:59:11.925 INFO:teuthology.orchestra.run.smithi113.stdout: "req_getvxattr_latency": { 2024-03-18T22:59:11.925 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.925 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.925 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.925 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:11.925 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.925 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_sessions": { 2024-03-18T22:59:11.925 INFO:teuthology.orchestra.run.smithi113.stdout: "session_count": 1, 2024-03-18T22:59:11.925 INFO:teuthology.orchestra.run.smithi113.stdout: "session_add": 2, 2024-03-18T22:59:11.926 INFO:teuthology.orchestra.run.smithi113.stdout: "session_remove": 1, 2024-03-18T22:59:11.926 INFO:teuthology.orchestra.run.smithi113.stdout: "sessions_open": 1, 2024-03-18T22:59:11.926 INFO:teuthology.orchestra.run.smithi113.stdout: "sessions_stale": 0, 2024-03-18T22:59:11.926 INFO:teuthology.orchestra.run.smithi113.stdout: "total_load": 83, 2024-03-18T22:59:11.926 INFO:teuthology.orchestra.run.smithi113.stdout: "average_load": 83, 2024-03-18T22:59:11.926 INFO:teuthology.orchestra.run.smithi113.stdout: "avg_session_uptime": 968, 2024-03-18T22:59:11.926 INFO:teuthology.orchestra.run.smithi113.stdout: "mdthresh_evicted": 0 2024-03-18T22:59:11.926 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.926 INFO:teuthology.orchestra.run.smithi113.stdout: "mempool": { 2024-03-18T22:59:11.926 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_bytes": 0, 2024-03-18T22:59:11.926 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_items": 0, 2024-03-18T22:59:11.926 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T22:59:11.926 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_items": 0, 2024-03-18T22:59:11.926 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T22:59:11.927 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_items": 0, 2024-03-18T22:59:11.927 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T22:59:11.927 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T22:59:11.927 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T22:59:11.927 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T22:59:11.927 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T22:59:11.927 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_items": 0, 2024-03-18T22:59:11.927 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T22:59:11.927 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T22:59:11.927 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_bytes": 0, 2024-03-18T22:59:11.927 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_items": 0, 2024-03-18T22:59:11.927 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_bytes": 0, 2024-03-18T22:59:11.927 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_items": 0, 2024-03-18T22:59:11.927 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T22:59:11.927 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T22:59:11.928 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T22:59:11.928 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T22:59:11.928 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T22:59:11.928 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_items": 0, 2024-03-18T22:59:11.928 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_bytes": 0, 2024-03-18T22:59:11.928 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_items": 0, 2024-03-18T22:59:11.928 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T22:59:11.928 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T22:59:11.928 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_bytes": 0, 2024-03-18T22:59:11.928 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_items": 0, 2024-03-18T22:59:11.928 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_bytes": 0, 2024-03-18T22:59:11.928 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_items": 0, 2024-03-18T22:59:11.928 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T22:59:11.928 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_items": 0, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_items": 0, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_bytes": 314285, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_items": 116, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_bytes": 0, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_items": 0, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_bytes": 0, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_items": 0, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_bytes": 0, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_items": 0, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_bytes": 0, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_items": 0, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_bytes": 9184, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_items": 177, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T22:59:11.929 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_items": 0, 2024-03-18T22:59:11.930 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_bytes": 0, 2024-03-18T22:59:11.930 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_items": 0, 2024-03-18T22:59:11.930 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_bytes": 117137, 2024-03-18T22:59:11.930 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_items": 640, 2024-03-18T22:59:11.930 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_bytes": 0, 2024-03-18T22:59:11.930 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_items": 0, 2024-03-18T22:59:11.930 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_bytes": 0, 2024-03-18T22:59:11.930 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_items": 0 2024-03-18T22:59:11.930 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.930 INFO:teuthology.orchestra.run.smithi113.stdout: "objecter": { 2024-03-18T22:59:11.930 INFO:teuthology.orchestra.run.smithi113.stdout: "op_active": 0, 2024-03-18T22:59:11.930 INFO:teuthology.orchestra.run.smithi113.stdout: "op_laggy": 0, 2024-03-18T22:59:11.930 INFO:teuthology.orchestra.run.smithi113.stdout: "op_send": 5895, 2024-03-18T22:59:11.930 INFO:teuthology.orchestra.run.smithi113.stdout: "op_send_bytes": 3679726, 2024-03-18T22:59:11.930 INFO:teuthology.orchestra.run.smithi113.stdout: "op_resend": 33, 2024-03-18T22:59:11.931 INFO:teuthology.orchestra.run.smithi113.stdout: "op_reply": 5895, 2024-03-18T22:59:11.931 INFO:teuthology.orchestra.run.smithi113.stdout: "op_latency": { 2024-03-18T22:59:11.931 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 5895, 2024-03-18T22:59:11.931 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 276.946669065, 2024-03-18T22:59:11.931 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.046979926 2024-03-18T22:59:11.931 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.931 INFO:teuthology.orchestra.run.smithi113.stdout: "op_inflight": 0, 2024-03-18T22:59:11.931 INFO:teuthology.orchestra.run.smithi113.stdout: "oplen_avg": { 2024-03-18T22:59:11.931 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 5895, 2024-03-18T22:59:11.931 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 6279 2024-03-18T22:59:11.931 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.931 INFO:teuthology.orchestra.run.smithi113.stdout: "op": 5895, 2024-03-18T22:59:11.931 INFO:teuthology.orchestra.run.smithi113.stdout: "op_r": 2830, 2024-03-18T22:59:11.931 INFO:teuthology.orchestra.run.smithi113.stdout: "op_w": 3065, 2024-03-18T22:59:11.931 INFO:teuthology.orchestra.run.smithi113.stdout: "op_rmw": 0, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "op_pg": 0, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_stat": 0, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_create": 10, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_read": 2, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_write": 1225, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_writefull": 61, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_writesame": 0, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_append": 0, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_zero": 0, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_truncate": 1, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_delete": 1417, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_mapext": 0, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_sparse_read": 0, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_clonerange": 0, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_getxattr": 2828, 2024-03-18T22:59:11.932 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_setxattr": 20, 2024-03-18T22:59:11.933 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_cmpxattr": 0, 2024-03-18T22:59:11.933 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_rmxattr": 0, 2024-03-18T22:59:11.933 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_resetxattrs": 0, 2024-03-18T22:59:11.933 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_call": 0, 2024-03-18T22:59:11.933 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_watch": 0, 2024-03-18T22:59:11.933 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_notify": 0, 2024-03-18T22:59:11.933 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T22:59:11.933 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_pgls": 0, 2024-03-18T22:59:11.933 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_pgls_filter": 0, 2024-03-18T22:59:11.933 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_other": 0, 2024-03-18T22:59:11.933 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_active": 0, 2024-03-18T22:59:11.933 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_send": 0, 2024-03-18T22:59:11.933 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_resend": 0, 2024-03-18T22:59:11.933 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_ping": 0, 2024-03-18T22:59:11.933 INFO:teuthology.orchestra.run.smithi113.stdout: "poolop_active": 0, 2024-03-18T22:59:11.934 INFO:teuthology.orchestra.run.smithi113.stdout: "poolop_send": 0, 2024-03-18T22:59:11.934 INFO:teuthology.orchestra.run.smithi113.stdout: "poolop_resend": 0, 2024-03-18T22:59:11.934 INFO:teuthology.orchestra.run.smithi113.stdout: "poolstat_active": 0, 2024-03-18T22:59:11.934 INFO:teuthology.orchestra.run.smithi113.stdout: "poolstat_send": 0, 2024-03-18T22:59:11.934 INFO:teuthology.orchestra.run.smithi113.stdout: "poolstat_resend": 0, 2024-03-18T22:59:11.934 INFO:teuthology.orchestra.run.smithi113.stdout: "statfs_active": 0, 2024-03-18T22:59:11.934 INFO:teuthology.orchestra.run.smithi113.stdout: "statfs_send": 0, 2024-03-18T22:59:11.934 INFO:teuthology.orchestra.run.smithi113.stdout: "statfs_resend": 0, 2024-03-18T22:59:11.934 INFO:teuthology.orchestra.run.smithi113.stdout: "command_active": 0, 2024-03-18T22:59:11.934 INFO:teuthology.orchestra.run.smithi113.stdout: "command_send": 0, 2024-03-18T22:59:11.934 INFO:teuthology.orchestra.run.smithi113.stdout: "command_resend": 0, 2024-03-18T22:59:11.934 INFO:teuthology.orchestra.run.smithi113.stdout: "map_epoch": 93, 2024-03-18T22:59:11.934 INFO:teuthology.orchestra.run.smithi113.stdout: "map_full": 0, 2024-03-18T22:59:11.935 INFO:teuthology.orchestra.run.smithi113.stdout: "map_inc": 4, 2024-03-18T22:59:11.935 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_sessions": 12, 2024-03-18T22:59:11.935 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_session_open": 12, 2024-03-18T22:59:11.935 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_session_close": 0, 2024-03-18T22:59:11.935 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_laggy": 0, 2024-03-18T22:59:11.935 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_wr": 553, 2024-03-18T22:59:11.935 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_rd": 0, 2024-03-18T22:59:11.935 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_del": 162 2024-03-18T22:59:11.935 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.935 INFO:teuthology.orchestra.run.smithi113.stdout: "oft": { 2024-03-18T22:59:11.935 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_objs": 1, 2024-03-18T22:59:11.935 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_kv_pairs": 0, 2024-03-18T22:59:11.935 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_updates": 0, 2024-03-18T22:59:11.935 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_removes": 0 2024-03-18T22:59:11.935 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: "osd-slow-ops": { 2024-03-18T22:59:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: "slow_ops_count": 0 2024-03-18T22:59:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: "purge_queue": { 2024-03-18T22:59:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing_ops": 0, 2024-03-18T22:59:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing_ops_high_water": 53, 2024-03-18T22:59:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing": 0, 2024-03-18T22:59:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing_high_water": 2, 2024-03-18T22:59:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executed_ops": 1405, 2024-03-18T22:59:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executed": 29, 2024-03-18T22:59:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_item_in_journal": 0 2024-03-18T22:59:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T22:59:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:11.936 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 104857600, 2024-03-18T22:59:11.937 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:11.937 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 12748, 2024-03-18T22:59:11.937 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 3692766, 2024-03-18T22:59:11.937 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:11.937 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 12748, 2024-03-18T22:59:11.937 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-18T22:59:11.937 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-18T22:59:11.937 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 12748, 2024-03-18T22:59:11.937 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 3692766, 2024-03-18T22:59:11.937 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:11.937 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.937 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.937 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.937 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:11.937 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.938 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-objecter_bytes": { 2024-03-18T22:59:11.938 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:11.938 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 104857600, 2024-03-18T22:59:11.938 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:11.938 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 0, 2024-03-18T22:59:11.938 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 0, 2024-03-18T22:59:11.938 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:11.938 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 0, 2024-03-18T22:59:11.938 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 5895, 2024-03-18T22:59:11.938 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 3682879, 2024-03-18T22:59:11.938 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 5895, 2024-03-18T22:59:11.938 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 3682879, 2024-03-18T22:59:11.938 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:11.938 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-objecter_ops": { 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 1024, 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 0, 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 0, 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 0, 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 5895, 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 5895, 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 5895, 2024-03-18T22:59:11.939 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 5895, 2024-03-18T22:59:11.940 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:11.940 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.940 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.940 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.940 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:11.940 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.940 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-write_buf_throttle": { 2024-03-18T22:59:11.940 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:11.940 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 3758096384, 2024-03-18T22:59:11.940 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:11.940 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 29, 2024-03-18T22:59:11.940 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 3153, 2024-03-18T22:59:11.940 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:11.940 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 29, 2024-03-18T22:59:11.940 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 2, 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 3153, 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-write_buf_throttle-0x5612f5a3a500": { 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 3758096384, 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 1727, 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 3490064, 2024-03-18T22:59:11.941 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:11.942 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 1727, 2024-03-18T22:59:11.942 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-18T22:59:11.942 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-18T22:59:11.942 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 1223, 2024-03-18T22:59:11.942 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 3490064, 2024-03-18T22:59:11.942 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:11.942 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:11.942 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:11.942 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:11.942 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:11.942 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:11.942 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-18T22:59:12.692 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:12 smithi113 ceph-mon[66775]: pgmap v847: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-18T22:59:12.717 INFO:tasks.check_counter:Daemon mds.i mds.exported=12 2024-03-18T22:59:12.717 INFO:tasks.check_counter:Daemon mds.i mds.imported=10 2024-03-18T22:59:12.717 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=44 2024-03-18T22:59:12.717 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=289 2024-03-18T22:59:12.717 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=1 2024-03-18T22:59:12.718 DEBUG:tasks.check_counter:Getting stats from l 2024-03-18T22:59:12.718 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 410ef414-e577-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-18T22:59:12.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:12 smithi103 ceph-mon[66377]: pgmap v847: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-18T22:59:12.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:12 smithi040 ceph-mon[62643]: pgmap v847: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-18T22:59:13.029 INFO:teuthology.orchestra.run.smithi113.stderr:Inferring config /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T22:59:14.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:14 smithi113 ceph-mon[66775]: pgmap v848: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:59:14.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:14 smithi103 ceph-mon[66377]: pgmap v848: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:59:14.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:14 smithi040 ceph-mon[62643]: pgmap v848: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:59:14.946 INFO:teuthology.orchestra.run.smithi113.stdout:{ 2024-03-18T22:59:14.946 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T22:59:14.946 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 4019, 2024-03-18T22:59:14.946 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 2735, 2024-03-18T22:59:14.946 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 1479520, 2024-03-18T22:59:14.946 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 1199403, 2024-03-18T22:59:14.946 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 11, 2024-03-18T22:59:14.946 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 4, 2024-03-18T22:59:14.946 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.208365279, 2024-03-18T22:59:14.946 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.070142723, 2024-03-18T22:59:14.946 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.157484606, 2024-03-18T22:59:14.947 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.018950421, 2024-03-18T22:59:14.947 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:59:14.947 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 2733, 2024-03-18T22:59:14.947 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.184099099, 2024-03-18T22:59:14.947 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000067361 2024-03-18T22:59:14.947 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.947 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:59:14.947 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 5848, 2024-03-18T22:59:14.947 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000929832, 2024-03-18T22:59:14.947 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000159 2024-03-18T22:59:14.947 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.947 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T22:59:14.947 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T22:59:14.947 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.948 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T22:59:14.948 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 1207, 2024-03-18T22:59:14.948 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 1270, 2024-03-18T22:59:14.948 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 1007550, 2024-03-18T22:59:14.948 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 756656, 2024-03-18T22:59:14.948 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 20, 2024-03-18T22:59:14.948 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 5, 2024-03-18T22:59:14.948 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.111094060, 2024-03-18T22:59:14.948 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.034825182, 2024-03-18T22:59:14.948 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.064270763, 2024-03-18T22:59:14.948 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.018535664, 2024-03-18T22:59:14.948 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:59:14.948 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 1269, 2024-03-18T22:59:14.948 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.246562199, 2024-03-18T22:59:14.948 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000194296 2024-03-18T22:59:14.949 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.949 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:59:14.949 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 892, 2024-03-18T22:59:14.949 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000235383, 2024-03-18T22:59:14.949 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000263 2024-03-18T22:59:14.949 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.949 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 687536, 2024-03-18T22:59:14.949 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 199856 2024-03-18T22:59:14.949 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.949 INFO:teuthology.orchestra.run.smithi113.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T22:59:14.949 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_messages": 1028, 2024-03-18T22:59:14.949 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages": 1153, 2024-03-18T22:59:14.949 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_bytes": 453804, 2024-03-18T22:59:14.949 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_bytes": 1391586, 2024-03-18T22:59:14.949 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_created_connections": 49, 2024-03-18T22:59:14.950 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_active_connections": 6, 2024-03-18T22:59:14.950 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_total_time": 0.101081558, 2024-03-18T22:59:14.950 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_send_time": 0.037866570, 2024-03-18T22:59:14.950 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_recv_time": 0.047886112, 2024-03-18T22:59:14.950 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_running_fast_dispatch_time": 0.017314494, 2024-03-18T22:59:14.950 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T22:59:14.950 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 1152, 2024-03-18T22:59:14.950 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.122412275, 2024-03-18T22:59:14.950 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000106260 2024-03-18T22:59:14.950 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.950 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_handle_ack_lat": { 2024-03-18T22:59:14.950 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 856, 2024-03-18T22:59:14.950 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000158777, 2024-03-18T22:59:14.950 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000185 2024-03-18T22:59:14.951 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.951 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_recv_encrypted_bytes": 4080, 2024-03-18T22:59:14.951 INFO:teuthology.orchestra.run.smithi113.stdout: "msgr_send_encrypted_bytes": 568736 2024-03-18T22:59:14.951 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.951 INFO:teuthology.orchestra.run.smithi113.stdout: "cct": { 2024-03-18T22:59:14.951 INFO:teuthology.orchestra.run.smithi113.stdout: "total_workers": 1, 2024-03-18T22:59:14.951 INFO:teuthology.orchestra.run.smithi113.stdout: "unhealthy_workers": 0 2024-03-18T22:59:14.951 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.951 INFO:teuthology.orchestra.run.smithi113.stdout: "finisher-MDSRank": { 2024-03-18T22:59:14.951 INFO:teuthology.orchestra.run.smithi113.stdout: "queue_len": 0, 2024-03-18T22:59:14.951 INFO:teuthology.orchestra.run.smithi113.stdout: "complete_latency": { 2024-03-18T22:59:14.951 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 365, 2024-03-18T22:59:14.951 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.072175414, 2024-03-18T22:59:14.951 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000197740 2024-03-18T22:59:14.952 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:14.952 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.952 INFO:teuthology.orchestra.run.smithi113.stdout: "finisher-PurgeQueue": { 2024-03-18T22:59:14.952 INFO:teuthology.orchestra.run.smithi113.stdout: "queue_len": 0, 2024-03-18T22:59:14.952 INFO:teuthology.orchestra.run.smithi113.stdout: "complete_latency": { 2024-03-18T22:59:14.952 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 935, 2024-03-18T22:59:14.952 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.033885162, 2024-03-18T22:59:14.952 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000036240 2024-03-18T22:59:14.952 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:14.952 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.952 INFO:teuthology.orchestra.run.smithi113.stdout: "mds": { 2024-03-18T22:59:14.952 INFO:teuthology.orchestra.run.smithi113.stdout: "request": 177, 2024-03-18T22:59:14.952 INFO:teuthology.orchestra.run.smithi113.stdout: "reply": 132, 2024-03-18T22:59:14.952 INFO:teuthology.orchestra.run.smithi113.stdout: "reply_latency": { 2024-03-18T22:59:14.953 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 132, 2024-03-18T22:59:14.953 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.219216802, 2024-03-18T22:59:14.953 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.001660733 2024-03-18T22:59:14.953 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.953 INFO:teuthology.orchestra.run.smithi113.stdout: "slow_reply": 0, 2024-03-18T22:59:14.953 INFO:teuthology.orchestra.run.smithi113.stdout: "forward": 45, 2024-03-18T22:59:14.953 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_fetch_complete": 0, 2024-03-18T22:59:14.953 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_fetch_keys": 0, 2024-03-18T22:59:14.953 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_commit": 11, 2024-03-18T22:59:14.953 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_split": 0, 2024-03-18T22:59:14.953 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_merge": 0, 2024-03-18T22:59:14.953 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes": 21, 2024-03-18T22:59:14.953 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_top": 4, 2024-03-18T22:59:14.953 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_bottom": 17, 2024-03-18T22:59:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_pin_tail": 0, 2024-03-18T22:59:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_pinned": 13, 2024-03-18T22:59:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_expired": 171, 2024-03-18T22:59:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: "inodes_with_caps": 0, 2024-03-18T22:59:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: "caps": 0, 2024-03-18T22:59:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: "subtrees": 4, 2024-03-18T22:59:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse": 442, 2024-03-18T22:59:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_hit": 290, 2024-03-18T22:59:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_forward": 45, 2024-03-18T22:59:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_discover": 86, 2024-03-18T22:59:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_dir_fetch": 0, 2024-03-18T22:59:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_remote_ino": 0, 2024-03-18T22:59:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: "traverse_lock": 0, 2024-03-18T22:59:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: "load_cent": 0, 2024-03-18T22:59:14.954 INFO:teuthology.orchestra.run.smithi113.stdout: "q": 0, 2024-03-18T22:59:14.955 INFO:teuthology.orchestra.run.smithi113.stdout: "exported": 1, 2024-03-18T22:59:14.955 INFO:teuthology.orchestra.run.smithi113.stdout: "exported_inodes": 0, 2024-03-18T22:59:14.955 INFO:teuthology.orchestra.run.smithi113.stdout: "imported": 1, 2024-03-18T22:59:14.955 INFO:teuthology.orchestra.run.smithi113.stdout: "imported_inodes": 18, 2024-03-18T22:59:14.955 INFO:teuthology.orchestra.run.smithi113.stdout: "openino_dir_fetch": 0, 2024-03-18T22:59:14.955 INFO:teuthology.orchestra.run.smithi113.stdout: "openino_backtrace_fetch": 72, 2024-03-18T22:59:14.955 INFO:teuthology.orchestra.run.smithi113.stdout: "openino_peer_discover": 0, 2024-03-18T22:59:14.955 INFO:teuthology.orchestra.run.smithi113.stdout: "root_rfiles": 49, 2024-03-18T22:59:14.955 INFO:teuthology.orchestra.run.smithi113.stdout: "root_rbytes": 729704, 2024-03-18T22:59:14.955 INFO:teuthology.orchestra.run.smithi113.stdout: "root_rsnaps": 0, 2024-03-18T22:59:14.955 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_backtrace_fetch": 72, 2024-03-18T22:59:14.955 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_set_tag": 0, 2024-03-18T22:59:14.955 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T22:59:14.955 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_inotable_repaired": 0, 2024-03-18T22:59:14.955 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_dir_inodes": 0, 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: "scrub_file_inodes": 72, 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_inode_file_caps": 0, 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_revoke": 57, 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_grant": 72, 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_caps": 179, 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_caps_dirty": 103, 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_cap_release": 1, 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: "process_request_cap_release": 36 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_cache": { 2024-03-18T22:59:14.956 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_update": 1, 2024-03-18T22:59:14.957 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_update_receipt": 309, 2024-03-18T22:59:14.957 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_try_discover": 196, 2024-03-18T22:59:14.957 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_send_discover": 225, 2024-03-18T22:59:14.957 INFO:teuthology.orchestra.run.smithi113.stdout: "dir_handle_discover": 1, 2024-03-18T22:59:14.957 INFO:teuthology.orchestra.run.smithi113.stdout: "num_strays": 0, 2024-03-18T22:59:14.957 INFO:teuthology.orchestra.run.smithi113.stdout: "num_strays_delayed": 0, 2024-03-18T22:59:14.957 INFO:teuthology.orchestra.run.smithi113.stdout: "num_strays_enqueuing": 0, 2024-03-18T22:59:14.957 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_created": 18, 2024-03-18T22:59:14.957 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_enqueued": 18, 2024-03-18T22:59:14.957 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_reintegrated": 0, 2024-03-18T22:59:14.957 INFO:teuthology.orchestra.run.smithi113.stdout: "strays_migrated": 0, 2024-03-18T22:59:14.957 INFO:teuthology.orchestra.run.smithi113.stdout: "num_recovering_processing": 0, 2024-03-18T22:59:14.957 INFO:teuthology.orchestra.run.smithi113.stdout: "num_recovering_enqueued": 0, 2024-03-18T22:59:14.957 INFO:teuthology.orchestra.run.smithi113.stdout: "num_recovering_prioritized": 0, 2024-03-18T22:59:14.957 INFO:teuthology.orchestra.run.smithi113.stdout: "recovery_started": 0, 2024-03-18T22:59:14.958 INFO:teuthology.orchestra.run.smithi113.stdout: "recovery_completed": 0, 2024-03-18T22:59:14.958 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_quiesce_path": 0, 2024-03-18T22:59:14.958 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_quiesce_inode": 0, 2024-03-18T22:59:14.958 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T22:59:14.958 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_exportdir": 1, 2024-03-18T22:59:14.958 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_flush": 0, 2024-03-18T22:59:14.958 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_fragmentdir": 0, 2024-03-18T22:59:14.958 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_fragstats": 0, 2024-03-18T22:59:14.958 INFO:teuthology.orchestra.run.smithi113.stdout: "ireq_inodestats": 0 2024-03-18T22:59:14.958 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.958 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_log": { 2024-03-18T22:59:14.958 INFO:teuthology.orchestra.run.smithi113.stdout: "evlrg": 0, 2024-03-18T22:59:14.958 INFO:teuthology.orchestra.run.smithi113.stdout: "evadd": 225, 2024-03-18T22:59:14.958 INFO:teuthology.orchestra.run.smithi113.stdout: "evex": 0, 2024-03-18T22:59:14.959 INFO:teuthology.orchestra.run.smithi113.stdout: "evtrm": 0, 2024-03-18T22:59:14.959 INFO:teuthology.orchestra.run.smithi113.stdout: "ev": 225, 2024-03-18T22:59:14.959 INFO:teuthology.orchestra.run.smithi113.stdout: "evexg": 0, 2024-03-18T22:59:14.959 INFO:teuthology.orchestra.run.smithi113.stdout: "evexd": 0, 2024-03-18T22:59:14.959 INFO:teuthology.orchestra.run.smithi113.stdout: "segadd": 2, 2024-03-18T22:59:14.959 INFO:teuthology.orchestra.run.smithi113.stdout: "segex": 0, 2024-03-18T22:59:14.959 INFO:teuthology.orchestra.run.smithi113.stdout: "segtrm": 0, 2024-03-18T22:59:14.959 INFO:teuthology.orchestra.run.smithi113.stdout: "seg": 2, 2024-03-18T22:59:14.959 INFO:teuthology.orchestra.run.smithi113.stdout: "segmjr": 2, 2024-03-18T22:59:14.959 INFO:teuthology.orchestra.run.smithi113.stdout: "segexg": 0, 2024-03-18T22:59:14.959 INFO:teuthology.orchestra.run.smithi113.stdout: "segexd": 0, 2024-03-18T22:59:14.959 INFO:teuthology.orchestra.run.smithi113.stdout: "expos": 4194304, 2024-03-18T22:59:14.959 INFO:teuthology.orchestra.run.smithi113.stdout: "wrpos": 4467424, 2024-03-18T22:59:14.959 INFO:teuthology.orchestra.run.smithi113.stdout: "rdpos": 0, 2024-03-18T22:59:14.960 INFO:teuthology.orchestra.run.smithi113.stdout: "jlat": { 2024-03-18T22:59:14.960 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 98, 2024-03-18T22:59:14.960 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 3.833139822, 2024-03-18T22:59:14.960 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.039113671 2024-03-18T22:59:14.960 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.960 INFO:teuthology.orchestra.run.smithi113.stdout: "replayed": 0 2024-03-18T22:59:14.960 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.960 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_mem": { 2024-03-18T22:59:14.960 INFO:teuthology.orchestra.run.smithi113.stdout: "ino": 26, 2024-03-18T22:59:14.960 INFO:teuthology.orchestra.run.smithi113.stdout: "ino+": 130, 2024-03-18T22:59:14.960 INFO:teuthology.orchestra.run.smithi113.stdout: "ino-": 104, 2024-03-18T22:59:14.960 INFO:teuthology.orchestra.run.smithi113.stdout: "dir": 25, 2024-03-18T22:59:14.960 INFO:teuthology.orchestra.run.smithi113.stdout: "dir+": 144, 2024-03-18T22:59:14.960 INFO:teuthology.orchestra.run.smithi113.stdout: "dir-": 119, 2024-03-18T22:59:14.960 INFO:teuthology.orchestra.run.smithi113.stdout: "dn": 21, 2024-03-18T22:59:14.961 INFO:teuthology.orchestra.run.smithi113.stdout: "dn+": 228, 2024-03-18T22:59:14.961 INFO:teuthology.orchestra.run.smithi113.stdout: "dn-": 207, 2024-03-18T22:59:14.961 INFO:teuthology.orchestra.run.smithi113.stdout: "cap": 0, 2024-03-18T22:59:14.961 INFO:teuthology.orchestra.run.smithi113.stdout: "cap+": 19, 2024-03-18T22:59:14.961 INFO:teuthology.orchestra.run.smithi113.stdout: "cap-": 19, 2024-03-18T22:59:14.961 INFO:teuthology.orchestra.run.smithi113.stdout: "rss": 57144, 2024-03-18T22:59:14.961 INFO:teuthology.orchestra.run.smithi113.stdout: "heap": 223612 2024-03-18T22:59:14.961 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.961 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_server": { 2024-03-18T22:59:14.961 INFO:teuthology.orchestra.run.smithi113.stdout: "dispatch_client_request": 206, 2024-03-18T22:59:14.961 INFO:teuthology.orchestra.run.smithi113.stdout: "dispatch_server_request": 2, 2024-03-18T22:59:14.961 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_request": 177, 2024-03-18T22:59:14.961 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_client_session": 15, 2024-03-18T22:59:14.961 INFO:teuthology.orchestra.run.smithi113.stdout: "handle_peer_request": 23, 2024-03-18T22:59:14.962 INFO:teuthology.orchestra.run.smithi113.stdout: "req_create_latency": { 2024-03-18T22:59:14.962 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.962 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.962 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.962 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.962 INFO:teuthology.orchestra.run.smithi113.stdout: "req_getattr_latency": { 2024-03-18T22:59:14.962 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.962 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.962 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.962 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.962 INFO:teuthology.orchestra.run.smithi113.stdout: "req_getfilelock_latency": { 2024-03-18T22:59:14.962 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.962 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.962 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.963 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.963 INFO:teuthology.orchestra.run.smithi113.stdout: "req_link_latency": { 2024-03-18T22:59:14.963 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.963 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.963 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.963 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.963 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookup_latency": { 2024-03-18T22:59:14.963 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 44, 2024-03-18T22:59:14.963 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.041759982, 2024-03-18T22:59:14.963 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000949090 2024-03-18T22:59:14.963 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.963 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookuphash_latency": { 2024-03-18T22:59:14.963 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.963 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.964 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.964 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.964 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupino_latency": { 2024-03-18T22:59:14.964 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.964 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.964 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.964 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.964 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupname_latency": { 2024-03-18T22:59:14.964 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.964 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.964 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.964 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.964 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupparent_latency": { 2024-03-18T22:59:14.964 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.964 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.965 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.965 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.965 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lookupsnap_latency": { 2024-03-18T22:59:14.965 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.965 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.965 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.965 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.965 INFO:teuthology.orchestra.run.smithi113.stdout: "req_lssnap_latency": { 2024-03-18T22:59:14.965 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.965 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.965 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.966 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.966 INFO:teuthology.orchestra.run.smithi113.stdout: "req_mkdir_latency": { 2024-03-18T22:59:14.966 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.966 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.966 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.966 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.966 INFO:teuthology.orchestra.run.smithi113.stdout: "req_mknod_latency": { 2024-03-18T22:59:14.966 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.966 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.967 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.967 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.967 INFO:teuthology.orchestra.run.smithi113.stdout: "req_mksnap_latency": { 2024-03-18T22:59:14.967 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.967 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.967 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.967 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.967 INFO:teuthology.orchestra.run.smithi113.stdout: "req_open_latency": { 2024-03-18T22:59:14.967 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 68, 2024-03-18T22:59:14.967 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.045529326, 2024-03-18T22:59:14.967 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000669548 2024-03-18T22:59:14.967 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.967 INFO:teuthology.orchestra.run.smithi113.stdout: "req_readdir_latency": { 2024-03-18T22:59:14.967 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 2, 2024-03-18T22:59:14.968 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.001029949, 2024-03-18T22:59:14.968 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000514974 2024-03-18T22:59:14.968 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.968 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rename_latency": { 2024-03-18T22:59:14.968 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.968 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.968 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.968 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.968 INFO:teuthology.orchestra.run.smithi113.stdout: "req_renamesnap_latency": { 2024-03-18T22:59:14.968 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.968 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.968 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.968 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.968 INFO:teuthology.orchestra.run.smithi113.stdout: "req_snapdiff_latency": { 2024-03-18T22:59:14.968 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.969 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.969 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.969 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.969 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rmdir_latency": { 2024-03-18T22:59:14.969 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.969 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.969 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.969 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.969 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rmsnap_latency": { 2024-03-18T22:59:14.969 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.969 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.969 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.969 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.969 INFO:teuthology.orchestra.run.smithi113.stdout: "req_rmxattr_latency": { 2024-03-18T22:59:14.969 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.970 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.970 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.970 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.970 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setattr_latency": { 2024-03-18T22:59:14.970 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.970 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.970 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.970 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.970 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setdirlayout_latency": { 2024-03-18T22:59:14.970 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.970 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.970 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.970 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.970 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setfilelock_latency": { 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setlayout_latency": { 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: "req_setxattr_latency": { 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: "req_symlink_latency": { 2024-03-18T22:59:14.971 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: "req_unlink_latency": { 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 18, 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.130897545, 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.007272085 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: "cap_revoke_eviction": 0, 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: "cap_acquisition_throttle": 0, 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: "req_getvxattr_latency": { 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:14.972 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.973 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_sessions": { 2024-03-18T22:59:14.973 INFO:teuthology.orchestra.run.smithi113.stdout: "session_count": 1, 2024-03-18T22:59:14.973 INFO:teuthology.orchestra.run.smithi113.stdout: "session_add": 2, 2024-03-18T22:59:14.973 INFO:teuthology.orchestra.run.smithi113.stdout: "session_remove": 1, 2024-03-18T22:59:14.973 INFO:teuthology.orchestra.run.smithi113.stdout: "sessions_open": 1, 2024-03-18T22:59:14.973 INFO:teuthology.orchestra.run.smithi113.stdout: "sessions_stale": 0, 2024-03-18T22:59:14.973 INFO:teuthology.orchestra.run.smithi113.stdout: "total_load": 48, 2024-03-18T22:59:14.973 INFO:teuthology.orchestra.run.smithi113.stdout: "average_load": 48, 2024-03-18T22:59:14.973 INFO:teuthology.orchestra.run.smithi113.stdout: "avg_session_uptime": 970, 2024-03-18T22:59:14.973 INFO:teuthology.orchestra.run.smithi113.stdout: "mdthresh_evicted": 0 2024-03-18T22:59:14.973 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.973 INFO:teuthology.orchestra.run.smithi113.stdout: "mempool": { 2024-03-18T22:59:14.973 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_bytes": 0, 2024-03-18T22:59:14.973 INFO:teuthology.orchestra.run.smithi113.stdout: "bloom_filter_items": 0, 2024-03-18T22:59:14.974 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T22:59:14.974 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_alloc_items": 0, 2024-03-18T22:59:14.974 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T22:59:14.974 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_data_items": 0, 2024-03-18T22:59:14.974 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T22:59:14.974 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T22:59:14.974 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T22:59:14.974 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T22:59:14.974 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T22:59:14.974 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_other_items": 0, 2024-03-18T22:59:14.974 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T22:59:14.974 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T22:59:14.974 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_bytes": 0, 2024-03-18T22:59:14.974 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_extent_items": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_bytes": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_blob_items": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_fsck_items": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_bytes": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_txc_items": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_bytes": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluestore_writing_items": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_bytes": 0, 2024-03-18T22:59:14.975 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_items": 0, 2024-03-18T22:59:14.976 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T22:59:14.976 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_reader_items": 0, 2024-03-18T22:59:14.976 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T22:59:14.976 INFO:teuthology.orchestra.run.smithi113.stdout: "bluefs_file_writer_items": 0, 2024-03-18T22:59:14.976 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_bytes": 314321, 2024-03-18T22:59:14.976 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_anon_items": 127, 2024-03-18T22:59:14.976 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_bytes": 0, 2024-03-18T22:59:14.976 INFO:teuthology.orchestra.run.smithi113.stdout: "buffer_meta_items": 0, 2024-03-18T22:59:14.976 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_bytes": 0, 2024-03-18T22:59:14.976 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_items": 0, 2024-03-18T22:59:14.976 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_bytes": 0, 2024-03-18T22:59:14.976 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_mapbl_items": 0, 2024-03-18T22:59:14.976 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_bytes": 0, 2024-03-18T22:59:14.976 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_pglog_items": 0, 2024-03-18T22:59:14.977 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_bytes": 9184, 2024-03-18T22:59:14.977 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_items": 177, 2024-03-18T22:59:14.977 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T22:59:14.977 INFO:teuthology.orchestra.run.smithi113.stdout: "osdmap_mapping_items": 0, 2024-03-18T22:59:14.977 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_bytes": 0, 2024-03-18T22:59:14.977 INFO:teuthology.orchestra.run.smithi113.stdout: "pgmap_items": 0, 2024-03-18T22:59:14.977 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_bytes": 105245, 2024-03-18T22:59:14.977 INFO:teuthology.orchestra.run.smithi113.stdout: "mds_co_items": 408, 2024-03-18T22:59:14.977 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_bytes": 0, 2024-03-18T22:59:14.977 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_1_items": 0, 2024-03-18T22:59:14.977 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_bytes": 0, 2024-03-18T22:59:14.977 INFO:teuthology.orchestra.run.smithi113.stdout: "unittest_2_items": 0 2024-03-18T22:59:14.977 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.977 INFO:teuthology.orchestra.run.smithi113.stdout: "objecter": { 2024-03-18T22:59:14.977 INFO:teuthology.orchestra.run.smithi113.stdout: "op_active": 0, 2024-03-18T22:59:14.978 INFO:teuthology.orchestra.run.smithi113.stdout: "op_laggy": 0, 2024-03-18T22:59:14.978 INFO:teuthology.orchestra.run.smithi113.stdout: "op_send": 1216, 2024-03-18T22:59:14.978 INFO:teuthology.orchestra.run.smithi113.stdout: "op_send_bytes": 295127, 2024-03-18T22:59:14.978 INFO:teuthology.orchestra.run.smithi113.stdout: "op_resend": 33, 2024-03-18T22:59:14.978 INFO:teuthology.orchestra.run.smithi113.stdout: "op_reply": 1216, 2024-03-18T22:59:14.978 INFO:teuthology.orchestra.run.smithi113.stdout: "op_latency": { 2024-03-18T22:59:14.978 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 1216, 2024-03-18T22:59:14.978 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 140.060300439, 2024-03-18T22:59:14.978 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.115181168 2024-03-18T22:59:14.978 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.978 INFO:teuthology.orchestra.run.smithi113.stdout: "op_inflight": 0, 2024-03-18T22:59:14.978 INFO:teuthology.orchestra.run.smithi113.stdout: "oplen_avg": { 2024-03-18T22:59:14.978 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 1216, 2024-03-18T22:59:14.978 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 1289 2024-03-18T22:59:14.979 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.979 INFO:teuthology.orchestra.run.smithi113.stdout: "op": 1216, 2024-03-18T22:59:14.979 INFO:teuthology.orchestra.run.smithi113.stdout: "op_r": 73, 2024-03-18T22:59:14.979 INFO:teuthology.orchestra.run.smithi113.stdout: "op_w": 1143, 2024-03-18T22:59:14.979 INFO:teuthology.orchestra.run.smithi113.stdout: "op_rmw": 0, 2024-03-18T22:59:14.979 INFO:teuthology.orchestra.run.smithi113.stdout: "op_pg": 0, 2024-03-18T22:59:14.979 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_stat": 0, 2024-03-18T22:59:14.979 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_create": 10, 2024-03-18T22:59:14.979 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_read": 1, 2024-03-18T22:59:14.979 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_write": 99, 2024-03-18T22:59:14.979 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_writefull": 29, 2024-03-18T22:59:14.979 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_writesame": 0, 2024-03-18T22:59:14.979 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_append": 0, 2024-03-18T22:59:14.979 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_zero": 0, 2024-03-18T22:59:14.979 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_truncate": 0, 2024-03-18T22:59:14.980 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_delete": 948, 2024-03-18T22:59:14.980 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_mapext": 0, 2024-03-18T22:59:14.980 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_sparse_read": 0, 2024-03-18T22:59:14.980 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_clonerange": 0, 2024-03-18T22:59:14.980 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_getxattr": 72, 2024-03-18T22:59:14.980 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_setxattr": 20, 2024-03-18T22:59:14.980 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_cmpxattr": 0, 2024-03-18T22:59:14.980 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_rmxattr": 0, 2024-03-18T22:59:14.980 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_resetxattrs": 0, 2024-03-18T22:59:14.980 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_call": 0, 2024-03-18T22:59:14.980 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_watch": 0, 2024-03-18T22:59:14.980 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_notify": 0, 2024-03-18T22:59:14.980 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T22:59:14.980 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_pgls": 0, 2024-03-18T22:59:14.980 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_pgls_filter": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "osdop_other": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_active": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_send": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_resend": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "linger_ping": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "poolop_active": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "poolop_send": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "poolop_resend": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "poolstat_active": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "poolstat_send": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "poolstat_resend": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "statfs_active": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "statfs_send": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "statfs_resend": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "command_active": 0, 2024-03-18T22:59:14.981 INFO:teuthology.orchestra.run.smithi113.stdout: "command_send": 0, 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: "command_resend": 0, 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: "map_epoch": 93, 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: "map_full": 0, 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: "map_inc": 4, 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_sessions": 12, 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_session_open": 12, 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_session_close": 0, 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: "osd_laggy": 0, 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_wr": 90, 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_rd": 0, 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_del": 20 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: "oft": { 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_objs": 1, 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_kv_pairs": 0, 2024-03-18T22:59:14.982 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_updates": 0, 2024-03-18T22:59:14.983 INFO:teuthology.orchestra.run.smithi113.stdout: "omap_total_removes": 0 2024-03-18T22:59:14.983 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.983 INFO:teuthology.orchestra.run.smithi113.stdout: "osd-slow-ops": { 2024-03-18T22:59:14.983 INFO:teuthology.orchestra.run.smithi113.stdout: "slow_ops_count": 0 2024-03-18T22:59:14.983 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.983 INFO:teuthology.orchestra.run.smithi113.stdout: "purge_queue": { 2024-03-18T22:59:14.983 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing_ops": 0, 2024-03-18T22:59:14.983 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing_ops_high_water": 52, 2024-03-18T22:59:14.983 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing": 0, 2024-03-18T22:59:14.983 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executing_high_water": 1, 2024-03-18T22:59:14.983 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executed_ops": 936, 2024-03-18T22:59:14.983 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_executed": 18, 2024-03-18T22:59:14.983 INFO:teuthology.orchestra.run.smithi113.stdout: "pq_item_in_journal": 0 2024-03-18T22:59:14.983 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.984 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T22:59:14.984 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:14.984 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 104857600, 2024-03-18T22:59:14.984 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:14.984 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 6250, 2024-03-18T22:59:14.984 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 2355848, 2024-03-18T22:59:14.984 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:14.984 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 6250, 2024-03-18T22:59:14.984 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-18T22:59:14.984 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-18T22:59:14.984 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 6250, 2024-03-18T22:59:14.984 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 2355848, 2024-03-18T22:59:14.984 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:14.984 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.984 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.985 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.985 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:14.985 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.985 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-objecter_bytes": { 2024-03-18T22:59:14.985 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:14.985 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 104857600, 2024-03-18T22:59:14.985 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:14.985 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 0, 2024-03-18T22:59:14.985 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 0, 2024-03-18T22:59:14.985 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:14.985 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 0, 2024-03-18T22:59:14.985 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 1216, 2024-03-18T22:59:14.985 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 297089, 2024-03-18T22:59:14.985 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 1216, 2024-03-18T22:59:14.985 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 297089, 2024-03-18T22:59:14.986 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:14.986 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.986 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.986 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.986 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:14.986 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.986 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-objecter_ops": { 2024-03-18T22:59:14.986 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:14.986 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 1024, 2024-03-18T22:59:14.986 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:14.986 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 0, 2024-03-18T22:59:14.986 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 0, 2024-03-18T22:59:14.986 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:14.986 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 0, 2024-03-18T22:59:14.987 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 1216, 2024-03-18T22:59:14.987 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 1216, 2024-03-18T22:59:14.987 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 1216, 2024-03-18T22:59:14.987 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 1216, 2024-03-18T22:59:14.987 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:14.987 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.987 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.987 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.987 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:14.987 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.987 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-write_buf_throttle": { 2024-03-18T22:59:14.987 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:14.987 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 3758096384, 2024-03-18T22:59:14.987 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:14.987 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 18, 2024-03-18T22:59:14.988 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 1962, 2024-03-18T22:59:14.988 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:14.988 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 18, 2024-03-18T22:59:14.988 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-18T22:59:14.988 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-18T22:59:14.988 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 1, 2024-03-18T22:59:14.988 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 1962, 2024-03-18T22:59:14.988 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:14.988 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.988 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.988 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.988 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:14.988 INFO:teuthology.orchestra.run.smithi113.stdout: }, 2024-03-18T22:59:14.988 INFO:teuthology.orchestra.run.smithi113.stdout: "throttle-write_buf_throttle-0x5651ffa12500": { 2024-03-18T22:59:14.988 INFO:teuthology.orchestra.run.smithi113.stdout: "val": 0, 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: "max": 3758096384, 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: "get_started": 0, 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: "get": 225, 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: "get_sum": 273120, 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_fail": 0, 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: "get_or_fail_success": 225, 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: "take": 0, 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: "take_sum": 0, 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: "put": 98, 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: "put_sum": 273120, 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: "wait": { 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: "avgcount": 0, 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: "sum": 0.000000000, 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: "avgtime": 0.000000000 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout: } 2024-03-18T22:59:14.989 INFO:teuthology.orchestra.run.smithi113.stdout:} 2024-03-18T22:59:15.793 INFO:tasks.check_counter:Daemon mds.l mds.exported=1 2024-03-18T22:59:15.793 INFO:tasks.check_counter:Daemon mds.l mds.imported=1 2024-03-18T22:59:15.793 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=1 2024-03-18T22:59:15.793 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=309 2024-03-18T22:59:15.794 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-03-18T22:59:15.794 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-18T22:59:15.804 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-18T22:59:15.804 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:59:15.804 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T22:59:15.962 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:59:15.962 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T22:59:16.021 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:59:16.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.177+0000 7f2805897640 1 -- 172.21.15.40:0/3558641499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2800102340 msgr2=0x7f2800104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:16.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.177+0000 7f2805897640 1 --2- 172.21.15.40:0/3558641499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2800102340 0x7f2800104730 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f27e8009900 tx=0x7f27e802bc30 comp rx=0 tx=0).stop 2024-03-18T22:59:16.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.177+0000 7f2805897640 1 -- 172.21.15.40:0/3558641499 shutdown_connections 2024-03-18T22:59:16.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.177+0000 7f2805897640 1 --2- 172.21.15.40:0/3558641499 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f28001075a0 0x7f28001099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.177+0000 7f2805897640 1 --2- 172.21.15.40:0/3558641499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2800104c70 0x7f2800107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.177+0000 7f2805897640 1 --2- 172.21.15.40:0/3558641499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2800102340 0x7f2800104730 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.177+0000 7f2805897640 1 -- 172.21.15.40:0/3558641499 >> 172.21.15.40:0/3558641499 conn(0x7f28000fbd50 msgr2=0x7f28000fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:16.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.177+0000 7f2805897640 1 -- 172.21.15.40:0/3558641499 shutdown_connections 2024-03-18T22:59:16.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.177+0000 7f2805897640 1 -- 172.21.15.40:0/3558641499 wait complete. 2024-03-18T22:59:16.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.177+0000 7f2805897640 1 Processor -- start 2024-03-18T22:59:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f2805897640 1 -- start start 2024-03-18T22:59:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f2805897640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2800102340 0x7f2800197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f2805897640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2800104c70 0x7f2800197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f2805897640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28001075a0 0x7f280019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f2805897640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f280010d3a0 con 0x7f2800102340 2024-03-18T22:59:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f2805897640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f280010d220 con 0x7f28001075a0 2024-03-18T22:59:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f2805897640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f280010d520 con 0x7f2800104c70 2024-03-18T22:59:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f27feffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2800102340 0x7f2800197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f27feffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2800102340 0x7f2800197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49824/0 (socket says 172.21.15.40:49824) 2024-03-18T22:59:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f27feffd640 1 -- 172.21.15.40:0/250938774 learned_addr learned my addr 172.21.15.40:0/250938774 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f27fe7fc640 1 --2- 172.21.15.40:0/250938774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2800104c70 0x7f2800197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f27ff7fe640 1 --2- 172.21.15.40:0/250938774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28001075a0 0x7f280019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f27feffd640 1 -- 172.21.15.40:0/250938774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2800104c70 msgr2=0x7f2800197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f27feffd640 1 --2- 172.21.15.40:0/250938774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2800104c70 0x7f2800197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f27feffd640 1 -- 172.21.15.40:0/250938774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28001075a0 msgr2=0x7f280019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f27feffd640 1 --2- 172.21.15.40:0/250938774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28001075a0 0x7f280019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f27feffd640 1 -- 172.21.15.40:0/250938774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f280019f6d0 con 0x7f2800102340 2024-03-18T22:59:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f27feffd640 1 --2- 172.21.15.40:0/250938774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2800102340 0x7f2800197ab0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f27e8006010 tx=0x7f27e8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.178+0000 7f2804895640 1 -- 172.21.15.40:0/250938774 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e8043020 con 0x7f2800102340 2024-03-18T22:59:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.179+0000 7f2805897640 1 -- 172.21.15.40:0/250938774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f280019f960 con 0x7f2800102340 2024-03-18T22:59:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.179+0000 7f2804895640 1 -- 172.21.15.40:0/250938774 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27e80043d0 con 0x7f2800102340 2024-03-18T22:59:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.179+0000 7f2805897640 1 -- 172.21.15.40:0/250938774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f280019ff00 con 0x7f2800102340 2024-03-18T22:59:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.179+0000 7f2804895640 1 -- 172.21.15.40:0/250938774 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e8003b20 con 0x7f2800102340 2024-03-18T22:59:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.179+0000 7f2805897640 1 -- 172.21.15.40:0/250938774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2800068a60 con 0x7f2800102340 2024-03-18T22:59:16.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.181+0000 7f2804895640 1 -- 172.21.15.40:0/250938774 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f27e8046020 con 0x7f2800102340 2024-03-18T22:59:16.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.181+0000 7f2804895640 1 --2- 172.21.15.40:0/250938774 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f27ec0a2800 0x7f27ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:16.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.181+0000 7f27fe7fc640 1 --2- 172.21.15.40:0/250938774 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f27ec0a2800 0x7f27ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:16.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.181+0000 7f2804895640 1 -- 172.21.15.40:0/250938774 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f27e80f1650 con 0x7f2800102340 2024-03-18T22:59:16.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.182+0000 7f27fe7fc640 1 --2- 172.21.15.40:0/250938774 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f27ec0a2800 0x7f27ec0a4cc0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2800198fc0 tx=0x7f27f0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:16.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.185+0000 7f2804895640 1 -- 172.21.15.40:0/250938774 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f27e804d020 con 0x7f2800102340 2024-03-18T22:59:16.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.370+0000 7f2805897640 1 -- 172.21.15.40:0/250938774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28000fd840 con 0x7f2800102340 2024-03-18T22:59:16.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.372+0000 7f2804895640 1 -- 172.21.15.40:0/250938774 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24908 (secure 0 0 0) 0x7f28000fd840 con 0x7f2800102340 2024-03-18T22:59:16.373 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:59:16.373 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":93,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:42:44.941873+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:59:16.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.374+0000 7f2805897640 1 -- 172.21.15.40:0/250938774 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f27ec0a2800 msgr2=0x7f27ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:16.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.374+0000 7f2805897640 1 --2- 172.21.15.40:0/250938774 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f27ec0a2800 0x7f27ec0a4cc0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2800198fc0 tx=0x7f27f0005f90 comp rx=0 tx=0).stop 2024-03-18T22:59:16.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.374+0000 7f2805897640 1 -- 172.21.15.40:0/250938774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2800102340 msgr2=0x7f2800197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:16.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.374+0000 7f2805897640 1 --2- 172.21.15.40:0/250938774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2800102340 0x7f2800197ab0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f27e8006010 tx=0x7f27e8005e40 comp rx=0 tx=0).stop 2024-03-18T22:59:16.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.374+0000 7f2805897640 1 -- 172.21.15.40:0/250938774 shutdown_connections 2024-03-18T22:59:16.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.374+0000 7f2805897640 1 --2- 172.21.15.40:0/250938774 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f27ec0a2800 0x7f27ec0a4cc0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.374+0000 7f2805897640 1 --2- 172.21.15.40:0/250938774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28001075a0 0x7f280019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.374+0000 7f2805897640 1 --2- 172.21.15.40:0/250938774 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f2800104c70 0x7f2800197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.374+0000 7f2805897640 1 --2- 172.21.15.40:0/250938774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2800102340 0x7f2800197ab0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.374+0000 7f2805897640 1 -- 172.21.15.40:0/250938774 >> 172.21.15.40:0/250938774 conn(0x7f28000fbd50 msgr2=0x7f2800102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:16.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.374+0000 7f2805897640 1 -- 172.21.15.40:0/250938774 shutdown_connections 2024-03-18T22:59:16.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.374+0000 7f2805897640 1 -- 172.21.15.40:0/250938774 wait complete. 2024-03-18T22:59:16.386 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T22:36:07.060691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T22:40:39.925147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T22:40:40.773757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T22:40:44.068183+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T22:59:16.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T22:59:16.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.491+0000 7fd0296ea640 1 -- 172.21.15.40:0/4132884948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd024101830 msgr2=0x7fd024101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.491+0000 7fd0296ea640 1 --2- 172.21.15.40:0/4132884948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd024101830 0x7fd024101c50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd01c009860 tx=0x7fd01c02bb70 comp rx=0 tx=0).stop 2024-03-18T22:59:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.491+0000 7fd0296ea640 1 -- 172.21.15.40:0/4132884948 shutdown_connections 2024-03-18T22:59:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.491+0000 7fd0296ea640 1 --2- 172.21.15.40:0/4132884948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd0241033e0 0x7fd024109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.491+0000 7fd0296ea640 1 --2- 172.21.15.40:0/4132884948 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd024102a20 0x7fd024102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.491+0000 7fd0296ea640 1 --2- 172.21.15.40:0/4132884948 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd024101830 0x7fd024101c50 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.491+0000 7fd0296ea640 1 -- 172.21.15.40:0/4132884948 >> 172.21.15.40:0/4132884948 conn(0x7fd0240fcdc0 msgr2=0x7fd0240ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.491+0000 7fd0296ea640 1 -- 172.21.15.40:0/4132884948 shutdown_connections 2024-03-18T22:59:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.491+0000 7fd0296ea640 1 -- 172.21.15.40:0/4132884948 wait complete. 2024-03-18T22:59:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.492+0000 7fd0296ea640 1 Processor -- start 2024-03-18T22:59:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.492+0000 7fd0296ea640 1 -- start start 2024-03-18T22:59:16.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.492+0000 7fd0296ea640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd024101830 0x7fd024197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.492+0000 7fd0296ea640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd024102a20 0x7fd024197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.492+0000 7fd0296ea640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0241033e0 0x7fd02419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.492+0000 7fd0296ea640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fd02410d330 con 0x7fd024102a20 2024-03-18T22:59:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.492+0000 7fd0296ea640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd02410d1b0 con 0x7fd024101830 2024-03-18T22:59:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.492+0000 7fd0296ea640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fd02410d4b0 con 0x7fd0241033e0 2024-03-18T22:59:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.492+0000 7fd0227fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd024102a20 0x7fd024197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.492+0000 7fd0227fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd024102a20 0x7fd024197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49842/0 (socket says 172.21.15.40:49842) 2024-03-18T22:59:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.492+0000 7fd022ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd024101830 0x7fd024197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.492+0000 7fd0227fc640 1 -- 172.21.15.40:0/1481752839 learned_addr learned my addr 172.21.15.40:0/1481752839 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.492+0000 7fd0237fe640 1 --2- 172.21.15.40:0/1481752839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0241033e0 0x7fd02419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.493+0000 7fd0227fc640 1 -- 172.21.15.40:0/1481752839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0241033e0 msgr2=0x7fd02419f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.493+0000 7fd0227fc640 1 --2- 172.21.15.40:0/1481752839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0241033e0 0x7fd02419f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.493+0000 7fd0227fc640 1 -- 172.21.15.40:0/1481752839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd024101830 msgr2=0x7fd024197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.493+0000 7fd0227fc640 1 --2- 172.21.15.40:0/1481752839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd024101830 0x7fd024197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.493+0000 7fd0227fc640 1 -- 172.21.15.40:0/1481752839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd02419f6d0 con 0x7fd024102a20 2024-03-18T22:59:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.493+0000 7fd022ffd640 1 --2- 172.21.15.40:0/1481752839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd024101830 0x7fd024197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:59:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.493+0000 7fd0237fe640 1 --2- 172.21.15.40:0/1481752839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0241033e0 0x7fd02419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:59:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.493+0000 7fd0227fc640 1 --2- 172.21.15.40:0/1481752839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd024102a20 0x7fd024197ff0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fd00c00e9c0 tx=0x7fd00c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.493+0000 7fd01bfff640 1 -- 172.21.15.40:0/1481752839 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c00cde0 con 0x7fd024102a20 2024-03-18T22:59:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.493+0000 7fd01bfff640 1 -- 172.21.15.40:0/1481752839 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd00c0035f0 con 0x7fd024102a20 2024-03-18T22:59:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.493+0000 7fd0296ea640 1 -- 172.21.15.40:0/1481752839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd02419f9c0 con 0x7fd024102a20 2024-03-18T22:59:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.493+0000 7fd01bfff640 1 -- 172.21.15.40:0/1481752839 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c010690 con 0x7fd024102a20 2024-03-18T22:59:16.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.493+0000 7fd0296ea640 1 -- 172.21.15.40:0/1481752839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd02419ff10 con 0x7fd024102a20 2024-03-18T22:59:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.494+0000 7fd0296ea640 1 -- 172.21.15.40:0/1481752839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd024108c70 con 0x7fd024102a20 2024-03-18T22:59:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.495+0000 7fd01bfff640 1 -- 172.21.15.40:0/1481752839 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd00c0037a0 con 0x7fd024102a20 2024-03-18T22:59:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.496+0000 7fd01bfff640 1 --2- 172.21.15.40:0/1481752839 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0100a2800 0x7fd0100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.496+0000 7fd022ffd640 1 --2- 172.21.15.40:0/1481752839 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0100a2800 0x7fd0100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.496+0000 7fd01bfff640 1 -- 172.21.15.40:0/1481752839 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd00c0cf190 con 0x7fd024102a20 2024-03-18T22:59:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.496+0000 7fd022ffd640 1 --2- 172.21.15.40:0/1481752839 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0100a2800 0x7fd0100a4cc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd01c02c180 tx=0x7fd01c0349a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:16.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.501+0000 7fd01bfff640 1 -- 172.21.15.40:0/1481752839 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd00c009510 con 0x7fd024102a20 2024-03-18T22:59:16.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:16 smithi040 ceph-mon[62643]: pgmap v849: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:59:16.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:16 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/250938774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:59:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.682+0000 7fd0296ea640 1 -- 172.21.15.40:0/1481752839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd024062420 con 0x7fd024102a20 2024-03-18T22:59:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.683+0000 7fd01bfff640 1 -- 172.21.15.40:0/1481752839 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fd00c09bde0 con 0x7fd024102a20 2024-03-18T22:59:16.683 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-18T22:59:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.685+0000 7fd0296ea640 1 -- 172.21.15.40:0/1481752839 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0100a2800 msgr2=0x7fd0100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.685+0000 7fd0296ea640 1 --2- 172.21.15.40:0/1481752839 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0100a2800 0x7fd0100a4cc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd01c02c180 tx=0x7fd01c0349a0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.685+0000 7fd0296ea640 1 -- 172.21.15.40:0/1481752839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd024102a20 msgr2=0x7fd024197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.685+0000 7fd0296ea640 1 --2- 172.21.15.40:0/1481752839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd024102a20 0x7fd024197ff0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fd00c00e9c0 tx=0x7fd00c00ee80 comp rx=0 tx=0).stop 2024-03-18T22:59:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.685+0000 7fd0296ea640 1 -- 172.21.15.40:0/1481752839 shutdown_connections 2024-03-18T22:59:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.685+0000 7fd0296ea640 1 --2- 172.21.15.40:0/1481752839 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fd0100a2800 0x7fd0100a4cc0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.685+0000 7fd0296ea640 1 --2- 172.21.15.40:0/1481752839 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fd0241033e0 0x7fd02419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.685+0000 7fd0296ea640 1 --2- 172.21.15.40:0/1481752839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd024102a20 0x7fd024197ff0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.685+0000 7fd0296ea640 1 --2- 172.21.15.40:0/1481752839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd024101830 0x7fd024197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.685+0000 7fd0296ea640 1 -- 172.21.15.40:0/1481752839 >> 172.21.15.40:0/1481752839 conn(0x7fd0240fcdc0 msgr2=0x7fd024105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.685+0000 7fd0296ea640 1 -- 172.21.15.40:0/1481752839 shutdown_connections 2024-03-18T22:59:16.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.685+0000 7fd0296ea640 1 -- 172.21.15.40:0/1481752839 wait complete. 2024-03-18T22:59:16.696 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T22:59:16.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.801+0000 7f076b91d640 1 -- 172.21.15.40:0/457524014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07641022d0 msgr2=0x7f0764108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:16.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.801+0000 7f076b91d640 1 --2- 172.21.15.40:0/457524014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07641022d0 0x7f0764108b60 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f0760009ca0 tx=0x7f076002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.801+0000 7f076b91d640 1 -- 172.21.15.40:0/457524014 shutdown_connections 2024-03-18T22:59:16.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.801+0000 7f076b91d640 1 --2- 172.21.15.40:0/457524014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07641022d0 0x7f0764108b60 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.801+0000 7f076b91d640 1 --2- 172.21.15.40:0/457524014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0764101910 0x7f0764101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.801+0000 7f076b91d640 1 --2- 172.21.15.40:0/457524014 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0764100720 0x7f0764100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.801+0000 7f076b91d640 1 -- 172.21.15.40:0/457524014 >> 172.21.15.40:0/457524014 conn(0x7f07640fbcb0 msgr2=0x7f07640fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.801+0000 7f076b91d640 1 -- 172.21.15.40:0/457524014 shutdown_connections 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.801+0000 7f076b91d640 1 -- 172.21.15.40:0/457524014 wait complete. 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.802+0000 7f076b91d640 1 Processor -- start 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.802+0000 7f076b91d640 1 -- start start 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.802+0000 7f076b91d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0764100720 0x7f0764197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.802+0000 7f076b91d640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0764101910 0x7f0764197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.802+0000 7f076b91d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07641022d0 0x7f076419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.802+0000 7f076b91d640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f076410d370 con 0x7f0764100720 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.802+0000 7f076b91d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f076410d1f0 con 0x7f07641022d0 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.802+0000 7f076b91d640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f076410d4f0 con 0x7f0764101910 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.802+0000 7f07696b9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0764100720 0x7f0764197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.802+0000 7f07696b9640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0764100720 0x7f0764197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49856/0 (socket says 172.21.15.40:49856) 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.802+0000 7f07696b9640 1 -- 172.21.15.40:0/3075111514 learned_addr learned my addr 172.21.15.40:0/3075111514 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.802+0000 7f0768eb8640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0764101910 0x7f0764197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.802+0000 7f0769eba640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07641022d0 0x7f076419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.803+0000 7f07696b9640 1 -- 172.21.15.40:0/3075111514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0764101910 msgr2=0x7f0764197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.803+0000 7f07696b9640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0764101910 0x7f0764197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.803+0000 7f07696b9640 1 -- 172.21.15.40:0/3075111514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07641022d0 msgr2=0x7f076419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.803+0000 7f07696b9640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07641022d0 0x7f076419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.803+0000 7f07696b9640 1 -- 172.21.15.40:0/3075111514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076419f670 con 0x7f0764100720 2024-03-18T22:59:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.803+0000 7f0768eb8640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0764101910 0x7f0764197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:59:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.803+0000 7f0769eba640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07641022d0 0x7f076419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:59:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.803+0000 7f07696b9640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0764100720 0x7f0764197a50 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f075c00e9c0 tx=0x7f075c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.803+0000 7f075a7fc640 1 -- 172.21.15.40:0/3075111514 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075c00cd80 con 0x7f0764100720 2024-03-18T22:59:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.803+0000 7f075a7fc640 1 -- 172.21.15.40:0/3075111514 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f075c0035f0 con 0x7f0764100720 2024-03-18T22:59:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.803+0000 7f076b91d640 1 -- 172.21.15.40:0/3075111514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f076419f960 con 0x7f0764100720 2024-03-18T22:59:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.803+0000 7f075a7fc640 1 -- 172.21.15.40:0/3075111514 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075c010690 con 0x7f0764100720 2024-03-18T22:59:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.803+0000 7f076b91d640 1 -- 172.21.15.40:0/3075111514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f076419fe60 con 0x7f0764100720 2024-03-18T22:59:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.804+0000 7f076b91d640 1 -- 172.21.15.40:0/3075111514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0764107ab0 con 0x7f0764100720 2024-03-18T22:59:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.806+0000 7f075a7fc640 1 -- 172.21.15.40:0/3075111514 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f075c0037a0 con 0x7f0764100720 2024-03-18T22:59:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.806+0000 7f075a7fc640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f07500a2800 0x7f07500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.806+0000 7f0768eb8640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f07500a2800 0x7f07500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.806+0000 7f075a7fc640 1 -- 172.21.15.40:0/3075111514 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f075c0cf4c0 con 0x7f0764100720 2024-03-18T22:59:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.807+0000 7f0768eb8640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f07500a2800 0x7f07500a4cc0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0764198f60 tx=0x7f0754005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:16.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.810+0000 7f075a7fc640 1 -- 172.21.15.40:0/3075111514 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f075c09c170 con 0x7f0764100720 2024-03-18T22:59:16.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:16 smithi113 ceph-mon[66775]: pgmap v849: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:59:16.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:16 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/250938774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:59:16.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:16 smithi103 ceph-mon[66377]: pgmap v849: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T22:59:16.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:16 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/250938774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:59:16.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.993+0000 7f076b91d640 1 -- 172.21.15.40:0/3075111514 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f076402d3a0 con 0x7f0764100720 2024-03-18T22:59:16.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.994+0000 7f075a7fc640 1 -- 172.21.15.40:0/3075111514 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f075c09ba70 con 0x7f0764100720 2024-03-18T22:59:16.994 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:59:16.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.996+0000 7f076b91d640 1 -- 172.21.15.40:0/3075111514 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f07500a2800 msgr2=0x7f07500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:16.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.996+0000 7f076b91d640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f07500a2800 0x7f07500a4cc0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0764198f60 tx=0x7f0754005f90 comp rx=0 tx=0).stop 2024-03-18T22:59:16.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.996+0000 7f076b91d640 1 -- 172.21.15.40:0/3075111514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0764100720 msgr2=0x7f0764197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:16.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.996+0000 7f076b91d640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0764100720 0x7f0764197a50 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f075c00e9c0 tx=0x7f075c00ee80 comp rx=0 tx=0).stop 2024-03-18T22:59:16.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.996+0000 7f076b91d640 1 -- 172.21.15.40:0/3075111514 shutdown_connections 2024-03-18T22:59:16.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.996+0000 7f076b91d640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f07500a2800 0x7f07500a4cc0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.996+0000 7f076b91d640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f07641022d0 0x7f076419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.996+0000 7f076b91d640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f0764101910 0x7f0764197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.996+0000 7f076b91d640 1 --2- 172.21.15.40:0/3075111514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0764100720 0x7f0764197a50 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:16.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.996+0000 7f076b91d640 1 -- 172.21.15.40:0/3075111514 >> 172.21.15.40:0/3075111514 conn(0x7f07640fbcb0 msgr2=0x7f0764104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:16.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.996+0000 7f076b91d640 1 -- 172.21.15.40:0/3075111514 shutdown_connections 2024-03-18T22:59:16.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:16.996+0000 7f076b91d640 1 -- 172.21.15.40:0/3075111514 wait complete. 2024-03-18T22:59:17.007 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T22:59:17.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.112+0000 7fe666d52640 1 -- 172.21.15.40:0/3143454107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe660100720 msgr2=0x7fe660100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:17.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.112+0000 7fe666d52640 1 --2- 172.21.15.40:0/3143454107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe660100720 0x7fe660100b40 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fe654009900 tx=0x7fe65402bc30 comp rx=0 tx=0).stop 2024-03-18T22:59:17.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.112+0000 7fe666d52640 1 -- 172.21.15.40:0/3143454107 shutdown_connections 2024-03-18T22:59:17.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.112+0000 7fe666d52640 1 --2- 172.21.15.40:0/3143454107 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe6601022d0 0x7fe660108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.112+0000 7fe666d52640 1 --2- 172.21.15.40:0/3143454107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe660101910 0x7fe660101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.112+0000 7fe666d52640 1 --2- 172.21.15.40:0/3143454107 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe660100720 0x7fe660100b40 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.112+0000 7fe666d52640 1 -- 172.21.15.40:0/3143454107 >> 172.21.15.40:0/3143454107 conn(0x7fe6600fbcb0 msgr2=0x7fe6600fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.112+0000 7fe666d52640 1 -- 172.21.15.40:0/3143454107 shutdown_connections 2024-03-18T22:59:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.112+0000 7fe666d52640 1 -- 172.21.15.40:0/3143454107 wait complete. 2024-03-18T22:59:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.112+0000 7fe666d52640 1 Processor -- start 2024-03-18T22:59:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.112+0000 7fe666d52640 1 -- start start 2024-03-18T22:59:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe666d52640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe660100720 0x7fe660069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe666d52640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe660101910 0x7fe66019c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe666d52640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6601022d0 0x7fe66019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe666d52640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fe66010c250 con 0x7fe6601022d0 2024-03-18T22:59:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe666d52640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe66010c0d0 con 0x7fe660101910 2024-03-18T22:59:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe666d52640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fe66010c3d0 con 0x7fe660100720 2024-03-18T22:59:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe6652ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6601022d0 0x7fe66019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe6652ef640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6601022d0 0x7fe66019ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49862/0 (socket says 172.21.15.40:49862) 2024-03-18T22:59:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe6652ef640 1 -- 172.21.15.40:0/505410930 learned_addr learned my addr 172.21.15.40:0/505410930 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe664aee640 1 --2- 172.21.15.40:0/505410930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe660100720 0x7fe660069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe65ffff640 1 --2- 172.21.15.40:0/505410930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe660101910 0x7fe66019c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe6652ef640 1 -- 172.21.15.40:0/505410930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe660100720 msgr2=0x7fe660069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe6652ef640 1 --2- 172.21.15.40:0/505410930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe660100720 0x7fe660069510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe6652ef640 1 -- 172.21.15.40:0/505410930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe660101910 msgr2=0x7fe66019c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe6652ef640 1 --2- 172.21.15.40:0/505410930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe660101910 0x7fe66019c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe6652ef640 1 -- 172.21.15.40:0/505410930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe66019f6e0 con 0x7fe6601022d0 2024-03-18T22:59:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe6652ef640 1 --2- 172.21.15.40:0/505410930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6601022d0 0x7fe66019ee20 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fe65800eb10 tx=0x7fe65800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe65dffb640 1 -- 172.21.15.40:0/505410930 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe658004b20 con 0x7fe6601022d0 2024-03-18T22:59:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe666d52640 1 -- 172.21.15.40:0/505410930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe66019f9d0 con 0x7fe6601022d0 2024-03-18T22:59:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe65dffb640 1 -- 172.21.15.40:0/505410930 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe658004cc0 con 0x7fe6601022d0 2024-03-18T22:59:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.113+0000 7fe65dffb640 1 -- 172.21.15.40:0/505410930 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe658010690 con 0x7fe6601022d0 2024-03-18T22:59:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.114+0000 7fe666d52640 1 -- 172.21.15.40:0/505410930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe66019ff20 con 0x7fe6601022d0 2024-03-18T22:59:17.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.114+0000 7fe666d52640 1 -- 172.21.15.40:0/505410930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe660107bb0 con 0x7fe6601022d0 2024-03-18T22:59:17.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.115+0000 7fe65dffb640 1 -- 172.21.15.40:0/505410930 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe658005020 con 0x7fe6601022d0 2024-03-18T22:59:17.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.116+0000 7fe65dffb640 1 --2- 172.21.15.40:0/505410930 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe6500a2800 0x7fe6500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:17.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.116+0000 7fe664aee640 1 --2- 172.21.15.40:0/505410930 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe6500a2800 0x7fe6500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:17.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.116+0000 7fe65dffb640 1 -- 172.21.15.40:0/505410930 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe6580cf570 con 0x7fe6601022d0 2024-03-18T22:59:17.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.117+0000 7fe664aee640 1 --2- 172.21.15.40:0/505410930 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe6500a2800 0x7fe6500a4cc0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe654005ab0 tx=0x7fe654005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:17.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.120+0000 7fe65dffb640 1 -- 172.21.15.40:0/505410930 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe65809c1c0 con 0x7fe6601022d0 2024-03-18T22:59:17.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.304+0000 7fe666d52640 1 -- 172.21.15.40:0/505410930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe660062420 con 0x7fe6601022d0 2024-03-18T22:59:17.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.304+0000 7fe65dffb640 1 -- 172.21.15.40:0/505410930 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fe65809bac0 con 0x7fe6601022d0 2024-03-18T22:59:17.305 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-18T22:59:17.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.306+0000 7fe666d52640 1 -- 172.21.15.40:0/505410930 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe6500a2800 msgr2=0x7fe6500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:17.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.306+0000 7fe666d52640 1 --2- 172.21.15.40:0/505410930 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe6500a2800 0x7fe6500a4cc0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe654005ab0 tx=0x7fe654005fd0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.306+0000 7fe666d52640 1 -- 172.21.15.40:0/505410930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6601022d0 msgr2=0x7fe66019ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:17.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.306+0000 7fe666d52640 1 --2- 172.21.15.40:0/505410930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6601022d0 0x7fe66019ee20 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fe65800eb10 tx=0x7fe65800c2d0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.306+0000 7fe666d52640 1 -- 172.21.15.40:0/505410930 shutdown_connections 2024-03-18T22:59:17.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.306+0000 7fe666d52640 1 --2- 172.21.15.40:0/505410930 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fe6500a2800 0x7fe6500a4cc0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.306+0000 7fe666d52640 1 --2- 172.21.15.40:0/505410930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6601022d0 0x7fe66019ee20 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.306+0000 7fe666d52640 1 --2- 172.21.15.40:0/505410930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe660101910 0x7fe66019c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.306+0000 7fe666d52640 1 --2- 172.21.15.40:0/505410930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fe660100720 0x7fe660069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.306+0000 7fe666d52640 1 -- 172.21.15.40:0/505410930 >> 172.21.15.40:0/505410930 conn(0x7fe6600fbcb0 msgr2=0x7fe660104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:17.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.306+0000 7fe666d52640 1 -- 172.21.15.40:0/505410930 shutdown_connections 2024-03-18T22:59:17.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.306+0000 7fe666d52640 1 -- 172.21.15.40:0/505410930 wait complete. 2024-03-18T22:59:17.317 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T22:59:17.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.429+0000 7fb97dee8640 1 -- 172.21.15.40:0/968700749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb978100740 msgr2=0x7fb978100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:17.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.429+0000 7fb97dee8640 1 --2- 172.21.15.40:0/968700749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb978100740 0x7fb978100b60 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009900 tx=0x7fb96c02bc30 comp rx=0 tx=0).stop 2024-03-18T22:59:17.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.429+0000 7fb97dee8640 1 -- 172.21.15.40:0/968700749 shutdown_connections 2024-03-18T22:59:17.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.429+0000 7fb97dee8640 1 --2- 172.21.15.40:0/968700749 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9781022f0 0x7fb978108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.429+0000 7fb97dee8640 1 --2- 172.21.15.40:0/968700749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb978101930 0x7fb978101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.429+0000 7fb97dee8640 1 --2- 172.21.15.40:0/968700749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb978100740 0x7fb978100b60 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.429+0000 7fb97dee8640 1 -- 172.21.15.40:0/968700749 >> 172.21.15.40:0/968700749 conn(0x7fb9780fbcd0 msgr2=0x7fb9780fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:17.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.429+0000 7fb97dee8640 1 -- 172.21.15.40:0/968700749 shutdown_connections 2024-03-18T22:59:17.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.429+0000 7fb97dee8640 1 -- 172.21.15.40:0/968700749 wait complete. 2024-03-18T22:59:17.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.430+0000 7fb97dee8640 1 Processor -- start 2024-03-18T22:59:17.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.430+0000 7fb97dee8640 1 -- start start 2024-03-18T22:59:17.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.430+0000 7fb97dee8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb978100740 0x7fb9781046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:17.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.430+0000 7fb97dee8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb978101930 0x7fb978102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:17.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.430+0000 7fb97dee8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9781022f0 0x7fb978103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:17.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.430+0000 7fb97dee8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fb97810c270 con 0x7fb978101930 2024-03-18T22:59:17.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.430+0000 7fb97dee8640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb97810c0f0 con 0x7fb978100740 2024-03-18T22:59:17.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.430+0000 7fb97dee8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fb97810c3f0 con 0x7fb9781022f0 2024-03-18T22:59:17.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.430+0000 7fb976ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb978101930 0x7fb978102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:17.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.430+0000 7fb976ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb978101930 0x7fb978102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49886/0 (socket says 172.21.15.40:49886) 2024-03-18T22:59:17.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.430+0000 7fb9777fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb978100740 0x7fb9781046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:17.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.430+0000 7fb976ffd640 1 -- 172.21.15.40:0/1724664930 learned_addr learned my addr 172.21.15.40:0/1724664930 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:17.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.430+0000 7fb977fff640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9781022f0 0x7fb978103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:17.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.431+0000 7fb976ffd640 1 -- 172.21.15.40:0/1724664930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9781022f0 msgr2=0x7fb978103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:17.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.431+0000 7fb976ffd640 1 --2- 172.21.15.40:0/1724664930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9781022f0 0x7fb978103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.431+0000 7fb976ffd640 1 -- 172.21.15.40:0/1724664930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb978100740 msgr2=0x7fb9781046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:17.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.431+0000 7fb976ffd640 1 --2- 172.21.15.40:0/1724664930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb978100740 0x7fb9781046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.431+0000 7fb976ffd640 1 -- 172.21.15.40:0/1724664930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb978103ae0 con 0x7fb978101930 2024-03-18T22:59:17.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.431+0000 7fb9777fe640 1 --2- 172.21.15.40:0/1724664930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb978100740 0x7fb9781046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:59:17.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.431+0000 7fb976ffd640 1 --2- 172.21.15.40:0/1724664930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb978101930 0x7fb978102d10 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fb96800e9f0 tx=0x7fb96800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:17.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.431+0000 7fb974ff9640 1 -- 172.21.15.40:0/1724664930 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96800cd80 con 0x7fb978101930 2024-03-18T22:59:17.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.431+0000 7fb97dee8640 1 -- 172.21.15.40:0/1724664930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9781a1be0 con 0x7fb978101930 2024-03-18T22:59:17.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.431+0000 7fb974ff9640 1 -- 172.21.15.40:0/1724664930 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9680108e0 con 0x7fb978101930 2024-03-18T22:59:17.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.431+0000 7fb974ff9640 1 -- 172.21.15.40:0/1724664930 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968018920 con 0x7fb978101930 2024-03-18T22:59:17.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.431+0000 7fb97dee8640 1 -- 172.21.15.40:0/1724664930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9781a2180 con 0x7fb978101930 2024-03-18T22:59:17.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.432+0000 7fb97dee8640 1 -- 172.21.15.40:0/1724664930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9781a2350 con 0x7fb978101930 2024-03-18T22:59:17.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.434+0000 7fb974ff9640 1 -- 172.21.15.40:0/1724664930 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb968010a90 con 0x7fb978101930 2024-03-18T22:59:17.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.434+0000 7fb974ff9640 1 --2- 172.21.15.40:0/1724664930 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9600a2800 0x7fb9600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:17.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.434+0000 7fb974ff9640 1 -- 172.21.15.40:0/1724664930 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb9680cf140 con 0x7fb978101930 2024-03-18T22:59:17.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.434+0000 7fb9777fe640 1 --2- 172.21.15.40:0/1724664930 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9600a2800 0x7fb9600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:17.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.435+0000 7fb9777fe640 1 --2- 172.21.15.40:0/1724664930 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9600a2800 0x7fb9600a4cc0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb96c005ab0 tx=0x7fb96c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:17.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.442+0000 7fb974ff9640 1 -- 172.21.15.40:0/1724664930 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb96809b7a0 con 0x7fb978101930 2024-03-18T22:59:17.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.629+0000 7fb97dee8640 1 -- 172.21.15.40:0/1724664930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb9781a2520 con 0x7fb978101930 2024-03-18T22:59:17.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.629+0000 7fb974ff9640 1 -- 172.21.15.40:0/1724664930 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fb96809f840 con 0x7fb978101930 2024-03-18T22:59:17.630 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:59:17.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.631+0000 7fb97dee8640 1 -- 172.21.15.40:0/1724664930 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9600a2800 msgr2=0x7fb9600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:17.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.631+0000 7fb97dee8640 1 --2- 172.21.15.40:0/1724664930 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9600a2800 0x7fb9600a4cc0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fb96c005ab0 tx=0x7fb96c005fd0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.631+0000 7fb97dee8640 1 -- 172.21.15.40:0/1724664930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb978101930 msgr2=0x7fb978102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:17.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.631+0000 7fb97dee8640 1 --2- 172.21.15.40:0/1724664930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb978101930 0x7fb978102d10 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fb96800e9f0 tx=0x7fb96800eeb0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.631+0000 7fb97dee8640 1 -- 172.21.15.40:0/1724664930 shutdown_connections 2024-03-18T22:59:17.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.631+0000 7fb97dee8640 1 --2- 172.21.15.40:0/1724664930 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fb9600a2800 0x7fb9600a4cc0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.631+0000 7fb97dee8640 1 --2- 172.21.15.40:0/1724664930 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fb9781022f0 0x7fb978103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.631+0000 7fb97dee8640 1 --2- 172.21.15.40:0/1724664930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb978101930 0x7fb978102d10 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.631+0000 7fb97dee8640 1 --2- 172.21.15.40:0/1724664930 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb978100740 0x7fb9781046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.631+0000 7fb97dee8640 1 -- 172.21.15.40:0/1724664930 >> 172.21.15.40:0/1724664930 conn(0x7fb9780fbcd0 msgr2=0x7fb978069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:17.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.632+0000 7fb97dee8640 1 -- 172.21.15.40:0/1724664930 shutdown_connections 2024-03-18T22:59:17.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.632+0000 7fb97dee8640 1 -- 172.21.15.40:0/1724664930 wait complete. 2024-03-18T22:59:17.642 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:17 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1481752839' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:59:17.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:17 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3075111514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:59:17.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:17 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/505410930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:59:17.644 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T22:59:17.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.756+0000 7f7a1ee27640 1 -- 172.21.15.40:0/3779035012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a18101910 msgr2=0x7f7a18101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.756+0000 7f7a1ee27640 1 --2- 172.21.15.40:0/3779035012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a18101910 0x7f7a18101d90 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f7a04009900 tx=0x7f7a0402bc30 comp rx=0 tx=0).stop 2024-03-18T22:59:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.756+0000 7f7a1ee27640 1 -- 172.21.15.40:0/3779035012 shutdown_connections 2024-03-18T22:59:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.756+0000 7f7a1ee27640 1 --2- 172.21.15.40:0/3779035012 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a181022d0 0x7f7a18108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.756+0000 7f7a1ee27640 1 --2- 172.21.15.40:0/3779035012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a18101910 0x7f7a18101d90 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.756+0000 7f7a1ee27640 1 --2- 172.21.15.40:0/3779035012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a18100720 0x7f7a18100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.756+0000 7f7a1ee27640 1 -- 172.21.15.40:0/3779035012 >> 172.21.15.40:0/3779035012 conn(0x7f7a180fbcb0 msgr2=0x7f7a180fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.756+0000 7f7a1ee27640 1 -- 172.21.15.40:0/3779035012 shutdown_connections 2024-03-18T22:59:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.756+0000 7f7a1ee27640 1 -- 172.21.15.40:0/3779035012 wait complete. 2024-03-18T22:59:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.757+0000 7f7a1ee27640 1 Processor -- start 2024-03-18T22:59:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.757+0000 7f7a1ee27640 1 -- start start 2024-03-18T22:59:17.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.757+0000 7f7a1ee27640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a18100720 0x7f7a18197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.757+0000 7f7a1ee27640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a18101910 0x7f7a18197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.757+0000 7f7a1ee27640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a181022d0 0x7f7a1819f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.757+0000 7f7a1ee27640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1810d370 con 0x7f7a181022d0 2024-03-18T22:59:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.757+0000 7f7a1ee27640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1810d1f0 con 0x7f7a18100720 2024-03-18T22:59:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.757+0000 7f7a1ee27640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1810d4f0 con 0x7f7a18101910 2024-03-18T22:59:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.757+0000 7f7a1d3c4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a181022d0 0x7f7a1819f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.757+0000 7f7a1d3c4640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a181022d0 0x7f7a1819f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49908/0 (socket says 172.21.15.40:49908) 2024-03-18T22:59:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.757+0000 7f7a1cbc3640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a18100720 0x7f7a18197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.757+0000 7f7a1d3c4640 1 -- 172.21.15.40:0/1586745838 learned_addr learned my addr 172.21.15.40:0/1586745838 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:17.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.757+0000 7f7a17fff640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a18101910 0x7f7a18197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.758+0000 7f7a1d3c4640 1 -- 172.21.15.40:0/1586745838 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a18101910 msgr2=0x7f7a18197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.758+0000 7f7a1d3c4640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a18101910 0x7f7a18197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.758+0000 7f7a1d3c4640 1 -- 172.21.15.40:0/1586745838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a18100720 msgr2=0x7f7a18197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.758+0000 7f7a1d3c4640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a18100720 0x7f7a18197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.758+0000 7f7a1d3c4640 1 -- 172.21.15.40:0/1586745838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a1819f6a0 con 0x7f7a181022d0 2024-03-18T22:59:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.758+0000 7f7a1cbc3640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a18100720 0x7f7a18197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:59:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.758+0000 7f7a17fff640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a18101910 0x7f7a18197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:59:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.758+0000 7f7a1d3c4640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a181022d0 0x7f7a1819f030 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f7a1000eb10 tx=0x7f7a1000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.758+0000 7f7a15ffb640 1 -- 172.21.15.40:0/1586745838 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a10004b20 con 0x7f7a181022d0 2024-03-18T22:59:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.758+0000 7f7a15ffb640 1 -- 172.21.15.40:0/1586745838 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7a10004cc0 con 0x7f7a181022d0 2024-03-18T22:59:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.758+0000 7f7a1ee27640 1 -- 172.21.15.40:0/1586745838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a1819f990 con 0x7f7a181022d0 2024-03-18T22:59:17.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.758+0000 7f7a15ffb640 1 -- 172.21.15.40:0/1586745838 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a10010690 con 0x7f7a181022d0 2024-03-18T22:59:17.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.758+0000 7f7a1ee27640 1 -- 172.21.15.40:0/1586745838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a1819feb0 con 0x7f7a181022d0 2024-03-18T22:59:17.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.759+0000 7f7a1ee27640 1 -- 172.21.15.40:0/1586745838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a18068610 con 0x7f7a181022d0 2024-03-18T22:59:17.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.760+0000 7f7a15ffb640 1 -- 172.21.15.40:0/1586745838 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7a10005020 con 0x7f7a181022d0 2024-03-18T22:59:17.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.761+0000 7f7a15ffb640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7a080a2800 0x7f7a080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:17.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.761+0000 7f7a1cbc3640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7a080a2800 0x7f7a080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:17.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.761+0000 7f7a15ffb640 1 -- 172.21.15.40:0/1586745838 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7a100cf570 con 0x7f7a181022d0 2024-03-18T22:59:17.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.761+0000 7f7a1cbc3640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7a080a2800 0x7f7a080a4cc0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7a18101770 tx=0x7f7a0c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:17.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.765+0000 7f7a15ffb640 1 -- 172.21.15.40:0/1586745838 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7a1009c1c0 con 0x7f7a181022d0 2024-03-18T22:59:17.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:17 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1481752839' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:59:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:17 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3075111514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:59:17.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:17 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/505410930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:59:17.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:17 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1481752839' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:59:17.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:17 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3075111514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:59:17.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:17 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/505410930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:59:17.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.959+0000 7f7a1ee27640 1 -- 172.21.15.40:0/1586745838 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7a18062420 con 0x7f7a181022d0 2024-03-18T22:59:17.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.959+0000 7f7a15ffb640 1 -- 172.21.15.40:0/1586745838 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v93) ==== 81+0+591 (secure 0 0 0) 0x7f7a1009bac0 con 0x7f7a181022d0 2024-03-18T22:59:17.960 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/347668672 2024-03-19T22:33:42.721571+0000 2024-03-18T22:59:17.960 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/4237962144 2024-03-19T22:33:42.721571+0000 2024-03-18T22:59:17.960 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/1454961757 2024-03-19T22:33:15.826478+0000 2024-03-18T22:59:17.960 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3047621005 2024-03-19T22:33:15.826478+0000 2024-03-18T22:59:17.960 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/2121391701 2024-03-19T22:33:15.826478+0000 2024-03-18T22:59:17.961 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/3876208039 2024-03-19T22:33:42.721571+0000 2024-03-18T22:59:17.961 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/3876208039 2024-03-19T22:33:42.721571+0000 2024-03-18T22:59:17.961 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/2591529486 2024-03-19T22:33:42.721571+0000 2024-03-18T22:59:17.961 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/1153335782 2024-03-19T22:33:15.826478+0000 2024-03-18T22:59:17.961 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/1153335782 2024-03-19T22:33:15.826478+0000 2024-03-18T22:59:17.961 INFO:teuthology.orchestra.run.smithi040.stderr:listed 10 entries 2024-03-18T22:59:17.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.961+0000 7f7a1ee27640 1 -- 172.21.15.40:0/1586745838 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7a080a2800 msgr2=0x7f7a080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:17.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.961+0000 7f7a1ee27640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7a080a2800 0x7f7a080a4cc0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7a18101770 tx=0x7f7a0c005f90 comp rx=0 tx=0).stop 2024-03-18T22:59:17.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.961+0000 7f7a1ee27640 1 -- 172.21.15.40:0/1586745838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a181022d0 msgr2=0x7f7a1819f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:17.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.961+0000 7f7a1ee27640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a181022d0 0x7f7a1819f030 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f7a1000eb10 tx=0x7f7a1000c2d0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.962+0000 7f7a1ee27640 1 -- 172.21.15.40:0/1586745838 shutdown_connections 2024-03-18T22:59:17.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.962+0000 7f7a1ee27640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f7a080a2800 0x7f7a080a4cc0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.962+0000 7f7a1ee27640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a181022d0 0x7f7a1819f030 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.962+0000 7f7a1ee27640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f7a18101910 0x7f7a18197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.962+0000 7f7a1ee27640 1 --2- 172.21.15.40:0/1586745838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a18100720 0x7f7a18197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:17.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.962+0000 7f7a1ee27640 1 -- 172.21.15.40:0/1586745838 >> 172.21.15.40:0/1586745838 conn(0x7f7a180fbcb0 msgr2=0x7f7a18104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:17.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.962+0000 7f7a1ee27640 1 -- 172.21.15.40:0/1586745838 shutdown_connections 2024-03-18T22:59:17.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:17.962+0000 7f7a1ee27640 1 -- 172.21.15.40:0/1586745838 wait complete. 2024-03-18T22:59:17.974 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-18T22:59:17.975 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T22:59:17.997 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:59:18.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 -- 172.21.15.40:0/529878403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d2c1033f0 msgr2=0x7f8d2c109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:18.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 --2- 172.21.15.40:0/529878403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d2c1033f0 0x7f8d2c109c80 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f8d24009be0 tx=0x7f8d2402bef0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 -- 172.21.15.40:0/529878403 shutdown_connections 2024-03-18T22:59:18.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 --2- 172.21.15.40:0/529878403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d2c1033f0 0x7f8d2c109c80 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 --2- 172.21.15.40:0/529878403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d2c102a30 0x7f8d2c102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 --2- 172.21.15.40:0/529878403 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d2c101840 0x7f8d2c101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 -- 172.21.15.40:0/529878403 >> 172.21.15.40:0/529878403 conn(0x7f8d2c0fcdd0 msgr2=0x7f8d2c0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:18.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 -- 172.21.15.40:0/529878403 shutdown_connections 2024-03-18T22:59:18.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 -- 172.21.15.40:0/529878403 wait complete. 2024-03-18T22:59:18.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 Processor -- start 2024-03-18T22:59:18.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 -- start start 2024-03-18T22:59:18.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d2c101840 0x7f8d2c197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:18.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d2c102a30 0x7f8d2c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:18.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d2c1033f0 0x7f8d2c19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:18.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c10d3a0 con 0x7f8d2c1033f0 2024-03-18T22:59:18.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c10d220 con 0x7f8d2c102a30 2024-03-18T22:59:18.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.149+0000 7f8d33085640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c10d520 con 0x7f8d2c101840 2024-03-18T22:59:18.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d31622640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d2c1033f0 0x7f8d2c19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:18.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d30e21640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d2c101840 0x7f8d2c197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:18.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d2bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d2c102a30 0x7f8d2c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:18.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d31622640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d2c1033f0 0x7f8d2c19efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49928/0 (socket says 172.21.15.40:49928) 2024-03-18T22:59:18.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d30e21640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d2c101840 0x7f8d2c197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.113:3300/0 says I am v2:172.21.15.40:34628/0 (socket says 172.21.15.40:34628) 2024-03-18T22:59:18.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d31622640 1 -- 172.21.15.40:0/1909913395 learned_addr learned my addr 172.21.15.40:0/1909913395 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:18.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d31622640 1 -- 172.21.15.40:0/1909913395 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d2c101840 msgr2=0x7f8d2c197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:18.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d31622640 1 --2- 172.21.15.40:0/1909913395 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d2c101840 0x7f8d2c197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d31622640 1 -- 172.21.15.40:0/1909913395 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d2c102a30 msgr2=0x7f8d2c197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:18.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d31622640 1 --2- 172.21.15.40:0/1909913395 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d2c102a30 0x7f8d2c197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d31622640 1 -- 172.21.15.40:0/1909913395 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2c19f640 con 0x7f8d2c1033f0 2024-03-18T22:59:18.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d30e21640 1 --2- 172.21.15.40:0/1909913395 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d2c101840 0x7f8d2c197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:59:18.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d31622640 1 --2- 172.21.15.40:0/1909913395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d2c1033f0 0x7f8d2c19efd0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f8d24034fd0 tx=0x7f8d24033c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:18.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d29ffb640 1 -- 172.21.15.40:0/1909913395 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d24043020 con 0x7f8d2c1033f0 2024-03-18T22:59:18.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d29ffb640 1 -- 172.21.15.40:0/1909913395 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d24003e20 con 0x7f8d2c1033f0 2024-03-18T22:59:18.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d33085640 1 -- 172.21.15.40:0/1909913395 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d2c19f8d0 con 0x7f8d2c1033f0 2024-03-18T22:59:18.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d29ffb640 1 -- 172.21.15.40:0/1909913395 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d24034430 con 0x7f8d2c1033f0 2024-03-18T22:59:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.150+0000 7f8d33085640 1 -- 172.21.15.40:0/1909913395 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d2c19fe20 con 0x7f8d2c1033f0 2024-03-18T22:59:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.151+0000 7f8d33085640 1 -- 172.21.15.40:0/1909913395 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d2c108b00 con 0x7f8d2c1033f0 2024-03-18T22:59:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.152+0000 7f8d29ffb640 1 -- 172.21.15.40:0/1909913395 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8d24046020 con 0x7f8d2c1033f0 2024-03-18T22:59:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.153+0000 7f8d29ffb640 1 --2- 172.21.15.40:0/1909913395 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8d1c0a2800 0x7f8d1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.153+0000 7f8d29ffb640 1 -- 172.21.15.40:0/1909913395 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8d240f1cc0 con 0x7f8d2c1033f0 2024-03-18T22:59:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.153+0000 7f8d30e21640 1 --2- 172.21.15.40:0/1909913395 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8d1c0a2800 0x7f8d1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:18.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.153+0000 7f8d30e21640 1 --2- 172.21.15.40:0/1909913395 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8d1c0a2800 0x7f8d1c0a4cc0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c102890 tx=0x7f8d20005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:18.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.157+0000 7f8d29ffb640 1 -- 172.21.15.40:0/1909913395 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8d2404d020 con 0x7f8d2c1033f0 2024-03-18T22:59:18.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.341+0000 7f8d33085640 1 -- 172.21.15.40:0/1909913395 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d2c02d370 con 0x7f8d2c1033f0 2024-03-18T22:59:18.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.343+0000 7f8d29ffb640 1 -- 172.21.15.40:0/1909913395 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24908 (secure 0 0 0) 0x7f8d240be970 con 0x7f8d2c1033f0 2024-03-18T22:59:18.344 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:59:18.344 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":93,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:42:44.941873+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:59:18.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.345+0000 7f8d33085640 1 -- 172.21.15.40:0/1909913395 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8d1c0a2800 msgr2=0x7f8d1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:18.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.345+0000 7f8d33085640 1 --2- 172.21.15.40:0/1909913395 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8d1c0a2800 0x7f8d1c0a4cc0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c102890 tx=0x7f8d20005f90 comp rx=0 tx=0).stop 2024-03-18T22:59:18.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.345+0000 7f8d33085640 1 -- 172.21.15.40:0/1909913395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d2c1033f0 msgr2=0x7f8d2c19efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:18.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.345+0000 7f8d33085640 1 --2- 172.21.15.40:0/1909913395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d2c1033f0 0x7f8d2c19efd0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f8d24034fd0 tx=0x7f8d24033c40 comp rx=0 tx=0).stop 2024-03-18T22:59:18.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.345+0000 7f8d33085640 1 -- 172.21.15.40:0/1909913395 shutdown_connections 2024-03-18T22:59:18.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.345+0000 7f8d33085640 1 --2- 172.21.15.40:0/1909913395 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f8d1c0a2800 0x7f8d1c0a4cc0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.345+0000 7f8d33085640 1 --2- 172.21.15.40:0/1909913395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d2c1033f0 0x7f8d2c19efd0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.345+0000 7f8d33085640 1 --2- 172.21.15.40:0/1909913395 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d2c102a30 0x7f8d2c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.345+0000 7f8d33085640 1 --2- 172.21.15.40:0/1909913395 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f8d2c101840 0x7f8d2c197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.345+0000 7f8d33085640 1 -- 172.21.15.40:0/1909913395 >> 172.21.15.40:0/1909913395 conn(0x7f8d2c0fcdd0 msgr2=0x7f8d2c105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:18.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.345+0000 7f8d33085640 1 -- 172.21.15.40:0/1909913395 shutdown_connections 2024-03-18T22:59:18.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.345+0000 7f8d33085640 1 -- 172.21.15.40:0/1909913395 wait complete. 2024-03-18T22:59:18.357 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T22:36:07.060691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T22:40:39.925147+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T22:40:40.773757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T22:40:44.068183+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T22:59:18.358 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T22:59:18.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.470+0000 7ff1c2ca1640 1 -- 172.21.15.40:0/2447712960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1bc100720 msgr2=0x7ff1bc100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:18.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.470+0000 7ff1c2ca1640 1 --2- 172.21.15.40:0/2447712960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1bc100720 0x7ff1bc100b40 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0009900 tx=0x7ff1b002bc30 comp rx=0 tx=0).stop 2024-03-18T22:59:18.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.470+0000 7ff1c2ca1640 1 -- 172.21.15.40:0/2447712960 shutdown_connections 2024-03-18T22:59:18.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.470+0000 7ff1c2ca1640 1 --2- 172.21.15.40:0/2447712960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1bc1022d0 0x7ff1bc108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.470+0000 7ff1c2ca1640 1 --2- 172.21.15.40:0/2447712960 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1bc101910 0x7ff1bc101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.470+0000 7ff1c2ca1640 1 --2- 172.21.15.40:0/2447712960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1bc100720 0x7ff1bc100b40 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.470+0000 7ff1c2ca1640 1 -- 172.21.15.40:0/2447712960 >> 172.21.15.40:0/2447712960 conn(0x7ff1bc0fbcb0 msgr2=0x7ff1bc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:18.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.471+0000 7ff1c2ca1640 1 -- 172.21.15.40:0/2447712960 shutdown_connections 2024-03-18T22:59:18.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.471+0000 7ff1c2ca1640 1 -- 172.21.15.40:0/2447712960 wait complete. 2024-03-18T22:59:18.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.471+0000 7ff1c2ca1640 1 Processor -- start 2024-03-18T22:59:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.471+0000 7ff1c2ca1640 1 -- start start 2024-03-18T22:59:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.471+0000 7ff1c2ca1640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1bc100720 0x7ff1bc199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.471+0000 7ff1c2ca1640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1bc101910 0x7ff1bc19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.471+0000 7ff1c2ca1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1bc1022d0 0x7ff1bc1a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.471+0000 7ff1c2ca1640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff1bc10b090 con 0x7ff1bc100720 2024-03-18T22:59:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.471+0000 7ff1c2ca1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff1bc10af10 con 0x7ff1bc1022d0 2024-03-18T22:59:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.471+0000 7ff1c2ca1640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff1bc10b210 con 0x7ff1bc101910 2024-03-18T22:59:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.471+0000 7ff1c0a3d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1bc100720 0x7ff1bc199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.471+0000 7ff1c0a3d640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1bc100720 0x7ff1bc199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49954/0 (socket says 172.21.15.40:49954) 2024-03-18T22:59:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.471+0000 7ff1c0a3d640 1 -- 172.21.15.40:0/484698356 learned_addr learned my addr 172.21.15.40:0/484698356 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.471+0000 7ff1c123e640 1 --2- 172.21.15.40:0/484698356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1bc1022d0 0x7ff1bc1a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.471+0000 7ff1bbfff640 1 --2- 172.21.15.40:0/484698356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1bc101910 0x7ff1bc19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.472+0000 7ff1c0a3d640 1 -- 172.21.15.40:0/484698356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1bc101910 msgr2=0x7ff1bc19a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:18.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.472+0000 7ff1c0a3d640 1 --2- 172.21.15.40:0/484698356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1bc101910 0x7ff1bc19a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.472+0000 7ff1c0a3d640 1 -- 172.21.15.40:0/484698356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1bc1022d0 msgr2=0x7ff1bc1a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.472+0000 7ff1c0a3d640 1 --2- 172.21.15.40:0/484698356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1bc1022d0 0x7ff1bc1a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.472+0000 7ff1c0a3d640 1 -- 172.21.15.40:0/484698356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1bc1a1950 con 0x7ff1bc100720 2024-03-18T22:59:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.472+0000 7ff1c0a3d640 1 --2- 172.21.15.40:0/484698356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1bc100720 0x7ff1bc199d40 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0000c00 tx=0x7ff1b0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.472+0000 7ff1b9ffb640 1 -- 172.21.15.40:0/484698356 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0043020 con 0x7ff1bc100720 2024-03-18T22:59:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.472+0000 7ff1b9ffb640 1 -- 172.21.15.40:0/484698356 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff1b0004350 con 0x7ff1bc100720 2024-03-18T22:59:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.472+0000 7ff1c2ca1640 1 -- 172.21.15.40:0/484698356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1bc1a1be0 con 0x7ff1bc100720 2024-03-18T22:59:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.472+0000 7ff1b9ffb640 1 -- 172.21.15.40:0/484698356 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b0003b10 con 0x7ff1bc100720 2024-03-18T22:59:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.472+0000 7ff1c2ca1640 1 -- 172.21.15.40:0/484698356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1bc1a2180 con 0x7ff1bc100720 2024-03-18T22:59:18.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.473+0000 7ff1c2ca1640 1 -- 172.21.15.40:0/484698356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1bc068610 con 0x7ff1bc100720 2024-03-18T22:59:18.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.475+0000 7ff1b9ffb640 1 -- 172.21.15.40:0/484698356 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff1b0046020 con 0x7ff1bc100720 2024-03-18T22:59:18.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.475+0000 7ff1b9ffb640 1 --2- 172.21.15.40:0/484698356 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1ac0a2880 0x7ff1ac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:18.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.476+0000 7ff1b9ffb640 1 -- 172.21.15.40:0/484698356 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff1b00f1370 con 0x7ff1bc100720 2024-03-18T22:59:18.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.476+0000 7ff1bbfff640 1 --2- 172.21.15.40:0/484698356 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1ac0a2880 0x7ff1ac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:18.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.476+0000 7ff1bbfff640 1 --2- 172.21.15.40:0/484698356 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1ac0a2880 0x7ff1ac0a4d40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8009bd0 tx=0x7ff1a8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:18.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.479+0000 7ff1b9ffb640 1 -- 172.21.15.40:0/484698356 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff1b004d020 con 0x7ff1bc100720 2024-03-18T22:59:18.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:18 smithi040 ceph-mon[62643]: pgmap v850: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T22:59:18.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:18 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1724664930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:59:18.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:18 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1586745838' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T22:59:18.644 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:18 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1909913395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:59:18.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.662+0000 7ff1c2ca1640 1 -- 172.21.15.40:0/484698356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff1bc02d1f0 con 0x7ff1bc100720 2024-03-18T22:59:18.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.662+0000 7ff1b9ffb640 1 -- 172.21.15.40:0/484698356 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7ff1b00bdfc0 con 0x7ff1bc100720 2024-03-18T22:59:18.663 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-18T22:59:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.664+0000 7ff1c2ca1640 1 -- 172.21.15.40:0/484698356 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1ac0a2880 msgr2=0x7ff1ac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.664+0000 7ff1c2ca1640 1 --2- 172.21.15.40:0/484698356 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1ac0a2880 0x7ff1ac0a4d40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff1a8009bd0 tx=0x7ff1a8005f90 comp rx=0 tx=0).stop 2024-03-18T22:59:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.664+0000 7ff1c2ca1640 1 -- 172.21.15.40:0/484698356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1bc100720 msgr2=0x7ff1bc199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.664+0000 7ff1c2ca1640 1 --2- 172.21.15.40:0/484698356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1bc100720 0x7ff1bc199d40 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0000c00 tx=0x7ff1b0005dc0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.664+0000 7ff1c2ca1640 1 -- 172.21.15.40:0/484698356 shutdown_connections 2024-03-18T22:59:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.664+0000 7ff1c2ca1640 1 --2- 172.21.15.40:0/484698356 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1ac0a2880 0x7ff1ac0a4d40 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.664+0000 7ff1c2ca1640 1 --2- 172.21.15.40:0/484698356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1bc1022d0 0x7ff1bc1a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.664+0000 7ff1c2ca1640 1 --2- 172.21.15.40:0/484698356 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff1bc101910 0x7ff1bc19a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.664+0000 7ff1c2ca1640 1 --2- 172.21.15.40:0/484698356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1bc100720 0x7ff1bc199d40 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.664+0000 7ff1c2ca1640 1 -- 172.21.15.40:0/484698356 >> 172.21.15.40:0/484698356 conn(0x7ff1bc0fbcb0 msgr2=0x7ff1bc104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.664+0000 7ff1c2ca1640 1 -- 172.21.15.40:0/484698356 shutdown_connections 2024-03-18T22:59:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.664+0000 7ff1c2ca1640 1 -- 172.21.15.40:0/484698356 wait complete. 2024-03-18T22:59:18.677 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T22:59:18.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.787+0000 7f605d612640 1 -- 172.21.15.40:0/2762506352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60581022d0 msgr2=0x7f6058108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:18.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.787+0000 7f605d612640 1 --2- 172.21.15.40:0/2762506352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60581022d0 0x7f6058108b60 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f6048009ca0 tx=0x7f604802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.787+0000 7f605d612640 1 -- 172.21.15.40:0/2762506352 shutdown_connections 2024-03-18T22:59:18.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.787+0000 7f605d612640 1 --2- 172.21.15.40:0/2762506352 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60581022d0 0x7f6058108b60 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.787+0000 7f605d612640 1 --2- 172.21.15.40:0/2762506352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6058101910 0x7f6058101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.787+0000 7f605d612640 1 --2- 172.21.15.40:0/2762506352 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6058100720 0x7f6058100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.787+0000 7f605d612640 1 -- 172.21.15.40:0/2762506352 >> 172.21.15.40:0/2762506352 conn(0x7f60580fbcb0 msgr2=0x7f60580fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.788+0000 7f605d612640 1 -- 172.21.15.40:0/2762506352 shutdown_connections 2024-03-18T22:59:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.788+0000 7f605d612640 1 -- 172.21.15.40:0/2762506352 wait complete. 2024-03-18T22:59:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.788+0000 7f605d612640 1 Processor -- start 2024-03-18T22:59:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.788+0000 7f605d612640 1 -- start start 2024-03-18T22:59:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.788+0000 7f605d612640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6058100720 0x7f6058197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.788+0000 7f605d612640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6058101910 0x7f6058197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.788+0000 7f605d612640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60581022d0 0x7f605819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.788+0000 7f605d612640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f605810c250 con 0x7f6058100720 2024-03-18T22:59:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.788+0000 7f605d612640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f605810c0d0 con 0x7f60581022d0 2024-03-18T22:59:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.788+0000 7f605d612640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f605810c3d0 con 0x7f6058101910 2024-03-18T22:59:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.788+0000 7f6056ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6058100720 0x7f6058197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.789+0000 7f6056ffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6058100720 0x7f6058197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60688/0 (socket says 172.21.15.40:60688) 2024-03-18T22:59:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.789+0000 7f6056ffd640 1 -- 172.21.15.40:0/1840666923 learned_addr learned my addr 172.21.15.40:0/1840666923 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.789+0000 7f60577fe640 1 --2- 172.21.15.40:0/1840666923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60581022d0 0x7f605819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.789+0000 7f60567fc640 1 --2- 172.21.15.40:0/1840666923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6058101910 0x7f6058197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.789+0000 7f6056ffd640 1 -- 172.21.15.40:0/1840666923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6058101910 msgr2=0x7f6058197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.789+0000 7f6056ffd640 1 --2- 172.21.15.40:0/1840666923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6058101910 0x7f6058197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.789+0000 7f6056ffd640 1 -- 172.21.15.40:0/1840666923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60581022d0 msgr2=0x7f605819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.789+0000 7f6056ffd640 1 --2- 172.21.15.40:0/1840666923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60581022d0 0x7f605819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.789+0000 7f6056ffd640 1 -- 172.21.15.40:0/1840666923 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605819f690 con 0x7f6058100720 2024-03-18T22:59:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.789+0000 7f6056ffd640 1 --2- 172.21.15.40:0/1840666923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6058100720 0x7f6058197a70 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f605000e9c0 tx=0x7f605000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.789+0000 7f604ffff640 1 -- 172.21.15.40:0/1840666923 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605000cd80 con 0x7f6058100720 2024-03-18T22:59:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.789+0000 7f605d612640 1 -- 172.21.15.40:0/1840666923 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605819f980 con 0x7f6058100720 2024-03-18T22:59:18.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.789+0000 7f604ffff640 1 -- 172.21.15.40:0/1840666923 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60500035f0 con 0x7f6058100720 2024-03-18T22:59:18.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.789+0000 7f604ffff640 1 -- 172.21.15.40:0/1840666923 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6050010690 con 0x7f6058100720 2024-03-18T22:59:18.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.789+0000 7f605d612640 1 -- 172.21.15.40:0/1840666923 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605819fed0 con 0x7f6058100720 2024-03-18T22:59:18.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.790+0000 7f605d612640 1 -- 172.21.15.40:0/1840666923 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60581a0100 con 0x7f6058100720 2024-03-18T22:59:18.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.791+0000 7f604ffff640 1 -- 172.21.15.40:0/1840666923 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f60500037a0 con 0x7f6058100720 2024-03-18T22:59:18.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.792+0000 7f604ffff640 1 --2- 172.21.15.40:0/1840666923 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f60440a2800 0x7f60440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:18.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.792+0000 7f60567fc640 1 --2- 172.21.15.40:0/1840666923 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f60440a2800 0x7f60440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:18.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.792+0000 7f604ffff640 1 -- 172.21.15.40:0/1840666923 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f60500cf400 con 0x7f6058100720 2024-03-18T22:59:18.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.792+0000 7f60567fc640 1 --2- 172.21.15.40:0/1840666923 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f60440a2800 0x7f60440a4cc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f60400095f0 tx=0x7f6040005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:18.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.796+0000 7f604ffff640 1 -- 172.21.15.40:0/1840666923 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f605009c050 con 0x7f6058100720 2024-03-18T22:59:18.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:18 smithi113 ceph-mon[66775]: pgmap v850: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T22:59:18.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:18 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1724664930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:59:18.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:18 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1586745838' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T22:59:18.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:18 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1909913395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:59:18.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:18 smithi103 ceph-mon[66377]: pgmap v850: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-18T22:59:18.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:18 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1724664930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:59:18.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:18 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1586745838' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T22:59:18.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:18 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1909913395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:59:18.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.980+0000 7f605d612640 1 -- 172.21.15.40:0/1840666923 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f60581a02d0 con 0x7f6058100720 2024-03-18T22:59:18.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.981+0000 7f604ffff640 1 -- 172.21.15.40:0/1840666923 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f605009b950 con 0x7f6058100720 2024-03-18T22:59:18.981 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:59:18.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.983+0000 7f605d612640 1 -- 172.21.15.40:0/1840666923 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f60440a2800 msgr2=0x7f60440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:18.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.983+0000 7f605d612640 1 --2- 172.21.15.40:0/1840666923 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f60440a2800 0x7f60440a4cc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f60400095f0 tx=0x7f6040005f90 comp rx=0 tx=0).stop 2024-03-18T22:59:18.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.983+0000 7f605d612640 1 -- 172.21.15.40:0/1840666923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6058100720 msgr2=0x7f6058197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:18.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.983+0000 7f605d612640 1 --2- 172.21.15.40:0/1840666923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6058100720 0x7f6058197a70 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f605000e9c0 tx=0x7f605000ee80 comp rx=0 tx=0).stop 2024-03-18T22:59:18.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.983+0000 7f605d612640 1 -- 172.21.15.40:0/1840666923 shutdown_connections 2024-03-18T22:59:18.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.983+0000 7f605d612640 1 --2- 172.21.15.40:0/1840666923 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f60440a2800 0x7f60440a4cc0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.983+0000 7f605d612640 1 --2- 172.21.15.40:0/1840666923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60581022d0 0x7f605819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.983+0000 7f605d612640 1 --2- 172.21.15.40:0/1840666923 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6058101910 0x7f6058197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.983+0000 7f605d612640 1 --2- 172.21.15.40:0/1840666923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6058100720 0x7f6058197a70 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:18.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.983+0000 7f605d612640 1 -- 172.21.15.40:0/1840666923 >> 172.21.15.40:0/1840666923 conn(0x7f60580fbcb0 msgr2=0x7f6058068980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:18.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.983+0000 7f605d612640 1 -- 172.21.15.40:0/1840666923 shutdown_connections 2024-03-18T22:59:18.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:18.983+0000 7f605d612640 1 -- 172.21.15.40:0/1840666923 wait complete. 2024-03-18T22:59:18.995 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T22:59:19.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.101+0000 7f6aa2948640 1 -- 172.21.15.40:0/871929174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a9c101920 msgr2=0x7f6a9c101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:19.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.101+0000 7f6aa2948640 1 --2- 172.21.15.40:0/871929174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a9c101920 0x7f6a9c101da0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f6a90009900 tx=0x7f6a9002bc30 comp rx=0 tx=0).stop 2024-03-18T22:59:19.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.101+0000 7f6aa2948640 1 -- 172.21.15.40:0/871929174 shutdown_connections 2024-03-18T22:59:19.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.101+0000 7f6aa2948640 1 --2- 172.21.15.40:0/871929174 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a9c1022e0 0x7f6a9c108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.101+0000 7f6aa2948640 1 --2- 172.21.15.40:0/871929174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a9c101920 0x7f6a9c101da0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.101+0000 7f6aa2948640 1 --2- 172.21.15.40:0/871929174 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a9c100730 0x7f6a9c100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.101+0000 7f6aa2948640 1 -- 172.21.15.40:0/871929174 >> 172.21.15.40:0/871929174 conn(0x7f6a9c0fbcc0 msgr2=0x7f6a9c0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.101+0000 7f6aa2948640 1 -- 172.21.15.40:0/871929174 shutdown_connections 2024-03-18T22:59:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.101+0000 7f6aa2948640 1 -- 172.21.15.40:0/871929174 wait complete. 2024-03-18T22:59:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.101+0000 7f6aa2948640 1 Processor -- start 2024-03-18T22:59:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa2948640 1 -- start start 2024-03-18T22:59:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa2948640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a9c100730 0x7f6a9c06a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa2948640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a9c101920 0x7f6a9c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:19.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa2948640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a9c1022e0 0x7f6a9c19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:19.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa2948640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c10c260 con 0x7f6a9c1022e0 2024-03-18T22:59:19.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa2948640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c10c0e0 con 0x7f6a9c101920 2024-03-18T22:59:19.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa2948640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9c10c3e0 con 0x7f6a9c100730 2024-03-18T22:59:19.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa0ee5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a9c1022e0 0x7f6a9c19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:19.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa0ee5640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a9c1022e0 0x7f6a9c19ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60698/0 (socket says 172.21.15.40:60698) 2024-03-18T22:59:19.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa0ee5640 1 -- 172.21.15.40:0/1398164947 learned_addr learned my addr 172.21.15.40:0/1398164947 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:19.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6a9bfff640 1 --2- 172.21.15.40:0/1398164947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a9c100730 0x7f6a9c06a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:19.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6a9b7fe640 1 --2- 172.21.15.40:0/1398164947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a9c101920 0x7f6a9c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:19.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa0ee5640 1 -- 172.21.15.40:0/1398164947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a9c100730 msgr2=0x7f6a9c06a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:19.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa0ee5640 1 --2- 172.21.15.40:0/1398164947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a9c100730 0x7f6a9c06a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa0ee5640 1 -- 172.21.15.40:0/1398164947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a9c101920 msgr2=0x7f6a9c19c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:19.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa0ee5640 1 --2- 172.21.15.40:0/1398164947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a9c101920 0x7f6a9c19c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa0ee5640 1 -- 172.21.15.40:0/1398164947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a9c19f730 con 0x7f6a9c1022e0 2024-03-18T22:59:19.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6a9bfff640 1 --2- 172.21.15.40:0/1398164947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a9c100730 0x7f6a9c06a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:59:19.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.102+0000 7f6aa0ee5640 1 --2- 172.21.15.40:0/1398164947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a9c1022e0 0x7f6a9c19ee20 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f6a94009fd0 tx=0x7f6a9400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:19.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.103+0000 7f6a997fa640 1 -- 172.21.15.40:0/1398164947 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a94017070 con 0x7f6a9c1022e0 2024-03-18T22:59:19.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.103+0000 7f6a997fa640 1 -- 172.21.15.40:0/1398164947 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a94003e20 con 0x7f6a9c1022e0 2024-03-18T22:59:19.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.103+0000 7f6aa2948640 1 -- 172.21.15.40:0/1398164947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a9c19f9c0 con 0x7f6a9c1022e0 2024-03-18T22:59:19.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.103+0000 7f6a997fa640 1 -- 172.21.15.40:0/1398164947 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a94010460 con 0x7f6a9c1022e0 2024-03-18T22:59:19.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.103+0000 7f6aa2948640 1 -- 172.21.15.40:0/1398164947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a9c19fee0 con 0x7f6a9c1022e0 2024-03-18T22:59:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.104+0000 7f6aa2948640 1 -- 172.21.15.40:0/1398164947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a9c107b10 con 0x7f6a9c1022e0 2024-03-18T22:59:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.105+0000 7f6a997fa640 1 -- 172.21.15.40:0/1398164947 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6a94003970 con 0x7f6a9c1022e0 2024-03-18T22:59:19.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.106+0000 7f6a997fa640 1 --2- 172.21.15.40:0/1398164947 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a840a2880 0x7f6a840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.106+0000 7f6a997fa640 1 -- 172.21.15.40:0/1398164947 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6a940ceb20 con 0x7f6a9c1022e0 2024-03-18T22:59:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.106+0000 7f6a9bfff640 1 --2- 172.21.15.40:0/1398164947 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a840a2880 0x7f6a840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.107+0000 7f6a9bfff640 1 --2- 172.21.15.40:0/1398164947 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a840a2880 0x7f6a840a4d40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c101780 tx=0x7f6a8c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:19.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.110+0000 7f6a997fa640 1 -- 172.21.15.40:0/1398164947 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6a940cee20 con 0x7f6a9c1022e0 2024-03-18T22:59:19.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.293+0000 7f6aa2948640 1 -- 172.21.15.40:0/1398164947 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6a9c02d380 con 0x7f6a9c1022e0 2024-03-18T22:59:19.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.294+0000 7f6a997fa640 1 -- 172.21.15.40:0/1398164947 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f6a9409b200 con 0x7f6a9c1022e0 2024-03-18T22:59:19.294 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-18T22:59:19.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.295+0000 7f6aa2948640 1 -- 172.21.15.40:0/1398164947 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a840a2880 msgr2=0x7f6a840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:19.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.295+0000 7f6aa2948640 1 --2- 172.21.15.40:0/1398164947 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a840a2880 0x7f6a840a4d40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c101780 tx=0x7f6a8c005f90 comp rx=0 tx=0).stop 2024-03-18T22:59:19.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.295+0000 7f6aa2948640 1 -- 172.21.15.40:0/1398164947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a9c1022e0 msgr2=0x7f6a9c19ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:19.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.295+0000 7f6aa2948640 1 --2- 172.21.15.40:0/1398164947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a9c1022e0 0x7f6a9c19ee20 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f6a94009fd0 tx=0x7f6a9400b260 comp rx=0 tx=0).stop 2024-03-18T22:59:19.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.295+0000 7f6aa2948640 1 -- 172.21.15.40:0/1398164947 shutdown_connections 2024-03-18T22:59:19.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.295+0000 7f6aa2948640 1 --2- 172.21.15.40:0/1398164947 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f6a840a2880 0x7f6a840a4d40 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.295+0000 7f6aa2948640 1 --2- 172.21.15.40:0/1398164947 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a9c1022e0 0x7f6a9c19ee20 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.295+0000 7f6aa2948640 1 --2- 172.21.15.40:0/1398164947 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a9c101920 0x7f6a9c19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.295+0000 7f6aa2948640 1 --2- 172.21.15.40:0/1398164947 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f6a9c100730 0x7f6a9c06a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.295+0000 7f6aa2948640 1 -- 172.21.15.40:0/1398164947 >> 172.21.15.40:0/1398164947 conn(0x7f6a9c0fbcc0 msgr2=0x7f6a9c104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:19.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.296+0000 7f6aa2948640 1 -- 172.21.15.40:0/1398164947 shutdown_connections 2024-03-18T22:59:19.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.296+0000 7f6aa2948640 1 -- 172.21.15.40:0/1398164947 wait complete. 2024-03-18T22:59:19.309 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T22:59:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.415+0000 7fdc4acb2640 1 -- 172.21.15.40:0/4194101562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc44103400 msgr2=0x7fdc44109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.415+0000 7fdc4acb2640 1 --2- 172.21.15.40:0/4194101562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc44103400 0x7fdc44109c90 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009ca0 tx=0x7fdc3c02c0f0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.415+0000 7fdc4acb2640 1 -- 172.21.15.40:0/4194101562 shutdown_connections 2024-03-18T22:59:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.415+0000 7fdc4acb2640 1 --2- 172.21.15.40:0/4194101562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc44103400 0x7fdc44109c90 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.415+0000 7fdc4acb2640 1 --2- 172.21.15.40:0/4194101562 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc44102a40 0x7fdc44102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.415+0000 7fdc4acb2640 1 --2- 172.21.15.40:0/4194101562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc44101850 0x7fdc44101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.415+0000 7fdc4acb2640 1 -- 172.21.15.40:0/4194101562 >> 172.21.15.40:0/4194101562 conn(0x7fdc440fcdc0 msgr2=0x7fdc440ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.415+0000 7fdc4acb2640 1 -- 172.21.15.40:0/4194101562 shutdown_connections 2024-03-18T22:59:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.415+0000 7fdc4acb2640 1 -- 172.21.15.40:0/4194101562 wait complete. 2024-03-18T22:59:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.416+0000 7fdc4acb2640 1 Processor -- start 2024-03-18T22:59:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.416+0000 7fdc4acb2640 1 -- start start 2024-03-18T22:59:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.416+0000 7fdc4acb2640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc44101850 0x7fdc44197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.416+0000 7fdc4acb2640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc44102a40 0x7fdc44197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.416+0000 7fdc4acb2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc44103400 0x7fdc4419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.416+0000 7fdc4acb2640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4410d380 con 0x7fdc44103400 2024-03-18T22:59:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.416+0000 7fdc4acb2640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4410d200 con 0x7fdc44101850 2024-03-18T22:59:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.416+0000 7fdc4acb2640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4410d500 con 0x7fdc44102a40 2024-03-18T22:59:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.416+0000 7fdc4924f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc44103400 0x7fdc4419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.416+0000 7fdc48a4e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc44101850 0x7fdc44197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.416+0000 7fdc4924f640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc44103400 0x7fdc4419f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60712/0 (socket says 172.21.15.40:60712) 2024-03-18T22:59:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.416+0000 7fdc4924f640 1 -- 172.21.15.40:0/3783959787 learned_addr learned my addr 172.21.15.40:0/3783959787 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.416+0000 7fdc43fff640 1 --2- 172.21.15.40:0/3783959787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc44102a40 0x7fdc44197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.417+0000 7fdc4924f640 1 -- 172.21.15.40:0/3783959787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc44102a40 msgr2=0x7fdc44197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.417+0000 7fdc4924f640 1 --2- 172.21.15.40:0/3783959787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc44102a40 0x7fdc44197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.417+0000 7fdc4924f640 1 -- 172.21.15.40:0/3783959787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc44101850 msgr2=0x7fdc44197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.417+0000 7fdc4924f640 1 --2- 172.21.15.40:0/3783959787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc44101850 0x7fdc44197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.417+0000 7fdc4924f640 1 -- 172.21.15.40:0/3783959787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc4419f680 con 0x7fdc44103400 2024-03-18T22:59:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.417+0000 7fdc4924f640 1 --2- 172.21.15.40:0/3783959787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc44103400 0x7fdc4419f010 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c02c700 tx=0x7fdc3c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.417+0000 7fdc41ffb640 1 -- 172.21.15.40:0/3783959787 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c041020 con 0x7fdc44103400 2024-03-18T22:59:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.417+0000 7fdc41ffb640 1 -- 172.21.15.40:0/3783959787 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc3c003e20 con 0x7fdc44103400 2024-03-18T22:59:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.417+0000 7fdc4acb2640 1 -- 172.21.15.40:0/3783959787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc4419f910 con 0x7fdc44103400 2024-03-18T22:59:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.417+0000 7fdc41ffb640 1 -- 172.21.15.40:0/3783959787 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c039040 con 0x7fdc44103400 2024-03-18T22:59:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.417+0000 7fdc4acb2640 1 -- 172.21.15.40:0/3783959787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc4419fe50 con 0x7fdc44103400 2024-03-18T22:59:19.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.418+0000 7fdc4acb2640 1 -- 172.21.15.40:0/3783959787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc44108b90 con 0x7fdc44103400 2024-03-18T22:59:19.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.419+0000 7fdc41ffb640 1 -- 172.21.15.40:0/3783959787 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdc3c044020 con 0x7fdc44103400 2024-03-18T22:59:19.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.420+0000 7fdc41ffb640 1 --2- 172.21.15.40:0/3783959787 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc340a2880 0x7fdc340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:19.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.420+0000 7fdc48a4e640 1 --2- 172.21.15.40:0/3783959787 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc340a2880 0x7fdc340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:19.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.420+0000 7fdc41ffb640 1 -- 172.21.15.40:0/3783959787 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fdc3c0f13b0 con 0x7fdc44103400 2024-03-18T22:59:19.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.420+0000 7fdc48a4e640 1 --2- 172.21.15.40:0/3783959787 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc340a2880 0x7fdc340a4d40 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fdc441028a0 tx=0x7fdc38005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:19.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.423+0000 7fdc41ffb640 1 -- 172.21.15.40:0/3783959787 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdc3c04b020 con 0x7fdc44103400 2024-03-18T22:59:19.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.606+0000 7fdc4acb2640 1 -- 172.21.15.40:0/3783959787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fdc4402d1f0 con 0x7fdc44103400 2024-03-18T22:59:19.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.606+0000 7fdc41ffb640 1 -- 172.21.15.40:0/3783959787 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fdc3c0be000 con 0x7fdc44103400 2024-03-18T22:59:19.607 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-18T22:59:19.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.608+0000 7fdc4acb2640 1 -- 172.21.15.40:0/3783959787 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc340a2880 msgr2=0x7fdc340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:19.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.608+0000 7fdc4acb2640 1 --2- 172.21.15.40:0/3783959787 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc340a2880 0x7fdc340a4d40 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fdc441028a0 tx=0x7fdc38005f90 comp rx=0 tx=0).stop 2024-03-18T22:59:19.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.608+0000 7fdc4acb2640 1 -- 172.21.15.40:0/3783959787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc44103400 msgr2=0x7fdc4419f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:19.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.608+0000 7fdc4acb2640 1 --2- 172.21.15.40:0/3783959787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc44103400 0x7fdc4419f010 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c02c700 tx=0x7fdc3c0328a0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.608+0000 7fdc4acb2640 1 -- 172.21.15.40:0/3783959787 shutdown_connections 2024-03-18T22:59:19.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.608+0000 7fdc4acb2640 1 --2- 172.21.15.40:0/3783959787 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdc340a2880 0x7fdc340a4d40 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.608+0000 7fdc4acb2640 1 --2- 172.21.15.40:0/3783959787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdc44103400 0x7fdc4419f010 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.608+0000 7fdc4acb2640 1 --2- 172.21.15.40:0/3783959787 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdc44102a40 0x7fdc44197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.608+0000 7fdc4acb2640 1 --2- 172.21.15.40:0/3783959787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdc44101850 0x7fdc44197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.608+0000 7fdc4acb2640 1 -- 172.21.15.40:0/3783959787 >> 172.21.15.40:0/3783959787 conn(0x7fdc440fcdc0 msgr2=0x7fdc44105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:19.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.609+0000 7fdc4acb2640 1 -- 172.21.15.40:0/3783959787 shutdown_connections 2024-03-18T22:59:19.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.609+0000 7fdc4acb2640 1 -- 172.21.15.40:0/3783959787 wait complete. 2024-03-18T22:59:19.620 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T22:59:19.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.723+0000 7ff193f77640 1 -- 172.21.15.40:0/3665014724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18c1022d0 msgr2=0x7ff18c108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.723+0000 7ff193f77640 1 --2- 172.21.15.40:0/3665014724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18c1022d0 0x7ff18c108b60 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7ff184009900 tx=0x7ff18402bc30 comp rx=0 tx=0).stop 2024-03-18T22:59:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.723+0000 7ff193f77640 1 -- 172.21.15.40:0/3665014724 shutdown_connections 2024-03-18T22:59:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.723+0000 7ff193f77640 1 --2- 172.21.15.40:0/3665014724 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18c1022d0 0x7ff18c108b60 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.723+0000 7ff193f77640 1 --2- 172.21.15.40:0/3665014724 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff18c101910 0x7ff18c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.723+0000 7ff193f77640 1 --2- 172.21.15.40:0/3665014724 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff18c100720 0x7ff18c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.723+0000 7ff193f77640 1 -- 172.21.15.40:0/3665014724 >> 172.21.15.40:0/3665014724 conn(0x7ff18c0fbcb0 msgr2=0x7ff18c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.724+0000 7ff193f77640 1 -- 172.21.15.40:0/3665014724 shutdown_connections 2024-03-18T22:59:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.724+0000 7ff193f77640 1 -- 172.21.15.40:0/3665014724 wait complete. 2024-03-18T22:59:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.724+0000 7ff193f77640 1 Processor -- start 2024-03-18T22:59:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.724+0000 7ff193f77640 1 -- start start 2024-03-18T22:59:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.724+0000 7ff193f77640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18c100720 0x7ff18c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.724+0000 7ff193f77640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff18c101910 0x7ff18c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.724+0000 7ff193f77640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff18c1022d0 0x7ff18c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.724+0000 7ff193f77640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c10c250 con 0x7ff18c100720 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.724+0000 7ff193f77640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c10c0d0 con 0x7ff18c1022d0 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.724+0000 7ff193f77640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c10c3d0 con 0x7ff18c101910 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.724+0000 7ff191d13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18c100720 0x7ff18c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.724+0000 7ff191d13640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18c100720 0x7ff18c197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60724/0 (socket says 172.21.15.40:60724) 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.724+0000 7ff191d13640 1 -- 172.21.15.40:0/714506900 learned_addr learned my addr 172.21.15.40:0/714506900 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.724+0000 7ff191512640 1 --2- 172.21.15.40:0/714506900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff18c101910 0x7ff18c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.724+0000 7ff192514640 1 --2- 172.21.15.40:0/714506900 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff18c1022d0 0x7ff18c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.725+0000 7ff191d13640 1 -- 172.21.15.40:0/714506900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff18c101910 msgr2=0x7ff18c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.725+0000 7ff191d13640 1 --2- 172.21.15.40:0/714506900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff18c101910 0x7ff18c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.725+0000 7ff191d13640 1 -- 172.21.15.40:0/714506900 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff18c1022d0 msgr2=0x7ff18c19f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.725+0000 7ff191d13640 1 --2- 172.21.15.40:0/714506900 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff18c1022d0 0x7ff18c19f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.725+0000 7ff191d13640 1 -- 172.21.15.40:0/714506900 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff18c19f6b0 con 0x7ff18c100720 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.725+0000 7ff191d13640 1 --2- 172.21.15.40:0/714506900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18c100720 0x7ff18c197ae0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff17c00e9c0 tx=0x7ff17c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.725+0000 7ff18affd640 1 -- 172.21.15.40:0/714506900 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17c00cd80 con 0x7ff18c100720 2024-03-18T22:59:19.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.725+0000 7ff193f77640 1 -- 172.21.15.40:0/714506900 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff18c19f9a0 con 0x7ff18c100720 2024-03-18T22:59:19.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.725+0000 7ff18affd640 1 -- 172.21.15.40:0/714506900 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff17c0035f0 con 0x7ff18c100720 2024-03-18T22:59:19.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.725+0000 7ff18affd640 1 -- 172.21.15.40:0/714506900 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17c010690 con 0x7ff18c100720 2024-03-18T22:59:19.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.725+0000 7ff193f77640 1 -- 172.21.15.40:0/714506900 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff18c19fea0 con 0x7ff18c100720 2024-03-18T22:59:19.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.726+0000 7ff193f77640 1 -- 172.21.15.40:0/714506900 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff18c1a00d0 con 0x7ff18c100720 2024-03-18T22:59:19.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.727+0000 7ff18affd640 1 -- 172.21.15.40:0/714506900 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff17c0037a0 con 0x7ff18c100720 2024-03-18T22:59:19.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.728+0000 7ff18affd640 1 --2- 172.21.15.40:0/714506900 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1740a2800 0x7ff1740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:19.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.728+0000 7ff191512640 1 --2- 172.21.15.40:0/714506900 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1740a2800 0x7ff1740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:19.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.728+0000 7ff18affd640 1 -- 172.21.15.40:0/714506900 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ff17c0cf4c0 con 0x7ff18c100720 2024-03-18T22:59:19.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.728+0000 7ff191512640 1 --2- 172.21.15.40:0/714506900 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1740a2800 0x7ff1740a4cc0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff18c198fa0 tx=0x7ff180009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:19.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.731+0000 7ff18affd640 1 -- 172.21.15.40:0/714506900 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff17c09c170 con 0x7ff18c100720 2024-03-18T22:59:19.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:19 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/484698356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:59:19.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:19 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1840666923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:59:19.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:19 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1398164947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:59:19.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:19 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/484698356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:59:19.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:19 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1840666923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:59:19.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:19 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1398164947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:59:19.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:19 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/484698356' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T22:59:19.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:19 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1840666923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T22:59:19.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:19 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1398164947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T22:59:19.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.926+0000 7ff193f77640 1 -- 172.21.15.40:0/714506900 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff18c1a02a0 con 0x7ff18c100720 2024-03-18T22:59:19.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.927+0000 7ff18affd640 1 -- 172.21.15.40:0/714506900 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v93) ==== 81+0+591 (secure 0 0 0) 0x7ff17c09ba70 con 0x7ff18c100720 2024-03-18T22:59:19.927 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/347668672 2024-03-19T22:33:42.721571+0000 2024-03-18T22:59:19.927 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/4237962144 2024-03-19T22:33:42.721571+0000 2024-03-18T22:59:19.927 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/1454961757 2024-03-19T22:33:15.826478+0000 2024-03-18T22:59:19.927 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/3047621005 2024-03-19T22:33:15.826478+0000 2024-03-18T22:59:19.927 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/2121391701 2024-03-19T22:33:15.826478+0000 2024-03-18T22:59:19.928 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/3876208039 2024-03-19T22:33:42.721571+0000 2024-03-18T22:59:19.928 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/3876208039 2024-03-19T22:33:42.721571+0000 2024-03-18T22:59:19.928 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:0/2591529486 2024-03-19T22:33:42.721571+0000 2024-03-18T22:59:19.928 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6800/1153335782 2024-03-19T22:33:15.826478+0000 2024-03-18T22:59:19.928 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40:6801/1153335782 2024-03-19T22:33:15.826478+0000 2024-03-18T22:59:19.928 INFO:teuthology.orchestra.run.smithi040.stderr:listed 10 entries 2024-03-18T22:59:19.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.929+0000 7ff193f77640 1 -- 172.21.15.40:0/714506900 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1740a2800 msgr2=0x7ff1740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:19.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.929+0000 7ff193f77640 1 --2- 172.21.15.40:0/714506900 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1740a2800 0x7ff1740a4cc0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff18c198fa0 tx=0x7ff180009500 comp rx=0 tx=0).stop 2024-03-18T22:59:19.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.929+0000 7ff193f77640 1 -- 172.21.15.40:0/714506900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18c100720 msgr2=0x7ff18c197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:19.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.929+0000 7ff193f77640 1 --2- 172.21.15.40:0/714506900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18c100720 0x7ff18c197ae0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7ff17c00e9c0 tx=0x7ff17c00ee80 comp rx=0 tx=0).stop 2024-03-18T22:59:19.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.929+0000 7ff193f77640 1 -- 172.21.15.40:0/714506900 shutdown_connections 2024-03-18T22:59:19.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.929+0000 7ff193f77640 1 --2- 172.21.15.40:0/714506900 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7ff1740a2800 0x7ff1740a4cc0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.929+0000 7ff193f77640 1 --2- 172.21.15.40:0/714506900 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff18c1022d0 0x7ff18c19f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.929+0000 7ff193f77640 1 --2- 172.21.15.40:0/714506900 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7ff18c101910 0x7ff18c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.929+0000 7ff193f77640 1 --2- 172.21.15.40:0/714506900 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18c100720 0x7ff18c197ae0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:19.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.929+0000 7ff193f77640 1 -- 172.21.15.40:0/714506900 >> 172.21.15.40:0/714506900 conn(0x7ff18c0fbcb0 msgr2=0x7ff18c0fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:19.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.929+0000 7ff193f77640 1 -- 172.21.15.40:0/714506900 shutdown_connections 2024-03-18T22:59:19.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:19.929+0000 7ff193f77640 1 -- 172.21.15.40:0/714506900 wait complete. 2024-03-18T22:59:19.940 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi040.front.sepia.ceph.com... 2024-03-18T22:59:19.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:59:19.940 DEBUG:teuthology.orchestra.run.smithi040:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-18T22:59:19.991 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T22:59:19.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[113620]: fuse finished with error 0 and tester_r 0 2024-03-18T22:59:20.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:20 smithi113 ceph-mon[66775]: pgmap v851: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:59:20.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:20 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3783959787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:59:20.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:20 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/714506900' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T22:59:20.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:20 smithi103 ceph-mon[66377]: pgmap v851: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:59:20.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:20 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3783959787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:59:20.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:20 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/714506900' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T22:59:20.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:20 smithi040 ceph-mon[62643]: pgmap v851: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T22:59:20.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:20 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3783959787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T22:59:20.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:20 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/714506900' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T22:59:22.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:22 smithi113 ceph-mon[66775]: pgmap v852: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-18T22:59:22.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:22 smithi103 ceph-mon[66377]: pgmap v852: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-18T22:59:22.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:22 smithi040 ceph-mon[62643]: pgmap v852: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-18T22:59:24.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:24 smithi113 ceph-mon[66775]: pgmap v853: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T22:59:24.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:24 smithi103 ceph-mon[66377]: pgmap v853: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T22:59:24.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:24 smithi040 ceph-mon[62643]: pgmap v853: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T22:59:26.093 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-18T22:59:26.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:59:26.094 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T22:59:26.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:59:26.112 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-18T22:59:26.224 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T22:59:26.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:59:26.225 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T22:59:26.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T22:59:26.255 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link delete ceph-brx 2024-03-18T22:59:26.349 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-18T22:59:26.360 INFO:tasks.cephadm:Teardown begin 2024-03-18T22:59:26.360 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T22:59:26.360 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:59:26.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.464+0000 7f29a3636640 1 -- 172.21.15.40:0/2985359819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c1022f0 msgr2=0x7f299c108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:26.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.464+0000 7f29a3636640 1 --2- 172.21.15.40:0/2985359819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c1022f0 0x7f299c108b80 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f2994009ca0 tx=0x7f299402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:59:26.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.464+0000 7f29a3636640 1 -- 172.21.15.40:0/2985359819 shutdown_connections 2024-03-18T22:59:26.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.464+0000 7f29a3636640 1 --2- 172.21.15.40:0/2985359819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c1022f0 0x7f299c108b80 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:26.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.464+0000 7f29a3636640 1 --2- 172.21.15.40:0/2985359819 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299c101930 0x7f299c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:26.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.464+0000 7f29a3636640 1 --2- 172.21.15.40:0/2985359819 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f299c100740 0x7f299c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:26.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.464+0000 7f29a3636640 1 -- 172.21.15.40:0/2985359819 >> 172.21.15.40:0/2985359819 conn(0x7f299c0fbcb0 msgr2=0x7f299c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.465+0000 7f29a3636640 1 -- 172.21.15.40:0/2985359819 shutdown_connections 2024-03-18T22:59:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.465+0000 7f29a3636640 1 -- 172.21.15.40:0/2985359819 wait complete. 2024-03-18T22:59:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.465+0000 7f29a3636640 1 Processor -- start 2024-03-18T22:59:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.465+0000 7f29a3636640 1 -- start start 2024-03-18T22:59:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.465+0000 7f29a3636640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c100740 0x7f299c197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.465+0000 7f29a3636640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299c101930 0x7f299c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.465+0000 7f29a3636640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f299c1022f0 0x7f299c19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.465+0000 7f29a3636640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f299c10d350 con 0x7f299c100740 2024-03-18T22:59:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.465+0000 7f29a3636640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f299c10d1d0 con 0x7f299c101930 2024-03-18T22:59:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.465+0000 7f29a3636640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f299c10d4d0 con 0x7f299c1022f0 2024-03-18T22:59:26.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.465+0000 7f29a13d2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c100740 0x7f299c197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.465+0000 7f29a13d2640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c100740 0x7f299c197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60738/0 (socket says 172.21.15.40:60738) 2024-03-18T22:59:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.465+0000 7f29a13d2640 1 -- 172.21.15.40:0/698204001 learned_addr learned my addr 172.21.15.40:0/698204001 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.465+0000 7f29a0bd1640 1 --2- 172.21.15.40:0/698204001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299c101930 0x7f299c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.465+0000 7f29a1bd3640 1 --2- 172.21.15.40:0/698204001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f299c1022f0 0x7f299c19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.466+0000 7f29a13d2640 1 -- 172.21.15.40:0/698204001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f299c1022f0 msgr2=0x7f299c19efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.466+0000 7f29a13d2640 1 --2- 172.21.15.40:0/698204001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f299c1022f0 0x7f299c19efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.466+0000 7f29a13d2640 1 -- 172.21.15.40:0/698204001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299c101930 msgr2=0x7f299c197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.466+0000 7f29a13d2640 1 --2- 172.21.15.40:0/698204001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299c101930 0x7f299c197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.466+0000 7f29a13d2640 1 -- 172.21.15.40:0/698204001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299c19f640 con 0x7f299c100740 2024-03-18T22:59:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.466+0000 7f29a13d2640 1 --2- 172.21.15.40:0/698204001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c100740 0x7f299c197a20 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f299800e9c0 tx=0x7f299800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.466+0000 7f29927fc640 1 -- 172.21.15.40:0/698204001 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299800cd80 con 0x7f299c100740 2024-03-18T22:59:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.466+0000 7f29927fc640 1 -- 172.21.15.40:0/698204001 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29980035f0 con 0x7f299c100740 2024-03-18T22:59:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.466+0000 7f29927fc640 1 -- 172.21.15.40:0/698204001 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2998010690 con 0x7f299c100740 2024-03-18T22:59:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.466+0000 7f29a3636640 1 -- 172.21.15.40:0/698204001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299c19f930 con 0x7f299c100740 2024-03-18T22:59:26.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.466+0000 7f29a3636640 1 -- 172.21.15.40:0/698204001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f299c19fe30 con 0x7f299c100740 2024-03-18T22:59:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.467+0000 7f29a3636640 1 -- 172.21.15.40:0/698204001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f299c068a60 con 0x7f299c100740 2024-03-18T22:59:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.468+0000 7f29927fc640 1 -- 172.21.15.40:0/698204001 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f29980037a0 con 0x7f299c100740 2024-03-18T22:59:26.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.469+0000 7f29927fc640 1 --2- 172.21.15.40:0/698204001 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f29880a2800 0x7f29880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:26.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.469+0000 7f29927fc640 1 -- 172.21.15.40:0/698204001 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f29980cf350 con 0x7f299c100740 2024-03-18T22:59:26.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.469+0000 7f29a0bd1640 1 --2- 172.21.15.40:0/698204001 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f29880a2800 0x7f29880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:26.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.470+0000 7f29a0bd1640 1 --2- 172.21.15.40:0/698204001 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f29880a2800 0x7f29880a4cc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f299c198f30 tx=0x7f298c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:26.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.473+0000 7f29927fc640 1 -- 172.21.15.40:0/698204001 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f299809bfa0 con 0x7f299c100740 2024-03-18T22:59:26.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.696+0000 7f29a3636640 1 -- 172.21.15.40:0/698204001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f299c107b00 con 0x7f299c100740 2024-03-18T22:59:26.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.696+0000 7f29927fc640 1 -- 172.21.15.40:0/698204001 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11970 (secure 0 0 0) 0x7f299809b8a0 con 0x7f299c100740 2024-03-18T22:59:26.698 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:59:26.698 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:58:01.473332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24509,24481,24473,24388,24401]},"id":1}]} 2024-03-18T22:59:26.698 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 49 2024-03-18T22:59:26.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.699+0000 7f29a3636640 1 -- 172.21.15.40:0/698204001 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f29880a2800 msgr2=0x7f29880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:26.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.699+0000 7f29a3636640 1 --2- 172.21.15.40:0/698204001 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f29880a2800 0x7f29880a4cc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f299c198f30 tx=0x7f298c005f90 comp rx=0 tx=0).stop 2024-03-18T22:59:26.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.699+0000 7f29a3636640 1 -- 172.21.15.40:0/698204001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c100740 msgr2=0x7f299c197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:26.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.699+0000 7f29a3636640 1 --2- 172.21.15.40:0/698204001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c100740 0x7f299c197a20 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f299800e9c0 tx=0x7f299800ee80 comp rx=0 tx=0).stop 2024-03-18T22:59:26.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.699+0000 7f29a3636640 1 -- 172.21.15.40:0/698204001 shutdown_connections 2024-03-18T22:59:26.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.699+0000 7f29a3636640 1 --2- 172.21.15.40:0/698204001 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f29880a2800 0x7f29880a4cc0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:26.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.699+0000 7f29a3636640 1 --2- 172.21.15.40:0/698204001 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f299c1022f0 0x7f299c19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:26.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.699+0000 7f29a3636640 1 --2- 172.21.15.40:0/698204001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f299c101930 0x7f299c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:26.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.699+0000 7f29a3636640 1 --2- 172.21.15.40:0/698204001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f299c100740 0x7f299c197a20 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:26.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.699+0000 7f29a3636640 1 -- 172.21.15.40:0/698204001 >> 172.21.15.40:0/698204001 conn(0x7f299c0fbcb0 msgr2=0x7f299c104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:26.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.699+0000 7f29a3636640 1 -- 172.21.15.40:0/698204001 shutdown_connections 2024-03-18T22:59:26.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.699+0000 7f29a3636640 1 -- 172.21.15.40:0/698204001 wait complete. 2024-03-18T22:59:26.711 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T22:59:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.813+0000 7f1e2d722640 1 -- 172.21.15.40:0/3311224195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e28101930 msgr2=0x7f1e28101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.813+0000 7f1e2d722640 1 --2- 172.21.15.40:0/3311224195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e28101930 0x7f1e28101db0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009900 tx=0x7f1e1002bc30 comp rx=0 tx=0).stop 2024-03-18T22:59:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.813+0000 7f1e2d722640 1 -- 172.21.15.40:0/3311224195 shutdown_connections 2024-03-18T22:59:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.813+0000 7f1e2d722640 1 --2- 172.21.15.40:0/3311224195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e281022f0 0x7f1e28108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.813+0000 7f1e2d722640 1 --2- 172.21.15.40:0/3311224195 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e28101930 0x7f1e28101db0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.813+0000 7f1e2d722640 1 --2- 172.21.15.40:0/3311224195 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e28100740 0x7f1e28100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.813+0000 7f1e2d722640 1 -- 172.21.15.40:0/3311224195 >> 172.21.15.40:0/3311224195 conn(0x7f1e280fbcb0 msgr2=0x7f1e280fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.813+0000 7f1e2d722640 1 -- 172.21.15.40:0/3311224195 shutdown_connections 2024-03-18T22:59:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.813+0000 7f1e2d722640 1 -- 172.21.15.40:0/3311224195 wait complete. 2024-03-18T22:59:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.814+0000 7f1e2d722640 1 Processor -- start 2024-03-18T22:59:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.814+0000 7f1e2d722640 1 -- start start 2024-03-18T22:59:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.814+0000 7f1e2d722640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e28100740 0x7f1e281046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.814+0000 7f1e2d722640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e28101930 0x7f1e28102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.814+0000 7f1e2d722640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e281022f0 0x7f1e28103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:26.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.814+0000 7f1e2d722640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2810c250 con 0x7f1e28101930 2024-03-18T22:59:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.814+0000 7f1e2d722640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2810c0d0 con 0x7f1e281022f0 2024-03-18T22:59:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.814+0000 7f1e2d722640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1e2810c3d0 con 0x7f1e28100740 2024-03-18T22:59:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.814+0000 7f1e267fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e28101930 0x7f1e28102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.814+0000 7f1e267fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e28101930 0x7f1e28102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60748/0 (socket says 172.21.15.40:60748) 2024-03-18T22:59:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.814+0000 7f1e267fc640 1 -- 172.21.15.40:0/1456255816 learned_addr learned my addr 172.21.15.40:0/1456255816 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.814+0000 7f1e277fe640 1 --2- 172.21.15.40:0/1456255816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e281022f0 0x7f1e28103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.814+0000 7f1e26ffd640 1 --2- 172.21.15.40:0/1456255816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e28100740 0x7f1e281046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.815+0000 7f1e267fc640 1 -- 172.21.15.40:0/1456255816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e28100740 msgr2=0x7f1e281046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.815+0000 7f1e267fc640 1 --2- 172.21.15.40:0/1456255816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e28100740 0x7f1e281046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.815+0000 7f1e267fc640 1 -- 172.21.15.40:0/1456255816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e281022f0 msgr2=0x7f1e28103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.815+0000 7f1e267fc640 1 --2- 172.21.15.40:0/1456255816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e281022f0 0x7f1e28103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.815+0000 7f1e267fc640 1 -- 172.21.15.40:0/1456255816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e28103ae0 con 0x7f1e28101930 2024-03-18T22:59:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.815+0000 7f1e267fc640 1 --2- 172.21.15.40:0/1456255816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e28101930 0x7f1e28102d10 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f1e10006010 tx=0x7f1e10005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.815+0000 7f1e1ffff640 1 -- 172.21.15.40:0/1456255816 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e10043020 con 0x7f1e28101930 2024-03-18T22:59:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.815+0000 7f1e1ffff640 1 -- 172.21.15.40:0/1456255816 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1e100043d0 con 0x7f1e28101930 2024-03-18T22:59:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.815+0000 7f1e2d722640 1 -- 172.21.15.40:0/1456255816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e281a1bf0 con 0x7f1e28101930 2024-03-18T22:59:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.815+0000 7f1e1ffff640 1 -- 172.21.15.40:0/1456255816 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e10003b80 con 0x7f1e28101930 2024-03-18T22:59:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.815+0000 7f1e2d722640 1 -- 172.21.15.40:0/1456255816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e281a2140 con 0x7f1e28101930 2024-03-18T22:59:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.816+0000 7f1e2d722640 1 -- 172.21.15.40:0/1456255816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e281a2310 con 0x7f1e28101930 2024-03-18T22:59:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.817+0000 7f1e1ffff640 1 -- 172.21.15.40:0/1456255816 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1e10046020 con 0x7f1e28101930 2024-03-18T22:59:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.818+0000 7f1e1ffff640 1 --2- 172.21.15.40:0/1456255816 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1e140a2800 0x7f1e140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.818+0000 7f1e26ffd640 1 --2- 172.21.15.40:0/1456255816 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1e140a2800 0x7f1e140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.818+0000 7f1e1ffff640 1 -- 172.21.15.40:0/1456255816 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f1e100f1650 con 0x7f1e28101930 2024-03-18T22:59:26.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.818+0000 7f1e26ffd640 1 --2- 172.21.15.40:0/1456255816 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1e140a2800 0x7f1e140a4cc0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1e28101790 tx=0x7f1e20005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:26.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:26.822+0000 7f1e1ffff640 1 -- 172.21.15.40:0/1456255816 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1e1004d020 con 0x7f1e28101930 2024-03-18T22:59:26.858 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:26 smithi103 ceph-mon[66377]: pgmap v854: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-03-18T22:59:26.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:26 smithi040 ceph-mon[62643]: pgmap v854: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-03-18T22:59:27.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.045+0000 7f1e2d722640 1 -- 172.21.15.40:0/1456255816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e281a26a0 con 0x7f1e28101930 2024-03-18T22:59:27.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.046+0000 7f1e1ffff640 1 -- 172.21.15.40:0/1456255816 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11970 (secure 0 0 0) 0x7f1e281a26a0 con 0x7f1e28101930 2024-03-18T22:59:27.047 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:59:27.048 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14517,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/823346277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":823346277},{"type":"v1","addr":"172.21.15.40:6835","nonce":823346277}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14529,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6837/3562391432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6836","nonce":3562391432},{"type":"v1","addr":"172.21.15.40:6837","nonce":3562391432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T22:40:41.572773+0000","modified":"2024-03-18T22:58:01.473332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24509,"mds_1":24481,"mds_2":24473,"mds_3":24401,"mds_4":24388},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14592":{"gid":14592,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6839/4264194754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6838","nonce":4264194754},{"type":"v1","addr":"172.21.15.40:6839","nonce":4264194754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14604":{"gid":14604,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.40:6841/1807327423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6840","nonce":1807327423},{"type":"v1","addr":"172.21.15.40:6841","nonce":1807327423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24388":{"gid":24388,"name":"l","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.113:6833/2086888629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6832","nonce":2086888629},{"type":"v1","addr":"172.21.15.113:6833","nonce":2086888629}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.103:6833/3761932784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3761932784},{"type":"v1","addr":"172.21.15.103:6833","nonce":3761932784}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/2536694691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":2536694691},{"type":"v1","addr":"172.21.15.103:6835","nonce":2536694691}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6835/4040610039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6834","nonce":4040610039},{"type":"v1","addr":"172.21.15.113:6835","nonce":4040610039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.113:6837/4212429112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6836","nonce":4212429112},{"type":"v1","addr":"172.21.15.113:6837","nonce":4212429112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.103:6837/1713853423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1713853423},{"type":"v1","addr":"172.21.15.103:6837","nonce":1713853423}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.113:6839/2986827418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6838","nonce":2986827418},{"type":"v1","addr":"172.21.15.113:6839","nonce":2986827418}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/2609728202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2609728202},{"type":"v1","addr":"172.21.15.103:6839","nonce":2609728202}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24509,24481,24473,24388,24401]},"id":1}]} 2024-03-18T22:59:27.048 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 49 2024-03-18T22:59:27.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.048+0000 7f1e2d722640 1 -- 172.21.15.40:0/1456255816 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1e140a2800 msgr2=0x7f1e140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:27.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.048+0000 7f1e2d722640 1 --2- 172.21.15.40:0/1456255816 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1e140a2800 0x7f1e140a4cc0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1e28101790 tx=0x7f1e20005f90 comp rx=0 tx=0).stop 2024-03-18T22:59:27.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.048+0000 7f1e2d722640 1 -- 172.21.15.40:0/1456255816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e28101930 msgr2=0x7f1e28102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:27.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.048+0000 7f1e2d722640 1 --2- 172.21.15.40:0/1456255816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e28101930 0x7f1e28102d10 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f1e10006010 tx=0x7f1e10005e40 comp rx=0 tx=0).stop 2024-03-18T22:59:27.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.048+0000 7f1e2d722640 1 -- 172.21.15.40:0/1456255816 shutdown_connections 2024-03-18T22:59:27.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.048+0000 7f1e2d722640 1 --2- 172.21.15.40:0/1456255816 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1e140a2800 0x7f1e140a4cc0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.048+0000 7f1e2d722640 1 --2- 172.21.15.40:0/1456255816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e281022f0 0x7f1e28103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.048+0000 7f1e2d722640 1 --2- 172.21.15.40:0/1456255816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e28101930 0x7f1e28102d10 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.048+0000 7f1e2d722640 1 --2- 172.21.15.40:0/1456255816 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1e28100740 0x7f1e281046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.048+0000 7f1e2d722640 1 -- 172.21.15.40:0/1456255816 >> 172.21.15.40:0/1456255816 conn(0x7f1e280fbcb0 msgr2=0x7f1e28068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:27.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.048+0000 7f1e2d722640 1 -- 172.21.15.40:0/1456255816 shutdown_connections 2024-03-18T22:59:27.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.048+0000 7f1e2d722640 1 -- 172.21.15.40:0/1456255816 wait complete. 2024-03-18T22:59:27.061 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T22:59:27.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:26 smithi113 ceph-mon[66775]: pgmap v854: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-03-18T22:59:27.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.163+0000 7f5a77ae8640 1 -- 172.21.15.40:0/2033755500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a70101d00 msgr2=0x7f5a70102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:27.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.163+0000 7f5a77ae8640 1 --2- 172.21.15.40:0/2033755500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a70101d00 0x7f5a70102120 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009860 tx=0x7f5a6c02bb70 comp rx=0 tx=0).stop 2024-03-18T22:59:27.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.163+0000 7f5a77ae8640 1 -- 172.21.15.40:0/2033755500 shutdown_connections 2024-03-18T22:59:27.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.163+0000 7f5a77ae8640 1 --2- 172.21.15.40:0/2033755500 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a701013f0 0x7f5a70108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.163+0000 7f5a77ae8640 1 --2- 172.21.15.40:0/2033755500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a70100a30 0x7f5a70100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.163+0000 7f5a77ae8640 1 --2- 172.21.15.40:0/2033755500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a70101d00 0x7f5a70102120 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.163+0000 7f5a77ae8640 1 -- 172.21.15.40:0/2033755500 >> 172.21.15.40:0/2033755500 conn(0x7f5a700fb460 msgr2=0x7f5a700fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:27.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.163+0000 7f5a77ae8640 1 -- 172.21.15.40:0/2033755500 shutdown_connections 2024-03-18T22:59:27.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.163+0000 7f5a77ae8640 1 -- 172.21.15.40:0/2033755500 wait complete. 2024-03-18T22:59:27.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a77ae8640 1 Processor -- start 2024-03-18T22:59:27.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a77ae8640 1 -- start start 2024-03-18T22:59:27.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a77ae8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a70100a30 0x7f5a70197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:27.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a77ae8640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a701013f0 0x7f5a70197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:27.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a77ae8640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a70101d00 0x7f5a7019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:27.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a77ae8640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7010c2a0 con 0x7f5a70101d00 2024-03-18T22:59:27.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a77ae8640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7010c120 con 0x7f5a70100a30 2024-03-18T22:59:27.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a77ae8640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7010c420 con 0x7f5a701013f0 2024-03-18T22:59:27.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a75884640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a70100a30 0x7f5a70197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:27.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a75083640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a701013f0 0x7f5a70197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:27.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a75884640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a70100a30 0x7f5a70197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:33316/0 (socket says 172.21.15.40:33316) 2024-03-18T22:59:27.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a76085640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a70101d00 0x7f5a7019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:27.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a75884640 1 -- 172.21.15.40:0/3337583973 learned_addr learned my addr 172.21.15.40:0/3337583973 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:27.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a75884640 1 -- 172.21.15.40:0/3337583973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a701013f0 msgr2=0x7f5a70197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:27.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a75884640 1 --2- 172.21.15.40:0/3337583973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a701013f0 0x7f5a70197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a75884640 1 -- 172.21.15.40:0/3337583973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a70101d00 msgr2=0x7f5a7019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:27.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a75884640 1 --2- 172.21.15.40:0/3337583973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a70101d00 0x7f5a7019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a75884640 1 -- 172.21.15.40:0/3337583973 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7019f6c0 con 0x7f5a70100a30 2024-03-18T22:59:27.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a75083640 1 --2- 172.21.15.40:0/3337583973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a701013f0 0x7f5a70197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:59:27.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.164+0000 7f5a76085640 1 --2- 172.21.15.40:0/3337583973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a70101d00 0x7f5a7019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:59:27.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.165+0000 7f5a75884640 1 --2- 172.21.15.40:0/3337583973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a70100a30 0x7f5a70197aa0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c005940 tx=0x7f5a6c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:27.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.165+0000 7f5a66ffd640 1 -- 172.21.15.40:0/3337583973 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c041020 con 0x7f5a70100a30 2024-03-18T22:59:27.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.165+0000 7f5a66ffd640 1 -- 172.21.15.40:0/3337583973 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a6c0035f0 con 0x7f5a70100a30 2024-03-18T22:59:27.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.165+0000 7f5a77ae8640 1 -- 172.21.15.40:0/3337583973 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a7019f950 con 0x7f5a70100a30 2024-03-18T22:59:27.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.165+0000 7f5a66ffd640 1 -- 172.21.15.40:0/3337583973 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c039040 con 0x7f5a70100a30 2024-03-18T22:59:27.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.165+0000 7f5a77ae8640 1 -- 172.21.15.40:0/3337583973 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a7019fe40 con 0x7f5a70100a30 2024-03-18T22:59:27.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.166+0000 7f5a77ae8640 1 -- 172.21.15.40:0/3337583973 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a701a0010 con 0x7f5a70100a30 2024-03-18T22:59:27.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.167+0000 7f5a66ffd640 1 -- 172.21.15.40:0/3337583973 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5a6c044020 con 0x7f5a70100a30 2024-03-18T22:59:27.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.167+0000 7f5a66ffd640 1 --2- 172.21.15.40:0/3337583973 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5a5c0a2800 0x7f5a5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:27.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.168+0000 7f5a75083640 1 --2- 172.21.15.40:0/3337583973 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5a5c0a2800 0x7f5a5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:27.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.168+0000 7f5a66ffd640 1 -- 172.21.15.40:0/3337583973 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f5a6c0f0bd0 con 0x7f5a70100a30 2024-03-18T22:59:27.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.168+0000 7f5a75083640 1 --2- 172.21.15.40:0/3337583973 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5a5c0a2800 0x7f5a5c0a4cc0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5a70198fb0 tx=0x7f5a60009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:27.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.175+0000 7f5a66ffd640 1 -- 172.21.15.40:0/3337583973 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5a6c04b020 con 0x7f5a70100a30 2024-03-18T22:59:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.363+0000 7f5a77ae8640 1 -- 172.21.15.40:0/3337583973 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5a701a01e0 con 0x7f5a70100a30 2024-03-18T22:59:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.364+0000 7f5a66ffd640 1 -- 172.21.15.40:0/3337583973 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24908 (secure 0 0 0) 0x7f5a6c0bd9a0 con 0x7f5a70100a30 2024-03-18T22:59:27.366 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-18T22:59:27.366 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":93,"fsid":"410ef414-e577-11ee-95c9-87774f69a715","created":"2024-03-18T22:32:48.032793+0000","modified":"2024-03-18T22:42:44.941873+0000","last_up_change":"2024-03-18T22:39:53.930770+0000","last_in_change":"2024-03-18T22:39:34.099782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T22:36:07.060691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T22:40:39.925147+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T22:40:40.773757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T22:40:44.068183+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0f712e31-408e-427e-8826-2afdfc70bf80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6803","nonce":1816003248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6805","nonce":1816003248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6809","nonce":1816003248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":1816003248},{"type":"v1","addr":"172.21.15.40:6807","nonce":1816003248}]},"public_addr":"172.21.15.40:6803/1816003248","cluster_addr":"172.21.15.40:6805/1816003248","heartbeat_back_addr":"172.21.15.40:6809/1816003248","heartbeat_front_addr":"172.21.15.40:6807/1816003248","state":["exists","up"]},{"osd":1,"uuid":"5b2d7485-ba8b-4ce0-9ddf-be6a384a63b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6801","nonce":2851705755}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6803","nonce":2851705755}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6807","nonce":2851705755}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2851705755},{"type":"v1","addr":"172.21.15.103:6805","nonce":2851705755}]},"public_addr":"172.21.15.103:6801/2851705755","cluster_addr":"172.21.15.103:6803/2851705755","heartbeat_back_addr":"172.21.15.103:6807/2851705755","heartbeat_front_addr":"172.21.15.103:6805/2851705755","state":["exists","up"]},{"osd":2,"uuid":"69ba7362-43de-46f4-a5fa-b8492210fa27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6800","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6801","nonce":954444822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6802","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6803","nonce":954444822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6806","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6807","nonce":954444822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6804","nonce":954444822},{"type":"v1","addr":"172.21.15.113:6805","nonce":954444822}]},"public_addr":"172.21.15.113:6801/954444822","cluster_addr":"172.21.15.113:6803/954444822","heartbeat_back_addr":"172.21.15.113:6807/954444822","heartbeat_front_addr":"172.21.15.113:6805/954444822","state":["exists","up"]},{"osd":3,"uuid":"627e5c6b-419e-4c32-9207-0447a88e18d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6811","nonce":3224857891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6813","nonce":3224857891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6817","nonce":3224857891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":3224857891},{"type":"v1","addr":"172.21.15.40:6815","nonce":3224857891}]},"public_addr":"172.21.15.40:6811/3224857891","cluster_addr":"172.21.15.40:6813/3224857891","heartbeat_back_addr":"172.21.15.40:6817/3224857891","heartbeat_front_addr":"172.21.15.40:6815/3224857891","state":["exists","up"]},{"osd":4,"uuid":"34f3434e-5092-4639-9c2c-f64a196ea4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6809","nonce":3777182460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6811","nonce":3777182460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6815","nonce":3777182460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":3777182460},{"type":"v1","addr":"172.21.15.103:6813","nonce":3777182460}]},"public_addr":"172.21.15.103:6809/3777182460","cluster_addr":"172.21.15.103:6811/3777182460","heartbeat_back_addr":"172.21.15.103:6815/3777182460","heartbeat_front_addr":"172.21.15.103:6813/3777182460","state":["exists","up"]},{"osd":5,"uuid":"f9617fbc-5b2c-4e57-a926-b1b3a0953e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6808","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6809","nonce":3666840324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6810","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6811","nonce":3666840324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6814","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6815","nonce":3666840324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6812","nonce":3666840324},{"type":"v1","addr":"172.21.15.113:6813","nonce":3666840324}]},"public_addr":"172.21.15.113:6809/3666840324","cluster_addr":"172.21.15.113:6811/3666840324","heartbeat_back_addr":"172.21.15.113:6815/3666840324","heartbeat_front_addr":"172.21.15.113:6813/3666840324","state":["exists","up"]},{"osd":6,"uuid":"42df7e48-779a-4aee-bf6b-8c9fc8627d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6818","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6819","nonce":1125393763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6820","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6821","nonce":1125393763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6824","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6825","nonce":1125393763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6822","nonce":1125393763},{"type":"v1","addr":"172.21.15.40:6823","nonce":1125393763}]},"public_addr":"172.21.15.40:6819/1125393763","cluster_addr":"172.21.15.40:6821/1125393763","heartbeat_back_addr":"172.21.15.40:6825/1125393763","heartbeat_front_addr":"172.21.15.40:6823/1125393763","state":["exists","up"]},{"osd":7,"uuid":"1f3ce8a7-42f4-4582-9343-e34951f6b3a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6817","nonce":542253261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6819","nonce":542253261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6823","nonce":542253261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":542253261},{"type":"v1","addr":"172.21.15.103:6821","nonce":542253261}]},"public_addr":"172.21.15.103:6817/542253261","cluster_addr":"172.21.15.103:6819/542253261","heartbeat_back_addr":"172.21.15.103:6823/542253261","heartbeat_front_addr":"172.21.15.103:6821/542253261","state":["exists","up"]},{"osd":8,"uuid":"66b85f48-4e20-4563-b495-b50e8fe79c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6816","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6817","nonce":2089621708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6818","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6819","nonce":2089621708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6822","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6823","nonce":2089621708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6820","nonce":2089621708},{"type":"v1","addr":"172.21.15.113:6821","nonce":2089621708}]},"public_addr":"172.21.15.113:6817/2089621708","cluster_addr":"172.21.15.113:6819/2089621708","heartbeat_back_addr":"172.21.15.113:6823/2089621708","heartbeat_front_addr":"172.21.15.113:6821/2089621708","state":["exists","up"]},{"osd":9,"uuid":"e46e9424-c131-4559-9a43-55c7cf9c8eeb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6827","nonce":4067616401}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6828","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6829","nonce":4067616401}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6833","nonce":4067616401}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4067616401},{"type":"v1","addr":"172.21.15.40:6831","nonce":4067616401}]},"public_addr":"172.21.15.40:6827/4067616401","cluster_addr":"172.21.15.40:6829/4067616401","heartbeat_back_addr":"172.21.15.40:6833/4067616401","heartbeat_front_addr":"172.21.15.40:6831/4067616401","state":["exists","up"]},{"osd":10,"uuid":"df88c341-1ebe-45e5-aa2e-f8a0696c3770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6825","nonce":1414934819}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6827","nonce":1414934819}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6831","nonce":1414934819}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":1414934819},{"type":"v1","addr":"172.21.15.103:6829","nonce":1414934819}]},"public_addr":"172.21.15.103:6825/1414934819","cluster_addr":"172.21.15.103:6827/1414934819","heartbeat_back_addr":"172.21.15.103:6831/1414934819","heartbeat_front_addr":"172.21.15.103:6829/1414934819","state":["exists","up"]},{"osd":11,"uuid":"97b51a30-f636-4149-a06a-f536f1671467","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6824","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6825","nonce":287160143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6826","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6827","nonce":287160143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6830","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6831","nonce":287160143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.113:6828","nonce":287160143},{"type":"v1","addr":"172.21.15.113:6829","nonce":287160143}]},"public_addr":"172.21.15.113:6825/287160143","cluster_addr":"172.21.15.113:6827/287160143","heartbeat_back_addr":"172.21.15.113:6831/287160143","heartbeat_front_addr":"172.21.15.113:6829/287160143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:17.792719+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:35:41.609319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:04.960858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:29.815372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:36:54.267651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:18.691307+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:37:44.112773+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:09.526488+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:38:34.774359+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:00.750908+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:26.851179+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T22:39:52.013401+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":6}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":2,"to":1}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:0/347668672":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/4237962144":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/1454961757":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/3047621005":"2024-03-19T22:33:15.826478+0000","172.21.15.40:0/2121391701":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/3876208039":"2024-03-19T22:33:42.721571+0000","172.21.15.40:0/2591529486":"2024-03-19T22:33:42.721571+0000","172.21.15.40:6800/1153335782":"2024-03-19T22:33:15.826478+0000","172.21.15.40:6801/1153335782":"2024-03-19T22:33:15.826478+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T22:59:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.367+0000 7f5a77ae8640 1 -- 172.21.15.40:0/3337583973 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5a5c0a2800 msgr2=0x7f5a5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.367+0000 7f5a77ae8640 1 --2- 172.21.15.40:0/3337583973 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5a5c0a2800 0x7f5a5c0a4cc0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5a70198fb0 tx=0x7f5a60009260 comp rx=0 tx=0).stop 2024-03-18T22:59:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.367+0000 7f5a77ae8640 1 -- 172.21.15.40:0/3337583973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a70100a30 msgr2=0x7f5a70197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:27.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.367+0000 7f5a77ae8640 1 --2- 172.21.15.40:0/3337583973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a70100a30 0x7f5a70197aa0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c005940 tx=0x7f5a6c0042f0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.367+0000 7f5a77ae8640 1 -- 172.21.15.40:0/3337583973 shutdown_connections 2024-03-18T22:59:27.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.367+0000 7f5a77ae8640 1 --2- 172.21.15.40:0/3337583973 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f5a5c0a2800 0x7f5a5c0a4cc0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.367+0000 7f5a77ae8640 1 --2- 172.21.15.40:0/3337583973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a70101d00 0x7f5a7019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.367+0000 7f5a77ae8640 1 --2- 172.21.15.40:0/3337583973 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f5a701013f0 0x7f5a70197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.367+0000 7f5a77ae8640 1 --2- 172.21.15.40:0/3337583973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a70100a30 0x7f5a70197aa0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.367+0000 7f5a77ae8640 1 -- 172.21.15.40:0/3337583973 >> 172.21.15.40:0/3337583973 conn(0x7f5a700fb460 msgr2=0x7f5a700ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:27.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.367+0000 7f5a77ae8640 1 -- 172.21.15.40:0/3337583973 shutdown_connections 2024-03-18T22:59:27.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.367+0000 7f5a77ae8640 1 -- 172.21.15.40:0/3337583973 wait complete. 2024-03-18T22:59:27.380 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T22:59:27.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.482+0000 7f57f14ea640 1 -- 172.21.15.40:0/888748528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ec1022d0 msgr2=0x7f57ec108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:27.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.482+0000 7f57f14ea640 1 --2- 172.21.15.40:0/888748528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ec1022d0 0x7f57ec108b60 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f57e4009ca0 tx=0x7f57e402c0f0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.482+0000 7f57f14ea640 1 -- 172.21.15.40:0/888748528 shutdown_connections 2024-03-18T22:59:27.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.482+0000 7f57f14ea640 1 --2- 172.21.15.40:0/888748528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ec1022d0 0x7f57ec108b60 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.482+0000 7f57f14ea640 1 --2- 172.21.15.40:0/888748528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57ec101910 0x7f57ec101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.482+0000 7f57f14ea640 1 --2- 172.21.15.40:0/888748528 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57ec100720 0x7f57ec100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.482+0000 7f57f14ea640 1 -- 172.21.15.40:0/888748528 >> 172.21.15.40:0/888748528 conn(0x7f57ec0fbcb0 msgr2=0x7f57ec0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:27.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.482+0000 7f57f14ea640 1 -- 172.21.15.40:0/888748528 shutdown_connections 2024-03-18T22:59:27.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.482+0000 7f57f14ea640 1 -- 172.21.15.40:0/888748528 wait complete. 2024-03-18T22:59:27.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.483+0000 7f57f14ea640 1 Processor -- start 2024-03-18T22:59:27.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.483+0000 7f57f14ea640 1 -- start start 2024-03-18T22:59:27.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.483+0000 7f57f14ea640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ec100720 0x7f57ec1046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:27.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.483+0000 7f57f14ea640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57ec101910 0x7f57ec102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:27.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.483+0000 7f57f14ea640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57ec1022d0 0x7f57ec103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:27.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.483+0000 7f57f14ea640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f57ec10c230 con 0x7f57ec100720 2024-03-18T22:59:27.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.483+0000 7f57f14ea640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57ec10c0b0 con 0x7f57ec1022d0 2024-03-18T22:59:27.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.483+0000 7f57f14ea640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f57ec10c3b0 con 0x7f57ec101910 2024-03-18T22:59:27.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.483+0000 7f57eaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ec100720 0x7f57ec1046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:27.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.483+0000 7f57eaffd640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ec100720 0x7f57ec1046b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60792/0 (socket says 172.21.15.40:60792) 2024-03-18T22:59:27.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.483+0000 7f57eaffd640 1 -- 172.21.15.40:0/2253469720 learned_addr learned my addr 172.21.15.40:0/2253469720 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:27.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.483+0000 7f57e27fc640 1 --2- 172.21.15.40:0/2253469720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57ec101910 0x7f57ec102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:27.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.483+0000 7f57eb7fe640 1 --2- 172.21.15.40:0/2253469720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57ec1022d0 0x7f57ec103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:27.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.484+0000 7f57eaffd640 1 -- 172.21.15.40:0/2253469720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57ec101910 msgr2=0x7f57ec102cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:27.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.484+0000 7f57eaffd640 1 --2- 172.21.15.40:0/2253469720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57ec101910 0x7f57ec102cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.484+0000 7f57eaffd640 1 -- 172.21.15.40:0/2253469720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57ec1022d0 msgr2=0x7f57ec103230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:27.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.484+0000 7f57eaffd640 1 --2- 172.21.15.40:0/2253469720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57ec1022d0 0x7f57ec103230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.484+0000 7f57eaffd640 1 -- 172.21.15.40:0/2253469720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57ec103ac0 con 0x7f57ec100720 2024-03-18T22:59:27.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.484+0000 7f57eaffd640 1 --2- 172.21.15.40:0/2253469720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ec100720 0x7f57ec1046b0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00e9c0 tx=0x7f57dc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:27.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.484+0000 7f57e8ff9640 1 -- 172.21.15.40:0/2253469720 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc00cd80 con 0x7f57ec100720 2024-03-18T22:59:27.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.484+0000 7f57e8ff9640 1 -- 172.21.15.40:0/2253469720 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57dc0035f0 con 0x7f57ec100720 2024-03-18T22:59:27.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.484+0000 7f57f14ea640 1 -- 172.21.15.40:0/2253469720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57ec1a1b50 con 0x7f57ec100720 2024-03-18T22:59:27.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.484+0000 7f57e8ff9640 1 -- 172.21.15.40:0/2253469720 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc010690 con 0x7f57ec100720 2024-03-18T22:59:27.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.484+0000 7f57f14ea640 1 -- 172.21.15.40:0/2253469720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57ec1a20a0 con 0x7f57ec100720 2024-03-18T22:59:27.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.485+0000 7f57f14ea640 1 -- 172.21.15.40:0/2253469720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57ec1a2270 con 0x7f57ec100720 2024-03-18T22:59:27.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.486+0000 7f57e8ff9640 1 -- 172.21.15.40:0/2253469720 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f57dc0037a0 con 0x7f57ec100720 2024-03-18T22:59:27.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.487+0000 7f57e8ff9640 1 --2- 172.21.15.40:0/2253469720 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57d80a2800 0x7f57d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:27.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.487+0000 7f57e8ff9640 1 -- 172.21.15.40:0/2253469720 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f57dc0cf240 con 0x7f57ec100720 2024-03-18T22:59:27.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.487+0000 7f57e27fc640 1 --2- 172.21.15.40:0/2253469720 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57d80a2800 0x7f57d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:27.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.487+0000 7f57e27fc640 1 --2- 172.21.15.40:0/2253469720 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57d80a2800 0x7f57d80a4cc0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f57ec104420 tx=0x7f57d4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:27.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.491+0000 7f57e8ff9640 1 -- 172.21.15.40:0/2253469720 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f57dc09bef0 con 0x7f57ec100720 2024-03-18T22:59:27.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:27 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/698204001' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:59:27.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:27 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1456255816' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:59:27.643 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:27 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/3337583973' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:59:27.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.714+0000 7f57f14ea640 1 -- 172.21.15.40:0/2253469720 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f57ec1a2ac0 con 0x7f57ec100720 2024-03-18T22:59:27.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.717+0000 7f57e8ff9640 1 -- 172.21.15.40:0/2253469720 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v50) ==== 161+0+0 (secure 0 0 0) 0x7f57dc09b7f0 con 0x7f57ec100720 2024-03-18T22:59:27.719 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T22:59:27.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.720+0000 7f57f14ea640 1 -- 172.21.15.40:0/2253469720 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57d80a2800 msgr2=0x7f57d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:27.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.720+0000 7f57f14ea640 1 --2- 172.21.15.40:0/2253469720 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57d80a2800 0x7f57d80a4cc0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f57ec104420 tx=0x7f57d4005f90 comp rx=0 tx=0).stop 2024-03-18T22:59:27.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.720+0000 7f57f14ea640 1 -- 172.21.15.40:0/2253469720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ec100720 msgr2=0x7f57ec1046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:27.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.720+0000 7f57f14ea640 1 --2- 172.21.15.40:0/2253469720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ec100720 0x7f57ec1046b0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00e9c0 tx=0x7f57dc00ee80 comp rx=0 tx=0).stop 2024-03-18T22:59:27.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.720+0000 7f57f14ea640 1 -- 172.21.15.40:0/2253469720 shutdown_connections 2024-03-18T22:59:27.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.720+0000 7f57f14ea640 1 --2- 172.21.15.40:0/2253469720 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f57d80a2800 0x7f57d80a4cc0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.720+0000 7f57f14ea640 1 --2- 172.21.15.40:0/2253469720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57ec1022d0 0x7f57ec103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.720+0000 7f57f14ea640 1 --2- 172.21.15.40:0/2253469720 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f57ec101910 0x7f57ec102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.720+0000 7f57f14ea640 1 --2- 172.21.15.40:0/2253469720 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f57ec100720 0x7f57ec1046b0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.720+0000 7f57f14ea640 1 -- 172.21.15.40:0/2253469720 >> 172.21.15.40:0/2253469720 conn(0x7f57ec0fbcb0 msgr2=0x7f57ec0689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:27.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.720+0000 7f57f14ea640 1 -- 172.21.15.40:0/2253469720 shutdown_connections 2024-03-18T22:59:27.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.720+0000 7f57f14ea640 1 -- 172.21.15.40:0/2253469720 wait complete. 2024-03-18T22:59:27.735 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T22:59:27.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.854+0000 7fdf1d722640 1 -- 172.21.15.40:0/3965715026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf181022d0 msgr2=0x7fdf18108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:27.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.854+0000 7fdf1d722640 1 --2- 172.21.15.40:0/3965715026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf181022d0 0x7fdf18108b60 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fdf08009ca0 tx=0x7fdf0802c0f0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.854+0000 7fdf1d722640 1 -- 172.21.15.40:0/3965715026 shutdown_connections 2024-03-18T22:59:27.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.854+0000 7fdf1d722640 1 --2- 172.21.15.40:0/3965715026 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf181022d0 0x7fdf18108b60 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.854+0000 7fdf1d722640 1 --2- 172.21.15.40:0/3965715026 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf18101910 0x7fdf18101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.854+0000 7fdf1d722640 1 --2- 172.21.15.40:0/3965715026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf18100720 0x7fdf18100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.854+0000 7fdf1d722640 1 -- 172.21.15.40:0/3965715026 >> 172.21.15.40:0/3965715026 conn(0x7fdf180fbcb0 msgr2=0x7fdf180fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:27.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.854+0000 7fdf1d722640 1 -- 172.21.15.40:0/3965715026 shutdown_connections 2024-03-18T22:59:27.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.854+0000 7fdf1d722640 1 -- 172.21.15.40:0/3965715026 wait complete. 2024-03-18T22:59:27.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.854+0000 7fdf1d722640 1 Processor -- start 2024-03-18T22:59:27.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.854+0000 7fdf1d722640 1 -- start start 2024-03-18T22:59:27.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf1d722640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf18100720 0x7fdf18197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:27.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf1d722640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf18101910 0x7fdf18197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:27.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf1d722640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf181022d0 0x7fdf1819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:27.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf1d722640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1810c250 con 0x7fdf181022d0 2024-03-18T22:59:27.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf1d722640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1810c0d0 con 0x7fdf18101910 2024-03-18T22:59:27.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf1d722640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1810c3d0 con 0x7fdf18100720 2024-03-18T22:59:27.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf177fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf181022d0 0x7fdf1819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:27.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf177fe640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf181022d0 0x7fdf1819efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60834/0 (socket says 172.21.15.40:60834) 2024-03-18T22:59:27.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf177fe640 1 -- 172.21.15.40:0/1428457423 learned_addr learned my addr 172.21.15.40:0/1428457423 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:27.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf167fc640 1 --2- 172.21.15.40:0/1428457423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf18101910 0x7fdf18197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:27.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf16ffd640 1 --2- 172.21.15.40:0/1428457423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf18100720 0x7fdf18197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:27.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf177fe640 1 -- 172.21.15.40:0/1428457423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf18100720 msgr2=0x7fdf18197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:27.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf177fe640 1 --2- 172.21.15.40:0/1428457423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf18100720 0x7fdf18197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf177fe640 1 -- 172.21.15.40:0/1428457423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf18101910 msgr2=0x7fdf18197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:27.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf177fe640 1 --2- 172.21.15.40:0/1428457423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf18101910 0x7fdf18197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:27.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf177fe640 1 -- 172.21.15.40:0/1428457423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf1819f640 con 0x7fdf181022d0 2024-03-18T22:59:27.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf167fc640 1 --2- 172.21.15.40:0/1428457423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf18101910 0x7fdf18197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T22:59:27.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf177fe640 1 --2- 172.21.15.40:0/1428457423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf181022d0 0x7fdf1819efd0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fdf0802c700 tx=0x7fdf080328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:27.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf0ffff640 1 -- 172.21.15.40:0/1428457423 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf08041020 con 0x7fdf181022d0 2024-03-18T22:59:27.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf0ffff640 1 -- 172.21.15.40:0/1428457423 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf08003e20 con 0x7fdf181022d0 2024-03-18T22:59:27.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf1d722640 1 -- 172.21.15.40:0/1428457423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf1819f8d0 con 0x7fdf181022d0 2024-03-18T22:59:27.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf0ffff640 1 -- 172.21.15.40:0/1428457423 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf08039040 con 0x7fdf181022d0 2024-03-18T22:59:27.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.855+0000 7fdf1d722640 1 -- 172.21.15.40:0/1428457423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf1819fdc0 con 0x7fdf181022d0 2024-03-18T22:59:27.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.856+0000 7fdf1d722640 1 -- 172.21.15.40:0/1428457423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf181a0020 con 0x7fdf181022d0 2024-03-18T22:59:27.862 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/698204001' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:59:27.862 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1456255816' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:59:27.862 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/3337583973' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:59:27.863 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-k[94994]: ignoring --setuser ceph since I am not root 2024-03-18T22:59:27.863 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-k[94994]: ignoring --setgroup ceph since I am not root 2024-03-18T22:59:27.863 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-mds[94998]: -- 172.21.15.103:0/2611717060 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5626f2fefa40 con 0x5626f2fc2800 2024-03-18T22:59:27.863 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-k[94994]: starting mds.k at 2024-03-18T22:59:27.863 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-b[95936]: ignoring --setuser ceph since I am not root 2024-03-18T22:59:27.864 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-b[95936]: ignoring --setgroup ceph since I am not root 2024-03-18T22:59:27.864 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-b[95936]: starting mds.b at 2024-03-18T22:59:27.864 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-h[95619]: ignoring --setuser ceph since I am not root 2024-03-18T22:59:27.864 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-h[95619]: ignoring --setgroup ceph since I am not root 2024-03-18T22:59:27.864 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-mds[95623]: -- 172.21.15.103:0/3400293683 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562f53c27a40 con 0x562f53bfa800 2024-03-18T22:59:27.864 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-h[95619]: starting mds.h at 2024-03-18T22:59:27.865 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-e[95315]: ignoring --setuser ceph since I am not root 2024-03-18T22:59:27.865 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-e[95315]: ignoring --setgroup ceph since I am not root 2024-03-18T22:59:27.865 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-mds[95319]: -- 172.21.15.103:0/1608688428 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eb589aba40 con 0x55eb5897e800 2024-03-18T22:59:27.865 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 22:59:27 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-e[95315]: starting mds.e at 2024-03-18T22:59:27.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.857+0000 7fdf0ffff640 1 -- 172.21.15.40:0/1428457423 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdf08044020 con 0x7fdf181022d0 2024-03-18T22:59:27.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.858+0000 7fdf0ffff640 1 --2- 172.21.15.40:0/1428457423 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf040a2880 0x7fdf040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:27.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.858+0000 7fdf16ffd640 1 --2- 172.21.15.40:0/1428457423 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf040a2880 0x7fdf040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:27.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.858+0000 7fdf0ffff640 1 -- 172.21.15.40:0/1428457423 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9675+0+0 (secure 0 0 0) 0x7fdf080f1840 con 0x7fdf181022d0 2024-03-18T22:59:27.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.858+0000 7fdf16ffd640 1 --2- 172.21.15.40:0/1428457423 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf040a2880 0x7fdf040a4d40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fdf10005fd0 tx=0x7fdf1000b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:27.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:27.861+0000 7fdf0ffff640 1 -- 172.21.15.40:0/1428457423 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdf0804b020 con 0x7fdf181022d0 2024-03-18T22:59:28.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/698204001' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:59:28.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1456255816' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T22:59:28.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/3337583973' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T22:59:28.080 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-i[98437]: ignoring --setuser ceph since I am not root 2024-03-18T22:59:28.080 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-i[98437]: ignoring --setgroup ceph since I am not root 2024-03-18T22:59:28.081 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-mds[98441]: -- 172.21.15.113:0/1706760982 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x559e9fb2e4e0 con 0x559e9fb04800 2024-03-18T22:59:28.081 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-mds[98441]: -- 172.21.15.113:0/1706760982 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559e9fb31a40 con 0x559e9fb04800 2024-03-18T22:59:28.081 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-i[98437]: starting mds.i at 2024-03-18T22:59:28.081 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-f[97804]: ignoring --setuser ceph since I am not root 2024-03-18T22:59:28.081 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-f[97804]: ignoring --setgroup ceph since I am not root 2024-03-18T22:59:28.081 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-mds[97808]: -- 172.21.15.113:0/2933680285 <== mon.1 v2:172.21.15.113:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562470b65a40 con 0x562470b38c00 2024-03-18T22:59:28.081 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-f[97804]: starting mds.f at 2024-03-18T22:59:28.082 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-c[98113]: ignoring --setuser ceph since I am not root 2024-03-18T22:59:28.082 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-c[98113]: ignoring --setgroup ceph since I am not root 2024-03-18T22:59:28.082 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-mds[98117]: -- 172.21.15.113:0/756986578 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562d794f5a40 con 0x562d794c8c00 2024-03-18T22:59:28.082 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-c[98113]: starting mds.c at 2024-03-18T22:59:28.082 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-l[97489]: ignoring --setuser ceph since I am not root 2024-03-18T22:59:28.082 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-l[97489]: ignoring --setgroup ceph since I am not root 2024-03-18T22:59:28.082 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-mds[97493]: -- 172.21.15.113:0/663099384 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fbf76da4e0 con 0x55fbf76b0800 2024-03-18T22:59:28.082 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-mds[97493]: -- 172.21.15.113:0/663099384 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fbf76dda40 con 0x55fbf76b0800 2024-03-18T22:59:28.082 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 22:59:27 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-l[97489]: starting mds.l at 2024-03-18T22:59:28.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.087+0000 7fdf1d722640 1 -- 172.21.15.40:0/1428457423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fdf181a03b0 con 0x7fdf181022d0 2024-03-18T22:59:28.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:27 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a[62639]: 2024-03-18T22:59:27.714+0000 7f354dd5d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T22:59:28.143 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 22:59:27 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-a[101365]: ignoring --setuser ceph since I am not root 2024-03-18T22:59:28.144 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 22:59:27 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-a[101365]: ignoring --setgroup ceph since I am not root 2024-03-18T22:59:28.144 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 22:59:27 smithi040 ceph-mds[101369]: -- 172.21.15.40:0/3466002309 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b9eefe3a40 con 0x55b9eefb6800 2024-03-18T22:59:28.144 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 22:59:27 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-a[101365]: starting mds.a at 2024-03-18T22:59:28.144 INFO:journalctl@ceph.mds.g.smithi040.stdout:Mar 18 22:59:27 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-g[101001]: ignoring --setuser ceph since I am not root 2024-03-18T22:59:28.144 INFO:journalctl@ceph.mds.g.smithi040.stdout:Mar 18 22:59:27 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-g[101001]: ignoring --setgroup ceph since I am not root 2024-03-18T22:59:28.144 INFO:journalctl@ceph.mds.g.smithi040.stdout:Mar 18 22:59:27 smithi040 ceph-mds[101005]: -- 172.21.15.40:0/2941208197 <== mon.0 v2:172.21.15.40:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55910807da40 con 0x559108050800 2024-03-18T22:59:28.144 INFO:journalctl@ceph.mds.g.smithi040.stdout:Mar 18 22:59:27 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-g[101001]: starting mds.g at 2024-03-18T22:59:28.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.719+0000 7fdf0ffff640 1 -- 172.21.15.40:0/1428457423 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v51) ==== 104+0+0 (secure 0 0 0) 0x7fdf080be2e0 con 0x7fdf181022d0 2024-03-18T22:59:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.722+0000 7fdf1d722640 1 -- 172.21.15.40:0/1428457423 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf040a2880 msgr2=0x7fdf040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.722+0000 7fdf1d722640 1 --2- 172.21.15.40:0/1428457423 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf040a2880 0x7fdf040a4d40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fdf10005fd0 tx=0x7fdf1000b410 comp rx=0 tx=0).stop 2024-03-18T22:59:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.722+0000 7fdf1d722640 1 -- 172.21.15.40:0/1428457423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf181022d0 msgr2=0x7fdf1819efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.722+0000 7fdf1d722640 1 --2- 172.21.15.40:0/1428457423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf181022d0 0x7fdf1819efd0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fdf0802c700 tx=0x7fdf080328a0 comp rx=0 tx=0).stop 2024-03-18T22:59:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.722+0000 7fdf1d722640 1 -- 172.21.15.40:0/1428457423 shutdown_connections 2024-03-18T22:59:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.722+0000 7fdf1d722640 1 --2- 172.21.15.40:0/1428457423 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7fdf040a2880 0x7fdf040a4d40 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.722+0000 7fdf1d722640 1 --2- 172.21.15.40:0/1428457423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf181022d0 0x7fdf1819efd0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:28.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.722+0000 7fdf1d722640 1 --2- 172.21.15.40:0/1428457423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdf18101910 0x7fdf18197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:28.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.722+0000 7fdf1d722640 1 --2- 172.21.15.40:0/1428457423 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7fdf18100720 0x7fdf18197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:28.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.722+0000 7fdf1d722640 1 -- 172.21.15.40:0/1428457423 >> 172.21.15.40:0/1428457423 conn(0x7fdf180fbcb0 msgr2=0x7fdf180fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:28.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.722+0000 7fdf1d722640 1 -- 172.21.15.40:0/1428457423 shutdown_connections 2024-03-18T22:59:28.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.722+0000 7fdf1d722640 1 -- 172.21.15.40:0/1428457423 wait complete. 2024-03-18T22:59:28.741 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T22:59:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.854+0000 7f1d5eb32640 1 -- 172.21.15.40:0/481351821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d581022d0 msgr2=0x7f1d58108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.854+0000 7f1d5eb32640 1 --2- 172.21.15.40:0/481351821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d581022d0 0x7f1d58108b60 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f1d50009ca0 tx=0x7f1d5002c0f0 comp rx=0 tx=0).stop 2024-03-18T22:59:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.854+0000 7f1d5eb32640 1 -- 172.21.15.40:0/481351821 shutdown_connections 2024-03-18T22:59:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.854+0000 7f1d5eb32640 1 --2- 172.21.15.40:0/481351821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d581022d0 0x7f1d58108b60 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.854+0000 7f1d5eb32640 1 --2- 172.21.15.40:0/481351821 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d58101910 0x7f1d58101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.854+0000 7f1d5eb32640 1 --2- 172.21.15.40:0/481351821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d58100720 0x7f1d58100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.854+0000 7f1d5eb32640 1 -- 172.21.15.40:0/481351821 >> 172.21.15.40:0/481351821 conn(0x7f1d580fbcb0 msgr2=0x7f1d580fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.854+0000 7f1d5eb32640 1 -- 172.21.15.40:0/481351821 shutdown_connections 2024-03-18T22:59:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.854+0000 7f1d5eb32640 1 -- 172.21.15.40:0/481351821 wait complete. 2024-03-18T22:59:28.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.854+0000 7f1d5eb32640 1 Processor -- start 2024-03-18T22:59:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.854+0000 7f1d5eb32640 1 -- start start 2024-03-18T22:59:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5eb32640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d58100720 0x7f1d581979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5eb32640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d58101910 0x7f1d58197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5eb32640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d581022d0 0x7f1d5819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5eb32640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f1d5810c230 con 0x7f1d58100720 2024-03-18T22:59:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5eb32640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1d5810c0b0 con 0x7f1d58101910 2024-03-18T22:59:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5eb32640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f1d5810c3b0 con 0x7f1d581022d0 2024-03-18T22:59:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5c8ce640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d58100720 0x7f1d581979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5c8ce640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d58100720 0x7f1d581979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57146/0 (socket says 172.21.15.40:57146) 2024-03-18T22:59:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5c8ce640 1 -- 172.21.15.40:0/2367776497 learned_addr learned my addr 172.21.15.40:0/2367776497 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:28.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d57fff640 1 --2- 172.21.15.40:0/2367776497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d58101910 0x7f1d58197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:28.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5d0cf640 1 --2- 172.21.15.40:0/2367776497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d581022d0 0x7f1d5819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:28.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5c8ce640 1 -- 172.21.15.40:0/2367776497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d581022d0 msgr2=0x7f1d5819ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:28.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5c8ce640 1 --2- 172.21.15.40:0/2367776497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d581022d0 0x7f1d5819ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:28.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5c8ce640 1 -- 172.21.15.40:0/2367776497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d58101910 msgr2=0x7f1d58197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:28.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5c8ce640 1 --2- 172.21.15.40:0/2367776497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d58101910 0x7f1d58197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:28.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5c8ce640 1 -- 172.21.15.40:0/2367776497 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d5819f5e0 con 0x7f1d58100720 2024-03-18T22:59:28.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d57fff640 1 --2- 172.21.15.40:0/2367776497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d58101910 0x7f1d58197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:59:28.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5c8ce640 1 --2- 172.21.15.40:0/2367776497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d58100720 0x7f1d581979c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c00c970 tx=0x7f1d4c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:28.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d55ffb640 1 -- 172.21.15.40:0/2367776497 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c007c10 con 0x7f1d58100720 2024-03-18T22:59:28.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d55ffb640 1 -- 172.21.15.40:0/2367776497 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d4c007db0 con 0x7f1d58100720 2024-03-18T22:59:28.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d5eb32640 1 -- 172.21.15.40:0/2367776497 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d5819f8d0 con 0x7f1d58100720 2024-03-18T22:59:28.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.855+0000 7f1d55ffb640 1 -- 172.21.15.40:0/2367776497 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c00f870 con 0x7f1d58100720 2024-03-18T22:59:28.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.856+0000 7f1d5eb32640 1 -- 172.21.15.40:0/2367776497 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d5819fe70 con 0x7f1d58100720 2024-03-18T22:59:28.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.857+0000 7f1d5eb32640 1 -- 172.21.15.40:0/2367776497 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d58068610 con 0x7f1d58100720 2024-03-18T22:59:28.863 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:28 smithi103 ceph-mon[66377]: pgmap v855: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-03-18T22:59:28.863 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:28 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2253469720' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T22:59:28.863 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:28 smithi103 ceph-mon[66377]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T22:59:28.863 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:28 smithi103 ceph-mon[66377]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T22:59:28.863 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:28 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2253469720' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T22:59:28.863 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:28 smithi103 ceph-mon[66377]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T22:59:28.864 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:28 smithi103 ceph-mon[66377]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T22:59:28.864 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:28 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1428457423' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T22:59:28.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.859+0000 7f1d55ffb640 1 -- 172.21.15.40:0/2367776497 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1d4c00fa10 con 0x7f1d58100720 2024-03-18T22:59:28.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.859+0000 7f1d55ffb640 1 --2- 172.21.15.40:0/2367776497 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1d480a2880 0x7f1d480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:28.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.859+0000 7f1d57fff640 1 --2- 172.21.15.40:0/2367776497 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1d480a2880 0x7f1d480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:28.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.860+0000 7f1d55ffb640 1 -- 172.21.15.40:0/2367776497 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9675+0+0 (secure 0 0 0) 0x7f1d4c0cf420 con 0x7f1d58100720 2024-03-18T22:59:28.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.860+0000 7f1d57fff640 1 --2- 172.21.15.40:0/2367776497 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1d480a2880 0x7f1d480a4d40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1d58198ed0 tx=0x7f1d4400a5c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:28.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:28.864+0000 7f1d55ffb640 1 -- 172.21.15.40:0/2367776497 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1d4c09bf50 con 0x7f1d58100720 2024-03-18T22:59:28.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:28 smithi040 ceph-mon[62643]: pgmap v855: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-03-18T22:59:28.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:28 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2253469720' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T22:59:28.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:28 smithi040 ceph-mon[62643]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T22:59:28.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:28 smithi040 ceph-mon[62643]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T22:59:28.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:28 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2253469720' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T22:59:28.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:28 smithi040 ceph-mon[62643]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T22:59:28.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:28 smithi040 ceph-mon[62643]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T22:59:28.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:28 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1428457423' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T22:59:28.981 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:28 smithi113 ceph-mon[66775]: pgmap v855: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-03-18T22:59:28.981 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:28 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2253469720' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T22:59:28.981 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:28 smithi113 ceph-mon[66775]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T22:59:28.981 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:28 smithi113 ceph-mon[66775]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T22:59:28.982 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:28 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2253469720' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T22:59:28.982 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:28 smithi113 ceph-mon[66775]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T22:59:28.982 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:28 smithi113 ceph-mon[66775]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T22:59:28.982 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:28 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1428457423' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T22:59:29.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.049+0000 7f1d5eb32640 1 -- 172.21.15.40:0/2367776497 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1d581a0100 con 0x7f1d58100720 2024-03-18T22:59:29.729 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-03-18T22:59:29.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.727+0000 7f1d55ffb640 1 -- 172.21.15.40:0/2367776497 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v95) ==== 181+0+0 (secure 0 0 0) 0x7f1d4c09b850 con 0x7f1d58100720 2024-03-18T22:59:29.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.730+0000 7f1d5eb32640 1 -- 172.21.15.40:0/2367776497 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1d480a2880 msgr2=0x7f1d480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:29.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.730+0000 7f1d5eb32640 1 --2- 172.21.15.40:0/2367776497 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1d480a2880 0x7f1d480a4d40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f1d58198ed0 tx=0x7f1d4400a5c0 comp rx=0 tx=0).stop 2024-03-18T22:59:29.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.730+0000 7f1d5eb32640 1 -- 172.21.15.40:0/2367776497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d58100720 msgr2=0x7f1d581979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:29.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.730+0000 7f1d5eb32640 1 --2- 172.21.15.40:0/2367776497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d58100720 0x7f1d581979c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c00c970 tx=0x7f1d4c00ce30 comp rx=0 tx=0).stop 2024-03-18T22:59:29.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.730+0000 7f1d5eb32640 1 -- 172.21.15.40:0/2367776497 shutdown_connections 2024-03-18T22:59:29.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.730+0000 7f1d5eb32640 1 --2- 172.21.15.40:0/2367776497 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f1d480a2880 0x7f1d480a4d40 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:29.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.730+0000 7f1d5eb32640 1 --2- 172.21.15.40:0/2367776497 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f1d581022d0 0x7f1d5819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:29.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.730+0000 7f1d5eb32640 1 --2- 172.21.15.40:0/2367776497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d58101910 0x7f1d58197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:29.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.730+0000 7f1d5eb32640 1 --2- 172.21.15.40:0/2367776497 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d58100720 0x7f1d581979c0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:29.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.730+0000 7f1d5eb32640 1 -- 172.21.15.40:0/2367776497 >> 172.21.15.40:0/2367776497 conn(0x7f1d580fbcb0 msgr2=0x7f1d580fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:29.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.730+0000 7f1d5eb32640 1 -- 172.21.15.40:0/2367776497 shutdown_connections 2024-03-18T22:59:29.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.730+0000 7f1d5eb32640 1 -- 172.21.15.40:0/2367776497 wait complete. 2024-03-18T22:59:29.747 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T22:59:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.857+0000 7f3b4913b640 1 -- 172.21.15.40:0/4016217476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b44101910 msgr2=0x7f3b44101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.857+0000 7f3b4913b640 1 --2- 172.21.15.40:0/4016217476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b44101910 0x7f3b44101d90 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f3b38009900 tx=0x7f3b3802bc30 comp rx=0 tx=0).stop 2024-03-18T22:59:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.857+0000 7f3b4913b640 1 -- 172.21.15.40:0/4016217476 shutdown_connections 2024-03-18T22:59:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.857+0000 7f3b4913b640 1 --2- 172.21.15.40:0/4016217476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b441022d0 0x7f3b44108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.857+0000 7f3b4913b640 1 --2- 172.21.15.40:0/4016217476 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b44101910 0x7f3b44101d90 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.857+0000 7f3b4913b640 1 --2- 172.21.15.40:0/4016217476 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b44100720 0x7f3b44100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.857+0000 7f3b4913b640 1 -- 172.21.15.40:0/4016217476 >> 172.21.15.40:0/4016217476 conn(0x7f3b440fbcb0 msgr2=0x7f3b440fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.857+0000 7f3b4913b640 1 -- 172.21.15.40:0/4016217476 shutdown_connections 2024-03-18T22:59:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.857+0000 7f3b4913b640 1 -- 172.21.15.40:0/4016217476 wait complete. 2024-03-18T22:59:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.858+0000 7f3b4913b640 1 Processor -- start 2024-03-18T22:59:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.858+0000 7f3b4913b640 1 -- start start 2024-03-18T22:59:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.858+0000 7f3b4913b640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b44100720 0x7f3b441046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.858+0000 7f3b4913b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b44101910 0x7f3b44102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.858+0000 7f3b4913b640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b441022d0 0x7f3b44103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.858+0000 7f3b4913b640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4410c230 con 0x7f3b441022d0 2024-03-18T22:59:29.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.858+0000 7f3b4913b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4410c0b0 con 0x7f3b44101910 2024-03-18T22:59:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.858+0000 7f3b4913b640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4410c3b0 con 0x7f3b44100720 2024-03-18T22:59:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.859+0000 7f3b4259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b44101910 0x7f3b44102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.859+0000 7f3b4259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b44101910 0x7f3b44102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.40:56624/0 (socket says 172.21.15.40:56624) 2024-03-18T22:59:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.859+0000 7f3b4259c640 1 -- 172.21.15.40:0/1436808022 learned_addr learned my addr 172.21.15.40:0/1436808022 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.859+0000 7f3b4359e640 1 --2- 172.21.15.40:0/1436808022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b441022d0 0x7f3b44103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.859+0000 7f3b42d9d640 1 --2- 172.21.15.40:0/1436808022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b44100720 0x7f3b441046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.859+0000 7f3b4259c640 1 -- 172.21.15.40:0/1436808022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b44100720 msgr2=0x7f3b441046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.859+0000 7f3b4259c640 1 --2- 172.21.15.40:0/1436808022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b44100720 0x7f3b441046b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.859+0000 7f3b4259c640 1 -- 172.21.15.40:0/1436808022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b441022d0 msgr2=0x7f3b44103230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.859+0000 7f3b4259c640 1 --2- 172.21.15.40:0/1436808022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b441022d0 0x7f3b44103230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.859+0000 7f3b4259c640 1 -- 172.21.15.40:0/1436808022 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b44103ac0 con 0x7f3b44101910 2024-03-18T22:59:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.859+0000 7f3b4259c640 1 --2- 172.21.15.40:0/1436808022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b44101910 0x7f3b44102cf0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3b38006010 tx=0x7f3b380042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.860+0000 7f3b2bfff640 1 -- 172.21.15.40:0/1436808022 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b38043020 con 0x7f3b44101910 2024-03-18T22:59:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.860+0000 7f3b4913b640 1 -- 172.21.15.40:0/1436808022 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b441a1b70 con 0x7f3b44101910 2024-03-18T22:59:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.860+0000 7f3b4913b640 1 -- 172.21.15.40:0/1436808022 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b441a2110 con 0x7f3b44101910 2024-03-18T22:59:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.860+0000 7f3b2bfff640 1 -- 172.21.15.40:0/1436808022 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b38004aa0 con 0x7f3b44101910 2024-03-18T22:59:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.860+0000 7f3b2bfff640 1 -- 172.21.15.40:0/1436808022 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b38003c60 con 0x7f3b44101910 2024-03-18T22:59:29.864 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T22:59:29.865 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T22:59:29.865 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: Cluster is now healthy 2024-03-18T22:59:29.865 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1428457423' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T22:59:29.865 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.103:6833/1518575009,v1:172.21.15.103:6835/1518575009] up:boot 2024-03-18T22:59:29.865 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.103:6832/900254621,v1:172.21.15.103:6834/900254621] up:boot 2024-03-18T22:59:29.865 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.103:6836/1790329871,v1:172.21.15.103:6838/1790329871] up:boot 2024-03-18T22:59:29.865 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.103:6837/1238992455,v1:172.21.15.103:6839/1238992455] up:boot 2024-03-18T22:59:29.865 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.113:6833/2325620757,v1:172.21.15.113:6836/2325620757] up:boot 2024-03-18T22:59:29.865 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.113:6832/1878955320,v1:172.21.15.113:6835/1878955320] up:boot 2024-03-18T22:59:29.865 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.40:6838/1724914336,v1:172.21.15.40:6839/1724914336] up:boot 2024-03-18T22:59:29.865 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.113:6837/1408309056,v1:172.21.15.113:6839/1408309056] up:boot 2024-03-18T22:59:29.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.113:6834/387383242,v1:172.21.15.113:6838/387383242] up:boot 2024-03-18T22:59:29.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: mds.? [v2:172.21.15.40:6840/855241432,v1:172.21.15.40:6841/855241432] up:boot 2024-03-18T22:59:29.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: fsmap 12 up:standby 2024-03-18T22:59:29.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T22:59:29.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T22:59:29.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T22:59:29.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T22:59:29.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T22:59:29.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T22:59:29.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T22:59:29.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T22:59:29.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T22:59:29.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T22:59:29.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:29 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2367776497' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T22:59:29.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.861+0000 7f3b4913b640 1 -- 172.21.15.40:0/1436808022 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b30005290 con 0x7f3b44101910 2024-03-18T22:59:29.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.863+0000 7f3b2bfff640 1 -- 172.21.15.40:0/1436808022 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3b38046020 con 0x7f3b44101910 2024-03-18T22:59:29.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.864+0000 7f3b2bfff640 1 --2- 172.21.15.40:0/1436808022 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3b2c0a2880 0x7f3b2c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:29.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.864+0000 7f3b42d9d640 1 --2- 172.21.15.40:0/1436808022 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3b2c0a2880 0x7f3b2c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:29.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.864+0000 7f3b2bfff640 1 -- 172.21.15.40:0/1436808022 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9084+0+0 (secure 0 0 0) 0x7f3b380f0f20 con 0x7f3b44101910 2024-03-18T22:59:29.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.864+0000 7f3b42d9d640 1 --2- 172.21.15.40:0/1436808022 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3b2c0a2880 0x7f3b2c0a4d40 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3b44101770 tx=0x7f3b3400b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:29.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:29.868+0000 7f3b2bfff640 1 -- 172.21.15.40:0/1436808022 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3b3804d020 con 0x7f3b44101910 2024-03-18T22:59:29.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T22:59:29.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T22:59:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: Cluster is now healthy 2024-03-18T22:59:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1428457423' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T22:59:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.103:6833/1518575009,v1:172.21.15.103:6835/1518575009] up:boot 2024-03-18T22:59:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.103:6832/900254621,v1:172.21.15.103:6834/900254621] up:boot 2024-03-18T22:59:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.103:6836/1790329871,v1:172.21.15.103:6838/1790329871] up:boot 2024-03-18T22:59:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.103:6837/1238992455,v1:172.21.15.103:6839/1238992455] up:boot 2024-03-18T22:59:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.113:6833/2325620757,v1:172.21.15.113:6836/2325620757] up:boot 2024-03-18T22:59:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.113:6832/1878955320,v1:172.21.15.113:6835/1878955320] up:boot 2024-03-18T22:59:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.40:6838/1724914336,v1:172.21.15.40:6839/1724914336] up:boot 2024-03-18T22:59:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.113:6837/1408309056,v1:172.21.15.113:6839/1408309056] up:boot 2024-03-18T22:59:29.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.113:6834/387383242,v1:172.21.15.113:6838/387383242] up:boot 2024-03-18T22:59:29.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: mds.? [v2:172.21.15.40:6840/855241432,v1:172.21.15.40:6841/855241432] up:boot 2024-03-18T22:59:29.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: fsmap 12 up:standby 2024-03-18T22:59:29.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T22:59:29.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T22:59:29.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T22:59:29.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T22:59:29.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T22:59:29.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T22:59:29.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T22:59:29.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T22:59:29.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T22:59:29.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T22:59:29.895 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:29 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2367776497' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T22:59:30.013 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T22:59:30.014 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T22:59:30.014 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: Cluster is now healthy 2024-03-18T22:59:30.014 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1428457423' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T22:59:30.014 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.103:6833/1518575009,v1:172.21.15.103:6835/1518575009] up:boot 2024-03-18T22:59:30.014 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.103:6832/900254621,v1:172.21.15.103:6834/900254621] up:boot 2024-03-18T22:59:30.014 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.103:6836/1790329871,v1:172.21.15.103:6838/1790329871] up:boot 2024-03-18T22:59:30.014 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.103:6837/1238992455,v1:172.21.15.103:6839/1238992455] up:boot 2024-03-18T22:59:30.014 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.113:6833/2325620757,v1:172.21.15.113:6836/2325620757] up:boot 2024-03-18T22:59:30.014 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.113:6832/1878955320,v1:172.21.15.113:6835/1878955320] up:boot 2024-03-18T22:59:30.014 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.40:6838/1724914336,v1:172.21.15.40:6839/1724914336] up:boot 2024-03-18T22:59:30.015 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.113:6837/1408309056,v1:172.21.15.113:6839/1408309056] up:boot 2024-03-18T22:59:30.015 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.113:6834/387383242,v1:172.21.15.113:6838/387383242] up:boot 2024-03-18T22:59:30.015 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: mds.? [v2:172.21.15.40:6840/855241432,v1:172.21.15.40:6841/855241432] up:boot 2024-03-18T22:59:30.015 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: fsmap 12 up:standby 2024-03-18T22:59:30.015 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T22:59:30.015 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T22:59:30.015 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T22:59:30.015 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T22:59:30.015 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T22:59:30.015 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T22:59:30.015 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T22:59:30.015 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T22:59:30.016 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T22:59:30.016 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: from='mgr.14148 172.21.15.40:0/3920815336' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T22:59:30.016 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:29 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2367776497' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T22:59:30.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.068+0000 7f3b4913b640 1 -- 172.21.15.40:0/1436808022 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3b30005580 con 0x7f3b44101910 2024-03-18T22:59:30.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.737+0000 7f3b2bfff640 1 -- 172.21.15.40:0/1436808022 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v96) ==== 169+0+0 (secure 0 0 0) 0x7f3b380bdc70 con 0x7f3b44101910 2024-03-18T22:59:30.739 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-03-18T22:59:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.741+0000 7f3b4913b640 1 -- 172.21.15.40:0/1436808022 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3b2c0a2880 msgr2=0x7f3b2c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.741+0000 7f3b4913b640 1 --2- 172.21.15.40:0/1436808022 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3b2c0a2880 0x7f3b2c0a4d40 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3b44101770 tx=0x7f3b3400b390 comp rx=0 tx=0).stop 2024-03-18T22:59:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.741+0000 7f3b4913b640 1 -- 172.21.15.40:0/1436808022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b44101910 msgr2=0x7f3b44102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.741+0000 7f3b4913b640 1 --2- 172.21.15.40:0/1436808022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b44101910 0x7f3b44102cf0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3b38006010 tx=0x7f3b380042f0 comp rx=0 tx=0).stop 2024-03-18T22:59:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.741+0000 7f3b4913b640 1 -- 172.21.15.40:0/1436808022 shutdown_connections 2024-03-18T22:59:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.741+0000 7f3b4913b640 1 --2- 172.21.15.40:0/1436808022 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f3b2c0a2880 0x7f3b2c0a4d40 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.741+0000 7f3b4913b640 1 --2- 172.21.15.40:0/1436808022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b441022d0 0x7f3b44103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.741+0000 7f3b4913b640 1 --2- 172.21.15.40:0/1436808022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b44101910 0x7f3b44102cf0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:30.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.741+0000 7f3b4913b640 1 --2- 172.21.15.40:0/1436808022 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f3b44100720 0x7f3b441046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.741+0000 7f3b4913b640 1 -- 172.21.15.40:0/1436808022 >> 172.21.15.40:0/1436808022 conn(0x7f3b440fbcb0 msgr2=0x7f3b440fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.742+0000 7f3b4913b640 1 -- 172.21.15.40:0/1436808022 shutdown_connections 2024-03-18T22:59:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.742+0000 7f3b4913b640 1 -- 172.21.15.40:0/1436808022 wait complete. 2024-03-18T22:59:30.759 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-18T22:59:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.910+0000 7f51ed777640 1 -- 172.21.15.40:0/753598960 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51e8101950 msgr2=0x7f51e8101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.910+0000 7f51ed777640 1 --2- 172.21.15.40:0/753598960 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51e8101950 0x7f51e8101dd0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009860 tx=0x7f51d802bb70 comp rx=0 tx=0).stop 2024-03-18T22:59:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.910+0000 7f51ed777640 1 -- 172.21.15.40:0/753598960 shutdown_connections 2024-03-18T22:59:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.910+0000 7f51ed777640 1 --2- 172.21.15.40:0/753598960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51e8102310 0x7f51e8108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.910+0000 7f51ed777640 1 --2- 172.21.15.40:0/753598960 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51e8101950 0x7f51e8101dd0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.910+0000 7f51ed777640 1 --2- 172.21.15.40:0/753598960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e8100760 0x7f51e8100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.910+0000 7f51ed777640 1 -- 172.21.15.40:0/753598960 >> 172.21.15.40:0/753598960 conn(0x7f51e80fbcd0 msgr2=0x7f51e80fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.910+0000 7f51ed777640 1 -- 172.21.15.40:0/753598960 shutdown_connections 2024-03-18T22:59:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.910+0000 7f51ed777640 1 -- 172.21.15.40:0/753598960 wait complete. 2024-03-18T22:59:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.911+0000 7f51ed777640 1 Processor -- start 2024-03-18T22:59:30.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.911+0000 7f51ed777640 1 -- start start 2024-03-18T22:59:30.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.911+0000 7f51ed777640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51e8100760 0x7f51e81046f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:30.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.911+0000 7f51ed777640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e8101950 0x7f51e8102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:30.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.911+0000 7f51ed777640 1 --2- >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51e8102310 0x7f51e8103270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.911+0000 7f51ed777640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 -- 0x7f51e810c290 con 0x7f51e8101950 2024-03-18T22:59:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.911+0000 7f51ed777640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f51e810c110 con 0x7f51e8100760 2024-03-18T22:59:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.911+0000 7f51ed777640 1 -- --> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] -- mon_getmap magic: 0 -- 0x7f51e810c410 con 0x7f51e8102310 2024-03-18T22:59:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.911+0000 7f51e67fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e8101950 0x7f51e8102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.911+0000 7f51e67fc640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e8101950 0x7f51e8102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57178/0 (socket says 172.21.15.40:57178) 2024-03-18T22:59:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.911+0000 7f51e67fc640 1 -- 172.21.15.40:0/951356792 learned_addr learned my addr 172.21.15.40:0/951356792 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-18T22:59:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.911+0000 7f51e77fe640 1 --2- 172.21.15.40:0/951356792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51e8102310 0x7f51e8103270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.912+0000 7f51e6ffd640 1 --2- 172.21.15.40:0/951356792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51e8100760 0x7f51e81046f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.912+0000 7f51e67fc640 1 -- 172.21.15.40:0/951356792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51e8102310 msgr2=0x7f51e8103270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.912+0000 7f51e67fc640 1 --2- 172.21.15.40:0/951356792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51e8102310 0x7f51e8103270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.912+0000 7f51e67fc640 1 -- 172.21.15.40:0/951356792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51e8100760 msgr2=0x7f51e81046f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.912+0000 7f51e67fc640 1 --2- 172.21.15.40:0/951356792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51e8100760 0x7f51e81046f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.912+0000 7f51e67fc640 1 -- 172.21.15.40:0/951356792 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e8103b00 con 0x7f51e8101950 2024-03-18T22:59:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.912+0000 7f51e6ffd640 1 --2- 172.21.15.40:0/951356792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51e8100760 0x7f51e81046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T22:59:30.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.912+0000 7f51e67fc640 1 --2- 172.21.15.40:0/951356792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e8101950 0x7f51e8102d30 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f51d8005940 tx=0x7f51d80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:30.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.912+0000 7f51cffff640 1 -- 172.21.15.40:0/951356792 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d8041020 con 0x7f51e8101950 2024-03-18T22:59:30.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.912+0000 7f51ed777640 1 -- 172.21.15.40:0/951356792 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51e81a1be0 con 0x7f51e8101950 2024-03-18T22:59:30.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.912+0000 7f51ed777640 1 -- 172.21.15.40:0/951356792 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51e81a2180 con 0x7f51e8101950 2024-03-18T22:59:30.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.912+0000 7f51cffff640 1 -- 172.21.15.40:0/951356792 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51d802cbc0 con 0x7f51e8101950 2024-03-18T22:59:30.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.912+0000 7f51cffff640 1 -- 172.21.15.40:0/951356792 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d8039040 con 0x7f51e8101950 2024-03-18T22:59:30.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.914+0000 7f51ed777640 1 -- 172.21.15.40:0/951356792 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51d4005290 con 0x7f51e8101950 2024-03-18T22:59:30.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.916+0000 7f51cffff640 1 -- 172.21.15.40:0/951356792 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f51d8044020 con 0x7f51e8101950 2024-03-18T22:59:30.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.916+0000 7f51cffff640 1 --2- 172.21.15.40:0/951356792 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51d00a2800 0x7f51d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T22:59:30.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.916+0000 7f51cffff640 1 -- 172.21.15.40:0/951356792 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8496+0+0 (secure 0 0 0) 0x7f51d80efcf0 con 0x7f51e8101950 2024-03-18T22:59:30.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.917+0000 7f51e6ffd640 1 --2- 172.21.15.40:0/951356792 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51d00a2800 0x7f51d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T22:59:30.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.917+0000 7f51e6ffd640 1 --2- 172.21.15.40:0/951356792 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51d00a2800 0x7f51d00a4cc0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f51e81017b0 tx=0x7f51dc009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T22:59:30.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:30.923+0000 7f51cffff640 1 -- 172.21.15.40:0/951356792 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f51d804b030 con 0x7f51e8101950 2024-03-18T22:59:31.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:30 smithi113 ceph-mon[66775]: pgmap v857: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-03-18T22:59:31.079 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:30 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/2367776497' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T22:59:31.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:30 smithi113 ceph-mon[66775]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T22:59:31.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:30 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/1436808022' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T22:59:31.080 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:30 smithi113 ceph-mon[66775]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T22:59:31.108 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:30 smithi103 ceph-mon[66377]: pgmap v857: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-03-18T22:59:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:30 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/2367776497' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T22:59:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:30 smithi103 ceph-mon[66377]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T22:59:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:30 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/1436808022' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T22:59:31.109 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:30 smithi103 ceph-mon[66377]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T22:59:31.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:31.110+0000 7f51ed777640 1 -- 172.21.15.40:0/951356792 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f51d4005580 con 0x7f51e8101950 2024-03-18T22:59:31.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:30 smithi040 ceph-mon[62643]: pgmap v857: 129 pgs: 129 active+clean; 14 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-03-18T22:59:31.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:30 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/2367776497' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T22:59:31.143 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:30 smithi040 ceph-mon[62643]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T22:59:31.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:30 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/1436808022' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T22:59:31.144 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:30 smithi040 ceph-mon[62643]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T22:59:31.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:31.740+0000 7f51cffff640 1 -- 172.21.15.40:0/951356792 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v97) ==== 178+0+0 (secure 0 0 0) 0x7f51d80bcc90 con 0x7f51e8101950 2024-03-18T22:59:31.741 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data_ec' removed 2024-03-18T22:59:31.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:31.742+0000 7f51ed777640 1 -- 172.21.15.40:0/951356792 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51d00a2800 msgr2=0x7f51d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:31.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:31.742+0000 7f51ed777640 1 --2- 172.21.15.40:0/951356792 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51d00a2800 0x7f51d00a4cc0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f51e81017b0 tx=0x7f51dc009260 comp rx=0 tx=0).stop 2024-03-18T22:59:31.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:31.742+0000 7f51ed777640 1 -- 172.21.15.40:0/951356792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e8101950 msgr2=0x7f51e8102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T22:59:31.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:31.742+0000 7f51ed777640 1 --2- 172.21.15.40:0/951356792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e8101950 0x7f51e8102d30 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f51d8005940 tx=0x7f51d80042f0 comp rx=0 tx=0).stop 2024-03-18T22:59:31.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:31.742+0000 7f51ed777640 1 -- 172.21.15.40:0/951356792 shutdown_connections 2024-03-18T22:59:31.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:31.743+0000 7f51ed777640 1 --2- 172.21.15.40:0/951356792 >> [v2:172.21.15.40:6800/3109254132,v1:172.21.15.40:6801/3109254132] conn(0x7f51d00a2800 0x7f51d00a4cc0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:31.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:31.743+0000 7f51ed777640 1 --2- 172.21.15.40:0/951356792 >> [v2:172.21.15.113:3300/0,v1:172.21.15.113:6789/0] conn(0x7f51e8102310 0x7f51e8103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:31.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:31.743+0000 7f51ed777640 1 --2- 172.21.15.40:0/951356792 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51e8101950 0x7f51e8102d30 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:31.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:31.743+0000 7f51ed777640 1 --2- 172.21.15.40:0/951356792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f51e8100760 0x7f51e81046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T22:59:31.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:31.743+0000 7f51ed777640 1 -- 172.21.15.40:0/951356792 >> 172.21.15.40:0/951356792 conn(0x7f51e80fbcd0 msgr2=0x7f51e8069bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T22:59:31.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:31.743+0000 7f51ed777640 1 -- 172.21.15.40:0/951356792 shutdown_connections 2024-03-18T22:59:31.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-18T22:59:31.743+0000 7f51ed777640 1 -- 172.21.15.40:0/951356792 wait complete. 2024-03-18T22:59:31.764 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T22:59:31.836 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T22:59:31.900 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T22:59:31.959 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:31 smithi113 ceph-mon[66775]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T22:59:31.959 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:31 smithi113 ceph-mon[66775]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T22:59:31.959 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:31 smithi113 ceph-mon[66775]: from='client.? 172.21.15.40:0/951356792' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T22:59:31.961 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-18T22:59:31.961 DEBUG:teuthology.orchestra.run.smithi040:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T22:59:31.994 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T22:59:32.009 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:31 smithi103 ceph-mon[66377]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T22:59:32.009 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:31 smithi103 ceph-mon[66377]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T22:59:32.009 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:31 smithi103 ceph-mon[66377]: from='client.? 172.21.15.40:0/951356792' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T22:59:32.027 DEBUG:teuthology.orchestra.run.smithi113:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T22:59:32.045 INFO:tasks.cephadm:Stopping all daemons... 2024-03-18T22:59:32.045 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-18T22:59:32.045 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.a 2024-03-18T22:59:32.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:31 smithi040 ceph-mon[62643]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T22:59:32.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:31 smithi040 ceph-mon[62643]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T22:59:32.069 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:31 smithi040 ceph-mon[62643]: from='client.? 172.21.15.40:0/951356792' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T22:59:32.393 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:32 smithi040 systemd[1]: Stopping Ceph mon.a for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:59:32.893 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:32 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a[62639]: 2024-03-18T22:59:32.611+0000 7f3553568700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T22:59:32.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:32 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a[62639]: 2024-03-18T22:59:32.611+0000 7f3553568700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-18T22:59:32.894 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:32 smithi040 podman[150751]: 2024-03-18 22:59:32.65414716 +0000 UTC m=+0.194596047 container died 6bd216c61bbfb5251c558b872e05d9af454c3acbc349d40bdd04261e1280e3ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-18T22:59:33.503 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:33 smithi040 podman[150751]: 2024-03-18 22:59:33.160073213 +0000 UTC m=+0.700522104 container cleanup 6bd216c61bbfb5251c558b872e05d9af454c3acbc349d40bdd04261e1280e3ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-18T22:59:33.504 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:33 smithi040 bash[150751]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a 2024-03-18T22:59:33.704 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.a.service' 2024-03-18T22:59:33.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:33 smithi040 podman[150763]: 2024-03-18 22:59:33.503865202 +0000 UTC m=+0.848800431 container remove 6bd216c61bbfb5251c558b872e05d9af454c3acbc349d40bdd04261e1280e3ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-18T22:59:33.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:33 smithi040 systemd[1]: ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-18T22:59:33.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:33 smithi040 systemd[1]: Stopped Ceph mon.a for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T22:59:33.768 INFO:journalctl@ceph.mon.a.smithi040.stdout:Mar 18 22:59:33 smithi040 systemd[1]: ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.a.service: Consumed 24.602s CPU time. 2024-03-18T22:59:33.805 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T22:59:33.805 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-18T22:59:33.805 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-18T22:59:33.805 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.b 2024-03-18T22:59:34.108 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:33 smithi103 systemd[1]: Stopping Ceph mon.b for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:59:34.559 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:34 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-b[66371]: 2024-03-18T22:59:34.194+0000 7f7089625700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T22:59:34.559 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:34 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-b[66371]: 2024-03-18T22:59:34.194+0000 7f7089625700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-18T22:59:34.559 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:34 smithi103 podman[101009]: 2024-03-18 22:59:34.296697173 +0000 UTC m=+0.221835147 container died c4d19a7c8bce501eaa9ac8915650d185f51530098229c2d32188d2f86154f423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-18T22:59:34.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:34 smithi103 podman[101009]: 2024-03-18 22:59:34.661986649 +0000 UTC m=+0.587124626 container cleanup c4d19a7c8bce501eaa9ac8915650d185f51530098229c2d32188d2f86154f423 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-18T22:59:34.859 INFO:journalctl@ceph.mon.b.smithi103.stdout:Mar 18 22:59:34 smithi103 bash[101009]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-b 2024-03-18T22:59:35.087 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.b.service' 2024-03-18T22:59:35.146 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T22:59:35.147 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-18T22:59:35.147 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-18T22:59:35.147 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.c 2024-03-18T22:59:35.509 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:35 smithi113 systemd[1]: Stopping Ceph mon.c for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:59:35.829 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:35 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-c[66755]: 2024-03-18T22:59:35.508+0000 7ff702785700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T22:59:35.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:35 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-c[66755]: 2024-03-18T22:59:35.508+0000 7ff702785700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-18T22:59:35.830 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:35 smithi113 podman[104257]: 2024-03-18 22:59:35.723701175 +0000 UTC m=+0.335043514 container died 3c9a2ff1f2041c946902d53fa437fcd818a051ccfd170fd7eadca1acdf7d4ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-03-18T22:59:36.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:36 smithi113 podman[104257]: 2024-03-18 22:59:36.134922476 +0000 UTC m=+0.746264794 container cleanup 3c9a2ff1f2041c946902d53fa437fcd818a051ccfd170fd7eadca1acdf7d4ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-18T22:59:36.329 INFO:journalctl@ceph.mon.c.smithi113.stdout:Mar 18 22:59:36 smithi113 bash[104257]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mon-c 2024-03-18T22:59:36.571 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mon.c.service' 2024-03-18T22:59:36.627 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T22:59:36.627 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-18T22:59:36.627 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-18T22:59:36.628 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mgr.x 2024-03-18T22:59:36.846 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:59:36 smithi040 systemd[1]: Stopping Ceph mgr.x for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:59:37.392 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:59:37 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x[62875]: 2024-03-18T22:59:37.097+0000 7fa7ea087700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T22:59:37.392 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:59:37 smithi040 podman[150865]: 2024-03-18 22:59:37.130720857 +0000 UTC m=+0.305165877 container died ea399b57c2c6d8d33433f00ec9eabf65ebeb05ad9a15a5e4c55dee1e6f40a6fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-18T22:59:37.643 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:59:37 smithi040 podman[150865]: 2024-03-18 22:59:37.491793441 +0000 UTC m=+0.666238460 container cleanup ea399b57c2c6d8d33433f00ec9eabf65ebeb05ad9a15a5e4c55dee1e6f40a6fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T22:59:37.643 INFO:journalctl@ceph.mgr.x.smithi040.stdout:Mar 18 22:59:37 smithi040 bash[150865]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-x 2024-03-18T22:59:37.929 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-18T22:59:37.968 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T22:59:37.968 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-18T22:59:37.968 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-18T22:59:37.968 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mgr.y 2024-03-18T22:59:38.295 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:59:38 smithi103 systemd[1]: Stopping Ceph mgr.y for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:59:38.601 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:59:38 smithi103 podman[101134]: 2024-03-18 22:59:38.295757021 +0000 UTC m=+0.137846145 container died 1cad52968eff6701ce3e140fd0a99616d9451d84a5cd6e79d705268bb1ebea9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True) 2024-03-18T22:59:38.859 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:59:38 smithi103 podman[101134]: 2024-03-18 22:59:38.684320436 +0000 UTC m=+0.526409563 container cleanup 1cad52968eff6701ce3e140fd0a99616d9451d84a5cd6e79d705268bb1ebea9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-18T22:59:38.859 INFO:journalctl@ceph.mgr.y.smithi103.stdout:Mar 18 22:59:38 smithi103 bash[101134]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-y 2024-03-18T22:59:39.248 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-18T22:59:39.285 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T22:59:39.285 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-18T22:59:39.285 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-18T22:59:39.285 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mgr.z 2024-03-18T22:59:39.579 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:59:39 smithi113 systemd[1]: Stopping Ceph mgr.z for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:59:39.954 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:59:39 smithi113 podman[104381]: 2024-03-18 22:59:39.635542928 +0000 UTC m=+0.142213193 container died 94352a482c0bd225209cd86341d3ef289c9236b1a5409f7e84b674822f2a1add (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-18T22:59:40.292 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:59:40 smithi113 podman[104381]: 2024-03-18 22:59:40.041063813 +0000 UTC m=+0.547734119 container cleanup 94352a482c0bd225209cd86341d3ef289c9236b1a5409f7e84b674822f2a1add (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-18T22:59:40.293 INFO:journalctl@ceph.mgr.z.smithi113.stdout:Mar 18 22:59:40 smithi113 bash[104381]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mgr-z 2024-03-18T22:59:40.410 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-18T22:59:40.446 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T22:59:40.446 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-18T22:59:40.447 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-18T22:59:40.447 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.0 2024-03-18T22:59:40.893 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:59:40 smithi040 systemd[1]: Stopping Ceph osd.0 for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:59:41.393 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:59:40 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-0[74673]: 2024-03-18T22:59:40.895+0000 7f0332b22700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T22:59:41.394 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:59:40 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-0[74673]: 2024-03-18T22:59:40.895+0000 7f0332b22700 -1 osd.0 97 *** Got signal Terminated *** 2024-03-18T22:59:41.394 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:59:40 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-0[74673]: 2024-03-18T22:59:40.895+0000 7f0332b22700 -1 osd.0 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T22:59:46.640 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:59:46 smithi040 podman[150986]: 2024-03-18 22:59:46.350936435 +0000 UTC m=+5.712535920 container died c8b687eb004296dcddb07c0da7a503facfcb3d7a6468d05b38b7d0bd2513abe8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True) 2024-03-18T22:59:46.893 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:59:46 smithi040 podman[150986]: 2024-03-18 22:59:46.739285506 +0000 UTC m=+6.100884995 container cleanup c8b687eb004296dcddb07c0da7a503facfcb3d7a6468d05b38b7d0bd2513abe8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-18T22:59:46.893 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:59:46 smithi040 bash[150986]: ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-0 2024-03-18T22:59:47.157 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:59:47 smithi040 podman[151005]: 2024-03-18 22:59:47.041308344 +0000 UTC m=+0.666945888 container remove c8b687eb004296dcddb07c0da7a503facfcb3d7a6468d05b38b7d0bd2513abe8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-18T22:59:47.695 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:59:47 smithi040 podman[151102]: 2024-03-18 22:59:47.313152591 +0000 UTC m=+0.017742521 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T22:59:47.696 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:59:47 smithi040 podman[151102]: 2024-03-18 22:59:47.428426321 +0000 UTC m=+0.133016240 container create 7036b64f8f8a3d60d1be34448459125027e724e413d2aab9ec3c46b5e8c05927 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-18T22:59:47.696 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:59:47 smithi040 podman[151102]: 2024-03-18 22:59:47.695646978 +0000 UTC m=+0.400236907 container init 7036b64f8f8a3d60d1be34448459125027e724e413d2aab9ec3c46b5e8c05927 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-03-18T22:59:48.143 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:59:47 smithi040 podman[151102]: 2024-03-18 22:59:47.699318206 +0000 UTC m=+0.403908123 container start 7036b64f8f8a3d60d1be34448459125027e724e413d2aab9ec3c46b5e8c05927 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-18T22:59:48.143 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:59:47 smithi040 podman[151102]: 2024-03-18 22:59:47.778807364 +0000 UTC m=+0.483397287 container attach 7036b64f8f8a3d60d1be34448459125027e724e413d2aab9ec3c46b5e8c05927 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T22:59:48.643 INFO:journalctl@ceph.osd.0.smithi040.stdout:Mar 18 22:59:48 smithi040 podman[151102]: 2024-03-18 22:59:48.237781934 +0000 UTC m=+0.942371854 container died 7036b64f8f8a3d60d1be34448459125027e724e413d2aab9ec3c46b5e8c05927 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-18T22:59:49.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.0.service' 2024-03-18T22:59:49.063 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T22:59:49.064 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-18T22:59:49.064 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-18T22:59:49.064 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.1 2024-03-18T22:59:49.359 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:59:49 smithi103 systemd[1]: Stopping Ceph osd.1 for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:59:49.859 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:59:49 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-1[70958]: 2024-03-18T22:59:49.563+0000 7f50a9a97700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T22:59:49.859 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:59:49 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-1[70958]: 2024-03-18T22:59:49.563+0000 7f50a9a97700 -1 osd.1 97 *** Got signal Terminated *** 2024-03-18T22:59:49.859 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:59:49 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-1[70958]: 2024-03-18T22:59:49.563+0000 7f50a9a97700 -1 osd.1 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T22:59:55.524 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:59:55 smithi103 podman[101257]: 2024-03-18 22:59:55.208835986 +0000 UTC m=+5.840180351 container died f3670ae93f49e3d9747d3fc98f66d6cc8e0813736fa104573344a1c95b1108f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T22:59:55.858 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:59:55 smithi103 podman[101257]: 2024-03-18 22:59:55.623513361 +0000 UTC m=+6.254857745 container cleanup f3670ae93f49e3d9747d3fc98f66d6cc8e0813736fa104573344a1c95b1108f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-1, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-18T22:59:55.858 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:59:55 smithi103 bash[101257]: ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-1 2024-03-18T22:59:56.108 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:59:55 smithi103 podman[101274]: 2024-03-18 22:59:55.858426453 +0000 UTC m=+0.651521580 container remove f3670ae93f49e3d9747d3fc98f66d6cc8e0813736fa104573344a1c95b1108f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-1, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=) 2024-03-18T22:59:56.547 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:59:56 smithi103 podman[101369]: 2024-03-18 22:59:56.189871622 +0000 UTC m=+0.018158388 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T22:59:56.548 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:59:56 smithi103 podman[101369]: 2024-03-18 22:59:56.301471625 +0000 UTC m=+0.129758379 container create 7fab98fc1afcf5a442ac23e577c628965b7bf19e00ba20be4bde14c4b958df16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-18T22:59:56.859 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:59:56 smithi103 podman[101369]: 2024-03-18 22:59:56.547898109 +0000 UTC m=+0.376184877 container init 7fab98fc1afcf5a442ac23e577c628965b7bf19e00ba20be4bde14c4b958df16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-18T22:59:56.859 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:59:56 smithi103 podman[101369]: 2024-03-18 22:59:56.551167055 +0000 UTC m=+0.379453808 container start 7fab98fc1afcf5a442ac23e577c628965b7bf19e00ba20be4bde14c4b958df16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-18T22:59:56.859 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:59:56 smithi103 podman[101369]: 2024-03-18 22:59:56.714737137 +0000 UTC m=+0.543023900 container attach 7fab98fc1afcf5a442ac23e577c628965b7bf19e00ba20be4bde14c4b958df16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-18T22:59:58.109 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:59:57 smithi103 podman[101369]: 2024-03-18 22:59:57.909348769 +0000 UTC m=+1.737635539 container died 7fab98fc1afcf5a442ac23e577c628965b7bf19e00ba20be4bde14c4b958df16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-18T22:59:58.511 INFO:journalctl@ceph.osd.1.smithi103.stdout:Mar 18 22:59:58 smithi103 podman[101399]: 2024-03-18 22:59:58.494734507 +0000 UTC m=+0.580090391 container remove 7fab98fc1afcf5a442ac23e577c628965b7bf19e00ba20be4bde14c4b958df16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T22:59:58.514 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.1.service' 2024-03-18T22:59:58.555 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T22:59:58.555 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-18T22:59:58.556 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-18T22:59:58.556 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.2 2024-03-18T22:59:58.912 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 22:59:58 smithi113 systemd[1]: Stopping Ceph osd.2 for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T22:59:59.330 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 22:59:58 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-2[71925]: 2024-03-18T22:59:58.910+0000 7f15b53b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T22:59:59.330 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 22:59:58 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-2[71925]: 2024-03-18T22:59:58.910+0000 7f15b53b9700 -1 osd.2 97 *** Got signal Terminated *** 2024-03-18T22:59:59.330 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 22:59:58 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-2[71925]: 2024-03-18T22:59:58.910+0000 7f15b53b9700 -1 osd.2 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T23:00:04.796 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 23:00:04 smithi113 podman[104508]: 2024-03-18 23:00:04.339691532 +0000 UTC m=+5.553727055 container died 0c44a058bdf84198c32c9abfee4bce7ac9429d885215d75fbeb69fcfa67801a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-18T23:00:05.080 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 23:00:04 smithi113 podman[104508]: 2024-03-18 23:00:04.89513365 +0000 UTC m=+6.109169173 container cleanup 0c44a058bdf84198c32c9abfee4bce7ac9429d885215d75fbeb69fcfa67801a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-18T23:00:05.080 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 23:00:04 smithi113 bash[104508]: ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-2 2024-03-18T23:00:05.432 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 23:00:05 smithi113 podman[104537]: 2024-03-18 23:00:05.138412552 +0000 UTC m=+0.758004426 container remove 0c44a058bdf84198c32c9abfee4bce7ac9429d885215d75fbeb69fcfa67801a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-18T23:00:05.734 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 23:00:05 smithi113 podman[104628]: 2024-03-18 23:00:05.334799988 +0000 UTC m=+0.019054868 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T23:00:05.734 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 23:00:05 smithi113 podman[104628]: 2024-03-18 23:00:05.456461397 +0000 UTC m=+0.140716270 container create cc84cb80ab84d090dcb144088e77dc62e8453cc21940c3bda948fedf41ac27ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS) 2024-03-18T23:00:05.735 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 23:00:05 smithi113 podman[104628]: 2024-03-18 23:00:05.734590837 +0000 UTC m=+0.418845716 container init cc84cb80ab84d090dcb144088e77dc62e8453cc21940c3bda948fedf41ac27ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-18T23:00:06.079 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 23:00:05 smithi113 podman[104628]: 2024-03-18 23:00:05.738318324 +0000 UTC m=+0.422573205 container start cc84cb80ab84d090dcb144088e77dc62e8453cc21940c3bda948fedf41ac27ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-03-18T23:00:06.080 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 23:00:05 smithi113 podman[104628]: 2024-03-18 23:00:05.884784119 +0000 UTC m=+0.569039037 container attach cc84cb80ab84d090dcb144088e77dc62e8453cc21940c3bda948fedf41ac27ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-18T23:00:07.004 INFO:journalctl@ceph.osd.2.smithi113.stdout:Mar 18 23:00:06 smithi113 podman[104628]: 2024-03-18 23:00:06.698558979 +0000 UTC m=+1.382813856 container died cc84cb80ab84d090dcb144088e77dc62e8453cc21940c3bda948fedf41ac27ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-18T23:00:07.313 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.2.service' 2024-03-18T23:00:07.356 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:00:07.356 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-18T23:00:07.356 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-18T23:00:07.356 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.3 2024-03-18T23:00:07.643 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 23:00:07 smithi040 systemd[1]: Stopping Ceph osd.3 for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:00:08.143 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 23:00:07 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-3[81228]: 2024-03-18T23:00:07.735+0000 7f27dfb0b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:00:08.144 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 23:00:07 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-3[81228]: 2024-03-18T23:00:07.735+0000 7f27dfb0b700 -1 osd.3 97 *** Got signal Terminated *** 2024-03-18T23:00:08.144 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 23:00:07 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-3[81228]: 2024-03-18T23:00:07.735+0000 7f27dfb0b700 -1 osd.3 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T23:00:13.643 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 23:00:13 smithi040 podman[151211]: 2024-03-18 23:00:13.222973024 +0000 UTC m=+5.665569338 container died ee8635ed7f43d6a0c14dd783e78d65796927560fd0abb237112e610bcc94488b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-3, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-18T23:00:13.955 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 23:00:13 smithi040 podman[151211]: 2024-03-18 23:00:13.753299458 +0000 UTC m=+6.195895775 container cleanup ee8635ed7f43d6a0c14dd783e78d65796927560fd0abb237112e610bcc94488b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-3, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-18T23:00:13.955 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 23:00:13 smithi040 bash[151211]: ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-3 2024-03-18T23:00:14.356 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 23:00:13 smithi040 podman[151240]: 2024-03-18 23:00:13.95506416 +0000 UTC m=+0.722713046 container remove ee8635ed7f43d6a0c14dd783e78d65796927560fd0abb237112e610bcc94488b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-18T23:00:14.626 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 23:00:14 smithi040 podman[151331]: 2024-03-18 23:00:14.259429416 +0000 UTC m=+0.017866256 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T23:00:14.626 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 23:00:14 smithi040 podman[151331]: 2024-03-18 23:00:14.381226643 +0000 UTC m=+0.139663508 container create 2261a0d75ba71918fa9540dd20b64b1477e4262b3c4e4f2036567c91b42c30b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-3-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T23:00:14.893 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 23:00:14 smithi040 podman[151331]: 2024-03-18 23:00:14.62580385 +0000 UTC m=+0.384240691 container init 2261a0d75ba71918fa9540dd20b64b1477e4262b3c4e4f2036567c91b42c30b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-3-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-03-18T23:00:14.893 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 23:00:14 smithi040 podman[151331]: 2024-03-18 23:00:14.629276983 +0000 UTC m=+0.387713814 container start 2261a0d75ba71918fa9540dd20b64b1477e4262b3c4e4f2036567c91b42c30b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T23:00:14.893 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 23:00:14 smithi040 podman[151331]: 2024-03-18 23:00:14.727008666 +0000 UTC m=+0.485445506 container attach 2261a0d75ba71918fa9540dd20b64b1477e4262b3c4e4f2036567c91b42c30b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T23:00:15.246 INFO:journalctl@ceph.osd.3.smithi040.stdout:Mar 18 23:00:14 smithi040 podman[151331]: 2024-03-18 23:00:14.925533071 +0000 UTC m=+0.683969889 container died 2261a0d75ba71918fa9540dd20b64b1477e4262b3c4e4f2036567c91b42c30b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T23:00:15.563 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.3.service' 2024-03-18T23:00:15.610 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:00:15.610 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-18T23:00:15.610 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-18T23:00:15.610 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.4 2024-03-18T23:00:15.990 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 23:00:15 smithi103 systemd[1]: Stopping Ceph osd.4 for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:00:16.359 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 23:00:15 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-4[77442]: 2024-03-18T23:00:15.988+0000 7f969492a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:00:16.359 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 23:00:15 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-4[77442]: 2024-03-18T23:00:15.988+0000 7f969492a700 -1 osd.4 97 *** Got signal Terminated *** 2024-03-18T23:00:16.359 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 23:00:15 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-4[77442]: 2024-03-18T23:00:15.988+0000 7f969492a700 -1 osd.4 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T23:00:22.004 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 23:00:21 smithi103 podman[101487]: 2024-03-18 23:00:21.678411518 +0000 UTC m=+5.875359588 container died b151016593fd446d8fee68dbbda21e302fb7b9d221e7b21be5071b0788164f2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T23:00:22.349 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 23:00:22 smithi103 podman[101487]: 2024-03-18 23:00:22.097706141 +0000 UTC m=+6.294654211 container cleanup b151016593fd446d8fee68dbbda21e302fb7b9d221e7b21be5071b0788164f2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-4, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T23:00:22.350 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 23:00:22 smithi103 bash[101487]: ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-4 2024-03-18T23:00:22.608 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 23:00:22 smithi103 podman[101503]: 2024-03-18 23:00:22.349566949 +0000 UTC m=+0.672861282 container remove b151016593fd446d8fee68dbbda21e302fb7b9d221e7b21be5071b0788164f2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-4, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-18T23:00:22.996 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 23:00:22 smithi103 podman[101594]: 2024-03-18 23:00:22.74266734 +0000 UTC m=+0.112929686 container create 85f5579c5bd71311e24bbdd3981ede19f86667d36248fa7985835a132d524d50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T23:00:22.996 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 23:00:22 smithi103 podman[101594]: 2024-03-18 23:00:22.647560032 +0000 UTC m=+0.017822387 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T23:00:23.292 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 23:00:22 smithi103 podman[101594]: 2024-03-18 23:00:22.996069098 +0000 UTC m=+0.366331454 container init 85f5579c5bd71311e24bbdd3981ede19f86667d36248fa7985835a132d524d50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T23:00:23.292 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 23:00:22 smithi103 podman[101594]: 2024-03-18 23:00:22.999714144 +0000 UTC m=+0.369976490 container start 85f5579c5bd71311e24bbdd3981ede19f86667d36248fa7985835a132d524d50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-18T23:00:23.292 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 23:00:23 smithi103 podman[101594]: 2024-03-18 23:00:23.096248841 +0000 UTC m=+0.466511208 container attach 85f5579c5bd71311e24bbdd3981ede19f86667d36248fa7985835a132d524d50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T23:00:23.608 INFO:journalctl@ceph.osd.4.smithi103.stdout:Mar 18 23:00:23 smithi103 podman[101594]: 2024-03-18 23:00:23.291948297 +0000 UTC m=+0.662210645 container died 85f5579c5bd71311e24bbdd3981ede19f86667d36248fa7985835a132d524d50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-18T23:00:23.880 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.4.service' 2024-03-18T23:00:23.921 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:00:23.922 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-18T23:00:23.922 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-18T23:00:23.922 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.5 2024-03-18T23:00:24.240 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 23:00:23 smithi113 systemd[1]: Stopping Ceph osd.5 for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:00:24.579 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 23:00:24 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-5[78843]: 2024-03-18T23:00:24.239+0000 7f452e9b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:00:24.579 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 23:00:24 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-5[78843]: 2024-03-18T23:00:24.239+0000 7f452e9b1700 -1 osd.5 97 *** Got signal Terminated *** 2024-03-18T23:00:24.580 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 23:00:24 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-5[78843]: 2024-03-18T23:00:24.239+0000 7f452e9b1700 -1 osd.5 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T23:00:30.079 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 23:00:29 smithi113 podman[104733]: 2024-03-18 23:00:29.794275974 +0000 UTC m=+5.675177797 container died dd9c38d46eea3f4840a9e7adb04d8344dbd8f463922a12996714d765d8dc37b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-5, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2) 2024-03-18T23:00:30.579 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 23:00:30 smithi113 podman[104733]: 2024-03-18 23:00:30.314700735 +0000 UTC m=+6.195602564 container cleanup dd9c38d46eea3f4840a9e7adb04d8344dbd8f463922a12996714d765d8dc37b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-18T23:00:30.580 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 23:00:30 smithi113 bash[104733]: ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-5 2024-03-18T23:00:30.923 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 23:00:30 smithi113 podman[104751]: 2024-03-18 23:00:30.591178878 +0000 UTC m=+0.770191049 container remove dd9c38d46eea3f4840a9e7adb04d8344dbd8f463922a12996714d765d8dc37b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-18T23:00:31.229 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 23:00:30 smithi113 podman[104843]: 2024-03-18 23:00:30.830668699 +0000 UTC m=+0.021646646 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T23:00:31.229 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 23:00:30 smithi113 podman[104843]: 2024-03-18 23:00:30.933988194 +0000 UTC m=+0.124966136 container create cfec07388ff998269ff0106af5907b575e0c53a3b9bc442601c1f74797a7ad62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-18T23:00:31.229 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 23:00:31 smithi113 podman[104843]: 2024-03-18 23:00:31.170029326 +0000 UTC m=+0.361007278 container init cfec07388ff998269ff0106af5907b575e0c53a3b9bc442601c1f74797a7ad62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-03-18T23:00:31.229 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 23:00:31 smithi113 podman[104843]: 2024-03-18 23:00:31.173062219 +0000 UTC m=+0.364040162 container start cfec07388ff998269ff0106af5907b575e0c53a3b9bc442601c1f74797a7ad62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-18T23:00:31.229 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 23:00:31 smithi113 podman[104843]: 2024-03-18 23:00:31.228356554 +0000 UTC m=+0.419334505 container attach cfec07388ff998269ff0106af5907b575e0c53a3b9bc442601c1f74797a7ad62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T23:00:31.580 INFO:journalctl@ceph.osd.5.smithi113.stdout:Mar 18 23:00:31 smithi113 podman[104843]: 2024-03-18 23:00:31.463063603 +0000 UTC m=+0.654041550 container died cfec07388ff998269ff0106af5907b575e0c53a3b9bc442601c1f74797a7ad62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-5-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T23:00:32.076 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.5.service' 2024-03-18T23:00:32.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:00:32.118 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-18T23:00:32.118 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-18T23:00:32.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.6 2024-03-18T23:00:32.416 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 23:00:32 smithi040 systemd[1]: Stopping Ceph osd.6 for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:00:32.893 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 23:00:32 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-6[87962]: 2024-03-18T23:00:32.415+0000 7f740bf85700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:00:32.894 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 23:00:32 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-6[87962]: 2024-03-18T23:00:32.415+0000 7f740bf85700 -1 osd.6 97 *** Got signal Terminated *** 2024-03-18T23:00:32.894 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 23:00:32 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-6[87962]: 2024-03-18T23:00:32.415+0000 7f740bf85700 -1 osd.6 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T23:00:38.300 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 23:00:38 smithi040 podman[151436]: 2024-03-18 23:00:38.036813381 +0000 UTC m=+5.715379041 container died b6980037c0105e5d964c2f0f6e3c3e154f80fabedbaaf17722bfaa248b0d91aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-18T23:00:38.643 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 23:00:38 smithi040 podman[151436]: 2024-03-18 23:00:38.440795473 +0000 UTC m=+6.119361141 container cleanup b6980037c0105e5d964c2f0f6e3c3e154f80fabedbaaf17722bfaa248b0d91aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-6, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2) 2024-03-18T23:00:38.643 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 23:00:38 smithi040 bash[151436]: ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-6 2024-03-18T23:00:39.109 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 23:00:38 smithi040 podman[151453]: 2024-03-18 23:00:38.672457506 +0000 UTC m=+0.633042887 container remove b6980037c0105e5d964c2f0f6e3c3e154f80fabedbaaf17722bfaa248b0d91aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-6, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T23:00:39.371 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 23:00:39 smithi040 podman[151547]: 2024-03-18 23:00:39.011421381 +0000 UTC m=+0.018041473 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T23:00:39.371 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 23:00:39 smithi040 podman[151547]: 2024-03-18 23:00:39.110171805 +0000 UTC m=+0.116791877 container create 4875ba14a83b586de83c55dda8f4eb965c7eb88e26e0a7151ff15d3eed5123cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-18T23:00:39.372 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 23:00:39 smithi040 podman[151547]: 2024-03-18 23:00:39.371862961 +0000 UTC m=+0.378483039 container init 4875ba14a83b586de83c55dda8f4eb965c7eb88e26e0a7151ff15d3eed5123cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T23:00:39.643 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 23:00:39 smithi040 podman[151547]: 2024-03-18 23:00:39.375336824 +0000 UTC m=+0.381956896 container start 4875ba14a83b586de83c55dda8f4eb965c7eb88e26e0a7151ff15d3eed5123cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T23:00:39.644 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 23:00:39 smithi040 podman[151547]: 2024-03-18 23:00:39.446392648 +0000 UTC m=+0.453012747 container attach 4875ba14a83b586de83c55dda8f4eb965c7eb88e26e0a7151ff15d3eed5123cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T23:00:39.991 INFO:journalctl@ceph.osd.6.smithi040.stdout:Mar 18 23:00:39 smithi040 podman[151547]: 2024-03-18 23:00:39.679067721 +0000 UTC m=+0.685687785 container died 4875ba14a83b586de83c55dda8f4eb965c7eb88e26e0a7151ff15d3eed5123cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-18T23:00:40.302 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.6.service' 2024-03-18T23:00:40.346 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:00:40.346 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-18T23:00:40.347 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-18T23:00:40.347 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.7 2024-03-18T23:00:40.661 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 23:00:40 smithi103 systemd[1]: Stopping Ceph osd.7 for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:00:41.108 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 23:00:40 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-7[84132]: 2024-03-18T23:00:40.661+0000 7f4a82646700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:00:41.109 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 23:00:40 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-7[84132]: 2024-03-18T23:00:40.661+0000 7f4a82646700 -1 osd.7 97 *** Got signal Terminated *** 2024-03-18T23:00:41.109 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 23:00:40 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-7[84132]: 2024-03-18T23:00:40.661+0000 7f4a82646700 -1 osd.7 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T23:00:46.587 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 23:00:46 smithi103 podman[101700]: 2024-03-18 23:00:46.30860758 +0000 UTC m=+5.764442358 container died 68bd9931a8c13887bb80d808eef69295cff069885be817fe36a4f41cb7d8c15a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True) 2024-03-18T23:00:46.859 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 23:00:46 smithi103 podman[101700]: 2024-03-18 23:00:46.678545038 +0000 UTC m=+6.134379815 container cleanup 68bd9931a8c13887bb80d808eef69295cff069885be817fe36a4f41cb7d8c15a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-18T23:00:46.859 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 23:00:46 smithi103 bash[101700]: ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-7 2024-03-18T23:00:47.303 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 23:00:46 smithi103 podman[101730]: 2024-03-18 23:00:46.971935042 +0000 UTC m=+0.663747233 container remove 68bd9931a8c13887bb80d808eef69295cff069885be817fe36a4f41cb7d8c15a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-7, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T23:00:47.584 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 23:00:47 smithi103 podman[101824]: 2024-03-18 23:00:47.205949059 +0000 UTC m=+0.018496101 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T23:00:47.584 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 23:00:47 smithi103 podman[101824]: 2024-03-18 23:00:47.32279023 +0000 UTC m=+0.135337256 container create 29261836f4217e74b90c1c9ab0e5ad8fbaa42eecc69f2288b37db172734d760b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-18T23:00:47.859 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 23:00:47 smithi103 podman[101824]: 2024-03-18 23:00:47.58439199 +0000 UTC m=+0.396939032 container init 29261836f4217e74b90c1c9ab0e5ad8fbaa42eecc69f2288b37db172734d760b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD) 2024-03-18T23:00:47.859 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 23:00:47 smithi103 podman[101824]: 2024-03-18 23:00:47.587933003 +0000 UTC m=+0.400480032 container start 29261836f4217e74b90c1c9ab0e5ad8fbaa42eecc69f2288b37db172734d760b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-18T23:00:47.859 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 23:00:47 smithi103 podman[101824]: 2024-03-18 23:00:47.66707348 +0000 UTC m=+0.479620509 container attach 29261836f4217e74b90c1c9ab0e5ad8fbaa42eecc69f2288b37db172734d760b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T23:00:48.211 INFO:journalctl@ceph.osd.7.smithi103.stdout:Mar 18 23:00:47 smithi103 podman[101824]: 2024-03-18 23:00:47.875664944 +0000 UTC m=+0.688211965 container died 29261836f4217e74b90c1c9ab0e5ad8fbaa42eecc69f2288b37db172734d760b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-18T23:00:48.526 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.7.service' 2024-03-18T23:00:48.566 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:00:48.567 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-18T23:00:48.567 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-18T23:00:48.567 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.8 2024-03-18T23:00:49.082 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 23:00:48 smithi113 systemd[1]: Stopping Ceph osd.8 for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:00:49.082 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 23:00:48 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-8[85673]: 2024-03-18T23:00:48.841+0000 7f6ac3942700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:00:49.082 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 23:00:48 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-8[85673]: 2024-03-18T23:00:48.841+0000 7f6ac3942700 -1 osd.8 97 *** Got signal Terminated *** 2024-03-18T23:00:49.082 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 23:00:48 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-8[85673]: 2024-03-18T23:00:48.841+0000 7f6ac3942700 -1 osd.8 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T23:00:49.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:00:49 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:00:49.204+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:00:50.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:00:50 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:00:50.194+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:00:50.893 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:00:50 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:00:50.407+0000 7f6d0cea9700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:41.372447+0000 front 2024-03-18T22:59:41.372498+0000 (oldest deadline 2024-03-18T23:00:49.573297+0000) 2024-03-18T23:00:51.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:00:51 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:00:51.202+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:00:51.643 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:00:51 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:00:51.377+0000 7f6d0cea9700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:41.372447+0000 front 2024-03-18T22:59:41.372498+0000 (oldest deadline 2024-03-18T23:00:49.573297+0000) 2024-03-18T23:00:52.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:00:52 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:00:52.180+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:00:52.643 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:00:52 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:00:52.336+0000 7f6d0cea9700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:41.372447+0000 front 2024-03-18T22:59:41.372498+0000 (oldest deadline 2024-03-18T23:00:49.573297+0000) 2024-03-18T23:00:53.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:00:53 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:00:53.136+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:00:53.643 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:00:53 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:00:53.286+0000 7f6d0cea9700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:41.372447+0000 front 2024-03-18T22:59:41.372498+0000 (oldest deadline 2024-03-18T23:00:49.573297+0000) 2024-03-18T23:00:54.580 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 23:00:54 smithi113 podman[104948]: 2024-03-18 23:00:54.3917089 +0000 UTC m=+5.630685447 container died 4846d388d3c624b296d06b4beb2e7d851fd4009c41cb4f8c03c4a6f58d454e5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-8, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-18T23:00:54.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:00:54 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:00:54.164+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:00:54.643 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:00:54 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:00:54.306+0000 7f6d0cea9700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:41.372447+0000 front 2024-03-18T22:59:41.372498+0000 (oldest deadline 2024-03-18T23:00:49.573297+0000) 2024-03-18T23:00:55.033 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 23:00:54 smithi113 podman[104948]: 2024-03-18 23:00:54.807918109 +0000 UTC m=+6.046894663 container cleanup 4846d388d3c624b296d06b4beb2e7d851fd4009c41cb4f8c03c4a6f58d454e5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-8, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T23:00:55.034 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 23:00:54 smithi113 bash[104948]: ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-8 2024-03-18T23:00:55.330 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:00:55 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:00:55.178+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:00:55.330 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 23:00:55 smithi113 podman[104965]: 2024-03-18 23:00:55.034013429 +0000 UTC m=+0.636952544 container remove 4846d388d3c624b296d06b4beb2e7d851fd4009c41cb4f8c03c4a6f58d454e5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-8, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-03-18T23:00:55.643 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:00:55 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:00:55.289+0000 7f6d0cea9700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:41.372447+0000 front 2024-03-18T22:59:41.372498+0000 (oldest deadline 2024-03-18T23:00:49.573297+0000) 2024-03-18T23:00:55.737 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 23:00:55 smithi113 podman[105056]: 2024-03-18 23:00:55.374622839 +0000 UTC m=+0.018385191 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T23:00:55.737 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 23:00:55 smithi113 podman[105056]: 2024-03-18 23:00:55.49324566 +0000 UTC m=+0.137008057 container create 5880b52179227ad218350863d4ea3236f74f933dd2210e82c385d3c4ad5e49f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T23:00:56.030 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 23:00:55 smithi113 podman[105056]: 2024-03-18 23:00:55.737491662 +0000 UTC m=+0.381254018 container init 5880b52179227ad218350863d4ea3236f74f933dd2210e82c385d3c4ad5e49f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-18T23:00:56.030 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 23:00:55 smithi113 podman[105056]: 2024-03-18 23:00:55.741093395 +0000 UTC m=+0.384855724 container start 5880b52179227ad218350863d4ea3236f74f933dd2210e82c385d3c4ad5e49f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T23:00:56.030 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 23:00:55 smithi113 podman[105056]: 2024-03-18 23:00:55.820562391 +0000 UTC m=+0.464324738 container attach 5880b52179227ad218350863d4ea3236f74f933dd2210e82c385d3c4ad5e49f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True) 2024-03-18T23:00:56.309 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:00:56 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:00:56.165+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:00:56.309 INFO:journalctl@ceph.osd.8.smithi113.stdout:Mar 18 23:00:56 smithi113 podman[105056]: 2024-03-18 23:00:56.031162038 +0000 UTC m=+0.674924376 container died 5880b52179227ad218350863d4ea3236f74f933dd2210e82c385d3c4ad5e49f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS) 2024-03-18T23:00:56.643 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:00:56 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:00:56.287+0000 7f6d0cea9700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:41.372447+0000 front 2024-03-18T22:59:41.372498+0000 (oldest deadline 2024-03-18T23:00:49.573297+0000) 2024-03-18T23:00:56.668 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.8.service' 2024-03-18T23:00:56.708 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:00:56.708 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-18T23:00:56.709 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-18T23:00:56.709 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.9 2024-03-18T23:00:56.995 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:00:56 smithi040 systemd[1]: Stopping Ceph osd.9 for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:00:57.246 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:00:56 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:00:56.993+0000 7f6d108b5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:00:57.246 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:00:56 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:00:56.993+0000 7f6d108b5700 -1 osd.9 97 *** Got signal Terminated *** 2024-03-18T23:00:57.246 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:00:56 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:00:56.993+0000 7f6d108b5700 -1 osd.9 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T23:00:57.246 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:00:57 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:00:57.245+0000 7f6d0cea9700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:41.372447+0000 front 2024-03-18T22:59:41.372498+0000 (oldest deadline 2024-03-18T23:00:49.573297+0000) 2024-03-18T23:00:57.581 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:00:57 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:00:57.165+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:00:58.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:00:58 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:00:58.135+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:00:58.643 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:00:58 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:00:58.199+0000 7f6d0cea9700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:41.372447+0000 front 2024-03-18T22:59:41.372498+0000 (oldest deadline 2024-03-18T23:00:49.573297+0000) 2024-03-18T23:00:59.359 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:00:58 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:00:58.970+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:52.098075+0000 front 2024-03-18T22:59:52.098044+0000 (oldest deadline 2024-03-18T23:00:57.998233+0000) 2024-03-18T23:00:59.480 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:00:59 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:00:59.124+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:00:59.643 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:00:59 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:00:59.197+0000 7f6d0cea9700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:41.372447+0000 front 2024-03-18T22:59:41.372498+0000 (oldest deadline 2024-03-18T23:00:49.573297+0000) 2024-03-18T23:01:00.358 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:00:59 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:00:59.974+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:52.098075+0000 front 2024-03-18T22:59:52.098044+0000 (oldest deadline 2024-03-18T23:00:57.998233+0000) 2024-03-18T23:01:00.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:00 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:00.102+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:00.643 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:01:00 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:01:00.215+0000 7f6d0cea9700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:41.372447+0000 front 2024-03-18T22:59:41.372498+0000 (oldest deadline 2024-03-18T23:00:49.573297+0000) 2024-03-18T23:01:00.643 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:01:00 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:01:00.215+0000 7f6d0cea9700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:49.573880+0000 front 2024-03-18T22:59:49.573798+0000 (oldest deadline 2024-03-18T23:01:00.174402+0000) 2024-03-18T23:01:01.358 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:00 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:00.986+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:52.098075+0000 front 2024-03-18T22:59:52.098044+0000 (oldest deadline 2024-03-18T23:00:57.998233+0000) 2024-03-18T23:01:01.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:01 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:01.138+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:01.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:01 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:01.138+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:01.643 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:01:01 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:01:01.229+0000 7f6d0cea9700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:41.372447+0000 front 2024-03-18T22:59:41.372498+0000 (oldest deadline 2024-03-18T23:00:49.573297+0000) 2024-03-18T23:01:01.643 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:01:01 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9[94952]: 2024-03-18T23:01:01.229+0000 7f6d0cea9700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:49.573880+0000 front 2024-03-18T22:59:49.573798+0000 (oldest deadline 2024-03-18T23:01:00.174402+0000) 2024-03-18T23:01:02.359 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:02 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:02.029+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:52.098075+0000 front 2024-03-18T22:59:52.098044+0000 (oldest deadline 2024-03-18T23:00:57.998233+0000) 2024-03-18T23:01:02.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:02 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:02.112+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:02.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:02 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:02.112+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:02.893 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:01:02 smithi040 podman[151664]: 2024-03-18 23:01:02.520669291 +0000 UTC m=+5.612564034 container died 5aa5fddb7c22a9af0e814f7544893155e6aca59af42e297c7d3e312e5db26227 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-18T23:01:03.220 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:01:02 smithi040 podman[151664]: 2024-03-18 23:01:02.994381291 +0000 UTC m=+6.086276036 container cleanup 5aa5fddb7c22a9af0e814f7544893155e6aca59af42e297c7d3e312e5db26227 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-18T23:01:03.220 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:01:02 smithi040 bash[151664]: ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9 2024-03-18T23:01:03.329 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:03 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:03.074+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:03.329 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:03 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:03.074+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:03.358 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:03 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:03.059+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:52.098075+0000 front 2024-03-18T22:59:52.098044+0000 (oldest deadline 2024-03-18T23:00:57.998233+0000) 2024-03-18T23:01:03.521 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:01:03 smithi040 podman[151692]: 2024-03-18 23:01:03.220488026 +0000 UTC m=+0.701439063 container remove 5aa5fddb7c22a9af0e814f7544893155e6aca59af42e297c7d3e312e5db26227 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-03-18T23:01:03.775 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:01:03 smithi040 podman[151785]: 2024-03-18 23:01:03.52143348 +0000 UTC m=+0.100574290 container create 1e9d83dbb305f1ef97fd08787d7096dfb2e694f5f6c99a3f4b13a5a290b5d7f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-18T23:01:03.775 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:01:03 smithi040 podman[151785]: 2024-03-18 23:01:03.438553752 +0000 UTC m=+0.017694523 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T23:01:03.775 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:01:03 smithi040 podman[151785]: 2024-03-18 23:01:03.775634674 +0000 UTC m=+0.354775438 container init 1e9d83dbb305f1ef97fd08787d7096dfb2e694f5f6c99a3f4b13a5a290b5d7f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-18T23:01:04.080 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:01:03 smithi040 podman[151785]: 2024-03-18 23:01:03.77885359 +0000 UTC m=+0.357994359 container start 1e9d83dbb305f1ef97fd08787d7096dfb2e694f5f6c99a3f4b13a5a290b5d7f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux ) 2024-03-18T23:01:04.081 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:01:03 smithi040 podman[151785]: 2024-03-18 23:01:03.85828581 +0000 UTC m=+0.437426592 container attach 1e9d83dbb305f1ef97fd08787d7096dfb2e694f5f6c99a3f4b13a5a290b5d7f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-18T23:01:04.358 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:04 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:04.037+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:52.098075+0000 front 2024-03-18T22:59:52.098044+0000 (oldest deadline 2024-03-18T23:00:57.998233+0000) 2024-03-18T23:01:04.363 INFO:journalctl@ceph.osd.9.smithi040.stdout:Mar 18 23:01:04 smithi040 podman[151785]: 2024-03-18 23:01:04.081135386 +0000 UTC m=+0.660276160 container died 1e9d83dbb305f1ef97fd08787d7096dfb2e694f5f6c99a3f4b13a5a290b5d7f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-9-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T23:01:04.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:04 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:04.120+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:04.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:04 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:04.120+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:04.668 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.9.service' 2024-03-18T23:01:04.708 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:01:04.709 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-18T23:01:04.709 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-18T23:01:04.709 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.10 2024-03-18T23:01:04.990 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:04 smithi103 systemd[1]: Stopping Ceph osd.10 for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:01:05.359 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:04 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:04.988+0000 7f540a49b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:01:05.359 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:04 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:04.988+0000 7f540a49b700 -1 osd.10 97 *** Got signal Terminated *** 2024-03-18T23:01:05.359 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:04 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:04.988+0000 7f540a49b700 -1 osd.10 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T23:01:05.359 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:05 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:05.064+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:52.098075+0000 front 2024-03-18T22:59:52.098044+0000 (oldest deadline 2024-03-18T23:00:57.998233+0000) 2024-03-18T23:01:05.359 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:05 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:05.064+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T22:59:59.699179+0000 front 2024-03-18T22:59:59.699043+0000 (oldest deadline 2024-03-18T23:01:04.399234+0000) 2024-03-18T23:01:05.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:05 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:05.163+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:05.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:05 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:05.163+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:05.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:05 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:05.163+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T23:00:00.163129+0000 front 2024-03-18T23:00:00.163003+0000 (oldest deadline 2024-03-18T23:01:04.863161+0000) 2024-03-18T23:01:06.358 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:06 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:06.063+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:52.098075+0000 front 2024-03-18T22:59:52.098044+0000 (oldest deadline 2024-03-18T23:00:57.998233+0000) 2024-03-18T23:01:06.359 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:06 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:06.063+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T22:59:59.699179+0000 front 2024-03-18T22:59:59.699043+0000 (oldest deadline 2024-03-18T23:01:04.399234+0000) 2024-03-18T23:01:06.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:06 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:06.198+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:06.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:06 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:06.198+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:06.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:06 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:06.198+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T23:00:00.163129+0000 front 2024-03-18T23:00:00.163003+0000 (oldest deadline 2024-03-18T23:01:04.863161+0000) 2024-03-18T23:01:07.358 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:07 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:07.039+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:52.098075+0000 front 2024-03-18T22:59:52.098044+0000 (oldest deadline 2024-03-18T23:00:57.998233+0000) 2024-03-18T23:01:07.358 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:07 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:07.039+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T22:59:59.699179+0000 front 2024-03-18T22:59:59.699043+0000 (oldest deadline 2024-03-18T23:01:04.399234+0000) 2024-03-18T23:01:07.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:07 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:07.181+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:07.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:07 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:07.181+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:07.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:07 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:07.181+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T23:00:00.163129+0000 front 2024-03-18T23:00:00.163003+0000 (oldest deadline 2024-03-18T23:01:04.863161+0000) 2024-03-18T23:01:08.358 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:08 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:08.054+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:52.098075+0000 front 2024-03-18T22:59:52.098044+0000 (oldest deadline 2024-03-18T23:00:57.998233+0000) 2024-03-18T23:01:08.359 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:08 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:08.054+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T22:59:59.699179+0000 front 2024-03-18T22:59:59.699043+0000 (oldest deadline 2024-03-18T23:01:04.399234+0000) 2024-03-18T23:01:08.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:08 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:08.199+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:08.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:08 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:08.199+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:08.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:08 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:08.199+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T23:00:00.163129+0000 front 2024-03-18T23:00:00.163003+0000 (oldest deadline 2024-03-18T23:01:04.863161+0000) 2024-03-18T23:01:09.358 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:09 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:09.006+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:52.098075+0000 front 2024-03-18T22:59:52.098044+0000 (oldest deadline 2024-03-18T23:00:57.998233+0000) 2024-03-18T23:01:09.359 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:09 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10[90701]: 2024-03-18T23:01:09.006+0000 7f5402a87700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T22:59:59.699179+0000 front 2024-03-18T22:59:59.699043+0000 (oldest deadline 2024-03-18T23:01:04.399234+0000) 2024-03-18T23:01:09.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:09 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:09.241+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:09.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:09 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:09.241+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:09.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:09 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:09.241+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T23:00:00.163129+0000 front 2024-03-18T23:00:00.163003+0000 (oldest deadline 2024-03-18T23:01:04.863161+0000) 2024-03-18T23:01:10.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:10 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:10.230+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:10.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:10 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:10.230+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:10.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:10 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:10.230+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T23:00:00.163129+0000 front 2024-03-18T23:00:00.163003+0000 (oldest deadline 2024-03-18T23:01:04.863161+0000) 2024-03-18T23:01:10.740 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:10 smithi103 podman[101944]: 2024-03-18 23:01:10.437760473 +0000 UTC m=+5.532813841 container died e6156fe9b109dd34e9ce0ff201370dcefcc0e3d0c466c1c6980a465119875d84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T23:01:11.057 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:10 smithi103 podman[101944]: 2024-03-18 23:01:10.814785416 +0000 UTC m=+5.909838765 container cleanup e6156fe9b109dd34e9ce0ff201370dcefcc0e3d0c466c1c6980a465119875d84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-03-18T23:01:11.057 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:10 smithi103 bash[101944]: ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10 2024-03-18T23:01:11.358 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:11 smithi103 podman[101963]: 2024-03-18 23:01:11.057316187 +0000 UTC m=+0.611625853 container remove e6156fe9b109dd34e9ce0ff201370dcefcc0e3d0c466c1c6980a465119875d84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T23:01:11.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:11 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:11.256+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:11.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:11 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:11.256+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:11.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:11 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:11.256+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T23:00:00.163129+0000 front 2024-03-18T23:00:00.163003+0000 (oldest deadline 2024-03-18T23:01:04.863161+0000) 2024-03-18T23:01:11.619 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:11 smithi103 podman[102051]: 2024-03-18 23:01:11.265406353 +0000 UTC m=+0.017788241 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T23:01:11.619 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:11 smithi103 podman[102051]: 2024-03-18 23:01:11.366674049 +0000 UTC m=+0.119055944 container create d93a8021f81c25a9f0b91b13452d85475bf39abcd42a7cb3014b15ca6047854c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-03-18T23:01:11.619 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:11 smithi103 podman[102051]: 2024-03-18 23:01:11.619187322 +0000 UTC m=+0.371569217 container init d93a8021f81c25a9f0b91b13452d85475bf39abcd42a7cb3014b15ca6047854c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-18T23:01:11.921 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:11 smithi103 podman[102051]: 2024-03-18 23:01:11.622512131 +0000 UTC m=+0.374894029 container start d93a8021f81c25a9f0b91b13452d85475bf39abcd42a7cb3014b15ca6047854c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T23:01:11.922 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:11 smithi103 podman[102051]: 2024-03-18 23:01:11.719202325 +0000 UTC m=+0.471584216 container attach d93a8021f81c25a9f0b91b13452d85475bf39abcd42a7cb3014b15ca6047854c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T23:01:12.359 INFO:journalctl@ceph.osd.10.smithi103.stdout:Mar 18 23:01:11 smithi103 podman[102051]: 2024-03-18 23:01:11.922216735 +0000 UTC m=+0.674598626 container died d93a8021f81c25a9f0b91b13452d85475bf39abcd42a7cb3014b15ca6047854c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T23:01:12.488 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.10.service' 2024-03-18T23:01:12.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:01:12.531 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-18T23:01:12.531 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-18T23:01:12.531 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.11 2024-03-18T23:01:12.561 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:12 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:12.294+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:12.561 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:12 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:12.294+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:12.561 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:12 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:12.294+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T23:00:00.163129+0000 front 2024-03-18T23:00:00.163003+0000 (oldest deadline 2024-03-18T23:01:04.863161+0000) 2024-03-18T23:01:12.829 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:12 smithi113 systemd[1]: Stopping Ceph osd.11 for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:01:12.830 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:12 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:12.809+0000 7f2d2ea50700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:01:12.830 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:12 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:12.809+0000 7f2d2ea50700 -1 osd.11 97 *** Got signal Terminated *** 2024-03-18T23:01:12.830 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:12 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:12.809+0000 7f2d2ea50700 -1 osd.11 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T23:01:13.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:13 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:13.257+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:13.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:13 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:13.257+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:13.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:13 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:13.257+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T23:00:00.163129+0000 front 2024-03-18T23:00:00.163003+0000 (oldest deadline 2024-03-18T23:01:04.863161+0000) 2024-03-18T23:01:14.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:14 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:14.230+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:14.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:14 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:14.230+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:14.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:14 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:14.230+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T23:00:00.163129+0000 front 2024-03-18T23:00:00.163003+0000 (oldest deadline 2024-03-18T23:01:04.863161+0000) 2024-03-18T23:01:15.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:15 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:15.232+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:15.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:15 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:15.232+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:15.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:15 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:15.232+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T23:00:00.163129+0000 front 2024-03-18T23:00:00.163003+0000 (oldest deadline 2024-03-18T23:01:04.863161+0000) 2024-03-18T23:01:16.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:16 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:16.261+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:16.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:16 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:16.261+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:16.580 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:16 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:16.261+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T23:00:00.163129+0000 front 2024-03-18T23:00:00.163003+0000 (oldest deadline 2024-03-18T23:01:04.863161+0000) 2024-03-18T23:01:17.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:17 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:17.269+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.40:6806 osd.0 since back 2024-03-18T22:59:44.960985+0000 front 2024-03-18T22:59:44.961022+0000 (oldest deadline 2024-03-18T23:00:49.061021+0000) 2024-03-18T23:01:17.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:17 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:17.269+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-03-18T22:59:54.262801+0000 front 2024-03-18T22:59:54.262832+0000 (oldest deadline 2024-03-18T23:01:00.162724+0000) 2024-03-18T23:01:17.579 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:17 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11[92694]: 2024-03-18T23:01:17.269+0000 7f2d2703c700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.113:6804 osd.2 since back 2024-03-18T23:00:00.163129+0000 front 2024-03-18T23:00:00.163003+0000 (oldest deadline 2024-03-18T23:01:04.863161+0000) 2024-03-18T23:01:18.708 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:18 smithi113 podman[105184]: 2024-03-18 23:01:18.388222675 +0000 UTC m=+5.660714825 container died e2838393f202e46c6f9574b35f8490d46bcbc110c94a0592983d3602c4be6868 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-18T23:01:19.080 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:18 smithi113 podman[105184]: 2024-03-18 23:01:18.708877827 +0000 UTC m=+5.981369983 container cleanup e2838393f202e46c6f9574b35f8490d46bcbc110c94a0592983d3602c4be6868 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-18T23:01:19.080 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:18 smithi113 bash[105184]: ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11 2024-03-18T23:01:19.080 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:18 smithi113 podman[105203]: 2024-03-18 23:01:18.918466906 +0000 UTC m=+0.530208074 container remove e2838393f202e46c6f9574b35f8490d46bcbc110c94a0592983d3602c4be6868 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.vendor=CentOS) 2024-03-18T23:01:19.621 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:19 smithi113 podman[105297]: 2024-03-18 23:01:19.336162635 +0000 UTC m=+0.114389254 container create ce8855c0f55b796280ff4baa242e381b8a9079b15a491ee0460331ef28adb33d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-18T23:01:19.621 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:19 smithi113 podman[105297]: 2024-03-18 23:01:19.240128708 +0000 UTC m=+0.018355336 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-18T23:01:19.621 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:19 smithi113 podman[105297]: 2024-03-18 23:01:19.546753788 +0000 UTC m=+0.324980407 container init ce8855c0f55b796280ff4baa242e381b8a9079b15a491ee0460331ef28adb33d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-18T23:01:19.622 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:19 smithi113 podman[105297]: 2024-03-18 23:01:19.549927297 +0000 UTC m=+0.328153916 container start ce8855c0f55b796280ff4baa242e381b8a9079b15a491ee0460331ef28adb33d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-18T23:01:19.622 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:19 smithi113 podman[105297]: 2024-03-18 23:01:19.621563152 +0000 UTC m=+0.399789793 container attach ce8855c0f55b796280ff4baa242e381b8a9079b15a491ee0460331ef28adb33d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T23:01:20.079 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:19 smithi113 podman[105297]: 2024-03-18 23:01:19.838434484 +0000 UTC m=+0.616661104 container died ce8855c0f55b796280ff4baa242e381b8a9079b15a491ee0460331ef28adb33d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-18T23:01:20.324 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.11.service' 2024-03-18T23:01:20.390 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:20 smithi113 podman[105323]: 2024-03-18 23:01:20.308131672 +0000 UTC m=+0.461347122 container remove ce8855c0f55b796280ff4baa242e381b8a9079b15a491ee0460331ef28adb33d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-osd-11-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T23:01:20.390 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:20 smithi113 systemd[1]: ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.11.service: Deactivated successfully. 2024-03-18T23:01:20.390 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:20 smithi113 systemd[1]: Stopped Ceph osd.11 for 410ef414-e577-11ee-95c9-87774f69a715. 2024-03-18T23:01:20.391 INFO:journalctl@ceph.osd.11.smithi113.stdout:Mar 18 23:01:20 smithi113 systemd[1]: ceph-410ef414-e577-11ee-95c9-87774f69a715@osd.11.service: Consumed 6min 5.719s CPU time. 2024-03-18T23:01:20.406 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:01:20.406 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-18T23:01:20.406 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-18T23:01:20.406 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.a 2024-03-18T23:01:20.724 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 23:01:20 smithi040 systemd[1]: Stopping Ceph mds.a for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:01:21.143 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 23:01:20 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-a[101365]: 2024-03-18T23:01:20.722+0000 7fa739558700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:01:21.143 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 23:01:20 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-a[101365]: 2024-03-18T23:01:20.722+0000 7fa739558700 -1 mds.a *** got signal Terminated *** 2024-03-18T23:01:24.115 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 23:01:23 smithi040 podman[151901]: 2024-03-18 23:01:23.832064646 +0000 UTC m=+3.227315052 container died 555cd76a8073ffd1b6aa943eb519d0e052ff16a9d20f306f982c4a1114f86fcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-a, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-18T23:01:24.393 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 23:01:24 smithi040 podman[151901]: 2024-03-18 23:01:24.189339601 +0000 UTC m=+3.584590005 container cleanup 555cd76a8073ffd1b6aa943eb519d0e052ff16a9d20f306f982c4a1114f86fcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-a, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-18T23:01:24.393 INFO:journalctl@ceph.mds.a.smithi040.stdout:Mar 18 23:01:24 smithi040 bash[151901]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-a 2024-03-18T23:01:24.688 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.a.service' 2024-03-18T23:01:24.724 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:01:24.724 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-18T23:01:24.724 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-18T23:01:24.724 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.d 2024-03-18T23:01:25.143 INFO:journalctl@ceph.mds.d.smithi040.stdout:Mar 18 23:01:24 smithi040 systemd[1]: Stopping Ceph mds.d for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:01:25.143 INFO:journalctl@ceph.mds.d.smithi040.stdout:Mar 18 23:01:25 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-d[100403]: 2024-03-18T23:01:25.051+0000 7ff82070b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:01:25.143 INFO:journalctl@ceph.mds.d.smithi040.stdout:Mar 18 23:01:25 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-d[100403]: 2024-03-18T23:01:25.051+0000 7ff82070b700 -1 mds.d *** got signal Terminated *** 2024-03-18T23:01:27.426 INFO:journalctl@ceph.mds.d.smithi040.stdout:Mar 18 23:01:27 smithi040 podman[152022]: 2024-03-18 23:01:27.162728662 +0000 UTC m=+2.224555083 container died 9ac863149aff646b3173d74a875d4bbf4fd9ac7875ef92dc704e29981fdf6c4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-18T23:01:27.702 INFO:journalctl@ceph.mds.d.smithi040.stdout:Mar 18 23:01:27 smithi040 podman[152022]: 2024-03-18 23:01:27.426786456 +0000 UTC m=+2.488612876 container cleanup 9ac863149aff646b3173d74a875d4bbf4fd9ac7875ef92dc704e29981fdf6c4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-d, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T23:01:27.702 INFO:journalctl@ceph.mds.d.smithi040.stdout:Mar 18 23:01:27 smithi040 bash[152022]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-d 2024-03-18T23:01:27.841 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.d.service' 2024-03-18T23:01:27.876 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:01:27.876 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-18T23:01:27.876 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-18T23:01:27.877 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.g 2024-03-18T23:01:28.393 INFO:journalctl@ceph.mds.g.smithi040.stdout:Mar 18 23:01:27 smithi040 systemd[1]: Stopping Ceph mds.g for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:01:28.394 INFO:journalctl@ceph.mds.g.smithi040.stdout:Mar 18 23:01:28 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-g[101001]: 2024-03-18T23:01:28.205+0000 7f1328167700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:01:28.394 INFO:journalctl@ceph.mds.g.smithi040.stdout:Mar 18 23:01:28 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-g[101001]: 2024-03-18T23:01:28.205+0000 7f1328167700 -1 mds.g *** got signal Terminated *** 2024-03-18T23:01:32.141 INFO:journalctl@ceph.mds.g.smithi040.stdout:Mar 18 23:01:31 smithi040 podman[152144]: 2024-03-18 23:01:31.825332206 +0000 UTC m=+3.723966929 container died 0741cacc890bc0e91e744bef5b4ae26af380623f59de0f201f49f028f483b40e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-g, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T23:01:32.393 INFO:journalctl@ceph.mds.g.smithi040.stdout:Mar 18 23:01:32 smithi040 podman[152144]: 2024-03-18 23:01:32.222575205 +0000 UTC m=+4.121209951 container cleanup 0741cacc890bc0e91e744bef5b4ae26af380623f59de0f201f49f028f483b40e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-g, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-18T23:01:32.393 INFO:journalctl@ceph.mds.g.smithi040.stdout:Mar 18 23:01:32 smithi040 bash[152144]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-g 2024-03-18T23:01:32.599 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.g.service' 2024-03-18T23:01:32.634 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:01:32.634 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-18T23:01:32.634 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-18T23:01:32.634 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.j 2024-03-18T23:01:32.893 INFO:journalctl@ceph.mds.j.smithi040.stdout:Mar 18 23:01:32 smithi040 systemd[1]: Stopping Ceph mds.j for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:01:33.393 INFO:journalctl@ceph.mds.j.smithi040.stdout:Mar 18 23:01:32 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-j[100052]: 2024-03-18T23:01:32.984+0000 7f8697427700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:01:33.394 INFO:journalctl@ceph.mds.j.smithi040.stdout:Mar 18 23:01:32 smithi040 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-j[100052]: 2024-03-18T23:01:32.984+0000 7f8697427700 -1 mds.j *** got signal Terminated *** 2024-03-18T23:01:36.585 INFO:journalctl@ceph.mds.j.smithi040.stdout:Mar 18 23:01:36 smithi040 podman[152264]: 2024-03-18 23:01:36.31518743 +0000 UTC m=+3.458598773 container died 312e652f8959935dc31afb21cfcd6e841f95e598c04a466acd1a93039d406396 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-j, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-18T23:01:36.893 INFO:journalctl@ceph.mds.j.smithi040.stdout:Mar 18 23:01:36 smithi040 podman[152264]: 2024-03-18 23:01:36.684528003 +0000 UTC m=+3.827939345 container cleanup 312e652f8959935dc31afb21cfcd6e841f95e598c04a466acd1a93039d406396 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-j, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T23:01:36.893 INFO:journalctl@ceph.mds.j.smithi040.stdout:Mar 18 23:01:36 smithi040 bash[152264]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-j 2024-03-18T23:01:37.068 DEBUG:teuthology.orchestra.run.smithi040:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.j.service' 2024-03-18T23:01:37.101 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:01:37.101 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-18T23:01:37.101 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-18T23:01:37.101 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.b 2024-03-18T23:01:37.383 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 23:01:37 smithi103 systemd[1]: Stopping Ceph mds.b for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:01:37.858 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 23:01:37 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-b[95936]: 2024-03-18T23:01:37.382+0000 7f5df4c6a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:01:37.859 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 23:01:37 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-b[95936]: 2024-03-18T23:01:37.382+0000 7f5df4c6a700 -1 mds.b *** got signal Terminated *** 2024-03-18T23:01:40.109 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 23:01:39 smithi103 podman[102166]: 2024-03-18 23:01:39.801429642 +0000 UTC m=+2.504567919 container died 670bfc4263edee67d786551cbece1203495faad959ff7fe53e39f9cad9c2ff69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-b, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-18T23:01:40.443 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 23:01:40 smithi103 podman[102166]: 2024-03-18 23:01:40.225595333 +0000 UTC m=+2.928733603 container cleanup 670bfc4263edee67d786551cbece1203495faad959ff7fe53e39f9cad9c2ff69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5) 2024-03-18T23:01:40.443 INFO:journalctl@ceph.mds.b.smithi103.stdout:Mar 18 23:01:40 smithi103 bash[102166]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-b 2024-03-18T23:01:40.652 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.b.service' 2024-03-18T23:01:40.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:01:40.688 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-18T23:01:40.688 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-18T23:01:40.688 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.e 2024-03-18T23:01:41.013 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 23:01:40 smithi103 systemd[1]: Stopping Ceph mds.e for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:01:41.358 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 23:01:41 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-e[95315]: 2024-03-18T23:01:41.011+0000 7f4f49e03700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:01:41.358 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 23:01:41 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-e[95315]: 2024-03-18T23:01:41.012+0000 7f4f49e03700 -1 mds.e *** got signal Terminated *** 2024-03-18T23:01:44.072 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 23:01:43 smithi103 podman[102288]: 2024-03-18 23:01:43.807655299 +0000 UTC m=+2.896267781 container died 2ede6c30917ebeebc014eacdafb23849a8dd1c76b65c462a1cc9a8bd8a8db610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-e, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307) 2024-03-18T23:01:44.358 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 23:01:44 smithi103 podman[102288]: 2024-03-18 23:01:44.171435238 +0000 UTC m=+3.260047727 container cleanup 2ede6c30917ebeebc014eacdafb23849a8dd1c76b65c462a1cc9a8bd8a8db610 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-03-18T23:01:44.359 INFO:journalctl@ceph.mds.e.smithi103.stdout:Mar 18 23:01:44 smithi103 bash[102288]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-e 2024-03-18T23:01:44.552 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.e.service' 2024-03-18T23:01:44.590 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:01:44.590 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-18T23:01:44.590 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-18T23:01:44.590 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.h 2024-03-18T23:01:44.858 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 23:01:44 smithi103 systemd[1]: Stopping Ceph mds.h for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:01:45.358 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 23:01:44 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-h[95619]: 2024-03-18T23:01:44.916+0000 7fb442212700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:01:45.359 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 23:01:44 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-h[95619]: 2024-03-18T23:01:44.916+0000 7fb442212700 -1 mds.h *** got signal Terminated *** 2024-03-18T23:01:48.085 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 23:01:47 smithi103 podman[102422]: 2024-03-18 23:01:47.818039891 +0000 UTC m=+3.008164941 container died ae455a62f9386640c6a1a3ce012ffbf8d20686a4482703c2531b435991845573 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-h, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T23:01:48.359 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 23:01:48 smithi103 podman[102422]: 2024-03-18 23:01:48.16766119 +0000 UTC m=+3.357786231 container cleanup ae455a62f9386640c6a1a3ce012ffbf8d20686a4482703c2531b435991845573 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-h, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0) 2024-03-18T23:01:48.359 INFO:journalctl@ceph.mds.h.smithi103.stdout:Mar 18 23:01:48 smithi103 bash[102422]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-h 2024-03-18T23:01:48.538 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.h.service' 2024-03-18T23:01:48.574 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:01:48.574 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-18T23:01:48.574 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-18T23:01:48.575 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.k 2024-03-18T23:01:48.862 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 23:01:48 smithi103 systemd[1]: Stopping Ceph mds.k for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:01:49.359 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 23:01:48 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-k[94994]: 2024-03-18T23:01:48.904+0000 7fed4cb87700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:01:49.359 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 23:01:48 smithi103 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-k[94994]: 2024-03-18T23:01:48.904+0000 7fed4cb87700 -1 mds.k *** got signal Terminated *** 2024-03-18T23:01:52.080 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 23:01:51 smithi103 podman[102543]: 2024-03-18 23:01:51.798835069 +0000 UTC m=+3.005576275 container died 3acf8ab35b6bf453877ccb8a1d855dd8ba418707a3a7734438e8bd6cf81ada55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-k, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-18T23:01:52.358 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 23:01:52 smithi103 podman[102543]: 2024-03-18 23:01:52.163434281 +0000 UTC m=+3.370175495 container cleanup 3acf8ab35b6bf453877ccb8a1d855dd8ba418707a3a7734438e8bd6cf81ada55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-18T23:01:52.359 INFO:journalctl@ceph.mds.k.smithi103.stdout:Mar 18 23:01:52 smithi103 bash[102543]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-k 2024-03-18T23:01:52.525 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.k.service' 2024-03-18T23:01:52.559 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:01:52.560 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-18T23:01:52.560 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-18T23:01:52.560 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.c 2024-03-18T23:01:52.898 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 18 23:01:52 smithi113 systemd[1]: Stopping Ceph mds.c for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:01:53.155 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 18 23:01:52 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-c[98113]: 2024-03-18T23:01:52.897+0000 7f1224a05700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:01:53.155 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 18 23:01:52 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-c[98113]: 2024-03-18T23:01:52.897+0000 7f1224a05700 -1 mds.c *** got signal Terminated *** 2024-03-18T23:01:56.090 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 18 23:01:55 smithi113 podman[105402]: 2024-03-18 23:01:55.82916111 +0000 UTC m=+3.066186149 container died 6830d5373e1363809acf1a7c52e637b9699fdedbac8558f95ec738310e4c4df0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-c, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-18T23:01:56.464 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 18 23:01:56 smithi113 podman[105402]: 2024-03-18 23:01:56.179991794 +0000 UTC m=+3.417016825 container cleanup 6830d5373e1363809acf1a7c52e637b9699fdedbac8558f95ec738310e4c4df0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307) 2024-03-18T23:01:56.464 INFO:journalctl@ceph.mds.c.smithi113.stdout:Mar 18 23:01:56 smithi113 bash[105402]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-c 2024-03-18T23:01:56.609 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.c.service' 2024-03-18T23:01:56.648 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:01:56.648 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-18T23:01:56.648 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-18T23:01:56.648 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.f 2024-03-18T23:01:57.079 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 23:01:56 smithi113 systemd[1]: Stopping Ceph mds.f for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:01:57.079 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 23:01:56 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-f[97804]: 2024-03-18T23:01:56.974+0000 7f4f41abd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:01:57.079 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 23:01:56 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-f[97804]: 2024-03-18T23:01:56.974+0000 7f4f41abd700 -1 mds.f *** got signal Terminated *** 2024-03-18T23:02:00.116 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 23:01:59 smithi113 podman[105536]: 2024-03-18 23:01:59.830377426 +0000 UTC m=+2.964042057 container died a6d4038b9ceab36e5f3c0c59a48034e5fbc496c5a036e17f7da718cefce343e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-f, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T23:02:00.116 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 23:02:00 smithi113 podman[105536]: 2024-03-18 23:02:00.116821479 +0000 UTC m=+3.250486119 container cleanup a6d4038b9ceab36e5f3c0c59a48034e5fbc496c5a036e17f7da718cefce343e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-18T23:02:00.376 INFO:journalctl@ceph.mds.f.smithi113.stdout:Mar 18 23:02:00 smithi113 bash[105536]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-f 2024-03-18T23:02:00.531 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.f.service' 2024-03-18T23:02:00.566 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:02:00.567 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-18T23:02:00.567 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-18T23:02:00.567 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.i 2024-03-18T23:02:00.887 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 23:02:00 smithi113 systemd[1]: Stopping Ceph mds.i for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:02:01.329 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 23:02:00 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-i[98437]: 2024-03-18T23:02:00.887+0000 7f4f176b7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:02:01.330 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 23:02:00 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-i[98437]: 2024-03-18T23:02:00.887+0000 7f4f176b7700 -1 mds.i *** got signal Terminated *** 2024-03-18T23:02:04.079 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 23:02:03 smithi113 podman[105656]: 2024-03-18 23:02:03.827472023 +0000 UTC m=+3.044147568 container died 124c2cdc599d40bbf23c3b1922404257e247b2687696f9df684e7504eea43be5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-i, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T23:02:04.396 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 23:02:04 smithi113 podman[105656]: 2024-03-18 23:02:04.187201616 +0000 UTC m=+3.403877156 container cleanup 124c2cdc599d40bbf23c3b1922404257e247b2687696f9df684e7504eea43be5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-i, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T23:02:04.396 INFO:journalctl@ceph.mds.i.smithi113.stdout:Mar 18 23:02:04 smithi113 bash[105656]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-i 2024-03-18T23:02:04.600 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.i.service' 2024-03-18T23:02:04.634 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:02:04.634 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-18T23:02:04.634 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-18T23:02:04.635 DEBUG:teuthology.orchestra.run.smithi113:> sudo systemctl stop ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.l 2024-03-18T23:02:04.949 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 23:02:04 smithi113 systemd[1]: Stopping Ceph mds.l for 410ef414-e577-11ee-95c9-87774f69a715... 2024-03-18T23:02:05.329 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 23:02:04 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-l[97489]: 2024-03-18T23:02:04.948+0000 7f876f23c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T23:02:05.329 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 23:02:04 smithi113 ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-l[97489]: 2024-03-18T23:02:04.948+0000 7f876f23c700 -1 mds.l *** got signal Terminated *** 2024-03-18T23:02:08.129 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 23:02:07 smithi113 podman[105776]: 2024-03-18 23:02:07.850812926 +0000 UTC m=+2.995673804 container died 382717cf9b9cc87d190d47aebb02266367fdc1344d08baa55c8cee14f5a1434d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-l, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-18T23:02:08.579 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 23:02:08 smithi113 podman[105776]: 2024-03-18 23:02:08.190491145 +0000 UTC m=+3.335352030 container cleanup 382717cf9b9cc87d190d47aebb02266367fdc1344d08baa55c8cee14f5a1434d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-l, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T23:02:08.579 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 23:02:08 smithi113 bash[105776]: ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-l 2024-03-18T23:02:08.579 INFO:journalctl@ceph.mds.l.smithi113.stdout:Mar 18 23:02:08 smithi113 podman[105791]: 2024-03-18 23:02:08.366432823 +0000 UTC m=+0.518105657 container remove 382717cf9b9cc87d190d47aebb02266367fdc1344d08baa55c8cee14f5a1434d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-410ef414-e577-11ee-95c9-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-18T23:02:08.618 DEBUG:teuthology.orchestra.run.smithi113:> sudo pkill -f 'journalctl -f -n 0 -u ceph-410ef414-e577-11ee-95c9-87774f69a715@mds.l.service' 2024-03-18T23:02:08.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T23:02:08.651 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-18T23:02:08.652 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 410ef414-e577-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T23:02:08.850 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T23:02:11.932 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 410ef414-e577-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T23:02:12.124 INFO:teuthology.orchestra.run.smithi103.stdout:Deleting cluster with fsid: 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T23:02:15.391 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 410ef414-e577-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T23:02:15.758 INFO:teuthology.orchestra.run.smithi113.stdout:Deleting cluster with fsid: 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T23:02:18.751 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T23:02:18.786 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T23:02:18.821 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T23:02:18.850 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-18T23:02:18.852 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143/remote/smithi040/crash 2024-03-18T23:02:18.853 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/crash -- . 2024-03-18T23:02:18.906 INFO:teuthology.orchestra.run.smithi040.stderr:tar: /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T23:02:18.906 INFO:teuthology.orchestra.run.smithi040.stderr:tar: Error is not recoverable: exiting now 2024-03-18T23:02:18.908 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143/remote/smithi103/crash 2024-03-18T23:02:18.909 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/crash -- . 2024-03-18T23:02:19.036 INFO:teuthology.orchestra.run.smithi103.stderr:tar: /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T23:02:19.036 INFO:teuthology.orchestra.run.smithi103.stderr:tar: Error is not recoverable: exiting now 2024-03-18T23:02:19.038 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143/remote/smithi113/crash 2024-03-18T23:02:19.039 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/crash -- . 2024-03-18T23:02:19.088 INFO:teuthology.orchestra.run.smithi113.stderr:tar: /var/lib/ceph/410ef414-e577-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T23:02:19.089 INFO:teuthology.orchestra.run.smithi113.stderr:tar: Error is not recoverable: exiting now 2024-03-18T23:02:19.090 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-18T23:02:19.090 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T23:02:19.172 INFO:teuthology.orchestra.run.smithi040.stdout:2024-03-18T22:34:41.931759+0000 mon.a (mon.0) 207 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T23:02:19.173 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-18T23:02:19.173 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[SEC\]' /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T23:02:19.209 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]' /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T23:02:19.285 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[WRN\]' /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T23:02:19.360 INFO:teuthology.orchestra.run.smithi040.stdout:2024-03-18T22:34:41.931759+0000 mon.a (mon.0) 207 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T23:02:19.360 INFO:tasks.cephadm:Compressing logs... 2024-03-18T23:02:19.360 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T23:02:19.404 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T23:02:19.406 DEBUG:teuthology.orchestra.run.smithi113:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T23:02:19.447 INFO:teuthology.orchestra.run.smithi040.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T23:02:19.447 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-18T23:02:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-18T23:02:19.455 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T23:02:19.455 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-18T23:02:19.455 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-18T23:02:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-18T23:02:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-18T23:02:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-18T23:02:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T23:02:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.log 2024-03-18T23:02:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T23:02:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-18T23:02:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-18T23:02:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-18T23:02:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T23:02:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-03-18T23:02:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-18T23:02:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.log: /var/log/ceph/ceph-client.admin.log: 87.3% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T23:02:19.464 INFO:teuthology.orchestra.run.smithi103.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T23:02:19.464 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-18T23:02:19.477 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T23:02:19.477 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-18T23:02:19.477 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-18T23:02:19.477 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-18T23:02:19.478 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-18T23:02:19.478 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-18T23:02:19.478 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-18T23:02:19.478 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-18T23:02:19.478 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-18T23:02:19.478 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T23:02:19.478 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.log 2024-03-18T23:02:19.479 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T23:02:19.479 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-18T23:02:19.480 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T23:02:19.497 INFO:teuthology.orchestra.run.smithi113.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T23:02:19.497 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-18T23:02:19.507 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mon.b.log: /var/log/ceph/cephadm.log: 82.7% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T23:02:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T23:02:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-18T23:02:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-18T23:02:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-18T23:02:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-18T23:02:19.513 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-18T23:02:19.514 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-18T23:02:19.514 INFO:teuthology.orchestra.run.smithi113.stderr:gzip -5/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.5.log: --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T23:02:19.514 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.log 2024-03-18T23:02:19.514 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-18T23:02:19.514 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-18T23:02:19.514 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T23:02:19.514 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-18T23:02:19.514 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T23:02:19.523 INFO:teuthology.orchestra.run.smithi040.stderr: 94.0% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-18T23:02:19.525 INFO:teuthology.orchestra.run.smithi040.stderr: 82.4% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T23:02:19.585 INFO:teuthology.orchestra.run.smithi103.stderr: 90.4% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T23:02:19.632 INFO:teuthology.orchestra.run.smithi040.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-18T23:02:19.641 INFO:teuthology.orchestra.run.smithi113.stderr:/var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mon.c.log: /var/log/ceph/cephadm.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T23:02:19.642 INFO:teuthology.orchestra.run.smithi113.stderr: 82.6% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T23:02:19.669 INFO:teuthology.orchestra.run.smithi113.stderr: 87.3% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T23:02:19.681 INFO:teuthology.orchestra.run.smithi103.stderr: 93.0% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-18T23:02:19.694 INFO:teuthology.orchestra.run.smithi113.stderr: 90.4% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T23:02:19.694 INFO:teuthology.orchestra.run.smithi103.stderr: 91.0% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T23:02:19.695 INFO:teuthology.orchestra.run.smithi113.stderr: 93.1% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-18T23:02:19.696 INFO:teuthology.orchestra.run.smithi113.stderr: 94.1% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-18T23:02:19.715 INFO:teuthology.orchestra.run.smithi103.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T23:02:19.717 INFO:teuthology.orchestra.run.smithi040.stderr: 90.2% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T23:02:19.755 INFO:teuthology.orchestra.run.smithi103.stderr: 87.3% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T23:02:19.763 INFO:teuthology.orchestra.run.smithi103.stderr: 93.0% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-18T23:02:19.768 INFO:teuthology.orchestra.run.smithi040.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T23:02:19.834 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T23:02:19.862 INFO:teuthology.orchestra.run.smithi040.stderr: 94.7% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-18T23:02:19.872 INFO:teuthology.orchestra.run.smithi040.stderr: 94.5% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-18T23:02:19.953 INFO:teuthology.orchestra.run.smithi113.stderr: 91.0% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T23:02:19.957 INFO:teuthology.orchestra.run.smithi103.stderr: 93.1% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-18T23:02:20.737 INFO:teuthology.orchestra.run.smithi103.stderr: 94.0% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-18T23:02:21.605 INFO:teuthology.orchestra.run.smithi040.stderr: 94.0% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-18T23:02:21.858 INFO:teuthology.orchestra.run.smithi113.stderr: 94.7% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-18T23:02:21.861 INFO:teuthology.orchestra.run.smithi040.stderr: 90.3% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-18T23:02:22.806 INFO:teuthology.orchestra.run.smithi113.stderr: 92.9% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-18T23:02:23.110 INFO:teuthology.orchestra.run.smithi103.stderr: 91.9% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-18T23:02:24.725 INFO:teuthology.orchestra.run.smithi113.stderr: 91.9% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-18T23:02:25.828 INFO:teuthology.orchestra.run.smithi113.stderr: 93.4% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-18T23:02:30.740 INFO:teuthology.orchestra.run.smithi040.stderr: 90.4% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-18T23:02:45.218 INFO:teuthology.orchestra.run.smithi103.stderr: 93.7% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-18T23:03:14.427 INFO:teuthology.orchestra.run.smithi103.stderr: 91.7% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-18T23:03:32.855 INFO:teuthology.orchestra.run.smithi113.stderr: 91.6% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-18T23:05:19.954 INFO:teuthology.orchestra.run.smithi103.stderr: 91.7% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-18T23:05:37.294 INFO:teuthology.orchestra.run.smithi113.stderr: 91.1% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-18T23:06:20.335 INFO:teuthology.orchestra.run.smithi103.stderr: 91.6% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-18T23:06:50.816 INFO:teuthology.orchestra.run.smithi103.stderr: 91.3% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-18T23:06:50.819 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-03-18T23:06:50.819 INFO:teuthology.orchestra.run.smithi103.stderr:real 4m31.398s 2024-03-18T23:06:50.819 INFO:teuthology.orchestra.run.smithi103.stderr:user 5m34.227s 2024-03-18T23:06:50.819 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m11.691s 2024-03-18T23:06:57.915 INFO:teuthology.orchestra.run.smithi113.stderr: 91.5% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-18T23:07:15.692 INFO:teuthology.orchestra.run.smithi113.stderr: 91.6% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-18T23:07:15.694 INFO:teuthology.orchestra.run.smithi113.stderr: 2024-03-18T23:07:15.695 INFO:teuthology.orchestra.run.smithi113.stderr:real 4m56.272s 2024-03-18T23:07:15.695 INFO:teuthology.orchestra.run.smithi113.stderr:user 6m5.165s 2024-03-18T23:07:15.695 INFO:teuthology.orchestra.run.smithi113.stderr:sys 0m13.319s 2024-03-18T23:17:12.238 INFO:teuthology.orchestra.run.smithi040.stderr: 91.6% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-18T23:21:56.037 INFO:teuthology.orchestra.run.smithi040.stderr: 91.5% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-18T23:22:27.789 INFO:teuthology.orchestra.run.smithi040.stderr: 90.6% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-18T23:23:57.560 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/410ef414-e577-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-18T23:25:40.698 INFO:teuthology.orchestra.run.smithi040.stderr: 96.4% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-03-18T23:25:40.700 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-03-18T23:25:40.700 INFO:teuthology.orchestra.run.smithi040.stderr:real 23m21.286s 2024-03-18T23:25:40.700 INFO:teuthology.orchestra.run.smithi040.stderr:user 9m57.101s 2024-03-18T23:25:40.700 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m26.937s 2024-03-18T23:25:40.700 INFO:tasks.cephadm:Archiving logs... 2024-03-18T23:25:40.702 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143/remote/smithi040/log 2024-03-18T23:25:40.703 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T23:27:09.602 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143/remote/smithi103/log 2024-03-18T23:27:09.604 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T23:28:08.159 DEBUG:teuthology.misc:Transferring archived files from smithi113:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143/remote/smithi113/log 2024-03-18T23:28:08.161 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T23:29:10.930 INFO:tasks.cephadm:Removing cluster... 2024-03-18T23:29:10.930 DEBUG:teuthology.orchestra.run.smithi040:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 410ef414-e577-11ee-95c9-87774f69a715 --force 2024-03-18T23:29:11.432 INFO:teuthology.orchestra.run.smithi040.stdout:Deleting cluster with fsid: 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T23:29:12.181 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 410ef414-e577-11ee-95c9-87774f69a715 --force 2024-03-18T23:29:12.376 INFO:teuthology.orchestra.run.smithi103.stdout:Deleting cluster with fsid: 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T23:29:13.086 DEBUG:teuthology.orchestra.run.smithi113:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 410ef414-e577-11ee-95c9-87774f69a715 --force 2024-03-18T23:29:13.277 INFO:teuthology.orchestra.run.smithi113.stdout:Deleting cluster with fsid: 410ef414-e577-11ee-95c9-87774f69a715 2024-03-18T23:29:14.011 INFO:tasks.cephadm:Removing cephadm ... 2024-03-18T23:29:14.011 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T23:29:14.060 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T23:29:14.077 DEBUG:teuthology.orchestra.run.smithi113:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T23:29:14.092 INFO:tasks.cephadm:Teardown complete 2024-03-18T23:29:14.093 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-18T23:29:14.103 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-18T23:29:14.103 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T23:29:14.106 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T23:29:14.120 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T23:29:14.238 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-18T23:29:14.252 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-18T23:29:14.252 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T23:29:14.255 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T23:29:14.257 DEBUG:teuthology.orchestra.run.smithi113:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T23:29:14.270 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-03-18T23:29:14.271 INFO:teuthology.orchestra.run.smithi113.stderr:bash: line 1: ntpq: command not found 2024-03-18T23:29:14.288 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-03-18T23:29:14.294 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T23:29:14.294 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-03-18T23:29:14.294 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 +1322us[+1322us] +/- 126ms 2024-03-18T23:29:14.294 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv02.front.sepia.ceph.com 3 6 377 26 -128us[ -113us] +/- 59ms 2024-03-18T23:29:14.294 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -558us[ -544us] +/- 67ms 2024-03-18T23:29:14.294 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T23:29:14.297 INFO:teuthology.orchestra.run.smithi113.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T23:29:14.298 INFO:teuthology.orchestra.run.smithi113.stdout:=============================================================================== 2024-03-18T23:29:14.298 INFO:teuthology.orchestra.run.smithi113.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 +1398us[+1399us] +/- 126ms 2024-03-18T23:29:14.298 INFO:teuthology.orchestra.run.smithi113.stdout:^* hv02.front.sepia.ceph.com 3 6 377 26 -118us[ -117us] +/- 59ms 2024-03-18T23:29:14.298 INFO:teuthology.orchestra.run.smithi113.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 -547us[ -546us] +/- 67ms 2024-03-18T23:29:14.298 INFO:teuthology.orchestra.run.smithi113.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T23:29:14.328 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T23:29:14.328 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-03-18T23:29:14.328 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +1317us[+1317us] +/- 127ms 2024-03-18T23:29:14.329 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -129us[ -137us] +/- 59ms 2024-03-18T23:29:14.329 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 -556us[ -564us] +/- 67ms 2024-03-18T23:29:14.329 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T23:29:14.330 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-18T23:29:14.340 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-18T23:29:14.341 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-18T23:29:14.357 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T23:29:14.540 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T23:29:14.696 DEBUG:teuthology.orchestra.run.smithi113:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T23:29:14.874 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T23:29:14.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T23:29:14.909 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T23:29:14.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T23:29:14.942 DEBUG:teuthology.orchestra.run.smithi113:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T23:29:14.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T23:29:14.972 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-18T23:29:14.983 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-18T23:29:15.016 INFO:teuthology.task.internal:Duration was 5148.246403 seconds 2024-03-18T23:29:15.016 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-18T23:29:15.049 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-18T23:29:15.049 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T23:29:15.052 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T23:29:15.054 DEBUG:teuthology.orchestra.run.smithi113:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T23:29:15.119 INFO:teuthology.orchestra.run.smithi113.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T23:29:15.129 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T23:29:15.168 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T23:29:15.666 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-18T23:29:15.667 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2024-03-18T23:29:15.667 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T23:29:15.699 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-03-18T23:29:15.699 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T23:29:15.727 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi113.front.sepia.ceph.com 2024-03-18T23:29:15.727 DEBUG:teuthology.orchestra.run.smithi113:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T23:29:15.754 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-18T23:29:15.754 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T23:29:15.757 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T23:29:15.770 DEBUG:teuthology.orchestra.run.smithi113:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T23:29:15.823 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-18T23:29:15.823 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T23:29:15.841 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T23:29:15.844 DEBUG:teuthology.orchestra.run.smithi113:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T23:29:16.253 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-18T23:29:16.263 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-18T23:29:16.264 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T23:29:16.291 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T23:29:16.319 DEBUG:teuthology.orchestra.run.smithi113:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T23:29:16.347 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-18T23:29:16.358 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T23:29:16.360 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T23:29:16.362 DEBUG:teuthology.orchestra.run.smithi113:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T23:29:16.388 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-03-18T23:29:16.415 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2024-03-18T23:29:16.415 INFO:teuthology.orchestra.run.smithi113.stdout:kernel.core_pattern = core 2024-03-18T23:29:16.444 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T23:29:16.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T23:29:16.457 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T23:29:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T23:29:16.474 DEBUG:teuthology.orchestra.run.smithi113:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T23:29:16.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T23:29:16.498 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-18T23:29:16.509 INFO:teuthology.task.internal:Transferring archived files... 2024-03-18T23:29:16.510 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143/remote/smithi040 2024-03-18T23:29:16.510 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T23:29:16.557 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143/remote/smithi103 2024-03-18T23:29:16.557 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T23:29:16.596 DEBUG:teuthology.misc:Transferring archived files from smithi113:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143/remote/smithi113 2024-03-18T23:29:16.597 DEBUG:teuthology.orchestra.run.smithi113:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T23:29:16.639 INFO:teuthology.task.internal:Removing archive directory... 2024-03-18T23:29:16.639 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T23:29:16.642 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T23:29:16.644 DEBUG:teuthology.orchestra.run.smithi113:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T23:29:16.686 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-18T23:29:16.700 INFO:teuthology.task.internal:Not uploading archives. 2024-03-18T23:29:16.700 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-18T23:29:16.725 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-18T23:29:16.725 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T23:29:16.727 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T23:29:16.729 DEBUG:teuthology.orchestra.run.smithi113:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T23:29:16.739 INFO:teuthology.orchestra.run.smithi040.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 23:29 /home/ubuntu/cephtest 2024-03-18T23:29:16.741 INFO:teuthology.orchestra.run.smithi103.stdout: 265311 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 23:29 /home/ubuntu/cephtest 2024-03-18T23:29:16.744 INFO:teuthology.orchestra.run.smithi113.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 23:29 /home/ubuntu/cephtest 2024-03-18T23:29:16.754 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-18T23:29:16.766 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-18T23:29:16.825 INFO:teuthology.nuke:Checking targets against current locks 2024-03-18T23:29:16.852 DEBUG:teuthology.nuke:shortname: smithi040 2024-03-18T23:29:16.852 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T23:29:16.877 DEBUG:teuthology.nuke:shortname: smithi103 2024-03-18T23:29:16.877 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T23:29:16.881 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 21:34:47.270026', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN23q79valGzZSCezmtu3H5G8OKoj+SkFhT8hOd86nc1Zovg3u2otRf4t8Mn3Ftf38CLYEaa5077x/XGvfhx+Sw='} 2024-03-18T23:29:16.900 DEBUG:teuthology.nuke:shortname: smithi113 2024-03-18T23:29:16.900 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T23:29:16.904 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 21:34:47.271578', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMmiz9uEAhVVpzeDVbCeY00XNBRR3ChEsH59yC2w7169XrawDvSiIBvRQCrtYE05HBaPvA1KDu7DHgxdfJydLm0='} 2024-03-18T23:29:16.927 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi113.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609143', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 21:34:47.268353', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFwzKdcwBBvp3d0PPUob2oMiEIIKsMpovmEzwap4MfAZGl6sa4edg2xdBAVqXtWC4dtzB5GC+YIsL6fdlM8lrew='} 2024-03-18T23:29:16.936 INFO:teuthology.orchestra.console.smithi040:Power off 2024-03-18T23:29:16.936 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T23:29:16.950 INFO:teuthology.orchestra.console.smithi103:Power off 2024-03-18T23:29:16.951 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T23:29:16.964 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2024-03-18T23:29:16.964 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T23:29:16.973 DEBUG:teuthology.orchestra.console.smithi103:power off output: Chassis Power Control: Down/Off 2024-03-18T23:29:16.974 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T23:29:16.990 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-03-18T23:29:16.998 INFO:teuthology.orchestra.console.smithi113:Power off 2024-03-18T23:29:16.998 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T23:29:17.007 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-03-18T23:29:17.022 DEBUG:teuthology.orchestra.console.smithi113:power off output: Chassis Power Control: Down/Off 2024-03-18T23:29:17.022 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T23:29:17.043 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2024-03-18T23:29:20.992 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T23:29:21.007 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T23:29:21.045 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T23:29:21.101 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-03-18T23:29:21.117 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-03-18T23:29:21.153 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2024-03-18T23:29:25.105 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T23:29:25.118 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T23:29:25.154 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T23:29:25.214 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-03-18T23:29:25.227 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-03-18T23:29:25.264 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is on 2024-03-18T23:29:25.315 INFO:teuthology.orchestra.console.smithi040:Power off completed 2024-03-18T23:29:29.232 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T23:29:29.265 DEBUG:teuthology.orchestra.console.smithi113:pexpect command: ipmitool -H smithi113.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T23:29:29.343 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is off 2024-03-18T23:29:29.375 DEBUG:teuthology.orchestra.console.smithi113:check power output: Chassis Power is off 2024-03-18T23:29:29.445 INFO:teuthology.orchestra.console.smithi103:Power off completed 2024-03-18T23:29:29.476 INFO:teuthology.orchestra.console.smithi113:Power off completed 2024-03-18T23:29:29.582 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} duration: 5148.246403217316 failure_reason: '"2024-03-18T22:34:41.931759+0000 mon.a (mon.0) 207 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-03-18T23:29:29.582 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T23:29:29.666 INFO:teuthology.run:FAIL